动态 LLM 路由器往往选错了方向

# Dynamic LLM Routers are Often Misguided
Source: [https://arxiv.org/html/2610.02762](https://arxiv.org/html/2610.02762)
###### Abstract

Dynamic LLM routers promise to cut inference costs by sending each query to the cheapest model that can answer it correctly\. We analyze six commercial routers across 14 settings on a diverse benchmark spanning eight task categories, finding that none of them outperforms a router that randomly selects between two well\-chosen models at matched cost\. Some underperform by more than 10 percentage points\. We trace this gap to four patterns prevalent across routers: difficulty blindness, length reversal, semantic matching, and roster suboptimality\. We show that the first three are what the standard objective rewards: cost–accuracy Pareto efficiency on realized costs favors escalating moderately hard queries over the hardest ones, shorter queries over longer ones, and routing by a query’s source over its difficulty\. We also argue that the two assumptions that would justify large rosters, model granularity and model specialization, do not hold empirically\. We propose an alternative evaluation methodology that does not reward these patterns, and as a proof of concept, we design a simple two\-model router that avoids all four\. Nevertheless, its gain over random routing is limited, because a well\-chosen roster leaves little to route\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2610.02762v1/intro_pitfalls.png)Figure 1:Four patterns of current LLM routers\. \(a\)*Difficulty blindness:*a strong model’s \(Opus 5 \(high\)\) advantage over a weak one \(gpt\-oss\-20b \(low\)\) peaks at moderately hard queries and shrinks at the expensive end, so routers under\-escalate hard queries\. \(b\)*Length reversal:*routing by difficulty \(blue\) spends more on queries with longer answers, while some commercial routers \(orange\) spend less\. \(c\)*Semantic matching:*routers often send most of each data source to one model regardless of query difficulty \(OpenRouter auto \(medium\) is shown as a representative example\)\. \(d\)*Roster suboptimality:*only a small number of models are at the cost\-accuracy frontier, yet many commercial routers spread queries across many non\-frontier models\.The central premise of dynamic LLM routing is that the computational expense of inference should scale with query complexity, sending each query to the cheapest model capable of answering it correctly\. While some routing architectures attempt to surpass standalone performance by exploiting model specializations\([Liu et al\., 2026](https://arxiv.org/html/2610.02762#bib.bib36);[Tang et al\., 2026](https://arxiv.org/html/2610.02762#bib.bib54)\), the primary goal of routing is cost reduction\. We focus on the simplest of these cases: a router for single\-turn, general\-purpose user traffic that achieves near\-frontier performance at a substantial discount\.

Despite the intuitive logic of dynamic routing, studies have consistently revealed that learned routers often fail to outperform simple baselines\([Hu et al\., 2024](https://arxiv.org/html/2610.02762#bib.bib19);[Lu et al\., 2026b](https://arxiv.org/html/2610.02762#bib.bib55);[Li et al\., 2026](https://arxiv.org/html/2610.02762#bib.bib44)\)\. Prior work has noted three of the four patterns we study: routers route without regard to query difficulty\([Lu et al\., 2026b](https://arxiv.org/html/2610.02762#bib.bib55)\)\(*difficulty blindness*\), collapse each domain onto a single model\([Li et al\., 2026](https://arxiv.org/html/2610.02762#bib.bib44)\)\(*semantic matching*\), and use rosters that are too large\([Li et al\., 2026](https://arxiv.org/html/2610.02762#bib.bib44)\)\(*roster suboptimality*\)\. We identify a fourth,*length reversal*: routers prefer to escalate queries with short answers, even though these tend to be easier\.

Our contribution is to explain these patterns\. Three of the four are what the conventional evaluation objective rewards\. The standard benchmarks\([Hu et al\., 2024](https://arxiv.org/html/2610.02762#bib.bib19);[Lu et al\., 2026a](https://arxiv.org/html/2610.02762#bib.bib20);[Li et al\., 2026](https://arxiv.org/html/2610.02762#bib.bib44)\)assess routers for Pareto efficiency on cost and accuracy\. However, a router that maximizes this objective escalates a query when an expensive model is much more likely than the cheap one to answer it, and this gap is small on the hardest queries since both models likely fail\. A Pareto\-efficient router therefore has little reason to escalate the most difficult queries \(difficulty blindness\)\. In addition, when answer lengths vary, the same router prefers to escalate queries with shorter expected answers, because escalating them costs less \(length reversal\)\. If the evaluation dataset includes many diverse benchmarks of varying difficulty, a router can perform well on evaluations just by predicting the source identity \(semantic matching\)\. Answer length and source are also easier to estimate than difficulty, and under this objective they score just as well or better, so routers have little reason to route on difficulty at all\. In its stead, we propose an alternative router evaluation methodology that does not reward these behaviors\. We attribute the fourth pattern, roster suboptimality, to two assumptions, that intermediate models fill useful price points and that models specialize, and show that neither holds up empirically\. We find that a roster of three \(and, in most cases, just two\) models closely approximates the best achievable accuracy of routers that send harder queries to strong models \(§[6](https://arxiv.org/html/2610.02762#S6)\)\.

Finally, we show that a router can avoid all four patterns by directing difficult queries to the stronger model, but on our model set this offers no accuracy gain over random routing at matched cost, because a well\-chosen roster leaves little to route\.

## 2Related Work

#### Dynamic routing\.

Many routers train a supervised predictor for a model’s answer quality or correctness\([Ding et al\., 2024](https://arxiv.org/html/2610.02762#bib.bib34);[Lu et al\., 2024](https://arxiv.org/html/2610.02762#bib.bib35);[Chen et al\., 2024](https://arxiv.org/html/2610.02762#bib.bib41);[Feng et al\., 2025](https://arxiv.org/html/2610.02762#bib.bib42);[Zhuang et al\., 2025](https://arxiv.org/html/2610.02762#bib.bib43);[Varshney et al\., 2026](https://arxiv.org/html/2610.02762#bib.bib31);[Bao et al\., 2026](https://arxiv.org/html/2610.02762#bib.bib21)\), while others route on human pairwise preferences\([Ong et al\., 2025](https://arxiv.org/html/2610.02762#bib.bib33)\)\. Semantic routers use BERT\-style embedders to classify queries into clusters or categories, assigning each cluster to a single model\([Jitkrittum et al\., 2026](https://arxiv.org/html/2610.02762#bib.bib39);[Zhang et al\., 2025a](https://arxiv.org/html/2610.02762#bib.bib38);[Zhang et al\., 2025b](https://arxiv.org/html/2610.02762#bib.bib37);[Liu et al\., 2026](https://arxiv.org/html/2610.02762#bib.bib36)\)\.

#### Router evaluation\.

RouterBench\([Hu et al\., 2024](https://arxiv.org/html/2610.02762#bib.bib19)\), LLMRouterBench\([Li et al\., 2026](https://arxiv.org/html/2610.02762#bib.bib44)\), RouterArena\([Lu et al\., 2026a](https://arxiv.org/html/2610.02762#bib.bib20)\), and TwinRouterBench\([Yang et al\., 2026](https://arxiv.org/html/2610.02762#bib.bib45)\)all evaluate routers by cost\-accuracy Pareto efficiency using realized generation costs\. We argue that this objective rewards difficulty blindness, length reversal, and semantic matching\.

#### Router patterns\.

RouterBench\([Hu et al\., 2024](https://arxiv.org/html/2610.02762#bib.bib19)\)found that none of its learned routers outperform a “zero router” that randomly interpolates between Pareto\-efficient models, which is also the baseline we adopt\.[Lu et al\. \(2026b\)](https://arxiv.org/html/2610.02762#bib.bib55)find that routers learn coarse rather than query\-specific signals, and[Li et al\. \(2026\)](https://arxiv.org/html/2610.02762#bib.bib44)that routers do little better than sending each dataset to its best model and that larger rosters show diminishing returns\. We find that these patterns persist in commercial routers\.

#### Difficulty and capability\.

Item response theory \(IRT\) describes pass rate as a function of a student \(model\)’s latent ability and an item \(query\)’s characteristics\([Lord, 1952](https://arxiv.org/html/2610.02762#bib.bib48);[Rasch, 1960](https://arxiv.org/html/2610.02762#bib.bib49);[Birnbaum, 1968](https://arxiv.org/html/2610.02762#bib.bib50);[Reckase, 2009](https://arxiv.org/html/2610.02762#bib.bib51)\)\. It has been used to estimate full\-benchmark scores from a small subset of items\([Polo et al\., 2024](https://arxiv.org/html/2610.02762#bib.bib52);[Kipnis et al\., 2025](https://arxiv.org/html/2610.02762#bib.bib53)\), to place models on a single capability scale\([Ho et al\., 2025](https://arxiv.org/html/2610.02762#bib.bib26)\), and to predict pass rates for routing\([Song et al\., 2025](https://arxiv.org/html/2610.02762#bib.bib25)\)\. We use the Rasch \(one\-parameter\) model to define model capabilities and query difficulties, and the multidimensional extension to test for specialization\. Prior evidence suggests that LLM performance across benchmarks is low\- or one\-dimensional\([Burnell et al\., 2023](https://arxiv.org/html/2610.02762#bib.bib47);[Ruan et al\., 2024](https://arxiv.org/html/2610.02762#bib.bib27);[Ilić and Gignac, 2024](https://arxiv.org/html/2610.02762#bib.bib28);[Ho et al\., 2025](https://arxiv.org/html/2610.02762#bib.bib26)\), and we likewise find specialization too weak to exploit at routing time\. Finally, several papers predict difficulty or correctness from model activations\([Lee et al\., 2025](https://arxiv.org/html/2610.02762#bib.bib30);[Cencerrado et al\., 2025](https://arxiv.org/html/2610.02762#bib.bib29);[Varshney et al\., 2026](https://arxiv.org/html/2610.02762#bib.bib31);[Ghasemabadi and Niu, 2025](https://arxiv.org/html/2610.02762#bib.bib46);[Lugoloobi et al\., 2026](https://arxiv.org/html/2610.02762#bib.bib57)\), and we build our own difficulty predictor on these methods\.

## 3Setup

#### Notation\.

Letℳ\\mathcal\{M\}be a*model set*, the set of models we evaluate, and let𝒬\\mathcal\{Q\}be a set of queries\. For modeli∈ℳi\\in\\mathcal\{M\}and queryj∈𝒬j\\in\\mathcal\{Q\},yi​j∈\{0,1\}y\_\{ij\}\\in\\\{0,1\\\}is whether the model answers correctly, andci​jc\_\{ij\}is the cost of generating its answer, including both input and output tokens\. We definec¯i=1\|𝒬\|​∑jci​j\\overline\{c\}\_\{i\}=\\frac\{1\}\{\|\\mathcal\{Q\}\|\}\\sum\_\{j\}c\_\{ij\}as the model’s mean cost per query\. A*router*r:𝒬→ℛr:\\mathcal\{Q\}\\to\\mathcal\{R\}assigns each query to a model in its rosterℛ⊆ℳ\\mathcal\{R\}\\subseteq\\mathcal\{M\}, with accuracy1\|𝒬\|​∑jyr⁡\(j\),j\\frac\{1\}\{\|\\mathcal\{Q\}\|\}\\sum\_\{j\}y\_\{r\(j\),j\}and cost1\|𝒬\|​∑jcr⁡\(j\),j\\frac\{1\}\{\|\\mathcal\{Q\}\|\}\\sum\_\{j\}c\_\{r\(j\),j\}\. A*random router*over two models\{i,k\}\\\{i,k\\\}selectsiiwith probabilityppandkkotherwise, so its expected accuracy and cost arepp\-weighted averages of the two models’ values, and varyingpptraces a line segment between them\. We define a query’s*difficulty*asdj=1−1\|ℳ\|​∑iyi​jd\_\{j\}=1\-\\frac\{1\}\{\|\\mathcal\{M\}\|\}\\sum\_\{i\}y\_\{ij\}, the fraction of models that fail it\. A model’s*ability*aia\_\{i\}is its Artificial Analysis Intelligence Index score\([Artificial Analysis, 2026](https://arxiv.org/html/2610.02762#bib.bib23)\), which we only use to order models\. A query’s*expected answer length*isℓj=1\|ℳ\|​∑ini​j\\ell\_\{j\}=\\frac\{1\}\{\|\\mathcal\{M\}\|\}\\sum\_\{i\}n\_\{ij\}, whereni​jn\_\{ij\}is the number of output tokens modeliigenerates for queryjj\. Every query belongs to a*data source*gjg\_\{j\}\(Table[2](https://arxiv.org/html/2610.02762#A1.T2)\)\. We also fit a Rasch model with capabilityθi\\theta\_\{i\}per model and a difficulty per query\. The fitted difficulties induce the same ordering asdjd\_\{j\}, so with a mild abuse of notation, we writedjd\_\{j\}for them as well\. Capabilitiesθi\\theta\_\{i\}are fit from our outcomes and distinct fromaia\_\{i\}\.

#### Commercial routers\.

We analyze six existing commercial routers across 14 settings\. We evaluate commercial rather than academic routers because they can be used off the shelf and presumably draw on the best available techniques\. The developers include general platforms \(OpenRouter, Microsoft Azure, and vLLM\) and routing specialists \(Not Diamond, Nadir, and Orca\)\. For each router and query, we record the selected model then independently generate and evaluate its generation using default OpenRouter settings and costs, so all costs and accuracies are measured the same way\. Appendix[A](https://arxiv.org/html/2610.02762#A1)describes each router, the input format, and several notable omissions because they are not direct comparisons\.

#### Datasets\.

Our data comprise 17 benchmarks across eight categories \(Table[2](https://arxiv.org/html/2610.02762#A1.T2)\), with 2,508 training samples and 800 evaluation samples\. The categories are coding, instruction\-following, knowledge, math, question\-answering \(QA\), office, tool use, and Humanity’s Last Exam \(HLE\) \(a challenging benchmark to assess the frontier of model capability\), together spanning a wide sweep of LLM use cases\. Each category has 100 evaluation samples and contributes equally to evaluation scores\. All sources admit verifiable rewards except for GDPval, which provides detailed rubrics for LLM judges, and Humanity’s Last Exam, where the judge only has to compare the provided answer with the answer key\.

#### Models and evaluation\.

We generate and grade responses across 26 models \(§[A](https://arxiv.org/html/2610.02762#A1)\)\. We selected 20 of them in early June 2026 to cover the frontier at the time, and added six in mid\-August as new models were released, giving us two snapshots of the model frontier\. Unless stated otherwise, results use all 26 models\. We use Gemini 3\.7 Flash and Opus 5 \(high\) throughout this paper because the former is strong for its price point and the latter has the highest accuracy\. To ensure that our observations are not limited to this time frame, we replicate findings on RouterBench\([Hu et al\., 2024](https://arxiv.org/html/2610.02762#bib.bib19)\)\(11 models, 34,381 queries\), SPROUT\([Somerstep et al\., 2025](https://arxiv.org/html/2610.02762#bib.bib24)\)\(13 models, 44,241 queries\), and LLMRouterBench\([Li et al\., 2026](https://arxiv.org/html/2610.02762#bib.bib44)\)\(33 models, 3,185 queries\), using queries where every model has a graded answer\.

## 4Common Router Patterns

Table 1:Comparison between commercial routers and a random \{Gemini 3\.7 Flash, Opus 5 \(high\)\} router at matched cost \(§[3](https://arxiv.org/html/2610.02762#S3)\), with our router \(§[7](https://arxiv.org/html/2610.02762#S7)\) below\.Δ\\Deltais the router’s accuracy minus the random router’s, with a 95% paired Wald interval \(which is narrower for Nadir because both it and the random baseline route many queries to Opus 5\)\.γ⁡\(d,a\)\\gamma\(d,a\)is Goodman\-Kruskal’s gamma between query difficultyddand the chosen model’s abilityaa\(§[F](https://arxiv.org/html/2610.02762#A6)\);γ⁡\(ℓ,c\)\\gamma\(\\ell,c\)is between expected answer length and the chosen model’s mean cost;*Source AMI*is the adjusted mutual information between source and routing choice, the share of routing decisions explained by the source \(§[B](https://arxiv.org/html/2610.02762#A2)\);*Within\-sourceγ⁡\(d,a\)\\gamma\(d,a\)*is the mean per\-sourceγ⁡\(d,a\)\\gamma\(d,a\)\. These intervals are 95% jackknife intervals over queries\. Our router is fit out of distribution \(§[A](https://arxiv.org/html/2610.02762#A1)\), and Table[7](https://arxiv.org/html/2610.02762#A4.T7)gives in\-distribution values\. A bold point estimate indicates a “pass”: not significantly below zero forΔ\\Deltaandγ⁡\(ℓ,c\)\\gamma\(\\ell,c\), and significantly above zero forγ⁡\(d,a\)\\gamma\(d,a\), globally or within source\. AMI is reported for context but not graded\.AccuracyDifficultyblindnessLengthreversalSemantic matchingRouterΔ\\Delta\(pp\)γ⁡\(d,a\)\\gamma\(d,a\)γ⁡\(ℓ,c\)\\gamma\(\\ell,c\)Source AMIWithin\-sourceγ⁡\(d,a\)\\gamma\(d,a\)OR auto \(med\)−5\.5\-5\.5±\\pm2\.12\.1\+0\.08\+0\.08±\\pm0\.080\.08−0\.24\-0\.24±\\pm0\.070\.070\.490\.49±\\pm0\.040\.04−0\.12\-0\.12±\\pm0\.120\.12OR auto \(high\)−4\.0\-4\.0±\\pm2\.02\.0−0\.16\-0\.16±\\pm0\.060\.06\+0\.02\\boldsymbol\{\+0\.02\}±\\pm0\.050\.050\.590\.59±\\pm0\.040\.04−0\.02\-0\.02±\\pm0\.120\.12OR auto \(xhigh\)−3\.3\-3\.3±\\pm2\.02\.0−0\.01\-0\.01±\\pm0\.060\.06\+0\.08\\boldsymbol\{\+0\.08\}±\\pm0\.050\.050\.600\.60±\\pm0\.040\.04\+0\.00\+0\.00±\\pm0\.120\.12OR auto\-β\\beta\(med\)−7\.7\-7\.7±\\pm2\.52\.5\+0\.00\+0\.00±\\pm0\.090\.09−0\.10\-0\.10±\\pm0\.080\.080\.560\.56±\\pm0\.050\.05−0\.10\-0\.10±\\pm0\.170\.17OR auto\-β\\beta\(high\)−8\.8\-8\.8±\\pm2\.52\.5−0\.07\-0\.07±\\pm0\.090\.09−0\.16\-0\.16±\\pm0\.080\.080\.540\.54±\\pm0\.060\.06−0\.08\-0\.08±\\pm0\.170\.17OR auto\-β\\beta\(xhigh\)−2\.0\-2\.0±\\pm1\.91\.9−0\.03\-0\.03±\\pm0\.060\.06−0\.01\\boldsymbol\{\-0\.01\}±\\pm0\.050\.050\.570\.57±\\pm0\.040\.04\+0\.01\+0\.01±\\pm0\.120\.12Azure \(balanced\)−8\.4\-8\.4±\\pm2\.62\.6\+0\.10\\boldsymbol\{\+0\.10\}±\\pm0\.070\.07\+0\.01\\boldsymbol\{\+0\.01\}±\\pm0\.060\.060\.360\.36±\\pm0\.040\.04\+0\.10\+0\.10±\\pm0\.100\.10Azure \(quality\)−2\.2\-2\.2±\\pm1\.91\.9\+0\.18\\boldsymbol\{\+0\.18\}±\\pm0\.070\.07−0\.19\-0\.19±\\pm0\.060\.060\.370\.37±\\pm0\.040\.04−0\.08\-0\.08±\\pm0\.110\.11ND \(cost\)−8\.8\-8\.8±\\pm2\.72\.7\+0\.07\+0\.07±\\pm0\.080\.08\+0\.29\\boldsymbol\{\+0\.29\}±\\pm0\.070\.070\.680\.68±\\pm0\.040\.04−0\.03\-0\.03±\\pm0\.150\.15ND \(quality\)−4\.1\-4\.1±\\pm2\.22\.2−0\.12\-0\.12±\\pm0\.080\.08−0\.25\-0\.25±\\pm0\.070\.070\.600\.60±\\pm0\.030\.03−0\.01\-0\.01±\\pm0\.130\.13vLLM SR−1\.0\\boldsymbol\{\-1\.0\}±\\pm1\.81\.8\+0\.15\\boldsymbol\{\+0\.15\}±\\pm0\.060\.06\+0\.10\\boldsymbol\{\+0\.10\}±\\pm0\.050\.050\.790\.79±\\pm0\.040\.04\+0\.11\+0\.11±\\pm0\.170\.17Nadir \(2\-tier\)−1\.4\-1\.4±\\pm1\.11\.1\+0\.12\+0\.12±\\pm0\.140\.14\+0\.71\\boldsymbol\{\+0\.71\}±\\pm0\.070\.070\.450\.45±\\pm0\.080\.08\+0\.21\+0\.21±\\pm0\.210\.21Nadir \(3\-tier\)−1\.8\-1\.8±\\pm1\.21\.2\+0\.12\\boldsymbol\{\+0\.12\}±\\pm0\.110\.11\+0\.63\\boldsymbol\{\+0\.63\}±\\pm0\.070\.070\.390\.39±\\pm0\.060\.06\+0\.17\\boldsymbol\{\+0\.17\}±\\pm0\.160\.16Orca \(adaptive\)−10\.5\-10\.5±\\pm2\.82\.8\+0\.05\+0\.05±\\pm0\.060\.06\+0\.04\\boldsymbol\{\+0\.04\}±\\pm0\.050\.05−0\.01\-0\.01±\\pm0\.010\.01\+0\.06\+0\.06±\\pm0\.070\.07Ours \($10/1k q\.\)−0\.2\\boldsymbol\{\-0\.2\}±\\pm0\.20\.2\+0\.40\\boldsymbol\{\+0\.40\}±\\pm0\.180\.18\+0\.64\\boldsymbol\{\+0\.64\}±\\pm0\.140\.140\.360\.36±\\pm0\.080\.08\+0\.48\\boldsymbol\{\+0\.48\}±\\pm0\.230\.23Ours \($20/1k q\.\)−0\.7\\boldsymbol\{\-0\.7\}±\\pm0\.70\.7\+0\.29\\boldsymbol\{\+0\.29\}±\\pm0\.110\.11\+0\.49\\boldsymbol\{\+0\.49\}±\\pm0\.100\.100\.480\.48±\\pm0\.080\.08\+0\.53\\boldsymbol\{\+0\.53\}±\\pm0\.160\.16Ours \($40/1k q\.\)−0\.7\\boldsymbol\{\-0\.7\}±\\pm1\.01\.0\+0\.20\\boldsymbol\{\+0\.20\}±\\pm0\.100\.10\+0\.39\\boldsymbol\{\+0\.39\}±\\pm0\.080\.080\.370\.37±\\pm0\.070\.07\+0\.29\\boldsymbol\{\+0\.29\}±\\pm0\.170\.17A router should outperform any random routing scheme at matched costs, but we find that a simple router that randomly chooses between \{Gemini 3\.7 Flash, Opus 5 \(high\)\} significantly outperforms every commercial router except for vLLM Semantic Router, which was fit on our training data \(Table[1](https://arxiv.org/html/2610.02762#S4.T1)\)\. Even when we restrict the random baseline’s roster to models each router had access to, no commercial router can outperform it, so this performance gap is not because we are using newer and more cost\-efficient models\.111All routers include the GPT 5\.6x series \(released July 9, 2026\), and some include Gemini 3\.7 Flash \(released August 13, 2026\)\. For routers without Gemini 3\.7 Flash, the random router over \{GPT 5\.6 Luna \(high\), GPT 5\.6 Sol \(xhigh\)\} is at least as strong \(Table[6](https://arxiv.org/html/2610.02762#A4.T6)\)\.We find that every commercial router that we evaluated exhibits several of the following four patterns:

#### Difficulty blindness\.

Across all routers, routing choices correlate at most weakly with query difficulty, and we argue that the standard objective gives them little reason to\. We measure the Goodman\-Kruskal gamma \(§[B](https://arxiv.org/html/2610.02762#A2)\) between query difficultydjd\_\{j\}and the chosen model’s abilityaia\_\{i\}, and find that no router exceedsγ=\+0\.20\\gamma=\+0\.20\.

#### Length reversal\.

Queries with longer answers tend to be more difficult, withγ=\+0\.52\\gamma=\+0\.52between answer length and difficulty in our evaluation set, so a router responding to difficulty should send longer\-answer queries to stronger models\. However,γ\\gammabetween a query’s expected answer lengthℓj\\ell\_\{j\}and the chosen model’s mean costc¯i\\overline\{c\}\_\{i\}is significantly negative for five commercial router settings\.

#### Semantic matching\.

Queries within a source vary in difficulty, but most commercial routers send nearly all of a source to one model \(Figure[15](https://arxiv.org/html/2610.02762#A5.F15)\), which suggests that they are routing by source identity\. We measure this in two ways: AMI \(adjusted mutual information\) between the source identity and routing choices \(see §[B](https://arxiv.org/html/2610.02762#A2)\), and mean within\-sourceγ\\gammabetween query difficultydjd\_\{j\}and the chosen model’s abilityaia\_\{i\}\. vLLM SR, which routes by category, has the highest AMI\. Orca achieves essentially zero AMI because it spreads each source over dozens of models\.

#### Roster suboptimality\.

A roster should contain the most cost\-efficient models and nothing else, and it is suboptimal if it omits one of them or if it includes others, which only add ways to route wrongly\. This suggests a simple test: if a router underperforms a random router over two well\-chosen models, either its roster or its routing policy must be suboptimal, and most commercial routers fail this test \(Table[1](https://arxiv.org/html/2610.02762#S4.T1)\)\. Most rosters include many models that do not help, while Nadir’s omits frontier models, routing only among Claude models\. Its cheaper tiers contain Haiku 4\.5 and Sonnet 5, neither of which is on the frontier\.222At the time of access, Nadir offered Haiku 4\.5, Sonnet 5, Opus 5, and Fable 5 for each tier\. Any cost\-saving configuration must use Haiku or Sonnet as the cheap tier, which are not on the Pareto frontier\. Our configurations are the only cost\-saving ones the platform allows\.

No router correlates with difficulty aboveγ=\+0\.20\\gamma=\+0\.20, five settings exhibit significant length reversal, most are far more source\-concentrated than difficulty\-based routing produces \(Table[7](https://arxiv.org/html/2610.02762#A4.T7)\) while being blind to within\-source difficulty, and most fail the roster test\. The next two sections examine the origins of these patterns\.

## 5Rethinking Pareto Efficiency

\(a\)Difficulty\-based escalation\.\(b\)Answer length\-based escalation\.
Figure 2:Accuracy against cost when escalating a 20% band of queries from the cheap to expensive model based on different metrics\. \(a\) escalates based on the difficulty of the query based on model pass rates across the whole model set, while \(b\) escalates based on output length, also averaged over the model set\. In \(b\), we mark the reference point corresponding to hardest\-20% escalation\. 95% half\-widths do not exceed 3\.5 pp, and in \(a\), the 60–80 band outperforms the 80–100 band by6\.3±3\.06\.3\\pm 3\.0pp\. The same trends hold across other datasets \(Figure[13](https://arxiv.org/html/2610.02762#A5.F13)\)\.The standard router evaluation methodology aims for Pareto efficiency on cost and accuracy\. Many routers optimize for this objective directly with a rule of the formarg⁡maxi⁡q^i​j−λ⋅c^i​j\\arg\\max\_\{i\}\\hat\{q\}\_\{ij\}\-\\lambda\\cdot\\hat\{c\}\_\{ij\}, whereq^i​j\\hat\{q\}\_\{ij\}is modelii’s predicted accuracy on queryjj,c^i​j\\hat\{c\}\_\{ij\}is the predicted cost, and the cost penaltyλ\\lambdatrades off between the two\. When the predictions are accurate, this rule is Pareto\-optimal for expected accuracy under an expected\-cost budget\([Jitkrittum et al\., 2023](https://arxiv.org/html/2610.02762#bib.bib56);[Dekoninck et al\., 2025](https://arxiv.org/html/2610.02762#bib.bib40)\)\. Whether or not commercial routers use this rule, they are likely measured against the same objective\. We show that this version of Pareto efficiency encourages some undesirable behaviors\.

### 5\.1Difficulty blindness

If only 20% of queries can be escalated, the natural choice is the hardest 20%, but this is not Pareto efficient\. Among 20% difficulty bands, escalating the 40–60th or 60–80th percentiles is Pareto\-efficient but escalating the 80–100th is not, both on our evaluation set \(Figure[2](https://arxiv.org/html/2610.02762#S5.F2)\) and on other datasets \(Figure[13](https://arxiv.org/html/2610.02762#A5.F13)\)\. This is because Pareto efficiency focuses on the tradeoff between cost and accuracy, and the strong model surpasses the weak model the most on moderately difficult queries\. On the easiest queries, the marginal gain is small because the weak model is likely to succeed, and on the most challenging queries, the gain is also small because both models are likely to fail\. Under a Rasch model with capabilitiesθ1\>θ2\\theta\_\{1\}\>\\theta\_\{2\}, the gain from escalation isσ⁡\(θ1−d\)−σ⁡\(θ2−d\)\\sigma\(\\theta\_\{1\}\-d\)\-\\sigma\(\\theta\_\{2\}\-d\), which peaks atd=\(θ1\+θ2\)/2d=\(\\theta\_\{1\}\+\\theta\_\{2\}\)/2and vanishes to 0 asd→±∞d\\to\\pm\\infty\.

### 5\.2Length reversal

Pareto efficiency also rewards escalating queries with shorter answers, as each answer contributes equally to accuracy but unequally to cost, and we see this when we escalate instead based on expected answer length \(Figure[2](https://arxiv.org/html/2610.02762#S5.F2)b\)\. On our evaluation dataset, we can achieve 59\.5% accuracy at $1\.03/1k queries by escalating the 0–20th length percentiles, just 1\.6 pp below escalating the hardest 20% at 1\.5% of the price, and outperform the hardest 20% in accuracy at 46% of the cost by escalating the 60–80th length percentiles\. We call this behavior*accuracy hacking*, in which a router invokes the smarter model when it is cheapest to do so\.

This phenomenon explains why we sometimes observe length reversal among commercial routers\. The CARROT design\([Somerstep et al\., 2025](https://arxiv.org/html/2610.02762#bib.bib24)\), in which a router predicts both the model’s answer quality and length, directly encourages this behavior by penalizing longer answers from expensive models\. A router that uses CARROT with a high cost penaltyλ\\lambdawill likely exhibit length reversal\.

### 5\.3Semantic matching

\(a\)Realized per\-query costs\.\(b\)Uniform per\-model costs\.
Figure 3:Accuracy at each maximum budget using three routing methods\.*Random*is random routing between the two frontier models adjacent to the budget\.*Difficulty*is a router with perfect knowledge of query difficulty, choosing its roster of up to five models and its thresholds on the evaluation set itself \(§[6](https://arxiv.org/html/2610.02762#S6)\)\.*Source*maps each data source to a single model, chosen on the training set\. In \(a\), each query is charged what the answer actually cost, while in \(b\), each query is charged that model’s average inference cost\. Since a map chosen on the training set costs a different amount on the evaluation set, we fit source maps on a dense grid of training budgets and use the highest\-budget map that stays under each evaluation budget\. Half\-widths for the gaps between difficulty and random routing are at most 1\.6 pp, and those between difficulty and source routing are at most 2\.1 pp\.Semantic matching can look attractive using realized per\-query costs, where source routing outperforms random routing at the two cheapest budgets \(Figure[3](https://arxiv.org/html/2610.02762#S5.F3)\)\. But when we use uniform per\-model costs, it either underperforms or ties, suggesting that much of semantic matching’s apparent advantage at these ranges was in accuracy hacking\. Answer length tracks source closely, and source is easier to predict than difficulty, so source routing can look better on this metric than difficulty routing\. Semantic matching is particularly brittle because the training dataset’s composition determines the router’s choices, which might generalize poorly\. For example, if the math questions in the training set tend to be challenging, then a router learns to choose a strong model for all math questions, even trivial ones\.

### 5\.4A revised methodology

A robust evaluation methodology should reward the routing behavior we want\. Two of the three corrections follow from the analysis above, and the first depends on the following judgment\. On a query no model can solve, the standard objective is indifferent between cheap and expensive models, since both fail\. We take the position that users would prefer if the more capable model attempted it, and would query that model themselves if they knew the weaker model had been used\. Partial credit could make this difference measurable, but assigning it is difficult, so we adopt our assumption as a design choice and treat correction 1 as optional for readers who disagree\. We propose the following corrections for the patterns:

1. 1\.Difficulty blindness:Escalation on the most difficult queries should be directly encouraged, not penalized\. We achieve this by partitioning our evaluation set into a*hard band*H⊆𝒬H\\subseteq\\mathcal\{Q\}and an*easy band*E⊆𝒬E\\subseteq\\mathcal\{Q\}, whereH​∪˙​E=𝒬H\\,\\dot\{\\cup\}\\,E=\\mathcal\{Q\}, and optimizing for Pareto efficiency jointly on three metrics: high hard\-band escalation rate, high easy\-band accuracy, and low easy\-band cost\.
2. 2\.Length reversal:A router chooses which model answers but not how long the answer is, so we charge each answer from modeliiits mean costc¯i\\overline\{c\}\_\{i\}rather than the realized costci​jc\_\{ij\}, just as every answer contributes equally to accuracy\.
3. 3\.Semantic matching:When we predict a model’s pass rate, we should evaluate the predictor’s meanγ\\gammawithin any given data source, not over the entire dataset\. Then, a predictor that assigns a single pass rate to each source cannot score well\.

Note that we do not assess a router by its accuracy or cost on the hard band since we want it to escalate difficult queries, even impossible ones, to stronger and more expensive models\. The size and composition of the hard band is subjective, depending both on the models in the roster and the target price point of the router\. In our case, we define the hard band as any query with at least a 50% failure rate \(H=\{j:dj≥0\.5\}H=\\\{j:d\_\{j\}\\geq 0\.5\\\}\), which encompasses 26\.2% of the evaluation set\.

## 6Why Are Rosters Large?

Most commercial routers we evaluate route among many models that are not on the cost\-accuracy frontier\. We believe two appealing but empirically unjustified assumptions explain why\.

### 6\.1Model granularity

Figure 4:For each budget, the highest accuracy a router can achieve with perfect knowledge of query difficulty, choosing the roster and thresholds on the evaluation set itself, for rosters ofKKmodels, where harder queries are sent to more capable models\. Accuracies above the strongest standalone model reflect overfitting to the evaluation set, since stronger models typically outperform weaker ones across all difficulty bands \(Figure[11](https://arxiv.org/html/2610.02762#A5.F11)\)\.If we only route between a cheap and an expensive model, then a medium\-difficulty query has two poor options: the cheap model, which risks a wrong answer, or the expensive model, which costs far more\. We could resolve this with an intermediate model, allowing us to match each difficulty level to the appropriate price level\. At the limit, we would have a large roster, where each model is matched to a narrow difficulty range\.

Empirically, however, few models appear on the cost\-accuracy frontier in any model set, and even fewer are useful for routing \(Figures[7](https://arxiv.org/html/2610.02762#A5.F7)and[12](https://arxiv.org/html/2610.02762#A5.F12)\)\. In the 26\-model set, Gemini 3\.7 Flash only trails Opus 5 \(high\) by 1\.6 pp while maintaining a substantial 96% discount\. Even when a router knows each query’s true difficulty and chooses its roster and thresholds on the evaluation set itself, which is the most favorable setting possible, a roster of two models is never more than 1 pp below the best roster of any size \(Figure[4](https://arxiv.org/html/2610.02762#S6.F4)\)\. Larger rosters are also more fragile, with more thresholds to tune from noisy difficulty estimates, so the small gains they show here, measured in\-sample, may not generalize out of distribution\.

### 6\.2Model specialization

Even if two models have the same overall capability, they may specialize in different areas\. If so, a router should route on domain as well as difficulty, and the roster should include models with different specializations to cover different task types\. We test this hypothesis using item response theory \(IRT\): if models do not specialize, then each can be described by a single capabilityθi\\theta\_\{i\}and each query by a difficultydjd\_\{j\}, where modeliipasses queryjjwith probabilityσ⁡\(θi−dj\)\\sigma\(\\theta\_\{i\}\-d\_\{j\}\)\. This is the Rasch model and it fits our outcomes well \(Figure[11](https://arxiv.org/html/2610.02762#A5.F11)\)\. If models do specialize, then a model’s propensitiespi∈ℝkp\_\{i\}\\in\\mathbb\{R\}^\{k\}interact with a query’s featuresfj∈ℝkf\_\{j\}\\in\\mathbb\{R\}^\{k\}beyond capability and difficulty alone, givingσ⁡\(θi−dj\+pi⋅fj\)\\sigma\(\\theta\_\{i\}\-d\_\{j\}\+p\_\{i\}\\cdot f\_\{j\}\)\(as in[Song et al\., 2025](https://arxiv.org/html/2610.02762#bib.bib25), but with separate capability and difficulty parameters\)\.

To establish an upper bound on this approach, we fit the model parametersθi\\theta\_\{i\}andpip\_\{i\}on the evaluation set directly\. For a given queryjj, the difficultydjd\_\{j\}and featuresfjf\_\{j\}are fit with one model’s outcome masked at a time, and we evaluate on the masked model’s outcome to prevent overfitting\. We also estimate query features from Qwen3\.5\-27B\([Qwen Team, 2026](https://arxiv.org/html/2610.02762#bib.bib18)\)activations, or using a combination of masked pass rates and activations\. These settings are highly generous, since a deployed router can only route on query features\.

\(a\)Easy\-band accuracy against easy\-band cost\.\(b\)Hard\-band escalation against easy\-band cost\.
Figure 5:Our router on \{Gemini 3\.7 Flash, Opus 5 \(high\)\} evaluated using our revised methodology from §[5](https://arxiv.org/html/2610.02762#S5), where all costs are per\-model mean costs and the hard band contains queries with at least a 50% failure rate\. Random is random routing between the two models\. Ours \(ID\) and \(OOD\) are our router fit in and out of distribution \(§[A](https://arxiv.org/html/2610.02762#A1)\)\. Difficulty oracle uses the true difficulty scores to route\. On \(a\), 95% confidence interval half\-widths for the gap with random do not exceed 1\.2 pp, with all deviation from random routing remaining within noise; on \(b\), they do not exceed 7\.5 pp, with our router surpassing random routing on hard\-band escalation outside noise across $6–139/1k queries ID, and $42–73/1k and $121–139/1k queries OOD\.Specialization effects are weak even in these generous settings, where increasingkkfrom 0 to 8 reduces NLL only from0\.27±0\.020\.27\\pm 0\.02to0\.24±0\.020\.24\\pm 0\.02\(Figure[6](https://arxiv.org/html/2610.02762#A5.F6)\)\. This is unsurprising, since frontier models are trained as generalists and prior work found that LLM performance across benchmarks is well described by a single capability score\([Ruan et al\., 2024](https://arxiv.org/html/2610.02762#bib.bib27);[Ilić and Gignac, 2024](https://arxiv.org/html/2610.02762#bib.bib28);[Ho et al\., 2025](https://arxiv.org/html/2610.02762#bib.bib26)\)\. Specialization can still matter when routing among frontier models\([Tang et al\., 2026](https://arxiv.org/html/2610.02762#bib.bib54)\), or on older, more heterogeneous model sets such as SPROUT, where source\-based routing outperforms difficulty\-based routing \(Figure[14](https://arxiv.org/html/2610.02762#A5.F14)e–f\)\. But in a roster of cheap and expensive models, differences in capability dominate\.

## 7Avoiding These Patterns

We build a proof\-of\-concept router to show that the four patterns are avoidable\. We choose \{Gemini 3\.7 Flash, Opus 5 \(high\)\} as our roster, the same model pair we used in our random baseline\. To predict difficulty, we compare five methods on Qwen3\.5\-27B by within\-sourceγ⁡\(d^,d\)\\gamma\(\\hat\{d\},d\)and select SharedTrunkNet\([Varshney et al\., 2026](https://arxiv.org/html/2610.02762#bib.bib31)\)\(§[C](https://arxiv.org/html/2610.02762#A3)\)\. A threshold on predicted difficulty determines the price point, with queries above the threshold routed to Opus 5 \(high\) and everything else to Gemini 3\.7 Flash\.

On the easy band, both models in our roster solve nearly all queries, and all routed accuracies are within noise of random routing \(Figure[5](https://arxiv.org/html/2610.02762#S6.F5)a\)\. On the hard band, Opus 5 \(high\) leads Gemini 3\.7 Flash by7\.2±7\.67\.2\\pm 7\.6pp, which is within noise, so there is little room for accuracy gains through routing on this roster\. Our router escalates hard\-band queries more often than random ID, but the gap is mostly within noise OOD, suggesting that source exposure helps a router know when to escalate\. Under the standard methodology, its accuracy is within noise of random routing at every budget \(Table[5](https://arxiv.org/html/2610.02762#A4.T5)\)\.

When we replace Gemini 3\.7 Flash with the much weaker gpt\-oss\-20b \(low\), both our router and the difficulty oracle underperform random routing at every budget, and the oracle does worst \(Table[7](https://arxiv.org/html/2610.02762#A4.T7)\), since the hardest queries tend to also be the longest\. Under our revised methodology, difficulty\-based routing on this roster significantly outperforms random routing at most budgets on the easy band \(Figure[9](https://arxiv.org/html/2610.02762#A5.F9)\)\.

Our router avoids the four patterns \(Table[1](https://arxiv.org/html/2610.02762#S4.T1)\)\. Its source AMI is comparable to that of the commercial routers, but its within\-sourceγ⁡\(d,a\)\\gamma\(d,a\)is far higher, which indicates that it distinguishes queries within a source by difficulty\. With a well\-chosen roster, the primary advantage of routing is better\-than\-random escalation in distribution in the hard band, and the roster, not the router, determines nearly all of the accuracy\.

## 8Discussion

In this work, we identify four patterns prevalent across commercial LLM routers: difficulty blindness, length reversal, semantic matching, and roster suboptimality\. As a proof of concept, we develop a simple router between two models which avoids them, although it offers little measurable performance gain\. Even with perfect knowledge of query difficulty, rosters larger than two models gain at most 1 pp, and on a well\-chosen pair, a router that avoids the other three patterns still performs within noise of random routing\. When a commercial router underperforms random routing, its roster is therefore the more likely cause\.

Our analysis suggests that the problem of routing is a challenging one, without a reliable way of determining which queries to send to each model\. Moreover, this challenge may become greater with future generations of models which are increasingly capable and optimized for long\-horizon tasks\. In that case, it may be even more difficult to predict based on the query alone whether the future cheap but \(by today’s standards\) extremely capable models will fail a task\.

## Limitations

#### Single\-turn routing\.

We focus solely on single\-turn routing, but this myopic approach applied per\-turn can hurt performance since with KV caching, switching models is not free\. We do not propose any solution to multi\-turn routing; this paper focuses on single\-turn routing and shows that, even in this simpler case, router designs are often suboptimal\.

#### Dataset composition\.

We attempted to make our training and evaluation datasets as diverse as possible, but we relied entirely on benchmarks because of their verifiability, and benchmark tasks may not resemble real user traffic\. We only generate each outcome once, which can be noisy since model outcomes on a query vary between samples, and we did not test the stability of pass/fail outcomes over samples\. Finally, our datasets only include English queries, although model capabilities can differ between languages\([Shi et al\., 2023](https://arxiv.org/html/2610.02762#bib.bib58);[Ahuja et al\., 2023](https://arxiv.org/html/2610.02762#bib.bib60);[Singh et al\., 2025](https://arxiv.org/html/2610.02762#bib.bib61);[Kang et al\., 2026](https://arxiv.org/html/2610.02762#bib.bib59)\)\. We leave these explorations to future work\.

#### Future models may specialize\.

We only observe weak specialization among the models we tested\. However, the model development landscape may shift to focus on specialized or personalized models whose capabilities vary greatly by domain, in which case future routers might benefit from larger rosters\.

#### Weak difficulty predictions\.

We find it difficult to predict query difficulty, especially OOD, which limits the efficacy of routing\. Although we do not claim to have found the optimal difficulty predictor, its OOD performance within a source does not improve with a tenfold increase in training data \(Figure[10](https://arxiv.org/html/2610.02762#A5.F10)\), which suggests the limit lies in how much a query reveals about its difficulty rather than in our method or data\. In addition, this task may become more challenging as model capabilities improve and it becomes increasingly difficult to predict which challenging tasks are harder than others\.

## References

- Ahujaet al\.\(2023\)K\. Ahuja, H\. Diddee, R\. Hada, M\. Ochieng, K\. Ramesh, P\. Jain, A\. Nambi, T\. Ganu, S\. Segal, M\. Ahmed, K\. Bali, and S\. SitaramMEGA: multilingual evaluation of generative AI\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 4232–4267\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.258/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.258)Cited by:[Dataset composition\.](https://arxiv.org/html/2610.02762#Sx1.SS0.SSS0.Px2.p1.1)\.
- Artificial Analysis \(2026\)Artificial AnalysisArtificial analysis intelligence index\.Note:[https://artificialanalysis\.ai/evaluations/artificial\-analysis\-intelligence\-index](https://artificialanalysis.ai/evaluations/artificial-analysis-intelligence-index)Accessed: 2026\-08\-31Cited by:[§3](https://arxiv.org/html/2610.02762#S3.SS0.SSS0.Px1.p1.1)\.
- Baoet al\.\(2026\)Z\. Bao, F\. Tian, C\. Zhang, Z\. Chen, X\. Ma, and Y\. ShiOrcaRouter: a production\-oriented llm router with hybrid offline\-online learning\.External Links:2605\.30736,[Link](https://arxiv.org/abs/2605.30736)Cited by:[item 6](https://arxiv.org/html/2610.02762#A1.I3.i6.p1.1),[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px1.p1.1)\.
- Birnbaum \(1968\)A\. BirnbaumSome latent trait models and their use in inferring an examinee’s ability\.InStatistical Theories of Mental Test Scores,F\. M\. Lord and M\. R\. Novick \(Eds\.\),pp\. 397–479\.Cited by:[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px4.p1.1)\.
- Burgeset al\.\(2005\)C\. Burges, T\. Shaked, E\. Renshaw, A\. Lazier, M\. Deeds, N\. Hamilton, and G\. HullenderLearning to rank using gradient descent\.InProceedings of the 22nd International Conference on Machine Learning,ICML ’05,New York, NY, USA,pp\. 89–96\.External Links:ISBN 1595931805,[Link](https://doi.org/10.1145/1102351.1102363),[Document](https://dx.doi.org/10.1145/1102351.1102363)Cited by:[item 3](https://arxiv.org/html/2610.02762#A3.I1.i3.p1.1)\.
- Burnellet al\.\(2023\)R\. Burnell, H\. Hao, A\. R\. A\. Conway, and J\. H\. OralloRevealing the structure of language model capabilities\.External Links:2306\.10062,[Link](https://arxiv.org/abs/2306.10062)Cited by:[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px4.p1.1)\.
- Cencerradoet al\.\(2025\)I\. V\. M\. Cencerrado, A\. P\. Masdemont, A\. G\. Hawthorne, D\. D\. Africa, and L\. PacchiardiNo answer needed: predicting llm answer accuracy from question\-only linear probes\.External Links:2509\.10625,[Link](https://arxiv.org/abs/2509.10625)Cited by:[item 2](https://arxiv.org/html/2610.02762#A3.I1.i2.p1.1),[Appendix C](https://arxiv.org/html/2610.02762#A3.p1.1),[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px4.p1.1)\.
- Center for AI Safetyet al\.\(2026\)Center for AI Safety, Scale AI, and HLE Contributors ConsortiumA benchmark of expert\-level academic questions to assess AI capabilities\.Nature649,pp\. 1139–1146\.External Links:[Document](https://dx.doi.org/10.1038/s41586-025-09962-4),2501\.14249,[Link](https://arxiv.org/abs/2501.14249)Cited by:[Table 2](https://arxiv.org/html/2610.02762#A1.T2.2.20.2.1.1)\.
- Chenet al\.\(2021\)M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. de Oliveira Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. ZarembaEvaluating large language models trained on code\.External Links:2107\.03374,[Link](https://arxiv.org/abs/2107.03374)Cited by:[Table 2](https://arxiv.org/html/2610.02762#A1.T2.2.3.2.1.1)\.
- Chenet al\.\(2024\)S\. Chen, W\. Jiang, B\. Lin, J\. Kwok, and Y\. ZhangRouterDC: query\-based router by dual contrastive learning for assembling large language models\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=7RQvjayHrM)Cited by:[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px1.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. SchulmanTraining verifiers to solve math word problems\.CoRRabs/2110\.14168\.External Links:[Link](https://arxiv.org/abs/2110.14168),2110\.14168Cited by:[Table 2](https://arxiv.org/html/2610.02762#A1.T2.2.12.2.1.1)\.
- Dekonincket al\.\(2025\)J\. Dekoninck, M\. Baader, and M\. VechevA unified approach to routing and cascading for LLMs\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=AAl89VNNy1)Cited by:[§5](https://arxiv.org/html/2610.02762#S5.p1.1)\.
- Dinget al\.\(2024\)D\. Ding, A\. Mallick, C\. Wang, R\. Sim, S\. Mukherjee, V\. Rühle, L\. V\. S\. Lakshmanan, and A\. H\. AwadallahHybrid LLM: cost\-efficient and quality\-aware query routing\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=02f3mUtqnM)Cited by:[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px1.p1.1)\.
- Fenget al\.\(2025\)T\. Feng, Y\. Shen, and J\. YouGraphRouter: a graph\-based router for LLM selections\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=eU39PDsZtT)Cited by:[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px1.p1.1)\.
- Ghasemabadi and Niu \(2025\)A\. Ghasemabadi and D\. NiuCan llms predict their own failures? self\-awareness via internal circuits\.External Links:2512\.20578,[Link](https://arxiv.org/abs/2512.20578)Cited by:[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px4.p1.1)\.
- Hendryckset al\.\(2021a\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring massive multitask language understanding\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=d7KBjmI3GmQ)Cited by:[Table 2](https://arxiv.org/html/2610.02762#A1.T2.2.8.2.1.1)\.
- Hendryckset al\.\(2021b\)D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. SteinhardtMeasuring mathematical problem solving with the MATH dataset\.InThirty\-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track \(Round 2\),External Links:[Link](https://openreview.net/forum?id=7Bywt2mQsCe)Cited by:[Table 2](https://arxiv.org/html/2610.02762#A1.T2.2.11.2.1.1)\.
- Hoet al\.\(2025\)A\. Ho, J\. Denain, D\. Atanasov, S\. Albanie, and R\. ShahA rosetta stone for ai benchmarks\.Note:Accessed: 2026\-09\-02External Links:[Link](https://epoch.ai/publications/a-rosetta-stone-for-ai-benchmarks)Cited by:[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px4.p1.1),[§6\.2](https://arxiv.org/html/2610.02762#S6.SS2.p3.1)\.
- Huet al\.\(2024\)Q\. J\. Hu, J\. Bieker, X\. Li, N\. Jiang, B\. Keigwin, G\. Ranganath, K\. Keutzer, and S\. K\. UpadhyayRouterBench: a benchmark for multi\-LLM routing system\.InAgentic Markets Workshop at ICML 2024,External Links:[Link](https://openreview.net/forum?id=IVXmV8Uxwh)Cited by:[§1](https://arxiv.org/html/2610.02762#S1.p2.1),[§1](https://arxiv.org/html/2610.02762#S1.p3.1),[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px3.p1.1),[§3](https://arxiv.org/html/2610.02762#S3.SS0.SSS0.Px4.p1.1)\.
- Ilić and Gignac \(2024\)D\. Ilić and G\. E\. GignacEvidence of interrelated cognitive\-like capabilities in large language models: indications of artificial general intelligence or achievement?\.Intelligence106,pp\. 101858\.External Links:ISSN 0160\-2896,[Document](https://dx.doi.org/10.1016/j.intell.2024.101858),[Link](https://www.sciencedirect.com/science/article/pii/S0160289624000527)Cited by:[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px4.p1.1),[§6\.2](https://arxiv.org/html/2610.02762#S6.SS2.p3.1)\.
- Jitkrittumet al\.\(2023\)W\. Jitkrittum, N\. Gupta, A\. K\. Menon, H\. Narasimhan, A\. S\. Rawat, and S\. KumarWhen does confidence\-based cascade deferral suffice?\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=4KZhZJSPYU)Cited by:[§5](https://arxiv.org/html/2610.02762#S5.p1.1)\.
- Jitkrittumet al\.\(2026\)W\. Jitkrittum, H\. Narasimhan, A\. S\. Rawat, J\. Juneja, C\. Wang, Z\. Wang, A\. Go, C\. Lee, P\. Shenoy, R\. Panigrahy, A\. K\. Menon, and S\. KumarUniversal model routing for efficient LLM inference\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=ka82fvJ5f1)Cited by:[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px1.p1.1)\.
- Joshiet al\.\(2017\)M\. Joshi, E\. Choi, D\. Weld, and L\. ZettlemoyerTriviaQA: a large scale distantly supervised challenge dataset for reading comprehension\.InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),R\. Barzilay and M\. Kan \(Eds\.\),Vancouver, Canada,pp\. 1601–1611\.External Links:[Link](https://aclanthology.org/P17-1147/),[Document](https://dx.doi.org/10.18653/v1/P17-1147)Cited by:[Table 2](https://arxiv.org/html/2610.02762#A1.T2.2.14.2.1.1)\.
- Kanget al\.\(2026\)D\. Kang, S\. Hwang, D\. Kim, H\. Kim, and G\. LeeWhy do multilingual reasoning gaps emerge in reasoning language models?\.InFindings of the Association for Computational Linguistics: ACL 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 31684–31716\.External Links:[Link](https://aclanthology.org/2026.findings-acl.1586/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.1586),ISBN 979\-8\-89176\-395\-1Cited by:[Dataset composition\.](https://arxiv.org/html/2610.02762#Sx1.SS0.SSS0.Px2.p1.1)\.
- Kipniset al\.\(2025\)A\. Kipnis, K\. Voudouris, L\. M\. S\. Buschoff, and E\. SchulzMetabench \- a sparse benchmark of reasoning and knowledge in large language models\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=4T33izzFpK)Cited by:[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px4.p1.1)\.
- Kwiatkowskiet al\.\(2019\)T\. Kwiatkowski, J\. Palomaki, O\. Redfield, M\. Collins, A\. Parikh, C\. Alberti, D\. Epstein, I\. Polosukhin, J\. Devlin, K\. Lee, K\. Toutanova, L\. Jones, M\. Kelcey, M\. Chang, A\. M\. Dai, J\. Uszkoreit, Q\. Le, and S\. PetrovNatural questions: a benchmark for question answering research\.Transactions of the Association for Computational Linguistics7,pp\. 453–466\.External Links:ISSN 2307\-387X,[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00276),[Link](https://doi.org/10.1162/tacl_a_00276)Cited by:[Table 2](https://arxiv.org/html/2610.02762#A1.T2.2.13.2.1.1)\.
- Leeet al\.\(2025\)S\. Lee, Q\. Yin, C\. T\. Leong, J\. Zhang, Y\. Gong, S\. Ni, M\. Yang, and X\. ShenProbing the difficulty perception mechanism of large language models\.External Links:2510\.05969,[Link](https://arxiv.org/abs/2510.05969)Cited by:[item 1](https://arxiv.org/html/2610.02762#A3.I1.i1.p1.1),[Appendix C](https://arxiv.org/html/2610.02762#A3.p1.1),[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px4.p1.1)\.
- Liet al\.\(2026\)H\. Li, Y\. Zhang, Z\. Guo, C\. Wang, S\. Tang, Q\. Zhang, Y\. Chen, B\. Qi, P\. Ye, L\. Bai, Z\. Wang, and S\. HuLLMRouterBench: a massive benchmark and unified framework for LLM routing\.InFindings of the Association for Computational Linguistics: ACL 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 37733–37754\.External Links:[Link](https://aclanthology.org/2026.findings-acl.1881/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.1881),ISBN 979\-8\-89176\-395\-1Cited by:[§1](https://arxiv.org/html/2610.02762#S1.p2.1),[§1](https://arxiv.org/html/2610.02762#S1.p3.1),[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px3.p1.1),[§3](https://arxiv.org/html/2610.02762#S3.SS0.SSS0.Px4.p1.1)\.
- Lightmanet al\.\(2024\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=v8L0pN6EOi)Cited by:[Table 2](https://arxiv.org/html/2610.02762#A1.T2.2.11.2.1.1)\.
- Liuet al\.\(2026\)X\. Liu, H\. Chen, S\. Lu, Y\. Ovadia, G\. Wen, H\. Wu, Z\. Tan, J\. Zhang, S\. Zedan, Y\. Kerido, L\. Weiss, H\. Zhang, B\. Yu, A\. Balum, N\. Limoy, A\. Samara, B\. Fan, B\. Salisbury, R\. Cook, Z\. Wang, Q\. Pan, R\. Khan, A\. Goswami, H\. H\. Zhang, S\. Wang, Z\. Tang, F\. Han, Z\. Hassan, J\. Zheng, A\. Changrani, X\. Liu, and B\. HeVLLM semantic router: signal driven decision routing for mixture\-of\-modality models\.External Links:2603\.04444,[Link](https://arxiv.org/abs/2603.04444)Cited by:[§1](https://arxiv.org/html/2610.02762#S1.p1.1),[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px1.p1.1)\.
- Lord \(1952\)F\. M\. LordA theory of test scores\.Psychometric Monograph\(7\)\.Cited by:[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px4.p1.1)\.
- Luet al\.\(2024\)K\. Lu, H\. Yuan, R\. Lin, J\. Lin, Z\. Yuan, C\. Zhou, and J\. ZhouRouting to the expert: efficient reward\-guided ensemble of large language models\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 1964–1974\.External Links:[Link](https://aclanthology.org/2024.naacl-long.109/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.109)Cited by:[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px1.p1.1)\.
- Luet al\.\(2026a\)Y\. Lu, R\. Liu, J\. Yuan, X\. Cui, S\. Zhang, H\. Liu, and J\. XingRouterArena: an open platform for comprehensive comparison of LLM routers\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=9HsaIi4ngF)Cited by:[§1](https://arxiv.org/html/2610.02762#S1.p3.1),[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px2.p1.1)\.
- Luet al\.\(2026b\)Y\. Lu, Q\. Zhang, S\. Zhang, Z\. Yu, Z\. Wang, H\. Chen, and J\. XingThe routing plateau: understanding and breaking the accuracy limits of llm routers\.External Links:2606\.07587,[Link](https://arxiv.org/abs/2606.07587)Cited by:[§1](https://arxiv.org/html/2610.02762#S1.p2.1),[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px3.p1.1)\.
- Lugoloobiet al\.\(2026\)W\. Lugoloobi, T\. Foster, W\. Bankes, and C\. RussellLLMs encode their failures: predicting success from pre\-generation activations\.InWorkshop on Latent & Implicit Thinking – Going Beyond CoT Reasoning,External Links:[Link](https://openreview.net/forum?id=8rlQrTUi9t)Cited by:[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px4.p1.1)\.
- Maet al\.\(2024\)Z\. Ma, B\. Zhang, J\. Zhang, J\. Yu, X\. Zhang, X\. Zhang, S\. Luo, X\. Wang, and J\. TangSpreadsheetBench: towards challenging real world spreadsheet manipulation\.InThe Thirty\-Eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=KYxzmRLF6i)Cited by:[Table 2](https://arxiv.org/html/2610.02762#A1.T2.2.16.2.1.1)\.
- Muennighoffet al\.\(2024\)N\. Muennighoff, Q\. Liu, A\. R\. Zebaze, Q\. Zheng, B\. Hui, T\. Y\. Zhuo, S\. Singh, X\. Tang, L\. V\. Werra, and S\. LongpreOctoPack: instruction tuning code large language models\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=mw1PWNSWZP)Cited by:[Table 2](https://arxiv.org/html/2610.02762#A1.T2.2.4.2.1.1)\.
- Odenaet al\.\(2021\)A\. Odena, C\. Sutton, D\. M\. Dohan, E\. Jiang, H\. Michalewski, J\. Austin, M\. P\. Bosma, M\. Nye, M\. Terry, and Q\. V\. LeProgram synthesis with large language models\.InarXiv preprint,External Links:2108\.07732,[Link](https://arxiv.org/abs/2108.07732)Cited by:[Table 2](https://arxiv.org/html/2610.02762#A1.T2.2.5.2.1.1)\.
- Onget al\.\(2025\)I\. Ong, A\. Almahairi, V\. Wu, W\. Chiang, T\. Wu, J\. E\. Gonzalez, M\. W\. Kadous, and I\. StoicaRouteLLM: learning to route LLMs from preference data\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=8sSqNntaMr)Cited by:[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px1.p1.1)\.
- Osés Grijalbaet al\.\(2024\)J\. Osés Grijalba, L\. A\. Ureña\-López, E\. Martínez Cámara, and J\. Camacho\-ColladosQuestion answering over tabular data with DataBench: a large\-scale empirical evaluation of LLMs\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),N\. Calzolari, M\. Kan, V\. Hoste, A\. Lenci, S\. Sakti, and N\. Xue \(Eds\.\),Torino, Italia,pp\. 13471–13488\.External Links:[Link](https://aclanthology.org/2024.lrec-main.1179/)Cited by:[Table 2](https://arxiv.org/html/2610.02762#A1.T2.2.15.2.1.1)\.
- Patilet al\.\(2025\)S\. G\. Patil, H\. Mao, F\. Yan, C\. C\. Ji, V\. Suresh, I\. Stoica, and J\. E\. GonzalezThe berkeley function calling leaderboard \(BFCL\): from tool use to agentic evaluation of large language models\.InProceedings of the 42nd International Conference on Machine Learning,ICML’25\.Cited by:[Table 2](https://arxiv.org/html/2610.02762#A1.T2.2.18.2.1.1)\.
- Patwardhanet al\.\(2026\)T\. Patwardhan, R\. Dias, E\. Proehl, G\. Kim, M\. Wang, O\. Watkins, S\. P\. Fishman, M\. Aljubeh, P\. Thacker, L\. Fauconnet, N\. S\. Kim, S\. Miserendino, G\. Chabot, D\. Li, P\. Chao, M\. Sharman, A\. Barr, A\. Glaese, and J\. TworekGDPval: evaluating AI model performance on real\-world economically valuable tasks\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=hcuEdq6eKD)Cited by:[Table 2](https://arxiv.org/html/2610.02762#A1.T2.2.17.2.1.1)\.
- Poloet al\.\(2024\)F\. M\. Polo, L\. Weber, L\. Choshen, Y\. Sun, G\. Xu, and M\. YurochkinTinyBenchmarks: evaluating LLMs with fewer examples\.InForty\-first International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=qAml3FpfhG)Cited by:[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px4.p1.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.5: towards native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[§6\.2](https://arxiv.org/html/2610.02762#S6.SS2.p2.1)\.
- Rasch \(1960\)G\. RaschProbabilistic models for some intelligence and attainment tests\.Danmarks Paedagogiske Institut,Copenhagen\.Cited by:[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px4.p1.1)\.
- Reckase \(2009\)M\. D\. ReckaseMultidimensional item response theory\.Statistics for Social and Behavioral Sciences,Springer,New York, NY\.External Links:[Document](https://dx.doi.org/10.1007/978-0-387-89976-3)Cited by:[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px4.p1.1)\.
- Reinet al\.\(2024\)D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. BowmanGPQA: a graduate\-level google\-proof Q&A benchmark\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=Ti67584b98)Cited by:[Table 2](https://arxiv.org/html/2610.02762#A1.T2.2.10.2.1.1)\.
- Ruanet al\.\(2024\)Y\. Ruan, C\. J\. Maddison, and T\. HashimotoObservational scaling laws and the predictability of language model performance\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=On5WIN7xyD)Cited by:[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px4.p1.1),[§6\.2](https://arxiv.org/html/2610.02762#S6.SS2.p3.1)\.
- Shiet al\.\(2023\)F\. Shi, M\. Suzgun, M\. Freitag, X\. Wang, S\. Srivats, S\. Vosoughi, H\. W\. Chung, Y\. Tay, S\. Ruder, D\. Zhou, D\. Das, and J\. WeiLanguage models are multilingual chain\-of\-thought reasoners\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=fR3wGCk-IXp)Cited by:[Dataset composition\.](https://arxiv.org/html/2610.02762#Sx1.SS0.SSS0.Px2.p1.1)\.
- Singhet al\.\(2025\)S\. Singh, A\. Romanou, C\. Fourrier, D\. I\. Adelani, J\. G\. Ngui, D\. Vila\-Suero, P\. Limkonchotiwat, K\. Marchisio, W\. Q\. Leong, Y\. Susanto, R\. Ng, S\. Longpre, S\. Ruder, W\. Ko, A\. Bosselut, A\. Oh, A\. Martins, L\. Choshen, D\. Ippolito, E\. Ferrante, M\. Fadaee, B\. Ermis, and S\. HookerGlobal MMLU: understanding and addressing cultural and linguistic biases in multilingual evaluation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 18761–18799\.External Links:[Link](https://aclanthology.org/2025.acl-long.919/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.919),ISBN 979\-8\-89176\-251\-0Cited by:[Dataset composition\.](https://arxiv.org/html/2610.02762#Sx1.SS0.SSS0.Px2.p1.1)\.
- Somerstepet al\.\(2025\)S\. Somerstep, F\. M\. Polo, A\. F\. M\. de Oliveira, P\. Mangal, M\. Silva, O\. Bhardwaj, M\. Yurochkin, and S\. MaityCARROT: a cost aware rate optimal router\.External Links:2502\.03261,[Link](https://arxiv.org/abs/2502.03261)Cited by:[§3](https://arxiv.org/html/2610.02762#S3.SS0.SSS0.Px4.p1.1),[§5\.2](https://arxiv.org/html/2610.02762#S5.SS2.p2.1)\.
- Songet al\.\(2025\)W\. Song, Z\. Huang, C\. Cheng, W\. Gao, B\. Xu, G\. Zhao, F\. Wang, and R\. WuIRT\-router: effective and interpretable multi\-LLM routing via item response theory\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 15629–15644\.External Links:[Link](https://aclanthology.org/2025.acl-long.761/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.761),ISBN 979\-8\-89176\-251\-0Cited by:[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px4.p1.1),[§6\.2](https://arxiv.org/html/2610.02762#S6.SS2.p1.1)\.
- Tanget al\.\(2026\)Y\. Tang, E\. Cetin, J\. Xu, Q\. Sun, S\. Nielsen, V\. Richard, H\. Goda, I\. Tymchenko, N\. Nguyen, H\. Lee, M\. Ashiga, S\. Kotyan, S\. Kuroki, and T\. ClanuwatSakana fugu technical report\.External Links:2606\.21228,[Link](https://arxiv.org/abs/2606.21228)Cited by:[§A\.3](https://arxiv.org/html/2610.02762#A1.SS3.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.02762#S1.p1.1),[§6\.2](https://arxiv.org/html/2610.02762#S6.SS2.p3.1)\.
- Varshneyet al\.\(2026\)T\. Varshney, A\. Surla, M\. Xu, G\. V\. Krishnan, M\. Jeblick, D\. Austin, N\. Vaidya, and D\. OnofrioLLM router: rethinking routing with prefill activations\.External Links:2603\.20895,[Link](https://arxiv.org/abs/2603.20895)Cited by:[item 4](https://arxiv.org/html/2610.02762#A3.I1.i4.p1.1),[item 5](https://arxiv.org/html/2610.02762#A3.I1.i5.p1.1),[Appendix C](https://arxiv.org/html/2610.02762#A3.p1.1),[Appendix C](https://arxiv.org/html/2610.02762#A3.p2.1),[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px4.p1.1),[§7](https://arxiv.org/html/2610.02762#S7.p1.1)\.
- Wanget al\.\(2024\)Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang, T\. Li, M\. Ku, K\. Wang, A\. Zhuang, R\. Fan, X\. Yue, and W\. ChenMMLU\-pro: a more robust and challenging multi\-task language understanding benchmark\.InProceedings of the 38th International Conference on Neural Information Processing Systems,NeurIPS ’24,Red Hook, NY, USA\.External Links:ISBN 9798331314385Cited by:[Table 2](https://arxiv.org/html/2610.02762#A1.T2.2.9.2.1.1)\.
- Yanget al\.\(2026\)P\. Yang, W\. Chen, T\. Yang, P\. Feng, J\. Xing, W\. Guo, Y\. Yao, Y\. Han, H\. Li, X\. Wang, Z\. WANG, J\. Xiao, A\. Yang, L\. Ai, E\. Yang, and T\. SHITwinRouterBench: fast static and live dynamic evaluation for realistic agentic LLM routing\.InRLEval: Methods and Reinforcement Learning Environments for Evaluating AI Agents,External Links:[Link](https://openreview.net/forum?id=UWvHJtnmc2)Cited by:[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px2.p1.1)\.
- Yaoet al\.\(2025\)S\. Yao, N\. Shinn, P\. Razavi, and K\. R\. Narasimhanτ\\tau\-Bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=roNSXZpUDN)Cited by:[Table 2](https://arxiv.org/html/2610.02762#A1.T2.2.19.2.1.1)\.
- Zhanget al\.\(2025a\)Y\. Zhang, H\. Li, J\. Chen, H\. Zhang, P\. Ye, L\. Bai, and S\. HuBeyond gpt\-5: making llms cheaper and better via performance\-efficiency optimized routing\.InProceedings of the 2025 7th International Conference on Distributed Artificial Intelligence,DAI ’25,New York, NY, USA,pp\. 122–129\.External Links:ISBN 9798400722752,[Link](https://doi.org/10.1145/3772429.3772445),[Document](https://dx.doi.org/10.1145/3772429.3772445)Cited by:[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2025b\)Y\. Zhang, H\. Li, C\. Wang, L\. Chen, Q\. Zhang, P\. Ye, S\. Feng, D\. Wang, Z\. Wang, X\. Wang, J\. Xu, L\. Bai, W\. Ouyang, and S\. HuThe avengers: a simple recipe for uniting smaller language models to challenge proprietary giants\.External Links:2505\.19797,[Link](https://arxiv.org/abs/2505.19797)Cited by:[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px1.p1.1)\.
- Zhouet al\.\(2023\)J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. HouInstruction\-following evaluation for large language models\.External Links:2311\.07911,[Link](https://arxiv.org/abs/2311.07911)Cited by:[Table 2](https://arxiv.org/html/2610.02762#A1.T2.2.7.2.1.1)\.
- Zhuanget al\.\(2025\)R\. Zhuang, T\. Wu, Z\. Wen, A\. Li, J\. Jiao, and K\. RamchandranEmbedLLM: learning compact representations of large language models\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Fs9EabmQrJ)Cited by:[§2](https://arxiv.org/html/2610.02762#S2.SS0.SSS0.Px1.p1.1)\.
- Zhuoet al\.\(2025\)T\. Y\. Zhuo, V\. M\. Chien, J\. Chim, H\. Hu, W\. Yu, R\. Widyasari, I\. N\. B\. Yusuf, H\. Zhan, J\. He, I\. Paul, S\. Brunner, C\. Gong, J\. Hoang, A\. R\. Zebaze, X\. Hong, W\. Li, J\. Kaddour, M\. Xu, Z\. Zhang, P\. Yadav, N\. Jain, A\. Gu, Z\. Cheng, J\. Liu, Q\. Liu, Z\. Wang, D\. Lo, B\. Hui, N\. Muennighoff, D\. Fried, X\. Du, H\. de Vries, and L\. V\. WerraBigCodeBench: benchmarking code generation with diverse function calls and complex instructions\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=YrycTjllL0)Cited by:[Table 2](https://arxiv.org/html/2610.02762#A1.T2.2.6.2.1.1)\.

## Appendix ASetup

### A\.1Dataset

Table 2:Data sources in our training and evaluation datasets, with the number of training and evaluation samples from each source\.SamplesCategorySourceDescriptionTrainEvalCodingHumanEval\([Chen et al\., 2021](https://arxiv.org/html/2610.02762#bib.bib1)\)Python function synthesis with unit tests82822525HumanEvalFix\([Muennighoff et al\., 2024](https://arxiv.org/html/2610.02762#bib.bib10)\)Bug\-fix coding tasks with unit tests82822525MBPP\([Odena et al\., 2021](https://arxiv.org/html/2610.02762#bib.bib2)\)Short Python programming tasks1501502525BigCodeBench\([Zhuo et al\., 2025](https://arxiv.org/html/2610.02762#bib.bib3)\)Harder practical coding scenarios1501502525InstructionIFEval\([Zhou et al\., 2023](https://arxiv.org/html/2610.02762#bib.bib4)\)Instruction\-following and formatting constraints150150100100KnowledgeMMLU\([Hendrycks et al\., 2021a](https://arxiv.org/html/2610.02762#bib.bib5)\)General knowledge MCQ1501503434MMLU\-Pro\([Wang et al\., 2024](https://arxiv.org/html/2610.02762#bib.bib6)\)Harder MMLU\-style benchmark1501503333GPQA\([Rein et al\., 2024](https://arxiv.org/html/2610.02762#bib.bib7)\)Graduate\-level domain QA99993333MathMATH\([Hendrycks et al\., 2021b](https://arxiv.org/html/2610.02762#bib.bib62);[Lightman et al\., 2024](https://arxiv.org/html/2610.02762#bib.bib8)\)Competition\-style math reasoning1501505050GSM8K\([Cobbe et al\., 2021](https://arxiv.org/html/2610.02762#bib.bib9)\)Grade\-school word problems1501505050QANatural Questions\([Kwiatkowski et al\., 2019](https://arxiv.org/html/2610.02762#bib.bib11)\)Open\-domain factual QA1501505050TriviaQA\([Joshi et al\., 2017](https://arxiv.org/html/2610.02762#bib.bib12)\)Trivia\-style short\-answer QA1491495050OfficeDataBench\([Osés Grijalba et al\., 2024](https://arxiv.org/html/2610.02762#bib.bib13)\)Office/data\-analysis tasks1501503030SpreadsheetBench\([Ma et al\., 2024](https://arxiv.org/html/2610.02762#bib.bib14)\)Spreadsheet manipulation/reasoning1601604040GDPval\([Patwardhan et al\., 2026](https://arxiv.org/html/2610.02762#bib.bib15)\)Knowledge\-work tasks1201203030Tool useBFCL\([Patil et al\., 2025](https://arxiv.org/html/2610.02762#bib.bib16)\)Function/API tool\-use2502505050Tau\-bench\([Yao et al\., 2025](https://arxiv.org/html/2610.02762#bib.bib17)\)Interactive tool\-use tasks1001005050HLEHumanity’s Last Exam\([Center for AI Safety et al\., 2026](https://arxiv.org/html/2610.02762#bib.bib22)\)Difficult exam\-style reasoning \(tools allowed\)116116100100Total2,5082\{,\}508800800Table[2](https://arxiv.org/html/2610.02762#A1.T2)lists the sources for our dataset, grouped into eight diverse categories\. We generate each model’s response using the provider\-default temperature\. All benchmarks have verifiable rewards except for GDPval, which comprises just 3\.8% of the evaluation set, and Humanity’s Last Exam, where the judge only has to compare the final answer with the answer key\. For GDPval, we use the provided rubric, counting any response with a score of at least 0\.5 as a pass\. For Humanity’s Last Exam, each model answers in an agentic loop with web search using the Tavily API and a sandboxed Python interpreter, for up to 15 tool calls\. We block any search results from[huggingface\.co](https://huggingface.co/),[lastexam\.ai](https://lastexam.ai/), and[github\.com](https://github.com/), which may contain answers\. Both of these datasets are judged using Opus 4\.8\.

### A\.2Model set

Our first model set, selected based on the frontier in June 2026, comprises:

1. 1\.Claude Haiku 4\.5
2. 2\.Claude Sonnet 4\.6
3. 3\.Claude Opus 4\.8
4. 4\.GPT 5\.4
5. 5\.GPT 5\.4 Mini
6. 6\.GPT 5\.5
7. 7\.GPT 5\.5 \(medium\)
8. 8\.gpt\-oss\-20b \(low\)
9. 9\.gpt\-oss\-120b
10. 10\.DeepSeek V3
11. 11\.DeepSeek V4 Flash
12. 12\.DeepSeek V4 Flash \(thinking\)
13. 13\.DeepSeek V4 Pro
14. 14\.DeepSeek V4 Pro \(thinking\)
15. 15\.GLM\-5\.1
16. 16\.GLM\-5\.2
17. 17\.GLM\-5\.2 \(high\)
18. 18\.Gemini 3\.1 Flash
19. 19\.Gemini 3\.1 Flash \(low\)
20. 20\.Gemini 3\.1 Pro

Our second model set, revised in August 2026, incorporates six new model releases:

1. 1\.DeepSeek V4 Flash 07\-31 \(thinking\)
2. 2\.GPT 5\.6 Luna \(high\)
3. 3\.Gemini 3\.7 Flash
4. 4\.Grok 4\.6 \(high\)
5. 5\.GPT 5\.6 Sol \(xhigh\)
6. 6\.Claude Opus 5 \(high\)

Note that we omitted Claude Fable 5 because it is substantially more expensive than any of the models in our list\.

### A\.3Commercial routers

We evaluate the following commercial routers:

1. 1\.OpenRouter:Classifies the incoming query and routes based on which models are called most frequently for such tasks by OpenRouter users in the past seven days\. We consider the medium, high, and xhigh cost bands\. We evaluate bothopenrouter/autoandopenrouter/auto\-beta, where the latter is an early\-access track with “new routing behaviors\.”
2. 2\.Azure Model Router:Analyzes the query and selects a model by “\[estimating\] which model in the pool delivers the best result for \[the\] specific prompt\.” We assess the balanced and quality settings\.
3. 3\.Not Diamond:Similar to Azure Model Router, choosing a model based on cost and estimated performance\. It offers cost and quality settings, and we evaluate both\.
4. 4\.vLLM Semantic Router:Uses a fine\-tuned ModernBERT model to assign each query to a user\-defined category, and selects the model with the highest observed accuracy on that category\. We define categories using the eight categories \(not data sources\) in our data, and we feed in all our training data from our 26\-model set\.
5. 5\.Nadir Router:Routes based on cost and estimated performance using customizable model configurations for three tiers: simple, medium, and complex\. The medium tier is optional, so the router functions as a two\- or three\-tier escalation ladder, making it the closest analog to our routing policies for comparison\. We assess two versions: two\-tier with Claude Haiku 4\.5 and Opus 5, and three\-tier with Claude Haiku 4\.5, Sonnet 5, and Opus 5\.
6. 6\.OrcaRouter:A multi\-armed bandit router described in[Bao et al\. \(2026\)](https://arxiv.org/html/2610.02762#bib.bib21)\. Its adaptive mode uses an online LinUCB algorithm to “\[converge\] on the best option from real production traffic, weighing quality, cost, latency, and reliability\.” The adaptive mode requires a warm\-up phase, so we initially send 1,000 training samples before the evaluation samples so the router has time to converge first\.

Among these, the OpenRouter routers, vLLM Semantic Router, and Nadir are not perfect comparisons with our router\. The OpenRouter router families choose the most popular model for each task type, meaning they route by popularity instead of predicted quality\. The vLLM Semantic Router only focuses on accuracy and offers no cost\-saving mechanism\. Nadir does not choose its own roster\. At the time of access, Nadir only offered Haiku 4\.5, Sonnet 5, Opus 5, and Fable 5 as available models for each tier, and the user must configure the ladder\. We used \{Haiku 4\.5, Opus 5\} and \{Haiku 4\.5, Sonnet 5, Opus 5\} because these are the configurations whose costs fall in the same range as the other commercial routers\. \{Sonnet 5, Opus 5\} would be more expensive, and Fable 5 more expensive than Opus 5\.

#### Router calling\.

We accessed each router between August 25–26, 2026\. Every router, including our own, received the same input: the query text as a single user message without tool schemas, since the routing endpoints reject or ignore them\. For each commercial router’s output, we extract the chosen model and generate our own completion using OpenRouter, grading our generated output and storing the cost\. This process ensures consistency across all evaluated routers\. For models outside our model set, we estimate the per\-model mean costc¯i\\overline\{c\}\_\{i\}from 100 queries stratified by source rather than from the full evaluation set\.

The Azure content filter rejected three queries across IFEval, MMLU, and MMLU\-Pro, so we compute its scores and our baseline comparison across the remaining 797 queries\. Orca routed 37 queries to models not available on OpenRouter\. For seven of these \(5 for gpt\-5\-search\-api and 2 for gpt\-3\.5\-turbo\-1106\), we successfully called them using the OpenAI API\. The remaining 30 \(gpt\-5\-chat\-latest, gpt\-5\.1\-chat\-latest, gpt\-5\.2\-chat\-latest, gemini\-robotics\-er\-1\.6\-preview, qwen3\-max\-preview\) are deprecated across both platforms, and we score the router on the remaining 770\. Given that these models are deprecated, it is unlikely that they would have improved Orca’s routing performance\.

#### Omitted routers\.

The Cursor, Factory, and Entelligence routers are all designed to select models per\-turn within multi\-turn coding agent sessions, whereas our routers make per\-query decisions over general traffic\. The Ramp Router does not route per query\. Merge Gateway focuses on live cost, throughput, and latency conditions, rather than per\-query predicted success rates\. Sakana Fugu\([Tang et al\., 2026](https://arxiv.org/html/2610.02762#bib.bib54)\)predicts answer success but does not expose the routing decision, making it difficult to accurately compare its underlying model costs with our router\. Its Sakana Fugu Ultra counterpart is an orchestrator that handles more than just routing\. AWS Bedrock Intelligent Prompt Routing uses a customizable two\-tier scheme, but the only available models are outdated \(Claude 3\.5, Llama 3\.x, and Amazon Nova series\)\.

### A\.4Router training

#### ID and OOD fits\.

We train the difficulty predictor in two ways\. In the in\-distribution \(ID\) setting, we fit one predictor using the full training set, which covers the same eight categories as the evaluation set\. In the out\-of\-distribution \(OOD\) setting, we fit eight predictors, each on the training data from seven categories, and score each evaluation query with the predictor that never saw that category\. The OOD scores are therefore the concatenation of eight separately fit models, and we apply a single threshold across all of them, making this setting doubly conservative: not only are we evaluating OOD, but also the eight fits are not calibrated relative to each other, so a threshold tuned for one can over\- or under\-escalate on the other\. The threshold is chosen so the router’s cost on the evaluation set equals the stated price point, which depends on whether we use realized or per\-model mean costs\. Note that this uses evaluation costs but not evaluation outcomes\.

## Appendix BDefinitions

#### Goodman\-Kruskal’s gamma\.

For two ordinal variables, consider every pair of queries that differs on both\. A pair is*concordant*if the two variables order the pair the same way, and*discordant*otherwise\.γ\\gammais the number of concordant minus discordant pairs divided by their sum\. Pairs tied on either variable are excluded\. By comparison, Spearman’sρ\\rhotreats tied values as sharing a rank, so when the dependent variable takes on fewer discrete values \(as in the case of our router, with only two models in the roster\), the maximum achievable value is3​r​\(1−r\)\\sqrt\{3r\(1\-r\)\}, whererris the escalation rate\. At 10% escalation rate, a perfect two\-model router achieves onlyρ≈0\.52\\rho\\approx 0\.52butγ=1\\gamma=1\.

#### Source AMI\.

Source adjusted mutual information \(AMI\) measures how much of the uncertainty in the router’s choice is explained by the query’s source\. We arrange the router’s choices as a table of counts, with one row per source and one column per chosen model\. Letns​in\_\{si\}be the number of queries from sourcesssent to modelii,ns=∑ins​in\_\{s\}=\\sum\_\{i\}n\_\{si\},ni=∑sns​in\_\{i\}=\\sum\_\{s\}n\_\{si\}, andn=∑s,ins​in=\\sum\_\{s,i\}n\_\{si\}\. The entropyHHof the chosen models and the mutual informationIIbetween source and chosen model are

H\\displaystyle H=−∑ininlognin,\\displaystyle=\-\\sum\_\{i\}\\frac\{n\_\{i\}\}\{n\}\\log\\frac\{n\_\{i\}\}\{n\},I\\displaystyle I=∑s∑ins​in​log⁡n​ns​ins​ni,\\displaystyle=\\sum\_\{s\}\\sum\_\{i\}\\frac\{n\_\{si\}\}\{n\}\\log\\frac\{n\\,n\_\{si\}\}\{n\_\{s\}\\,n\_\{i\}\},where terms withns​i=0n\_\{si\}=0are zero\. Since0≤I≤H0\\leq I\\leq H, the AMI is given by

I−𝔼⁡\(I\)H−𝔼⁡\(I\),\\frac\{I\-\\mathbb\{E\}\(I\)\}\{H\-\\mathbb\{E\}\(I\)\},where𝔼⁡\(I\)\\mathbb\{E\}\(I\)is the expected mutual information if the queries were shuffled at random between models while keeping every source’s size and every model’s count fixed\. This way, the AMI is 0 in expectation if knowing the source tells us nothing about the router’s picks, and 1 if the source fully determines the model\.

## Appendix CRouter Design

We explore a range of methods for predicting query difficulty, using Qwen3\.5\-27B as the base model since the Qwen family of models is known to be competitive at predicting difficulty\([Cencerrado et al\., 2025](https://arxiv.org/html/2610.02762#bib.bib29);[Lee et al\., 2025](https://arxiv.org/html/2610.02762#bib.bib30);[Varshney et al\., 2026](https://arxiv.org/html/2610.02762#bib.bib31)\)\. We choose the 27B variant because a full 27B prefill is more costly than a smaller probe, but still a fraction of the inference cost of any model in the roster\. We encode each query using Qwen’s chat template\. We analyze a range of methods using final\-token activations to estimate query difficulty:

1. 1\.Linear probe:Train a single linear head on a layer’s activations to predict the failure rate\([Lee et al\., 2025](https://arxiv.org/html/2610.02762#bib.bib30)\);
2. 2\.Difference\-of\-means probe:Compute the mean\-correct minus mean\-incorrect activation direction for each model, and average the projections\([Cencerrado et al\., 2025](https://arxiv.org/html/2610.02762#bib.bib29)\);
3. 3\.Linear RankNet:Fit a linear scoring direction using source\-balanced pairs of questions whose failure rates differ by at least 0\.3 and optimize the pairwise logistic ranking loss\([Burges et al\., 2005](https://arxiv.org/html/2610.02762#bib.bib32)\);
4. 4\.PCA logistic probes:For each model, select an upper\-half layer using Fisher separation, reduce it to 100 principal components, and fit a correctness classifier, outputting the average over all model pass probabilities\([Varshney et al\., 2026](https://arxiv.org/html/2610.02762#bib.bib31)\);
5. 5\.SharedTrunkNet:Concatenate all model\-specific PCA\-100 representations and train a neural network across ten seeds, averaging the best five ensembles\([Varshney et al\., 2026](https://arxiv.org/html/2610.02762#bib.bib31)\)\.

Table 3:Comparison between difficulty prediction methods by global and mean per\-sourceγ⁡\(d^,d\)\\gamma\(\\hat\{d\},d\), the Goodman\-Kruskal correlation between predicted and true query difficulty\. We consider both the in\-distribution \(ID\) and out\-of\-distribution \(OOD\) fits, described in Appendix[A](https://arxiv.org/html/2610.02762#A1)\. Confidence intervals are 95% jackknife intervals over evaluation queries\. Bold marks the highest point estimate in each column\.IDOODMethodGlobalSourceGlobalSourceLinear probe0\.290\.29±\\pm0\.050\.050\.090\.09±\\pm0\.070\.070\.090\.09±\\pm0\.050\.050\.050\.05±\\pm0\.070\.07Difference of means0\.410\.41±\\pm0\.040\.040\.270\.27±\\pm0\.070\.07−0\.03\-0\.03±\\pm0\.050\.050\.250\.25±\\pm0\.070\.07RankNet0\.280\.28±\\pm0\.050\.050\.280\.28±\\pm0\.070\.070\.23\\boldsymbol\{0\.23\}±\\pm0\.050\.050\.090\.09±\\pm0\.080\.08PCA logistic0\.580\.58±\\pm0\.040\.040\.380\.38±\\pm0\.070\.070\.140\.14±\\pm0\.050\.050\.29\\boldsymbol\{0\.29\}±\\pm0\.070\.07SharedTrunkNet0\.59\\boldsymbol\{0\.59\}±\\pm0\.040\.040\.39\\boldsymbol\{0\.39\}±\\pm0\.070\.070\.120\.12±\\pm0\.050\.050\.280\.28±\\pm0\.070\.07PCA logistic and SharedTrunkNet are tied within noise on each metric\. Since we cannot distinguish between these two methods on our data, we defer to the reasoning in[Varshney et al\. \(2026\)](https://arxiv.org/html/2610.02762#bib.bib31), which argues for SharedTrunkNet because it shares information across targets, calibrates scores on a comparable scale, and computes correctness probabilities in a single forward pass\.

Interestingly, the highest achievable globalγ⁡\(d^,d\)\\gamma\(\\hat\{d\},d\)is substantially lower OOD than ID, whereas the per\-source correlation suffers less\. This observation suggests that the primary benefit of seeing a source in training is learning its average difficulty, since the probe’s ability to order queries within a source generalizes relatively well\.

## Appendix DSupplementary Tables

This section contains supplementary tables to the main text\.

Table 4:Costs and accuracies of commercial routers on our evaluation dataset\.*Chosen*is the number of distinct models the router chose, and*Roster*is the number of models in its roster\. OR: OpenRouter; ND: Not Diamond; SR: Semantic Router\.ModelsRouterCost \($/1k q\.\)Acc\. \(%\)ChosenRosterOR auto \(med\)42\.9542\.9576\.176\.177415415∗\*OR auto \(high\)68\.8368\.8377\.977\.91010415415∗\*OR auto \(xhigh\)59\.0459\.0478\.578\.577415415∗\*OR auto\-β\\beta\(med\)9\.549\.5473\.573\.577415415∗\*OR auto\-β\\beta\(high\)23\.5023\.5072\.572\.566415415∗\*OR auto\-β\\beta\(xhigh\)56\.6256\.6279\.879\.81010415415∗\*Azure \(balanced\)15\.6915\.6972\.872\.813132121Azure \(quality\)52\.0952\.0979\.379\.310102121ND \(cost\)22\.9122\.9172\.572\.555132132ND \(quality\)27\.4127\.4177\.277\.255132132vLLM SR68\.9768\.9780\.980\.9662626Nadir \(2\-tier\)119\.87119\.8781\.181\.12222†\\daggerNadir \(3\-tier\)114\.83114\.8380\.680\.63333†\\daggerOrca \(adaptive\)29\.2329\.2371\.071\.09696187187
- ∗\*The roster size for OpenRouter routers is unknown, and 415 reflects the full pool of models available on OpenRouter at the time of evaluation\.
- †\\daggerNadir’s roster size equals the number of tiers we configured, not the full list of models the platform supports\.

Table 5:Our router evaluated as in Table[1](https://arxiv.org/html/2610.02762#S4.T1), using accuracy over the entire evaluation set with realized generation costs\.*ID*and*OOD*are the in\-distribution and out\-of\-distribution fits\.*Difficulty oracle*is a router with perfect knowledge of query difficulty that escalates the hardest queries first\. Intervals are 95% paired Wald intervals\. A bold point estimate indicates that the router is not significantly below the random router\.Δ\\Deltavs\. random router \(pp\)Price pointRandom acc\. \(%\)Ours \(ID\)Ours \(OOD\)Difficulty oracleGemini 3\.7 Flash→\\rightarrowClaude Opus 5 \(high\)$10/1k q\.81\.281\.2−0\.1\\boldsymbol\{\-0\.1\}±\\pm0\.30\.3−0\.2\\boldsymbol\{\-0\.2\}±\\pm0\.20\.2−0\.1\\boldsymbol\{\-0\.1\}±\\pm0\.10\.1$20/1k q\.81\.381\.3−0\.2\\boldsymbol\{\-0\.2\}±\\pm0\.40\.4−0\.7\\boldsymbol\{\-0\.7\}±\\pm0\.70\.7−0\.2\\boldsymbol\{\-0\.2\}±\\pm0\.20\.2$40/1k q\.81\.581\.5−0\.4\\boldsymbol\{\-0\.4\}±\\pm0\.80\.8−0\.7\\boldsymbol\{\-0\.7\}±\\pm1\.01\.0\+0\.2\\boldsymbol\{\+0\.2\}±\\pm0\.90\.9gpt\-oss\-20b \(low\)→\\rightarrowClaude Opus 5 \(high\)$10/1k q\.57\.457\.4−1\.5\-1\.5±\\pm0\.50\.5−0\.7\\boldsymbol\{\-0\.7\}±\\pm0\.70\.7−1\.9\-1\.9±\\pm0\.20\.2$20/1k q\.59\.359\.3−2\.8\-2\.8±\\pm0\.80\.8−1\.9\-1\.9±\\pm1\.01\.0−3\.8\-3\.8±\\pm0\.50\.5$40/1k q\.63\.263\.2−4\.6\-4\.6±\\pm1\.31\.3−0\.7\\boldsymbol\{\-0\.7\}±\\pm1\.71\.7−6\.4\-6\.4±\\pm1\.11\.1Table 6:Comparison between commercial routers and random routers at matched costs, as in Table[1](https://arxiv.org/html/2610.02762#S4.T1), but also matched to models available at the time of each commercial router’s last update\. Every router’s roster includes the GPT 5\.6x series or models released after, and some include Gemini 3\.7 Flash, which was released later\. Commercial routers whose rosters do not contain Gemini 3\.7 Flash at the time of access are compared against \{GPT 5\.6 Luna \(high\), GPT 5\.6 Sol \(xhigh\)\}, while those that contain it are compared against \{Gemini 3\.7 Flash, Opus 5 \(high\)\}\. The shaded cell in each row is the comparison matched to that router’s roster\. A bold point estimate indicates that the router is not significantly below the random router\. Five routers are within noise of a random router using models available at the time, and all other routers underperform\.Δ\\Deltavs\. random router \(pp\)RouterCost \($/1k q\.\)Acc\. \(%\)Luna / SolFlash / OpusOR auto \(med\)42\.9542\.9576\.176\.1−3\.9\-3\.9±\\pm2\.22\.2−5\.5\-5\.5±\\pm2\.12\.1OR auto \(high\)68\.8368\.8377\.977\.9−2\.9\-2\.9±\\pm2\.32\.3−4\.0\-4\.0±\\pm2\.02\.0OR auto \(xhigh\)59\.0459\.0478\.578\.5−2\.2\\boldsymbol\{\-2\.2\}±\\pm2\.22\.2−3\.3\-3\.3±\\pm2\.02\.0OR auto\-β\\beta\(med\)9\.549\.5473\.573\.5−4\.1\-4\.1±\\pm2\.52\.5−7\.7\-7\.7±\\pm2\.52\.5OR auto\-β\\beta\(high\)23\.5023\.5072\.572\.5−6\.1\-6\.1±\\pm2\.32\.3−8\.8\-8\.8±\\pm2\.52\.5OR auto\-β\\beta\(xhigh\)56\.6256\.6279\.879\.8−1\.0\\boldsymbol\{\-1\.0\}±\\pm1\.81\.8−2\.0\-2\.0±\\pm1\.91\.9Azure \(balanced\)15\.6915\.6972\.872\.8−5\.1\-5\.1±\\pm2\.52\.5−8\.4\-8\.4±\\pm2\.62\.6Azure \(quality\)52\.0952\.0979\.379\.3−1\.3\\boldsymbol\{\-1\.3\}±\\pm1\.71\.7−2\.2\-2\.2±\\pm1\.91\.9ND \(cost\)22\.9122\.9172\.572\.5−6\.0\-6\.0±\\pm2\.52\.5−8\.8\-8\.8±\\pm2\.72\.7ND \(quality\)27\.4127\.4177\.277\.2−1\.6\\boldsymbol\{\-1\.6\}±\\pm2\.12\.1−4\.1\-4\.1±\\pm2\.22\.2vLLM SR68\.9768\.9780\.980\.9\+0\.1\\boldsymbol\{\+0\.1\}±\\pm2\.12\.1−1\.0\\boldsymbol\{\-1\.0\}±\\pm1\.81\.8Nadir \(2\-tier\)119\.87119\.8781\.181\.1\+0\.4\\boldsymbol\{\+0\.4\}±\\pm2\.42\.4−1\.4\-1\.4±\\pm1\.11\.1Nadir \(3\-tier\)114\.83114\.8380\.680\.6−0\.1\\boldsymbol\{\-0\.1\}±\\pm2\.42\.4−1\.8\-1\.8±\\pm1\.21\.2Orca \(adaptive\)29\.2329\.2371\.071\.0−8\.0\-8\.0±\\pm2\.62\.6−10\.5\-10\.5±\\pm2\.82\.8Table 7:Pattern metrics for our router fit in distribution and for a difficulty oracle that escalates the hardest queries by true difficulty, at the same three price points as Table[1](https://arxiv.org/html/2610.02762#S4.T1)\. The oracle’sγ⁡\(d,a\)\\gamma\(d,a\)is 1 by construction\. Intervals are 95% jackknife intervals over evaluation queries\. A bold point estimate indicates a “pass,” as in Table[1](https://arxiv.org/html/2610.02762#S4.T1)\.DifficultyblindnessLengthreversalSemantic matchingRouterγ⁡\(d,a\)\\gamma\(d,a\)γ⁡\(ℓ,c\)\\gamma\(\\ell,c\)Source AMIWithin\-sourceγ⁡\(d,a\)\\gamma\(d,a\)Ours, ID \($10/1k q\.\)\+0\.81\\boldsymbol\{\+0\.81\}±\\pm0\.080\.08\+0\.92\\boldsymbol\{\+0\.92\}±\\pm0\.040\.040\.460\.46±\\pm0\.070\.07\+0\.24\+0\.24±\\pm0\.290\.29Ours, ID \($20/1k q\.\)\+0\.76\\boldsymbol\{\+0\.76\}±\\pm0\.060\.06\+0\.67\\boldsymbol\{\+0\.67\}±\\pm0\.120\.120\.490\.49±\\pm0\.070\.07\+0\.20\\boldsymbol\{\+0\.20\}±\\pm0\.190\.19Ours, ID \($40/1k q\.\)\+0\.74\\boldsymbol\{\+0\.74\}±\\pm0\.050\.05\+0\.61\\boldsymbol\{\+0\.61\}±\\pm0\.080\.080\.560\.56±\\pm0\.060\.06\+0\.31\\boldsymbol\{\+0\.31\}±\\pm0\.180\.18Difficulty oracle \($10/1k q\.\)\+1\.00\\boldsymbol\{\+1\.00\}±\\pm0\.000\.00\+0\.25\\boldsymbol\{\+0\.25\}±\\pm0\.270\.270\.190\.19±\\pm0\.110\.11\+1\.00\\boldsymbol\{\+1\.00\}±\\pm0\.000\.00Difficulty oracle \($20/1k q\.\)\+1\.00\\boldsymbol\{\+1\.00\}±\\pm0\.000\.00\+0\.31\\boldsymbol\{\+0\.31\}±\\pm0\.150\.150\.250\.25±\\pm0\.080\.08\+1\.00\\boldsymbol\{\+1\.00\}±\\pm0\.000\.00Difficulty oracle \($40/1k q\.\)\+1\.00\\boldsymbol\{\+1\.00\}±\\pm0\.000\.00\+0\.51\\boldsymbol\{\+0\.51\}±\\pm0\.090\.090\.310\.31±\\pm0\.070\.07\+1\.00\\boldsymbol\{\+1\.00\}±\\pm0\.000\.00
## Appendix ESupplementary Figures

This section contains supplementary figures to the main text\.

Figure 6:Negative log\-likelihood \(NLL\) of the MIRT model as the number of interaction dimensionskkgrows\. The fits use observed pass rates \(*outcomes*\), Qwen3\.5\-27B activations \(*activations*\), or a combination of these \(*both*\)\.k=0k=0is the Rasch model\. 2 standard error bars do not exceed 0\.02, and the lines for*outcomes*and*both*are identical\.Figure 7:Standalone costs and accuracies of each model in the 26\-model set\. The gray line is the cost\-accuracy frontier, which linearly interpolates between models on the Pareto frontier\. gpt\-oss\-20b \(low\) is marked in pink, Gemini 3\.7 Flash in yellow, and Opus 5 \(high\) in green\. Error bars are 95% confidence intervals over queries\.Figure 8:Histogram of average answer lengthℓj\\ell\_\{j\}in characters for each dataset\. Most of RouterBench’s queries are multiple\-choice questions\. Our dataset spans the widest range of answer lengths, which may be why no previous evaluation reported length reversal\.\(a\)Easy\-band accuracy against easy\-band cost\.\(b\)Easy\-band accuracy gap with random routing\.
Figure 9:\(a\) is a replication of Figure[5](https://arxiv.org/html/2610.02762#S6.F5)\(a\) with \{gpt\-oss\-20b \(low\), Opus 5 \(high\)\} as the roster\. \(b\) shows the improvement of difficulty\-based routing methods against the random routing baseline on the easy band\. Note that the hard\-band escalation against easy\-band cost figure is the same shape as in Figure[5](https://arxiv.org/html/2610.02762#S6.F5)\(b\), since we are using the same router, just with a different roster\. On \(a\), 95% half\-widths do not exceed 3\.6 pp; on \(b\), 95% half\-widths do not exceed 2\.5 pp, and both ours \(ID\) and difficulty oracle achieve statistically significant accuracy gaps when easy\-band costs range from $6–139/1k queries, while ours \(OOD\) achieves significant gains over most of $11–64/1k queries\.Figure 10:Scaling ofγ⁡\(d^,d\)\\gamma\(\\hat\{d\},d\)predictions using the SharedTrunkNet architecture as a function of the training size\. We plot both global and source correlations ID and OOD, with 95% jackknife confidence intervals\. For points trained on less than the full training set, we average over three seeds\.Figure 11:Observed versus predicted pass rates for each model using the Rasch model\. We group evaluation queries into ten bins by fitted difficulty, so each point represents a difficulty band\. Error bars are 95% bootstrapped confidence intervals\. Because all models are evaluated on the same queries with the same difficulty predictions, each predicted line is linearly offset in logit space\. The fact that each predicted line closely matches observed outcomes indicates that the Rasch model fits outcomes well\.\(a\)20\-model set: standalone models\.\(b\)20\-model set: router accuracies\.\(c\)RouterBench: standalone models\.\(d\)RouterBench: router accuracies\.\(e\)SPROUT: standalone models\.\(f\)SPROUT: router accuracies\.\(g\)LLMRouterBench: standalone models\.\(h\)LLMRouterBench: router accuracies\.
Figure 12:Replication of Figures[7](https://arxiv.org/html/2610.02762#A5.F7)and[4](https://arxiv.org/html/2610.02762#S6.F4)for the 20\-model set, RouterBench, SPROUT, and LLMRouterBench, demonstrating that small roster sizes suffice\. Note that some models in LLMRouterBench are listed with $0\.00 inference costs\. Half\-widths for roster accuracies do not exceed 3\.5 pp \(20\-model set\), 0\.5 pp \(RouterBench\), 0\.4 pp \(SPROUT\), and 1\.5 pp \(LLMRouterBench\)\.\(a\)20\-model set: difficulty\-based escalation\.\(b\)20\-model set: length\-based escalation\.\(c\)RouterBench: difficulty\-based escalation\.\(d\)RouterBench: length\-based escalation\.\(e\)SPROUT: difficulty\-based escalation\.\(f\)SPROUT: length\-based escalation\.\(g\)LLMRouterBench: difficulty\-based escalation\.\(h\)LLMRouterBench: length\-based escalation\.
Figure 13:Replication of Figure[2](https://arxiv.org/html/2610.02762#S5.F2)for all other datasets\. We pair the cheapest model with the top\-scoring model so accuracy and cost differences are most visible\. For RouterBench, we route between Mistral 7B and GPT\-4, and we do not include the endpoint stars because they are too far from the area of focus\. The trends discussed in Section[5](https://arxiv.org/html/2610.02762#S5)hold across datasets except for RouterBench on length\-based escalation, where query outputs differ little in length so there is less headroom for accuracy hacking \(Figure[8](https://arxiv.org/html/2610.02762#A5.F8)\)\. The maximum half\-widths for each dataset are 3\.5 pp \(20\-model set\), 0\.9 pp \(RouterBench\), 0\.4 pp \(SPROUT\), and 1\.7 pp \(LLMRouterBench\)\. Escalating the 60–80 difficulty band significantly exceeds 80–100 on accuracy on every dataset except RouterBench\.\(a\)20\-model set: realized costs\.\(b\)20\-model set: uniform costs\.\(c\)RouterBench: realized costs\.\(d\)RouterBench: uniform costs\.\(e\)SPROUT: realized costs\.\(f\)SPROUT: uniform costs\.\(g\)LLMRouterBench: realized costs\.\(h\)LLMRouterBench: uniform costs\.
Figure 14:Replication of Figure[3](https://arxiv.org/html/2610.02762#S5.F3)for all other datasets\. For the three benchmarks, we split the dataset into training and evaluation halves\. 95% half\-widths do not exceed 3\.2 pp \(20\-model set\), 0\.7 pp \(RouterBench\), 0\.4 pp \(SPROUT\), and 2\.1 pp \(LLMRouterBench\)\. On the 20\-model set, difficulty routing underperforms random at $2\.50/1k queries because many of the most difficult queries also have the longest outputs, and this pattern reverses using uniform costs, an example of accuracy hacking\. On SPROUT, source\-based routing outperforms difficulty\-based routing, suggesting that its model set exhibits some noticeable specialization\.![Refer to caption](https://arxiv.org/html/2610.02762v1/source_tables_all.png)Figure 15:Heatmaps of each router’s choices over each data source\. We compare all commercial router tiers, our routers at three price points ID and OOD, and a difficulty oracle using two models at the same price points\. Models are ranked in increasing order of abilityaia\_\{i\}\. Note that router choices can only be meaningfully compared over identical rosters\.
## Appendix FModel Ability Scores

In Tables[1](https://arxiv.org/html/2610.02762#S4.T1)and[7](https://arxiv.org/html/2610.02762#A4.T7), a model’s abilityaais its score on the Artificial Analysis Intelligence Index \(AAII\) version 4\.1\.1\. Sinceγ\\gammaonly depends on how models are ordered, a small error in one score only matters if it changes the order of two models that a router actually uses\.

For every chosen model, we find the AAII listing that best matches the reasoning effort\. Table[8](https://arxiv.org/html/2610.02762#A6.T8)lists the 26 models from our model set\. As discussed in Appendix[A](https://arxiv.org/html/2610.02762#A1), we store each commercial router’s chosen model on each query and call the model on OpenRouter using our generation configuration to ensure fairness\. For some models, we send no effort setting and get the provider’s default, and we try to infer the default from the model’s output length\. For example, GPT\-5\.5 with no setting writes about as much as when we ask for medium effort \(1,323 vs 1,311 tokens on average\), so we give it the medium score\. Table[9](https://arxiv.org/html/2610.02762#A6.T9)lists the other 76 models that commercial routers picked\. For these, we do not have output\-length evidence, so when the model ID does not name an effort, we use the default effort listed on AA\.

We match AAII scores for each model using the first available option:

1. 1\.Exact:AA scores this model at the same effort\.
2. 2\.Only published effort:AA only includes one score for the model across effort levels, so we use that score\.
3. 3\.Default effort:The model ID does not name an effort, so we use the default AA effort\.
4. 4\.Interpolated:AA does not list the model at our chosen effort level, so we take the midpoint of the effort levels just above and below it\.
5. 5\.Converted from v4\.3\.2:AA only provides v4\.3\.2 scores, and we convert to a v4\.1\.1 score usingv4\.1\.1=−2\.21\+1\.72⋅v4\.3\.2\\text\{v4\.1\.1\}=\-2\.21\+1\.72\\cdot\\text\{v4\.3\.2\}, which we fit on 26 models that both versions score\.
6. 6\.Converted from v4\.1:We follow the same procedure as for v4\.3\.2, converting viav4\.1\.1=v4\.1\+1\.00\\text\{v4\.1\.1\}=\\text\{v4\.1\}\+1\.00, which is the average gap across nine models that both versions score\.
7. 7\.Stand\-in:AA does not score the model, so we use the closest model with a score\.
8. 8\.Rank only:Only applies to Mistral NeMo, which has no AA score in any version\. Our capability estimates for it indicate that it is the weakest model chosen by Not Diamond, so we give it a score of 4, below that of every other model\.

Although the AAII scores are not exact, the imperfect matches cannot explain the gap inγ⁡\(d,a\)\\gamma\(d,a\)between commercial routers and our router \(Table[1](https://arxiv.org/html/2610.02762#S4.T1)\)\. Of the routed queries, 55% go to a model with an exact AA match and another 31% go to a model scored at AA’s default or only published effort\. To test how much these scores matter, we perturb every inexactly matched model’s score by an independent random amount of up to±5\\pm 5points and recomputeγ\\gammaover 1,000 draws\. Across all routers, the reported values span−0\.16\-0\.16to\+0\.18\+0\.18, and across all 1,000 draws they span−0\.19\-0\.19to\+0\.19\+0\.19\. Even under the most favorable ordering of each router’s models, no commercial router exceedsγ=\+0\.27\\gamma=\+0\.27, well below the\+0\.40\+0\.40of our router OOD and\+0\.81\+0\.81ID at $10/1k queries\.

Table 8:AAII scores for the 26 models in our model set, sorted by AAII\.*Effort*is the reasoning setting we sent,*AA entry used*is the matched Artificial Analysis listing, and*Match*is the matching method\.ModelEffortAA entry usedAAIIMatchClaude Opus 5 \(high\)highClaude Opus 5 \(high\)6161ExactGrok 4\.6 \(high\)highGrok 4\.6 \(high\)6161ExactGPT 5\.6 Sol \(xhigh\)xhighGPT\-5\.6 Sol \(xhigh\)5959ExactClaude Opus 4\.8highClaude Opus 4\.8 \(max\)5757Only published effortGemini 3\.7 FlashnoneGemini 3\.7 Flash \(high\)5656Default effortGLM\-5\.2none \(behaves like high\)GLM\-5\.2 \(max\)5353Only published effortGLM\-5\.2 \(high\)highGLM\-5\.2 \(max\)5353Only published effortDeepSeek V4 Flash 0731 \(thinking\)4k\-token budgetDeepSeek V4 Flash 0731 \(max\)5252Only published effortGPT 5\.5none \(defaults to medium\)GPT\-5\.5 \(medium\)5151ExactGPT 5\.5 \(medium\)mediumGPT\-5\.5 \(medium\)5151ExactGemini 3\.1 PrononeGemini 3\.1 Pro Preview4848ExactGPT 5\.4none \(defaults to medium\)GPT\-5\.4 \(medium\)47\.847\.8Interpolated between low and xhighGPT 5\.6 Luna \(high\)highGPT\-5\.6 Luna \(high\)4747ExactDeepSeek V4 Pronone \(reasons anyway\)DeepSeek V4 Pro \(Reasoning, High\)44\.144\.1Converted from v4\.1 \(43\.1\)DeepSeek V4 Pro \(thinking\)4k\-token budgetDeepSeek V4 Pro \(Reasoning, High\)44\.144\.1Converted from v4\.1 \(43\.1\)GLM\-5\.1noneGLM\-5\.14141ExactDeepSeek V4 Flashnone \(reasons anyway\)DeepSeek V4 Flash \(Reasoning, High\)38\.538\.5Converted from v4\.1 \(37\.5\)DeepSeek V4 Flash \(thinking\)4k\-token budgetDeepSeek V4 Flash \(Reasoning, High\)38\.538\.5Converted from v4\.1 \(37\.5\)Claude Sonnet 4\.6no thinkingClaude Sonnet 4\.6 \(Non\-reasoning, high\)3737ExactGPT 5\.4 Mininone \(defaults to medium\)GPT\-5\.4 mini \(medium\)30\.830\.8Converted from v4\.1 \(29\.8\)Gemini 3\.1 Flash \(low\)lowGemini 3\.1 Flash Lite Preview2626Only published effortGemini 3\.1 FlashnoneGemini 3\.1 Flash Lite Preview2626ExactDeepSeek V3noneDeepSeek V3\.2 \(Non\-reasoning\)25\.425\.4Converted from v4\.3\.2 \(16\)Claude Haiku 4\.5no thinkingClaude 4\.5 Haiku \(Non\-reasoning\)2424Exactgpt\-oss\-120bnone \(defaults to medium\)gpt\-oss\-120b \(medium\)19\.519\.5Interpolated between low and highgpt\-oss\-20b \(low\)lowgpt\-oss\-20b \(low\)1515Converted from v4\.3\.2 \(10\)Table 9:AAII scores for each model outside our model set chosen by a commercial router, sorted by AAII\.*Model ID*is the ID returned by the router,*Queries*is the number of queries the model answered across all commercial routers,*AA entry used*is the matched Artificial Analysis listing, and*Match*is the matching method\.Model IDQueriesAA entry usedAAIIMatchclaude\-fable\-511Claude Fable 56262Default effortglm\-5\.355GLM\-5\.3 \(max\)6060Default effortkimi\-k3735735Kimi K3 \(max\)6060Default effortqwen3\.8\-max11Qwen3\.8 Max5858Exactgpt\-5\.6\-sol\-high1,7461,746GPT\-5\.6 Sol \(high\)5757Exactmuse\-spark\-1\.233Muse Spark 1\.2 \(xhigh\)5757Default effortgpt\-5\.5\-pro1010GPT\-5\.5 \(xhigh\)5656Stand\-in: the non\-Pro version, a lower boundgrok\-4\.5202202Grok 4\.5 \(high\)5656Default effortclaude\-opus\-4\-777Claude Opus 4\.7 \(max\)5555Default effortclaude\-sonnet\-5451451Claude Sonnet 5 \(max\)5555Only published effortdeepseek\-v4\-pro\-08131616DeepSeek V4 Pro 0813 \(max\)5353Exactmuse\-spark\-1\.166Muse Spark 1\.1 \(xhigh\)5353Default effortgpt\-5\.4\-pro1010GPT\-5\.4 \(xhigh\)52\.452\.4Stand\-in: the non\-Pro version, a lower bound, converted from v4\.1 \(51\.4\)deepseek\-v4\-flash\-vision\-exp88DeepSeek V4 Flash 0731 \(max\)5252Stand\-in: the text model it matchesgemini\-3\.6\-flash66Gemini 3\.6 Flash5252Exactqwen3\.8\-27b2828Qwen3\.8 27B \(xhigh\)5252Default effortgemini\-3\.1\-pro\-preview\-customtools11Gemini 3\.1 Pro Preview4848Stand\-in: the same model without tool tuninggemini\-pro\-latest77Gemini 3\.1 Pro Preview4848Exactgemini\-3\.5\-flash1818Gemini 3\.5 Flash \(medium\)4747Only published effortgpt\-5\.6\-terra\-medium136136GPT\-5\.6 Terra \(medium\)4747Exactqwen3\.7\-max1111Qwen3\.7 Max4747Exactgpt\-5\.3\-codex1010GPT\-5\.3 Codex \(xhigh\)4646Default effortKimi\-K2\.611Kimi K2\.64545Exactminimax\-m399MiniMax\-M34545Exactqwen3\.6\-plus66Qwen3\.6 Plus44\.344\.3Converted from v4\.3\.2 \(27\)gpt\-5\.21515GPT\-5\.2 \(xhigh\)4343Default effortgpt\-5\.2\-pro88GPT\-5\.2 \(xhigh\)4343Stand\-in: the non\-Pro version, a lower boundkimi\-k2\.7\-code77Kimi K2\.7 Code4343Exacthy366Hy34242Exactglm\-522GLM\-54141Exactgpt\-5\.2\-codex66GPT\-5\.2 Codex \(xhigh\)4141Default effortgpt\-5\.4\-nano2525GPT\-5\.4 nano \(xhigh\)4040Default effortclaude\-opus\-4\-611Claude Opus 4\.6 \(Non\-reasoning, high\)3939Default effortgemini\-3\-flash\-preview273273Gemini 3 Flash \(reasoning\)3939Stand\-in: the released versionminimax\-m2\.71414MiniMax\-M2\.73939Exactqwen3\.7\-plus44Qwen3\.7 Plus3939Exactgrok\-4\.20369369Grok 4\.20 0309 v23838Exactgrok\-4\.388Grok 4\.3 \(medium\)3838Default effortgemini\-3\.5\-flash\-lite55Gemini 3\.5 Flash\-Lite3737Exactgpt\-5\.188GPT\-5\.1 \(high\)3737Default effortclaude\-opus\-4\-544Claude Opus 4\.5 \(Non\-reasoning\)3636Default effortgpt\-5\.1\-codex77GPT\-5\.1 Codex \(high\)3636Default effortglm\-4\.777GLM 4\.73535Exactgpt\-5100100GPT\-5 \(high\)3535Default effortgpt\-5\-pro88GPT\-5 \(high\)3535Stand\-in: the non\-Pro version, a lower boundqwen3\.5\-27b55Qwen3\.5\-27B3535Exactminimax\-m2\.51515MiniMax\-M2\.53434Exactqwen3\.5\-397b\-a17b77Qwen3\.5 397B A17B3434Exactqwen3\.5\-plus\-2026042066Qwen3\.5 397B A17B3434Stand\-in: the open model behind the hosted APIqwen3\.5\-122b\-a10b88Qwen3\.5 122B A10B3333Exactqwen3\.6\-35b\-a3b1313Qwen3\.6 35B A3B3232Exactqwen3\.6\-flash2020Qwen3\.6 35B A3B3232Stand\-in: the open model behind the hosted APIgpt\-5\.1\-codex\-mini66GPT\-5\.1\-Codex\-Mini3131Exactclaude\-sonnet\-4\.54040Claude 4\.5 Sonnet \(Non\-reasoning\)30\.530\.5Converted from v4\.3\.2 \(19\)qwen3\.5\-35b\-a3b1212Qwen3\.5 35B A3B \(Reasoning\)30\.530\.5Converted from v4\.3\.2 \(19\)gemma\-4\-31B\-it55Gemma 4 31B3030Exactgemini\-2\.5\-pro44Gemini 2\.5 Pro2626Exactgemini\-3\.1\-flash\-lite44Gemini 3\.1 Flash Lite Preview2626Stand\-in: the preview versiongemma\-4\-26b\-a4b\-it1818Gemma 4 26B A4B2626Exactgpt\-5\-mini101101GPT\-5 mini \(high\)2626Default effortqwen3\-max66Qwen3 Max2424Exactqwen3\.5\-flash\-02\-231010Qwen3\.5 35B A3B \(Non\-reasoning\)23\.623\.6Stand\-in: non\-reasoning variant, converted from v4\.3\.2 \(15\)glm\-4\.644GLM 4\.62323Exactqwen3\-vl\-235b\-a22b\-thinking1010Qwen3 VL 235B A22B \(Reasoning\)20\.220\.2Converted from v4\.3\.2 \(13\)gpt\-4\.166GPT\-4\.12020Exactgpt\-5\-nano2727GPT\-5 nano \(high\)2020Default effortgpt\-5\-search\-api55GPT\-5 \(minimal\)16\.716\.7Stand\-in: GPT\-5 without reasoning, converted from v4\.3\.2 \(11\)gpt\-4\.1\-mini1515GPT\-4\.1 Mini1515Exactgemini\-2\.5\-flash2424Gemini 2\.5 Flash1414Exactqwen3\-vl\-235b\-a22b\-instruct99Qwen3 VL 235B A22B Instruct1414Exactgpt\-4o\-2024\-08\-0666GPT\-4o \(Aug ’24\)11\.611\.6Converted from v4\.3\.2 \(8\)qwen3\-vl\-8b\-thinking66Qwen3 VL 8B \(Reasoning\)11\.611\.6Converted from v4\.3\.2 \(8\)gpt\-4o44GPT\-4o \(Nov\)1111Exactgpt\-4o\-2024\-11\-201010GPT\-4o \(Nov\)1111Exactgpt\-4\.1\-nano2121GPT\-4\.1 Nano1010Exactsonar150150Sonar99Exactgpt\-4\-turbo33GPT\-4 Turbo88Exactqwen3\-vl\-8b\-instruct66Qwen3 VL 8B Instruct88Exactgemini\-2\.5\-flash\-lite2121Gemini 2\.5 Flash Lite77Exactgpt\-4o\-mini88GPT\-4o\-mini77Exactgpt\-3\.5\-turbo77GPT\-3\.5 Turbo6\.46\.4Converted from v4\.3\.2 \(5\)gpt\-3\.5\-turbo\-110622GPT\-3\.5 Turbo6\.46\.4Converted from v4\.3\.2 \(5\)gpt\-3\.5\-turbo\-16k66GPT\-3\.5 Turbo6\.46\.4Converted from v4\.3\.2 \(5\)open\-mistral\-nemo182182none44Rank only: below every other Not Diamond pick