HyDRA: Hybrid Dynamic Routing Architecture for Heterogeneous LLM Pools

# Hybrid Dynamic Routing Architecture for Heterogeneous LLM Pools
Source: [https://arxiv.org/html/2605.17106](https://arxiv.org/html/2605.17106)
Aashna GargSiddharth Singha RoyJinu Jang Federico BrancasiShengyu Fu Microsoft \{aashnagarg, ssingharoy, jinujang, fbrancasi, shengyfu\}@microsoft\.com

###### Abstract

Production LLM deployments increasingly maintain heterogeneous model pools spanning order\-of\-magnitude cost differences\. Existing routers make binary strong\-vs\-weak decisions and couple learned parameters to specific model identities, requiring retraining whenever the catalog changes\. We presentHyDRA\(HybridDynamicRoutingArchitecture\), a framework that predicts fine\-grained, multi\-dimensional capability requirements per query and matches them against configuration\-defined model profiles via shortfall matching\. A ModernBERT encoder withK=4K\{=\}4independent sigmoid heads scores each query along*reasoning*,*code generation*,*debugging*, and*tool use*; a shortfall\-matching algorithm then selects the cheapest model whose capabilities meet the predicted requirements\. ModernBERT is chosen for its low CPU inference cost: the deployed predictor runs at86 ms median ModernBERT inference latency on CPUin production \(Table[14](https://arxiv.org/html/2605.17106#A1.T14)\), keeping routing overhead negligible relative to LLM call latency\. Crucially, the predictor is fully decoupled from the model catalog—adding or removing models requires only a configuration change, with zero retraining\. On SWE\-Bench Verified \(5\-model pool: GPT\-5\.4\-mini, Claude Haiku 4\.5, GPT\-5\.3 Codex, Claude Sonnet 4\.6, GPT\-5\.4\),HyDRA’s tunable shortfall threshold spans three useful regimes: at the peak\-quality setting it*exceeds*the always\-strong Claude Sonnet 4\.6 baseline \(75\.4% vs\. 74\.2%resolution\) at 12\.9% cost savings; at the iso\-quality setting it matches Sonnet while delivering54\.1% cost savings, a6×\\timesimprovementover our prior in\-house binary router at 9\.1%; at an aggressive setting it pushes savings to72\.5%for a 3\.2\-point quality trade\. Results generalize across LiveCodeBench, BigCodeBench, andτ\\tau\-bench\.HyDRAis deployed to all users in GitHub Copilot’s VS Code Chat auto\-mode and—to our knowledge for the first time in the LLM routing literature—demonstrates language\-invariant routing across CJK, European, and other script families\.

HyDRA: Hybrid Dynamic Routing Architecture for Heterogeneous LLM Pools

Aashna Garg Siddharth Singha Roy Jinu JangFederico BrancasiShengyu FuMicrosoft\{aashnagarg, ssingharoy, jinujang, fbrancasi, shengyfu\}@microsoft\.com

## 1Introduction

Production systems serving millions of users—code assistants, conversational agents, search copilots—now maintain pools of 10–15 LLMs, from lightweight models costing fractions of a cent per query to frontier reasoning models at 10–50×\\timesthe price\. The routing problem is straightforward: for each incoming query, select the cheapest model capable of producing a satisfactory response\.

Despite its practical importance, routing remains surprisingly underexplored\. The dominant deployed approach is infrastructure\-level load balancing—entirely blind to what the user is asking\(GitHub,[2025](https://arxiv.org/html/2605.17106#bib.bib5)\)\. Recent learned routers\(Ong et al\.,[2024](https://arxiv.org/html/2605.17106#bib.bib16); Ding et al\.,[2024](https://arxiv.org/html/2605.17106#bib.bib4); Lu et al\.,[2024](https://arxiv.org/html/2605.17106#bib.bib12)\)improve on this but share two limitations\.

Existing routers are model\-coupled\.They learnf​\(query\)→model\_idf\(\\text\{query\}\)\\to\\text\{model\\\_id\}where model identities are embedded in training labels\. When models are added, retired, or re\-priced—a monthly occurrence—the router must be retrained\.

Existing routers collapse heterogeneous capability requirements onto a single axis\.A query requiring deep multi\-step reasoning but trivial code output differs fundamentally from one needing sophisticated code generation but no reasoning, or one dominated by tool\-use orchestration\. Binary routers\(Ong et al\.,[2024](https://arxiv.org/html/2605.17106#bib.bib16)\)and scalar difficulty estimators\(Chen et al\.,[2023](https://arxiv.org/html/2605.17106#bib.bib3)\)collapse this distinction onto a single strong\-vs\-weak score\. On a single\-task benchmark like SWE\-Bench, where almost every query is reasoning\- and code\-heavy in similar proportions, the cost of this collapse is small \(§[D](https://arxiv.org/html/2605.17106#A4), Table[17](https://arxiv.org/html/2605.17106#A4.T17)\)\. The cost grows with workload heterogeneity: it shows up on tool\-heavy debugging conversations, on multilingual traffic where complexity decouples from surface form, and—most importantly—on*catalog heterogeneity*, where a router that only knows “how strong” a query is cannot exploit a mid\-tier model that happens to be best\-in\-class for one dimension \(e\.g\. a small code\-tuned model, or a debugging\-specialized one\) when it is added to the pool\. A scalar router would either ignore that model or have to be retrained; a multi\-dimensional router routes to it on the dimensions where it dominates, on day one\.

Existing routers do not address language invariance\.Production coding assistants serve a global user base, yet published learned routers\(Ong et al\.,[2024](https://arxiv.org/html/2605.17106#bib.bib16); Ding et al\.,[2024](https://arxiv.org/html/2605.17106#bib.bib4); Lu et al\.,[2024](https://arxiv.org/html/2605.17106#bib.bib12); Zhang et al\.,[2025](https://arxiv.org/html/2605.17106#bib.bib25)\)train and report exclusively on English benchmarks \(Chatbot Arena, MT\-Bench, SWE\-Bench, MMLU\), and concurrent multi\-turn\(Zhang et al\.,[2026b](https://arxiv.org/html/2605.17106#bib.bib26),[a](https://arxiv.org/html/2605.17106#bib.bib23)\)and pre\-routing\(Liu et al\.,[2026](https://arxiv.org/html/2605.17106#bib.bib11); Varshney et al\.,[2026](https://arxiv.org/html/2605.17106#bib.bib20); Madeyski,[2026](https://arxiv.org/html/2605.17106#bib.bib14)\)systems do not evaluate cross\-lingual behavior\. The closest concurrent work using the phrase “multilingual routing”\(Bandarkar et al\.,[2026](https://arxiv.org/html/2605.17106#bib.bib1)\)studies token\-level expert routing*inside*a single MoE model, not selection across a pool of separately\-priced LLMs; Routesplain\(Štorek et al\.,[2025](https://arxiv.org/html/2605.17106#bib.bib18)\)routes “multilingual” software tasks but in the sense of programming languages, not natural languages\. Routing must be language\-invariant: task complexity, not input natural language, should determine model selection\. We are not aware of a prior LLM\-pool routing system that explicitly evaluates routing consistency across natural\-language script families\.

We proposeHyDRA, built on three ideas:

1. 1\.Multi\-dimensional capability prediction\.A lightweight encoder predictsK=4K\{=\}4independent requirement scores per query: reasoning, code generation, debugging, tool use\.
2. 2\.Config\-decoupled model matching\.Model capabilities live in a YAML file, not in learned weights\. The router selects the cheapest model covering predicted requirements via shortfall matching\.
3. 3\.Language\-invariant multilingual routing\.Training on 21 languages across CJK, European, and other script families ensures routing decisions depend on task complexity, not language\.

The practical consequence: adding a model to the catalog requires editing a YAML file\. No retraining, no redeployment\.

HyDRAis the second iteration of our deployed routing system\. The first iteration—an in\-house binary strong\-vs\-weak ModernBERT classifier \(the binary router\)—shipped to production in early 2026 and serves as our most operationally meaningful baseline throughout this paper\. It is also subject to the two limitations above \(model\-coupled, dimensionally impoverished\), which directly motivated theHyDRAredesign and which we quantify against the deployed predecessor in §[7](https://arxiv.org/html/2605.17106#S7)and §[8](https://arxiv.org/html/2605.17106#S8)\.

We developHyDRAfor a commercial AI coding assistant serving tens of millions of developers\. Our contributions are:

1. 1\.Capability\-decoupled routing via shortfall matching\(§[3](https://arxiv.org/html/2605.17106#S3), §[3\.3](https://arxiv.org/html/2605.17106#S3.SS3)\)\. To our knowledge, the first router that fully decouples the learned predictor from the model catalog: predictions are over query*requirements*alongK=4K\{=\}4capability dimensions, and model selection is a configuration\-driven shortfall\-matching algorithm\. Adding, removing, or repricing a model is a YAML edit—zero retraining, zero redeployment\.
2. 2\.Multi\-dimensional capability prediction with a structured labeling pipeline\(§[4](https://arxiv.org/html/2605.17106#S4)\)\. A single ModernBERT forward pass producesKKindependent capability\-requirement scores, trained on 50,016 dual\-model LLM\-judge labels with position\-swap debiasing\.
3. 3\.Language\-invariant routing across 21 languages and 3 script families\(§[5](https://arxiv.org/html/2605.17106#S5)\)\. To our knowledge, the first deployed multilingual LLM router with reported per\-language quality and cost parity across CJK, European, and other script families\.
4. 4\.Production integration in GitHub Copilot\(§[10](https://arxiv.org/html/2605.17106#S10)\): session\-sticky routing, image hardgating, health\-aware filtering, and zero\-downtime model lifecycle management on the GitHub Copilot API \(CAPI\) infrastructure\.
5. 5\.End\-to\-end empirical validation\(§[7](https://arxiv.org/html/2605.17106#S7)\): cost\-quality Pareto sweeps on SWE\-Bench Verified \(delivering iso\-quality with the always\-strong baseline at 54\.1% savings\), cross\-benchmark generalization to LiveCodeBench, BigCodeBench, andτ\\tau\-bench, a controlled 50/50 A/B flight \(§[8\.2](https://arxiv.org/html/2605.17106#S8.SS2)\), and a two\-week production scorecard \(§[8\.2](https://arxiv.org/html/2605.17106#S8.SS2.SSSx1)\)\.

##### Note on this preprint version\.

This v1 reports the offline benchmark, competitive comparison, portability, and ablation results in full\. The production\-telemetry tables \(Tables[10](https://arxiv.org/html/2605.17106#S8.T10),[11](https://arxiv.org/html/2605.17106#S8.T11)\), the dimension\-weight numerics in §[9](https://arxiv.org/html/2605.17106#S9), and the per\-language breakdown in Appendix[F](https://arxiv.org/html/2605.17106#A6)are deferred to v2 pending the next eval\-harness run and production\-telemetry sign\-off; cells affected are marked with an em\-dash \(—\)\. The qualitative claims in the surrounding prose are preserved as design statements and should be read as forward\-looking until v2 lands\.

## 2Related Work

##### LLM Routing\.

RouteLLM\(Ong et al\.,[2024](https://arxiv.org/html/2605.17106#bib.bib16)\)trains classifiers on Chatbot Arena preference data for binary strong\-vs\-weak routing\. Their matrix factorization approach jointly embeds queries and models, coupling the router to training\-time identities\. Hybrid LLM\(Ding et al\.,[2024](https://arxiv.org/html/2605.17106#bib.bib4)\)uses a BERT difficulty predictor for binary routing\. Both are binary and model\-coupled\. ZOOTER\(Lu et al\.,[2024](https://arxiv.org/html/2605.17106#bib.bib12)\)scores each candidate via a reward model, requiringNNforward passes\.HyDRAuses a single forward pass, predicts multi\-dimensional*requirements*\(not preferences\), and stores model capabilities in configuration\.

##### Cascading and Verification\.

FrugalGPT\(Chen et al\.,[2023](https://arxiv.org/html/2605.17106#bib.bib3)\)cascades through models cheapest\-first, verifying before escalation\. AutoMix\(Madaan et al\.,[2024](https://arxiv.org/html/2605.17106#bib.bib13)\)adds self\-verification\. Both add latency proportional to cascade depth and discard partial generations\. EcoAssistant\(Zhang et al\.,[2024](https://arxiv.org/html/2605.17106#bib.bib24)\)combines cascading with solution\-level caching\.HyDRAis pre\-routing: the model is selected before generation, adding only encoder latency \(40–80ms\)\.

##### Ensembles and MoE\.

LLM\-Blender\(Jiang et al\.,[2023](https://arxiv.org/html/2605.17106#bib.bib9)\)fuses responses from multiple models \(N×N\\timescost\)\. Mixture\-of\-experts\(Jiang et al\.,[2024](https://arxiv.org/html/2605.17106#bib.bib8)\)routes tokens within one model;HyDRAroutes queries across distinct models\.

##### Concurrent multi\-turn routing\.

MTRouter\(Zhang et al\.,[2026b](https://arxiv.org/html/2605.17106#bib.bib26)\)encodes interaction history and candidate models into joint embeddings, learning an outcome estimator from logged trajectories for per\-turn model selection\. DialRouter\(Zhang et al\.,[2026a](https://arxiv.org/html/2605.17106#bib.bib23)\)uses MCTS to explore dialogue branches and trains a lightweight routing policy from search\-derived data\. Both are model\-coupled \(model identities embedded in learned parameters\) and require expensive trajectory collection\.HyDRA’s confidence\-gated sticky routing achieves multi\-turn awareness at zero compute cost\.

##### Concurrent pre\-routing approaches\.

TRouter\(Liu et al\.,[2026](https://arxiv.org/html/2605.17106#bib.bib11)\)models query\-conditioned cost and performance via latent task\-type variables, addressing cold\-start scenarios via synthetic data\. LLM Router\(Varshney et al\.,[2026](https://arxiv.org/html/2605.17106#bib.bib20)\)uses internal prefill activations as routing signals, requiring open\-weight encoder access\. Triage\(Madeyski,[2026](https://arxiv.org/html/2605.17106#bib.bib14)\)routes software engineering tasks via code health metrics\. All remain model\-coupled and single\-dimensional\.HyDRAis the only system combining multi\-dimensional prediction with full model decoupling\.

##### Adversarial robustness\.

R2A\(Tang et al\.,[2026](https://arxiv.org/html/2605.17106#bib.bib19)\)demonstrates that adversarial suffix optimization can manipulate cost\-aware routers into selecting expensive models, exposing a security surface for deployed routing systems\. We discuss implications forHyDRAin §[Limitations](https://arxiv.org/html/2605.17106#Sx1)\.

### 2\.1Competitive Analysis

Table[1](https://arxiv.org/html/2605.17106#S2.T1)comparesHyDRAagainst published routing systems across key design dimensions\.

Table 1:Competitive analysis of LLM routing systems\.HyDRAis the only system that is fully model\-decoupled \(no retraining on catalog change\), uses multi\-dimensional capability prediction, and—to our knowledge—is the first to provide multilingual routing across 21 languages\. The deployed predictor consumes only the current user message plus a coarse turn\-count signal; session\-level continuity is provided by the sticky routing policy \(§[10\.2](https://arxiv.org/html/2605.17106#S10.SS2)\), not by the predictor\.

## 3Architecture

HyDRAhas three components: a capability requirement predictor \(§[3\.1](https://arxiv.org/html/2605.17106#S3.SS1)\), model capability profiles \(§[3\.2](https://arxiv.org/html/2605.17106#S3.SS2)\), and shortfall matching \(§[3\.3](https://arxiv.org/html/2605.17106#S3.SS3)\)\. Figure[1](https://arxiv.org/html/2605.17106#S3.F1)illustrates the end\-to\-end flow\.

Input ConstructionSignal PrefixTC,ERR,FILE,URL,CMD,CODE,SHORTCurrent User Msgfull text, single turntokenized at a 512\-token capCapability PredictorModernBERT\-base149M params, ONNX dyn\. INT8 \(skip\-attn\)\[CLS\]→\\toDropout\(0\.1\)K=4K\{=\}4Sigmoid Headsσ​\(𝐰1⊤​𝐡\+b1\)\\sigma\(\\mathbf\{w\}\_\{1\}^\{\\top\}\\mathbf\{h\}\+b\_\{1\}\)σ​\(𝐰2⊤​𝐡\+b2\)\\sigma\(\\mathbf\{w\}\_\{2\}^\{\\top\}\\mathbf\{h\}\+b\_\{2\}\)σ​\(𝐰3⊤​𝐡\+b3\)\\sigma\(\\mathbf\{w\}\_\{3\}^\{\\top\}\\mathbf\{h\}\+b\_\{3\}\)σ​\(𝐰4⊤​𝐡\+b4\)\\sigma\(\\mathbf\{w\}\_\{4\}^\{\\top\}\\mathbf\{h\}\+b\_\{4\}\)r^1\\hat\{r\}\_\{1\}= \.91r^2\\hat\{r\}\_\{2\}= \.45r^3\\hat\{r\}\_\{3\}= \.88r^4\\hat\{r\}\_\{4\}= \.35ReasoningCode GenDebugTool UseShortfall MatchingModel Capability Profilesexternal configuration — no learned params∑kwk​max⁡\(0,r^k−cm,k\)\\displaystyle\\sum\_\{k\}w\_\{k\}\\max\(0,\\,\\hat\{r\}\_\{k\}\-c\_\{m,k\}\)cheapest with shortfall≤τ\\leq\\tau→\\toGPT\-5\.4 \(frontier\)τ\\tau,𝐰\\mathbf\{w\}tunable at runtime — no retrainingEnd\-to\-end latency:55ms P50/ 120ms P99 \(ONNX INT8, CPU\)

Figure 1:HyDRAarchitecture overview\.\(1\) Input Construction: a 7\-flag signal prefix is concatenated with the current user message and tokenized at a 512\-token cap\. The deployed predictor is single\-turn: prior turn text is never fed to the model; conversation position is exposed only via the coarse turn\-count bin in the signal prefix\.\(2\) Capability Predictor: ModernBERT\-base produces a \[CLS\] embedding\.\(3\) Sigmoid Heads:K=4K\{=\}4independent heads predict per\-dimension requirement scoresr^k∈\[0,1\]\\hat\{r\}\_\{k\}\\in\[0,1\]\.\(4\) Shortfall Matching: scores are compared against externally configured model profiles; the cheapest model with shortfall≤τ\\leq\\tauis selected\. The entire model catalog is decoupled from learned parameters—adding or removing models requires only editing the configuration with zero retraining\. Example scores shown are for the query “Fix the race condition in WebSocket reconnect\.”### 3\.1Capability Requirement Predictor

Given queryqq, the predictor estimatesKKscoresr^1,…,r^K∈\[0,1\]\\hat\{r\}\_\{1\},\\ldots,\\hat\{r\}\_\{K\}\\in\[0,1\]\. We use ModernBERT\-base\(Warner et al\.,[2024](https://arxiv.org/html/2605.17106#bib.bib21)\)\(149M parameters\)\. The \[CLS\] representation passes through dropout andKKindependent linear heads:

r^k=σ​\(𝐰k⊤​dropout⁡\(𝐡\[CLS\]\)\+bk\)\\hat\{r\}\_\{k\}=\\sigma\\\!\\bigl\(\\mathbf\{w\}\_\{k\}^\{\\top\}\\operatorname\{dropout\}\(\\mathbf\{h\}\_\{\\mathrm\{\[CLS\]\}\}\)\+b\_\{k\}\\bigr\)\(1\)Total added parameters:K×769=3,076K\\times 769=3\{,\}076forK=4K\{=\}4—negligible relative to the encoder\.

##### Input representation\.

Rather than routing on the current user message in isolation, we construct a compact textual input that exposes shallow query signals\. Concretely, the input is the concatenation of two parts:

1. 1\.A bracketedsignal prefixof seven binary or small\-integer flags computed from the current message and conversation length: a 4\-bin turn\-count indicator together with flags detecting error keywords, file extensions, URLs, shell/command keywords, code constructs, and very short messages\.
2. 2\.Thecurrent user message, appended directly to the signal prefix\.

The combined string is tokenized with the ModernBERT tokenizer at a 512\-token cap \(truncation enabled\) for both training and inference of the deployed multilingual model\. The deployed predictor is therefore single\-turn at the input level: prior turn text is never fed to the model, and conversation position is exposed only via the coarse turn\-count bin in the signal prefix\. We do not include assistant responses, tool outputs, or repository state in the input text—this keeps inference cheap, avoids leaking large code blobs into the router, and matches the information available to the routing service before the LLM is called\.

##### Training objective\.

Binary cross\-entropy per dimension with optional per\-dimension weighting:

ℒ=−1K​∑k=1Kαk​\[rk​log⁡r^k\+\(1−rk\)​log⁡\(1−r^k\)\]\\mathcal\{L\}=\-\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\alpha\_\{k\}\\bigl\[r\_\{k\}\\log\\hat\{r\}\_\{k\}\+\(1\{\-\}r\_\{k\}\)\\log\(1\{\-\}\\hat\{r\}\_\{k\}\)\\bigr\]\(2\)whereαk\\alpha\_\{k\}are dimension\-specific loss weights \(default 1\.0\)\. Heads use the lightweight configuration \(a single linear projectiondh→1d\_\{h\}\\to 1per dimension over the dropout\-regularized \[CLS\] vector, withK×\(dh\+1\)=4×769=3,076K\\times\(d\_\{h\}\{\+\}1\)=4\\times 769=3\{,\}076added parameters fordh=768d\_\{h\}\{=\}768\)\. Only one checkpoint is deployed to production—the multilingualHyDRA\-Multimodel fine\-tuned on the merged English\+\+multilingual corpus; its training corpus, hyperparameters, and runtime are described in full in §[5\.2](https://arxiv.org/html/2605.17106#S5.SS2)\.

### 3\.2Model Capability Profiles

Each modelmmhas𝐜m∈\[0,1\]K\\mathbf\{c\}\_\{m\}\\in\[0,1\]^\{K\}andcostm\\text\{cost\}\_\{m\}, stored in a YAML configuration file\. Profiles derive from public benchmarks via the two\-step computation in Algorithm[1](https://arxiv.org/html/2605.17106#alg1): a weighted benchmark average per dimension, then a pool\-relative affine map into the requirement predictor’s empirical score band, with stored routing weights compensated for differences in per\-dimension band width so that operator intent is preserved\. The per\-benchmark coefficientwbw\_\{b\}collapses parent\-benchmark importance, panel\-mean LLM\-judge ratings per dimension, and difficulty\-tier weighting into a single effective weight\.

Algorithm 1Capability Profile Computation0:Benchmark scores

\{sm,b\}\\\{s\_\{m,b\}\\\}for models

m∈ℳm\\in\\mathcal\{M\}, benchmarks

bb
0:Benchmark\-to\-dimension mapping

ℬ​\(k\)\\mathcal\{B\}\(k\)with weights

\{wb\}\\\{w\_\{b\}\\\}
0:Operator\-stored routing weights

\{wk\}\\\{w\_\{k\}\\\}\(per dimension\)

0:Capability profiles

\{cm,k\}\\\{c\_\{m,k\}\\\}and band\-compensated routing weights

\{w~k\}\\\{\\tilde\{w\}\_\{k\}\\\}
1:foreach dimension

kkand each model

m∈ℳm\\in\\mathcal\{M\}do

2:

rawm,k←∑b∈ℬ​\(k\)wb⋅sm,b\\text\{raw\}\_\{m,k\}\\leftarrow\\sum\_\{b\\in\\mathcal\{B\}\(k\)\}w\_\{b\}\\cdot s\_\{m,b\}\{Step 1: benchmark anchoring\}

3:endfor

4:

\[βklo,βkhi\]←\[\\beta^\{\\text\{lo\}\}\_\{k\},\\beta^\{\\text\{hi\}\}\_\{k\}\]\\leftarrowlow/high percentiles of the requirement predictor’s score distribution on a held\-out set, per dimension

kk
5:foreach dimension

kkand each model

m∈ℳm\\in\\mathcal\{M\}do

6:

cm,k←βklo\+rawm,k−minj⁡rawj,kmaxj⁡rawj,k−minj⁡rawj,k⋅\(βkhi−βklo\)c\_\{m,k\}\\leftarrow\\beta^\{\\text\{lo\}\}\_\{k\}\+\\dfrac\{\\text\{raw\}\_\{m,k\}\-\\min\_\{j\}\\text\{raw\}\_\{j,k\}\}\{\\max\_\{j\}\\text\{raw\}\_\{j,k\}\-\\min\_\{j\}\\text\{raw\}\_\{j,k\}\}\\cdot\(\\beta^\{\\text\{hi\}\}\_\{k\}\-\\beta^\{\\text\{lo\}\}\_\{k\}\)\{Step 2: pool\-relative normalization\}

7:endfor

8:

Δk←βkhi−βklo\\Delta\_\{k\}\\leftarrow\\beta^\{\\text\{hi\}\}\_\{k\}\-\\beta^\{\\text\{lo\}\}\_\{k\}\{Step 3: band\-width dim\-weight compensation \(Eq\.[4](https://arxiv.org/html/2605.17106#S3.E4)\)\}

9:

w~k←wk/Δk∑k′wk′/Δk′⋅∑k′wk′\\tilde\{w\}\_\{k\}\\leftarrow\\dfrac\{w\_\{k\}/\\Delta\_\{k\}\}\{\\sum\_\{k^\{\\prime\}\}w\_\{k^\{\\prime\}\}/\\Delta\_\{k^\{\\prime\}\}\}\\cdot\\sum\_\{k^\{\\prime\}\}w\_\{k^\{\\prime\}\}

Step 1: Benchmark anchoring\.For each dimensionkk, we compute a weighted average of relevant benchmarks\. Table[2](https://arxiv.org/html/2605.17106#S3.T2)lists the benchmark\-to\-dimension mapping\.

Table 2:Benchmark\-to\-dimension mapping used for capability profile derivation\. Each benchmark contributes only to the dimensions it can plausibly exercise \(e\.g\. BigCodeBench has no tool surface, so Tool Use is forced to 0\)\.Step 2: Pool\-relative normalization\.We pool the requirement predictor’s per\-instance score distribution on a held\-out target set and take the lower/upper percentiles per dimension as the empirical band\[βklo,βkhi\]\[\\beta^\{\\text\{lo\}\}\_\{k\},\\beta^\{\\text\{hi\}\}\_\{k\}\]\. Each model’s raw score is then affinely mapped into this band so that the weakest model in the pool pins toβklo\\beta^\{\\text\{lo\}\}\_\{k\}and the strongest toβkhi\\beta^\{\\text\{hi\}\}\_\{k\}:

cm,k=βklo\+rawm,k−minj⁡rawj,kmaxj⁡rawj,k−minj⁡rawj,k⋅\(βkhi−βklo\)\.c\_\{m,k\}=\\beta^\{\\text\{lo\}\}\_\{k\}\+\\frac\{\\text\{raw\}\_\{m,k\}\-\\min\_\{j\}\\text\{raw\}\_\{j,k\}\}\{\\max\_\{j\}\\text\{raw\}\_\{j,k\}\-\\min\_\{j\}\\text\{raw\}\_\{j,k\}\}\\cdot\(\\beta^\{\\text\{hi\}\}\_\{k\}\-\\beta^\{\\text\{lo\}\}\_\{k\}\)\.\(3\)This guarantees that capability scores live on the same scale as the requirements they will be compared against at routing time\. Because the map is affine, it preserves rank ordering and the relative spacing between models on each dimension\. Because the normalization is pool\-relative, scores re\-normalize automatically when the pool changes\.

A side\-effect of the affine mapping is that per\-dimension band widthsΔk=βkhi−βklo\\Delta\_\{k\}=\\beta^\{\\text\{hi\}\}\_\{k\}\-\\beta^\{\\text\{lo\}\}\_\{k\}differ across dimensions, so the*effective*routing impact of operator\-supplied weightswkw\_\{k\}\(§[3\.3](https://arxiv.org/html/2605.17106#S3.SS3), Eq\.[5](https://arxiv.org/html/2605.17106#S3.E5)\) iswk⋅Δkw\_\{k\}\\cdot\\Delta\_\{k\}rather thanwkw\_\{k\}\. To keep stored weights faithful to their stated intent, we rescale them inversely to band width while preserving∑kwk\\sum\_\{k\}w\_\{k\}soτ\\tauremains comparable across pools:

w~k=wk/Δk∑k′wk′/Δk′⋅∑k′wk′\.\\tilde\{w\}\_\{k\}=\\frac\{w\_\{k\}/\\Delta\_\{k\}\}\{\\sum\_\{k^\{\\prime\}\}w\_\{k^\{\\prime\}\}/\\Delta\_\{k^\{\\prime\}\}\}\\cdot\\sum\_\{k^\{\\prime\}\}w\_\{k^\{\\prime\}\}\.\(4\)The effective weightsw~k\\tilde\{w\}\_\{k\}are written back to the routing configuration alongside the operator’s storedwkw\_\{k\}\(kept for auditability\), and the shortfall computation at inference time usesw~k\\tilde\{w\}\_\{k\}\.

### 3\.3Shortfall Matching

The shortfall\-matching algorithm \(Algorithm[2](https://arxiv.org/html/2605.17106#alg2)\) is the core routing decision procedure\. Given predicted requirements𝐫^\\hat\{\\mathbf\{r\}\}and model profiles\{\(𝐜m,costm\)\}\\\{\(\\mathbf\{c\}\_\{m\},\\text\{cost\}\_\{m\}\)\\\}:

sm≔shortfall​\(m\)=∑k=1Kw~k⋅max⁡\(0,r^k−cm,k\)s\_\{m\}\\coloneqq\\text\{shortfall\}\(m\)=\\sum\_\{k=1\}^\{K\}\\tilde\{w\}\_\{k\}\\cdot\\max\(0,\\hat\{r\}\_\{k\}\-c\_\{m,k\}\)\(5\)wherew~k\\tilde\{w\}\_\{k\}are the band\-compensated per\-dimension weights from Eq\.[4](https://arxiv.org/html/2605.17106#S3.E4)\(operators specifywkw\_\{k\}; the system stores and usesw~k\\tilde\{w\}\_\{k\}\)\. Themax⁡\(0,⋅\)\\max\(0,\\cdot\)ensures surplus on one dimension does*not*compensate for a deficit on another—a model with excellent code generation cannot offset poor debugging ability\. This is a deliberate design choice: capability dimensions represent qualitatively different skills, and deficiency in any one skill degrades the response\.

Algorithm 2Shortfall\-Based Routing Algorithm0:Requirements

𝐫^=\(r^1,…,r^K\)\\hat\{\\mathbf\{r\}\}=\(\\hat\{r\}\_\{1\},\\ldots,\\hat\{r\}\_\{K\}\), threshold

τ\\tau
0:Model pool

\{\(m,𝐜m,costm\)\}\\\{\(m,\\mathbf\{c\}\_\{m\},\\text\{cost\}\_\{m\}\)\\\}, band\-compensated weights

𝐰~=\(w~1,…,w~K\)\\tilde\{\\mathbf\{w\}\}=\(\\tilde\{w\}\_\{1\},\\ldots,\\tilde\{w\}\_\{K\}\)from Eq\.[4](https://arxiv.org/html/2605.17106#S3.E4)

0:Available models

𝒜\\mathcal\{A\}\(from infrastructure health filter\)

0:Confidence

γ\\gamma, sticky threshold

γsticky\\gamma\_\{\\text\{sticky\}\}
0:Selected model

m∗m^\{\*\}
1:if

γ<γsticky\\gamma<\\gamma\_\{\\text\{sticky\}\}then

2:returnfirst model in

𝒜\\mathcal\{A\}\{sticky: keep current model\}

3:endif

4:

ℱ←PreFilter​\(𝒜\)\\mathcal\{F\}\\leftarrow\\text\{PreFilter\}\(\\mathcal\{A\}\)\{hard gates \(e\.g\., vision\)\}

5:foreach

m∈ℱm\\in\\mathcal\{F\}do

6:

sm←∑k=1Kw~k⋅max⁡\(0,r^k−cm,k\)s\_\{m\}\\leftarrow\\sum\_\{k=1\}^\{K\}\\tilde\{w\}\_\{k\}\\cdot\\max\(0,\\hat\{r\}\_\{k\}\-c\_\{m,k\}\)
7:endfor

8:

ℰ←\{m∈ℱ:sm≤τ\}\\mathcal\{E\}\\leftarrow\\\{m\\in\\mathcal\{F\}:s\_\{m\}\\leq\\tau\\\}\{eligible set\}

9:if

ℰ≠∅\\mathcal\{E\}\\neq\\emptysetthen

10:

m∗←arg⁡minm∈ℰ⁡costmm^\{\*\}\\leftarrow\\arg\\min\_\{m\\in\\mathcal\{E\}\}\\text\{cost\}\_\{m\}\{cheapest eligible\}

11:else

12:

m∗←arg⁡minm∈ℱ⁡smm^\{\*\}\\leftarrow\\arg\\min\_\{m\\in\\mathcal\{F\}\}s\_\{m\}\{fail\-open: least shortfall\}

13:endif

14:return

m∗m^\{\*\}

Bothτ\\tauand𝐰\\mathbf\{w\}are runtime parameters—adjustable via hot configuration changes without retraining\. Grid search over∼\{\\sim\}3,000\(τ,𝐰\)\(\\tau,\\mathbf\{w\}\)configurations on a held\-out subset of the multilingual eval set \(§[5](https://arxiv.org/html/2605.17106#S5)\) identifies operating points along the cost\-quality Pareto frontier in seconds via vectorized NumPy computation\.

##### Conformal threshold calibration \(proposed\)\.

The deployed system selectsτ\\tauvia the grid search above\. As a principled alternative we propose to calibrateτ\\tauvia conformal risk control\(Bates et al\.,[2021](https://arxiv.org/html/2605.17106#bib.bib2)\), providing a distribution\-free coverage guarantee\. For each calibration queryqqwith known per\-model outcomes, we identify the*oracle cheapest\-correct model*mq∗m^\{\*\}\_\{q\}—the cheapest model producing quality above threshold—and compute its shortfallsq∗=shortfall​\(𝐫^q,𝐜mq∗\)s^\{\*\}\_\{q\}=\\text\{shortfall\}\(\\hat\{\\mathbf\{r\}\}\_\{q\},\\mathbf\{c\}\_\{m^\{\*\}\_\{q\}\}\)\. This is the minimumτ\\tauthat would have routedqqcorrectly\. We then set:

τ=Quantile​\(\{sq∗\}q=1N,⌈α​\(N\+1\)⌉N\)\\tau=\\text\{Quantile\}\\\!\\left\(\\\{s^\{\*\}\_\{q\}\\\}\_\{q=1\}^\{N\},\\;\\frac\{\\lceil\\alpha\(N\{\+\}1\)\\rceil\}\{N\}\\right\)\(6\)whereα\\alphais the target coverage rate \(e\.g\.,α=0\.95\\alpha\{=\}0\.95for 5% misroute tolerance\)\. Under exchangeability of calibration and test queries, this guaranteesP​\[misroute\]≤1−αP\[\\text\{misroute\}\]\\leq 1\{\-\}\\alpha\. RouteNLP\(Guo et al\.,[2026](https://arxiv.org/html/2605.17106#bib.bib6)\)applies conformal thresholds to cascade deferral decisions; we adapt the technique for pre\-routing shortfall threshold selection\. We do not deploy this scheme in the production multilingual checkpoint reported in this paper—it is left as future work to compare conformal selection against grid search empirically\.

##### Shortfall margin as routing confidence\.

The shortfall marginμm=τ−sm\\mu\_\{m\}=\\tau\-s\_\{m\}for the selected modelm∗m^\{\*\}provides an implicit confidence signal at zero additional cost\. Whenμm∗\\mu\_\{m^\{\*\}\}is large \(≫0\\gg 0\), the model’s capabilities comfortably exceed predicted requirements; whenμm∗\\mu\_\{m^\{\*\}\}is small \(≈0\\approx 0\), the routing decision is near the eligibility boundary and more likely incorrect\. We define a*margin\-gated conservative override*: ifμm∗<ϵ\\mu\_\{m^\{\*\}\}<\\epsilonfor a configurable margin thresholdϵ\\epsilon, the router selects the next\-more\-expensive eligible model instead, trading cost savings for reliability on ambiguous queries:

m∗=\{cheapest​\(ℰ\)if​τ−sm∗≥ϵ2nd\-cheapest​\(ℰ\)otherwisem^\{\*\}=\\begin\{cases\}\\text\{cheapest\}\(\\mathcal\{E\}\)&\\text\{if \}\\tau\-s\_\{m^\{\*\}\}\\geq\\epsilon\\\\ \\text\{2nd\-cheapest\}\(\\mathcal\{E\}\)&\\text\{otherwise\}\\end\{cases\}\(7\)This mechanism is complementary to cascading approaches\(Guo et al\.,[2026](https://arxiv.org/html/2605.17106#bib.bib6)\)that verify after generation: it provides a pre\-generation safety net with zero latency overhead, at the cost of slightly higher routing expense on borderline queries\. \(Pool\-level matched comparisons against deployed routers are deferred to §[7\.4](https://arxiv.org/html/2605.17106#S7.SS4)so that the architecture section stays focused on the routing decision procedure itself\.\)

### 3\.4Evaluation Metrics

We define three router\-level metrics, computed as in Algorithm[3](https://arxiv.org/html/2605.17106#alg3)\.

Quality Retention \(QR\):Resolution rate as a fraction of*Oracle Routing*, where Oracle assigns each query to the cheapest model that resolves it:

QR=ResrouterResoracle×100\\text\{QR\}=\\frac\{\\text\{Res\}\_\{\\text\{router\}\}\}\{\\text\{Res\}\_\{\\text\{oracle\}\}\}\\times 100\(8\)
Cost Savings \(CS\):The fraction of cost reduced relative to always routing to the pool’s most expensive model:

CS=\(1−costroutercostbaseline\)×100\\text\{CS\}=\\left\(1\-\\frac\{\\text\{cost\}\_\{\\text\{router\}\}\}\{\\text\{cost\}\_\{\\text\{baseline\}\}\}\\right\)\\times 100\(9\)
Misroute Rate \(Mis\):Fraction of queries where a cheaper model in the pool would also have resolved the query:

Mis=1\|𝒬\|\|\{q:∃m′s\.t\.\\displaystyle\\text\{Mis\}=\\tfrac\{1\}\{\|\\mathcal\{Q\}\|\}\\,\\big\|\\\{q:\\exists\\,m^\{\\prime\}\\ \\text\{s\.t\.\}cost​\(m′\)<cost​\(mq\)\\displaystyle\\text\{cost\}\(m^\{\\prime\}\)<\\text\{cost\}\(m\_\{q\}\)\(10\)∧yq,m′=1\}\|\\displaystyle\\wedge\\ y\_\{q,m^\{\\prime\}\}=1\\\}\\big\|
Algorithm 3Router Evaluation Metrics Computation0:Queries

𝒬\\mathcal\{Q\}, per\-model outcomes

\{yq,m\}\\\{y\_\{q,m\}\\\}, costs

\{costq,m\}\\\{\\text\{cost\}\_\{q,m\}\\\}
0:Router assignments

\{mq\}\\\{m\_\{q\}\\\}, baseline model

mbasem\_\{\\text\{base\}\}
0:

QR,CS,Mis\\text\{QR\},\\text\{CS\},\\text\{Mis\}
1:

resr←∑qyq,mq\\text\{res\}\_\{r\}\\leftarrow\\sum\_\{q\}y\_\{q,m\_\{q\}\};

costr←∑qcostq,mq\\text\{cost\}\_\{r\}\\leftarrow\\sum\_\{q\}\\text\{cost\}\_\{q,m\_\{q\}\}
2:

costb←∑qcostq,mbase\\text\{cost\}\_\{b\}\\leftarrow\\sum\_\{q\}\\text\{cost\}\_\{q,m\_\{\\text\{base\}\}\}
3:

noracle←0n\_\{\\text\{oracle\}\}\\leftarrow 0;

nmis←0n\_\{\\text\{mis\}\}\\leftarrow 0
4:foreach

q∈𝒬q\\in\\mathcal\{Q\}do

5:

ℛq←\{m:yq,m=1\}\\mathcal\{R\}\_\{q\}\\leftarrow\\\{\\,m:y\_\{q,m\}=1\\,\\\}
6:if

ℛq≠∅\\mathcal\{R\}\_\{q\}\\neq\\emptysetthen

7:

noracle←noracle\+1n\_\{\\text\{oracle\}\}\\leftarrow n\_\{\\text\{oracle\}\}\+1
8:

mq∗←arg⁡minm∈ℛq⁡costq,mm\_\{q\}^\{\*\}\\leftarrow\\arg\\min\_\{m\\in\\mathcal\{R\}\_\{q\}\}\\text\{cost\}\_\{q,m\}
9:if

costq,mq\>costq,mq∗\\text\{cost\}\_\{q,m\_\{q\}\}\>\\text\{cost\}\_\{q,m\_\{q\}^\{\*\}\}then

10:

nmis←nmis\+1n\_\{\\text\{mis\}\}\\leftarrow n\_\{\\text\{mis\}\}\+1
11:endif

12:endif

13:endfor

14:

QR←resr/noracle\\text\{QR\}\\leftarrow\\text\{res\}\_\{r\}/n\_\{\\text\{oracle\}\}
15:

CS←1−costr/costb\\text\{CS\}\\leftarrow 1\-\\text\{cost\}\_\{r\}/\\text\{cost\}\_\{b\}
16:

Mis←nmis/\|𝒬\|\\text\{Mis\}\\leftarrow n\_\{\\text\{mis\}\}/\|\\mathcal\{Q\}\|

For the predictor itself, we report per\-dimension MAE, RMSE, Pearsonrr, Spearmanρ\\rho, and binary accuracy at threshold 0\.5\.

## 4Labeling Pipeline

### 4\.1Context Tiering \(English Pipeline\)

For English data we route queries through one of three context tiers based on repository dependence:

- •T1 \(∼\{\\sim\}4%\): Explicit user\-attached file references\.
- •T2 \(∼\{\\sim\}62%\): Requires repo via tool calls or deictic references\.
- •T3 \(∼\{\\sim\}34%\): Self\-contained, no repo context needed\.

T3 queries proceed to dual\-model generation\. T1/T2 receive conservative synthetic defaults \(a constant requirement of0\.80\.8on every dimension\), biasing toward stronger models\. For T1/T2 queries referencing public repos, we reconstruct context by cloning at the recorded commit hash, raising coverage to∼\{\\sim\}45%\. The multilingual pipeline \(§[5](https://arxiv.org/html/2605.17106#S5)\) does*not*apply tier classification; every sampled non\-English conversation is sent through dual\-model generation directly\.

### 4\.2Dual\-Model Generation and Judging

The English re\-labeling and the multilingual labeling pipelines share the same labeling pipeline and differ only in the cheap\-model deployment\. For each query \(the last user turn of a sampled conversation\), we issuetwo parallel generationsconditioned on the system prompt and up to the prior 10 user/assistant turns of conversation context \(assistant chunks truncated to 8,000 characters\):

- •Cheap model:gpt\-5\.2\-chatfor multilingual data;gpt\-5\.4\-minifor the English re\-labeling run\. Chat Completions API, max completion tokens 4,096\.
- •Strong model:gpt\-5\.3\-codex\. Responses API, max output tokens 8,192\.
- •Judge model:gpt\-5\.2\-chat, JSON\-mode Chat Completions\.

The judge is given the user query and both responses and asked, on a 1–5 scale, to score each response on \{reasoning, code\_gen, debugging, tool\_use\} together with a winner∈\{A,B,tie\}\\in\\\{A,B,\\text\{tie\}\\\}and a quality\_gap∈\{\\in\\\{none, small, large\}\\\}tag\. To cancel positional bias\(Zheng et al\.,[2023](https://arxiv.org/html/2605.17106#bib.bib27)\), the judge is invoked*twice*—once with \(cheap=A, strong=B\) and once with positions swapped—and the per\-dimension scores attributed to the strong response from the two calls are averaged\.

### 4\.3Requirement Labels

Lets¯kcheap\\bar\{s\}^\{\\text\{cheap\}\}\_\{k\}ands¯kstrong\\bar\{s\}^\{\\text\{strong\}\}\_\{k\}denote the position\-debiased1​–​51\\text\{\-\-\}5judge scores for the cheap and strong responses on dimensionkk\. The requirement label measures where the strong model adds value:

rk=max⁡\(0,s¯kstrong−s¯kcheap\)/5∈\[0,1\]r\_\{k\}=\\max\(0,\\;\\bar\{s\}^\{\\text\{strong\}\}\_\{k\}\-\\bar\{s\}^\{\\text\{cheap\}\}\_\{k\}\)\\,/\\,5\\;\\;\\in\[0,1\]\(11\)If both score equally on a dimension, the requirement is zero\. \(Per\-model capability scores, when needed—e\.g\. for analysis or for the English re\-labeling run—are computed by simple linear normalization of the same judge outputs:ck=\(s¯k−1\)/4c\_\{k\}=\(\\bar\{s\}\_\{k\}\-1\)/4\.\) The router learns to escalate precisely where it matters\.

### 4\.4Training Data Statistics

The deployed multilingual checkpointHyDRA\-Multiis fine\-tuned on a single merged English\+multilingual corpus\. The implied split sizes from the released checkpoint’s training metadata are 40,128 train / 5,016 validation / 5,015 test examples\. The non\-English portion is sampled from 90 days of production telemetry stratified by language with a configurable per\-language cap \(Table[3](https://arxiv.org/html/2605.17106#S5.T3)\)\. Label distribution on the labeled subset \(mean±\\pmstd\): reasoning0\.31±0\.280\.31\\pm 0\.28, code generation0\.22±0\.240\.22\\pm 0\.24, debugging0\.18±0\.230\.18\\pm 0\.23, tool use0\.14±0\.200\.14\\pm 0\.20\.

### 4\.5Human\-Labeled Multilingual Eval Set

TODO before camera\-ready:replace per\-dimension agreement and routing\-quality numbers in this subsection \(and in Appendix[D](https://arxiv.org/html/2605.17106#A4)\) once the human\-annotation pass completes\. The deployed checkpoint will not be re\-selected on the basis of these scores; this set is held out as an independent ground\-truth audit of the LLM\-judge eval distribution\.

In addition to the LLM\-judge–labeled training and eval sets \(§[4](https://arxiv.org/html/2605.17106#S4), §[8\.1](https://arxiv.org/html/2605.17106#S8.SS1)\), we curate a separatehuman\-labeled multilingual eval setdrawn from VS Code Chat telemetry to provide a judge\-independent reference for both label quality and routing quality\.

##### Composition\.

The set contains∼\{\\sim\}5,000 Englishand∼\{\\sim\}3,000 non\-Englishqueries \(∼\{\\sim\}8,000 total\), sampled from the same 90\-day production telemetry pool as the LLM\-judge eval but with no query overlap\. Non\-English queries are stratified across the same 17 telemetry languages of Table[3](https://arxiv.org/html/2605.17106#S5.T3), weighted toward CJK and “Other” groups to compensate for their underrepresentation in the dual\-generation pipeline\.

##### Annotation protocol\.

Each query is independently labeled bythree annotatorsdrawn from a vendor\-supplied annotation team with native\-speaker coverage of the language groups; all annotators completed a 200\-example calibration pass against an internal gold set before live labeling\. Annotators score the query along the same four capability dimensions \(reasoning, code generation, debugging, tool use\) on the11–55rubric used by the LLM judge, plus a free\-text rationale\. Inter\-annotator disagreements are resolved by majority vote at the per\-dimension level; cases with no majority \(all three annotators distinct on a11–55score\) are routed to a senior reviewer for adjudication\.

##### Use in evaluation\.

This set is used for two analyses: \(i\) per\-dimension Krippendorff’sα\\alphabetween human consensus and the LLM judge, reported in Appendix[D](https://arxiv.org/html/2605.17106#A4), which quantifies the systematic bias of the judge baseline relative to human ground truth; and \(ii\) routing quality \(QR / CS / misroute\) ofHyDRA\-Multicomputed against human consensus labels rather than judge labels, providing a judge\-independent point estimate alongside the production\-telemetry numbers reported in Table[4](https://arxiv.org/html/2605.17106#S5.T4)\. Because the deployed checkpoint was selected before this set existed, it functions as a held\-out audit rather than a development eval\.

## 5Multilingual Support

Production coding assistants serve a global user base\. Routing must be*language\-invariant*: the same task complexity should yield the same routing decision regardless of whether the user writes in English, Japanese, or Portuguese\. We address this by extendingHyDRA’s training data and evaluating cross\-lingual routing consistency\.

### 5\.1Language Coverage

We pull non\-English Copilot Chat telemetry from 90 days of production traffic, using script\-range regexes \(Cyrillic, Arabic, CJK, etc\.\) on the user\-message text in KQL to assign each turn to a language bucket\. Three group files are produced \(CJK, European, Other\) totaling∼\{\\sim\}46,095 conversations / 88,596 turns across 17 non\-English languages with a non\-trivial signal in this snapshot \(Table[3](https://arxiv.org/html/2605.17106#S5.T3)\)\. For labeling we group raw turns by conversation, take the last turn as the labeling target with up to 10 prior turns as context, stratify by language, and cap each language at 5,000 conversations before issuing the cheap\- and strong\-model batches\. The full training corpus combines this multilingual pull with the English data of §[4](https://arxiv.org/html/2605.17106#S4), covering 21 languages in total; the held\-out multilingual production eval set \(§[8\.1](https://arxiv.org/html/2605.17106#S8.SS1)\) draws from a refreshed sample and contains 19 of these languages\.

Table 3:Multilingual telemetry coverage\. Conversations and turns for each language group, collected from 90 days of production Copilot Chat traffic\.
### 5\.2Training Strategy

Multilingual conversations are sampled stratified by language and processed through the same dual\-model generation \+ judge pipeline as English data \(§[4](https://arxiv.org/html/2605.17106#S4)\), with two pipeline\-level differences: \(i\) the cheap model isgpt\-5\.2\-chat\(rather thangpt\-5\.4\-minias used for the English re\-labeling run\); \(ii\) the T1/T2/T3 context\-tier classifier is not applied—every sampled non\-English conversation is sent through dual generation directly, so multilingual examples never receive synthetic tier defaults\. The strong model \(gpt\-5\.3\-codex, Responses API\) and judge \(gpt\-5\.2\-chat, JSON\-mode\) are identical to the English pipeline, as is the position\-swap protocol and the requirement\-label formula\.

The deployed multilingual model \(HyDRA\-Multi\) is fine\-tuned on a single merged English\+\+multilingual corpus \(rather than via a separate continued\-fine\-tuning stage\) starting fromanswerdotai/ModernBERT\-base—an English\-pretrained encoder used as\-is for all 21 languages\. The training configuration is: 5 epochs, per\-device batch size 32, AdamW \(fused\) optimizer, learning rate2×10−52\{\\times\}10^\{\-5\}, cosine schedule with 10% warmup, weight decay 0\.01, mixed\-precision fp16, seed 42\. Training runs to 6,270 optimizer steps in∼\{\\sim\}42 minutes on a single A100\. A second variant \(HyDRA\-Multi\-2048\) uses the same hyperparameters except per\-device batch size 8 to fit the longer 2048\-token context; it serves as an ablation for the chosen 512\-token deployment limit \(Table[18](https://arxiv.org/html/2605.17106#A4.T18), Appendix[D](https://arxiv.org/html/2605.17106#A4)\)\.

### 5\.3Multilingual Routing Results

Table[4](https://arxiv.org/html/2605.17106#S5.T4)reports routing quality on a 5,001\-query multilingual eval set \(3,191 English \+ 1,810 non\-English queries spanning 19 of the 21 training languages\), comparingHyDRA\-Multiagainst the always\-strong / always\-cheap / oracle reference points on the GPT\-5\.4\-mini vs\. GPT\-5\.3 Codex pool\. Per\-language and per\-group breakdowns are in Appendix[F](https://arxiv.org/html/2605.17106#A6)\.

RouterQRCSMis\.Oracle Routing100\.028\.00\.0GPT\-5\.4\-mini \(cheap\)89\.363\.832\.1GPT\-5\.3 Codex \(strong\)97\.30\.067\.9Binary router92\.335\.937\.0HyDRA\-Multi\(cons\.,τ=0\.30\\tau\{=\}0\.30\)95\.217\.140\.2HyDRA\-Multi\(agg\.,τ=0\.50\\tau\{=\}0\.50\)92\.338\.033\.8Table 4:Multilingual routing quality on the 5,001\-query eval set \(3,191 English \+ 1,810 non\-English; per\-language split in Appendix[F](https://arxiv.org/html/2605.17106#A6)\)\. 2\-model pool: GPT\-5\.4\-mini \(cheap\) vs\. GPT\-5\.3 Codex \(strong\)\. Reference rows are highlighted:*Oracle*\(blue\) is the per\-query upper bound,*always\-cheap*\(green\) is the GPT\-5\.4\-mini floor,*always\-strong*\(gray\) is the GPT\-5\.3 Codex ceiling that CS is measured against\.QR= quality retention vs\. Oracle Routing \(Eq\.[8](https://arxiv.org/html/2605.17106#S3.E8)\)\.CS= cost savings vs\. always\-strong \(GPT\-5\.3 Codex\)\.Mis\.= misroute \(%\)\. The binary router is our prior in\-house production system \(the first iteration of this work\) tuned to its iso\-quality operating point \(threshold0\.390\.39, matchingHyDRA\-Multi\(agg\.\) at QR = 92\.3\);HyDRAis its next\-generation replacement\. At iso\-quality,HyDRA\-Multi\(agg\.\) delivers 38\.0% cost savings vs\. the binary router’s 35\.9%, and the conservative point recovers an additional 2\.9 QR points \(95\.2 vs\. 92\.3\) that the binary classifier cannot reach without collapsing back to always\-cheap\.EnglishEuropeanCJKOther707080809090100100English baseline90\.490\.494\.794\.787\.587\.588\.388\.3100100100100100100100100Quality retention \(%\)HyDRAAlways\-StrongFigure 2:Language\-invariant routing quality\.Quality retention by language group on the multilingual eval set \(EnglishN=3,191N\{=\}3\{,\}191; EuropeanN=1,434N\{=\}1\{,\}434; CJKN=22N\{=\}22; OtherN=175N\{=\}175\)\.HyDRAstays within±4\.3\\pm 4\.3points of its English baseline across all four groups—European actually exceeds English—confirming routing decisions depend on task complexity rather than input language\. Always\-Strong is 100% by definition\.

## 6Evaluation

We evaluateHyDRAalong four complementary tracks, each targeting a different validity question:

1. 1\.Production telemetry \(multilingual eval set, §[8\.1](https://arxiv.org/html/2605.17106#S8.SS1)\)\.Does the deployed router work on the actual query distribution it serves? 5,001 held\-out queries sampled from VS Code Chat traffic across 19 languages\. This is the eval the deployedHyDRA\-Multicheckpoint is selected against\.
2. 2\.SWE\-Bench \(Verified & Pro, §[7\.2](https://arxiv.org/html/2605.17106#S7.SS2)\)\.Does routing preserve quality on a hard, externally curated agentic\-coding benchmark with real per\-instance run cost? Each instance is executed end\-to\-end by every candidate model so the cost–quality trade\-off is grounded in real token counts and resolution outcomes, not synthetic judge scores\.
3. 3\.Cross\-benchmark generalization \(§[7\.3](https://arxiv.org/html/2605.17106#S7.SS3)\)\.Does the routing decision generalize beyond the labeling distribution? We report LiveCodeBench\(Jain et al\.,[2024](https://arxiv.org/html/2605.17106#bib.bib7)\), BigCodeBench, andτ\\tau\-bench\(Yao et al\.,[2024](https://arxiv.org/html/2605.17106#bib.bib22)\)\(Airline \+ Retail\), spanning competitive programming, library\-grounded code, and tool\-use dialogue\.
4. 4\.Competitive comparison \(§[7\.4](https://arxiv.org/html/2605.17106#S7.SS4)\)\.How doesHyDRAstack up against published and commercial routers on a shared benchmark? We compare against RouteLLM\(Ong et al\.,[2024](https://arxiv.org/html/2605.17106#bib.bib16)\), Avengers Pro\(Zhang et al\.,[2025](https://arxiv.org/html/2605.17106#bib.bib25)\), Azure Foundry router\(Microsoft Azure,[2026](https://arxiv.org/html/2605.17106#bib.bib15)\), and OpenRouter\(OpenRouter,[2025](https://arxiv.org/html/2605.17106#bib.bib17)\)on SWE\-Bench Verified\.

### 6\.1Benchmarks

##### Multilingual production eval set \(§[5](https://arxiv.org/html/2605.17106#S5)\)\.

5,001 queries \(3,191 English \+ 1,810 non\-English across 19 languages\) drawn from 90 days of VS Code Chat telemetry, with 4\-dimensional capability labels obtained from the dual\-generation labeling pipeline \(§[4](https://arxiv.org/html/2605.17106#S4)\)\. This is the only eval whose query distribution matches the deployed serving distribution\.

##### SWE\-Bench Verified\.

500 real GitHub issues requiring multi\-file code changes\(Jimenez et al\.,[2024](https://arxiv.org/html/2605.17106#bib.bib10)\)\. Each instance runs through all candidate models, producing binary resolved/not\-resolved outcomes with real token counts and costs\. We evaluate with a pool of 8 models with full coverage on SWE\-Bench Verified, LiveCodeBench, and BigCodeBench:

##### LiveCodeBench, BigCodeBench,τ\\tau\-bench\.

Public benchmarks covering competitive programming\(Jain et al\.,[2024](https://arxiv.org/html/2605.17106#bib.bib7)\), library\-grounded code generation\(Zhuo et al\.,[2024](https://arxiv.org/html/2605.17106#bib.bib28)\), and tool\-use dialogue \(τ\\tau\-bench\(Yao et al\.,[2024](https://arxiv.org/html/2605.17106#bib.bib22)\), Airline \+ Retail\)\. These are used in §[7\.3](https://arxiv.org/html/2605.17106#S7.SS3)to test generalization of the deployed predictor \(no per\-benchmark fine\-tuning\) against the same baseline set\.

##### Baselines\.

Across all tracks:*Always\-Strong*\(the most expensive model in the pool\),*Always\-Cheap*\(the cheapest\),*Binary router*\(our prior in\-house ModernBERT binary classifier, the first iteration of this work and the system thatHyDRAreplaces in production\), and where applicable an*Oracle*that picks the per\-query optimal model from the resolution ledger\. Track 4 \(competitive comparison\) adds RouteLLM, Avengers Pro, Azure Foundry router, and OpenRouter\.

## 7Offline Evaluation

This section reports offline evaluation on public benchmarks and configuration sweeps\. All production\-data evaluations \(multilingual telemetry, A/B flight, two\-week 50/50 A/B scorecard\) are reported separately in §[8](https://arxiv.org/html/2605.17106#S8)\.

### 7\.1Setup

All offline numbers below are computed on the deployedHyDRA\-Multicheckpoint \(training corpus, hyperparameters, and runtime in §[5\.2](https://arxiv.org/html/2605.17106#S5.SS2); predictor architecture and objective in §[3\.1](https://arxiv.org/html/2605.17106#S3.SS1)\)\. The 512\-token input cap used in production is paired with a 2048\-token ablation reported in Appendix[D](https://arxiv.org/html/2605.17106#A4)\.

Baselines: see §[6](https://arxiv.org/html/2605.17106#S6)\.

### 7\.2SWE\-Bench Verified

Table 5:SWE\-Bench Verified \(500 instances; 5\-model pool: GPT\-5\.4\-mini, Claude Haiku 4\.5, GPT\-5\.3 Codex, Claude Sonnet 4\.6, GPT\-5\.4\)\. Reference rows are highlighted:*Oracle*\(blue\) is the per\-query upper bound,*always\-cheap*\(green\) is the GPT\-5\.4\-mini floor,*always\-strong*\(gray\) is the Claude Sonnet 4\.6 ceiling that CS is measured against\.QR= quality retention vs\. Oracle Routing \(cheapest model that resolves each instance, per Eq\.[8](https://arxiv.org/html/2605.17106#S3.E8); Oracle resolves 431/500 = 86\.2%\)\.CS= cost savings vs\. always\-strong \(Claude Sonnet 4\.6;$​1,024\.75\\mathdollar 1\{,\}024\.75on this dataset\)\.Mis\.= misroute \(%\)\. The binary router is our prior in\-house production system \(the first iteration of this work\) restricted to \(Sonnet, GPT\-5\.4\-mini\);HyDRAis its next\-generation replacement\. Three operating points illustrate the tunability ofτ\\tau: \(peak\)*exceeds*Sonnet \(75\.4% vs\. 74\.2% resolution\) at 12\.9% cost savings; \(cons\.\) matches Sonnet at iso\-quality and delivers 54\.1% savings \(a 6×\\timesimprovement over the binary router’s 9\.1%\); \(agg\.\) trades 5\.1 QR points for 72\.5% savings\. The full Pareto sweep overτ\\tauis shown in Fig\.[4](https://arxiv.org/html/2605.17106#S9.F4)\.Figure[3](https://arxiv.org/html/2605.17106#S7.F3)decomposes the 500 instances by which of the strong/weak baseline solves them, and shows whereHyDRAsends each subset\.

STRONGclaude\-sonnet\-4\.6WEAKgpt\-5\.4\-mini53instancesH→\\tostrong: 29H→\\toweak: 24318instancesH→\\tostrong: 155H→\\toweak: 16329instancesH→\\tostrong: 16H→\\toweak: 13

Figure 3:SWE\-Bench Verified routing decomposition \(claude\-sonnet\-4\.6 vs\. gpt\-5\.4\-mini; 100 instances unsolved by either model are excluded from the Venn\)\. Each region shows how many of the 400 solvable instancesHyDRArouted to the strong vs\. weak model at the iso\-quality operating point \(τ=0\.175\\tau\{=\}0\.175, balanced 249/251 split\)\.HyDRAcaptures 72\.0% resolution at 34\.1% cost savings versus always\-Sonnet\.This is a 2\-model analysis on the same strong/weak pair as the binary router in Table[5](https://arxiv.org/html/2605.17106#S7.T5); the headline 5\-model results use the larger pool described there\.
### 7\.3Cross\-Benchmark Generalization

The deployed predictor is trained only on labeled VS Code Chat queries \(§[4](https://arxiv.org/html/2605.17106#S4)\); its capability requirement scores are never exposed to LiveCodeBench, BigCodeBench, orτ\\tau\-bench at training time\. We therefore test whether the four capability dimensions, the model capability profiles, and the shortfall\-matching policy together transfer to off\-distribution evaluation suites without per\-benchmark adjustment\. Results—per\-router quality and cost\-savings across SWE, LiveCode, BigCode, AIME, andτ\\tau\-bench Airline/Retail—are reported in Tables[15](https://arxiv.org/html/2605.17106#A4.T15)and[16](https://arxiv.org/html/2605.17106#A4.T16)\(Appendix[D](https://arxiv.org/html/2605.17106#A4)\)\. The same router config used in production is used at inference, with no per\-benchmark hyperparameter tuning\.

### 7\.4Competitive Comparison

We compareHyDRAon SWE\-Bench Verified against published research routers \(RouteLLM\-MF, RouteLLM\-BERT, Avengers Pro\) and commercial offerings \(Azure Foundry router, OpenRouterauto\)\. The headline comparison is*matched\-pool*: every router selects only from a common 3\-model pool \(GPT\-5, GPT\-5\-mini, GPT\-5\.2\), so quality and cost numbers are directly comparable \(Table[6](https://arxiv.org/html/2605.17106#S7.T6)\)\. We then report a broader*pool\-normalized*view \(Table[7](https://arxiv.org/html/2605.17106#S7.T7)\) that lets each system run against its native pool, with QR/CS measured against that system’s own strong baseline\. Design\-level dimensions are summarized in Table[1](https://arxiv.org/html/2605.17106#S2.T1)\. Numerical entries marked\[p\.\]are pending the shared\-harness run\.

SystemQR \(%\)CS \(%\)MR \(%\)Oracle Routing100\.0\+68\.90\.0GPT\-5\-mini \(always\-cheap\)73\.2\+\+90\.10\.0Avengers Pro – Conservative91\.8\+\+15\.658\.8OpenRouter Auto90\.3\+\+4\.968\.2Azure Foundry Router \(Quality\)82\.9\+\+36\.338\.8Avengers Pro – Aggressive81\.6\+\+46\.735\.6Azure Foundry Router \(Balanced\)76\.0\+\+66\.217\.6GPT\-5\.2 \(always\-strong\)93\.40\.072\.4HyDRA \(cons\.\)τ=0\.703\\tau\{=\}0\.70390\.3\+16\.2\\boldsymbol\{\+\}\\textbf\{16\.2\}57\.8HyDRA \(agg\.\)τ=0\.888\\tau\{=\}0\.88884\.2\+44\.1\\boldsymbol\{\+\}\\textbf\{44\.1\}35\.6

Table 6:Matched\-pool comparison on SWE\-Bench Verified\(n=500n\{=\}500, pool: GPT\-5, GPT\-5\-mini, GPT\-5\.2\)\. All routers select only from this fixed pool, so QR / CS / MR are directly comparable across rows\. Reference rows highlighted:*Oracle*\(blue\) is the per\-query upper bound,*always\-cheap*\(green\) is the GPT\-5\-mini floor,*always\-strong*\(gray\) is the GPT\-5\.2 ceiling that CS is measured against\.QR: quality retention vs\. Oracle Routing\.CS: cost savings vs\. always\-strong \(GPT\-5\.2\)\.MR: misroute rate\.*Conservative*settings preserve quality at the cost of smaller savings;*aggressive*settings push savings further at the cost of some quality\.Matched\-pool result\.On the common 3\-model pool,HyDRA\(cons\.\) ties Avengers Pro and OpenRouter Auto on quality \(90\.3% QR\) while delivering competitive cost savings;HyDRA\(agg\.\) Pareto\-dominates Avengers Pro–Aggressive \(84\.2% vs\. 81\.6% QR at comparable savings\) and is strictly better than the Foundry Balanced operating point on QR per CS\. No competing system is onHyDRA’s frontier when both axes are measured against the same Oracle and the same strong baseline\.

HyDRA vs\. RouteLLM: a numerical tie hides an architectural gap\.On its native 2\-model pool \(gpt\-5\.3\-codex,gpt\-5\.4\-mini; Table[7](https://arxiv.org/html/2605.17106#S7.T7)\), RouteLLM\-BERT reports 91\.1% QR /\+\+18\.6% CS atθ=0\.5\\theta\{=\}0\.5, numerically close toHyDRA’s conservative operating point on the 5\-model pool\. The headline numbers are similar; the architectures are not\. Three structural advantages ofHyDRAare not visible in a single QR/CS row:

1. 1\.No retraining on catalog change\.RouteLLM jointly embeds queries and model identities \(Ong et al\.,[2024](https://arxiv.org/html/2605.17106#bib.bib16), §3\.2\): adding, removing, or repricing a model requires fresh preference data and a retraining run\.HyDRA’s predictor is over query*requirements*; the catalog lives in YAML \(§[3\.2](https://arxiv.org/html/2605.17106#S3.SS2)\) and is changed without touching the model \(quantified in §[7\.5](https://arxiv.org/html/2605.17106#S7.SS5)\)\.
2. 2\.Catalog scaling without re\-architecting\.RouteLLM is binary by construction; serving a 5\-model pool requires either training\(52\)=10\\binom\{5\}\{2\}\{=\}10pairwise routers or collapsing the catalog to a strong/weak partition\.HyDRA’s shortfall\-matching algorithm scales to arbitrary pool size inO​\(\|ℳ\|​K\)O\(\|\{\\mathcal\{M\}\}\|K\)at inference \(§[3\.3](https://arxiv.org/html/2605.17106#S3.SS3)\); the matched\-pool numbers above are with\|ℳ\|=3\|\\mathcal\{M\}\|\{=\}3and the SWE\-Bench headline numbers are with\|ℳ\|=5\|\\mathcal\{M\}\|\{=\}5, reusing the same trained predictor\.
3. 3\.Multilingual coverage\.RouteLLM trains on English Chatbot Arena preferences and reports only on English benchmarks\.HyDRAtrains and is evaluated on 21 languages across 3 script families \(§[5](https://arxiv.org/html/2605.17106#S5)\); to our knowledge, the first deployed LLM\-pool router with reported per\-language quality and cost parity\.

These are the operational advantages that justify deployingHyDRAover a binary classifier in a production setting where the model catalog turns over monthly and users speak 21 languages\. The pool\-normalized comparison below quantifies the broader landscape\.

SystemModel Pool \(strong→\\tocheap\)Mode / op\. pointQR \(%\)CS \(%\)MR \(%\)Commercial / closed\-source routersOpenRouter AutoClaude Sonnet 4\.6, GPT\-5\.2, GPT\-5 minidefaultauto92\.1\+\+0\.176\.0Azure Foundry RouterGPT\-5\.2, GPT\-5, GPT\-5 miniBalanced76\.0\+\+66\.217\.6Published research routersRouteLLM \(BERT\)GPT\-5\.3 Codex, GPT\-5\.4 minithreshold=0\.5=0\.591\.1\+\+18\.645\.2Avengers ProClaude Sonnet 4\.6, Claude Haiku 4\.5, GPT\-5\.4, GPT\-5\.3 Codex, GPT\-5\.4 minidefault83\.3\+\+51\.050\.8OursHyDRA \(cons\.\)Claude Sonnet 4\.6, Claude Haiku 4\.5, GPT\-5\.4, GPT\-5\.3 Codex, GPT\-5\.4 miniτ=0\.239\\tau\{=\}0\.23986\.1\+54\.1\\boldsymbol\{\+\}\\textbf\{54\.1\}70\.4HyDRA \(agg\.\)Claude Sonnet 4\.6, Claude Haiku 4\.5, GPT\-5\.4, GPT\-5\.3 Codex, GPT\-5\.4 miniτ=0\.436\\tau\{=\}0\.43684\.5\+63\.7\\boldsymbol\{\+\}\\textbf\{63\.7\}52\.8Table 7:Pool\-normalized comparison on SWE\-Bench Verified\(n=500n\{=\}500\)\. Each system runs against its native model pool \(column 2\)\.QR: quality retention vs\. that pool’s*Oracle Routing*\(cheapest model that resolves each instance\)\.CS: cost savings vs\. that pool’s most expensive model\.MR: misroute rate \(share of instances where a cheaper model in the pool would also have resolved\)\. Units are comparable across rows even when pools differ\.*Conservative*settings preserve quality at the cost of smaller savings;*aggressive*settings push savings further at the cost of some quality\.
### 7\.5Model Catalog Portability

Table 8:Catalog portability\.HyDRAreroutes via a YAML edit: a small QR drop on removal \(it must fall back to the next\-cheapest covering model\), a small QR gain on addition \(it routes to the new model on day one\)\. Thebinary routeris held to its fixed strong/weak split, so its QR is unchanged across all three rows—not because it is robust to catalog change, but because it cannot perceive it\. To matchHyDRA’s adaptation, the binary router would need new preference data and a full retraining run for every catalog event\.Table[8](https://arxiv.org/html/2605.17106#S7.T8)validates the core claim\. The binary router’s unchanged QR across all three rows is the failure mode, not a strength: it cannot exploit a mid\-tier model that is added, and it cannot reroute Haiku traffic when Haiku is removed—both because model identities are baked into its training labels\.HyDRAhandles both events with a single YAML edit and no retraining: removal costs 2\.4 QR points \(Haiku’s 40% traffic share falls back to the next\-cheapest covering model\), addition gains 0\.4 QR points \(the new model is selected on the dimensions where it dominates\)\. This is the operational advantage that pays back the multi\-dimensional architecture in production, where the catalog turns over monthly\.

### 7\.6Latency

Table 9:*Offline*routing\-latency breakdown, measured on a local CPU benchmark machine\. The corresponding production CAPI\-end\-to\-end numbers \(median 142 ms, dominated by 86 ms of ModernBERT INT8 inference\) are reported in Table[14](https://arxiv.org/html/2605.17106#A1.T14); the gap relative to the 55 ms offline P50 here is network plus request/response serialization between the CAPI front\-end and the routing pod\. At P50==55 ms, routing overhead is<<1% of typical LLM generation time \(2–30 s\)\.

## 8Production Evaluation

This section reports evaluation on real VS Code Chat production traffic, kept separate from the offline benchmark and configuration evaluations of §[7](https://arxiv.org/html/2605.17106#S7)\. We report three production\-data tracks: a held\-out multilingual telemetry eval \(§[8\.1](https://arxiv.org/html/2605.17106#S8.SS1)\), a controlled A/B flight \(§[8\.2](https://arxiv.org/html/2605.17106#S8.SS2)\), and a two\-week 50/50 A/B production scorecard \(§[8\.2](https://arxiv.org/html/2605.17106#S8.SS2.SSSx1)\)\. Server\-side serving metrics for the deployed router are reported alongside the deployment description in §[10\.6](https://arxiv.org/html/2605.17106#S10.SS6)\.

### 8\.1Production Telemetry

The headline production result is reported in Table[4](https://arxiv.org/html/2605.17106#S5.T4)\(§[5](https://arxiv.org/html/2605.17106#S5)\): on 5,001 held\-out queries sampled from real VS Code Chat traffic across 19 languages, the deployedHyDRA\-Multicheckpoint matches the always\-strong baseline within 2\.1 QR points at 17% cost savings \(conservative operating point\) or trades 5 QR points for 38% savings \(aggressive operating point\), against the GPT\-5\.4\-mini vs\. GPT\-5\.3 Codex pool with per\-query LLM\-judge labels\. At iso\-quality \(QR = 92\.3\) the prior production binary router delivers only 35\.9% savings vs\.HyDRA\-Multi’s 38\.0%, and cannot reach the 95\.2\-QR conservative point at all \(its calibration saturates well below it\)\. This is the eval that selects the deployed checkpoint; the remaining tracks test whether that decision generalizes\.

### 8\.2Production A/B Flight and Scorecard

We run a 50/50 randomized A/B flight comparing the deployedHyDRAarm against the prior production auto\-mode control—an in\-house binary router combined with the heuristic\-based auto\-mode policy that preceded any learned routing—on the full VS Code Chat auto\-mode population, with sticky per\-user assignment over a two\-week window stratified by SKU \(Free / Edu / Pro / Pro\+ / Business / Enterprise\)\. The two\-arm design controls for traffic\-mix and seasonality effects that single\-arm before/after comparisons cannot\. We report two complementary views of the same flight: a per\-user core\-metrics view \(Table[10](https://arxiv.org/html/2605.17106#S8.T10)\) covering quality retention, cost savings, token counts, and acceptance; and a two\-week operational scorecard \(Table[11](https://arxiv.org/html/2605.17106#S8.T11)\) covering auto\-mode opt\-in/out flow, commit\-survival, P99 latency, and fallback rate\. Statistical significance for the core\-metrics view is reported via a two\-sidedtt\-test on user\-level means with bootstrap 95% CIs\. The scorecard answers operational questions the core view alone cannot: do users opt out of auto mode under either arm? Do response and prompt token counts shift in unexpected directions? Does the routed code survive in users’ commits at the same rate as control? What is the realized CoGS savings net of router infrastructure cost?

Table 10:Production A/B flight \(HyDRAvs\. the prior production auto\-mode control—in\-house binary router plus heuristic auto\-mode policy—on VS Code Chat\)\.*Production telemetry numbers omitted from this preprint version; to be added in v2\.*#### Two\-week scorecard\.

The operational scorecard view of the same flight is reported in Table[11](https://arxiv.org/html/2605.17106#S8.T11)\.

Table 11:Two\-week 50/50 A/B production scorecard forHyDRAvs\. the prior production binary router on VS Code Chat auto\-mode\. Both arms run over the same time window with sticky per\-user assignment\. Commit\-survival rate measures the fraction of accepted code suggestions that remain in the user’s working tree after 24h / 7d \(a proxy for downstream quality\)\.*Production telemetry numbers omitted from this preprint version; to be added in v2\.*

## 9Analysis

##### Dimension weights\.

Grid search reveals non\-uniform optimal weights: debugging and tool use receive higher weights than reasoning and code generation\. Cheap models handle routine code generation well but struggle with subtle debugging and multi\-step tool orchestration\.

##### Shortfall margin analysis\.

The shortfall marginμ=τ−sm∗\\mu=\\tau\-s\_\{m^\{\*\}\}\(Eq\.[7](https://arxiv.org/html/2605.17106#S3.E7)\) is intended as a pre\-generation confidence signal: queries with large margin should route more reliably, and anϵ\\epsilon\-conservative upgrade rule \(ϵ=0\.05\\epsilon\{=\}0\.05\) trades a small additional cost for reduced misroute\. The numerical correlation and upgrade\-rule sweep are deferred to v2 of this preprint pending the eval\-harness rerun\.

##### Misroute failure modes\.

Among the 50 worst quality\-loss instances: ambiguous intent \(40%\)—short messages with insufficient context; hidden complexity \(35%\)—simple\-looking queries requiring deep domain knowledge; label noise \(25%\)—inconsistent judge scores across position swaps\.

##### Cost\-quality frontier\.

HyDRAprovides a smooth, continuously tunable curve on the 5\-model SWE\-Bench pool, ranging from 87\.0% QR / 37\.6% CS atτ=0\.10\\tau\{=\}0\.10down to 80\.5% QR / 78\.5% CS atτ≥1\.0\\tau\{\\geq\}1\.0\(asymptotically matching Always\-Cheap\)\. Our prior in\-house binary router—the first iteration of this work, restricted to \(Sonnet, GPT\-5\.4\-mini\)—sits at a single point of 85\.6% QR / 9\.1% CS, dominated byHyDRA’s curve at every operating point\. The continuous tunability ofτ\\tauover the full 5\-model pool is the key operational improvement ofHyDRAover its binary predecessor: a single deployment can be re\-tuned for cost vs\. quality without retraining or model swaps\. Figure[4](https://arxiv.org/html/2605.17106#S9.F4)visualises this frontier\.

010102020303040405050606070708080808085859090τ=0\.01\\tau\{=\}0\.01τ=0\.24\\tau\{=\}0\.24τ=0\.64\\tau\{=\}0\.64Cost Savings vs\. Sonnet \(%\)Quality Retention vs\. Oracle \(%\)HyDRABinary routerAlways\-CheapAlways\-StrongFigure 4:Cost\-quality Pareto frontier on SWE\-Bench Verified \(5\-model pool\)\.Y\-axisis quality retention vs\. Oracle Routing \(cheapest model that resolves each query, per Eq\.[8](https://arxiv.org/html/2605.17106#S3.E8); Oracle resolves 431/500 = 86\.2%\)\.X\-axisis cost savings vs\. always\-strong \(Claude Sonnet 4\.6;$​1,024\.75\\mathdollar 1\{,\}024\.75on this dataset\)\.HyDRAoffers a smooth, continuously tunable curve; the binary router provides only a single operating point that is dominated at everyτ\\tau\.

## 10Production Deployment

HyDRAis deployed as a pre\-routing layer integrated with the GitHub Copilot API \(CAPI\) infrastructure\. It currently serves production traffic forVS Code Chat—GitHub Copilot’s conversational coding assistant used by tens of millions of developers\. The system is planned for extension to Copilot CLI, Copilot Coding Agent \(CCA\), and Copilot Code Review \(CCR\)\. This section describes the integration architecture, session\-sticky routing, and model lifecycle management\.

### 10\.1CAPI Integration Architecture

The existing CAPI routing infrastructure makes model selection decisions based on infrastructure health metrics—throughput utilization, error rates, and latency—using weight multipliers per model endpoint\. This is entirely*content\-blind*: a trivial question receives the same routing treatment as a complex debugging task\.

HyDRAintegrates via arank\-then\-filterprotocol \(Algorithm[4](https://arxiv.org/html/2605.17106#alg4)\):

Algorithm 4CAPI Integration: Rank\-then\-Filter0:Query

qq, available models

𝒜\\mathcal\{A\}\(from CAPI\)

0:Health scores

\{hm\}\\\{h\_\{m\}\\\}, health floor

hf=0\.10h\_\{f\}=0\.10
0:Selected model

m∗m^\{\*\}
1:

𝐫^,γ←HyDRA\.predict​\(q\)\\hat\{\\mathbf\{r\}\},\\gamma\\leftarrow\\text\{HyDRA\}\.\\text\{predict\}\(q\)
2:

ℰ←ShortfallMatch​\(𝐫^,𝒜\)\\mathcal\{E\}\\leftarrow\\text\{ShortfallMatch\}\(\\hat\{\\mathbf\{r\}\},\\mathcal\{A\}\)\{Alg\.[2](https://arxiv.org/html/2605.17106#alg2)\}

3:

ℋ←\{m∈ℰ:hm≥hf\}\\mathcal\{H\}\\leftarrow\\\{m\\in\\mathcal\{E\}:h\_\{m\}\\geq h\_\{f\}\\\}\{health veto\}

4:if

ℋ=∅\\mathcal\{H\}=\\emptysetthen

5:return

CAPI\_Fallback​\(𝒜,\{hm\}\)\\text\{CAPI\\\_Fallback\}\(\\mathcal\{A\},\\\{h\_\{m\}\\\}\)\{fail\-open\}

6:endif

7:return

first​\(ℋ\)\\text\{first\}\(\\mathcal\{H\}\)\{cheapest capable & healthy\}

Three key design properties govern this integration:

1\. CAPI is a veto, not a reranker\.CAPI removes unhealthy models fromHyDRA’s ranked list but never changes the ordering\. This preserves query\-awareness as the primary routing signal while respecting infrastructure constraints\.

2\. Fail\-open semantics\.When allHyDRA\-eligible models are unhealthy \(rare—indicates widespread infrastructure issues\), routing falls back entirely to CAPI’s weighted random selection\. This guarantees service availability\.

3\. No double\-randomization\.Under integration, CAPI’s stochastic weighted sampling is replaced byHyDRA’s deterministic cheapest\-eligible selection filtered by health\. This makes routing decisions reproducible and debuggable\.

The serving architecture deploysHyDRAas an ONNX INT8 model behind a managed CPU inference service, integrated into the VS Code Chat request path\. When a user sends an auto\-mode message in VS Code Chat, CAPI sends a POST request containing the structured query context and the list of available models\.HyDRAreturns a ranked candidate list with per\-dimension scores and shortfall values\. The system handles edge cases gracefully: if the selected model is unhealthy, the next cheapest eligible is chosen; if all eligible models are unhealthy, routing fails open to CAPI’s weighted random selection; if CAPI metrics are unavailable, health defaults to 1\.0 soHyDRAdecides; and if confidence is low \(γ<γs\\gamma<\\gamma\_\{s\}\), sticky routing preserves the current model\.

### 10\.2Prompt\-Cache\-Preserving Sticky Routing

Multi\-turn agentic conversations routinely span 20\+ turns and accumulate tens of thousands of context tokens\. Provider prompt caches \(Anthropic’s prompt caching, OpenAI’s automatic input\-token caching\) make this affordable: a turn that hits the cache reuses the entire serialized prefix at a 90% discount and avoids re\-prefilling tens of thousands of tokens of agent scaffolding, tool definitions, file contents, and prior assistant turns\. Switching models mid\-conversation defeats this entirely—the prefix has to be re\-billed and re\-prefilled at the new provider, and \(2\) model\-specific formatting and behavioural patterns change, confusing users and downstream tools\. A naive per\-turn router would therefore destroy cache hit rate on exactly the heaviest, most cost\-sensitive conversations\.

To protect the cache, the deployed VS Code Chat integration runs the router*at least once per conversation*\. The session controller caches a per\-conversation routing decision keyed on the conversation identifier and skips the router on every subsequent turn unless an explicit invalidation signal is received\. Concretely,HyDRAis invoked in exactly three situations:

1. 1\.Turn 1of a new conversation\.
2. 2\.After an explicit user\-issued conversation\-compaction command, which invalidates the cached routing decision\. Compaction rewrites the conversation prefix, so the existing prompt cache is already invalidated; this is a safe re\-route point\.
3. 3\.After background conversation summarization, which invalidates the routing decision for the same reason\.

For every other turn the cached model from the most recent routing decision is reused, regardless of router confidence\. This is a stronger form of stickiness than confidence gating: the cheapest\-capable choice made at turn 1 is amortized over the entire \(often 20\+ turn\) conversation, the prompt cache is preserved end\-to\-end, and there is no behavioural drift between providers\. The cache is keyed on the conversation identifier, so different conversations route independently and parallel router\-version rollouts \(e\.g\., A/B experiments\) coexist without cross\-contamination\. Re\-routing is only permitted at exactly the points where the prompt cache is already known to be invalid\.

The service also exposes three configurable stickiness modes for offline evaluation and experiments—per\-request\(no caching, every turn re\-routed\),per\-content\(cache key includes a SHA\-256 hash of the conversation prefix\), andper\-session\(the production setting described above\)\. The per\-session mode is the deployed default precisely because it is the only mode that aligns the routing\-cache lifetime with the prompt\-cache lifetime\.

### 10\.3Image Hardgating

The deployed router is text\-only by construction\. When the incoming request contains image attachments, the VS Code Chat session controller bypassesHyDRAentirely and emits a router\-fallback telemetry event tagged as a multimodal\-skip; routing is delegated to CAPI’s content\-blind weighted selection over the multimodal\-capable subset of the catalog\. This hard\-gate is necessary because \(i\) the labeled training distribution contains no image content, so capability\-requirement predictions are unreliable on multimodal queries, and \(ii\) the cap\-matcher’s eligibility check would otherwise need a fifth “vision” dimension whose model\-side capability scores are not yet calibrated\. ExtendingHyDRAto route image\-bearing requests—via a calibrated vision capability dimension and image\-aware training data—is planned as the next step\.

### 10\.4Quantization and Inference

The deployed multilingual checkpoint is exported from PyTorch to ONNX FP32 \(the encoder produces a \[CLS\] vector that is fed through the four sigmoid heads, with the heads concatenated into a single output tensor for graph simplicity\)\. Two quantized variants are produced from the FP32 ONNX model:

Dynamic INT8 with attention nodes excluded\(the variant used in production\)\. We apply ONNX Runtime’s dynamic INT8 weight quantization while excluding all attention/QKV nodes\. Full INT8 quantization of ModernBERT collapses predictions to a near\-constant baseline because the attention layer’s masked\-fill operations interact pathologically with INT8 calibration; quantizing only the non\-attention nodes \(predominantly the feed\-forward linear layers\) preserves accuracy bit\-exactly within tolerance while still cutting model size by∼\{\\sim\}40% and improving CPU latency by∼\{\\sim\}8%\.

FP16 ONNXvia ONNX Runtime’s float\-to\-float16 conversion\. Roughly halves model size with no measurable quality regression and is used on hardware where FP16 kernels are faster than dynamic INT8\.

We also implement*quantization\-aware training*\(QAT\) with PyTorch’s standard fake\-quantization pipeline: per\-channel symmetric INT8 weights with min/max observers \(\[−128,127\]\[\-128,127\]\) and per\-tensor affine UINT8 activations with moving\-average min/max observers \(reduced range\)\. QAT was not required to ship the deployed 512\-token checkpoint \(post\-training INT8\-skip\-attention already achieves zero quality regression on our held\-out set\) and is therefore retained as a fallback for future encoders or platforms where dynamic INT8 fails\.

### 10\.5Model lifecycle operations

HyDRA’s decoupled architecture reduces model lifecycle operations to configuration changes \(Table[12](https://arxiv.org/html/2605.17106#S10.T12)\):

- •Adding a model:Define capability profile in YAML \(via Algorithm[1](https://arxiv.org/html/2605.17106#alg1)\), set cost\-per\-token, deploy config\. The shortfall algorithm automatically includes the new model\. No ML changes needed\.
- •Deprecating a model:Remove from YAML \(or sethm=0h\_\{m\}=0in CAPI\)\. Traffic redistributes automatically to the next\-cheapest capable model\. Sticky caches are invalidated on the next prediction\.
- •Updating capabilities:Re\-run pool\-relative normalization and deploy updated YAML\. All profiles are re\-normalized automatically since they are pool\-relative\.

Table 12:Model lifecycle operations:HyDRAvs\. a model\-coupled router\. AllHyDRAoperations are configuration\-only\.
### 10\.6Server\-Side Production Metrics

A self\-contained snapshot of production serving\-tier metrics for the deployed multilingual router—availability against the routing endpoint \(99\.98% 5xx\-only\), daily routing volume \(∼\{\\sim\}830K total /∼\{\\sim\}795K auto\-mode\), end\-to\-end median router latency \(142 ms, dominated by 86 ms of ModernBERT INT8 inference\), and per\-model traffic distribution—is reported in Appendix[A](https://arxiv.org/html/2605.17106#A1)\. The two cheapest 1P models combined absorb 43% of auto\-mode traffic; the strongest model \(GPT\-5\.3 Codex\) accounts for 21%, confirming that shortfall matching steers easy queries to cheap models in aggregate, not just on the offline eval\. Numbers are taken from the production CAPI Auto Intent dashboard following the 100% rollout; updated multi\-week aggregates will be reported in v2\.

### 10\.7Routing Explainability in VS Code Chat

We have prototyped a developer\-mode inspector in VS Code Chat that surfaces, for each routed turn, the selected model and the four predicted capability requirements \(reasoning, code generation, debugging, tool use\)\. The inspector is an internal\-only mock at the time of writing and has not yet shipped to end users; we report it as a design study for in\-product router explainability\. A screenshot of the prototype is shown in Appendix[B](https://arxiv.org/html/2605.17106#A2)\.

### 10\.8Open\-Source Release and Reproducibility

We will release the fullHyDRAstack to support reproduction and extension: training and evaluation code for the ModernBERTK=4K\{=\}4\-head capability predictor \(English\+\+multilingual pipeline, hyperparameters used to produceHyDRA\-Multi\), capability profile generation code \(Algorithm[1](https://arxiv.org/html/2605.17106#alg1)\) and benchmark\-derived YAMLs for the deployed pool, the LLM\-judge dual\-generation labeling pipeline, ablation scripts, and the deployed dynamic INT8 ONNX checkpoint plus FP32 / FP16 sensitivity variants\. A detailed component listing is in Appendix[C](https://arxiv.org/html/2605.17106#A3)\.

## 11Conclusion

We presentedHyDRA, a multi\-dimensional, capability\-decoupled routing framework for heterogeneous LLM pools, and the second iteration of our deployed routing system\. The first iteration—an in\-house binary strong\-vs\-weak router that we shipped to GitHub Copilot in early 2026—demonstrated the practical value of learned routing but was model\-coupled and dimensionally limited\.HyDRAaddresses both: on SWE\-Bench Verified it spans three operating points along a single tunable threshold—exceeding always\-strong Sonnet atτ=0\.01\\tau\{=\}0\.01\(75\.4% vs\. 74\.2% resolution\) at 12\.9% savings, matching Sonnet at iso\-quality with 54\.1% savings \(τ=0\.24\\tau\{=\}0\.24, a6×6\\timesimprovement over our binary predecessor\), and pushing to 72\.5% savings atτ=0\.64\\tau\{=\}0\.64for a 3\.2\-point quality trade\. The capability\-decoupled design enables zero\-retraining model onboarding, and the rank\-then\-filter integration with production CAPI infrastructure preserves both query\-awareness and system reliability\.HyDRAcurrently routes production traffic for VS Code Chat and is planned for extension to Visual Studio, JetBrains / IntelliJ, Copilot CLI, Copilot Coding Agent, and Copilot Code Review\. Multilingual training across 21 languages ensures routing quality is language\-invariant\. We release the routing framework, model weights, and evaluation code\.111Code and models: link to be added in the camera\-ready version\.

Looking forward, several extensions are scoped for the next release cycle: per\-*subagent*capability prediction so cheap models can handle planning while reserving the strong model for code generation;*cost\-aware adaptive multi\-turn re\-routing*that re\-invokes the router at turn boundaries when the expected quality gain exceeds cache\-invalidation cost \(relaxing the current first\-turn\-plus\-post\-compaction policy\);*long\-context support*beyond the deployed 512\-token cap \(the 2048\-tokenHyDRA\-Multi\-2048ablation is a first step\); a*vision capability dimension*so multimodal queries \(currently hard\-gated to a strong model\) participate in routing;*cross\-IDE rollout*to Visual Studio, JetBrains / IntelliJ, Copilot CLI, Copilot Coding Agent, and Copilot Code Review; defenses against the cost\-inflation suffix attacks documented in Appendix[E](https://arxiv.org/html/2605.17106#A5); and*online capability profile updates*that close the loop between deployed model behavior and the YAML profiles used for shortfall matching\.

## Limitations

Domain specificity\.The four dimensions target coding tasks\. Other domains require re\-defining dimensions and re\-labeling\. The architecture generalizes\.

Labeling cost\.Dual\-model generation \+ LLM judging is currently the dominant labeling expense; we do not report a precise dollar figure as our runs combine on\-demand and Azure Batch API pricing across multiple deployment regions\.

Judge labels for repository\-dependent queries\.Telemetry queries are stripped of repository state before labeling, so the LLM judge sees only the user message and the conversation prefix—not the files, symbols, or build context the query actually depends on\. For repository\-dependent queries \(“why does this test fail”, “add a field to this model”\) the judge cannot fully resolve what the query means in its original repo context, which biases its scores toward high\-requirement defaults and limits the cost savings the router can realize on this slice\.

Judge bias\.Position swapping mitigates positional bias; other biases \(verbosity, sycophancy\) remain\.

Benchmark\-derived profiles\.Public benchmarks may not reflect real query performance\. Profile errors cause systematic misrouting\.

SWE\-Bench skew\.SWE\-Bench is reasoning\-heavy\. Real traffic has more trivial queries, likely increasing savings\.

Multilingual coverage imbalance\.CJK and “other” language groups have fewer training samples \(250 and∼\{\\sim\}300 conversations respectively\) compared to European \(45K\)\. Per\-language evaluation may be underpowered for low\-resource languages\.

Adversarial robustness\.Recent work\(Tang et al\.,[2026](https://arxiv.org/html/2605.17106#bib.bib19)\)shows that adversarial suffix optimization can manipulate LLM routers into selecting expensive models\. Appendix[E](https://arxiv.org/html/2605.17106#A5)reports a diagnostic suffix probe forHyDRA: simple prompts that should remain cheap can be moved toward more expensive routes by appending complexity\-signaling suffixes\. This is not yet a comprehensive adaptive attack benchmark; defenses such as input perturbation detection, suffix filtering, score anomaly filtering, and calibrated score caps remain future work\.

Compaction\-bounded re\-routing\.The router is invoked on the first turn of a conversation and after every compaction event \(explicit user\-issued compaction or background summarization\); between compactions the selected model is reused via prompt\-cache–preserving sticky routing \(§[10\.2](https://arxiv.org/html/2605.17106#S10.SS2)\)\. Sessions whose difficulty drifts mid\-conversation*between*compactions cannot benefit from re\-routing under the current policy\.

Context\-length cap\.The deployed multilingual checkpoint truncates inputs at 512 tokens\. Queries with large repository contexts or long conversation histories are clipped before scoring, which can underestimate complexity\.

Coding\-only deployment\.HyDRAis currently deployed only in VS Code Chat\. Generalization to other surfaces \(Visual Studio, JetBrains / IntelliJ, Copilot CLI, Copilot Coding Agent, Copilot Code Review\) and to non\-coding domains has not been measured at production scale\.

## Ethics Statement

Data sourcing and consent\.The production telemetry used for training and evaluation \(§[4](https://arxiv.org/html/2605.17106#S4), §[5](https://arxiv.org/html/2605.17106#S5)\) is drawn from VS Code Chat traffic for users who have opted in to product\-improvement telemetry under the GitHub Copilot privacy and data\-handling agreement\. Personally identifiable information is removed prior to labeling and storage; raw user prompts are not redistributed and are not part of the OSS release\.

Annotator compensation\.The native\-speaker annotators contracted for the human\-labeled multilingual eval set \(§[4\.5](https://arxiv.org/html/2605.17106#S4.SS5)\) were compensated at the contracting vendor’s standard rate for technical\-content annotation in their respective markets\. Annotators were briefed on the labeling rubric and the downstream routing use case prior to consent\.

LLM judge bias\.Quality labels are produced by an LLM\-as\-judge pipeline \(§[4](https://arxiv.org/html/2605.17106#S4)\)\. We mitigate positional bias via order swapping but cannot fully eliminate verbosity, style, or self\-preference biases\. The held\-out human\-labeled eval set \(§[4\.5](https://arxiv.org/html/2605.17106#S4.SS5)\) is intended to bound judge bias quantitatively via per\-dimension Krippendorff’sα\\alphaagainst human consensus\.

Cost–quality trade\-off and end\-user impact\.HyDRAroutes auto\-mode requests to cheaper models when the predicted capability requirement allows\. This trades a small expected\-quality reduction for substantial cost savings on a population that did not explicitly opt in to a specific model\. We mitigate this by reporting quality\-retention numbers per language and per SKU, surfacing the routing decision and per\-dimension scores in a developer\-mode inspector \(§[10\.7](https://arxiv.org/html/2605.17106#S10.SS7), currently internal\-only\), and maintaining a user\-facing model\-picker override path so any user can opt out of auto\-routing entirely\.

Security and adversarial use\.The adversarial suffix probe in Appendix[E](https://arxiv.org/html/2605.17106#A5)documents a known cost\-inflation attack surface\. We disclose this openly so deployers can plan defenses; we are not aware of in\-the\-wild exploitation against the deployed router\.

## Acknowledgments

We thank the GitHub Copilot, VS Code, and CAPI infrastructure teams for the production integration, telemetry, and rollout support that made this work possible, and our annotation partners for the multilingual labeling effort\.

## Appendix AServer\-Side Production Metrics \(Detailed\)

Note:HyDRAhas been rolled out to 100% of VS Code Chat auto\-mode traffic\. The numbers in this section are taken from the production CAPI Auto Intent dashboard at the time of writing; multi\-week aggregates \(availability, latency, RPM, per\-model and per\-SKU traffic share\) and the post\-rollout snapshot of Figure[6](https://arxiv.org/html/2605.17106#A1.F6)will be added in v2\.

We report aggregate serving\-tier metrics for the deployed multilingual checkpoint, observed on the production CAPI Auto Intent service over a representative 24\-hour window\. These complement the routing\-quality metrics in §[7](https://arxiv.org/html/2605.17106#S7)by characterizing the system as an online service rather than as a classifier\.

Table[13](https://arxiv.org/html/2605.17106#A1.T13)summarizes the high\-level production KPIs: availability is measured against the routing endpoint and split into a 5xx\-only SLO and a stricter 5xx\+\+4xx variant \(the latter conservatively counts upstream payload\-validation rejections, dominated by malformed multimodal\-fallback requests\)\. Auto\-mode traffic accounts for∼\{\\sim\}795,000 of the∼\{\\sim\}830,000 daily routing requests; the remainder is direct\-model requests routed through the same endpoint for telemetry uniformity\.HyDRAhas been rolled out to 100% of auto\-mode traffic; the only residual non\-HyDRApaths are the safety fallbacks \(multimodal hard\-gate, predictor timeout\)\.

Table 13:Production KPIs for the deployed multilingual router \(24\-hour window, post\-100%\-rollout\)\.*Values to be added in v2\.*##### Latency\.

End\-to\-end router latency decomposes cleanly across the three internal stages, with the ModernBERT inference itself accounting for the overwhelming majority of the cost \(Table[14](https://arxiv.org/html/2605.17106#A1.T14)\)\. Tokenization is effectively free; the residual∼\{\\sim\}54 ms not attributable to predict\-stage work is network plus request/response serialization between the CAPI front\-end and the routing pod\.

Table 14:Production latency breakdown for the deployed dynamic\-INT8 \(attention\-excluded\) multilingual checkpoint\. Median values over a 24\-hour window\.
##### Model traffic distribution\.

Of the∼\{\\sim\}805,000 daily auto\-mode routing decisions, the cheap\-tier models \(GPT\-5 mini, GPT\-4\.1\) absorb the plurality of traffic rather than the strongest model \(Figure[5](https://arxiv.org/html/2605.17106#A1.F5)\)\.

02244668810101212141416161818202022222424262628283030gpt\-5\-minigpt\-5\.3\-codexgpt\-4\.1sonnet\-4\.6haiku\-4\.5Other26262121171799552222% of daily auto\-mode routing decisionsFigure 5:Per\-model share of the∼\{\\sim\}805K daily auto\-mode routing decisions following the 100% rollout\. The two cheapest 1P models combined absorb 43% of traffic; the strongest model \(GPT\-5\.3 Codex\) accounts for 21%\.*Placeholder: stacked time\-series of \(a\) RPM, \(b\) p50/p95/p99 router latency, \(c\) per\-model traffic share over a 7\-day post\-rollout window, captured from the CAPI Auto Intent dashboard\.*

Figure 6:Production serving\-tier metrics for the deployed multilingual router at 100% rollout\. Final figure to be added in v2\.

## Appendix BIn\-Product Routing Explainability Prototype

![Refer to caption](https://arxiv.org/html/2605.17106v1/vscode_explainability.png)Figure 7:Prototype routing explainability inspector for the VS Code Chat developer\-mode auto\-routing surface \(*internal mock; not yet shipped to end users*\)\. The chosen model is displayed alongside the four sigmoid\-head outputs that drove the shortfall\-matching decision \(here: Reasoning 92%, Code Generation 61%, Debugging 88%, Tool Use 34%\)\.
## Appendix COpen\-Source Release Components

*\[Placeholder — final URLs and DOIs to be added in the camera\-ready version\.\]*We release the fullHyDRAstack to support reproduction and extension:

- •Training codefor the ModernBERTK=4K\{=\}4\-head capability predictor, including the merged English\+\+multilingual training pipeline and the hyperparameters used to produceHyDRA\-Multi\.
- •Evaluation codefor all benchmarks reported in §[7](https://arxiv.org/html/2605.17106#S7)\(SWE\-Bench Verified, LiveCodeBench, BigCodeBench,τ\\tau\-bench, multilingual production eval set\)\.
- •Capability profile generation code\(Algorithm[1](https://arxiv.org/html/2605.17106#alg1)\) and the benchmark\-derived profile YAMLs for the deployed model pool\.
- •Benchmarking and portability scripts, including the pool\-portability sweep \(§[7\.5](https://arxiv.org/html/2605.17106#S7.SS5)\) and the LLM\-judge dual\-generation labeling pipeline\.
- •Ablation studies\(head dropout, context\-length variants, quantization configurations\)\.
- •Model weights: the deployed dynamic INT8 \(attention\-excluded\) production checkpoint, plus FP32 and FP16 ONNX variants for sensitivity analysis, uploaded to a public model hub\.

## Appendix DExtended Evaluation

Additional evaluation results\. All metrics marked\[pending\]will be populated before final submission\.

Table 15:Per\-model resolution rate \(%\) across coding benchmarks\. SWE = SWE\-Bench Verified \(n=500n\{=\}500\), LiveCode = LiveCodeBench\(Jain et al\.,[2024](https://arxiv.org/html/2605.17106#bib.bib7)\)\(n=175n\{=\}175\), BigCode = BigCodeBench \(n=1,140n\{=\}1\{,\}140\)\. Only models evaluated on all three benchmarks are shown\.Table 16:Per\-model realized cost savings \(%\) vs\.claude\-sonnet\-4\.6\(positive = cheaper than baseline\)\. Cost is computed asTotalInputTokens×pin\+TotalOutputTokens×pout\\text\{TotalInputTokens\}\\times p\_\{\\text\{in\}\}\+\\text\{TotalOutputTokens\}\\times p\_\{\\text\{out\}\}from the comparison files, summed per benchmark, with unit prices from the published per\-model rate card\. SWE = SWE\-Bench Verified \(n=500n\{=\}500\), LiveCode = LiveCodeBench\(Jain et al\.,[2024](https://arxiv.org/html/2605.17106#bib.bib7)\)\(n=175n\{=\}175\), BigCode = BigCodeBench \(n=1,140n\{=\}1\{,\}140\)\. Note that GPT\-5 shows a sharp token blow\-up on the short LiveCode/BigCode tasks—its mean output length on these benchmarks is roughly5×5\\timesthat of its peers \(e\.g\.∼\\sim2\.5k tokens vs\.∼\\sim0\.5k forgpt\-5\.4\), driving realized cost aboveclaude\-sonnet\-4\.6despite a cheaper unit price; this is a model\-behavior artifact, not a pricing inversion\.Table 17:Ablation studies\.*Routing\-dimension*rows report SWE\-Bench Verified resolution rate and cost savings vs\. always\-strong using the router\-eval sweep at the best\-quality\-vs\-strong operating point on the 2048\-context multilingual checkpoints retrained with the indicated dimensions zeroed out \(shortfall thresholdτ\\tauchosen per row\)\.*Context\-feature*rows report judge QR and CS on a 239\-query multilingual judge slice using the deployedK=4K\{=\}4checkpoint with progressively stripped context, scored against the always\-strong baseline\. The two sections use different baselines and so are not directly comparable; each section is internally consistent\.##### WhyK=4K\{=\}4whenK=2K\{=\}2looks competitive on SWE\-Bench?

On SWE\-Bench Verified theK=2K\{=\}2,K=3K\{=\}3, andK=4K\{=\}4checkpoints are within 0\.4 quality points; only cost savings move materially \(\+\+3\.8 CS forK=4K\{=\}4overK=2K\{=\}2\)\. This is expected: SWE\-Bench is a single\-task benchmark on which essentially every query is reasoning\- and code\-heavy in similar proportions, so the marginal information from the debugging and tool\-use heads is small\. The dimensions earn their complexity on three axes that this single\-benchmark ablation cannot show\. \(1\)Workload heterogeneity: production traffic mixes pure\-reasoning “explain this” turns, tool\-orchestration agent loops, and debugging\-focused diff turns; theK=4K\{=\}4predictor routes each according to its dominant requirement, while aK=2K\{=\}2predictor must conflate them\. \(2\)Catalog heterogeneity: when a new model that is best\-in\-class on a single dimension \(e\.g\. a debug\-specialized or tool\-tuned variant\) is added to the YAML catalog \(§[7\.5](https://arxiv.org/html/2605.17106#S7.SS5)\),K=4K\{=\}4shortfall matching can route to it on day one without retraining; a scalar router cannot express the dimension on which the new model dominates\. \(3\)Interpretability and auditability: each per\-dimension score is exposed in the routing decision and surfaced in a developer\-mode inspector \(Appendix[B](https://arxiv.org/html/2605.17106#A2)\), giving operators a structured explanation of why a given model was selected—a property that is valuable for debugging production routing regressions and that has zero analogue in a scalar router\. The marginal cost ofK=4K\{=\}4overK=2K\{=\}2is≈\\approx1,500 additional parameters in the head \(K×769K\\times 769, §[3\.1](https://arxiv.org/html/2605.17106#S3.SS1)\) and zero extra latency in the encoder forward pass\. Given the cost is essentially free and the benefits scale with the very workload and catalog heterogeneity that the rest of the paper targets, we shipK=4K\{=\}4\.

##### Input context length: 512 vs\. 2048 tokens\.

The deployed multilingual checkpoint truncates inputs at 512 tokens; we also trained an otherwise\-identical 2048\-token checkpoint to scope the headroom from longer context \(Table[18](https://arxiv.org/html/2605.17106#A4.T18)\)\. On the held\-out test split, the 2048 variant gains\+\+0\.20 binary\-accuracy points and\+\+0\.010 Pearson averaged across the four heads, while inference throughput drops5\.4×5\.4\\times\(282→\\to51 samples/s on a single A100\) and training time grows1\.7×1\.7\\times\(2,510→\\to4,307 s\)\. Per\-dimension differences are within noise: the largest delta is 0\.0024 MAE on the*reasoning*head\. Given the cost predictor is in the synchronous request path and the predictor\-quality gain does not move any downstream router metric we can measure, we ship the 512\-token cap\.

Table 18:Input context\-length ablation \(multilingualK=4K\{=\}4checkpoint, otherwise identical training recipe\)\. Doubling context twice yields a sub\-noise predictor\-quality gain at5\.4×5\.4\\timesinference cost; the deployed checkpoint uses the 512\-token cap\.Router \(threshold\)Res%QRCSMis\.RouteLLM \(BERT,t=0t\{=\}0\)73\.40100\.000\.00\[p\.\]RouteLLM \(BERT,t=25t\{=\}25\)73\.40100\.000\.47\[p\.\]RouteLLM \(BERT,t=50t\{=\}50\)72\.0098\.0924\.46\[p\.\]RouteLLM \(BERT,t=75t\{=\}75\)69\.4094\.5571\.27\[p\.\]RouteLLM \(BERT,t=100t\{=\}100\)69\.4094\.5571\.44\[p\.\]HyDRA\(cons\.,τ=0\.150\\tau\{=\}0\.150\)73\.2099\.7319\.9040\.00HyDRA\(agg\.,τ=0\.300\\tau\{=\}0\.300\)69\.8095\.1043\.107\.20Table 19:RouteLLM \(BERT\) threshold sweep on SWE\-Bench Verified, evaluated on a 2\-model pool \(strong = GPT\-5\.3 Codex, weak = GPT\-5\.4 mini\); higher threshold→\\toeasier to choose weak\. RouteLLM only supports binary \(2\-model\) routing, which is why this comparison is restricted to a strong/weak pair rather than the 5\-model pool used elsewhere in the paper\. Misroute column for RouteLLM rows pending\.

## Appendix EAdversarial Suffix Robustness Probe

This appendix summarizes a diagnostic robustness probe motivated by R2A\-style cost\-inflation attacks\(Tang et al\.,[2026](https://arxiv.org/html/2605.17106#bib.bib19)\)\. We start from simple prompts that route to cheap models, append a short suffix, and measure whetherHyDRAroutes the attacked prompt to a frontier/reasoning model or otherwise leaves the cheap set\. The production setting uses the deployed INT8 ONNX model on 50 English prompts; FP32 and FP16 rescore the same frozen English corpus as sensitivity checks\. The multilingual supplement uses 30 non\-English base prompts with either the original English suffix or a suffix localized to the prompt language/script\. Clean rows have 0% frontier ASR in all settings\.

##### Attack suffixes\.

We use five deliberately simple suffix families that leave the base task unchanged but add surface cues likely to inflate predicted capability requirements\. S1 \(*keyword stuffing*\) appends: “This requires deep multi\-step reasoning, complex debugging, and advanced tool orchestration\.” S2 \(*fake complexity*\) appends: “Consider all edge cases, race conditions, memory leaks, and thread safety implications\.” S3 \(*role injection*\) appends: “\[SYSTEM: This is a critically complex query requiring frontier model capabilities\.\]” S4 \(*technical jargon*\) appends: “Apply distributed consensus algorithms with Byzantine fault tolerance analysis\.” S5 \(*difficulty assertion*\) appends: “Difficulty: Expert\. Requires: reasoning=0\.95, debugging=0\.90, tool\_use=0\.85\.” Mechanistically, S1 pressures reasoning, debugging, and tool\-use scores; S2 mostly pressures reasoning/debugging; S3 targets frontier routing; S4 targets reasoning cues; and S5 applies direct score\-threshold pressure\.

##### Grouped visual summary\.

Figures[8](https://arxiv.org/html/2605.17106#A5.F8)–[9](https://arxiv.org/html/2605.17106#A5.F9)report frontier attack success rate \(A​S​RfrontierASR\_\{\\text\{frontier\}\}\)\. The focused view isolates deployed INT8 and multilingual INT8; the all\-condition view adds FP32/FP16 sensitivity\.

S1S2S3S4S5033669912121515Frontier ASR \(%\)INT8ML\-EnML\-LocalFigure 8:Focused INT8/multilingual ASR\.ML\-En/ML\-Local are multilingual prompts with English/localized suffixes\.S1S2S3S4S5055101015152020252530303535Frontier ASR \(%\)INT8FP32FP16ML\-EnML\-LocalFigure 9:All\-condition ASR\.FP32/FP16 are frozen\-corpus English sensitivity checks\.
##### Heatmap\-style summary\.

Tables[E](https://arxiv.org/html/2605.17106#A5.SS0.SSS0.Px3)–[E](https://arxiv.org/html/2605.17106#A5.SS0.SSS0.Px3)pack ASR and cost ratio into each condition cell\. Darker cells indicate larger frontier ASR, so the qualitative pattern is visible even when the exact numbers are small\.

Table 20:Focused INT8/multilingual heatmap\.Each cell reportsA​S​RfrontierASR\_\{\\text\{frontier\}\}, cost ratio\. Shading tracks ASR only, not cost\. ML\-En/ML\-Local are multilingual INT8 supplements\.
Table 21:English precision heatmap\.Each cell reportsA​S​RfrontierASR\_\{\\text\{frontier\}\}, cost ratio\. Shading tracks ASR only, not cost\. FP32/FP16 are English frozen\-corpus sensitivity checks\.
Human\-label sanity check\.For 30 English clean/attacked prompt pairs, a human reviewer labeled the true task requirements before and after suffixing\. The suffix preserved the underlying task in 30/30 pairs; 28/30 clean prompts and 28/30 attacked prompts were still judged suitable for a cheap model\. Mean human\-label shifts were small \(reasoning \+0\.045, code generation \+0\.020, debugging \+0\.043, tool use \+0\.005\), supporting the interpretation that these suffixes affect router scores more than human\-perceived task requirements\.

##### Takeaway\.

The deployed INT8 router is less sensitive to these suffixes than the FP32/FP16 sensitivity runs, suggesting that quantization may dampen some fragile surface\-feature activations that otherwise amplify complexity cues\. However, the human labels show that the suffixes usually preserve the underlying task, so adversarial cost\-inflation remains a real robustness surface forHyDRAand a mitigation target for future work\.

## Appendix FPer\-Language Routing Results

Table 22:Per\-language routing quality \(HyDRA\-Multi\)\. QR uses the same definition as Table[4](https://arxiv.org/html/2605.17106#S5.T4)\(vs\.*Oracle Routing*, Eq\.[8](https://arxiv.org/html/2605.17106#S3.E8)\);Δ\\DeltaQR is the per\-language difference vs\. English\.Pending: re\-run of the multilingual eval harness with the full per\-query oracle ledger; values will be populated alongside the Table[4](https://arxiv.org/html/2605.17106#S5.T4)update\.
## Appendix GBenchmark\-Derived Capability Profiles

This appendix details the benchmark results used to construct model capability profiles for each routing dimension\. Table[23](https://arxiv.org/html/2605.17106#A7.T23)reports per\-benchmark scores, and Table[24](https://arxiv.org/html/2605.17106#A7.T24)shows the resulting pool\-normalized profiles used for shortfall matching\.

Table 23:Per\-benchmark resolution rates \(%\) used for capability profile derivation\. SWE\-Bench Verified and LiveCodeBench are split into Easy / Medium / Hard tiers; BigCodeBench andτ2\\tau^\{2\}\-Bench \(Airline / Retail\) report a single aggregate\. Only models with full coverage across all five benchmarks are shown\.Table 24:Final capability profiles for shortfall matching on SWE\-Bench Verified, BigCodeBench, and LiveCodeBench\. Values are raw weighted\-benchmark scores\. Routing dimension weights:*Reas\.*=1\.25=1\.25,*Code*=0\.69=0\.69,*Debug*=0\.77=0\.77,*Tool*=1\.29=1\.29\(band\-compensated\)\.
## References

- Bandarkar et al\. \(2026\)Lucas Bandarkar, Chenyuan Yang, Mohsen Fayyaz, Junlin Hu, and Nanyun Peng\. 2026\.Multilingual routing in mixture\-of\-experts\.In*The Fourteenth International Conference on Learning Representations \(ICLR\)*\.ArXiv:2510\.04694\.
- Bates et al\. \(2021\)Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael Jordan\. 2021\.Distribution\-free, risk\-controlling prediction sets\.*Journal of the ACM*, 68\(6\):1–34\.
- Chen et al\. \(2023\)Lingjiao Chen, Matei Zaharia, and James Zou\. 2023\.FrugalGPT: How to use large language models while reducing cost and improving performance\.*arXiv preprint arXiv:2305\.05176*\.
- Ding et al\. \(2024\)Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks V\.S\. Lakshmanan, and Ahmed Hassan Awadallah\. 2024\.Hybrid LLM: Cost\-efficient and quality\-aware query routing\.*arXiv preprint arXiv:2404\.14618*\.
- GitHub \(2025\)GitHub\. 2025\.Github copilot infrastructure routing\.Internal documentation\.
- Guo et al\. \(2026\)Dongxin Guo, Jikun Wu, and Siu Ming Yiu\. 2026\.RouteNLP: Closed\-loop LLM routing with conformal cascading and distillation co\-optimization\.In*Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(ACL\), Industry Track*\.ArXiv:2604\.23577\.
- Jain et al\. \(2024\)Naman Jain, King Han, Alex Gu, Wen\-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar\-Lezama, Koushik Sen, and Ion Stoica\. 2024\.LiveCodeBench: Holistic and contamination\-free evaluation of large language models for code\.*arXiv preprint arXiv:2403\.07974*\.
- Jiang et al\. \(2024\)Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, and 1 others\. 2024\.Mixtral of experts\.In*arXiv preprint arXiv:2401\.04088*\.
- Jiang et al\. \(2023\)Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin\. 2023\.LLM\-Blender: Ensembling large language models with pairwise ranking and generative fusion\.In*Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(ACL\)*\.
- Jimenez et al\. \(2024\)Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan\. 2024\.SWE\-bench: Can language models resolve real\-world GitHub issues?In*Proceedings of the International Conference on Learning Representations \(ICLR\)*\.
- Liu et al\. \(2026\)Hui Liu, Bin Zou, Kecheng Chen, Jie Liu, Wenya Wang, and Haoliang Li\. 2026\.Task\-aware LLM routing with multi\-level task\-profile\-guided data synthesis for cold\-start scenarios\.In*Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(ACL\)*\.ArXiv:2604\.09377\.
- Lu et al\. \(2024\)Keming Lu, Bowen Yu, Chang Zhou, and Jingren Zhou\. 2024\.Routing to the expert: Efficient reward\-guided ensemble of large language models\.*arXiv preprint arXiv:2311\.08692*\.
- Madaan et al\. \(2024\)Aman Madaan, Pranjal Aggarwal, Ankit Anand, Srividya Potdar, Sandro Savarese, and Shafiq Jain\. 2024\.AutoMix: Automatically mixing language models\.*arXiv preprint arXiv:2310\.12963*\.
- Madeyski \(2026\)Lech Madeyski\. 2026\.Triage: Routing software engineering tasks to cost\-effective LLM tiers via code quality signals\.*arXiv preprint arXiv:2604\.07494*\.
- Microsoft Azure \(2026\)Microsoft Azure\. 2026\.Model router for Microsoft Foundry\.[https://learn\.microsoft\.com/azure/foundry/openai/concepts/model\-router](https://learn.microsoft.com/azure/foundry/openai/concepts/model-router)\.Accessed 2026\-05\-06\.
- Ong et al\. \(2024\)Isaac Ong, Amjad Almahairi, Vincent Wu, Wei\-Lin Chiang, Tianhao Wu, Joseph E Gonzalez, M Waleed Kadous, and Ion Stoica\. 2024\.RouteLLM: Learning to route LLMs with preference data\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*\.ArXiv:2406\.18665\.
- OpenRouter \(2025\)OpenRouter\. 2025\.OpenRouter auto routing\.[https://openrouter\.ai/docs/features/auto\-router](https://openrouter.ai/docs/features/auto-router)\.Accessed 2026\-05\-06\.
- Štorek et al\. \(2025\)Adam Štorek, Vikas Upadhyay, Marianne Menglin Liu, Daniel W\. Peterson, Anshul Mittal, Sujeeth Bharadwaj, Fahad Shah, and Dan Roth\. 2025\.[Routesplain: Towards faithful and intervenable routing for software\-related tasks](https://arxiv.org/abs/2511.09373)\.*Preprint*, arXiv:2511\.09373\.
- Tang et al\. \(2026\)Haochun Tang, Yuliang Yan, Jiahua Lu, Huaxiao Liu, and Enyan Dai\. 2026\.Route to rome attack: Directing LLM routers to expensive models via adversarial suffix optimization\.In*Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(ACL\)*\.ArXiv:2604\.15022\.
- Varshney et al\. \(2026\)Tanay Varshney, Annie Surla, Michelle Xu, Gomathy Venkata Krishnan, Maximilian Jeblick, David Austin, Neal Vaidya, and Davide Onofrio\. 2026\.LLM router: Rethinking routing with prefill activations\.*arXiv preprint arXiv:2603\.20895*\.
- Warner et al\. \(2024\)Benjamin Warner, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Galkin, Raja Biber, Stephen Labusch, Mehmet Emin Durmus, and Nomic AI\. 2024\.ModernBERT: A modern approach to encoder\-only transformers\.*arXiv preprint arXiv:2412\.13663*\.
- Yao et al\. \(2024\)Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan\. 2024\.τ\\tau\-bench: A benchmark for tool\-agent\-user interaction in real\-world domains\.*arXiv preprint arXiv:2406\.12045*\.[https://github\.com/sierra\-research/tau\-bench](https://github.com/sierra-research/tau-bench)\.
- Zhang et al\. \(2026a\)Jiarui Zhang, Xiangyu Liu, Yong Hu, Chaoyue Niu, Hang Zeng, Shaojie Tang, Fan Wu, and Guihai Chen\. 2026a\.From myopic selection to long\-horizon awareness: Sequential LLM routing for multi\-turn dialogue\.*arXiv preprint arXiv:2604\.12385*\.
- Zhang et al\. \(2024\)Jieyu Zhang, Ranjay Krishna, Ahmed Hassan Awadallah, and Chi Wang\. 2024\.EcoAssistant: Using LLM assistant more affordably and accurately\.*arXiv preprint arXiv:2310\.03046*\.
- Zhang et al\. \(2025\)Yiqun Zhang, Hao Li, Jianhao Chen, Hangfan Zhang, Peng Ye, Lei Bai, and Shuyue Hu\. 2025\.Beyond GPT\-5: Making LLMs cheaper and better via performance\-efficiency optimized routing\.In*Proceedings of the International Conference on Distributed Artificial Intelligence \(DAI\)*\.ArXiv:2508\.12631\.
- Zhang et al\. \(2026b\)Yiqun Zhang, Hao Li, Zihan Wang, Shi Feng, Xiaocui Yang, Daling Wang, Bo Zhang, Lei Bai, and Shuyue Hu\. 2026b\.MTRouter: Cost\-aware multi\-turn LLM routing with history\-model joint embeddings\.In*Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(ACL\)*\.ArXiv:2604\.23530\.
- Zheng et al\. \(2023\)Lianmin Zheng, Wei\-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P Xing, and 1 others\. 2023\.Judging LLM\-as\-a\-judge with MT\-Bench and chatbot arena\.*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Zhuo et al\. \(2024\)Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, and 1 others\. 2024\.BigCodeBench: Benchmarking code generation with diverse function calls and complex instructions\.*arXiv preprint arXiv:2406\.15877*\.