VDAR-Router: Adaptive LLMs Routing via Verbalized Query Difficulty Analysis Retrieval

# Adaptive LLMs Routing via Verbalized Query Difficulty Analysis Retrieval
Source: [https://arxiv.org/html/2607.18098](https://arxiv.org/html/2607.18098)
Yu\-Chien Tang,†Jun\-Chen Hung,†Wen\-Chih Peng, An\-Zi Yen National Yang Ming Chiao Tung University Department of Computer Science Hsinchu, Taiwan tommytyc\.cs10@nycu\.edu\.tw, andyhung\.cs13@nycu\.edu\.tw, wcpeng@cs\.nycu\.edu\.tw, azyen@nycu\.edu\.tw

###### Abstract

Large language models are increasingly used in practical systems, making efficient model selection important for reducing deployment cost\. LLM routing has emerged as a practical solution for allocating each input query to an appropriate model under a desired cost\-performance trade\-off\. Existing routing methods often estimate model suitability from the surface semantics or embedding similarity of the input query\. However, such methods may ignore the underlying difficulty of a query, leading to suboptimal routing decisions\. To address the challenge, we propose VDAR\-Router, a difficulty\-aware retrieval\-based routing framework\. For each input query, VDAR\-Router first generates an explicit difficulty analysis\. It then retrieves historical examples with similar difficulty profiles\. Based on the retrieved records, it estimates candidate model suitability and selects the model using a reward function that considers both performance and cost\. Experiments on three datasets show that VDAR\-Router consistently achieves better cost\-performance trade\-offs than existing baselines\. These results demonstrate the effectiveness of difficulty\-aware retrieval for training\-free LLM routing\. Case studies further show that explicit query analysis helps retrieve more relevant examples and supports more reliable routing decisions\.111Our code is publicly available at[https://anonymous\.4open\.science/r/vdar\-router](https://anonymous.4open.science/r/vdar-router)

VDAR\-Router: Adaptive LLMs Routing viaVerbalized QueryDifficultyAnalysisRetrieval

Yu\-Chien Tang,†Jun\-Chen Hung,†Wen\-Chih Peng, An\-Zi YenNational Yang Ming Chiao Tung UniversityDepartment of Computer Science Hsinchu, Taiwantommytyc\.cs10@nycu\.edu\.tw, andyhung\.cs13@nycu\.edu\.tw,wcpeng@cs\.nycu\.edu\.tw, azyen@nycu\.edu\.tw

11footnotetext:These authors contributed equally to this work\.## 1Introduction

Large language models \(LLMs\) have evolved rapidly and achieved strong performance across a broad range of tasks, including summarization, code generation, and mathematical reasoning\(Xueet al\.,[2024](https://arxiv.org/html/2607.18098#bib.bib13); Wanget al\.,[2024](https://arxiv.org/html/2607.18098#bib.bib56); Tianet al\.,[2025](https://arxiv.org/html/2607.18098#bib.bib12); Singhet al\.,[2026](https://arxiv.org/html/2607.18098#bib.bib55)\)\. At the same time, LLMs are increasingly embedded in agentic workflows that plan actions, invoke tools, retrieve information, inspect code, and make multi\-step decisions\(Yaoet al\.,[2023](https://arxiv.org/html/2607.18098#bib.bib50); Schicket al\.,[2023](https://arxiv.org/html/2607.18098#bib.bib51); Qinet al\.,[2024](https://arxiv.org/html/2607.18098#bib.bib52); Zhenget al\.,[2024](https://arxiv.org/html/2607.18098#bib.bib53)\), with emerging systems such as OpenClaw222[https://openclaw\.ai/](https://openclaw.ai/)reflecting this trend in practical applications\. These workflows often require a sequence of heterogeneous model calls, ranging from simple information extraction to complex reasoning or code editing\. However, always invoking the strongest model can quickly exhaust user budgets or service quotas, while relying on cheaper but weak models may lead to poor responses to difficult queries\. As models differ in their architectures, training data, parameter scales, and capability profiles, they exhibit distinct strengths, costs, latency, and response characteristics across tasks\(Barandoniet al\.,[2026](https://arxiv.org/html/2607.18098#bib.bib19)\), posing a unique challenge in dynamically identifying the most appropriate model within budget for a variety of user queries\.

![Refer to caption](https://arxiv.org/html/2607.18098v1/x1.png)Figure 1:An example of using difficulty analysis to select model instead of using raw query\.A growing body of work has studied LLM routing as a way to strike a balance between response quality and inference cost\. One common direction addresses routing in a binary manner, where simpler queries are assigned to a cheap model and more complex queries are routed to a stronger one\(Chenet al\.,[2024a](https://arxiv.org/html/2607.18098#bib.bib1); Dinget al\.,[2024](https://arxiv.org/html/2607.18098#bib.bib2); Onget al\.,[2025](https://arxiv.org/html/2607.18098#bib.bib5)\)\. Building on this formulation, recent methods train learned routers from labeled examples, preference data, or benchmark performance records to predict the most suitable model for each query\(Chenet al\.,[2024b](https://arxiv.org/html/2607.18098#bib.bib4); Fenget al\.,[2025a](https://arxiv.org/html/2607.18098#bib.bib6); Daiet al\.,[2024](https://arxiv.org/html/2607.18098#bib.bib7); Wanget al\.,[2025](https://arxiv.org/html/2607.18098#bib.bib8)\)\. Other approaches further incorporate retrieval or difficulty estimation, comparing an incoming query with previously observed examples or explicitly modeling query difficulty with similar queries to support routing decisions\(Stripeliset al\.,[2024](https://arxiv.org/html/2607.18098#bib.bib41); Songet al\.,[2025](https://arxiv.org/html/2607.18098#bib.bib35)\)\. Nevertheless, most of these existing methods still heavily depend on the query itself as the primary routing signal\. The router is often expected to learn or retrieve from query representations that mainly encode surface\-level semantics, such as the topic, wording, or task category of the input, while semantic similarity may not always reflect the difficulty characteristics relevant to model selection\.

Based on these findings, we aim to observe the capability required to answer and the difficulty of the query to produce a fine\-grained knowledge of the query\. We hypothesize that if the queries with similar difficulty share similar skills to correctly answer, each model must have similar response quality on these queries, thus providing reliable evidence for model routing\. As illustrated in Figure[1](https://arxiv.org/html/2607.18098#S1.F1), two arithmetic queries can look unrelated in raw text, but both require only simple one\-step reasoning\. Conversely, two website\-generation queries may appear similar in semantics, but one may only require a simple personal webpage while the other demands more complex functionality, planning, and implementation details\. In these cases, if the router understands the performance of each model on queries with similar difficulty analysis, it can better capture the capability ranking of the incoming query and select the most proper one\.

In this paper, we propose VDAR\-Router, a difficulty\-aware retrieval\-based framework for LLM routing\. Given an input query, VDAR\-Router first analyzes its difficulty characteristics and uses the resulting analysis representation to retrieve historically observed queries with similar routing\-relevant challenges\. It then estimates model suitability from the historical performance or preference signals associated with the retrieved examples\. Finally, the routing decision is made by jointly considering response quality and inference cost\. This design allows VDAR\-Router to reuse routing evidence across queries with shared difficulty patterns, while distinguishing superficially similar queries that require different model capabilities\.

In sum, our contributions are threefold:

- •We introduce a novel LLM routing approach by incorporating the verbalized diagnosis of query difficulty as the retrieval objectives\. The analysis result can also serve as human\-interpretable metadata for each routing decision\.
- •We propose VDAR\-Router, a plug\-and\-play retrieval\-based routing framework\. By matching the difficulty characteristics of a test query with historical queries and considering cost to rerank model capability orders, VDAR\-Router enables model suitability estimation without requiring any additional training\.
- •Experimental results on three model selection datasets demonstrate that VDAR\-Router can effectively recognize the relative model capability rankings, providing a refreshing view for practitioners in LLM routing\.

## 2Related Work

![Refer to caption](https://arxiv.org/html/2607.18098v1/x2.png)Figure 2:Overview of the VDAR\-Router workflow\.Efficient LLM routing, which aims to direct an incoming query to the most appropriate model, has recently been explored extensively\. Early work was mainly motivated by the cost\-quality balance in LLM deployment by selecting the answer model from a cheap but weak model and an expensive but strong one\(Chenet al\.,[2024a](https://arxiv.org/html/2607.18098#bib.bib1); Dinget al\.,[2024](https://arxiv.org/html/2607.18098#bib.bib2); Onget al\.,[2025](https://arxiv.org/html/2607.18098#bib.bib5)\)\. These works established the core intuition of LLM routing: different queries require different levels of model capability, and routing can exploit this heterogeneity to reduce cost\.

Recent works build router framework in a supervised manner\. RouterDC\(Chenet al\.,[2024b](https://arxiv.org/html/2607.18098#bib.bib4)\)uses dual contrastive learning to learn query and model representations for LLM selection\. GraphRouter\(Fenget al\.,[2025a](https://arxiv.org/html/2607.18098#bib.bib6)\)represents tasks, queries, and LLMs as a heterogeneous graph and formulates routing as an edge prediction problem\. Some researchers also leverage bandit\-based learning to optimize the routing strategy\(Daiet al\.,[2024](https://arxiv.org/html/2607.18098#bib.bib7); Wanget al\.,[2025](https://arxiv.org/html/2607.18098#bib.bib8)\)\. However, a common limitation of these methods is their reliance on training an additional neural model and the limiting interpretability of each routing decision, which can complicate real\-world deployment\.

Another line of research explores the adoption of the retrieval method to assess whether a LLM candidate is capable of answering a query\.Stripeliset al\.\([2024](https://arxiv.org/html/2607.18098#bib.bib41)\)employs KNN retrieval and selects the LLM achieving the highest performance on retrieved queries\. IRT\-Router\(Songet al\.,[2025](https://arxiv.org/html/2607.18098#bib.bib35)\)shares similar idea, and further applies Item Response Theory \(IRT\) to estimate the numeric query difficulty and calibrate LLMs performance ranking\. Instead of using query embedding, we conduct retrieval over the embedding of verbalized query difficulty analysis to better recognize the performance of each LLMs on queries with similar capability requirement\.

## 3Method

### 3\.1Preliminary

#### 3\.1\.1Problem Formulation

We formally define a training set of queriesQt=\{qt1,qt2,…\}Q\_\{t\}=\\\{q\_\{t\}^\{1\},q\_\{t\}^\{2\},\.\.\.\\\}, a set of candidate LLMs,M=\{m1,m2,…,\}M=\\\{m^\{1\},m^\{2\},\.\.\.,\\\}, and the response performance and cost onQtQ\_\{t\}of each LLM\. The goal of router is to learn fromQtQ\_\{t\}to intelligently select the optimal modelm∈Mm\\in Mfor incoming queryqq, balancing both response quality and cost\.

#### 3\.1\.2VDAR\-Router Overview

We present a training\-free and easy\-to\-implement framework, VDAR\-Router, to address the LLM routing problem through difficulty\-aware query\-model matching\. As illustrated in Figure[2](https://arxiv.org/html/2607.18098#S2.F2), an LLM agentDifficulty Analystis designed to determine the proficiency levels across various capability dimensions required for an LLM to correctly answer a given queryqq\. We first offline collect the difficulty analyses of training queries by prompting Difficulty Analyst, and embed the results in a database \(Section[3\.2](https://arxiv.org/html/2607.18098#S3.SS2)\)\. During routing, after producing the difficulty analysis for incoming queries, we retrieve the most similar difficulty analyses and ranking the candidate models by their performance and cost on the corresponding retrieved queries \(Section[3\.3](https://arxiv.org/html/2607.18098#S3.SS3)\)\. In this way, the capability order of each model is accordingly aligned with its answer correctness of queries with similar difficulty level, and thus enables a more adaptive routing decision\.

### 3\.2Difficulty Analysis Collection

The goal of the Difficulty Analyst is to estimate the difficulty characteristics of an input query by identifying the capabilities required to solve it\. We hypothesize that explicitly reasoning over the required capability dimensions and their corresponding proficiency levels enables the router to build a more fine\-grained understanding of the query, thus improving LLM selection\(Chenet al\.,[2025](https://arxiv.org/html/2607.18098#bib.bib25); Yuet al\.,[2025](https://arxiv.org/html/2607.18098#bib.bib26)\)\. To this end, we design the Difficulty Analyst, denoted asℳ\\mathcal\{M\}, to infer the extent of capabilities necessary for answering a given query\. Inspired byMinaeeet al\.\([2024](https://arxiv.org/html/2607.18098#bib.bib23)\)andShiet al\.\([2025](https://arxiv.org/html/2607.18098#bib.bib24)\), we consider seven capability dimensions: reasoning, comprehension, instruction following, agentic capabilities, knowledge retrieval, coding, and multilingual ability\. These dimensions are integrated into the system prompt ofℳ\\mathcal\{M\}as analytic targets\. Given an input queryqq, the agent produces a query difficulty analysis:

a=ℳ​\(q\),a=\\mathcal\{M\}\(q\),\(1\)whereaadescribes the difficulty characteristics and required capabilities of queryqqacross the predefined dimensions\.

To collect the difficulty analysis for routing, we apply the same analysis process to each queryqtiq\_\{t\}^\{i\}in training setQtQ\_\{t\}\. The generated difficulty analysisatia\_\{t\}^\{i\}is encoded by an embedding modelℰ\\mathcal\{E\}and indexed in database for later retrieval\. This design allows VDAR\-Router to retrieve examples according to difficulty similarity rather than query similarity\. Consequently, lexically different queries may still be retrieved together for model capability reference when they require similar capabilities, and superficially similar queries with different difficulty profiles can be separated\.333The prompt of Difficulty Analyst is in Appendix[A](https://arxiv.org/html/2607.18098#A1)\.

### 3\.3Test\-time Difficulty\-aware Routing

At runtime, given an incoming queryqq, the Difficulty Analyst first produces its difficulty analysisaa\. The generated analysis is then embedded by the same embedding modelℰ\\mathcal\{E\}and is used to retrieve top\-kksimilar analysesara\_\{r\}with similar difficulty characteristics from the database\. The retrievedara\_\{r\}are then mapped back to the corresponding queriesqr∈Qtq\_\{r\}\\in Q\_\{t\}and the response performance of each LLM, where the performance ranking represents the capability order of each model on similar difficulty queries\. However, only selecting the model performed the best onqrq\_\{r\}may ignore the impact of inference cost and always seek the strongest but expensive models\(Jitkrittumet al\.,[2026](https://arxiv.org/html/2607.18098#bib.bib66); Varangot\-Reilleet al\.,[2026](https://arxiv.org/html/2607.18098#bib.bib67)\)\. In light of this, we propose to rerank the order of models by integrating the cost and calculating rewardRR:

R​\(m∣q\)=1k​∑i=1kα⋅pm​\(qri\)−β⋅cm​\(qri\),R\(m\\mid q\)=\\frac\{1\}\{k\}\\sum\_\{i=1\}^\{k\}\\alpha\\cdot p\_\{m\}\(q\_\{r\}^\{i\}\)\-\\beta\\cdot c\_\{m\}\(q\_\{r\}^\{i\}\),\(2\)wherepm​\(qri\)p\_\{m\}\(q\_\{r\}^\{i\}\)denotes the performance of modelmmonqriq\_\{r\}^\{i\}, andcm​\(qri\)c\_\{m\}\(q\_\{r\}^\{i\}\)denotes the cost ofmmonqriq\_\{r\}^\{i\}linearly normalized to\[0,1\]\[0,1\]\. The coefficientsα\\alphaandβ\\betacontrol the trade\-off between performance and cost, withα\+β=1\\alpha\+\\beta=1\. Finally, the router selects the candidate model with the highest reward:

m^=arg⁡maxm∈M⁡R​\(m∣q\)\.\\hat\{m\}=\\arg\\max\_\{m\\in M\}R\(m\\mid q\)\.\(3\)

## 4Experimental Setup

Table 1:Results on RouterBench and LLMRouterBench dataset under differentα\\alphaandβ\\betasettings\. Perf\. stands for Performance, and Cost stands for Total Cost\. The best result is inboldface, while the second best isunderlined\.### 4\.1Implementation Details

Across all experiments and data preprocessing, we set random seed to 42, temperature to 0, retrieval sizekkto 30, and LLMs decoding strategy to greedy for reproducibility\. All the experiments are conducted on two NVIDIA RTX A6000 GPUs\.

### 4\.2Datasets

We evaluate our routing method on three datasets: RouterBench\(Huet al\.,[2024](https://arxiv.org/html/2607.18098#bib.bib3)\), LLMRouterBench\(Liet al\.,[2026](https://arxiv.org/html/2607.18098#bib.bib40)\), and ArenaExpert5K\(Arena,[2025](https://arxiv.org/html/2607.18098#bib.bib44)\)\. RouterBench and LLMRouterBench represent regular routing settings, where the performance of candidate LLMs on each query is available\. In contrast, ArenaExpert5K is used to assess whether each router can accurately predict the relative capability ranking of each model\.

##### Dataset Preprocessing\.

For RouterBench, we sample 10,000 and 11 candidate models from the original dataset to construct our experimental subset\. For LLMRouterBench, we sample 12,732 queries and 11 candidate models\. The sampling is performed before the train\-test split, so both splits are drawn from the same data distribution\. For ArenaExpert5K, we sample 2,678 queries and 105 candidate models\. Detailed preprocessing steps and candidate models are listed in Appendix[B\.1](https://arxiv.org/html/2607.18098#A2.SS1)\.

##### Data Split\.

Each dataset is partitioned into an 80% training setQtQ\_\{t\}and a 20% test setQQ\. The training setQtQ\_\{t\}is used to build the Difficulty Analysis DB and to provide the retrieval pool or training data for the compared routers\. The test setQQis used exclusively for evaluation\. All routers are trained and evaluated using the same split within each dataset to ensure a fair comparison\.

### 4\.3Metrics

For RouterBench and LLMRouterBench, we followSonget al\.\([2025](https://arxiv.org/html/2607.18098#bib.bib35)\)andFenget al\.\([2025a](https://arxiv.org/html/2607.18098#bib.bib6)\)and evaluate each routing method using Performance, Total Cost, and Reward, where Performance and Total Cost measure the answer correctness and total inference cost incurred by the selected model for each routing decision respectively\. Reward measures the performance\-cost balance achieved by the selected model, as defined in Equation[2](https://arxiv.org/html/2607.18098#S3.E2)\. We experiment in twoα\\alphaandβ\\betasettings to simulate different scenarios of cost\-performance trade\-off in real\-world deployment\.

For ArenaExpert5K, we use a different evaluation protocol since the dataset contains only pairwise human preference between models rather than the answer correctness for all models, making the reward metric incomputable\. A pairwise win of modelAAover modelBBonly indicates thatAAis preferred toBBfor that specific comparison, and does not necessarily imply thatAAis globally better across all queries or against all other candidate models\. In this scenario, we report Pairwise Accuracy, Spearman Correlation, and Total Cost\. Pairwise Accuracy measures whether the router correctly predicts the preferred model in a held\-out pairwise comparison, and Spearman Correlation measures the consistency between the model ranking induced by the router and the Elo ranking estimated from the training comparisons\. Total Cost is computed based on the sum of the inference cost of each model selected by the router\. Details are in Appendix[B\.2](https://arxiv.org/html/2607.18098#A2.SS2)\.

### 4\.4Difficulty Analyst LLMs

To assess the VDAR\-Router with different size of LLM as Difficulty Analyst, we use Gemma4\-31B\(Google DeepMind,[2026](https://arxiv.org/html/2607.18098#bib.bib45); Farabet and Lacombe,[2026](https://arxiv.org/html/2607.18098#bib.bib46)\)for RouterBench and LLMRouterBench, and Qwen3\.5\-4B\(Qwen Team,[2026](https://arxiv.org/html/2607.18098#bib.bib47)\)for ArenaExpert5K\. To further evaluate the effect of smaller language model in real\-world deployment, we also experiment with Qwen3\.5\-2B\(Qwen Team,[2026](https://arxiv.org/html/2607.18098#bib.bib47)\)on all datasets\. For embedding model, we use Qwen3\-Embedding\-0\.6B\(Zhanget al\.,[2025](https://arxiv.org/html/2607.18098#bib.bib34)\)for all datasets\.

### 4\.5Baselines

We compare five routing methods in our experiments: KNN\(Stripeliset al\.,[2024](https://arxiv.org/html/2607.18098#bib.bib41)\), RouteLLM\(Onget al\.,[2025](https://arxiv.org/html/2607.18098#bib.bib5)\), RouterDC\(Chenet al\.,[2024b](https://arxiv.org/html/2607.18098#bib.bib4)\), ICL\-Router\(Wanget al\.,[2026](https://arxiv.org/html/2607.18098#bib.bib42)\), and IRT\-Router\(Songet al\.,[2025](https://arxiv.org/html/2607.18098#bib.bib35)\)\. For KNN and RouterDC, we use the implementation provided by thellmrouterpackage\(Fenget al\.,[2025b](https://arxiv.org/html/2607.18098#bib.bib49)\)\. For RouteLLM, we select the introduced MLP classifier variant in our experiment\. For IRT\-Router and ICL\-Router, we implement based on the source code\. Specifically, we adopt Qwen3\-embedding\-0\.6B for IRT\-Router query embedding retrieval and use LoRA\(Huet al\.,[2022](https://arxiv.org/html/2607.18098#bib.bib57)\)to finetune Qwen3\-4B\(Team,[2025](https://arxiv.org/html/2607.18098#bib.bib58)\)for ICL\-Router\. We also report two simple baselines on RouterBench and LLMRouterBench: always routing to the smallest LLM or the largest one \(denoted as Small/Large LLM\)\. Details are in Appendix[C](https://arxiv.org/html/2607.18098#A3)\.

## 5Experimental Results and Discussion

Our experiments aim to answer the following research questions \(RQs\):

RQ1:How is the routing performance of VDAR\-Router compared to baselines? \(Section[5\.1](https://arxiv.org/html/2607.18098#S5.SS1)\)

RQ2:How does the hyper\-parameterkkandα\\alphaaffect the routing performance of VDAR\-Router? \(Section[5\.2](https://arxiv.org/html/2607.18098#S5.SS2)\)

RQ3:What evidence is retrieved during the VDAR\-Router routing? \(Section[5\.3](https://arxiv.org/html/2607.18098#S5.SS3)\)

RQ4:Are the retrieved top\-kkqueries really having a difficulty\-level similar to the incoming query? \(Section[5\.4](https://arxiv.org/html/2607.18098#S5.SS4)\)

### 5\.1RQ1: Quantitative Results

Table[1](https://arxiv.org/html/2607.18098#S4.T1)summarizes the results on RouterBench and LLMRouterBench\. On RouterBench, VDAR\-Router strikes the best performance\-cost balance under both settings\. Although it does not obtain the highest raw performance, it substantially reduces inference cost compared to high\-performance baselines such as ICL\-Router and RouterDC, leading to better overall reward\. This suggests that the proposed method is not simply biased toward always selecting the strongest model, but can make more cost\-effective routing decisions by incorporating reward ranking\. A similar trend is observed on LLMRouterBench\. VDAR\-Router again achieves the best reward under both cost\-performance preferences\. Compared with the RouteLLM and ICL\-Router, which reach lowest cost and highest performance respectively, VDAR\-Router better compromises performance and cost\. In addition, this advantage remains consistent when using a smaller 2B model as the Difficulty Analyst, suggesting that VDAR\-Router can achieve strong routing performance across different analyst model sizes\. This indicates that difficulty\-aware retrieval provides a useful routing signal beyond directly trained routing models or surface\-level query similarity\.

Table 2:Results on ArenaExpert5K dataset\. Acc\. stands for Pairwise Accuracy, SP stands for Spearman Correlation, and Cost stands for Total Cost\.Table[2](https://arxiv.org/html/2607.18098#S5.T2)reports the results on ArenaExpert5K, where the target is to test whether each router can capture the nuance of model capability ranking while only pairwise human preference supervision is available\. VDAR\-Router again achieves the best performance across all metrics\. Compared with the strongest baselines, VDAR\-Router not only predicts pairwise preferences more accurately, but also produces a more reliable global ranking among candidate models\. This improvement also holds when replacing the Difficulty Analyst with Qwen3\.5\-2B, where VDAR\-Router continues to outperform all baselines in both pairwise accuracy and global ranking correlation\. This indicates that VDAR\-Router can better recognize the relative capability ranking of each models against incoming query\.

Table 3:Results of 50%:50% train\-test split ratio\. Reward metric is calculated withα=0\.8\\alpha=0\.8andβ=0\.2\\beta=0\.2\.##### Impact of Data Split Ratio\.

To dive deeper into the efficiency and effectiveness of VDAR\-Router, we further experiment with a 50%:50% train\-test split ratio on each dataset to simulate a scenario with fewer training queries available\. As shown in Table[3](https://arxiv.org/html/2607.18098#S5.T3), IRT\-Router shows relatively competitive accuracy and correlation on ArenaExpert5K, indicating that adopting IRT and query embedding retrieval to estimate query difficulty can provide useful routing signals\. However, VDAR\-Router still outperforms all baseline routers across all datasets, consistently demonstrating better performance\-cost balance and producing a more reliable model capability ranking\. This result shows that difficulty analysis retrieval and reward ranking can more efficiently discern the capability difference between each model with fewerQtQ\_\{t\}, thereby benefiting model routing\.

##### Ablation Studies\.

To understand the importance of each component in VDAR\-Router, we conduct two ablation studies: \(1\) removing the reward ranking and directly selecting the model with the highest estimated performance; and \(2\) replacing the difficulty analysis retrieval with query retrieval\. The results are summarized in Table[4](https://arxiv.org/html/2607.18098#S5.T4), and we can see that both variants lead to weaker routing performance compared with the full VDAR\-Router\. When reward ranking is removed, the framework will only favor models with higher performance without explicitly considering inference cost\. Although this strategy preserves competitive raw performance, it loses the ability to balance performance improvement against additional cost, leading to weaker reward scores\. When the difficulty\-analysis embedding is replaced by the raw query embedding, the router becomes more dependent on surface\-level semantic similarity\. As a result, it retrieves queries that are textually similar but require different capabilities, or miss queries that are lexically different but share similar difficulty characteristics\. This shows that difficulty analysis provides a more suitable representation for retrieving examples that are relevant to model capability\.

Table 4:Results of ablative experiments\. Reward metric is calculated withα=0\.8\\alpha=0\.8andβ=0\.2\\beta=0\.2\.

### 5\.2RQ2: Effects of Hyper\-parameter

![Refer to caption](https://arxiv.org/html/2607.18098v1/x3.png)\(a\)RouterBench
![Refer to caption](https://arxiv.org/html/2607.18098v1/x4.png)\(b\)LLMRouterBench
![Refer to caption](https://arxiv.org/html/2607.18098v1/x5.png)\(c\)ArenaExpert5K

Figure 3:Routing performance under differentkkandα\\alpha\.We analyze the effect of the retrieval sizekkand the reward weightsα\\alphaby settingk∈\{5,10,15,20,25,30\}k\\in\\\{5,10,15,20,25,30\\\}andα∈\{0\.3,0\.4,0\.5,0\.6,0\.7,0\.8,0\.9\}\\alpha\\in\\\{0\.3,0\.4,0\.5,0\.6,0\.7,0\.8,0\.9\\\}, and the results are plotted in Figure[3](https://arxiv.org/html/2607.18098#S5.F3)\. It can be observed that largerα\\alpha, which implies placing more emphasis on response quality during routing, generally leads to better performance and pairwise accuracy across all settings\. This is consistent with our design intuition that integrating the reward score during retrieval for model ranking can dynamically control the performance\-cost balance\.

We further analyze the effect of the retrieval sizekk\. On RouterBench and LLMRouterBench, increasingkkgenerally yields stronger and more stable performance, suggesting that aggregating evidence from a broader set of difficulty\-similar examples reduces the variance of model suitability estimation\(Liet al\.,[2025](https://arxiv.org/html/2607.18098#bib.bib68)\)\. This supports our intuition that queries with similar difficulty profiles tend to induce similar model capability rankings\. In contrast, the trend is less monotonic on ArenaExpert5K\. We attribute this to its pairwise human preference supervision, where labels depend not only on query difficulty but also on response\-relative factors such as style, verbosity, helpfulness, and subjective preference biases\. Thus, enlarging the retrieval neighborhood may introduce examples that are difficulty\-aligned but preference\-heterogeneous, reducing the marginal gain from increasingkk\. This observation is in line with previous work on incorporating user personal preference\(Daiet al\.,[2025](https://arxiv.org/html/2607.18098#bib.bib69); Tranet al\.,[2025](https://arxiv.org/html/2607.18098#bib.bib70)\)and points to a promising future direction of augmenting difficulty\-aware retrieval with preference\-aware or response\-aware signals for subjective pairwise evaluation\.

### 5\.3RQ3: Case Study on Retrieval Targets

Query:Find a closed form for∑k=0n\(\(2​k\+1\)5​\(2​kk\)​\(2​n−2​kn−k\)\)\.\\textstyle\\sum\_\{k=0\}^\{n\}\\left\(\(2k\+1\)^\{5\}\\binom\{2k\}\{k\}\\binom\{2n\-2k\}\{n\-k\}\\right\)\.Query Difficulty:weighted binomial convolution,central binomial coefficients,convolution identities,generating functions,polynomial decomposition,symbolic manipulation,algebraic drift\.KNN Routerqwen3\-235b\-a22b\-2507 \(✗\)R1:Find the sum of all integerskksuch that\(234\)\+\(235\)=\(24k\)\\binom\{23\}\{4\}\+\\binom\{23\}\{5\}=\\binom\{24\}\{k\}\.R2:Find the functionffwith the lowest complexity such that∑m=0n\(−1\)m​\(nm\)​Γ​\(32\+n\)Γ​\(32\+n−m\)≤C​f​\(n\)\\textstyle\\sum\_\{m=0\}^\{n\}\(\-1\)^\{m\}\{n\\choose m\}\\frac\{\\Gamma\(\\frac\{3\}\{2\}\+n\)\}\{\\Gamma\(\\frac\{3\}\{2\}\+n\-m\)\}\\leq Cf\(n\)for alln∈ℕn\\in\\mathbb\{N\}\.R3:Find a closed expression of the infinite product∏n=0∞\(1−e−\(2​n\+1\)​π\)\\textstyle\\prod\_\{n=0\}^\{\\infty\}\(1\-e^\{\-\(2n\+1\)\\pi\}\)\.VDAR Routerglm\-4\.6 \(✓\)R1:multi\-step mathematical insight,recognition of hidden identity,polynomial coefficient reasoning,combinatorial counting\.R2:summation\-order manipulation,counting repeated terms,discrete combinatorial reasoning,symbolic derivation\.R3:specialized combinatorial identities,binomial symmetry,symbolic reasoning, constraint verification\.Figure 4:Case study comparing raw query retrieval and difficulty\-aware retrieval\. The analysis has been truncated and summarized for better readability\.R​iRidenotes theii\-th retrieved example ranked by similarity\.To qualitatively uncover the difference between raw query retrieval and difficulty\-aware retrieval, we conduct a case study on a sample from LLMRouterBench and depict the result in Figure[4](https://arxiv.org/html/2607.18098#S5.F4)\.444Full text of the case study is in Appendix[F](https://arxiv.org/html/2607.18098#A6)\.It can be seen that the standard KNN router retrieves examples that are superficially similar to the input query, sharing lexical and structural patterns such as mathematical notation, summations, binomial coefficients, and closed\-form requests\. However, these retrieved queries do not necessarily require the same underlying reasoning capabilities\. Their similarity mainly reflects topical or notational overlap, which provides limited evidence for estimating which model is suitable for the target query\. In contrast, VDAR\-Router retrieves examples using embeddings of difficulty analyses\. Specifically, the retrieved analyses emphasize similar capability requirements, such as recognizing hidden identities, applying polynomial or coefficient\-based reasoning, performing combinatorial summation, and conducting symbolic manipulation\. These capability\-aligned neighbors provide more informative evidence for estimating model performance on queries requiring similar reasoning skills\. As a result, VDAR\-Router selectsglm\-4\.6, one of the oracle models for this query\. This suggests that difficulty\-analysis embeddings offer a more suitable retrieval space for routing, where neighborhood similarity better reflects model\-relevant capability requirements rather than surface\-form overlap\.

### 5\.4RQ4: Test\-time Difficulty Alignment of Retrieved Queries

![Refer to caption](https://arxiv.org/html/2607.18098v1/x6.png)\(a\)RouterBench
![Refer to caption](https://arxiv.org/html/2607.18098v1/x7.png)\(b\)LLMRouterBench

Figure 5:Rasch difficulty difference between retrieved queries and incoming query\.Finally, we quantitatively examine whether difficulty\-aware retrieval actually retrieves examples with similar difficulty\. For this analysis, we estimate numeric query difficulty using a Bayesian Rasch model\(Rasch,[1993](https://arxiv.org/html/2607.18098#bib.bib43); Scarlatoset al\.,[2025](https://arxiv.org/html/2607.18098#bib.bib59)\), which is commonly used in IRT\(Poloet al\.,[2024](https://arxiv.org/html/2607.18098#bib.bib62); Fernandezet al\.,[2026](https://arxiv.org/html/2607.18098#bib.bib61)\)and Knowledge Tracing\(Zhouet al\.,[2025](https://arxiv.org/html/2607.18098#bib.bib60)\)to model item difficulty\.555Details of Rasch difficulty estimation are in Appendix[D](https://arxiv.org/html/2607.18098#A4)\.We measure the difference of Rasch difficulty between a test queryqqand a retrieved training queryq′q^\{\\prime\}as\|b​\(q\)−b​\(q′\)\|\|b\(q\)\-b\(q^\{\\prime\}\)\|, whereb​\(⋅\)b\(\\cdot\)denotes the estimated Rasch difficulty, and a smaller value indicates better difficulty alignment\. We visualize the distribution of this difficulty gap using an empirical cumulative distribution function \(ECDF\)\. In this plot, thexx\-axis represents the absolute Rasch difficulty gap, and theyy\-axis represents the proportion of retrieved samples whose difficulty gap is smaller than or equal to the correspondingxxvalue\. Therefore, a curve that rises earlier and stays closer to the upper\-left corner indicates that more retrieved examples have similar difficulty\. We compare the proposed difficulty\-analysis\-based retrieval with two baselines: query\-embedding retrieval, which follows the retrieval strategy used in IRT\-Router as well as KNN Router, and random retrieval, which randomly samples training queries\.

As shown in Figure[5](https://arxiv.org/html/2607.18098#S5.F5), the proposed retrieval method retrieves examples with smaller difficulty gap than both baselines\. Compared with query\-embedding retrieval, using difficulty analysis as the retrieval target results in retrieved examples that are more closely aligned with the test query in terms of estimated difficulty\. Compared with random retrieval, the improvement further confirms that the observed alignment is not simply due to chance\. This consolidates the design motivation of VDAR\-Router that historical examples are more useful for routing when they reflect similar difficulty characteristics to the incoming query\.

## 6Conclusion

In this work, we introduce VDAR\-Router, a difficulty\-aware retrieval\-based framework for LLM routing\. VDAR\-Router analyzes the difficulty characteristics and the required capability for a correct model response of an input query, retrieves historically observed examples with similar difficulty patterns, and ranks candidate model suitability based on their performance and cost of the retrieval results\. This design facilitates a training\-free and more human\-interpretable routing decision, while reducing the dependence on explicit routing labels, complete prompt\-level scores, or additional router training\. VDAR\-Router consistently achieves the strongest cost\-performance trade\-off on three datasets, showcasing the effectiveness of our framework\. We believe our results can serve as a stepping stone for more research on adaptive and difficulty\-aware model selection, and several future directions could be further explored under VDAR\-Router framework, such as structured or latent difficulty representations and stronger integration with sparse preference\-based supervision\.

## Limitations

While our proposed framework demonstrates strong results, several limitations remain, pointing to directions for future work\.

##### Test\-time difficulty analysis generation\.

We note that the success of the entire framework highly depends on the generation and retrieval accuracy of the difficulty analysis, which may introduce additional latency\. Although we have demonstrated that adopting smaller model can still reach better routing performance than all baseline methods, future research will be further explored to mitigate this test\-time inference cost, such as finetuning a small language model to generate latent difficulty analysis tokens\.

##### Difficulty similarity and preference heterogeneity\.

We assume that queries with similar difficulty profiles tend to induce similar model capability rankings\. However, this assumption may not always hold in preference\-based settings, where labels can depend not only on task difficulty but also on response style, verbosity, formatting, helpfulness, and subjective preference biases\. This may explain why increasing the retrieval size does not always lead to monotonic improvement on ArenaExpert5K\. Future work could combine difficulty\-aware retrieval with preference\-aware or response\-aware signals to better handle subjective pairwise evaluation scenarios\.

## References

- Arena\-expert\-5k\.Hugging Face\.Note:[https://huggingface\.co/datasets/lmarena\-ai/arena\-expert\-5k](https://huggingface.co/datasets/lmarena-ai/arena-expert-5k)Accessed: 2026\-05\-15Cited by:[§4\.2](https://arxiv.org/html/2607.18098#S4.SS2.p1.1)\.
- S\. Barandoni, L\. Cascone, E\. Marrale, S\. Puccio, and F\. Chiarello \(2026\)Automating customer needs analysis: a comparative study of large language models in the travel industry\.Computers in Industry176,pp\. 104448\.External Links:ISSN 0166\-3615,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.compind.2026.104448),[Link](https://www.sciencedirect.com/science/article/pii/S0166361526000151)Cited by:[§1](https://arxiv.org/html/2607.18098#S1.p1.1)\.
- J\. C\. Chen, S\. Yun, E\. Stengel\-Eskin, T\. Chen, and M\. Bansal \(2025\)Symbolic mixture\-of\-experts: adaptive skill\-based routing for heterogeneous reasoning\.arXiv preprint arXiv:2503\.05641\.Cited by:[§3\.2](https://arxiv.org/html/2607.18098#S3.SS2.p1.3)\.
- L\. Chen, M\. Zaharia, and J\. Zou \(2024a\)FrugalGPT: how to use large language models while reducing cost and improving performance\.Transactions on Machine Learning Research\.Note:Featured CertificationExternal Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=cSimKw5p6R)Cited by:[§1](https://arxiv.org/html/2607.18098#S1.p2.1),[§2](https://arxiv.org/html/2607.18098#S2.p1.1)\.
- S\. Chen, W\. Jiang, B\. Lin, J\. Kwok, and Y\. Zhang \(2024b\)RouterDC: query\-based router by dual contrastive learning for assembling large language models\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=7RQvjayHrM)Cited by:[§1](https://arxiv.org/html/2607.18098#S1.p2.1),[§2](https://arxiv.org/html/2607.18098#S2.p2.1),[§4\.5](https://arxiv.org/html/2607.18098#S4.SS5.p1.1),[Table 1](https://arxiv.org/html/2607.18098#S4.T1.16.16.21.5.1)\.
- X\. Dai, J\. Li, X\. Liu, A\. Yu, and J\. C\. S\. Lui \(2024\)Cost\-effective online multi\-llm selection with versatile reward models\.External Links:2405\.16587,[Link](https://arxiv.org/abs/2405.16587)Cited by:[§1](https://arxiv.org/html/2607.18098#S1.p2.1),[§2](https://arxiv.org/html/2607.18098#S2.p2.1)\.
- Z\. Dai, T\. Feng, and J\. You \(2025\)PersonalizedRouter: personalized LLM routing via graph\-based user preference modeling\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=W80eE3ArAl)Cited by:[§5\.2](https://arxiv.org/html/2607.18098#S5.SS2.p2.3)\.
- D\. Ding, A\. Mallick, C\. Wang, R\. Sim, S\. Mukherjee, V\. Rühle, L\. V\. S\. Lakshmanan, and A\. H\. Awadallah \(2024\)Hybrid LLM: cost\-efficient and quality\-aware query routing\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=02f3mUtqnM)Cited by:[§1](https://arxiv.org/html/2607.18098#S1.p2.1),[§2](https://arxiv.org/html/2607.18098#S2.p1.1)\.
- C\. Farabet and O\. Lacombe \(2026\)Gemma 4: byte for byte, the most capable open models\.Note:[https://blog\.google/innovation\-and\-ai/technology/developers\-tools/gemma\-4/](https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4/)Accessed: 2026\-05\-15Cited by:[§4\.4](https://arxiv.org/html/2607.18098#S4.SS4.p1.1)\.
- T\. Feng, Y\. Shen, and J\. You \(2025a\)GraphRouter: a graph\-based router for LLM selections\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=eU39PDsZtT)Cited by:[§1](https://arxiv.org/html/2607.18098#S1.p2.1),[§2](https://arxiv.org/html/2607.18098#S2.p2.1),[§4\.3](https://arxiv.org/html/2607.18098#S4.SS3.p1.2)\.
- T\. Feng, H\. Zhang, Z\. Lei, H\. Yue, C\. Lin, G\. Liu, and J\. You \(2025b\)LLMRouter: an open\-source library for llm routing\.Note:[https://github\.com/ulab\-uiuc/LLMRouter](https://github.com/ulab-uiuc/LLMRouter)GitHub repositoryCited by:[§4\.5](https://arxiv.org/html/2607.18098#S4.SS5.p1.1)\.
- N\. Fernandez, B\. Kveton, R\. A\. Rossi, A\. Lan, and Z\. Wang \(2026\)RADAR: reasoning\-ability and difficulty\-aware routing for reasoning LLMs\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=CB6Ds5T4ae)Cited by:[§5\.4](https://arxiv.org/html/2607.18098#S5.SS4.p1.7)\.
- Google DeepMind \(2026\)Gemma 4 model card\.Note:[https://ai\.google\.dev/gemma/docs/core/model\_card\_4](https://ai.google.dev/gemma/docs/core/model_card_4)Accessed: 2026\-05\-15Cited by:[§4\.4](https://arxiv.org/html/2607.18098#S4.SS4.p1.1)\.
- E\. J\. Hu, yelong shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§4\.5](https://arxiv.org/html/2607.18098#S4.SS5.p1.1)\.
- Q\. J\. Hu, J\. Bieker, X\. Li, N\. Jiang, B\. Keigwin, G\. Ranganath, K\. Keutzer, and S\. K\. Upadhyay \(2024\)RouterBench: a benchmark for multi\-LLM routing system\.InAgentic Markets Workshop at ICML 2024,External Links:[Link](https://openreview.net/forum?id=IVXmV8Uxwh)Cited by:[§4\.2](https://arxiv.org/html/2607.18098#S4.SS2.p1.1)\.
- W\. Jitkrittum, H\. Narasimhan, A\. S\. Rawat, J\. Juneja, C\. Wang, Z\. Wang, A\. Go, C\. Lee, P\. Shenoy, R\. Panigrahy, A\. K\. Menon, and S\. Kumar \(2026\)Universal model routing for efficient LLM inference\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=ka82fvJ5f1)Cited by:[§3\.3](https://arxiv.org/html/2607.18098#S3.SS3.p1.9)\.
- H\. Li, Y\. Zhang, Z\. Guo, C\. Wang, S\. Tang, Q\. Zhang, Y\. Chen, B\. Qi, P\. Ye, L\. Bai, Z\. Wang, and S\. Hu \(2026\)LLMRouterBench: a massive benchmark and unified framework for llm routing\.arXiv preprint arXiv:2601\.07206\.Cited by:[§4\.2](https://arxiv.org/html/2607.18098#S4.SS2.p1.1)\.
- S\. Li, L\. Stenzel, C\. Eickhoff, and S\. A\. Bahrainian \(2025\)Enhancing retrieval\-augmented generation: a study of best practices\.InProceedings of the 31st International Conference on Computational Linguistics,O\. Rambow, L\. Wanner, M\. Apidianaki, H\. Al\-Khalifa, B\. D\. Eugenio, and S\. Schockaert \(Eds\.\),Abu Dhabi, UAE,pp\. 6705–6717\.External Links:[Link](https://aclanthology.org/2025.coling-main.449/)Cited by:[§5\.2](https://arxiv.org/html/2607.18098#S5.SS2.p2.3)\.
- S\. Minaee, T\. Mikolov, N\. Nikzad, M\. Chenaghlu, R\. Socher, X\. Amatriain, and J\. Gao \(2024\)Large language models: a survey\.arXiv preprint arXiv:2402\.06196\.Cited by:[§3\.2](https://arxiv.org/html/2607.18098#S3.SS2.p1.3)\.
- I\. Ong, A\. Almahairi, V\. Wu, W\. Chiang, T\. Wu, J\. E\. Gonzalez, M\. W\. Kadous, and I\. Stoica \(2025\)RouteLLM: learning to route LLMs from preference data\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=8sSqNntaMr)Cited by:[§1](https://arxiv.org/html/2607.18098#S1.p2.1),[§2](https://arxiv.org/html/2607.18098#S2.p1.1),[§4\.5](https://arxiv.org/html/2607.18098#S4.SS5.p1.1),[Table 1](https://arxiv.org/html/2607.18098#S4.T1.16.16.22.6.1)\.
- F\. M\. Polo, L\. Weber, L\. Choshen, Y\. Sun, G\. Xu, and M\. Yurochkin \(2024\)TinyBenchmarks: evaluating LLMs with fewer examples\.InForty\-first International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=qAml3FpfhG)Cited by:[§5\.4](https://arxiv.org/html/2607.18098#S5.SS4.p1.7)\.
- Y\. Qin, S\. Hu, Y\. Lin, W\. Chen, N\. Ding, G\. Cui, Z\. Zeng, X\. Zhou, Y\. Huang, C\. Xiao, C\. Han, Y\. R\. Fung, Y\. Su, H\. Wang, C\. Qian, R\. Tian, K\. Zhu, S\. Liang, X\. Shen, B\. Xu, Z\. Zhang, Y\. Ye, B\. Li, Z\. Tang, J\. Yi, Y\. Zhu, Z\. Dai, L\. Yan, X\. Cong, Y\. Lu, W\. Zhao, Y\. Huang, J\. Yan, X\. Han, X\. Sun, D\. Li, J\. Phang, C\. Yang, T\. Wu, H\. Ji, G\. Li, Z\. Liu, and M\. Sun \(2024\)Tool learning with foundation models\.ACM Comput\. Surv\.57\(4\)\.External Links:ISSN 0360\-0300,[Link](https://doi.org/10.1145/3704435),[Document](https://dx.doi.org/10.1145/3704435)Cited by:[§1](https://arxiv.org/html/2607.18098#S1.p1.1)\.
- Qwen Team \(2026\)Qwen3\.5: towards native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[§4\.4](https://arxiv.org/html/2607.18098#S4.SS4.p1.1)\.
- G\. Rasch \(1993\)Probabilistic models for some intelligence and attainment tests\.\.ERIC\.Cited by:[§5\.4](https://arxiv.org/html/2607.18098#S5.SS4.p1.7)\.
- A\. Scarlatos, N\. Fernandez, C\. Ormerod, S\. Lottridge, and A\. Lan \(2025\)SMART: simulated students aligned with item response theory for question difficulty prediction\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 25071–25094\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1274/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1274),ISBN 979\-8\-89176\-332\-6Cited by:[§5\.4](https://arxiv.org/html/2607.18098#S5.SS4.p1.7)\.
- T\. Schick, J\. Dwivedi\-Yu, R\. Dessi, R\. Raileanu, M\. Lomeli, E\. Hambro, L\. Zettlemoyer, N\. Cancedda, and T\. Scialom \(2023\)Toolformer: language models can teach themselves to use tools\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=Yacmpz84TH)Cited by:[§1](https://arxiv.org/html/2607.18098#S1.p1.1)\.
- H\. Shi, T\. Zheng, W\. Wang, B\. Xu, C\. Li, C\. Chan, T\. Fan, Y\. Song, and Q\. Yang \(2025\)INFERENCEDYNAMICS: efficient routing across llms through structured capability and knowledge profiling\.arXiv preprint arXiv:2505\.16303\.Cited by:[§3\.2](https://arxiv.org/html/2607.18098#S3.SS2.p1.3)\.
- A\. Singh, A\. Fry, A\. Perelman, A\. Tart, A\. Ganesh, A\. El\-Kishky, A\. McLaughlin, A\. Low, A\. Ostrow, A\. Ananthram, A\. Nathan, A\. Luo, A\. Helyar, A\. Madry, A\. Efremov, A\. Spyra, A\. Baker\-Whitcomb, A\. Beutel, A\. Karpenko, A\. Makelov, A\. Neitz, A\. Wei, A\. Barr, A\. Kirchmeyer, A\. Ivanov, A\. Christakis, A\. Gillespie, A\. Tam, A\. Bennett, A\. Wan, A\. Huang, A\. M\. Sandjideh, A\. Yang, A\. Kumar, A\. Saraiva, A\. Vallone, A\. Gheorghe, A\. G\. Garcia, A\. Braunstein, A\. Liu, A\. Schmidt, A\. Mereskin, A\. Mishchenko, A\. Applebaum, A\. Rogerson, A\. Rajan, A\. Wei, A\. Kotha, A\. Srivastava, A\. Agrawal, A\. Vijayvergiya, A\. Tyra, A\. Nair, A\. Nayak, B\. Eggers, B\. Ji, B\. Hoover, B\. Chen, B\. Chen, B\. Barak, B\. Minaiev, B\. Hao, B\. Baker, B\. Lightcap, B\. McKinzie, B\. Wang, B\. Quinn, B\. Fioca, B\. Hsu, B\. Yang, B\. Yu, B\. Zhang, B\. Brenner, C\. R\. Zetino, C\. Raymond, C\. Lugaresi, C\. Paz, C\. Hudson, C\. Whitney, C\. Li, C\. Chen, C\. Cole, C\. Voss, C\. Ding, C\. Shen, C\. Huang, C\. Colby, C\. Hallacy, C\. Koch, C\. Lu, C\. Kaplan, C\. Kim, C\. Minott\-Henriques, C\. Frey, C\. Yu, C\. Czarnecki, C\. Reid, C\. Wei, C\. Decareaux, C\. Scheau, C\. Zhang, C\. Forbes, D\. Tang, D\. Goldberg, D\. Roberts, D\. Palmie, D\. Kappler, D\. Levine, D\. Wright, D\. Leo, D\. Lin, D\. Robinson, D\. Grabb, D\. Chen, D\. Lim, D\. Salama, D\. Bhattacharjee, D\. Tsipras, D\. Li, D\. Yu, D\. Strouse, D\. Williams, D\. Hunn, E\. Bayes, E\. Arbus, E\. Akyurek, E\. Y\. Le, E\. Widmann, E\. Yani, E\. Proehl, E\. Sert, E\. Cheung, E\. Schwartz, E\. Han, E\. Jiang, E\. Mitchell, E\. Sigler, E\. Wallace, E\. Ritter, E\. Kavanaugh, E\. Mays, E\. Nikishin, F\. Li, F\. P\. Such, F\. de Avila Belbute Peres, F\. Raso, F\. Bekerman, F\. Tsimpourlas, F\. Chantzis, F\. Song, F\. Zhang, G\. Raila, G\. McGrath, G\. Briggs, G\. Yang, G\. Parascandolo, G\. Chabot, G\. Kim, G\. Zhao, G\. Valiant, G\. Leclerc, H\. Salman, H\. Wang, H\. Sheng, H\. Jiang, H\. Wang, H\. Jin, H\. Sikchi, H\. Schmidt, H\. Aspegren, H\. Chen, H\. Qiu, H\. Lightman, I\. Covert, I\. Kivlichan, I\. Silber, I\. Sohl, I\. Hammoud, I\. Clavera, I\. Lan, I\. Akkaya, I\. Kostrikov, I\. Kofman, I\. Etinger, I\. Singal, J\. Hehir, J\. Huh, J\. Pan, J\. Wilczynski, J\. Pachocki, J\. Lee, J\. Quinn, J\. Kiros, J\. Kalra, J\. Samaroo, J\. Wang, J\. Wolfe, J\. Chen, J\. Wang, J\. Harb, J\. Han, J\. Wang, J\. Zhao, J\. Chen, J\. Yang, J\. Tworek, J\. Chand, J\. Landon, J\. Liang, J\. Lin, J\. Liu, J\. Wang, J\. Tang, J\. Yin, J\. Jang, J\. Morris, J\. Flynn, J\. Ferstad, J\. Heidecke, J\. Fishbein, J\. Hallman, J\. Grant, J\. Chien, J\. Gordon, J\. Park, J\. Liss, J\. Kraaijeveld, J\. Guay, J\. Mo, J\. Lawson, J\. McGrath, J\. Vendrow, J\. Jiao, J\. Lee, J\. Steele, J\. Wang, J\. Mao, K\. Chen, K\. Hayashi, K\. Xiao, K\. Salahi, K\. Wu, K\. Sekhri, K\. Sharma, K\. Singhal, K\. Li, K\. Nguyen, K\. Gu\-Lemberg, K\. King, K\. Liu, K\. Stone, K\. Yu, K\. Ying, K\. Georgiev, K\. Lim, K\. Tirumala, K\. Miller, L\. Ahmad, L\. Lv, L\. Clare, L\. Fauconnet, L\. Itow, L\. Yang, L\. Romaniuk, L\. Anise, L\. Byron, L\. Pathak, L\. Maksin, L\. Lo, L\. Ho, L\. Jing, L\. Wu, L\. Xiong, L\. Mamitsuka, L\. Yang, L\. McCallum, L\. Held, L\. Bourgeois, L\. Engstrom, L\. Kuhn, L\. Feuvrier, L\. Zhang, L\. Switzer, L\. Kondraciuk, L\. Kaiser, M\. Joglekar, M\. Singh, M\. Shah, M\. Stratta, M\. Williams, M\. Chen, M\. Sun, M\. Cayton, M\. Li, M\. Zhang, M\. Aljubeh, M\. Nichols, M\. Haines, M\. Schwarzer, M\. Gupta, M\. Shah, M\. Y\. Guan, M\. Huang, M\. Dong, M\. Wang, M\. Glaese, M\. Carroll, M\. Lampe, M\. Malek, M\. Sharman, M\. Zhang, M\. Wang, M\. Pokrass, M\. Florian, M\. Pavlov, M\. Wang, M\. Chen, M\. Wang, M\. Feng, M\. Bavarian, M\. Lin, M\. Abdool, M\. Rohaninejad, N\. Soto, N\. Staudacher, N\. LaFontaine, N\. Marwell, N\. Liu, N\. Preston, N\. Turley, N\. Ansman, N\. Blades, N\. Pancha, N\. Mikhaylin, N\. Felix, N\. Handa, N\. Rai, N\. Keskar, N\. Brown, O\. Nachum, O\. Boiko, O\. Murk, O\. Watkins, O\. Gleeson, P\. Mishkin, P\. Lesiewicz, P\. Baltescu, P\. Belov, P\. Zhokhov, P\. Pronin, P\. Guo, P\. Thacker, Q\. Liu, Q\. Yuan, Q\. Liu, R\. Dias, R\. Puckett, R\. Arora, R\. T\. Mullapudi, R\. Gaon, R\. Miyara, R\. Song, R\. Aggarwal, R\. Marsan, R\. Yemiru, R\. Xiong, R\. Kshirsagar, R\. Nuttall, R\. Tsiupa, R\. Eldan, R\. Wang, R\. James, R\. Ziv, R\. Shu, R\. Nigmatullin, S\. Jain, S\. Talaie, S\. Altman, S\. Arnesen, S\. Toizer, S\. Toyer, S\. Miserendino, S\. Agarwal, S\. Yoo, S\. Heon, S\. Ethersmith, S\. Grove, S\. Taylor, S\. Bubeck, S\. Banesiu, S\. Amdo, S\. Zhao, S\. Wu, S\. Santurkar, S\. Zhao, S\. R\. Chaudhuri, S\. Krishnaswamy, Shuaiqi, Xia, S\. Cheng, S\. Anadkat, S\. P\. Fishman, S\. Tobin, S\. Fu, S\. Jain, S\. Mei, S\. Egoian, S\. Kim, S\. Golden, S\. Mah, S\. Lin, S\. Imm, S\. Sharpe, S\. Yadlowsky, S\. Choudhry, S\. Eum, S\. Sanjeev, T\. Khan, T\. Stramer, T\. Wang, T\. Xin, T\. Gogineni, T\. Christianson, T\. Sanders, T\. Patwardhan, T\. Degry, T\. Shadwell, T\. Fu, T\. Gao, T\. Garipov, T\. Sriskandarajah, T\. Sherbakov, T\. Korbak, T\. Kaftan, T\. Hiratsuka, T\. Wang, T\. Song, T\. Zhao, T\. Peterson, V\. Kharitonov, V\. Chernova, V\. Kosaraju, V\. Kuo, V\. Pong, V\. Verma, V\. Petrov, W\. Jiang, W\. Zhang, W\. Zhou, W\. Xie, W\. Zhan, W\. McCabe, W\. DePue, W\. Ellsworth, W\. Bain, W\. Thompson, X\. Chen, X\. Qi, X\. Xiang, X\. Shi, Y\. Dubois, Y\. Yu, Y\. Khakbaz, Y\. Wu, Y\. Qian, Y\. T\. Lee, Y\. Chen, Y\. Zhang, Y\. Xiong, Y\. Tian, Y\. Cha, Y\. Bai, Y\. Yang, Y\. Yuan, Y\. Li, Y\. Zhang, Y\. Yang, Y\. Jin, Y\. Jiang, Y\. Wang, Y\. Wang, Y\. Liu, Z\. Stubenvoll, Z\. Dou, Z\. Wu, and Z\. Wang \(2026\)OpenAI gpt\-5 system card\.External Links:2601\.03267,[Link](https://arxiv.org/abs/2601.03267)Cited by:[§1](https://arxiv.org/html/2607.18098#S1.p1.1)\.
- W\. Song, Z\. Huang, C\. Cheng, W\. Gao, B\. Xu, G\. Zhao, F\. Wang, and R\. Wu \(2025\)IRT\-router: effective and interpretable multi\-LLM routing via item response theory\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 15629–15644\.External Links:[Link](https://aclanthology.org/2025.acl-long.761/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.761),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2607.18098#S1.p2.1),[§2](https://arxiv.org/html/2607.18098#S2.p3.1),[§4\.3](https://arxiv.org/html/2607.18098#S4.SS3.p1.2),[§4\.5](https://arxiv.org/html/2607.18098#S4.SS5.p1.1),[Table 1](https://arxiv.org/html/2607.18098#S4.T1.16.16.23.7.1)\.
- D\. Stripelis, Z\. Xu, Z\. Hu, A\. D\. Shah, H\. Jin, Y\. Yao, J\. Zhang, T\. Zhang, S\. Avestimehr, and C\. He \(2024\)TensorOpera router: a multi\-model router for efficient LLM inference\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track,F\. Dernoncourt, D\. Preoţiuc\-Pietro, and A\. Shimorina \(Eds\.\),Miami, Florida, US,pp\. 452–462\.External Links:[Link](https://aclanthology.org/2024.emnlp-industry.34/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-industry.34)Cited by:[§1](https://arxiv.org/html/2607.18098#S1.p2.1),[§2](https://arxiv.org/html/2607.18098#S2.p3.1),[§4\.5](https://arxiv.org/html/2607.18098#S4.SS5.p1.1),[Table 1](https://arxiv.org/html/2607.18098#S4.T1.16.16.20.4.1)\.
- Q\. Team \(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§4\.5](https://arxiv.org/html/2607.18098#S4.SS5.p1.1)\.
- Y\. Tian, Y\. Tang, K\. Wang, A\. Yen, and W\. Peng \(2025\)Template\-based financial report generation in agentic and decomposed information retrieval\.InProceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval,SIGIR ’25,New York, NY, USA,pp\. 2706–2710\.External Links:ISBN 9798400715921,[Link](https://doi.org/10.1145/3726302.3730253),[Document](https://dx.doi.org/10.1145/3726302.3730253)Cited by:[§1](https://arxiv.org/html/2607.18098#S1.p1.1)\.
- C\. Tran, S\. Paracha, A\. Hafeez, and S\. Chen \(2025\)Arch\-router: aligning llm routing with human preferences\.arXiv preprint arXiv:2506\.16655\.Cited by:[§5\.2](https://arxiv.org/html/2607.18098#S5.SS2.p2.3)\.
- C\. Varangot\-Reille, C\. Bouvard, and A\. Gourru \(2026\)Generalising LLM routing using past performance retrieval: a few\-shot router is sufficient\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 4: Student Research Workshop\),S\. Baez Santamaria, S\. A\. Somayajula, and A\. Yamaguchi \(Eds\.\),Rabat, Morocco,pp\. 304–319\.External Links:[Link](https://aclanthology.org/2026.eacl-srw.22/),[Document](https://dx.doi.org/10.18653/v1/2026.eacl-srw.22),ISBN 979\-8\-89176\-383\-8Cited by:[§3\.3](https://arxiv.org/html/2607.18098#S3.SS3.p1.9)\.
- C\. Wang, H\. Li, Y\. Zhang, L\. Chen, J\. Chen, P\. Jian, Q\. Zhang, and S\. Hu \(2026\)ICL\-router: in\-context learned model representations for llm routing\.Proceedings of the AAAI Conference on Artificial Intelligence40\(39\),pp\. 33413–33421\.External Links:[Link](https://ojs.aaai.org/index.php/AAAI/article/view/40628),[Document](https://dx.doi.org/10.1609/aaai.v40i39.40628)Cited by:[§4\.5](https://arxiv.org/html/2607.18098#S4.SS5.p1.1),[Table 1](https://arxiv.org/html/2607.18098#S4.T1.16.16.24.8.1)\.
- X\. Wang, Y\. Liu, W\. Cheng, X\. Zhao, Z\. Chen, W\. Yu, Y\. Fu, and H\. Chen \(2025\)MixLLM: dynamic routing in mixed large language models\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 10912–10922\.External Links:[Link](https://aclanthology.org/2025.naacl-long.545/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.545),ISBN 979\-8\-89176\-189\-6Cited by:[§1](https://arxiv.org/html/2607.18098#S1.p2.1),[§2](https://arxiv.org/html/2607.18098#S2.p2.1)\.
- Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang, T\. Li, M\. Ku, K\. Wang, A\. Zhuang, R\. Fan, X\. Yue, and W\. Chen \(2024\)MMLU\-pro: a more robust and challenging multi\-task language understanding benchmark\.InThe Thirty\-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=y10DM6R2r3)Cited by:[§1](https://arxiv.org/html/2607.18098#S1.p1.1)\.
- S\. Xue, Z\. Huang, J\. Liu, X\. Lin, Y\. Ning, B\. Jin, X\. Li, and Q\. Liu \(2024\)Decompose, analyze and rethink: solving intricate problems with human\-like reasoning cycle\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=NPKZF1WDjZ)Cited by:[§1](https://arxiv.org/html/2607.18098#S1.p1.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. R\. Narasimhan, and Y\. Cao \(2023\)ReAct: synergizing reasoning and acting in language models\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=WE_vluYUL-X)Cited by:[§1](https://arxiv.org/html/2607.18098#S1.p1.1)\.
- S\. Yu, Y\. Zhang, Z\. Wang, J\. Yoon, and M\. Bansal \(2025\)MEXA: towards general multimodal reasoning with dynamic multi\-expert aggregation\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 22636–22652\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.1233/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.1233),ISBN 979\-8\-89176\-335\-7Cited by:[§3\.2](https://arxiv.org/html/2607.18098#S3.SS2.p1.3)\.
- Y\. Zhang, M\. Li, D\. Long, X\. Zhang, H\. Lin, B\. Yang, P\. Xie, A\. Yang, D\. Liu, J\. Lin, F\. Huang, and J\. Zhou \(2025\)Qwen3 embedding: advancing text embedding and reranking through foundation models\.arXiv preprint arXiv:2506\.05176\.Cited by:[§4\.4](https://arxiv.org/html/2607.18098#S4.SS4.p1.1)\.
- T\. Zheng, G\. Zhang, T\. Shen, X\. Liu, B\. Y\. Lin, J\. Fu, W\. Chen, and X\. Yue \(2024\)OpenCodeInterpreter: integrating code generation with execution and refinement\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 12834–12859\.External Links:[Link](https://aclanthology.org/2024.findings-acl.762/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.762)Cited by:[§1](https://arxiv.org/html/2607.18098#S1.p1.1)\.
- Y\. Zhou, Z\. Lv, S\. Zhang, and J\. Chen \(2025\)Disentangled knowledge tracing for alleviating cognitive bias\.InProceedings of the ACM on Web Conference 2025,WWW ’25,New York, NY, USA,pp\. 2633–2645\.External Links:ISBN 9798400712746,[Link](https://doi.org/10.1145/3696410.3714607),[Document](https://dx.doi.org/10.1145/3696410.3714607)Cited by:[§5\.4](https://arxiv.org/html/2607.18098#S5.SS4.p1.7)\.

## Appendix

## Appendix ADifficulty Analysis Prompt

The system prompt we use for the Difficulty Analyst is shown in Figure[6](https://arxiv.org/html/2607.18098#A1.F6)\.

Your role as an assistant is to analyze the difficulty of a given query for a large language model through a systematic long thinking process analysis\. You will be provided with the user query\. You need to evaluate the incoming query on several key dimensions: reasoning, comprehension, instruction following, agentic, knowledge retrieval, coding, multilingual\. For each dimension, elaborate on the specific challenges and required capabilities\. Please structure your response into Summary\. In the Summary section, based on the analysis, explorations, and reflections from the Think section, systematically present the summary you think for the query difficulty\. The summary should remain a clear, concise expression style and detail necessary difficulty description to reach the conclusion, formatted as follows: <summary\> final formatted, precise, and clear summary </summary\>\. Now, try to analyze the following query through the above guidelines:

Figure 6:Prompt used for query difficulty analysis\.
## Appendix BArenaExpert5K Evaluation Details

### B\.1Dataset Preprocess

We first normalize each model name into its corresponding OpenRouter666[https://openrouter\.ai/](https://openrouter.ai/)model identifier and retrieve its per\-token API price\. If a model name cannot be converted into a valid OpenRouter identifier, or if its pricing information is unavailable, we use the price ofopenai/gpt\-4o\-minias the fallback price\.

In addition, we remove records labeled asboth badortie, since these cases do not provide a clear preference signal for comparing the two candidate answers\. For each remaining pairwise comparison, we convert the pair into two model\-level records: the winning model is assigned a reward of 1, while the losing model is assigned a reward of 0\. To reduce potential distribution imbalance, we also ensure that every candidate model appears at least once in both the training and test splits\.

We further filter out prompt pairs with empty answers and those whoseevaluation\_orderis not equal to 1, retaining only meaningful first\-order preference records for training and testing\. After preprocessing, the final ArenaExpert5K subset contains 2,678 queries and 5,500 answers across 105 models\. The complete candidate models of each dataset are summarized in Table[5](https://arxiv.org/html/2607.18098#A6.T5)\.

### B\.2Evaluation Metrics

##### Pairwise Accuracy\.

Each test instance contains a queryqiq\_\{i\}, two candidate models, and a preference label\. Letmi\+m\_\{i\}^\{\+\}be the preferred model andmi−m\_\{i\}^\{\-\}be the non\-preferred model\. Pairwise accuracy is computed as:

PairwiseAcc=1N​∑i=1N𝕀​\[R​\(mi\+∣qi\)\>R​\(mi−∣qi\)\],\\mathrm\{PairwiseAcc\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbb\{I\}\\left\[R\(m\_\{i\}^\{\+\}\\mid q\_\{i\}\)\>R\(m\_\{i\}^\{\-\}\\mid q\_\{i\}\)\\right\],\(4\)whereR​\(m∣qi\)R\(m\\mid q\_\{i\}\)is the router reward score\.

##### Spearman Correlation\.

For each queryqiq\_\{i\}, the router produces a model rankingπiR\\pi\_\{i\}^\{R\}according to the reward scores\. We compare this ranking with the reference rankingπtrainElo\\pi^\{\\mathrm\{Elo\}\}\_\{\\mathrm\{train\}\}, which is constructed from the preference records in the training split using Elo rating\. The Spearman correlation for queryqiq\_\{i\}is computed as:

ρi=Spearman​\(πiR,πtrainElo\)\.\\rho\_\{i\}=\\mathrm\{Spearman\}\\left\(\\pi\_\{i\}^\{R\},\\pi^\{\\mathrm\{Elo\}\}\_\{\\mathrm\{train\}\}\\right\)\.\(5\)
The final score is averaged over all test queries:

SpearmanCorr=1N​∑i=1Nρi\.\\mathrm\{SpearmanCorr\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\rho\_\{i\}\.\(6\)

##### Evaluation Cost\.

During evaluation, we report the total routing cost of the top\-1 selected models\. Since the output length of the selected model is unavailable, we only compute the input token cost\. This simulates the observable API cost incurred when the router sends each query to its predicted top\-1 model\.

Let

m^i=arg⁡maxm∈M⁡R​\(m∣qi\)\\hat\{m\}\_\{i\}=\\arg\\max\_\{m\\in M\}R\(m\\mid q\_\{i\}\)\(7\)be the model selected by the router for queryqiq\_\{i\}\. The total evaluation cost is computed as:

TotalCost=∑i=1Nnin​\(qi\)⋅pin​\(m^i\),\\mathrm\{TotalCost\}=\\sum\_\{i=1\}^\{N\}n\_\{\\mathrm\{in\}\}\(q\_\{i\}\)\\cdot p\_\{\\mathrm\{in\}\}\(\\hat\{m\}\_\{i\}\),\(8\)wherenin​\(qi\)n\_\{\\mathrm\{in\}\}\(q\_\{i\}\)is the input token length of queryqiq\_\{i\}, andpin​\(m^i\)p\_\{\\mathrm\{in\}\}\(\\hat\{m\}\_\{i\}\)is the input token price of the selected model\.

### B\.3Runtime Cost Estimation

During routing, the router estimates the expected cost of each candidate model using both input and output token costs\. Since the actual output length is unavailable before generation, we approximate it using the average output length of the retrieved examples\.

For queryqqand candidate modelmm, the estimated runtime cost is:

Cost^​\(q,m\)=nin​\(q\)⋅pin​\(m\)\+n^out​\(q,m\)⋅pout​\(m\),\\widehat\{\\mathrm\{Cost\}\}\(q,m\)=n\_\{\\mathrm\{in\}\}\(q\)\\cdot p\_\{\\mathrm\{in\}\}\(m\)\+\\widehat\{n\}\_\{\\mathrm\{out\}\}\(q,m\)\\cdot p\_\{\\mathrm\{out\}\}\(m\),\(9\)wherenin​\(q\)n\_\{\\mathrm\{in\}\}\(q\)is the input token length,pin​\(m\)p\_\{\\mathrm\{in\}\}\(m\)andpout​\(m\)p\_\{\\mathrm\{out\}\}\(m\)are the input and output token prices, andn^out​\(q,m\)\\widehat\{n\}\_\{\\mathrm\{out\}\}\(q,m\)is the estimated output length\.

The estimated output length is computed as:

n^out​\(q,m\)=1\|ℐm​\(q\)\|​∑i∈ℐm​\(q\)nout​\(qi,m\),\\widehat\{n\}\_\{\\mathrm\{out\}\}\(q,m\)=\\frac\{1\}\{\|\\mathcal\{I\}\_\{m\}\(q\)\|\}\\sum\_\{i\\in\\mathcal\{I\}\_\{m\}\(q\)\}n\_\{\\mathrm\{out\}\}\(q\_\{i\},m\),\(10\)whereℐm​\(q\)\\mathcal\{I\}\_\{m\}\(q\)denotes the retrieved examples associated with modelmm\.

The estimated runtime costCost^​\(q,m\)\\widehat\{\\mathrm\{Cost\}\}\(q,m\)is then used as the cost term in the original reward function \(Equation[2](https://arxiv.org/html/2607.18098#S3.E2)\) to compute the final routing score for each candidate model\.

## Appendix CBaseline Details

### C\.1KNN

KNN is a retrieval\-based router that performs model selection based on similar queries in the training set\. For each input query, it retrieves the top\-kknearest training examples using query embeddings\. The router then aggregates the model performance or preference signals from the retrieved examples to estimate the suitability of each candidate model\. In our experiments, KNN uses the original query embedding for retrieval\. This differs from VDAR\-Router, which performs retrieval over difficulty analysis representations rather than the raw query representation\.

### C\.2RouteLLM

RouteLLM is a supervised routing framework for cost\-performance\-aware model selection\. Among the router variants discussed in RouteLLM, we adopt the MLP\-based router in our experiments\. The router takes the query representation as input and predicts a score over candidate models\. During inference, the candidate model with the highest predicted score is selected as the routing result\.

### C\.3RouterDC

RouterDC is a learned router that models the compatibility between queries and candidate models\. It learns query\-side and model\-side representations and predicts how suitable each model is for a given input query\. During inference, RouterDC computes a compatibility score for each candidate model and selects the model with the highest score\.

### C\.4IRT\-Router

IRT\-Router predicts the performance of each candidate LLM on a given query using an item\-response\-theory\-based model\. Following the original formulation, it represents both queries and LLMs as embeddings, where the LLM embedding is constructed from the model profile\. The router then estimates the interaction between query attributes and LLM abilities to predict model performance\.

The original paper proposes two variants, MIRT\-Router and NIRT\-Router\. MIRT\-Router uses a multidimensional IRT formulation, while NIRT\-Router further incorporates a relevance vector to associate queries with ability dimensions\. The predicted performance is finally combined with fixed model cost to determine the routed model\. In our experiments, we employ the MIRT\-Router variant\.

### C\.5ICL\-Router

ICL\-Router is a trainable routing method that represents candidate models through in\-context learned capability profiles\. It uses an embedding model and a projector to convert queries into vector representations, which are then interpreted by an LLM\-based router\. The method is trained in two stages: query reconstruction training aligns the projected query vectors with the router’s semantic space, while model routing training teaches the router to predict whether a candidate model can correctly answer a query\. For each candidate model, ICL\-Router constructs a capability profile from representative query\-performance pairs\. At inference time, the router combines the input query vector with each model’s capability profile and predicts the probability that the model will answer correctly\. The model with the highest predicted probability is selected\.

## Appendix DRasch Difficulty Estimation Implementation

We estimate query\-level difficulty with a Bayesian Rasch model, treating each query as an item and each candidate LLM as an examinee\. Given the observed correctness scorexm​q∈\[0,1\]x\_\{mq\}\\in\[0,1\]of modelmmon queryqq, we model the response probability asp​\(xm​q=1∣θm,b​\(q\)\)=σ​\(θm−b​\(q\)\)p\(x\_\{mq\}=1\\mid\\theta\_\{m\},b\(q\)\)=\\sigma\(\\theta\_\{m\}\-b\(q\)\), whereθm\\theta\_\{m\}denotes the latent ability of modelmmandb​\(q\)b\(q\)denotes the latent difficulty of queryqq\. Since our routing labels may be fractional success rates rather than binary outcomes, we optimize the likelihood using binary cross\-entropy with continuous targets\. We place independent zero\-mean Gaussian priors on both model abilities and query difficulties, and approximate the posterior with a mean\-field Gaussian variational distribution\. The variational parameters are optimized by minimizing the stochastic negative ELBO, which consists of a minibatch\-scaled binary cross\-entropy term and closed\-form KL regularization against the Gaussian priors\.

To place training and test queries on a common Rasch scale, we use a two\-stage estimation procedure\. First, we jointly infer model abilities and training\-query difficulties from the training routing responses\. Second, we freeze the learned posterior mean of each model ability and infer only the difficulties of test queries using the test routing responses\. This anchoring step ensures that both training and test query difficulties are measured relative to the same set of model abilities, making them comparable for our downstream proximity analysis\. We use the posterior mean ofb​\(q\)b\(q\)as the final Rasch difficulty score for each query, where larger values indicate harder queries\. Algorithm[1](https://arxiv.org/html/2607.18098#algorithm1)summarizes the fitting procedure\.

## Appendix EUse of AI assistants

We use AI to help our coding and grammar correction in our paper writing\. We do not use AI to help us in related paper surveys\.

## Appendix FFull Case Study

The label “Ranknn” corresponds toRnR\_\{n\}in Figure[4](https://arxiv.org/html/2607.18098#S5.F4), denoting thenn\-th retrieved example ranked by retrieval similarity\.

### F\.1Question

Input QuestionFind a closed form for the following sum:∑k=0n\(\(2​k\+1\)5​\(2​kk\)​\(2​n−2​kn−k\)\)\\sum\_\{k=0\}^\{n\}\(\(2k\+1\)^\{5\}\\binom\{2k\}\{k\}\\binom\{2n\-2k\}\{n\-k\}\)

Difficulty Analysisbased on the provided guidelines and the context of previous analyses, here is the systematic difficulty analysis for the query: \*\*"find a closed form for the following sum:∑k=0n\(\(2​k\+1\)5​\(2​kk\)​\(2​n−2​kn−k\)\)\\sum\_\{k=0\}^\{n\}\(\(2k\+1\)^\{5\}\\binom\{2k\}\{k\}\\binom\{2n\-2k\}\{n\-k\}\)"\*\*\#\#\# \*\*overall difficulty assessment: high\*\*this query is a sophisticated mathematical challenge that moves beyond simple pattern matching or basic formula application\. it requires a synthesis of advanced combinatorial identities, symbolic manipulation, and rigorous verification\.—\#\#\# \*\*dimension\-by\-dimension analysis\*\*\#\#\#\# \*\*1\. knowledge retrieval \(high difficulty\)\*\* the model cannot rely on a simple lookup table for this specific sum\. it requires retrieval of several specialized mathematical tools: \* \*\*central binomial coefficients:\*\* the model must recognize the properties of\(2​kk\)\\binom\{2k\}\{k\}and\(2​n−2​kn−k\)\\binom\{2n\-2k\}\{n\-k\}\. \* \*\*convolution identities:\*\* the core of the problem relies on the identity∑k=0n\(2​kk\)​\(2​n−2​kn−k\)=4n\\sum\_\{k=0\}^\{n\}\\binom\{2k\}\{k\}\\binom\{2n\-2k\}\{n\-k\}=4^\{n\}\. the model must retrieve this specific identity as the "base" for the more complex sum\. \* \*\*generating functions:\*\* to handle the\(2​k\+1\)5\(2k\+1\)^\{5\}term, the model may need to retrieve knowledge of generating functions for central binomial coefficients, specifically the identity11−4​x=∑n=0∞\(2​nn\)​xn\\frac\{1\}\{\\sqrt\{1\-4x\}\}=\\sum\_\{n=0\}^\{\\infty\}\\binom\{2n\}\{n\}x^\{n\}\.\#\#\#\# \*\*2\. reasoning \(very high difficulty\)\*\* the primary challenge is the polynomial term\(2​k\+1\)5\(2k\+1\)^\{5\}\. this requires a multi\-step reasoning process: \* \*\*polynomial decomposition:\*\* the model must decide how to handle\(2​k\+1\)5\(2k\+1\)^\{5\}\. it cannot simply distribute the power; it must express\(2​k\+1\)5\(2k\+1\)^\{5\}in terms of falling factorials or operators \(likex​dd​xx\\frac\{d\}\{dx\}\) that interact cleanly with the binomial coefficients\. \* \*\*operator application:\*\* if using generating functions, the model must apply the operator\(2​x​dd​x\+1\)5\(2x\\frac\{d\}\{dx\}\+1\)^\{5\}to the generating function of the convolution\. this involves repeated differentiation and algebraic simplification, where the risk of a "calculation hallucination" is extremely high\. \* \*\*symmetry exploitation:\*\* the model must recognize the symmetry betweenkkandn−kn\-kto simplify the sum, potentially splitting the polynomial into symmetric and anti\-symmetric parts to reduce the number of terms to calculate\.\#\#\#\# \*\*3\. comprehension \(moderate difficulty\)\*\* \* \*\*latex parsing:\*\* the model must correctly parse the latex notation, specifically the exponents and the binomial coefficients\. \* \*\*goal identification:\*\* the term "closed form" implies that the final answer must be a function ofnnwithout any summation symbols \(∑\\sum\), which is a strict constraint on the output format\.\#\#\#\# \*\*4\. instruction following \(low difficulty\)\*\* the instruction is a straightforward mathematical request\. the difficulty lies not in understanding \*what\* to do, but in the \*execution\* of the mathematical derivation\.\#\#\#\# \*\*5\. coding/agentic \(n/a\)\*\* unless the model decides to use a python interpreter \(like sympy\) to solve the sum, this is a purely symbolic reasoning task\. if the model uses a tool, the difficulty shifts to correctly formulating the sympy code to handle the summation of binomial products\.—\#\#\# \*\*critical success factors & potential failure points\*\*\* \*\*critical success factor:\*\* the ability to relate the sum to the identity∑\(2​kk\)​\(2​n−2​kn−k\)=4n\\sum\\binom\{2k\}\{k\}\\binom\{2n\-2k\}\{n\-k\}=4^\{n\}and then systematically apply the\(2​k\+1\)5\(2k\+1\)^\{5\}weight\. \* \*\*potential failure point \(algebraic drift\):\*\* the expansion of\(2​k\+1\)5\(2k\+1\)^\{5\}creates many terms\. a model is likely to make a sign error or a coefficient error during the expansion or the subsequent simplification\. \* \*\*potential failure point \(over\-simplification\):\*\* the model might attempt to use a general formula for∑kp​\(2​kk\)​\(2​n−2​kn−k\)\\sum k^\{p\}\\binom\{2k\}\{k\}\\binom\{2n\-2k\}\{n\-k\}that it "remembers" incorrectly, leading to a plausible\-looking but wrong closed form\.\#\#\# \*\*summary table\*\*\| dimension \| difficulty \| key challenge \| \| :— \| :— \| :— \| \| \*\*reasoning\*\* \| very high \| handling the 5th\-degree polynomial weight via operators or identities\. \| \| \*\*knowledge retrieval\*\* \| high \| recalling specific binomial convolution identities and generating functions\. \| \| \*\*comprehension\*\* \| moderate \| correct interpretation of latex and "closed form" requirement\. \| \| \*\*instruction following\*\* \| low \| simple direct request\. \| \| \*\*overall\*\* \| \*\*high\*\* \| requires advanced symbolic manipulation and precision\. \|

### F\.2VDAR\-Router Retrieved Result

Rank 1Retrieved QueryLet∏i=019\(1\+x3i\+x2⋅3i\+x3⋅3i\)=∑kak​xk\\prod\_\{i=0\}^\{19\}\\left\(1\+x^\{3^\{i\}\}\+x^\{2\\cdot 3^\{i\}\}\+x^\{3\\cdot 3^\{i\}\}\\right\)=\\sum\_\{k\}a\_\{k\}x^\{k\}\. Find∑kak2\\sum\_\{k\}a\_\{k\}^\{2\}\.Retrieved Difficulty Analysis\*\*Query Difficulty Analysis Summary\*\*\*\*Overall Difficulty Level: HIGH\*\*This mathematical problem presents significant challenges for a large language model across multiple dimensions:\*\*1\. Reasoning \(HIGH\)\*\* \- Requires multi\-step mathematical insight to connect polynomial coefficients to combinatorial counting \- Needs recognition of the identity∑kak2=\[x0\]​P​\(x\)​P​\(x−1\)\\sum\_\{k\}a\_\{k\}^\{2\}=\[x^\{0\}\]P\(x\)P\(x^\{\-1\}\)\- Involves understanding modified base\-3 representation structure where digits range from 0\-3 \- The mathematical derivation is non\-obvious and requires competition\-level insight\*\*2\. Comprehension \(MODERATE\-HIGH\)\*\* \- Must parse complex mathematical notation with nested products, exponents, and summations \- Requires understanding the conceptual meaning of polynomial coefficients in this context \- The relationship between the given expression and target quantity is not immediately apparent\*\*3\. Knowledge Retrieval \(HIGH\)\*\* \- Needs specialized knowledge of polynomial algebra and coefficient properties \- Requires access to Parseval\-like identities for coefficient sums \- Must understand combinatorial counting techniques in polynomial products \- This is advanced mathematical knowledge beyond general training data\*\*4\. Instruction Following \(LOW\)\*\* \- Query is clearly stated with well\-defined objectives \- No ambiguity in what needs to be computed\*\*5\. Agentic/Coding/Multilingual \(LOW\)\*\* \- No tool use, planning, or external information gathering required \- Pure mathematical derivation with no code generation \- Single language with standard mathematical notation\*\*Key Challenge\*\*: The primary difficulty is the mathematical reasoning and knowledge required to recognize and apply the correct mathematical identity\. Once the insight is found, the computation is straightforward \(4204^\{20\}\), but finding that insight requires sophisticated mathematical thinking at an advanced competition level\.\*\*Estimated Difficulty\*\*: This represents an AIME/IMO\-level competition mathematics problem that would challenge most general\-purpose LLMs without specialized mathematical training or reasoning capabilities\.

Rank 2Retrieved QueryGiven:σ2​\(n\)=∑i=1ni2=∑i=1n∑j=1i\(2​j−1\)\\sigma\_\{2\}\(n\)=\\sum\_\{i=1\}^\{n\}i^\{2\}=\\sum\_\{i=1\}^\{n\}\\sum\_\{j=1\}^\{i\}\(2j\-1\)\. This double sum can be interpreted as a single Lebesgue sum:σ2​\(n\)=∑j=1n\(2​j−1\)​μ2​\(j\)\.\\sigma\_\{2\}\(n\)=\\sum\_\{j=1\}^\{n\}\(2j\-1\)\\mu\_\{2\}\(j\)\.determine tha form of the functionμ2​\(j\)\\mu\_\{2\}\(j\)Retrieved Difficulty Analysis\*\*Query Difficulty Analysis Summary\*\*\*\*Overall Difficulty Level: Moderate to High\*\*\*\*Key Difficulty Dimensions:\*\*1\. \*\*Mathematical Reasoning \(High Difficulty\)\*\*: Requires multi\-step logical deduction to convert a double summation to a single summation\. The model must recognize that for each j from 1 to n, the term\(2​j−1\)\(2j\-1\)appears in the inner sum for alli≥ji\\geq j, givingμ2​\(j\)=\(n−j\+1\)\\mu\_\{2\}\(j\)=\(n\-j\+1\)\. This requires understanding summation order manipulation and counting functions\.2\. \*\*Mathematical Comprehension \(Moderate Difficulty\)\*\*: Requires correct parsing of LaTeX notation and understanding the relationship between the three equation forms\. The term "Lebesgue sum" in discrete context may cause confusion as it typically refers to continuous measure theory\.3\. \*\*Knowledge Retrieval \(Moderate Difficulty\)\*\*: Requires specialized knowledge of summation manipulation techniques, discrete mathematics, and combinatorial counting principles not commonly found in general training data\.4\. \*\*Instruction Following \(Low Difficulty\)\*\*: The instruction is clear and direct \- determine the form ofμ2​\(j\)\\mu\_\{2\}\(j\)\- but requires mathematical derivation rather than simple information extraction\.5\. \*\*Pattern Recognition \(Moderate Difficulty\)\*\*: Requires recognizing the counting pattern in the nested summation structure where each j value appears multiple times across different i values\.\*\*Critical Success Factors:\*\* \- Correct interpretation of double summation limits \- Accurate counting of term occurrences across nested sums \- Understanding of discrete counting functions in summation contexts\*\*Potential Failure Points:\*\* \- Misinterpreting the relationship between i and j in the double sum \- Confusing continuous Lebesgue measure with discrete counting weights \- Arithmetic errors in deriving the counting functionμ2​\(j\)\\mu\_\{2\}\(j\)\*\*Expected Solution:\*\*μ2​\(j\)=\(n−j\+1\)\\mu\_\{2\}\(j\)=\(n\-j\+1\)forj=1,2,…,nj=1,2,\.\.\.,n

Rank 3Retrieved QueryFind the sum of all integerskksuch that\(234\)\+\(235\)=\(24k\)\\binom\{23\}\{4\}\+\\binom\{23\}\{5\}=\\binom\{24\}\{k\}\.Retrieved Difficulty AnalysisThe query presents a \*\*medium difficulty\*\* challenge for a Large Language Model, primarily driven by \*\*Knowledge Retrieval\*\* and \*\*Reasoning\*\* dimensions\.\*\*Key Difficulty Factors:\*\* 1\. \*\*Specialized Knowledge:\*\* Requires specific recall of combinatorial identities, specifically Pascal’s Identity and the Symmetry Property of Binomial Coefficients\. This is not general knowledge but domain\-specific mathematical curriculum\. 2\. \*\*Reasoning Complexity:\*\* The task involves multi\-step symbolic manipulation\. The critical challenge is recognizing that the equation yields \*\*two\*\* valid integer solutions forkkdue to symmetry \(\(nk\)=\(nn−k\)\\binom\{n\}\{k\}=\\binom\{n\}\{n\-k\}\), rather than just one\. A model must avoid the common bias of assuming a unique solution\. 3\. \*\*Constraint Verification:\*\* The model must ensure the identified integers fall within the valid domain \(0≤k≤n0\\leq k\\leq n\) and sum them correctly\. 4\. \*\*Instruction Following:\*\* The meta\-task requires analyzing the difficulty without directly solving the problem, demanding a clear separation between the analysis process and the mathematical solution\.\*\*Overall Assessment:\*\* The query tests an LLM’s ability to apply precise mathematical rules and handle edge cases in symmetry, rather than simple pattern matching\. Success depends on accurate symbolic reasoning and comprehensive domain knowledge retrieval\.

### F\.3KNN\-Router Retrieved Result

Rank 1Retrieved QueryFind the sum of all integerskksuch that\(234\)\+\(235\)=\(24k\)\\binom\{23\}\{4\}\+\\binom\{23\}\{5\}=\\binom\{24\}\{k\}\.

Rank 2Retrieved Queryfind the function f with the lowest complexity such that there exists a constant C\>0 with∑m=0n\(−1\)m​\(nm\)​Γ​\(32\+n\)Γ​\(32\+n−m\)≤C​f​\(n\)\\sum\_\{m=0\}^\{n\}\(\-1\)^\{m\}\{n\\choose m\}\\frac\{\\Gamma\(\\frac\{3\}\{2\}\+n\)\}\{\\Gamma\(\\frac\{3\}\{2\}\+n\-m\)\}\\leq Cf\(n\)for alln∈Nn\\in N

Rank 3Retrieved QueryFind a closed expression of the infinite product∏n=0∞\(1−e−\(2​n\+1\)​π\)\\prod\_\{n=0\}^\{\\infty\}\(1\-e^\{\-\(2n\+1\)\\pi\}\)\.

Table 5:Candidate Models across each datasets\.Algorithm 1Bayesian Rasch difficulty estimation\. Stage 1 jointly fits model abilities and training\-query difficulties\. Stage 2 fixes the learned model\-ability means and optimizes only test\-query difficulties on test responses\.mu\_theta=center\(logit\(model\_success\_rate\)\)

mu\_beta=center\(\-logit\(query\_success\_rate\)\)

rho\_theta=full\(\[M\],softplus\_inv\(0\.1\)\)

rho\_beta=full\(\[Q\],softplus\_inv\(0\.1\)\)

params=\[mu\_theta,rho\_theta,mu\_beta,rho\_beta\]

opt=Adam\(params,lr=lr\)

forstepinrange\(T\):

idx=randint\(0,N,size=B\)

qi,mi,yi=query\_ids\[idx\],model\_ids\[idx\],labels\[idx\]

s\_theta=softplus\(rho\_theta\)\+eps

s\_beta=softplus\(rho\_beta\)\+eps

theta=mu\_theta\[mi\]\+s\_theta\[mi\]\*randn\(B\)

beta=mu\_beta\[qi\]\+s\_beta\[qi\]\*randn\(B\)

nll=bce\_with\_logits\(theta\-beta,yi,reduction=’sum’\)

kl\_theta=gaussian\_kl\(mu\_theta,s\_theta,sigma\_theta\)

kl\_beta=gaussian\_kl\(mu\_beta,s\_beta,sigma\_beta\)

loss=\(N/B\)\*nll\+kl\_theta\+kl\_beta

opt\.zero\_grad\(\);loss\.backward\(\)

clip\_grad\_norm\_\(params,max\_norm=10\.0\)

opt\.step\(\)

returnmu\_theta,softplus\(rho\_theta\)\+eps,mu\_beta,softplus\(rho\_beta\)\+eps