Pro-Router: Token-Aware Progressive Model Routing with Adaptive Edge-Cloud Collaboration for Efficient Multimodal LLM Inference

# Pro-Router: Token-Aware Progressive Model Routing with Adaptive Edge-Cloud Collaboration for Efficient Multimodal LLM Inference
Source: [https://arxiv.org/html/2608.28726](https://arxiv.org/html/2608.28726)
###### Abstract

The remarkable performance of multimodal large language models \(MLLMs\) comes at the cost of substantial computational overhead, posing significant challenges to real\-time deployment and cost effectiveness\. Existing model routing approaches either decide from coarse request\-level features alone or spend one or several extra language model passes to inspect the generated response, leaving the token\-level uncertainty signals that emerge during generation unused\. To address these limitations, we propose Pro\-Router, a token\-aware progressive model routing method with adaptive edge\-cloud collaboration for efficient multimodal LLM inference\. Pro\-Router employs a two\-stage progressive decision mechanism\. First, a lightweight prompt pre\-scorer module performs rapid pre\-screening before token generation begins, guiding apparently simple requests to small models\. Second, a token\-aware verifier reads the sampling probability distribution of each token the small model generates, estimating the model’s confidence in its own output to determine, per request, whether the answer ships or escalates to the cloud\-based high\-precision model\. Furthermore, we design an adaptive edge\-cloud serving pipeline that sizes every dispatch to each device’s measured service rate, so both the edge and the cloud tiers stay fully utilized without manual parameter tuning and are not impacted by the network latency\. Extensive experiments on multiple multimodal benchmark datasets and models demonstrate the effectiveness of Pro\-Router\. Compared to other methods, it achieves the highest routing accuracy and improves routing speed by more than 10×\\times\. Its serving pipeline also reaches more than 75% higher end\-to\-end throughput than the existing model routing pipeline\. Our code is available athttps://github\.com/xinyuangui2/pro\-router\.

1Anyscale

2Department of Computer Science and Engineering, Mississippi State University

3Institute of Computing Technology, Chinese Academy of Sciences

4Institute of AI for Industries, Chinese Academy of Sciences

## Introduction

Multimodal large language models \(MLLMs\) define the state of the art in visual question answering, document and chart understanding, and multimodal reasoning\([Bai et al\. 2025](https://arxiv.org/html/2608.28726#bib.bib3);[Li et al\. 2025](https://arxiv.org/html/2608.28726#bib.bib27);[Agrawal et al\. 2024](https://arxiv.org/html/2608.28726#bib.bib2)\), and they are increasingly the default backend of production multimodal services\. Serving them is expensive\. State\-of\-the\-art large models have tens of billions of parameters, autoregressive decoding streams all of them through the accelerator for every generated token\([Kwon et al\. 2023](https://arxiv.org/html/2608.28726#bib.bib26);[Dao et al\. 2022](https://arxiv.org/html/2608.28726#bib.bib13)\), and a single image expands into hundreds or thousands of tokens that inflate prefill compute and KV\-cache footprint\([Chen et al\. 2024b](https://arxiv.org/html/2608.28726#bib.bib8)\), so these models effectively live on expensive cloud GPUs\. Yet a large share of real traffic does not need them\. A multimodal small language model \(MSLM\), an order of magnitude smaller, answers many requests equally well\([Ong et al\. 2025](https://arxiv.org/html/2608.28726#bib.bib38);[Ding et al\. 2024](https://arxiv.org/html/2608.28726#bib.bib14);[Chen, Zaharia, and Zou 2024](https://arxiv.org/html/2608.28726#bib.bib7)\), and such a model fits comfortably on a much cheaper edge device\.

![Refer to caption](https://arxiv.org/html/2608.28726v1/paper_figure1.png)Figure 1:The two options existing routing methods take\. \(a\) Request\-only routers score the difficulty of each request from the prompt alone, before any token is generated; requests under the threshold go to the small model and the rest go directly to the large model\. \(b\) Response\-based routers let the small model answer first and score the shipping confidence of each response; confident responses ship, and the rest escalate to the large model for regeneration\.Existing model routing methods split by when they decide \(Figure[1](https://arxiv.org/html/2608.28726#Sx1.F1)\)\. Request\-only routers decide from the request alone, before any token is generated\([Ong et al\. 2025](https://arxiv.org/html/2608.28726#bib.bib38);[Ding et al\. 2024](https://arxiv.org/html/2608.28726#bib.bib14);[Tang et al\. 2025](https://arxiv.org/html/2608.28726#bib.bib44)\); they are cheap, but inaccurate, because the request by itself does not reveal whether the small model can actually answer it\. Response\-based routers decide after the small model generates its answer\([Chen, Zaharia, and Zou 2024](https://arxiv.org/html/2608.28726#bib.bib7);[Kadavath et al\. 2022](https://arxiv.org/html/2608.28726#bib.bib25);[Aggarwal et al\. 2024](https://arxiv.org/html/2608.28726#bib.bib1)\), which is far more informative, but they spend one or several passes of another language model per request, and this extra compute kills the gains of model routing\. In addition, edge\-cloud collaboration has been studied alongside model routing\([Ding et al\. 2024](https://arxiv.org/html/2608.28726#bib.bib14);[Tang et al\. 2025](https://arxiv.org/html/2608.28726#bib.bib44)\)and LLM serving in general\([Zhang et al\. 2025](https://arxiv.org/html/2608.28726#bib.bib55);[Jin and Wu 2025](https://arxiv.org/html/2608.28726#bib.bib23);[Hao et al\. 2024](https://arxiv.org/html/2608.28726#bib.bib19);[Wang et al\. 2024a](https://arxiv.org/html/2608.28726#bib.bib48)\)\. The small models are deployed on cheap edge devices close to the client, the expensive large models are deployed on cloud devices, and a good routing algorithm utilizes both tiers to sustain their throughput\. However, these existing collaboration pipelines interact between the edge and the cloud frequently and keep the two tiers dependent on each other, so network fluctuation stalls whichever tier is waiting and wastes its GPU cycles\.

To address the above problems, we propose a token\-aware progressive model routing framework with adaptive edge\-cloud collaboration for efficient multimodal LLM inference, named Pro\-Router\. A lightweight prompt pre\-scorer guides the incoming traffic before any token is generated, so complicated requests are likely to be routed directly to the large model\. A token\-aware verifier then reads the per\-token sampling distributions that the small model’s own decoding already produces and decides, per request, whether the answer ships or escalates\. We also propose an adaptive edge\-cloud collaboration pipeline: a global scheduler orders requests by difficulty, feeds the small models from the easy end and the large models from the escalations and the hard end, and sizes every dispatch to each device’s reported service rate\. The main contributions can be summarized as follows\.

- •We propose progressive routing, a two\-stage method that decides both before and after small model decoding\. A lightweight prompt pre\-scorer guides initial traffic from the prompt alone, and a token\-aware verifier reads the decoding token distributions to make the final ship\-or\-escalate decision\.
- •We build an adaptive edge\-cloud collaboration pipeline that fully utilizes the edge and cloud devices, is unaffected by high network latency, and scales to many devices, all without manual tuning\.
- •We run extensive experiments across 15 benchmarks, two modalities, three models, and four baselines against a real 72B cloud target\. Our router achieves the highest routing accuracy while producing its routing signal more than 10×\\timesfaster than prior methods, and our pipeline reaches more than 75% higher end\-to\-end throughput than the existing model routing pipeline, holds it under extreme network latency, and scales linearly to more machines\.

## Related Work

Model routing serves each request with the model that can handle it, and it descends from selective prediction and the reject option\([Chow 1970](https://arxiv.org/html/2608.28726#bib.bib10);[Geifman and El\-Yaniv 2017](https://arxiv.org/html/2608.28726#bib.bib16);[El\-Yaniv and Wiener 2010](https://arxiv.org/html/2608.28726#bib.bib15)\)and from learning to defer\([Mozannar and Sontag 2020](https://arxiv.org/html/2608.28726#bib.bib36);[Cortes, DeSalvo, and Mohri 2016](https://arxiv.org/html/2608.28726#bib.bib12)\)\. Existing methods split into two groups by what they read to decide\. Request\-only routers decide from the request alone\. RouteLLM\([Ong et al\. 2025](https://arxiv.org/html/2608.28726#bib.bib38)\)trains its router on human preference data, Hybrid LLM\([Ding et al\. 2024](https://arxiv.org/html/2608.28726#bib.bib14)\)predicts the quality gap between the small and the large model from the query, and RouterDC\([Chen et al\. 2024c](https://arxiv.org/html/2608.28726#bib.bib9)\)learns a contrastive query encoder that selects among candidate LLMs\. PerLLM\([Yang et al\. 2024](https://arxiv.org/html/2608.28726#bib.bib52)\)accommodates diverse user requirements regarding accuracy and cost through edge\-cloud collaboration\. Recent work reaches the multimodal setting\. ECVL\-ROUTER\([Tang et al\. 2025](https://arxiv.org/html/2608.28726#bib.bib44)\)selects between edge and cloud VLMs under scenario requirements, and AVR\([Liu et al\. 2026](https://arxiv.org/html/2608.28726#bib.bib29)\)routes the steps of computer\-use agents between small and large VLMs by semantic difficulty\. Because these routers never observe the small model’s actual answer, they stay cheap, but their accuracy suffers from not knowing the response\.

Response\-based routing methods decide after the small model answers\. FrugalGPT\([Chen, Zaharia, and Zou 2024](https://arxiv.org/html/2608.28726#bib.bib7)\)trains a DistilBERT\-based scorer on the answer*text*, capturing its surface form\. The work in\([Hu et al\. 2026b](https://arxiv.org/html/2608.28726#bib.bib22)\)leverages MLLMs to provide semantically enhanced adaptive routing for edge\-cloud\. SAEC\([Tian and Yang 2026](https://arxiv.org/html/2608.28726#bib.bib45)\)proposes a scene\-aware enhanced edge\-cloud collaborative routing framework\. AIVD\([Hu et al\. 2026a](https://arxiv.org/html/2608.28726#bib.bib21)\)can dynamically balance accuracy and efficiency through flexible routing for large and small models\. Moa\-off\([Yang et al\. 2025](https://arxiv.org/html/2608.28726#bib.bib51)\)introduces a modality\-aware heterogeneous routing mechanism that adaptively distributes computation between edge and cloud\. P\(True\)\([Kadavath et al\. 2022](https://arxiv.org/html/2608.28726#bib.bib25)\)re\-asks the model whether its own answer is true, and AutoMix\([Aggarwal et al\. 2024](https://arxiv.org/html/2608.28726#bib.bib1)\)samples a few\-shot self\-verifier several times to estimate a verification probability\. Observing the response makes these methods more accurate, but each spends one or several extra language model passes per request\.

![Refer to caption](https://arxiv.org/html/2608.28726v1/paper_figure2.png)Figure 2:The workflow of the proposed Pro\-Router\.
## Method

Figure[2](https://arxiv.org/html/2608.28726#Sx2.F2)illustrates the overall design and workflow of Pro\-Router, which performs efficient multimodal LLM inference by dynamically routing each request between edge\-deployed multimodal small language models \(MSLM\) and cloud\-based multimodal large language models \(MLLM\)\. Concretely, Pro\-Router follows a two\-stage progressive routing mechanism\. The first stage is a lightweight prompt pre\-scorer module that analyzes each incoming request before generation and assigns it an early difficulty estimate; requests enter a scored input buffer ordered by this estimate, and the easiest are served by the MSLMs running on the edge devices\. The second stage is a token\-aware verifier that, during edge decoding, reads the per\-token sampling distributions the MSLM already produces and accumulates the model’s confidence in its own output\. From this signal, Pro\-Router decides per request whether to accept the edge answer or escalate it into an escalated buffer\. The escalated requests, together with the hard requests in the scored input buffer, are sent to the cloud\-based MLLMs for high\-precision inference\. Meanwhile, an adaptive edge\-cloud collaboration pipeline keeps both the edge and cloud devices highly utilized by tracking their measured throughput and sizing each dispatch accordingly\. Through this progressive routing and adaptive serving strategy, Pro\-Router achieves high routing accuracy at low latency while utilizing the edge and cloud devices efficiently and robustly across different network latencies and cluster sizes\.

![Refer to caption](https://arxiv.org/html/2608.28726v1/paper_figure3.png)Figure 3:The structure of the token\-aware verifier\.### Token\-Aware Progressive Routing

We formalize the routing objective by defining a gain function that encapsulates the inherent trade\-offs among inference accuracy, response latency, and system resource utilization\. The deployment architecture consists ofNcN\_\{c\}cloud devices, each provisioned with a nominal capacityLiL\_\{i\}\(in requests per second\), andNeN\_\{e\}edge devices with capacitiesSjS\_\{j\}, respectively\. Owing to dynamic network conditions and fluctuating workloads, the actual available fractions of these capacities are characterized byηLi,ηSj∈\[0,1\]\\eta\_\{L\_\{i\}\},\\eta\_\{S\_\{j\}\}\\in\[0,1\], denoting the effective utilization ratios for each cloud and edge device\. Each request the small model answers receives a shipping confidenceppand ships whenp≥τp\\geq\\tau, where the thresholdτ\\tauis fixed in advance by calibration on a validation pool\. Let the ship rate bes=Pr\[p≥τ\]s=\\Pr\[\\,p\\geq\\tau\\,\], the probability that the verifier accepts the small model’s answer so the request is served entirely at the edge; the remaining1−s1\-sescalate to the cloud\. Among the shipped requests the conditional accuracy isa=Pr⁡\[correct∣p≥τ\]a=\\Pr\[\\,\\text\{correct\}\\mid p\\geq\\tau\\,\], and each wrongly shipped answer incurs a penaltyλ\\lambda\. The pipeline then gains:

G\\displaystyle G=∑i=1NcηLi​Li\+∑j=1NeηSj​Sj⋅s⁡\(a−λ⁡\(1−a\)\)\\displaystyle=\\sum\_\{i=1\}^\{N\_\{c\}\}\\eta\_\{L\_\{i\}\}L\_\{i\}\+\\sum\_\{j=1\}^\{N\_\{e\}\}\\eta\_\{S\_\{j\}\}S\_\{j\}\\cdot s\\,\(a\-\\lambda\(1\-a\)\)\(1\)=∑i=1NcηLi​Li\+\(∑j=1NeηSj​Sj\)​s​\(a−λ⁡\(1−a\)\)\\displaystyle=\\sum\_\{i=1\}^\{N\_\{c\}\}\\eta\_\{L\_\{i\}\}L\_\{i\}\+\\left\(\\sum\_\{j=1\}^\{N\_\{e\}\}\\eta\_\{S\_\{j\}\}S\_\{j\}\\right\)s\\,\(a\-\\lambda\(1\-a\)\)per second, where the second line factorsssandaaout because one verifier serves every small model device, so both are optimized globally and do not vary across machines\.

Equation[1](https://arxiv.org/html/2608.28726#Sx3.E1)increases in two directions, a higher ship ratessand a higher routing accuracyaa\. The ship rate rises when easy requests reach the small models and hard ones go straight to the large models, a decision available*before*generation; the accuracy rises when the module making the final decision is itself accurate, which needs evidence available only*after*generation\. Two directions at two moments call for two mechanisms, which is what progressive routing provides: a lightweight prompt pre\-scorer scoring request difficulty before generation, and a token\-aware verifier deciding after it\.

Existing prompt\-only routers run a language model over the prompt\([Ong et al\. 2025](https://arxiv.org/html/2608.28726#bib.bib38);[Ding et al\. 2024](https://arxiv.org/html/2608.28726#bib.bib14);[Tang et al\. 2025](https://arxiv.org/html/2608.28726#bib.bib44)\), showing that the prompt already carries a strong difficulty signal, but such a model is overkill here: the pre\-scorer only has to be fast and approximately right because it guides traffic rather than deciding it, and any model competing for the accelerator lowersηS\\eta\_\{S\}in Equation[1](https://arxiv.org/html/2608.28726#Sx3.E1)\. We therefore keep the pre\-scorer off the accelerator, using three cheap features: the TF\-IDF vector of the prompt text\([Salton and Buckley 1988](https://arxiv.org/html/2608.28726#bib.bib40)\), the image count, and the prompt length\. TF\-IDF captures the lexical shape of difficulty, such as the question style and the domain vocabulary; the image count is a multimodal difficulty proxy that costs no vision encoding; and the prompt length restores the size signal that TF\-IDF normalization removes\. The features pass through a trained logistic regression,

ppre​\(x\)=σ⁡\(w⊤​ϕ​\(x\)\+b\),p\_\{\\text\{pre\}\}\(x\)\\;=\\;\\sigma\\\!\\big\(w^\{\\top\}\\phi\(x\)\+b\\big\),\(2\)whereϕ⁡\(x\)\\phi\(x\)concatenates the three features of requestxx\. The regression output works as a difficulty guidance that routes the traffic downstream\.

The pre\-scorer is only guidance; whether an answer ships is decided by the token\-aware verifier\. Previous methods pass the response through another language model or through the small model itself\([Chen, Zaharia, and Zou 2024](https://arxiv.org/html/2608.28726#bib.bib7);[Kadavath et al\. 2022](https://arxiv.org/html/2608.28726#bib.bib25);[Aggarwal et al\. 2024](https://arxiv.org/html/2608.28726#bib.bib1)\), spending fresh compute to re\-estimate what the small model already revealed: every decoding step emits a distribution over the vocabulary stating how certain the model was about the token it produced\([Kadavath et al\. 2022](https://arxiv.org/html/2608.28726#bib.bib25)\)\. Discarding that signal and paying a second model to recover it lowersηS\\eta\_\{S\}for no informational gain\. Our verifier reads it directly\. To materialize the token sampling distributions, we build an uncertainty measurement sequence from four features of each generated tokentt, the log\-probability of the emitted tokenyty\_\{t\}, the maximum probability, the negative entropy of the top\-KKprobability mass, and the position fraction,

ℓt\\displaystyle\\ell\_\{t\}=log⁡pt​\(yt\),\\displaystyle=\\log p\_\{t\}\(y\_\{t\}\),\(3\)mt\\displaystyle m\_\{t\}=maxv⁡pt​\(v\),\\displaystyle=\\max\\nolimits\_\{v\}p\_\{t\}\(v\),ht\\displaystyle h\_\{t\}=−ℋt,\\displaystyle=\-\\mathcal\{H\}\_\{t\},ρt\\displaystyle\\rho\_\{t\}=t/T,\\displaystyle=t/T,wherept​\(v\)p\_\{t\}\(v\)is the decoding distribution over candidate tokensvvat stepttandℋt=−∑v∈top\-​Kpt\(v\)logpt\(v\)\\mathcal\{H\}\_\{t\}=\-\\sum\\limits\_\{v\\in\\text\{top\-\}K\}p\_\{t\}\(v\)\\log p\_\{t\}\(v\)is the Shannon entropy of the top\-KKprobability mass, withK=20K=20\. The entropy is low when the distribution concentrates on a few candidate tokens, so the top\-KKentropy featurehth\_\{t\}rises when the small model is certain at steptt\. Stackingxt=\(ℓt,mt,ht,ρt\)x\_\{t\}=\(\\ell\_\{t\},m\_\{t\},h\_\{t\},\\rho\_\{t\}\)over theTTgenerated tokens gives the input sequenceX∈ℝT×4X\\in\\mathbb\{R\}^\{T\\times 4\}\. The whole feature stack is a side product of the small model’s own decoding and requires no extra model compute\. After a linear projection with positional encoding\([Vaswani et al\. 2017](https://arxiv.org/html/2608.28726#bib.bib46)\), a transformer encoder applies the sequence interaction, as shown in Figure[3](https://arxiv.org/html/2608.28726#Sx3.F3)\. It is followed by a mean pool and a sigmoid layer that outputs one confidence score, the shipping confidence of the answer\. Both components run entirely on the CPU of the edge device hosting the small model\.

### Adaptive Edge\-Cloud Collaboration Pipeline

While progressive routing reduces the average per\-request computation cost, the actual end\-to\-end throughput of a hybrid edge\-cloud system is governed by how efficiently the serving pipeline itself orchestrates heterogeneous devices\. Naive dispatch strategies often leave parts of the cluster underutilized: fast edge devices idle while waiting for cloud responses, or cloud GPUs remain idle because the edge verifier becomes a bottleneck\. We therefore design a closed\-loop pipeline that explicitly models device capabilities, accounts for communication overhead, and dynamically balances load across the edge and cloud tiers\. Our cluster consists of two tiers\. The*edge tier*comprisesNeN\_\{e\}devices, each equipped with CPUs and GPUs\. The CPUs handle the lightweight prompt pre\-scorer and the token\-aware verifier, while the GPUs run the MSLM\. The*cloud tier*consists ofNcN\_\{c\}powerful GPU servers running the MLLM\. All devices are connected via a network with heterogeneous latency\. We model the service capacity of a deviceiias its measured inference throughputμi\\mu\_\{i\}\(requests processed per second\)\. For the cloud tier, the effective throughput is further constrained by the network round\-trip time \(RTT\)RR, since each request dispatched to the cloud incurs a communication delay before the first token is generated\.

Algorithm 1Adaptive Edge\-Cloud Collaboration Pipeline1:Edge devices

𝒟edge\\mathcal\{D\}\_\{\\text\{edge\}\}\(MSLM on GPU, pre\-scorer/verifier on CPU\), cloud devices

𝒟cloud\\mathcal\{D\}\_\{\\text\{cloud\}\}\(MLLM on GPU\)

2:Adaptive dispatch for all requests

3:Maintain scored buffer

ℬin\\mathcal\{B\}\_\{\\text\{in\}\}, escalated buffer

ℬesc\\mathcal\{B\}\_\{\\text\{esc\}\}, EMA throughput

μ¯i\\bar\{\\mu\}\_\{i\}
4:loop

5:Pre\-scorer on edge CPU scores each incoming request into

ℬin\\mathcal\{B\}\_\{\\text\{in\}\}
6:Collect

\(μi,βi\)\(\\mu\_\{i\},\\beta\_\{i\}\)from all devices; update

μ¯i←α​μi\+\(1−α\)​μ¯i\\bar\{\\mu\}\_\{i\}\\leftarrow\\alpha\\mu\_\{i\}\+\(1\-\\alpha\)\\bar\{\\mu\}\_\{i\}
7:Compute

ni←⌈Tw​μ¯i⌉\+Bin\_\{i\}\\leftarrow\\lceil T\_\{w\}\\bar\{\\mu\}\_\{i\}\\rceil\+B\_\{i\}if

βi=0\\beta\_\{i\}=0, else

ni←0n\_\{i\}\\leftarrow 0
8:foreach

i∈𝒟edgei\\in\\mathcal\{D\}\_\{\\text\{edge\}\}do

9:Dispatch

PopTop​\(ℬin,ni\)\\text\{PopTop\}\(\\mathcal\{B\}\_\{\\text\{in\}\},n\_\{i\}\)to edge MSLM for inference & verification

10:endfor

11:foreach

j∈𝒟cloudj\\in\\mathcal\{D\}\_\{\\text\{cloud\}\}do

12:

b​a​t​c​h←PopTop​\(ℬesc,nj\)batch\\leftarrow\\text\{PopTop\}\(\\mathcal\{B\}\_\{\\text\{esc\}\},n\_\{j\}\)
13:

b​a​t​c​h←b​a​t​c​h∪PopBottom​\(ℬin,nj−\|b​a​t​c​h\|\)batch\\leftarrow batch\\cup\\text\{PopBottom\}\(\\mathcal\{B\}\_\{\\text\{in\}\},n\_\{j\}\-\|batch\|\)
14:Dispatch

b​a​t​c​hbatchto cloud MLLM

15:endfor

16:Update

TwT\_\{w\}adaptively based on network RTT and cluster throughput

17:endloop

A naive serving pipeline sends every request to the edge first and escalates only after the verifier rejects, so its throughput is bounded by

Λ≤∑i=1Neμiedge\+min⁡\{∑j=1Ncμjcloud,NcR\+1/μ¯cloud\},\\Lambda\\leq\\sum\_\{i=1\}^\{N\_\{e\}\}\\mu\_\{i\}^\{\\text\{edge\}\}\+\\min\\left\\\{\\sum\_\{j=1\}^\{N\_\{c\}\}\\mu\_\{j\}^\{\\text\{cloud\}\},\\;\\frac\{N\_\{c\}\}\{R\+1/\\bar\{\\mu\}\_\{\\text\{cloud\}\}\}\\right\\\},\(4\)whereμiedge\\mu\_\{i\}^\{\\text\{edge\}\}andμjcloud\\mu\_\{j\}^\{\\text\{cloud\}\}are the per\-device throughputs andμ¯cloud\\bar\{\\mu\}\_\{\\text\{cloud\}\}the average cloud inference rate\. The ideal throughput instead*adds*the two tiers,Λ⋆=∑i=1Neμiedge\+∑j=1Ncμjcloud\\Lambda^\{\\star\}=\\sum\_\{i=1\}^\{N\_\{e\}\}\\mu\_\{i\}^\{\\text\{edge\}\}\+\\sum\_\{j=1\}^\{N\_\{c\}\}\\mu\_\{j\}^\{\\text\{cloud\}\}, reached only when every device runs at its own service rate at the same time\. To approach it the pipeline must assign each device a workload proportional to its actual service rate, so that the round\-trip timeRRis not in the critical path\. We introduce a global scheduler \(Algorithm[1](https://arxiv.org/html/2608.28726#alg1)\) that operates over fixed time windows of lengthTwT\_\{w\}\. For each deviceii, the scheduler maintains an exponentially weighted moving average \(EMA\) of its recent throughput:

μ¯i\(t\)←α⋅μi\(t−1\)\+\(1−α\)⋅μ¯i\(t−1\),\\bar\{\\mu\}\_\{i\}^\{\(t\)\}\\leftarrow\\alpha\\cdot\\mu\_\{i\}^\{\(t\-1\)\}\+\(1\-\\alpha\)\\cdot\\bar\{\\mu\}\_\{i\}^\{\(t\-1\)\},\(5\)whereμi\(t−1\)\\mu\_\{i\}^\{\(t\-1\)\}is the measured throughput in the previous window andα∈\(0,1\)\\alpha\\in\(0,1\)is the smoothing factor\. This adaptive tracking captures performance fluctuations due to request diversity and dynamic KV cache pressure\.

At the beginning of each window, the scheduler computes the number of requests to dispatch to deviceiias

ni=⌈Tw⋅μ¯i⌉\+Bi,n\_\{i\}=\\left\\lceil T\_\{w\}\\cdot\\bar\{\\mu\}\_\{i\}\\right\\rceil\+B\_\{i\},\(6\)whereBiB\_\{i\}is a small lookahead margin that prevents the device’s local buffer from draining prematurely, ensuring continuous utilization\. By dispatchingnin\_\{i\}requests, every device receives enough work to sustain it for the entire window, thereby aligning each device’s busy time and eliminating the bottleneck\-idle pattern described in Equation[4](https://arxiv.org/html/2608.28726#Sx3.E4)\.

The global scheduler runs on a dedicated edge CPU node and maintains two logical queues\. At each window, the scheduler fills batches from both buffers as follows:

- •Edge MSLM devicesreceive requests from the*top*of the scored input buffer \(lowest difficulty scores\), since easy requests are most suitable for the lightweight MSLM and yield high ship rates\.
- •Cloud MLLM devicesfirst drain the escalated buffer \(requests already rejected by the verifier\), then take requests from the*bottom*of the scored input buffer \(highest difficulty scores\), ensuring that challenging requests receive the cloud’s full capacity\.

Each device runs one local scheduler as the middle layer between its inference engine and the global scheduler\. It monitors the GPU KV\-cache utilizationui∈\[0,1\]u\_\{i\}\\in\[0,1\]and the buffer occupancyqiq\_\{i\}, the number of pending requests, against two configurable thresholdsθq,θu∈\(0,1\)\\theta\_\{q\},\\theta\_\{u\}\\in\(0,1\)\. To avoid overcommitting the engine and triggering costly preemptions it calls the engine only whileui<θuu\_\{i\}<\\theta\_\{u\}; to avoid holding more requests than the device can drain, it raises a backpressure flagβi=1\\beta\_\{i\}=1whenqi\>θq⋅storage limitq\_\{i\}\>\\theta\_\{q\}\\cdot\\text\{storage limit\}, which is reported to the global scheduler together with the response and the updatedμ¯i\\bar\{\\mu\}\_\{i\}\. Whenβi=1\\beta\_\{i\}=1the global scheduler skips dispatching to deviceiiin the next window, letting it drain its buffer\.

## Experiments

### Evaluation Setup

##### Datasets\.

We evaluate on a suite of 15 benchmarks spanning both modalities\. The nine single\-image benchmarks are C18, A\-OKVQA\([Schwenk et al\. 2022](https://arxiv.org/html/2608.28726#bib.bib42)\), MathVerse\([Zhang et al\. 2024](https://arxiv.org/html/2608.28726#bib.bib56)\), MMStar\([Chen et al\. 2024a](https://arxiv.org/html/2608.28726#bib.bib6)\), HallusionBench\([Guan et al\. 2024](https://arxiv.org/html/2608.28726#bib.bib17)\), RealWorldQA\([xAI 2024](https://arxiv.org/html/2608.28726#bib.bib50)\), OCRBench\([Liu et al\. 2024b](https://arxiv.org/html/2608.28726#bib.bib31)\), MM\-Vet\([Yu et al\. 2024](https://arxiv.org/html/2608.28726#bib.bib53)\), and MathVision\([Wang et al\. 2024b](https://arxiv.org/html/2608.28726#bib.bib49)\); the two multi\-image benchmarks are MileBench\([Song et al\. 2024](https://arxiv.org/html/2608.28726#bib.bib43)\)and MuirBench\([Wang et al\. 2025](https://arxiv.org/html/2608.28726#bib.bib47)\); and the four text benchmarks are MMLU\([Hendrycks et al\. 2021](https://arxiv.org/html/2608.28726#bib.bib20)\), GSM8K\([Cobbe et al\. 2021](https://arxiv.org/html/2608.28726#bib.bib11)\), CoQA\([Reddy, Chen, and Manning 2019](https://arxiv.org/html/2608.28726#bib.bib39)\), and TriviaQA\([Joshi et al\. 2017](https://arxiv.org/html/2608.28726#bib.bib24)\)\. C18 is a stratified mix of five single\-image sources, drawn from ChartQA\([Masry et al\. 2022](https://arxiv.org/html/2608.28726#bib.bib33)\), DocVQA\([Mathew, Karatzas, and Jawahar 2021](https://arxiv.org/html/2608.28726#bib.bib34)\), MathVista\([Lu et al\. 2024](https://arxiv.org/html/2608.28726#bib.bib32)\), MMBench\([Liu et al\. 2024a](https://arxiv.org/html/2608.28726#bib.bib30)\), and MMMU\([Yue et al\. 2024](https://arxiv.org/html/2608.28726#bib.bib54)\)\. Table[1](https://arxiv.org/html/2608.28726#Sx4.T1)lists the suite; test splits range from 106 to 682 records\. All answers are graded for semantic equivalence against the gold answer by a strong LLM judge\([Zheng et al\. 2023](https://arxiv.org/html/2608.28726#bib.bib57)\), with unsure verdicts counted as incorrect\. To align with the real\-world serving scenario, where a single verifier must serve all kinds of incoming data\([Gulrajani and Lopez\-Paz 2021](https://arxiv.org/html/2608.28726#bib.bib18)\), we deploy a single pooled verifier trained on all 15 benchmarks\. At deployment, the operating point follows the coverage\-based rule of selective prediction\([Geifman and El\-Yaniv 2017](https://arxiv.org/html/2608.28726#bib.bib16)\), fixing a ship fractionqqand calibrating the threshold on a validation split as in the method section\.

Table 1:The 15\-benchmark evaluation suite, with eleven vision\-language benchmarks \(nine single\-image, two multi\-image\) and four text benchmarks\.
##### Implementation Details\.

We evaluate three small models, Qwen2\.5\-VL\-7B, LLaVA\-OneVision\-7B, and Pixtral\-12B\. For the large model, we use Qwen2\.5\-VL\-72B\. Serving runs onMMp4d nodes \(eight A100 GPUs each\) as the cloud tier andNNg5\.12 nodes \(four A10G GPUs each\) as the edge tier, withMMandNNswept in the scalability study\. We implement the pipeline on Ray\([Moritz et al\. 2018](https://arxiv.org/html/2608.28726#bib.bib35)\)\. Each device is one Ray actor that wraps its local scheduler, its local buffer, and a vLLM engine\([Kwon et al\. 2023](https://arxiv.org/html/2608.28726#bib.bib26)\), and the global scheduler reaches the actors through asynchronous RPC\. To study the network conditions, we inject extra one\-way delay into every RPC between the global scheduler and the devices\. The verifier is embedded in the vLLM code, so it knows exactly when each request finishes and fetches that request’s token sampling distributions directly\.

##### Baselines\.

We compare against four baselines covering the main classes a router can come from, namely RouteLLM\([Ong et al\. 2025](https://arxiv.org/html/2608.28726#bib.bib38)\), a pretrained 278M query\-only router; FrugalGPT\([Chen, Zaharia, and Zou 2024](https://arxiv.org/html/2608.28726#bib.bib7)\), a DistilBERT\-66M\([Sanh et al\. 2019](https://arxiv.org/html/2608.28726#bib.bib41)\)scorer fine\-tuned on the small model’s answer text; P\(True\)\([Kadavath et al\. 2022](https://arxiv.org/html/2608.28726#bib.bib25)\)and AutoMix\([Aggarwal et al\. 2024](https://arxiv.org/html/2608.28726#bib.bib1)\), self\-verification baselines where P\(True\) asks the small model once whether its own answer is true and AutoMix samples its few\-shot self\-verifier eight times \(its default\) to estimate a verification probability\. Each baseline is reproduced faithfully from its paper with verbatim templates and graded by the same LLM judge on the same labels\.

##### Metrics\.

For routing accuracy we report the area under the receiver operating characteristic curve \(AUROC\) and the area under the shipping accuracy curve, both standard for confidence and uncertainty measures\([Geifman and El\-Yaniv 2017](https://arxiv.org/html/2608.28726#bib.bib16);[El\-Yaniv and Wiener 2010](https://arxiv.org/html/2608.28726#bib.bib15);[Lin, Trivedi, and Sun 2024](https://arxiv.org/html/2608.28726#bib.bib28)\)\. AUROC\([Bradley 1997](https://arxiv.org/html/2608.28726#bib.bib4)\)is the probability that the routing signal scores a randomly chosen correct small model answer above a randomly chosen incorrect one; 0\.5 is chance and 1 is a perfect ranking\. The shipping accuracy is the rate of good responses among those the small model ships, as a function of the ship rate\. We compute it under two labels, the*correctness label*, where a shipped response counts if the LLM judge grades it correct against the gold, with area AUARC\([Nadeem, Zucker, and Hanczar 2009](https://arxiv.org/html/2608.28726#bib.bib37)\); and the*pairwise label*, where a shipped response counts if it wins or ties against the large model’s response, with area AUACC\([Chen et al\. 2023](https://arxiv.org/html/2608.28726#bib.bib5)\)\. For the accuracy\-cost frontier we use the performance gap recovered \(PGR\), the standard metric of the routing literature\([Ong et al\. 2025](https://arxiv.org/html/2608.28726#bib.bib38);[Ding et al\. 2024](https://arxiv.org/html/2608.28726#bib.bib14)\),

PGR=accpipeline−accsmall modelacclarge model−accsmall model,\\mathrm\{PGR\}\\;=\\;\\frac\{\\mathrm\{acc\}\_\{\\text\{pipeline\}\}\-\\mathrm\{acc\}\_\{\\text\{small model\}\}\}\{\\mathrm\{acc\}\_\{\\text\{large model\}\}\-\\mathrm\{acc\}\_\{\\text\{small model\}\}\},\(7\)which shows the fraction of the small\-to\-large quality gap that routing recovers\. For serving efficiency we report end\-to\-end throughput, the number of requests the whole pipeline completes per second measured at steady state after a warm\-up ramp\.

### Experiment Results

#### Token\-Aware Verifier Accuracy and Latency\.

We first compare the routing accuracy of our verifier against the baselines \(Figure[4](https://arxiv.org/html/2608.28726#Sx4.F4)\)\. Across the suite our verifier has the highest mean AUROC on every small model, ahead of all four baselines, and the lead holds on macro PGR and on the shipping accuracy under both the correctness label \(AUARC\) and the pairwise label \(AUACC\); the agreement between the two labels shows the lead does not depend on how correctness is defined\. We attribute the lead both to what the verifier reads and to how it reads it\. The four per\-token statistics capture the model’s belief state at the moment it produced each token, and the verifier consumes them as a trajectory rather than a summary, so a run of low\-confidence tokens in the middle of an answer stays visible instead of being averaged away\. The baselines read weaker evidence\. RouteLLM decides from the prompt alone and never observes the answer, which is why it is the lowest among them; FrugalGPT reads the answer text and captures its surface form but not the uncertainty behind it; and the self\-verification baselines re\-ask the model, which conflates the answer’s correctness with the model’s willingness to endorse it\.

Figure 4:Routing accuracy summaries by small model, our verifier and the four baselines, showing \(a\) mean AUROC, \(b\) macro PGR, \(c\) AUARC, and \(d\) AUACC, all averaged over the benchmarks\. Ours is highest on every metric and every model\.Moreover, our verifier is nearly free to run where every alternative pays a visible price \(Figure[5](https://arxiv.org/html/2608.28726#Sx4.F5)\)\. Producing the routing signal under a steady decode load adds 2 to 3 ms for our verifier\. The scorer baselines add 47 to 119 ms, since RouteLLM’s 278M router and FrugalGPT’s DistilBERT scorer each run a forward on the GPU that steals compute and memory bandwidth from the decode batches; measured against the cheapest of them, our signal is 19 to 28×\\timesfaster\. The self\-verification baselines instead pay whole generation passes, one for P\(True\) and eight for AutoMix, each re\-encoding the image, and their signal costs run from two seconds to nearly two minutes per request\. Those passes also run on the same accelerator as the small model, so they contend with it for compute and KV\-cache memory, which makes the situation worse than the raw latency suggests\. The gap follows the size of the computation each signal runs\. Our verifier is a≈\\approx67k\-parameter head on the CPU, four orders of magnitude smaller than the 66M to 278M side models the scorers add and far smaller than the billions of parameters a self\-verification pass streams; the head is small enough that it neither competes with the small model for the accelerator nor stalls the decode batches\.

Figure 5:Per\-request latency to produce the routing signal, on a log scale\. The bars for P\(True\) and AutoMix are clipped at 200 ms with their true values annotated; our verifier adds 2 to 3 ms, 19 to 28×\\timesless than the cheapest alternative\.
#### End\-to\-End Throughput\.

Figure 6:\(a\) End\-to\-end pipeline throughput by routing signal with two edge devices\. \(b\) End\-to\-end pipeline throughput by pipeline implementation, our pipeline versus the Ray Serve pipeline running the same routing signal\.We now serve the whole pipeline with two edge devices and one cloud device and compare the end\-to\-end throughput of each routing signal \(Figure[6](https://arxiv.org/html/2608.28726#Sx4.F6)a\)\. Our method leads for every model, 1\.16 to 1\.28×\\timesthe strongest baseline signal and up to 2\.57×\\timesthe weakest\. The throughput improvement comes from two factors\. First, the prompt pre\-scorer raises the ship rate\. Sorting the input buffer by the pre\-scorer score and dispatching the high\-scoring front first raises the served ship rate that multiplies the edge contribution\. Second, the verifier itself keeps the edge tier fast\. Unlike the baselines, our verifier runs on the CPU, in parallel with the small model’s decoding, so it adds almost zero overhead to the small model’s throughput and the full serving capacity goes to generating responses rather than to the routing decision\.

We further compare our edge\-cloud collaboration pipeline against the Ray Serve pipeline that expresses the same ship\-or\-escalate policy as a deployment graph in a serving framework\([Moritz et al\. 2018](https://arxiv.org/html/2608.28726#bib.bib35)\)\(Figure[6](https://arxiv.org/html/2608.28726#Sx4.F6)b\)\. Our pipeline is 1\.77 to 1\.79×\\timesfaster on every model\. The Ray Serve pipeline instead sends every request to the small model first and escalates afterward, which leaves the large model under\-utilized, makes the small model the bottleneck, and drives high concurrency on the small model that triggers frequent KV\-cache preemption\. By decoupling routing decisions from scheduling decisions and continuously adapting to each device’s real\-time service rate, our pipeline effectively turns the heterogeneous edge\-cloud cluster into a balanced ensemble where neither tier becomes the system’s bottleneck\.

#### Robustness and Scalability\.

We measure the throughput under network latency of 0, 200, and 1000 ms \(Figure[7](https://arxiv.org/html/2608.28726#Sx4.F7)a\)\. Because the global scheduler dispatches throughput\-sized batches and every device serves from its local buffer, added round\-trip latency does not stall the engines\. The pipeline keeps 90 to 95% of its un\-delayed throughput at 200 ms and 90 to 96% even at 1000 ms across the three models\.

Figure 7:Pipeline robustness and scalability\. \(a\) End\-to\-end throughput under one\-way network latency of 0, 200, and 1000 ms stays at 90% or more of the un\-delayed pipeline for all three models\. \(b\) End\-to\-end throughput in requests per second scales linearly in the number of edge devicesNNfor bothM=1M=1andM=3M=3cloud devices; the square markers atN=0N=0are the cloud\-only deployments\.Finally, we extend the deployment to more devices \(Figure[7](https://arxiv.org/html/2608.28726#Sx4.F7)b\)\. Adding small model devices against a single large model scales the pipeline’s throughput linearly inNNon every model, with no saturation knee, and adding large model devices scales the same way at all 18 measured points\. The largest deployment,M=3M=3andN=4N=4, reaches 5\.8 to 6\.9×\\timesthe throughput of a single cloud device\. These results confirm that our pipeline’s closed\-loop adaptation effectively decouples throughput from network latency and device count: by continuously aligning each device’s workload with its measured service rate, the system avoids the idling and contention that typically plague static or sequential deployments, enabling near\-linear scalability across both tiers without manual reconfiguration\.

#### Ablation Studies\.

We ablate three design dimensions: verifier input, verifier architecture, and pipeline optimizations in Table[2](https://arxiv.org/html/2608.28726#Sx4.T2)\. For the verifier, per\-token sampling distribution features consistently outperform hidden states \(35843584dims\) across all models, confirming that sampling distributions encode more direct confidence signals than raw representations\. Architecturally, our two\-layer transformer achieves the best accuracy\-throughput trade\-off, surpassing a quantile\-summary MLP by up to11\.411\.4AUROC points while matching or exceeding deeper variants with less overhead\. For pipeline optimizations, both are indispensable: removing batched dispatch collapses throughput from124124to2828req/s \(4\.4×4\.4\\timesdrop\), and disabling the KV\-cache admission gate inflates latency from754754to16341634ms due to frequent preemptions\. These results validate that token\-aware progressive routing and adaptive edge\-cloud collaboration pipeline jointly enable high routing accuracy and efficient edge\-cloud serving\.

VariantAUROCThrpt\.LatencyQwenLLaVA\-OVPixtral\(req/s\)\(ms\)Pro\-Router \(full\)0\.8050\.7530\.685124754Verifier inputfeaturessampling distribution feats \(ours\)0\.8050\.7530\.685124754hidden states \(3584 / token\)0\.7580\.6470\.67395932Verifierarchitecture2\-layer transformer \(ours\)0\.8050\.7530\.685124754quantile summary \+ MLP0\.6910\.7010\.6691187831 layer0\.7950\.7460\.6831307634 layers0\.7990\.7480\.677119736Pipelineoptimizationsfull pipeline \(ours\)0\.8050\.7530\.685124754w/o batched dispatch0\.8050\.7530\.685281038w/o KV\-cache gate0\.8050\.7530\.6851121634

Table 2:Ablation study\. AUROC per model; end\-to\-end throughput \(req/s\) and end\-to\-end serving latency \(ms\)\. Each block varies one design choice, and \(ours\) marks the configuration used in Pro\-Router\.

## Conclusion

In this paper, we proposed Pro\-Router, a token\-aware progressive routing method paired with an adaptive edge\-cloud pipeline\. First, the pre\-scorer raises the ship rate by guiding easy requests to the small models\. The token\-aware verifier then decides at almost zero overhead, and the adaptive pipeline keeps both tiers utilized without manual tuning\. Across 15 benchmarks and three models, our verifier is the most accurate signal and decides over 10×\\timesfaster, while our pipeline reaches more than 75% higher throughput than the existing model routing pipeline\.

## References

- Aggarwal et al\. \(2024\)Aggarwal, P\.; Madaan, A\.; Anand, A\.; Potharaju, S\. P\.; Mishra, S\.; Zhou, P\.; Gupta, A\.; Rajagopal, D\.; Kappaganthu, K\.; Yang, Y\.; Upadhyay, S\.; Faruqui, M\.; and Mausam\. 2024\.AutoMix: Automatically Mixing Language Models\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Agrawal et al\. \(2024\)Agrawal, P\.; Antoniak, S\.; Hanna, E\. B\.; et al\. 2024\.Pixtral 12B\.arXiv:2410\.07073\.
- Bai et al\. \(2025\)Bai, S\.; Chen, K\.; Liu, X\.; Wang, J\.; Ge, W\.; Song, S\.; et al\. 2025\.Qwen2\.5\-VL Technical Report\.arXiv:2502\.13923\.
- Bradley \(1997\)Bradley, A\. P\. 1997\.The Use of the Area Under the ROC Curve in the Evaluation of Machine Learning Algorithms\.*Pattern Recognition*, 30\(7\): 1145–1159\.
- Chen et al\. \(2023\)Chen, J\.; Yoon, J\.; Ebrahimi, S\.; Arik, S\. Ö\.; Pfister, T\.; and Jha, S\. 2023\.Adaptation with Self\-Evaluation to Improve Selective Prediction in LLMs\.In*Findings of Empirical Methods in Natural Language Processing \(EMNLP\)*\.
- Chen et al\. \(2024a\)Chen, L\.; Li, J\.; Dong, X\.; Zhang, P\.; Zang, Y\.; Chen, Z\.; et al\. 2024a\.Are We on the Right Way for Evaluating Large Vision\-Language Models?In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Chen, Zaharia, and Zou \(2024\)Chen, L\.; Zaharia, M\.; and Zou, J\. 2024\.FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance\.*Transactions on Machine Learning Research*\.
- Chen et al\. \(2024b\)Chen, L\.; Zhao, H\.; Liu, T\.; Bai, S\.; Lin, J\.; Zhou, C\.; and Chang, B\. 2024b\.An Image is Worth 1/2 Tokens After Layer 2: Plug\-and\-Play Inference Acceleration for Large Vision\-Language Models\.In*European Conference on Computer Vision \(ECCV\)*\.
- Chen et al\. \(2024c\)Chen, S\.; Jiang, W\.; Lin, B\.; Kwok, J\. T\.; and Zhang, Y\. 2024c\.RouterDC: Query\-Based Router by Dual Contrastive Learning for Assembling Large Language Models\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Chow \(1970\)Chow, C\. K\. 1970\.On Optimum Recognition Error and Reject Tradeoff\.*IEEE Transactions on Information Theory*, 16\(1\): 41–46\.
- Cobbe et al\. \(2021\)Cobbe, K\.; Kosaraju, V\.; Bavarian, M\.; Chen, M\.; Jun, H\.; Kaiser, L\.; et al\. 2021\.Training Verifiers to Solve Math Word Problems\.*arXiv preprint arXiv:2110\.14168*\.
- Cortes, DeSalvo, and Mohri \(2016\)Cortes, C\.; DeSalvo, G\.; and Mohri, M\. 2016\.Learning with Rejection\.In*International Conference on Algorithmic Learning Theory \(ALT\)*\.
- Dao et al\. \(2022\)Dao, T\.; Fu, D\. Y\.; Ermon, S\.; Rudra, A\.; and Ré, C\. 2022\.FlashAttention: Fast and Memory\-Efficient Exact Attention with IO\-Awareness\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Ding et al\. \(2024\)Ding, D\.; Mallick, A\.; Wang, C\.; Sim, R\.; Mukherjee, S\.; Rühle, V\.; Lakshmanan, L\. V\. S\.; and Awadallah, A\. H\. 2024\.Hybrid LLM: Cost\-Efficient and Quality\-Aware Query Routing\.In*International Conference on Learning Representations \(ICLR\)*\.
- El\-Yaniv and Wiener \(2010\)El\-Yaniv, R\.; and Wiener, Y\. 2010\.On the Foundations of Noise\-free Selective Classification\.*Journal of Machine Learning Research*, 11: 1605–1641\.
- Geifman and El\-Yaniv \(2017\)Geifman, Y\.; and El\-Yaniv, R\. 2017\.Selective Classification for Deep Neural Networks\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Guan et al\. \(2024\)Guan, T\.; Liu, F\.; Wu, X\.; Xian, R\.; Li, Z\.; Liu, X\.; et al\. 2024\.HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision\-Language Models\.In*IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*\.
- Gulrajani and Lopez\-Paz \(2021\)Gulrajani, I\.; and Lopez\-Paz, D\. 2021\.In Search of Lost Domain Generalization\.In*International Conference on Learning Representations \(ICLR\)*\.
- Hao et al\. \(2024\)Hao, Z\.; Jiang, H\.; Jiang, S\.; Ren, J\.; and Cao, T\. 2024\.Hybrid SLM and LLM for Edge\-Cloud Collaborative Inference\.In*Workshop on Edge and Mobile Foundation Models \(EdgeFM@MobiSys\)*\.
- Hendrycks et al\. \(2021\)Hendrycks, D\.; Burns, C\.; Basart, S\.; Zou, A\.; Mazeika, M\.; Song, D\.; and Steinhardt, J\. 2021\.Measuring Massive Multitask Language Understanding\.In*International Conference on Learning Representations \(ICLR\)*\.
- Hu et al\. \(2026a\)Hu, Y\.; Yang, Z\.; Zhao, C\.; Guo, Q\.; Gao, M\.; Li, P\.; and Ji, W\. 2026a\.AIVD: Adaptive Edge\-Cloud Collaboration for Accurate and Efficient Industrial Visual Detection\.*arXiv preprint arXiv:2601\.04734*\.
- Hu et al\. \(2026b\)Hu, Y\.; Yang, Z\.; Zhao, C\.; and Ji, W\. 2026b\.Adaptive guidance semantically enhanced via multimodal llm for edge\-cloud object detection\.In*ICASSP 2026\-2026 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\)*, 12962–12966\. IEEE\.
- Jin and Wu \(2025\)Jin, H\.; and Wu, Y\. 2025\.CE\-CoLLM: Efficient and Adaptive Large Language Models Through Cloud\-Edge Collaboration\.In*IEEE International Conference on Web Services \(ICWS\)*\.
- Joshi et al\. \(2017\)Joshi, M\.; Choi, E\.; Weld, D\. S\.; and Zettlemoyer, L\. 2017\.TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension\.In*Annual Meeting of the Association for Computational Linguistics \(ACL\)*\.
- Kadavath et al\. \(2022\)Kadavath, S\.; Conerly, T\.; Askell, A\.; Henighan, T\.; Drain, D\.; Perez, E\.; Schiefer, N\.; Hatfield\-Dodds, Z\.; DasSarma, N\.; Tran\-Johnson, E\.; et al\. 2022\.Language Models \(Mostly\) Know What They Know\.*arXiv preprint arXiv:2207\.05221*\.
- Kwon et al\. \(2023\)Kwon, W\.; Li, Z\.; Zhuang, S\.; Sheng, Y\.; Zheng, L\.; Yu, C\. H\.; Gonzalez, J\. E\.; Zhang, H\.; and Stoica, I\. 2023\.Efficient Memory Management for Large Language Model Serving with PagedAttention\.In*ACM Symposium on Operating Systems Principles \(SOSP\)*\.
- Li et al\. \(2025\)Li, B\.; Zhang, Y\.; Guo, D\.; Zhang, R\.; Li, F\.; Zhang, H\.; et al\. 2025\.LLaVA\-OneVision: Easy Visual Task Transfer\.*Transactions on Machine Learning Research*\.
- Lin, Trivedi, and Sun \(2024\)Lin, Z\.; Trivedi, S\.; and Sun, J\. 2024\.Generating with Confidence: Uncertainty Quantification for Black\-box Large Language Models\.*Transactions on Machine Learning Research*\.
- Liu et al\. \(2026\)Liu, X\.; He, B\.; Liu, X\.; Luo, A\.; Zhang, H\.; and Chen, H\. 2026\.Adaptive Vision\-Language Model Routing for Computer Use Agents\.arXiv:2603\.12823\.
- Liu et al\. \(2024a\)Liu, Y\.; Duan, H\.; Zhang, Y\.; Li, B\.; Zhang, S\.; Zhao, W\.; et al\. 2024a\.MMBench: Is Your Multi\-modal Model an All\-around Player?In*European Conference on Computer Vision \(ECCV\)*\.
- Liu et al\. \(2024b\)Liu, Y\.; Li, Z\.; Huang, M\.; Yang, B\.; Yu, W\.; Li, C\.; et al\. 2024b\.OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models\.*Science China Information Sciences \(SCIS\)*, 67\(12\)\.
- Lu et al\. \(2024\)Lu, P\.; Bansal, H\.; Xia, T\.; Liu, J\.; Li, C\.; Hajishirzi, H\.; et al\. 2024\.MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts\.In*International Conference on Learning Representations \(ICLR\)*\.
- Masry et al\. \(2022\)Masry, A\.; Long, D\. X\.; Tan, J\. Q\.; Joty, S\.; and Hoque, E\. 2022\.ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning\.In*Findings of the Association for Computational Linguistics \(ACL\)*\.
- Mathew, Karatzas, and Jawahar \(2021\)Mathew, M\.; Karatzas, D\.; and Jawahar, C\. V\. 2021\.DocVQA: A Dataset for VQA on Document Images\.In*IEEE/CVF Winter Conference on Applications of Computer Vision \(WACV\)*\.
- Moritz et al\. \(2018\)Moritz, P\.; Nishihara, R\.; Wang, S\.; Tumanov, A\.; Liaw, R\.; Liang, E\.; Elibol, M\.; Yang, Z\.; Paul, W\.; Jordan, M\. I\.; and Stoica, I\. 2018\.Ray: A Distributed Framework for Emerging AI Applications\.In*USENIX Symposium on Operating Systems Design and Implementation \(OSDI\)*\.
- Mozannar and Sontag \(2020\)Mozannar, H\.; and Sontag, D\. 2020\.Consistent Estimators for Learning to Defer to an Expert\.In*International Conference on Machine Learning \(ICML\)*\.
- Nadeem, Zucker, and Hanczar \(2009\)Nadeem, M\. S\. A\.; Zucker, J\.\-D\.; and Hanczar, B\. 2009\.Accuracy\-Rejection Curves \(ARCs\) for Comparing Classification Methods with a Reject Option\.In*Machine Learning in Systems Biology \(JMLR Workshop and Conference Proceedings\)*\.
- Ong et al\. \(2025\)Ong, I\.; Almahairi, A\.; Wu, V\.; Chiang, W\.\-L\.; Wu, T\.; Gonzalez, J\. E\.; Kadous, M\. W\.; and Stoica, I\. 2025\.RouteLLM: Learning to Route LLMs from Preference Data\.In*International Conference on Learning Representations \(ICLR\)*\.
- Reddy, Chen, and Manning \(2019\)Reddy, S\.; Chen, D\.; and Manning, C\. D\. 2019\.CoQA: A Conversational Question Answering Challenge\.*Transactions of the Association for Computational Linguistics \(TACL\)*, 7: 249–266\.
- Salton and Buckley \(1988\)Salton, G\.; and Buckley, C\. 1988\.Term\-weighting Approaches in Automatic Text Retrieval\.*Information Processing & Management*, 24\(5\): 513–523\.
- Sanh et al\. \(2019\)Sanh, V\.; Debut, L\.; Chaumond, J\.; and Wolf, T\. 2019\.DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter\.*arXiv preprint arXiv:1910\.01108*\.
- Schwenk et al\. \(2022\)Schwenk, D\.; Khandelwal, A\.; Clark, C\.; Marino, K\.; and Mottaghi, R\. 2022\.A\-OKVQA: A Benchmark for Visual Question Answering Using World Knowledge\.In*European Conference on Computer Vision \(ECCV\)*\.
- Song et al\. \(2024\)Song, D\.; Chen, S\.; Chen, G\. H\.; Yu, F\.; Wan, X\.; and Wang, B\. 2024\.MileBench: Benchmarking MLLMs in Long Context\.In*Conference on Language Modeling \(COLM\)*\.
- Tang et al\. \(2025\)Tang, X\.; Han, Y\.; Gou, F\.; Zhao, W\.; Meng, X\.; Yu, Y\.; Zhang, J\.; Shi, Y\.; Wang, Y\.; and Zhang, T\. 2025\.ECVL\-ROUTER: Scenario\-Aware Routing for Vision\-Language Models\.arXiv:2510\.27256\.
- Tian and Yang \(2026\)Tian, Y\.; and Yang, Z\. 2026\.SAEC: scene\-aware enhanced edge\-cloud collaborative industrial vision inspection with multimodal LLM\.In*ICASSP 2026\-2026 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\)*, 19747–19751\. IEEE\.
- Vaswani et al\. \(2017\)Vaswani, A\.; Shazeer, N\.; Parmar, N\.; Uszkoreit, J\.; Jones, L\.; Gomez, A\. N\.; Kaiser, Ł\.; and Polosukhin, I\. 2017\.Attention Is All You Need\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Wang et al\. \(2025\)Wang, F\.; Fu, X\.; Huang, J\. Y\.; Li, Z\.; Liu, Q\.; Liu, X\.; et al\. 2025\.MuirBench: A Comprehensive Benchmark for Robust Multi\-image Understanding\.In*International Conference on Learning Representations \(ICLR\)*\.
- Wang et al\. \(2024a\)Wang, G\.; Liu, J\.; Li, C\.; Zhang, Y\.; Ma, J\.; et al\. 2024a\.Cloud\-Device Collaborative Learning for Multimodal Large Language Models\.In*IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*\.
- Wang et al\. \(2024b\)Wang, K\.; Pan, J\.; Shi, W\.; Lu, Z\.; Zhan, M\.; and Li, H\. 2024b\.Measuring Multimodal Mathematical Reasoning with MATH\-Vision Dataset\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- xAI \(2024\)xAI\. 2024\.RealWorldQA: A Benchmark for Real\-World Spatial Understanding\.https://x\.ai/news/grok\-1\.5v\.
- Yang et al\. \(2025\)Yang, Z\.; Guo, Q\.; Hu, Y\.; Zhao, C\.; Zhang, C\.; Zhao, J\.; and Ji, W\. 2025\.Moa\-off: Adaptive heterogeneous modality\-aware offloading with edge\-cloud collaboration for efficient multimodal llm inference\.*arXiv preprint arXiv:2509\.16995*\.
- Yang et al\. \(2024\)Yang, Z\.; Yang, Y\.; Zhao, C\.; Guo, Q\.; He, W\.; and Ji, W\. 2024\.Perllm: Personalized inference scheduling with edge\-cloud collaboration for diverse llm services\.*arXiv preprint arXiv:2405\.14636*\.
- Yu et al\. \(2024\)Yu, W\.; Yang, Z\.; Li, L\.; Wang, J\.; Lin, K\.; Liu, Z\.; Wang, X\.; and Wang, L\. 2024\.MM\-Vet: Evaluating Large Multimodal Models for Integrated Capabilities\.In*International Conference on Machine Learning \(ICML\)*\.
- Yue et al\. \(2024\)Yue, X\.; Ni, Y\.; Zhang, K\.; Zheng, T\.; Liu, R\.; Zhang, G\.; et al\. 2024\.MMMU: A Massive Multi\-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI\.In*IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*\.
- Zhang et al\. \(2025\)Zhang, M\.; Shen, X\.; Cao, J\.; Cui, Z\.; and Jiang, S\. 2025\.EdgeShard: Efficient LLM Inference via Collaborative Edge Computing\.*IEEE Internet of Things Journal*, 12\(10\): 13119–13131\.
- Zhang et al\. \(2024\)Zhang, R\.; Jiang, D\.; Zhang, Y\.; Lin, H\.; Guo, Z\.; Qiu, P\.; et al\. 2024\.MathVerse: Does Your Multi\-modal LLM Truly See the Diagrams in Visual Math Problems?In*European Conference on Computer Vision \(ECCV\)*\.
- Zheng et al\. \(2023\)Zheng, L\.; Chiang, W\.\-L\.; Sheng, Y\.; Zhuang, S\.; Wu, Z\.; Zhuang, Y\.; Lin, Z\.; Li, Z\.; Li, D\.; Xing, E\. P\.; Zhang, H\.; Gonzalez, J\. E\.; and Stoica, I\. 2023\.Judging LLM\-as\-a\-Judge with MT\-Bench and Chatbot Arena\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.