DLLG: Dynamic Logit-Level Gating of LLM Experts

# DLLG: Dynamic Logit-Level Gating of LLM Experts
Source: [https://arxiv.org/html/2606.04378](https://arxiv.org/html/2606.04378)
Zhaoyang ZhangXiaoze LiuYantao ShenShuli JiangShuo YangWei XiaZhuowen TuStefano Soatto

###### Abstract

Leveraging multiple specialized LLMs can combine complementary strengths, but existing approaches trade adaptability for stability: routing commits prematurely, heuristic ensembling depends on fragile proxies, and parameter merging introduces interference\. We propose DLLG \(Dynamic Logit\-Level Gating\), a dynamic logit\-level ensembling framework that learns token\-level expert fusion from sparse response\-level supervision\. A lightweight gating module predicts step\-wise fusion weights, linking trajectory\-level correctness to generation without token\-level labels or expert retraining\. Across diverse reasoning and code benchmarks, DLLG consistently outperforms strong routing, heuristic ensembling, and parameter\-merging baselines across model scales, highlighting learned logit\-level fusion as a robust and scalable paradigm for integrating specialized experts\.

Machine Learning, ICML

![[Uncaptioned image]](https://arxiv.org/html/2606.04378v1/x1.png)

Figure 1:Comparison of expert combination strategies\. Routing relies on hard, early expert selection and fails when subtasks vary within a response\. Heuristic ensembling uses inference\-time proxy signals that are often misaligned with task correctness\. Parameter merging statically fuses expert weights, sacrificing modularity and inducing interference\. In contrast, DLLG \(ours\) performs dynamic, token\-level logit fusion using learned gating conditioned on expert hidden states, enabling fine\-grained and recoverable expert utilization while preserving expert modularity\.
## 1Introduction

The landscape of Large Language Models \(LLMs\) has witnessed a proliferation of*specialized models*, each optimized for narrow domains such as mathematical reasoning or code generation\(Huiet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib39); Yanget al\.,[2024a](https://arxiv.org/html/2606.04378#bib.bib38)\)\. While such specialists can outperform general\-purpose models\(Achiamet al\.,[2023](https://arxiv.org/html/2606.04378#bib.bib36); Touvronet al\.,[2023](https://arxiv.org/html/2606.04378#bib.bib37)\)within their niches, relying on any single expert limits coverage across diverse tasks\. This motivates a central question:*how can we integrate independently trained specialists into a unified system that exploits their complementary strengths*\(Chenet al\.,[2025](https://arxiv.org/html/2606.04378#bib.bib23); Yanget al\.,[2024b](https://arxiv.org/html/2606.04378#bib.bib27)\)?

Prior work offers multiple directions, but existing approaches expose a persistent tension among*adaptivity*,*robustness*, and*practicality*\.Routing\-based methodsselect one expert per input\(Šakotaet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib20); Zhuanget al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib16); Zhanget al\.,[2025](https://arxiv.org/html/2606.04378#bib.bib17); Wanget al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib22); Dinget al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib18); Nguyenet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib19); Panet al\.,[2025](https://arxiv.org/html/2606.04378#bib.bib21); Onget al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib24)\), yet coarse early commitments are difficult to revise when expertise shifts within a single response; moreover, many require router training or calibration on benchmark data to ensure expert diversity\(Wanget al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib22); Zhanget al\.,[2025](https://arxiv.org/html/2606.04378#bib.bib17); Chenet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib15); Onget al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib24)\), an assumption rarely met in deployment\.Token\-level ensemblingenables finer\-grained combination\(Yuet al\.,[2024b](https://arxiv.org/html/2606.04378#bib.bib9); Huanget al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib12); Xuet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib13); Mavromatiset al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib11); Yaoet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib10); Wickset al\.,[2025](https://arxiv.org/html/2606.04378#bib.bib14)\), from uniform averaging\(Yuet al\.,[2024b](https://arxiv.org/html/2606.04378#bib.bib9)\)to heuristics such as perplexity weighting\(Mavromatiset al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib11)\)or top\-kktoken unions\(Yaoet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib10)\), but these schemes remain largely empirical and rely on proxy signals that may not align with specialization or correctness\.Parameter\-space merging\(e\.g\., model souping\(Wortsmanet al\.,[2022](https://arxiv.org/html/2606.04378#bib.bib25)\)and task arithmetic\(Ilharcoet al\.,[2022](https://arxiv.org/html/2606.04378#bib.bib26)\)\) yields a single model but sacrifices flexibility and often suffers destructive interference when combining disparate experts, making hyperparameter choices brittle\(Yanget al\.,[2024b](https://arxiv.org/html/2606.04378#bib.bib27)\)\. Finally, whileMixture\-of\-Experts\(MoE\) architectures scale capacity via sparse gating\(Chenet al\.,[2023](https://arxiv.org/html/2606.04378#bib.bib44); Shazeeret al\.,[2017](https://arxiv.org/html/2606.04378#bib.bib42); Daiet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib43); Jianget al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib45)\), they typically require end\-to\-end joint training of the gate and experts, limiting their ability to leverage independently pre\-trained, off\-the\-shelf specialists\. Consequently, a key hurdle remains: effectively unifying these “black\-box” experts into a single system without the prohibitive cost of expert retraining\.

In this work, we introduceDLLG\(DynamicLogit\-LevelGating\), a dynamic logit\-level ensembling framework that learns fine\-grained expert utilization from sparse supervision\. To avoid the premature commitment of routing, DLLG performs autoregressive soft fusion: at each decoding step, a lightweight gate conditions on the prompt, the partial prefix, and trajectory\-level hidden states from all experts to produce step\-specific mixture weights for logit aggregation\. To replace brittle token\-level heuristics, DLLG learns the fusion rule via supervision\. We train the gate with teacher forcing using response\-level correctness labels from automatic verifiers \(computed independently of the gate\) and broadcast them to all tokens of the reference response\. DLLG preserves expert modularity by freezing all expert parameters, avoiding interference and eliminating the need for test\-time supervision or online rollouts\.

Unlike MoE, which typically requires joint training of experts and gating, DLLG treats specialists as plug\-and\-play frozen components and learns logit\-level fusion from sparse supervision, achieving MoE\-like adaptivity without expert retraining\. Our contributions are summarized as follows:

1. 1\.We propose DLLG, a dynamic logit\-level ensembling framework that bridges token\-level generation and sparse response\-level supervision\. Using teacher\-forced correctness labels, DLLG learns fine\-grained expert utilization without token\-level annotation or online RL\.
2. 2\.We analyze the learned token\-level fusion weights and show that the proposed gating mechanism dynamically adapts expert contribution during rollout, adjusting the contribution of different models in accordance with their specialization as the generation context evolves\.
3. 3\.We evaluate DLLG on diverse reasoning and code benchmarks \(GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2606.04378#bib.bib1)\), Minerva Math\(Lewkowyczet al\.,[2022](https://arxiv.org/html/2606.04378#bib.bib2)\), MATH\(Hendryckset al\.,[2021](https://arxiv.org/html/2606.04378#bib.bib3)\), Code\-R1\(Liu and Zhang,[2025](https://arxiv.org/html/2606.04378#bib.bib4)\), HumanEval\(Chen,[2021](https://arxiv.org/html/2606.04378#bib.bib5)\), MBPP\(Austinet al\.,[2021](https://arxiv.org/html/2606.04378#bib.bib6)\), BBH\(Suzgunet al\.,[2023](https://arxiv.org/html/2606.04378#bib.bib7)\), BigCodeBench\(Zhuoet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib8)\)\), showing consistent gains over routing, heuristic ensembling, and parameter\-merging baselines across model scales\.

## 2Related Work

We categorize existing strategies for combining specialized LLMs into four main paradigms: token\-level aggregation, routing\-based selection, parameter\-space merging, and MoE\.

![Refer to caption](https://arxiv.org/html/2606.04378v1/x2.png)Figure 2:Overview of DLLG, illustrating training, inference, and the gating model architecture in a unified pipeline\.\(a\) Training:Frozen specialized LLMs are conditioned on ground\-truth prefixes under teacher forcing to produce hidden states, which are fed into a lightweight gating model\. Response\-level correctness signals, obtained from automatic verifiers applied to expert rollouts, supervise the gating model via an MSE objective\.\(b\) Inference:At each decoding step, the gating model predicts token\-level fusion weights from expert hidden states, and expert logits are softly combined through logit\-level fusion for autoregressive generation\.\(c\) Gating model:Hidden states from all experts are concatenated, projected into a shared embedding space, and processed by a gating stem with LoRA adapters and KV caching\. A weight prediction head outputs token\-wise fusion weights, while all expert models remain frozen\.![Refer to caption](https://arxiv.org/html/2606.04378v1/x3.png)Figure 3:Token\-level fusion behavior on a representative Code\-R1 example, where the gating model dynamically adjusts expert fusion weights, with math\-specialized experts dominating early reasoning stages and code\-specialized experts becoming more prominent during code generation\.##### Token\-level aggregation

combines model outputs at the finest granularity and typically falls into two main categories:*static*or*heuristic*\.*Static approaches*, such as uniform averaging\(Yuet al\.,[2024b](https://arxiv.org/html/2606.04378#bib.bib9)\), benefit from stability but lack the flexibility to account for varying expert specializations\.*Heuristic approaches*attempt to weight models dynamically using inference\-time proxies, such as perplexity\(Mavromatiset al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib11)\), confidence scores\(Yuet al\.,[2024b](https://arxiv.org/html/2606.04378#bib.bib9)\), or top\-k token agreement\(Yaoet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib10)\)\. However, these metrics are empirical proxies that often fail to align with true task correctness\(Nunezet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib34)\)\. Additionally, while some works focus onvocabulary alignmentfor heterogeneous models\(Huanget al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib12); Xuet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib13); Wickset al\.,[2025](https://arxiv.org/html/2606.04378#bib.bib14)\), they generally rely on simple fusion rules after projection\. In contrast to heuristic or static ensembling approaches, our method learns token\-level fusion weights directly from response\-level supervision, providing a principled alternative that adapts expert contributions at a finer granularity while retaining the robustness advantages of logit\-level ensembling\.

##### Routing\-based Expert Selection

aims to assign inputs to the most suitable model, typically operating at the coarse granularity of the entire prompt or response\(Chenet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib15); Zhuanget al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib16); Zhanget al\.,[2025](https://arxiv.org/html/2606.04378#bib.bib17); Wanget al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib22); Dinget al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib18); Nguyenet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib19); Šakotaet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib20); Panet al\.,[2025](https://arxiv.org/html/2606.04378#bib.bib21)\)\. Existing methods primarily rely on*feature\-based predictors*trained on prompt embeddings\(Chenet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib15); Zhuanget al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib16); Zhanget al\.,[2025](https://arxiv.org/html/2606.04378#bib.bib17)\)or*statistical priors*derived from benchmark performance or preference signals\(Wanget al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib22); Onget al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib24)\)\. While some approaches focus on optimizing the performance\-cost trade\-off\(Dinget al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib18); Nguyenet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib19)\), they generally suffer from two limitations\. First, routing involves*hard, premature commitment*: selecting a single expert early in generation leads to unrecoverable errors if the chosen model fails on a specific sub\-task\. Second, many routers require extensive*benchmark\-specific calibration*or test\-set statistics to generalize\(Wanget al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib22); Zhanget al\.,[2025](https://arxiv.org/html/2606.04378#bib.bib17)\), an assumption that rarely holds in realistic deployment\. In contrast, DLLG avoids hard selection entirely, utilizing soft, autoregressive fusion to adapt dynamically without relying on test\-time ground truth\.

##### Parameter\-Space Merging

constructs a single model by fusing weights from multiple experts, typically assuming a shared architecture and initialization\. Common techniques range from*averaging\-based methods*like model souping and SLERP\(Wortsmanet al\.,[2022](https://arxiv.org/html/2606.04378#bib.bib25); Shoemake,[1985](https://arxiv.org/html/2606.04378#bib.bib31); Grove and Karcher,[1973](https://arxiv.org/html/2606.04378#bib.bib32)\)to*arithmetic operations*on task vectors\(Ilharcoet al\.,[2022](https://arxiv.org/html/2606.04378#bib.bib26); Yadavet al\.,[2023](https://arxiv.org/html/2606.04378#bib.bib28); Yuet al\.,[2024a](https://arxiv.org/html/2606.04378#bib.bib29); Deepet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib30)\)\. While computationally efficient at inference time, these approaches suffer from inherent*static interference*: merging parameters optimized for distinct objectives often leads to performance degradation due to conflicting gradient directions\(Liet al\.,[2023](https://arxiv.org/html/2606.04378#bib.bib33); Yanget al\.,[2024b](https://arxiv.org/html/2606.04378#bib.bib27)\)\. Furthermore, the merged model is fixed after composition, sacrificing*modularity*and the ability to dynamically leverage specific expert strengths during generation\. By focusing on inference\-time logit fusion, DLLG bypasses parameter interference entirely, preserving the specialized capabilities of frozen experts\.

##### Mixture\-of\-Experts \(MoE\)

is a foundational technique for increasing model capacity by activating only a subset of parameters per input through a router\(Chenet al\.,[2023](https://arxiv.org/html/2606.04378#bib.bib44); Shazeeret al\.,[2017](https://arxiv.org/html/2606.04378#bib.bib42); Daiet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib43); Jianget al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib45)\)\. While effective, standard MoE models rely on the simultaneous optimization of experts and routers to ensure balanced specialization\(Caiet al\.,[2025](https://arxiv.org/html/2606.04378#bib.bib46)\)\. In contrast, DLLG operates in a post\-hoc ensembling regime where the experts are already fully specialized for distinct domains, such as math or code\. By shifting the fusion to the logit level during inference, DLLG inherits the dynamic flexibility of MoE’s gating logic but bypasses the need for expensive end\-to\-end training rollouts\. This distinguishes DLLG from existing routing methods that suffer from premature commitment, as our soft fusion allows for recoverable expert contributions at every decoding step\.

## 3Method

In this section, we present DLLG \(Dynamic Logit\-Level Gating\), a dynamic logit\-level ensembling framework for combining multiple specialized language models\. DLLG learns to assign token\-level fusion weights under sparse response\-level supervision, enabling fine\-grained and adaptive expert contribution without requiring token\-level annotations, hard expert selection, or expert retraining\. The key idea is to train a lightweight gating model that predicts token\-wise weights from expert trajectory representations, and to combine expert outputs through soft logit\-level fusion at inference time\. An overview of the training and inference pipeline of DLLG is shown in[Figure2](https://arxiv.org/html/2606.04378#S2.F2)\.

We first formalize the problem setting and the ensembling objective in[Section3\.1](https://arxiv.org/html/2606.04378#S3.SS1)\.[Section3\.2](https://arxiv.org/html/2606.04378#S3.SS2)introduces our logit\-level fusion formulation, followed by the architecture of the gating model in[Section3\.3](https://arxiv.org/html/2606.04378#S3.SS3)\.[Section3\.4](https://arxiv.org/html/2606.04378#S3.SS4)describes how the gating model is trained using response\-level correctness signals under a teacher\-forcing regime\. Finally,[Section3\.5](https://arxiv.org/html/2606.04378#S3.SS5)details the inference procedure, where experts are combined autoregressively using the learned token\-level fusion weights\.

### 3\.1Problem Formulation

We consider a setting withKKspecialized large language models \(experts\)\{ℰi\}i=1K\\\{\\mathcal\{E\}\_\{i\}\\\}\_\{i=1\}^\{K\}from the same model family, sharing a common tokenizer and vocabulary\. Letxxdenote an input prompt, and lety=\(y1,…,yT\)y=\(y\_\{1\},\\ldots,y\_\{T\}\)denote the corresponding output token sequence of lengthTT, where eachyt∈𝒱y\_\{t\}\\in\\mathcal\{V\}and𝒱\\mathcal\{V\}is the shared vocabulary across all experts\. We usey<t=\(y1,…,yt−1\)y\_\{<t\}=\(y\_\{1\},\\ldots,y\_\{t\-1\}\)to denote the prefix up to stept−1t\-1\.

Given the input promptxxand prefixy<ty\_\{<t\}, each expertℰi\\mathcal\{E\}\_\{i\}produces a hidden state𝐡t\(i\)\\mathbf\{h\}\_\{t\}^\{\(i\)\}and a next\-token logit vectorℓt\(i\)∈ℝ\|𝒱\|\\boldsymbol\{\\ell\}\_\{t\}^\{\(i\)\}\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\}autoregressively\. Our goal is to combine the outputs of these experts to generate a single output sequence that leverages their complementary strengths\.

We assume access to response\-level supervision for each expert, indicating whether the expert produces a correct response for a given input under an automatic verifier or a task\-specific evaluation criterion\. This supervision is defined at the response level and does not provide token\-level annotations\. The central challenge is therefore to learn fine\-grained, token\-level expert utilization under such sparse supervision\.

### 3\.2Logit\-Level Ensembling Framework

At decoding steptt, each expertℰi\\mathcal\{E\}\_\{i\}produces a next\-token logit vectorℓt\(i\)∈ℝ\|𝒱\|\\boldsymbol\{\\ell\}\_\{t\}^\{\(i\)\}\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\}and a corresponding hidden state𝐡t\(i\)\\mathbf\{h\}\_\{t\}^\{\(i\)\}, conditioned on the input promptxxand the prefixy<ty\_\{<t\}\. We combine expert outputs at the logit level by computing a weighted sum:

ℓt=∑i=1Kwt,i​ℓt\(i\),\\boldsymbol\{\\ell\}\_\{t\}=\\sum\_\{i=1\}^\{K\}w\_\{t,i\}\\,\\boldsymbol\{\\ell\}\_\{t\}^\{\(i\)\},\(1\)
wherewt,iw\_\{t,i\}denotes the token\-level fusion weight assigned to expertiiat steptt\. The combined logitsℓt\\boldsymbol\{\\ell\}\_\{t\}are then used for next\-token prediction\.

We do not require the fusion weights to sum to one\. Fusion weights are produced independently for each expert \(e\.g\., via a sigmoid transformation\) and may optionally be normalized without affecting the formulation\. This design enables soft and flexible expert contributions, in contrast to hard expert selection\.

### 3\.3Gating Model Architecture

The core of DLLG is a lightweight gating model that predicts token\-level fusion weights from expert trajectory representations\. An overview of the gating architecture is shown in[Figure2](https://arxiv.org/html/2606.04378#S2.F2)\(c\)\. At each decoding steptt, the gating model processes hidden states produced by all experts and outputs step\-specific fusion weights used for logit\-level aggregation\.

##### Expert representation aggregation\.

At decoding steptt, each expertℰi\\mathcal\{E\}\_\{i\}produces a hidden state𝐡t\(i\)∈ℝd\\mathbf\{h\}\_\{t\}^\{\(i\)\}\\in\\mathbb\{R\}^\{d\}\. We concatenate expert hidden states along the feature dimension and project them into a shared embedding space:

𝐡tcat=Proj​\(Concat​\(\{𝐡t\(i\)\}i=1K\)\),\\mathbf\{h\}\_\{t\}^\{\\text\{cat\}\}=\\text\{Proj\}\\\!\\left\(\\text\{Concat\}\\left\(\\\{\\mathbf\{h\}\_\{t\}^\{\(i\)\}\\\}\_\{i=1\}^\{K\}\\right\)\\right\),\(2\)whereProj​\(⋅\)\\text\{Proj\}\(\\cdot\)denotes a learnable projection module\. In practice, this projection is implemented as a low\-rank projection to reduce parameter count while preserving cross\-expert information\.

##### Gating backbone\.

The projected representation is processed by a gating stem model:

𝐡tgate=Mstem​\(𝐡tcat\),\\mathbf\{h\}\_\{t\}^\{\\text\{gate\}\}=M\_\{\\text\{stem\}\}\\\!\\left\(\\mathbf\{h\}\_\{t\}^\{\\text\{cat\}\}\\right\),\(3\)which models trajectory\-level context using an autoregressive architecture with cached key–value states\. We instantiateMstemM\_\{\\text\{stem\}\}usingQwen2\.5\-0\.5B\-Instructand fine\-tune it via LoRA adapters while keeping the backbone parameters frozen\. We additionally provide an ablation study on different gating backbone architectures in[Table3](https://arxiv.org/html/2606.04378#S4.T3)\.

##### Prediction of fusion weights\.

A lightweight prediction head maps the gating representation to unnormalized fusion scores, which are transformed into non\-negative fusion weights via a sigmoid function:

\{wt,i\}i=1K=σ​\(Head​\(𝐡tgate\)\),\\\{w\_\{t,i\}\\\}\_\{i=1\}^\{K\}=\\sigma\\\!\\left\(\\text\{Head\}\\\!\\left\(\\mathbf\{h\}\_\{t\}^\{\\text\{gate\}\}\\right\)\\right\),\(4\)whereσ​\(⋅\)\\sigma\(\\cdot\)is applied element\-wise\. The resulting fusion weights are used directly for logit\-level ensembling without enforcing a simplex constraint\.

In\-DomainOut\-of\-DomainMethodGSM8KMinervaMathMATHCode\-R1HumanEvalMBPPBBHAvgExpertsQwen2\.5\-0\.5B\-Instruct49\.2224\.1146\.007\.0232\.3233\.4027\.78\\cellcolorgray\!1831\.41Qwen2\.5\-Coder\-0\.5B\-Instruct31\.2511\.1622\.007\.0254\.8835\.8027\.20\\cellcolorgray\!1827\.04Dolphin3\.0\-Qwen2\.5\-0\.5B42\.9713\.8436\.407\.4442\.0728\.0026\.16\\cellcolorgray\!1828\.13Model MergingLinear\(Wortsmanet al\.,[2022](https://arxiv.org/html/2606.04378#bib.bib25)\)46\.0918\.3042\.208\.7135\.9833\.2029\.39\\cellcolorgray\!1830\.55SLERP48\.4416\.5241\.007\.5841\.4631\.6027\.43\\cellcolorgray\!1830\.58Task Arithmetic\(Ilharcoet al\.,[2022](https://arxiv.org/html/2606.04378#bib.bib26)\)26\.565\.8014\.205\.4843\.9033\.0029\.51\\cellcolorgray\!1822\.64RoutingRouterDC\(Chenet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib15)\)48\.4415\.1842\.207\.7241\.4628\.4026\.16\\cellcolorgray\!1829\.94EmbedLLM\(Zhuanget al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib16)\)47\.6619\.2044\.608\.7132\.3133\.8027\.78\\cellcolorgray\!1830\.58Logits EnsembleGaC\(Yuet al\.,[2024b](https://arxiv.org/html/2606.04378#bib.bib9)\)50\.7816\.0742\.408\.8545\.1235\.4030\.79\\cellcolorgray\!1832\.77Entropy Weighting41\.4115\.1841\.809\.1345\.1236\.0027\.66\\cellcolorgray\!1830\.90Pack of LLMs\(Mavromatiset al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib11)\)44\.5315\.1842\.208\.9945\.7335\.2030\.09\\cellcolorgray\!1831\.70Token Maj\-Voting45\.3116\.0743\.006\.6043\.3033\.2027\.20\\cellcolorgray\!1830\.67UniTe\(Yaoet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib10)\)50\.7815\.6342\.609\.2745\.7335\.4030\.90\\cellcolorgray\!1832\.90Ours52\.3419\.6443\.409\.5545\.7335\.0030\.56\\cellcolorgray\!1833\.75

Table 1:Performance comparison across in\-domain and out\-of\-domain benchmarks at the 0\.5B scale \(0\-shot\)\. Avg is the unweighted mean across all benchmarks\.Boldandunderlinedenote the best and second\-best results except experts, respectively\.

### 3\.4Training with Response\-Level Supervision

Training the gating model is challenging due to the absence of token\-level supervision\. We address this by leveraging response\-level correctness signals as sparse supervision\.

For each training example and expertℰi\\mathcal\{E\}\_\{i\}, we obtain a binary correctness labelsi∈\{0,1\}s\_\{i\}\\in\\\{0,1\\\}, indicating whether the expert’s generated response is correct according to an automatic verifier or a task\-specific evaluation criterion\. These response\-level labels are computed independently of the gating model and are broadcast to all token positions of the corresponding ground\-truth response during training\.

We train the gating model in a teacher\-forcing regime, where expert hidden states𝐡t\(i\)\\mathbf\{h\}\_\{t\}^\{\(i\)\}are computed conditioned on the ground\-truth prefixy<ty\_\{<t\}\. At each token positiontt, the gating model predicts fusion weightswt,iw\_\{t,i\}, which are supervised using the response\-level signalsis\_\{i\}\. We minimize the following mean squared error objective:

ℒ=1T​∑t=1T∑i=1K\(wt,i−si\)2,\\mathcal\{L\}=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\sum\_\{i=1\}^\{K\}\\big\(w\_\{t,i\}\-s\_\{i\}\\big\)^\{2\},\(5\)whereTTdenotes the length of the target output sequence\. The loss is averaged over all token positions and experts and it encourages the predicted token\-level weights to align with the response\-level correctness signal while allowing variation across tokens\.

Teacher forcing decouples the learning of token\-level fusion weights from the stochasticity of autoregressive generation, providing a low\-variance and semantically aligned training signal under sparse supervision\.

### 3\.5Inference Procedure

At inference time, DLLG operates autoregressively\. Given the current prefixy<ty\_\{<t\}, all experts produce next\-token logits and hidden states\. The gating model predicts step\-specific fusion weights based on the expert representations, and the final logits are obtained via the logit\-level combination in[Section3\.2](https://arxiv.org/html/2606.04378#S3.SS2)\. The next token is sampled or selected from the combined logits, and the process repeats until termination\.

This inference procedure enables soft and recoverable expert utilization at each decoding step, avoiding premature commitment to a single expert\. Since expert parameters remain unchanged, the method incurs minimal additional overhead and can be applied to arbitrary sets of specialized experts within a shared\-tokenizer setting\. When sufficient parallel GPU resources are available, the additional wall\-clock latency over single\-expert decoding can be limited, as experts are executed concurrently\.

## 4Experiments

In\-DomainOut\-of\-DomainMethodGSM8KMinervaMathMATHCode\-R1HumanEvalMBPPBBHBigCodeBenchAvgExpertsQwen2\.5\-1\.5B\-Instruct67\.9730\.3669\.4015\.0353\.6649\.8038\.193\.40\\cellcolorgray\!1840\.98Qwen2\.5\-Math\-1\.5B\-Instruct82\.8152\.6885\.604\.6337\.2030\.4026\.620\.00\\cellcolorgray\!1839\.99Qwen2\.5\-Coder\-1\.5B\-Instruct62\.5024\.5557\.6015\.3164\.6353\.6034\.265\.40\\cellcolorgray\!1839\.73Model MergingLinear\(Wortsmanet al\.,[2022](https://arxiv.org/html/2606.04378#bib.bib25)\)67\.9725\.4564\.8013\.7650\.6145\.8035\.304\.70\\cellcolorgray\!1838\.55SLERP50\.7822\.3254\.4012\.6456\.1050\.8034\.614\.10\\cellcolorgray\!1835\.72Task Arithmetic\(Ilharcoet al\.,[2022](https://arxiv.org/html/2606.04378#bib.bib26)\)50\.0011\.6135\.0012\.3646\.9542\.0031\.023\.40\\cellcolorgray\!1829\.04RoutingRouterDC\(Chenet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib15)\)82\.8151\.3483\.8013\.6265\.2453\.4029\.986\.10\\cellcolorgray\!1848\.29EmbedLLM\(Zhuanget al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib16)\)81\.2551\.3484\.6018\.2654\.2749\.2033\.686\.80\\cellcolorgray\!1847\.43Logits EnsembleGaC\(Yuet al\.,[2024b](https://arxiv.org/html/2606.04378#bib.bib9)\)81\.2545\.0980\.2016\.7160\.9850\.6034\.842\.70\\cellcolorgray\!1846\.55Entropy Weighting80\.4748\.6682\.4018\.2662\.1951\.6033\.104\.10\\cellcolorgray\!1847\.60Pack of LLMs\(Mavromatiset al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib11)\)82\.0344\.6482\.6018\.1264\.0252\.6034\.264\.10\\cellcolorgray\!1847\.80Token Maj\-Voting78\.1238\.3973\.8017\.5662\.8052\.0035\.423\.40\\cellcolorgray\!1845\.19UniTe\(Yaoet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib10)\)80\.4745\.9880\.2018\.1261\.5950\.0035\.531\.40\\cellcolorgray\!1846\.66Ours82\.8151\.7984\.4019\.9665\.2452\.6036\.114\.10\\cellcolorgray\!1849\.63

Table 2:Performance comparison across in\-domain and out\-of\-domain benchmarks at the 1\.5B scale \(0\-shot\)\. Avg is the unweighted mean across all benchmarks\.Boldandunderlinedenote the best and second\-best results except experts, respectively\.### 4\.1Implementation Details

We evaluate DLLG using multiple specialized language models from the Qwen2\.5 family\(Qwenet al\.,[2025](https://arxiv.org/html/2606.04378#bib.bib40); Huiet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib39); Yanget al\.,[2024a](https://arxiv.org/html/2606.04378#bib.bib38)\), ensuring a shared tokenizer and aligned vocabularies\. The gating model is built on top of the Qwen2\.5\-0\.5B\-Instruct model and trained with LoRA\(Huet al\.,[2022](https://arxiv.org/html/2606.04378#bib.bib41)\)\. During training, we adopt teacher forcing and optimize the gating model using a mean squared error objective\. At inference time, all experts run in parallel and are combined autoregressively through soft logit\-level fusion as described in[Section3](https://arxiv.org/html/2606.04378#S3)\. All experiments are conducted on 8 NVIDIA RTX H200 GPUs\. For more details, please refer to Appendix[AppendixA](https://arxiv.org/html/2606.04378#A1)\.

#### 4\.1\.1Training Data

We collect training data from GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2606.04378#bib.bib1)\), MATH\(Hendryckset al\.,[2021](https://arxiv.org/html/2606.04378#bib.bib3)\)and Code\-R1\(Liu and Zhang,[2025](https://arxiv.org/html/2606.04378#bib.bib4)\)training set, resulting in approximately 24K training examples covering mathematical reasoning and code generation\. The response\-level supervision signals are obtained using task\-specific evaluation criteria or automatic verifiers, depending on the benchmark\.

Table 3:Ablation on gating model architecture under the 1\.5B scale setting\. Aux\-LLM denotes a lightweight auxiliary language model \(Qwen\-0\.5B\-Instruct\), and LowRank denotes an additional low\-rank projection applied to expert representations\.
#### 4\.1\.2Benchmarks and Baselines

We evaluate our method on a diverse set of reasoning and code generation benchmarks, covering both in\-domain and out\-of\-domain settings\. In\-domain benchmarks include GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2606.04378#bib.bib1)\), MinervaMath\(Lewkowyczet al\.,[2022](https://arxiv.org/html/2606.04378#bib.bib2)\), MATH\(Hendryckset al\.,[2021](https://arxiv.org/html/2606.04378#bib.bib3)\), and Code\-R1\(Liu and Zhang,[2025](https://arxiv.org/html/2606.04378#bib.bib4)\)\. Out\-of\-domain benchmarks include HumanEval\(Chen,[2021](https://arxiv.org/html/2606.04378#bib.bib5)\), MBPP\(Austinet al\.,[2021](https://arxiv.org/html/2606.04378#bib.bib6)\), BBH\(Suzgunet al\.,[2023](https://arxiv.org/html/2606.04378#bib.bib7)\), and BigCodeBench\(Zhuoet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib8)\)\. For fair comparison, we utilized lm\-evaluation\-harness and official evaluation kits of Qwen2\.5\-Math, Code\-R1 and BigCodeBench\.

We compare DLLG against a broad range of baselines, including individual parameter\-space model merging methods \(Linear\(Wortsmanet al\.,[2022](https://arxiv.org/html/2606.04378#bib.bib25)\), SLERP\(Shoemake,[1985](https://arxiv.org/html/2606.04378#bib.bib31)\), Task Arithmetic\(Ilharcoet al\.,[2022](https://arxiv.org/html/2606.04378#bib.bib26)\)\), routing\-based approaches \(RouterDC\(Chenet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib15)\), EmbedLLM\(Zhuanget al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib16)\)\), and logit\-level ensembling methods such as uniform averaging \(GaC\(Yuet al\.,[2024b](https://arxiv.org/html/2606.04378#bib.bib9)\)\), entropy\-based weighting, Pack of LLMs\(Mavromatiset al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib11)\), token majority voting, and UniTe\(Yaoet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib10)\)\. All baselines are evaluated under the same 0\-shot setting for fair comparison\.

### 4\.2Multi\-Scale Experiments

#### 4\.2\.10\.5B Experiments

[Table1](https://arxiv.org/html/2606.04378#S3.T1)reports results at the smaller 0\.5B scale, which serves as a challenging low\-capacity setting where individual experts are relatively weak\. Despite this constraint, DLLG achieves the best average performance among all baselines, demonstrating that the benefits of learned token\-level fusion extend beyond high\-capacity regimes\.

Compared to routing\-based and heuristic ensembling methods, DLLG exhibits more robust behavior in this low\-capacity setting\. When experts are error\-prone or unevenly specialized, brittle routing decisions and unreliable proxy\-based weighting strategies can severely degrade performance\. In contrast, DLLG learns to allocate expert contributions dynamically at the token level, mitigating early mistakes and enabling recoverable expert utilization throughout generation\.

Overall, these results indicate that DLLG remains effective even when individual experts are weak, highlighting its robustness in low\-capacity regimes and motivating evaluation at larger model scales\.

#### 4\.2\.21\.5B Experiments

[Table2](https://arxiv.org/html/2606.04378#S4.T2)summarizes the results at the 1\.5B scale across both in\-domain and out\-of\-domain benchmarks\. At this larger scale, DLLG consistently achieves the best average performance and ranks among the top methods on nearly all individual tasks, confirming that the advantages observed at smaller scales persist as model capacity increases\.

Compared to routing\-based methods, DLLG demonstrates substantially more stable performance across diverse benchmarks\. While routing approaches such as RouterDC\(Chenet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib15)\)and EmbedLLM\(Zhuanget al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib16)\)perform competitively on selected in\-domain tasks, their performance degrades when expert strengths vary within a single response, as in mixed reasoning or code\-generation benchmarks\. This limitation stems from hard or coarse\-grained expert selection: once an expert is chosen early in the generation process, errors introduced by suboptimal routing decisions are difficult to recover\. In contrast, DLLG performs soft, token\-level fusion throughout decoding, allowing expert contributions to adapt dynamically as the generation context evolves\.

Relative to logit\-level ensembling baselines, DLLG consistently outperforms uniform averaging and heuristic weighting strategies\(Mavromatiset al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib11); Yaoet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib10)\), including entropy\-based weighting and perplexity\-driven methods\. These approaches rely on empirical proxies that are only loosely correlated with task correctness and fail to capture fine\-grained specialization at the token level\. By contrast, DLLG learns fusion weights directly from response\-level correctness supervision, enabling more effective exploitation of expert complementarity across both reasoning\-intensive tasks \(e\.g\., GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2606.04378#bib.bib1)\), MinervaMath\(Lewkowyczet al\.,[2022](https://arxiv.org/html/2606.04378#bib.bib2)\), MATH\(Hendryckset al\.,[2021](https://arxiv.org/html/2606.04378#bib.bib3)\)\) and code\-generation benchmarks \(e\.g\., Code\-R1\(Liu and Zhang,[2025](https://arxiv.org/html/2606.04378#bib.bib4)\), HumanEval\(Chen,[2021](https://arxiv.org/html/2606.04378#bib.bib5)\)\)\.

Finally, parameter\-space model merging methods\(Wortsmanet al\.,[2022](https://arxiv.org/html/2606.04378#bib.bib25); Shoemake,[1985](https://arxiv.org/html/2606.04378#bib.bib31); Ilharcoet al\.,[2022](https://arxiv.org/html/2606.04378#bib.bib26)\)underperform inference\-time ensembling approaches by a substantial margin\. This gap highlights the challenges of static interference when merging experts with heterogeneous specializations\. Since merged models are fixed after composition, they lack the flexibility to adapt expert contributions during generation, whereas DLLG preserves expert modularity and dynamically integrates expert outputs at inference time\.

### 4\.3Ablation Study

To better understand the design choices of DLLG, we conduct an ablation study on the gating model architecture under the 1\.5B scale setting, with results summarized in[Table3](https://arxiv.org/html/2606.04378#S4.T3)\.

We first observe that simple gating designs, such as a standalone MLP head, fail to achieve competitive performance\. This indicates that naively predicting fusion weights without sufficiently expressive expert representations is insufficient for effective token\-level fusion\. Introducing a cross\-attention\-based gating head improves performance, confirming the importance of modeling interactions between experts\.

Further gains are achieved by incorporating a lightweight auxiliary language model \(Aux\-LLM\) into the gating mechanism\. Conditioning the gating model on richer trajectory\-level representations enables more accurate estimation of expert utility, particularly on reasoning\-heavy benchmarks such as MinervaMath\(Lewkowyczet al\.,[2022](https://arxiv.org/html/2606.04378#bib.bib2)\)and MATH\(Hendryckset al\.,[2021](https://arxiv.org/html/2606.04378#bib.bib3)\)\. Adding a low\-rank projection layer further stabilizes training and improves out\-of\-domain generalization, suggesting that controlled dimensionality reduction helps mitigate overfitting to specific expert behaviors\.

Overall, the full DLLG configuration achieves the strongest and most balanced performance across both in\-domain and out\-of\-domain benchmarks\. These results confirm that effective token\-level fusion requires not only response\-level supervision, but also a sufficiently expressive yet lightweight gating architecture capable of modeling expert trajectories without introducing excessive overhead\.

### 4\.4Analysis of Token\-Level Fusion

To better understand how DLLG utilizes different experts during generation, we analyze the learned token\-level fusion behavior through qualitative visualization\.[Figure3](https://arxiv.org/html/2606.04378#S2.F3)presents a representative example from the Code\-R1\(Liu and Zhang,[2025](https://arxiv.org/html/2606.04378#bib.bib4)\)benchmark, illustrating how expert contributions evolve across decoding steps\.

As shown in the top panel of[Figure3](https://arxiv.org/html/2606.04378#S2.F3), different experts dominate the fused logits at different token positions\. In the early stages of generation, when the response primarily involves mathematical reasoning or symbolic manipulation, the math\-specialized expert tends to receive higher fusion weights\. As the generation progresses and transitions into code synthesis and implementation details, the gating model gradually shifts emphasis toward the code\-specialized expert\. This behavior highlights the limitation of coarse\-grained routing\(Chenet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib15); Zhuanget al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib16)\)or static ensembling\(Mavromatiset al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib11); Yaoet al\.,[2024](https://arxiv.org/html/2606.04378#bib.bib10)\), which cannot revise early decisions when subtask requirements change within a single response\.

The bottom panel visualizes the normalized fusion weights predicted by the gating model over decoding steps\. Rather than remaining static or collapsing to a single expert, the weights vary smoothly and adapt to the evolving generation context\. This demonstrates that DLLG learns fine\-grained, context\-aware expert fusion aligned with subtask structure, enabling recoverable expert utilization throughout generation\.

Additional qualitative examples exhibiting similar fusion patterns across different inputs are provided in Appendix[AppendixB](https://arxiv.org/html/2606.04378#A2)\.

## 5Conclusion

In this work, we introduced DLLG, a dynamic logit\-level ensembling framework that learns token\-level fusion weights from sparse response\-level supervision\. By training a lightweight gating model under a teacher\-forcing regime and combining expert outputs through soft logit\-level fusion, DLLG enables fine\-grained and recoverable expert utilization without requiring token\-level annotations, hard expert selection, or expert retraining\.

Across a broad range of reasoning and code generation benchmarks and multiple model scales, DLLG consistently outperforms strong routing, heuristic ensembling, and parameter\-space merging baselines, demonstrating robust improvements across both in\-domain and out\-of\-domain settings\. Our analysis further shows that the learned fusion weights evolve dynamically during rollout, enabling effective and recoverable expert utilization within a single response\.

DLLG focuses on the fusion mechanism itself and assumes aligned token\-level representations among experts\. Extending the framework to settings with heterogeneous tokenizers and integrating vocabulary alignment techniques is a natural direction for future work\. More broadly, we believe that learning fine\-grained expert utilization from weak supervision offers a promising and practical paradigm for leveraging increasingly specialized language models\.

## Impact Statement

This paper presents a method for improving the robustness and adaptability of large language models through learned logit\-level ensembling\. The primary goal of this work is to advance the field of machine learning by enabling more effective utilization of specialized models without additional supervision or retraining\.

As with many techniques that improve the performance and flexibility of language models, the proposed approach may contribute to downstream applications that rely on such models\. While these applications could have broad societal impacts depending on their specific use cases, we do not identify any unique or direct ethical concerns introduced by the method itself beyond those commonly associated with large language models\. We therefore believe that the societal implications of this work are consistent with those of existing research in model ensembling and language model deployment\.

## References

- J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2606.04378#S1.p1.1)\.
- J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le,et al\.\(2021\)Program synthesis with large language models\.arXiv preprint arXiv:2108\.07732\.Cited by:[item 3](https://arxiv.org/html/2606.04378#S1.I1.i3.p1.1),[§4\.1\.2](https://arxiv.org/html/2606.04378#S4.SS1.SSS2.p1.1)\.
- W\. Cai, J\. Jiang, F\. Wang, J\. Tang, S\. Kim, and J\. Huang \(2025\)A survey on mixture of experts in large language models\.IEEE Transactions on Knowledge and Data Engineering\.Cited by:[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px4.p1.1)\.
- M\. Chen \(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[item 3](https://arxiv.org/html/2606.04378#S1.I1.i3.p1.1),[§4\.1\.2](https://arxiv.org/html/2606.04378#S4.SS1.SSS2.p1.1),[§4\.2\.2](https://arxiv.org/html/2606.04378#S4.SS2.SSS2.p3.1)\.
- S\. Chen, W\. Jiang, B\. Lin, J\. Kwok, and Y\. Zhang \(2024\)Routerdc: query\-based router by dual contrastive learning for assembling large language models\.Advances in Neural Information Processing Systems37,pp\. 66305–66328\.Cited by:[§1](https://arxiv.org/html/2606.04378#S1.p2.1),[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2606.04378#S3.T1.2.1.1.1.1.1.1.12.12.1),[§4\.1\.2](https://arxiv.org/html/2606.04378#S4.SS1.SSS2.p2.1),[§4\.2\.2](https://arxiv.org/html/2606.04378#S4.SS2.SSS2.p2.1),[§4\.4](https://arxiv.org/html/2606.04378#S4.SS4.p2.1),[Table 2](https://arxiv.org/html/2606.04378#S4.T2.2.1.1.1.1.1.1.12.12.1)\.
- T\. Chen, Z\. Zhang, A\. Jaiswal, S\. Liu, and Z\. Wang \(2023\)Sparse moe as the new dropout: scaling dense and self\-slimmable transformers\.arXiv preprint arXiv:2303\.01610\.Cited by:[§1](https://arxiv.org/html/2606.04378#S1.p2.1),[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px4.p1.1)\.
- Z\. Chen, J\. Li, P\. Chen, Z\. Li, K\. Sun, Y\. Luo, Q\. Mao, M\. Li, L\. Xiao, D\. Yang,et al\.\(2025\)Harnessing multiple large language models: a survey on llm ensemble\.arXiv preprint arXiv:2502\.18036\.Cited by:[§1](https://arxiv.org/html/2606.04378#S1.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[item 3](https://arxiv.org/html/2606.04378#S1.I1.i3.p1.1),[§4\.1\.1](https://arxiv.org/html/2606.04378#S4.SS1.SSS1.p1.1),[§4\.1\.2](https://arxiv.org/html/2606.04378#S4.SS1.SSS2.p1.1),[§4\.2\.2](https://arxiv.org/html/2606.04378#S4.SS2.SSS2.p3.1)\.
- D\. Dai, C\. Deng, C\. Zhao, R\. Xu, H\. Gao, D\. Chen, J\. Li, W\. Zeng, X\. Yu, Y\. Wu,et al\.\(2024\)Deepseekmoe: towards ultimate expert specialization in mixture\-of\-experts language models\.arXiv preprint arXiv:2401\.06066\.Cited by:[§1](https://arxiv.org/html/2606.04378#S1.p2.1),[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px4.p1.1)\.
- P\. T\. Deep, R\. Bhardwaj, and S\. Poria \(2024\)Della\-merging: reducing interference in model merging through magnitude\-based sampling\.arXiv preprint arXiv:2406\.11617\.Cited by:[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px3.p1.1)\.
- D\. Ding, A\. Mallick, C\. Wang, R\. Sim, S\. Mukherjee, V\. Ruhle, L\. V\. Lakshmanan, and A\. H\. Awadallah \(2024\)Hybrid llm: cost\-efficient and quality\-aware query routing\.arXiv preprint arXiv:2404\.14618\.Cited by:[§1](https://arxiv.org/html/2606.04378#S1.p2.1),[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px2.p1.1)\.
- K\. Grove and H\. Karcher \(1973\)How to conjugate c 1\-close group actions\.Mathematische Zeitschrift132\(1\),pp\. 11–20\.Cited by:[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px3.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021\)Measuring mathematical problem solving with the math dataset\.arXiv preprint arXiv:2103\.03874\.Cited by:[item 3](https://arxiv.org/html/2606.04378#S1.I1.i3.p1.1),[§4\.1\.1](https://arxiv.org/html/2606.04378#S4.SS1.SSS1.p1.1),[§4\.1\.2](https://arxiv.org/html/2606.04378#S4.SS1.SSS2.p1.1),[§4\.2\.2](https://arxiv.org/html/2606.04378#S4.SS2.SSS2.p3.1),[§4\.3](https://arxiv.org/html/2606.04378#S4.SS3.p3.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.\(2022\)Lora: low\-rank adaptation of large language models\.\.ICLR1\(2\),pp\. 3\.Cited by:[§4\.1](https://arxiv.org/html/2606.04378#S4.SS1.p1.1)\.
- Y\. Huang, X\. Feng, B\. Li, Y\. Xiang, H\. Wang, T\. Liu, and B\. Qin \(2024\)Ensemble learning for heterogeneous large language models with deep parallel collaboration\.Advances in Neural Information Processing Systems37,pp\. 119838–119860\.Cited by:[§1](https://arxiv.org/html/2606.04378#S1.p2.1),[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px1.p1.1)\.
- B\. Hui, J\. Yang, Z\. Cui, J\. Yang, D\. Liu, L\. Zhang, T\. Liu, J\. Zhang, B\. Yu, K\. Lu,et al\.\(2024\)Qwen2\. 5\-coder technical report\.arXiv preprint arXiv:2409\.12186\.Cited by:[§1](https://arxiv.org/html/2606.04378#S1.p1.1),[§4\.1](https://arxiv.org/html/2606.04378#S4.SS1.p1.1)\.
- G\. Ilharco, M\. T\. Ribeiro, M\. Wortsman, S\. Gururangan, L\. Schmidt, H\. Hajishirzi, and A\. Farhadi \(2022\)Editing models with task arithmetic\.arXiv preprint arXiv:2212\.04089\.Cited by:[§1](https://arxiv.org/html/2606.04378#S1.p2.1),[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px3.p1.1),[Table 1](https://arxiv.org/html/2606.04378#S3.T1.2.1.1.1.1.1.1.10.10.1),[§4\.1\.2](https://arxiv.org/html/2606.04378#S4.SS1.SSS2.p2.1),[§4\.2\.2](https://arxiv.org/html/2606.04378#S4.SS2.SSS2.p4.1),[Table 2](https://arxiv.org/html/2606.04378#S4.T2.2.1.1.1.1.1.1.10.10.1)\.
- A\. Q\. Jiang, A\. Sablayrolles, A\. Roux, A\. Mensch, B\. Savary, C\. Bamford, D\. S\. Chaplot, D\. d\. l\. Casas, E\. B\. Hanna, F\. Bressand,et al\.\(2024\)Mixtral of experts\.arXiv preprint arXiv:2401\.04088\.Cited by:[§1](https://arxiv.org/html/2606.04378#S1.p2.1),[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px4.p1.1)\.
- A\. Lewkowycz, A\. Andreassen, D\. Dohan, E\. Dyer, H\. Michalewski, V\. Ramasesh, A\. Slone, C\. Anil, I\. Schlag, T\. Gutman\-Solo,et al\.\(2022\)Solving quantitative reasoning problems with language models\.Advances in neural information processing systems35,pp\. 3843–3857\.Cited by:[item 3](https://arxiv.org/html/2606.04378#S1.I1.i3.p1.1),[§4\.1\.2](https://arxiv.org/html/2606.04378#S4.SS1.SSS2.p1.1),[§4\.2\.2](https://arxiv.org/html/2606.04378#S4.SS2.SSS2.p3.1),[§4\.3](https://arxiv.org/html/2606.04378#S4.SS3.p3.1)\.
- W\. Li, Y\. Peng, M\. Zhang, L\. Ding, H\. Hu, and L\. Shen \(2023\)Deep model fusion: a survey\.arXiv preprint arXiv:2309\.15698\.Cited by:[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Liu and L\. Zhang \(2025\)Code\-r1: reproducing r1 for code with reliable rewards\.Note:[https://github\.com/ganler/code\-r1](https://github.com/ganler/code-r1)Cited by:[item 3](https://arxiv.org/html/2606.04378#S1.I1.i3.p1.1),[§4\.1\.1](https://arxiv.org/html/2606.04378#S4.SS1.SSS1.p1.1),[§4\.1\.2](https://arxiv.org/html/2606.04378#S4.SS1.SSS2.p1.1),[§4\.2\.2](https://arxiv.org/html/2606.04378#S4.SS2.SSS2.p3.1),[§4\.4](https://arxiv.org/html/2606.04378#S4.SS4.p1.1)\.
- C\. Mavromatis, P\. Karypis, and G\. Karypis \(2024\)Pack of llms: model fusion at test\-time via perplexity optimization\.arXiv preprint arXiv:2404\.11531\.Cited by:[§1](https://arxiv.org/html/2606.04378#S1.p2.1),[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2606.04378#S3.T1.2.1.1.1.1.1.1.17.17.1),[§4\.1\.2](https://arxiv.org/html/2606.04378#S4.SS1.SSS2.p2.1),[§4\.2\.2](https://arxiv.org/html/2606.04378#S4.SS2.SSS2.p3.1),[§4\.4](https://arxiv.org/html/2606.04378#S4.SS4.p2.1),[Table 2](https://arxiv.org/html/2606.04378#S4.T2.2.1.1.1.1.1.1.17.17.1)\.
- Q\. H\. Nguyen, T\. Dao, D\. C\. Hoang, J\. Decugis, S\. Manchanda, N\. V\. Chawla, and K\. D\. Doan \(2024\)Metallm: a high\-performant and cost\-efficient dynamic framework for wrapping llms\.arXiv preprint arXiv:2407\.10834\.Cited by:[§1](https://arxiv.org/html/2606.04378#S1.p2.1),[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px2.p1.1)\.
- E\. Nunez, L\. Zancato, B\. Bowman, A\. Golatkar, W\. Xia, and S\. Soatto \(2024\)Expansion span: combining fading memory and retrieval in hybrid state space models\.arXiv preprint arXiv:2412\.13328\.Cited by:[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px1.p1.1)\.
- I\. Ong, A\. Almahairi, V\. Wu, W\. Chiang, T\. Wu, J\. E\. Gonzalez, M\. W\. Kadous, and I\. Stoica \(2024\)Routellm: learning to route llms with preference data\.arXiv preprint arXiv:2406\.18665\.Cited by:[§1](https://arxiv.org/html/2606.04378#S1.p2.1),[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Pan, K\. Zhang, Y\. Zhao, and Y\. Han \(2025\)Route to reason: adaptive routing for llm and reasoning strategy selection\.arXiv preprint arXiv:2505\.19435\.Cited by:[§1](https://arxiv.org/html/2606.04378#S1.p2.1),[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px2.p1.1)\.
- Qwen, :, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu \(2025\)Qwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[§4\.1](https://arxiv.org/html/2606.04378#S4.SS1.p1.1)\.
- M\. Šakota, M\. Peyrard, and R\. West \(2024\)Fly\-swat or cannon? cost\-effective language model choice via meta\-modeling\.InProceedings of the 17th ACM International Conference on Web Search and Data Mining,pp\. 606–615\.Cited by:[§1](https://arxiv.org/html/2606.04378#S1.p2.1),[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px2.p1.1)\.
- N\. Shazeer, A\. Mirhoseini, K\. Maziarz, A\. Davis, Q\. Le, G\. Hinton, and J\. Dean \(2017\)Outrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.arXiv preprint arXiv:1701\.06538\.Cited by:[§1](https://arxiv.org/html/2606.04378#S1.p2.1),[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px4.p1.1)\.
- K\. Shoemake \(1985\)Animating rotation with quaternion curves\.InProceedings of the 12th annual conference on Computer graphics and interactive techniques,pp\. 245–254\.Cited by:[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px3.p1.1),[§4\.1\.2](https://arxiv.org/html/2606.04378#S4.SS1.SSS2.p2.1),[§4\.2\.2](https://arxiv.org/html/2606.04378#S4.SS2.SSS2.p4.1)\.
- M\. Suzgun, N\. Scales, N\. Schärli, S\. Gehrmann, Y\. Tay, H\. W\. Chung, A\. Chowdhery, Q\. Le, E\. Chi, D\. Zhou,et al\.\(2023\)Challenging big\-bench tasks and whether chain\-of\-thought can solve them\.InFindings of the Association for Computational Linguistics: ACL 2023,pp\. 13003–13051\.Cited by:[item 3](https://arxiv.org/html/2606.04378#S1.I1.i3.p1.1),[§4\.1\.2](https://arxiv.org/html/2606.04378#S4.SS1.SSS2.p1.1)\.
- H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar,et al\.\(2023\)Llama: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§1](https://arxiv.org/html/2606.04378#S1.p1.1)\.
- Y\. Wang, X\. Zhang, J\. Zhao, S\. Wen, P\. Feng, S\. Liao, L\. Huang, and W\. Wu \(2024\)Bench\-coe: a framework for collaboration of experts from benchmark\.arXiv preprint arXiv:2412\.04167\.Cited by:[§1](https://arxiv.org/html/2606.04378#S1.p2.1),[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px2.p1.1)\.
- R\. Wicks, K\. Ravisankar, X\. Yang, P\. Koehn, and M\. Post \(2025\)Token\-level ensembling of models with different vocabularies\.arXiv preprint arXiv:2502\.21265\.Cited by:[§1](https://arxiv.org/html/2606.04378#S1.p2.1),[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px1.p1.1)\.
- M\. Wortsman, G\. Ilharco, S\. Y\. Gadre, R\. Roelofs, R\. Gontijo\-Lopes, A\. S\. Morcos, H\. Namkoong, A\. Farhadi, Y\. Carmon, S\. Kornblith,et al\.\(2022\)Model soups: averaging weights of multiple fine\-tuned models improves accuracy without increasing inference time\.InInternational conference on machine learning,pp\. 23965–23998\.Cited by:[§1](https://arxiv.org/html/2606.04378#S1.p2.1),[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px3.p1.1),[Table 1](https://arxiv.org/html/2606.04378#S3.T1.2.1.1.1.1.1.1.8.8.1),[§4\.1\.2](https://arxiv.org/html/2606.04378#S4.SS1.SSS2.p2.1),[§4\.2\.2](https://arxiv.org/html/2606.04378#S4.SS2.SSS2.p4.1),[Table 2](https://arxiv.org/html/2606.04378#S4.T2.2.1.1.1.1.1.1.8.8.1)\.
- Y\. Xu, J\. Lu, and J\. Zhang \(2024\)Bridging the gap between different vocabularies for llm ensemble\.arXiv preprint arXiv:2404\.09492\.Cited by:[§1](https://arxiv.org/html/2606.04378#S1.p2.1),[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px1.p1.1)\.
- P\. Yadav, D\. Tam, L\. Choshen, C\. A\. Raffel, and M\. Bansal \(2023\)Ties\-merging: resolving interference when merging models\.Advances in Neural Information Processing Systems36,pp\. 7093–7115\.Cited by:[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Yang, B\. Zhang, B\. Hui, B\. Gao, B\. Yu, C\. Li, D\. Liu, J\. Tu, J\. Zhou, J\. Lin,et al\.\(2024a\)Qwen2\. 5\-math technical report: toward mathematical expert model via self\-improvement\.arXiv preprint arXiv:2409\.12122\.Cited by:[§1](https://arxiv.org/html/2606.04378#S1.p1.1),[§4\.1](https://arxiv.org/html/2606.04378#S4.SS1.p1.1)\.
- E\. Yang, L\. Shen, G\. Guo, X\. Wang, X\. Cao, J\. Zhang, and D\. Tao \(2024b\)Model merging in llms, mllms, and beyond: methods, theories, applications and opportunities\.arXiv preprint arXiv:2408\.07666\.Cited by:[§1](https://arxiv.org/html/2606.04378#S1.p1.1),[§1](https://arxiv.org/html/2606.04378#S1.p2.1),[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Yao, H\. Wu, M\. Liu, S\. Luo, X\. Han, J\. Liu, Z\. Guo, and L\. Song \(2024\)Determine\-then\-ensemble: necessity of top\-k union for large language model ensembling\.arXiv preprint arXiv:2410\.03777\.Cited by:[§1](https://arxiv.org/html/2606.04378#S1.p2.1),[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2606.04378#S3.T1.2.1.1.1.1.1.1.19.19.1),[§4\.1\.2](https://arxiv.org/html/2606.04378#S4.SS1.SSS2.p2.1),[§4\.2\.2](https://arxiv.org/html/2606.04378#S4.SS2.SSS2.p3.1),[§4\.4](https://arxiv.org/html/2606.04378#S4.SS4.p2.1),[Table 2](https://arxiv.org/html/2606.04378#S4.T2.2.1.1.1.1.1.1.19.19.1)\.
- L\. Yu, B\. Yu, H\. Yu, F\. Huang, and Y\. Li \(2024a\)Language models are super mario: absorbing abilities from homologous models as a free lunch\.InForty\-first International Conference on Machine Learning,Cited by:[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Yu, C\. C\. Kuo, Y\. Ziqi, C\. Yucheng, and Y\. Li \(2024b\)Breaking the ceiling of the llm community by treating token generation as a classification for ensembling\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 1826–1839\.Cited by:[§1](https://arxiv.org/html/2606.04378#S1.p2.1),[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2606.04378#S3.T1.2.1.1.1.1.1.1.15.15.1),[§4\.1\.2](https://arxiv.org/html/2606.04378#S4.SS1.SSS2.p2.1),[Table 2](https://arxiv.org/html/2606.04378#S4.T2.2.1.1.1.1.1.1.15.15.1)\.
- Y\. Zhang, H\. Li, C\. Wang, L\. Chen, Q\. Zhang, P\. Ye, S\. Feng, D\. Wang, Z\. Wang, X\. Wang,et al\.\(2025\)The avengers: a simple recipe for uniting smaller language models to challenge proprietary giants\.arXiv preprint arXiv:2505\.19797\.Cited by:[§1](https://arxiv.org/html/2606.04378#S1.p2.1),[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px2.p1.1)\.
- R\. Zhuang, T\. Wu, Z\. Wen, A\. Li, J\. Jiao, and K\. Ramchandran \(2024\)EmbedLLM: learning compact representations of large language models\.arXiv preprint arXiv:2410\.02223\.Cited by:[§1](https://arxiv.org/html/2606.04378#S1.p2.1),[§2](https://arxiv.org/html/2606.04378#S2.SS0.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2606.04378#S3.T1.2.1.1.1.1.1.1.13.13.1),[§4\.1\.2](https://arxiv.org/html/2606.04378#S4.SS1.SSS2.p2.1),[§4\.2\.2](https://arxiv.org/html/2606.04378#S4.SS2.SSS2.p2.1),[§4\.4](https://arxiv.org/html/2606.04378#S4.SS4.p2.1),[Table 2](https://arxiv.org/html/2606.04378#S4.T2.2.1.1.1.1.1.1.13.13.1)\.
- T\. Y\. Zhuo, M\. C\. Vu, J\. Chim, H\. Hu, W\. Yu, R\. Widyasari, I\. N\. B\. Yusuf, H\. Zhan, J\. He, I\. Paul,et al\.\(2024\)Bigcodebench: benchmarking code generation with diverse function calls and complex instructions\.arXiv preprint arXiv:2406\.15877\.Cited by:[item 3](https://arxiv.org/html/2606.04378#S1.I1.i3.p1.1),[§4\.1\.2](https://arxiv.org/html/2606.04378#S4.SS1.SSS2.p1.1)\.

## Appendix AMore Implementation Details

### A\.1Training

The gating model is trained using the AdamW optimizer with a learning rate of1×10−31\\times 10^\{\-3\}\. Training is performed for a total of 1,000 optimization steps\. All LoRA adapters are configured with rank 16\. The projection rank is set to be 64\. The batch size is fixed to 32 across all experiments\. Unless otherwise specified, all remaining optimization hyperparameters follow standard settings\.

### A\.2Evaluation

All evaluations are conducted under a strictly zero\-shot setting\.

Due to the computational cost of multi\-model logit\-level ensemble inference, we evaluate several large benchmarks on fixed subsets\. Unless otherwise specified, subsets are selected by taking the firstNNexamples from the original benchmark order, without any filtering based on model outputs\.

For benchmarks evaluated usinglm\-evaluation\-harness, we report results ongsm8k\_cot\(first 128 examples\),minerva\_math\(first 32 examples for each sub\-task\),humaneval\_instruct\(full set\),mbpp\_instruct\(full set\), andbbh\_zeroshot\(first 32 examples for each sub\-task\)\. The MATH benchmark is evaluated using the official Qwen2\.5\-Math evaluation toolkit on the first 500 examples\. For BigCodeBench, we evaluate on both thecompletesplit and thehardsubset\.

For Code\-R1, we evaluate all 712 examples using the official evaluation framework with a minor modification to the reward function\. Specifically, the weights of all format\-related reward components are set to zero, and correctness is determined solely based on the extracted code and its execution results\. This setting isolates functional correctness from formatting\-related factors\.

Across all benchmarks, the maximum generation length is fixed to 3,072 tokens\. All evaluations apply the default chat templates provided by the corresponding evaluation toolkit for each task\.

## Appendix BMore Fusion Weights Visualizations

To further illustrate the behavior of the learned token\-level fusion in DLLG, we present additional qualitative visualizations of fusion weights across different problem instances\. These examples complement the analysis in[Section4\.4](https://arxiv.org/html/2606.04378#S4.SS4)and demonstrate that the observed fusion patterns are consistent across diverse inputs\.

Figure[4](https://arxiv.org/html/2606.04378#A2.F4)shows representative examples from reasoning\- and code\-centric tasks\. For each example, the top panel visualizes token\-level expert dominance in the fused output, while the bottom panel plots the normalized fusion weights predicted by the gating model over decoding steps\.

A clear and recurring pattern emerges across examples\. In the early stages of generation, when the response primarily involves mathematical reasoning or symbolic manipulation, the math\-specialized expert receives higher fusion weights and dominates the combined logits\. As the generation transitions into code synthesis, implementation details, or executable logic, the gating model progressively shifts emphasis toward the code\-specialized expert\. This transition occurs smoothly over tokens rather than through abrupt switches, reflecting fine\-grained and recoverable expert utilization\.

Importantly, the fusion weights do not collapse to static values or a single dominant expert\. Instead, DLLG dynamically reallocates expert influence in a context\-dependent manner, enabling different experts to contribute where they are most effective within a single response\. These qualitative results further support that the learned gating mechanism captures subtask structure and generalizes beyond individual examples, consistently performing token\-level fusion aligned with the evolving generation context\.

![Refer to caption](https://arxiv.org/html/2606.04378v1/x4.png)Figure 4:Additional visualizations of token\-level weights in DLLG across different examples\. For each example, the top panel highlights token\-wise expert dominance in the fused output, and the bottom panel shows the normalized fusion weights over decoding steps\. Across examples, math\-specialized experts tend to dominate during early reasoning\-heavy segments, while code\-specialized experts become more prominent during later code\-generation phases\. The smooth transitions between experts illustrate DLLG’s ability to perform context\-aware, token\-level fusion rather than static weighting or hard expert selection\.
## Appendix CLimitations

### C\.1Latency and resource cost

A practical limitation of DLLG is its increased inference\-time resource cost\. Unlike standard single\-model decoding, DLLG requires all experts to be executed at every decoding step before the next token can be selected\. In our implementation, experts are executed in parallel across multiple GPUs, which helps limit the additional wall\-clock latency compared with single\-expert decoding\. However, this comes at the cost of higher aggregate computation, memory usage, and hardware requirements\. If sufficient parallel resources are not available, executing experts sequentially would directly increase per\-token latency\.

These resource costs define the scope of our empirical comparisons\. Our goal is to study whether specialized models can be dynamically combined at the token level, rather than to establish compute\-optimality against a single larger model under a matched inference budget\. We therefore do not claim that DLLG is preferable to scaling a single model under the same total compute or memory budget\. Compute\-matched comparisons with larger single models are left to future work\.

### C\.2Serving limitations

DLLG is currently implemented on top of HuggingFace Transformers, which provides explicit control over the rollout loop and simplifies KV\-cache management\. This is important because DLLG requires all experts to advance in lock\-step: at each decoding step, it collects expert logits and hidden states, predicts token\-level fusion weights, merges logits before sampling, and feeds the same generated token back to all experts\.

This design is not directly compatible with off\-the\-shelfvLLMserving\. WhilevLLMis highly optimized for standard single\-model inference, its rollout loop, scheduling, sampling, and KV\-cache management are largely encapsulated inside the engine\. DLLG, in contrast, requires a cross\-model synchronization point before every sampling step and aligned KV caches across all experts\. Running experts as separatevLLMengines would further introduce token\-level communication overhead and complicate cache alignment under continuous batching\. We therefore leave optimizedvLLM\-style serving support for DLLG to future work\.