IntBMoE: 在专家组合中集成区块级条件的全参与式混合专家模型

# IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts
Source: [https://arxiv.org/html/2609.21346](https://arxiv.org/html/2609.21346)
Longfei Xu11footnotemark:1††thanks:Project lead\.Zheng Liu11footnotemark:1Kaikui LiuXiangxiang ChuDreamX, Alibaba Group

###### Abstract

Mixture\-of\-Experts \(MoE\) scales capacity, but existing designs cannot set three quantities independently\. For a single token,participationis how many experts contribute knowledge to its output,executionis how many are actually computed \(compute cost\), andmaterializationis how many expert\-sized parameter sets must be built and stored \(memory cost\)\. Sparse routing keeps execution and materialization low, but shrinks participation: for each token, only a few experts contribute\. Dense output\-mixing restores full participation, but its execution grows with the number of experts\. Parameter\-merging keeps execution at one expert, but its materialization grows with the number of routing decisions\.

We propose IntBMoE, a block\-conditioned MoE that decouples all three by pairing dense expert composition with sparse block execution\. Its blocks come from a small learned codebook, one per entry\. At each internal layer, a lightweight hypernetwork merges all expert bases in that layer’s pool into one composed expert\.Participation is full, because every composed expert draws on the entire pool\.Execution stays sparse, because a router sends each token to only a few blocks\.Materialization is bounded, because the codebook, not the input, fixes how many blocks exist\. Dual\-Path Residual Gating \(DPRG\) further couples two independently composed paths through multiplicative gating\.

Experiments on image classification show consistent gains over representative sparse and dense MoE baselines\. Additional experiments on language modeling and sequential recommendation validate its generalization beyond vision\. IntBMoE is fully deployed in AMap’s generative recommendation system, serving hundreds of millions of users under a60​ms60\\,\\mathrm\{ms\}latency budget, with a2\.4%2\.4\\%relative UVCTR gain in online A/B testing\. Our code is available at[https://github\.com/AMAP\-ML/DreamX\-Rec/](https://github.com/AMAP-ML/DreamX-Rec/)\.

## 1Introduction

Mixture\-of\-Experts \(MoE\) architectures have been widely adopted in language models[Lepikhin et al\. \(2020\)](https://arxiv.org/html/2609.21346#bib.bib10);[Fedus et al\. \(2022\)](https://arxiv.org/html/2609.21346#bib.bib4);[Jiang et al\. \(2024\)](https://arxiv.org/html/2609.21346#bib.bib8);[Dai et al\. \(2024\)](https://arxiv.org/html/2609.21346#bib.bib1), vision models[Riquelme et al\. \(2021\)](https://arxiv.org/html/2609.21346#bib.bib19);[Fan et al\. \(2022\)](https://arxiv.org/html/2609.21346#bib.bib3), multimodal models[Mustafa et al\. \(2022\)](https://arxiv.org/html/2609.21346#bib.bib15);[Xue et al\. \(2023\)](https://arxiv.org/html/2609.21346#bib.bib27);[Li et al\. \(2024\)](https://arxiv.org/html/2609.21346#bib.bib11), and recommendation systems[Deng et al\. \(2025\)](https://arxiv.org/html/2609.21346#bib.bib2);[Zhu et al\. \(2025\)](https://arxiv.org/html/2609.21346#bib.bib35)\. A standard MoE layer contains a pool of experts and a router that determines how the experts process each token\. Existing designs follow three main strategies\.*Sparse\-routing methods*execute only a small number of selected experts[Shazeer et al\. \(2017\)](https://arxiv.org/html/2609.21346#bib.bib21);[Lepikhin et al\. \(2020\)](https://arxiv.org/html/2609.21346#bib.bib10);[Fedus et al\. \(2022\)](https://arxiv.org/html/2609.21346#bib.bib4);[Dai et al\. \(2024\)](https://arxiv.org/html/2609.21346#bib.bib1)\.*Dense output\-mixing methods*execute every expert and combine their outputs[Ma et al\. \(2018\)](https://arxiv.org/html/2609.21346#bib.bib13);[Tang et al\. \(2020\)](https://arxiv.org/html/2609.21346#bib.bib22)\.*Parameter\-merging methods*combine the parameters of all experts into a single composite expert and then execute it[Muqeeth et al\. \(2023\)](https://arxiv.org/html/2609.21346#bib.bib14);[Zhong et al\. \(2024\)](https://arxiv.org/html/2609.21346#bib.bib33)\.

We analyze these approaches along three dimensions\.Participation: how many experts contribute knowledge to a token’s output?Execution: how many of them must actually be computed, and so how much compute?Materialization: how many expert\-sized parameter sets must be built and stored \(one for every distinct routing decision\), and so how much memory? In principle the three are independent, yet existing designs couple them, as summarized in Figure[1](https://arxiv.org/html/2609.21346#S1.F1), producing three common trade\-offs:

Figure 1:Comparison of MoE design strategies\. Filled circles indicate that a strategy satisfies the corresponding property, while hollow circles indicate otherwise\.- •Sparse execution limits participation\.Sparse\-routing methods bound execution by activating only a few experts for each token\. The decision narrows participation: non\-selected experts neither affect the token’s output nor receive a learning signal from it\.
- •Full participation requires dense execution\.Dense output\-mixing methods combine the outputs of all experts for each token\. Because every expert must be evaluated, execution cost grows with the number of experts\.
- •Single\-expert execution increases materialization\.Parameter\-merging methods combine the full expert pool into one composed expert and execute it once\. Every distinct routing decision, however, needs its own copy of those composed weights, so the number of expert\-sized parameter sets built at run time grows with the number of routing units\.

Taken together, these trade\-offs leave a central question:Can every token benefit from the full expert pool without requiring dense execution or unbounded parameter materialization?Existing MoE formulations cannot satisfy all three requirements because they couple the construction of expert transformations with their execution on tokens\. To break this coupling, we propose IntBMoE, a block\-conditioned MoE architecture that separates the two stages\. The full expert pool first constructs a bounded set of reusable transformations, after which each token independently selects and executes only a few of them\. IntBMoE thereby achieves pool\-wide expert participation, sparse execution, and bounded parameter materialization\.

Our main contributions are summarized as follows:

- •Decoupling participation, execution, and materialization\.We propose IntBMoE to set the three quantities independently\. A small codebook of learned embeddings defines the blocks a module can use, and a shared hypernetwork builds each one layer by layer, merging that layer’s expert bases into a single composed expert\. A router then sends each token to only a few of these blocks\. Participation is therefore pool\-wide, execution stays sparse, and materialization is bounded by the codebook\.
- •Expressive expert composition\.We introduce Dual\-Path Residual Gating \(DPRG\), which merges each block’s expert bases twice, into a value path and a gate path whose product is nonlinear in those bases\. DPRG therefore adds expressiveness without enlarging the expert pool\.
- •Visual evaluation and cross\-domain generalization\.We compare IntBMoE with representative sparse and dense MoE baselines on ImageNet\-1K and observe consistent improvements\. Results on language modeling and sequential recommendation further show that the same architecture is effective across modalities and application domains\. IntBMoE has also been fully deployed in AMap’s generative recommendation system, serving hundreds of millions of users under a strict60​ms60\\,\\mathrm\{ms\}latency budget and delivering a2\.4%2\.4\\%relative UVCTR improvement in large\-scale online A/B testing\.

## 2Related Work

### 2\.1Sparse Mixture\-of\-Experts

Sparsely gated MoEs activate only a subset of experts for each token[Shazeer et al\. \(2017\)](https://arxiv.org/html/2609.21346#bib.bib21)\. GShard[Lepikhin et al\. \(2020\)](https://arxiv.org/html/2609.21346#bib.bib10)combines Top\-2 expert routing with automatic sharding for large\-scale Transformer training, while Switch Transformer[Fedus et al\. \(2022\)](https://arxiv.org/html/2609.21346#bib.bib4)simplifies the routing rule to Top\-1 selection\. V\-MoE[Riquelme et al\. \(2021\)](https://arxiv.org/html/2609.21346#bib.bib19)extends token\-choice sparse routing to Vision Transformers by routing image patch tokens to a small number of experts\. Expert Choice[Zhou et al\. \(2022\)](https://arxiv.org/html/2609.21346#bib.bib34)reverses the assignment direction\. Each expert selects a fixed\-capacity set of tokens, balancing expert workloads while allowing a variable number of assignments per token\. DeepSeekMoE[Dai et al\. \(2024\)](https://arxiv.org/html/2609.21346#bib.bib1)improves expert organization through fine\-grained expert segmentation and shared\-expert isolation\. DeepSeek\-V3[Liu et al\. \(2024\)](https://arxiv.org/html/2609.21346#bib.bib12)retains this fine\-grained architecture and introduces an auxiliary\-loss\-free routing bias to improve load balance\.

Subsequent work modifies expert structure and routing while preserving sparse execution\. D2\-MoE[Gu et al\. \(2025\)](https://arxiv.org/html/2609.21346#bib.bib5)decomposes pretrained experts into a shared base and compressed expert\-specific deltas, after which sparse routing activates only the selected deltas\. ReLU\-routing ReMoE[Wang et al\. \(2024\)](https://arxiv.org/html/2609.21346#bib.bib26)replaces discontinuous Top\-kkselection with continuous ReLU gates and regularizes their sparsity and load balance\. LapSum SoftMoE[Zasada et al\. \(2026\)](https://arxiv.org/html/2609.21346#bib.bib30)instead uses a truncated soft Top\-kkrelaxation and learns how to allocate an overall expert\-computation budget across layers\. Dense2MoE[Zheng et al\. \(2025\)](https://arxiv.org/html/2609.21346#bib.bib32)extends sparse selection to model depth\. Its Mixture of Blocks executes only a subset of existing Transformer blocks\.

Recent adaptive sparse MoEs relax fixed choices for the number of experts maintained per layer and activated per token\. DynMoE[Guo et al\. \(2025\)](https://arxiv.org/html/2609.21346#bib.bib6)adjusts the expert pool based on token–expert routing coverage and uses Top\-any routing\. MASS[Park & Park \(2026\)](https://arxiv.org/html/2609.21346#bib.bib17)expands the expert pool using gradient\-based semantic drift detection and uses Top\-pprouting\.

Together, these methods improve scalability, routing efficiency, and expert organization\. However, expert participation remains coupled to execution\. Each token can benefit only from the experts selected and evaluated for it\.

### 2\.2Dense Output Mixing

Dense output\-mixing methods evaluate all experts and combine their outputs with learned routing weights\. MMoE[Ma et al\. \(2018\)](https://arxiv.org/html/2609.21346#bib.bib13)shares an expert pool across tasks and learns a task\-specific gate to combine the expert outputs\. PLE[Tang et al\. \(2020\)](https://arxiv.org/html/2609.21346#bib.bib22)stacks multiple extraction layers containing shared and task\-specific experts, progressively separating shared knowledge from task\-specific information\. Both allow every expert in the relevant pool to contribute to an output, but doing so requires computing every expert’s output, causing the execution cost to grow with the number of experts\.

Soft MoE[Puigcerver et al\. \(2024\)](https://arxiv.org/html/2609.21346#bib.bib18)provides a distinct slot\-based variant\. It softly aggregates input tokens into a fixed set of slots, processes each slot with its assigned expert, and maps the processed slots back to individual tokens\. Because each slot mixes all input tokens, the original formulation does not preserve causality and is not directly applicable to autoregressive prediction\.μ\\muMoE[Oldfield et al\. \(2024\)](https://arxiv.org/html/2609.21346#bib.bib16)instead takes a factorized approach\. It represents expert weights as a tensor and computes their mixture using CP or Tensor Ring factorization\. This avoids materializing the full tensor and evaluating experts separately\.

### 2\.3Parameter Merging

Parameter\-merging methods achieve full expert participation in a different way\. SMEAR[Muqeeth et al\. \(2023\)](https://arxiv.org/html/2609.21346#bib.bib14)constructs one composite expert by taking a routing\-weighted average of all experts and then executes the merged expert on the input\. In its example\-level form, all tokens in a sequence share the same merged expert\. This prevents the composition from adapting to individual tokens\. A composition derived from the complete sequence also uses future information, so it cannot be applied directly to causal prediction\. In its token\-level form, SMEAR produces a separate composition for each token\. This enables token\-specific adaptation but requires a full\-pool parameter merge per token, substantially increasing materialization cost\. Lory[Zhong et al\. \(2024\)](https://arxiv.org/html/2609.21346#bib.bib33)reduces the cost of this operation through causal segment\-level routing\. A composition derived from the preceding segment is reused by all tokens in the current segment\. During generation, a prompt\-conditioned composition is reused within the request\. This reduces the frequency of parameter merging, but all tokens within a segment share the same composition, limiting token\-level adaptation\.

DSFNet[Yu et al\. \(2025\)](https://arxiv.org/html/2609.21346#bib.bib29)performs input\-conditioned parameter merging to construct scenario\-specific network parameters, using gates to linearly combine parameter sets from disentangled factor\-scenario branches\. In contrast, IntBMoE preconstructs a finite set of input\-independent blocks and adapts to each token through sparse block routing\.

### 2\.4Hypernetworks and Parameter Generation

Hypernetworks[Ha et al\. \(2017\)](https://arxiv.org/html/2609.21346#bib.bib7)generate the parameters of a target network from learned or dynamically produced conditioning embeddings\. Directly generating full weight matrices can be memory\-intensive\. When the conditioning signal is input\-dependent, the parameters must also be regenerated as the signal changes\.

HyperMoE[Zhao et al\. \(2024\)](https://arxiv.org/html/2609.21346#bib.bib31)applies this idea to sparse MoE\. It encodes information associated with a token’s unselected experts and generates a HyperExpert for that token\. The HyperExpert is executed alongside the selected experts, providing an additional token\-conditioned path while the unselected experts remain inactive\. IntBMoE conditions its hypernetwork on a finite codebook of learned, input\-independent block embeddings\. The hypernetwork generates compact coefficients that combine a shared pool of expert bases, rather than directly generating full weight matrices\. Because the conditioning set is finite and input\-independent, the composed blocks can be precomputed and reused across routing decisions\. IntBMoE can therefore be viewed as a basis\-constrained hypernetwork with bounded parameter materialization\.

## 3Preliminaries

Let𝐱t∈ℝd\\mathbf\{x\}\_\{t\}\\in\\mathbb\{R\}^\{d\}denote the representation of input elementtt, and letfe​\(⋅,𝜽e\)f\_\{e\}\(\\cdot;\\bm\{\\theta\}\_\{e\}\)denote experteewith parameters𝜽e\\bm\{\\theta\}\_\{e\}\. A conventional sparse MoE maintainsEEexperts and uses a router to selectkkof them:

𝐫t=grouter​\(𝐱t\),𝐩t=softmax⁡\(𝐫t\),𝐲tsparse=∑e∈TopK⁡\(𝐫t,k\)pt,e​fe​\(𝐱t,𝜽e\)\.\\mathbf\{r\}\_\{t\}=g\_\{\\mathrm\{router\}\}\(\\mathbf\{x\}\_\{t\}\),\\qquad\\mathbf\{p\}\_\{t\}=\\operatorname\{softmax\}\(\\mathbf\{r\}\_\{t\}\),\\qquad\\mathbf\{y\}\_\{t\}^\{\\mathrm\{sparse\}\}=\\sum\_\{e\\in\\TopK\(\\mathbf\{r\}\_\{t\},k\)\}p\_\{t,e\}f\_\{e\}\(\\mathbf\{x\}\_\{t\};\\bm\{\\theta\}\_\{e\}\)\.\(1\)Here,grouter:ℝd→ℝEg\_\{\\mathrm\{router\}\}:\\mathbb\{R\}^\{d\}\\rightarrow\\mathbb\{R\}^\{E\}is the routing network,𝐫t\\mathbf\{r\}\_\{t\}contains its scores for theEEexperts, andpt,ep\_\{t,e\}is the routing probability assigned to expertee\.TopK⁡\(𝐫t,k\)\\TopK\(\\mathbf\{r\}\_\{t\},k\)returns the indices of thekkhighest\-scoring experts\. Sparse routing evaluates onlykkexpert networks for the token\.

A direct dense output mixture instead aggregates all expert outputs,

𝐲tdense=∑e=1Ept,e​fe​\(𝐱t,𝜽e\)\.\\mathbf\{y\}\_\{t\}^\{\\mathrm\{dense\}\}=\\sum\_\{e=1\}^\{E\}p\_\{t,e\}f\_\{e\}\(\\mathbf\{x\}\_\{t\};\\bm\{\\theta\}\_\{e\}\)\.\(2\)AllEEexperts can therefore affect the token, but every expert must be executed\. Slot\-based variants change the unit of expert computation from individual tokens to learned token mixtures, but still process the full set of expert\-associated slots\.

Parameter\-merging methods obtain full expert participation without separately executing every expert\. Letuudenote a routing unit, which may be a token, segment, or sequence\. We useau,ea\_\{u,e\}for the coefficient assigned to expertee\. These methods construct

𝜽¯u=∑e=1Eau,e​𝜽e,𝐲tmerge=f⁡\(𝐱t,𝜽¯u\)\.\\bar\{\\bm\{\\theta\}\}\_\{u\}=\\sum\_\{e=1\}^\{E\}a\_\{u,e\}\\bm\{\\theta\}\_\{e\},\\qquad\\mathbf\{y\}\_\{t\}^\{\\mathrm\{merge\}\}=f\(\\mathbf\{x\}\_\{t\};\\bar\{\\bm\{\\theta\}\}\_\{u\}\)\.\(3\)The token is processed by one composite expert, but each routing unit requires its own expert\-sized parameter set, constructed by combining all experts\. If a layer containsGGdistinct routing units, parameter synthesis costsO⁡\(G​E​\|𝜽\|\)O\(GE\|\\bm\{\\theta\}\|\), where\|𝜽\|\|\\bm\{\\theta\}\|denotes the size of one expert\. Here,*materialization*refers to constructing these derived parameter sets at runtime\.

## 4Method

### 4\.1Architecture Overview

Figure 2:Overview of IntBMoE\. The left panel shows independent IntBMoE modules integrated into a multi\-layer Transformer backbone\. The right panel shows block\-level expert parameter synthesis and token\-level Top\-kkblock routing and execution\.IntBMoE integrates into a Transformer backbone[Vaswani et al\. \(2017\)](https://arxiv.org/html/2609.21346#bib.bib25)by replacing its FFN sublayer\. Each IntBMoE module composes reusable multi\-layer blocks from shared expert pools and sparsely routes each token to a few blocks, as illustrated in Figure[2](https://arxiv.org/html/2609.21346#S4.F2)\. At the block level,KKlearned codebook embeddings produce composition coefficients\. These coefficients combine layer\-specific expert pools intoKKtoken\-independentLL\-layer blocks\. At the token level, a router independently selects the Top\-kkblocks for each token\. Each selected block applies block\-conditioned feature filtering and processes the token sequentially through itsLLcomposed experts\. The selected outputs are weighted by their routing probabilities and combined with the output of an always\-active shared SwiGLU expert\. The following subsections describe these components in detail\.

### 4\.2Block\-Level Parameter Synthesis

Block synthesis starts fromKKlearned codebook embeddings, one per candidate block\. A shared hypernetwork maps each embedding to value and gate composition coefficients, which combine the layer\-wise expert pools intoKKreusable multi\-layer blocks\. The token router then selects among these blocks, as described in the next subsection\.

Concretely, each IntBMoE module maintains a codebook ofKKlearned embeddings,

𝒞=\{𝐜b∈ℝdc\}b=1K\.\\mathcal\{C\}=\\\{\\mathbf\{c\}\_\{b\}\\in\\mathbb\{R\}^\{d\_\{c\}\}\\\}\_\{b=1\}^\{K\}\.\(4\)Each embedding𝐜b\\mathbf\{c\}\_\{b\}identifies oneLL\-layer block and is used to synthesize its parameters\. The codebook therefore defines theKKblocks available to the token router\.

We use column vectors throughout; linear maps act by left multiplication\. Letd0=dL=dd\_\{0\}=d\_\{L\}=d, withdℓd\_\{\\ell\}denoting the output dimension of internal layerℓ\\ell\. For eachℓ∈\{1,…,L\}\\ell\\in\\\{1,\\ldots,L\\\}, the module maintains a layer\-specific pool ofEEexpert bases shared across theKKblocks,

𝒫\(ℓ\)=\{\(We\(ℓ\),𝐛e\(ℓ\)\)\}e=1E,We\(ℓ\)∈ℝdℓ×dℓ−1,𝐛e\(ℓ\)∈ℝdℓ\.\\mathcal\{P\}^\{\(\\ell\)\}=\\\{\(W\_\{e\}^\{\(\\ell\)\},\\mathbf\{b\}\_\{e\}^\{\(\\ell\)\}\)\\\}\_\{e=1\}^\{E\},\\quad W\_\{e\}^\{\(\\ell\)\}\\in\\mathbb\{R\}^\{d\_\{\\ell\}\\times d\_\{\\ell\-1\}\},\\quad\\mathbf\{b\}\_\{e\}^\{\(\\ell\)\}\\in\\mathbb\{R\}^\{d\_\{\\ell\}\}\.\(5\)
Here,𝒫\(ℓ\)\\mathcal\{P\}^\{\(\\ell\)\}denotes the expert pool at internal layerℓ\\ell, and\(We\(ℓ\),𝐛e\(ℓ\)\)\(W\_\{e\}^\{\(\\ell\)\},\\mathbf\{b\}\_\{e\}^\{\(\\ell\)\}\)is itsee\-th expert\. These routed experts are distinct from the always\-active shared expert introduced later\. The pools are independent across internal layers and backbone layers\.

The block hypernetworkhϕh\_\{\\phi\}is shared across theKKcodebook entries within one IntBMoE module\. It takes only the learned block embedding𝐜b\\mathbf\{c\}\_\{b\}as input and uses a linear–LayerNorm–ReLU trunk followed by two linear output heads\. Its hidden representation is

𝐪b=ReLU⁡\(LN⁡\(A​𝐜b\+𝐚\)\),\\mathbf\{q\}\_\{b\}=\\operatorname\{ReLU\}\\\!\\left\(\\operatorname\{LN\}\(A\\mathbf\{c\}\_\{b\}\+\\mathbf\{a\}\)\\right\),\(6\)whereA∈ℝdh×dcA\\in\\mathbb\{R\}^\{d\_\{h\}\\times d\_\{c\}\}and𝐚∈ℝdh\\mathbf\{a\}\\in\\mathbb\{R\}^\{d\_\{h\}\}are the trainable weight matrix and bias of the input projection\. The resulting representation satisfies𝐪b∈ℝdh\\mathbf\{q\}\_\{b\}\\in\\mathbb\{R\}^\{d\_\{h\}\}\. Two linear output heads map𝐪b\\mathbf\{q\}\_\{b\}to the value and gate composition coefficients𝜶bv\\bm\{\\alpha\}\_\{b\}^\{v\}and𝜶bg\\bm\{\\alpha\}\_\{b\}^\{g\}, respectively:

𝜶bv=Av𝐪b\+𝐚v,𝜶bg=Ag𝐪b\+𝐚g,𝜶bv,𝜶bg∈ℝE\.\\bm\{\\alpha\}\_\{b\}^\{v\}=A\_\{v\}\\mathbf\{q\}\_\{b\}\+\\mathbf\{a\}\_\{v\},\\qquad\\bm\{\\alpha\}\_\{b\}^\{g\}=A\_\{g\}\\mathbf\{q\}\_\{b\}\+\\mathbf\{a\}\_\{g\},\\qquad\\bm\{\\alpha\}\_\{b\}^\{v\},\\bm\{\\alpha\}\_\{b\}^\{g\}\\in\\mathbb\{R\}^\{E\}\.\(7\)Here,Av,Ag∈ℝE×dhA\_\{v\},A\_\{g\}\\in\\mathbb\{R\}^\{E\\times d\_\{h\}\}and𝐚v,𝐚g∈ℝE\\mathbf\{a\}\_\{v\},\\mathbf\{a\}\_\{g\}\\in\\mathbb\{R\}^\{E\}are the trainable weights and biases of the two output heads\.

The coefficients are not normalized by softmax or sigmoid\. They may be negative and need not sum to one, allowing composition over the linear span of the expert bases rather than restricting it to their convex hull\. We instead apply a variance\-preserving factor of1/E1/\\sqrt\{E\}when composing the expert bases, keeping the scale of the composed parameters approximately stable as the expert pool grows\. For pathp∈\{v,g\}p\\in\\\{v,g\\\}, the composed parameters at every internal layer are

Wb,p\(ℓ\)=1E​∑e=1Eαb,ep​We\(ℓ\),𝐛b,p\(ℓ\)=1E​∑e=1Eαb,ep​𝐛e\(ℓ\)\.W\_\{b,p\}^\{\(\\ell\)\}=\\frac\{1\}\{\\sqrt\{E\}\}\\sum\_\{e=1\}^\{E\}\\alpha^\{p\}\_\{b,e\}W\_\{e\}^\{\(\\ell\)\},\\qquad\\mathbf\{b\}\_\{b,p\}^\{\(\\ell\)\}=\\frac\{1\}\{\\sqrt\{E\}\}\\sum\_\{e=1\}^\{E\}\\alpha^\{p\}\_\{b,e\}\\mathbf\{b\}\_\{e\}^\{\(\\ell\)\}\.\(8\)
Thus, the hypernetwork produces one recipe\(𝜶bv,𝜶bg\)\(\\bm\{\\alpha\}\_\{b\}^\{v\},\\bm\{\\alpha\}\_\{b\}^\{g\}\)for blockbb\. At each internal layer, this recipe combines that layer’s expert pool into separate value and gate parameter sets\. Repeating this process over allLLinternal layers constructs the completeLL\-layer block\. Becausehϕh\_\{\\phi\}does not depend on token representations, theKKcomposed blocks can be precomputed and shared across all tokens\.

### 4\.3Token\-Level Block Routing

Routing is performed independently for each token\. For tokentt, the router uses its representation𝐱t\\mathbf\{x\}\_\{t\}to produce one score for each block and selects the Top\-kkblocks:

𝐫t=gblock​\-​router​\(𝐱t\),ℬt=TopK⁡\(𝐫t,k\),\\mathbf\{r\}\_\{t\}=g\_\{\\mathrm\{block\\text\{\-\}router\}\}\(\\mathbf\{x\}\_\{t\}\),\\qquad\\mathcal\{B\}\_\{t\}=\\TopK\(\\mathbf\{r\}\_\{t\},k\),\(9\)whereℬt\\mathcal\{B\}\_\{t\}is the set of indices of thekkblocks selected for tokentt\. The block routergblock​\-​router:ℝd→ℝKg\_\{\\mathrm\{block\\text\{\-\}router\}\}:\\mathbb\{R\}^\{d\}\\rightarrow\\mathbb\{R\}^\{K\}outputs a score vector𝐫t∈ℝK\\mathbf\{r\}\_\{t\}\\in\\mathbb\{R\}^\{K\}and is implemented as a two\-layer ReLU MLP by default\. The routing weight of a selected blockbbis

πt,b=softmax​\(𝐫t\)b,b∈ℬt\.\\pi\_\{t,b\}=\\mathrm\{softmax\}\(\\mathbf\{r\}\_\{t\}\)\_\{b\},\\qquad b\\in\\mathcal\{B\}\_\{t\}\.\(10\)

### 4\.4Block\-Conditioned Feature Filtering

Before entering a selected block, the token representation𝐱t\\mathbf\{x\}\_\{t\}is filtered using that block’s codebook embedding𝐜b\\mathbf\{c\}\_\{b\}:

𝐦t,b=sigmoid⁡\(𝐖f​\[𝐱t∥𝐜b\]\+𝐛f\),𝐳t,b\(0\)=𝐱t⊙𝐦t,b\.\\mathbf\{m\}\_\{t,b\}=\\operatorname\{sigmoid\}\\left\(\\mathbf\{W\}\_\{f\}\[\\mathbf\{x\}\_\{t\}\\mathbin\{\\\|\}\\mathbf\{c\}\_\{b\}\]\+\\mathbf\{b\}\_\{f\}\\right\),\\qquad\\mathbf\{z\}^\{\(0\)\}\_\{t,b\}=\\mathbf\{x\}\_\{t\}\\odot\\mathbf\{m\}\_\{t,b\}\.\(11\)Here,\[⋅∥⋅\]\[\\cdot\\mathbin\{\\\|\}\\cdot\]denotes vertical concatenation of column vectors,𝐖f∈ℝd×\(d\+dc\)\\mathbf\{W\}\_\{f\}\\in\\mathbb\{R\}^\{d\\times\(d\+d\_\{c\}\)\}and𝐛f∈ℝd\\mathbf\{b\}\_\{f\}\\in\\mathbb\{R\}^\{d\}are the learnable parameters of a feature\-filtering layer shared across blocks, and𝐦t,b∈\(0,1\)d\\mathbf\{m\}\_\{t,b\}\\in\(0,1\)^\{d\}is a soft feature\-wise mask\. The filtered representation𝐳t,b\(0\)∈ℝd\\mathbf\{z\}^\{\(0\)\}\_\{t,b\}\\in\\mathbb\{R\}^\{d\}is the input to the first internal layer of blockbbfor tokentt\. This gives different blocks distinct views of the same token before their composed transformations are applied\.

### 4\.5Dual\-Path Residual Gating

Although each parameter path in equation[8](https://arxiv.org/html/2609.21346#S4.E8)is composed linearly from the expert bases, DPRG introduces a nonlinear interaction between two independently composed paths\. It couples the value and gate paths through residual multiplicative modulation\.

For internal layerℓ\\elland input𝐳t,b\(ℓ−1\)\\mathbf\{z\}^\{\(\\ell\-1\)\}\_\{t,b\}, the DPRG transformation is

𝐯t,b\(ℓ\)\\displaystyle\\mathbf\{v\}^\{\(\\ell\)\}\_\{t,b\}=Wb,v\(ℓ\)​𝐳t,b\(ℓ−1\)\+𝐛b,v\(ℓ\),\\displaystyle=W\_\{b,v\}^\{\(\\ell\)\}\\mathbf\{z\}^\{\(\\ell\-1\)\}\_\{t,b\}\+\\mathbf\{b\}\_\{b,v\}^\{\(\\ell\)\},\(12\)𝐠t,b\(ℓ\)\\displaystyle\\mathbf\{g\}^\{\(\\ell\)\}\_\{t,b\}=RMSNorm⁡\(Wb,g\(ℓ\)​𝐳t,b\(ℓ−1\)\+𝐛b,g\(ℓ\)\),\\displaystyle=\\operatorname\{RMSNorm\}\\\!\\left\(W\_\{b,g\}^\{\(\\ell\)\}\\mathbf\{z\}^\{\(\\ell\-1\)\}\_\{t,b\}\+\\mathbf\{b\}\_\{b,g\}^\{\(\\ell\)\}\\right\),\(13\)𝐳~t,b\(ℓ\)\\displaystyle\\widetilde\{\\mathbf\{z\}\}^\{\(\\ell\)\}\_\{t,b\}=𝐯t,b\(ℓ\)⊙\(𝟏\+λ​SiLU⁡\(𝐠t,b\(ℓ\)\)\)\.\\displaystyle=\\mathbf\{v\}^\{\(\\ell\)\}\_\{t,b\}\\odot\\left\(\\mathbf\{1\}\+\\lambda\\,\\operatorname\{SiLU\}\(\\mathbf\{g\}^\{\(\\ell\)\}\_\{t,b\}\)\\right\)\.\(14\)LayerNorm is applied between consecutive internal layers:

𝐳t,b\(ℓ\)=\{LN⁡\(𝐳~t,b\(ℓ\)\),ℓ<L,𝐳~t,b\(ℓ\),ℓ=L\.\\mathbf\{z\}^\{\(\\ell\)\}\_\{t,b\}=\\begin\{cases\}\\operatorname\{LN\}\(\\widetilde\{\\mathbf\{z\}\}^\{\(\\ell\)\}\_\{t,b\}\),&\\ell<L,\\\\ \\widetilde\{\\mathbf\{z\}\}^\{\(\\ell\)\}\_\{t,b\},&\\ell=L\.\\end\{cases\}\(15\)
Thus, each layer processes the token state produced by the preceding layer, andFb​\(𝐱t\)=𝐳t,b\(L\)F\_\{b\}\(\\mathbf\{x\}\_\{t\}\)=\\mathbf\{z\}^\{\(L\)\}\_\{t,b\}is the final output of blockbb\. The learnable residual scaleλ\\lambdais shared across the internal layers of an IntBMoE module\. DPRG increases the expressiveness of each composed block by coupling two compositions of the same expert pool, while adding only a constant factor to its parameter\-synthesis and execution costs\.

### 4\.6Output Aggregation and Shared Expert

Selected block outputs are aggregated using routing probabilities:

𝐲troute=∑b∈ℬtπt,b​Fb​\(𝐱t\)\.\\mathbf\{y\}^\{\\mathrm\{route\}\}\_\{t\}=\\sum\_\{b\\in\\mathcal\{B\}\_\{t\}\}\\pi\_\{t,b\}F\_\{b\}\(\\mathbf\{x\}\_\{t\}\)\.\(16\)We add an always\-active shared SwiGLU expertSSto model components that need not be differentiated by block routing:

𝐲t=𝐲troute\+S⁡\(𝐱t\)\.\\mathbf\{y\}\_\{t\}=\\mathbf\{y\}^\{\\mathrm\{route\}\}\_\{t\}\+S\(\\mathbf\{x\}\_\{t\}\)\.\(17\)
The shared path captures common information, allowing the routed blocks to focus on transformations that benefit from token\-dependent selection\.

### 4\.7Complexity and Inference Caching

Consider an IntBMoE module processingTTvalid tokens\. Let each block containLLlayers, where layerℓ\\ellmapsdℓ−1d\_\{\\ell\-1\}todℓd\_\{\\ell\}, and defineD=∑ℓ=1Ldℓ−1​dℓD=\\sum\_\{\\ell=1\}^\{L\}d\_\{\\ell\-1\}d\_\{\\ell\}as the total matrix size of one multi\-layer expert basis\. Composing allKKblocks fromEEbases costsO⁡\(K​E​D\)O\(KED\), and executing thekkselected blocks for all tokens costsO⁡\(T​k​D\)O\(TkD\)\. Ignoring lower\-order routing operations and the constant factor from the two DPRG paths, the total uncached composition\-and\-routed\-execution cost isO⁡\(K​E​D\+T​k​D\)O\(KED\+TkD\)\. The corresponding amortized per\-token cost isO⁡\(K​E​D/T\+k​D\)O\(KED/T\+kD\)\. The composition term remains linear inEE, but it is incurred once forKKreusable blocks rather than once per token\.

Block composition depends only on the learned block embeddings, hypernetwork, and expert bases\. Once the model parameters are fixed for inference, all composed block parameters can be constructed once and cached, removing theO⁡\(K​E​D\)O\(KED\)composition term from request\-time computation\. The dominant routed\-block cost is thenO⁡\(T​k​D\)O\(TkD\), in addition to the router, feature filter, and shared expert\. Consequently, for fixedKKandkk, the request\-time computation of cached IntBMoE does not increase with the expert\-pool sizeEE\.

## 5Visual Experiments

### 5\.1Experimental Setup

ImageNet\-1K[Russakovsky et al\. \(2015\)](https://arxiv.org/html/2609.21346#bib.bib20)is our primary benchmark\. It contains 1\.28 million training images and 50,000 validation images from 1,000 classes\. All methods use an eight\-layer DeiT\-Tiny\-style backbone[Touvron et al\. \(2021\)](https://arxiv.org/html/2609.21346#bib.bib23)and are trained from scratch on the official training split\. We report the mean Top\-1 and Top\-5 validation accuracy over three random seeds\.

We compare IntBMoE with the dense backbone and two families of MoE methods\. Sparse\-routing baselines include Switch Transformer[Fedus et al\. \(2022\)](https://arxiv.org/html/2609.21346#bib.bib4), DeepSeek\-V3 MoE[Liu et al\. \(2024\)](https://arxiv.org/html/2609.21346#bib.bib12), ReMoE[Wang et al\. \(2024\)](https://arxiv.org/html/2609.21346#bib.bib26), V\-MoE[Riquelme et al\. \(2021\)](https://arxiv.org/html/2609.21346#bib.bib19), Expert Choice[Zhou et al\. \(2022\)](https://arxiv.org/html/2609.21346#bib.bib34), DynMoE[Guo et al\. \(2025\)](https://arxiv.org/html/2609.21346#bib.bib6), and MASS[Park & Park \(2026\)](https://arxiv.org/html/2609.21346#bib.bib17)\. Dense\-participation baselines include Soft MoE[Puigcerver et al\. \(2024\)](https://arxiv.org/html/2609.21346#bib.bib18), SMEAR[Muqeeth et al\. \(2023\)](https://arxiv.org/html/2609.21346#bib.bib14), Lory[Zhong et al\. \(2024\)](https://arxiv.org/html/2609.21346#bib.bib33), andμ\\muMoE[Oldfield et al\. \(2024\)](https://arxiv.org/html/2609.21346#bib.bib16)\. Complete experimental configurations and implementation details are provided in Appendix[A](https://arxiv.org/html/2609.21346#A1)\.

### 5\.2Experimental Results

Table 1:Main results on ImageNet\-1K\.↑\\uparrowindicates that higher values are better\. Best results are highlighted with ablue background, and second\-best results are shown inblue\. Params denotes the total number of trainable parameters in the model; Activated Params denotes the number of trainable parameters activated as a single patch token passes through the entire model\. FLOPs denotes the inference computational cost per image, measured with a batch size of 1\.#### Main results\.

Table[1](https://arxiv.org/html/2609.21346#S5.T1)presents the primary comparison\. IntBMoE achieves 73\.76% Top\-1 and 91\.48% Top\-5 accuracy\. Relative to the dense backbone, these scores represent gains of 7\.36 and 3\.79 percentage points, respectively\. IntBMoE also outperforms all sparse\-routing and dense\-participation MoE baselines, exceeding the strongest competitor, SMEAR, by 1\.98 Top\-1 percentage points and 1\.15 Top\-5 percentage points\.

All MoE methods use comparable total parameter budgets of approximately 24M\. IntBMoE activates 23\.111M parameters because the full expert pool participates in constructing its reusable blocks\. Despite full expert participation, each token executes only its selected blocks\. The total inference cost is 4\.063 GFLOPs per image without caching and 3\.457 GFLOPs per image with caching, measured with a batch size of 1\.

#### Ablation studies\.

To assess the contribution of each component, Table[2](https://arxiv.org/html/2609.21346#S5.T2)reports ablations on ImageNet\-1K\. The*1\-Layer*variant replaces each two\-layer block with a single layer and enlarges its expert pool to preserve the total number of expert parameters\.*Fixedλ\\lambda*makesλ\\lambdanon\-learnable and sets it to 1\.*w/o Gate*removes the composed gate path and multiplicative modulation, reducing equation[14](https://arxiv.org/html/2609.21346#S4.E14)to𝐳~t,b\(ℓ\)=𝐯t,b\(ℓ\)\\widetilde\{\\mathbf\{z\}\}^\{\(\\ell\)\}\_\{t,b\}=\\mathbf\{v\}^\{\(\\ell\)\}\_\{t,b\}\.*w/o Shared*removes the shared SwiGLU expert, while*w/o Filter*bypasses block\-conditioned feature filtering by setting𝐳t,b\(0\)=𝐱t\\mathbf\{z\}^\{\(0\)\}\_\{t,b\}=\\mathbf\{x\}\_\{t\}\. Finally,*Softmax Coeff\.*replaces our unconstrained, variance\-scaled coefficient formulation with separately softmax\-normalized value and gate coefficients\.

Table 2:Component ablations on ImageNet\-1K\.Removing the gate path causes the largest performance drop, followed by collapsing each two\-layer block into a parameter\-matched single layer\. The shared expert and feature filter provide smaller but consistent gains\. Both fixingλ\\lambdaand softmax\-normalizing the composition coefficients underperform the full formulation\.

### 5\.3Sensitivity to Architectural Hyperparameters

We vary the codebook sizeKK, the number of selected blockskk, the expert\-pool sizeEEat each internal layer, and the block depthLLon ImageNet\-1K\. We change one hyperparameter at a time from the default configuration\(K,E,k,L\)=\(8,16,2,2\)\(K,E,k,L\)=\(8,16,2,2\)\. For theK=1K=1setting, we setk=1k=1\. Figure[3](https://arxiv.org/html/2609.21346#S5.F3)reports the resulting Top\-1 accuracies\.

Figure 3:Sensitivity of IntBMoE to architectural hyperparameters on ImageNet\-1K\. Each panel reports Top\-1 accuracy while varying one hyperparameter; higher values are better\. All panels use the same vertical scale\. Orange points and gray horizontal lines mark the default configuration\(K,E,k,L\)=\(8,16,2,2\)\(K,E,k,L\)=\(8,16,2,2\)and its accuracy, respectively\.Increasing eitherKKorEEimproves accuracy, but further gains are small beyond their default values\. ExpandingKKfrom 8 to 32 increases Top\-1 accuracy by only 0\.07 percentage points, and expandingEEfrom 16 to 64 increases it by only 0\.15 percentage points\. Increasingkkconsistently improves Top\-1 accuracy\. We usek=2k=2to obtain a substantial gain overk=1k=1while limiting per\-token computation\. For block depth,L=2L=2achieves the highest Top\-1 accuracy\.

### 5\.4Effective Full\-Pool Expert Participation

Although every block is constructed from all expert bases, this design alone does not show that every expert makes a meaningful contribution to model performance\. We therefore evaluate the contribution of each expert basis on ImageNet\-1K by removing it from all block compositions in one MoE layer at a time\. We examine all 16 expert bases in Layers 0, 2, 4, and 6, producing 64 removal settings in total, and evaluate the original checkpoint without retraining\. Removing an expert may reduce the magnitude of the composed weights simply because fewer expert bases remain\. To isolate the contribution of the removed expert, we compensate for this scale change by replacing1/E1/\\sqrt\{E\}with1/E−11/\\sqrt\{E\-1\}\.

Figure 4:Top\-1 accuracy drop on ImageNet\-1K after removing individual expert bases from one MoE layer at a time\. The vertical axis uses a logarithmic scale\.Figure[4](https://arxiv.org/html/2609.21346#S5.F4)shows that removing any of the examined expert bases reduces Top\-1 accuracy\. The mean drops at Layers 0, 2, 4, and 6 are 1\.60, 0\.49, 1\.17, and 0\.78 percentage points, respectively\. Contributions are highly uneven in Layer 0, where removing E1, E12, or E15 lowers accuracy by 4\.10, 4\.99, and 9\.15 percentage points, while the effects of most other bases are much smaller\. The removal effects become more balanced in later layers, ranging from 0\.26 to 0\.74 percentage points in Layer 2, 0\.60 to 1\.95 percentage points in Layer 4, and 0\.45 to 1\.04 percentage points in Layer 6\. This suggests that the earliest MoE layer relies strongly on a few expert bases, whereas later layers distribute useful contributions more evenly across their pools\. Even the smallest decrease is 0\.26 percentage points, indicating that every expert basis contributes to the model\. Together, the results support the conclusion that the model makes effective use of the full expert pool throughout its MoE layers\.

### 5\.5Block Routing and Block\-Specific Composition Coefficients

Figure 5:Class\-conditioned block routing for the same eight representative ImageNet\-1K classes at Transformer layers 4 and 6\. Each cell shows the token\-assignment proportion for a class and block minus the uniform allocation of 12\.5%, in percentage points\. Positive \(red\) and negative \(blue\) values indicate above\- and below\-uniform use of a block, respectively\. Each Top\-kkselection is counted as one token–block assignment\.#### Class\-conditioned block allocation\.

On ImageNet\-1K, we compute the proportion of patch\-token routing assignments received by each block within each ground\-truth class\. Figure[5](https://arxiv.org/html/2609.21346#S5.F5)shows deviations from the uniform 12\.5% allocation for the same eight representative classes at layers 4 and 6\.

Different image classes favor different blocks within the same layer, and these preferences often change with depth\. For example, at layer 4, hen favorsb4b\_\{4\}, whereas Boston bull favorsb6b\_\{6\}\. At layer 6, their preferred blocks shift tob0b\_\{0\}andb3b\_\{3\}, respectively\. These patterns show that routing is both class\-conditioned and depth\-dependent\.

Figure 6:Expert\-composition coefficients learned by the eight blocks in Transformer layers 0, 2, 4, and 6 on ImageNet\-1K, ordered from shallow to deep along the backbone\. Rows represent blocks and columns represent expert bases\. Each block’s coefficient vector isℓ2\\ell\_\{2\}\-normalized only for visualization\. Red and blue denote positive and negative coefficients, respectively\.
#### Block\-specific composition coefficients\.

We visualize the expert\-composition coefficients generated for every block on ImageNet\-1K, separately for the value and gate paths\. As shown in Figure[6](https://arxiv.org/html/2609.21346#S5.F6), blocks within the same layer assign different positive and negative weights to the same expert bases\. The patterns also vary across layers and between the two DPRG paths\. Because all blocks in a layer compose the same ordered expert pools, these differences show that the codebook entries learn block\-specific composition recipes rather than collapsing to identical fusion coefficients\.

The degree of recipe differentiation increases with network depth\. For the value path, the mean pairwise cosine similarity between block recipes decreases monotonically from 0\.796 at layer 0 to 0\.079, 0\.019, and 0\.010 at layers 2, 4, and 6, respectively\. Gate\-path similarity follows the same trend, decreasing from 0\.726 to 0\.126, 0\.083, and 0\.043\. Thus, the shallowest layer largely shares an expert\-composition pattern across blocks, whereas deeper layers construct substantially more diverse recipes\. This trend is consistent with shallow visual features being more general and deeper representations benefiting from finer block\-specific expert composition\.

### 5\.6Inference Cost and Caching

Figure[7](https://arxiv.org/html/2609.21346#S5.F7)shows how the inference memory and FLOPs of cached and uncached IntBMoE change as the number of experts increases in the ImageNet\-1K setup\. Without caching, peak inference memory increases from54\.5754\.57to627\.83627\.83MB and inference cost increases from3\.4953\.495to8\.3058\.305GFLOPs as the expert\-pool sizeEEgrows from 1 to 128\. With caching, they remain constant at104\.33104\.33MB and3\.4573\.457GFLOPs, respectively\. Caching introduces a fixed memory cost for the precomposed blocks, so it uses more memory for small expert pools, but becomes more memory\-efficient from 16 experts onward\. These results show that caching decouples request\-time memory and computation from the size of the expert pool\. During training, all images in a minibatch share the same composed blocks\. The composition is therefore performed once per batch, and its cost is amortized across the images in that batch\.

Figure 7:Peak inference memory and inference FLOPs of cached and uncached IntBMoE as the number of expertsEEincreases\. Both quantities are measured for inference on a single image with a batch size of 1 using the ImageNet\-1K model configuration\. The horizontal axis uses alog2\\log\_\{2\}scale\.

## 6Generalization to Language Modeling and Recommendation

#### Tasks and datasets\.

We use MiniPile[Kaddour \(2023\)](https://arxiv.org/html/2609.21346#bib.bib9)and IntTravel[Yan et al\. \(2026\)](https://arxiv.org/html/2609.21346#bib.bib28)to test whether the same architecture generalizes to language modeling and sequential recommendation\. MiniPile uses an 18\-layer Llama\-style causal Transformer[Touvron et al\. \(2023\)](https://arxiv.org/html/2609.21346#bib.bib24), while IntTravel uses a causal Transformer\. MiniPile is a 6\-GB subset of the deduplicated Pile\. We evaluate causal language modeling on MiniPile using token\-level cross\-entropy and perplexity\. IntTravel contains 162\.8 million users, 7\.3 million POIs, and 4\.13 billion interactions\. We use its “Where” task to predict the destination POI of the next journey and report HR@1, HR@5, and NDCG@5\.

#### Experimental results\.

Table[3](https://arxiv.org/html/2609.21346#S6.T3)compares IntBMoE with the dense backbone, sparse\-routing methods, and dense\-participation methods on MiniPile and IntTravel\. For these causal prediction tasks, we make Soft MoE causal by constructing each slot using only the current and preceding tokens\. SMEAR uses token\-level composition\. Complete model, training, routing, and baseline configurations are provided in Appendix[A](https://arxiv.org/html/2609.21346#A1)\.

Table 3:Generalization results on MiniPile and IntTravel\.↑\\uparrowindicates that higher values are better, and↓\\downarrowindicates that lower values are better\. Best results per column are highlighted with ablue background, and second\-best results are shown inblue\.On MiniPile, IntBMoE achieves a test loss of 2\.6802 and a PPL of 14\.5878\. Relative toμ\\muMoE \(CP\), the strongest baseline, IntBMoE reduces PPL by 2\.9%\. Relative to the dense backbone, the reduction is 12\.4%\. On IntTravel, IntBMoE achieves the highest mean HR@1, HR@5, and NDCG@5\. These results support generalization to autoregressive language modeling and sequential recommendation\.

Table 4:Component ablations on MiniPile and IntTravel\.Table[4](https://arxiv.org/html/2609.21346#S6.T4)reports component ablations on both MiniPile and IntTravel\. The gate path has the largest impact on MiniPile and is among the most influential components on IntTravel\. The parameter\-matched 1\-Layer variant also underperforms the full model, showing that the two\-layer block structure provides benefits beyond expert parameter count\. The shared expert yields smaller but consistent gains\. Fixingλ\\lambdaand removing feature filtering hurt both autoregressive tasks, while the unconstrained, variance\-scaled coefficients consistently outperform softmax\-normalized composition\. These results confirm that the main design choices remain effective beyond vision\.

#### Online evaluation and deployment\.

We further evaluate cached IntBMoE in AMap’s POI recommendation service for the initial map screen displayed when a user opens the app\. The production generative recommender predicts the Top\-10 POIs that the user is likely to visit and uses them to determine the map viewport and zoom level\. Because this service requires a response within60​ms60\\,\\mathrm\{ms\}, we precompute and cache the input\-independent composed block parameters, removing block synthesis from the request\-time path\.

We conducted a one\-week online A/B experiment in which the control group used the existing production model without an MoE module and the treatment group used cached IntBMoE\. The experiment handled approximately 5,000 queries per second\. Running on Alibaba T\-Head PPUs, cached IntBMoE achieved an average latency of19​ms19\\,\\mathrm\{ms\}and a P99 latency of38​ms38\\,\\mathrm\{ms\}, remaining within the serving budget while delivering a 2\.4% relative improvement in UVCTR\. Following the experiment, IntBMoE was fully deployed to serve production traffic\.

## 7Conclusion

Existing MoE designs couple expert participation, execution, and materialization: sparse routing limits participation, dense output mixing increases execution, and parameter merging increases materialization\. We introduced IntBMoE to resolve this coupling, allowing every token to benefit from the full expert pool without dense execution or unbounded parameter materialization\. IntBMoE separates reusable block construction from token\-level execution\. Each candidate block composes the full pool of expert bases, while Top\-kkrouting executes only a few blocks per token, and a finite codebook bounds the number of materialized blocks independently of the number of routing decisions\. Within each block, DPRG couples independently composed value and gate paths through residual multiplicative interactions\. On the primary ImageNet\-1K benchmark, IntBMoE consistently outperforms the evaluated sparse and dense\-participation MoE baselines\. Analysis on ImageNet\-1K shows that different image classes induce distinct block\-allocation profiles\. It also shows that blocks within each layer learn distinct expert\-composition recipes\. Auxiliary experiments on MiniPile and IntTravel further demonstrate generalization to language modeling and sequential recommendation\. For efficient serving, the composed blocks can be cached to remove block synthesis from the request\-time path\. The full deployment of IntBMoE in AMap’s latency\-critical generative recommendation system further demonstrates its practical utility\.

## References

- Dai et al\. \(2024\)Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al\.Deepseekmoe: Towards ultimate expert specialization in mixture\-of\-experts language models\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 1280–1297, 2024\.
- Deng et al\. \(2025\)Jiaxin Deng, Shiyao Wang, Kuo Cai, Lejian Ren, Qigen Hu, Weifeng Ding, Qiang Luo, and Guorui Zhou\.Onerec: Unifying retrieve and rank with generative recommender and iterative preference alignment\.*arXiv preprint arXiv:2502\.18965*, 2025\.
- Fan et al\. \(2022\)Zhiwen Fan, Rishov Sarkar, Ziyu Jiang, Tianlong Chen, Kai Zou, Yu Cheng, Cong Hao, Zhangyang Wang, et al\.M3vit: Mixture\-of\-experts vision transformer for efficient multi\-task learning with model\-accelerator co\-design\.*Advances in Neural Information Processing Systems*, 35:28441–28457, 2022\.
- Fedus et al\. \(2022\)William Fedus, Barret Zoph, and Noam Shazeer\.Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity\.*Journal of Machine Learning Research*, 23\(120\):1–39, 2022\.
- Gu et al\. \(2025\)Hao Gu, Wei Li, Lujun Li, Qiyuan Zhu, Mark Lee, Shengjie Sun, Wei Xue, and Yike Guo\.Delta decompression for moe\-based llms compression\.*arXiv preprint arXiv:2502\.17298*, 2025\.
- Guo et al\. \(2025\)Yongxin Guo, Zhenglin Cheng, Xiaoying Tang, Zhaopeng Tu, and Tao Lin\.Dynamic mixture of experts: An auto\-tuning approach for efficient transformer models\.In*International Conference on Learning Representations*, volume 2025, pp\. 79643–79672, 2025\.
- Ha et al\. \(2017\)David Ha, Andrew M Dai, and Quoc V Le\.Hypernetworks\.In*International conference on learning representations*, 2017\.
- Jiang et al\. \(2024\)Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al\.Mixtral of experts\.*arXiv preprint arXiv:2401\.04088*, 2024\.
- Kaddour \(2023\)Jean Kaddour\.The minipile challenge for data\-efficient language models\.*arXiv preprint arXiv:2304\.08442*, 2023\.
- Lepikhin et al\. \(2020\)Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen\.Gshard: Scaling giant models with conditional computation and automatic sharding\.*arXiv preprint arXiv:2006\.16668*, 2020\.
- Li et al\. \(2024\)Jiachen Li, Xinyao Wang, Sijie Zhu, Chia\-Wen Kuo, Lu Xu, Fan Chen, Jitesh Jain, Humphrey Shi, and Longyin Wen\.Cumo: Scaling multimodal llm with co\-upcycled mixture\-of\-experts\.*Advances in Neural Information Processing Systems*, 37:131224–131246, 2024\.
- Liu et al\. \(2024\)Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al\.Deepseek\-v3 technical report\.*arXiv preprint arXiv:2412\.19437*, 2024\.
- Ma et al\. \(2018\)Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi\.Modeling task relationships in multi\-task learning with multi\-gate mixture\-of\-experts\.In*Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining*, pp\. 1930–1939, 2018\.
- Muqeeth et al\. \(2023\)Mohammed Muqeeth, Haokun Liu, and Colin Raffel\.Soft merging of experts with adaptive routing\.*arXiv preprint arXiv:2306\.03745*, 2023\.
- Mustafa et al\. \(2022\)Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby\.Multimodal contrastive learning with limoe: the language\-image mixture of experts\.*Advances in Neural Information Processing Systems*, 35:9564–9576, 2022\.
- Oldfield et al\. \(2024\)James Oldfield, Markos Georgopoulos, Grigorios G Chrysos, Christos Tzelepis, Yannis Panagakis, Mihalis A Nicolaou, Jiankang Deng, and Ioannis Patras\.Multilinear mixture of experts: Scalable expert specialization through factorization\.*Advances in Neural Information Processing Systems*, 37:53022–53063, 2024\.
- Park & Park \(2026\)Sumin Park and Noseong Park\.How many experts are enough? towards optimal semantic specialization for mixture\-of\-experts\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 40, pp\. 24792–24800, 2026\.
- Puigcerver et al\. \(2024\)Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, and Neil Houlsby\.From sparse to soft mixtures of experts\.In*International Conference on Learning Representations*, volume 2024, pp\. 28435–28445, 2024\.
- Riquelme et al\. \(2021\)Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby\.Scaling vision with sparse mixture of experts\.*Advances in Neural Information Processing Systems*, 34:8583–8595, 2021\.
- Russakovsky et al\. \(2015\)Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al\.Imagenet large scale visual recognition challenge\.*International journal of computer vision*, 115\(3\):211–252, 2015\.
- Shazeer et al\. \(2017\)Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean\.Outrageously large neural networks: The sparsely\-gated mixture\-of\-experts layer\.*arXiv preprint arXiv:1701\.06538*, 2017\.
- Tang et al\. \(2020\)Hongyan Tang, Junning Liu, Ming Zhao, and Xudong Gong\.Progressive layered extraction \(ple\): A novel multi\-task learning \(mtl\) model for personalized recommendations\.In*Proceedings of the 14th ACM conference on recommender systems*, pp\. 269–278, 2020\.
- Touvron et al\. \(2021\)Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou\.Training data\-efficient image transformers & distillation through attention\.In*International conference on machine learning*, pp\. 10347–10357\. PMLR, 2021\.
- Touvron et al\. \(2023\)Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie\-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al\.Llama: Open and efficient foundation language models\.*arXiv preprint arXiv:2302\.13971*, 2023\.
- Vaswani et al\. \(2017\)Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin\.Attention is all you need\.*Advances in neural information processing systems*, 30, 2017\.
- Wang et al\. \(2024\)Ziteng Wang, Jun Zhu, and Jianfei Chen\.Remoe: Fully differentiable mixture\-of\-experts with relu routing\.*arXiv preprint arXiv:2412\.14711*, 2024\.
- Xue et al\. \(2023\)Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo\.Raphael: Text\-to\-image generation via large mixture of diffusion paths\.*Advances in Neural Information Processing Systems*, 36:41693–41706, 2023\.
- Yan et al\. \(2026\)Huimin Yan, Longfei Xu, Junjie Sun, Zheng Liu, Wei Luo, Kaikui Liu, and Xiangxiang Chu\.Inttravel: A real\-world dataset and generative framework for integrated multi\-task travel recommendation\.*arXiv preprint arXiv:2602\.11664*, 2026\.
- Yu et al\. \(2025\)Jiahao Yu, Yihai Duan, Longfei Xu, Chao Chen, Shuliang Liu, Kaikui Liu, Fan Yang, Xiangxiang Chu, and Ning Guo\.Dsfnet: Learning disentangled scenario factorization for multi\-scenario route ranking\.In*Companion Proceedings of the ACM on Web Conference 2025*, pp\. 567–576, 2025\.
- Zasada et al\. \(2026\)Mikołaj Zasada, Łukasz Struski, Jacek Tabor, and Marcin Kurdziel\.Softmoe: Soft differentiable routing for mixture\-of\-experts in llms\.*arXiv preprint arXiv:2606\.17952*, 2026\.
- Zhao et al\. \(2024\)Hao Zhao, Zihan Qiu, Huijia Wu, Zili Wang, Zhaofeng He, and Jie Fu\.Hypermoe: Towards better mixture of experts via transferring among experts\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 10605–10618, 2024\.
- Zheng et al\. \(2025\)Youwei Zheng, Yuxi Ren, Xin Xia, Xuefeng Xiao, and Xiaohua Xie\.Dense2moe: Restructuring diffusion transformer to moe for efficient text\-to\-image generation\.In*2025 IEEE/CVF International Conference on Computer Vision \(ICCV\)*, pp\. 18661–18670\. IEEE, 2025\.
- Zhong et al\. \(2024\)Zexuan Zhong, Mengzhou Xia, Danqi Chen, and Mike Lewis\.Lory: Fully differentiable mixture\-of\-experts for autoregressive language model pre\-training\.*arXiv preprint arXiv:2405\.03133*, 2024\.
- Zhou et al\. \(2022\)Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al\.Mixture\-of\-experts with expert choice routing\.*Advances in Neural Information Processing Systems*, 35:7103–7114, 2022\.
- Zhu et al\. \(2025\)Jie Zhu, Zhifang Fan, Xiaoxie Zhu, Yuchen Jiang, Hangyu Wang, Xintian Han, Haoran Ding, Xinmin Wang, Wenlin Zhao, Zhen Gong, et al\.Rankmixer: Scaling up ranking models in industrial recommenders\.In*Proceedings of the 34th ACM International Conference on Information and Knowledge Management*, pp\. 6309–6316, 2025\.

## Appendix ADetailed Experimental Configurations

### A\.1Common Experimental Protocol

All experiments use eight Alibaba T\-Head PPUs, each with 96 GB of memory\. Within each dataset, all methods share the backbone, input processing, task head, optimization schedule, seed set, and expert initialization; only the FFN or MoE module differs\. ImageNet\-1K uses seeds 214797, 531770, and 635451, MiniPile uses seeds 1, 145306, and 145849, and IntTravel uses seeds 42, 4981213, and 211231244\.

Table[5](https://arxiv.org/html/2609.21346#A1.T5)summarizes the model dimensions, input sizes, and training lengths\. Here,HHanddddenote the number of Transformer layers and the hidden size\.nheadsn\_\{\\mathrm\{heads\}\},dheadd\_\{\\mathrm\{head\}\}, anddrd\_\{r\}denote the number of attention heads, the head dimension, and the router hidden size\.EEis the number of expert bases in each IntBMoE internal layer and the routed\-expert count for baselines with a shared path\. BS, Input, andλ0\\lambda\_\{0\}denote the per\-device batch size, the image resolution or maximum sequence length, and IntBMoE’s initial residual scale, respectively\.

Table 5:Dataset\-specific model and training configurations\.
### A\.2Dataset\-Specific Configurations

#### ImageNet\-1K\.

We use a DeiT\-Tiny\-style backbone[Touvron et al\. \(2021\)](https://arxiv.org/html/2609.21346#bib.bib23)shortened to eight Transformer layers\. MoE modules replace the FFNs in alternating layers \(0, 2, 4, and 6\) for patch tokens, while the class token uses a separate dense FFN\. We use AdamW with\(β1,β2\)=\(0\.9,0\.999\)\(\\beta\_\{1\},\\beta\_\{2\}\)=\(0\.9,0\.999\)and weight decay 0\.05\. The learning rate warms up from10−610^\{\-6\}to5×10−45\\times 10^\{\-4\}over five epochs and then follows cosine decay to10−510^\{\-5\}\. Standard dropout, attention dropout, patch dropout, and position dropout are disabled, while stochastic depth uses a maximum DropPath rate of 0\.1\. The augmentation pipeline applies a224×224224\\times 224bicubic random resized crop with scale\[0\.08,1\.0\]\[0\.08,1\.0\]and aspect ratio\[0\.75,1\.33\]\[0\.75,1\.33\], random horizontal flipping with probability 0\.5, RandAugment using two operations per image, magnitude 9, and a magnitude standard deviation of 0\.5, and pixel\-mode random erasing with probability 0\.25 and one erased region\. Mixup \(α=0\.8\\alpha=0\.8\) and CutMix \(α=1\.0\\alpha=1\.0\) are applied in batch mode with a combined probability of 1\.0 and equal selection probabilities, and label smoothing is 0\.1\.

#### MiniPile\.

We use a Llama\-style causal Transformer[Touvron et al\. \(2023\)](https://arxiv.org/html/2609.21346#bib.bib24)and replace the FFN in every layer with an MoE module\. We train for one epoch on 1\.523 billion tokens and evaluate on a 15\.0\-million\-token test split\. Text is tokenized with a byte\-level BPE vocabulary of 32,000\. Four\-step gradient accumulation gives an effective batch size of 256 sequences across eight devices\. We use AdamW with\(β1,β2\)=\(0\.9,0\.95\)\(\\beta\_\{1\},\\beta\_\{2\}\)=\(0\.9,0\.95\)and weight decay 0\.1\. The learning rate warms up to4×10−44\\times 10^\{\-4\}over the first 10% of training and then decays cosinusoidally to4×10−54\\times 10^\{\-5\}\.

#### IntTravel\.

We use a causal Transformer for sequential recommendation and replace every FFN and the prediction head with MoE modules\. We use AdamW with\(β1,β2\)=\(0\.9,0\.999\)\(\\beta\_\{1\},\\beta\_\{2\}\)=\(0\.9,0\.999\), weight decay10−610^\{\-6\}, and a constant learning rate of10−410^\{\-4\}\.

### A\.3MoE Configurations and Fair Comparison

#### Expert\-parameter matching\.

We use the parameter count of the backbone’s original Transformer FFN, which is replaced by the MoE module, as the target for a single expert\. Each internal IntBMoE layer retains a separate pool ofEEexpert bases, and each basis contains one weight matrix\. For parameter accounting, we pair same\-index bases from the two internal layers and match their combined parameter count to this target\. For the baselines, one complete FFN or SwiGLU expert is matched to the same target by counting all of its weight matrices\. This matching sets the input–intermediate–output dimensions of each two\-layer IntBMoE block tod→→dd\\\!\\rightarrow\\\!2844\\\!\\rightarrow\\\!don MiniPile andd→4​d→dd\\\!\\rightarrow\\\!4d\\\!\\rightarrow\\\!don ImageNet\-1K and IntTravel\. We also match the always\-active shared SwiGLU expert to the same parameter target by adjusting its intermediate width\. To keep the total expert parameter count comparable across methods, a baseline with a shared path usesEErouted experts and one shared expert, whereas a baseline without one usesE\+1E\+1routed experts\. For the CP and Tensor Ring variants ofμ\\muMoE, we choose the factorization ranks that make the total trainable parameter count of the full model as close as possible to that of IntBMoE\. Specifically, we solve the CP rankRRunder this parameter budget; for Tensor Ring, we fixR1=R2=4R\_\{1\}=R\_\{2\}=4and solveR3R\_\{3\}\.

#### Baseline configurations\.

Switch Transformer follows its original Top\-1 routing rule\. Other Top\-kktoken\-choice baselines use Top\-2 routing when their original configurations do not specifykk\. Expert Choice retains expert\-to\-token routing with a capacity factor of 2\.μ\\muMoE usesentmax1\.5\\mathrm\{entmax\}\_\{1\.5\}routing, and MASS uses Top\-pprouting withp=0\.5p=0\.5\.

Soft MoE uses one slot per image token on ImageNet\-1K\. On MiniPile and IntTravel, we make Soft MoE causal by constructing each slot only from the current and preceding tokens\. We use one slot per expert to limit computation\. SMEAR uses example\-level composition for images and token\-level composition for autoregressive tasks\. Lory uses a segment size of 16\.

For DynMoE and MASS, the initial expert\-pool sizes on IntTravel, ImageNet\-1K, and MiniPile are 16, 8, and 4, respectively\. Their maximum pool sizes match the expert counts used by the other methods\. We create all expert parameters in advance up to these maxima\. DynMoE initializes each expert’s threshold to correspond to a selection probability of 0\.5 and uses a routing temperature of 1\.0\. Every 300 optimizer steps, it removes unused experts and opens a new expert when some tokens activate no expert\. MASS detects semantic drift over the previous 200 optimizer steps and expands the pool using significance and similarity thresholds of 0\.01 and 0\.001\. Its redundancy regularizer has a coefficient of 0\.01\.

#### IntBMoE configuration\.

Each module containsK=8K=8codebook blocks and routes each token tok=2k=2blocks\. The block\-embedding dimension, router hidden dimension, and hypernetwork hidden dimension are 32, 64, and 16, respectively\. Each block contains two internal DPRG layers\. These layers use equally sized, layer\-specific expert pools and share the same pair of value and gate composition coefficients\. An always\-active SwiGLU expert provides the shared path\. We initialize the expert matrices with PyTorch’s defaultnn\.Linearscheme and set all expert biases to zero\. This small initial weight scale limits perturbations from the composed transformations early in training\.