Expert Routing for Communication-Efficient MoE via Finite Expert Banks

# Expert Routing for Communication-Efficient MoE via Finite Expert Banks
Source: [https://arxiv.org/html/2605.05278](https://arxiv.org/html/2605.05278)
###### Abstract

Resource\-efficient machine learning increasingly uses sparse Mixture\-of\-Experts \(MoE\) architectures, where the gate acts as both a learning component and a routing interface controlling computation, communication, and accuracy\. Motivated by finite\-rate interpretations of MoE gating, we treat the gate as a stochastic channel and useI​\(X;T\)I\(X;T\)to quantify the routing information available to the selected expert\. To make the associated information quantities tractable beyond synthetic examples, we develop a finite\-bank MNIST construction using pretrained CNN experts and a discrete, data\-dependent selection rule\. Since the selected model belongs to a finite candidate set, the algorithmic mutual informationI​\(S;W\)I\(S;W\)admits a closed\-form discrete\-entropy estimator from the empirical posteriorq​\(W\|S\)q\(W\|S\)\. Sweeping a data\-dependence parameterα\\alpha, we observe thatI^​\(S;W\)\\widehat\{I\}\(S;W\)monotonically tracks the generalization gap, while the Xu\-Raginsky bound exhibits the expected looseness\. We also compare with a uniform union\-bound baseline and introduce an empirical estimator ofI​\(X;T\)I\(X;T\)together with a Blahut\-Arimoto procedure for tracing an accuracy\-rate curve over the expert bank\. The proposed framework provides a practical tool for analyzing resource\-aware MoE inference systems and for interpretingI​\(X;T\)I\(X;T\)andD​\(Rg\)D\(R\_\{g\}\)as design proxies for efficient expert routing\.

## IIntroduction

MoE architectures combine specialized predictors through a gating mechanism that either softly weights or discretely selects experts\[[1](https://arxiv.org/html/2605.05278#bib.bib1),[2](https://arxiv.org/html/2605.05278#bib.bib2)\]\. Sparse MoE models are now widely used in large neural architectures because only a small subset of experts is activated for each input, enabling scalable and resource\-efficient machine learning and AI\[[3](https://arxiv.org/html/2605.05278#bib.bib3)\]\. Such resource\-aware inference mechanisms are particularly relevant in communication\- and computation\-constrained signal processing systems, including aeronautics and aerospace applications where onboard processing, bandwidth, latency, and energy budgets are limited\. A complementary theoretical question is how the gate should be interpreted when expert routing is subject to communication, privacy, or compression constraints\.

A recent communication\-theoretic formulation treats the gate as a stochastic channelX→TX\\to Tand uses the gating mutual informationRg=I​\(X;T\)R\_\{g\}=I\(X;T\)as an operational information rate\. In this view,RgR\_\{g\}controls how much input information reaches the expert bank, while the learning algorithm’s dependence on the sample is measured byI​\(S;Θ\)I\(S;\\Theta\)\[[4](https://arxiv.org/html/2605.05278#bib.bib4)\], whereΘ\\Thetacollects the full MoE parameters\. The resulting risk decomposition combines a rate\-distortion termD​\(Rg\)D\(R\_\{g\}\)with an information\-theoretic generalization penalty\. This view is related to classical information\-theoretic generalization bounds\[[5](https://arxiv.org/html/2605.05278#bib.bib5),[6](https://arxiv.org/html/2605.05278#bib.bib6)\], communication\-limited learning\[[7](https://arxiv.org/html/2605.05278#bib.bib7),[8](https://arxiv.org/html/2605.05278#bib.bib8)\], hierarchical decision\-making with information constraints\[[9](https://arxiv.org/html/2605.05278#bib.bib9),[10](https://arxiv.org/html/2605.05278#bib.bib10),[11](https://arxiv.org/html/2605.05278#bib.bib11),[12](https://arxiv.org/html/2605.05278#bib.bib12)\], and risk analyses for MoE models\[[13](https://arxiv.org/html/2605.05278#bib.bib13),[14](https://arxiv.org/html/2605.05278#bib.bib14)\]\. It is also aligned with the design of resource\-efficient AI systems for edge inference, distributed sensing, autonomous platforms, and aeronautical or aerospace signal\-processing pipelines, where routing decisions may represent not only neural\-network choices but also communication and computation decisions\.

A natural concern is that synthetic experiments, while useful for isolating information\-rate effects, do not by themselves demonstrate how the framework applies to standard deep\-learning benchmarks\. The difficulty is not conceptual, but rather statistical and computational: For a CNN with on the order of10510^\{5\}continuous\-valued parameters, the parameter variable is high\-dimensional, the random training sample induces a continuous posterior over weights, andI​\(S;Θ\)I\(S;\\Theta\),I​\(X;T\)I\(X;T\), andD​\(Rg\)D\(R\_\{g\}\)are typically not available in closed form\. Consequently, any benchmark\-oriented extension must be constructed so that these quantities remain either directly measurable or meaningfully approximable\.

This paper provides such an extension\. We consider a finite bank of pretrained CNN experts on MNIST together with a discrete selection rule that maps a training sampleSSto a posterior distributionq​\(W\|S\)q\(W\|S\)over candidate models, whereWWnow denotes the index\-valued selected candidate \(a deliberate restriction of the more generalΘ\\Thetaabove\)\. This setup is sufficiently rich to be benchmark\-relevant, yet finite enough to allow exact entropy\-based evaluation ofI​\(S;W\)I\(S;W\)\. In contrast to prior work that estimatesI​\(S;Θ\)I\(S;\\Theta\)for stochastic gradient methods via PAC\-Bayes or noisy\-iterate analyses\[[15](https://arxiv.org/html/2605.05278#bib.bib15),[16](https://arxiv.org/html/2605.05278#bib.bib16)\], our finite\-bank construction makesI​\(S;W\)I\(S;W\)computable from a closed\-form discrete entropy, at the cost of restrictingWWto a pretrained candidate set\. We additionally develop a plug\-in estimator forI​\(X;T\)I\(X;T\)under discrete MoE gating and instantiate the corresponding Blahut\-Arimoto rate\-distortion solver on the same expert bank\. From a systems perspective, this provides a tractable way to study how routing information can be used as a design proxy for resource\-aware MoE inference\.

Our contributions are as follows\.

1. 1\.We introduce a finite\-bank CNN protocol on MNIST that recasts deep model selection as a tractable information\-theoretic experiment for resource\-efficient MoE inference\.
2. 2\.We derive a Monte Carlo estimator forI​\(S;W\)I\(S;W\)induced by theα\\alpha\-mixture selection rule used in practice, together with a Miller\-Madow bias analysis and a bootstrap confidence interval\.
3. 3\.We compare the resulting bound against a uniform union\-bound baseline over the finite bank, and report the looseness factor explicitly\.
4. 4\.We instantiate the input\-dependent finite\-rate routing extension empirically using a Blahut\-Arimoto solver on the fixed expert bank, producing an empirical rate\-distortion curveD^​\(ρ\)\\widehat\{D\}\(\\rho\)\.

Organization:The remainder of the paper is organized as follows\. Section[II](https://arxiv.org/html/2605.05278#S2)reviews the finite\-rate MoE framework and the Xu\-Raginsky generalization bound\. Section[III](https://arxiv.org/html/2605.05278#S3)introduces the finite\-bank CNN protocol on MNIST and theα\\alpha\-mixture selection rule\. Section[IV](https://arxiv.org/html/2605.05278#S4)derives the discrete\-entropy estimator forI​\(S;W\)I\(S;W\), analyzes its bias and variance, and compares it against a union\-bound baseline\. Section[V](https://arxiv.org/html/2605.05278#S5)extends the construction to input\-dependent finite\-rate routing via a Blahut\-Arimoto solver\. Section[VI](https://arxiv.org/html/2605.05278#S6)reports the experimental results, and Sections[VII](https://arxiv.org/html/2605.05278#S7)and[VIII](https://arxiv.org/html/2605.05278#S8)discuss limitations and conclude\.

## IIFinite\-Rate MoE Background

Let\(X,Y\)∼𝒟\(X,Y\)\\sim\\mathcal\{D\}denote an input\-label pair and considernnexperts\{hg​\(⋅;Wg\)\}g=1n\\\{h\_\{g\}\(\\cdot;W\_\{g\}\)\\\}\_\{g=1\}^\{n\}\. The gate maps an inputxxto a probability vector over experts and samples a routing variableT∈\[n\]T\\in\[n\]\. LetΘ=\(Wgate,Wexp\)\\Theta=\(W\_\{\\mathrm\{gate\}\},W\_\{\\mathrm\{exp\}\}\)withWexp=\(W1,…,Wn\)W\_\{\\mathrm\{exp\}\}=\(W\_\{1\},\\ldots,W\_\{n\}\)collect the full MoE parameters\. We assume throughout that the loss takes values in\[0,1\]\[0,1\], as required by the Xu\-Raginsky bound\. The population and empirical risks are

R​\(Θ\)=𝔼\(X,Y\)∼𝒟​𝔼T∼PWgate\(⋅\|X\)​\[ℓ​\(hT​\(X;WT\),Y\)\],\\displaystyle R\(\\Theta\)=\\mathbb\{E\}\_\{\(X,Y\)\\sim\\mathcal\{D\}\}\\mathbb\{E\}\_\{T\\sim P\_\{W\_\{\\mathrm\{gate\}\}\}\(\\cdot\|X\)\}\\left\[\\ell\\left\(h\_\{T\}\(X;W\_\{T\}\),Y\\right\)\\right\],\(1\)RS​\(Θ\)\\displaystyle R\_\{S\}\(\\Theta\)=1m​∑j=1m𝔼T∼PWgate\(⋅\|xj\)​\[ℓ​\(hT​\(xj;WT\),yj\)\],\\displaystyle=\\frac\{1\}\{m\}\\sum\_\{j=1\}^\{m\}\\mathbb\{E\}\_\{T\\sim P\_\{W\_\{\\mathrm\{gate\}\}\}\(\\cdot\|x\_\{j\}\)\}\\left\[\\ell\\left\(h\_\{T\}\(x\_\{j\};W\_\{T\}\),y\_\{j\}\\right\)\\right\],\(2\)whereS=\{\(xj,yj\)\}j=1m∼𝒟mS=\\\{\(x\_\{j\},y\_\{j\}\)\\\}\_\{j=1\}^\{m\}\\sim\\mathcal\{D\}^\{m\}\. The gate is interpreted as a channelP​\(T\|X\)P\(T\|X\)with achieved gating rate

Rg\\displaystyle R\_\{g\}≜I​\(X;T\)\.\\displaystyle\\triangleq I\(X;T\)\.\(3\)The corresponding single\-letter rate\-distortion objective is

D​\(Rg\)\\displaystyle D\(R\_\{g\}\)≜infP​\(T\|X\):I​\(X;T\)≤Rg𝔼​\[ℓ​\(hT​\(X;WT\),Y\)\],\\displaystyle\\triangleq\\inf\_\{P\(T\|X\):\\,I\(X;T\)\\leq R\_\{g\}\}\\mathbb\{E\}\\left\[\\ell\\left\(h\_\{T\}\(X;W\_\{T\}\),Y\\right\)\\right\],\(4\)where the expert bank is fixed when evaluating the infimum\. In high\-dimensional deep networks, \([4](https://arxiv.org/html/2605.05278#S2.E4)\) is rarely solved exactly; nevertheless, it is useful as a design proxy for regularized gates\.

The information\-theoretic generalization component is controlled byI​\(S;Θ\)I\(S;\\Theta\)\. Specializing the Xu\-Raginsky bound\[[5](https://arxiv.org/html/2605.05278#bib.bib5)\]to the MoE rule gives

\|𝔼​\[R​\(Θ\)\]−𝔼​\[RS​\(Θ\)\]\|\\displaystyle\\left\|\\mathbb\{E\}\[R\(\\Theta\)\]\-\\mathbb\{E\}\[R\_\{S\}\(\\Theta\)\]\\right\|≤2m​I​\(S;Θ\)\.\\displaystyle\\leq\\sqrt\{\\frac\{2\}\{m\}I\(S;\\Theta\)\}\.\(5\)If the learned gate is empirically near\-optimal for the rate\-distortion objective at achieved rateRgR\_\{g\}, i\.e\.,

𝔼​\[RS​\(Θ\)\]\\displaystyle\\mathbb\{E\}\[R\_\{S\}\(\\Theta\)\]≤D​\(Rg\)\+δm,\\displaystyle\\leq D\(R\_\{g\}\)\+\\delta\_\{m\},\(6\)then combining \([5](https://arxiv.org/html/2605.05278#S2.E5)\) and \([6](https://arxiv.org/html/2605.05278#S2.E6)\) yields

𝔼​\[R​\(Θ\)\]\\displaystyle\\mathbb\{E\}\[R\(\\Theta\)\]≤D​\(Rg\)\+δm\+2m​I​\(S;Θ\)\.\\displaystyle\\leq D\(R\_\{g\}\)\+\\delta\_\{m\}\+\\sqrt\{\\frac\{2\}\{m\}I\(S;\\Theta\)\}\.\(7\)Equation \([7](https://arxiv.org/html/2605.05278#S2.E7)\) separates an expressivity/communication term,D​\(Rg\)D\(R\_\{g\}\), from an estimation term,I​\(S;Θ\)I\(S;\\Theta\)\. We emphasize that condition \([6](https://arxiv.org/html/2605.05278#S2.E6)\) is a design assumption that is*not*verified in our finite\-bank experiment; the experiment isolates the estimation term\.

## IIIFinite\-Bank CNN Protocol on MNIST

To makeI​\(S;Θ\)I\(S;\\Theta\)tractable, we restrictΘ\\Thetato a finite, pretrained candidate set, so that the index of the selected candidate plays the role of the learned parameter\. With a slight abuse of notation we will writeWWfor this index\-valued random variable below\.

### III\-ACandidate Bank

The benchmark extension uses a finite bank ofRRpretrained CNN classifiers

𝒲R\\displaystyle\\mathcal\{W\}\_\{R\}≜\{W1,…,WR\}\.\\displaystyle\\triangleq\\\{W\_\{1\},\\ldots,W\_\{R\}\\\}\.\(8\)In the implementation, each candidate uses the same compact CNN architecture, consisting of two convolutional layers with1616and3232filters, respectively, each followed by max\-pooling, a fully connected layer with6464hidden units, and a1010\-class softmax output\. Each candidate is pretrained on an independently drawn subset of10,00010\{,\}000MNIST training images \(sampled with replacement across candidates\) for one epoch using Adam with a learning rate of10−310^\{\-3\}and independent random initialization\. This procedure yields candidates with test accuracies in the range0\.890\.89\-0\.920\.92and pairwise prediction disagreement rates of approximately0\.060\.06\-0\.100\.10on the MNIST test set, indicating nontrivial diversity\. The default configuration is summarized in Table[I](https://arxiv.org/html/2605.05278#S3.T1)\.

For a sampleSSof sizemm, the empirical0\-11error of candidaterris

R^S​\(Wr\)\\displaystyle\\widehat\{R\}\_\{S\}\(W\_\{r\}\)≜1m​∑j=1m𝟏​\{arg⁡maxc∈\{0,…,9\}⁡fc​\(xj;Wr\)≠yj\},\\displaystyle\\triangleq\\frac\{1\}\{m\}\\sum\_\{j=1\}^\{m\}\\mathbf\{1\}\\left\\\{\\arg\\max\_\{c\\in\\\{0,\\ldots,9\\\}\}f\_\{c\}\(x\_\{j\};W\_\{r\}\)\\neq y\_\{j\}\\right\\\},\(9\)wherefc​\(x;Wr\)f\_\{c\}\(x;W\_\{r\}\)is the softmax score assigned to classccby candidaterr\.

TABLE I:MNIST finite\-bank CNN results\. Entropies and mutual information are reported in nats\.TABLE II:Effect of the mixture parameterα\\alphaon the information\-generalization term\.
### III\-BData\-Dependent Discrete Selection

GivenSS, the empirical\-risk\-minimizing candidate is

r⋆​\(S\)\\displaystyle r^\{\\star\}\(S\)≜arg​min1≤r≤RR^S​\(Wr\),\\displaystyle\\triangleq\\mathop\{\\mathrm\{arg\\,min\}\}\_\{1\\leq r\\leq R\}\\widehat\{R\}\_\{S\}\(W\_\{r\}\),\(10\)where ties \(which arise rarely underm=256m=256\) are broken by selecting the smallest index\. The randomized learning rule used in the code is anα\\alpha\-mixture posterior over the finite bank:

qα​\(r\|S\)\\displaystyle q\_\{\\alpha\}\(r\|S\)≜1−αR\+α​1​\{r=r⋆​\(S\)\},r∈\{1,…,R\}\.\\displaystyle\\triangleq\\frac\{1\-\\alpha\}\{R\}\+\\alpha\\,\\mathbf\{1\}\\\{r=r^\{\\star\}\(S\)\\\},\\quad r\\in\\\{1,\\ldots,R\\\}\.\(11\)The selected indexWWis then sampled according to

W\|S\\displaystyle W\\,\|\\,S∼qα\(⋅\|S\)\.\\displaystyle\\sim q\_\{\\alpha\}\(\\cdot\|S\)\.\(12\)This construction interpolates between a sample\-independent learner atα=0\\alpha=0and the empirical\-risk minimizer atα=1\\alpha=1\. Increasingα\\alphaincreases the dependence ofWWonSSand therefore increasesI​\(S;W\)I\(S;W\)\.

### III\-CSensitivity to Bank Construction

To assess robustness we variedR∈\{10,25,50\}R\\in\\\{10,25,50\\\}and the per\-candidate pretraining subset size in\{5​k,10​k,20​k\}\\\{5\\text\{k\},10\\text\{k\},20\\text\{k\}\\\}\. The qualitative trend reported in Table[II](https://arxiv.org/html/2605.05278#S3.T2)\(monotone increase ofI^​\(S;W\)\\widehat\{I\}\(S;W\)withα\\alpha\) was preserved across all configurations; absolute values ofI^​\(S;W\)\\widehat\{I\}\(S;W\)scaled approximately aslog⁡R\\log R, consistent with the entropy ceilingH​\(W\)≤log⁡RH\(W\)\\leq\\log R\. The default valueR=25R=25was chosen as a balance between informational diversity and Monte Carlo estimation cost\. The sample sizem=256m=256was chosen so that the bound proxy2​I/m\\sqrt\{2I/m\}remains in the same numerical range as the observed gap, allowing a meaningful side\-by\-side comparison\.

MNISTsampleSSEvaluateR^S​\(Wr\)\\widehat\{R\}\_\{S\}\(W\_\{r\}\)Buildqα​\(r\|S\)q\_\{\\alpha\}\(r\|S\)SampleWWFinite CNN bank\{Wr\}r=1R\\\{W\_\{r\}\\\}\_\{r=1\}^\{R\}Reportgap,H​\(W\)H\(W\),H​\(W\|S\)H\(W\|S\)Figure 1:Finite\-bank MNIST protocol\. A small sampleSSis used to score pretrained CNN candidates, form the posteriorqα​\(r\|S\)q\_\{\\alpha\}\(r\|S\), sample a model, and estimate the information\-generalization term\.

## IVEstimatingI​\(S;W\)I\(S;W\)in the Finite Bank

BecauseWWtakes values in a finite set,I​\(S;W\)I\(S;W\)can be estimated directly\. The exact identity is

I​\(S;W\)\\displaystyle I\(S;W\)=H​\(W\)−H​\(W\|S\)\.\\displaystyle=H\(W\)\-H\(W\|S\)\.\(13\)LetS1,…,SMS\_\{1\},\\ldots,S\_\{M\}be independent Monte Carlo training samples, and write

qi,r\\displaystyle q\_\{i,r\}≜qα​\(r\|Si\)\.\\displaystyle\\triangleq q\_\{\\alpha\}\(r\|S\_\{i\}\)\.\(14\)The empirical marginal distribution of the selected model is

p^r\\displaystyle\\widehat\{p\}\_\{r\}≜1M​∑i=1Mqi,r,r=\{1,…,R\}\.\\displaystyle\\triangleq\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}q\_\{i,r\},\\quad r=\\\{1,\\ldots,R\\\}\.\(15\)The plug\-in entropy estimates are

H^​\(W\)\\displaystyle\\widehat\{H\}\(W\)≜−∑r=1Rp^r​log⁡p^r,\\displaystyle\\triangleq\-\\sum\_\{r=1\}^\{R\}\\widehat\{p\}\_\{r\}\\log\\widehat\{p\}\_\{r\},\(16\)H^​\(W\|S\)\\displaystyle\\widehat\{H\}\(W\|S\)≜−1M​∑i=1M∑r=1Rqi,r​log⁡qi,r\.\\displaystyle\\triangleq\-\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}\\sum\_\{r=1\}^\{R\}q\_\{i,r\}\\log q\_\{i,r\}\.\(17\)Thus the finite\-bank mutual\-information estimate is

I^​\(S;W\)\\displaystyle\\widehat\{I\}\(S;W\)≜H^​\(W\)−H^​\(W\|S\)\.\\displaystyle\\triangleq\\widehat\{H\}\(W\)\-\\widehat\{H\}\(W\|S\)\.\(18\)Since the posterior in \([11](https://arxiv.org/html/2605.05278#S3.E11)\) has one large mass andR−1R\-1equal smaller masses, its conditional entropy is constant across samples:

H\(qα\(⋅\|S\)\)\\displaystyle H\(q\_\{\\alpha\}\(\\cdot\|S\)\)=−\(α\+1−αR\)​log⁡\(α\+1−αR\)\\displaystyle=\-\\left\(\\alpha\+\\frac\{1\-\\alpha\}\{R\}\\right\)\\log\\left\(\\alpha\+\\frac\{1\-\\alpha\}\{R\}\\right\)−\(R−1\)​1−αR​log⁡\(1−αR\)\.\\displaystyle\\quad\-\(R\-1\)\\frac\{1\-\\alpha\}\{R\}\\log\\left\(\\frac\{1\-\\alpha\}\{R\}\\right\)\.\(19\)Therefore, variation inI^​\(S;W\)\\widehat\{I\}\(S;W\)comes entirely from the marginal distributionp^r\\widehat\{p\}\_\{r\}, i\.e\., from how often each candidate is the empirical\-risk minimizer over random samples\.

### IV\-AEstimator Properties

The plug\-in entropy estimatorH^​\(W\)\\widehat\{H\}\(W\)is biased downward; the Miller\-Madow correction is of order\(R−1\)/\(2​M\)\(R\-1\)/\(2M\), which evaluates to approximately0\.040\.04nats in our default configuration \(R=25R=25,M=300M=300\)\. This is small relative to the measuredI^​\(S;W\)≈1\.38\\widehat\{I\}\(S;W\)\\approx 1\.38nats, but we report the corrected estimate as a robustness check\. A nonparametric bootstrap over theMMMonte Carlo runs gives a95%95\\%confidence interval of approximately\[1\.34,1\.42\]\[1\.34,1\.42\]nats forI^​\(S;W\)\\widehat\{I\}\(S;W\)atα=0\.7\\alpha=0\.7, indicating that variation across theα\\alphasweep in Table[II](https://arxiv.org/html/2605.05278#S3.T2)substantially exceeds estimator uncertainty\.

### IV\-BGeneralization Gap and Bounds

The empirical generalization gap measured by the code is

gen^\\displaystyle\\widehat\{\\mathrm\{gen\}\}≜1M​∑i=1M\[Rtest​\(Wi\)−R^Si​\(Wi\)\],\\displaystyle\\triangleq\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}\\left\[R\_\{\\mathrm\{test\}\}\(W\_\{i\}\)\-\\widehat\{R\}\_\{S\_\{i\}\}\(W\_\{i\}\)\\right\],\(20\)whereWiW\_\{i\}is sampled fromqα\(⋅\|Si\)q\_\{\\alpha\}\(\\cdot\|S\_\{i\}\)\. The corresponding information term is

B^MI\\displaystyle\\widehat\{B\}\_\{\\mathrm\{MI\}\}≜2​I^​\(S;W\)m\.\\displaystyle\\triangleq\\sqrt\{\\frac\{2\\,\\widehat\{I\}\(S;W\)\}\{m\}\}\.\(21\)

### IV\-CBaseline: Union Bound Over the Finite Bank

BecauseWWis index\-valued in\{1,…,R\}\\\{1,\\ldots,R\\\}, an immediate non\-information\-theoretic baseline is the classical union bound for finite hypothesis classes,

BUB\\displaystyle B\_\{\\mathrm\{UB\}\}≜log⁡R2​m\.\\displaystyle\\triangleq\\sqrt\{\\frac\{\\log R\}\{2m\}\}\.\(22\)For our default configuration this givesBUB≈0\.079B\_\{\\mathrm\{UB\}\}\\approx 0\.079, compared withB^MI≈0\.104\\widehat\{B\}\_\{\\mathrm\{MI\}\}\\approx 0\.104atα=0\.7\\alpha=0\.7andB^MI≈0\.140\\widehat\{B\}\_\{\\mathrm\{MI\}\}\\approx 0\.140atα=1\\alpha=1\. The union bound is therefore tighter for the deterministic ERM rule, as expected:I^​\(S;W\)→log⁡R\\widehat\{I\}\(S;W\)\\to\\log Rasα→1\\alpha\\to 1, soB^MI→2​log⁡R/m\\widehat\{B\}\_\{\\mathrm\{MI\}\}\\to\\sqrt\{2\\log R/m\}, exactly4=2×\\sqrt\{4\}=2\\timesthe union bound\. The information\-theoretic bound becomes competitive precisely when randomization in the gate \(smallα\\alpha\) reducesI​\(S;W\)I\(S;W\)belowlog⁡R\\log R, which is the regime of interest for finite\-rate MoE: the value of the MI bound is not in beating the union bound on the deterministic ERM, but in tracking how the gap depends continuously on the gate’s randomization, which the union bound cannot\.

## VAdding Input\-Dependent Finite\-Rate Routing

The finite\-bank CNN protocol can be extended into a genuine MoE routing experiment by treating the CNNs as experts and adding a discrete gatepθ​\(t\|x\)p\_\{\\theta\}\(t\|x\)overt∈\{1,…,R\}t\\in\\\{1,\\ldots,R\\\}\. For a held\-out set\{xi\}i=1N\\\{x\_\{i\}\\\}\_\{i=1\}^\{N\}, the empirical expert marginal is

π^t\\displaystyle\\widehat\{\\pi\}\_\{t\}≜1N​∑i=1Npθ​\(t\|xi\),t=1,…,R\.\\displaystyle\\triangleq\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}p\_\{\\theta\}\(t\|x\_\{i\}\),\\quad t=1,\\ldots,R\.\(23\)The plug\-in estimator of the routing information is

I^​\(X;T\)\\displaystyle\\widehat\{I\}\(X;T\)≜1N​∑i=1N∑t=1Rpθ​\(t\|xi\)​log⁡pθ​\(t\|xi\)π^t\.\\displaystyle\\triangleq\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\sum\_\{t=1\}^\{R\}p\_\{\\theta\}\(t\|x\_\{i\}\)\\log\\frac\{p\_\{\\theta\}\(t\|x\_\{i\}\)\}\{\\widehat\{\\pi\}\_\{t\}\}\.\(24\)Equation \([24](https://arxiv.org/html/2605.05278#S5.E24)\) is the practical estimator corresponding to the discrete formula in the finite\-rate MoE framework\.

Letℓi,t\\ell\_\{i,t\}denote the loss of experttton sample\(xi,yi\)\(x\_\{i\},y\_\{i\}\):

ℓi,t\\displaystyle\\ell\_\{i,t\}≜ℓ​\(ht​\(xi;Wt\),yi\)\.\\displaystyle\\triangleq\\ell\\left\(h\_\{t\}\(x\_\{i\};W\_\{t\}\),y\_\{i\}\\right\)\.\(25\)The empirical rate\-regularized gate objective is

ℒ^λ​\(θ\)\\displaystyle\\widehat\{\\mathcal\{L\}\}\_\{\\lambda\}\(\\theta\)≜1N​∑i=1N∑t=1Rpθ​\(t\|xi\)​ℓi,t\+λ​I^​\(X;T\)\.\\displaystyle\\triangleq\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\sum\_\{t=1\}^\{R\}p\_\{\\theta\}\(t\|x\_\{i\}\)\\ell\_\{i,t\}\+\\lambda\\,\\widehat\{I\}\(X;T\)\.\(26\)Equivalently, one may solve the constrained empirical problem

D^​\(ρ\)\\displaystyle\\widehat\{D\}\(\\rho\)≜minpθ​\(t\|x\):I^​\(X;T\)≤ρ⁡1N​∑i=1N∑t=1Rpθ​\(t\|xi\)​ℓi,t\.\\displaystyle\\triangleq\\min\_\{p\_\{\\theta\}\(t\|x\):\\,\\widehat\{I\}\(X;T\)\\leq\\rho\}\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\sum\_\{t=1\}^\{R\}p\_\{\\theta\}\(t\|x\_\{i\}\)\\ell\_\{i,t\}\.\(27\)When no neural parameterization is imposed on the gate and all conditional distributionsp​\(t\|xi\)p\(t\|x\_\{i\}\)are allowed, a Blahut\-Arimoto\-style update\[[17](https://arxiv.org/html/2605.05278#bib.bib17)\]has the form

p\(k\+1\)​\(t\|xi\)\\displaystyle p^\{\(k\+1\)\}\(t\|x\_\{i\}\)=πt\(k\)​exp⁡\[−ℓi,t/λ\]∑s=1Rπs\(k\)​exp⁡\[−ℓi,s/λ\],\\displaystyle=\\frac\{\\pi\_\{t\}^\{\(k\)\}\\exp\\left\[\-\\ell\_\{i,t\}/\\lambda\\right\]\}\{\\sum\_\{s=1\}^\{R\}\\pi\_\{s\}^\{\(k\)\}\\exp\\left\[\-\\ell\_\{i,s\}/\\lambda\\right\]\},\(28\)πt\(k\+1\)\\displaystyle\\pi\_\{t\}^\{\(k\+1\)\}=1N​∑i=1Np\(k\+1\)​\(t\|xi\)\.\\displaystyle=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}p^\{\(k\+1\)\}\(t\|x\_\{i\}\)\.\(29\)
### V\-AEmpirical Rate\-Distortion Curve

We instantiate \([28](https://arxiv.org/html/2605.05278#S5.E28)\)\-\([29](https://arxiv.org/html/2605.05278#S5.E29)\) on a held\-out subset ofN=2,000N=2\{,\}000MNIST test images using the sameR=25R=25candidates and the0\-11loss matrix\{ℓi,t\}\\\{\\ell\_\{i,t\}\\\}\. Sweepingλ\\lambdaover a logarithmic grid in\[10−3,101\]\[10^\{\-3\},10^\{1\}\]traces an empirical rate\-distortion curveD^​\(ρ\)\\widehat\{D\}\(\\rho\)\. Atλ→∞\\lambda\\to\\inftythe gate collapses to a single expert \(the one with the lowest average loss\), givingI^​\(X;T\)≈0\\widehat\{I\}\(X;T\)\\approx 0andD^​\(0\)\\widehat\{D\}\(0\)equal to that expert’s mean error\. Atλ→0\\lambda\\to 0the gate routes each input to its best expert, attainingI^​\(X;T\)\\widehat\{I\}\(X;T\)near its empirical maximum and the lowest achievable distortion\. Intermediateλ\\lambdavalues trace a strictly decreasing convex curve, consistent with the rate\-distortion theory\. Reporting code and the resulting curve are provided alongside the implementation\.

## VIExperimental Details and Reporting

The implementation follows the workflow in Fig\.[1](https://arxiv.org/html/2605.05278#S3.F1)\. It first pretrainsR=25R=25CNN candidates as described in Section III\. It then repeats the following procedure overM=300M=300random samples of sizem=256m=256: compute each candidate’s empirical error, formqα​\(r\|S\)q\_\{\\alpha\}\(r\|S\)withα=0\.7\\alpha=0\.7, sample a candidate, evaluate both train and test errors, and accumulate the entropy estimates in \([16](https://arxiv.org/html/2605.05278#S4.E16)\)\-\([18](https://arxiv.org/html/2605.05278#S4.E18)\)\.

For conference reporting, the following quantities are included:

R¯train\\displaystyle\\overline\{R\}\_\{\\mathrm\{train\}\}≜1M​∑i=1MR^Si​\(Wi\),\\displaystyle\\triangleq\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}\\widehat\{R\}\_\{S\_\{i\}\}\(W\_\{i\}\),\(30\)R¯test\\displaystyle\\overline\{R\}\_\{\\mathrm\{test\}\}≜1M​∑i=1MRtest​\(Wi\),\\displaystyle\\triangleq\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}R\_\{\\mathrm\{test\}\}\(W\_\{i\}\),\(31\)G¯\\displaystyle\\overline\{G\}≜R¯test−R¯train,\\displaystyle\\triangleq\\overline\{R\}\_\{\\mathrm\{test\}\}\-\\overline\{R\}\_\{\\mathrm\{train\}\},\(32\)G¯abs\\displaystyle\\overline\{G\}\_\{\\mathrm\{abs\}\}≜1M​∑i=1M\|Rtest​\(Wi\)−R^Si​\(Wi\)\|\.\\displaystyle\\triangleq\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}\\left\|R\_\{\\mathrm\{test\}\}\(W\_\{i\}\)\-\\widehat\{R\}\_\{S\_\{i\}\}\(W\_\{i\}\)\\right\|\.\(33\)
![Refer to caption](https://arxiv.org/html/2605.05278v1/x1.png)Figure 2:Empirical distribution of the MNIST finite\-bank generalization gapRtest​\(W\)−R^S​\(W\)R\_\{\\mathrm\{test\}\}\(W\)\-\\widehat\{R\}\_\{S\}\(W\), with vertical lines marking the mean gap and the mutual\-information bound proxy±2​I^​\(S;W\)/m\\pm\\sqrt\{2\\widehat\{I\}\(S;W\)/m\}\.### VI\-ABound Looseness

The most informative empirical observation is theα\\alpha\-sweep in Table[II](https://arxiv.org/html/2605.05278#S3.T2)\. Asα\\alphaincreases from0to11,I^​\(S;W\)\\widehat\{I\}\(S;W\)rises monotonically from0to approximately2\.532\.53nats and the bound proxyB^MI\\widehat\{B\}\_\{\\mathrm\{MI\}\}rises from0to0\.1400\.140, while the mean signed gapG¯\\overline\{G\}rises only from−0\.009\-0\.009to0\.0100\.010\. The MI bound is therefore loose by a factor of roughly1414\-20×20\\timesin absolute terms across the sweep\. This degree of looseness is consistent with known properties of the Xu\-Raginsky bound\[[5](https://arxiv.org/html/2605.05278#bib.bib5)\]and is partly addressed by tighter individual\-sample variants\[[6](https://arxiv.org/html/2605.05278#bib.bib6)\]; we view the looseness as a calibration property of the bound rather than a defect of the estimator\. The decisive empirical signature is qualitative: the bound’s monotone tracking ofα\\alphamatches the monotone behavior of the gap, which is what one would hope an information\-theoretic bound to capture in this regime\.

### VI\-BInterpreting the Benchmark

The finite\-bank MNIST experiment should be read as an engineering validation of the information\-generalization mechanism rather than as a claim that the exact Shannon rate\-distortion function of a high\-dimensional neural MoE has been computed\. The main reason is that the finite bank makesI​\(S;W\)I\(S;W\)observable\. For unrestricted deep networks, the parameter variableΘ\\Thetais high\-dimensional and continuous, andI​\(S;Θ\)I\(S;\\Theta\)is usually intractable\. Practical substitutes include compression\-based bounds, PAC\-Bayesian estimates, and variational mutual\-information estimators such as MINE\[[18](https://arxiv.org/html/2605.05278#bib.bib18)\]\. Similarly,I​\(X;T\)I\(X;T\)is directly computable for discrete gates through \([24](https://arxiv.org/html/2605.05278#S5.E24)\), but it is only a proxy for physical capacity unless the gate is actually transmitted over a channel with a known capacity constraint\.

## VIIDiscussion

### VII\-ATwo Information Quantities, Two Roles

The proposed benchmark extension clarifies the roles of two different information quantities\. The termI​\(S;W\)I\(S;W\)measures the data dependence of the learning algorithm\. In the finite\-bank experiment, this dependence is adjustable throughα\\alpha: asα\\alphaincreases, the learner more often selects the empirical\-risk minimizer, soH​\(W\|S\)H\(W\|S\)decreases and the bound \([21](https://arxiv.org/html/2605.05278#S4.E21)\) increases\. This reproduces the basic behavior of the synthetic experiments while using a standard image\-classification dataset\.

The routing termI​\(X;T\)I\(X;T\)has a different interpretation\. It measures how much input information the gate communicates to the expert bank at inference time\. It can be reduced through entropy regularization, noisy routing, local differential privacy\[[19](https://arxiv.org/html/2605.05278#bib.bib19)\], or explicit capacity constraints\. Reducing this term improves communication efficiency and may improve robustness, but it can increase the distortion termD​\(Rg\)D\(R\_\{g\}\)because the gate has less information with which to route examples to specialized experts\. This separation is useful in communication\-constrained applications such as federated MoE, edge inference, split inference, coded computing, and aerospace systems with strict bandwidth and latency budgets\[[20](https://arxiv.org/html/2605.05278#bib.bib20),[21](https://arxiv.org/html/2605.05278#bib.bib21)\], where the gate is not merely a neural\-network module but also a communication interface\.

### VII\-BLimitations

Three limitations should be made explicit\.First, MNIST is a comparatively easy benchmark; the magnitudes of both the gap and the bound are small in absolute terms, and behavior on harder datasets \(CIFAR\-10, ImageNet subsets\) may differ\.Second, the finite\-bank construction restrictsWWto a discrete set, which makesI​\(S;W\)I\(S;W\)tractable but rules out claims about the continuous parameterI​\(S;Θ\)I\(S;\\Theta\)of an end\-to\-end trained MoE; bridging this gap requires variational or compression\-based estimators\.Third, the Xu\-Raginsky bound is known to be loose by a constant factor; the looseness we observe \(Section[VI\-A](https://arxiv.org/html/2605.05278#S6.SS1)\) is consistent with this and is not a deficiency of the finite\-bank estimator\. Tightening the bound would require switching to individual\-sample mutual\-information bounds\[[6](https://arxiv.org/html/2605.05278#bib.bib6)\]or PAC\-Bayesian alternatives\.

## VIIIConclusion

This paper proposed a deep\-benchmark extension of finite\-rate MoE gating in which the algorithmic mutual informationI​\(S;W\)I\(S;W\)is computable in closed form on MNIST\. Motivated by resource\-efficient machine learning and AI, the framework interprets MoE gating as both a learning mechanism and a routing interface for computation\- and communication\-constrained inference systems, including aeronautics and aerospace signal\-processing applications\. Across a sweep of the data\-dependence parameterα\\alphawe observed the predicted monotone relationship betweenI^​\(S;W\)\\widehat\{I\}\(S;W\)and the generalization gap, while the absolute gap remained1414\-20×20\\timesbelow the bound proxy, consistent with the known constant looseness of Xu\-Raginsky\. We compared against a uniform union bound and identified the regime—randomized gates withα<1\\alpha<1—in which the information\-theoretic bound carries information that the union bound cannot\. We further instantiated the input\-dependent routing extension via a Blahut\-Arimoto solver on the fixed expert bank, producing an empirical rate\-distortion curveD^​\(ρ\)\\widehat\{D\}\(\\rho\)over the same MNIST candidates\. The resulting accuracy\-rate viewpoint provides a tractable design proxy for resource\-aware MoE inference, where routing information can be related to communication, latency, and computational constraints\. Future work should train the input\-dependent gate end\-to\-end under the empirical objective \([26](https://arxiv.org/html/2605.05278#S5.E26)\), sweep the routing\-rate budgetρ\\rhoin \([27](https://arxiv.org/html/2605.05278#S5.E27)\), and compare the resulting accuracy\-rate curves across MNIST, Fashion\-MNIST, CIFAR\-10, larger sparse\-MoE architectures, and signal\-processing tasks arising in edge, aeronautical, and aerospace systems\.

## References

- \[1\]R\. A\. Jacobs, M\. I\. Jordan, S\. J\. Nowlan, and G\. E\. Hinton, “Adaptive mixtures of local experts,”*Neural Computation*, vol\. 3, no\. 1, pp\. 79–87, 1991\.
- \[2\]M\. I\. Jordan and R\. A\. Jacobs, “Hierarchical mixtures of experts and the em algorithm,”*Neural Computation*, vol\. 6, no\. 2, pp\. 181–214, 1994\.
- \[3\]W\. Fedus, B\. Zoph, and N\. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,”*Journal of Machine Learning Research*, vol\. 23, no\. 120, pp\. 1–39, 2022\.
- \[4\]A\. Khalesi and M\. R\. D\. Salehi, “Mixture\-of\-experts under finite\-rate gating: Communication–generalization trade\-offs,”*IEEE Communications Letters*, 2026\.
- \[5\]A\. Xu and M\. Raginsky, “Information\-theoretic analysis of generalization capability of learning algorithms,” in*Advances in Neural Information Processing Systems*, vol\. 30, 2017\.
- \[6\]Y\. Bu, S\. Zou, and V\. V\. Veeravalli, “Tightening mutual information\-based bounds on generalization error,”*IEEE Journal on Selected Areas in Information Theory*, vol\. 1, no\. 1, pp\. 121–130, 2020\.
- \[7\]Y\. Polyanskiy and Y\. Wu,*Information Theory: From Coding to Learning*\. Cambridge University Press, 2022, draft manuscript\.
- \[8\]O\. Shamir, “Fundamental limits of online and distributed learning,” in*International Conference on Machine Learning*\. PMLR, 2014, pp\. 1314–1322\.
- \[9\]H\. Hihn, S\. Gottwald, and D\. A\. Braun, “An information\-theoretic online learning principle for specialization in hierarchical decision\-making systems,” in*2019 IEEE 58th Conference on Decision and Control \(CDC\)*\. IEEE, 2019, pp\. 3677–3684\.
- \[10\]H\. Hihn and D\. A\. Braun, “Hierarchically structured task\-agnostic continual learning,”*Machine Learning*, vol\. 112, no\. 2, pp\. 655–686, 2023\.
- \[11\]A\. Khalesi and M\. R\. Deylam Salehi, “Typical solutions of multi\-user linearly\-decomposable distributed computing,”*IEEE Networking Letters*, vol\. 8, pp\. 10–13, 2026\.
- \[12\]Z\. He, M\. R\. D\. Salehi, D\. Malak, and P\. A\. Stavrou, “Learning\-augmented perfectly secure collaborative matrix multiplication,” in*Proc\. IEEE Int\. Symp\. Inf\. Theory \(ISIT\)*, Jun\. 2026\.
- \[13\]A\. Azran and R\. Meir, “Data dependent risk bounds for hierarchical mixture of experts classifiers,” in*International Conference on Computational Learning Theory*\. Springer, 2004, pp\. 427–441\.
- \[14\]W\. Akretche, F\. LeBlanc, and M\. Marchand, “Tighter risk bounds for mixtures of experts,”*arXiv preprint arXiv:2410\.10397*, 2024\.
- \[15\]A\. Pensia, V\. Jog, and P\.\-L\. Loh, “Generalization error bounds for noisy, iterative algorithms,” in*Proc\. IEEE Int\. Symp\. Inf\. Theory \(ISIT\)*, 2018, pp\. 546–550\.
- \[16\]J\. Negrea, M\. Haghifam, G\. K\. Dziugaite, A\. Khisti, and D\. M\. Roy, “Information\-theoretic generalization bounds for SGLD via data\-dependent estimates,” in*Advances in Neural Information Processing Systems\(NeurIPS\)*, vol\. 32, 2019\.
- \[17\]R\. E\. Blahut, “An hypothesis testing approach to information theory,” Ph\.D\. dissertation, Cornell University, 1972\.
- \[18\]M\. I\. Belghazi, A\. Baratin, S\. Rajeshwar, S\. Ozair, Y\. Bengio, A\. Courville, and D\. Hjelm, “Mutual information neural estimation,” in*International Conference on Machine Learning*\. PMLR, 2018, pp\. 531–540\.
- \[19\]J\. C\. Duchi, M\. I\. Jordan, and M\. J\. Wainwright, “Local privacy and statistical minimax rates,” in*2013 IEEE 54th Annual Symposium on Foundations of Computer Science*\. IEEE, 2013, pp\. 429–438\.
- \[20\]Y\. Cao, W\. Yu, W\. Ren, and G\. Chen, “An overview of recent progress in the study of distributed multi\-agent coordination,”*IEEE Transactions on Industrial Informatics*, vol\. 9, no\. 1, pp\. 427–438, 2013\.
- \[21\]Y\. Zeng, Q\. Wu, and R\. Zhang, “Accessing from the sky: A tutorial on uav communications for 5g and beyond,”*Proceedings of the IEEE*, vol\. 107, no\. 12, pp\. 2327–2375, 2019\.