Mixture of Channel Experts: Static Sparse Supports with Input-Adaptive Mixing for Pointwise Projections

# Static Sparse Supports withInput-Adaptive Mixing for Pointwise Projections
Source: [https://arxiv.org/html/2608.23794](https://arxiv.org/html/2608.23794)
## Mixture of Channel Experts: Static Sparse Supports with Input\-Adaptive Mixing for Pointwise Projections

###### Abstract

Mixture\-of\-Experts \(MoE\) scales language models by routing each input through a small set of independently parameterized experts\. We show that copying this design into convolutional networks fails for a structural reason: parallel convolutional experts that read the same input channels learn nearly identical filters\. We therefore move the expert axis from*operator duplication*to*channel selection*\. We introduce*Mixture of Channel Experts \(MoCE\)*, a structured sparse channel\-mixing layer, inspired by MoE, that replaces pointwise \(1×11\\times 1\) channel\-reduction projections\. In MoCE, an*expert*is a single output channel with a learned sparse support ofk≪Ck\\ll Cinput channels\. The selected channels are combined by a softmax whose temperature is predicted per input, so each expert can move between mean\-like and max\-like aggregation\. A residual expert summarizes the unselected channels, and a load\-balancing loss keeps channel coverage complete\. MoCE replaces a dense projection whose cost is quadratic inCCwith a mechanism whose relative cost scales ask/Ck/C, and the predicted savings hold in measured wall\-clock time\. Across ResNet backbones on ImageNet\-1K and CIFAR\-100, transfer learning, EfficientViT, and a strong modern training recipe, MoCE matches or exceeds dense baselines and prior channel\-selection methods while reducing MACs by 16\.7% and end\-to\-end latency\.

School of Computer Science, Ariel University, Israel

elianroy\.iluk@msmail\.ariel\.ac\.il, gilba@ariel\.ac\.il

## 1Introduction

Pointwise projections mix channels independently at every spatial location and appear throughout modern vision backbones: as the entrance and exit stages of ResNet bottlenecks, as the projections of inverted bottlenecks and ConvNeXt\-style blocks, and—since a token\-wise linear map is a1×11\\times 1convolution—as the feed\-forward projections of vision transformers\. Their cost isCin​Cout​H​WC\_\{\\mathrm\{in\}\}C\_\{\\mathrm\{out\}\}HW, so channel width directly increases both arithmetic and parameter count\. This makes them a natural target for structured sparsification, provided the resulting operator remains efficient on hardware\.

A direct convolutional analogue of Mixture\-of\-Experts \(MoE\) duplicates an operator and routes each input to a subset of parallel experts\. In a controlled ResNet\-50/CIFAR\-100 diagnostic with eight convolutional experts sharing the same input representation and objective, the learned kernels are strongly aligned: the mean off\-diagonal cosine similarity is0\.880\.88, with most pairs between0\.850\.85and0\.970\.97\(Fig\.[1](https://arxiv.org/html/2608.23794#S1.F1)\)\. A comparable collapse has been reported for upcycled transformer MoE, where experts remain close to the original dense weights and to one another\([Huang et al\. 2025](https://arxiv.org/html/2608.23794#bib.bib11)\)\. Together these observations indicate that operator duplication can spend parameters without producing useful specialization when experts receive the same representation and supervision\. We therefore move specialization from duplicated operators to sparse channel supports\.*Mixture of Channel Experts*\(MoCE\) replaces a dense pointwise projection with one aggregation unit per output channel \(Fig\.[2](https://arxiv.org/html/2608.23794#S3.F2)\)\. Each routed expert selects a learned top\-kksubset of the input channels and forms a convex mixture over that support\. The support is static at inference, enabling fixed gathers and packed execution\. A lightweight gate predicts one temperature per expert and example; the temperature changes how concentrated the expert’s learned channel preferences are, without changing the selected support or the ordering of its weights\. One residual output aggregates channels not selected by any routed expert\.

#### Where input dependence pays\.

The design question that organizes this paper is whether the input signal is better spent on choosing*which*channels an expert reads or on choosing*how*it combines them\. The dynamic alternative is the more expressive of the two: a router must score every candidate channel before selecting, so it reads the full pooled descriptor and adds a low\-rank offset to the routing logits, which changes both which channels are read and how they are weighted\. The temperature gate, by contrast, reads only thekkdescriptors already on the expert’s support and emits a single scalar\. Despite this advantage, we show that for example, on CIFAR\-100 the dynamic variant improves the accuracy by\+0\.03\+0\.03points while making the routing\-and\-gather path7\.13×7\.13\\timesslower, because the memory\-access pattern becomes data dependent\. However, removing the temperature gate reduces accuracy by1\.191\.19points\. That the more expressive route produces no meaningful gain in the evaluated run, while removing the more constrained one costs over a point, is what makes static supports the better operating point, and it is why MoCE remains statically schedulable\.

MoCE is inspired by expert\-side routing, but it is not a capacity\-scaling MoE: an expert is an output channel rather than an independently parameterized network\. Its closest methodological context is structured sparse projection, channel routing, dynamic channel selection, and channel attention\. The contribution is the combination of learned static support, minimal input\-adaptive mixing, residual coverage, and an implementation whose savings are visible in wall\-clock measurements\. Across ResNet and EfficientViT backbones, sparse supports preserve or improve the dense accuracy–efficiency operating point at1717–21%21\\%fewer MACs, with deployed\-parameter reductions of1717–21%21\\%on the ResNet variants, and an end\-to\-end speedup that is smaller than the MAC reduction for reasons an arithmetic\-intensity analysis makes explicit\.

![Refer to caption](https://arxiv.org/html/2608.23794v1/cosine_similarity.jpg)Figure 1:Parallel convolutional experts sharing an input\(ResNet\-50, CIFAR\-100\): pairwise kernel cosine similarity\.

## 2Related Work

#### Sparse experts and routing\.

Sparse MoE activates a subset of experts for each input, making routing balance and stability central concerns\([Shazeer et al\. 2017](https://arxiv.org/html/2608.23794#bib.bib22);[Lepikhin et al\. 2021](https://arxiv.org/html/2608.23794#bib.bib15);[Fedus, Zoph, and Shazeer 2022](https://arxiv.org/html/2608.23794#bib.bib4)\)\. Expert\-Choice routing reverses token\-to\-expert assignment and gives experts explicit capacity\([Zhou et al\. 2022](https://arxiv.org/html/2608.23794#bib.bib27)\); differentiable and soft alternatives reduce discontinuities in hard assignment\([Hazimeh et al\. 2021](https://arxiv.org/html/2608.23794#bib.bib7);[Puigcerver et al\. 2024](https://arxiv.org/html/2608.23794#bib.bib19)\)\. Vision MoE primarily applies these ideas to token routing in transformers\([Riquelme et al\. 2021](https://arxiv.org/html/2608.23794#bib.bib20);[Liu et al\. 2024](https://arxiv.org/html/2608.23794#bib.bib17)\)\. MoCE borrows expert\-side support selection and aggregate balancing, but applies them along the channel axis of a pointwise projection: channels play the role that tokens play in MoE, and each expert selects its own top\-kkof them\. Token features vary with every input and are therefore ordinarily routed dynamically; whether channel membership benefits similarly from dynamic routing is what our membership experiment tests\.

#### Structured sparse pointwise projection\.

Structured sparsity learning removes groups of weights during training\([Wen et al\. 2016](https://arxiv.org/html/2608.23794#bib.bib24)\)\. CondenseNet learns group\-sparse1×11\\times 1connectivity and freezes it for inference\([Huang et al\. 2018](https://arxiv.org/html/2608.23794#bib.bib10)\); movement\-based and wiring\-based methods also learn which connections survive optimization\([Sanh, Wolf, and Rush 2020](https://arxiv.org/html/2608.23794#bib.bib21);[Wortsman, Farhadi, and Rastegari 2019](https://arxiv.org/html/2608.23794#bib.bib25)\)\. MoCE shares the static\-support principle but constrains each routed output to normalized nonnegative weights and retains one scalar of input dependence that modulates their concentration\. It should therefore be read as a sparse projection with an adaptive mixing rule, rather than as a replacement for the broader structured\-pruning literature\.

#### Conditional computation in CNNs\.

Channel gating, dynamic pruning, and feature suppression select or reweight channels per input\([Hua et al\. 2019](https://arxiv.org/html/2608.23794#bib.bib9);[Gao et al\. 2019](https://arxiv.org/html/2608.23794#bib.bib6);[Bejnordi et al\. 2020](https://arxiv.org/html/2608.23794#bib.bib1);[Gao et al\. 2024](https://arxiv.org/html/2608.23794#bib.bib5)\)\. Dynamic group connectivity and joint spatial–channel gating change the active operator structure\([Su et al\. 2020](https://arxiv.org/html/2608.23794#bib.bib23);[Li et al\. 2021](https://arxiv.org/html/2608.23794#bib.bib16)\)\. Pick\-or\-Mix performs fine\-grained per\-input channel sampling\([Kumar et al\. 2024](https://arxiv.org/html/2608.23794#bib.bib14)\), while Squeeze\-and\-Excitation recalibrates channels from global descriptors\([Hu, Shen, and Sun 2018](https://arxiv.org/html/2608.23794#bib.bib8)\)\. CondConv and Dynamic Convolution form input\-dependent mixtures of dense kernels\([Yang et al\. 2019](https://arxiv.org/html/2608.23794#bib.bib26);[Chen et al\. 2020](https://arxiv.org/html/2608.23794#bib.bib2)\)\. MoCE instead removes channel connectivity and keeps the data\-dependent component scalar and support preserving\.

## 3Mixture of Channel Experts

![Refer to caption](https://arxiv.org/html/2608.23794v1/moce.png)Figure 2:MoCE layer\.E−1E\\\!\-\\\!1routed experts over learned static supports, plus one residual aggregate\.### 3\.1Setting and Scope

LetX∈ℝB×C×H×WX\\in\\mathbb\{R\}^\{B\\times C\\times H\\times W\}and consider a pointwise map fromCCinput channels toE≤CE\\leq Coutputs\. A dense layer computes

Yo​\(h,w\)=∑c=1CWo,c​Xc​\(h,w\)Y\_\{o\}\(h,w\)=\\sum\_\{c=1\}^\{C\}W\_\{o,c\}X\_\{c\}\(h,w\)\(1\)withC​E​H​WCEHWMACs\. MoCE emits the same shape usingE−1E\-1routed experts and one residual aggregate\.

MoCE applies to any pointwise projection withE≤CE\\leq C; the ratios=C/Es=C/Eis a property of the layer, not of the method\. In ResNet we replace the bottleneck entrance projection, wheressis44for most layers and11or22at stage boundaries\. In EfficientViT we replace the second feed\-forward projection of each ConvMlp block, wheres=2s=2\. The method does not replace spatial convolution\.

### 3\.2Learning Static Supports

Routing logitsL∈ℝ\(E−1\)×CL\\in\\mathbb\{R\}^\{\(E\-1\)\\times C\}assign one preference vector to each routed expert\. Its support is

𝒮e=TopK⁡\(Le,k\)\.\\mathcal\{S\}\_\{e\}=\\operatorname\{TopK\}\(L\_\{e\},k\)\.\(2\)The forward pass uses this hard support\. The task loss differentiates through the selected logit values used by the mixer, while the full\-support coverage loss in Sec\.[3\.4](https://arxiv.org/html/2608.23794#S3.SS4)differentiates through every logit\. No gradient is assigned through the discrete indices themselves; supports change when the continuous logits reorder during training\. At inference, each𝒮e\\mathcal\{S\}\_\{e\}is precomputed and fixed, so the layer requires neither top\-kksearch nor data\-dependent branching, and its memory\-access pattern can be scheduled ahead of time\.

### 3\.3Input\-Adaptive Mixing

Global average pooling givesz=GAP⁡\(X\)∈ℝB×Cz=\\operatorname\{GAP\}\(X\)\\in\\mathbb\{R\}^\{B\\times C\}\. A two\-layer gateg:ℝk→ℝg:\\mathbb\{R\}^\{k\}\\rightarrow\\mathbb\{R\}, shared by the experts within a layer, receives thekkdescriptors on the expert’s own support and predicts

τe​\(X\)=τmin\+\(τmax−τmin\)​σ​\(g⁡\(z𝒮e\)\)\.\\tau\_\{e\}\(X\)=\\tau\_\{\\min\}\+\(\\tau\_\{\\max\}\-\\tau\_\{\\min\}\)\\,\\sigma\\\!\\left\(g\(z\_\{\\mathcal\{S\}\_\{e\}\}\)\\right\)\.\(3\)Becauseggis not permutation invariant, the ordering of its input is part of the specification: thekkdescriptors are supplied in order of descending routing logit, so positioniiof the gate input always corresponds to the expert’sii\-th most preferred channel\. This shared positional meaning is what makes a single gate sensible across experts, and the same rule is applied to the example\-dependent support of Sec\.[4\.5](https://arxiv.org/html/2608.23794#S4.SS5)\. The selected logits define mixing coefficients

ae,i​\(X\)=exp⁡\(Le,i/τe​\(X\)\)∑j∈𝒮eexp⁡\(Le,j/τe​\(X\)\),i∈𝒮e,a\_\{e,i\}\(X\)=\\frac\{\\exp\\\!\\big\(L\_\{e,i\}/\\tau\_\{e\}\(X\)\\big\)\}\{\\sum\_\{j\\in\\mathcal\{S\}\_\{e\}\}\\exp\\\!\\big\(L\_\{e,j\}/\\tau\_\{e\}\(X\)\\big\)\},\\quad i\\in\\mathcal\{S\}\_\{e\},\(4\)and the routed output is

Ye​\(h,w\)=∑i∈𝒮eae,i​\(X\)​Xi​\(h,w\)\.Y\_\{e\}\(h,w\)=\\sum\_\{i\\in\\mathcal\{S\}\_\{e\}\}a\_\{e,i\}\(X\)\\,X\_\{i\}\(h,w\)\.\(5\)The weights are shared over spatial locations and vary across examples only throughτe​\(X\)\\tau\_\{e\}\(X\)\. Small temperatures concentrate mass on the selected channel with the largest routing logit; large temperatures approach a uniform average over the support\. The temperature therefore changes the entropy of fixed routing preferences, not the maximum activation value\. For a fixedLeL\_\{e\}, the reachable weight vectors trace a one\-dimensional curve on the\(k−1\)\(k\\\!\-\\\!1\)\-simplex along which the ordering of the weights is invariant: per expert and example, the adaptive capacity of the layer is exactly one scalar, which is why it is nearly free to compute\.

For any positive constantτ\\tau,softmax⁡\(Le/τ\)\\operatorname\{softmax\}\(L\_\{e\}/\\tau\)can be represented assoftmax⁡\(Le′\)\\operatorname\{softmax\}\(L^\{\\prime\}\_\{e\}\)withLe′=Le/τL^\{\\prime\}\_\{e\}=L\_\{e\}/\\tau, and positive rescaling leaves top\-kkunchanged\. Consequently, theτ≡1\\tau\\equiv 1ablation has the same static representational class as any model with a learned input\-independent temperature\. Optimization and interaction with the coverage term can still differ, so we use this equivalence to define a function\-class\-matched static control rather than to claim a causal proof\. The measured gap nevertheless provides direct evidence that example\-dependent concentration is useful beyond static sparse mixing\. Two further properties keep the temperature from being redundant in the full model: it depends on the input throughz𝒮ez\_\{\\mathcal\{S\}\_\{e\}\}, which no reparameterization of a staticLLcan express; and it dividesLLin Eq\.[4](https://arxiv.org/html/2608.23794#S3.E4)but not in the coverage objective, so it decouples how sharply an expert mixes from how sharply the regularizer perceives it to mix\.

Equation[5](https://arxiv.org/html/2608.23794#S3.E5)follows the softmax\-mixing principle of attention, with the weights derived from learned routing preferences rather than query–key interactions and computed over a sparse static support\. It is also a convex mixture, and therefore strictly less expressive than an unconstrained signed projection on the same support; we treat this as a structured regularizer with an execution benefit, and Sec\.[4\.5](https://arxiv.org/html/2608.23794#S4.SS5)reports how much of the dense projection’s function is actually required\. Routing behavior is visualized in Figure[3](https://arxiv.org/html/2608.23794#S4.F3)\. Figure[4](https://arxiv.org/html/2608.23794#S4.F4)shows the distribution of learned temperatures over all MoCE experts\.

### 3\.4Residual Coverage and Regularization

Let𝒰=\{1,…,C\}∖⋃e𝒮e\\mathcal\{U\}=\\\{1,\\ldots,C\\\}\\setminus\\bigcup\_\{e\}\\mathcal\{S\}\_\{e\}\. The residual output is

Yres​\(h,w\)=\{\|𝒰\|−1​∑i∈𝒰Xi​\(h,w\),\|𝒰\|\>0,0,\|𝒰\|=0\.Y\_\{\\mathrm\{res\}\}\(h,w\)=\\begin\{cases\}\|\\mathcal\{U\}\|^\{\-1\}\\sum\_\{i\\in\\mathcal\{U\}\}X\_\{i\}\(h,w\),&\|\\mathcal\{U\}\|\>0,\\\\ 0,&\|\\mathcal\{U\}\|=0\.\\end\{cases\}\(6\)Thus every input channel contributes either through a routed expert or through a compressed residual summary\. On average,7\.7%7\.7\\%of input channels remain uncovered across the replaced layers\. Replacing the mean with a temperature\-gated aggregation over𝒰\\mathcal\{U\}changed accuracy negligibly, so we keep the cheaper form\.

To discourage concentration on a small channel subset, definepe,c=Softmax⁡\(Le\)cp\_\{e,c\}=\\operatorname\{Softmax\}\(L\_\{e\}\)\_\{c\}anduc=\(E−1\)−1​∑epe,cu\_\{c\}=\(E\-1\)^\{\-1\}\\sum\_\{e\}p\_\{e,c\}\. We optimize

ℒcov=∑c\(uc−u¯\)2u¯2\+ϵ,u¯=C−1​∑cuc,\\mathcal\{L\}\_\{\\mathrm\{cov\}\}=\\frac\{\\sum\_\{c\}\(u\_\{c\}\-\\bar\{u\}\)^\{2\}\}\{\\bar\{u\}^\{2\}\+\\epsilon\},\\qquad\\bar\{u\}=C^\{\-1\}\\sum\_\{c\}u\_\{c\},\(7\)withℒ=ℒtask\+λ​ℒcov\\mathcal\{L\}=\\mathcal\{L\}\_\{\\mathrm\{task\}\}\+\\lambda\\mathcal\{L\}\_\{\\mathrm\{cov\}\}\. This regularizer equalizes aggregate soft usage; it does not guarantee disjoint hard supports, and Fig\.[3](https://arxiv.org/html/2608.23794#S4.F3)reports the resulting overlap and usage empirically\. Its contribution to coverage alone should not be overstated: with\(E−1\)​k\(E\-1\)kselections spread overCCchannels, even independent uniform supports would leave only a modest fraction uncovered, so the regularizer improves an already\-high baseline\. It also supplies gradients to unselected logits and thereby permits their ordering, and hence the hard supports, to change during training\.

### 3\.5Cost, Break\-Even, and Bandwidth

The dominant per\-layer spatial work is

MMoCE=\(E−1\)​k​H​W\+C​H​W\+\|𝒰\|​H​W\+Mg,M\_\{\\mathrm\{MoCE\}\}=\(E\-1\)kHW\+CHW\+\|\\mathcal\{U\}\|HW\+M\_\{g\},\(8\)where the first term is routed mixing, the second is global pooling, the third is the residual reduction, andMgM\_\{g\}is the small non\-spatial gate\. Relative toMdense=C​E​H​WM\_\{\\mathrm\{dense\}\}=CEHW, the leading term is approximatelyk/Ck/C; with\|𝒰\|≤C\|\\mathcal\{U\}\|\\leq C, a conservative upper bound on the ratio is

MMoCEMdense≲kC\+2E\.\\frac\{M\_\{\\mathrm\{MoCE\}\}\}\{M\_\{\\mathrm\{dense\}\}\}\\lesssim\\frac\{k\}\{C\}\+\\frac\{2\}\{E\}\.\(9\)The residual and pooling overheads scale only linearly inCCwhile the dense projection scales asC​ECE, so the advantage grows with input width\. Using the measured\|𝒰\|≈0\.077​C\|\\mathcal\{U\}\|\\approx 0\.077Cinstead of the worst case tightens the second term to≈1\.08/E\\approx 1\.08/E; all reported MoCE MAC values use the conservative bound\.

#### Break\-even\.

Parity requires\(E−1\)​k\+2​C=C​E\(E\-1\)k\+2C=CE, giving

k⋆=C⁡\(E−2\)E−1\.k^\{\\star\}=\\frac\{C\(E\-2\)\}\{E\-1\}\.\(10\)For the commons=4s=4projections this isk⋆=252k^\{\\star\}=252atC=256C=256andk⋆=2044k^\{\\star\}=2044atC=2048C=2048\. The threshold is smallest at the64→6464\\rightarrow 64stage\-entry projection, wherek⋆=63k^\{\\star\}=63, and exceeds250250for every other replaced layer\. Atk=8k=8the layer therefore operates well below parity everywhere, which means that within this design the ceiling onkkis representational rather than computational—a point the sparsity sweep in Sec\.[4\.6](https://arxiv.org/html/2608.23794#S4.SS6)confirms from the other direction\.

#### Instantiation\.

On ResNet\-50 the sixteen replaced projections account for706\.5706\.5M spatial MACs,17\.2%17\.2\\%of the network budget, and MoCE reduces them to20\.120\.1M: a35×35\\timesreduction on the replaced operators and16\.7%16\.7\\%of the whole network \(Table[1](https://arxiv.org/html/2608.23794#S3.T1)\)\. Per\-layer costs follow directly from Eq\.[8](https://arxiv.org/html/2608.23794#S3.E8); the omitted non\-spatial gate contributes0\.000270\.00027G and does not affect the rounded totals\. The saving grows to20\.8%20\.8\\%on ResNet\-152, since deeper backbones place proportionally more replaced projections in wide stages\.

#### Storage\.

Training stores the full routing matrix,\(E−1\)​C\(E\-1\)Cvalues per layer, so that supports can migrate; at that point MoCE is not a parameter reduction, and its training\-time count sits within0\.050\.05M of the dense model\. Deployment retains only the\(E−1\)​k\(E\-1\)kselected logits, their integer indices, and the shared gate\. In ResNet\-50, the4\.334\.33M weights of the replaced dense projections are replaced at deployment by approximately30\.130\.1k retained logits and6060KB of 16\-bit support indices, reducing the complete model from25\.5625\.56M to21\.2621\.26M learned parameter values; the corresponding reduction is19\.6%19\.6\\%on ResNet\-101 and20\.5%20\.5\\%on ResNet\-152\. We report both training and deployment counts throughout, and the index overhead wherever total storage is at issue\.

#### Why the time saving is smaller than the MAC saving\.

Both the dense projection and MoCE move\(C\+E\)​H​W\(C\+E\)HWactivation elements per example\. Sparsification removes arithmetic but not traffic: the pooling reads allCCchannels and the residual reads\|𝒰\|\|\\mathcal\{U\}\|of them, so activation traffic remainsΘ⁡\(C​H​W\)\\Theta\(CHW\)independently ofkk\. A roofline\-style estimate from these idealized traffic counts, at the batch size512512used in Sec\.[4\.7](https://arxiv.org/html/2608.23794#S4.SS7)where weight traffic is heavily amortized, places the dense2048→5122048\\rightarrow 512projection near205205FLOP/byte in FP32 and MoCE near1\.61\.6FLOP/byte, against an accelerator balance point around1313FLOP/byte\. This suggests that MoCE shifts the projection from a compute\-dominated toward a bandwidth\-dominated regime, which would explain why a128×128\\timesMAC reduction on the deepest replaced projection yields a3\.0×3\.0\\timesmeasured speedup\. We present this as an explanatory estimate rather than a profiler\-verified attribution; its practical implication is that reducing activation traffic—fusing the pooling into the gather, and evaluating the gate as a single batched product—offers more headroom than further reductions inkk\.

Table 1:Cost and storage, accounted as in Sec\.[3\.5](https://arxiv.org/html/2608.23794#S3.SS5): MACs are network totals with each replaced projection counted as\[\(E−1\)​k\+2​C\]​H​W\[\(E\{\-\}1\)k\+2C\]HWunder the conservative bound\|𝒰\|=C\|\\mathcal\{U\}\|\{=\}C\. MoCE parameters are training/deployment counts of learned values\.

## 4Experiments

### 4\.1Setup

We evaluate ImageNet\-1K\([Deng et al\. 2009](https://arxiv.org/html/2608.23794#bib.bib3)\), CIFAR\-100\([Krizhevsky 2009](https://arxiv.org/html/2608.23794#bib.bib13)\), ImageNet\-to\-CIFAR transfer, and EfficientViT\([Liu et al\. 2023](https://arxiv.org/html/2608.23794#bib.bib18)\)\. ImageNet models train for120120epochs with batch size512512, SGD with momentum0\.90\.9and weight decay10−410^\{\-4\}, base learning rate0\.10\.1with cosine decay, and standard random\-resized\-crop to2242224^\{2\}with horizontal flipping; no label smoothing, mixup, or CutMix is used\. Transfer fine\-tuning uses2020epochs at learning rate0\.050\.05; during fine\-tuning the support indices are frozen while the selected logits, temperature gates, and all remaining network parameters are updated\. We deliberately adopt this original\-style ResNet recipe rather than a high\-accuracy one: it matches the protocol under which the compared methods were developed, and it isolates the architectural change from the training pipeline\. Our75\.98%75\.98\\%ResNet\-50 baseline is consistent with standard original\-recipe implementations\.

Accuracies in the main tables are means and standard deviations over three seeds; ablation entries are single runs and are described as such throughout\. With three runs we make no claim of statistical significance; we interpret differences that are small relative to the observed run\-to\-run variation as accuracy preservation rather than as improvement\. Bold typeface is used only for deterministic quantities \(MACs and parameter counts\), never for accuracies\.

All ResNet bottleneck entrance projections are replaced \(1616,3333, and5050layers for ResNet\-50/101/152\), and BatchNorm and the surrounding block structure are unchanged\. Unless stated otherwise,k=8k=8,τ∈\[0\.2,6\.0\]\\tau\\in\[0\.2,6\.0\], andλ=5×10−4\\lambda=5\\times 10^\{\-4\}on ImageNet and0\.050\.05on CIFAR\-100\. Routing logits are initialized i\.i\.d\. from a zero\-mean Gaussian, and the temperature gate is a two\-layer MLP of hidden widthkkwith ReLU and biases, shared across the experts of a layer\. EfficientViT usesk=64k=64and replaces the second2​d→d2d\\rightarrow dfeed\-forward projection in each ConvMlp block, in bothffn0andffn1across all stages, leaving expansion, activation, and attention modules unchanged\. CIFAR\-100 models trained from scratch use a CIFAR\-style stem, a single3×33\\times 3convolution with no max\-pooling, giving stage resolutions of32/16/8/432/16/8/4\. The supplement gives the complete training recipes for the CIFAR\-from\-scratch and EfficientViT settings—epochs, batch size, optimizer, learning\-rate schedule, warm\-up, weight decay, and augmentation—together with the routing\-logit initialization variance, the Gumbel temperature schedule used for the dynamic ablation, the warm\-up and timed\-iteration counts for the latency measurements, and the GPU model with framework, CUDA, and cuDNN versions\.

### 4\.2ImageNet\-1K

Table[2](https://arxiv.org/html/2608.23794#S4.T2)reports matched dense and MoCE models\. On ResNet\-50 the Top\-1 point estimate increases by0\.730\.73points while MACs fall by16\.7%16\.7\\%and deployed learned parameters by16\.8%16\.8\\%\. On ResNet\-101 accuracy is preserved within the observed run\-to\-run variation at19\.7%19\.7\\%lower MACs, and on ResNet\-152 the point estimate increases by0\.460\.46points at20\.8%20\.8\\%lower MACs\. Because MoCE improves or preserves accuracy while reducing computation and deployment storage simultaneously, the relevant comparison is the joint operating point rather than the accuracy delta alone\.

Table 2:ImageNet\-1K; accuracies are mean±\\pmstandard deviation over three seeds\. MoCE parameters are training/deployment counts of learned values; deployment additionally stores packed support indices \(≈60\\approx 60KB for ResNet\-50\)\.
### 4\.3Conditional Channel Methods

Table[3](https://arxiv.org/html/2608.23794#S4.T3)compares methods retrained under the same ResNet\-50 protocol\. MoCE and SE have essentially the same accuracy point estimate, but SE adds resources whereas MoCE removes them: MoCE reaches SE\-level accuracy with6\.86\.8M fewer deployed learned parameters and17%17\\%fewer MACs\. CondConv’s parameter cost is consistent with our motivation that duplicating dense operators can be an inefficient route to specialization\. Pick\-or\-Mix reaches slightly lower MACs; MoCE has the higher accuracy point estimate and comparable measured batch time\. These comparisons establish the trade\-off against conditional channel methods, while the distinction from static structured sparse projections is architectural rather than a claim of universal superiority\.

Table 3:Matched ResNet\-50 comparison on ImageNet\-1K\. MoCE parameters are training/deployment counts of learned values\.
### 4\.4CIFAR\-100, Transfer, and EfficientViT

Table[4](https://arxiv.org/html/2608.23794#S4.T4)reports CIFAR\-100 trained from scratch and ImageNet\-to\-CIFAR transfer\. MACs fall by17\.217\.2–21\.0%21\.0\\%throughout\. The accuracy point estimate increases clearly for ResNet\-50 \(\+0\.91\+0\.91from scratch,\+1\.03\+1\.03transferring to CIFAR\-100\) and the margin narrows with depth, with ResNet\-152 preserved within observed variation in both from\-scratch CIFAR\-100 and transfer to CIFAR\-100\. Transfer is the more informative setting for the design claim, since supports are learned on ImageNet and reused unchanged: they remain effective after fine\-tuning on a different label space, which is what one expects if a channel’s role is comparatively stable across inputs and tasks\.

Table[5](https://arxiv.org/html/2608.23794#S4.T5)reports EfficientViT, where the replaced layer is the second feed\-forward projection rather than a bottleneck entrance\. MACs fall by1818–22%22\\%on M5/M3/M2, and all three accuracy differences are small relative to the observed run\-to\-run variation, so we read the result as accuracy preservation at lower computation—sufficient to establish that MoCE applies beyond ResNet\-style backbones to token\-wise linear projections, without claiming an accuracy gain there\.

Table 4:CIFAR\-100 from scratch \(left\) and ImageNet→\\rightarrowCIFAR transfer \(right\); three seeds\.Table 5:MoCE in EfficientViT on CIFAR\-100 \(k=64k\{=\}64, second feed\-forward projection,s=2s\{=\}2\), three seeds\. Parameter counts are training\-time; the deployment packing reported for ResNet is not applied here\.
### 4\.5Where Input Dependence Pays

The upper half of Table[6](https://arxiv.org/html/2608.23794#S4.T6)contrasts two ways of using the input: to choose which channels an expert reads, or to choose how it combines them\. All entries in this subsection are single runs on ResNet\-50/CIFAR\-100\.

#### Input\-conditioned support selection\.

The dynamic variant augments the full MoCE model: it retains the input\-adaptive temperature and the residual aggregate, and replaces only the static support of Eq\.[2](https://arxiv.org/html/2608.23794#S3.E2)with an input\-conditioned one\. Both downstream components follow the example\-dependent support: the temperature is evaluated fromz𝒮e​\(X\)z\_\{\\mathcal\{S\}\_\{e\}\(X\)\}, and the residual set is recomputed per example as𝒰⁡\(X\)=\{1,…,C\}∖⋃e𝒮e​\(X\)\\mathcal\{U\}\(X\)=\\\{1,\\ldots,C\\\}\\setminus\\bigcup\_\{e\}\\mathcal\{S\}\_\{e\}\(X\)\. Because a router must score every candidate channel before selecting, it reads the full pooled descriptorz=GAP⁡\(X\)z=\\operatorname\{GAP\}\(X\)and adds a rank\-ddfactorization of the expert\-by\-channel offset matrix to the routing logits:

Le​\(X\)c=Le,c\+⟨qe⊙h⁡\(z\),vc⟩,𝒮e​\(X\)=TopK⁡\(Le​\(X\),k\),L\_\{e\}\(X\)\_\{c\}=L\_\{e,c\}\+\\big\\langle q\_\{e\}\\odot h\(z\),\\,v\_\{c\}\\big\\rangle,\\quad\\mathcal\{S\}\_\{e\}\(X\)=\\operatorname\{TopK\}\\\!\\big\(L\_\{e\}\(X\),k\\big\),\(11\)whereh⁡\(z\)=ReLU⁡\(Wh​z\+bh\)∈ℝdh\(z\)=\\mathrm\{ReLU\}\(W\_\{h\}z\+b\_\{h\}\)\\in\\mathbb\{R\}^\{d\}is a shared bottleneck of the descriptor,vc∈ℝdv\_\{c\}\\in\\mathbb\{R\}^\{d\}is a learned channel embedding andqe∈ℝdq\_\{e\}\\in\\mathbb\{R\}^\{d\}a learned expert embedding\. The low\-rank form is what keeps a per\-expert, per\-channel offset affordable: a separate dense network per expert would emit\(E−1\)​C\(E\-1\)Cvalues per example and be far larger than the projection it routes\. The router consists ofWh∈ℝd×CW\_\{h\}\\in\\mathbb\{R\}^\{d\\times C\}andbh∈ℝdb\_\{h\}\\in\\mathbb\{R\}^\{d\}, the channel embeddings\{vc\}c=1C\\\{v\_\{c\}\\\}\_\{c=1\}^\{C\}, and the expert embeddings\{qe\}e=1E−1\\\{q\_\{e\}\\\}\_\{e=1\}^\{E\-1\}, so its parameter count per layer isC​d\+d\+C​d\+\(E−1\)​d=2​C​d\+E​dCd\+d\+Cd\+\(E\-1\)d=2Cd\+Edexactly; no further biases are used\. Withd=32d=32this is0\.960\.96M over the sixteen ResNet\-50 layers\. The input\-conditioned logitsLe​\(X\)L\_\{e\}\(X\)are used both for selection and, in place ofLe,iL\_\{e,i\}in Eq\.[4](https://arxiv.org/html/2608.23794#S3.E4), for the mixing coefficients, so the variant makes both membership and mixing weights example dependent\.

During training the forward pass uses hard top\-kkassignments and gradients are supplied by a straight\-through Gumbel\-softmax relaxation of the discrete selection\([Jang, Gu, and Poole 2017](https://arxiv.org/html/2608.23794#bib.bib12)\); the input dependence comes from the router, not from the sampling noise, which supplies only the relaxation\. At evaluation the variant uses deterministic hard per\-example top\-kkwith no noise and no relaxation, and the reported timing is for that inference path\.

In the evaluated run this variant changes accuracy by\+0\.03\+0\.03points \(79\.3879\.38vs\.79\.3579\.35, mean over 3 seeds\), while the routing\-and\-gather path rises from1\.91\.9to13\.513\.5ms—a factor of7\.137\.13\. These figures time the selection and gather stages only: for MoCE the fixed indexed gather, and for the dynamic variant the router evaluation, hard per\-example top\-kk, and per\-example gather\. They exclude temperature prediction, the channel aggregation of Eq\.[5](https://arxiv.org/html/2608.23794#S3.E5), and the construction and aggregation of𝒰\\mathcal\{U\}, which are common to both and unchanged in form\. The comparison is therefore a like\-for\-like measurement of the path that dynamic membership alters, not a complete layer latency\.

#### Input\-conditioned aggregation\.

Removing the temperature gate and fixingτ≡1\\tau\\equiv 1reduces accuracy by1\.191\.19points\. Since the static control retains fully learnable logits and the complete static function class described in Sec\.[3\.3](https://arxiv.org/html/2608.23794#S3.SS3), this is consistent with example\-dependent concentration mattering beyond a better constant sharpness\.

#### Reading the two together\.

The comparison is deliberately unequal and favors the dynamic variant, which reads the fullCC\-dimensional descriptor, carries0\.960\.96M extra parameters, and makes the mixing weights example dependent as well, whereas the temperature gate reads only thekkdescriptors on the expert’s support and emits one scalar\. It therefore evaluates two practical implementations rather than providing a capacity\-matched isolation\. Its force is that the strictly more expressive and more expensive route produces no meaningful gain in the evaluated run while forfeiting static execution, whereas removing the far more constrained route—one shared two\-layer gate—reduced accuracy by1\.191\.19points\. These results suggest that, in the evaluated design, input dependence is more effective when applied to aggregation than to support membership\. Removing the residual aggregate reduced accuracy by a further0\.740\.74points\.

#### How much of the dense projection is needed\.

The support\-design rows deserve more attention than a routine ablation\.*Deterministic fixed supports*assign channels cyclically without learning:

𝒮ecyc=\{1\+\(\(\(e−1\)k\+j\)modC\):j=0,…,k−1\},\\mathcal\{S\}\_\{e\}^\{\\mathrm\{cyc\}\}=\\Big\\\{\\,1\+\\big\(\(\(e\-1\)k\+j\)\\bmod C\\big\)\\;:\\;j=0,\\ldots,k\-1\\,\\Big\\\},\(12\)which gives approximately uniform channel usage and controlled overlap—exact uniformity would requireCCto divide\(E−1\)​k\(E\-1\)k—with logits and temperature still trained\. This reaches78\.27%78\.27\\%against the dense projection’s78\.44%78\.44\\%\. A nonnegative convex mixture of eight channels per output, with supports assigned by a fixed rule, therefore recovers the dense projection to within0\.170\.17points while removing97%97\\%of its MACs\. Learning the supports adds a further1\.081\.08points and overtakes the dense projection\. The*evaluated random assignment*—each expert drawskkdistinct channels uniformly without replacement, independently across experts, with a single draw held fixed for the run—performs poorly at61\.30%61\.30\\%\. Read together, the three rows indicate that the choice of support matters and is not benign: the random assignment we evaluate degrades the representation severely, a deterministic balanced assignment nearly preserves it, and a learned one improves on the dense projection\. They also suggest substantial redundancy in these projections, since eight ofCCchannels per output suffice to match the dense layer\.

### 4\.6Sparsity, Coverage, and Temperature

The lower half of Table[6](https://arxiv.org/html/2608.23794#S4.T6)reports the three hyperparameter sweeps, also single runs\. Sparsity behaves non\-monotonically aroundk=8k=8: smaller supports \(k=4k=4\) limit expert capacity, and larger ones \(k≥10k\\geq 10\) reduce specialization\. Since break\-even lies atk⋆≥63k^\{\\star\}\\geq 63for every replaced projection \(Sec\.[3\.5](https://arxiv.org/html/2608.23794#S3.SS5)\),k=8k=8sits far below the cost\-parity point, so the binding constraint onkkis representational, not computational—a useful property, because it means the operating point was not chosen to satisfy a compute budget\.

For the coverage weight, the defaultλ=5×10−4\\lambda=5\\times 10^\{\-4\}gives the best point estimate\. Differences across the interior of the sweep are small relative to the run\-to\-run variation observed in the main tables and we do not read a trend into them; only the endpoints are informative, in that too weak a weight permits concentration and too strong a weight over\-constrains routing\. Temperature bounds behave as the reparameterization argument in Sec\.[3\.3](https://arxiv.org/html/2608.23794#S3.SS3)predicts: a global rescaling ofLLis absorbed, so the interval matters only as a constraint, and the widest setting\[0\.1,8\]\[0\.1,8\]is the worst\. We adopt\[0\.2,6\.0\]\[0\.2,6\.0\], which spans the useful range while remaining numerically stable\.

Table 6:Ablations on ResNet\-50; all entries are single runs\. Components, support design, and temperature bounds on CIFAR\-100; sparsitykkand coverage weightλ\\lambdaon ImageNet\-1K\. Timings cover the inference\-time routing and gather path only\.Table 7:FP32 batch time \(ms\), batch512512, ResNet\-50 eval mode; medians after warm\-up with device synchronization around each measurement\. PiX and MoCE medians fall within their spread; IQRs, hardware, and library versions are in the supplement\.![Refer to caption](https://arxiv.org/html/2608.23794v1/channels_and_experts.png)Figure 3:Routing in the first MoCE layer\.Top left: top\-kkoverlap between experts\. Top right: cosine similarity of routing\-logit rows\. Bottom: experts selecting each channel, against the uniform expectation\.![Refer to caption](https://arxiv.org/html/2608.23794v1/temp_regimes.png)Figure 4:Learned temperatures across experts,spanning concentrated, intermediate, and near\-uniform regimes rather than collapsing to one mode\.
### 4\.7Measured Batch Execution Time

Table[7](https://arxiv.org/html/2608.23794#S4.T7)reports FP32 batch execution time at batch size512512under one common evaluation setup, with mixed precision, graph compilation, and CUDA graphs disabled\. MoCE and Pick\-or\-Mix are effectively tied end to end, and both are faster than the dense model\. Across all replaced operators, MoCE reduces time from19\.6319\.63to15\.6015\.60ms\. The advantage increases in wide projections: the representative2048→5122048\\rightarrow 512module runs at0\.331×0\.331\\timesthe dense time, consistent with thek/Ck/Cscaling of Eq\.[9](https://arxiv.org/html/2608.23794#S3.E9)asCCgrows from256256to20482048\.

The4\.6%4\.6\\%network\-level gain is smaller than the MAC reduction\. Section[3\.5](https://arxiv.org/html/2608.23794#S3.SS5)offers an explanation rather than a demonstration: the sparse path still reads the input activation tensor, so an idealized arithmetic\-intensity estimate places the operator on the bandwidth\-dominated side of the roofline, where runtime is governed by traffic thatkkdoes not reduce\. The honest efficiency claim at the network level is therefore a1717–21%21\\%MAC reduction with a4\.6%4\.6\\%end\-to\-end speedup, and the analysis suggests that closing the remainder is more likely to come from memory\-traffic fusion than from smaller supports\.

## 5Conclusion

MoCE replaces a dense pointwise projection with channel supports that are learned during training and frozen afterwards, together with one input\-dependent temperature per expert\. Across the evaluated settings it preserves or improves accuracy while reducing computation, deployment storage, and measured execution time\. The design finding is that input dependence is worth more for adjusting how a fixed set of channels is mixed than for changing which channels are read: a strictly more expressive input\-conditioned router gained nothing measurable at7\.13×7\.13\\timesthe routing\-and\-gather cost, while removing the far cheaper temperature gate cost1\.191\.19points in the same run\. Two results invite follow\-up: a fixed cyclic assignment of eight channels per output already matches the dense projection to within0\.20\.2points, and once sparsified the layer is bounded by memory traffic rather than arithmetic\.

## References

- Bejnordi et al\. \(2020\)Bejnordi, B\. E\.; et al\. 2020\.Batch\-Shaping For Learning Cconditional Channel Gated Networks\.In*International Conference on Learning Representations \(ICLR\)*\.
- Chen et al\. \(2020\)Chen, Y\.; et al\. 2020\.Dynamic Convolution: Attention over Convolution Kernels\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*\.
- Deng et al\. \(2009\)Deng, J\.; Dong, W\.; Socher, R\.; Li, L\.\-J\.; Li, K\.; and Fei\-Fei, L\. 2009\.ImageNet: A Large\-Scale Hierarchical Image Database\.In*Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\)*\.
- Fedus, Zoph, and Shazeer \(2022\)Fedus, W\.; Zoph, B\.; and Shazeer, N\. 2022\.Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity\.In*International Conference on Learning Representations \(ICLR\)*\.
- Gao et al\. \(2024\)Gao, S\.; Zhang, Y\.; Huang, F\.; and Huang, H\. 2024\.BilevelPruning: Unified Dynamic and Static Channel Pruning for Convolutional Neural Networks\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 16090–16100\.
- Gao et al\. \(2019\)Gao, X\.; et al\. 2019\.Dynamic Channel Pruning: Feature Boosting and Suppression\.In*International Conference on Learning Representations \(ICLR\)*\.
- Hazimeh et al\. \(2021\)Hazimeh, H\.; Zhao, Z\.; Chowdhery, A\.; Hong, L\.; and Chi, E\. H\. 2021\.DSelect\-k: Differentiable Selection in the Mixture of Experts with Applications to Multi\-Task Learning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Hu, Shen, and Sun \(2018\)Hu, J\.; Shen, L\.; and Sun, G\. 2018\.Squeeze\-and\-Excitation Networks\.In*Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\)*\.
- Hua et al\. \(2019\)Hua, W\.; Zhou, Y\.; De Sa, C\.; and Zhang, Z\. 2019\.Channel Gating Neural Networks\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Huang et al\. \(2018\)Huang, G\.; Liu, S\.; van der Maaten, L\.; and Weinberger, K\. Q\. 2018\.CondenseNet: An Efficient DenseNet Using Learned Group Convolutions\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2752–2761\.
- Huang et al\. \(2025\)Huang, Y\.; Ye, P\.; Huang, C\.; Cao, J\.; Zhang, L\.; Li, B\.; Yu, G\.; and Chen, T\. 2025\.DeRS: Towards Extremely Efficient Upcycled Mixture\-of\-Experts Models\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 10056–10066\.
- Jang, Gu, and Poole \(2017\)Jang, E\.; Gu, S\.; and Poole, B\. 2017\.Categorical Reparameterization with Gumbel\-Softmax\.In*International Conference on Learning Representations \(ICLR\)*\.
- Krizhevsky \(2009\)Krizhevsky, A\. 2009\.Learning Multiple Layers of Features from Tiny Images\.Technical Report TR\-2009, University of Toronto\.
- Kumar et al\. \(2024\)Kumar, A\.; Kim, D\.; Park, J\.; and Behera, L\. 2024\.Pick\-or\-Mix: Dynamic Channel Sampling for ConvNets\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*\.
- Lepikhin et al\. \(2021\)Lepikhin, D\.; Lee, H\.; Xu, Y\.; Chen, D\.; Firat, O\.; Huang, Y\.; Krikun, M\.; Shazeer, N\.; and Chen, Z\. 2021\.GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding\.In*International Conference on Learning Representations \(ICLR\)*\.
- Li et al\. \(2021\)Li, F\.; Li, G\.; He, X\.; and Cheng, J\. 2021\.Dynamic Dual Gating Neural Networks\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\)*\.
- Liu et al\. \(2024\)Liu, T\.; Blondel, M\.; Riquelme, C\.; and Puigcerver, J\. 2024\.Routers in Vision Mixture of Experts: An Empirical Study\.*arXiv preprint arXiv:2401\.15969*\.
- Liu et al\. \(2023\)Liu, X\.; et al\. 2023\.EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*\.
- Puigcerver et al\. \(2024\)Puigcerver, J\.; et al\. 2024\.From Sparse To Soft Mixtures Of Experts\.In*International Conference on Learning Representations \(ICLR\)*\.
- Riquelme et al\. \(2021\)Riquelme, C\.; Puigcerver, J\.; Mustafa, B\.; Neumann, M\.; and Jenatton, R\. 2021\.Scaling Vision with Sparse Mixture of Experts\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Sanh, Wolf, and Rush \(2020\)Sanh, V\.; Wolf, T\.; and Rush, A\. M\. 2020\.Movement Pruning: Adaptive Sparsity by Fine\-Tuning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, volume 33, 20378–20389\.
- Shazeer et al\. \(2017\)Shazeer, N\.; Mirhoseini, A\.; Maziarz, K\.; Davis, A\.; Le, Q\. V\.; Hinton, G\.; and Dean, J\. 2017\.Outrageously Large Neural Networks: The Sparsely\-Gated Mixture\-of\-Experts Layer\.In*International Conference on Learning Representations \(ICLR\)*\.
- Su et al\. \(2020\)Su, H\.; et al\. 2020\.Dynamic Group Convolution for Accelerating Convolutional Neural Networks\.In*European Conference on Computer Vision \(ECCV\)*\.
- Wen et al\. \(2016\)Wen, W\.; Wu, C\.; Wang, Y\.; Chen, Y\.; and Li, H\. 2016\.Learning Structured Sparsity in Deep Neural Networks\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, volume 29\.
- Wortsman, Farhadi, and Rastegari \(2019\)Wortsman, M\.; Farhadi, A\.; and Rastegari, M\. 2019\.Discovering Neural Wirings\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, volume 32\.
- Yang et al\. \(2019\)Yang, B\.; et al\. 2019\.CondConv: Conditionally Parameterized Convolutions for Efficient Inference\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Zhou et al\. \(2022\)Zhou, Y\.; Lei, T\.; Liu, H\.; and Huang, Y\. 2022\.Mixture\-of\-Experts with Expert Choice Routing\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.