SpecDrop: Parameter-Free Category-Conditioned Routing for Modular Specialization

# SpecDrop: Parameter-Free Category-Conditioned Routing for Modular Specialization
Source: [https://arxiv.org/html/2608.04084](https://arxiv.org/html/2608.04084)
Boyao Wang Machine Learning Department Carnegie Mellon University bryanw2@cs\.cmu\.edu &Zhihan Lei Machine Learning Department Carnegie Mellon University lexl@cs\.cmu\.edu

###### Abstract

Modular networks such as mixture\-of\-experts \(MoE\) pursue specialization through learned routers, gates, and load\-balancing losses, yet at matched total\-parameter budgets learned routers can underperform equal\-weight No\-Routing baselines\. Is the bottleneck the routing algorithm, or the alignment between training\-signal granularity and the target categories? Across four settings spanning vision and language, we find the answer tracks partition granularity, not router design\. We probe the question with SpecDrop, a fixed parameter\-free routing scheme: each ofKKbranches receives weightpap\_\{\\mathrm\{a\}\}for its assigned category and a small leakagepi\>0p\_\{\\mathrm\{i\}\}\{\>\}0otherwise, merged through a category\-independent fixed denominator, with no learned routing parameters and no auxiliary losses; the category label is required at inference\. On vision tasks where each image has one superclass label \(CIFAR\-100 on ResNet\-110; ImageNet\-1K on ViT\-S/16\), SpecDrop reaches79\.23%\\mathbf\{79\.23\\%\}on CIFAR\-100 and79\.89%\\mathbf\{79\.89\\%\}on ImageNet\-1K, exceeding parameter\-matched baselines that do not use the label \(\+4\.75\+4\.75over dense on CIFAR\-100;\+6\.53\+6\.53over the No\-Routing\+SE control on ImageNet\-1K\)\. These gains quantify what category supervision buys when deployed through routing — not an advantage over label\-aware deployments of the baselines: given the same label, masking a dense model’s outputs is stronger for accuracy alone \(85\.285\.2/83\.783\.7\)\. SpecDrop’s contribution is converting the label into trained\-in modular structure:58%58\\%/100%100\\%branch–category alignment, and masking gains of0\.000\.00\(CIFAR\) /\+1\.06\+1\.06\(ImageNet\) — the output\-space restriction is largely internalized during training\. SpecDrop also reaches a higher top\-1 than every label\-free multi\-branch routing baseline we evaluate at this parameter budget\. On fuzzy partitions, where training units span multiple categories \(SlimPajama\-6B language modeling with a 30M Transformer; SuperNI instruction tuning over Llama\-3\.2\-1B with LoRA\), the routing mechanism reduces to the matched No\-Routing controls within seed noise, the null our thesis predicts\. Granularity alignment, not algorithm choice, localizes when routing helps\. Code:[https://github\.com/Beryex/SpecDrop](https://github.com/Beryex/SpecDrop)\.

## 1Introduction

Modular neural networks, including mixture\-of\-experts \(MoE\)\(Jacobs et al\.,[1991](https://arxiv.org/html/2608.04084#bib.bib20); Jiang et al\.,[2024](https://arxiv.org/html/2608.04084#bib.bib21)\), parallel branches\(Szegedy et al\.,[2015](https://arxiv.org/html/2608.04084#bib.bib36); Xie et al\.,[2017](https://arxiv.org/html/2608.04084#bib.bib45)\), and multi\-head attention\(Vaswani et al\.,[2017](https://arxiv.org/html/2608.04084#bib.bib39)\), decompose model capacity across multiple parameter subsets, each meant to develop concentrated expertise on a portion of the input distribution\. When achieved, specialization yields functional decomposition: distinct experts can be inspected, ablated, or selectively deployed\.

Three lines of work attempt to recover specialization:*\(A\) Learned routing*with auxiliary losses\(Fedus et al\.,[2022](https://arxiv.org/html/2608.04084#bib.bib13)\),*\(B\) Fixed\-rule routing*with deterministic gates\(Roller et al\.,[2021](https://arxiv.org/html/2608.04084#bib.bib30)\), and*\(C\) Input\-dependent dropout*at the neuron level\(Ba and Frey,[2013](https://arxiv.org/html/2608.04084#bib.bib2)\)or block level\(Fan et al\.,[2020](https://arxiv.org/html/2608.04084#bib.bib11)\)\. Yet specialization remains elusive:Wang et al\. \([2026](https://arxiv.org/html/2608.04084#bib.bib41)\)document a persistent “standing committee” of generalist experts across sparse MoE variants\. All three intervene on the routing function or activation pattern, treating the training signal as a fixed input\.

![Refer to caption](https://arxiv.org/html/2608.04084v1/x1.png)Figure 1:Per\-branch pruning sensitivity across four settings: alignment quality predicts when category\-conditioned routing helps\.In each panel,xx\-axis indexes branches,yy\-axis categories; cell darkness shows performance drop when branchkkis removed for category\-ccsamples\. Diag\-argmaxN/MN/Mcounts categories whose most\-pruning\-sensitive branch matches the assigned branch under round\-robin \(seed4242\)\. The aligned vision partitions show dominant diagonals \(ViT BREEDS46/4646/46; CIFAR13/2013/20\), the anti\-aligned SuperNI/LoRA setting none \(0/150/15, counted over the1515of2020clusters with held\-out test tasks, App\.[E\.12](https://arxiv.org/html/2608.04084#A5.SS12)\), and fuzzy SlimPajama specializes \(6/76/7; its Book domain lacks validation coverage, App\.[E\.16](https://arxiv.org/html/2608.04084#A5.SS16)\) without an aggregate PPL gain — so the matched\-supervision gains \(Tabs\.[1](https://arxiv.org/html/2608.04084#S5.T1)–[4](https://arxiv.org/html/2608.04084#S5.T4)\) are large on the aligned partitions and null on the fuzzy ones, tracking partition granularity rather than diagonal counts alone \(Sec\.[5\.3](https://arxiv.org/html/2608.04084#S5.SS3)\)\. Each setting independently normalized; cross\-panel intensity not directly comparable — the near\-blank ViT No\-Routing panel means no branch–category cell rises above noise there\.As Figure[1](https://arxiv.org/html/2608.04084#S1.F1)shows, equal\-weight No\-Routing baselines fail to develop branch\-category specialization across all four settings despite identical architecture\. To address this gap, we propose*SpecDrop*: a fixed, parameter\-free dropout schedule that conditions each module’s activation probability on the input’s category tag \(the assigned module at high probabilitypap\_\{\\mathrm\{a\}\}, the rest at a small leakagepi\>0p\_\{\\mathrm\{i\}\}\{\>\}0, merged through a category\-independent denominator\)\. The construction has zero learned routing parameters and zero auxiliary losses, yet produces specialization that tracks category clarity\.

Whether this category\-conditioned specialization translates to performance gains depends on partition alignment: we call a partition*aligned*when each training unit carries one clean category label, and*fuzzy*when training units span multiple categories\. On aligned vision partitions \(CIFAR\-100 and ImageNet\-1K, where each image has one superclass\), SpecDrop exceeds the parameter\-matched baselines that do not use the label and reaches the highest top\-1 among the multi\-branch routing baselines we evaluate\. These gains quantify what category supervision buys when deployed through routing, not an advantage over label\-aware deployments: an information\-matched masking control \(Sec\.[6](https://arxiv.org/html/2608.04084#S6)\) shows that, given the same label, masking a dense model’s outputs is stronger for accuracy alone; what routing adds is converting the label into trained\-in modular structure\. On fuzzy partitions \(SlimPajama\-6B language modeling and SuperNI/LoRA instruction tuning over Llama\-3\.2\-1B\), SpecDrop reduces to matched\-architecture baselines within seed noise\.

Contributions\.*\(1\) SpecDrop*: a fixed, parameter\-free dropout schedule that conditions module activation on the input’s category tag, with zero learned routing parameters and zero auxiliary losses\.*\(2\) A category\-independent fixed denominatorSS*\(Prop\.[1](https://arxiv.org/html/2608.04084#Thmproposition1)\) that makes the train and test forward passes match exactly and calibrates merged\-branch magnitude to single\-branch scale, so the optional shared expert composes co\-equally with the routed mixture\.*\(3\) Empirical validation and attribution*: on aligned vision partitions SpecDrop exceeds parameter\-matched label\-free baselines \(\+4\.75\+4\.75over dense on CIFAR\-100,\+6\.53\+6\.53over the matched No\-Routing\+SE on ImageNet\-1K\) and leads the multi\-branch routing baselines we evaluate, while an information\-matched masking control separates the label’s share from routing’s and locates SpecDrop’s contribution in the induced structure; on fuzzy NLP/LoRA partitions it ties matched\-architecture baselines, identifying alignment as the binding condition\.

![Refer to caption](https://arxiv.org/html/2608.04084v1/x2.png)Figure 2:SpecDrop method overview\.Each input’s category tag selects one preferred module via a*fixed, pre\-set*assignment matrix𝐀\\mathbf\{A\}\(round\-robin, never updated by gradient descent\); the preferred module receives activation probabilitypap\_\{\\mathrm\{a\}\}while the rest receive a small leakagepi\>0p\_\{\\mathrm\{i\}\}\{\>\}0\. Routed branches are merged through a category\-independent denominatorS=pa\+\(K−1\)​piS=p\_\{\\mathrm\{a\}\}\+\(K\{\-\}1\)p\_\{\\mathrm\{i\}\}, with an optional always\-on shared expert added after normalization \(deployed CIFAR:\(pa,pi,K\)=\(0\.7,0\.3,20\)\(p\_\{\\mathrm\{a\}\},p\_\{\\mathrm\{i\}\},K\)\{=\}\(0\.7,0\.3,20\), merge weights\(0\.109,0\.047\)=\(pa,pi\)/S\(0\.109,0\.047\)=\(p\_\{\\mathrm\{a\}\},p\_\{\\mathrm\{i\}\}\)/S\)\. Zero learned routing parameters, zero auxiliary losses\.
## 2Related Work

Three research lines target specialization in modular networks: learned routing, fixed\-rule routing, and input\-dependent dropout\.

#### Learned routing\.

A learned function decides which experts each input activates, typically with auxiliary losses to enforce balanced utilization\. Switch\(Fedus et al\.,[2022](https://arxiv.org/html/2608.04084#bib.bib13)\)uses top\-1 sparse routing; V\-MoE\(Riquelme et al\.,[2021](https://arxiv.org/html/2608.04084#bib.bib29)\)scales per\-token top\-kkrouting to vision, and Soft MoE\(Puigcerver et al\.,[2024](https://arxiv.org/html/2608.04084#bib.bib28)\), the same family’s slot\-based successor, replaces hard top\-kkwith continuous slot assignment; Expert Choice\(Zhou et al\.,[2022](https://arxiv.org/html/2608.04084#bib.bib46)\)inverts routing direction so each expert chooses its tokens; ReMoE\(Wang et al\.,[2025](https://arxiv.org/html/2608.04084#bib.bib43)\)uses ReLU for adaptive sparsity; Mod\-Squad\(Chen et al\.,[2023b](https://arxiv.org/html/2608.04084#bib.bib5)\)adds a mutual\-information loss for specialization\. A parallel line progressively simplifies the routing machinery itself: StableMoE\(Dai et al\.,[2022](https://arxiv.org/html/2608.04084#bib.bib6)\)freezes routing after a distillation stage because routing fluctuation harms training; ST\-MoE\(Zoph et al\.,[2022](https://arxiv.org/html/2608.04084#bib.bib47)\)regularizes router logits for stability; auxiliary\-loss\-free balancing\(Wang et al\.,[2024](https://arxiv.org/html/2608.04084#bib.bib40)\)removes the balancing loss in favor of a bias correction\. SpecDrop sits at the limit point of this simplification trajectory, with routing frozen from step 0, no router parameters, and no balancing machinery; it quantifies what this zero\-fluctuation limit buys when, and only when, the routing signal is category\-aligned\.

#### Fixed\-rule routing\.

Deterministic rules replace learned gates\. Hash Layers\(Roller et al\.,[2021](https://arxiv.org/html/2608.04084#bib.bib30)\)hash tokens to experts; MaskMoE\(Su et al\.,[2024](https://arxiv.org/html/2608.04084#bib.bib34)\)assigns a fixed random binary mask per vocabulary token, with token frequency controlling the number of visible experts; COMET\(Shaier et al\.,[2025](https://arxiv.org/html/2608.04084#bib.bib32)\)uses fixed random projection withkk\-WTA selection\. Data\-domain variants route by tags: DEMix\(Gururangan et al\.,[2022](https://arxiv.org/html/2608.04084#bib.bib16)\)assigns each domain a dedicated expert, while Branch\-Train variants\(Li et al\.,[2022](https://arxiv.org/html/2608.04084#bib.bib23); Sukhbaatar et al\.,[2024](https://arxiv.org/html/2608.04084#bib.bib35)\)train separate experts per partition\. Hash rules give up category alignment, while hard domain routing setspinactive=0p\_\{\\text\{inactive\}\}\{=\}0and severs cross\-group gradient flow\. SpecDrop conditions on category structure while preservingpinactive\>0p\_\{\\text\{inactive\}\}\{\>\}0for cross\-category transfer\.

#### Input\-dependent dropout\.

Activations are stochastically dropped for regularization\. Standout\(Ba and Frey,[2013](https://arxiv.org/html/2608.04084#bib.bib2)\), Information/Contextual Dropout\(Achille and Soatto,[2018](https://arxiv.org/html/2608.04084#bib.bib1); Fan et al\.,[2021](https://arxiv.org/html/2608.04084#bib.bib12)\), Example\-Tied Dropout\(Maini et al\.,[2023](https://arxiv.org/html/2608.04084#bib.bib26)\), and NSDropout\(Shunk,[2022](https://arxiv.org/html/2608.04084#bib.bib33)\)operate at the neuron level; Stochastic Depth\(Huang et al\.,[2016](https://arxiv.org/html/2608.04084#bib.bib19)\), LayerDrop\(Fan et al\.,[2020](https://arxiv.org/html/2608.04084#bib.bib11)\), and SMoE\-Dropout\(Chen et al\.,[2023a](https://arxiv.org/html/2608.04084#bib.bib4)\)at the block or layer level\. SpecDrop instead conditions module activation on category\. Even so, specialization remains elusive across MoE variants —Wang et al\. \([2026](https://arxiv.org/html/2608.04084#bib.bib41)\)document a persistent “standing committee” of generalist experts under sophisticated load\-balancing; SpecDrop responds by intervening on training\-signal granularity instead of the routing function\.

## 3Method

SpecDrop conditions each module’s activation on the input’s category tag \(Figure[2](https://arxiv.org/html/2608.04084#S1.F2)\)\. It combines four elements:*\(i\)*a round\-robin assignment matrix mapping categories to preferred modules,*\(ii\)*a category\-conditioned dropout schedule with two probabilitiespa\>pi\>0p\_\{\\mathrm\{a\}\}\>p\_\{\\mathrm\{i\}\}\>0,*\(iii\)*a fixed\-denominator merge that calibrates branch magnitudes, and*\(iv\)*an optional always\-on shared expert\. We detail each below, then extend to imbalanced categories \(§[3\.5](https://arxiv.org/html/2608.04084#S3.SS5)\) and specify the warmup schedule \(§[3\.6](https://arxiv.org/html/2608.04084#S3.SS6)\)\.

### 3\.1Problem Setup

Consider a deep neural network with a shared feature extractorfstemf\_\{\\text\{stem\}\}, followed byKKparallel modules \(branches\)\{gk\}k=1K\\\{g\_\{k\}\\\}\_\{k=1\}^\{K\}, and a shared classifier headfheadf\_\{\\text\{head\}\}\. Let𝒞=\{1,2,…,M\}\\mathcal\{C\}=\\\{1,2,\\ldots,M\\\}denote the set of data categories \(e\.g\., superclasses in image classification, domain tags in language modeling\); each training sample\(x,y\)\(x,y\)has an associated category labelc​\(x\)∈𝒞c\(x\)\\in\\mathcal\{C\}\. During standard training without routing, allKKmodules process every input:

y^=fhead​\(1K​∑k=1Kgk​\(fstem​\(x\)\)\)\.\\hat\{y\}=f\_\{\\text\{head\}\}\\\!\\left\(\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}g\_\{k\}\\bigl\(f\_\{\\text\{stem\}\}\(x\)\\bigr\)\\right\)\.\(1\)

### 3\.2Category\-Conditioned Modular Dropout

#### Assignment matrix\.

We define a binary assignment matrix𝐀∈\{0,1\}M×K\\mathbf\{A\}\\in\\\{0,1\\\}^\{M\\times K\}, whereAc​k=1A\_\{ck\}=1indicates that modulekkis*assigned*to categorycc\. In the simplest case, with 1\-based indexingc∈\{1,…,M\}c\\in\\\{1,\\ldots,M\\\}andk∈\{1,…,K\}k\\in\\\{1,\\ldots,K\\\}, we use round\-robin assignment:Ac​k=𝟏​\[\(\(c−1\)modK\)\+1=k\]A\_\{ck\}=\\mathbf\{1\}\[\(\(c\-1\)\\bmod K\)\+1=k\], which gives each module⌊M/K⌋\\lfloor M/K\\rflooror⌈M/K⌉\\lceil M/K\\rceilassigned categories and an exactly equalM/KM/KwhenK∣MK\\mid M\(the case in our deployed settings,K∈\{20,46,7,20\}K\{\\in\}\\\{20,46,7,20\\\}\)\.

#### Activation probabilities\.

For a training sample with categorycc, the activation probability of modulekkis:

pk​\(c\)=Ac​k⋅pactive\+\(1−Ac​k\)⋅pinactive,p\_\{k\}\(c\)=A\_\{ck\}\\cdot p\_\{\\text\{active\}\}\+\(1\-A\_\{ck\}\)\\cdot p\_\{\\text\{inactive\}\},\(2\)wherepactive∈\(0,1\]p\_\{\\text\{active\}\}\\in\(0,1\]is the keep probability for assigned modules, andpinactive∈\[0,pactive\)p\_\{\\text\{inactive\}\}\\in\[0,p\_\{\\text\{active\}\}\)is the keep probability for unassigned modules \(abbreviatedpap\_\{\\mathrm\{a\}\}andpip\_\{\\mathrm\{i\}\}throughout\)\. During training, each modulekkis independently activated viamk∼Bernoulli​\(pk​\(c\)\)m\_\{k\}\\sim\\text\{Bernoulli\}\(p\_\{k\}\(c\)\)\. At inference,mk=pk​\(c\)m\_\{k\}=p\_\{k\}\(c\)deterministically\.

#### Rationale\.

The nonzeropinactive\>0p\_\{\\text\{inactive\}\}\>0preserves cross\-category gradient flow rather than hard isolation; Theorem[1](https://arxiv.org/html/2608.04084#Thmtheorem1)formalizes the resulting specialization\.

### 3\.3Fixed\-Denominator Merge

To combine theKKbranch outputs\{hk=gk​\(fstem​\(x\)\)\}k=1K\\\{h\_\{k\}=g\_\{k\}\(f\_\{\\text\{stem\}\}\(x\)\)\\\}\_\{k=1\}^\{K\}, we use a*fixed denominator*SS:

output=∑k=1Kmk⋅hkS,S=∑k=1Kpk​\(c\)=pactive\+\(K−1\)⋅pinactive,\\text\{output\}=\\frac\{\\sum\_\{k=1\}^\{K\}m\_\{k\}\\cdot h\_\{k\}\}\{S\},\\qquad S=\\sum\_\{k=1\}^\{K\}p\_\{k\}\(c\)=p\_\{\\text\{active\}\}\+\(K\-1\)\\cdot p\_\{\\text\{inactive\}\},\(3\)whereSSis*category\-independent*under round\-robin assignment\. The fixed denominator serves a dual role\.\(i\) Train–test consistency\.Proposition[1](https://arxiv.org/html/2608.04084#Thmproposition1)shows exact match at the merge layer, in contrast to the naive stochastic denominator∑kmk\\sum\_\{k\}m\_\{k\}whose Jensen bias𝔼​\[∑kmk​hk/∑kmk\]≠∑kpk​hk/∑kpk\\mathbb\{E\}\[\\sum\_\{k\}m\_\{k\}h\_\{k\}/\\sum\_\{k\}m\_\{k\}\]\\neq\\sum\_\{k\}p\_\{k\}h\_\{k\}/\\sum\_\{k\}p\_\{k\}introduces train–test mismatch \(a Bernoulli\-variant concern, App\.[A](https://arxiv.org/html/2608.04084#A1); the deployed soft variant relies on*\(ii\)*\)\.\(ii\) Magnitude calibration\.The convex\-combination weights place the merged output at single\-branch scale, allowing the shared expert to be added co\-equally \(Sec\.[3\.4](https://arxiv.org/html/2608.04084#S3.SS4)\)\.

### 3\.4Canonical Instantiation: Soft SpecDrop

The activation probabilitiespk​\(c\)p\_\{k\}\(c\)admit two instantiations whose forward outputs match in expectation:*Stochastic SpecDrop*samplesmk∼Bernoulli​\(pk​\(c\)\)m\_\{k\}\\sim\\mathrm\{Bernoulli\}\(p\_\{k\}\(c\)\)\(Appendix[A](https://arxiv.org/html/2608.04084#A1)\), while the deployed*Soft SpecDrop*usespk​\(c\)p\_\{k\}\(c\)directly as deterministic soft weights at both training and inference \(no mask sampling\); we retain “SpecDrop” for the family name while emphasizing that the canonical variant is fixed category\-conditioned gating, not stochastic dropout\. We adopt Soft SpecDrop as our canonical variant and augment it with an*always\-on shared expert \(SE\)*gSEg\_\{\\mathrm\{SE\}\}, a module disjoint from theKKrouted branches whose outputhSE=gSE​\(fstem​\(x\)\)h\_\{\\mathrm\{SE\}\}=g\_\{\\mathrm\{SE\}\}\(f\_\{\\mathrm\{stem\}\}\(x\)\)is added*after*the fixed\-denominator normalization:

output=∑k=1Kpk​\(c\)​hkS\+hSE\.\\text\{output\}=\\frac\{\\sum\_\{k=1\}^\{K\}p\_\{k\}\(c\)\\,h\_\{k\}\}\{S\}\\;\+\\;h\_\{\\mathrm\{SE\}\}\.\(4\)Soft SpecDrop removes Bernoulli mask\-sampling variance, so thepk​\(c\)/Sp\_\{k\}\(c\)/Sscaling factor in Theorem[1](https://arxiv.org/html/2608.04084#Thmtheorem1)is exact at training time \(thepa/pip\_\{a\}/p\_\{i\}ratio still requires the stated base\-gradient symmetry\); settinggSE≡0g\_\{\\mathrm\{SE\}\}\{\\equiv\}0recovers the pure routed\-branches merge \(CIFAR\-100 config\), while the always\-on SE is retained by default on imbalanced NLP\.

### 3\.5Per\-Category Routing

Under imbalanced category frequencies\{πc\}\\\{\\pi\_\{c\}\\\}\(∑cπc=1\\sum\_\{c\}\\pi\_\{c\}\{=\}1\) we generalize\(pa,pi\)\(p\_\{\\mathrm\{a\}\},p\_\{\\mathrm\{i\}\}\)to per\-category\(pac,pic\)\(p\_\{\\mathrm\{a\}\}^\{c\},p\_\{\\mathrm\{i\}\}^\{c\}\)withgapc=\(pa−pi\)⋅\[\(1−πc\)/\(1−1/M\)\]β\\mathrm\{gap\}\_\{c\}=\(p\_\{\\mathrm\{a\}\}\-p\_\{\\mathrm\{i\}\}\)\\cdot\[\(1\{\-\}\\pi\_\{c\}\)/\(1\{\-\}1/M\)\]^\{\\beta\}, whereβ≥0\\beta\\geq 0controls how strongly the per\-category gap is amplified for rare categories \(largerβ\\beta= more amplification\):

pac=SK\+gapc⋅K−1K,pic=SK−gapcK,Sc≜pac\+\(K−1\)​pic=S​∀c,β,πcp\_\{\\mathrm\{a\}\}^\{c\}=\\tfrac\{S\}\{K\}\+\\mathrm\{gap\}\_\{c\}\\cdot\\tfrac\{K\-1\}\{K\},\\qquad p\_\{\\mathrm\{i\}\}^\{c\}=\\tfrac\{S\}\{K\}\-\\tfrac\{\\mathrm\{gap\}\_\{c\}\}\{K\},\\qquad S^\{c\}\\;\\triangleq\\;p\_\{\\mathrm\{a\}\}^\{c\}\+\(K\{\-\}1\)\\,p\_\{\\mathrm\{i\}\}^\{c\}\\;=\\;S\\;\\;\\forall\\,c,\\beta,\\pi\_\{c\}\(5\)\(invariance proof App\.[C\.3](https://arxiv.org/html/2608.04084#A3.SS3)\)\. Eq\.[3](https://arxiv.org/html/2608.04084#S3.E3)and Prop\.[1](https://arxiv.org/html/2608.04084#Thmproposition1)apply*verbatim*; only the per\-sample weights change\. Atβ=0\\beta\{=\}0or in balanced settings \(πc=1/M\\pi\_\{c\}\{=\}1/M\), Eq\.[5](https://arxiv.org/html/2608.04084#S3.E5)reduces to the scalar form\(pa,pi\)\(p\_\{\\mathrm\{a\}\},p\_\{\\mathrm\{i\}\}\)\.

### 3\.6Training Schedule

pap\_\{\\mathrm\{a\}\}ramps fromS/KS/K\(yielding a uniform1/K1/Kmerge at warmup start\) to its target value via a cosine schedule over a fractionwr∈\[0,1\]w\_\{r\}\\in\[0,1\]of total training;pip\_\{\\mathrm\{i\}\}follows the coupled inversepi​\(t\)=\(S−pa​\(t\)\)/\(K−1\)p\_\{\\mathrm\{i\}\}\(t\)=\(S\-p\_\{\\mathrm\{a\}\}\(t\)\)/\(K\-1\)to keepSSconstant\. The warmup is applied per\-epoch for vision settings and per\-step for language settings, matching the learning\-rate decay schedule\. At inference,mk=pk​\(c\)m\_\{k\}=p\_\{k\}\(c\)deterministically and the forward pass matches Eq\.[4](https://arxiv.org/html/2608.04084#S3.E4)\.

## 4Theoretical Analysis

We state three formal results that characterize SpecDrop’s specialization properties\. All proofs, remarks, and a bias–variance argument for soft\-vs\-hard routing are in Appendix[C](https://arxiv.org/html/2608.04084#A3)\.

###### Theorem 1\(Gradient Concentration\)\.

Under SpecDrop \(Eq\.[2](https://arxiv.org/html/2608.04084#S3.E2)\), the expected gradient magnitude for modulekkon an assigned categorycc\(Ac​k=1A\_\{ck\}\{=\}1\) versus an unassigned categoryc′c^\{\\prime\}\(Ac′​k=0A\_\{c^\{\\prime\}k\}\{=\}0\) satisfies𝔼m​\[‖∂ℒ/∂θk‖∣c\]/𝔼m​\[‖∂ℒ/∂θk‖∣c′\]=pactive/pinactive\\mathbb\{E\}\_\{m\}\[\\\|\\partial\\mathcal\{L\}/\\partial\\theta\_\{k\}\\\|\\mid c\]\\,/\\,\\mathbb\{E\}\_\{m\}\[\\\|\\partial\\mathcal\{L\}/\\partial\\theta\_\{k\}\\\|\\mid c^\{\\prime\}\]=p\_\{\\textup\{active\}\}/p\_\{\\textup\{inactive\}\}, assuming \(i\) the gradient norm is independent of the mask given the input \(*mask\-independence*\), and \(ii\) at initialization, all categories produce equal\-magnitude base gradients \(*category\-symmetry at initialization*\)\.

###### Proposition 1\(Fixed vs\. Stochastic Denominator\)\.

LetS=pactive\+\(K−1\)​pinactiveS=p\_\{\\textup\{active\}\}\+\(K\-1\)p\_\{\\textup\{inactive\}\}; under round\-robin,S=∑kpk​\(c\)S=\\sum\_\{k\}p\_\{k\}\(c\)is*category\-independent*\.\(a\)The fixed\-denominator merge matches train and test exactly:𝔼m​\[∑kmk​hk/S\]=∑kpk​\(c\)​hk/S\\mathbb\{E\}\_\{m\}\[\\sum\_\{k\}m\_\{k\}h\_\{k\}/S\]=\\sum\_\{k\}p\_\{k\}\(c\)h\_\{k\}/Sfor all\{hk\}\\\{h\_\{k\}\\\}andcc, with weights\(pk​\(c\)/S\)k\(p\_\{k\}\(c\)/S\)\_\{k\}forming a convex combination\.\(b\)The naive stochastic merge∑kmk​hk/∑kmk\\sum\_\{k\}m\_\{k\}h\_\{k\}/\\sum\_\{k\}m\_\{k\}\(with the convention that the output is𝟎\\mathbf\{0\}when∑kmk=0\\sum\_\{k\}m\_\{k\}\{=\}0\) departs from the test\-time forward pass:𝔼m​\[X/N\]=𝔼​\[X\]​𝔼​\[1/N\]\+Cov​\(X,1/N\)\\mathbb\{E\}\_\{m\}\[X/N\]=\\mathbb\{E\}\[X\]\\,\\mathbb\{E\}\[1/N\]\+\\mathrm\{Cov\}\(X,1/N\)whereX=∑kmk​hkX\{=\}\\sum\_\{k\}m\_\{k\}h\_\{k\}andN=∑kmkN\{=\}\\sum\_\{k\}m\_\{k\}, with both an inverse\-denominator Jensen gap and non\-zero covariance\.

###### Theorem 2\(Routing\-Indicator Mutual Information\)\.

LetZk∈\{0,1\}Z\_\{k\}\\in\\\{0,1\\\}be the activation indicator for modulekkunder round\-robin assignment withK\|MK\\,\|\\,M\(i\.e\.,KKdividesMM\) andCCuniform on\{1,…,M\}\\\{1,\\ldots,M\\\}\. ThenI​\(Zk;C\)=1K​DKL​\(pactive∥p¯\)\+K−1K​DKL​\(pinactive∥p¯\)I\(Z\_\{k\};C\)=\\tfrac\{1\}\{K\}D\_\{\\textup\{KL\}\}\(p\_\{\\textup\{active\}\}\\\|\\bar\{p\}\)\+\\tfrac\{K\-1\}\{K\}D\_\{\\textup\{KL\}\}\(p\_\{\\textup\{inactive\}\}\\\|\\bar\{p\}\)exactly, withp¯=\(pactive\+\(K−1\)​pinactive\)/K\\bar\{p\}=\(p\_\{\\textup\{active\}\}\+\(K\-1\)p\_\{\\textup\{inactive\}\}\)/K; this quantity vanishes iffpactive=pinactivep\_\{\\textup\{active\}\}=p\_\{\\textup\{inactive\}\}\(random dropout\)\.

lossℒ\\mathcal\{L\}g1g\_\{1\}g2g\_\{2\}g3g\_\{3\}g4g\_\{4\}pa/S\\nicefrac\{\{p\_\{\\mathrm\{a\}\}\}\}\{\{S\}\}pi/S\\nicefrac\{\{p\_\{\\mathrm\{i\}\}\}\}\{\{S\}\}sample of categoryccwithAc​1=1A\_\{c1\}\{=\}1: assigned branchg1g\_\{1\}, unassignedg2,…,gKg\_\{2\},\\ldots,g\_\{K\}\(shown:K=4K\{=\}4\)Figure 3:The gradient\-concentration mechanism \(Thm\.[1](https://arxiv.org/html/2608.04084#Thmtheorem1)\)\.Backpropagated gradient magnitude for branchkkscales with its activation probabilitypk​\(c\)/Sp\_\{k\}\(c\)/S, so over training each branch accumulatespa/pip\_\{\\mathrm\{a\}\}/p\_\{\\mathrm\{i\}\}more gradient signal from its assigned categories than from unassigned ones, while the nonzeropip\_\{\\mathrm\{i\}\}preserves cross\-category flow\.#### From theory to design\.

Thm\.[1](https://arxiv.org/html/2608.04084#Thmtheorem1)predictspactive/pinactivep\_\{\\textup\{active\}\}/p\_\{\\textup\{inactive\}\}as the specialization\-driving gradient ratio \(Fig\.[3](https://arxiv.org/html/2608.04084#S4.F3)\), directionally consistent with the1\.87×1\.87\\timespost\-training pruning\-sensitivity ratio on CIFAR \(Sec\.[5\.6](https://arxiv.org/html/2608.04084#S5.SS6)\)\. Prop\.[1](https://arxiv.org/html/2608.04084#Thmproposition1)justifies the fixed\-denominator merge through exact train–test consistency, Jensen\-bias quantification, and magnitude calibration for the shared expert \(Sec\.[3\.4](https://arxiv.org/html/2608.04084#S3.SS4)\)\. Thm\.[2](https://arxiv.org/html/2608.04084#Thmtheorem2)grounds the Bernoulli variant information\-theoretically \(App\.[A](https://arxiv.org/html/2608.04084#A1)\); Soft specialization is measured empirically by pruning sensitivity\.

## 5Experiments

We evaluate SpecDrop across four settings spanning vision and language; we call a partition*balanced*when training counts are equal across categories and*imbalanced*otherwise\. All numbers are mean±\\pmstd over 3 seeds \(42, 123, 456\)\. Alongside each setting’s primary metric we report*branch–category alignment*\(Align, %\): the percentage of categories whose most\-pruning\-sensitive branch coincides with its assigned branch; a high value indicates that the trained model’s emergent specialization matches the imposed partition\. Align is a property of the trained model, distinct from the data\-side partition\-alignment axis of Sec\.[1](https://arxiv.org/html/2608.04084#S1): a partition can be fuzzy while Align is high, as on SlimPajama\. Per\-setting details \(datasets, backbones, baselines, training\) are given in the corresponding subsections; full per\-baseline protocols, branch derivations, BREEDS construction, ROUGE\-L selection metric, and additional disclosures are in App\.[B](https://arxiv.org/html/2608.04084#A2)\.

### 5\.1CIFAR\-100 Results

#### Setup\.

We evaluate on CIFAR\-100\(Krizhevsky,[2009](https://arxiv.org/html/2608.04084#bib.bib22)\)using itsM=K=20M\{=\}K\{=\}20balanced superclasses on ResNet\-110\(He et al\.,[2016](https://arxiv.org/html/2608.04084#bib.bib17)\)\(∼\\sim1\.7M params\), comparing Soft SpecDrop against dense ResNet\-110, Stochastic Depth\(Huang et al\.,[2016](https://arxiv.org/html/2608.04084#bib.bib19)\), Example\-Tied Dropout\(Maini et al\.,[2023](https://arxiv.org/html/2608.04084#bib.bib26)\), Contextual Dropout\(Fan et al\.,[2021](https://arxiv.org/html/2608.04084#bib.bib12)\), and an architecture\-matched No\-Routing variant\. All methods train with SGD at learning rate0\.10\.1on a cosine schedule, batch128128, for200200epochs\.

Table 1:CIFAR\-100 baseline comparison\.*HardCategory*\(pa=1,pi=0p\_\{\\mathrm\{a\}\}\{=\}1,p\_\{\\mathrm\{i\}\}\{=\}0, DEMix\-style\) isolates soft routing’spi\>0p\_\{\\mathrm\{i\}\}\{\>\}0contribution; both Soft SpecDrop and HardCategory consume the2020\-superclass label at inference\. No\-Routing’s63\.0863\.08reflects narrow\-branch capacity \(Sec\.[5\.1](https://arxiv.org/html/2608.04084#S5.SS1)\); Align chance level is5%5\\%atK=20K\{=\}20\.MethodBackbone\#ParamsTop\-1 \(%\)↑\\uparrowAlign \(%\)↑\\uparrowResNet\-110dense ResNet\-1101\.737M74\.48±0\.1374\.48\\pm 0\.13—Stochastic Depth1\.737M75\.80±0\.1675\.80\\pm 0\.16—Example\-Tied Dropout1\.737M63\.68±0\.9763\.68\\pm 0\.97—Contextual Dropout1\.764M70\.25±0\.2870\.25\\pm 0\.28—No\-RoutingMultiBranchK=20K\{=\}201\.721M63\.08±0\.0463\.08\\pm 0\.043\.3±2\.93\.3\\pm 2\.9HardCategory1\.721M57\.67±3\.4857\.67\\pm 3\.48100\.0±0\.0\\mathbf\{100\.0\\pm 0\.0\}Soft SpecDrop1\.721M79\.23±0\.17\\mathbf\{79\.23\\pm 0\.17\}58\.3±14\.458\.3\\pm 14\.4
#### Results\.

Table[1](https://arxiv.org/html/2608.04084#S5.T1)reports CIFAR\-100 top\-1 accuracy across five baselines and Soft SpecDrop\. Soft SpecDrop achieves79\.23±0\.17%\\mathbf\{79\.23\\pm 0\.17\}\\%, exceeding dense ResNet\-110 by\+4\.75\+4\.75and Stochastic Depth by\+3\.43\+3\.43\.

#### Routing\-isolated comparison and HardCategory ablation\.

The architecture\-matched No\-Routing baseline atK=20K\{=\}20\(63\.08%63\.08\\%, constrained by narrow\-branch capacity, channels\[4,7,14\]\[4,7,14\]\) isolates the routing contribution under the partition\-aligned regime: Soft SpecDrop’s\+16\.15\\mathbf\{\+16\.15\}over it comes from the fixed\-denominator merge with category\-conditioned dropout\. HardCategory uses one\-hot routing instead of soft on the same metadata, collapsing to57\.67±3\.4857\.67\\pm 3\.48\(5\.415\.41below No\-Routing,20×20\\timesours’ std\); our Align reaches58\.3%58\.3\\%\(vs5%5\\%chance,3\.3%3\.3\\%No\-Routing\) while HardCategory’s tautological100%100\\%co\-occurs with the worst top\-1, isolating soft leakagepi\>0p\_\{i\}\{\>\}0as the active ingredient \(not metadata access or alignment\)\. Baseline implementations in App\.[B](https://arxiv.org/html/2608.04084#A2)\.

### 5\.2ImageNet ViT Results

#### Setup\.

We evaluate on ImageNet\-1K\(Deng et al\.,[2009](https://arxiv.org/html/2608.04084#bib.bib8)\)partitioned intoM=46M\{=\}46imbalanced supercategories via our recursive expansion of the BREEDS\(Santurkar et al\.,[2021](https://arxiv.org/html/2608.04084#bib.bib31)\)curated WordNet hierarchy with hyperparametersT=60T\{=\}60\(max\-leaves\) andC=10C\{=\}10\(max\-children\); the full algorithm and per\-supercategory sizes are in App\.[B\.3](https://arxiv.org/html/2608.04084#A2.SS3)\. All methods are based on ViT\-Small/16\(Dosovitskiy et al\.,[2021](https://arxiv.org/html/2608.04084#bib.bib9); Touvron et al\.,[2021](https://arxiv.org/html/2608.04084#bib.bib38)\)\(∼\\sim22M params\)\. We compare Soft SpecDrop against dense ViT\-S/16, Soft MoE\(Puigcerver et al\.,[2024](https://arxiv.org/html/2608.04084#bib.bib28)\)in a tuned configuration \(paper\-canonical second\-half placement, learning rate5×10−45\{\\times\}10^\{\-4\}, selected by the dedicated sweep of App\.[E\.7](https://arxiv.org/html/2608.04084#A5.SS7)\) plus a compute\-matched variant whose parameters are unconstrained, an auxiliary\-loss\-free top\-kkrouter\(Wang et al\.,[2024](https://arxiv.org/html/2608.04084#bib.bib40)\)capacity\-identical to our Mod\-Squad configuration, Mod\-Squad\(Chen et al\.,[2023b](https://arxiv.org/html/2608.04084#bib.bib5)\)\(FFN\-only, adapted to ImageNet BREEDS\), COMET\(Shaier et al\.,[2025](https://arxiv.org/html/2608.04084#bib.bib32)\), and an architecture\-matched No\-Routing\+SE baseline\. All methods train with AdamW at learning rate2\.5×10−42\.5\{\\times\}10^\{\-4\}\(the tuned Soft MoE at its swept5×10−45\{\\times\}10^\{\-4\}\) on a cosine schedule, batch256256, for100100epochs of the DeiT short recipe \(no EMA, no RepeatedAugmentation\)\.

#### Results\.

Soft SpecDrop reaches79\.89±0\.18%\\mathbf\{79\.89\\pm 0\.18\}\\%top\-1 on ImageNet\-1K BREEDS \(Tab\.[2](https://arxiv.org/html/2608.04084#S5.T2)\),\+6\.53\+6\.53over the matched\-supervision No\-Routing\+SE baseline and\+3\.20\+3\.20over the strongest learned router, the tuned Soft MoE \(76\.69±0\.7076\.69\\pm 0\.70, itself above dense\); the SE contributes\+2\.06\+2\.06\. The compute\-matched Soft MoE \(dense\-level MACs, parameters unconstrained\) lands at66\.7266\.72, below the bare No\-Routing control: the deployed configuration’s constraint was placement, not compute \(App\.[E\.7](https://arxiv.org/html/2608.04084#A5.SS7)\)\.

Table 2:ImageNet\-1K BREEDS\-46 on MultiBranch ViT\-Small/16 \(henceforth MB\-ViT\), with every baseline’s parameter and compute status stated per row\.*Parameter status*vs dense ViT\-S \(22\.05122\.051M\): tuned Soft MoE\+0\.7%\+0\.7\\%, ALF router\+0\.6%\+0\.6\\%, Mod\-Squad\+1\.0%\+1\.0\\%, COMET\+0\.0%\+0\.0\\%, No\-Routing\+0\.2%\+0\.2\\%, No\-Routing\+\+SE and ours\+1\.0%\+1\.0\\%, all within±1%\\pm 1\\%; the compute\-matched Soft MoE’s parameters are*unconstrained by design*\(481481M,21\.8×21\.8\\timesdense\)\.*Compute status*is the MACs column \(App\.[F\.3](https://arxiv.org/html/2608.04084#A6.SS3)\): ours and the No\-Routing\+\+SE control are compute\-identical \(4\.254\.25G, matching dense\); the deployed all\-blocks Soft MoE \(63\.06±0\.2263\.06\\pm 0\.22, retained as an appendix ablation, App\.[E\.7](https://arxiv.org/html/2608.04084#A5.SS7)\) runs at1\.561\.56G under parameter matching, and the compute\-matched variant restores dense\-level MACs\. Soft MoE rows use a dedicated SoftMoEViT implementation \(Puigcerver et al\.,[2024](https://arxiv.org/html/2608.04084#bib.bib28), Alg\. 1\); the tuned row uses the paper\-canonical second\-half placement and lr5×10−45\{\\times\}10^\{\-4\}from a dedicated sweep \(App\.[E\.7](https://arxiv.org/html/2608.04084#A5.SS7)\), the only one any method received\. The ALF router\(Wang et al\.,[2024](https://arxiv.org/html/2608.04084#bib.bib40)\)is capacity\-identical to Mod\-Squad, isolating the balancing mechanism\. The comparison to dense is not supervision\-controlled \(BREEDS labels are target\-derived; Sec\.[5\.2](https://arxiv.org/html/2608.04084#S5.SS2)\)\.MethodBackbone\#ParamsMACs \(G\)Top\-1 \(%\)↑\\uparrowAlign \(%\)↑\\uparrowViT\-Small/16Dense ViT\-S/1622\.051M4\.2576\.38±0\.1576\.38\\pm 0\.15—Soft MoE \(tuned\)SoftMoEViTN=32N\{=\}322nd\-half22\.196M2\.9176\.69±0\.7076\.69\\pm 0\.70—Soft MoE \(comp\.\-matched\)SoftMoEViTN=32N\{=\}32wide481\.2M4\.2666\.72±0\.6366\.72\\pm 0\.63—ALF top\-kkrouterMoE\-ViTN=16N\{=\}16, top\-2222\.194M4\.2771\.09±0\.3271\.09\\pm 0\.32—Mod\-Squad22\.267M4\.2770\.11±0\.6170\.11\\pm 0\.611\.4±1\.31\.4\\pm 1\.3COMETMultiBranchK=46K\{=\}4622\.051M5\.6571\.02±0\.1771\.02\\pm 0\.17—No\-Routing22\.092M4\.2271\.30±0\.2271\.30\\pm 0\.225\.1±4\.55\.1\\pm 4\.5No\-RoutingMultiBranchK=46K\{=\}46\+\+SE22\.263M4\.2573\.36±0\.2973\.36\\pm 0\.292\.9±2\.52\.9\\pm 2\.5Soft SpecDrop22\.263M4\.2579\.89±0\.18\\mathbf\{79\.89\\pm 0\.18\}100\.0±0\.0\\mathbf\{100\.0\\pm 0\.0\}

#### Routing\-isolated comparison and BREEDS label\-leak\.

The proper isolated baseline is MB\-ViT No\-Routing\+\+SE: ours beats it by\+6\.53\\mathbf\{\+6\.53\}under matched supervision; relative to bare No\-Routing \(71\.3071\.30\), the SE contributes\+2\.06\+2\.06and the routing mechanism contributes the remaining\+6\.53\+6\.53on top of matched\-SE\. Pruning\-sensitivity confirms mechanism\-driven: ours100%100\\%Align \(vs2\.2%2\.2\\%chance atK=46K\{=\}46\); matched\-SE No\-Routing2\.9%2\.9\\%, Mod\-Squad1\.4%1\.4\\%— partition\-aligned specialization, not capacity\. The\+3\.51\+3\.51over dense is not directly comparable: BREEDS supercategories are constructed from fine labels\(Santurkar et al\.,[2021](https://arxiv.org/html/2608.04084#bib.bib31)\), leaking supervision that dense does not consume\.

### 5\.3NLP Domain\-Conditioned Routing

#### Setup\.

We evaluate on SlimPajama\-6B language modeling using itsM=K=7M\{=\}K\{=\}7imbalanced document domains as categories \(∼15×\{\\sim\}15\\timesmax/min ratio\), with all methods based on a 6\-layer / 384\-hidden / 6\-head Transformer LM \(∼\\sim30M params\)\. We compare Soft SpecDrop against the dense Transformer\(Vaswani et al\.,[2017](https://arxiv.org/html/2608.04084#bib.bib39)\), Switch\(Fedus et al\.,[2022](https://arxiv.org/html/2608.04084#bib.bib13)\), Hash Layers\(Roller et al\.,[2021](https://arxiv.org/html/2608.04084#bib.bib30)\), SMoE\-Dropout\(Chen et al\.,[2023a](https://arxiv.org/html/2608.04084#bib.bib4)\), DEMix\(Gururangan et al\.,[2022](https://arxiv.org/html/2608.04084#bib.bib16)\), and an architecture\-matched No\-Routing\+\+SE baseline\. All methods train with AdamW at learning rate3×10−43\{\\times\}10^\{\-4\}on a cosine schedule in bf16, batch3232sequences of512512tokens, for1010epochs over500500M unique tokens\.

#### Results\.

Table[3](https://arxiv.org/html/2608.04084#S5.T3)reports validation perplexity\. Soft SpecDrop achieves45\.38±0\.02\\mathbf\{45\.38\\pm 0\.02\}PPL; ours trails matched\-SE No\-Routing by\+0\.10\+0\.10PPL \(per\-seed deltas\+0\.05/\+0\.02/\+0\.22\+0\.05/\+0\.02/\+0\.22, App\.[E](https://arxiv.org/html/2608.04084#A5)\), within the seed\-noise envelope\. Within the multi\-branch architecture, the always\-on shared expert contributes1\.521\.52PPL \(matched\-SE45\.2845\.28vs bare No\-Routing46\.8046\.80\); ours nonetheless achieves the lowest perplexity among learned and fixed\-rule routers\.

Table 3:SlimPajama\-6B 7\-domain language modeling on a 30M Transformer LM\. SMoE\-Dropout uses the gradual\-kkschedule of[Chen et al\.](https://arxiv.org/html/2608.04084#bib.bib4)\(their Fig\. 2\) starting from a single active expert\.MethodBackbone\#ParamsVal PPL↓\\downarrowAlign \(%\)↑\\uparrowDense TransformerDense Transformer LM30\.143M44\.80±0\.05\\mathbf\{44\.80\\pm 0\.05\}—Hash LayersMultiBranchK=8K\{=\}830\.159M52\.05±0\.0652\.05\\pm 0\.0644\.4±9\.644\.4\\pm 9\.6SMoE\-DropoutMultiBranchK=16K\{=\}1630\.214M67\.32±0\.5067\.32\\pm 0\.505\.6±9\.65\.6\\pm 9\.6SwitchMultiBranchK=32K\{=\}3230\.288M49\.54±0\.2049\.54\\pm 0\.200\.0±0\.00\.0\\pm 0\.0DEMixMultiBranchK=7K\{=\}730\.175M53\.31±0\.0853\.31\\pm 0\.08100\.0±0\.0\\mathbf\{100\.0\\pm 0\.0\}No\-Routing30\.175M46\.80±0\.1146\.80\\pm 0\.1111\.1±9\.611\.1\\pm 9\.6No\-RoutingMultiBranchK=7K\{=\}7\+\+SE30\.168M45\.28±0\.1045\.28\\pm 0\.105\.6±9\.65\.6\\pm 9\.6Soft SpecDrop30\.168M45\.38±0\.0245\.38\\pm 0\.0294\.4±9\.694\.4\\pm 9\.6
#### Routing\-isolated tie and granularity alignment\.

SlimPajama’s77document\-level domains form a fuzzy partition: over the full validation set \(9,7669\{,\}766chunks\),56\.1%56\.1\\%of512512\-token chunks span≥2\{\\geq\}2BGE\-KMeans clusters \(App\.[E\.15](https://arxiv.org/html/2608.04084#A5.SS15)\); the tie with the mechanism\-OFF reference is the predicted outcome under our granularity\-alignment thesis\. The misalignment operates at training time and at distribution level: per\-chunk purity is uncorrelated with the per\-chunk cross\-entropy difference against the matched control \(Pearsonr=−0\.001r\{=\}\-0\.001,p=0\.95p\{=\}0\.95, App\.[E\.15](https://arxiv.org/html/2608.04084#A5.SS15)\), consistent with branch specialization forming over the whole training distribution rather than per evaluation chunk\. Pruning\-sensitivity rules out mechanism failure: ours94\.4%94\.4\\%Align \(vs14\.3%14\.3\\%chance,5\.6%5\.6\\%matched\-SE\) — the tied PPL reflects partition fuzziness, not mechanism failure to engage\. Within the tie, our cross\-seedσmean=0\.014\\sigma\_\{\\text\{mean\}\}\{=\}0\.014is4×4\\timestighter than the matched\-SE scalar’s0\.0570\.057, a stability benefit from step\-granularity warmup \(per\-seed and per\-domain PPL in App\.[E\.16](https://arxiv.org/html/2608.04084#A5.SS16)\)\.

### 5\.4SuperNI Instruction\-Tuning Results \(Llama\-3\.2\-1B \+ LoRA\)

#### Setup\.

We evaluate on SuperNI\(Wang et al\.,[2022](https://arxiv.org/html/2608.04084#bib.bib42)\)instruction tuning overM=K=20M\{=\}K\{=\}20imbalanced task clusters constructed by frequency\-cutoff over the SuperNI*Domains*field \(top\-19 most frequent normalized root domains plus a*miscellaneous*bucket; algorithm in App\.[B\.4](https://arxiv.org/html/2608.04084#A2.SS4)\), evaluating on 119 held\-out tasks\. All methods run as adapters on a frozen Llama\-3\.2\-1B base \(∼\\sim225M trainable LoRA parameters per method\)\. We compare Soft SpecDrop against single LoRA\(Hu et al\.,[2022](https://arxiv.org/html/2608.04084#bib.bib18)\), LoRAMoE\(Dou et al\.,[2024](https://arxiv.org/html/2608.04084#bib.bib10)\), MoCLE\(Gou et al\.,[2024](https://arxiv.org/html/2608.04084#bib.bib15)\), HydraLoRA\(Tian et al\.,[2024](https://arxiv.org/html/2608.04084#bib.bib37)\), and an architecture\-matched No\-Routing\+\+SE baseline\. All methods train with AdamW at learning rate2×10−42\{\\times\}10^\{\-4\}on a cosine schedule, at an effective batch of128128\(88per device×\\times1616accumulation steps\), for33epochs on the SuperNI training mix\.

#### Results\.

Table[4](https://arxiv.org/html/2608.04084#S5.T4)reports SuperNI ROUGE\-L F1\. Soft SpecDrop achieves0\.5106±0\.003\\mathbf\{0\.5106\\pm 0\.003\}, within seed noise of HydraLoRA \(0\.51530\.5153\) and LoRAMoE \(0\.50790\.5079\) in\-distribution; the routing\-only contribution over matched\-SE No\-Routing \(0\.50940\.5094\) is\+0\.0012\+0\.0012, statistically zero on this fuzzy partition\. Pruning\-sensitivity confirms no method specializes at the imposedK=20K\{=\}20partition \(all six near5%5\\%chance, ours2\.2%2\.2\\%, highest6\.7%6\.7\\%\) — unlike NLP’s94\.4%94\.4\\%alignment, the SuperNI partition is fuzzy enough that the mechanism fails to engage, a predicted null under our thesis \(cf\. the a\-priori embedding diagnostic, App\.[E\.15](https://arxiv.org/html/2608.04084#A5.SS15)\)\.

Table 4:SuperNI held\-out instruction tuning \(119 test tasks; ROUGE\-L computed on a1010\-instance\-per\-task sub\-sample due to greedy\-decoding cost, while training and CE\-loss eval use the canonical100100instances per task\) on Llama\-3\.2\-1B with LoRA adapters\. \#Params is trainable adapter only; the frozen Llama base \(∼\\sim1\.24B\) is excluded\. Per\-method capacity\-match configurations in App\.[B](https://arxiv.org/html/2608.04084#A2)\.MethodBackbone\#ParamsROUGE\-L F1↑\\uparrowAlign \(%\)↑\\uparrowSingle LoRASingle LoRAr=320r\{=\}320225\.44M0\.4754±0\.0070\.4754\\pm 0\.007—MoCLEMultiBranchK=5K\{=\}5224\.67M0\.4924±0\.0100\.4924\\pm 0\.0100\.0±0\.00\.0\\pm 0\.0LoRAMoEMultiBranchK=6K\{=\}6225\.31M0\.5079±0\.0020\.5079\\pm 0\.0022\.2±3\.82\.2\\pm 3\.8HydraLoRAMultiBranchK=8K\{=\}8226\.56M0\.5153±0\.003\\mathbf\{0\.5153\\pm 0\.003\}0\.0±0\.00\.0\\pm 0\.0No\-RoutingMultiBranchK=20K\{=\}20225\.44M0\.4993±0\.0110\.4993\\pm 0\.0116\.7±6\.76\.7\\pm 6\.7No\-RoutingMultiBranchK=20K\{=\}20\+\+SE221\.92M0\.5094±0\.0070\.5094\\pm 0\.0070\.0±0\.00\.0\\pm 0\.0Soft SpecDrop221\.92M0\.5106±0\.0030\.5106\\pm 0\.0032\.2±3\.82\.2\\pm 3\.8
#### Per\-task analysis\.

A per\-task split of47/46/2647/46/26ours\-wins / HydraLoRA\-wins / ties across119119SuperNI held\-out tasks is statistically indistinguishable from random allocation \(χ2\\chi^\{2\}omnibus fails to rejectH0H\_\{0\}, App\.[E\.14](https://arxiv.org/html/2608.04084#A5.SS14)\); the win\-pattern is not concentrated by task cluster\.

### 5\.5Hyperparameter Ablations

![Refer to caption](https://arxiv.org/html/2608.04084v1/x3.png)Figure 4:Hyperparameter ablations across four settings\.Row 1: activation probabilitypap\_\{\\mathrm\{a\}\}\. Row 2: imbalance\-amplification exponentβ\\betaon the imbalanced settings; on balanced CIFAR\-100 \(equal training images per category\), the per\-category formula reduces to a scalar for allβ\\beta, so we sweep the warmup ratiowrw\_\{r\}instead\. Row 3: shared\-expert capacity ratioXX\. Each panel: mean over 3 seeds, error bars±1​σ\\pm 1\\sigma\. The deployed operating point is marked with a vertical line and a filled red marker; hollow markers denote the algebraic mechanism\-OFF point \(pa=pip\_\{\\mathrm\{a\}\}\{=\}p\_\{\\mathrm\{i\}\}, excluded a priori\); the dotted reference line carries the best value of the preceding row’s sweep\.We sweep three core hyperparameters in sequence to identify deployed operating points \(Fig\.[4](https://arxiv.org/html/2608.04084#S5.F4)\)\.Activation probabilitypap\_\{\\mathrm\{a\}\}\(Row 1\) is searched first, withpip\_\{\\mathrm\{i\}\}coupled as1−pa1\{\-\}p\_\{\\mathrm\{a\}\}\(all sweeps and deployed operating points obey this coupling\), excludingpa=0\.5p\_\{\\mathrm\{a\}\}\{=\}0\.5a priori \(wherepa=pip\_\{\\mathrm\{a\}\}\{=\}p\_\{\\mathrm\{i\}\}collapses to the mechanism\-OFF reference, voiding downstream search\)\. The remaining sweep places the optimalpap\_\{\\mathrm\{a\}\}at an intermediate value in every setting \(0\.70\.7on CIFAR\-100,0\.60\.6on ImageNet and SlimPajama, and0\.80\.8on SuperNI/LoRA\), with both endpoints worse, confirming that the cross\-category leakagepi\>0p\_\{\\mathrm\{i\}\}\{\>\}0is load\-bearing rather than no\-routing or hard one\-hot\.Warmup ratiowrw\_\{r\}and imbalance amplificationβ\\beta\(Row 2\): on balanced CIFAR\-100 the per\-category formula degenerates to a scalar across allβ\\beta, so we sweepwrw\_\{r\}instead and adoptwr=1\.0w\_\{r\}\{=\}1\.0\(cosine LR\-aligned\); imbalancedβ\\betapeaks atβ=4\\beta\{=\}4\(SlimPajama\) andβ=1\\beta\{=\}1\(ImageNet, SuperNI/LoRA\)\.Shared\-expert capacityXX\(Row 3\): imbalanced prefer SE on \(X∈\[0\.5,2\.0\]X\\in\[0\.5,2\.0\]\), balanced CIFAR\-100 prefers off \(X=0X\{=\}0\)\.

### 5\.6Cross\-Setting Specialization Analysis

We measure per\-branch specialization via pruning sensitivity \(Δk,c\\Delta\_\{k,c\}from zero\-ablating branchkk\), visualized as heatmaps in Figure[1](https://arxiv.org/html/2608.04084#S1.F1); per\-method Align values are in Tabs\.[1](https://arxiv.org/html/2608.04084#S5.T1)–[4](https://arxiv.org/html/2608.04084#S5.T4)\. Disabling routing at inference \(uniform1/K1/Kmasks\) ranks the four settings via the resultingΔ\\Delta:−69\.83\-69\.83acc on CIFAR,−8\.61\-8\.61on ImageNet,\+0\.37\+0\.37PPL on SlimPajama,\+0\.0014\+0\.0014ROUGE on SuperNI/LoRA — the mechanism is load\-bearing on aligned partitions and vestigial on fuzzy ones\.

Table 5:SE/routing decomposition across the four settings\.SEΔ\\Deltais the shared expert’s architectural contribution \(No\-Routing→\\to\+\+SE, available to every method\); routingΔ\\Deltais the mechanism’s marginal contribution under matched supervision \(\+\+SE→\\toours, identical parameters and compute\)\. Deployed CIFAR uses no SE \(X=0X\{=\}0, App\.[A\.1](https://arxiv.org/html/2608.04084#A1.SS1)\), so its routingΔ\\Deltais measured from bare No\-Routing\.The routing share is\+16\.15\+16\.15and\+6\.53\+6\.53on the aligned vision partitions, versus\+0\.10\+0\.10PPL and\+0\.0012\+0\.0012ROUGE\-L \(both within seed noise\) on the fuzzy ones \(Tab\.[5](https://arxiv.org/html/2608.04084#S5.T5)\)\.

On SuperNI, LoRA branches specialize on a partition different from the imposedK=20K\{=\}20\(0/150/15but2\.1×2\.1\\timessigned diag/off at s42, App\.[E\.12](https://arxiv.org/html/2608.04084#A5.SS12)\); specialization is necessary but not sufficient when the imposed partition mismatches the underlying data structure\. An a\-priori embedding\-validity diagnostic \(App\.[E\.15](https://arxiv.org/html/2608.04084#A5.SS15)\) corroborates the modality split from data alone: BGE text\-chunk embeddings yield silhouette0\.0310\.031\(continuous manifold\) while DINOv2 image embeddings yield0\.0690\.069\(discrete clusters\), predicting which modalities support categorical routing without consulting task metrics\.

## 6Conclusion

We have argued that specialization in modular networks is governed by the alignment between training\-signal granularity and the target categories, not the routing function alone\. Across four settings, the mechanism’s contribution is qualitatively positive on aligned partitions and indistinguishable from architecture\-matched controls on fuzzy ones: aligned CIFAR\-100 superclass \(\+16\.15\+16\.15over the matched No\-Routing control\) and ViT BREEDS \(\+6\.53\+6\.53over matched\-SE\) yield clear gains; fuzzy SlimPajama\-6B \(\+0\.10\+0\.10PPL, tied\) and anti\-aligned SuperNI/LoRA \(mean tie\) show no significant routing contribution\. The HardCategory ablation isolates the active ingredient: one\-hot routing collapses5\.415\.41below No\-Routing on CIFAR despite reaching tautological100%100\\%branch–category alignment by construction, so the load\-bearing component is the cross\-category leakagepi\>0p\_\{\\mathrm\{i\}\}\{\>\}0, not metadata access or alignment itself\. Granularity alignment, not algorithm choice, localizes when routing helps\.

#### What the label buys: the masking control and its scope\.

An information\-matched control sharpens what the vision accuracy gains mean\. Given the same label at inference, simply masking a dense model’s logits to its fine classes is stronger for accuracy alone: masked dense reaches85\.2385\.23on CIFAR\-100 and83\.6583\.65on ImageNet, above ours at79\.2379\.23/79\.8979\.89\(3\-seed control, App\.[E\.3](https://arxiv.org/html/2608.04084#A5.SS3)\)\. Where the output space is hierarchically partitioned by the category and the deployment goal is only accuracy under a trusted label, logit masking is the stronger and cheaper mechanism at the scales we test; SpecDrop’s contribution is inducing modular structure under matched supervision, and the accuracy comparisons in the paper \(against No\-Routing controls without inference\-time masking\) should be read in that scope\. Applying the same masking to SpecDrop’s own outputs adds exactly0\.000\.00on CIFAR \(its predictions leave the given superclass once in30,00030\{,\}000across three seeds, versus16\.3%16\.3\\%of the time for dense\) and\+1\.06\+1\.06on ImageNet: the output\-space restriction is largely internalized during training, and what remains beyond it is the trained\-in structure of Tabs\.[1](https://arxiv.org/html/2608.04084#S5.T1)–[2](https://arxiv.org/html/2608.04084#S5.T2)\(branch–category alignment, per\-category pruning, selective deployment\) that a monolithic model, masked or not, has no substructure to support\. The control itself exists only where the output space is hierarchically partitioned by the category; on SlimPajama and SuperNI the output space \(a shared vocabulary, free\-form text\) admits no category masking\.

#### The matched\-supervision margin as an instrument\.

The\+6\.53\+6\.53margin over the matched\-SE control is a category\-oracle ceiling on what category structure buys through routing at this scale, and learned routers can be scored against it\. The tuned Soft MoE recovers\+3\.33\+3\.33of the ceiling \(76\.6976\.69vs73\.3673\.36\), almost exactly half, while SpecDrop retains a3\.203\.20\-point margin at matched parameters; the ALF router \(71\.0971\.09\) and Mod\-Squad \(70\.1170\.11\) remain below the73\.3673\.36control \(App\.[E\.7](https://arxiv.org/html/2608.04084#A5.SS7)\)\.

#### Limitations\.

Three axes frame our results\. On scale, our experiments span 30M language models, with a 125M 3\-seed replication verifying the same direction \(App\.[E\.10](https://arxiv.org/html/2608.04084#A5.SS10)\); the≥10\\geq 10B Mixture\-of\-Experts and≥1\\geq 1B Chinchilla\-optimal regimes remain open\. On inference metadata, the category label is required at deployment, and on the two vision settings the label is target\-derived \(CIFAR\-100 superclass is a coarsening of the fine label; ImageNet/BREEDS supercategories are constructed from fine labels\), so the matched\-architecture comparisons \(No\-Routing and No\-Routing\+SE on the same partition\) rather than the dense reference are the supervision\-controlled comparisons\. When the label must be predicted, the label\-quality curve \(App\.[E\.4](https://arxiv.org/html/2608.04084#A5.SS4)\) makes the applicability condition quantitative\. SpecDrop stays above the architecture\-matched control down to≈80%\{\\approx\}80\\%category\-label accuracy, a bar that training\-free predictors \(the dense model’s own prediction coarsened through the hierarchy;≈2×\{\\approx\}2\\timesdeployment cost, App\.[E\.4](https://arxiv.org/html/2608.04084#A5.SS4)\) clear on both vision settings: the83\.8%83\.8\\%CIFAR predictor retains\+6\.2\+6\.2over the matched control \(69\.369\.3vs63\.0863\.08\), and the88\.5%88\.5\\%ImageNet predictor retains\+0\.9\+0\.9\(74\.2574\.25vs73\.3673\.36\)\. Beating the dense reference instead requires≈92\{\\approx\}92–94%94\\%label accuracy, which no capacity\-comparable predictor reaches on CIFAR’s 5\-classes\-per\-category partition \(a coarse head fine\-tuned from the dense checkpoint attains∼84\.5%\{\\sim\}84\.5\\%\); under predicted labels the vision settings do not clear that higher bar — that too is our own measurement, and it is why the applicability condition \(a trusted category label available at inference\) is load\-bearing\. On theory, our results characterize the construction rather than provide tight bounds\.

#### Scope on dense prediction\.

For segmentation, detection, or VQA, a single input contains multiple categories, so whole\-input tags break the one\-clean\-tag alignment condition; the thesis makes a falsifiable prediction there: no gain over matched controls without region\-level tags, the same reduction\-to\-control observed on SlimPajama and SuperNI\. The alignment condition is restorable: per\-pixel class labels coarsen to per\-region superclass tags, and our merge already applies the mask per sample \(a\(B,1,K,1\)\(B,1,K,1\)broadcast over tokens\), so a region\-level tag promotes it to per\-token\(B,T,K\)\(B,T,K\), the fixed\-signal analogue of V\-MoE’s per\-token routing\. Testing whether region\-level tags recover the vision\-classification gains is the direct falsification test of the thesis on dense prediction\.

#### Impact statement\.

The granularity\-alignment characterization guides when category\-conditioned routing yields gains; deployments should ensure the category signal does not encode sensitive demographic or proxy attributes, since routing could amplify upstream bias\.

#### Future work\.

Token\-level tag attribution could recover the routing signal lost to chunk\-level coarsening \(56\.1%56\.1\\%of512512\-token chunks span≥2\{\\geq\}2BGE clusters,n=9,766n\{=\}9\{,\}766, App\.[E\.15](https://arxiv.org/html/2608.04084#A5.SS15)\); scaling to Chinchilla\-optimal regimes would test the thesis at scale; predictor co\-training would address the inference\-metadata caveat\.*\(i\) Closed\-form auxiliary\-loss\-free balancing*\(bridging Sec\.[2](https://arxiv.org/html/2608.04084#S2)’s trajectory\): with a fixed assignment, branch load is computable in closed form from category frequencies, and the per\-categoryβ\\beta\-amplification already plays, statically, the role ofWang et al\. \([2024](https://arxiv.org/html/2608.04084#bib.bib40)\)’s online bias correction; combining a category prior with their bias update is the natural synthesis when category structure is informative but imperfect\.*\(ii\) Fixed\-to\-learned handover*: StableMoE\(Dai et al\.,[2022](https://arxiv.org/html/2608.04084#bib.bib6)\)distills a learned router into a frozen one; the mirror\-image curriculum \(train under the fixed signal, then hand off to a learned router needing no labels at inference\) would import SpecDrop’s specialization into standard MoE deployment\.

## References

- Achille and Soatto \[2018\]Alessandro Achille and Stefano Soatto\.Information dropout: Learning optimal representations through noisy computation\.*IEEE Transactions on Pattern Analysis and Machine Intelligence*, 40\(12\):2897–2905, 2018\.
- Ba and Frey \[2013\]Jimmy Ba and Brendan Frey\.Adaptive dropout for training deep neural networks\.In*Advances in Neural Information Processing Systems \(NIPS\)*, volume 26, 2013\.
- Béna and Goodman \[2025\]Gabriel Béna and Dan F M Goodman\.Dynamics of specialization in neural modules under resource constraints\.*Nature Communications*, 16\(1\):187, 2025\.doi:10\.1038/s41467\-024\-55188\-9\.
- Chen et al\. \[2023a\]Tianlong Chen, Zhenyu Zhang, Ajay Jaiswal, Shiwei Liu, and Zhangyang Wang\.Sparse MoE as the new dropout: Scaling dense and self\-slimmable transformers\.In*International Conference on Learning Representations \(ICLR\)*, 2023a\.
- Chen et al\. \[2023b\]Zitian Chen, Yikang Shen, Mingyu Ding, Zhenfang Chen, Hengshuang Zhao, Erik Learned\-Miller, and Chuang Gan\.Mod\-squad: Designing mixtures of experts as modular multi\-task learners\.In*IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2023b\.
- Dai et al\. \[2022\]Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, and Furu Wei\.StableMoE: Stable routing strategy for mixture of experts\.In*Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(ACL\)*, pages 7085–7095, 2022\.
- Dai et al\. \[2024\]Damai Dai, Chengqi Deng, Chenggang Zhao, R X Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al\.DeepSeekMoE: Towards ultimate expert specialization in mixture\-of\-experts language models\.In*Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2024\.
- Deng et al\. \[2009\]Jia Deng, Wei Dong, Richard Socher, Li\-Jia Li, Kai Li, and Li Fei\-Fei\.ImageNet: A large\-scale hierarchical image database\.In*IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 248–255, 2009\.
- Dosovitskiy et al\. \[2021\]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby\.An image is worth 16x16 words: Transformers for image recognition at scale\.In*International Conference on Learning Representations \(ICLR\)*, 2021\.
- Dou et al\. \[2024\]Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, et al\.LoRAMoE: Alleviating world knowledge forgetting in large language models via MoE\-style plugin\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\)*, pages 1932–1945, 2024\.
- Fan et al\. \[2020\]Angela Fan, Edouard Grave, and Armand Joulin\.Reducing transformer depth on demand with structured dropout\.In*International Conference on Learning Representations \(ICLR\)*, 2020\.
- Fan et al\. \[2021\]Xinjie Fan, Shujian Zhang, Korawat Tanwisuth, Xiaoning Qian, and Mingyuan Zhou\.Contextual dropout: An efficient sample\-dependent dropout module\.In*International Conference on Learning Representations \(ICLR\)*, 2021\.
- Fedus et al\. \[2022\]William Fedus, Barret Zoph, and Noam Shazeer\.Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity\.*Journal of Machine Learning Research*, 23\(120\):1–39, 2022\.
- Fernando et al\. \[2017\]Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra\.PathNet: Evolution channels gradient descent in super neural networks\.*arXiv preprint arXiv:1701\.08734*, 2017\.
- Gou et al\. \[2024\]Yunhao Gou, Zhili Liu, Kai Chen, Lanqing Hong, Hang Xu, Xin Jiang, Aoxue Li, Zhenguo Li, Dit\-Yan Yeung, James T Kwok, and Yu Zhang\.Mixture of cluster\-conditional LoRA experts for vision\-language instruction tuning\.*arXiv preprint arXiv:2312\.12379*, 2024\.
- Gururangan et al\. \[2022\]Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A Smith, and Luke Zettlemoyer\.DEMix layers: Disentangling domains for modular language modeling\.In*Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\)*, 2022\.
- He et al\. \[2016\]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun\.Deep residual learning for image recognition\.In*IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 770–778, 2016\.
- Hu et al\. \[2022\]Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen\.LoRA: Low\-rank adaptation of large language models\.In*International Conference on Learning Representations \(ICLR\)*, 2022\.
- Huang et al\. \[2016\]Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger\.Deep networks with stochastic depth\.In*European Conference on Computer Vision \(ECCV\)*, pages 646–661\. Springer, 2016\.
- Jacobs et al\. \[1991\]Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton\.Adaptive mixtures of local experts\.*Neural Computation*, 3\(1\):79–87, 1991\.
- Jiang et al\. \[2024\]Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al\.Mixtral of experts\.*arXiv preprint arXiv:2401\.04088*, 2024\.
- Krizhevsky \[2009\]Alex Krizhevsky\.Learning multiple layers of features from tiny images\.Technical Report TR\-2009, University of Toronto, 2009\.
- Li et al\. \[2022\]Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyer\.Branch\-train\-merge: Embarrassingly parallel training of expert language models\.*arXiv preprint arXiv:2208\.03306*, 2022\.
- Liu et al\. \[2023\]Zhuang Liu, Zhiqiu Xu, Joseph Jin, Zhiqiang Shen, and Trevor Darrell\.Dropout reduces underfitting\.In*International Conference on Machine Learning \(ICML\)*, 2023\.
- Ma et al\. \[2018\]Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi\.Modeling task relationships in multi\-task learning with multi\-gate mixture\-of\-experts\.In*ACM SIGKDD International Conference on Knowledge Discovery and Data Mining*, pages 1930–1939, 2018\.
- Maini et al\. \[2023\]Pratyush Maini, Michael C Mozer, Hanie Sedghi, Zachary C Lipton, J Zico Kolter, and Chiyuan Zhang\.Can neural network memorization be localized?In*International Conference on Machine Learning \(ICML\)*, 2023\.
- Mallya et al\. \[2018\]Arun Mallya, Dillon Davis, and Svetlana Lazebnik\.Piggyback: Adapting a single network to multiple tasks by learning to mask weights\.In*European Conference on Computer Vision \(ECCV\)*, pages 72–88, 2018\.
- Puigcerver et al\. \[2024\]Joan Puigcerver, Carlos Riquelme, Basil Mustafa, and Neil Houlsby\.From sparse to soft mixtures of experts\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.
- Riquelme et al\. \[2021\]Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby\.Scaling vision with sparse mixture of experts\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, volume 34, pages 8583–8595, 2021\.
- Roller et al\. \[2021\]Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston\.Hash layers for large sparse models\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2021\.
- Santurkar et al\. \[2021\]Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry\.BREEDS: Benchmarks for subpopulation shift\.In*International Conference on Learning Representations \(ICLR\)*, 2021\.
- Shaier et al\. \[2025\]Sagi Shaier, Francisco Pereira, Katharina von der Wense, Lawrence E Hunter, and Matt Jones\.More experts than galaxies: Conditionally\-overlapping experts with biologically\-inspired fixed routing\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.
- Shunk \[2022\]Joshua Shunk\.Neuron\-specific dropout: A deterministic regularization technique to prevent neural networks from overfitting and reduce dependence on large training samples\.*arXiv preprint arXiv:2201\.06938*, 2022\.
- Su et al\. \[2024\]Zhenpeng Su, Zijia Lin, Xue Bai, Xing Wu, Yizhe Xiong, Haoran Lian, Guangyuan Ma, Hui Chen, Guiguang Ding, Wei Zhou, and Songlin Hu\.MaskMoE: Boosting token\-level learning via routing mask in mixture\-of\-experts\.*arXiv preprint arXiv:2407\.09816*, 2024\.
- Sukhbaatar et al\. \[2024\]Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Rozière, Jacob Kahn, Daniel Li, Wen\-tau Yih, Jason Weston, and Xian Li\.Branch\-train\-MiX: Mixing expert LLMs into a mixture\-of\-experts LLM\.In*Conference on Language Modeling \(COLM\)*, 2024\.
- Szegedy et al\. \[2015\]Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich\.Going deeper with convolutions\.In*IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 1–9, 2015\.
- Tian et al\. \[2024\]Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Chengzhong Xu\.HydraLoRA: An asymmetric LoRA architecture for efficient fine\-tuning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024\.
- Touvron et al\. \[2021\]Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou\.Training data\-efficient image transformers & distillation through attention\.In*International Conference on Machine Learning \(ICML\)*, 2021\.
- Vaswani et al\. \[2017\]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin\.Attention is all you need\.In*Advances in Neural Information Processing Systems \(NIPS\)*, 2017\.
- Wang et al\. \[2024\]Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai\.Auxiliary\-loss\-free load balancing strategy for mixture\-of\-experts\.*arXiv preprint arXiv:2408\.15664*, 2024\.
- Wang et al\. \[2026\]Yan Wang, Yitao Xu, Nanhan Shen, Jinyan Su, Jimin Huang, and Zining Zhu\.The illusion of specialization: Unveiling the domain\-invariant “standing committee” in mixture\-of\-experts models\.*arXiv preprint arXiv:2601\.03425*, 2026\.
- Wang et al\. \[2022\]Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, et al\.Super\-NaturalInstructions: Generalization via declarative instructions on 1600\+ NLP tasks\.In*Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2022\.
- Wang et al\. \[2025\]Ziteng Wang, Jun Zhu, and Jianfei Chen\.ReMoE: Fully differentiable mixture\-of\-experts with ReLU routing\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.
- Wortsman et al\. \[2020\]Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi\.Supermasks in superposition\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2020\.
- Xie et al\. \[2017\]Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He\.Aggregated residual transformations for deep neural networks\.In*IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 1492–1500, 2017\.
- Zhou et al\. \[2022\]Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Zhifeng Chen, Quoc V Le, and James Laudon\.Mixture\-of\-experts with expert choice routing\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2022\.
- Zoph et al\. \[2022\]Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus\.ST\-MoE: Designing stable and transferable sparse expert models\.*arXiv preprint arXiv:2202\.08906*, 2022\.

## Appendix AStochastic SpecDrop Formulation and Additional Ablations

This appendix contains \(i\) the Stochastic \(Bernoulli\) SpecDrop variant against which the canonical Soft variant is benchmarked, \(ii\) a four\-corner mask×\\timesdenominator ablation at the ResNet\-110 / CIFAR\-100 scale, and \(iii\) a random\-permutation robustness check on the assignment matrix𝐀\\mathbf\{A\}\. All results support the main\-text claim that the category\-asymmetric*deterministic*soft signal is the active ingredient; the Bernoulli variant and the specific𝐀\\mathbf\{A\}ordering are not\.

### A\.1Full Phase A/B/C Hyperparameter Ablation

Section[5\.1](https://arxiv.org/html/2608.04084#S5.SS1.SSS0.Px3)summarizes the 3\-phase hyperparameter ablation\. The full per\-value sweep is below; take\-aways follow the same pattern described in the main text\.

Table 6:Full Soft SpecDrop hyperparameter ablation on CIFAR\-100\.MultiBranchResNet110,K=20K\{=\}20, 200 epochs, 3 seeds\. Phase A:pactivep\_\{\\text\{active\}\}sweep atwr=0w\_\{r\}\{=\}0\(no warmup\) with a1\.0×1\.0\\timesshared expert\. Phase B: SE capacity ratio at\(pa,wr\)=\(0\.7,1\.0\)\(p\_\{a\},w\_\{r\}\)\{=\}\(0\.7,1\.0\)\(cosine warmup, deployed\)\. Bold = phase\-wise best\. The Phase B0×0\\timesSE value79\.23±0\.1779\.23\\pm 0\.17is the deployed number reported in Tab\.[1](https://arxiv.org/html/2608.04084#S5.T1); the1\.821\.82gap from Phase A’spa=0\.7p\_\{a\}\{=\}0\.7optimum \(77\.4177\.41\) combines the cosine\-warmup contribution \(\+0\.90\+0\.90at the same1\.0×1\.0\\timesSE, Phase B\) with the effect of removing the SE \(\+0\.92\+0\.92\)\.#### Extended take\-aways\.

Phase A directly tests the central theoretical prediction: a non\-trivial cross\-category leakage term \(pinactive\>0p\_\{\\text\{inactive\}\}\{\>\}0\) is necessary — both fully uniform \(pa=0\.5p\_\{a\}\{=\}0\.5, collapse to62\.00%62\.00\\%\) and fully hard routing \(pa=1\.0p\_\{a\}\{=\}1\.0,76\.95%76\.95\\%\) are strictly worse than the dual\-probability optimumpa=0\.7p\_\{a\}\{=\}0\.7\. Phase B reveals a modality\-dependent finding: an always\-on shared expert helps on imbalanced domains \(NLP, Section[5\.3](https://arxiv.org/html/2608.04084#S5.SS3)\) but dilutes the routing signal on balanced CIFAR\-100\.

### A\.2Stochastic SpecDrop

Section[3\.4](https://arxiv.org/html/2608.04084#S3.SS4)of the main text presents Soft SpecDrop \(deterministic soft weighting\) as our canonical variant; here we document the original Bernoulli sampling scheme for completeness\. For each training sample with categorycc, we draw an independent Bernoulli mask per module:

mk∼Bernoulli​\(pk​\(c\)\),outputtrain=∑kmk​hkS\+hSE,m\_\{k\}\\sim\\mathrm\{Bernoulli\}\\bigl\(p\_\{k\}\(c\)\\bigr\),\\qquad\\text\{output\}\_\{\\text\{train\}\}=\\frac\{\\sum\_\{k\}m\_\{k\}\\,h\_\{k\}\}\{S\}\\;\+\\;h\_\{\\mathrm\{SE\}\},\(6\)whereS=pactive\+\(K−1\)​pinactiveS=p\_\{\\text\{active\}\}\+\(K\-1\)p\_\{\\text\{inactive\}\}is the fixed constant of Prop\.[1](https://arxiv.org/html/2608.04084#Thmproposition1)andhSEh\_\{\\mathrm\{SE\}\}is the shared\-expert output defined in Eq\.[4](https://arxiv.org/html/2608.04084#S3.E4)\(added*after*÷S\\div S, by the same design as Soft SpecDrop; set to zero when no shared expert is used\)\. At inference,mk=pk​\(c\)m\_\{k\}=p\_\{k\}\(c\)deterministically, giving the same output expression as Soft SpecDrop\. The stochastic masking acts as a regularizer analogous to standard dropout, and the train–test expectations match by Prop\.[1](https://arxiv.org/html/2608.04084#Thmproposition1)\(a\) \(linearity under the fixed denominator\)\. Soft SpecDrop removes Bernoulli mask\-sampling variance, so thepk​\(c\)/Sp\_\{k\}\(c\)/Sscaling in Thm\.[1](https://arxiv.org/html/2608.04084#Thmtheorem1)is exact at training time \(thepa/pip\_\{a\}/p\_\{i\}ratio still requires the stated base\-gradient symmetry\); this motivates the choice of Soft as the default variant in the main experiments\.

### A\.3Mask×\\timesDenominator Ablation

We decompose Soft SpecDrop into its orthogonal design axes: whether masks are*stochastic*\(Bernoullimkm\_\{k\}\) or*deterministic*\(soft weightpk​\(c\)p\_\{k\}\(c\)\), and whether the merge denominator is*fixed*\(÷S\\div S\) or*stochastic*\(÷∑kmk\\div\\sum\_\{k\}m\_\{k\}\)\. Proposition[1](https://arxiv.org/html/2608.04084#Thmproposition1)singles out the fixed denominator as the unique category\-independent constant that gives exact train–test consistency; the naive stochastic denominator incurs a closed\-form Jensen bias\. Table[7](https://arxiv.org/html/2608.04084#A1.T7)isolates all four combinations at the paper\-faithful scale \(MultiBranchResNet110,K=20K\{=\}20, 200 epochs, 3 seeds\), plus two random\-dropout references that remove the category\-conditioned matrix𝐀\\mathbf\{A\}entirely\.

Table 7:Mask×\\timesdenominator ablationon CIFAR\-100 \(MultiBranchResNet110,K=20K\{=\}20, 200 epochs; 3 seeds for all rows, with standard deviation reported only for the deployed Soft \+ fixedSSrow\)\. The Soft \+ fixedSSrow uses the deployed canonical config\(pa,wr,SE\)=\(0\.7,1\.0,0×\)\(p\_\{a\},w\_\{r\},\\text\{SE\}\)\{=\}\(0\.7,1\.0,0\\times\)matching Tab\.[1](https://arxiv.org/html/2608.04084#S5.T1); the Bernoulli \(Stochastic\) variants use\(pactive,pinactive\)=\(0\.9,0\.1\)\(p\_\{\\text\{active\}\},p\_\{\\text\{inactive\}\}\)\{=\}\(0\.9,0\.1\)tuned for the Bernoulli regime per App\.[A](https://arxiv.org/html/2608.04084#A1); random\-dropout \(no\-𝐀\\mathbf\{A\}\) comparisons remove category conditioning entirely\. The decomposition isolates deterministic weighting, fixed\-denominator, and category conditioning along three design axes; mask probabilities are tuned per regime as listed above, so cross\-regime comparisons additionally absorb a probability shift\.MaskDenominatorTrain–test consistent?Top\-1 \(%\)Soft \(deterministicpk​\(c\)p\_\{k\}\(c\)\)fixedSSyes \(trivially, Prop\.[1](https://arxiv.org/html/2608.04084#Thmproposition1)\)79\.23±0\.17\\mathbf\{79\.23\\pm 0\.17\}Stochastic \(Bernoullimkm\_\{k\}\)fixedSSyes \(Prop\.[1](https://arxiv.org/html/2608.04084#Thmproposition1)\)64\.5564\.55Stochastic \(Bernoullimkm\_\{k\}\)stochastic∑kmk\\sum\_\{k\}m\_\{k\}no \(Jensen bias\)67\.5467\.54Random dropout \(no𝐀\\mathbf\{A\}\)fixedSSyes58\.9758\.97Random dropout \(no𝐀\\mathbf\{A\}\)stochastic∑kmk\\sum\_\{k\}m\_\{k\}no \(Jensen bias\)59\.5359\.53#### Take\-aways\.

Two quantitative effects decompose the method\.*\(i\) Deterministic soft weighting is the dominant contributor\.*Within category\-conditioned variants at fixedSS, replacing the deterministicpk​\(c\)p\_\{k\}\(c\)with Bernoulli sampling costs14\.68%14\.68\\%\(79\.23→64\.5579\.23\\to 64\.55\) — the single largest effect in the table\.*\(ii\) Category conditioning contributes independently on top of soft weighting\.*Comparing the two Bernoulli\-masked variants at fixedSS*with*vs\.*without*category conditioning isolates the effect of𝐀\\mathbf\{A\}:64\.5564\.55vs\.58\.97=\+5\.58%58\.97=\+5\.58\\%\. The fixed\-versus\-stochastic\-denominator distinction, theoretically predicted to favor fixed via Jensen\-bias elimination, is less clean empirically in the Bernoulli regime: stoch\_fixed64\.5564\.55vs\. stoch\_naive67\.5467\.54\(direction inverted from the theoretical prediction, consistent with the Bernoulli experiments’ own∼2%\\sim 2\\%seed\-to\-seed variance dominating the bias correction\)\. In the deterministic regime — where we operate in Soft SpecDrop — the fixed denominatorSSapplies by linearity \(Prop\.[1](https://arxiv.org/html/2608.04084#Thmproposition1)\(a\)\) and the Jensen\-bias issue does not arise\. The random\-dropout floor \(58\.9758\.97/59\.5359\.53\) confirms that without𝐀\\mathbf\{A\}, the mask mechanism alone is worse than the63\.0863\.08No\-Routing baseline by∼4%\\sim 4\\%: it is the*asymmetric signal shaped by categories*, not the masking itself, that drives the gain\.

### A\.4Assignment\-Matrix Geometry: Round\-Robin vs\. Random Permutation

To rule out the hypothesis that the specific round\-robin assignment𝐀rr\\mathbf\{A\}\_\{\\text\{rr\}\}is responsible for the gain, we replace𝐀rr\\mathbf\{A\}\_\{\\text\{rr\}\}with a random permutation𝐀rand\\mathbf\{A\}\_\{\\text\{rand\}\}\(each category still assigned to exactly one module, but the category\-to\-module mapping is a uniform random bijection; assignment seeds42/123/45642/123/456\)\. Random permutation gives78\.69±0\.27%78\.69\\pm 0\.27\\%versus round\-robin’s79\.23±0\.17%79\.23\\pm 0\.17\\%— a gap of−0\.54%\-0\.54\\%\(≈1\.7​σ\{\\approx\}1\.7\\sigmaunder the two configurations’ combined seed variance\)\. The gradient asymmetrypactive/pinactivep\_\{\\text\{active\}\}/p\_\{\\text\{inactive\}\}is the active ingredient; the method requires only that𝐀\\mathbf\{A\}be a bijection, not a carefully chosen semantic grouping\.

## Appendix BBaseline Implementation Details

We describe the implementation of each baseline used in the main comparisons\. The CIFAR\-100 baselines \(Section[5\.1](https://arxiv.org/html/2608.04084#S5.SS1)\) all run on the dense ResNet\-110 backbone exactly as in their original papers, with no multi\-branch wrapping\. The NLP baselines \(Section[5\.3](https://arxiv.org/html/2608.04084#S5.SS3)\) are paper\-canonical MoE architectures, each implemented as a dedicated model class; the algorithm\-plugin pathway used by SpecDrop is bypassed because routing is intrinsic to these models\.

### B\.1CIFAR\-100 baselines

#### Stochastic Depth\[Huang et al\.,[2016](https://arxiv.org/html/2608.04084#bib.bib19)\]\.

Block\-level Bernoulli drop on dense ResNet\-110, with the linear survival schedulepℓ=1−ℓL​\(1−pL\)p\_\{\\ell\}=1\-\\frac\{\\ell\}\{L\}\(1\-p\_\{L\}\)andpL=0\.5p\_\{L\}\\\!=\\\!0\.5exactly as in Equation 4 of the original paper\. Survival probabilities are stored on the modules and consumed inside eachBasicBlock\.forward; the residual identity is preserved when a block is dropped \(noReLUon the shortcut\)\. At inference the per\-block expected scaling is applied\.

#### Example\-Tied Dropout\[Maini et al\.,[2023](https://arxiv.org/html/2608.04084#bib.bib26)\]\.

Per\-example fixed binary masks over a memorization\-channel subset, applied per\-block within the dense ResNet\-110 residual path\. We follow the paper’s deployment protocol from its Section 6\.2: at test time the memorization neurons are zeroed out, leaving only the generalization channels active\. This protocol improves test accuracy on the 9\-layer ResNet of the original paper, but on a 54\-layer ResNet\-110 it removes25%25\\%of the convolutional capacity at every block; we report the paper\-faithful protocol rather than tuning it off\-protocol\.

#### Contextual Dropout\[Fan et al\.,[2021](https://arxiv.org/html/2608.04084#bib.bib12)\]\.

Gaussian variant with the paper’s scaled sigmoidσt​\(α\)=σ​\(0\.01​α\)\\sigma\_\{t\}\(\\alpha\)=\\sigma\(0\.01\\alpha\)as a numerical stabilizer, placed inside the residual path between the two convolutions, matching the WRN placement of Figure 6 of the paper\. The context network is the smallest variant from the paper\.

#### No\-Routing\.

The same MultiBranchResNet110 architecture used by Soft SpecDrop, with all branch weights fixed to1/K1/Kfor every input\. This isolates the cost of the multi\-branch architecture from the routing signal: any gap between Soft SpecDrop and No\-Routing is attributable purely to the category\-conditioned routing and the fixed\-denominator merge\.

### B\.2NLP baselines

All NLP baselines share the same backbone \(6\-layer Transformer LM, hidden384384,66heads, max sequence length512512, vocab50,25750\{,\}257\) and the same training recipe \(Section[5\.3](https://arxiv.org/html/2608.04084#S5.SS3)\); only the FFN and routing differ\. Each baseline is implemented as a stand\-alone model class so that routing is intrinsic to the architecture rather than a plug\-in mask layer\.

#### Switch Transformer\[Fedus et al\.,[2022](https://arxiv.org/html/2608.04084#bib.bib13)\]\.

Paper\-canonicalN=32N\\\!=\\\!32experts with FFN hidden4848each \(exact total parameter match to dense FFN15361536\)\. Each token is routed to the top\-11expert via a learned linear router; the selected expert’s output is scaled by its softmax gate valuept​\[i∗\]p\_\{t\}\[i^\{\*\}\]\(Fedus 2022 §2\.1\), so gradients flow through the gate naturally\. The auxiliary load\-balance loss \(paper Eq\. 4\) is added with weight0\.010\.01\.

#### Hash Layers\[Roller et al\.,[2021](https://arxiv.org/html/2608.04084#bib.bib30)\]\.

Scaled\-downN=8N\\\!=\\\!8experts with FFN hidden192192each \(parameter\-matched; the original paper’s smallest configuration isN=16N\{=\}16\)\. Each token is mapped to one expert by a fixed random hash table over the vocabulary; the hash table is drawn once at init from a fixed seed and frozen\. No routing parameters and no auxiliary loss; per\-layer hash tables use offset seeds\.

#### SMoE\-Dropout\[Chen et al\.,[2023a](https://arxiv.org/html/2608.04084#bib.bib4)\]\.

Paper Figure 5 settingN=16N\\\!=\\\!16experts with FFN hidden9696each\. Routing uses a fixed random Linear projection \(no learned parameters\); top\-kkselection with softmax\-normalized weights, wherekkfollows a linear schedule from11toKKover training \([Chen et al\.](https://arxiv.org/html/2608.04084#bib.bib4),kt=kinit\+\(K−kinit\)⋅t/Tk\_\{t\}=k\_\{\\text\{init\}\}\+\(K\-k\_\{\\text\{init\}\}\)\\cdot t/T\)\. We disable the per\-expert Bernoulli dropout \(expert\_drop\_prob=0\.00\.0\) to match the paper\.

#### DEMix\[Gururangan et al\.,[2022](https://arxiv.org/html/2608.04084#bib.bib16)\]\.

One FFN per domain \(N=7N\\\!=\\\!7\), FFN hidden220220each \(parameter\-matched\)\. Hard per\-domain routing using the document’s domain tag during training; at inference we use the mixture\-of\-experts inference ofGururangan et al\. \[[2022](https://arxiv.org/html/2608.04084#bib.bib16)\]\(§5\.2\) with uniform mixture weights\.

#### Dense Transformer\.

A single FFN of hidden15361536, no routing\. This is the parameter and compute reference for every multi\-branch method\.

#### No\-Routing \(NLP\)\.

K=7K\\\!=\\\!7FFN branches of width220220each, combined by uniform1/K1/Kweighting at every layer\. Architecturally identical to the multi\-branch SpecDrop variant but with the routing signal removed; bounds the best perplexity any routing algorithm can achieve at this branch count and width\.

#### Batch\-size disclosure \(NLP\)\.

All NLP perplexity numbers \(Sec\.[5\.3](https://arxiv.org/html/2608.04084#S5.SS3), Table[3](https://arxiv.org/html/2608.04084#S5.T3)\) are from training runs with per\-device batch size3232sequences of length512512\(16,38416\{,\}384tokens per optimizer step\)\. We verified the method rankings of Table[3](https://arxiv.org/html/2608.04084#S5.T3)are stable to a doubling of the per\-device batch\.

#### NLP training\-regime disclosure \(multi\-epoch over Chinchilla\-optimal\)\.

All NLP experiments \(Sec\.[5\.3](https://arxiv.org/html/2608.04084#S5.SS3), mini\-ablation Sec\.[5\.5](https://arxiv.org/html/2608.04084#S5.SS5), exploratory App\.[E\.15](https://arxiv.org/html/2608.04084#A5.SS15)\) train for1010epochs on500500M unique SlimPajama tokens, totaling55B token\-passes \(≈167\\approx 167tokens per parameter at3030M scale, well above the Chinchilla\-optimal∼20\\sim 20tokens per parameter\); the125125M scale\-up \(App\.[E\.10](https://arxiv.org/html/2608.04084#A5.SS10)\) preserves this regime to isolate model size as the only varying factor\. We retain the1010\-epoch regime to preserve internal consistency across the NLP\-side ablation cells \(Sec\.[5\.5](https://arxiv.org/html/2608.04084#S5.SS5), App\.[E\.6](https://arxiv.org/html/2608.04084#A5.SS6)\); a11\-epoch sanity verification at the same total\-token budget is reported in App\.[E\.11](https://arxiv.org/html/2608.04084#A5.SS11)\.All NLP baselines \(Dense, No\-Routing, No\-Routing\+SE, Switch, Hash Layers, DEMix, SMoE\-Dropout, ours\) use the identical1010\-epoch regime, so cross\-method rankings of Table[3](https://arxiv.org/html/2608.04084#S5.T3)are internally fair\. We acknowledge this regime is over\-trained relative to standard11\-epoch LM\-pretraining practice \(Switch, GPT\-3, LLaMA, Chinchilla\); a3030M×\\times11\-epoch sanity verification of the ours\-vs\-matched\-SE\-scalar tie is reported in App\.[E\.11](https://arxiv.org/html/2608.04084#A5.SS11), and the125125M scale\-up of App\.[E\.10](https://arxiv.org/html/2608.04084#A5.SS10)shows the same conditional\-negative direction at4×4\\timesscale\. The granularity\-alignment thesis itself \(Sec\.[6](https://arxiv.org/html/2608.04084#S6)\) is anchored to data\-modality properties — intra\-chunk heterogeneity \(56\.1%56\.1\\%of512512\-token chunks span≥2\\geq 2BGE clusters atk=7k\{=\}7over the full9,7669\{,\}766\-chunk validation set, App\.[E\.15](https://arxiv.org/html/2608.04084#A5.SS15)\) and silhouette modality asymmetry \(BGEsmax=0\.031s\_\{\\max\}\{=\}0\.031vs DINOv2 CIFAR0\.0690\.069\) — which are properties of the data and embeddings, independent of training\-epoch count\.

#### Hyperparameter selection\.

Our method’s operating point is identified via a three\-stage sequential ablation \(Sec\.[5\.1](https://arxiv.org/html/2608.04084#S5.SS1.SSS0.Px3)for CIFAR, Sec\.[5\.5](https://arxiv.org/html/2608.04084#S5.SS5)for NLP\) at33seeds per cell\. Baselines use the paper\-canonical hyperparameters specified in their original publications \(Switch LB weight0\.010\.01, Hash Layers frozen random seeds, SMoE\-Dropoutexpert\_drop\_prob=0=0, DEMix hard domain routing, Mod\-Squadmi\_weight=0\.001=0\.001, COMETp\_keep=0\.25=0\.25\)\.

### B\.3ImageNet BREEDS\-46 Construction Algorithm

TheM=46M\{=\}46supercategory partition is derived from the BREEDS\[Santurkar et al\.,[2021](https://arxiv.org/html/2608.04084#bib.bib31)\]curated WordNet hierarchy via a recursive expansion controlled by two parameters,TT\(max\-leaves before forced expansion\) andCC\(max\-children allowed for an expansion to be accepted\)\. Starting from BREEDS’ level\-3 nodes \(2929groups covering most of the10001000ImageNet\-1K fine classes\), for each node whose number of ImageNet leaves exceedsTTwe attempt to expand into its WordNet children; the expansion is accepted only when the node has at mostCCnon\-empty child groups, which prevents degenerate splits where one child holds nearly all leaves and the rest become singletons \(e\.g\.,*carnivore*has∼25\\sim 25children but only*dog*carries substantial mass\)\.

WithT=60T\{=\}60,C=10C\{=\}10as our depth/branching cutoffs, this recursion produces4545groups covering890890classes; the remaining110110ImageNet classes that do not appear in the BREEDS curated tree are pooled into a single*miscellaneous*group, yieldingM=46M\{=\}46\. The resulting partition is imbalanced: the largest group is*carnivore*\(158158classes\), followed by*miscellaneous*\(110110\),*man\-made structure*\(6262\),*bird*\(5959\), and*equipment*\(4949\); a long tail of2121groups holds only22–88classes each \(e\.g\.,*aquatic mammal*,*marsupial*,*vascular plant*\)\. TheseTT,CCvalues are our hyperparameters for the recursive expansion \(the BREEDS curated tree itself comes from Santurkar et al\.\); at fixedC=10C\{=\}10, lowering the cutoff toT=20T\{=\}20refines the partition toK=73K\{=\}73, while raising it toT=100T\{=\}100leavesK=46K\{=\}46unchanged \(the only remaining tree group above that cutoff,*carnivore*, does not admit an expansion into≤C\{\\leq\}Cnon\-empty subgroups\)\.

### B\.4SuperNI K=20 Clustering Construction

TheM=K=20M\{=\}K\{=\}20task clusters used in Section[5\.4](https://arxiv.org/html/2608.04084#S5.SS4)are constructed by frequency\-cutoff over the SuperNI\[Wang et al\.,[2022](https://arxiv.org/html/2608.04084#bib.bib42)\]*Domains*field, not byKK\-means or sentence\-embedding clustering\.

Algorithm\.For each task in the756756\-task English training split: \(i\) take its first*Domains*entry and normalize it by extracting the root segment of the hierarchical path \(e\.g\.,*“Commonsense→\\toConcepts and Relations→\\toSocial Commonsense”*becomes*“Commonsense”*\), yielding∼72\\sim 72unique normalized root domains across the train split; \(ii\) count root\-domain frequency over training tasks; \(iii\) retain the top\-1919most frequent root domains as cluster IDs0–1818, with all remaining tasks assigned to cluster ID1919\(*miscellaneous*\)\. At test time, the119119held\-out tasks map to clusters via the same normalize\-and\-lookup function; tasks whose normalized root domain is unseen in the train\-frequency table fall back to the miscellaneous cluster\.

This is the BREEDS\-analog convention \(frequency\-cutoff with a miscellaneous bucket; cf\. App\.[B\.3](https://arxiv.org/html/2608.04084#A2.SS3)\), chosen for determinism and consistency with our ImageNet partition\. We did not performKK\-means over Wang et al\.’s task definitions; that is a reasonable alternative left to future work\. The choiceK=20K\{=\}20matches the CIFAR\-100 superclass count for cross\-setting comparison, and the trailing miscellaneous cluster holds3030of the119119test tasks, large enough to avoid being dominated by a single task family\.

## Appendix CProofs

### C\.1Proof of Theorem[1](https://arxiv.org/html/2608.04084#Thmtheorem1)\(Gradient Concentration\)

###### Proof\.

Letz=∑kmk​hk/Sz=\\sum\_\{k\}m\_\{k\}h\_\{k\}/Sdenote the merged routed output, soy^=fhead​\(z\)\\hat\{y\}=f\_\{\\text\{head\}\}\(z\)\. By the chain rule,∂ℒ∂θk=∂ℒ∂z⋅mkS⋅∂hk∂θk\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\theta\_\{k\}\}=\\frac\{\\partial\\mathcal\{L\}\}\{\\partial z\}\\cdot\\frac\{m\_\{k\}\}\{S\}\\cdot\\frac\{\\partial h\_\{k\}\}\{\\partial\\theta\_\{k\}\}\. Taking norms and then expectations overmm\(under the mask\-independence assumption \(i\)\):

𝔼m\[∥∂ℒ∂θk∥\|c\]=pk​\(c\)S⋅𝔼x\|c\[∥∂ℒ∂z⋅∂hk∂θk∥\]\.\\mathbb\{E\}\_\{m\}\\\!\\left\[\\\!\\left\\\|\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\theta\_\{k\}\}\\right\\\|\\;\\middle\|\\;c\\right\]=\\frac\{p\_\{k\}\(c\)\}\{S\}\\cdot\\mathbb\{E\}\_\{x\|c\}\\\!\\left\[\\\!\\left\\\|\\frac\{\\partial\\mathcal\{L\}\}\{\\partial z\}\\cdot\\frac\{\\partial h\_\{k\}\}\{\\partial\\theta\_\{k\}\}\\right\\\|\\right\]\.For assignedcc\(Ac​k=1A\_\{ck\}\{=\}1\),pk​\(c\)=pactivep\_\{k\}\(c\)=p\_\{\\text\{active\}\}; for unassignedc′c^\{\\prime\}\(Ac′​k=0A\_\{c^\{\\prime\}k\}\{=\}0\),pk​\(c′\)=pinactivep\_\{k\}\(c^\{\\prime\}\)=p\_\{\\text\{inactive\}\}\. Under assumption \(ii\) the base gradient expectations𝔼x\|c​\[⋅\]\\mathbb\{E\}\_\{x\|c\}\[\\cdot\]and𝔼x\|c′​\[⋅\]\\mathbb\{E\}\_\{x\|c^\{\\prime\}\}\[\\cdot\]coincide, and the ratio reduces topactive/pinactivep\_\{\\text\{active\}\}/p\_\{\\text\{inactive\}\}\. ∎

The independence assumption in Theorem[1](https://arxiv.org/html/2608.04084#Thmtheorem1)is a first\-order approximation: in practice, the mask realizationmkm\_\{k\}affects the merged output and thus the loss landscape, creating higher\-order dependencies\. The equal\-base\-gradient assumption \(ii\) is also a first\-order initial\-condition idealization: once specialization develops it is violated in the direction that would only widen the predicted ratio\. However, the linear scalingpk​\(c\)/Sp\_\{k\}\(c\)/Sdominates the gradient expectation, and our empirical proxy is consistent with this first\-order prediction: the diagonal\-to\-off\-diagonal pruning\-sensitivity ratio \(a post\-training loss\-recovery proxy\) measured at\(pactive,pinactive\)=\(0\.7,0\.3\)\(p\_\{\\text\{active\}\},p\_\{\\text\{inactive\}\}\)=\(0\.7,0\.3\)is1\.87×1\.87\\times, close to the theoreticalpactive/pinactive=2\.33×p\_\{\\text\{active\}\}/p\_\{\\text\{inactive\}\}=2\.33\\times\(Section[5\.6](https://arxiv.org/html/2608.04084#S5.SS6)\); the small shortfall is attributable to the higher\-order mask\-realization dependencies this first\-order approximation drops together with the proxy gap \(gradient\-flow→\\toend\-of\-training pruning sensitivity is heuristic, not a tight bridge\)\.

### C\.2Proof of Proposition[1](https://arxiv.org/html/2608.04084#Thmproposition1)\(Fixed vs\. Stochastic Denominator\)

###### Proof\.

Part \(a\)\.By linearity of expectation, for any constantα\>0\\alpha\>0we have𝔼m​\[∑kmk​hk/α\]=∑kpk​\(c\)​hk/α\\mathbb\{E\}\_\{m\}\[\\sum\_\{k\}m\_\{k\}h\_\{k\}/\\alpha\]=\\sum\_\{k\}p\_\{k\}\(c\)h\_\{k\}/\\alpha\. Settingα=S\\alpha=Smatches the test\-time forward pass∑kpk​\(c\)​hk/S\\sum\_\{k\}p\_\{k\}\(c\)h\_\{k\}/Sexactly\. Under round\-robin assignment, exactly one module per category haspk=pactivep\_\{k\}=p\_\{\\text\{active\}\}and the otherK−1K\-1havepk=pinactivep\_\{k\}=p\_\{\\text\{inactive\}\}, soS=∑kpk​\(c\)=pactive\+\(K−1\)​pinactiveS=\\sum\_\{k\}p\_\{k\}\(c\)=p\_\{\\text\{active\}\}\+\(K\-1\)p\_\{\\text\{inactive\}\}is the same scalar for every categorycc\. Finally,∑k\(pk​\(c\)/S\)=1\\sum\_\{k\}\(p\_\{k\}\(c\)/S\)=1for allcc, so the merge is a proper convex combination\.

Part \(b\)\.WritingX=∑kmk​hkX=\\sum\_\{k\}m\_\{k\}h\_\{k\}andY=1/∑kmkY=1/\\sum\_\{k\}m\_\{k\}, we have𝔼​\[X​Y\]=𝔼​\[X\]​𝔼​\[Y\]\+Cov​\(X,Y\)\\mathbb\{E\}\[XY\]=\\mathbb\{E\}\[X\]\\mathbb\{E\}\[Y\]\+\\text\{Cov\}\(X,Y\)\.XXandYYare both functions of the same mask draws, soCov​\(X,Y\)≠0\\text\{Cov\}\(X,Y\)\\neq 0for generic\{hk\}\\\{h\_\{k\}\\\}\. A second\-order Taylor expansion of1/N1/NaroundμN=𝔼​\[∑kmk\]=S\\mu\_\{N\}=\\mathbb\{E\}\[\\sum\_\{k\}m\_\{k\}\]=Sgives𝔼​\[1/N\]≈1/S\+Var​\(N\)/S3\\mathbb\{E\}\[1/N\]\\approx 1/S\+\\mathrm\{Var\}\(N\)/S^\{3\}, so the relative magnitude of the Jensen bias scales asVar​\(∑kmk\)/S2\\mathrm\{Var\}\(\\sum\_\{k\}m\_\{k\}\)/S^\{2\}\. ∎

### C\.3Per\-CategorySS\-Invariance \(Extension to Eq\.[5](https://arxiv.org/html/2608.04084#S3.E5)\)

The per\-category schedule of Sec\.[3\.5](https://arxiv.org/html/2608.04084#S3.SS5)generalizes\(pa,pi\)\(p\_\{\\mathrm\{a\}\},p\_\{\\mathrm\{i\}\}\)to\(pac,pic\)\(p\_\{\\mathrm\{a\}\}^\{c\},p\_\{\\mathrm\{i\}\}^\{c\}\)while preservingSc=SS^\{c\}\{=\}Sexactly\.

###### Lemma 1\(Per\-CategorySS\-Invariance\)\.

Letgapc=\(pa−pi\)​\[\(1−πc\)/\(1−1/M\)\]β\\mathrm\{gap\}\_\{c\}\{=\}\(p\_\{\\mathrm\{a\}\}\{\-\}p\_\{\\mathrm\{i\}\}\)\\,\[\(1\{\-\}\\pi\_\{c\}\)/\(1\{\-\}1/M\)\]^\{\\beta\}withβ≥0\\beta\{\\geq\}0,\{πc\}\\\{\\pi\_\{c\}\\\}any probability distribution on𝒞\\mathcal\{C\}, and\(pac,pic\)=\(S/K\+gapc​\(K−1\)/K,S/K−gapc/K\)\(p\_\{\\mathrm\{a\}\}^\{c\},p\_\{\\mathrm\{i\}\}^\{c\}\)\{=\}\(S/K\{\+\}\\mathrm\{gap\}\_\{c\}\(K\{\-\}1\)/K,\\,S/K\{\-\}\\mathrm\{gap\}\_\{c\}/K\)per Eq\.[5](https://arxiv.org/html/2608.04084#S3.E5)\. ThenSc≜pac\+\(K−1\)​pic=SS^\{c\}\\,\\triangleq\\,p\_\{\\mathrm\{a\}\}^\{c\}\+\(K\{\-\}1\)\\,p\_\{\\mathrm\{i\}\}^\{c\}\\,=\\,Sfor everycc,β\\beta, and\{πc\}\\\{\\pi\_\{c\}\\\}\.

###### Proof\.

By direct expansion,Sc=SK\+gapc​K−1K\+\(K−1\)​\[SK−gapcK\]=SK⋅K\+gapc​K−1K−gapc​K−1K=SS^\{c\}=\\tfrac\{S\}\{K\}\+\\mathrm\{gap\}\_\{c\}\\tfrac\{K\{\-\}1\}\{K\}\+\(K\{\-\}1\)\\bigl\[\\tfrac\{S\}\{K\}\-\\tfrac\{\\mathrm\{gap\}\_\{c\}\}\{K\}\\bigr\]=\\tfrac\{S\}\{K\}\\\!\\cdot\\\!K\+\\mathrm\{gap\}\_\{c\}\\tfrac\{K\{\-\}1\}\{K\}\-\\mathrm\{gap\}\_\{c\}\\tfrac\{K\{\-\}1\}\{K\}=S\. ∎

###### Corollary 1\(Prop\.[1](https://arxiv.org/html/2608.04084#Thmproposition1)extends verbatim to per\-category\)\.

Replacingpk​\(c\)p\_\{k\}\(c\)in Eq\.[3](https://arxiv.org/html/2608.04084#S3.E3)with the per\-categorypk​\(c\)=Ac​k​pac\+\(1−Ac​k\)​picp\_\{k\}\(c\)\{=\}A\_\{ck\}p\_\{\\mathrm\{a\}\}^\{c\}\+\(1\{\-\}A\_\{ck\}\)p\_\{\\mathrm\{i\}\}^\{c\}preserves both parts of Prop\.[1](https://arxiv.org/html/2608.04084#Thmproposition1): \(a\)𝔼m​\[∑kmk​hk/S\]=∑kpk​\(c\)​hk/S\\mathbb\{E\}\_\{m\}\[\\sum\_\{k\}m\_\{k\}h\_\{k\}/S\]\{=\}\\sum\_\{k\}p\_\{k\}\(c\)h\_\{k\}/Sremains exact becauseSc=SS^\{c\}\{=\}SmakesSSstill category\-independent and thus valid as a fixed constant; \(b\) the naive\-stochastic denominator still incurs the Jensen bias, now with Bernoulli varianceVarc​\(N\)=pac​\(1−pac\)\+\(K−1\)​pic​\(1−pic\)\\mathrm\{Var\}\_\{c\}\(N\)\{=\}p\_\{\\mathrm\{a\}\}^\{c\}\(1\{\-\}p\_\{\\mathrm\{a\}\}^\{c\}\)\+\(K\{\-\}1\)p\_\{\\mathrm\{i\}\}^\{c\}\(1\{\-\}p\_\{\\mathrm\{i\}\}^\{c\}\)that becomes per\-category\. The gradient concentration ratio of Thm\.[1](https://arxiv.org/html/2608.04084#Thmtheorem1)generalizes toρc=pac/pic\\rho\_\{c\}\{=\}p\_\{\\mathrm\{a\}\}^\{c\}/p\_\{\\mathrm\{i\}\}^\{c\}; the MI bound of Thm\.[2](https://arxiv.org/html/2608.04084#Thmtheorem2)for Soft SpecDrop \(whereZkZ\_\{k\}is deterministic argmax\) is unaffected since the argmax depends only on𝐀\\mathbf\{A\}, not on\(pac,pic\)\(p\_\{\\mathrm\{a\}\}^\{c\},p\_\{\\mathrm\{i\}\}^\{c\}\)\.

### C\.4Proof of Theorem[2](https://arxiv.org/html/2608.04084#Thmtheorem2)\(Routing\-Indicator Mutual Information\)

###### Proof\.

We use the assumptions of the theorem: round\-robin assignment withK\|MK\\,\|\\,M\(so each module is assigned to exactlyM/KM/Kcategories\) and uniformCC\. By definition,I​\(Zk;C\)=H​\(Zk\)−H​\(Zk\|C\)I\(Z\_\{k\};C\)=H\(Z\_\{k\}\)\-H\(Z\_\{k\}\|C\)\. Under these two assumptions, exactly1/K1/Kof categories assignpactivep\_\{\\text\{active\}\}to modulekkand\(K−1\)/K\(K\{\-\}1\)/Kassignpinactivep\_\{\\text\{inactive\}\}, so the conditional entropy factorizes:H​\(Zk\|C\)=1K​Hb​\(pactive\)\+K−1K​Hb​\(pinactive\)H\(Z\_\{k\}\|C\)=\\frac\{1\}\{K\}H\_\{b\}\(p\_\{\\text\{active\}\}\)\+\\frac\{K\-1\}\{K\}H\_\{b\}\(p\_\{\\text\{inactive\}\}\), whereHb​\(p\)=−p​log⁡p−\(1−p\)​log⁡\(1−p\)H\_\{b\}\(p\)=\-p\\log p\-\(1\-p\)\\log\(1\-p\)is the binary entropy\. The marginalP​\(Zk=1\)=p¯P\(Z\_\{k\}=1\)=\\bar\{p\}, soH​\(Zk\)=Hb​\(p¯\)H\(Z\_\{k\}\)=H\_\{b\}\(\\bar\{p\}\)\. ThusI​\(Zk;C\)=Hb​\(p¯\)−1K​Hb​\(pactive\)−K−1K​Hb​\(pinactive\)I\(Z\_\{k\};C\)=H\_\{b\}\(\\bar\{p\}\)\-\\frac\{1\}\{K\}H\_\{b\}\(p\_\{\\text\{active\}\}\)\-\\frac\{K\-1\}\{K\}H\_\{b\}\(p\_\{\\text\{inactive\}\}\)\. Applying the identityHb​\(p¯\)−Hb​\(p\)=DKL​\(p∥p¯\)\+\(p¯−p\)​log⁡1−p¯p¯H\_\{b\}\(\\bar\{p\}\)\-H\_\{b\}\(p\)=D\_\{\\text\{KL\}\}\(p\\\|\\bar\{p\}\)\+\(\\bar\{p\}\-p\)\\log\\frac\{1\-\\bar\{p\}\}\{\\bar\{p\}\}to eachp∈\{pactive,pinactive\}p\\in\\\{p\_\{\\text\{active\}\},p\_\{\\text\{inactive\}\}\\\}and combining,I​\(Zk;C\)=1K​DKL​\(pactive∥p¯\)\+K−1K​DKL​\(pinactive∥p¯\)\+log⁡1−p¯p¯⋅\[1K​\(p¯−pactive\)\+K−1K​\(p¯−pinactive\)\]\.I\(Z\_\{k\};C\)=\\tfrac\{1\}\{K\}D\_\{\\text\{KL\}\}\(p\_\{\\text\{active\}\}\\\|\\bar\{p\}\)\+\\tfrac\{K\-1\}\{K\}D\_\{\\text\{KL\}\}\(p\_\{\\text\{inactive\}\}\\\|\\bar\{p\}\)\+\\log\\\!\\frac\{1\-\\bar\{p\}\}\{\\bar\{p\}\}\\cdot\\\!\\bigl\[\\tfrac\{1\}\{K\}\(\\bar\{p\}\-p\_\{\\text\{active\}\}\)\+\\tfrac\{K\-1\}\{K\}\(\\bar\{p\}\-p\_\{\\text\{inactive\}\}\)\\bigr\]\.The bracketed remainder equalsp¯−\[1K​pactive\+K−1K​pinactive\]=0\\bar\{p\}\-\[\\tfrac\{1\}\{K\}p\_\{\\text\{active\}\}\+\\tfrac\{K\-1\}\{K\}p\_\{\\text\{inactive\}\}\]=0by the definition ofp¯\\bar\{p\}, yielding the*exact*equality \(no convexity bound needed\)\. Whenpactive=pinactivep\_\{\\text\{active\}\}=p\_\{\\text\{inactive\}\},p¯=pactive\\bar\{p\}=p\_\{\\text\{active\}\}and all KL terms vanish\. ∎

## Appendix DExtended Related Work

#### Multi\-branch architectures and task\-specific masks\.

PathNet\[Fernando et al\.,[2017](https://arxiv.org/html/2608.04084#bib.bib14)\]uses evolutionary selection of module pathways—described as “evolutionary dropout”—for continual learning\. Piggyback\[Mallya et al\.,[2018](https://arxiv.org/html/2608.04084#bib.bib27)\]and Supermasks\-in\-Superposition\[Wortsman et al\.,[2020](https://arxiv.org/html/2608.04084#bib.bib44)\]learn task\-specific binary masks over shared weights for task adaptation \(incremental and continual settings respectively\), without separate parameters per task\. MMoE\[Ma et al\.,[2018](https://arxiv.org/html/2608.04084#bib.bib25)\]uses task\-specific learned gates over shared experts for multi\-task learning, forming a natural comparison axis with SpecDrop: MMoE learns*soft per\-task weights*over modules end\-to\-end \(adding trainable parameters per task\), while SpecDrop uses a*fixed binary assignment*with a deterministic soft\-weight readout \(zero extra parameters, no auxiliary losses\)\. All these mask\-based methods learn routing*after*or*during*training for task adaptation, whereas SpecDrop uses a*fixed, predetermined*assignment matrix*during training*to shape what each module learns from scratch\.

#### Theoretical foundations\.

Béna and Goodman \[[2025](https://arxiv.org/html/2608.04084#bib.bib3)\]recently showed that structural modularity alone does not guarantee functional specialization without appropriate constraints—a finding that directly motivates SpecDrop’s explicit specialization pressure through category\-conditioned dropout\. Our theoretical analysis \(Section[4](https://arxiv.org/html/2608.04084#S4)\) builds on this insight, proving that the combination of category conditioning with nonzero cross\-category gradient flow \(pinactive\>0p\_\{\\text\{inactive\}\}\>0\) creates provable specialization guarantees that neither structural modularity nor random dropout achieve independently\. The interaction between dropout and optimization dynamics\[Liu et al\.,[2023](https://arxiv.org/html/2608.04084#bib.bib24)\]further suggests that stochastic masking can either help or hinder depending on timing and magnitude, consistent with our finding that deterministic category conditioning outperforms stochastic variants\.

## Appendix EAdditional Experimental Results

### E\.1CIFAR Pruning\-Sensitivity Heatmap and Routing\-Level MI \(sanity check\)

Section[5\.6](https://arxiv.org/html/2608.04084#S5.SS6)of the main text reports the diag\-argmax and ratio summary statistics for CIFAR pruning sensitivity\. Figure[5](https://arxiv.org/html/2608.04084#A5.F5)below shows the full heatmap\.

![Refer to caption](https://arxiv.org/html/2608.04084v1/x4.png)Figure 5:Pruning sensitivity on CIFAR\-100 \(K=20K\{=\}20\)\.Rows: superclasses; cols: branches; cell darkness∝Δk,c\\propto\\Delta\_\{k,c\}\(Sec\.[5\.6](https://arxiv.org/html/2608.04084#S5.SS6)\); blue outlines \(left panel\) mark each superclass’s round\-robin assigned branch\.*Left*: Soft SpecDrop \(diag\-argmax13/2013/20, ratio1\.87×1\.87\\times\)\.*Right*: No\-Routing \(diag\-argmax1/201/20, ratio0\.94×0\.94\\times\)\. Same backbone — the structural difference is attributable to routing alone\.For completeness we also report the categorical argmax\-branch quantityI​\(Zargmax;C\)I\(Z\_\{\\textup\{argmax\}\};C\)whereZargmax=arg⁡maxk⁡pk​\(C\)∈\{1,…,K\}Z\_\{\\textup\{argmax\}\}\{=\}\\arg\\max\_\{k\}p\_\{k\}\(C\)\\in\\\{1,\\ldots,K\\\}: for Soft SpecDrop this is the branch with the largestpk​\(c\)p\_\{k\}\(c\), and for No\-Routing it is undefined by symmetry \(reported as the uniform argmax, which collapses to a single branch\)\. The Soft SpecDrop routing schedule is a deterministic one\-to\-one mapc↦k​\(c\)c\\mapsto k\(c\)atK=20K\{=\}20, soI​\(Zargmax;C\)=log⁡KI\(Z\_\{\\textup\{argmax\}\};C\)=\\log Kis reached by construction; No\-Routing’s uniform weights giveI​\(Zargmax;C\)=0I\(Z\_\{\\textup\{argmax\}\};C\)=0by symmetry\. Table[8](https://arxiv.org/html/2608.04084#A5.T8)confirms both values empirically to machine precision\. This contrast is a sanity check on the routing layer and does*not*by itself imply that the branch*parameters*have specialized — that is what the pruning\-sensitivity test of Sec\.[5\.6](https://arxiv.org/html/2608.04084#S5.SS6)measures\.

Table 8:Routing\-level mutual informationon CIFAR\-100 \(MultiBranchResNet110,K=20K\{=\}20, 200 epochs, 3 seeds 42/123/456\)\. Exact by construction — no seed variance\.
### E\.2CIFAR Fine\-Label Oracle \(K=20K\{=\}20with fine\-label routing\)

To bound the upper endpoint of the partition\-alignment spectrum \(cf\. Sec\.[6](https://arxiv.org/html/2608.04084#S6)\), we run Soft SpecDrop withroute\_label\_type=fine\\texttt\{route\\\_label\\\_type\}\{=\}\\texttt\{fine\}on CIFAR\-100:M=100M\{=\}100fine labels,K=20K\{=\}20branches, round\-robin so each branch covers55fine labels\. This is*deliberately leaky*: whenfine\_label=cluster\_id\\texttt\{fine\\\_label\}\{=\}\\texttt\{cluster\\\_id\}is known at routing time, the mechanism receives a11\-of\-100100oracle at every routing site\.

#### This is not a deployable comparison\.

The\+14\.12\+14\.12top\-1 lift reflects*oracle\-routing*: at training and inference time the routing key equals the answer\. We report it as the empirical upper\-bound endpoint of the partition\-alignment spectrum — when partition→\\toanswer, accuracy approaches the oracle’s per\-class capacity \(∼93%\\sim 93\\%on CIFAR\-100\)\. This complements the four other settings \(ViT BREEDS46/4646/46aligned\+6\.53\+6\.53; CIFAR\-100 superclass13/2013/20partial\+4\.75\+4\.75; NLP SlimPajama6/76/7fuzzy\+0\.10\+0\.10PPL; LoRA SuperNI0/150/15anti\-aligned tied\) to characterize the binding condition: the mechanism’s contribution is directionally consistent with alignment quality across these five points\. We do*not*claim this as evidence of method efficacy in any deployable sense; it is a conceptual anchor for the alignment thesis \(Sec\.[6](https://arxiv.org/html/2608.04084#S6)\)\.

### E\.3Information\-Matched Logit\-Masking Control

The scope statement of Sec\.[6](https://arxiv.org/html/2608.04084#S6)rests on an information\-matched control: at inference, restrict each model’s logits to the fine classes of the given category \(superclass on CIFAR\-100, BREEDS supercategory on ImageNet\) and renormalize\. This gives every method, including those trained without the label, the identical inference\-time information SpecDrop consumes\. All rows are 3\-seed means \(42/123/456\), evaluated on the full test/validation splits \(10,00010\{,\}000and50,00050\{,\}000images\)\.

Table 9:Logit\-masking control\(3 seeds\)\.*Masked*restricts output logits to the given category’s fine classes at inference; unmasked entries use the checkpoints of the provenance note below and may differ slightly from Tabs\.[1](https://arxiv.org/html/2608.04084#S5.T1)–[2](https://arxiv.org/html/2608.04084#S5.T2)\(by up to0\.150\.15\)\. Given the same label, masking the dense model is the strongest deployment for accuracy alone on both datasets; masking changes SpecDrop by exactly0\.000\.00on CIFAR and\+1\.06\+1\.06on ImageNet, showing the output\-space restriction is largely internalized during training\. On ImageNet the masked No\-Routing\+\+SE control \(81\.4481\.44\) also exceeds masked ours \(80\.9580\.95\); on CIFAR this reverses \(78\.4778\.47vs79\.2379\.23\)\.#### Internalization statistic\.

The CIFAR\+0\.00\+0\.00is exact at every seed: across three seeds \(30,00030\{,\}000test predictions\), SpecDrop predicts outside the given superclass once \(0/1/00/1/0per seed\), versus16\.3%16\.3\\%of the time for the dense model \(16\.43/16\.32/16\.01%16\.43/16\.32/16\.01\\%per seed\), so the mask has nothing left to remove\.

#### Checkpoint provenance\.

Two seed\-42 checkpoints were unavailable and retrained from the stored configs before evaluation: the CIFAR dense s42 retrain reaches73\.9773\.97\(original74\.4174\.41; cross\-torch\-version drift\) and the ImageNet dense s42 retrain reaches76\.2376\.23unmasked \(original76\.4576\.45at 2 seeds; the retrained seed’s masked gain,\+7\.28\+7\.28, matches the other seeds\)\. Table entries use the retrained checkpoints; no conclusion depends on the drift\.

### E\.4Label\-Quality Curve and Predicted\-Label Operating Points

The break\-even thresholds of Sec\.[6](https://arxiv.org/html/2608.04084#S6)read off a label\-quality→\\toperformance curve: CIFAR\-100 top\-1 as a function of category\-label accuracy under symmetric label corruption \(eval\-only, 3 seeds\), together with realistic predicted\-label operating points \(the dense baseline’s own fine\-class prediction coarsened through the hierarchy; no extra training\)\.

Table 10:Label\-quality curve\(CIFAR\-100, 3 seeds, eval\-only\)\. Label accuracy under symmetric corruption at rateppis1−1920​p1\-\\tfrac\{19\}\{20\}p\(5%5\\%atp=1p\{=\}1, chance\)\. The*predicted*column is a realistic operating point using the dense model’s coarsened fine\-class prediction as the label\. SpecDrop stays above the architecture\-matched No\-Routing control \(63\.0863\.08\) down to≈80%\{\\approx\}80\\%label accuracy and above dense \(74\.4874\.48\) down to≈92\{\\approx\}92–94%94\\%\.#### ImageNet operating point\.

The corresponding predictor on ImageNet \(the dense ViT’s fine\-class prediction coarsened through the BREEDS hierarchy\) is88\.5%88\.5\\%accurate and gives74\.25±0\.1074\.25\\pm 0\.10, retaining\+0\.9\+0\.9over the matched\-SE No\-Routing control \(73\.3673\.36\) under fully predicted labels; ImageNet’s supercategories average2222fine classes, so the coarsened predictor clears the≈80%\{\\approx\}80\\%bar comfortably \(on CIFAR the83\.8%83\.8\\%predictor retains\+6\.2\+6\.2over the matched control,69\.369\.3vs63\.0863\.08\)\. On CIFAR’s finer55\-classes\-per\-category partition, predictors at the dense model’s own capacity measure only∼84%\{\\sim\}84\\%\(coarsened83\.8%83\.8\\%; a coarse head fine\-tuned from the dense checkpoint reaches∼84\.5%\{\\sim\}84\.5\\%\), so no capacity\-comparable predictor reaches the≈92\{\\approx\}92–94%94\\%dense break\-even there; the applicability condition of Sec\.[6](https://arxiv.org/html/2608.04084#S6)is load\-bearing on such partitions\.

#### Deployment cost\.

Both vision predictors share the same construction: the dense baseline’s own fine\-class prediction, coarsened through the respective hierarchy \(the CIFAR\-100 superclass tree; the BREEDS supercategory partition\), with no extra training\. Where the category tag does not arrive for free \(in our NLP and LoRA settings it does, as domain tags and task clusters\), predicting it adds a second dense\-scale forward pass —≈2×\{\\approx\}2\\timesthe single\-model inference cost of App\.[F\.3](https://arxiv.org/html/2608.04084#A6.SS3)— on CIFAR\-100 and ImageNet alike\.

### E\.5NLP Per\-Domain Reliance Decomposition

#### Per\-domain reliance decomposition \(source\-domain labels,β=0\\beta\{=\}0/scalar, Phase A,K=7K\{=\}7, 100M\)\.

Each domain’s branch sensitivity is decomposed into “own\-branch”Δk​\(d\),d\\Delta\_\{k\(d\),d\}versus “other\-branches”∑k≠k​\(d\)Δk,d\\sum\_\{k\{\\neq\}k\(d\)\}\\Delta\_\{k,d\}\. The resulting own\-reliance ratio anti\-correlates with data volume:

Large web\-scrape domains \(CommonCrawl, C4\) behave as*generalists*that the network’s whole branch pool helps represent; smaller, lexically narrower domains \(ArXiv, Wikipedia\) behave as*specialists*whose assigned branch is nearly solely responsible for their predictions; Github and StackExchange sit between\. The asymmetry is orthogonal to the PPL–specialization decoupling: the model supports a generalist/specialist split internally without translating it to an aggregate PPL gain over the matched scalar baseline\.

### E\.6NLP Mini\-Ablation \(full\)

Section[5\.5](https://arxiv.org/html/2608.04084#S5.SS5)of the main text identifies the final NLP operating point via a three\-phase sequential search at 100M tokens \(step\-wise warmup throughout,K=7K\{=\}7uniform branches, 3 seeds\)\. The complete per\-cell tables are below\.

#### Phase 3a:pap\_\{\\mathrm\{a\}\}sweep at SE anchors \(step warmup,β=1\\beta\{=\}1\)\.

The SE=0\.5 column reports only the degeneratepa=0\.5p\_\{\\mathrm\{a\}\}\{=\}0\.5cell because it serves as the*matched\-SE scalar baseline*against the final ours operating point\.

#### Phase 3b:β\\betasweep at\(pa,X\)=\(0\.6,1\)\(p\_\{\\mathrm\{a\}\},X\)\{=\}\(0\.6,1\)\.

Spread acrossβ\\betais0\.040\.04PPL, all within oneσmean\\sigma\_\{\\text\{mean\}\};β=0\\beta\{=\}0\(scalar\-per\-cat reference\) is the*highest*mean, weak but direction\-consistent evidence that per\-category differentiation helps at 100M\.

#### Phase 3c:XX\(SE\-ratio\) sweep at\(pa,β\)=\(0\.6,4\)\(p\_\{\\mathrm\{a\}\},\\beta\)\{=\}\(0\.6,4\)\.

Clean U\-curve with minimum atX=0\.5X\{=\}0\.5, the configuration reported in Table[3](https://arxiv.org/html/2608.04084#S5.T3)\.

#### 100M matched\-SE tie, seed\-paired\.

Ours at the final operating point gives55\.01±0\.1055\.01\\pm 0\.10\(s=42: 55\.15, s=123: 54\.98, s=456: 54\.90\); the matched\-SE scalar gives55\.01±0\.1455\.01\\pm 0\.14\(s=42: 55\.20, s=123: 54\.93, s=456: 54\.90\)\. Seed\-paired differences\(−0\.05,\+0\.05,0\.00\)\(\-0\.05,\+0\.05,0\.00\)sum to zero;z=0\.00​σz\{=\}0\.00\\sigma, exactly tied\. The\+0\.10\+0\.10PPL gap that emerges at 500M \(Table[3](https://arxiv.org/html/2608.04084#S5.T3)\) is a longer\-horizon effect, not a 100M effect\.

#### Warmup\-granularity: step vs epoch at 100M\.

A side comparison at\(pa,β,X\)=\(0\.6,2,0\.5\)\(p\_\{\\mathrm\{a\}\},\\beta,X\)\{=\}\(0\.6,2,0\.5\)compares per\-step vs per\-epochpap\_\{\\mathrm\{a\}\}warmup at 100M\. Mean PPL differs by−0\.03\-0\.03\(step wins, within noise\); 3\-seedσ\\sigmadrops from0\.150\.15\(epoch\) to0\.080\.08\(step\), a2×2\\timesreduction that becomes4×4\\timesat 500M \(Sec\.[5\.3](https://arxiv.org/html/2608.04084#S5.SS3)\)\. At the degeneratepa=0\.5p\_\{\\mathrm\{a\}\}\{=\}0\.5point, step and epoch are mathematically identical; we verify numerically that the residual drift between archived \(epoch\) and new \(step\) 3a runs is at most0\.090\.09PPL across thepa≤0\.9p\_\{\\mathrm\{a\}\}\{\\leq\}0\.9cells, within theσmean≤0\.12\\sigma\_\{\\text\{mean\}\}\{\\leq\}0\.12envelope and attributable to CUDA non\-determinism \(the hard\-routingpa=1\.0p\_\{\\mathrm\{a\}\}\{=\}1\.0cell, farthest from the degenerate point, drifts\+0\.59\+0\.59\)\.

### E\.7ViT ImageNet Mini\-Ablation

The ViT operating point of Sec\.[5\.2](https://arxiv.org/html/2608.04084#S5.SS2)is identified by a three\-phase sequential search on a20%20\\%stratified ImageNet subset \(per\-class\), validated on the full5050K validation split,K=46K\{=\}46uniform branches,wr=1\.0w\_\{r\}\{=\}1\.0cosine per\-epoch warmup,33seeds\.

#### Phase 5a:pap\_\{\\mathrm\{a\}\}sweep atX=1X\{=\}1,β=1\\beta\{=\}1\.

pa=0\.5p\_\{\\mathrm\{a\}\}\{=\}0\.5excluded*a priori*\(algebraic mechanism\-OFF\)\. Strict argmax atpa=0\.6p\_\{\\mathrm\{a\}\}\{=\}0\.6\.

#### Phase 5b:β\\betasweep atpa=0\.6p\_\{\\mathrm\{a\}\}\{=\}0\.6,X=1X\{=\}1\.

β\\betais a flat lever \(spread0\.230\.23within seed noise\),β=1\\beta\{=\}1strict argmax\.

#### Phase 5c:XX\(SE\-ratio\) sweep at\(pa,β\)=\(0\.6,1\)\(p\_\{\\mathrm\{a\}\},\\beta\)\{=\}\(0\.6,1\)\.

XXis the only ViT axis with significant signal:X=2\.0X\{=\}2\.0provides\+0\.94\+0\.94overX=1\.0X\{=\}1\.0, indicating BREEDS’ moderate semantic dispersion benefits from a2×2\\times\-capacity shared\-expert\. The full\-data ViT main\-table number \(79\.8979\.89, Table[2](https://arxiv.org/html/2608.04084#S5.T2)\) at\(pa,β,X\)=\(0\.6,1,2\.0\)\(p\_\{\\mathrm\{a\}\},\\beta,X\)\{=\}\(0\.6,1,2\.0\)confirms the20%20\\%\-subset selection transfers to full ImageNet\-1K\.

#### Mini\-ablation decomposition \(mechanism necessity,β\\beta\-plateau, SE\-monotonic\)\.

Phases 5a/5b/5c jointly decompose the contribution of each axis on the balanced BREEDS partition\. Starting from\(pa,β,X\)=\(0\.5,1,1\)\(p\_\{\\mathrm\{a\}\},\\beta,X\)\{=\}\(0\.5,1,1\)\(mechanism\-OFF,42\.9142\.91top\-1\), three findings emerge\.*\(i\) Mechanism necessity*:pa=0\.5→0\.6p\_\{\\mathrm\{a\}\}\{=\}0\.5\\to 0\.6is\+12\.61\\mathbf\{\+12\.61\}top\-1 \(42\.91→55\.5242\.91\\to 55\.52\), a discrete jump from mech\-OFF to mech\-ON with architecture and SE held constant\. This is the cleanest single\-table evidence that the routing mechanism itself contributes, not just theK=46K\{=\}46multi\-branch architecture — on ViT BREEDS the mechanism is responsible for the bulk of the gain over arch\-matched baselines \(cf\. Tab\.[2](https://arxiv.org/html/2608.04084#S5.T2)headline\+6\.53\+6\.53over matched\-SE at full ImageNet\)\.*\(ii\)β\\beta\-plateau*:β∈\{0,1,2,4\}\\beta\{\\in\}\\\{0,1,2,4\\\}all within0\.230\.23top\-1 \(statistically tied\), consistent with BREEDS being a balanced partition \(each supercategory roughly equal weight\) where the per\-category amplification term has no leverage;β\\betamatters only when the input distribution is meaningfully non\-uniform \(NLP/LoRA, whereβ=4\\beta\{=\}4/11are selected\)\.*\(iii\) SE\-monotonic*:X∈\{0,0\.5,1,2\}X\{\\in\}\\\{0,0\.5,1,2\\\}is monotonic,\+3\.06\\mathbf\{\+3\.06\}fromX=0X\{=\}0toX=2X\{=\}2, a secondary boost\. Combined: on the balanced ViT BREEDS partition, the routing mechanism contributes the bulk of the gain, SE provides a secondary monotonic boost, and per\-category amplification is statistically inactive — consistent with its motivation as an imbalance\-correction term\.

#### ImageNet\-BREEDS label\-leak disclosure\.

The BREEDS supercategory is derived from each image’s fine\-grained ImageNet\-1K label\[Santurkar et al\.,[2021](https://arxiv.org/html/2608.04084#bib.bib31)\]\. Our routing observes a label\-derived signal that the learned\-routing baselines \(Mod\-Squad, Soft MoE, COMET\) do not\. We acknowledge this asymmetry; the matched\-SE No\-Routing baseline \(uniform1/K1/K\+ SE on the same K=46 branches with the same supercategory pipeline\) provides a controlled mechanism\-OFF reference under the same supervision and is the appropriate isolated comparison for the routing\-mechanism contribution\.

#### 20%→100%20\\%\{\\to\}100\\%subset transfer\.

Sec\.[5\.2](https://arxiv.org/html/2608.04084#S5.SS2)reports the final operating point\(pa,β,X\)=\(0\.6,1,2\.0\)\(p\_\{\\mathrm\{a\}\},\\beta,X\)\{=\}\(0\.6,1,2\.0\)at full ImageNet\-1K, identified on the20%20\\%stratified subset\. We acknowledge that per\-subset argmax may differ from full\-data argmax \(NLP precedent: 100M argmin≠\\neq500M argmin in Phase 3c, where the metric is PPL\); the full\-data result for ours79\.89±0\.1879\.89\\pm 0\.18is reported\.

#### DeiT short\-recipe disclosure\.

We train all ViT methods with a shortened DeiT recipe: 100 epochs, no model EMA, no RepeatedAugmentation\. Full DeiT 300\-ep \+ EMA \+ RA reaches∼79\.85%\\sim 79\.85\\%for ViT\-S/16 \(timm reference\); our short\-recipe dense ViT\-Small reaches76\.38%76\.38\\%\. All methods share this recipe identically, so relative orderings are preserved; the full DeiT 300\-ep\+EMA\+RA recipe is out of scope for this work\.

#### Mod\-Squad FFN\-only adaptation\.

The original Mod\-Squad\[Chen et al\.,[2023b](https://arxiv.org/html/2608.04084#bib.bib5)\]targets multi\-task vision \(Taskonomy, PASCAL\-Context\) with MoE applied to both attention and FFN\. We adapt it to single\-task ImageNet by treating the BREEDS\-46 supercategories as the “tasks” for the mutual\-information loss, and we restrict MoE to FFN blocks only; the original paper’s ablation suggests this FFN\-only restriction underestimates Mod\-Squad by∼0\.3\\sim 0\.3–11top\-1\. We accept this asymmetry rather than re\-tune\.

#### Soft MoE tuning study \(placement, learning rate, granularity, compute\)\.

The deployed all\-blocks Soft MoE configuration reaches63\.06±0\.2263\.06\\pm 0\.22at full protocol; a dedicated tuning study shows this number is a placement artifact, and Table[2](https://arxiv.org/html/2608.04084#S5.T2)accordingly reports the tuned variant as the primary Soft MoE row\. On the same20%20\\%stratified selection subset used for our own operating\-point search above, we sweep eight Soft MoE variants plus the ALF router and a capacity\-identical Mod\-Squad pair, every row at 3 seeds:

One change dominates: the paper\-canonical second\-half placement is worth\+14\.1\+14\.1on the subset by itself \(51\.4151\.41vs37\.2737\.27\), the learning rate adds\+1\.2\+1\.2on top \(52\.6352\.63vs51\.4151\.41\), and no other single change moves the number by more than\+2\.1\+2\.1; the lr1×10−41\{\\times\}10^\{\-4\}and compute\-matched variants hurt \(−6\.0\-6\.0and−1\.7\-1\.7\)\. \(The reference row’s±0\.32\\pm 0\.32is a sample standard deviation; the same three runs appear as56\.46±0\.2656\.46\\pm 0\.26in Phase 5c above under its population convention\.\) Carried to the full protocol \(identical to Tab\.[2](https://arxiv.org/html/2608.04084#S5.T2): full data,100100epochs, 3 seeds\), the tuned combination reaches76\.69±0\.7076\.69\\pm 0\.70\(per\-seed75\.92/77\.30/76\.8675\.92/77\.30/76\.86; the seed\-456 configuration was accidentally scheduled twice, and we report the run whose checkpoint and results artifacts are retained,76\.8676\.86— the displaced duplicate’s log records76\.7376\.73, a0\.130\.13same\-seed replication gap\), above dense; the compute\-matched variant lands at66\.72±0\.6366\.72\\pm 0\.63, below the bare No\-Routing control while consuming dense\-level compute, so the binding constraint of the deployed configuration was placement, not compute; and the ALF router reaches71\.09±0\.3271\.09\\pm 0\.32vs its capacity\-identical Mod\-Squad pair’s70\.1170\.11\(\+0\.98\+0\.98at full protocol,\+2\.23\+2\.23on the subset, consistent in direction\), so the bias\-corrected balancing helps, while both remain below the73\.3673\.36matched\-SE control\. The sweep gave Soft MoE a placement and learning\-rate search that no other method in Table[2](https://arxiv.org/html/2608.04084#S5.T2), including ours, received\.

#### Compute\-matched and ALF configurations\.

The compute\-matched Soft MoE widens experts from4848to16001600hidden dimensions at55slots per expert, with parameters unconstrained \(481481M\), restoring dense\-level per\-image compute \(4\.264\.26GMACs vs dense4\.254\.25; the deployed all\-blocks variant runs at1\.561\.56GMACs,37%37\\%of dense\)\. The ALF router followsWang et al\. \[[2024](https://arxiv.org/html/2608.04084#bib.bib40)\]: top\-22selection overN=16N\{=\}16experts of hidden9696\(capacity\-identical to our Mod\-Squad configuration\) with bias\-corrected, auxiliary\-loss\-free load balancing \(bias update rate0\.0010\.001\)\. Per\-method MACs for all rows are in App\.[F\.3](https://arxiv.org/html/2608.04084#A6.SS3)\.

#### Training curves \(deployed vs tuned vs ours\)\.

Figure[6](https://arxiv.org/html/2608.04084#A5.F6)plots test top\-1 across training for the deployed Soft MoE, the tuned Soft MoE, and Soft SpecDrop \(3\-seed means from per\-epoch histories\)\. All three converge healthily under the shared100100\-epoch budget that every Table[2](https://arxiv.org/html/2608.04084#S5.T2)method received\. The tuned baseline dominates the deployed one at every epoch, consistent with the placement finding; it also leads Soft SpecDrop through epoch2525before Soft SpecDrop overtakes it by epoch5050, and the ordering is stable across the final quarter of training \(epoch\-7575gap3\.93\.9vs final3\.23\.2\)\.

![Refer to caption](https://arxiv.org/html/2608.04084v1/x5.png)Figure 6:ImageNet BREEDS\-46 test top\-1 across trainingfor the deployed all\-blocks Soft MoE, the tuned Soft MoE \(canonical second\-half placement\+\+lr5×10−45\{\\times\}10^\{\-4\}\), and Soft SpecDrop\. Lines are 3\-seed means; shaded bands span seed min/max\. Milestones \(3\-seed means\) at epochs10/25/50/75/10010/25/50/75/100: deployed26\.3/41\.8/52\.9/60\.0/63\.026\.3/41\.8/52\.9/60\.0/63\.0, tuned42\.1/57\.5/66\.8/73\.6/76\.742\.1/57\.5/66\.8/73\.6/76\.7, ours33\.3/56\.1/72\.4/77\.5/79\.933\.3/56\.1/72\.4/77\.5/79\.9\.

### E\.8LoRA SuperNI Mini\-Ablation

The LoRA operating point of Sec\.[5\.4](https://arxiv.org/html/2608.04084#S5.SS4)is identified by a three\-phase sequential search on a20%20\\%stratified subset of SuperNI training tasks,K=20K\{=\}20uniform branches,wr=1\.0w\_\{r\}\{=\}1\.0cosine per\-step warmup,33seeds, ROUGE\-L F1 selection metric end\-to\-end\.

#### Phase 8a:pap\_\{\\mathrm\{a\}\}sweep atX=1X\{=\}1,β=1\\beta\{=\}1\.

pa∈\{0\.5,0\.6,0\.7,0\.8,0\.9,1\.0\}p\_\{\\mathrm\{a\}\}\{\\in\}\\\{0\.5,0\.6,0\.7,0\.8,0\.9,1\.0\\\}all within0\.00300\.0030ROUGE\-L;pa=0\.5p\_\{\\mathrm\{a\}\}\{=\}0\.5\(mechanism\-OFF\) andpa=1\.0p\_\{\\mathrm\{a\}\}\{=\}1\.0\(hard routing\) excluded*a priori*; strict non\-degenerate argmax atpa=0\.8p\_\{\\mathrm\{a\}\}\{=\}0\.8\(0\.4784±0\.00900\.4784\\pm 0\.0090\)\. The flatpap\_\{\\mathrm\{a\}\}\-curve at full\-data fine\-tune is itself diagnostic: the routing axis is muted in this regime, consistent with the LoRA decomposition \(Sec\.[5\.4](https://arxiv.org/html/2608.04084#S5.SS4.SSS0.Px2)\) attributing only∼3%\\sim 3\\%of our lift to routing\.

#### Phase 8b:β\\betasweep at\(pa,X\)=\(0\.8,1\)\(p\_\{\\mathrm\{a\}\},X\)\{=\}\(0\.8,1\)\.

β∈\{0,1,2,4\}\\beta\{\\in\}\\\{0,1,2,4\\\}all within0\.00260\.0026ROUGE\-L;β=1\\beta\{=\}1strict argmax \(no per\-category amplification benefit\)\.

#### Phase 8c:XXsweep at\(pa,β\)=\(0\.8,1\)\(p\_\{\\mathrm\{a\}\},\\beta\)\{=\}\(0\.8,1\)\.

The first axis with real signal in LoRA mini\-ablation \(∼1​σ\{\\sim\}1\\sigmabetweenX=0X\{=\}0andX=1X\{=\}1\)\. Strict argmax atX=1\.0X\{=\}1\.0, consistent with full\-data Table[4](https://arxiv.org/html/2608.04084#S5.T4)\.

#### ROUGE\-L F1 selection metric\.

We use ROUGE\-L F1 per[Wang et al\.](https://arxiv.org/html/2608.04084#bib.bib42)’s Tk\-Instruct canonical end\-to\-end: best\.pt selected by argmax ROUGE\-L over training epochs \(per\-epoch generation eval\); BESTpa/β/Xp\_\{\\mathrm\{a\}\}/\\beta/Xall argmax on 3\-seed mean ROUGE\-L; main\-table reporting on argmax\-ROUGE\-L checkpoint\. A single metric across \(1\) checkpoint selection, \(2\) hyperparameter selection, and \(3\) baseline comparison ensures consistency\.

#### cluster\-id\-at\-inference requirement\.

SpecDrop\-LoRA reads the cluster ID from each batch at inference; for SuperNI held\-out tasks every task carries an official[Wang et al\.](https://arxiv.org/html/2608.04084#bib.bib42)Domain label, so this is unambiguous\. Table[4](https://arxiv.org/html/2608.04084#S5.T4)reports only tasks with known cluster assignments\.

#### 225225M /18%18\\%trainable budget justification\.

Our LoRA configuration \(K=20K\{=\}20branches attached to all77linear projections in each transformer block; rankr=16r\{=\}16for the no\-SE variant,r=15r\{=\}15plus a rank\-1515shared expert as deployed\) yields≈225\\approx 225M trainable parameters,≈18%\\approx 18\\%of the Llama\-3\.2\-1B base\. This budget is substantially larger than single\-LoRA defaults \(typically<<1% atr=8r\{=\}8\) but comparable to recent multi\-branch PEFT methods: LoRAMoE\[Dou et al\.,[2024](https://arxiv.org/html/2608.04084#bib.bib10)\]atK=6×r=92K\{=\}6\{\\times\}r\{=\}92uses7%7\\%of Llama2\-7B; MoCLE\[Gou et al\.,[2024](https://arxiv.org/html/2608.04084#bib.bib15)\]atE=4\+1×r=63E\{=\}4\{\+\}1\{\\times\}r\{=\}63uses∼5%\\sim 5\\%\. All66multi\-branch baselines are strictly budget\-matched to225225M \(±3%\\pm 3\\%\)\. We do not call this “PEFT” in the abstract or intro; the regime is non\-typical, and method comparisons remain fair within it\.

#### LoRAMoE rank\-extrapolation disclosure\.

LoRAMoE’s native rank range\[Dou et al\.,[2024](https://arxiv.org/html/2608.04084#bib.bib10)\]isr∈\{4,8,16\}r\{\\in\}\\\{4,8,16\\\}; our budget\-matching constraint forcesr=76r\{=\}76on Llama\-3\.2\-1B\. This is well outside the original paper’s tested range and may dilute Dou et al\. 2024’s load\-balance loss, which was tuned for lower\-rank regimes\. A sanity comparison at nativer=8r\{=\}8\(outside our225225M budget\) is deferred\.

#### HydraLoRA rank\-extrapolation disclosure\.

HydraLoRA’s main result\[Tian et al\.,[2024](https://arxiv.org/html/2608.04084#bib.bib37)\]\(Tian et al\. 2024 Table 2, LLaMA\-2\-7B\) usesN=3N\{=\}3B\-heads at rankr=8r\{=\}8, withN=4N\{=\}4reported as empirically optimal in their ablation \(Section 4\.5 and Figure 8,N∈\{1,…,5\}N\\in\\\{1,\\ldots,5\\\}\)\. We useN=8N\{=\}8at rankr=67r\{=\}67on Llama\-3\.2\-1B to match our225225M trainable\-parameter budget; rank\-matched atr=8r\{=\}8would yield only∼27\\sim 27M trainable \(∼12%\\sim 12\\%of our budget\) and starve HydraLoRA’s capacity\. We do not perform Tian’skk\-means warm\-start initialization \(we use standard zero\-init gate, Kaiming𝐀\\mathbf\{A\}, zero𝐁\\mathbf\{B\}\);N=8N\{=\}8, the larger rank, and the no\-warm\-start choice are our deviations from the canonical setup\.

#### LoRAMoE site\-coverage confound\.

Dou et al\. 2024 attaches LoRAMoE to FFN sites only \(33linears per block\); our LoRAMoE adaptation, like ours/HydraLoRA/MoCLE, attaches to all77linears for capacity\-matched comparison\. We attempted a77\-site LoRAMoE variant matching their rank but it OOM’d atbatch==8 on 32GB; reproducing atbatch=4\\text\{batch\}=4withaccum=32\\text\{accum\}=32was deferred under time constraint\. The single\-baseline coverage difference is acknowledged\.

#### LoRA77\-linear attachment \+ GQA \+ MoCLE adaptations\.

For fair comparison, all66multi\-branch LoRA methods attach adapters to all77linear projections \(q/k/v/o \+ gate/up/down\) per transformer block, regardless of original\-paper defaults \(Hu 2022’s single LoRA typically attaches to q/v only; QLoRA\-recommended default\)\. Llama\-3\.2\-1B uses Grouped\-Query Attention: q/o\_proj have dim20482048, but k/v\_proj have dim512512\(88KV heads×\\times6464head\_dim\); a rank\-rrLoRA represents0\.6%0\.6\\%on q\_proj but2\.5%2\.5\\%on v\_proj\. This does not affect param budgeting \(all methods share base dimensions\), but we report it for completeness\. MoCLE’s originalK=64K\{=\}64gate\-cluster classes are adapted toK=20K\{=\}20to match our cluster partition \(App\.[B\.4](https://arxiv.org/html/2608.04084#A2.SS4)\), with the55experts \(E=4\+1E\{=\}4\{\+\}1universal\) accessed by all2020cluster IDs through a dense2020\-way softmax gate \(consistent with Gou 2024’s own ablation showingK\>MK\{\>\}Mimproves only marginally\)\.

### E\.9LoRA Aggregated Per\-Method Results \(3 seeds\)

Aggregated ROUGE\-L F1 and Exact\-match \(mean±\\pmstd over seeds4242/123123/456456\) for the SuperNI main\-table comparison \(Sec\.[5\.4](https://arxiv.org/html/2608.04084#S5.SS4), Table[4](https://arxiv.org/html/2608.04084#S5.T4)\); per\-seed JSON artifacts are released alongside the code\.

### E\.10NLP Scaling Check \(125125M Transformer\)

We replicate the 30M training regime at125125M scale \(GPT\-2\-small architecture:1212layers, hidden768768,1212heads;500500M unique SlimPajama tokens×\\times1010epochs; AdamWlr​3×10−4\\text\{lr\}\\,3\{\\times\}10^\{\-4\}cosine, bf16, batch16×51216\\times 512\(halved from3030M batch32×51232\\times 512due to memory\),33seeds42/123/45642/123/456\) to test whether the3030M ours\-vs\-matched\-SE\-scalar tie persists across∼\\sim4×4\\timesscale\. We chose this regime over a Chinchilla\-optimal11\-epoch run at125125M to preserve internal cross\-scale comparability; the only varying factor between3030M and125125M is model size\.

#### Reading\.

At125125M, ours and the matched\-SE scalar both drop∼25%\\sim 25\\%absolute PPL relative to3030M, confirming both benefit from scale\. The conditional\-tie direction at3030M \(Δ=\+0\.10\\Delta\{=\}\{\+\}0\.10on33seeds\) is preserved at125125M \(Δ=\+0\.17\\Delta\{=\}\{\+\}0\.17on33seeds, same sign in3/33/3paired seeds\), consistent with the granularity\-alignment thesis predicting no differential routing gain on this fuzzy partition at either scale\.

### E\.11NLP11\-Epoch Regime Sanity \(3030M\)

To verify that the1010\-epoch multi\-epoch regime \(App\.[B\.2](https://arxiv.org/html/2608.04084#A2.SS2.SSS0.Px8)\) does not bias the cross\-method orderings of Tab\.[3](https://arxiv.org/html/2608.04084#S5.T3), we rerun all88Tab\.[3](https://arxiv.org/html/2608.04084#S5.T3)methods at3030M×\\times11epoch on the same500500M\-unique SlimPajama cache \(≈17\\approx 17tokens per parameter, Chinchilla\-near\-optimal\),33seeds each \(2424cells\), regenerating identically to the main configuration except for epoch count\.

#### Reading \(regime\-invariant tie\)\.

The headline ours\-vs\-matched\-SE\-scalarΔ\\Deltashrinks from\+0\.10\+0\.10PPL \(1010\-ep,1\.64​σ1\.64\\sigma,3/33/3seeds lose\) to−0\.02\\mathbf\{\-0\.02\}PPL \(11\-ep,0\.13​σ0\.13\\sigma,2/32/3seeds win\) — sign\-flipped but*still tied within seed noise*\. Cross\-method orderings are preserved within seed noise: ours and matched\-SE swap positions2↔32\\leftrightarrow 3\(within0\.020\.02PPL at11\-ep\), and Switch and No\-Routing\-no\-SE swap positions4↔54\\leftrightarrow 5\(multi\-branch architectures benefit slightly more from over\-training\); Hash, DEMix, and SMoE\-Dropout retain their1010\-ep ranks\. Within multi\-branch, ours and matched\-SE move*together*\(\+16\.53\+16\.53vs\+16\.65\+16\.65PPL gain from1010\-ep→\\to11\-ep\), confirming the routing mechanism does not differentially benefit from extra epochs\. The conditional\-tie on NLP SlimPajama is therefore a property of the data partition’s fuzziness, not of the multi\-epoch training regime\.

### E\.12LoRA Specialization Heatmap

Per\-cluster×\\timesper\-branch zero\-ablation onours\_lora\_s42,K=20K\{=\}20[Wang et al\.](https://arxiv.org/html/2608.04084#bib.bib42)domains, evaluated on1515test\-split clusters \(5 clusters had no held\-out tasks\)\.

#### Summary statistics\.

Diagonal hits0/150/15\(0%\);max⁡\|Δ\|=\+0\.100\\max\|\\Delta\|\{=\}\+0\.100ROUGE\-L \(cluster1515×\\timesbranch44, off\-diagonal\); mean\|Δ\|=0\.014\|\\Delta\|\{=\}0\.014; diag meanΔ=\+0\.0103\\Delta\{=\}\+0\.0103, off\-diag meanΔ=\+0\.0049\\Delta\{=\}\+0\.0049;diag/off\-diag ratio2\.1×2\.1\\times; sign skew152152positive /4141zero /107107negative \(of300300cells\)\.

#### Reading\.

Branches carry cluster\-specific information \(real signal,2\.1×2\.1\\timesratio,max⁡\|Δ\|\\max\|\\Delta\|well above off\-diagonal mean0\.0050\.005\) but are*anti\-aligned*with the imposed round\-robin assignment\. Three lines of evidence: \(1\) The0/150/15diagonal hits are not noise — for no covered cluster does its assigned branch produce the most\-negativeΔ\\Delta\(i\.e\., contribute most when present\); diagonalΔ\\Deltamedians at\+0\.003\+0\.003, with99positive /55negative /11zero across1515clusters\. \(2\) Top\-55most\-negative cells \(ablation hurts most⇒\\Rightarrowstrongest contribution\) are all off\-diagonal: cluster1313relies on branches\{19,15\}\\\{19,15\\\}; cluster99relies on\{0,3,19\}\\\{0,3,19\\\}\. \(3\) Top\-55most\-positive cells \(ablation*helps*⇒\\Rightarrowbranch hurts cluster\) are all off\-diagonal: cluster1515hurt by\{4,11\}\\\{4,11\\\}\(Δ=\+0\.10\\Delta\{=\}\+0\.10each\); cluster1818hurt by\{0,10\}\\\{0,10\\\}\. This is consistent with the LoRAΔ≈0\\Delta\{\\approx\}0tie under uniform\-mask inference \(Sec\.[5\.6](https://arxiv.org/html/2608.04084#S5.SS6)\): branches*do*specialize, but the specialization structure does not match the imposedK=20K\{=\}20[Wang et al\.](https://arxiv.org/html/2608.04084#bib.bib42)partition, so routing\-weighted output averages contributions that don’t align with the cluster the input is in\. This is a richer negative result than “no specialization” — the Soft SpecDrop mechanism is mechanically active, but the imposed K=20 task partition is not the right partition\. At a different K or a non\-Wang\-2022 clustering, alignment may be recovered \(left as an explicit open question\)\.

### E\.13LoRA Per\-Task ROUGE\-L Breakdown \(F\)

s42,119119SuperNI held\-out tasks,ipt=10\\text\{ipt\}\{=\}10instances per task \([Wang et al\.](https://arxiv.org/html/2608.04084#bib.bib42)Tk\-Instruct codebase default\), top\-3 LoRA methods\. Mean ROUGE\-L: ours0\.51370\.5137, HydraLoRA0\.51170\.5117, MB\-LoRA no\-routing0\.49620\.4962\(within1\.0​σ1\.0\\sigmaof 3\-seed main table, consistent\)\.

#### Sorted\-Δ\\Deltasplit \(ours−\-HydraLoRA per task\)\.

4747tasks ours wins,4646HydraLoRA wins,2626tied — near\-balanced split, meanΔ≈\+0\.002\\Delta\{\\approx\}\+0\.002\.

*Right tail \(ours wins, cluster\-aligned tasks\):*

- •task1390\_wscfixed\_coreference\(Δ=\+0\.40\\Delta\{=\}\{\+\}0\.40\)
- •task202\_mnli\_contradiction\_classification\(Δ=\+0\.40\\Delta\{=\}\{\+\}0\.40\)
- •task1158\_bard\_analogical\_reasoning\_manipulating\_items\(Δ=\+0\.40\\Delta\{=\}\{\+\}0\.40\)
- •task936\_defeasible\_nli\_snli\_classification\(Δ=\+0\.30\\Delta\{=\}\{\+\}0\.30\)
- •task033\_winogrande\_answer\_generation\(Δ=\+0\.30\\Delta\{=\}\{\+\}0\.30\)

*Left tail \(HydraLoRA wins, cross\-cluster reasoning tasks\):*

- •task1387\_anli\_r3\_entailment\(Δ=−0\.50\\Delta\{=\}\{\-\}0\.50\)
- •task233\_iirc\_link\_exists\_classification\(Δ=−0\.40\\Delta\{=\}\{\-\}0\.40\)
- •task200\_mnli\_entailment\_classification\(Δ=−0\.40\\Delta\{=\}\{\-\}0\.40\)
- •task020\_mctaco\_span\_based\_question\(Δ=−0\.30\\Delta\{=\}\{\-\}0\.30\)
- •task1152\_bard\_analogical\_reasoning\_causation\(Δ=−0\.20\\Delta\{=\}\{\-\}0\.20\)

#### Interpretation\.

The split is the fine\-grained signature of the main\-table tie: ours specializes when the task aligns with one of ourK=20K\{=\}20clusters \(coreference, classification within a domain family\), HydraLoRA wins on cross\-cluster reasoning \(multi\-hop entailment, causal inference, link prediction\) because its asymmetric A/B \+ gate routes a single example to a knowledge\-mixture rather than one cluster\. TheK=20K\{=\}20categorical structure helps when the task lives inside one cluster, hurts when it spans clusters\.

### E\.14Per\-Task Win\-Pattern×\\timesCluster Association \(χ2\\chi^\{2\}onn=119n\{=\}119tasks\)

We extend the granularity\-alignment test fromn=4n\{=\}4settings ton=119n\{=\}119tasks by cross\-tabulating the per\-task ours\-vs\-HydraLoRA win/loss/tie \(App\.[E\.13](https://arxiv.org/html/2608.04084#A5.SS13)\) with the[Wang et al\.](https://arxiv.org/html/2608.04084#bib.bib42)K=20K\{=\}20cluster\_id and applying aχ2\\chi^\{2\}omnibus test plus per\-cluster Fisher exact tests\.

Both omnibus tests fail to rejectH0H\_\{0\}\(random allocation\), and no individual per\-cluster Fisher reachesp<0\.05p<0\.05\(closest: cluster1313atp=0\.079p\{=\}0\.079,n=5n\{=\}5\)\. Per\-cluster heterogeneity exists descriptively \(cluster1313,n=5n\{=\}5, ours wins80%80\\%; cluster33,n=11n\{=\}11, HydraLoRA wins73%73\\%\) but is not statistically distinguishable from chance at the available per\-cluster sample sizes \(median per\-clustern=5n\{=\}5;χ2\\chi^\{2\}with small expected counts loses power\)\.

#### Independent corroboration of the LoRA anti\-alignment finding\.

The null is double\-corroborated by App\.[E\.12](https://arxiv.org/html/2608.04084#A5.SS12)\(0/150/15diagonal hits with2\.1×2\.1\\timesdiag/off ratio\): two independent diagnostics — per\-cluster zero\-ablation and per\-task Fisher win\-association — both indicate that the Soft SpecDrop mechanism is mechanically active on LoRA \(App\.[E\.13](https://arxiv.org/html/2608.04084#A5.SS13)shows real47/46/2647/46/26per\-task differentiation;max⁡\|Δ\|=0\.10\\max\|\\Delta\|\{=\}0\.10ROUGE\-L well above noise\) but the imposedK=20K\{=\}20[Wang et al\.](https://arxiv.org/html/2608.04084#bib.bib42)partition does not match the underlying task feature structure\. The granularity\-alignment thesis at fine task grain is consistent with — but does not statistically prove — the cross\-setting trend; the partition itself is the binding constraint, as the cross\-setting and now per\-task evidence jointly confirm\.

### E\.15Embedding\-Structure Diagnostic: BGE vs DINOv2 Modality Asymmetry

To characterize whether SlimPajama text chunks carry the discrete cluster structure that categorical routing presupposes, we run a 3\-metric cluster\-validity scan \(silhouette, Calinski\-Harabasz, Davies\-Bouldin\) on BGE\-large\-en\-v1\.5 embeddings of195195K512512\-token SlimPajama chunks overk∈\[2,50\]k\\in\[2,50\], with DINOv2\-base embeddings of CIFAR\-100 images \(5050K samples\) as a contrastive image\-modality reference\.

#### BGE embeddings lack discrete cluster structure\.

The BGE scan gives no consensus optimum: silhouette argmax atk=3k\{=\}3with value0\.0310\.031, below the Kaufman\-Rousseeuw0\.250\.25“substantial structure” threshold; CH near\-monotone\-decreasing; DB argmin atk=47k\{=\}47via singleton\-artifact\. DINOv2 embeddings of CIFAR\-100 images give silhouettesmax=0\.069s\_\{\\max\}\{=\}0\.069atk=50k\{=\}50with healthy clusters \(minimum cluster size364364, no singletons\)\. BGE text\-chunk embeddings of web text form a*continuous manifold*; DINOv2 image embeddings form mildly but discretely clustered structure\. The modality asymmetry directly supports the granularity\-mismatch interpretation of Sec\.[6](https://arxiv.org/html/2608.04084#S6): categorical routing is not a universal mechanism but one that presupposes a categorizable data modality\.

#### Full\-corpus intra\-chunk mixture and per\-chunk correlation\.

For all9,7669\{,\}766SlimPajama validation chunks we compute \(i\) the intra\-chunk topic mixture, slicing each512512\-token chunk into88sub\-windows of6464tokens, BGE\-embedding each sub\-window, and assigning it to thek=7k\{=\}7train\-fit clusters; and \(ii\) the per\-chunk cross\-entropy differenceΔ​CE\\Delta\\text\{CE\}between the matched\-SE No\-Routing control and ours \(33seeds each\)\.56\.1%\\mathbf\{56\.1\\%\}of chunks span≥2\{\\geq\}2clusters \(an earlier300300\-chunk estimate gave53%53\\%; this is the full\-set value\), quantifying the partition’s fuzziness\. The per\-chunk correlation between purity andΔ​CE\\Delta\\text\{CE\}is null \(Pearsonr=−0\.001r\{=\}\-0\.001,p=0\.95p\{=\}0\.95; Spearmanρ=−0\.001\\rho\{=\}\-0\.001,p=0\.94p\{=\}0\.94\), and homogeneous chunks \(43\.9%43\.9\\%of the set\) versus mixed chunks show statistically identical meanΔ​CE\\Delta\\text\{CE\}\(−0\.0023\-0\.0023vs−0\.0021\-0\.0021\)\. This null is the expected signature of a*training\-time, distribution\-level*property rather than an inference\-time, per\-chunk one: branch specialization forms over the whole training distribution, so a chunk being homogeneous at evaluation does not retroactively give it a specialized branch\. The evidence for the granularity\-alignment thesis is therefore cross\-setting — partitions that are clean throughout training \(CIFAR, BREEDS\) yield gains, fuzzy ones yield ties — robust to training protocol \(App\.[E\.11](https://arxiv.org/html/2608.04084#A5.SS11)\) and scale \(App\.[E\.10](https://arxiv.org/html/2608.04084#A5.SS10)\)\.

### E\.16Per\-Seed and Per\-Domain Results

Tables[11](https://arxiv.org/html/2608.04084#A5.T11)–[13](https://arxiv.org/html/2608.04084#A5.T13)report per\-seed top\-1/PPL for the CIFAR and NLP main tables and per\-domain PPL for three representative methods\.

Table 11:Per\-seed top\-1 accuracy \(%\) on CIFAR\-100 for the faithful baseline comparison of Table[1](https://arxiv.org/html/2608.04084#S5.T1)\(MultiBranchResNet110 / dense ResNet\-110,∼\\sim1\.7M params, 200 epochs, RTX 5090\)\.Table 12:Per\-seed validation perplexity on SlimPajama\-6B for the NLP comparison of Table[3](https://arxiv.org/html/2608.04084#S5.T3)\(30M\-parameter Transformer LM, 500M tokens, 10 epochs, RTX 5090, three seeds 42/123/456\)\. The matched No\-Routing\+SE seed\-456 value45\.1445\.14is the lowest of the three matched\-SE seeds; per\-seed \(ours−\-matched\-SE\) PPL differences are\+0\.05/\+0\.02/\+0\.22\+0\.05/\+0\.02/\+0\.22at s42/s123/s456, so the aggregate\+0\.10\+0\.10PPL gap \(ours behind matched\-SE\) is dominated by s456\.Table 13:Per\-domain validation perplexity on SlimPajama\-6B \(seed 42, 500M tokens, 10 epochs\) for the three methods shown\. Absolute per\-domain difficulty varies by an order of magnitude across domains\. Book domain \(4\.1%4\.1\\%of training data\) is omitted: the val split contains too few Book chunks at the500500M\-token scale for a stable per\-domain PPL estimate\.

## Appendix FArchitecture and Implementation Details

### F\.1MultiBranchResNet110 Architecture

For CIFAR\-100 experiments at ResNet\-110 scale, we useMultiBranchResNet110where all three layer groups are branched, mirroring the every\-layer MoE design of modern LLMs\[Jiang et al\.,[2024](https://arxiv.org/html/2608.04084#bib.bib21)\]:

conv1 \(shared\)→K×layer1→merge→K×layer2→merge→K×layer3→merge→FC\\text\{conv1 \(shared\)\}\\to K\\\!\\times\\\!\\text\{layer1\}\\to\\text\{merge\}\\to K\\\!\\times\\\!\\text\{layer2\}\\to\\text\{merge\}\\to K\\\!\\times\\\!\\text\{layer3\}\\to\\text\{merge\}\\to\\text\{FC\}Withnum\_blocks=18\\,\{=\}\\,18, this gives ResNet\-110 depth \(3 groups×\\times18 blocks×\\times2 convs \+ 2 = 110 layers\)\. Branch channel widths are auto\-computed to match single\-branch ResNet\-110 parameter count \(∼\\sim1\.74M\): withK=20K\\\!\{=\}\\\!20, branch channels are\[4,7,14\]\[4,7,14\]per layer group\. The always\-on branch \(a branch withpactive=1\.0p\_\{\\text\{active\}\}=1\.0for all categories\) uses the same channel widths as routed branches but may have a different block count, controlling capacity without requiring dimension projection\. This is a natural extension of the routing framework—not a separate architecture—and is parameter\-matched by reducing routed branch widths accordingly\. The design is inspired by the shared expert in DeepSeekMoE\[Dai et al\.,[2024](https://arxiv.org/html/2608.04084#bib.bib7)\]\.

### F\.2Efficient Multi\-Branch Computation

A naive implementation ofKKparallel branches uses a Pythonfor\-loop, launchingKKsequential CUDA kernels per layer\. WithK=20K\\\!\{=\}\\\!20branches, 3 layer groups, 18 blocks per group, and the standard ResNetBasicBlock’s 2 convolutions per block, this results in3×20×18×2=2,1603\\times 20\\times 18\\times 2=2\{,\}160sequential kernel launches per forward pass—over an order of magnitude slower than a single\-branch ResNet\-110 on a modern GPU\.

#### CNN: Grouped convolution\.

We fuse allKKbranch convolutions into a single grouped convolution \(groups=K\\,\{=\}\\,K\)\. The shared input is repeated along the channel dimension:\(B,C,H,W\)→\(B,K​C,H,W\)\(B,C,H,W\)\\to\(B,KC,H,W\)\. Each group processes itsCCinput channels independently with its own filter set, producing\(B,K​Cout,H,W\)\(B,KC\_\{\\text\{out\}\},H,W\)in one CUDA kernel call\.BatchNorm\(K​Cout\)\(K\\,C\_\{\\text\{out\}\}\)naturally provides per\-branch normalization since each group ofCoutC\_\{\\text\{out\}\}output channels has independent statistics \(γ\\gamma,β\\beta, running mean/var\)\. This reduces kernel launches from2,1602\{,\}160to∼108\\sim 108\(two grouped convs per block,5454blocks total\)\.

#### NLP: Batched einsum\.

For transformer FFN branches, we stackKKweight matrices into tensors𝐖1∈ℝK×F×D\\mathbf\{W\}\_\{1\}\\in\\mathbb\{R\}^\{K\\times F\\times D\}and𝐖2∈ℝK×D×F\\mathbf\{W\}\_\{2\}\\in\\mathbb\{R\}^\{K\\times D\\times F\}, then compute all branches simultaneously viatorch\.einsum:

𝐇\\displaystyle\\mathbf\{H\}=GELU​\(einsum​\(‘btd,kfd→btkf’,𝐗,𝐖1\)\+𝐛1\),\\displaystyle=\\text\{GELU\}\(\\texttt\{einsum\}\(\\text\{\`btd,kfd\}\\to\\text\{btkf'\},\\mathbf\{X\},\\mathbf\{W\}\_\{1\}\)\+\\mathbf\{b\}\_\{1\}\),𝐎\\displaystyle\\mathbf\{O\}=einsum​\(‘btkf,kdf→btkd’,𝐇,𝐖2\)\+𝐛2\\displaystyle=\\texttt\{einsum\}\(\\text\{\`btkf,kdf\}\\to\\text\{btkd'\},\\mathbf\{H\},\\mathbf\{W\}\_\{2\}\)\+\\mathbf\{b\}\_\{2\}This replacesKKsequential matrix multiplications with two batched operations\.

#### Additional optimizations\.

We use mixed\-precision training \(torch\.amp\) andtorch\.compilefor kernel fusion\. Combined with grouped convolution, these achieve11\.0×\\timesspeedupover the naive for\-loop implementation, reducing per\-batch time from 1,387ms to 126ms \(measured on an NVIDIA A100 during development\)\.

#### Equivalence verification\.

Both optimizations are mathematically equivalent to the naive implementation: \(1\) forward outputs match withinatol=1e\-5on CPU \(tests/test\_grouped\_conv\.py\); \(2\) gradient differences pass a1e\-3threshold \(typical observed magnitude<10−4<10^\{\-4\}, the threshold accommodates cuDNN nondeterminism\); \(3\) a 20\-epoch training comparison yields final accuracy within seed noise \(verified byarchive/tests/test\_grouped\_training\_equivalence\.py\)\.

Table 14:Computational overhead of multi\-branch architectures vs\. single\-branch baselines\. All times are forward\+backward per batch, measured on an NVIDIA A100 during implementation development; the speedup columns compare implementations on the same device\. End\-to\-end training wall\-clock on the RTX 5090 production hardware is reported in App\.[F\.3](https://arxiv.org/html/2608.04084#A6.SS3)\.

### F\.3Per\-Method Compute \(MACs\) and Wall\-Clock

We report per\-forward compute for every method in the four main tables as*multiply–accumulate operations*\(MACs\): one MAC is one multiplication plus one addition, soFLOPs≈2×MACs\\text\{FLOPs\}\\approx 2\\times\\text\{MACs\}\. MACs are counted withfvcore’s operator\-level counter, the convention used by the ViT/DeiT/Soft MoE reference implementations\. SpecDrop’s fixed routing adds zero MACs \(no router network\); we claim no compute advantage\.

Table 15:Per\-method MACs across the four settings\.CIFAR: per32×3232\{\\times\}32image; ImageNet: per224×224224\{\\times\}224image; SlimPajama: per512512\-token sequence; SuperNI: adapter add\-on per token as % of the frozen Llama\-3\.2\-1B base \(∼1\.24\{\\sim\}1\.24GMACs/token, shared by all methods\)\. The deployed all\-blocks Soft MoE runs at37%37\\%of dense compute under parameter matching; the tuned second\-half variant at68%68\\%; the compute\-matched variant restores dense\-level MACs with parameters unconstrained\.SettingMethodMACsCIFAR\-100dense\-backbone baselines255\.3MMultiBranchK=20K\{=\}20\(No\-Routing, HardCategory, ours\)287\.6M \(\+12\.7%\+12\.7\\%\)ImageNet\-1KDense ViT\-S/164\.25GSoft SpecDrop \(ours\) / No\-Routing\+\+SE4\.25G \(±0\.0%\\pm 0\.0\\%\)No\-Routing \(no SE\)4\.22GMod\-Squad / ALF top\-kkrouter4\.27GCOMET5\.65GSoft MoE: deployed / tuned / comp\.\-matched1\.56G / 2\.91G / 4\.26GSlimPajamaall eight methods,15\.9315\.93–15\.9715\.97Gwithin0\.3%0\.3\\%SuperNI/LoRASingle LoRA / No\-Routing\+18\.2%\+18\.2\\%Soft SpecDrop \(ours\)\+18\.0%\+18\.0\\%LoRAMoE / MoCLE\+18\.1%\+18\.1\\%/\+7\.2%\+7\.2\\%HydraLoRA\+30\.6%\+30\.6\\%Table 16:Measured wall\-clock, all four settings\(3\-seed mean total training hours on RTX 5090, bf16; ratios vs the per\-setting dense/single reference\)\. Ours is within∼0\.5%\{\\sim\}0\.5\\%of the architecture\-matched No\-Routing control on three of four settings and within7%7\\%on CIFAR: the routing rule adds little to no measurable cost; the cost is the multi\-branch architecture, shared by ours and the No\-Routing controls\. The three new ImageNet baselines \(bottom\) were trained on identical\-model GPUs on a separate node; per\-epoch seconds are directly comparable to the deployed Soft MoE’s502502s\.