Hierarchical Copula-Gumbel-Top-\texorpdfstring{$K$}{K} Routing: Two-Sided Dependence Control for Frozen Mixture-of-Experts at Fixed Per-Token Routing Laws

# Two-Sided Dependence Control for Frozen Mixture-of-Experts at Fixed Per-Token Routing Laws
Source: [https://arxiv.org/html/2607.28670](https://arxiv.org/html/2607.28670)
## Hierarchical Copula\-Gumbel\-Top\-KKRouting: Two\-Sided Dependence Control for Frozen Mixture\-of\-Experts at Fixed Per\-Token Routing Laws

Richard Yi Da Xu Hong Kong Baptist University and TadReamk Limited xuyida@hkbu\.edu\.hk,richard@tadreamk\.com

###### Abstract

A stochastic Gumbel\-Top\-KKrouter defines, for every token of a mixture\-of\-experts \(MoE\) model, a*routing law*: a distribution over ordered expert lists and mixture weights\. We ask which*joint*distributions over the routing choices of different tokens are reachable while every individual token’s complete routing law is held exactly fixed\. We give a two\-sided construction,*Hierarchical Copula\-Gumbel\-Top\-KK*\(H\-CGA\)\. Within a group of related tokens, an exchangeable Gaussian copula positively correlates the Gumbel perturbations at each expert coordinate, which can increase within\-group expert\-set coherence\. Across disjoint pairs of groups, a tunable antithetic construction introduces a selectable amount of negative dependence\. We prove that both operations leave each token’s ordered Top\-KKsample, mixture weights, and inclusion probabilities identical in distribution to independent routing*at a routing layer conditioned on its pre\-routing logits*; conditional expected expert traffic is preserved as a consequence\. We characterize the resulting trade\-off: positive within\-group coupling can only inflate the variance of realized expert loads relative to independent routing, while nonnegative cross\-group opposition can only reduce it relative to flat coupling at the same within\-group strength\. Coherence and load dispersion are thus controlled by two complementary dependence dials on the invariance constraint surface\. Because the base model is untouched, the dials can be driven by a small controller over frozen features, trainable with a score\-function estimator: the frozen network is evaluated only in the forward direction, and gradients are confined to the controller\. An initial small\-scale pilot validates the mechanism and the training route, but does not establish task\-level fine\-tuning gains\.

## 1Introduction

Sparse MoE layers scale language models by evaluating only a few experts for each token\(Shazeeret al\.,[2017](https://arxiv.org/html/2607.28670#bib.bib1); Lepikhinet al\.,[2021](https://arxiv.org/html/2607.28670#bib.bib2); Feduset al\.,[2022](https://arxiv.org/html/2607.28670#bib.bib3)\)\. Modern MoEs commonly use top\-KKrouting: a router scores all experts, sends a token to itsKKhighest\-scoring experts, and combines their outputs\. Under the standard stochastic formulation, Gumbel\-Top\-KKsampling\(Koolet al\.,[2019](https://arxiv.org/html/2607.28670#bib.bib8)\), the router assigns each token a routing law—a Plackett–Luce distribution over ordered expert lists\. Nearly all work on MoE routing modifies this per\-token law: fine\-tuning changes the logits, auxiliary losses reshape the gates, and similarity\-aware routers alter individual selections\(Nguyenet al\.,[2025](https://arxiv.org/html/2607.28670#bib.bib10); Omiet al\.,[2025](https://arxiv.org/html/2607.28670#bib.bib11)\)\.

We study a different, largely unexamined degree of freedom\. Holding every token’s routing law*exactly*fixed, the joint distribution over the choices of different tokens is still free: Sklar’s theorem separates marginals from dependence\(Sklar,[1959](https://arxiv.org/html/2607.28670#bib.bib13); Nelsen,[2006](https://arxiv.org/html/2607.28670#bib.bib14)\), and the routing choices of a frozen MoE are a collection of discrete marginals awaiting a dependence structure\. This paper asks:*which joint routing behaviors are reachable on this invariance constraint surface, and what do the reachable extremes trade off?*

The question has practical stakes in both directions of dependence\.*Positive*cross\-token dependence makes the tokens of a phrase, entity, or code identifier more likely to reuse the same experts—local coherence and fewer distinct experts per group\. This may improve expert locality, depending on the execution system\. But bunching grouped tokens onto shared experts makes their inclusion counts positively correlated, so realized per\-expert loads become burstier even though conditional expected loads at that layer are provably unchanged\.*Negative*cross\-group dependence pushes the other way: anticorrelating the demand of different groups reduces the variance of realized loads\. We show both directions are simultaneously available in one hierarchical construction, and both preserve every token’s routing law\.

We proposeH\-CGA, shown in Figure[1](https://arxiv.org/html/2607.28670#S1.F1)\. The frozen router produces logitsℓt​e\\ell\_\{te\}for tokenttand expertee; ordinary stochastic routing adds independent Gumbel noises and takes the topKKperturbed logits\.H\-CGAorganizes the noise hierarchically\. Within a groupggof tokens, the noises at each expert coordinate share a group latentζg​e\\zeta\_\{ge\}through an exchangeable Gaussian copula: related tokens receive positively correlated perturbations, expert by expert\. Across a disjoint pair of groups\(g,g′\)\(g,g^\{\\prime\}\), the latents have a tunable antithetic relationship with opposition strengthαg,g′∈\[0,1\]\\alpha\_\{g,g^\{\\prime\}\}\\in\[0,1\]\. Atαg,g′=1\\alpha\_\{g,g^\{\\prime\}\}=1, whatever random push a pair member gives its tokens toward expertee, its partner receives the opposite push; atαg,g′=0\\alpha\_\{g,g^\{\\prime\}\}=0, the two groups are independent\. The noise vector of any single token retains its original i\.i\.d\. Gumbel distribution throughout\.

The method leaves each token’s marginal selection distribution unchanged and acts only on the cross\-token dependence of the routing noise: positive coordination raises the tendency of related tokens to make matching random selections, while negative coordination reduces the tendency of distinct groups to do so simultaneously\. For the stochastic router studied here, this dependence can be introduced without altering any single token’s routing law\.

frozen hiddenstateshth\_\{t\}frozen routerlogitsℓt\\ell\_\{t\}trainable controllerag,ρg,αg,g′a\_\{g\},\\rho\_\{g\},\\alpha\_\{g,g^\{\\prime\}\}hierarchical copulaGumbel noisesfrozenGumbel\-Top\-KKfrozen expertsand weighted sumFigure 1:H\-CGAas a routing\-side dependence layer\. Only the controller is trainable\. The base router, experts, and mixture rule are frozen\. Positive within\-group coupling and tunable cross\-group opposition are both generated in the hierarchical copula stage; the pairing is a fixed, pre\-sampling design choice, while the controller may set its opposition strengthαg,g′\\alpha\_\{g,g^\{\\prime\}\}\. Atag=0a\_\{g\}=0the sampler is independent Gumbel\-Top\-KKrouting exactly\.#### Contributions\.

1. 1\.We introduceH\-CGA, a hierarchical copula layer over the routing noise of a frozen stochastic Gumbel\-Top\-KKMoE that controls cross\-token dependence in both directions—positive within\-group coupling for local expert\-set coherence and tunable negative cross\-group coupling for load\-variance control—without touching router logits, experts, or any per\-token routing law\.
2. 2\.We prove*full per\-token routing\-law invariance*for the entire hierarchy: the selected ordered Top\-KKlist and gate\-based mixture weights of every token have the same distribution as under independent Gumbel\-Top\-KKrouting\. Consequently, conditional expected expert\-inclusion counts at that layer are preserved \(Theorem[1](https://arxiv.org/html/2607.28670#Thmtheorem1), Corollaries[1](https://arxiv.org/html/2607.28670#Thmcorollary1)and[2](https://arxiv.org/html/2607.28670#Thmcorollary2)\)\.
3. 3\.We characterize the coherence–dispersion trade\-off on the invariance surface \(Proposition[1](https://arxiv.org/html/2607.28670#Thmproposition1)\): relative to independent routing, flat positive coupling can only increase the variance of realized expert loads; relative to flat coupling at the same strength, every nonnegative cross\-group opposition strength can only decrease it\. Conditional expected loads are identical in all schemes\.
4. 4\.We formulate routing\-only adaptation as an application: a small controller reads frozen features and sets the dependence dials, and is trainable with a score\-function estimator that evaluates the frozen base model only in the forward direction\.

## 2Background and Problem Setting

#### Top\-KKMoE routing\.

Let a frozen router map token representationhth\_\{t\}to logitsℓt=\(ℓt​1,…,ℓt​E\)\\ell\_\{t\}=\(\\ell\_\{t1\},\\ldots,\\ell\_\{tE\}\)\. A deterministic top\-KKrouter selects theKKlargest entries\. We instead use the standard stochastic Gumbel\-Top\-KKlaw: drawγt​e​∼iid​Gumbel​\(0,1\)\\gamma\_\{te\}\\overset\{\\mathrm\{iid\}\}\{\\sim\}\\mathrm\{Gumbel\}\(0,1\)and let

\(rt​1,…,rt​K\)=TopKe⁡\{ℓt​e\+γt​e\},St=\{rt​1,…,rt​K\}\.\(r\_\{t1\},\\ldots,r\_\{tK\}\)=\\operatorname\{TopK\}\_\{e\}\\\{\\ell\_\{te\}\+\\gamma\_\{te\}\\\},\\qquad S\_\{t\}=\\\{r\_\{t1\},\\ldots,r\_\{tK\}\\\}\.\(1\)The ordered list is a Plackett–Luce sample without replacement\(Koolet al\.,[2019](https://arxiv.org/html/2607.28670#bib.bib8)\)\. We use gate\-based mixture weights

wt​e=exp⁡\(ℓt​e\)​1​\[e∈St\]∑j∈Stexp⁡\(ℓt​j\),yt=∑e∈Stwt​e​fe​\(ht\)\.w\_\{te\}=\\frac\{\\exp\(\\ell\_\{te\}\)\\,\\mathbf\{1\}\[e\\in S\_\{t\}\]\}\{\\sum\_\{j\\in S\_\{t\}\}\\exp\(\\ell\_\{tj\}\)\},\\qquad y\_\{t\}=\\sum\_\{e\\in S\_\{t\}\}w\_\{te\}f\_\{e\}\(h\_\{t\}\)\.\(2\)Other weight rules that are deterministic functions of the complete Gumbel\-noise vector can be used as well\.

#### What “routing law” means here\.

Conditioned on a token’s frozen logits, its routing law is the joint distribution of its ordered Top\-KKlist and its weights in \([2](https://arxiv.org/html/2607.28670#S2.E2)\)\. It describes what can happen to one token under stochastic routing\. It does not specify how the random choices of two different tokens co\-vary, nor does it fix realized batch loads\.

#### Scope and level of the guarantee\.

The reference router in this paper is*independent stochastic*Gumbel\-Top\-KK, not deterministic top\-KK\. A deterministic router has a degenerate routing law, so no nontrivial dependence can be introduced while preserving that law exactly\. Capacity clipping, token dropping, and expert\-choice allocation occur after the choices in \([1](https://arxiv.org/html/2607.28670#S2.E1)\); they are outside the invariance results below\. All invariance and expected\-load claims are*layer\-local*: they condition on the hidden states and logits entering one routing layer\. If coupling changes the joint hidden\-state distribution at an earlier layer, later\-layer logits may change as well\. Therefore the results do not by themselves establish end\-to\-end invariance of a multi\-layer MoE\.

#### Relation to PEFT\.

Conventional MoE parameter\-efficient fine\-tuning attaches weight adapters to experts or learns an additional adapter router\(Huet al\.,[2022](https://arxiv.org/html/2607.28670#bib.bib6); Liuet al\.,[2026](https://arxiv.org/html/2607.28670#bib.bib9)\); both allocate trainable*representational*capacity and change what individual tokens prefer\. The framework studied here exposes a different, orthogonal budget: with all base parametersΘbase\\Theta\_\{\\mathrm\{base\}\}frozen—embeddings, attention, router, and experts—the only trainable object is a controllerϕ\\phiover the*dependence*of routing randomness, ranging from one scalar per MoE layer to a small MLP\. Routing\-only adaptation \(Section[3\.5](https://arxiv.org/html/2607.28670#S3.SS5)\) is thus an application of the dependence framework, not its definition\. It is not a conventional weight\-adapter PEFT method: by construction, it cannot change any token’s marginal preference for an expert or its expected inclusion probability\.

## 3Hierarchical Copula\-Gumbel\-Top\-KK

This section introduces cross\-token coordination into a stochastic router while leaving each token’s marginal routing law unchanged\. Throughout, the marginal routing law denotes the distribution of a single token’s ordered expert list under repeated routing, and the joint routing law denotes the dependence structure among the selections of distinct tokens routed together; the frozen router fixes the former, andH\-CGAacts only on the latter\.

The construction proceeds in four stages\. Section[3\.1](https://arxiv.org/html/2607.28670#S3.SS1)forms groups of related tokens and fixes a coupling strength before any routing noise is drawn; Section[3\.2](https://arxiv.org/html/2607.28670#S3.SS2)induces correlated Gumbel noise within a group; Section[3\.3](https://arxiv.org/html/2607.28670#S3.SS3)establishes that every token retains its original stochastic Top\-KKlaw; and Section[3\.4](https://arxiv.org/html/2607.28670#S3.SS4)introduces a tunably antithetic shared signal between paired groups to counteract the load burstiness produced by positive within\-group coordination\.

### 3\.1A pre\-routing coupling controller

Partition a sequence into disjoint candidate groupsgg\. In the minimal version, these are fixed windows ofmmadjacent tokens\. A controller reads only frozen, pre\-routing featuressgs\_\{g\}, such as the mean hidden state, mean gate entropy, within\-group gate similarity, and boundary indicators:

ag=σ​\(ϕ​\(sg\)\)∈\[0,1\],ρg=ag​ρmax,0≤ρmax<1\.a\_\{g\}=\\sigma\(\\phi\(s\_\{g\}\)\)\\in\[0,1\],\\qquad\\rho\_\{g\}=a\_\{g\}\\rho\_\{\\max\},\\qquad 0\\leq\\rho\_\{\\max\}<1\.\(3\)Hereσ\\sigmais the logistic sigmoid, so thatag∈\[0,1\]a\_\{g\}\\in\[0,1\]\. The correlation used by the sampler isρg\\rho\_\{g\}, capped below one byρmax\\rho\_\{\\max\}\. Keepingρmax<1\\rho\_\{\\max\}<1leaves each token with private randomness and keeps the Gaussian construction below nondegenerate\.

#### A second dial between paired groups\.

Once groups are matched into disjoint pairs, the controller may also choose an opposition strength

αg,g′=σ​\(ϕpair​\(stopgrad⁡\(sg\),stopgrad⁡\(sg′\)\)\)∈\[0,1\]\.\\alpha\_\{g,g^\{\\prime\}\}=\\sigma\\\!\\left\(\\phi\_\{\\mathrm\{pair\}\}\(\\operatorname\{stopgrad\}\(s\_\{g\}\),\\operatorname\{stopgrad\}\(s\_\{g^\{\\prime\}\}\)\)\\right\)\\in\[0,1\]\.\(4\)The within\-group strengthρg\\rho\_\{g\}and between\-group strengthαg,g′\\alpha\_\{g,g^\{\\prime\}\}have different jobs\.ρg\\rho\_\{g\}makes tokens in the same group share more randomness;αg,g′\\alpha\_\{g,g^\{\\prime\}\}determines how strongly the two groups’ shared random signals oppose each other\. A fixed value of either dial is also valid; learning both is optional\.

For a single expert coordinate,H\-CGAmay assign the tokens of a group a partly shared random perturbation toward that expert when their frozen router scores are similar\. This perturbation does not increase the expert’s score, alter the router weights, or force any token to select the expert; it affects only the random tie\-breaking component of otherwise unchanged routing decisions\. Atρg=0\\rho\_\{g\}=0these perturbations are independent, and asρg\\rho\_\{g\}approaches one the shared component dominates, while each token retains the same marginal noise distribution\.

The controller must be evaluated before routing noise is drawn\. Atag=0a\_\{g\}=0,ρg=0\\rho\_\{g\}=0and the rule reduces exactly to independent stochastic Top\-KK\. Groups may be chosen adaptively from frozen inputs, and groups may further be matched into disjoint pairs with tunable opposition \(Section[3\.4](https://arxiv.org/html/2607.28670#S3.SS4)\), provided that group membership*and*the pairing are fixed before any routing noise is sampled and groups remain disjoint\.

#### Why “pre\-routing” is required\.

The group, its coupling strength, and its optional paired partner may depend on information already available to the frozen model, such as hidden states or gate similarity\. They must not depend on the Gumbel draws or on the selected experts from the current routing operation\. Selecting a group after seeing a favorable random outcome would bias the distribution and destroy the preservation result\.

### 3\.2Copula\-correlated Gumbel perturbations

For every experteeand groupgg, independently draw a shared latentζg​e∼𝒩​\(0,1\)\\zeta\_\{ge\}\\sim\\mathcal\{N\}\(0,1\)and per\-token noisesϵt​e∼𝒩​\(0,1\)\\epsilon\_\{te\}\\sim\\mathcal\{N\}\(0,1\):

yt​e=ρg​ζg​e\+1−ρg​ϵt​e,ut​e=Φ​\(yt​e\),γt​e=−log⁡\[−log⁡\(ut​e\)\]\.y\_\{te\}=\\sqrt\{\\rho\_\{g\}\}\\,\\zeta\_\{ge\}\+\\sqrt\{1\-\\rho\_\{g\}\}\\,\\epsilon\_\{te\},\\qquad u\_\{te\}=\\Phi\(y\_\{te\}\),\\qquad\\gamma\_\{te\}=\-\\log\[\-\\log\(u\_\{te\}\)\]\.\(5\)
#### Distributional properties of the sampler\.

In \([5](https://arxiv.org/html/2607.28670#S3.E5)\),yt​ey\_\{te\}is a convex combination of the shared latentζg​e\\zeta\_\{ge\}, common to every token in groupggat expertee, and the private noiseϵt​e\\epsilon\_\{te\}, specific to tokentt\. The coefficientsρg\\sqrt\{\\rho\_\{g\}\}and1−ρg\\sqrt\{1\-\\rho\_\{g\}\}are chosen so thatyt​ey\_\{te\}is marginally standard normal for every token, withρg\\rho\_\{g\}setting the fraction of variance attributable to the shared source:ρg=0\\rho\_\{g\}=0under independent routing, andρg\\rho\_\{g\}increases with coordination\. The subsequent transformations alter the marginal shape but not the dependence pattern:ut​e=Φ​\(yt​e\)u\_\{te\}=\\Phi\(y\_\{te\}\)is uniform on\[0,1\]\[0,1\], and the inverse Gumbel c\.d\.f\. mapsut​eu\_\{te\}to a standard Gumbel variateγt​e\\gamma\_\{te\}, as required by Gumbel\-Top\-KKrouting\. This Gaussian\-to\-uniform\-to\-Gumbel pipeline is the copula: it preserves each token’s marginal Gumbel law while inducing dependence across tokens\.

The draws are independent across expert coordinatesee\. This detail is essential\. A token needs independent Gumbel perturbations across experts for its usual ranked Top\-KKdistribution to remain valid\. We then apply the unchanged Top\-KKselection and mixture rule in \([1](https://arxiv.org/html/2607.28670#S2.E1)\) and \([2](https://arxiv.org/html/2607.28670#S2.E2)\)\.

Positiveρg\\rho\_\{g\}correlates the random perturbations received by the same expert across tokens\. When related tokens have similar frozen logits, this raises their chance of including the same experts\. It does not guarantee same\-expert inclusion for dissimilar gates\. And it has a quantifiable cost: bunching a group’s tokens onto shared experts makes their inclusion counts positively correlated, so the*variance*of realized per\-expert loads grows even though conditional expected loads at this layer are exactly preserved\. Proposition[1](https://arxiv.org/html/2607.28670#Thmproposition1)makes both statements precise: the within\-group and between\-group dials jointly control coherence and load dispersion, and Section[3\.4](https://arxiv.org/html/2607.28670#S3.SS4)supplies the second pole\.

### 3\.3What is preserved

#### The key distinction\.

The construction deliberately changes how tokens’ random choices move together\. It does not change the distribution of the random vector seen by one token\. For example, two adjacent tokens may now select the same expert more often, but if either token is considered alone and routed repeatedly, it has exactly the same probabilities for every ordered Top\-KKlist as before\. The following theorem formalizes this statement\.

###### Theorem 1\(Per\-token Top\-KKrouting\-law invariance\)\.

Condition on all frozen hidden states, router logits, group memberships, and controller outputs\. Suppose thatH\-CGAuses \([5](https://arxiv.org/html/2607.28670#S3.E5)\) and that the copula draws are independent across expert coordinates\. For every tokentt,

\(γt​1,…,γt​E\)​=𝑑​\(G1,…,GE\),Ge​∼iid​Gumbel​\(0,1\)\.\(\\gamma\_\{t1\},\\ldots,\\gamma\_\{tE\}\)\\overset\{d\}\{=\}\(G\_\{1\},\\ldots,G\_\{E\}\),\\qquad G\_\{e\}\\overset\{\\mathrm\{iid\}\}\{\\sim\}\\mathrm\{Gumbel\}\(0,1\)\.\(6\)Consequently, the ordered list\(rt​1,…,rt​K\)\(r\_\{t1\},\\ldots,r\_\{tK\}\), selected setStS\_\{t\}, and weights\(wt​1,…,wt​E\)\(w\_\{t1\},\\ldots,w\_\{tE\}\)have exactly the same conditional distribution as under the independent Gumbel\-Top\-KKrouter\.

###### Proof\.

For everyt,et,e,yt​ey\_\{te\}in \([5](https://arxiv.org/html/2607.28670#S3.E5)\) is standard normal, sout​eu\_\{te\}is uniform andγt​e\\gamma\_\{te\}is standard Gumbel\. Different coordinateseeuse independent\(ζg​e,ϵt​e\)\(\\zeta\_\{ge\},\\epsilon\_\{te\}\)draws, so\(γt​1,…,γt​E\)\(\\gamma\_\{t1\},\\ldots,\\gamma\_\{tE\}\)is i\.i\.d\. Gumbel\. The Top\-KKmap and the gate\-weight map in \([1](https://arxiv.org/html/2607.28670#S2.E1)\)–\([2](https://arxiv.org/html/2607.28670#S2.E2)\) are deterministic functions of this vector and the fixed logits, which proves the result\. ∎

#### What the theorem does and does not say\.

The theorem is conditional on the hidden states and logits entering this routing layer\. In plain terms, once the frozen router has supplied the scores for the current tokens, replacing independent noise byH\-CGAnoise does not change the probability distribution of any one token’s route\. It does change the joint distribution of several routes\. In a model with multiple coupled MoE layers, that change can in turn affect later hidden states and later logits\. Thus this is a precise layer\-local safety guarantee, not a claim that the entire multi\-layer network has an unchanged output distribution\.

###### Corollary 1\(Expected inclusion load is invariant\)\.

LetNe=∑t𝟏​\[e∈St\]N\_\{e\}=\\sum\_\{t\}\\mathbf\{1\}\[e\\in S\_\{t\}\]be the number of tokens that include expertee\. Under the conditions of Theorem[1](https://arxiv.org/html/2607.28670#Thmtheorem1),

𝔼​\[Ne∣\{ℓt\}t,\{ag,ρg\}g\]=∑tℙ​\(e∈St∣ℓt;independent Gumbel\-Top\-​K\)\.\\mathbb\{E\}\[N\_\{e\}\\mid\\\{\\ell\_\{t\}\\\}\_\{t\},\\\{a\_\{g\},\\rho\_\{g\}\\\}\_\{g\}\]=\\sum\_\{t\}\\mathbb\{P\}\\\!\\left\(e\\in S\_\{t\}\\mid\\ell\_\{t\};\\,\\text\{independent Gumbel\-Top\-\}K\\right\)\.\(7\)The right\-hand side is the conditional expected inclusion count of the frozen independent Gumbel\-Top\-KKrouter\. In particular, it does not depend on the coupling strengths, groups, or fixed pairing once the layer’s incoming logits are fixed\.

###### Proof\.

Apply Theorem[1](https://arxiv.org/html/2607.28670#Thmtheorem1)to the indicator𝟏​\[e∈St\]\\mathbf\{1\}\[e\\in S\_\{t\}\]and sum over tokens\. No independence across tokens is required for linearity of expectation\. ∎

#### Interpretation\.

The corollary follows from a simple accounting rule: expected load is the sum of each token’s expert\-inclusion probability, and those individual probabilities are unchanged\. It is stronger than preserving a softmax coordinate\. Under Gumbel\-Top\-KK, softmax values are not themselves expert\-inclusion probabilities\.H\-CGApreserves the complete base distribution over the ordered expert list and thus every inclusion probability implied by that distribution\.

Expected load is not the same as realized load: realized per\-batch counts fluctuate around this conditional mean\.H\-CGApreserves the conditional average over many routing draws, but it does*not*preserve realized loads, load variance, capacity overflow, or a deterministic top\-KKmodel’s output exactly\.

### 3\.4Hierarchical dependence control

Positive coupling inside one group encourages local agreement, but that same agreement can make the group’s demand for an expert arrive in a burst\. The second level of the hierarchy acts*between*groups\. It provides an opposing signal whose strength can be selected: if one group receives a shared random push toward an expert, its matched partner can receive an independent, partly opposite, or fully opposite shared push\.

Match groups into disjoint pairs\(g,g′\)\(g,g^\{\\prime\}\), fixed before sampling\. In the minimal version, consecutive non\-overlapping windows can be paired\. More generally, any fixed pre\-routing matching rule is valid\. Figure[2](https://arxiv.org/html/2607.28670#S3.F2)illustrates the maximally opposed case,αg,g′=1\\alpha\_\{g,g^\{\\prime\}\}=1\. For each pair and each expert coordinateee, draw independentζg​e,ηg′​e∼𝒩​\(0,1\)\\zeta\_\{ge\},\\eta\_\{g^\{\\prime\}e\}\\sim\\mathcal\{N\}\(0,1\)and set

Groupgg: lower hierarchytokent1t\_\{1\}tokent2t\_\{2\}tokent3t\_\{3\}shared latentζg​e\\zeta\_\{ge\}same shared signalpositive within\-group dependenceshared latentζg′​e\\zeta\_\{g^\{\\prime\}e\}tokent1′t^\{\\prime\}\_\{1\}tokent2′t^\{\\prime\}\_\{2\}tokent3′t^\{\\prime\}\_\{3\}Groupg′g^\{\\prime\}: lower hierarchysame shared signalpositive within\-group dependencehigher hierarchyαg,g′=1\\alpha\_\{g,g^\{\\prime\}\}=1: opposite group signalsFigure 2:The two\-level dependence structure for one expert coordinateee\. At the lower level, all tokens in each group share that group’s latent, creating positive dependence*within*Groupggand within Groupg′g^\{\\prime\}\. At the higher level,αg,g′\\alpha\_\{g,g^\{\\prime\}\}controls the opposition between the two group latents\. The diagram showsαg,g′=1\\alpha\_\{g,g^\{\\prime\}\}=1, where paired groups receive fully opposite shared random pushes\. Atαg,g′=0\\alpha\_\{g,g^\{\\prime\}\}=0, they are independent\. Tokens inside either individual group remain positively coordinated\. Private token noisesϵt​e\\epsilon\_\{te\}are omitted for clarity\.Figure[2](https://arxiv.org/html/2607.28670#S3.F2)shows the key point: opposition belongs to the*relationship between groups*, not to the relationship among tokens inside Groupg′g^\{\\prime\}\. Both groups have positive within\-group dependence; only the higher hierarchy controls how strongly their shared signals oppose one another\.

ζg′​e=−αg,g′​ζg​e\+1−αg,g′2​ηg′​e,0≤αg,g′≤1,\\zeta\_\{g^\{\\prime\}e\}=\-\\alpha\_\{g,g^\{\\prime\}\}\\zeta\_\{ge\}\+\\sqrt\{1\-\\alpha\_\{g,g^\{\\prime\}\}^\{2\}\}\\,\\eta\_\{g^\{\\prime\}e\},\\qquad 0\\leq\\alpha\_\{g,g^\{\\prime\}\}\\leq 1,\(8\)leaving the within\-group construction \([5](https://arxiv.org/html/2607.28670#S3.E5)\) and all per\-token noisesϵt​e\\epsilon\_\{te\}unchanged and independent\. Whatever random push the pair’s first member gives its tokens toward expertee, the second member receives an opposing shared component of strengthαg,g′\\alpha\_\{g,g^\{\\prime\}\}\. Atαg,g′=0\\alpha\_\{g,g^\{\\prime\}\}=0, the two group latents are independent; atαg,g′=1\\alpha\_\{g,g^\{\\prime\}\}=1, \([8](https://arxiv.org/html/2607.28670#S3.E8)\) becomes the fully antithetic ruleζg′​e=−ζg​e\\zeta\_\{g^\{\\prime\}e\}=\-\\zeta\_\{ge\}\. Pairing supports strong local negative dependence without requiring a jointly negative equicorrelated vector across all groups, whose feasible correlation is bounded below by−1/\(G−1\)\-1/\(G\-1\)and vanishes as the number of groups grows\.

#### Why this does not break the one\-token law\.

For anyαg,g′∈\[0,1\]\\alpha\_\{g,g^\{\\prime\}\}\\in\[0,1\], the two terms on the right of \([8](https://arxiv.org/html/2607.28670#S3.E8)\) combine independent standard normals with squared coefficients that sum to one\. The partner group’s shared signal is therefore still standard normal when viewed alone\. Each token continues to combine that signal with private noise in exactly the way used in \([5](https://arxiv.org/html/2607.28670#S3.E5)\)\. The following corollary records the consequence\.

###### Corollary 2\(Hierarchical routing\-law invariance\)\.

Under the paired\-latent rule \([8](https://arxiv.org/html/2607.28670#S3.E8)\), for any fixedαg,g′∈\[0,1\]\\alpha\_\{g,g^\{\\prime\}\}\\in\[0,1\], the conclusion of Theorem[1](https://arxiv.org/html/2607.28670#Thmtheorem1)and Corollary[1](https://arxiv.org/html/2607.28670#Thmcorollary1)continue to hold for every token\.

###### Proof\.

Equation \([8](https://arxiv.org/html/2607.28670#S3.E8)\) is standard normal for every pair and expert coordinate\. Pairs use independent draws across expert coordinates, and each token belongs to exactly one group, hence touches exactly one shared latent per expert coordinate\. Everyyt​ey\_\{te\}is therefore still standard normal with independent coordinates acrossee, and the proof of Theorem[1](https://arxiv.org/html/2607.28670#Thmtheorem1)applies verbatim\. ∎

###### Proposition 1\(Coherence–dispersion trade\-off\)\.

Condition on all logits, group memberships, pairings, within\-group coupling strengths, and opposition strengthsαg,g′∈\[0,1\]\\alpha\_\{g,g^\{\\prime\}\}\\in\[0,1\], and fix an expertee\. WriteXg=∑t∈g𝟏​\[e∈St\]X\_\{g\}=\\sum\_\{t\\in g\}\\mathbf\{1\}\[e\\in S\_\{t\}\]andNe=∑gXgN\_\{e\}=\\sum\_\{g\}X\_\{g\}\. Then the conditional expected loads𝔼​\[Ne\]\\mathbb\{E\}\[N\_\{e\}\]are identical under independent routing, flat coupling \(independentζg​e\\zeta\_\{ge\}across groups\), and the tunable paired construction, and:

1. \(i\)under flat coupling with any strengthsρg≥0\\rho\_\{g\}\\geq 0,Var⁡\(Ne\)\\operatorname\{Var\}\(N\_\{e\}\)is at least its value under independent routing;
2. \(ii\)under paired coupling with the sameρg\\rho\_\{g\}and anyαg,g′∈\[0,1\]\\alpha\_\{g,g^\{\\prime\}\}\\in\[0,1\],Var⁡\(Ne\)\\operatorname\{Var\}\(N\_\{e\}\)is at most its value under flat coupling\.

Part \(i\) states that positive within\-group coupling cannot make an expert’s conditional load less variable than under independent routing, which is the cost of local coherence\. Part \(ii\) states that adding cross\-group opposition to such coupling cannot make that load more variable than flat coupling at the same strengths\. The two constructions coincide atα=0\\alpha=0and are fully antithetic atα=1\\alpha=1; the opposition is a partial counterbalance, not a guarantee of lower variance than independent routing\.

###### Proof sketch\.

Both parts follow from the association inequality for functions of independent random variables\(Esaryet al\.,[1967](https://arxiv.org/html/2607.28670#bib.bib15)\)after one sign change of coordinates; the full argument is in the appendix\. For \(i\), given the group’s latent vector, token inclusion indicators within a group are conditionally independent with conditional means that are coordinatewise monotone in a common transformed latent, hence pairwise nonnegatively correlated, so within\-group count variance can only grow\. For \(ii\), after averaging over a partner group’s independent residual noise, its conditional mean is coordinatewise nonincreasing inWgW\_\{g\}\. The paired counts therefore have nonpositive covariance; unpaired groups are independent, so total variance can only shrink\. ∎

Two qualifications delimit the claim\. First, the two bounds run in opposite directions from different baselines: the hierarchical scheme reduces variance relative to*flat coupling at the sameρ\\rho*, not necessarily below the independent\-routing baseline; the dials interpolate, they do not dominate\. Second, the magnitude of both effects depends on how strongly inclusion probabilities respond to the shared latents, which varies with the gate distribution; the proposition signs the effects but does not quantify them, and heterogeneous gates can make either effect small\.

The higher level also changes*cross\-group*joint statistics\. Increasingαg,g′\\alpha\_\{g,g^\{\\prime\}\}strengthens the opposing shared component; the pilot’s synthetic check measures the resulting paired\-boundary overlap directly\. Flat coupling \(αg,g′=0\\alpha\_\{g,g^\{\\prime\}\}=0\) leaves their shared latents independent\. Applications that rely on cross\-group co\-occurrence patterns should treatαg,g′\\alpha\_\{g,g^\{\\prime\}\}and the pairing as active design choices, not free lunches\.

### 3\.5Routing\-only adaptation

This subsection explains one possible use of the dependence mechanism\. It does not claim that routing\-only adaptation is effective on every task\. The fixed\-coupling mechanism above needs no learning at all: a user can choose a window size, a pairing rule, and values ofρg\\rho\_\{g\}andαg,g′\\alpha\_\{g,g^\{\\prime\}\}\. Adaptation asks the separate question of whether a small controller can choose the coupling strength from frozen features while all ordinary model weights remain fixed\.

LetYYcollect the correlated standard\-normal variablesyt​ey\_\{te\}in \([5](https://arxiv.org/html/2607.28670#S3.E5)\) across the groups in an input\. Conditional on the frozen features and a fixed pairing,YYhas a Gaussian densitypϕ​\(Y∣x\)p\_\{\\phi\}\(Y\\mid x\)whose within\-group correlations are set byρg=ρmax​σ​\(ϕ​\(stopgrad⁡\(sg\)\)\)\\rho\_\{g\}=\\rho\_\{\\max\}\\sigma\(\\phi\(\\operatorname\{stopgrad\}\(s\_\{g\}\)\)\)and whose paired cross\-group correlations are set by \([4](https://arxiv.org/html/2607.28670#S3.E4)\)\. Thestopgrad⁡\(⋅\)\\operatorname\{stopgrad\}\(\\cdot\)operation guarantees that the controller’s inputs provide no gradient path into the backbone or router, so the routing\-only property would survive even if the base parameters were trainable\. WithΘbase\\Theta\_\{\\mathrm\{base\}\}frozen, we optimize onlyϕ\\phi:

minϕ⁡𝔼\(x,y~\),Y∼pϕ​\(Y∣x\)​\[ℒ​\(FΘbase​\(x;Y\),y~\)\]\.\\min\_\{\\phi\}\\;\\mathbb\{E\}\_\{\(x,\\tilde\{y\}\),\\,Y\\sim p\_\{\\phi\}\(Y\\mid x\)\}\\left\[\\mathcal\{L\}\\big\(F\_\{\\Theta\_\{\\mathrm\{base\}\}\}\(x;Y\),\\tilde\{y\}\\big\)\\right\]\.\(9\)Herey~\\tilde\{y\}denotes the target, avoiding a collision with the pre\-Gumbel normal variableYY\. The discrete Top\-KKmap prevents ordinary pathwise differentiation through the selected set: a tiny change in a noise value usually changes no selected expert, then abruptly changes the selected set at a ranking boundary\. Instead, forL=ℒ​\(FΘbase​\(x;Y\),y~\)L=\\mathcal\{L\}\(F\_\{\\Theta\_\{\\mathrm\{base\}\}\}\(x;Y\),\\tilde\{y\}\), a score\-function estimator\(Williams,[1992](https://arxiv.org/html/2607.28670#bib.bib7)\)uses

∇ϕ𝔼\[L∣x\]=𝔼\[\(L−b\(x\)\)∇ϕlogpϕ\(Y∣x\)\|x\],\\nabla\_\{\\phi\}\\mathbb\{E\}\[L\\mid x\]=\\mathbb\{E\}\\\!\\left\[\\bigl\(L\-b\(x\)\\bigr\)\\nabla\_\{\\phi\}\\log p\_\{\\phi\}\(Y\\mid x\)\\,\\middle\|\\,x\\right\],\(10\)whereb​\(x\)b\(x\)is any stop\-gradient baseline that does not depend on the realized routing noise, for example a moving\-average or leave\-one\-out baseline\. In the score term,YYis treated as the sampled observation; the gradient differentiates the closed\-form Gaussian log\-density with respect to the controller parameters\. Thus the base model is evaluated only in the forward direction, while the backward pass updates onlyϕ\\phi\. No gradient through the discrete Top\-KKoperation is assumed or required\.

#### Score\-function estimator\.

Rather than differentiating through the discrete selection, the estimator reweights each sampled noise configuration by its centered loss and adjusts the controller to raise or lower the probability of that configuration accordingly\. The term∇ϕlog⁡pϕ​\(Y∣x\)\\nabla\_\{\\phi\}\\log p\_\{\\phi\}\(Y\\mid x\)measures the sensitivity of the sampled correlated noise’s log\-density to the controller’s coupling strength, and the baselineb​\(x\)b\(x\)subtracts a reference loss to reduce the variance of the update\. The frozen base model contributes only the forward loss, and all trainable gradients terminate at the controller\.

#### The density for the hierarchical sampler\.

For an unpaired group ofmmtokens and one expert coordinate, the covariance is\(1−ρg\)​Im\+ρg​𝟏𝟏𝖳\(1\-\\rho\_\{g\}\)I\_\{m\}\+\\rho\_\{g\}\\mathbf\{1\}\\mathbf\{1\}^\{\\mathsf\{T\}\}\. For an equal\-size paired group\(g,g′\)\(g,g^\{\\prime\}\), the corresponding2​m2m\-dimensional covariance is

Σg,g′=\[\(1−ρg\)​Im\+ρg​𝟏𝟏𝖳−αg,g′​ρg​ρg′​11𝖳−αg,g′​ρg​ρg′​11𝖳\(1−ρg′\)​Im\+ρg′​𝟏𝟏𝖳\]\.\\Sigma\_\{g,g^\{\\prime\}\}=\\begin\{bmatrix\}\(1\-\\rho\_\{g\}\)I\_\{m\}\+\\rho\_\{g\}\\mathbf\{1\}\\mathbf\{1\}^\{\\mathsf\{T\}\}&\-\\alpha\_\{g,g^\{\\prime\}\}\\sqrt\{\\rho\_\{g\}\\rho\_\{g^\{\\prime\}\}\}\\,\\mathbf\{1\}\\mathbf\{1\}^\{\\mathsf\{T\}\}\\\\ \-\\alpha\_\{g,g^\{\\prime\}\}\\sqrt\{\\rho\_\{g\}\\rho\_\{g^\{\\prime\}\}\}\\,\\mathbf\{1\}\\mathbf\{1\}^\{\\mathsf\{T\}\}&\(1\-\\rho\_\{g^\{\\prime\}\}\)I\_\{m\}\+\\rho\_\{g^\{\\prime\}\}\\mathbf\{1\}\\mathbf\{1\}^\{\\mathsf\{T\}\}\\end\{bmatrix\}\.\(11\)The full densitypϕp\_\{\\phi\}is the product of these Gaussian blocks across expert coordinates and disjoint groups or pairs\. It is nonsingular becauseρmax<1\\rho\_\{\\max\}<1\. Thus \([10](https://arxiv.org/html/2607.28670#S3.E10)\) applies to both the flat and tunable paired variants when the relevant block log\-density is used\. Pairing itself remains fixed before sampling; learning a discrete pairing would require a separate discrete gradient estimator\.

Readers need not use this covariance expression to understand or run the fixed\-coupling router\. It is included to make the learning claim precise: when paired groups are present, their noises must be scored jointly rather than as two independent groups\. Atαg,g′=0\\alpha\_\{g,g^\{\\prime\}\}=0the off\-diagonal blocks vanish and the density reduces to independent group blocks; atαg,g′=1\\alpha\_\{g,g^\{\\prime\}\}=1it is the fully antithetic density\.

#### The baseline used in the pilot\.

Equation \([10](https://arxiv.org/html/2607.28670#S3.E10)\) is exactly unbiased for a baseline that does not depend on the scored sample’s routing noise\. The small pilot uses the batch\-mean loss as a practical baseline\. Because that mean includes the scored sample, its expectation is a\(1−1/B\)\(1\-1/B\)\-scaled version of the score\-function gradient for batch sizeBB; a leave\-one\-out baseline would remove this finite\-batch scaling\. This choice affects optimization efficiency, not the routing\-law theorem\.

A biased continuous Top\-KKrelaxation may instead be used during optimization, while evaluating the exact discrete rule\. The routing\-law results concern the exact execution distribution, not a training relaxation\. The controller may set both within\-group strengthsρg\\rho\_\{g\}and paired\-group opposition strengthsαg,g′\\alpha\_\{g,g^\{\\prime\}\}\.

#### Where a learning signal can arise\.

Marginal preservation means that an objective depending only on one token’s isolated routing distribution cannot identifyϕ\\phi: that distribution does not change\. Any useful signal must arise from joint effects, for example several routed positions interacting through later attention, multiple MoE layers compounding, or an explicitly joint routing objective\. Consequently, ordinary per\-token cross\-entropy can provide a weak or noisy signal in a fully frozen model\. This is a prediction of the construction, not a failure of the estimator, and motivates the cautious pilot in Section[5](https://arxiv.org/html/2607.28670#S5)\.

## 4Related Work

#### MoE routing\.

Sparse MoE systems commonly use top\-KKselection or capacity\-aware assignment\(Shazeeret al\.,[2017](https://arxiv.org/html/2607.28670#bib.bib1); Lepikhinet al\.,[2021](https://arxiv.org/html/2607.28670#bib.bib2); Feduset al\.,[2022](https://arxiv.org/html/2607.28670#bib.bib3); Zhouet al\.,[2022](https://arxiv.org/html/2607.28670#bib.bib4); Lewiset al\.,[2021](https://arxiv.org/html/2607.28670#bib.bib5)\)\. SeqTopK redistributes a sequence\-level expert budget\(Wenet al\.,[2025](https://arxiv.org/html/2607.28670#bib.bib12)\); it changes the number of experts assigned to individual tokens\. Similarity\-aware routers modify gates or selections using token relationships\(Nguyenet al\.,[2025](https://arxiv.org/html/2607.28670#bib.bib10); Omiet al\.,[2025](https://arxiv.org/html/2607.28670#bib.bib11)\)\.H\-CGAaddresses a different degree of freedom: it holds each token’s stochastic Top\-KKrouting law fixed and modifies only joint assignments across tokens, in either direction\.

#### Routing\-aware PEFT\.

LoRA\(Huet al\.,[2022](https://arxiv.org/html/2607.28670#bib.bib6)\)and related PEFT methods adapt weights while freezing most of a backbone\. Routed PEFT learns or reuses routing to activate adapter modules\(Liuet al\.,[2026](https://arxiv.org/html/2607.28670#bib.bib9)\)\. These approaches allocate trainable*representational*capacity\. The controller of Section[3\.5](https://arxiv.org/html/2607.28670#S3.SS5)instead adds no expert\-weight adapter: it is a small trainable map into the dependence structure of a frozen base router’s sampling noise\.

#### Dependent discrete sampling and variance reduction\.

The Gumbel\-Top\-KKtrick samples a ranked set without replacement\(Koolet al\.,[2019](https://arxiv.org/html/2607.28670#bib.bib8)\);Huijbenet al\.\([2023](https://arxiv.org/html/2607.28670#bib.bib18)\)survey structured extensions of the Gumbel\-max mechanism\. Copulas separate joint dependence from marginals\(Sklar,[1959](https://arxiv.org/html/2607.28670#bib.bib13); Nelsen,[2006](https://arxiv.org/html/2607.28670#bib.bib14)\)\. Antithetic variates are a classical variance\-reduction device\(Hammersley and Morton,[1956](https://arxiv.org/html/2607.28670#bib.bib16)\), and antithetic constructions also underlie low\-variance gradient estimators for discrete variables\(Yin and Zhou,[2019](https://arxiv.org/html/2607.28670#bib.bib17)\); the association inequality ofEsaryet al\.\([1967](https://arxiv.org/html/2607.28670#bib.bib15)\)supplies the monotonicity argument we use\. Our construction combines these elements at the routing\-noise level of an MoE: coordinate\-wise copulas induce cross\-token dependence of either sign while retaining the i\.i\.d\. Gumbel vector required by each token’s original Gumbel\-Top\-KKlaw\.

## 5Initial Frozen\-Base Pilot

This pilot is deliberately small\. Its purpose is to test whether the exact sampler, the invariance checks, and the controller\-only training route work together\. It is*not*evidence that routing\-only adaptation improves a pretrained MoE on a downstream task\.

#### Setup\.

We trained one 15\.8M\-parameter, six\-layer decoder\-only Top\-2 MoE language model on 10M TinyStories tokens\. It has eight experts in each of three MoE layers and processes sequences of length 256\. Groups are fixed, non\-overlapping windows ofm=4m=4adjacent tokens; within\-window Jaccard means the average Top\-2 set Jaccard similarity of adjacent positions inside those windows\. We then froze every base parameter and evaluated the copula mechanism on the held\-out validation split\. The main fixed\-coupling comparison usesρ=0\.6\\rho=0\.6andα=0\\alpha=0, so the reported full\-model rows measure lower\-level positive coupling only\.

The learned\-controller experiment is intentionally a restricted special case of Section[3\.5](https://arxiv.org/html/2607.28670#S3.SS5): it learns one constantρ\\rhoper MoE layer \(three parameters total\), rather than a feature\-conditioned, per\-group mapϕ​\(sg\)\\phi\(s\_\{g\}\)\. It trains for 2M additional in\-distribution tokens using the score\-function route in \([10](https://arxiv.org/html/2607.28670#S3.E10)\),ρmax=0\.95\\rho\_\{\\max\}=0\.95, and three seeds\. The learned\-controller and table rows useα=0\\alpha=0; the higher\-level dial is evaluated separately below with fixed coupling only, so no run in this pilot measures a learnedα\\alphaor an end\-to\-end variance\-reduction benefit\.

#### Routing\-law checks\.

In a separate synthetic fixed\-logit test with four tokens, 60,000 draws, andρ=0\.8\\rho=0\.8, the largest difference between the empirical frequency of an ordered Top\-2 list under independent and copula routing was0\.00350\.0035; the largest difference in an expert\-inclusion frequency was2\.2×10−42\.2\\times 10^\{\-4\}\. These are Monte\-Carlo checks, not proofs; Theorem[1](https://arxiv.org/html/2607.28670#Thmtheorem1)gives the exact conditional result\. In this deliberately similar\-logit synthetic setting, adjacent\-token Top\-2 Jaccard overlap rose from0\.530\.53to0\.770\.77\. These values are not directly comparable with the full\-model metrics in Table[1](https://arxiv.org/html/2607.28670#S5.T1)\.

We also test the new higher\-level dial on identical synthetic logits, which isolate the shared\-noise effect from gate differences\. Atρ=0\.6\\rho=0\.6, paired\-boundary Jaccard overlap decreases from0\.3890\.389atα=0\\alpha=0to0\.3030\.303atα=0\.5\\alpha=0\.5and0\.2180\.218atα=1\\alpha=1, while within\-window overlap remains approximately constant \(0\.6010\.601,0\.6010\.601, and0\.6000\.600\)\. A separateα=1\\alpha=1law check gives maximum ordered\-list and inclusion\-frequency deviations of0\.00280\.0028and0\.00340\.0034, respectively\. These synthetic checks validate thatα\\alphachanges the intended cross\-group statistic without altering the observed per\-token law; they are not end\-to\-end evidence of load balancing\.

The same pattern holds end to end on the frozen model\. A single\-seed evaluation\-only sweep overρ∈\{0\.6,0\.9\}\\rho\\in\\\{0\.6,0\.9\\\}andα∈\{0,0\.5,1\}\\alpha\\in\\\{0,0\.5,1\\\}lowers paired\-window\-boundary Jaccard from0\.2100\.210to0\.1700\.170to0\.1370\.137atρ=0\.6\\rho=0\.6, and from0\.2100\.210to0\.1520\.152to0\.1100\.110atρ=0\.9\\rho=0\.9, while within\-window Jaccard is unchanged to within3×10−43\\times 10^\{\-4\}\(0\.3140\.314and0\.3890\.389, respectively\) and validation cross\-entropy varies by at most3\.1×10−43\.1\\times 10^\{\-4\}across all six cells\. Theα\\alphadial therefore moves only its target statistic in a full multi\-layer model as well\. Aggregate load CV again changes very little, and, as above, that summary does not test Proposition[1](https://arxiv.org/html/2607.28670#Thmproposition1)\.

Table 1:Initial frozen\-base pilot\. Values after±\\pmare standard deviations over three seeds; the two fixed rows are one mechanism\-check run each\. The pilot is not a task\-adaptation benchmark\.
#### Findings and limits\.

Fixed positive coupling substantially changes the*joint*routing statistics: it raises within\-window overlap and reduces the number of distinct experts used in a window\. The observed validation cross\-entropy difference is small in this one\-seed check, but layer\-local routing\-law invariance does not predict sequence\-level cross\-entropy invariance\. This is mechanism evidence, not a quality\-improvement claim\. The observed aggregate load CV also changes very little\. It is a finite\-sample summary across experts, rather than a direct estimate of the conditional variance in Proposition[1](https://arxiv.org/html/2607.28670#Thmproposition1), so it does not test that proposition’s signed comparison\.

The three\-scalar controller trains stably and preserves aggregate inclusion frequencies to within2\.4×10−42\.4\\times 10^\{\-4\}of the independent estimate in this evaluation\. This is an empirical observation, not an end\-to\-end invariance guarantee\. Its learned strengths range from0\.040\.04to0\.270\.27without a reproducible pattern, and it provides no validation cross\-entropy gain\. The much smaller Jaccard change than the fixed\-ρ\\rhorow is consistent with the learned strengths staying well below0\.60\.6and with the weak joint signal discussed above\. Router\-LoRA attains lower cross\-entropy, but it has 528 times as many trainable parameters, optimizes the base balance regularizer in addition to cross\-entropy, and changes the routing distribution itself: its maximum observed inclusion\-frequency shift is5\.8×10−35\.8\\times 10^\{\-3\}\. It is therefore an illustrative conventional routing\-adaptation reference, not a matched\-budget or matched\-objective control\.

The pilot leaves the central empirical questions open: whether learned dependence helps on a real domain shift, whether anα\\alphasweep reduces capacity overflows in practice, and whether fewer distinct experts translate into measurable hardware savings\. A paper\-level evaluation should answer these questions on a pretrained stochastic Top\-KKMoE with capacity\-aware measurements\.

## 6Discussion and Limitations

H\-CGAis a dependence layer for stochastic routers, not a universal drop\-in replacement for every MoE\. It is exactly plug\-compatible only with a stochastic Gumbel\-Top\-KKbase router; converting a deterministic pretrained Top\-KKrouter to stochastic Gumbel\-Top\-KKchanges its base behavior even at zero coupling\. Capacity clipping, token dropping, and expert\-choice allocation act after sampling and are outside the invariance results; under a hard capacity, the burstiness signed by Proposition[1](https://arxiv.org/html/2607.28670#Thmproposition1)\(i\) is exactly the quantity that causes overflow, which is one motivation for the higher\-level opposition dial\. The proposition signs but does not quantify either variance effect, and for heterogeneous gate distributions the magnitudes may be small\. The initial pilot in Section[5](https://arxiv.org/html/2607.28670#S5)checks the positive\-coupling mechanism and theα\\alphadial’s routing statistics, both synthetically and end to end on the frozen model, but it does not measure conditional load variance, capacity overflows, or a systems benefit\. Finally, the method requiresEEGumbel perturbations per token; although routers usually already score all experts, the sampling and cross\-token coordination overhead must be measured\. Marginal preservation is a safety and identifiability property, not an accuracy theorem: it does not by itself imply better task performance, communication, specialization, or realized load balance\.

In summary, holding every token’s Top\-KKrouting law fixed leaves a usable design space—the joint dependence of routing across tokens\. A hierarchical copula reaches both poles of this space, coherence within groups and dispersion across them, with provable invariance of every per\-token routing quantity and a signed characterization of the load\-variance trade\-off, and it exposes these controls to a small controller trainable without updating the frozen base\. These guarantees are conditional and routing\-layer\-local, and do not by themselves render a multi\-layer MoE end\-to\-end invariant\.

## Appendix: Proof Details

#### Independence across expert coordinates\.

Independence across expert coordinates is essential in Theorem[1](https://arxiv.org/html/2607.28670#Thmtheorem1)\. Correlatingγt​e\\gamma\_\{te\}andγt​e′\\gamma\_\{te^\{\\prime\}\}within a token would alter the Gumbel\-Top\-KKranking law and invalidate the theorem\. Similarly, choosing group membership, the pairing,ρg\\rho\_\{g\}, orαg,g′\\alpha\_\{g,g^\{\\prime\}\}after observing routing noise can select on the random variables and need not preserve their uniform/Gumbel margins; the controller and pairing must be measurable with respect to frozen, pre\-noise quantities only\. The paired\-latent rule \([8](https://arxiv.org/html/2607.28670#S3.E8)\) must likewise act coordinate\-wise: couplingζg′​e\\zeta\_\{g^\{\\prime\}e\}toζg​e′\\zeta\_\{ge^\{\\prime\}\}fore≠e′e\\neq e^\{\\prime\}would couple expert coordinates within a token and break the theorem\.

#### Proof of Proposition[1](https://arxiv.org/html/2607.28670#Thmproposition1)\.

Fix experteeand condition throughout on logits, groups, pairings, and strengths\. For a groupgg, collect its latents intoζg=\(ζg​1,…,ζg​E\)\\zeta\_\{g\}=\(\\zeta\_\{g1\},\\ldots,\\zeta\_\{gE\}\)and define the transformed vectorWgW\_\{g\}byWg​e=ζg​eW\_\{ge\}=\\zeta\_\{ge\}andWg​e′=−ζg​e′W\_\{ge^\{\\prime\}\}=\-\\zeta\_\{ge^\{\\prime\}\}fore′≠ee^\{\\prime\}\\neq e\. The coordinates ofWgW\_\{g\}are independent symmetric standard normals\. For a tokent∈gt\\in g, the indicator𝟏​\[e∈St\]\\mathbf\{1\}\[e\\in S\_\{t\}\]is, for fixed values of all other noise, nondecreasing in the perturbed logitℓt​e\+γt​e\\ell\_\{te\}\+\\gamma\_\{te\}and nonincreasing in each competitor’s perturbed logit; sinceγt​e\\gamma\_\{te\}is increasing inyt​ey\_\{te\}, which is increasing inζg​e\\zeta\_\{ge\}, and each competitor coordinate is increasing inζg​e′\\zeta\_\{ge^\{\\prime\}\}, the conditional meanφt​\(Wg\)=ℙ​\(e∈St∣Wg\)\\varphi\_\{t\}\(W\_\{g\}\)=\\mathbb\{P\}\(e\\in S\_\{t\}\\mid W\_\{g\}\)\(integrating out the token noisesϵ\\epsilon\) is nondecreasing in every coordinate ofWgW\_\{g\}\. GivenWgW\_\{g\}, the indicators of distinct tokens inggare conditionally independent, because they involve disjoint sets ofϵ\\epsilondraws\.

\(i\) Fort≠t′t\\neq t^\{\\prime\}ingg,Cov⁡\(𝟏​\[e∈St\],𝟏​\[e∈St′\]\)=Cov⁡\(φt​\(Wg\),φt′​\(Wg\)\)≥0\\operatorname\{Cov\}\(\\mathbf\{1\}\[e\\in S\_\{t\}\],\\mathbf\{1\}\[e\\in S\_\{t^\{\\prime\}\}\]\)=\\operatorname\{Cov\}\(\\varphi\_\{t\}\(W\_\{g\}\),\\varphi\_\{t^\{\\prime\}\}\(W\_\{g\}\)\)\\geq 0, since independent coordinates are associated and both functions are coordinatewise nondecreasing\(Esaryet al\.,[1967](https://arxiv.org/html/2607.28670#bib.bib15)\)\. HenceVar⁡\(Xg\)≥∑t∈gVar⁡\(𝟏​\[e∈St\]\)\\operatorname\{Var\}\(X\_\{g\}\)\\geq\\sum\_\{t\\in g\}\\operatorname\{Var\}\(\\mathbf\{1\}\[e\\in S\_\{t\}\]\), and the individual variances equal their independent\-routing values by Theorem[1](https://arxiv.org/html/2607.28670#Thmtheorem1)\. Groups are independent under flat coupling, so summing overggproves \(i\)\.

\(ii\) Under the paired rule \([8](https://arxiv.org/html/2607.28670#S3.E8)\), letVg′V\_\{g^\{\\prime\}\}be the transformed independent standard\-normal vector formed fromηg′​e\\eta\_\{g^\{\\prime\}e\}\.

Wg′=−αg,g′​Wg\+1−αg,g′2​Vg′\.W\_\{g^\{\\prime\}\}=\-\\alpha\_\{g,g^\{\\prime\}\}W\_\{g\}\+\\sqrt\{1\-\\alpha\_\{g,g^\{\\prime\}\}^\{2\}\}\\,V\_\{g^\{\\prime\}\}\.Given\(Wg,Vg′\)\(W\_\{g\},V\_\{g^\{\\prime\}\}\), the countsXgX\_\{g\}andXg′X\_\{g^\{\\prime\}\}are conditionally independent because theirϵ\\epsilondraws are disjoint\. Their conditional means areΦg​\(Wg\)=∑t∈gφt​\(Wg\)\\Phi\_\{g\}\(W\_\{g\}\)=\\sum\_\{t\\in g\}\\varphi\_\{t\}\(W\_\{g\}\), which is coordinatewise nondecreasing, andΦg′​\(−αg,g′​Wg\+1−αg,g′2​Vg′\)\\Phi\_\{g^\{\\prime\}\}\(\-\\alpha\_\{g,g^\{\\prime\}\}W\_\{g\}\+\\sqrt\{1\-\\alpha\_\{g,g^\{\\prime\}\}^\{2\}\}V\_\{g^\{\\prime\}\}\)\. Averaging the latter overVg′V\_\{g^\{\\prime\}\}gives a function ofWgW\_\{g\}that is coordinatewise nonincreasing forαg,g′≥0\\alpha\_\{g,g^\{\\prime\}\}\\geq 0\. Therefore

Cov⁡\(Xg,Xg′\)\\displaystyle\\operatorname\{Cov\}\(X\_\{g\},X\_\{g^\{\\prime\}\}\)=Cov⁡\(Φg​\(Wg\),𝔼Vg′​\[Φg′​\(−αg,g′​Wg\+1−αg,g′2​Vg′\)∣Wg\]\)\\displaystyle=\\operatorname\{Cov\}\\\!\\left\(\\Phi\_\{g\}\(W\_\{g\}\),\\mathbb\{E\}\_\{V\_\{g^\{\\prime\}\}\}\[\\Phi\_\{g^\{\\prime\}\}\(\-\\alpha\_\{g,g^\{\\prime\}\}W\_\{g\}\+\\sqrt\{1\-\\alpha\_\{g,g^\{\\prime\}\}^\{2\}\}V\_\{g^\{\\prime\}\}\)\\mid W\_\{g\}\]\\right\)≤0\\displaystyle\\leq 0by the same association inequality applied to a coordinatewise nondecreasing and a coordinatewise nonincreasing function\. The marginal law of each group is unchanged by the pairing, distinct pairs are independent, and unpaired covariances vanish, soVar⁡\(Ne\)=∑gVar⁡\(Xg\)\+2​∑\(g,g′\)Cov⁡\(Xg,Xg′\)\\operatorname\{Var\}\(N\_\{e\}\)=\\sum\_\{g\}\\operatorname\{Var\}\(X\_\{g\}\)\+2\\sum\_\{\(g,g^\{\\prime\}\)\}\\operatorname\{Cov\}\(X\_\{g\},X\_\{g^\{\\prime\}\}\)is at most its flat\-coupling value\. Expected loads agree in all schemes by Corollaries[1](https://arxiv.org/html/2607.28670#Thmcorollary1)and[2](https://arxiv.org/html/2607.28670#Thmcorollary2), conditional on the fixed logits in this proof\. ∎

## References

- Association of random variables, with applications\.The Annals of Mathematical Statistics38\(5\),pp\. 1466–1474\.Cited by:[§3\.4](https://arxiv.org/html/2607.28670#S3.SS4.SSS0.Px1.2.p1.1),[§4](https://arxiv.org/html/2607.28670#S4.SS0.SSS0.Px3.p1.2),[Proof of Proposition1\.](https://arxiv.org/html/2607.28670#Sx1.SS0.SSS0.Px2.p2.5)\.
- W\. Fedus, B\. Zoph, and N\. Shazeer \(2022\)Switch transformers: scaling to trillion parameter models with simple and efficient sparsity\.Journal of Machine Learning Research23\(120\),pp\. 1–39\.Cited by:[§1](https://arxiv.org/html/2607.28670#S1.p1.3),[§4](https://arxiv.org/html/2607.28670#S4.SS0.SSS0.Px1.p1.2)\.
- J\. M\. Hammersley and K\. W\. Morton \(1956\)A new Monte Carlo technique: antithetic variates\.Mathematical Proceedings of the Cambridge Philosophical Society52\(3\),pp\. 449–475\.Cited by:[§4](https://arxiv.org/html/2607.28670#S4.SS0.SSS0.Px3.p1.2)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.28670#S2.SS0.SSS0.Px4.p1.2),[§4](https://arxiv.org/html/2607.28670#S4.SS0.SSS0.Px2.p1.1)\.
- I\. A\. M\. Huijben, W\. Kool, M\. B\. Paulus, and R\. J\. G\. van Sloun \(2023\)A review of the Gumbel\-max trick and its extensions for discrete stochasticity in machine learning\.IEEE Transactions on Pattern Analysis and Machine Intelligence45\(2\),pp\. 1353–1371\.Cited by:[§4](https://arxiv.org/html/2607.28670#S4.SS0.SSS0.Px3.p1.2)\.
- W\. Kool, H\. van Hoof, and M\. Welling \(2019\)Stochastic beams and where to find them: the gumbel\-top\-kktrick for sampling sequences without replacement\.InInternational Conference on Machine Learning,pp\. 3499–3508\.Cited by:[§1](https://arxiv.org/html/2607.28670#S1.p1.3),[§2](https://arxiv.org/html/2607.28670#S2.SS0.SSS0.Px1.p1.7),[§4](https://arxiv.org/html/2607.28670#S4.SS0.SSS0.Px3.p1.2)\.
- D\. Lepikhin, H\. Lee, Y\. Xu, D\. Chen, O\. Firat, Y\. Huang, M\. Krikun, N\. Shazeer, and Z\. Chen \(2021\)GShard: scaling giant models with conditional computation and automatic sharding\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.28670#S1.p1.3),[§4](https://arxiv.org/html/2607.28670#S4.SS0.SSS0.Px1.p1.2)\.
- M\. Lewis, S\. Bhosale, T\. Dettmers, N\. Goyal, and L\. Zettlemoyer \(2021\)BASE layers: simplifying training of large, sparse models\.InInternational Conference on Machine Learning,pp\. 6265–6274\.Cited by:[§4](https://arxiv.org/html/2607.28670#S4.SS0.SSS0.Px1.p1.2)\.
- Y\. Liu, Y\. Ma, Y\. Lu, S\. Chen, Z\. Ding, and V\. Tresp \(2026\)Parameter\-efficient routed fine\-tuning: mixture\-of\-experts demands mixture of adaptation modules\.InFindings of the Association for Computational Linguistics: EACL 2026,pp\. 4439–4457\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.findings-eacl.232)Cited by:[§2](https://arxiv.org/html/2607.28670#S2.SS0.SSS0.Px4.p1.2),[§4](https://arxiv.org/html/2607.28670#S4.SS0.SSS0.Px2.p1.1)\.
- R\. B\. Nelsen \(2006\)An introduction to copulas\.Springer\.Cited by:[§1](https://arxiv.org/html/2607.28670#S1.p2.1),[§4](https://arxiv.org/html/2607.28670#S4.SS0.SSS0.Px3.p1.2)\.
- T\. Nguyen, N\. N\. Tran, K\. Nguyen, and R\. G\. Baraniuk \(2025\)Improving routing in sparse mixture of experts with graph of tokens\.arXiv preprint arXiv:2505\.00792\.Cited by:[§1](https://arxiv.org/html/2607.28670#S1.p1.3),[§4](https://arxiv.org/html/2607.28670#S4.SS0.SSS0.Px1.p1.2)\.
- N\. Omi, S\. Sen, and A\. Farhadi \(2025\)Load balancing mixture of experts with similarity preserving routers\.arXiv preprint arXiv:2506\.14038\.Cited by:[§1](https://arxiv.org/html/2607.28670#S1.p1.3),[§4](https://arxiv.org/html/2607.28670#S4.SS0.SSS0.Px1.p1.2)\.
- N\. Shazeer, A\. Mirhoseini, K\. Maziarz, A\. Davis, Q\. Le, G\. Hinton, and J\. Dean \(2017\)Outrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.28670#S1.p1.3),[§4](https://arxiv.org/html/2607.28670#S4.SS0.SSS0.Px1.p1.2)\.
- A\. Sklar \(1959\)Fonctions de répartition ànndimensions et leurs marges\.Publications de l’Institut de Statistique de l’Université de Paris8,pp\. 229–231\.Cited by:[§1](https://arxiv.org/html/2607.28670#S1.p2.1),[§4](https://arxiv.org/html/2607.28670#S4.SS0.SSS0.Px3.p1.2)\.
- T\. Wen, Y\. Wang, A\. Feng, L\. Ma, X\. Liu, Y\. Wang, L\. Guo, B\. Chen, S\. Jegelka, and C\. You \(2025\)Route experts by sequence, not by token\.arXiv preprint arXiv:2511\.06494\.Cited by:[§4](https://arxiv.org/html/2607.28670#S4.SS0.SSS0.Px1.p1.2)\.
- R\. J\. Williams \(1992\)Simple statistical gradient\-following algorithms for connectionist reinforcement learning\.Machine Learning8\(3–4\),pp\. 229–256\.Cited by:[§3\.5](https://arxiv.org/html/2607.28670#S3.SS5.p2.12)\.
- M\. Yin and M\. Zhou \(2019\)ARM: augment\-REINFORCE\-merge gradient for stochastic binary networks\.InInternational Conference on Learning Representations,Cited by:[§4](https://arxiv.org/html/2607.28670#S4.SS0.SSS0.Px3.p1.2)\.
- Y\. Zhou, T\. Lei, H\. Liu, N\. Du, Y\. Huang, V\. Zhao, A\. M\. Dai, Z\. Chen, Q\. V\. Le, and J\. Laudon \(2022\)Mixture\-of\-experts with expert choice routing\.InAdvances in Neural Information Processing Systems,Cited by:[§4](https://arxiv.org/html/2607.28670#S4.SS0.SSS0.Px1.p1.2)\.