Is MoE Routing a Huffman Code? Discovering the Frequency-Diversity Law in Chain-of-Thought

# Is MoE Routing a Huffman Code? Discovering the Frequency-Diversity Law in Chain-of-Thought
Source: [https://arxiv.org/html/2607.20427](https://arxiv.org/html/2607.20427)
###### Abstract

Mixture\-of\-Experts architectures have revolutionized scaling, yet the underlying logic of their routing remains a black box\. In this paper, we uncover a fundamental governing principle:*MoE routing is not merely selection, but a manifestation of Huffman Coding\.*We introduce theFrequency\-Diversity Law, revealing that state\-of\-the\-art models, such as Phi\-3\.5\-MoE, Gemma\-4\-27B\-A4B, spontaneously act as information\-theoretic engines\. These models allocate sparse expert resources for common tokens while invoking high\-diversity expert committees for rare, complex tasks found in chain\-of\-thought trajectories\. However, we identify a critical redundancy trap in Qwen3\.5\-35B\-A3B: when effective sparsity \(k/Eeffk/E\_\{\\text\{eff\}\}\) is sufficiently low, load\-balancing inadvertently imposes functional redundancy, masking the underlying Huffman efficiency signal\. To bridge this gap, we proposeSubset Difference Pruning, a surgical strategy to eliminate functional duplicates\. We demonstrate that pruning does not degrade reasoning; instead, it unleashes the model’s latent Huffman efficiency, forcing the logic to collapse into streamlined, high\-density paths\. Our findings suggest that the next generation of MoEs should move beyond forced load\-balancing towardMinimum Description Length \(MDL\) optimality, assigning shorter expert\-routing codes to high\-frequency information and longer, more diverse codes to low\-frequency information, thereby transforming routing from a heuristic into a principled compression engine\.

Mixture\-of\-Expert, Model Reasoning

## 1Introduction

Mixture\-of\-Experts \(MoE\) architectures have emerged as a dominant paradigm for scaling Large Language Models \(LLMs\)\(Shazeeret al\.,[2017](https://arxiv.org/html/2607.20427#bib.bib34); Jianget al\.,[2024](https://arxiv.org/html/2607.20427#bib.bib2); Daiet al\.,[2024](https://arxiv.org/html/2607.20427#bib.bib38)\), primarily due to their ability to decouple model capacity from computational cost\. At the heart of this efficiency is the Gating Network \(Router\), which dynamically selects a small subset of experts to process each input token\(Feduset al\.,[2022](https://arxiv.org/html/2607.20427#bib.bib37)\)\. Fundamentally, this process is an exercise inresource allocation\(Clarket al\.,[2022](https://arxiv.org/html/2607.20427#bib.bib39)\): the model must decide, in real\-time, how to distribute its limited*active*parameters to best represent the vast complexity of world knowledge\.

This mechanism of selective activation mirrors the core principles ofInformation Theory, specificallySource Coding\(Shannon,[1948](https://arxiv.org/html/2607.20427#bib.bib33)\)\. Huffman Coding\(Huffman,[1952](https://arxiv.org/html/2607.20427#bib.bib32)\), one of the cornerstone of this field, serves as the benchmark for data compression and communication efficiency\. Its essence lies in allocating resources based on symbol probability: assigning shorter codes to high\-frequency information and longer codes to low\-frequency information\. If an LLM is viewed as a predictive system aimed at the “ultimate compression” of world knowledge\(Hutter,[2005](https://arxiv.org/html/2607.20427#bib.bib35)\), its MoE reasoning trajectory should not be random; rather, it should reflect the frequency distribution of semantic logic, achieving an optimal balance between information entropy and physical resources\.\(Shannon,[1959](https://arxiv.org/html/2607.20427#bib.bib36); Shazeeret al\.,[2017](https://arxiv.org/html/2607.20427#bib.bib34)\)

Despite this natural connection, the potential alignment between MoE routing and Huffman coding principles remains relatively under\-explored\. While existing literature has extensively examined scaling laws\(Krajewskiet al\.,[2024](https://arxiv.org/html/2607.20427#bib.bib31)\), expert specialization\(Zhouet al\.,[2022](https://arxiv.org/html/2607.20427#bib.bib40); Xueet al\.,[2024](https://arxiv.org/html/2607.20427#bib.bib50)\), and routing stability\(Lepikhinet al\.,[2021](https://arxiv.org/html/2607.20427#bib.bib1)\), an information\-theoretic characterization of routing efficiency from the perspective of source coding has yet to be fully established\.

To verify our hypothesis that optimized MoEs naturally gravitate towards Huffman\-like efficiency, we conducted quantitative experiments on state\-of\-the\-art models such as Gemma\-4\-27B\-A4B and Phi\-3\.5\-MoE\. As illustrated in Figure[1](https://arxiv.org/html/2607.20427#S1.F1), we discovered a significant*Huffman Phenomenon*: the number of activated experts \(mean unique experts/layer\) exhibits an extremely strong positive correlation with task rarity \(−log⁡p​\(o\)\-\\log p\(o\)\), with a Spearman’sρ=1\.00\\rho=1\.00\. This confirms that driven by the minimization of prediction loss, these models have automatically optimized routing efficiency by utilizing a few core experts for general logic and invoking a more diverse combination of experts only when encountering rare and complex tasks\.

Nevertheless, the determination of expert numbers in current MoE designs remains largely empirical\(Zoph,[2022](https://arxiv.org/html/2607.20427#bib.bib17); Krajewskiet al\.,[2024](https://arxiv.org/html/2607.20427#bib.bib31)\)\. Modern training paradigms typically prioritize suppressing “expert collapse” by introducing auxiliary load\-balancing loss to ensure uniform communication across the expert pool\(Yanget al\.,[2025](https://arxiv.org/html/2607.20427#bib.bib4); Kamathet al\.,[2025](https://arxiv.org/html/2607.20427#bib.bib5); Feduset al\.,[2022](https://arxiv.org/html/2607.20427#bib.bib37); Omiet al\.,[2025](https://arxiv.org/html/2607.20427#bib.bib58)\)\. Our empirical analysis of Qwen3\.5\-35B\-A3B reveals that when expert sparsity \(k/Eeffk/E\_\{\\text\{eff\}\}\) is excessively small, these balancing pressures may inadvertently compel the model to overallocate gradients to underutilized parameters \(Table[4](https://arxiv.org/html/2607.20427#A4.T4)\)\. This manifests as a proliferation of*functional clones*\(Puigcerveret al\.,[2023](https://arxiv.org/html/2607.20427#bib.bib45)\), which are redundant experts with high semantic overlap that exist primarily to satisfy balancing objectives rather than to provide unique informational gain\.

This leads to a pivotal inquiry:*Is the conventional reliance on aggressive load\-balancing an optimal scaling strategy, or does it fundamentally obstruct the emergence of Huffman\-efficient routing?*We argue that routing essentially promotes compression:the expert activation pattern selected by a router functions as a compact code for the semantic operations unfolding in a CoT\(Weiet al\.,[2022](https://arxiv.org/html/2607.20427#bib.bib56)\)trace\. Motivated by this perspective, our investigation reveals that while models like Qwen3\.5\-35B\-A3B maintain high active expert counts, the resulting code is often inefficient due to functional redundancy\.

To test whether removing such redundancy can unleash latent Huffman efficiency, we introduceSubset Difference Pruning\(SDP\), a surgical strategy designed to refine the model’s routing topology\. Our findings indicate that strategically removing moderately redundant experts effectively reshapes the model’s reasoning path\. Experimental results demonstrate that despite trajectory shifts caused by pruning, the model exhibits a robust re\-encoding capability, collapsing logic into a more streamlined efficient path that aligns better with Huffman principles\.

The contribution of this research lies in:

1. 1\.Phenomenon:We identify theHuffman encoding phenomenonin MoE routing, revealing the temporal compression of expert patterns during CoT\.
2. 2\.Universality:We establish theFrequency\-Diversity Law, proving that frequency\-diversity correspondence in expert routing is a universal structural property of optimized MoE models rather than a model\-specific coincidence\.
3. 3\.Methodology:We proposeSubset Difference Pruningand identify the effective sparsity \(k/Eeffk/E\_\{\\text\{eff\}\}\) range for routing to operate as a genuine MDL\-optimal compression engine\.

Huffman test:u¯\(o\)∝−log⁡p​\(o\)\\bar\{u\}^\{\(o\)\}\\propto\-\\log p\(o\) ![Refer to caption](https://arxiv.org/html/2607.20427v1/figs/huffman_scatter.png)

Figure 1:Quantitative test of the Frequency\-Diversity Law \(Eq\. \([4](https://arxiv.org/html/2607.20427#S3.E4)\)\) on Gemma\-4\-27B\-A4B and Phi\-3\.5\-MoE\.Each point is one operation typeo∈𝒪o\\in\\mathcal\{O\};xx\-axis: rarity\(−log⁡p​\(o\)\)\(\{\-\}\\log p\(o\)\);yy\-axis: mean unique experts per layer \(u¯\(o\)\\bar\{u\}^\{\(o\)\}\); dashed line: OLS fit\. Both models achieve Spearmanρ=1\.00\\rho=1\.00\(p<0\.005p<0\.005\), confirmingu¯\(o\)∝−log⁡p​\(o\)\\bar\{u\}^\{\(o\)\}\\propto\{\-\}\\log p\(o\)\.
## 2Related Work

### 2\.1Sparse Mixture\-of\-Experts \(MoE\)

Sparse MoEs\(Shazeeret al\.,[2017](https://arxiv.org/html/2607.20427#bib.bib34)\)utilize top\-kkrouting to achieve high parameter counts with constant computational costs\. While early scaling focused on massive deployments\(Lepikhinet al\.,[2021](https://arxiv.org/html/2607.20427#bib.bib1); Feduset al\.,[2022](https://arxiv.org/html/2607.20427#bib.bib37)\), recent architectures emphasize refined expert granularity\.Yanget al\.\([2025](https://arxiv.org/html/2607.20427#bib.bib4)\)andKamathet al\.\([2025](https://arxiv.org/html/2607.20427#bib.bib5)\)introduced diverse expert configurations, such as shared\-expert isolation, to optimize utilization\. We analyze modern benchmarks, Phi\-3\.5\-MoE\(Abdinet al\.,[2024](https://arxiv.org/html/2607.20427#bib.bib3)\), Gemma\-4\-27B\-A4B\(Kamathet al\.,[2025](https://arxiv.org/html/2607.20427#bib.bib5)\), and Qwen3\.5\-35B\-A3B\(Yanget al\.,[2025](https://arxiv.org/html/2607.20427#bib.bib4)\), which represent distinct balances between expert pool size and routing sparsity \(k/E∈\{3\.1%,6\.25%,12\.5%\}k/E\\in\\\{3\.1\\%,6\.25\\%,12\.5\\%\\\}\), providing a spectrum to study information\-theoretic routing properties\.

### 2\.2Routing Interpretability and Dynamics

Interpretability research often treats MoE experts as specialized key\-value memories\(Gevaet al\.,[2021](https://arxiv.org/html/2607.20427#bib.bib20); Elhageet al\.,[2022](https://arxiv.org/html/2607.20427#bib.bib21)\)\. Previous studies explored whether experts encode semantic or syntactic features\(Artetxeet al\.,[2022](https://arxiv.org/html/2607.20427#bib.bib18); Herbstet al\.,[2026](https://arxiv.org/html/2607.20427#bib.bib47)\)but largely relied on static, single\-token snapshots\. Recent systematic analyses have begun to uncover macro\-structures in routing, such as layer\-wise specialization and “Language Routing Isolation,” where experts are partitioned by linguistic resource levels and families\(Chenet al\.,[2026](https://arxiv.org/html/2607.20427#bib.bib55); Zhenget al\.,[2026](https://arxiv.org/html/2607.20427#bib.bib54)\)\. However, these insights remain focused on the spatial distribution of experts across model depth\. Our work departs from this by analyzing thetemporal dynamicsof routing, specifically across CoT trajectories\. We bridge a critical gap by linking routing decisions to Huffman coding, demonstrating that routers spontaneously optimize for frequency\-dependent expert allocation when given adequate expert sparsity\.

### 2\.3Information\-Theoretic Compression

The Minimum Description Length \(MDL\) principle\(Rissanen,[1978](https://arxiv.org/html/2607.20427#bib.bib28)\)provides a unified framework for weight pruning\(Hanet al\.,[2015](https://arxiv.org/html/2607.20427#bib.bib25)\)and quantization\(Frantaret al\.,[2022](https://arxiv.org/html/2607.20427#bib.bib27)\)\. While LLMs are recognized as general\-purpose compressors\(Delétanget al\.,[2024](https://arxiv.org/html/2607.20427#bib.bib29)\), and concurrent work like RFID\-MoE\(Miet al\.,[2026](https://arxiv.org/html/2607.20427#bib.bib51)\)uses routing frequency for weight\-SVD compression, our approach is distinct\. We focus onrouting path description lengthrather than parameter weights\. By drawing on optimal source coding\(Cover and Thomas,[2006](https://arxiv.org/html/2607.20427#bib.bib30)\), we argue that inefficient models \(e\.g\., Qwen3\.5\-35B\-A3B\) suffer from functional redundancy where forced load\-balancing imposes a sub\-optimal prior on the routing distribution\. Our Subset Difference Pruning targets this “routing codebook” redundancy, recovering MDL\-optimality without further training\.

## 3Temporal Expert Combination Encoding: Information\-Theoretic Analysis

This section formalizes the claim that MoE routing implements Huffman coding\. We define expert activation patterns as combinatorial codewords, connect them to information theory via mutual information, and derive the*Frequency\-Diversity Law*as a falsifiable prediction\. We then show this law extends temporally along individual CoT traces\. Throughout, we adopt the thought\-step segmentation convention fromZhouet al\.\([2025](https://arxiv.org/html/2607.20427#bib.bib23)\), treating each segment of a CoT trace demarcated by newlines as a distinct reasoning step\.

#### Expert Set and Unique Expert Count\.

For each reasoning stepiiin a CoT trace, define the*expert set*at layerllas the union of all activated expert indices across the tokens of that step:

ℰi\(l\)=⋃t∈𝒯itop​\-​k⁡\(𝐠t\(l\)\),\\mathcal\{E\}\_\{i\}^\{\(l\)\}=\\bigcup\_\{t\\in\\mathcal\{T\}\_\{i\}\}\\operatorname\{top\\text\{\-\}k\}\\\!\\left\(\\mathbf\{g\}\_\{t\}^\{\(l\)\}\\right\),\(1\)where𝐠t\(l\)\\mathbf\{g\}\_\{t\}^\{\(l\)\}is the gating weight vector at tokenttand layerll, and𝒯i\\mathcal\{T\}\_\{i\}denotes the token indices belonging to stepii\. The*unique expert count*at stepiiis denoted byui=\|ℰi\(l\)\|u\_\{i\}=\|\\mathcal\{E\}\_\{i\}^\{\(l\)\}\|, serving as the code\-length surrogate in the analysis below\.

Unique Expert Count Trajectory – MATH by level

![Refer to caption](https://arxiv.org/html/2607.20427v1/figs/unique_expert_traj_gemma.png)\(a\)Gemma\-4\-27B\-A4B
![Refer to caption](https://arxiv.org/html/2607.20427v1/figs/unique_expert_traj_phi-moe.png)\(b\)Phi\-3\.5\-MoE
![Refer to caption](https://arxiv.org/html/2607.20427v1/figs/unique_expert_traj_qwen.png)\(c\)Qwen3\.5\-35B\-A3B

Figure 2:Unique expert count trajectory across normalized reasoning steps \(MATH\) for Gemma\-4\-27B\-A4B, Phi\-3\.5\-MoE, and Qwen3\.5\-35B\-A3B\.
#### Router as an Information\-Theoretic Encoder\.

We interpret the MoE router as a*semantic encoder*: the gating mechanism maps the input at stepiito a discrete expert activation patternℰi\(l\)\\mathcal\{E\}\_\{i\}^\{\(l\)\}, which functions as a combinatorial codeword drawn from the expert index set\{1,…,E\}\\\{1,\\ldots,E\\\}\.

LetO∈𝒪O\\in\\mathcal\{O\}denote the*semantic operation type*of a reasoning step, where𝒪=\{add/subtract,multiply/divide,problem setup,other\}\\mathcal\{O\}=\\\{\\texttt\{add/subtract\},\\,\\texttt\{multiply/divide\},\\,\\texttt\{problem setup\},\\,\\texttt\{other\}\\\}111These four categories are selected because LLMs are inherently limited in performing highly complex arithmetic computations; problem setup and the four basic arithmetic operations represent the core cognitive capabilities required for LLM Chain\-of\-Thought reasoning on mathematical problems\.is classified per step using a lightweight keyword classifier \(Appendix[I](https://arxiv.org/html/2607.20427#A9)\), andp​\(o\)p\(o\)denotes the empirical frequency of typeo∈𝒪o\\in\\mathcal\{O\}\. The quality of routing is measured by the*mutual information*betweenOOand the induced activation patternℰ\(l\)\\mathcal\{E\}^\{\(l\)\}:

I​\(O;ℰ\(l\)\)=H​\(O\)−H​\(O∣ℰ\(l\)\),I\\\!\\left\(O;\\,\\mathcal\{E\}^\{\(l\)\}\\right\)=H\(O\)\-H\\\!\\left\(O\\mid\\mathcal\{E\}^\{\(l\)\}\\right\),\(2\)whereH​\(O\)=−∑o∈𝒪p​\(o\)​log⁡p​\(o\)H\(O\)=\-\\sum\_\{o\\in\\mathcal\{O\}\}p\(o\)\\log p\(o\)is the semantic entropy andH​\(O∣ℰ\(l\)\)H\(O\\mid\\mathcal\{E\}^\{\(l\)\}\)is the residual uncertainty after observing the routing pattern\. A well\-optimized router maximizesI​\(O;ℰ\(l\)\)I\(O;\\mathcal\{E\}^\{\(l\)\}\), ensuring the activated expert set is as informative as possible about the underlying semantic operation\.

#### From Mutual Information to Huffman Optimality\.

A key result from information theory\(Cover and Thomas,[2006](https://arxiv.org/html/2607.20427#bib.bib30)\)is that maximizing mutual information under a mean activation constraint is equivalent to minimizing the average description length of the source\. Interpretinguiu\_\{i\}as the*code length*assigned to stepii, the Huffman lower bound gives:

𝔼​\[ui\]≥H​\(O\)/log⁡\(Ek\),\\mathbb\{E\}\[u\_\{i\}\]\\;\\geq\\;H\(O\)/\\log\\binom\{E\}\{k\},\(3\)with equality when shorter codes \(smalleruiu\_\{i\}\) are assigned to more frequent operation types\. This yields theFrequency\-Diversity Lawas a falsifiable prediction:

u¯\(o\)∝−log⁡p​\(o\),\\bar\{u\}^\{\(o\)\}\\;\\propto\\;\{\-\}\\log p\(o\),\(4\)whereu¯\(o\)=𝔼​\[ui∣type⁡\(i\)=o\]\\bar\{u\}^\{\(o\)\}=\\mathbb\{E\}\[u\_\{i\}\\mid\\operatorname\{type\}\(i\)=o\]is the mean unique expert count for steps of typeo∈𝒪o\\in\\mathcal\{O\}\. Equation \([4](https://arxiv.org/html/2607.20427#S3.E4)\) states that the router assigns shorter combinatorial codes to more frequent operation types, satisfying the Huffman optimality condition that maximizesI​\(O;ℰ\(l\)\)I\(O;\\mathcal\{E\}^\{\(l\)\}\)under the mean activation constraint\.

Table 1:Routing architecture characteristics and their effect on expert space
pruning manifestation\.ModelEEkkk/Ek/EPruning PatternGemma\-4\-27B\-A4B12886\.25%Monotonic decrease inuiu\_\{i\}Phi\-3\.5\-MoE16212\.5%Oscillating \(cluster pairs\)Qwen3\.5\-35B\-A3B25683\.1%Sparsity masks signal \(Huffman unmeasurable\)
#### Temporal Compression: Expert Diversity Decays Along the Reasoning Trace\.

Beyond the cross\-type Huffman correspondence, we observe a complementary*temporal*phenomenon within individual CoT traces\. As illustrated in Figure[2](https://arxiv.org/html/2607.20427#S3.F2), the unique expert countuiu\_\{i\}typically peaks during the early reasoning phases before exhibiting a systematic contraction in the latter half of the trace as the indexiiincreases\.

We interpret this as an information convergence process\. In the early steps of a CoT trace, the reasoning state is highly uncertain: the model must explore a broad semantic space to parse the problem and identify relevant sub\-tasks\. This corresponds to high conditional entropyH​\(O∣step​i\)H\(O\\mid\\text\{step \}i\), which, according to the Huffman correspondence in Eq\. \([4](https://arxiv.org/html/2607.20427#S3.E4)\), induces larger expert sets \(longer codes\)\. As reasoning progresses, the model incrementally resolves this uncertainty, concentrating probability mass onto a narrowing set of operation types\. The resulting reduction in semantic entropy manifests as a contraction of the activated expert set, with later steps relying on a compact core of high\-frequency experts\. Formally, this temporal dynamic can be expressed as:

𝔼​\[ui\]≈α⋅H​\(O∣step​i\)\+β,α\>0,\\mathbb\{E\}\[u\_\{i\}\]\\;\\approx\\;\\alpha\\cdot H\\\!\\left\(O\\mid\\text\{step \}i\\right\)\+\\beta,\\quad\\alpha\>0,\(5\)whereβ\\betais the baseline expert count when semantic uncertainty is fully resolved \(H=0H=0\), andα\>0\\alpha\>0quantifies the router’s sensitivity: each additional nat of operation\-type uncertainty inducesα\\alphamore unique experts per layer\. We estimateH^​\(O∣step​i\)\\hat\{H\}\(O\\mid\\text\{step \}i\)empirically by partitioning the normalized trace into ten equal\-width position bins and computing the entropy of the empirical operation\-type distribution within each bin\.

Figure[3](https://arxiv.org/html/2607.20427#S3.F3)validates Eq\. \([5](https://arxiv.org/html/2607.20427#S3.E5)\) on GSM8K: Gemma\-4\-27B\-A4B confirms the linear relationship \(α=7\.36\\alpha\{=\}7\.36,r=0\.773r\{=\}0\.773,p=0\.009p\{=\}0\.009\); Phi and Qwen deviate for the architecture\-specific reasons discussed in Section[3](https://arxiv.org/html/2607.20427#S3.SS0.SSS0.Px5)and Section[4](https://arxiv.org/html/2607.20427#S4)respectively, consistent with the Huffman signal being unmeasurable in those regimes\. Equation \([5](https://arxiv.org/html/2607.20427#S3.E5)\) thus characterizes efficient MoE routing as a temporal compression process that continuously adapts code length to the evolving information content of the reasoning trajectory\.

#### Architecture\-Specific Routing Patterns\.

While the Frequency\-Diversity Law serves as a universal governing principle, its empirical manifestation is strictly modulated by the expert sparsity ratiok/Ek/E\(Table[1](https://arxiv.org/html/2607.20427#S3.T1)\)\.

Gemma\-4\-27B\-A4B\(k/E=6\.25%k/E\{=\}6\.25\\%\) exhibits the most direct form of temporal compression, characterized by a clean monotonic decline in the unique expert countuiu\_\{i\}; in this regime, early and late reasoning steps are distinguishable from the raw signal\.

Phi\-3\.5\-MoE\(k/E=12\.5%k/E\{=\}12\.5\\%\) presents a more complex temporal trajectory due to its inherent expert\-index clustering \(Table[1](https://arxiv.org/html/2607.20427#S3.T1)\); here, temporal compression manifests as the routing state settling into stable, low\-entropy cluster pairs rather than a simple reduction in cardinality\.

In contrast,Qwen3\.5\-35B\-A3Bemploys the sparsest activation ratio \(k/E=3\.1%k/E\{=\}3\.1\\%\), where the Huffman signal becomes statistically unmeasurable\. As Section[4](https://arxiv.org/html/2607.20427#S4)reveals, the root cause is*functional redundancy*: the vast majority of Qwen’s 256 experts are near\-duplicates, so routing codes expend bits selecting*which equivalent expert*rather than encoding*which semantic operation*\. Table[1](https://arxiv.org/html/2607.20427#S3.T1)summarizes the three routing regimes\.

#### Quantitative Validation\.

Figure[1](https://arxiv.org/html/2607.20427#S1.F1)reports the relationship between operation\-type frequencyp​\(o\)p\(o\)and mean unique expert countu¯\(o\)\\bar\{u\}^\{\(o\)\}\(Eq\.[4](https://arxiv.org/html/2607.20427#S3.E4)\) across all three architectures\.

For Gemma\-4\-27B\-A4B and Phi\-3\.5\-MoE, the four operation types conform to the predicted Huffman correspondence: both models attain Spearmanρ=1\.00\\rho=1\.00\(p<0\.05p<0\.05\)\. As shown in Figure[1](https://arxiv.org/html/2607.20427#S1.F1), the mean expert countu¯\(o\)\\bar\{u\}^\{\(o\)\}for each operation type coincides with the reverse frequency ranking \(−log⁡p​\(o\)\-\\log p\(o\)\), consistent with Eq\. \([4](https://arxiv.org/html/2607.20427#S3.E4)\)\. Qwen3\.5\-35B\-A3B does not follow this pattern; we defer its analysis to Section[4](https://arxiv.org/html/2607.20427#S4), where its extreme activation sparsity \(k/E=3\.1%k/E=3\.1\\%\) is shown to induce a qualitatively distinct routing regime\. A further corroborating finding is reported in the appendix: routing fingerprints are content\-independent \(intra\-type Jaccard\>\>inter\-type across all 12 model–dataset pairs,p≤10−163p\\leq 10^\{\-163\}; Appendix[I](https://arxiv.org/html/2607.20427#A9)\)\.

Temporal Compression:u¯i\\bar\{u\}\_\{i\}vs\.H^​\(O∣step​i\)\\hat\{H\}\(O\\mid\\text\{step\}\\,i\)— GSM8K ![Refer to caption](https://arxiv.org/html/2607.20427v1/x1.png)

Figure 3:u¯i\\bar\{u\}\_\{i\}vs\. empiricalH^​\(O∣step​i\)\\hat\{H\}\(O\\mid\\text\{step\}\\,i\)on GSM8K \(ten decile bins per model; dashed line: OLS fit\)\. Gemma confirms the linear relationship in Eq\. \([5](https://arxiv.org/html/2607.20427#S3.E5)\) \(α=7\.36\\alpha\{=\}7\.36,β=30\.49\\beta\{=\}30\.49,r=0\.773r\{=\}0\.773,p=0\.009p\{=\}0\.009\)\. Phi \(r=−0\.460r\{=\}\{\-\}0\.460,p=0\.18p\{=\}0\.18\) deviates because its oscillating trajectory makesH^\\hat\{H\}a poor linear predictor\. Qwen \(r≈0r\{\\approx\}0\) shows no relationship, consistent with functional redundancy decoupling expert count from operation\-type entropy\.

## 4Diving Deeper into Qwen: Expert Redundancy and Effective sparsity

The analysis in Section[3](https://arxiv.org/html/2607.20427#S3.SS0.SSS0.Px5)attributes Qwen3\.5\-35B\-A3B’s unmeasurable Huffman signal to its extreme routing sparsity\. This raises a natural follow\-up question:although 256 expertsformallyexist, are all of them functionally distinct?If many experts are redundant \(activating on the same operation types\) then the*effective*expert pool may be much smaller than 256, and the truek/Ek/Eratio correspondingly higher\.

### 4\.1Discovering Expert Redundancy via Co\-Activation Profiles

To test this hypothesis, we build a14\-dimensional co\-activation profilefor every \(layer, expert\) pair from correct CoT traces across four datasets \(GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2607.20427#bib.bib12)\), MATH\(Hendryckset al\.,[2021](https://arxiv.org/html/2607.20427#bib.bib13)\), AQuA\(Linget al\.,[2017](https://arxiv.org/html/2607.20427#bib.bib14)\), CompMath\-MCQ\(Loet al\.,[2025](https://arxiv.org/html/2607.20427#bib.bib52)\)\)\. Each profile encodes 4 operation\-type dimensions \(fraction of activations per semantic type:problem setup,add/subtract,multiply/divide,other\) and 10 stage dimensions \(fraction per decile of the normalized trajectory, stepi/\(n−1\)i/\(n\{\-\}1\)\), normalized independently so both aspects contribute equally \(construction details in Appendix[D](https://arxiv.org/html/2607.20427#A4)\)\.

We then compute the pairwise cosine similarity between all expert profiles within each layer\. Table[4](https://arxiv.org/html/2607.20427#A4.T4)in Appendix[D](https://arxiv.org/html/2607.20427#A4)reports the resulting similarity statistics across all 40 layers\.

The figures are striking: across every layer, the*average*pairwise cosine similarity between expert profiles exceeds 0\.886, and the 90th\-percentile similarity exceeds 0\.983\. In other words, the vast majority of Qwen’s 256 experts are nearly indistinguishable in terms of*which operation types they activate on*and*when in the reasoning chain they fire*\. Despite load\-balancing loss during training forcing all experts to be approximately equally utilized, their functional roles \(as captured by temporal co\-activation structure\) are highly redundant\.

### 4\.2Pruning Redundant Experts and Testing for Huffman Emergence

#### Pruning method\.

We identify redundant expert pairs via a greedy similarity\-based procedure applied independently per layer:

1. 1\.Compute the256×256256\{\\times\}256cosine similarity matrix from co\-activation profiles\.
2. 2\.Iteratively find the most similar active pair\(i,j\)\(i,j\); remove the expert with the lower total activation frequency\.
3. 3\.Repeat until the target pruning fraction is reached\.

At inference time, the removed experts are masked via a forward hook on the gate module: their router logits are set to−∞\-\\inftybefore top\-kkselection, so they are never dispatched to\. No weights are modified; the procedure is entirely training\-free\. Full algorithmic details are given in Appendix[F](https://arxiv.org/html/2607.20427#A6)\. If the redundancy hypothesis is correct, pruning should incur minimal accuracy loss and crucially raisek/Eeffk/E\_\{\\text\{eff\}\}enough to make the Huffman signal detectable \(see Appendix[E](https://arxiv.org/html/2607.20427#A5)for effective sparsity at each level\)\. Table[2](https://arxiv.org/html/2607.20427#S4.T2)reports accuracy and Huffmanrron GSM8K under each condition\.

Table 2:GSM8K accuracy and Huffman correlation for Qwen3\.5\-35B\-A3B under SDP\.PruningEeffE\_\{\\text\{eff\}\}GSM8K Acc\.HuffmanrrAvg\. Tokens0% \(baseline\)25693\.6%−0\.631\-0\.6311,65810% expert pruned23091\.9%\+0\.567\+0\.5671,89920% expert pruned20575\.7%\+0\.533\+0\.5332,339Qwen3\.5\-35B\-A3B — Huffman test:u¯\(o\)∝−log⁡p​\(o\)\\bar\{u\}^\{\(o\)\}\\propto\-\\log p\(o\) ![Refer to caption](https://arxiv.org/html/2607.20427v1/x2.png)

Figure 4:Huffman scatter plots for Qwen3\.5\-35B\-A3B under 0% \(baseline\), 10%, and 20% Subset Difference Pruning, evaluated on GSM8K\.
#### Results\.

Figure[4](https://arxiv.org/html/2607.20427#S4.F4)shows the Huffman scatter for all three conditions evaluated on GSM8K dataset\. The baseline \(0%, 256 experts\) yields Pearsonr=−0\.631r=\-0\.631, indicating that*frequent*operation types activate*more*unique experts per step, which is an anti\-Huffman signature\. The root cause is Qwen’s*inverted\-U trajectory*\(Figure[2](https://arxiv.org/html/2607.20427#S3.F2)\): unlike Gemma and Phi, which show monotonic decline, Qwen’s unique expert count rises during mid\-trace exploration then collapses as the model converges toward a solution\. Because operation types are not uniformly distributed across trace positions; for instance,problem setupkeywords \(*let, given that, define, etc\.*\) appear disproportionately in the lower\-u¯\\bar\{u\}phases while arithmetic steps dominate the high\-u¯\\bar\{u\}exploration peak,*temporal position within the trace*, rather than operation\-type frequency, becomes the primary predictor of expert set size\. Consequently, the rarest operation type \(*setup*,p=0\.095p\{=\}0\.095\) receives the*lowest*mean expert count \(u¯=45\.3\\bar\{u\}\{=\}45\.3\), directly inverting the Huffman expectation \(full per\-type statistics in Appendix[J](https://arxiv.org/html/2607.20427#A10)\)\. After masking the middle\-redundancy tier, both pruned conditions flip to positive correlation \(r=\+0\.57r\\\!=\\\!\+0\.57at 10% pruning;r=\+0\.53r\\\!=\\\!\+0\.53at 20%\), confirming that the Huffman phenomenon emerges once effective sparsity is raised\. The 10%\-pruned model retains near\-baseline accuracy \(91\.9% vs\. 93\.6%\), suggesting this level sits near the boundary of tolerable redundancy removal, whereas 20% pruning incurs a more substantial drop \(75\.7%\), likely because the router can no longer adequately compensate for the missing capacity on harder problems\.

Pruned Expert Proportion — Qwen3\.5\-35B\-A3B ![Refer to caption](https://arxiv.org/html/2607.20427v1/x3.png)

Figure 5:Left: Share of total expert activations partitioned by redundancy tier\. Tier\-1 \(top 10% most redundant experts,∼26\{\\sim\}26per layer\) accounts for 12\.6% of activations, while Tier\-2 \(the subsequent 10% tier\) contributes an additional 11\.0%\. Right: Comparison of expert count removed versus activations lost for each pruning condition\. The “10% expert pruned” case removes Tier\-1 \(10\.2% of pool\), whereas the “20% expert pruned” case removes the combined Tier\-1\+2 \(19\.9% of pool\)\. In both settings, the share of lost activations exceeds the share of removed experts, confirming that these experts are actively utilized functional redundancies rather than inactive artifacts\.

### 4\.3What Gets Pruned, and Why It Matters

#### Activation share of pruned experts\.

The SDP algorithm iteratively removes the lower\-frequency member from the most similar expert pairs\. Figure[5](https://arxiv.org/html/2607.20427#S4.F5)compares activation loss across tiers: the Tier\-1 experts \(10\.2% of pool\) account for12\.6%of activations, while the combined Tier\-1\+2 \(19\.9% of pool\) accounts for23\.6%\. Because the activation share consistently exceeds the expert count share, these units are clearly not dormant relics; rather, they are active but functionally redundant components\. At inference time, the router successfully redistributes their traffic to surviving experts within the same functional cluster, maintaining coverage despite the reduced architectural flexibility\.

#### Functional profile t\-SNE\.

To verify that pruned experts do not occupy a narrow, easily\-separable functional niche, Figure[6](https://arxiv.org/html/2607.20427#S4.F6)plots t\-SNE embeddings of the 14\-dimensional functional profiles for all 10,185 active \(layer, expert\) pairs\. Tier\-1 \(orange\) and tier\-1\+2 \(red\) points areuniformly scatteredacross the entire embedding and show no isolated cluster that separates them from the retained population\. This geometric evidence reinforces the redundancy interpretation: the removed experts are near\-duplicate copies of retained experts, distributed throughout every functional region of the routing space rather than concentrated in any single operation\-type or temporal niche\. Taken together, the activation\-share and t\-SNE results confirm that Qwen’s 256\-expert pool contains substantial built\-in redundancy and that Subset Difference Pruning selectively removes it while preserving functional diversity\.

From an information\-theoretic perspective, this redundancy is a symptom of*insufficient routing information density*: a large share of each routing codeword’s bits are consumed identifying*which functionally identical expert*was selected, rather than encoding*which semantic operation*is being performed\. This explains both the anti\-Huffman signature \(Table[2](https://arxiv.org/html/2607.20427#S4.T2)\) and the diffuse t\-SNE geometry: when many experts are interchangeable, the router’s choices carry low mutual information with the operation type, and the resulting routing fingerprints are scattered rather than clustering semantically by operation\.

#### Huffman Compliance as a Routing Health Diagnostic\.

The Huffman correlationrr\(Eq\.[4](https://arxiv.org/html/2607.20427#S3.E4)\) provides a lightweight, task\-free probe for expert pool quality: a positiverrconfirms the router is operating as an MDL\-optimal encoder, while a negativerrsignals a*redundancy trap*, where the effective sparsityk/Eeffk/E\_\{\\text\{eff\}\}is too low for the Huffman law to manifest, typically because load\-balancing has filled the expert pool with functional near\-duplicates\. Crucially, this diagnostic requires only routing traces on a small held\-out set and no accuracy benchmarks, making it applicable at any point in training or at deployment to assess whether the expert pool’s functional diversity is properly aligned with the model’s semantic operation distribution\.

Expert Functional Profile t\-SNE — Qwen3\.5\-35B\-A3B ![Refer to caption](https://arxiv.org/html/2607.20427v1/x4.png)

Figure 6:t\-SNE visualization of expert functional profiles in Qwen3\.5\-35B\-A3B\. The 14\-dimensional profiles are computed from 782 traces across all active \(layer, expert\) pairs\. Panels illustrate experts removed via \(left\) 10% Subset Difference Pruning \(orange, 1,040 pairs\) and \(right\) 20% pruning \(red, 2,040 pairs\), overlaid on retained experts \(grey\)\. The uniform distribution of pruned experts suggests functional redundancy across the entire routing space\.

## 5Conclusion

This study confirms that the MoE routing mechanism functions as a spontaneously evolvedHuffman Encoder\. Addressing the central hypothesis posed in the introduction:*Routing is Compression*, our findings yield three key insights:

- •Universality of the Huffman Phenomenon:In models like Gemma\-4\-27B\-A4B and Phi\-3\.5\-MoE, routing decisions exhibit peak information\-theoretic efficiency\. The monotonic increase of expert diversity with task rarity \(−log⁡p​\(o\)\-\\log p\(o\)\), evidenced by a Spearman’sρ=1\.00\\rho=1\.00, proves that highly optimized models naturally allocate longer routing paths to complex semantic logic to minimize the MDL\.
- •Antagonism between Redundancy and Efficiency:The anomaly in Qwen3\.5\-35B\-A3B reveals the pitfalls of aggressive load\-balancing\. Forced utilization diversity often results in functional redundancy \(pairwise cosine similarity\>0\.90\>0\.90\), which obscures the latent Huffman signal and deviates from optimal source coding principles\.
- •Structural Re\-encoding via Pruning:ThroughSubset Difference Pruning, we successfully removed the intermediate\-redundancy tier of experts\. This intervention reshaped Qwen’s reasoning trajectories from an anti\-Huffman state to an efficient encoding path \(Pearsonrrimproved to\+0\.57\+0\.57\) with negligible performance loss \(−1\.7%\-1\.7\\%on GSM8K\)\.

In conclusion, routing efficiency hinges on effective sparsity \(k/Eeffk/E\_\{\\text\{eff\}\}\)\. By maintaining a 5–13% ratio, MoE routers transcend mere load\-balancing to act as genuine semantic compression engines\. This validates the MDL principle as a governing law for routing paths,offering a theoretical blueprint for building high\-performance, low\-redundancy reasoning models\.Beyond post\-hoc analysis, the Huffman correlationrropens a path toward training\-time self\-monitoring: ifrrturns persistently negative during training, it signals that load balancing is generating functional duplicates faster than the model develops meaningful expert specialization—a diagnostic that requires no downstream evaluation\. Limitations are discussed in Appendix[K](https://arxiv.org/html/2607.20427#A11)\.

## References

- M\. Abdin, J\. Aneja, H\. Awadalla, A\. Awadallah, A\. A\. Awan, N\. Bach, A\. Bahree, A\. Bakhtiari, J\. Bao,et al\.\(2024\)Phi\-3 technical report: a highly capable language model locally on your phone\.arXiv preprint arXiv:2404\.14219\.Cited by:[§2\.1](https://arxiv.org/html/2607.20427#S2.SS1.p1.2)\.
- M\. Artetxe, S\. Bhosale, N\. Goyal, T\. Mihaylov, M\. Ott, S\. Shleifer, X\. V\. Lin, J\. Du, S\. Iyer, R\. Pasunuru,et al\.\(2022\)Efficient large scale language modeling with mixtures of experts\.InProceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§2\.2](https://arxiv.org/html/2607.20427#S2.SS2.p1.1)\.
- Y\. Chen, Z\. Cai, X\. Ji, W\. Zhao, A\. Zhang, X\. Wang, and T\. Chua \(2026\)Understanding multilingualism in mixture\-of\-experts llms: routing mechanism, expert specialization, and layerwise steering\.arXiv preprint arXiv:2601\.14050\.Cited by:[§2\.2](https://arxiv.org/html/2607.20427#S2.SS2.p1.1)\.
- A\. Clark, D\. de Las Casas, A\. Guy, A\. Mensch, M\. Paganini, J\. Hoffmann, B\. Damoc, B\. Hechtman, T\. Cai, S\. Borgeaud,et al\.\(2022\)Unified scaling laws for routed language models\.InInternational conference on machine learning,pp\. 4057–4086\.Cited by:[§1](https://arxiv.org/html/2607.20427#S1.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[Appendix K](https://arxiv.org/html/2607.20427#A11.p1.1),[§4\.1](https://arxiv.org/html/2607.20427#S4.SS1.p1.1)\.
- T\. M\. Cover and J\. A\. Thomas \(2006\)Elements of information theory\.2nd edition,Wiley\-Interscience\.Cited by:[§2\.3](https://arxiv.org/html/2607.20427#S2.SS3.p1.1),[§3](https://arxiv.org/html/2607.20427#S3.SS0.SSS0.Px3.p1.2)\.
- D\. Dai, C\. Deng, C\. Zhao, R\. Xu, H\. Gao, D\. Chen, J\. Li, W\. Zeng, X\. Yu,et al\.\(2024\)Deepseekmoe: towards ultimate expert specialization in mixture\-of\-experts language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1280–1297\.Cited by:[§1](https://arxiv.org/html/2607.20427#S1.p1.1)\.
- G\. Delétang, A\. Ruoss, P\. Duquenne, E\. Catt, T\. Genewein, C\. Mattern, J\. Grau\-Moya, L\. K\. Wenliang, M\. Aitchison, L\. Orseau, M\. Hutter, and J\. Veness \(2024\)Language modeling is compression\.InInternational Conference on Learning Representations,Cited by:[§2\.3](https://arxiv.org/html/2607.20427#S2.SS3.p1.1)\.
- N\. Elhage, T\. Hume, C\. Olsson, N\. Nanda, T\. Henighan, S\. Johnston, S\. ElShowk, N\. Joseph, N\. DasSarma, B\. Mann,et al\.\(2022\)Softmax linear units\.Transformer Circuits Thread\.Note:[https://transformer\-circuits\.pub/2022/solu/index\.html](https://transformer-circuits.pub/2022/solu/index.html)Cited by:[§2\.2](https://arxiv.org/html/2607.20427#S2.SS2.p1.1)\.
- W\. Fedus, B\. Zoph, and N\. Shazeer \(2022\)Switch transformers: scaling to trillion parameter models with simple and efficient sparsity\.Journal of Machine Learning Research23\(120\),pp\. 1–39\.Cited by:[§1](https://arxiv.org/html/2607.20427#S1.p1.1),[§1](https://arxiv.org/html/2607.20427#S1.p5.1),[§2\.1](https://arxiv.org/html/2607.20427#S2.SS1.p1.2)\.
- E\. Frantar, S\. Ashkboos, T\. Hoefler, and D\. Alistarh \(2022\)GPTQ: accurate post\-training quantization for generative pre\-trained transformers\.arXiv preprint arXiv:2210\.17323\.Cited by:[§2\.3](https://arxiv.org/html/2607.20427#S2.SS3.p1.1)\.
- M\. Geva, R\. Schuster, J\. Berant, and O\. Levy \(2021\)Transformer feed\-forward layers are key\-value memories\.InProceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§2\.2](https://arxiv.org/html/2607.20427#S2.SS2.p1.1)\.
- S\. Han, J\. Pool, J\. Tran, and W\. J\. Dally \(2015\)Learning both weights and connections for efficient neural networks\.InAdvances in Neural Information Processing Systems,Vol\.28\.Cited by:[§2\.3](https://arxiv.org/html/2607.20427#S2.SS3.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021\)Measuring mathematical problem solving with the MATH dataset\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[Appendix K](https://arxiv.org/html/2607.20427#A11.p1.1),[§4\.1](https://arxiv.org/html/2607.20427#S4.SS1.p1.1)\.
- J\. Herbst, J\. H\. Lee, and S\. Wermter \(2026\)The expert strikes back: interpreting mixture\-of\-experts language models at expert level\.arXiv preprint arXiv:2604\.02178\.Cited by:[§2\.2](https://arxiv.org/html/2607.20427#S2.SS2.p1.1)\.
- D\. A\. Huffman \(1952\)A method for the construction of minimum\-redundancy codes\.Proceedings of the IRE40\(9\),pp\. 1098–1101\.Cited by:[§1](https://arxiv.org/html/2607.20427#S1.p2.1)\.
- M\. Hutter \(2005\)Universal artificial intelligence: sequential decisions based on algorithmic probability\.Vol\.300,Springer\.Cited by:[§1](https://arxiv.org/html/2607.20427#S1.p2.1)\.
- A\. Q\. Jiang, A\. Sablayrolles, A\. Roux, A\. Mensch, B\. Savary, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, E\. B\. Hanna,et al\.\(2024\)Mixtral of experts\.arXiv preprint arXiv:2401\.04088\.Cited by:[§1](https://arxiv.org/html/2607.20427#S1.p1.1)\.
- G\. T\. A\. Kamath, J\. Ferret, S\. Pathak, N\. Vieillard, R\. Merhej, S\. Perrin, T\. Matejovicova, A\. Ram’e, M\. Rivière, L\. Rouillard, T\. Mesnard, G\. Cideron, J\. Grill, S\. Ramos, E\. Yvinec, M\. Casbon, E\. Pot, I\. Penchev, G\. Liu, F\. Visin, K\. Kenealy, L\. Beyer, X\. Zhai, A\. Tsitsulin, R\. I\. Busa\-Fekete, A\. Feng, N\. Sachdeva, B\. Coleman, Y\. Gao, B\. Mustafa, I\. Barr, E\. Parisotto, D\. Tian, M\. Eyal, C\. Cherry, J\. Peter, D\. Sinopalnikov, S\. Bhupatiraju, R\. Agarwal, M\. Kazemi, D\. Malkin, R\. Kumar, D\. Vilar, I\. Brusilovsky, J\. Luo, A\. Steiner, A\. Friesen, A\. Sharma, A\. Sharma, A\. M\. Gilady, A\. Goedeckemeyer, A\. Saade, A\. Kolesnikov, A\. Bendebury, A\. Abdagic, A\. Vadi, A\. Gyorgy, A\. S\. Pinto, A\. Das, A\. Bapna, A\. Miech, A\. Yang, A\. Paterson, A\. Shenoy, A\. Chakrabarti, B\. Piot, B\. Wu, B\. Shahriari, B\. Petrini, C\. Chen, C\. L\. Lan, C\. A\. Choquette\-Choo, C\. Carey, C\. Brick, D\. Deutsch, D\. Eisenbud, D\. Cattle, D\. Cheng, D\. Paparas, D\. S\. Sreepathihalli, D\. Reid, D\. Tran, D\. Zelle, E\. Noland, E\. Huizenga, E\. Kharitonov, F\. Liu, G\. Amirkhanyan, G\. Cameron, H\. Hashemi, H\. Klimczak\-Pluci’nska, H\. Singh, H\. Mehta, H\. T\. Lehri, H\. Hazimeh, I\. Ballantyne, I\. Szpektor, I\. Nardini, J\. Pouget\-Abadie, J\. Chan, J\. Stanton, J\. M\. Wieting, J\. Lai, J\. Orbay, J\. Fernandez, J\. Newlan, J\. Ji, J\. Singh, K\. Black, K\. Yu, K\. Hui, K\. Vodrahalli, K\. Greff, L\. Qiu, M\. Valentine, M\. Coelho, M\. Ritter, M\. Hoffman, M\. Watson, M\. Chaturvedi, M\. Moynihan, M\. Ma, N\. Babar, N\. Noy, N\. Byrd, N\. Roy, N\. Momchev, N\. Chauhan, O\. Bunyan, P\. Botarda, P\. Caron, P\. K\. Rubenstein, P\. Culliton, P\. Schmid, P\. G\. Sessa, P\. Xu, P\. Stańczyk, P\. D\. Tafti, R\. Shivanna, R\. Wu, R\. Pan, R\. A\. Rokni, R\. Willoughby, R\. Vallu, R\. Mullins, S\. Jerome, S\. Smoot, S\. Girgin, S\. Iqbal, S\. Reddy, S\. Sheth, S\. Põder, S\. Bhatnagar, S\. R\. Panyam, S\. Eiger, S\. Zhang, T\. Liu, T\. Yacovone, T\. Liechty, U\. Kalra, U\. Evci, V\. Misra, V\. Roseberry, V\. Feinberg, V\. Kolesnikov, W\. Han, W\. Kwon, X\. Chen, Y\. Chow, Y\. Zhu, Z\. Wei, Z\. Egyed, V\. Cotruta, M\. Giang, P\. Kirk, A\. Rao, J\. Lo, E\. Moreira, L\. G\. Martins, O\. Sanseviero, L\. Gonzalez, Z\. Gleicher, T\. Warkentin, V\. S\. Mirrokni, E\. Senter, E\. Collins, J\. Barral, Z\. Ghahramani, R\. Hadsell, Y\. Matias, D\. Sculley, S\. Petrov, N\. Fiedel, N\. Shazeer, O\. Vinyals, J\. Dean, D\. Hassabis, K\. Kavukcuoglu, C\. Farabet, E\. Buchatskaya, J\. Alayrac, R\. Anil, D\. Lepikhin, S\. Borgeaud, O\. Bachem, A\. Joulin, A\. Andreev, C\. Hardin, R\. Dadashi, and L\. Hussenot \(2025\)Gemma 3 technical report\.ArXivabs/2503\.19786\.External Links:[Link](https://api.semanticscholar.org/CorpusID:277313563)Cited by:[§1](https://arxiv.org/html/2607.20427#S1.p5.1),[§2\.1](https://arxiv.org/html/2607.20427#S2.SS1.p1.2)\.
- J\. Krajewski, J\. Ludziejewski, K\. Adamczewski, M\. Pióro, M\. Krutul, S\. Antoniak, K\. Ciebiera, K\. Król, T\. Odrzygóźdź, P\. Sankowski,et al\.\(2024\)Scaling laws for fine\-grained mixture of experts\.arXiv preprint arXiv:2402\.07871\.Cited by:[§1](https://arxiv.org/html/2607.20427#S1.p3.1),[§1](https://arxiv.org/html/2607.20427#S1.p5.1)\.
- D\. Lepikhin, H\. Lee, Y\. Xu, D\. Chen, O\. Firat, Y\. Huang, M\. Krikun, N\. Shazeer, and Z\. Chen \(2021\)GShard: scaling giant models with conditional computation and automatic sharding\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2607.20427#S1.p3.1),[§2\.1](https://arxiv.org/html/2607.20427#S2.SS1.p1.2)\.
- W\. Ling, D\. Yogatama, C\. Dyer, and P\. Blunsom \(2017\)Program induction by rationale generation: learning to solve and explain algebraic word problems\.InProceedings of the Association for Computational Linguistics \(ACL\),Cited by:[Appendix K](https://arxiv.org/html/2607.20427#A11.p1.1),[§4\.1](https://arxiv.org/html/2607.20427#S4.SS1.p1.1)\.
- K\. M\. Lo, Z\. Huang, Z\. Qiu, Z\. Wang, and J\. Fu \(2025\)A closer look into mixture\-of\-experts in large language models\.InFindings of the Association for Computational Linguistics: NAACL 2025,pp\. 4427–4447\.Cited by:[Appendix K](https://arxiv.org/html/2607.20427#A11.p1.1),[§4\.1](https://arxiv.org/html/2607.20427#S4.SS1.p1.1)\.
- Z\. Mi, Y\. Chen, P\. Zhao, X\. Yu, H\. Wang, Y\. Wang, and S\. Huang \(2026\)Effective moe\-based llm compression by exploiting heterogeneous inter\-group experts routing frequency and information density\.arXiv preprint arXiv:2602\.09316\.Cited by:[§2\.3](https://arxiv.org/html/2607.20427#S2.SS3.p1.1)\.
- N\. Omi, S\. Sen, and A\. Farhadi \(2025\)Load balancing mixture of experts with similarity preserving routers\.arXiv preprint arXiv:2506\.14038\.Cited by:[§1](https://arxiv.org/html/2607.20427#S1.p5.1)\.
- J\. Puigcerver, C\. R\. Ruiz, B\. Mustafa, and N\. Houlsby \(2023\)From sparse to soft mixtures of experts\.InThe Twelfth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.20427#S1.p5.1)\.
- J\. Rissanen \(1978\)Modeling by shortest data description\.Automatica14\(5\),pp\. 465–471\.Cited by:[§2\.3](https://arxiv.org/html/2607.20427#S2.SS3.p1.1)\.
- C\. E\. Shannon \(1948\)A mathematical theory of communication\.Bell System Technical Journal27\(3\),pp\. 379–423\.Cited by:[§1](https://arxiv.org/html/2607.20427#S1.p2.1)\.
- C\. E\. Shannon \(1959\)Coding theorems for a discrete source with a fidelity criterion\.IRE International Convention Record,pp\. 142–163\.Cited by:[§1](https://arxiv.org/html/2607.20427#S1.p2.1)\.
- N\. Shazeer, A\. Mirhoseini, K\. Maziarz, A\. Davis, Q\. Le, G\. Hinton, and J\. Dean \(2017\)Outrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2607.20427#S1.p1.1),[§1](https://arxiv.org/html/2607.20427#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.20427#S2.SS1.p1.2)\.
- L\. Van der Maaten and G\. Hinton \(2008\)Visualizing data using t\-sne\.Journal of machine learning research9\(11\)\.Cited by:[Appendix C](https://arxiv.org/html/2607.20427#A3.SS0.SSS0.Px1.p1.4)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. H\. Chi, Q\. V\. Le, and D\. Zhou \(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in Neural Information Processing Systems35,pp\. 24824–24837\.Cited by:[§1](https://arxiv.org/html/2607.20427#S1.p6.1.2)\.
- F\. Xue, Z\. Zheng, Y\. Fu, J\. Ni, Z\. Zheng, W\. Zhou, and Y\. You \(2024\)OpenMoE: an early effort on open mixture\-of\-experts language models\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 55625–55655\.Cited by:[§1](https://arxiv.org/html/2607.20427#S1.p3.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§1](https://arxiv.org/html/2607.20427#S1.p5.1),[§2\.1](https://arxiv.org/html/2607.20427#S2.SS1.p1.2)\.
- K\. Zheng, W\. Huang, J\. Huo, Z\. Li, H\. P\. Zou, Y\. Yan, X\. Zou, J\. Li, J\. Li, H\. Zhang,et al\.\(2026\)Unveiling language routing isolation in multilingual moe models for interpretable subnetwork adaptation\.arXiv preprint arXiv:2604\.03592\.Cited by:[§2\.2](https://arxiv.org/html/2607.20427#S2.SS2.p1.1)\.
- Y\. Zhou, T\. Lei, H\. Liu, N\. Du, Y\. Huang, V\. Zhao, A\. Dai, Z\. Chen, Q\. Le, and J\. Laudon \(2022\)Mixture\-of\-experts with expert choice routing\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 7103–7114\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2607.20427#S1.p3.1)\.
- Z\. Zhou, Z\. Zhu, X\. Li, M\. Galkin, X\. Feng, S\. Koyejo, and B\. Han \(2025\)Landscape of thoughts: visualizing the reasoning process of large language models\.arXiv preprint arXiv:2503\.22165\.Cited by:[§3](https://arxiv.org/html/2607.20427#S3.p1.1)\.
- B\. Zoph \(2022\)Designing effective sparse expert models\.In2022 IEEE International Parallel and Distributed Processing Symposium Workshops \(IPDPSW\),Vol\.,pp\. 1044–1044\.External Links:[Document](https://dx.doi.org/10.1109/IPDPSW55747.2022.00171)Cited by:[§1](https://arxiv.org/html/2607.20427#S1.p5.1)\.

## Appendix AModel Architecture Details

Table[3](https://arxiv.org/html/2607.20427#A1.T3)summarizes the key routing parameters for each model\.

Table 3:MoE architecture parameters used in this work\.Model\# Layers\# ExpertsTop\-kkGate typeGemma\-4\-27B\-A4B301288Linear projection \(softmax\)Phi\-3\.5\-MoE32162Softmax routerQwen3\.5\-35B\-A3B402568Softmax gate
## Appendix BArchitecture\-Specific Routing Patterns: Extended Discussion

The three models span qualitatively distinct routing regimes determined by their expert sparsity ratiok/Ek/E\.

Gemma\-4\-27B\-A4Bactivatesk=8k\{=\}8out ofE=128E\{=\}128experts \(k/E=6\.25%k/E\{=\}6\.25\\%\), an intermediate sparsity that creates a natural balance between exploration and convergence\. Early reasoning steps engage a genuinely diverse expert set \(highusu\_\{s\}\), while late steps settle onto a smaller, stable subset; crucially, both behaviors are directly and monotonically reflected in the raw unique countusu\_\{s\}without requiring normalization or smoothing\. The unique expert count and the temporal compression signal are, in Gemma\-4’s case, essentially the same measure, making it the clearest empirical demonstration of the Huffman temporal compression phenomenon\.

Phi\-3\.5\-MoEroutes each token to exactlyk=2k\{=\}2experts fromE=16E\{=\}16, givingk/E=12\.5%k/E\{=\}12\.5\\%—the coarsest ratio among our models\. With only 16 total experts, individual tokens can cover a large fraction of the pool within a single step\. We observe a characteristic*odd/even clustering*phenomenon: the router tends to pair experts from distinct index groups \(e\.g\.,\{0,1\}\\\{0,1\\\},\{2,3\}\\\{2,3\\\}, …\) and alternates between these clusters across tokens\. This produces an*oscillating*unique\-expert trajectory rather than a monotonic decline\. Temporal compression in Phi\-3\.5\-MoE therefore manifests as the model settling into a stable, repeating cluster pair in later steps rather than a gradual count reduction; detecting convergence requires observing the trajectory over a window of multiple steps rather than any single\-step count\.

Qwen3\.5\-35B\-A3B\(k/E=3\.1%k/E\{=\}3\.1\\%\) represents the sparsest regime\. While its routing does show statistically significant operation\-type specialization \(Table[7](https://arxiv.org/html/2607.20427#A9.T7)\), the Huffman frequency–diversity signal remains unmeasurable: temporal position within the trace dominates operation\-type frequency as a predictor ofuiu\_\{i\}\. As Section[4](https://arxiv.org/html/2607.20427#S4)reveals, the underlying cause is functional redundancy—near\-duplicate experts are repeatedly co\-activated regardless of semantic operation, so routing codes expend bits selecting*which equivalent expert*rather than encoding*which semantic operation*\.

## Appendix Ct\-SNE Visualizations of Routing State

t\-SNE of Expert Routing Fingerprints

![Refer to caption](https://arxiv.org/html/2607.20427v1/x5.png)\(a\)Gemma\-4\-27B\-A4B
![Refer to caption](https://arxiv.org/html/2607.20427v1/x6.png)\(b\)Phi\-3\.5\-MoE
![Refer to caption](https://arxiv.org/html/2607.20427v1/x7.png)\(c\)Qwen3\.5\-35B\-A3B

Figure 7:t\-SNE projections of expert routing fingerprints colored by normalized step position \(0=early purple, 1=late yellow\)\. While Gemma\-4\-27B\-A4B and Phi\-3\.5\-MoE exhibit smooth temporal transitions, Qwen3\.5\-35B\-A3B displays distinct, isolated clusters at the extremes\. These represent thesetup phase\(early purple cluster\) and thefinal answer output\(late yellow cluster\), where the model utilizes theminimum number of experts\. This high sparsity at the start and end of the reasoning trace results in highly specific routing fingerprints that manifest as extreme clusters, whereas the intermediate steps show scattered patterns due to functional redundancy among its experts\.Figure[7](https://arxiv.org/html/2607.20427#A3.F7)shows t\-SNE projections of the expert activation patterns for all three models\. Each point represents the routing fingerprint of a single reasoning step, colored by its normalized position along the reasoning trace \(a continuous gradient from early to late steps\)\. The smooth color transitions visible in Gemma\-4\-27B\-A4B and Phi\-3\.5\-MoE provide geometric evidence that routing states evolve in an orderly temporal trajectory throughout the reasoning trace\.

#### Feature construction and embedding procedure\.

For each reasoning stepii, we construct a binary indicator vector𝐯i∈\{0,1\}L×E\\mathbf\{v\}\_\{i\}\\in\\\{0,1\\\}^\{L\\times E\}recording which experts were activated at each of theLLsensitive layers, whereEEis the total number of experts per layer\. To reduce dimensionality before embedding, we first apply PCA to retain 50 principal components, then apply t\-SNE\(Van der Maaten and Hinton,[2008](https://arxiv.org/html/2607.20427#bib.bib57)\)with perplexity 30, 1,000 iterations, and random seed 42 \(scikit\-learn defaults otherwise\)\. Points are colored by normalized step position along the reasoning trace, using a continuous colormap from early \(position 0\) to late \(position 1\) steps\.

#### Geometric Corroboration of the Huffman Hypothesis\.

The cluster topology in Figure[7](https://arxiv.org/html/2607.20427#A3.F7)provides geometric support for the Frequency\-Diversity Law\. In Gemma\-4\-27B\-A4B and Phi\-3\.5\-MoE, colors transition smoothly across the embedding from early to late positions, confirming that late\-stage routing states occupy a qualitatively distinct, more concentrated region of the expert combination space—consistent with the convergence toward shorter combinatorial codes predicted by Huffman optimality\. The smoothness of this color gradient varies systematically across architectures in a manner directly attributable to their sparsity ratiok/Ek/E: higher sparsity \(Phi,k/E=12\.5%k/E\{=\}12\.5\\%\) produces a sharper, more globally ordered gradient, while intermediate sparsity \(Gemma,k/E=6\.25%k/E\{=\}6\.25\\%\) yields a more gradual but still coherent transition\. Qwen3\.5\-35B\-A3B, with the sparsest ratio \(k/E=3\.1%k/E\{=\}3\.1\\%\), shows the least spatial separation\. This is best understood through two compounding effects revealed by Section[4](https://arxiv.org/html/2607.20427#S4)\. First, the functional profiles of the vast majority of Qwen’s 256 experts are nearly identical \(pairwise cosine similarity\>0\.90\>\{0\.90\}across all 40 layers\): many experts specialize in precisely the same operation types and temporal stages, forming large redundant clusters\. Second, within each such cluster, the router has no principled reason to select the same subset of experts for the same operation type—it can route through any of several functionally equivalent members\. As a result, steps performing the same operation \(e\.g\., multiplication\) may activate entirely different expert subsets across different problems, even though those subsets are functionally interchangeable\. The routing fingerprints for same\-operation steps are therefore scattered rather than clustered in t\-SNE space—not because the temporal compression phenomenon is absent, but because the routing code is spending most of its bits encoding*which of many equivalent experts*was chosen rather than*which semantic operation*is being performed\. This is precisely the low information density diagnosis: the code length is high but the information content is not, because the expert pool contains far more functional redundancy than semantic diversity\. Removing the redundant middle tier via Subset Difference Pruning reduces the pool to more functionally distinct experts, concentrating information content per routing bit and allowing the Huffman signal to emerge\.

## Appendix DQwen3\.5\-35B\-A3B Expert Functional Profile Analysis

### Pairwise Cosine Similarity

Table 4:Pairwise cosine similarity between co\-activation profiles of all 256 experts, aggregated across 40 layers of Qwen3\.5\-35B\-A3B\. All statistics are computed over the 782\-trace corpus; “active experts” counts experts seen at least once\.StatisticMinMeanMaxActive experts per layer243254\.6256Pairwise sim\. \(mean\)0\.8860\.9080\.949Pairwise sim\. \(P90\)0\.9830\.9870\.993Pairwise sim\. \(P99\)0\.9940\.9960\.998
### t\-SNE of Expert Functional Profiles by Pruning Tier

To verify that pruned experts do not occupy a narrow, easily\-separable functional niche, Figure[6](https://arxiv.org/html/2607.20427#S4.F6)\(main text\) plots t\-SNE embeddings of the 14\-dimensional functional profiles for all 10,185 active \(layer, expert\) pairs\. Tier\-1 \(orange\) and tier\-1\+2 \(red\) points areuniformly scatteredacross the entire embedding and show no isolated cluster separating them from the retained population\. This geometric evidence reinforces the redundancy interpretation from the similarity table above: the removed experts are near\-duplicate copies of retained experts, distributed throughout every functional region of the routing space rather than concentrated in any single operation\-type or temporal niche\.

Crucially, the scattering is not a simple consequence of high\-dimensional sparsity\. The same functional profiles that yield pairwise cosine similarity\>0\.90\>\{0\.90\}\(Table[4](https://arxiv.org/html/2607.20427#A4.T4)\) also produce overlapping t\-SNE clusters: because many experts specialize in precisely the same operation types and temporal stages, the router has no principled preference for any particular subset when routing a given step\. The routing code therefore spends bits encoding*which of many equivalent experts*was selected rather than*which semantic operation*is being performed—a direct symptom of insufficient routing information density\. Subset Difference Pruning reduces the pool to a more functionally distinct set, allowing each routing bit to carry more semantic information and enabling the Huffman signal to emerge\.

## Appendix EExpert pruning levels for Qwen3\.5\-35B\-A3B

Table 5:Expert pruning levels for Qwen3\.5\-35B\-A3B and the resulting effective sparsity ratios\. The columnk/Eeffk/E\_\{\\text\{eff\}\}measures routing sparsity in the pruned model; Gemma\-4’s ratio \(6\.25%\) is provided for reference\.PruningPruned / layerEeffE\_\{\\text\{eff\}\}k/Eeffk/E\_\{\\text\{eff\}\}Gemma\-4 ref\.10%252313\.5%6\.25%20%512053\.9%30%761804\.4%50%1281286\.2%
## Appendix FSubset Difference Pruning: Algorithmic Details

We identify redundant experts through a hierarchical similarity\-based procedure applied independently per layer to preserve core model capabilities\.

1. 1\.Profile similarity matrix\.Compute the256×256256\{\\times\}256cosine similarity matrix from co\-activation profiles \(Section[4\.1](https://arxiv.org/html/2607.20427#S4.SS1)\) to quantify semantic redundancy between all expert pairs\.
2. 2\.Establish a Preservation Tier\.Designate the top 10% of experts by total activation frequency \(∼25\{\\sim\}25per layer\) as a “protected core” that is exempt from pruning\. These are the experts most consistently activated across all operation types and reasoning stages\.
3. 3\.Greedy Redundancy Removal\.Among the remaining unprotected experts, iteratively identify the most similar pair\(i,j\)\(i,j\)and remove the expert with the lower total activation frequency\. This step removes functional near\-duplicates while retaining the more informative member of each similar pair\.
4. 4\.Repeat until target fraction reached\.Continue greedy removal until the desired total pruning fraction \(e\.g\., 20% or 30%\) is achieved\.

At inference time, the removed experts are masked via a forward hook on the gate module: their router logits are set to−∞\-\\inftybefore top\-kkselection, ensuring they are never dispatched to\. This forces the router to redistribute load among the remaining representative experts*without modifying any weights*\.

The “Subset Difference” in the method’s name refers to the mask construction: the set of experts pruned at levelL2L\_\{2\}is the*set difference*of the experts selected at two adjacent pruning thresholds,setL2∖setL1\\text\{set\}\_\{L\_\{2\}\}\\setminus\\text\{set\}\_\{L\_\{1\}\}, deliberately skipping the top\-10% core tier\. This asymmetric removal targets the*middle*of the redundancy spectrum—experts that are neither maximally representative nor maximally rare—which our analysis identifies as the source of functional overlap in Qwen3\.5\-35B\-A3B\.

## Appendix GFull Unique Expert Trajectory Figures

Figures[8](https://arxiv.org/html/2607.20427#A7.F8)–[10](https://arxiv.org/html/2607.20427#A7.F10)present the unique expert count trajectory for all three models across all four datasets\. Thexx\-axis is the normalized reasoning step position \(0 = first step, 1 = last step\); theyy\-axis is mean unique experts per sensitive layer per step; the dashed line is an OLS linear trend with slope and Pearsonrrannotated\.

Unique Expert Count Trajectory of Gemma\-4\-27B\-A4B

![Refer to caption](https://arxiv.org/html/2607.20427v1/figs/gemma_gsm8k_fig2_unique_expert_trajectory.png)\(a\)GSM8K
![Refer to caption](https://arxiv.org/html/2607.20427v1/figs/gemma_aqua_fig2_unique_expert_trajectory.png)\(b\)AQuA
![Refer to caption](https://arxiv.org/html/2607.20427v1/figs/gemma_compmath_mcq_fig2_unique_expert_trajectory.png)\(c\)CompMath\-MCQ

Figure 8:Unique expert count trajectory forGemma\-4\-27B\-A4Bacross three datasets \(GSM8K, AQuA, and CompMath\-MCQ\); the MATH trajectory appears in Figure[2](https://arxiv.org/html/2607.20427#S3.F2)\. All datasets show consistent monotonic decline\.Unique Expert Count Trajectory of Phi\-3\.5\-MoE

![Refer to caption](https://arxiv.org/html/2607.20427v1/figs/phi_gsm8k_fig2_unique_expert_trajectory.png)\(a\)GSM8K
![Refer to caption](https://arxiv.org/html/2607.20427v1/figs/phi_aqua_fig2_unique_expert_trajectory.png)\(b\)AQuA
![Refer to caption](https://arxiv.org/html/2607.20427v1/figs/phi_compmath_mcq_fig2_unique_expert_trajectory.png)\(c\)CompMath\-MCQ

Figure 9:Unique expert count trajectory forPhi\-3\.5\-MoEacross three datasets \(GSM8K, AQuA, and CompMath\-MCQ\); the MATH trajectory appears in Figure[2](https://arxiv.org/html/2607.20427#S3.F2)\. The oscillating but overall declining pattern is consistent across both open\-ended and MCQ tasks\.Unique Expert Count Trajectory of Qwen3\.5\-35B\-A3B

![Refer to caption](https://arxiv.org/html/2607.20427v1/figs/qwen_gsm8k_fig2_unique_expert_trajectory.png)\(a\)GSM8K
![Refer to caption](https://arxiv.org/html/2607.20427v1/figs/qwen_aqua_fig2_unique_expert_trajectory.png)\(b\)AQuA
![Refer to caption](https://arxiv.org/html/2607.20427v1/figs/qwen_compmath_mcq_fig2_unique_expert_trajectory.png)\(c\)CompMath\-MCQ

Figure 10:Unique expert count trajectory forQwen3\.5\-35B\-A3Bacross three datasets \(GSM8K, AQuA, and CompMath\-MCQ\); the MATH trajectory appears in Figure[2](https://arxiv.org/html/2607.20427#S3.F2)\. The inverted\-U pattern \(exploration then late collapse\) is consistent across tasks, reflecting the model’s divergent exploration strategy under fine\-grained \(k/E=3\.1%k/E=3\.1\\%\) routing\.
## Appendix HEmpirical Validation of Temporal Compression \(Eq\.[5](https://arxiv.org/html/2607.20427#S3.E5)\)

We estimateH^​\(O∣step​i\)\\hat\{H\}\(O\\mid\\text\{step\}\\,i\)by partitioning each trace into ten equal\-width normalized\-position bins and computing the entropy of the empirical operation\-type distribution within each bin\. Figure[3](https://arxiv.org/html/2607.20427#S3.F3)\(main paper\) plots the resulting\(H^,u¯i\)\(\\hat\{H\},\\bar\{u\}\_\{i\}\)pairs for GSM8K\.

## Appendix IExpert Routing Specialization by Operation Type

The Frequency\-Diversity Law \(Section[3](https://arxiv.org/html/2607.20427#S3)\) establishes that frequent operations receive shorter routing codes—fewer active experts per step\. A natural follow\-up question is whether these codes are merely*compact*, or also*consistent*: do steps of the same operation type, drawn from entirely unrelated problems, activate the same expert subsets?

#### Code Consistency Across Problem Instances\.

Beyond the frequency\-diversity correspondence, the “short codes” assigned to late\-stage reasoning steps are not arbitrary: they encode the*type of arithmetic operation*in a content\-independent manner\. Steps performing division drawn from unrelated problems activate nearly identical expert subsets \(JJup to0\.9470\.947\), whereas cross\-operation pairs yield substantially lower overlap \(Table[8](https://arxiv.org/html/2607.20427#A9.T8)\)\. This*routing fingerprint consistency*property is a direct implication of Huffman optimality: each semantic operation typeo∈𝒪o\\in\\mathcal\{O\}is assigned a stable, reproducible routing fingerprintℰ∗​\(o\)\\mathcal\{E\}^\{\*\}\(o\)that is invariant to surface\-level problem content\. Crucially, this property holds across all three architectures, indicating that code consistency is a structural consequence of routing optimisation rather than any architecture\-specific inductive bias\.

To quantify this, we classify each thought step by semantic operation type \(add/subtract, multiply/divide, problem setup, restatement, other\) using a lightweight keyword classifier whose trigger words are listed in Table[6](https://arxiv.org/html/2607.20427#A9.T6)\. Routing overlap is measured by the pairwise Jaccard similarity between expert sets:

Table 6:Keyword triggers for the operation\-type classifier\. A step is assigned to the first matching category in the order shown; steps matching none of the listed keywords are labelledother\.CategoryTrigger keywordssetuplet, let’s, suppose, given that, we know, we have,step 1, first, define, note thatadd/subtractadd, subtract, sum, total, plus, minus, difference,increase, decrease, more than, less than,\+,\-multiply/dividemultiply, times, product,×\\times, per, each, rate,fraction, divide, divided, ratio, percent,%J​\(ℰa\(l\),ℰb\(l\)\)=\|ℰa\(l\)∩ℰb\(l\)\|\|ℰa\(l\)∪ℰb\(l\)\|\.J\\\!\\left\(\\mathcal\{E\}\_\{a\}^\{\(l\)\},\\,\\mathcal\{E\}\_\{b\}^\{\(l\)\}\\right\)=\\frac\{\\left\|\\mathcal\{E\}\_\{a\}^\{\(l\)\}\\cap\\mathcal\{E\}\_\{b\}^\{\(l\)\}\\right\|\}\{\\left\|\\mathcal\{E\}\_\{a\}^\{\(l\)\}\\cup\\mathcal\{E\}\_\{b\}^\{\(l\)\}\\right\|\}\.\(6\)We then compute for each model–dataset pair:

- •J¯intra\\bar\{J\}\_\{\\text\{intra\}\}: mean pairwise Jaccard between steps of the*same*operation type drawn from*different*problems\.
- •J¯inter\\bar\{J\}\_\{\\text\{inter\}\}: mean pairwise Jaccard between arithmetic and non\-arithmetic \(explain/restate\) steps\.

A positiveΔ=J¯intra−J¯inter\\Delta=\\bar\{J\}\_\{\\text\{intra\}\}\-\\bar\{J\}\_\{\\text\{inter\}\}confirms that routing encodes operation type beyond problem identity\.

Table[7](https://arxiv.org/html/2607.20427#A9.T7)showsΔ\>0\\Delta\>0in all 12 model–dataset combinations, with Mann–Whitneyp≤10−163p\\leq 10^\{\-163\}in every case\. The absoluteJ¯\\bar\{J\}values vary across models because they reflect the expert sparsity ratiok/Ek/E\(Phi:2/16=12\.5%2/16=12\.5\\%; Gemma:8/128=6\.25%8/128=6\.25\\%; Qwen:8/256=3\.1%8/256=3\.1\\%\)—higher sparsity means higher expected baseline overlap—but the*relative*gapΔ∈\[\+0\.02,\+0\.10\]\\Delta\\in\[\+0\.02,\+0\.10\]is consistent across all architectures\.

Table 7:Expert routing specialization by arithmetic operation type \(Mann–Whitney U test\)\.Δ\>0\\Delta\>0in all 12 cases confirms that the router encodes*what computation is performed*, independent of problem content and model architecture\. AbsoluteJ¯\\bar\{J\}values scale with sparsity ratiok/Ek/E\(shown asEEcolumn\)\.ModelDatasetJ¯intra\\bar\{J\}\_\{\\text\{intra\}\}J¯inter\\bar\{J\}\_\{\\text\{inter\}\}Δ\\Deltapp\-valueEEGemma\-4\-27B\-A4BGSM8K0\.5260\.475\+0\.051<10−300<10^\{\-300\}128MATH0\.5430\.502\+0\.040<10−300<10^\{\-300\}128AQuA0\.5280\.475\+0\.053<10−300<10^\{\-300\}128CompMath\-MCQ0\.5300\.458\+0\.072<10−300<10^\{\-300\}128Phi\-3\.5\-MoEGSM8K0\.8090\.755\+0\.055<10−300<10^\{\-300\}16MATH0\.7190\.701\+0\.018<10−163<10^\{\-163\}16AQuA0\.7750\.727\+0\.048<10−300<10^\{\-300\}16CompMath\-MCQ0\.7850\.689\+0\.095<10−300<10^\{\-300\}16Qwen3\.5\-35B\-A3BGSM8K0\.3310\.259\+0\.072<10−300<10^\{\-300\}256MATH0\.3090\.268\+0\.042<10−300<10^\{\-300\}256AQuA0\.3150\.236\+0\.079<10−300<10^\{\-300\}256CompMath\-MCQ0\.3420\.251\+0\.090<10−300<10^\{\-300\}256
#### Qualitative illustration\.

Table[8](https://arxiv.org/html/2607.20427#A9.T8)shows the highest\-Jaccard routing matches found between steps from*entirely different problems*performing the same arithmetic operation\. Despite sharing no numbers, variables, or narrative context, these step pairs activate nearly identical expert subsets—with Jaccard up toJ=0\.947J=0\.947\(Phi\-3\.5\-MoE on GSM8K\)\. Two steps that both say “divide both sides byccto solve forxx” \(differentcc, different equations, different problems\) route through virtually the same expert combination: the router has learned a stable, problem\-agnostic “division” code—exactly what the Huffman analogy predicts\.

Table 8:Highest\-Jaccard cross\-problem routing matches for same\-type arithmetic steps\. Step pairs are drawn from unrelated questions; text is truncated at 60 characters\. High Jaccard across unrelated problems confirms operation\-type encoding is content\-independent\.ModelDatasetJJStep pair \(truncated\)Gemma\-4\-27B\-A4BMATH0\.866“Divide both sides by 2 to solve forxx\.”vs\.“Divide both sides by 3 to solve fornn\.”Phi\-3\.5\-MoEGSM8K0\.947“Step 4: Calculate the total amount of time Melissa spends on…”vs\.“Step 4: Calculate the overall probability that Marcus won’t…”Qwen3\.5\-35B\-A3BCOMPMATH0\.658“The determinant is calculated as 3 multiplied by 5 minus…”vs\.“The determinant is computed as 2 times 14 minus−5\-5times−5\-5\.”

## Appendix JQuantitative Huffman Validation:u¯\(o\)∝−log⁡p​\(o\)\\bar\{u\}^\{\(o\)\}\\propto\{\-\}\\log p\(o\)

This appendix reports the full numerical results for the quantitative Huffman test \(Eq\.[4](https://arxiv.org/html/2607.20427#S3.E4)in the main paper\) and explains why Qwen3\.5\-35B\-A3B is excluded from the claim\.

#### Setup\.

We classify each thought step into one of four types using a keyword classifier:*add/subtract*\(keywords: sum, plus, minus, difference, …\),*multiply/divide*\(keywords: times, product, percent, ratio, …\),*setup*\(keywords: let, given, suppose, define, …\), and*other*\. For each model we compute the empirical frequencyp​\(o\)p\(o\)and the mean unique expert countu¯\(o\)\\bar\{u\}^\{\(o\)\}across all correct traces and all four datasets\.

In total this yields756 correct traces\(6,688 classified steps, excluding restate\) for Gemma\-4\-27B\-A4B, and618 correct traces\(4,807 classified steps\) for Phi\-3\.5\-MoE\. Each scatter point in Figure[1](https://arxiv.org/html/2607.20427#S1.F1)therefore aggregates hundreds to thousands of individual step observations, providing strong statistical backing for the Spearmanρ=1\.00\\rho=1\.00result\.

#### Gemma\-4\-27B\-A4B and Phi\-3\.5\-MoE: perfect Huffman rank order\.

Table[9](https://arxiv.org/html/2607.20427#A10.T9)reports the per\-type statistics for the two models that satisfy Eq\.[4](https://arxiv.org/html/2607.20427#S3.E4)\. Both models achieveSpearmanρ=1\.00\\rho=1\.00\(p<0\.005p<0\.005\) and Pearsonr\>0\.97r\>0\.97\(p<0\.04p<0\.04\)\. The rank orderu¯​\(other\)<u¯​\(add/sub\)<u¯​\(mult/div\)<u¯​\(setup\)\\bar\{u\}\(\\text\{other\}\)<\\bar\{u\}\(\\text\{add/sub\}\)<\\bar\{u\}\(\\text\{mult/div\}\)<\\bar\{u\}\(\\text\{setup\}\)mirrors frequency rank in reverse across both architectures, confirming the Huffman prediction is not model\-specific\.

Table 9:Per\-type statistics for the quantitative Huffman test \(aggregated over all four datasets, correct traces only\)\.nn= number of classified steps \(restate excluded\);p​\(o\)p\(o\)= empirical frequency;u¯\(o\)\\bar\{u\}^\{\(o\)\}= mean unique experts per layer\. Gemma: 756 questions, 6,688 steps total; Phi: 618 questions, 4,807 steps total; Qwen: shown for reference only \(does not satisfy Eq\.[4](https://arxiv.org/html/2607.20427#S3.E4)\)\.ModelOperation typennp​\(o\)p\(o\)−log⁡p​\(o\)\-\\log p\(o\)u¯\(o\)\\bar\{u\}^\{\(o\)\}Gemma\-4\-27B\-A4BOther28400\.3740\.9837\.3Add/Subtract17360\.2291\.4740\.9Mult/Divide14390\.1901\.6641\.4Setup6730\.0892\.4253\.6Phi\-3\.5\-MoEOther16960\.3041\.1911\.6Add/Subtract11320\.2031\.5912\.2Setup10470\.1881\.6712\.5Mult/Divide9320\.1671\.7912\.6Qwen3\.5\-35B\-A3BOther34560\.4120\.8958\.3Add/Subtract17560\.2091\.5666\.9Mult/Divide14370\.1711\.7662\.2Setup7990\.0952\.3545\.3
#### Why Qwen3\.5\-35B\-A3B violates Eq\.[4](https://arxiv.org/html/2607.20427#S3.E4)\.

WithE=256E\{=\}256experts andk=8k\{=\}8\(k/E=3\.1%k/E=3\.1\\%\), the expected pairwise overlap between any two independently\-sampled steps is nearly zero\. Table[9](https://arxiv.org/html/2607.20427#A10.T9)shows the resulting disorder:*setup*steps \(the rarest type,p=0\.095p\{=\}0\.095\) have the*lowest*u¯\\bar\{u\}\(45\.3\), while*add/subtract*steps \(more common,p=0\.209p\{=\}0\.209\) have the*highest*u¯\\bar\{u\}\(66\.9\)\. As explained in Section[4\.2](https://arxiv.org/html/2607.20427#S4.SS2), the root cause is Qwen’s inverted\-U trajectory: expert diversity peaks mid\-trace and collapses toward the end, so the position of each operation type along this curve—not its frequency—determinesu¯\(o\)\\bar\{u\}^\{\(o\)\}\. Qwen therefore exhibits the qualitative temporal compression pattern \(inverted\-U exploration followed by late\-stage collapse\) but does not satisfy the quantitative frequency–diversity relationship of Eq\.[4](https://arxiv.org/html/2607.20427#S3.E4)\.

## Appendix KLimitations

*\(1\) Domain scope\.*All experiments are restricted to mathematical reasoning benchmarks \(GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2607.20427#bib.bib12)\), MATH\(Hendryckset al\.,[2021](https://arxiv.org/html/2607.20427#bib.bib13)\), AQuA\(Linget al\.,[2017](https://arxiv.org/html/2607.20427#bib.bib14)\), CompMath\-MCQ\(Loet al\.,[2025](https://arxiv.org/html/2607.20427#bib.bib52)\)\)\. Whether the Frequency\-Diversity Law generalizes to code generation, factual question\-answering, or open\-ended generation remains an open question\.

*\(2\) Operation\-type classifier\.*We classify reasoning steps via a keyword\-based heuristic into four types\. With only\|𝒪\|=4\|\\mathcal\{O\}\|\{=\}4categories, the Pearson correlation in the Huffman scatter is estimated from four points per model; the result is not statistically significant for Qwen’s baseline \(p=0\.37p\{=\}0\.37\) and only marginally so for the two Huffman\-compliant models \(p<0\.05p\{<\}0\.05\)\. A richer, model\-assisted classifier producing more fine\-grained operation types would increase statistical power\.

*\(3\) Benchmark scope for Qwen pruning\.*The pruning evaluation \(Table[2](https://arxiv.org/html/2607.20427#S4.T2)\) is conducted on GSM8K; accuracy on the full 1,319\-question benchmark and on the other three datasets \(MATH, AQuA, CompMath\-MCQ\) under each pruning level remains to be evaluated\.

*\(4\) Huffmanrras a training\-time signal\.*Our results suggest thatrrcould be monitored during training as an early indicator of redundancy formation: a persistent negativerrwould signal that load\-balancing regularization is creating functional near\-duplicates faster than the model develops meaningful expert specialization\. Whether onlinerrmonitoring can guide adaptive regularization schedules \(or serve as a stopping criterion for load\-balancing loss\) remains an open question for future work\.