Beyond the Previous Layer: Residual Predictive Structure in Sparse MoE Routing

# Beyond the Previous Layer:Residual Predictive Structure in Sparse MoE Routing
Source: [https://arxiv.org/html/2609.17940](https://arxiv.org/html/2609.17940)
\\paperurl

https://github\.com/withfanta/moe\-routing\-dynamics/tree/main/code

Hao LiAffiliation:Department of Informatics, Graduate School of Informatics and Engineering, The University of Electro\-Communications, Tokyo, Japan Source Code:[https://github\.com/withfanta/moe\-routing\-dynamics/tree/main/code](https://github.com/withfanta/moe-routing-dynamics/tree/main/code)Yasuyuki TaharaAffiliation:Department of Informatics, Graduate School of Informatics and Engineering, The University of Electro\-Communications, Tokyo, Japan Source Code:[https://github\.com/withfanta/moe\-routing\-dynamics/tree/main/code](https://github.com/withfanta/moe-routing-dynamics/tree/main/code)Yuichi Sei\*Affiliation:Department of Informatics, Graduate School of Informatics and Engineering, The University of Electro\-Communications, Tokyo, Japan Source Code:[https://github\.com/withfanta/moe\-routing\-dynamics/tree/main/code](https://github.com/withfanta/moe-routing-dynamics/tree/main/code)

###### Abstract

Sparse mixture\-of\-experts models route each token through a sequence of expert selections\. We ask whether the immediately preceding selection adequately summarizes this trajectory for predicting the next router\. Using frozen OLMoE and JetMoE models, we measure the held\-out predictive gain from earlier expert selections while retaining the most recent selection as a common baseline\. In OLMoE, extending the history from one to eleven layers raises router\-logitR2R^\{2\}from 0\.59879 to 0\.66544\. A preregistered JetMoE replication yields four\-layer gains of 0\.14275 and 0\.20528 at two target depths, with paired bootstrap intervals above zero\. These gains survive nonlinear decoding: adding history to a small multilayer perceptron improvesR2R^\{2\}by 0\.17137 and 0\.21861, whereas nonlinear decoding of the recent state alone adds 0\.00139 and 0\.00936 over a linear probe\. Parameter\-matched controls preserve the advantage, and cross\-fitted history residuals predict target residuals withR2R^\{2\}of 0\.20549 and 0\.23556\. These findings identify residual predictive structure in expert\-selection trajectories beyond adjacent\-layer persistence\.

11footnotetext:Corresponding author: Yuichi Sei\. E\-mails:[r2540015@gl\.cc\.uec\.ac\.jp](mailto:[email protected]);[tahara@uec\.ac\.jp](mailto:[email protected]);[seiuny@uec\.ac\.jp](mailto:[email protected])\.
ORCID: Hao Li,[0009\-0004\-2623\-286X](https://orcid.org/0009-0004-2623-286X); Yasuyuki Tahara,[0000\-0002\-1939\-4455](https://orcid.org/0000-0002-1939-4455); Yuichi Sei,[0000\-0002\-2552\-6717](https://orcid.org/0000-0002-2552-6717)\.## 1Introduction

A token traversing a sparse mixture\-of\-experts \(MoE\) model acquires an expert\-selection trajectory across depth\. The immediately preceding selection offers a natural summary of that trajectory\. How much predictive structure remains in earlier selections once this recent state is already available? Answering this question distinguishes adjacent\-layer persistence from the additional value of a deeper routing history\.

Cross\-layer routing structure has motivated both analysis and architecture design\. Read\-ME decouples routing from the backbone to support pre\-gating\[[1](https://arxiv.org/html/2609.17940#bib.bib4)\]; RMoE propagates a recurrent routing state across layers\[[2](https://arxiv.org/html/2609.17940#bib.bib3)\]; and PathMoE shares router parameters within consecutive blocks\[[3](https://arxiv.org/html/2609.17940#bib.bib5)\]\. Labzin et al\. align router\-control subspaces and study shared linear dynamics\[[4](https://arxiv.org/html/2609.17940#bib.bib6)\]\. Expert paths are also recorded and replayed during MoE reinforcement learning\[[5](https://arxiv.org/html/2609.17940#bib.bib7)\], while HeRo uses routing history for dynamic layer skipping\[[6](https://arxiv.org/html/2609.17940#bib.bib8)\]\. Our question concerns the incremental predictive value of older selections in existing pretrained sparse MoEs, with the immediately preceding selection explicitly included in both sides of the comparison\.

The evidence follows a sequence of increasingly specific questions\. Selected\-expert provenance predicts later routing better than fused expert outputs\. Decomposing that provenance identifies expert selection as the dominant accessible signal\. Holding the most recent selection fixed then reveals an additional history gain in OLMoE\[[7](https://arxiv.org/html/2609.17940#bib.bib1)\], which a preregistered JetMoE\[[8](https://arxiv.org/html/2609.17940#bib.bib2)\]experiment replicates at two depths\. Finally, nonlinear decoders and cross\-fitted residual prediction test whether the gain arises simply because a linear probe cannot decode the recent state\.

We make three contributions: \(1\) an evaluation of routing\-history value beyond an explicit previous\-layer baseline; \(2\) evidence across two pretrained MoE architectures, including a preregistered replication; and \(3\) nonlinear and parameter\-matched controls, together with residual prediction, that preserve the history effect\. The resulting measurement provides a concrete target for understanding which aspects of a routing trajectory a local selection state summarizes\.

Table 1:Where the predictive signal comes from\.Held\-out router\-logitR2R^\{2\}in OLMoE\. EIPC retains selected contributions in identity slots; EPD separates selection identity from contribution content\. EIPC and EPD use different samples and compression budgets, so comparisons are within each panel\. The EPD single\-layer curve uses 16 PCA components per representation\.EIPC: provenance \(64 components\)TargetFusedIdentityShuffledL80\.190160\.29939\-0\.00149L120\.287050\.462110\.00396TargetId\.−\-FusedId\.−\-Shuf\.L80\.109230\.30088L120\.175060\.45815Mean0\.142150\.37952EPD: L12 \(32 components\)RepresentationR2R^\{2\}Fused0\.20114Selection path0\.66969Content by router rank0\.06732Full provenance0\.34007Path \+ content \(post\-hoc\)0\.66728
EPD: single\-layer prediction of L12Source layerSelection pathContent10\.142530\.0091420\.25597\-0\.0072130\.35859\-0\.0060540\.416640\.0055650\.40384\-0\.0104860\.44991\-0\.0081470\.44797\-0\.0065780\.493810\.0213990\.538500\.00420100\.572160\.00700110\.599270\.01017Mean0\.425380\.00173

## 2Measuring Historical Value

### 2\.1State, target, and comparison

For a fixed token, letSlS\_\{l\}be the binary vector indicating the experts selected at layerll, and letglg\_\{l\}be that layer’s native router\-logit vector\. Layer numbers are one\-based\. History here runs across model depth for the same token\. It does not denote earlier tokens in the sequence\.

A probeFkF\_\{k\}predictsglg\_\{l\}from thekkimmediately preceding selection states, denotedSl−k:l−1S\_\{l\-k:l\-1\}\. We evaluate

ℛl\(k\)=RTEST2\(Fk\(Sl−k:l−1\),gl\),\\mathcal\{R\}\_\{l\}\(k\)=R^\{2\}\_\{\\mathrm\{TEST\}\}\\\!\\left\(F\_\{k\}\(S\_\{l\-k:l\-1\}\),g\_\{l\}\\right\),\(1\)and measure the historical gain

Δl​\(k\)=ℛl​\(k\)−ℛl​\(1\)\.\\Delta\_\{l\}\(k\)=\\mathcal\{R\}\_\{l\}\(k\)\-\\mathcal\{R\}\_\{l\}\(1\)\.\(2\)The one\-layer baseline measures adjacent\-layer predictability\. A positiveΔl​\(k\)\\Delta\_\{l\}\(k\)measures the improvement available to the tested probe when it also receives older selections\. Throughout,R2R^\{2\}is the uniform average of the coordinate\-wise scores\. Target logits are centered within each sample and standardized using FIT statistics\.

### 2\.2Frozen evaluation and decoder controls

All backbones are frozen, non\-quantized, and evaluated in FP16:OLMoE\-1B\-7B\-0125\(revision9b0c1aa8\) andjetmoe\-8b\(d8fd02cc\)\. OLMoE selects eight of 64 MLP experts; JetMoE selects two of eight\. We use WikiText\-103\-raw\-v1’s training split\[[9](https://arxiv.org/html/2609.17940#bib.bib9)\], partitioned into disjoint FIT and TEST blocks for the probes\. Each 129\-token block contributes one experimental token at position 127 of its 128\-token context\. Holdout is relative to probe fitting; backbone pretraining overlap is unassessed\. Full model identifiers, revisions, and protocols are in the linked code archive\.

EIPC uses 1,024 FIT and 1,024 TEST samples; EPD and its post\-hoc RMO analysis share 512/256 samples\. EPD excludes the earlier EIPC and XEC blocks\. JetMoE replication RMC uses 512/256 samples; NHD reuses those exact captures and split\. All scalers and principal component analysis \(PCA\) transforms are fitted on FIT\. Linear probes use ridge regression withα=1\\alpha=1\.

For the nonlinear comparison, writeR=Sl−1R=S\_\{l\-1\},H=Sl−4:l−2H=S\_\{l\-4:l\-2\}, andY=glY=g\_\{l\}\. We compare predictors ofYYfromRRand from\[H;R\]\[H;R\]\. The recent state has eight raw binary coordinates and older history has 24; NHD uses no PCA\. Both MLPs have one 32\-unit GELU hidden layer\. A 77\-unit recent\-only MLP matches the history model’s parameter count to within three parameters \(1,317 versus 1,320\)\. Training uses full\-batch AdamW, learning rate10−310^\{\-3\}, weight decay10−410^\{\-4\}, at most 300 epochs, and validation patience 30\. A fixed 384/128 split within FIT selects checkpoints; initialization seeds are 42, 123, and 2026\.

Table 2:History improves prediction beyond the previous layer\.Top: held\-outR2R^\{2\}for nested history windows;kkincludes the immediately preceding layer\. Middle: OLMoE gains overk=1k=1and between consecutive windows\. Bottom: JetMoE gains, with 95% paired bootstrap intervals for the preregisteredk=4k=4comparison \(10,000 resamples\)\. Dashes denote unavailable windows\.Model / targetk=1k=1k=2k=2k=4k=4k=8k=8k=11k=11OLMoE / L120\.598790\.623200\.647050\.662650\.66544JetMoE / L120\.209240\.349820\.351990\.38275–JetMoE / L200\.138980\.249360\.344250\.35463–OLMoE cumulative gain–0\.024410\.048260\.063870\.06665OLMoE stepwise gain–0\.024410\.023850\.015610\.00279
JetMoE targetΔ2\\Delta\_\{2\}Δ4\\Delta\_\{4\}Δ8\\Delta\_\{8\}95% CI forΔ4\\Delta\_\{4\}L120\.140570\.142750\.17351\[0\.10998, 0\.17916\]L200\.110380\.205280\.21565\[0\.16891, 0\.24396\]

To examine the remaining variation directly, separate MLPs predictYYandHHfromRR\. Their residuals are

ϵY=Y−f⁡\(R\),ϵH=H−h⁡\(R\),\\epsilon\_\{Y\}=Y\-f\(R\),\\qquad\\epsilon\_\{H\}=H\-h\(R\),\(3\)whereffandhhare the respective recent\-state predictors\. Two\-fold cross\-fitting constructs FIT residuals: each 256\-sample fold is predicted by models fitted on the other fold, with validation internal to that training fold\. A ridge probe learnsϵH↦ϵY\\epsilon\_\{H\}\\mapsto\\epsilon\_\{Y\}\. TEST residualizers use the original FIT training/validation protocol; all residualizers use seed 42\. A fixed permutation of TEST history residuals provides a descriptive control\. This measures residual predictability relative to the fitted decoder family\.

## 3Evidence

### 3\.1Expert selection carries the accessible signal

EIPC compares three histories of selected\-expert contributions: their fused sum, vectors retained in expert\-identity slots, and the same vectors with identity slots shuffled\. Contributions are projected from 2,048 to 32 dimensions before each full history receives a common 64\-component PCA budget\. Identity preservation improvesR2R^\{2\}over fusion by 0\.10923 and 0\.17506; its mean gain is 0\.14215 \(Table[1](https://arxiv.org/html/2609.17940#S1.T1)\)\. The shuffled comparison measures accessibility under compression, since the fused sum remains recoverable before compression\.

EPD separates binary selection paths, rank\-ordered contribution content, fused outputs, and full provenance under a 32\-component budget\. The path alone reachesR2=0\.66969R^\{2\}=0\.66969, while content reaches 0\.06732\. Adding content to the path gives 0\.66728\. Single\-layer path prediction generally strengthens toward the target, from 0\.14253 to 0\.59927, while content remains close to zero\. The full\-provenance representation retains 15\.18% of its variance under PCA, versus 52\.17% for the path\. These results identify selection identity as the strongest accessible predictor and motivate testing how much of its signal comes from the previous layer\.

Table 3:Nonlinear decoding preserves the historical advantage \(NHD\)\.Raw\-state ridge scores and MLP scores for all three initialization seeds\.AAis the recent\-only MLP gain over recent\-only ridge;BBis the history MLP gain over the recent\-only MLP;BmatchedB\_\{\\mathrm\{matched\}\}uses the parameter\-matched recent\-only MLP\. Residual scores predictϵY\\epsilon\_\{Y\}fromϵH\\epsilon\_\{H\}and use a different target from the joint prediction scores\. The permutation control shuffles TEST history residuals with the fitted residual probe held fixed\.TargetDecoder / inputParametersSeed 42Seed 123Seed 2026Mean / fixedL12Ridge /RR––––0\.20972L12Ridge /\[H;R\]\[H;R\]––––0\.37188L12MLP /RR5520\.208710\.212060\.212570\.21111L12Matched MLP /RR1,3170\.220510\.217420\.216330\.21809L12MLP /\[H;R\]\[H;R\]1,3200\.380180\.386770\.380510\.38249L20Ridge /RR––––0\.13948L20Ridge /\[H;R\]\[H;R\]––––0\.35382L20MLP /RR5520\.146780\.152140\.147600\.14884L20Matched MLP /RR1,3170\.146020\.152100\.146880\.14834L20MLP /\[H;R\]\[H;R\]1,3200\.368190\.363390\.370780\.36745
TargetLinear gainAABBBmatchedB\_\{\\mathrm\{matched\}\}ResidualR2R^\{2\}PermutedR2R^\{2\}L120\.162160\.001390\.171370\.164400\.20549\-0\.33325L200\.214340\.009360\.218610\.219120\.23556\-0\.32149

### 3\.2Older selections add value across architectures

RMO holds the OLMoE Layer\-11 representation fixed while extending history toward earlier layers\. The recent state and each older\-history window receive separate 16\-component PCA transforms\. Every ridge input has 32 coordinates; the baseline pads the older block with zeros\. Prediction of Layer 12 increases from 0\.59879 to 0\.66544 \(Table[2](https://arxiv.org/html/2609.17940#S2.T2)\)\. Gains arrive most strongly in the nearby layers; extending from eight to eleven historical layers adds 0\.00279\. This exploratory analysis uses the already\-observed EPD samples\.

RMC tests the same comparison in JetMoE at preregistered target Layers 12 and 20\. The recent eight\-dimensional selection vector remains unchanged; older history receives eight PCA components, yielding a 16\-dimensional input for every history window\. The frozen primary comparison isk=4k=4againstk=1k=1\. ItsΔ4\\Delta\_\{4\}values are 0\.14275 and 0\.20528, and both paired bootstrap intervals exclude zero\. All 10,000 resamples give positive differences at both targets \(seed 314159\)\. Both targets satisfy the frozen rule: a positive baseline,Δ4≥0\.02\\Delta\_\{4\}\\geq 0\.02, and an interval lower bound above zero\.

JetMoE has separate attention and MLP mixtures\. The captured MLP router was identified by object identity and matched direct recomputation from its input exactly \(maximum absolute logit difference00\); it differed from the attention router in every block\. Thus the replication concerns MLP expert selection throughout\.

### 3\.3The gain survives nonlinear decoding

One explanation for a linear history gain is that older selections expose structure already present nonlinearly in the recent state\. NHD tests that explanation on the frozen JetMoE captures with uncompressed selection vectors\. Recent\-only MLPs improve over ridge by 0\.00139 and 0\.00936, whereas adding history improves the MLP by 0\.17137 and 0\.21861 \(Table[3](https://arxiv.org/html/2609.17940#S3.T3)\)\. The corresponding parameter\-matched gains are 0\.16440 and 0\.21912\. Every initialization gives a positive history gain against both recent\-only controls\.

Cross\-fitted history residuals also predict the target residuals:R2=0\.20549R^\{2\}=0\.20549and0\.235560\.23556\. Permuting TEST history residuals reduces these scores to \-0\.33325 and \-0\.32149\. The joint and residual results agree: the tested nonlinear recent\-state decoders leave substantial predictive structure accessible from older selections\. Both targets receive the frozen descriptive classificationRESIDUAL\-HISTORY\-VALUE\.

## 4Discussion and Scope

Expert trajectories predict routing, older selections add value given the latest selection, and stronger recent\-only decoders retain the gap\. Across the OLMoE exploration and JetMoE replication, the local selection pattern is an incomplete predictive summary under these probes\.

The state definition matters\.RRrecords selected identities, rather than the full hidden representation or the preceding router’s continuous logits\. The findings establish model\-relative residual predictive value; they do not establish causal memory, a formal Markov order, or an information\-theoretic proof thatHHcontains information mathematically absent fromRR\. History comparisons cover two models, three target/model combinations, one corpus, and one position per block\. NHD reuses the replication samples\. The fixed PCA budget changes the representation as the history window grows, so these curves do not identify a memory\-decay law\. Mapping effective historical depth across models and layers remains future work\.

#### Prediction and actionability\.

The closed XEC experiment evaluated a five\-action Layer\-12 routing policy on OLMoE \(2,048 training, 512 validation, 1,024 TEST samples; three seeds\)\. Mean next\-token NLLs were 2\.692297 for native routing, 2\.690444 for a current\-only policy, 2\.689735 for a fused cache, 2\.690791 for an expert cache, and 2\.647312 for the five\-action oracle\. Expert\-cache differences versus native, current\-only, and fused routing were respectively−0\.001506\-0\.001506,\+0\.000347\+0\.000347, and\+0\.001056\+0\.001056; their 95% paired intervals were\[−0\.005743,0\.002760\]\[\-0\.005743,0\.002760\],\[−0\.000761,0\.001670\]\[\-0\.000761,0\.001670\], and\[−0\.000023,0\.002156\]\[\-0\.000023,0\.002156\]\. The experiment did not demonstrate reliable NLL improvement\. Validation selected the untrained initialization for every learned policy, which bounds the interpretation to this training and decision setup\. The result distinguishes measurable historical structure from its successful use in routing control\.

## 5Conclusion

Earlier expert selections improve held\-out routing prediction beyond the previous\-layer baseline in OLMoE and JetMoE\. Nonlinear and parameter\-matched decoders preserve the gain, and residualized history predicts the residual target\. These results identify predictive structure in routing trajectories that the tested recent\-state summaries leave unexplained\.

## References

- \[1\]\(2024\)Read\-ME: Refactorizing LLMs as Router\-Decoupled Mixture of Experts with System Co\-Design\.External Links:2410\.19123,[Link](https://arxiv.org/abs/2410.19123)Cited by:[§1](https://arxiv.org/html/2609.17940#S1.p2.1)\.
- \[2\]Z\. Qiu, Z\. Huang, S\. Cheng, Y\. Zhou, Z\. Wang, I\. Titov, and J\. Fu\(2025\)Layerwise Recurrent Router for Mixture\-of\-Experts\.External Links:2408\.06793,[Link](https://arxiv.org/abs/2408.06793)Cited by:[§1](https://arxiv.org/html/2609.17940#S1.p2.1)\.
- \[3\]Z\. Gu, T\. Likhomanenko, V\. Thilak, J\. Ramapuram, and N\. Jaitly\(2026\)Path\-Constrained Mixture\-of\-Experts\.External Links:2603\.18297,[Link](https://arxiv.org/abs/2603.18297)Cited by:[§1](https://arxiv.org/html/2609.17940#S1.p2.1)\.
- \[4\]K\. Labzin, S\. Kulibaba, A\. Dzhalilov, and A\. Gorokhov\(2026\)Evidence for Shared Routing Geometry and Dynamics in Sparse Mixture\-of\-Experts\.External Links:2609\.02404,[Link](https://arxiv.org/abs/2609.02404)Cited by:[§1](https://arxiv.org/html/2609.17940#S1.p2.1)\.
- \[5\]H\. He, Z\. Lin, X\. Liu, P\. Cheng, Y\. Lu, and Y\. Gong\(2026\)Expert\-Space Exploration in MoE Reinforcement Learning\.External Links:2609\.13058,[Link](https://arxiv.org/abs/2609.13058)Cited by:[§1](https://arxiv.org/html/2609.17940#S1.p2.1)\.
- \[6\]H\. Lin, W\. Wan, and K\. Wang\(2026\)Do Dynamic Routers Need Memory? HeRo: History\-Aware Routing for Efficient LLM Inference\.External Links:2609\.08189,[Link](https://arxiv.org/abs/2609.08189)Cited by:[§1](https://arxiv.org/html/2609.17940#S1.p2.1)\.
- \[7\]N\. Muennighoff, L\. Soldaini, D\. Groeneveld, K\. Lo, J\. Morrison, S\. Min, W\. Shi, P\. Walsh, O\. Tafjord, N\. Lambert, Y\. Gu, S\. Arora, A\. Bhagia, D\. Schwenk, D\. Wadden, A\. Wettig, B\. Hui, T\. Dettmers, D\. Kiela, A\. Farhadi, N\. A\. Smith, P\. W\. Koh, A\. Singh, and H\. Hajishirzi\(2025\)OLMoE: Open Mixture\-of\-Experts Language Models\.External Links:2409\.02060,[Link](https://arxiv.org/abs/2409.02060)Cited by:[§1](https://arxiv.org/html/2609.17940#S1.p3.1)\.
- \[8\]Y\. Shen, Z\. Guo, T\. Cai, and Z\. Qin\(2024\)JetMoE: Reaching Llama2 Performance with 0\.1M Dollars\.External Links:2404\.07413,[Link](https://arxiv.org/abs/2404.07413)Cited by:[§1](https://arxiv.org/html/2609.17940#S1.p3.1)\.
- \[9\]S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher\(2016\)Pointer Sentinel Mixture Models\.External Links:1609\.07843,[Link](https://arxiv.org/abs/1609.07843)Cited by:[§2\.2](https://arxiv.org/html/2609.17940#S2.SS2.p1.1)\.