Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Study

# Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall A Pre-Registered Negative Result, with a Systems Measurement Study
Source: [https://arxiv.org/html/2608.18261](https://arxiv.org/html/2608.18261)
Shriniwas Ramesh SuramThanks:ORCID:[0009\-0009\-0452\-9407](https://orcid.org/0009-0009-0452-9407)\. Work conducted independently on personal hardware \(single RTX 3070\)\.Affiliation:University of Cumberlands, Williamsburg, USAEmail:[ssuram36954@ucumberlands\.edu](mailto:)

August 2026

###### Abstract

Serving a 235B\-parameter Mixture\-of\-Experts \(MoE\) model on a single commodity GPU \(8 GB VRAM\) is bottlenecked not by compute but by*memory bandwidth*: autoregressive decode must stream each token’s active expert weights out of whichever memory tier holds them, and on consumer hardware most experts live on an SSD an order of magnitude slower than RAM\. We first quantify this*bandwidth wall*empirically on Qwen3\-235B\-A22B \(Q4\_K\_M, 134 GB\): measured decode is0\.440\.44tok/s warm, in exact agreement with a bytes\-per\-token÷\\divbandwidth model, and a plausible request\-batching scheme that should amortize one disk sweep over many streams instead*collapses*at batch 32 due to paging thrash\. We then buildllama\-moe\-trace, a zero\-model\-surgery router\-telemetry tool, and measure MoE routing structure on Qwen3\-30B\-A3B: expert reuse between adjacent tokens is2\.0×2\.0\\timeschance, 95% of traffic flows through 52\.5% of experts, and code\-domain routing is nearly orthogonal to prose/math/medical\. A least\-recently\-used cache holding only 13\.4% of experts already serves 66% of requests\. Motivated by this, we ask whether routing cacheability is a*trainable*property: we pre\-register an experiment training 137M\-parameter MoE language models with auxiliary*locality*and*domain*router losses, with joint success criteria on both cache\-miss reduction*and*language\-modeling quality\. The mechanism works—locality training cuts cache misses up to 60% and domain training reaches a 99% static\-pin hit rate—but*every configuration fails the pre\-registered≤1%\\leq\\\!1\\%perplexity gate*: miss reduction and perplexity cost are tightly coupled, and no loss weight threads the joint bar at this scale\. Concurrent work \(StickyMoE;[Kayyam 2026](https://arxiv.org/html/2608.18261#bib.bib9)\) proposes the same adjacent\-token routing\-consistency loss and reports it as nearly free—even*improving*perplexity \(up to−4\.1%\-4\.1\\%\)—on single\-domain \(WikiText\-2\), sub\-25M\-parameter models\. On a harder multi\-domain corpus at 137M we instead find the locality tax real and the≤1%\\leq\\\!1\\%gate unmet; our contribution is this*pre\-registered, stricter\-criterion, multi\-domain*evaluation, a domain\-partitioning arm, and the edge\-serving measurements above\. We report this negative result in full\. A 340M rung tests whether the tax shrinks with scale: it does not \(it rises slightly\), though undertraining at a matched budget leaves this suggestive\. We further show that training\-free cache\-aware rerouting*stacks*with trained locality—together reaching∼\\sim80% cache\-miss reduction at≤3\.4%\\leq\\\!3\.4\\%perplexity at both sizes, far cheaper than either alone— while domain\-primed prefetching does not help\. All code, traces, and the pre\-registration are released\.

## 1Introduction

Mixture\-of\-Experts \(MoE\) architectures decouple a model’s*total*parameter count from the parameters*active*per token: a router sends each token to a small top\-kksubset of many experts\[[Shazeer et al\. 2017](https://arxiv.org/html/2608.18261#bib.bib20),[Fedus et al\. 2022](https://arxiv.org/html/2608.18261#bib.bib5)\]\. This sparsity is what makes frontier open models such as DeepSeek\-V3\[[DeepSeek\-AI 2024](https://arxiv.org/html/2608.18261#bib.bib3)\]and Qwen3\[[Yang et al\. 2025](https://arxiv.org/html/2608.18261#bib.bib28)\]runnable at all outside a datacenter: Qwen3\-235B\-A22B stores 235 B parameters but activates only∼\\sim22 B per token\. Yet “runnable” and “fast” are different claims\. On a commodity desktop—the setting of this paper, an 8 GB RTX 3070 with 32 GB of system RAM and a consumer NVMe SSD—the 4\-bit model is 134 GB and cannot fit the fast memory tiers, so most experts are streamed from disk on demand\.

Our central observation, developed in §[3](https://arxiv.org/html/2608.18261#S3), is that autoregressive*decode is memory\-bandwidth\-bound*: the arithmetic units spend most of their time waiting for weights to arrive, so throughput is governed by*bytes touched per token*divided by*bandwidth to those bytes*\[[Sheng et al\. 2023](https://arxiv.org/html/2608.18261#bib.bib21),[Xiao et al\. 2024](https://arxiv.org/html/2608.18261#bib.bib25)\]\. Every intervention available—batching, caching, placement, or changing the model—attacks one of those two terms and nothing else\. This paper works through them, empirically, on real hardware, and ends with an original attempt to attack the*bytes*term at training time\.

#### Contributions\.

1. 1\.A quantified bandwidth wall\(§[3](https://arxiv.org/html/2608.18261#S3)\): measurements on Qwen3\-235B\-A22B matching a first\-principles model to within 6%, and a measured*negative*result—request batching, which should amortize disk sweeps, collapses at batch 32 inllama\.cpp\.
2. 2\.A routing\-telemetry instrument and profile\(§[4](https://arxiv.org/html/2608.18261#S4)\):llama\-moe\-tracecaptures router decisions with weights untouched; on Qwen3\-30B\-A3B we measure2\.0×2\.0\\timestemporal locality, a 52\.5% working set, and near\-orthogonal per\-domain expert sets, and simulate cache hit rates\.
3. 3\.Path\-Mapped Serving and its ceiling\(§[5](https://arxiv.org/html/2608.18261#S5)\): a placement policy that improves decode22–7×7\\timesat full accuracy but provably cannot reach interactive speed on this hardware\.
4. 4\.A pre\-registered, stricter\-criterion test of training\-time locality\(§[6](https://arxiv.org/html/2608.18261#S6)–[7](https://arxiv.org/html/2608.18261#S7)\): the adjacent\-token routing\-consistency loss was introduced concurrently by StickyMoE\[[Kayyam 2026](https://arxiv.org/html/2608.18261#bib.bib9)\], which reports it as nearly free \(even*improving*perplexity\) on single\-domain, sub\-25M\-parameter models\. We independently and*pre\-registrably*evaluate it \(plus a new domain\-partitioning loss\) under joint miss\-and\-quality criteria on a multi\-domain corpus and find it fails a strict≤1%\\leq\\\!1\\%perplexity bar at 137M scale—in tension with StickyMoE’s near\-free result—and we chart the full dose\-response and a scale study that probes the reconciliation\.

We deliberately hold accuracy fixed throughout: we never quantize below the model’s shipped 4\-bit weights nor distill, so “same accuracy” means bit\-exact with the served model\.

## 2Background

#### Transformer decode\.

A decoder LM maps tokens to embeddings and applies a stack of blocks, each an attention sub\-layer \(token mixing\) followed by a feed\-forward network \(FFN\) applied per token\. The FFN holds the majority of parameters\. During decode the model emits one token at a time; each step reads the active weights once and does anO⁡\(d2\)O\(d^\{2\}\)matrix–vector product per layer—low arithmetic intensity, so the step is bound by weight\-read bandwidth, not FLOPs\.

#### Mixture\-of\-Experts\.

An MoE layer replaces the single FFN withEEexpert FFNs and a router that, per token, selects the top\-kkby a learned gate\[[Shazeer et al\. 2017](https://arxiv.org/html/2608.18261#bib.bib20),[Fedus et al\. 2022](https://arxiv.org/html/2608.18261#bib.bib5),[Lepikhin et al\. 2020](https://arxiv.org/html/2608.18261#bib.bib10)\]\. Storage scales withEE; per\-token cost scales withkk\. A load\-balancing auxiliary loss prevents expert collapse\[[Fedus et al\. 2022](https://arxiv.org/html/2608.18261#bib.bib5)\]\. Fine\-grained variants raiseEEand lower the activation ratio\[[Dai et al\. 2024](https://arxiv.org/html/2608.18261#bib.bib2)\]\. Figure[1](https://arxiv.org/html/2608.18261#S2.F1)contrasts dense and MoE layers\.

#### The memory hierarchy\.

Table[1](https://arxiv.org/html/2608.18261#S2.T1)gives the three tiers on our machine\. Bandwidth falls∼\\sim180×\\timesfrom VRAM to SSD while capacity rises; a model that overflows the fast tiers is served at the speed of the slow one\. This is the same constraint that motivates offloaded\-inference systems\[[Sheng et al\. 2023](https://arxiv.org/html/2608.18261#bib.bib21),[Xue et al\. 2024b](https://arxiv.org/html/2608.18261#bib.bib27),[Xue et al\. 2024a](https://arxiv.org/html/2608.18261#bib.bib26)\]\.

Table 1:Memory hierarchy on the test machine \(i9\-12900, RTX 3070 8 GB, 32 GB DDR4, WD SN530 NVMe\)\. The 4\-bit 235B model is 134 GB; only∼\\sim40 GB fits the fast tiers\.tokenrouter \(top\-2\)E1✓\\checkmarkE5✓\\checkmarkE2E6⋯\\cdotsE128storeEEexperts→\\;\\to\\;235 B totalcomputekkexperts→\\;\\to\\;22 B activeFigure 1:An MoE layer\. The router activatesk=2k\{=\}2ofE=128E\{=\}128experts per token \(Qwen3 uses top\-8\)\. Idle experts \(grey\) cost storage but not per\-token bandwidth—the sparsity that makes edge serving conceivable\.

## 3The Bandwidth Wall

#### Baseline\.

We serve Qwen3\-235B\-A22B \(Q4\_K\_M, 3\-part GGUF, 134 GB\) withllama\.cpp\(\-ngl 99 \-ncmoe 94\), streaming experts from NVMe\. Each token activates∼\\sim22 B parameters≈\\approx12 GB of 4\-bit expert reads\. Table[2](https://arxiv.org/html/2608.18261#S3.T2)reports measured throughput\. Decode is0\.440\.44tok/s warm; dividing the 12 GB/token read by the measured rate recovers∼\\sim2\.4 GB/s, exactly the SSD’s cold sequential ceiling\. The first\-principles modeltok/s=bandwidth/bytes\-per\-token\\text\{tok/s\}=\\text\{bandwidth\}/\\text\{bytes\-per\-token\}predicts0\.200\.20tok/s from cold disk; the machine is disk\-bandwidth\-bound and already optimally configured\.

Table 2:Serving measurements, Qwen3\-235B\-A22B Q4\_K\_M\. Aggregate throughput at batchBBacross concurrent streams\. The union\-of\-experts model predicts aggregate rate should*rise*withBB; it does toB=8B\{=\}8then collapses\.
#### The 50×\\timesgap\.

A usable 10 tok/s at 12 GB/token requires 120 GB/s sustained to the weights\. The SSD delivers 2\.4 GB/s—a 50×\\timesshortfall that no software removes \(Figure[2](https://arxiv.org/html/2608.18261#S3.F2)left\)\. This frames the rest of the paper: to go faster we must either cut bytes\-per\-token or raise bandwidth\-to\-bytes\.

#### A measured negative result: batching\.

Reading all 134 GB once takes∼\\sim56 s\. If many decode streams share that sweep, aggregate throughput should scale with batch, since at high concurrency the union of experts needed approaches the whole model\. The union model matches measurement to within 6% atB=1,8B\{=\}1,8\(Table[2](https://arxiv.org/html/2608.18261#S3.T2)\) but atB=32B\{=\}32throughput*falls*to 0\.087 tok/s: 32 concurrent streams’ page\-fault storms push the OS memory manager into thrash \(effective IO∼\\sim0\.35 GB/s\)\. The idea is sound;llama\.cppcannot express it aboveB≈8B\\\!\\approx\\\!8\. Realising it would require a purpose\-built sequential\-sweep engine—future work\.

SSDDDR4targetVRAM10010^\{0\}10110^\{1\}10210^\{2\}0\.880\.883\.913\.914\.794\.79GB/s \(log\)\(a\)Bandwidth vs\. the 120 GB/s needed for 10 tok/s\. SSD is 50×\\timesshort\.B=1B=8B=32000\.10\.10\.20\.20\.30\.30\.40\.4aggregate tok/smeasuredpredicted\(b\)Batching collapse: prediction tracks toB=8B\{=\}8, then paging thrash\.
Figure 2:The bandwidth wall \(left\) and the batching collapse \(right\)\.

## 4Measuring Routing Structure

To do better than blind streaming we need to know*which*experts fire, for what text\. No serving engine exposes this, so we builtllama\-moe\-trace, a∼\\sim120\-line addition tollama\.cpp’s eval\-callback that records the router’s top\-kkselection \(ffn\_moe\_topk\) for every layer and token,*with weights untouched*\. A subtlety cost us a false start: this tensor is a non\-contiguous view \(the top\-kkslice of a 128\-wide argsort\), so a flat copy read whole argsort rows and produced perfectly uniform garbage—every expert appearing exactlykktimes, with sub\-chance reuse\. We detected this from those anomalies before drawing any conclusion and fixed it with a stride\-honoring copy \(logged in our corrections record\)\.

We trace Qwen3\-30B\-A3B \(48 layers, 128 experts, top\-8\) over 8,000 tokens each of prose, code, math, and medical text\. Table[3](https://arxiv.org/html/2608.18261#S4.T3)summarizes\. Expert reuse between adjacent tokens is2\.0×2\.0\\timeschance \(temporal locality\); 95% of traffic flows through 52\.5% of experts; and per\-domain expert sets are structured—code overlaps the others by only 0\.11–0\.16 \(Jaccard\-style similarity\) versus 0\.33–0\.42 among prose/math/medical \(Table[4](https://arxiv.org/html/2608.18261#S4.T4)\), consistent with measured Qwen3 specialization\[[Dai et al\. 2024](https://arxiv.org/html/2608.18261#bib.bib2)\]\. Replaying traces through a simulated per\-layer expert cache \(Figure[3](https://arxiv.org/html/2608.18261#S4.F3)\), a least\-recently\-used policy holding just 13\.4% of experts—the fraction our 18 GB fast\-memory budget represents of the 235B— already serves 65\.9% of requests, beating static pinning \(59\.2%\) because the exploitable signal is temporal; a clairvoyant Belady oracle reaches 79\.1%\.

Table 3:Routing profile of Qwen3\-30B\-A3B \(llama\-moe\-trace, 8k tokens×\\times4 domains\)\. Cache hit rates at the 235B’s 13\.4% fast\-memory budget \(layer 24, code domain\)\.Table 4:Cross\-domain expert\-usage similarity \(Qwen3\-30B\-A3B\)\. Code’s expert set is nearly disjoint from the others\.10102020303040405050606070700\.40\.40\.60\.60\.80\.811fast\-memory budget \(% of experts\)cache hit rateBelady \(oracle\)LRUstatic pinFigure 3:Expert\-cache hit rate vs\. fast\-memory budget \(Qwen3\-30B, layer 24, code\)\. At the 235B’s 13\.4% budget \(dashed\), LRU already achieves 66% and beats static pinning—the signal is temporal, favouring a dynamic resident set\.
## 5Path\-Mapped Serving and Its Ceiling

Path\-Mapped Serving \(PMS\) holds the currently\-hot experts in fast memory and streams cold misses from disk, leaving the model file untouched \(bit\-exact accuracy; a rare input costs a wait, never a wrong answer\)\. Related offloading systems predict and prefetch experts across tiers\[[Xue et al\. 2024b](https://arxiv.org/html/2608.18261#bib.bib27),[Xue et al\. 2024a](https://arxiv.org/html/2608.18261#bib.bib26),[Tang et al\. 2024](https://arxiv.org/html/2608.18261#bib.bib24),[Yi et al\. 2023](https://arxiv.org/html/2608.18261#bib.bib29),[Zhong et al\. 2024](https://arxiv.org/html/2608.18261#bib.bib31)\]; PMS uses our measured heat\-maps as the placement policy\. Projected to the 235B \(94 MoE layers,∼\\sim10\.6 MB/expert, 18 GB fast memory, 2\.4 GB/s IO\), miss traffic falls to∼\\sim2\.7 GB/token at LRU rates, giving∼\\sim0\.9 tok/s single\-stream—a22–7×7\\timesimprovement over baseline at full accuracy\.

The ceiling is hard: single\-stream 10 tok/s needs∼\\sim95% hit rate, which Figure[3](https://arxiv.org/html/2608.18261#S4.F3)shows requires∼\\sim52% of experts \(∼\\sim70 GB\) resident—4×4\\timesthis machine’s fast memory\.*Placement alone cannot reach interactive speed here\.*This exhausts the “raise bandwidth\-to\-bytes” attack on fixed hardware and motivates attacking*bytes\-per\-token*at the source: the architecture\.

## 6Training Routers for Cacheability

The2\.0×2\.0\\timeslocality that makes LRU work \(§[4](https://arxiv.org/html/2608.18261#S4)\) is*accidental*—no production router is trained for it\. We ask: is cacheability a trainable property obtainable*at no accuracy cost*? We pre\-registered the design \(hypotheses, arms, metrics, and pass/fail thresholds frozen before any run\) to make a null result meaningful\.

#### Relation to concurrent work\.

Independently and concurrently, StickyMoE\[[Kayyam 2026](https://arxiv.org/html/2608.18261#bib.bib9)\]proposed the same adjacent\-token routing\-consistency loss \(§[D](https://arxiv.org/html/2608.18261#A4),ℒloc\\mathcal\{L\}\_\{\\text\{loc\}\}\) and reports it as a favorable, quality\-*improving*technique: on small \(8\.88\.8M\) and medium \(2222M\) MoE LMs on the single\-domain WikiText\-2 corpus, its soft loss cuts the expert switch rate up to59%59\\%and reduces cache misses up to3\.92×3\.92\\timeswhile perplexity*improves on the medium \(2222M\) model*\(up to−4\.1%\-4\.1\\%atλ=0\.05\\lambda\{=\}0\.05\) and is essentially preserved on the small \(8\.88\.8M\) model \(\+0\.1%\+0\.1\\%at thatλ\\lambda; best−0\.5%\-0\.5\\%\)\. Our study differs in the corpus \(a harder 3\-domain prose/code/math mix\), the criterion \(a pre\-registered strict joint miss/quality bar\), and a domain\-confinement arm they do not consider—and in conclusion: at 137M on multi\-domain data we do*not*clear the strict bar \(see §[7](https://arxiv.org/html/2608.18261#S7)\)\. We read the two as complementary but in tension: StickyMoE shows the mechanism can be quality\-preserving on single\-domain, sub\-25M models; we show it is not free on a harder multi\-domain 137M setting, and flag the corpus and scale dependence that would reconcile them\.

#### Losses\.

All arms use the Switch load\-balance loss\. We add two differentiable statements of what a cache wants, wherept∈ΔEp\_\{t\}\\in\\Delta^\{E\}is the router’s softmax distribution at tokentt:

ℒloc\\displaystyle\\mathcal\{L\}\_\{\\text\{loc\}\}=1L​∑ℓmeant\(1−⟨ptℓ,pt−1ℓ⟩\)\\displaystyle=\\tfrac\{1\}\{L\}\\sum\_\{\\ell\}\\ \\operatorname\*\{mean\}\_\{t\}\\bigl\(1\-\\langle p\_\{t\}^\{\\ell\},\\,p\_\{t\-1\}^\{\\ell\}\\rangle\\bigr\)\(temporal reuse\)\(1\)ℒdom\\displaystyle\\mathcal\{L\}\_\{\\text\{dom\}\}=1L∑ℓmeant∑e∉𝒮⁡\(dt\)pt,eℓ\\displaystyle=\\tfrac\{1\}\{L\}\\sum\_\{\\ell\}\\ \\operatorname\*\{mean\}\_\{t\}\\ \\textstyle\\sum\_\{e\\notin\\mathcal\{S\}\(d\_\{t\}\)\}p\_\{t,e\}^\{\\ell\}\(domain confinement\)\(2\)where𝒮⁡\(d\)\\mathcal\{S\}\(d\)is the expert slice allowed for domaindd\(4 exclusive\+\+4 shared of 16\)\.ℒloc\\mathcal\{L\}\_\{\\text\{loc\}\}rewards routing the same experts to adjacent tokens;ℒdom\\mathcal\{L\}\_\{\\text\{dom\}\}rewards keeping a domain within its slice so a single slice can be pinned and served at full speed \(Figure[4](https://arxiv.org/html/2608.18261#S6.F4)\)\.

ℒLM\\mathcal\{L\}\_\{\\text\{LM\}\}\+α\+\\ \\alphaℒbalance\\mathcal\{L\}\_\{\\text\{balance\}\}\+λ\+\\ \\lambdaℒloc\\mathcal\{L\}\_\{\\text\{loc\}\}\+μ\+\\ \\muℒdom\\mathcal\{L\}\_\{\\text\{dom\}\}higher hit rate @ equal PPLFigure 4:The sticky\-moe objective\. Arm A uses onlyℒbalance\\mathcal\{L\}\_\{\\text\{balance\}\}; arm B addsλ​ℒloc\\lambda\\mathcal\{L\}\_\{\\text\{loc\}\}; arm C addsμ​ℒdom\\mu\\mathcal\{L\}\_\{\\text\{dom\}\}\. The claim under test is the box on the right*at no perplexity cost*\.
#### Setup\.

A 137M\-parameter decoder MoE \(dmodel=384d\_\{\\text\{model\}\}\{=\}384, 8 layers,E=16E\{=\}16, top\-2, expertdff=768d\_\{\\text\{ff\}\}\{=\}768, GPT\-2 BPE\), the largest that trains comfortably in 8 GB\. Data: 300M tokens, 100M each of prose \(WikiText\-103\), code \(codeparrot\-clean\), and math \(OpenWebMath\), SHA\-256 manifested\. Arms:A\(balance only\),B\(\+ℒloc\+\\mathcal\{L\}\_\{\\text\{loc\}\},λ\\lambdaswept\),C\(\+ℒdom\+\\mathcal\{L\}\_\{\\text\{dom\}\},μ=0\.1\\mu\{=\}0\.1\)\. Router traces are exported in the format of §[4](https://arxiv.org/html/2608.18261#S4)and scored by the*same*cache simulator—one instrument from frontier model to toy\. Amended budgets \(below\): 200M tokens per main arm, two seeds on A and the chosen B\.

#### Pre\-registered criteria\.

RQ1\(locality\):≥\\geq30% reduction in LRU miss/token at 25% capacity*and*validation perplexity within\+\+1% of arm A\.RQ2\(domain\):≥\\geq90% static\-pin hit@50% within\-domain*and*perplexity within\+\+2%\. The joint \(metric*and*quality\) form is deliberate: a cacheability gain that costs accuracy does not count\.

#### Engineering notes \(for reproducibility\)\.

Three issues were logged as amendments before any results were unblinded: \(i\) anN⁡\(0,1\)N\(0,1\)embedding init made the tied\-logit loss start at 373 instead of∼\\sim10\.8 \(fixed to std 0\.02\); \(ii\) a padded\-batched\-GEMM expert dispatch ran2\.8×2\.8\\times*slower*than a per\-expert loop under early\-training imbalance and was reverted \(parity test retained\); \(iii\) consequently token budgets were trimmed from 300M to 200M per arm to keep wall\-clock feasible on one GPU\. Throughput was 7,000–8,900 tok/s on the RTX 3070\.

## 7Results

Table[5](https://arxiv.org/html/2608.18261#S7.T5)gives all arms\. The baseline \(arm A, two seeds\) has perplexity 31\.9, LRU miss/token@25% of 8\.21, and static hit@50% of 0\.741\.

Table 5:Main results \(200M tokens/arm\)\. Metrics averaged over prose/code/math\. “hit” is the fraction of expert\-loads already resident\.#### RQ1 is refuted\.

The locality loss produces a clean, monotonic, seed\-stable dose\-response \(Figure[5](https://arxiv.org/html/2608.18261#S7.F5)\): largerλ\\lambdayields larger miss reduction*and*larger perplexity cost, tightly coupled\. No weight satisfies both gates\.λ=0\.02\\lambda\{=\}0\.02preserves quality \(\+\+0\.3% PPL\) but cuts misses only 15% \(<<30%\);λ=0\.05\\lambda\{=\}0\.05cuts misses 59–60% \(two seeds\) but costs\+\+2\.1–3\.1% PPL \(\>\>1%\);λ=0\.03\\lambda\{=\}0\.03, the predicted crossover, lands in the dead zone at\+\+24% /\+\+1\.8%, failing both\. The two objectives trade at a rate that excludes the\{≥30%,≤1%\}\\\{\{\\geq\}30\\%,\\ \{\\leq\}1\\%\\\}corner at this scale\.

#### RQ2 is refuted on quality\.

Domain confinement drives the static\-pin hit rate to a near\-perfect 0\.991 \(easily clearing 0\.90\)—one domain slice can be pinned and served almost miss\-free—but at\+\+3\.1% perplexity, over the\+\+2% gate\.

000\.50\.5111\.51\.5222\.52\.5333\.53\.5444\.54\.5555\.55\.5⋅10−2\\cdot 10^\{\-2\}00202040406060miss red\.RQ1 floor 30%locality weightλ\\lambdamiss reduction \(%\)00112233PPL costRQ1 gate 1%PPL cost \(%\)Figure 5:Dose\-response of the locality loss \(200M tokens\)\. Miss reduction \(left, blue\) and perplexity cost \(right, red\) rise together\. The RQ1 pass region needs miss reduction above 30%*and*PPL cost below 1%—the curves never satisfy both simultaneously\. \(Two axes shown only to display the coupling; they are not a dual\-scale claim about one series\.\)
### 7\.1Training\-free rerouting, and its complementarity with trained locality

The dose\-response above trains the router for locality\. A cache can also be helped*at inference time only*, with no retraining: when a token’s top\-ranked expert is not resident but a cached expert scores within a relative toleranceτ\\tauof it, we route to the cached expert instead—the training\-free strategy of[Skliar et al\. 2025](https://arxiv.org/html/2608.18261#bib.bib22), trading a controlled slice of routing fidelity for cache hits\. Sweepingτ\\tautraces a perplexity\-versus\-miss frontier from any fixed checkpoint\. Numbers here use an independent held\-out multi\-domain stream with the same per\-layer LRU cache \(25% of experts\); we report relative effects, and absolute perplexities differ slightly from Table[5](https://arxiv.org/html/2608.18261#S7.T5)’s 500k\-token evaluation\.

First, on the*baseline*model \(arm A, never trained for locality\) training\-free rerouting reaches∼\\sim30% miss reduction at\+\+0\.8% perplexity on one seed \(\+\+2\.0% on the other—this low\-τ\\tauregime is seed\-sensitive\), at zero training cost; but pushed harder it degrades fast on both seeds \(50% reduction costs\+\+4\.0–5\.8%; 74% costs\+\+25%\), because rerouting an untrained router forces genuinely worse experts\.

Second, and the main point: the two mechanisms are*complementary*, and this holds across both seeds\. Applying the sameτ\\tausweep on top of the locality\-*trained*model \(λ=0\.05\\lambda\{=\}0\.05\) is nearly free\. Training\-time locality alone gives 58–61% miss reduction at\+\+1\.4–2\.2% PPL; addingτ=0\.5\\tau\{=\}0\.5raises this to80–82% reduction at\+\+1\.9–3\.1%, andτ=0\.7\\tau\{=\}0\.7to 86–88% \(Figure[6](https://arxiv.org/html/2608.18261#S7.F6); ranges are the two seeds\)\. The identicalτ=0→0\.5\\tau\{=\}0\{\\to\}0\.5intervention that costs\+\+4\.0–5\.8% PPL on the baseline costs only\+\+0\.5–0\.9% on the trained model—a 5–8×\\timessmaller marginal cost\. Training for locality co\-adapts experts into mutually substitutable neighbourhoods \(expert redundancy and mergeability are well documented—similar experts can be merged with little loss\[[Li et al\. 2024a](https://arxiv.org/html/2608.18261#bib.bib12)\]—and routing specialisation strengthens over training\[[Mouzouni 2026](https://arxiv.org/html/2608.18261#bib.bib18)\]\), so inference\-time substitution within a neighbourhood barely perturbs the output\. Neither mechanism alone threads a high\-reduction, low\-cost point; together they reach∼\\sim80% fewer misses at an operating point that no single setting in Table[5](https://arxiv.org/html/2608.18261#S7.T5)attains\. This complementarity replicates at 340M \(Table[6](https://arxiv.org/html/2608.18261#S8.T6)\): theτ​0→0\.5\\tau 0\{\\to\}0\.5intervention costs\+0\.9%\+0\.9\\%perplexity on the trained model versus\+7\.7%\+7\.7\\%on the baseline, and stacking reaches 82% miss reduction at\+3\.4%\+3\.4\\%—so it is not a single\-scale artifact\. As a preliminary third case, the same locality training makes the second expert’s load frequently*omittable*: eliding it whenever its renormalised gate falls belowϵ=0\.02\\epsilon\{=\}0\.02\(a resident, load\-free test\) drops∼\\sim12% of expert loads \(prose/code/math12\.6/18\.5/3\.8%12\.6/18\.5/3\.8\\%\) at a near\-lossless≤0\.05%\{\\leq\}0\.05\\%perplexity change on the trained model, versus essentially none on the baseline\.

We also tested a*domain\-primed*variant \(cf\. data\-aware offloading,[Zhang et al\. 2025](https://arxiv.org/html/2608.18261#bib.bib30)\)—prefetching a domain’s experts into the cache at each boundary of a mixed prose/code/math stream—and found no measurable benefit: an LRU cache re\-warms withinO⁡\(cap\)O\(\\text\{cap\}\)tokens of a switch, so the boundary miss burst amortises to nothing over realistic \(≥\\geq1k\-token\) domain runs \(mixed\-stream miss/token equals the single\-domain value to within noise\)\. The domain\-switch cache penalty, plausible a priori, is negligible; multi\-domain difficulty is a*training*phenomenon \(the tax above\), not a serving\-time caching one\.

001010202030304040505060607070808090901001000010102020≤\\leq1% gateexpert\-cache miss reduction vs baseline \(%\)perplexity cost \(%\)training\-free \(baseline model\)stacked \(trained\+\+reroute\)training\-time \(τ=0\\tau\{=\}0\)Figure 6:Complementarity of trained and inference\-time locality\. Training\-free rerouting on the baseline \(blue\) is cheap only up to∼\\sim30% miss reduction, then diverges\. On the locality\-trained model the same rerouting \(green\) stays flat and low, reaching 80% miss reduction at\+\+2\.4% perplexity—far beyond training\-time alone \(squares, max 59%\) or training\-free alone\. The experts’ co\-adaptation makes inference\-time substitution nearly free\.

## 8Discussion

The mechanism is real: training changes routing exactly as intended, with large \(≤\\leq60% fewer misses; 99% static hit\), monotonic, seed\-stable effects\. But it is*not free*at 137M parameters—a genuine perplexity tax breaks the pre\-registered bars everywhere\. We take the joint\-criterion refutation at face value; pre\-registration is what stops a large\-but\-costly effect from being reported as a win by choosing a favourable metric post hoc\.

#### Does the tax shrink with scale? A 340M rung says not yet\.

A 16\-expert, 137M model has little spare capacity: forcing the router to reuse or confine experts removes representational freedom the model was using for language modelling, so quality drops\. Sparsity scaling suggests the tax should shrink withEEand total parameters\[[Dai et al\. 2024](https://arxiv.org/html/2608.18261#bib.bib2),[DeepSeek\-AI 2024](https://arxiv.org/html/2608.18261#bib.bib3)\], giving a sharp, pre\-registered hypothesis:*the perplexity cost of a fixed cacheability gain decreases with model scale*\. We tested it directly with a 340M rung \(Table[6](https://arxiv.org/html/2608.18261#S8.T6)\), matched to the 137M runs at the same 200M\-token budget andλ=0\.05\\lambda\{=\}0\.05\. It does*not*shrink: at a fixed∼\\sim58% cache\-miss reduction the locality tax*rises*from\+2\.0%\+2\.0\\%\(137M\) to\+2\.5%\+2\.5\\%\(340M\)\. Two caveats keep this suggestive rather than decisive: at a matched token budget the 340M model is more undertrained \(its baseline perplexity is higher, §[9](https://arxiv.org/html/2608.18261#S9)\), and the ladder has only two rungs\. A compute\-optimal and a≥1\\geq\\\!1B run are the definitive test\. Concurrent work is consistent with a scale effect existing in principle—Oracle\-MoE\[[Zhou et al\. 2025](https://arxiv.org/html/2608.18261#bib.bib32)\]trains for locality across a 200M–2B ladder without task\-performance loss—but on standard, not multi\-domain strict\-bar, evaluation\. Crucially, the*complementarity*result \(§[7\.1](https://arxiv.org/html/2608.18261#S7.SS1)\)*does*replicate at 340M: training\-free rerouting stays∼\\sim9×\\timescheaper on the locality\-trained model and stacking reaches 82% miss reduction at\+3\.4%\+3\.4\\%perplexity—so the cheapest\-when\-trained\-and\-rerouted finding is not a 137M artifact\.

Table 6:Scale rung\. Locality tax \(arm A baseline vs arm Bλ=0\.05\\lambda\{=\}0\.05atτ=0\\tau\{=\}0\) and the stacked training\-free rerouting point \(τ=0\.5\\tau\{=\}0\.5on arm B\), at 137M and 340M; same 200M\-token budget, cap=25%\{=\}25\\%experts, LRU, seed 1\. The tax does not shrink with scale; the stacking advantage persists\.

## 9Limitations

\(i\)Scale: two rungs \(137M, 340M\), both below compute\-optimal; the 340M locality tax did not shrink \(Table[6](https://arxiv.org/html/2608.18261#S8.T6)\), but a≥1\\geq\\\!1B rung and a compute\-optimal run remain the definitive test\. \(ii\)Proxy: the routing profile \(§[4](https://arxiv.org/html/2608.18261#S4)\) is Qwen3\-30B, a same\-family proxy for the 235B; a same\-artifact trace on the 235B GGUF is future work\. \(iii\)Undertraining: 200M tokens is far below compute\-optimal, so absolute perplexities are high—comparisons are fair \(identical budgets across arms\) but absolute quality is not the claim\. \(iv\)Single hardware: bf16, one consumer GPU\. \(v\) The batched\-serving and PMS speed projections are modelled from measured IO and cache rates, not end\-to\-end deployed\.

## 10Related Work

MoE and routing\.Sparsely\-gated MoE\[[Shazeer et al\. 2017](https://arxiv.org/html/2608.18261#bib.bib20)\], Switch\[[Fedus et al\. 2022](https://arxiv.org/html/2608.18261#bib.bib5)\], GShard\[[Lepikhin et al\. 2020](https://arxiv.org/html/2608.18261#bib.bib10)\], and fine\-grained DeepSeekMoE\[[Dai et al\. 2024](https://arxiv.org/html/2608.18261#bib.bib2)\]establish the total/active split and load balancing; routing analyses\[[Dai et al\. 2024](https://arxiv.org/html/2608.18261#bib.bib2)\]document the specialization we exploit\.Training\-time routing locality \(closest prior/concurrent work\)\.StickyMoE\[[Kayyam 2026](https://arxiv.org/html/2608.18261#bib.bib9)\]introduces the adjacent\-token routing\-consistency loss we study here and reports it favorably; Oracle\-MoE\[[Zhou et al\. 2025](https://arxiv.org/html/2608.18261#bib.bib32)\]likewise*trains*for locality—routing in an attention\-derived “oracle space”—and reports gains across a 200M–2B size ladder without task\-performance loss, while post\-hoc router fine\-tuning \(ReMoE\[[Zhu et al\. 2026](https://arxiv.org/html/2608.18261#bib.bib33)\],\+\+26% reuse; MELINOE\[[Raje et al\. 2026](https://arxiv.org/html/2608.18261#bib.bib19)\], up to3×3\\timesfewer transfers\) adapts a pretrained router for reuse rather than co\-adapting experts and routing from step one\. Cache\-conditional*routing*\[[Skliar et al\. 2025](https://arxiv.org/html/2608.18261#bib.bib22)\]is a training\-free, cache\-aware routing strategy that cuts mobile cache misses\>\>50% at 0\.1–3% PPL \(surpassing the oracle cache bound\), and expert\-locality decode routing\[[Choi et al\. 2026](https://arxiv.org/html/2608.18261#bib.bib1)\]exploits locality at serving time; and[Liang et al\. 2026](https://arxiv.org/html/2608.18261#bib.bib14)measure this “local routing consistency” across 20 MoE models, finding it trades off against load balance—corroborating why a locality intervention carries a capacity cost\. Our contribution relative to these is the pre\-registered, strict\-criterion evaluation, the domain\-confinement arm, a 137M→\\to340M scale rung \(the tax does not shrink\), and the training\-time×\\timestraining\-free stacking result\.Edge MoE and expert offloading\.PowerInfer/PowerInfer\-2\[[Song et al\. 2024](https://arxiv.org/html/2608.18261#bib.bib23),[Xue et al\. 2024b](https://arxiv.org/html/2608.18261#bib.bib27)\], EdgeMoE\[[Yi et al\. 2023](https://arxiv.org/html/2608.18261#bib.bib29)\], MoE\-Infinity\[[Xue et al\. 2024a](https://arxiv.org/html/2608.18261#bib.bib26)\], HOBBIT\[[Tang et al\. 2024](https://arxiv.org/html/2608.18261#bib.bib24)\], AdapMoE\[[Zhong et al\. 2024](https://arxiv.org/html/2608.18261#bib.bib31)\], Fiddler\[[Kamahori et al\. 2025](https://arxiv.org/html/2608.18261#bib.bib8)\], cross\-layer gate prediction \(Fate;[Fang et al\. 2025](https://arxiv.org/html/2608.18261#bib.bib4)\), predictive caching with token scheduling \(ExpertFlow, predating cache\-conditional routing;[He et al\. 2026](https://arxiv.org/html/2608.18261#bib.bib7)\), and related systems predict, prefetch, cache, or CPU\-GPU\-orchestrate experts across the hierarchy—the lineage of PMS; MoE\-Infinity’s activation tracing is the direct ancestor of our tracer, and Fiddler’s CPU\-side expert execution is an alternative to the disk\-streaming regime we measure; see[Liu et al\. 2025](https://arxiv.org/html/2608.18261#bib.bib16)for a system\-stack survey of MoE inference optimisation\.Offloaded inference and the bandwidth bound\.FlexGen\[[Sheng et al\. 2023](https://arxiv.org/html/2608.18261#bib.bib21)\]and ZeRO\-Inference characterise memory\-bound generation\.Complementary axes we hold fixed\.Quantization\[[Frantar et al\. 2023](https://arxiv.org/html/2608.18261#bib.bib6),[Lin et al\. 2024](https://arxiv.org/html/2608.18261#bib.bib15),[Ma et al\. 2024](https://arxiv.org/html/2608.18261#bib.bib17)\]and KV\-cache compression\[[Xiao et al\. 2024](https://arxiv.org/html/2608.18261#bib.bib25)\]attack bytes\-per\-token by changing precision or state; speculative decoding\[[Leviathan et al\. 2023](https://arxiv.org/html/2608.18261#bib.bib11),[Li et al\. 2024b](https://arxiv.org/html/2608.18261#bib.bib13)\]trades compute for bandwidth—orthogonal to training\-time routing and deliberately out of scope here\. A full, PDF\-verified bibliography accompanies this draft\.

## 11Conclusion

On commodity hardware, serving a 235B MoE is a memory\-bandwidth problem, and the speed ceiling is set by bytes\-per\-token over bandwidth\-to\-bytes\. We measured the wall, built an instrument to see the routing structure inside it, and tested whether that structure can be trained to be more cacheable\. It can—but not for free at small scale, and our pre\-registered joint criteria are not met by any configuration\. We publish the negative result in full because it converts a vague hope \(“train routers to be cache\-friendly”\) into a precise, falsifiable scaling claim, and because the discipline of reporting it is the point\.

#### Reproducibility\.

#### Use of AI assistance\.

The author used an AI coding assistant to help implement the experimental harness and to draft and edit the manuscript\. All experimental design decisions, the pre\-registration, the runs, and the interpretation are the author’s, who takes full responsibility for the content and has verified every reported number against the released artifacts\.

## Appendix APer\-Domain Results

Table[5](https://arxiv.org/html/2608.18261#S7.T5)averages over domains; Table[7](https://arxiv.org/html/2608.18261#A1.T7)breaks the baseline \(arm A, two seeds\) and theλ=0\.05\\lambda\{=\}0\.05locality arm \(two seeds\) out by domain\. The perplexity tax is*not*uniform: it is concentrated in the hardest domain \(prose,\+\+3\.0%\) and is essentially zero for the most predictable one \(code, PPL≈\\approx4\.7 both arms\), while the cacheability gain is large and consistent everywhere \(∼\\sim0\.30 reuse gain in every domain\)\. A model with almost no perplexity headroom on prose pays for the routing constraint there; where the task is easy \(code\) the router can be made sticky for free\. This domain\-dependence is itself evidence for the small\-capacity explanation in §[9](https://arxiv.org/html/2608.18261#S9)\.

Table 7:Per\-domain metrics, baseline \(A\) vs\. locality \(λ=0\.05\\lambda\{=\}0\.05\), each averaged over 2 seeds\. “hit@25%” is LRU; “static@50%” is static\-pin\.
## Appendix BAll Runs

Table[8](https://arxiv.org/html/2608.18261#A2.T8)lists every training run, including the 50M\-token sanity andλ\\lambda\-triage runs used to size the main study\. The sanity/triage rows were run before the 200M main arms and are not used in the verdict\.

Table 8:Complete run log\. Metrics averaged over prose/code/math\.
## Appendix CHyperparameters

Table 9:Hyperparameters \(identical across arms; only the router\-loss weightsλ,μ\\lambda,\\mudiffer\)\.
## Appendix DRouter Losses \(exact form\)

For a batch of routing distributionsptℓ∈ΔEp\_\{t\}^\{\\ell\}\\in\\Delta^\{E\}at tokentt, layerℓ\\ell, with top\-kkindicatorztℓ∈\{0,1\}Ez\_\{t\}^\{\\ell\}\\in\\\{0,1\\\}^\{E\}:

ℒbalance\\displaystyle\\mathcal\{L\}\_\{\\text\{balance\}\}=1L​∑ℓE​∑efeℓ​Peℓ,feℓ=1k​T​∑tzt,eℓ,Peℓ=1T​∑tpt,eℓ\\displaystyle=\\tfrac\{1\}\{L\}\\sum\_\{\\ell\}E\\sum\_\{e\}f\_\{e\}^\{\\ell\}\\,P\_\{e\}^\{\\ell\},\\hskip 10\.00002ptf\_\{e\}^\{\\ell\}=\\tfrac\{1\}\{kT\}\\\!\\sum\_\{t\}z\_\{t,e\}^\{\\ell\},\\ \\ P\_\{e\}^\{\\ell\}=\\tfrac\{1\}\{T\}\\\!\\sum\_\{t\}p\_\{t,e\}^\{\\ell\}\(Switch\)ℒloc\\displaystyle\\mathcal\{L\}\_\{\\text\{loc\}\}=1L​∑ℓ1T−1​∑t≥2\(1−⟨ptℓ,pt−1ℓ⟩\)\\displaystyle=\\tfrac\{1\}\{L\}\\sum\_\{\\ell\}\\tfrac\{1\}\{T\-1\}\\sum\_\{t\\geq 2\}\\bigl\(1\-\\langle p\_\{t\}^\{\\ell\},p\_\{t\-1\}^\{\\ell\}\\rangle\\bigr\)\(reuse\)ℒdom\\displaystyle\\mathcal\{L\}\_\{\\text\{dom\}\}=1L​∑ℓ1T​∑t∑e∉𝒮⁡\(dt\)pt,eℓ\\displaystyle=\\tfrac\{1\}\{L\}\\sum\_\{\\ell\}\\tfrac\{1\}\{T\}\\sum\_\{t\}\\sum\_\{e\\notin\\mathcal\{S\}\(d\_\{t\}\)\}p\_\{t,e\}^\{\\ell\}\(confinement\)Total loss:ℒ=ℒLM\+α​ℒbalance\+λ​ℒloc\+μ​ℒdom\\mathcal\{L\}=\\mathcal\{L\}\_\{\\text\{LM\}\}\+\\alpha\\mathcal\{L\}\_\{\\text\{balance\}\}\+\\lambda\\mathcal\{L\}\_\{\\text\{loc\}\}\+\\mu\\mathcal\{L\}\_\{\\text\{dom\}\}\. The domain slice𝒮⁡\(d\)\\mathcal\{S\}\(d\)for domaindd\(of 3\) is 4 exclusive experts plus 4 shared, ofE=16E\{=\}16\.

## Appendix ECache Simulation

We replay each router trace \(per layer, an\(T,k\)\(T,k\)array of expert ids\) through an independent per\-layer cache of capacityc=⌈cap⋅E⌉c=\\lceil\\text\{cap\}\\cdot E\\rceiland count demand misses; reported miss/token sums over layers and normalises byL⋅kL\\cdot kexpert\-loads\. Policies:LRU\(evict least\-recently\-used\),LFU\(least\-frequently\-used over cumulative counts\),static\(pin the top\-ccexperts by frequency in a warmup prefix, no eviction\), andBelady\(clairvoyant: evict the resident expert whose next use is farthest in the future—an unachievable upper bound\)\. The same simulator scores the frontier\-model traces of §[4](https://arxiv.org/html/2608.18261#S4)and the trained\-model traces of §[7](https://arxiv.org/html/2608.18261#S7), so a single instrument spans both\.

## Appendix FCorrections Log

We record every error caught during the study, per the reproducibility stance:

1. 1\.“5\.3 GB/s NVMe” was warm page\-cache inflation; the cold sequential rate is 2\.4 GB/s \(used throughout\)\.
2. 2\.Batching throughput prediction held toB=8B\{=\}8then broke atB=32B\{=\}32\(paging thrash\); reported as a negative result, not tuned away\.
3. 3\.ffn\_moe\_topkis a non\-contiguous argsort view; a flat copy read garbage \(every expert appearing exactlykktimes, sub\-chance reuse\)\. Fixed with a stride\-honoring copy; a first profiling run was discarded\.
4. 4\.Embedding initN⁡\(0,1\)N\(0,1\)made the tied\-logit loss start at 373 instead of∼\\sim10\.8; fixed to std 0\.02\.
5. 5\.A padded\-batched\-GEMM expert dispatch ran2\.8×2\.8\\timesslower than a per\-expert loop under early\-training imbalance; reverted \(parity test kept\), budgets trimmed 300M→\\to200M/arm\. Logged before any results were unblinded\.
6. 6\.The analysis hit\-rate was initially miscomputed \(summed misses over all layers but normalised by one layer’s slots, giving negative rates\); fixed to÷\(L⋅k\)\\div\(L\\cdot k\)\.

## Appendix GReproducibility Artifacts

Released: the frozen pre\-registration \(DESIGN\.md\); training, analysis, and orchestration code; thellama\-moe\-tracetool; 7 unit tests covering loss shapes, gradient flow, the constant\-routing limits ofℒloc\\mathcal\{L\}\_\{\\text\{loc\}\}andℒdom\\mathcal\{L\}\_\{\\text\{dom\}\}, the domain\-mask layout, determinism, and grouped\-vs\-loop dispatch parity; all router traces \(\.npz\); and per\-runconfig\.jsonwith seed, loss weights, token budget, and data\-shard SHA\-256 hashes\. Every number in this paper is regenerable from these\. Code and artifacts:[https://github\.com/Shriniwas410/cacheable\-by\-design](https://github.com/Shriniwas410/cacheable-by-design)\.

## References

- Choi et al\. \[2026\]Sangjin Choi, Sukmin Cho, Yifan Xiong, Ziyue Yang, Youngjin Kwon, and Peng Cheng\.ELDR: Expert\-locality\-aware decode routing for PD\-disaggregated MoE serving\.*arXiv preprint arXiv:2607\.00466*, 2026\.URL[https://arxiv\.org/abs/2607\.00466](https://arxiv.org/abs/2607.00466)\.
- Dai et al\. \[2024\]Damai Dai, Chengqi Deng, Chenggang Zhao, R\. X\. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y\. Wu, Zhenda Xie, Y\. K\. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang\.Deepseekmoe: Towards ultimate expert specialization in mixture\-of\-experts language models\.*arXiv preprint arXiv:2401\.06066*, 2024\.URL[https://arxiv\.org/abs/2401\.06066](https://arxiv.org/abs/2401.06066)\.
- DeepSeek\-AI \[2024\]DeepSeek\-AI\.Deepseek\-v3 technical report\.*arXiv preprint arXiv:2412\.19437*, 2024\.URL[https://arxiv\.org/abs/2412\.19437](https://arxiv.org/abs/2412.19437)\.
- Fang et al\. \[2025\]Zhiyuan Fang, Zicong Hong, Yuegui Huang, Yufeng Lyu, Wuhui Chen, Yue Yu, Fan Yu, and Zibin Zheng\.Fate: Fast edge inference of mixture\-of\-experts models via cross\-layer gate\.*arXiv preprint arXiv:2502\.12224*, 2025\.URL[https://arxiv\.org/abs/2502\.12224](https://arxiv.org/abs/2502.12224)\.
- Fedus et al\. \[2022\]William Fedus, Barret Zoph, and Noam Shazeer\.Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity\.*Journal of Machine Learning Research \(JMLR\)*, 23\(120\):1–39, 2022\.URL[https://arxiv\.org/abs/2101\.03961](https://arxiv.org/abs/2101.03961)\.
- Frantar et al\. \[2023\]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh\.Gptq: Accurate post\-training quantization for generative pre\-trained transformers\.In*International Conference on Learning Representations \(ICLR\)*, 2023\.URL[https://arxiv\.org/abs/2210\.17323](https://arxiv.org/abs/2210.17323)\.
- He et al\. \[2026\]Xin He, Shunkang Zhang, Kaijie Tang, Shaohuai Shi, Yuxin Wang, Zihao Zeng, Zhenheng Tang, Xiaowen Chu, Haiyan Yin, Ivor W\. Tsang, and Yew Soon Ong\.ExpertFlow: Efficient mixture\-of\-experts inference via predictive expert caching and token scheduling\.In*Proceedings of the 63rd ACM/IEEE Design Automation Conference \(DAC\)*, 2026\.URL[https://arxiv\.org/abs/2410\.17954](https://arxiv.org/abs/2410.17954)\.
- Kamahori et al\. \[2025\]Keisuke Kamahori, Tian Tang, Yile Gu, Kan Zhu, and Baris Kasikci\.Fiddler: Cpu\-gpu orchestration for fast inference of mixture\-of\-experts models\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.URL[https://arxiv\.org/abs/2402\.07033](https://arxiv.org/abs/2402.07033)\.
- Kayyam \[2026\]Ali Kayyam\.Sticky routing: Training MoE models for memory\-efficient inference\.*arXiv preprint arXiv:2607\.08780*, 2026\.URL[https://arxiv\.org/abs/2607\.08780](https://arxiv.org/abs/2607.08780)\.
- Lepikhin et al\. \[2020\]Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen\.Gshard: Scaling giant models with conditional computation and automatic sharding\.*arXiv preprint arXiv:2006\.16668*, 2020\.URL[https://arxiv\.org/abs/2006\.16668](https://arxiv.org/abs/2006.16668)\.
- Leviathan et al\. \[2023\]Yaniv Leviathan, Matan Kalman, and Yossi Matias\.Fast inference from transformers via speculative decoding\.In*Proceedings of the 40th International Conference on Machine Learning \(ICML\)*, 2023\.URL[https://arxiv\.org/abs/2211\.17192](https://arxiv.org/abs/2211.17192)\.
- Li et al\. \[2024a\]Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi\-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen\.Merge, then compress: Demystify efficient SMoE with hints from its routing policy\.In*International Conference on Learning Representations \(ICLR\)*, 2024a\.URL[https://arxiv\.org/abs/2310\.01334](https://arxiv.org/abs/2310.01334)\.
- Li et al\. \[2024b\]Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang\.Eagle: Speculative sampling requires rethinking feature uncertainty\.*arXiv preprint arXiv:2401\.15077*, 2024b\.URL[https://arxiv\.org/abs/2401\.15077](https://arxiv.org/abs/2401.15077)\.
- Liang et al\. \[2026\]Jingcong Liang, Siyuan Wang, Miren Tian, Yitong Li, Duyu Tang, and Zhongyu Wei\.Not all models suit expert offloading: On local routing consistency of mixture\-of\-expert models\.In*International Conference on Learning Representations \(ICLR\)*, 2026\.URL[https://arxiv\.org/abs/2505\.16056](https://arxiv.org/abs/2505.16056)\.
- Lin et al\. \[2024\]Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei\-Ming Chen, Wei\-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han\.Awq: Activation\-aware weight quantization for llm compression and acceleration\.In*Proceedings of Machine Learning and Systems \(MLSys\)*, 2024\.URL[https://arxiv\.org/abs/2306\.00978](https://arxiv.org/abs/2306.00978)\.
- Liu et al\. \[2025\]Jiacheng Liu, Peng Tang, Wenfeng Wang, Yuhang Ren, Xiaofeng Hou, Pheng\-Ann Heng, Minyi Guo, and Chao Li\.A survey on inference optimization techniques for mixture of experts models\.*ACM Computing Surveys*, 2025\.URL[https://arxiv\.org/abs/2412\.14219](https://arxiv.org/abs/2412.14219)\.
- Ma et al\. \[2024\]Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei\.The era of 1\-bit llms: All large language models are in 1\.58 bits\.*arXiv preprint arXiv:2402\.17764*, 2024\.URL[https://arxiv\.org/abs/2402\.17764](https://arxiv.org/abs/2402.17764)\.
- Mouzouni \[2026\]Charafeddine Mouzouni\.Three phases of expert routing: How load balance evolves during mixture\-of\-experts training\.*arXiv preprint arXiv:2604\.04230*, 2026\.URL[https://arxiv\.org/abs/2604\.04230](https://arxiv.org/abs/2604.04230)\.
- Raje et al\. \[2026\]Arian Raje, Anupam Nayak, and Gauri Joshi\.MELINOE: Fine\-tuning enables memory\-efficient inference for mixture\-of\-experts models\.*arXiv preprint arXiv:2602\.11192*, 2026\.URL[https://arxiv\.org/abs/2602\.11192](https://arxiv.org/abs/2602.11192)\.
- Shazeer et al\. \[2017\]Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean\.Outrageously large neural networks: The sparsely\-gated mixture\-of\-experts layer\.*arXiv preprint arXiv:1701\.06538*, 2017\.URL[https://arxiv\.org/abs/1701\.06538](https://arxiv.org/abs/1701.06538)\.
- Sheng et al\. \[2023\]Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y\. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E\. Gonzalez, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang\.Flexgen: High\-throughput generative inference of large language models with a single gpu\.*arXiv preprint arXiv:2303\.06865*, 2023\.URL[https://arxiv\.org/abs/2303\.06865](https://arxiv.org/abs/2303.06865)\.
- Skliar et al\. \[2025\]Andrii Skliar, Ties van Rozendaal, Romain Lepert, Todor Boinovski, Mart van Baalen, Markus Nagel, Paul Whatmough, and Babak Ehteshami Bejnordi\.Mixture of cache\-conditional experts for efficient mobile device inference\.*Transactions on Machine Learning Research \(TMLR\)*, 2025\.URL[https://arxiv\.org/abs/2412\.00099](https://arxiv.org/abs/2412.00099)\.
- Song et al\. \[2024\]Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen\.Powerinfer: Fast large language model serving with a consumer\-grade gpu\.In*Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles \(SOSP\)*, 2024\.URL[https://arxiv\.org/abs/2312\.12456](https://arxiv.org/abs/2312.12456)\.
- Tang et al\. \[2024\]Peng Tang, Jiacheng Liu, Xiaofeng Hou, Yifei Pu, Jing Wang, Pheng\-Ann Heng, Chao Li, and Minyi Guo\.Hobbit: A mixed precision expert offloading system for fast moe inference\.*arXiv preprint arXiv:2411\.01433*, 2024\.URL[https://arxiv\.org/abs/2411\.01433](https://arxiv.org/abs/2411.01433)\.
- Xiao et al\. \[2024\]Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis\.Efficient streaming language models with attention sinks\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.URL[https://arxiv\.org/abs/2309\.17453](https://arxiv.org/abs/2309.17453)\.
- Xue et al\. \[2024a\]Leyang Xue, Yao Fu, Zhan Lu, Luo Mai, and Mahesh Marina\.Moe\-infinity: Efficient moe inference on personal machines with sparsity\-aware expert cache\.*arXiv preprint arXiv:2401\.14361*, 2024a\.URL[https://arxiv\.org/abs/2401\.14361](https://arxiv.org/abs/2401.14361)\.
- Xue et al\. \[2024b\]Zhenliang Xue, Yixin Song, Zeyu Mi, Xinrui Zheng, Yubin Xia, and Haibo Chen\.Powerinfer\-2: Fast large language model inference on a smartphone\.*arXiv preprint arXiv:2406\.06282*, 2024b\.URL[https://arxiv\.org/abs/2406\.06282](https://arxiv.org/abs/2406.06282)\.
- Yang et al\. \[2025\]An Yang et al\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*, 2025\.URL[https://arxiv\.org/abs/2505\.09388](https://arxiv.org/abs/2505.09388)\.
- Yi et al\. \[2023\]Rongjie Yi, Liwei Guo, Shiyun Wei, Ao Zhou, Shangguang Wang, and Mengwei Xu\.Edgemoe: Empowering sparse large language models on mobile devices\.*arXiv preprint arXiv:2308\.14352*, 2023\.URL[https://arxiv\.org/abs/2308\.14352](https://arxiv.org/abs/2308.14352)\.
- Zhang et al\. \[2025\]Yujie Zhang, Shivam Aggarwal, and Tulika Mitra\.DAOP: Data\-aware offloading and predictive pre\-calculation for efficient moe inference\.*arXiv preprint arXiv:2501\.10375*, 2025\.URL[https://arxiv\.org/abs/2501\.10375](https://arxiv.org/abs/2501.10375)\.
- Zhong et al\. \[2024\]Shuzhang Zhong, Ling Liang, Yuan Wang, Runsheng Wang, Ru Huang, and Meng Li\.Adapmoe: Adaptive sensitivity\-based expert gating and management for efficient moe inference\.*arXiv preprint arXiv:2408\.10284*, 2024\.URL[https://arxiv\.org/abs/2408\.10284](https://arxiv.org/abs/2408.10284)\.
- Zhou et al\. \[2025\]Jixian Zhou, Fang Dong, Ruijun Huang, Hengjie Cao, Mengyi Chen, Yifeng Yang, Anrui Chen, Mingzhi Dong, Yujiang Wang, Dongsheng Li, David A\. Clifton, Qin Lv, Rui Zhu, Chun Zhang, Fan Yang, Tun Lu, Ning Gu, and Li Shang\.Oracle\-MoE: Locality\-preserving routing in the oracle space for memory\-constrained large language model inference\.In*Proceedings of the 42nd International Conference on Machine Learning \(ICML\)*, volume 267 of*PMLR*, 2025\.URL[https://proceedings\.mlr\.press/v267/zhou25b\.html](https://proceedings.mlr.press/v267/zhou25b.html)\.
- Zhu et al\. \[2026\]Xiongwei Zhu, Xiaojian Liao, Tianyang Jiang, Yusen Zhang, Liang Wang, and Limin Xiao\.ReMoE: Boosting expert reuse through router fine\-tuning in memory\-constrained moe llm inference\.*arXiv preprint arXiv:2605\.27081*, 2026\.URL[https://arxiv\.org/abs/2605\.27081](https://arxiv.org/abs/2605.27081)\.