MESH: Memory-Efficient Sinkhorn Optimization for Mixture-of-Experts Training
Summary
This paper introduces MESH, a memory-efficient Sinkhorn-based optimizer for Mixture-of-Experts (MoE) training that restores temporal momentum without storing full optimizer state, reducing memory by 62.5% while maintaining competitive evaluation loss compared to AdamW.
View Cached Full Text
Cached at: 08/06/26, 07:49 AM
# MESH: Memory-Efficient Sinkhorn Optimization for Mixture-of-Experts Training
Source: [https://arxiv.org/html/2608.04407](https://arxiv.org/html/2608.04407)
###### Abstract
Memory\-efficient matrix optimizers such as Sinkhorn gradient descent remove most AdamW optimizer state for dense Transformer matrices, but direct application to Mixture\-of\-Experts \(MoE\) training is unreliable\. We study this failure in a controlled 110M\-parameter nanowhale DeepSeek\-style MoE pretraining setting\. A SAGE/Sinkhorn hybrid reduces optimizer state from 0\.883GB to 0\.331GB but degrades evaluation loss to 3\.8265, far above the AdamW baselines observed in the same setup \(3\.58–3\.64 across the seeds we study\)\. We show that routed MoE expert matrices are the dominant failure point: their gradients are conditional, temporally varying, and poorly served by stateless Sinkhorn normalization\. We proposeMESH, a hidden\-momentum Sinkhorn update for MoE experts\.MESHrestores a temporal first\-moment signal through the gradient\-buffer lifecycle, without storing the expert first moment as optimizer state\.MESH\-Bis an optional block\-preconditioned variant that adds a coarse neuron/block inverse\-RMS multiplier\. Across ablations, temporal smoothing before matrix normalization is the primary causal ingredient; block/neuron preconditioning can improve the memory\-quality frontier, but is not established as universally necessary\. In two additional seeds,MESHandMESH\-Breduce optimizer\-state memory by 62\.5% and peak PyTorch CUDA allocation by about 12\.6% relative to AdamW, with a modest evaluation\-loss gap\. Full\-state diagnostic variants recover AdamW\-like performance in ablations, supporting the conclusion that MoE experts need temporal smoothing, but not necessarily full coordinate\-wise AdamW state\.
Index Terms—Mixture\-of\-Experts, memory\-efficient optimization, Sinkhorn optimization, hidden momentum, language model pretraining
## 1Introduction
AdamW remains the default optimizer for language\-model pretraining because it combines temporal first\-moment smoothing with coordinate\-wise adaptive scaling\[[1](https://arxiv.org/html/2608.04407#bib.bib1),[2](https://arxiv.org/html/2608.04407#bib.bib2)\]\. Its memory cost, however, is substantial: AdamW stores two full state tensors per trainable parameter\. This motivates optimizers that reduce or remove moment state\. Sinkhorn\-style matrix optimizers normalize gradient matrices by alternating row and column operations, yielding stateless updates for dense two\-dimensional weights\[[4](https://arxiv.org/html/2608.04407#bib.bib4)\]\. SAGE\-style hybrids further recognize that embeddings are difficult for purely stateless methods and assign light\-state updates to vocabulary tensors while leaving dense matrices to Sinkhorn\[[5](https://arxiv.org/html/2608.04407#bib.bib5)\]\.
These role assignments were developed mainly for dense Transformer settings\. We ask whether they transfer to Mixture\-of\-Experts \(MoE\) language models\. The answer is negative: a routed expert matrix is two\-dimensional, but its gradient is not the same object as a dense MLP gradient\. Each expert receives a router\-selected token subset, weighted by learned gate scores, and this conditional distribution changes across training\. In a nanowhale 110M DeepSeek\-style MoE model, a direct SAGE/Sinkhorn hybrid lowers optimizer state to 0\.331GB but reaches only 3\.8265 evaluation loss, far worse than AdamW\. Applying SAGE to experts is worse still, and reducing Sinkhorn scale, fusing expert tensors, or using route\-confidence damping does not close the gap\.
The key question is therefore not whether Sinkhorn can be memory\-efficient, but what it lacks for routed MoE experts\. Our ablations indicate that the primary missing ingredient is temporal smoothing before matrix normalization\. Stateless Sinkhorn computesSinkhorn\(Gt\)\\mathrm\{Sinkhorn\}\(G\_\{t\}\)from the instantaneous expert gradient\. In a routed expert,GtG\_\{t\}is a high\-variance, conditional sample of an expert\-specific token distribution\. Because Sinkhorn is nonlinear,Sinkhorn\(𝔼\[Gt\]\)\\mathrm\{Sinkhorn\}\(\\mathbb\{E\}\[G\_\{t\}\]\)and𝔼\[Sinkhorn\(Gt\)\]\\mathbb\{E\}\[\\mathrm\{Sinkhorn\}\(G\_\{t\}\)\]can behave differently; the latter amplifies step\-to\-step routing noise\. The optimizer failure is an ordering problem: normalize\-then\-average behaves differently from average\-then\-normalize\.MESHconverts the update toSinkhorn\(Ht\)\\mathrm\{Sinkhorn\}\(H\_\{t\}\), whereHtH\_\{t\}averages the expert’s gradient history before matrix normalization\.
We proposeMESH, a MoE\-aware, hidden\-momentum Sinkhorn optimizer\.MESHkeeps the SAGE/Sinkhorn hybrid for vocabulary and non\-expert matrices, but replaces stateless expert Sinkhorn with hidden\-momentum expert Sinkhorn\. Its expert first moment is maintained through the gradient\-buffer lifecycle rather than as an optimizer\-state tensor\. We also evaluateMESH\-B, which adds a coarse block/neuron inverse\-RMS multiplier\. The main empirical message is deliberately conservative: MoE experts primarily need temporal smoothing before matrix normalization; block\-level adaptivity can help, especially in seed\-42 diagnostics, but is not the sole causal ingredient\.
Our contributions are:
- •We identify routed MoE experts as the failure point of direct SAGE/Sinkhorn optimization in nanowhale pretraining\.
- •We introduceMESH, a hidden\-momentum Sinkhorn expert update that restores a first\-moment signal without adding explicit expert first\-moment optimizer state\.
- •We evaluateMESH\-B, an optional block\-preconditioned extension, and show that the unblocked and block\-preconditioned hidden\-momentum variants give similar multi\-seed memory\-quality trade\-offs\.
- •We provide negative ablations showing that second\-moment\-only, sign/momentum\-only, confidence\-only, scale\-only, and fused\-Sinkhorn interventions do not explain the gap\.
## 2Related Work
### 2\.1Adaptive and memory\-efficient optimization
Adam and AdamW maintain first\- and second\-moment estimates for every parameter\[[1](https://arxiv.org/html/2608.04407#bib.bib1),[2](https://arxiv.org/html/2608.04407#bib.bib2)\]\. This robustness costs two state tensors per parameter\. Memory\-efficient Adam variants reduce this cost in different ways: Adam\-mini uses fewer learning\-rate resources by exploiting block structure\[[7](https://arxiv.org/html/2608.04407#bib.bib7)\], and HMAdam/HMAdamW remove the explicit first\-moment tensor by storing a momentum\-like quantity in the gradient buffer\[[6](https://arxiv.org/html/2608.04407#bib.bib6)\]\.MESHuses the latter idea only for MoE expert first moments and combines it with Sinkhorn\-style matrix normalization\.
### 2\.2Matrix normalization and SAGE hybrids
Sinkhorn and gradient multi\-normalization optimizers compute matrix updates through row and column normalization\[[3](https://arxiv.org/html/2608.04407#bib.bib3),[4](https://arxiv.org/html/2608.04407#bib.bib4)\]\. Their appeal is that a large dense matrix can be updated without storing AdamW’s two full state tensors\. SAGE extends this strategy by treating embeddings as a separate difficult class\[[5](https://arxiv.org/html/2608.04407#bib.bib5)\]\. Our work identifies a second difficult class: routed MoE expert matrices\.
### 2\.3Mixture\-of\-Experts models
Sparse MoE layers route tokens to a subset of experts, increasing model capacity without activating every parameter on every token\[[8](https://arxiv.org/html/2608.04407#bib.bib8),[9](https://arxiv.org/html/2608.04407#bib.bib9),[10](https://arxiv.org/html/2608.04407#bib.bib10)\]\. This routing changes the optimizer problem\. Expert gradients are conditional on router decisions and gate weights, so a routed expert matrix is not equivalent to a dense MLP matrix even when it has the same shape\.MESHis designed for this expert\-specific gradient structure\.
## 3Background and Diagnosis
### 3\.1Sinkhorn updates for dense matrices
LetW∈ℝm×nW\\in\\mathbb\{R\}^\{m\\times n\}be a matrix parameter andGt=∇WℒtG\_\{t\}=\\nabla\_\{W\}\\mathcal\{L\}\_\{t\}its gradient at steptt\. A Sinkhorn\-style matrix optimizer forms an update direction by repeatedly normalizing row and column norms,
Ut=Sinkhorn\(Gt;K,γ\),U\_\{t\}=\\mathrm\{Sinkhorn\}\(G\_\{t\};K,\\gamma\),\(1\)whereKKis the number of normalization rounds andγ\\gammais a scalar update scale\. This update is attractive because it is stateless: it uses the current matrix gradient and stores no first or second moment for dense matrices\.
### 3\.2Why routed experts are not ordinary dense matrices
A routed MoE expert weight is also a matrix, but its gradient has a different sampling structure\. In a top\-kkMoE layer, experteereceives only tokens selected by the router:
Ge,t=∑i:e∈topk\(xi,t\)ai,e,tgi,e,t,G\_\{e,t\}=\\sum\_\{i:e\\in\\mathrm\{topk\}\(x\_\{i\},t\)\}a\_\{i,e,t\}\\,g\_\{i,e,t\},\(2\)whereai,e,ta\_\{i,e,t\}is the gate weight\. ThusGe,tG\_\{e,t\}is a gradient under the conditional distributionp\(x∣e∈topk\(x,t\)\)p\(x\\mid e\\in\\mathrm\{topk\}\(x,t\)\), not under the full mini\-batch distribution\. This conditional distribution changes as the router and hidden states change during training\.
This matters because Sinkhorn is nonlinear\. Dense training can often tolerateSinkhorn\(Gt\)\\mathrm\{Sinkhorn\}\(G\_\{t\}\)becauseGtG\_\{t\}is a reasonably stable estimate of a full\-batch dense\-matrix gradient\. In routed experts, however, the sequenceGe,tG\_\{e,t\}carries routing\-induced temporal noise\. Applying Sinkhorn before smoothing computesSinkhorn\(Ge,t\)\\mathrm\{Sinkhorn\}\(G\_\{e,t\}\)for each noisy expert sample\. Momentum instead approximates applying Sinkhorn to a temporally averaged expert signal\. This distinction is the central diagnosis of the paper\.
### 3\.3Hypotheses we test
We considered several explanations for the expert failure\. Raw route\-count sparsity could reduce effective batch size; gate\-weight skew could produce effective sparsity even when every expert is selected; semantic routing could make expert gradients non\-i\.i\.d\.; Sinkhorn could be sensitive to expert tensor partitioning; or stateless Sinkhorn could simply lack temporal smoothing\. Our experiments rule out the first four as complete explanations\. The remaining supported hypothesis is that routed experts require a first\-moment signal before matrix normalization\. Coarse block/neuron preconditioning can improve this update, but the multi\-seed results indicate that the first moment is the primary causal ingredient\.
## 4MESH
### 4\.1Problem: MoE expert gradients are not dense\-matrix gradients
LetWeW\_\{e\}be an expert matrix andGe,t=∇WeℒtG\_\{e,t\}=\\nabla\_\{W\_\{e\}\}\\mathcal\{L\}\_\{t\}its gradient at steptt\. A dense MLP matrix receives gradient contributions from essentially all tokens in a minibatch\. A routed expert instead receives
Ge,t=∑i∈ℛe,tai,e,tgi\(We\),G\_\{e,t\}=\\sum\_\{i\\in\\mathcal\{R\}\_\{e,t\}\}a\_\{i,e,t\}\\,g\_\{i\}\(W\_\{e\}\),\(3\)whereℛe,t\\mathcal\{R\}\_\{e,t\}is the token subset selected for experteeandai,e,ta\_\{i,e,t\}is the gate weight\. Both the subset and the weights change with the router and hidden states\. Thus the expert gradient is a temporally varying conditional estimate\. Applying stateless Sinkhorn directly toGe,tG\_\{e,t\}uses
Ue,t=Sinkhorn\(Ge,t;K,τ\),U\_\{e,t\}=\\mathrm\{Sinkhorn\}\(G\_\{e,t\};K,\\tau\),\(4\)which normalizes the instantaneous matrix without remembering whether the expert signal is persistent or a one\-step routing fluctuation\.
MESH\-Badditionally multiplies the Sinkhorn direction by a coarse mean\-normalized block/neuron inverse\-RMS factor\.
Fig\. 1:Core MESH mechanism\. The failure is an ordering issue: normalizing each instantaneous routed\-expert gradient is not equivalent to normalizing a temporally averaged expert signal\.
### 4\.2MESH: hidden momentum before Sinkhorn
MESHchanges only the MoE expert role\. The primary algorithm,MESH, maintains an unnormalized expert momentum accumulator
He,t=β1He,t−1\+Ge,t\.H\_\{e,t\}=\\beta\_\{1\}H\_\{e,t\-1\}\+G\_\{e,t\}\.\(5\)This differs by a constant scale from the conventional Adam first moment\. We useHe,tH\_\{e,t\}directly because Sinkhorn normalization and mean\-normalized block multipliers are invariant to global scaling up to the explicit update scale\. The expert matrix direction is then
Ue,t=Sinkhorn\(He,t;K,τ\),U\_\{e,t\}=\\mathrm\{Sinkhorn\}\(H\_\{e,t\};K,\\tau\),\(6\)whereKKis the number of alternating row/column normalization rounds andτ\\tauis the Sinkhorn update scale\.
The hidden\-momentum implementation storesHe,tH\_\{e,t\}outside the optimizer\-state dictionary\. After an optimizer step, the expert gradient buffer is stashed\. Immediately before the next backward pass, the buffer is reattached asp\.grad←β1He,tp\.grad\\leftarrow\\beta\_\{1\}H\_\{e,t\}\. Backward accumulation adds the current gradient, yieldingHe,t\+1H\_\{e,t\+1\}\. This prepare\-before\-backward lifecycle was necessary in Trainer/Accelerate/DDP\-style loops, where a naivezero\_gradbridge can be bypassed by wrapped\-model gradient clearing\. Diagnostics in our final implementation report approximately one prepare call per optimizer step, buffer\-shadow cosine near one, and relative error around10−810^\{\-8\}\.
### 4\.3MESH\-B: optional block/neuron scaling
MESHcan be used as hidden\-momentum Sinkhorn alone\. We also evaluate a block\-scaled variant, denotedMESH\-B, which applies a coarse inverse\-RMS multiplier to the Sinkhorn direction\. For expertw1,w3w\_\{1\},w\_\{3\}matrices we use row statistics; forw2w\_\{2\}we use column statistics\. LetR\(He,t\)R\(H\_\{e,t\}\)denote this block RMS\. The defaultMESH\-Bstatistic is computed from the smoothed sourceHe,tH\_\{e,t\}rather than a separate raw\-gradient stream:
Ve,t=β2Ve,t−1\+\(1−β2\)R\(He,t\)2\.V\_\{e,t\}=\\beta\_\{2\}V\_\{e,t\-1\}\+\(1\-\\beta\_\{2\}\)R\(H\_\{e,t\}\)^\{2\}\.\(7\)We useHHfor this statistic because the best seed\-42 diagnostic usedV\(H\)V\(H\), and because it keeps the hidden\-momentum implementation simple without storing an additional raw\-gradient stream\. R aw\-gradient block statistics are a natural alternative, but we leave a systematic comparison to future work\.
The multiplier is clipped and mean\-normalized,
Qe,t=clip\(\(Ve,t\+ϵ\)−p/2mean\(\(Ve,t\+ϵ\)−p/2\),qmin,qmax\),Q\_\{e,t\}=\\mathrm\{clip\}\\left\(\\frac\{\(V\_\{e,t\}\+\\epsilon\)^\{\-p/2\}\}\{\\mathrm\{mean\}\(\(V\_\{e,t\}\+\\epsilon\)^\{\-p/2\}\)\},q\_\{\\min\},q\_\{\\max\}\\right\),\(8\)and the expert update is
We,t\+1=We,t−ηQe,t⊙Ue,t−ηλWe,t,W\_\{e,t\+1\}=W\_\{e,t\}\-\\eta\\,Q\_\{e,t\}\\odot U\_\{e,t\}\-\\eta\\lambda W\_\{e,t\},\(9\)withQe,tQ\_\{e,t\}broadcast over the appropriate block dimension\. The block multiplier is intentionally coarse\. It is not a full coordinate\-wise second moment and is not the central causal claim of the paper\. Our results suggest that temporal smoothing is the primary requirement, while block/neuron scaling can improve the frontier in some settings\.
MESH family role assignment\.
Fig\. 2:MESHkeeps the SAGE/Sinkhorn hybrid for non\-expert parameters and replaces only the MoE expert update\.MESHis hidden\-momentum expert Sinkhorn;MESH\-Badds optional block preconditioning\. Full\-state variants are diagnostic controls\.
### 4\.4Full\-state diagnostic controls
We use full\-state variants only for diagnosis\.MESH\-Fullstores the same unnormalized expert momentumHe,tH\_\{e,t\}explicitly in optimizer state rather than in the gradient buffer\.MESH\-B\-Fulladditionally stores the block statistic used byMESH\-B\. Comparing hidden and full\-state variants isolates the systems effect of hidden momentum from the optimizer effect of temporal smoothing\. In our experiments, full\-state variants are often stronger quality diagnostics, whileMESHandMESH\-Bare the memory\-efficient algorithms\.
## 5Experiments
### 5\.1Setup
We use the HuggingFace nanowhale training codebase, which implements a 110M\-parameter DeepSeek\-V4\-style MoE language model\[[11](https://arxiv.org/html/2608.04407#bib.bib11)\]\. The model has eight layers, four routed experts and one shared expert per layer, hidden size 320, and an untied 129,280\-token vocabulary\. The vocabulary matrices are unusually large for this model size: input embedding plus LM head contain 82\.75M of 110\.37M trainable parameters\.
Unless noted, runs train from scratch on FineWeb\-Edu for 5,000 optimizer steps with sequence length 2048, per\-device batch size 8, gradient accumulation 4, and eight data\-parallel workers\. This corresponds to an estimated 524,288 tokens per optimizer step and 2\.62B model tokens per run\. Evaluation uses 512 held\-out examples\. Because several differences are in the 0\.01–0\.02 loss range, we interpret small gaps as suggestive rather than conclusive\.
SAGE/Sinkhorn andMESHconfigurations follow the official SAGE\-style hyperparameters unless stated otherwise: learning rate2×10−32\\times 10^\{\-3\},β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.99\\beta\_\{2\}=0\.99, weight decay 0\.01, cosine schedule, 10% warmup, Sinkhorn scaleτ=10\\tau=10, andK=5K=5normalization rounds\. The AdamW baseline uses the nanowhale\-tuned configuration used in the project logs: learning rate6×10−46\\times 10^\{\-4\},β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.95\\beta\_\{2\}=0\.95, weight decay 0\.1, cosine schedule, and 3% warmup\. We report evaluation loss, mean token accuracy, persistent optimizer\-state memory, and peak PyTorch CUDA allocation\.
Table 1:Main multi\-seed comparison\. These are the runs for which post\-fix hidden\-momentum results are available at seeds 100 and 512\.MESHis hidden\-momentum Sinkhorn without block preconditioning\.MESH\-BisMESHwith a block/neuron inverse\-RMS multiplier\. State denotes persistent optimizer\-state memory\.Table 2:Seed\-42 diagnostic comparison\. These single\-seed ablations should not be directly averaged with Table[1](https://arxiv.org/html/2608.04407#S5.T1); they identify mechanisms\.
### 5\.2Main results
Table[1](https://arxiv.org/html/2608.04407#S5.T1)shows thatMESH\-Bachieves a strong memory\-quality trade\-off but does not outperform AdamW in the additional seeds\. Relative to AdamW,MESH\-Breduces optimizer\-state memory by 62\.5% and peak PyTorch CUDA allocation by about 12\.6%, but trails AdamW by about 0\.049 evaluation loss\.MESHwithout the block multiplier is close toMESH\-B, suggesting that hidden temporal smoothing is the dominant ingredient and that block preconditioning is an incremental improvement in these seeds\.
Table[2](https://arxiv.org/html/2608.04407#S5.T2)separates diagnostic seed\-42 ablations from the multi\-seed table\. In this seed, full\-stateMESH\-Bvariants recover AdamW\-like quality with about half of AdamW’s optimizer\-state memory\. We use these runs as mechanistic evidence rather than as final multi\-seed performance claims\.
### 5\.3Where the SAGE/Sinkhorn hybrid fails
Table 3:Expert role ablation, seed 42\.Table[3](https://arxiv.org/html/2608.04407#S5.T3)shows that routed experts are the dominant failure point\. Converting only routed experts to AdamW recovers most of the gap, while converting only the shared expert helps less\. Applying SAGE to all experts worsens training, indicating that the embedding\-oriented SAGE update is not a suitable expert replacement\.
### 5\.4What the expert update needs
Table 4:Expert update decomposition, seed 42\. These results motivateMESHbut do not imply that every component is universally necessary\.Table[4](https://arxiv.org/html/2608.04407#S5.T4)shows that adding a temporal first moment to expert Sinkhorn is the major improvement\. V\-only, sign\-only, and factored\-v controls are poor\. Block/neuron preconditioning can improve the best single\-seed results, but the closeness ofMESHandMESH\-Bin Table[1](https://arxiv.org/html/2608.04407#S5.T1)prevents a stronger claim that block preconditioning is universally required\.
### 5\.5Memory accounting
AdamW stores two full moment states for all 110\.37M parameters, producing 0\.883GB optimizer state\. Direct SAGE/Sinkhorn stores state mainly for the 82\.75M vocabulary/SAGE parameters, producing 0\.331GB\.MESHkeeps this same optimizer\-state size: the expert first moment is carried by the gradient\-buffer lifecycle and counted separately as a 0\.098GB hidden buffer, not as optimizer state\. Although the hidden buffer is excluded from optimizer\-state accounting because it lives in the gradient\-buffer lifecycle, it is still part of the persistent training memory and is reflected in CUDA peak measurements\.MESH\-Badds only small block statistics at this model size, so its reported optimizer\-state memory remains 0\.331GB at three\-decimal precision\.MESH\-B\-Fullstores the expert first moment explicitly, increasing state to about 0\.430GB\. Peak CUDA allocation decreases less than optimizer\-state memory because parameters, activations, communication buffers, gradient buffers, temporary update tensors, and allocator effects remain\.
## 6Analysis and Limitations
### 6\.1Why routed experts needed momentum
The routed expert failure is best understood as an ordering problem\. Stateless Sinkhorn appliesSinkhorn\(Gt\)\\mathrm\{Sinkhorn\}\(G\_\{t\}\)at every step\.MESHappliesSinkhorn\(Ht\)\\mathrm\{Sinkhorn\}\(H\_\{t\}\), whereHtH\_\{t\}averages the expert gradient sequence over time\. Because Sinkhorn is nonlinear,Sinkhorn\(𝔼G\)\\mathrm\{Sinkhorn\}\(\\mathbb\{E\}G\)is not equivalent to𝔼\[Sinkhorn\(G\)\]\\mathbb\{E\}\[\\mathrm\{Sinkhorn\}\(G\)\]\. For routed experts, this difference is large: averaging first filters routing\-induced temporal noise, and only then applies matrix normalization\. This explains why momentum Sinkhorn improves substantially over instantaneous Sinkhorn, even before adding a block multiplier\.
The need for momentum is specific to the routed path\. A shared expert is active for every token and behaves more like a dense MLP path\. In contrast, a routed expert sees a router\-conditioned token subset with learned gate weights\. Its gradient is therefore a sequence of conditional samples rather than a sequence of dense minibatch estimates\. Momentum tracks the expert’s persistent direction across these samples\.
### 6\.2What block preconditioning contributes
Block/neuron inverse\-RMS scaling improves some single\-seed settings\. In seed 42,MESH\-B\-Fullreaches 3\.6187 eval loss, and theV\(H\)V\(H\)variant reaches 3\.6102\. However, in seeds 100 and 512, hidden\-momentum Sinkhorn without block preconditioning is close to the block\-preconditionedMESH\-Bresult\. We therefore treat block preconditioning as a useful coarse adaptive scale, not as a conclusively necessary component\. The safest conclusion is that MoE expert optimization primarily requires temporal smoothing; block/neuron adaptivity can improve the frontier and deserves further study\.
### 6\.3Residual gap to AdamW
The strongest AdamW seeds remain better thanMESHorMESH\-Bin the current experiment\. This residual gap may not be caused by expert optimization alone\. The nanowhale configuration is vocabulary\-heavy: input embedding plus untied LM head account for approximately 75% of trainable parameters\.MESHleaves those matrices on SAGE, while AdamW adapts them with full first and second moments\. Non\-expert dense matrices also remain on stateless Sinkhorn\. Future role ablations should therefore keepMESHon experts while assigning only vocabulary matrices or only non\-expert dense matrices to AdamW\.
### 6\.4Limitations
The study is intentionally narrow\. It uses one 110M MoE architecture, one streaming corpus, 5,000\-step runs, and primarily 512\-example evaluations\. Several gaps are small enough that larger evaluation sets and more seeds are needed\. We also do not claim thatMESHoutperforms AdamW\. Rather,MESHprovides a substantially lower\-state point on the memory\-quality frontier and identifies temporal smoothing before matrix normalization as a key missing ingredient for MoE expert Sinkhorn optimization\.
## 7Conclusion
We presentedMESH, a hidden\-momentum Sinkhorn optimizer for MoE expert matrices, andMESH\-B, an optional block\-preconditioned variant\. The study shows that direct SAGE/Sinkhorn optimization fails on routed MoE experts because the expert gradient sequence is conditional and temporally noisy\. The missing ingredient is primarily temporal smoothing before matrix normalization\. Hidden momentum supplies this signal without storing an explicit expert first\-moment tensor in optimizer state\. The resulting algorithms reduce optimizer\-state memory substantially relative to AdamW while recovering much of the quality lost by stateless Sinkhorn\. The remaining gap to AdamW likely involves non\-expert role assignment, especially vocabulary matrices, and motivates future work on memory\-efficient updates for the entire MoE parameter taxonomy\.
## References
- \[1\]D\. P\. Kingma and J\. Ba, “Adam: A method for stochastic optimization,” in*International Conference on Learning Representations*, 2015\.
- \[2\]I\. Loshchilov and F\. Hutter, “Decoupled weight decay regularization,” in*International Conference on Learning Representations*, 2019\.
- \[3\]R\. Sinkhorn, “A relationship between arbitrary positive matrices and doubly stochastic matrices,”*Annals of Mathematical Statistics*, vol\. 35, no\. 2, pp\. 876–879, 1964\.
- \[4\]M\. Scetbon, C\. Ma, W\. Gong, and E\. Meeds, “Gradient multi\-normalization for efficient LLM training,”*OpenReview*, 2025\. Available:[https://openreview\.net/forum?id=oanhUGY6un](https://openreview.net/forum?id=oanhUGY6un)
- \[5\]W\. Lee and H\.\-T\. Kim, “SAGE: Sign\-adaptive gradient for memory\-efficient LLM optimization,”*arXiv:2604\.07663*, 2026\.
- \[6\]M\. A\. Sadeghi, “Eliminating the first moment state in Adam optimizer,”*OpenReview*, 2025\. Available:[https://openreview\.net/forum?id=xRxh48OAAM](https://openreview.net/forum?id=xRxh48OAAM)
- \[7\]Y\. Zhang*et al\.*, “Adam\-mini: Use fewer learning rates to gain more,”*arXiv:2406\.16793*, 2024\.
- \[8\]N\. Shazeer*et al\.*, “Outrageously large neural networks: The sparsely\-gated mixture\-of\-experts layer,” in*International Conference on Learning Representations*, 2017\.
- \[9\]D\. Lepikhin*et al\.*, “GShard: Scaling giant models with conditional computation and automatic sharding,” in*International Conference on Learning Representations*, 2021\.
- \[10\]W\. Fedus, B\. Zoph, and N\. Shazeer, “Switch Transformers: Scaling to trillion parameter models with simple and efficient sparsity,”*Journal of Machine Learning Research*, 2022\.
- \[11\]Hugging Face, “nanowhale: A 110M\-parameter DeepSeek\-style language model,” GitHub repository, 2025\. Available:[https://github\.com/huggingface/nanowhale](https://github.com/huggingface/nanowhale)
## Appendix AAdditional Experimental Results
### A\.1Full result tables
Table[5](https://arxiv.org/html/2608.04407#A1.T5)contains the multi\-seed comparison used in the main text\. Table[6](https://arxiv.org/html/2608.04407#A1.T6)collects single\-seed expert\-update ablations\. Table[7](https://arxiv.org/html/2608.04407#A1.T7)collects routing controls\. Table[8](https://arxiv.org/html/2608.04407#A1.T8)lists negative or partial causal interventions\. AdamW baselines use learning rate6×10−46\\times 10^\{\-4\},β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.95\\beta\_\{2\}=0\.95, weight decay 0\.1, cosine decay, and 3% warmup\. SAGE/MESH\-style runs use learning rate2×10−32\\times 10^\{\-3\},β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.99\\beta\_\{2\}=0\.99, weight decay 0\.01, cosine decay, 10% warmup, Sinkhorn scale 10, and five Sinkhorn rounds unless otherwise stated\.
Table 5:Post\-fix hidden\-momentum multi\-seed comparison\.MESHdenotes hidden\-momentum expert Sinkhorn without block preconditioning;MESH\-Bdenotes the optional block\-preconditioned variant\.Table 6:Expert optimizer decomposition, seed 42 unless marked otherwise\. Full\-state variants are diagnostic controls; the proposed memory\-efficient algorithms are hidden\-momentum variants in Table[5](https://arxiv.org/html/2608.04407#A1.T5)\.Table 7:Routing and dense\-like controls\. Top\-k and routing interventions do not remove the SAGE/Sinkhorn gap\.Table 8:Negative and partial interventions\.
## Appendix BPseudocode and Implementation Details
### B\.1Sinkhorn normalization
Algorithm 1: SinkhornNormalize\(G,K,ϵ\)\(G,K,\\epsilon\)
3:
X←X/\(‖Xr:‖2\+ϵ\)X\\leftarrow X/\(\\\|X\_\{r:\}\\\|\_\{2\}\+\\epsilon\)\{row normalization\}
4:
X←X/\(‖X:c‖2\+ϵ\)X\\leftarrow X/\(\\\|X\_\{:c\}\\\|\_\{2\}\+\\epsilon\)\{column normalization\}
5:endfor
### B\.2MESH expert step
Algorithm 2: MESH expert update
0:expert parameter
WW, hidden stash
HH, learning rate
η\\eta, weight decay
λ\\lambda, Sinkhorn scale
γ\\gamma
1:Before backward:if
HHexists, set
grad\(W\)←β1H\\mathrm\{grad\}\(W\)\\leftarrow\\beta\_\{1\}H; otherwise set
grad\(W\)←0\\mathrm\{grad\}\(W\)\\leftarrow 0\.
2:Run backward\. The gradient slot now contains
Ht=β1Ht−1\+GtH\_\{t\}=\\beta\_\{1\}H\_\{t\-1\}\+G\_\{t\}\.
3:
Dt←SinkhornNormalize\(Ht,K,ϵ\)D\_\{t\}\\leftarrow\\mathrm\{SinkhornNormalize\}\(H\_\{t\},K,\\epsilon\)\.
4:
W←W−ηγDt−ηλWW\\leftarrow W\-\\eta\\gamma D\_\{t\}\-\\eta\\lambda W\.
5:Stash
H←detach\(Ht\)H\\leftarrow\\mathrm\{detach\}\(H\_\{t\}\)outside the optimizer\-state dictionary\.
6:Clear the gradient slot normally\.
### B\.3Optional MESH\-B block\-preconditioned expert step
Algorithm 3: MESH\-B expert update
0:expert parameter
WW, hidden accumulator
HtH\_\{t\}, block state
VV, power
pp
1:
Dt←SinkhornNormalize\(Ht,K,ϵ\)D\_\{t\}\\leftarrow\\mathrm\{SinkhornNormalize\}\(H\_\{t\},K,\\epsilon\)\.
2:
Bt←ρ\(Ht\)B\_\{t\}\\leftarrow\\rho\(H\_\{t\}\), where
ρ\\rhois row RMS for
W1/W3W\_\{1\}/W\_\{3\}and column RMS for
W2W\_\{2\}\.
3:
V←β2V\+\(1−β2\)Bt2V\\leftarrow\\beta\_\{2\}V\+\(1\-\\beta\_\{2\}\)B\_\{t\}^\{2\}\.
4:
P←clip\(\(V\+ϵ\)−p/2mean\(\(V\+ϵ\)−p/2\),cmin,cmax\)P\\leftarrow\\mathrm\{clip\}\\left\(\\frac\{\(V\+\\epsilon\)^\{\-p/2\}\}\{\\mathrm\{mean\}\(\(V\+\\epsilon\)^\{\-p/2\}\)\},c\_\{\\min\},c\_\{\\max\}\\right\)\.
5:
W←W−ηγ\(P⊙Dt\)−ηλWW\\leftarrow W\-\\eta\\gamma\(P\\odot D\_\{t\}\)\-\\eta\\lambda W\.
We useHtH\_\{t\}rather than a separate raw\-gradient stream for the default block statistic\. This matches the strongest seed\-42 diagnostic, denotedV\(H\)V\(H\)in the tables, and avoids storing an additional full raw\-gradient stream in the hidden\-momentum implementation\.
### B\.4Full\-state diagnostic variants
The full\-state diagnostics,MESH\-FullandMESH\-B\-Full, storeHtH\_\{t\}explicitly in optimizer state rather than through the gradient buffer\.MESH\-Fullis otherwise identical to Algorithm 2, whileMESH\-B\-Fullis otherwise identical to Algorithm 3\. We use these variants to separate the value of temporal smoothing from the engineering question of whether hidden momentum can faithfully reproduce the explicit first moment\.
### B\.5Hidden momentum scale convention
The hidden buffer obeysHt=β1Ht−1\+GtH\_\{t\}=\\beta\_\{1\}H\_\{t\-1\}\+G\_\{t\}, while a conventional Adam\-style first moment usesMt=β1Mt−1\+\(1−β1\)GtM\_\{t\}=\\beta\_\{1\}M\_\{t\-1\}\+\(1\-\\beta\_\{1\}\)G\_\{t\}\. ThusHtH\_\{t\}is an unnormalized first\-moment accumulator\. Since Sinkhorn normalization and mean\-normalized block multipliers are invariant to global scaling up to the explicit update scale, we useHtH\_\{t\}directly\.
### B\.6Trainer lifecycle
A naiveoptimizer\.zero\_grad\(\)override was insufficient in our training stack because the Trainer, Accelerate, or distributed wrapper can clear gradients through model\-level paths\. The final implementation uses a prepare\-before\-backward hook\. The optimizer stashes hidden expert buffers after each step; immediately before backward, the trainer reattaches the decayed stash to the gradient slot\. Diagnostics on full runs report approximately one prepare call per optimizer step, buffer\-shadow cosine near one, and relative error around10−810^\{\-8\}\.
### B\.7Role counts in the main MESH configurations
In the mainMESHconfiguration, 85 tensors with 82\.747M parameters use SAGE, 65 non\-expert dense tensors with 3\.046M parameters use plain Sinkhorn, and 120 expert tensors with 24\.576M parameters use hidden\-momentum expert Sinkhorn\. InMESH\-B, the same 120 expert tensors additionally use the block/neuron multiplier\. The hidden expert buffer is estimated at 0\.098GB\.MESH\-Bhas the same expert coverage and adds small block statistics\. The optimizer\-state footprint is 0\.331GB at three\-decimal precision, dominated by the vocabulary/SAGE states\.Similar Articles
Where Should Optimizer State Live? Tiered State Allocation for Memory-Efficient Mixture-of-Experts Training
SkewAdam is a novel optimizer for mixture-of-experts models that tier allocates optimizer state across backbone, experts, and router, reducing memory footprint to 2.6% of AdamW while achieving better validation perplexity in controlled comparisons.
PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference
PuzzleMoE introduces a pairwise dual-mask expert merging algorithm and bit-level packing technique to compress large Mixture-of-Experts models, maintaining performance while reducing storage and accelerating inference.
Keeping Large MoE Training Within Fixed GPU Memory (20 minute read)
This paper introduces methods to bound four key memory peaks in large Mixture-of-Experts training, enabling training at 1M context length with fixed GPU memory and up to 10.4× throughput improvement over baselines.
Mixture of Experts (MoEs) in Transformers
Hugging Face blog post explaining Mixture of Experts (MoEs) architecture in Transformers, covering the shift from dense to sparse models, weight loading optimizations, expert parallelism, and training techniques for MoE-based language models.
Flattening Every Memory Peak in Long-Context Mixture-of-Experts Training
This paper introduces techniques to manage memory peaks in training large Mixture-of-Experts models with long context lengths, including Pipelined LLEP, Ring-DTP, SCO, and OffloadStreamAdamW, which enable fixed GPU working sets and improve throughput up to 10.4x.