Where Should Optimizer State Live? Tiered State Allocation for Memory-Efficient Mixture-of-Experts Training
Summary
SkewAdam is a novel optimizer for mixture-of-experts models that tier allocates optimizer state across backbone, experts, and router, reducing memory footprint to 2.6% of AdamW while achieving better validation perplexity in controlled comparisons.
View Cached Full Text
Cached at: 07/22/26, 10:42 AM
Paper page - Where Should Optimizer State Live? Tiered State Allocation for Memory-Efficient Mixture-of-Experts Training
Source: https://huggingface.co/papers/2607.19058
Abstract
Optimizerstateisthelargestsinglelineiteminthememorybudgetofmixture-of-experts(MoE)training:ona6.78B-parameterMoElanguagemodel,AdamWkeeps50.6GBoffirstandsecondmomentstoupdate12.6GBofbfloat16weights.WestudySkewAdam,anoptimizerbuiltontheobservationthatthethreeparameterpopulationsofanMoE-thedensebackbone,theexperts,andtherouter-differenoughinsizeandgradientstatisticsthattheyshouldnotreceivethesamestate.SkewAdamkeepsfloat32momentumplusafactoredsecondmomentforthebackbone(5%ofparameters),afactoredsecondmomentalonefortheexperts(95%),andanexactsecondmomentfortherouter(<0.01%).Theresultingstateoccupies1.29GB,2.6%ofAdamW’s,andpeaktrainingmemoryfallsfrom81.4GBto31.3GB,withinthebudgetofa40GBaccelerator.Inacontrolledcomparisonfromidenticalinitializationsover82Mtokens,SkewAdamreachesvalidationperplexity108.4,aheadofAdamW(126.8),Muon(120.2),andLion(393.7),andsettlesrouterloadbalancetowithin1%ofitsuniformfloor.Theallocationisnotwhatearnsthatperplexity:atierablationmatchesitwithtwentytimesthestate,andAdafactor,whichsharesthefactoredestimatorbutdropsmomentum,plateaus40pointsbehind.Thetiersbuymemoryatnocosttoaccuracy;theaccuracycomesfromkeepingmomentum,whichauniformoptimizersharestoo.Sweepingthebaselines’learningratesnarrowsbutdoesnotclosethegap:thebesttunedAdamWreaches118.5,tunedAdafactor139.7.Whereoptimizerstatelives,theseresultssuggest,mattersatleastasmuchashowmuchofitthereis.
View arXiv pageView PDFGitHub1Add to collection
Get this paper in your agent:
hf papers read 2607\.19058
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2607.19058 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2607.19058 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2607.19058 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
SkewAdam: A tiered optimizer that cuts MoE state memory by 97% (fits a 6.7B MoE on a 40GB GPU) [R]
SkewAdam is a tiered optimizer that reduces memory usage for MoE state by 97%, enabling a 6.7B MoE model to fit on a single 40GB GPU.
MESH: Memory-Efficient Sinkhorn Optimization for Mixture-of-Experts Training
This paper introduces MESH, a memory-efficient Sinkhorn-based optimizer for Mixture-of-Experts (MoE) training that restores temporal momentum without storing full optimizer state, reducing memory by 62.5% while maintaining competitive evaluation loss compared to AdamW.
Flattening Every Memory Peak in Long-Context Mixture-of-Experts Training
This paper introduces techniques to manage memory peaks in training large Mixture-of-Experts models with long context lengths, including Pipelined LLEP, Ring-DTP, SCO, and OffloadStreamAdamW, which enable fixed GPU working sets and improve throughput up to 10.4x.
Gefen: Optimized Stochastic Optimizer
Gefen is a memory-efficient optimizer that reduces AdamW's memory footprint by ~8x by automatically sharing second-moment estimates and quantizing the first moment using a learned codebook, while maintaining AdamW-level performance.
Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts
The paper proposes LOOM, a scalable recipe for Looped Mixture-of-Experts models that stabilizes recurrence via residual scaling and input re-injection while diversifying expert usage with per-loop routers, achieving stable scaling to 9-12 loops and improved perplexity and zero-shot accuracy on 100M-1.7B models.