SkewAdam: A tiered optimizer that cuts MoE state memory by 97% (fits a 6.7B MoE on a 40GB GPU) [R]
Summary
SkewAdam is a tiered optimizer that reduces memory usage for MoE state by 97%, enabling a 6.7B MoE model to fit on a single 40GB GPU.
Similar Articles
Where Should Optimizer State Live? Tiered State Allocation for Memory-Efficient Mixture-of-Experts Training
SkewAdam is a novel optimizer for mixture-of-experts models that tier allocates optimizer state across backbone, experts, and router, reducing memory footprint to 2.6% of AdamW while achieving better validation perplexity in controlled comparisons.
Auto-fit vs tuned MoE offload: 564 → 1330 pp tok/s, unchanged decode (Qwen3.6-35B-A3B Q6 / RTX 3090)
A developer benchmarks Qwen3.6-35B-A3B Q6 on an RTX 3090, showing that offloading eight MoE expert layers to CPU and increasing batch sizes improves prompt processing by 2.36× (564→1330 tok/s) with no decode speed regression, using evolutionary search to find the tuning config.
LRMoE: Demand-Side Load Re-Calibration for Efficient MoE Training on Memory-Constrained Nodes
LRMoE is a memory-aware MoE training system that jointly recalibrates backward demand and balances retained workload on memory-constrained GPU nodes, reducing cumulative MoE-layer training time by 21.2-44.8% on Qwen3-30B-A3B and GLM4.5-106B-A12B compared to SYMI.
Gefen: Optimized Stochastic Optimizer
Gefen is a memory-efficient optimizer that reduces AdamW's memory footprint by ~8x by automatically sharing second-moment estimates and quantizing the first moment using a learned codebook, while maintaining AdamW-level performance.
We swapped AdamW's optimizer states for a Fast Fourier Transform (FFT) to cut VRAM in half. Anyone else trying non-quantization methods?
A team claims to halve optimizer-state VRAM by transforming AdamW gradient states into the frequency domain with an FFT, dropping low-energy frequencies instead of using quantization, trading extra compute per step for near-zero OOM risk. They're sharing their Colab environment and baseline weights and asking whether others are exploring non-quantization VRAM-reduction methods.