SkewAdam: A tiered optimizer that cuts MoE state memory by 97% (fits a 6.7B MoE on a 40GB GPU) [R]

Reddit r/MachineLearning Papers

Summary

SkewAdam is a tiered optimizer that reduces memory usage for MoE state by 97%, enabling a 6.7B MoE model to fit on a single 40GB GPU.

No content available
Original Article

Similar Articles

Gefen: Optimized Stochastic Optimizer

arXiv cs.LG

Gefen is a memory-efficient optimizer that reduces AdamW's memory footprint by ~8x by automatically sharing second-moment estimates and quantizing the first moment using a learned codebook, while maintaining AdamW-level performance.

We swapped AdamW's optimizer states for a Fast Fourier Transform (FFT) to cut VRAM in half. Anyone else trying non-quantization methods?

Reddit r/LocalLLaMA

A team claims to halve optimizer-state VRAM by transforming AdamW gradient states into the frequency domain with an FFT, dropping low-energy frequencies instead of using quantization, trading extra compute per step for near-zero OOM risk. They're sharing their Colab environment and baseline weights and asking whether others are exploring non-quantization VRAM-reduction methods.