Tag
SkewAdam is a tiered optimizer that reduces memory usage for MoE state by 97%, enabling a 6.7B MoE model to fit on a single 40GB GPU.