We swapped AdamW's optimizer states for a Fast Fourier Transform (FFT) to cut VRAM in half. Anyone else trying non-quantization methods?

Reddit r/LocalLLaMA Papers
optimizer-memory fft vram-reduction fine-tuning llm compression open-collaboration

Summary

A team claims to halve optimizer-state VRAM by transforming AdamW gradient states into the frequency domain with an FFT, dropping low-energy frequencies instead of using quantization, trading extra compute per step for near-zero OOM risk. They're sharing their Colab environment and baseline weights and asking whether others are exploring non-quantization VRAM-reduction methods.

Hey everyone, Like most of you, we have been fighting constant OOM errors while trying to fine-tune 8B and 70B models on consumer GPUs. The AdamW optimizer states are always the biggest bottleneck. We didn't want to rely on aggressive 8-bit quantization because we were seeing degradation in convergence, so we tried an experiment: tackling the optimizer states in the frequency domain. The methodology: Instead of storing the full gradients, we transform them using an FFT. This isolates the high-energy signal from the noise. We dynamically drop the low-impact frequencies and compress the state. When we inverse-transform back, it maintains the directional integrity but uses roughly 50% less VRAM. The catch: Running FFT operations adds compute overhead. It takes slightly longer per step, but the trade-off is completely avoiding OOM crashes and pushing batch sizes way up on standard hardware. We are currently giving out access to our internal Colab environment and baseline weights to anyone who wants to poke holes in our math or try to break it. We are really curious if anyone else here is exploring frequency-domain stuff or other non-quantization methods for VRAM reduction?
Original Article

Similar Articles

I built a PyTorch optimizer for large GPU workloads — looking for feedback[P]

Reddit r/MachineLearning

A developer shares ApexW, an experimental PyTorch optimizer using compact optimizer states and a Triton-based execution path, claiming ~1.5x faster optimizer steps and halved optimizer-state memory on large-tensor workloads — while openly noting regressions on many-small-tensor models and minimal end-to-end gains.

Gefen: Optimized Stochastic Optimizer

arXiv cs.LG

Gefen is a memory-efficient optimizer that reduces AdamW's memory footprint by ~8x by automatically sharing second-moment estimates and quantizing the first moment using a learned codebook, while maintaining AdamW-level performance.

autotrust/GEV-26B-Decide-NVFP4

Hugging Face Models Trending

autotrust/GEV-26B-Decide-NVFP4 is an NVFP4 4-bit quantized release of GEV-26B-Decide that reduces weights from 49.5 GB to 18 GB (−66% GPU memory) by quantizing only the 3,840 routed expert MLPs, with benchmark results within run-to-run noise of the bf16 model and support for native W4A4 inference on Blackwell GPUs via vLLM.