Gefen: Optimized Stochastic Optimizer

arXiv cs.LG Papers

Summary

Gefen is a memory-efficient optimizer that reduces AdamW's memory footprint by ~8x by automatically sharing second-moment estimates and quantizing the first moment using a learned codebook, while maintaining AdamW-level performance.

arXiv:2606.13894v1 Announce Type: new Abstract: AdamW is a default optimizer for modern deep learning, but its first and second moment states add roughly two parameter-sized buffers to training memory. We propose Gefen, a memory-efficient optimizer that automatically shares second-moment estimates across parameter blocks and quantizes the first moment using a learned codebook, thereby reducing AdamW's memory footprint by ~8x while maintaining the same performance, corresponding to a reduction of 6.5 GiB per billion parameters. The method is motivated by a theoretical result showing that large mixed Hessian entries constrain the ratio of squared gradients toward one, suggesting that Hessian-aligned parameters are natural candidates for sharing second-moment statistics. Since computing Hessians is impractical at scale, Gefen infers block structure from the initial squared gradients, requiring no architecture-specific metadata or hyperparameters beyond AdamW defaults. Gefen learns an exact histogram-based dynamic-programming quantization codebook and reuses the same blocks for first-moment scaling. Across diverse experiments, Gefen achieves the lowest peak optimizer memory among the compared AdamW-like methods while maintaining AdamW-level performance. In FSDP and DDP training, the reduced memory footprint enables larger microbatches and improves throughput significantly over AdamW, providing a practical drop-in replacement with lower memory usage that can increase throughput and enable training larger models or using larger batch sizes. We provide the complete Python implementation, including fused CUDA kernels at https://github.com/ndvbd/Gefen
Original Article
View Cached Full Text

Cached at: 06/15/26, 09:08 AM

# Gefen: Optimized Stochastic Optimizer
Source: [https://arxiv.org/html/2606.13894](https://arxiv.org/html/2606.13894)
Nadav Benedek Reichman University &Tomer Koren Tel Aviv University, Google Research &Ohad Fried Reichman University

###### Abstract

AdamW is a default optimizer for modern deep learning, but its first and second moment states add roughly two parameter\-sized buffers to training memory\. We propose Gefen, a memory\-efficient optimizer that automatically shares second\-moment estimates across parameter blocks and quantizes the first moment using a learned codebook, thereby reducing AdamW’s memory footprint by∼8×\\mathord\{\\sim\}8\\timeswhile maintaining the same performance, corresponding to a reduction of 6\.5 GiB per billion parameters\. The method is motivated by a theoretical result showing that large mixed Hessian entries constrain the ratio of squared gradients toward one, suggesting that Hessian\-aligned parameters are natural candidates for sharing second\-moment statistics\. Since computing Hessians is impractical at scale, Gefen infers block structure from the initial squared gradients, requiring no architecture\-specific metadata or hyperparameters beyond AdamW defaults\. Gefen learns an exact histogram\-based dynamic\-programming quantization codebook and reuses the same blocks for first\-moment scaling\. Across diverse experiments, Gefen achieves the lowest peak optimizer memory among the compared AdamW\-like methods while maintaining AdamW\-level performance\. In FSDP and DDP training, the reduced memory footprint enables larger microbatches and improves throughput significantly over AdamW, providing a practical drop\-in replacement with lower memory usage that can increase throughput and enable training larger models or using larger batch sizes\. We provide the complete Python implementation, including fused CUDA kernels at[https://github\.com/ndvbd/Gefen](https://github.com/ndvbd/Gefen)\.

## 1Introduction

Adam\(Kingma & Ba,[2014](https://arxiv.org/html/2606.13894#bib.bib15)\)and AdamW\(Loshchilov & Hutter,[2017](https://arxiv.org/html/2606.13894#bib.bib24)\)are widely used workhorse optimizers in deep learning due to their strong and consistent performance across a broad range of architectures and datasets\. However, because they store both first and second moment moving averages, their optimizer state alone adds roughly twice the parameter memory\.

Reducing optimizer memory can enable training larger models or using larger batch sizes, which can in turn improve training throughput and final model quality\. Therefore, we are motivated to design an optimizer that maintains AdamW\-like performance while reducing memory usage\.

For example, Adam\-mini\(Zhang et al\.,[2025](https://arxiv.org/html/2606.13894#bib.bib37)\)is a recent variant of Adam that reduces optimizer memory bysharingsecond\-moment values using manual rules that mimic the block diagonal structure of parameters’ Hessians, but a theoretical justification forwhyHessian\-aligned grouping is beneficial remains missing\. In practice, users may need to provide architecture\-specific information \(e\.g\., the number of attention heads\), and grouping rules depend on tensor names, which may vary across implementations\. Quantized optimizers\(Dettmers et al\.,[2021](https://arxiv.org/html/2606.13894#bib.bib8); Li et al\.,[2023](https://arxiv.org/html/2606.13894#bib.bib18)\)reduce memory, but rely on manually crafted, suboptimal codebooks and introduce an implicit hyperparameter through the quantization block size\.

Inspired by these methods, our goal is to remove manual grouping requirements, avoid Adam\-mini’s tensor\-name\-dependent decisions, provide stronger theoretical grounding for grouping parameters with high Hessian\-related affinity, validate on both LLMs and non\-LLM networks, and further reduce the memory footprint beyond Adam\-mini while maintaining general applicability and AdamW\-level performance\.

InLABEL:sec:desiderata, we describe the desiderata for an ideal optimizer, which we use to guide our design and evaluation\. In[Section3](https://arxiv.org/html/2606.13894#S3), we provide theoretical analysis showing that high Hessian affinity implies similar squared gradients, which justifies our grouping strategy\. In[Section4](https://arxiv.org/html/2606.13894#S4), we present our algorithm, which automatically groups parameters and shares second moments within these groups\. Empirically, our method achieves the lowest memory footprint among the compared methods across a variety of models and datasets, while maintaining comparable or improved quality and throughput, as shown in[Section5](https://arxiv.org/html/2606.13894#S5)\. Readers familiar with AdamW may wish to skip directly to[Section3](https://arxiv.org/html/2606.13894#S3)\.

## 2Preliminaries and Related Work

##### Adam and AdamW\.

Combining ideas from AdaGrad\(Duchi et al\.,[2011](https://arxiv.org/html/2606.13894#bib.bib9)\)and RMSProp\(Tieleman & Hinton,[2012](https://arxiv.org/html/2606.13894#bib.bib35)\), Adam is an adaptive first\-order optimizer that maintains exponential moving averages of the gradient and squared gradient\. At steptt, with gradientgt=∇θℒ​\(θt\)g\_\{t\}=\\nabla\_\{\\theta\}\\mathcal\{L\}\(\\theta\_\{t\}\), Adam computes

mt\\displaystyle m\_\{t\}=β1​mt−1\+\(1−β1\)​gt\\displaystyle=\\beta\_\{1\}m\_\{t\-1\}\+\(1\-\\beta\_\{1\}\)g\_\{t\}\(EMA of first moment; momentum term\)\(1\)vt\\displaystyle v\_\{t\}=β2​vt−1\+\(1−β2\)​gt2\\displaystyle=\\beta\_\{2\}v\_\{t\-1\}\+\(1\-\\beta\_\{2\}\)g\_\{t\}^\{2\}\(EMA of second raw moment\)\(2\)then applies bias correction, which is needed because both EMAs are initialized at zero and are therefore biased toward zero at early steps,

m^t\\displaystyle\\hat\{m\}\_\{t\}=mt1−β1t,v^t=vt1−β2t\\displaystyle=\\frac\{m\_\{t\}\}\{1\-\\beta\_\{1\}^\{t\}\},\\qquad\\hat\{v\}\_\{t\}=\\frac\{v\_\{t\}\}\{1\-\\beta\_\{2\}^\{t\}\}\(Bias correction\)\(3\)and updates parameters as

θt\+1=θt−η​m^tv^t\+ϵ\\displaystyle\\theta\_\{t\+1\}=\\theta\_\{t\}\-\\eta\\frac\{\\hat\{m\}\_\{t\}\}\{\\sqrt\{\\hat\{v\}\_\{t\}\}\+\\epsilon\}\(4\)AdamW uses the same adaptive moments, but applies weight decay directly in parameter space rather than injecting anL2L\_\{2\}penalty term into the gradient:

θt\+1=θt−η​m^tv^t\+ϵ−η​λ​θt\\displaystyle\\theta\_\{t\+1\}=\\theta\_\{t\}\-\\eta\\frac\{\\hat\{m\}\_\{t\}\}\{\\sqrt\{\\hat\{v\}\_\{t\}\}\+\\epsilon\}\-\\eta\\lambda\\theta\_\{t\}\(5\)This decoupling avoids interactions between adaptive gradient scaling and regularization strength, and typically yields better generalization and more stable hyperparameter tuning\. In modern deep learning practice, Adam and AdamW are the de facto standard optimizers\. They keep first and second moment states, typically in FP32, so optimizer\-state memory is roughly88bytes per parameter\. For example, for a 13B\-parameter model, Adam\-style optimizer states alone require about13×109×8≈9713\\times 10^\{9\}\\times 8\\approx 97GiB; this can make training more difficult on memory\-constrained GPUs and force smaller batch sizes, which may also reduce training throughput\.

##### Quantized optimizer states\.

Recent work reduces optimizer memory by quantizing Adam\-style moment states\. Adam8bit\(Dettmers et al\.,[2021](https://arxiv.org/html/2606.13894#bib.bib8)\)flattens tensors into fixed\-size blocks \(B=2048B=2048\), quantizes both first and second moments, and stores per\-block absolute\-max scaling \(recomputed each step\) together with 8\-bit values per element\. Its Dynamic Tree Quantization encodes normalized values in\[−1,1\]\[\-1,1\], providing higher precision near zero while still representing larger magnitudes\. However, this design has practical limitations: \(1\) the dynamic\-tree quantizer can be wasteful due to duplicated codebook entries, \(2\) in practice the method is not applicable to all tensor types \(e\.g\., embeddings should not be quantized\), so it is not fully plug\-and\-play as a universal drop\-in, and \(3\) the fixed user\-chosen block size introduces an additional hyperparameter burden\. Adam4bit\(Li et al\.,[2023](https://arxiv.org/html/2606.13894#bib.bib18)\)pushes state quantization to 4 bits with improved handling of outliers \(including finer\-grained and asymmetric treatments for moment statistics\), further reducing memory while preserving competitive accuracy in many settings\. The method uses linear quantization and includes a block\-size hyperparameter, which defaults to128128in the authors’ implementation\.

SeeLABEL:app:extended\_relatedfor an extended related work discussion, including additional optimizers and gradient communication compression methods\. In this work, we focus on general\-purpose optimizers that require no manual configuration, additional hyperparameters, or learning\-rate retuning, and that can be used as drop\-in replacements for AdamW across training regimes while achieving a significantly lower memory footprint\.

## 3Theoretical Analysis

AdamW maintains, for each parameter, an exponential moving average of its squared gradient\. We analyze conditions under which two weights with a large\-magnitude mixed Hessian entry also have similar squared\-gradient magnitudes, making them natural candidates for sharing their second\-moment estimates\. Specifically, for parameter entries indexed by\(k,l\)\(k,l\)and\(k′,l′\)\(k^\{\\prime\},l^\{\\prime\}\), we study when

\(∂L∂W​\[k,l\]\)2and\(∂L∂W​\[k′,l′\]\)2\\left\(\\frac\{\\partial L\}\{\\partial W\[k,l\]\}\\right\)^\{2\}\\quad\\text\{and\}\\quad\\left\(\\frac\{\\partial L\}\{\\partial W\[k^\{\\prime\},l^\{\\prime\}\]\}\\right\)^\{2\}are close\. Such a relationship motivates grouping parameter pairs with large Hessian\-based coupling and sharing their second\-moment estimates\.

###### Theorem 3\.1\(Large Hessian entries contract squared\-gradient ratios\)\.

The theorem implies that for feasible\|H\|≤A\|H\|\\leq A, increasing the mixed Hessian magnitude tightens the admissible range of the squared\-gradient ratio toward11\. The same form of contraction also holds for second\-layer weights that share the same output coordinate and for hidden\-bias terms, whereas output\-bias Hessian entries are either zero or constant\. The theoretical derivation above is presented for a simple two\-layer MLP inLABEL:app:theory\-proof\. However, the same behavior is also observed empirically in tensors from real network architectures, including attention tensors and convolutional layers, as shown in Appendix[A](https://arxiv.org/html/2606.13894#A1)\.

## 4Algorithm

Algorithm 1Gefen: our proposed algorithm for optimized stochastic optimization\. Default settings are identical to AdamW\. All operations on vectors are element\-wise\. The algorithm has a fused CUDA implementation to avoid unnecessary temporary memory allocations\.1:Require:

α,λ∈ℝ\\alpha,\\lambda\\in\\mathbb\{R\}: Stepsize and Weight decay,

β1,β2∈\[0,1\)\\beta\_\{1\},\\beta\_\{2\}\\in\[0,1\): Decay rates for moment estimates\.

f​\(θ\)f\(\\theta\): Stochastic objective function with parameters

θ\\theta\. Initialize

t←0t\\leftarrow 0,

𝑮0=∇θft​\(𝜽0\)\\bm\{G\}\_\{0\}=\\nabla\_\{\\theta\}f\_\{t\}\(\\bm\{\\theta\}\_\{0\}\)
2:foreach parameter\-tensor

θ\\thetarun[Automatic Block Partitioning](https://arxiv.org/html/2606.13894#alg2)using

𝑮0\\bm\{G\}\_\{0\}
3:Run[Exact DP Quantization Codebook Learning](https://arxiv.org/html/2606.13894#alg3)\.

4:foreach parameter\-tensor

θ\\thetado

5:Initialize quantized 1stmomentum codebook

𝒎¯0←𝟎\\bm\{\\bar\{m\}\}\_\{0\}\\leftarrow\\bm\{0\}per weight and

‖𝒎0‖∞\|\|\\bm\{m\}\_\{0\}\|\|\_\{\\infty\}per block

6:Initialize a single 2ndmoment scalar

𝒗¯0←0\\bar\{\\bm\{v\}\}\_\{0\}\\leftarrow 0per block

7:repeat

8:

t←t\+1t\\leftarrow t\+1
9:foreach parameter\-tensor

θ\\thetado

10:

𝒈t←∇θft​\(𝜽t−1\)\\bm\{g\}\_\{t\}\\leftarrow\\nabla\_\{\\theta\}f\_\{t\}\(\\bm\{\\theta\}\_\{t\-1\}\)⊳\\trianglerightGet gradients for the current parameter

11:

𝒎t←β1⋅dequantize​\(𝒎¯t−1,‖𝒎t−1‖∞\)\+\(1−β1\)⋅𝒈t\\bm\{m\}\_\{t\}\\leftarrow\\beta\_\{1\}\\cdot\\textsc\{dequantize\}\(\\bm\{\\bar\{m\}\}\_\{t\-1\},\|\|\\bm\{m\}\_\{t\-1\}\|\|\_\{\\infty\}\)\+\(1\-\\beta\_\{1\}\)\\cdot\\bm\{g\}\_\{t\}⊳\\trianglerightUpdate biased momentum estimate

12:

𝒎¯t,‖𝒎t‖∞←Quantize​\(𝒎t\)\\bm\{\\bar\{m\}\}\_\{t\},\|\|\\bm\{m\}\_\{t\}\|\|\_\{\\infty\}\\leftarrow\\textsc\{Quantize\}\(\\bm\{m\}\_\{t\}\)⊳\\trianglerightQuantize and recalculate absmax per block

13:

𝒎^t←𝒎t/\(1−β1t\)\\hat\{\\bm\{m\}\}\_\{t\}\\leftarrow\\bm\{m\}\_\{t\}/\(1\-\\beta\_\{1\}^\{t\}\)⊳\\trianglerightBias correction for momentum

14:

𝒗t←β2⋅𝒗t−1\+\(1−β2\)⋅m​e​a​n​\(𝒈t2\)\\bm\{v\}\_\{t\}\\leftarrow\\beta\_\{2\}\\cdot\\bm\{v\}\_\{t\-1\}\+\(1\-\\beta\_\{2\}\)\\cdot mean\(\\bm\{g\}^\{2\}\_\{t\}\)⊳\\trianglerightUpdate biased second moment using a mean per block

15:

𝒗^t←𝒗t/\(1−β2t\)\\hat\{\\bm\{\{v\}\}\}\_\{t\}\\leftarrow\\bm\{v\}\_\{t\}/\(1\-\\beta\_\{2\}^\{t\}\)⊳\\trianglerightBias correction of compact second moment estimate

16:

𝜽t←𝜽t−1−α⋅\(𝒎^t/\(𝒗^t\+ϵ\)\+λ​𝜽t−1\)\\bm\{\\theta\}\_\{t\}\\leftarrow\\bm\{\\theta\}\_\{t\-1\}\-\\alpha\\cdot\\left\(\\hat\{\\bm\{m\}\}\_\{t\}/\(\\sqrt\{\\hat\{\\bm\{v\}\}\_\{t\}\}\+\\epsilon\)\+\\lambda\\bm\{\\theta\}\_\{t\-1\}\\right\)⊳\\trianglerightCompact𝒗^t\\hat\{\\bm\{\{v\}\}\}\_\{t\}are broadcasted to full shape

17:untilstopping criterion is met

18:returnoptimized parameters

𝜽t\\bm\{\\theta\}\_\{t\}

Algorithm 2Automatic Block Partitioning1:Require:first\-step gradient

𝒈0\\bm\{g\}\_\{0\}for a parameter tensor,

n=\|𝒈0\|n=\|\\bm\{g\}\_\{0\}\|
2:Flatten

𝒈0\\bm\{g\}\_\{0\}and let

𝒫\\mathcal\{P\}be all proper divisors of

nn, sorted increasingly

3:

Eprev←∅E\_\{\\mathrm\{prev\}\}\\leftarrow\\emptyset,

Δ⋆←10−12\\Delta^\{\\star\}\\leftarrow 10^\{\-12\},

p⋆←1p^\{\\star\}\\leftarrow 1
4:foreach candidate period

p∈𝒫p\\in\\mathcal\{P\}do

5:Reshape

𝒈02\\bm\{g\}\_\{0\}^\{2\}into

𝑩∈ℝ\(n/p\)×p\\bm\{B\}\\in\\mathbb\{R\}^\{\(n/p\)\\times p\}
6:

E​\(p\)←meani⁡\(Var​\(𝑩i,:\)\)E\(p\)\\leftarrow\\sqrt\{\\operatorname\{mean\}\_\{i\}\\left\(\\mathrm\{Var\}\(\\bm\{B\}\_\{i,:\}\)\\right\)\}⊳\\triangleright[Equation6](https://arxiv.org/html/2606.13894#S4.E6)

7:if

Eprev≠∅E\_\{\\mathrm\{prev\}\}\\neq\\emptysetthen

8:

Δ←E​\(p\)−Eprev\\Delta\\leftarrow E\(p\)\-E\_\{\\mathrm\{prev\}\}
9:if

Δ<Δ⋆\\Delta<\\Delta^\{\\star\}then

10:

Δ⋆←Δ\\Delta^\{\\star\}\\leftarrow\\Delta,

p⋆←pp^\{\\star\}\\leftarrow p
11:

Eprev←E​\(p\)E\_\{\\mathrm\{prev\}\}\\leftarrow E\(p\)
12:return

p⋆p^\{\\star\}if

p⋆≥8p^\{\\star\}\\geq 8, otherwise

11

Algorithm 3Exact DP Quantization Codebook Learning1:Require:gradients

\{𝒈0\(θ\)\}\\\{\\bm\{g\}^\{\(\\theta\)\}\_\{0\}\\\}, automatic periods

\{pθ\}\\\{p\_\{\\theta\}\\\}, number of codebook entries

kk
2:Initialize histogram counts

𝒄∈ℝ16​k\\bm\{c\}\\in\\mathbb\{R\}^\{16k\}over

\[−1,1\]\[\-1,1\]
3:foreach parameter tensor

θ\\thetado

4:Reshape

𝒈0\(θ\)\\bm\{g\}^\{\(\\theta\)\}\_\{0\}into blocks

𝑩∈ℝ\(nθ/pθ\)×pθ\\bm\{B\}\\in\\mathbb\{R\}^\{\(n\_\{\\theta\}/p\_\{\\theta\}\)\\times p\_\{\\theta\}\}
5:Normalize each nonzero block by its maximum absolute value:

𝒁i,:←𝑩i,:/‖𝑩i,:‖∞\\bm\{Z\}\_\{i,:\}\\leftarrow\\bm\{B\}\_\{i,:\}/\\\|\\bm\{B\}\_\{i,:\}\\\|\_\{\\infty\}
6:Accumulate all entries of

𝒁\\bm\{Z\}into histogram counts

𝒄\\bm\{c\}
7:Let

\{\(mi,ci\)\}i=1b\\\{\(m\_\{i\},c\_\{i\}\)\\\}\_\{i=1\}^\{b\}be the nonempty histogram bin centers and counts, sorted by

mim\_\{i\}
8:Compute prefix sums of

cic\_\{i\},

ci​mic\_\{i\}m\_\{i\}, and

ci​mi2c\_\{i\}m\_\{i\}^\{2\}
9:Define interval squared error cost

C​\(bℓ,br,k′\)C\(b\_\{\\ell\},b\_\{r\},k^\{\\prime\}\)using center

−1\-1if

k′=1k^\{\\prime\}=1, center

11if

k′=kk^\{\\prime\}=k, and otherwise the weighted mean of bins

bℓ,…,brb\_\{\\ell\},\\ldots,b\_\{r\}as codebook center⊳\\trianglerightForce extreme codebook entries

10:

Dbr,k′:=D\_\{b\_\{r\},k^\{\\prime\}\}:=the cost for quantizing the first

brb\_\{r\}bins using the first

k′k^\{\\prime\}codepoints; initialize

D0,0←0D\_\{0,0\}\\leftarrow 0and all other entries to

∞\\infty
11:for

k′=1,…,kk^\{\\prime\}=1,\\ldots,kdo

12:for

br=k′,…,bb\_\{r\}=k^\{\\prime\},\\ldots,bdo

13:

Dbr,k′←minbℓ≤br⁡Dbℓ−1,k′−1\+C​\(bℓ,br,k′\)D\_\{b\_\{r\},k^\{\\prime\}\}\\leftarrow\\min\_\{b\_\{\\ell\}\\leq b\_\{r\}\}\\;D\_\{b\_\{\\ell\}\-1,k^\{\\prime\}\-1\}\+C\(b\_\{\\ell\},b\_\{r\},k^\{\\prime\}\)
14:Store the minimizing split point

15:Backtrack the stored splits to recover the

kkcodebook entries

16:returnthe sorted codebook

Computing the Hessian directly in large models is not tractable\. Therefore, we present[Algorithm1](https://arxiv.org/html/2606.13894#alg1), a method that partitions each parameter tensor into blocks whose squared gradients are similar, and shares the moving average of the second moment within each block in a fully automatic way, without relying on user\-specified configuration\. We further reduce the memory footprint of the first moment using alearnableandexactdynamic programming quantization, whichreusesthe same block partitioning and therefore does not introduce an additional scaling constant\.

### 4\.1Automatic Partitioning

As shown in[Figure1](https://arxiv.org/html/2606.13894#S4.F1)and byCollobert \([2004](https://arxiv.org/html/2606.13894#bib.bib5)\), the Hessians of some parameters exhibit block\-diagonal structure\. Splitting this parameter into 4 blocks, corresponding to the number of attention heads in this example, groups together parameters with stronger Hessian coupling, which, as suggested by the theorem and empirical evidence, leads to more similar squared gradients\. Splitting into 16 groups provides finer granularity, as shown in the figure\. However, computing the two\-dimensional Hessian of mixed gradients is not tractable in large models\. We therefore use the observed structure of the squared gradients: the same 16\-block partition produces regions with more similar squared\-gradient magnitudes\. This observation motivates our one\-dimensional squared\-gradient\-based algorithm for partitioning parameters into blocks, without computing the Hessian directly\.

![Refer to caption](https://arxiv.org/html/2606.13894v1/x1.png)Figure 1:Hessian of a toy transformer attention\-query parameter, showing block\-diagonal structure, and the corresponding squared gradients of the same parameter below\. Qualitatively, both exhibit a visible 16\-block structure\.For each parameter tensor, Gefen infers the sharing pattern from the first iteration gradient only\. Let𝒈0∈ℝn\\bm\{g\}\_\{0\}\\in\\mathbb\{R\}^\{n\}be theflattenedgradient of a parameter tensor at the first optimization step\. For every proper divisorppofnn, we reshape𝒈02\\bm\{g\}\_\{0\}^\{2\}into consecutive blocks of sizeppand measure their within\-block heterogeneity:

E​\(p\)=pn​∑i=1n/pVar​\(\[g0,\(i−1\)​p\+12,…,g0,i​p2\]\)\.E\(p\)=\\sqrt\{\\frac\{p\}\{n\}\\sum\_\{i=1\}^\{n/p\}\\mathrm\{Var\}\\\!\\left\(\\left\[g\_\{0,\(i\-1\)p\+1\}^\{2\},\\ldots,g\_\{0,ip\}^\{2\}\\right\]\\right\)\}\.\(6\)The number\-of\-divisors functionτ​\(n\)\\tau\(n\)satisfies the classical boundτ​\(n\)≤n\(ln⁡2\+o​\(1\)\)/ln⁡ln⁡n\\tau\(n\)\\leq n^\{\(\\ln 2\+o\(1\)\)/\\ln\\ln n\}, showing that the number of divisors grows subpolynomially innn\(Landau,[1909](https://arxiv.org/html/2606.13894#bib.bib17)\), so the number of candidate periods is low\. Small values ofE​\(p\)E\(p\)indicate that the squared gradients inside each candidate block are close, and therefore that replacing the element\-wise second moment by a single shared scalar is expected to introduce little distortion\. Ideally, we want largeppfor reduced memory and smallE​\(p\)E\(p\)for better optimization, but these two goals are in tension\.

One might expectE​\(p\)E\(p\)to increase asppgrows, since larger blocks impose more sharing\. Surprisingly, however, some larger divisors align better with the structure of the Hessian and therefore produce a sharp drop inE​\(p\)E\(p\)\. Instead of choosing the smallest value ofE​\(p\)E\(p\)directly, which would over\-favor tiny blocks, Gefen chooses the first pronounced improvement in this curve\.

We sort the candidate periods in increasing order and choose the period with the largest drop, falling back top=1p=1if no meaningful drop is found or if the selected period is smaller than88, since such small blocks provide little memory benefit\.

The selected period is fixed for the remainder of training\.[Algorithm2](https://arxiv.org/html/2606.13894#alg2)provides the pseudocode of the algorithm\. The period we found has dual use: it is used to share the second moment, and also to quantize the first moment, as described in the next section\.

### 4\.2Quantization

In our quest for a better optimizer, we seek to further reduce the memory footprint of the first moment through quantization\. Unlike Adam8bit and Adam4bit, which rely on fixed, hand\-designed quantization schemes, with a block\-size hyperparameter for storing the scale factor, and a quantization technique that is not guaranteed to be optimal, we design alearnableandexactdynamic programming quantization that automaticallyreusesthe block partitioning described earlier, thereby not introducing an additional constant for the quantization block size\.

#### 4\.2\.1Lloyd\-Max

Lloyd\-Max\(Lloyd,[1982](https://arxiv.org/html/2606.13894#bib.bib23); Max,[1960](https://arxiv.org/html/2606.13894#bib.bib27)\)is a classical algorithm for scalar quantization\. It iteratively alternates between assigning each value to the nearest codebook entry and updating each codebook entry to be the mean of the values assigned to it\. However, we identify two major problems with this approach\.

The first problem iscontraction\. Because Lloyd\-Max \(LM\) learns its codebook from the observed values, the two extreme codebook entries are not necessarily fixed at the endpoints of the normalized range\[−1,1\]\[\-1,1\]\. If we recompute the quantization scale at every optimizer step, this can create an undesirable feedback loop: after dequantization, the largest stored value is bounded by the largest learned codebook entry, which may be strictly smaller than11\. The next absmax estimate is then smaller as well, causing the effective quantization range to shrink over time\. To solve this problem, we force the codebook to always include the two extreme entries−1\-1and11, which guarantees that the quantization range is preserved and prevents contraction\.

However, another issue is that we found LM to be sensitive to the codebook initialization strategy: linear spacing or initialization by quantiles can lead to different results, with no consistent winner\. SeeLABEL:app:exact\-dp\-quantizationfor more details\.

#### 4\.2\.2Exact Quantization of Momentum Using Histogram\-Based Dynamic Programming

One dimensional Lloyd\-Max quantization is not guaranteed to converge to the optimal codebook solution, and is sensitive to the codebook initialization strategy \(LABEL:app:exact\-dp\-quantization\)\. We therefore seek to find an optimal solution\. The quantization problem in general is NP\-hard\(Aloise et al\.,[2009](https://arxiv.org/html/2606.13894#bib.bib1); Dasgupta & Freund,[2009](https://arxiv.org/html/2606.13894#bib.bib6); Mahajan et al\.,[2012](https://arxiv.org/html/2606.13894#bib.bib25)\), however, inspired byWang & Song \([2011](https://arxiv.org/html/2606.13894#bib.bib36)\)we design an optimal dynamic programming solution that operates on the histogram of the first step gradients and runs inO​\(b2​k\)O\(b^\{2\}k\), wherebbis the number of bins in the histogram andkkis the number of quantization levels\. Histogram calculation is performed in CUDA kernels, and dynamic programming is performed on the CPU using Numba\(Lam et al\.,[2015](https://arxiv.org/html/2606.13894#bib.bib16)\)allowing the codebook learning process, which takes place only once at the beginning of the training, to finish quickly\. See[Algorithms3](https://arxiv.org/html/2606.13894#alg3)andLABEL:app:dp\_detailsfor more details on the algorithm\. SeeLABEL:app:exact\-dp\-quantizationfor comparison of the exact method vs the Lloyd\-Max heuristic, and[Section6](https://arxiv.org/html/2606.13894#S6)for ablations\.

## 5Experiments

We evaluate Gefen across the desiderata presented above: peak memory footprint, performance, throughput, and distributability, using a variety of model architectures, model sizes, and datasets\. Both in theory and in implementation, Gefen reduces the peak and persistent memory footprint of the optimizer state by8×8\\times, while achieving performance similar to AdamW\. In the evaluated distributed\-training configurations, Gefen improves FSDP throughput by 56% compared to AdamW and enables the DDP setting where AdamW cannot fit even a microbatch size of 1\. Alternatively, one can use the reduced memory footprint to increase model size\. Implementation details and hyperparameters are found in[AppendixB](https://arxiv.org/html/2606.13894#A2)\.

### 5\.1Throughput and Distributability

![Refer to caption](https://arxiv.org/html/2606.13894v1/x2.png)

\(a\) Llama 3\-1\.5B with FSDP

![Refer to caption](https://arxiv.org/html/2606.13894v1/x3.png)

\(b\) GPT2\-1B

Figure 2:Training throughput in two distributed settings\. \(a\) With FSDP, Gefen increases throughput by 56%\. \(b\) For GPT2\-1B, Gefen enables training where AdamW is infeasible and improves throughput by 21% over Adam\-mini; similar behavior is observed in single\-machine training\.We next evaluate whether the reduced memory footprint translates into practical throughput gains and whether the method is compatible with distributed training schemes\. Since optimizer\-step time is only one component of the full training iteration, we measure end\-to\-end training throughput in tokens per second under fixed global batch size and sequence length\. In both distributed settings, the lower optimizer\-state memory of Gefen allows a larger microbatch to fit on each GPU, reducing gradient accumulation overhead and increasing throughput\. As shown in[Figure2](https://arxiv.org/html/2606.13894#S5.F2), this yields a 56% throughput improvement over other methods for Llama 3\-1\.5B with FSDP sharding on C4, by allowing the microbatch size per GPU to increase from 1 to 2, which is not possible with the other methods due to their larger memory footprint\. Adam4bit and Adam8bit implementations do not support FSDP sharding\(Liang et al\.,[2025](https://arxiv.org/html/2606.13894#bib.bib20)\)\. For the GPT2\-1B model, under both DDP on two 24GiB GPUs and single\-machine training, AdamW is infeasible even with a microbatch size of 1\. Adam\-mini supports a microbatch size of 1, whereas Gefen supports a microbatch size of 2, yielding a 21% throughput improvement over Adam\-mini while making this setting feasible relative to AdamW\.

## 6Ablation Study

## 7Conclusion

[Figure3](https://arxiv.org/html/2606.13894#S7.F3)summarizes the main results of this work\. Among the baselines that reduce AdamW’s peak memory footprint by2×2\\timesor more, Gefen satisfies the desiderata of a practical optimizer: it matches AdamW’s performance, requires no learning\-rate adjustment or additional hyperparameters, and achieves the lowest memory footprint \(8×8\\timesless than AdamW\) and the highest throughput\.

![Refer to caption](https://arxiv.org/html/2606.13894v1/x4.png)Figure 3:Left: qualitative comparison of optimizer desiderata\. Middle: peak optimizer memory relative to model size, where lower is better; Gefen uses0\.25×0\.25\\timesmodel size, an88%88\\%reduction compared with AdamW\. Right: end\-to\-end training throughput, where higher is better; the reduced optimizer memory of Gefen enables a larger microbatch, achieving a56%56\\%improvement over AdamW in the FSDP setting and enabling the DDP setting with a microbatch size of 2, where AdamW cannot fit even a microbatch size of 1\.
## References

- Aloise et al\. \(2009\)Daniel Aloise, Amit Deshpande, Pierre Hansen, and Preyas Popat\.Np\-hardness of euclidean sum\-of\-squares clustering\.*Machine learning*, 75\(2\):245–248, 2009\.
- Anil et al\. \(2019\)Rohan Anil, Vineet Gupta, Tomer Koren, and Yoram Singer\.Memory efficient adaptive optimization\.*Advances in Neural Information Processing Systems*, 32, 2019\.
- Björck & Bowie \(1971\)Åke Björck and Clary Bowie\.An iterative algorithm for computing the best estimate of an orthogonal matrix\.*SIAM Journal on Numerical Analysis*, 8\(2\):358–364, 1971\.
- Chen et al\. \(2023\)Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho\-Jui Hsieh, Yifeng Lu, et al\.Symbolic discovery of optimization algorithms\.*Advances in neural information processing systems*, 36:49205–49233, 2023\.
- Collobert \(2004\)Ronan Collobert\.Large scale machine learning\.2004\.
- Dasgupta & Freund \(2009\)Sanjoy Dasgupta and Yoav Freund\.Random projection trees for vector quantization\.*IEEE Transactions on Information Theory*, 55\(7\):3229–3242, 2009\.
- Deng et al\. \(2009\)Jia Deng, Wei Dong, Richard Socher, Li\-Jia Li, Kai Li, and Li Fei\-Fei\.Imagenet: A large\-scale hierarchical image database\.In*2009 IEEE conference on computer vision and pattern recognition*, pp\. 248–255\. IEEE, 2009\.
- Dettmers et al\. \(2021\)Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer\.8\-bit optimizers via block\-wise quantization\.*arXiv preprint arXiv:2110\.02861*, 2021\.
- Duchi et al\. \(2011\)John Duchi, Elad Hazan, and Yoram Singer\.Adaptive subgradient methods for online learning and stochastic optimization\.*Journal of machine learning research*, 12\(7\), 2011\.
- He et al\. \(2016\)Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun\.Deep residual learning for image recognition\.In*Proceedings of the IEEE conference on computer vision and pattern recognition*, pp\. 770–778, 2016\.
- Higham \(2008\)Nicholas J Higham\.*Functions of matrices: theory and computation*\.SIAM, 2008\.
- Huang et al\. \(2019\)Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al\.Gpipe: Efficient training of giant neural networks using pipeline parallelism\.*Advances in neural information processing systems*, 32, 2019\.
- Jiang et al\. \(2024\)Shuoran Jiang, Qingcai Chen, Youcheng Pan, Yang Xiang, Yukang Lin, Xiangping Wu, Chuanyi Liu, and Xiaobao Song\.Zo\-adamu optimizer: Adapting perturbation by the momentum and uncertainty in zeroth\-order optimization\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 38, pp\. 18363–18371, 2024\.
- Jordan \(2024\)Keller Jordan\.Muon: An optimizer for hidden layers in neural networks\.[https://kellerjordan\.github\.io/posts/muon/](https://kellerjordan.github.io/posts/muon/), 2024\.Blog post\.
- Kingma & Ba \(2014\)Diederik P Kingma and Jimmy Ba\.Adam: A method for stochastic optimization\.*arXiv preprint arXiv:1412\.6980*, 2014\.
- Lam et al\. \(2015\)Siu Kwan Lam, Antoine Pitrou, and Stanley Seibert\.Numba: A llvm\-based python jit compiler\.In*Proceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC*, pp\. 1–6, 2015\.
- Landau \(1909\)Edmund Landau\.*Handbuch der Lehre von der Verteilung der Primazahlen*, volume 1\.BG Teubner, 1909\.
- Li et al\. \(2023\)Bingrui Li, Jianfei Chen, and Jun Zhu\.Memory efficient optimizers with 4\-bit states\.*Advances in Neural Information Processing Systems*, 36:15136–15171, 2023\.
- Li et al\. \(2020\)Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al\.Pytorch distributed: Experiences on accelerating data parallel training\.*arXiv preprint arXiv:2006\.15704*, 2020\.
- Liang et al\. \(2025\)Wanchao Liang, Tianyu Liu, Less Wright, Will Constable, Andrew Gu, Chien\-Chin Huang, Iris Zhang, Wei Feng, Howard Huang, Junjie Wang, Sanket Purandare, Gokul Nadathur, and Stratos Idreos\.Torchtitan: One\-stop pytorch native solution for production ready LLM pretraining\.In*The Thirteenth International Conference on Learning Representations*, 2025\.URL[https://openreview\.net/forum?id=SFN6Wm7YBI](https://openreview.net/forum?id=SFN6Wm7YBI)\.
- Liu et al\. \(2024\)Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma\.Sophia: A scalable stochastic second\-order optimizer for language model pre\-training\.In*International Conference on Learning Representations*, volume 2024, pp\. 1621–1650, 2024\.
- Liu et al\. \(2026\)Yong Liu, Zirui Zhu, Chaoyu Gong, Minhao Cheng, Cho\-Jui Hsieh, and Yang You\.Sparse mezo: Less parameters for better performance in zeroth\-order llm fine\-tuning\.*Advances in Neural Information Processing Systems*, 38:21139–21169, 2026\.
- Lloyd \(1982\)Stuart Lloyd\.Least squares quantization in pcm\.*IEEE transactions on information theory*, 28\(2\):129–137, 1982\.
- Loshchilov & Hutter \(2017\)Ilya Loshchilov and Frank Hutter\.Decoupled weight decay regularization\.*arXiv preprint arXiv:1711\.05101*, 2017\.
- Mahajan et al\. \(2012\)Meena Mahajan, Prajakta Nimbhorkar, and Kasturi Varadarajan\.The planar k\-means problem is np\-hard\.*Theoretical computer science*, 442:13–21, 2012\.
- Malladi et al\. \(2023\)Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora\.Fine\-tuning language models with just forward passes\.*Advances in Neural Information Processing Systems*, 36:53038–53075, 2023\.
- Max \(1960\)Joel Max\.Quantizing for minimum distortion\.*IRE Transactions on Information Theory*, 6\(1\):7–12, 1960\.
- Raffel et al\. \(2020\)Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu\.Exploring the limits of transfer learning with a unified text\-to\-text transformer\.*Journal of machine learning research*, 21\(140\):1–67, 2020\.
- Rajbhandari et al\. \(2020\)Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He\.Zero: Memory optimizations toward training trillion parameter models\.In*SC20: international conference for high performance computing, networking, storage and analysis*, pp\. 1–16\. IEEE, 2020\.
- Reddi et al\. \(2019\)Sashank J Reddi, Satyen Kale, and Sanjiv Kumar\.On the convergence of adam and beyond\.*arXiv preprint arXiv:1904\.09237*, 2019\.
- Shazeer & Stern \(2018\)Noam Shazeer and Mitchell Stern\.Adafactor: Adaptive learning rates with sublinear memory cost\.In*International conference on machine learning*, pp\. 4596–4604\. PMLR, 2018\.
- Shoeybi et al\. \(2019\)Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro\.Megatron\-lm: Training multi\-billion parameter language models using model parallelism\.*arXiv preprint arXiv:1909\.08053*, 2019\.
- Spall \(2002\)James C Spall\.Multivariate stochastic approximation using a simultaneous perturbation gradient approximation\.*IEEE transactions on automatic control*, 37\(3\):332–341, 2002\.
- Tang et al\. \(2021\)Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, and Yuxiong He\.1\-bit adam: Communication efficient large\-scale training with adam’s convergence speed\.In*International Conference on Machine Learning*, pp\. 10118–10129\. PMLR, 2021\.
- Tieleman & Hinton \(2012\)Tijmen Tieleman and Geoffrey Hinton\.Lecture 6\.5—RMSProp: Divide the gradient by a running average of its recent magnitude\.COURSERA: Neural Networks for Machine Learning, 2012\.
- Wang & Song \(2011\)Haizhou Wang and Mingzhou Song\.Ckmeans\. 1d\. dp: optimal k\-means clustering in one dimension by dynamic programming\.*The R journal*, 3\(2\):29, 2011\.
- Zhang et al\. \(2025\)Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Diederik Durk Kingma, Yinyu Ye, Zhi\-Quan Luo, and Ruoyu Sun\.Adam\-mini: Use fewer learning rates to gain more\.In*International Conference on Learning Representations*, volume 2025, pp\. 28033–28063, 2025\.
- Zhao et al\. \(2023\)Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien\-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al\.Pytorch fsdp: experiences on scaling fully sharded data parallel\.*arXiv preprint arXiv:2304\.11277*, 2023\.
- Zhu et al\. \(2024\)Hanqing Zhu, Zhenyu Zhang, Wenyan Cong, Xi Liu, Sem Park, Vikas Chandra, Bo Long, David Z\. Pan, Zhangyang Wang, and Jinwon Lee\.Apollo: Sgd\-like memory, adamw\-level performance, 2024\.URL[https://arxiv\.org/abs/2412\.05270](https://arxiv.org/abs/2412.05270)\.

###### Contents

1. [1Introduction](https://arxiv.org/html/2606.13894#S1)
2. [2Preliminaries and Related Work](https://arxiv.org/html/2606.13894#S2)
3. [3Theoretical Analysis](https://arxiv.org/html/2606.13894#S3)
4. [4Algorithm](https://arxiv.org/html/2606.13894#S4)1. [4\.1Automatic Partitioning](https://arxiv.org/html/2606.13894#S4.SS1) 2. [4\.2Quantization](https://arxiv.org/html/2606.13894#S4.SS2)1. [4\.2\.1Lloyd\-Max](https://arxiv.org/html/2606.13894#S4.SS2.SSS1) 2. [4\.2\.2Exact Quantization of Momentum Using Histogram\-Based Dynamic Programming](https://arxiv.org/html/2606.13894#S4.SS2.SSS2)
5. [5Experiments](https://arxiv.org/html/2606.13894#S5)1. [5\.1Throughput and Distributability](https://arxiv.org/html/2606.13894#S5.SS1)
6. [6Ablation Study](https://arxiv.org/html/2606.13894#S6)
7. [7Conclusion](https://arxiv.org/html/2606.13894#S7)
8. [References](https://arxiv.org/html/2606.13894#bib)
9. [AEmpirical Evidence That High Hessian Implies Close Squared Gradients](https://arxiv.org/html/2606.13894#A1)
10. [BHyperparameters](https://arxiv.org/html/2606.13894#A2)

## Appendix AEmpirical Evidence That High Hessian Implies Close Squared Gradients

We trained small models and examined the relationship between the squared gradients of selected weight entries within several parameter tensors and the corresponding Hessian entries\. The models are intentionally small, since computing the Hessian for large models is difficult, but they all tell the same empirical story\. This matches the theory in[Theorem3\.1](https://arxiv.org/html/2606.13894#S3.Thmtheorem1): when the Hessian magnitude between two weights is high, the ratio between their squared gradients tends to be close to11\.

In[Figure5](https://arxiv.org/html/2606.13894#A1.F5), as the Hessian entry moves away from zero in either direction, the ratio of the squared gradients, shown on a log scale, approaches one, marked by the horizontal dashed line\. The same phenomenon appears both when the two weights share the same first dimension and when they do not\.

![Refer to caption](https://arxiv.org/html/2606.13894v1/x5.png)

\(a\)

![Refer to caption](https://arxiv.org/html/2606.13894v1/x6.png)

\(b\)

Figure 4:Squared\-gradient ratio on a log scale vs\. Hessian for GPT\-toy at the beginning of training: \(a\) first\-layer query matrix; \(b\) fourth\-layer value matrix\. As the Hessian entry moves away from zero, the squared\-gradient ratio approaches one, as predicted by the theorem\.
![Refer to caption](https://arxiv.org/html/2606.13894v1/x7.png)

\(a\)

![Refer to caption](https://arxiv.org/html/2606.13894v1/x8.png)

\(b\)

Figure 5:Squared\-gradient ratio on a log scale vs\. Hessian for CNN on MNIST at the beginning of training: \(a\)conv1\.weight; \(b\)conv1\.bias\. The same phenomenon appears in both the weight and bias parameters, as in the transformer model\.

Another example, this time for a convolutional architecture, is shown in[Figure5](https://arxiv.org/html/2606.13894#A1.F5)\. The same phenomenon appears there, even though the networks are very different\.

## Appendix BHyperparameters

##### Llama Pre\-training\.

ForLABEL:fig:gpt2\_and\_llama\_train\_curvewe pre\-trained Llama 3\-1\.5B using the TorchTitan codebase\(Liang et al\.,[2025](https://arxiv.org/html/2606.13894#bib.bib20)\)on the C4 dataset\(Raffel et al\.,[2020](https://arxiv.org/html/2606.13894#bib.bib28)\)\. We use weight decay coefficientλ=0\.0\\lambda=0\.0,ϵ=10−8\\epsilon=10^\{\-8\},β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.999\\beta\_\{2\}=0\.999, and learning rate=10−4=10^\{\-4\}\. For the learning\-rate schedule, we use warmup for 1% of the total training steps, followed by linear decay\. This matches the default scheduler used in the TorchTitan codebase\. We use sequence length 2048 and global batch size 256, with per\-GPU batch size 2 and 64 gradient accumulation steps on 2 GeForce RTX 3090 GPUs\.

##### GPT\-2 training on OpenWebText\.

We use the nanoGPT codebase111[nanoGPT repository](https://github.com/karpathy/nanoGPT/tree/master)to train GPT\-2\-125M on OpenWebText \(LABEL:fig:gpt2\_and\_llama\_train\_curve\)\. We use the recommended hyperparameters:seq\_len = 1024, batch size=480=480, weight decay coefficientλ=0\.1\\lambda=0\.1,ϵ=10−8\\epsilon=10^\{\-8\},β1=0\.9\\beta\_\{1\}=0\.9, andβ2=0\.95\\beta\_\{2\}=0\.95\. We use a cosine\-decay learning\-rate schedule with 2000 warmup iterations\. For GPT\-2\-small, we use the peak learning rate recommended byLiu et al\. \([2024](https://arxiv.org/html/2606.13894#bib.bib21)\), which is reported to be optimal based on grid search\. The chosen peak learning rate is6×10−46\\times 10^\{\-4\}\. The minimum learning rate is3×10−53\\times 10^\{\-5\}\. For SM3, this learning rate caused divergence, so we switched to a more stable learning rate of 0\.225\. For Adafactor and SM3, we added momentum with the same value for a fair comparison\. In general, SM3 and Adafactor use different learning rates than AdamW, which makes them less suitable as drop\-in replacements, but we report them for completeness\. More accurate learning rates for SM3 and Adafactor may bring these optimizers closer to AdamW\.

##### ResNet\.

ForLABEL:fig:resnet\_imagenet\_train\_curvewe use the official PyTorch implementation codebase222[PyTorch torchvision ResNet implementation](https://github.com/pytorch/vision/blob/main/torchvision/models/resnet.py)to train ResNet18\(He et al\.,[2016](https://arxiv.org/html/2606.13894#bib.bib10)\)on ImageNet\(Deng et al\.,[2009](https://arxiv.org/html/2606.13894#bib.bib7)\)\. We use a cosine\-decay learning\-rate schedule, 90 epochs,β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.999\\beta\_\{2\}=0\.999,ϵ=10−8\\epsilon=10^\{\-8\}, and weight decay10−410^\{\-4\}\. For ResNet18, we use batch size=256=256and peak learning rate=0\.005=0\.005\.

##### CNN for MNIST\.

We use the official PyTorch implementation codebase333[PyTorch MNIST example implementation](https://github.com/pytorch/examples/tree/main/mnist)to train a CNN on MNIST on an RTX PRO 6000 Blackwell Max\-Q GPU \(LABEL:tab:mnist\_validation\_loss\)\. The network consists of two convolutional layers with ReLU activations, followed by max pooling and dropout, and then two fully connected layers ending in a 10\-class digit classifier\. We use batch size=64=64, test batch size=1000=1000, and train for 4 epochs\. For all optimizers, we useβ1=0\.9\\beta\_\{1\}=0\.9,β2=0\.999\\beta\_\{2\}=0\.999, and learning rate=0\.001=0\.001\. We use aStepLRscheduler with step size11andγ=0\.7\\gamma=0\.7, decaying the learning rate after each epoch\. Standard deviations are computed over three consecutive seeds\.

##### Throughput\.

For FSDP in[Figure2](https://arxiv.org/html/2606.13894#S5.F2)we used Llama 3\-1\.5B on C4 with the default sequence length 2048, global batch size 256, and two RTX 3090 GPUs\. In this setting, AdamW and Adam\-mini support a microbatch size of 1, whereas Gefen supports a microbatch size of 2\. For DDP in[Figure2](https://arxiv.org/html/2606.13894#S5.F2)we used GPT2\-1B on OpenWebText with the default sequence length 1024, global batch size 120, and two RTX 3090 GPUs\. In this setting, AdamW cannot fit a microbatch size of 1, Adam\-mini supports a microbatch size of 1, and Gefen supports a microbatch size of 2\.

Similar Articles

We swapped AdamW's optimizer states for a Fast Fourier Transform (FFT) to cut VRAM in half. Anyone else trying non-quantization methods?

Reddit r/LocalLLaMA

A team claims to halve optimizer-state VRAM by transforming AdamW gradient states into the frequency domain with an FFT, dropping low-energy frequencies instead of using quantization, trading extra compute per step for near-zero OOM risk. They're sharing their Colab environment and baseline weights and asking whether others are exploring non-quantization VRAM-reduction methods.