Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning
Summary
Proposes EISAM, a new optimizer that extends Sharpness-Aware Minimization using an extragradient step to find flatter minima, improving generalization and robustness while reducing sensitivity to hyperparameters. Outperforms SGD, Adam, and SAM on benchmarks.
View Cached Full Text
Cached at: 07/08/26, 04:46 AM
# Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning Source: [https://arxiv.org/abs/2607.06151](https://arxiv.org/abs/2607.06151) [View PDF](https://arxiv.org/pdf/2607.06151) > Abstract:Generalization remains a pivotal challenge in deep learning, where traditional optimizers like Stochastic Gradient Descent \(SGD\) often converge to sharp minima, leading to overfitting and reduced performance on unseen data\. Building on Sharpness\-Aware Minimization \(SAM\), for seeking flat minima associated with improved generalization, we propose the Extragradient\-Inspired Sharpness\-Aware Minimization \(EISAM\), a novel optimizer that enhances generalization via the extragradient technique\. EISAM uses a two\-step update process: a prediction step investigating the geometry of the loss landscape and a perturbation step that refines updates with a base optimizer\. This approach achieves better generalization performance than SAM\. Crucially, EISAM reduces sensitivity to the perturbation radius, enhancing robustness, and simplifying the tuning across diverse settings\. Extensive experiments on benchmark datasets demonstrate that EISAM consistently outperforms SGD, Adaptive Moment Estimation \(Adam\), and SAM in test accuracy and training efficiency across various architectures\. Theoretical analysis further confirms that EISAM tightens the generalization bound by steering parameters toward flatter minima with reduced curvature\. Accompanied by a thorough hyperparameter analysis, EISAM offers practical tuning guidance, establishing it as a robust, scalable, and broadly applicable optimization solution that advances both the theory and practice in deep learning\. ## Submission history From: Yao Fu \[[view email](https://arxiv.org/show-email/e9698c5f/2607.06151)\] **\[v1\]**Tue, 7 Jul 2026 11:25:46 UTC \(30,880 KB\)
Similar Articles
Gradient-Energy Guided Block-Wise Perturbations for Sharpness-Aware Minimization
The paper proposes GEAR-SAM, which adaptively allocates the perturbation budget across network blocks using an exponential moving average of squared gradients, improving generalization without additional computational overhead.
Closed-Form Steepest Descent Direction toward Flat Minima: Reducing Upper Bounds on the Loss Hessian Eigenspectrum in Neural Networks
Derives the closed-form gradient of the Wolkowicz-Styan upper bound on the loss Hessian eigenspectrum to guide neural network training toward flat minima, and introduces Hessian Spectral Range (HSR) Regularization. Numerical experiments show that HSR narrows the Hessian eigenvalue range, avoids sharp minima and saddle points, and achieves flat solutions comparable to Sharpness-Aware Minimization (SAM).
From Perturbation Correction to Geometry-Aware Sampling: Sharpness-Guided Equilibrium Sampling for Balanced Flat Minima in Long-Tailed Learning
Introduces Sharpness-Guided Equilibrium Sampling (SGS) that dynamically adjusts sampling probabilities using sharpness estimates to achieve balanced flat minima in long-tailed learning, achieving significant gains on CIFAR-100 LT and ImageNet-LT.
Mini-batch Noise Lowers Sharpness via Dominant-Subspace Fluctuations
This paper argues that the dominant subspace of the Hessian, while contributing little to loss reduction, plays a key role in reducing sharpness during mini-batch SGD. It derives a sharpness correction term induced by mini-batch noise in the dominant directions.
Gradient Smoothing: Coupling Layer-wise Updates for Improved Optimization
Introduces Depth-wise Gradient Augmentation, a general optimization paradigm that transforms block-wise optimizer updates along depth dimension. The method, Gradient Smoothing, improves optimization and generalization across diverse architectures including transformers and diffusion models.