Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning

arXiv cs.LG Papers
deep-learning optimization generalization sharpness-aware-minimization extragradient flat-minima

Summary

Proposes EISAM, a new optimizer that extends Sharpness-Aware Minimization using an extragradient step to find flatter minima, improving generalization and robustness while reducing sensitivity to hyperparameters. Outperforms SGD, Adam, and SAM on benchmarks.

arXiv:2607.06151v1 Announce Type: new Abstract: Generalization remains a pivotal challenge in deep learning, where traditional optimizers like Stochastic Gradient Descent (SGD) often converge to sharp minima, leading to overfitting and reduced performance on unseen data. Building on Sharpness-Aware Minimization (SAM), for seeking flat minima associated with improved generalization, we propose the Extragradient-Inspired Sharpness-Aware Minimization (EISAM), a novel optimizer that enhances generalization via the extragradient technique. EISAM uses a two-step update process: a prediction step investigating the geometry of the loss landscape and a perturbation step that refines updates with a base optimizer. This approach achieves better generalization performance than SAM. Crucially, EISAM reduces sensitivity to the perturbation radius, enhancing robustness, and simplifying the tuning across diverse settings. Extensive experiments on benchmark datasets demonstrate that EISAM consistently outperforms SGD, Adaptive Moment Estimation (Adam), and SAM in test accuracy and training efficiency across various architectures. Theoretical analysis further confirms that EISAM tightens the generalization bound by steering parameters toward flatter minima with reduced curvature. Accompanied by a thorough hyperparameter analysis, EISAM offers practical tuning guidance, establishing it as a robust, scalable, and broadly applicable optimization solution that advances both the theory and practice in deep learning.
Original Article
View Cached Full Text

Cached at: 07/08/26, 04:46 AM 

# Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning
Source: [https://arxiv.org/abs/2607.06151](https://arxiv.org/abs/2607.06151)
[View PDF](https://arxiv.org/pdf/2607.06151)

> Abstract:Generalization remains a pivotal challenge in deep learning, where traditional optimizers like Stochastic Gradient Descent \(SGD\) often converge to sharp minima, leading to overfitting and reduced performance on unseen data\. Building on Sharpness\-Aware Minimization \(SAM\), for seeking flat minima associated with improved generalization, we propose the Extragradient\-Inspired Sharpness\-Aware Minimization \(EISAM\), a novel optimizer that enhances generalization via the extragradient technique\. EISAM uses a two\-step update process: a prediction step investigating the geometry of the loss landscape and a perturbation step that refines updates with a base optimizer\. This approach achieves better generalization performance than SAM\. Crucially, EISAM reduces sensitivity to the perturbation radius, enhancing robustness, and simplifying the tuning across diverse settings\. Extensive experiments on benchmark datasets demonstrate that EISAM consistently outperforms SGD, Adaptive Moment Estimation \(Adam\), and SAM in test accuracy and training efficiency across various architectures\. Theoretical analysis further confirms that EISAM tightens the generalization bound by steering parameters toward flatter minima with reduced curvature\. Accompanied by a thorough hyperparameter analysis, EISAM offers practical tuning guidance, establishing it as a robust, scalable, and broadly applicable optimization solution that advances both the theory and practice in deep learning\.

## Submission history

From: Yao Fu \[[view email](https://arxiv.org/show-email/e9698c5f/2607.06151)\] **\[v1\]**Tue, 7 Jul 2026 11:25:46 UTC \(30,880 KB\)

Similar Articles

Closed-Form Steepest Descent Direction toward Flat Minima: Reducing Upper Bounds on the Loss Hessian Eigenspectrum in Neural Networks

arXiv cs.LG

Derives the closed-form gradient of the Wolkowicz-Styan upper bound on the loss Hessian eigenspectrum to guide neural network training toward flat minima, and introduces Hessian Spectral Range (HSR) Regularization. Numerical experiments show that HSR narrows the Hessian eigenvalue range, avoids sharp minima and saddle points, and achieves flat solutions comparable to Sharpness-Aware Minimization (SAM).

Gradient Smoothing: Coupling Layer-wise Updates for Improved Optimization

arXiv cs.LG

Introduces Depth-wise Gradient Augmentation, a general optimization paradigm that transforms block-wise optimizer updates along depth dimension. The method, Gradient Smoothing, improves optimization and generalization across diverse architectures including transformers and diffusion models.

Convergence of Steepest Descent and Adam under Non-Uniform Smoothness

arXiv cs.LG

This paper generalizes non-uniform smoothness assumptions to objectives whose curvature is affine in the objective value, proving convergence rates for steepest descent and diagonal variants of RMSProp and Adam, with applications to logistic regression and neural networks.

Certification of Machine Learning Models via Directional Sharpness

arXiv cs.LG

This paper introduces directional sharpness, a new metric for certifying the generalization performance of machine learning models that is both efficient to compute and more reliable than existing proxies like test accuracy or traditional sharpness, even when training deviates from prescribed procedures.

Edge of Stability Selectively Shapes Learning Across the Data Distribution

arXiv cs.LG

MIT researchers show that the edge of stability (EoS) in neural network training is not merely a global optimization phenomenon but selectively redistributes learning across subsets of the training distribution, amplifying progress on some data groups while suppressing others. They identify two key conditions governing this allocation: gradient alignment with the top Hessian eigenvector and sustained non-vanishing gradient magnitude.