Gradient-Energy Guided Block-Wise Perturbations for Sharpness-Aware Minimization

# Gradient-Energy Guided Block-Wise Perturbations for Sharpness-Aware Minimization
Source: [https://arxiv.org/html/2607.18306](https://arxiv.org/html/2607.18306)
Zhen Huang, Jiaxin Deng, and Junbiao PangZ\. Huang, J\. Deng, and J\. Pang are with the Faculty of Information Technology, Beijing University of Technology, Beijing, China \(e\-mail: huangzhen@emails\.bjut\.edu\.cn; dengjiaxin@emails\.bjut\.edu\.cn; junbiao\_pang@bjut\.edu\.cn\)\.Corresponding author: Junbiao Pang\.

###### Abstract

Sharpness\-Aware Minimization \(SAM\) improves generalization by minimizing the worst\-case loss in a local parameter neighborhood\. Standard SAM implicitly allocates its global perturbation budget across parameter blocks according to instantaneous minibatch gradient norms\. Such an allocation can be noisy and may not reflect the sensitivity that blocks accumulate throughout training\. We propose Gradient\-Energy Adaptive Radius SAM \(GEAR\-SAM\), which maintains an exponential moving average \(EMA\) of squared block gradients as a lightweight, curvature\-related sensitivity signal and allocates the fixed SAM budget through a closed\-form constrained optimization\. GEAR\-SAM preserves the global SAM radius, requires no Hessian\-vector products or explicit Fisher estimation, and adds only scalar state beyond SAM\. Experiments on image classification, transfer learning, noisy\-label learning, and partition studies demonstrate improved generalization and robustness across architectures and tasks\. More broadly, GEAR\-SAM provides a dynamic view of sharpness\-aware optimization: a fixed perturbation budget should be redistributed as the sensitivity of functional network blocks evolves during training\.

## IIntroduction

Deep neural networks often generalize better when optimization converges to flatter minima\[[13](https://arxiv.org/html/2607.18306#bib.bib3),[12](https://arxiv.org/html/2607.18306#bib.bib5),[1](https://arxiv.org/html/2607.18306#bib.bib7)\]\. Sharpness\-Aware Minimization \(SAM\) explicitly encourages such solutions by minimizing the worst\-case loss in a local neighborhood of the current parameters\[[6](https://arxiv.org/html/2607.18306#bib.bib8)\]\. Due to its simplicity and effectiveness, SAM has become a widely used training strategy and has inspired many follow\-up methods, including ASAM\[[17](https://arxiv.org/html/2607.18306#bib.bib12)\], GSAM\[[31](https://arxiv.org/html/2607.18306#bib.bib13)\], Fisher SAM\[[14](https://arxiv.org/html/2607.18306#bib.bib14)\], F\-SAM\[[19](https://arxiv.org/html/2607.18306#bib.bib17)\], and BSAM\[[4](https://arxiv.org/html/2607.18306#bib.bib18)\]\.

Despite its success, standard SAM makes its perturbation decision from the current stochastic gradient\. When the network is partitioned into functional blocks, this decision implicitly assigns a different portion of the global perturbation budget to each block, but the allocation is determined only by the current minibatch\. This is a dynamic issue: the optimization roles and perturbation sensitivities of network blocks can evolve throughout training, whereas an instantaneous gradient can be dominated by minibatch noise or transient updates\. Thus, a fixed global radius should not imply a fixed or purely instantaneous allocation across the network\.

To construct a stable dynamic allocation, the signal should reflect more than a transient gradient magnitude\. Directly using curvature information during training is difficult: Hessian matrices are infeasible to store for modern networks, and online Hessian\-vector or Hessian\-block estimation introduces additional computation\. Fisher information provides a natural curvature\-related alternative and has been used in geometry\-aware SAM methods\[[14](https://arxiv.org/html/2607.18306#bib.bib14)\]\. Nevertheless, explicit Fisher estimation is still costly\. Motivated by the connection between squared gradients, diagonal empirical Fisher information, and expected Hessian information under standard likelihood assumptions\[[16](https://arxiv.org/html/2607.18306#bib.bib42),[9](https://arxiv.org/html/2607.18306#bib.bib43)\], we use an online second\-moment statistic as a lightweight curvature\-related signal\.

In this work, we revisit SAM as a dynamic block\-wise budget\-allocation method\. We show that standard SAM induces a different effective radius for each block, but this implicit allocation depends only on instantaneous minibatch gradient norms\. GEAR\-SAM instead uses an exponential moving average \(EMA\) of squared block gradients as a persistent sensitivity score and allocates the fixed SAM perturbation budget according to this historical signal\. The resulting perturbation preserves the global SAM budget while assigning larger radii to blocks that have remained sensitive over the recent optimization history\.

Our contributions are summarized as follows:

- •We analyze SAM from a second\-order and dynamic block\-wise perspective, showing that its practical perturbation is first\-order optimal but ignores curvature\-related block sensitivity and temporal persistence\.
- •We propose GEAR\-SAM, which uses EMA\-based historical gradient energy as a persistent Fisher\-related second\-moment signal and derives a closed\-form block\-wise radius allocation under a fixed SAM budget\.
- •We conduct experiments on image classification, transfer learning, label\-noise robustness, and partition and parameter studies, demonstrating the effectiveness and stability of the proposed allocation strategy\.

## IIRelated Work

### II\-ASharpness\-Aware Minimization

SAM improves generalization by optimizing the loss at adversarially perturbed parameters within a local neighborhood\[[6](https://arxiv.org/html/2607.18306#bib.bib8)\]\. The method is simple and effective, but its practical perturbation is computed using a first\-order approximation of the inner maximization\. A large body of work has therefore studied how to improve SAM from different perspectives\. ASAM addresses the scale dependency of sharpness\[[17](https://arxiv.org/html/2607.18306#bib.bib12)\]\. GSAM introduces a surrogate gap to better guide flat\-minima search\[[31](https://arxiv.org/html/2607.18306#bib.bib13)\]\. F\-SAM analyzes the role of stochastic\-gradient components in SAM perturbations\[[19](https://arxiv.org/html/2607.18306#bib.bib17)\]\. BSAM introduces bilateral sharpness by considering both max\-sharpness and min\-sharpness\[[4](https://arxiv.org/html/2607.18306#bib.bib18)\]\. Other methods improve the efficiency of SAM through stochastic perturbation, perturbation reuse, sample selection, or parallelization\[[5](https://arxiv.org/html/2607.18306#bib.bib15),[20](https://arxiv.org/html/2607.18306#bib.bib16),[11](https://arxiv.org/html/2607.18306#bib.bib31),[25](https://arxiv.org/html/2607.18306#bib.bib32),[27](https://arxiv.org/html/2607.18306#bib.bib33)\]\. These methods mainly modify the sharpness objective, perturbation direction, or computational procedure\. In contrast, GEAR\-SAM focuses on how a fixed perturbation budget is allocated across parameter blocks\.

### II\-BStructure\-aware SAM

Although SAM is usually formulated over the full parameter vector, recent work shows that its behavior depends strongly on network structure\. Perturbing normalization layers can retain much of the benefit of full\-parameter SAM\[[24](https://arxiv.org/html/2607.18306#bib.bib19)\], and the role of normalization has been studied theoretically and empirically\[[3](https://arxiv.org/html/2607.18306#bib.bib20)\]\. Effective SAM updates may require layer\-wise perturbation scaling\[[7](https://arxiv.org/html/2607.18306#bib.bib21)\], while sparse or selective perturbations can further improve SAM\[[22](https://arxiv.org/html/2607.18306#bib.bib22),[30](https://arxiv.org/html/2607.18306#bib.bib23),[18](https://arxiv.org/html/2607.18306#bib.bib24),[2](https://arxiv.org/html/2607.18306#bib.bib41)\]\. These findings indicate that treating the network as a single undifferentiated parameter vector can overlook useful structure\.

These studies motivate structure\-aware perturbation design\. Most existing methods either select a subset of parameters or modify scaling at the layer level\. GEAR\-SAM instead partitions the network into functional blocks and continuously allocates a fixed global perturbation budget among them according to historical block sensitivity\. It therefore changes neither the perturbation geometry nor the set of perturbed parameters\.

### II\-CCurvature Proxies for Hessian

Sharpness\-aware training is closely related to local curvature\. Hessian\-based quantities, such as the largest eigenvalue or trace, are widely used to characterize flatness and generalization\[[13](https://arxiv.org/html/2607.18306#bib.bib3),[12](https://arxiv.org/html/2607.18306#bib.bib5),[10](https://arxiv.org/html/2607.18306#bib.bib6),[1](https://arxiv.org/html/2607.18306#bib.bib7)\]\. Recent studies also connect SAM with stability, first\-order flatness, and explicit curvature regularization\[[21](https://arxiv.org/html/2607.18306#bib.bib10),[29](https://arxiv.org/html/2607.18306#bib.bib27),[26](https://arxiv.org/html/2607.18306#bib.bib30)\]\. However, explicit Hessian computation is expensive for modern neural networks\.

Fisher information provides a practical curvature\-related alternative\. Fisher SAM uses Fisher geometry to define SAM neighborhoods\[[14](https://arxiv.org/html/2607.18306#bib.bib14)\], while Fisher Mask SAM uses Fisher information to estimate parameter importance\[[30](https://arxiv.org/html/2607.18306#bib.bib23)\]\. Nevertheless, explicit Fisher estimation is still costly, and empirical Fisher should not be treated as an exact Hessian estimator\[[16](https://arxiv.org/html/2607.18306#bib.bib42)\]\. Adam\-style squared\-gradient statistics provide an efficient online second\-moment signal\[[15](https://arxiv.org/html/2607.18306#bib.bib25)\]and have been connected to diagonal empirical Fisher information\[[9](https://arxiv.org/html/2607.18306#bib.bib43)\]\. Motivated by this connection, GEAR\-SAM uses block\-wise historical gradient energy as a lightweight curvature\-related proxy for adaptive radius allocation\.

## IIIMethod

### III\-AMotivation

SAM solves the following local min\-max problem:

min𝐰⁡max‖ϵ‖2≤ρ⁡L​\(𝐰\+ϵ\)\.\\min\_\{\\mathbf\{w\}\}\\max\_\{\\\|\\boldsymbol\{\\epsilon\}\\\|\_\{2\}\\leq\\rho\}L\(\\mathbf\{w\}\+\\boldsymbol\{\\epsilon\}\)\.\(1\)At iterationtt, let

𝐠t=∇𝐰Lℬt​\(𝐰t\),𝐇t=∇𝐰2Lℬt​\(𝐰t\)\.\\mathbf\{g\}\_\{t\}=\\nabla\_\{\\mathbf\{w\}\}L\_\{\\mathcal\{B\}\_\{t\}\}\(\\mathbf\{w\}\_\{t\}\),\\quad\\mathbf\{H\}\_\{t\}=\\nabla\_\{\\mathbf\{w\}\}^\{2\}L\_\{\\mathcal\{B\}\_\{t\}\}\(\\mathbf\{w\}\_\{t\}\)\.\(2\)The practical perturbation used by SAM is derived from the first\-order approximation

Lℬt​\(𝐰t\+ϵ\)≈Lℬt​\(𝐰t\)\+𝐠t⊤​ϵ\.L\_\{\\mathcal\{B\}\_\{t\}\}\(\\mathbf\{w\}\_\{t\}\+\\boldsymbol\{\\epsilon\}\)\\approx L\_\{\\mathcal\{B\}\_\{t\}\}\(\\mathbf\{w\}\_\{t\}\)\+\\mathbf\{g\}\_\{t\}^\{\\top\}\\boldsymbol\{\\epsilon\}\.\(3\)Solving the corresponding inner maximization gives

ϵtSAM=ρ​𝐠t‖𝐠t‖2\.\\boldsymbol\{\\epsilon\}^\{\\mathrm\{SAM\}\}\_\{t\}=\\rho\\frac\{\\mathbf\{g\}\_\{t\}\}\{\\\|\\mathbf\{g\}\_\{t\}\\\|\_\{2\}\}\.\(4\)for𝐠t≠𝟎\\mathbf\{g\}\_\{t\}\\neq\\mathbf\{0\}; if the full gradient is zero, we define the first\-order perturbation as zero\. Thus, Eq\. \([4](https://arxiv.org/html/2607.18306#S3.E4)\) is optimal for the linearized sharpness objective\. This follows directly from the Cauchy–Schwarz inequality:𝐠t⊤​ϵ≤‖𝐠t‖2​‖ϵ‖2≤ρ​‖𝐠t‖2\\mathbf\{g\}\_\{t\}^\{\\top\}\\boldsymbol\{\\epsilon\}\\leq\\\|\\mathbf\{g\}\_\{t\}\\\|\_\{2\}\\\|\\boldsymbol\{\\epsilon\}\\\|\_\{2\}\\leq\\rho\\\|\\mathbf\{g\}\_\{t\}\\\|\_\{2\}, with equality whenϵ\\boldsymbol\{\\epsilon\}is parallel to𝐠t\\mathbf\{g\}\_\{t\}\.

However, the local sharpness objective also contains curvature\. A second\-order approximation gives

Lℬt​\(𝐰t\+ϵ\)≈Lℬt​\(𝐰t\)\+𝐠t⊤​ϵ\+12​ϵ⊤​𝐇t​ϵ⏟Qt​\(ϵ\)\.L\_\{\\mathcal\{B\}\_\{t\}\}\(\\mathbf\{w\}\_\{t\}\+\\boldsymbol\{\\epsilon\}\)\\approx L\_\{\\mathcal\{B\}\_\{t\}\}\(\\mathbf\{w\}\_\{t\}\)\+\\mathbf\{g\}\_\{t\}^\{\\top\}\\boldsymbol\{\\epsilon\}\+\\underbrace\{\\frac\{1\}\{2\}\\boldsymbol\{\\epsilon\}^\{\\top\}\\mathbf\{H\}\_\{t\}\\boldsymbol\{\\epsilon\}\}\_\{Q\_\{t\}\(\\boldsymbol\{\\epsilon\}\)\}\.\(5\)LetQt​\(ϵ\)Q\_\{t\}\(\\boldsymbol\{\\epsilon\}\)denote the quadratic terms in Eq\. \([5](https://arxiv.org/html/2607.18306#S3.E5)\)\. The maximization ofQtQ\_\{t\}over the norm ball is a classical trust\-region problem\[[23](https://arxiv.org/html/2607.18306#bib.bib45)\]\. When the associated optimality system is nonsingular, its solution has the form

ϵt⋆=\(2​λt​𝐈−𝐇t\)−1​𝐠t,\\boldsymbol\{\\epsilon\}\_\{t\}^\{\\star\}=\(2\\lambda\_\{t\}\\mathbf\{I\}\-\\mathbf\{H\}\_\{t\}\)^\{\-1\}\\mathbf\{g\}\_\{t\},\(6\)whereλt\\lambda\_\{t\}is chosen to satisfy the trust\-region conditions\. The complete conditions, including the singular hard case, are given in the appendix\. Equation \([6](https://arxiv.org/html/2607.18306#S3.E6)\) shows that the desirable perturbation is shaped by both gradient𝐠t\\mathbf\{g\}\_\{t\}and curvature𝐇t\\mathbf\{H\}\_\{t\}\. Thus, SAM is first\-order optimal, but it is generally not optimal for the second\-order sharpness objective\.

### III\-BBlock\-Wise View of SAM

Computing the Hessian information in Eq\. \([6](https://arxiv.org/html/2607.18306#S3.E6)\) online is costly\. We therefore study how to allocate the fixed SAM radius across parameter blocks while preserving the current gradient direction within each block\. Let the parameters be divided intoBBdisjoint blocks:

𝐰=\(𝐰1,𝐰2,…,𝐰B\),\\mathbf\{w\}=\(\\mathbf\{w\}\_\{1\},\\mathbf\{w\}\_\{2\},\\ldots,\\mathbf\{w\}\_\{B\}\),\(7\)with the corresponding gradient decomposition

𝐠t=\(𝐠1,t,𝐠2,t,…,𝐠B,t\)\.\\mathbf\{g\}\_\{t\}=\(\\mathbf\{g\}\_\{1,t\},\\mathbf\{g\}\_\{2,t\},\\ldots,\\mathbf\{g\}\_\{B,t\}\)\.\(8\)Architecture\-aware block partition\.We use a coarse, architecture\-aware partition rather than treating every parameter tensor as an independent group\. A block is a self\-contained architecture\-level computation unit, or a contiguous group of such units that operates at the same representation scale\. Thus, a coarse block can contain multiple residual units; the input stem and final classifier are separate blocks\. Every trainable parameter belongs to exactly one block\.

All trainable affine normalization parameters belonging to a computation unit are assigned to the same block as its weight layers; non\-trainable normalization buffers are not perturbed\. This treatment is important because normalization parameters are not interchangeable with arbitrary small parameter subsets\. In particular, perturbing only affine normalization parameters can retain, and sometimes exceed, the benefit of full\-parameter SAM\[[24](https://arxiv.org/html/2607.18306#bib.bib19)\]\. Assigning these parameters to a separate radius would split the scale\-modulating normalization operation from the weights whose activations it normalizes\. We therefore use the aggregate gradient energy of the complete functional block as its allocation signal\. This is an architecture\-aware design principle, not a claim that a single partition granularity is universally optimal; alternative block granularities are evaluated in Table[VI](https://arxiv.org/html/2607.18306#Sx1.T6)\.

From Eq\. \([4](https://arxiv.org/html/2607.18306#S3.E4)\), the perturbation applied to blockbbis

ϵb,tSAM=ρ​𝐠b,t∑j=1B‖𝐠j,t‖22\.\\boldsymbol\{\\epsilon\}^\{\\mathrm\{SAM\}\}\_\{b,t\}=\\rho\\frac\{\\mathbf\{g\}\_\{b,t\}\}\{\\sqrt\{\\sum\_\{j=1\}^\{B\}\\\|\\mathbf\{g\}\_\{j,t\}\\\|\_\{2\}^\{2\}\}\}\.\(9\)This can be written as a block\-wise radius times a normalized block direction:

ϵb,tSAM=rb,tSAM​𝐠b,t‖𝐠b,t‖2,rb,tSAM=ρ​‖𝐠b,t‖2∑j=1B‖𝐠j,t‖22\.\\boldsymbol\{\\epsilon\}^\{\\mathrm\{SAM\}\}\_\{b,t\}=r^\{\\mathrm\{SAM\}\}\_\{b,t\}\\frac\{\\mathbf\{g\}\_\{b,t\}\}\{\\\|\\mathbf\{g\}\_\{b,t\}\\\|\_\{2\}\},\\quad r^\{\\mathrm\{SAM\}\}\_\{b,t\}=\\rho\\frac\{\\\|\\mathbf\{g\}\_\{b,t\}\\\|\_\{2\}\}\{\\sqrt\{\\sum\_\{j=1\}^\{B\}\\\|\\mathbf\{g\}\_\{j,t\}\\\|\_\{2\}^\{2\}\}\}\.\(10\)Equation \([10](https://arxiv.org/html/2607.18306#S3.E10)\) reveals that SAM already performs block\-wise radius allocation\. Its allocation score is the instantaneous minibatch gradient norm‖𝐠b,t‖2\\\|\\mathbf\{g\}\_\{b,t\}\\\|\_\{2\}\. This allocation is consistent with the first\-order inner maximization, but it does not encode the curvature term in Eq\. \([5](https://arxiv.org/html/2607.18306#S3.E5)\)\.

Observation 1 \(Implicit SAM allocation\)\.For any partition of parameters into blocks and any nonzero full gradient, the SAM perturbation in Eq\. \([4](https://arxiv.org/html/2607.18306#S3.E4)\) is equivalent to a block\-wise perturbation whose block radius is given by Eq\. \([10](https://arxiv.org/html/2607.18306#S3.E10)\)\. Moreover, these block radii preserve the global SAM budget, i\.e\.,∑b=1B\(rb,tSAM\)2=ρ2\\sum\_\{b=1\}^\{B\}\(r^\{\\mathrm\{SAM\}\}\_\{b,t\}\)^\{2\}=\\rho^\{2\}\. The derivation is given in the appendix\. This observation makes explicit that SAM is already an implicit allocation method, but the allocation signal is only the instantaneous gradient norm\. If the full gradient is zero, SAM uses zero perturbation under our convention\. When only a block gradient is zero, its SAM radius and perturbation are defined as zero\.

Figure[1](https://arxiv.org/html/2607.18306#S3.F1)illustrates this dynamic allocation on ResNet\-18 during the first 40 epochs, when the allocation changes most rapidly\. Although both methods use the same global perturbation budget at every iteration, GEAR\-SAM more quickly reduces the share assigned to the classifier and assigns larger shares to Layer3 and Layer4\. The complete trajectories are reported in the appendix\. This early redistribution motivates an allocation signal that summarizes recent optimization history rather than reacting only to the current minibatch\.

![Refer to caption](https://arxiv.org/html/2607.18306v1/x1.png)Figure 1:Early block\-wise perturbation dynamics of SAM and GEAR\-SAM on ResNet\-18 trained on CIFAR\-100\. The plots show the first 40 epochs for the classifier and the two deeper feature blocks\. Each panel reports the normalized perturbation\-budget shareρb,t2/ρ2\\rho\_\{b,t\}^\{2\}/\\rho^\{2\}at epoch 40\.To see the second\-order information missing from the instantaneous allocation, write the block perturbation as

ϵb,t=rb,t​𝐮b,t,𝐮b,t=𝐠b,t‖𝐠b,t‖2\.\\boldsymbol\{\\epsilon\}\_\{b,t\}=r\_\{b,t\}\\mathbf\{u\}\_\{b,t\},\\quad\\mathbf\{u\}\_\{b,t\}=\\frac\{\\mathbf\{g\}\_\{b,t\}\}\{\\\|\\mathbf\{g\}\_\{b,t\}\\\|\_\{2\}\}\.\(11\)Here𝐮b,t\\mathbf\{u\}\_\{b,t\}is defined for blocks with nonzero gradient; zero\-gradient blocks contribute no first\-order direction under our convention\. LetΔ​Lt=Lℬt​\(𝐰t\+ϵt\)−Lℬt​\(𝐰t\)\\Delta L\_\{t\}=L\_\{\\mathcal\{B\}\_\{t\}\}\(\\mathbf\{w\}\_\{t\}\+\\boldsymbol\{\\epsilon\}\_\{t\}\)\-L\_\{\\mathcal\{B\}\_\{t\}\}\(\\mathbf\{w\}\_\{t\}\)denote the perturbed minibatch\-loss increase\. Applying a second\-order Taylor expansion and, for interpretation, omitting cross\-block Hessian terms gives

Δ​Lt\\displaystyle\\Delta L\_\{t\}≈∑b=1Brb,t​‖𝐠b,t‖2\+12​∑b=1Brb,t2​𝐮b,t⊤​𝐇b​b,t​𝐮b,t\.\\displaystyle\\approx\\sum\_\{b=1\}^\{B\}r\_\{b,t\}\\\|\\mathbf\{g\}\_\{b,t\}\\\|\_\{2\}\+\\frac\{1\}\{2\}\\sum\_\{b=1\}^\{B\}r\_\{b,t\}^\{2\}\\mathbf\{u\}\_\{b,t\}^\{\\top\}\\mathbf\{H\}\_\{bb,t\}\\mathbf\{u\}\_\{b,t\}\.\(12\)Therefore, an informative block\-wise allocation should consider not only the current gradient magnitude but also persistent block sensitivity related to curvature\.

### III\-CGradient Energy as a Curvature\-related Proxy

The ideal block sensitivity in Eq\. \([12](https://arxiv.org/html/2607.18306#S3.E12)\) would involve Hessian\-block information such as𝐮b,t⊤​𝐇b​b,t​𝐮b,t\\mathbf\{u\}\_\{b,t\}^\{\\top\}\\mathbf\{H\}\_\{bb,t\}\\mathbf\{u\}\_\{b,t\}, trace, or spectral quantities\. These quantities are too expensive to estimate at every iteration\. We instead use squared\-gradient statistics as a lightweight online proxy\.

To distinguish per\-example and minibatch quantities, let𝐳t​\(x\)=∇𝐰ℓ​\(𝐰t;x\)\\mathbf\{z\}\_\{t\}\(x\)=\\nabla\_\{\\mathbf\{w\}\}\\ell\(\\mathbf\{w\}\_\{t\};x\)denote the gradient of one examplexx\. Define its mean and second\-moment matrix,

𝝁t=𝔼x​\[𝐳t​\(x\)\],𝐂t=𝔼x​\[𝐳t​\(x\)​𝐳t​\(x\)⊤\]\.\\boldsymbol\{\\mu\}\_\{t\}=\\mathbb\{E\}\_\{x\}\[\\mathbf\{z\}\_\{t\}\(x\)\],\\qquad\\mathbf\{C\}\_\{t\}=\\mathbb\{E\}\_\{x\}\[\\mathbf\{z\}\_\{t\}\(x\)\\mathbf\{z\}\_\{t\}\(x\)^\{\\top\}\]\.\(13\)For a negative log\-likelihood,𝐂t\\mathbf\{C\}\_\{t\}equals the Fisher information when the expectation is taken under the model distribution\. Under the training distribution, it is an empirical\-Fisher\-related gradient second moment and should not be identified with the Hessian in general\[[16](https://arxiv.org/html/2607.18306#bib.bib42)\]\. Under standard likelihood regularity and model\-matching assumptions, however, the Fisher equals the expected Hessian\. We use only this curvature\-related connection, not an exact Hessian equivalence\.

Assume that the minibatch loss is an average overmmindependently sampled examples\. For blockbb,

𝐠b,t=1m​∑i=1m𝐳b,t​\(xi\)\.\\mathbf\{g\}\_\{b,t\}=\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}\\mathbf\{z\}\_\{b,t\}\(x\_\{i\}\)\.\(14\)Conditioned on𝐰t\\mathbf\{w\}\_\{t\}, its squared norm satisfies the exact identity

𝔼ℬt​‖𝐠b,t‖22=\\displaystyle\\mathbb\{E\}\_\{\\mathcal\{B\}\_\{t\}\}\\\|\\mathbf\{g\}\_\{b,t\}\\\|\_\{2\}^\{2\}=\{\}1m​Tr⁡\(𝐂b​b,t\)\\displaystyle\\frac\{1\}\{m\}\\operatorname\{Tr\}\(\\mathbf\{C\}\_\{bb,t\}\)\(15\)\+\(1−1m\)​‖𝝁b,t‖22\.\\displaystyle\+\\left\(1\-\\frac\{1\}\{m\}\\right\)\\\|\\boldsymbol\{\\mu\}\_\{b,t\}\\\|\_\{2\}^\{2\}\.Thus, the squared minibatch gradient is not an unbiased empirical\-Fisher trace estimator\. Near a stationary region, where‖𝝁b,t‖2\\\|\\boldsymbol\{\\mu\}\_\{b,t\}\\\|\_\{2\}is small,m​𝔼​‖𝐠b,t‖22m\\,\\mathbb\{E\}\\\|\\mathbf\{g\}\_\{b,t\}\\\|\_\{2\}^\{2\}approaches the block\-wise second\-moment trace; the common factormmcancels after block normalization\. Away from stationarity, the statistic also retains a mean\-gradient contribution, making it a mixed first\- and second\-moment sensitivity signal\.

For blockbb, we define the instantaneous gradient energy as

eb,t=‖𝐠b,t‖22,e\_\{b,t\}=\\\|\\mathbf\{g\}\_\{b,t\}\\\|\_\{2\}^\{2\},\(16\)and maintain its exponential moving average:

sb,t=β​sb,t−1\+\(1−β\)​eb,t,β∈\[0,1\)\.s\_\{b,t\}=\\beta s\_\{b,t\-1\}\+\(1\-\\beta\)e\_\{b,t\},\\quad\\beta\\in\[0,1\)\.\(17\)Because summation and EMA updates are linear,sb,ts\_\{b,t\}is equivalent to summing Adam\-style per\-parameter second moments within blockbb\[[15](https://arxiv.org/html/2607.18306#bib.bib25),[9](https://arxiv.org/html/2607.18306#bib.bib43)\]\. We deliberately retain the total second\-moment mass rather than divide by the block dimension: it is additive over a partition and represents the aggregate sensitivity of a block under a globalℓ2\\ell\_\{2\}perturbation budget\.

Lemma 1 \(EMA estimation\)\.Assume that the stochastic energieseb,te\_\{b,t\}are locally stationary with meane¯b\\bar\{e\}\_\{b\}\. Withsb,0=0s\_\{b,0\}=0, the EMA satisfies

𝔼​\[sb,t\]=\(1−βt\)​e¯b\.\\mathbb\{E\}\[s\_\{b,t\}\]=\(1\-\\beta^\{t\}\)\\bar\{e\}\_\{b\}\.\(18\)The common bias factor1−βt1\-\\beta^\{t\}cancels in the normalized allocation rule\. The appendix also shows that, under an independent\-noise idealization, EMA reduces the variance of the instantaneous energy\. We therefore usesb,ts\_\{b,t\}as a persistent aggregate sensitivity score, not as an exact Fisher or Hessian estimator\.

### III\-DGradient\-Energy Adaptive Radius Allocation

Given the block sensitivity scores\{sb,t\}b=1B\\\{s\_\{b,t\}\\\}\_\{b=1\}^\{B\}, we allocate the global SAM radiusρ\\rhoacross blocks\. Letrb,tr\_\{b,t\}denote the radius assigned to blockbb\. We require the same global budget as SAM:

∑b=1Brb,t2≤ρ2,rb,t≥0\.\\sum\_\{b=1\}^\{B\}r\_\{b,t\}^\{2\}\\leq\\rho^\{2\},\\quad r\_\{b,t\}\\geq 0\.\(19\)This leaves the question of how to transform sensitivity scores into radii\. We use a simple surrogate allocation principle: larger sensitivity should receive a larger radius, the global budget should remain controlled, and the rule should not introduce additional block\-wise hyperparameters\. These requirements do not uniquely solve the second\-order inner problem\. We instantiate them with a scale\-invariant linear alignment surrogate:

max\{rb,t\}b=1B∑b=1Bsb,trb,t,s\.t\.∑b=1Brb,t2≤ρ2,rb,t≥0\.\\max\_\{\\\{r\_\{b,t\}\\\}\_\{b=1\}^\{B\}\}\\sum\_\{b=1\}^\{B\}s\_\{b,t\}r\_\{b,t\},\\quad\\mathrm\{s\.t\.\}\\quad\\sum\_\{b=1\}^\{B\}r\_\{b,t\}^\{2\}\\leq\\rho^\{2\},\\quad r\_\{b,t\}\\geq 0\.\(20\)Equation \([20](https://arxiv.org/html/2607.18306#S3.E20)\) is a budget\-preserving surrogate rather than an exact second\-order solution\. We align radii directly with the second\-moment masssb,ts\_\{b,t\}, which emphasizes blocks with persistently large aggregate sensitivity, is invariant to a common rescaling of all scores, and avoids an additional exponent hyperparameter\.

Proposition 1 \(Closed\-form allocation\)\.If the nonnegative score vector is nonzero, the unique radius vector solving Eq\. \([20](https://arxiv.org/html/2607.18306#S3.E20)\) is

rb,t⋆=ρ​sb,t∑j=1Bsj,t2\.r\_\{b,t\}^\{\\star\}=\\rho\\frac\{s\_\{b,t\}\}\{\\sqrt\{\\sum\_\{j=1\}^\{B\}s\_\{j,t\}^\{2\}\}\}\.\(21\)The proof is given in the appendix\. We denote the resulting block radius byρb,t=rb,t⋆\\rho\_\{b,t\}=r\_\{b,t\}^\{\\star\}\.

The perturbation direction within each block is kept as the current gradient direction, giving

ϵb,t=ρb,t​𝐠b,t‖𝐠b,t‖2\.\\boldsymbol\{\\epsilon\}\_\{b,t\}=\\rho\_\{b,t\}\\frac\{\\mathbf\{g\}\_\{b,t\}\}\{\\\|\\mathbf\{g\}\_\{b,t\}\\\|\_\{2\}\}\.\(22\)The idealized analysis assumes nonzero block gradients\. In implementation, we use the stabilized direction𝐮~b,t=𝐠b,t/\(‖𝐠b,t‖2\+δ\)\\widetilde\{\\mathbf\{u\}\}\_\{b,t\}=\\mathbf\{g\}\_\{b,t\}/\(\\\|\\mathbf\{g\}\_\{b,t\}\\\|\_\{2\}\+\\delta\)with a smallδ\>0\\delta\>0; a zero\-gradient block therefore contributes no perturbation\. The complete perturbation is

ϵt=\(ϵ1,t,ϵ2,t,…,ϵB,t\)\.\\boldsymbol\{\\epsilon\}\_\{t\}=\(\\boldsymbol\{\\epsilon\}\_\{1,t\},\\boldsymbol\{\\epsilon\}\_\{2,t\},\\ldots,\\boldsymbol\{\\epsilon\}\_\{B,t\}\)\.\(23\)
The allocated radii exactly exhaust the radius budget,

∑b=1Bρb,t2=ρ2\.\\sum\_\{b=1\}^\{B\}\\rho\_\{b,t\}^\{2\}=\\rho^\{2\}\.\(24\)With exact unit directions, Eq\. \([24](https://arxiv.org/html/2607.18306#S3.E24)\) also gives‖ϵt‖2=ρ\\\|\\boldsymbol\{\\epsilon\}\_\{t\}\\\|\_\{2\}=\\rho\. With the stabilized implementation,‖𝐮~b,t‖2≤1\\\|\\widetilde\{\\mathbf\{u\}\}\_\{b,t\}\\\|\_\{2\}\\leq 1, and hence‖ϵt‖2≤ρ\\\|\\boldsymbol\{\\epsilon\}\_\{t\}\\\|\_\{2\}\\leq\\rho\. Therefore, GEAR\-SAM never enlarges the SAM neighborhood\. Its effect comes from reallocating the same radius budget according to historical curvature\-related block sensitivity\.

### III\-EAlgorithm and Complexity

After constructingϵt\\boldsymbol\{\\epsilon\}\_\{t\}, GEAR\-SAM follows the standard SAM update rule\. The perturbed gradient is computed as

𝐆t=∇𝐰Lℬt​\(𝐰t\+ϵt\)\.\\mathbf\{G\}\_\{t\}=\\nabla\_\{\\mathbf\{w\}\}L\_\{\\mathcal\{B\}\_\{t\}\}\(\\mathbf\{w\}\_\{t\}\+\\boldsymbol\{\\epsilon\}\_\{t\}\)\.\(25\)With SGD as the base optimizer, the parameter update is

𝐰t\+1=𝐰t−η​𝐆t\.\\mathbf\{w\}\_\{t\+1\}=\\mathbf\{w\}\_\{t\}\-\\eta\\mathbf\{G\}\_\{t\}\.\(26\)Other base optimizers can also be used in the same way\.

Algorithm 1GEAR\-SAM0:Training data, model parameters

𝐰\\mathbf\{w\}, learning rate

η\\eta, SAM radius

ρ\\rho, EMA factor

β\\beta, stability constant

δ\\delta
1:Initialize

sb=0s\_\{b\}=0for each block

b=1,…,Bb=1,\\ldots,B
2:foreach iteration

ttdo

3:Sample a minibatch

ℬt\\mathcal\{B\}\_\{t\}
4:Compute block\-wise gradients on

ℬt\\mathcal\{B\}\_\{t\}
5:Update block sensitivity scores according to Eq\. \([17](https://arxiv.org/html/2607.18306#S3.E17)\)

6:ifthe score vector is nonzerothen

7:Allocate block radii according to Eq\. \([21](https://arxiv.org/html/2607.18306#S3.E21)\)

8:else

9:Set

ρb,t=0\\rho\_\{b,t\}=0for all blocks

10:endif

11:Construct stabilized block perturbations as described after Eq\. \([22](https://arxiv.org/html/2607.18306#S3.E22)\)

12:Compute the perturbed gradient according to Eq\. \([25](https://arxiv.org/html/2607.18306#S3.E25)\)

13:Update the parameters according to Eq\. \([26](https://arxiv.org/html/2607.18306#S3.E26)\)

14:endfor

GEAR\-SAM has the same number of forward–backward passes as SAM: one gradient computation constructs the perturbation and a second computes the gradient at the perturbed point\. Computing the block energies requires reductions over the model gradients, withO​\(d\)O\(d\)arithmetic forddparameters, comparable to the global\-norm reduction already used by SAM\. The persistent state consists of onlyBBscalar EMA values, followed byO​\(B\)O\(B\)normalization\. Thus GEAR\-SAM does not change the dominant forward–backward complexity and introduces only lightweight reductions and scalar state beyond SAM\.

Different from sparse or selective SAM methods, GEAR\-SAM does not decide whether a layer should be perturbed\. Instead, it continuously allocates radii across blocks under a fixed global budget\. Different from Fisher\-based SAM variants, it does not modify the perturbation geometry or require explicit Fisher matrix estimation\. Thus, GEAR\-SAM provides a lightweight plug\-in allocation mechanism for sharpness\-aware optimization\.

## IVExperiments

### IV\-AImage Classification

#### IV\-A1Setup

To evaluate GEAR\-SAM’s effectiveness, we conduct experiments on the CIFAR\-10 and CIFAR\-100 datasets across a variety of architectures, i\.e\., ResNet\-18, WideResNet\-28\-10, and PyramidNet\-110\. ResNet\-18 and WideResNet\-28\-10 are trained for 200 epochs, while PyramidNet\-110 is trained for 300 epochs\. For ResNet\-18 and WideResNet\-28\-10, the initial learning rate is set to 0\.05 with a cosine schedule, and the momentum and weight decay are set to 0\.9 and 0\.001, respectively\. For PyramidNet\-110, the initial learning rate is 0\.1, and the momentum and weight decay are set to 0\.9 and 0\.0005, respectively\. For CIFAR\-10,ρ\\rhois set to 0\.1, while for CIFAR\-100, it is set to 0\.2\. The EMA factorβ\\betais set to 0\.9 for both datasets\.

We apply the architecture\-aware block partition described in the method section\. ResNet\-18 is divided into the stem, four coarse residual blocks \(layer1–layer4\), and the classifier\. WideResNet\-28\-10 and PyramidNet\-110 are divided into the stem, three coarse representation blocks, and the classifier\. For GEAR\-SAM, we conduct three independent runs with different random seeds and report the average accuracy and standard deviation\.

We use vanilla SGD and SAM as baselines\. To comprehensively evaluate the performance, we also include ASAM, FisherSAM, F\-SAM, SGD\-SALR, Unified VaSSO, SSAM\-F, and SSAM\-D for comparison\. These methods are follow\-up works of SAM that aim to enhance generalization\. For ASAM, FisherSAM, and F\-SAM, we report the results in F\-SAM\[[19](https://arxiv.org/html/2607.18306#bib.bib17)\]\. These three methods were run for 300 epochs on PyramidNet\-110, and we denote them as ASAM\(300\), FisherSAM\(300\), and F\-SAM\(300\), respectively\. In addition, we compare several efficient SAM variants, including LookSAM, ESAM, and SAF\. For SGD\-SALR, Unified VaSSO, SSAM\-F, SSAM\-D, LookSAM, ESAM, SAF, SGD, SAM, and BSAM, we directly use the results reported in BSAM\[[4](https://arxiv.org/html/2607.18306#bib.bib18)\]\.

#### IV\-A2Results

Table[I](https://arxiv.org/html/2607.18306#S4.T1)reports the test accuracy on CIFAR\-10 and CIFAR\-100\. GEAR\-SAM consistently improves upon SGD across all architectures and datasets\. On CIFAR\-10, its largest gain over SGD is obtained with PyramidNet\-110, where accuracy increases by 0\.71%; it also improves standard SAM by 0\.09% and 0\.16% on ResNet\-18 and PyramidNet\-110, respectively\. On CIFAR\-100, the improvements are considerably larger\. GEAR\-SAM exceeds SGD by 1\.85%, 2\.97%, and 3\.52%, and exceeds SAM by 0\.66%, 0\.82%, and 0\.84% on ResNet\-18, WideResNet\-28\-10, and PyramidNet\-110, respectively\. This indicates that the proposed allocation is effective across networks with different depth and width configurations\.

Compared with other SAM variants, GEAR\-SAM achieves the best accuracy for all three architectures on CIFAR\-100, including both methods designed to enhance generalization and efficient SAM variants\. On CIFAR\-10, it remains competitive, while SSAM\-D gives the highest accuracy on ResNet\-18 and WideResNet\-28\-10 and BSAM gives the highest accuracy on PyramidNet\-110\. The smaller differences among methods on CIFAR\-10 suggest that performance is already close to saturation on this dataset\. In contrast, the consistent advantage on CIFAR\-100 shows that historical block sensitivity becomes more useful as the classification problem requires finer inter\-class discrimination\. Overall, these results support the effectiveness of using a stable curvature\-related signal to guide perturbation allocation across heterogeneous network blocks\.

TABLE I:Test accuracy \(%\) comparison of various networks on CIFAR\-10 and CIFAR\-100\.

### IV\-BTop Eigenvalues of Hessian

The maximum Hessian eigenvalue is commonly viewed as a potential sharpness/flatness metric\[[12](https://arxiv.org/html/2607.18306#bib.bib5),[10](https://arxiv.org/html/2607.18306#bib.bib6),[1](https://arxiv.org/html/2607.18306#bib.bib7)\]\. To offer a more quantitative assessment of sharpness, we compare the maximum Hessian eigenvalues obtained from SGD, SAM, and GEAR\-SAM\. The resulting spectra provide complementary evidence about both the sharpest local direction and the overall distribution of dominant curvature\.

![Refer to caption](https://arxiv.org/html/2607.18306v1/x2.png)

![Refer to caption](https://arxiv.org/html/2607.18306v1/x3.png)

![Refer to caption](https://arxiv.org/html/2607.18306v1/x4.png)

Figure 2:Histograms of the top\-50 Hessian eigenvalues of ResNet\-18 trained on CIFAR\-100 with \(a\) SGD, \(b\) SAM, and \(c\) GEAR\-SAM\. Each panel also reports the maximum eigenvalue and the mean and variance of the top\-50 eigenvalues\.We analyze the Hessian spectra of ResNet\-18 trained on CIFAR\-100 with SGD, SAM, and GEAR\-SAM using the CIFAR\-100 test set\. We compute the Hessian spectrum using PyHessian\[[28](https://arxiv.org/html/2607.18306#bib.bib44)\]\. For each method, we report the maximum eigenvalue of the Hessian, a histogram of the top\-50 Hessian eigenvalues, and the mean and variance of these top\-50 Hessian eigenvalues\. As shown in Fig\.[2](https://arxiv.org/html/2607.18306#S4.F2), both SAM and GEAR\-SAM substantially shift the dominant Hessian spectrum toward smaller values compared with SGD\. GEAR\-SAM obtains the smallest maximum eigenvalue, reducing it from 365\.78 for SGD and 200\.72 for SAM to 192\.92\. Its variance is also slightly lower than that of SAM \(1633\.00 versus 1656\.25\), suggesting that GEAR\-SAM suppresses extreme high\-curvature directions without broadening the dominant spectrum\. GEAR\-SAM does not outperform SAM on every spectral statistic: its top\-50 mean is 55\.16, compared with 52\.48 for SAM\. This result indicates that the benefit of block\-wise radius allocation is concentrated in reducing the sharpest local direction rather than uniformly shrinking all dominant eigenvalues\. Overall, the spectrum supports a more precise conclusion: GEAR\-SAM reaches a curvature profile comparable to SAM while further reducing the worst\-case local sharpness measured by the maximum Hessian eigenvalue\.

### IV\-CRobustness to Label Noise

Since prior studies have shown that SAM is robust to label noise, this section evaluates the effect of applying GEAR\-SAM in the classical noisy\-label setting for CIFAR\-10 and CIFAR\-100\. We evaluate GEAR\-SAM under symmetric label noise by randomly flipping training labels\[[8](https://arxiv.org/html/2607.18306#bib.bib40),[4](https://arxiv.org/html/2607.18306#bib.bib18)\]\. The training settings are the same as those in the image classification experiments, except that the perturbation radii of both SAM and GEAR\-SAM are set toρ=0\.05\\rho=0\.05for all noise rates on both datasets\.

As shown in Table[II](https://arxiv.org/html/2607.18306#S4.T2), for CIFAR\-10, the accuracy of models optimized with SGD decreases rapidly as the noise rate increases, dropping to only 28\.91% when the noise rate reaches 80%\. In contrast, the accuracy of models optimized with SAM and GEAR\-SAM decreases more slowly as the label noise increases, and both methods maintain over 70% accuracy at the 80% noise rate\. GEAR\-SAM achieves the best results at all four noise rates, reaching 92\.90%, 90\.91%, 87\.21%, and 79\.15%, respectively\.

For CIFAR\-100, we observe a similar overall trend\. At the 20% noise rate, SAM obtains slightly higher accuracy than GEAR\-SAM\. When the noise rate rises to 40%, 60%, and 80%, however, GEAR\-SAM achieves 57\.86%, 50\.45%, and 35\.96%, respectively, consistently outperforming both SGD and SAM\. In particular, when the noise rate reaches 80%, SAM falls below SGD, whereas GEAR\-SAM still maintains substantially higher accuracy\. These observations indicate that GEAR\-SAM can converge effectively in the presence of extensive incorrect supervision, facilitates learning from the remaining clean labels, and mitigates overfitting to noisy labels\. Overall, GEAR\-SAM improves the robustness of sharpness\-aware training and provides stronger generalization under moderate and severe label noise\.

TABLE II:Test accuracy \(%\) of ResNet\-18 under symmetric label noise\.
### IV\-DTransfer Learning

Transfer learning leverages a model trained on one task to improve performance on a related task\. By fine\-tuning a pretrained model or using its learned representations, transfer learning can reduce the training time and data requirement of the target task, and often improves performance when the target dataset is limited\. Previous studies have demonstrated the effectiveness of SAM and its variants in transfer learning\[[6](https://arxiv.org/html/2607.18306#bib.bib8),[19](https://arxiv.org/html/2607.18306#bib.bib17)\]\. In this section, we evaluate the performance of GEAR\-SAM in transfer learning tasks\.

Specifically, we apply SGD, SAM, and GEAR\-SAM to fine\-tune EfficientNet\-B0 and ResNet\-50 pretrained on ImageNet\. Weights are initialized from publicly available checkpoints, except for the final classification layer, which is resized according to the number of target classes and randomly initialized\. We train the models for 30 epochs with a batch size of 128\. The initial learning rate is set to 0\.01 with cosine learning rate decay\. Weight decay is set to1×10−51\\times 10^\{\-5\}for EfficientNet\-B0 and1×10−41\\times 10^\{\-4\}for ResNet\-50\. For SAM and GEAR\-SAM, we use SGD as the base optimizer\. The perturbation radius of SAM is set to 0\.05\. For GEAR\-SAM, the perturbation radiusρ\\rhois set to 0\.2, and the EMA factorβ\\betais set to 0\.9\. We do not use any data augmentation for Flowers102, Stanford Cars, and Oxford\-IIIT Pet\. For CIFAR\-10 and CIFAR\-100, we employ the same data augmentations as the image classification experiments\.

As shown in Table[III](https://arxiv.org/html/2607.18306#S4.T3), SAM exhibits stronger generalization than SGD on both EfficientNet\-B0 and ResNet\-50, and GEAR\-SAM further improves SAM in most transfer\-learning tasks\. On EfficientNet\-B0, GEAR\-SAM achieves the best results on all five datasets, with particularly large gains on Flowers102, Stanford Cars, and Oxford\-IIIT Pet\. On ResNet\-50, GEAR\-SAM obtains the best accuracy on CIFAR\-100, Flowers102, and Stanford Cars, while remaining competitive on CIFAR\-10 and Oxford\-IIIT Pet\. These results indicate that the proposed block\-wise radius allocation is effective not only when training from scratch, but also when adapting pretrained representations to downstream tasks\.

TABLE III:Test accuracy \(%\) on transfer\-learning benchmarks\.Additional results comparing alternative block partitions are provided in the appendix\.

### IV\-EParameter Studies

We examine the sensitivity of GEAR\-SAM to its main hyperparameters\. The perturbation radiusρ\\rhocontrols the size of the neighborhood, while the EMA factorβ\\betacontrols the stability of the block sensitivity estimate\. Tables[IV](https://arxiv.org/html/2607.18306#S4.T4)and[V](https://arxiv.org/html/2607.18306#S4.T5)show that GEAR\-SAM performs best aroundρ=0\.2\\rho=0\.2andβ=0\.9\\beta=0\.9on ResNet\-18 with CIFAR\-100\. Smaller radii provide weaker sharpness regularization, while larger radii may harm optimization\. Smallerβ\\betavalues make the allocation close to instantaneous gradients, while larger values adapt too slowly\.

TABLE IV:Effect of perturbation radiusρ\\rhoon CIFAR\-100 with ResNet\-18\.TABLE V:Effect of EMA factorβ\\betaon CIFAR\-100 with ResNet\-18\.

## VConclusion

In this paper, we revisited SAM from a curvature\-aware block\-wise allocation perspective\. We showed that the practical SAM perturbation is optimal for the first\-order inner maximization, but the second\-order sharpness objective indicates that curvature information should also guide perturbations\. We further showed that standard SAM already performs an implicit block\-wise radius allocation, where the radius of each block depends only on instantaneous minibatch gradient norms\. Based on these observations, we proposed GEAR\-SAM, which estimates block sensitivity using EMA\-based gradient energy and allocates the fixed SAM radius through a closed\-form constrained alignment rule\. GEAR\-SAM preserves the global SAM budget and requires no explicit Hessian or Fisher computation\. Experiments on image classification, transfer learning, noisy\-label learning, and partition and parameter studies demonstrate that the proposed allocation strategy improves the effectiveness and robustness of SAM\-based optimization\.

## References

- \[1\]P\. Chaudhari, A\. Choromanska, S\. Soatto, Y\. LeCun, C\. Baldassi, C\. Borgs, J\. Chayes, L\. Sagun, and R\. Zecchina\(2019\)Entropy\-sgd: biasing gradient descent into wide valleys\.Journal of Statistical Mechanics: Theory and Experiment2019\(12\),pp\. 124018\.Cited by:[§I](https://arxiv.org/html/2607.18306#S1.p1.1),[§II\-C](https://arxiv.org/html/2607.18306#S2.SS3.p1.1),[§IV\-B](https://arxiv.org/html/2607.18306#S4.SS2.p1.1)\.
- \[2\]\(2026\)Sparse layer sharpness\-aware minimization for efficient fine\-tuning\.arXiv preprint arXiv:2602\.09395\.Cited by:[§II\-B](https://arxiv.org/html/2607.18306#S2.SS2.p1.1)\.
- \[3\]Y\. Dai, K\. Ahn, and S\. Sra\(2023\)The crucial role of normalization in sharpness\-aware minimization\.Advances in Neural Information Processing Systems36,pp\. 67741–67770\.Cited by:[§II\-B](https://arxiv.org/html/2607.18306#S2.SS2.p1.1)\.
- \[4\]J\. Deng, J\. Pang, B\. Zhang, and Q\. Huang\(2026\)Bilateral sharpness\-aware minimization for flatter minima\.IEEE Transactions on Neural Networks and Learning Systems\.Cited by:[§I](https://arxiv.org/html/2607.18306#S1.p1.1),[§II\-A](https://arxiv.org/html/2607.18306#S2.SS1.p1.1),[§IV\-A1](https://arxiv.org/html/2607.18306#S4.SS1.SSS1.p3.3),[§IV\-C](https://arxiv.org/html/2607.18306#S4.SS3.p1.1)\.
- \[5\]J\. Du, H\. Yan, J\. Feng, J\. T\. Zhou, L\. Zhen, R\. S\. M\. Goh, and V\. Y\. Tan\(2021\)Efficient sharpness\-aware minimization for improved training of neural networks\.arXiv preprint arXiv:2110\.03141\.Cited by:[§II\-A](https://arxiv.org/html/2607.18306#S2.SS1.p1.1)\.
- \[6\]P\. Foret, A\. Kleiner, H\. Mobahi, and B\. Neyshabur\(2020\)Sharpness\-aware minimization for efficiently improving generalization\.arXiv preprint arXiv:2010\.01412\.Cited by:[§I](https://arxiv.org/html/2607.18306#S1.p1.1),[§II\-A](https://arxiv.org/html/2607.18306#S2.SS1.p1.1),[§IV\-D](https://arxiv.org/html/2607.18306#S4.SS4.p1.1)\.
- \[7\]M\. Haas, J\. Xu, V\. Cevher, and L\. C\. Vankadara\(2024\)μ\\muP2: effective sharpness aware minimization requires layerwise perturbation scaling\.arXiv preprint arXiv:2411\.00075\.Cited by:[§II\-B](https://arxiv.org/html/2607.18306#S2.SS2.p1.1)\.
- \[8\]J\. Huang, L\. Qu, R\. Jia, and B\. Zhao\(2019\)O2U\-net: a simple noisy label detection approach for deep neural networks\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 3326–3334\.Cited by:[§IV\-C](https://arxiv.org/html/2607.18306#S4.SS3.p1.1)\.
- \[9\]D\. Hwang\(2024\)FAdam: adam is a natural gradient optimizer using diagonal empirical fisher information\.arXiv preprint arXiv:2405\.12807\.Cited by:[§I](https://arxiv.org/html/2607.18306#S1.p3.1),[§II\-C](https://arxiv.org/html/2607.18306#S2.SS3.p2.1),[§III\-C](https://arxiv.org/html/2607.18306#S3.SS3.p4.4)\.
- \[10\]S\. Jastrzębski, Z\. Kenton, D\. Arpit, N\. Ballas, A\. Fischer, Y\. Bengio, and A\. Storkey\(2017\)Three factors influencing minima in sgd\.arXiv preprint arXiv:1711\.04623\.Cited by:[§II\-C](https://arxiv.org/html/2607.18306#S2.SS3.p1.1),[§IV\-B](https://arxiv.org/html/2607.18306#S4.SS2.p1.1)\.
- \[11\]W\. Jiang, H\. Yang, Y\. Zhang, and J\. Kwok\(2023\)An adaptive policy to employ sharpness\-aware minimization\.arXiv preprint arXiv:2304\.14647\.Cited by:[§II\-A](https://arxiv.org/html/2607.18306#S2.SS1.p1.1)\.
- \[12\]Y\. Jiang, B\. Neyshabur, H\. Mobahi, D\. Krishnan, and S\. Bengio\(2019\)Fantastic generalization measures and where to find them\.arXiv preprint arXiv:1912\.02178\.Cited by:[§I](https://arxiv.org/html/2607.18306#S1.p1.1),[§II\-C](https://arxiv.org/html/2607.18306#S2.SS3.p1.1),[§IV\-B](https://arxiv.org/html/2607.18306#S4.SS2.p1.1)\.
- \[13\]N\. S\. Keskar, D\. Mudigere, J\. Nocedal, M\. Smelyanskiy, and P\. T\. P\. Tang\(2016\)On large\-batch training for deep learning: generalization gap and sharp minima\.arXiv preprint arXiv:1609\.04836\.Cited by:[§I](https://arxiv.org/html/2607.18306#S1.p1.1),[§II\-C](https://arxiv.org/html/2607.18306#S2.SS3.p1.1)\.
- \[14\]M\. Kim, D\. Li, S\. X\. Hu, and T\. Hospedales\(2022\)Fisher sam: information geometry and sharpness aware minimisation\.InInternational Conference on Machine Learning,pp\. 11148–11161\.Cited by:[§I](https://arxiv.org/html/2607.18306#S1.p1.1),[§I](https://arxiv.org/html/2607.18306#S1.p3.1),[§II\-C](https://arxiv.org/html/2607.18306#S2.SS3.p2.1)\.
- \[15\]D\. P\. Kingma and J\. Ba\(2014\)Adam: a method for stochastic optimization\.arXiv preprint arXiv:1412\.6980\.Cited by:[§II\-C](https://arxiv.org/html/2607.18306#S2.SS3.p2.1),[§III\-C](https://arxiv.org/html/2607.18306#S3.SS3.p4.4)\.
- \[16\]F\. Kunstner, P\. Hennig, and L\. Balles\(2019\)Limitations of the empirical fisher approximation for natural gradient descent\.Advances in neural information processing systems32\.Cited by:[§I](https://arxiv.org/html/2607.18306#S1.p3.1),[§II\-C](https://arxiv.org/html/2607.18306#S2.SS3.p2.1),[§III\-C](https://arxiv.org/html/2607.18306#S3.SS3.p2.3)\.
- \[17\]J\. Kwon, J\. Kim, H\. Park, and I\. K\. Choi\(2021\)Asam: adaptive sharpness\-aware minimization for scale\-invariant learning of deep neural networks\.InInternational conference on machine learning,pp\. 5905–5914\.Cited by:[§I](https://arxiv.org/html/2607.18306#S1.p1.1),[§II\-A](https://arxiv.org/html/2607.18306#S2.SS1.p1.1)\.
- \[18\]S\. Lee\(2024\)Layer\-wise adaptive gradient norm penalizing method for efficient and accurate deep learning\.InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,pp\. 1518–1529\.Cited by:[§II\-B](https://arxiv.org/html/2607.18306#S2.SS2.p1.1)\.
- \[19\]T\. Li, P\. Zhou, Z\. He, X\. Cheng, and X\. Huang\(2024\)Friendly sharpness\-aware minimization\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 5631–5640\.Cited by:[§I](https://arxiv.org/html/2607.18306#S1.p1.1),[§II\-A](https://arxiv.org/html/2607.18306#S2.SS1.p1.1),[§IV\-A1](https://arxiv.org/html/2607.18306#S4.SS1.SSS1.p3.3),[§IV\-D](https://arxiv.org/html/2607.18306#S4.SS4.p1.1)\.
- \[20\]Y\. Liu, S\. Mai, X\. Chen, C\. Hsieh, and Y\. You\(2022\)Towards efficient and scalable sharpness\-aware minimization\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 12360–12370\.Cited by:[§II\-A](https://arxiv.org/html/2607.18306#S2.SS1.p1.1)\.
- \[21\]P\. M\. Long and P\. L\. Bartlett\(2024\)Sharpness\-aware minimization and the edge of stability\.Journal of Machine Learning Research25\(179\),pp\. 1–20\.Cited by:[§II\-C](https://arxiv.org/html/2607.18306#S2.SS3.p1.1)\.
- \[22\]P\. Mi, L\. Shen, T\. Ren, Y\. Zhou, X\. Sun, R\. Ji, and D\. Tao\(2022\)Make sharpness\-aware minimization stronger: a sparsified perturbation approach\.Advances in Neural Information Processing Systems35,pp\. 30950–30962\.Cited by:[§II\-B](https://arxiv.org/html/2607.18306#S2.SS2.p1.1)\.
- \[23\]J\. J\. Moré and D\. C\. Sorensen\(1983\)Computing a trust region step\.SIAM Journal on Scientific and Statistical Computing4\(3\),pp\. 553–572\.Cited by:[§III\-A](https://arxiv.org/html/2607.18306#S3.SS1.p2.2),[Trust\-region Conditions and SAM Decomposition](https://arxiv.org/html/2607.18306#Sx1.SSx1.p1.4)\.
- \[24\]M\. Mueller, T\. Vlaar, D\. Rolnick, and M\. Hein\(2023\)Normalization layers are all that sharpness\-aware minimization needs\.Advances in Neural Information Processing Systems36,pp\. 69228–69252\.Cited by:[§II\-B](https://arxiv.org/html/2607.18306#S2.SS2.p1.1),[§III\-B](https://arxiv.org/html/2607.18306#S3.SS2.p2.1)\.
- \[25\]R\. Ni, P\. Chiang, J\. Geiping, M\. Goldblum, A\. G\. Wilson, and T\. Goldstein\(2022\)K\-sam: sharpness\-aware minimization at the speed of sgd\.arXiv preprint arXiv:2210\.12864\.Cited by:[§II\-A](https://arxiv.org/html/2607.18306#S2.SS1.p1.1)\.
- \[26\]T\. Wu, T\. Luo, and D\. C\. Wunsch II\(2024\)Cr\-sam: curvature regularized sharpness\-aware minimization\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 6144–6152\.Cited by:[§II\-C](https://arxiv.org/html/2607.18306#S2.SS3.p1.1)\.
- \[27\]W\. Xie, T\. Pethick, and V\. Cevher\(2024\)Sampa: sharpness\-aware minimization parallelized\.Advances in Neural Information Processing Systems37,pp\. 51333–51357\.Cited by:[§II\-A](https://arxiv.org/html/2607.18306#S2.SS1.p1.1)\.
- \[28\]Z\. Yao, A\. Gholami, K\. Keutzer, and M\. W\. Mahoney\(2020\)Pyhessian: neural networks through the lens of the hessian\.In2020 IEEE international conference on big data \(Big data\),pp\. 581–590\.Cited by:[§IV\-B](https://arxiv.org/html/2607.18306#S4.SS2.p2.1)\.
- \[29\]X\. Zhang, R\. Xu, H\. Yu, H\. Zou, and P\. Cui\(2023\)Gradient norm aware minimization seeks first\-order flatness and improves generalization\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 20247–20257\.Cited by:[§II\-C](https://arxiv.org/html/2607.18306#S2.SS3.p1.1)\.
- \[30\]Q\. Zhong, L\. Ding, L\. Shen, P\. Mi, J\. Liu, B\. Du, and D\. Tao\(2022\)Improving sharpness\-aware minimization with fisher mask for better generalization on language models\.InFindings of the Association for Computational Linguistics: EMNLP 2022,pp\. 4064–4085\.Cited by:[§II\-B](https://arxiv.org/html/2607.18306#S2.SS2.p1.1),[§II\-C](https://arxiv.org/html/2607.18306#S2.SS3.p2.1)\.
- \[31\]J\. Zhuang, B\. Gong, L\. Yuan, Y\. Cui, H\. Adam, N\. Dvornek, S\. Tatikonda, J\. Duncan, and T\. Liu\(2022\)Surrogate gap minimization improves sharpness\-aware training\.arXiv preprint arXiv:2203\.08065\.Cited by:[§I](https://arxiv.org/html/2607.18306#S1.p1.1),[§II\-A](https://arxiv.org/html/2607.18306#S2.SS1.p1.1)\.

## Appendix

This appendix provides additional details supporting the design of GEAR\-SAM\. It gives complete derivations for the trust\-region motivation, the stochastic gradient second moment, the EMA estimator, and the budget\-preserving allocation rule\. These results establish exact properties of the proposed surrogate while keeping separate the assumptions used to motivate its curvature\-related sensitivity signal\.

### Trust\-region Conditions and SAM Decomposition

For the quadratic approximation, define

Qt​\(ϵ\)=𝐠t⊤​ϵ\+12​ϵ⊤​𝐇t​ϵ\.Q\_\{t\}\(\\boldsymbol\{\\epsilon\}\)=\\mathbf\{g\}\_\{t\}^\{\\top\}\\boldsymbol\{\\epsilon\}\+\\frac\{1\}\{2\}\\boldsymbol\{\\epsilon\}^\{\\top\}\\mathbf\{H\}\_\{t\}\\boldsymbol\{\\epsilon\}\.\(27\)The Lagrangian for maximizingQtQ\_\{t\}subject to‖ϵ‖22≤ρ2\\\|\\boldsymbol\{\\epsilon\}\\\|\_\{2\}^\{2\}\\leq\\rho^\{2\}is

ℒ​\(ϵ,λ\)=Qt​\(ϵ\)−λ​\(‖ϵ‖22−ρ2\),λ≥0\.\\mathcal\{L\}\(\\boldsymbol\{\\epsilon\},\\lambda\)=Q\_\{t\}\(\\boldsymbol\{\\epsilon\}\)\-\\lambda\(\\\|\\boldsymbol\{\\epsilon\}\\\|\_\{2\}^\{2\}\-\\rho^\{2\}\),\\quad\\lambda\\geq 0\.\(28\)By the classical trust\-region optimality theorem\[[23](https://arxiv.org/html/2607.18306#bib.bib45)\], a global maximizerϵt⋆\\boldsymbol\{\\epsilon\}\_\{t\}^\{\\star\}admits a multiplierλt\\lambda\_\{t\}satisfying

𝐠t\+𝐇t​ϵt⋆\\displaystyle\\mathbf\{g\}\_\{t\}\+\\mathbf\{H\}\_\{t\}\\boldsymbol\{\\epsilon\}\_\{t\}^\{\\star\}=2​λt​ϵt⋆,\\displaystyle=2\\lambda\_\{t\}\\boldsymbol\{\\epsilon\}\_\{t\}^\{\\star\},\(29\)‖ϵt⋆‖22\\displaystyle\\\|\\boldsymbol\{\\epsilon\}\_\{t\}^\{\\star\}\\\|\_\{2\}^\{2\}≤ρ2,\\displaystyle\\leq\\rho^\{2\},λt​\(‖ϵt⋆‖22−ρ2\)\\displaystyle\\lambda\_\{t\}\(\\\|\\boldsymbol\{\\epsilon\}\_\{t\}^\{\\star\}\\\|\_\{2\}^\{2\}\-\\rho^\{2\}\)=0,\\displaystyle=0,2​λt​𝐈−𝐇t\\displaystyle 2\\lambda\_\{t\}\\mathbf\{I\}\-\\mathbf\{H\}\_\{t\}⪰0\.\\displaystyle\\succeq 0\.These conditions are also sufficient\. To see this, let𝐝=ϵ−ϵt⋆\\mathbf\{d\}=\\boldsymbol\{\\epsilon\}\-\\boldsymbol\{\\epsilon\}\_\{t\}^\{\\star\}for any feasibleϵ\\boldsymbol\{\\epsilon\}\. Stationarity gives

Qt​\(ϵ\)−Qt​\(ϵt⋆\)\\displaystyle Q\_\{t\}\(\\boldsymbol\{\\epsilon\}\)\-Q\_\{t\}\(\\boldsymbol\{\\epsilon\}\_\{t\}^\{\\star\}\)=2​λt​\(ϵt⋆\)⊤​𝐝\+12​𝐝⊤​𝐇t​𝐝\\displaystyle=2\\lambda\_\{t\}\(\\boldsymbol\{\\epsilon\}\_\{t\}^\{\\star\}\)^\{\\top\}\\mathbf\{d\}\+\\frac\{1\}\{2\}\\mathbf\{d\}^\{\\top\}\\mathbf\{H\}\_\{t\}\\mathbf\{d\}\(30\)≤−12​𝐝⊤​\(2​λt​𝐈−𝐇t\)​𝐝≤0,\\displaystyle\\leq\-\\frac\{1\}\{2\}\\mathbf\{d\}^\{\\top\}\(2\\lambda\_\{t\}\\mathbf\{I\}\-\\mathbf\{H\}\_\{t\}\)\\mathbf\{d\}\\leq 0,where the first inequality follows from feasibility and complementary slackness\. If2​λt​𝐈−𝐇t≻02\\lambda\_\{t\}\\mathbf\{I\}\-\\mathbf\{H\}\_\{t\}\\succ 0, stationarity yields Eq\. \([6](https://arxiv.org/html/2607.18306#S3.E6)\)\. If it is singular, a null\-space component may be required; this is the classical hard case\. GEAR\-SAM does not attempt to solve either case online\.

Under the block\-diagonal interpretation in the main text, letab,t=‖𝐠b,t‖2a\_\{b,t\}=\\\|\\mathbf\{g\}\_\{b,t\}\\\|\_\{2\}andqb,t=𝐮b,t⊤​𝐇b​b,t​𝐮b,tq\_\{b,t\}=\\mathbf\{u\}\_\{b,t\}^\{\\top\}\\mathbf\{H\}\_\{bb,t\}\\mathbf\{u\}\_\{b,t\}\. For an active block coordinate withrb,t\>0r\_\{b,t\}\>0, the KKT stationarity condition becomes

ab,t\+qb,t​rb,t=2​λt​rb,t\.a\_\{b,t\}\+q\_\{b,t\}r\_\{b,t\}=2\\lambda\_\{t\}r\_\{b,t\}\.\(31\)Whenever2​λt\>maxb⁡qb,t2\\lambda\_\{t\}\>\\max\_\{b\}q\_\{b,t\}, this gives

rb,t=ab,t2​λt−qb,t\.r\_\{b,t\}=\\frac\{a\_\{b,t\}\}\{2\\lambda\_\{t\}\-q\_\{b,t\}\}\.\(32\)For a fixed feasible multiplier and comparableab,ta\_\{b,t\}, the expression increases with directional curvatureqb,tq\_\{b,t\}\. It motivates using curvature\-related block information but does not derive GEAR\-SAM’s surrogate allocation rule\.

For completeness, when𝐠t≠𝟎\\mathbf\{g\}\_\{t\}\\neq\\mathbf\{0\}, Observation 1 follows directly from the orthogonal parameter partition:

‖𝐠t‖22=∑b=1B‖𝐠b,t‖22\.\\\|\\mathbf\{g\}\_\{t\}\\\|\_\{2\}^\{2\}=\\sum\_\{b=1\}^\{B\}\\\|\\mathbf\{g\}\_\{b,t\}\\\|\_\{2\}^\{2\}\.\(33\)Substituting Eq\. \([10](https://arxiv.org/html/2607.18306#S3.E10)\) gives

∑b=1B\(rb,tSAM\)2=ρ2​∑b‖𝐠b,t‖22∑j‖𝐠j,t‖22=ρ2\.\\sum\_\{b=1\}^\{B\}\(r^\{\\mathrm\{SAM\}\}\_\{b,t\}\)^\{2\}=\\rho^\{2\}\\frac\{\\sum\_\{b\}\\\|\\mathbf\{g\}\_\{b,t\}\\\|\_\{2\}^\{2\}\}\{\\sum\_\{j\}\\\|\\mathbf\{g\}\_\{j,t\}\\\|\_\{2\}^\{2\}\}=\\rho^\{2\}\.\(34\)If𝐠t=𝟎\\mathbf\{g\}\_\{t\}=\\mathbf\{0\}, SAM uses the zero perturbation by convention, and the block\-wise radii are all set to zero\.

### Minibatch Gradient Second Moment

Condition on𝐰t\\mathbf\{w\}\_\{t\}and let𝐳b,i=∇𝐰bℓ​\(𝐰t;xi\)\\mathbf\{z\}\_\{b,i\}=\\nabla\_\{\\mathbf\{w\}\_\{b\}\}\\ell\(\\mathbf\{w\}\_\{t\};x\_\{i\}\)be IID per\-example gradients with mean𝝁b,t\\boldsymbol\{\\mu\}\_\{b,t\}and second\-moment block𝐂b​b,t\\mathbf\{C\}\_\{bb,t\}\. For a minibatch mean𝐠b,t=m−1​∑i=1m𝐳b,i\\mathbf\{g\}\_\{b,t\}=m^\{\-1\}\\sum\_\{i=1\}^\{m\}\\mathbf\{z\}\_\{b,i\},

𝔼​‖𝐠b,t‖22\\displaystyle\\mathbb\{E\}\\\|\\mathbf\{g\}\_\{b,t\}\\\|\_\{2\}^\{2\}=1m2​𝔼​‖∑i=1m𝐳b,i‖22\\displaystyle=\\frac\{1\}\{m^\{2\}\}\\mathbb\{E\}\\left\\\|\\sum\_\{i=1\}^\{m\}\\mathbf\{z\}\_\{b,i\}\\right\\\|\_\{2\}^\{2\}\(35\)=1m2​\(m​𝔼​‖𝐳b,1‖22\+m​\(m−1\)​‖𝝁b,t‖22\)\\displaystyle=\\frac\{1\}\{m^\{2\}\}\\left\(m\\,\\mathbb\{E\}\\\|\\mathbf\{z\}\_\{b,1\}\\\|\_\{2\}^\{2\}\+m\(m\-1\)\\\|\\boldsymbol\{\\mu\}\_\{b,t\}\\\|\_\{2\}^\{2\}\\right\)=1m​Tr⁡\(𝐂b​b,t\)\+\(1−1m\)​‖𝝁b,t‖22\.\\displaystyle=\\frac\{1\}\{m\}\\operatorname\{Tr\}\(\\mathbf\{C\}\_\{bb,t\}\)\+\\left\(1\-\\frac\{1\}\{m\}\\right\)\\\|\\boldsymbol\{\\mu\}\_\{b,t\}\\\|\_\{2\}^\{2\}\.This proves Eq\. \([15](https://arxiv.org/html/2607.18306#S3.E15)\)\. For negative log\-likelihoods,𝐂t\\mathbf\{C\}\_\{t\}is the Fisher only when its expectation is taken under the model distribution\. Under the empirical data distribution it is a gradient second moment, and neither it nor the minibatch quantity above is generally equal to the Hessian\. Near stationarity, the mean\-gradient term is small and the second\-moment trace dominates up to the common factor1/m1/m\.

### EMA Expectation and Variance

Unrolling Eq\. \([17](https://arxiv.org/html/2607.18306#S3.E17)\) withsb,0=0s\_\{b,0\}=0gives

sb,t=\(1−β\)​∑k=0t−1βk​eb,t−k\.s\_\{b,t\}=\(1\-\\beta\)\\sum\_\{k=0\}^\{t\-1\}\\beta^\{k\}e\_\{b,t\-k\}\.\(36\)Under local stationarity,𝔼​\[eb,t−k\]=e¯b\\mathbb\{E\}\[e\_\{b,t\-k\}\]=\\bar\{e\}\_\{b\}throughout this window\. Therefore,

𝔼​\[sb,t\]=\(1−β\)​e¯b​∑k=0t−1βk=\(1−βt\)​e¯b,\\mathbb\{E\}\[s\_\{b,t\}\]=\(1\-\\beta\)\\bar\{e\}\_\{b\}\\sum\_\{k=0\}^\{t\-1\}\\beta^\{k\}=\(1\-\\beta^\{t\}\)\\bar\{e\}\_\{b\},\(37\)which proves Eq\. \([18](https://arxiv.org/html/2607.18306#S3.E18)\)\. If the centered stochastic energies are additionally independent over iterations with varianceσb2\\sigma\_\{b\}^\{2\}, then

Var⁡\(sb,t\)\\displaystyle\\operatorname\{Var\}\(s\_\{b,t\}\)=\(1−β\)2​σb2​∑k=0t−1β2​k\\displaystyle=\(1\-\\beta\)^\{2\}\\sigma\_\{b\}^\{2\}\\sum\_\{k=0\}^\{t\-1\}\\beta^\{2k\}\(38\)=1−β1\+β​\(1−β2​t\)​σb2\.\\displaystyle=\\frac\{1\-\\beta\}\{1\+\\beta\}\(1\-\\beta^\{2t\}\)\\sigma\_\{b\}^\{2\}\.Thus, under this idealized independent\-noise model, the asymptotic variance is reduced by\(1−β\)/\(1\+β\)\(1\-\\beta\)/\(1\+\\beta\)relative to an instantaneous energy\. The formula is an estimator property under the stated assumptions, not a claim that optimization gradients are exactly stationary or independent\.

### Closed\-form Allocation and Budget Preservation

Let𝐬t≠𝟎\\mathbf\{s\}\_\{t\}\\neq\\mathbf\{0\}be nonnegative\. For every feasible radius vector𝐫t\\mathbf\{r\}\_\{t\}, Cauchy–Schwarz gives

𝐬t⊤​𝐫t≤‖𝐬t‖2​‖𝐫t‖2≤ρ​‖𝐬t‖2\.\\mathbf\{s\}\_\{t\}^\{\\top\}\\mathbf\{r\}\_\{t\}\\leq\\\|\\mathbf\{s\}\_\{t\}\\\|\_\{2\}\\\|\\mathbf\{r\}\_\{t\}\\\|\_\{2\}\\leq\\rho\\\|\\mathbf\{s\}\_\{t\}\\\|\_\{2\}\.\(39\)Equality in both inequalities is attained uniquely at

𝐫t⋆=ρ​𝐬t‖𝐬t‖2,\\mathbf\{r\}\_\{t\}^\{\\star\}=\\rho\\frac\{\\mathbf\{s\}\_\{t\}\}\{\\\|\\mathbf\{s\}\_\{t\}\\\|\_\{2\}\},\(40\)which is nonnegative and proves Proposition 1\. It also gives

∑b=1B\(rb,t⋆\)2=ρ2\.\\sum\_\{b=1\}^\{B\}\(r\_\{b,t\}^\{\\star\}\)^\{2\}=\\rho^\{2\}\.\(41\)For exact unit block directions, the concatenated perturbation therefore has normρ\\rho\. With the stabilized directions used in Algorithm[1](https://arxiv.org/html/2607.18306#alg1), each direction has norm at most one, so the realized perturbation instead satisfies‖ϵt‖2≤ρ\\\|\\boldsymbol\{\\epsilon\}\_\{t\}\\\|\_\{2\}\\leq\\rho\. A common positive rescaling of all sensitivity scores leaves𝐫t⋆\\mathbf\{r\}\_\{t\}^\{\\star\}unchanged\. If𝐬t=𝟎\\mathbf\{s\}\_\{t\}=\\mathbf\{0\}, the alignment objective is identically zero over the feasible set\. In this degenerate case, Algorithm[1](https://arxiv.org/html/2607.18306#alg1)sets all radii to zero, which is feasible and avoids injecting an arbitrary perturbation before any block\-sensitivity signal is available\.

### Coarse Block Grouping

Let a block containKKtensors with energy estimatese1,t,…,eK,te\_\{1,t\},\\ldots,e\_\{K,t\}\. Suppose each tensor\-level estimate can be decomposed asei,t=μi\+ξi,te\_\{i,t\}=\\mu\_\{i\}\+\\xi\_\{i,t\}, whereξi,t\\xi\_\{i,t\}is zero\-mean minibatch noise\. Tensor\-wise grouping assigns a separate radius to every noisy estimate\. In contrast, coarse block grouping uses the aggregate

eb,t=∑i=1Kei,t=∑i=1Kμi\+∑i=1Kξi,t\.e\_\{b,t\}=\\sum\_\{i=1\}^\{K\}e\_\{i,t\}=\\sum\_\{i=1\}^\{K\}\\mu\_\{i\}\+\\sum\_\{i=1\}^\{K\}\\xi\_\{i,t\}\.\(42\)If the noise terms are weakly correlated, the relative fluctuation of the aggregate score is smaller than that of individual tensor scores\. For example, with independent noise of varianceσ2\\sigma^\{2\}and comparable meansμ\\mu, the coefficient of variation changes fromσ/μ\\sigma/\\mufor a tensor\-level score toσ/\(K​μ\)\\sigma/\(\\sqrt\{K\}\\mu\)for the aggregated block score\. Coarse grouping therefore improves the stability of online allocation\.

Coarse grouping also preserves the joint perturbation direction of tensors that operate at the same representation scale\. Tensor\-wise grouping can normalize each tensor independently and fragment a functional block into many small allocation decisions\. Coarse block grouping avoids this fragmentation while still distinguishing parts of the network with different spatial resolution, channel width, and optimization roles\. This provides the theoretical rationale for the architecture\-aware partition used in GEAR\-SAM\.

### Structured Partitioning Analysis

We further investigate alternative block partitions\. Tensor\-wise grouping assigns an independent sensitivity score and radius to every weight or bias tensor\. Our default coarse block\-wise grouping assigns one score to each architecture\-level functional block, such aslayer1in ResNet\-18\. Fine block\-wise grouping assigns a separate score to every residual block, while keeping the stem and classifier as separate blocks\.

As shown in Table[VI](https://arxiv.org/html/2607.18306#Sx1.T6), all structured partitions clearly outperform standard SAM\. The default coarse block\-wise GEAR\-SAM improves SAM from 81\.08 to 81\.83 on ResNet\-18 and from 84\.71 to 85\.49 on WideResNet\-28\-10\. Fine block\-wise grouping performs slightly better still, reaching 81\.84 and 85\.60, respectively\. These results show that the benefit of GEAR\-SAM is not tied to a single manually selected partition\. More broadly, preserving functional units while avoiding excessively small groups appears important for reliable online allocation\. Tensor\-wise grouping also improves SAM, but its larger number of independent allocation variables can introduce noisier sensitivity estimates and excessive allocation freedom\. The small advantage of fine blocks suggests that the most effective granularity can depend on the architecture; learning an adaptive block partition is an interesting direction for future work\.

TABLE VI:Comparison of partition strategies on CIFAR\-100\.
### Full Block\-Wise Perturbation Dynamics

For completeness, Fig\.[3](https://arxiv.org/html/2607.18306#Sx1.F3)reports the normalized budget shares of all six coarse blocks over the full 200\-epoch training run\. The trajectories provide the long\-horizon context for the early redistribution shown in Fig\.[1](https://arxiv.org/html/2607.18306#S3.F1): the allocation is not a uniform rescaling of SAM, but a redistribution of the same global perturbation budget\.

![Refer to caption](https://arxiv.org/html/2607.18306v1/x5.png)Figure 3:Full block\-wise perturbation dynamics of SAM and GEAR\-SAM on ResNet\-18 trained on CIFAR\-100\. Each panel shows the normalized perturbation\-budget shareρb,t2/ρ2\\rho\_\{b,t\}^\{2\}/\\rho^\{2\}of one network block\. The gray region denotes the final 20 epochs, and each annotation reports the corresponding SAM and GEAR\-SAM averages\. All panels use the same vertical scale\.