Mini-batch Noise Lowers Sharpness via Dominant-Subspace Fluctuations

# Mini-batch Noise Lowers Sharpness via Dominant-Subspace Fluctuations
Source: [https://arxiv.org/html/2607.23012](https://arxiv.org/html/2607.23012)
\\hldauthor\\Name

Junho So\\Emailjhso@ajou\.ac\.kr \\NameDongwook Shin\\Emaildws@ajou\.ac\.kr \\addrDepartment of Mathematics, Ajou University, South Korea

###### Abstract

During SGD training, the gradients often align strongly with the dominant subspace spanned by the top\-kkeigenvectors of the Hessian of the loss\. While this seems to naturally imply that loss reduction mainly occurs within this space, prior work has shown that updates within this dominant subspace make no meaningful progress in reducing the loss\. In this work, we argue that the dominant subspace is better understood not as the main space for loss reduction, but as a key subspace for explaining the sharpness dynamics of mini\-batch SGD\. To explain the role of the dominant subspace in reducing top\-kksharpness, we show how the averaged gradient over fluctuations in the dominant directions produces a sharpness correction term, and derive a sharpness correction term induced by mini\-batch noise in the dominant directions\. Experimental results show that adding the derived correction term to GD brings the sharpness evolution of GD closer to that of SGD\.

## 1Introduction

Understanding the dynamics of training deep neural networks is one of the main topics of machine learning\. During stochastic gradient descent \(SGD\) training, it has been observed that the Hessian of the loss often has a small number of large outlier eigenvalues\(sagun2017eigenvalues;DBLP:journals/corr/SagunEGDB17;pmlr\-v97\-ghorbani19b;pmlr\-v97\-papyan19a;JMLR:v21:20\-933\), and training gradients are known to align strongly with thedominant subspace, which is spanned by the top\-kkHessian eigenvectors\(gurari2018gradientdescenthappenstiny;pmlr\-v97\-ghorbani19b\)\. This suggests that the effective training dynamics of SGD may be low\-dimensional, and it naturally leads to the expectation that loss reduction mainly occurs within this space\.

However, prior work\(song2025does\)reports results that contradict this interpretation\. Specifically, when the SGD update is projected onto the dominant subspace, training achieves no meaningful loss reduction\. In contrast, when the update is projected onto thebulk subspace, which is the orthogonal complement of the dominant subspace, the loss decreases similarly to standard SGD\. Does the dominant subspace therefore contribute nothing to training?

In this paper, we argue that the main role of the dominant subspace lies in reducing the top\-kksharpness of the loss landscape\. Specifically, we observe that the dominant\-projected update significantly reduces sharpness even though it has little impact on loss reduction\. Then, through experiments involving controlled perturbations along different subspaces \(dominantandrandom\), we show that only perturbations in the dominant directions reduce sharpness\. To explain this effect, we first show that averaging the gradient over fluctuations in the dominant directions yields a deterministic sharpness correction term\. We then derive the deterministic correction term induced by mini\-batch noise in the dominant directions\. Finally, we empirically show that adding this correction term to GD produces sharpness dynamics similar to SGD\.

## 2Setup and Motivation

![Refer to caption](https://arxiv.org/html/2607.23012v1/x1.png)

![Refer to caption](https://arxiv.org/html/2607.23012v1/x2.png)

![Refer to caption](https://arxiv.org/html/2607.23012v1/x3.png)

Figure 1:The dominant component does not reduce loss but reduces sharpness\.\(a\) Training gradients align with the dominant subspace\. \(b\) Dom\-SGD fails to reduce loss, while Bulk\-SGD continues to train\. \(c\) Dom\-SGD lowersSkS\_\{k\}, whereas Bulk\-SGD maintains higherSkS\_\{k\}than SGD\.#### Setup\.

LetL:ℝd→ℝL:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}be the loss function\. For a mini\-batchBBwith mini\-batch lossLBL\_\{B\}, define the mini\-batch noise asξB​\(θ\):=∇LB​\(θ\)−∇L​\(θ\)\\xi\_\{B\}\(\\theta\):=\\nabla L\_\{B\}\(\\theta\)\-\\nabla L\(\\theta\), where𝔼B​\[ξB​\(θ\)∣θ\]=0\\mathbb\{E\}\_\{B\}\[\\xi\_\{B\}\(\\theta\)\\mid\\theta\]=0\. We write mini\-batch SGD asθt\+1=θt−η​\(∇L​\(θt\)\+ξBt​\(θt\)\)\\theta\_\{t\+1\}=\\theta\_\{t\}\-\\eta\(\\nabla L\(\\theta\_\{t\}\)\+\\xi\_\{B\_\{t\}\}\(\\theta\_\{t\}\)\), and full\-batch GD asθt\+1=θt−η​∇L​\(θt\)\\theta\_\{t\+1\}=\\theta\_\{t\}\-\\eta\\nabla L\(\\theta\_\{t\}\)\. Let\(λi​\(θ\),ei​\(θ\)\)\(\\lambda\_\{i\}\(\\theta\),e\_\{i\}\(\\theta\)\)be the eigenpairs ofH​\(θ\)=∇2L​\(θ\)H\(\\theta\)=\\nabla^\{2\}L\(\\theta\), ordered so thatλ1​\(θ\)≥⋯≥λd​\(θ\)\\lambda\_\{1\}\(\\theta\)\\geq\\cdots\\geq\\lambda\_\{d\}\(\\theta\)\. We define thedominant subspaceas the space spanned by the top\-kkeigenvectors,Edom​\(θ\)=span​\{e1​\(θ\),…,ek​\(θ\)\}E\_\{\\mathrm\{dom\}\}\(\\theta\)=\\mathrm\{span\}\\\{e\_\{1\}\(\\theta\),\\ldots,e\_\{k\}\(\\theta\)\\\}, and define its orthogonal complementEbulk​\(θ\)=Edom​\(θ\)⟂E\_\{\\mathrm\{bulk\}\}\(\\theta\)=E\_\{\\mathrm\{dom\}\}\(\\theta\)^\{\\perp\}as thebulk subspace\. We also define the projections onto these spaces as the dominant projectionPdom​\(θ\)P\_\{\\mathrm\{dom\}\}\(\\theta\)and the bulk projectionPbulk​\(θ\)=I−Pdom​\(θ\)P\_\{\\mathrm\{bulk\}\}\(\\theta\)=I\-P\_\{\\mathrm\{dom\}\}\(\\theta\)\. We define top\-kksharpness asSk​\(θ\)=∑j=1kλj​\(θ\)S\_\{k\}\(\\theta\)=\\sum\_\{j=1\}^\{k\}\\lambda\_\{j\}\(\\theta\)\. We restrict our focus to the stable learning\-rate regime, whereλ1​\(θ\)<2/η\\lambda\_\{1\}\(\\theta\)<2/\\etais maintained during training\.

### 2\.1Motivation and Main Observation

Let us define the dominant alignment metricχk​\(θ\)=‖Pdom​\(θ\)​∇L​\(θ\)‖/‖∇L​\(θ\)‖\\chi\_\{k\}\(\\theta\)=\\\|P\_\{\\mathrm\{dom\}\}\(\\theta\)\\nabla L\(\\theta\)\\\|/\\\|\\nabla L\(\\theta\)\\\|, which quantifies the relative magnitude of the training\-loss gradient lying in the dominant subspace along the SGD trajectory\. As shown in Figure[1](https://arxiv.org/html/2607.23012#S2.F1)\(a\), the SGD training gradient is strongly concentrated in the dominant subspace from early in training, and this tendency becomes stronger at later stages\(gurari2018gradientdescenthappenstiny;pmlr\-v97\-ghorbani19b\)\. This observation suggests that SGD dynamics are closely related to the dominant subspace\.

![Refer to caption](https://arxiv.org/html/2607.23012v1/x4.png)Figure 2:River\-valley intuition\.However, as shown in Figure[1](https://arxiv.org/html/2607.23012#S2.F1)\(b\), Dom\-SGD, which projects the SGD update onto the dominant subspace, does not substantially reduce the training loss\. Instead, Bulk\-SGD, which keeps only the bulk component, reduces the loss as effectively as SGD\(song2025does\)\. Prior work\(song2025does;wen2025understanding\)explains this phenomenon using the river\-valley intuition\. In this view, as shown in Figure[2](https://arxiv.org/html/2607.23012#S2.F2), the dominant directions correspond to the high\-curvature valley walls, where updates do not reduce the loss, whereas the bulk directions follow the valley floor, where loss reduction mainly occurs\.

Nevertheless, prior work has reported that models trained with Bulk\-SGD alone fail to fully recover the final accuracy gains achieved by SGD\(zakarin2025acceleratingneuralnetworktraining\)\. This suggests that the dominant subspace, while not primarily responsible for loss reduction, may still play a meaningful role in training\. Its contribution may instead lie in aspects of the optimization dynamics that are not directly captured by loss reduction alone\.

In this work, we argue that the dominant subspace plays an important role in reducing the top\-kksharpness of the loss landscape\. As shown in Figure[1](https://arxiv.org/html/2607.23012#S2.F1)\(c\), Bulk\-SGD follows almost the same loss curve as SGD, but its top\-kksharpness becomes higher than that of SGD\. In contrast, Dom\-SGD keeps the loss high while substantially reducing top\-kksharpness, suggesting that the dominant subspace is connected to sharpness reduction rather than loss reduction\. In the next section, we present experiments examining whether motion along the dominant directions is in fact the factor that reduces top\-kksharpness\.

## 3Dominant\-Subspace Fluctuations Reduce Sharpness

![Refer to caption](https://arxiv.org/html/2607.23012v1/x5.png)

![Refer to caption](https://arxiv.org/html/2607.23012v1/x6.png)

![Refer to caption](https://arxiv.org/html/2607.23012v1/x7.png)

Figure 3:Dominant perturbations reduce top\-kksharpness\.We compare GD, GD with dominant mean\-zero perturbations \(GD\+Dom\), and GD with random mean\-zero perturbations \(GD\+Rand\)\. The two perturbation runs are matched in rank and noise scale\(k=10,ρ=0\.1\)\(k=10,\\rho=0\.1\)\. \(a\) Dominant perturbations induce strong dominant alignment\. \(b\) Only dominant perturbations reduceSkS\_\{k\}\. \(c\) Both perturbed runs leave the GD loss curve nearly unchanged\. All experiments are run on an MLP\.The previous observation showed that the dominant subspace is related to sharpness reduction rather than loss reduction\. However, this simple comparison between Dom\-SGD and Bulk\-SGD does not reveal whether this sharpness reduction comes from thegradient component in the dominant subspaceor fromstochastic motion within that subspace\.

To separate these effects, we conduct a controlled noise injection experiment that keeps the mean update equal to full\-batch GD while adding only mean\-zero perturbations in different subspaces\. Specifically, we compare the perturbed GD dynamicsθt\+1=θt−η​\(∇L​\(θt\)\+ζt\)\\theta\_\{t\+1\}=\\theta\_\{t\}\-\\eta\(\\nabla L\(\\theta\_\{t\}\)\+\\zeta\_\{t\}\)withζt\\zeta\_\{t\}sampled from different subspaces\. For the dominant perturbation, we setζt∼𝒩​\(0,ρ2​Pdom​\(θt\)\)\\zeta\_\{t\}\\sim\\mathcal\{N\}\(0,\\rho^\{2\}P\_\{\\mathrm\{dom\}\}\(\\theta\_\{t\}\)\), and for the random perturbation, we setζt∼𝒩​\(0,ρ2​Prand\)\\zeta\_\{t\}\\sim\\mathcal\{N\}\(0,\\rho^\{2\}P\_\{\\mathrm\{rand\}\}\), wherePrandP\_\{\\mathrm\{rand\}\}is a randomkk\-dimensional orthogonal projection fixed at initialization\.

First, as shown in Figure[3](https://arxiv.org/html/2607.23012#S3.F3)\(c\), GD, GD with random perturbations \(GD\+Rand\), and GD with dominant perturbations \(GD\+Dom\) follow almost the same training\-loss curve, indicating that these perturbations do not significantly alter the rate of loss reduction\. However, as shown in Figure[3](https://arxiv.org/html/2607.23012#S3.F3)\(a\) and[3](https://arxiv.org/html/2607.23012#S3.F3)\(b\), their trajectories exhibit substantial geometric differences: GD and GD\+Rand maintain lowχk\\chi\_\{k\}and highSkS\_\{k\}, whereas GD\+Dom drivesχk\\chi\_\{k\}close to 1 and maintains a lower top\-kksharpnessSkS\_\{k\}than either GD or GD\+Rand\.

These results show two things\. First, as suggested by the previous observation, alignment with the dominant subspace does not imply that loss reduction occurs along the dominant directions\. Second, and more importantly, stochastic motion alone within the dominant subspace can actually affect sharpness dynamics while barely changing the loss curve\. Together, these results indicate that the role of the dominant subspace lies not in providing a direction for loss reduction, but rather in changing thelocal geometry associated with the dominant directions, especially top\-kksharpness\. In the next section, we show that this effect arises from the interaction between the covariance of a mean\-zero displacement and local variations of the Hessian\.

## 4Sharpness Correction Term

In this section, we show that when the parameter fluctuates in the dominant directions around a reference point, averaging the gradient leaves asharpness correction term\. To show this, we consider a displacementδ∈Edom​\(θc\)\\delta\\in E\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\)from a reference pointθc\\theta\_\{c\}, with𝔼​\[δ\]=0\\mathbb\{E\}\[\\delta\]=0, and write the nearby parameter asθc\+δ\\theta\_\{c\}\+\\delta\. Expanding∇L​\(θc\+δ\)\\nabla L\(\\theta\_\{c\}\+\\delta\)using a Taylor expansion aroundθc\\theta\_\{c\}, we get:

∇L​\(θc\+δ\)=∇L​\(θc\)\+H​\(θc\)​δ\+12​∇\(δ⊤​H​δ\)⁡\(θc\)\+O​\(‖δ‖3\)\.\\nabla L\(\\theta\_\{c\}\+\\delta\)=\\nabla L\(\\theta\_\{c\}\)\+H\(\\theta\_\{c\}\)\\delta\+\\frac\{1\}\{2\}\\nabla\(\\delta^\{\\top\}H\\delta\)\(\\theta\_\{c\}\)\+O\(\\\|\\delta\\\|^\{3\}\)\.\(1\)The first term in Eq\. \([1](https://arxiv.org/html/2607.23012#S4.E1)\) is the gradient at the reference point, the second term is the linear term in the displacement, and the third term is a second\-order term that captures the local variation of the Hessian\. Averaging this overδ\\deltagives:

𝔼δ​\[∇L​\(θc\+δ\)\]=∇L​\(θc\)\+H​\(θc\)​𝔼​\[δ\]⏟=0\+12​∇Tr⁡\(H​\(θc\)​C\)\+O​\(𝔼​‖δ‖3\),C:=𝔼​\[δ​δ⊤\]\.\\mathbb\{E\}\_\{\\delta\}\[\\nabla L\(\\theta\_\{c\}\+\\delta\)\]=\\nabla L\(\\theta\_\{c\}\)\+\\underbrace\{H\(\\theta\_\{c\}\)\\mathbb\{E\}\[\\delta\]\}\_\{=\\,0\}\+\\frac\{1\}\{2\}\\nabla\\operatorname\{Tr\}\\\!\\left\(H\(\\theta\_\{c\}\)C\\right\)\+O\(\\mathbb\{E\}\\\|\\delta\\\|^\{3\}\),\\qquad C:=\\mathbb\{E\}\[\\delta\\delta^\{\\top\}\]\.\(2\)Here,CCis held fixed when taking the derivative\. As shown in Eq\. \([2](https://arxiv.org/html/2607.23012#S4.E2)\), the averaged gradient is the sum of the gradient at the reference point and an additional term12​∇Tr⁡\(H​\(θc\)​C\)\\frac\{1\}\{2\}\\nabla\\operatorname\{Tr\}\\\!\\left\(H\(\\theta\_\{c\}\)C\\right\)\. This term is determined by the displacement covarianceCCand acts to lower the covariance\-weighted curvature represented byTr⁡\(H​C\)\\operatorname\{Tr\}\(HC\)\. WhenCCis supported in the dominant subspace, it lowers the weighted curvature associated with top\-kksharpness, so we refer to it as asharpness correction term\.

#### Mini\-batch\-induced displacement covariance\.

Finally, we derive the displacement covariance induced by mini\-batch noise in the local recursion and substitute it into the sharpness correction term\. This shows that the sharpness effect of mini\-batch noise can be represented as a deterministic correction term\.

To define the local displacement created by mini\-batch noise around a reference pointθc\\theta\_\{c\}, we compare the mini\-batch SGD update with the full\-batch GD update\. Consider a local stateθc\+δs\\theta\_\{c\}\+\\delta\_\{s\}, whereδs∈Edom​\(θc\)\\delta\_\{s\}\\in E\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\), and letPc:=Pdom​\(θc\)P\_\{c\}:=P\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\)\. The next displacementδs\+1\\delta\_\{s\+1\}is defined as

δs\+1:=Pc\(θmb\+−θgd\+\),θmb\+:=θc\+δs−η∇LBs\(θc\+δs\),θgd\+:=θc−η∇L\(θc\)\.\\delta\_\{s\+1\}:=P\_\{c\}\(\\theta\_\{\\mathrm\{mb\}\}^\{\+\}\-\\theta\_\{\\mathrm\{gd\}\}^\{\+\}\),\\qquad\\theta\_\{\\mathrm\{mb\}\}^\{\+\}:=\\theta\_\{c\}\+\\delta\_\{s\}\-\\eta\\nabla L\_\{B\_\{s\}\}\(\\theta\_\{c\}\+\\delta\_\{s\}\),\\qquad\\theta\_\{\\mathrm\{gd\}\}^\{\+\}:=\\theta\_\{c\}\-\\eta\\nabla L\(\\theta\_\{c\}\)\.We then linearize the mini\-batch gradient atθc\+δs\\theta\_\{c\}\+\\delta\_\{s\}aroundθc\\theta\_\{c\}:

∇LBs​\(θc\+δs\)≈∇L​\(θc\)\+Hc​δs\+ξBs​\(θc\),Hc:=H​\(θc\)\.\\nabla L\_\{B\_\{s\}\}\(\\theta\_\{c\}\+\\delta\_\{s\}\)\\approx\\nabla L\(\\theta\_\{c\}\)\+H\_\{c\}\\delta\_\{s\}\+\\xi\_\{B\_\{s\}\}\(\\theta\_\{c\}\),\\qquad H\_\{c\}:=H\(\\theta\_\{c\}\)\.Substituting this into the definition ofδs\+1\\delta\_\{s\+1\}gives

δs\+1=Pc​\(δs−η​Hc​δs−η​ξBs​\(θc\)\)=Ac​δs−η​Pc​ξBs​\(θc\),Ac:=Pc​\(I−η​Hc\)​Pc\.\\delta\_\{s\+1\}=P\_\{c\}\\bigl\(\\delta\_\{s\}\-\\eta H\_\{c\}\\delta\_\{s\}\-\\eta\\xi\_\{B\_\{s\}\}\(\\theta\_\{c\}\)\\bigr\)=A\_\{c\}\\delta\_\{s\}\-\\eta P\_\{c\}\\xi\_\{B\_\{s\}\}\(\\theta\_\{c\}\),\\qquad A\_\{c\}:=P\_\{c\}\(I\-\\eta H\_\{c\}\)P\_\{c\}\.\(3\)Taking second moments in Eq\. \([3](https://arxiv.org/html/2607.23012#S4.E3)\) gives

Cs\+1=Ac​Cs​Ac⊤\+η2​Σdom​\(θc\),\(Cs:=𝔼​\[δs​δs⊤\]\),\(Σdom​\(θc\):=Pc​Cov⁡\(ξB​\(θc\)\)​Pc\)\.C\_\{s\+1\}=A\_\{c\}C\_\{s\}A\_\{c\}^\{\\top\}\+\\eta^\{2\}\\Sigma\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\),\\qquad\(C\_\{s\}:=\\mathbb\{E\}\[\\delta\_\{s\}\\delta\_\{s\}^\{\\top\}\]\),\\quad\(\\Sigma\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\):=P\_\{c\}\\operatorname\{Cov\}\(\\xi\_\{B\}\(\\theta\_\{c\}\)\)P\_\{c\}\)\.Here,Σdom\\Sigma\_\{\\mathrm\{dom\}\}denotes the mini\-batch noise covariance projected onto the dominant subspace\. LetCmbC\_\{\\mathrm\{mb\}\}be the stationary limit of the covariance sequenceCsC\_\{s\}\. ThenCmb=Ac​Cmb​Ac⊤\+η2​ΣdomC\_\{\\mathrm\{mb\}\}=A\_\{c\}C\_\{\\mathrm\{mb\}\}A\_\{c\}^\{\\top\}\+\\eta^\{2\}\\Sigma\_\{\\mathrm\{dom\}\}\. Therefore, substitutingCmbC\_\{\\mathrm\{mb\}\}into Eq\. \([2](https://arxiv.org/html/2607.23012#S4.E2)\) gives the following:

𝔼​\[∇L​\(θc\+δ\)\]≈∇L​\(θc\)\+12​∇Tr⁡\(H​\(θc\)​Cmb\),Cmb=η2​∑ℓ=0∞Acℓ​Σdom​\(Ac⊤\)ℓ\.\\displaystyle\\mathbb\{E\}\[\\nabla L\(\\theta\_\{c\}\+\\delta\)\]\\approx\\nabla L\(\\theta\_\{c\}\)\+\\frac\{1\}\{2\}\\nabla\\operatorname\{Tr\}\\\!\\left\(H\(\\theta\_\{c\}\)C\_\{\\mathrm\{mb\}\}\\right\),\\qquad C\_\{\\mathrm\{mb\}\}=\\eta^\{2\}\\sum\_\{\\ell=0\}^\{\\infty\}A\_\{c\}^\{\\ell\}\\Sigma\_\{\\mathrm\{dom\}\}\(A\_\{c\}^\{\\top\}\)^\{\\ell\}\.
Thus,CmbC\_\{\\mathrm\{mb\}\}is not merely the mini\-batch noise covariance, but the displacement covariance produced by the local recursion, and therefore it also depends on the local Hessian structure and the learning rate\. With this covariance, the effect of mini\-batch noise on the averaged gradient is represented as a deterministic correction term \(see Appendix[C](https://arxiv.org/html/2607.23012#A3)for the detailed derivation\)\.

## 5Experiments

![Refer to caption](https://arxiv.org/html/2607.23012v1/x8.png)

![Refer to caption](https://arxiv.org/html/2607.23012v1/x9.png)

![Refer to caption](https://arxiv.org/html/2607.23012v1/x10.png)

Figure 4:The sharpness correction term brings GD closer to mini\-batch SGD in top\-kksharpness\.Across MLP, CNN, and Transformer, full\-batch GD reaches higher top\-kksharpnessSkS\_\{k\}, whereas adding the derived sharpness correction term to GD brings its top\-kksharpness curve closer to that of mini\-batch SGD\.In this section, we empirically test whether the sharpness correction term derived in Section[4](https://arxiv.org/html/2607.23012#S4)reproduces the low sharpness of mini\-batch SGD\. For the experiments, we compare the sharpness dynamics of full\-batch GD, mini\-batch SGD, and corrected GD across three model architectures: MLP, CNN, and Transformer\(vaswani2017attention\)\. Here, corrected GD is obtained by adding our sharpness correction term to GD,θt\+1=θt−η​∇L​\(θt\)\+η​bcorr​\(θt\)\\theta\_\{t\+1\}=\\theta\_\{t\}\-\\eta\\nabla L\(\\theta\_\{t\}\)\+\\eta b\_\{\\mathrm\{corr\}\}\(\\theta\_\{t\}\), wherebcorr​\(θ\):=−12​∇ϑTr⁡\(H​\(ϑ\)​Cmb​\(θ\)\)\|ϑ=θb\_\{\\mathrm\{corr\}\}\(\\theta\):=\-\\frac\{1\}\{2\}\\nabla\_\{\\vartheta\}\\operatorname\{Tr\}\\\!\\left\(H\(\\vartheta\)C\_\{\\mathrm\{mb\}\}\(\\theta\)\\right\)\\big\|\_\{\\vartheta=\\theta\}\. Detailed experimental settings are given in Appendix[D](https://arxiv.org/html/2607.23012#A4)\.

Figure[4](https://arxiv.org/html/2607.23012#S5.F4)shows that full\-batch GD moves toward higherSkS\_\{k\}than mini\-batch SGD in all three models, whereas corrected GD stays much closer to the low sharpness level of SGD\. In particular, for the MLP, corrected GD almost overlaps with SGD, and for the CNN and Transformer, it also keepsSkS\_\{k\}lower than GD and substantially reduces thesharpness gapwith SGD\.

This result shows that the sharpness correction term we derived in Section[4](https://arxiv.org/html/2607.23012#S4)explains much of the sharpness dynamics of SGD, empirically showing that the dominant subspace helps explain the lower top\-kksharpness of mini\-batch SGD, rather than directly providing a direction for reducing the loss\.

## 6Conclusion

Prior work bysong2025doesshows that the dominant subspace, the top\-kkeigenspace of the loss Hessian, does not provide meaningful directions for loss reduction, leaving its explanatory role unclear\. In this paper, we show that this dominant subspace plays an important role in reducing top\-kksharpness under SGD\. Specifically, we show that perturbations within the dominant subspace reduce top\-kksharpness, and derive the sharpness correction term induced by mini\-batch noise\. Empirically, we show that adding this correction term to GD brings its sharpness evolution closer to that of mini\-batch SGD, demonstrating that the dominant subspace matters not as a direction for loss reduction, but as the subspace through which mini\-batch noise lowers sharpness\.

## References

## Appendix ARelated Work

#### Hessian spectra and dominant alignment\.

The training\-loss Hessian of a neural network often has a structure that separates into a small number of large outlier eigenvalues and a broad bulk spectrum\(sagun2017eigenvalues;DBLP:journals/corr/SagunEGDB17;pmlr\-v97\-ghorbani19b;pmlr\-v97\-papyan19a;JMLR:v21:20\-933\)\. It has also been repeatedly observed that, during training, the training\-loss gradient is strongly aligned with a low\-dimensional subspace spanned by a small number of eigenvectors corresponding to the large eigenvalues of the Hessian\(gurari2018gradientdescenthappenstiny;pmlr\-v97\-ghorbani19b\)\. These observations suggest that SGD dynamics in a high\-dimensional parameter space are closely related to low\-dimensional structure\.

#### Projected\-update experiments\.

song2025doesconducted projected\-update experiments to test whether SGD can still reduce the training loss when its updates are restricted to the top Hessian eigenspace, referred to as the dominant subspace\. Surprisingly, Dom\-SGD, which trains using only the SGD update projected onto the dominant subspace, barely reduces the training loss\. In contrast, Bulk\-SGD, which trains using the SGD update with its dominant component removed, continues to reduce the loss similarly to standard SGD\. To address this phenomenon,song2025doesinterpret the loss landscape as an ill\-conditioned valley\. In this view, the dominant directions correspond to high\-curvature valley walls, so Dom\-SGD mostly moves across the walls and has difficulty reducing the loss, whereas Bulk\-SGD follows the valley floor and continues to train\. This intuition is also similar to the river\-valley loss landscape\(wen2025understanding\)\.deng2026suspiciousalignmentsgdfinegrainedfurther analyze projected SGD updates by deriving step\-size conditions for loss decrease\. However, these loss\-based explanations account for why a dominant\-projected update has difficulty making loss progress, but not why the dominant subspace is connected to lower top\-kksharpness\. This question is the starting point of our work\.

#### Sharpness and flatness\.

Sharpness and flatness have long been studied as loss\-landscape geometry for explaining generalization\. Early work suggested that flat minima, where the loss does not change much under small perturbations around the solution, are associated with better generalization\(10\.1162/neco\.1997\.9\.1\.1\)\. Later, it was also observed that large\-batch training can converge to sharper minimizers than small\-batch training and show a generalization gap, with worse test performance despite similar training loss\(keskar2017on\)\. This view also led to optimization methods that directly search for flat minima\. A representative method is SAM, and since then, related variants and theoretical analyses have also been actively studied\(foret2021sharpnessaware;pmlr\-v139\-kwon21b;pmlr\-v162\-andriushchenko22a;JMLR:v24:23\-043;wen2023how\)\. On the other hand, since sharpness\-based measures can be sensitive to parameterization or experimental settings, there is also a view that sharpness alone has limits in explaining generalization\(pmlr\-v70\-dinh17b;Jiang\*2020Fantastic\)\.

#### SGD noise and implicit regularization\.

SGD noise has been studied as a source of implicit regularization related to flatness and sharpness\. Prior work has studied SGD noise through the stationary distribution of constant\-step SGD\(mandt2018stochasticgradientdescentapproximate\), or approximated SGD using SDEs or stochastic modified equations\(pmlr\-v70\-li17f;JMLR:v20:17\-526;li2021on\)\. Along with this view, other studies explored how the scale and covariance structure of SGD noise affect optimization and generalization\(jastrzębski2018three;pmlr\-v97\-zhu19e;pmlr\-v108\-thomas20a;pmlr\-v108\-wen20a\)\. In particular, some analyses suggest that this covariance structure is related to sharp directions and affects flat\-minima selection\(NEURIPS2022\_1e55c38d\)\. There is also work showing that mini\-batch SGD noise near a zero\-loss manifold induces a drift toward lower sharpness of the loss\(li2022what\)\. Like this line of work, we study how SGD noise affects sharpness, but we focus on the displacement covariance induced in the dominant subspace rather than on the mini\-batch noise covariance itself\.

#### Edge of stability\.

In large\-step\-size GD, the largest Hessian eigenvalue is observed to increase up to around the stability threshold2/η2/\\eta, after which training continues without diverging, although the loss becomes non\-monotonic\. This phenomenon is called the Edge of Stability \(EoS\)\(cohen2021gradient\)\. Subsequent theoretical analyses showed that motion caused by instability in the top Hessian direction can lower sharpness through higher\-order terms\(damian2023selfstabilizationimplicitbiasgradient\), and that the time\-averaged trajectory of large\-step GD near EoS can be described by a deterministic flow\(cohen2025understanding\)\. Our mechanism is closely related to these analyses in that motion in high\-curvature directions leaves a higher\-order correction after averaging\. However, the source of the correction is different\. While these works study oscillations that arise in large\-step GD, we focus on fluctuations induced by mini\-batch noise in SGD\.

## Appendix BNotation and Assumptions

In this section, we fix the notation and assumptions used in Appendix[C](https://arxiv.org/html/2607.23012#A3)\.

### B\.1Notation

#### Optimization setup\.

We useL:ℝd→ℝL:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}to denote the training loss\. For a mini\-batchBB, we writeLBL\_\{B\}for the corresponding mini\-batch loss\.

We write

H​\(θ\):=∇2L​\(θ\)H\(\\theta\):=\\nabla^\{2\}L\(\\theta\)for the Hessian matrix atθ\\theta\. For a mini\-batchBB, we define the mini\-batch noise by

ξB​\(θ\):=∇LB​\(θ\)−∇L​\(θ\),𝔼B​\[ξB​\(θ\)∣θ\]=0\.\\xi\_\{B\}\(\\theta\):=\\nabla L\_\{B\}\(\\theta\)\-\\nabla L\(\\theta\),\\qquad\\mathbb\{E\}\_\{B\}\[\\xi\_\{B\}\(\\theta\)\\mid\\theta\]=0\.We denote the corresponding noise covariance by

Σξ​\(θ\):=CovB⁡\(ξB​\(θ\)∣θ\)\.\\Sigma\_\{\\xi\}\(\\theta\):=\\operatorname\{Cov\}\_\{B\}\(\\xi\_\{B\}\(\\theta\)\\mid\\theta\)\.Mini\-batch SGD with learning rateη\>0\\eta\>0is written as

θt\+1=θt−η​∇LBt​\(θt\)=θt−η​\(∇L​\(θt\)\+ξBt​\(θt\)\)\.\\theta\_\{t\+1\}=\\theta\_\{t\}\-\\eta\\nabla L\_\{B\_\{t\}\}\(\\theta\_\{t\}\)=\\theta\_\{t\}\-\\eta\\bigl\(\\nabla L\(\\theta\_\{t\}\)\+\\xi\_\{B\_\{t\}\}\(\\theta\_\{t\}\)\\bigr\)\.

#### Dominant Hessian subspace\.

We denote the eigenpairs ofH​\(θ\)H\(\\theta\)by\(λi​\(θ\),ei​\(θ\)\)i=1d\(\\lambda\_\{i\}\(\\theta\),e\_\{i\}\(\\theta\)\)\_\{i=1\}^\{d\}, ordered as

λ1​\(θ\)≥λ2​\(θ\)≥⋯≥λd​\(θ\)\.\\lambda\_\{1\}\(\\theta\)\\geq\\lambda\_\{2\}\(\\theta\)\\geq\\cdots\\geq\\lambda\_\{d\}\(\\theta\)\.We define the dominant subspace as

Edom​\(θ\):=span⁡\{e1​\(θ\),…,ek​\(θ\)\},E\_\{\\mathrm\{dom\}\}\(\\theta\):=\\operatorname\{span\}\\\{e\_\{1\}\(\\theta\),\\ldots,e\_\{k\}\(\\theta\)\\\},and writePdom​\(θ\)P\_\{\\mathrm\{dom\}\}\(\\theta\)for the orthogonal projection ontoEdom​\(θ\)E\_\{\\mathrm\{dom\}\}\(\\theta\)\. We also write

Ebulk​\(θ\):=Edom​\(θ\)⟂,Pbulk​\(θ\):=I−Pdom​\(θ\)\.E\_\{\\mathrm\{bulk\}\}\(\\theta\):=E\_\{\\mathrm\{dom\}\}\(\\theta\)^\{\\perp\},\\qquad P\_\{\\mathrm\{bulk\}\}\(\\theta\):=I\-P\_\{\\mathrm\{dom\}\}\(\\theta\)\.We define the top\-kksharpness as

Sk​\(θ\):=∑i=1kλi​\(θ\)\.S\_\{k\}\(\\theta\):=\\sum\_\{i=1\}^\{k\}\\lambda\_\{i\}\(\\theta\)\.

#### Local displacements\.

For a reference pointθc\\theta\_\{c\}, we write a nearby parameter as

whereδ\\deltadenotes the local displacement\. In the local averaging calculations,δ\\deltais mean\-zero, and we write

C​\(θc\)=𝔼​\[δ​δ⊤\]C\(\\theta\_\{c\}\)=\\mathbb\{E\}\[\\delta\\delta^\{\\top\}\]for its displacement covariance\. When the displacement is supported onEdom​\(θc\)E\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\), the covarianceCCis supported on the same subspace\.

#### Local linear map\.

For a reference pointθc\\theta\_\{c\}, we define

Ac:=Pdom​\(θc\)​\(I−η​H​\(θc\)\)​Pdom​\(θc\),A\_\{c\}:=P\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\)\\bigl\(I\-\\eta H\(\\theta\_\{c\}\)\\bigr\)P\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\),viewed as a linear map onEdom​\(θc\)E\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\)\. This is the linear part of the local recursion derived in Appendix[C\.3](https://arxiv.org/html/2607.23012#A3.SS3)\.

### B\.2Local Assumptions

We use the following local assumptions at each reference pointθc\\theta\_\{c\}\.

#### Assumption 1 \(Local Taylor regularity\)\.

Aroundθc\\theta\_\{c\}, the lossLLis locallyC4C^\{4\}\. This is used to apply the Taylor expansion of the gradient and to control the local remainder terms\.

#### Assumption 2 \(Eigengap\)\.

When derivatives ofSkS\_\{k\}orPdomP\_\{\\mathrm\{dom\}\}are used, we assume

λk​\(θc\)\>λk\+1​\(θc\)\.\\lambda\_\{k\}\(\\theta\_\{c\}\)\>\\lambda\_\{k\+1\}\(\\theta\_\{c\}\)\.This ensures that the top\-kkeigenspace is separated from the rest of the spectrum nearθc\\theta\_\{c\}\.

#### Assumption 3 \(Moments and mini\-batch noise\)\.

We use the following moment and noise assumptions\.

\(i\) Local displacement moments\.The local displacements used in the averaging calculations satisfy

𝔼​\[δ\]=0,𝔼​‖δ‖3<∞\.\\mathbb\{E\}\[\\delta\]=0,\\qquad\\mathbb\{E\}\\\|\\delta\\\|^\{3\}<\\infty\.In the local recursion, we initialize with

𝔼​\[δ0\]=0\.\\mathbb\{E\}\[\\delta\_\{0\}\]=0\.
\(ii\) Mini\-batch noise\.The mini\-batch noise satisfies

𝔼B​\[ξB​\(θc\)∣θc\]=0,‖Σξ​\(θc\)‖<∞\.\\mathbb\{E\}\_\{B\}\[\\xi\_\{B\}\(\\theta\_\{c\}\)\\mid\\theta\_\{c\}\]=0,\\qquad\\\|\\Sigma\_\{\\xi\}\(\\theta\_\{c\}\)\\\|<\\infty\.
\(iii\) Local noise freezing\.For the local recursion, we write the noise variation as

ξBs​\(θc\+δs\)=ξBs​\(θc\)\+rξ,s,\\xi\_\{B\_\{s\}\}\(\\theta\_\{c\}\+\\delta\_\{s\}\)=\\xi\_\{B\_\{s\}\}\(\\theta\_\{c\}\)\+r\_\{\\xi,s\},and ignorerξ,sr\_\{\\xi,s\}when linearizing the mini\-batch gradient aroundθc\\theta\_\{c\}\.

#### Assumption 4 \(Local stability\)\.

When taking the stationary limit of the covariance recursion, we assume that the selected dominant eigenvalues satisfy

0<λk​\(θc\)≤λ1​\(θc\)<2η\.0<\\lambda\_\{k\}\(\\theta\_\{c\}\)\\leq\\lambda\_\{1\}\(\\theta\_\{c\}\)<\\frac\{2\}\{\\eta\}\.Since the local linear mapAcA\_\{c\}has eigenvalues1−η​λi​\(θc\)1\-\\eta\\lambda\_\{i\}\(\\theta\_\{c\}\)onEdom​\(θc\)E\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\), this implies

## Appendix CDerivation of the Sharpness Correction Term

In this section, we derive in more detail the sharpness correction term presented briefly in the main text\. Section[4](https://arxiv.org/html/2607.23012#S4)showed that when the local displacement lies in the dominant directions and has zero mean, averaging the gradient leaves a sharpness correction term, and described how this correction is induced by mini\-batch noise\. Here, we give the same calculation in more detail and show how the dominant perturbation experiment in Section[3](https://arxiv.org/html/2607.23012#S3)and mini\-batch noise each lead to a sharpness correction term through their displacement covariance\. The notation and assumptions follow Appendix[B](https://arxiv.org/html/2607.23012#A2)\.

First, in Appendix[C\.1](https://arxiv.org/html/2607.23012#A3.SS1), we show that averaging∇L​\(θc\+δ\)\\nabla L\(\\theta\_\{c\}\+\\delta\)over a local displacement with zero mean leaves a covariance\-weighted curvature term: the sharpness correction term\. Next, in Appendix[C\.2](https://arxiv.org/html/2607.23012#A3.SS2), we specialize this sharpness correction term to the dominant perturbation experiment in Section[3](https://arxiv.org/html/2607.23012#S3), and show why it lowers top\-kksharpness in that case\. Finally, in Appendix[C\.3](https://arxiv.org/html/2607.23012#A3.SS3), we compute the displacement covariance induced by mini\-batch noise through a local recursion, and show how it determines the mini\-batch\-induced sharpness correction term\.

### C\.1Sharpness correction from local displacement

We compute the covariance\-weighted curvature term that remains when we average the gradient around a reference point\. To do this, fix a reference pointθc\\theta\_\{c\}, and letδ\\deltabe a local displacement aroundθc\\theta\_\{c\}with𝔼​\[δ\]=0\\mathbb\{E\}\[\\delta\]=0\. We write the nearby parameter asθc\+δ\\theta\_\{c\}\+\\delta, and define the displacement covariance by

C​\(θc\):=𝔼​\[δ​δ⊤\]\.C\(\\theta\_\{c\}\):=\\mathbb\{E\}\[\\delta\\delta^\{\\top\}\]\.
Motivated by the dominant perturbation experiment in Section[3](https://arxiv.org/html/2607.23012#S3), we consider the case where the displacement aroundθc\\theta\_\{c\}lies in the dominant subspace\. That is, we takeδ∈Edom​\(θc\)\\delta\\in E\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\), and so we viewCCas a covariance supported onEdom​\(θc\)E\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\)\.

Now expand the gradient atθc\+δ\\theta\_\{c\}\+\\deltaaroundθc\\theta\_\{c\}\. The second\-order Taylor expansion of the gradient is

∇L​\(θc\+δ\)=∇L​\(θc\)\+H​\(θc\)​δ\+12​∇\(δ⊤​H​δ\)⁡\(θc\)\+R3​\(θc,δ\)\.\\nabla L\(\\theta\_\{c\}\+\\delta\)=\\nabla L\(\\theta\_\{c\}\)\+H\(\\theta\_\{c\}\)\\delta\+\\frac\{1\}\{2\}\\nabla\(\\delta^\{\\top\}H\\delta\)\(\\theta\_\{c\}\)\+R\_\{3\}\(\\theta\_\{c\},\\delta\)\.\(4\)In Eq\. \([4](https://arxiv.org/html/2607.23012#A3.E4)\), the first term is the gradient at the reference point, and the second term is the linear term in the displacement\. The third term is a second\-order term that captures the local variation of the Hessian aroundθc\\theta\_\{c\}, andR3​\(θc,δ\)R\_\{3\}\(\\theta\_\{c\},\\delta\)is the higher\-order remainder\.

Now average Eq\. \([4](https://arxiv.org/html/2607.23012#A3.E4)\) overδ\\delta\. Since𝔼​\[δ\]=0\\mathbb\{E\}\[\\delta\]=0, the linear term vanishes after averaging:

𝔼δ​\[H​\(θc\)​δ\]=H​\(θc\)​𝔼δ​\[δ\]=0\.\\mathbb\{E\}\_\{\\delta\}\[H\(\\theta\_\{c\}\)\\delta\]=H\(\\theta\_\{c\}\)\\mathbb\{E\}\_\{\\delta\}\[\\delta\]=0\.After this cancellation, the only nontrivial term left to average is the second\-order term containing the Hessian variation\. To compute this, letϑ\\varthetabe a temporary variable and write

δ⊤​H​\(ϑ\)​δ=Tr⁡\(H​\(ϑ\)​δ​δ⊤\)\.\\delta^\{\\top\}H\(\\vartheta\)\\delta=\\operatorname\{Tr\}\\\!\\left\(H\(\\vartheta\)\\delta\\delta^\{\\top\}\\right\)\.Then, sinceC​\(θc\)=𝔼​\[δ​δ⊤\]C\(\\theta\_\{c\}\)=\\mathbb\{E\}\[\\delta\\delta^\{\\top\}\],

𝔼δ​\[δ⊤​H​\(ϑ\)​δ\]=Tr⁡\(H​\(ϑ\)​C​\(θc\)\)\.\\mathbb\{E\}\_\{\\delta\}\[\\delta^\{\\top\}H\(\\vartheta\)\\delta\]=\\operatorname\{Tr\}\\\!\\left\(H\(\\vartheta\)C\(\\theta\_\{c\}\)\\right\)\.Here,C​\(θc\)C\(\\theta\_\{c\}\)is held fixed when taking the derivative with respect toϑ\\vartheta\. Therefore,

𝔼δ​\[∇\(δ⊤​H​δ\)⁡\(θc\)\]=∇ϑTr⁡\(H​\(ϑ\)​C​\(θc\)\)\|ϑ=θc\.\\mathbb\{E\}\_\{\\delta\}\\left\[\\nabla\(\\delta^\{\\top\}H\\delta\)\(\\theta\_\{c\}\)\\right\]=\\left\.\\nabla\_\{\\vartheta\}\\operatorname\{Tr\}\\\!\\left\(H\(\\vartheta\)C\(\\theta\_\{c\}\)\\right\)\\right\|\_\{\\vartheta=\\theta\_\{c\}\}\.Thus, the averaged gradient is

𝔼δ​\[∇L​\(θc\+δ\)\]=∇L​\(θc\)\+12​∇ϑTr⁡\(H​\(ϑ\)​C​\(θc\)\)\|ϑ=θc\+ℛ3​\(θc\),\\mathbb\{E\}\_\{\\delta\}\[\\nabla L\(\\theta\_\{c\}\+\\delta\)\]=\\nabla L\(\\theta\_\{c\}\)\+\\frac\{1\}\{2\}\\left\.\\nabla\_\{\\vartheta\}\\operatorname\{Tr\}\\\!\\left\(H\(\\vartheta\)C\(\\theta\_\{c\}\)\\right\)\\right\|\_\{\\vartheta=\\theta\_\{c\}\}\+\\mathcal\{R\}\_\{3\}\(\\theta\_\{c\}\),\(5\)where

ℛ3​\(θc\):=𝔼δ​\[R3​\(θc,δ\)\]\.\\mathcal\{R\}\_\{3\}\(\\theta\_\{c\}\):=\\mathbb\{E\}\_\{\\delta\}\[R\_\{3\}\(\\theta\_\{c\},\\delta\)\]\.Under Assumptions 1 and 3 in Appendix[B\.2](https://arxiv.org/html/2607.23012#A2.SS2), this remainder isO​\(𝔼δ​‖δ‖3\)O\(\\mathbb\{E\}\_\{\\delta\}\\\|\\delta\\\|^\{3\}\)\.

As shown in Eq\. \([5](https://arxiv.org/html/2607.23012#A3.E5)\), averaging the gradient around the reference point adds a covariance\-weighted curvature term to the gradient at the reference point\. We refer to this additional term as a sharpness correction term\. This term is determined once the covarianceCCis specified\. We first consider the covariance that corresponds to the dominant perturbation experiment in Section[3](https://arxiv.org/html/2607.23012#S3), and then derive the covariance created by mini\-batch noise\.

### C\.2Sharpness correction from dominant perturbations

For the dominant perturbation experiment in Section[3](https://arxiv.org/html/2607.23012#S3), we specify the covariance that enters the sharpness correction term in Eq\. \([5](https://arxiv.org/html/2607.23012#A3.E5)\)\. We then show that the resulting correction lowers top\-kksharpness\.

A dominant perturbation can be viewed as the case where the covariance is isotropic inside the dominant subspace\. That is, for some scalar varianceσ2\>0\\sigma^\{2\}\>0, let

C=σ2​Pdom​\(θc\)C=\\sigma^\{2\}P\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\)\(6\)
We substitute the covariance in Eq\. \([6](https://arxiv.org/html/2607.23012#A3.E6)\) into the correction term in Eq\. \([5](https://arxiv.org/html/2607.23012#A3.E5)\)\. Under Assumption 2 in Appendix[B\.2](https://arxiv.org/html/2607.23012#A2.SS2),SkS\_\{k\}is locally differentiable, and the standard eigenvalue derivative formula gives

∇ϑTr⁡\(H​\(ϑ\)​Pdom​\(θc\)\)\|ϑ=θc=∇Sk​\(θc\)\.\\left\.\\nabla\_\{\\vartheta\}\\operatorname\{Tr\}\\\!\\left\(H\(\\vartheta\)P\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\)\\right\)\\right\|\_\{\\vartheta=\\theta\_\{c\}\}=\\nabla S\_\{k\}\(\\theta\_\{c\}\)\.Therefore, by Eq\. \([6](https://arxiv.org/html/2607.23012#A3.E6)\),

∇ϑTr⁡\(H​\(ϑ\)​C\)\|ϑ=θc=σ2​∇Sk​\(θc\)\.\\left\.\\nabla\_\{\\vartheta\}\\operatorname\{Tr\}\\\!\\left\(H\(\\vartheta\)C\\right\)\\right\|\_\{\\vartheta=\\theta\_\{c\}\}=\\sigma^\{2\}\\nabla S\_\{k\}\(\\theta\_\{c\}\)\.\(7\)Substituting this into Eq\. \([5](https://arxiv.org/html/2607.23012#A3.E5)\) gives

𝔼δ​\[∇L​\(θc\+δ\)\]=∇L​\(θc\)\+σ22​∇Sk​\(θc\)\+ℛ3​\(θc\)\.\\mathbb\{E\}\_\{\\delta\}\[\\nabla L\(\\theta\_\{c\}\+\\delta\)\]=\\nabla L\(\\theta\_\{c\}\)\+\\frac\{\\sigma^\{2\}\}\{2\}\\nabla S\_\{k\}\(\\theta\_\{c\}\)\+\\mathcal\{R\}\_\{3\}\(\\theta\_\{c\}\)\.\(8\)As shown in Eq\. \([8](https://arxiv.org/html/2607.23012#A3.E8)\), when the covariance is isotropic inside the dominant subspace, the averaged gradient has the form of the gradient at the reference point plus the termσ22​∇Sk​\(θc\)\\frac\{\\sigma^\{2\}\}\{2\}\\nabla S\_\{k\}\(\\theta\_\{c\}\)\. Thus, in the negative\-gradient update, the term−σ22​∇Sk​\(θc\)\-\\frac\{\\sigma^\{2\}\}\{2\}\\nabla S\_\{k\}\(\\theta\_\{c\}\)is added on top of the full\-gradient term, and this term acts in the direction that lowersSkS\_\{k\}\. This explains why the dominant perturbation in Section[3](https://arxiv.org/html/2607.23012#S3)shows lower top\-kksharpness than GD\.

### C\.3Sharpness correction from mini\-batch noise

In Appendix[C\.2](https://arxiv.org/html/2607.23012#A3.SS2), we considered the case where the displacement covariance can be specified directly, as in the dominant perturbation experiment in Section[3](https://arxiv.org/html/2607.23012#S3)\. For mini\-batch noise, this covariance is not directly given\. We now derive it from a local recursion around a reference point\.

#### Local recursion\.

Fix a reference pointθc\\theta\_\{c\}\. To model the local displacement created by mini\-batch noise around this point, we introduce a local recursion aroundθc\\theta\_\{c\}, indexed byss\. At local stepss, we write the local state asθc\+δs\\theta\_\{c\}\+\\delta\_\{s\}, withδs∈Edom​\(θc\)\\delta\_\{s\}\\in E\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\), and draw a fresh mini\-batchBsB\_\{s\}\.

To define one step of this local recursion, we take the difference between the mini\-batch update fromθc\+δs\\theta\_\{c\}\+\\delta\_\{s\}and the full\-batch GD update fromθc\\theta\_\{c\}, and project it onto the dominant subspace\. First, define the mini\-batch update atθc\+δs\\theta\_\{c\}\+\\delta\_\{s\}and the full\-batch GD update atθc\\theta\_\{c\}as

θmb\+:=θc\+δs−η​∇LBs​\(θc\+δs\),\\theta\_\{\\mathrm\{mb\}\}^\{\+\}:=\\theta\_\{c\}\+\\delta\_\{s\}\-\\eta\\nabla L\_\{B\_\{s\}\}\(\\theta\_\{c\}\+\\delta\_\{s\}\),\(9\)θgd\+:=θc−η​∇L​\(θc\)\\theta\_\{\\mathrm\{gd\}\}^\{\+\}:=\\theta\_\{c\}\-\\eta\\nabla L\(\\theta\_\{c\}\)\(10\)respectively\. Then the next displacement in the local recursion is defined as

δs\+1:=Pdom​\(θc\)​\(θmb\+−θgd\+\)\.\\delta\_\{s\+1\}:=P\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\)\(\\theta\_\{\\mathrm\{mb\}\}^\{\+\}\-\\theta\_\{\\mathrm\{gd\}\}^\{\+\}\)\.\(11\)That is, after subtracting the full\-batch GD step fromθc\\theta\_\{c\}, we project the remaining local difference onto the dominant subspace and use the result asδs\+1\\delta\_\{s\+1\}\.

Next, we approximate the mini\-batch gradient∇LBs​\(θc\+δs\)\\nabla L\_\{B\_\{s\}\}\(\\theta\_\{c\}\+\\delta\_\{s\}\)in Eq\. \([9](https://arxiv.org/html/2607.23012#A3.E9)\) around the reference pointθc\\theta\_\{c\}\. By the definition of mini\-batch noise,ξB​\(θ\):=∇LB​\(θ\)−∇L​\(θ\)\\xi\_\{B\}\(\\theta\):=\\nabla L\_\{B\}\(\\theta\)\-\\nabla L\(\\theta\), we have

∇LBs​\(θc\+δs\)=∇L​\(θc\+δs\)\+ξBs​\(θc\+δs\)\.\\nabla L\_\{B\_\{s\}\}\(\\theta\_\{c\}\+\\delta\_\{s\}\)=\\nabla L\(\\theta\_\{c\}\+\\delta\_\{s\}\)\+\\xi\_\{B\_\{s\}\}\(\\theta\_\{c\}\+\\delta\_\{s\}\)\.\(12\)Now approximate the two terms in Eq\. \([12](https://arxiv.org/html/2607.23012#A3.E12)\) aroundθc\\theta\_\{c\}as follows:

∇L​\(θc\+δs\)=∇L​\(θc\)\+H​\(θc\)​δs\+O​\(‖δs‖2\),ξBs​\(θc\+δs\)≈ξBs​\(θc\)\.\\nabla L\(\\theta\_\{c\}\+\\delta\_\{s\}\)=\\nabla L\(\\theta\_\{c\}\)\+H\(\\theta\_\{c\}\)\\delta\_\{s\}\+O\(\\\|\\delta\_\{s\}\\\|^\{2\}\),\\qquad\\xi\_\{B\_\{s\}\}\(\\theta\_\{c\}\+\\delta\_\{s\}\)\\approx\\xi\_\{B\_\{s\}\}\(\\theta\_\{c\}\)\.Therefore, after substituting this local approximation into Eq\. \([12](https://arxiv.org/html/2607.23012#A3.E12)\) and dropping the higher\-order term, we get

∇LBs​\(θc\+δs\)≈∇L​\(θc\)\+H​\(θc\)​δs\+ξBs​\(θc\)\.\\nabla L\_\{B\_\{s\}\}\(\\theta\_\{c\}\+\\delta\_\{s\}\)\\approx\\nabla L\(\\theta\_\{c\}\)\+H\(\\theta\_\{c\}\)\\delta\_\{s\}\+\\xi\_\{B\_\{s\}\}\(\\theta\_\{c\}\)\.\(13\)Eq\. \([13](https://arxiv.org/html/2607.23012#A3.E13)\) shows that the mini\-batch gradient atθc\+δs\\theta\_\{c\}\+\\delta\_\{s\}is approximated by the full\-gradient term atθc\\theta\_\{c\}, the linear response to the local displacement, and the mini\-batch noise term atθc\\theta\_\{c\}\.

Now use Eq\. \([13](https://arxiv.org/html/2607.23012#A3.E13)\) to computeδs\+1\\delta\_\{s\+1\}\. By Eq\. \([9](https://arxiv.org/html/2607.23012#A3.E9)\) and Eq\. \([11](https://arxiv.org/html/2607.23012#A3.E11)\),

δs\+1\\displaystyle\\delta\_\{s\+1\}≈Pdom​\(θc\)​\[θc\+δs−η​\(∇L​\(θc\)\+H​\(θc\)​δs\+ξBs​\(θc\)\)−\(θc−η​∇L​\(θc\)\)\]\\displaystyle\\approx P\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\)\\Big\[\\theta\_\{c\}\+\\delta\_\{s\}\-\\eta\\bigl\(\\nabla L\(\\theta\_\{c\}\)\+H\(\\theta\_\{c\}\)\\delta\_\{s\}\+\\xi\_\{B\_\{s\}\}\(\\theta\_\{c\}\)\\bigr\)\-\\bigl\(\\theta\_\{c\}\-\\eta\\nabla L\(\\theta\_\{c\}\)\\bigr\)\\Big\]\(14\)=Pdom​\(θc\)​\(δs−η​H​\(θc\)​δs−η​ξBs​\(θc\)\)\\displaystyle=P\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\)\\bigl\(\\delta\_\{s\}\-\\eta H\(\\theta\_\{c\}\)\\delta\_\{s\}\-\\eta\\xi\_\{B\_\{s\}\}\(\\theta\_\{c\}\)\\bigr\)=Ac​δs−η​Pdom​\(θc\)​ξBs​\(θc\)\.\\displaystyle=A\_\{c\}\\delta\_\{s\}\-\\eta P\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\)\\xi\_\{B\_\{s\}\}\(\\theta\_\{c\}\)\.Eq\. \([14](https://arxiv.org/html/2607.23012#A3.E14)\) is the local recursion around the reference pointθc\\theta\_\{c\}\. Here,Ac​δsA\_\{c\}\\delta\_\{s\}is the term by which the previous local displacement is carried forward throughAcA\_\{c\}, and−η​Pdom​\(θc\)​ξBs​\(θc\)\-\\eta P\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\)\\xi\_\{B\_\{s\}\}\(\\theta\_\{c\}\)is the term by which the current mini\-batch noise is newly injected into the dominant subspace\.

#### Displacement covariance recursion\.

Eq\. \([14](https://arxiv.org/html/2607.23012#A3.E14)\) gives the recursion for the local displacementδs\\delta\_\{s\}\. Now we compute the local displacement covariance induced by this recursion\. First, define thess\-th local displacement covariance as

Cs:=𝔼​\[δs​δs⊤\]\.C\_\{s\}:=\\mathbb\{E\}\[\\delta\_\{s\}\\delta\_\{s\}^\{\\top\}\]\.\(15\)Next, define the dominant noise covariance at the reference pointθc\\theta\_\{c\}as

Σdom​\(θc\):=Pdom​\(θc\)​Σξ​\(θc\)​Pdom​\(θc\)\.\\Sigma\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\):=P\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\)\\Sigma\_\{\\xi\}\(\\theta\_\{c\}\)P\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\)\.\(16\)
Since eachBsB\_\{s\}is a mini\-batch drawn freshly at the local step,ξBs​\(θc\)\\xi\_\{B\_\{s\}\}\(\\theta\_\{c\}\)is independent ofδs\\delta\_\{s\}, and𝔼​\[ξBs​\(θc\)∣θc\]=0\\mathbb\{E\}\[\\xi\_\{B\_\{s\}\}\(\\theta\_\{c\}\)\\mid\\theta\_\{c\}\]=0\. From these conditions, the cross terms satisfy

𝔼​\[δs​ξBs​\(θc\)⊤\]=𝔼​\[δs​𝔼​\[ξBs​\(θc\)⊤∣δs,θc\]\]=0,𝔼​\[ξBs​\(θc\)​δs⊤\]=0\.\\mathbb\{E\}\[\\delta\_\{s\}\\xi\_\{B\_\{s\}\}\(\\theta\_\{c\}\)^\{\\top\}\]=\\mathbb\{E\}\\\!\\left\[\\delta\_\{s\}\\,\\mathbb\{E\}\[\\xi\_\{B\_\{s\}\}\(\\theta\_\{c\}\)^\{\\top\}\\mid\\delta\_\{s\},\\theta\_\{c\}\]\\right\]=0,\\qquad\\mathbb\{E\}\[\\xi\_\{B\_\{s\}\}\(\\theta\_\{c\}\)\\delta\_\{s\}^\{\\top\}\]=0\.Therefore, computingCs\+1=𝔼​\[δs\+1​δs\+1⊤\]C\_\{s\+1\}=\\mathbb\{E\}\[\\delta\_\{s\+1\}\\delta\_\{s\+1\}^\{\\top\}\]from Eq\. \([14](https://arxiv.org/html/2607.23012#A3.E14)\) gives

Cs\+1\\displaystyle C\_\{s\+1\}=𝔼​\[δs\+1​δs\+1⊤\]\\displaystyle=\\mathbb\{E\}\[\\delta\_\{s\+1\}\\delta\_\{s\+1\}^\{\\top\}\]=𝔼​\[\(Ac​δs−η​Pdom​\(θc\)​ξBs​\(θc\)\)​\(Ac​δs−η​Pdom​\(θc\)​ξBs​\(θc\)\)⊤\]\\displaystyle=\\mathbb\{E\}\\\!\\left\[\\left\(A\_\{c\}\\delta\_\{s\}\-\\eta P\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\)\\xi\_\{B\_\{s\}\}\(\\theta\_\{c\}\)\\right\)\\left\(A\_\{c\}\\delta\_\{s\}\-\\eta P\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\)\\xi\_\{B\_\{s\}\}\(\\theta\_\{c\}\)\\right\)^\{\\top\}\\right\]=Ac​Cs​Ac⊤\+η2​Pdom​\(θc\)​Σξ​\(θc\)​Pdom​\(θc\)\.\\displaystyle=A\_\{c\}C\_\{s\}A\_\{c\}^\{\\top\}\+\\eta^\{2\}P\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\)\\Sigma\_\{\\xi\}\(\\theta\_\{c\}\)P\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\)\.Thus, by the definition in Eq\. \([16](https://arxiv.org/html/2607.23012#A3.E16)\),

Cs\+1=Ac​Cs​Ac⊤\+η2​Σdom​\(θc\)\.C\_\{s\+1\}=A\_\{c\}C\_\{s\}A\_\{c\}^\{\\top\}\+\\eta^\{2\}\\Sigma\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\)\.\(17\)Eq\. \([17](https://arxiv.org/html/2607.23012#A3.E17)\) is the covariance recursion showing how the dominant noise covarianceΣdom​\(θc\)\\Sigma\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\)induces the local displacement covariance through the local recursion\.

#### Stationary displacement covariance\.

We now derive the stationary displacement covariance created by the local recursion around the reference pointθc\\theta\_\{c\}\.

Iterating Eq\. \([17](https://arxiv.org/html/2607.23012#A3.E17)\) gives

Cs=Acs​C0​\(Ac⊤\)s\+η2​∑ℓ=0s−1Acℓ​Σdom​\(θc\)​\(Ac⊤\)ℓ\.C\_\{s\}=A\_\{c\}^\{s\}C\_\{0\}\(A\_\{c\}^\{\\top\}\)^\{s\}\+\\eta^\{2\}\\sum\_\{\\ell=0\}^\{s\-1\}A\_\{c\}^\{\\ell\}\\Sigma\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\)\(A\_\{c\}^\{\\top\}\)^\{\\ell\}\.\(18\)By Assumption 4 in Appendix[B\.2](https://arxiv.org/html/2607.23012#A2.SS2),ρ​\(Ac\)<1\\rho\(A\_\{c\}\)<1, soAcs​C0​\(Ac⊤\)s→0A\_\{c\}^\{s\}C\_\{0\}\(A\_\{c\}^\{\\top\}\)^\{s\}\\to 0, and the second term in Eq\. \([18](https://arxiv.org/html/2607.23012#A3.E18)\) also converges ass→∞s\\to\\infty\. We denote its limit by

Cmb​\(θc\):=lims→∞CsC\_\{\\mathrm\{mb\}\}\(\\theta\_\{c\}\):=\\lim\_\{s\\to\\infty\}C\_\{s\}and call it the mini\-batch\-induced displacement covariance\. ThenCmb​\(θc\)C\_\{\\mathrm\{mb\}\}\(\\theta\_\{c\}\)satisfies

Cmb​\(θc\)=Ac​Cmb​\(θc\)​Ac⊤\+η2​Σdom​\(θc\)\.C\_\{\\mathrm\{mb\}\}\(\\theta\_\{c\}\)=A\_\{c\}C\_\{\\mathrm\{mb\}\}\(\\theta\_\{c\}\)A\_\{c\}^\{\\top\}\+\\eta^\{2\}\\Sigma\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\)\.\(19\)Eq\. \([19](https://arxiv.org/html/2607.23012#A3.E19)\) is a discrete Lyapunov equation forCmb​\(θc\)C\_\{\\mathrm\{mb\}\}\(\\theta\_\{c\}\)\. Equivalently,Cmb​\(θc\)C\_\{\\mathrm\{mb\}\}\(\\theta\_\{c\}\)is given by thes→∞s\\to\\inftylimit of Eq\. \([18](https://arxiv.org/html/2607.23012#A3.E18)\), namely

Cmb​\(θc\)=η2​∑ℓ=0∞Acℓ​Σdom​\(θc\)​\(Ac⊤\)ℓ\.C\_\{\\mathrm\{mb\}\}\(\\theta\_\{c\}\)=\\eta^\{2\}\\sum\_\{\\ell=0\}^\{\\infty\}A\_\{c\}^\{\\ell\}\\Sigma\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\)\(A\_\{c\}^\{\\top\}\)^\{\\ell\}\.\(20\)Eq\. \([20](https://arxiv.org/html/2607.23012#A3.E20)\) shows thatCmb​\(θc\)C\_\{\\mathrm\{mb\}\}\(\\theta\_\{c\}\)is formed by accumulating the dominant noise covarianceΣdom​\(θc\)\\Sigma\_\{\\mathrm\{dom\}\}\(\\theta\_\{c\}\)through the local recursion, with its value determined by the noise covariance, the local Hessian structure, and the learning rate\. This covariance summarizes the effect of mini\-batch noise in the dominant subspace\.

#### Mini\-batch\-induced sharpness correction\.

Now setC=Cmb​\(θc\)C=C\_\{\\mathrm\{mb\}\}\(\\theta\_\{c\}\)in Eq\. \([5](https://arxiv.org/html/2607.23012#A3.E5)\)\. Then the averaged gradient is

𝔼δ​\[∇L​\(θc\+δ\)\]=∇L​\(θc\)\+12​∇ϑTr⁡\(H​\(ϑ\)​Cmb​\(θc\)\)\|ϑ=θc\+ℛ3​\(θc\)\.\\mathbb\{E\}\_\{\\delta\}\[\\nabla L\(\\theta\_\{c\}\+\\delta\)\]=\\nabla L\(\\theta\_\{c\}\)\+\\frac\{1\}\{2\}\\left\.\\nabla\_\{\\vartheta\}\\operatorname\{Tr\}\\\!\\left\(H\(\\vartheta\)C\_\{\\mathrm\{mb\}\}\(\\theta\_\{c\}\)\\right\)\\right\|\_\{\\vartheta=\\theta\_\{c\}\}\+\\mathcal\{R\}\_\{3\}\(\\theta\_\{c\}\)\.\(21\)Eq\. \([21](https://arxiv.org/html/2607.23012#A3.E21)\) shows that the mini\-batch\-induced displacement covarianceCmb​\(θc\)C\_\{\\mathrm\{mb\}\}\(\\theta\_\{c\}\)leaves an additional term in the averaged gradient\. We refer to this term as the mini\-batch\-induced sharpness correction term\. For the isotropic covariance in Section[C\.2](https://arxiv.org/html/2607.23012#A3.SS2), this term is proportional to∇Sk​\(θc\)\\nabla S\_\{k\}\(\\theta\_\{c\}\)\. In contrast,Cmb​\(θc\)C\_\{\\mathrm\{mb\}\}\(\\theta\_\{c\}\)is generally not isotropic, so the correction term in Eq\. \([21](https://arxiv.org/html/2607.23012#A3.E21)\) is not simply proportional to∇Sk​\(θc\)\\nabla S\_\{k\}\(\\theta\_\{c\}\)\.

Therefore, in Section[5](https://arxiv.org/html/2607.23012#S5), we apply the corresponding sharpness correction to full\-batch GD and compare the resulting corrected GD with mini\-batch SGD\. Specifically, Figure[4](https://arxiv.org/html/2607.23012#S5.F4)compares full\-batch GD, mini\-batch SGD, and corrected GD, all starting from the same initialization, and shows that this correction reduces the sharpness gap between GD and SGD\.

## Appendix DExperimental Details

In this section, we summarize the codebase, datasets, architectures, optimization settings, Hessian computation, and correction computation used in the main experiments\.

### D\.1Implementation

The experimental code is based on the supplementary implementation provided withsong2025does\. This implementation follows the experimental setup and codebase ofcohen2021gradient\. The experiments were run on an internal server with88NVIDIA RTX 3090 GPUs\.

On top of this codebase, we added the estimation of the displacement covarianceCmbC\_\{\\mathrm\{mb\}\}, the correction termbcorrb\_\{\\mathrm\{corr\}\}, and corrected GD used in this paper\. For Hessian eigenspace computation and higher\-order automatic differentiation, we followed the numerical conventions ofcohen2025understanding\. In particular, for the Transformer experiments, we use a vanilla PyTorch LayerNorm implementation instead of the default PyTorchnn\.LayerNorm, which was needed to compute the third\-order derivatives in the correction term\.

### D\.2Datasets and Architectures

Table[1](https://arxiv.org/html/2607.23012#A4.T1)summarizes the datasets and architectures used in our experiments\.

Table 1:Dataset–architecture pairs used in our experiments\.For MNIST\-5k and CIFAR10\-5k, we use the first5,0005\{,\}000training examples\. For SST2\-1k, we use the first1,0001\{,\}000training examples\. All experiments use only the training split\. The MLP and CNN settings followcohen2021gradient, while the Transformer setting followsdamian2023selfstabilizationimplicitbiasgradient\.

### D\.3Training Setup

All main runs use vanilla mini\-batch SGD or full\-batch GD\. The learning rate is kept constant, and we do not use momentum, weight decay, warmup, learning\-rate decay, or early stopping\. The default loss is the MSE loss with one\-hot labels\. Gradients, Hessian\-vector products, Hessian eigenvalues, and covariance estimates are all computed using the1/N1/N\-normalized training loss\.

Unless stated otherwise, we follow the stable learning\-rate \(GF\) regime ofsong2025does\. We use a batch size of5050, with learning rate0\.010\.01for the MLP and0\.0010\.001for the CNN and Transformer\. Each main run is trained for20,00020\{,\}000steps\.

### D\.4Hessian and Correction Computation

Hessian\-related quantities are computed at analysis checkpoints placed every100100training steps\. At each checkpoint, we compute the top Hessian eigenpairs using Hessian\-vector products and LOBPCG\(doi:10\.1137/S1064827500366124\)\. We compute2020eigenpairs at each checkpoint and use the top\-kkeigenvectors as the dominant\-subspace basis\. We usek=10k=10for MNIST\-5k and CIFAR10\-5k, andk=2k=2for SST2\-1k\.

At the same checkpoint, we estimate the mini\-batch noise covariance on the dominant Hessian basis using100100newly sampled mini\-batches of size5050\. The quantitiesCmbC\_\{\\mathrm\{mb\}\}andbcorrb\_\{\\mathrm\{corr\}\}are then computed as described in Appendix[C\.3](https://arxiv.org/html/2607.23012#A3.SS3)\. In corrected GD,bcorrb\_\{\\mathrm\{corr\}\}is recomputed at each checkpoint and kept fixed until the next checkpoint\.

### D\.5Reported Quantities and Baselines

At the analysis checkpoints above, we record training loss, gradients, top Hessian eigenvalues, top\-kksharpnessSkS\_\{k\}, and dominant alignmentχk\\chi\_\{k\}\. The main comparisons are mini\-batch SGD, full\-batch GD, and corrected GD\. In the projected\-update experiments, we also report Dom\-SGD and Bulk\-SGD\. In the perturbation experiments, we compare dominant perturbation with random perturbation matched in rank and total variance\.

## Appendix EAdditional Experiments

In this section, we provide additional experimental results complementing Section[5](https://arxiv.org/html/2607.23012#S5)\. We first examine how the sharpness evolution changes when the batch size is changed\. We then reverse the direction of the correction termbcorrb\_\{\\mathrm\{corr\}\}to check whether the sharpness\-reducing effect depends on the direction of the correction\.

### E\.1Batch Size Sweep

We compare mini\-batch SGD, full\-batch GD, and corrected GD obtained by applying the sharpness correction term to full\-batch GD under different batch sizes in Figure[5](https://arxiv.org/html/2607.23012#A5.F5)\.

![Refer to caption](https://arxiv.org/html/2607.23012v1/x11.png)

![Refer to caption](https://arxiv.org/html/2607.23012v1/x12.png)

![Refer to caption](https://arxiv.org/html/2607.23012v1/x13.png)

Figure 5:Corrected GD across batch sizes\.\(a\) MLP on MNIST\-5k, \(b\) CNN on CIFAR10\-5k, and \(c\) Transformer on SST2\-1k\. Solid lines denote mini\-batch SGD, and dashed lines denote corrected GD\. As the batch size increases, theSkS\_\{k\}curve of SGD moves closer to full\-batch GD, and corrected GD follows the same trend\.
### E\.2Correction Direction

We reverse the direction of the correction termbcorrb\_\{\\mathrm\{corr\}\}and compare it with the original corrected GD\. This experiment shows that sharpness reduction depends on the direction of the derived correction term\. Figures[7](https://arxiv.org/html/2607.23012#A5.F7)and[7](https://arxiv.org/html/2607.23012#A5.F7)show the corresponding sharpness and loss curves\.

![Refer to caption](https://arxiv.org/html/2607.23012v1/x14.png)

Figure 6:Effect of the correction direction on MLP / MNIST\-5k\.We compare GD, SGD, corrected GD, and GD with the reversed correction\. \(a\) Top\-kksharpnessSkS\_\{k\}\. \(b\) Training loss\. The derived correction lowers sharpness relative to GD, whereas reversing its direction increases sharpness relative to GD\. The loss curves remain largely unchanged\.![Refer to caption](https://arxiv.org/html/2607.23012v1/x15.png)

Figure 7:Effect of the correction direction on CNN / CIFAR10\-5k\.We compare GD, SGD, corrected GD, and GD with the reversed correction\. \(a\) Top\-kksharpnessSkS\_\{k\}\. \(b\) Training loss\. The derived correction lowers sharpness relative to GD, whereas reversing its direction increases sharpness relative to GD\. The loss curves remain largely unchanged\.