SAM-on-the-Curve: Sharpness-Aware Mode Connectivity for Robust Weight-Space Interpolation

# SAM-on-the-Curve: Sharpness-Aware Mode Connectivity for Robust Weight-Space Interpolation
Source: [https://arxiv.org/html/2609.17748](https://arxiv.org/html/2609.17748)
###### Abstract

Deep neural networks that are independently trained to similar performance can be connected by low\-loss parametric curves in weight space, a phenomenon known as Mode Connectivity \(MC\)\. This geometric property underpins practical techniques such as weight averaging, model ensembling, and model merging\. We argue that low\-loss connectivity is an incomplete geometric criterion: it controls loss only along a one\-dimensional trajectory while leaving the surrounding weight\-space neighborhood unconstrained, so the optimized curve may traverse sharp ridges that become fragile under distribution shift\. We therefore reformulate mode connectivity as a neighborhood\-robust path optimization problem, seeking a curve whose entire local neighborhood maintains low loss\. We propose Sharp Mode Connectivity \(SMC\), which applies a first\-order sharpness\-aware approximation to the resulting minimax functional, enforcing flatness along the entire curve rather than only on it\. We derive a practical optimization algorithm for connectivity paths under this sharpness\-aware objective\. Under severe blur corruptions from CIFAR\-10\-C, SMC achieves up to 6\.09% absolute accuracy improvement over standard MC\. Remarkably, SMC produces negative loss barriers, meaning that models obtained at interior points of the optimized path can outperform the average endpoint loss\. These results, validated across ResNet\-18, VGG16\-BN, and ViT\-Tiny on CIFAR\-10 and ImageNet\-100, establish path\-wise flatness as a practical principle for robust weight\-space interpolation\.

###### Index Terms:

Mode Connectivity, Sharpness\-Aware Minimization, Loss Landscape Geometry, Generalization, Distribution Shift, Flat Minima

††impactstatement:As deep learning is increasingly deployed in safety\-critical applications such as autonomous driving, medical diagnosis, and scientific instrumentation, the ability to guarantee robust generalization under real\-world distribution shifts becomes a societal imperative, not merely an academic benchmark\. This work contributes a principled geometric framework that enables the construction of inherently robust model ensembles and weight\-averaged models by ensuring that the entire weight\-space path connecting independently trained solutions lies within flat, low\-loss valleys\. For practitioners, this means more reliable predictions when operating conditions deviate from training data, directly impacting the trustworthiness of AI\-assisted decisions in healthcare, infrastructure monitoring, and environmental sensing\. For the research community, this work bridges two influential but previously disconnected theoretical perspectives \(Mode Connectivity and Sharpness\-Aware Minimization\), opening new avenues in path\-wise geometric regularization and, potentially, federated model merging \(where client models must be combined through weight\-space operations\), as well as a deeper understanding of how architectural inductive biases shape the geometry of the loss landscape\.## IIntroduction

Weight\-space interpolation has become a central operation in modern deep learning\. Stochastic Weight Averaging \(SWA\)\[[1](https://arxiv.org/html/2609.17748#bib.bib17)\]averages SGD iterates along training trajectories to find wider optima\. Fast Geometric Ensembling \(FGE\)\[[2](https://arxiv.org/html/2609.17748#bib.bib1)\]collects diverse checkpoints along low\-loss curves discovered by cyclical learning rates\. Model soups\[[3](https://arxiv.org/html/2609.17748#bib.bib24)\]combine independently fine\-tuned parameter vectors to improve accuracy without increasing inference cost\. All of these methods rely on combining or interpolating parameters across trained solutions in weight space, so the geometric structure of the region connecting these solutions can influence the robustness of the resulting models\.

Mode Connectivity \(MC\)\[[2](https://arxiv.org/html/2609.17748#bib.bib1),[4](https://arxiv.org/html/2609.17748#bib.bib16)\]provides the foundational path\-finding framework: given two independently trained solutionsw1,w2∈ℝdw\_\{1\},w\_\{2\}\\in\\mathbb\{R\}^\{d\}, MC optimizes a parametric curveϕθ\\phi\_\{\\theta\}connecting them by minimizing the expected loss along the path\. However, this low\-loss connectivity is an incomplete geometric criterion because it controls the loss strictly on the one\-dimensional trajectory while leaving the surrounding weight\-space neighborhood unconstrained\. Consequently, the optimized path may thread through narrow, sharp ridges where the loss rises steeply under perturbation, making it fragile under distribution shift\. We therefore reformulate mode connectivity as a neighborhood\-robust path optimization problem, seeking a curve whose entire local neighborhood maintains low loss\.

![Refer to caption](https://arxiv.org/html/2609.17748v1/figures/diagram-smc-2.png)Figure 1:Conceptual overview of SMC\. Two independently trained solutionsw1w\_\{1\}andw2w\_\{2\}are connected by a standard MC path \(red, dashed\), which may traverse sharp ridges, and an SMC path \(blue, solid\), which is encouraged to pass through flatter regions by the sharpness\-aware perturbation objective of radiusρ\\rho\.We proposeSharp Mode Connectivity\(SMC\), which replaces the pointwise loss at each curve location with the worst\-case loss inside a surrounding perturbation neighborhood, so the whole curve is driven into a flat region of the loss landscape \(Fig\.[1](https://arxiv.org/html/2609.17748#S1.F1); formalized in Section[III](https://arxiv.org/html/2609.17748#S3)\)\. Empirically, this makes the loss profile along the curve far more uniform: on CIFAR\-10 with ResNet\-18, the variance of the loss across points sampled along the SMC path is0\.00210\.0021, versus0\.0560\.056for standard MC \(a26\.7×26\.7\\timesreduction\), reflecting a nearly flat, valley\-like profile rather than a sharp point spike\. Under severe distribution shifts from the CIFAR\-10\-C benchmark\[[5](https://arxiv.org/html/2609.17748#bib.bib8)\], the path midpoint achieves up to\+6\.09%\+6\.09\\%absolute accuracy improvement over standard MC\. We focus on the midpoint here to evaluate the robustness of the connectivity, whereas peak accuracy is reported later when seeking the single best model along the path\. Furthermore, SMC produces*negative loss barriers*\.

Our principal contributions are:

- •Neighborhood\-robust mode connectivity\.We identify a limitation of conventional mode connectivity: minimizing loss on a one\-dimensional interpolation trajectory leaves the surrounding weight\-space neighborhood unconstrained\. We formulate SMC as a minimax path functional that seeks low\-loss neighborhoods along the connectivity curve\.
- •SAM\-on\-the\-Curve optimization\.We derive a practical stochastic algorithm that optimizes Bézier control points using first\-order sharpness\-aware perturbations at sampled curve locations, extending sharpness\-aware optimization from isolated parameter vectors to continuous weight\-space trajectories\.
- •Geometric and robustness analysis\.Across CNN and Transformer architectures, we show that sharpness\-aware path optimization reduces both longitudinal path\-loss variation and local neighborhood sensitivity\. Under CIFAR\-10\-C distribution shifts, these geometric changes are associated with improved interpolation robustness and path\-interior solutions that can outperform the endpoint average\.

The remainder of this paper is organized as follows\. Section[II](https://arxiv.org/html/2609.17748#S2)reviews related work\. Section[III](https://arxiv.org/html/2609.17748#S3)introduces the necessary preliminaries on Mode Connectivity and SAM, identifies the research gap, and presents the proposed SMC framework and the SAM\-on\-the\-Curve algorithm\. Section[IV](https://arxiv.org/html/2609.17748#S4)reports comprehensive experimental results\. Section[V](https://arxiv.org/html/2609.17748#S5)concludes and discusses future directions\.

## IIRelated Work

### II\-AGeneralization and Loss Landscape Geometry

Understanding why overparameterized DNNs generalize despite having the capacity to memorize random labels\[[6](https://arxiv.org/html/2609.17748#bib.bib12)\]has motivated a shift toward geometric explanations of generalization\. A central finding is that the*shape*of minima in the loss landscape, rather than merely the loss value, is among the strongest predictors of test performance\[[7](https://arxiv.org/html/2609.17748#bib.bib18)\]\.

Hochreiter and Schmidhuber\[[8](https://arxiv.org/html/2609.17748#bib.bib13)\]first established that flat minima, whose loss remains stable under parameter perturbations, are associated with shorter description lengths and better generalization\. Keskar et al\.\[[9](https://arxiv.org/html/2609.17748#bib.bib14)\]provided complementary empirical evidence showing that large\-batch SGD converges to sharp, narrow minima with degraded test accuracy, whereas small\-batch training favors flatter regions\. Neyshabur et al\.\[[10](https://arxiv.org/html/2609.17748#bib.bib15)\]formalized norm\-based generalization bounds connecting parameter geometry to the gap between training and test error, further grounding the flatness and generalization link in learning theory\.

Li et al\.\[[11](https://arxiv.org/html/2609.17748#bib.bib19)\]introduced filter\-normalized loss landscape visualization, enabling meaningful geometric comparisons across architectures and training regimes\. Their analysis confirmed that architectural choices such as skip connections \(e\.g\., ResNets\[[12](https://arxiv.org/html/2609.17748#bib.bib11)\]\) produce markedly smoother loss surfaces than architectures without residual paths \(e\.g\., VGG\[[13](https://arxiv.org/html/2609.17748#bib.bib10)\]\), a finding that influences both optimization dynamics and the feasibility of weight\-space interpolation methods discussed below\.

### II\-BSharpness\-Aware Optimization

Motivated by the flatness and generalization connection, Foret et al\.\[[14](https://arxiv.org/html/2609.17748#bib.bib5)\]proposedSharpness\-Aware Minimization\(SAM\), which replaces pointwise loss minimization with a min\-max objective that penalizes the worst\-case loss within a perturbation ball of radiusρ\\rho\. The inner maximization is approximated via a first\-order Taylor expansion, yielding a two\-step procedure that consistently improves generalization on CIFAR\-10, CIFAR\-100, and ImageNet\[[14](https://arxiv.org/html/2609.17748#bib.bib5)\]\. Subsequent extensions include ASAM\[[15](https://arxiv.org/html/2609.17748#bib.bib20)\], which introduces adaptive sharpness invariant to parameter rescaling, and GSAM\[[16](https://arxiv.org/html/2609.17748#bib.bib21)\], which minimizes a surrogate gap related to the dominant Hessian eigenvalue\. Despite these advances, all SAM variants optimize flatness at*individual parameter vectors*; none extend sharpness\-aware regularization to*continuous paths*through weight space\.

### II\-CMode Connectivity and Weight\-Space Geometry

Garipov et al\.\[[2](https://arxiv.org/html/2609.17748#bib.bib1)\]and Draxler et al\.\[[4](https://arxiv.org/html/2609.17748#bib.bib16)\]independently demonstrated that SGD\-found minima are not isolated: independently trained solutions can be connected by simple parametric curves along which training loss and test error remain nearly constant\. This discovery enabledFast Geometric Ensembling\(FGE\)\[[2](https://arxiv.org/html/2609.17748#bib.bib1)\]andStochastic Weight Averaging\(SWA\)\[[1](https://arxiv.org/html/2609.17748#bib.bib17)\], both of which exploit the low\-loss valley structure to improve generalization within a single training budget\. Maddox et al\.\[[17](https://arxiv.org/html/2609.17748#bib.bib25)\]extended SWA to Bayesian inference via SWAG\. Ren et al\.\[[18](https://arxiv.org/html/2609.17748#bib.bib4)\]further extended conventional curve\-based mode connectivity to Bézier surfaces\.

The mode connectivity perspective has since intersected with model merging\. Entezari et al\.\[[19](https://arxiv.org/html/2609.17748#bib.bib22)\]conjectured that accounting for permutation symmetry would eliminate loss barriers on linear interpolation paths; Ainsworth et al\.\[[20](https://arxiv.org/html/2609.17748#bib.bib23)\]validated this withGit Re\-Basin\. Wortsman et al\.\[[3](https://arxiv.org/html/2609.17748#bib.bib24)\]showed that simple weight averaging of fine\-tuned models \(Model Soups\) improves accuracy without increasing inference cost\. Parameter\-space connectivity has also been explored for machine unlearning\[[21](https://arxiv.org/html/2609.17748#bib.bib2)\]and evolutionary attacks\[[22](https://arxiv.org/html/2609.17748#bib.bib3)\]\. However, all of these methods optimize parameter\-space structures using standard loss functions that do not explicitly account for the curvature of the surrounding loss surface\.

## IIIProposed Method

This section develops the Sharp Mode Connectivity \(SMC\) framework\. We first review how Mode Connectivity constructs low\-loss paths between independently trained solutions, then introduce Sharpness\-Aware Minimization and the gap it addresses, and finally derive the SMC algorithm that integrates both perspectives\.

### III\-AHow to Build a Low\-Loss Path: Mode Connectivity

Garipov et al\.\[[2](https://arxiv.org/html/2609.17748#bib.bib1)\]demonstrated that independently trained DNN solutions can be connected by simple parametric curves along which both training loss and test error remain nearly constant\. Given a labeled dataset𝒮=\{\(xi,yi\)\}i=1n\\mathcal\{S\}=\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{n\}drawn from an unknown distribution𝒟\\mathcal\{D\}over𝒳×𝒴\\mathcal\{X\}\\times\\mathcal\{Y\}, a DNNf⁡\(⋅,w\):𝒳→ℝCf\(\\cdot\\,;w\):\\mathcal\{X\}\\to\\mathbb\{R\}^\{C\}with parametersw∈ℝdw\\in\\mathbb\{R\}^\{d\}incurs the empirical riskℒ𝒮​\(w\)=1n​∑i=1nℓ⁡\(f⁡\(xi,w\),yi\)\\mathcal\{L\}\_\{\\mathcal\{S\}\}\(w\)=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\ell\(f\(x\_\{i\};w\),y\_\{i\}\), whereℓ\\elldenotes the cross\-entropy loss\. Letw1,w2∈ℝdw\_\{1\},w\_\{2\}\\in\\mathbb\{R\}^\{d\}denote two independently trained solutions\. Given two fixed endpoints, Mode Connectivity seeks a parametric curveϕθ:\[0,1\]→ℝd\\phi\_\{\\theta\}:\[0,1\]\\to\\mathbb\{R\}^\{d\}satisfying the boundary conditions

ϕθ​\(0\)=w1,ϕθ​\(1\)=w2,\\phi\_\{\\theta\}\(0\)=w\_\{1\},\\quad\\phi\_\{\\theta\}\(1\)=w\_\{2\},\(1\)whereθ\\thetadenotes learnable curve parameters \(e\.g\., control points\)\. The ideal objective minimizes the arc\-length\-normalized integral of the loss along the curve; however, since gradients of this objective are intractable, the practical formulation samples t from the uniform distributionU⁡\(0,1\)U\(0,1\):

ℒpath​\(θ\)=𝔼t∼U⁡\(0,1\)​\[ℒ𝒮​\(ϕθ​\(t\)\)\]\.\\mathcal\{L\}\_\{\\mathrm\{path\}\}\(\\theta\)=\\mathbb\{E\}\_\{t\\sim U\(0,1\)\}\\big\[\\mathcal\{L\}\_\{\\mathcal\{S\}\}\(\\phi\_\{\\theta\}\(t\)\)\\big\]\.\(2\)We employ acubic Bézier curvewith two control pointsθ=\(θ1,θ2\)\\theta=\(\\theta\_\{1\},\\theta\_\{2\}\)to parametrize the path:

ϕθ​\(t\)=\(1−t\)3​w1\+3​t​\(1−t\)2​θ1\+3​t2​\(1−t\)​θ2\+t3​w2\.\\phi\_\{\\theta\}\(t\)=\(1\-t\)^\{3\}w\_\{1\}\+3t\(1\-t\)^\{2\}\\theta\_\{1\}\+3t^\{2\}\(1\-t\)\\theta\_\{2\}\+t^\{3\}w\_\{2\}\.\(3\)
The choice of a cubic Bézier curve balances expressiveness and parsimony\. Linear interpolation \(ϕ⁡\(t\)=\(1−t\)​w1\+t​w2\\phi\(t\)=\(1\-t\)w\_\{1\}\+tw\_\{2\}\) provides no learnable degrees of freedom, so the path is fully determined by the endpoints and no optimization is possible\. A quadratic Bézier curve introduces only one learnable control point, so the geometry of the early and late portions of the path cannot be adjusted independently\. The cubic Bézier curve uses two control points and is therefore the lowest\-order Bézier parameterization that provides separate control over the path near the two endpoints, which is useful for navigating asymmetric and non\-convex loss landscapes\. Higher\-order Bézier curves \(≥3\{\\color\[rgb\]\{0,0\.6,0\}\\geq 3\}control points\) would increase the dimensionality of the optimization problem inℝd\\mathbb\{R\}^\{d\}without empirical evidence of improved path quality, and risk overfitting to noise in the sampled loss landscape\.

To optimize the curve parametersθ\\theta, we require the gradient of the path objective \([2](https://arxiv.org/html/2609.17748#S3.E2)\)\. This derivation proceeds in three steps\. First, differentiating under the expectation yields

∇θℒpath​\(θ\)=𝔼t∼U⁡\(0,1\)​\[∇θℒ𝒮​\(ϕθ​\(t\)\)\]\.\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\mathrm\{path\}\}\(\\theta\)=\\mathbb\{E\}\_\{t\\sim U\(0,1\)\}\\\!\\Big\[\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\mathcal\{S\}\}\\\!\\big\(\\phi\_\{\\theta\}\(t\)\\big\)\\Big\]\.\(4\)Since the lossℒ𝒮\\mathcal\{L\}\_\{\\mathcal\{S\}\}is defined as a function of the weightsw=ϕθ​\(t\)∈ℝdw=\\phi\_\{\\theta\}\(t\)\\in\\mathbb\{R\}^\{d\}and the curveϕθ\\phi\_\{\\theta\}maps the parametersθ\\thetainto weight space, we apply the chain rule to decompose the gradient:

∇θℒ𝒮​\(ϕθ​\(t\)\)=\(∂ϕθ​\(t\)∂θ\)⊤⏟curve Jacobian​∇wℒ𝒮​\(ϕθ​\(t\)\)⏟loss gradient in weight space\.\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\mathcal\{S\}\}\\\!\\big\(\\phi\_\{\\theta\}\(t\)\\big\)=\\underbrace\{\\left\(\\frac\{\\partial\\phi\_\{\\theta\}\(t\)\}\{\\partial\\theta\}\\right\)^\{\\\!\\top\}\}\_\{\\text\{curve Jacobian\}\}\\underbrace\{\\nabla\_\{w\}\\mathcal\{L\}\_\{\\mathcal\{S\}\}\\\!\\big\(\\phi\_\{\\theta\}\(t\)\\big\)\}\_\{\\text\{loss gradient in weight space\}\}\.\(5\)Here∇wℒ𝒮​\(ϕθ​\(t\)\)\\nabla\_\{w\}\\mathcal\{L\}\_\{\\mathcal\{S\}\}\(\\phi\_\{\\theta\}\(t\)\)is the standard loss gradient evaluated at the weight vectorϕθ​\(t\)\\phi\_\{\\theta\}\(t\), and∂ϕθ​\(t\)∂θ\\frac\{\\partial\\phi\_\{\\theta\}\(t\)\}\{\\partial\\theta\}is the Jacobian of the Bézier parameterization \([3](https://arxiv.org/html/2609.17748#S3.E3)\) with respect to the control points, whose entries are the basis polynomials3​t​\(1−t\)23t\(1\-t\)^\{2\}and3​t2​\(1−t\)3t^\{2\}\(1\-t\)evaluated attt\. Substituting \([5](https://arxiv.org/html/2609.17748#S3.E5)\) into \([4](https://arxiv.org/html/2609.17748#S3.E4)\) gives the complete path gradient:

∇θℒpath​\(θ\)=𝔼t∼U⁡\(0,1\)​\[\(∂ϕθ​\(t\)∂θ\)⊤​∇wℒ𝒮​\(ϕθ​\(t\)\)\]\.\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\mathrm\{path\}\}\(\\theta\)=\\mathbb\{E\}\_\{t\\sim U\(0,1\)\}\\left\[\\left\(\\frac\{\\partial\\phi\_\{\\theta\}\(t\)\}\{\\partial\\theta\}\\right\)^\{\\top\}\\nabla\_\{w\}\\mathcal\{L\}\_\{\\mathcal\{S\}\}\\big\(\\phi\_\{\\theta\}\(t\)\\big\)\\right\]\.

### III\-BHow to Find Flat Minima: Sharpness\-Aware Minimization

The geometry of loss landscape minima strongly influences generalization: sharp minima, where the loss increases rapidly under small perturbations, tend to generalize poorly, whereas flat minima remain low\-loss throughout a local neighborhood\[[14](https://arxiv.org/html/2609.17748#bib.bib5)\]\. Sharpness\-Aware Minimization \(SAM\) formalizes this insight by seeking parameters whose entireℓ2\\ell\_\{2\}\-ball of radiusρ\>0\\rho\>0attains low loss\. The*sharpness\-aware loss*is defined as

ℒ𝒮SAM​\(w\)≜max‖ϵ‖2≤ρ⁡ℒ𝒮​\(w\+ϵ\),\\mathcal\{L\}\_\{\\mathcal\{S\}\}^\{\\mathrm\{SAM\}\}\(w\)\\triangleq\\max\_\{\\\|\\epsilon\\\|\_\{2\}\\leq\\rho\}\\mathcal\{L\}\_\{\\mathcal\{S\}\}\(w\+\\epsilon\),\(6\)whereϵ∈ℝd\\epsilon\\in\\mathbb\{R\}^\{d\}is a perturbation vector andρ\\rhocontrols the neighborhood size\. Since solving the inner maximization exactly is intractable, a first\-order Taylor expansion yields the closed\-form approximation for the optimal perturbation:

ϵ^​\(w\)=ρ​∇wℒ𝒮​\(w\)‖∇wℒ𝒮​\(w\)‖2\.\\hat\{\\epsilon\}\(w\)=\\rho\\frac\{\\nabla\_\{w\}\\mathcal\{L\}\_\{\\mathcal\{S\}\}\(w\)\}\{\\\|\\nabla\_\{w\}\\mathcal\{L\}\_\{\\mathcal\{S\}\}\(w\)\\\|\_\{2\}\}\.\(7\)The SAM gradient is then approximated as∇wℒ𝒮​\(w\+ϵ^​\(w\)\)\\nabla\_\{w\}\\mathcal\{L\}\_\{\\mathcal\{S\}\}\(w\+\\hat\{\\epsilon\}\(w\)\), dropping second\-order terms for computational efficiency\. This results in a two\-step procedure at each training iteration: first, compute the ascent perturbation using Equation \([7](https://arxiv.org/html/2609.17748#S3.E7)\), then evaluate the gradient at the perturbed point\.

These two formulations are complementary but have been studied in isolation: MC discovers low\-loss curves\[[2](https://arxiv.org/html/2609.17748#bib.bib1),[4](https://arxiv.org/html/2609.17748#bib.bib16)\]but imposes no constraint on sharpness along them, while SAM\[[14](https://arxiv.org/html/2609.17748#bib.bib5)\]enforces flatness within a local neighborhood of individual parameter vectors without extending to continuous trajectories\. To our knowledge, no existing method integrates sharpness constraints into the path\-finding objective\.

This gap motivates the central question of this paper:*can we find a connectivity path between independently trained solutions that is not only low\-loss but also uniformly flat within a local neighborhood along its entire trajectory?*We answer this question affirmatively in the following subsection\.

### III\-CHow to Find the Flat Path: Sharp Mode Connectivity

#### III\-C1Sharpness\-Aware Path Functional

Instead of evaluating the standard empirical lossℒ𝒮​\(ϕθ​\(t\)\)\\mathcal\{L\}\_\{\\mathcal\{S\}\}\(\\phi\_\{\\theta\}\(t\)\)at each point along the curve, we evaluate the worst\-case loss within aρ\\rho\-neighborhood\. The*sharpness\-aware path functional*is defined as

minθ⁡Jsharp​\(θ\)=∫01max‖ϵ‖2≤ρ⁡ℒ𝒮​\(ϕθ​\(t\)\+ϵ\)​𝑑t\.\\min\_\{\\theta\}J\_\{\\mathrm\{sharp\}\}\(\\theta\)=\\int\_\{0\}^\{1\}\\max\_\{\\\|\\epsilon\\\|\_\{2\}\\leq\\rho\}\\mathcal\{L\}\_\{\\mathcal\{S\}\}\\big\(\\phi\_\{\\theta\}\(t\)\+\\epsilon\\big\)\\,dt\.\(8\)This objective encourages points along the path, on average overtt, to lie in neighborhoods with low worst\-case loss, extending SAM’s local minimax reasoning from individual parameter vectors to a continuous connectivity trajectory\. Minimizing \([8](https://arxiv.org/html/2609.17748#S3.E8)\) encourages the optimizer to discover wide, smooth valleys connecting the modesw1w\_\{1\}andw2w\_\{2\}, rather than paths that merely achieve low loss at each point but may traverse sharp ridges vulnerable to distributional perturbation\.

#### III\-C2SAM\-on\-the\-Curve Approximation

Direct evaluation of the inner maximization in \([8](https://arxiv.org/html/2609.17748#S3.E8)\) is intractable\. Applying the first\-order SAM approximation at each curve point, the perturbation is

ϵ^​\(ϕθ​\(t\)\)=ρ​∇wℒ𝒮​\(ϕθ​\(t\)\)‖∇wℒ𝒮​\(ϕθ​\(t\)\)‖2\.\\hat\{\\epsilon\}\(\\phi\_\{\\theta\}\(t\)\)=\\rho\\frac\{\\nabla\_\{w\}\\mathcal\{L\}\_\{\\mathcal\{S\}\}\(\\phi\_\{\\theta\}\(t\)\)\}\{\\\|\\nabla\_\{w\}\\mathcal\{L\}\_\{\\mathcal\{S\}\}\(\\phi\_\{\\theta\}\(t\)\)\\\|\_\{2\}\}\.\(9\)Two approximations make this objective tractable\. First, the inner maximization at each curve point is replaced by the standard first\-order SAM perturbation \([9](https://arxiv.org/html/2609.17748#S3.E9)\), which computes the steepest ascent direction within theρ\\rho\-ball\. Second, the continuous integral overt∈\[0,1\]t\\in\[0,1\]is estimated via Monte Carlo sampling:KKpoints\{tk\}k=1K\\\{t\_\{k\}\\\}\_\{k=1\}^\{K\}are drawn uniformly from\[0,1\]\[0,1\]and the functional is approximated by their average\. These two approximations yield the practical objective:

Jsharp​\(θ\)≈1K​∑k=1Kℒ𝒮​\(ϕθ​\(tk\)\+ϵ^​\(ϕθ​\(tk\)\)\)\.J\_\{\\mathrm\{sharp\}\}\(\\theta\)\\approx\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathcal\{L\}\_\{\\mathcal\{S\}\}\\Big\(\\phi\_\{\\theta\}\(t\_\{k\}\)\+\\hat\{\\epsilon\}\\big\(\\phi\_\{\\theta\}\(t\_\{k\}\)\\big\)\\Big\)\.\(10\)

#### III\-C3Gradient with Respect to Curve Parameters

Letwt=ϕθ​\(t\)w\_\{t\}=\\phi\_\{\\theta\}\(t\)and define the SAM gradient at the perturbed point asgSAM​\(wt\)=∇wℒ𝒮​\(wt\+ϵ^​\(wt\)\)g\_\{\\mathrm\{SAM\}\}\(w\_\{t\}\)=\\nabla\_\{w\}\\mathcal\{L\}\_\{\\mathcal\{S\}\}\(w\_\{t\}\+\\hat\{\\epsilon\}\(w\_\{t\}\)\)\. By the chain rule:

∇θJsharp​\(θ\)≈𝔼t∼U⁡\(0,1\)​\[\(∂ϕθ​\(t\)∂θ\)⊤​gSAM​\(wt\)\],\\nabla\_\{\\theta\}J\_\{\\mathrm\{sharp\}\}\(\\theta\)\\approx\\mathbb\{E\}\_\{t\\sim U\(0,1\)\}\\left\[\\left\(\\frac\{\\partial\\phi\_\{\\theta\}\(t\)\}\{\\partial\\theta\}\\right\)^\{\\\!\\top\}g\_\{\\mathrm\{SAM\}\}\(w\_\{t\}\)\\right\],\(11\)where derivatives throughϵ^​\(wt\)\\hat\{\\epsilon\}\(w\_\{t\}\)are omitted for computational efficiency, following standard SAM practice\. Each update ofθ\\thetarequires: \(i\) computing∇wℒ𝒮​\(wt\)\\nabla\_\{w\}\\mathcal\{L\}\_\{\\mathcal\{S\}\}\(w\_\{t\}\), \(ii\) constructingϵ^​\(wt\)\\hat\{\\epsilon\}\(w\_\{t\}\), \(iii\) evaluating the loss gradient at the perturbed point, and \(iv\) multiplying by the curve Jacobian∂ϕθ​\(t\)/∂θ\\partial\\phi\_\{\\theta\}\(t\)/\\partial\\theta\. This procedure is computationally analogous to SAM, requiring two backpropagations per sampledtt, but operates on a curve in weight space rather than on a single parameter vector\.

#### III\-C4Complete Algorithm

Algorithm[1](https://arxiv.org/html/2609.17748#alg1)implements the SMC framework as a practical optimization procedure\. The goal is to find curve parametersθ\\thetathat yield a low\-loss connectivity path whose local neighborhood is uniformly flat, since flat regions in the loss landscape are associated with better generalization\[[14](https://arxiv.org/html/2609.17748#bib.bib5)\]\.

Algorithm 1Sharp Mode Connectivity \(SMC\)0:Endpoints

w1,w2w\_\{1\},w\_\{2\}, perturbation radius

ρ\\rho, learning rate

η\\eta, samples per step

KK, stabilizer

δ\\delta
Initialize curve parameters

θ=\(θ1,θ2\)\\theta=\(\\theta\_\{1\},\\theta\_\{2\}\)\{Cubic Bézier control points\}

whilenot convergeddo

Sample mini\-batch

ℬ\\mathcal\{B\}
Sample

\{tk\}k=1K∼U⁡\(0,1\)\\\{t\_\{k\}\\\}\_\{k=1\}^\{K\}\\sim U\(0,1\)
Gθ←0G\_\{\\theta\}\\leftarrow 0

for

k=1k=1to

KKdo

wk←ϕθ​\(tk\)w\_\{k\}\\leftarrow\\phi\_\{\\theta\}\(t\_\{k\}\)\{Interpolate via \([3](https://arxiv.org/html/2609.17748#S3.E3)\)\}

gk←∇wℒℬ​\(wk\)g\_\{k\}\\leftarrow\\nabla\_\{w\}\\mathcal\{L\}\_\{\\mathcal\{B\}\}\(w\_\{k\}\)\{Ascent direction\}

ϵ^k←ρ​gk/\(‖gk‖2\+δ\)\\hat\{\\epsilon\}\_\{k\}\\leftarrow\\rho\\,g\_\{k\}/\(\\\|g\_\{k\}\\\|\_\{2\}\+\\delta\)\{SAM perturbation\}

g~k←∇wℒℬ​\(wk\+ϵ^k\)\\tilde\{g\}\_\{k\}\\leftarrow\\nabla\_\{w\}\\mathcal\{L\}\_\{\\mathcal\{B\}\}\(w\_\{k\}\+\\hat\{\\epsilon\}\_\{k\}\)\{Descent gradient\}

Gθ←Gθ\+\(∂ϕθ​\(tk\)/∂θ\)⊤​g~kG\_\{\\theta\}\\leftarrow G\_\{\\theta\}\+\\left\(\\partial\\phi\_\{\\theta\}\(t\_\{k\}\)/\\partial\\theta\\right\)^\{\\\!\\top\}\\tilde\{g\}\_\{k\}

endfor

Gθ←Gθ/KG\_\{\\theta\}\\leftarrow G\_\{\\theta\}/K

θ←θ−η​Gθ\\theta\\leftarrow\\theta\-\\eta\\,G\_\{\\theta\}

endwhile

returnOptimized curve parameters

θ\\theta

The algorithm uses a single mini\-batchℬ\\mathcal\{B\}for both the ascent \(perturbation\) and descent \(update\) steps, ensuring consistency between the two gradient computations and reducing variance from batch\-level stochasticity\. The stabilizerδ\\deltaprevents division by zero in the perturbation normalization\.

## IVExperimental Results

### IV\-AModels and Datasets

We evaluate the proposed framework across three architectures spanning distinct design families:

- •ResNet\-18\[[12](https://arxiv.org/html/2609.17748#bib.bib11)\]: Modified for CIFAR\-10 with a3×33\\times 3initial convolution \(stride 1, padding 1, no bias\) replacing the standard7×77\\times 7convolution, and an identity mapping replacing the max\-pooling layer to accommodate the32×3232\\times 32spatial dimensions\. Serves as the primary architecture for all ablation studies\.
- •VGG16\-BN\[[13](https://arxiv.org/html/2609.17748#bib.bib10)\]: Standard VGG16 with Batch Normalization, providing a purely sequential architecture without residual connections\. Tests whether the framework generalizes beyond residual networks\.
- •ViT\-Tiny\[[23](https://arxiv.org/html/2609.17748#bib.bib9)\]: A compact Vision Transformer with patch embedding and multi\-head self\-attention, testing attention\-based architectures\.

Experiments are conducted on two datasets:

- •CIFAR\-10: 60,000 images in 10 classes \(32×3232\\times 32\)\. Used for all three architectures and all ablation studies\.
- •ImageNet\-100: A 100\-class subset of ImageNet \(224×224224\\times 224\)\. Used with ViT\-Tiny to test Mode Connectivity on higher\-resolution data with attention\-based architectures\.

Out\-of\-distribution evaluation uses theCIFAR\-10\-Cbenchmark\[[5](https://arxiv.org/html/2609.17748#bib.bib8),[24](https://arxiv.org/html/2609.17748#bib.bib6)\], comprising 15 corruption types at 5 severity levels\. We focus on the “Blur” category \(Gaussian, Defocus, Glass, and Motion blur\) as these systematically degrade spatial frequency information\.

### IV\-BImplementation Details

Batch Normalization statistics require special handling: interpolated running statistics are mathematically invalid for non\-linear networks\. We performrecalibrationby running 100 gradient\-free forward passes on clean training data at each evaluation pointttto recompute valid running means and variances\. This recalibration uses only clean data to prevent target distribution leakage\. The hyperparameters are summarized in Table[I](https://arxiv.org/html/2609.17748#S4.T1)\. All experiments use fixed random seeds to ensure full reproducibility\.

Table I:Hyperparameters for Sharp Mode Connectivity
### IV\-CEvaluation Protocol

#### IV\-C1Peak vs\. Midpoint Accuracy

We report two accuracy metrics along the path:*peak accuracy*\(maxt⁡Acc​\(t\)\\max\_\{t\}\\text\{Acc\}\(t\)\) to identify the single best model for ensembling and model soups, and*midpoint accuracy*\(t=0\.5t=0\.5\) to assess connectivity robustness, as the midpoint is furthest from the endpoints and most vulnerable to barriers and distribution shifts\.

#### IV\-C2Midpoint Loss Barrier

We define the midpoint loss barrier as:

Δ​ℒbarrier=ℒ⁡\(ϕθ​\(0\.5\)\)−12​\(ℒ⁡\(w1\)\+ℒ⁡\(w2\)\)\.\\Delta\\mathcal\{L\}\_\{\\mathrm\{barrier\}\}=\\mathcal\{L\}\\big\(\\phi\_\{\\theta\}\(0\.5\)\\big\)\-\\frac\{1\}\{2\}\\Big\(\\mathcal\{L\}\(w\_\{1\}\)\+\\mathcal\{L\}\(w\_\{2\}\)\\Big\)\.\(12\)A value of zero indicates perfect barrier\-free interpolation\. Positive values indicate sharp ridges; negative values indicate that the midpoint loss is lower than the average endpoint loss\.

#### IV\-C3Point\-wise Sharpness \(SAM Gap\)

We measure the sharpness of the loss surface at each curve pointϕθ​\(t\)\\phi\_\{\\theta\}\(t\)using the SAM Gap:

ΔSAM​\(t\)=max‖ϵ‖2≤ρ⁡ℒ𝒮​\(ϕθ​\(t\)\+ϵ\)−ℒ𝒮​\(ϕθ​\(t\)\)\.\\Delta\_\{\\mathrm\{SAM\}\}\(t\)=\\max\_\{\\\|\\epsilon\\\|\_\{2\}\\leq\\rho\}\\mathcal\{L\}\_\{\\mathcal\{S\}\}\\big\(\\phi\_\{\\theta\}\(t\)\+\\epsilon\\big\)\-\\mathcal\{L\}\_\{\\mathcal\{S\}\}\\big\(\\phi\_\{\\theta\}\(t\)\\big\)\.\(13\)This quantity measures how much the loss increases under worst\-case perturbation at positionttalong the curve\. A high SAM Gap indicates a sharp ridge; a low SAM Gap indicates a flat valley\. We report both the path\-averaged SAM GapΔ¯SAM=1\|T\|​∑t∈TΔSAM​\(t\)\\bar\{\\Delta\}\_\{\\mathrm\{SAM\}\}=\\frac\{1\}\{\|T\|\}\\sum\_\{t\\in T\}\\Delta\_\{\\mathrm\{SAM\}\}\(t\)and the midpoint valueΔSAM​\(0\.5\)\\Delta\_\{\\mathrm\{SAM\}\}\(0\.5\)\.

### IV\-DSanity Checks

SAM optimizer\.We verify the standalone SAM optimizer by comparing it against baseline SGD on ResNet\-18/CIFAR\-10 \(150 epochs, batch size 128, cosine annealing,ρ=0\.05\\rho=0\.05\)\. As shown in Fig\.[2](https://arxiv.org/html/2609.17748#S4.F2), SAM consistently outperforms SGD in test accuracy while both achieve near\-89% training accuracy, confirming that SAM finds flatter, better\-generalizing minima\.

![Refer to caption](https://arxiv.org/html/2609.17748v1/figures/sam-acc-1.png)Figure 2:SGD vs\. SAM training metrics on ResNet\-18/CIFAR\-10\. SAM achieves higher test accuracy, validating its sharpness\-aware optimization\.Mode Connectivity\.Two independent ResNet\-18 models \(w1w\_\{1\}: 91\.68%,w2w\_\{2\}: 92\.44%\) are connected via an optimized cubic Bézier curve\. After Batch Normalization recalibration, the path exhibits uniformly high performance: test accuracy remains above 90% across the entire trajectory \(Fig\.[3](https://arxiv.org/html/2609.17748#S4.F3)\), confirming the existence of a smooth, low\-loss connection between independently trained solutions\.

![Refer to caption](https://arxiv.org/html/2609.17748v1/figures/mc_loss_curve_bn.png)\(a\)Test loss along path
![Refer to caption](https://arxiv.org/html/2609.17748v1/figures/mc_acc_curve_bn.png)\(b\)Test accuracy along path

Figure 3:Mode Connectivity verification on ResNet\-18/CIFAR\-10: test loss \(a\) and accuracy \(b\) along the optimized cubic Bézier path after BN recalibration\. The path maintains\>\>90% accuracy across the entire trajectory, confirming a low\-loss, barrier\-free connection between independently trained solutions\.
### IV\-EClean Connectivity: Standard MC vs\. SMC

We first compare the standard MC path against the SMC path on clean test data across three architectures: ResNet\-18, VGG16\-BN, and ViT\-Tiny\.

#### IV\-E1CNN Architectures \(ResNet\-18 and VGG16\-BN\)

Fig\.[4](https://arxiv.org/html/2609.17748#S4.F4)presents the clean connectivity landscape for both CNN architectures\.

Figure 4:Clean connectivity landscape on CIFAR\-10\.Left: Cross\-entropy loss vs\. path parameterttfor the SMC model \(blue\) and standard MC \(red\) across both ResNet\-18 and VGG16\-BN architectures\.Right: Test accuracy \(%\)\. The SMC maintains a flat “U\-shape” loss profile and higher accuracy across the full trajectory, confirming that the benefit generalizes beyond residual architectures\.Fig\.[5](https://arxiv.org/html/2609.17748#S4.F5)plots the point\-wise sharpness \(SAM Gap, Eq\.[13](https://arxiv.org/html/2609.17748#S4.E13)\) at each curve location for both methods on ResNet\-18\.

![Refer to caption](https://arxiv.org/html/2609.17748v1/figures/sam_gap_profile.png)Figure 5:Point\-wise sharpness \(SAM Gap\) along the connectivity path for ResNet\-18 on CIFAR\-10\. The standard MC path \(red, dashed\) exhibits a sharp spike at the midpoint \(ΔSAM​\(0\.5\)=0\.6916\\Delta\_\{\\mathrm\{SAM\}\}\(0\.5\)=0\.6916\), revealing that the curve threads through a narrow ridge att=0\.5t=0\.5\. The SMC path \(blue, solid\) suppresses sharpness across the entire trajectory, peaking at only0\.35940\.3594at the midpoint, a48%48\\%reduction\. Both paths share identical endpoints \(t=0t=0andt=1t=1\), confirming that the sharpness reduction arises entirely from the path\-interior optimization\.ResNet\-18 \(CIFAR\-10\): The standard MC trajectory exhibits a distinct “V\-shape” with a loss peak of 0\.2851 at the midpoint\. In contrast, the SMC model maintains a remarkably stable, deep “U\-shape” with substantially lower loss throughout\. The standard MC model drops to 92\.07% accuracy at the midpoint, while the SMC maintains 92\.33%\. Notably, both frameworks discover a region of superior generalization att=0\.90t=0\.90, where the SMC achieves 94\.00%, surpassing both independently trained endpoints\.

VGG16\-BN \(CIFAR\-10\): As shown inFig\.[4](https://arxiv.org/html/2609.17748#S4.F4), SMC achieves a flatter loss profile and higher accuracy along the entire path\. The SMC attains a path\-averaged accuracy of90\.22%90\.22\\%vs\.89\.61%89\.61\\%for standard MC, with path loss reduced from 0\.406 to 0\.299, a26\.4%26\.4\\%reduction\. Despite lacking skip connections that smooth the loss landscape, VGG16\-BN exhibits clear mode connectivity, confirming that the SAM\-based framework generalizes beyond residual architectures\.

###### Definition 1\(Path Loss Flatness\)\.

We define the*path loss flatness*as the variance of the loss evaluated at uniformly spaced points along the curve:

F⁡\(θ\)=Vart∈T⁡\[ℒ𝒮​\(ϕθ​\(t\)\)\]=1\|T\|​∑t∈T\(ℒ𝒮​\(ϕθ​\(t\)\)−ℒ¯\)2,F\(\\theta\)=\\operatorname\{Var\}\_\{t\\in T\}\\big\[\\mathcal\{L\}\_\{\\mathcal\{S\}\}\(\\phi\_\{\\theta\}\(t\)\)\\big\]=\\frac\{1\}\{\|T\|\}\\sum\_\{t\\in T\}\\Big\(\\mathcal\{L\}\_\{\\mathcal\{S\}\}\(\\phi\_\{\\theta\}\(t\)\)\-\\bar\{\\mathcal\{L\}\}\\Big\)^\{2\},\(14\)whereTTis a set of uniformly spaced evaluation points along the curve andℒ¯=1\|T\|​∑t∈Tℒ𝒮​\(ϕθ​\(t\)\)\\bar\{\\mathcal\{L\}\}=\\frac\{1\}\{\|T\|\}\\sum\_\{t\\in T\}\\mathcal\{L\}\_\{\\mathcal\{S\}\}\(\\phi\_\{\\theta\}\(t\)\)is the path\-averaged loss\. Lower values indicate a more uniform loss profile along the path: the curve traverses a consistently deep valley rather than oscillating between high\- and low\-loss regions\.

Table[II](https://arxiv.org/html/2609.17748#S4.T2)summarizes the clean baseline performance across all three architectures\. The loss flatness metric \(Definition[1](https://arxiv.org/html/2609.17748#Thmmydef1)\) quantifies the uniformity of the loss profile along the path\. For ResNet\-18, the SMC achieves a26\.7×26\.7\\timesimprovement in flatness \(0\.0021 vs\. 0\.056\), confirming that SAM’s min\-max regularization successfully enforces uniform curvature constraints across the connectivity manifold\.

The SAM Gap profile \(Fig\.[5](https://arxiv.org/html/2609.17748#S4.F5)\) reveals the geometric mechanism behind this improvement\. Along the standard MC path, point\-wise sharpness rises sharply toward the midpoint, peaking atΔSAM​\(0\.5\)=0\.6916\\Delta\_\{\\mathrm\{SAM\}\}\(0\.5\)=0\.6916, indicating that the path traverses an increasingly sharp ridge as it departs from either endpoint\. This is precisely the location where accuracy degrades most under distribution shift \(Section[IV\-F](https://arxiv.org/html/2609.17748#S4.SS6)\)\. The SMC path maintains substantially lower sharpness throughout, with the midpoint SAM Gap reduced to0\.35940\.3594\(a48%48\\%reduction\) and the path\-averaged SAM Gap reduced from0\.34990\.3499to0\.22710\.2271\(a35%35\\%reduction\)\. This provides direct empirical evidence that SMC’s sharpness\-aware objective flattens the loss surface not only on the curve but in its transverse neighborhood, establishing the causal link between path\-wise flatness and the OOD robustness gains reported in Section[IV\-F](https://arxiv.org/html/2609.17748#S4.SS6)\.

Table II:Clean Baseline Performance Metrics
#### IV\-E2Vision Transformer \(ViT\-Tiny\)

The ViT\-Tiny results \(Fig\.[6](https://arxiv.org/html/2609.17748#S4.F6)\) reveal a distinct connectivity structure compared to CNNs\. While both architectures exhibit low\-loss connectivity away from the midpoint, ViT\-Tiny displays a residual midpoint barrier: accuracy dips to∼\\sim77% on CIFAR\-10 and∼\\sim68% on ImageNet\-100 att=0\.5t=0\.5, compared to the∼\\sim87% and∼\\sim73% at the endpoints, respectively\. Crucially, however, the SMC model consistently outperforms standard MC across both datasets\. On CIFAR\-10, the SMC achieves a path\-averaged accuracy of 86\.08% vs\. 85\.84% for standard MC, with lower path loss \(0\.485 vs\. 0\.541\)\. On ImageNet\-100, the advantage is more pronounced: 74\.32% vs\. 73\.32% path accuracy \(1\.003 vs\. 1\.120 path loss\)\. The SMC also discovers points along the path that surpass both endpoints, reaching 87\.63% att=0\.85t=0\.85on CIFAR\-10 and 75\.65% att=0\.30t=0\.30on ImageNet\-100, confirming that the sharpness\-aware path functional identifies superior solutions even in attention\-based architectures\.

Figure 6:ViT\-Tiny clean connectivity on CIFAR\-10 \(blue\) and ImageNet\-100 \(red\)\. Solid lines denote the SMC path; dashed lines denote standard MC\.Left: Cross\-entropy loss along the path parametertt\.Right: Test accuracy \(%\)\. The SMC achieves lower loss and higher accuracy along the path interior on both datasets\.These results reframe the ViT connectivity landscape: rather than a failure of the framework, the residual midpoint barrier reflects an architectural property of attention mechanisms, while the consistent SMC advantage demonstrates that sharpness\-aware path optimization remains effective across architecture families\. We proceed to stress\-test the CNN architectures under severe distribution shifts\.

### IV\-FOut\-of\-Distribution Robustness \(CIFAR\-10\-C\)

We evaluate the SMC framework under severe distribution shifts using blur corruptions from CIFAR\-10\-C at five severity levels, comparing ResNet\-18 and VGG16\-BN\.

#### IV\-F1Accuracy Under Distribution Shift

Fig\.[7](https://arxiv.org/html/2609.17748#S4.F7)plots the accuracy across severity levels for all four blur types\. The SMC model consistently outperforms standard MC, with the advantage generally growing with corruption severity\. Under Gaussian blur \(Severity 5\), the SMC maintains 35\.98% vs\. 30\.27% for standard MC; under Defocus blur, 52\.44% vs\. 46\.35%\.

Figure 7:ResNet\-18 midpoint accuracy across blur corruption severities for Gaussian, Defocus, Glass, and Motion blur on CIFAR\-10\-C\. The SMC consistently outperforms standard MC, with the advantage generally scaling with severity\. The clean baseline is marked with a dashed gray line\.Table[III](https://arxiv.org/html/2609.17748#S4.T3)quantifies the Severity 5 advantage across both CNN architectures\. The maximum absolute improvement is\+6\.09%\+6\.09\\%under Defocus blur for ResNet\-18 and\+4\.08%\+4\.08\\%for VGG16\-BN\.

Table III:Midpoint Accuracy Under Severe Shift \(Severity 5\): ResNet\-18 and VGG16\-BNTable[IV](https://arxiv.org/html/2609.17748#S4.T4)reports the SMC accuracy advantage across all severity levels for both architectures, revealing a broadly monotonic scaling trend: as corruption intensifies, the advantage of flat\-basin connectivity generally increases\. The trend is strongest for Gaussian, Defocus, and Motion blur, though Glass blur deviates with non\-monotonic reversals\. Notably, the VGG16\-BN SMC consistently outperforms standard MC from Severity 1 onward, whereas for ResNet\-18 the advantage emerges at Severity 2, suggesting that purely sequential architectures are more uniformly sensitive to sharpness\-aware regularization\.

Table IV:Midpoint SMC Accuracy Advantage \(\+%\+\\%\) Over Standard MC Across Severities
#### IV\-F2Loss Barrier Analysis

Fig\.[8](https://arxiv.org/html/2609.17748#S4.F8)plots the loss barrier \([12](https://arxiv.org/html/2609.17748#S4.E12)\) averaged across blur types as severity increases for both ResNet\-18 and VGG16\-BN\. For both architectures, the standard MC barrier remains near zero or trends slightly positive, indicating that the narrow connecting valley collapses under distribution shift\. Remarkably, the SMC barrier plunges into deeply negative territory: ResNet\-18 reaches−0\.47\-0\.47and VGG16\-BN reaches−0\.90\-0\.90at Severity 5, demonstrating that the optimized midpoint achieves lower loss than the average of the independently trained endpoint losses under corruption\. We note that because the barrier is defined relative to the endpoint average \(Eq\.[12](https://arxiv.org/html/2609.17748#S4.E12)\), a negative value does not guarantee that the midpoint outperforms both endpoints individually; however, the consistently large negative magnitudes across corruption types suggest that the midpoint occupies a genuinely more robust region of the loss landscape\. The deeper VGG16\-BN barriers suggest that purely sequential architectures benefit even more from flat\-basin connectivity, plausibly because the absence of residual skip connections produces sharper ridges between independently trained solutions\[[11](https://arxiv.org/html/2609.17748#bib.bib19)\], making sharpness\-aware regularization more impactful\. The wider confidence bands at high severity indicate greater variability across blur types\.

![Refer to caption](https://arxiv.org/html/2609.17748v1/figures/3_barrier_at_peaks.png)Figure 8:Loss barrier \(averaged over blur types\) vs\. corruption severity for ResNet\-18 and VGG16\-BN on CIFAR\-10\-C\. The SMC maintains deeply negative barriers for both architectures, while standard MC barriers remain near zero\. VGG16\-BN SMC barriers are consistently deeper than ResNet\-18, suggesting purely sequential architectures benefit more from flat\-basin connectivity\. Shaded regions indicate±1\\pm 1std across blur types\.Table[V](https://arxiv.org/html/2609.17748#S4.T5)reports per\-corruption Severity 5 barriers for both architectures\. Under Gaussian blur, the ResNet\-18 SMC achieves−1\.3582\-1\.3582vs\.\+0\.3793\+0\.3793for standard MC, a profound geometric advantage\. The VGG16\-BN SMC replicates this pattern with deeply negative barriers across all corruption types, confirming architectural transferability\.

Table V:Midpoint Loss Barrier at Severity 5: ResNet\-18 and VGG16\-BN
#### IV\-F3Interpolation Advantage Dynamics

Fig\.[9](https://arxiv.org/html/2609.17748#S4.F9)plots the pointwise accuracy advantageAdv​\(t\)=AccSMC​\(t\)−AccMC​\(t\)\\text\{Adv\}\(t\)=\\text\{Acc\}\_\{\\text\{SMC\}\}\(t\)\-\\text\{Acc\}\_\{\\text\{MC\}\}\(t\)across the full path for Severity 3 and 5 under all four blur types\. Volatile positive spikes at the midpoint and quartiles indicate regions where standard MC is most vulnerable under interpolation\. VGG16\-BN exhibits amplified advantages, peaking at\+6%\+6\\%under Gaussian and Defocus blur, roughly double the ResNet\-18 peaks\. Glass blur yields the smallest, noisiest gains for both architectures\.

Figure 9:Interpolation advantage \(AccSMC−AccMC\\text\{Acc\}\_\{\\text\{SMC\}\}\-\\text\{Acc\}\_\{\\text\{MC\}\}\) along the full path for ResNet\-18 \(solid\) and VGG16\-BN \(dashed\) under Severity 3 and 5 across four blur types on CIFAR\-10\-C\. Volatile positive spikes and quartiles reveal where standard MC is most fragile\. VGG16\-BN exhibits amplified advantages \(up to\+6%\+6\\%under Gaussian and Defocus blur\), while Glass blur yields the smallest, noisiest gains for both architectures\.

### IV\-GAblation: Impact of SAM Perturbation Radiusρ\\rho

The preceding OOD results demonstrate that the SMC framework consistently improves robustness, but the magnitude of this improvement depends on the SAM perturbation radiusρ\\rho\. To isolate this effect, we sweepρ∈\{0\.01,0\.02,0\.05,0\.1,0\.2,0\.5\}\\rho\\in\\\{0\.01,0\.02,0\.05,0\.1,0\.2,0\.5\\\}with frozen endpoints and evaluate under Gaussian blur at Severity 3 and 5 for both ResNet\-18 and VGG16\-BN\. Fig\.[10](https://arxiv.org/html/2609.17748#S4.F10)summarizes the resulting trade\-off between the loss barrier and midpoint accuracy for ResNet\-18\.

Figure 10:ResNet\-18 dual\-axis plot showing the loss barrier and midpoint accuracy across varying SAM perturbation radiiρ\\rhounder Gaussian blur \(CIFAR\-10\-C\)\. Increasingρ\\rhoimproves connectivity up to an optimal range, beyond which excessive noise collapses the path\.![Refer to caption](https://arxiv.org/html/2609.17748v1/resnet18_landscape_combined.png)Figure 11:ResNet\-18 accuracy landscape heatmaps across the path parameter \(tt\) and SAM perturbation radius \(ρ\\rho\) under Gaussian blur\. An optimalρ\\rhorange is observed; excessive budgets collapse the path\. The wider optimal range at Severity 5 confirms that stronger regularization is needed under more severe distribution shift\.![Refer to caption](https://arxiv.org/html/2609.17748v1/vgg16_landscape_combined.png)Figure 12:VGG16\-BN accuracy landscape heatmaps across the path parameter \(tt\) and SAM perturbation radius \(ρ\\rho\) under Gaussian blur\. The pattern mirrors ResNet\-18, confirming that the benefit is architecture\-agnostic\.At Severity 3, increasingρ\\rhostrictly improves the connectivity bridge up toρ=0\.1\\rho=0\.1, after which the excessive noise injection collapses the path\. At Severity 5, the optimizer requires stronger regularization, and increasingρ\\rhoover a broader range consistently enhances performance\. This pattern holds across both architectures \(Figs\.[11](https://arxiv.org/html/2609.17748#S4.F11)and[12](https://arxiv.org/html/2609.17748#S4.F12)\), with moderate perturbation budgets \(ρ≈0\.05\\rho\\approx 0\.05to0\.10\.1\) generating the optimal accuracy landscapes\. These results confirm that the wide\-valley constraint is critical under severe distribution shift and that the benefit is architecture\-agnostic\.

### IV\-HVision Transformer Connectivity

Having established that SMC consistently benefits CNN architectures across corruption types and perturbation budgets, we now examine the computational cost and architectural implications for attention\-based architectures\. The clean connectivity results for ViT\-Tiny on CIFAR\-10 and ImageNet\-100 were presented in Section[IV\-E](https://arxiv.org/html/2609.17748#S4.SS5)\(Table[II](https://arxiv.org/html/2609.17748#S4.T2)and Fig\.[6](https://arxiv.org/html/2609.17748#S4.F6)\), where the SMC consistently outperformed standard MC despite a residual midpoint barrier absent in CNNs\.

Table[VI](https://arxiv.org/html/2609.17748#S4.T6)reports the computational cost\. The SMC incurs a5\.6×5\.6\\timesper\-epoch overhead due to SAM’s two\-backpropagation structure, with 200 training epochs \(vs\. 50 for CNNs\) to accommodate attention\-based architectures\.

Table VI:ViT\-Tiny Training Time and Computational Cost \(CIFAR\-10\)One plausible contributing factor to the residual midpoint barrier in ViT\-Tiny is the permutation symmetry of multi\-head self\-attention: query\-key\-value projections can be arbitrarily reordered across heads without changing the network function, so two independently trained ViTs may encode functionally equivalent but weight\-space permuted representations\. Cubic interpolation between such permuted representations may partially disrupt learned token\-mixing structure\. However, we have not empirically verified this conjecture; testing it by applying permutation\-alignment techniques \(e\.g\., Git Re\-Basin\[[20](https://arxiv.org/html/2609.17748#bib.bib23)\], optimal transport\) prior to path optimization is a promising direction for eliminating this residual barrier\. A systematic OOD evaluation of ViT\-Tiny under CIFAR\-10\-C corruptions, analogous to the CNN analysis in Section[IV\-F](https://arxiv.org/html/2609.17748#S4.SS6), remains an important direction for future work\.

### IV\-IDiscussion

The deeply negative loss barriers under distribution shift \(Table[V](https://arxiv.org/html/2609.17748#S4.T5)\) arise from asymmetric optimization: endpoints are trained with standard SGD without flatness constraints, while every interior point is optimized byJsharp​\(θ\)J\_\{\\mathrm\{sharp\}\}\(\\theta\), which penalizes worst\-case loss within aρ\\rho\-ball\. Under distribution shift, the non\-regularized endpoints degrade faster than the SAM\-regularized interior, producing negative barriers that suggest the midpoint occupies a more robust region of the loss landscape than the endpoint average\. An important caveat is that our framework guarantees*weight\-space*flatness \(max‖ϵ‖2≤ρ⁡ℒ⁡\(w\+ϵ\)≈ℒ⁡\(w\)\\max\_\{\\\|\\epsilon\\\|\_\{2\}\\leq\\rho\}\\mathcal\{L\}\(w\+\\epsilon\)\\approx\\mathcal\{L\}\(w\)\), which need not imply*input\-space*robustness \(max‖δ‖∞≤ε⁡ℒ⁡\(x\+δ,w\)\\max\_\{\\\|\\delta\\\|\_\{\\infty\}\\leq\\varepsilon\}\\mathcal\{L\}\(x\+\\delta;w\)\)\. Practitioners should consider this distinction when deploying interpolated models in safety\-critical applications\.

## VConclusion

This paper proposed SMC, which integrates SAM’s minimax objective into Mode Connectivity’s path\-finding formulation to discover parametric curves in weight space that are simultaneously low\-loss and uniformly flat\. Experiments across CNN \(ResNet\-18, VGG16\-BN\) and Transformer \(ViT\-Tiny\) architectures on CIFAR\-10 and ImageNet\-100 show that SMC consistently produces flatter connectivity paths, yields meaningful accuracy gains under severe distribution shift, and discovers path\-interior solutions that can outperform the average endpoint loss, as evidenced by deeply negative loss barriers\. These results establish path\-wise flatness as a practical geometric principle for robust generalization\.

The current evaluation is limited to a single endpoint pair per architecture and does not ablate SAM\-trained endpoints with standard MC or permutation\-aligned linear interpolation\. Future work includes adaptive perturbation scheduling along the curve, integration of permutation\-alignment methods to reduce the residual ViT midpoint barrier, and evaluation of adversarial robustness along the optimized path using gradient\-based attacks \(e\.g\., PGD, FGSM\[[25](https://arxiv.org/html/2609.17748#bib.bib7)\]\) to determine whether weight\-space flatness confers any resistance to input\-space perturbations\.

## References

- \[1\]P\. Izmailov, D\. Podoprikhin, T\. Garipov, D\. Vetrov, and A\. G\. Wilson\(2018\)Averaging weights leads to wider optima and better generalization\.InConference on Uncertainty in Artificial Intelligence \(UAI\),Cited by:[§I](https://arxiv.org/html/2609.17748#S1.p1.1),[§II\-C](https://arxiv.org/html/2609.17748#S2.SS3.p1.1)\.
- \[2\]T\. Garipov, P\. Izmailov, D\. Podoprikhin, D\. Vetrov, and A\. G\. Wilson\(2018\)Loss surfaces, mode connectivity, and fast ensembling of dnns\.External Links:1802\.10026Cited by:[§I](https://arxiv.org/html/2609.17748#S1.p1.1),[§I](https://arxiv.org/html/2609.17748#S1.p2.1),[§II\-C](https://arxiv.org/html/2609.17748#S2.SS3.p1.1),[§III\-A](https://arxiv.org/html/2609.17748#S3.SS1.p1.1),[§III\-B](https://arxiv.org/html/2609.17748#S3.SS2.p2.1)\.
- \[3\]M\. Wortsman, G\. Ilharco, S\. Y\. Gadre, R\. Roelofs, R\. Gontijo\-Lopes, A\. S\. Morcos, H\. Namkoong, A\. Farhadi, Y\. Carmon, S\. Kornblith, and L\. Schmidt\(2022\)Model soups: averaging weights of multiple fine\-tuned models improves accuracy without increasing inference time\.InProceedings of the 39th International Conference on Machine Learning \(ICML\),Cited by:[§I](https://arxiv.org/html/2609.17748#S1.p1.1),[§II\-C](https://arxiv.org/html/2609.17748#S2.SS3.p2.1)\.
- \[4\]F\. Draxler, K\. Veschgini, M\. Salmhofer, and F\. Hamprecht\(2018\)Essentially no barriers in neural network energy landscape\.InProceedings of the International Conference on Machine Learning \(ICML\),Cited by:[§I](https://arxiv.org/html/2609.17748#S1.p2.1),[§II\-C](https://arxiv.org/html/2609.17748#S2.SS3.p1.1),[§III\-B](https://arxiv.org/html/2609.17748#S3.SS2.p2.1)\.
- \[5\]D\. Hendrycks and T\. Dietterich\(2019\)Benchmarking neural network robustness to common corruptions and perturbations\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§I](https://arxiv.org/html/2609.17748#S1.p3.1),[§IV\-A](https://arxiv.org/html/2609.17748#S4.SS1.p1.3)\.
- \[6\]C\. Zhang, S\. Bengio, M\. Hardt, B\. Recht, and O\. Vinyals\(2017\)Understanding deep learning requires rethinking generalization\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§II\-A](https://arxiv.org/html/2609.17748#S2.SS1.p1.1)\.
- \[7\]Y\. Jiang, B\. Neyshabur, H\. Mobahi, D\. Krishnan, and S\. Bhojanapalli\(2020\)Fantastic generalization measures and where to find them\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§II\-A](https://arxiv.org/html/2609.17748#S2.SS1.p1.1)\.
- \[8\]S\. Hochreiter and J\. Schmidhuber\(1997\)Flat minima\.Neural Computation9\(1\),pp\. 1–42\.Cited by:[§II\-A](https://arxiv.org/html/2609.17748#S2.SS1.p2.1)\.
- \[9\]N\. S\. Keskar, D\. Mudigere, J\. Nocedal, M\. Smelyanskiy, and P\. T\. P\. Tang\(2017\)On large\-batch training for deep learning: generalization gap and sharp minima\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§II\-A](https://arxiv.org/html/2609.17748#S2.SS1.p2.1)\.
- \[10\]B\. Neyshabur, S\. Bhojanapalli, D\. McAllester, and N\. Srebro\(2017\)Exploring generalization in deep nets\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§II\-A](https://arxiv.org/html/2609.17748#S2.SS1.p2.1)\.
- \[11\]H\. Li, Z\. Xu, G\. Taylor, C\. Studer, and T\. Goldstein\(2018\)Visualizing the loss landscape of neural nets\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.31\.Cited by:[§II\-A](https://arxiv.org/html/2609.17748#S2.SS1.p3.1),[§IV\-F2](https://arxiv.org/html/2609.17748#S4.SS6.SSS2.p1.1)\.
- \[12\]K\. He, X\. Zhang, S\. Ren, and J\. Sun\(2016\)Deep residual learning for image recognition\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 770–778\.Cited by:[§II\-A](https://arxiv.org/html/2609.17748#S2.SS1.p3.1),[1st item](https://arxiv.org/html/2609.17748#S4.I1.i1.p1.1)\.
- \[13\]K\. Simonyan and A\. Zisserman\(2015\)Very deep convolutional networks for large\-scale image recognition\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§II\-A](https://arxiv.org/html/2609.17748#S2.SS1.p3.1),[2nd item](https://arxiv.org/html/2609.17748#S4.I1.i2.p1.1)\.
- \[14\]P\. Foret, A\. Kleiner, H\. Mobahi, and B\. Neyshabur\(2020\)Sharpness\-aware minimization for efficiently improving generalization\.External Links:2010\.01412Cited by:[§II\-B](https://arxiv.org/html/2609.17748#S2.SS2.p1.1),[§III\-B](https://arxiv.org/html/2609.17748#S3.SS2.p1.1),[§III\-B](https://arxiv.org/html/2609.17748#S3.SS2.p2.1),[§III\-C4](https://arxiv.org/html/2609.17748#S3.SS3.SSS4.p1.1)\.
- \[15\]J\. Kwon, J\. Kim, H\. Park, and I\. K\. Choi\(2021\)ASAM: adaptive sharpness\-aware minimization for scale\-invariant learning of deep neural networks\.InProceedings of the 38th International Conference on Machine Learning \(ICML\),Vol\.139,pp\. 5905–5914\.Cited by:[§II\-B](https://arxiv.org/html/2609.17748#S2.SS2.p1.1)\.
- \[16\]J\. Zhuang, B\. Gong, L\. Yuan, Y\. Cui, H\. Adam, N\. C\. Dvornek, S\. Tatikonda, J\. S\. Duncan, and T\. Liu\(2022\)Surrogate gap minimization improves sharpness\-aware training\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=edONMAnhLu-)Cited by:[§II\-B](https://arxiv.org/html/2609.17748#S2.SS2.p1.1)\.
- \[17\]W\. J\. Maddox, P\. Izmailov, T\. Garipov, D\. P\. Vetrov, and A\. G\. Wilson\(2019\)A simple baseline for Bayesian uncertainty in deep learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),pp\. 13153–13164\.Cited by:[§II\-C](https://arxiv.org/html/2609.17748#S2.SS3.p1.1)\.
- \[18\]J\. Ren, P\. Chen, and R\. Wang\(2025\)Revisiting mode connectivity in neural networks with bezier surface\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 101867–101884\.Cited by:[§II\-C](https://arxiv.org/html/2609.17748#S2.SS3.p1.1)\.
- \[19\]R\. Entezari, H\. Sedghi, O\. Saukh, and B\. Neyshabur\(2022\)The role of permutation invariance in linear mode connectivity of neural networks\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=1K27E1gU-w)Cited by:[§II\-C](https://arxiv.org/html/2609.17748#S2.SS3.p2.1)\.
- \[20\]S\. K\. Ainsworth, J\. Hayase, and S\. Srinivasa\(2023\)Git re\-basin: merging models modulo permutation symmetries\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§II\-C](https://arxiv.org/html/2609.17748#S2.SS3.p2.1),[§IV\-H](https://arxiv.org/html/2609.17748#S4.SS8.p3.1)\.
- \[21\]Y\. Shi and R\. Wang\(2025\)Exploring nonlinear pathway in parameter space for machine unlearning\.arXiv preprint arXiv:2505\.10859\.Cited by:[§II\-C](https://arxiv.org/html/2609.17748#S2.SS3.p2.1)\.
- \[22\]H\. S\. Kim, G\. Luo, C\. Chen, B\. Wang, Y\. Duan, and R\. Wang\(2026\)MoCo\-ea: exploiting adversarial mode connectivity for efficient evolutionary attacks\.arXiv preprint arXiv:2605\.18919\.Cited by:[§II\-C](https://arxiv.org/html/2609.17748#S2.SS3.p2.1)\.
- \[23\]A\. Dosovitskiy, L\. Beyer, A\. Kolesnikov, D\. Weissenborn, X\. Zhai, T\. Unterthiner, M\. Dehghani, M\. Minderer, G\. Heigold, S\. Gelly, J\. Uszkoreit, and N\. Houlsby\(2021\)An image is worth 16x16 words: transformers for image recognition at scale\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[3rd item](https://arxiv.org/html/2609.17748#S4.I1.i3.p1.1)\.
- \[24\]D\. Hendrycks and T\. Dietterich\(2019\)CIFAR\-10\-c \[data set\]\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.2535967),[Link](https://doi.org/10.5281/zenodo.2535967)Cited by:[§IV\-A](https://arxiv.org/html/2609.17748#S4.SS1.p1.3)\.
- \[25\]H\. Kim\(2020\)Torchattacks: a pytorch repository for adversarial attacks\.arXiv preprint arXiv:2010\.01950\.External Links:[Link](https://github.com/Harry24k/adversarial-attacks-pytorch)Cited by:[§V](https://arxiv.org/html/2609.17748#S5.p2.1)\.