How Far Can Sharpness and Complexity Jointly Explain Generalization?

# How Far Can Sharpness and Complexity Jointly Explain Generalization?
Source: [https://arxiv.org/html/2606.29043](https://arxiv.org/html/2606.29043)
###### Abstract

Sharpness and complexity are two central factors in the generalization analysis of deep neural networks\. Existing quantitative evaluations of generalization measures have largely focused on individual scalar measures, leaving the joint explanatory power of sharpness and complexity largely unexplored\. This work studies how far sharpness and complexity can*jointly*explain generalization\. We use linear regression and introduce a Pareto\-based analysis to quantitatively evaluate the joint explanatory power of these two factors\. Beyond the existing parameter\-level definitions, we further propose realizations of sharpness and complexity that are closer to function space and less dependent on raw parameter representations\. We find that function\-oriented definitions of these two quantities expand the explanatory scope of the two\-factor view beyond what is achieved by existing parameter\-level metrics\. Overall, our results support the sharpness\-complexity perspective as an informative lens for understanding generalization across diverse settings\. At the same time, the remaining failures indicate that whether this two\-factor view can serve as a complete theory of generalization remains open\.

## 1Introduction

Modern deep networks often achieve strong predictive performance in highly overparameterized regimes\. This empirical success has exposed a persistent gap between practical performance and theoretical understanding\. Classical capacity\-based viewpoints cannot explain the fact that highly overparameterized networks can easily fit random labels, while the same architectures often generalize well on structured data\[[38](https://arxiv.org/html/2606.29043#bib.bib1)\]\. This tension has motivated a broad line of work seeking more refined explanations for generalization, including implicit biases induced by architectures and optimization algorithms\[[31](https://arxiv.org/html/2606.29043#bib.bib4),[37](https://arxiv.org/html/2606.29043#bib.bib5),[13](https://arxiv.org/html/2606.29043#bib.bib6),[23](https://arxiv.org/html/2606.29043#bib.bib7),[36](https://arxiv.org/html/2606.29043#bib.bib8)\], norm\- and margin\-based bounds\[[32](https://arxiv.org/html/2606.29043#bib.bib37),[4](https://arxiv.org/html/2606.29043#bib.bib38),[29](https://arxiv.org/html/2606.29043#bib.bib36)\], PAC\-Bayes analyses\[[25](https://arxiv.org/html/2606.29043#bib.bib17),[26](https://arxiv.org/html/2606.29043#bib.bib18),[27](https://arxiv.org/html/2606.29043#bib.bib19),[10](https://arxiv.org/html/2606.29043#bib.bib20),[30](https://arxiv.org/html/2606.29043#bib.bib35)\], sharpness and flatness\[[15](https://arxiv.org/html/2606.29043#bib.bib9),[18](https://arxiv.org/html/2606.29043#bib.bib10),[9](https://arxiv.org/html/2606.29043#bib.bib11),[11](https://arxiv.org/html/2606.29043#bib.bib15),[3](https://arxiv.org/html/2606.29043#bib.bib14)\], double descent and benign overfitting\[[28](https://arxiv.org/html/2606.29043#bib.bib42),[5](https://arxiv.org/html/2606.29043#bib.bib43)\], and large\-scale empirical studies of generalization measures\[[33](https://arxiv.org/html/2606.29043#bib.bib2),[16](https://arxiv.org/html/2606.29043#bib.bib3)\]\.

Despite their diversity, many successful approaches are built on two recurring factors: sharpness and complexity\.*Sharpness*\(S\)\(S\)characterizes the sensitivity of a learned solution to perturbations and is commonly quantified through Hessian\-based metrics\[[9](https://arxiv.org/html/2606.29043#bib.bib11),[17](https://arxiv.org/html/2606.29043#bib.bib12),[21](https://arxiv.org/html/2606.29043#bib.bib13)\]or perturbation\-induced loss\-increase metrics, such as Sharpness\-Aware Minimization \(SAM\)\[[11](https://arxiv.org/html/2606.29043#bib.bib15)\]\.*Complexity*\(C\)\(C\)captures the size or information content of the learned model or its hypothesis class, and plays a central role in classical capacity\-based bounds, PAC and PAC\-Bayes theory, and many modern generalization analyses\. Common complexity metrics include weight norm, path norm\[[32](https://arxiv.org/html/2606.29043#bib.bib37)\], spectral norm product\[[4](https://arxiv.org/html/2606.29043#bib.bib38)\], and posterior–prior KL divergence\[[25](https://arxiv.org/html/2606.29043#bib.bib17),[26](https://arxiv.org/html/2606.29043#bib.bib18),[12](https://arxiv.org/html/2606.29043#bib.bib25),[29](https://arxiv.org/html/2606.29043#bib.bib36)\]\.

The practical importance of these two factors is perhaps best illustrated by SAM\[[11](https://arxiv.org/html/2606.29043#bib.bib15),[20](https://arxiv.org/html/2606.29043#bib.bib16),[3](https://arxiv.org/html/2606.29043#bib.bib14)\]\. While SAM explicitly regularizes sharpness, its strongest empirical performance is typically achieved when combined with weight decay, one of the most widely used complexity regularizers\[[19](https://arxiv.org/html/2606.29043#bib.bib39),[22](https://arxiv.org/html/2606.29043#bib.bib40),[39](https://arxiv.org/html/2606.29043#bib.bib41)\]\. More broadly, many successful training algorithms and theoretical frameworks can be interpreted as balancing sharpness and complexity in different forms\.

Despite the widespread use of sharpness and complexity in both theory and practice, their*joint explanatory power*has not beenquantitativelyexamined\. A closely related study is Jiang et al\.\[[16](https://arxiv.org/html/2606.29043#bib.bib3)\], which conducted a large\-scale evaluation of generalization measures and reported largely pessimistic results on the predictive power of existing measures\. However, their analysis primarily treated each measure as an individual scalar predictor, leaving open how much generalization can be explained by sharpness and complexity jointly\.

In this work, we study the two\-factor view of generalization and ask the following central questions:

To what extent do sharpness and complexity jointly explain generalization?\(∗\*\)and, closely related,

How should sharpness and complexity be defined to maximize the explanatory power of the two\-factor view?

Understanding these questions has both theoretical and practical implications\. Theoretically, it clarifies whether sharpness and complexity provide a sufficient lens for explaining generalization, or whether additional factors are needed beyond sharpness and complexity\. It also helps distinguish failures caused by inadequate metric definitions from failures that reflect intrinsic limitations of the two\-factor framework\. In practice, this understanding can guide the development of more reliable methods for model comparison, hyperparameter selection, and training regularization\. When sharpness and complexity metrics have strong explanatory power, they can inform the design of penalties that encourage better generalization; when they fail, they point to directions for incorporating additional factors beyond the two factors\.

Notably, the objective of this work is not simply to propose better metrics, but to use these metrics as probes for studying*the explanatory scope*of the two\-factor view\.

### 1\.1Related Work

Previous work has largely studied generalization measures as individual scalar quantities\.

Neyshabur et al\.\[[30](https://arxiv.org/html/2606.29043#bib.bib35)\]investigated several explanations for generalization in deep learning, including norm\-based control, sharpness, and robustness\. Their work highlighted the importance of scale normalization and connected sharpness\-based reasoning with PAC\-Bayes theory, making it closely related to the two\-factor view\. However, their study mainly focuses on individual generalization measures, rather than the joint explanatory power of sharpness and complexity as a two\-factor system\.

Jiang et al\.\[[16](https://arxiv.org/html/2606.29043#bib.bib3)\]conducted a large\-scale evaluation of different generalization measures for deep networks, covering many commonly used quantities such as VC\-dimension, norm\- or margin\-based bounds, PAC\-Bayes bounds, sharpness, and path norm\. Their reported results are quite pessimistic: most evaluated measures exhibited weak correlation with the actual generalization gap\. Although some of their evaluated measures are computed from both sharpness and complexity, the two factors are typically combined through a prescribed functional form and then evaluated as a single scalar quantity\. Therefore, the poor correlation of such a scalar measure does not necessarily imply that sharpness and complexity are uninformative; it may instead indicate that the chosen functional form combines the two factors in an ineffective way\. This leaves open a more fundamental question: how much of generalization can be*jointly*explained by sharpness and complexity*without*committing in advance to a specific scalar combination?

In comparison, this work focuses on the two\-factor view rather than on individual scalar generalization measures\. We study how much of the generalization gap can be explained by*the pair*of quantities\(S,C\)\(S,C\)\. Importantly, we*do not assume a known functional form*for how these two factors combine, since this relationship may be complex and may vary across settings\. Our goal is to examine*how far the two factors are jointly informative*in predicting the true generalization gap\.

A key feature of our study is that we do not restrict the evaluation to models within a single architecture or dataset\. Instead, we explicitly examine whether sharpness and complexity can support meaningful comparisons*across*architectures and datasets\. Consistent with the observation of Jiang et al\.\[[16](https://arxiv.org/html/2606.29043#bib.bib3)\], we further find that single\-factor view has relatively limited explanatory power; see Section[5\.2\.1](https://arxiv.org/html/2606.29043#S5.SS2.SSS1)\.

### 1\.2Main Contributions

- •We empirically study the explanatory power of the two\-factor view across a range of architectures and metric realizations\. Our main findings are reported in Section[5\.2](https://arxiv.org/html/2606.29043#S5.SS2)and summarized as follows\. - –Existing parameter\-level realizations of sharpness and complexity can explain generalization reasonably well in some architectures, such as ResNet18 and WideResNet, but lose explanatory power in others, such as no\-BN ResNet18 and no\-BN WideResNet111BN denotes batch normalization\.\. This shows that the two\-factor view is informative, but its explanatory scope can be limited under parameter\-level metrics\. - –When sharpness and complexity are defined closer to function space, the explanatory boundary of the two\-factor framework can be pushed further: in several settings where parameter\-level metrics fail, function\-oriented metrics recover good explanatory power\. This suggests that some failures are due to limitations of the metric definitions rather than limitations of the two\-factor view itself\. More broadly, it indicates that with more appropriate realizations of sharpness and complexity, the explanatory scope of the two\-factor view may have considerable room for improvement\. - –At the same time, the proposed function\-oriented metrics fail in certain architectures, such as ViT\. These remaining failures are themselves informative\. They suggest that the proposed new realization of the two\-factor view is useful but not yet complete: either more adequate metric definitions are needed to further extend its explanatory scope, or additional factors beyond sharpness and complexity are necessary for a complete explanation of generalization\.
- •As an evaluation tool, we introduce Pareto analysis for directly testing the expected monotonic relationship between generalization factors\(S,C\)\(S,C\)and the generalization gap, complementing standard regression\-based evaluation criterion\.

These findings*quantitatively*characterize the extent to which generalization can be explained through the two\-factor lens and identify the regimes in which this view succeeds or potentially breaks down\. Practically, metrics with strong explanatory power can serve as diagnostic tools for comparing trained models, selecting hyperparameters, and designing regularization penalties that encourage better generalization\. Moreover, by comparing parameter\-level and function\-oriented metrics, our study sheds light on how sharpness and complexity can be measured in overparameterized networks\.

We*emphasize*that the proposed Pareto analysis and the function\-oriented metrics are only probes for investigating the explanatory power of the two\-factor view, rather than the core contributions of this paper\. Other evaluation criteria and alternative definitions of these metrics may also be suitable\.

Our investigation is organized around*three central questions*, Question \([∗\*‣1](https://arxiv.org/html/2606.29043#S1.Ex1)\), Question \([∗⁣∗\*\*‣3\.1](https://arxiv.org/html/2606.29043#S3.Ex8)\), and Question \([∗⁣∗⁣∗\*\\\!\*\\\!\*‣3\.2\.2](https://arxiv.org/html/2606.29043#S3.Ex9)\)\. The focus is an empirical and conceptual analysis of when the two\-factor view succeeds, when it fails, and what these successes and failures reveal about the scope of the sharpness\-complexity framework for explaining deep\-network generalization\.

##### Paper organization\.

The remainder of the paper is organized as follows\. Section[2](https://arxiv.org/html/2606.29043#S2)reviews existing sharpness and complexity metrics\. Section[3](https://arxiv.org/html/2606.29043#S3)analyzes why existing metrics are insufficient and introduces our methodological contribution, Pareto analysis\. Section[4](https://arxiv.org/html/2606.29043#S4)introduces the function\-oriented realizations of sharpness and complexity\. Section[5](https://arxiv.org/html/2606.29043#S5)reports the main empirical results\.

## 2Review of Existing Sharpness and Complexity Metrics

Throughout the paper, we use the following notation\. LetZ=\{zi\}i=1mZ=\\\{z\_\{i\}\\\}\_\{i=1\}^\{m\}be a training dataset of sizemm, where each examplezi=\(𝐱i,yi\)z\_\{i\}=\(\\mathbf\{x\}\_\{i\},y\_\{i\}\)are drawn i\.i\.d\. from an unknown data distribution𝒟\\mathcal\{D\}\. Here,𝐱i\\mathbf\{x\}\_\{i\}denotes the input andyiy\_\{i\}denotes the corresponding true label\. For a modelhhand a loss functionℓ​\(h;z\)\\ell\(h;z\), we define the empirical \(or training\) loss onZZand the generalization \(or population\) loss under𝒟\\mathcal\{D\}as

ℓ​\(h;Z\):=1m​∑i=1mℓ​\(h;zi\),ℓ​\(h;𝒟\):=𝔼Z∼𝒟m​ℓ​\(h;Z\)\.\\displaystyle\\ell\(h;Z\)=\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}\\ell\(h;z\_\{i\}\)\\;,\\quad\\ell\(h;\\mathcal\{D\}\)=\\mathbb\{E\}\_\{Z\\sim\\mathcal\{D\}^\{m\}\}\\,\\ell\(h;Z\)\.When the model is parameterized byθ\\theta, we writefθf\_\{\\theta\}for the represented function, andθ^\\hat\{\\theta\}for the trained parameter vector\. We denote the Hessian of the empirical loss atθ^\\hat\{\\theta\}by

Hθ^=∇θ2ℓ​\(fθ^;Z\)\.H\_\{\\hat\{\\theta\}\}=\\nabla\_\{\\theta\}^\{2\}\\ell\(f\_\{\\hat\{\\theta\}\};Z\)\.
### 2\.1Parameterization Ambiguity

Deep networks are typically represented by parameters, but this representation is often non\-unique\. For a model parameter vectorθ\\theta, letfθf\_\{\\theta\}denote the function represented by the network\. A*function\-preserving reparameterization*is a transformation of the parameters that changesθ\\thetabut leaves the represented function unchanged\. That is, two different parameter vectorsθ\\thetaandθ′\\theta^\{\\prime\}may satisfy

fθ′=fθ,f\_\{\\theta^\{\\prime\}\}=f\_\{\\theta\}\\,,even though parameter\-dependent quantities evaluated atθ\\thetaandθ′\\theta^\{\\prime\}can be very different\. We refer to this non\-uniqueness as*parameterization ambiguity*\.

There are many sources of parameterization ambiguity in overparameterized deep networks\. One small but common class is given by rescalings, specifically, the layer\-wise rescaling symmetries in positively homogeneous networks\. For example, in ReLU networks, scaling one layer by a positiveα\\alphaand compensating in a following layer by1/α1/\\alphacan leave the represented function unchanged, but substantially changes the parameter\-dependent quantities\. Such layer\-wise rescaling symmetries are*only one special case*of all types of function\-preserving reparameterizations in overparameterized networks; see Figure[1](https://arxiv.org/html/2606.29043#S2.F1)\. Many other function\-preserving transformations may arise from overparameterization, architectural structure, skip connections, normalization layers, or other redundancies\.

Ideally, a generalization factor metric should be invariant under any function\-preserving reparameterization, since such transformations do not change the predictor or its generalization behavior\. However, eliminating all sources of parameterization ambiguity is difficult\. A more attainable requirement in practice is*invariance under simple rescalings*\. Therefore, in this work we focus on scale\-invariant†\\dagger†\\dagger†\\daggerThroughout this work, scale invariance refers to invariance under the layer\-wise rescaling symmetries of positively homogeneous networks\.metrics as existing baselines\.

Nevertheless, scale invariance removes only one source of parameterization ambiguity\. Metrics that are scale\-invariant can still depend on other aspects of the parameter representation, and therefore are not guaranteed to remain comparable across general architectures\. This limitation motivates our later search for metrics that are defined closer to the function space, rather than relying only on parameter\-level invariance\.

![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/venn_figure.png)Figure 1:Illustration of parameterization ambiguity\. The layer\-wise rescaling symmetries in positively homogeneous networks form only one special case of all types of function\-preserving reparameterizations\.
### 2\.2Existing Sharpness and Complexity Metrics

Existing refinements of sharpness and complexity metrics have mainly focused around making them scale\-invariant†\. We summarize their properties and applicability across different network architectures in Table[1](https://arxiv.org/html/2606.29043#S2.T1)\.

##### Determinant of Hessian\.

A natural way to aggregate local curvature is to use determinant\-based sharpness metrics, such as

Sdet​\(θ^\)=det​\(Hθ^\),S\_\{\\mathrm\{det\}\}\(\\hat\{\\theta\}\)=\\mathrm\{det\}\\big\(H\_\{\\hat\{\\theta\}\}\\big\),\(1\)or its damped log\-determinant versionSlog⁡det​\(θ^\)=log⁡det​\(Hθ^\+ϵ​I\),S\_\{\\log\\mathrm\{det\}\}\(\\hat\{\\theta\}\)=\\log\\mathrm\{det\}\\big\(H\_\{\\hat\{\\theta\}\}\+\\epsilon I\\big\),whereϵ\>0\\epsilon\>0is a damping parameter\. These quantities are motivated by the local volume of the loss basin: a larger determinant corresponds to larger aggregate curvature, a smaller local volume around the solution, and therefore a sharper minimum\.

Determinant of Hessian can remain invariant under special volume\-preserving rescalings where Hessian trace and top eigenvalue do not\. However, it is generally not invariant under the layer\-wise rescaling symmetries in positively homogeneous networks\. It is also very expensive to compute\. Moreover, the Hessian itself may be singular or indefinite, making the determinant unstable or even ill\-defined as a sharpness metric\. For these reasons, we do not use determinant of Hessian as our main existing baseline\.

##### Adaptive SAM\.

SAM captures sharpness throughthe worst\-case loss increasein a local neighborhood of the trained solution\[[11](https://arxiv.org/html/2606.29043#bib.bib15)\]\. Given trained model weightsθ^\\hat\{\\theta\}, the SAM sharpness is

SSAM​\(θ^\):=max‖ϵ‖2≤ρ⁡\[ℓ​\(fθ^\+ϵ;Z\)−ℓ​\(fθ^;Z\)\],S\_\{\\mathrm\{SAM\}\}\(\\hat\{\\theta\}\):=\\max\_\{\\\|\\epsilon\\\|\_\{2\}\\leq\\rho\}\\left\[\\ell\(f\_\{\\hat\{\\theta\}\+\\epsilon\};Z\)\-\\ell\(f\_\{\\hat\{\\theta\}\};Z\)\\right\],whereρ\>0\\rho\>0is a hyperparameter controlling the neighborhood/perturbation radius\.

Adaptive SAM \(adapS\)\[[20](https://arxiv.org/html/2606.29043#bib.bib16)\]replaces the fixed neighborhood in SAM with a parameter\-adaptive perturbation region,

SadapS​\(θ^\):=max‖Tθ^−1​ϵ‖2≤ρ⁡\[ℓ​\(fθ^\+ϵ;Z\)−ℓ​\(fθ^;Z\)\],S\_\{\\mathrm\{adapS\}\}\(\\hat\{\\theta\}\):=\\max\_\{\\\|T\_\{\\hat\{\\theta\}\}^\{\-1\}\\epsilon\\\|\_\{2\}\\leq\\rho\}\\left\[\\ell\(f\_\{\\hat\{\\theta\}\+\\epsilon\};Z\)\-\\ell\(f\_\{\\hat\{\\theta\}\};Z\)\\right\],\(2\)whereTθ^T\_\{\\hat\{\\theta\}\}is a diagonal matrix whose entries are proportional to parameter magnitudes, for exampleTθ^=diag​\(\|θ^1\|,⋯,\|θ^d\|\)T\_\{\\hat\{\\theta\}\}\\\!=\\\!\\mathrm\{diag\}\(\|\\hat\{\\theta\}\_\{1\}\|,\\cdots,\|\\hat\{\\theta\}\_\{d\}\|\)\.

The key idea of adapS is that the perturbation scale changes with the parameter magnitude\. Equivalently, it measures relative perturbations rather than absolute Euclidean perturbations\. This adaptive geometry makes it invariant to coordinate\-wise function\-preserving rescalings, and the layer\-wise rescaling symmetries in positively homogeneous networks are one special case\. Using a local quadratic approximation, adapS can be interpreted as measuring the largest eigenvalue of an adaptive Hessian:

SadapS​\(θ^\)≈ρ22​λmax​\(Tθ^​Hθ^​Tθ^\),S\_\{\\mathrm\{adapS\}\}\(\\hat\{\\theta\}\)\\approx\\frac\{\\rho^\{2\}\}\{2\}\\;\\lambda\_\{\\max\}\\left\(T\_\{\\hat\{\\theta\}\}H\_\{\\hat\{\\theta\}\}T\_\{\\hat\{\\theta\}\}\\right\),and therefore it captures*the worst\-direction curvature*under a*parameter\-adaptive*geometry\.

##### Path norm\.

Path norm is a norm\-based complexity measure that sums the contribution of all input\-output paths through a network\[[32](https://arxiv.org/html/2606.29043#bib.bib37)\]\. For a feedforward network, let𝒱\\mathcal\{V\}denote the set of all paths from input units to output units\. The path norm is defined as

Cpath​\(θ^\):=∑v∈𝒱∏i∈vθ^i2,C\_\{\\mathrm\{path\}\}\(\\hat\{\\theta\}\):=\\sum\_\{v\\in\\mathcal\{V\}\}\\prod\_\{i\\in v\}\{\\hat\{\\theta\}\}\_\{i\}^\{\\,2\},\(3\)or equivalently as the squaredℓ2\\ell\_\{2\}\-norm of the vector of path products\. The design intuition is that a network should be considered complex if many input\-output paths have large effective strength\. Unlike the original weight norm, path norm is invariant to layer\-wise rescaling symmetries in positively homogeneous networks\.

##### Spectral norm product\.

Spectral norm product measures complexity through the product of operator norms of the weight matrices\[[4](https://arxiv.org/html/2606.29043#bib.bib38)\]\. For anLL\-layer network with weight matricesW1,…,WLW\_\{1\},\\dots,W\_\{L\}, it is defined as

Cspec​\(θ^\):=∏j=1L‖Wj‖2,C\_\{\\mathrm\{spec\}\}\(\\hat\{\\theta\}\):=\\prod\_\{j=1\}^\{L\}\\\|W\_\{j\}\\\|\_\{2\},\(4\)where‖Wj‖2\\\|W\_\{j\}\\\|\_\{2\}is the spectral norm of layerjj\. This quantity controls the Lipschitz constant of the network up to the Lipschitz constants of the activation functions\. Thus, smaller spectral norm product suggests a more controlled input\-output mapping\. In positively homogeneous networks, the product form is invariant to compensating layer\-wise rescaling symmetries\.

MetricVGGsResNetsViTScale Invariant†det​\(Hθ\)\\mathrm\{det\}\(H\_\{\\theta\}\)\([1](https://arxiv.org/html/2606.29043#S2.E1)\)defined, expensivedefined, expensivedefined, very expensivenoadapS \([2](https://arxiv.org/html/2606.29043#S2.E2)\)well\-definedextendableextendableyespath norm \([3](https://arxiv.org/html/2606.29043#S2.E3)\)well\-definedextendableno standard definitionyesspec norm product \([4](https://arxiv.org/html/2606.29043#S2.E4)\)well\-definedextendableno standard definitionyes

Table 1:Properties and applicability of existing sharpness and complexity metrics in different network architectures\.†Scale invariance refers to invariance under the layer\-wise rescaling symmetries of positively homogeneous networks\.

## 3Are the Existing Metrics Sufficient?

### 3\.1A Linear Regression View

DatasetArchitecture\(𝐒,𝐂\)\\mathbf\{\(S,C\)\}Metric PairScaleInvariant†Regression PerformanceR2↑R^\{2\}\\uparrow, coefficientfailed/poor/moderate/goodCIFAR\-10ResNet18\(tr\(H\) ,ℓ2\\,\\ell\_\{2\}norm\)no0\.83,\(\+,\+\)\(\+,\+\)\(adapS, spec norm\)yes0\.56,\(\+,−\)\(\+,\-\)\(adapS, path norm\)yes0\.93,\(\+,\+\)\(\+,\+\)VGG13\(tr\(H\) ,ℓ2\\,\\ell\_\{2\}norm\)no0\.76,\(\+,\+\)\(\+,\+\)\(adapS, spec norm\)yes0\.38,\(\+,\+\)\(\+,\+\)\(adapS, path norm\)yes0\.85,\(\+,\+\)\(\+,\+\)CIFAR\-100ResNet34\(tr\(H\) ,ℓ2\\,\\ell\_\{2\}norm\)no0\.84,\(\+,\+\)\(\+,\+\)\(adapS, spec norm\)yes0\.57,\(\+,\+\)\(\+,\+\)\(adapS, path norm\)yes0\.87,\(\+,\+\)\(\+,\+\)VGG19\(tr\(H\) ,ℓ2\\,\\ell\_\{2\}norm\)no0\.76,\(\+,\+\)\(\+,\+\)\(adapS, spec norm\)yes0\.34,\(\+,−\)\(\+,\-\)\(adapS, path norm\)yes0\.35,\(\+,\+\)\(\+,\+\)WideResNet\(tr\(H\) ,ℓ2\\,\\ell\_\{2\}norm\)no0\.93,\(\+,\+\)\(\+,\+\)\(adapS, spec norm\)yes0\.56,\(\+,−\)\(\+,\-\)\(adapS, path norm\)yes0\.89,\(\+,\+\)\(\+,\+\)CIFAR\-10no\-BN ResNet18\(tr\(H\) ,ℓ2\\,\\ell\_\{2\}norm\)no0\.04,\(−,−\)\(\-,\-\)\(adapS, spec norm\)yes0\.19,\(−,−\)\(\-,\-\)\(adapS, path norm\)yes0\.07,\(−,−\)\(\-,\-\)no\-BN VGG13\(tr\(H\) ,ℓ2\\,\\ell\_\{2\}norm\)no0\.69,\(\+,−\)\(\+,\-\)\(adapS, spec norm\)yes0\.60,\(\+,\+\)\(\+,\+\)\(adapS, path norm\)yes0\.60,\(\+,\+\)\(\+,\+\)CIFAR\-100no\-BN ResNet34–––no\-BN VGG19–––CIFAR\-100no\-BN WideResNet\(tr\(H\) ,ℓ2\\,\\ell\_\{2\}norm\)no0\.17,\(−,\+\)\(\-,\+\)\(adapS, spec norm\)yes0\.42,\(−,\+\)\(\-,\+\)\(adapS, path norm\)yes0\.25,\(−,\+\)\(\-,\+\)

Table 2:Linear regressionresults of existing\(S,C\)\(S,C\)metric pairs\. The two blocks correspond to normalization\-equipped and normalization\-disabled settings, respectively\.ℓ2\\ell\_\{2\}norm is the norm of weights\. Since CIFAR\-100, no\-BN ResNet34 and no\-BN VGG19 are barely trainable, we are unable to obtain enough valid trained runs and the corresponding entries are left blank\. Strong result is suggested by*highR2R^\{2\}\(↑\)\(\\uparrow\)together with positive coefficients*\. The last column summarizes the regression performance of each evaluated metric pair:*failed*indicates the regression returns negative coefficients, and*poor/moderate/good*is assigned according toR2R^\{2\}cutoffs at 0\.50 and 0\.80\. Based on these results, we choose \(adapS, path norm\) to serve as thebaseline metric pair, as it achieves the best results among the existing scale\-invariant†metrics\.For a fixed dataset and network, we obtain*a collection of trained networks*by sweeping the training hyperparameters \(learning rate, batch size, weight decay, and momentum\), and we treat each trained model as one regression point\. For theii\-th trained model, letSiS\_\{i\}andCiC\_\{i\}be its sharpness and complexity values under the evaluated sharpness and complexity metrics, and letGapi\\mathrm\{Gap\}\_\{i\}be the generalization gap calculated as its test loss minus its training loss\. We fit

Gapi=β0\+βS​𝑺𝒊\+βC​𝑪𝒊\+εi,\\mathrm\{Gap\}\_\{i\}=\\beta\_\{0\}\+\\beta\_\{S\}\\boldsymbol\{S\_\{i\}\}\+\\beta\_\{C\}\\boldsymbol\{C\_\{i\}\}\+\\varepsilon\_\{i\},and report theR2R^\{2\}and the sign of regression coefficientsβS,βC\\beta\_\{S\},\\beta\_\{C\}\. A highR2R^\{2\}indicates that the evaluated\(S,C\)\(S,C\)metrics have strong explanatory power for generalization under the linear modeling assumption\.

Table[2](https://arxiv.org/html/2606.29043#S3.T2)presents the regression results of the existing sharpness–complexity metric pairs\. Several observations can be made\.

First, in normalization\-equipped architectures, the scale\-*variant*metric pair \(tr\(H\),ℓ2\\ell\_\{2\}norm\) achieves predictability comparable to that of the scale\-*invariant*pair \(adapS, path norm\)\. This suggests that enforcing scale\-invariance removes one source of parameterization ambiguity, but does not by itself substantially improve the explanatory power of the two\-factor view\. This is consistent with previous discussion in Section[2\.1](https://arxiv.org/html/2606.29043#S2.SS1): simple rescaling symmetries form only one small class of parameterization ambiguity, while many other function\-preserving reparameterizations remain unresolved\.

Second, among the existing scale\-invariant metrics, the pair \(adapS, path norm\) performs reasonably well on normalization\-equipped ResNet18, VGG13, ResNet34, and WideResNet\. This indicates that these metrics can capture meaningful variations in generalization within certain architecture families\. By contrast, replacing path norm with the spectral norm product often leads to much weaker, and sometimes failed, predictability\. This observation is consistent with the large\-scale study of Jiang et al\.\[[16](https://arxiv.org/html/2606.29043#bib.bib3)\], which found that spectral\-norm\-based complexity measures can correlate poorly, and sometimes negatively, with generalization, while path norm exhibits more favorable correlation across hyperparameter variations\. Therefore, we use \(adapS, path norm\) as the existing scale\-invariantbaseline metric pairin our later experiments\.

However, the explanatory power of existing metrics is far from consistent\. Even the best existing baseline \(adapS, path norm\) can lose predictability when normalization is removed\. For example, for no\-BN ResNet18 and no\-BN WideResNet, the regressionR2R^\{2\}drops below0\.300\.30, and the fitted coefficients become even negative\. These results suggest that scale invariance alone is insufficient to guarantee consistent explanatory power across architectures and training settings\.

At first glance, these failures may appear to provide pessimistic evidence against the two\-factor view\. However, an important question remains:

Does poor regression performance imply that the two\-factor view fail to explain generalization, or\(∗⁣∗\*\*\)does it indicate that their relationship to generalization is not well captured by a linear model?In other words, are the observed failures caused by the two\-factor framework itself, or by the linearity assumption imposed by the regression evaluation?

This question motivates ourmethodological contribution: an order\-based evaluation tool that makes no linearity assumption\.

### 3\.2A Pareto Analysis View

#### 3\.2\.1Pareto Contradiction Rate \(PCR\)

To avoid making any linear or other particular functional assumption, we introducePareto analysis, which does not assume fitting a parametric model\. Its main idea is to check*whether the ordering induced by the\(S,C\)\(S,C\)metrics agrees with the ordering of generalization performances*\. Pareto analysis only relies on themonotonic relationshipsuggested by the two\-factor view: if modeliihas both lower sharpness and lower complexity than modeljj, then modeliishould generalize better\.

Pareto analysis starts by representing a collection of trained models as points in the two\-dimensional sharpness–complexity plane\. Each trained model is represented as a point\(Si,Ci\)\(S\_\{i\},C\_\{i\}\), whereSiS\_\{i\}andCiC\_\{i\}denote its sharpness and complexity values under the evaluated metrics\. The point is annotated by its generalization gap\.

Under the two\-factor view, models with*smaller generalization gaps*are expected to lie toward*the lower\-left region*of the plane\. The lower\-left envelope of the point cloud forms an*empirical Pareto frontier*\. A point is dominated if there exists another point with both lower sharpness and lower complexity\. The empirical Pareto frontier consists of all non\-dominated points and represents the best attainable sharpness–complexity trade\-offs among the collection of trained models; see Figure[2](https://arxiv.org/html/2606.29043#S3.F2)and Section[4\.1\.2](https://arxiv.org/html/2606.29043#S4.SS1.SSS2)\. In principle, models on the Pareto frontier should have relatively small generalization gaps, because \(horizontally\) for a given level of complexity they achieve smaller sharpness, or \(vertically\) for a given level of sharpness they achieve smaller complexity\. The Pareto plot provides a*visualization*of the expected explanatory power of the evaluated\(S,C\)\(S,C\)metric pair:*metrics with good explanatory power*should place well\-generalizing models closer to the lower\-left Pareto frontier, while poorly\-generalizing models should lie farther away from it\.

SSCCnon\-dominated pointsempirical Pareto frontierdominated pointsFigure 2:Each point denotes a trained model in the\(S,C\)\(S,C\)\-plane\. A point is non\-dominated if no other point dominates it\.*Empirical Pareto frontier*is the set of non\-dominated points, forming a lower\-left envelope\.##### Pairwise Comparison\.

Given two trained modelsiiandjj, we say\(i,j\)\(i,j\)is acandidate pairif one model has both lower sharpness and lower complexity than the other\. For example, modeliidominates modeljjin the plane ifSi<Sj​and​Ci<Cj\.S\_\{i\}\\\!<\\\!S\_\{j\}\\text\{ and \}C\_\{i\}\\\!<\\\!C\_\{j\}\.Geometrically, modeliilies toward*the lower\-left*of modeljj\. Under the expected*monotonic*relationship, modeliishould have a*better*generalization performanceGapi<Gapj\\mathrm\{Gap\}\_\{i\}\\\!<\\\!\\mathrm\{Gap\}\_\{j\}\. We restrict attention to such candidate pairs because only these pairs induce a clear expected ordering from the monotonic two\-factor view\. For pairs where one model has lower sharpness but higher complexity, or vice versa, the ordering is unclear without assuming a specific functional form for howSSandCCcombine; as emphasized in Section[1\.1](https://arxiv.org/html/2606.29043#S1.SS1), our analysis do not assume such a functional form\.

A candidate pair is called acontradictory pairif this expected ordering is violated\. That is, if modeliihas both lower sharpness and lower complexity than modeljj, but has a worse generalization performance,Si<Sj,Ci<Cj,and​Gapi\>Gapj,S\_\{i\}\\\!<\\\!S\_\{j\},\\,C\_\{i\}\\\!<\\\!C\_\{j\},\\text\{ and \}\\mathrm\{Gap\}\_\{i\}\\\!\>\\\!\\mathrm\{Gap\}\_\{j\},then the pair\(i,j\)\(i,j\)is counted as a Pareto contradiction\. Such a contradictory pair indicates that the\(S,C\)\(S,C\)\-ordering under the evaluated metrics disagrees with the expected generalization ordering\.

To quantify the degree of disagreement, we define the*Pareto contradiction rate*as

PCR:=\#​\{contradictory pairs\}\#​\{candidate pairs\}\.\\mathrm\{PCR\}:=\\frac\{\\\#\\\{\\text\{contradictory pairs\}\\\}\}\{\\\#\\\{\\text\{candidate pairs\}\\\}\}\.\(5\)
A lower PCR indicates better agreement between the\(S,C\)\(S,C\)\-ordering and generalization ordering, and hence stronger explanatory power of the evaluated metric pair\. Conversely, a high PCR suggests frequent violations of the monotonic relationship expected from the two\-factor view\. Figure[3](https://arxiv.org/html/2606.29043#S3.F3)provides an illustration\.

![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/PCR001_cricled.png)\(a\)*candidate pairs*:\{AC,BC,BD\}, count 3\.
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/PCR_demonstration2.png)\(b\)*contradictory pairs*: None, count 0\.PCR=03=0%\\text\{PCR\}=\\frac\{0\}\{3\}=0\\%\.
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/pcr_demonstration3.png)\(c\)*contradictory pairs*:\{BC,BD\}, count 2\.PCR=23=66\.7%\\text\{PCR\}=\\frac\{2\}\{3\}=66\.7\\%\.

Figure 3:Illustration of computingPCRon an artificial collection of trained models\{A,B,C,D\}\\\{A,B,C,D\\\}\. Each point represents a trained model in the sharpness–complexity plane and is annotated by its generalization gap\. Pareto analysis checks whether the\(S,C\)\(S,C\)\-ordering indicated by*point locations*agrees with the generalization ordering annotated by*generalization gaps*; the degree of disagreement is quantified by PCR\.![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c10_resnet18_adapS_pathnorm.png)\(a\)CIFAR\-10, ResNet18\.PCR=9\.4%\\text\{PCR\}\\\!=\\\!9\.4\\%\.
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c10_vgg13_adapS_pathnorm.png)\(b\)CIFAR\-10, VGG13\.PCR=22\.2%\\text\{PCR\}\\\!=\\\!22\.2\\%\.
Normalization\-equipped networks\.

![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c100_resnet34_adapS_pathnorm.png)\(c\)CIFAR\-100,ResNet34\.PCR=18\.5%\\text\{PCR\}\\\!=\\\!18\.5\\%\.
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c100_vgg19_adapS_pathnorm.png)\(d\)CIFAR\-100,VGG19\.PCR=37\.0%\\text\{PCR\}\\\!=\\\!37\.0\\%\.
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c100_wideresnet_adapS_pathnorm.png)\(e\)CIFAR\-100,WideResNet\.PCR=8\.2%\\text\{PCR\}\\\!=\\\!8\.2\\%\.
Normalization\-equipped networks\.

![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c10_nobnresnet18_adapS_pathnorm_dark.png)\(f\)CIFAR\-10, no\-BN ResNet18\.
PCR=77\.8%\\text\{PCR\}\\\!=\\\!77\.8\\%\.
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c10_nobnvgg13_adapS_pathnorm_dark.png)\(g\)CIFAR\-10, no\-BN VGG13\.
PCR=63\.4%\\text\{PCR\}\\\!=\\\!63\.4\\%\.
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c100_nobn_wideresnet_adapS_pathnorm_dark.png)\(h\)CIFAR\-100, no\-BN WideResNet\.
PCR=47\.3%\\text\{PCR\}\\\!=\\\!47\.3\\%\.
Normalization\-disabled networks\.

Figure 4:Pareto analysisof existing baseline metric pair\(adapS, path norm\),*complementing*the regression results shown in Table[2](https://arxiv.org/html/2606.29043#S3.T2)\. Each point represents a trained model, annotated by its generalization gap\. The first two rows show normalization\-equipped networks, and the last row shows normalization\-disabled networks\. A*lower PCR*\(↓\)\(\\downarrow\)indicates*better*agreement with the expected monotonic relationship\.

#### 3\.2\.2Evaluating Existing Baseline Metric Pair via Pareto Analysis

Question \([∗⁣∗\*\*‣3\.1](https://arxiv.org/html/2606.29043#S3.Ex8)\) highlights one possible explanation for those poor regression performance of the existing baseline metric pair: the relationship between\(S,C\)\(S,C\)and generalization may be substantially nonlinear\. If this were the main issue, then lowR2R^\{2\}would not necessarily invalidate the two\-factor view; it would only indicate that linear regression is an inadequate evaluation tool\.

To investigate this possibility, Figure[4](https://arxiv.org/html/2606.29043#S3.F4)reportsPareto analysis resultsof the existing baseline metric pair\(adapS,path​norm\)\(\\mathrm\{adapS\},\\mathrm\{path\\ norm\}\)\. Unlike linear regression, Pareto analysis does not assume any linearity\. Instead, PCR directly checks the expected*monotonic*ordering: models with lowerSSand lowerCCshould have smaller generalization gaps\. For better readability, Table[3](https://arxiv.org/html/2606.29043#S3.T3)brings the regression and Pareto analysis results together\.

Nevertheless, the PCR results remain largely pessimistic\. The baseline metric pair \(adap, path norm\) achieves only moderate PCR on some normalization\-equipped networks, and the PCR becomes even more problematic once normalization is disabled\. For instance, PCR is37\.0%37\.0\\%on BN\-equipped VGG19 , and rises to77\.8%77\.8\\%on no\-BN ResNet18,63\.4%63\.4\\%on no\-BN VGG13, and47\.3%47\.3\\%on no\-BN WideResNet\. Since a high PCR means that a large fraction of pairwise comparisons violate the expected monotonic ordering, these results indicate that the*failures of existing baseline metric pair persists*even under an order\-based evaluation that makes no linearity assumption\.

The Pareto analysis results provide a partial answer to Question \([∗⁣∗\*\*‣3\.1](https://arxiv.org/html/2606.29043#S3.Ex8)\): the failures shown in Table[2](https://arxiv.org/html/2606.29043#S3.T2)cannot be attributed solely to the linearity assumption of regression\. Even under the order\-based evaluation tool, which does not impose linearity, the existing baseline metric pair still fails to consistently predict generalization behavior\. Therefore, the issue is not simply that the relationship may be nonlinear; rather, the existing metric definitions may fail to predict generalization\.

This naturally leads to the next question:

Are these failures caused by limitations of the two\-factor view itself, or\(∗⁣∗⁣∗\*\\\!\*\\\!\*\)by limitations of the current definitions of sharpness and complexity?
DatasetArchitectureExisting Baseline\(𝐒,𝐂\)\\mathbf\{\(S,C\)\}Metric PairRegressionR2↑R^\{2\}\\uparrow, coefficientPareto AnalysisPCR↓\\downarrowCIFAR\-10ResNet18\(adapS, path norm\)0\.93,\(\+,\+\)\(\+,\+\)9\.4%VGG13\(adapS, path norm\)0\.85,\(\+,\+\)\(\+,\+\)22\.2%CIFAR\-100ResNet34\(adapS, path norm\)0\.87,\(\+,\+\)\(\+,\+\)18\.5%VGG19\(adapS, path norm\)0\.35,\(\+,\+\)\(\+,\+\)37\.0%WideResNet\(adapS, path norm\)0\.89,\(\+,\+\)\(\+,\+\)8\.2%CIFAR\-10no\-BN ResNet18\(adapS, path norm\)0\.07,\(−,−\)\(\-,\-\)77\.8%no\-BN VGG13\(adapS, path norm\)0\.60,\(\+,\+\)\(\+,\+\)63\.4%CIFAR\-100no\-BN WideResNet\(adapS, path norm\)0\.25,\(−,\+\)\(\-,\+\)47\.3%

Table 3:Two\-factor explanatory power under the existing baseline metric pair \(adapS, path norm\), evaluated using two complementary tools: linear regression, where*higher*R2​\(↑\)R^\{2\}\(\\uparrow\)is better, and Pareto analysis, where*lower*PCR\(↓\)\(\\downarrow\)is better\. This table brings together the regression results from Table[2](https://arxiv.org/html/2606.29043#S3.T2)and PCR results from Figure[4](https://arxiv.org/html/2606.29043#S3.F4), showing that the observed failures cannot be attributed solely to the linear modeling assumption\.

## 4Moving Beyond Purely Parameter\-Level Metrics

As discussed in Section[2\.1](https://arxiv.org/html/2606.29043#S2.SS1), requiring scale invariance can remove one source of ambiguity, but it does not eliminate all possible parameterization ambiguities\. This issue becomes especially relevant for architectures with skip\-connection designs, such as ResNets, where functional equivalence can arise from interactions that are*not*reducible to a simple scaling symmetry\. We provide an example to illustrate this\.

Consider a scalar two\-layer residual block

fθ​\(x\)=x\+a​σ​\(b​x\),f\_\{\\theta\}\(x\)=x\+a\\,\\sigma\(bx\),whereσ\\sigmais the ReLU activation\. The two parameterizations

θ1=\(a,b\)=\(1,0\),θ2=\(a,b\)=\(0,1\)\\theta\_\{1\}=\(a,b\)=\(1,0\),\\qquad\\theta\_\{2\}=\(a,b\)=\(0,1\)both implement the identity function, and thusfθ1​\(x\)=fθ2​\(x\)=xf\_\{\\theta\_\{1\}\}\(x\)=f\_\{\\theta\_\{2\}\}\(x\)=xfor allxx\. However, these two parameterizations are not connected by the simple rescaling symmetry

\(a,b\)↦\(aα,α​b\),α\>0\.\(a,b\)\\mapsto\\left\(\\frac\{a\}\{\\alpha\},\\alpha b\\right\),\\qquad\\alpha\>0\.No choice ofα\>0\\alpha\>0can transform\(1,0\)\(1,0\)into\(0,1\)\(0,1\)\. This toy example shows that even in a minimal residual block, functional equivalence can arise from parameterization ambiguities beyond simple rescalings\.

This motivates the use of function\-level metrics, which compare models directly through*the induced predictive behavior*rather than through the raw parameter representation\. To address Question \([∗⁣∗⁣∗\*\\\!\*\\\!\*‣3\.2\.2](https://arxiv.org/html/2606.29043#S3.Ex9)\), we introduce function\-oriented realizations of sharpness and complexity:*Bayes sharpness*and*functional KL divergence*\.

### 4\.1Inspiration from PAC\-Bayes Bounds

The idea of the two\-factor view and the proposed function\-oriented metric definitions are best explained through PAC\-Bayes bounds\. Readers familiar with PAC\-Bayes theory may skip this section and proceed directly to Section[4\.2](https://arxiv.org/html/2606.29043#S4.SS2)\.

#### 4\.1\.1PAC\-Bayes Theory

PAC\-Bayes theory studies*probabilistic predictors*\. Letℋ\\mathcal\{H\}be a hypothesis space, where eachh∈ℋh\\in\\mathcal\{H\}is a predictor mapping inputs to outputs, such as a neural network with fixed weights\. A*prior*distribution𝒫\\mathcal\{P\}overℋ\\mathcal\{H\}encodes a data\-*independent*belief about which predictors are likely before observing the training data\. After observing the training data, one chooses a*posterior*distribution𝒬\\mathcal\{Q\}, which represents a data\-*dependent*distribution over predictors\. In practice,𝒬\\mathcal\{Q\}is often chosen to concentrate around the trained model\.

The generalization performance of the probabilistic predictor,h∼𝒬h\\sim\\mathcal\{Q\}, is measured by its*expected*generalization \(or population\) loss:

L​\(𝒬\):=𝔼h∼𝒬​\[ℓ​\(h;𝒟\)\]=𝔼h∼𝒬​𝔼Z∼𝒟m​\[ℓ​\(h;Z\)\]\.L\(\\mathcal\{Q\}\)\\\!:=\\\!\\mathbb\{E\}\_\{h\\sim\{\\mathcal\{Q\}\}\}\\\!\\big\[\\ell\(h;\\mathcal\{D\}\)\\big\]\\\!=\\\!\\mathbb\{E\}\_\{h\\sim\{\\mathcal\{Q\}\}\}\\mathbb\{E\}\_\{Z\\sim\\mathcal\{D\}^\{m\}\}\\big\[\\,\\ell\(h;Z\)\\,\\big\]\.A PAC\-Bayes bound upper bounds this quantity in terms of the*expected*empirical \(or training\) loss

L^​\(𝒬\):=𝔼h∼𝒬​\[ℓ​\(h;Z\)\],\\hat\{L\}\(\\mathcal\{Q\}\):=\\mathbb\{E\}\_\{h\\sim\\mathcal\{Q\}\}\\big\[\\ell\(h;Z\)\\big\],\(6\)and the posterior–prior KL divergenceKL\(𝒬\|\|𝒫\)\\mathrm\{KL\}\(\\mathcal\{Q\}\|\|\\mathcal\{P\}\)\.

For example, a classical PAC\-Bayes bound \(section 3\.2\.2 of\[[2](https://arxiv.org/html/2606.29043#bib.bib23)\]\) states that, with probability at least1−δ1\-\\delta, the following inequality holds for all posteriors𝒬\\mathcal\{Q\}:

L​\(𝒬\)≤L^​\(𝒬\)\+KL\(𝒬\|\|𝒫\)\+ln2​mδ2​m\.L\(\\mathcal\{Q\}\)\\leq\\hat\{L\}\(\\mathcal\{Q\}\)\+\\sqrt\{\\frac\{\\mathrm\{KL\}\(\\mathcal\{Q\}\|\|\\mathcal\{P\}\)\+\\ln\\frac\{2\\sqrt\{m\}\}\{\\delta\}\}\{2m\}\}\.\(7\)Here,mmis the size of training data, andδ\\deltais the failure probability\.

Beyond the illustrative form in \([7](https://arxiv.org/html/2606.29043#S4.E7)\), PAC\-Bayes bounds have many variations\[[24](https://arxiv.org/html/2606.29043#bib.bib24),[2](https://arxiv.org/html/2606.29043#bib.bib23)\]\. For bounded losses, these include classical McAllester\-type\[[25](https://arxiv.org/html/2606.29043#bib.bib17),[26](https://arxiv.org/html/2606.29043#bib.bib18)\], Seeger\-type\[[35](https://arxiv.org/html/2606.29043#bib.bib21)\], and Catoni\-type bounds\[[7](https://arxiv.org/html/2606.29043#bib.bib27),[8](https://arxiv.org/html/2606.29043#bib.bib29)\]\. Extensions to unbounded losses require exponential\-moment or tail\-control conditions\[[1](https://arxiv.org/html/2606.29043#bib.bib30),[14](https://arxiv.org/html/2606.29043#bib.bib31),[6](https://arxiv.org/html/2606.29043#bib.bib32),[34](https://arxiv.org/html/2606.29043#bib.bib33),[40](https://arxiv.org/html/2606.29043#bib.bib34)\]\. Although these bounds differ in their specific functional forms, they*share the same basic structure*: the expected generalization loss is controlled by an expected empirical loss term and a posterior–prior complexity term\.

For deep networks, the modelhhis usually represented through parameters,h=fθh=f\_\{\\theta\}\. This leads to two related but distinct levels of description\. Inparameter space, we denote the prior and posterior distributions by𝒫θ\\mathcal\{P\}\_\{\\theta\}and𝒬θ\\mathcal\{Q\}\_\{\\theta\}\. Through the mapθ↦fθ\\theta\\mapsto f\_\{\\theta\}, they induce correspondingfunction spacedistributions, denoted by𝒫f\\mathcal\{P\}\_\{f\}and𝒬f\\mathcal\{Q\}\_\{f\}\. This distinction is important to our work: parameter\-level quantities may depend on the chosen representation of the same predictor, whereas function\-level quantities aim to compare models through their induced predictive behavior\.

#### 4\.1\.2Origin of Two Factors\.

As discussed above, PAC\-Bayes bounds control the expected generalization lossL​\(𝒬\)L\(\\mathcal\{Q\}\)mainly through two terms:

*the expected empirical loss*L^​\(𝒬\)\\hat\{L\}\(\\mathcal\{Q\}\)and*the posterior\-prior deviation*KL​\(𝒬∥𝒫\)\\mathrm\{KL\}\(\\mathcal\{Q\}\\\|\\mathcal\{P\}\)\.

The first term is closely related tosharpness\. Applying Taylor expansion in the parameter space toL^​\(𝒬\)\\hat\{L\}\(\\mathcal\{Q\}\), we obtain

L^​\(𝒬θ\)=𝔼θ∼𝒬θ​\[ℓ​\(fθ;Z\)\]≈ℓ​\(fθ^;Z\)⏟empirical loss\+12​Tr​\(∇θ2ℓ​\(fθ^;Z\)​Σθ\),\\hat\{L\}\(\\mathcal\{Q\}\_\{\\theta\}\)\\,=\\,\\mathbb\{E\}\_\{\\theta\\sim\\mathcal\{Q\}\_\{\\theta\}\}\\big\[\\ell\(f\_\{\\theta\};Z\)\\big\]\\;\\approx\\underbrace\{\\ell\(f\_\{\\hat\{\\theta\}\};Z\)\}\_\{\\text\{empirical loss\}\}\+\\;\\,\\tfrac\{1\}\{2\}\\mathrm\{Tr\}\\big\(\\nabla\_\{\\theta\}^\{2\}\\ell\(f\_\{\\hat\{\\theta\}\};Z\)\\,\\Sigma\_\{\\theta\}\\big\),\(8\)where the posterior𝒬θ\\mathcal\{Q\}\_\{\\theta\}is centered at the trained model weightsθ^\\hat\{\\theta\}with covarianceΣθ\\Sigma\_\{\\theta\}; see derivation of this local approximation in Appendix[B](https://arxiv.org/html/2606.29043#A2)\. This shows that, when the empirical loss reaches nearly zero, the dominant term depends on the*local curvature*around the learned solutionθ^\\hat\{\\theta\}, which gives intuition of the factor sharpness, and motivates Hessian\-based sharpness metrics\.

The second term is the origin of factorcomplexity:KL​\(𝒬∥𝒫\)\\mathrm\{KL\}\(\\mathcal\{Q\}\\\|\\mathcal\{P\}\)measures how much deviation is needed to move from the prior to the learned posterior\.

In particular, under a Gaussian posterior and prior, the KL divergence admits a closed form\. Suppose𝒬θ=𝒩​\(θ^,Σθ\)\\mathcal\{Q\}\_\{\\theta\}=\\mathcal\{N\}\(\\hat\{\\theta\},\\Sigma\_\{\\theta\}\)and𝒫θ=𝒩​\(ξθ,𝒯θ\)\\mathcal\{P\}\_\{\\theta\}=\\mathcal\{N\}\(\\xi\_\{\\theta\},\\mathcal\{T\}\_\{\\theta\}\),

KL​\(𝒬θ∥𝒫θ\)=12​\(−d\+Tr​\(Σθ​𝒯θ−1\)\+ln⁡det\(𝒯θ\)det\(Σθ\)\+\(θ^−ξθ\)⊤​𝒯θ−1​\(θ^−ξθ\)\),\\mathrm\{KL\}\(\\mathcal\{Q\}\_\{\\theta\}\\\|\\mathcal\{P\}\_\{\\theta\}\)=\\frac\{1\}\{2\}\\Big\(\-\\\!d\+\\mathrm\{Tr\}\\big\(\\Sigma\_\{\\theta\}\\;\\mathcal\{T\_\{\\theta\}\}^\{\-1\}\\big\)\+\\ln\\frac\{\\det\(\\mathcal\{T\_\{\\theta\}\}\)\}\{\\det\(\\Sigma\_\{\\theta\}\)\}\+\(\\hat\{\\theta\}\-\\xi\_\{\\theta\}\)^\{\\top\}\{\\mathcal\{T\_\{\\theta\}\}^\{\-1\}\}\(\\hat\{\\theta\}\-\\xi\_\{\\theta\}\)\\Big\)\\;,\(9\)whereddis the dimension of the parameter vector\. If we further assume that the posterior and prior share the same isotropic covarianceΣθ=𝒯θ=σ2​I\\Sigma\_\{\\theta\}=\\mathcal\{T\}\_\{\\theta\}=\\sigma^\{2\}I, and prior is zero\-centeredξθ=0\\xi\_\{\\theta\}\\\!=\\\!0, then the covariance\-mismatch terms vanish, and the KL divergence reduces to a scaled squared weight norm

KL​\(𝒬θ∥𝒫θ\)=12​σ2​‖θ^‖2\.\\mathrm\{KL\}\(\\mathcal\{Q\}\_\{\\theta\}\\\|\\mathcal\{P\}\_\{\\theta\}\)=\\frac\{1\}\{2\\sigma^\{2\}\}\\\|\\hat\{\\theta\}\\\|^\{2\}\.\(10\)This explains why*weight decay*in SGD\[[19](https://arxiv.org/html/2606.29043#bib.bib39)\]can be viewed as a simple parameter\-space complexity measure\.

Recall the generalization gap is defined as

Gap⁡\(h\):=L​\(𝒬\)⏟expected generalization loss−ℓ​\(h;Z\)⏟empirical loss\.\\operatorname\{Gap\}\(h\):=\\underbrace\{L\(\\mathcal\{Q\}\)\}\_\{\\text\{expected generalization loss\}\}\-\\qquad\\underbrace\{\\ell\(h;Z\)\}\_\{\\text\{empirical loss\}\}\.\(11\)Plugging the Taylor expansion \([8](https://arxiv.org/html/2606.29043#S4.E8)\) into the PAC\-Bayes bound \([7](https://arxiv.org/html/2606.29043#S4.E7)\), we have

Gap​\(fθ\)\\displaystyle\\text\{Gap\}\(f\_\{\\theta\}\)≲12​Tr​\(∇θ2ℓ​\(fθ^;Z\)​Σθ\)\+KL​\(𝒬∥𝒫\)\+log⁡2​mδm\.\\displaystyle\\lesssim\\;\\;\\tfrac\{1\}\{2\}\\mathrm\{Tr\}\\big\(\\nabla\_\{\\theta\}^\{2\}\\ell\(f\_\{\\hat\{\\theta\}\};Z\)\\,\\Sigma\_\{\\theta\}\\big\)\+\\sqrt\{\\tfrac\{\\mathrm\{KL\}\(\\mathcal\{Q\}\\\|\\mathcal\{P\}\)\+\\log\\frac\{2\\sqrt\{m\}\}\{\\delta\}\}\{m\}\}\.\(12\)
For other types of PAC\-Bayes bounds, the relationship may be nonlinear, but should still bemonotonic: models with both lower sharpness\(S\)\(S\)and lower complexity\(C\)\(C\)are expected to have smaller generalization gaps, and therefore generalize better\.

The PAC\-Bayes bound \([7](https://arxiv.org/html/2606.29043#S4.E7)\) also reveals a natural tension betweenSSandCC\. For a fixed trained model, increasing the posterior covarianceΣθ\\Sigma\_\{\\theta\}generally increases the expected empirical loss termL^​\(𝒬\)\\hat\{L\}\(\\mathcal\{Q\}\)through the curvature\-weighted quantityTr​\(∇θ2ℓ​\(fθ^;Z\)​Σθ\)\\mathrm\{Tr\}\\big\(\\nabla\_\{\\theta\}^\{2\}\\ell\(f\_\{\\hat\{\\theta\}\};Z\)\\Sigma\_\{\\theta\}\\big\), but it typically lowers the KL complexity; for example, in the case of Gaussian prior–posterior with shared isotropic covariance, the KL divergence‖θ^‖2/\(2​σ2\)\\\|\\hat\{\\theta\}\\\|^\{2\}/\(2\\sigma^\{2\}\)decreases asΣθ\\Sigma\_\{\\theta\}increases\. This illustrates an intrinsic*sharpness–complexity trade\-off*: reducing one term may come at the cost of increasing the other\. This trade\-off is consistent with the Pareto analysis, where the empirical Pareto frontier \(Figure[2](https://arxiv.org/html/2606.29043#S3.F2)\) captures the best attainable compromises, and appears as a lower\-left envelope in the\(S,C\)\(S,C\)\-plane\.

### 4\.2Bayes Sharpness

Classical sharpness metrics are often defined through the Hessian of the empirical loss\. From the perspective of Taylor expansion, this means approximating the local loss landscape by its second\-order term and discarding higher\-order information\. Instead, one can measure sharpness directly through the posterior\-averaged empirical loss increase:

𝔼θ∼𝒬θ^​\[ℓ​\(fθ;Z\)\]⏟expected empirical loss−ℓ​\(fθ^;Z\)⏟empirical loss\.\\underbrace\{\\mathbb\{E\}\_\{\\theta\\sim\\mathcal\{Q\}\_\{\\hat\{\\theta\}\}\}\\big\[\\ell\(f\_\{\\theta\};Z\)\\big\]\}\_\{\\text\{expected empirical loss\}\}\\quad\-\\quad\\underbrace\{\\ell\(f\_\{\\hat\{\\theta\}\};Z\)\}\_\{\\text\{empirical loss\}\}\.\(13\)This quantity*captures*the actual change in empirical loss induced by sampled predictors and therefore*retains*higher\-order information about the local loss landscape\. In this sense, it is function\-oriented: although the perturbations are sampled in parameter space, the measured quantity is the resulting change in empirical loss, which reflects the behavior of the induced predictors\.

However, since the posterior covariance is still specified in parameter space, we use an*adaptive posterior*to ensure scale invariance under layer\-wise rescaling symmetries\. Specifically, the standard deviation of each parameter is chosen to be proportional to the magnitude of the corresponding parameter:

𝒬θ^,σadap=𝒩​\(θ^,σ2​diag⁡\(θ^12,…,θ^d2\)\)\.\\mathcal\{Q\}\_\{\\hat\{\\theta\},\\sigma\}^\{\\mathrm\{adap\}\}=\\mathcal\{N\}\\left\(\\hat\{\\theta\},\\,\\sigma^\{2\}\\operatorname\{diag\}\(\\hat\{\\theta\}\_\{1\}^\{2\},\\ldots,\\hat\{\\theta\}\_\{d\}^\{2\}\)\\right\)\.\(14\)This covariance assigns larger perturbations to parameters with larger magnitudes, which is the source of adaptivity in this posterior\.

Using this adaptive posterior, we define*Bayes sharpness*\(bayesS\) as the normalized posterior\-averaged loss increase:

SbayesS:=𝔼θ∼𝒬θ^,σadap​\[ℓ​\(fθ;Z\)\]−ℓ​\(fθ^;Z\)σ2\.S\_\{\\mathrm\{bayesS\}\}:=\\frac\{\\mathbb\{E\}\_\{\\theta\\sim\\mathcal\{Q\}\_\{\\hat\{\\theta\},\\sigma\}^\{\\mathrm\{adap\}\}\}\\big\[\\ell\(f\_\{\\theta\};Z\)\\big\]\-\\ell\(f\_\{\\hat\{\\theta\}\};Z\)\}\{\\sigma^\{2\}\}\.\(15\)The normalization byσ2\\sigma^\{2\}makes bayesS a loss increase rate with respect to the perturbation scale, improving comparability across different choices ofσ\\sigma\. Detailed discussion of bayesS is deferred to Appendix[C](https://arxiv.org/html/2606.29043#A3), and an ablation study comparing anisotropic and isotropic covariance choices is provided in Appendix[E\.2](https://arxiv.org/html/2606.29043#A5.SS2)\.

### 4\.3Function\-Level Complexity

We propose a function\-level complexity metric, which compares posterior and prior distributions after mapping parameter samples to their induced network outputs\.

Specifically, let𝒫θ\\mathcal\{P\}\_\{\\theta\}and𝒬θ\\mathcal\{Q\}\_\{\\theta\}denote the parameter\-space prior and posterior distributions\. Given a fixed reference datasetD=\{x1,…,xn\}D=\\\{x\_\{1\},\\dots,x\_\{n\}\\\}, we define a function\-output mapFDF\_\{D\}, which maps a parameter vectorθ\\thetato the vectorized model outputs onDD\.

FD:Θ→ℝn​c,FD​\(θ\):=vec⁡\(fθ​\(D\)\),F\_\{D\}:\\Theta\\to\\mathbb\{R\}^\{nc\},\\qquad F\_\{D\}\(\\theta\):=\\operatorname\{vec\}\\bigl\(f\_\{\\theta\}\(D\)\\bigr\),whereccis the model output dimension\.

Through this map, the parameter\-space prior and posterior induce corresponding function\-space distributions\. Samplingθ∼𝒫θ\\theta\\\!\\sim\\\!\\mathcal\{P\}\_\{\\theta\}and evaluating the network onDDinduces a prior distribution over function outputs, while samplingθ∼𝒬θ\\theta\\\!\\sim\\\!\\mathcal\{Q\}\_\{\\theta\}gives a posterior distribution over the function outputs\. We denote the induced function\-level distributions as𝒫fD:=\(FD\)\#​𝒫θ\\mathcal\{P\}\_\{f\}^\{D\}:=\(F\_\{D\}\)\_\{\\\#\}\\mathcal\{P\}\_\{\\theta\}and𝒬fD:=\(FD\)\#​𝒬θ\\mathcal\{Q\}\_\{f\}^\{D\}:=\(F\_\{D\}\)\_\{\\\#\}\\mathcal\{Q\}\_\{\\theta\}, where\#\\\#is the push forward map\. The*functional KL divergence*is defined as

KLfunc:=KL​\(𝒬fD∥𝒫fD\)\.\\mathrm\{KL\}\_\{\\mathrm\{func\}\}:=\\mathrm\{KL\}\\big\(\\mathcal\{Q\}\_\{f\}^\{D\}\\,\\\|\\,\\mathcal\{P\}\_\{f\}^\{D\}\\big\)\.\(16\)Conceptually, this metric measures how far the learned posterior is from the prior*in terms of the function outputs they induce*on the reference data, rather than in terms of the raw parameter values\. Therefore, it is less tied to a specific parameter representation and is better aligned with the reduced predictive behavior of the model\.

#### 4\.3\.1Practical Estimation ofKLfunc\\mathrm\{KL\}\_\{\\mathrm\{func\}\}

In practice, we assume the push\-forward function\-level distributions𝒬fD\\mathcal\{Q\}\_\{f\}^\{D\}and𝒫fD\\mathcal\{P\}\_\{f\}^\{D\}are both i\.i\.d\. Gaussian, whose parameters can be estimated by Monte Carlo sampling\.

Specifically, we draw parameter samples

θ𝒬\(k\)∼𝒬θ,θ𝒫\(k\)∼𝒫θ,k=1,…,K,\\theta\_\{\\mathcal\{Q\}\}^\{\(k\)\}\\sim\\mathcal\{Q\}\_\{\\theta\},\\qquad\\theta\_\{\\mathcal\{P\}\}^\{\(k\)\}\\sim\\mathcal\{P\}\_\{\\theta\},\\qquad k=1,\\dots,K,evaluate each sampled model on the reference datasetDD, and collect the resulting function\-output vectors\. This produces two point clouds in the function\-output space: one induced by the posterior and one induced by the prior\. We approximate both point clouds by diagonal Gaussian distributions and compute the corresponding KL divergence \([9](https://arxiv.org/html/2606.29043#S4.E9)\)\. This gives the empirical*estimator*

KL^func=12​∑j=1df\[−1\+\(v^𝒬f\)j\(v^𝒫f\)j\+ln⁡\(\(v^𝒫f\)j\(v^𝒬f\)j\)\+\(\(μ^𝒬f\)j−\(μ^𝒫f\)j\)2\(v^𝒫f\)j\],\\widehat\{\\mathrm\{KL\}\}\_\{\\mathrm\{func\}\}=\\frac\{1\}\{2\}\\sum\_\{j=1\}^\{d\_\{f\}\}\\left\[\-1\+\\frac\{\(\\widehat\{v\}\_\{\\mathcal\{Q\}\_\{f\}\}\)\_\{j\}\}\{\(\\widehat\{v\}\_\{\\mathcal\{P\}\_\{f\}\}\)\_\{j\}\}\+\\ln\\left\(\\frac\{\(\\widehat\{v\}\_\{\\mathcal\{P\}\_\{f\}\}\)\_\{j\}\}\{\(\\widehat\{v\}\_\{\\mathcal\{Q\}\_\{f\}\}\)\_\{j\}\}\\right\)\+\\frac\{\\big\(\(\\widehat\{\\mu\}\_\{\\mathcal\{Q\}\_\{f\}\}\)\_\{j\}\-\(\\widehat\{\\mu\}\_\{\\mathcal\{P\}\_\{f\}\}\)\_\{j\}\\big\)^\{2\}\}\{\(\\widehat\{v\}\_\{\\mathcal\{P\}\_\{f\}\}\)\_\{j\}\}\\right\],\(17\)wheredfd\_\{f\}is the dimension of the function\-output representationdf=n​cd\_\{f\}=nc\. Here\(μ^𝒬f\)j\(\\widehat\{\\mu\}\_\{\\mathcal\{Q\}\_\{f\}\}\)\_\{j\}and\(v^𝒬f\)j\(\\widehat\{v\}\_\{\\mathcal\{Q\}\_\{f\}\}\)\_\{j\}are the coordinate\-wise empirical mean and variance of theKKposterior function\-output samples, while\(μ^𝒫f\)j\(\\widehat\{\\mu\}\_\{\\mathcal\{P\}\_\{f\}\}\)\_\{j\}and\(v^𝒫f\)j\(\\widehat\{v\}\_\{\\mathcal\{P\}\_\{f\}\}\)\_\{j\}are the corresponding quantities for the prior function\-output samples\.

##### SimplifiedKLfunc\\mathrm\{KL\}\_\{\\mathrm\{func\}\}estimator\.

Inspired by the way*weight decay*arises from a zero\-centered Gaussian prior with a shared prior–posterior covariance in parameter space \([10](https://arxiv.org/html/2606.29043#S4.E10)\), we introduce an analogous simplification in function space\.

We assume that the induced function\-output prior is zero\-centered,

\(μ^𝒫f\)j=0,j=1,…,df\(\\widehat\{\\mu\}\_\{\\mathcal\{P\}\_\{f\}\}\)\_\{j\}=0,\\qquad j=1,\\ldots,d\_\{f\}and that the prior and posterior share the same isotropic function\-output variance,

\(v^𝒫f\)j=\(v^𝒬f\)j=σf2,j=1,…,df\.\(\\widehat\{v\}\_\{\\mathcal\{P\}\_\{f\}\}\)\_\{j\}=\(\\widehat\{v\}\_\{\\mathcal\{Q\}\_\{f\}\}\)\_\{j\}=\\sigma\_\{f\}^\{2\},\\qquad j=1,\\ldots,d\_\{f\}\.Under these simplifying assumptions, Eq\. \([17](https://arxiv.org/html/2606.29043#S4.E17)\) reduces to a function\-space approximation:

KL^func≈12​σf2​∑j=1df\(μ^𝒬f\)j2,\\widehat\{\\mathrm\{KL\}\}\_\{\\mathrm\{func\}\}\\approx\\frac\{1\}\{2\\sigma\_\{f\}^\{2\}\}\\sum\_\{j=1\}^\{d\_\{f\}\}\\big\(\\widehat\{\\mu\}\_\{\\mathcal\{Q\}\_\{f\}\}\\big\)\_\{j\}^\{2\}\\,,\(18\)whereσf\\sigma\_\{f\}is a fixed scalar whose value does not affect the regression and PCR analysis results\. In practice, the meanμ^𝒬f\\widehat\{\\mu\}\_\{\\mathcal\{Q\}\_\{f\}\}can further be approximated by theFD​\(θ^\)F\_\{D\}\(\\hat\{\\theta\}\), by assuming the posterior to be centered around the trained model as in the derivation of the weight decay\. As a result,

KL^func∼‖FD​\(θ^\)‖2\.\\widehat\{\\mathrm\{KL\}\}\_\{\\mathrm\{func\}\}\\sim\\\|F\_\{D\}\(\\hat\{\\theta\}\)\\\|^\{2\}\.\(19\)
We use Eq\. \([19](https://arxiv.org/html/2606.29043#S4.E19)\) as the*practical recipe*to estimateKLfunc\{\\mathrm\{KL\}\}\_\{\\mathrm\{func\}\}, which is fully function level\. Implementation details are provided in Appendix[D\.2](https://arxiv.org/html/2606.29043#A4.SS2)\.

## 5Empirical Results

### 5\.1Interpreting the Two Evaluation Tools Together

Linear regression and Pareto analysis provide complementary tools to evaluate the explanatory power of an\(S,C\)\(S,C\)metric pair\. Interpreting them together leads to several possible outcomes\.

1. i\)*Low PCR, highR2R^\{2\}, and positive regression coefficients\.*This is the ideal case\. It suggests that the\(S,C\)\(S,C\)metrics have strong explanatory power and that the dominant relationship between sharpness–complexity and generalization is well captured by a linear model\.
2. ii\)*Low PCR, lowR2R^\{2\}\.*This suggests that the\(S,C\)\(S,C\)metrics induce a consistent monotonic ordering with generalization, but that the relationship is not well described by a linear model\. In this case, the two\-factor framework remains explanatory, but the dependence may be nonlinear\.
3. iii\)*High or moderate PCR, highR2R^\{2\}, and positive regression coefficients\.*In this case, the\(S,C\)\(S,C\)metrics have useful linear explanatory power, but the elevated PCR indicates substantial ordering violations\. This suggests that the linear trend captures part of the relationship, while important information may still be missing\.
4. iv\)*High PCR, lowR2R^\{2\}, or negative regression coefficients\.*This indicates that the evaluated\(S,C\)\(S,C\)metrics are poor realizations of sharpness and complexity in the given setting\. They neither provide a strong linear explanation nor induce a monotonic ordering consistent with the expected relationship between lower sharpness–complexity and better generalization\.

We note that low PCR rarely comes with a negative sign of the regression coefficients for the following reason\. A low PCR indicates strong agreement between\(S,C\)\(S,C\)\-ordering and the generalization ordering: models with both smaller sharpness and smaller complexity tend to have smaller generalization gap\. In such cases, the empirical Pareto frontier \(Figure[2](https://arxiv.org/html/2606.29043#S3.F2)\) usually reflects a trade\-off between the two factors, and therefore the frontier itself has a negative slope in the\(S,C\)\(S,C\)\-plane\. This negative slope does not imply a negative regression coefficient for either factor; rather, it is consistent with positive marginal effects of bothSSandCCon the generalization gap\. Thus, when PCR is low, negative regression coefficients are expected to be rare\.

##### Greater emphasis on PCR\.

The joint interpretation above*places greater weight on PCR*than onR2R^\{2\}, because PCR does not rely on a linear modeling assumption\. For example, a result with low PCR but lowR2R^\{2\}is still considered favorable, as it suggests a strong agreement on the monotonic ordering even if the relationship is not well captured by a linear model\. In contrast, a result with high PCR and lowR2R^\{2\}indicates poor explanatory power under both the order\-based and regression\-based evaluations\. Quantitatively, in the main results in Table[4](https://arxiv.org/html/2606.29043#S5.T4), the “good” category requires a stringent PCR threshold\(≤10%\)\(\\leq 10\\%\)together with a relatively mild regression requirement\(R2≥0\.6,\(\+,\+\)\)\\big\(R^\{2\}\\\!\\geq\\\!0\.6,\(\+,\+\)\\big\)\.

### 5\.2Main Results and Interpretations

Our main empirical results are reported in Table[4](https://arxiv.org/html/2606.29043#S5.T4)\. We evaluate the existing parameter\-level baseline metric pair and the proposed function\-oriented metric pair under*three settings*\.

The*upper*block reports results on normalization\-equipped architectures, while the*middle*block reports results on normalization\-disabled architectures\. In the*lower*block, we evaluate cross\-architecture and cross\-dataset explanatory power by pooling trained models from different architectures into a single collection\. This is inspired by the fact that the PAC\-Bayes bounds are not inherently restricted to a single architecture\. The hypothesis spaceℋ\\cal\{H\}can contain predictors from multiple architectures, as long as the prior is defined over this common space before observing the data\. This makes the PAC\-Bayes bound conceptually compatible in cross\-architecture comparisons, which we use as a stress\-test, checking whether a metric pair remains predictive in mixed\-architecture settings, even when it performs well within each architecture individually\.

We now take a closer look at the observations from Table[4](https://arxiv.org/html/2606.29043#S5.T4)\.

- ⊳\\trianglerightIn the*upper*block, which considers normalization\-*equipped*architectures, the new metric pair visibly improvesR2R^\{2\}and PCR in settings where the existing baseline metric pair is weak\. For example, on VGG19, \(bayesS, func norm\) reduces PCR from37\.0%37\.0\\%to10\.5%10\.5\\%and increasesR2R^\{2\}from 0\.35 to 0\.65\. On architectures where \(adapS, path norm\) already works reasonably well, \(bayesS, func norm\) remains competitive, often achieving lower PCR with comparableR2R^\{2\}\. However, the new metric remains weak on ViT, where a negative regression coefficient and a PCR of20\.8%20\.8\\%were obtained, indicating further improvements of the metrics may be needed\.
- ⊳\\trianglerightIn the*middle*block, which considers normalization\-*disabled*architectures, the new metric pair greatly increases the explanatory power of the two factors\. For no\-BN ResNet18, the existing baseline \(adapS, path norm\) collapses, withPCR=77\.8%,R2=0\.07\\mathrm\{PCR\}\\\!=\\\!77\.8\\%,\\,R^\{2\}\\\!=\\\!0\.07, and negative regression coefficients\. In contrast, \(bayesS, func norm\) achievesPCR=1\.2%,R2=0\.83\\mathrm\{PCR\}\\\!=\\\!1\.2\\%,\\,R^\{2\}\\\!=\\\!0\.83, and positive coefficients\. Similar improvements are observed for no\-BN WideResNet and no\-BN VGG13\. These results suggest that defining the\(S,C\)\(S,C\)metrics closer to function space can extend the explanatory scope of the two\-factor view when parameter\-level metrics are insufficient\. Interestingly, although \(bayesS, func norm\) fails on normalization\-equipped ViT, it shows good explanatory power on LayerNormalization\-diabled ViT, withPCR=13\.6%,R2=0\.87\\mathrm\{PCR\}=13\.6\\%,R^\{2\}=0\.87\.
- ⊳\\trianglerightIn the*lower*block, corresponding to cross\-architecture and cross\-dataset settings, the mixed collections further test whether a metric pair remains comparable*across different model families and datasets*\. In the CIFAR\-10 Mixed Three setting, the existing baseline loses explanatory power, whereas the function\-oriented pair \(bayesS, func norm\) recovers good performance withPCR=5\.1%,R2=0\.74\\mathrm\{PCR\}=5\.1\\%,R^\{2\}=0\.74\. Another informative setting is Mixed Four\(cross\-data\), which pools ResNet18, VGG13 trained on CIFAR\-10 together with ResNet34 and WideResNet trained on CIFAR\-100\. Although the existing baseline \(adapS, path norm\) works well on these architectures individually, it loses explanatory power when they are combined into one collection\. Surprisingly, \(bayesS, func norm\) remains predictive in this mixed setting, achievingPCR=3\.3%,R2=0\.68\\mathrm\{PCR\}=3\.3\\%,R^\{2\}=0\.68\. This suggests that \(adapS, path norm\) may provide useful within\-architecture rankings, but its values are not always comparable across architectures or datasets\. The new metric pair is less tied to raw parameter representations and therefore provides a more stable basis for cross\-architecture comparison\. In the Mixed Eight\(cross\-data\) setting, 278 valid trained models from eight different architectures are pooled together and treated as one collection; the PCR is visualized in Figure[6\(m\)](https://arxiv.org/html/2606.29043#S5.F6.sf13)\. The function\-oriented metric pair \(bayesS, func norm\) still achieves a relatively low PCR of 8\.7%, but its regressionR2R^\{2\}drops to 0\.44\. This suggests that the monotonic ordering is partly preserved, while the quantitative relationship is no longer well captured by a simple linear model\.

Figure[4](https://arxiv.org/html/2606.29043#S3.F4),[5](https://arxiv.org/html/2606.29043#S5.F5), and[6](https://arxiv.org/html/2606.29043#S5.F6)provide visualizations of the PCR column of Table[4](https://arxiv.org/html/2606.29043#S5.T4)\.

DatasetNetworkArchitecture\(𝐒,𝐂\)\\mathbf\{\(S,C\)\}Metric PairRegressionR2↑R^\{2\}\\uparrow, coefficientPCR↓\\downarrowVisualized inFig\.[4](https://arxiv.org/html/2606.29043#S3.F4),[5](https://arxiv.org/html/2606.29043#S5.F5),[6](https://arxiv.org/html/2606.29043#S5.F6)ExplanatoryPowerfailed/poor/moderate/good/excellentCIFAR\-10ResNet18\(adapS, path norm\)0\.93,\(\+,\+\)\(\+,\+\)9\.4%\(bayesS, func norm\)0\.68,\(\+,\+\)\(\+,\+\)2\.8%*\(ours\)*VGG13\(adapS, path norm\)0\.85,\(\+,\+\)\(\+,\+\)22\.2%\(bayesS, func norm\)0\.72,\(\+,\+\)\(\+,\+\)3\.5%*\(ours\)*CIFAR\-100ResNet34\(adapS, path norm\)0\.87,\(\+,\+\)\(\+,\+\)18\.5%\(bayesS, func norm\)0\.78,\(\+,\+\)\(\+,\+\)6\.4%*\(ours\)*VGG19\(adapS, path norm\)0\.35,\(\+,\+\)\(\+,\+\)37\.0%\(bayesS, func norm\)0\.65,\(\+,\+\)\(\+,\+\)10\.5%*\(ours\)*WideResNet\(adapS, path norm\)0\.89,\(\+,\+\)\(\+,\+\)8\.2%\(bayesS, func norm\)0\.91,\(\+,\+\)\(\+,\+\)1\.0%*\(ours\)*ViT\(adapS, path norm\)–––\(bayesS, func norm\)0\.88,\(−,\+\)\(\-,\+\)20\.8%*\(ours\)*CIFAR\-10no\-BN ResNet18\(adapS, path norm\)0\.07,\(−,−\)\(\-,\-\)77\.8%\(bayesS, func norm\)0\.83,\(\+,\+\)\(\+,\+\)1\.2%*\(ours\)*no\-BN VGG13\(adapS, path norm\)0\.60,\(\+,\+\)\(\+,\+\)63\.4%\(bayesS, func norm\)0\.70,\(\+,\+\)\(\+,\+\)11\.4%*\(ours\)*CIFAR\-100no\-BN ResNet34\(adapS, path norm\)–––\(bayesS, func norm\)–––no\-BN VGG19\(adapS, path norm\)–––\(bayesS, func norm\)–––no\-BN WideResNet\(adapS, path norm\)0\.25,\(−,\+\)\(\-,\+\)47\.3%\(bayesS, func norm\)0\.71,\(\+,\+\)\(\+,\+\)17\.9%*\(ours\)*no\-LN ViT\(adapS, path norm\)–––\(bayesS, func norm\)0\.87,\(\+,\+\)\(\+,\+\)13\.6%*\(ours\)*CIFAR\-10Mixed Two\(adapS, path norm\)0\.84,\(\+,\+\)\(\+,\+\)19\.1%\(bayesS, func norm\)0\.70,\(\+,\+\)\(\+,\+\)4\.0%*\(ours\)*Mixed Three\(adapS, path norm\)0\.22,\(−,\+\)\(\-,\+\)59\.4%\(bayesS, func norm\)0\.74,\(\+,\+\)\(\+,\+\)5\.1%*\(ours\)*CIFAR\-100Mixed Two\(adapS, path norm\)0\.75,\(\+,\+\)\(\+,\+\)17\.8%\(bayesS, func norm\)0\.65,\(\+,\+\)\(\+,\+\)9\.9%*\(ours\)*Mixed Three\(adapS, path norm\)0\.34,\(−,\+\)\(\-,\+\)28\.0%\(bayesS, func norm\)0\.80,\(\+,\+\)\(\+,\+\)6\.1%*\(ours\)*CIFAR\-10&100Mixed Four\(cross\-data\)\(adapS, path norm\)0\.34,\(\+,\+\)\(\+,\+\)26\.2%\(bayesS, func norm\)0\.68,\(\+,\+\)\(\+,\+\)3\.3%*\(ours\)*Mixed Six\(cross\-data\)\(adapS, path norm\)0\.10,\(\+,\+\)\(\+,\+\)44\.3%\(bayesS, func norm\)0\.79,\(\+,\+\)\(\+,\+\)2\.1%*\(ours\)*Mixed Eight\(cross\-data\)\(adapS, path norm\)–––\(bayesS, func norm\)0\.44,\(\+,\+\)\(\+,\+\)8\.7%*\(ours\)*

Table 4:Main results\.Thelinear regressionandPareto analysisfor existing baseline metric pair and the*proposed function\-oriented*metric pair\. The*upper, middle*, and*lower*blocks correspond to normalization\-equipped, normalization\-disabled, and cross\-architecture/dataset mixed settings, respectively; the Mixed settings are specified in Table[5](https://arxiv.org/html/2606.29043#S5.T5)\. Since path norm has no standard comparable definitions for ViT, the corresponding entries are left blank\. The proposed functional complexityKLfunc\\mathrm\{KL\}\_\{\\mathrm\{func\}\}is denoted as func norm; see Appendix[D\.2](https://arxiv.org/html/2606.29043#A4.SS2)\. Visualizations of PCR are provided in Figure[4](https://arxiv.org/html/2606.29043#S3.F4),[5](https://arxiv.org/html/2606.29043#S5.F5), and[6](https://arxiv.org/html/2606.29043#S5.F6)\. The last column summarizes the explanatory power:*strong results require highR2​\(↑\)R^\{2\}\(\\uparrow\)and low PCR\(↓\)\(\\downarrow\)*;*failed*indicates regression returns negative coefficients,*good*indicatesPCR≤10%\\mathrm\{PCR\}\\\!\\leq\\\!10\\%andR2≥0\.6,\(\+,\+\)R^\{2\}\\\!\\geq\\\!0\.6,\(\+,\+\), intermediate cases are labeled as*poor*or*moderate*; the*categories*are visualized by the status\-dot bar and partially highlighted by shaded colors\.Conclusions:the proposed*\(bayesS, func norm\)*extends the explanatory scope of the two\-factor view in settings where the existing baseline \(adapS, path norm\) fails, while remaining failures such as ViT, show that the explanation is still incomplete\.DatasetMixed SettingNetworks being pooled togetherCIFAR\-10Mixed TwoResNet18, VGG13Mixed ThreeResNet18, VGG13, no\-BN VGG13CIFAR\-100Mixed TwoResNet34, WideResNetMixed ThreeResNet34, WideResNet, no\-BN WideResNetCIFAR\-10&100Mixed Four\(cross\-data\)ResNet18, VGG13, ResNet34, WideResNetMixed Six\(cross\-data\)ResNet18, VGG13, WideResNet w/ and w/o normalizationMixed Eight\(cross\-data\)ResNet18, VGG13, WideResNet, ViT w/ and w/o normalization

Table 5:Network combinations used in the Mixed settings reported in the*lower*block of Table[4](https://arxiv.org/html/2606.29043#S5.T4)\.##### A diagnostic look at the remaining failure on ViT\.

We do not have a definitive answer for why the function\-oriented metric pair*\(bayesS,KL*func*\\mathrm\{\\textit\{KL\}\}\_\{\\mathrm\{\\textit\{func\}\}\}\)*fails on ViT\. Nevertheless, we offer two conjectures\.

First, Bayes sharpness is not yet fully function\-level: its posterior is still defined through a scale\-invariant parameter\-space perturbation, and this perturbation geometry may not be well suited to ViT\. Since the negative regression coefficient in Table[4](https://arxiv.org/html/2606.29043#S5.T4)is associated with bayesS rather thanKLfunc\\mathrm\{KL\}\_\{\\mathrm\{func\}\}, the failure appears to be driven more by the sharpness component than by the functional complexity\.

A second conjecture comes from examining the PCR visualization in Figure[5\(f\)](https://arxiv.org/html/2606.29043#S5.F5.sf6)\. We observe that most trained ViT models have very large test losses222Pareto plots annotate each point by its generalization gap, computed as test loss minus training loss\. Since the training loss is nearly zero, large generalization gaps correspond to large test losses\., in many cases even larger than the trivial loss of a uniform predictor,ℓ=ln⁡100≈4\.6\\ell\\\!=\\\!\\ln 100\\\!\\approx\\\!4\.6\. This suggests that these models severely overfit the training data and are largely outside the meaningful generalization regime\. In such a regime, the PAC\-Bayes bound is likely to be highly loose or vacuous, and it is therefore unsurprising that the two\-factor view fails to explain the observed behavior\. Motivated by this observation, we conduct an ablation study in which ViT models are trained with data augmentation\. The results show that the explanatory power of the two\-factor view under \(bayesS,KLfunc\\mathrm\{KL\}\_\{\\mathrm\{func\}\}\) improves when the ViT test losses become less extreme: regression coefficients turn to positive withR2=0\.89R^\{2\}\\\!=\\\!0\.89, although the PCR remains mildly high at 15\.2%\. See Appendix[E\.3](https://arxiv.org/html/2606.29043#A5.SS3)for details\.

##### Taken together, our main results suggest the following:

- •The two\-factor framework is informative across a broad range of settings\. Moreover, defining sharpness and complexity closer to function space can extend its explanatory scope\.
- •A surprising finding is that, under the proposed function\-oriented metric pair, the explanatory scope of the two\-factor view can extend to certain cross\-architecture and cross\-dataset settings\.
- •Some failures remain, such as ViT, indicating that the current metric realizations do not yet provide a complete explanation of generalization\.

These results answer Question \([∗⁣∗⁣∗\*\\\!\*\\\!\*‣3\.2\.2](https://arxiv.org/html/2606.29043#S3.Ex9)\) in a twofold way\. On the one hand, some failures of existing metrics arise from limitations of the current parameter\-level definitions: when sharpness and complexity are defined closer to function level, the explanatory power of the two\-factor view visibly improves in most CNN settings\. On the other hand, the remaining failures on ViT show that, current metric realizations do not make the two\-factor view fully explain generalization in all regimes\.

Thus, our results leave open two possibilities\. Either the explanatory scope of the two\-factor view can be further extended \(perhaps even made complete\) through better definitions of sharpness and complexity, or the two\-factor framework is intrinsically incomplete and needs to be complemented by additional factors beyond\(S,C\)\(S,C\)\.

More detailed experimental protocols are provided in Appendix[A](https://arxiv.org/html/2606.29043#A1)\.

![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c10_resnet18_bayeS_funcnorm.png)\(a\)CIFAR\-10, ResNet18\.PCR=2\.8%\\text\{PCR\}\\\!=\\\!2\.8\\%\.
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c10_vgg13_bayeS_funcnorm.png)\(b\)CIFAR\-10, VGG13\.PCR=3\.5%\\text\{PCR\}\\\!=\\\!3\.5\\%
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c100_wideresnet_bayeS_funcnorm.png)\(c\)CIFAR\-100, WideResNet\.PCR=1\.0%\\text\{PCR\}\\\!=\\\!1\.0\\%
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c100_resnet34_bayeS_funcnorm.png)\(d\)CIFAR\-100, ResNet34\.PCR=6\.4%\\text\{PCR\}\\\!=\\\!6\.4\\%
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c100_vgg19_bayeS_funcnorm.png)\(e\)CIFAR\-100, VGG19\.PCR=10\.5%\\text\{PCR\}\\\!=\\\!10\.5\\%
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c100_vit_bayeS_funcnorm.png)\(f\)CIFAR\-100, ViT\.PCR=20\.8%\\text\{PCR\}\\\!=\\\!20\.8\\%
Normalization\-*equipped*networks, the*upper*block of Table[4](https://arxiv.org/html/2606.29043#S5.T4)\.

![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c10_nobnresnet18_bayeS_funcnorm_dark.png)\(g\)CIFAR\-10, no\-BN ResNet18\.PCR=1\.2%\\text\{PCR\}\\\!=\\\!1\.2\\%
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c10_nobnvgg13_bayeS_funcnorm_dark.png)\(h\)CIFAR\-10, no\-BN VGG13\.PCR=11\.4%\\text\{PCR\}\\\!=\\\!11\.4\\%
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c100_nobn_wideresnet_bayeS_funcnorm_dark.png)\(i\)CIFAR\-100, no\-BN WideResNet\.PCR=17\.9%\\text\{PCR\}\\\!=\\\!17\.9\\%
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c100_nobn_vit_bayeS_funcnorm_dark.png)\(j\)CIFAR\-100, no\-LN ViT\.PCR=13\.6%\\text\{PCR\}\\\!=\\\!13\.6\\%
Normalization\-*disabled*networks, the*middle*block of Table[4](https://arxiv.org/html/2606.29043#S5.T4)\.

Figure 5:Pareto analysisof the proposed function\-oriented metrics\(bayesS, func norm\)for the*upper*and*middle*block of Table[4](https://arxiv.org/html/2606.29043#S5.T4), visualizing the PCR column\. Visualizations of \(adapS, path norm\) have been shown in Figure[4](https://arxiv.org/html/2606.29043#S3.F4)\. Here, func norm denotes the functional complexityKLfunc\\mathrm\{KL\}\_\{\\mathrm\{func\}\}; see Appendix[D\.2](https://arxiv.org/html/2606.29043#A4.SS2)\. Each point represents a trained model, annotated by its generalization gap\. The*first two rows*show normalization\-equipped networks, and the*last two rows*show normalization\-disabled networks\. A*lower PCR*\(↓\)\(\\downarrow\)indicates*better*agreement with the expected monotonic relationship\.![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c10_mixedtwo_adapS_pathnorm.png)\(a\)Mixed Two\.PCR=19\.1%\\text\{PCR\}\\\!=\\\!19\.1\\%\.
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c10_mixedtwo_bayeS_funcnorm.png)\(b\)Mixed Two\.PCR=4\.0%\\text\{PCR\}\\\!=\\\!4\.0\\%\.
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c10_mixedthree_adapS_pathnorm.png)\(c\)Mixed Three\.PCR=59\.4%\\text\{PCR\}\\\!=\\\!59\.4\\%\.
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c10_mixedthree_bayeS_funcnorm.png)\(d\)Mixed Three\.PCR=5\.1%\\text\{PCR\}\\\!=\\\!5\.1\\%\.
CIFAR\-10\.

![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c100_mixedtwo_adapS_pathnorm.png)\(e\)Mixed Two\.PCR=17\.8%\\text\{PCR\}\\\!=\\\!17\.8\\%\.
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c100_mixedtwo_bayeS_funcnorm.png)\(f\)Mixed Two\.PCR=9\.9%\\text\{PCR\}\\\!=\\\!9\.9\\%\.
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c100_mixedthree_adapS_pathnorm.png)\(g\)Mixed Three\.PCR=28\.0%\\text\{PCR\}\\\!=\\\!28\.0\\%\.
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c100_mixedthree_bayeS_funcnorm.png)\(h\)Mixed Three\.PCR=6\.1%\\text\{PCR\}\\\!=\\\!6\.1\\%\.
CIFAR\-100\.

![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c10_100_mixedfour_adapS_pathnorm.png)\(i\)Mixed Four\.PCR=26\.2%\\text\{PCR\}\\\!=\\\!26\.2\\%\.
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c10_100_mixedfour_bayeS_funcnorm.png)\(j\)Mixed Four\.PCR=3\.3%\\text\{PCR\}\\\!=\\\!3\.3\\%\.
CIFAR\-10&100\.

![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c10_100_mixedsix_adapS_pathnorm.png)\(k\)Mixed Six\.PCR=44\.3%\\text\{PCR\}\\\!=\\\!44\.3\\%\.
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c10_100_mixedsix_bayeS_funcnorm.png)\(l\)Mixed Six\.PCR=2\.1%\\text\{PCR\}\\\!=\\\!2\.1\\%\.
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c10_100_mixedeight_bayeS_funcnorm.png)\(m\)Mixed Eight\.PCR=8\.7%\\text\{PCR\}\\\!=\\\!8\.7\\%\.
CIFAR\-10&100\.

Figure 6:Pareto analysisof the exiting baseline metrics\(adapS, path norm\)and the proposed metrics\(bayesS, func norm\)for the*lower*block \(Mixed settings\) of Table[4](https://arxiv.org/html/2606.29043#S5.T4), visualizing the PCR column\. Networks pooled in each mixed setting are specified in Table[5](https://arxiv.org/html/2606.29043#S5.T5)\. Here, func norm denotes the functional complexityKLfunc\\mathrm\{KL\}\_\{\\mathrm\{func\}\}; see Appendix[D\.2](https://arxiv.org/html/2606.29043#A4.SS2)\. Each point represents a trained model and is annotated by its generalization gap\. Since the Mixed Eight contains ViT, for which no standard comparable path norm definition is available, the corresponding Pareto plot is omitted\. Mixed Six and Mixed Eight pool 219 models from six architectures and 278 models from eight architectures, respectively; the large number of points makes the plots visually crowded\. A*lower PCR*\(↓\)\(\\downarrow\)indicates*better*agreement with the expected monotonic relationship\.
#### 5\.2\.1Single\-Factor View as a Baseline Comparison

Table[6](https://arxiv.org/html/2606.29043#S5.T6)reports single\-factor regression results, where each sharpness*or*complexity metric is used individually as*the sole predictor*of the generalization gap\. The results show that the explanatory power of single factorSSorCCis generally weak and highly inconsistent on different networks, regardless of whether the metric is defined at the parameter level or the function level\. This observation is consistent with the large\-scale empirical study of Jiang et al\.\[[16](https://arxiv.org/html/2606.29043#bib.bib3)\], and supports the view that a single\-factor perspective is insufficient for explaining generalization\.

DatasetNetworkArchitectureSingle\-Factor RegressionR2↑R^\{2\}\\uparrow, coefficientadapS*bayesS*path norm*func norm*CIFAR\-10ResNet180\.56,\(\+\)\(\+\)0\.46,\(\+\)\(\+\)0\.90,\(\+\)\(\+\)0\.14,\(\+\)\(\+\)VGG130\.35,\(\+\)\(\+\)0\.36,\(\+\)\(\+\)0\.84,\(\+\)\(\+\)0\.23,\(\+\)\(\+\)CIFAR\-100ResNet340\.52,\(\+\)\(\+\)0\.60,\(\+\)\(\+\)0\.82,\(\+\)\(\+\)0\.24,\(\+\)\(\+\)VGG190\.34,\(\+\)\(\+\)0\.30,\(\+\)\(\+\)0\.17,\(\+\)\(\+\)0\.17,\(\+\)\(\+\)WideResNet0\.53,\(\+\)\(\+\)0\.76,\(\+\)\(\+\)0\.78,\(\+\)\(\+\)0\.77,\(\+\)\(\+\)ViT0\.23,\(−\)\(\-\)0\.01,\(\+\)\(\+\)–0\.87,\(\+\)\(\+\)CIFAR\-10no\-BN ResNet180\.04,\(−\)\(\-\)0\.02,\(\+\)\(\+\)0\.00,\(−\)\(\-\)0\.76,\(\+\)\(\+\)no\-BN VGG130\.60,\(\+\)\(\+\)0\.63,\(\+\)\(\+\)0\.41,\(\+\)\(\+\)0\.50,\(\+\)\(\+\)CIFAR\-100no\-BN ResNet34––––no\-BN VGG19––––no\-BN WideResNet0\.02,\(−\)\(\-\)0\.02,\(−\)\(\-\)0\.22,\(\+\)\(\+\)0\.71,\(\+\)\(\+\)no\-BN ViT0\.02,\(\+\)\(\+\)0\.04,\(\+\)\(\+\)–0\.86,\(\+\)\(\+\)CIFAR\-10Mixed Two0\.43,\(\+\)\(\+\)0\.43,\(\+\)\(\+\)0\.74,\(\+\)\(\+\)0\.18,\(\+\)\(\+\)Mixed Three0\.01,\(\+\)\(\+\)0\.08,\(\+\)\(\+\)0\.22,\(\+\)\(\+\)0\.70,\(\+\)\(\+\)CIFAR\-100Mixed Two0\.36,\(\+\)\(\+\)0\.57,\(\+\)\(\+\)0\.69,\(\+\)\(\+\)0\.46,\(\+\)\(\+\)Mixed Three0\.00,\(\+\)\(\+\)0\.03,\(\+\)\(\+\)0\.33,\(\+\)\(\+\)0\.76,\(\+\)\(\+\)CIFAR\-10&100Mixed Six\(cross\-data\)0\.01,\(\+\)\(\+\)0\.09,\(\+\)\(\+\)0\.10,\(\+\)\(\+\)0\.75,\(\+\)\(\+\)Mixed Eight\(cross\-data\)0\.02,\(\+\)\(\+\)0\.04,\(\+\)\(\+\)–0\.42,\(\+\)\(\+\)

Table 6:Single\-factor regressionresults for the existing baseline metrics and*proposed function\-oriented*metrics\. The*upper*,*middle*, and*lower*blocks correspond to normalization\-equipped, normalization\-disabled, and cross\-architecture/dataset mixed settings, respectively\. The setup is identical to Table[4](https://arxiv.org/html/2606.29043#S5.T4), except that each sharpness or complexity metric is evaluated individually as a sole regression predictor\. Strong result is implied by*high*R2​\(↑\)R^\{2\}\(\\uparrow\)with a positive coefficient\. The results suggest that the single\-factor view is insufficient for explaining generalization, consistent with the findings of Jiang et al\.\[[16](https://arxiv.org/html/2606.29043#bib.bib3)\]\.

## 6Discussion

In this paper, we introduced Pareto analysis and new sharpness and complexity metrics as probes for addressing the*main question of interest*: how far can sharpness and complexity explain generalization?

Our main results in Table[4](https://arxiv.org/html/2606.29043#S5.T4)suggest a nuanced answer\. The two\-factor view is informative across a broad range of settings\. In many normalization\-equipped architectures, existing baseline metrics already capture meaningful variations in generalization\. However, their explanatory power is not consistent: they can fail when normalization is removed or when models from different architectures are pooled together\. Defining sharpness and complexity closer to function space extends the explanatory scope\. In particular, the proposed function\-oriented metric pair \(bayesS, func norm\) recovers explanatory power in several normalization\-disabled and mixed\-architecture settings where the existing parameter\-level baseline fails\. This suggests that some apparent failures of the two\-factor view are not failures of the framework itself, but rather failures of how sharpness and complexity are realized\.

At the same time, our results do not show that\(S,C\)\(S,C\)forms a complete explanation of generalization\. Some failures remain, leaving open two possibilities\. One possibility is that the explanatory scope of the two\-factor view can be further extended \(perhaps even made complete\) by developing better, more fully function\-level definitions of sharpness and complexity\. Another possibility is that the two\-factor framework is intrinsically incomplete, so additional factors beyond\(S,C\)\(S,C\)are needed for a comprehensive explanation\. Thus, our results*support the two\-factor view as an informative explanatory lens, while leaving open whether it can serve as a complete theory\.*

This work has several limitations\. First, the proposed sharpness metric, bayesS, is semi\-function\-level\. It measures posterior\-induced loss increase and therefore captures changes in predictive behavior, but the posterior involved is still defined in parameter space\. In contrast, the proposed complexity metricKLfunc\\mathrm\{KL\}\_\{\\mathrm\{func\}\}is function\-level, because it is computed from distributions over function outputs induced by parameter samples\. A more fully function\-level sharpness metric remains an important direction for future work\. Second, although we provide a diagnostic explanation for the remaining ViT failure, the issue is not fully resolved\. Data augmentation mitigates the extremely large test losses and improves regression predictability, but the test losses remain large relative to the trivial loss of a uniform predictor, and the PCR remains mildly high\. This suggests that some ViT models may still lie outside a meaningful generalization regime, where PAC\-Bayes bounds can be loose or vacuous and where the two\-factor view is difficult to evaluate reliably\. Finally, our experiments are limited to image classification tasks and a finite collection of architectures\. These limitations naturally point to meaningful directions for further exploring the boundary and potential extensions of the two\-factor framework\.

## References

- \[1\]P\. Alquier, J\. Ridgway, and N\. Chopin\(2016\)Properties of variational approximations of gibbs posteriors\.Journal of Machine Learning Research17\(236\),pp\. 1–41\.Cited by:[§4\.1\.1](https://arxiv.org/html/2606.29043#S4.SS1.SSS1.p4.1)\.
- \[2\]P\. Alquier\(2024\)User\-friendly introduction to pac\-bayes bounds\.Foundations and Trends in Machine Learning17\(2\),pp\. 174–303\.External Links:[Document](https://dx.doi.org/10.1561/2200000100)Cited by:[§4\.1\.1](https://arxiv.org/html/2606.29043#S4.SS1.SSS1.p3.2),[§4\.1\.1](https://arxiv.org/html/2606.29043#S4.SS1.SSS1.p4.1)\.
- \[3\]M\. Andriushchenko and N\. Flammarion\(2022\)Towards understanding sharpness\-aware minimization\.InInternational Conference on Machine Learning,pp\. 639–668\.Cited by:[§1](https://arxiv.org/html/2606.29043#S1.p1.1),[§1](https://arxiv.org/html/2606.29043#S1.p3.1)\.
- \[4\]P\. L\. Bartlett, D\. J\. Foster, and M\. Telgarsky\(2017\)Spectrally\-normalized margin bounds for neural networks\.Advances in neural information processing systems30\.Cited by:[§1](https://arxiv.org/html/2606.29043#S1.p1.1),[§1](https://arxiv.org/html/2606.29043#S1.p2.2),[§2\.2](https://arxiv.org/html/2606.29043#S2.SS2.SSS0.Px4.p1.2)\.
- \[5\]P\. L\. Bartlett, P\. M\. Long, G\. Lugosi, and A\. Tsigler\(2020\)Benign overfitting in linear regression\.Proceedings of the National Academy of Sciences117\(48\),pp\. 30063–30070\.Cited by:[§1](https://arxiv.org/html/2606.29043#S1.p1.1)\.
- \[6\]I\. Casado, L\. A\. Ortega, A\. Pérez, and A\. R\. Masegosa\(2024\)PAC\-bayes\-chernoff bounds for unbounded losses\.Advances in Neural Information Processing Systems37\.Cited by:[§4\.1\.1](https://arxiv.org/html/2606.29043#S4.SS1.SSS1.p4.1)\.
- \[7\]O\. Catoni\(2004\)Statistical learning theory and stochastic optimization: ecole d’ete de probabilites de saint\-flour xxxi–2001\.Vol\.1851,Springer\.Cited by:[§4\.1\.1](https://arxiv.org/html/2606.29043#S4.SS1.SSS1.p4.1)\.
- \[8\]O\. Catoni\(2007\)PAC\-bayesian supervised classification: the thermodynamics of statistical learning\.arXiv preprint arXiv:0712\.0248\.Cited by:[§4\.1\.1](https://arxiv.org/html/2606.29043#S4.SS1.SSS1.p4.1)\.
- \[9\]L\. Dinh, R\. Pascanu, S\. Bengio, and Y\. Bengio\(2017\)Sharp minima can generalize for deep nets\.InInternational Conference on Machine Learning,pp\. 1019–1028\.Cited by:[§1](https://arxiv.org/html/2606.29043#S1.p1.1),[§1](https://arxiv.org/html/2606.29043#S1.p2.2)\.
- \[10\]G\. K\. Dziugaite and D\. M\. Roy\(2017\)Computing nonvacuous generalization bounds for deep \(stochastic\) neural networks with many more parameters than training data\.InProceedings of the Thirty\-Third Conference on Uncertainty in Artificial Intelligence,Cited by:[§1](https://arxiv.org/html/2606.29043#S1.p1.1)\.
- \[11\]P\. Foret, A\. Kleiner, H\. Mobahi, and B\. Neyshabur\(2020\)Sharpness\-aware minimization for efficiently improving generalization\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2606.29043#S1.p1.1),[§1](https://arxiv.org/html/2606.29043#S1.p2.2),[§1](https://arxiv.org/html/2606.29043#S1.p3.1),[§2\.2](https://arxiv.org/html/2606.29043#S2.SS2.SSS0.Px2.p1.1)\.
- \[12\]B\. Guedj\(2019\)A primer on pac\-bayesian learning\.arXiv preprint arXiv:1901\.05353\.Cited by:[§1](https://arxiv.org/html/2606.29043#S1.p2.2)\.
- \[13\]S\. Gunasekar, J\. D\. Lee, D\. Soudry, and N\. Srebro\(2018\)Implicit bias of gradient descent on linear convolutional networks\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2606.29043#S1.p1.1)\.
- \[14\]M\. Haddouche, B\. Guedj, O\. Rivasplata, and J\. Shawe\-Taylor\(2021\)PAC\-bayes unleashed: generalisation bounds with unbounded losses\.Entropy23\(10\),pp\. 1330\.Cited by:[§4\.1\.1](https://arxiv.org/html/2606.29043#S4.SS1.SSS1.p4.1)\.
- \[15\]S\. Hochreiter and J\. Schmidhuber\(1997\)Flat minima\.InNeural Computation,Vol\.9,pp\. 1–42\.Cited by:[§1](https://arxiv.org/html/2606.29043#S1.p1.1)\.
- \[16\]Y\. Jiang, B\. Neyshabur, H\. Mobahi, D\. Krishnan, and S\. Bengio\(2019\)Fantastic generalization measures and where to find them\.InInternational Conference on Learning Representations,Cited by:[§1\.1](https://arxiv.org/html/2606.29043#S1.SS1.p3.1),[§1\.1](https://arxiv.org/html/2606.29043#S1.SS1.p5.1),[§1](https://arxiv.org/html/2606.29043#S1.p1.1),[§1](https://arxiv.org/html/2606.29043#S1.p4.1),[§3\.1](https://arxiv.org/html/2606.29043#S3.SS1.p4.1),[§5\.2\.1](https://arxiv.org/html/2606.29043#S5.SS2.SSS1.p1.2),[Table 6](https://arxiv.org/html/2606.29043#S5.T6),[Table 6](https://arxiv.org/html/2606.29043#S5.T6.64.1)\.
- \[17\]S\. Kaur, J\. M\. Cohen, and Z\. C\. Lipton\(2023\)On the maximum hessian eigenvalue and generalization\.InProceedings of the Thirty\-Ninth Conference on Uncertainty in Artificial Intelligence,pp\. 1048–1059\.Cited by:[§1](https://arxiv.org/html/2606.29043#S1.p2.2)\.
- \[18\]N\. S\. Keskar, D\. Mudigere, J\. Nocedal, M\. Smelyanskiy, and P\. T\. P\. Tang\(2017\)On large\-batch training for deep learning: generalization gap and sharp minima\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2606.29043#S1.p1.1)\.
- \[19\]A\. Krogh and J\. Hertz\(1991\)A simple weight decay can improve generalization\.InAdvances in Neural Information Processing Systems,Vol\.4\.Cited by:[§1](https://arxiv.org/html/2606.29043#S1.p3.1),[§4\.1\.2](https://arxiv.org/html/2606.29043#S4.SS1.SSS2.p4.6)\.
- \[20\]J\. Kwon, J\. Kim, H\. Park, and I\. K\. Choi\(2021\)ASAM: adaptive sharpness\-aware minimization for scale\-invariant learning of deep neural networks\.InInternational Conference on Machine Learning,pp\. 5905–5914\.Cited by:[§1](https://arxiv.org/html/2606.29043#S1.p3.1),[§2\.2](https://arxiv.org/html/2606.29043#S2.SS2.SSS0.Px2.p2.3)\.
- \[21\]Y\. Liu, S\. Yu, and T\. Lin\(2022\)Regularizing deep neural networks with stochastic estimators of hessian trace\.arXiv preprint arXiv:2208\.05924\.Cited by:[§1](https://arxiv.org/html/2606.29043#S1.p2.2)\.
- \[22\]I\. Loshchilov and F\. Hutter\(2019\)Decoupled weight decay regularization\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2606.29043#S1.p3.1)\.
- \[23\]K\. Lyu and J\. Li\(2020\)Gradient descent maximizes the margin of homogeneous neural networks\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2606.29043#S1.p1.1)\.
- \[24\]A\. Maurer\(2004\)A note on the PAC bayesian theorem\.External Links:cs/0411099Cited by:[§4\.1\.1](https://arxiv.org/html/2606.29043#S4.SS1.SSS1.p4.1)\.
- \[25\]D\. A\. McAllester\(1998\)Some pac\-bayesian theorems\.InProceedings of the eleventh annual conference on Computational learning theory,pp\. 230–234\.Cited by:[§1](https://arxiv.org/html/2606.29043#S1.p1.1),[§1](https://arxiv.org/html/2606.29043#S1.p2.2),[§4\.1\.1](https://arxiv.org/html/2606.29043#S4.SS1.SSS1.p4.1)\.
- \[26\]D\. A\. McAllester\(1999\)PAC\-bayesian model averaging\.InProceedings of the twelfth annual conference on Computational learning theory,pp\. 164–170\.Cited by:[§1](https://arxiv.org/html/2606.29043#S1.p1.1),[§1](https://arxiv.org/html/2606.29043#S1.p2.2),[§4\.1\.1](https://arxiv.org/html/2606.29043#S4.SS1.SSS1.p4.1)\.
- \[27\]D\. A\. McAllester\(2003\)Simplified pac\-bayesian margin bounds\.InLearning Theory and Kernel Machines: 16th Annual Conference on Learning Theory and 7th Kernel Workshop, COLT/Kernel 2003, Washington, DC, USA, August 24\-27, 2003\. Proceedings,pp\. 203–215\.Cited by:[§1](https://arxiv.org/html/2606.29043#S1.p1.1)\.
- \[28\]P\. Nakkiran, G\. Kaplun, Y\. Bansal, T\. Yang, B\. Barak, and I\. Sutskever\(2020\)Deep double descent: where bigger models and more data hurt\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2606.29043#S1.p1.1)\.
- \[29\]B\. Neyshabur, S\. Bhojanapalli, D\. McAllester, and N\. Srebro\(2018\)A pac\-bayesian approach to spectrally\-normalized margin bounds for neural networks\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2606.29043#S1.p1.1),[§1](https://arxiv.org/html/2606.29043#S1.p2.2)\.
- \[30\]B\. Neyshabur, S\. Bhojanapalli, D\. McAllester, and N\. Srebro\(2017\)Exploring generalization in deep learning\.Advances in neural information processing systems30\.Cited by:[§1\.1](https://arxiv.org/html/2606.29043#S1.SS1.p2.1),[§1](https://arxiv.org/html/2606.29043#S1.p1.1)\.
- \[31\]B\. Neyshabur, R\. Tomioka, and N\. Srebro\(2014\)In search of the real inductive bias: on the role of implicit regularization in deep learning\.arXiv preprint arXiv:1412\.6614\.Cited by:[§1](https://arxiv.org/html/2606.29043#S1.p1.1)\.
- \[32\]B\. Neyshabur, R\. Tomioka, and N\. Srebro\(2015\)Norm\-based capacity control in neural networks\.InConference on Learning Theory,pp\. 1376–1401\.Cited by:[§1](https://arxiv.org/html/2606.29043#S1.p1.1),[§1](https://arxiv.org/html/2606.29043#S1.p2.2),[§2\.2](https://arxiv.org/html/2606.29043#S2.SS2.SSS0.Px3.p1.1)\.
- \[33\]B\. Recht, R\. Roelofs, L\. Schmidt, and V\. Shankar\(2019\)Do imagenet classifiers generalize to imagenet?\.InProceedings of the 36th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.97,pp\. 5389–5400\.Cited by:[§1](https://arxiv.org/html/2606.29043#S1.p1.1)\.
- \[34\]B\. Rodríguez\-Gálvez, R\. Thobaben, and M\. Skoglund\(2024\)More pac\-bayes bounds: from bounded losses, to losses with general tail behaviors, to anytime validity\.Journal of Machine Learning Research25\(192\),pp\. 1–78\.Cited by:[§4\.1\.1](https://arxiv.org/html/2606.29043#S4.SS1.SSS1.p4.1)\.
- \[35\]M\. Seeger\(2002\)PAC\-bayesian generalisation error bounds for gaussian process classification\.InJournal of Machine Learning Research,pp\. 233–269\.Cited by:[§4\.1\.1](https://arxiv.org/html/2606.29043#S4.SS1.SSS1.p4.1)\.
- \[36\]S\. L\. Smith, B\. Dherin, D\. G\. T\. Barrett, and S\. De\(2021\)On the origin of implicit regularization in stochastic gradient descent\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2606.29043#S1.p1.1)\.
- \[37\]D\. Soudry, E\. Hoffer, M\. S\. Nacson, S\. Gunasekar, and N\. Srebro\(2018\)The implicit bias of gradient descent on separable data\.Journal of Machine Learning Research19\(70\),pp\. 1–57\.Cited by:[§1](https://arxiv.org/html/2606.29043#S1.p1.1)\.
- \[38\]C\. Zhang, S\. Bengio, M\. Hardt, B\. Recht, and O\. Vinyals\(2017\)Understanding deep learning requires rethinking generalization\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2606.29043#S1.p1.1)\.
- \[39\]G\. Zhang, C\. Wang, B\. Xu, and R\. Grosse\(2019\)Three mechanisms of weight decay regularization\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2606.29043#S1.p3.1)\.
- \[40\]X\. Zhang, A\. Ghosh, G\. Liu, and R\. Wang\(2024\)Improving generalization of complex models under unbounded loss using PAC\-Bayes bounds\.Transactions on Machine Learning Research\.Cited by:[§4\.1\.1](https://arxiv.org/html/2606.29043#S4.SS1.SSS1.p4.1)\.

## Appendix AExperimental Protocol

We report several implementation choices that are important for conducting and interpreting the linear regression and Pareto analysis\.

1. 1\.Broad hyperparameter sweeps\.For each network architecture, we perform a broad hyperparameter sweep and analyze the resulting pool of valid trained models, rather than selecting a single tuned configuration\. We vary four training hyperparameters: batch size, learning rate, weight decay, and momentum\. The specific hyperparameter ranges may differ across architectures to account for their training stability and optimization behavior\. However, for each architecture, we*retain at least 40 valid trained models*\. This ensures that both regression and Pareto analysis are conducted on a sufficiently diverse collection of attainable solutions, rather than on a small or cherry\-picked set of runs\.
2. 2\.Filtering under\-trained models\.To avoid confounding effects from under\-trained models, we retain only runs that achieve nearly zero training loss, using the thresholdtraining loss≤0\.01\\text\{training loss\}\\\!\\leq\\\!0\.01\. Runs that do not meet this threshold are discarded\. This filtering step is important because it ensures that the measured generalization gap primarily reflects differences in generalization behavior, rather than artifacts of incomplete training or optimization failure\.
3. 3\.Fixed reference data for metric computation\.All evaluated metrics are computed on fixed reference batches for each dataset\. The same reference data are used across models within the same comparison group\. This reduces data\-sampling noise and ensures that differences in metric values are attributable to the trained models rather than to different evaluation batches\.
4. 4\.Generalization gap as the consistent target\.Throughout the regression and Pareto analyses, the generalization gap is computed as test loss minus training loss\. Using a consistent gap definition across all models ensures that the target variable is comparable across architectures and metric pairs\.
5. 5\.Cross\-architecture and cross\-data mixtures\.For the mixed\-architecture settings \(lower block in Table[4](https://arxiv.org/html/2606.29043#S5.T4)\), we pool the valid trained models from multiple different architectures and treat the pooled set as one collection for regression and Pareto analyses\. This serves as a stress\-test, checking whether the evaluated metrics remain comparable across architectures and datasets\.

## Appendix BOrigin of the Factor Sharpness

Let the posterior centered at the trained model weightsθ^\\hat\{\\theta\}with covarianceΣθ\\Sigma\_\{\\theta\}, we can the write the expected posterior empirical loss as:

L^​\(𝒬θ\)=𝔼θ∼𝒬θ​\[ℓ​\(fθ;Z\)\]=θ=θ^\+Δ​θ𝔼Δ​θ∼𝒬~θ​\[ℓ​\(fθ^\+Δ​θ;Z\)\],\\displaystyle\\hat\{L\}\(\\mathcal\{Q\}\_\{\\theta\}\)=\\mathbb\{E\}\_\{\\theta\\sim\\mathcal\{Q\}\_\{\\theta\}\}\\big\[\\ell\(f\_\{\\theta\};Z\)\\big\]\\stackrel\{\{\\scriptstyle\\theta=\\hat\{\\theta\}\+\\Delta\\theta\}\}\{\{=\}\}\\mathbb\{E\}\_\{\\Delta\\theta\\sim\\widetilde\{\\mathcal\{Q\}\}\_\{\\theta\}\}\\big\[\\ell\(f\_\{\\hat\{\\theta\}\+\\Delta\\theta\};Z\)\\big\],where𝒬~θ\\widetilde\{\\mathcal\{Q\}\}\_\{\\theta\}is the distribution ofΔ​θ=θ−θ^\\Delta\\theta=\\theta\-\\hat\{\\theta\}, satisfying

𝔼𝒬~θ​\[Δ​θ\]=0,𝔼𝒬~θ​\[Δ​θ​Δ​θ⊤\]=Σθ\.\\mathbb\{E\}\_\{\\widetilde\{\\mathcal\{Q\}\}\_\{\\theta\}\}\[\\Delta\\theta\]=0,\\qquad\\mathbb\{E\}\_\{\\widetilde\{\\mathcal\{Q\}\}\_\{\\theta\}\}\[\\Delta\\theta\\Delta\\theta^\{\\top\}\]=\\Sigma\_\{\\theta\}\.Applying a second\-order Taylor expansion aroundθ^\\hat\{\\theta\}, we obtain

L^​\(𝒬θ\)\\displaystyle\\hat\{L\}\(\\mathcal\{Q\}\_\{\\theta\}\)=𝔼Δ​θ∼𝒬~θ​\[ℓ​\(fθ^\+Δ​θ;Z\)\]\\displaystyle=\\mathbb\{E\}\_\{\\Delta\\theta\\sim\\widetilde\{\\mathcal\{Q\}\}\_\{\\theta\}\}\\big\[\\ell\(f\_\{\\hat\{\\theta\}\+\\Delta\\theta\};Z\)\\big\]=𝔼Δ​θ∼𝒬~θ​\[ℓ​\(fθ^;Z\)\+Δ​θ⊤​∇θℓ​\(fθ^;Z\)\+12​Δ​θ⊤​∇θ2ℓ​\(fθ^;Z\)​Δ​θ\+o​\(‖Δ​θ‖2\)\]\\displaystyle=\\mathbb\{E\}\_\{\\Delta\\theta\\sim\\widetilde\{\\mathcal\{Q\}\}\_\{\\theta\}\}\\Big\[\\ell\(f\_\{\\hat\{\\theta\}\};Z\)\+\\Delta\\theta^\{\\top\}\\nabla\_\{\\theta\}\\ell\(f\_\{\\hat\{\\theta\}\};Z\)\+\\tfrac\{1\}\{2\}\\Delta\\theta^\{\\top\}\\nabla\_\{\\theta\}^\{2\}\\ell\(f\_\{\\hat\{\\theta\}\};Z\)\\Delta\\theta\+o\(\\\|\\Delta\\theta\\\|^\{2\}\)\\Big\]≈ℓ​\(fθ^;Z\)\+12​Tr⁡\(∇θ2ℓ​\(fθ^;Z\)​𝔼Δ​θ∼𝒬~θ​\[Δ​θ​Δ​θ⊤\]\)\\displaystyle\\approx\\ell\(f\_\{\\hat\{\\theta\}\};Z\)\+\\tfrac\{1\}\{2\}\\operatorname\{Tr\}\\left\(\\nabla\_\{\\theta\}^\{2\}\\ell\(f\_\{\\hat\{\\theta\}\};Z\)\\,\\mathbb\{E\}\_\{\\Delta\\theta\\sim\\widetilde\{\\mathcal\{Q\}\}\_\{\\theta\}\}\[\\Delta\\theta\\Delta\\theta^\{\\top\}\]\\right\)=ℓ​\(fθ^;Z\)⏟empirical loss\+12​Tr⁡\(∇θ2ℓ​\(fθ^;Z\)​Σθ\)\.\\displaystyle=\\underbrace\{\\ell\(f\_\{\\hat\{\\theta\}\};Z\)\}\_\{\\text\{empirical loss\}\}\+\\tfrac\{1\}\{2\}\\operatorname\{Tr\}\\Big\(\\nabla\_\{\\theta\}^\{2\}\\ell\(f\_\{\\hat\{\\theta\}\};Z\)\\,\\Sigma\_\{\\theta\}\\Big\)\.

## Appendix CMore on Bayes Sharpness

Bayes sharpness \(bayesS\) \([15](https://arxiv.org/html/2606.29043#S4.E15)\) is defined as the normalized posterior expected increase in empirical loss:

SbayesS:=𝔼θ∼𝒬θ​\[ℓ​\(fθ;Z\)\]−ℓ​\(fθ^;Z\)σ2,S\_\{\\mathrm\{bayesS\}\}:=\\frac\{\\mathbb\{E\}\_\{\\theta\\sim\\mathcal\{Q\}\_\{\\theta\}\}\\big\[\\ell\(f\_\{\\theta\};Z\)\\big\]\-\\ell\(f\_\{\\hat\{\\theta\}\};Z\)\}\{\\sigma^\{2\}\},\(20\)whereσ\\sigmacontrols the magnitude of the posterior covarianceΣθ\\Sigma\_\{\\theta\}, i\.e\., the perturbation scale\. Bayes sharpness admits the approximation:

SbayesS≈12​σ2​Tr⁡\(Hθ^​Σθ\)=12​Tr⁡\(Hθ^​diag⁡\(θ^2\)\),S\_\{\\mathrm\{bayesS\}\}\\approx\\frac\{1\}\{2\\sigma^\{2\}\}\\operatorname\{Tr\}\\\!\\big\(H\_\{\\hat\{\\theta\}\}\\Sigma\_\{\\theta\}\\big\)=\\frac\{1\}\{2\}\\operatorname\{Tr\}\\\!\\big\(H\_\{\\hat\{\\theta\}\}\\operatorname\{diag\}\(\\hat\{\\theta\}^\{2\}\)\\big\),\(21\)whereHθ^=∇θ2ℓ​\(fθ^;Z\)\.H\_\{\\hat\{\\theta\}\}=\\nabla\_\{\\theta\}^\{2\}\\ell\(f\_\{\\hat\{\\theta\}\};Z\)\.

### C\.1Interpretation

BayesS differs from Hessian trace and adapS in the following ways\.

- •The factordiag⁡\(θ^2\)\\operatorname\{diag\}\(\\hat\{\\theta\}^\{2\}\)makes the perturbation scale adaptive to the magnitude of each parameter\. This is the source of the scale\-aware behavior of bayesS\. Under simple rescalings, the adaptive covariance rescales together with the parameters, making the resulting loss\-increase quantity more stable than the ordinary Hessian trace\.
- •AdapS \([2](https://arxiv.org/html/2606.29043#S2.E2)\) captures a worst\-case loss increase over an adaptive neighborhood, whereas bayesS \([20](https://arxiv.org/html/2606.29043#A3.E20)\) measures an*expected*loss increase under an adaptive posterior\. AdapS is closer to a worst\-direction sharpness measure, while bayesS is more aligned with the posterior expected empirical loss term in PAC\-Bayes bounds\.
- •The worst\-case perturbation in adapS is governed by the sharpest direction of the adaptive neighborhood\. In this sense, adapS can be viewed as an adaptive analogue of a top\-eigenvalue sharpness metric\. In contrast, bayesS*averages the loss increase*under posterior perturbations\. With the adaptive posterior, bayesS capturesTr⁡\(Hθ^​diag⁡\(θ^2\)\)\.\\operatorname\{Tr\}\\big\(H\_\{\\hat\{\\theta\}\}\\operatorname\{diag\}\(\\hat\{\\theta\}^\{2\}\)\\big\)\.Hence, bayesS can be viewed as an*adaptive trace\-type*sharpness metric\. Instead of gonerned by the single sharpest direction, it aggregates curvature over posterior\-relevant directions, making it less sensitive to outlier curvature directions that may dominate worst\-case sharpness metrics\.

This distinction is especially important when we try to evaluate the explanatory power of\(S,C\)\(S,C\)in the cross\-architecture mixed settings \(the lower block of Table[4](https://arxiv.org/html/2606.29043#S5.T4)and Figure[6](https://arxiv.org/html/2606.29043#S5.F6)\)\. Different architectures may have very different parameter dimensions and parameter interactions, making direct comparison of worst\-direction sharpness unreliable\. By contrast, bayesS uses posterior*expected*loss increase and summarizes adaptive curvature in a trace\-like manner, providing a more distributional and PAC\-Bayes\-aligned notion of sharpness\.

### C\.2Estimation of BayesS

In practice, bayesS is estimated by Monte Carlo sampling from the adaptive posterior\. Given a trained model with weightsθ^\\hat\{\\theta\}, we first evaluate the empirical lossℓ​\(fθ^;Z\)\\ell\(f\_\{\\hat\{\\theta\}\};Z\)on the training datasetZ=\{zi\}i=1mZ=\\\{z\_\{i\}\\\}\_\{i=1\}^\{m\}

ℓ​\(fθ^;Z\)=1m​∑i=1mℓ​\(fθ^;zi\)\.\\ell\(f\_\{\\hat\{\\theta\}\};Z\)=\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}\\ell\(f\_\{\\hat\{\\theta\}\};z\_\{i\}\)\.We then draw parameter samples \(perturbations\) from the adaptive posterior

θ\(k\)∼𝒩​\(θ^,σ2​diag⁡\(θ^2\)\),k=1,…,K\.\\theta^\{\(k\)\}\\sim\\mathcal\{N\}\\big\(\\hat\{\\theta\},\\sigma^\{2\}\\operatorname\{diag\}\(\\hat\{\\theta\}^\{2\}\)\\big\),\\qquad k=1,\\dots,K\.Each sampled parameter vectorθ\(k\)\\theta^\{\(k\)\}is loaded into the network and evaluated on the same dataset\. The empirical estimator of bayesS is

S^bayesS=\(1K​∑k=1Kℓ​\(fθ\(k\);Z\)−ℓ​\(fθ^;Z\)\)/σ2\.\\widehat\{S\}\_\{\\mathrm\{bayesS\}\}=\\Big\(\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\ell\(f\_\{\\theta^\{\(k\)\}\};Z\)\-\\ell\(f\_\{\\hat\{\\theta\}\};Z\)\\Big\)/\\sigma^\{2\}\.
Estimating bayesS requires choosing a perturbation scaleσ\\sigma\. However, as explained in Section[4\.2](https://arxiv.org/html/2606.29043#S4.SS2), the normalization byσ2\\sigma^\{2\}removes the leading\-order dependence on this scale\. This can be seen from the Taylor approximation \([21](https://arxiv.org/html/2606.29043#A3.E21)\):SbayesS≈12​Tr⁡\(Hθ^​diag⁡\(θ^2\)\),S\_\{\\mathrm\{bayesS\}\}\\approx\\frac\{1\}\{2\}\\operatorname\{Tr\}\\big\(H\_\{\\hat\{\\theta\}\}\\operatorname\{diag\}\(\\hat\{\\theta\}^\{2\}\)\\big\),which no longer explicitly depends onσ\\sigma\.

## Appendix DMore on Functional KL Divergence

### D\.1Interpretation

The key distinction from parameter\-level complexity is thatKLfunc\\mathrm\{KL\}\_\{\\mathrm\{func\}\}does not compare parameter vectors directly\. Instead, it compares the function\-output distributions induced by the parameter prior and posterior on a fixed reference dataset\. As a result, it is less tied to a specific parameter representation and more directly reflects how far the learned predictor \(posterior\) is from the prior in terms of input\-output behavior\. This makesKLfunc\\mathrm\{KL\}\_\{\\mathrm\{func\}\}a more appropriate complexity metric when parameterization ambiguity is a central concern\.

### D\.2Two Ways to ImplementKLfunc\\mathrm\{KL\}\_\{\\mathrm\{func\}\}Estimation

##### Implementation A \(stochastic functional KL estimation\)\.

To directly compute \([18](https://arxiv.org/html/2606.29043#S4.E18)\)

KL^func≈12​σf2​∑j=1df\(μ^𝒬f\)j2,\\widehat\{\\mathrm\{KL\}\}\_\{\\mathrm\{func\}\}\\approx\\frac\{1\}\{2\\sigma\_\{f\}^\{2\}\}\\sum\_\{j=1\}^\{d\_\{f\}\}\\big\(\\widehat\{\\mu\}\_\{\\mathcal\{Q\}\_\{f\}\}\\big\)\_\{j\}^\{2\}\\,,we draw stochastic samples from the posterior

θ𝒬\(k\)∼𝒬θ=𝒩​\(θ^,Σθ\),k=1,…,K,\\theta\_\{\\mathcal\{Q\}\}^\{\(k\)\}\\sim\\mathcal\{Q\}\_\{\\theta\}=\\mathcal\{N\}\(\\hat\{\\theta\},\\Sigma\_\{\\theta\}\),\\qquad k=1,\\dots,K,evaluate the corresponding sampled models on the reference datasetDD, and estimate the posterior function\-output means\(μ^𝒬f\)j\(\\widehat\{\\mu\}\_\{\\mathcal\{Q\}\_\{f\}\}\)\_\{j\}\. This implementation uses Monte Carlo sampling and requires choosingσ\\sigma, which controls the magnitude of the poseterior parameter perturbations\. We use an adaptive posterior with anisotropic covarianceΣθ=σ2​diag​\(θ^2\)\\Sigma\_\{\\theta\}=\\sigma^\{2\}\\mathrm\{diag\}\(\\hat\{\\theta\}^\{2\}\)and setσ=0\.01\\sigma=0\.01in our experiments\.

##### Implementation B \(deterministic functional norm proxy\)\.

Since the parameter posterior is centered at the trained weightsθ^\\hat\{\\theta\}, the induced function\-level posterior is concentrated around the trained functionfθ^f\_\{\\hat\{\\theta\}\}\. If we ignore the stochastic posterior sampling, thenμ^𝒬f\\widehat\{\\mu\}\_\{\\mathcal\{Q\}\_\{f\}\}in Eq\. \([18](https://arxiv.org/html/2606.29043#S4.E18)\) can be approximated by

μ^𝒬f≈FD​\(θ^\)=vec⁡\(fθ^​\(D\)\),\\widehat\{\\mu\}\_\{\\mathcal\{Q\}\_\{f\}\}\\approx F\_\{D\}\(\\hat\{\\theta\}\)=\\operatorname\{vec\}\(f\_\{\\hat\{\\theta\}\}\(D\)\),and therefore the entire mean\-mismatch term can be viewed as a quantity that is proportional to the squared norm of the trained function\-outputs\. This motivates a deterministic proxy, i\.e\., Eq\. \([19](https://arxiv.org/html/2606.29043#S4.E19)\):

KL^func​=Eq\. \([18](https://arxiv.org/html/2606.29043#S4.E18)\)​12​σf2​∑j=1df\(μ^𝒬f\)j2∼function norm:=‖vec⁡\(fθ^​\(D\)\)‖2\.\\widehat\{\\mathrm\{KL\}\}\_\{\\mathrm\{func\}\}\\overset\{\\text\{Eq\.~\\eqref\{def:funcKL\_estimator\_simplified\}\}\}\{=\}\\frac\{1\}\{2\\sigma\_\{f\}^\{2\}\}\\sum\_\{j=1\}^\{d\_\{f\}\}\\big\(\\widehat\{\\mu\}\_\{\\mathcal\{Q\}\_\{f\}\}\\big\)\_\{j\}^\{2\}\\,\\sim\\,\\text\{function norm\}:=\\left\\\|\\operatorname\{vec\}\(f\_\{\\hat\{\\theta\}\}\(D\)\)\\right\\\|^\{2\}\.Unlike implementation A, this function norm proxy does not require Monte Carlo sampling or choosing the parameter perturbation scaleσ\\sigma, and is therefore deterministic and more computationally efficient\.

##### Relationship between the two implementations\.

We evaluate both implementation A and B\. When computed on the same reference dataset, they empirically produce highly consistent regression and Pareto analyses results\. We view the stochastic approximation \([18](https://arxiv.org/html/2606.29043#S4.E18)\) as the more principled implementation, while the deterministic \([19](https://arxiv.org/html/2606.29043#S4.E19)\) serves as a simplified and computationally efficient proxy\.

We report the deterministic proxy*func norm*\(implementation B\) in our main results in Table[4](https://arxiv.org/html/2606.29043#S5.T4), and defer the stochastic approximation*func KL*\(implementation A\) to the ablation study in Appendix[E\.1](https://arxiv.org/html/2606.29043#A5.SS1)\.

## Appendix EAblation Study

### E\.1Two Implementations ofKLfunc\\mathrm\{KL\}\_\{\\mathrm\{func\}\}Estimation

##### Ablation study 1\.

Table[7](https://arxiv.org/html/2606.29043#A5.T7)presents an ablation study on the implementation of the proposed function\-level complexity metric, and its combination with different sharpness metrics\.

The results first show that the stochastic approximation \(implementation A,*func KL*\) and the deterministic proxy \(implementation B,*func norm*\) lead to highly consistent performance\. This supports the use of the deterministic proxy in the main experiments as a computationally simpler realization of the functional complexity\. Second, when complexity is measured at the function level, it pairs well with both adapS and bayesS\. This indicates that a substantial part of the improved explanatory power comes from moving the complexity metric from parameter\-level to function\-level\. Comparing the two sharpness metrics,*bayesS*is more favorable than adapS overall; see reasons in Appendix[C\.1](https://arxiv.org/html/2606.29043#A3.SS1)\. Therefore, we use\(bayesS, func norm\)as our main proposed function\-oriented metric pair\.

A visualization of the PCR column of Table[7](https://arxiv.org/html/2606.29043#A5.T7)is provided in Figure[7](https://arxiv.org/html/2606.29043#A5.F7)\.

### E\.2Posterior Covariance Choice: Anisotropic v\.s\. Isotropic

##### Ablation study 2\.

Table[8](https://arxiv.org/html/2606.29043#A5.T8)studies the effect of the anisotropic/isotropic posterior choice\. We compare an isotropic posteriorΣθ=σ2​I\\Sigma\_\{\\theta\}=\\sigma^\{2\}I, with the adaptive anisotropic posteriorΣθ=σ2​diag⁡\(θ^2\)\\Sigma\_\{\\theta\}=\\sigma^\{2\}\\operatorname\{diag\}\(\\hat\{\\theta\}^\{2\}\)\. The results show that the anisotropic posterior generally outperforms\. In several settings, the isotropic version yields negative coefficients or noticeably higher PCR, indicating less consistent agreement with the expected monotonic relationship between\(S,C\)\(S,C\)and generalization\. By contrast, the anisotropic posterior often produces positive coefficients, good regression predictability, and low PCR\. This supports the use of*adaptive posterior*, which provides a more stable and scale\-aware realization of both Bayes sharpness and functional KL divergence\.

A visualization of the PCR column of Table[8](https://arxiv.org/html/2606.29043#A5.T8)is provided in Figure[7](https://arxiv.org/html/2606.29043#A5.F7)\.

DatasetNetworkArchitectureEvaluated\(𝐒,𝐂\)\\mathbf\{\(S,C\)\}MetricsRegressionR2↑R^\{2\}\\uparrow, coefficientPCR↓\\downarrowVisualized in Fig\.[7](https://arxiv.org/html/2606.29043#A5.F7)\.CIFAR\-10ResNet18\(adapS, func KL\)0\.73,\(\+,\+\)\(\+,\+\)1\.2%\(adapS, func norm\)0\.73,\(\+,\+\)\(\+,\+\)1\.2%\(bayesS, func KL\)0\.68,\(\+,\+\)\(\+,\+\)2\.8%\(bayesS, func norm\)0\.68,\(\+,\+\)\(\+,\+\)2\.8%VGG13\(adapS, func KL\)0\.58,\(\+,\+\)\(\+,\+\)4\.9%\(adapS, func norm\)0\.58,\(\+,\+\)\(\+,\+\)4\.9%\(bayesS, func KL\)0\.72,\(\+,\+\)\(\+,\+\)3\.5%\(bayesS, func norm\)0\.72,\(\+,\+\)\(\+,\+\)3\.5%CIFAR\-100WideResNet\(adapS, func KL\)0\.83,\(\+,\+\)\(\+,\+\)1\.5%\(adapS, func norm\)0\.83,\(\+,\+\)\(\+,\+\)1\.5%\(bayesS, func KL\)0\.91,\(\+,\+\)\(\+,\+\)1\.0%\(bayesS, func norm\)0\.91,\(\+,\+\)\(\+,\+\)1\.0%ViT\(adapS, func KL\)0\.91,\(−,\+\)\(\-,\+\)18\.9%\(adapS, func norm\)0\.91,\(−,\+\)\(\-,\+\)18\.4%\(bayesS, func KL\)0\.88,\(−,\+\)\(\-,\+\)21\.2%\(bayesS, func norm\)0\.88,\(−,\+\)\(\-,\+\)20\.8%CIFAR\-10no\-BN ResNet18\(adapS, func KL\)0\.80,\(\+,\+\)\(\+,\+\)0\.5%\(adapS, func norm\)0\.80,\(\+,\+\)\(\+,\+\)0\.5%\(bayesS, func KL\)0\.83,\(\+,\+\)\(\+,\+\)1\.2%\(bayesS, func norm\)0\.83,\(\+,\+\)\(\+,\+\)1\.2%no\-BN VGG13\(adapS, func KL\)0\.70,\(\+,\+\)\(\+,\+\)17\.1%\(adapS, func norm\)0\.70,\(\+,\+\)\(\+,\+\)17\.1%\(bayesS, func KL\)0\.69,\(\+,\+\)\(\+,\+\)11\.4%\(bayesS, func norm\)0\.69,\(\+,\+\)\(\+,\+\)11\.4%CIFAR\-100no\-BN WideResNet\(adapS, func KL\)0\.71,\(\+,\+\)\(\+,\+\)20\.9%\(adapS, func norm\)0\.71,\(\+,\+\)\(\+,\+\)20\.7%\(bayesS, func KL\)0\.71,\(\+,\+\)\(\+,\+\)17\.9%\(bayesS, func norm\)0\.71,\(\+,\+\)\(\+,\+\)17\.9%no\-LN ViT\(adapS, func KL\)0\.86,\(\+,\+\)\(\+,\+\)12\.5%\(adapS, func norm\)0\.86,\(\+,\+\)\(\+,\+\)12\.6%\(bayesS, func KL\)0\.87,\(\+,\+\)\(\+,\+\)13\.6%\(bayesS, func norm\)0\.87,\(\+,\+\)\(\+,\+\)13\.6%

Table 7:Ablation study 1\.Regression and Pareto analysis comparing: \(i\) two implementations of the proposed functional complexity metric,func KL\(implementation A, stochastic approximation \([18](https://arxiv.org/html/2606.29043#S4.E18)\)\) andfunc norm\(implementation B, deterministic proxy \([19](https://arxiv.org/html/2606.29043#S4.E19)\)\) ; \(ii\) two sharpness metrics,adapSandbayesS\. For posterior\-involved metrics, bayesS and func KL, here we use the adaptive anisotropic posterior\. The results support the use of\(bayesS, func norm\)as our main proposed function\-oriented metric pair reported in Table[4](https://arxiv.org/html/2606.29043#S5.T4)\. A partial visualization of the PCR column is provided in Figure[7](https://arxiv.org/html/2606.29043#A5.F7)\.DatasetNetworkArchitecture\(𝐒,𝐂\)\\mathbf\{\(S,C\)\}Metric PairRegressionR2↑R^\{2\}\\uparrow, coefficientPCR↓\\downarrowVisualized in Fig\.[7](https://arxiv.org/html/2606.29043#A5.F7)\.CIFAR\-10ResNet18\(bayesS iso, func KL iso\)0\.23,\(\+,−\)\(\+,\-\)23\.6%\(bayesS iso, func norm\)0\.44,\(\+,\+\)\(\+,\+\)1\.1%\(bayesS, func KL\)0\.68,\(\+,\+\)\(\+,\+\)2\.8%\(bayesS, func norm\)0\.68,\(\+,\+\)\(\+,\+\)2\.8%VGG13\(bayesS iso, func KL iso\)0\.02,\(−,\+\)\(\-,\+\)27\.9%\(bayesS iso, func norm\)0\.24,\(−,\+\)\(\-,\+\)2\.8%\(bayesS, func KL\)0\.72,\(\+,\+\)\(\+,\+\)3\.5%\(bayesS, func norm\)0\.72,\(\+,\+\)\(\+,\+\)3\.5%CIFAR\-100WideResNet\(bayesS iso, func KL iso\)0\.88,\(\+,−\)\(\+,\-\)14\.0%\(bayesS iso, func norm\)0\.83,\(\+,\+\)\(\+,\+\)4\.3%\(bayesS, func KL\)0\.91,\(\+,\+\)\(\+,\+\)1\.0%\(bayesS, func norm\)0\.91,\(\+,\+\)\(\+,\+\)1\.0%ViT\(bayesS iso, func KL iso\)0\.90,\(−,\+\)\(\-,\+\)11\.0%\(bayesS iso, func norm\)0\.90,\(−,\+\)\(\-,\+\)20\.0%\(bayesS, func KL\)0\.88,\(−,\+\)\(\-,\+\)21\.2%\(bayesS, func norm\)0\.88,\(−,\+\)\(\-,\+\)20\.8%CIFAR\-10no\-BN ResNet18\(bayesS iso, func KL iso\)0\.82,\(−,\+\)\(\-,\+\)2\.0%\(bayesS iso, func norm\)0\.85,\(\+,\+\)\(\+,\+\)0\.4%\(bayesS, func KL\)0\.83,\(\+,\+\)\(\+,\+\)1\.2%\(bayesS, func norm\)0\.83,\(\+,\+\)\(\+,\+\)1\.2%no\-BN VGG13\(bayesS iso, func KL iso\)0\.70,\(\+,\+\)\(\+,\+\)0\.3%\(bayesS iso, func norm\)0\.72,\(\+,\+\)\(\+,\+\)0\.5%\(bayesS, func KL\)0\.69,\(\+,\+\)\(\+,\+\)11\.4%\(bayesS, func norm\)0\.69,\(\+,\+\)\(\+,\+\)11\.4%CIFAR\-100no\-BN WideResNet\(bayesS iso, func KL iso\)0\.30,\(−,\+\)\(\-,\+\)56\.6%\(bayesS iso, func norm\)0\.71,\(\+,\+\)\(\+,\+\)23\.5%\(bayesS, func KL\)0\.71,\(\+,\+\)\(\+,\+\)17\.9%\(bayesS, func norm\)0\.71,\(\+,\+\)\(\+,\+\)17\.9%no\-LN ViT\(bayesS iso, func KL iso\)0\.94,\(−,\+\)\(\-,\+\)20\.4%\(bayesS iso, func norm\)0\.86,\(−,\+\)\(\-,\+\)13\.0%\(bayesS, func KL\)0\.87,\(\+,\+\)\(\+,\+\)13\.6%\(bayesS, func norm\)0\.87,\(\+,\+\)\(\+,\+\)13\.6%

Table 8:Ablation study 2\.Regression and Pareto analysis comparing the posterior covariance choices,isotropicv\.s\.anisotropic, for the proposed function\-level metrics\. The isotropic posterior uses covarianceΣθ=σ2​I\\Sigma\_\{\\theta\}=\\sigma^\{2\}I, while the anisotropic posterior uses covarianceΣθ=σ2​diag⁡\(θ^2\)\\Sigma\_\{\\theta\}=\\sigma^\{2\}\\operatorname\{diag\}\(\\hat\{\\theta\}^\{2\}\)\. Metrics with the suffix ‘iso’ are computed using the isotropic posterior, whereas metrics without it are computed using the adaptive anisotropic posterior\. This covariance choice affects posterior\-related metrics: the bayesS and \(stochastic\) func KL\. The results support the use of*adaptive anisotropic posterior*in the proposed function\-oriented metrics\. A partial visualization of the PCR column is provided in Figure[7](https://arxiv.org/html/2606.29043#A5.F7)\.![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c10_resnet18_adapS_funckl.png)\(a\)\(adapS, func KL\)\.PCR=1\.2%\\text\{PCR\}\\\!=1\.2\\%\\\!\.
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c10_resnet18_bayeS_funckl_iso.png)\(b\)\(bayeS iso, func KL iso\)\.PCR=23\.6%\\text\{PCR\}\\\!=23\.6\\%\\\!\.
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c10_resnet18_bayeS_funckl.png)\(c\)\(bayeS, func KL\)\.PCR=2\.8%\\text\{PCR\}\\\!=2\.8\\%\\\!\.
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c10_resnet18_adapS_funcnorm.png)\(d\)\(adapS, func norm\)\.PCR=1\.2%\\text\{PCR\}\\\!=1\.2\\%\\\!\.
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c10_resnet18_bayeS_funcnorm_iso.png)\(e\)\(bayesS iso, func norm\)\.PCR=1\.1%\\text\{PCR\}\\\!=1\.1\\%\\\!\.
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c10_resnet18_bayeS_funcnorm.png)\(f\)*\(bayesS, func norm\)\.*PCR=2\.8%\\text\{PCR\}\\\!=2\.8\\%\\\!\.
This is the one used in the main results\.

Figure 7:Pareto analysiscomparing: \(i\) two implementations of the proposed functional complexity metric,func KL\(implementation A, stochastic approximation \([18](https://arxiv.org/html/2606.29043#S4.E18)\)\) andfunc norm\(implementation B, deterministic proxy \([19](https://arxiv.org/html/2606.29043#S4.E19)\)\); \(ii\) the use ofisotropicv\.s\.anisotropicposterior; \(iii\) two sharpness metrics,adapSandbayesS\. For posterior\-related metrics, the bayesS and func KL, with suffix ‘iso’ is computed using isotropic covariance, whereas without it is using adaptive anisotropic covariance\. These plots visualize the PCR column of CIFAR\-10 ResNet18 in Table[7](https://arxiv.org/html/2606.29043#A5.T7)and[8](https://arxiv.org/html/2606.29043#A5.T8)\.

### E\.3Diagnosing the Failure on ViT

##### Ablation study 3\.

In our main experimental results in Table[4](https://arxiv.org/html/2606.29043#S5.T4), the proposed function\-oriented metric pair*\(bayesS, func norm\)*fails on CIFAR\-100 ViT, despite performing well in several normalization\-disabled and mixed\-architecture settings\. To better understand this failure, we examine the second conjecture provided in Section[5\.2](https://arxiv.org/html/2606.29043#S5.SS2.SSS0.Px1)\.

Recall that all models are filtered to have nearly zero training loss, so a large generalization gap directly corresponds to a large test loss\. We observe that in the ViT setting without data augmentation \(Figure[5\(f\)](https://arxiv.org/html/2606.29043#S5.F5.sf6)\), many trained models have test losses even larger than the trivial loss of a uniform predictor,ℓ=ln⁡100≈4\.6\\ell=\\ln 100\\approx 4\.6, suggesting severe overfitting rather than meaningful generalization behavior\. Motivated by this observation, we conduct a simple ablation study comparing CIFAR\-100 ViT models*trained with and without data augmentation*\. The training setup, filtering criterion, and evaluated metric pair are kept the same; the only intended change is whether data augmentation is enabled during training\. The goal is not to fully resolve the ViT failure, but to test whether the failure is partly driven by the extreme overfitting behavior of ViT models trained without augmentation\.

Table[9](https://arxiv.org/html/2606.29043#A5.T9)first reports the training and test loss statistics for ViT models trained with and without data augmentation\. The comparison shows that data augmentation mitigates, but does not fully eliminate, the extremely large test losses observed in the no\-augmentation setting\.

Table[10](https://arxiv.org/html/2606.29043#A5.T10)presents the corresponding regression and Pareto analysis results for the proposed function\-oriented metric pair*\(bayesS, func norm\)*\. Figure[8](https://arxiv.org/html/2606.29043#A5.F8)provides the PCR visualizations for the two settings\.

Dataset, NetworkData Aug\.LossminmeanmaxCIFAR\-100, ViTFalsetrain loss0\.00030\.00150\.0091test loss2\.76004\.78879\.0100Truetrain loss0\.00050\.00110\.0030test loss2\.8900*3\.7067**7\.2000*Table 9:Loss statisticsfor CIFAR\-100 ViT models with and without data augmentation\. All reported models satisfy the filtering criterion of nearly zero training loss \(≤0\.01\\leq 0\.01\)\. Without data augmentation, the mean and maximum test losses are4\.784\.78and9\.019\.01, respectively, exceeding the trivial loss of a uniform predictor,ln⁡100≈4\.6\\ln 100\\\!\\approx\\\!4\.6\. Enabling data augmentation reduces them to3\.703\.70and7\.207\.20, indicating that data augmentation mitigates, but does not fully eliminate, the severe overfitting of ViT on CIFAR\-100\.Dataset, NetworkDataAug\.\(𝐒,𝐂\)\\mathbf\{\(S,C\)\}Metric PairRegressionR2↑R^\{2\}\\uparrow, coefficientPCR↓\\downarrowVisualized inFig\.[8](https://arxiv.org/html/2606.29043#A5.F8)\.ExplanatoryPowerfailed/poor/moderate/good/excellentCIFAR\-100, ViTFalse\(bayesS, func norm\)0\.88,\(−,\+\)\(\-,\+\)20\.8%True\(bayesS, func norm\)0\.89,\(\+,\+\)\(\+,\+\)15\.2%

Table 10:Ablation study 3\.Regression and Pareto analysis for CIFAR\-100 ViT with and without data augmentation\. We evaluate the proposed function\-oriented metric pair \(bayesS, func norm\) under the two training settings\. Without data augmentation, the metric pair fails due to a negative regression coefficient and PCR=20\.8%=20\.8\\%\. With data augmentation, regression coefficients become positive,R2R^\{2\}remains high at0\.890\.89, and PCR decreases to15\.2%15\.2\\%, improving the explanatory power from*failed*to*moderate*\. These results suggest that the ViT failure is possibly associated with the extremely large test losses, which place the trained models outside a meaningful generalization regime\.![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c100_vit_bayeS_funcnorm_darkcolor.png)\(a\)ViT, data augmentation=False\.PCR=20\.8%\\text\{PCR\}\\\!=\\\!20\.8\\%\.
![Refer to caption](https://arxiv.org/html/2606.29043v1/figs/c100_vitDataAug_bayes_funcnorm.png)\(b\)ViT, data augmentation=True\.PCR=15\.2%\\text\{PCR\}\\\!=\\\!15\.2\\%\.

Figure 8:Pareto analysisresults for CIFAR\-100 ViT with and without data augmentation, serving as visualizations of the PCR column of Table[10](https://arxiv.org/html/2606.29043#A5.T10)\.