PAC--Bayes Bounds on Quotient Parameter Spaces: Geometry-induced Implicit-Bias Priors

# PAC–Bayes Bounds on Quotient Parameter Spaces: Geometry-induced Implicit-Bias Priors
Source: [https://arxiv.org/html/2607.18422](https://arxiv.org/html/2607.18422)
Nicola Aladrah1Fabio Anselmi1,2 1Department of Mathematics, Informatics and Geoscience, University of Trieste, Via Valerio 12/1, 34127 Trieste, Italy 2McGovern Institute, MIT, Main Street, Cambridge, MA 02139, USA

###### Abstract

Overparameterized models often have continuous parameter symmetries, so different parameters define the same predictor\. We show that PAC–Bayesian analysis should be performed on the quotient predictor space: pushing a prior and posterior to the quotient preserves the empirical and population Gibbs risks while removing the nonnegative KL contribution caused solely by how the two distributions differ among parameterizations of the same predictor\.

Quotienting alone does not determine which prior to use\. We construct a canonical choice of one parameterization for each predictor and account for the geometric volume of its equivalent parameterizations\. This transforms a neutral reference prior into a data\-independent prior that reflects the model’s implicit bias\. It approximates the ideal but inadmissible posterior\-matched prior, which would minimize the KL term by depending on the training data\. The resulting certificate is tighter exactly when this geometry\-induced prior has smaller KL divergence from the learned quotient posterior than the neutral prior\.

We test this prediction in Fourier regression with a Hadamard parameterization and in Query\-Key attention, using ordinary SGD without an explicit regularizer\. The implicit\-bias prior reduces the mean quotient\-space KL by40\.69%40\.69\\%and the mean PAC–Bayes certificate by21\.40%21\.40\\%in the Fourier\-Hadamard experiment\. The smaller, prior\-scale\-dependent improvement in Query\-Key attention confirms the predicted conditional nature of the effect\.

## 1Introduction

Modern learning models usually operate in overparameterized regimes, where the number of trainable parameters exceeds the number of observations\. Such models may contain large families of parameters that represent exactly the same predictor, as occurs, for example, in machine learning models with parameter symmetries\[[1](https://arxiv.org/html/2607.18422#bib.bib1),[15](https://arxiv.org/html/2607.18422#bib.bib2)\]\. To make this precise, let𝒳\\mathcal\{X\}be the input space and𝒴\\mathcal\{Y\}the output or label space\. Letℬ\\mathcal\{B\}denote the parameter space, and letb∈ℬb\\in\\mathcal\{B\}be a parameter, such as the collection of weights of a model\. Eachbbdefines a predictorfb:𝒳→𝒴f\_\{b\}:\\mathcal\{X\}\\to\\mathcal\{Y\}, which maps an inputx∈𝒳x\\in\\mathcal\{X\}to a predictionfb​\(x\)∈𝒴f\_\{b\}\(x\)\\in\\mathcal\{Y\}\. The empirical risk is the average loss offbf\_\{b\}on the observed training sample, whereas the population risk is its expected loss under the data\-generating distribution\. Letbbandb′b^\{\\prime\}be two parameters related by such symmetry\. Iffb=fb′f\_\{b\}=f\_\{b^\{\\prime\}\}as functions on𝒳\\mathcal\{X\}, then they induce identical empirical and population risks for every loss that depends only on the predictor\. This motivates our central goal: to formulate PAC–Bayesian complexity so that it reflects differences between predictors rather than differences between parameters that represent the same predictor\. A PAC–Bayesian analysis performed directly on parameter space can nevertheless assign different prior and posterior masses tobbandb′b^\{\\prime\}, thereby charging their discrepancy in the Kullback–Leibler divergence\. Its complexity term may therefore depend on representational degrees of freedom that have no effect on prediction or risk\.

PAC–Bayesian theory bounds the population risk of a randomized predictor in terms of its empirical risk and a complexity penaltyKL⁡\(Q∥P\)\\operatorname\{KL\}\(Q\\\|P\), wherePPis a prior independent of the certification sample andQQis a data\-dependent posterior over hypotheses\[[7](https://arxiv.org/html/2607.18422#bib.bib3),[9](https://arxiv.org/html/2607.18422#bib.bib4)\]\. How should this framework be formulated for modern overparameterized models whose parameter spaces may contain multiple representations of the same predictor? When the parameter space itself is taken as the hypothesis space, the KL divergence can distinguish probability mass assigned to different parameters that define the same predictor\. The resulting certificate consequently mixes predictor\-level complexity with variation among equivalent parameter representatives\. To address this problem, we apply the McAllester PAC–Bayes inequality\[[8](https://arxiv.org/html/2607.18422#bib.bib17)\]on the quotient hypothesis space, where all the parameter representing the same predictor are identified\. In this way, the symmetry structure of an overparameterized model becomes part of the specification of the PAC–Bayesian hypothesis space\.

The symmetry considered here acts on parameter space rather than on the data\. To make this concrete, consider the regression predictor,fa,c​\(x\)=\(a​c\)​xf\_\{a,c\}\(x\)=\(ac\)x, where the regression weight isw=a​cw=ac\. The transformation,\(a,c\)↦\(λ,a,λ−1​c\)\(a,c\)\\mapsto\(\\lambda,a,\\lambda^\{\-1\}c\), forλ\>0\\lambda\>0, leavesww, and therefore the predictor unchanged\. For a fixedww, the set,𝒪w=\{\(λ​a,λ−1​c\):λ\>0\}\\mathcal\{O\}\_\{w\}=\\\{\(\\lambda a,\\lambda^\{\-1\}c\):\\lambda\>0\\\}, is𝒪w\\mathcal\{O\}\_\{w\}\-orbit or the symmetry orbit: its elements are different parameter values representing the same regression function\. In general, for every group elementggand parameterbb,fg⋅b=fbf\_\{g\\cdot b\}=f\_\{b\}\. The statistically relevant hypothesis is therefore the equivalence class of the parameters that implement one predictor\. We represent this hypothesis space by the quotient of parameter space under the symmetry action, which collapses each symmetry orbit to a single quotient element\. In the regression example, the whole orbit𝒪w\\mathcal\{O\}\_\{w\}is represented by the single weightww\. Geometrically, this identification is illustrated by the hyperbolic symmetry orbits and the representative symmetry\-breaking slice shown in Fig\. 1 of the previous work\[[1](https://arxiv.org/html/2607.18422#bib.bib1)\]\. This operation removes non\-identifiable representational degrees of freedom\. It neither imposes an invariance on the data distribution nor averages predictions from distinct predictors\.

This quotient construction directly impacts the PAC–Bayesian complexity term\. Let a parameter\-space prior and posterior be pushed forward through the quotient map so that probability mass is assigned to the predictor classes represented by the parameters\. Collapsing each symmetry orbit into a single quotient point does not change the predictor represented by that orbit\. Hence, computing the empirical or population risk before this identification gives the same result as computing it afterward, and their posterior averages are unchanged\. The Kullback–Leibler divergence behaves differently\. On the parameter space, it measures how differently the prior and posterior distribute probability among the parameters that belongs to the same orbit\. Once the orbit is collapsed to a single quotient point, this representational discrepancy disappears\. The KL divergence on the quotient space therefore cannot be greater than the KL divergence on the parameter space\. Applying the same PAC–Bayes inequality on the quotient thus preserves the risk terms while removing complexity that comes solely from differences between equivalent parameter representations\.

This quotient construction is the first main contribution of work\. It provides a general PAC–Bayesian reduction for predictor\-preserving parameter symmetries: the complexity term cannot increase when equivalent parameterizations are identified, and it decreases whenever the posterior and prior distribute probability differently along symmetry orbits\. This perspective follows in spirit the symmetry\-based PAC–Bayesian analysis developed in\[[6](https://arxiv.org/html/2607.18422#bib.bib11)\]\. Here, however, the symmetry is formulated specifically on the parameter space: it relates different parameterizations of one predictor rather than transformations of the inputs or outputs\. This parameter\-space viewpoint connects naturally to the geometric mechanism of implicit bias prior induced by symmetry breaking, developed in\[[1](https://arxiv.org/html/2607.18422#bib.bib1)\]\.

The quotient reduction, however, does not by itself select a preferred prior among the possible priors on the quotient space\. This leads to our second contribution\. The geometry of the symmetry provides a canonical way to represent each set of equivalent parameters by a single point\. It also assigns a geometric weight to each predictor according to the volume of its redundant parameterizations\. This weight defines animplicit\-bias prioron the quotient space, which defines a preference determined by the model architecture rather than by the data or by an added training penalty\. This prior gives a tighter PAC–Bayes certificate than a neutral quotient prior only if it has smaller Kullback–Leibler divergence from the learned posterior\. We evaluate this explicit posterior\-dependent condition numerically\. Thus, quotienting gives a general complexity reduction, while the additional benefit of the implicit\-bias prior must be verified for the learned posterior\.

Our results distinguish two effects\. First, identifying the parameters that represent the same predictor leaves the risk terms unchanged and cannot increase the complexity term for any prior independent of the certification sample and any posterior\. Second, choosing the implicit\-bias prior does not necessarily improve the certificate\. It helps only when this prior is closer to the chosen posterior, in KL divergence, than the neutral reference prior\. We derive the exact condition for this comparison and test it in controlled numerical experiments\.

### 1\.1Related Work

PAC–Bayesian theory gives high\-probability bounds on the population risk of randomized predictors by comparing a data\-dependent posterior with a prior chosen independently of the sample\[[7](https://arxiv.org/html/2607.18422#bib.bib3),[9](https://arxiv.org/html/2607.18422#bib.bib4)\]\. Subsequent work has developed PAC–Bayes\-kl inequalities, alternative loss\-dependent bounds, and formulations for settings such as linear regression and Gaussian\-process classification\[[10](https://arxiv.org/html/2607.18422#bib.bib10),[4](https://arxiv.org/html/2607.18422#bib.bib12),[2](https://arxiv.org/html/2607.18422#bib.bib8),[11](https://arxiv.org/html/2607.18422#bib.bib9)\]\. Our analysis takes a different starting point\. Rather than modifying the PAC–Bayes inequality or introducing another loss\-dependent bound, we examine how the hypothesis space and the prior should be defined when the parameterization contains multiple representations of the same predictor\. We therefore use the symmetry structure of the parameter space to pass to a quotient hypothesis space and then use the associated geometric correction to construct a prior on that quotient\. Our analysis uses a bounded certification loss and evaluates the posterior–prior KL divergence explicitly\.

Parameter symmetries are a central source of non\-identifiability in modern learning models\. Continuous rescaling and change\-of\-basis symmetries can leave the represented predictor unchanged while modifying optimization trajectories, local geometry, and parameter\-space descriptions of complexity\[[15](https://arxiv.org/html/2607.18422#bib.bib2),[14](https://arxiv.org/html/2607.18422#bib.bib13),[12](https://arxiv.org/html/2607.18422#bib.bib14),[13](https://arxiv.org/html/2607.18422#bib.bib15),[5](https://arxiv.org/html/2607.18422#bib.bib16)\]\. In a complementary geometric analysis of stochastic learning, symmetry breaking yields a volume correction in the reduced distribution and thereby induces an effective preference among equivalent parameter representations that can be treated as the implicit bias of the model architecture\[[1](https://arxiv.org/html/2607.18422#bib.bib1)\]\. We use this correction for a different purpose: to construct and compare quotient\-space priors for PAC–Bayesian certification\.

A related PAC–Bayesian literature studied symmetries of the learning problem, where a group acts on inputs and outputs and the hypothesis class is required to be invariant or equivariant\. PAC–Bayesian generalization benefits have been established for compact group symmetries with invariant data distributions\[[6](https://arxiv.org/html/2607.18422#bib.bib11)\], and subsequently extended to non\-compact groups and non\-invariant data distributions\[[3](https://arxiv.org/html/2607.18422#bib.bib6)\]\. These works concern how transformed inputs and outputs are related by a predictor\. Our setting is different: the symmetry acts only on the parameters and leaves the predictor unchanged\. For Hadamard parameterization and other learning models that admit symmetries, the appropriate operation is therefore to identify parameter representatives of one predictor before evaluating the KL divergence, rather than to average over transformed inputs or impose predictor equivariance\.

Quotient constructions and reparameterization are also common in learning models with non\-identifiable parameters\. The PAC–Bayesian approach is more specific: KL divergence is defined between probability measures on the selected hypothesis space\. When equivalent parameters are treated as distinct elements of the parameter\-space, the KL term can include posterior–prior mismatch within a predictor equivalence class\. The quotient formulation makes this contribution explicit by preserving posterior\-averaged risks under pushforward while contracting the KL term\. It therefore separates predictor\-level complexity from representational complexity\.

## 2PAC–Bayesian bound on quotient parameter space

Letℬ∘\\mathcal\{B\}^\{\\circ\}denote the regular part of the parameter space, equipped with its Borelσ\\sigma\-algebra\. Each parameterb∈ℬ∘b\\in\\mathcal\{B\}^\{\\circ\}defines a measurable predictorfb:𝒳→𝒴f\_\{b\}:\\mathcal\{X\}\\to\\mathcal\{Y\}, where𝒳\\mathcal\{X\}is the input space and𝒴\\mathcal\{Y\}is the output space\. Let𝒟\\mathcal\{D\}be the data\-generating distribution on𝒳×𝒴\\mathcal\{X\}\\times\\mathcal\{Y\}, and letS=\(Z1,…,Zn\)∼𝒟nS=\(Z\_\{1\},\\ldots,Z\_\{n\}\)\\sim\\mathcal\{D\}^\{n\}forZi=\(Xi,Yi\)Z\_\{i\}=\(X\_\{i\},Y\_\{i\}\), be a certification sample of sizenn\. We use a measurable lossℓ:𝒴×𝒴→\[0,1\]\\ell:\\mathcal\{Y\}\\times\\mathcal\{Y\}\\to\[0,1\]\. Forb∈ℬ∘b\\in\\mathcal\{B\}^\{\\circ\}, its population and empirical risks are

Rb​\(b\)\\displaystyle R^\{b\}\(b\):=𝔼\(X,Y\)∼𝒟​\[ℓ​\(fb​\(X\),Y\)\],\\displaystyle:=\\mathbb\{E\}\_\{\(X,Y\)\\sim\\mathcal\{D\}\}\\left\[\\ell\\bigl\(f\_\{b\}\(X\),Y\\bigr\)\\right\],\(1\)R^Sb​\(b\)\\displaystyle\\widehat\{R\}\_\{S\}^\{b\}\(b\):=1n​∑i=1nℓ​\(fb​\(Xi\),Yi\)\.\\displaystyle:=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\ell\\bigl\(f\_\{b\}\(X\_\{i\}\),Y\_\{i\}\\bigr\)\.\(2\)
LetPbP^\{b\}be a prior probability measure onℬ∘\\mathcal\{B\}^\{\\circ\}, chosen independently of the certification sampleSS, and letQbQ^\{b\}be a posterior probability measure onℬ∘\\mathcal\{B\}^\{\\circ\}, which may depend onSS\. For a confidence parameterδ∈\(0,1\)\\delta\\in\(0,1\), the McAllester PAC–Bayes bound states that, with probability at least1−δ1\-\\deltaover the draw ofSS,

𝔼b∼Qb​\[Rb\]\\displaystyle\\mathbb\{E\}\_\{b\\sim Q^\{b\}\}\[R^\{b\}\]≤𝔼b∼Qb​\[R^Sb\]\+Compn,δ⁡\(KL⁡\(Qb∥Pb\)\)\.\\displaystyle\\leq\\mathbb\{E\}\_\{b\\sim Q^\{b\}\}\[\\widehat\{R\}\_\{S\}^\{b\}\]\+\\operatorname\{Comp\}\_\{n,\\delta\}\(\\operatorname\{KL\}\(Q^\{b\}\\\|P^\{b\}\)\)\.\(3\)where,

Compn,δ⁡\(KL⁡\(Qb∥Pb\)\)\\displaystyle\\operatorname\{Comp\}\_\{n,\\delta\}\(\\operatorname\{KL\}\(Q^\{b\}\\\|P^\{b\}\)\)=\\displaystyle=KL⁡\(Qb∥Pb\)\+log⁡\(1/δ\)\+log⁡n\+22​n−1\\displaystyle\\sqrt\{\\frac\{\\operatorname\{KL\}\(Q^\{b\}\\\|P^\{b\}\)\+\\log\(1/\\delta\)\+\\log n\+2\}\{2n\-1\}\}\(4\)HereKL⁡\(Qb∥Pb\)\\operatorname\{KL\}\(Q^\{b\}\\\|P^\{b\}\)is the Kullback–Leibler divergence between the posterior and the prior\[[8](https://arxiv.org/html/2607.18422#bib.bib17)\]\. We denote the complexity term byCompn,δ\(KL\(⋅∥⋅\)\)\\operatorname\{Comp\}\_\{n,\\delta\}\(\\operatorname\{KL\}\(\\cdot\\\|\\cdot\)\)\.

To introduce the notion of quotient, we present first the symmetry reduction\. Let𝒢\\mathcal\{G\}be a group acting measurably onℬ∘\\mathcal\{B\}^\{\\circ\}, and letU:=B∘/𝒢U:=B^\{\\circ\}/\\mathcal\{G\}be the corresponding quotient space\. Its elements are the𝒢\\mathcal\{G\}\-orbits of the parameters\. We assume thatUUis a measurable space and denote the quotient map byϕ:B∘→U\\phi:B^\{\\circ\}\\to U, whereϕ​\(b\)=\[b\]\\phi\(b\)=\[b\]\. Here\[b\]\[b\]is the orbit containingbb\. The quotient prior and posterior are the pushforwardsPu:=ϕ\#​PbP^\{u\}:=\\phi\_\{\\\#\}P^\{b\}andQu:=ϕ\#​QbQ^\{u\}:=\\phi\_\{\\\#\}Q^\{b\}\. Thus,PuP^\{u\}andQuQ^\{u\}assign probability to symmetry orbits rather than to individual parameter representatives\. SincePbP^\{b\}is independent ofSS, its pushforwardPuP^\{u\}is also independent ofSSand is therefore an appropriate PAC–Bayesian prior on the quotient space\.

To justify replacing the parameter space by its quotient, we must check the two parts of the PAC–Bayesian certificate separately\. First, we assume that identifying equivalent parameters does not change the empirical or population risk\. Second, we must compare the KL divergence before and after the identification\. The first point holds whenever the symmetry changes only the parameter representation and leaves the predictor unchanged\.

###### Assumption\(Parameter\-space symmetry\)\.

For everyg∈𝒢g\\in\\mathcal\{G\}andb∈ℬ∘b\\in\\mathcal\{B\}^\{\\circ\},

fg⋅b=fb\.\\displaystyle f\_\{g\\cdot b\}=f\_\{b\}\.\(5\)Moreover, there exists a measurable predictor mapu↦fuu\\mapsto f\_\{u\}onUUsuch that

fb=fϕ​\(b\),b∈ℬ∘\.\\displaystyle f\_\{b\}=f\_\{\\phi\(b\)\},\\qquad b\\in\\mathcal\{B\}^\{\\circ\}\.\(6\)Thus, the group action changes only the parameter representative\. It does not act on the input, output, or predictor\. Consequently, the loss, population risk, and empirical risk are constant on each𝒢\\mathcal\{G\}\-orbit and depend onbbonly throughu=ϕ​\(b\)u=\\phi\(b\)\.

The risks can be evaluated directly on the quotient space, and hence their posterior averages can therefore be computed equivalently in either space\.

###### Proposition 1\.

Under the parameter\-space symmetry assumption, define, foru∈Uu\\in U,

Ru​\(u\)\\displaystyle R^\{u\}\(u\):=\\displaystyle:=𝔼\(X,Y\)∼𝒟​\[ℓ​\(fu​\(X\),Y\)\],\\displaystyle\\mathbb\{E\}\_\{\(X,Y\)\\sim\\mathcal\{D\}\}\\left\[\\ell\\bigl\(f\_\{u\}\(X\),Y\\bigr\)\\right\],\(7\)R^Su​\(u\)\\displaystyle\\widehat\{R\}\_\{S\}^\{u\}\(u\):=\\displaystyle:=1n​∑i=1nℓ​\(fu​\(Xi\),Yi\)\.\\displaystyle\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\ell\\bigl\(f\_\{u\}\(X\_\{i\}\),Y\_\{i\}\\bigr\)\.\(8\)These quotient risks are measurable and satisfy

Rb=Ru∘ϕ,R^Sb=R^Su∘ϕ\.\\displaystyle R^\{b\}=R^\{u\}\\circ\\phi,\\qquad\\widehat\{R\}\_\{S\}^\{b\}=\\widehat\{R\}\_\{S\}^\{u\}\\circ\\phi\.\(9\)Consequently, for every probability measureQbQ^\{b\}onℬ∘\\mathcal\{B\}^\{\\circ\}and its pushforwardQu=ϕ\#​QbQ^\{u\}=\\phi\_\{\\\#\}Q^\{b\},

∫ℬ∘Rb​\(b\)​Qb​\(d​b\)\\displaystyle\\int\_\{\\mathcal\{B\}^\{\\circ\}\}R^\{b\}\(b\)\\,Q^\{b\}\(db\)=\\displaystyle=∫URu​\(u\)​Qu​\(d​u\),\\displaystyle\\int\_\{U\}R^\{u\}\(u\)\\,Q^\{u\}\(du\),\(10\)∫ℬ∘R^Sb​\(b\)​Qb​\(d​b\)\\displaystyle\\int\_\{\\mathcal\{B\}^\{\\circ\}\}\\widehat\{R\}\_\{S\}^\{b\}\(b\)\\,Q^\{b\}\(db\)=\\displaystyle=∫UR^Su​\(u\)​Qu​\(d​u\)\.\\displaystyle\\int\_\{U\}\\widehat\{R\}\_\{S\}^\{u\}\(u\)\\,Q^\{u\}\(du\)\.\(11\)

Instead the parameter\-space KL divergence is not invariant since it can include differences between the prior and posterior in how they distribute probability among different parameters belonging to the same symmetry orbit\. The quotient map removes this within\-orbit information\.

###### Proposition 2\.

AssumeQb≪PbQ^\{b\}\\ll P^\{b\}, and assume additionally thatℬ∘\\mathcal\{B\}^\{\\circ\}and𝒰\\mathcal\{U\}are standard Borel spaces\. ThenQu≪PuQ^\{u\}\\ll P^\{u\}\. LetQb\(⋅∣u\)Q^\{b\}\(\\cdot\\mid u\)andPb\(⋅∣u\)P^\{b\}\(\\cdot\\mid u\)be regular conditional distributions ofQbQ^\{b\}andPbP^\{b\}, respectively, givenu=ϕ​\(b\)u=\\phi\(b\)\. Then

KL⁡\(Qb∥Pb\)=KL⁡\(Qu∥Pu\)\+Δϕ,\\operatorname\{KL\}\(Q^\{b\}\\\|P^\{b\}\)=\\operatorname\{KL\}\(Q^\{u\}\\\|P^\{u\}\)\+\\Delta\_\{\\phi\},\(12\)where

Δϕ:=∫𝒰KL\(Qb\(⋅∣u\)∥Pb\(⋅∣u\)\)Qu\(du\)≥0\.\\Delta\_\{\\phi\}:=\\int\_\{\\mathcal\{U\}\}\\operatorname\{KL\}\\\!\\left\(Q^\{b\}\(\\cdot\\mid u\)\\,\\\|\\,P^\{b\}\(\\cdot\\mid u\)\\right\)\\,Q^\{u\}\(du\)\\geq 0\.\(13\)Consequently,

KL⁡\(Qu∥Pu\)≤KL⁡\(Qb∥Pb\)\.\\operatorname\{KL\}\(Q^\{u\}\\\|P^\{u\}\)\\leq\\operatorname\{KL\}\(Q^\{b\}\\\|P^\{b\}\)\.\(14\)

This extra contribution, Eq\. \([13](https://arxiv.org/html/2607.18422#S2.E13)\), is nonnegative, so passing to the quotient leaves the two risk terms unchanged and cannot increase the complexity term\. We can therefore apply the same PAC–Bayesian inequality on the quotient space\.

###### Theorem 1\.

Suppose Propositions[1](https://arxiv.org/html/2607.18422#Thmproposition1)and[2](https://arxiv.org/html/2607.18422#Thmproposition2)holds\. In particular, thatPbP^\{b\}is independent ofSS, thatQb≪PbQ^\{b\}\\ll P^\{b\}, and that the parameter\-space symmetry assumption holds\. LetPu=ϕ\#​PbP^\{u\}=\\phi\_\{\\\#\}P^\{b\}andQu=ϕ\#​QbQ^\{u\}=\\phi\_\{\\\#\}Q^\{b\}\. Consider a PAC–Bayesian inequality whose certificate is nondecreasing in the KL divergence\. Applied onUU, it yields, with probability at least1−δ1\-\\deltaover the draw of the certification sampleSS,

𝔼u∼Qu​\[Ru\]≤𝔼u∼Qu​\[R^Su\]\+Compn,δ⁡\(KL⁡\(Qu∥Pu\)\),\\displaystyle\\mathbb\{E\}\_\{u\\sim Q^\{u\}\}\[R^\{u\}\]\\leq\\mathbb\{E\}\_\{u\\sim Q^\{u\}\}\[\\widehat\{R\}\_\{S\}^\{u\}\]\+\\operatorname\{Comp\}\_\{n,\\delta\}\\\!\\left\(\\operatorname\{KL\}\(Q^\{u\}\\\|P^\{u\}\)\\right\),\(15\)whereCompn,δ\\operatorname\{Comp\}\_\{n,\\delta\}is nondecreasing\. Moreover,

𝔼u∼Qu​\[Ru\]=𝔼b∼Qb​\[Rb\],𝔼u∼Qu​\[R^Su\]=𝔼b∼Qb​\[R^Sb\],\\displaystyle\\begin\{split\}\\mathbb\{E\}\_\{u\\sim Q^\{u\}\}\[R^\{u\}\]&=\\mathbb\{E\}\_\{b\\sim Q^\{b\}\}\[R^\{b\}\],\\\\ \\mathbb\{E\}\_\{u\\sim Q^\{u\}\}\[\\widehat\{R\}\_\{S\}^\{u\}\]&=\\mathbb\{E\}\_\{b\\sim Q^\{b\}\}\[\\widehat\{R\}\_\{S\}^\{b\}\],\\end\{split\}\(16\)KL⁡\(Qu∥Pu\)=KL⁡\(Qb∥Pb\)−Δϕ\.\\operatorname\{KL\}\(Q^\{u\}\\\|P^\{u\}\)=\\operatorname\{KL\}\(Q^\{b\}\\\|P^\{b\}\)\-\\Delta\_\{\\phi\}\.\(17\)where,Δϕ\\Delta\_\{\\phi\}is defined in Eq\. \([13](https://arxiv.org/html/2607.18422#S2.E13)\)\. Consequently, the quotient certificate is no larger than the certificate obtained by applying the same PAC–Bayesian inequality onℬ∘\\mathcal\{B\}^\{\\circ\}withPbP^\{b\}andQbQ^\{b\}\.

This theorem shows that, for a fixed parameter\-space prior and posterior, passing to the quotient preserves the empirical Gibbs risk and cannot increase the KL complexity term\. Thus, for any PAC–Bayesian bound whose complexity term is nondecreasing in the KL divergence, quotienting yields a certificate no larger than the corresponding parameter\-space certificate\. Having established this general PAC–Bayesian advantage of removing redundant parameterizations, we now focus on the quotient space itself\. The remaining question is how to select a sample\-independent prior on the quotient that is well aligned with the learned quotient posterior\. In the next section, we introduce the implicit\-bias prior and derive a prior\-improvement criterion that provides a particular and sufficient comparison with a neutral reference prior\. We evaluate this criterion using numerical experiments in Hadamard and Query\-Key attention settings\.

## 3Implicit\-bias priors

The quotient result in Theorem[1](https://arxiv.org/html/2607.18422#Thmtheorem1)does not tell which prior is most appropriate\. The missing principle comes from our previous geometric analysis of symmetry breaking in redundant stochastic parametrizations\[[1](https://arxiv.org/html/2607.18422#bib.bib1)\]\. There, we showed that the symmetry of a parameterization determines a geometry\-dependent correction, which can be used to construct an implicit\-bias prior\. It identifies, in closed form, a preference that emerges from the optimization dynamics despite not being imposed as part of the training objective\.

###### Theorem 2\(Symmetry\-breaking induced implicit bias\[[1](https://arxiv.org/html/2607.18422#bib.bib1)\]\)\.

Letℬ∘\\mathcal\{B\}^\{\\circ\}be a smooth Riemannian manifold with metricgg, and let anmm\-dimensional Lie group𝒢\\mathcal\{G\}act smoothly, freely, and properly onℬ∘\\mathcal\{B\}^\{\\circ\}by predictor\-preserving transformations\. Letχ:ℬ∘→ℝm\\chi:\\mathcal\{B\}^\{\\circ\}\\to\\mathbb\{R\}^\{m\}be a smooth symmetry\-breaking map such that0is a regular value andSχ:=χ−1​\(0\)S\_\{\\chi\}:=\\chi^\{\-1\}\(0\)is a transverse slice intersecting each orbit once in the region under consideration\. Define

\(Gχ​\(b\)\)i​j:=⟨∇χi​\(b\),∇χj​\(b\)⟩g\.\\bigl\(G\_\{\\chi\}\(b\)\\bigr\)\_\{ij\}:=\\left\\langle\\nabla\\chi\_\{i\}\(b\),\\nabla\\chi\_\{j\}\(b\)\\right\\rangle\_\{g\}\.\(18\)The stationary density induced onSχS\_\{\\chi\}is proportional to

exp⁡\[−βσ2​L​\(b\)\]​detGχ​\(b\)−1/2,\\exp\\\!\\left\[\-\\frac\{\\beta\}\{\\sigma^\{2\}\}L\(b\)\\right\]\\det G\_\{\\chi\}\(b\)^\{\-1/2\},\(19\)with respect to the induced surface measure\. Equivalently, the symmetry\-reduced dynamics are associated with the effective loss

Leff​\(b\)=L​\(b\)\+σ22​β​log​detGχ​\(b\)\.L\_\{\\mathrm\{eff\}\}\(b\)=L\(b\)\+\\frac\{\\sigma^\{2\}\}\{2\\beta\}\\log\\det G\_\{\\chi\}\(b\)\.\(20\)

This theorem identifies the geometric factordetGχ−1/2\\det G\_\{\\chi\}^\{\-1/2\}that represents the implicit bias\. Letsχ:U→Sχs\_\{\\chi\}:U\\to S\_\{\\chi\}denote the section induced by the slice, and define the corresponding implicit\-bias loss

Lχ​\(u\):=12​log​detGχ​\(sχ​\(u\)\)\.L\_\{\\chi\}\(u\):=\\frac\{1\}\{2\}\\log\\det G\_\{\\chi\}\\bigl\(s\_\{\\chi\}\(u\)\\bigr\)\.\(21\)We use this factor to construct a PAC–Bayesian prior\. Given a sample\-independent reference priorP0uP\_\{0\}^\{u\}such that

0<Zχ:=∫Ue−Lχ​\(u\)​P0u​\(d​u\)<∞,0<Z\_\{\\chi\}:=\\int\_\{U\}e^\{\-L\_\{\\chi\}\(u\)\}P\_\{0\}^\{u\}\(du\)<\\infty,\(22\)define

PIBu​\(d​u\):=e−Lχ​\(u\)Zχ​P0u​\(d​u\)\.P\_\{\\mathrm\{IB\}\}^\{u\}\(du\):=\\frac\{e^\{\-L\_\{\\chi\}\(u\)\}\}\{Z\_\{\\chi\}\}P\_\{0\}^\{u\}\(du\)\.\(23\)The remaining question is whether this geometric reweighting actually improves the PAC–Bayesian certificate\. The following proposition is the second main result of our analysis\. It gives an exact criterion for when the implicit\-bias prior has a smaller KL divergence from a given quotient posterior than the reference prior\.

###### Proposition 3\(Prior\-improvement criterion\)\.

LetP0uP\_\{0\}^\{u\}be a sample\-independent reference prior onUU, and letPIBuP\_\{\\mathrm\{IB\}\}^\{u\}be the implicit\-bias prior defined in Eq\. \([23](https://arxiv.org/html/2607.18422#S3.E23)\)\. LetQu≪P0uQ^\{u\}\\ll P\_\{0\}^\{u\}be a quotient posterior such that𝔼u∼Qu​\[Lχ\]<∞\\mathbb\{E\}\_\{u\\sim Q^\{u\}\}\[L\_\{\\chi\}\]<\\infty\. Define the prior\-improvement criterion

Δ​KL⁡\(Qu∥PIBu\):=KL⁡\(Qu∥PIBu\)−KL⁡\(Qu∥P0u\)\.\\Delta\\operatorname\{KL\}\(Q^\{u\}\\\|P\_\{\\mathrm\{IB\}\}^\{u\}\):=\\operatorname\{KL\}\(Q^\{u\}\\\|P\_\{\\mathrm\{IB\}\}^\{u\}\)\-\\operatorname\{KL\}\(Q^\{u\}\\\|P\_\{0\}^\{u\}\)\.\(24\)Then

Δ​KL⁡\(Qu∥PIBu\)=𝔼u∼Qu​\[Lχ\]\+log⁡Zχ\.\\Delta\\operatorname\{KL\}\(Q^\{u\}\\\|P\_\{\\mathrm\{IB\}\}^\{u\}\)=\\mathbb\{E\}\_\{u\\sim Q^\{u\}\}\[L\_\{\\chi\}\]\+\\log Z\_\{\\chi\}\.\(25\)Moreover, consider any PAC–Bayesian certificate onUUwhose complexity term is nondecreasing in the KL divergence\. For the same quotient posteriorQuQ^\{u\}, replacingP0uP\_\{0\}^\{u\}byPIBuP\_\{\\mathrm\{IB\}\}^\{u\}leaves the empirical Gibbs risk unchanged\. Therefore, whenever

Δ​KL⁡\(Qu∥PIBu\)≤0,\\Delta\\operatorname\{KL\}\(Q^\{u\}\\\|P\_\{\\mathrm\{IB\}\}^\{u\}\)\\leq 0,\(26\)the certificate obtained withPIBuP\_\{\\mathrm\{IB\}\}^\{u\}is no larger than the certificate obtained withP0uP\_\{0\}^\{u\}\.

### 3\.1Case study: Hadamard parametrization

Consider the Hadamard parametrization of the weightsww, wuch thatw=a⊙cw=a\\odot c, on the regular parameter spaceℬ∘=\(ℝ×\)2​d\\mathcal\{B\}^\{\\circ\}=\(\\mathbb\{R\}^\{\\times\}\)^\{2d\}\. The group𝒢=\(ℝ×\)d\\mathcal\{G\}=\(\\mathbb\{R\}^\{\\times\}\)^\{d\}acts by coordinatewise rescaling,

λ⋅\(a,c\):=\(λ⊙a,λ−1⊙c\),λ∈𝒢\.\\displaystyle\\lambda\\cdot\(a,c\):=\\left\(\\lambda\\odot a,\\,\\lambda^\{\-1\}\\odot c\\right\),\\qquad\\lambda\\in\\mathcal\{G\}\.\(27\)This action preserves the weightsww, since\(λ⊙a\)⊙\(λ−1⊙c\)=a⊙c\(\\lambda\\odot a\)\\odot\(\\lambda^\{\-1\}\\odot c\)=a\\odot c\. Hence the quotient is identified withU=\(ℝ×\)dU=\(\\mathbb\{R\}^\{\\times\}\)^\{d\}, and the quotient map isϕ​\(a,c\)=a⊙c=w\\phi\(a,c\)=a\\odot c=w\. Following the symmetry\-breaking construction of\[[1](https://arxiv.org/html/2607.18422#bib.bib1)\], for the canonical slice

χi​\(a,c\):=12​\(ai2−ci2\)=0,i=1,…,d,\\displaystyle\\chi\_\{i\}\(a,c\):=\\frac\{1\}\{2\}\(a\_\{i\}^\{2\}\-c\_\{i\}^\{2\}\)=0,\\qquad i=1,\\ldots,d,\(28\)To choose one representative on each orbit, we impose the branch conventionai\>0a\_\{i\}\>0\. We next compute the geometric factor generated by this slice\. Under the Euclidean metric onℬ∘\\mathcal\{B\}^\{\\circ\}, the gradient of theii\-th constraint has nonzero componentsaia\_\{i\}and−ci\-c\_\{i\}\. Consequently, the constraint Gram matrix is diagonal:

\(Gχ​\(a,c\)\)i​j=δi​j​\(ai2\+ci2\)\.\\displaystyle\\bigl\(G\_\{\\chi\}\(a,c\)\\bigr\)\_\{ij\}=\\delta\_\{ij\}\(a\_\{i\}^\{2\}\+c\_\{i\}^\{2\}\)\.\(29\)Evaluating it on the balanced section we have

detGχ​\(sχ​\(w\)\)\\displaystyle\\det G\_\{\\chi\}\\bigl\(s\_\{\\chi\}\(w\)\\bigr\)=\\displaystyle=∏i=1d2​\|wi\|\.\\displaystyle\\prod\_\{i=1\}^\{d\}2\|w\_\{i\}\|\.\(30\)The resulting implicit\-bias loss is therefore

Lχ​\(w\)\\displaystyle L\_\{\\chi\}\(w\)=\\displaystyle=12​∑i=1dlog⁡\(2​\|wi\|\)\.\\displaystyle\\frac\{1\}\{2\}\\sum\_\{i=1\}^\{d\}\\log\\bigl\(2\|w\_\{i\}\|\\bigr\)\.\(31\)LetP0wP\_\{0\}^\{w\}be a reference prior onU=\(ℝ×\)dU=\(\\mathbb\{R\}^\{\\times\}\)^\{d\}, fixed independently of the certification sample, and assume that the following normalizing constant is finite:

ZIB:=∫U∏i=1d\|wi\|−1/2​P0w​\(d​w\)\.\\displaystyle Z\_\{\\mathrm\{IB\}\}:=\\int\_\{U\}\\prod\_\{i=1\}^\{d\}\|w\_\{i\}\|^\{\-1/2\}\\,P\_\{0\}^\{w\}\(dw\)\.\(32\)The exact Hadamard implicit\-bias prior is

PIBw​\(d​w\)\\displaystyle P\_\{\\mathrm\{IB\}\}^\{w\}\(dw\):=\\displaystyle:=1ZIB​∏i=1d\|wi\|−1/2​P0w​\(d​w\)\.\\displaystyle\\frac\{1\}\{Z\_\{\\mathrm\{IB\}\}\}\\prod\_\{i=1\}^\{d\}\|w\_\{i\}\|^\{\-1/2\}\\,P\_\{0\}^\{w\}\(dw\)\.\(33\)Here, the constant factor2−d/22^\{\-d/2\}fromexp⁡\[−Lχ​\(w\)\]\\exp\[\-L\_\{\\chi\}\(w\)\]has been absorbed intoZIBZ\_\{\\mathrm\{IB\}\}\. Thus, the symmetry\-induced logarithmic loss in Eq\. \([31](https://arxiv.org/html/2607.18422#S3.E31)\) enters the PAC–Bayesian analysis as a modified prior on the weightsww\.

Table 1:PAC–Bayesian certificates at the reference scales0=20s\_\{0\}=20\. Entries are mean±\\pmsample standard deviation over 40 optimizer seeds\.

## 4Experiments

We evaluate the prior\-improvement criterion from Proposition[3](https://arxiv.org/html/2607.18422#Thmproposition3)in two controlled settings: sparse Fourier regression with a Hadamard parameterization and a single\-head Query\-Key attention teacher–student model\. In both experiments, training uses only the ordinary mean\-squared\-error objective\. No implicit\-bias term is added during training\. After a burn\-in period \(post\-burn\-in\), we record SGD iterates and use them to fit an isotropic Gaussian posterior\. For each fitted posterior, the posterior distribution and empirical Gibbs risk are held fixed while the neutral and implicit\-bias priors are compared\. The complete certification protocol is given in Appendix[B](https://arxiv.org/html/2607.18422#A2)\.

For a fixed fitted posteriorQQand neutral\-prior scales0s\_\{0\}, we measure

Δ​KL⁡\(Q∥PIB\(s0\)\)\\displaystyle\\Delta\\operatorname\{KL\}\(Q\\\|P^\{\(s\_\{0\}\)\}\_\{\\mathrm\{IB\}\}\):=KL⁡\(Q∥PIB,ε\(s0\)\)−KL⁡\(Q∥P0\(s0\)\)\\displaystyle:=\\operatorname\{KL\}\\\!\\left\(Q\\middle\\\|P\_\{\\mathrm\{IB\},\\varepsilon\}^\{\(s\_\{0\}\)\}\\right\)\-\\operatorname\{KL\}\\\!\\left\(Q\\middle\\\|P\_\{0\}^\{\(s\_\{0\}\)\}\\right\)\(34\)=𝔼Q​\[Lε\]\+log⁡Zε​\(s0\)\.\\displaystyle=\\mathbb\{E\}\_\{Q\}\[L\_\{\\varepsilon\}\]\+\\log Z\_\{\\varepsilon\}\(s\_\{0\}\)\.\(35\)Consequently,Δ​KL⁡\(Q∥PIB\(s0\)\)<0\\Delta\\operatorname\{KL\}\(Q\\\|P^\{\(s\_\{0\}\)\}\_\{\\mathrm\{IB\}\}\)<0is exactly the condition under which the implicit\-bias prior decreases the prior\-dependent KL term\.

### 4\.1Sparse Fourier regression

The data\-generating predictor is a sparse Fourier series in a cosine basis of dimensiond=64d=64\. Its coefficient vectorw⋆∈ℝ64w^\{\\star\}\\in\\mathbb\{R\}^\{64\}has three nonzero entries,w3⋆=1\.2w^\{\\star\}\_\{3\}=1\.2,w9⋆=−0\.9w^\{\\star\}\_\{9\}=\-0\.9andw17⋆=0\.7w^\{\\star\}\_\{17\}=0\.7\. For inputstit\_\{i\}drawn uniformly from\[0,1\]\[0,1\], letΦ​\(ti\)∈ℝ64\\Phi\(t\_\{i\}\)\\in\\mathbb\{R\}^\{64\}denote the corresponding cosine\-feature vector\. We generatentrain=32n\_\{\\mathrm\{train\}\}=32training data as

yi\\displaystyle y\_\{i\}=\\displaystyle=Φ​\(ti\)⊤​w⋆\+ξi,\\displaystyle\\Phi\(t\_\{i\}\)^\{\\top\}w^\{\\star\}\+\\xi\_\{i\},\(36\)whereξi\\xi\_\{i\}is Gaussian noise with standard deviation equal to5%5\\%of the largest absolute noiseless training response\. An independent certification sample of sizencert=20,000n\_\{\\mathrm\{cert\}\}=20\{,\}000is drawn from the same distribution\.

We compare two linear models\. The naive dense model usesw∈ℝ64w\\in\\mathbb\{R\}^\{64\}and predicts

fw​\(t\)\\displaystyle f\_\{w\}\(t\)=\\displaystyle=Φ​\(t\)⊤​w\.\\displaystyle\\Phi\(t\)^\{\\top\}w\.\(37\)The Hadamard model uses\(a,c\)∈ℝ64×ℝ64\(a,c\)\\in\\mathbb\{R\}^\{64\}\\times\\mathbb\{R\}^\{64\}and predicts

fa,c​\(t\)\\displaystyle f\_\{a,c\}\(t\)=\\displaystyle=Φ​\(t\)⊤​\(a⊙c\)\.\\displaystyle\\Phi\(t\)^\{\\top\}\(a\\odot c\)\.\(38\)Both models are trained for70,00070\{,\}000SGD steps\. We use batch size1616, learning rate10−310^\{\-3\}, zero momentum, and zero weight decay\. No implicit\-bias term is added to the training objective\. For each optimizer seed, the weightwworw=a⊙cw=a\\odot c, is retained every200200updates from update60,00060\{,\}000through update70,00070\{,\}000\. This givesT=51T=51post\-burn\-in trajectory snapshots\.

For the naive model, the retained weight isw\(t\)w^\{\(t\)\}\. For the Hadamard model, it isw\(t\)=a\(t\)⊙b\(t\)w^\{\(t\)\}=a^\{\(t\)\}\\odot b^\{\(t\)\}, rather than the redundant parameter pair\(a\(t\),b\(t\)\)\(a^\{\(t\)\},b^\{\(t\)\}\)\. These retained SGD states are used to fit the posterior\. They are not treated as independent posterior samples\.

The neutral reference prior on the quotient space is the isotropic GaussianP0w,\(s0\)=𝒩​\(0,s02​I64\)P^\{w,\(s\_\{0\}\)\}\_\{0\}=\\mathcal\{N\}\(0,s\_\{0\}^\{2\}I\_\{64\}\)\. The implicit\-bias prior is obtained by tilting this reference prior with the implicit\-bias loss\. The corresponding implicit\-bias prior and its normalizing constant are constructed after training\. Their numerical evaluation is described in Appendix[B\.2](https://arxiv.org/html/2607.18422#A2.SS2)\.

### 4\.2Query\-Key attention

We consider a single\-head attention predictor with sequence lengthL=8L=8, embedding dimensionde=16d\_\{e\}=16, and head dimensiondh=8d\_\{h\}=8\. Its parameters are

θ=\(WQ,WK,WV\),WQ,WK,WV∈ℝde×dh\.\\theta=\(W\_\{Q\},W\_\{K\},W\_\{V\}\),\\qquad W\_\{Q\},W\_\{K\},W\_\{V\}\\in\\mathbb\{R\}^\{d\_\{e\}\\times d\_\{h\}\}\.For an input sequenceX∈ℝL×deX\\in\\mathbb\{R\}^\{L\\times d\_\{e\}\}, the predictor is

fθ​\(X\):=softmax⁡\(\(X​WQ\)​\(X​WK\)𝖳dh\)​X​WV,\\displaystyle f\_\{\\theta\}\(X\):=\\operatorname\{softmax\}\\\!\\left\(\\frac\{\(XW\_\{Q\}\)\(XW\_\{K\}\)^\{\\mathsf\{T\}\}\}\{\\sqrt\{d\_\{h\}\}\}\\right\)XW\_\{V\},\(39\)where the softmax is applied row\-wise\. A frozen teacher of the same form generates the labels

Y=fθ⋆​\(X\)\+E,\\displaystyle Y=f\_\{\\theta^\{\\star\}\}\(X\)\+E,\(40\)where the entries ofXXare independent standard Gaussian variables and the entries ofEEare independent centered Gaussian variables with standard deviation0\.020\.02\.

Posterior fitting begins after60%60\\%of training\. Every200200updates from update60,00060\{,\}000through update100,000100\{,\}000, the Query and Key matrices are mapped to the implicit\-bias loss,Lχ,εL\_\{\\chi,\\varepsilon\}, and the vectorz\(t\)=vec⁡\(W¯Q\(t\),W¯K\(t\),WV\(t\)\)∈ℝ384z^\{\(t\)\}=\\operatorname\{vec\}\\left\(\\bar\{W\}\_\{Q\}^\{\(t\)\},\\bar\{W\}\_\{K\}^\{\(t\)\},W\_\{V\}^\{\(t\)\}\\right\)\\in\\mathbb\{R\}^\{384\}\. is retained\. This givesT=201T=201trajectory snapshots\. An isotropic GaussianQzQ^\{z\}is fitted to these vectors\. The snapshots are observations from one correlated SGD trajectory and are used only to estimate the mean and scalar variance ofQzQ^\{z\}\. They are not treated as independent posterior samples as illustrated in Appendix[B\.3](https://arxiv.org/html/2607.18422#A2.SS3)\.

For a fixed fitted posteriorQzQ^\{z\}, the neutral reference prior isP0z,\(s0\)=𝒩​\(0,s02​I384\)P\_\{0\}^\{z,\(s\_\{0\}\)\}=\\mathcal\{N\}\(0,s\_\{0\}^\{2\}I\_\{384\}\)\. The implicit\-bias prior is obtained by tilting this Gaussian byexp⁡\[−Lχ,ε\]\\exp\[\-L\_\{\\chi,\\varepsilon\}\]\. Both certificates use the sameQzQ^\{z\}and the same empirical Gibbs risk\. Their difference therefore isolates the prior\-dependent KL term\.

We train using plain SGD\. Each update uses a newly generated batch of6464teacher examples\. We use learning rate10−310^\{\-3\}, zero momentum, zero weight decay, and100,000100\{,\}000steps\. Similarly to the Fourier experiment, no implicit\-bias term is added to the training objective\.

### 4\.3Results

Table[1](https://arxiv.org/html/2607.18422#S3.T1)reports the results at the reference scales0=20s\_\{0\}=20\. For the fitted Fourier Hadamard posterior, replacing the neutral prior by the regularized implicit\-bias prior reduces the mean KL divergence from585\.574684585\.574684to347\.300316347\.300316, a reduction of40\.69%40\.69\\%\. The corresponding mean certificate decreases from0\.1278630\.127863to0\.1004980\.100498, a reduction of21\.40%21\.40\\%\.

Table 2:Mean±\\pmsample standard deviation of the prior\-improvement quantity,Δ​KL⁡\(Qu∥PIBu\)\\Delta\\operatorname\{KL\}\(Q^\{u\}\\\|P^\{u\}\_\{\\mathrm\{IB\}\}\), over4040optimizer seeds\. Negative values indicate that the implicit\-bias prior has the smaller KL divergence\.
These two rows use the same posterior and empirical Gibbs risk, so the difference is caused entirely by the prior\-dependent KL term\.

For the Query\-Key implicit\-bias prior ats0=20s\_\{0\}=20, the mean KL divergence decreases from2288\.3193322288\.319332to2263\.2640582263\.264058, a reduction of1\.09%1\.09\\%\. The mean certificate decreases from0\.3013330\.301333to0\.3000240\.300024, a reduction of0\.43%0\.43\\%\. Thus, the prior improvement is substantially smaller in the Query\-Key experiment than in the Fourier experiment comparison\. This is not a surprising result\. The smaller relative improvement in can be understood from the number of symmetry directions included in the implicit\-bias term,dhd\_\{h\}\. In the Fourier\-Hadamard experiment, the correction acts across alld=64d=64independently\. While in the Query\-Key experiment, the evaluated correction acts only along thedh=8d\_\{h\}=8head\-wise symmetry directions, while the neutral\-prior KL is computed for the full384384\-dimensional vector containingWQW\_\{Q\},WKW\_\{K\}, andWVW\_\{V\}\. This explain why the total KL reduction is much smaller as the implicit\-bias loss modifies only a restricted part of the complexity term\.

Table[2](https://arxiv.org/html/2607.18422#S4.T2)shows that the meanΔ​KL⁡\(Qu∥PIBu\)\\Delta\\operatorname\{KL\}\(Q^\{u\}\\\|P^\{u\}\_\{\\mathrm\{IB\}\}\)is negative at all 11 tested scales in the Fourier\-Hadamard experiment\. In the Query\-Key experiment, its mean is positive ats0∈\{0\.25,0\.5,0\.75\}s\_\{0\}\\in\\\{0\.25,0\.5,0\.75\\\}and negative at the tested scaless0∈\{1,1\.25,1\.5,2,5,7,10,20\}s\_\{0\}\\in\\\{1,1\.25,1\.5,2,5,7,10,20\\\}\. Therefore, within the tested grid, the implicit\-bias prior does not improve the Query\-Key comparison at the three smallest scales\.

## 5Conclusion

This work separates predictor\-level complexity from representational complexity in PAC–Bayesian analysis\. For a predictor\-preserving parameter symmetry, pushing a parameter\-space prior and posterior through the quotient map leaves their posterior\-averaged empirical and population risks unchanged\. The KL divergence cannot increase, and its exact reduction is the conditional posterior–prior divergence within symmetry orbits\.

The geometric construction addresses the separate problem of selecting a prior on the quotient space\. Because the parameterization possesses a predictor\-preserving symmetry, the symmetry\-breaking geometry induces an additional preference in the SGD dynamics, implicitly, even though no explicit regularizer is added to the training objective\. This preference is represented by a volume correction, which defines an implicit\-bias prior through the tilting of a normalizable reference prior\. We show that this symmetry\-induced preference can improve the PAC–Bayes certificate without any intervention in the training procedure: when the learned posterior is sufficiently aligned with the implicit\-bias prior, the prior\-dependent KL term decreases and the certificate becomes tighter, precisely whenΔ​KL⁡\(Q∥PIB\)<0\\Delta\\operatorname\{KL\}\(Q\\\|P\_\{\\mathrm\{IB\}\}\)<0\. The improvement is therefore implicit and conditional, arising from the interaction between the symmetry\-induced geometry and the posterior produced by SGD\.

The experiments evaluate the posterior\-dependent prior comparison rather than the quotienting inequality itself\. In the Fourier\-Hadamard experiment, the regularized implicit\-bias prior has a smaller mean KL divergence than the neutral prior at every tested scale\. In the Query\-Key experiment, the mean KL difference is positive at the three tested scales below11and negative at every tested scale from11to2020\. The result therefore depends on both the learned posterior and the reference\-prior scale\.

This perspective suggests a connection with inverse learning of implicit bias\. Given a family of predictor\-preserving parameterizations, one could seek a design whose induced implicit\-bias prior is well matched to a specified class of posteriors\. Such a procedure must choose the parametrization and prior family independently of the certification sample\. Developing sample\-independent procedures for this inverse\-learning problem and constructing Bayesian quotient posteriors are natural directions for future work\.

## Data and Code Availability

The data and code used for the numerical experiments are available from the authors upon reasonable request\.

## References

- \[1\]N\. Aladrah, E\. Ballarin, M\. Biagetti, A\. Ansuini, A\. d’Onofrio, and F\. Anselmi\(2026\)Understanding and inverse design of implicit bias in stochastic learning: a geometric perspective\.arXiv preprint arXiv:2601\.06597\.Cited by:[§B\.3](https://arxiv.org/html/2607.18422#A2.SS3.p2.11),[§1\.1](https://arxiv.org/html/2607.18422#S1.SS1.p2.1),[§1](https://arxiv.org/html/2607.18422#S1.p1.15),[§1](https://arxiv.org/html/2607.18422#S1.p3.13),[§1](https://arxiv.org/html/2607.18422#S1.p5.1),[§3\.1](https://arxiv.org/html/2607.18422#S3.SS1.p1.8),[§3](https://arxiv.org/html/2607.18422#S3.p1.1),[Theorem 2](https://arxiv.org/html/2607.18422#Thmtheorem2)\.
- \[2\]P\. Alquier, J\. Ridgway, and N\. Chopin\(2016\)On the properties of variational approximations of gibbs posteriors\.Journal of Machine Learning Research17\(236\),pp\. 1–41\.External Links:[Link](http://jmlr.org/papers/v17/15-290.html)Cited by:[§1\.1](https://arxiv.org/html/2607.18422#S1.SS1.p1.1)\.
- \[3\]A\. Beck and P\. Ochs\(2025\)Symmetries in pac\-bayesian learning\.arXiv preprint arXiv:2510\.17303\.Cited by:[§1\.1](https://arxiv.org/html/2607.18422#S1.SS1.p3.1)\.
- \[4\]O\. Catoni\(2007\)PAC\-bayesian supervised classification: the thermodynamics of statistical learning\.arXiv preprint arXiv:0712\.0248\.Cited by:[§1\.1](https://arxiv.org/html/2607.18422#S1.SS1.p1.1)\.
- \[5\]C\. Godfrey, D\. Brown, T\. Emerson, and H\. Kvinge\(2022\)On the symmetries of deep learning models and their internal representations\.Advances in Neural Information Processing Systems35,pp\. 11893–11905\.Cited by:[§1\.1](https://arxiv.org/html/2607.18422#S1.SS1.p2.1)\.
- \[6\]C\. Lyle, M\. van der Wilk, M\. Kwiatkowska, Y\. Gal, and B\. Bloem\-Reddy\(2020\)On the benefits of invariance in neural networks\.arXiv preprint arXiv:2005\.00178\.Cited by:[§1\.1](https://arxiv.org/html/2607.18422#S1.SS1.p3.1),[§1](https://arxiv.org/html/2607.18422#S1.p5.1)\.
- \[7\]D\. A\. McAllester\(1998\)Some pac\-bayesian theorems\.InProceedings of the eleventh annual conference on Computational learning theory,pp\. 230–234\.Cited by:[§1\.1](https://arxiv.org/html/2607.18422#S1.SS1.p1.1),[§1](https://arxiv.org/html/2607.18422#S1.p2.3)\.
- \[8\]D\. A\. McAllester\(2003\)PAC\-bayesian stochastic model selection\.Machine Learning51\(1\),pp\. 5–21\.Cited by:[§1](https://arxiv.org/html/2607.18422#S1.p2.3),[§2](https://arxiv.org/html/2607.18422#S2.p2.11)\.
- \[9\]D\. McAllester\(2013\)A pac\-bayesian tutorial with a dropout bound\.arXiv preprint arXiv:1307\.2118\.Cited by:[§1\.1](https://arxiv.org/html/2607.18422#S1.SS1.p1.1),[§1](https://arxiv.org/html/2607.18422#S1.p2.3)\.
- \[10\]M\. Seeger\(2002\)PAC\-bayesian generalisation error bounds for gaussian process classification\.Journal of machine learning research3\(Oct\),pp\. 233–269\.Cited by:[§1\.1](https://arxiv.org/html/2607.18422#S1.SS1.p1.1)\.
- \[11\]V\. Shalaeva, A\. F\. Esfahani, P\. Germain, and M\. Petreczky\(2020\)Improved pac\-bayesian bounds for linear regression\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.34,pp\. 5660–5667\.Cited by:[§1\.1](https://arxiv.org/html/2607.18422#S1.SS1.p1.1)\.
- \[12\]B\. Simsek, F\. Ged, A\. Jacot, F\. Spadaro, C\. Hongler, W\. Gerstner, and J\. Brea\(2021\)Geometry of the loss landscape in overparameterized neural networks: symmetries and invariances\.InInternational Conference on Machine Learning,pp\. 9722–9732\.Cited by:[§1\.1](https://arxiv.org/html/2607.18422#S1.SS1.p2.1)\.
- \[13\]B\. Zhao, R\. M\. Gower, R\. Walters, and R\. Yu\(2024\)Improving convergence and generalization using parameter symmetries\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 55008–55035\.Cited by:[§1\.1](https://arxiv.org/html/2607.18422#S1.SS1.p2.1)\.
- \[14\]B\. Zhao, R\. Walters, and R\. Yu\(2025\)Symmetry in neural network parameter spaces\.arXiv preprint arXiv:2506\.13018\.Cited by:[§1\.1](https://arxiv.org/html/2607.18422#S1.SS1.p2.1)\.
- \[15\]L\. Ziyin, Y\. Xu, T\. Poggio, and I\. Chuang\(2025\)Parameter symmetry potentially unifies deep learning theory\.arXiv preprint arXiv:2502\.05300\.Cited by:[§1\.1](https://arxiv.org/html/2607.18422#S1.SS1.p2.1),[§1](https://arxiv.org/html/2607.18422#S1.p1.15)\.

## Appendix AProofs

This appendix proves the quotient\-risk identities, the KL decomposition along symmetry orbits, and the prior\-comparison identity used in the main text\. The proofs are presented in the order of the corresponding propositions and theorem\.

### A\.1Proposition[1](https://arxiv.org/html/2607.18422#Thmproposition1)

By the parameter\-space symmetry assumption, there exists a measurable predictor familyu↦fuu\\mapsto f^\{u\}such thatfb=fϕ​\(b\)f^\{b\}=f^\{\\phi\(b\)\}for everyb∈ℬ∘b\\in\\mathcal\{B\}^\{\\circ\}\. Hence, for everyb∈ℬ∘b\\in\\mathcal\{B\}^\{\\circ\},

Rb​\(b\)\\displaystyle R^\{b\}\(b\)=𝔼\(X,Y\)∼𝒟​\[ℓ​\(fϕ​\(b\)​\(X\),Y\)\]=Ru​\(ϕ​\(b\)\),\\displaystyle=\\mathbb\{E\}\_\{\(X,Y\)\\sim\\mathcal\{D\}\}\\left\[\\ell\\\!\\left\(f^\{\\phi\(b\)\}\(X\),Y\\right\)\\right\]=R^\{u\}\(\\phi\(b\)\),\(A\.1\)R^Sb​\(b\)\\displaystyle\\widehat\{R\}\_\{S\}^\{b\}\(b\)=1n​∑i=1nℓ​\(fϕ​\(b\)​\(Xi\),Yi\)=R^Su​\(ϕ​\(b\)\)\.\\displaystyle=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\ell\\\!\\left\(f^\{\\phi\(b\)\}\(X\_\{i\}\),Y\_\{i\}\\right\)=\\widehat\{R\}\_\{S\}^\{u\}\(\\phi\(b\)\)\.\(A\.2\)LetQu=ϕ\#​QbQ^\{u\}=\\phi\_\{\\\#\}Q^\{b\}\. For every integrable measurableh:U→ℝh:U\\to\\mathbb\{R\}, the defining property of the pushforward gives

∫Uh​\(u\)​Qu​\(d​u\)=∫ℬ∘h​\(ϕ​\(b\)\)​Qb​\(d​b\)\.\\int\_\{U\}h\(u\)\\,Q^\{u\}\(du\)=\\int\_\{\\mathcal\{B\}^\{\\circ\}\}h\(\\phi\(b\)\)\\,Q^\{b\}\(db\)\.\(A\.3\)Applying this identity toh=Ruh=R^\{u\}andh=R^Suh=\\widehat\{R\}\_\{S\}^\{u\}, and using the identities above, proves the two posterior\-risk equalities\.

### A\.2Proposition[2](https://arxiv.org/html/2607.18422#Thmproposition2)

LetA⊆UA\\subseteq Ube measurable and suppose thatPu​\(A\)=0P^\{u\}\(A\)=0\. SincePu=ϕ\#​PbP^\{u\}=\\phi\_\{\\\#\}P^\{b\},Pb​\(ϕ−1​\(A\)\)=Pu​\(A\)=0P^\{b\}\\\!\\left\(\\phi^\{\-1\}\(A\)\\right\)=P^\{u\}\(A\)=0\. BecauseQb≪PbQ^\{b\}\\ll P^\{b\},Qu​\(A\)=Qb​\(ϕ−1​\(A\)\)=0Q^\{u\}\(A\)=Q^\{b\}\\\!\\left\(\\phi^\{\-1\}\(A\)\\right\)=0\. HenceQu≪PuQ^\{u\}\\ll P^\{u\}\.

Sinceℬ∘\\mathcal\{B\}^\{\\circ\}andUUare standard Borel spaces, and givenu=ϕ​\(b\)u=\\phi\(b\)the disintegrations,

Qb​\(d​b\)=Qb​\(d​b∣u\)​Qu​\(d​u\),\\displaystyle Q^\{b\}\(db\)=Q^\{b\}\(db\\mid u\)\\,Q^\{u\}\(du\),Pb​\(d​b\)=Pb​\(d​b∣u\)​Pu​\(d​u\)\.\\displaystyle P^\{b\}\(db\)=P^\{b\}\(db\\mid u\)\\,P^\{u\}\(du\)\.\(A\.4\)Moreover,Qb≪PbQ^\{b\}\\ll P^\{b\}andQu≪PuQ^\{u\}\\ll P^\{u\}imply thatQb\(⋅∣u\)≪Pb\(⋅∣u\)Q^\{b\}\(\\cdot\\mid u\)\\ll P^\{b\}\(\\cdot\\mid u\)forQuQ^\{u\}\-almost everyuu\.

The Radon–Nikodym derivative therefore factorizesQbQ^\{b\}\-almost surely as

d​Qbd​Pb​\(b\)=d​Qud​Pu​\(u\)​dQb\(⋅∣u\)dPb\(⋅∣u\)​\(b\)\.\\frac\{dQ^\{b\}\}\{dP^\{b\}\}\(b\)=\\frac\{dQ^\{u\}\}\{dP^\{u\}\}\(u\)\\frac\{dQ^\{b\}\(\\cdot\\mid u\)\}\{dP^\{b\}\(\\cdot\\mid u\)\}\(b\)\.\(A\.5\)Taking logarithms and integrating with respect toQbQ^\{b\}, followed by the disintegration ofQbQ^\{b\}, gives the relative\-entropy chain rule

KL⁡\(Qb∥Pb\)\\displaystyle\\operatorname\{KL\}\(Q^\{b\}\\\|P^\{b\}\)=KL⁡\(Qu∥Pu\)\\displaystyle=\\operatorname\{KL\}\(Q^\{u\}\\\|P^\{u\}\)\+∫UKL\(Qb\(⋅∣u\)∥Pb\(⋅∣u\)\)Qu\(du\)\.\\displaystyle\+\\int\_\{U\}\\operatorname\{KL\}\\\!\\left\(Q^\{b\}\(\\cdot\\mid u\)\\,\\middle\\\|\\,P^\{b\}\(\\cdot\\mid u\)\\right\)Q^\{u\}\(du\)\.\(A\.6\)The second term is nonnegative, which proves the claimed inequality\.

### A\.3Theorem[1](https://arxiv.org/html/2607.18422#Thmtheorem1)

SincePbP^\{b\}is independent of the certification sampleSSandϕ\\phiis fixed independently ofSS, its pushforwardPu=ϕ\#​PbP^\{u\}=\\phi\_\{\\\#\}P^\{b\}is also independent ofSS\. ThereforePuP^\{u\}is an admissible PAC–Bayes prior onUU\. Applying the selected PAC–Bayes inequality onUUgives, with probability at least1−δ1\-\\delta,

𝔼u∼Qu​\[Ru\]≤𝔼u∼Qu​\[R^Su\]\+Compn,δ​\(KL⁡\(Qu∥Pu\)\)\.\\mathbb\{E\}\_\{u\\sim Q^\{u\}\}\[R^\{u\}\]\\leq\\mathbb\{E\}\_\{u\\sim Q^\{u\}\}\[\\widehat\{R\}\_\{S\}^\{u\}\]\+\\mathrm\{Comp\}\_\{n,\\delta\}\\\!\\left\(\\operatorname\{KL\}\(Q^\{u\}\\\|P^\{u\}\)\\right\)\.\(A\.7\)By Proposition[1](https://arxiv.org/html/2607.18422#Thmproposition1),

𝔼u∼Qu​\[Ru\]\\displaystyle\\mathbb\{E\}\_\{u\\sim Q^\{u\}\}\[R^\{u\}\]=𝔼b∼Qb​\[Rb\],\\displaystyle=\\mathbb\{E\}\_\{b\\sim Q^\{b\}\}\[R^\{b\}\],𝔼u∼Qu​\[R^Su\]\\displaystyle\\mathbb\{E\}\_\{u\\sim Q^\{u\}\}\[\\widehat\{R\}\_\{S\}^\{u\}\]=𝔼b∼Qb​\[R^Sb\]\.\\displaystyle=\\mathbb\{E\}\_\{b\\sim Q^\{b\}\}\[\\widehat\{R\}\_\{S\}^\{b\}\]\.\(A\.8\)By Proposition[2](https://arxiv.org/html/2607.18422#Thmproposition2),

KL⁡\(Qu∥Pu\)≤KL⁡\(Qb∥Pb\)\.\\operatorname\{KL\}\(Q^\{u\}\\\|P^\{u\}\)\\leq\\operatorname\{KL\}\(Q^\{b\}\\\|P^\{b\}\)\.\(A\.9\)SinceCompn,δ\\mathrm\{Comp\}\_\{n,\\delta\}is nondecreasing,

Compn,δ​\(KL⁡\(Qu∥Pu\)\)≤Compn,δ​\(KL⁡\(Qb∥Pb\)\)\.\\mathrm\{Comp\}\_\{n,\\delta\}\\\!\\left\(\\operatorname\{KL\}\(Q^\{u\}\\\|P^\{u\}\)\\right\)\\leq\\mathrm\{Comp\}\_\{n,\\delta\}\\\!\\left\(\\operatorname\{KL\}\(Q^\{b\}\\\|P^\{b\}\)\\right\)\.\(A\.10\)Combining these identities and inequalities yields

𝔼u∼Qu​\[R^Su\]\+Compn,δ​\(KL⁡\(Qu∥Pu\)\)≤𝔼b∼Qb​\[R^Sb\]\+Compn,δ​\(KL⁡\(Qb∥Pb\)\),\\displaystyle\\begin\{aligned\} &\\mathbb\{E\}\_\{u\\sim Q^\{u\}\}\[\\widehat\{R\}\_\{S\}^\{u\}\]\+\\mathrm\{Comp\}\_\{n,\\delta\}\\\!\\left\(\\operatorname\{KL\}\(Q^\{u\}\\\|P^\{u\}\)\\right\)\\\\ &\\qquad\\leq\\mathbb\{E\}\_\{b\\sim Q^\{b\}\}\[\\widehat\{R\}\_\{S\}^\{b\}\]\+\\mathrm\{Comp\}\_\{n,\\delta\}\\\!\\left\(\\operatorname\{KL\}\(Q^\{b\}\\\|P^\{b\}\)\\right\),\\end\{aligned\}\(A\.11\)which proves the theorem\.

### A\.4Proposition[3](https://arxiv.org/html/2607.18422#Thmproposition3)

By construction,

d​PIBud​P0u​\(u\)=e−Lχ​\(u\)Zχ\.\\frac\{dP^\{u\}\_\{\\mathrm\{IB\}\}\}\{dP^\{u\}\_\{0\}\}\(u\)=\\frac\{e^\{\-L\_\{\\chi\}\(u\)\}\}\{Z\_\{\\chi\}\}\.\(A\.12\)SinceLχL\_\{\\chi\}is finiteP0uP^\{u\}\_\{0\}\-almost everywhere and0<Zχ<∞0<Z\_\{\\chi\}<\\infty, this density is strictly positiveP0uP^\{u\}\_\{0\}\-almost everywhere\. HenceQu≪P0uQ^\{u\}\\ll P^\{u\}\_\{0\}impliesQu≪PIBuQ^\{u\}\\ll P^\{u\}\_\{\\mathrm\{IB\}\}\. Moreover,

d​P0ud​PIBu​\(u\)=Zχ​eLχ​\(u\)\.\\frac\{dP^\{u\}\_\{0\}\}\{dP^\{u\}\_\{\\mathrm\{IB\}\}\}\(u\)=Z\_\{\\chi\}e^\{L\_\{\\chi\}\(u\)\}\.\(A\.13\)Therefore,

d​Qud​PIBu​\(u\)=d​Qud​P0u​\(u\)​d​P0ud​PIBu​\(u\),\\frac\{dQ^\{u\}\}\{dP^\{u\}\_\{\\mathrm\{IB\}\}\}\(u\)=\\frac\{dQ^\{u\}\}\{dP^\{u\}\_\{0\}\}\(u\)\\frac\{dP^\{u\}\_\{0\}\}\{dP^\{u\}\_\{\\mathrm\{IB\}\}\}\(u\),\(A\.14\)and hence

log⁡d​Qud​PIBu​\(u\)=log⁡d​Qud​P0u​\(u\)\+Lχ​\(u\)\+log⁡Zχ\.\\log\\frac\{dQ^\{u\}\}\{dP^\{u\}\_\{\\mathrm\{IB\}\}\}\(u\)=\\log\\frac\{dQ^\{u\}\}\{dP^\{u\}\_\{0\}\}\(u\)\+L\_\{\\chi\}\(u\)\+\\log Z\_\{\\chi\}\.\(A\.15\)The assumptionsKL⁡\(Qu∥P0u\)<∞\\operatorname\{KL\}\(Q^\{u\}\\\|P^\{u\}\_\{0\}\)<\\inftyand𝔼u∼Qu​\[\|Lχ\|\]<∞\\mathbb\{E\}\_\{u\\sim Q^\{u\}\}\[\|L\_\{\\chi\}\|\]<\\inftypermit integration with respect toQuQ^\{u\}\. Thus,

KL⁡\(Qu∥PIBu\)=KL⁡\(Qu∥P0u\)\+𝔼u∼Qu​\[Lχ\]\+log⁡Zχ\.\\operatorname\{KL\}\(Q^\{u\}\\\|P^\{u\}\_\{\\mathrm\{IB\}\}\)=\\operatorname\{KL\}\(Q^\{u\}\\\|P^\{u\}\_\{0\}\)\+\\mathbb\{E\}\_\{u\\sim Q^\{u\}\}\[L\_\{\\chi\}\]\+\\log Z\_\{\\chi\}\.\(A\.16\)Rearranging gives

Δ​KL⁡\(Qu\)=𝔼u∼Qu​\[Lχ\]\+log⁡Zχ\.\\Delta\\operatorname\{KL\}\(Q^\{u\}\)=\\mathbb\{E\}\_\{u\\sim Q^\{u\}\}\[L\_\{\\chi\}\]\+\\log Z\_\{\\chi\}\.\(A\.17\)For a fixed quotient posteriorQuQ^\{u\}, replacingP0uP^\{u\}\_\{0\}byPIBuP^\{u\}\_\{\\mathrm\{IB\}\}does not change either the empirical or the population Gibbs risk\. IfΔ​KL⁡\(Qu\)≤0\\Delta\\operatorname\{KL\}\(Q^\{u\}\)\\leq 0, then

KL⁡\(Qu∥PIBu\)≤KL⁡\(Qu∥P0u\)\.\\operatorname\{KL\}\(Q^\{u\}\\\|P^\{u\}\_\{\\mathrm\{IB\}\}\)\\leq\\operatorname\{KL\}\(Q^\{u\}\\\|P^\{u\}\_\{0\}\)\.\(A\.18\)Monotonicity of the PAC–Bayes complexity term then yields the stated comparison of certificate expressions\.

## Appendix BExperimental details

### B\.1Posterior construction and certification protocol

For retained trajectory vectorsz\(1\),…,z\(T\)∈ℝpz^\{\(1\)\},\\ldots,z^\{\(T\)\}\\in\\mathbb\{R\}^\{p\}, we fit

Qz\\displaystyle Q^\{z\}=𝒩​\(z¯,s^Q2​Ip\),\\displaystyle=\\mathcal\{N\}\(\\bar\{z\},\\widehat\{s\}\_\{Q\}^\{2\}I\_\{p\}\),\(B\.1\)z¯\\displaystyle\\bar\{z\}=1T​∑t=1Tz\(t\),\\displaystyle=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}z^\{\(t\)\},\(B\.2\)s^Q2\\displaystyle\\widehat\{s\}\_\{Q\}^\{2\}=1p​\(T−1\)​∑t=1T∥z\(t\)−z¯∥22\.\\displaystyle=\\frac\{1\}\{p\(T\-1\)\}\\sum\_\{t=1\}^\{T\}\\lVert z^\{\(t\)\}\-\\bar\{z\}\\rVert\_\{2\}^\{2\}\.\(B\.3\)The fitted posterior is constructed separately for every run\. The retained SGD iterates are correlated trajectory observations and are used only to estimatez¯\\bar\{z\}ands^Q2\\widehat\{s\}\_\{Q\}^\{2\}\.

For each tested scales0s\_\{0\}, the neutral prior isP0z,\(s0\)=𝒩​\(0,s02​Ip\)P\_\{0\}^\{z,\(s\_\{0\}\)\}=\\mathcal\{N\}\(0,s\_\{0\}^\{2\}I\_\{p\}\)\. Its KL divergence fromQzQ^\{z\}is evaluated analytically\. Both experiments use the clipped squared certification loss

ℓM​\(y^,y\)=min⁡\{\(y^−y\)2M2,1\},M=2,\\ell\_\{M\}\(\\widehat\{y\},y\)=\\min\\left\\\{\\frac\{\(\\widehat\{y\}\-y\)^\{2\}\}\{M^\{2\}\},1\\right\\\},\\qquad M=2,\(B\.4\)an independent certification sample of size20,00020\{,\}000, confidence levelδ=0\.05\\delta=0\.05, and8,1928\{,\}192draws from the fitted posterior to estimate the empirical Gibbs risk\. For a fixed run, changings0s\_\{0\}changes the two priors but not the fitted posterior distribution or its empirical Gibbs risk\.

### B\.2Fourier–Hadamard implicit\-bias loss

For both models, the posterior is by the weightsw∈ℝ64w\\in\\mathbb\{R\}^\{64\}\. It is fitted fromT=51T=51SGD states retained every200200updates between updates60,00060\{,\}000and70,00070\{,\}000\. For the Hadamard model, the retained state isw=a⊙cw=a\\odot c, rather than the redundant pair\(a,c\)\(a,c\)\.

The regularized implicit\-bias loss forε=10−3\\varepsilon=10^\{\-3\}is

Lχ,ε​\(w\)=12​∑i=164log⁡\(\|wi\|\+ε\)\.L\_\{\\chi,\\varepsilon\}\(w\)=\\frac\{1\}\{2\}\\sum\_\{i=1\}^\{64\}\\log\(\|w\_\{i\}\|\+\\varepsilon\)\.\(B\.5\)The corresponding implicit\-bias prior is

PIB,εw,\(s0\)​\(d​w\)=e−Lχ,ε​\(w\)ZIB,ε​\(s0\)​P0w,\(s0\)​\(d​w\)\.P\_\{\\mathrm\{IB\},\\varepsilon\}^\{w,\(s\_\{0\}\)\}\(dw\)=\\frac\{e^\{\-L\_\{\\chi,\\varepsilon\}\(w\)\}\}\{Z\_\{\\mathrm\{IB\},\\varepsilon\}\(s\_\{0\}\)\}P\_\{0\}^\{w,\(s\_\{0\}\)\}\(dw\)\.\(B\.6\)Because the Gaussian reference prior and the implicit\-bias loss factorize over weights, the normalizer reduces to a one\-dimensional integral\. This integral is approximated by trapezoidal quadrature on\[−160,160\]\[\-160,160\]using800,001800\{,\}001equally spaced points\. The expectation𝔼Q​\[Lχ,ε\]\\mathbb\{E\}\_\{Q\}\[L\_\{\\chi,\\varepsilon\}\]is estimated with8,1928\{,\}192posterior draws\.

### B\.3Query–Key implicit\-bias loss

For the query–key model, the weights arez=vec⁡\(WQ,WK,WV\)∈ℝ384z=\\operatorname\{vec\}\(W\_\{Q\},W\_\{K\},W\_\{V\}\)\\in\\mathbb\{R\}^\{384\}\. The posterior is fitted fromT=201T=201balanced trajectory vectors retained every200200updates between updates60,00060\{,\}000and100,000100\{,\}000\. Each vectorz∈ℝ384z\\in\\mathbb\{R\}^\{384\}contains the balanced Query and Key matrices together withWVW\_\{V\}\.

To define the balanced vector, letqjq\_\{j\}andkjk\_\{j\}denote thejj\-th columns ofWQW\_\{Q\}andWKW\_\{K\}, respectively\. For nonzero columns, define

q¯j=qj​‖kj‖2‖qj‖2,k¯j=kj​‖qj‖2‖kj‖2,j=1,…,dh\.\\bar\{q\}\_\{j\}=q\_\{j\}\\sqrt\{\\frac\{\\\|k\_\{j\}\\\|\_\{2\}\}\{\\\|q\_\{j\}\\\|\_\{2\}\}\},\\qquad\\bar\{k\}\_\{j\}=k\_\{j\}\\sqrt\{\\frac\{\\\|q\_\{j\}\\\|\_\{2\}\}\{\\\|k\_\{j\}\\\|\_\{2\}\}\},\\qquad j=1,\\ldots,d\_\{h\}\.\(B\.7\)This transformation preserves the Query–Key predictor and satisfies the balanced condition\[[1](https://arxiv.org/html/2607.18422#bib.bib1)\],

‖q¯j‖2=‖k¯j‖2\.\\\|\\bar\{q\}\_\{j\}\\\|\_\{2\}=\\\|\\bar\{k\}\_\{j\}\\\|\_\{2\}\.\(B\.8\)We then define

bal⁡\(z\)=vec⁡\(W¯Q,W¯K,WV\),\\operatorname\{bal\}\(z\)=\\operatorname\{vec\}\(\\bar\{W\}\_\{Q\},\\bar\{W\}\_\{K\},W\_\{V\}\),\(B\.9\)whereW¯Q\\bar\{W\}\_\{Q\}andW¯K\\bar\{W\}\_\{K\}have columnsq¯j\\bar\{q\}\_\{j\}andk¯j\\bar\{k\}\_\{j\}\. The corresponding regularized implicit\-bias loss, forε=10−3\\varepsilon=10^\{\-3\}is

Lχ,ε​\(bal⁡\(z\)\)=12​∑j=1dhlog⁡\(‖q¯j‖22\+‖k¯j‖22\+ε\)\.L\_\{\\chi,\\varepsilon\}\(\\operatorname\{bal\}\(z\)\)=\\frac\{1\}\{2\}\\sum\_\{j=1\}^\{d\_\{h\}\}\\log\\\!\\left\(\\\|\\bar\{q\}\_\{j\}\\\|\_\{2\}^\{2\}\+\\\|\\bar\{k\}\_\{j\}\\\|\_\{2\}^\{2\}\+\\varepsilon\\right\)\.\(B\.10\)
The expectation of this loss underQzQ^\{z\}is estimated using8,1928\{,\}192posterior draws\. The normalizing constantZIB​\(s0\)Z\_\{\\mathrm\{IB\}\}\(s\_\{0\}\)is estimated using16,38416\{,\}384independent draws from the neutral priorP0z,\(s0\)P\_\{0\}^\{z,\(s\_\{0\}\)\}\.