Population loss in shallow ReLU networks: Bias & families of critical points

arXiv cs.LG Papers

Summary

This paper derives a formula for population loss in shallow ReLU networks with bias, showing that bias strictly decreases loss and extends known families of critical points.

arXiv:2609.30661v1 Announce Type: new Abstract: The main result presented is a formula for the population loss in the student-teacher kernel model that is applicable to shallow ReLU networks with bias. This extends previous work of Choo and Saul (2009) and Brutzkus and Globerson (2017). The formula makes essential use of Owen's T-function. The necessary theory of the T-function is given and a high precision coding using MPFR for the T-function, based on an algorithm of Komelj (2023), is available on request. It is shown that various families of spurious minima described in past papers of Arjevani and the author extend to biased networks and that the loss is always strictly decreased when bias is added. The change in landscape geometry caused by adding bias appears to be relatively mild. Only the simplest examples are described in this paper where it is assumed that the number of inputs is equal to the number of neurons (this restriction is for reasons of length). A review of relevant previous results on unbiased networks is included. Aside from Gaussian statistics, the main mathematical tools and ideas come from analytic geometry (analytic and subanalytic sets, the Curve Selection Lemma).
Original Article
View Cached Full Text

Cached at: 09/29/26, 09:42 AM

# Population loss in shallow ReLU networks:Bias & families of critical points
Source: [https://arxiv.org/html/2609.30661](https://arxiv.org/html/2609.30661)
Michael FieldAddress:Michael Field, Department of Mechanical Engineering, UC Santa Barbara, CA 93106Email address:[mikefield@gmail\.com](mailto:[email protected])

Date: September 25, 2026

###### Abstract\.

The main result presented is a formula for the population loss in the student\-teacher kernel model that is applicable to shallow ReLU networks with bias\. This extends previous work of Choo & Saul \(2009\) and Brutzkus and Globerson \(2017\)\. The formula makes essential use of Owen’sTT\-function\. The necessary theory of theTTfunction is given and a high precision coding using*MPFR*for theTTfunction, based on an algorithm of Komelj \(2023\), is available on request\. It is shown that various families of spurious minima described in past papers of Arjevani and the author extend to biased networks and that the loss is always strictly decreased when bias is added\. The change in landscape geometry caused by adding bias appears to be relatively mild\. Only the simplest examples are described in this paper where it is assumed that the number of inputs is equal to the number of neurons \(this restriction is for reasons of length\)\. A review of relevant previous results on unbiased networks is included\. Aside from Gaussian statistics, the main mathematical tools and ideas come from analytic geometry \(analytic and subanalytic sets, the Curve Selection Lemma\)\.

###### Contents

1. [1Introduction](https://arxiv.org/html/2609.30661#S1)
2. [2Background on neural networks and machine learning](https://arxiv.org/html/2609.30661#S2)1. [2\.1Shallow Neural Networks](https://arxiv.org/html/2609.30661#S2.SS1) 2. [2\.2Statistical learning and a reduction](https://arxiv.org/html/2609.30661#S2.SS2)
3. [3Bias](https://arxiv.org/html/2609.30661#S3)1. [3\.1Computation offfif the network is not biased⋆](https://arxiv.org/html/2609.30661#S3.SS1) 2. [3\.2Evaluation of the defining integrals forf⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\): I](https://arxiv.org/html/2609.30661#S3.SS2) 3. [3\.3Evaluation of the defining integrals forf⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\): II](https://arxiv.org/html/2609.30661#S3.SS3)
4. [4The OwenTT\-function](https://arxiv.org/html/2609.30661#S4)1. [4\.1The Generalized OwenTT\-function\[21\]](https://arxiv.org/html/2609.30661#S4.SS1)
5. [5Evaluation of the defining integrals forf⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\): III](https://arxiv.org/html/2609.30661#S5)
6. [6Evaluation off⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)concluded](https://arxiv.org/html/2609.30661#S6)1. [The constant termβ¯​γ¯​∫D1\\overline\{\\beta\}\\overline\{\\gamma\}\\int\_\{D\}1](https://arxiv.org/html/2609.30661#S6.SS0.SSSx1) 2. [The linear term−∫D\(γ¯cos\(θ\)\+β¯\)x1\+γ¯sin\(θ\)x2\-\\int\_\{D\}\(\\overline\{\\gamma\}\\cos\(\\theta\)\+\\overline\{\\beta\}\)x\_\{1\}\+\\overline\{\\gamma\}\\sin\(\\theta\)x\_\{2\}](https://arxiv.org/html/2609.30661#S6.SS0.SSSx2) 3. [The first quadratic term∫Dsin⁡\(θ\)​x1​x2\\int\_\{D\}\\sin\(\\theta\)x\_\{1\}x\_\{2\}](https://arxiv.org/html/2609.30661#S6.SS0.SSSx3) 4. [The second quadratic term∫Dcos⁡\(θ\)​x12\\int\_\{D\}\\cos\(\\theta\)x\_\{1\}^\{2\}](https://arxiv.org/html/2609.30661#S6.SS0.SSSx4) 5. [The formula forf⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)](https://arxiv.org/html/2609.30661#S6.SS0.SSSx5) 6. [6\.1Notational conventions & formulas forf⁡\(𝐰i,𝐯j\),f⁡\(𝐰i,𝐰j\)f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{j\}\),f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{j\}\)](https://arxiv.org/html/2609.30661#S6.SS1)
7. [7Gradient off⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)with respect to\(𝐰,β¯\)\(\{\{\\mathbf\{w\}\}\},\\overline\{\\beta\}\): I](https://arxiv.org/html/2609.30661#S7)1. [Notational conventions](https://arxiv.org/html/2609.30661#S7.SS0.SSSx1) 2. [Some frequently used derivatives](https://arxiv.org/html/2609.30661#S7.SS0.SSSx2) 3. [7\.1Derivatives ofH⁡\(β¯,γ¯,λ\)H\(\\overline\{\\beta\},\\overline\{\\gamma\},\\lambda\)andH⁡\(β¯i,β¯j,Λi​j\)H\(\\overline\{\\beta\}\_\{i\},\\overline\{\\beta\}\_\{j\},\\Lambda\_\{ij\}\),i≠ji\\neq j](https://arxiv.org/html/2609.30661#S7.SS1)
8. [8Computation of∂f∂β¯\\frac\{\\partial f\}\{\\partial\\overline\{\\beta\}\}](https://arxiv.org/html/2609.30661#S8)
9. [9Computation of∂f∂𝐰\\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\}\(normalized biasβ¯\\overline\{\\beta\}\)](https://arxiv.org/html/2609.30661#S9)1. [9\.1The formula for∂f∂𝐰​\(𝐰,𝐯\)\\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\}\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)\(normalized biasβ¯\\overline\{\\beta\}\)](https://arxiv.org/html/2609.30661#S9.SS1) 2. [9\.2Angles close to00andπ\\pi](https://arxiv.org/html/2609.30661#S9.SS2) 3. [9\.3Computation of∂f∂𝐰\\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\}\(biasβ\\beta\) I: derivatives](https://arxiv.org/html/2609.30661#S9.SS3) 4. [9\.4The computation of∂f∂𝐰\\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\}\(biasOPENβ\)\\beta\): II](https://arxiv.org/html/2609.30661#S9.SS4)
10. [10The formula for∂ℒ∂𝐰\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\{\{\\mathbf\{w\}\}\}\}\(no symmetry assumption\)](https://arxiv.org/html/2609.30661#S10)1. [10\.1Table defining terms used in the formula forgrad​\(ℒ\)​\(𝐖\)\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}\(\{\{\\mathbf\{W\}\}\}\)](https://arxiv.org/html/2609.30661#S10.SS1) 2. [10\.2The terms𝒰i​j,…,𝒴i​j\\mathcal\{U\}\_\{ij\},\\ldots,\\mathcal\{Y\}\_\{ij\}](https://arxiv.org/html/2609.30661#S10.SS2)
11. [11The formula for∂ℒ∂β¯\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\}](https://arxiv.org/html/2609.30661#S11)1. [11\.1The𝜷¯\\boldsymbol\{\\overline\{\\beta\}\}\-derivative ofℒ\\mathcal\{L\}at𝜷¯=𝟎\\boldsymbol\{\\overline\{\\beta\}\}=\\mathbf\{0\}](https://arxiv.org/html/2609.30661#S11.SS1)
12. [12Symmetry andgrad​\(ℒ\)\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}in a network with bias](https://arxiv.org/html/2609.30661#S12)1. [12\.1Symmetries of the lossℒ\\mathcal\{L\}in a biased network](https://arxiv.org/html/2609.30661#S12.SS1) 2. [12\.2Isotropy groups of critical points when𝐕=Id\{\{\\mathbf\{V\}\}\}=I\_\{d\}](https://arxiv.org/html/2609.30661#S12.SS2) 3. [12\.3Families of subgroups ofΔ​Sd⊂Skr×Sdc\\Delta S\_\{d\}\\subset S^\{r\}\_\{k\}\\times S^\{c\}\_\{d\}](https://arxiv.org/html/2609.30661#S12.SS3)1. [Review of notation and conventions](https://arxiv.org/html/2609.30661#S12.SS3.SSSx1) 2. [Sketch of the proof of Lemma](https://arxiv.org/html/2609.30661#S12.SS3.SSSx2) 3. [Properties of rowtypes](https://arxiv.org/html/2609.30661#S12.SS3.SSSx3) 4. [12\.4Coordinatization ofM​\(k,d\)G𝐪ΔM\(k,d\)^\{G\_\{\\mathbf\{q\}\}^\{\\Delta\}\}&M​\(k,d\+1\)G𝐪ΔM\(k,d\+1\)^\{G\_\{\\mathbf\{q\}\}^\{\\Delta\}\}](https://arxiv.org/html/2609.30661#S12.SS4) 5. [12\.5Critical point equations forgrad​\(ℒ\)\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}onM​\(k,d\+1\)G𝐪ΔM\(k,d\+1\)^\{G^\{\\Delta\}\_\{\\mathbf\{q\}\}\}](https://arxiv.org/html/2609.30661#S12.SS5) 6. [12\.6Critical point equations associated to rowtypei∉𝐫𝐬i\\notin\{\{\\mathbf\{r\_\{s\}\}\}\}](https://arxiv.org/html/2609.30661#S12.SS6) 7. [12\.7Critical point equations associated to rowtypei∈𝐫𝐬i\\in\{\{\\mathbf\{r\_\{s\}\}\}\}](https://arxiv.org/html/2609.30661#S12.SS7) 8. [12\.8The terms𝐀i,𝐁i\{\\mathbf\{A\}\}^\{i\},\{\\mathbf\{B\}\}^\{i\}wheni∈𝐫𝐬i\\in\{\{\\mathbf\{r\_\{s\}\}\}\}and𝐕=Id\{\{\\mathbf\{V\}\}\}=I\_\{d\}](https://arxiv.org/html/2609.30661#S12.SS8) 9. [12\.9Population loss with symmetric target](https://arxiv.org/html/2609.30661#S12.SS9)1. [Termsf⁡\(𝐰,𝐰′\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{w\}\}\}^\{\\prime\}\)](https://arxiv.org/html/2609.30661#S12.SS9.SSSx1) 2. [Termsf⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)](https://arxiv.org/html/2609.30661#S12.SS9.SSSx2) 3. [Termsf⁡\(𝐯,𝐯′\)f\(\{\{\\mathbf\{v\}\}\},\{\{\\mathbf\{v\}\}\}^\{\\prime\}\)](https://arxiv.org/html/2609.30661#S12.SS9.SSSx3) 10. [12\.10Derivatives ofℒ\\mathcal\{L\}with respect to𝜷¯\{\\boldsymbol\{\\overline\{\\beta\}\}\}, indexed by rowtype](https://arxiv.org/html/2609.30661#S12.SS10)
13. [13Review of results for symmetric targetIdI\_\{d\}](https://arxiv.org/html/2609.30661#S13)1. [13\.1Fixed point spaces and fractional power series](https://arxiv.org/html/2609.30661#S13.SS1)1. [Discussion](https://arxiv.org/html/2609.30661#S13.SS1.SSSx1) 2. [13\.2A family of spurious minima with isotropyΔ​Sd\\Delta S\_\{d\},k=dk=d](https://arxiv.org/html/2609.30661#S13.SS2) 3. [13\.3Fractional power series \(FPS\): types I and II](https://arxiv.org/html/2609.30661#S13.SS3)
14. [14Computing∂ℒ∂β¯​\(𝐖,𝟎\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\}\(\{\{\\mathbf\{W\}\}\},\\mathbf\{0\}\): symmetric & asymmetric targets](https://arxiv.org/html/2609.30661#S14)1. [14\.1Computation of∂ℒ∂𝜷¯​\(𝐖,𝟎\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\boldsymbol\{\\overline\{\\beta\}\}\}\(\{\{\\mathbf\{W\}\}\},\\mathbf\{0\}\): a symmetric example](https://arxiv.org/html/2609.30661#S14.SS1)1. [Inner products of𝐰0\{\{\\mathbf\{w\}\}\}^\{0\}with𝐰i\{\{\\mathbf\{w\}\}\}\_\{i\}and𝐯j\{\{\\mathbf\{v\}\}\}^\{j\},j∈𝐝j\\in\{\\mathbf\{d\}\}](https://arxiv.org/html/2609.30661#S14.SS1.SSSx1) 2. [Computation of∂ℒ∂β¯0​\(𝔠​\(d\),𝟎\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{0\}\}\(\\mathfrak\{c\}\(d\),\\mathbf\{0\}\)for specific values ofdd](https://arxiv.org/html/2609.30661#S14.SS1.SSSx2) 2. [14\.2Results for isotropyΔ​Sd\\Delta S\_\{d\}, type I \(type A in\[4\]\)](https://arxiv.org/html/2609.30661#S14.SS2) 3. [14\.3Results for isotropyΔ​Sd−1\\Delta S\_\{d\-1\}, type I,m=0,1,2m=0,1,2](https://arxiv.org/html/2609.30661#S14.SS3) 4. [14\.4Results for isotropyΔ​Sd−1\\Delta S\_\{d\-1\}, type II,m=0,1,2m=0,1,2](https://arxiv.org/html/2609.30661#S14.SS4) 5. [14\.5Results for isotropyGd−2,2Δ=Δ⁡\(Sd−2×S2\)G^\{\\Delta\}\_\{d\-2,2\}=\\Delta\(S\_\{d\-2\}\\times S\_\{2\}\), type I,m=0,1,2m=0,1,2](https://arxiv.org/html/2609.30661#S14.SS5) 6. [14\.6Results for isotropyΔ⁡\(Sd−2×S2\)\\Delta\(S\_\{d\-2\}\\times S\_\{2\}\), type II,m=0,1m=0,1](https://arxiv.org/html/2609.30661#S14.SS6) 7. [14\.7Results for isotropyΔ⁡\(Sd−p×Sp\)\\Delta\(S\_\{d\-p\}\\times S\_\{p\}\), type II,p\>2p\>2](https://arxiv.org/html/2609.30661#S14.SS7) 8. [14\.8Asymmetric targets and fractional power series](https://arxiv.org/html/2609.30661#S14.SS8)1. [Path following deformations of the target](https://arxiv.org/html/2609.30661#S14.SS8.SSSx1)
15. [15Families of spurious minima in biased ReLU networks](https://arxiv.org/html/2609.30661#S15)1. [15\.1IsotropyΔ​Sd\\Delta S\_\{d\},d=kd=k: Type I in a network with bias](https://arxiv.org/html/2609.30661#S15.SS1) 2. [15\.2Numerics](https://arxiv.org/html/2609.30661#S15.SS2)1. [The loss](https://arxiv.org/html/2609.30661#S15.SS2.SSSx1) 3. [15\.3Path based analysis](https://arxiv.org/html/2609.30661#S15.SS3) 4. [15\.4IsotropyΔ⁡\(Sd−1×S1\)\\Delta\(S\_\{d\-1\}\\times S\_\{1\}\),q1=1q\_\{1\}=1,d=kd=k, Types I and II](https://arxiv.org/html/2609.30661#S15.SS4)1. [Type I family of spurious with isotropyΔ⁡\(Sd−1×S1\)=defΔ​Sd−1\\Delta\(S\_\{d\-1\}\\times S\_\{1\}\)\\stackrel\{\{\\scriptstyle\\mathrm\{def\}\}\}\{\{=\}\}\\Delta S\_\{d\-1\}](https://arxiv.org/html/2609.30661#S15.SS4.SSSx1) 5. [15\.5IsotropyΔ⁡\(Sd−2×S2\)\\Delta\(S\_\{d\-2\}\\times S\_\{2\}\),q1=2q\_\{1\}=2,d=kd=k, Types I and II](https://arxiv.org/html/2609.30661#S15.SS5)1. [Type I family of spurious minima with isotropyΔ⁡\(Sd−2×S2\)\\Delta\(S\_\{d\-2\}\\times S\_\{2\}\)](https://arxiv.org/html/2609.30661#S15.SS5.SSSx1) 2. [Type II family of spurious minima with isotropyΔ⁡\(Sd−2×S2\)\\Delta\(S\_\{d\-2\}\\times S\_\{2\}\)](https://arxiv.org/html/2609.30661#S15.SS5.SSSx2) 6. [15\.6IsotropyΔ⁡\(Sd−3×S3\)\\Delta\(S\_\{d\-3\}\\times S\_\{3\}\),q1=3q\_\{1\}=3,d=kd=k, type II](https://arxiv.org/html/2609.30661#S15.SS6) 7. [15\.7Comments on numerical methods: comparison with bias free networks](https://arxiv.org/html/2609.30661#S15.SS7)
16. [AReview of the error functionerf​\(x\)\{\\text\{\\rm erf\}\(\{x\}\)\}](https://arxiv.org/html/2609.30661#A1)
17. [BHigh precision computation of Owen’sTT\-function](https://arxiv.org/html/2609.30661#A2)
18. [CGeneralizedTT\-function](https://arxiv.org/html/2609.30661#A3)
19. [References](https://arxiv.org/html/2609.30661#bib)

## 1\.Introduction

In a past collaboration with Yossi Arjevani, a detailed analysis was made of families of spurious minima that appeared in a bias\-free shallow student\-teacher ReLU network with the identity matrix as target\[[1](https://arxiv.org/html/2609.30661#bib.bib1),[2](https://arxiv.org/html/2609.30661#bib.bib2),[3](https://arxiv.org/html/2609.30661#bib.bib3),[4](https://arxiv.org/html/2609.30661#bib.bib4),[5](https://arxiv.org/html/2609.30661#bib.bib5),[6](https://arxiv.org/html/2609.30661#bib.bib6)\]\. The investigation, initiated by Arjevani, was motivated in part by earlier work of Safran & Shamir\[[22](https://arxiv.org/html/2609.30661#bib.bib22)\]who undertook a numerical investigation of spurious minima in low dimensional bias free ReLU networks withd≤20d\\leq 20inputs,kkneurons,d≤k≤d\+2d\\leq k\\leq d\+2, and the identity matrixIdI\_\{d\}as target\. The authors remarked that the spurious minima they found always had a symmetric structure \(formally, after a permutation of rows, the solutions were fixed by a subgroup of the symmetric groupSdS\_\{d\}acting diagonally on the rows and columns of the parameter matrix𝐖∈M⁡\(d,d\)\{\{\\mathbf\{W\}\}\}\\in M\(d,d\)\)\. As Arjevani observed, the symmetric structure was associated to group invariance properties of the loss inherited from the symmetry of the target\.

Using this symmetric structure, one can apply representation theory of the symmetric group to determine the spectrum of the Hessian; in particular, the asymptotics asd→∞d\{\\rightarrow\}\\infty\[[2](https://arxiv.org/html/2609.30661#bib.bib2),[5](https://arxiv.org/html/2609.30661#bib.bib5)\]\. These results made essential use of a result in\[[4](https://arxiv.org/html/2609.30661#bib.bib4)\]that each family of spurious minima had \(computable\) fractional power series expansions ind−12d^\{\-\\frac\{1\}\{2\}\}ord−14d^\{\-\\frac\{1\}\{4\}\}\(the latter only if the number of neuronskkwas greater thandd\)\. Along similar lines, it was possible to give precise asymptotics for the loss and show that there were ‘good’ \(type II\) and ‘bad’ \(type I\) spurious minima\. The type II spurious minima had rapid decay of the loss to zero, while for the type I families the loss converged to a strictly positive constant asd→∞d\{\\rightarrow\}\\infty\. Some of this work is reviewed in more detail in Section[13](https://arxiv.org/html/2609.30661#S13)\.

Taking account of the applications, the discussion of symmetry breaking will be in the framework of gradient dynamics and bifurcation\. Specifically by gradient dynamics of the loss in the student\-teacher model and the creation or annihilation of local minima\. Given a symmetric model, there are two different types of symmetry breaking that can occur\. If the symmetry of the model is fixed, then as model parameters are varied new local minima may appear typically with less symmetry\. This is characteristic of*symmetry breaking*\. In symmetric bifurcation theory this phenomenon occurs when a group orbit of high symmetry critical points defining, for example, local minima, bifurcates into a group orbit of critical points defining local minima but each with less symmetry\. However, this does*not*happen in an obvious way in our setting: the global minima do not bifurcate\. Instead, new \(local\) minima appear through, for example, bifurcation on subspace representations associated to the symmetric group on the parameter space \(see\[[6](https://arxiv.org/html/2609.30661#bib.bib6), §4\]for examples\)\. Basically, this is bifurcation from a saddle to a local minimum and is characteristic of all the examples described later in Section 15\.

On the other hand,*forced*symmetry breaking occurs when the symmetry of the*model*is reduced, for example by a generic perturbation of the targetIdI\_\{d\}\. This will sharply reduce the number of critical points defining the*global*minimum with similar effects on all the local minima\. As long as the original critical points defining local minima are non\-degenerate and the perturbation is not too large, the perturbed critical points will continue to define local minima\.

Because the critical points defining local minima in the symmetric case are typically non\-degenerate for sufficiently largedd, the spectral properties persist under perturbations of the target matrixIdI\_\{d\}\. More formally, pick ap×pp\\times pmatrixAAwhich is not too far from the identityIpI\_\{p\}and ford≥pd\\geq p, embedAAas the lowerp×pp\\times pdiagonal block inIdI\_\{d\}to obtain a ‘target’ matrixTTwhich has less symmetry thanIdI\_\{d\}\(typicallySd−pS\_\{d\-p\}under the diagonal action on the upperd−pd\-pdiagonal block\)\. In this setup, basic results such as the existence of fractional power series for families of critical points, carry over to the partially asymmetric case\. Computation is straightforward using standard path following techniques\.

However, there is a problem\. As pointed out in the recent paper by Zhang*et al\.*\[[24](https://arxiv.org/html/2609.30661#bib.bib24)\], the space of functions that can be modelled by a bias free ReLU network is always a*proper*closed vector subspace of the space of continuous functions111This result is presumably well known\[[20](https://arxiv.org/html/2609.30661#bib.bib20)\]—and is not the main focus of the paper by Zhang*et al\.*—but reading their preprint was the trigger for this paper\.\. In summary, without further work, no inferences can be drawn about stability in a biased ReLU network under forced symmetry breaking on the basis of results on bias free ReLU networks\. The aim of this paper is to develop the appropriate tools that show that the examples found previously for bias free networks do carry over to biased networks\. But the correspondence is non\-trivial since a local minimum of a bias free network is generally not a critical point of a biased network, with zero bias\.

So as to keep the paper to a reasonable length, the focus will be on examples withk=dk=d\(not ‘over\-specified’ in the terminology of\[[5](https://arxiv.org/html/2609.30661#bib.bib5)\]\) and there are no computations relating to the Hessian spectrum\.

###### Remarks 1\.1\.

\(1\) Adding bias to the network adds strong constraints on the existence of families of spurious minima\. This is on account of multiple terms in the formula for the loss which contain the expressions−γ¯​cot⁡\(λ\)\+β¯​csc⁡\(λ\)\-\\overline\{\\gamma\}\\cot\(\\lambda\)\+\\overline\{\\beta\}\\csc\(\\lambda\), whereγ¯,β¯\\overline\{\\gamma\},\\overline\{\\beta\}are respectively \(normalized\) biases for the teacher and student network andλ\\lambdais the angle between the corresponding rows of the target and parameter matrix\. Restricting to the casek=dk=d, the typical behavior asd→∞d\{\\rightarrow\}\\inftyis that every row of the parameter matrix converges to±\\pma row of the target \(this holds for targets in a neighbourhood ofIdI\_\{d\}\)\. Sincecot⁡\(λ\)→±∞\\cot\(\\lambda\)\{\\rightarrow\}\\pm\\inftyandcsc⁡\(λ\)→\+∞\\csc\(\\lambda\)\{\\rightarrow\}\+\\inftyasθ→0\+\\theta\{\\rightarrow\}0\+orπ−\\pi\-, this imposes strong constraints onβ¯=β¯​\(d\)\\overline\{\\beta\}=\\overline\{\\beta\}\(d\)asd→∞d\{\\rightarrow\}\\infty\. In particular,\|γ¯​\(d\)−β¯​\(d\)\|→0\|\\overline\{\\gamma\}\(d\)\-\\overline\{\\beta\}\(d\)\|\{\\rightarrow\}0fast enough asd→∞d\{\\rightarrow\}\\inftyelse the term−γ¯​cot⁡\(λ\)\+β¯​csc⁡\(λ\)\-\\overline\{\\gamma\}\\cot\(\\lambda\)\+\\overline\{\\beta\}\\csc\(\\lambda\)will diverge to±∞\\pm\\infty\. A natural question is to ask about the effects of noise and random perturbations even whenβ¯​\(d\)−γ¯​\(d\)\\overline\{\\beta\}\(d\)\-\\overline\{\\gamma\}\(d\)has good asymptotics\. \(2\) From a mathematical perspective, the framework described in this paper shows that there are interesting connections to analytic geometry and potentially complex analysis\. Further, unlike when sigmoidal functions are used to build the activation function, much of the geometry is attractive and natural\. Perhaps this setting has the potential to form a bridge between the limiting regimes, whenddand/orkk→∞\{\\rightarrow\}\\infty, and the natural regimesd,k<∞d,k<\\infty\.✠\\maltese

We conclude with a summary of the paper by section\.

Section[2](https://arxiv.org/html/2609.30661#S2)starts with a short review of shallow neural networks \(with and without bias\), notational conventions \(which are strictly followed and avoid use of the transpose\) and the fundamental approximation theorem for shallow neural networks\. The*population loss*or just the*loss*is defined as an ergodic limit of the empirical loss, using Gaussian statistics\. The section concludes with a review of the results of Choo & Saul\[[10](https://arxiv.org/html/2609.30661#bib.bib10)\]and Brutzkus & Globerson\[[8](https://arxiv.org/html/2609.30661#bib.bib8)\]together with remarks about the subanalyticity of the loss function in the bias free case\.

Sections 3 through 11 are devoted to obtaining explicit formulas for the loss and gradient of the loss in a shallow ReLU network with bias\. Section 3 starts with an introduction to the problem of finding an expression for the loss\. This amounts to evaluation of the expectation

f⁡\(\(𝐰,β\),\(𝐯,γ\)\)=‖𝐰‖​‖𝐯‖​𝔼𝐱∼𝒩​\(σ⁡\(𝐰¯​𝐱−β¯\)​σ​\(𝐯¯​𝐱−γ¯\)\)f\(\(\{\{\\mathbf\{w\}\}\},\\beta\),\(\{\{\\mathbf\{v\}\}\},\\gamma\)\)=\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\\mathbb\{E\}\_\{\{\\mathbf\{x\}\}\\sim\\mathcal\{N\}\}\(\\sigma\(\\overline\{\{\{\\mathbf\{w\}\}\}\}\{\\mathbf\{x\}\}\-\\overline\{\\beta\}\)\\sigma\(\\overline\{\{\{\\mathbf\{v\}\}\}\}\{\\mathbf\{x\}\}\-\\overline\{\\gamma\}\)\)where we use the*normalized*biasesβ¯=β/‖𝐰‖\\overline\{\\beta\}=\\beta/\\\|\{\{\\mathbf\{w\}\}\}\\\|,γ¯=γ/‖𝐯‖\\overline\{\\gamma\}=\\gamma/\\\|\{\{\\mathbf\{v\}\}\}\\\|,𝐰¯=𝐰/\|𝐰∥\\overline\{\{\{\\mathbf\{w\}\}\}\}=\{\{\\mathbf\{w\}\}\}/\|\{\{\\mathbf\{w\}\}\}\\\|,𝐯¯=𝐯/‖𝐯‖\\overline\{\{\{\\mathbf\{v\}\}\}\}=\{\{\\mathbf\{v\}\}\}/\\\|\{\{\\mathbf\{v\}\}\}\\\|\. and𝒩\\mathcal\{N\}denotes the standard normal distributionN⁡\(0,Id\)N\(0,I\_\{d\}\)onℝd\{\\mathbb\{R\}\}^\{d\}\. Rather than using polar coordinates \(the method used in\[[10](https://arxiv.org/html/2609.30661#bib.bib10)\]\), we present a proof in 3\.1 of Choo & Saul’s result using Cartesian coordinates onℝ2\{\\mathbb\{R\}\}^\{2\}as this indicates some of the difficulties of handling the general case in a simple situation where the result is known\. The reader will note how all the terms involving error functions, exponentials, cosecants and cotangents disappear in the final steps of the computation \(remarkable cancellations also occur in the computation of the gradient of the loss in a biased ReLU network\)\.

Sections 3\.2, 3\.3 evaluate the defining integrals forf⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)with one exception:

π2​∫γ¯∞e−t22​erf​\(−t​cot⁡\(θ\)\+β¯​csc⁡\(θ\)2\)​𝑑t\\sqrt\{\\frac\{\\pi\}\{2\}\}\\int\_\{\\overline\{\\gamma\}\}^\{\\infty\}e^\{\-\\frac\{t^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-t\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}dtThis integral cannot be evaluated in terms of ‘standard functions’\. However, it can be evaluated in terms of two instances of Owen’sTT\-function, three inverse tangents and one error function and this is the topic of Section[4](https://arxiv.org/html/2609.30661#S4)and Appendix C\. In Appendix B we indicate how high precision computation of theTT\-function can be done rapidly using an algorithm due to Komelj\[[11](https://arxiv.org/html/2609.30661#bib.bib11)\]\(see Appendix C\)\.

Using results on theTT\-function, we complete the derivation of a computable formula forf⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)in Sections 5 and 6\.

In Sections 7, 8 and 9, the gradient offfis computed with respect to𝐰\{\{\\mathbf\{w\}\}\}and𝜷¯\{\\boldsymbol\{\\overline\{\\beta\}\}\}\. A few brief notes are included on changes in the formulas if we useβ\\betarather thanβ¯=β/‖𝐰‖\\overline\{\\beta\}=\\beta/\\\|\{\{\\mathbf\{w\}\}\}\\\|as the bias parameter\.

In Section 10, the formula for the gradient of the lossℒ\\mathcal\{L\}with respect to𝐰\{\{\\mathbf\{w\}\}\}is given; in Section 11 is the formula for the gradient of the loss with respect to𝜷¯\\boldsymbol\{\\overline\{\\beta\}\}\. In addition, formulas for the𝜷¯\\boldsymbol\{\\overline\{\\beta\}\}derivative of the loss at𝜷¯=0\\boldsymbol\{\\overline\{\\beta\}\}=0are given\. This is approximately the half\-way point of the paper and readers not interested in applications related to symmetry can safely omit reading the remaining sections \(but perhaps not the appendices\)\.

In Section 12, we review notation and terminology for symmetry in a bias free ReLU network with the high symmetry targetIdI\_\{d\}\. Next is a description of symmetries of the lossℒ\\mathcal\{L\}in a biased network with target\(Id,𝜸¯\)∈M⁡\(d,d\+1\)\(I\_\{d\},\{\\boldsymbol\{\\overline\{\\gamma\}\}\}\)\\in M\(d,d\+1\), where𝜸¯∈M⁡\(d,1\)\{\\boldsymbol\{\\overline\{\\gamma\}\}\}\\in M\(d,1\)is the matrix of biases which are assumed equal\. The remainder of the section is devoted to a description of how we quantify the symmetry of a critical point using the isotropy group of the action ofΔ​Sd\\Delta S\_\{d\}onM⁡\(d,d\+1\)M\(d,d\+1\)and the vital role played by the associated fixed point spaces which allow a big reduction in the dimension of the space needed when we search for families of critical points of the loss\. For example, the fixed point spaceM​\(d,d\+1\)Δ​SdM\(d,d\+1\)^\{\\Delta S\_\{d\}\}is 3\-dimensional and is used to find type I families of critical points with isotropyΔ​Sd\\Delta S\_\{d\}\. If we search for families with isotropyΔ⁡\(Sd−q×Sq\)\\Delta\(S\_\{d\-q\}\\times S\_\{q\}\), whereq≥2q\\geq 2and fixed, then the associated fixed point space is of dimension88compared with thed2\+dd^\{2\}\+d, the dimension ofM⁡\(d,d\+1\)M\(d,d\+1\)\. Typically, Newton’s method is used to find critical points by treatingddas a real parameter \(it is, if we restrict to the fixed point space and is fast\)\. An alternative, is to use gradient descent on the fixed point space\.

In the remainder of Section 12, we develop the necessary results to give the formula for the gradient of the loss \(Proposition[12\.15](https://arxiv.org/html/2609.30661#S12.Thmlemma15)\)\. Although the presence of symmetry greatly simplifies the computation of critical points, the formula for the gradient is significantly more complex than the general formula for the asymmetric case given towards the end of Section 10\. The section ends with necessary details about the computation ofℒ\\mathcal\{L\}in the symmetric case as well as the formula for the derivative ofℒ\\mathcal\{L\}with respect toβ¯i\\overline\{\\beta\}\_\{i\}\.

Section 13 is a brief review of relevant past work on the bias free case and the role of fractional power series in parameterizingdd\-dependant families of critical points \(types I and II\)\.

In Section 14, examples are given of the computation of∂ℒ∂β¯​\(𝐖,𝟎\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\}\(\{\{\\mathbf\{W\}\}\},\\mathbf\{0\}\), where𝐖\{\{\\mathbf\{W\}\}\}is a critical point of the loss in the zero bias network\. Included are examples wherek\>dk\>dor the target has an asymmetric diagonal block\. In every case,∂ℒ∂β¯​\(𝐖,𝟎\)≠0\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\}\(\{\{\\mathbf\{W\}\}\},\\mathbf\{0\}\)\\neq 0\. These results imply that the extension of results obtained for unbiased networks to biased networks is non\-trivial\. For example, adding bias might annihilate the family of spurious minima and result in the point\(𝐖,𝟎\)\(\{\{\\mathbf\{W\}\}\},\{\\mathbf\{0\}\}\)lying on a gradient trajectory convergent to a critical point defining the global minimum\. However, it does not—at least in the cases so far analysed\.

In Section 15, we give the results of numerical investigations that show that families of spurious minima, and critical points generally, do persist when we add bias and limiting values asd→∞d\{\\rightarrow\}\\inftyare the same as those in the unbiased case\. Moreover, adding bias reduces the loss \(as indeed it should\) but asd→∞d\{\\rightarrow\}\\infty, the difference converges rapidly to zero,

Appendices A, B & C respectively review needed formulas for the error function, give details on the high precision computation of theTT\-function and lastly describe an alternative derivation of the key formula \([4\.18](https://arxiv.org/html/2609.30661#S4.E18)\)\[[21](https://arxiv.org/html/2609.30661#bib.bib21)\]for the ”gneralizedTT\-function using only Owen’s results\[[18](https://arxiv.org/html/2609.30661#bib.bib18)\]\.

## 2\.Background on neural networks and machine learning

### 2\.1\.Shallow Neural Networks

Roughly speaking, a shallow neural network consists of an*input*layer, one222More generally, a ‘small’ number of hidden layers*hidden layer*comprised of “neurons”, and an*output*layer\. The number of layers is defined as the number of hidden layers plus one\. If the output layer is not counted–that is, the weighted summation over the hidden neuron outputs is not included—then the network is said to have1⁤121\\frac\{1\}\{2\}layers\. See Figure[2\.1](https://arxiv.org/html/2609.30661#S2.SS1)for a 2\-layer neural network with one hidden layer\.

![[Uncaptioned image]](https://arxiv.org/html/2609.30661v1/nn.png)The*width*of the network is the maximal number of nodes in a layer\. Deep neural networks typically have many hidden layers of relatively small widths whereas shallow networks may have very large widths\. In this work the emphasis is on shallow networks though results are likely relevant for deep networks\.

Referring to Figure[2\.1](https://arxiv.org/html/2609.30661#S2.SS1), suppose that the input is𝐱=\(x1,…,xd\)∈ℝd\{\\mathbf\{x\}\}=\(x\_\{1\},\\ldots,x\_\{d\}\)\\in\{\\mathbb\{R\}\}^\{d\}, and that we are given a set ofkkaffine linear mapsAi:ℝd→ℝA\_\{i\}:\{\\mathbb\{R\}\}^\{d\}\{\\rightarrow\}\{\\mathbb\{R\}\},𝐚∈\(ℝk\)⋆≈M⁡\(1,k\)\\mathbf\{a\}\\in\(\{\\mathbb\{R\}\}^\{k\}\)^\{\\star\}\\approx M\(1,k\)and a continuousσ:ℝ→ℝ\\sigma:\{\\mathbb\{R\}\}\{\\rightarrow\}\{\\mathbb\{R\}\}—the*activation*333In this work, only ReLU activation is used:σ⁡\(t\)=max⁡\{0,t\}\\sigma\(t\)=\\max\\\{0,t\\\}\.\. The neural network defines the mapF^:ℝd→ℝ\\widehat\{F\}:\{\\mathbb\{R\}\}^\{d\}\{\\rightarrow\}\{\\mathbb\{R\}\}by

F^​\(𝐱\)=∑i∈\[k\]ai​σ​\(Ai​\(𝐱\)\)\\widehat\{F\}\(\{\\mathbf\{x\}\}\)=\\sum\_\{i\\in\[k\]\}a\_\{i\}\\sigma\(A\_\{i\}\(\{\\mathbf\{x\}\}\)\)Suppose thatAi​\(𝐱\)=∑jwi​j​xj−βiA\_\{i\}\(\{\\mathbf\{x\}\}\)=\\sum\_\{j\}w\_\{ij\}x\_\{j\}\-\\beta\_\{i\}\. Then\[wi​j\]=𝐖∈M⁡\(k,d\)\[w\_\{ij\}\]=\{\{\\mathbf\{W\}\}\}\\in M\(k,d\)defines the*parameter matrix*and\[βi\]=𝜷∈M⁡\(k,1\)\[\\beta\_\{i\}\]=\\boldsymbol\{\\beta\}\\in M\(k,1\)is the vector of neuron*biases*or*offsets*\. Note that it is common to writeσ⁡\(𝐰𝐱\+β\)\\sigma\(\{\{\\mathbf\{w\}\}\}\{\\mathbf\{x\}\}\+\\beta\)rather thanσ⁡\(𝐰𝐱−β\)\\sigma\(\{\{\\mathbf\{w\}\}\}\{\\mathbf\{x\}\}\-\\beta\)as done here \(following\[[20](https://arxiv.org/html/2609.30661#bib.bib20)\]\)\.

Let𝐰i∈\(ℝd\)⋆\{\{\\mathbf\{w\}\}\}^\{i\}\\in\(\{\\mathbb\{R\}\}^\{d\}\)^\{\\star\}denote rowiiof𝐖\{\{\\mathbf\{W\}\}\},i∈\[k\]i\\in\[k\]\. Then

F^​\(𝐱\)=∑i∈\[k\]ai​σ​\(𝐰i​𝐱−βi\)=𝐚​σ​\(𝐖𝐱−𝜷\),\\widehat\{F\}\(\{\\mathbf\{x\}\}\)=\\sum\_\{i\\in\[k\]\}a\_\{i\}\\sigma\(\{\{\\mathbf\{w\}\}\}^\{i\}\{\\mathbf\{x\}\}\-\\beta\_\{i\}\)=\{\\mathbf\{a\}\}\\sigma\(\{\{\\mathbf\{W\}\}\}\{\\mathbf\{x\}\}\-\\boldsymbol\{\\beta\}\),whereσ⁡\(𝐖𝐱−𝜷\)=def\(σ⁡\(𝐰1​𝐱−β1\),…,σ⁡\(𝐰k​𝐱−βk\)\)∈ℝk\\sigma\(\{\{\\mathbf\{W\}\}\}\{\\mathbf\{x\}\}\-\\boldsymbol\{\\beta\}\)\\stackrel\{\{\\scriptstyle\\mathrm\{def\}\}\}\{\{=\}\}\(\\sigma\(\{\{\\mathbf\{w\}\}\}^\{1\}\{\\mathbf\{x\}\}\-\\beta\_\{1\}\),\\ldots,\\sigma\(\{\{\\mathbf\{w\}\}\}^\{k\}\{\\mathbf\{x\}\}\-\\beta\_\{k\}\)\)\\in\{\\mathbb\{R\}\}^\{k\}and𝐚​σ​\(W​𝐱−𝜷\)\{\\mathbf\{a\}\}\\sigma\(W\{\\mathbf\{x\}\}\-\\boldsymbol\{\\beta\}\)is matrix multiplication \(or evaluation of a linear functional\)\. In summary, neuroniihas biasβi∈ℝ\\beta\_\{i\}\\in\{\\mathbb\{R\}\}and \(input\) parameters𝐰i=\(wi​1,⋯,wi​d\)∈\(ℝd\)⋆\{\{\\mathbf\{w\}\}\}^\{i\}=\(w\_\{i1\},\\cdots,w\_\{id\}\)\\in\(\{\\mathbb\{R\}\}^\{d\}\)^\{\\star\}\. The notation generalizes straightforwardly to mapsF:ℝn1→ℝn2F:\{\\mathbb\{R\}\}^\{n\_\{1\}\}\{\\rightarrow\}\{\\mathbb\{R\}\}^\{n\_\{2\}\},n2\>1n\_\{2\}\>1, as well as to deep neural networks\.

Given the activation functionσ\\sigma, letℳ⁡\(σ\)⊂C0​\(ℝd\)\\mathcal\{M\}\(\\sigma\)\\subset C^\{0\}\(\{\\mathbb\{R\}\}^\{d\}\)be the vector space spanned by\{σ\(𝐰𝐱−β\)\|𝐰∈\(ℝd\)⋆,β∈ℝ\}\\\{\\sigma\(\{\{\\mathbf\{w\}\}\}\{\\mathbf\{x\}\}\-\\beta\)\\mbox\{$\\;\|\\;$\}\{\{\\mathbf\{w\}\}\}\\in\(\{\\mathbb\{R\}\}^\{d\}\)^\{\\star\},\\beta\\in\{\\mathbb\{R\}\}\\\}\. We recall the fundamental approximation theorem for shallow neural nets\.

###### Theorem 2\.1\(\[[20](https://arxiv.org/html/2609.30661#bib.bib20), Theorem 3\.1\]\)\.

Letσ:ℝ→ℝ\\sigma:\{\\mathbb\{R\}\}\{\\rightarrow\}\{\\mathbb\{R\}\}be a continuous function and giveC0​\(ℝd\)C^\{0\}\(\{\\mathbb\{R\}\}^\{d\}\)the topology of uniform convergence on compact subsets\. Thenℳ⁡\(σ\)\\mathcal\{M\}\(\\sigma\)is dense inC0​\(ℝd\)C^\{0\}\(\{\\mathbb\{R\}\}^\{d\}\)iffσ\\sigmais*not*a polynomial\.

###### Remark 2\.2\.

The approximation theorem fails if we replace*affine linear*maps by*linear*maps—that is, all biases are set to zero andℳ⋆​\(σ\)\\mathcal\{M\}^\{\\star\}\(\\sigma\)is defined to be the vector space spanned by\{σ⁡\(𝐰𝐱\)\|𝐰∈\(ℝd\)⋆\}\\\{\\sigma\(\{\{\\mathbf\{w\}\}\}\{\\mathbf\{x\}\}\)\\mbox\{$\\;\|\\;$\}\{\{\\mathbf\{w\}\}\}\\in\(\{\\mathbb\{R\}\}^\{d\}\)^\{\\star\}\\\}: the closure ofℳ⋆​\(σ\)\\mathcal\{M\}^\{\\star\}\(\\sigma\)has no interior points\. For details we refer to the recent paper by Zhang, Saxe and Latham\[[24](https://arxiv.org/html/2609.30661#bib.bib24)\]\. The main aim of the present work is to address the question of whether or not results obtained for the restricted class of bias free ReLU networks extend to class of ReLU networks with bias covered by the approximation theorem\. For example, the existence of families of spurious minima of various types in bias free student teacher models\[[22](https://arxiv.org/html/2609.30661#bib.bib22),[2](https://arxiv.org/html/2609.30661#bib.bib2),[4](https://arxiv.org/html/2609.30661#bib.bib4),[5](https://arxiv.org/html/2609.30661#bib.bib5)\]\(see Section[13](https://arxiv.org/html/2609.30661#S13)for a review of theory for examples with symmetry\)\. Since the closure ofℳ⋆​\(σ\)\\mathcal\{M\}^\{\\star\}\(\\sigma\)has no interior points, and non\-degenerate critical points of a bias free network appear never to be critical points of the associated biased network, with bias set to zero \(see Section[14](https://arxiv.org/html/2609.30661#S14)\), the question concerning the existence of these families in a ReLU network with bias cannot be inferred using simple stability \(persistence under perturbation\) arguments\. The answer, as indicated at the end of the paper, is positive but non\-trivial\.✠\\maltese

### 2\.2\.Statistical learning and a reduction

Suppose given a physical system that we believe can be modelled by a functionF:ℝd→ℝF:\{\\mathbb\{R\}\}^\{d\}\{\\rightarrow\}\{\\mathbb\{R\}\}\. Based on experimental data, it is required to construct a ‘good’ approximation to the conjectured modelFF\. To this end, assume given a ‘random’ sequence of points\(𝐱i\)∈ℝd\(\{\\mathbf\{x\}\}^\{i\}\)\\in\{\\mathbb\{R\}\}^\{d\}and corresponding value sequence\(yi\)=\(F⁡\(𝐱i\)\)\(y^\{i\}\)=\(F\(\{\\mathbf\{x\}\}^\{i\}\)\)and train a neural network with this data\. Roughly speaking, initialize the network with an appropriate444For example, Xavier\[[22](https://arxiv.org/html/2609.30661#bib.bib22)\]random distribution of parameters𝐖,𝐚\{\{\\mathbf\{W\}\}\},\\mathbf\{a\}and biases𝜷\\boldsymbol\{\\beta\}\. ComputeF^​\(𝐱i\)\\widehat\{F\}\(\{\\mathbf\{x\}\}^\{i\}\),i≥1i\\geq 1, and then improve the estimates ofF^​\(xi\)−yi\\widehat\{F\}\(x^\{i\}\)\-y\_\{i\}using gradient descent and back propagation on an appropriate cost function \(see below\)\.

This process is difficult to handle theoretically since the functionFFis unknown and the implied optimization is non\-convex\. There is also the matter of a lack of analyticity \(the ReLU activation functionσ⁡\(t\)=max⁡\{0,t\}\\sigma\(t\)=\\max\\\{0,t\\\}most commonly used is only piecewise analytic\)\. A more realistic approach is to study the*realizable*case where the data is generated by a known functionFF\. To this end, assume a trained neural network \(the ‘teacher’\) and use that to train a second neural network \(the ‘student’\)\. Here we are given the parameter matrix𝐕\{\{\\mathbf\{V\}\}\}, bias vector𝜸\\boldsymbol\{\\gamma\}and linear functional𝐛\\mathbf\{b\}defining the output for the teacher network\. Assume the teacher and student networks both haveddinputs\. If the teacher network hask′k^\{\\prime\}neurons, then\(𝐕,𝜸\)∈M⁡\(k′,d\+1\)\(\{\{\\mathbf\{V\}\}\},\\boldsymbol\{\\gamma\}\)\\in M\(k^\{\\prime\},d\+1\), where the column matrix𝜸∈M⁡\(k′,1\)\\boldsymbol\{\\gamma\}\\in M\(k^\{\\prime\},1\)gives the neuron biases and𝐛∈M⁡\(1,k\)\\mathbf\{b\}\\in M\(1,k\)\. DefineF:ℝd→ℝF:\{\\mathbb\{R\}\}^\{d\}\{\\rightarrow\}\{\\mathbb\{R\}\}by

F⁡\(𝐱\)=F⁡\(𝐱,𝐕,𝜸,𝐛\)=∑i∈\[k\]bi​σ​\(𝐯i​𝐱−γi\)=𝐛​σ​\(𝐕𝐱−𝜸\)=yF\(\{\\mathbf\{x\}\}\)=F\(\{\\mathbf\{x\}\};\{\{\\mathbf\{V\}\}\},\\boldsymbol\{\\gamma\},\\mathbf\{b\}\)=\\sum\_\{i\\in\[k\]\}b\_\{i\}\\sigma\(\{\{\\mathbf\{v\}\}\}^\{i\}\{\\mathbf\{x\}\}\-\\gamma\_\{i\}\)=\{\\mathbf\{b\}\}\\sigma\(\{\{\\mathbf\{V\}\}\}\{\\mathbf\{x\}\}\-\\boldsymbol\{\\gamma\}\)=yHere𝐱\{\\mathbf\{x\}\}will be sampled according to a given distribution𝒟\\mathcal\{D\}onℝd\{\\mathbb\{R\}\}^\{d\}, typically the standard norm distribution𝒩⁡\(0,Id\)\\mathcal\{N\}\(0,I\_\{d\}\)\. If we assumenntraining points𝐱1,…,𝐱n\{\\mathbf\{x\}\}\_\{1\},\\ldots,\{\\mathbf\{x\}\}\_\{n\}and setF⁡\(𝐱i\)=yiF\(\{\\mathbf\{x\}\}\_\{i\}\)=y\_\{i\}, then the problem is to find𝐖,𝜷,𝐚\{\{\\mathbf\{W\}\}\},\\boldsymbol\{\\beta\},\\mathbf\{a\}for the student network that minimize the*training*\(or*empirical*\) loss:

1n​\(12​∑i\(F^​\(𝐱i,𝐖,𝜷\)−yi\)2\)\\frac\{1\}\{n\}\\left\(\\frac\{1\}\{2\}\\sum\_\{i\}\\big\(\\widehat\{F\}\(\{\\mathbf\{x\}\}\_\{i\};\{\{\\mathbf\{W\}\}\},\\boldsymbol\{\\beta\}\)\-y\_\{i\}\\big\)^\{2\}\\right\)Assume that the student haskkneurons and so𝐖∈M⁡\(k,d\)\{\{\\mathbf\{W\}\}\}\\in M\(k,d\),𝜷∈M⁡\(k,1\)\\boldsymbol\{\\beta\}\\in M\(k,1\)and𝐚∈M⁡\(1,k\)\\mathbf\{a\}\\in M\(1,k\)\. Lettingn→∞n\{\\rightarrow\}\\infty, leads to the problem of minimizing the*population loss*555Also called the*population risk*\. Typically we just refer to the ‘loss’:

ℒ⁡\(𝐖,𝐕\)=12​𝔼𝐱∼𝒟​\(F^​\(𝐱,𝐖\)−F⁡\(𝐱,𝐕\)\)2,\\mathcal\{L\}\(\{\{\\mathbf\{W\}\}\},\{\{\\mathbf\{V\}\}\}\)=\\frac\{1\}\{2\}\\mathbb\{E\}\_\{\{\\mathbf\{x\}\}\\sim\\mathcal\{D\}\}\\big\(\\widehat\{F\}\(\{\\mathbf\{x\}\},\{\{\\mathbf\{W\}\}\}\)\-F\(\{\\mathbf\{x\}\};\{\{\\mathbf\{V\}\}\}\)\\big\)^\{2\},whereF^​\(𝐱\)=F^​\(𝐱,𝐖,𝜷,𝐚\)\\widehat\{F\}\(\{\\mathbf\{x\}\}\)=\\widehat\{F\}\(\{\\mathbf\{x\}\};\{\{\\mathbf\{W\}\}\},\\boldsymbol\{\\beta\},\\mathbf\{a\}\)\. Here, as elsewhere we usually writeF^​\(𝐱\)\\widehat\{F\}\(\{\\mathbf\{x\}\}\)orF^​\(𝐱,𝐖\)\\widehat\{F\}\(\{\\mathbf\{x\}\};\{\{\\mathbf\{W\}\}\}\)rather thanF^​\(𝐱,𝐖,𝜷,𝐚\)\\widehat\{F\}\(\{\\mathbf\{x\}\};\{\{\\mathbf\{W\}\}\},\\boldsymbol\{\\beta\},\\mathbf\{a\}\); similarly forF⁡\(𝐱\)F\(\{\\mathbf\{x\}\}\)\.

This reduces the problem to one of non\-convex optimization ofℒ=ℒ⁡\(𝐖\)\\mathcal\{L\}=\\mathcal\{L\}\(\{\{\\mathbf\{W\}\}\}\)using gradient descent\. Further simplifications are possible and often used\. For example, assuming biases are constant and set to zero and the functional𝐚\\mathbf\{a\}is constant\. By way of illustration, in Brutzkus & Globerson’s work on no\-overlap convolutional networks\[[8](https://arxiv.org/html/2609.30661#bib.bib8)\], the networks studied are bias free, with ReLU activation, thekkneurons have the same number of inputs, none of which overlap, and the output is given byF⁡\(𝐱,𝐰\)=1k​∑iσ⁡\(𝐰,𝐱i\)F\(\{\\mathbf\{x\}\},\{\{\\mathbf\{w\}\}\}\)=\\frac\{1\}\{k\}\\sum\_\{i\}\\sigma\(\{\{\\mathbf\{w\}\}\},\{\\mathbf\{x\}\}\_\{i\}\)—average pooling\.

It is also natural to take𝐚=𝐛=𝟏k\\mathbf\{a\}=\\mathbf\{b\}=\\mathbf\{1\}\_\{k\}\[[22](https://arxiv.org/html/2609.30661#bib.bib22),[2](https://arxiv.org/html/2609.30661#bib.bib2)\]\. Since ReLU activation is positively homogeneous we can rescale rows of𝐖\{\{\\mathbf\{W\}\}\}and𝜷\\boldsymbol\{\\beta\}so thatℒ⁡\(𝐖,𝐕\)\\mathcal\{L\}\(\{\{\\mathbf\{W\}\}\},\{\{\\mathbf\{V\}\}\}\)is unchanged butai∈\{±1,0\}a\_\{i\}\\in\\\{\\pm 1,0\\\}, for alli∈\[k\]i\\in\[k\]\. Under these conditions,𝐖,𝜷\{\{\\mathbf\{W\}\}\},\\boldsymbol\{\\beta\}and𝐚\\mathbf\{a\}are uniquely determined up to row permutations666Of course, we may remove rows𝐰i\{\{\\mathbf\{w\}\}\}^\{i\}ifai=0a\_\{i\}=0\.\.

Assume the target network is bias free and the target matrix of parameters is𝐕∈M\(d\.d\)\{\{\\mathbf\{V\}\}\}\\in M\(d\.d\)and𝐕\{\{\\mathbf\{V\}\}\}has no zero rows\. Take𝐛=𝟏d\\mathbf\{b\}=\\mathbf\{1\}\_\{d\}\. This choice simplifies the optimization process \(the network is said to have1⁤121\\frac\{1\}\{2\}layers\)\. Letk≥dk\\geq dand𝐖∈M⁡\(k,d\)\{\{\\mathbf\{W\}\}\}\\in M\(k,d\)be a choice of initialization of the parameter matrix for the student network\. Again assume no zero rows for𝐖\{\{\\mathbf\{W\}\}\}, bias free and𝐚=𝟏k\\mathbf\{a\}=\\mathbf\{1\}\_\{k\}\. Assume ReLU activation and bias free\. The loss is given by

ℒ⁡\(𝐖,𝐕\)=12​𝔼𝐱∼𝒩⁡\(0,Id\)​\(∑i∈\[k\]σ⁡\(𝐰i​𝐱\)−∑j∈\[d\]σ⁡\(𝐯j​𝐱\)\)2,\\mathcal\{L\}\(\{\{\\mathbf\{W\}\}\},\{\{\\mathbf\{V\}\}\}\)=\\frac\{1\}\{2\}\\mathbb\{E\}\_\{\{\\mathbf\{x\}\}\\sim\\mathcal\{N\}\(0,I\_\{d\}\)\}\\left\(\\sum\_\{i\\in\[k\]\}\\sigma\(\{\{\\mathbf\{w\}\}\}^\{i\}\{\\mathbf\{x\}\}\)\-\\sum\_\{j\\in\[d\]\}\\sigma\(\{\{\\mathbf\{v\}\}\}^\{j\}\{\\mathbf\{x\}\}\)\\right\)^\{2\},where𝔼\\mathbb\{E\}is the expectation over the zero mean, unit variance Gaussian distribution onℝd\{\\mathbb\{R\}\}^\{d\}\. Obviouslyℒ⁡\(𝐖,𝐕\)=0\\mathcal\{L\}\(\{\{\\mathbf\{W\}\}\},\{\{\\mathbf\{V\}\}\}\)=0if𝐖=𝐕\{\{\\mathbf\{W\}\}\}=\{\{\\mathbf\{V\}\}\}\. The same is true if we replace𝐖\{\{\\mathbf\{W\}\}\}by any row permutation of𝐖\{\{\\mathbf\{W\}\}\}; similarly for𝐕\{\{\\mathbf\{V\}\}\}\.

Following Brutzkus & Globerson, define

f⁡\(𝐰,𝐯\)=𝔼𝐱∼𝒩⁡\(0,Id\)​σ​\(𝐰𝐱\)​σ​\(𝐯𝐱\),𝐰,𝐯∈\(ℝd\)⋆f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)=\\mathbb\{E\}\_\{\{\\mathbf\{x\}\}\\sim\\mathcal\{N\}\(0,I\_\{d\}\)\}\\;\\sigma\(\{\{\\mathbf\{w\}\}\}\{\\mathbf\{x\}\}\)\\sigma\(\{\{\\mathbf\{v\}\}\}\{\\mathbf\{x\}\}\),\\;\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\\in\(\{\\mathbb\{R\}\}^\{d\}\)^\{\\star\}and note that

ℒ⁡\(𝐖,𝐕\)=12​\[∑i,j​i∈\[k\]f⁡\(𝐰i,𝐰j\)−2​∑i∈\[k\],j∈\[d\]f⁡\(𝐰i,𝐯j\)\+12​∑i,j∈\[d\]f⁡\(𝐯i,𝐯j\)\]\\mathcal\{L\}\(\{\{\\mathbf\{W\}\}\},\{\{\\mathbf\{V\}\}\}\)=\\frac\{1\}\{2\}\\left\[\\sum\_\{i,ji\\in\[k\]\}f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{j\}\)\-2\\sum\_\{i\\in\[k\],j\\in\[d\]\}f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{j\}\)\+\\frac\{1\}\{2\}\\sum\_\{i,j\\in\[d\]\}f\(\{\{\\mathbf\{v\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{j\}\)\\right\]Using elementary methods, Choo & Saul\[[10](https://arxiv.org/html/2609.30661#bib.bib10)\]proved that

\(2\.1\)f⁡\(𝐰,𝐯\)=‖𝐰‖​‖𝐯‖2​π​\(sin⁡\(θ\)\+\(π−θ\)​cos⁡\(θ\)\),f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)=\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\pi\}\\left\(\\sin\(\\theta\)\+\(\\pi\-\\theta\)\\cos\(\\theta\)\\right\),whereθ\\thetais the angle between𝐰\{\{\\mathbf\{w\}\}\}and𝐯\{\{\\mathbf\{v\}\}\}\. Of course, if either𝐯\{\{\\mathbf\{v\}\}\}or𝐰\{\{\\mathbf\{w\}\}\}is zero,f⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)is zero\. Obviously,ffis a continuous function of\(𝐰,𝐯\)\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)onℝd×ℝd\{\\mathbb\{R\}\}^\{d\}\\times\{\\mathbb\{R\}\}^\{d\}and is analytic precisely on the open and dense domain\{\(𝐰,𝐯\)\|⟨𝐰,𝐯⟩≠±‖𝐰‖​‖𝐯‖\}⊂ℝd×ℝd\\\{\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)\\mbox\{$\\;\|\\;$\}\\langle\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\\rangle\\neq\\pm\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\\\}\\subset\{\\mathbb\{R\}\}^\{d\}\\times\{\\mathbb\{R\}\}^\{d\}\. In particular, if𝐰,𝐯\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}are parallel,ffis not analytic at\(𝐰,𝐯\)\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)\.

Moreover,ffis a*subanalytic*function\. That is, the closure of the graph offfis locally the projection of a relatively compact semianalytic set\[[19](https://arxiv.org/html/2609.30661#bib.bib19), §2\],\[[7](https://arxiv.org/html/2609.30661#bib.bib7)\]\. This follows easily since the sum and product of subanalytic functions are subanalytic and the Euclidean norm∥⁣∥\\\|\\;\\\|is trivially a subanalytic function of𝐰∈ℝd\{\{\\mathbf\{w\}\}\}\\in\{\\mathbb\{R\}\}^\{d\}\. Hence, it suffices to show that⟨𝐰,𝐯⟩/‖𝐰‖​‖𝐯‖\\langle\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\\rangle/\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|is subanalytic and this is elementary using the Cauchy\-Schwartz inequality\. For our applications,𝐰,𝐯\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}will never be zero and so proving subanalyticity amounts to analyzing limiting behaviour as⟨𝐰,𝐯⟩→0\\langle\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\\rangle\{\\rightarrow\}0, where𝐰,𝐯\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}are bounded away from zero\.

The subanalyticity offfimplies strong regularity results onffthat are exploited in\[[4](https://arxiv.org/html/2609.30661#bib.bib4),[5](https://arxiv.org/html/2609.30661#bib.bib5),[6](https://arxiv.org/html/2609.30661#bib.bib6)\]when the target𝐕=Id\{\{\\mathbf\{V\}\}\}=I\_\{d\}and methods from the analytic geometry of stratified sets can be used to prove the existence of fractional power series \(FPS\) representations of families of critical points parametrized byddand precise results on the Hessian spectrum \(theFPSresults are discussed in more detail in Section[13](https://arxiv.org/html/2609.30661#S13)\)\.

The main aim of this work is to provide the framework for extending results on spurious minima and fractional power series to biased networks\. The key steps involve generalizing the formula of Choo & Saul \([2\.1](https://arxiv.org/html/2609.30661#S2.E1)\) and quantifying the regularity of the associated population loss which is quite different from the bias free case\.

## 3\.Bias

We find an explicit formula for the loss when bias is included\. Our approach initially follows that of Choo & Saul\[[10](https://arxiv.org/html/2609.30661#bib.bib10)\]for zero bias \(see\[[4](https://arxiv.org/html/2609.30661#bib.bib4)\]for details in the case of ‘leaky ReLU’\)\.

Let∥⁣∥\\\|\\;\\\|and⟨,⟩\\langle\\;,\\;\\rangledenote the standard Euclidean norm and inner product\. Given𝐰∈\(ℝd\)⋆\{\{\\mathbf\{w\}\}\}\\in\(\{\\mathbb\{R\}\}^\{d\}\)^\{\\star\},𝐱∈ℝd\{\\mathbf\{x\}\}\\in\{\\mathbb\{R\}\}^\{d\},𝐰𝐱\{\{\\mathbf\{w\}\}\}\{\\mathbf\{x\}\}is evaluation of𝐰\{\{\\mathbf\{w\}\}\}at𝐱\{\\mathbf\{x\}\}\(matrix multiplication;𝐰⁡\(𝐱\)\{\{\\mathbf\{w\}\}\}\(\{\\mathbf\{x\}\}\)if it is necessary to emphasize that𝐰𝐱\{\{\\mathbf\{w\}\}\}\{\\mathbf\{x\}\}is the linear functional𝐰\{\{\\mathbf\{w\}\}\}evaluated at𝐱\{\\mathbf\{x\}\}\)\. If\(ℝd\)⋆\(\{\\mathbb\{R\}\}^\{d\}\)^\{\\star\}is identified withℝd\{\\mathbb\{R\}\}^\{d\}using the inner product, then𝐰𝐱=⟨𝐰,𝐱⟩=𝐰T​𝐱\{\{\\mathbf\{w\}\}\}\{\\mathbf\{x\}\}=\\langle\{\{\\mathbf\{w\}\}\},\{\\mathbf\{x\}\}\\rangle=\{\{\\mathbf\{w\}\}\}^\{T\}\{\\mathbf\{x\}\}\(matrix multiplication\) but we avoid the use of the transpose here and only use the inner product to define the angle between𝐰,𝐯\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\(row vectors\)\. To emphasize:𝐱\{\\mathbf\{x\}\}is a column vector and𝐰\{\{\\mathbf\{w\}\}\}is a row vector \(linear functional or dual vector\)\. Care needs to be taken with the notation especially because of the use of \(matrix\) group representation theory where the formal theory is most transparently expressed in terms of matrices and vectors rather than vectorized quantities\.

Let𝐰,𝐯∈\(ℝd\)⋆\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\\in\(\{\\mathbb\{R\}\}^\{d\}\)^\{\\star\}andβ,γ∈ℝ\\beta,\\gamma\\in\{\\mathbb\{R\}\}\. Following the approach for the bias free case, we want to evaluate

f⁡\(\(𝐰,β\),\(𝐯,γ\)\)=def𝔼𝐱∼𝒩​\(σ⁡\(𝐰𝐱−β\)​σ​\(𝐯𝐱−γ\)\),f\(\(\{\{\\mathbf\{w\}\}\},\\beta\),\(\{\{\\mathbf\{v\}\}\},\\gamma\)\)\\stackrel\{\{\\scriptstyle\\mathrm\{def\}\}\}\{\{=\}\}\\mathbb\{E\}\_\{\{\\mathbf\{x\}\}\\sim\\mathcal\{N\}\}\(\\sigma\(\{\{\\mathbf\{w\}\}\}\{\\mathbf\{x\}\}\-\\beta\)\\sigma\(\{\{\\mathbf\{v\}\}\}\{\\mathbf\{x\}\}\-\\gamma\)\),where𝐱∈ℝd\{\\mathbf\{x\}\}\\in\{\\mathbb\{R\}\}^\{d\}and𝒩=𝒩⁡\(0,Id\)\\mathcal\{N\}=\\mathcal\{N\}\(0,I\_\{d\}\)\(standard normal\)\. Usually we writef⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)and omit explicit dependence on the bias parameters\.

###### Remark 3\.1\.

Unlike the bias free case, if𝐰=𝟎\{\{\\mathbf\{w\}\}\}=\{\\mathbf\{0\}\},𝐯≠𝟎\{\{\\mathbf\{v\}\}\}\\neq\{\\mathbf\{0\}\}, thenf⁡\(𝐰,𝐯\)=0f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)=0only ifβ≥0\\beta\\geq 0; similarly if𝐯=𝟎\{\{\\mathbf\{v\}\}\}=\{\\mathbf\{0\}\}\. In particular,f⁡\(𝟎,𝟎\)≠0f\(\{\\mathbf\{0\}\},\{\\mathbf\{0\}\}\)\\neq 0iffβ,γ<0\\beta,\\gamma<0\. We generally assume that𝐖\{\{\\mathbf\{W\}\}\}and𝐕\{\{\\mathbf\{V\}\}\}have no zero rows\.✠\\maltese

Assume𝐰,𝐯≠𝟎\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\\neq\{\\mathbf\{0\}\}\. Write

\(3\.2\)𝐰𝐱−β=\|𝐰\|\(𝐰¯​𝐱−β¯\)𝐯𝐱−γ=\|𝐯\|\(𝐯¯​𝐱−γ¯\)\{\{\\mathbf\{w\}\}\}\{\\mathbf\{x\}\}\-\\beta=\\\|\{\{\\mathbf\{w\}\}\}\\\|\(\\overline\{\{\{\\mathbf\{w\}\}\}\}\{\\mathbf\{x\}\}\-\\bar\{\\beta\}\)\\quad\{\{\\mathbf\{v\}\}\}\{\\mathbf\{x\}\}\-\\gamma=\\\|\{\{\\mathbf\{v\}\}\}\\\|\(\\overline\{\{\{\\mathbf\{v\}\}\}\}\{\\mathbf\{x\}\}\-\\overline\{\\gamma\}\)where

𝐰¯=𝐰‖𝐰‖,𝐯¯=𝐯‖𝐯‖,β¯=β‖𝐰‖,γ¯=γ‖𝐯‖\\overline\{\{\{\\mathbf\{w\}\}\}\}=\\frac\{\{\{\\mathbf\{w\}\}\}\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\},\\;\\;\\overline\{\{\{\\mathbf\{v\}\}\}\}=\\frac\{\{\{\\mathbf\{v\}\}\}\}\{\\\|\{\{\\mathbf\{v\}\}\}\\\|\},\\;\\;\\overline\{\\beta\}=\\frac\{\\beta\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\},\\;\\;\\overline\{\\gamma\}=\\frac\{\\gamma\}\{\\\|\{\{\\mathbf\{v\}\}\}\\\|\}
It follows that

\(3\.3\)f⁡\(\(𝐰,β\),\(𝐯,γ\)\)=‖𝐰‖​‖𝐯‖​𝔼𝐱∼𝒩​\(σ⁡\(𝐰¯​𝐱−β¯\)​σ​\(𝐯¯​𝐱−γ¯\)\)f\(\(\{\{\\mathbf\{w\}\}\},\\beta\),\(\{\{\\mathbf\{v\}\}\},\\gamma\)\)=\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\\mathbb\{E\}\_\{\{\\mathbf\{x\}\}\\sim\\mathcal\{N\}\}\(\\sigma\(\\overline\{\{\{\\mathbf\{w\}\}\}\}\{\\mathbf\{x\}\}\-\\overline\{\\beta\}\)\\sigma\(\\overline\{\{\{\\mathbf\{v\}\}\}\}\{\\mathbf\{x\}\}\-\\overline\{\\gamma\}\)\)
###### Remark 3\.2\.

In future, we regardβ¯,γ¯\\overline\{\\beta\},\\overline\{\\gamma\}as the \(normalized\) bias parameters\. This will not change the critical point structure but gives a significant simplification of computations and formulas\. Of course, sinceβ=‖𝐰‖​β¯\\beta=\\\|\{\{\\mathbf\{w\}\}\}\\\|\\overline\{\\beta\}, we can always recover the biasβ\\beta\.✠\\maltese

If𝐰,𝐯\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}are parallel and not zero, computations are elementary, see Lemmas[3\.5](https://arxiv.org/html/2609.30661#S3.Thmlemma5),[3\.6](https://arxiv.org/html/2609.30661#S3.Thmlemma6)\. So assume𝐰,𝐯\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}are not parallel; in particular,𝐰,𝐯≠𝟎\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\\neq\\mathbf\{0\}\. Choose the orthogonal coordinate system on the 2\-plane spanned by\{𝐰,𝐯\}\\\{\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\\\}so that

𝐯¯=\(1,0,…,0\),𝐰¯=\(cos⁡\(θ\),sin⁡\(θ\),0,…,0\)\\overline\{\{\{\\mathbf\{v\}\}\}\}=\(1,0,\\ldots,0\),\\quad\\overline\{\{\{\\mathbf\{w\}\}\}\}=\(\\cos\(\\theta\),\\sin\(\\theta\),0,\\ldots,0\)whereθ=θ𝐰,𝐯∈\(0,π\)\\theta=\\theta\_\{\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\}\\in\(0,\\pi\)is the angle between𝐰\{\{\\mathbf\{w\}\}\}and𝐯\{\{\\mathbf\{v\}\}\}defined by

θ=cos−1⁡\(⟨𝐰,𝐯⟩‖𝐰‖​‖𝐯‖\)\\theta=\\cos^\{\-1\}\\left\(\\frac\{\\langle\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\\rangle\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\\right\)Given𝐱∈ℝd\{\\mathbf\{x\}\}\\in\{\\mathbb\{R\}\}^\{d\},

𝐰𝐱=\|𝐰\|\(x1​cos⁡\(θ\)\+x2​sin⁡\(θ\)\),𝐯𝐱=‖𝐯‖​x1\{\{\\mathbf\{w\}\}\}\{\\mathbf\{x\}\}=\\\|\{\{\\mathbf\{w\}\}\}\\\|\(x\_\{1\}\\cos\(\\theta\)\+x\_\{2\}\\sin\(\\theta\)\),\\;\\;\{\{\\mathbf\{v\}\}\}\{\\mathbf\{x\}\}=\\\|\{\{\\mathbf\{v\}\}\}\\\|x\_\{1\}The termσ⁡\(𝐰⋅𝐱−β\)​σ​\(𝐯⋅𝐱−γ\)\\sigma\(\{\{\\mathbf\{w\}\}\}\\cdot\{\\mathbf\{x\}\}\-\\beta\)\\sigma\(\{\{\\mathbf\{v\}\}\}\\cdot\{\\mathbf\{x\}\}\-\\gamma\)contributes to the expectation iffσ⁡\(𝐰⋅𝐱−β\),σ⁡\(𝐯⋅𝐱−γ\)\\sigma\(\{\{\\mathbf\{w\}\}\}\\cdot\{\\mathbf\{x\}\}\-\\beta\),\\sigma\(\{\{\\mathbf\{v\}\}\}\\cdot\{\\mathbf\{x\}\}\-\\gamma\)are both strictly positive\. Setting

F⁡\(x1,x2\)=\(x1​cos⁡\(θ\)\+x2​sin⁡\(θ\)−β¯\)​\(x1−γ¯\)F\(x\_\{1\},x\_\{2\}\)=\(x\_\{1\}\\cos\(\\theta\)\+x\_\{2\}\\sin\(\\theta\)\-\\bar\{\\beta\}\)\(x\_\{1\}\-\\overline\{\\gamma\}\)andp𝒩=e−x12\+x222p\_\{\\mathcal\{N\}\}=e^\{\-\\frac\{x\_\{1\}^\{2\}\+x\_\{2\}^\{2\}\}\{2\}\}\(so∫ℝ2p𝒩​d​x1​d​x2=2​π\\int\_\{\{\\mathbb\{R\}\}^\{2\}\}p\_\{\\mathcal\{N\}\}dx\_\{1\}dx\_\{2\}=2\\pi\), we have

\(3\.4\)f⁡\(𝐰,𝐯\)=‖𝐰‖​‖𝐯‖2​π​∫DθF⁡\(x1,x2\)​p𝒩​\(x1,x2\)​d​x1​d​x2,f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)=\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\pi\}\\int\_\{D\_\{\\theta\}\}F\(x\_\{1\},x\_\{2\}\)p\_\{\\mathcal\{N\}\}\(x\_\{1\},x\_\{2\}\)dx\_\{1\}dx\_\{2\},where

Dθ=D=\{\(x1,x2\)∈ℝ2\|x1cos\(θ\)\+x2sin\(θ\)−β¯,x1−γ¯≥0\}D\_\{\\theta\}=D=\\\{\(x\_\{1\},x\_\{2\}\)\\in\{\\mathbb\{R\}\}^\{2\}\\mbox\{$\\;\|\\;$\}x\_\{1\}\\cos\(\\theta\)\+x\_\{2\}\\sin\(\\theta\)\-\\bar\{\\beta\},x\_\{1\}\-\\overline\{\\gamma\}\\geq 0\\\}
In figure[1](https://arxiv.org/html/2609.30661#S3.F1), we show the domain of integration ifγ¯<0<β¯\\overline\{\\gamma\}<0<\\overline\{\\beta\}andθ∈\(0,π/2\]\\theta\\in\(0,\\pi/2\]\. The boundary ofDDis defined by the linesx1=γ¯x\_\{1\}=\\overline\{\\gamma\}andx1​cos⁡\(θ\)\+x2​sin⁡\(θ\)=β¯x\_\{1\}\\cos\(\\theta\)\+x\_\{2\}\\sin\(\\theta\)=\\overline\{\\beta\}\(denotedLLin the figure\)\.

![Refer to caption](https://arxiv.org/html/2609.30661v1/D1.png)Figure 1\.The region of integration in thex1,x2x\_\{1\},x\_\{2\}\-plane with boundary shown in red andb¯<0<β¯\\bar\{b\}<0<\\bar\{\\beta\}andθ∈\(0,π/2\]\\theta\\in\(0,\\pi/2\]\. Asθ\\thetaincreases, the lineLLdefined byx1​cos⁡\(θ\)\+x2​sin⁡\(θ\)=β¯x\_\{1\}\\cos\(\\theta\)\+x\_\{2\}\\sin\(\\theta\)=\\bar\{\\beta\}rotates counter clockwise so that whenθ∈\(π/2,π\)\\theta\\in\(\\pi/2,\\pi\),LLintersects the negativex1x\_\{1\}\-axis\.###### Remarks 3\.3\.

\(1\) It is clear from \([3\.4](https://arxiv.org/html/2609.30661#S3.E4)\) that ifθ∈\(0,π\)\\theta\\in\(0,\\pi\), thenf⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)is a continuous function of𝐰,𝐯,β¯\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\},\\overline\{\\beta\}andγ¯\\overline\{\\gamma\}\. Just as for zero bias, it is easy to see thatffwill not be differentiable with respect to𝐰\{\{\\mathbf\{w\}\}\}if𝐰=0\{\{\\mathbf\{w\}\}\}=0and will be analytic \(in𝐰,𝐯,β¯\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\},\\overline\{\\beta\}andγ¯\\overline\{\\gamma\}\) provided that𝐰,𝐯\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}are not parallel:‖𝐰‖​‖𝐯‖≠⟨𝐰,𝐯⟩\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\\neq\\langle\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\\rangle\(which implies𝐰,𝐯\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}are non\-zero\)\. For differentiability see also Remarks[3\.3](https://arxiv.org/html/2609.30661#S3.Thmlemma3)\(3\)\. \(2\) Suppose that𝐰,𝐯\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}are are non\-zero and parallel withθ=0\\theta=0\. Observe that asθ→0\+\\theta\{\\rightarrow\}0\+, the lineLLconverges tox1=β¯x\_\{1\}=\\overline\{\\beta\}\. Consequently,

D0=\{\(x1,x2\)\|x1≥max⁡\{γ¯,β¯\}\}D\_\{0\}=\\\{\(x\_\{1\},x\_\{2\}\)\\mbox\{$\\;\|\\;$\}x\_\{1\}\\geq\\max\\\{\\overline\{\\gamma\},\\overline\{\\beta\}\\\}\\\}and*not*x1≥γ¯x\_\{1\}\\geq\\overline\{\\gamma\}unlessγ¯≥β¯\\overline\{\\gamma\}\\geq\\overline\{\\beta\}\. If instead we consider the limiting domainDπD\_\{\\pi\}obtained by lettingθ→π−\\theta\{\\rightarrow\}\\pi\-, thenLLconverges tox1=−β¯x\_\{1\}=\-\\overline\{\\beta\}and

Dπ=\{∅,if​γ¯\+β¯\>0\[γ¯,−β¯\]×ℝ⊂ℝ2,ifγ¯\+β¯≤0\.D\_\{\\pi\}=\\begin\{cases\}&\\emptyset,\\;\\text\{if \}\\overline\{\\gamma\}\+\\overline\{\\beta\}\>0\\\\ &\[\\overline\{\\gamma\},\-\\overline\{\\beta\}\]\\times\{\\mathbb\{R\}\}\\subset\{\\mathbb\{R\}\}^\{2\},\\;\\text\{if \}\\overline\{\\gamma\}\+\\overline\{\\beta\}\\leq 0\.\\end\{cases\}In particular,f⁡\(𝐰,𝐯\)=0f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)=0ifθ=π\\theta=\\piandγ¯\+β¯≥0\\overline\{\\gamma\}\+\\overline\{\\beta\}\\geq 0\. However, although the limits ofDθD\_\{\\theta\}asθ→0\+,π−\\theta\{\\rightarrow\}0\+,\\pi\-might suggest a discontinuity, it will be shown later that‖𝐰‖​‖𝐯‖​∫DθF​p𝒩\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\\int\_\{D\_\{\\theta\}\}Fp\_\{\\mathcal\{N\}\}is a continuous function of𝐰,𝐯,β,γ\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\},\\beta,\\gamma\. In particular, the limit of∫DθF​p𝒩\\int\_\{D\_\{\\theta\}\}Fp\_\{\\mathcal\{N\}\}asθ→0\+\\theta\{\\rightarrow\}0\+exists and is equal to∫D0F​p𝒩\\int\_\{D\_\{0\}\}Fp\_\{\\mathcal\{N\}\}; similarly for the limit asθ→π−\\theta\{\\rightarrow\}\\pi\-\. \(3\) In the case of zero bias, Safran, Yelundai & Shamir\[[23](https://arxiv.org/html/2609.30661#bib.bib23)\]prove that the lossℒ\\mathcal\{L\}isC2C^\{2\}at𝐖∈M⁡\(k,d\)\{\{\\mathbf\{W\}\}\}\\in M\(k,d\)provided that no row of𝐖\{\{\\mathbf\{W\}\}\}is zero andk≥dk\\geq d\. However, ifk\>dk\>d, there will be zero eigenvalues at critical points defining the global minima*op\. cit\.*,\[[5](https://arxiv.org/html/2609.30661#bib.bib5), Appendix A\.7\]\. This is so even though the Hessian is continuous and all eigenvalues are positive\. This degeneracy is quite apparent when doing numerical investigations in the over\-parametrized case\. Of course, all eigenvalues at a global minimum are positive\. However, ifk≥d\+1k\\geq d\+1, there typically exist many non\-degenerate critical points including spurious minima\. \(4\) Both domainDθD\_\{\\theta\}and integrandFFin \([3\.4](https://arxiv.org/html/2609.30661#S3.E4)\) depend on the choice𝐯=\|𝐯\|\(1,0,…,0\)\{\{\\mathbf\{v\}\}\}=\\\|\{\{\\mathbf\{v\}\}\}\\\|\(1,0,\\ldots,0\)\. If instead we take𝐰=\|𝐰\|\(1,0,…,0\)\{\{\\mathbf\{w\}\}\}=\\\|\{\{\\mathbf\{w\}\}\}\\\|\(1,0,\\ldots,0\), thenθ\\thetais unchanged and𝐯¯=\|𝐯\|\(cos⁡\(θ\),sin⁡\(θ\),0,…,0\)\\overline\{\{\{\\mathbf\{v\}\}\}\}=\\\|\{\{\\mathbf\{v\}\}\}\\\|\(\\cos\(\\theta\),\\sin\(\\theta\),0,\\ldots,0\)\. The effect is to swapβ¯\\overline\{\\beta\}andγ¯\\overline\{\\gamma\}in the expressions forFFandDθD\_\{\\theta\}\. This leads to two terms in the expression we obtain forf⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)depending on which vector was chosen to be a multiple of the unit vector \(of course, the value off⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)is unchanged\)\. The associated identity leads to a surprising and important symmetry in\(β¯,γ¯\)\(\\overline\{\\beta\},\\overline\{\\gamma\}\)that underpins much of our analysis\.✠\\maltese

###### Example 3\.4\.

If𝐯=𝐰\{\{\\mathbf\{v\}\}\}=\{\{\\mathbf\{w\}\}\}andβ=γ\\beta=\\gamma\. thenθ=0\\theta=0,β¯=γ¯\\overline\{\\beta\}=\\overline\{\\gamma\}andD0=\[γ¯,∞\)×ℝD\_\{0\}=\[\\overline\{\\gamma\},\\infty\)\\times\{\\mathbb\{R\}\}\. The problem of computingffreduces to computing12​π​∫β¯∞\(x2−2​β¯​x−β¯2\)​e−x22​𝑑x\\frac\{1\}\{\\sqrt\{2\\pi\}\}\\int\_\{\\overline\{\\beta\}\}^\{\\infty\}\(x^\{2\}\-2\\overline\{\\beta\}x\-\\overline\{\\beta\}^\{2\}\)e^\{\-\\frac\{x^\{2\}\}\{2\}\}dxgiving

\(3\.5\)f⁡\(𝐰,𝐰\)=‖𝐰‖2​\(β¯2\+1\)2​erfc​\(β¯2\)−‖𝐰‖22​π​β¯​e−β¯22,f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{w\}\}\}\)=\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|^\{2\}\(\\overline\{\\beta\}^\{2\}\+1\)\}\{2\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|^\{2\}\}\{\\sqrt\{2\\pi\}\}\\overline\{\\beta\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\},where*erfc*​\(\)\{\\text\{\\emph\{erfc\}\}\(\)\}is the complementary error function \(see Appendix[A](https://arxiv.org/html/2609.30661#A1)\)\. Sinceβ¯=γ¯\\overline\{\\beta\}=\\overline\{\\gamma\},D0=limθ→0\+Dθ⊂ℝ2D\_\{0\}=\\lim\_\{\\theta\{\\rightarrow\}0\+\}D\_\{\\theta\}\\subset\{\\mathbb\{R\}\}^\{2\}\.※

###### Lemma 3\.5\.

Suppose that𝐰,𝐯\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}are parallel and non\-zero withθ=0\\theta=0\. Ifβ,γ∈ℝ\\beta,\\gamma\\in\{\\mathbb\{R\}\}, then

\(3\.6\)f⁡\(𝐰,𝐯\)=‖𝐰‖​‖𝐯‖​\(β¯​γ¯\+1\)2​erfc​\(M2\)−m​‖𝐯‖​‖𝐰‖2​π​e−M22,f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)=\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\(\\overline\{\\beta\}\\overline\{\\gamma\}\+1\)\}\{2\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{M\}\{\\sqrt\{2\}\}\}\\right\)\}\-\\frac\{m\\\|\{\{\\mathbf\{v\}\}\}\\\|\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\{\\sqrt\{2\\pi\}\}e^\{\-\\frac\{M^\{2\}\}\{2\}\},whereM=max⁡\{γ¯,β¯\}M=\\max\\\{\\overline\{\\gamma\},\\overline\{\\beta\}\\\},m=min⁡\{γ¯,β¯\}m=\\min\\\{\\overline\{\\gamma\},\\overline\{\\beta\}\\\}\.

###### Proof\.

As in Example[3\.4](https://arxiv.org/html/2609.30661#S3.Thmlemma4), the result follows by evaluating the elementary integral∫max⁡\{γ¯,β¯\}∞\(x12−\(γ¯\+β¯\)​x1\+β¯​γ¯\)​e−x22​𝑑x\\int\_\{\\max\\\{\\overline\{\\gamma\},\\overline\{\\beta\}\\\}\}^\{\\infty\}\(x\_\{1\}^\{2\}\-\(\\overline\{\\gamma\}\+\\overline\{\\beta\}\)x\_\{1\}\+\\overline\{\\beta\}\\overline\{\\gamma\}\)e^\{\-\\frac\{x^\{2\}\}\{2\}\}dx\. ∎

###### Lemma 3\.6\.

Suppose that𝐰,𝐯\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}are parallel and non\-zero withθ=π\\theta=\\pi\. Ifβ,γ∈ℝ\\beta,\\gamma\\in\{\\mathbb\{R\}\}, then

1. \(1\)Forγ¯≤−β¯\\overline\{\\gamma\}\\leq\-\\overline\{\\beta\}, \(3\.7\)f⁡\(𝐰,𝐯\)=‖𝐰‖​‖𝐯‖​\(1−β¯​γ¯\)4​\[erf​\(β¯2\)\+erf​\(γ¯2\)\]−‖𝐰‖​‖𝐯‖2​2​π​\[γ¯​e−β¯22\+β¯​e−γ¯22\]\\begin\{aligned\} f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)=&\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\(1\-\\overline\{\\beta\}\\overline\{\\gamma\}\)\}\{4\}\\left\[\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ &\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\left\[\\overline\{\\gamma\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\+\\overline\{\\beta\}e^\{\-\\frac\{\\overline\{\\gamma\}^\{2\}\}\{2\}\}\\right\]\\end\{aligned\}
2. \(2\)Forγ¯≥−β¯\\overline\{\\gamma\}\\geq\-\\overline\{\\beta\},f⁡\(𝐰,𝐯\)=0f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)=0\.

###### Proof\.

Similar to that of Lemma[3\.5](https://arxiv.org/html/2609.30661#S3.Thmlemma5)using Remarks[3\.3](https://arxiv.org/html/2609.30661#S3.Thmlemma3)\(2\)\. ∎

###### Remarks 3\.7\.

\(1\) Lemmas[3\.5](https://arxiv.org/html/2609.30661#S3.Thmlemma5),[3\.6](https://arxiv.org/html/2609.30661#S3.Thmlemma6)imply thatf⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)is a continuous function ofβ¯,γ¯\\overline\{\\beta\},\\overline\{\\gamma\}\(andβ,γ\\beta,\\gamma\) onD0D\_\{0\}andDπD\_\{\\pi\}\. It may be shown thatffisC1C^\{1\}inβ¯,γ¯\\overline\{\\beta\},\\overline\{\\gamma\}on these subspaces but notC2C^\{2\}\. \(2\) Note the vanishing offfifγ¯≥−β¯\\overline\{\\gamma\}\\geq\-\\overline\{\\beta\}in Lemma[3\.6](https://arxiv.org/html/2609.30661#S3.Thmlemma6)\(2\)\. If the network is not biased, thenf⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)is zero iff𝐰\{\{\\mathbf\{w\}\}\}is a non\-zero negative multiple of𝐯\{\{\\mathbf\{v\}\}\}\.✠\\maltese

### 3\.1\.Computation offfif the network is not biased⋆

Before completing the computation off⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)for biased ReLU networks in section[3\.2](https://arxiv.org/html/2609.30661#S3.SS2)–[6](https://arxiv.org/html/2609.30661#S6), the reader may find it helpful to review the computation for bias free networks without the use of polar coordinates\[[10](https://arxiv.org/html/2609.30661#bib.bib10)\]\.

Ifβ=γ=0\\beta=\\gamma=0, then

f⁡\(𝐰,𝐯\)\\displaystyle f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)=\\displaystyle=‖𝐰‖​‖𝐯‖2​π​∫D\(x12​cos⁡\(θ\)\+x1​x2​sin⁡\(θ\)\)​e−x12\+x222​d​x1​d​x2\\displaystyle\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\pi\}\\int\_\{D\}\(x\_\{1\}^\{2\}\\cos\(\\theta\)\+x\_\{1\}x\_\{2\}\\sin\(\\theta\)\)e^\{\-\\frac\{x\_\{1\}^\{2\}\+x\_\{2\}^\{2\}\}\{2\}\}dx\_\{1\}dx\_\{2\}=\\displaystyle=‖𝐰‖​‖𝐯‖2​π​∫0∞\(∫x~2∞x12​cos⁡\(θ\)\+x1​x2​sin⁡\(θ\)​e−x222​d​x2\)​e−x122​d​x1,\\displaystyle\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\pi\}\\int\_\{0\}^\{\\infty\}\\left\(\\int\_\{\\widetilde\{x\}\_\{2\}\}^\{\\infty\}x\_\{1\}^\{2\}\\cos\(\\theta\)\+x\_\{1\}x\_\{2\}\\sin\(\\theta\)e^\{\-\\frac\{x\_\{2\}^\{2\}\}\{2\}\}dx\_\{2\}\\right\)e^\{\-\\frac\{x\_\{1\}^\{2\}\}\{2\}\}dx\_\{1\},wherex~2=−x1​cot⁡\(θ\)\\widetilde\{x\}\_\{2\}=\-x\_\{1\}\\cot\(\\theta\)\(see Figure[1](https://arxiv.org/html/2609.30661#S3.F1)\)\. We evaluate the double integral directly, rather than use the transformation to polar coordinates, so that we can see how the angle term\(π−θ\)​cos⁡\(θ\)\(\\pi\-\\theta\)\\cos\(\\theta\)enters into the equation forf⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)\. Of course, there is no scope for a simple \(or useful\) application of polar coordinates ifβ​γ≠0\\beta\\gamma\\neq 0\.

Write

∫x~2∞x12​cos⁡\(θ\)\+x1​x2​sin⁡\(θ\)​e−x222​d​x2=cos⁡\(θ\)​x12​A11\+sin⁡\(θ\)​x1​A12\\int\_\{\\widetilde\{x\}\_\{2\}\}^\{\\infty\}x\_\{1\}^\{2\}\\cos\(\\theta\)\+x\_\{1\}x\_\{2\}\\sin\(\\theta\)e^\{\-\\frac\{x\_\{2\}^\{2\}\}\{2\}\}dx\_\{2\}=\\cos\(\\theta\)x\_\{1\}^\{2\}A\_\{11\}\+\\sin\(\\theta\)x\_\{1\}A\_\{12\}where

A11=∫x~2∞e−x222​d​x2A12=∫x~2∞x2​e−x222​d​x2A\_\{11\}=\\int\_\{\\widetilde\{x\}\_\{2\}\}^\{\\infty\}e^\{\-\\frac\{x\_\{2\}^\{2\}\}\{2\}\}dx\_\{2\}\\quad A\_\{12\}=\\int\_\{\\widetilde\{x\}\_\{2\}\}^\{\\infty\}x\_\{2\}e^\{\-\\frac\{x\_\{2\}^\{2\}\}\{2\}\}dx\_\{2\}An elementary integration gives

A12=∫x~2∞x2​e−x222​d​x2=e−x12​cot2⁡\(θ\)2A\_\{12\}=\\int\_\{\\widetilde\{x\}\_\{2\}\}^\{\\infty\}x\_\{2\}e^\{\-\\frac\{x\_\{2\}^\{2\}\}\{2\}\}dx\_\{2\}=e^\{\-\\frac\{x\_\{1\}^\{2\}\\cot^\{2\}\(\\theta\)\}\{2\}\}and so the contribution to the integral overDDwill be

\(3\.8\)sin⁡\(θ\)​∫0∞x1​e−x12​cot2⁡\(θ\)2​e−x122​d​x1=sin⁡\(θ\)​∫0∞x1​e−x12​csc2⁡\(θ\)2​d​x1=sin3⁡\(θ\)\.\\sin\(\\theta\)\\int\_\{0\}^\{\\infty\}x\_\{1\}e^\{\-\\frac\{x\_\{1\}^\{2\}\\cot^\{2\}\(\\theta\)\}\{2\}\}e^\{\-\\frac\{x\_\{1\}^\{2\}\}\{2\}\}dx\_\{1\}=\\sin\(\\theta\)\\int\_\{0\}^\{\\infty\}x\_\{1\}e^\{\-\\frac\{x\_\{1\}^\{2\}\\csc^\{2\}\(\\theta\)\}\{2\}\}dx\_\{1\}=\\sin^\{3\}\(\\theta\)\.Turning to the first term in the integral overDD,

A11=π2​erfc​\(−x1​cot⁡\(θ\)2\)=π2​\[1−erf​\(−x1​cot⁡\(θ\)2\)\]A\_\{11\}=\\sqrt\{\\frac\{\\pi\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\-x\_\{1\}\\cot\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}=\\sqrt\{\\frac\{\\pi\}\{2\}\}\\left\[1\-\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-x\_\{1\}\\cot\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]and so we must evaluate

F⁡\(θ\)\\displaystyle F\(\\theta\)=\\displaystyle=π2​∫0∞x12​\(1−erf​\(−x1​cot⁡\(θ\)2\)\)​e−x122​d​x1\\displaystyle\\sqrt\{\\frac\{\\pi\}\{2\}\}\\int\_\{0\}^\{\\infty\}x\_\{1\}^\{2\}\\big\(1\-\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-x\_\{1\}\\cot\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}\\big\)e^\{\-\\frac\{x\_\{1\}^\{2\}\}\{2\}\}dx\_\{1\}=\\displaystyle=π2−π2​∫0∞x12​erf​\(−x1​cot⁡\(θ\)2\)​e−x122​d​x1,\\displaystyle\\frac\{\\pi\}\{2\}\-\\sqrt\{\\frac\{\\pi\}\{2\}\}\\int\_\{0\}^\{\\infty\}x\_\{1\}^\{2\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-x\_\{1\}\\cot\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}e^\{\-\\frac\{x\_\{1\}^\{2\}\}\{2\}\}dx\_\{1\},where the last equality follows from \([A\.65](https://arxiv.org/html/2609.30661#A1.E65)\)\. Sincecot⁡\(π/2\)=0\\cot\(\\pi/2\)=0,

\(3\.9\)F⁡\(π/2\)=π2F\(\\pi/2\)=\\frac\{\\pi\}\{2\}By \([A\.56](https://arxiv.org/html/2609.30661#A1.E56)\),erf​\(a​x\)=2π​∫0a​xe−t2​𝑑t\{\\text\{\\rm erf\}\(\{ax\}\)\}=\\frac\{2\}\{\\sqrt\{\\pi\}\}\\int\_\{0\}^\{ax\}e^\{\-t^\{2\}\}dtand so sincedd​θ​cot⁡\(θ\)=−csc2⁡\(θ\)\\frac\{d\}\{d\\theta\}\\cot\(\\theta\)=\-\\csc^\{2\}\(\\theta\)

∂∂θ​erf​\(−x1​cot⁡\(θ\)2\)=2π​x1​csc2⁡\(θ\)​e−x12​cot2⁡\(θ\)2\\frac\{\\partial\}\{\\partial\\theta\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-x\_\{1\}\\cot\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}=\\sqrt\{\\frac\{2\}\{\\pi\}\}x\_\{1\}\\csc^\{2\}\(\\theta\)e^\{\\frac\{\-x^\{2\}\_\{1\}\\cot^\{2\}\(\\theta\)\}\{2\}\}Using1\+cot2⁡\(θ\)=csc2⁡\(θ\)1\+\\cot^\{2\}\(\\theta\)=\\csc^\{2\}\(\\theta\),

F′​\(θ\)\\displaystyle F^\{\\prime\}\(\\theta\)=\\displaystyle=−csc2\(θ\)∫0∞x13e−x12​csc2⁡\(θ\)2dx1\\displaystyle\-\\csc^\{2\}\(\\theta\)\\int\_\{0\}^\{\\infty\}x\_\{1\}^\{3\}e^\{\-\\frac\{x\_\{1\}^\{2\}\\csc^\{2\}\(\\theta\)\}\{2\}\}dx\_\{1\}=\\displaystyle=−2​sin2⁡\(θ\)=−1\+cos⁡\(2​θ\)\.\\displaystyle\-2\\sin^\{2\}\(\\theta\)=\-1\+\\cos\(2\\theta\)\.where the last line follows from \([A\.66](https://arxiv.org/html/2609.30661#A1.E66)\)\. Integrating and usingF⁡\(π/2\)=π2F\(\\pi/2\)=\\frac\{\\pi\}\{2\}, gives

F⁡\(θ\)=\(π−θ\)\+sin⁡\(θ\)​cos⁡\(θ\)F\(\\theta\)=\(\\pi\-\\theta\)\+\\sin\(\\theta\)\\cos\(\\theta\)Combining the computations, the integral involvingA12A\_\{12\}\(resp\.A11A\_\{11\}\) contributessin3⁡\(θ\)\\sin^\{3\}\(\\theta\)\(resp\.\(π−θ\)​cos⁡\(θ\)\+sin⁡\(θ\)​cos2⁡\(θ\)\(\\pi\-\\theta\)\\cos\(\\theta\)\+\\sin\(\\theta\)\\cos^\{2\}\(\\theta\)\) and so

∫D\(x12​cos⁡\(θ\)\+x1​x2​sin⁡\(θ\)\)​e−x12\+x222​d​x1​d​x2\\displaystyle\\int\_\{D\}\(x\_\{1\}^\{2\}\\cos\(\\theta\)\+x\_\{1\}x\_\{2\}\\sin\(\\theta\)\)e^\{\-\\frac\{x\_\{1\}^\{2\}\+x\_\{2\}^\{2\}\}\{2\}\}dx\_\{1\}dx\_\{2\}=\\displaystyle=sin⁡\(θ\)\+\(π−θ\)​cos⁡\(θ\)\\displaystyle\\sin\(\\theta\)\+\(\\pi\-\\theta\)\\cos\(\\theta\)
###### Remarks 3\.8\.

\(1\) The key to the computation lies in evaluating the integral∫0∞erf​\(−x​cot⁡\(θ\)2\)​x2​e−x22​𝑑x\\int\_\{0\}^\{\\infty\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-x\\cot\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}x^\{2\}e^\{\-\\frac\{x^\{2\}\}\{2\}\}dx\. Using integration by parts, this is easily reduced to evaluating∫0∞erf​\(−x​cot⁡\(θ\)2\)​e−x22​𝑑x\\int\_\{0\}^\{\\infty\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-x\\cot\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}e^\{\-\\frac\{x^\{2\}\}\{2\}\}dxwhich can be done by the same method used above\. \(2\) For the bias free network, it sufficed to compute the integrals ofx1​x2x\_\{1\}x\_\{2\}andx12x\_\{1\}^\{2\}overDDwith respect to the normal distributionp𝒩p\_\{\\mathcal\{N\}\}\. When we add bias, there are five integrals to compute overDD:x12,x1​x2,x1,x2x\_\{1\}^\{2\},x\_\{1\}x\_\{2\},x\_\{1\},x\_\{2\}and11\. Here the more difficult integrals are those ofx12x\_\{1\}^\{2\}and11both of which use Owen’sTT\-function \(see sections[4](https://arxiv.org/html/2609.30661#S4),[4\.1](https://arxiv.org/html/2609.30661#S4.SS1)\)\.✠\\maltese

### 3\.2\.Evaluation of the defining integrals forf⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\): I

Our approach will be to represent∫D\\int\_\{D\}as the double integral

∫γ¯∞\(∫x~2∞F⁡\(x1,x2\)​e−x222​d​x2\)​e−x122​d​x1,\\int\_\{\\overline\{\\gamma\}\}^\{\\infty\}\\left\(\\int\_\{\\widetilde\{x\}\_\{2\}\}^\{\\infty\}F\(x\_\{1\},x\_\{2\}\)e^\{\-\\frac\{x\_\{2\}^\{2\}\}\{2\}\}dx\_\{2\}\\right\)e^\{\-\\frac\{x\_\{1\}^\{2\}\}\{2\}\}dx\_\{1\},wherex~2=β¯​csc⁡\(θ\)−x1​cot⁡\(θ\)\\widetilde\{x\}\_\{2\}=\\overline\{\\beta\}\\,\\csc\(\\theta\)\-x\_\{1\}\\cot\(\\theta\)\(see Figure[1](https://arxiv.org/html/2609.30661#S3.F1)\),θ∈\(0,π\)\\theta\\in\(0,\\pi\), and

F⁡\(x1,x2\)=\(x1​cos⁡\(θ\)\+x2​sin⁡\(θ\)−β¯\)​\(x1−γ¯\)\.F\(x\_\{1\},x\_\{2\}\)=\(x\_\{1\}\\cos\(\\theta\)\+x\_\{2\}\\sin\(\\theta\)\-\\overline\{\\beta\}\)\(x\_\{1\}\-\\overline\{\\gamma\}\)\.In order to compute the double integral, it suffices, by linearity of the integral, to compute the integral overDDfor each of the polynomial functionsx12,x1​x2,x1,x2,1x\_\{1\}^\{2\},x\_\{1\}x\_\{2\},x\_\{1\},x\_\{2\},1since

F⁡\(x1,x2\)=cos⁡\(θ\)​x12\+sin⁡\(θ\)​x1​x2−\(γ¯​cos⁡\(θ\)\+β¯\)​x1−γ¯​sin⁡\(θ\)​x2\+β¯​γ¯F\(x\_\{1\},x\_\{2\}\)=\\cos\(\\theta\)x\_\{1\}^\{2\}\+\\sin\(\\theta\)x\_\{1\}x\_\{2\}\-\(\\overline\{\\gamma\}\\cos\(\\theta\)\+\\overline\{\\beta\}\)x\_\{1\}\-\\overline\{\\gamma\}\\sin\(\\theta\)x\_\{2\}\+\\overline\{\\beta\}\\overline\{\\gamma\}
###### Example 3\.9\.

Ifθ=π/2\\theta=\\pi/2, the integral ofFFis easily computed since∫DF=∫γ¯∞\(∫β¯∞\[x1​x2−β¯​x1−γ¯​x2\+γ¯​β¯\]​e−x222​d​x2\)​e−x122​d​x1\\int\_\{D\}F=\\int\_\{\\overline\{\\gamma\}\}^\{\\infty\}\\left\(\\int\_\{\\overline\{\\beta\}\}^\{\\infty\}\[x\_\{1\}x\_\{2\}\-\\overline\{\\beta\}x\_\{1\}\-\\overline\{\\gamma\}x\_\{2\}\+\\overline\{\\gamma\}\\overline\{\\beta\}\]e^\{\-\\frac\{x\_\{2\}^\{2\}\}\{2\}\}dx\_\{2\}\\right\)e^\{\-\\frac\{x\_\{1\}^\{2\}\}\{2\}\}dx\_\{1\}and

1. \(1\)∫Dx1​x2=e−β¯2\+γ¯22\\int\_\{D\}x\_\{1\}x\_\{2\}=e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\+\\overline\{\\gamma\}^\{2\}\}\{2\}\}\.
2. \(2\)∫Dx1=π2​e−γ¯22​erfc​\(β¯2\)\\int\_\{D\}x\_\{1\}=\\sqrt\{\\frac\{\\pi\}\{2\}\}e^\{\-\\frac\{\\overline\{\\gamma\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\.
3. \(3\)∫Dx2=π2​e−β¯22​erfc​\(γ¯2\)\\int\_\{D\}x\_\{2\}=\\sqrt\{\\frac\{\\pi\}\{2\}\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\.
4. \(4\)∫D1=π2​erfc​\(γ¯2\)​erfc​\(β¯2\)\\int\_\{D\}1=\\frac\{\\pi\}\{2\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\.

Hence ifθ=π/2\\theta=\\pi/2,f⁡\(𝐰,𝐯\)=‖𝐰‖​‖𝐯‖2​π​∫DFf\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)=\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\pi\}\\int\_\{D\}Fis given by

f⁡\(𝐰,𝐯\)=‖𝐰‖​‖𝐯‖​β¯​γ¯4​erfc​\(γ¯2\)​erfc​\(β¯2\)−‖𝐰‖​‖𝐯‖2​2​π​\(β¯​e−γ¯22​erfc​\(β¯2\)\+γ​e−β¯22​erfc​\(γ¯2\)\)\+‖𝐰‖​‖𝐯‖2​π​e−β¯2\+γ¯22\\begin\{aligned\} f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)=\\;\\;&\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\\overline\{\\beta\}\\overline\{\\gamma\}\}\{4\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\\\ &\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\left\(\\overline\{\\beta\}e^\{\-\\frac\{\\overline\{\\gamma\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\gamma e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\)\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\pi\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\+\\overline\{\\gamma\}^\{2\}\}\{2\}\}\\end\{aligned\}The symmetryf⁡\(𝐰,𝐯\)=f⁡\(𝐯,𝐰\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)=f\(\{\{\\mathbf\{v\}\}\},\{\{\\mathbf\{w\}\}\}\)is reflected in the\(β¯,γ¯\)↔\(γ¯,β¯\)\(\\overline\{\\beta\},\\overline\{\\gamma\}\)\\leftrightarrow\(\\overline\{\\gamma\},\\overline\{\\beta\}\)invariance of the formula\.※

In the remainder of this section∫D\\int\_\{D\}is computed for the integrandsx2,x1,x1​x2x\_\{2\},x\_\{1\},x\_\{1\}x\_\{2\}\. Results are given in terms of standard functions: the exponential and error function\. Computations are valid forθ∈\(0,π\)\\theta\\in\(0,\\pi\)\. The limiting casesθ→0\+,π−\\theta\{\\rightarrow\}0\+,\\pi\-are given only for the integrandx1x\_\{1\}since the contribution to the integral ofFFfrom the the termsx1​x2​sin⁡\(θ\),−γ¯​x2​sin⁡\(θ\)x\_\{1\}x\_\{2\}\\sin\(\\theta\),\-\\overline\{\\gamma\}x\_\{2\}\\sin\(\\theta\)is zero in the limit on account of the factorsin⁡\(θ\)\\sin\(\\theta\)\.

Throughout, we set−x1​cot⁡\(θ\)\+β¯​csc⁡\(θ\)=x~2​\(x1\)=x~2\-x\_\{1\}\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)=\\widetilde\{x\}\_\{2\}\(x\_\{1\}\)=\\widetilde\{x\}\_\{2\}\(x1x\_\{1\}is omitted if clear from the context\)\. For future reference, we record the limits ofx~2​\(x1\)\\widetilde\{x\}\_\{2\}\(x\_\{1\}\)asθ→0\+,π−\\theta\{\\rightarrow\}0\+,\\pi\-\.

limθ→0\+x~2​\(x1\)=\{\+∞,x1<β¯0,x1=β¯−∞,x1\>β¯limθ→π−x~2​\(x1\)=\{\+∞,x1\>−β¯0,x1=−β¯−∞,x1<−β¯\\lim\_\{\\theta\{\\rightarrow\}0\+\}\\widetilde\{x\}\_\{2\}\(x\_\{1\}\)=\\begin\{cases\}&\+\\infty,\\;x\_\{1\}<\\overline\{\\beta\}\\\\ &0,\\;\\;\\quad x\_\{1\}=\\overline\{\\beta\}\\\\ &\-\\infty,\\;x\_\{1\}\>\\overline\{\\beta\}\\end\{cases\}\\qquad\\lim\_\{\\theta\{\\rightarrow\}\\pi\-\}\\widetilde\{x\}\_\{2\}\(x\_\{1\}\)=\\begin\{cases\}&\+\\infty,\\;x\_\{1\}\>\-\\overline\{\\beta\}\\\\ &0,\\;x\_\{1\}=\-\\overline\{\\beta\}\\\\ &\-\\infty,\\;x\_\{1\}<\-\\overline\{\\beta\}\\end\{cases\}We frequently use the abbreviationsA=−cot⁡\(θ\)A=\-\\cot\(\\theta\)andB=csc⁡\(θ\)B=\\csc\(\\theta\)\.

###### Lemma 3\.10\.

∫Dx2=π2​e−β¯22​sin⁡\(θ\)​erfc​\(γ¯​csc⁡\(θ\)−β¯​cot⁡\(θ\)2\)\\int\_\{D\}x\_\{2\}=\\sqrt\{\\frac\{\\pi\}\{2\}\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\\sin\(\\theta\)\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\\csc\(\\theta\)\-\\overline\{\\beta\}\\cot\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}

###### Proof\.

∫Dx2=∫γ¯∞\(∫x~2∞x2​e−x222​d​x2\)​e−x122​d​x1\\int\_\{D\}x\_\{2\}=\\int\_\{\\overline\{\\gamma\}\}^\{\\infty\}\\left\(\\int\_\{\\widetilde\{x\}\_\{2\}\}^\{\\infty\}x\_\{2\}e^\{\-\\frac\{x\_\{2\}^\{2\}\}\{2\}\}dx\_\{2\}\\right\)e^\{\-\\frac\{x\_\{1\}^\{2\}\}\{2\}\}dx\_\{1\}\. Clearly,

∫x~2∞x2​e−x222​d​x2\\displaystyle\\int\_\{\\widetilde\{x\}\_\{2\}\}^\{\\infty\}x\_\{2\}e^\{\-\\frac\{x\_\{2\}^\{2\}\}\{2\}\}dx\_\{2\}=\\displaystyle=e−x~222=e−\(x1​cot⁡\(θ\)−β¯​csc⁡\(θ\)\)22\\displaystyle e^\{\-\\frac\{\\widetilde\{x\}\_\{2\}^\{2\}\}\{2\}\}=e^\{\-\\frac\{\(x\_\{1\}\\cot\(\\theta\)\-\\overline\{\\beta\}\\csc\(\\theta\)\)^\{2\}\}\{2\}\}Hence

∫Dx2=∫γ¯∞e−\(x1​cot⁡\(θ\)−β¯​csc⁡\(θ\)\)22e−x122dx1=∫γ¯∞e−\(Bx1\+Aβ¯\)2/2e−β¯22dx1\\int\_\{D\}x\_\{2\}=\\int\_\{\\overline\{\\gamma\}\}^\{\\infty\}e^\{\-\\frac\{\(x\_\{1\}\\cot\(\\theta\)\-\\overline\{\\beta\}\\csc\(\\theta\)\)^\{2\}\}\{2\}\}e^\{\-\\frac\{x\_\{1\}^\{2\}\}\{2\}\}dx\_\{1\}=\\int\_\{\\overline\{\\gamma\}\}^\{\\infty\}e^\{\-\(Bx\_\{1\}\+A\\overline\{\\beta\}\)^\{2\}/2\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}dx\_\{1\}The result follows from \([A\.63](https://arxiv.org/html/2609.30661#A1.E63)\) sincecsc⁡\(θ\)=B\>0\\csc\(\\theta\)=B\>0on\(0,π\)\(0,\\pi\)\. ∎

###### Lemma 3\.11\.

1. \(1\)Ifθ∈\(0,π\)\\theta\\in\(0,\\pi\), then ∫Dθx1=π2​e−γ¯22​erfc​\(−γ¯​cot⁡\(θ\)\+β¯​csc⁡\(θ\)2\)\+π2​e−β¯22​cos⁡\(θ\)​erfc​\(γ¯​csc⁡\(θ\)−β¯​cot⁡\(θ\)2\)\\begin\{aligned\} \\hskip 25\.29494pt\\int\_\{D\_\{\\theta\}\}x\_\{1\}=&\\sqrt\{\\frac\{\\pi\}\{2\}\}e^\{\-\\frac\{\\overline\{\\gamma\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\-\\overline\{\\gamma\}\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}\\\\ &\+\\sqrt\{\\frac\{\\pi\}\{2\}\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\\cos\(\\theta\)\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\\,\\csc\(\\theta\)\-\\overline\{\\beta\}\\cot\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}\\end\{aligned\}
2. \(2\)SetM=max⁡\{γ¯,β¯\}M=\\max\\\{\\overline\{\\gamma\},\\overline\{\\beta\}\\\}\. Ifθ=0\\theta=0, thenD0=\[M,∞\)×ℝD\_\{0\}=\[M,\\infty\)\\times\{\\mathbb\{R\}\}and ∫D0x1=limθ→0\+∫Dθx1=2​π​e−M22\\int\_\{D\_\{0\}\}x\_\{1\}=\\lim\_\{\\theta\{\\rightarrow\}0\+\}\\int\_\{D\_\{\\theta\}\}x\_\{1\}=\\sqrt\{2\\pi\}e^\{\-\\frac\{M^\{2\}\}\{2\}\}
3. \(3\)Ifθ=π\\theta=\\piandγ¯≤−β¯\\overline\{\\gamma\}\\leq\-\\overline\{\\beta\}, thenDπ=\[γ¯,−β¯\]×ℝD\_\{\\pi\}=\[\\overline\{\\gamma\},\-\\overline\{\\beta\}\]\\times\{\\mathbb\{R\}\}and ∫Dπx1=limθ→π−∫Dθx1=2​π​\[e−γ¯22−e−β¯22\]\\int\_\{D\_\{\\pi\}\}x\_\{1\}=\\lim\_\{\\theta\{\\rightarrow\}\\pi\-\}\\int\_\{D\_\{\\theta\}\}x\_\{1\}=\\sqrt\{2\\pi\}\\big\[e^\{\-\\frac\{\\overline\{\\gamma\}^\{2\}\}\{2\}\}\-e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\\big\]Ifθ=π\\theta=\\piandγ¯≥−β¯\\overline\{\\gamma\}\\geq\-\\overline\{\\beta\}, then∫Dπx1=limθ→π−∫Dθx1=0\\int\_\{D\_\{\\pi\}\}x\_\{1\}=\\lim\_\{\\theta\{\\rightarrow\}\\pi\-\}\\int\_\{D\_\{\\theta\}\}x\_\{1\}=0\.

For \(2\) \(resp\. \(3\)\) see also Lemma[3\.5](https://arxiv.org/html/2609.30661#S3.Thmlemma5)\(resp\. Lemma[3\.6](https://arxiv.org/html/2609.30661#S3.Thmlemma6)\)\.

###### Proof\.

\(1\)∫Dx1=∫γ¯∞\(∫x~2∞e−x222​d​x2\)​x1​e−x122​d​x1\\int\_\{D\}x\_\{1\}=\\int\_\{\\overline\{\\gamma\}\}^\{\\infty\}\\left\(\\int\_\{\\widetilde\{x\}\_\{2\}\}^\{\\infty\}e^\{\-\\frac\{x\_\{2\}^\{2\}\}\{2\}\}dx\_\{2\}\\right\)x\_\{1\}e^\{\-\\frac\{x\_\{1\}^\{2\}\}\{2\}\}dx\_\{1\}\. By \([A\.63](https://arxiv.org/html/2609.30661#A1.E63)\),

∫x~2∞e−x222​d​x2\\displaystyle\\int\_\{\\widetilde\{x\}\_\{2\}\}^\{\\infty\}e^\{\-\\frac\{x\_\{2\}^\{2\}\}\{2\}\}dx\_\{2\}=\\displaystyle=π2​\[1−erf​\(x~22\)\]\\displaystyle\\sqrt\{\\frac\{\\pi\}\{2\}\}\\big\[1\-\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\widetilde\{x\}\_\{2\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\big\]=\\displaystyle=π2​\[1−erf​\(−x1​cot⁡\(θ\)\+β¯​csc⁡\(θ\)2\)\]\\displaystyle\\sqrt\{\\frac\{\\pi\}\{2\}\}\\left\[1\-\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-x\_\{1\}\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]It remains to compute

π2​∫γ¯∞x1​\[1−erf​\(−x1​cot⁡\(θ\)\+β¯​csc⁡\(θ\)2\)\]​e−x122​d​x1\\displaystyle\\sqrt\{\\frac\{\\pi\}\{2\}\}\\int^\{\\infty\}\_\{\\overline\{\\gamma\}\}x\_\{1\}\\left\[1\-\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-x\_\{1\}\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]e^\{\-\\frac\{x\_\{1\}^\{2\}\}\{2\}\}dx\_\{1\}=\\displaystyle=π2​e−γ¯22−\\displaystyle\\sqrt\{\\frac\{\\pi\}\{2\}\}e^\{\-\\frac\{\\overline\{\\gamma\}^\{2\}\}\{2\}\}\-π2​∫γ¯∞x1​e−x122​erf​\(−x1​cot⁡\(θ\)\+β¯​csc⁡\(θ\)2\)​d​x1\\displaystyle\\hskip 21\.68121pt\\sqrt\{\\frac\{\\pi\}\{2\}\}\\int\_\{\\overline\{\\gamma\}\}^\{\\infty\}x\_\{1\}e^\{\-\\frac\{x\_\{1\}^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-x\_\{1\}\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}dx\_\{1\}=def\\displaystyle\\stackrel\{\{\\scriptstyle\\mathrm\{def\}\}\}\{\{=\}\}I1−I2,\\displaystyle I\_\{1\}\-I\_\{2\},whereI1=π2​e−γ¯22I\_\{1\}=\\sqrt\{\\frac\{\\pi\}\{2\}\}e^\{\-\\frac\{\\overline\{\\gamma\}^\{2\}\}\{2\}\}\. Integrating by parts and using \([A\.60](https://arxiv.org/html/2609.30661#A1.E60)\)\.

I2\\displaystyle I\_\{2\}=\\displaystyle=π2​e−γ¯22​erf​\(A​γ¯\+B​β¯2\)−cot⁡\(θ\)​∫γ¯∞e−x122​e−\(A​x1\+B​β¯\)22​d​x1\\displaystyle\\sqrt\{\\frac\{\\pi\}\{2\}\}e^\{\-\\frac\{\\overline\{\\gamma\}^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{A\\overline\{\\gamma\}\+B\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\\cot\(\\theta\)\\int\_\{\\overline\{\\gamma\}\}^\{\\infty\}e^\{\-\\frac\{x\_\{1\}^\{2\}\}\{2\}\}e^\{\-\\frac\{\(Ax\_\{1\}\+B\\overline\{\\beta\}\)^\{2\}\}\{2\}\}dx\_\{1\}Nowe−x122e−\(A​x1\+B​β¯\)22=e−\(Bx1\+Aβ¯\)2/2e−β¯22e^\{\-\\frac\{x\_\{1\}^\{2\}\}\{2\}\}e^\{\-\\frac\{\(Ax\_\{1\}\+B\\overline\{\\beta\}\)^\{2\}\}\{2\}\}=e^\{\-\(Bx\_\{1\}\+A\\overline\{\\beta\}\)^\{2\}/2\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}and so, by \([A\.63](https://arxiv.org/html/2609.30661#A1.E63)\), witha=csc⁡\(θ\)/2\>0a=\\csc\(\\theta\)/\\sqrt\{2\}\>0on\(0,π\)\(0,\\pi\)andb=−cot\(θ\)/2b=\-\\cot\(\\theta\)/\\sqrt\{2\},

I2=π2​\[e−γ¯22​erf​\(A​γ¯\+B​β¯2\)−e−β¯22​cos⁡\(θ\)​erfc​\(B​γ¯\+A​β¯2\)\]\\hskip\-21\.68121ptI\_\{2\}=\\sqrt\{\\frac\{\\pi\}\{2\}\}\\left\[e^\{\\frac\{\-\\overline\{\\gamma\}^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{A\\overline\{\\gamma\}\+B\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\\cos\(\\theta\)\{\\text\{\\rm erfc\}\\left\(\{\\frac\{B\\overline\{\\gamma\}\+A\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]and the result follows\. \(2\) For the evaluation oflimθ→0\+∫Dθx1\\lim\_\{\\theta\{\\rightarrow\}0\+\}\\int\_\{D\_\{\\theta\}\}x\_\{1\}, the main step is to show that ifγ¯<β¯\\overline\{\\gamma\}<\\overline\{\\beta\}, thenlimθ→0\+∫Dθ∩\(\[γ¯,β¯\]×ℝ\)x1=0\\lim\_\{\\theta\{\\rightarrow\}0\+\}\\int\_\{D\_\{\\theta\}\\cap\(\[\\overline\{\\gamma\},\\overline\{\\beta\}\]\\times\{\\mathbb\{R\}\}\)\}x\_\{1\}=0\. Referring to Figure[2](https://arxiv.org/html/2609.30661#S3.F2),LθL\_\{\\theta\}is the line defining the lower boundary ofDθD\_\{\\theta\}and is given by the equationx1​cos⁡\(θ\)\+x2​sin⁡\(θ\)=β¯x\_\{1\}\\cos\(\\theta\)\+x\_\{2\}\\sin\(\\theta\)=\\overline\{\\beta\}\. The lineLθL\_\{\\theta\}intersectsx1=β¯x\_\{1\}=\\overline\{\\beta\}atX2​\(θ\)=X2=β¯​\(cot⁡\(θ\)−csc⁡\(θ\)\)X\_\{2\}\(\\theta\)=X\_\{2\}=\\overline\{\\beta\}\(\\cot\(\\theta\)\-\\csc\(\\theta\)\)\. HenceX2​\(θ\)X\_\{2\}\(\\theta\)is monotone function ofθ\\thetaandlimθ→0\+X2​\(θ\)=0\\lim\_\{\\theta\{\\rightarrow\}0\+\}X\_\{2\}\(\\theta\)=0\. In the figure,β¯\>0\\overline\{\\beta\}\>0and soX2​\(θ\)<0X\_\{2\}\(\\theta\)<0andX2​\(θ\)X\_\{2\}\(\\theta\)is strictly monotone increasing\. On the other hand,LθL\_\{\\theta\}meetsx1=γ¯x\_\{1\}=\\overline\{\\gamma\}at the pointY2​\(θ\)=β¯​csc⁡\(θ\)−γ¯​cot⁡\(θ\)Y\_\{2\}\(\\theta\)=\\overline\{\\beta\}\\csc\(\\theta\)\-\\overline\{\\gamma\}\\cot\(\\theta\)and sinceβ¯​csc⁡\(θ\)\>γ¯​cot⁡\(θ\)\\overline\{\\beta\}\\csc\(\\theta\)\>\\overline\{\\gamma\}\\cot\(\\theta\)on\(0,π/2\)\(0,\\pi/2\),limθ→0\+Y2​\(θ\)=\+∞\\lim\_\{\\theta\{\\rightarrow\}0\+\}Y\_\{2\}\(\\theta\)=\+\\infty\.

There are two approaches\. \(1\) A direct proof that ifβ¯≥γ¯\\overline\{\\beta\}\\geq\\overline\{\\gamma\}then

limθ→0\+∫γ¯β¯\(∫x~2∞e−x222​d​x2\)​x1​e−x122​d​x1=0\\lim\_\{\\theta\{\\rightarrow\}0\+\}\\int\_\{\\overline\{\\gamma\}\}^\{\\overline\{\\beta\}\}\\left\(\\int\_\{\\widetilde\{x\}\_\{2\}\}^\{\\infty\}e^\{\-\\frac\{x\_\{2\}^\{2\}\}\{2\}\}dx\_\{2\}\\right\)x\_\{1\}e^\{\-\\frac\{x\_\{1\}^\{2\}\}\{2\}\}dx\_\{1\}=0Alternatively, \(2\) A Proof thatlimθ→0\+∫γ¯β¯erfc​\(−x1​cot⁡\(θ\)\+β¯​csc⁡\(θ\)2\)=0\\lim\_\{\\theta\{\\rightarrow\}0\+\}\\int\_\{\\overline\{\\gamma\}\}^\{\\overline\{\\beta\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\-x\_\{1\}\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}=0ifβ¯\>γ¯\\overline\{\\beta\}\>\\overline\{\\gamma\}\. We use the first method here; later we use the second method\.

![Refer to caption](https://arxiv.org/html/2609.30661v1/int_region.png)Figure 2\.The region of integration on the strip\[γ¯,β¯\]×ℝ\[\\overline\{\\gamma\},\\overline\{\\beta\}\]\\times\{\\mathbb\{R\}\},θ\\thetanear zero andβ¯<γ¯\\overline\{\\beta\}<\\overline\{\\gamma\}\.For the first approach, it is easy to see that the integral over the regionAAin Figure[2](https://arxiv.org/html/2609.30661#S3.F2)is bounded byke−C2/2ke^\{\-C^\{2\}/2\}, wherekkis a constant independent ofCC, and the integral over the triangular regionBBconverges to zero asθ→0\+\\theta\{\\rightarrow\}0\+\. Formally, givenε\>0\\varepsilon\>0, chooseCCso that the integral overAAis bounded byε/2\\varepsilon/2, then chooseκ⁡\(ε\)\>0\\kappa\(\\varepsilon\)\>0, so that the integral over the triangular regionBBis bounded byε/2\\varepsilon/2forθ∈\(0,κ⁡\(ε\)\)\\theta\\in\(0,\\kappa\(\\varepsilon\)\)\.

Alternatively,limθ→0\+erfc​\(−x1​cot⁡\(θ\)\+β¯​csc⁡\(θ\)2\)=0\\lim\_\{\\theta\{\\rightarrow\}0\+\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\-x\_\{1\}\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}=0uniformly forx1∈\[γ¯,β¯−ε\]x\_\{1\}\\in\[\\overline\{\\gamma\},\\overline\{\\beta\}\-\\varepsilon\],ε\>0\\varepsilon\>0, and the error arising from the integral over\[β¯−ε,β¯\]\[\\overline\{\\beta\}\-\\varepsilon,\\overline\{\\beta\}\]is bounded by a constant multiple ofε\\varepsilon\.

Either approach gives the explicit formula forD0D\_\{0\}\. \(3\) The proof is similar to that of \(2\) and is omitted\. ∎

Neither of the termsx1x\_\{1\}orx2x\_\{2\}occurs in the bias free case\. However,x1​x2x\_\{1\}x\_\{2\}does occur if there is zero bias\.

###### Lemma 3\.12\.

\(3\.10\)∫Dx1​x2=e−β¯22​sin⁡\(θ\)​π2​β¯​cos⁡\(θ\)​erfc​\(γ¯​csc⁡\(θ\)−β¯​cot⁡\(θ\)2\)\+e−β¯22​sin2⁡\(θ\)​e−\(γ¯​csc⁡\(θ\)−β¯​cot⁡\(θ\)\)22\\begin\{aligned\} \\int\_\{D\}x\_\{1\}x\_\{2\}=&e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\\sin\(\\theta\)\\sqrt\{\\frac\{\\pi\}\{2\}\}\\overline\{\\beta\}\\cos\(\\theta\)\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\\csc\(\\theta\)\-\\overline\{\\beta\}\\cot\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}\\\\ &\+e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\\sin^\{2\}\(\\theta\)e^\{\-\\frac\{\(\\overline\{\\gamma\}\\csc\(\\theta\)\-\\overline\{\\beta\}\\cot\(\\theta\)\)^\{2\}\}\{2\}\}\\end\{aligned\}

###### Proof\.

We have

∫Dx1​x2=∫γ¯∞\(∫x~2∞x2​e−x222​d​x2\)​x1​e−x122​d​x1\\int\_\{D\}x\_\{1\}x\_\{2\}=\\int\_\{\\overline\{\\gamma\}\}^\{\\infty\}\\left\(\\int\_\{\\widetilde\{x\}\_\{2\}\}^\{\\infty\}x\_\{2\}e^\{\-\\frac\{x\_\{2\}^\{2\}\}\{2\}\}dx\_\{2\}\\right\)x\_\{1\}e^\{\-\\frac\{x\_\{1\}^\{2\}\}\{2\}\}dx\_\{1\}As in Lemma[3\.10](https://arxiv.org/html/2609.30661#S3.Thmlemma10),

∫x~2∞x2​e−x222​d​x2=e−\(−x1​cot⁡\(θ\)\+β¯​csc⁡\(θ\)\)22\\int\_\{\\widetilde\{x\}\_\{2\}\}^\{\\infty\}x\_\{2\}e^\{\-\\frac\{x\_\{2\}^\{2\}\}\{2\}\}dx\_\{2\}=e^\{\-\\frac\{\(\-x\_\{1\}\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)\)^\{2\}\}\{2\}\}Hence

∫Dx1​x2=∫γ¯∞x1​e−x122​e−\(−x1​cot⁡\(θ\)\+β¯​csc⁡\(θ\)\)22​d​x1\\int\_\{D\}x\_\{1\}x\_\{2\}=\\int\_\{\\overline\{\\gamma\}\}^\{\\infty\}x\_\{1\}e^\{\-\\frac\{x\_\{1\}^\{2\}\}\{2\}\}e^\{\-\\frac\{\(\-x\_\{1\}\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)\)^\{2\}\}\{2\}\}dx\_\{1\}Now

−x122−\(−x1​cot⁡\(θ\)\+β¯​csc⁡\(θ\)\)22=−β22−\(x1​csc⁡\(θ\)−β¯​cot⁡\(θ\)\)22\-\\frac\{x\_\{1\}^\{2\}\}\{2\}\-\\frac\{\(\-x\_\{1\}\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)\)^\{2\}\}\{2\}=\-\\frac\{\\beta^\{2\}\}\{2\}\-\\frac\{\(x\_\{1\}\\csc\(\\theta\)\-\\overline\{\\beta\}\\cot\(\\theta\)\)^\{2\}\}\{2\}and so

∫Dx1​x2=e−β¯22​∫γ¯∞x1​e−\(x1​csc⁡\(θ\)−β¯​cot⁡\(θ\)\)22​d​x1\\int\_\{D\}x\_\{1\}x\_\{2\}=e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\\int\_\{\\overline\{\\gamma\}\}^\{\\infty\}x\_\{1\}e^\{\-\\frac\{\(x\_\{1\}\\csc\(\\theta\)\-\\overline\{\\beta\}\\cot\(\\theta\)\)^\{2\}\}\{2\}\}dx\_\{1\}Apply Example[A\.3](https://arxiv.org/html/2609.30661#A1.Thmlemma3)witha=csc⁡\(θ\)2a=\\frac\{\\csc\(\\theta\)\}\{\\sqrt\{2\}\}andb=−β¯​cot⁡\(θ\)2b=\-\\frac\{\\overline\{\\beta\}\\cot\(\\theta\)\}\{\\sqrt\{2\}\}to deduce that

∫Dx1​x2=e−β¯22​sin⁡\(θ\)​\[π2​β¯​cos⁡\(θ\)​erfc​\(B​γ¯\+A​β¯2\)\+sin⁡\(θ\)​e−\(B​γ¯\+A​β¯\)22\]\\int\_\{D\}x\_\{1\}x\_\{2\}=e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\\sin\(\\theta\)\\left\[\\sqrt\{\\frac\{\\pi\}\{2\}\}\\overline\{\\beta\}\\cos\(\\theta\)\{\\text\{\\rm erfc\}\\left\(\{\\frac\{B\\overline\{\\gamma\}\+A\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\sin\(\\theta\)e^\{\-\\frac\{\(B\\overline\{\\gamma\}\+A\\overline\{\\beta\}\)^\{2\}\}\{2\}\}\\right\]sincecsc⁡\(θ\)\>0\\csc\(\\theta\)\>0on\(0,π\)\(0,\\pi\)\.

On account of the factorsin⁡\(θ\)\\sin\(\\theta\), there is no need to analyze the limits asθ→0\+,π−\\theta\{\\rightarrow\}0\+,\\pi\-\. ∎

### 3\.3\.Evaluation of the defining integrals forf⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\): II

In this section, we start the analysis of∫D1\\int\_\{D\}1and∫Dx12\\int\_\{D\}x\_\{1\}^\{2\}forθ∈\(0,π\)\\theta\\in\(0,\\pi\)and find the limits of these integrals asθ→0\+,π−\\theta\{\\rightarrow\}0\+,\\pi\-\.

The methods of the previous section show that forθ∈\(0,π\)\\theta\\in\(0,\\pi\)

∫D1\\displaystyle\\int\_\{D\}1=\\displaystyle=∫γ¯∞\(∫x~2∞1​e−x222​d​x2\)​e−x122​d​x1\\displaystyle\\int\_\{\\overline\{\\gamma\}\}^\{\\infty\}\\left\(\\int^\{\\infty\}\_\{\\widetilde\{x\}\_\{2\}\}1e^\{\-\\frac\{x\_\{2\}^\{2\}\}\{2\}\}dx\_\{2\}\\right\)e^\{\-\\frac\{x\_\{1\}^\{2\}\}\{2\}\}dx\_\{1\}=\\displaystyle=π2​∫γ¯∞\(1−erf​\(−x1​cot⁡\(θ\)\+β¯​csc⁡\(θ\)2\)\)​e−x122​d​x1\\displaystyle\\sqrt\{\\frac\{\\pi\}\{2\}\}\\int\_\{\\overline\{\\gamma\}\}^\{\\infty\}\\left\(1\-\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-x\_\{1\}\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\)e^\{\-\\frac\{x\_\{1\}^\{2\}\}\{2\}\}dx\_\{1\}=\\displaystyle=π2​erfc​\(γ¯2\)−π2​∫γ¯∞e−x122​erf​\(−x1​cot⁡\(θ\)\+β¯​csc⁡\(θ\)2\)​d​x1\\displaystyle\\frac\{\\pi\}\{2\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\\sqrt\{\\frac\{\\pi\}\{2\}\}\\int\_\{\\overline\{\\gamma\}\}^\{\\infty\}e^\{\-\\frac\{x\_\{1\}^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-x\_\{1\}\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}dx\_\{1\}
For the integral ofx12x\_\{1\}^\{2\}overDDwe have

∫Dx12\\displaystyle\\int\_\{D\}x\_\{1\}^\{2\}=\\displaystyle=∫γ¯∞\(∫x~2∞1​e−x222​d​x2\)​x12​e−x122​d​x1\\displaystyle\\int\_\{\\overline\{\\gamma\}\}^\{\\infty\}\\left\(\\int^\{\\infty\}\_\{\\widetilde\{x\}\_\{2\}\}1e^\{\-\\frac\{x\_\{2\}^\{2\}\}\{2\}\}dx\_\{2\}\\right\)x\_\{1\}^\{2\}e^\{\-\\frac\{x\_\{1\}^\{2\}\}\{2\}\}dx\_\{1\}=\\displaystyle=π2​∫γ¯∞\(1−erf​\(−x1​cot⁡\(θ\)\+β¯​csc⁡\(θ\)2\)\)​x12​e−x122​d​x1\\displaystyle\\sqrt\{\\frac\{\\pi\}\{2\}\}\\int\_\{\\overline\{\\gamma\}\}^\{\\infty\}\\left\(1\-\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-x\_\{1\}\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\)x\_\{1\}^\{2\}e^\{\-\\frac\{x\_\{1\}^\{2\}\}\{2\}\}dx\_\{1\}=\\displaystyle=π2​γ¯​e−γ¯22\+π2​erfc​\(γ¯2\)−\\displaystyle\\sqrt\{\\frac\{\\pi\}\{2\}\}\\overline\{\\gamma\}e^\{\-\\frac\{\\overline\{\\gamma\}^\{2\}\}\{2\}\}\+\\frac\{\\pi\}\{2\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-π2​∫γ¯∞x12​e−x122​erf​\(−x1​cot⁡\(θ\)\+β¯​csc⁡\(θ\)2\)​d​x1\\displaystyle\\sqrt\{\\frac\{\\pi\}\{2\}\}\\int\_\{\\overline\{\\gamma\}\}^\{\\infty\}x\_\{1\}^\{2\}e^\{\-\\frac\{x\_\{1\}^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-x\_\{1\}\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}dx\_\{1\}
Using \([A\.68](https://arxiv.org/html/2609.30661#A1.E68)\), we have

π2​∫γ¯∞x12​e−x122​erf​\(−x1​cot⁡\(θ\)\+β¯​csc⁡\(θ\)2\)​d​x1=π2​γ¯​e−γ¯22​erf​\(−γ¯​cot⁡\(θ\)\+β¯​csc⁡\(θ\)2\)−π2​β¯​cos2⁡\(θ\)​e−β¯22​\[1−erf​\(γ¯​csc⁡\(θ\)−β¯​cot⁡\(θ\)2\)\]−sin⁡\(θ\)​cos⁡\(θ\)​e−β¯22​e−\(γ¯​csc⁡\(θ\)−β¯​cot⁡\(θ\)\)22\+π2∫γ¯∞e−x122erf\(−x1​cot⁡\(θ\)\+β¯​csc⁡\(θ\)2\)dx1\\sqrt\{\\frac\{\\pi\}\{2\}\}\\int\_\{\\overline\{\\gamma\}\}^\{\\infty\}x\_\{1\}^\{2\}e^\{\-\\frac\{x\_\{1\}^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-x\_\{1\}\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}dx\_\{1\}=\\\\ \\begin\{aligned\} &\\sqrt\{\\frac\{\\pi\}\{2\}\}\\overline\{\\gamma\}e^\{\-\\frac\{\\overline\{\\gamma\}^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-\\overline\{\\gamma\}\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}\\\\ &\-\\sqrt\{\\frac\{\\pi\}\{2\}\}\\overline\{\\beta\}\\cos^\{2\}\(\\theta\)e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\\left\[1\-\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\gamma\}\\csc\(\\theta\)\-\\overline\{\\beta\}\\cot\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ &\-\\sin\(\\theta\)\\cos\(\\theta\)e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}e^\{\-\\frac\{\(\\overline\{\\gamma\}\\csc\(\\theta\)\-\\overline\{\\beta\}\\cot\(\\theta\)\)^\{2\}\}\{2\}\}\\\\ &\+\\sqrt\{\\frac\{\\pi\}\{2\}\}\\int\_\{\\overline\{\\gamma\}\}^\{\\infty\}e^\{\-\\frac\{x\_\{1\}^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-x\_\{1\}\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}dx\_\{1\}\\end\{aligned\}Hence

∫Dx12\\displaystyle\\int\_\{D\}x\_\{1\}^\{2\}=\\displaystyle=π2​γ¯​e−γ¯22​erfc​\(−γ¯​cot⁡\(θ\)\+β¯​csc⁡\(θ\)2\)\+π2​erfc​\(γ¯2\)\\displaystyle\\sqrt\{\\frac\{\\pi\}\{2\}\}\\overline\{\\gamma\}e^\{\-\\frac\{\\overline\{\\gamma\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\-\\overline\{\\gamma\}\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\frac\{\\pi\}\{2\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+π2​β¯​cos2⁡\(θ\)​e−β¯22​erfc​\(γ¯​csc⁡\(θ\)−β¯​cot⁡\(θ\)2\)\\displaystyle\+\\sqrt\{\\frac\{\\pi\}\{2\}\}\\overline\{\\beta\}\\cos^\{2\}\(\\theta\)e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\\csc\(\\theta\)\-\\overline\{\\beta\}\\cot\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}\+sin⁡\(θ\)​cos⁡\(θ\)​e−β¯22​e−\(γ¯​csc⁡\(θ\)−β¯​cot⁡\(θ\)\)22\\displaystyle\+\\sin\(\\theta\)\\cos\(\\theta\)e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}e^\{\-\\frac\{\(\\overline\{\\gamma\}\\csc\(\\theta\)\-\\overline\{\\beta\}\\cot\(\\theta\)\)^\{2\}\}\{2\}\}−π2∫γ¯∞e−x122erf\(−x1​cot⁡\(θ\)\+β¯​csc⁡\(θ\)2\)dx1\\displaystyle\-\\sqrt\{\\frac\{\\pi\}\{2\}\}\\int\_\{\\overline\{\\gamma\}\}^\{\\infty\}e^\{\-\\frac\{x\_\{1\}^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-x\_\{1\}\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}dx\_\{1\}
The integral

\(3\.11\)π2​∫γ¯∞e−t22​erf​\(−t​cot⁡\(θ\)\+β¯​csc⁡\(θ\)2\)​𝑑t\\sqrt\{\\frac\{\\pi\}\{2\}\}\\int\_\{\\overline\{\\gamma\}\}^\{\\infty\}e^\{\-\\frac\{t^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-t\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}dtoccurs in both formulas obtained above for∫D1\\int\_\{D\}1and∫Dx12\\int\_\{D\}x\_\{1\}^\{2\}\. In general,∫b∞e−t22​erf​\(a​t\+b2\)​𝑑t\\int\_\{b\}^\{\\infty\}e^\{\-\\frac\{t^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{at\+b\}\{\\sqrt\{2\}\}\}\\right\)\}dtcannot be expressed in terms of standard elementary functions\. However, there are some exceptions:

\(3\.12\)π2​∫0∞e−t22​erf​\(a​t2\)​𝑑t\\displaystyle\\sqrt\{\\frac\{\\pi\}\{2\}\}\\int\_\{0\}^\{\\infty\}e^\{\-\\frac\{t^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{at\}\{\\sqrt\{2\}\}\}\\right\)\}dt=\\displaystyle=tan−1⁡\(a\)\\displaystyle\\tan^\{\-1\}\(a\)\(3\.13\)π2​∫−∞∞e−t22​erf​\(a​t\+b2\)​𝑑t\\displaystyle\\sqrt\{\\frac\{\\pi\}\{2\}\}\\int\_\{\-\\infty\}^\{\\infty\}e^\{\-\\frac\{t^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{at\+b\}\{\\sqrt\{2\}\}\}\\right\)\}dt=\\displaystyle=π​erf​\(b2​\(a2\+1\)\)\\displaystyle\\pi\\,\{\\text\{\\rm erf\}\\left\(\{\\frac\{b\}\{\\sqrt\{2\(a^\{2\}\+1\)\}\}\}\\right\)\}\(3\.14\)limθ→0\+π2​∫β¯∞e−t22​erf​\(−t​cot⁡\(θ\)\+β¯​csc⁡\(θ\)2\)​𝑑t=−π2​erfc​\(β¯2\)\\lim\_\{\\theta\{\\rightarrow\}0\+\}\\sqrt\{\\frac\{\\pi\}\{2\}\}\\int\_\{\\overline\{\\beta\}\}^\{\\infty\}e^\{\-\\frac\{t^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-t\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}dt=\-\\frac\{\\pi\}\{2\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}Ifa=−cot⁡\(θ\)a=\-\\cot\(\\theta\),b=β¯​csc⁡\(θ\)b=\\overline\{\\beta\}\\csc\(\\theta\), then \([3\.12](https://arxiv.org/html/2609.30661#S3.E12)\) equalsθ−π2\\theta\-\\frac\{\\pi\}\{2\}and \([3\.13](https://arxiv.org/html/2609.30661#S3.E13)\) is equal toπ​erf​\(β¯2\)\\pi\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\.

Notwithstanding the lack of a simple representation of \([3\.11](https://arxiv.org/html/2609.30661#S3.E11)\), we can evaluate the limit of the integral asθ→0\+,π−\\theta\{\\rightarrow\}0\+,\\pi\-\.

###### Lemma 3\.13\.

SetIθ=π2​∫γ¯∞e−t22​erf​\(−t​cot⁡\(θ\)\+β¯​csc⁡\(θ\)2\)​𝑑tI\_\{\\theta\}=\\sqrt\{\\frac\{\\pi\}\{2\}\}\\int\_\{\\overline\{\\gamma\}\}^\{\\infty\}e^\{\-\\frac\{t^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-t\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}dt\.

1. \(1\)The limit asθ→0\+\\theta\{\\rightarrow\}0\+ limθ→0\+Iθ=\{=π2\[2erf\(β¯2\)−erf\(γ¯2\)−1\],ifβ¯≥γ¯\.=−π2erfc\(γ¯2\),ifβ¯≤γ¯\.\\lim\_\{\\theta\{\\rightarrow\}0\+\}I\_\{\\theta\}=\\begin\{cases\}&=\\frac\{\\pi\}\{2\}\\left\[2\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-1\\right\],\\;\\text\{if \}\\overline\{\\beta\}\\geq\\overline\{\\gamma\}\.\\\\ &=\-\\frac\{\\pi\}\{2\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\},\\;\\text\{if \}\\overline\{\\beta\}\\leq\\overline\{\\gamma\}\.\\end\{cases\}
2. \(2\)The limit asθ→π−\\theta\{\\rightarrow\}\\pi\- limθ→π−Iθ=\{=π2\[2erf\(β¯2\)\+erf\(γ¯2\)\+1\],ifγ¯\+β¯≤0\.=π2erfc\(γ¯2\),ifγ¯\+β¯≥0\.\\lim\_\{\\theta\{\\rightarrow\}\\pi\-\}I\_\{\\theta\}=\\begin\{cases\}&=\\frac\{\\pi\}\{2\}\\left\[2\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+1\\right\],\\;\\text\{if \}\\overline\{\\gamma\}\+\\overline\{\\beta\}\\leq 0\.\\\\ &=\\frac\{\\pi\}\{2\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\},\\;\\text\{if \}\\overline\{\\gamma\}\+\\overline\{\\beta\}\\geq 0\.\\end\{cases\}

###### Proof\.

We prove \(1\); details for \(2\) are similar\. Forθ∈\(0,π/4\]\\theta\\in\(0,\\pi/4\], define

fθ​\(t\)\\displaystyle f\_\{\\theta\}\(t\)=\\displaystyle=−t​cot⁡\(θ\)\+β¯​csc⁡\(θ\)2\\displaystyle\\frac\{\-t\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\sqrt\{2\}\}=\\displaystyle=\(β¯−t\)​csc⁡\(θ\)2\+t⁡\(csc⁡\(θ\)−cot⁡\(θ\)\)2,t∈ℝ\.\\displaystyle\\frac\{\(\\overline\{\\beta\}\-t\)\\csc\(\\theta\)\}\{\\sqrt\{2\}\}\+\\frac\{t\(\\csc\(\\theta\)\-\\cot\(\\theta\)\)\}\{\\sqrt\{2\}\},\\;t\\in\{\\mathbb\{R\}\}\.Observe that

0<t​\(csc⁡\(θ\)−cot⁡\(θ\)\)2=t​tan⁡\(θ/2\)2≤C​t​θ,θ∈\(0,π/4\],0<t\\frac\{\(\\csc\(\\theta\)\-\\cot\(\\theta\)\)\}\{\\sqrt\{2\}\}=t\\frac\{\\tan\(\\theta/2\)\}\{\\sqrt\{2\}\}\\leq Ct\\theta,\\theta\\in\(0,\\pi/4\],whereC\>0C\>0is independent ofttandθ∈\(0,π/4\]\\theta\\in\(0,\\pi/4\]\. Consequently, on any closed intervalJJnot containingβ¯\\overline\{\\beta\}, we can always chooseθ0∈\(0,π/4\]\\theta\_\{0\}\\in\(0,\\pi/4\]such thatfθf\_\{\\theta\}is strictly monotone decreasing onJJfor allθ∈\(0,θ0\]\\theta\\in\(0,\\theta\_\{0\}\]\.

Suppose thatβ¯≥γ¯\\overline\{\\beta\}\\geq\\overline\{\\gamma\}\. Givenε\>0\\varepsilon\>0, chooseδε\>0\\delta\_\{\\varepsilon\}\>0andθ0∈\(0,π/4\]\\theta\_\{0\}\\in\(0,\\pi/4\]so that

π2​∫\[β¯−δε,β¯\]e−t22​𝑑t<ε/2\\sqrt\{\\frac\{\\pi\}\{2\}\}\\int\_\{\[\\overline\{\\beta\}\-\\delta\_\{\\varepsilon\},\\overline\{\\beta\}\]\}e^\{\-\\frac\{t^\{2\}\}\{2\}\}dt<\\varepsilon/2andfθ​\(t\)f\_\{\\theta\}\(t\)is strictly monotone decreasing on\[γ¯,β¯−δε\]\[\\overline\{\\gamma\},\\overline\{\\beta\}\-\\delta\_\{\\varepsilon\}\]for allθ∈\(0,θ0\]\\theta\\in\(0,\\theta\_\{0\}\]\. Hence the contribution toIθI\_\{\\theta\}from the integral over\[β¯−δε,β¯\]\[\\overline\{\\beta\}\-\\delta\_\{\\varepsilon\},\\overline\{\\beta\}\]is less thanε/2\\varepsilon/2\. Using the monotonicity offθf\_\{\\theta\}on\[γ¯,β¯−δε\]\[\\overline\{\\gamma\},\\overline\{\\beta\}\-\\delta\_\{\\varepsilon\}\], chooseθε∈\(0,θ0\]\\theta\_\{\\varepsilon\}\\in\(0,\\theta\_\{0\}\]so thatπ2​\|fθ​\(t\)−1\|<ε/2​\|β¯−γ¯\|\\sqrt\{\\frac\{\\pi\}\{2\}\}\|f\_\{\\theta\}\(t\)\-1\|<\\varepsilon/2\|\\overline\{\\beta\}\-\\overline\{\\gamma\}\|forθ∈\(0,θε\]\\theta\\in\(0,\\theta\_\{\\varepsilon\}\]\. Hence for allθ∈\(0,θε\)\\theta\\in\(0,\\theta\_\{\\varepsilon\}\),

π2​\|∫\[γ¯,β¯\]e−t22​erf​\(−t​cot⁡\(θ\)\+β¯​csc⁡\(θ\)2\)​𝑑t−∫\[γ¯,β¯\]e−t22​𝑑t\|<ε,\\sqrt\{\\frac\{\\pi\}\{2\}\}\\left\|\\int\_\{\[\\overline\{\\gamma\},\\overline\{\\beta\}\]\}e^\{\-\\frac\{t^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-t\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}dt\-\\int\_\{\[\\overline\{\\gamma\},\\overline\{\\beta\}\]\}e^\{\-\\frac\{t^\{2\}\}\{2\}\}dt\\right\|<\\varepsilon,proving the first part of \(1\)\. The proof of the second part of \(1\) is similar but simpler\. ∎

###### Remark 3\.14\.

LetI0=I0​\(β¯,γ¯\)I\_\{0\}=I\_\{0\}\(\\overline\{\\beta\},\\overline\{\\gamma\}\)denote the limiting valuelimθ→0\+Iθ\\lim\_\{\\theta\{\\rightarrow\}0\+\}I\_\{\\theta\}and similarly defineIπI\_\{\\pi\}, thenIθI\_\{\\theta\}is a continuous function of\(β¯,γ¯,θ\)∈ℝ2×\[0,π\]\(\\overline\{\\beta\},\\overline\{\\gamma\},\\theta\)\\in\{\\mathbb\{R\}\}^\{2\}\\times\[0,\\pi\]\.✠\\maltese

###### Lemma 3\.15\.

Ifθ∈\{0,π\}\\theta\\in\\\{0,\\pi\\\}, the integrals∫Dθ1,x12\\int\_\{D\_\{\\theta\}\}1,x\_\{1\}^\{2\}can be explicitly computed and

1. \(1\)∫D01\\displaystyle\\int\_\{D\_\{0\}\}1=\\displaystyle=limθ→0\+∫Dθ1=π​erfc​\(M2\)\\displaystyle\\lim\_\{\\theta\{\\rightarrow\}0\+\}\\int\_\{D\_\{\\theta\}\}1=\\pi\\,\{\\text\{\\rm erfc\}\\left\(\{\\frac\{M\}\{\\sqrt\{2\}\}\}\\right\)\}∫Dπ1\\displaystyle\\int\_\{D\_\{\\pi\}\}1=\\displaystyle=limθ→π−∫Dθ1=\{−π⁡\[erf​\(β¯2\)\+erf​\(γ¯2\)\],if​γ¯\+β¯≤00,if​γ¯\+β¯≥0\\displaystyle\\lim\_\{\\theta\{\\rightarrow\}\\pi\-\}\\int\_\{D\_\{\\theta\}\}1=\\begin\{cases\}&\-\\pi\\left\[\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\],\\;\\text\{if \}\\overline\{\\gamma\}\+\\overline\{\\beta\}\\leq 0\\\\ &0,\\;\\text\{if \}\\overline\{\\gamma\}\+\\overline\{\\beta\}\\geq 0\\end\{cases\}
2. \(2\)∫D0x12\\displaystyle\\int\_\{D\_\{0\}\}x\_\{1\}^\{2\}=\\displaystyle=limθ→0\+∫Dθx12=2​π​M​e−M22\+π​erfc​\(M2\)\\displaystyle\\lim\_\{\\theta\{\\rightarrow\}0\+\}\\int\_\{D\_\{\\theta\}\}x\_\{1\}^\{2\}=\\sqrt\{2\\pi\}Me^\{\-\\frac\{M^\{2\}\}\{2\}\}\+\\pi\\,\{\\text\{\\rm erfc\}\\left\(\{\\frac\{M\}\{\\sqrt\{2\}\}\}\\right\)\}∫Dπx12\\displaystyle\\int\_\{D\_\{\\pi\}\}x\_\{1\}^\{2\}=\\displaystyle=limθ→π−∫Dθx12=\{2​π​\(γ¯​e−γ¯22\+β¯​e−β¯22\)−π⁡\[erf​\(β¯2\)\+erf​\(γ¯2\)\],if​γ¯\+β¯≤00,if​γ¯\+β¯≥0\\displaystyle\\lim\_\{\\theta\{\\rightarrow\}\\pi\-\}\\int\_\{D\_\{\\theta\}\}x\_\{1\}^\{2\}=\\begin\{cases\}&\\sqrt\{2\\pi\}\\big\(\\overline\{\\gamma\}e^\{\-\\frac\{\\overline\{\\gamma\}^\{2\}\}\{2\}\}\+\\overline\{\\beta\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\\big\)\\\\ &\-\\pi\\left\[\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\],\\;\\text\{if \}\\overline\{\\gamma\}\+\\overline\{\\beta\}\\leq 0\\\\ &0,\\;\\text\{if \}\\overline\{\\gamma\}\+\\overline\{\\beta\}\\geq 0\\end\{cases\}whereM=max⁡\{β¯,γ¯\}M=\\max\\\{\\overline\{\\beta\},\\overline\{\\gamma\}\\\}\.

###### Proof\.

The proof follows along the lines of the proof of Lemma[3\.11](https://arxiv.org/html/2609.30661#S3.Thmlemma11)starting from the integral \([3\.11](https://arxiv.org/html/2609.30661#S3.E11)\)\. ∎

###### Remark 3\.16\.

Using the results of Lemmas[3\.11](https://arxiv.org/html/2609.30661#S3.Thmlemma11)and[3\.15](https://arxiv.org/html/2609.30661#S3.Thmlemma15), we recover the results of Lemmas[3\.5](https://arxiv.org/html/2609.30661#S3.Thmlemma5)and[3\.6](https://arxiv.org/html/2609.30661#S3.Thmlemma6)\.✠\\maltese

## 4\.The OwenTT\-function

The T\-functionT⁡\(h,a\)T\(h,a\)was introduced by Owen in 1956\[[17](https://arxiv.org/html/2609.30661#bib.bib17)\]and is defined by

T⁡\(h,a\)=12​π​∫0ae−h2​\(1\+x2\)21\+x2​𝑑x,h,a∈ℝT\(h,a\)=\\frac\{1\}\{2\\pi\}\\int\_\{0\}^\{a\}\\frac\{e^\{\-\\frac\{h^\{2\}\(1\+x^\{2\}\)\}\{2\}\}\}\{1\+x^\{2\}\}dx,\\;\\;h,a\\in\{\\mathbb\{R\}\}The T\-function gives the probability of the eventX\>0∧0<Y<a​XX\>0\\wedge 0<Y<aXwhereXXandYYare independent standard normal variables\. LetG⁡\(x\)G\(x\)denote the CDF12​π​∫−∞xe−t22​𝑑t\\frac\{1\}\{\\sqrt\{2\\pi\}\}\\int\_\{\-\\infty\}^\{x\}e^\{\-\\frac\{t^\{2\}\}\{2\}\}dtfor the standard normal density\. Some important properties of theTT\-function from Owen\[[18](https://arxiv.org/html/2609.30661#bib.bib18), Table II\]:

T\(h,a\)=T\(−h,a\),T\(h\.−a\)=−T\(h,a\),T\(0,a\)=12​πtan−1\(a\)\\displaystyle T\(h,a\)=T\(\-h,a\),\\;T\(h\.\-a\)=\-T\(h,a\),\\;T\(0,a\)=\\frac\{1\}\{2\\pi\}\\tan^\{\-1\}\(a\)T⁡\(h,a\)\+T⁡\(h​a,a−1\)=\{G⁡\(h\)/2\+G⁡\(a​h\)/2−G⁡\(h\)​G​\(a​h\),if​a≥0G⁡\(h\)/2\+G⁡\(a​h\)/2−G⁡\(h\)​G​\(a​h\)−1/4,if​a<0\\displaystyle T\(h,a\)\+T\(ha,a^\{\-1\}\)=\\begin\{cases\}&G\(h\)/2\+G\(ah\)/2\-G\(h\)G\(ah\),\\;\\text\{if \}a\\geq 0\\\\ &G\(h\)/2\+G\(ah\)/2\-G\(h\)G\(ah\)\-1/4,\\;\\text\{if \}a<0\\end\{cases\}\(4\.15\)T⁡\(h,1\)\\displaystyle T\(h,1\)=\\displaystyle=G⁡\(h\)​\(1−G⁡\(h\)\)/2=18​\(1−erf​\(h2\)2\)\\displaystyle G\(h\)\(1\-G\(h\)\)/2=\\frac\{1\}\{8\}\\left\(1\-\{\\text\{\\rm erf\}\\left\(\{\\frac\{h\}\{\\sqrt\{2\}\}\}\\right\)\}^\{2\}\\right\)\(4\.16\)T⁡\(h,∞\)\\displaystyle T\(h,\\infty\)=\\displaystyle=\(1−G⁡\(\|h\|\)/2=14​erfc​\(\|h\|2\)CLOSE\\displaystyle\(1\-G\(\|h\|\)/2=\\frac\{1\}\{4\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\|h\|\}\{\\sqrt\{2\}\}\}\\right\)\}
In\[[18](https://arxiv.org/html/2609.30661#bib.bib18)\], there are tables of integrals of functions related to the standard normal density, both univariate and multivariate\. See also Appendix A \([A\.57](https://arxiv.org/html/2609.30661#A1.E57)—[A\.58](https://arxiv.org/html/2609.30661#A1.E58)\) for the relation between the cumulative density \(CDF\) and error function\.

The result from\[[18](https://arxiv.org/html/2609.30661#bib.bib18)\]of special interest here is the indefinite integral from Table 1, formula 10,010\.3 \(p\. 403\)\.

\(4\.17\)∫G′​\(x\)​G​\(a​x\+b\)​dx=T⁡\(x,bx​1\+a2\)\+T⁡\(b1\+a2,x​1\+a2b\)−T⁡\(x,a​x\+bx\)−T⁡\(b1\+a2,a​b\+x⁡\(1\+a2\)b\)\+G⁡\(x\)​G​\(b1\+a2\)\\int G^\{\\prime\}\(x\)G\(ax\+b\)dx=T\\left\(x,\\frac\{b\}\{x\\sqrt\{1\+a^\{2\}\}\}\\right\)\\\\ \\begin\{aligned\} &\+T\\left\(\\frac\{b\}\{\\sqrt\{1\+a^\{2\}\}\},\\frac\{x\\sqrt\{1\+a^\{2\}\}\}\{b\}\\right\)\-T\\left\(x,\\frac\{ax\+b\}\{x\}\\right\)\\\\ &\-T\\left\(\\frac\{b\}\{\\sqrt\{1\+a^\{2\}\}\},\\frac\{ab\+x\(1\+a^\{2\}\)\}\{b\}\\right\)\+G\(x\)G\\left\(\\frac\{b\}\{\\sqrt\{1\+a^\{2\}\}\}\\right\)\\end\{aligned\}Note that for consistency with our notation, we have switchedaaandbbin\[[18](https://arxiv.org/html/2609.30661#bib.bib18)\]\. That is, we writea​x\+bax\+brather than thea\+b​xa\+bxused in\[[18](https://arxiv.org/html/2609.30661#bib.bib18)\]\.

In our context, it is more natural to express results in terms of error functions as opposed to the CDFG⁡\(x\)G\(x\)\. Indeed, this emphasis leads to a more attractive and concise formula that better fits our applications\. We refer to Appendix[C](https://arxiv.org/html/2609.30661#A3)for more details on Owen’s formula and how to derive the formula described below*directly*from \([4\.17](https://arxiv.org/html/2609.30661#S4.E17)\)\.

### 4\.1\.The Generalized OwenTT\-function\[[21](https://arxiv.org/html/2609.30661#bib.bib21)\]

The generalizedTT\-function is the analytic functionT^\{\\widehat\{T\}\}onℝ3\{\\mathbb\{R\}\}^\{3\}defined by

\(4\.18\)T^​\(h,a,b\)=12​2​π​∫h∞e−t22​erf​\(a​t\+b2\)​𝑑t\\begin\{aligned\} \{\\widehat\{T\}\}\(h,a,b\)=&\\frac\{1\}\{2\\sqrt\{2\\pi\}\}\\int\_\{h\}^\{\\infty\}e^\{\-\\frac\{t^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{at\+b\}\{\\sqrt\{2\}\}\}\\right\)\}dt\\end\{aligned\}
Forθ∈\(0,π\)\\theta\\in\(0,\\pi\), the integral we need for the evaluation of∫Dθ1\\int\_\{D\_\{\\theta\}\}1and∫Dθx12\\int\_\{D\_\{\\theta\}\}x\_\{1\}^\{2\}has a simple relationship to the generalizedTT\-function:

π2​∫γ¯∞e−t22​erf​\(−t​cot⁡\(θ\)\+β¯​csc⁡\(θ\)2\)​𝑑t=2​π​T^​\(γ¯,−cot⁡\(θ\),β¯​csc⁡\(θ\)\)\\sqrt\{\\frac\{\\pi\}\{2\}\}\\int\_\{\\overline\{\\gamma\}\}^\{\\infty\}e^\{\-\\frac\{t^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-t\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}dt=2\\pi\{\\widehat\{T\}\}\(\\overline\{\\gamma\},\-\\cot\(\\theta\),\\overline\{\\beta\}\\csc\(\\theta\)\)The definition of the generalizedTT\-function appears in\[[21](https://arxiv.org/html/2609.30661#bib.bib21)\]777The symbol used isTTrather thanT^\{\\widehat\{T\}\}butT^\{\\widehat\{T\}\}is preferred here to avoid possible confusion with Owen’sTT\-function\.where it is shown that

\(4\.19\)T^​\(h,a,b\)=12​π​\[tan−1⁡\(a\)−tan−1⁡\(a\+bh\)−tan−1⁡\(a\+h⁡\(1\+a2\)b\)\]\+T⁡\(h,a\+bh\)\+T⁡\(b1\+a2,a\+h⁡\(1\+a2\)b\)\+14​erf​\(b2​\(1\+a2\)\)\\begin\{aligned\} \\hskip\-9\.39545pt\{\\widehat\{T\}\}\(h,a,b\)=&\\frac\{1\}\{2\\pi\}\\left\[\\tan^\{\-1\}\(a\)\-\\tan^\{\-1\}\\left\(a\+\\frac\{b\}\{h\}\\right\)\-\\tan^\{\-1\}\\left\(a\+\\frac\{h\(1\+a^\{2\}\)\}\{b\}\\right\)\\right\]\\\\ &\+T\\left\(h,a\+\\frac\{b\}\{h\}\\right\)\+T\\left\(\\frac\{b\}\{\\sqrt\{1\+a^\{2\}\}\},a\+\\frac\{h\(1\+a^\{2\}\)\}\{b\}\\right\)\\\\ &\+\\frac\{1\}\{4\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{b\}\{\\sqrt\{2\(1\+a^\{2\}\)\}\}\}\\right\)\}\\end\{aligned\}Four of the six functions on the right hand side are not continuous functions on all ofℝ3\{\\mathbb\{R\}\}^\{3\}—the continuous \(analytic\) exceptions beingtan−1⁡\(a\)\\tan^\{\-1\}\(a\)and14​erf​\(b2​\(1\+a2\)\)\\frac\{1\}\{4\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{b\}\{\\sqrt\{2\(1\+a^\{2\}\)\}\}\}\\right\)\}\. Consequently, \([4\.19](https://arxiv.org/html/2609.30661#S4.E19)\) is a decomposition of an analytic function of\(h,a,b\)\(h,a,b\)into a sum of six functions, four of which are not continuous, let alone analytic, onℝ3\{\\mathbb\{R\}\}^\{3\}\.

###### Lemma 4\.1\.

RegardT^​\(h,a,b\)\{\\widehat\{T\}\}\(h,a,b\)as given by \([4\.19](https://arxiv.org/html/2609.30661#S4.E19)\)\.

1. \(1\)limb→0T^​\(h,a,b\)=T⁡\(h,a\)\\lim\_\{b\{\\rightarrow\}0\}\{\\widehat\{T\}\}\(h,a,b\)=T\(h,a\),h≠0h\\neq 0\.
2. \(2\)limh→0T^​\(h,a,b\)=14​erf​\(b2​\(1\+a2\)\)\+T⁡\(b1\+a2,a\)\\lim\_\{h\{\\rightarrow\}0\}\{\\widehat\{T\}\}\(h,a,b\)=\\frac\{1\}\{4\}\{\\text\{\\rm erf\}\(\{\\frac\{b\}\{\\sqrt\{2\(1\+a^\{2\}\)\}\}\}\)\}\+T\\big\(\\frac\{b\}\{\\sqrt\{1\+a^\{2\}\}\},a\\big\),b≠0b\\neq 0\.

In particular,T^​\(0,a,b\)=14​erf​\(b2​\(1\+a2\)\)\+T⁡\(b1\+a2,a\)\{\\widehat\{T\}\}\(0,a,b\)=\\frac\{1\}\{4\}\{\\text\{\\rm erf\}\(\{\\frac\{b\}\{\\sqrt\{2\(1\+a^\{2\}\)\}\}\}\)\}\+T\\big\(\\frac\{b\}\{\\sqrt\{1\+a^\{2\}\}\},a\\big\)andT^​\(h,a,0\)=T​\(h,a\)\{\\widehat\{T\}\}\(h,a,0\)=T\(h,a\), for alla,b,h∈ℝa,b,h\\in\{\\mathbb\{R\}\}\.

###### Proof\.

We prove \(1\), the proof of \(2\) is similar\.

limb→0\+T^​\(h,a,b\)\\displaystyle\\lim\_\{b\{\\rightarrow\}0\_\{\+\}\}\{\\widehat\{T\}\}\(h,a,b\)=\\displaystyle=12​π​\(tan−1⁡\(a\)−tan−1⁡\(a\)−π2​sgn​\(h\)\)\\displaystyle\\frac\{1\}\{2\\pi\}\\left\(\\tan^\{\-1\}\(a\)\-\\tan^\{\-1\}\(a\)\-\\frac\{\\pi\}\{2\}\\mbox\{sgn\}\(h\)\\right\)\+T⁡\(h,a\)\+14​sgn​\(h\)\+0=T⁡\(h,a\)\\displaystyle\+\\;T\(h,a\)\+\\frac\{1\}\{4\}\\mbox\{sgn\}\(h\)\+0=T\(h,a\)The same argument proveslimb→0−T^​\(h,a,b\)=T⁡\(h,a\)\\lim\_\{b\{\\rightarrow\}0\_\{\-\}\}\{\\widehat\{T\}\}\(h,a,b\)=T\(h,a\)and the final statement follows by the continuity ofT^​\(h,a,b\)\{\\widehat\{T\}\}\(h,a,b\), ∎

###### Remarks 4\.2\.

\(1\) The formula forT^​\(h,a,0\)\{\\widehat\{T\}\}\(h,a,0\)\(resp\.T^​\(0,a,b\)\{\\widehat\{T\}\}\(0,a,b\)\) is written in terms of the CDFGGand is in\[[18](https://arxiv.org/html/2609.30661#bib.bib18), 1\.010\.1\]\(resp\.\[[18](https://arxiv.org/html/2609.30661#bib.bib18), 10\.010\.6\]\)\. The argument forlimb→0\+T^​\(h,a,b\)\\lim\_\{b\{\\rightarrow\}0\+\}\{\\widehat\{T\}\}\(h,a,b\)is in\[[21](https://arxiv.org/html/2609.30661#bib.bib21)\]\. The result forT^​\(h,a,0\)\{\\widehat\{T\}\}\(h,a,0\)can be proved directly by differentiating the integral forT^​\(h,a,0\)\{\\widehat\{T\}\}\(h,a,0\)with respect toaa, then integrating with respect tott, and finally with respect toaato recover the formula forT⁡\(h,a\)T\(h,a\)\. \(2\) Using the continuity ofT^​\(h,a,b\)\{\\widehat\{T\}\}\(h,a,b\), it follows easily from Lemma[4\.1](https://arxiv.org/html/2609.30661#S4.Thmlemma1)thatT^​\(0,a,0\)=12​π​tan−1⁡\(a\)\{\\widehat\{T\}\}\(0,a,0\)=\\frac\{1\}\{2\\pi\}\\tan^\{\-1\}\(a\)\(cf\. \([3\.12](https://arxiv.org/html/2609.30661#S3.E12)\)\)\. \(3\) Notwithstanding Lemma[4\.1](https://arxiv.org/html/2609.30661#S4.Thmlemma1), the appearance of discontinuous terms in \([4\.19](https://arxiv.org/html/2609.30661#S4.E19)\) indicates the need for care in numerical simulations involving theTT\-function as well as in analytical studies of bifurcation from zero bias\. See also Remarks[5\.1](https://arxiv.org/html/2609.30661#S5.Thmlemma1)\(3\)\.✠\\maltese

## 5\.Evaluation of the defining integrals forf⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\): III

Forθ∈\(0,π\)\\theta\\in\(0,\\pi\),

\(5\.20\)π2​∫γ¯∞e−t22​erf​\(−t​cot⁡\(θ\)\+β¯​csc⁡\(θ\)2\)​𝑑t=2​π​T^​\(γ¯,−cot⁡\(θ\),β¯​csc⁡\(θ\)\),\\sqrt\{\\frac\{\\pi\}\{2\}\}\\int\_\{\\overline\{\\gamma\}\}^\{\\infty\}e^\{\-\\frac\{t^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-t\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}dt=2\\pi\{\\widehat\{T\}\}\(\\overline\{\\gamma\},\-\\cot\(\\theta\),\\overline\{\\beta\}\\csc\(\\theta\)\),where the integral on the left occurs twice in our computations off⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)\. Simplifying notation, defineH⁡\(β¯,γ¯,θ\)H\(\\overline\{\\beta\},\\overline\{\\gamma\},\\theta\)by

H\(β¯,γ¯,θ\)=2πT^\(γ¯,−cot\(θ\),β¯csc\(θ\)\),\(β¯,γ¯\)∈ℝ2,θ∈\(0\.π\)\.H\(\\overline\{\\beta\},\\overline\{\\gamma\},\\theta\)=2\\pi\{\\widehat\{T\}\}\(\\overline\{\\gamma\},\-\\cot\(\\theta\),\\overline\{\\beta\}\\csc\(\\theta\)\),\\;\(\\overline\{\\beta\},\\overline\{\\gamma\}\)\\in\{\\mathbb\{R\}\}^\{2\},\\;\\theta\\in\(0\.\\pi\)\.
Applying the results from the previous section, we have

\(5\.21\)H\(β¯,OPENγ¯,θ\)=\+tan−1⁡\(−cot⁡\(θ\)\)−\[tan−1⁡\(−cot⁡\(θ\)\+β¯​csc⁡\(θ\)γ¯\)\+tan−1⁡\(−cot⁡\(θ\)\+γ¯​csc⁡\(θ\)β¯\)\]\+2​π​\[T⁡\(γ¯,−cot⁡\(θ\)\+β¯​csc⁡\(θ\)γ¯\)\+T⁡\(β¯,−cot⁡\(θ\)\+γ¯​csc⁡\(θ\)β¯\)\]\+π2​erf​\(β¯2\)\\begin\{aligned\} H\(\\overline\{\\beta\},&\\overline\{\\gamma\},\\theta\)=\\\\ &\+\\tan^\{\-1\}\(\-\\cot\(\\theta\)\)\\\\ &\-\\left\[\\tan^\{\-1\}\\left\(\-\\cot\(\\theta\)\+\\frac\{\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\overline\{\\gamma\}\}\\right\)\+\\tan^\{\-1\}\\left\(\-\\cot\(\\theta\)\+\\frac\{\\overline\{\\gamma\}\\csc\(\\theta\)\}\{\\overline\{\\beta\}\}\\right\)\\right\]\\\\ &\+2\\pi\\left\[T\\left\(\\overline\{\\gamma\},\-\\cot\(\\theta\)\+\\frac\{\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\overline\{\\gamma\}\}\\right\)\+T\\left\(\\overline\{\\beta\},\-\\cot\(\\theta\)\+\\frac\{\\overline\{\\gamma\}\\csc\(\\theta\)\}\{\\overline\{\\beta\}\}\\right\)\\right\]\\\\ &\+\\frac\{\\pi\}\{2\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\end\{aligned\}
###### Remarks 5\.1\.

\(1\) The sum of the first three rows of the expression forHHare unchanged if we permuteγ¯,β¯\\overline\{\\gamma\},\\overline\{\\beta\}\. Although the last row does not have this symmetry, the symmetry will reappear when we evaluate∫D1,∫Dx12\\int\_\{D\}1,\\int\_\{D\}x\_\{1\}^\{2\}: the symmetry is a reflection of the trivial symmetryf⁡\(𝐰,𝐯\)=f⁡\(𝐯,𝐰\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)=f\(\{\{\\mathbf\{v\}\}\},\{\{\\mathbf\{w\}\}\}\)\. A consequence of the symmetry is the identity

\(5\.22\)H⁡\(β¯,γ¯,θ\)\+π2​erf​\(γ¯2\)=H⁡\(γ¯,β¯,θ\)\+π2​erf​\(β¯2\)H\(\\overline\{\\beta\},\\overline\{\\gamma\},\\theta\)\+\\frac\{\\pi\}\{2\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}=H\(\\overline\{\\gamma\},\\overline\{\\beta\},\\theta\)\+\\frac\{\\pi\}\{2\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}See also Remarks[3\.3](https://arxiv.org/html/2609.30661#S3.Thmlemma3)\(3\)\. \(2\) AlthoughHHis a continuous function of\(β¯,γ¯,θ\)\(\\overline\{\\beta\},\\overline\{\\gamma\},\\theta\)\(since this is true for the defining integral \([5\.20](https://arxiv.org/html/2609.30661#S5.E20)\)\), the terms 2—5 in the above decomposition of the generalizedTT\-function in terms of inverse tangents and Owen’sTT\-function may not be defined whenβ¯​γ¯=0\\overline\{\\beta\}\\overline\{\\gamma\}=0\. However, the approach of Lemma[4\.1](https://arxiv.org/html/2609.30661#S4.Thmlemma1)can be used to handle cases where only one ofβ¯,γ¯\\overline\{\\beta\},\\overline\{\\gamma\}is zero\. See also Proposition[5\.3](https://arxiv.org/html/2609.30661#S5.Thmlemma3)below\. \(3\) The functionHHis*not*subanalytic onℝ2×\(0,π\)\{\\mathbb\{R\}\}^\{2\}\\times\(0,\\pi\)even though it is analytic \(that is, the graph is not subanalytic\)\. It is, however, subanalytic onℝ2×\[a,b\]\{\\mathbb\{R\}\}^\{2\}\\times\[a,b\]for any closed interval\[a,b\]⊂\(0,π\)\[a,b\]\\subset\(0,\\pi\)\. Although this appears to be a significant barrier to the extension of the fractional power series results for bias free networks \(\[[2](https://arxiv.org/html/2609.30661#bib.bib2)\]—\[[5](https://arxiv.org/html/2609.30661#bib.bib5)\]\) to networks with bias, this turns out not to be the case\.✠\\maltese

It is sometimes possible to extract useful information about the inverse tangent terms in the above expression forHH\. First, an elementary lemma\.

###### Lemma 5\.2\.

\(Notation of \([5\.21](https://arxiv.org/html/2609.30661#S5.E21)\)\)

1. \(1\)tan−1⁡\(−cot⁡\(θ\)\)=θ−π2,θ∈\[0,π\]\\tan^\{\-1\}\(\-\\cot\(\\theta\)\)=\\theta\-\\frac\{\\pi\}\{2\},\\;\\theta\\in\[0,\\pi\]\.
2. \(2\)Ifθ∈\[0,π\]\\theta\\in\[0,\\pi\], then tan−1⁡\(−cot⁡\(θ\)\+csc⁡\(θ\)\)=θ2,tan−1⁡\(−cot⁡\(θ\)−csc⁡\(θ\)\)=θ−π2\\tan^\{\-1\}\(\-\\cot\(\\theta\)\+\\csc\(\\theta\)\)=\\frac\{\\theta\}\{2\},\\;\\;\\tan^\{\-1\}\(\-\\cot\(\\theta\)\-\\csc\(\\theta\)\)=\\frac\{\\theta\-\\pi\}\{2\}where the left hand side is defined aslimθ→0\+\\lim\_\{\\theta\{\\rightarrow\}0\+\}ifθ=0\\theta=0andlimθ→π−\\lim\_\{\\theta\{\\rightarrow\}\\pi\-\}ifθ=π\\theta=\\pi\.

###### Proof\.

For \(2\) observe that−cot⁡\(θ\)\+csc⁡\(θ\)=tan⁡\(θ/2\)\-\\cot\(\\theta\)\+\\csc\(\\theta\)=\\tan\(\\theta/2\)and−cot⁡\(θ\)−csc⁡\(θ\)=−cot⁡\(θ/2\)=tan⁡\(θ−π2\)\-\\cot\(\\theta\)\-\\csc\(\\theta\)=\-\\cot\(\\theta/2\)=\\tan\(\\frac\{\\theta\-\\pi\}\{2\}\)\. ∎

###### Proposition 5\.3\.

1. \(1\)H⁡\(β¯,0,θ\)=π2​erf​\(β¯2\)\+2​π​T​\(β¯,−cot⁡\(θ\)\)H\(\\overline\{\\beta\},0,\\theta\)=\\frac\{\\pi\}\{2\}\{\\text\{\\rm erf\}\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\)\}\+2\\pi T\(\\overline\{\\beta\},\-\\cot\(\\theta\)\)\.
2. \(2\)H⁡\(0,γ¯,θ\)=2​π​T​\(γ¯,−cot⁡\(θ\)\)H\(0,\\overline\{\\gamma\},\\theta\)=2\\pi T\(\\overline\{\\gamma\},\-\\cot\(\\theta\)\)\.
3. \(3\)H⁡\(β¯,β¯,θ\)=4​π​T​\(β¯,tan⁡\(θ2\)\)−π2​erfc​\(β¯2\)H\(\\overline\{\\beta\},\\overline\{\\beta\},\\theta\)=4\\pi T\(\\overline\{\\beta\},\\tan\(\\frac\{\\theta\}\{2\}\)\)\-\\frac\{\\pi\}\{2\}\{\\text\{\\rm erfc\}\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\)\}and \(5\.23\)∂H∂β¯​\(β¯,β¯,θ\)=π2​e−β¯22​\[2​erfc​\(β¯​\(−cot⁡\(θ\)\+csc⁡\(θ\)\)2\)−1\]\\frac\{\\partial H\}\{\\partial\\overline\{\\beta\}\}\(\\overline\{\\beta\},\\overline\{\\beta\},\\theta\)=\\sqrt\{\\frac\{\\pi\}\{2\}\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\\left\[2\\,\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\\big\(\-\\cot\(\\theta\)\+\\csc\(\\theta\)\\big\)\}\{\\sqrt\{2\}\}\}\\right\)\}\-1\\right\]

###### Proof\.

Statements \(1,2\) are immediate from Lemma[4\.1](https://arxiv.org/html/2609.30661#S4.Thmlemma1)and \(3\) is substitution in \([5\.21](https://arxiv.org/html/2609.30661#S5.E21)\) and Lemma[5\.2](https://arxiv.org/html/2609.30661#S5.Thmlemma2)\. Finally \([5\.23](https://arxiv.org/html/2609.30661#S5.E23)\) follows by noting that

∂H∂β¯=−2β¯e−β¯22∫0tan⁡\(θ/2\)eOPEN−β¯​t\)2/2dt\+π2e−β¯22\\frac\{\\partial H\}\{\\partial\\overline\{\\beta\}\}=\-2\\overline\{\\beta\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\\int\_\{0\}^\{\\tan\(\\theta/2\)\}e^\{\-\\overline\{\\beta\}t\)^\{2\}/2\}dt\+\\sqrt\{\\frac\{\\pi\}\{2\}\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}The integral is equal toβ¯−1​∫0β¯​tan⁡\(θ/2\)e−s22​𝑑s\\overline\{\\beta\}^\{\-1\}\\int\_\{0\}^\{\\overline\{\\beta\}\\tan\(\\theta/2\)\}e^\{\-\\frac\{s^\{2\}\}\{2\}\}dsand the result follows sinceβ¯​tan⁡\(θ/2\)=β¯​\(−cot⁡\(θ\)\+csc⁡\(θ\)\)\\overline\{\\beta\}\\tan\(\\theta/2\)=\\overline\{\\beta\}\\big\(\-\\cot\(\\theta\)\+\\csc\(\\theta\)\\big\)\(proof of Lemma[5\.2](https://arxiv.org/html/2609.30661#S5.Thmlemma2)\(2\)\)\. ∎

###### Lemma 5\.4\(Results forlimθ→0\+,π−\\lim\_\{\\theta\{\\rightarrow\}0\+,\\pi\-\}I\)\.

Letβ¯,γ¯∈ℝ\\overline\{\\beta\},\\overline\{\\gamma\}\\in\{\\mathbb\{R\}\}be non\-zero\.

1. \(1\)Limits asθ→0\+\\theta\{\\rightarrow\}0\+\. limθ→0\+tan−1⁡\(−cot⁡\(θ\)\+β¯γ¯​csc⁡\(θ\)\)=\{π2,if​β¯γ¯\>10,if​β¯=γ¯−π2,if​β¯γ¯<1\.\\lim\_\{\\theta\{\\rightarrow\}0\+\}\\tan^\{\-1\}\\big\(\-\\cot\(\\theta\)\+\\frac\{\\overline\{\\beta\}\}\{\\overline\{\\gamma\}\}\\csc\(\\theta\)\\big\)=\\begin\{cases\}&\\frac\{\\pi\}\{2\},\\;\\text\{if \}\\frac\{\\overline\{\\beta\}\}\{\\overline\{\\gamma\}\}\>1\\\\ &0,\\;\\text\{if \}\\overline\{\\beta\}=\\overline\{\\gamma\}\\\\ &\-\\frac\{\\pi\}\{2\},\\;\\text\{if \}\\frac\{\\overline\{\\beta\}\}\{\\overline\{\\gamma\}\}<1\.\\end\{cases\}\(Ifβ¯\\overline\{\\beta\}andγ¯\\overline\{\\gamma\}are interchanged, cases 1 and 3 are switched\.\)
2. \(2\)Limits asθ→π−\\theta\{\\rightarrow\}\\pi\-\. limθ→π−tan−1⁡\(−cot⁡\(θ\)\+β¯γ¯​csc⁡\(θ\)\)=\{π2,if​β¯γ¯\>0π2,if​β¯γ¯∈\(−1,0\)0,if​β¯=−γ¯−π2,if​β¯γ¯<−1\\lim\_\{\\theta\{\\rightarrow\}\\pi\-\}\\tan^\{\-1\}\(\-\\cot\(\\theta\)\+\\frac\{\\overline\{\\beta\}\}\{\\overline\{\\gamma\}\}\\csc\(\\theta\)\)=\\begin\{cases\}&\\frac\{\\pi\}\{2\},\\;\\text\{if \}\\frac\{\\overline\{\\beta\}\}\{\\overline\{\\gamma\}\}\>0\\\\ &\\frac\{\\pi\}\{2\},\\;\\text\{if \}\\frac\{\\overline\{\\beta\}\}\{\\overline\{\\gamma\}\}\\in\(\-1,0\)\\\\ &0,\\;\\text\{if \}\\overline\{\\beta\}=\-\\overline\{\\gamma\}\\\\ &\-\\frac\{\\pi\}\{2\},\\;\\text\{if \}\\frac\{\\overline\{\\beta\}\}\{\\overline\{\\gamma\}\}<\-1\\end\{cases\}\(Ifβ¯\\overline\{\\beta\}andγ¯\\overline\{\\gamma\}are interchanged, cases 2 and 4 are switched\.\)

###### Proof\.

Elementary and omitted\. ∎

###### Remark 5\.5\.

For the limit asθ→0\+\\theta\{\\rightarrow\}0\+\. the sum of both terms is always00; for the limit asθ→π−\\theta\{\\rightarrow\}\\pi\-, the sum is zero ifβ¯γ¯<0\\frac\{\\overline\{\\beta\}\}\{\\overline\{\\gamma\}\}<0, otherwise it isπ\\pi\.✠\\maltese

In the limiting casesθ→0\+,π−\\theta\{\\rightarrow\}0\+,\\pi\-, useful information can be given about the terms in the decomposition of the generalizedTT\-function \([5\.21](https://arxiv.org/html/2609.30661#S5.E21)\)\. OwenTT\-function terms\.

###### Lemma 5\.6\(Results forlimθ→0\+,π−\\lim\_\{\\theta\{\\rightarrow\}0\+,\\pi\-\}II\)\.

Suppose thatγ¯≠0\\overline\{\\gamma\}\\neq 0\.

1. \(1\)Limit asθ→0\+\\theta\{\\rightarrow\}0\+\. limθ→0\+T⁡\(γ¯,−cot⁡\(θ\)\+β¯​csc⁡\(θ\)γ¯\)=\{14​erfc​\(\|γ¯\|2\)​if​β¯/γ¯\>10,if​β¯=γ¯−14erfc\(\|γ¯\|2\)ifβ¯/γ¯<1\\lim\_\{\\theta\{\\rightarrow\}0\+\}T\\big\(\\overline\{\\gamma\},\-\\cot\(\\theta\)\+\\frac\{\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\overline\{\\gamma\}\}\\big\)=\\begin\{cases\}&\\frac\{1\}\{4\}\\,\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\|\\overline\{\\gamma\}\|\}\{\\sqrt\{2\}\}\}\\right\)\}\\;\\text\{if \}\\overline\{\\beta\}/\\overline\{\\gamma\}\>1\\\\ &0,\\;\\text\{if \}\\overline\{\\beta\}=\\overline\{\\gamma\}\\\\ &\-\\frac\{1\}\{4\}\\,\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\|\\overline\{\\gamma\}\|\}\{\\sqrt\{2\}\}\}\\right\)\}\\;\\text\{if \}\\overline\{\\beta\}/\\overline\{\\gamma\}<1\\end\{cases\}\(Ifβ¯\\overline\{\\beta\}andγ¯\\overline\{\\gamma\}are interchanged, cases 1 and 3 are switched—assumingβ¯≠0\\overline\{\\beta\}\\neq 0\.\)
2. \(2\)Limit asθ→π−\\theta\{\\rightarrow\}\\pi\-\. limθ→π−T⁡\(γ¯,−cot⁡\(θ\)\+β¯​csc⁡\(θ\)γ¯\)=\{14​erfc​\(\|γ¯\|2\)​if​β¯/γ¯\>014erfc\(\|γ¯\|2\)ifβ¯/γ¯∈\(−1,0\]0​if​β¯=−γ¯−14−erfc​\(\|γ¯\|2\)​if​β¯/γ¯<−1\\lim\_\{\\theta\{\\rightarrow\}\\pi\-\}T\\big\(\\overline\{\\gamma\},\-\\cot\(\\theta\)\+\\frac\{\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\overline\{\\gamma\}\}\\big\)=\\begin\{cases\}&\\frac\{1\}\{4\}\\,\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\|\\overline\{\\gamma\}\|\}\{\\sqrt\{2\}\}\}\\right\)\}\\;\\text\{if \}\\overline\{\\beta\}/\\overline\{\\gamma\}\>0\\\\ &\\frac\{1\}\{4\}\\,\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\|\\overline\{\\gamma\}\|\}\{\\sqrt\{2\}\}\}\\right\)\}\\;\\text\{if \}\\overline\{\\beta\}/\\overline\{\\gamma\}\\in\(\-1,0\]\\\\ &0\\;\\text\{if \}\\overline\{\\beta\}=\-\\overline\{\\gamma\}\\\\ &\-\\frac\{1\}\{4\}\\,\-\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\|\\overline\{\\gamma\}\|\}\{\\sqrt\{2\}\}\}\\right\)\}\\;\\text\{if \}\\overline\{\\beta\}/\\overline\{\\gamma\}<\-1\\end\{cases\}\(Ifβ¯\\overline\{\\beta\}andγ¯\\overline\{\\gamma\}are interchanged, cases 2 and 4 are switched—assumingβ¯≠0\\overline\{\\beta\}\\neq 0\.\)

###### Proof\.

Straightforward using \([4\.16](https://arxiv.org/html/2609.30661#S4.E16)\)\. ∎

###### Remark 5\.7\.

Lemmas[5\.4](https://arxiv.org/html/2609.30661#S5.Thmlemma4),[5\.6](https://arxiv.org/html/2609.30661#S5.Thmlemma6)can be used to give a simple direct proof of the continuity ofHHinβ¯,γ¯\\overline\{\\beta\},\\overline\{\\gamma\}whenθ∈\{0,π\}\\theta\\in\\\{0,\\pi\\\}andβ​γ¯≠0\\beta\\overline\{\\gamma\}\\neq 0\. Note that analyticity ofHHfails ifθ∈\{0,π\}\\theta\\in\\\{0,\\pi\\\}, as in the bias free case\.✠\\maltese

It follows from \([5\.21](https://arxiv.org/html/2609.30661#S5.E21)\) that if we defineS⁡\(β¯,γ¯,θ\)=H⁡\(β¯,γ¯,θ\)−π2​erf​\(β¯2\)S\(\\overline\{\\beta\},\\overline\{\\gamma\},\\theta\)=H\(\\overline\{\\beta\},\\overline\{\\gamma\},\\theta\)\-\\frac\{\\pi\}\{2\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}, then forβ¯​γ¯≠0\\overline\{\\beta\}\\overline\{\\gamma\}\\neq 0,

\(5\.24\)S⁡\(β¯,γ¯,θ\)=tan−1⁡\(−cot⁡\(θ\)\)−\[tan−1⁡\(−cot⁡\(θ\)\+β¯​csc⁡\(θ\)γ¯\)\+tan−1⁡\(−cot⁡\(θ\)\+γ¯​csc⁡\(θ\)β¯\)\]\+2​π​\[T⁡\(γ¯,−cot⁡\(θ\)\+β¯​csc⁡\(θ\)γ¯\)\+T⁡\(β¯,−cot⁡\(θ\)\+γ¯​csc⁡\(θ\)β¯\)\]\\begin\{split\}&\\hskip 28\.90755ptS\(\\overline\{\\beta\},\\overline\{\\gamma\},\\theta\)=\\tan^\{\-1\}\(\-\\cot\(\\theta\)\)\{\}\-\\\\ &\\left\[\\tan^\{\-1\}\\left\(\-\\cot\(\\theta\)\+\\frac\{\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\overline\{\\gamma\}\}\\right\)\+\\tan^\{\-1\}\\left\(\-\\cot\(\\theta\)\+\\frac\{\\overline\{\\gamma\}\\csc\(\\theta\)\}\{\\overline\{\\beta\}\}\\right\)\\right\]\\\\ &\+2\\pi\\left\[T\\left\(\\overline\{\\gamma\},\-\\cot\(\\theta\)\+\\frac\{\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\overline\{\\gamma\}\}\\right\)\+T\\left\(\\overline\{\\beta\},\-\\cot\(\\theta\)\+\\frac\{\\overline\{\\gamma\}\\csc\(\\theta\)\}\{\\overline\{\\beta\}\}\\right\)\\right\]\\end\{split\}The functionS⁡\(β¯,γ¯,θ\)S\(\\overline\{\\beta\},\\overline\{\\gamma\},\\theta\)is symmetric inβ¯,γ¯\\overline\{\\beta\},\\overline\{\\gamma\}:S⁡\(β¯,γ¯,θ\)=S⁡\(γ¯,β¯,θ\)S\(\\overline\{\\beta\},\\overline\{\\gamma\},\\theta\)=S\(\\overline\{\\gamma\},\\overline\{\\beta\},\\theta\)\. Sinceπ2​erf​\(β¯2\)\\frac\{\\pi\}\{2\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}is real analytic inβ¯∈ℝ\\overline\{\\beta\}\\in\{\\mathbb\{R\}\}and the integral definingHHis real analytic onℝ2×\(0,π\)\{\\mathbb\{R\}\}^\{2\}\\times\(0,\\pi\), we have

###### Proposition 5\.8\.

\(Notation and assumptions as above\.\)S⁡\(β¯,γ¯,θ\)S\(\\overline\{\\beta\},\\overline\{\\gamma\},\\theta\)is real analytic and bounded onℝ2×\(0,π\)\{\\mathbb\{R\}\}^\{2\}\\times\(0,\\pi\)\. Moreover,

1. \(1\)S⁡\(β¯,0,θ\)=2​π​T​\(β¯,−cot⁡\(θ\)\)S\(\\overline\{\\beta\},0,\\theta\)=2\\pi T\(\\overline\{\\beta\},\-\\cot\(\\theta\)\),β¯≠0\\overline\{\\beta\}\\neq 0\.
2. \(2\)S⁡\(0,γ¯,θ\)=2​π​T​\(γ¯,−cot⁡\(θ\)\)S\(0,\\overline\{\\gamma\},\\theta\)=2\\pi T\(\\overline\{\\gamma\},\-\\cot\(\\theta\)\),γ¯≠0\\overline\{\\gamma\}\\neq 0\.
3. \(3\)S⁡\(β¯,β¯,θ\)=4​π​T​\(β¯,tan⁡\(θ/2\)\)−π/2S\(\\overline\{\\beta\},\\overline\{\\beta\},\\theta\)=4\\pi T\(\\overline\{\\beta\},\\tan\(\\theta/2\)\)\-\\pi/2,β¯≠0\\overline\{\\beta\}\\neq 0\.
4. \(4\)S⁡\(0,0,θ\)=θ−π2S\(0,0,\\theta\)=\\theta\-\\frac\{\\pi\}\{2\}\.

###### Proof\.

For \(1–4\), use Lemma[5\.2](https://arxiv.org/html/2609.30661#S5.Thmlemma2)\. ∎

###### Remarks 5\.9\.

\(1\) AlthoughSSis real analytic and bounded onℝ2×\(0,π\)\{\\mathbb\{R\}\}^\{2\}\\times\(0,\\pi\), it is not true thatSSis subanalytic\. In particular, the closure of the graph ofSSis not subanalytic\. \(2\) The right hand side of \([5\.24](https://arxiv.org/html/2609.30661#S5.E24)\) comprises four*discontinuous*functions and one analytic function—tan−1⁡\(−cot⁡\(θ\)\)\\tan^\{\-1\}\(\-\\cot\(\\theta\)\)\(cf\. Lemmas[5\.4](https://arxiv.org/html/2609.30661#S5.Thmlemma4),[5\.6](https://arxiv.org/html/2609.30661#S5.Thmlemma6)\)\.✠\\maltese

It is helpful and suggestive to expressS⁡\(β¯,γ¯,θ\)S\(\\overline\{\\beta\},\\overline\{\\gamma\},\\theta\)in terms of “angles”\. To this end, defineθ~=θ~​\(β¯,γ¯,θ\)\\widetilde\{\\theta\}=\\widetilde\{\\theta\}\(\\overline\{\\beta\},\\overline\{\\gamma\},\\theta\)by the equation

\(5\.25\)S⁡\(β¯,γ¯,θ\)=θ~−π2S\(\\overline\{\\beta\},\\overline\{\\gamma\},\\theta\)=\\widetilde\{\\theta\}\-\\frac\{\\pi\}\{2\}and note that

\(5\.26\)θ~\\displaystyle\\hskip 36\.135pt\{\\widetilde\{\\theta\}\}=\\displaystyle=H⁡\(β¯,γ¯,θ\)\+π2​erfc​\(β¯2\)\\displaystyle H\(\\overline\{\\beta\},\\overline\{\\gamma\},\\theta\)\+\\frac\{\\pi\}\{2\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\(5\.27\)=\\displaystyle=π2​∫γ¯∞e−t22​erf​\(−t​cot⁡\(θ\)\+β¯​csc⁡\(θ\)2\)​𝑑t\+π2​erfc​\(β¯2\)\\displaystyle\\sqrt\{\\frac\{\\pi\}\{2\}\}\\int\_\{\\overline\{\\gamma\}\}^\{\\infty\}e^\{\-\\frac\{t^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-t\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}dt\+\\frac\{\\pi\}\{2\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\(5\.28\)=\\displaystyle=π2​∫β¯∞e−t22​erf​\(−t​cot⁡\(θ\)\+γ¯​csc⁡\(θ\)2\)​𝑑t\+π2​erfc​\(γ¯2\)\\displaystyle\\sqrt\{\\frac\{\\pi\}\{2\}\}\\int\_\{\\overline\{\\beta\}\}^\{\\infty\}e^\{\-\\frac\{t^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-t\\cot\(\\theta\)\+\\overline\{\\gamma\}\\csc\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}dt\+\\frac\{\\pi\}\{2\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}
Ifβ¯=γ¯=0\\overline\{\\beta\}=\\overline\{\\gamma\}=0, thenS⁡\(0,0,θ\)=θ−π2S\(0,0,\\theta\)=\\theta\-\\frac\{\\pi\}\{2\}\(Proposition[5\.8](https://arxiv.org/html/2609.30661#S5.Thmlemma8)\) and soθ~=θ\{\\widetilde\{\\theta\}\}=\\theta\. In the sequel, the variable dependence ofθ~\\widetilde\{\\theta\}will be clear from the context and we always writeθ~\\widetilde\{\\theta\}and omit the variable dependence\.

###### Example 5\.10\.

Using either Lemmas[5\.2](https://arxiv.org/html/2609.30661#S5.Thmlemma2)and[5\.4](https://arxiv.org/html/2609.30661#S5.Thmlemma4)or Proposition[5\.8](https://arxiv.org/html/2609.30661#S5.Thmlemma8),

limθ→0\+H⁡\(β¯,β¯,θ\)\\displaystyle\\lim\_\{\\theta\{\\rightarrow\}0\+\}H\(\\overline\{\\beta\},\\overline\{\\beta\},\\theta\)=\\displaystyle=S⁡\(β¯,β¯,0\)\+π2​erf​\(β¯2\)\\displaystyle S\(\\overline\{\\beta\},\\overline\{\\beta\},0\)\+\\frac\{\\pi\}\{2\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}=\\displaystyle=−π2​erfc​\(β¯2\)\\displaystyle\-\\frac\{\\pi\}\{2\}\\,\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}andθ~=0\\widetilde\{\\theta\}=0\. Here, exactly two of the terms in \([5\.21](https://arxiv.org/html/2609.30661#S5.E21)\) are non\-zero: term 1 \(which is−π2\-\\frac\{\\pi\}\{2\}\) and the final term\. A similar analysis can be done forlimθ→π−H⁡\(β¯,−β¯,θ\)=0\\lim\_\{\\theta\{\\rightarrow\}\\pi\-\}H\(\\overline\{\\beta\},\-\\overline\{\\beta\},\\theta\)=0\(see also Lemma[3\.6](https://arxiv.org/html/2609.30661#S3.Thmlemma6)\)\.※

## 6\.Evaluation off⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)concluded

Throughout this section, takeA=−cot⁡\(θ\)A=\-\\cot\(\\theta\),B=csc⁡\(θ\)B=\\csc\(\\theta\)and assumeθ∈\(0,π\)\\theta\\in\(0,\\pi\)\.

#### The constant termβ¯​γ¯​∫D1\\overline\{\\beta\}\\overline\{\\gamma\}\\int\_\{D\}1

We have

β¯​γ¯​∫D1\\displaystyle\\overline\{\\beta\}\\overline\{\\gamma\}\\int\_\{D\}1=\\displaystyle=β¯​γ¯​\[π2​erfc​\(γ¯2\)−\(θ~−π2\)−π2​erf​\(β¯2\)\]\\displaystyle\\overline\{\\beta\}\\overline\{\\gamma\}\\left\[\\frac\{\\pi\}\{2\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\\big\(\\widetilde\{\\theta\}\-\\frac\{\\pi\}\{2\}\\big\)\-\\frac\{\\pi\}\{2\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]=\\displaystyle=β¯​γ¯​\(π2​\[erfc​\(γ¯2\)\+erfc​\(β¯2\)\]−θ~\)\\displaystyle\\overline\{\\beta\}\\overline\{\\gamma\}\\left\(\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\\widetilde\{\\theta\}\\right\)*Check:*β¯​γ¯​∫D1\\overline\{\\beta\}\\overline\{\\gamma\}\\int\_\{D\}1is\(β¯,γ¯\)\(\\overline\{\\beta\},\\overline\{\\gamma\}\)\-symmetric\.

#### The linear term−∫D\(γ¯cos\(θ\)\+β¯\)x1\+γ¯sin\(θ\)x2\-\\int\_\{D\}\(\\overline\{\\gamma\}\\cos\(\\theta\)\+\\overline\{\\beta\}\)x\_\{1\}\+\\overline\{\\gamma\}\\sin\(\\theta\)x\_\{2\}

−∫D\(γ¯​cos⁡\(θ\)\+β¯\)​x1\+γ¯​sin⁡\(θ\)​x2=−π2​e−γ¯22​\[\(γ¯​cos⁡\(θ\)\+β¯\)​erfc​\(A​γ¯\+B​β¯2\)\]−π2​e−β¯22​\[\(γ¯\+β¯​cos⁡\(θ\)\)​erfc​\(A​β¯\+B​γ¯2\)\]\\begin\{aligned\} \-\\int\_\{D\}&\(\\overline\{\\gamma\}\\cos\(\\theta\)\+\\overline\{\\beta\}\)x\_\{1\}\+\\overline\{\\gamma\}\\sin\(\\theta\)x\_\{2\}=\\\\ &\-\\sqrt\{\\frac\{\\pi\}\{2\}\}e^\{\-\\frac\{\\overline\{\\gamma\}^\{2\}\}\{2\}\}\\left\[\(\\overline\{\\gamma\}\\cos\(\\theta\)\+\\overline\{\\beta\}\)\{\\text\{\\rm erfc\}\\left\(\{\\frac\{A\\overline\{\\gamma\}\+B\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ &\-\\sqrt\{\\frac\{\\pi\}\{2\}\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\\left\[\(\\overline\{\\gamma\}\+\\overline\{\\beta\}\\cos\(\\theta\)\)\{\\text\{\\rm erfc\}\\left\(\{\\frac\{A\\overline\{\\beta\}\+B\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\end\{aligned\}*Check:*−∫D\(γ¯cos\(θ\)\+β¯\)x1\+γ¯sin\(θ\)x2\-\\int\_\{D\}\(\\overline\{\\gamma\}\\cos\(\\theta\)\+\\overline\{\\beta\}\)x\_\{1\}\+\\overline\{\\gamma\}\\sin\(\\theta\)x\_\{2\}is\(β¯,γ¯\)\(\\overline\{\\beta\},\\overline\{\\gamma\}\)\-symmetric\.

#### The first quadratic term∫Dsin⁡\(θ\)​x1​x2\\int\_\{D\}\\sin\(\\theta\)x\_\{1\}x\_\{2\}

∫Dsin⁡\(θ\)​x1​x2=sin2⁡\(θ\)​e−β¯22​\[π2​β¯​cos⁡\(θ\)​erfc​\(A​β¯\+B​γ¯2\)\+sin⁡\(θ\)​e−\(A​β¯\+B​γ¯\)22\]\\int\_\{D\}\\sin\(\\theta\)x\_\{1\}x\_\{2\}=\\sin^\{2\}\(\\theta\)e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\\left\[\\sqrt\{\\frac\{\\pi\}\{2\}\}\\overline\{\\beta\}\\cos\(\\theta\)\{\\text\{\\rm erfc\}\\left\(\{\\frac\{A\\overline\{\\beta\}\+B\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\sin\(\\theta\)e^\{\-\\frac\{\(A\\overline\{\\beta\}\+B\\overline\{\\gamma\}\)^\{2\}\}\{2\}\}\\right\]This is consistent with the result for∫Dsin⁡\(θ\)​x1​x2\\int\_\{D\}\\sin\(\\theta\)x\_\{1\}x\_\{2\}when the bias is zero\. Obviously this term is not\(β¯,γ¯\)\(\\overline\{\\beta\},\\overline\{\\gamma\}\)\-symmetric\. Consequently, the sum of the quadratic terms∫Dsin⁡\(θ\)​x1​x2\+∫Dcos⁡\(θ\)​x12\\int\_\{D\}\\sin\(\\theta\)x\_\{1\}x\_\{2\}\+\\int\_\{D\}\\cos\(\\theta\)x\_\{1\}^\{2\}must be\(β¯,γ¯\)\(\\overline\{\\beta\},\\overline\{\\gamma\}\)\-symmetric\.

#### The second quadratic term∫Dcos⁡\(θ\)​x12\\int\_\{D\}\\cos\(\\theta\)x\_\{1\}^\{2\}

∫Dcos⁡\(θ\)​x12=cos⁡\(θ\)​π2​\(1−erf​\(γ¯2\)\)\+π2​\[cos⁡\(θ\)​γ¯​e−γ¯22​erfc​\(A​γ¯\+B​β¯2\)\]\+π2​\[cos3⁡\(θ\)​β¯​e−β¯22​erfc​\(A​β¯\+B​γ¯2\)\]\+sin⁡\(θ\)​cos2⁡\(θ\)​e−β¯22​e−\(A​β¯\+B​γ¯\)22−cos⁡\(θ\)​\(θ~−π2​erfc​\(β¯2\)\)−cos⁡\(θ\)​π2​erf​\(β¯2\)\\begin\{aligned\} \\int\_\{D\}\\cos\(\\theta\)x\_\{1\}^\{2\}=&\\cos\(\\theta\)\\frac\{\\pi\}\{2\}\\big\(1\-\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\big\)\\\\ &\+\\sqrt\{\\frac\{\\pi\}\{2\}\}\\left\[\\cos\(\\theta\)\\overline\{\\gamma\}e^\{\-\\frac\{\\overline\{\\gamma\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{A\\overline\{\\gamma\}\+B\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ &\+\\sqrt\{\\frac\{\\pi\}\{2\}\}\\left\[\\cos^\{3\}\(\\theta\)\\overline\{\\beta\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{A\\overline\{\\beta\}\+B\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ &\+\\sin\(\\theta\)\\cos^\{2\}\(\\theta\)e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}e^\{\-\\frac\{\(A\\overline\{\\beta\}\+B\\overline\{\\gamma\}\)^\{2\}\}\{2\}\}\\\\ &\-\\cos\(\\theta\)\\big\(\\widetilde\{\\theta\}\-\\frac\{\\pi\}\{2\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\big\)\-\\cos\(\\theta\)\\frac\{\\pi\}\{2\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\end\{aligned\}It is convenient to define functionsk,Kk,Kof two biases and one angle by

k⁡\(β¯,γ¯,θ\)\\displaystyle k\(\\overline\{\\beta\},\\overline\{\\gamma\},\\theta\)=\\displaystyle=β¯22\+\(A​β¯\+B​γ¯\)22=γ¯22\+\(A​γ¯\+B​β¯\)22\\displaystyle\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\+\\frac\{\(A\\overline\{\\beta\}\+B\\overline\{\\gamma\}\)^\{2\}\}\{2\}=\\frac\{\\overline\{\\gamma\}^\{2\}\}\{2\}\+\\frac\{\(A\\overline\{\\gamma\}\+B\\overline\{\\beta\}\)^\{2\}\}\{2\}=\\displaystyle=\(β¯2\+γ¯2\)​csc2⁡\(θ\)−2​β¯​γ¯​cot⁡\(θ\)​csc⁡\(θ\)2\\displaystyle\\frac\{\(\\overline\{\\beta\}^\{2\}\+\\overline\{\\gamma\}^\{2\}\)\\csc^\{2\}\(\\theta\)\-2\\overline\{\\beta\}\\overline\{\\gamma\}\\cot\(\\theta\)\\csc\(\\theta\)\}\{2\}We use capitalKKto indicate a functionK⁡\(β¯i,β¯j,Θ\)K\(\\overline\{\\beta\}\_\{i\},\\overline\{\\beta\}\_\{j\},\\Theta\)whereΘ\\Thetawill now be an angle between two rows𝐰i,𝐰j\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{j\}of𝐖\{\{\\mathbf\{W\}\}\}as opposed to the angle between a row of𝐖\{\{\\mathbf\{W\}\}\}and a row of𝐕\{\{\\mathbf\{V\}\}\}\.

After using some elementary trigonometric identities, we find that if‖𝐰‖,‖𝐯‖≠0\\\|\{\{\\mathbf\{w\}\}\}\\\|,\\\|\{\{\\mathbf\{v\}\}\}\\\|\\neq 0, then

∫Dcos⁡\(θ\)​x12\+sin⁡\(θ\)​x1​x2=cos⁡\(θ\)​π2​\[erfc​\(γ¯2\)\+erfc​\(β¯2\)\]−cos⁡\(θ\)​θ~\+π2​cos⁡\(θ\)​\[β¯​e−β¯22\+γ¯​e−γ¯22\]−π2​cos⁡\(θ\)​\[β¯​e−β¯22​erf​\(A​β¯\+B​γ¯2\)\+γ¯​e−γ¯22​erf​\(A​γ¯\+B​β¯2\)\]\+sin⁡\(θ\)​e−k⁡\(β¯,γ¯,θ\)\\begin\{aligned\} \\int\_\{D\}&\\cos\(\\theta\)x\_\{1\}^\{2\}\+\\sin\(\\theta\)x\_\{1\}x\_\{2\}=\\\\ &\\cos\(\\theta\)\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\\cos\(\\theta\)\\widetilde\{\\theta\}\\\\ &\+\\sqrt\{\\frac\{\\pi\}\{2\}\}\\cos\(\\theta\)\\left\[\\overline\{\\beta\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\+\\overline\{\\gamma\}e^\{\-\\frac\{\\overline\{\\gamma\}^\{2\}\}\{2\}\}\\right\]\\\\ &\-\\sqrt\{\\frac\{\\pi\}\{2\}\}\\cos\(\\theta\)\\left\[\\overline\{\\beta\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{A\\overline\{\\beta\}\+B\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\overline\{\\gamma\}e^\{\-\\frac\{\\overline\{\\gamma\}^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{A\\overline\{\\gamma\}\+B\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ &\+\\sin\(\\theta\)e^\{\-k\(\\overline\{\\beta\},\\overline\{\\gamma\},\\theta\)\}\\end\{aligned\}*Checks:*∫Dcos⁡\(θ\)​x12\+sin⁡\(θ\)​x1​x2\\int\_\{D\}\\cos\(\\theta\)x\_\{1\}^\{2\}\+\\sin\(\\theta\)x\_\{1\}x\_\{2\}is\(β¯,γ¯\)\(\\overline\{\\beta\},\\overline\{\\gamma\}\)\-symmetric and ifγ¯,β¯=0\\overline\{\\gamma\},\\overline\{\\beta\}=0,∫Dcos⁡\(θ\)​x12\+sin⁡\(θ\)​x1​x2=sin⁡\(θ\)\+\(π−θ\)​cos⁡\(θ\)\\int\_\{D\}\\cos\(\\theta\)x\_\{1\}^\{2\}\+\\sin\(\\theta\)x\_\{1\}x\_\{2\}=\\sin\(\\theta\)\+\(\\pi\-\\theta\)\\cos\(\\theta\)\.

#### The formula forf⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)

For this we use the expressions computed above and the abbreviationsA=−cot⁡\(θ\),B=csc⁡\(θ\)A=\-\\cot\(\\theta\),B=\\csc\(\\theta\)\. Many terms cancel and we find that

\(6\.29\)f\(𝐰,OPEN𝐯\)=‖𝐰‖​‖𝐯‖​\[β¯​γ¯\+cos⁡\(θ\)\]2​π​\(π2​\[erfc​\(γ¯2\)\+erfc​\(β¯2\)\]−θ~\)−‖𝐰‖​‖𝐯‖2​2​π​\[γ¯​e−β¯22​erfc​\(A​β¯\+B​γ¯2\)\+β¯​e−γ¯22​erfc​\(A​γ¯\+B​β¯2\)\]\+‖𝐰‖​‖𝐯‖2​π​sin⁡\(θ\)​e−k⁡\(β¯,γ¯,θ\)\\begin\{aligned\} f\(\{\{\\mathbf\{w\}\}\},&\{\{\\mathbf\{v\}\}\}\)=\\\\ &\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\[\\overline\{\\beta\}\\overline\{\\gamma\}\+\\cos\(\\theta\)\]\}\{2\\pi\}\\left\(\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\\widetilde\{\\theta\}\\right\)\\\\ &\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\left\[\\overline\{\\gamma\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{A\\overline\{\\beta\}\+B\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\overline\{\\beta\}e^\{\-\\frac\{\\overline\{\\gamma\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{A\\overline\{\\gamma\}\+B\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\pi\}\\sin\(\\theta\)e^\{\-k\(\\overline\{\\beta\},\\overline\{\\gamma\},\\theta\)\}\\end\{aligned\}
###### Remark 6\.1\.

Using \([6\.29](https://arxiv.org/html/2609.30661#S6.E29)\) forf⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\), it is a useful exercise to deduce the result forf⁡\(𝐰,𝐰\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{w\}\}\}\)given in \([3\.5](https://arxiv.org/html/2609.30661#S3.E5)\) as well as Lemmas[3\.5](https://arxiv.org/html/2609.30661#S3.Thmlemma5),[3\.6](https://arxiv.org/html/2609.30661#S3.Thmlemma6)covering the casesθ∈\{0,π\}\\theta\\in\\\{0,\\pi\\\}\.✠\\maltese

The section concludes with some examples and computations used later\. Some preliminary notational conventions are needed first\.

### 6\.1\.Notational conventions & formulas forf⁡\(𝐰i,𝐯j\),f⁡\(𝐰i,𝐰j\)f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{j\}\),f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{j\}\)

\. Givend∈ℕd\\in\\mbox\{$\\mathbb\{N\}$\}, define𝐝=\{0,1,…,d−1\}\{\\mathbf\{d\}\}=\\\{0,1,\\ldots,d\-1\\\}and\[d\]=\{1,2,…,d\}\[d\]=\\\{1,2,\\ldots,d\\\}\. We fix a matrix𝐕∈M⁡\(d,d\)\{\{\\mathbf\{V\}\}\}\\in M\(d,d\)and denote the rows by𝐯0,…,𝐯d−1\{\{\\mathbf\{v\}\}\}^\{0\},\\ldots,\{\{\\mathbf\{v\}\}\}^\{d\-1\}\. Later we refer to𝐕\{\{\\mathbf\{V\}\}\}as the “target”: the hoped for end result of the learning process \(gradient descent in our set up\)\. Generally, it is assumed that𝐕\{\{\\mathbf\{V\}\}\}has no parallel rows\. Typically, we take𝐕\{\{\\mathbf\{V\}\}\}to be the identity matrix and\{𝐯i\}i∈𝐝\\\{\{\{\\mathbf\{v\}\}\}^\{i\}\\\}\_\{i\\in\{\\mathbf\{d\}\}\}to be the standard orthonormal basis of\(ℝd\)⋆≈ℝd\(\{\\mathbb\{R\}\}^\{d\}\)^\{\\star\}\\approx\{\\mathbb\{R\}\}^\{d\}\(by the Euclidean inner product\)\.

Next, let𝐖∈M⁡\(k,d\)\{\{\\mathbf\{W\}\}\}\\in M\(k,d\),k≥d∈ℕk\\geq d\\in\\mbox\{$\\mathbb\{N\}$\}\. Assume𝐖\{\{\\mathbf\{W\}\}\}has no parallel rows and label rows of𝐖\{\{\\mathbf\{W\}\}\}by𝐤\{\\mathbf\{k\}\}:𝐰0,…,𝐰k−1\{\{\\mathbf\{w\}\}\}^\{0\},\\ldots,\{\{\\mathbf\{w\}\}\}^\{k\-1\}\. Given rows𝐰i,𝐰j\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{j\}in𝐖\{\{\\mathbf\{W\}\}\},𝐯j\{\{\\mathbf\{v\}\}\}^\{j\}in𝐕\{\{\\mathbf\{V\}\}\}, define

Λi​j=cos−1⁡\(⟨𝐰i,𝐰j⟩‖𝐰i‖​‖𝐰j‖\),λi​j=cos−1⁡\(⟨𝐰i,𝐯j⟩‖𝐰i‖​‖𝐯j‖\)\.\\Lambda\_\{ij\}=\\cos^\{\-1\}\\left\(\\frac\{\\langle\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{j\}\\rangle\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\}\\right\),\\;\\lambda\_\{ij\}=\\cos^\{\-1\}\\left\(\\frac\{\\langle\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{j\}\\rangle\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|\}\\right\)\.Note thatΛi​j=Λj​i\\Lambda\_\{ij\}=\\Lambda\_\{ji\}, alli,j∈𝐤i,j\\in\{\\mathbf\{k\}\}and thatΛi​i=0\\Lambda\_\{ii\}=0\. This is not the case for the anglesλi​j\\lambda\_\{ij\}; indeed,λj​i\\lambda\_\{ji\}is not be defined ifj\>dj\>d\.

It is helpful to develop a more concise notation for the terms that occur inf⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)\. Continuing with our assumptions on𝐖,𝐕\{\{\\mathbf\{W\}\}\},\{\{\\mathbf\{V\}\}\},

1. \(a1\)Fori,j∈𝐤i,j\\in\{\\mathbf\{k\}\},j≠ij\\neq i, define Zi​j=−cot⁡\(Λi​j\)​β¯j\+csc⁡\(Λi​j\)​β¯i,Z^i​j=−cot⁡\(Λi​j\)​β¯i\+csc⁡\(Λi​j\)​β¯jZ\_\{ij\}=\-\\cot\(\\Lambda\_\{ij\}\)\\overline\{\\beta\}\_\{j\}\+\\csc\(\\Lambda\_\{ij\}\)\\overline\{\\beta\}\_\{i\},\\qquad\{\\widehat\{Z\}\}\_\{ij\}=\-\\cot\(\\Lambda\_\{ij\}\)\\overline\{\\beta\}\_\{i\}\+\\csc\(\\Lambda\_\{ij\}\)\\overline\{\\beta\}\_\{j\}K⁡\(β¯i,β¯j,Λi​j\)=12​\(\(β¯i2\+β¯j2\)​csc2⁡\(Λi​j\)−2​β¯i​β¯j​csc⁡\(Λi​j\)​cot⁡\(Λi​j\)\)K\(\\overline\{\\beta\}\_\{i\},\\overline\{\\beta\}\_\{j\},\\Lambda\_\{ij\}\)=\\frac\{1\}\{2\}\\big\(\(\\overline\{\\beta\}\_\{i\}^\{2\}\+\\overline\{\\beta\}\_\{j\}^\{2\}\)\\csc^\{2\}\(\\Lambda\_\{ij\}\)\-2\\overline\{\\beta\}\_\{i\}\\overline\{\\beta\}\_\{j\}\\csc\(\\Lambda\_\{ij\}\)\\cot\(\\Lambda\_\{ij\}\)\\big\)
2. \(a2\)Fori∈𝐤,j∈𝐝i\\in\{\\mathbf\{k\}\},j\\in\{\\mathbf\{d\}\}, define zi​j=−cot⁡\(λi​j\)​γ¯j\+csc⁡\(λi​j\)​β¯i,z^i​j=−cot⁡\(λi​j\)​β¯i\+csc⁡\(λi​j\)​γ¯jz\_\{ij\}=\-\\cot\(\\lambda\_\{ij\}\)\\overline\{\\gamma\}\_\{j\}\+\\csc\(\\lambda\_\{ij\}\)\\overline\{\\beta\}\_\{i\},\\qquad\{\\widehat\{z\}\}\_\{ij\}=\-\\cot\(\\lambda\_\{ij\}\)\\overline\{\\beta\}\_\{i\}\+\\csc\(\\lambda\_\{ij\}\)\\overline\{\\gamma\}\_\{j\}k⁡\(β¯i,γ¯j,λi​j\)=12​\(\(β¯i2\+γ¯j2\)​csc2⁡\(λi​j\)−2​β¯i​γ¯j​csc⁡\(λi​j\)​cot⁡\(λi​j\)\)k\(\\overline\{\\beta\}\_\{i\},\\overline\{\\gamma\}\_\{j\},\\lambda\_\{ij\}\)=\\frac\{1\}\{2\}\\big\(\(\\overline\{\\beta\}\_\{i\}^\{2\}\+\\overline\{\\gamma\}\_\{j\}^\{2\}\)\\csc^\{2\}\(\\lambda\_\{ij\}\)\-2\\overline\{\\beta\}\_\{i\}\\overline\{\\gamma\}\_\{j\}\\csc\(\\lambda\_\{ij\}\)\\cot\(\\lambda\_\{ij\}\)\\big\)

With these conventions,

f\(𝐰i,OPEN𝐯j\)=‖𝐰i‖​‖𝐯j‖​\[β¯i​γ¯j\+cos⁡\(λi​j\)\]2​π​\(π2​\[erfc​\(β¯i2\)\+erfc​\(γ¯j2\)\]−λ~i​j\)−‖𝐰i‖​‖𝐯j‖2​2​π​\[γ¯j​e−β¯i22​erfc​\(z^i​j2\)\+β¯i​e−γ¯j22​erfc​\(zi​j2\)\]\+‖𝐰i‖​‖𝐯j‖2​π​sin⁡\(λi​j\)​e−k⁡\(β¯i,γ¯j,λi​j\),i∈𝐤,j∈𝐝,f\(𝐰i,OPEN𝐰j\)=‖𝐰i‖​‖𝐰j‖​\[β¯i​β¯j\+cos⁡\(Λi​j\)\]2​π​\(π2​\[erfc​\(β¯i2\)\+erfc​\(β¯j2\)\]−Λ~i​j\)−‖𝐰i‖​‖𝐰j‖2​2​π​\[β¯j​e−β¯i22​erfc​\(Z^i​j2\)\+β¯i​e−β¯j22​erfc​\(Zi​j2\)\]\+‖𝐰i‖​‖𝐰j‖2​π​sin⁡\(Λi​j\)​e−K⁡\(β¯i,β¯j,Λi​j\),i,j∈𝐤,j≠i\\begin\{aligned\} f\(\{\{\\mathbf\{w\}\}\}^\{i\},&\{\{\\mathbf\{v\}\}\}^\{j\}\)=\\\\ &\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|\[\\overline\{\\beta\}\_\{i\}\\overline\{\\gamma\}\_\{j\}\+\\cos\(\\lambda\_\{ij\}\)\]\}\{2\\pi\}\\left\(\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\{\\widetilde\{\\lambda\}\}\_\{ij\}\\right\)\\\\ &\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\left\[\\overline\{\\gamma\}\_\{j\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\{\\widehat\{z\}\}\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\overline\{\\beta\}\_\{i\}e^\{\-\\frac\{\\overline\{\\gamma\}\_\{j\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{z\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|\}\{2\\pi\}\\sin\(\\lambda\_\{ij\}\)e^\{\-k\(\\overline\{\\beta\}\_\{i\},\\overline\{\\gamma\}\_\{j\},\\lambda\_\{ij\}\)\},\\;\\;i\\in\{\\mathbf\{k\}\},j\\in\{\\mathbf\{d\}\},\\\\ f\(\{\{\\mathbf\{w\}\}\}^\{i\},&\{\{\\mathbf\{w\}\}\}^\{j\}\)=\\\\ &\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\[\\overline\{\\beta\}\_\{i\}\\overline\{\\beta\}\_\{j\}\+\\cos\(\\Lambda\_\{ij\}\)\]\}\{2\\pi\}\\left\(\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\{\\widetilde\{\\Lambda\}\}\_\{ij\}\\right\)\\\\ &\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\left\[\\overline\{\\beta\}\_\{j\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\{\\widehat\{Z\}\}\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\overline\{\\beta\}\_\{i\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{j\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{Z\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\}\{2\\pi\}\\sin\(\\Lambda\_\{ij\}\)e^\{\-K\(\\overline\{\\beta\}\_\{i\},\\overline\{\\beta\}\_\{j\},\\Lambda\_\{ij\}\)\},\\;\\;i,j\\in\{\\mathbf\{k\}\},j\\neq i\\end\{aligned\}\(Fori=ji=j, see Example[3\.4](https://arxiv.org/html/2609.30661#S3.Thmlemma4), equation \([3\.5](https://arxiv.org/html/2609.30661#S3.E5)\) and also \([6\.30](https://arxiv.org/html/2609.30661#S6.E30)\) below\.\)

Ifγ¯i=0\\overline\{\\gamma\}\_\{i\}=0, for alli∈𝐝i\\in\{\\mathbf\{d\}\}—the only case considered in this paper—then fori∈𝐤i\\in\{\\mathbf\{k\}\}andj∈𝐝j\\in\{\\mathbf\{d\}\}

f\(𝐰i,OPEN𝐯j\)=‖𝐰i‖​‖𝐯j‖​cos⁡\(λi​j\)2​π​\(π−λ~i​j−π2​erf​\(β¯i2\)\)−‖𝐰i‖​‖𝐯j‖2​2​π​β¯i​erfc​\(csc⁡\(λi​j\)​β¯i2\)\+‖𝐰i‖​‖𝐯j‖2​π​sin⁡\(λi​j\)​e−β¯i2​csc2⁡\(λi​j\)2\\begin\{aligned\} f\(\{\{\\mathbf\{w\}\}\}^\{i\},&\{\{\\mathbf\{v\}\}\}^\{j\}\)=\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|\\cos\(\\lambda\_\{ij\}\)\}\{2\\pi\}\\left\(\\pi\-\{\\widetilde\{\\lambda\}\}\_\{ij\}\-\\frac\{\\pi\}\{2\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\)\\\\ &\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\overline\{\\beta\}\_\{i\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\csc\(\\lambda\_\{ij\}\)\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|\}\{2\\pi\}\\sin\(\\lambda\_\{ij\}\)e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\\csc^\{2\}\(\\lambda\_\{ij\}\)\}\{2\}\}\\end\{aligned\}When symmetry is present, we may haveβ¯i=β¯j\\overline\{\\beta\}\_\{i\}=\\overline\{\\beta\}\_\{j\},i≠ji\\neq j\. Settingβ¯i,β¯j=β¯\\overline\{\\beta\}\_\{i\},\\overline\{\\beta\}\_\{j\}=\\overline\{\\beta\}andΛi​j=Θ\\Lambda\_\{ij\}=\\Theta, the formula forf⁡\(𝐰i,𝐰j\)f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{j\}\)is

f⁡\(𝐰i,𝐰j\)=‖𝐰i‖​‖𝐰j‖​\[β¯2\+cos⁡\(Θ\)\]2​π​\(π​erfc​\(β¯2\)−Θ~\)−‖𝐰i‖​‖𝐰j‖2​π​β¯​e−β¯22​erfc​\(β⁡\(−cot⁡\(Θ\)\+csc⁡\(Θ\)\)2\)\+‖𝐰i‖​‖𝐰j‖2​π​sin⁡\(Θ\)​e−β¯2​csc⁡\(Θ\)​\(csc⁡\(Θ\)−cot⁡\(Θ\)\)​i,j∈𝐤,j≠i\\begin\{aligned\} f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{j\}\)=&\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\[\\overline\{\\beta\}^\{2\}\+\\cos\(\\Theta\)\]\}\{2\\pi\}\\left\(\\pi\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\{\\widetilde\{\\Theta\}\}\\right\)\\\\ &\\hskip\-28\.90755pt\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\}\{\\sqrt\{2\\pi\}\}\\overline\{\\beta\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\beta\\big\(\-\\cot\(\\Theta\)\+\\csc\(\\Theta\)\\big\)\}\{\\sqrt\{2\}\}\}\\right\)\}\\\\ &\\hskip\-28\.90755pt\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\}\{2\\pi\}\\sin\(\\Theta\)e^\{\-\\overline\{\\beta\}^\{2\}\\csc\(\\Theta\)\\big\(\\csc\(\\Theta\)\-\\cot\(\\Theta\)\\big\)\}\\;\\;i,j\\in\{\\mathbf\{k\}\},j\\neq i\\end\{aligned\}
###### Example 6\.2\.

Let\{𝐮i\}i∈𝐝\\\{\{\{\\mathbf\{u\}\}\}^\{i\}\\\}\_\{i\\in\{\\mathbf\{d\}\}\}be an orthogonal basis ofℝd\{\\mathbb\{R\}\}^\{d\}such that each𝐮i\{\{\\mathbf\{u\}\}\}^\{i\}is a non\-zero multiple of𝐯i\{\{\\mathbf\{v\}\}\}^\{i\},i∈𝐝i\\in\{\\mathbf\{d\}\}\. Fori≠ji\\neq j, the angleϕi​j\\phi\_\{ij\}between𝐮i\{\{\\mathbf\{u\}\}\}^\{i\}and𝐮j\{\{\\mathbf\{u\}\}\}^\{j\}\(or𝐯j\{\{\\mathbf\{v\}\}\}^\{j\}\) isπ/2\\pi/2\. Fori∈𝐝i\\in\{\\mathbf\{d\}\},

\(6\.30\)f⁡\(𝐮i,𝐮i\)=‖𝐮i‖22​\(β¯i2\+1\)​erfc​\(β¯i2\)−‖𝐮i‖22​π​β¯i​e−β¯i22f\(\{\{\\mathbf\{u\}\}\}^\{i\},\{\{\\mathbf\{u\}\}\}^\{i\}\)=\\frac\{\\\|\{\{\\mathbf\{u\}\}\}^\{i\}\\\|^\{2\}\}\{2\}\\big\(\\overline\{\\beta\}\_\{i\}^\{2\}\+1\\big\)\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\\frac\{\\\|\{\{\\mathbf\{u\}\}\}^\{i\}\\\|^\{2\}\}\{\\sqrt\{2\\pi\}\}\\overline\{\\beta\}\_\{i\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}Fori,j∈𝐝i,j\\in\{\\mathbf\{d\}\},i≠ji\\neq j,

\(6\.31\)f\(𝐮i,OPEN𝐮j\)=‖𝐮i‖​‖𝐮j‖4​β¯i​β¯j​erfc​\(β¯i2\)​erfc​\(β¯j2\)−‖𝐮i‖​‖𝐮j‖2​2​π​\[β¯j​e−β¯i22​erfc​\(β¯j2\)\+β¯i​e−β¯j22​erfc​\(β¯i2\)\]\+‖𝐮i‖​‖𝐮j‖2​π​e−K⁡\(β¯i,β¯j,π/2\),\\begin\{aligned\} f\(\{\{\\mathbf\{u\}\}\}^\{i\},&\{\{\\mathbf\{u\}\}\}^\{j\}\)=\\frac\{\\\|\{\{\\mathbf\{u\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{u\}\}\}^\{j\}\\\|\}\{4\}\\overline\{\\beta\}\_\{i\}\\overline\{\\beta\}\_\{j\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\\\ &\-\\frac\{\\\|\{\{\\mathbf\{u\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{u\}\}\}^\{j\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\left\[\\overline\{\\beta\}\_\{j\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\overline\{\\beta\}\_\{i\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{j\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{u\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{u\}\}\}^\{j\}\\\|\}\{2\\pi\}e^\{\-K\(\\overline\{\\beta\}\_\{i\},\\overline\{\\beta\}\_\{j\},\\pi/2\)\},\\end\{aligned\}whereK⁡\(β¯i,β¯j,π/2\)=12​\(β¯i2\+β¯j2\)K\(\\overline\{\\beta\}\_\{i\},\\overline\{\\beta\}\_\{j\},\\pi/2\)=\\frac\{1\}\{2\}\(\\overline\{\\beta\}\_\{i\}^\{2\}\+\\overline\{\\beta\}\_\{j\}^\{2\}\)and we used2π​H​\(β¯i,β¯j,π/2\)=erfc​\(β¯j2\)​erf​\(β¯i2\)\\frac\{2\}\{\\pi\}H\(\\overline\{\\beta\}\_\{i\},\\overline\{\\beta\}\_\{j\},\\pi/2\)=\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\. The reader may confirm that the result is consistent with the alternative description in terms ofϕ~i​j\\widetilde\{\\phi\}\_\{ij\}\.※

## 7\.Gradient off⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)with respect to\(𝐰,β¯\)\(\{\{\\mathbf\{w\}\}\},\\overline\{\\beta\}\): I

Our strategy will be to minimize the number of evaluations of theTTfunction\. In particular, when we consider derivatives ofH⁡\(β¯,γ¯,θ\)=2​π​T​\(γ¯,−cot⁡\(θ\),β¯​csc⁡\(θ\)\)H\(\\overline\{\\beta\},\\overline\{\\gamma\},\\theta\)=2\\pi T\(\\overline\{\\gamma\},\-\\cot\(\\theta\),\\overline\{\\beta\}\\csc\(\\theta\)\), we differentiate the defining integral \([3\.11](https://arxiv.org/html/2609.30661#S3.E11)\) rather than the OwenTT\-functions and inverse tangents that appear in \([5\.21](https://arxiv.org/html/2609.30661#S5.E21)\)\.

#### Notational conventions

If there areddinputs andkk\-neurons and there is non\-zero bias, then the parameter space will beM⁡\(k,d\+1\)M\(k,d\+1\)\. Ifi∈𝐤i\\in\{\\mathbf\{k\}\}, and𝐖∈M⁡\(k,d\)⊂M⁡\(k,d\+1\)\{\{\\mathbf\{W\}\}\}\\in M\(k,d\)\\subset M\(k,d\+1\), then rowiiof𝐖\{\{\\mathbf\{W\}\}\}is denoted by𝐰i\{\{\\mathbf\{w\}\}\}^\{i\}and

𝐰i=\(wi​0,…,wi​d−1\)∈\(ℝd\)⋆\{\{\\mathbf\{w\}\}\}^\{i\}=\(w\_\{i0\},\\ldots,w\_\{id\-1\}\)\\in\(\{\\mathbb\{R\}\}^\{d\}\)^\{\\star\}Note that𝐰i\{\{\\mathbf\{w\}\}\}^\{i\}is a row vector—linear functional\. Thus the matrix multiplication𝐰i​𝐱\{\{\\mathbf\{w\}\}\}^\{i\}\{\\mathbf\{x\}\}is well defined for all vectors𝐱∈ℝd\{\\mathbf\{x\}\}\\in\{\\mathbb\{R\}\}^\{d\}\. With bias included, we sometimes define𝐖~∈M⁡\(k,d\+1\)\{\\widetilde\{\{\{\\mathbf\{W\}\}\}\}\}\\in M\(k,d\+1\)by𝐖~=\[𝐖,𝜷¯\]\{\\widetilde\{\{\{\\mathbf\{W\}\}\}\}\}=\[\{\{\\mathbf\{W\}\}\},\\boldsymbol\{\\overline\{\\beta\}\}\]where𝜷¯\\boldsymbol\{\\overline\{\\beta\}\}is thek×1k\\times 1column matrix of biases and rowiiof𝐖~\{\\widetilde\{\{\{\\mathbf\{W\}\}\}\}\}is denoted by𝐰~i\\widetilde\{\{\{\\mathbf\{w\}\}\}\}^\{i\}so that

𝐰~i=\(wi​0,…,wi​d−1,β¯i\)∈\(ℝd\)⋆×ℝ,\\widetilde\{\{\{\\mathbf\{w\}\}\}\}^\{i\}=\(w\_\{i0\},\\ldots,w\_\{id\-1\},\\overline\{\\beta\}\_\{i\}\)\\in\(\{\\mathbb\{R\}\}^\{d\}\)^\{\\star\}\\times\{\\mathbb\{R\}\},whereβ¯i\\overline\{\\beta\}\_\{i\}is the \(normalized\) bias associated to neuronii\. Mostly we stick to𝐖,𝐰\{\{\\mathbf\{W\}\}\},\{\{\\mathbf\{w\}\}\}when the meaning is clear from the context\.

We regard∂/∂𝐰\\partial/\\partial\{\{\\mathbf\{w\}\}\}as shorthand for the gradient operator in the variable𝐰\{\{\\mathbf\{w\}\}\}\. That is,∂f∂𝐰\\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\}is the derivativeD​f𝐰∈\(ℝ⋆\)dDf\_\{\{\\mathbf\{w\}\}\}\\in\(\{\\mathbb\{R\}\}^\{\\star\}\)^\{d\}followed by identification ofD​f𝐰Df\_\{\{\\mathbf\{w\}\}\}with the point inℝd⊂ℝd\+1\{\\mathbb\{R\}\}^\{d\}\\subset\{\\mathbb\{R\}\}^\{d\+1\}defined by the Euclidean inner product\. Note that absence of bold face forwwindicates the standard partial derivative\. Setgrad​\(f\)​\(𝐰~\)=\(∂f∂𝐰,∂f∂𝜷¯\)\{\\text\{\\rm grad\}\(\{f\}\)\}\(\\widetilde\{\{\{\\mathbf\{w\}\}\}\}\)=\(\\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\},\\frac\{\\partial f\}\{\\partial\\boldsymbol\{\\overline\{\\beta\}\}\}\)\.

###### Remark 7\.1\.

From Section[12](https://arxiv.org/html/2609.30661#S12), the focus will be on networks with a symmetric target and thenλ\\lambdais used for the angle between𝐰\{\{\\mathbf\{w\}\}\}and𝐯\{\{\\mathbf\{v\}\}\}inf⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)andΛ\\Lambdafor angle between𝐰,𝐰′\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{w\}\}\}^\{\\prime\}inf⁡\(𝐰,𝐰′\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{w\}\}\}^\{\\prime\}\)when there is no symmetry relation\. Moreover,θ\\theta\(resp\.Θ\\Theta\) will be used for the angle between𝐰\{\{\\mathbf\{w\}\}\}and𝐯\{\{\\mathbf\{v\}\}\}\(resp\.𝐰\{\{\\mathbf\{w\}\}\}and𝐰′\{\{\\mathbf\{w\}\}\}^\{\\prime\}\) when the associated rows have the same symmetry or*rowtype*\(see Section[12](https://arxiv.org/html/2609.30661#S12)\)\. Sinceθ\\thetais often used to denote angles between rows in the literature \(for example,\[[10](https://arxiv.org/html/2609.30661#bib.bib10),[9](https://arxiv.org/html/2609.30661#bib.bib9)\]where the networks are bias free\), it seemed best to start by following this convention\. However, from now onλ\\lambdawill generally be used for the angle between a row of𝐖\{\{\\mathbf\{W\}\}\}and a row of𝐕\{\{\\mathbf\{V\}\}\}andΛ\\Lambdafor angles between \(different\) rows of𝐖\{\{\\mathbf\{W\}\}\}\.✠\\maltese

###### Example 7\.2\.

Recall from Example[3\.4](https://arxiv.org/html/2609.30661#S3.Thmlemma4)that

f⁡\(𝐰,𝐰\)=‖𝐰‖2​\(β¯2\+1\)2​erfc​\(β¯2\)−‖𝐰‖22​π​β¯​e−β¯22f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{w\}\}\}\)=\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|^\{2\}\(\\overline\{\\beta\}^\{2\}\+1\)\}\{2\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|^\{2\}\}\{\\sqrt\{2\\pi\}\}\\overline\{\\beta\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}Computing we find that

\(7\.32\)∂f∂𝐰​\(𝐰,𝐰\)\\displaystyle\\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\}\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{w\}\}\}\)=\\displaystyle=\[\(β¯2\+1\)​erfc​\(β¯2\)−2π​β¯​e−β¯22\]​𝐰\\displaystyle\\left\[\(\\overline\{\\beta\}^\{2\}\+1\)\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\\sqrt\{\\frac\{2\}\{\\pi\}\}\\overline\{\\beta\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\\right\]\{\{\\mathbf\{w\}\}\}\(7\.33\)∂f∂β¯\\displaystyle\\frac\{\\partial f\}\{\\partial\\overline\{\\beta\}\}=\\displaystyle=‖𝐰‖2​\[β¯​erfc​\(β¯2\)−2π​e−β¯22\]\\displaystyle\\\|\{\{\\mathbf\{w\}\}\}\\\|^\{2\}\\left\[\\overline\{\\beta\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\\sqrt\{\\frac\{2\}\{\\pi\}\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\\right\]and so, setting\(𝐰,β¯\)=𝐰~∈M⁡\(d\+1,k\)\(\{\{\\mathbf\{w\}\}\},\\overline\{\\beta\}\)=\\widetilde\{\{\{\\mathbf\{w\}\}\}\}\\in M\(d\+1,k\)\.

grad​\(f\)​\(𝐰~\)=erfc​\(β¯2\)​\(\(β¯2\+1\)​𝐰,‖𝐰‖2​β¯\)−2π​e−β¯22​\(β¯​𝐰,1\)\{\\text\{\\rm grad\}\(\{f\}\)\}\(\\widetilde\{\{\{\\mathbf\{w\}\}\}\}\)=\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\big\(\(\\overline\{\\beta\}^\{2\}\+1\)\{\{\\mathbf\{w\}\}\},\\\|\{\{\\mathbf\{w\}\}\}\\\|^\{2\}\\overline\{\\beta\}\)\-\\sqrt\{\\frac\{2\}\{\\pi\}\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\\big\(\\overline\{\\beta\}\{\{\\mathbf\{w\}\}\},1\\big\)※

If𝐮∈ℝn\{\{\\mathbf\{u\}\}\}\\in\{\\mathbb\{R\}\}^\{n\}\(or\(ℝn\)⋆\(\{\\mathbb\{R\}\}^\{n\}\)^\{\\star\}\) is non\-zero, define𝐮¯=𝐮‖𝐮‖\\overline\{\{\{\\mathbf\{u\}\}\}\}=\\frac\{\{\{\\mathbf\{u\}\}\}\}\{\\\|\{\{\\mathbf\{u\}\}\}\\\|\}\.

#### Some frequently used derivatives

Suppose𝐰,𝐯∈ℝd\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\\in\{\\mathbb\{R\}\}^\{d\}are not parallel \(and therefore non\-zero\),cos⁡\(θ\)=⟨𝐰,𝐯⟩‖𝐰‖​‖𝐯‖\\cos\(\\theta\)=\\frac\{\\langle\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\\rangle\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\}and

\(7\.34\)𝐣=cot⁡\(θ\)​𝐰¯−csc⁡\(θ\)​𝐯¯\{\\mathbf\{j\}\}=\\cot\(\\theta\)\\overline\{\{\{\\mathbf\{w\}\}\}\}\-\\csc\(\\theta\)\\overline\{\{\{\\mathbf\{v\}\}\}\}Similarly define𝐉=𝐉⁡\(𝐰,𝐰′\)∈ℝd\{\\mathbf\{J\}\}=\{\\mathbf\{J\}\}\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{w\}\}\}^\{\\prime\}\)\\in\{\\mathbb\{R\}\}^\{d\}when𝐰,𝐰′\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{w\}\}\}^\{\\prime\}are distinct rows of𝐖\{\{\\mathbf\{W\}\}\}\. The following derivatives will be used repeatedly in this and subsequent sections\.

∂∂𝐰​‖𝐰‖\\displaystyle\\frac\{\\partial\}\{\\partial\{\{\\mathbf\{w\}\}\}\}\\\|\{\{\\mathbf\{w\}\}\}\\\|=\\displaystyle=𝐰¯,∂∂𝐰​1‖𝐰‖=−𝐰¯‖𝐰‖2,∂θ∂𝐰=𝐣‖𝐰‖\\displaystyle\\overline\{\{\{\\mathbf\{w\}\}\}\},\\quad\\frac\{\\partial\}\{\\partial\{\{\\mathbf\{w\}\}\}\}\\frac\{1\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}=\-\\frac\{\\overline\{\{\{\\mathbf\{w\}\}\}\}\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|^\{2\}\},\\quad\\frac\{\\partial\\theta\}\{\\partial\{\{\\mathbf\{w\}\}\}\}=\\frac\{\{\\mathbf\{j\}\}\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}∂cos⁡\(θ\)∂𝐰\\displaystyle\\frac\{\\partial\\,\\cos\(\\theta\)\}\{\\partial\{\{\\mathbf\{w\}\}\}\}=\\displaystyle=−sin⁡\(θ\)‖𝐰‖​𝐣,∂sin⁡\(θ\)∂𝐰=cos⁡\(θ\)‖𝐰‖​𝐣\\displaystyle\-\\frac\{\\sin\(\\theta\)\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\{\\mathbf\{j\}\},\\quad\\frac\{\\partial\\,\\sin\(\\theta\)\}\{\\partial\{\{\\mathbf\{w\}\}\}\}=\\frac\{\\cos\(\\theta\)\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\{\\mathbf\{j\}\}∂cot⁡\(θ\)∂𝐰\\displaystyle\\frac\{\\partial\\,\\cot\(\\theta\)\}\{\\partial\{\{\\mathbf\{w\}\}\}\}=\\displaystyle=−csc2⁡\(θ\)‖𝐰‖​𝐣,∂csc⁡\(θ\)∂𝐰=−cot⁡\(θ\)​csc⁡\(θ\)‖𝐰‖​𝐣\\displaystyle\-\\frac\{\\csc^\{2\}\(\\theta\)\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\{\\mathbf\{j\}\},\\quad\\frac\{\\partial\\,\\csc\(\\theta\)\}\{\\partial\{\{\\mathbf\{w\}\}\}\}=\-\\frac\{\\cot\(\\theta\)\\csc\(\\theta\)\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\{\\mathbf\{j\}\}
###### Lemma 7\.3\.

\(Notation and assumptions as above\.\)

1. \(1\)𝐣⁡\(𝐰,𝐯\)⟂𝐰\{\\mathbf\{j\}\}\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)\\perp\{\{\\mathbf\{w\}\}\},‖𝐣‖=1\\\|\{\\mathbf\{j\}\}\\\|=1,
2. \(2\)𝐉⁡\(𝐰,𝐰′\)⟂𝐰\{\\mathbf\{J\}\}\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{w\}\}\}^\{\\prime\}\)\\perp\{\{\\mathbf\{w\}\}\},‖𝐉‖=1\\\|\{\\mathbf\{J\}\}\\\|=1\.
3. \(3\)\{𝐰¯,𝐣\}\\\{\\overline\{\{\{\\mathbf\{w\}\}\}\},\{\\mathbf\{j\}\}\\\}is an orthonormal basis for the space spanned by𝐰,𝐯\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\.

The proof is left as an exercise for the reader\.

### 7\.1\.Derivatives ofH⁡\(β¯,γ¯,λ\)H\(\\overline\{\\beta\},\\overline\{\\gamma\},\\lambda\)andH⁡\(β¯i,β¯j,Λi​j\)H\(\\overline\{\\beta\}\_\{i\},\\overline\{\\beta\}\_\{j\},\\Lambda\_\{ij\}\),i≠ji\\neq j

The variableγ¯\\overline\{\\gamma\}is not used in computing the gradient off⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)since the𝐯\{\{\\mathbf\{v\}\}\}\-variables are part of the target and fixed\. For the termf⁡\(𝐰i,𝐰j\)f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{j\}\),i≠ji\\neq j, we need to consider the derivatives ofH⁡\(β¯i,β¯j,Λi​j\)H\(\\overline\{\\beta\}\_\{i\},\\overline\{\\beta\}\_\{j\},\\Lambda\_\{ij\}\)with respect toβ¯i\\overline\{\\beta\}\_\{i\}andβ¯j\\overline\{\\beta\}\_\{j\}—including the case whenβ¯i=β¯j\\overline\{\\beta\}\_\{i\}=\\overline\{\\beta\}\_\{j\}which is important when there is symmetry\.

###### Proposition 7\.4\.

\(Notation and assumptions as above\.\)

1. \(1\)Derivatives ofH⁡\(β¯,γ¯,λ\)H\(\\overline\{\\beta\},\\overline\{\\gamma\},\\lambda\)\. \(7\.35\)∂H∂β¯\\displaystyle\\frac\{\\partial H\}\{\\partial\\overline\{\\beta\}\}=\\displaystyle=π2​e−β¯22​erfc​\(−cot⁡\(λ\)​β¯\+csc⁡\(λ\)​γ¯2\)\\displaystyle\\sqrt\{\\frac\{\\pi\}\{2\}\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\-\\cot\(\\lambda\)\\overline\{\\beta\}\+\\csc\(\\lambda\)\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\(7\.36\)∂H∂𝐰\\displaystyle\\frac\{\\partial H\}\{\\partial\{\{\\mathbf\{w\}\}\}\}=\\displaystyle=e−k⁡\(β¯,γ¯,λ\)‖𝐰‖​𝐣\\displaystyle\\frac\{e^\{\-k\(\\overline\{\\beta\},\\overline\{\\gamma\},\\lambda\)\}\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\{\\mathbf\{j\}\}
2. \(2\)Derivatives ofH⁡\(β¯i,β¯j,Λi​j\)H\(\\overline\{\\beta\}\_\{i\},\\overline\{\\beta\}\_\{j\},\\Lambda\_\{ij\}\),i≠ji\\neq j\. \(7\.37\)∂H∂β¯i\\displaystyle\\frac\{\\partial H\}\{\\partial\\overline\{\\beta\}\_\{i\}\}=\\displaystyle=π2​e−β¯i22​erfc​\(−cot⁡\(Λi​j\)​β¯i\+csc⁡\(Λi​j\)​β¯j2\)\\displaystyle\\sqrt\{\\frac\{\\pi\}\{2\}\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\-\\cot\(\\Lambda\_\{ij\}\)\\overline\{\\beta\}\_\{i\}\+\\csc\(\\Lambda\_\{ij\}\)\\overline\{\\beta\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\(7\.38\)∂H∂β¯j\\displaystyle\\frac\{\\partial H\}\{\\partial\\overline\{\\beta\}\_\{j\}\}=\\displaystyle=−π2​e−β¯j22​erf​\(−cot⁡\(Λi​j\)​β¯j\+csc⁡\(Λi​j\)​β¯i2\)\\displaystyle\-\\sqrt\{\\frac\{\\pi\}\{2\}\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{j\}^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-\\cot\(\\Lambda\_\{ij\}\)\\overline\{\\beta\}\_\{j\}\+\\csc\(\\Lambda\_\{ij\}\)\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\(7\.39\)∂H∂𝐰i\\displaystyle\\frac\{\\partial H\}\{\\partial\{\{\\mathbf\{w\}\}\}^\{i\}\}=\\displaystyle=e−K⁡\(β¯i,β¯j,Λi​j\)‖𝐰i‖​𝐉i​j\\displaystyle\\frac\{e^\{\-K\(\\overline\{\\beta\}\_\{i\},\\overline\{\\beta\}\_\{j\},\\Lambda\_\{ij\}\)\}\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\{\\mathbf\{J\}\}\_\{ij\}
3. \(3\)Derivative ofH⁡\(β¯,β¯,Θ\)H\(\\overline\{\\beta\},\\overline\{\\beta\},\\Theta\)\. \(7\.40\)∂H∂β¯\\displaystyle\\frac\{\\partial H\}\{\\partial\\overline\{\\beta\}\}=\\displaystyle=π2​e−β¯22​\[2​erfc​\(β¯​\(−cot⁡\(Θ\)\+csc⁡\(Θ\)\)2\)−1\]\\displaystyle\\sqrt\{\\frac\{\\pi\}\{2\}\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\\left\[2\\,\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\\big\(\-\\cot\(\\Theta\)\+\\csc\(\\Theta\)\\big\)\}\{\\sqrt\{2\}\}\}\\right\)\}\-1\\right\]

###### Proof\.

\(1\) We have

∂H∂β¯​\(β¯,γ¯,λ\)=∂∂β¯π2∫∞γ¯e−t2/2erf\(−t​cot⁡\(λ\)\+csc⁡\(λ\)​β¯2\)dt=π2∫∞γ¯e−t2/22​csc⁡\(λ\)2​πe−\(tcot\(λ\)−β¯csc\(λ\)\)2/2dt=e−β¯22∫∞γ¯csc\(λ\)e−\(tcsc\(λ\)−β¯cot\(λ\)\)2/2dt=π2​e−β¯22​erfc​\(−cot⁡\(λ\)​β¯\+csc⁡\(θ\)​γ¯2\)\\begin\{aligned\} \\frac\{\\partial H\}\{\\partial\\overline\{\\beta\}\}\(\\overline\{\\beta\},\\overline\{\\gamma\},\\lambda\)&=\\frac\{\\partial\}\{\\partial\\overline\{\\beta\}\}\\sqrt\{\\frac\{\\pi\}\{2\}\}\\int^\{\\infty\}\_\{\\overline\{\\gamma\}\}e^\{\-t^\{2\}/2\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-t\\cot\(\\lambda\)\+\\csc\(\\lambda\)\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}dt\\\\ &=\\sqrt\{\\frac\{\\pi\}\{2\}\}\\int^\{\\infty\}\_\{\\overline\{\\gamma\}\}e^\{\-t^\{2\}/2\}\\frac\{2\\csc\(\\lambda\)\}\{\\sqrt\{2\\pi\}\}e^\{\-\(t\\cot\(\\lambda\)\-\\overline\{\\beta\}\\csc\(\\lambda\)\)^\{2\}/2\}dt\\\\ &=e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\\int^\{\\infty\}\_\{\\overline\{\\gamma\}\}\\csc\(\\lambda\)e^\{\-\(t\\csc\(\\lambda\)\-\\overline\{\\beta\}\\cot\(\\lambda\)\)^\{2\}/2\}dt\\\\ &=\\sqrt\{\\frac\{\\pi\}\{2\}\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\-\\cot\(\\lambda\)\\overline\{\\beta\}\+\\csc\(\\theta\)\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\end\{aligned\}where the last line follows from \([A\.63](https://arxiv.org/html/2609.30661#A1.E63)\)\.

In order to prove \([7\.36](https://arxiv.org/html/2609.30661#S7.E36)\), differentiateHHwith respect to𝐰\{\{\\mathbf\{w\}\}\}to obtain

∂H∂𝐰=∫γ¯∞e−t22​\[t​L−M\]​e−\(−t​cot⁡\(λ\)\+csc⁡\(λ\)​β¯\)22​𝑑t=IL−IM,\\frac\{\\partial H\}\{\\partial\{\{\\mathbf\{w\}\}\}\}=\\int\_\{\\overline\{\\gamma\}\}^\{\\infty\}e^\{\-\\frac\{t^\{2\}\}\{2\}\}\\big\[tL\-M\\big\]e^\{\-\\frac\{\(\-t\\cot\(\\lambda\)\+\\csc\(\\lambda\)\\overline\{\\beta\}\)^\{2\}\}\{2\}\}dt=I\_\{L\}\-I\_\{M\},where

L=csc2⁡\(λ\)​∂λ∂𝐰=csc2⁡\(λ\)‖𝐰‖​𝐣​and​M=β¯​csc⁡\(λ\)​cot⁡\(λ\)​∂λ∂𝐰=β¯​csc⁡\(λ\)​cot⁡\(λ\)‖𝐰‖​𝐣L=\\csc^\{2\}\(\\lambda\)\\frac\{\\partial\\lambda\}\{\\partial\{\{\\mathbf\{w\}\}\}\}=\\frac\{\\csc^\{2\}\(\\lambda\)\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\{\\mathbf\{j\}\}\\;\\text\{and \}M=\\overline\{\\beta\}\\csc\(\\lambda\)\\cot\(\\lambda\)\\frac\{\\partial\\lambda\}\{\\partial\{\{\\mathbf\{w\}\}\}\}=\\overline\{\\beta\}\\frac\{\\csc\(\\lambda\)\\cot\(\\lambda\)\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\{\\mathbf\{j\}\}SettingA=−cot⁡\(λ\)A=\-\\cot\(\\lambda\),B=csc⁡\(λ\)B=\\csc\(\\lambda\), evaluate the integralsIL,IMI\_\{L\},I\_\{M\}using Example[A\.3](https://arxiv.org/html/2609.30661#A1.Thmlemma3)and \([A\.63](https://arxiv.org/html/2609.30661#A1.E63)\):

IL\\displaystyle I\_\{L\}=\\displaystyle=e−β¯22‖𝐰‖​e−\(A​β¯\+B​γ¯\)22​𝐣−π2​β¯​e−β¯22‖𝐰‖​erfc​\(A​β¯\+B​γ¯2\)​cot⁡\(λ\)​𝐣\\displaystyle\\frac\{e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}e^\{\-\\frac\{\(A\\overline\{\\beta\}\+B\\overline\{\\gamma\}\)^\{2\}\}\{2\}\}\{\\mathbf\{j\}\}\-\\sqrt\{\\frac\{\\pi\}\{2\}\}\\overline\{\\beta\}\\frac\{e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{A\\overline\{\\beta\}\+B\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\cot\(\\lambda\)\\,\{\\mathbf\{j\}\}−IM\\displaystyle\-I\_\{M\}=\\displaystyle=π2​β¯​e−β¯22‖𝐰‖​erfc​\(A​β¯\+B​γ¯2\)​cot⁡\(λ\)​𝐣\\displaystyle\\sqrt\{\\frac\{\\pi\}\{2\}\}\\overline\{\\beta\}\\frac\{e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{A\\overline\{\\beta\}\+B\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\cot\(\\lambda\)\\,\{\\mathbf\{j\}\}and so∂H∂𝐰=e−k⁡\(β¯,γ¯,λ\)‖𝐰‖​𝐣\.\\frac\{\\partial H\}\{\\partial\{\{\\mathbf\{w\}\}\}\}=\\frac\{e^\{\-k\(\\overline\{\\beta\},\\overline\{\\gamma\},\\lambda\)\}\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\{\\mathbf\{j\}\}\.

\(2\) The proofs of \([7\.37](https://arxiv.org/html/2609.30661#S7.E37),[7\.38](https://arxiv.org/html/2609.30661#S7.E38),[7\.39](https://arxiv.org/html/2609.30661#S7.E39)\) are similar to those of \(1\) and omitted \(note that \([7\.38](https://arxiv.org/html/2609.30661#S7.E38)\) follows easily from \([7\.37](https://arxiv.org/html/2609.30661#S7.E37)\) using \([5\.22](https://arxiv.org/html/2609.30661#S5.E22)\)\)\.

\(3\) Simply deduced from \([7\.37](https://arxiv.org/html/2609.30661#S7.E37),[7\.38](https://arxiv.org/html/2609.30661#S7.E38)\)\. Alternatively, the result follows from Proposition[5\.3](https://arxiv.org/html/2609.30661#S5.Thmlemma3)\(3\)\. ∎

SinceH⁡\(β¯,γ¯,λ\)=λ~−π2​erfc​\(β¯2\)H\(\\overline\{\\beta\},\\overline\{\\gamma\},\\lambda\)=\\widetilde\{\\lambda\}\-\\frac\{\\pi\}\{2\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\(see \([5\.26](https://arxiv.org/html/2609.30661#S5.E26)\)\), the next result follows trivially from Proposition[7\.4](https://arxiv.org/html/2609.30661#S7.Thmlemma4)\.

###### Proposition 7\.5\.

\(Notation and assumptions as above\.\)

1. \(1\)β¯\\overline\{\\beta\}and𝐰\{\{\\mathbf\{w\}\}\}derivatives ofλ~​\(β¯,γ¯,λ\)\{\\widetilde\{\\lambda\}\}\(\\overline\{\\beta\},\\overline\{\\gamma\},\\lambda\) ∂λ~∂β¯=−π2​e−β¯22​erf​\(−cot⁡\(λ\)​β¯\+csc⁡\(λ\)​γ¯2\),∂λ~∂𝐰=e−k⁡\(β¯,γ¯,λ\)‖𝐰‖​𝐣\\frac\{\\partial\{\\widetilde\{\\lambda\}\}\}\{\\partial\\overline\{\\beta\}\}=\-\\sqrt\{\\frac\{\\pi\}\{2\}\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-\\cot\(\\lambda\)\\overline\{\\beta\}\+\\csc\(\\lambda\)\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\},\\quad\\frac\{\\partial\{\\widetilde\{\\lambda\}\}\}\{\\partial\{\{\\mathbf\{w\}\}\}\}=\\frac\{e^\{\-k\(\\overline\{\\beta\},\\overline\{\\gamma\},\\lambda\)\}\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\{\\mathbf\{j\}\}
2. \(2\)β¯i,β¯j\\overline\{\\beta\}\_\{i\},\\overline\{\\beta\}\_\{j\}and𝐰\{\{\\mathbf\{w\}\}\}derivatives ofΛ~i​j​\(β¯i,β¯j,Λi​j\)\{\\widetilde\{\\Lambda\}\}\_\{ij\}\(\\overline\{\\beta\}\_\{i\},\\overline\{\\beta\}\_\{j\},\\Lambda\_\{ij\}\) ∂Λ~i​j∂β¯i\\displaystyle\\frac\{\\partial\{\\widetilde\{\\Lambda\}\}\_\{ij\}\}\{\\partial\\overline\{\\beta\}\_\{i\}\}=\\displaystyle=−π2​e−β¯i22​erf​\(−cot⁡\(Λi​j\)​β¯i\+csc⁡\(Λi​j\)​β¯j2\)\\displaystyle\-\\sqrt\{\\frac\{\\pi\}\{2\}\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-\\cot\(\\Lambda\_\{ij\}\)\\overline\{\\beta\}\_\{i\}\+\\csc\(\\Lambda\_\{ij\}\)\\overline\{\\beta\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}∂Λ~i​j∂β¯j\\displaystyle\\frac\{\\partial\{\\widetilde\{\\Lambda\}\}\_\{ij\}\}\{\\partial\\overline\{\\beta\}\_\{j\}\}=\\displaystyle=−π2​e−β¯j22​erf​\(−cot⁡\(Λi​j\)​β¯j\+csc⁡\(Λi​j\)​β¯i2\)\\displaystyle\-\\sqrt\{\\frac\{\\pi\}\{2\}\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{j\}^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-\\cot\(\\Lambda\_\{ij\}\)\\overline\{\\beta\}\_\{j\}\+\\csc\(\\Lambda\_\{ij\}\)\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}∂Λ~i​j∂𝐰\\displaystyle\\frac\{\\partial\{\\widetilde\{\\Lambda\}\}\_\{ij\}\}\{\\partial\{\{\\mathbf\{w\}\}\}\}=\\displaystyle=e−K⁡\(β¯i,β¯j,Λi​j\)‖𝐰‖​𝐉i​j\\displaystyle\\frac\{e^\{\-K\(\\overline\{\\beta\}\_\{i\},\\overline\{\\beta\}\_\{j\},\\Lambda\_\{ij\}\)\}\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\{\\mathbf\{J\}\}\_\{ij\}

###### Remarks 7\.6\.

\(1\) The expression for∂H∂𝐰\\frac\{\\partial H\}\{\\partial\{\{\\mathbf\{w\}\}\}\}is singular ifθ∈\{0,π\}\\theta\\in\\\{0,\\pi\\\}even for zero bias: a similar term appears if we follow the same analysis in section[3\.1](https://arxiv.org/html/2609.30661#S3.SS1)where the kernel for zero bias is derived without using polar coordinates\. That singular term is cancelled out by another𝐰\{\{\\mathbf\{w\}\}\}\-derivative in the expression for∂f∂𝐰\\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\}; if bias is non\-zero it will not be\. \(2\) The additional term−π2​β¯​e−β¯22‖𝐰‖​erfc​\(A​β¯\+B​γ¯2\)​𝐰¯\-\\sqrt\{\\frac\{\\pi\}\{2\}\}\\,\\overline\{\\beta\}\\frac\{e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{A\\overline\{\\beta\}\+B\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\overline\{\{\{\\mathbf\{w\}\}\}\}appears in the formula for∂H∂𝐰\\frac\{\\partial H\}\{\\partial\{\{\\mathbf\{w\}\}\}\}ifβ\\betais the bias parameter\. Of course,∂H∂β=1‖𝐰‖​∂H∂β¯\\frac\{\\partial H\}\{\\partial\\beta\}=\\frac\{1\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\\frac\{\\partial H\}\{\\partial\\overline\{\\beta\}\}\.✠\\maltese

## 8\.Computation of∂f∂β¯\\frac\{\\partial f\}\{\\partial\\overline\{\\beta\}\}

###### Lemma 8\.1\.

\(Notations and assumptions of previous section\)\. \(1\) Assuming𝐰i,𝐯j\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{j\}are not parallel,

∂f∂β¯i​\(𝐰i,𝐯j\)=‖𝐰i‖​‖𝐯j‖​γ¯j2​π​\(π2​\[erfc​\(γ¯j2\)\+erfc​\(β¯i2\)\]−λ~i​j\)−‖𝐰i‖​‖𝐯j‖2​2​π​\[e−γ¯j22​erfc​\(A​γ¯j\+B​β¯i2\)\+cos⁡\(λi​j\)​e−β¯i22​erfc​\(A​β¯i\+B​γ¯j2\)\]\\begin\{aligned\} \\hskip 14\.45377pt\\frac\{\\partial f\}\{\\partial\\overline\{\\beta\}\_\{i\}\}\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{j\}\)&=\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|\\overline\{\\gamma\}\_\{j\}\}\{2\\pi\}\\left\(\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\{\\widetilde\{\\lambda\}\}\_\{ij\}\\right\)\\\\ &\\hskip\-43\.36243pt\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\left\[e^\{\-\\frac\{\\overline\{\\gamma\}\_\{j\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{A\\overline\{\\gamma\}\_\{j\}\+B\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\cos\(\\lambda\_\{ij\}\)e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{A\\overline\{\\beta\}\_\{i\}\+B\\overline\{\\gamma\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\end\{aligned\}\(2\) Ifγ¯j=0\\overline\{\\gamma\}\_\{j\}=0, then

∂f∂β¯i​\(𝐰i,𝐯j\)=−‖𝐰i‖​‖𝐯j‖2​2​π​\[erfc​\(csc⁡\(λi​j\)​β¯i2\)\+cos⁡\(λi​j\)​e−β¯i22​erfc​\(−cot⁡\(λi​j\)​β¯i2\)\]\\begin\{aligned\} \\hskip\-14\.45377pt\\frac\{\\partial f\}\{\\partial\\overline\{\\beta\}\_\{i\}\}\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{j\}\)&=\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\csc\(\\lambda\_\{ij\}\)\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\cos\(\\lambda\_\{ij\}\)e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\-\\cot\(\\lambda\_\{ij\}\)\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\end\{aligned\}

###### Proof\.

Omitting sub\- and superscripts, a straightforward computation gives

∂f∂β¯​\(𝐰,𝐯\)=‖𝐰‖​‖𝐯‖​γ¯2​π​\(π2​\[erfc​\(γ¯2\)\+erfc​\(β¯2\)\]−λ~\)−‖𝐰‖​‖𝐯‖​\(β¯​γ¯\+cos⁡\(λ\)\)2​2​π​e−β¯22​erfc​\(A​β¯\+B​γ¯2\)−‖𝐰‖​‖𝐯‖2​2​π​\[−β¯​γ¯​e−β¯22​erfc​\(A​β¯\+B​γ¯2\)\+e−γ¯22​erfc​\(A​γ¯\+B​β¯2\)\]\+‖𝐰‖​‖𝐯‖2​π​\[A​γ¯​e−β¯22​e−\(A​β¯\+B​γ¯\)22\+B​β¯​e−γ¯22​e−\(A​γ¯\+B​β¯\)22\]−‖𝐰‖​‖𝐯‖2​π​\(β¯​csc⁡\(λ\)−γ¯​cot⁡\(λ\)\)​e−k⁡\(β¯,γ¯,λ\)\\begin\{aligned\} \\frac\{\\partial f\}\{\\partial\\overline\{\\beta\}\}\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)&=\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\\overline\{\\gamma\}\}\{2\\pi\}\\left\(\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\\widetilde\{\\lambda\}\\right\)\\\\ &\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\(\\overline\{\\beta\}\\overline\{\\gamma\}\+\\cos\(\\lambda\)\)\}\{2\\sqrt\{2\\pi\}\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{A\\overline\{\\beta\}\+B\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\\\ &\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\left\[\-\\overline\{\\beta\}\\overline\{\\gamma\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{A\\overline\{\\beta\}\+B\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+e^\{\-\\frac\{\\overline\{\\gamma\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{A\\overline\{\\gamma\}\+B\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\pi\}\\left\[A\\overline\{\\gamma\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}e^\{\-\\frac\{\(A\\overline\{\\beta\}\+B\\overline\{\\gamma\}\)^\{2\}\}\{2\}\}\+B\\overline\{\\beta\}e^\{\-\\frac\{\\overline\{\\gamma\}^\{2\}\}\{2\}\}e^\{\-\\frac\{\(A\\overline\{\\gamma\}\+B\\overline\{\\beta\}\)^\{2\}\}\{2\}\}\\right\]\\\\ &\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\pi\}\\big\(\\overline\{\\beta\}\\csc\(\\lambda\)\-\\overline\{\\gamma\}\\cot\(\\lambda\)\\big\)e^\{\-k\(\\overline\{\\beta\},\\overline\{\\gamma\},\\lambda\)\}\\end\{aligned\}After some cancellation—the last two lines and terms involvingβ¯​γ¯\\overline\{\\beta\}\\overline\{\\gamma\}—the result follows\. ∎

###### Proposition 8\.2\.

\(1\) Assumingi≠ji\\neq jand𝐰i,𝐰j\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{j\}are not parallel,

∂f∂β¯i​\(𝐰i,𝐰j\)=‖𝐰i‖​‖𝐰j‖​β¯j2​π​\(π2​\[erfc​\(β¯i2\)\+erfc​\(β¯j2\)\]−Λ~i​j\)−‖𝐰i‖​‖𝐰j‖2​2​π​\[e−β¯j22​erfc​\(A​β¯j\+B​β¯i2\)\+cos⁡\(Λi​j\)​e−β¯i22​erfc​\(A​β¯i\+B​β¯j2\)\]\\begin\{aligned\} \\hskip 14\.45377pt\\frac\{\\partial f\}\{\\partial\\overline\{\\beta\}\_\{i\}\}\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{j\}\)&=\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\\overline\{\\beta\}\_\{j\}\}\{2\\pi\}\\left\(\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\{\\widetilde\{\\Lambda\}\}\_\{ij\}\\right\)\\\\ &\\hskip\-43\.36243pt\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\left\[e^\{\-\\frac\{\\overline\{\\beta\}\_\{j\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{A\\overline\{\\beta\}\_\{j\}\+B\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\cos\(\\Lambda\_\{ij\}\)e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{A\\overline\{\\beta\}\_\{i\}\+B\\overline\{\\beta\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\end\{aligned\}\(2\) Ifβ¯i=β¯j=β¯\\overline\{\\beta\}\_\{i\}=\\overline\{\\beta\}\_\{j\}=\\overline\{\\beta\}, then

∂f∂β¯​\(𝐰i,𝐰j\)=‖𝐰i‖​‖𝐰j‖​β¯π​\(π​erfc​\(β¯2\)−Λ~i​j\)−‖𝐰i‖​‖𝐰j‖2​π​\[e−β¯22​erfc​\(β¯​\(−cot⁡\(Λi​j\)\+csc⁡\(Λi​j\)CLOSE2\)​\(1\+cos⁡\(Λi​j\)\)\]\\begin\{aligned\} \\hskip 14\.45377pt\\frac\{\\partial f\}\{\\partial\\overline\{\\beta\}\}\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{j\}\)&=\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\\overline\{\\beta\}\}\{\\pi\}\\left\(\\pi\\,\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\{\\widetilde\{\\Lambda\}\}\_\{ij\}\\right\)\\\\ &\\hskip\-43\.36243pt\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\}\{\\sqrt\{2\\pi\}\}\\left\[e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\(\-\\cot\(\\Lambda\_\{ij\}\)\+\\csc\(\\Lambda\_\{ij\}\)\}\{\\sqrt\{2\}\}\}\\right\)\}\(1\+\\cos\(\\Lambda\_\{ij\}\)\)\\right\]\\end\{aligned\}

###### Proof\.

The proof of \(1\) is formally identical to that of Lemma[8\.1](https://arxiv.org/html/2609.30661#S8.Thmlemma1)\(1\)\. The proof of \(2\) amounts to adding the derivatives offfwith respect toβ¯i\\overline\{\\beta\}\_\{i\}andβ¯j\\overline\{\\beta\}\_\{j\}and then settingβ¯,β¯j=β¯\\overline\{\\beta\},\\overline\{\\beta\}\_\{j\}=\\overline\{\\beta\}\. Of special interest is the term∂Λ~i​j∂β¯\\frac\{\\partial\{\\widetilde\{\\Lambda\}\}\_\{ij\}\}\{\\partial\\overline\{\\beta\}\}\. It is shown in section[5](https://arxiv.org/html/2609.30661#S5)that

Λ~i​j=S⁡\(β¯,β¯,Λi​j\)\+π2=H⁡\(β¯,β¯,Λi​j\)\+π2​erfc​\(β¯2\)\{\\widetilde\{\\Lambda\}\}\_\{ij\}=S\(\\overline\{\\beta\},\\overline\{\\beta\},\\Lambda\_\{ij\}\)\+\\frac\{\\pi\}\{2\}=H\(\\overline\{\\beta\},\\overline\{\\beta\},\\Lambda\_\{ij\}\)\+\\frac\{\\pi\}\{2\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}and so

∂Λ~i​j∂β¯=−2​π2​e−β¯22​erf​\(β¯​\(−cot⁡\(Λi​j\)\+csc⁡\(Λi​j\)\)2\)\\frac\{\\partial\{\\widetilde\{\\Lambda\}\}\_\{ij\}\}\{\\partial\\overline\{\\beta\}\}=\-2\\sqrt\{\\frac\{\\pi\}\{2\}\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\(\-\\cot\(\\Lambda\_\{ij\}\)\+\\csc\(\\Lambda\_\{ij\}\)\)\}\{\\sqrt\{2\}\}\}\\right\)\}by Proposition[5\.3](https://arxiv.org/html/2609.30661#S5.Thmlemma3), \([5\.23](https://arxiv.org/html/2609.30661#S5.E23)\)\. Adding the derivative ofπ​erfc​\(β¯2\)\\pi\\,\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}to−∂Λ~i​j∂β¯\-\\frac\{\\partial\{\\widetilde\{\\Lambda\}\}\_\{ij\}\}\{\\partial\\overline\{\\beta\}\}gives the coefficient ofcos⁡\(Λi​j\)\\cos\(\\Lambda\_\{ij\}\)in the displayed formula\. ∎

###### Examples 8\.3\.

\(1\) Ifβ¯=γ¯=0\\overline\{\\beta\}=\\overline\{\\gamma\}=0, then

∂f∂β¯​\(𝐰,𝐯\)=−‖𝐰‖​‖𝐯‖2​2​π​\(1\+cos⁡\(λ\)\)\\frac\{\\partial f\}\{\\partial\\overline\{\\beta\}\}\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)=\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\sqrt\{2\\pi\}\}\(1\+\\cos\(\\lambda\)\)If𝐯=𝐰\{\{\\mathbf\{v\}\}\}=\{\{\\mathbf\{w\}\}\}andγ¯=β¯\\overline\{\\gamma\}=\\overline\{\\beta\}, it follows from Example[3\.4](https://arxiv.org/html/2609.30661#S3.Thmlemma4)that ifβ¯=0\\overline\{\\beta\}=0then

∂f∂β¯​\(𝐰,𝐰\)=−2​‖𝐰‖22​π\\frac\{\\partial f\}\{\\partial\\overline\{\\beta\}\}\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{w\}\}\}\)=\-\\frac\{2\\\|\{\{\\mathbf\{w\}\}\}\\\|^\{2\}\}\{\\sqrt\{2\\pi\}\}\(2\) Suppose𝐰,𝐯≠0\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\\neq 0are parallel\. Using Lemma[3\.13](https://arxiv.org/html/2609.30661#S3.Thmlemma13), we have

1. \(1\)Ifλ=0\\lambda=0: ∂f∂β¯=\{‖𝐰‖​‖𝐯‖​γ¯2​erfc​\(β¯2\)−‖𝐰‖​‖𝐯‖2​π​e−β¯22,if​β¯≥γ¯‖𝐰‖​‖𝐯‖​γ¯2​erfc​\(γ¯2\)−‖𝐰‖​‖𝐯‖2​π​e−γ¯22,if​β¯≤γ¯\\frac\{\\partial f\}\{\\partial\\overline\{\\beta\}\}=\\begin\{cases\}&\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\\overline\{\\gamma\}\}\{2\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{\\sqrt\{2\\pi\}\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\},\\;\\text\{if \}\\overline\{\\beta\}\\geq\\overline\{\\gamma\}\\\\ &\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\\overline\{\\gamma\}\}\{2\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{\\sqrt\{2\\pi\}\}e^\{\-\\frac\{\\overline\{\\gamma\}^\{2\}\}\{2\}\},\\;\\text\{if \}\\overline\{\\beta\}\\leq\\overline\{\\gamma\}\\end\{cases\}
2. \(2\)Ifλ=π\\lambda=\\pi: ∂f∂β¯=\{‖𝐰‖​‖𝐯‖​γ¯2​\[erf​\(β¯2\)−erf​\(γ¯2\)\]\+‖𝐰‖​‖𝐯‖2​π​e−β¯22,if​β¯\+γ¯≥0−‖𝐰‖​‖𝐯‖​γ¯​erf​\(γ¯2\)\+‖𝐰‖​‖𝐯‖2​π​e−γ¯22,if​β¯\+γ¯≤0\\frac\{\\partial f\}\{\\partial\\overline\{\\beta\}\}=\\begin\{cases\}&\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\\overline\{\\gamma\}\}\{2\}\\left\[\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{\\sqrt\{2\\pi\}\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\},\\;\\text\{if \}\\overline\{\\beta\}\+\\overline\{\\gamma\}\\geq 0\\\\ &\-\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\\overline\{\\gamma\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{\\sqrt\{2\\pi\}\}e^\{\-\\frac\{\\overline\{\\gamma\}^\{2\}\}\{2\}\},\\;\\text\{if \}\\overline\{\\beta\}\+\\overline\{\\gamma\}\\leq 0\\end\{cases\}

※

###### Remark 8\.4\.

It follows from the last example thatffis aC1C^\{1\}function ofβ¯\\overline\{\\beta\}\(orβ\\beta\) ifλ∈\{0,π\}\\lambda\\in\\\{0,\\pi\\\}andffis analytic inβ¯\\overline\{\\beta\}\(orβ\\beta\) ifλ=0\\lambda=0\(resp\.λ=π\\lambda=\\pi\) provided thatβ¯≠γ¯\\overline\{\\beta\}\\neq\\overline\{\\gamma\}\(resp\.β¯≠−γ¯\\overline\{\\beta\}\\neq\-\\overline\{\\gamma\}\)\. Of course, real analyticity offfinβ¯\\overline\{\\beta\}holds without restriction ifλ∈\(0,π\)\\lambda\\in\(0,\\pi\)and𝐰,𝐯≠0\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\\neq 0\.✠\\maltese

## 9\.Computation of∂f∂𝐰\\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\}\(normalized biasβ¯\\overline\{\\beta\}\)

We supplement the list of derivatives given in Section[7](https://arxiv.org/html/2609.30661#S7)before Lemma[7\.3](https://arxiv.org/html/2609.30661#S7.Thmlemma3)with

∂∂𝐰​erf​\(A​γ¯\+B​β¯2\)=2π​e−\(Aγ¯\+Bβ¯\)2/2‖𝐰‖​\[csc⁡\(λ\)​\(A​β¯\+B​γ¯\)\]​𝐣∂∂𝐰​erf​\(A​𝜷¯\+B​γ¯2\)=2π​e−\(Aβ¯\+Bγ¯\)2/2‖𝐰‖​\[csc⁡\(λ\)​\(A​γ¯\+B​β¯\)\]​𝐣∂k∂𝐰​\(β¯,γ¯,λ\)=csc⁡\(λ\)‖𝐰‖​\[\(A​β¯\+B​γ¯\)​\(A​γ¯\+B​β¯\)\]​𝐣∂∂𝐰​e−k=−∂k∂𝐰​e−k,∂H∂𝐰=e−k⁡\(β¯,γ¯,λ\)‖𝐰‖​𝐣\\begin\{aligned\} \\frac\{\\partial\}\{\\partial\{\{\\mathbf\{w\}\}\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{A\\overline\{\\gamma\}\+B\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}=&\\sqrt\{\\frac\{2\}\{\\pi\}\}\\frac\{e^\{\-\(A\\overline\{\\gamma\}\+B\\overline\{\\beta\}\)^\{2\}/2\}\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\\big\[\\csc\(\\lambda\)\\left\(A\\overline\{\\beta\}\+B\\overline\{\\gamma\}\\right\)\\big\]\{\\mathbf\{j\}\}\\\\ \\frac\{\\partial\}\{\\partial\{\{\\mathbf\{w\}\}\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{A\{\\boldsymbol\{\\overline\{\\beta\}\}\}\+B\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}=&\\sqrt\{\\frac\{2\}\{\\pi\}\}\\frac\{e^\{\-\(A\\overline\{\\beta\}\+B\\overline\{\\gamma\}\)^\{2\}/2\}\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\\big\[\\csc\(\\lambda\)\\left\(A\\overline\{\\gamma\}\+B\\overline\{\\beta\}\\right\)\\big\]\{\\mathbf\{j\}\}\\\\ \\frac\{\\partial k\}\{\\partial\{\{\\mathbf\{w\}\}\}\}\(\\overline\{\\beta\},\\overline\{\\gamma\},\\lambda\)=&\\frac\{\\csc\(\\lambda\)\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\\left\[\(A\\overline\{\\beta\}\+B\\overline\{\\gamma\}\)\(A\\overline\{\\gamma\}\+B\\overline\{\\beta\}\)\\right\]\{\\mathbf\{j\}\}\\\\ \\frac\{\\partial\}\{\\partial\{\{\\mathbf\{w\}\}\}\}e^\{\-k\}=&\-\\frac\{\\partial k\}\{\\partial\{\{\\mathbf\{w\}\}\}\}e^\{\-k\},\\quad\\frac\{\\partial H\}\{\\partial\{\{\\mathbf\{w\}\}\}\}=\\frac\{e^\{\-k\(\\overline\{\\beta\},\\overline\{\\gamma\},\\lambda\)\}\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\{\\mathbf\{j\}\}\\end\{aligned\}\(A=−cot⁡\(λ\),B=csc⁡\(λ\)A=\-\\cot\(\\lambda\),B=\\csc\(\\lambda\)andλ\\lambdais the angle between𝐰\{\{\\mathbf\{w\}\}\}and𝐯\{\{\\mathbf\{v\}\}\}\.\)

∂λ~i​j∂β¯i\\displaystyle\\frac\{\\partial\{\\widetilde\{\\lambda\}\}\_\{ij\}\}\{\\partial\\overline\{\\beta\}\_\{i\}\}=\\displaystyle=−π2​e−β¯i22​erf​\(−cot⁡\(λi​j\)​β¯i\+csc⁡\(λi​j\)​γ¯j2\)∂λ~i​j∂𝐰=e−k⁡\(β¯i,γ¯j,λi​j\)‖𝐰‖​𝐣\\displaystyle\-\\sqrt\{\\frac\{\\pi\}\{2\}\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-\\cot\(\\lambda\_\{ij\}\)\\overline\{\\beta\}\_\{i\}\+\\csc\(\\lambda\_\{ij\}\)\\overline\{\\gamma\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\quad\\frac\{\\partial\{\\widetilde\{\\lambda\}\}\_\{ij\}\}\{\\partial\{\{\\mathbf\{w\}\}\}\}=\\frac\{e^\{\-k\(\\overline\{\\beta\}\_\{i\},\\overline\{\\gamma\}\_\{j\},\\lambda\_\{ij\}\)\}\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\{\\mathbf\{j\}\}∂Λ~i​j∂β¯i\\displaystyle\\frac\{\\partial\{\\widetilde\{\\Lambda\}\}\_\{ij\}\}\{\\partial\\overline\{\\beta\}\_\{i\}\}=\\displaystyle=−π2​e−β¯i22​erf​\(−cot⁡\(Λi​j\)​β¯i\+csc⁡\(Λi​j\)​β¯j2\)∂Λ~i​j∂𝐰=e−K⁡\(β¯i,β¯j,Λi​j\)‖𝐰‖​𝐉i​j\\displaystyle\-\\sqrt\{\\frac\{\\pi\}\{2\}\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-\\cot\(\\Lambda\_\{ij\}\)\\overline\{\\beta\}\_\{i\}\+\\csc\(\\Lambda\_\{ij\}\)\\overline\{\\beta\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\quad\\frac\{\\partial\{\\widetilde\{\\Lambda\}\}\_\{ij\}\}\{\\partial\{\{\\mathbf\{w\}\}\}\}=\\frac\{e^\{\-K\(\\overline\{\\beta\}\_\{i\},\\overline\{\\beta\}\_\{j\},\\Lambda\_\{ij\}\)\}\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\{\\mathbf\{J\}\}\_\{ij\}Herej∈𝐝,i∈𝐤j\\in\{\\mathbf\{d\}\},i\\in\{\\mathbf\{k\}\}andλi​j\\lambda\_\{ij\}\(resp\.Λi​j\\Lambda\_\{ij\}\) is the angle between𝐰i\{\{\\mathbf\{w\}\}\}^\{i\}and𝐯j\{\{\\mathbf\{v\}\}\}^\{j\}\(resp\.𝐰j\{\{\\mathbf\{w\}\}\}^\{j\},j≠ij\\neq i\)\. Note that the partial derivatives ofΛ~i​j\{\\widetilde\{\\Lambda\}\}\_\{ij\}with respect toβ¯i,β¯j\\overline\{\\beta\}\_\{i\},\\overline\{\\beta\}\_\{j\}will not generally be equal unlessβ¯i=β¯j\\overline\{\\beta\}\_\{i\}=\\overline\{\\beta\}\_\{j\}\(see Section[8](https://arxiv.org/html/2609.30661#S8)for more details and note that the𝐰\{\{\\mathbf\{w\}\}\}\-derivatives ofΛ~i​j,Λ~j​i\{\\widetilde\{\\Lambda\}\}\_\{ij\},\{\\widetilde\{\\Lambda\}\}\_\{ji\}are equal\)\)\.

### 9\.1\.The formula for∂f∂𝐰​\(𝐰,𝐯\)\\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\}\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)\(normalized biasβ¯\\overline\{\\beta\}\)

\. A straightforward computation of the gradient gives

∂f∂𝐰=‖𝐯‖2​π​\(π2​\[erfc​\(γ¯2\)\+erfc​\(β¯2\)\]−λ~\)​\[\(β¯​γ¯\+cos⁡\(λ\)\)​𝐰¯−sin⁡\(λ\)​𝐣\]−‖𝐯‖2​π​e−k​\(β¯​γ¯\+cos⁡\(λ\)\)​𝐣−‖𝐯‖2​2​π​\[β¯​e−γ¯22​erfc​\(A​γ¯\+B​β¯2\)\+γ¯​e−β¯22​erfc​\(A​β¯\+B​γ¯2\)\]​𝐰¯\+‖𝐯‖​e−k2​π​\[γ¯​csc⁡\(λ\)​\(A​γ¯\+B​β¯\)\+β¯​csc⁡\(λ\)​\(A​β¯\+B​γ¯\)\]​𝐣\+‖𝐯‖2​π​e−k​\(sin⁡\(λ\)​𝐰¯\+cos⁡\(λ\)​𝐣\)−‖𝐯‖​e−k2​π​\[\(A​β¯\+B​γ¯\)​\(A​γ¯\+B​β¯\)\]​𝐣\\begin\{aligned\} \\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\}=&\\frac\{\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\pi\}\\left\(\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\\widetilde\{\\lambda\}\\right\)\\big\[\(\\overline\{\\beta\}\\overline\{\\gamma\}\+\\cos\(\\lambda\)\)\\overline\{\{\{\\mathbf\{w\}\}\}\}\-\\sin\(\\lambda\)\{\\mathbf\{j\}\}\\big\]\\\\ &\-\\frac\{\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\pi\}e^\{\-k\}\\big\(\\overline\{\\beta\}\\overline\{\\gamma\}\+\\cos\(\\lambda\)\\big\)\{\\mathbf\{j\}\}\\\\ &\-\\frac\{\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\left\[\\overline\{\\beta\}e^\{\-\\frac\{\\overline\{\\gamma\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{A\\overline\{\\gamma\}\+B\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\overline\{\\gamma\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{A\\overline\{\\beta\}\+B\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\overline\{\{\{\\mathbf\{w\}\}\}\}\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{v\}\}\}\\\|e^\{\-k\}\}\{2\\pi\}\\big\[\\overline\{\\gamma\}\\csc\(\\lambda\)\(A\\overline\{\\gamma\}\+B\\overline\{\\beta\}\)\+\\overline\{\\beta\}\\csc\(\\lambda\)\(A\\overline\{\\beta\}\+B\\overline\{\\gamma\}\)\\big\]\{\\mathbf\{j\}\}\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\pi\}e^\{\-k\}\\big\(\\sin\(\\lambda\)\\overline\{\{\{\\mathbf\{w\}\}\}\}\+\\cos\(\\lambda\)\{\\mathbf\{j\}\}\\big\)\\\\ &\-\\frac\{\\\|\{\{\\mathbf\{v\}\}\}\\\|e^\{\-k\}\}\{2\\pi\}\\big\[\(A\\overline\{\\beta\}\+B\\overline\{\\gamma\}\)\(A\\overline\{\\gamma\}\+B\\overline\{\\beta\}\)\\big\]\{\\mathbf\{j\}\}\\end\{aligned\}In the last line, thecsc⁡\(λ\)\\csc\(\\lambda\)term that comes from∂k∂𝐰\\frac\{\\partial k\}\{\\partial\{\{\\mathbf\{w\}\}\}\}is multiplied bysin⁡\(λ\)\\sin\(\\lambda\)and so the final expression is quadratic in bothβ¯,γ¯\\overline\{\\beta\},\\overline\{\\gamma\}andcsc⁡\(λ\)\\csc\(\\lambda\)andcot⁡\(λ\)\\cot\(\\lambda\), as is the corresponding expression on line 4\. After some rearrangement and two major cancellations—most notably lines 4 & 6 together with the first term in line 2—we obtain

###### Proposition 9\.1\.

\(Notation and assumptions as above\.\)

∂f∂𝐰=‖𝐯‖2​π​\(π2​\[erfc​\(γ¯2\)\+erfc​\(β¯2\)\]−λ~\)​\(β¯​γ¯​𝐰¯\+𝐯¯\)\+‖𝐯‖2​π​e−k​sin⁡\(λ\)​𝐰¯−‖𝐯‖2​2​π​\[β¯​e−γ¯22​erfc​\(A​γ¯\+B​β¯2\)\+γ¯​e−β¯22​erfc​\(A​β¯\+B​γ¯2\)\]​𝐰¯\\begin\{aligned\} \\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\}=&\\frac\{\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\pi\}\\left\(\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\\widetilde\{\\lambda\}\\right\)\(\\overline\{\\beta\}\\overline\{\\gamma\}\\overline\{\{\{\\mathbf\{w\}\}\}\}\+\\overline\{\{\{\\mathbf\{v\}\}\}\}\)\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\pi\}e^\{\-k\}\\sin\(\\lambda\)\\overline\{\{\{\\mathbf\{w\}\}\}\}\\\\ &\-\\frac\{\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\left\[\\overline\{\\beta\}e^\{\-\\frac\{\\overline\{\\gamma\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{A\\overline\{\\gamma\}\+B\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\overline\{\\gamma\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{A\\overline\{\\beta\}\+B\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\overline\{\{\{\\mathbf\{w\}\}\}\}\\end\{aligned\}

###### Remarks 9\.2\.

\(1\) If we takeβ¯=γ¯=0\\overline\{\\beta\}=\\overline\{\\gamma\}=0\(zero bias\), then the last three terms are zero and the first two terms give the formula for gradf⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)derived by Brutzkus and Globerson\[[8](https://arxiv.org/html/2609.30661#bib.bib8)\]:

∂f∂𝐰=‖𝐯‖2​π​\(sin⁡\(λ\)​𝐰¯\+\(π−λ\)​𝐯¯\)\\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\}=\\frac\{\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\pi\}\\big\(\\sin\(\\lambda\)\\overline\{\{\{\\mathbf\{w\}\}\}\}\+\(\\pi\-\\lambda\)\\overline\{\{\{\\mathbf\{v\}\}\}\}\\big\)The computation usesH⁡\(0,0,λ\)=λ−π2H\(0,0,\\lambda\)=\\lambda\-\\frac\{\\pi\}\{2\}\. Note that in Brutzkus and Globerson the second term is written12​π​\(π−λ\)​𝐯\\frac\{1\}\{2\\pi\}\(\\pi\-\\lambda\)\{\{\\mathbf\{v\}\}\}\. \(2\) Terms in𝐣\{\\mathbf\{j\}\}do not appear\. This is because of the cancellation of lines 4 & 6 together with the first term in line 2\. The only place where𝐣\{\\mathbf\{j\}\}is expanded ascot⁡\(λ\)​𝐰¯−csc⁡\(λ\)​𝐯¯\\cot\(\\lambda\)\\overline\{\{\{\\mathbf\{w\}\}\}\}\-\\csc\(\\lambda\)\\overline\{\{\{\\mathbf\{v\}\}\}\}is in line 1\. This results in the replacement of the factorOPEN\(β¯​γ¯\+cos⁡\(λ\)\)​𝐰¯−sin⁡\(λ\)​𝐣\)\(\\overline\{\\beta\}\\overline\{\\gamma\}\+\\cos\(\\lambda\)\)\\overline\{\{\{\\mathbf\{w\}\}\}\}\-\\sin\(\\lambda\)\{\\mathbf\{j\}\}\)by\(β¯​γ¯​𝐰¯\+𝐯¯\)\(\\overline\{\\beta\}\\overline\{\\gamma\}\\overline\{\{\{\\mathbf\{w\}\}\}\}\+\\overline\{\{\{\\mathbf\{v\}\}\}\}\)\. Thus the gradient is a linear combination of𝐯¯\\overline\{\{\{\\mathbf\{v\}\}\}\}and𝐰¯\\overline\{\{\{\\mathbf\{w\}\}\}\}\. That is not a surprise since𝐣\{\\mathbf\{j\}\}is a linear combination of𝐯¯\\overline\{\{\{\\mathbf\{v\}\}\}\}and𝐰¯\\overline\{\{\{\\mathbf\{w\}\}\}\}and also, of course, since the gradient∂f∂𝐰\\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\}is obviously a point in the space spanned by𝐰,𝐯\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\. What is surprising is the relative simplicity of the formula for∂f∂𝐰\\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\}together with the absence of products and powers ofcot⁡\(λ\)\\cot\(\\lambda\)andcsc⁡\(λ\)\\csc\(\\lambda\)except for the exponential terme−ke^\{\-k\}\.✠\\maltese

All examples in this paper assumeγ¯=0\\overline\{\\gamma\}=0and, for future reference, the next proposition gives the gradients off⁡\(𝐰i,𝐯j\)f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{j\}\)andf⁡\(𝐰i,𝐰j\),i≠jf\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{j\}\),i\\neq junder the assumption thatγ¯=0\\overline\{\\gamma\}=0\.

###### Proposition 9\.3\.

Assumeγ¯=0\\overline\{\\gamma\}=0and follow the notation conventions of section[6\.1](https://arxiv.org/html/2609.30661#S6.SS1)\.

1. \(1\)Fori∈𝐤,j∈𝐝i\\in\{\\mathbf\{k\}\},j\\in\{\\mathbf\{d\}\}: ∂f∂𝐰​\(𝐰i,𝐯j\)=‖𝐯j‖2​π​\[π−λ~i​j−π2​erf​\(β¯i2\)\]​𝐯¯j\+‖𝐯j‖2​π​\[e−ki​j​sin⁡\(λi​j\)−π2​β¯i​erfc​\(csc⁡\(λi​j\)​β¯i2\)\]​𝐰¯i\\begin\{aligned\} \\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\}\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{j\}\)=&\\frac\{\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|\}\{2\\pi\}\\left\[\\pi\-\{\\widetilde\{\\lambda\}\}\_\{ij\}\-\\frac\{\\pi\}\{2\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\overline\{\{\{\\mathbf\{v\}\}\}\}^\{j\}\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|\}\{2\\pi\}\\left\[e^\{\-k\_\{ij\}\}\\sin\(\\lambda\_\{ij\}\)\-\\sqrt\{\\frac\{\\pi\}\{2\}\}\\overline\{\\beta\}\_\{i\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\csc\(\\lambda\_\{ij\}\)\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\overline\{\{\{\\mathbf\{w\}\}\}\}^\{i\}\\end\{aligned\}whereki​j=12​β¯i2​csc2⁡\(λi​j\)k\_\{ij\}=\\frac\{1\}\{2\}\\overline\{\\beta\}\_\{i\}^\{2\}\\csc^\{2\}\(\\lambda\_\{ij\}\)\.
2. \(2\)Fori,j∈𝐤i,j\\in\{\\mathbf\{k\}\},i≠ji\\neq j: ∂f∂𝐰​\(𝐰i,𝐰j\)=‖𝐰j‖2​π​\[π−Λ~i​j−π2​erf​\(β¯i2\)−π2​erf​\(β¯j2\)\]​\(β¯i​β¯j​𝐰¯i\+𝐰¯j\)\+‖𝐰j‖2​πe−Ki​j​sin⁡\(Λi​j\)​𝐰¯i−‖𝐰j‖2​2​π\[β¯i​e−β¯j22​erfc​\(Zi​j2\)\+β¯j​e−β¯i22​erfc​\(Z^i​j2\)\]​𝐰¯i\\begin\{aligned\} \\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\}\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{j\}\)=&\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\}\{2\\pi\}\\left\[\\pi\-\{\\widetilde\{\\Lambda\}\}\_\{ij\}\-\\frac\{\\pi\}\{2\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\\frac\{\\pi\}\{2\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\(\\overline\{\\beta\}\_\{i\}\\overline\{\\beta\}\_\{j\}\\overline\{\{\{\\mathbf\{w\}\}\}\}^\{i\}\+\\overline\{\{\{\\mathbf\{w\}\}\}\}^\{j\}\)\\\\ \+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\}\{2\\pi\}&e^\{\-K\_\{ij\}\}\\sin\(\\Lambda\_\{ij\}\)\\overline\{\{\{\\mathbf\{w\}\}\}\}^\{i\}\\\\ \-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\}\{2\\sqrt\{2\\pi\}\}&\\left\[\\overline\{\\beta\}\_\{i\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{j\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{Z\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\overline\{\\beta\}\_\{j\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\{\\widehat\{Z\}\}\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\overline\{\{\{\\mathbf\{w\}\}\}\}^\{i\}\\end\{aligned\}whereZi​j=−cot⁡\(Λi​j\)​β¯j\+csc⁡\(Λi​j\)​β¯i=Z^j​iZ\_\{ij\}=\-\\cot\(\\Lambda\_\{ij\}\)\\overline\{\\beta\}\_\{j\}\+\\csc\(\\Lambda\_\{ij\}\)\\overline\{\\beta\}\_\{i\}=\{\\widehat\{Z\}\}\_\{ji\}\.
3. \(3\)Fori∈𝐤i\\in\{\\mathbf\{k\}\}, ∂f∂β¯i​\(𝐰i,𝐰i\)\\displaystyle\\frac\{\\partial f\}\{\\partial\\overline\{\\beta\}\_\{i\}\}\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{i\}\)=\\displaystyle=\[1\+β¯i2​erfc​\(βi2\)−erf​\(βi2\)−2π​β¯i​e−β¯i22\]​𝐰i\\displaystyle\\left\[1\+\\overline\{\\beta\}\_\{i\}^\{2\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\beta\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\beta\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\\sqrt\{\\frac\{2\}\{\\pi\}\}\\overline\{\\beta\}\_\{i\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\\right\]\{\{\\mathbf\{w\}\}\}^\{i\}

###### Proof\.

For theβ¯\\overline\{\\beta\}\-derivative off⁡\(𝐰,𝐰\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{w\}\}\}\)use \([3\.5](https://arxiv.org/html/2609.30661#S3.E5)\)\. ∎

###### Example 9\.4\.

Takeγ¯=0\\overline\{\\gamma\}=0and recall from Lemma[8\.1](https://arxiv.org/html/2609.30661#S8.Thmlemma1)\(2\) that

∂f∂β¯​\(𝐰,𝐯\)=−‖𝐰‖​‖𝐯‖2​2​π​\[erfc​\(csc⁡\(λ\)​β¯2\)\+cos⁡\(λ\)​e−β¯22​erfc​\(−cot⁡\(λ\)​β¯2\)\]\\begin\{aligned\} \\hskip\-14\.45377pt\\frac\{\\partial f\}\{\\partial\\overline\{\\beta\}\}\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)&=\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\csc\(\\lambda\)\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\cos\(\\lambda\)e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\-\\cot\(\\lambda\)\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\end\{aligned\}where we have set𝐰i=𝐰,𝐯j=𝐯\{\{\\mathbf\{w\}\}\}^\{i\}=\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}^\{j\}=\{\{\\mathbf\{v\}\}\}andλi​j=λ\\lambda\_\{ij\}=\\lambda, Computing∂2f∂𝐰​∂β¯\\frac\{\\partial^\{2\}f\}\{\\partial\{\{\\mathbf\{w\}\}\}\\partial\\overline\{\\beta\}\}gives

∂2f∂𝐰​∂β¯=−‖𝐯‖2​2​π​\[erfc​\(csc⁡\(λ\)​β¯2\)\+cos⁡\(λ\)​e−β¯22​erfc​\(−cot⁡\(λ\)​β¯2\)\]​𝐰¯\+‖𝐯‖2​2​π​\[sin⁡\(λ\)​e−β¯22​erfc​\(−cot⁡\(λ\)​β¯2\)\]​𝐣=−‖𝐯‖2​2​π​\[erfc​\(csc⁡\(λ\)​β¯2\)​𝐰¯\+‖𝐯‖2​2​π​e−β¯22​erfc​\(−cot⁡\(λ\)​β¯2\)​𝐯¯\]\\begin\{aligned\} \\frac\{\\partial^\{2\}f\}\{\\partial\{\{\\mathbf\{w\}\}\}\\partial\\overline\{\\beta\}\}=&\-\\frac\{\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\csc\(\\lambda\)\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\cos\(\\lambda\)e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\-\\cot\(\\lambda\)\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\overline\{\{\{\\mathbf\{w\}\}\}\}\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\left\[\\sin\(\\lambda\)e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\-\\cot\(\\lambda\)\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\{\\mathbf\{j\}\}\\\\ =&\-\\frac\{\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\csc\(\\lambda\)\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\overline\{\{\{\\mathbf\{w\}\}\}\}\+\\frac\{\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\sqrt\{2\\pi\}\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\-\\cot\(\\lambda\)\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\overline\{\{\{\\mathbf\{v\}\}\}\}\\right\]\\end\{aligned\}It is a good exercise/check to verify that∂2f∂𝐰​∂β¯=∂2f∂β¯​∂𝐰\\frac\{\\partial^\{2\}f\}\{\\partial\{\{\\mathbf\{w\}\}\}\\partial\\overline\{\\beta\}\}=\\frac\{\\partial^\{2\}f\}\{\\partial\\overline\{\\beta\}\\partial\{\{\\mathbf\{w\}\}\}\}\.※

It follows from Propositions[9\.1](https://arxiv.org/html/2609.30661#S9.Thmlemma1),[9\.3](https://arxiv.org/html/2609.30661#S9.Thmlemma3)that

\(9\.41\)∂f∂𝐰​\(𝐰i,𝐯j\)\\displaystyle\\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\}\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{j\}\)=\\displaystyle=‖𝐯‖2​π​𝒰i​j​𝐰¯i\+12​π​𝒱i​j​𝐯j,i∈𝐤,j∈𝐝\\displaystyle\\frac\{\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\pi\}\\mathcal\{U\}\_\{ij\}\\overline\{\{\{\\mathbf\{w\}\}\}\}^\{i\}\+\\frac\{1\}\{2\\pi\}\\mathcal\{V\}\_\{ij\}\{\{\\mathbf\{v\}\}\}^\{j\},\\;i\\in\{\\mathbf\{k\}\},j\\in\{\\mathbf\{d\}\}\(9\.42\)∂f∂𝐰​\(𝐰i,𝐰j\)\\displaystyle\\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\}\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{j\}\)=\\displaystyle=‖𝐯‖2​π​𝒲i​j​𝐰¯i\+12​π​𝒴i​j​𝐰j,i,j∈𝐤,i≠j\\displaystyle\\frac\{\\\|\{\{\\mathbf\{v\}\}\}\\\|\}\{2\\pi\}\\mathcal\{W\}\_\{ij\}\\overline\{\{\{\\mathbf\{w\}\}\}\}^\{i\}\+\\frac\{1\}\{2\\pi\}\\mathcal\{Y\}\_\{ij\}\{\{\\mathbf\{w\}\}\}^\{j\},\\;i,j\\in\{\\mathbf\{k\}\},i\\neq jwhere

𝒰i​j=−π2​\[β¯i​e−γ¯j22​erfc​\(zi​j2\)\+γ¯j​e−β¯i22​erfc​\(z^i​j2\)\]\+β¯i​γ¯j​\(π2​\[erfc​\(γ¯j2\)\+erfc​\(β¯i2\)\]−λ~i​j\)\+e−ki​j​sin⁡\(λi​j\)𝒱i​j=\(π2​\[erfc​\(γ¯j2\)\+erfc​\(β¯i2\)\]−λ~i​j\)𝒲i​j=−π2​\[β¯i​e−β¯j22​erfc​\(Zi​j2\)\+β¯j​e−β¯i22​erfc​\(Z^i​j2\)\]\+β¯i​β¯j​\(π2​\[erfc​\(β¯j2\)\+erfc​\(β¯i2\)\]−Λ~i​j\)\+e−Ki​j​sin⁡\(Λi​j\)𝒴i​j=\(π2​\[erfc​\(γ¯j2\)\+erfc​\(β¯i2\)\]−Λ~i​j\)\\begin\{aligned\} \\mathcal\{U\}\_\{ij\}=&\-\\sqrt\{\\frac\{\\pi\}\{2\}\}\\left\[\\overline\{\\beta\}\_\{i\}e^\{\-\\frac\{\\overline\{\\gamma\}\_\{j\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{z\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\overline\{\\gamma\}\_\{j\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\{\\widehat\{z\}\}\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ &\+\\overline\{\\beta\}\_\{i\}\\overline\{\\gamma\}\_\{j\}\\left\(\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\{\\widetilde\{\\lambda\}\}\_\{ij\}\\right\)\+e^\{\-k\_\{ij\}\}\\sin\(\\lambda\_\{ij\}\)\\\\ \\mathcal\{V\}\_\{ij\}=&\\left\(\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\{\\widetilde\{\\lambda\}\}\_\{ij\}\\right\)\\\\ \\mathcal\{W\}\_\{ij\}=&\-\\sqrt\{\\frac\{\\pi\}\{2\}\}\\left\[\\overline\{\\beta\}\_\{i\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{j\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{Z\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\overline\{\\beta\}\_\{j\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\{\\widehat\{Z\}\}\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ &\+\\overline\{\\beta\}\_\{i\}\\overline\{\\beta\}\_\{j\}\\left\(\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\{\\widetilde\{\\Lambda\}\}\_\{ij\}\\right\)\+e^\{\-K\_\{ij\}\}\\sin\(\\Lambda\_\{ij\}\)\\\\ \\mathcal\{Y\}\_\{ij\}=&\\left\(\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\{\\widetilde\{\\Lambda\}\}\_\{ij\}\\right\)\\end\{aligned\}\(Notational conventions of section[6\.1](https://arxiv.org/html/2609.30661#S6.SS1)\.\)

### 9\.2\.Angles close to00andπ\\pi

In the examples studied in this paper, the angle between two rows of the parameter matrix𝐖\{\{\\mathbf\{W\}\}\}, or between two rows of the target matrix𝐕\{\{\\mathbf\{V\}\}\}, at a critical point of the loss will be bounded away from00andπ\\pi\. However, angles between a row of𝐖\{\{\\mathbf\{W\}\}\}and a row of𝐕\{\{\\mathbf\{V\}\}\}may be close to00orπ\\pi\. Indeed, fordddependent families of critical points, it is often the case that each𝐖\{\{\\mathbf\{W\}\}\}row converges to±\\pma𝐕\{\{\\mathbf\{V\}\}\}row asd→∞d\{\\rightarrow\}\\infty\. This has the potential to cause problems with the exponential factorse−ki​je^\{\-k\_\{ij\}\}\(see the remark below\)\. The next two lemmas show that under reasonable hypotheses this potential pathology can be controlled\.

###### Lemma 9\.5\.

Assumeθ∈\(0,π\)\\theta\\in\(0,\\pi\)and is either near zero orπ\\pi\.

1. \(1\)csc⁡\(θ\)−cot⁡\(θ\)=tan⁡\(θ2\)=θ2\+θ324\+O⁡\(θ5\)\\csc\(\\theta\)\-\\cot\(\\theta\)=\\tan\\left\(\\frac\{\\theta\}\{2\}\\right\)=\\frac\{\\theta\}\{2\}\+\\frac\{\\theta^\{3\}\}\{24\}\+O\(\\theta^\{5\}\)\.
2. \(2\)csc⁡\(π−θ\)\+cot⁡\(π−θ\)=tan⁡\(π−θ2\)=\(π−θ\)2\+\(π−θ\)324\+O⁡\(\(π−θ\)5\)\\csc\(\\pi\-\\theta\)\+\\cot\(\\pi\-\\theta\)=\\tan\\left\(\\frac\{\\pi\-\\theta\}\{2\}\\right\)=\\frac\{\(\\pi\-\\theta\)\}\{2\}\+\\frac\{\(\\pi\-\\theta\)^\{3\}\}\{24\}\+O\(\(\\pi\-\\theta\)^\{5\}\)\.

###### Proof\.

Elementary trigonometry and the Maclaurin series fortan\\tan\. ∎

###### Lemma 9\.6\.

Setδ¯±=β¯±γ¯\\overline\{\\delta\}\_\{\\pm\}=\\overline\{\\beta\}\\pm\\overline\{\\gamma\},Δ±=csc⁡\(θ\)±c​o​t​\(θ\)\\Delta\_\{\\pm\}=\\csc\(\\theta\)\\pm cot\(\\theta\),z=−γ¯​cot⁡\(θ\)\+β¯​csc⁡\(θ\)z=\-\\overline\{\\gamma\}\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)andz^=−β¯​cot⁡\(θ\)\+γ¯​csc⁡\(θ\)\{\\widehat\{z\}\}=\-\\overline\{\\beta\}\\cot\(\\theta\)\+\\overline\{\\gamma\}\\csc\(\\theta\)\.

1. \(1\)Forθ\\thetanear zero γ¯​z\+β¯​z^\\displaystyle\\overline\{\\gamma\}z\+\\overline\{\\beta\}\{\\widehat\{z\}\}=\\displaystyle=β¯​γ¯​Δ−−δ¯−2​cot⁡\(θ\)\\displaystyle\\overline\{\\beta\}\\overline\{\\gamma\}\\Delta\_\{\-\}\-\\overline\{\\delta\}\_\{\-\}^\{2\}\\cot\(\\theta\)
2. \(2\)Forπ−θ\\pi\-\\thetanear zero γ¯​z\+β¯​z^\\displaystyle\\overline\{\\gamma\}z\+\\overline\{\\beta\}\{\\widehat\{z\}\}=\\displaystyle=β¯​γ¯​Δ\+−δ¯\+2​cot⁡\(θ\)\\displaystyle\\overline\{\\beta\}\\overline\{\\gamma\}\\Delta\_\{\+\}\-\\overline\{\\delta\}\_\{\+\}^\{2\}\\cot\(\\theta\)

###### Proof\.

Straightforward substitution\. ∎

###### Remark 9\.7\.

Lemma[9\.6](https://arxiv.org/html/2609.30661#S9.Thmlemma6)is important as it implies the Curve Selection Lemma\[[7](https://arxiv.org/html/2609.30661#bib.bib7),[14](https://arxiv.org/html/2609.30661#bib.bib14)\]\(analytic arc, limiting at boundary point\) will hold in cuspidal regions with cusp vertices atλ=0\\lambda=0orλ=π\\lambda=\\pi\) and cusps tangent at the vertex toγ¯−β¯=0\\overline\{\\gamma\}\-\\overline\{\\beta\}=0, orγ¯\+β¯=0\\overline\{\\gamma\}\+\\overline\{\\beta\}=0, even though there is the exponential factore−ke^\{\-k\}\. In general, this is*not*so for o\-minimal structures where practical computability also appears to be extremely challenging if we leave the \(sub\)analytic domain\. In other words, we do not expect to see infinite \(indd\) \(analytic\) families of critical points if we get too close to points where subanalyticity breaks down\. The result suggests analytically why \(spurious\) minima in symmetric and asymmetric systems are often characterized by convergence of rows to±\\pma target row asd→∞d\{\\rightarrow\}\\infty\. More generally convergence can be to a non\-zero row which is parallel to more than one target row \(whenk\>dk\>d\[[5](https://arxiv.org/html/2609.30661#bib.bib5), Example 2\(2\)\]\)\. Possibly adding bias, and so breaking subanalyticity for the equations, is an advantage as it makes it harder to have convergence to a row which is not parallel to a target row\. Perhaps the word ‘spurious’ should be reserved for critical point families that break down for largedd\.✠\\maltese

### 9\.3\.Computation of∂f∂𝐰\\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\}\(biasβ\\beta\) I: derivatives

We list the derivatives used that differ from those in the list for normalized bias\.

∂β¯∂𝐰\\displaystyle\\frac\{\\partial\\overline\{\\beta\}\}\{\\partial\{\{\\mathbf\{w\}\}\}\}=\\displaystyle=−β¯‖𝐰‖​𝐰¯,∂e−β¯22∂𝐰=β¯2​e−β¯22‖𝐰‖​𝐰¯\\displaystyle\-\\frac\{\\overline\{\\beta\}\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\\overline\{\{\{\\mathbf\{w\}\}\}\},\\quad\\frac\{\\partial e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\}\{\\partial\{\{\\mathbf\{w\}\}\}\}=\\frac\{\\overline\{\\beta\}^\{2\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\\overline\{\{\{\\mathbf\{w\}\}\}\}∂erf​\(β¯/2\)∂𝐰\\displaystyle\\frac\{\\partial\\,\{\\text\{\\rm erf\}\(\{\\overline\{\\beta\}/\\sqrt\{2\}\}\)\}\}\{\\partial\{\{\\mathbf\{w\}\}\}\}=\\displaystyle=−2π​β¯​e−β¯22‖𝐰‖​𝐰¯\\displaystyle\-\\sqrt\{\\frac\{2\}\{\\pi\}\}\\frac\{\\overline\{\\beta\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\\overline\{\{\{\\mathbf\{w\}\}\}\}∂∂𝐰erf​\(A​β¯\+B​γ¯2\)=2π​e−\(A​β¯\+B​γ¯\)22‖𝐰‖​\(csc⁡\(λ\)​\[A​γ¯\+B​β¯\]​𝐣\+cot⁡\(λ\)​β¯​𝐰¯\)∂∂𝐰erf​\(A​γ¯\+B​β¯2\)=2π​e−\(A​γ¯\+B​β¯\)22‖𝐰‖​\(csc⁡\(λ\)​\[A​β¯\+B​γ¯\]​𝐣−csc⁡\(λ\)​β¯​𝐰¯\)∂∂𝐰k⁡\(β¯,γ¯,λ\)=−β¯​csc⁡\(λ\)‖𝐰‖​\[A​γ¯\+B​β¯\]​𝐰¯\+csc⁡\(λ\)‖𝐰‖​\[\(A​β¯\+B​γ¯\)​\(A​γ¯\+B​β¯\)\]​𝐣∂H∂𝐰=e−k⁡\(β¯,γ¯,λ\)‖𝐰‖​𝐣−π2​β¯​e−β¯22‖𝐰‖​erfc​\(A​β¯\+B​γ¯2\)​𝐰¯\\begin\{aligned\} \\frac\{\\partial\}\{\\partial\{\{\\mathbf\{w\}\}\}\}&\{\\text\{\\rm erf\}\\left\(\{\\frac\{A\\overline\{\\beta\}\+B\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}=\\sqrt\{\\frac\{2\}\{\\pi\}\}\\frac\{e^\{\-\\frac\{\(A\\overline\{\\beta\}\+B\\overline\{\\gamma\}\)^\{2\}\}\{2\}\}\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\\left\(\\csc\(\\lambda\)\\left\[A\\overline\{\\gamma\}\+B\\overline\{\\beta\}\\right\]\{\\mathbf\{j\}\}\+\\cot\(\\lambda\)\\overline\{\\beta\}\\,\\overline\{\{\{\\mathbf\{w\}\}\}\}\\right\)\\\\ \\frac\{\\partial\}\{\\partial\{\{\\mathbf\{w\}\}\}\}&\{\\text\{\\rm erf\}\\left\(\{\\frac\{A\\overline\{\\gamma\}\+B\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}=\\sqrt\{\\frac\{2\}\{\\pi\}\}\\frac\{e^\{\-\\frac\{\(A\\overline\{\\gamma\}\+B\\overline\{\\beta\}\)^\{2\}\}\{2\}\}\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\\left\(\\csc\(\\lambda\)\\left\[A\\overline\{\\beta\}\+B\\overline\{\\gamma\}\\right\]\{\\mathbf\{j\}\}\-\\csc\(\\lambda\)\\overline\{\\beta\}\\,\\overline\{\{\{\\mathbf\{w\}\}\}\}\\right\)\\\\ \\frac\{\\partial\}\{\\partial\{\{\\mathbf\{w\}\}\}\}&k\(\\overline\{\\beta\},\\overline\{\\gamma\},\\lambda\)=\-\\frac\{\\overline\{\\beta\}\\csc\(\\lambda\)\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\\left\[A\\overline\{\\gamma\}\+B\\overline\{\\beta\}\\right\]\\overline\{\{\{\\mathbf\{w\}\}\}\}\+\\frac\{\\csc\(\\lambda\)\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\\left\[\(A\\overline\{\\beta\}\+B\\overline\{\\gamma\}\)\(A\\overline\{\\gamma\}\+B\\overline\{\\beta\}\)\\right\]\{\\mathbf\{j\}\}\\\\ \\frac\{\\partial H\}\{\\partial\{\{\\mathbf\{w\}\}\}\}&=\\frac\{e^\{\-k\(\\overline\{\\beta\},\\overline\{\\gamma\},\\lambda\)\}\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\{\\mathbf\{j\}\}\-\\sqrt\{\\frac\{\\pi\}\{2\}\}\\frac\{\\overline\{\\beta\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{A\\overline\{\\beta\}\+B\\overline\{\\gamma\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\overline\{\{\{\\mathbf\{w\}\}\}\}\\end\{aligned\}

### 9\.4\.The computation of∂f∂𝐰\\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\}\(biasOPENβ\)\\beta\): II

Although our emphasis is on normalized bias, it is straightforward to give a formula for∂f∂𝐰\\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\}when the bias isβ\\beta\. If∂f∂𝐰~\\widetilde\{\\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\}\}denotes the gradient when the bias isβ\\beta, then

\(9\.43\)∂f∂𝐰~=∂f∂𝐰−β¯​𝐰¯‖𝐰‖​∂f∂β¯=∂f∂𝐰−β¯​𝐰¯​∂f∂β\\widetilde\{\\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\}\}=\\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\}\-\\frac\{\\overline\{\\beta\}\\overline\{\{\{\\mathbf\{w\}\}\}\}\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\\frac\{\\partial f\}\{\\partial\\overline\{\\beta\}\}=\\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\}\-\\overline\{\\beta\}\\overline\{\{\{\\mathbf\{w\}\}\}\}\\frac\{\\partial f\}\{\\partial\\beta\}This can be proved straightforwardly using the formula for∂f∂𝐰\\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\}when the bias isβ¯\\overline\{\\beta\}or directly using the results listed in section[9\.3](https://arxiv.org/html/2609.30661#S9.SS3)\.

## 10\.The formula for∂ℒ∂𝐰\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\{\{\\mathbf\{w\}\}\}\}\(no symmetry assumption\)

Assume given𝐖∈M⁡\(k,d\)\{\{\\mathbf\{W\}\}\}\\in M\(k,d\)and a target matrix𝐕∈M⁡\(d,d\)\{\{\\mathbf\{V\}\}\}\\in M\(d,d\)\. Denote rowiiof𝐖\{\{\\mathbf\{W\}\}\}\(resp\. rowjjof𝐯\{\{\\mathbf\{v\}\}\}\) by𝐰i\{\{\\mathbf\{w\}\}\}^\{i\}\(resp\.𝐯j\{\{\\mathbf\{v\}\}\}^\{j\}\)\. We extend the notational conventions developed at the end of section[3](https://arxiv.org/html/2609.30661#S3)\. Recall thatλi​j\\lambda\_\{ij\}denotes the angle between𝐰i\{\{\\mathbf\{w\}\}\}^\{i\}and𝐯j\{\{\\mathbf\{v\}\}\}^\{j\},i∈𝐤,j∈𝐝i\\in\{\\mathbf\{k\}\},j\\in\{\\mathbf\{d\}\}, andΛi​j\\Lambda\_\{ij\}denotes the angle between𝐰i\{\{\\mathbf\{w\}\}\}^\{i\}and𝐰j\{\{\\mathbf\{w\}\}\}^\{j\},i,j∈𝐤i,j\\in\{\\mathbf\{k\}\}\. Assume no parallel rows\. In particular:λi​j∉\{0,π\}\\lambda\_\{ij\}\\notin\\\{0,\\pi\\\},i∈𝐤,j∈𝐝i\\in\{\\mathbf\{k\}\},j\\in\{\\mathbf\{d\}\}andΛi​j∉\{0,π\}\\Lambda\_\{ij\}\\notin\\\{0,\\pi\\\},i,j∈𝐤,i≠ji,j\\in\{\\mathbf\{k\}\},i\\neq j\.

### 10\.1\.Table defining terms used in the formula forgrad​\(ℒ\)​\(𝐖\)\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}\(\{\{\\mathbf\{W\}\}\}\)

Zi​j\\displaystyle Z\_\{ij\}=\\displaystyle=−cot⁡\(Λi​j\)​β¯j\+csc⁡\(Λi​j\)​β¯iZ^i​j=−cot⁡\(Λi​j\)​β¯i\+csc⁡\(Λi​j\)​β¯j\\displaystyle\-\\cot\(\\Lambda\_\{ij\}\)\\overline\{\\beta\}\_\{j\}\+\\csc\(\\Lambda\_\{ij\}\)\\overline\{\\beta\}\_\{i\}\\quad\{\\widehat\{Z\}\}\_\{ij\}=\-\\cot\(\\Lambda\_\{ij\}\)\\overline\{\\beta\}\_\{i\}\+\\csc\(\\Lambda\_\{ij\}\)\\overline\{\\beta\}\_\{j\}zi​j\\displaystyle z\_\{ij\}=\\displaystyle=−cot⁡\(λi​j\)​γ¯j\+csc⁡\(λi​j\)​β¯iz^i​j=−cot⁡\(λi​j\)​β¯i\+csc⁡\(λi​j\)​γ¯j\\displaystyle\-\\cot\(\\lambda\_\{ij\}\)\\overline\{\\gamma\}\_\{j\}\+\\csc\(\\lambda\_\{ij\}\)\\overline\{\\beta\}\_\{i\}\\quad\{\\widehat\{z\}\}\_\{ij\}=\-\\cot\(\\lambda\_\{ij\}\)\\overline\{\\beta\}\_\{i\}\+\\csc\(\\lambda\_\{ij\}\)\\overline\{\\gamma\}\_\{j\}Ki​j\\displaystyle K\_\{ij\}=\\displaystyle=K⁡\(β¯i,β¯j,Λi​j\)=12​\[\(β¯i2\+β¯j2\)​csc2⁡\(Λi​j\)−2​β¯i​β¯j​csc⁡\(Λi​j\)​cot⁡\(Λi​j\)\]\\displaystyle K\(\\overline\{\\beta\}\_\{i\},\\overline\{\\beta\}\_\{j\},\\Lambda\_\{ij\}\)=\\frac\{1\}\{2\}\\left\[\(\\overline\{\\beta\}\_\{i\}^\{2\}\+\\overline\{\\beta\}\_\{j\}^\{2\}\)\\csc^\{2\}\(\\Lambda\_\{ij\}\)\-2\\overline\{\\beta\}\_\{i\}\\overline\{\\beta\}\_\{j\}\\csc\(\\Lambda\_\{ij\}\)\\cot\(\\Lambda\_\{ij\}\)\\right\]ki​j\\displaystyle k\_\{ij\}=\\displaystyle=k⁡\(β¯i,γ¯j,λi​j\)=12​\[\(β¯i2\+γ¯j2\)​csc2⁡\(λi​j\)−2​β¯i​γ¯j​csc⁡\(λi​j\)​cot⁡\(λi​j\)\]\\displaystyle k\(\\overline\{\\beta\}\_\{i\},\\overline\{\\gamma\}\_\{j\},\\lambda\_\{ij\}\)=\\frac\{1\}\{2\}\\left\[\(\\overline\{\\beta\}\_\{i\}^\{2\}\+\\overline\{\\gamma\}\_\{j\}^\{2\}\)\\csc^\{2\}\(\\lambda\_\{ij\}\)\-2\\overline\{\\beta\}\_\{i\}\\overline\{\\gamma\}\_\{j\}\\csc\(\\lambda\_\{ij\}\)\\cot\(\\lambda\_\{ij\}\)\\right\]Li​j\\displaystyle L\_\{ij\}=\\displaystyle=−π2​\[erf​\(β¯i2\)\+erf​\(β¯j2\)\],ℓi​j=−π2​\[erf​\(β¯i2\)\+erf​\(γ¯j2\)\]\\displaystyle\-\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\],\\;\\ell\_\{ij\}=\-\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\gamma\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]Ei​jX\\displaystyle E^\{X\}\_\{ij\}=\\displaystyle=−π2​\[β¯i​e−β¯j22​erfc​\(Zi​j2\)\+β¯j​e−β¯i22​erfc​\(Z^i​j2\)\]\\displaystyle\-\\sqrt\{\\frac\{\\pi\}\{2\}\}\\left\[\\overline\{\\beta\}\_\{i\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{j\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{Z\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\overline\{\\beta\}\_\{j\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\{\\widehat\{Z\}\}\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]ei​jX\\displaystyle e^\{X\}\_\{ij\}=\\displaystyle=−π2​\[β¯i​e−γ¯j22​erfc​\(zi​j2\)\+γ¯j​e−β¯i22​erfc​\(z^i​j2\)\]\\displaystyle\-\\sqrt\{\\frac\{\\pi\}\{2\}\}\\left\[\\overline\{\\beta\}\_\{i\}e^\{\-\\frac\{\\overline\{\\gamma\}\_\{j\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{z\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\overline\{\\gamma\}\_\{j\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\{\\widehat\{z\}\}\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]Xi\\displaystyle X\_\{i\}=\\displaystyle=β¯i2​erfc​\(β¯i2\)−erf​\(β¯i2\)−2π​β¯i​e−β¯i22\\displaystyle\\overline\{\\beta\}\_\{i\}^\{2\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\\sqrt\{\\frac\{2\}\{\\pi\}\}\\overline\{\\beta\}\_\{i\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}The capitalized terms are defined fori,j∈𝐤i,j\\in\{\\mathbf\{k\}\},j≠ij\\neq i\. Lower case terms are defined fori∈𝐤i\\in\{\\mathbf\{k\}\},j∈𝐝j\\in\{\\mathbf\{d\}\}\. The termsKi​j,Li​j,Ei​jXK\_\{ij\},L\_\{ij\},E\_\{ij\}^\{X\}are symmetric ini,ji,jand, with the exception ofLi​jL\_\{ij\}, the diagonal terms are undefined—though it is safe to set these terms \(apart fromLi​iL\_\{ii\}\) to zero, which is the limiting value asΛi​i→0\\Lambda\_\{ii\}\{\\rightarrow\}0\. Note thatZj​i=Z^i​jZ\_\{ji\}=\{\\widehat\{Z\}\}\_\{ij\}but thatzj​iz\_\{ji\}is generally*not*equal toz^i​j\{\\widehat\{z\}\}\_\{ij\}unless, for example,λi​j=λj​i\\lambda\_\{ij\}=\\lambda\_\{ji\}\(λj​i\\lambda\_\{ji\}is not even defined ifi∉𝐝i\\notin\{\\mathbf\{d\}\}\)\.

### 10\.2\.The terms𝒰i​j,…,𝒴i​j\\mathcal\{U\}\_\{ij\},\\ldots,\\mathcal\{Y\}\_\{ij\}

From Section[9](https://arxiv.org/html/2609.30661#S9)we have

∂f∂𝐰​\(𝐰i,𝐯j\)\\displaystyle\\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\}\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{j\}\)=\\displaystyle=‖𝐯j‖2​π​𝒰i​j​𝐰¯i\+12​π​𝒱i​j​𝐯j,i∈𝐤,j∈𝐝\\displaystyle\\frac\{\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|\}\{2\\pi\}\\mathcal\{U\}\_\{ij\}\\overline\{\{\{\\mathbf\{w\}\}\}\}^\{i\}\+\\frac\{1\}\{2\\pi\}\\mathcal\{V\}\_\{ij\}\{\{\\mathbf\{v\}\}\}^\{j\},\\;i\\in\{\\mathbf\{k\}\},j\\in\{\\mathbf\{d\}\}∂f∂𝐰​\(𝐰i,𝐰j\)\\displaystyle\\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}\}\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{j\}\)=\\displaystyle=‖𝐰j‖2​π​𝒲i​j​𝐰¯i\+12​π​𝒴i​j​𝐰j,i,j∈𝐤,i≠j\\displaystyle\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\}\{2\\pi\}\\mathcal\{W\}\_\{ij\}\\overline\{\{\{\\mathbf\{w\}\}\}\}^\{i\}\+\\frac\{1\}\{2\\pi\}\\mathcal\{Y\}\_\{ij\}\{\{\\mathbf\{w\}\}\}^\{j\},\\;i,j\\in\{\\mathbf\{k\}\},i\\neq j∂f∂𝐰i​\(𝐰i,𝐰i\)\\displaystyle\\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}^\{i\}\}\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{i\}\)=\\displaystyle=𝒳i​𝐰i,i∈𝐤\\displaystyle\\mathcal\{X\}\_\{i\}\{\{\\mathbf\{w\}\}\}^\{i\},\\;i\\in\{\\mathbf\{k\}\}where

𝒰i​j=−π2​\[β¯i​e−γ¯j22​erfc​\(zi​j2\)\+γ¯j​e−β¯i22​erfc​\(z^i​j2\)\]\+β¯i​γ¯j​\(π2​\[erfc​\(γ¯j2\)\+erfc​\(β¯i2\)\]−λ~i​j\)\+e−ki​j​sin⁡\(λi​j\)=ei​jX\+β¯i​γ¯j​\(ℓi​j\+π−λ~i​j\)\+e−ki​j​sin⁡\(λi​j\)𝒱i​j=\(π2​\[erfc​\(γ¯j2\)\+erfc​\(β¯i2\)\]−λ~i​j\)=ℓi​j\+π−λ~i​j𝒲i​j=−π2​\[β¯i​e−β¯j22​erfc​\(Zi​j2\)\+β¯j​e−β¯i22​erfc​\(Z^i​j2\)\]\+β¯i​β¯j​\(π2​\[erfc​\(β¯j2\)\+erfc​\(β¯i2\)\]−Λ~i​j\)\+e−Ki​j​sin⁡\(Λi​j\)=Ei​jX\+β¯i​β¯j​\(Li​j\+π−Λ~i​j\)\+e−Ki​j​sin⁡\(Λi​j\)𝒴i​j=π2​\[erfc​\(β¯i2\)\+erfc​\(β¯j2\)\]−Λ~i​j=Li​j\+π−Λ~i​j𝒳i=\(β¯i2\+1\)​erfc​\(β¯i2\)−2π​β¯i​e−β¯i22=1\+Xi\\begin\{aligned\} \\mathcal\{U\}\_\{ij\}=&\-\\sqrt\{\\frac\{\\pi\}\{2\}\}\\left\[\\overline\{\\beta\}\_\{i\}e^\{\-\\frac\{\\overline\{\\gamma\}\_\{j\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{z\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\overline\{\\gamma\}\_\{j\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\{\\widehat\{z\}\}\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ &\+\\overline\{\\beta\}\_\{i\}\\overline\{\\gamma\}\_\{j\}\\left\(\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\{\\widetilde\{\\lambda\}\}\_\{ij\}\\right\)\+e^\{\-k\_\{ij\}\}\\sin\(\\lambda\_\{ij\}\)\\\\ &=e\_\{ij\}^\{X\}\+\\overline\{\\beta\}\_\{i\}\\overline\{\\gamma\}\_\{j\}\\big\(\\ell\_\{ij\}\+\\pi\-\{\\widetilde\{\\lambda\}\}\_\{ij\}\\big\)\+e^\{\-k\_\{ij\}\}\\sin\(\\lambda\_\{ij\}\)\\\\ \\mathcal\{V\}\_\{ij\}=&\\left\(\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\\widetilde\{\\lambda\}\_\{ij\}\\right\)\\\\ =&\\ell\_\{ij\}\+\\pi\-\\widetilde\{\\lambda\}\_\{ij\}\\\\ \\mathcal\{W\}\_\{ij\}=&\-\\sqrt\{\\frac\{\\pi\}\{2\}\}\\left\[\\overline\{\\beta\}\_\{i\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{j\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{Z\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\overline\{\\beta\}\_\{j\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\{\\widehat\{Z\}\}\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ &\+\\overline\{\\beta\}\_\{i\}\\overline\{\\beta\}\_\{j\}\\left\(\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\{\\widetilde\{\\Lambda\}\}\_\{ij\}\\right\)\+e^\{\-K\_\{ij\}\}\\sin\(\\Lambda\_\{ij\}\)\\\\ &=E\_\{ij\}^\{X\}\+\\overline\{\\beta\}\_\{i\}\\overline\{\\beta\}\_\{j\}\\big\(L\_\{ij\}\+\\pi\-\{\\widetilde\{\\Lambda\}\}\_\{ij\}\\big\)\+e^\{\-K\_\{ij\}\}\\sin\(\\Lambda\_\{ij\}\)\\\\ \\mathcal\{Y\}\_\{ij\}=&\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\\widetilde\{\\Lambda\}\_\{ij\}\\\\ =&L\_\{ij\}\+\\pi\-\\widetilde\{\\Lambda\}\_\{ij\}\\\\ \\mathcal\{X\}\_\{i\}=&\(\\overline\{\\beta\}\_\{i\}^\{2\}\+1\)\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\\sqrt\{\\frac\{2\}\{\\pi\}\}\\overline\{\\beta\}\_\{i\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}=1\+X\_\{i\}\\end\{aligned\}Fori∈𝐤i\\in\{\\mathbf\{k\}\}, denote componentiiofgrad​\(ℒ\)​\(𝐖\)\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}\(\{\{\\mathbf\{W\}\}\}\)bygrad​\(ℒ\)​\(𝐖\)i\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}\(\{\{\\mathbf\{W\}\}\}\)^\{i\}\(a row vector\)\. Then

grad​\(ℒ\)​\(𝐖\)i=𝒳i2​𝐰i\+12​π​\(∑j∈𝐤,j≠i‖𝐰j‖​𝒲i​j‖𝐰i‖−∑j∈𝐝‖𝐯j‖​𝒰i​j‖𝐰i‖\)​𝐰i\+12​π​\(∑j∈𝐤,j≠i𝒴i​j​𝐰j−∑j∈𝐝𝒱i​j​𝐯j\)\\begin\{aligned\} \{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}\(\{\{\\mathbf\{W\}\}\}\)^\{i\}&=\\frac\{\\mathcal\{X\}\_\{i\}\}\{2\}\{\{\\mathbf\{w\}\}\}^\{i\}\+\\frac\{1\}\{2\\pi\}\\left\(\\sum\_\{j\\in\{\\mathbf\{k\}\},j\\neq i\}\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\\mathcal\{W\}\_\{ij\}\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\-\\sum\_\{j\\in\{\\mathbf\{d\}\}\}\\frac\{\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|\\mathcal\{U\}\_\{ij\}\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\\right\)\{\{\\mathbf\{w\}\}\}^\{i\}\\\\ &\\hskip 43\.36243pt\+\\frac\{1\}\{2\\pi\}\\left\(\\sum\_\{j\\in\{\\mathbf\{k\}\},j\\neq i\}\\mathcal\{Y\}\_\{ij\}\{\{\\mathbf\{w\}\}\}^\{j\}\-\\sum\_\{j\\in\{\\mathbf\{d\}\}\}\\mathcal\{V\}\_\{ij\}\{\{\\mathbf\{v\}\}\}^\{j\}\\right\)\\end\{aligned\}Using the expressions for𝒰i​j,…,𝒴i​j\\mathcal\{U\}\_\{ij\},\\ldots,\\mathcal\{Y\}\_\{ij\}and𝒳i\\mathcal\{X\}\_\{i\}, we may write

\(10\.44\)grad​\(ℒ\)​\(𝐖\)i=Γi​𝐰i−𝐀i\+𝐁i\+𝛀\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}\(\{\{\\mathbf\{W\}\}\}\)^\{i\}=\\Gamma\_\{i\}\{\{\\mathbf\{w\}\}\}^\{i\}\-\\mathbf\{A\}^\{i\}\+\\mathbf\{B\}^\{i\}\+\{\\boldsymbol\{\\Omega\}\}wheregrad​\(ℒ\)​\(𝐖\)i\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}\(\{\{\\mathbf\{W\}\}\}\)^\{i\}is theiith row ofgrad​\(ℒ\)​\(𝐖\)∈M​\(k,d\)\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}\(\{\{\\mathbf\{W\}\}\}\)\\in M\(k,d\)and

Γi\\displaystyle\\Gamma\_\{i\}=\\displaystyle=Xi2\+12​π​\[∑j∈𝐤,j≠i‖𝐰j‖​e−Ki​j​sin⁡\(Λi​j\)‖𝐰i‖−∑j∈𝐝‖𝐯j‖​e−ki​j​sin⁡\(λi​j\)‖𝐰i‖\]\\displaystyle\\frac\{X\_\{i\}\}\{2\}\+\\frac\{1\}\{2\\pi\}\\left\[\\sum\_\{j\\in\{\\mathbf\{k\}\},j\\neq i\}\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|e^\{\-K\_\{ij\}\}\\sin\(\\Lambda\_\{ij\}\)\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\-\\sum\_\{j\\in\{\\mathbf\{d\}\}\}\\frac\{\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|e^\{\-k\_\{ij\}\}\\sin\(\\lambda\_\{ij\}\)\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\\right\]\+12​π​\[∑j∈𝐤,j≠i‖𝐰j‖​Ei​jX‖𝐰i‖−∑j∈𝐝‖𝐯j‖​ei​jX‖𝐰i‖\]\\displaystyle\+\\frac\{1\}\{2\\pi\}\\left\[\\sum\_\{j\\in\{\\mathbf\{k\}\},j\\neq i\}\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|E\_\{ij\}^\{X\}\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\-\\sum\_\{j\\in\{\\mathbf\{d\}\}\}\\frac\{\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|e\_\{ij\}^\{X\}\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\\right\]\+β¯i2​π​\[∑j∈𝐤,j≠iβ¯j​‖𝐰j‖​\(Li​j\+\(π−Λ~i​j\)\)‖𝐰i‖−∑j∈𝐝γ¯j​‖𝐯j‖​\(ℓi​j\+\(π−λ~i​j\)\)‖𝐰i‖\]\\displaystyle\+\\frac\{\\overline\{\\beta\}\_\{i\}\}\{2\\pi\}\\left\[\\sum\_\{j\\in\{\\mathbf\{k\}\},j\\neq i\}\\frac\{\\overline\{\\beta\}\_\{j\}\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\\big\(L\_\{ij\}\+\(\\pi\-\{\\widetilde\{\\Lambda\}\}\_\{ij\}\)\\big\)\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\-\\sum\_\{j\\in\{\\mathbf\{d\}\}\}\\frac\{\\overline\{\\gamma\}\_\{j\}\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|\(\\ell\_\{ij\}\+\(\\pi\-\{\\widetilde\{\\lambda\}\}\_\{ij\}\)\)\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\\right\]𝐀i\\displaystyle\{\\mathbf\{A\}\}^\{i\}=\\displaystyle=12​π​\(∑j∈𝐤,j≠iΛ~i​j​𝐰j−∑j∈𝐝λ~i​j​𝐯j\)𝐁i=12​π​\(∑j∈𝐤,j≠iLi​j​𝐰j−∑j∈𝐝ℓi​j​𝐯j\)\\displaystyle\\frac\{1\}\{2\\pi\}\\left\(\\sum\_\{j\\in\{\\mathbf\{k\}\},j\\neq i\}\{\\widetilde\{\\Lambda\}\}\_\{ij\}\{\{\\mathbf\{w\}\}\}^\{j\}\-\\sum\_\{j\\in\{\\mathbf\{d\}\}\}\{\\widetilde\{\\lambda\}\}\_\{ij\}\{\{\\mathbf\{v\}\}\}^\{j\}\\right\)\\quad\{\\mathbf\{B\}\}^\{i\}=\\frac\{1\}\{2\\pi\}\\left\(\\sum\_\{j\\in\{\\mathbf\{k\}\},j\\neq i\}L\_\{ij\}\{\{\\mathbf\{w\}\}\}^\{j\}\-\\sum\_\{j\\in\{\\mathbf\{d\}\}\}\\ell\_\{ij\}\{\{\\mathbf\{v\}\}\}^\{j\}\\right\)𝛀\\displaystyle\{\\boldsymbol\{\\Omega\}\}=\\displaystyle=12​\(∑j∈𝐤𝐰j−∑j∈𝐝𝐯j\)\\displaystyle\\frac\{1\}\{2\}\\left\(\\sum\_\{j\\in\{\\mathbf\{k\}\}\}\{\{\\mathbf\{w\}\}\}^\{j\}\-\\sum\_\{j\\in\{\\mathbf\{d\}\}\}\{\{\\mathbf\{v\}\}\}^\{j\}\\right\)
Setting all biases to zero,𝒰i​j=sin⁡\(λi​j\)\\mathcal\{U\}\_\{ij\}=\\sin\(\\lambda\_\{ij\}\),𝒱i​j=π−λi​j\\mathcal\{V\}\_\{ij\}=\\pi\-\\lambda\_\{ij\},𝒲i​j=sin⁡\(Λi​j\)\\mathcal\{W\}\_\{ij\}=\\sin\(\\Lambda\_\{ij\}\),𝒴i​j=π−Λi​j\\mathcal\{Y\}\_\{ij\}=\\pi\-\\Lambda\_\{ij\},Xi=0X\_\{i\}=0, and𝐁i=𝟎\{\\mathbf\{B\}\}^\{i\}=\\mathbf\{0\}\. Substitution in the formula forgrad​\(ℒ\)​\(𝐖\)i\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}\(\{\{\\mathbf\{W\}\}\}\)^\{i\}gives the result for the bias free case\[[8](https://arxiv.org/html/2609.30661#bib.bib8)\]\(see also\[[22](https://arxiv.org/html/2609.30661#bib.bib22)\]and\[[4](https://arxiv.org/html/2609.30661#bib.bib4), Prop\. 4\.11\]\):

grad​\(ℒ\)\(𝐖\)i=12​π​\[∑j∈𝐤‖𝐰j‖​sin⁡\(Λi​j\)‖𝐰i‖−∑j∈𝐝‖𝐯j‖​sin⁡\(λi​j\)‖𝐰i‖\]​𝐰i−12​π​\(∑j∈𝐤Λi​j​𝐰j−∑j∈𝐝λi​j​𝐯j\)\+12​\(∑j∈𝐤𝐰j−∑j∈𝐝𝐯j\)\\begin\{aligned\} \{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}&\(\{\{\\mathbf\{W\}\}\}\)^\{i\}=\\frac\{1\}\{2\\pi\}\\left\[\\sum\_\{j\\in\{\\mathbf\{k\}\}\}\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\\sin\(\\Lambda\_\{ij\}\)\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\\ \-\\sum\_\{j\\in\{\\mathbf\{d\}\}\}\\frac\{\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|\\sin\(\\lambda\_\{ij\}\)\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\\right\]\{\{\\mathbf\{w\}\}\}^\{i\}\\\\ &\-\\frac\{1\}\{2\\pi\}\\left\(\\sum\_\{j\\in\{\\mathbf\{k\}\}\}\\Lambda\_\{ij\}\{\{\\mathbf\{w\}\}\}^\{j\}\-\\sum\_\{j\\in\{\\mathbf\{d\}\}\}\\lambda\_\{ij\}\{\{\\mathbf\{v\}\}\}^\{j\}\\right\)\+\\frac\{1\}\{2\}\\left\(\\sum\_\{j\\in\{\\mathbf\{k\}\}\}\{\{\\mathbf\{w\}\}\}^\{j\}\-\\sum\_\{j\\in\{\\mathbf\{d\}\}\}\{\{\\mathbf\{v\}\}\}^\{j\}\\right\)\\end\{aligned\}
###### Remarks 10\.1\.

\(1\) Note thatΛi​i=0\\Lambda\_\{ii\}=0, so that‖𝐰i‖​sin⁡\(Λi​i\)=0\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\sin\(\\Lambda\_\{ii\}\)=0in the first sum\. The row vector𝛀\\boldsymbol\{\\Omega\}is common to both the biased and bias free equations and the “angle” terms𝐀i\{\\mathbf\{A\}\}^\{i\}have the same structure for both biased and bias free equations\. \(2\) Write𝐀i=𝐔i−𝐕i\\mathbf\{A\}^\{i\}=\\mathbf\{U\}^\{i\}\-\\mathbf\{V\}^\{i\}where

𝐓i\\displaystyle\\mathbf\{T\}^\{i\}=\\displaystyle=\(T0i,…,Td−1i\)=12​π​∑ℓ∈𝐤,ℓ≠iΛ~i​ℓ​𝐰ℓ\\displaystyle\(T^\{i\}\_\{0\},\\ldots,T^\{i\}\_\{d\-1\}\)=\\frac\{1\}\{2\\pi\}\\sum\_\{\\ell\\in\{\\mathbf\{k\}\},\\ell\\neq i\}\{\\widetilde\{\\Lambda\}\}\_\{i\\ell\}\{\{\\mathbf\{w\}\}\}^\{\\ell\}𝐔i\\displaystyle\\mathbf\{U\}^\{i\}=\\displaystyle=\(U0i,…,Ud−1i\)=12​π​∑ℓ∈𝐝λ~i​ℓ​𝐯ℓ\\displaystyle\(U^\{i\}\_\{0\},\\ldots,U^\{i\}\_\{d\-1\}\)=\\frac\{1\}\{2\\pi\}\\sum\_\{\\ell\\in\{\\mathbf\{d\}\}\}\{\\widetilde\{\\lambda\}\}\_\{i\\ell\}\{\{\\mathbf\{v\}\}\}^\{\\ell\}The components of𝐓i\\mathbf\{T\}^\{i\}and𝐔i\\mathbf\{U\}^\{i\}are given by

Tji=12​π​∑ℓ∈𝐤,ℓ≠iΛ~i​ℓ​wℓ​jUji=12​π​∑ℓ∈𝐝λ~i​ℓ​vℓ​j,j∈𝐝T^\{i\}\_\{j\}=\\frac\{1\}\{2\\pi\}\\sum\_\{\\ell\\in\{\\mathbf\{k\}\},\\ell\\neq i\}\{\\widetilde\{\\Lambda\}\}\_\{i\\ell\}w\_\{\\ell j\}\\quad U^\{i\}\_\{j\}=\\frac\{1\}\{2\\pi\}\\sum\_\{\\ell\\in\{\\mathbf\{d\}\}\}\{\\widetilde\{\\lambda\}\}\_\{i\\ell\}v\_\{\\ell j\},\\;j\\in\{\\mathbf\{d\}\}If the target𝐕\{\{\\mathbf\{V\}\}\}is the identity matrix, thenUji=12​π​λ~i​j,j∈𝐝\.U^\{i\}\_\{j\}=\\frac\{1\}\{2\\pi\}\{\\widetilde\{\\lambda\}\}\_\{ij\},\\;j\\in\{\\mathbf\{d\}\}\.Similar remarks hold for the row vectors𝐁i\{\\mathbf\{B\}\}^\{i\}\. All of this is straightforward for the general case but becomes less obvious, but more interesting, in the presence of symmetry when the target matrix typically has a diagonal blockIpI\_\{p\},2≤p≤d2\\leq p\\leq d\. We refer Section[14](https://arxiv.org/html/2609.30661#S14)for more details and examples, including ones where𝐕\{\{\\mathbf\{V\}\}\}contains an asymmetric block\. In Section[15\.1](https://arxiv.org/html/2609.30661#S15.SS1)we give details for the case when the target isIdI\_\{d\}and families of spurious minima have a symmetry \(isotropy\) ranging fromΔ​Sd\\Delta S\_\{d\}toΔ⁡\(Sd−q×Sq\)\\Delta\(S\_\{d\-q\}\\times S\_\{q\}\),q≥1q\\geq 1\. \(3\) The usual convexity property holds for solutions of the critical point equation\. That is, suppose\(𝐖,𝜷¯\)∈M⁡\(k,d\+1\)\(\{\{\\mathbf\{W\}\}\},\\boldsymbol\{\\overline\{\\beta\}\}\)\\in M\(k,d\+1\)is a solution of the critical point equation and𝐰i\{\{\\mathbf\{w\}\}\}^\{i\}is a row of𝐖\{\{\\mathbf\{W\}\}\}with associated biasβ¯i\\overline\{\\beta\}\_\{i\}\. Letλ,μ\>0\\lambda,\\mu\>0withλ\+μ=1\\lambda\+\\mu=1\. Define\(𝐖⋆,𝜷¯⋆\)∈M⁡\(k\+1,d\+1\)\(\{\{\\mathbf\{W\}\}\}^\{\\star\},\\boldsymbol\{\\overline\{\\beta\}\}^\{\\star\}\)\\in M\(k\+1,d\+1\)by replacing𝐰i\{\{\\mathbf\{w\}\}\}^\{i\}by the pair of rows\(λ​𝐰i,μ​𝐰i\)\(\\lambda\{\{\\mathbf\{w\}\}\}^\{i\},\\mu\{\{\\mathbf\{w\}\}\}^\{i\}\), both with biasβ¯i\\overline\{\\beta\}\_\{i\}\. Then\(𝐖⋆,𝜷¯⋆\)∈M⁡\(k\+1,d\+1\)\(\{\{\\mathbf\{W\}\}\}^\{\\star\},\\boldsymbol\{\\overline\{\\beta\}\}^\{\\star\}\)\\in M\(k\+1,d\+1\)is a solution of the corresponding critical point equation\. For more generalities on this well\-known phenomenon, which implies that the set of critical points defining global minima form a simplicial complex of degenerate solutions whenk\>dk\>d, see\[[5](https://arxiv.org/html/2609.30661#bib.bib5), A\.7\]where the phenomenon is referred to as “fossilization”: every time a neuron is added the old critical points remain but in a degenerate state spread out over a simplicial complex\.✠\\maltese

## 11\.The formula for∂ℒ∂β¯\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\}

We continue with the previous notation and assumptions\. In particular, letℒ\\mathcal\{L\}denote the loss and regardℒ\\mathcal\{L\}as a function of𝐖∈M⁡\(k,d\)\{\{\\mathbf\{W\}\}\}\\in M\(k,d\),𝜷¯∈M⁡\(k,1\)\\boldsymbol\{\\overline\{\\beta\}\}\\in M\(k,1\)\.

###### Proposition 11\.1\.

Fori∈𝐤i\\in\{\\mathbf\{k\}\},

∂ℒ∂β¯i=‖𝐰i‖22​\[β¯i​erfc​\(β¯i2\)−2π​e−β¯i22\]\+‖𝐰i‖2​π∑j∈𝐤,j≠i∥𝐰j∥β¯j\(π2\[erfc\(β¯i2\)\+erfc\(β¯j2\)\]−Λ~i​j\)−‖𝐰i‖2​2​π∑j∈𝐤,j≠i∥𝐰j∥\[e−β¯j22erfc\(Zi​j2\)\+cos\(Λi​j\)e−β¯i22erfc\(Z^i​j2\)\]−‖𝐰i‖2​π∑j∈𝐝∥𝐯j∥γ¯j\(π2\[erfc\(β¯i2\)\+erfc\(γ¯j2\)\]−λ~i​j\)\+‖𝐰i‖2​2​π∑j∈𝐝∥𝐯j∥\[e−γ¯j22erfc\(zi​j2\)\+cos\(λi​j\)e−β¯i22erfc\(z^i​j2\)\]\\begin\{aligned\} \\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{i\}\}&=\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|^\{2\}\}\{2\}\\left\[\\overline\{\\beta\}\_\{i\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\\sqrt\{\\frac\{2\}\{\\pi\}\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\\right\]\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\{2\\pi\}\\sum\_\{j\\in\{\\mathbf\{k\}\},j\\neq i\}\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\\overline\{\\beta\}\_\{j\}\\left\(\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\{\\widetilde\{\\Lambda\}\}\_\{ij\}\\right\)\\\\ &\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\sum\_\{j\\in\{\\mathbf\{k\}\},j\\neq i\}\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\\left\[e^\{\-\\frac\{\\overline\{\\beta\}\_\{j\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{Z\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\cos\(\\Lambda\_\{ij\}\)e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\{\\widehat\{Z\}\}\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ &\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\{2\\pi\}\\sum\_\{j\\in\{\\mathbf\{d\}\}\}\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|\\overline\{\\gamma\}\_\{j\}\\left\(\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\{\\widetilde\{\\lambda\}\}\_\{ij\}\\right\)\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\sum\_\{j\\in\{\\mathbf\{d\}\}\}\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|\\left\[e^\{\-\\frac\{\\overline\{\\gamma\}\_\{j\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{z\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\cos\(\\lambda\_\{ij\}\)e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\{\\widehat\{z\}\}\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\end\{aligned\}

###### Proof\.

Using \([7\.33](https://arxiv.org/html/2609.30661#S7.E33)\) and Lemma[8\.1](https://arxiv.org/html/2609.30661#S8.Thmlemma1), we have

∂f∂β¯i​\(𝐰i,𝐰i\)=‖𝐰i‖2​\[β¯i​erfc​\(β¯i2\)−2π​e−β¯i22\]∂f∂β¯i​\(𝐰i,𝐰j\)=‖𝐰i‖​‖𝐰j‖​β¯j2​π​\(π2​\[erfc​\(β¯i2\)\+erfc​\(β¯j2\)\]−Λ~i​j\)−‖𝐰i‖​‖𝐰j‖2​2​π​\[e−β¯j22​erfc​\(Zi​j2\)\+cos⁡\(Λi​j\)​e−β¯i22​erfc​\(Z^i​j2\)\]∂f∂β¯i​\(𝐰i,𝐯j\)=‖𝐰i‖​‖𝐯j‖​γ¯j2​π​\(π2​\[erfc​\(β¯i2\)\+erfc​\(γ¯j2\)\]−λ~i​j\)−‖𝐰i‖​‖𝐯j‖2​2​π​\[e−γ¯j22​erfc​\(zi​j2\)\+cos⁡\(λi​j\)​e−β¯i22​erfc​\(z^i​j2\)\]\\begin\{aligned\} \\frac\{\\partial f\}\{\\partial\\overline\{\\beta\}\_\{i\}\}\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{i\}\)&=\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|^\{2\}\\left\[\\overline\{\\beta\}\_\{i\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\\sqrt\{\\frac\{2\}\{\\pi\}\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\\right\]\\\\ \\frac\{\\partial f\}\{\\partial\\overline\{\\beta\}\_\{i\}\}\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{j\}\)&=\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\\overline\{\\beta\}\_\{j\}\}\{2\\pi\}\\left\(\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\{\\widetilde\{\\Lambda\}\}\_\{ij\}\\right\)\\\\ &\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\left\[e^\{\-\\frac\{\\overline\{\\beta\}\_\{j\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{Z\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\cos\(\\Lambda\_\{ij\}\)e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\{\\widehat\{Z\}\}\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ \\frac\{\\partial f\}\{\\partial\\overline\{\\beta\}\_\{i\}\}\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{j\}\)&=\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|\\overline\{\\gamma\}\_\{j\}\}\{2\\pi\}\\left\(\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\{\\widetilde\{\\lambda\}\}\_\{ij\}\\right\)\\\\ &\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\left\[e^\{\-\\frac\{\\overline\{\\gamma\}\_\{j\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{z\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\cos\(\\lambda\_\{ij\}\)e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\{\\widehat\{z\}\}\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\end\{aligned\}The result follows straightforwardly\. ∎

###### Remark 11\.2\.

Since∂f∂β=1‖𝐰‖​∂f∂β¯\\frac\{\\partial f\}\{\\partial\\beta\}=\\frac\{1\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\}\\frac\{\\partial f\}\{\\partial\\overline\{\\beta\}\}, it follows from \([9\.43](https://arxiv.org/html/2609.30661#S9.E43)\) that

\(11\.45\)∂f∂𝐰i~\\displaystyle\\widetilde\{\\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}^\{i\}\}\}=\\displaystyle=∂f∂𝐰i−β¯​𝐰¯i‖𝐰i‖​∂f∂β¯i\\displaystyle\\frac\{\\partial f\}\{\\partial\{\{\\mathbf\{w\}\}\}^\{i\}\}\-\\frac\{\\overline\{\\beta\}\\overline\{\{\{\\mathbf\{w\}\}\}\}^\{i\}\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\\frac\{\\partial f\}\{\\partial\\overline\{\\beta\}\_\{i\}\}\(11\.46\)∂f∂βj\\displaystyle\\frac\{\\partial f\}\{\\partial\\beta\_\{j\}\}=\\displaystyle=1‖𝐰j‖​∂f∂β¯j\\displaystyle\\frac\{1\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\}\\frac\{\\partial f\}\{\\partial\\overline\{\\beta\}\_\{j\}\}
Define the coordinate transformationCCofM⁡\(k,d\+1\)M\(k,d\+1\)by

C⁡\(𝐖,𝜷¯\)=\(𝐖,\(‖𝐰i‖​β¯i\)\)=\(𝐖,𝜷\),\(𝐖,𝜷¯\)∈M⁡\(k,d\+1\),C\(\{\{\\mathbf\{W\}\}\},\\boldsymbol\{\\overline\{\\beta\}\}\)=\(\{\{\\mathbf\{W\}\}\},\(\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\overline\{\\beta\}\_\{i\}\)\)=\(\{\{\\mathbf\{W\}\}\},\\boldsymbol\{\\beta\}\),\\;\(\{\{\\mathbf\{W\}\}\},\\boldsymbol\{\\overline\{\\beta\}\}\)\\in M\(k,d\+1\),where𝐖\{\{\\mathbf\{W\}\}\}is always assumed to have no zero rows\. Equations \([11\.45](https://arxiv.org/html/2609.30661#S11.E45),[11\.46](https://arxiv.org/html/2609.30661#S11.E46)\) imply that non\-degenerate critical points ofℒ\\mathcal\{L\}\(\(𝐖,𝜷¯\)\(\{\{\\mathbf\{W\}\}\},\\boldsymbol\{\\overline\{\\beta\}\}\)\-variables\) are mapped byCCto non\-degenerate critical points ofℒ~\\widetilde\{\\mathcal\{L\}\}\(\(𝐖,𝜷\)\(\{\{\\mathbf\{W\}\}\},\\boldsymbol\{\\beta\}\)\-variables\)\. The linearizations ofgrad​\(ℒ\)\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}andgrad​\(ℒ~\)\{\\text\{\\rm grad\}\(\{\\widetilde\{\\mathcal\{L\}\}\}\)\}at corresponding critical points are similar matrices and so have the same spectrum—this a standard result about linearizations of vector fields at a zero \(rather than the Hessian of a gradient vector field at a critical point\) and is appropriate here as we reduce to studying vector fields on fixed point spaces\. An important consequence is that it will be no loss of generality to work with\(𝐖,𝜷¯\)\(\{\{\\mathbf\{W\}\}\},\\boldsymbol\{\\overline\{\\beta\}\}\)\-variables rather than\(𝐖,𝜷\)\(\{\{\\mathbf\{W\}\}\},\\boldsymbol\{\\beta\}\)\-variables\. If we identify the fixed point space withℝn\{\\mathbb\{R\}\}^\{n\},*Euclidean inner product*, the induced vector fields onℝn\{\\mathbb\{R\}\}^\{n\}will not be gradient unless we either choose a non\-standard inner product onℝn\{\\mathbb\{R\}\}^\{n\}or we modify the identification \(best option\)—see Remarks[13\.1](https://arxiv.org/html/2609.30661#S13.Thmlemma1)\(2\)\.✠\\maltese

###### Remark 11\.3\(Discussion\)\.

Of special interest is the value of∂ℒ∂𝜷¯\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\boldsymbol\{\\overline\{\\beta\}\}\}when the bias vectors𝜷¯,𝜸¯\\boldsymbol\{\\overline\{\\beta\}\},\\boldsymbol\{\\overline\{\\gamma\}\}are zero\. The problem is to see how the critical point and gradient structure change when we add bias to the activation function\. In particular, what happens when the bias is zero in a biased network?888Abuse of language—a “biased network” is a “network with bias”, parameterized byM⁡\(k,d\+1\)M\(k,d\+1\)\.\.

For example, is a critical point of the bias free network999A network without bias variable, parameterized byM⁡\(k,d\)M\(k,d\)\., a critical point of the biased network, with bias set to zero? Is the space of bias free networks an invariant subspace ofM⁡\(k,d\+1\)M\(k,d\+1\)under gradient dynamics? If invariance holds, then a critical point𝐖∈M⁡\(k,d\)\{\{\\mathbf\{W\}\}\}\\in M\(k,d\)of the loss \(no bias\) necessarily defines a critical point\(𝐖,𝟎\)∈M⁡\(k,d\+1\)\(\{\{\\mathbf\{W\}\}\},\\mathbf\{0\}\)\\in M\(k,d\+1\)for the loss of the biased network\. Note that if𝐖∈M⁡\(k,d\)\{\{\\mathbf\{W\}\}\}\\in M\(k,d\)is a critical point of the bias free network, then\(𝐖,𝟎\)∈M⁡\(k,d\+1\)\(\{\{\\mathbf\{W\}\}\},\\mathbf\{0\}\)\\in M\(k,d\+1\)is a critical point for the loss of the biased network iff∂ℒ/∂𝜷¯​\(𝐖,𝟎\)=0\\partial\\mathcal\{L\}/\\partial\\boldsymbol\{\\overline\{\\beta\}\}\(\{\{\\mathbf\{W\}\}\},\\mathbf\{0\}\)=0\(a necessary condition for invariance\)\.

As it turns out, in*all*the examples we have investigated, if𝐖\{\{\\mathbf\{W\}\}\}is a non\-degenerate critical point ofℒ\\mathcal\{L\}, then\(𝐖,𝟎\)∈M⁡\(k,d\+1\)\(\{\{\\mathbf\{W\}\}\},\\mathbf\{0\}\)\\in M\(k,d\+1\)is*not*a critical point in the biased network\. In particular, the invariance property fails\. For details and examples see Section[14](https://arxiv.org/html/2609.30661#S14)\.

As a consequence the only tractable transition from critical point of a bias free network to the corresponding biased network would appear to be if*the derivatives∂ℒ∂β¯\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\}are non\-zero but ‘small’\.*Otherwise put, no drastic changes in the landscape geometry near the critical point\. That at least allows the possibility of a critical point of the biased network nearby\. As we show in Section[14](https://arxiv.org/html/2609.30661#S14), the derivatives∂ℒ∂β¯\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\}are small—asymptotically they decay to zero like a fractional power ofd−1d^\{\-1\}—and the derivatives for small values ofddare typically significantly smaller than suggested by the asymptotics\.

If there is a natural connection between spurious minima in corresponding bias free and biased networks, then one would expect that the loss at a spurious minimum in the biased network should be smaller than that in the bias free network\. Indeed, it would be counter intuitive if adding extra structure, in this case bias, resulted in degraded performance \(increased loss\)\. The point is illustrated with examples in Section[15](https://arxiv.org/html/2609.30661#S15)\.✠\\maltese

### 11\.1\.The𝜷¯\\boldsymbol\{\\overline\{\\beta\}\}\-derivative ofℒ\\mathcal\{L\}at𝜷¯=𝟎\\boldsymbol\{\\overline\{\\beta\}\}=\\mathbf\{0\}

###### Lemma 11\.4\.

\(Notation and assumptions as above\.\) Assume𝛄¯=𝟎\\boldsymbol\{\\overline\{\\gamma\}\}=\\mathbf\{0\}\. For alli∈𝐤i\\in\{\\mathbf\{k\}\},𝐖∈M⁡\(k,d\)\{\{\\mathbf\{W\}\}\}\\in M\(k,d\),

∂ℒ∂β¯i​\(𝐖,𝟎\)=‖𝐰i‖2​2​π​\[∑j∈𝐝‖𝐯j‖​\(1\+cos⁡\(λi​j\)\)−∑j∈𝐤‖𝐰j‖​\(1\+cos⁡\(Λi​j\)\)\]\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{i\}\}\(\{\{\\mathbf\{W\}\}\},\\mathbf\{0\}\)=\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\left\[\\sum\_\{j\\in\{\\mathbf\{d\}\}\}\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|\(1\+\\cos\(\\lambda\_\{ij\}\)\)\-\\sum\_\{j\\in\{\\mathbf\{k\}\}\}\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\(1\+\\cos\(\\Lambda\_\{ij\}\)\)\\right\]

###### Proof\.

Substitute𝜸¯=𝟎\\boldsymbol\{\\overline\{\\gamma\}\}=\\mathbf\{0\},𝜷¯=𝟎\\boldsymbol\{\\overline\{\\beta\}\}=\\mathbf\{0\}in the formula for theβ¯\\overline\{\\beta\}\-derivative ofℒ\\mathcal\{L\}\(Proposition[11\.1](https://arxiv.org/html/2609.30661#S11.Thmlemma1), The term−‖𝐰‖22​π\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}\\\|^\{2\}\}\{\\sqrt\{2\\pi\}\}corresponding to∂f∂β¯i​\(𝐰i,𝐰i\)\\frac\{\\partial f\}\{\\partial\\overline\{\\beta\}\_\{i\}\}\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{i\}\)\(see Examples[8\.3](https://arxiv.org/html/2609.30661#S8.Thmlemma3)\(1\)\) is accounted for sinceΛi​i=0\\Lambda\_\{ii\}=0and socos⁡\(Λi​i\)=1\\cos\(\\Lambda\_\{ii\}\)=1\. ∎

###### Remark 11\.5\.

The formula for∂ℒ∂β¯i​\(𝐖,𝟎\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{i\}\}\(\{\{\\mathbf\{W\}\}\},\\mathbf\{0\}\)can be written as a sum and difference of inner products of rows of𝐖\{\{\\mathbf\{W\}\}\}and𝐕\{\{\\mathbf\{V\}\}\}and a sum and difference of norms of rows\. This is trivial since the cosine of the angle between row𝐯\{\{\\mathbf\{v\}\}\}and𝐰\{\{\\mathbf\{w\}\}\}is⟨𝐰,𝐯⟩‖𝐰‖​‖𝐯‖\\frac\{\\langle\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\\rangle\}\{\\\|\{\{\\mathbf\{w\}\}\}\\\|\\\|\{\{\\mathbf\{v\}\}\}\\\|\}and the denominator is cleared since it is multiplied by‖𝐰‖\\\|\{\{\\mathbf\{w\}\}\}\\\|,‖𝐯‖\\\|\{\{\\mathbf\{v\}\}\}\\\|; similarly for angles between rows of𝐖\{\{\\mathbf\{W\}\}\}\. That is, the expression for∂ℒ∂β¯i​\(𝐖,𝟎\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{i\}\}\(\{\{\\mathbf\{W\}\}\},\\mathbf\{0\}\)is equal to

12​2​π​\(‖𝐰i‖​\[∑j∈𝐝‖𝐯j‖−∑j∈𝐤‖𝐰j‖\]\+\[∑j∈𝐝⟨𝐰i,𝐯j⟩−∑j∈𝐤⟨𝐰i,𝐰j⟩\]\)\\frac\{1\}\{2\\sqrt\{2\\pi\}\}\\left\(\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\big\[\\sum\_\{j\\in\{\\mathbf\{d\}\}\}\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|\-\\sum\_\{j\\in\{\\mathbf\{k\}\}\}\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\\big\]\+\\big\[\\sum\_\{j\\in\{\\mathbf\{d\}\}\}\\langle\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{j\}\\rangle\-\\sum\_\{j\\in\{\\mathbf\{k\}\}\}\\langle\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{j\}\\rangle\\big\]\\right\)Computationally this is better to work with than using the formula with angles since no division or angle computation is required\.✠\\maltese

## 12\.Symmetry andgrad​\(ℒ\)\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}in a network with bias

This section starts by reviewing terminology and notation for symmetry in shallow ReLU networks with a high symmetry target \(proofs for bias free networks can be found in\[[5](https://arxiv.org/html/2609.30661#bib.bib5), Appendix A\]and\[[2](https://arxiv.org/html/2609.30661#bib.bib2),[3](https://arxiv.org/html/2609.30661#bib.bib3)\]\)\. The remainder of the section is devoted to developing definitions and basic theory for symmetric biased networks; in particular, the critical point equations for a large class of symmetric biased networks\.

The mathematical background and overhead required for the analytic investigation of networks with a high symmetry target is significant\. However, the advantage of working in this setting is the ability to prove sharp analytical results which persist under symmetry breaking perturbation\. As one example, we mention computation of the Hessian spectrum at critical points for large \(very large\) values ofddand applications to networks with*asymmetric*targets which are not too far from the high symmetry target used in this section \(see Section[14](https://arxiv.org/html/2609.30661#S14)for examples\)\. However, in the present paper we do not present any detailed analysis of the Hessian spectrum \(for the basic theory see\[[3](https://arxiv.org/html/2609.30661#bib.bib3)\]and also\[[5](https://arxiv.org/html/2609.30661#bib.bib5)\]\)\.

Symmetry properties of the lossℒ\\mathcal\{L\}related to symmetries of the target were first described in\[[1](https://arxiv.org/html/2609.30661#bib.bib1)\]\(see also\[[4](https://arxiv.org/html/2609.30661#bib.bib4), §4\.3\]\)\. Generally we follow the notation previously described and developed from\[[5](https://arxiv.org/html/2609.30661#bib.bib5), Appendix A\]\. In particular, each row𝐰\{\{\\mathbf\{w\}\}\}of a matrix𝐖∈M⁡\(k,d\)\{\{\\mathbf\{W\}\}\}\\in M\(k,d\)is regarded as a linear functional \(row vector\) and no use is made of the transpose operator\. That is,𝐰∈\(ℝd\)⋆\{\{\\mathbf\{w\}\}\}\\in\(\{\\mathbb\{R\}\}^\{d\}\)^\{\\star\}and𝐰⁡\(𝐱\)=𝐰𝐱\{\{\\mathbf\{w\}\}\}\(\{\\mathbf\{x\}\}\)=\{\{\\mathbf\{w\}\}\}\{\\mathbf\{x\}\}\(matrix multiplication\) is a well\-defined real number for all𝐱∈ℝd\{\\mathbf\{x\}\}\\in\{\\mathbb\{R\}\}^\{d\}\. This is the natural approach in the context of group representation theory \(a key tool in the analysis of the Hessian\) and also avoids unnecessary notational complexity resulting from the use of the transpose\.

### 12\.1\.Symmetries of the lossℒ\\mathcal\{L\}in a biased network

Recall from Section 3 that if𝐖∈M⁡\(k,d\)\{\{\\mathbf\{W\}\}\}\\in M\(k,d\),𝐕∈M⁡\(d,d\)\{\{\\mathbf\{V\}\}\}\\in M\(d,d\)have no zero rows—a standing assumption in what follows—𝜸¯∈M⁡\(1,d\)\\boldsymbol\{\\overline\{\\gamma\}\}\\in M\(1,d\)and𝜷¯∈M⁡\(1,k\)\\boldsymbol\{\\overline\{\\beta\}\}\\in M\(1,k\)are \(normalized\) bias vectors, then

\(12\.47\)ℒ⁡\(𝐖,𝐕\)\\displaystyle\\hskip 28\.90755pt\\mathcal\{L\}\(\{\{\\mathbf\{W\}\}\},\{\{\\mathbf\{V\}\}\}\)=\\displaystyle=12​𝔼𝐱∈𝒩⁡\(0,Id\)​\(∑i∈𝐤‖𝐰i‖​σ​\(𝐰¯i​𝐱−β¯i\)−∑j∈𝐝‖𝐯j‖​σ​\(𝐯¯j​𝐱−γ¯j\)\)2\\displaystyle\\frac\{1\}\{2\}\\mathbb\{E\}\_\{\{\\mathbf\{x\}\}\\in\\mathcal\{N\}\(0,I\_\{d\}\)\}\\left\(\\sum\_\{i\\in\{\\mathbf\{k\}\}\}\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\sigma\(\\overline\{\{\{\\mathbf\{w\}\}\}\}^\{i\}\{\\mathbf\{x\}\}\-\\overline\{\\beta\}\_\{i\}\)\-\\sum\_\{j\\in\{\\mathbf\{d\}\}\}\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|\\sigma\(\\overline\{\{\{\\mathbf\{v\}\}\}\}^\{j\}\{\\mathbf\{x\}\}\-\\overline\{\\gamma\}\_\{j\}\)\\right\)^\{2\}\(12\.48\)=\\displaystyle=12​∑i,j∈𝐤f⁡\(𝐰i,𝐰j\)−∑i∈𝐤,j∈𝐝f⁡\(𝐰i,𝐯j\)\+12​∑i,j∈𝐝f⁡\(𝐯i,𝐯j\)\\displaystyle\\frac\{1\}\{2\}\\sum\_\{i,j\\in\{\\mathbf\{k\}\}\}f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{j\}\)\-\\sum\_\{i\\in\{\\mathbf\{k\}\},j\\in\{\\mathbf\{d\}\}\}f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{j\}\)\+\\frac\{1\}\{2\}\\sum\_\{i,j\\in\{\\mathbf\{d\}\}\}f\(\{\{\\mathbf\{v\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{j\}\)
wheref⁡\(𝐰i,𝐯j\)=‖𝐰i‖​‖𝐯j‖​𝔼𝐱∈𝒩⁡\(0,Id\)​\(σ⁡\(𝐰¯i​𝐱−β¯i\)​σ​\(𝐯¯j​𝐱−γ¯j\)\)f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{j\}\)=\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|\\mathbb\{E\}\_\{\{\\mathbf\{x\}\}\\in\\mathcal\{N\}\(0,I\_\{d\}\)\}\\big\(\\sigma\(\\overline\{\{\{\\mathbf\{w\}\}\}\}^\{i\}\{\\mathbf\{x\}\}\-\\overline\{\\beta\}\_\{i\}\)\\sigma\(\\overline\{\{\{\\mathbf\{v\}\}\}\}^\{j\}\{\\mathbf\{x\}\}\-\\overline\{\\gamma\}^\{j\}\)\\big\)with similar expressions for thef⁡\(𝐰i,𝐰j\)f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{j\}\)andf⁡\(𝐯i,𝐯j\)f\(\{\{\\mathbf\{v\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{j\}\)terms \(for explicit formulas, see Section[6](https://arxiv.org/html/2609.30661#S6)\)\.

When the target\(𝐕,𝜸¯\)\(\{\{\\mathbf\{V\}\}\},\\boldsymbol\{\\overline\{\\gamma\}\}\)is fixed, we regardℒ\\mathcal\{L\}as a function of\(𝐖,𝜷\)\(\{\{\\mathbf\{W\}\}\},\\boldsymbol\{\\beta\}\)and writeℒ⁡\(𝐖\)\\mathcal\{L\}\(\{\{\\mathbf\{W\}\}\}\)rather thanℒ⁡\(𝐖,𝐕\)\\mathcal\{L\}\(\{\{\\mathbf\{W\}\}\},\{\{\\mathbf\{V\}\}\}\)\.

Note that\(𝐖,𝜷¯\)∈M⁡\(k,d\+1\)\(\{\{\\mathbf\{W\}\}\},\\boldsymbol\{\\overline\{\\beta\}\}\)\\in M\(k,d\+1\),\(𝐕,𝜸¯\)∈M⁡\(d,d\+1\)\(\{\{\\mathbf\{V\}\}\},\\boldsymbol\{\\overline\{\\gamma\}\}\)\\in M\(d,d\+1\)\. The natural action of the symmetric groupSdS\_\{d\}onM⁡\(k,d\)M\(k,d\)permuting columns extends to anSdS\_\{d\}\-action onM⁡\(k,d\+1\)M\(k,d\+1\)which fixes the last column𝜷¯\\boldsymbol\{\\overline\{\\beta\}\}\. Similarly for the column actionSdS\_\{d\}onM⁡\(d,d\)M\(d,d\)andM⁡\(d,d\+1\)M\(d,d\+1\)and𝜸¯\\boldsymbol\{\\overline\{\\gamma\}\}\. We sometimes writeSdcS\_\{d\}^\{c\}to emphasize the action permutes columns\. The naturalSkS\_\{k\}action permuting rows ofM⁡\(k,d\)M\(k,d\)extends to anSkS\_\{k\}action onM⁡\(k,d\+1\)M\(k,d\+1\)which permutes the entries ofOPEN𝜷¯\)\\boldsymbol\{\\overline\{\\beta\}\}\)\. Similarly for theSkS\_\{k\}actions permuting rows ofM⁡\(k,d\)M\(k,d\)andM⁡\(k,d\+1\)M\(k,d\+1\)\. Here we sometimes writeSkrS\_\{k\}^\{r\}orSdrS\_\{d\}^\{r\}to emphasize the action permutes rows\.

The orthogonal groupO⁡\(d\)O\(d\)acts on the acts on the right ofM⁡\(k,d\)M\(k,d\)andM⁡\(d,d\)M\(d,d\):

g​𝐖=𝐖∘g−1,𝐖∈M⁡\(k,d\),g∈O​\(d\)\.g\{\{\\mathbf\{W\}\}\}=\{\{\\mathbf\{W\}\}\}\\circ g^\{\-1\},\\;\{\{\\mathbf\{W\}\}\}\\in M\(k,d\),g\\in\\text\{O\}\(d\)\.In particular, since the group ofd×dd\\times dpermutation matrices is a subgroup ofO​\(d\)\\text\{O\}\(d\), we can represent the action ofSdcS^\{c\}\_\{d\}onM⁡\(k,d\)M\(k,d\)andM⁡\(d,d\)M\(d,d\)as an orthogonal action by the group of permutation matrices\. The orthogonal action byO​\(d\)\\text\{O\}\(d\)onM⁡\(k,d\)M\(k,d\)extends in the obvious way to the orthogonal action byO​\(d\)⊂O​\(d\+1\)\\text\{O\}\(d\)\\subset\\text\{O\}\(d\+1\)onM⁡\(k,d\+1\)M\(k,d\+1\)that fixesOPEN𝜷\)\\boldsymbol\{\\beta\}\)\. Similar remarks hold for theO​\(d\)\\text\{O\}\(d\)actions onM⁡\(d,d\)M\(d,d\),M⁡\(d,d\+1\)M\(d,d\+1\)\.

Since the Gaussian distribution onℝd\{\\mathbb\{R\}\}^\{d\}is invariant by the orthogonal groupO⁡\(d\)O\(d\),ℒ\\mathcal\{L\}isO⁡\(d\)O\(d\)\-invariant:

ℒ⁡\(g​𝐖,g​𝐕\)=ℒ⁡\(𝐖,𝐕\),g∈O⁡\(d\),\\mathcal\{L\}\(g\{\{\\mathbf\{W\}\}\},g\{\{\\mathbf\{V\}\}\}\)=\\mathcal\{L\}\(\{\{\\mathbf\{W\}\}\},\{\{\\mathbf\{V\}\}\}\),\\;g\\in O\(d\),whereO⁡\(d\)O\(d\)acts on the right as defined above\. Henceℒ\\mathcal\{L\}isSdcS^\{c\}\_\{d\}\-invariant*as a function of\(𝐖,𝐕\)\(\{\{\\mathbf\{W\}\}\},\{\{\\mathbf\{V\}\}\}\)*\.

Trivially,ℒ⁡\(𝐖,𝐕\)\\mathcal\{L\}\(\{\{\\mathbf\{W\}\}\},\{\{\\mathbf\{V\}\}\}\)is invariant by the action ofSkr×SdrS\_\{k\}^\{r\}\\times S\_\{d\}^\{r\}onM⁡\(k,d\)×M⁡\(d,d\)M\(k,d\)\\times M\(d,d\): the finite sums in \([12\.47](https://arxiv.org/html/2609.30661#S12.E47)\) are independent of the order of summation of the terms\. In particularℒ⁡\(𝐖\)\\mathcal\{L\}\(\{\{\\mathbf\{W\}\}\}\)isSkrS\_\{k\}^\{r\}\-invariant\. Althoughℒ⁡\(𝐖,𝐕\)\\mathcal\{L\}\(\{\{\\mathbf\{W\}\}\},\{\{\\mathbf\{V\}\}\}\)is invariant by the action ofSdcS^\{c\}\_\{d\},ℒ⁡\(𝐖\)\\mathcal\{L\}\(\{\{\\mathbf\{W\}\}\}\)will not be invariant under the action of a non\-trivial column symmetry unless, for example, a column symmetry of𝐕\{\{\\mathbf\{V\}\}\}can be represented as a row symmetry\. If𝐕=Id\{\{\\mathbf\{V\}\}\}=I\_\{d\}, then this is so for every column symmetry since if the transposition\(a,b\)∈Sd\(a,b\)\\in S\_\{d\},a≠ba\\neq b, then\(a,b\)c​Id=\(a,b\)r​Id\(a,b\)^\{c\}I\_\{d\}=\(a,b\)^\{r\}I\_\{d\}\. Making use of the right action byO⁡\(d\)O\(d\)it follows that if𝐕=Id\{\{\\mathbf\{V\}\}\}=I\_\{d\}, thenℒ:M⁡\(k,d\)→ℝ\\mathcal\{L\}:M\(k,d\)\{\\rightarrow\}\{\\mathbb\{R\}\}isSkr×SdcS\_\{k\}^\{r\}\\times S\_\{d\}^\{c\}\-invariant\. Consequently,grad​\(ℒ\):M⁡\(k,d\)→M⁡\(k,d\)\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}:M\(k,d\)\{\\rightarrow\}M\(k,d\)isSkr×SdcS\_\{k\}^\{r\}\\times S\_\{d\}^\{c\}\-equivariant:

grad​\(ℒ\)​\(g​𝐖\)=g​grad​\(ℒ\)​\(𝐔\),for all​g∈Skr×Sdc,𝐖∈M⁡\(k,d\)\.\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}\(g\{\{\\mathbf\{W\}\}\}\)=g\\,\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}\(\{\{\\mathbf\{U\}\}\}\),\\;\\text\{for all \}g\\in S\_\{k\}^\{r\}\\times S\_\{d\}^\{c\},\\;\{\{\\mathbf\{W\}\}\}\\in M\(k,d\)\.\(The gradient is defined and continuous provided𝐖\{\{\\mathbf\{W\}\}\}has no zero rows\[[4](https://arxiv.org/html/2609.30661#bib.bib4), §4\.4\]\.\) If the network is biased and the gradient is taken with respect to𝐰\{\{\\mathbf\{w\}\}\}and𝜷¯\\boldsymbol\{\\overline\{\\beta\}\}variables, then the gradient is equivariant with respect to theSkr×SdcS\_\{k\}^\{r\}\\times S\_\{d\}^\{c\}\-action onM⁡\(k,d\+1\)M\(k,d\+1\)\(the biases for𝐕=Id\{\{\\mathbf\{V\}\}\}=I\_\{d\}must all be equal for this to be true\)\.

### 12\.2\.Isotropy groups of critical points when𝐕=Id\{\{\\mathbf\{V\}\}\}=I\_\{d\}

Assume that𝐕=Id∈M⁡\(d,d\)\{\{\\mathbf\{V\}\}\}=I\_\{d\}\\in M\(d,d\)and that𝐕\{\{\\mathbf\{V\}\}\}is augmented by a column𝜸¯\\boldsymbol\{\\overline\{\\gamma\}\}of identical constantsγ¯0\\overline\{\\gamma\}\_\{0\}if the network is biased,

SetGk,d=G=Skr×SdcG\_\{k,d\}=G=S^\{r\}\_\{k\}\\times S^\{c\}\_\{d\}\. If𝐖∈M⁡\(k,d\)\{\{\\mathbf\{W\}\}\}\\in M\(k,d\), respectively\(𝐖,𝜷¯\)∈M⁡\(k,d\+1\)\(\{\{\\mathbf\{W\}\}\},\\boldsymbol\{\\overline\{\\beta\}\}\)\\in M\(k,d\+1\), define the*isotropy subgroup*G𝐖G\_\{\{\\mathbf\{W\}\}\}for the action ofGGat𝐖\{\{\\mathbf\{W\}\}\}, respectively\(𝐖,𝜷¯\)\(\{\{\\mathbf\{W\}\}\},\\boldsymbol\{\\overline\{\\beta\}\}\), by

G𝐖=\{g∈G\|g​𝐖=𝐖\},resp\.​\{g∈G\|g⁡\(𝐖,𝜷¯\)=\(𝐖,𝜷¯\)\}G\_\{\{\\mathbf\{W\}\}\}=\\\{g\\in G\\mbox\{$\\;\|\\;$\}g\{\{\\mathbf\{W\}\}\}=\{\{\\mathbf\{W\}\}\}\\\},\\;\\text\{resp\.\}\\;\\\{g\\in G\\mbox\{$\\;\|\\;$\}g\(\{\{\\mathbf\{W\}\}\},\\boldsymbol\{\\overline\{\\beta\}\}\)=\(\{\{\\mathbf\{W\}\}\},\\boldsymbol\{\\overline\{\\beta\}\}\)\\\}The groupG𝐖G\_\{\{\\mathbf\{W\}\}\}measures the symmetry of the point𝐖\{\{\\mathbf\{W\}\}\}with respect to the action ofG=Skr×SdcG=S^\{r\}\_\{k\}\\times S^\{c\}\_\{d\}and is an important invariant of critical points of the loss when𝐕=Id\{\{\\mathbf\{V\}\}\}=I\_\{d\}\. With𝐕∈M⁡\(d,d\)\{\{\\mathbf\{V\}\}\}\\in M\(d,d\),

G𝐕=Δ​Sd=def\{\(g,g\)\|g∈Sd\}⊂Sdr×SdcG\_\{\{\\mathbf\{V\}\}\}=\\Delta S\_\{d\}\\stackrel\{\{\\scriptstyle\\mathrm\{def\}\}\}\{\{=\}\}\\\{\(g,g\)\\mbox\{$\\;\|\\;$\}g\\in S\_\{d\}\\\}\\subset S^\{r\}\_\{d\}\\times S^\{c\}\_\{d\}andG𝐕G\_\{\{\\mathbf\{V\}\}\}is maximal \(up to conjugacy\) for isotropy subgroups of points𝐖∈M⁡\(d,d\)\{\{\\mathbf\{W\}\}\}\\in M\(d,d\)provided that neither two rows of𝐖\{\{\\mathbf\{W\}\}\}nor two columns of𝐖\{\{\\mathbf\{W\}\}\}are equal \(for example, thed×dd\\times dmatrixℐd,d\\mathcal\{I\}\_\{d,d\}all of whose entries are11has isotropyGd,dG\_\{d,d\}\)\. Note that under our conventions for non parallel rows,𝐕\{\{\\mathbf\{V\}\}\}is not defined as an element ofM⁡\(k,d\)M\(k,d\)ifk\>dk\>d\. However,Δ​Sd\\Delta S\_\{d\}is naturally embedded as a subgroup ofSkr×SdcS^\{r\}\_\{k\}\\times S^\{c\}\_\{d\}fork\>dk\>d\(that is, by embeddingSdrS^\{r\}\_\{d\}as the subgroup ofSkrS^\{r\}\_\{k\}permuting the firstddrows\.

###### Remark 12\.1\.

Supposek=dk=dand𝐖\{\{\\mathbf\{W\}\}\}is a non\-degenerate critical point\. If a single neuron is added to the network then typically𝐖\{\{\\mathbf\{W\}\}\}will be fossilized in the new parameter space \(see Remarks[10\.1](https://arxiv.org/html/2609.30661#S10.Thmlemma1)\(3\)\) as a 1\-dimensional simplex of degenerate critical points and𝐖\{\{\\mathbf\{W\}\}\}will be replaced by a new non\-degenerate critical point inM⁡\(d\+1,d\)M\(d\+1,d\)\(this can be path connected back to the original point by deforming the gradient fork=d\+1k=d\+1back tok=dk=d\)\. Similar results hold for allk\>dk\>d\. However, this is not the case for a critical point giving the global minimum zero of the loss; this is replaced by a 1\-dimensional connected simplicial complex each point of which defines the global minimum and no new non\-degenerate critical points defining the global minimum are created\.✠\\maltese

Our interest here lies indd\-dependent families of critical points ofgrad​\(ℒ\)\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}which have isotropy a subgroup ofΔ​Sd\\Delta S\_\{d\}\(strictly, conjugate to a subgroup ofΔ​Sd\\Delta S\_\{d\}\)\.

For any subgroupHHofGG, define the*fixed point space*for the action ofHHonM⁡\(k,d\)M\(k,d\)by

M\(k,d\)H=\{𝐖\|h𝐖=𝐖,for allh∈H\}⊂M\(k,d\)\.M\(k,d\)^\{H\}=\\\{\{\{\\mathbf\{W\}\}\}\\mbox\{$\\;\|\\;$\}h\{\{\\mathbf\{W\}\}\}=\{\{\\mathbf\{W\}\}\},\\;\\text\{for all \}h\\in H\\\}\\subset M\(k,d\)\.The fixed point spaceM​\(k,d\)HM\(k,d\)^\{H\}is a vector subspace ofM⁡\(k,d\)M\(k,d\)which is invariant under the flow of any equivariant vector field onM⁡\(k,d\)M\(k,d\)\. More precisely, wherever the vector field is defined on on the fixed point spaceM⁡\(k,d\)M\(k,d\), it will be tangent toM​\(k,d\)HM\(k,d\)^\{H\}\. Consequently, gradient descent forgrad​\(ℒ\)\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}which is initialized onM​\(k,d\)HM\(k,d\)^\{H\}stays onM​\(k,d\)HM\(k,d\)^\{H\}\.

We similarly define the fixed point spaceM​\(k,d\+1\)HM\(k,d\+1\)^\{H\}for biased networks\. All the properties stated above forM​\(k,d\)HM\(k,d\)^\{H\}continue to hold forM​\(k,d\+1\)HM\(k,d\+1\)^\{H\}\.

The invariance of the fixed point space by gradient dynamics is crucial: in the search for critical points, it allows the search to start on the typically low dimensional fixed point space\.

### 12\.3\.Families of subgroups ofΔ​Sd⊂Skr×Sdc\\Delta S\_\{d\}\\subset S^\{r\}\_\{k\}\\times S^\{c\}\_\{d\}

Givenf∈ℕf\\in\\mbox\{$\\mathbb\{N\}$\}, supposeqi∈ℕq\_\{i\}\\in\\mbox\{$\\mathbb\{N\}$\},i∈\[f−1\]i\\in\[f\-1\]and thatq1≥q2≥…≥qf−1≥1q\_\{1\}\\geq q\_\{2\}\\geq\\ldots\\geq q\_\{f\-1\}\\geq 1\(the decreasing assumption is not strictly necessary but it makes the exposition simpler and can always be obtained by an appropriate permutation of rows and columns\)\. Givend\>∑i∈\[f−1\]qid\>\\sum\_\{i\\in\[f\-1\]\}q\_\{i\}, setq0=d−∑i∈\[f−1\]qiq\_\{0\}=d\-\\sum\_\{i\\in\[f\-1\]\}q\_\{i\}\. Choosed0∈ℕd\_\{0\}\\in\\mbox\{$\\mathbb\{N\}$\}such thatq0≥q1q\_\{0\}\\geq q\_\{1\}, alld≥d0d\\geq d\_\{0\}—in particular,q1,…,qf−1q\_\{1\},\\ldots,q\_\{f\-1\}will be constants andq0q\_\{0\}depends onddandq1,…,qf−1q\_\{1\},\\ldots,q\_\{f\-1\}\.

Note that if1=q0≥q1≥…≥qf−1≥11=q\_\{0\}\\geq q\_\{1\}\\geq\\ldots\\geq q\_\{f\-1\}\\geq 1, then the isotropy is trivial—the*asymmetric case*\. Although we give examples of asymmetry later, we usually assume thatq0\>1q\_\{0\}\>1and so there will be non\-trivial symmetries\.

Givenf∈ℕf\\in\\mbox\{$\\mathbb\{N\}$\},q0,…,qf−1≥1q\_\{0\},\\ldots,q\_\{f\-1\}\\geq 1define

\(12\.49\)G𝐪Δ=Gq0,⋯,qf−1Δ=Δ⁡\(∏i∈𝐟Sqi\)⊂Δ​Sd,G^\{\\Delta\}\_\{\\mathbf\{q\}\}=G^\{\\Delta\}\_\{q\_\{0\},\\cdots,q\_\{f\-1\}\}=\\Delta\\left\(\\prod\_\{i\\in\{\\mathbf\{f\}\}\}S\_\{q\_\{i\}\}\\right\)\\subset\\Delta S\_\{d\},and regardG𝐪ΔG^\{\\Delta\}\_\{\\mathbf\{q\}\}as acting onM⁡\(k,d\),k≥dM\(k,d\),k\\geq d\.

###### Remarks 12\.2\.

\(1\) If we assume thatq1\>…\>qf−1≥1q\_\{1\}\>\\ldots\>q\_\{f\-1\}\\geq 1andd0d\_\{0\}is chosen sufficiently large so thatq0\>q1q\_\{0\}\>q\_\{1\}, the row order is uniquely determined up to rowdd\(see \(4\) below\)\. \(2\) Iff=2f=2, andq0≠q1q\_\{0\}\\neq q\_\{1\}, thenG𝐪ΔG^\{\\Delta\}\_\{\\mathbf\{q\}\}is a maximal proper subgroup ofΔ​Sd\\Delta S\_\{d\}\. A necessary condition for maximality isq0≠q1q\_\{0\}\\neq q\_\{1\}\(see\[[4](https://arxiv.org/html/2609.30661#bib.bib4), §5\.3\]for precise statements and references\)\. Ifq0=q1\>1q\_\{0\}=q\_\{1\}\>1, then the maximal proper subgroup is the wreath productC2≀SpC\_\{2\}\\wr S\_\{p\}, whereC2≈S2C\_\{2\}\\approx S\_\{2\}is the cyclic group of order22andp=q0=q1p=q\_\{0\}=q\_\{1\}\. If there are more than two factors,G𝐪ΔG^\{\\Delta\}\_\{\\mathbf\{q\}\}is not a maximal proper subgroup\. It is possible to have non\-degenerate critical points𝐖∈M⁡\(2​d,2​d\)\{\{\\mathbf\{W\}\}\}\\in M\(2d,2d\)such thatq0=q1=dq\_\{0\}=q\_\{1\}=d,G𝐖=G𝐪ΔG\_\{\{\\mathbf\{W\}\}\}=G^\{\\Delta\}\_\{\\mathbf\{q\}\}, the firstddrows of𝐖\{\{\\mathbf\{W\}\}\}are not parallel to any of the remainingddrows and there is no symmetry relation between the two sets of rows\. \(3\) If𝐖\{\{\\mathbf\{W\}\}\}is a critical point ofgrad​\(ℒ\)\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}with isotropyG𝐪ΔG^\{\\Delta\}\_\{\\mathbf\{q\}\}then, by theGG\-equivariance ofgrad​\(ℒ\)\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}, there are\|G\|/\|G𝐪Δ\|=k\!​d\!/\(q0\!​…​qf−1\!\)\|G\|/\|G^\{\\Delta\}\_\{\\mathbf\{q\}\}\|=k\!d\!/\(q\_\{0\}\!\\ldots q\_\{f\-1\}\!\)critical points on theΓ\\Gamma\-orbit of𝐖\{\{\\mathbf\{W\}\}\}\. \(4\) Ifk\>d\+1k\>d\+1andqf−1≥2q\_\{f\-1\}\\geq 2, setp=k−dp=k\-d, and ifk≥d\+1k\\geq d\+1andq2\>q1=1q\_\{2\}\>q\_\{1\}=1, setp=k−d−1p=k\-d\-1\. Noting the standing assumption of no parallel rows, ifG𝐖=G𝐪ΔG\_\{\{\\mathbf\{W\}\}\}=G^\{\\Delta\}\_\{\\mathbf\{q\}\}, then there is a free action bySprS^\{r\}\_\{p\}on the lastpprows of𝐖\{\{\\mathbf\{W\}\}\}\(and so onM​\(k,d\)G𝐪ΔM\(k,d\)^\{G^\{\\Delta\}\_\{\\mathbf\{q\}\}\}\)\. \(5\) Ifqf−Ra=qf−Ra\+1=…=qf−1=1q\_\{f\-R\_\{a\}\}=q\_\{f\-R\_\{a\}\+1\}=\\ldots=q\_\{f\-1\}=1andqi\>1q\_\{i\}\>1,i<f−Rai<f\-R\_\{a\}, there is anRa×RaR\_\{a\}\\times R\_\{a\}\-diagonal asymmetric block in𝐖\{\{\\mathbf\{W\}\}\}\. Here it is natural to setG𝐖=Δ⁡\(∏i∈𝐟−𝐑𝐚Sqi\)⊂Δ​Sd−a⊂Δ​SdG\_\{\{\\mathbf\{W\}\}\}=\\Delta\\left\(\\prod\_\{i\\in\{\\mathbf\{f\-R\_\{a\}\}\}\}S\_\{q\_\{i\}\}\\right\)\\subset\\Delta S\_\{d\-a\}\\subset\\Delta S\_\{d\}\.✠\\maltese

#### Review of notation and conventions

Assumef∈ℕf\\in\\mbox\{$\\mathbb\{N\}$\}and we are given integersq1≥q2≥…≥qf−1≥1q\_\{1\}\\geq q\_\{2\}\\geq\\ldots\\geq q\_\{f\-1\}\\geq 1\. Choosed0∈ℕd\_\{0\}\\in\\mbox\{$\\mathbb\{N\}$\}so thatq0=d−∑i∈\[f−1\]qi≥2q\_\{0\}=d\-\\sum\_\{i\\in\[f\-1\]\}q\_\{i\}\\geq 2for alld≥d0d\\geq d\_\{0\}\. LetG𝐪Δ⊂Δ​SdG^\{\\Delta\}\_\{\\mathbf\{q\}\}\\subset\\Delta S\_\{d\}be defined as in \([12\.49](https://arxiv.org/html/2609.30661#S12.E49)\)\. Setm=k−dm=k\-dand assume thatmmis constant ford≥d0d\\geq d\_\{0\}\.

SetR=f\+mR=f\+m—the number of*rowtypes*: one for each factor inG𝐪ΔG^\{\\Delta\}\_\{\\mathbf\{q\}\}and one for each additional row\. LetRsR\_\{s\}be the number of values ofi∈𝐟i\\in\{\\mathbf\{f\}\}for whichqi\>1q\_\{i\}\>1\. Sinceq0≥2q\_\{0\}\\geq 2,Rs∈\[f\]R\_\{s\}\\in\[f\]\. Let𝐫𝐬=\{i∈𝐟\|qi\>1\}\{\{\\mathbf\{r\_\{s\}\}\}\}=\\\{i\\in\{\\mathbf\{f\}\}\\mbox\{$\\;\|\\;$\}q\_\{i\}\>1\\\}\. Since we assume\(qi\)i∈𝐟\(q\_\{i\}\)\_\{i\\in\{\\mathbf\{f\}\}\}is a decreasing sequence,𝐫𝐬=𝐑𝐬\{\{\\mathbf\{r\_\{s\}\}\}\}=\{\\mathbf\{R\_\{s\}\}\}\. Let𝐫𝐚=\{i∈𝐟\|qi=1\}\{\{\\mathbf\{r\_\{a\}\}\}\}=\\\{i\\in\{\\mathbf\{f\}\}\\mbox\{$\\;\|\\;$\}q\_\{i\}=1\\\}and letRa=f−RsR\_\{a\}=f\-R\_\{s\}count the number of factors withqi=1q\_\{i\}=1\. Obviously,𝐫𝐚∪𝐫𝐬=𝐟\{\{\\mathbf\{r\_\{a\}\}\}\}\\cup\{\{\\mathbf\{r\_\{s\}\}\}\}=\{\\mathbf\{f\}\}and𝐫𝐚=𝐟∖𝐫𝐬\{\{\\mathbf\{r\_\{a\}\}\}\}=\{\\mathbf\{f\}\}\\smallsetminus\{\{\\mathbf\{r\_\{s\}\}\}\}\. The rowtypes in𝐫𝐚\{\{\\mathbf\{r\_\{a\}\}\}\}define the asymmetric diagonal block\. Finally, define𝐫𝐦=\{i∈𝐑\|i≥f\}\{\{\\mathbf\{r\_\{m\}\}\}\}=\\\{i\\in\{\\mathbf\{R\}\}\\mbox\{$\\;\|\\;$\}i\\geq f\\\}; that is, the rowtypes associated to neurons with rowtype at leastff\.

Recall from Section[7](https://arxiv.org/html/2609.30661#S7)that we label rows of𝐖∈M⁡\(k,d\)\{\{\\mathbf\{W\}\}\}\\in M\(k,d\)from00tok−1k\-1rather than from11tokk\. That is, fori∈𝐤i\\in\{\\mathbf\{k\}\},𝐰i\{\{\\mathbf\{w\}\}\}^\{i\}is rowiiof𝐖\{\{\\mathbf\{W\}\}\}\. We have a similar convention for rowtypes\. For example, if𝐖∈M​\(d,d\)G𝐪Δ\{\{\\mathbf\{W\}\}\}\\in M\(d,d\)^\{G^\{\\Delta\}\_\{\{\\mathbf\{q\}\}\}\}\(orM​\(d,d\+1\)G𝐪ΔM\(d,d\+1\)^\{G^\{\\Delta\}\_\{\{\\mathbf\{q\}\}\}\}if the network is biased\), there is 1 rowtype00and the first row of𝐖\{\{\\mathbf\{W\}\}\}is𝐰0\{\{\\mathbf\{w\}\}\}^\{0\}\. In general,𝐰i\{\{\\mathbf\{w\}\}\}^\{i\}typically represents the first𝐖\{\{\\mathbf\{W\}\}\}row of rowtypeii; that is𝐰I⁡\(i\)\{\{\\mathbf\{w\}\}\}^\{I\(i\)\}\. The context will always make the meaning clear\.

Ifdim​\(M​\(k,d\)G𝐪Δ\)=N\\text\{dim\}\(M\(k,d\)^\{G^\{\\Delta\}\_\{\{\\mathbf\{q\}\}\}\}\)=N\(unbiased\), then

dim​\(M​\(k,d\+1\)Δ𝐪\)=N\+R,biased\\text\{dim\}\\left\(M\(k,d\+1\)^\{\\Delta\_\{\\mathbf\{q\}\}\}\\right\)=N\+R,\\;\\;\\text\{biased\}
###### Remark 12\.3\.

The convention of indexing from00rather than11is made to reduce confusion and error when coding arrays\. If the reader prefers,𝐝\{\\mathbf\{d\}\}can be interpreted as\[d\]\[d\]; this results in few changes to the text\.✠\\maltese

###### Lemma 12\.4\(\[[5](https://arxiv.org/html/2609.30661#bib.bib5), Appendix A\]& below\)\.

\(Notation and assumptions as above\.\) Ford≥d0d\\geq d\_\{0\},

dim​\(M​\(k,d\)G𝐪Δ\)\\displaystyle\\text\{dim\}\\big\(M\(k,d\)^\{G^\{\\Delta\}\_\{\\mathbf\{q\}\}\}\\big\)=\\displaystyle=f⁡\(f\+m\)\+Rs\\displaystyle f\(f\+m\)\+R\_\{s\}dim​\(M​\(k,d\+1\)G𝐪Δ\)\\displaystyle\\text\{dim\}\\big\(M\(k,d\+1\)^\{G^\{\\Delta\}\_\{\\mathbf\{q\}\}\}\\big\)=\\displaystyle=\(f\+1\)​\(f\+m\)\+Rs\\displaystyle\(f\+1\)\(f\+m\)\+R\_\{s\}In particular,

1. \(1\)Iff=1f=1, then dim​\(M​\(k,d\)G𝐪Δ\)\\displaystyle\\text\{dim\}\\big\(M\(k,d\)^\{G^\{\\Delta\}\_\{\\mathbf\{q\}\}\}\\big\)=\\displaystyle=2\+m​\(unbiased\)\\displaystyle 2\+m\\;\\text\{\\rm\(unbiased\)\}dim​\(M​\(k,d\+1\)G𝐪Δ\)\\displaystyle\\text\{dim\}\\big\(M\(k,d\+1\)^\{G^\{\\Delta\}\_\{\\mathbf\{q\}\}\}\\big\)=\\displaystyle=3\+2​m​\(biased\)\\displaystyle 3\+2m\\;\\text\{\\rm\(biased\)\}
2. \(2\)Iff=2f=2andq1=1q\_\{1\}=1, then dim​\(M​\(k,d\)G𝐪Δ\)\\displaystyle\\text\{dim\}\\big\(M\(k,d\)^\{G^\{\\Delta\}\_\{\\mathbf\{q\}\}\}\\big\)=\\displaystyle=5\+2​m​\(unbiased\)\\displaystyle 5\+2m\\;\\text\{\\rm\(unbiased\)\}dim​\(M​\(k,d\+1\)G𝐪Δ\)\\displaystyle\\text\{dim\}\\big\(M\(k,d\+1\)^\{G^\{\\Delta\}\_\{\\mathbf\{q\}\}\}\\big\)=\\displaystyle=7\+3​m​\(biased\)\\displaystyle 7\+3m\\;\\text\{\\rm\(biased\)\}
3. \(3\)Iff=2f=2andq1\>1q\_\{1\}\>1, then dim​\(M​\(k,d\)G𝐪Δ\)\\displaystyle\\text\{dim\}\\big\(M\(k,d\)^\{G^\{\\Delta\}\_\{\\mathbf\{q\}\}\}\\big\)=\\displaystyle=6\+2​m​\(unbiased\)\\displaystyle 6\+2m\\;\\text\{\\rm\(unbiased\)\}dim​\(M​\(k,d\+1\)G𝐪Δ\)\\displaystyle\\text\{dim\}\\big\(M\(k,d\+1\)^\{G^\{\\Delta\}\_\{\\mathbf\{q\}\}\}\\big\)=\\displaystyle=8\+3​m​\(biased\)\\displaystyle 8\+3m\\;\\text\{\\rm\(biased\)\}

#### Sketch of the proof of Lemma[12\.4](https://arxiv.org/html/2609.30661#S12.Thmlemma4)

Start by assuming bias free\. Observe thatSmr⊂SkrS\_\{m\}^\{r\}\\subset S\_\{k\}^\{r\}acts \(freely\) on the lastmmrows ofM⁡\(k,d\)M\(k,d\)and so acts on the fixed point space and does not enter into the computation of the dimension\. Suppose thatk=dk=d\. We have a natural block decomposition ofℬ=\{Bi​j\|i∈𝐑,j∈𝐟\}\\mathcal\{B\}=\\\{\{B\}\_\{ij\}\\mbox\{$\\;\|\\;$\}i\\in\{\\mathbf\{R\}\},j\\in\{\\mathbf\{f\}\}\\\}ofM⁡\(d,d\)M\(d,d\)which is preserved by the action ofG𝐪ΔG^\{\\Delta\}\_\{\\mathbf\{q\}\}\. Specifically, the diagonal blocksBi​iB\_\{ii\}will be of sizeqi×qiq\_\{i\}\\times q\_\{i\}and invariant byΔ​Sqi⊂G𝐪Δ×Smr\\Delta S\_\{q\_\{i\}\}\\subset G^\{\\Delta\}\_\{\\mathbf\{q\}\}\\times S^\{r\}\_\{m\}\. The off diagonal blocksBi​jB\_\{ij\},i≠ji\\neq j, will be invariant bySqir×SqjcS\_\{q\_\{i\}\}^\{r\}\\times S^\{c\}\_\{q\_\{j\}\}\. Since the action ofSqir×SqjcS\_\{q\_\{i\}\}^\{r\}\\times S^\{c\}\_\{q\_\{j\}\}onBi​jB\_\{ij\}is transitive ifi≠ji\\neq j, it follows thatBi​jB\_\{ij\}is fixed bySqir×SqjcS\_\{q\_\{i\}\}^\{r\}\\times S^\{c\}\_\{q\_\{j\}\}iff all entries are equal\. On the other hand, the action on diagonal blocksBi​iB\_\{ii\}has two group orbits ifqi\>1q\_\{i\}\>1: the set of diagonal elements and the set of off\-diagonal elements\. Ifk\>dk\>d, we have to consider the action induced bySdS\_\{d\}on columns\. Here each of thek−dk\-dadditional rows has a block decomposition into1×qj1\\times q\_\{j\}matricesBi​jB\_\{ij\}andSqjcS^\{c\}\_\{q\_\{j\}\}acts transitively onBi​jB\_\{ij\}\. Each of these blocks contributes 1 to the dimension of the fixed point space\.

In general, we have

dim​\(Bi​jG𝐪Δ\)=\{2,ifi=j∈𝐫𝐬1,otherwise\\text\{dim\}\\big\(B\_\{ij\}^\{G^\{\\Delta\}\_\{\\mathbf\{q\}\}\}\\big\)=\\begin\{cases\}&2,\\;\\text\{if $i=j\\in\{\{\\mathbf\{r\_\{s\}\}\}\}$\}\\\\ &1,\\;\\text\{otherwise\}\\end\{cases\}This argument suffices to prove the bias free case\. Adding bias increases the dimension by the number of rowtypes which isf\+mf\+m\. ∎

###### Remark 12\.5\.

Suppose thatk=dk=dandRa≥1R\_\{a\}\\geq 1so that there is at least oneqiq\_\{i\}equal to11\. After a permutation of rows and columns we may assume thatq0≥q1≥…≥qf−1≥1q\_\{0\}\\geq q\_\{1\}\\geq\\ldots\\geq q\_\{f\-1\}\\geq 1\. LetP=∪f−Ra≤i,j<fBi​jP=\\cup\_\{f\-R\_\{a\}\\leq i,j<f\}B\_\{ij\}\. ThenPPis anRa×RaR\_\{a\}\\times R\_\{a\}diagonal block for whichdim​\(PG𝐪Δ\)=Ra2\\text\{dim\}\\big\(P^\{G^\{\\Delta\}\_\{\\mathbf\{q\}\}\}\\big\)=R\_\{a\}^\{2\}\. Since there are no symmetry constraints onPP—the row and column actions of\(S1\)f−Rs\(S\_\{1\}\)^\{f\-R\_\{s\}\}onPPare trivial—PPcan be regarded as an asymmetric block with no symmetry\. In particular, ifRs=0R\_\{s\}=0, there are no symmetry constraints at all onP∈M⁡\(d,d\)P\\in M\(d,d\)\(d=fd=f\)\. If we have a non\-degenerate symmetric critical point𝐖\{\{\\mathbf\{W\}\}\}ofgrad​\(ℒ\)\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}and deform𝐕\{\{\\mathbf\{V\}\}\}to an asymmetric target \(no non\-trivial symmetries ofΔ​Sd\\Delta S\_\{d\}\), then as we deform𝐕\{\{\\mathbf\{V\}\}\},𝐖\{\{\\mathbf\{W\}\}\}will deform to an asymmetric non\-degenerate critical point\. An example is given later\. Similar comments hold whenm\>0m\>0and𝐖∈M⁡\(k,d\)\{\{\\mathbf\{W\}\}\}\\in M\(k,d\)and/or when the network is biased\.✠\\maltese

#### Properties of rowtypes

Assume first thatk=dk=d\(som=0m=0\) and the network is unbiased\. Let𝐖∈M⁡\(d,d\)\{\{\\mathbf\{W\}\}\}\\in M\(d,d\)have isotropy subgroupG=G𝐪Δ⊂Δ​SdG=G^\{\\Delta\}\_\{\\mathbf\{q\}\}\\subset\\Delta S\_\{d\}whereq0≥q1≥…≥qf−1≥1q\_\{0\}\\geq q\_\{1\}\\geq\\ldots\\geq q\_\{f\-1\}\\geq 1\. As described above, there is natural block decompositionℬ=\{Bi​j\|i,j∈𝐟\}\\mathcal\{B\}=\\\{B\_\{ij\}\\mbox\{$\\;\|\\;$\}i,j\\in\{\\mathbf\{f\}\}\\\}of𝐖\{\{\\mathbf\{W\}\}\}with diagonal blocksBi​iB\_\{ii\}consisting ofqi×qiq\_\{i\}\\times q\_\{i\}\-matrices which are fixed byΔ​Sqi\\Delta S\_\{q\_\{i\}\}\. The action is non\-trivial iffi∈𝐫𝐬i\\in\{\{\\mathbf\{r\_\{s\}\}\}\}\(equivalently,i∈\[0,Rs−1\]i\\in\[0,R\_\{s\}\-1\]\)\. A row𝐰\{\{\\mathbf\{w\}\}\}ofW∈M​\(d,d\)GW\\in M\(d,d\)^\{G\}is of rowtypei∈𝐟i\\in\{\\mathbf\{f\}\}iff𝐰\{\{\\mathbf\{w\}\}\}meets the diagonal blockBi​iB\_\{ii\}\. Rows of rowtypeiiare permuted byΔ​Sqi\\Delta S\_\{q\_\{i\}\}\.

Suppose now thatk=d\+mk=d\+m,m\>0m\>0and that the isotropy subgroup of𝐖\{\{\\mathbf\{W\}\}\}isG=G𝐪ΔG=G^\{\\Delta\}\_\{\\mathbf\{q\}\}as above\. It is always assumed that there are no parallel rows and so the lastmmrows of𝐖\{\{\\mathbf\{W\}\}\}are not symmetrically related to any of the firstddrows of𝐖\{\{\\mathbf\{W\}\}\}\.

Thef\+mf\+mrowtypes of𝐖\{\{\\mathbf\{W\}\}\}are indexed by𝐑\{\\mathbf\{R\}\}\. Ifi∈𝐫𝐬i\\in\{\{\\mathbf\{r\_\{s\}\}\}\}, thenqi\>1q\_\{i\}\>1and the set ofqiq\_\{i\}rows of rowtypeiiare permuted byΔ​Sqi\\Delta S\_\{q\_\{i\}\}\. The set of rowtypes for which there is exactly one row of each rowtype is parameterized by \{i∈𝐑\|i≥R−Rs=f\+m−Rs\}i\\in\{\\mathbf\{R\}\}\\mbox\{$\\;\|\\;$\}i\\geq R\-R\_\{s\}=f\+m\-R\_\{s\}\\\}\. Note that ifqi=1q\_\{i\}=1,i∈𝐟i\\in\{\\mathbf\{f\}\}, theni∈\[𝐫𝐬,f\)i\\in\[\{\{\\mathbf\{r\_\{s\}\}\}\},f\)and ifi≥fi\\geq f, then𝐰d−f\+i\{\{\\mathbf\{w\}\}\}^\{d\-f\+i\}will be the unique𝐖\{\{\\mathbf\{W\}\}\}row of rowtypeii\.

Supposei∈𝐫𝐬i\\in\{\{\\mathbf\{r\_\{s\}\}\}\}so thatqi\>1q\_\{i\}\>1and that𝐰a,𝐰b\{\{\\mathbf\{w\}\}\}^\{a\},\{\{\\mathbf\{w\}\}\}^\{b\}are both of rowtypeii\. SinceΔ​Sqi\\Delta S\_\{q\_\{i\}\}acts transitively on the set of rows of rowtypeii, it follows by equivariance that \(a\)‖𝐰a‖=‖𝐰b‖\\\|\{\{\\mathbf\{w\}\}\}^\{a\}\\\|=\\\|\{\{\\mathbf\{w\}\}\}^\{b\}\\\|\. \(b\) the angleΘi\\Theta\_\{i\}between𝐰a,𝐰b\{\{\\mathbf\{w\}\}\}^\{a\},\{\{\\mathbf\{w\}\}\}^\{b\}is independent of the choice ofa≠ba\\neq b; similar results hold forΛi​j\\Lambda\_\{ij\}, wherejjis not of rowtypeii\. Similarly, the angleθi\\theta\_\{i\}between𝐰a,𝐯b\{\{\\mathbf\{w\}\}\}^\{a\},\{\{\\mathbf\{v\}\}\}^\{b\}, whereb≠ab\\neq abut𝐰a,𝐰b\{\{\\mathbf\{w\}\}\}^\{a\},\{\{\\mathbf\{w\}\}\}^\{b\}are of the same rowtype, is independent of the choice ofa≠ba\\neq b\. Similar results hold for the anglesλi​j\\lambda\_\{ij\}, wherejjis not of rowtypeiiori=ji=j\(that is𝐰i,𝐯i\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{i\}are in the same row\)\.

### 12\.4\.Coordinatization ofM​\(k,d\)G𝐪ΔM\(k,d\)^\{G\_\{\\mathbf\{q\}\}^\{\\Delta\}\}&M​\(k,d\+1\)G𝐪ΔM\(k,d\+1\)^\{G\_\{\\mathbf\{q\}\}^\{\\Delta\}\}

First assume the unbiased case\. There is a natural linear isomorphismϕ\\phi\(but*not*isometry—see Remark[12\.7](https://arxiv.org/html/2609.30661#S12.Thmlemma7)at the end of this section\):

ϕ:M​\(k,d\)G𝐪Δ≈\(∏i∈𝐑ℝf\)×ℝrs\\phi:M\(k,d\)^\{G\_\{\\mathbf\{q\}\}^\{\\Delta\}\}\\approx\\left\(\\prod\_\{i\\in\{\\mathbf\{R\}\}\}\{\\mathbb\{R\}\}^\{f\}\\right\)\\times\{\\mathbb\{R\}\}^\{r\_\{s\}\}andϕ\\phiis used to coordinatizeM​\(k,d\)G𝐪ΔM\(k,d\)^\{G\_\{\\mathbf\{q\}\}^\{\\Delta\}\}\. In particular, we compute vector fields on the fixed point space usingϕ\\phicoordinates\.

Given𝐖∈M​\(k,d\)G𝐪Δ\{\{\\mathbf\{W\}\}\}\\in M\(k,d\)^\{G^\{\\Delta\}\_\{\\mathbf\{q\}\}\}, leti∈𝐑i\\in\{\\mathbf\{R\}\},j∈𝐟j\\in\{\\mathbf\{f\}\}\. Ifi≠ji\\neq j, defineξji\\xi^\{i\}\_\{j\}to be any element ofBi​jG𝐪ΔB\_\{ij\}^\{G\_\{\\mathbf\{q\}\}^\{\\Delta\}\}\. Ifi=ji=j, defineξii\\xi^\{i\}\_\{i\}to be any diagonal element ofBi​iG𝐪ΔB\_\{ii\}^\{G\_\{\\mathbf\{q\}\}^\{\\Delta\}\}\. Ifi=ji=jandqi\>1q\_\{i\}\>1—that is,i∈𝐫𝐬i\\in\{\{\\mathbf\{r\_\{s\}\}\}\}—defineξ⋆ii\\xi\_\{\\star i\}^\{i\}to be any off\-diagonal element ofBi​iG𝐪ΔB\_\{ii\}^\{G\_\{\\mathbf\{q\}\}^\{\\Delta\}\},

The construction defines the required linear isomorphism

𝐖↦\(𝝃,𝝃⋆\)∈\(∏i∈𝐑ℝf\)×ℝrs\{\{\\mathbf\{W\}\}\}\\mapsto\(\{\\boldsymbol\{\\xi\}\},\{\\boldsymbol\{\\xi\}\}\_\{\\star\}\)\\in\\left\(\\prod\_\{i\\in\{\\mathbf\{R\}\}\}\{\\mathbb\{R\}\}^\{f\}\\right\)\\times\{\\mathbb\{R\}\}^\{r\_\{s\}\}All rows of𝐖\{\{\\mathbf\{W\}\}\}of rowtypei∈𝐟i\\in\{\\mathbf\{f\}\}get mapped to the same point\(\(ξji\)j∈𝐟,ξi⋆i\)\(\(\\xi^\{i\}\_\{j\}\)\_\{j\\in\{\\mathbf\{f\}\}\},\\xi^\{i\}\_\{i\\star\}\)\(the ‘star’ term is omitted ifqi=1q\_\{i\}=1\)\. In particular the superscriptiiis the rowtype of the rows parameterized by\(\(ξ0i,…,ξf−1i\),ξ⋆ii\)\(\(\\xi\_\{0\}^\{i\},\\ldots,\\xi\_\{f\-1\}^\{i\}\),\\xi\_\{\\star i\}^\{i\}\)or\(ξ0i,…,ξf−1i\)\(\\xi\_\{0\}^\{i\},\\ldots,\\xi\_\{f\-1\}^\{i\}\), ifi∉𝐫𝐬i\\notin\{\{\\mathbf\{r\_\{s\}\}\}\}\.

For biased networks,M​\(k,d\+1\)G𝐪Δ≈M​\(k,d\)G𝐪Δ⊕M​\(1,d\)G𝐪rM\(k,d\+1\)^\{G^\{\\Delta\}\_\{\\mathbf\{q\}\}\}\\approx M\(k,d\)^\{G^\{\\Delta\}\_\{\\mathbf\{q\}\}\}\\oplus M\(1,d\)^\{G\_\{\{\\mathbf\{q\}\}\}^\{r\}\}, whereG𝐪rG\_\{\{\\mathbf\{q\}\}\}^\{r\}is the projection ofG𝐪ΔG\_\{\\mathbf\{q\}\}^\{\\Delta\}onto∏Sqir⊂Sdr\\prod S\_\{q\_\{i\}\}^\{r\}\\subset S\_\{d\}^\{r\}\.

###### Example 12\.6\.

Suppose that𝐖∈M⁡\(k,d\+1\)\{\{\\mathbf\{W\}\}\}\\in M\(k,d\+1\),G𝐖=Δ⁡\(Sd−1×S1\)G\_\{\{\\mathbf\{W\}\}\}=\\Delta\(S\_\{d\-1\}\\times S\_\{1\}\)andm=1m=1\. Assuming normalized bias,dim​\(M​\(k,d\+1\)G\(q0,1\)Δ\)=10\\text\{dim\}\(M\(k,d\+1\)^\{G^\{\\Delta\}\_\{\(q\_\{0\},1\)\}\}\)=10and

𝐖=\[ξ00ξ⋆00…ξ⋆00ξ10β¯0ξ⋆00ξ00…ξ⋆00ξ10β¯0………………………………ξ⋆00ξ⋆00…ξ00ξ10β¯0ξ01ξ01…ξ01ξ11β¯1ξ02ξ02…ξ02ξ12β¯2\]\{\{\\mathbf\{W\}\}\}=\\left\[\\begin\{matrix\}\\xi^\{0\}\_\{0\}&\\xi\_\{\\star 0\}^\{0\}&\\ldots&\\xi\_\{\\star 0\}^\{0\}&\\xi^\{0\}\_\{1\}&\\overline\{\\beta\}\_\{0\}\\\\ \\xi^\{0\}\_\{\\star 0\}&\\xi\_\{0\}^\{0\}&\\ldots&\\xi^\{0\}\_\{\\star 0\}&\\xi^\{0\}\_\{1\}&\\overline\{\\beta\}\_\{0\}\\\\ \\ldots&\\ldots&\\ldots&\\ldots&\\ldots&\\ldots\\\\ \\ldots&\\ldots&\\ldots&\\ldots&\\ldots&\\ldots\\\\ \\xi^\{0\}\_\{\\star 0\}&\\xi\_\{\\star 0\}^\{0\}&\\ldots&\\xi^\{0\}\_\{0\}&\\xi^\{0\}\_\{1\}&\\overline\{\\beta\}\_\{0\}\\\\ \\xi^\{1\}\_\{0\}&\\xi^\{1\}\_\{0\}&\\ldots&\\xi^\{1\}\_\{0\}&\\xi^\{1\}\_\{1\}&\\overline\{\\beta\}\_\{1\}\\\\ \\xi^\{2\}\_\{0\}&\\xi^\{2\}\_\{0\}&\\ldots&\\xi^\{2\}\_\{0\}&\\xi^\{2\}\_\{1\}&\\overline\{\\beta\}\_\{2\}\\end\{matrix\}\\right\]The penultimate row corresponds to theS1S\_\{1\}factor and the final row to the additional neuron\. In this case

ϕ⁡\(𝐖,β¯\)=\(\(ξ00,ξ10,β¯0,ξ01,ξ11,β¯1,ξ02,ξ12,β¯2\),ξ⋆00\)∈ℝ9×ℝ\\phi\(\{\{\\mathbf\{W\}\}\},\\overline\{\\beta\}\)=\(\(\\xi^\{0\}\_\{0\},\\xi^\{0\}\_\{1\},\\overline\{\\beta\}\_\{0\},\\xi^\{1\}\_\{0\},\\xi^\{1\}\_\{1\},\\overline\{\\beta\}\_\{1\},\\xi^\{2\}\_\{0\},\\xi^\{2\}\_\{1\},\\overline\{\\beta\}\_\{2\}\),\\xi\_\{\\star 0\}^\{0\}\)\\in\{\\mathbb\{R\}\}^\{9\}\\times\{\\mathbb\{R\}\}Ignoring the bias, the number of coordinates needed is33\(for the first row\) and22each for the last two rows\. A total of77coordinates, as required by Lemma[12\.4](https://arxiv.org/html/2609.30661#S12.Thmlemma4)\(2\)\. Three additional parameters are needed when bias is added\.※

###### Remark 12\.7\.

As indicated in the definition of the coordinatization ofM​\(k,d\)G𝐪ΔM\(k,d\)^\{G\_\{\\mathbf\{q\}\}^\{\\Delta\}\}&M​\(k,d\+1\)G𝐪ΔM\(k,d\+1\)^\{G\_\{\\mathbf\{q\}\}^\{\\Delta\}\}, the coordinatization mapϕ\\phiis not an isometry and so does not preserve the inner product\. In particular, we do not recovergrad​\(ℒ\)\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}if we compute the gradient ofℒ\\mathcal\{L\}inϕ\\phi\(Euclidean\) coordinates\. If we arrange matters so thatϕ\\phiis an isometry, then we do recovergrad​\(ℒ\)\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}\. There are two ways of doing this: \(a\) leaveϕ\\phiunchanged but change the inner product on the range space so thatϕ\\phiis an isometry, \(b\) redefineϕ\\phiso thatϕ\\phiis an isometry\. In case \(b\) the coordinateξ\\xicorresponding to ap×qp\\times qblock B, with all entriesξ\\xi, will be mapped top​q​ξ\\sqrt\{pq\}\\xirather thanξ\\xi\. A diagonal blockξ​Ip\\xi I\_\{p\}will be mapped top​ξ\\sqrt\{p\}\\xi\. In Example[12\.6](https://arxiv.org/html/2609.30661#S12.Thmlemma6), we modifyϕ\\phito the isometryϕ~:M​\(k,d\+1\)G\(q0,1\)Δ→ℝ9×ℝ\\widetilde\{\\phi\}:M\(k,d\+1\)^\{G^\{\\Delta\}\_\{\(q\_\{0\},1\)\}\}\{\\rightarrow\}\{\\mathbb\{R\}\}^\{9\}\\times\{\\mathbb\{R\}\}defined by

ϕ~​\(𝐖,β¯\)=\(\(α0​ξ00,α0​ξq0,α0​β¯0,α0​ξ01,ξ11,β¯1,α0​ξ02,ξ12,β¯2\),α1​ξ⋆00\)\\widetilde\{\\phi\}\(\{\{\\mathbf\{W\}\}\},\\overline\{\\beta\}\)=\(\(\\alpha\_\{0\}\\xi\_\{0\}^\{0\},\\alpha\_\{0\}\\xi\_\{q\}^\{0\},\\alpha\_\{0\}\\overline\{\\beta\}\_\{0\},\\alpha\_\{0\}\\xi^\{1\}\_\{0\},\\xi\_\{1\}^\{1\},\\overline\{\\beta\}\_\{1\},\\alpha\_\{0\}\\xi^\{2\}\_\{0\},\\xi^\{2\}\_\{1\},\\overline\{\\beta\}\_\{2\}\),\\alpha\_\{1\}\\xi\_\{\\star 0\}^\{0\}\)whereα0=d−1\\alpha\_\{0\}=\\sqrt\{d\-1\},α1=\(d−1\)​\(d−2\)\\alpha\_\{1\}=\\sqrt\{\(d\-1\)\(d\-2\)\}and, as usual, we take the Frobenius \(Euclidean\) norm onM⁡\(k,d\+1\)≈ℝk⁡\(d\+1\)M\(k,d\+1\)\\approx\{\\mathbb\{R\}\}^\{k\(d\+1\)\}\. However, we do not follow either scheme since both \(a\) and \(b\) add some complexity but yield no additional information\. Indeed, the linearization ofgrad​\(ℒ\)\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}at a critical point is similar to the linearization of the vector field inϕ\\phicoordinates by a diagonal matrix\. Moreover, Sylvester’s test for strictly positive spectrum applies to the linearization ofgrad​\(ℒ\)\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}inϕ\\phi\-coordinates with no change even though there is no Hessian \(that is, the linearization matrix is not usually symmetric\)\.✠\\maltese

### 12\.5\.Critical point equations forgrad​\(ℒ\)\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}onM​\(k,d\+1\)G𝐪ΔM\(k,d\+1\)^\{G^\{\\Delta\}\_\{\\mathbf\{q\}\}\}

In sections[10](https://arxiv.org/html/2609.30661#S10)&[11](https://arxiv.org/html/2609.30661#S11)formulas were given for the components ofgrad​\(ℒ\)\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}when there is bias but no symmetry\. Using the crucial notation of rowtype, the aim is to extend these results to take account of symmetry\. In this way, relatively simple critical point equations are obtained that are amenable to analysis and simulation for large values ofdd\(assuming, for example, thatq1,…,qf−1q\_\{1\},\\ldots,q\_\{f\-1\}are fixed iff\>1f\>1\)\. The first step is to expand the table of coefficients given in section[10\.1](https://arxiv.org/html/2609.30661#S10.SS1)to allow for rowtypes so that we can rewrite the formula for the components ofgrad​\(ℒ\)\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}in terms of rowtypes\. Since there will generally be multiple rows with the same rowtype, this will lead to a reduction of the number of critical point equations defined on the fixed point space\.

The coefficients defined in section[10\.1](https://arxiv.org/html/2609.30661#S10.SS1)were parametrized by a pair of rows \(one from𝐖\{\{\\mathbf\{W\}\}\}, the other from𝐖\{\{\\mathbf\{W\}\}\}or𝐕\{\{\\mathbf\{V\}\}\}\)\. We now work in terms of rowtypes and define families of coefficients for rows of rowtypeiiwherei∈𝐫𝐬i\\in\{\{\\mathbf\{r\_\{s\}\}\}\}\. There will be one set of critical point equations for each rowtype\. The number of equations for rowtypeiiwill depend on whether or noti∈𝐫𝐬i\\in\{\{\\mathbf\{r\_\{s\}\}\}\}—there will be one ‘extra’ equation ifi∈𝐫𝐬i\\in\{\{\\mathbf\{r\_\{s\}\}\}\}otherwise there will beffequations\. We start with the easiest casei∉𝐫𝐬i\\notin\{\{\\mathbf\{r\_\{s\}\}\}\}\.

### 12\.6\.Critical point equations associated to rowtypei∉𝐫𝐬i\\notin\{\{\\mathbf\{r\_\{s\}\}\}\}

Ifk=dk=d, the integersqiq\_\{i\}are defined for all the rowtypes\. Ifk=m\+dk=m\+d,m\>0m\>0, defineqj=1q\_\{j\}=1for each of themmadditional rows \(strictly, rowtypes\)𝐰j\{\{\\mathbf\{w\}\}\}^\{j\},j∈𝐫𝐦j\\in\{\{\\mathbf\{r\_\{m\}\}\}\}\. In what follows, networks are assumed to be biased and we usually write𝐖\{\{\\mathbf\{W\}\}\}\(or𝐖~\\widetilde\{\{\{\\mathbf\{W\}\}\}\}\) rather than\(𝐖,𝜷¯\)\(\{\{\\mathbf\{W\}\}\},\\boldsymbol\{\\overline\{\\beta\}\}\)\.

Given𝐖∈M​\(k,d\+1\)G𝐪Δ\{\{\\mathbf\{W\}\}\}\\in M\(k,d\+1\)^\{G^\{\\Delta\}\_\{\\mathbf\{q\}\}\}, denote the rowtype of𝐰j\{\{\\mathbf\{w\}\}\}^\{j\}\(rowjjof𝐖\{\{\\mathbf\{W\}\}\}\) byr⁡\(j\)∈𝐑r\(j\)\\in\{\\mathbf\{R\}\}\. Conversely, ifiiis a rowtype, let𝐢R\{\\mathbf\{i\}\}\_\{R\}denote the ordered set parameterizing the rows𝐰a\{\{\\mathbf\{w\}\}\}^\{a\}for whichr⁡\(a\)=ir\(a\)=i\. For example, takingG𝐪ΔG\_\{\\mathbf\{q\}\}^\{\\Delta\}andq2≥q1\>q2=1q\_\{2\}\\geq q\_\{1\}\>q\_\{2\}=1,

𝟎R\\displaystyle\{\\mathbf\{0\}\}\_\{R\}=\\displaystyle=\{0,1,…,q0−1\}\\displaystyle\\\{0,1,\\ldots,q\_\{0\}\-1\\\}𝟏R\\displaystyle\{\\mathbf\{1\}\}\_\{R\}=\\displaystyle=\{q0,1,…,q0\+q1−1\}\\displaystyle\\\{q\_\{0\},1,\\ldots,q\_\{0\}\+q\_\{1\}\-1\\\}𝟐R\\displaystyle\{\\mathbf\{2\}\}\_\{R\}=\\displaystyle=\{q0\+q1\}\\displaystyle\\\{q\_\{0\}\+q\_\{1\}\\\}Given the rowtypeii, letI⁡\(i\)I\(i\)denote the initial element of𝐢R\{\\mathbf\{i\}\}\_\{R\}\. Thus the rows of rowtypeiiin𝐖∈M​\(k,d\)G𝐪Δ\{\{\\mathbf\{W\}\}\}\\in M\(k,d\)^\{G^\{\\Delta\}\_\{\\mathbf\{q\}\}\}are𝐰I⁡\(i\),…,𝐰I⁡\(i\)\+qi−1\{\{\\mathbf\{w\}\}\}^\{I\(i\)\},\\ldots,\{\{\\mathbf\{w\}\}\}^\{I\(i\)\+q\_\{i\}\-1\}and∪i∈𝐑𝐢R=𝐤\\cup\_\{i\\in\{\\mathbf\{R\}\}\}\{\\mathbf\{i\}\}\_\{R\}=\{\\mathbf\{k\}\}\.

Generally, we label specific rows of𝐖\{\{\\mathbf\{W\}\}\}bya,b,…∈𝐤a,b,\\ldots\\in\{\\mathbf\{k\}\}*except that for the first rowI⁡\(i\)∈𝐢RI\(i\)\\in\{\\mathbf\{i\}\}\_\{R\}we often write𝐰i\{\{\\mathbf\{w\}\}\}^\{i\}*\. That is,𝐰i\{\{\\mathbf\{w\}\}\}^\{i\}may denote either a general row of rowtypeii*or*, more usually, the first row of rowtypeiiin𝐖\{\{\\mathbf\{W\}\}\}\. In order to find the critical point equations on the fixed point space in𝝃,𝝃⋆\{\\boldsymbol\{\\xi\}\},\{\\boldsymbol\{\\xi\}\}\_\{\\star\}coordinates, it suffices to determine the components ofgrad​\(ℒ\)​\(𝐖\)i\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}\(\{\{\\mathbf\{W\}\}\}\)^\{i\}, fori∈𝐑i\\in\{\\mathbf\{R\}\}\(in terms of row number, fori=0,q0,q0\+q1,…i=0,q\_\{0\},q\_\{0\}\+q\_\{1\},\\ldots\)\. Here𝐰i\{\{\\mathbf\{w\}\}\}^\{i\}will have coordinates\(\(ξji\)j∈𝐟,ξ⋆ii\)\\big\(\(\\xi^\{i\}\_\{j\}\)\_\{j\\in\{\\mathbf\{f\}\}\},\\xi\_\{\\star i\}^\{i\}\\big\)ifi∈𝐫𝐬i\\in\{\{\\mathbf\{r\_\{s\}\}\}\}, otherwise\(ξji\)j∈𝐟\(\\xi^\{i\}\_\{j\}\)\_\{j\\in\{\\mathbf\{f\}\}\}\. In other words, when working with𝝃\{\\boldsymbol\{\\xi\}\}coordinates on\(∏i∈𝐑ℝf\)×ℝrs\\left\(\\prod\_\{i\\in\{\\mathbf\{R\}\}\}\{\\mathbb\{R\}\}^\{f\}\\right\)\\times\{\\mathbb\{R\}\}^\{r\_\{s\}\}, rowtype is the natural parameter as opposed to the actual row number\.

We continue with the same definitions of𝒰i​j,…,𝒱i​j\\mathcal\{U\}\_\{ij\},\\ldots,\\mathcal\{V\}\_\{ij\}used in section[10\.2](https://arxiv.org/html/2609.30661#S10.SS2)except that we use an overline to emphasize that everything is in terms of rowtype and take account of multiplicities\.

Thus, fori∈𝐑i\\in\{\\mathbf\{R\}\}, define

𝒰¯i​j==ei​jX\+β¯i​γ¯j​\(ℓi​j\+π−λ~i​j\)\+e−ki​j​sin⁡\(λi​j\),j∈𝐟𝒱¯i​j=ℓi​j\+π−λ~i​j,j∈𝐟𝒲¯i​j=Ei​jX\+β¯i​β¯j​\(Li​j\+π−Λ~i​j\)\+e−Ki​j​sin⁡\(Λi​j\),j≠i𝒴¯i​j=Li​j\+π−Λ~i​j,j≠i𝒳¯i=1\+Xi\\begin\{aligned\} \\overline\{\\mathcal\{U\}\}\_\{ij\}=&=e\_\{ij\}^\{X\}\+\\overline\{\\beta\}\_\{i\}\\overline\{\\gamma\}\_\{j\}\\big\(\\ell\_\{ij\}\+\\pi\-\{\\widetilde\{\\lambda\}\}\_\{ij\}\\big\)\+e^\{\-k\_\{ij\}\}\\sin\(\\lambda\_\{ij\}\),\\;j\\in\{\\mathbf\{f\}\}\\\\ \\overline\{\\mathcal\{V\}\}\_\{ij\}=&\\ell\_\{ij\}\+\\pi\-\\widetilde\{\\lambda\}\_\{ij\},\\;j\\in\{\\mathbf\{f\}\}\\\\ \\overline\{\\mathcal\{W\}\}\_\{ij\}=&E\_\{ij\}^\{X\}\+\\overline\{\\beta\}\_\{i\}\\overline\{\\beta\}\_\{j\}\\big\(L\_\{ij\}\+\\pi\-\{\\widetilde\{\\Lambda\}\}\_\{ij\}\\big\)\+e^\{\-K\_\{ij\}\}\\sin\(\\Lambda\_\{ij\}\),\\;j\\neq i\\\\ \\overline\{\\mathcal\{Y\}\}\_\{ij\}=&L\_\{ij\}\+\\pi\-\\widetilde\{\\Lambda\}\_\{ij\},\\;j\\neq i\\\\ \\overline\{\\mathcal\{X\}\}\_\{i\}=&1\+X\_\{i\}\\end\{aligned\}Note that the conditionj≠ij\\neq iis on rowtypes and so ifi∈𝐫𝐬i\\in\{\{\\mathbf\{r\_\{s\}\}\}\}many terms are missed\. This point is addressed shortly\.

###### Definition 12\.8\.

Forj∈𝐫𝐬j\\in\{\{\\mathbf\{r\_\{s\}\}\}\}define

1. \(1\)𝐰Σj=∑r⁡\(a\)=j𝐰a\{\{\\mathbf\{w\}\}\}^\{j\}\_\{\\Sigma\}=\\sum\_\{r\(a\)=j\}\{\{\\mathbf\{w\}\}\}^\{a\}\.
2. \(2\)𝐯Σj=∑r⁡\(a\)=j𝐯a\{\{\\mathbf\{v\}\}\}^\{j\}\_\{\\Sigma\}=\\sum\_\{r\(a\)=j\}\{\{\\mathbf\{v\}\}\}^\{a\}\.

\(Ifj∈𝐑∖𝐫𝐬j\\in\{\\mathbf\{R\}\}\\smallsetminus\{\{\\mathbf\{r\_\{s\}\}\}\}, then𝐰Σj\{\{\\mathbf\{w\}\}\}^\{j\}\_\{\\Sigma\}is defined to be𝐰j\{\{\\mathbf\{w\}\}\}^\{j\}; similarly for𝐯Σj\{\{\\mathbf\{v\}\}\}^\{j\}\_\{\\Sigma\}\)\.

###### Remark 12\.9\.

The row vector𝐰Σj\{\{\\mathbf\{w\}\}\}^\{j\}\_\{\\Sigma\}may be written as a1×d1\\times d\-matrix with block decomposition\[\(aj​0\)q0\],…,\[\(aj​f−1\)qf−1\]\[\(a\_\{j0\}\)^\{q\_\{0\}\}\],\\ldots,\[\(a\_\{j\{f\-1\}\}\)^\{q\_\{f\-1\}\}\]where

aj​α=\{qj​ξαj,if​α≠jξjj\+\(qj−1\)​ξ⋆jj,if​α=ja\_\{j\\alpha\}=\\begin\{cases\}&q\_\{j\}\\xi^\{j\}\_\{\\alpha\},\\;\\text\{if \}\\alpha\\neq j\\\\ &\\xi^\{j\}\_\{j\}\+\(q\_\{j\}\-1\)\\xi\_\{\\star j\}^\{j\},\\;\\text\{if \}\\alpha=j\\end\{cases\}Of course, ifqj=1q\_\{j\}=1\(and soj∉𝐫𝐬j\\notin\{\{\\mathbf\{r\_\{s\}\}\}\}\), applying the definition, gives𝐰Σj=𝐰j\{\{\\mathbf\{w\}\}\}^\{j\}\_\{\\Sigma\}=\{\{\\mathbf\{w\}\}\}^\{j\},𝐯Σj=𝐯j\{\{\\mathbf\{v\}\}\}^\{j\}\_\{\\Sigma\}=\{\{\\mathbf\{v\}\}\}^\{j\}\.✠\\maltese

###### Proposition 12\.10\.

\(Notation and assumptions as above\.\) Fori∈𝐑∖𝐫𝐬i\\in\{\\mathbf\{R\}\}\\smallsetminus\{\{\\mathbf\{r\_\{s\}\}\}\}, componentiiofgrad​\(ℒ\)​\(𝐖\)\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}\(\{\{\\mathbf\{W\}\}\}\)is given by

\(12\.50\)grad​\(ℒ\)​\(𝐖\)i=Γi​𝐰i−𝐀i\+𝐁i\+𝛀\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}\(\{\{\\mathbf\{W\}\}\}\)^\{i\}=\\Gamma\_\{i\}\{\{\\mathbf\{w\}\}\}^\{i\}\-\{\\mathbf\{A\}\}^\{i\}\+\{\\mathbf\{B\}\}^\{i\}\+\\boldsymbol\{\\Omega\}where

Γi\\displaystyle\\Gamma\_\{i\}=\\displaystyle=Xi2\+12​π​\[∑j∈𝐑,j≠iqj​‖𝐰j‖​e−Ki​j​sin⁡\(Λi​j\)‖𝐰i‖−∑j∈𝐟,j≠iqj​‖𝐯j‖​e−ki​j​sin⁡\(λi​j\)‖𝐰i‖\]\\displaystyle\\frac\{X\_\{i\}\}\{2\}\+\\frac\{1\}\{2\\pi\}\\left\[\\sum\_\{j\\in\{\\mathbf\{R\}\},j\\neq i\}q\_\{j\}\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|e^\{\-K\_\{ij\}\}\\sin\(\\Lambda\_\{ij\}\)\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\-\\sum\_\{j\\in\{\\mathbf\{f\}\},j\\neq i\}q\_\{j\}\\frac\{\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|e^\{\-k\_\{ij\}\}\\sin\(\\lambda\_\{ij\}\)\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\\right\]\+12​π​\[∑j∈𝐑,j≠iqj​‖𝐰j‖​Ei​jX‖𝐰i‖−∑j∈𝐟,j≠iqj​‖𝐯j‖​ei​jX‖𝐰i‖\]−12​π​\(‖𝐯i‖​\(e−ki​i​sin⁡\(λi​i\)\+ei​iX\)‖𝐰i‖\)\\displaystyle\+\\frac\{1\}\{2\\pi\}\\left\[\\sum\_\{j\\in\{\\mathbf\{R\}\},j\\neq i\}q\_\{j\}\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|E\_\{ij\}^\{X\}\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\-\\sum\_\{j\\in\{\\mathbf\{f\}\},j\\neq i\}q\_\{j\}\\frac\{\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|e\_\{ij\}^\{X\}\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\\right\]\-\\frac\{1\}\{2\\pi\}\\left\(\\frac\{\\\|\{\{\\mathbf\{v\}\}\}^\{i\}\\\|\\big\(e^\{\-k\_\{ii\}\}\\sin\(\\lambda\_\{ii\}\)\+e\_\{ii\}^\{X\}\\big\)\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\\right\)\+β¯i2​π​\[∑j∈𝐑,j≠iqj​β¯j​‖𝐰j‖​\(Li​j\+\(π−Λ~i​j\)\)‖𝐰i‖−∑j∈𝐟,j≠iqj​γ¯j​‖𝐯j‖​\(ℓi​j\+\(π−λ~i​j\)\)‖𝐰i‖\]\\displaystyle\+\\frac\{\\overline\{\\beta\}\_\{i\}\}\{2\\pi\}\\left\[\\sum\_\{j\\in\{\\mathbf\{R\}\},j\\neq i\}q\_\{j\}\\frac\{\\overline\{\\beta\}\_\{j\}\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\\big\(L\_\{ij\}\+\(\\pi\-\{\\widetilde\{\\Lambda\}\}\_\{ij\}\)\\big\)\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\-\\sum\_\{j\\in\{\\mathbf\{f\}\},j\\neq i\}q\_\{j\}\\frac\{\\overline\{\\gamma\}\_\{j\}\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|\\big\(\\ell\_\{ij\}\+\(\\pi\-\{\\widetilde\{\\lambda\}\}\_\{ij\}\)\\big\)\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\\right\]−β¯i​γ¯i2​π​‖𝐯i‖​\(ℓi​i\+\(π−λ~i​i\)\)‖𝐰i‖\\displaystyle\-\\frac\{\\overline\{\\beta\}\_\{i\}\\overline\{\\gamma\}\_\{i\}\}\{2\\pi\}\\frac\{\\\|\{\{\\mathbf\{v\}\}\}^\{i\}\\\|\\big\(\\ell\_\{ii\}\+\(\\pi\-\{\\widetilde\{\\lambda\}\}\_\{ii\}\)\\big\)\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}𝐀i\\displaystyle\{\\mathbf\{A\}\}^\{i\}=\\displaystyle=12​π​\(∑j∈𝐑,j≠iΛ~i​j​𝐰Σj−∑j∈𝐟λ~i​j​𝐯Σj\)\\displaystyle\\frac\{1\}\{2\\pi\}\\left\(\\sum\_\{j\\in\{\\mathbf\{R\}\},j\\neq i\}\{\\widetilde\{\\Lambda\}\}\_\{ij\}\{\{\\mathbf\{w\}\}\}\_\{\\Sigma\}^\{j\}\-\\sum\_\{j\\in\{\\mathbf\{f\}\}\}\{\\widetilde\{\\lambda\}\}\_\{ij\}\{\{\\mathbf\{v\}\}\}\_\{\\Sigma\}^\{j\}\\right\)𝐁i\\displaystyle\{\\mathbf\{B\}\}^\{i\}=\\displaystyle=12​π​\(∑j∈𝐑,j≠iLi​j​𝐰Σj−∑j∈𝐟ℓi​j​𝐯Σj\)\\displaystyle\\frac\{1\}\{2\\pi\}\\left\(\\sum\_\{j\\in\{\\mathbf\{R\}\},j\\neq i\}L\_\{ij\}\{\{\\mathbf\{w\}\}\}\_\{\\Sigma\}^\{j\}\-\\sum\_\{j\\in\{\\mathbf\{f\}\}\}\\ell\_\{ij\}\{\{\\mathbf\{v\}\}\}\_\{\\Sigma\}^\{j\}\\right\)𝛀\\displaystyle\{\\boldsymbol\{\\Omega\}\}=\\displaystyle=12​\(∑j∈𝐑𝐰Σj−∑j∈𝐟𝐯Σj\)=12​\(∑a∈𝐤𝐰a−∑b∈𝐝𝐯b\)\\displaystyle\\frac\{1\}\{2\}\\left\(\\sum\_\{j\\in\{\\mathbf\{R\}\}\}\{\{\\mathbf\{w\}\}\}\_\{\\Sigma\}^\{j\}\-\\sum\_\{j\\in\{\\mathbf\{f\}\}\}\{\{\\mathbf\{v\}\}\}\_\{\\Sigma\}^\{j\}\\right\)=\\frac\{1\}\{2\}\\left\(\\sum\_\{a\\in\{\\mathbf\{k\}\}\}\{\{\\mathbf\{w\}\}\}^\{a\}\-\\sum\_\{b\\in\{\\mathbf\{d\}\}\}\{\{\\mathbf\{v\}\}\}^\{b\}\\right\)

###### Proof\.

Follows from the earlier description ofgrad​\(ℒ\)​\(𝐖\)i\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}\(\{\{\\mathbf\{W\}\}\}\)^\{i\}fori∈𝐤i\\in\{\\mathbf\{k\}\}\. Note that sincei∉𝐫𝐬i\\notin\{\{\\mathbf\{r\_\{s\}\}\}\},qi=1q\_\{i\}=1and so none of the displayed terms involve the anglesΘi,θi\\Theta\_\{i\},\\theta\_\{i\}which are only defined ifqi\>1q\_\{i\}\>1\. ∎

###### Remarks 12\.11\.

\(1\) The proposition is valid for any row ofgrad​\(ℒ\)​\(𝐖\)\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}\(\{\{\\mathbf\{W\}\}\}\)which is of rowtypei∉𝐫𝐬i\\notin\{\{\\mathbf\{r\_\{s\}\}\}\}\. However, it is easiest to think ofiias representing both the rowtype and the first row ingrad​\(ℒ\)​\(𝐖\)\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}\(\{\{\\mathbf\{W\}\}\}\)which is of rowtypeii; that is, rowI⁡\(i\)I\(i\)\. \(2\) The only angles that appear in the formula forgrad​\(ℒ\)​\(𝐖\)i\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}\(\{\{\\mathbf\{W\}\}\}\)^\{i\}wheni∉𝐫𝐬i\\notin\{\{\\mathbf\{r\_\{s\}\}\}\}are theΛi​j,λi​j\\Lambda\_\{ij\},\\lambda\_\{ij\}angles defined in Section[10](https://arxiv.org/html/2609.30661#S10)\.✠\\maltese

Ifi∉𝐫𝐬i\\notin\{\{\\mathbf\{r\_\{s\}\}\}\}, then the row vectors𝐀i,𝐁i\{\\mathbf\{A\}\}^\{i\},\{\\mathbf\{B\}\}^\{i\}have a block decomposition induced from the natural block decomposition onM​\(k,d\+1\)G𝐪ΔM\(k,d\+1\)^\{G^\{\\Delta\}\_\{\\mathbf\{q\}\}\}\. If𝐕=Id\{\{\\mathbf\{V\}\}\}=I\_\{d\}, then𝐀i=\[\(A0i\)q0,…,\(Af−1i\)qf−1\]\{\\mathbf\{A\}\}^\{i\}=\[\(A\_\{0\}^\{i\}\)^\{q\_\{0\}\},\\ldots,\(A\_\{f\-1\}^\{i\}\)^\{q\_\{f\-1\}\}\]where\(Aji\)qj\(A^\{i\}\_\{j\}\)^\{q\_\{j\}\}denotes thejjth1×qj1\\times q\_\{j\}block\. This consists ofAjiA^\{i\}\_\{j\}repeatedqjq\_\{j\}times where

Aji=\{12​π​\[∑α≠jqα​Λ~i​α​ξjα\+Λ~i​j​\(\(qj−1\)​ξ⋆jj\+ξjj\)−λ~i​j\],j≠i12​π​\[∑α≠iqα​Λ~i​α​ξiα−λ~i​i\],j=iA^\{i\}\_\{j\}=\\begin\{cases\}&\\frac\{1\}\{2\\pi\}\\left\[\\sum\_\{\\alpha\\neq j\}q\_\{\\alpha\}\{\\widetilde\{\\Lambda\}\}\_\{i\\alpha\}\\xi^\{\\alpha\}\_\{j\}\+\{\\widetilde\{\\Lambda\}\}\_\{ij\}\(\(q\_\{j\}\-1\)\\xi^\{j\}\_\{\\star j\}\+\\xi^\{j\}\_\{j\}\)\-\{\\widetilde\{\\lambda\}\}\_\{ij\}\\right\],\\;j\\neq i\\\\ &\\frac\{1\}\{2\\pi\}\\left\[\\sum\_\{\\alpha\\neq i\}q\_\{\\alpha\}\{\\widetilde\{\\Lambda\}\}\_\{i\\alpha\}\\xi^\{\\alpha\}\_\{i\}\-\{\\widetilde\{\\lambda\}\}\_\{ii\}\\right\],\\;j=i\\end\{cases\}Similar, but simpler, expressions hold forBjiB^\{i\}\_\{j\}providedi∉𝐫𝐬i\\notin\{\{\\mathbf\{r\_\{s\}\}\}\}\. Note that𝐀i\{\\mathbf\{A\}\}^\{i\}\(resp\.𝐁i\{\\mathbf\{B\}\}^\{i\}\) contains the term−12​π​λ~i​i​𝐯i\-\\frac\{1\}\{2\\pi\}\{\\widetilde\{\\lambda\}\}\_\{ii\}\{\{\\mathbf\{v\}\}\}^\{i\}\(resp\.−12​π​ℓi​i​𝐯i\-\\frac\{1\}\{2\\pi\}\\ell\_\{ii\}\{\{\\mathbf\{v\}\}\}^\{i\}\)\.

###### Example 12\.12\.

SupposeG𝐪Δ=Δ⁡\(Sd−1×S1\)G\_\{\\mathbf\{q\}\}^\{\\Delta\}=\\Delta\(S\_\{d\-1\}\\times S\_\{1\}\), so thatq0=d−1q\_\{0\}=d\-1,q1=1q\_\{1\}=1\. Assumem=0m=0and𝐕=Id\{\{\\mathbf\{V\}\}\}=I\_\{d\}\. Since1∉𝐫𝐬1\\notin\{\{\\mathbf\{r\_\{s\}\}\}\}, Proposition[12\.10](https://arxiv.org/html/2609.30661#S12.Thmlemma10)may be used to compute𝐀1\\mathbf\{A\}^\{1\}and𝐁1\\mathbf\{B\}^\{1\}\. We have

A01\\displaystyle\{A\}^\{1\}\_\{0\}=\\displaystyle=12​π​\[\(ξ00\+\(q0−1\)​ξ⋆00\)​Λ~10−λ~10\],A11=12​π​\[q0​ξ10​Λ~10−λ~11\]\\displaystyle\\frac\{1\}\{2\\pi\}\\left\[\\big\(\\xi^\{0\}\_\{0\}\+\(q\_\{0\}\-1\)\\xi^\{0\}\_\{\\star 0\}\\big\)\{\\widetilde\{\\Lambda\}\}\_\{10\}\-\{\\widetilde\{\\lambda\}\}\_\{10\}\\right\],\\;\{A\}^\{1\}\_\{1\}=\\frac\{1\}\{2\\pi\}\\left\[q\_\{0\}\\xi^\{0\}\_\{1\}\{\\widetilde\{\\Lambda\}\}\_\{10\}\-\{\\widetilde\{\\lambda\}\}\_\{11\}\\right\]B01\\displaystyle\{B\}^\{1\}\_\{0\}=\\displaystyle=12​π​\[\(ξ00\+\(q0−1\)​ξ⋆00\)​L10−ℓ10\],B11=12​π​\[q0​ξ10​L10−ℓ11\]\\displaystyle\\frac\{1\}\{2\\pi\}\\left\[\\big\(\\xi^\{0\}\_\{0\}\+\(q\_\{0\}\-1\)\\xi^\{0\}\_\{\\star 0\}\\big\)L\_\{10\}\-\\ell\_\{10\}\\right\],\\;\{B\}^\{1\}\_\{1\}=\\frac\{1\}\{2\\pi\}\\left\[q\_\{0\}\\xi^\{0\}\_\{1\}L\_\{10\}\-\\ell\_\{11\}\\right\]where

L10=−π2​\[erf​\(β¯12\)\+erf​\(β¯02\)\]L\_\{10\}=\-\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{1\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{0\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]ℓ10=−π2​\[erf​\(β¯12\)\+erf​\(γ¯02\)\],ℓ11=−π2​\[erf​\(β¯12\)\+erf​\(γ¯12\)\]\\ell\_\{10\}=\-\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{1\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\gamma\}\_\{0\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\],\\quad\\ell\_\{11\}=\-\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{1\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\gamma\}\_\{1\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]Note that the biases for𝐕\{\{\\mathbf\{V\}\}\}are all equal if\(𝐕,γ¯\)\(\{\{\\mathbf\{V\}\}\},\\overline\{\\gamma\}\)has isotropyΔ​Sd\\Delta S\_\{d\}and so𝜸¯\\boldsymbol\{\\overline\{\\gamma\}\}is fixed bySd−1r×S1rS\_\{d\-1\}^\{r\}\\times S\_\{1\}^\{r\}—the isotropy group for𝜷¯\\boldsymbol\{\\overline\{\\beta\}\}\.

Setting the biases to zero,𝐁1=𝟎\{\\mathbf\{B\}\}^\{1\}=\{\\mathbf\{0\}\}and

A01=12​π​\(Λ10​\(ξ00\+\(q0−1\)​ξ⋆00\)−λ10\),A11=12​π​\(q0​Λ10​ξ10−λ11\)\{A\}^\{1\}\_\{0\}=\\frac\{1\}\{2\\pi\}\\big\(\\Lambda\_\{10\}\(\\xi^\{0\}\_\{0\}\+\(q\_\{0\}\-1\)\\xi^\{0\}\_\{\\star 0\}\)\-\\lambda\_\{10\}\\big\),\\quad\{A\}^\{1\}\_\{1\}=\\frac\{1\}\{2\\pi\}\\big\(q\_\{0\}\\Lambda\_\{10\}\\xi^\{0\}\_\{1\}\-\\lambda\_\{11\}\)which is seen to be correct using the formula forgrad​\(ℒ\)\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}\[[4](https://arxiv.org/html/2609.30661#bib.bib4)\]\.※

### 12\.7\.Critical point equations associated to rowtypei∈𝐫𝐬i\\in\{\{\\mathbf\{r\_\{s\}\}\}\}

In the table below, we add new coefficients to the table of section[10\.1](https://arxiv.org/html/2609.30661#S10.SS1)\. These will all be scalars with upper and lower case having the same meaning as before\. Fori∈𝐫𝐬i\\in\{\{\\mathbf\{r\_\{s\}\}\}\}define

Zi\\displaystyle Z\_\{i\}=\\displaystyle=\(−cot⁡\(Θi\)\+csc⁡\(Θi\)\)​β¯i\\displaystyle\\big\(\-\\cot\(\\Theta\_\{i\}\)\+\\csc\(\\Theta\_\{i\}\)\\big\)\\overline\{\\beta\}\_\{i\}zi\\displaystyle z\_\{i\}=\\displaystyle=−cot⁡\(θi\)​γ¯i\+csc⁡\(θi\)​β¯i,z^i=−cot⁡\(θi\)​β¯i\+csc⁡\(θi\)​γ¯i\\displaystyle\-\\cot\(\\theta\_\{i\}\)\\overline\{\\gamma\}\_\{i\}\+\\csc\(\\theta\_\{i\}\)\\overline\{\\beta\}\_\{i\},\\;\{\\widehat\{z\}\}\_\{i\}=\-\\cot\(\\theta\_\{i\}\)\\overline\{\\beta\}\_\{i\}\+\\csc\(\\theta\_\{i\}\)\\overline\{\\gamma\}\_\{i\}Ki\\displaystyle K\_\{i\}=\\displaystyle=β¯i2​\(csc2⁡\(Θi\)−csc⁡\(Θi\)​cot⁡\(Θi\)\)\\displaystyle\\overline\{\\beta\}\_\{i\}^\{2\}\\big\(\\csc^\{2\}\(\\Theta\_\{i\}\)\-\\csc\(\\Theta\_\{i\}\)\\cot\(\\Theta\_\{i\}\)\\big\)ki\\displaystyle k\_\{i\}=\\displaystyle=12​\[\(β¯i2\+γ¯i2\)​csc2⁡\(θi\)−2​β¯i​γ¯i​csc⁡\(θi\)​cot⁡\(θi\)\]\\displaystyle\\frac\{1\}\{2\}\\big\[\(\\overline\{\\beta\}\_\{i\}^\{2\}\+\\overline\{\\gamma\}\_\{i\}^\{2\}\)\\csc^\{2\}\(\\theta\_\{i\}\)\-2\\overline\{\\beta\}\_\{i\}\\overline\{\\gamma\}\_\{i\}\\csc\(\\theta\_\{i\}\)\\cot\(\\theta\_\{i\}\)\\big\]Li\\displaystyle L\_\{i\}=\\displaystyle=−π​erf​\(β¯i2\),ℓi=−π2​\[erf​\(β¯i2\)\+erf​\(γ¯i2\)\]\\displaystyle\-\\pi\\,\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\},\\;\\ell\_\{i\}=\-\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\gamma\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]EiX\\displaystyle E\_\{i\}^\{X\}=\\displaystyle=−2​π​β¯i​e−β¯i22​erfc​\(Zi2\)\\displaystyle\-\\sqrt\{2\\pi\}\\overline\{\\beta\}\_\{i\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{Z\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}eiX\\displaystyle e\_\{i\}^\{X\}=\\displaystyle=−π2​\[β¯i​e−γ¯i22​erfc​\(zi2\)\+γ¯i​e−β¯i22​erfc​\(z^i2\)\]\\displaystyle\-\\sqrt\{\\frac\{\\pi\}\{2\}\}\\left\[\\overline\{\\beta\}\_\{i\}e^\{\-\\frac\{\\overline\{\\gamma\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{z\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\overline\{\\gamma\}\_\{i\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\{\\widehat\{z\}\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]In casei=ji=j, andi∈𝐫𝐬i\\in\{\{\\mathbf\{r\_\{s\}\}\}\}, define

𝒰¯i=eiX\+β¯i​γ¯i​\(ℓi\+π−θ~i\)\+e−ki​sin⁡\(θi\)𝒱¯i=ℓi\+π−θ~i𝒲¯i=EiX\+β¯i2​\(Li\+π−Θ~i\)\+e−Ki​sin⁡\(Θi\)𝒴¯i=Li\+π−Θ~i\\begin\{aligned\} \\overline\{\\mathcal\{U\}\}\_\{i\}=&e\_\{i\}^\{X\}\+\\overline\{\\beta\}\_\{i\}\\overline\{\\gamma\}\_\{i\}\\big\(\\ell\_\{i\}\+\\pi\-\{\\widetilde\{\\theta\}\}\_\{i\}\\big\)\+e^\{\-k\_\{i\}\}\\sin\(\\theta\_\{i\}\)\\quad\\overline\{\\mathcal\{V\}\}\_\{i\}=\\ell\_\{i\}\+\\pi\-\{\\widetilde\{\\theta\}\}\_\{i\}\\\\ \\overline\{\\mathcal\{W\}\}\_\{i\}=&E\_\{i\}^\{X\}\+\\overline\{\\beta\}^\{2\}\_\{i\}\\big\(L\_\{i\}\+\\pi\-\{\\widetilde\{\\Theta\}\}\_\{i\}\\big\)\+e^\{\-K\_\{i\}\}\\sin\(\\Theta\_\{i\}\)\\quad\\overline\{\\mathcal\{Y\}\}\_\{i\}=L\_\{i\}\+\\pi\-\\widetilde\{\\Theta\}\_\{i\}\\end\{aligned\}
###### Definition 12\.13\.

Fori∈𝐫𝐬i\\in\{\{\\mathbf\{r\_\{s\}\}\}\}, define

1. \(1\)𝐰𝚺⋆i=∑a∈𝐢R,a\>I⁡\(i\)𝐰a\{\{\\mathbf\{w\}\}\}^\{i\}\_\{\\boldsymbol\{\\Sigma^\{\\star\}\}\}=\\sum\_\{a\\in\{\\mathbf\{i\}\}\_\{R\},a\>I\(i\)\}\{\{\\mathbf\{w\}\}\}^\{a\}\.
2. \(2\)𝐯𝚺⋆i=∑a∈𝐢R,a\>I⁡\(i\)𝐯a\{\{\\mathbf\{v\}\}\}^\{i\}\_\{\\boldsymbol\{\\Sigma^\{\\star\}\}\}=\\sum\_\{a\\in\{\\mathbf\{i\}\}\_\{R\},a\>I\(i\)\}\{\{\\mathbf\{v\}\}\}^\{a\}\.

Both𝐰𝚺⋆i\{\{\\mathbf\{w\}\}\}^\{i\}\_\{\\boldsymbol\{\\Sigma^\{\\star\}\}\}and𝐯𝚺⋆i\{\{\\mathbf\{v\}\}\}^\{i\}\_\{\\boldsymbol\{\\Sigma^\{\\star\}\}\}are defined to be zero ifi∉𝐫𝐬i\\notin\{\{\\mathbf\{r\_\{s\}\}\}\}\.

###### Remark 12\.14\.

Unlike wheni∉𝐫𝐬i\\notin\{\{\\mathbf\{r\_\{s\}\}\}\}, the vectors𝐰𝚺⋆i\{\{\\mathbf\{w\}\}\}^\{i\}\_\{\\boldsymbol\{\\Sigma^\{\\star\}\}\}\(resp\.𝐯𝚺⋆i\{\{\\mathbf\{v\}\}\}^\{i\}\_\{\\boldsymbol\{\\Sigma^\{\\star\}\}\}\) are not column averages of the set of rows𝐰\{\{\\mathbf\{w\}\}\}\(resp\.𝐯\{\{\\mathbf\{v\}\}\}\) of rowtypeiisince𝐰I⁡\(i\)\{\{\\mathbf\{w\}\}\}^\{I\(i\)\}\(resp\.𝐯I⁡\(i\)\{\{\\mathbf\{v\}\}\}^\{I\(i\)\}\) is omitted\. As a result the block of𝐰𝚺⋆i\{\{\\mathbf\{w\}\}\}^\{i\}\_\{\\boldsymbol\{\\Sigma^\{\\star\}\}\}defined by the intersection with the diagonal block of𝐖\{\{\\mathbf\{W\}\}\}with isotropyΔ​Sqi\\Delta S\_\{q\_\{i\}\}is not a constant block—there are two different coordinate values\.✠\\maltese

###### Proposition 12\.15\.

\(Notation and assumptions as above\.\) Fori∈𝐫𝐬i\\in\{\{\\mathbf\{r\_\{s\}\}\}\}\(ori∈𝐑i\\in\{\\mathbf\{R\}\}\), componentiiofgrad​\(ℒ\)​\(𝐖\)\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}\(\{\{\\mathbf\{W\}\}\}\)is given by

\(12\.51\)grad​\(ℒ\)​\(𝐖\)i=Γi​𝐰i−𝐀i\+𝐁i\+𝛀\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}\(\{\{\\mathbf\{W\}\}\}\)^\{i\}=\\Gamma\_\{i\}\{\{\\mathbf\{w\}\}\}^\{i\}\-\{\\mathbf\{A\}\}^\{i\}\+\{\\mathbf\{B\}\}^\{i\}\+\\boldsymbol\{\\Omega\}whereΓi,𝐀i,𝐁i\\Gamma\_\{i\},\{\\mathbf\{A\}\}^\{i\},\{\\mathbf\{B\}\}^\{i\}and𝛀\\boldsymbol\{\\Omega\}are defined by:

Γi\\displaystyle\\Gamma\_\{i\}=\\displaystyle=Xi2\+12​π​\[\(qi−1\)​\(‖𝐰i‖​e−Ki​sin⁡\(Θi\)−‖𝐯i‖​e−ki​sin⁡\(θi\)\)−‖𝐯i‖​e−ki​i​sin⁡\(λi​i\)‖𝐰i‖\]\\displaystyle\\frac\{X\_\{i\}\}\{2\}\+\\frac\{1\}\{2\\pi\}\\left\[\\frac\{\(q\_\{i\}\-1\)\\big\(\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|e^\{\-K\_\{i\}\}\\sin\(\\Theta\_\{i\}\)\-\\\|\{\{\\mathbf\{v\}\}\}^\{i\}\\\|e^\{\-k\_\{i\}\}\\sin\(\\theta\_\{i\}\)\\big\)\-\\\|\{\{\\mathbf\{v\}\}\}^\{i\}\\\|e^\{\-k\_\{ii\}\}\\sin\(\\lambda\_\{ii\}\)\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\\right\]\+12​π​\[\(qi−1\)​\(‖𝐰i‖​EiX−‖𝐯i‖​eiX\)−‖𝐯i‖​ei​iX‖𝐰i‖\]\\displaystyle\+\\frac\{1\}\{2\\pi\}\\left\[\\frac\{\(q\_\{i\}\-1\)\\big\(\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|E\_\{i\}^\{X\}\-\\\|\{\{\\mathbf\{v\}\}\}^\{i\}\\\|e\_\{i\}^\{X\}\\big\)\-\\\|\{\{\\mathbf\{v\}\}\}^\{i\}\\\|e^\{X\}\_\{ii\}\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\\right\]\+β¯i2​π​\[\(qi−1\)​\[β¯i​‖𝐰i‖​\(Li\+\(π−Θ~i\)\)−γ¯i​‖𝐯i‖​\(ℓi\+\(π−θ~i\)\)\]−γ¯i​‖𝐯i‖​\(ℓi​i\+\(π−λ~i​i\)\)‖𝐰i‖\]\\displaystyle\+\\frac\{\\overline\{\\beta\}\_\{i\}\}\{2\\pi\}\\left\[\\frac\{\(q\_\{i\}\-1\)\\big\[\\overline\{\\beta\}\_\{i\}\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\big\(L\_\{i\}\+\(\\pi\-\{\\widetilde\{\\Theta\}\}\_\{i\}\)\\big\)\-\\overline\{\\gamma\}\_\{i\}\\\|\{\{\\mathbf\{v\}\}\}^\{i\}\\\|\\big\(\\ell\_\{i\}\+\(\\pi\-\{\\widetilde\{\\theta\}\}\_\{i\}\)\\big\)\\big\]\-\\overline\{\\gamma\}\_\{i\}\\\|\{\{\\mathbf\{v\}\}\}^\{i\}\\\|\\big\(\\ell\_\{ii\}\+\(\\pi\-\{\\widetilde\{\\lambda\}\}\_\{ii\}\)\\big\)\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\\right\]\+12​π​\[∑j∈𝐑,j≠iqj​‖𝐰j‖​e−Ki​j​sin⁡\(Λi​j\)‖𝐰i‖−∑j∈𝐟,j≠iqj​‖𝐯j‖​e−ki​j​sin⁡\(λi​j\)‖𝐰i‖\]\\displaystyle\+\\frac\{1\}\{2\\pi\}\\left\[\\sum\_\{j\\in\{\\mathbf\{R\}\},j\\neq i\}q\_\{j\}\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|e^\{\-K\_\{ij\}\}\\sin\(\\Lambda\_\{ij\}\)\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\-\\sum\_\{j\\in\{\\mathbf\{f\}\},j\\neq i\}q\_\{j\}\\frac\{\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|e^\{\-k\_\{ij\}\}\\sin\(\\lambda\_\{ij\}\)\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\\right\]\+12​π​\[∑j∈𝐑,j≠iqj​‖𝐰j‖​Ei​jX‖𝐰i‖−∑j∈𝐟,j≠iqj​‖𝐯j‖​ei​jX‖𝐰i‖\]\\displaystyle\+\\frac\{1\}\{2\\pi\}\\left\[\\sum\_\{j\\in\{\\mathbf\{R\}\},j\\neq i\}q\_\{j\}\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|E\_\{ij\}^\{X\}\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\-\\sum\_\{j\\in\{\\mathbf\{f\}\},j\\neq i\}q\_\{j\}\\frac\{\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|e\_\{ij\}^\{X\}\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\\right\]\+β¯i2​π​\[∑j∈𝐑,j≠iqj​β¯j​‖𝐰j‖​\(\(π−Λ~i​j\)\+Li​j\)‖𝐰i‖−∑j∈𝐟,j≠iqj​γ¯j​‖𝐯j‖​\(\(π−λ~i​j\)\+ℓi​j\)‖𝐰i‖\]\\displaystyle\+\\frac\{\\overline\{\\beta\}\_\{i\}\}\{2\\pi\}\\left\[\\sum\_\{j\\in\{\\mathbf\{R\}\},j\\neq i\}q\_\{j\}\\frac\{\\overline\{\\beta\}\_\{j\}\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\\big\(\(\\pi\-\{\\widetilde\{\\Lambda\}\}\_\{ij\}\)\+L\_\{ij\}\\big\)\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\-\\sum\_\{j\\in\{\\mathbf\{f\}\},j\\neq i\}q\_\{j\}\\frac\{\\overline\{\\gamma\}\_\{j\}\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|\\big\(\(\\pi\-\{\\widetilde\{\\lambda\}\}\_\{ij\}\)\+\\ell\_\{ij\}\\big\)\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\\right\]𝐀i\\displaystyle\{\\mathbf\{A\}\}^\{i\}=\\displaystyle=12​π​\(∑j∈𝐑,j≠iΛ~i​j​𝐰Σj−∑j∈𝐟,j≠iλ~i​j​𝐯Σj\)\+12​π​\(Θ~i​𝐰𝚺⋆i−θ~i​𝐯𝚺⋆i−λ~i​i​𝐯i\)\\displaystyle\\frac\{1\}\{2\\pi\}\\left\(\\sum\_\{j\\in\{\\mathbf\{R\}\},j\\neq i\}\{\\widetilde\{\\Lambda\}\}\_\{ij\}\{\{\\mathbf\{w\}\}\}\_\{\\Sigma\}^\{j\}\-\\sum\_\{j\\in\{\\mathbf\{f\}\},j\\neq i\}\{\\widetilde\{\\lambda\}\}\_\{ij\}\{\{\\mathbf\{v\}\}\}\_\{\\Sigma\}^\{j\}\\right\)\+\\frac\{1\}\{2\\pi\}\\big\(\{\\widetilde\{\\Theta\}\}\_\{i\}\{\{\\mathbf\{w\}\}\}^\{i\}\_\{\\boldsymbol\{\\Sigma^\{\\star\}\}\}\-\{\\widetilde\{\\theta\}\}\_\{i\}\{\{\\mathbf\{v\}\}\}^\{i\}\_\{\\boldsymbol\{\\Sigma^\{\\star\}\}\}\-\{\\widetilde\{\\lambda\}\}\_\{ii\}\{\{\\mathbf\{v\}\}\}^\{i\}\\big\)𝐁i\\displaystyle\{\\mathbf\{B\}\}^\{i\}=\\displaystyle=12​π​\(∑j∈𝐑,j≠iLi​j​𝐰Σj−∑j∈𝐟,j≠iℓi​j​𝐯Σj\)\+12​π​\(Li​𝐰𝚺⋆i−ℓi​𝐯𝚺⋆i−ℓi​i​𝐯i\)\\displaystyle\\frac\{1\}\{2\\pi\}\\left\(\\sum\_\{j\\in\{\\mathbf\{R\}\},j\\neq i\}L\_\{ij\}\{\{\\mathbf\{w\}\}\}\_\{\\Sigma\}^\{j\}\-\\sum\_\{j\\in\{\\mathbf\{f\}\},j\\neq i\}\\ell\_\{ij\}\{\{\\mathbf\{v\}\}\}\_\{\\Sigma\}^\{j\}\\right\)\+\\frac\{1\}\{2\\pi\}\\big\(L\_\{i\}\{\{\\mathbf\{w\}\}\}^\{i\}\_\{\\boldsymbol\{\\Sigma^\{\\star\}\}\}\-\\ell\_\{i\}\{\{\\mathbf\{v\}\}\}^\{i\}\_\{\\boldsymbol\{\\Sigma^\{\\star\}\}\}\-\\ell\_\{ii\}\{\{\\mathbf\{v\}\}\}^\{i\}\\big\)𝛀\\displaystyle\{\\boldsymbol\{\\Omega\}\}=\\displaystyle=12​\(∑j∈𝐑𝐰Σj−∑j∈𝐟𝐯Σj\)=12​\(∑a∈𝐤𝐰a−∑b∈𝐝𝐯b\)\\displaystyle\\frac\{1\}\{2\}\\left\(\\sum\_\{j\\in\{\\mathbf\{R\}\}\}\{\{\\mathbf\{w\}\}\}\_\{\\Sigma\}^\{j\}\-\\sum\_\{j\\in\{\\mathbf\{f\}\}\}\{\{\\mathbf\{v\}\}\}\_\{\\Sigma\}^\{j\}\\right\)=\\frac\{1\}\{2\}\\left\(\\sum\_\{a\\in\{\\mathbf\{k\}\}\}\{\{\\mathbf\{w\}\}\}^\{a\}\-\\sum\_\{b\\in\{\\mathbf\{d\}\}\}\{\{\\mathbf\{v\}\}\}^\{b\}\\right\)

###### Proof\.

Ifi∈𝐑i\\in\{\\mathbf\{R\}\}, then

grad​\(ℒ\)​\(𝐖\)i=\(X¯i2\+12​π​\[\(qi−1\)​\(‖𝐰i‖​𝒲¯i−‖𝐯i‖​𝒰¯i\)−‖𝐯i‖​𝒰¯i​i‖𝐰i‖\]\)​𝐰i\+12​π​\(∑j∈𝐑,j≠iqj​‖𝐰j‖​𝒲¯i​j‖𝐰i‖−∑j∈𝐟,j≠iqj​‖𝐯j‖​𝒰¯i​j‖𝐰i‖\)​𝐰i\+12​π​\[∑j∈𝐑,j≠i𝒴¯i​j​𝐰Σj\+𝒴¯i​𝐰𝚺i\]−12​π​\[∑j∈𝐟𝒱¯i​j​𝐯Σj\+𝒱¯i​𝐯𝚺i\]\\begin\{aligned\} \{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}\(\{\{\\mathbf\{W\}\}\}\)^\{i\}=&\\left\(\\frac\{\\overline\{X\}\_\{i\}\}\{2\}\+\\frac\{1\}\{2\\pi\}\\left\[\\frac\{\(q\_\{i\}\-1\)\\big\(\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\overline\{\\mathcal\{W\}\}\_\{i\}\-\\\|\{\{\\mathbf\{v\}\}\}^\{i\}\\\|\\overline\{\\mathcal\{U\}\}\_\{i\}\\big\)\-\\\|\{\{\\mathbf\{v\}\}\}^\{i\}\\\|\\overline\{\\mathcal\{U\}\}\_\{ii\}\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\\right\]\\right\)\{\{\\mathbf\{w\}\}\}^\{i\}\\\\ &\+\\frac\{1\}\{2\\pi\}\\left\(\\sum\_\{j\\in\{\\mathbf\{R\}\},j\\neq i\}q\_\{j\}\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\\overline\{\\mathcal\{W\}\}\_\{ij\}\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\-\\sum\_\{j\\in\{\\mathbf\{f\}\},j\\neq i\}q\_\{j\}\\frac\{\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|\\overline\{\\mathcal\{U\}\}\_\{ij\}\}\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\\right\)\{\{\\mathbf\{w\}\}\}^\{i\}\\\\ &\+\\frac\{1\}\{2\\pi\}\\left\[\\sum\_\{j\\in\{\\mathbf\{R\}\},j\\neq i\}\\overline\{\\mathcal\{Y\}\}\_\{ij\}\{\{\\mathbf\{w\}\}\}\_\{\\Sigma\}^\{j\}\+\\overline\{\\mathcal\{Y\}\}\_\{i\}\{\{\\mathbf\{w\}\}\}\_\{\\boldsymbol\{\\Sigma\}\}^\{i\}\\right\]\-\\frac\{1\}\{2\\pi\}\\left\[\\sum\_\{j\\in\{\\mathbf\{f\}\}\}\\overline\{\\mathcal\{V\}\}\_\{ij\}\{\{\\mathbf\{v\}\}\}\_\{\\Sigma\}^\{j\}\+\\overline\{\\mathcal\{V\}\}\_\{i\}\{\{\\mathbf\{v\}\}\}\_\{\\boldsymbol\{\\Sigma\}\}^\{i\}\\right\]\\end\{aligned\}where the terms involving𝒲¯i,𝒰¯i,𝒴¯i,𝒱¯i\\overline\{\\mathcal\{W\}\}\_\{i\},\\overline\{\\mathcal\{U\}\}\_\{i\},\\overline\{\\mathcal\{Y\}\}\_\{i\},\\overline\{\\mathcal\{V\}\}\_\{i\}are zero ifi∉𝐫𝐬i\\notin\{\{\\mathbf\{r\_\{s\}\}\}\}\. The proof now follows that of Proposition[12\.10](https://arxiv.org/html/2609.30661#S12.Thmlemma10)but using the additional terms𝒲¯i\\overline\{\\mathcal\{W\}\}\_\{i\},𝒰¯i\\overline\{\\mathcal\{U\}\}\_\{i\},𝒴¯i,𝒱¯i\\overline\{\\mathcal\{Y\}\}\_\{i\},\\overline\{\\mathcal\{V\}\}\_\{i\}wheni∈𝐫𝐬i\\in\{\{\\mathbf\{r\_\{s\}\}\}\}\(see below and Section[15\.1](https://arxiv.org/html/2609.30661#S15.SS1)for examples\)\. ∎

###### Remark 12\.16\.

The theorem is valid for alli∈𝐑i\\in\{\\mathbf\{R\}\}and general targets𝐕∈M⁡\(k,d\)\{\{\\mathbf\{V\}\}\}\\in M\(k,d\)\(assume no parallel rows\)\. Indeed, if there are no column symmetries then the statement of the theorem is*simpler*\. For example, there will be no second term in the expressions for𝐀i\{\\mathbf\{A\}\}^\{i\}and𝐁i\{\\mathbf\{B\}\}^\{i\}and𝐰Σi\{\{\\mathbf\{w\}\}\}^\{i\}\_\{\\Sigma\}gets replaced by𝐰i\{\{\\mathbf\{w\}\}\}^\{i\}and𝐯Σj\{\{\\mathbf\{v\}\}\}^\{j\}\_\{\\Sigma\}by𝐯j\{\{\\mathbf\{v\}\}\}^\{j\}—every row has a unique rowtype\. However, if there are no column symmetries then the computational cost of finding spurious minima increases rapidly withddas there is no scope for reducing the problem to a low dimensional fixed point space\. Granted the benefits of the symmetry assumption, there is necessarily a cost to that assumption: the equations are more complex\. However, it is possible to break the symmetry in a diagonal blockAAand use path based methods to examine the effects of forced symmetry breaking within the blockAA\. As long as the asymmetric block size remains fixed,ddcan be increased and it is possible to analyze the limiting behaviour asd→∞d\{\\rightarrow\}\\inftyto obtain FPS for families of critical points parametrized bydd\. Put another way, if we start with a symmetric family of spurious minima defined ford≥d0d\\geq d\_\{0\}and with isotropyΔ​Sd−1\\Delta S\_\{d\-1\}, it is possible to study numerically and analytically forced symmetry breaking to a family with isotropyΔ​Sd−a=Δ​Sd−a×S1a\\Delta S\_\{d\-a\}=\\Delta S\_\{d\-a\}\\times S\_\{1\}^\{a\}\. Again the equations are more complex than those of the fully asymmetric system but there is the promise of proving strong analytical results based on FPS—for example, asymptotics of the loss\. We indicate one or two examples in Section[14](https://arxiv.org/html/2609.30661#S14)\.✠\\maltese

### 12\.8\.The terms𝐀i,𝐁i\{\\mathbf\{A\}\}^\{i\},\{\\mathbf\{B\}\}^\{i\}wheni∈𝐫𝐬i\\in\{\{\\mathbf\{r\_\{s\}\}\}\}and𝐕=Id\{\{\\mathbf\{V\}\}\}=I\_\{d\}

Ifi∈𝐫𝐬i\\in\{\{\\mathbf\{r\_\{s\}\}\}\}, then the row vectors𝐀i,𝐁i\{\\mathbf\{A\}\}^\{i\},\{\\mathbf\{B\}\}^\{i\}have a block decomposition induced from the natural block decomposition onM​\(k,d\)G𝐪ΔM\(k,d\)^\{G^\{\\Delta\}\_\{\\mathbf\{q\}\}\}\. However, ifi∈𝐫𝐬i\\in\{\{\\mathbf\{r\_\{s\}\}\}\}, thenqi\>1q\_\{i\}\>1and blockiiwill intersect theqi×qiq\_\{i\}\\times q\_\{i\}\-diagonal block associated toii\(decomposition of𝐖∈M​\(k,d\)G𝐪Δ\{\{\\mathbf\{W\}\}\}\\in M\(k,d\)^\{G^\{\\Delta\}\_\{\\mathbf\{q\}\}\}into block diagonal form\) and the entries of this block will not all be equal\. Thus, the1×qi1\\times q\_\{i\}block of𝐀i\{\\mathbf\{A\}\}^\{i\}will be\[Aii,\(A⋆ii\)qi−1\]\[A^\{i\}\_\{i\},\(A^\{i\}\_\{\\star i\}\)^\{q\_\{i\}\-1\}\]with

Aii\\displaystyle A^\{i\}\_\{i\}=\\displaystyle=12​π​\[\(qi−1\)​Θ~i​ξ⋆ii\+∑α≠iqα​Λ~i​α​ξiα−λ~i​i\],qi≥1\\displaystyle\\frac\{1\}\{2\\pi\}\\left\[\(q\_\{i\}\-1\)\{\\widetilde\{\\Theta\}\}\_\{i\}\\xi^\{i\}\_\{\\star i\}\+\\sum\_\{\\alpha\\neq i\}q\_\{\\alpha\}\{\\widetilde\{\\Lambda\}\}\_\{i\\alpha\}\\xi^\{\\alpha\}\_\{i\}\-\{\\widetilde\{\\lambda\}\}\_\{ii\}\\right\],\\;\\;q\_\{i\}\\geq 1=\\displaystyle=12​π​\[∑α≠iqα​Λ~i​α​ξiα−λ~i​i\],if​qi=1\\displaystyle\\frac\{1\}\{2\\pi\}\\left\[\\sum\_\{\\alpha\\neq i\}q\_\{\\alpha\}\{\\widetilde\{\\Lambda\}\}\_\{i\\alpha\}\\xi^\{\\alpha\}\_\{i\}\-\{\\widetilde\{\\lambda\}\}\_\{ii\}\\right\],\\;\\;\\text\{if \}q\_\{i\}=1A⋆ii\\displaystyle A^\{i\}\_\{\\star i\}=\\displaystyle=12​π​\[Θ~i​ξii\+\(qi−2\)​Θ~i​ξ⋆ii\+∑α≠iqα​Λ~i​α​ξiα−θ~i\],qi≥2\\displaystyle\\frac\{1\}\{2\\pi\}\\left\[\{\\widetilde\{\\Theta\}\}\_\{i\}\\xi^\{i\}\_\{i\}\+\(q\_\{i\}\-2\)\{\\widetilde\{\\Theta\}\}\_\{i\}\\xi^\{i\}\_\{\\star i\}\+\\sum\_\{\\alpha\\neq i\}q\_\{\\alpha\}\{\\widetilde\{\\Lambda\}\}\_\{i\\alpha\}\\xi^\{\\alpha\}\_\{i\}\-\{\\widetilde\{\\theta\}\}\_\{i\}\\right\],\\;q\_\{i\}\\geq 2Finally, the angle termsAjiA^\{i\}\_\{j\}whenj≠ij\\neq i\.

Aji\\displaystyle A^\{i\}\_\{j\}=\\displaystyle=12​π​\[∑α≠jqα​Λ~i​α​ξjα\+\(qi−1\)​Θ~i​ξji\+Λ~i​j​\(\(qj−1\)​ξ⋆jj\+ξjj\)−λ~i​j\],qi≥2\\displaystyle\\frac\{1\}\{2\\pi\}\\left\[\\sum\_\{\\alpha\\neq j\}q\_\{\\alpha\}\{\\widetilde\{\\Lambda\}\}\_\{i\\alpha\}\\xi^\{\\alpha\}\_\{j\}\+\(q\_\{i\}\-1\)\{\\widetilde\{\\Theta\}\}\_\{i\}\\xi^\{i\}\_\{j\}\+\{\\widetilde\{\\Lambda\}\}\_\{ij\}\(\(q\_\{j\}\-1\)\\xi\_\{\\star j\}^\{j\}\+\\xi\_\{j\}^\{j\}\)\-\{\\widetilde\{\\lambda\}\}\_\{ij\}\\right\],\\;q\_\{i\}\\geq 2=\\displaystyle=12​π​\[∑α≠jqα​Λ~i​α​ξjα\+Λ~i​j​\(\(qj−1\)​ξ⋆jj\+ξjj\)−λ~i​j\],qi=1\\displaystyle\\frac\{1\}\{2\\pi\}\\left\[\\sum\_\{\\alpha\\neq j\}q\_\{\\alpha\}\{\\widetilde\{\\Lambda\}\}\_\{i\\alpha\}\\xi^\{\\alpha\}\_\{j\}\+\{\\widetilde\{\\Lambda\}\}\_\{ij\}\(\(q\_\{j\}\-1\)\\xi\_\{\\star j\}^\{j\}\+\\xi\_\{j\}^\{j\}\)\-\{\\widetilde\{\\lambda\}\}\_\{ij\}\\right\],\\;q\_\{i\}=1These expressions extend those given fori∉𝐫𝐬i\\notin\{\{\\mathbf\{r\_\{s\}\}\}\}and apply wheneveri∈𝐑i\\in\{\\mathbf\{R\}\},j∈𝐟j\\in\{\\mathbf\{f\}\}\.

The results for𝐁i\{\\mathbf\{B\}\}^\{i\},i∈𝐫𝐬i\\in\{\{\\mathbf\{r\_\{s\}\}\}\}, are similar:

Bii\\displaystyle B^\{i\}\_\{i\}=\\displaystyle=12​π​\[\(qi−1\)​Li​ξ⋆ii\+∑α≠iqα​Li​α​ξiα−ℓi\],qi≥1\\displaystyle\\frac\{1\}\{2\\pi\}\\left\[\(q\_\{i\}\-1\)L\_\{i\}\\xi^\{i\}\_\{\\star i\}\+\\sum\_\{\\alpha\\neq i\}q\_\{\\alpha\}L\_\{i\\alpha\}\\xi^\{\\alpha\}\_\{i\}\-\\ell\_\{i\}\\right\],\\;q\_\{i\}\\geq 1=\\displaystyle=\[∑α≠iqα​Li​α​ξiα−ℓi\],if​qi=1\\displaystyle\\left\[\\sum\_\{\\alpha\\neq i\}q\_\{\\alpha\}L\_\{i\\alpha\}\\xi^\{\\alpha\}\_\{i\}\-\\ell\_\{i\}\\right\],\\;\\text\{if \}q\_\{i\}=1B⋆ii\\displaystyle B^\{i\}\_\{\\star i\}=\\displaystyle=12​π​\[Li​ξii\+\(qi−2\)​Li​ξ⋆ii\+∑α≠iqα​Li​α​ξiα−ℓi\],qi≥2\\displaystyle\\frac\{1\}\{2\\pi\}\\left\[L\_\{i\}\\xi^\{i\}\_\{i\}\+\(q\_\{i\}\-2\)L\_\{i\}\\xi^\{i\}\_\{\\star i\}\+\\sum\_\{\\alpha\\neq i\}q\_\{\\alpha\}L\_\{i\\alpha\}\\xi^\{\\alpha\}\_\{i\}\-\\ell\_\{i\}\\right\],\\;q\_\{i\}\\geq 2Ifj≠ij\\neq i, then

Bji\\displaystyle B^\{i\}\_\{j\}=\\displaystyle=12​π​\[∑α≠jqα​Li​α​ξjα\+\(qi−1\)​Li​ξji\+Li​j​\(\(qj−1\)​ξ⋆jj\+ξjj\)−ℓi​j\]\\displaystyle\\frac\{1\}\{2\\pi\}\\left\[\\sum\_\{\\alpha\\neq j\}q\_\{\\alpha\}L\_\{i\\alpha\}\\xi^\{\\alpha\}\_\{j\}\+\(q\_\{i\}\-1\)L\_\{i\}\\xi^\{i\}\_\{j\}\+L\_\{ij\}\(\(q\_\{j\}\-1\)\\xi\_\{\\star j\}^\{j\}\+\\xi\_\{j\}^\{j\}\)\-\\ell\_\{ij\}\\right\]

### 12\.9\.Population loss with symmetric target

The section concludes with expressions for the population lossℒ\\mathcal\{L\}and theβ¯\\overline\{\\beta\}\-derivatives ofℒ\\mathcal\{L\}when the network is biased and𝐖∈M⁡\(k,d\)\{\{\\mathbf\{W\}\}\}\\in M\(k,d\)has isotropy isG𝐪Δ⊂Δ​SdG^\{\\Delta\}\_\{\{\\mathbf\{q\}\}\}\\subset\\Delta S\_\{d\}\. As above, the target𝐕\{\{\\mathbf\{V\}\}\}is assumed to be𝐈d∈M⁡\(d,d\)\{\\mathbf\{I\}\}\_\{d\}\\in M\(d,d\)\(zero bias\)\.k≥dk\\geq dand there areRRrowtypes,fffactors,rsr\_\{s\}factors that have symmetrySqiS\_\{q\_\{i\}\}withqi\>1q\_\{i\}\>1, and𝐑∖𝐟=𝐫𝐦\{\\mathbf\{R\}\}\\smallsetminus\{\\mathbf\{f\}\}=\{\\mathbf\{r\_\{m\}\}\}labels neurons corresponding to the lastk−dk\-drows of𝐖\{\{\\mathbf\{W\}\}\}\.

Recall that if𝐖∈M​\(k,d\)G𝐪Δ\{\{\\mathbf\{W\}\}\}\\in M\(k,d\)^\{G^\{\\Delta\}\_\{\{\\mathbf\{q\}\}\}\}andi∈𝐑i\\in\{\\mathbf\{R\}\}is a rowtype, then𝐰i\{\{\\mathbf\{w\}\}\}^\{i\}denotes the first row in𝐖\{\{\\mathbf\{W\}\}\}of rowtypeii\(that is,𝐰I⁡\(i\)\{\{\\mathbf\{w\}\}\}^\{I\(i\)\}\)\. Ifi∈𝐫𝐬i\\in\{\{\\mathbf\{r\_\{s\}\}\}\}, let𝐰~i\\widetilde\{\{\{\\mathbf\{w\}\}\}\}^\{i\}denote any row of rowtypeiiother than𝐰i\{\{\\mathbf\{w\}\}\}^\{i\}\. For example,𝐰I⁡\(i\)\+1\{\{\\mathbf\{w\}\}\}^\{I\(i\)\+1\}\. Ifqiq\_\{i\}denotes the number of rows of rowtypeii, thenqi=1q\_\{i\}=1iffi∉𝐫𝐬i\\notin\{\{\\mathbf\{r\_\{s\}\}\}\}\.

The population lossℒ\\mathcal\{L\}comprises termsf⁡\(𝐰,𝐰′\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{w\}\}\}^\{\\prime\}\),f⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)andf⁡\(𝐯,𝐯′\)f\(\{\{\\mathbf\{v\}\}\},\{\{\\mathbf\{v\}\}\}^\{\\prime\}\)\. Each of these terms can be described in terms of rowtype\.

#### Termsf⁡\(𝐰,𝐰′\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{w\}\}\}^\{\\prime\}\)

- \(1a\)Ifi∈𝐑i\\in\{\\mathbf\{R\}\}, then ∑a∈I⁡\(i\)f⁡\(𝐰a,𝐰a\)=qi​f​\(𝐰i,𝐰i\)\\sum\_\{a\\in I\(i\)\}f\(\{\{\\mathbf\{w\}\}\}^\{a\},\{\{\\mathbf\{w\}\}\}^\{a\}\)=q\_\{i\}f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{i\}\)
- \(1b\)Ifi∈𝐫𝐬i\\in\{\{\\mathbf\{r\_\{s\}\}\}\}, then ∑a,b∈I⁡\(i\),a≠bf⁡\(𝐰a,𝐰b\)=qi​\(qi−1\)​f​\(𝐰i,𝐰~i\)\\sum\_\{a,b\\in I\(i\),a\\neq b\}f\(\{\{\\mathbf\{w\}\}\}^\{a\},\{\{\\mathbf\{w\}\}\}^\{b\}\)=q\_\{i\}\(q\_\{i\}\-1\)f\(\{\{\\mathbf\{w\}\}\}^\{i\},\\widetilde\{\{\{\\mathbf\{w\}\}\}\}^\{i\}\)
- \(1c\)Ifi≠j,i,j∈𝐑i\\neq j,i,j\\in\{\\mathbf\{R\}\}, then ∑a∈I⁡\(i\),b∈I⁡\(j\)\(f⁡\(𝐰a,𝐰b\)\+f⁡\(𝐰b,𝐰a\)\)=2​qi​qj​f​\(𝐰i,𝐰j\)\\sum\_\{a\\in I\(i\),b\\in I\(j\)\}\\big\(f\(\{\{\\mathbf\{w\}\}\}^\{a\},\{\{\\mathbf\{w\}\}\}^\{b\}\)\+f\(\{\{\\mathbf\{w\}\}\}^\{b\},\{\{\\mathbf\{w\}\}\}^\{a\}\)\\big\)=2q\_\{i\}q\_\{j\}f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{j\}\)

#### Termsf⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)

- \(2a\)Ifi∈𝐟i\\in\{\\mathbf\{f\}\}, then ∑a∈I⁡\(i\)f⁡\(𝐰a,𝐯a\)=qi​f​\(𝐰i,𝐯i\)\\sum\_\{a\\in I\(i\)\}f\(\{\{\\mathbf\{w\}\}\}^\{a\},\{\{\\mathbf\{v\}\}\}^\{a\}\)=q\_\{i\}f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{i\}\)
- \(2b\)Ifi∈𝐫𝐬i\\in\{\{\\mathbf\{r\_\{s\}\}\}\}, then ∑a,b∈I⁡\(i\),a≠bf⁡\(𝐰a,𝐯b\)=qi​\(qi−1\)​f​\(𝐰i,𝐯~i\)\\sum\_\{a,b\\in I\(i\),a\\neq b\}f\(\{\{\\mathbf\{w\}\}\}^\{a\},\{\{\\mathbf\{v\}\}\}^\{b\}\)=q\_\{i\}\(q\_\{i\}\-1\)f\(\{\{\\mathbf\{w\}\}\}^\{i\},\\widetilde\{\{\{\\mathbf\{v\}\}\}\}^\{i\}\)
- \(2c\)Ifi∈𝐑,j∈𝐟i\\in\{\\mathbf\{R\}\},j\\in\{\\mathbf\{f\}\},i≠ji\\neq j, then ∑a∈I⁡\(i\),b∈I⁡\(j\)f⁡\(𝐰a,𝐯b\)=qi​qj​f​\(𝐰i,𝐯j\)\\sum\_\{a\\in I\(i\),b\\in I\(j\)\}f\(\{\{\\mathbf\{w\}\}\}^\{a\},\{\{\\mathbf\{v\}\}\}^\{b\}\)=q\_\{i\}q\_\{j\}f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{j\}\)

#### Termsf⁡\(𝐯,𝐯′\)f\(\{\{\\mathbf\{v\}\}\},\{\{\\mathbf\{v\}\}\}^\{\\prime\}\)

- \(3a\)Ifi∈𝐟i\\in\{\\mathbf\{f\}\}, then ∑a∈I⁡\(i\)f⁡\(𝐯a,𝐯a\)=qi​f​\(𝐯i,𝐯i\)\\sum\_\{a\\in I\(i\)\}f\(\{\{\\mathbf\{v\}\}\}^\{a\},\{\{\\mathbf\{v\}\}\}^\{a\}\)=q\_\{i\}f\(\{\{\\mathbf\{v\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{i\}\)
- \(3b\)Ifi∈𝐫𝐬i\\in\{\{\\mathbf\{r\_\{s\}\}\}\}, then ∑a,b∈I⁡\(i\),a≠bf⁡\(𝐯a,𝐯b\)=qi​\(qi−1\)​f​\(𝐯i,𝐯~i\)\\sum\_\{a,b\\in I\(i\),a\\neq b\}f\(\{\{\\mathbf\{v\}\}\}^\{a\},\{\{\\mathbf\{v\}\}\}^\{b\}\)=q\_\{i\}\(q\_\{i\}\-1\)f\(\{\{\\mathbf\{v\}\}\}^\{i\},\\widetilde\{\{\{\\mathbf\{v\}\}\}\}^\{i\}\)
- \(3c\)Ifi<j,i,j∈𝐟i<j,i,j\\in\{\\mathbf\{f\}\}, then ∑a∈I⁡\(i\)​b∈I⁡\(j\)f⁡\(𝐯a,𝐯b\)=∑a∈I⁡\(i\)​b∈I⁡\(j\)f⁡\(𝐯b,𝐯a\)=qi​qj​f​\(𝐯i,𝐯j\)\\sum\_\{a\\in I\(i\)b\\in I\(j\)\}f\(\{\{\\mathbf\{v\}\}\}^\{a\},\{\{\\mathbf\{v\}\}\}^\{b\}\)=\\sum\_\{a\\in I\(i\)b\\in I\(j\)\}f\(\{\{\\mathbf\{v\}\}\}^\{b\},\{\{\\mathbf\{v\}\}\}^\{a\}\)=q\_\{i\}q\_\{j\}f\(\{\{\\mathbf\{v\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{j\}\)

###### Remark 12\.17\.

It is assumed here that\{𝐯a\}a∈𝐝\\\{\{\{\\mathbf\{v\}\}\}^\{a\}\\\}\_\{a\\in\{\\mathbf\{d\}\}\}is an orthonormal basis of\(ℝd\)⋆≈ℝd\(\{\\mathbb\{R\}\}^\{d\}\)^\{\\star\}\\approx\{\\mathbb\{R\}\}^\{d\}and that the target network has bias zero, Hence,

∑a∈I⁡\(i\)f⁡\(𝐯a,𝐯a\)=qi2,∑a,b∈I⁡\(i\),a≠bf⁡\(𝐯a,𝐯b\)=qi​\(qi−1\)2​π,∑a∈I⁡\(i\)​i,b∈I⁡\(j\)f⁡\(𝐯a,𝐯b\)=qi​qj2​π\\sum\_\{a\\in I\(i\)\}f\(\{\{\\mathbf\{v\}\}\}^\{a\},\{\{\\mathbf\{v\}\}\}^\{a\}\)=\\frac\{q\_\{i\}\}\{2\},\\;\\sum\_\{a,b\\in I\(i\),a\\neq b\}f\(\{\{\\mathbf\{v\}\}\}^\{a\},\{\{\\mathbf\{v\}\}\}^\{b\}\)=\\frac\{q\_\{i\}\(q\_\{i\}\-1\)\}\{2\\pi\},\\;\\sum\_\{a\\in I\(i\)i,b\\in I\(j\)\}f\(\{\{\\mathbf\{v\}\}\}^\{a\},\{\{\\mathbf\{v\}\}\}^\{b\}\)=\\frac\{q\_\{i\}q\_\{j\}\}\{2\\pi\}It is not difficult to compute these sums if the target is the identity with non\-zero bias𝜸¯\\boldsymbol\{\\overline\{\\gamma\}\}fixed byG𝐪ΔG\_\{\{\\mathbf\{q\}\}\}^\{\\Delta\}\(use Example[6\.2](https://arxiv.org/html/2609.30661#S6.Thmlemma2)\)\.✠\\maltese

###### Proposition 12\.18\(Loss in terms of rowtype\)\.

\(Notation and assumptions as above\.\)

ℒ⁡\(𝐖\)\\displaystyle\\mathcal\{L\}\(\{\{\\mathbf\{W\}\}\}\)=\\displaystyle=∑i∈𝐟qi​\[f⁡\(𝐰i,𝐰i\)\+f⁡\(𝐯i,𝐯i\)2−f⁡\(𝐰i,𝐯i\)\]\\displaystyle\\sum\_\{i\\in\{\\mathbf\{f\}\}\}q\_\{i\}\\big\[\\frac\{f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{i\}\)\+f\(\{\{\\mathbf\{v\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{i\}\)\}\{2\}\-f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{i\}\)\\big\]\+∑i∈𝐫𝐬qi\(qi−1\)\[f⁡\(𝐰i,𝐰~i\)\+f⁡\(𝐯i,𝐯~i\)2−f\(𝐰i,𝐯~i\)\]\\displaystyle\\hskip\-36\.135pt\+\\sum\_\{i\\in\{\{\\mathbf\{r\_\{s\}\}\}\}\}q\_\{i\}\(q\_\{i\}\-1\)\\big\[\\frac\{f\(\{\{\\mathbf\{w\}\}\}^\{i\},\\widetilde\{\{\{\\mathbf\{w\}\}\}\}^\{i\}\)\+f\(\{\{\\mathbf\{v\}\}\}^\{i\},\\widetilde\{\{\{\\mathbf\{v\}\}\}\}^\{i\}\)\}\{2\}\-f\(\{\{\\mathbf\{w\}\}\}^\{i\},\\widetilde\{\{\{\\mathbf\{v\}\}\}\}^\{i\}\)\\big\]\+∑i,j∈𝐟,i<jqiqj\[\(f\(𝐰i,𝐰j\)\+f\(𝐯i,𝐯j\)\)−\(f\(𝐰i,𝐯j\)\+f\(𝐰j,𝐯i\)\)\]\\displaystyle\\hskip\-36\.135pt\+\\sum\_\{i,j\\in\{\\mathbf\{f\}\},i<j\}q\_\{i\}q\_\{j\}\\big\[\\big\(f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{j\}\)\+f\(\{\{\\mathbf\{v\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{j\}\)\\big\)\-\\big\(f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{j\}\)\+f\(\{\{\\mathbf\{w\}\}\}^\{j\},\{\{\\mathbf\{v\}\}\}^\{i\}\)\\big\)\\big\]\+∑i∈𝐫𝐦\[f⁡\(𝐰i,𝐰i\)2\+∑j∈𝐟qj\(f\(𝐰i\.𝐰j\)−f\(𝐰i,𝐯j\)\)\+∑j∈𝐫𝐦,j<if\(𝐰i,𝐰j\)\]\\displaystyle\\hskip\-36\.135pt\+\\sum\_\{i\\in\{\\mathbf\{r\_\{m\}\}\}\}\\left\[\\frac\{f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{i\}\)\}\{2\}\+\\sum\_\{j\\in\{\\mathbf\{f\}\}\}q\_\{j\}\\big\(f\(\{\{\\mathbf\{w\}\}\}^\{i\}\.\{\{\\mathbf\{w\}\}\}^\{j\}\)\-f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{j\}\)\\big\)\+\\sum\_\{j\\in\{\\mathbf\{r\_\{m\}\}\},j<i\}f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{j\}\)\\right\]

###### Proof\.

The first, second and third lines of the expression forℒ⁡\(𝐖\)\\mathcal\{L\}\(\{\{\\mathbf\{W\}\}\}\)follow respectively from the \(a\), \(b\) and \(c\) lines above; the final line accounts for the contribution from the extra neurons parametrized by𝐫𝐦\{\\mathbf\{r\_\{m\}\}\}and depends on the \(a\) and \(b\) lines\. ∎

### 12\.10\.Derivatives ofℒ\\mathcal\{L\}with respect to𝜷¯\{\\boldsymbol\{\\overline\{\\beta\}\}\}, indexed by rowtype

We continue with the assumptions of the previous subsection\. In particular,\(𝐖,𝜷¯\)∈M​\(k,d\+1\)G𝐪Δ\(\{\{\\mathbf\{W\}\}\},\{\\boldsymbol\{\\overline\{\\beta\}\}\}\)\\in M\(k,d\+1\)^\{G\_\{\{\\mathbf\{q\}\}\}^\{\\Delta\}\}and\{𝐯j\}j∈𝐝\\\{\{\{\\mathbf\{v\}\}\}^\{j\}\\\}\_\{j\\in\{\\mathbf\{d\}\}\}is an orthonormal basis of\(ℝd\)⋆\(\{\\mathbb\{R\}\}^\{d\}\)^\{\\star\}\. The formula for∂ℒ∂β¯i\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{i\}\}is given in Proposition[11\.1](https://arxiv.org/html/2609.30661#S11.Thmlemma1)\. However, the indexing is by row number rather than row type and there are no symmetry assumptions\. The next proposition gives the formula for derivatives ofℒ\\mathcal\{L\}with respect to𝜷¯\{\\boldsymbol\{\\overline\{\\beta\}\}\}in terms of rowtype\.

###### Proposition 12\.19\.

\(Notation and assumptions as above\.\) Fori∈𝐑i\\in\{\\mathbf\{R\}\}

qi−1​∂ℒ∂β¯i=‖𝐰i‖22​\[β¯i​erfc​\(β¯i2\)−2π​e−β¯i22\]\+‖𝐰i‖22​π​\(qi−1\)​β¯i​\(π​erfc​\(β¯i2\)−Θ~i\)−‖𝐰i‖2​π​γ¯i​\(qi​π2​\[erfc​\(β¯i2\)\+erfc​\(γ¯i2\)\]−\(λ~i​i\+\(qi−1\)​θ~i\)\),\(i∈𝐟\)−‖𝐰i‖22​2​π​\(qi−1\)​\[e−β¯i22​erfc​\(−βi​\(−cot⁡\(Θi\)\+csc⁡\(Θi\)\)2\)​\(1\+cos⁡\(Θi\)\)\]\+‖𝐰i‖2​2​π​\(qi−1\)​\[e−γ¯i22​erfc​\(zi2\)\+cos⁡\(θi\)​e−β¯i22​erfc​\(z^i2\)\]\+‖𝐰i‖2​2​π​\[e−γ¯i22​erfc​\(zi​i2\)\+cos⁡\(λi​i\)​e−β¯i22​erfc​\(z^i​i2\)\],\(i∈𝐟\)\+‖𝐰i‖2​π∑j∈𝐑,j≠iqj∥𝐰j∥β¯j\(π2\[erfc\(β¯i2\)\+erfc\(β¯j2\)\]−Λ~i​j\)−‖𝐰i‖2​2​π∑j∈𝐑,j≠iqj∥𝐰j∥\[e−β¯j22erfc\(Zi​j2\)\+cos\(Λi​j\)e−β¯i22erfc\(Z^i​j2\)\]−‖𝐰i‖2​π∑j∈𝐟,j≠iγ¯jqj\(π2\[erfc\(β¯i2\)\+erfc\(γ¯j2\)\]−λ~i​j\)\+‖𝐰i‖2​2​π∑j∈𝐟,j≠iqj\[e−γ¯j22erfc\(zi​j2\)\+cos\(λi​j\)e−β¯i22erfc\(z^i​j2\)\]\\begin\{aligned\} q\_\{i\}^\{\-1\}\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{i\}\}&=\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|^\{2\}\}\{2\}\\left\[\\overline\{\\beta\}\_\{i\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\\sqrt\{\\frac\{2\}\{\\pi\}\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\\right\]\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|^\{2\}\}\{2\\pi\}\(q\_\{i\}\-1\)\\overline\{\\beta\}\_\{i\}\\left\(\\pi\\,\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\{\\widetilde\{\\Theta\}\}\_\{i\}\\right\)\\\\ &\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\{2\\pi\}\\overline\{\\gamma\}\_\{i\}\\left\(q\_\{i\}\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\(\{\\widetilde\{\\lambda\}\}\_\{ii\}\+\(q\_\{i\}\-1\)\{\\widetilde\{\\theta\}\}\_\{i\}\)\\right\),\\;\(i\\in\{\\mathbf\{f\}\}\)\\\\ &\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|^\{2\}\}\{2\\sqrt\{2\\pi\}\}\(q\_\{i\}\-1\)\\left\[e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\-\\beta\_\{i\}\(\-\\cot\(\\Theta\_\{i\}\)\+\\csc\(\\Theta\_\{i\}\)\)\}\{\\sqrt\{2\}\}\}\\right\)\}\(1\+\\cos\(\\Theta\_\{i\}\)\)\\right\]\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\{2\\sqrt\{2\\pi\}\}\(q\_\{i\}\-1\)\\left\[e^\{\-\\frac\{\\overline\{\\gamma\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{z\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\cos\(\\theta\_\{i\}\)e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\{\\widehat\{z\}\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\left\[e^\{\-\\frac\{\\overline\{\\gamma\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{z\_\{ii\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\cos\(\\lambda\_\{ii\}\)e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\{\\widehat\{z\}\}\_\{ii\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\],\\;\(i\\in\{\\mathbf\{f\}\}\)\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\{2\\pi\}\\sum\_\{j\\in\{\\mathbf\{R\}\},j\\neq i\}q\_\{j\}\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\\overline\{\\beta\}\_\{j\}\\left\(\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\{\\widetilde\{\\Lambda\}\}\_\{ij\}\\right\)\\\\ &\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\sum\_\{j\\in\{\\mathbf\{R\}\},j\\neq i\}q\_\{j\}\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\\left\[e^\{\-\\frac\{\\overline\{\\beta\}\_\{j\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{Z\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\cos\(\\Lambda\_\{ij\}\)e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\{\\widehat\{Z\}\}\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ &\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\{2\\pi\}\\sum\_\{j\\in\{\\mathbf\{f\}\},j\\neq i\}\\overline\{\\gamma\}\_\{j\}q\_\{j\}\\left\(\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\gamma\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\{\\widetilde\{\\lambda\}\}\_\{ij\}\\right\)\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\sum\_\{j\\in\{\\mathbf\{f\}\},j\\neq i\}q\_\{j\}\\left\[e^\{\-\\frac\{\\overline\{\\gamma\}\_\{j\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{z\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\cos\(\\lambda\_\{ij\}\)e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\{\\widehat\{z\}\}\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ \\end\{aligned\}

###### Proof\.

The proof follows directly from Proposition[11\.1](https://arxiv.org/html/2609.30661#S11.Thmlemma1)\. Note that lines 3 & 6, ending in\(i∈𝐟\)\(i\\in\{\\mathbf\{f\}\}\), are defined to be zero ifi∉𝐟i\\notin\{\\mathbf\{f\}\}\. The lines 2,4 & 5 are zero ifi∉𝐫𝐬i\\notin\{\{\\mathbf\{r\_\{s\}\}\}\}\. ∎

###### Remark 12\.20\.

The termsZi​j,Z^i​j,zi​j,z^i​jZ\_\{ij\},\{\\widehat\{Z\}\}\_\{ij\},z\_\{ij\},\{\\widehat\{z\}\}\_\{ij\}are defined in Section[10\.1](https://arxiv.org/html/2609.30661#S10.SS1), andzi,z^iz\_\{i\},\{\\widehat\{z\}\}\_\{i\}at the beginning of Section[12\.7](https://arxiv.org/html/2609.30661#S12.SS7)\.✠\\maltese

###### Examples 12\.21\.

\(1\) If𝜸¯=𝟎\\boldsymbol\{\\overline\{\\gamma\}\}=\{\\mathbf\{0\}\}—the only case we investigate in this work—then

qi−1​∂ℒ∂β¯i=‖𝐰i‖22​\[β¯i​erfc​\(β¯i2\)−2π​e−β¯i22\]\+‖𝐰i‖22​π​\(qi−1\)​β¯i​\(π​erfc​\(β¯i2\)−Θ~i\)−‖𝐰i‖22​2​π​\(qi−1\)​\[e−β¯i22​erfc​\(−βi​\(−cot⁡\(Θi\)\+csc⁡\(Θi\)\)2\)​\(1\+cos⁡\(Θi\)\)\]\+‖𝐰i‖2​2​π​\(qi−1\)​\[erfc​\(zi2\)\+cos⁡\(θi\)​e−β¯i22​erfc​\(z^i2\)\]\+‖𝐰i‖2​2​π​\[e−γ¯i22​erfc​\(zi​i2\)\+cos⁡\(λi​i\)​e−β¯i22​erfc​\(z^i​i2\)\],\(i∈𝐟\)\+‖𝐰i‖2​π∑j∈𝐑,j≠iqj∥𝐰j∥β¯j\(π2\[erfc\(β¯i2\)\+erfc\(β¯j2\)\]−Λ~i​j\)−‖𝐰i‖2​2​π∑j∈𝐑,j≠iqj∥𝐰j∥\[e−β¯j22erfc\(Zi​j2\)\+cos\(Λi​j\)e−β¯i22erfc\(Z^i​j2\)\]\+‖𝐰i‖2​2​π∑j∈𝐟,j≠iqj\[erfc\(β¯i​csc⁡\(λi​j\)2\)\+cos\(λi​j\)e−β¯i22erfc\(−β¯i​cot⁡\(λi​j\)2\)\]\\begin\{aligned\} q\_\{i\}^\{\-1\}\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{i\}\}&=\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|^\{2\}\}\{2\}\\left\[\\overline\{\\beta\}\_\{i\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\\sqrt\{\\frac\{2\}\{\\pi\}\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\\right\]\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|^\{2\}\}\{2\\pi\}\(q\_\{i\}\-1\)\\overline\{\\beta\}\_\{i\}\\left\(\\pi\\,\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\{\\widetilde\{\\Theta\}\}\_\{i\}\\right\)\\\\ &\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|^\{2\}\}\{2\\sqrt\{2\\pi\}\}\(q\_\{i\}\-1\)\\left\[e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\-\\beta\_\{i\}\(\-\\cot\(\\Theta\_\{i\}\)\+\\csc\(\\Theta\_\{i\}\)\)\}\{\\sqrt\{2\}\}\}\\right\)\}\(1\+\\cos\(\\Theta\_\{i\}\)\)\\right\]\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\{2\\sqrt\{2\\pi\}\}\(q\_\{i\}\-1\)\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{z\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\cos\(\\theta\_\{i\}\)e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\{\\widehat\{z\}\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\left\[e^\{\-\\frac\{\\overline\{\\gamma\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{z\_\{ii\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\cos\(\\lambda\_\{ii\}\)e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\{\\widehat\{z\}\}\_\{ii\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\],\\;\(i\\in\{\\mathbf\{f\}\}\)\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\{2\\pi\}\\sum\_\{j\\in\{\\mathbf\{R\}\},j\\neq i\}q\_\{j\}\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\\overline\{\\beta\}\_\{j\}\\left\(\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{j\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\{\\widetilde\{\\Lambda\}\}\_\{ij\}\\right\)\\\\ &\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\sum\_\{j\\in\{\\mathbf\{R\}\},j\\neq i\}q\_\{j\}\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\\left\[e^\{\-\\frac\{\\overline\{\\beta\}\_\{j\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{Z\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\cos\(\\Lambda\_\{ij\}\)e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\{\\widehat\{Z\}\}\_\{ij\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\sum\_\{j\\in\{\\mathbf\{f\}\},j\\neq i\}q\_\{j\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{i\}\\csc\(\\lambda\_\{ij\}\)\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\cos\(\\lambda\_\{ij\}\)e^\{\-\\frac\{\\overline\{\\beta\}\_\{i\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\-\\overline\{\\beta\}\_\{i\}\\cot\(\\lambda\_\{ij\}\)\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\end\{aligned\}\(2\) Suppose𝜸¯=𝟎\\boldsymbol\{\\overline\{\\gamma\}\}=\{\\mathbf\{0\}\},k=dk=dandG𝐪Δ=Δ⁡\(Sq0×Sq1\)G\_\{\{\\mathbf\{q\}\}\}^\{\\Delta\}=\\Delta\(S\_\{q\_\{0\}\}\\times S\_\{q\_\{1\}\}\)\. whereq0≥q1≥1q\_\{0\}\\geq q\_\{1\}\\geq 1andd≥3d\\geq 3\. There are two rowtypes\. Sinceq0≥2q\_\{0\}\\geq 2,

q0−1​∂ℒ∂β¯0=‖𝐰0‖22​\[β¯0​erfc​\(β¯02\)−2π​e−β¯022\]\+‖𝐰0‖22​π​\(q0−1\)​β¯0​\(π​erfc​\(β¯02\)−Θ~0\)−‖𝐰0‖22​2​π​\(q0−1\)​\[e−β¯022​erfc​\(−β0​\(−cot⁡\(Θ0\)\+csc⁡\(Θ0\)\)2\)​\(1\+cos⁡\(Θ0\)\)\]\+‖𝐰0‖2​2​π​\(q0−1\)​\[erfc​\(z02\)\+cos⁡\(θ0\)​e−β¯022​erfc​\(z^02\)\]\+‖𝐰i‖2​2​π​\[erfc​\(z002\)\+cos⁡\(λ00\)​e−β¯022​erfc​\(z^002\)\]\+‖𝐰0‖2​π​q1​‖𝐰1‖​β¯1​\(π2​\[erfc​\(β¯02\)\+erfc​\(β¯12\)\]−Λ~01\)−‖𝐰0‖2​2​π​q1​‖𝐰1‖​\[e−β¯122​erfc​\(Z012\)\+cos⁡\(Λ01\)​e−β¯022​erfc​\(Z^012\)\]\+‖𝐰0‖2​2​π​q1​\[erfc​\(β¯0​csc⁡\(λ01\)2\)\+cos⁡\(λ01\)​e−β¯022​erfc​\(−β¯0​cot⁡\(λ01\)2\)\]\\begin\{aligned\} q\_\{0\}^\{\-1\}\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{0\}\}&=\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{0\}\\\|^\{2\}\}\{2\}\\left\[\\overline\{\\beta\}\_\{0\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{0\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\\sqrt\{\\frac\{2\}\{\\pi\}\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{0\}^\{2\}\}\{2\}\}\\right\]\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{0\}\\\|^\{2\}\}\{2\\pi\}\(q\_\{0\}\-1\)\\overline\{\\beta\}\_\{0\}\\left\(\\pi\\,\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{0\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\{\\widetilde\{\\Theta\}\}\_\{0\}\\right\)\\\\ &\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{0\}\\\|^\{2\}\}\{2\\sqrt\{2\\pi\}\}\(q\_\{0\}\-1\)\\left\[e^\{\-\\frac\{\\overline\{\\beta\}\_\{0\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\-\\beta\_\{0\}\(\-\\cot\(\\Theta\_\{0\}\)\+\\csc\(\\Theta\_\{0\}\)\)\}\{\\sqrt\{2\}\}\}\\right\)\}\(1\+\\cos\(\\Theta\_\{0\}\)\)\\right\]\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{0\}\\\|\}\{2\\sqrt\{2\\pi\}\}\(q\_\{0\}\-1\)\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{z\_\{0\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\cos\(\\theta\_\{0\}\)e^\{\-\\frac\{\\overline\{\\beta\}\_\{0\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\{\\widehat\{z\}\}\_\{0\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{z\_\{00\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\cos\(\\lambda\_\{00\}\)e^\{\-\\frac\{\\overline\{\\beta\}\_\{0\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\{\\widehat\{z\}\}\_\{00\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{0\}\\\|\}\{2\\pi\}q\_\{1\}\\\|\{\{\\mathbf\{w\}\}\}^\{1\}\\\|\\overline\{\\beta\}\_\{1\}\\left\(\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{0\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{1\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\{\\widetilde\{\\Lambda\}\}\_\{01\}\\right\)\\\\ &\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{0\}\\\|\}\{2\\sqrt\{2\\pi\}\}q\_\{1\}\\\|\{\{\\mathbf\{w\}\}\}^\{1\}\\\|\\left\[e^\{\-\\frac\{\\overline\{\\beta\}\_\{1\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{Z\_\{01\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\cos\(\\Lambda\_\{01\}\)e^\{\-\\frac\{\\overline\{\\beta\}\_\{0\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\{\\widehat\{Z\}\}\_\{01\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{0\}\\\|\}\{2\\sqrt\{2\\pi\}\}q\_\{1\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{0\}\\csc\(\\lambda\_\{01\}\)\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\cos\(\\lambda\_\{01\}\)e^\{\-\\frac\{\\overline\{\\beta\}\_\{0\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\-\\overline\{\\beta\}\_\{0\}\\cot\(\\lambda\_\{01\}\)\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\end\{aligned\}Ifq1=1q\_\{1\}=1,

∂ℒ∂β¯1=‖𝐰1‖22​\[β¯1​erfc​\(β¯12\)−2π​e−β¯122\]\+‖𝐰1‖2​π​q0​‖𝐰0‖​β¯0​\(π2​\[erfc​\(β¯02\)\+erfc​\(β¯12\)\]−Λ~10\)\+‖𝐰1‖2​2​π​\[erfc​\(z112\)\+cos⁡\(λ11\)​e−β¯122​erfc​\(z^112\)\]−‖𝐰1‖2​2​π​q0​‖𝐰0‖​\[e−β¯022​erfc​\(Z102\)\+cos⁡\(Λ10\)​e−β¯122​erfc​\(Z^102\)\]\+‖𝐰1‖2​2​π​q0​\[erfc​\(β¯1​csc⁡\(λ10\)2\)\+cos⁡\(λ10\)​e−β¯122​erfc​\(−β¯1​cot⁡\(λ10\)2\)\]\\begin\{aligned\} \\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{1\}\}&=\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{1\}\\\|^\{2\}\}\{2\}\\left\[\\overline\{\\beta\}\_\{1\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{1\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\\sqrt\{\\frac\{2\}\{\\pi\}\}e^\{\-\\frac\{\\overline\{\\beta\}\_\{1\}^\{2\}\}\{2\}\}\\right\]\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{1\}\\\|\}\{2\\pi\}q\_\{0\}\\\|\{\{\\mathbf\{w\}\}\}^\{0\}\\\|\\overline\{\\beta\}\_\{0\}\\left\(\\frac\{\\pi\}\{2\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{0\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{1\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\{\\widetilde\{\\Lambda\}\}\_\{10\}\\right\)\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{1\}\\\|\}\{2\\sqrt\{2\\pi\}\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{z\_\{11\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\cos\(\\lambda\_\{11\}\)e^\{\-\\frac\{\\overline\{\\beta\}\_\{1\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\{\\widehat\{z\}\}\_\{11\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ &\-\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{1\}\\\|\}\{2\\sqrt\{2\\pi\}\}q\_\{0\}\\\|\{\{\\mathbf\{w\}\}\}^\{0\}\\\|\\left\[e^\{\-\\frac\{\\overline\{\\beta\}\_\{0\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{Z\_\{10\}\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\cos\(\\Lambda\_\{10\}\)e^\{\-\\frac\{\\overline\{\\beta\}\_\{1\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\{\\widehat\{Z\}\}\_\{10\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ &\+\\frac\{\\\|\{\{\\mathbf\{w\}\}\}^\{1\}\\\|\}\{2\\sqrt\{2\\pi\}\}q\_\{0\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\_\{1\}\\csc\(\\lambda\_\{10\}\)\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\cos\(\\lambda\_\{10\}\)e^\{\-\\frac\{\\overline\{\\beta\}\_\{1\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\-\\overline\{\\beta\}\_\{1\}\\cot\(\\lambda\_\{10\}\)\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\end\{aligned\}Ifq1\>1q\_\{1\}\>1, the formula can be deduced from the result forq0q\_\{0\}by appropriate permutation of the indices\.

More details for the simplest caseG𝐪Δ=Δ​SdG\_\{\{\\mathbf\{q\}\}\}^\{\\Delta\}=\\Delta S\_\{d\}appear at the beginning of Section[15\.1](https://arxiv.org/html/2609.30661#S15.SS1)\.※

## 13\.Review of results for symmetric targetIdI\_\{d\}

We review parts of the theory for bias free networks when the target isIdI\_\{d\}andk≥dk\\geq d\. The emphasis is on the critical point equations and examples of infinite families of critical points parametrized bydd\. Details and proofs can be found in\[[4](https://arxiv.org/html/2609.30661#bib.bib4),[2](https://arxiv.org/html/2609.30661#bib.bib2),[3](https://arxiv.org/html/2609.30661#bib.bib3),[5](https://arxiv.org/html/2609.30661#bib.bib5)\]\. For each example, the corresponding critical point equations are given for the biased network under the assumption that the target has zero bias\.

### 13\.1\.Fixed point spaces and fractional power series

Supposek0−d0=m≥0k\_\{0\}\-d\_\{0\}=m\\geq 0and𝐖∈M⁡\(k0,d0\)\{\{\\mathbf\{W\}\}\}\\in M\(k\_\{0\},d\_\{0\}\)is a critical point ofℒ\\mathcal\{L\}of isotropyG𝐪ΔG^\{\\Delta\}\_\{\{\\mathbf\{q\}\}\}, whereq0\>q1≥…≥qf−1≥1q\_\{0\}\>q\_\{1\}\\geq\\ldots\\geq q\_\{f\-1\}\\geq 1\. Takingq1,…,qf−1q\_\{1\},\\ldots,q\_\{f\-1\}as constant andq0=d−∑i\>0qiq\_\{0\}=d\-\\sum\_\{i\>0\}q\_\{i\}\(variable\), it follows from section[12\.3](https://arxiv.org/html/2609.30661#S12.SS3)that ford≥d0d\\geq d\_\{0\}there is a natural linear isomorphism

M​\(k,d\)G𝐪Δ≈\(∏i∈𝐑ℝf\)×ℝrsM\(k,d\)^\{G\_\{\\mathbf\{q\}\}^\{\\Delta\}\}\\approx\\left\(\\prod\_\{i\\in\{\\mathbf\{R\}\}\}\{\\mathbb\{R\}\}^\{f\}\\right\)\\times\{\\mathbb\{R\}\}^\{r\_\{s\}\}which we use to coordinatizeM​\(k,d\)G𝐪ΔM\(k,d\)^\{G\_\{\\mathbf\{q\}\}^\{\\Delta\}\}\. Herersr\_\{s\}counts the number ofqi\>1q\_\{i\}\>1andR=f\+mR=f\+m\. Ifm\>0m\>0, It is convenient to defineqj=1q\_\{j\}=1forj∈\[f,m\+f−1\]j\\in\[f,m\+f\-1\]\. Denoting coordinates on\(∏i∈𝐑ℝf\)×ℝrs\\left\(\\prod\_\{i\\in\{\\mathbf\{R\}\}\}\{\\mathbb\{R\}\}^\{f\}\\right\)\\times\{\\mathbb\{R\}\}^\{r\_\{s\}\}by\(ξji\)i∈𝐑,j∈𝐟,\(ξ⋆ii\)i∈𝐫𝐬\(\\xi\_\{j\}^\{i\}\)\_\{i\\in\{\\mathbf\{R\}\},j\\in\{\\mathbf\{f\}\}\},\(\\xi\_\{\\star i\}^\{i\}\)\_\{i\\in\{\{\\mathbf\{r\_\{s\}\}\}\}\}, the critical point equations restricted to the flow invariant subspaceM​\(k,d\)G𝐪ΔM\(k,d\)^\{G\_\{\\mathbf\{q\}\}^\{\\Delta\}\}may be written in𝝃,𝝃⋆\{\\boldsymbol\{\\xi\}\},\{\\boldsymbol\{\\xi\}\}\_\{\\star\}coordinates as

\(13\.52\)12​π​\[Γi​ξji−Aji\]\+12​Ωj\\displaystyle\\frac\{1\}\{2\\pi\}\\left\[\\Gamma\_\{i\}\\xi\_\{j\}^\{i\}\-A\_\{j\}^\{i\}\\right\]\+\\frac\{1\}\{2\}\\Omega\_\{j\}=\\displaystyle=0,j∈𝐟\\displaystyle 0,\\;j\\in\{\\mathbf\{f\}\}\(13\.53\)12​π​\[Γi​ξ⋆ii−A⋆ii\]\+12​Ωi\\displaystyle\\frac\{1\}\{2\\pi\}\\left\[\\Gamma\_\{i\}\\xi\_\{\\star i\}^\{i\}\-A\_\{\\star i\}^\{i\}\\right\]\+\\frac\{1\}\{2\}\\Omega\_\{i\}=\\displaystyle=0,if​qi\>1,\\displaystyle 0,\\;\\text\{if \}q\_\{i\}\>1,where

Aji\\displaystyle A^\{i\}\_\{j\}=\\displaystyle=∑ℓ≠jqℓ​Λi​ℓ​ξjℓ\+\(qi−1\)​Θi​ξji\+Λi​j​\(\(qj−1\)​ξ⋆jj\+ξjj\)−λi​j,i≠j\\displaystyle\\sum\_\{\\ell\\neq j\}q\_\{\\ell\}\\Lambda\_\{i\\ell\}\\xi^\{\\ell\}\_\{j\}\+\(q\_\{i\}\-1\)\\Theta\_\{i\}\\xi^\{i\}\_\{j\}\+\\Lambda\_\{ij\}\(\(q\_\{j\}\-1\)\\xi\_\{\\star j\}^\{j\}\+\\xi\_\{j\}^\{j\}\)\-\\lambda\_\{ij\},\\;i\\neq jAii\\displaystyle A^\{i\}\_\{i\}=\\displaystyle=\(qi−1\)​Θi​ξ⋆ii\+∑jqj​Λi​j​ξij−λi​i,i∈𝐟\\displaystyle\(q\_\{i\}\-1\)\\Theta\_\{i\}\\xi^\{i\}\_\{\\star i\}\+\\sum\_\{j\}q\_\{j\}\\Lambda\_\{ij\}\\xi^\{j\}\_\{i\}\-\\lambda\_\{ii\},\\;i\\in\{\\mathbf\{f\}\}A⋆ii\\displaystyle A^\{i\}\_\{\\star i\}=\\displaystyle=Θi​ξii\+\(qi−2\)​Θi​ξ⋆ii\+∑jqj​Λi​j​ξij−θi,qi≥2\\displaystyle\\Theta\_\{i\}\\xi^\{i\}\_\{i\}\+\(q\_\{i\}\-2\)\\Theta\_\{i\}\\xi^\{i\}\_\{\\star i\}\+\\sum\_\{j\}q\_\{j\}\\Lambda\_\{ij\}\\xi^\{j\}\_\{i\}\-\\theta\_\{i\},\\;\\;q\_\{i\}\\geq 2
###### Remarks 13\.1\.

\(1\) The termsΩj\\Omega\_\{j\},j∈𝐟j\\in\{\\mathbf\{f\}\}, are defined in section[10\.2](https://arxiv.org/html/2609.30661#S10.SS2)\. \(2\) Every solution of \([13\.52](https://arxiv.org/html/2609.30661#S13.E52),[13\.53](https://arxiv.org/html/2609.30661#S13.E53)\) determines a critical point ofℒ\\mathcal\{L\}lying inM​\(k,d\)G𝐪ΔM\(k,d\)^\{G\_\{\\mathbf\{q\}\}^\{\\Delta\}\}\. The equations \([13\.52](https://arxiv.org/html/2609.30661#S13.E52),[13\.53](https://arxiv.org/html/2609.30661#S13.E53)\) on\(∏i∈𝐑ℝf\)×ℝrs\\left\(\\prod\_\{i\\in\{\\mathbf\{R\}\}\}\{\\mathbb\{R\}\}^\{f\}\\right\)\\times\{\\mathbb\{R\}\}^\{r\_\{s\}\}are not the gradient ofℒ\|M​\(k,d\)G𝐪Δ\\mathcal\{L\}\|M\(k,d\)^\{G\_\{\\mathbf\{q\}\}^\{\\Delta\}\}with respect to the standard Euclidean inner product on\(∏i∈𝐑ℝf\)×ℝrs\\left\(\\prod\_\{i\\in\{\\mathbf\{R\}\}\}\{\\mathbb\{R\}\}^\{f\}\\right\)\\times\{\\mathbb\{R\}\}^\{r\_\{s\}\}\. In order for that the map fromM​\(k,d\)G𝐪ΔM\(k,d\)^\{G\_\{\\mathbf\{q\}\}^\{\\Delta\}\}to\(∏i∈𝐑ℝf\)×ℝrs\\left\(\\prod\_\{i\\in\{\\mathbf\{R\}\}\}\{\\mathbb\{R\}\}^\{f\}\\right\)\\times\{\\mathbb\{R\}\}^\{r\_\{s\}\}needs to be an isometry with respect to the Euclidean \(Frobenius\) inner product onM​\(k,d\)G𝐪ΔM\(k,d\)^\{G\_\{\\mathbf\{q\}\}^\{\\Delta\}\}\. Relative to the natural block decomposition of matrices inℒ\|M​\(k,d\)G𝐪Δ\\mathcal\{L\}\|M\(k,d\)^\{G\_\{\\mathbf\{q\}\}^\{\\Delta\}\}, non diagonal blocksx​Iqi,qjxI\_\{q\_\{i\},q\_\{j\}\}will map tox​qi​qjx\\sqrt\{q\_\{i\}q\_\{j\}\}and aqi×qiq\_\{i\}\\times q\_\{i\}\-diagonal block with diagonal entryxxand off diagonal entryyywill map to\(qi​x,qi​\(qi−1\)​y\)\(\\sqrt\{q\_\{i\}\}x,\\sqrt\{q\_\{i\}\(q\_\{i\}\-1\)\}y\)\(omitted coordinates are zero\)\. However, there is no advantage in doing this since the linearization of the vector field is similar by a*diagonal*matrix to the Hessian of the loss\. In particular, Sylvester’s criterion applies to the linearization of the vector field\. Of course, this remark only applies to the set of eigenvalues associated to trivial representation ofG𝐪ΔG\_\{\\mathbf\{q\}\}^\{\\Delta\}on the fixed point space\. For the remaining eigenvalues methods based on representation theory may be used\[[3](https://arxiv.org/html/2609.30661#bib.bib3)\]\.✠\\maltese

#### Discussion

Since the fixed point space is independent ofd≥d0d\\geq d\_\{0\}, we may regardd≥d0d\\geq d\_\{0\}as a*real*parameter in \([13\.52](https://arxiv.org/html/2609.30661#S13.E52),[13\.53](https://arxiv.org/html/2609.30661#S13.E53)\) and consider the solution𝝃,𝝃⋆\{\\boldsymbol\{\\xi\}\},\{\\boldsymbol\{\\xi\}\}\_\{\\star\}of the parametrized family as a function ofdd\. The critical point equations are analytic on an open dense semi\-analytic subset ofM​\(k,d\)G𝐪ΔM\(k,d\)^\{G\_\{\\mathbf\{q\}\}^\{\\Delta\}\}\[[4](https://arxiv.org/html/2609.30661#bib.bib4)\]\. This suggests path following𝝃=𝝃⁡\(d\),𝝃⋆=𝝃⋆​\(d\)\{\\boldsymbol\{\\xi\}\}=\{\\boldsymbol\{\\xi\}\}\(d\),\{\\boldsymbol\{\\xi\}\}\_\{\\star\}=\{\\boldsymbol\{\\xi\}\}\_\{\\star\}\(d\)asddis increased\. This will work by the analytic version of the implicit function theorem provided that the Jacobian determinant along the path \(in the fixed point space\) is non\-singular\. Practically speaking, Newton’s method is used to find a discrete series of solutions asddis incremented\. Note that the trivial solution𝐕\{\{\\mathbf\{V\}\}\}does not lie in the domain of analyticity of the critical point\. Ifk=dk=d, it may be shown that the loss isC3C^\{3\}at𝐖=𝐕\{\{\\mathbf\{W\}\}\}=\{\{\\mathbf\{V\}\}\}\.

Using high precision computation, it is straightforward to use path following methods to obtain critical points for large values ofdd—for exampled=101000d=10^\{1000\}\. The numerical results indicate that in many cases there are fractional power series solutions \(FPS\) ind−pqd^\{\-\\frac\{p\}\{q\}\}for these families of critical points\. Ifk=dk=d, it is often relatively straightforward to prove the existence of FPS solutions for families of critical points\(𝝃,𝝃⋆\)d≥d1\(\{\\boldsymbol\{\\xi\}\},\{\\boldsymbol\{\\xi\}\}\_\{\\star\}\)\_\{d\\geq d\_\{1\}\}, whered1≥d0d\_\{1\}\\geq d\_\{0\}, as well as obtain low order coefficients in the FPS\. Examples of these computations, which depend on elementary methods using the implicit function theorem, are in\[[4](https://arxiv.org/html/2609.30661#bib.bib4),[5](https://arxiv.org/html/2609.30661#bib.bib5)\]\. Typically the denominatorqqis a power of22andp=1p=1\. The existence of these solutions follows from the general theory of subanalytic nets and the Curve Selection Lemma\. In practice, numerics can be used to give the initial constant terms of the FPS \(often given by±𝐯j\\pm\{\{\\mathbf\{v\}\}\}^\{j\}—the rows of the target\) and the exponentqqand this data suffices to work out low order terms of the FPS and apply the implicit function theorem\. For the examples given below, this process is usually easy ifk=dk=d, when the FPS are in powers ofd−12d^\{\-\\frac\{1\}\{2\}\}, but harder whenm\>0m\>0or there is significant asymmetry in the isotropy \(many terms withqi=1q\_\{i\}=1\) when low order coefficients may need to be computed numerically and the series may be in powers ofd−14d^\{\-\\frac\{1\}\{4\}\}implying slow convergence\.

Although the existence of FPS follows straightforwardly from general theory in bias free networks, this is*not*so for biased networks since the critical point equations are not subanalytic and so the Curve Selection Lemma and generalities on subanalytic stratifications may not apply\. Even in the simplest cases, this is an issue\. For example, suppose the target isT=IdT=I\_\{d\},k=dk=d, and the biasesγ¯i\\overline\{\\gamma\}\_\{i\}in the teacher network are all set toγ¯∈ℝ\\overline\{\\gamma\}\\in\{\\mathbb\{R\}\}\. Let𝒮=\(𝝃⁡\(d\),𝝃⋆​\(d\)\)d≥d0\\mathcal\{S\}=\(\{\\boldsymbol\{\\xi\}\}\(d\),\{\\boldsymbol\{\\xi\}\}\_\{\\star\}\(d\)\)\_\{d\\geq d\_\{0\}\}be a family of of spurious minima of isotropyΔ⁡\(Sd−q1×Sq1\)\\Delta\(S\_\{d\-q\_\{1\}\}\\times S\_\{q\_\{1\}\}\)\. Denote the normalized biases byβ¯i​\(d\)\\overline\{\\beta\}\_\{i\}\(d\),i∈𝟐i\\in\{\\mathbf\{2\}\}\. A necessary condition for there to be an FPS representation of𝒮\\mathcal\{S\}is that fori∈𝟐i\\in\{\\mathbf\{2\}\},limd→∞β¯i​\(d\)=±γ¯\\lim\_\{d\{\\rightarrow\}\\infty\}\\overline\{\\beta\}\_\{i\}\(d\)=\\pm\\overline\{\\gamma\}and that\|β¯i​\(d\)−γ¯\|=O⁡\(d−α\)\|\\overline\{\\beta\}\_\{i\}\(d\)\-\\overline\{\\gamma\}\|=O\(d^\{\-\\alpha\}\), for someα\>0\\alpha\>0\.*This is so, even ifγ¯=0\\overline\{\\gamma\}=0*\. If𝒮=\(𝝃⁡\(d\),𝝃⋆​\(d\)\)d≥d0\\mathcal\{S\}=\(\{\\boldsymbol\{\\xi\}\}\(d\),\{\\boldsymbol\{\\xi\}\}\_\{\\star\}\(d\)\)\_\{d\\geq d\_\{0\}\}is family of critical points for the*unbiased network*, then it will generally*not*be a family of critical points for the biased network even if the biases are all set to zero for the target \(see Section[14](https://arxiv.org/html/2609.30661#S14)\)\.

### 13\.2\.A family of spurious minima with isotropyΔ​Sd\\Delta S\_\{d\},k=dk=d

In the bias free case, the FPS is defined ford≥4d\\geq 4and ford≥8d\\geq 8defines a curve of spurious minima\. The FPS is given by convergent series for the componentsξ00,ξ⋆00\\xi\_\{0\}^\{0\},\\xi\_\{\\star 0\}^\{0\}of the critical point and is a power series ind−12d^\{\-\\frac\{1\}\{2\}\}:

ξ00=−1\+∑n=2∞cn​k−n2,ξ⋆0=∑n=2∞en​k−n2\\xi\_\{0\}^\{0\}=\-1\+\\sum\_\{n=2\}^\{\\infty\}c\_\{n\}k^\{\-\\frac\{n\}\{2\}\},\\quad\\xi\_\{\\star 0\}=\\sum\_\{n=2\}^\{\\infty\}e\_\{n\}k^\{\-\\frac\{n\}\{2\}\}where the initial coefficients are given by

c0=−1e0=0c1=e1=0c2=e2=2c3=e3=0c4=8π−4e4=4π−2\\begin\{matrix\}c\_\{0\}=\-1&e\_\{0\}=0&c\_\{1\}=e\_\{1\}=0&c\_\{2\}=e\_\{2\}=2&c\_\{3\}=e\_\{3\}=0\\\\ &&c\_\{4\}=\\frac\{8\}\{\\pi\}\-4&e\_\{4\}=\\frac\{4\}\{\\pi\}\-2&\\end\{matrix\}The series appears to converge to the critical point for allddfor which type A solutions exist but the existence proof\[[4](https://arxiv.org/html/2609.30661#bib.bib4), §8\.3\], based as it is on the implicit function theorem, only gives convergence for “sufficiently large”dd\. The family was originally designated as being of type A in\[[4](https://arxiv.org/html/2609.30661#bib.bib4)\]\. Subsequently, a new definition of type I was introduced for families with isotropyΔ⁡\(Sd−q1×Sq1\)\\Delta\(S\_\{d\-q\_\{1\}\}\\times S\_\{q\_\{1\}\}\), whereq1≥0q\_\{1\}\\geq 0\. Roughly speaking, a family is said to be of type I if the constant coefficient in the FPS forξ00\\xi^\{0\}\_\{0\}is−1\-1; the family is said to be of type II, if the constant coefficient in the FPS forξ00\\xi^\{0\}\_\{0\}is11and the critical point does not define the global minimum\. A characteristic feature of type I families is that the lossℒ=ℒ⁡\(d\)\\mathcal\{L\}=\\mathcal\{L\}\(d\)does not decay to zero asd→∞d\{\\rightarrow\}\\infty\. For the family with isotropyΔ​Sd\\Delta S\_\{d\}andk=dk=dit is shown in\[[4](https://arxiv.org/html/2609.30661#bib.bib4), §8\.6\]thatℒ⁡\(d\)=12−1π\+O⁡\(d−12\)\\mathcal\{L\}\(d\)=\\frac\{1\}\{2\}\-\\frac\{1\}\{\\pi\}\+O\(d^\{\-\\frac\{1\}\{2\}\}\)\.

### 13\.3\.Fractional power series \(FPS\): types I and II

We give two examples here to illustrate and fill out the details on fractional power series as well as introduce the notion of type I and II critical point families\.

###### Examples 13\.2\.

\(1\) We give the FPS for a family of spurious minima of type II\[[4](https://arxiv.org/html/2609.30661#bib.bib4), Thm\. 8\.1\]with isotropyΔ⁡\(Sd−1×S1\)≈Δ​Sd−1\\Delta\(S\_\{d\-1\}\\times S\_\{1\}\)\\approx\\Delta S\_\{d\-1\}andm=0m=0\. Taking note of Lemma[12\.4](https://arxiv.org/html/2609.30661#S12.Thmlemma4), the dimension of the fixed point space is 5 \(Example[12\.6](https://arxiv.org/html/2609.30661#S12.Thmlemma6), bias free case\)\. There is a family\(𝝃⁡\(d\)\)d≥3\(\{\\boldsymbol\{\\xi\}\}\(d\)\)\_\{d\\geq 3\}of critical points of isotropyΔ​Sd−1\\Delta S\_\{d\-1\}for which there is an FPS in1/d1/d\(power series ind−12d^\{\-\\frac\{1\}\{2\}\}\) for each component of the critical point𝝃⁡\(d\)\{\\boldsymbol\{\\xi\}\}\(d\):

ξ00\\displaystyle\\xi\_\{0\}^\{0\}=1\+∑n=4∞cn​d−n2=1\+8π​d−2−320​π3​π4​\(π−2\)​d−52\+O⁡\(d−3\)\\displaystyle=1\+\\sum\_\{n=4\}^\{\\infty\}c\_\{n\}d^\{\-\\frac\{n\}\{2\}\}=1\+\\frac\{8\}\{\\pi\}d^\{\-2\}\-\\frac\{320\\pi\}\{3\\pi^\{4\}\(\\pi\-2\)\}d^\{\-\\frac\{5\}\{2\}\}\+O\(d^\{\-3\}\)ξ⋆00\\displaystyle\\xi\_\{\\star 0\}^\{0\}=∑n=4∞en​d−n2=−4π​d−2−32π3​d−52\+O⁡\(d−3\)\\displaystyle=\\sum\_\{n=4\}^\{\\infty\}e\_\{n\}d^\{\-\\frac\{n\}\{2\}\}=\-\\frac\{4\}\{\\pi\}d^\{\-2\}\-\\frac\{32\}\{\\pi^\{3\}\}d^\{\-\\frac\{5\}\{2\}\}\+O\(d^\{\-3\}\)ξ10\\displaystyle\\xi\_\{1\}^\{0\}=∑n=2∞fn​d−n2=2​d−1\+0​d−32\+O⁡\(d−2\)\\displaystyle=\\sum\_\{n=2\}^\{\\infty\}f\_\{n\}d^\{\-\\frac\{n\}\{2\}\}=2d^\{\-1\}\+0d^\{\-\\frac\{3\}\{2\}\}\+O\(d^\{\-2\}\)ξ01\\displaystyle\\xi^\{1\}\_\{0\}=∑n=2∞gn​d−n2=4π​d−1\+32π3​d−32\+O⁡\(d−2\)\\displaystyle=\\sum\_\{n=2\}^\{\\infty\}g\_\{n\}d^\{\-\\frac\{n\}\{2\}\}=\\frac\{4\}\{\\pi\}d^\{\-1\}\+\\frac\{32\}\{\\pi^\{3\}\}d^\{\-\\frac\{3\}\{2\}\}\+O\(d^\{\-2\}\)ξ11\\displaystyle\\xi\_\{1\}^\{1\}=−1\+∑n=2∞hn​d−n2=−1\+2​\(1\+2π\)2​d−1\+64​π−7683​π4​\(π−2\)​d−32\+O⁡\(d−2\)\\displaystyle=\-1\+\\sum\_\{n=2\}^\{\\infty\}h\_\{n\}d^\{\-\\frac\{n\}\{2\}\}=\-1\+2\\left\(1\+\\frac\{2\}\{\\pi\}\\right\)^\{2\}d^\{\-1\}\+\\frac\{64\\pi\-768\}\{3\\pi^\{4\}\(\\pi\-2\)\}d^\{\-\\frac\{3\}\{2\}\}\+O\(d^\{\-2\}\)The lossℒ⁡\(𝝃⁡\(d\)\)=\(12−1π2\)​d−1\+O⁡\(d−32\)→0\\mathcal\{L\}\(\{\\boldsymbol\{\\xi\}\}\(d\)\)=\\left\(\\frac\{1\}\{2\}\-\\frac\{1\}\{\\pi^\{2\}\}\\right\)d^\{\-1\}\+O\(d^\{\-\\frac\{3\}\{2\}\}\)\{\\rightarrow\}0, asd→∞d\{\\rightarrow\}\\infty\[[4](https://arxiv.org/html/2609.30661#bib.bib4), §8\.5\]\. The series are convergent for sufficiently largedd\. In practice, it seems possible that the series are convergent for small values ofdd, sayd≥9d\\geq 9\(implying a radius of convergence of at least1/31/3\)\.

A family\(𝝃⁡\(d\)\)d≥d0\(\{\\boldsymbol\{\\xi\}\}\(d\)\)\_\{d\\geq d\_\{0\}\}of critical points with isotropy groupΔ​Sd−​p×Sp\\Delta S\_\{d\_\{\-\}p\}\\times S\_\{p\},d−p\>pd\-p\>p, is said to be of*type I*\(resp\.*type II*\) if it has an FPS such that asd→∞d\{\\rightarrow\}\\infty,ξ00​\(d\)→−1\\xi\_\{0\}^\{0\}\(d\)\{\\rightarrow\}\-1\(resp\.\+1\+1\)\. More generally, a critical point𝝃⁡\(d0\)\{\\boldsymbol\{\\xi\}\}\(d\_\{0\}\)is of type I \(resp\. type II\) if it lies on a family of critical points of type I \(resp\. II\)\. The convergence of the FPS at the specific valued0d\_\{0\}is immaterial: what matters is the behaviour asd→∞d\{\\rightarrow\}\\inftyand that the critical point lies on a family—as defined above\.

The family of critical points of type II described above is a family of spurious \(local, not global\) minima ford≥6d\\geq 6\. Ford∈\[3,5\]d\\in\[3,5\],𝝃⁡\(d\)\{\\boldsymbol\{\\xi\}\}\(d\)is not a spurious minimum ofℒ\\mathcal\{L\}but is a non\-degenerate critical point: a bifurcation occurs at aboutd=5\.58d=5\.58—transverse to the fixed point space and associated to the standard representation ofSd−1S\_\{d\-1\}\[[6](https://arxiv.org/html/2609.30661#bib.bib6)\]\.

For families of spurious minima of type II \(resp\. type I\), the lossℒ⁡\(𝝃⁡\(d\)\)\\mathcal\{L\}\(\{\\boldsymbol\{\\xi\}\}\(d\)\)typically converges to zero \(resp\. a strictly positive constant\) asd→∞d\{\\rightarrow\}\\infty\. \(2\) We describe the FPS for a type I family\(𝝃⁡\(d\)\)\(\{\\boldsymbol\{\\xi\}\}\(d\)\)of critical points with isotropyΔ​Sd−1\\Delta S\_\{d\-1\}andm=1m=1\. Aside from giving an example of a type I family defining spurious minima, the FPS will be in powers ofd−14d^\{\-\\frac\{1\}\{4\}\}and the constant terms in the FPS lie in\{0,±12,±1\}\\\{0,\\pm\\frac\{1\}\{2\},\\pm 1\\\}\. The presence of non\-trivial termsc​d−14cd^\{\-\\frac\{1\}\{4\}\}implies that convergence is slow\. Setting𝝃⁡\(d\)=𝝃∈M⁡\(d\+1,d\)\{\\boldsymbol\{\\xi\}\}\(d\)=\{\\boldsymbol\{\\xi\}\}\\in M\(d\+1,d\), we have

𝝃00\\displaystyle\{\\boldsymbol\{\\xi\}\}^\{0\}\_\{0\}=−1\+2​d−1\+π2​d−32\+O⁡\(d−74\)\\displaystyle=\-1\+2d^\{\-1\}\+\\frac\{\\pi\}\{2\}d^\{\-\\frac\{3\}\{2\}\}\+O\(d^\{\-\\frac\{7\}\{4\}\}\)𝝃⋆00\\displaystyle\{\\boldsymbol\{\\xi\}\}^\{0\}\_\{\\star 0\}=2​d−1−π−2​d−74\+O⁡\(d−2\)\\displaystyle=2d^\{\-1\}\-\\sqrt\{\\pi\-2\}d^\{\-\\frac\{7\}\{4\}\}\+O\(d^\{\-2\}\)𝝃10\\displaystyle\{\\boldsymbol\{\\xi\}\}\_\{1\}^\{0\}=d−1−6\+3​π4​π​π−2​d−54\+O⁡\(d−32\)\\displaystyle=d^\{\-1\}\-\\frac\{6\+3\\pi\}\{4\\pi\\sqrt\{\\pi\-2\}\}d^\{\-\\frac\{5\}\{4\}\}\+O\(d^\{\-\\frac\{3\}\{2\}\}\)𝝃01\\displaystyle\{\\boldsymbol\{\\xi\}\}^\{1\}\_\{0\}=π−22​d−34\+O⁡\(d−1\)\\displaystyle=\\frac\{\\sqrt\{\\pi\-2\}\}\{2\}d^\{\-\\frac\{3\}\{4\}\}\+O\(d^\{\-1\}\)𝝃11\\displaystyle\{\\boldsymbol\{\\xi\}\}^\{1\}\_\{1\}=12\+6\+3​π8​π​π−2​d−14\+O⁡\(d−12\)\\displaystyle=\\frac\{1\}\{2\}\+\\frac\{6\+3\\pi\}\{8\\pi\\sqrt\{\\pi\-2\}\}d^\{\-\\frac\{1\}\{4\}\}\+O\(d^\{\-\\frac\{1\}\{2\}\}\)𝝃02\\displaystyle\{\\boldsymbol\{\\xi\}\}^\{2\}\_\{0\}=π−22​d−34\+O⁡\(d−1\)\\displaystyle=\\frac\{\\sqrt\{\\pi\-2\}\}\{2\}d^\{\-\\frac\{3\}\{4\}\}\+O\(d^\{\-1\}\)𝝃12\\displaystyle\{\\boldsymbol\{\\xi\}\}^\{2\}\_\{1\}=−12\+6\+3​π8​π​π−2​d−14\+O⁡\(d−12\)\\displaystyle=\-\\frac\{1\}\{2\}\+\\frac\{6\+3\\pi\}\{8\\pi\\sqrt\{\\pi\-2\}\}d^\{\-\\frac\{1\}\{4\}\}\+O\(d^\{\-\\frac\{1\}\{2\}\}\)The lossℒ⁡\(𝝃⁡\(d\)\)=12−1π−43​π​d−12−\(π−2\)323​π​d−34\+O⁡\(d−1\)\\mathcal\{L\}\(\{\\boldsymbol\{\\xi\}\}\(d\)\)=\\frac\{1\}\{2\}\-\\frac\{1\}\{\\pi\}\-\\frac\{4\}\{3\\pi\}d^\{\-\\frac\{1\}\{2\}\}\-\\frac\{\(\\pi\-2\)^\{\\frac\{3\}\{2\}\}\}\{3\\pi\}d^\{\-\\frac\{3\}\{4\}\}\+O\(d^\{\-1\}\)\. We refer to\[[5](https://arxiv.org/html/2609.30661#bib.bib5), Appendix A\]and also for the computation of the FPS for type II whenm=1,2m=1,2and isotropy isΔ​Sd−1\\Delta S\_\{d\-1\}\.※

## 14\.Computing∂ℒ∂β¯​\(𝐖,𝟎\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\}\(\{\{\\mathbf\{W\}\}\},\\mathbf\{0\}\): symmetric & asymmetric targets

In this section, results are given on theβ¯\\overline\{\\beta\}\-derivatives ofℒ\\mathcal\{L\}at points\(𝐖,𝜷¯\)∈M⁡\(k,d\+1\)\(\{\{\\mathbf\{W\}\}\},\\boldsymbol\{\\overline\{\\beta\}\}\)\\in M\(k,d\+1\), where𝐖\{\{\\mathbf\{W\}\}\}is a critical point of the unbiased network and𝜷¯=𝟎\\boldsymbol\{\\overline\{\\beta\}\}=\\mathbf\{0\}\. The bias𝜸¯∈ℝd\\boldsymbol\{\\overline\{\\gamma\}\}\\in\{\\mathbb\{R\}\}^\{d\}associated to the target𝐕\{\{\\mathbf\{V\}\}\}, is also set to𝟎∈ℝd\\mathbf\{0\}\\in\{\\mathbb\{R\}\}^\{d\}\. The critical points𝐖\{\{\\mathbf\{W\}\}\}will define spurious minima inM⁡\(k,d\)M\(k,d\)but\(𝐖,𝟎\)\(\{\{\\mathbf\{W\}\}\},\\mathbf\{0\}\)will generally*not*be a critical point inM⁡\(k,d\+1\)M\(k,d\+1\)*unless*ℒ⁡\(𝐖\)=0\\mathcal\{L\}\(\{\{\\mathbf\{W\}\}\}\)=0and so defines the global minimum ofℒ\\mathcal\{L\}\. Under this condition, theβ¯\\overline\{\\beta\}\-derivative is zero \(𝐖\{\{\\mathbf\{W\}\}\}will be a degenerate critical point ifk\>dk\>d\[[5](https://arxiv.org/html/2609.30661#bib.bib5), Appendix A\]\)\. The examples presented include symmetric spurious minima of types I and II as described in\[[4](https://arxiv.org/html/2609.30661#bib.bib4),[2](https://arxiv.org/html/2609.30661#bib.bib2),[3](https://arxiv.org/html/2609.30661#bib.bib3),[5](https://arxiv.org/html/2609.30661#bib.bib5)\]\. All of these critical points have are associated to the symmetric target𝐕=Id\{\{\\mathbf\{V\}\}\}=I\_\{d\}and can be represented by FPS in1d\\frac\{1\}\{d\}\(typically the initial non\-constant term of lowest order will be a constant multiple of1/d121/d^\{\\frac\{1\}\{2\}\}or1/d1/d, but1/d141/d^\{\\frac\{1\}\{4\}\}is possible\)\. Two additional examples are given one of which is completely asymmetric \(both target and critical point\), the other “partially asymmetric” \(the target has significant asymmetry and critical points have an FPS representation\)\.

### 14\.1\.Computation of∂ℒ∂𝜷¯​\(𝐖,𝟎\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\boldsymbol\{\\overline\{\\beta\}\}\}\(\{\{\\mathbf\{W\}\}\},\\mathbf\{0\}\): a symmetric example

We start with an example illustrating how theFPSexpansion for a family of critical points can be used to estimateβ¯\\overline\{\\beta\}\-derivatives at𝜷¯=𝟎\\boldsymbol\{\\overline\{\\beta\}\}=\\mathbf\{0\}\.

###### Example 14\.1\.

Consider the type II family\(𝔠⁡\(d\)\)d≥3\(\\mathfrak\{c\}\(d\)\)\_\{d\\geq 3\}of critical points withk=dk=dand isotropyΔ​Sd−1\\Delta S\_\{d\-1\}described in Examples[13\.2](https://arxiv.org/html/2609.30661#S13.Thmlemma2)\(1\)\. Using theFPSexpansion and the formula given in Remark[11\.5](https://arxiv.org/html/2609.30661#S11.Thmlemma5), we estimate the decay ofOPEN∂ℒ∂β¯0​\(𝔠​\(d\),𝟎\)\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{0\}\}\(\\mathfrak\{c\}\(d\),\\mathbf\{0\}\)\)\.

###### Remark 14\.2\.

The symmetry assumption implies thatOPENOPEN∂ℒ∂β¯i​\(𝔠⁡\(d\),𝟎\)\)=∂ℒ∂β¯0​\(𝔠⁡\(d\),𝟎\)\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{i\}\}\(\\mathfrak\{c\}\(d\),\\mathbf\{0\}\)\)=\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{0\}\}\(\\mathfrak\{c\}\(d\),\\mathbf\{0\}\)\), ifi∈𝐝−𝟏i\\in\{\\mathbf\{d\-1\}\}\. In general,OPENOPEN∂ℒ∂β¯0​\(𝔠⁡\(d\),𝟎\)\)≠∂ℒ∂β¯d−1​\(𝔠⁡\(d\),𝟎\)\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{0\}\}\(\\mathfrak\{c\}\(d\),\\mathbf\{0\}\)\)\\neq\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{d\-1\}\}\(\\mathfrak\{c\}\(d\),\\mathbf\{0\}\)\)\. In the sequel, we typically parameterize theβ¯\\overline\{\\beta\}derivatives by rowtype—indeed, this is what we do when we solve the critical point equations for𝝃,𝝃⋆,𝜷¯\{\\boldsymbol\{\\xi\}\},\{\\boldsymbol\{\\xi\}\}\_\{\\star\},\{\\boldsymbol\{\\overline\{\\beta\}\}\}on the associated seven dimensional fixed point space as well as in the tables presented in the remainder of this section\.✠\\maltese

Letτi\\tau\_\{i\}denote the norm of a row of𝔠⁡\(d\)\\mathfrak\{c\}\(d\)of rowtypeii,i∈𝟐i\\in\{\\mathbf\{2\}\}\. Using theFPSfor the critical point given in Examples[13\.2](https://arxiv.org/html/2609.30661#S13.Thmlemma2)\(1\), we find that

τ0\\displaystyle\\tau\_\{0\}=\\displaystyle=1\+\(2\+8π\)​d−2−320​π3​π4​\(π−2\)​d−52\+O⁡\(d−3\)\\displaystyle 1\+\\left\(2\+\\frac\{8\}\{\\pi\}\\right\)d^\{\-2\}\-\\frac\{320\\pi\}\{3\\pi^\{4\}\(\\pi\-2\)\}d^\{\-\\frac\{5\}\{2\}\}\+O\(d^\{\-3\}\)τ1\\displaystyle\\tau\_\{1\}=\\displaystyle=1−\(2\+8π2\)​d−1\+320​π3​π4​\(π−2\)​d−32\+O⁡\(d−2\)\\displaystyle 1\-\\left\(2\+\\frac\{8\}\{\\pi^\{2\}\}\\right\)d^\{\-1\}\+\\frac\{320\\pi\}\{3\\pi^\{4\}\(\\pi\-2\)\}d^\{\-\\frac\{3\}\{2\}\}\+O\(d^\{\-2\}\)Since∑j∈𝐝‖𝐰j‖=\(d−1\)​τ0\+τ1\\sum\_\{j\\in\{\\mathbf\{d\}\}\}\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|=\(d\-1\)\\tau\_\{0\}\+\\tau\_\{1\}, it follows easily that

∑j∈𝐝‖𝐰j‖=d\+O⁡\(d−2\)\\sum\_\{j\\in\{\\mathbf\{d\}\}\}\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|=d\+O\(d^\{\-2\}\)and, since∑j∈𝐝‖𝐯j‖=d\\sum\_\{j\\in\{\\mathbf\{d\}\}\}\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|=d,

12​2​π​‖𝐰i‖​\[∑j∈𝐝‖𝐯j‖−∑j∈𝐤‖𝐰j‖\]=O⁡\(d−2\),i∈𝐝−𝟏\\frac\{1\}\{2\\sqrt\{2\\pi\}\}\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|\\big\[\\sum\_\{j\\in\{\\mathbf\{d\}\}\}\\\|\{\{\\mathbf\{v\}\}\}^\{j\}\\\|\-\\sum\_\{j\\in\{\\mathbf\{k\}\}\}\\\|\{\{\\mathbf\{w\}\}\}^\{j\}\\\|\\big\]=O\(d^\{\-2\}\),\\;\\;i\\in\{\\mathbf\{d\-1\}\}For the second term in the formula of Remark[11\.5](https://arxiv.org/html/2609.30661#S11.Thmlemma5), we need to estimate theFPSof inner products\. Specifically

#### Inner products of𝐰0\{\{\\mathbf\{w\}\}\}^\{0\}with𝐰i\{\{\\mathbf\{w\}\}\}\_\{i\}and𝐯j\{\{\\mathbf\{v\}\}\}^\{j\},j∈𝐝j\\in\{\\mathbf\{d\}\}

Computing, using theFPS,

⟨𝐰0,𝐰0⟩\\displaystyle\\langle\{\{\\mathbf\{w\}\}\}^\{0\},\{\{\\mathbf\{w\}\}\}^\{0\}\\rangle=\\displaystyle=τ02=1\+O⁡\(d−2\)\\displaystyle\\tau\_\{0\}^\{2\}=1\+O\(d^\{\-2\}\)\(d−2\)​⟨𝐰0,𝐰1⟩\\displaystyle\(d\-2\)\\langle\{\{\\mathbf\{w\}\}\}^\{0\},\{\{\\mathbf\{w\}\}\}^\{1\}\\rangle=\\displaystyle=\(−8π\)​d−1−64π3​d−32\+O⁡\(d−2\)\\displaystyle\\left\(\-\\frac\{8\}\{\\pi\}\\right\)d^\{\-1\}\-\\frac\{64\}\{\\pi^\{3\}\}d^\{\-\\frac\{3\}\{2\}\}\+O\(d^\{\-2\}\)⟨𝐰0,𝐰d⟩\\displaystyle\\langle\{\{\\mathbf\{w\}\}\}^\{0\},\{\{\\mathbf\{w\}\}\}^\{d\}\\rangle=\\displaystyle=\(4π−2\)​d−1\+32π3​d−32\+O⁡\(d−2\)\\displaystyle\\left\(\\frac\{4\}\{\\pi\}\-2\\right\)d^\{\-1\}\+\\frac\{32\}\{\\pi^\{3\}\}d^\{\-\\frac\{3\}\{2\}\}\+O\(d^\{\-2\}\)⟨𝐰0,𝐯0⟩\\displaystyle\\langle\{\{\\mathbf\{w\}\}\}^\{0\},\{\{\\mathbf\{v\}\}\}^\{0\}\\rangle=\\displaystyle=1\+O⁡\(d−2\)\\displaystyle 1\+O\(d^\{\-2\}\)\(d−2\)​⟨𝐰0,𝐯1⟩\\displaystyle\(d\-2\)\\langle\{\{\\mathbf\{w\}\}\}^\{0\},\{\{\\mathbf\{v\}\}\}^\{1\}\\rangle=\\displaystyle=−4π​d−1−32π3​d−32\+O⁡\(d−2\)\\displaystyle\-\\frac\{4\}\{\\pi\}d^\{\-1\}\-\\frac\{32\}\{\\pi^\{3\}\}d^\{\-\\frac\{3\}\{2\}\}\+O\(d^\{\-2\}\)⟨𝐰0,𝐯d⟩\\displaystyle\\langle\{\{\\mathbf\{w\}\}\}^\{0\},\{\{\\mathbf\{v\}\}\}^\{d\}\\rangle=\\displaystyle=2d\+O⁡\(d−2\)\\displaystyle\\frac\{2\}\{d\}\+O\(d^\{\-2\}\)Using these estimates, it follows easily that

12​2​π​\[∑j∈𝐝⟨𝐰i,𝐯j⟩−∑j∈𝐤⟨𝐰i,𝐰j⟩\]=O⁡\(d−2\)\.\\frac\{1\}\{2\\sqrt\{2\\pi\}\}\\big\[\\sum\_\{j\\in\{\\mathbf\{d\}\}\}\\langle\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{j\}\\rangle\-\\sum\_\{j\\in\{\\mathbf\{k\}\}\}\\langle\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{j\}\\rangle\\big\]=O\(d^\{\-2\}\)\.HenceOPEN∂ℒ∂β¯0​\(𝔠​\(d\),𝟎\)\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{0\}\}\(\\mathfrak\{c\}\(d\),\\mathbf\{0\}\)\)decays at least as fast asd−2d^\{\-2\}\. With the addition of one further term in each of the fiveFPSfor𝔠⁡\(d\)\\mathfrak\{c\}\(d\), the leading term in theFPSforOPEN∂ℒ∂β¯0​\(𝔠​\(d\),𝟎\)\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{0\}\}\(\\mathfrak\{c\}\(d\),\\mathbf\{0\}\)\)can be computed\. However, it is faster and easier to do a high precision numerical computations to verify that the leading term of theFPSfor the derivative is a multiple ofd−2d^\{\-2\}\.

A high precision computation of the derivative takingd=10512d=10^\{512\}gives the value of the leading coefficientD40D^\{0\}\_\{4\}ofd−2d^\{\-2\}in theFPSfor the derivative with respect toβ¯0\\overline\{\\beta\}\_\{0\}to be

D40\\displaystyle D^\{0\}\_\{4\}=\\displaystyle=0\.541586362042490362453470777718114374878787486392237162895683618​…\\displaystyle 0\.541586362042490362453470777718114374878787486392237162895683618\\ldots\(Accurate to 64 decimal places\.\)

#### Computation of∂ℒ∂β¯0​\(𝔠​\(d\),𝟎\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{0\}\}\(\\mathfrak\{c\}\(d\),\\mathbf\{0\}\)for specific values ofdd

Using high precision computations, we tabulate values of the derivative for specific choices ofdd:

Table 1\.Numerically computed values of∂ℒ∂β¯0​\(𝔠​\(d\),𝟎\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{0\}\}\(\\mathfrak\{c\}\(d\),\\mathbf\{0\}\)for a type II critical point withk=dk=dand isotropyΔ​Sd−1\\Delta S\_\{d\-1\}Observe that the results for small values ofddare smaller than the estimate given byD40​d−2D^\{0\}\_\{4\}d^\{\-2\}\. The estimate given byD40/62D^\{0\}\_\{4\}/6^\{2\}is approximately1\.5×10−21\.5\\times 10^\{\-2\}\. If instead we taked=4d=4, the true value is approximately2\.8025×10−22\.8025\\times 10^\{\-2\}while the estimateD40​d−2D^\{0\}\_\{4\}d^\{\-2\}gives3\.34×10−23\.34\\times 10^\{\-2\}\(the family is not defined ford=3d=3\)\.※

### 14\.2\.Results for isotropyΔ​Sd\\Delta S\_\{d\}, type I \(type A in\[[4](https://arxiv.org/html/2609.30661#bib.bib4)\]\)

We give results only form=0m=0\.

Table 2\.Numerically computed values of∂ℒ∂β¯0​\(𝔠​\(d\),𝟎\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{0\}\}\(\\mathfrak\{c\}\(d\),\\mathbf\{0\}\)for a type I critical point withk=dk=dand isotropyΔ​Sd\\Delta S\_\{d\}From the table it follows that the derivative decays liked−1d^\{\-1\}and high precision computation leads to the value for the initial coefficientD2D\_\{2\}in theFPSof the derivative:

D20\\displaystyle D^\{0\}\_\{2\}=\\displaystyle=0\.144967736664468798796892840195837787855643034035842961341133133​…\\displaystyle 0\.144967736664468798796892840195837787855643034035842961341133133\\ldots\(Correct to the 64 decimal places displayed\.\)

It is straightforward to obtain an explicit formula forD20D^\{0\}\_\{2\}in terms of the coefficients of theFPS\. The initial terms for theFPSof the critical points are easy to derive \(see\[[4](https://arxiv.org/html/2609.30661#bib.bib4), §8\.3\]\):

ξ00\\displaystyle\\xi^\{0\}\_\{0\}=\\displaystyle=−1\+2​d−1\+2​\(4π−2\)​d−2\+O⁡\(d−52\)\\displaystyle\-1\+2d^\{\-1\}\+2\\left\(\\frac\{4\}\{\\pi\}\-2\\right\)d^\{\-2\}\+O\(d^\{\-\\frac\{5\}\{2\}\}\)ξ⋆00\\displaystyle\\xi^\{0\}\_\{\\star 0\}=\\displaystyle=2​d−1\+\(4π−2\)​d−2\+O⁡\(d−52\)\\displaystyle 2d^\{\-1\}\+\\left\(\\frac\{4\}\{\\pi\}\-2\\right\)d^\{\-2\}\+O\(d^\{\-\\frac\{5\}\{2\}\}\)Using the formula of Remark[11\.5](https://arxiv.org/html/2609.30661#S11.Thmlemma5), a little algebra gives the exact value ofD20D^\{0\}\_\{2\}and the asymptotics for the derivative:

∂ℒ∂β¯0​\(𝔠⁡\(d\),𝟎\)=π−2π​2​π​d−1\+O⁡\(d−32\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{0\}\}\(\\mathfrak\{c\}\(d\),\\mathbf\{0\}\)=\\frac\{\\pi\-2\}\{\\pi\\sqrt\{2\\pi\}\}d^\{\-1\}\+O\(d^\{\-\\frac\{3\}\{2\}\}\)The numerically computed vale ofD20D^\{0\}\_\{2\}agrees withπ−2π​2​π\\frac\{\\pi\-2\}\{\\pi\\sqrt\{2\\pi\}\}to the number of decimal paces shown\.

Henceforth, tabular values are only given to 4 significant figures and the coefficients of the leading term for theFPSof derivatives are accurate to the3232decimal places displayed\. Except where noted, theFPSare series ind−12d^\{\-\\frac\{1\}\{2\}\}andDniD^\{i\}\_\{n\}will be the leading non\-zero coefficient ofd−n2d^\{\-\\frac\{n\}\{2\}\}in theFPSfor∂ℒ/∂β¯i\\partial\\mathcal\{L\}/\\partial\\overline\{\\beta\}\_\{i\}\. Thus ifn=2n=2, the derivatives decay liked−1d^\{\-1\}\.

### 14\.3\.Results for isotropyΔ​Sd−1\\Delta S\_\{d\-1\}, type I,m=0,1,2m=0,1,2

Values of the derivatives are given ford=10,103d=10,10^\{3\}and10610^\{6\}\.

Table 3\.Computed values of∂ℒ∂β¯i​\(𝔠​\(d\),𝟎\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{i\}\}\(\\mathfrak\{c\}\(d\),\\mathbf\{0\}\),i∈𝟐i\\in\{\\mathbf\{2\}\}, for a type I critical point withm=0m=0and isotropyΔ​Sd−1\\Delta S\_\{d\-1\}\.Initial coefficients for theFPSof derivatives:

D20\\displaystyle D\_\{2\}^\{0\}=\\displaystyle=0\.144967736664468798796892840195838​…\\displaystyle 0\.144967736664468798796892840195838\\ldotsD21\\displaystyle D\_\{2\}^\{1\}=\\displaystyle=0\.197646145812959995056847958971127​…\\displaystyle 0\.197646145812959995056847958971127\\ldots
Table 4\.Computed values of∂ℒ∂β¯i​\(𝔠​\(d\),𝟎\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{i\}\}\(\\mathfrak\{c\}\(d\),\\mathbf\{0\}\),i∈𝟑i\\in\{\\mathbf\{3\}\}, for a type I critical point withm=1m=1and isotropyΔ​Sd−1\\Delta S\_\{d\-1\}\.TheFPSis in powers ofd−14d^\{\-\\frac\{1\}\{4\}\}and decay of derivatives is either liked−1d^\{\-1\}\(D40D^\{0\}\_\{4\}\) ord−34d^\{\-\\frac\{3\}\{4\}\}\(D31,D32D^\{1\}\_\{3\},D^\{2\}\_\{3\}\)\. Initial coefficients of theFPSfor the derivatives:

D40\\displaystyle D\_\{4\}^\{0\}=\\displaystyle=0\.144967736664468798796892840195838​…\\displaystyle 0\.144967736664468798796892840195838\\ldotsD31\\displaystyle D\_\{3\}^\{1\}=\\displaystyle=0\.0774456350768987130272615429004301​…\\displaystyle 0\.0774456350768987130272615429004301\\ldotsD32\\displaystyle D\_\{3\}^\{2\}=\\displaystyle=0\.0774456350768987130272615429004301​…\\displaystyle 0\.0774456350768987130272615429004301\\ldots
Table 5\.Computed values of∂ℒ∂β¯i​\(𝔠​\(d\),𝟎\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{i\}\}\(\\mathfrak\{c\}\(d\),\\mathbf\{0\}\),i∈𝟒i\\in\{\\mathbf\{4\}\}, for a type I critical point withm=2m=2and isotropyΔ​Sd−1\\Delta S\_\{d\-1\}\.TheFPSis in powers ofd−14d^\{\-\\frac\{1\}\{4\}\}and decay of derivatives is either liked−1d^\{\-1\}\(D4iD\_\{4\}^\{i\}\) ord−34d^\{\-\\frac\{3\}\{4\}\}\(D3iD\_\{3\}^\{i\}\)\. Initial coefficients of theFPSfor the derivatives:

D40\\displaystyle D\_\{4\}^\{0\}=\\displaystyle=0\.14496773666446879879689284019584​…\\displaystyle 0\.14496773666446879879689284019584\\ldotsD31\\displaystyle D\_\{3\}^\{1\}=\\displaystyle=0\.077445635076898713027261542900430​…\\displaystyle 0\.077445635076898713027261542900430\\ldotsD32\\displaystyle D\_\{3\}^\{2\}=\\displaystyle=0\.077445635076898713027261542900430​…\\displaystyle 0\.077445635076898713027261542900430\\ldotsD43\\displaystyle D\_\{4\}^\{3\}=\\displaystyle=0\.18171273140268941937100246131777​…\\displaystyle 0\.18171273140268941937100246131777\\ldots

### 14\.4\.Results for isotropyΔ​Sd−1\\Delta S\_\{d\-1\}, type II,m=0,1,2m=0,1,2

The next three tables give results for type II critical points with isotropyΔ​Sd−1\\Delta S\_\{d\-1\}\.

Table 6\.Computed values of∂ℒ∂β¯i​\(𝔠​\(d\),𝟎\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{i\}\}\(\\mathfrak\{c\}\(d\),\\mathbf\{0\}\),i∈𝟐i\\in\{\\mathbf\{2\}\}, for a type II critical point withm=0m=0and isotropyΔ​Sd−1\\Delta S\_\{d\-1\}\.Initial coefficients for theFPSof derivatives:

D40\\displaystyle D\_\{4\}^\{0\}=\\displaystyle=0\.541586362042490362453470777718114​…\\displaystyle 0\.541586362042490362453470777718114\\ldotsD21\\displaystyle D\_\{2\}^\{1\}=\\displaystyle=−0\.237257064180446401333830561616386​…\\displaystyle\-0\.237257064180446401333830561616386\\ldots
Table 7\.Computed values of∂ℒ∂β¯i​\(𝔠​\(d\),𝟎\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{i\}\}\(\\mathfrak\{c\}\(d\),\\mathbf\{0\}\),i∈𝟑i\\in\{\\mathbf\{3\}\}, for a type I critical point withm=1m=1and isotropyΔ​Sd−1\\Delta S\_\{d\-1\}\.TheFPSis in powers ofd−12d^\{\-\\frac\{1\}\{2\}\}and decay of the the derivative is either liked−32d^\{\-\\frac\{3\}\{2\}\}ord−1d^\{\-1\}\. Initial coefficients of theFPSfor the derivatives:

D30\\displaystyle D^\{0\}\_\{3\}=\\displaystyle=−0\.0513640604753390711588488689614188​…\\displaystyle\-0\.0513640604753390711588488689614188\\ldotsD21\\displaystyle D^\{1\}\_\{2\}=\\displaystyle=−0\.213238489802580052077245332033039​…\\displaystyle\-0\.213238489802580052077245332033039\\ldotsD32\\displaystyle D^\{2\}\_\{3\}=\\displaystyle=−0\.237042923895697357373281965133800​…\\displaystyle\-0\.237042923895697357373281965133800\\ldots
###### Remark 14\.3\.

S2S\_\{2\}acts freely on the last two rows \(rowtypes 1, 2\)\. Interchanging these rows will swap the asymptotics\. In general, if isotropy isΔ⁡\(Sd−1×S1\)\\Delta\(S\_\{d\-1\}\\times S\_\{1\}\)andm\>0m\>0, there will be a free action ofSmS\_\{m\}on the lastm\+1m\+1rows and corresponding free action on the fixed point space\.✠\\maltese

Table 8\.Computed values of∂ℒ∂β¯i​\(𝔠​\(d\),𝟎\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{i\}\}\(\\mathfrak\{c\}\(d\),\\mathbf\{0\}\),i∈𝟒i\\in\{\\mathbf\{4\}\}, for a type II critical point withm=2m=2and isotropyΔ​Sd−1\\Delta S\_\{d\-1\}\.TheFPSis in powers ofd−12d^\{\-\\frac\{1\}\{2\}\}and derivatives decay liked−32d^\{\-\\frac\{3\}\{2\}\}ord−1d^\{\-1\}\. Initial coefficients of theFPSfor the derivatives:

D30\\displaystyle D^\{0\}\_\{3\}=\\displaystyle=−0\.0515991045178723939348354300977768​…\\displaystyle\-0\.0515991045178723939348354300977768\\ldotsD21\\displaystyle D^\{1\}\_\{2\}=\\displaystyle=−0\.213189482387164650802356811574747​…\\displaystyle\-0\.213189482387164650802356811574747\\ldotsD32\\displaystyle D^\{2\}\_\{3\}=\\displaystyle=−0\.187109834651767919653399769806042​…\\displaystyle\-0\.187109834651767919653399769806042\\ldotsD33\\displaystyle D^\{3\}\_\{3\}=\\displaystyle=−0\.0804839616208356631748165154342071​…\\displaystyle\-0\.0804839616208356631748165154342071\\ldots

### 14\.5\.Results for isotropyGd−2,2Δ=Δ⁡\(Sd−2×S2\)G^\{\\Delta\}\_\{d\-2,2\}=\\Delta\(S\_\{d\-2\}\\times S\_\{2\}\), type I,m=0,1,2m=0,1,2

For this example, asd→∞d\{\\rightarrow\}\\infty,ξ00→\+1\\xi\_\{0\}^\{0\}\{\\rightarrow\}\+1,ξ11→−1\\xi\_\{1\}^\{1\}\{\\rightarrow\}\-1and all off\-diagonal entries converge to zero\.

Table 9\.Computed values of∂ℒ∂β¯i​\(𝔠​\(d\),𝟎\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{i\}\}\(\\mathfrak\{c\}\(d\),\\mathbf\{0\}\),i∈𝟐i\\in\{\\mathbf\{2\}\}, for a type I critical point withm=0m=0and isotropyGd−2,2ΔG^\{\\Delta\}\_\{d\-2,2\}\.Initial coefficients of theFPSind−12d^\{\-\\frac\{1\}\{2\}\}for derivatives:

D20\\displaystyle D\_\{2\}^\{0\}=\\displaystyle=0\.144967736664468798796892840195838​…\\displaystyle 0\.144967736664468798796892840195838\\ldotsD21\\displaystyle D\_\{2\}^\{1\}=\\displaystyle=0\.197646145812959995056847958971127​…\\displaystyle 0\.197646145812959995056847958971127\\ldots
Table 10\.Computed values of∂ℒ∂β¯i​\(𝔠​\(d\),𝟎\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{i\}\}\(\\mathfrak\{c\}\(d\),\\mathbf\{0\}\),i∈𝟑i\\in\{\\mathbf\{3\}\}, for a type I critical point withm=1m=1and isotropyGd−2,2ΔG^\{\\Delta\}\_\{d\-2,2\}\.TheFPSis in powers ofd−12d^\{\-\\frac\{1\}\{2\}\}and decay of the the derivative is liked−1d^\{\-1\}\. Initial coefficients of theFPSfor the derivatives:

D20\\displaystyle D^\{0\}\_\{2\}=\\displaystyle=0\.144967736664468798796892840195838​…\\displaystyle 0\.144967736664468798796892840195838\\ldotsD21\\displaystyle D^\{1\}\_\{2\}=\\displaystyle=0\.304176867750552726817930542844234​…\\displaystyle 0\.304176867750552726817930542844234\\ldotsD22\\displaystyle D^\{2\}\_\{2\}=\\displaystyle=0\.477893257041395097510780683943154​…\\displaystyle 0\.477893257041395097510780683943154\\ldots
Table 11\.Computed values of∂ℒ∂β¯i​\(𝔠​\(d\),𝟎\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{i\}\}\(\\mathfrak\{c\}\(d\),\\mathbf\{0\}\),i∈𝟒i\\in\{\\mathbf\{4\}\}, for a type I critical point withm=2m=2and isotropyGd−2,2ΔG^\{\\Delta\}\_\{d\-2,2\}\.TheFPSis in powers ofd−12d^\{\-\\frac\{1\}\{2\}\}and derivatives decay liked−1d^\{\-1\}\. Initial coefficients of theFPSfor the derivatives:

D20\\displaystyle D^\{0\}\_\{2\}=\\displaystyle=0\.144967736664468798796892840195838​…\\displaystyle 0\.144967736664468798796892840195838\\ldotsD21\\displaystyle D^\{1\}\_\{2\}=\\displaystyle=0\.301137183047154477921400976012972​…\\displaystyle 0\.301137183047154477921400976012972\\ldotsD22\\displaystyle D^\{2\}\_\{2\}=\\displaystyle=0\.365035379908109906816379948976053​e​…\\displaystyle 0\.365035379908109906816379948976053e\\ldotsD23\\displaystyle D^\{3\}\_\{2\}=\\displaystyle=0\.134277248901558876075204468076051​…\\displaystyle 0\.134277248901558876075204468076051\\ldots

### 14\.6\.Results for isotropyΔ⁡\(Sd−2×S2\)\\Delta\(S\_\{d\-2\}\\times S\_\{2\}\), type II,m=0,1m=0,1

For this example, asd→∞d\{\\rightarrow\}\\infty,ξ00→\+1\\xi\_\{0\}^\{0\}\{\\rightarrow\}\+1,ξ11→−1\\xi\_\{1\}^\{1\}\{\\rightarrow\}\-1and all off\-diagonal entries converge to zero\. Only results are given form=0,1m=0,1as at the time of writing, we are unaware of a type II solution with anFPSwhich originates from the type II solution withm=1m=1described below\.

Table 12\.Computed values of∂ℒ∂β¯i​\(𝔠​\(d\),𝟎\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{i\}\}\(\\mathfrak\{c\}\(d\),\\mathbf\{0\}\),i∈𝟐i\\in\{\\mathbf\{2\}\}, for a type II critical point withm=0m=0and isotropyGd−2,2ΔG^\{\\Delta\}\_\{d\-2,2\}\.Initial coefficients of theFPSind−12d^\{\-\\frac\{1\}\{2\}\}for derivatives:

D40\\displaystyle D\_\{4\}^\{0\}=\\displaystyle=1\.08317272408498072490694155543623​…\\displaystyle 1\.08317272408498072490694155543623\\ldotsD21\\displaystyle D\_\{2\}^\{1\}=\\displaystyle=−0\.237257064180446401333830561616386​…\\displaystyle\-0\.237257064180446401333830561616386\\ldots
Table 13\.Computed values of∂ℒ∂β¯i​\(𝔠​\(d\),𝟎\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{i\}\}\(\\mathfrak\{c\}\(d\),\\mathbf\{0\}\),i∈𝟑i\\in\{\\mathbf\{3\}\}, for a type II critical point withm=1m=1and isotropyGd−2,2ΔG^\{\\Delta\}\_\{d\-2,2\}\.TheFPSis in powers ofd−12d^\{\-\\frac\{1\}\{2\}\}and decay of the the derivative is liked−1d^\{\-1\}\. Initial coefficients of theFPSfor the derivatives:

D30\\displaystyle D^\{0\}\_\{3\}=\\displaystyle=−0\.0735719750760874284557949071165949​…\\displaystyle\-0\.0735719750760874284557949071165949\\ldotsD21\\displaystyle D^\{1\}\_\{2\}=\\displaystyle=−0\.209265817903196378166307212398253​…\\displaystyle\-0\.209265817903196378166307212398253\\ldotsD32\\displaystyle D^\{2\}\_\{3\}=\\displaystyle=0\.406200155553695625169686284999401​…\\displaystyle 0\.406200155553695625169686284999401\\ldots

### 14\.7\.Results for isotropyΔ⁡\(Sd−p×Sp\)\\Delta\(S\_\{d\-p\}\\times S\_\{p\}\), type II,p\>2p\>2

Examples of critical point families are given which haveFPSrepresentations, isotropyGd−p,pΔG^\{\\Delta\}\_\{d\-p,p\},p\>2p\>2, and are such that diagonal entries converge to±1\\pm 1\. These solutions can all represented in a two parameter critical point family\(𝝃⁡\(p,q\)\)\(\{\\boldsymbol\{\\xi\}\}\(p,q\)\)with isotropyGp,qΔG^\{\\Delta\}\_\{p,q\}\. Ifddis even, there is a family\(𝝃⁡\(p,p\)\)\(\{\\boldsymbol\{\\xi\}\}\(p,p\)\)with isotropyGp,pΔG^\{\\Delta\}\_\{p,p\}such that half the diagonal entries converge to\+1\+1and half to−1\-1\. Here the isotropy is*not*a proper maximal subgroup ofS2​pS\_\{2p\}—the maximal proper subgroup isS2≀SpS\_\{2\}\\wr S\_\{p\}\.

Table 14\.Computed values of∂ℒ∂β¯i​\(𝔠​\(d\),𝟎\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{i\}\}\(\\mathfrak\{c\}\(d\),\\mathbf\{0\}\),i∈𝟐i\\in\{\\mathbf\{2\}\}, for type II critical points withp=3,4p=3,4andd−p=pd\-p=pIfd−p=pd\-p=p, the diagonal entries of the upper diagonal block converge to\+1\+1asd→∞d\{\\rightarrow\}\\infty\.Ifd−p=pd\-p=p, theFPSforβ¯\\overline\{\\beta\}derivatives are in powers ofd−12d^\{\-\\frac\{1\}\{2\}\}and derivatives decay liked−1d^\{\-1\}\. Initial coefficients of theFPSfor the derivatives:

D20\\displaystyle D^\{0\}\_\{2\}=\\displaystyle=0\.2119019935040464​…​D21=0\.1135488183150048​…\\displaystyle 0\.2119019935040464\\ldots\\;\\;D^\{1\}\_\{2\}=0\.1135488183150048\\ldots

### 14\.8\.Asymmetric targets and fractional power series

We give two examples where the target is notIdI\_\{d\}\. In the first example the target𝐓∈M⁡\(d,d\)\{\\mathbf\{T\}\}\\in M\(d,d\)is block diagonal with the upper diagonal block beingId−6I\_\{d\-6\}and the remaining blockA∈M⁡\(6,6\)A\\in M\(6,6\)having no symmetries with respect to the action ofS6r×S6cS\_\{6\}^\{r\}\\times S\_\{6\}^\{c\}\. The off diagonal blocks are matrices of zeroes\. The second example is an asymmetric matrix inM⁡\(10,10\)M\(10,10\)which is not fixed by any non\-identity element ofS10r×S10cS\_\{10\}^\{r\}\\times S\_\{10\}^\{c\}\.

#### Path following deformations of the target

Starting with the target𝐕=Id\{\{\\mathbf\{V\}\}\}=I\_\{d\}and a non\-degenerate critical point𝝃\{\\boldsymbol\{\\xi\}\}, the basic idea is to path follow𝝃\{\\boldsymbol\{\\xi\}\}as the target𝐕\{\{\\mathbf\{V\}\}\}is deformed to the target𝐓∈M⁡\(d,d\)\{\\mathbf\{T\}\}\\in M\(d,d\)\. Specifically, choose the linear path𝐓λ=λ​𝐓\+\(1−λ\)​𝐕\{\\mathbf\{T\}\}\_\{\\lambda\}=\\lambda\{\\mathbf\{T\}\}\+\(1\-\\lambda\)\{\{\\mathbf\{V\}\}\},λ∈\[0,1\]\\lambda\\in\[0,1\], from𝐕\{\{\\mathbf\{V\}\}\}to𝐓\{\\mathbf\{T\}\}\. Let𝝃~​\(λ\)\\widetilde\{\{\\boldsymbol\{\\xi\}\}\}\(\\lambda\)denote the conjectured continuous path of critical points joining𝝃~​\(0\)=def𝝃\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(0\)\\stackrel\{\{\\scriptstyle\\mathrm\{def\}\}\}\{\{=\}\}\{\\boldsymbol\{\\xi\}\}to the critical point𝝃~=def𝝃~​\(1\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\\stackrel\{\{\\scriptstyle\\mathrm\{def\}\}\}\{\{=\}\}\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(1\)for the target𝐓1=𝐓\{\\mathbf\{T\}\}\_\{1\}=\{\\mathbf\{T\}\}\. To this end, break\[0,1\]\[0,1\]intoNNequal subintervals each of length1/N1/N,N∈ℕN\\in\\mbox\{$\\mathbb\{N\}$\}\. We are given the critical point𝝃=𝝃~​\(0\)\{\\boldsymbol\{\\xi\}\}=\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(0\)associated to the target𝐓0=𝐕\{\\mathbf\{T\}\}\_\{0\}=\{\{\\mathbf\{V\}\}\}\. Proceeding inductively, suppose we have computed the critical point𝝃~​\(i\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(i\)associated to the target𝐓iN\{\\mathbf\{T\}\}\_\{\\frac\{i\}\{N\}\}, for alli≤n<Ni\\leq n<N\. The next step is to change the target to𝐓n\+1N\{\\mathbf\{T\}\}\_\{\\frac\{n\+1\}\{N\}\}, take the approximate solution𝝃~​\(n\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(n\)and then check that Newton’s method gives convergence to a critical point𝝃~​\(n\+1\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(n\+1\)associated to the target𝐓n\+1N\{\\mathbf\{T\}\}\_\{\\frac\{n\+1\}\{N\}\}\. The process terminates with𝝃~​\(N\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(N\)which will be a critical point associated to the target𝐓1=𝐓\{\\mathbf\{T\}\}\_\{1\}=\{\\mathbf\{T\}\}\. Suppose, for example, that𝝃\{\\boldsymbol\{\\xi\}\}a type II critical point\. Providedd≥6d\\geq 6\[[4](https://arxiv.org/html/2609.30661#bib.bib4), §8\],𝝃\{\\boldsymbol\{\\xi\}\}is a non\-degenerate spurious minimum\. Provided‖𝐓−𝐕‖\\\|\{\\mathbf\{T\}\}\-\{\{\\mathbf\{V\}\}\}\\\|is sufficiently small,𝝃~\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}will be a non\-degenerate spurious minimum since non\-degenerate critical points are stable under perturbation of the underlying dynamics\. However, since the bifurcation of𝝃\{\\boldsymbol\{\\xi\}\}from saddle to spurious minimum occurs neard=5\.6d=5\.6, the expectation is that forddnear66, path following is likely to be difficult in view of the complex dynamics close to the bifurcation \(see\[[6](https://arxiv.org/html/2609.30661#bib.bib6)\]for more on this\)\. However, for type II critical points convergence of eigenvalues as functions ofddis fast near the bifurcation\[[2](https://arxiv.org/html/2609.30661#bib.bib2)\]while for type I growth is relatively slow \(*op\. cit*\)\. As a result, path following is likely to be harder for type I critical points close to bifurcation \(neard=7d=7for type I\)\. This turns out to be the case \(see notes on the first example below\)\. A consequence is that it is sometimes helpful to use piecewise linear paths to avoid points close to bifurcation and/or Newton fails to converge\.

Even when target𝐓\{\\mathbf\{T\}\}and𝝃~​\(N\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(N\)have very few or no symmetries, it is still possible to obtainFPSexpansions for critical points and this is illustrated in both examples\.

###### Example 14\.4\(Partial asymmetry with a6×66\\times 6asymmetric block\)\.

As asymmetric block, take the matrixA∈M⁡\(6,6\)A\\in M\(6,6\)defined by

A=\[1\.030\.04−0\.020\.070\.0−0\.020\.0160\.96−0\.026−0\.0280\.0−0\.0260\.0560\.02761\.040\.00\.0760\.0−0\.03970\.03960\.01\.010\.0−0\.070\.040\.00\.0−0\.02770\.970\.010\.0112−0\.020\.05−0\.0030\.0381\.1\]A=\\left\[\\begin\{matrix\}1\.03&0\.04&\-0\.02&0\.07&0\.0&\-0\.02\\\\ 0\.016&0\.96&\-0\.026&\-0\.028&0\.0&\-0\.026\\\\ 0\.056&0\.0276&1\.04&0\.0&0\.076&0\.0\\\\ \-0\.0397&0\.0396&0\.0&1\.01&0\.0&\-0\.07\\\\ 0\.04&0\.0&0\.0&\-0\.0277&0\.97&0\.01\\\\ 0\.0112&\-0\.02&0\.05&\-0\.003&0\.038&1\.1\\end\{matrix\}\\right\]Ford≥8d\\geq 8, extend to the matrix𝐓d\{\\mathbf\{T\}\}\_\{d\}defined as the block diagonal matrix

𝐓d=\[Id−6,A\]∈M⁡\(d,d\)\{\\mathbf\{T\}\}\_\{d\}=\[I\_\{d\-6\},A\]\\in M\(d,d\)Take a value ofd≫6d\\gg 6—the value ofddat which the type II critical point bifurcates to a spurious minimum–for exampled=100d=100\. Using the path following method described above, compute the critical point𝝃~​\(100\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(100\)associated to the target𝐓100\{\\mathbf\{T\}\}\_\{100\}and then varyingddand using path following compute type II critical points for𝐓10,𝐓10n\{\\mathbf\{T\}\}\_\{10\},\{\\mathbf\{T\}\}\_\{10^\{n\}\},n∈\{6,7,8\}n\\in\\\{6,7,8\\\}\. As part this process, the 7 derivatives∂ℒ/∂β¯i\\partial\\mathcal\{L\}/\\partial\\overline\{\\beta\}\_\{i\},i∈𝟕i\\in\{\\mathbf\{7\}\}atβ¯=𝟎\\overline\{\\beta\}=\\mathbf\{0\}are also computed\.

Table 15\.Computed values of∂ℒ∂β¯i​\(𝔠​\(d\),𝟎\)\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{i\}\}\(\\mathfrak\{c\}\(d\),\\mathbf\{0\}\),i∈𝟑i\\in\{\\mathbf\{3\}\}, for a type II critical point withm=1m=1and isotropyGd−2,2ΔG^\{\\Delta\}\_\{d\-2,2\}\.Referring to Table[15](https://arxiv.org/html/2609.30661#S14.T15), the second column gives the derivative with respect toβ¯0\\overline\{\\beta\}\_\{0\}and the rapid decay reflects theΔ​Sd−6\\Delta S\_\{d\-6\}symmetry of theId−6I\_\{d\-6\}block\. All the decay rates are characteristic of type II\. The results of the table suggest anFPSfor the critical point in powers ofd−12d^\{\-\\frac\{1\}\{2\}\}\. On this basis, estimate numerically lower order coefficients in theFPSand then compute the associated critical point for the target withd=10512d=10^\{512\}\. We have the following estimate for the initial non\-zero coefficient of the seven derivatives\. This was computed takingd=10512d=10^\{512\}and the terms are accurate to the 16 significant figures displayed and consistent with the values shown in Table[15](https://arxiv.org/html/2609.30661#S14.T15),

D40=0\.6211900643271606D21=−0\.002193417454242254D22=−0\.008698020018753445D23=−0\.008698020018753445D24=0\.009751822809799718D25=−0\.014474815696883097D26=−0\.2878453022368189\\begin\{matrix\}D^\{0\}\_\{4\}&=&0\.6211900643271606&&\\\\ D^\{1\}\_\{2\}&=&\-0\.002193417454242254&D^\{2\}\_\{2\}&=\-0\.008698020018753445\\\\ D^\{3\}\_\{2\}&=&\-0\.008698020018753445&D^\{4\}\_\{2\}&=0\.009751822809799718\\\\ D^\{5\}\_\{2\}&=&\-0\.014474815696883097&D^\{6\}\_\{2\}&=\-0\.2878453022368189\\end\{matrix\}The loss decays liked−1d^\{\-1\}with initial coefficient0\.36076059​…0\.36076059\.\.\.\.

###### Remarks 14\.5\.

\(1\) Although we have not given numerical approximations for the critical points, it is worth remarking that asd→∞d\{\\rightarrow\}\\infty, the constant terms in theFPSconverge to\[Id−6,𝐓\]\[I\_\{d\-6\},\{\\mathbf\{T\}\}\]*except that the last row of components converges to the negative of the last row of𝐓\{\\mathbf\{T\}\}*\. That is, if we set𝝃~​\(d\)=𝝃~\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)=\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}, then𝝃j6~→−t6​j\\widetilde\{\{\\boldsymbol\{\\xi\}\}^\{6\}\_\{j\}\}\{\\rightarrow\}\-t\_\{6j\}, asd→∞d\{\\rightarrow\}\\infty,j=0,…,6j=0,\\ldots,6\. Similar statements are valid for asymmetric critical points resulting from deformation of critical points of type I\. \(2\) There is no difficulty in constructing a path fromd=10d=10tod=9d=9\. However,d=8d=8provides a challenge and even with a piecewise linear path with several linear segments, we were unable to reachd=8d=8\.✠\\maltese

###### Example 14\.6\(Full asymmetry ford=k=10d=k=10\)\.

Take as target the matrix𝐓10=𝐓∈M⁡\(10,10\)\{\\mathbf\{T\}\}\_\{10\}=\{\\mathbf\{T\}\}\\in M\(10,10\)defined by

\[1\.090\.1−0\.040\.020\.07−0\.10\.04−0\.080\.10\.010\.051\.070\.040\.010\.111−0\.008760\.020\.09−0\.030\.1110\.050\.031\.120\.04−0\.0430\.043\.25​e\-03−0\.10\.008−0\.0240\.040\.0080\.10\.970\.0050\.010\.032−0\.0040\.007560\.850\.025\.078​e\-05−0\.030\.061\.1117−0\.060\.0020\.015\.78​e\-050\.020\.0060\.090\.130\.0025\.6​e\-061\.036090\.070\.005−0\.137\.77​e\-04−0\.050\.09090\.02−0\.0070\.0880\.0121\.20\.0091−0\.070\.00090\.0−0\.0020\.020\.041−0\.1400\.070\.01\.30\.0030\.010\.0008−0\.10\.0006−0\.020\.0020\.0070\.040\.0031\.140\.0040\.00390\.10\.0070\.0020\.00070\.05−0\.030\.0050\.00\.95\]\\hskip\-21\.68121pt\\left\[\\begin\{matrix\}1\.09&0\.1&\-0\.04&0\.02&0\.07&\-0\.1&0\.04&\-0\.08&0\.1&0\.01\\\\ 0\.05&1\.07&0\.04&0\.01&0\.111&\-0\.00876&0\.02&0\.09&\-0\.03&0\.111\\\\ 0\.05&0\.03&1\.12&0\.04&\-0\.043&0\.04&3\.25\\text\{e\-03\}&\-0\.1&0\.008&\-0\.024\\\\ 0\.04&0\.008&0\.1&0\.97&0\.005&0\.01&0\.032&\-0\.004&0\.00756&0\.85\\\\ 0\.02&5\.078\\text\{e\-05\}&\-0\.03&0\.06&1\.1117&\-0\.06&0\.002&0\.01&5\.78\\text\{e\-05\}&0\.02\\\\ 0\.006&0\.09&0\.13&0\.002&5\.6\\text\{e\-06\}&1\.03609&0\.07&0\.005&\-0\.13&7\.77\\text\{e\-04\}\\\\ \-0\.05&0\.0909&0\.02&\-0\.007&0\.088&0\.012&1\.2&0\.0091&\-0\.07&0\.0009\\\\ 0\.0&\-0\.002&0\.02&0\.041&\-0\.140&0\.07&0\.0&1\.3&0\.003&0\.01\\\\ 0\.0008&\-0\.1&0\.0006&\-0\.02&0\.002&0\.007&0\.04&0\.003&1\.14&0\.004\\\\ 0\.0039&0\.1&0\.007&0\.002&0\.0007&0\.05&\-0\.03&0\.005&0\.0&0\.95\\end\{matrix\}\\right\]This matrix has no permutation symmetries\. It is also not a small perturbation ofI10I\_\{10\}—note in particular the matrix entryt3,9t\_\{3,9\}\(indexing from00to99\)\.

Starting with the target𝐕=I10\{\{\\mathbf\{V\}\}\}=I\_\{10\}and the spurious minimum the critical point of type II \(k=d=10k=d=10and isotropyΔ​S9\\Delta S\_\{9\}\), path follow𝝃\{\\boldsymbol\{\\xi\}\}as the target𝐕\{\{\\mathbf\{V\}\}\}is deformed to the target𝐓\{\\mathbf\{T\}\}as described previously\. Using 40 iterations per step, Newton’s method converges at every step and we obtain a path connecting𝝃\{\\boldsymbol\{\\xi\}\}to a critical point𝝃~​\(10\)=𝝃~\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(10\)=\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}associated to the target𝐓\{\\mathbf\{T\}\}\. The path following procedure was implemented takingN=100N=100andN=500N=500and a log kept of the convergence at the beginning of each step as well the error at the end of the step measured by the size of the components ofgrad​\(ℒ\)​\(𝝃~​\(i\)\)\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}\(\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(i\)\)\. Computations were done with20242024\-bit precision and the errors at the end of each step were always10−61610^\{\-616\}or better\. There were no indications of bifurcation or change in stability along the path and so it is likely that𝝃~\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}is a non\-degenerate spurious minimum\.

Assuming zero bias, the 10 derivatives∂ℒ/∂β¯i\\partial\\mathcal\{L\}/\\partial\\overline\{\\beta\}\_\{i\}are given to 16 significant figures by

D0=1\.603536471231742​e\-03D1=2\.714453782566770​e\-03D2=1\.352298945591707​e\-03D3=8\.185795884148845​e\-03D4=1\.603140416072457​e\-03D5=1\.956506389478921​e\-03D6=1\.484721381112565​e\-03D7=1\.950916845556952​e\-03D8=1\.810232124741424​e\-03D9=−1\.610119156961948​e\-03\\begin\{matrix\}D^\{0\}=1\.603536471231742\\text\{e\-03\}&D^\{1\}=2\.714453782566770\\text\{e\-03\}\\\\ D^\{2\}=1\.352298945591707\\text\{e\-03\}&D^\{3\}=8\.185795884148845\\text\{e\-03\}\\\\ D^\{4\}=1\.603140416072457\\text\{e\-03\}&D^\{5\}=1\.956506389478921\\text\{e\-03\}\\\\ D^\{6\}=1\.484721381112565\\text\{e\-03\}&D^\{7\}=1\.950916845556952\\text\{e\-03\}\\\\ D^\{8\}=1\.810232124741424\\text\{e\-03\}&D^\{9\}=\-1\.610119156961948\\text\{e\-03\}\\end\{matrix\}
###### Remark 14\.7\.

Even though the target is not close to being symmetric, the derivatives withd=10d=10are all smaller than10−210^\{\-2\}, as in previous symmetric examples\. Note howeverD3D^\{3\}, which is the largest derivative\. The larger value ofD3D^\{3\}is likely a consequence of the big symmetry breaking perturbation of0\.850\.85seen in the final entry of the fourth row of𝐓\{\\mathbf\{T\}\}\.✠\\maltese

Although𝐓10\{\\mathbf\{T\}\}\_\{10\}and𝝃~​\(10\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(10\)have no permutation symmetries \(that is, trivial isotropy under the action ofS10r×S10cS^\{r\}\_\{10\}\\times S^\{c\}\_\{10\}\), it is possible to regard𝝃~​\(10\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(10\)as analytically connected to a family𝝃~​\(d\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)defined ford≥12d\\geq 12and such that𝝃~​\(d\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)has isotropyΔ​Sd−10\\Delta S\_\{d\-10\}\. We explain the basic idea\. Starting with the matrix𝐓10\{\\mathbf\{T\}\}\_\{10\}we define𝐓10⋆\{\\mathbf\{T\}\}\_\{10\}^\{\\star\}to be the10×1010\\times 10matrix with diagonal blocksI2I\_\{2\}and𝐓8\{\\mathbf\{T\}\}\_\{8\}, where𝐓8\{\\mathbf\{T\}\}\_\{8\}is the8×88\\times 8\-matrix defined by the last 8 rows and columns of𝐓10\{\\mathbf\{T\}\}\_\{10\}\. The two off diagonal blocks of𝐓10⋆\{\\mathbf\{T\}\}\_\{10\}^\{\\star\}are zero matrices\. We take the linear path from𝐓10\{\\mathbf\{T\}\}\_\{10\}to𝐓10⋆\{\\mathbf\{T\}\}\_\{10\}^\{\\star\}and let𝝃~⋆​\(10\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}^\{\\star\}\(10\)denote the corresponding critical point obtained by path following\. The critical point𝝃~⋆\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}^\{\\star\}has isotropyΔ​S2\\Delta S\_\{2\}\. The fixed point space for the action ofΔ​S2\\Delta S\_\{2\}onM⁡\(10,10\)M\(10,10\)is of dimension8282\. On account of theS2S\_\{2\}symmetry, we can path follow𝝃~⋆​\(d\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}^\{\\star\}\(d\)asddis increased from1010to1212\. The isotropy of𝝃~⋆​\(12\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}^\{\\star\}\(12\)will beΔ​S4\\Delta S\_\{4\}\. Regard the10×1010\\times 10lower diagonal block of𝐓12⋆\{\\mathbf\{T\}\}\_\{12\}^\{\\star\}as asymmetric keeping theΔ​S2\\Delta S\_\{2\}symmetric blockI2I\_\{2\}from the first two rows and columns\. Now deform𝐓12⋆\{\\mathbf\{T\}\}\_\{12\}^\{\\star\}to𝐓12=\[I2,𝐓10\]\{\\mathbf\{T\}\}\_\{12\}=\[I\_\{2\},\{\\mathbf\{T\}\}\_\{10\}\]\. In practice, it is easier to taked=12d=12and path deform the type II critical point𝝃⁡\(12\)\{\\boldsymbol\{\\xi\}\}\(12\)with isotropyΔ​S11\\Delta S\_\{11\}to the critical point inM⁡\(12,12\)M\(12,12\)associated to the asymmetric target𝐓12\{\\mathbf\{T\}\}\_\{12\}and then further path deform this critical point to a critical point inM⁡\(12,12\)M\(12,12\)with isotropyΔ​S4\\Delta S\_\{4\}, target𝐓12⋆\{\\mathbf\{T\}\}\_\{12\}^\{\\star\}\. When computed, this critical point is𝝃~⋆​\(12\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}^\{\\star\}\(12\)and so the required path connection from𝝃~​\(10\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(10\)to𝝃~​\(12\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(12\)is the composition of the path from𝝃~​\(10\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(10\)to𝝃~⋆​\(12\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}^\{\\star\}\(12\)and the reversed path from𝝃~⋆​\(12\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}^\{\\star\}\(12\)to𝝃~​\(12\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(12\)\.

While there is noFPSfor the asymmetric example \(ddis fixed\), anFPScan be constructed if one extends tod\>12d\>12as described above\. Takingd=10128d=10^\{128\}, we find that the initial non\-zero terms of theFPSfor the derivatives are either multiples ofd−1d^\{\-1\}ord−12d^\{\-\\frac\{1\}\{2\}\}and, assuming101010The result still holds even if theFPSis in powers ofd−12nd^\{\-\\frac\{1\}\{2^\{n\}\}\},n\>1n\>1theFPSare power series ind−12d^\{\-\\frac\{1\}\{2\}\}, the initial non\-zero coefficients are:

D40=0\.2805800311544216D21=0\.001873922117631949D22=0\.03788347013872621D23=−0\.003893028296665770D24=0\.1570405665520425D25=0\.001836908105484236D26=0\.01117268780308763D27=−0\.005708669773701929D28=0\.002806305904052021D29=−0\.0004776786377640696D210=−0\.2088224377148879\\begin\{matrix\}D^\{0\}\_\{4\}&=&0\.2805800311544216&&&\\\\ D^\{1\}\_\{2\}&=&0\.001873922117631949&D^\{2\}\_\{2\}&=&0\.03788347013872621\\\\ D^\{3\}\_\{2\}&=&\-0\.003893028296665770&D^\{4\}\_\{2\}&=&0\.1570405665520425\\\\ D^\{5\}\_\{2\}&=&0\.001836908105484236&D^\{6\}\_\{2\}&=&0\.01117268780308763\\\\ D^\{7\}\_\{2\}&=&\-0\.005708669773701929&D^\{8\}\_\{2\}&=&0\.002806305904052021\\\\ D^\{9\}\_\{2\}&=&\-0\.0004776786377640696&D^\{10\}\_\{2\}&=&\-0\.2088224377148879\\end\{matrix\}

## 15\.Families of spurious minima in biased ReLU networks

In this section, the question of whether or not families of critical points and spurious minima that are seen in unbiased ReLU networks withdd\-inputs and target matrix𝐕=Id\{\{\\mathbf\{V\}\}\}=I\_\{d\}\[[4](https://arxiv.org/html/2609.30661#bib.bib4),[3](https://arxiv.org/html/2609.30661#bib.bib3),[5](https://arxiv.org/html/2609.30661#bib.bib5)\]extend naturally to biased ReLU networks\. This problem is examined in the simplest case of ReLU networks for which the number of neurons equals the number of inputs \(k=dk=d\)\. In particular, no investigation is made of the over\-specified case \(k\>dk\>d\) nor of the spectrum of the Hessian at critical points\. This is partly for reasons of length but primarily to keep the focus on the naturality of the extension \(and what that means\)\. The arguments given are based on high precision numerics and implicitly assume theFPSexist\. However, for the families presented below it is likely that more formal arguments, based on the implicit function, are relatively simple and similar to those given for unbiased networks\[[4](https://arxiv.org/html/2609.30661#bib.bib4)\],\[[5](https://arxiv.org/html/2609.30661#bib.bib5), Appendix A, remark 4\]\.

*It is assumed throughout that the target matrix has zero bias*—that is,γ¯j=0\\overline\{\\gamma\}\_\{j\}=0,j∈𝐝j\\in\{\\mathbf\{d\}\}\. Including examples where the target has bias would distract from the main question on the naturality of the extension\.

### 15\.1\.IsotropyΔ​Sd\\Delta S\_\{d\},d=kd=k: Type I in a network with bias

Two basic definitions inspired by high precision numerical results, are introduced in this subsection and play an important role in the remaining subsections\. Indeed, one definition motivates the formulation of an*ansatz*that leads to a formal proof for the existence ofFPSfor all the families of critical points in biased networks described below\.

We start with the simplest case: the family of spurious minima of type I \(or A\) described in\[[4](https://arxiv.org/html/2609.30661#bib.bib4)\]\. For this family there is exactly one rowtype00andq0=dq\_\{0\}=d\.

Critical points with isotropyΔ​Sd\\Delta S\_\{d\}in the type I family for the unbiased network \(resp\. biased network\) are denoted by𝝃⁡\(d\)=𝝃∈M​\(d,d\)Δ​Sd\{\\boldsymbol\{\\xi\}\}\(d\)=\{\\boldsymbol\{\\xi\}\}\\in M\(d,d\)^\{\\Delta S\_\{d\}\}\(resp\.𝝃~​\(d\)=𝝃~∈M​\(d,d\+1\)Δ​Sd\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)=\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\\in M\(d,d\+1\)^\{\\Delta S\_\{d\}\}\)—the previously introduced notation𝔠⁡\(d\)\\mathfrak\{c\}\(d\)\(resp\.𝔠⋆​\(d\)\\mathfrak\{c\}^\{\\star\}\(d\)\) is used when membership ofM⁡\(d,d\)M\(d,d\)\(resp\.M⁡\(d,d\+1\)M\(d,d\+1\)\) is emphasized rather than coordinatization of the fixed points space\.

Denote coordinates for𝝃\{\\boldsymbol\{\\xi\}\}\(resp\.𝝃~\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\) in the associated fixed point space by\(ξ0,ξ⋆0\)\(\\xi\_\{0\},\\xi\_\{\\star 0\}\)\(resp\.\(\(ξ0,ξ⋆0\),β¯\)\(\(\\xi\_\{0\},\\xi\_\{\\star 0\}\),\\overline\{\\beta\}\)\. Write𝝃⁡\(d\)\{\\boldsymbol\{\\xi\}\}\(d\),𝝃~​\(d\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)if the dependence onddneeds to be emphasized\. For either of these families, the index identifying rowtype is usually omitted since the isotropy isΔ​Sd\\Delta S\_\{d\}and there is only one rowtype\. For example, write‖𝐰i‖=τ0=τ\\\|\{\{\\mathbf\{w\}\}\}^\{i\}\\\|=\\tau\_\{0\}=\\tau,i∈𝐝i\\in\{\\mathbf\{d\}\}\. The bias for the network with parameter matrix𝐖\{\{\\mathbf\{W\}\}\}is denoted byβ¯\\overline\{\\beta\}and angles generally have no subscript*except*that we keep ‘0000’ terms such asλ00\\lambda\_\{00\}\(there is noλ01\\lambda\_\{01\}term since we assumeΔ​Sd\\Delta S\_\{d\}symmetry\)\.

Let𝐯0,…,𝐯d−1\{\{\\mathbf\{v\}\}\}^\{0\},\\ldots,\{\{\\mathbf\{v\}\}\}^\{d\-1\}define the standard orthonormal basis of\(ℝd\)⋆\(\{\\mathbb\{R\}\}^\{d\}\)^\{\\star\}\(identified with the dual spaceℝd\{\\mathbb\{R\}\}^\{d\}using the identification provided by the Euclidean inner product\)\. Define the three angles

Θ\\displaystyle\\Theta=\\displaystyle=cos−1⁡\(⟨𝐰i,𝐰j⟩/τ2\),i,j∈𝐝,i≠j\\displaystyle\\cos^\{\-1\}\\big\(\\langle\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{j\}\\rangle/\\tau^\{2\}\\big\),\\;i,j\\in\{\\mathbf\{d\}\},i\\neq jθ\\displaystyle\\theta=\\displaystyle=cos−1⁡\(⟨𝐰i,𝐯j⟩/τ\),i,j∈𝐝,i≠j\\displaystyle\\cos^\{\-1\}\\big\(\\langle\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{j\}\\rangle/\\tau\\big\),\\;i,j\\in\{\\mathbf\{d\}\},i\\neq jλ00\\displaystyle\\lambda\_\{00\}=\\displaystyle=cos−1⁡\(⟨𝐰i,𝐯i⟩/τ\),i∈𝐝\\displaystyle\\cos^\{\-1\}\\big\(\\langle\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{i\}\\rangle/\\tau\\big\),\\;i\\in\{\\mathbf\{d\}\}Following the conventions of section[12\.7](https://arxiv.org/html/2609.30661#S12.SS7), we have

Z=Z^\\displaystyle Z=\{\\widehat\{Z\}\}=\\displaystyle=β¯​\(−cot⁡\(Θ\)\+csc⁡\(Θ\)\),z=β¯​csc⁡\(θ\),z^=−β¯​cot⁡\(θ\)\\displaystyle\\overline\{\\beta\}\(\-\\cot\(\\Theta\)\+\\csc\(\\Theta\)\),\\quad z=\\overline\{\\beta\}\\csc\(\\theta\),\\quad\{\\widehat\{z\}\}=\-\\overline\{\\beta\}\\cot\(\\theta\)K\\displaystyle K=\\displaystyle=β¯2​csc⁡\(Θ\)​\(csc⁡\(Θ\)−cot⁡\(Θ\)\)k=12​\(β¯2​csc2⁡\(θ\)\)\\displaystyle\\overline\{\\beta\}^\{2\}\\csc\(\\Theta\)\(\\csc\(\\Theta\)\-\\cot\(\\Theta\)\)\\quad k=\\frac\{1\}\{2\}\\big\(\\overline\{\\beta\}^\{2\}\\csc^\{2\}\(\\theta\)\\big\)L\\displaystyle L=\\displaystyle=−π​erf​\(β¯2\),ℓ=−π2​erf​\(β¯2\)\\displaystyle\-\\pi\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\},\\quad\\ell=\-\\frac\{\\pi\}\{2\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}EX\\displaystyle E^\{X\}=\\displaystyle=−2​π​β¯​e−β¯22​erfc​\(Z2\)eX=−π2​β¯​erfc​\(z2\)\\displaystyle\-\\sqrt\{2\\pi\}\\overline\{\\beta\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{Z\}\{\\sqrt\{2\}\}\}\\right\)\}\\quad e^\{X\}=\-\\sqrt\{\\frac\{\\pi\}\{2\}\}\\overline\{\\beta\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{z\}\{\\sqrt\{2\}\}\}\\right\)\}Θ~\\displaystyle\{\\widetilde\{\\Theta\}\}=\\displaystyle=π2\+S⁡\(β¯,β¯,Θ\),θ~=π2\+S⁡\(β¯,0,θ\),λ~00=π2\+S⁡\(β¯,0,λ00\)\\displaystyle\\frac\{\\pi\}\{2\}\+S\(\\overline\{\\beta\},\\overline\{\\beta\},\\Theta\),\\quad\{\\widetilde\{\\theta\}\}=\\frac\{\\pi\}\{2\}\+S\(\\overline\{\\beta\},0,\\theta\),\\quad\{\\widetilde\{\\lambda\}\}\_\{00\}=\\frac\{\\pi\}\{2\}\+S\(\\overline\{\\beta\},0,\\lambda\_\{00\}\)z00\\displaystyle z\_\{00\}=\\displaystyle=β¯​csc⁡\(λ00\),z^00=−β¯​cot⁡\(λ00\)\\displaystyle\\overline\{\\beta\}\\csc\(\\lambda\_\{00\}\),\\quad\{\\widehat\{z\}\}\_\{00\}=\-\\overline\{\\beta\}\\cot\(\\lambda\_\{00\}\)k00\\displaystyle k\_\{00\}=\\displaystyle=12​β¯2​csc2⁡\(λ00\)\\displaystyle\\frac\{1\}\{2\}\\overline\{\\beta\}^\{2\}\\csc^\{2\}\(\\lambda\_\{00\}\)e00X\\displaystyle e^\{X\}\_\{00\}=\\displaystyle=−π2​β¯​erfc​\(z002\)ℓ00=−π2​erf​\(β¯2\)\\displaystyle\-\\sqrt\{\\frac\{\\pi\}\{2\}\}\\overline\{\\beta\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{z\_\{00\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\quad\\ \\ell\_\{00\}=\-\\frac\{\\pi\}\{2\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}Applying Proposition[12\.15](https://arxiv.org/html/2609.30661#S12.Thmlemma15)and the computations of section[12\.8](https://arxiv.org/html/2609.30661#S12.SS8)gives

Γ\\displaystyle\\Gamma=\\displaystyle=12​\[β¯2−\(1\+β¯2\)​erf​\(β¯2\)\]−12​π​β¯​e−β¯22\\displaystyle\\frac\{1\}\{2\}\\left\[\\overline\{\\beta\}^\{2\}\-\(1\+\\overline\{\\beta\}^\{2\}\)\{\\text\{\\rm erf\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\-\\frac\{1\}\{\\sqrt\{2\\pi\}\}\\overline\{\\beta\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\+12​π​\[\(d−1\)​\(τ​e−K​sin⁡\(Θ\)−e−k​sin⁡\(θ\)\)−e−k00​sin⁡\(λ00\)τ\]\\displaystyle\+\\frac\{1\}\{2\\pi\}\\left\[\\frac\{\(d\-1\)\\big\(\\tau e^\{\-K\}\\sin\(\\Theta\)\-e^\{\-k\}\\sin\(\\theta\)\\big\)\-e^\{\-k\_\{00\}\}\\sin\(\\lambda\_\{00\}\)\}\{\\tau\}\\right\]\+12​π​\[\(d−1\)​\(τ​EX−eX\)−e00Xτ\]\+β¯22​π​\(d−1\)​\[L\+\(π−Θ~\)\]\\displaystyle\+\\frac\{1\}\{2\\pi\}\\left\[\\frac\{\(d\-1\)\\big\(\\tau E^\{X\}\-e^\{X\}\\big\)\-e^\{X\}\_\{00\}\}\{\\tau\}\\right\]\+\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\\pi\}\(d\-1\)\\big\[L\+\(\\pi\-\{\\widetilde\{\\Theta\}\}\)\\big\]A0\\displaystyle A\_\{0\}=\\displaystyle=12​π​\(\(d−1\)​Θ~​ξ⋆0−λ~00\)A⋆0=12​π​\(Θ~​ξ0\+\(d−2\)​ξ⋆0​Θ~−θ~\)\\displaystyle\\frac\{1\}\{2\\pi\}\\big\(\(d\-1\)\{\\widetilde\{\\Theta\}\}\\xi\_\{\\star 0\}\-\{\\widetilde\{\\lambda\}\}\_\{00\}\\big\)\\quad A\_\{\\star 0\}=\\frac\{1\}\{2\\pi\}\\big\(\{\\widetilde\{\\Theta\}\}\\xi\_\{0\}\+\(d\-2\)\\xi\_\{\\star 0\}\{\\widetilde\{\\Theta\}\}\-\{\\widetilde\{\\theta\}\}\\big\)B0\\displaystyle B\_\{0\}=\\displaystyle=12​π​\(\(d−1\)​L​ξ⋆0−ℓ00\)B⋆0=12​π​\(L⁡\(ξ0\+\(d−2\)​ξ⋆0\)−ℓ00\)\\displaystyle\\frac\{1\}\{2\\pi\}\\big\(\(d\-1\)L\\xi\_\{\\star 0\}\-\\ell\_\{00\}\\big\)\\quad B\_\{\\star 0\}=\\frac\{1\}\{2\\pi\}\\big\(L\(\\xi\_\{0\}\+\(d\-2\)\\xi\_\{\\star 0\}\)\-\\ell\_\{00\}\\big\)Ω0\\displaystyle\\Omega\_\{0\}=\\displaystyle=12​\(ξ0\+\(d−1\)​ξ⋆0−1\)\\displaystyle\\frac\{1\}\{2\}\\big\(\\xi\_\{0\}\+\(d\-1\)\\xi\_\{\\star 0\}\-1\\big\)
Also needed is the derivative of the lossℒ\\mathcal\{L\}with respect to𝜷¯\\boldsymbol\{\\overline\{\\beta\}\}\.

###### Proposition 15\.1\.

\(Notation and assumptions as above; in particular,γ¯=0\\overline\{\\gamma\}=0\.\)

1d​∂ℒ∂β¯=τ22​\[β¯​erfc​\(β¯2\)−2π​e−β¯22\]\+\(d−1\)​τ2​β¯2​π​\[π​erfc​\(β¯2\)−Θ~\]\+τ2​2​π​\[erfc​\(β¯​csc⁡\(λ00\)2\)\+cos⁡\(λ00\)​e−β¯22​erfc​\(−β¯​cot⁡\(λ00\)2\)\]\+\(d−1\)​τ2​2​π​\[erfc​\(β¯​csc⁡\(θ\)2\)\+cos⁡\(θ\)​e−β¯22​erfc​\(−β¯​cot⁡\(θ\)2\)\]−\(d−1\)​τ22​2​π​\[e−β¯22​erfc​\(β¯​\(−cot⁡\(Θ\)\+csc⁡\(Θ\)\)2\)\]​\(1\+cos⁡\(Θ\)\)\\begin\{aligned\} \\hskip 14\.45377pt\\frac\{1\}\{d\}\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\}&=\\frac\{\\tau^\{2\}\}\{2\}\\left\[\\overline\{\\beta\}\\,\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\\sqrt\{\\frac\{2\}\{\\pi\}\}e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\\right\]\+\(d\-1\)\\frac\{\\tau^\{2\}\\overline\{\\beta\}\}\{2\\pi\}\\left\[\\pi\\,\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\}\{\\sqrt\{2\}\}\}\\right\)\}\-\{\\widetilde\{\\Theta\}\}\\right\]\\\\ &\\hskip\-43\.36243pt\+\\frac\{\\tau\}\{2\\sqrt\{2\\pi\}\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\\csc\(\\lambda\_\{00\}\)\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\cos\(\\lambda\_\{00\}\)e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\-\\overline\{\\beta\}\\cot\(\\lambda\_\{00\}\)\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ &\\hskip\-43\.36243pt\+\(d\-1\)\\frac\{\\tau\}\{2\\sqrt\{2\\pi\}\}\\left\[\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\cos\(\\theta\)e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\-\\overline\{\\beta\}\\cot\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\\\ &\\hskip\-43\.36243pt\-\(d\-1\)\\frac\{\\tau^\{2\}\}\{2\\sqrt\{2\\pi\}\}\\left\[e^\{\-\\frac\{\\overline\{\\beta\}^\{2\}\}\{2\}\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{\\overline\{\\beta\}\(\-\\cot\(\\Theta\)\+\\csc\(\\Theta\)\)\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\\big\(1\+\\cos\(\\Theta\)\\big\)\\end\{aligned\}

###### Proof\.

This follows from Proposition[12\.19](https://arxiv.org/html/2609.30661#S12.Thmlemma19)\. For a simple direct proof, it follows fromΔ​Sd\\Delta S\_\{d\}symmetry that

ℒ=d​f⁡\(𝐰0,𝐰0\)2\+d⁡\(d−1\)​f⁡\(𝐰0,𝐰1\)2−d​f​\(𝐰0,𝐯0\)−d⁡\(d−1\)​f​\(𝐰0,𝐯1\)\+K\\mathcal\{L\}=d\\frac\{f\(\{\{\\mathbf\{w\}\}\}^\{0\},\{\{\\mathbf\{w\}\}\}^\{0\}\)\}\{2\}\+d\(d\-1\)\\frac\{f\(\{\{\\mathbf\{w\}\}\}^\{0\},\{\{\\mathbf\{w\}\}\}^\{1\}\)\}\{2\}\-df\(\{\{\\mathbf\{w\}\}\}^\{0\},\{\{\\mathbf\{v\}\}\}^\{0\}\)\-d\(d\-1\)f\(\{\{\\mathbf\{w\}\}\}^\{0\},\{\{\\mathbf\{v\}\}\}^\{1\}\)\+Kwhere𝐰0,𝐰1\{\{\\mathbf\{w\}\}\}^\{0\},\{\{\\mathbf\{w\}\}\}^\{1\}are the first and second rows of𝐖\{\{\\mathbf\{W\}\}\}—any pair of distinct rows suffice—and the termKKdepends on𝐕\{\{\\mathbf\{V\}\}\}and is independent of𝐖\{\{\\mathbf\{W\}\}\}andβ¯\\overline\{\\beta\}\. Hence

1d∂ℒ∂β¯=∂∂β¯\[f⁡\(𝐰0,𝐰0\)2−f\(𝐰0,𝐯0\)\+\(d−1\)f⁡\(𝐰0,𝐰1\)2−\(d−1\)f\(𝐰0,𝐯1\)\)\]\\frac\{1\}\{d\}\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\}=\\frac\{\\partial\}\{\\partial\\overline\{\\beta\}\}\\left\[\\frac\{f\(\{\{\\mathbf\{w\}\}\}^\{0\},\{\{\\mathbf\{w\}\}\}^\{0\}\)\}\{2\}\-f\(\{\{\\mathbf\{w\}\}\}^\{0\},\{\{\\mathbf\{v\}\}\}^\{0\}\)\+\(d\-1\)\\frac\{f\(\{\{\\mathbf\{w\}\}\}^\{0\},\{\{\\mathbf\{w\}\}\}^\{1\}\)\}\{2\}\-\(d\-1\)f\(\{\{\\mathbf\{w\}\}\}^\{0\},\{\{\\mathbf\{v\}\}\}^\{1\}\)\\big\)\\right\]Using Lemma[8\.1](https://arxiv.org/html/2609.30661#S8.Thmlemma1), Proposition[8\.2](https://arxiv.org/html/2609.30661#S8.Thmlemma2)\(2\), and Example[7\.2](https://arxiv.org/html/2609.30661#S7.Thmlemma2)\([7\.33](https://arxiv.org/html/2609.30661#S7.E33)\), the partial derivative inβ¯\\overline\{\\beta\}is seen to be that of the statement of the proposition\. ∎

In future set

ℒ0=f⁡\(𝐰0,𝐰0\)2−f⁡\(𝐰0,𝐯0\)\+\(d−1\)​\[f⁡\(𝐰0,𝐰1\)2−f⁡\(𝐰0,𝐯1\)\]\\mathcal\{L\}^\{0\}=\\frac\{f\(\{\{\\mathbf\{w\}\}\}^\{0\},\{\{\\mathbf\{w\}\}\}^\{0\}\)\}\{2\}\-f\(\{\{\\mathbf\{w\}\}\}^\{0\},\{\{\\mathbf\{v\}\}\}^\{0\}\)\+\(d\-1\)\\left\[\\frac\{f\(\{\{\\mathbf\{w\}\}\}^\{0\},\{\{\\mathbf\{w\}\}\}^\{1\}\)\}\{2\}\-f\(\{\{\\mathbf\{w\}\}\}^\{0\},\{\{\\mathbf\{v\}\}\}^\{1\}\)\\right\]and regard theℒ\\mathcal\{L\}superscript as an index by rowtype \(see the remark following for the generalization to families with isotropyΔ⁡\(Sq​0×Sq​1\)\\Delta\(S\_\{q0\}\\times S\_\{q1\}\)\)\.

The equations for critical points with isotropyΔ​Sd\\Delta S\_\{d\}, target𝐕=Id\{\{\\mathbf\{V\}\}\}=I\_\{d\}andk=dk=d, may be written

Γ​ξ0−A0\+B0\+Ω0\\displaystyle\\Gamma\\xi\_\{0\}\-A\_\{0\}\+B\_\{0\}\+\\Omega\_\{0\}=\\displaystyle=0\\displaystyle 0Γ​ξ⋆0−A⋆0\+B⋆0\+Ω0\\displaystyle\\Gamma\\xi\_\{\\star 0\}\-A\_\{\\star 0\}\+B\_\{\\star 0\}\+\\Omega\_\{0\}=\\displaystyle=0\\displaystyle 0∂ℒ0∂β¯0\\displaystyle\\frac\{\\partial\\mathcal\{L\}^\{0\}\}\{\\partial\\overline\{\\beta\}\_\{0\}\}=\\displaystyle=0\\displaystyle 0
###### Remark 15\.2\(Needed only in subsequent sections\)\.

The definition ofℒ0\\mathcal\{L\}^\{0\}extends to problems with symmetry groupG𝐪ΔG^\{\\Delta\}\_\{\{\\mathbf\{q\}\}\}acting onM⁡\(k,d\)M\(k,d\), whereq0≥q1≥⋯qf−1≥1q\_\{0\}\\geq q\_\{1\}\\geq\\cdots q\_\{f\-1\}\\geq 1andk≥dk\\geq d\(see section[12\.3](https://arxiv.org/html/2609.30661#S12.SS3)\)\. Here there will beR=f\+mR=f\+mrowtypes, wherem=k−dm=k\-d\. Let𝐫𝐬⊂𝐟\{\{\\mathbf\{r\_\{s\}\}\}\}\\subset\{\\mathbf\{f\}\}denote the set ofqi\>1q\_\{i\}\>1so thatq0,…,qRs−1≥2q\_\{0\},\\ldots,q\_\{R\_\{s\}\-1\}\\geq 2and the remainingqiq\_\{i\}are all equal to one\. Recall the conventions on superscripts:𝐰i\{\{\\mathbf\{w\}\}\}^\{i\}will denote the*first*row of𝐖∈M⁡\(k,d\)\{\{\\mathbf\{W\}\}\}\\in M\(k,d\)which is of rowtypeiiand𝐰~i\\widetilde\{\{\{\\mathbf\{w\}\}\}\}^\{i\}will denote any row of𝐖\{\{\\mathbf\{W\}\}\}which is of rowtypeiibut is not𝐰i\{\{\\mathbf\{w\}\}\}^\{i\}\(that is, not the first row of rowtypeii\)\. Similar conventions hold for rows of𝐕\{\{\\mathbf\{V\}\}\}\.

Fori∈𝐫𝐬i\\in\{\{\\mathbf\{r\_\{s\}\}\}\}, define

ℒi=\(qi−1\)​\[f⁡\(𝐰i,𝐰~i\)2−f⁡\(𝐰i,𝐯~i\)\]\+\[f⁡\(𝐰i,𝐰i\)2−f⁡\(𝐰i,𝐯i\)\]\+∑j≠iqj\[f\(𝐰i,𝐰j\)−f\(𝐰i,𝐯j\)\]\\begin\{aligned\} \\mathcal\{L\}^\{i\}=&\(q\_\{i\}\-1\)\\left\[\\frac\{f\(\{\{\\mathbf\{w\}\}\}^\{i\},\\widetilde\{\{\{\\mathbf\{w\}\}\}\}^\{i\}\)\}\{2\}\-f\(\{\{\\mathbf\{w\}\}\}^\{i\},\\widetilde\{\{\{\\mathbf\{v\}\}\}\}^\{i\}\)\\right\]\+\\left\[\\frac\{f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{i\}\)\}\{2\}\-f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{i\}\)\\right\]\\\\ &\+\\sum\_\{j\\neq i\}q\_\{j\}\\left\[f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{j\}\)\-f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{j\}\)\\right\]\\end\{aligned\}Fori∈𝐟∖𝐫𝐬i\\in\{\\mathbf\{f\}\}\\smallsetminus\{\{\\mathbf\{r\_\{s\}\}\}\}, define

ℒi\\displaystyle\\mathcal\{L\}^\{i\}=\\displaystyle=\[f⁡\(𝐰i,𝐰i\)2−f⁡\(𝐰i,𝐯i\)\]\+∑j≠iqj​\[f⁡\(𝐰i,𝐰j\)−f⁡\(𝐰i,𝐯j\)\]\\displaystyle\\left\[\\frac\{f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{i\}\)\}\{2\}\-f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{i\}\)\\right\]\+\\sum\_\{j\\neq i\}q\_\{j\}\\left\[f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{j\}\)\-f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{j\}\)\\right\]Ifi∈𝐑∖𝐟i\\in\{\\mathbf\{R\}\}\\smallsetminus\{\\mathbf\{f\}\}, define

ℒi\\displaystyle\\mathcal\{L\}^\{i\}=\\displaystyle=f⁡\(𝐰i,𝐰i\)2\+∑j≠iqj​\[f⁡\(𝐰i,𝐰j\)−f⁡\(𝐰i,𝐯j\)\]\\displaystyle\\frac\{f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{i\}\)\}\{2\}\+\\sum\_\{j\\neq i\}q\_\{j\}\\left\[f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{w\}\}\}^\{j\}\)\-f\(\{\{\\mathbf\{w\}\}\}^\{i\},\{\{\\mathbf\{v\}\}\}^\{j\}\)\\right\]The critical point equations associated to theβ¯\\overline\{\\beta\}\-derivatives ofℒ\\mathcal\{L\}are then

∂ℒi∂β¯i\\displaystyle\\frac\{\\partial\\mathcal\{L\}^\{i\}\}\{\\partial\\overline\{\\beta\}\_\{i\}\}=\\displaystyle=0,i∈𝐑\\displaystyle 0,\\;i\\in\{\\mathbf\{R\}\}In each case,∂ℒi∂β¯i=1qi​∂ℒ∂β¯i\\frac\{\\partial\\mathcal\{L\}^\{i\}\}\{\\partial\\overline\{\\beta\}\_\{i\}\}=\\frac\{1\}\{q\_\{i\}\}\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\_\{i\}\}\.✠\\maltese

### 15\.2\.Numerics

As shown in Section[14\.2](https://arxiv.org/html/2609.30661#S14.SS2),∂ℒ∂β¯≠0\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\overline\{\\beta\}\}\\neq 0at a type I critical point of the bias free network\.

Starting with the biased network, with bias set to zero, and initial point\(𝝃⁡\(d\),0\)∈M​\(d,d\+1\)Δ​Sd≈ℝ3\(\{\\boldsymbol\{\\xi\}\}\(d\),0\)\\in M\(d,d\+1\)^\{\\Delta S\_\{d\}\}\\approx\{\\mathbb\{R\}\}^\{3\}, Newton’s method is applied to the critical point equations given above\. To this end,*MPFR*code is used, always with at least20482048\-bit precision \(higher for largedd\)\. Newton’s method converges rapidly for alld≥4d\\geq 4to a critical point𝝃~​\(d\)∈M​\(d,d\+1\)Δ​Sd≈ℝ3\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)\\in M\(d,d\+1\)^\{\\Delta S\_\{d\}\}\\approx\{\\mathbb\{R\}\}^\{3\}and is not a point defining the global minimum\. The smallest value ofddtested was44, the largest1051210^\{512\}\.

###### Remark 15\.3\.

Although we do not analyze the spectrum of the Hessian at𝝃~​\(d\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)here \(see below\), it is likely that the critical point defines a spurious minimum for large enoughdd\(most likelyd≥8d\\geq 8\)\. Indeed, for values of8≤d<308\\leq d<30, small random perturbation of the critical point inM⁡\(d,d\+1\)M\(d,d\+1\)\(notM​\(d,d\+1\)Δ​SdM\(d,d\+1\)^\{\\Delta S\_\{d\}\}\!\) always increase the loss\. This is a necessary condition for a non\-degenerate local minimum\. A more formal analysis of the Hessian spectrum can be made using the representation theoretic methods of\[[3](https://arxiv.org/html/2609.30661#bib.bib3),[5](https://arxiv.org/html/2609.30661#bib.bib5)\]: adding the single bias adds exactly one trivial representation and one standard representation to the isotypic decomposition of theΔ​Sd\\Delta S\_\{d\}\-action onM⁡\(d,d\)M\(d,d\)\(bias free parameter space\)\. The isotypic decomposition ofΔ​Sd\\Delta S\_\{d\}onM⁡\(d,d\+1\)M\(d,d\+1\), has 3 copies of the trivial representation \(the dimension of the fixed point space\), 4 copies of the standard representation and one copy each of the exterior square representation and the representation associated to the partition\(d−2,2\)\(d\-2,2\)\. The spectrum associated to the standard representations and the exterior square representation are of greatest interest\[[6](https://arxiv.org/html/2609.30661#bib.bib6),[5](https://arxiv.org/html/2609.30661#bib.bib5)\]\. Since the trivial factor is given by the fixed point space, it is unlikely \(see below\) that there is any change in the spectrum associated to this factor\. In any case, explicit computation of the 3 eigenvalues associated to the trivial representation is easily done using Newton’s method\. An example is given below for the type I family withΔ​Sd\\Delta S\_\{d\}symmetry\.✠\\maltese

#### The loss

In the unbiased network, the asymptotics of the loss\[[4](https://arxiv.org/html/2609.30661#bib.bib4), §8\.6\]are given by

ℒ⁡\(𝝃⁡\(d\)\)\\displaystyle\\mathcal\{L\}\(\{\\boldsymbol\{\\xi\}\}\(d\)\)=\\displaystyle=\(12−1π\)\+O⁡\(d−12\)\\displaystyle\\left\(\\frac\{1\}\{2\}\-\\frac\{1\}\{\\pi\}\\right\)\+O\(d^\{\-\{\\frac\{1\}\{2\}\}\}\)→\\displaystyle\{\\rightarrow\}\(12−1π\)≈0\.181690113816​…,as​d→∞\\displaystyle\\left\(\\frac\{1\}\{2\}\-\\frac\{1\}\{\\pi\}\\right\)\\approx 0\.181690113816\\ldots,\\;\\text\{as \}d\{\\rightarrow\}\\inftyLettingd→∞d\{\\rightarrow\}\\inftyin the biased network, numerical investigation up tod=10128d=10^\{128\}strongly suggests that

ℒ​\(𝝃~​\(d\)\)\\displaystyle\\mathcal\{L\}\(\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)\)→\\displaystyle\{\\rightarrow\}\(12−1π\)\+O⁡\(d−12\)\\displaystyle\\left\(\\frac\{1\}\{2\}\-\\frac\{1\}\{\\pi\}\\right\)\+O\(d^\{\-\{\\frac\{1\}\{2\}\}\}\)
Moreover, even though the behaviour asd→∞d\{\\rightarrow\}\\inftyappears the same in the biased and unbiased network, the loss at𝝃~​\(d\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)is*always strictly less*than the loss at𝝃⁡\(d\)\{\\boldsymbol\{\\xi\}\}\(d\)—see the tables below and note that if this were not always true, then either the computations would be in error or any connection between𝝃⁡\(d\)\{\\boldsymbol\{\\xi\}\}\(d\)and𝝃~​\(d\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)would likely involve complex landscape geometry\.

Define

Δu​b​\(d\)=ℒ⁡\(𝝃⁡\(d\)\)−ℒ⁡\(𝝃~​\(d\)\)\.\\Delta\_\{ub\}\(d\)=\\mathcal\{L\}\(\{\\boldsymbol\{\\xi\}\}\(d\)\)\-\\mathcal\{L\}\(\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)\)\.
Table 16\.Computed values of the loss for type I critical point of isotropyΔ​Sd\\Delta S\_\{d\}\. Loss is given to 5 or more significant figures and shows how the addition of bias to the student network always reduces the loss\. The third column gives the decrease of the loss when the network is biased\. This is always positive\. Included are the values obtained whend=4d=4\. However,𝝃⁡\(d\)\{\\boldsymbol\{\\xi\}\}\(d\)is not a spurious minima ford≤7d\\leq 7; the same is likely true for𝝃~​\(4\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(4\)\.The data presented above strongly suggests that the critical point families described not only haveFPSrepresentations but that lowest order terms are the same for unbiased and biased networks\. This is perhaps the strongest argument that𝝃~​\(d\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)is a natural extension of the family𝝃⁡\(d\)\{\\boldsymbol\{\\xi\}\}\(d\)\. There is more\.

###### Example 15\.4\.

Takingd=10128d=10^\{128\}, a high precision computation gives

Δu​b​\(10128\)=0\.044992875434563​…×10−128\>0\\Delta\_\{ub\}\(10^\{128\}\)=0\.044992875434563\\ldots\\times 10^\{\-128\}\>0indicating that the first*two*terms in theFPSexpansions forℒ⁡\(𝝃⁡\(d\)\)\\mathcal\{L\}\(\{\\boldsymbol\{\\xi\}\}\(d\)\)andℒ​\(𝝃~​\(d\)\)\\mathcal\{L\}\(\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)\)are equal\. In particular, the coefficients ofd−12d^\{\-\\frac\{1\}\{2\}\}are*equal and non\-zero*\(easily verified analytically\[[4](https://arxiv.org/html/2609.30661#bib.bib4)\]for the unbiased network\)\. That is,

Δu​b​\(d\)=O⁡\(d−1\)\\Delta\_\{ub\}\(d\)=O\(d^\{\-1\}\)and notO⁡\(d−12\)O\(d^\{\-\\frac\{1\}\{2\}\}\)as one might have expected\.※

Granted this example, it is natural to ask about the coefficients of initial terms in theFPSof coordinates of𝝃\{\\boldsymbol\{\\xi\}\}and𝝃~\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\. All the examples considered here haveFPSind−12d^\{\-\\frac\{1\}\{2\}\}\(this may not be so if there is over\-specification\[[5](https://arxiv.org/html/2609.30661#bib.bib5)\]\)\.

###### Definition 15\.5\.

Let∑j≥pi≥0αji​d−j2\\sum\_\{j\\geq p\_\{i\}\\geq 0\}\\alpha\_\{j\}^\{i\}d^\{\-\\frac\{j\}\{2\}\},i∈𝟐i\\in\{\\mathbf\{2\}\}, be a pair of fractional power series \(scalar valued\) withαpii≠0\\alpha^\{i\}\_\{p\_\{i\}\}\\neq 0,i∈𝟐i\\in\{\\mathbf\{2\}\}\. The series are said to be*nn\-matched*if

1. \(1\)p0=p1=defpp\_\{0\}=p\_\{1\}\\stackrel\{\{\\scriptstyle\\mathrm\{def\}\}\}\{\{=\}\}p\.
2. \(2\)αj0=αj1\\alpha^\{0\}\_\{j\}=\\alpha^\{1\}\_\{j\},j=p,…,p\+n−1j=p,\\ldots,p\+n\-1\.

Two vector valuedFPSarenn\-matched if they have the same range and all pairs of corresponding coordinates arennmatched \(the valuesp⁡\(i\)p\(i\)may and do vary with the coordinate chosen\)\. It is implicit thatnnis the largest integer for which the statement is true\.

More generally, given twoℝQ\{\\mathbb\{R\}\}^\{Q\}valuedFPSsuch that theiith coordinates arenin\_\{i\}matched withni≥0n\_\{i\}\\geq 0, for alli∈𝐐i\\in\{\\mathbf\{Q\}\}, theFPSare*𝐧\{\\mathbf\{n\}\}\-matched*with matching vector𝐧=\(n0,…,nQ−1\)\{\\mathbf\{n\}\}=\(n\_\{0\},\\ldots,n\_\{Q\-1\}\)\.

###### Examples 15\.6\.

\(1\) TheFPSforℒ⁡\(𝝃\)\\mathcal\{L\}\(\{\\boldsymbol\{\\xi\}\}\)andℒ⁡\(𝝃~\)\\mathcal\{L\}\(\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\)are 2\-matched but not 3\-matched \. \(2\) Making use of computations of𝝃\{\\boldsymbol\{\\xi\}\}and𝝃~\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}whend=10128d=10^\{128\}and ignoring theβ¯\\overline\{\\beta\}\-coordinate, we have

ξ0−ξ0~\\displaystyle\\xi\_\{0\}\-\\widetilde\{\\xi\_\{0\}\}=\\displaystyle=c0​10−129\\displaystyle c\_\{0\}10^\{\-129\}ξ⋆0−ξ~⋆0\\displaystyle\\xi\_\{\\star 0\}\-\\widetilde\{\\xi\}\_\{\\star 0\}=\\displaystyle=c⋆0​10−256\\displaystyle c\_\{\\star 0\}10^\{\-256\}wherec0≈7\.78c\_\{0\}\\approx 7\.78andc⋆0≈1\.27c\_\{\\star 0\}\\approx 1\.27\. Hence theFPSare22\-matched \(or𝟐=\(2,2\)\{\\mathbf\{2\}\}=\(2,2\)\-matched\)111111We have ignoredβ¯​\(d\)\\overline\{\\beta\}\(d\)but sinceβ¯​\(d\)=c​d−1\\overline\{\\beta\}\(d\)=cd^\{\-1\},c≈−6\.21c\\approx\-6\.21, this term is “22\-matched” to𝝃\{\\boldsymbol\{\\xi\}\}, withp0=p1=0p\_\{0\}=p\_\{1\}=0, if we define theβ¯\\overline\{\\beta\}\-component to be identically zero\.\. It follows from section[13\.2](https://arxiv.org/html/2609.30661#S13.SS2)that the coefficient ofd−12d^\{\-\\frac\{1\}\{2\}\}in theFPSforξ~0\\widetilde\{\\xi\}\_\{0\}and the coefficient ofd−32d^\{\-\\frac\{3\}\{2\}\}in theFPSofξ~⋆0\\widetilde\{\\xi\}\_\{\\star 0\}are both zero\.

The existence ofFPSfor this unbiased network is proved using elementary implicit function arguments\[[4](https://arxiv.org/html/2609.30661#bib.bib4)\]\. The numerics for𝝃~\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}strongly suggest the ansatz that the pair𝝃,𝝃~\{\\boldsymbol\{\\xi\}\},\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}is22\-matched\. If so, a significant number of the initial coefficients of theFPSfor𝝃~\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}are known exactly without having to do any computations on the coefficients of the biased network \(see results in section[13\.2](https://arxiv.org/html/2609.30661#S13.SS2)\)\. As a result, the expectation is that the implicit function theorem method can be extended to allow for dependence on a parameter—in this caseβ¯\\overline\{\\beta\}\. Along the line joining00toβ¯​\(d\)\\overline\{\\beta\}\(d\), the correspondingFPSshould always be 2\-matched to theFPSfor𝝃\{\\boldsymbol\{\\xi\}\}\.※

###### Remarks 15\.7\.

\(1\) Based on high precision numerics, the initial terms of theFPSexpansion forβ¯​\(d\)\\overline\{\\beta\}\(d\)is

β¯​\(d\)=−0\.62072950119515416243932018851173​…​d−1\+O⁡\(d−32\)\\overline\{\\beta\}\(d\)=\-0\.62072950119515416243932018851173\\ldots d^\{\-1\}\+O\(d^\{\-\\frac\{3\}\{2\}\}\)where the first 32 significant figures of the first non\-zero coefficient are shown and the coefficient ofd−32d^\{\-\\frac\{3\}\{2\}\}is strictly negative\. \(2\) For both biased and unbiased networks, two of the eigenvalues are of the formc0\+c1​d\+O⁡\(d−12\)c\_\{0\}\+c\_\{1\}d\+O\(d^\{\-\\frac\{1\}\{2\}\}\)where in one casec1=1/4c\_\{1\}=1/4for both biased and unbiased networks and in the other casec1\>0c\_\{1\}\>0but the value is different from that in the unbiased network\. \(3\) Ifd=4d=4, the three eigenvalues are0\.8811246​…,0\.0242543​…0\.8811246\\ldots,0\.0242543\\ldotsand0\.0661600​…0\.0661600\\ldots\. All are strictly positive \(the case for alld≥4d\\geq 4\)\.✠\\maltese

Not only are these strong results unexpected \(to the author at least\) but they suggest that explicit computation of the initial terms of theFPSalong the lines described in\[[4](https://arxiv.org/html/2609.30661#bib.bib4), §8\]may not be as challenging as it seems at first sight\. In particular, since the low order terms are the same, the implicit function theorem based method described in\[[4](https://arxiv.org/html/2609.30661#bib.bib4), §8\]should extend if we regardβ¯\\overline\{\\beta\}as a parameter\. Alternatively, using estimates from sections 4 and 5, there is the possibility of showing that a general proof of existence of anFPSin the biased case can be done within a cuspidal domain on which the loss is subanalytic and so existence can be deduced from the Curve Selection Lemma\[[7](https://arxiv.org/html/2609.30661#bib.bib7),[14](https://arxiv.org/html/2609.30661#bib.bib14)\]\.

### 15\.3\.Path based analysis

One criticism of the arguments presented above might be that we start with a critical point for the bias free network and then apply Newton to the biased network\. It is possible that on the first iteration of Newton there is convergence to a nearby critical point which is a saddle point for the loss\. Ideally one would like a gradient descent path from𝝃⁡\(d\)\{\\boldsymbol\{\\xi\}\}\(d\)to𝝃~​\(d\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)along which the loss is strictly monotone decreasing\. However, once𝝃~​\(d\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)has been computed, the variation in the loss along a straight line path from𝝃~​\(d\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)to𝝃⁡\(d\)\{\\boldsymbol\{\\xi\}\}\(d\)can be analyzed\. View the equations

\(15\.54\)Γ​ξ0−A0\+B0\+Ω0\\displaystyle\\Gamma\\xi\_\{0\}\-A\_\{0\}\+B\_\{0\}\+\\Omega\_\{0\}=\\displaystyle=0\\displaystyle 0\(15\.55\)Γ​ξ⋆0−A⋆0\+B⋆0\+Ω0\\displaystyle\\Gamma\\xi\_\{\\star 0\}\-A\_\{\\star 0\}\+B\_\{\\star 0\}\+\\Omega\_\{0\}=\\displaystyle=0\\displaystyle 0as equations for an ‘unbiased network’ depending on the parameterβ~∈\[0,β¯\]\\widetilde\{\\beta\}\\in\[0,\\overline\{\\beta\}\]\. Now𝝃⁡\(d\)=\(ξ0,ξ⋆0,0\)\{\\boldsymbol\{\\xi\}\}\(d\)=\(\\xi\_\{0\},\\xi\_\{\\star 0\},0\),𝝃~​\(d\)=\(ξ~0,ξ~0,⋆,β~\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)=\(\\widetilde\{\\xi\}\_\{0\},\\widetilde\{\\xi\}\_\{0,\\star\},\\widetilde\{\\beta\}\)and𝝃⁡\(d\)\{\\boldsymbol\{\\xi\}\}\(d\)\(resp\.\(ξ~0,ξ~0,⋆\)\(\\widetilde\{\\xi\}\_\{0\},\\widetilde\{\\xi\}\_\{0,\\star\}\)\) are solutions of these equations when the parameter is zero \(resp\.β~\\widetilde\{\\beta\}\)\. Test for the path by dividing\[0,β¯\]\[0,\\overline\{\\beta\}\]intoNNequal increments\. Setδ=β¯/N\\delta=\\overline\{\\beta\}/Nand use Newton to move fromβ¯⋆\\overline\{\\beta\}^\{\\star\}back to zero inNNsteps of lengthδ\\delta\. Examine the output at each step, especially for the initial iterations of Newton, to detect any anomalies that are suggestive of a bifurcation, change of stability or ‘jump’ in the critical point\. We have done this for several values ofdd\. In all cases convergence was to the original critical point in the bias free network\. If𝝃~\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}is a non\-degenerate spurious minimum, then the loss should be strictly monotone increasing near𝝃~\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(it is\) but possibly decreasing near𝝃\{\\boldsymbol\{\\xi\}\}\(for this example, it is not\)\. Of course, there is no reason why a curve of steepest descent linking𝝃\{\\boldsymbol\{\\xi\}\}to𝝃~\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}should be a straight line\. However, here there is only one rowtype, and it is likely that there is a unique trajectory joining\(𝝃⁡\(d\),0\)\(\{\\boldsymbol\{\\xi\}\}\(d\),0\)to𝝃~​\(d\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)along which the loss is strictly decreasing\.

### 15\.4\.IsotropyΔ⁡\(Sd−1×S1\)\\Delta\(S\_\{d\-1\}\\times S\_\{1\}\),q1=1q\_\{1\}=1,d=kd=k, Types I and II

#### Type I family of spurious with isotropyΔ⁡\(Sd−1×S1\)=defΔ​Sd−1\\Delta\(S\_\{d\-1\}\\times S\_\{1\}\)\\stackrel\{\{\\scriptstyle\\mathrm\{def\}\}\}\{\{=\}\}\\Delta S\_\{d\-1\}

Let𝝃⁡\(d\)\{\\boldsymbol\{\\xi\}\}\(d\)denote the type I family of critical points inM​\(d,d\)Δ​Sd−1M\(d,d\)^\{\\Delta S\_\{d\-1\}\}which give spurious minima ford≥8d\\geq 8\. Using Newton’s method, a corresponding family𝝃~​\(d\)∈M​\(d,d\+1\)Δ​Sd−1\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)\\in M\(d,d\+1\)^\{\\Delta S\_\{d\-1\}\}is constructed for the biased network\.

Recall thatΔu​b​\(d\)=ℒ⁡\(𝝃⁡\(d\)\)−ℒ⁡\(𝝃~​\(d\)\)\\Delta\_\{ub\}\(d\)=\\mathcal\{L\}\(\{\\boldsymbol\{\\xi\}\}\(d\)\)\-\\mathcal\{L\}\(\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)\)\.

Table 17\.Computed values of the loss for type I critical point of isotropyΔ​Sd−1\\Delta S\_\{d\-1\}\. Loss is given to 5 or more significant figures and shows how the addition of bias to the student network always reduces the loss\. The third column gives the decrease of the loss when the network is biased\. This is always positive\. The table entries are close to the corresponding values for the type I, isotropyΔ​Sd\\Delta S\_\{d\}example of the preceding section\.###### Example 15\.8\.

Takingd=10128d=10^\{128\}, a high precision computation gives

Δu​b​\(10128\)=0\.044992875434563​…×10−128\>0\\Delta\_\{ub\}\(10^\{128\}\)=0\.044992875434563\\ldots\\times 10^\{\-128\}\>0indicating that the first two terms in the FPS expansions ofℒ⁡\(𝝃⁡\(d\)\)\\mathcal\{L\}\(\{\\boldsymbol\{\\xi\}\}\(d\)\)andℒ​\(𝝃~​\(d\)\)\\mathcal\{L\}\(\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)\)are equal\. A more detailed analysis indicates that if the subscriptsAAandIIare used to distinguish the type A and type I network, then

\|Δu​bI​\(d\)−Δu​bA​\(d\)\|=c​d−2\+O⁡\(d−52\),\|\\Delta^\{I\}\_\{ub\}\(d\)\-\\Delta^\{A\}\_\{ub\}\(d\)\|=cd^\{\-2\}\+O\(d^\{\-\\frac\{5\}\{2\}\}\),wherec≈0\.0242080c\\approx 0\.0242080\.※

The families𝝃⁡\(d\)\{\\boldsymbol\{\\xi\}\}\(d\)and𝝃~​\(d\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)are\(2,0,2,2,2\)\(2,0,2,2,2\)\-matched\. The00\-matched term corresponds toξ10​\(d\)\\xi^\{0\}\_\{1\}\(d\)andξ~10​\(d\)\\widetilde\{\\xi\}^\{0\}\_\{1\}\(d\)which both have initial non\-zero term in theirFPSa multiple ofd−2d^\{\-2\}\. Hence all corresponding termsc​d−p2cd^\{\-\\frac\{p\}\{2\}\}in every component of the coordinates are equal provided thatp<4p<4\.

###### Remark 15\.9\.

Bothβ¯\\overline\{\\beta\}derivatives of the loss at\(𝝃,0\)\(\{\\boldsymbol\{\\xi\}\},0\)are strictly positive so the expectation is that both values ofβ¯\\overline\{\\beta\}at the critical point𝝃~\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}should be strictly negative\. This turns out to be true for alld≥10d\\geq 10\. The initial terms of theFPSforβ¯​\(d\)\\overline\{\\beta\}\(d\)are given by

β¯0​\(d\)\\displaystyle\\overline\{\\beta\}\_\{0\}\(d\)=\\displaystyle=−0\.62072950119515416243932018851173​…​d−1\+O⁡\(d−32\)\\displaystyle\-0\.62072950119515416243932018851173\\dots d^\{\-1\}\+O\(d^\{\-\\frac\{3\}\{2\}\}\)β¯1​\(d\)\\displaystyle\\overline\{\\beta\}\_\{1\}\(d\)=\\displaystyle=−1\.2006004478530293576268915492951​…​d−1\+O⁡\(d−32\)\\displaystyle\-1\.2006004478530293576268915492951\\ldots d^\{\-1\}\+O\(d^\{\-\\frac\{3\}\{2\}\}\)✠\\maltese

*Type II family of spurious with isotropyΔ​Sd−1\\Delta S\_\{d\-1\}*

Table 18\.Computed values of the loss for type II critical point of isotropyΔ​Sd−1\\Delta S\_\{d\-1\}\. Loss is given to 5 or more significant figures and shows how the addition of bias to the student network always reduces the loss\. The third column gives the decrease of the loss when the network is biased\. This is always positive\.###### Examples 15\.10\.

\(1\) Takingd=10128d=10^\{128\}, a high precision computation gives

Δu​b​\(d\)=c​d−2\+O⁡\(d−52\),\\Delta\_\{ub\}\(d\)=cd^\{\-2\}\+O\(d^\{\-\\frac\{5\}\{2\}\}\),wherec≈0\.619636515c\\approx 0\.619636515and it is assumed that𝝃,𝝃~\{\\boldsymbol\{\\xi\}\},\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}have FPS\. \(2\) The initial terms of anFPSexpansion forβ¯i​\(d\)\\overline\{\\beta\}\_\{i\}\(d\),i∈𝟐i\\in\{\\mathbf\{2\}\}are

β¯0​\(d\)\\displaystyle\\overline\{\\beta\}\_\{0\}\(d\)=\\displaystyle=−3\.9147584367443569485868567728717​…​d−2\+O⁡\(d−52\)\\displaystyle\-3\.9147584367443569485868567728717\\dots d^\{\-2\}\+O\(d^\{\-\\frac\{5\}\{2\}\}\)β¯1​\(d\)\\displaystyle\\overline\{\\beta\}\_\{1\}\(d\)=\\displaystyle=2\.6116672965535862283319971186917​…​d−1\+O⁡\(d−32\)\\displaystyle 2\.6116672965535862283319971186917\\ldots d^\{\-1\}\+O\(d^\{\-\\frac\{3\}\{2\}\}\)Note that, unlike type I, the initial terms are of opposite sign\. \(3\) The families𝝃⁡\(d\)\{\\boldsymbol\{\\xi\}\}\(d\)and𝝃~​\(d\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)are\(4,2,2,4,2\)\(4,2,2,4,2\)\-matched\. The44\-matched terms correspond to the “diagonal” coordinatesξ00,ξ~00\\xi^\{0\}\_\{0\},\\widetilde\{\\xi\}^\{0\}\_\{0\}andξ11,ξ~11\\xi^\{1\}\_\{1\},\\widetilde\{\\xi\}^\{1\}\_\{1\}and so in the correspondingFPSterms match up to the coefficients ofd−32d^\{\-\\frac\{3\}\{2\}\}but notd−2d^\{\-2\}\. The same is true forξ10​\(d\),ξ~10​\(d\)\\xi^\{0\}\_\{1\}\(d\),\\widetilde\{\\xi\}^\{0\}\_\{1\}\(d\)andξ01​\(d\),ξ~01​\(d\)\\xi^\{1\}\_\{0\}\(d\),\\widetilde\{\\xi\}^\{1\}\_\{0\}\(d\)On the other hand, the familiesξ⋆00​\(d\)\\xi^\{0\}\_\{\\star 0\}\(d\)andξ~⋆00​\(d\)\\widetilde\{\\xi\}^\{0\}\_\{\\star 0\}\(d\)agree up to and including terms ind−2d^\{\-2\}\.※

### 15\.5\.IsotropyΔ⁡\(Sd−2×S2\)\\Delta\(S\_\{d\-2\}\\times S\_\{2\}\),q1=2q\_\{1\}=2,d=kd=k, Types I and II

#### Type I family of spurious minima with isotropyΔ⁡\(Sd−2×S2\)\\Delta\(S\_\{d\-2\}\\times S\_\{2\}\)

Let𝝃⁡\(d\)\{\\boldsymbol\{\\xi\}\}\(d\)denote the type I family of critical points inM​\(d,d\)Δ⁡\(Sd−2×S2\)M\(d,d\)^\{\\Delta\(S\_\{d\-2\}\\times S\_\{2\}\)\}which give spurious minima for sufficiently largedd\. Using Newton’s method, a corresponding family𝝃~​\(d\)∈M​\(d,d\+1\)Δ⁡\(Sd−2×S2\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)\\in M\(d,d\+1\)^\{\\Delta\(S\_\{d\-2\}\\times S\_\{2\}\)\}is constructed for the biased network\.

Table 19\.Computed values of the loss for type I critical point of isotropyΔ⁡\(Sd−2×S2\)\\Delta\(S\_\{d\-2\}\\times S\_\{2\}\)\. Loss is given to 5 or more significant figures and shows how the addition of bias to the student network always reduces the loss\. The third column gives the decrease of the loss when the network is biased\.###### Examples 15\.11\.

\(1\) Takingd=10128d=10^\{128\}, a high precision computation gives

Δu​b​\(10128\)=0\.044992875434563089568371515024571​…×10−128\\Delta\_\{ub\}\(10^\{128\}\)=0\.044992875434563089568371515024571\\ldots\\times 10^\{\-128\}indicating that the first two terms in theFPSexpansions ofℒ⁡\(𝝃⁡\(d\)\)\\mathcal\{L\}\(\{\\boldsymbol\{\\xi\}\}\(d\)\)andℒ​\(𝝃~​\(d\)\)\\mathcal\{L\}\(\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)\)are equal\. \(2\) The initial terms of anFPSexpansion forβ¯i​\(d\)\\overline\{\\beta\}\_\{i\}\(d\),i∈𝟐i\\in\{\\mathbf\{2\}\}are

β¯0​\(d\)\\displaystyle\\overline\{\\beta\}\_\{0\}\(d\)=\\displaystyle=−0\.620729501195​…​d−1\+O⁡\(d−32\)\\displaystyle\-0\.620729501195\\ldots d^\{\-1\}\+O\(d^\{\-\\frac\{3\}\{2\}\}\)β¯1​\(d\)\\displaystyle\\overline\{\\beta\}\_\{1\}\(d\)=\\displaystyle=−1\.2006004478530​…​d−1\+O⁡\(d−32\)\\displaystyle\-1\.2006004478530\\ldots d^\{\-1\}\+O\(d^\{\-\\frac\{3\}\{2\}\}\)\(3\) As for type I,q1=1q\_\{1\}=1, both initial terms are negative and the families𝝃⁡\(d\)\{\\boldsymbol\{\\xi\}\}\(d\)and𝝃~​\(d\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)are\(2,0,2,2,2,2\)\(2,0,2,2,2,2\)\-matched\.※

#### Type II family of spurious minima with isotropyΔ⁡\(Sd−2×S2\)\\Delta\(S\_\{d\-2\}\\times S\_\{2\}\)

Let𝝃⁡\(d\)\{\\boldsymbol\{\\xi\}\}\(d\)denote the type II family of critical points inM​\(d,d\)Δ⁡\(Sd−2×S2\)M\(d,d\)^\{\\Delta\(S\_\{d\-2\}\\times S\_\{2\}\)\}which give spurious minima ford≥8d\\geq 8\. Using Newton’s method, a corresponding family𝝃~​\(d\)∈M​\(d,d\+1\)Δ⁡\(Sd−2×S2\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)\\in M\(d,d\+1\)^\{\\Delta\(S\_\{d\-2\}\\times S\_\{2\}\)\}is constructed for the biased network\.

Table 20\.Computed values of the loss for type II critical point of isotropyΔ⁡\(Sd−2×S2\)\\Delta\(S\_\{d\-2\}\\times S\_\{2\}\)\. Loss is given to 5 or more significant figures and shows how the addition of bias to the student network always reduces the loss\. The third column gives the decrease of the loss when the network is biased\.###### Examples 15\.12\.

\(1\) Takingd=10128d=10^\{128\}, a high precision computation gives

Δu​b​\(10128\)=0\.6196365153963871523492​…×10−256\\Delta\_\{ub\}\(10^\{128\}\)=0\.6196365153963871523492\\ldots\\times 10^\{\-256\}indicating that the first two terms in the FPS expansions ofℒ⁡\(𝝃⁡\(d\)\)\\mathcal\{L\}\(\{\\boldsymbol\{\\xi\}\}\(d\)\)andℒ​\(𝝃~​\(d\)\)\\mathcal\{L\}\(\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)\)are equal\. \(2\) The initial terms of anFPSexpansion forβ¯i​\(d\)\\overline\{\\beta\}\_\{i\}\(d\),i∈𝟐i\\in\{\\mathbf\{2\}\}are

β¯0​\(d\)\\displaystyle\\overline\{\\beta\}\_\{0\}\(d\)=\\displaystyle=−7\.829516873488713897737135457431​…​d−2\+O⁡\(d−52\)\\displaystyle\-7\.829516873488713897737135457431\\ldots d^\{\-2\}\+O\(d^\{\-\\frac\{5\}\{2\}\}\)β¯1​\(d\)\\displaystyle\\overline\{\\beta\}\_\{1\}\(d\)=\\displaystyle=2\.611667296553586228331997118692​…​d−1\+O⁡\(d−32\)\\displaystyle 2\.611667296553586228331997118692\\ldots d^\{\-1\}\+O\(d^\{\-\\frac\{3\}\{2\}\}\)and have opposite sign\. \(3\) Similar to the type II family of critical points with isotropyΔ​Sd−1\\Delta S\_\{d\-1\}, the families𝝃​\(d\),𝝃~​\(d\)\{\\boldsymbol\{\\xi\}\}\(d\),\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)are\(4,2,2,4,2,2\)\(4,2,2,4,2,2\)\-matched\.※

### 15\.6\.IsotropyΔ⁡\(Sd−3×S3\)\\Delta\(S\_\{d\-3\}\\times S\_\{3\}\),q1=3q\_\{1\}=3,d=kd=k, type II

The final family of examples has isotropyΔ⁡\(Sd−3×S3\)\\Delta\(S\_\{d\-3\}\\times S\_\{3\}\)\. Only the type II case is considered and this follows the pattern described above forq1=1,2q\_\{1\}=1,2\. Similar remarks hold for type I whenq1=3q\_\{1\}=3\.

The ‘seed’ for the type II family is the critical point withd=10d=10and isotropyΔ⁡\(S7×S3\)\\Delta\(S\_\{7\}\\times S\_\{3\}\)\. This is obtained by evolving along a path in the fixed point space in the unbiased network from the type II point with isotropyΔ⁡\(S8×S2\)\\Delta\(S\_\{8\}\\times S\_\{2\}\)to the type II critical point with isotropyΔ⁡\(S7×S3\)\\Delta\(S\_\{7\}\\times S\_\{3\}\)\. That is, if we write the isotropy asΔ⁡\(Sq0×Sq​1\)\\Delta\(S\_\{q\_\{0\}\}\\times S\_\{q1\}\), we vary\(q0,q1\)\(q\_\{0\},q\_\{1\}\)linearly along a line segment so thatq0\+q1=d=10q\_\{0\}\+q\_\{1\}=d=10,q0q\_\{0\}decreases from88to77, andq1q\_\{1\}increases from22to33\. This path lies in the 6\-dimensional fixed point spaceM​\(10,10\)Δ⁡\(S7×S3\)M\(10,10\)^\{\\Delta\(S\_\{7\}\\times S\_\{3\}\)\}\. Ifq1=1q\_\{1\}=1, the fixed point space is 5\-dimensional, the critical point equations are different and we cannot path evolve fromq1=1q\_\{1\}=1toq1=2q\_\{1\}=2\. The same method can be used for the biased network and this provides a simple consistency check \(is the result of incrementing first and then evolving the same as evolving first then incrementing?\)\.

Table 21\.Computed values of the loss for type II critical point of isotropyΔ⁡\(Sd−3×S3\)\\Delta\(S\_\{d\-3\}\\times S\_\{3\}\)\. Loss is given to 5 or more significant figures and shows how the addition of bias to the student network always reduces the loss\. The third column gives the decrease of the loss when the network is biased\.###### Examples 15\.13\.

\(1\) Using the result ford=109d=10^\{9\}and assuming the existence of the FPS,

Δu​b​\(109\)=0\.9295​…×10−18\+O⁡\(d−32\)\\Delta\_\{ub\}\(10^\{9\}\)=0\.9295\\ldots\\times 10^\{\-18\}\+O\(d^\{\-\\frac\{3\}\{2\}\}\)indicating that the first two terms in the FPS expansions ofℒ⁡\(𝝃⁡\(d\)\)\\mathcal\{L\}\(\{\\boldsymbol\{\\xi\}\}\(d\)\)andℒ​\(𝝃~​\(d\)\)\\mathcal\{L\}\(\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)\)are equal\. \(2\) Using computations ford=109d=10^\{9\}, the initial terms of anFPSexpansion forβ¯i​\(d\)\\overline\{\\beta\}\_\{i\}\(d\),i∈𝟐i\\in\{\\mathbf\{2\}\}, are

β¯0​\(d\)\\displaystyle\\overline\{\\beta\}\_\{0\}\(d\)=\\displaystyle=−11\.74​…​d−2\+O⁡\(d−52\)\\displaystyle\-11\.74\\ldots d^\{\-2\}\+O\(d^\{\-\\frac\{5\}\{2\}\}\)β¯1​\(d\)\\displaystyle\\overline\{\\beta\}\_\{1\}\(d\)=\\displaystyle=2\.612​…​d−1\+O⁡\(d−32\)\\displaystyle 2\.612\\ldots d^\{\-1\}\+O\(d^\{\-\\frac\{3\}\{2\}\}\)and have opposite sign\. \(3\) The families𝝃⁡\(d\)\{\\boldsymbol\{\\xi\}\}\(d\)and𝝃~​\(d\)\{\\widetilde\{\\boldsymbol\{\\xi\}\}\}\(d\)are\(4,2,2,4,2,2\)\(4,2,2,4,2,2\)\-matched\.※

### 15\.7\.Comments on numerical methods: comparison with bias free networks

Rather than using exact Newton’s method, we avoid computing the many second derivatives of the loss function by using approximations to the second derivatives\. Specifically, partial derivatives off​\(𝐱\)=grad​\(ℒ\)​\(𝐱\)f\(\{\\mathbf\{x\}\}\)=\{\\text\{\\rm grad\}\(\{\\mathcal\{L\}\}\)\}\(\{\\mathbf\{x\}\}\)are approximated by\[f⁡\(𝐱\+h​𝐞i\)−f⁡\(𝐱\)\]/h\[f\(\{\\mathbf\{x\}\}\+h\{\\mathbf\{e\}\}\_\{i\}\)\-f\(\{\\mathbf\{x\}\}\)\]/hwhere𝐱∈ℝn\{\\mathbf\{x\}\}\\in\{\\mathbb\{R\}\}^\{n\},\(𝐞1,…,𝐞n\)\(\{\\mathbf\{e\}\}\_\{1\},\\ldots,\{\\mathbf\{e\}\}\_\{n\}\)is an orthonormal basis ofℝn\{\\mathbb\{R\}\}^\{n\}andhhis a “small” scalar, dependent on the precision\. This method works well and reliably for bias\-free ReLU networks\. However, matters are more challenging when bias is introduced\. There are two potential problems\. \(1\) The presence of terms such as−γ¯i​cot⁡\(λi​i\)\+β¯i​csc⁡\(λi​i\)\-\\overline\{\\gamma\}\_\{i\}\\cot\(\\lambda\_\{ii\}\)\+\\overline\{\\beta\}\_\{i\}\\csc\(\\lambda\_\{ii\}\)whenλi​i\\lambda\_\{ii\}converges to00orπ\\piasd→∞d\{\\rightarrow\}\\infty\(as it does in all the examples\)\. This only has a happy ending if\|γ¯i−β¯i\|\|\\overline\{\\gamma\}\_\{i\}\-\\overline\{\\beta\}\_\{i\}\|decays fast enough to zero asd→∞d\{\\rightarrow\}\\infty\. In particular, ifγ¯i=0\\overline\{\\gamma\}\_\{i\}=0, thenβ¯i\\overline\{\\beta\}\_\{i\}needs to have a fast enough decay to offset the singular behaviour of either the cotangent or cosecant at00orπ\\pi\(see also Section[5](https://arxiv.org/html/2609.30661#S5)\)\. \(2\) Formula \([5\.21](https://arxiv.org/html/2609.30661#S5.E21)\) seems unlikely to behave well with respect to the approximate Newton method sketched above\.

With a little care in choosing the incrementhh, the second problem can be managed within the framework of the approximate Newton method, at least for values ofddthat are not too large and if theγ¯\\overline\{\\gamma\}terms are assumed to be zero\. As an example, ifd=10512d=10^\{512\}and we take the type I family of maximum isotropy spurious minima for the unbiased network and try to use Newton’s method, then the iteration fails ifh=10−480h=10^\{\-480\}but converges to the correct critical point ifh=10−600h=10^\{\-600\}\. Precision in both case was 16384 bit — approximately 4900 decimal places\. The iteration converges significantly faster ifh=10−1000h=10^\{\-1000\}with errors \(value of LHS of the associated critical point equation\) ranging from about10−440010^\{\-4400\}to10−493410^\{\-4934\}, much the same as withh=10−600h=10^\{\-600\}\.

As it is, there is a simple resolution of the problem obtained by using the easily computed exact derivatives of the integral that is being represented using theTT\-function \(see Propositions[7\.4](https://arxiv.org/html/2609.30661#S7.Thmlemma4)and[7\.5](https://arxiv.org/html/2609.30661#S7.Thmlemma5)\)\.

The first problem appears harder to manage but more interesting\. One possibility is that the result of using noisy data is not to have convergence to spurious minima when theβ¯i\\overline\{\\beta\}\_\{i\}terms are not decaying fast enough indd\(this is potentially a mechanism for removing spurious minima whenk\>dk\>d\)\.

Summarizing\. if approximate Newton routines are used, then it is important to choose the incrementshhcarefully so as not to leave the region of subanalyticity\. This is potentially a serious problem for approximating derivatives of terms depending on theTT\-function on account of the ratios ofβ¯\\overline\{\\beta\}terms that occur\.

The complete set of the critical point data used in this section is available on request as a “data\.h” file that can be incorporated into a high precision program\. Also available is an*MPFR*program that includes tested code for the computation of theTT\-function and the generalizedTT\-function \(see Appendix C\)\.

## Appendix AReview of the error functionerf​\(x\)\{\\text\{\\rm erf\}\(\{x\}\)\}

We state a number of elementary results on the error function that are used in Section[3](https://arxiv.org/html/2609.30661#S3)\.

Recall the definitions of the error and complementary error functions:

\(A\.56\)erf​\(x\)\\displaystyle\{\\text\{\\rm erf\}\(\{x\}\)\}=\\displaystyle=2π​∫0xe−t2​𝑑t,erfc​\(x\)=1−erf​\(x\)\\displaystyle\\frac\{2\}\{\\sqrt\{\\pi\}\}\\int\_\{0\}^\{x\}e^\{\-t^\{2\}\}dt,\\quad\{\\text\{\\rm erfc\}\(\{x\}\)\}=1\-\{\\text\{\\rm erf\}\(\{x\}\)\}
###### Remark A\.1\.

The error function is an*odd*function ofxxbut the complementary error function is neither odd nor even\. Integrals such as∫e−\(a​t\+b\)2​𝑑t\\int e^\{\-\(at\+b\)^\{2\}\}dthave invariance under the symmetry\(a,b\)↔\(−a,−b\)\(a,b\)\\leftrightarrow\(\-a,\-b\)and this carries through to the error function defined by the integral \(see \([A\.59](https://arxiv.org/html/2609.30661#A1.E59)\)—note the appearance ofaatwice\)\. This is not the case if we work with the complementary error function\.✠\\maltese

Related to the error function areG′​\(x\)=12​π​e−x22G^\{\\prime\}\(x\)=\\frac\{1\}\{\\sqrt\{2\\pi\}\}e^\{\-\\frac\{x^\{2\}\}\{2\}\}, the univariate density of the standard normal distribution, andG⁡\(x\)=∫−∞xG′​\(t\)​𝑑tG\(x\)=\\int\_\{\-\\infty\}^\{x\}G^\{\\prime\}\(t\)dtthe cumulative density function:

\(A\.57\)G⁡\(x\)\\displaystyle G\(x\)=\\displaystyle=12​\[1\+erf​\(x2\)\]\\displaystyle\\frac\{1\}\{2\}\\left\[1\+\{\\text\{\\rm erf\}\\left\(\{\\frac\{x\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\]\(A\.58\)dd​x​erf​\(x2\)\\displaystyle\\frac\{d\}\{dx\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{x\}\{\\sqrt\{2\}\}\}\\right\)\}=\\displaystyle=2​G′​\(x\)=2π​e−x22\\displaystyle 2G^\{\\prime\}\(x\)=\\sqrt\{\\frac\{2\}\{\\pi\}\}e^\{\-\\frac\{x^\{2\}\}\{2\}\}
Next, some elementary results used in Section[3](https://arxiv.org/html/2609.30661#S3)about integrals related to the error function \(see\[[12](https://arxiv.org/html/2609.30661#bib.bib12),[15](https://arxiv.org/html/2609.30661#bib.bib15),[16](https://arxiv.org/html/2609.30661#bib.bib16)\]for general tables\)\. To this end,a≠0a\\neq 0andbbwill denote real constants or real functions depending on a parameter\.

\(A\.59\)∫e−\(a​x\+b\)2​𝑑x\\displaystyle\\int e^\{\-\(ax\+b\)^\{2\}\}dx=\\displaystyle=π2​a​erf​\(a​x\+b\)\\displaystyle\\frac\{\\sqrt\{\\pi\}\}\{2a\}\{\\text\{\\rm erf\}\(\{ax\+b\}\)\}and so

\(A\.60\)dd​x​erf​\(a​x\+b\)=2​aπ​e−\(a​x\+b\)2\\frac\{d\}\{dx\}\{\\text\{\\rm erf\}\(\{ax\+b\}\)\}=\\frac\{2a\}\{\\sqrt\{\\pi\}\}e^\{\-\(ax\+b\)^\{2\}\}More generally, ifffis continuously differentiable,

\(A\.61\)dd​x​erf​\(f⁡\(x\)\)=2​f′​\(x\)π​e−f​\(x\)2\\frac\{d\}\{dx\}\{\\text\{\\rm erf\}\(\{f\(x\)\}\)\}=\\frac\{2f^\{\\prime\}\(x\)\}\{\\sqrt\{\\pi\}\}e^\{\-f\(x\)^\{2\}\}
###### Example A\.2\.

In terms of definite integrals, we have

\(A\.62\)∫0xe−\(a​t\+b\)2​𝑑t\\displaystyle\\int\_\{0\}^\{x\}e^\{\-\(at\+b\)^\{2\}\}dt=\\displaystyle=π2​a​\(erf​\(a​x\+b\)−erf​\(b\)\)\\displaystyle\\frac\{\\sqrt\{\\pi\}\}\{2a\}\\big\(\{\\text\{\\rm erf\}\(\{ax\+b\}\)\}\-\{\\text\{\\rm erf\}\(\{b\}\)\}\\big\)\(A\.63\)∫x∞e−\(a​t\+b\)2​𝑑t\\displaystyle\\int\_\{x\}^\{\\infty\}e^\{\-\(at\+b\)^\{2\}\}dt=\\displaystyle=π2​a​\(sgn​\(a\)−erf​\(a​x\+b\)\)\\displaystyle\\frac\{\\sqrt\{\\pi\}\}\{2a\}\\big\(\\mbox\{sgn\}\(a\)\-\{\\text\{\\rm erf\}\(\{ax\+b\}\)\}\\big\)Ifsgn​\(a\)=1\\mbox\{sgn\}\(a\)=1, thenπ2​a​\(sgn​\(a\)−erf​\(a​x\+b\)\)=erfc​\(a​x\+b\)\\frac\{\\sqrt\{\\pi\}\}\{2a\}\\big\(\\mbox\{sgn\}\(a\)\-\{\\text\{\\rm erf\}\(\{ax\+b\}\)\}\\big\)=\{\\text\{\\rm erfc\}\(\{ax\+b\}\)\}\.※

\(A\.64\)∫x​e−\(a​x\+b\)2​𝑑x=−12​a2​\[e−\(a​x\+b\)2\+π​b​erf​\(a​x\+b\)\]\\int xe^\{\-\(ax\+b\)^\{2\}\}dx=\-\\frac\{1\}\{2a^\{2\}\}\\big\[e^\{\-\(ax\+b\)^\{2\}\}\+\\sqrt\{\\pi\}b\\,\{\\text\{\\rm erf\}\(\{ax\+b\}\)\}\\big\]
###### Example A\.3\.

Using \([A\.64](https://arxiv.org/html/2609.30661#A1.E64)\), we have

∫X∞t​e−\(a​t\+b\)2​𝑑t=12​a2​\[e−\(a​X\+b\)2\+π​b​\[erf​\(a​X\+b\)−sgn​\(a\)\]\]\\int\_\{X\}^\{\\infty\}te^\{\-\(at\+b\)^\{2\}\}dt=\\frac\{1\}\{2a^\{2\}\}\\left\[e^\{\-\(aX\+b\)^\{2\}\}\+\\sqrt\{\\pi\}b\\big\[\{\\text\{\\rm erf\}\(\{aX\+b\}\)\}\-\\mbox\{sgn\}\(a\)\\big\]\\right\]Note the invariance under the symmetry\(a,b\)↔\(−a,−b\)\(a,b\)\\leftrightarrow\(\-a,\-b\)\.※

\(A\.65\)∫x2​e−\(a​x\+b\)2​dx=12​a3​\[\(b−a​x\)​e−\(a​x\+b\)2\+π​\(2​b2\+1\)2​erf​\(a​x\+b\)\]\\int x^\{2\}e^\{\-\(ax\+b\)^\{2\}\}dx=\\\\ \\frac\{1\}\{2a^\{3\}\}\\left\[\(b\-ax\)e^\{\-\(ax\+b\)^\{2\}\}\+\\frac\{\\sqrt\{\\pi\}\(2b^\{2\}\+1\)\}\{2\}\{\\text\{\\rm erf\}\(\{ax\+b\}\)\}\\right\]
\(A\.66\)∫x3​e−\(a​x\+b\)2​dx=−π2​a4​\(b3\+3​b2\)​erf​\(a​x\+b\)−12​a4​e−\(a​x\+b\)2​\[a2​x2−a​b​x\+b2\+1\]\\int x^\{3\}e^\{\-\(ax\+b\)^\{2\}\}dx=\\\\ \\begin\{aligned\} &\-\\frac\{\\sqrt\{\\pi\}\}\{2a^\{4\}\}\\big\(b^\{3\}\+\\frac\{3b\}\{2\}\\big\)\{\\text\{\\rm erf\}\(\{ax\+b\}\)\}\-\\frac\{1\}\{2a^\{4\}\}e^\{\-\(ax\+b\)^\{2\}\}\\left\[a^\{2\}x^\{2\}\-abx\+b^\{2\}\+1\\right\]\\end\{aligned\}Integrals involvingerf​\(a​x\+b\)\{\\text\{\\rm erf\}\(\{ax\+b\}\)\}can often be computed using integration by parts and \([A\.60](https://arxiv.org/html/2609.30661#A1.E60)\)\. For example,

\(A\.67\)∫0xerf​\(a​t\+b\)​dt=a−1​\[\(a​x\+b\)​erf​\(a​x\+b\)−b​erf​\(b\)\]\+1a​π​\[e−\(a​x\+b\)2−e−b2\]\\int\_\{0\}^\{x\}\{\\text\{\\rm erf\}\(\{at\+b\}\)\}dt=a^\{\-1\}\\left\[\(ax\+b\)\{\\text\{\\rm erf\}\(\{ax\+b\}\)\}\-b\\,\{\\text\{\\rm erf\}\(\{b\}\)\}\\right\]\\\\ \\begin\{aligned\} \+\\frac\{1\}\{a\\sqrt\{\\pi\}\}\\left\[e^\{\-\(ax\+b\)^\{2\}\}\-e^\{\-b^\{2\}\}\\right\]\\end\{aligned\}The following result, which uses Example[A\.3](https://arxiv.org/html/2609.30661#A1.Thmlemma3), is used for computingf⁡\(𝐰,𝐯\)f\(\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\)when there is non\-zero bias\.

\(A\.68\)∫X∞t2​erf​\(a​t\+b2\)​e−t22​dt=X​e−X22​erf​\(a​X\+b2\)\+a2​b\(a2\+1\)32​e−b22​\(1\+a2\)​\[erf​\(\(1\+a2\)​X\+b​a2​\(1\+a2\)\)−1\]\+2π​aa2\+1​e−b2\+\(\(1\+a2\)​X\+b​a\)22​\(1\+a2\)\+∫X∞e−t22​erf​\(a​t\+b2\)​𝑑t\\int\_\{X\}^\{\\infty\}t^\{2\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{at\+b\}\{\\sqrt\{2\}\}\}\\right\)\}e^\{\-\\frac\{t^\{2\}\}\{2\}\}dt=Xe^\{\-\\frac\{X^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{aX\+b\}\{\\sqrt\{2\}\}\}\\right\)\}\+\\\\ \\begin\{aligned\} &\\frac\{a^\{2\}b\}\{\(a^\{2\}\+1\)^\{\\frac\{3\}\{2\}\}\}e^\{\-\\frac\{b^\{2\}\}\{2\(1\+a^\{2\}\)\}\}\\left\[\{\\text\{\\rm erf\}\\left\(\{\\frac\{\(1\+a^\{2\}\)X\+ba\}\{\\sqrt\{2\(1\+a^\{2\}\)\}\}\}\\right\)\}\-1\\right\]\+\\\\ &\\sqrt\{\\frac\{2\}\{\\pi\}\}\\frac\{a\}\{a^\{2\}\+1\}e^\{\-\\frac\{b^\{2\}\+\(\(1\+a^\{2\}\)X\+ba\)^\{2\}\}\{2\(1\+a^\{2\}\)\}\}\+\\int\_\{X\}^\{\\infty\}e^\{\-\\frac\{t^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{at\+b\}\{\\sqrt\{2\}\}\}\\right\)\}dt\\end\{aligned\}
###### Remark A\.4\.

ForX∈\(−∞,∞\)X\\in\(\-\\infty,\\infty\), the integral∫X∞e−t22​erf​\(a​t\+b2\)​𝑑t\\int\_\{X\}^\{\\infty\}e^\{\-\\frac\{t^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{at\+b\}\{\\sqrt\{2\}\}\}\\right\)\}dtcannot be evaluated in terms of the error function and exponentials\. However, the integral can be represented in terms of the*Owen T\-function*\[[17](https://arxiv.org/html/2609.30661#bib.bib17),[18](https://arxiv.org/html/2609.30661#bib.bib18)\]—see Section[3](https://arxiv.org/html/2609.30661#S3)\.✠\\maltese

## Appendix BHigh precision computation of Owen’sTT\-function

High precision computations were done using the*MPFR*package which uses*GMP*and*GCC*\. As far as we know, Owen’sTT\-function is currently not available as a standard built\-in function in*MPFR*or any other high precision package\. However, it is available in several \(low precision\) packages; for example, the statistical software*R*\. Recently, a package was written by Komelj that uses the*Rmpmr*high precision extension of*R*\. This package uses his method for the computation of theTTfunction\[[11](https://arxiv.org/html/2609.30661#bib.bib11)\]and gives a fast high precision computation ofT⁡\(h,a\)T\(h,a\)for all values ofh,ah,a\. Based on the analysis in\[[11](https://arxiv.org/html/2609.30661#bib.bib11)\], we have written*MPFR*code for high precision computation of theTT\-function with a view to applications to biased ReLU networks\. We give the basic method below and refer the reader to Komelj’s article\[[11](https://arxiv.org/html/2609.30661#bib.bib11)\]for details we omit\.

The aim is to computeT⁡\(h,a\)T\(h,a\)forh,a∈ℝh,a\\in\{\\mathbb\{R\}\}\. We use the two algorithms \(40a\) and a slightly modified \(40c\) from\[[11](https://arxiv.org/html/2609.30661#bib.bib11), Theorem 5\]121212Ifa<−1a<\-1, apply the second algorithm to computeT⁡\(h,−a\)T\(h,\-a\)and thenT⁡\(h,a\)=−T⁡\(h,−a\)T\(h,a\)=\-T\(h,\-a\)\. Note that both \(40a,40c\) converge for alla∈ℝa\\in\{\\mathbb\{R\}\}but that \(40a\) \(resp\. \(40c\)\) converges fastest if\|a\|≤1\|a\|\\leq 1\(resp\.a\>1a\>1\)\. In the proof of\[[11](https://arxiv.org/html/2609.30661#bib.bib11), Theorem 5\], reference is made to Owen’s formula \(17\) and that should be \(15\) \(*op\. cit\.*\)\. As stated,\[[11](https://arxiv.org/html/2609.30661#bib.bib11), \(40c\)\]is not quite correct—this does not affect any of the numerical results in\[[11](https://arxiv.org/html/2609.30661#bib.bib11)\]which depend on an application of \(15\)\.\.

T⁡\(h,a\)\\displaystyle T\(h,a\)=\\displaystyle=t​a​n−1​\(a\)2​π−a2​π​\(1\+a2\)​∑k=0∞\(2​k\)\!\!\(2​k\+1\)\!\!​P​\(k\+1,q\)​\(a21\+a2\)k,if​\|a\|≤1,\\displaystyle\\frac\{tan^\{\-1\}\(a\)\}\{2\\pi\}\-\\frac\{a\}\{2\\pi\(1\+a^\{2\}\)\}\\sum\_\{k=0\}^\{\\infty\}\\frac\{\(2k\)\!\!\}\{\(2k\+1\)\!\!\}P\(k\+1,q\)\\left\(\\frac\{a^\{2\}\}\{1\+a^\{2\}\}\\right\)^\{k\},\\;\\text\{if \}\|a\|\\leq 1,=\\displaystyle=U⁡\(h,a\)−14\+t​a​n−1​\(a\)2​π\+a2​π​\(1\+a2\)​∑k=0∞\(2​k\)\!\!\(2​k\+1\)\!\!​P​\(k\+1,q\)​\(11\+a2\)k,if​a\>1\.\\displaystyle U\(h,a\)\-\\frac\{1\}\{4\}\+\\frac\{tan^\{\-1\}\(a\)\}\{2\\pi\}\+\\frac\{a\}\{2\\pi\(1\+a^\{2\}\)\}\\sum\_\{k=0\}^\{\\infty\}\\frac\{\(2k\)\!\!\}\{\(2k\+1\)\!\!\}P\(k\+1,q\)\\left\(\\frac\{1\}\{1\+a^\{2\}\}\\right\)^\{k\},\\;\\text\{if \}a\>1\.where

q\\displaystyle q=\\displaystyle=12​\(1\+a2\)​h2\\displaystyle\\frac\{1\}\{2\}\(1\+a^\{2\}\)h^\{2\}U⁡\(h,a\)\\displaystyle U\(h,a\)=\\displaystyle=14​\[1−erf​\(h2\)​erf​\(h​a2\)\],a≥0\\displaystyle\\frac\{1\}\{4\}\\left\[1\-\{\\text\{\\rm erf\}\\left\(\{\\frac\{h\}\{\\sqrt\{2\}\}\}\\right\)\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{ha\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\],\\;a\\geq 0andP⁡\(k\+1,q\)P\(k\+1,q\)is the*lower regularized Gamma function*defined fork≥0k\\geq 0by

P⁡\(k\+1,q\)=∫0qt−k​e−t​𝑑tΓ⁡\(k\+1\)∈\(0,1\)P\(k\+1,q\)=\\frac\{\\int\_\{0\}^\{q\}t^\{\-k\}e^\{\-t\}\\,dt\}\{\\Gamma\(k\+1\)\}\\in\(0,1\)Noting the elementary identitiesP⁡\(k\+1,q\)=P⁡\(k,q\)−qa​e−qΓ⁡\(k\+1\)P\(k\+1,q\)=P\(k,q\)\-\\frac\{q^\{a\}e^\{\-q\}\}\{\\Gamma\(k\+1\)\},P⁡\(1,q\)=1−e−qP\(1,q\)=1\-e^\{\-q\}, recursively define sequences\(Bk\)k≥0,\(Dk\)k≥0=\(P​\(k\+1,q\)k≥0CLOSE\(B\_\{k\}\)\_\{k\\geq 0\},\(D\_\{k\}\)\_\{k\\geq 0\}=\(P\(k\+1,q\)\_\{k\\geq 0\}by

1. \(1\)B0=e−q,D0=1−B0B\_\{0\}=e^\{\-q\},D\_\{0\}=1\-B\_\{0\}\.
2. \(2\)Bk\+1=q​Bkk\+1B\_\{k\+1\}=\\frac\{qB\_\{k\}\}\{k\+1\},Dk\+1=Dk−Bk\+1D\_\{k\+1\}=D\_\{k\}\-B\_\{k\+1\}

This leads directly to a recursive definition of the terms in the infinite series∑k=0∞\(2​k\)\!\!\(2​k\+1\)\!\!​P​\(k\+1,q\)​\(a21\+a2\)k\\sum\_\{k=0\}^\{\\infty\}\\frac\{\(2k\)\!\!\}\{\(2k\+1\)\!\!\}P\(k\+1,q\)\\left\(\\frac\{a^\{2\}\}\{1\+a^\{2\}\}\\right\)^\{k\}and the corresponding partial sumsSkS\_\{k\}\. Noting the positivity of the terms, and following\[[11](https://arxiv.org/html/2609.30661#bib.bib11)\], the computation is terminated whenSk\+1<SkS\_\{k\+1\}<S\_\{k\}and the sum is taken to beSkS\_\{k\}\. We refer to\[[11](https://arxiv.org/html/2609.30661#bib.bib11)\]for more details\. In our implementation, the partial sums are slightly different from those used in\[[11](https://arxiv.org/html/2609.30661#bib.bib11)\], On initialization, we also define the sequence\(\(2​k\)\!\!\(2​k\+1\)\!\!\)k≤10,000\(\\frac\{\(2k\)\!\!\}\{\(2k\+1\)\!\!\}\)\_\{k\\leq 10,000\}of double factorials\.

###### Remark B\.1\.

Although the upper and lower regularized Gamma functions are used in\[[11](https://arxiv.org/html/2609.30661#bib.bib11)\], the*MPFR*implementation of the upper and lower regularized Gamma functions is not used\. Indeed, as is commented in the*MPFR*manual\[[13](https://arxiv.org/html/2609.30661#bib.bib13), mpfr\_gamma\_inc, page 32\], the*MPFR*implementation of the incremental gamma function can be slow for large values of the arguments\.✠\\maltese

## Appendix CGeneralizedTT\-function

Recall from Section[3](https://arxiv.org/html/2609.30661#S3)that the generalizedTT\-function\[[21](https://arxiv.org/html/2609.30661#bib.bib21)\]is defined by

\(C\.69\)T⁡\(h,a,b\)=12​2​π​∫h∞e−t22​erf​\(a​t\+b2\)​𝑑tT\(h,a,b\)=\\frac\{1\}\{2\\sqrt\{2\\pi\}\}\\int\_\{h\}^\{\\infty\}e^\{\-\\frac\{t^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{at\+b\}\{\\sqrt\{2\}\}\}\\right\)\}dtand that

2​π​T​\(γ¯,−c​o​t​\(θ\),β¯​csc⁡\(θ\)\)=π2​∫γ¯∞e−t22​erf​\(−t​cot⁡\(θ\)\+β¯​csc⁡\(θ\)2\)​𝑑t2\\pi T\(\\overline\{\\gamma\},\-cot\(\\theta\),\\overline\{\\beta\}\\csc\(\\theta\)\)=\\sqrt\{\\frac\{\\pi\}\{2\}\}\\int\_\{\\overline\{\\gamma\}\}^\{\\infty\}e^\{\-\\frac\{t^\{2\}\}\{2\}\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{\-t\\cot\(\\theta\)\+\\overline\{\\beta\}\\csc\(\\theta\)\}\{\\sqrt\{2\}\}\}\\right\)\}dtIt follows from\[[21](https://arxiv.org/html/2609.30661#bib.bib21)\]that

T⁡\(h,a,b\)=12​π​\[tan−1⁡\(a\)−tan−1⁡\(a\+bh\)−tan−1⁡\(a\+h⁡\(1\+a2\)b\)\]\+T⁡\(h,a\+bh\)\+T⁡\(b1\+a2,a\+h⁡\(1\+a2\)b\)\+14​erf​\(b2​\(1\+a2\)\)\\begin\{aligned\} \\hskip\-9\.39545ptT\(h,a,b\)=&\\frac\{1\}\{2\\pi\}\\left\[\\tan^\{\-1\}\(a\)\-\\tan^\{\-1\}\\left\(a\+\\frac\{b\}\{h\}\\right\)\-\\tan^\{\-1\}\\left\(a\+\\frac\{h\(1\+a^\{2\}\)\}\{b\}\\right\)\\right\]\\\\ &\+T\\left\(h,a\+\\frac\{b\}\{h\}\\right\)\+T\\left\(\\frac\{b\}\{\\sqrt\{1\+a^\{2\}\}\},a\+\\frac\{h\(1\+a^\{2\}\)\}\{b\}\\right\)\\\\ &\+\\frac\{1\}\{4\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{b\}\{\\sqrt\{2\(1\+a^\{2\}\)\}\}\}\\right\)\}\\end\{aligned\}
On the other hand, Owen\[[18](https://arxiv.org/html/2609.30661#bib.bib18)\]gives a formula in terms of the CDF

∫G′​\(x\)​G​\(a​x\+b\)​dx=T⁡\(x,bx​1\+a2\)\+T⁡\(b1\+a2,x​1\+a2b\)−T⁡\(x,a​x\+bx\)−T⁡\(b1\+a2,a​b\+x⁡\(1\+a2\)b\)\+G⁡\(x\)​G​\(b1\+a2\)\\int G^\{\\prime\}\(x\)G\(ax\+b\)dx=T\\left\(x,\\frac\{b\}\{x\\sqrt\{1\+a^\{2\}\}\}\\right\)\\\\ \\begin\{aligned\} &\+T\\left\(\\frac\{b\}\{\\sqrt\{1\+a^\{2\}\}\},\\frac\{x\\sqrt\{1\+a^\{2\}\}\}\{b\}\\right\)\-T\\left\(x,\\frac\{ax\+b\}\{x\}\\right\)\\\\ &\-T\\left\(\\frac\{b\}\{\\sqrt\{1\+a^\{2\}\}\},\\frac\{ab\+x\(1\+a^\{2\}\)\}\{b\}\\right\)\+G\(x\)G\\left\(\\frac\{b\}\{\\sqrt\{1\+a^\{2\}\}\}\\right\)\\end\{aligned\}SinceG′​\(t\)=12​π​e−t22G^\{\\prime\}\(t\)=\\frac\{1\}\{\\sqrt\{2\\pi\}\}e^\{\-\\frac\{t^\{2\}\}\{2\}\}andG​\(t\)=12​\[1\+erf​\(t2\)\]G\(t\)=\\frac\{1\}\{2\}\\left\[1\+\{\\text\{\\rm erf\}\(\{\\frac\{t\}\{\\sqrt\{2\}\}\}\)\}\\right\],

∫h∞G′​\(t\)​G​\(a​t\+b\)​𝑑t=T⁡\(h,a,b\)\+14​erfc​\(h2\)\\int\_\{h\}^\{\\infty\}G^\{\\prime\}\(t\)G\(at\+b\)dt=T\(h,a,b\)\+\\frac\{1\}\{4\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{h\}\{\\sqrt\{2\}\}\}\\right\)\}and so

\(C\.70\)T⁡\(h,a,b\)=∫h∞G′​\(t\)​G​\(a​t\+b\)​𝑑t−14​erfc​\(h2\)T\(h,a,b\)=\\int\_\{h\}^\{\\infty\}G^\{\\prime\}\(t\)G\(at\+b\)dt\-\\frac\{1\}\{4\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{h\}\{\\sqrt\{2\}\}\}\\right\)\}Using Owen’s formula for the indefinite integral∫G′​\(x\)​G​\(a​x\+b\)​𝑑x\\int G^\{\\prime\}\(x\)G\(ax\+b\)dx, the contribution atx=∞x=\\inftyis

12​\[1\+erf​\(b2​\(1\+a2CLOSE\)\]\\frac\{1\}\{2\}\\left\[1\+\{\\text\{\\rm erf\}\\left\(\{\\frac\{b\}\{\\sqrt\{2\(1\+a^\{2\}\}\}\}\\right\)\}\\right\]and that atx=hx=his

T⁡\(h,bh​1\+a2\)\+T⁡\(b1\+a2,h​1\+a2b\)\\displaystyle T\\left\(h,\\frac\{b\}\{h\\sqrt\{1\+a^\{2\}\}\}\\right\)\+T\\left\(\\frac\{b\}\{\\sqrt\{1\+a^\{2\}\}\},\\frac\{h\\sqrt\{1\+a^\{2\}\}\}\{b\}\\right\)−T⁡\(h,a\+bh\)−T⁡\(b1\+a2,a\+h⁡\(1\+a2\)b\)\\displaystyle\-T\\left\(h,a\+\\frac\{b\}\{h\}\\right\)\-T\\left\(\\frac\{b\}\{\\sqrt\{1\+a^\{2\}\}\},a\+\\frac\{h\(1\+a^\{2\}\)\}\{b\}\\right\)\+14​\(1\+erf​\(h2\)\)​\(1\+erf​\(bOPEN2​\(1\+a2\)\)\)\)\\displaystyle\+\\frac\{1\}\{4\}\\left\(1\+\{\\text\{\\rm erf\}\\left\(\{\\frac\{h\}\{\\sqrt\{2\}\}\}\\right\)\}\\right\)\\left\(1\+\{\\text\{\\rm erf\}\\left\(\{\\frac\{b\}\{\\sqrt\{2\(1\+a^\{2\}\)\)\}\}\}\\right\)\}\\right\)Hence

T⁡\(h,a,b\)\\displaystyle T\(h,a,b\)=\\displaystyle=T⁡\(h,a\+bh\)\+T⁡\(b1\+a2,a\+h⁡\(1\+a2\)b\)\\displaystyle T\\left\(h,a\+\\frac\{b\}\{h\}\\right\)\+T\\left\(\\frac\{b\}\{\\sqrt\{1\+a^\{2\}\}\},a\+\\frac\{h\(1\+a^\{2\}\)\}\{b\}\\right\)−T⁡\(h,bh​1\+a2\)−T⁡\(b1\+a2,h​1\+a2b\)\\displaystyle\-T\\left\(h,\\frac\{b\}\{h\\sqrt\{1\+a^\{2\}\}\}\\right\)\-T\\left\(\\frac\{b\}\{\\sqrt\{1\+a^\{2\}\}\},\\frac\{h\\sqrt\{1\+a^\{2\}\}\}\{b\}\\right\)\+14​erf​\(b2​\(1\+a2\)\)​erfc​\(h2\)\\displaystyle\+\\frac\{1\}\{4\}\{\\text\{\\rm erf\}\\left\(\{\\frac\{b\}\{\\sqrt\{2\(1\+a^\{2\}\)\}\}\}\\right\)\}\{\\text\{\\rm erfc\}\\left\(\{\\frac\{h\}\{\\sqrt\{2\}\}\}\\right\)\}
Unlike the formula used in the text, this formula uses 4 instances of theTTfunction\. However, using\[[18](https://arxiv.org/html/2609.30661#bib.bib18), Table II, 2\.8,2\.9\], it can be shown that−T⁡\(h,bh​1\+a2\)−T⁡\(b1\+a2,h​1\+a2b\)\-T\\left\(h,\\frac\{b\}\{h\\sqrt\{1\+a^\{2\}\}\}\\right\)\-T\\left\(\\frac\{b\}\{\\sqrt\{1\+a^\{2\}\}\},\\frac\{h\\sqrt\{1\+a^\{2\}\}\}\{b\}\\right\)can be expressed in terms of inverse tangents and error functions and so obtain the formula given in\[[21](https://arxiv.org/html/2609.30661#bib.bib21)\]\.

As well as analytically, we have verified that the two expressions for the generalizedTT\-function give the same value forT⁡\(h,a,b\)T\(h,a,b\)to 615 decimal places when working in 2048 bit precision using*MPFR*\.

## References

- \[1\]Y Arjevani and M Field\. ‘On the principle of least symmetry breaking in shallow relu models’\. arXiv:1912\.11939\.
- \[2\]Y Arjevani and M Field\. ‘Analytic characterization of the hessian in shallow relu models: A tale of symmetry’,*NeurIPS 2020*\.
- \[3\]Y Arjevani and M Field\. ‘Analytic study of families of spurious minima in two\-layer ReLU neural networks: a tale of symmetry II’,*Proc\.NeurIPS 2021*, 15162\-–15174
- \[4\]Y Arjevani and M Field\.‘Symmetry and critical points for a model shallow neural network’,*Physica D*427\(2021\)\. arXiv:2003\.10576, 2020\.
- \[5\]Y Arjevani and M Field\. ‘Annihilation of Spurious Minima in Two\-Layer ReLU Networks’,*Proc\.NeurIPS 2022*
- \[6\]Y Arjevani and M Field\. ‘Equivariant bifurcation, quadratic equivariants, and symmetry breaking for the standard representation ofSkS\_\{k\}’,*Nonlinearity*35\(6\) \(2022\), 2809\.
- \[7\]E Bierstone and P Milman\. ‘Subanalytic Geometry’,*Model theory, algebra and geometry \(ed\. Haskell, D et al\.\)*MSRI, 39 \(1998\), 151–172
- \[8\]A Brutzkus and A Globerson\. ‘Globally optimal gradient descent for a convnet with gaussian inputs’,*Proc\. of the 34th Int\. Conf\. on Machine Learning*70\(2017\), 605–614\.
- \[9\]A Brutzkus, A Globerson, E Malach, & S Shalev\-Shwartz\. ‘SGD Learns Over\-parameterized Networks that Provably Generalize on Linearly Separable Data’, \(in*6th International Conference on Learning Representations*, ICLR 2018, Vancouver, BC, Canada, April 30– May 3, 2018, Conf\. Track Proc\., 2018\)\.
- \[10\]Y Cho and L K Saul\. ‘Kernel Methods for Deep Learning’,*Advances in neural information processing systems*\(2009\), 342–350\.
- \[11\]J Komelj\. ‘The Bivariate Normal Integral via Owen’sTTFunction as a Modified Euler’s Arctangent series’,*Amer\. Jnr\. of Comp\. Math\.*13\(4\) \(2023\), 476–504 \(also arXiv:2312\.00011v1\)\.
- \[12\]N E Korotkov\.*Table of integrals related to error function*\(based on Russian text by N E Korotkov, edited by A N Korotkov, 2019\)\.
- \[13\]GNU MPFR ‘The Multiple Precision Floating\-Point Reliable Library’, \(version 4\.2\.2, available atmpfr\.org/mpfr\-current/\#doc\)\.
- \[14\]J Milnor\.*Singular points of complex hypersurfaces*\(Annals of Math\. Studies 61, Princeton University Press, 1968\)\.
- \[15\]E W Ng and M Geller\. ‘A Table of Integrals of the Error Function’,*Jnr\. Research of the Nat\. Bureau of Stand\.—B\. Math\. Sci\.*73B\(1\) \(1969\), 1–26\.
- \[16\]E W Ng and M Geller\. ‘A Table of Integrals of the Error Function\. II\. Additions and Corrections\.’,*Jnr\. Research of the Nat\. Bureau of Stand\.—B\. Math\. Sci\.*75B\(3\) \(1971\), 149–163\.
- \[17\]D B Owen\. ‘Tables for computing bivariate normal probabilities’,*Ann\. Math\. Statist\.*27\(1956\), 1075–1090
- \[18\]D B Owen\. ‘A table of normal integrals’,*Commun\. Statist\.–Simula\. and Comput\.*9:4 \(1980\), 389–419\.*9th Innovations in Theoretical Computer Science Conference, ITCS 2018*, January 11\-14, 2018, Cambridge, MA, USA, 2018\), 22:1–22:19\.
- \[19\]A Parusiński\. ‘Subanaytic functions’,*Trans\. Amer\. Math\. Soc\.*,344\(2\) \(12994\), 583–595\.
- \[20\]A Pinkus, ‘Approximation theory of the MLP model in neural networks’,*Acta Numer\.*8\(1999\), 143–195\.
- \[21\]Przemo \(https://math\.stackexchange\.com/users/99778/przemo\), Generalized Owen’s T function, URL \(version: 2019\-03\-28\): https://math\.stackexchange\.com/q/3087504
- \[22\]I Safran and O Shamir\. ‘Spurious Local Minima are Common in Two\-Layer ReLU Neural Networks’,*Proc\. of the 35th Int\. Conf\. on Machine Learning*80\(2018\), 4433–4441 \(for data sets, see https://github\.com/ItaySafran/OneLayerGDconvergence\)\.
- \[23\]I Safran, G Yehudai, & P Shamir\. ‘The Effects of Mild Over\-parametrization on the Optimization Landscape of Shallow ReLU Neural Networks’,*Proc\. of Machine Learning Research*134\(2021\), 1\-–46\.
- \[24\]Y Zhang, A Saxe, & P E Latham\. ‘When are bias\-free ReLU networks effectively linear networks?’*arXiv:2406\.12615v3*\.

Similar Articles

Shallower ReLU Network Representations via Exact Linear Algebra

arXiv cs.LG

This paper improves theoretical bounds on the depth of ReLU networks needed to represent the maximum function, showing exact two-hidden-layer representations for up to 10 inputs and improved depth for larger n via exact linear algebra techniques.

Singular Learning and Occam's Razor in Deep Monomial Networks

arXiv cs.LG

This paper studies critical points in deep fully-connected networks with monomial activations, proving that for large activation degree, criticality occurs precisely at subnetworks, offering a mathematical perspective on implicit bias toward simpler functions.

The Implicit Bias of Depth: From Neural Collapse to Softmax Codes

arXiv cs.LG

This paper studies how depth alone induces an implicit low-rank bias in deep unconstrained feature models trained without regularization, shifting the optimal solution from neural collapse to softmax codes, and provides the first asymptotic and dynamic characterization of this bias under gradient descent with cross-entropy loss.