Randomly initialized autoencoders: fixed points and edge-of-chaos

# Randomly initialized autoencoders: fixed points and edge-of-chaos
Source: [https://arxiv.org/html/2608.14638](https://arxiv.org/html/2608.14638)
###### Abstract

In this paper we study autoencoders, a special class of deep neural nets \(DNNs\) whose performance can be characterized via their fixed points\. This perspective naturally raises questions of existence, stability, and basins of attraction of these fixed points\. These questions are addressed via the contractive properties of autoencoders, and are closely related to the notion of edge\-of\-chaos\.

Edge\-of\-chaos \(EoC\) is an important notion in the theory of DNNs\. It describes the critical regime separating ordered and chaotic signal propagation through a randomly initialized network\. Initialization at or near this critical regime offers several theoretical and practical advantages, including stability of the network w\.r\.t\. perturbations of the input\. EoC was previously introduced for broad classes of neural networks using mean\-field averaging methods\. In this paper we modify the notion of EoC for the study of autoencoders\. Specifically, we introduce local and global EoC for autoencoders that controls local \(small\) and global \(arbitrary\) perturbations of the input respectively\.

The study of stability of autoencoders falls within the scope of nonlinear problems in Random Matrix Theory \(RMT\)\. Our analysis of local EoC is based on spectral techniques of RMT, whereas global EoC is studied by employing Sudakov–Fernique inequality for Gaussian processes\.

## 1Introduction

### 1\.1Autoencoder DNNs: what and why?

Autoencoders \(AEs\) constitute an important class of deep neural networks \(DNNs\) designed to learn efficient representations of data \(e\.g\., image\) through compression and reconstruction\. We first recall the definition of a feedforward fully\-connected DNN\. For eachk=1,2,…,Lk=1,2,\\ldots,L, thekk\-th layer function is

Φ\(k\):ℝnk→ℝnk\+1,Φ\(k\)​\(s\)=λ​\(W\(k\)​s\+b\(k\)\),\\Phi^\{\(k\)\}:\\ \\mathbb\{R\}^\{n\_\{k\}\}\\to\\mathbb\{R\}^\{n\_\{k\+1\}\},\\quad\\Phi^\{\(k\)\}\(s\)=\\lambda\(W^\{\(k\)\}s\+b^\{\(k\)\}\),\(1\)whereW\(k\)W^\{\(k\)\}is annk\+1×nk\{n\_\{k\+1\}\\times\{n\_\{k\}\}\}matrix,b\(k\)∈ℝnk\+1b^\{\(k\)\}\\in\\mathbb\{R\}^\{n\_\{k\+1\}\}is a vector, andλ\\lambdais a nonlinear function applied coordinatewise\. In other words, for eachkk,Φ\(k\)\\Phi^\{\(k\)\}is a composition of an affine maps↦W\(k\)​s\+b\(k\)s\\mapsto W^\{\(k\)\}s\+b^\{\(k\)\}and a nonlinear functionλ:ℝ→ℝ\\lambda\\colon\\mathbb\{R\}\\to\\mathbb\{R\}applied to each coordinate ofW\(k\)​s\+b\(k\)W^\{\(k\)\}s\+b^\{\(k\)\}\. The matrixW\(k\)W^\{\(k\)\}is called a*weight matrix*, the vectorb\(k\)b^\{\(k\)\}is called a*bias vector*, and the functionλ\\lambdais called anactivation function\.

AnLL\-layer DNN function is defined as a composition:

𝐗​\(s\)=\(Φ\(L\)∘Φ\(L−1\)∘…∘Φ\(2\)∘Φ\(1\)\)​\(s\)\.\{\\bf X\}\(s\)=\(\\Phi^\{\(L\)\}\\circ\\Phi^\{\(L\-1\)\}\\circ\\ldots\\circ\\Phi^\{\(2\)\}\\circ\\Phi^\{\(1\)\}\)\(s\)\.\(2\)The vectorss∈ℝn1s\\in\\mathbb\{R\}^\{n\_\{1\}\}ands¯=𝐗​\(s\)∈ℝnL\+1\\bar\{s\}=\{\\bf X\}\(s\)\\in\\mathbb\{R\}^\{n\_\{L\+1\}\}are calledinputandoutput vectorscorrespondingly\.

Autoencoders \(AEs\) are a special class of DNNs when the input and output vectors have the same dimensionn1=nL\+1=Nn\_\{1\}=n\_\{L\+1\}=N\. An autoencoder consists of two DNN functions: an encoder𝐗E\{\\bf X\}\_\{E\}, and a decoder𝐗D\{\\bf X\}\_\{D\}, i\.e\.

𝐗A​E=𝐗D∘𝐗E\.\{\\bf X\}\_\{AE\}=\{\\bf X\}\_\{D\}\\circ\{\\bf X\}\_\{E\}\.\(3\)The encoder𝐗E\{\\bf X\}\_\{E\}maps the input𝒔\\boldsymbol\{s\}into a lower\-dimensional latent representation𝒛∈ℝn\\boldsymbol\{z\}\\in\\mathbb\{R\}^\{n\}, withn≪Nn\\ll N, while the decoder𝐗D\{\\bf X\}\_\{D\}reconstructs the original input from this latent space:

𝐗E​\(𝒔\)=𝒛,𝐗D​\(𝒛\)=𝒔¯\.\{\\bf X\}\_\{E\}\(\\boldsymbol\{s\}\)=\\boldsymbol\{z\},\\quad\{\\bf X\}\_\{D\}\(\\boldsymbol\{z\}\)=\\boldsymbol\{\\bar\{s\}\}\.\(4\)
XEX\_\{E\}XDX\_\{D\}Figure 1:A simple AE withN=7N=7,n=3n=3\. The blue and green nodes are the components of the input and output vectors, the red nodes form the latent layer\.A simple AE with a one\-layer encoder and decoder is represented in Fig\.[1](https://arxiv.org/html/2608.14638#S1.F1)\.

Autoencoders \(AEs\) have become a versatile tool in modern data analysis due to their ability to learn compact and informative representations of high dimensional data in an unsupervised manner\. One application of AEs is nonlinear dimensionality reduction, where AEs generalize classical techniques such as principal component analysis, see, e\.g\.,\[[7](https://arxiv.org/html/2608.14638#bib.bib7)\]\. In this case AE functionXA​EX\_\{AE\}\(see \([3](https://arxiv.org/html/2608.14638#S1.E3)\) \- \([4](https://arxiv.org/html/2608.14638#S1.E4)\)\) performs sequential mapping of input vector \(image\)𝒔\\boldsymbol\{s\}to latent layer vector𝒛\\boldsymbol\{z\}and then mapping of𝒛\\boldsymbol\{z\}back to𝒔\\boldsymbol\{s\}, i\.e\.,XA​EX\_\{AE\}acts as identity function\. Due to the fact that the latent layer is smaller or even much smaller than input and output layers, AE function cannot provide identity map on the entireℝN\\mathbb\{R\}^\{N\}\. Our goal is to provide a map which is small deviation from identity map on a low\-dimensional manifold\. If AE functionXA​EX\_\{AE\}provides contraction on a subsetSSof the manifold, then according to the Banach fixed point theorem there exist unique fixed point inSS:

𝐗A​E​\(s\)=s\{\\bf X\}\_\{AE\}\(s\)=s\(5\)with basin of attractionSS\. These fixed points correspond to images which are compressed and decompressed precisely, and the rest of input vectors fromSSform a class which can be restored approximately\. Hence, fixed points can be used as unique identifier \(label\) for all the input vectors lying in the basin of attraction\. For this reason, the criteria ofXA​EX\_\{AE\}contraction are important for the AE design and are studied in this work\.

Weight matricesW\(k\)W^\{\(k\)\}and bias vectorsb\(k\)b^\{\(k\)\}in \([1](https://arxiv.org/html/2608.14638#S1.E1)\) are determined by the choice of their entries\. This process is calledinitialization\. However, for a given choice ofW\(k\)W^\{\(k\)\}andb\(k\)b^\{\(k\)\}, AE may not have fixed points, and those appear after iterative adjustment of weights, see\[[2](https://arxiv.org/html/2608.14638#bib.bib2)\]\. This procedure is calledtrainingand it amounts to minimization of the so\-calledloss function, e\.g\.,\[[1](https://arxiv.org/html/2608.14638#bib.bib1),[9](https://arxiv.org/html/2608.14638#bib.bib9)\]\.

In the problem of image encoding/decoding the loss function can be chosen as:

ℒ​\(\{α\}\)=∑s∈T\|𝐗A​E​\(𝒔,\{α\}\)−𝒔\|,\{\\cal L\}\(\\\{\\alpha\\\}\)=\\sum\_\{s\\in T\}\|\{\\bf X\}\_\{AE\}\(\\boldsymbol\{s\},\\\{\\alpha\\\}\)\-\\boldsymbol\{s\}\|,\(6\)whereT⊂ST\\subset Sis a so\-called training set\. Here\{α\}\\\{\\alpha\\\}denotes the set of the entries of all weight matrices and bias vectors\.

The efficiency, e\.g\. speed and accuracy, of training critically depends on the initialization\[[8](https://arxiv.org/html/2608.14638#bib.bib8)\], and it was shown \(e\.g\.\[[25](https://arxiv.org/html/2608.14638#bib.bib25),[23](https://arxiv.org/html/2608.14638#bib.bib23)\]\) that the best training results can be achieved when the matrices are initializedrandomly\. Our main focus is how the choice of random initialization determines the existence, stability, and basins of attraction of the fixed points of AEs\. In this paper we consider AEs initialized with centered Gaussian weights and study the dependence of the behavior on the variance of the weights\.

Particular attention in the work is paid to the study of edge\-of\-chaos \(EoC\) for general feed\-forward DNNs \(see, e\.g\.,\[[21](https://arxiv.org/html/2608.14638#bib.bib21),[19](https://arxiv.org/html/2608.14638#bib.bib19),[14](https://arxiv.org/html/2608.14638#bib.bib14),[10](https://arxiv.org/html/2608.14638#bib.bib10),[5](https://arxiv.org/html/2608.14638#bib.bib5)\]\)\. This is a critical regime of parameter initialization which corresponds to the a transition between two DNN behaviors: stability and chaos\. When the initialization is chosen close to EoC, small perturbations in the input data neither vanish nor explode while the input propagates through the DNN, which leads to the most efficient transfer of information through the layers\. In the case of Gaussian initialization with the weights distributed asN​\(0,σ2/n\)N\(0,\\sigma^\{2\}/n\), EoC is the critical valueσc​r​i​t2\\sigma^\{2\}\_\{crit\}of the variance parameterσ2\\sigma^\{2\}such that forσ2<σc​r​i​t2\\sigma^\{2\}<\\sigma^\{2\}\_\{crit\}perturbations of the input vanish and forσ2\>σc​r​i​t2\\sigma^\{2\}\>\\sigma^\{2\}\_\{crit\}perturbations explode while propagating through the layers\.

The focus of our paper is on autoencoder DNNs\. Therefore, we introduce the notion of EoC specifically for autoencoders that allow one to study existence, stability, and basins of attraction of fixed points\. Moreover, stability of fixed points can be studied for small perturbations \(local stability\) or arbitrary perturbations \(global stability\)\. For this reason we introduce local and global EoC for autoencoders in discussions after Theorem[1\.1](https://arxiv.org/html/2608.14638#S1.Thmtheorem1)and Corollary[1\.4](https://arxiv.org/html/2608.14638#S1.Thmtheorem4)respectively\. Note that in the study of fixed points in neuroscience science literature for firing networks \(see, e\.g\.,\[[30](https://arxiv.org/html/2608.14638#bib.bib30),[32](https://arxiv.org/html/2608.14638#bib.bib32),[22](https://arxiv.org/html/2608.14638#bib.bib22),[24](https://arxiv.org/html/2608.14638#bib.bib24)\]\), it is sufficient to consider local stability\. For autoencoders, uniform convergence w\.r\.t\. inputs is important for the global stability\.

In the local stability analysis,Random Matrix Theory\(RMT\) techniques can be successfully applied to study input\-output Jacobian of the autoencoder function\. However, for the global stability the uniformity in inputssis required, which is essentially harder to capture using standard RMT approaches\. Instead, we develop novel techniques based on the theory of Gaussian processes andSudakov\-Fernique inequality\(Theorem[2\.1](https://arxiv.org/html/2608.14638#S2.Thmtheorem1), see\[[27](https://arxiv.org/html/2608.14638#bib.bib27),[28](https://arxiv.org/html/2608.14638#bib.bib28),[6](https://arxiv.org/html/2608.14638#bib.bib6)\]\)\.

### 1\.2Notation and main results

The principal object of our study is anLL\-layer autoencoder𝐗L,σ2\(N\)​\(s\)\{\\bf X\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\(s\)\([2](https://arxiv.org/html/2608.14638#S1.E2)\) with

Φk​\(s\)=ϕα​\(W\(k\)​s\+b\(k\)\),\\Phi\_\{k\}\(s\)=\\phi\_\{\\alpha\}\(W^\{\(k\)\}s\+b^\{\(k\)\}\)~,whereW\(k\)W^\{\(k\)\}is annk\+1×nk\{n\_\{k\+1\}\\times\{n\_\{k\}\}\}random matrix withN​\(0,σ2nk\)N\\left\(0,\\dfrac\{\\sigma^\{2\}\}\{n\_\{k\}\}\\right\)\-distributed entries,b\(k\)∈ℝnk\+1b^\{\(k\)\}\\in\\mathbb\{R\}^\{n\_\{k\+1\}\}is a random vector withN​\(0,σ~2\)N\(0,\\tilde\{\\sigma\}^\{2\}\), all the random variables are jointly independent, input and output dimensions are equal:n1=nL\+1=Nn\_\{1\}=n\_\{L\+1\}=N, and the activation functionϕα\\phi\_\{\\alpha\}\(0≤α<10\\leq\\alpha<1\) is given by

ϕα​\(x\)=\{x,x≥0;α​xx<0,\\phi\_\{\\alpha\}\(x\)=\\begin\{cases\}x,&x\\geq 0;\\\\ \\alpha x&x<0,\\end\{cases\}\(7\)\(known as ReLU forα=0\\alpha=0and LeakyReLU for0<α<10<\\alpha<1\)\. Let

𝐉L,σ2\(N\)​\(s\)=∂𝐗L,σ2\(N\)​\(s\)∂s\{\\bf J\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\)=\\dfrac\{\\partial\{\\bf X\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\(s\)\}\{\\partial s\}\(8\)be theinput\-output Jacobianof the neural network\. Our first result describes the typical behavior of‖𝐉L,σ2\(N\)​\(s\)‖\\\|\{\\bf J\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\)\\\|for a fixed vectorss\.

###### Theorem 1\.1

Assumeα=0\\alpha=0\(ReLU activation\)\. Assume that for eachk=2,3,…,Lk=2,3,\\ldots,Lthere existlimN→∞nkN=ck∈\(c;C\)\\lim\\limits\_\{N\\to\\infty\}\\dfrac\{n\_\{k\}\}\{N\}=c\_\{k\}\\in\(c;C\)for someC\>c\>0C\>c\>0\. Then, for anys∈ℝNs\\in\\mathbb\{R\}^\{N\}, we have almost surely

limL→∞limN→∞1L​log⁡‖𝐉L,σ2\(N\)​\(s\)‖=12​log⁡\(σ2/2\)\.\\lim\_\{L\\to\\infty\}\\lim\_\{N\\to\\infty\}\\frac\{1\}\{L\}\\log\\\|\{\\bf J\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\)\\\|=\\frac\{1\}\{2\}\\log\(\\sigma^\{2\}/2\)~\.\(9\)

To recast this result in the form reminiscent of the DNN terminology, we can define thelocal edge\-of\-chaos of autoencoderas the critical valueσL,crit,loc2\\sigma^\{2\}\_\{L,\\text\{crit\},\\text\{loc\}\}for which almost surely

limN→∞1L​log⁡‖𝐉L,σ2\(N\)​\(s\)‖=0\.\\lim\_\{N\\to\\infty\}\\frac\{1\}\{L\}\\log\\\|\{\\bf J\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\)\\\|=0~\.\(10\)Then Theorem[1\.1](https://arxiv.org/html/2608.14638#S1.Thmtheorem1)can be equivalently stated as

limL→∞σL,crit,loc2=2\.\\lim\_\{L\\to\\infty\}\\sigma^\{2\}\_\{L,\\text\{crit\},\\text\{loc\}\}=2~\.\(11\)
The notion of edge\-of\-chaos was introduced and studied for a wide class of DNNs, see e\.g\.,\[[21](https://arxiv.org/html/2608.14638#bib.bib21),[19](https://arxiv.org/html/2608.14638#bib.bib19),[14](https://arxiv.org/html/2608.14638#bib.bib14),[10](https://arxiv.org/html/2608.14638#bib.bib10),[5](https://arxiv.org/html/2608.14638#bib.bib5)\], and is defined as the critical valueσcrit2\\sigma^\{2\}\_\{\\mathrm\{crit\}\}of the varianceσ2\\sigma^\{2\}in the initializationN​\(0,σ2nk\)N\(0,\\dfrac\{\\sigma^\{2\}\}\{n\_\{k\}\}\)for which

𝔼​\{1N​Tr​\(𝐉\(k\)​\(s\)\)T​𝐉\(k\)​\(s\)\}→1asN→∞,\\mathbb\{E\}\\Bigl\\\{\\dfrac\{1\}\{N\}\\mathrm\{Tr\}\\,\(\{\\bf J\}^\{\(k\)\}\(s\)\)^\{T\}\{\\bf J\}^\{\(k\)\}\(s\)\\Bigr\\\}\\to 1\\quad\\text\{as $N\\to\\infty$\},\(12\)where𝐉\(k\)​\(s\)\{\\bf J\}^\{\(k\)\}\(s\)is the input\-output Jacobian of thekk\-th layer functionΦ\(k\)\\Phi^\{\(k\)\}of the DNN\. Note thatσcrit2\\sigma^\{2\}\_\{\\mathrm\{crit\}\}is independent ofkkbecause of the assumption that the diagonal components of Jacobian are independent ofkk, see e\.g\.,\[[19](https://arxiv.org/html/2608.14638#bib.bib19), Section 2\.2\]and\[[21](https://arxiv.org/html/2608.14638#bib.bib21), Section 3\]\.

The edge\-of\-chaos introduced in e\.g\.,\[[21](https://arxiv.org/html/2608.14638#bib.bib21),[19](https://arxiv.org/html/2608.14638#bib.bib19),[14](https://arxiv.org/html/2608.14638#bib.bib14),[10](https://arxiv.org/html/2608.14638#bib.bib10),[5](https://arxiv.org/html/2608.14638#bib.bib5)\]shows whether small \(local\) perturbations of the input data are dying out or exploding when propagating through multilayer DNN, while our notion of local EoC shows if a multilayer DNN has a local contraction property \(stability w\.r\.t\. small perturbations\)\. In contrast, the global EoC introduced below shows if a multilayer DNN has a contraction property on the entire set of data \(stability w\.r\.t\. arbitrary perturbations\)\.

Note that it was proved in\[[19](https://arxiv.org/html/2608.14638#bib.bib19), Subsection 3\.4\]that forσ2=σcrit2\\sigma^\{2\}=\\sigma^\{2\}\_\{\\mathrm\{crit\}\}convergence \([12](https://arxiv.org/html/2608.14638#S1.E12)\) also holds with𝐉L,σ2\(N\)\{\\bf J\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}in place of𝐉\(k\)\{\\bf J\}^\{\(k\)\}, i\.e\. edge\-of\-chaosσcrit2\\sigma^\{2\}\_\{\\mathrm\{crit\}\}can be defined in terms of the input\-output Jacobian of the entire DNN rather than single layer\. Then the standard inequality between Hilbert\-Schmidt and operator norms implies

σL,crit,loc2≤σcrit2\.\\sigma^\{2\}\_\{L,\\mathrm\{crit,\\,loc\}\}\\leq\\sigma^\{2\}\_\{\\mathrm\{crit\}\}\.\(13\)Moreover, since for ReLU activationσcrit2=2\\sigma^\{2\}\_\{\\mathrm\{crit\}\}=2\(it was proved for DNNs with layers of the same width, see\[[19](https://arxiv.org/html/2608.14638#bib.bib19), Table 1\]\), our result \([14](https://arxiv.org/html/2608.14638#S1.E14)\) yields

limL→∞σL,crit,loc2=σcrit2\\lim\_\{L\\to\\infty\}\\sigma^\{2\}\_\{L,\\text\{crit\},\\text\{loc\}\}=\\sigma^\{2\}\_\{\\mathrm\{crit\}\}~\(14\)for ReLU activation, that is for large number of layers these two values of EoC coincide\.

An important issue in applications of autoencoders is to prove that \([10](https://arxiv.org/html/2608.14638#S1.E10)\) holds uniformly inss, which will imply that forσ2<σcrit2\\sigma^\{2\}<\\sigma^\{2\}\_\{\\mathrm\{crit\}\}autoencoder𝐗L,σ2\(N\)\{\\bf X\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}is a contraction not only on a neighborhood of a given inputss, but also on the entireℝN\\mathbb\{R\}^\{N\}\.

The following second result provides a bound which is uniform in the initial vectorss\. In this theorem we allow arbitrary0≤α<10\\leq\\alpha<1\. On the other hand, we consider the particular case of no biases, i\.e\.σ~2=0\\tilde\{\\sigma\}^\{2\}=0\. In this case, it is sufficient to control‖𝐗​\(s\)‖\\\|\{\\bf X\}\(s\)\\\|rather than‖𝐉​\(s\)‖\\\|\{\\bf J\}\(s\)\\\|when studying global stability of a fixed point, see Corollary[1\.4](https://arxiv.org/html/2608.14638#S1.Thmtheorem4)below\.

###### Theorem 1\.2

Assume0≤α<10\\leq\\alpha<1,σ~2=0\\tilde\{\\sigma\}^\{2\}=0\(no bias\), and𝐗L,σ2\(N\)​\(s\)\{\\bf X\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\(s\)has layer widthsc​N≤nk≤C​NcN\\leq n\_\{k\}\\leq CNfor someC\>c\>0C\>c\>0\. Denoteck=nkN∈\[c;C\]c\_\{k\}=\\dfrac\{n\_\{k\}\}\{N\}\\in\[c;C\]and

βα\(L\)=\{2​L​\(1\+∑k=1L−11\+ck\+1ck\+1cL\)−1,ifα=0;L​\(1\+∑k=1L−11\+ck\+1ck⋅1\+α22\+1cL⋅1\+α22\)−1,if0<α<1\.\\beta\_\{\\alpha\}^\{\(L\)\}=\\begin\{cases\}\\sqrt\{2\}L\\left\(1\+\\sum\\limits\_\{k=1\}^\{L\-1\}\\sqrt\{1\+\\dfrac\{c\_\{k\+1\}\}\{c\_\{k\}\}\}\+\\sqrt\{\\dfrac\{1\}\{c\_\{L\}\}\}\\right\)^\{\-1\},\\quad\\text\{if $\\alpha=0$;\}\\\\ \\vskip 9\.0pt\\cr L\\left\(1\+\\sum\\limits\_\{k=1\}^\{L\-1\}\\sqrt\{1\+\\dfrac\{c\_\{k\+1\}\}\{c\_\{k\}\}\\cdot\\dfrac\{1\+\\alpha^\{2\}\}\{2\}\}\+\\sqrt\{\\dfrac\{1\}\{c\_\{L\}\}\}\\cdot\\sqrt\{\\dfrac\{1\+\\alpha^\{2\}\}\{2\}\}\\right\)^\{\-1\},\\quad\\text\{if $0<\\alpha<1$\.\}\\end\{cases\}\(15\)Then for anyε\>0\\varepsilon\>0there existsc~=c~​\(ε,L,c,C\)\>0\\widetilde\{c\}=\\widetilde\{c\}\(\\varepsilon,L,c,C\)\>0such that with probability≥1−exp⁡\{−c~​N​L\}\\geq 1\-\\exp\\\{\-\\widetilde\{c\}NL\\\}:

sups∈Ωα‖s‖=1‖𝐗L,σ2\(N\)​\(s\)‖≤\(σβα\(L\)\+ε\)L,\\sup\_\{\\begin\{subarray\}\{c\}s\\in\\Omega\_\{\\alpha\}\\\\ \\\|s\\\|=1\\end\{subarray\}\}\\\|\{\\bf X\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\)\\\|\\leq\\left\(\\dfrac\{\\sigma\}\{\\beta\_\{\\alpha\}^\{\(L\)\}\}\+\\varepsilon\\right\)^\{L\},\(16\)where

Ωα=\{ℝ\+N,ifα=0;ℝN,if0<α<1,ℝ\+N=\{\(s1,…,sN\)∈ℝN:s1,…,sN≥0\}\.\\Omega\_\{\\alpha\}=\\begin\{cases\}\\mathbb\{R\}^\{N\}\_\{\+\},\\quad\\text\{if $\\alpha=0$;\}\\\\ \\vskip 4\.0pt\\cr\\mathbb\{R\}^\{N\},\\quad\\text\{if $0<\\alpha<1$,\}\\end\{cases\}\\qquad\\mathbb\{R\}^\{N\}\_\{\+\}=\\\{\(s\_\{1\},\\ldots,s\_\{N\}\)\\in\\mathbb\{R\}^\{N\}\\colon s\_\{1\},\\ldots,s\_\{N\}\\geq 0\\\}\.\(17\)

The following two corollaries describe a special case of Theorem[1\.2](https://arxiv.org/html/2608.14638#S1.Thmtheorem2)which captures asymptotic behavior of autoencoders in the large\-depth limit\. Specifically, it provides a simple upper bound for the variance in the distribution of random initialization that insures that the autoencoder has unique stable fixed point\.

###### Corollary 1\.3

Assume0≤α<10\\leq\\alpha<1,σ~2=0\\tilde\{\\sigma\}^\{2\}=0\(no bias\), and𝐗L,σ2\(N\)\{\\bf X\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}has layer widthsn1=n2=…=nl−1=nl\+1=…=nL\+1=Nn\_\{1\}=n\_\{2\}=\\ldots=n\_\{l\-1\}=n\_\{l\+1\}=\\ldots=n\_\{L\+1\}=N,c​N≤nl<NcN\\leq n\_\{l\}<Nfor somel∈\[2;L\]l\\in\[2;L\]and fixedc∈\(0;1\)c\\in\(0;1\)\. Denote

βα=\{1,ifα=0;23\+α2,if0<α<1\.\\beta\_\{\\alpha\}=\\begin\{cases\}1,\\quad\\text\{if $\\alpha=0$;\}\\\\ \\vskip 7\.0pt\\cr\\sqrt\{\\dfrac\{2\}\{3\+\\alpha^\{2\}\}\},\\quad\\text\{if $0<\\alpha<1$\.\}\\end\{cases\}\(18\)Ifσ<βα\\sigma<\\beta\_\{\\alpha\}, then with probability≥1−C~​exp⁡\(−c~​N\)\\geq 1\-\\widetilde\{C\}\\exp\(\-\\widetilde\{c\}N\)\(whereC~,c~\>0\\widetilde\{C\},\\widetilde\{c\}\>0depend onσ2,c\\sigma^\{2\},c\) we have:

limL→∞sup‖s‖=1s∈Ωα‖𝐗L,σ2\(N\)​\(s\)‖=0,\\lim\_\{L\\to\\infty\}\\sup\_\{\\stackrel\{\{\\scriptstyle s\\in\\Omega\_\{\\alpha\}\}\}\{\{\\scriptscriptstyle\\\|s\\\|=1\}\}\}\\\|\{\\bf X\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\)\\\|=0~,\(19\)whereΩα\\Omega\_\{\\alpha\}defined as in \([17](https://arxiv.org/html/2608.14638#S1.E17)\)\.

Due to Corollary[1\.3](https://arxiv.org/html/2608.14638#S1.Thmtheorem3), ifσ<βα\\sigma<\\beta\_\{\\alpha\}then for large enoughL≥L~L\\geq\\widetilde\{L\}with probability≥1−C~​exp⁡\(−c~​N\)\\geq 1\-\\widetilde\{C\}\\exp\(\-\\widetilde\{c\}N\)we have

K:=sup‖s‖=1s∈Ωα‖𝐗L,σ2\(N\)​\(s\)‖<1\.K:=\\sup\_\{\\stackrel\{\{\\scriptstyle\\scriptstyle s\\in\\Omega\_\{\\alpha\}\}\}\{\{\\scriptstyle\\\|s\\\|=1\}\}\}\\\|\{\\bf X\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\)\\\|<1\.\(20\)Consider an iterative process

s0∈ℝN,sm=𝐗L,σ2\(N\)​\(sm−1\)\.s\_\{0\}\\in\\mathbb\{R\}^\{N\},\\quad s\_\{m\}=\{\\bf X\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\_\{m\-1\}\)\.\(21\)Notice thatsk∈Ωαs\_\{k\}\\in\\Omega\_\{\\alpha\}fork≥1k\\geq 1\. Then positive homogeneity ofXL,σ2\(N\)​\(s\)X\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\)implies

‖sm‖=‖\(𝐗L,σ2\(N\)∘𝐗L,σ2\(N\)∘…∘𝐗L,σ2\(N\)\)⏟m−1​times​\(s1\)‖≤Km−1​‖s1‖,\\\|s\_\{m\}\\\|=\\Bigl\\\|\\underbrace\{\(\{\\bf X\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\\circ\{\\bf X\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\\circ\\ldots\\circ\{\\bf X\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\)\}\_\{m\-1\\text\{ times\}\}\(s\_\{1\}\)\\Bigr\\\|\\leq K^\{m\-1\}\\\|s\_\{1\}\\\|,\(22\)which shows that

limm→∞sm=0\.\\lim\_\{m\\to\\infty\}s\_\{m\}=0\.\(23\)The observation above leads to the following result\.

###### Corollary 1\.4

Assume0≤α<10\\leq\\alpha<1,σ~2=0\\tilde\{\\sigma\}^\{2\}=0\(no bias\), and𝐗L,σ2\(N\)​\(s\)\{\\bf X\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\(s\)has layer widthsn1=n2=…=nl−1=nl\+1=…=nL\+1=Nn\_\{1\}=n\_\{2\}=\\ldots=n\_\{l\-1\}=n\_\{l\+1\}=\\ldots=n\_\{L\+1\}=N,c​N≤nl<NcN\\leq n\_\{l\}<Nfor somel∈\[2;L\]l\\in\[2;L\]and fixedc∈\(0;1\)c\\in\(0;1\)\. Ifσ<βα\\sigma<\\beta\_\{\\alpha\}defined in \([18](https://arxiv.org/html/2608.14638#S1.E18)\), then forL≥L~L\\geq\\widetilde\{L\}with probability≥1−C~​exp⁡\(−c~​N\)\\geq 1\-\\widetilde\{C\}\\exp\(\-\\widetilde\{c\}N\)\(whereL~\\widetilde\{L\},C~,c~\>0\\widetilde\{C\},\\widetilde\{c\}\>0depend onσ2,c\\sigma^\{2\},c\) autoencoder function𝐗L,σ2\(N\)​\(s\)\{\\bf X\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\)has unique stable fixed point0with basin of attractionℝN\\mathbb\{R\}^\{N\}\.

As we see, for autoencoders with ReLU/LeakyReLU activation and no biases global stability of the fixed point is controlled bysups∈Ωα,‖s‖=1‖𝐗L,σ2\(N\)​\(s\)‖\\sup\\limits\_\{s\\in\\Omega\_\{\\alpha\},\\\|s\\\|=1\}\\\|\{\\bf X\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\)\\\|\. For such autoencoders, we can defineglobal edge\-of\-chaosas the valueσL,crit,glob2\\sigma^\{2\}\_\{L,\\text\{crit\},\\text\{glob\}\}for which almost surely

limN→∞sups∈Ωα‖s‖=1‖𝐗L,σ2\(N\)​\(s\)‖=1\.\\lim\_\{N\\to\\infty\}\\sup\_\{\\begin\{subarray\}\{c\}s\\in\\Omega\_\{\\alpha\}\\\\ \\\|s\\\|=1\\end\{subarray\}\}\\\|\{\\bf X\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\)\\\|=1\.\(24\)Corollary[1\.3](https://arxiv.org/html/2608.14638#S1.Thmtheorem3)provides a lower bound on global edge\-of\-chaos in large\-depth limit:

lim infL→∞σL,crit,glob2≥βα2\.\\liminf\\limits\_\{L\\to\\infty\}\\sigma^\{2\}\_\{L,\\mathrm\{crit,glob\}\}\\geq\\beta\_\{\\alpha\}^\{2\}\.\(25\)
Unlike local EoC, which is based on pointwise convergence, global EoC is based onuniformbounds w\.r\.t\. the inputss\.

The paper is organized as follows\. Section 2 contains the proof of Theorem[1\.2](https://arxiv.org/html/2608.14638#S1.Thmtheorem2)in the ReLU caseα=0\\alpha=0\. Section 3 contains the proof of Theorem[1\.1](https://arxiv.org/html/2608.14638#S1.Thmtheorem1)\. The proof of Theorem[1\.2](https://arxiv.org/html/2608.14638#S1.Thmtheorem2)in the LeakyReLU case0<α<10<\\alpha<1follows the exact same steps as in Section 2 and is given in Appendix A\.

Acknowledgement\.The work of V\.S\. and R\.S\. is supported by Grant “International Multilateral Partnerships for Resilient Education and Science System in Ukraine” IMPRESS\-U: N7114 funded by US National Academy of Science and Office of Naval Research Global\. The work of V\.S\. is partially supported by NASU project F 26\-5\. The work of R\.S\. is partially supported by NASU project M 22\-13\. The work of L\.B\. is partially supported by the NSF Grant IMPRESS\-U: N2401227\. The work of S\.S\. and Y\.S\. is partially supported by the Or programme of the ISF under agreement number 4101/25\. The work of Y\.S\. is partially supported by by the European Research Council \(ERC\) under the European Union’s Horizon 2022 research and innovation program \(grant agreement No\. 101041711\)\.

## 2Proof of Theorem[1\.2](https://arxiv.org/html/2608.14638#S1.Thmtheorem2), ReLU caseα=0\\alpha=0

### 2\.1Auxiliary tools

The main technical tool of the proof is Sudakov–Fernique inequality \(\[[27](https://arxiv.org/html/2608.14638#bib.bib27),[28](https://arxiv.org/html/2608.14638#bib.bib28),[6](https://arxiv.org/html/2608.14638#bib.bib6)\]\):

###### Theorem 2\.1\(Sudakov–Fernique inequality\)

Let\{Pu\}u∈S\\\{P\_\{u\}\\\}\_\{u\\in S\}and\{Qu\}u∈S\\\{Q\_\{u\}\\\}\_\{u\\in S\}be centered, almost surely bounded Gaussian processes indexed by a setSS\. Assume that for allu,u′∈Su,u^\{\\prime\}\\in S,

𝔼​\(Pu−Pu′\)2≤𝔼​\(Qu−Qu′\)2\.\\mathbb\{E\}\\bigl\(P\_\{u\}\-P\_\{u^\{\\prime\}\}\\bigr\)^\{2\}\\ \\leq\\ \\mathbb\{E\}\\bigl\(Q\_\{u\}\-Q\_\{u^\{\\prime\}\}\\bigr\)^\{2\}\.\(29\)Then

𝔼​supu∈SPu≤𝔼​supu∈SQu\.\\mathbb\{E\}\\sup\_\{u\\in S\}P\_\{u\}\\ \\leq\\ \\mathbb\{E\}\\sup\_\{u\\in S\}Q\_\{u\}\.

We obtain an upper bound forsup‖𝐗L,σ2\(N\)​\(s\)‖\\sup\\\|\{\\bf X\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\)\\\|via the supremum of a certain Gaussian process\. The Sudakov–Fernique inequality will be used to estimate the expected supremum of that Gaussian process by another Gaussian process for which expected supremum is computable\.

We will also need a version of the Gaussian concentration inequality to show that the supremum of the Gaussian process is close to its expected value with high probability \(\[[3](https://arxiv.org/html/2608.14638#bib.bib3),[29](https://arxiv.org/html/2608.14638#bib.bib29)\]\):

###### Theorem 2\.2\(Borell–Tsirelson–Ibragimov–Sudakov \(Borell–TIS\) inequality\)

Let\{Pu\}u∈S\\\{P\_\{u\}\\\}\_\{u\\in S\}be a centered, almost surely bounded Gaussian process withM=supu∈S𝔼​Pu2<∞M=\\sup\\limits\_\{u\\in S\}\\mathbb\{E\}P\_\{u\}^\{2\}<\\infty\. Then for allt≥0t\\geq 0,

ℙ​\{supu∈SPu−𝔼​supu∈SPu≥t\}≤exp⁡\(−t22​M\)\.\\mathbb\{P\}\\Big\\\{\\sup\_\{u\\in S\}P\_\{u\}\-\\mathbb\{E\}\\sup\_\{u\\in S\}P\_\{u\}\\geq t\\Big\\\}\\leq\\exp\\\!\\left\(\-\\frac\{t^\{2\}\}\{2M\}\\right\)\.\(30\)

### 2\.2Outline of the proof

ConsiderLL\-layer autoencoder with ReLU activation, no biases, and Gaussian initialization:

𝐗L,σ2\(N\)​\(s\)=\(Φ\(L\)∘…∘Φ\(2\)∘Φ\(1\)\)​\(s\),Φ\(k\)​\(s\)=\(W\(k\)​s\)\+\.\{\\bf X\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\(s\)=\\left\(\\Phi^\{\(L\)\}\\circ\\ldots\\circ\\Phi^\{\(2\)\}\\circ\\Phi^\{\(1\)\}\\right\)\(s\),\\quad\\Phi^\{\(k\)\}\(s\)=\(W^\{\(k\)\}s\)\_\{\+\}\.\(31\)Here and below we use the notation

v\+=\(\(v1\)\+,…,\(vn\)\+\)​for​v∈ℝnv\_\{\+\}=\(\(v\_\{1\}\)\_\{\+\},\\ldots,\(v\_\{n\}\)\_\{\+\}\)\\text\{\\quad for \}v\\in\\mathbb\{R\}^\{n\}\(32\)for brevity\. Also denote

‖𝐗L,σ2\(N\)‖\+:=sups∈ℝ\+N‖s‖=1‖𝐗L,σ2\(N\)​\(s\)‖,\\\|\{\\bf X\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\\\|\_\{\+\}:=\\sup\_\{\\begin\{subarray\}\{c\}s\\in\\mathbb\{R\}^\{N\}\_\{\+\}\\\\ \\\|s\\\|=1\\end\{subarray\}\}\\\|\{\\bf X\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\)\\\|,\(33\)whereℝ\+N\\mathbb\{R\}^\{N\}\_\{\+\}is defined in \([17](https://arxiv.org/html/2608.14638#S1.E17)\)\.

We will perform the proof in four steps:

- •Step 1:Obtain an upper bound of the form \(‖𝐗L,σ2\(N\)‖\+\)1/L≤1L​supu∈SPu\(\\\|\{\\bf X\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\\\|\_\{\+\}\)^\{1/L\}\\leq\\dfrac\{1\}\{L\}\\sup\_\{u\\in S\}P\_\{u\}\(34\)for certain centered Gaussian process\{Pu\}u∈S\\\{P\_\{u\}\\\}\_\{u\\in S\}\.
- •Step 2:Find a centered Gaussian process\{Qu\}u∈S\\\{Q\_\{u\}\\\}\_\{u\\in S\}which majorizes\{Pu\}u∈S\\\{P\_\{u\}\\\}\_\{u\\in S\}in terms of \([29](https://arxiv.org/html/2608.14638#S2.E29)\) and such that𝔼​supu∈SQu\\mathbb\{E\}\\sup\\limits\_\{u\\in S\}Q\_\{u\}is computable\.
- •Step 3:Apply Sudakov\-Fernique inequality to obtain an upper bound1L​𝔼​supu∈SPu≤σβ0\(L\)\\dfrac\{1\}\{L\}\\mathbb\{E\}\\sup\\limits\_\{u\\in S\}P\_\{u\}\\leq\\dfrac\{\\sigma\}\{\\beta\_\{0\}^\{\(L\)\}\}\.
- •Step 4:Use Borell–TIS inequality to estimate the probability ℙ​\{1L​supu∈SPu−𝔼​\{1L​supu∈SPu\}\>ε\}\.\\mathbb\{P\}\\left\\\{\\dfrac\{1\}\{L\}\\sup\_\{u\\in S\}P\_\{u\}\-\\mathbb\{E\}\\Big\\\{\\dfrac\{1\}\{L\}\\sup\_\{u\\in S\}P\_\{u\}\\Big\\\}\>\\varepsilon\\right\\\}\.\(35\)Then apply such bound to obtain \([16](https://arxiv.org/html/2608.14638#S1.E16)\)\.

### 2\.3Step 1: Upper bound via Gaussian process

Fixs∈ℝ\+Ns\\in\\mathbb\{R\}^\{N\}\_\{\+\}with‖s‖=1\\\|s\\\|=1\. Define neuron vectors and their norms:

s\(0\):=s∈ℝ\+N,s\(k\):=Φ\(k\)​\(s\(k−1\)\)=\(W\(k\)​s\(k−1\)\)\+∈ℝ\+nk\+1,r\(k\):=‖s\(k\)‖\.s^\{\(0\)\}:=s\\in\\mathbb\{R\}^\{N\}\_\{\+\},\\quad s^\{\(k\)\}:=\\Phi^\{\(k\)\}\(s^\{\(k\-1\)\}\)=\(W^\{\(k\)\}s^\{\(k\-1\)\}\)\_\{\+\}\\in\\mathbb\{R\}^\{n\_\{k\+1\}\}\_\{\+\},\\quad r^\{\(k\)\}:=\\\|s^\{\(k\)\}\\\|\.\(36\)Thens\(L\)=𝐗L,σ2\(N\)​\(s\)s^\{\(L\)\}=\{\\bf X\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\(s\)and we can write

‖𝐗L,σ2\(N\)​\(s\)‖=r\(L\)=∏k=1Lr\(k\)r\(k−1\)\.\\\|\{\\bf X\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\(s\)\\\|=r^\{\(L\)\}=\\prod\_\{k=1\}^\{L\}\\dfrac\{r^\{\(k\)\}\}\{r^\{\(k\-1\)\}\}\.\(37\)Thus, we can rewrite‖𝐗L,σ2\(N\)‖\+\\\|\{\\bf X\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\\\|\_\{\+\}in the following way:

‖𝐗L,σ2\(N\)‖\+=sups∈ℝ\+N,‖s‖=1∏k=1Lr\(k\)r\(k−1\)\.\\\|\{\\bf X\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\\\|\_\{\+\}=\\sup\_\{s\\in\\mathbb\{R\}^\{N\}\_\{\+\},\\ \\\|s\\\|=1\}\\prod\_\{k=1\}^\{L\}\\dfrac\{r^\{\(k\)\}\}\{r^\{\(k\-1\)\}\}\.\(38\)Also define normalized neuron vectorsuk:=s\(k\)r\(k\)∈𝕊\+nk\+1−1u\_\{k\}:=\\dfrac\{s^\{\(k\)\}\}\{r^\{\(k\)\}\}\\in\\mathbb\{S\}^\{n\_\{k\+1\}\-1\}\_\{\+\}, where𝕊\+n−1=ℝ\+n∩𝕊n−1\\mathbb\{S\}^\{n\-1\}\_\{\+\}=\\mathbb\{R\}^\{n\}\_\{\+\}\\cap\\mathbb\{S\}^\{n\-1\}\. Then we can express norm gain ratiosr\(k\)r\(k−1\)\\dfrac\{r^\{\(k\)\}\}\{r^\{\(k\-1\)\}\}in terms ofuku\_\{k\}’s:

r\(k\)r\(k−1\)=1r\(k−1\)​‖\(W\(k\)​s\(k−1\)\)\+‖=1r\(k−1\)​‖\(W\(k\)​r\(k−1\)​uk−1\)\+‖=‖\(W\(k\)​uk−1\)\+‖\.\\dfrac\{r^\{\(k\)\}\}\{r^\{\(k\-1\)\}\}=\\dfrac\{1\}\{r^\{\(k\-1\)\}\}\\\|\(W^\{\(k\)\}s^\{\(k\-1\)\}\)\_\{\+\}\\\|=\\dfrac\{1\}\{r^\{\(k\-1\)\}\}\\\|\(W^\{\(k\)\}r^\{\(k\-1\)\}u\_\{k\-1\}\)\_\{\+\}\\\|=\\\|\(W^\{\(k\)\}u\_\{k\-1\}\)\_\{\+\}\\\|\.\(39\)Moreover,uku\_\{k\}’s satisfy the recurrence relation

u0=s,uk=\(W\(k\)​uk−1\)\+‖\(W\(k\)​uk−1\)\+‖\.u\_\{0\}=s,\\qquad u\_\{k\}=\\dfrac\{\(W^\{\(k\)\}u\_\{k\-1\}\)\_\{\+\}\}\{\\\|\(W^\{\(k\)\}u\_\{k\-1\}\)\_\{\+\}\\\|\}\.\(40\)It is easy to check that for any vectorv∈ℝnv\\in\\mathbb\{R\}^\{n\}we have\(v,v\+\)=‖v\+‖2\(v,v\_\{\+\}\)=\\\|v\_\{\+\}\\\|^\{2\}\. Applying this fact forv=W\(k\)​uk−1v=W^\{\(k\)\}u\_\{k\-1\}, we obtain

‖\(W\(k\)​uk−1\)\+‖=‖\(W\(k\)​uk−1\)\+‖2‖\(W\(k\)​uk−1\)\+‖=\(W\(k\)​uk−1,\(W\(k\)​uk−1\)\+\)‖\(W\(k\)​uk−1\)\+‖==\(W\(k\)​uk−1,\(W\(k\)​uk−1\)\+‖\(W\(k\)​uk−1\)\+‖\)=\(W\(k\)​uk−1,uk\)\.\\\|\(W^\{\(k\)\}u\_\{k\-1\}\)\_\{\+\}\\\|=\\dfrac\{\\\|\(W^\{\(k\)\}u\_\{k\-1\}\)\_\{\+\}\\\|^\{2\}\}\{\\\|\(W^\{\(k\)\}u\_\{k\-1\}\)\_\{\+\}\\\|\}=\\dfrac\{\(W^\{\(k\)\}u\_\{k\-1\},\(W^\{\(k\)\}u\_\{k\-1\}\)\_\{\+\}\)\}\{\\\|\(W^\{\(k\)\}u\_\{k\-1\}\)\_\{\+\}\\\|\}=\\\\ =\\left\(W^\{\(k\)\}u\_\{k\-1\},\\dfrac\{\(W^\{\(k\)\}u\_\{k\-1\}\)\_\{\+\}\}\{\\\|\(W^\{\(k\)\}u\_\{k\-1\}\)\_\{\+\}\\\|\}\\right\)=\(W^\{\(k\)\}u\_\{k\-1\},u\_\{k\}\)\.\(41\)Combining \([38](https://arxiv.org/html/2608.14638#S2.E38)\), \([39](https://arxiv.org/html/2608.14638#S2.E39)\) and \([41](https://arxiv.org/html/2608.14638#S2.E41)\), we obtain

‖𝐗L,σ2\(N\)‖\+=sups∈𝕊\+N−1∏k=1L\(W\(k\)​uk−1,uk\)\.\\\|\{\\bf X\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\\\|\_\{\+\}=\\sup\_\{s\\in\\mathbb\{S\}^\{N\-1\}\_\{\+\}\}\\prod\_\{k=1\}^\{L\}\(W^\{\(k\)\}u\_\{k\-1\},u\_\{k\}\)\.\(42\)
In order to obtain Gaussian process under the supremum, we first use AM\-GM inequality:

a1​a2​…​aL≤\(a1\+a2\+…\+aLL\)L​for​ak≥0\.a\_\{1\}a\_\{2\}\\ldots a\_\{L\}\\leq\\left\(\\dfrac\{a\_\{1\}\+a\_\{2\}\+\\ldots\+a\_\{L\}\}\{L\}\\right\)^\{L\}\\text\{\\quad for \}a\_\{k\}\\geq 0\.\(43\)We can takeak:=\(W\(k\)​uk−1,uk\)=‖\(W\(k\)​uk−1\)\+‖≥0a\_\{k\}:=\(W^\{\(k\)\}u\_\{k\-1\},u\_\{k\}\)=\\\|\(W^\{\(k\)\}u\_\{k\-1\}\)\_\{\+\}\\\|\\geq 0and obtain

‖𝐗L,σ2\(N\)‖\+≤sups∈𝕊\+N−1\(∑k=1L\(W\(k\)​uk−1,uk\)L\)L\.\\\|\{\\bf X\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\\\|\_\{\+\}\\leq\\sup\_\{s\\in\\mathbb\{S\}^\{N\-1\}\_\{\+\}\}\\left\(\\dfrac\{\\sum\\limits\_\{k=1\}^\{L\}\(W^\{\(k\)\}u\_\{k\-1\},u\_\{k\}\)\}\{L\}\\right\)^\{L\}\.\(44\)
In formula \([42](https://arxiv.org/html/2608.14638#S2.E42)\) the sequence ofuku\_\{k\}’s is defined by a single inputs∈𝕊\+N−1s\\in\\mathbb\{S\}^\{N\-1\}\_\{\+\}via the recurrence \([40](https://arxiv.org/html/2608.14638#S2.E40)\)\. In order to obtain a computable upper bound, we simply ‘forget’ about the dependencies betweenuku\_\{k\}’s, which can only increase the supremum\. This observation gives us an upper bound

‖𝐗L,σ2\(N\)‖\+≤\(1L​supuk∈𝕊\+nk\+1−1,k=0,1,…,L∑k=1L\(W\(k\)​uk−1,uk\)\)L\.\\\|\{\\bf X\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\\\|\_\{\+\}\\leq\\Bigg\(\\dfrac\{1\}\{L\}\\sup\_\{\\begin\{subarray\}\{c\}u\_\{k\}\\in\\mathbb\{S\}^\{n\_\{k\+1\}\-1\}\_\{\+\},\\\\ k=0,1,\\ldots,L\\end\{subarray\}\}\\quad\\sum\\limits\_\{k=1\}^\{L\}\(W^\{\(k\)\}u\_\{k\-1\},u\_\{k\}\)\\Bigg\)^\{L\}\.\(45\)
Denote

Pu0,u1,…,uL:=∑k=1L\(W\(k\)​uk−1,uk\)\.P\_\{u\_\{0\},u\_\{1\},\\ldots,u\_\{L\}\}:=\\sum\\limits\_\{k=1\}^\{L\}\(W^\{\(k\)\}u\_\{k\-1\},u\_\{k\}\)\.\(47\)Since the matricesW\(k\)W^\{\(k\)\}are independent and have independent Gaussian entries,Pu0,u1,…,uLP\_\{u\_\{0\},u\_\{1\},\\ldots,u\_\{L\}\}is a Gaussian random variable for any choice of\(u0,u1,…,uL\)∈𝕊\+n1−1×𝕊\+n2−1×…×𝕊\+nL\+1−1\(u\_\{0\},u\_\{1\},\\ldots,u\_\{L\}\)\\in\\mathbb\{S\}^\{n\_\{1\}\-1\}\_\{\+\}\\times\\mathbb\{S\}^\{n\_\{2\}\-1\}\_\{\+\}\\times\\ldots\\times\\mathbb\{S\}^\{n\_\{L\+1\}\-1\}\_\{\+\}\. Moreover,\{Pu0,u1,…,uL\}\(u0,…,uL\)∈𝕊\+n1−1×…×𝕊\+nL\+1−1\\\{P\_\{u\_\{0\},u\_\{1\},\\ldots,u\_\{L\}\}\\\}\_\{\(u\_\{0\},\\ldots,u\_\{L\}\)\\in\\mathbb\{S\}^\{n\_\{1\}\-1\}\_\{\+\}\\times\\ldots\\times\\mathbb\{S\}^\{n\_\{L\+1\}\-1\}\_\{\+\}\}is an almost surely bounded Gaussian process indexed by the set𝕊\+n1−1×…×𝕊\+nL\+1−1\\mathbb\{S\}^\{n\_\{1\}\-1\}\_\{\+\}\\times\\ldots\\times\\mathbb\{S\}^\{n\_\{L\+1\}\-1\}\_\{\+\}\.

In conclusion, we bounded‖𝐗L,σ2\(N\)‖\+\\\|\{\\bf X\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\\\|\_\{\+\}via supremum of certain Gaussian process\. In the following steps we will first estimate the expectation of that supremum, and then establish bounds on probability of deviation from expected value\. However, even expected value of supremum of such Gaussian process is impossible to compute directly, and that is where Sudakov\-Fernique comparison inequality is applied\.

### 2\.4Step 2: Majorizing Gaussian process

For the sake of brevity, let us denote𝐮:=\(u0,u1,…,uL\)\{\\bf u\}:=\(u\_\{0\},u\_\{1\},\\ldots,u\_\{L\}\)and𝐒\+:=𝕊\+n1−1×𝕊\+n2−1×…×𝕊\+nL\+1−1\{\\bf S\}\_\{\+\}:=\\mathbb\{S\}^\{n\_\{1\}\-1\}\_\{\+\}\\times\\mathbb\{S\}^\{n\_\{2\}\-1\}\_\{\+\}\\times\\ldots\\times\\mathbb\{S\}^\{n\_\{L\+1\}\-1\}\_\{\+\}\. We have an upper bound for‖𝐗L,σ2\(N\)‖\+\\\|\{\\bf X\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\\\|\_\{\+\}:

‖XL,σ2\(N\)‖\+≤\(1L​sup𝐮∈𝐒\+P𝐮\)L,P𝐮=∑k=1L\(W\(k\)​uk−1,uk\)\.\\\|X\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\\\|\_\{\+\}\\leq\\Bigg\(\\dfrac\{1\}\{L\}\\sup\_\{\{\\bf u\}\\in\{\\bf S\}\_\{\+\}\}P\_\{\\bf u\}\\Bigg\)^\{L\},\\qquad P\_\{\\bf u\}=\\sum\\limits\_\{k=1\}^\{L\}\(W^\{\(k\)\}u\_\{k\-1\},u\_\{k\}\)\.\(48\)The goal of this step is to obtain an upper bound for𝔼​\{sup𝐮∈𝐒\+P𝐮\}\\mathbb\{E\}\\left\\\{\\sup\\limits\_\{\{\\bf u\}\\in\{\\bf S\}\_\{\+\}\}P\_\{\\bf u\}\\right\\\}using Sudakov\-Fernique inequality\.

Let us construct a majorizing Gaussian process\. Consider independent Gaussian vectorsg0,g1,…,gLg\_\{0\},g\_\{1\},\\ldots,g\_\{L\}such that eachgk∈ℝnk\+1g\_\{k\}\\in\\mathbb\{R\}^\{n\_\{k\+1\}\}has i\.i\.d\. entries distributed asN​\(0,1\)N\(0,1\)\. Denote

Q𝐮=Qu0,u1,…,uL:=σ2n1​\(g0,u0\)\+∑k=1L−1σ2nk\+σ2nk\+1​\(gk,uk\)\+σ2nL​\(gL,uL\)\.Q\_\{\\bf u\}=Q\_\{u\_\{0\},u\_\{1\},\\ldots,u\_\{L\}\}:=\\sqrt\{\\dfrac\{\\sigma^\{2\}\}\{n\_\{1\}\}\}\(g\_\{0\},u\_\{0\}\)\+\\sum\_\{k=1\}^\{L\-1\}\\sqrt\{\\dfrac\{\\sigma^\{2\}\}\{n\_\{k\}\}\+\\dfrac\{\\sigma^\{2\}\}\{n\_\{k\+1\}\}\}\(g\_\{k\},u\_\{k\}\)\+\\sqrt\{\\dfrac\{\\sigma^\{2\}\}\{n\_\{L\}\}\}\(g\_\{L\},u\_\{L\}\)\.\(49\)One can easily check that\{Q𝐮\}𝐮∈𝐒\+\\\{Q\_\{\\bf u\}\\\}\_\{\{\\bf u\}\\in\{\\bf S\}\_\{\+\}\}is an almost surely bounded Gaussian process\. In order to check majorization condition \([29](https://arxiv.org/html/2608.14638#S2.E29)\), we need to compute𝔼​\{\(P𝐮−P𝐮′\)2\}\\mathbb\{E\}\\\{\(P\_\{\\bf u\}\-P\_\{\\bf u^\{\\prime\}\}\)^\{2\}\\\}and𝔼​\{\(Q𝐮−Q𝐮′\)2\}\\mathbb\{E\}\\\{\(Q\_\{\\bf u\}\-Q\_\{\\bf u^\{\\prime\}\}\)^\{2\}\\\}for arbitrary𝐮,𝐮′∈𝐒\+\{\\bf u\},\{\\bf u^\{\\prime\}\}\\in\{\\bf S\}\_\{\+\}\.

Observe thatP𝐮−P𝐮′P\_\{\\bf u\}\-P\_\{\\bf u^\{\\prime\}\}is a centered random variable, which means that𝔼​\{\(P𝐮−P𝐮′\)2\}\\mathbb\{E\}\\\{\(P\_\{\\bf u\}\-P\_\{\\bf u^\{\\prime\}\}\)^\{2\}\\\}is its variance\. Then independence of the matricesW\(l\)W^\{\(l\)\}implies that

𝔼​\{\(P𝐮−P𝐮′\)2\}=∑k=1L𝔼​\{\(\(W\(k\)​uk−1,uk\)−\(W\(k\)​uk−1′,uk′\)\)2\}==∑l=1L\(𝔼​\{\(W\(k\)​uk−1,uk\)2\}−2​𝔼​\{\(W\(k\)​uk−1,uk\)​\(W\(k\)​uk−1′,uk′\)\}\+𝔼​\{\(W\(k\)​uk−1′,uk′\)2\}\)\\mathbb\{E\}\\\{\(P\_\{\\bf u\}\-P\_\{\\bf u^\{\\prime\}\}\)^\{2\}\\\}=\\sum\_\{k=1\}^\{L\}\\mathbb\{E\}\\\{\\bigl\(\(W^\{\(k\)\}u\_\{k\-1\},u\_\{k\}\)\-\(W^\{\(k\)\}u\_\{k\-1\}^\{\\prime\},u\_\{k\}^\{\\prime\}\)\\bigr\)^\{2\}\\\}=\\\\ =\\sum\_\{l=1\}^\{L\}\\Bigl\(\\mathbb\{E\}\\\{\(W^\{\(k\)\}u\_\{k\-1\},u\_\{k\}\)^\{2\}\\\}\-2\\mathbb\{E\}\\\{\(W^\{\(k\)\}u\_\{k\-1\},u\_\{k\}\)\(W^\{\(k\)\}u\_\{k\-1\}^\{\\prime\},u\_\{k\}^\{\\prime\}\)\\\}\+\\mathbb\{E\}\\\{\(W^\{\(k\)\}u^\{\\prime\}\_\{k\-1\},u^\{\\prime\}\_\{k\}\)^\{2\}\\\}\\Bigr\)\(50\)We can writeW\(k\)=σ2nk​G\(k\)W^\{\(k\)\}=\\sqrt\{\\dfrac\{\\sigma^\{2\}\}\{n\_\{k\}\}\}G^\{\(k\)\}, whereG\(k\)∈ℝnk\+1×nkG^\{\(k\)\}\\in\\mathbb\{R\}^\{n\_\{k\+1\}\\times n\_\{k\}\}has i\.i\.d\. entriesN​\(0,1\)N\(0,1\)\. Straightforward computation shows that for a Gaussian matrixGGwith i\.i\.d\. entriesN​\(0,1\)N\(0,1\)and arbitrary deterministic vectorsa,b,c,da,b,c,dthe following identity holds:

𝔼​\{\(G​a,b\)​\(G​c,d\)\}=\(a,c\)​\(b,d\),\\mathbb\{E\}\\\{\(Ga,b\)\(Gc,d\)\\\}=\(a,c\)\(b,d\),\(51\)which implies

𝔼​\{\(P𝐮−P𝐮′\)2\}=∑k=1Lσ2nk​\(‖uk−1‖2​‖uk‖2−2​\(uk−1,uk−1′\)​\(uk,uk′\)\+‖uk−1′‖2​‖uk′‖2\)\\mathbb\{E\}\\\{\(P\_\{\\bf u\}\-P\_\{\\bf u^\{\\prime\}\}\)^\{2\}\\\}=\\sum\_\{k=1\}^\{L\}\\dfrac\{\\sigma^\{2\}\}\{n\_\{k\}\}\(\\\|u\_\{k\-1\}\\\|^\{2\}\\\|u\_\{k\}\\\|^\{2\}\-2\(u\_\{k\-1\},u\_\{k\-1\}^\{\\prime\}\)\(u\_\{k\},u\_\{k\}^\{\\prime\}\)\+\\\|u\_\{k\-1\}^\{\\prime\}\\\|^\{2\}\\\|u\_\{k\}^\{\\prime\}\\\|^\{2\}\)\(52\)Taking into account‖uk‖=‖uk′‖=1\\\|u\_\{k\}\\\|=\\\|u\_\{k\}^\{\\prime\}\\\|=1, we obtain final formula for𝔼​\{\(P𝐮−P𝐮′\)2\}\\mathbb\{E\}\\\{\(P\_\{\\bf u\}\-P\_\{\\bf u^\{\\prime\}\}\)^\{2\}\\\}:

𝔼​\{\(P𝐮−P𝐮′\)2\}=2​∑k=1Lσ2nk​\(1−\(uk−1,uk−1′\)​\(uk,uk′\)\)\\mathbb\{E\}\\\{\(P\_\{\\bf u\}\-P\_\{\\bf u^\{\\prime\}\}\)^\{2\}\\\}=2\\sum\_\{k=1\}^\{L\}\\dfrac\{\\sigma^\{2\}\}\{n\_\{k\}\}\\bigl\(1\-\(u\_\{k\-1\},u\_\{k\-1\}^\{\\prime\}\)\(u\_\{k\},u\_\{k\}^\{\\prime\}\)\\bigr\)\(53\)Note that\(uk−1,uk−1′\)≤1\(u\_\{k\-1\},u\_\{k\-1\}^\{\\prime\}\)\\leq 1,\(uk,uk′\)≤1\(u\_\{k\},u\_\{k\}^\{\\prime\}\)\\leq 1by Cauchy\-Schwarz\. A trivial inequality

1−x​y≤1−x\+1−y​for​x,y≤11\-xy\\leq 1\-x\+1\-y\\text\{\\quad for \}x,y\\leq 1\(54\)gives us an upper bound for𝔼​\{\(P𝐮−P𝐮′\)2\}\\mathbb\{E\}\\\{\(P\_\{\\bf u\}\-P\_\{\\bf u^\{\\prime\}\}\)^\{2\}\\\}:

𝔼​\{\(P𝐮−P𝐮′\)2\}≤2​∑k=1Lσ2nl​\(\(1−\(uk−1,uk−1′\)\)\+\(1−\(uk,uk′\)\)\)==2​σ2n1​\(1−\(u0,u0′\)\)\+2​∑k=1L−1\(σ2nk\+σ2nk\+1\)​\(1−\(uk,uk′\)\)\+2​σ2nL​\(1−\(uL,uL′\)\)\\mathbb\{E\}\\\{\(P\_\{\\bf u\}\-P\_\{\\bf u^\{\\prime\}\}\)^\{2\}\\\}\\leq 2\\sum\_\{k=1\}^\{L\}\\dfrac\{\\sigma^\{2\}\}\{n\_\{l\}\}\\Bigl\(\\bigl\(1\-\(u\_\{k\-1\},u\_\{k\-1\}^\{\\prime\}\)\\big\)\+\\bigl\(1\-\(u\_\{k\},u\_\{k\}^\{\\prime\}\)\\bigr\)\\Bigr\)=\\\\ =2\\dfrac\{\\sigma^\{2\}\}\{n\_\{1\}\}\(1\-\(u\_\{0\},u\_\{0\}^\{\\prime\}\)\)\+2\\sum\_\{k=1\}^\{L\-1\}\\left\(\\dfrac\{\\sigma^\{2\}\}\{n\_\{k\}\}\+\\dfrac\{\\sigma^\{2\}\}\{n\_\{k\+1\}\}\\right\)\\bigl\(1\-\(u\_\{k\},u\_\{k\}^\{\\prime\}\)\\bigr\)\+2\\dfrac\{\\sigma^\{2\}\}\{n\_\{L\}\}\\bigl\(1\-\(u\_\{L\},u\_\{L\}^\{\\prime\}\)\\bigr\)\(55\)Let us move on to𝔼​\{\(Q𝐮−Q𝐮′\)2\}\\mathbb\{E\}\\\{\(Q\_\{\\bf u\}\-Q\_\{\\bf u^\{\\prime\}\}\)^\{2\}\\\}\. Due to the independence ofgkg\_\{k\}we obtain

𝔼​\{\(Q𝐮−Q𝐮′\)2\}=σ2n1​𝔼​\{\(g0,u0−u0′\)2\}\+\+∑k=1L−1\(σ2nk\+σ2nk\+1\)​𝔼​\{\(gk,uk−uk′\)2\}\+σ2nL​𝔼​\{\(gL,uL−uL′\)2\}\.\\mathbb\{E\}\\\{\(Q\_\{\\bf u\}\-Q\_\{\\bf u^\{\\prime\}\}\)^\{2\}\\\}=\\dfrac\{\\sigma^\{2\}\}\{n\_\{1\}\}\\mathbb\{E\}\\\{\(g\_\{0\},u\_\{0\}\-u\_\{0\}^\{\\prime\}\)^\{2\}\\\}\+\\\\ \+\\sum\_\{k=1\}^\{L\-1\}\\left\(\\dfrac\{\\sigma^\{2\}\}\{n\_\{k\}\}\+\\dfrac\{\\sigma^\{2\}\}\{n\_\{k\+1\}\}\\right\)\\mathbb\{E\}\\\{\\bigl\(g\_\{k\},u\_\{k\}\-u\_\{k\}^\{\\prime\}\\bigr\)^\{2\}\\\}\+\\dfrac\{\\sigma^\{2\}\}\{n\_\{L\}\}\\mathbb\{E\}\\\{\(g\_\{L\},u\_\{L\}\-u\_\{L\}^\{\\prime\}\)^\{2\}\\\}\.\(56\)For a Gaussian vectorggwith i\.i\.d\. entriesN​\(0,1\)N\(0,1\)and an arbitrary deterministic vectorvvone have

𝔼​\{\(g,v\)2\}=‖v‖2\.\\mathbb\{E\}\\\{\(g,v\)^\{2\}\\\}=\\\|v\\\|^\{2\}\.\(57\)Recall that‖uk‖=‖uk′‖=1\\\|u\_\{k\}\\\|=\\\|u\_\{k\}^\{\\prime\}\\\|=1, which means that‖uk−uk′‖2=2​\(1−\(uk,uk′\)\)\\\|u\_\{k\}\-u\_\{k\}^\{\\prime\}\\\|^\{2\}=2\(1\-\(u\_\{k\},u\_\{k\}^\{\\prime\}\)\)\. Combining this with \([57](https://arxiv.org/html/2608.14638#S2.E57)\), we obtain

𝔼​\{\(Q𝐮−Q𝐮′\)2\}=2​σ2n1​\(1−\(u0,u0′\)\)\+2​∑k=1L−1\(σ2nk\+σ2nk\+1\)​\(1−\(uk,uk′\)\)\+2​σ2nL​\(1−\(uL,uL′\)\)\\mathbb\{E\}\\\{\(Q\_\{\\bf u\}\-Q\_\{\\bf u^\{\\prime\}\}\)^\{2\}\\\}=2\\dfrac\{\\sigma^\{2\}\}\{n\_\{1\}\}\(1\-\(u\_\{0\},u\_\{0\}^\{\\prime\}\)\)\+2\\sum\_\{k=1\}^\{L\-1\}\\left\(\\dfrac\{\\sigma^\{2\}\}\{n\_\{k\}\}\+\\dfrac\{\\sigma^\{2\}\}\{n\_\{k\+1\}\}\\right\)\\bigl\(1\-\(u\_\{k\},u\_\{k\}^\{\\prime\}\)\\bigr\)\+2\\dfrac\{\\sigma^\{2\}\}\{n\_\{L\}\}\\bigl\(1\-\(u\_\{L\},u\_\{L\}^\{\\prime\}\)\\bigr\)\(58\)The upper bound \([55](https://arxiv.org/html/2608.14638#S2.E55)\) and the identity \([58](https://arxiv.org/html/2608.14638#S2.E58)\) together yield that

𝔼​\{\(P𝐮−P𝐮′\)2\}≤𝔼​\{\(Q𝐮−Q𝐮′\)2\},\\mathbb\{E\}\\\{\(P\_\{\\bf u\}\-P\_\{\\bf u^\{\\prime\}\}\)^\{2\}\\\}\\leq\\mathbb\{E\}\\\{\(Q\_\{\\bf u\}\-Q\_\{\\bf u^\{\\prime\}\}\)^\{2\}\\\},\(59\)i\.e\. Gaussian processQ𝐮Q\_\{\\bf u\}majorizesP𝐮P\_\{\\bf u\}in terms of \([29](https://arxiv.org/html/2608.14638#S2.E29)\)\.

### 2\.5Step 3: Upper bound on expected supremum

In Step 2 we checked that Gaussian processesP𝐮P\_\{\\bf u\}andQ𝐮Q\_\{\\bf u\}satisfy condition \([29](https://arxiv.org/html/2608.14638#S2.E29)\)\. Then Sudakov\-Fernique inequality yields that

𝔼​\{sup𝐮∈𝐒\+P𝐮\}≤𝔼​\{sup𝐮∈𝐒\+Q𝐮\}\.\\mathbb\{E\}\\\{\\sup\_\{\{\\bf u\}\\in\{\\bf S\}\_\{\+\}\}P\_\{\\bf u\}\\\}\\leq\\mathbb\{E\}\\\{\\sup\_\{\{\\bf u\}\\in\{\\bf S\}\_\{\+\}\}Q\_\{\\bf u\}\\\}\.\(60\)We are left to compute𝔼​\{sup𝐮∈𝐒\+Q𝐮\}\\mathbb\{E\}\\\{\\sup\\limits\_\{\{\\bf u\}\\in\{\\bf S\}\_\{\+\}\}Q\_\{\\bf u\}\\\}to complete the upper bound for𝔼​\{sup𝐮∈𝐒\+P𝐮\}\\mathbb\{E\}\\\{\\sup\\limits\_\{\{\\bf u\}\\in\{\\bf S\}\_\{\+\}\}P\_\{\\bf u\}\\\}\. Observe that for any vectorv∈ℝnv\\in\\mathbb\{R\}^\{n\}one can compute the following supremum:

supu∈S\+n−1\(v,u\)=‖v\+‖\.\\sup\_\{u\\in S^\{n\-1\}\_\{\+\}\}\(v,u\)=\\\|v\_\{\+\}\\\|\.\(61\)This formula together with the definition ofQ𝐮Q\_\{\\bf u\}gives us

𝔼​\{sup𝐮∈𝐒\+Q𝐮\}=σ2n1​𝔼​\{‖\(g0\)\+‖\}\+∑k=1L−1σ2nk\+σ2nk\+1​𝔼​\{‖\(gk\)\+‖\}\+σ2nL​𝔼​\{‖\(gL\)\+‖\}\\mathbb\{E\}\\\{\\sup\\limits\_\{\{\\bf u\}\\in\{\\bf S\}\_\{\+\}\}Q\_\{\\bf u\}\\\}=\\sqrt\{\\dfrac\{\\sigma^\{2\}\}\{n\_\{1\}\}\}\\mathbb\{E\}\\\{\\\|\(g\_\{0\}\)\_\{\+\}\\\|\\\}\+\\sum\_\{k=1\}^\{L\-1\}\\sqrt\{\\dfrac\{\\sigma^\{2\}\}\{n\_\{k\}\}\+\\dfrac\{\\sigma^\{2\}\}\{n\_\{k\+1\}\}\}\\mathbb\{E\}\\\{\\\|\(g\_\{k\}\)\_\{\+\}\\\|\\\}\+\\sqrt\{\\dfrac\{\\sigma^\{2\}\}\{n\_\{L\}\}\}\\mathbb\{E\}\\\{\\\|\(g\_\{L\}\)\_\{\+\}\\\|\\\}\(62\)For a Gaussian vectorg=\(g\(1\),…,g\(n\)\)∈ℝng=\(g^\{\(1\)\},\\ldots,g^\{\(n\)\}\)\\in\\mathbb\{R\}^\{n\}with i\.i\.d\. entriesN​\(0,1\)N\(0,1\)one can use Cauchy\-Schwarz inequality to obtain

𝔼​\{‖g\+‖\}≤\(𝔼​\{‖g\+‖2\}\)1/2=\(∑j=1n𝔼​\{\(g\+\(j\)\)2\}\)1/2=n2\.\\mathbb\{E\}\\\{\\\|g\_\{\+\}\\\|\\\}\\leq\(\\mathbb\{E\}\\\{\\\|g\_\{\+\}\\\|^\{2\}\\\}\)^\{1/2\}=\\left\(\\sum\_\{j=1\}^\{n\}\\mathbb\{E\}\\\{\(g^\{\(j\)\}\_\{\+\}\)^\{2\}\\\}\\right\)^\{1/2\}=\\sqrt\{\\dfrac\{n\}\{2\}\}\.\(63\)Using this bound forg0,g1,…,gLg\_\{0\},g\_\{1\},\\ldots,g\_\{L\}and recalling thatn1=Nn\_\{1\}=N,nL\+1=Nn\_\{L\+1\}=N,nk=ck​\(N\)⋅Nn\_\{k\}=c\_\{k\}\(N\)\\cdot N, we obtain that \([62](https://arxiv.org/html/2608.14638#S2.E62)\) together with \([60](https://arxiv.org/html/2608.14638#S2.E60)\) implies

1L​𝔼​\{sup𝐮∈𝐒\+P𝐮\}≤σ⋅1\+∑k=1L−11\+ck\+1ck\+1cL2​L=σβ0\(L\)\.\\dfrac\{1\}\{L\}\\mathbb\{E\}\\\{\\sup\\limits\_\{\{\\bf u\}\\in\{\\bf S\}\_\{\+\}\}P\_\{\\bf u\}\\\}\\leq\\sigma\\cdot\\dfrac\{1\+\\sum\\limits\_\{k=1\}^\{L\-1\}\\sqrt\{1\+\\dfrac\{c\_\{k\+1\}\}\{c\_\{k\}\}\}\+\\sqrt\{\\dfrac\{1\}\{c\_\{L\}\}\}\}\{\\sqrt\{2\}\\,L\}=\\dfrac\{\\sigma\}\{\\beta\_\{0\}^\{\(L\)\}\}\.\(64\)

### 2\.6Step 4: Gaussian concentration

We are left to prove that there existsc~\>0\\widetilde\{c\}\>0such that

ℙ​\{1L​sup𝐮∈𝐒\+P𝐮−𝔼​\{1L​sup𝐮∈𝐒\+P𝐮\}≥ε\}≤e−c~​N​L\.\\mathbb\{P\}\\left\\\{\\dfrac\{1\}\{L\}\\sup\\limits\_\{\{\\bf u\}\\in\{\\bf S\}\_\{\+\}\}P\_\{\\bf u\}\-\\mathbb\{E\}\\Bigl\\\{\\dfrac\{1\}\{L\}\\sup\\limits\_\{\{\\bf u\}\\in\{\\bf S\}\_\{\+\}\}P\_\{\\bf u\}\\Bigr\\\}\\geq\\varepsilon\\right\\\}\\leq e^\{\-\\widetilde\{c\}NL\}\.\(65\)
We will use Borell–TIS inequality \(Theorem[2\.2](https://arxiv.org/html/2608.14638#S2.Thmtheorem2)\) to obtain \([65](https://arxiv.org/html/2608.14638#S2.E65)\)\. To this end we need to estimate𝔼​\{P𝐮2\}\\mathbb\{E\}\\\{P^\{2\}\_\{\\bf u\}\\\}\. Similarly to \([50](https://arxiv.org/html/2608.14638#S2.E50)\) and \([52](https://arxiv.org/html/2608.14638#S2.E52)\) we obtain

𝔼​\{P𝐮2\}=∑k=1L𝔼​\{\(W\(k\)​uk−1,uk\)2\}=∑k=1Lσ2nk​‖uk−1‖2​‖uk‖2=SN​for​S=∑k=1Lσ2ck≤L​σ2c\.\\mathbb\{E\}\\\{P^\{2\}\_\{\\bf u\}\\\}=\\sum\_\{k=1\}^\{L\}\\mathbb\{E\}\\\{\\bigl\(W^\{\(k\)\}u\_\{k\-1\},u\_\{k\}\\bigr\)^\{2\}\\\}=\\sum\_\{k=1\}^\{L\}\\dfrac\{\\sigma^\{2\}\}\{n\_\{k\}\}\\\|u\_\{k\-1\}\\\|^\{2\}\\\|u\_\{k\}\\\|^\{2\}=\\dfrac\{S\}\{N\}\\text\{\\quad for \}S=\\sum\_\{k=1\}^\{L\}\\dfrac\{\\sigma^\{2\}\}\{c\_\{k\}\}\\leq L\\dfrac\{\\sigma^\{2\}\}\{c\}\.\(66\)Thus, in terms of Theorem[2\.2](https://arxiv.org/html/2608.14638#S2.Thmtheorem2)M=sup𝐮∈𝐒\+𝔼​\{P𝐮2\}≤L​σ2N​cM=\\sup\\limits\_\{\{\\bf u\}\\in\{\\bf S\}\_\{\+\}\}\\mathbb\{E\}\\\{P^\{2\}\_\{\\bf u\}\\\}\\leq\\dfrac\{L\\sigma^\{2\}\}\{Nc\}\. Applying Theorem[2\.2](https://arxiv.org/html/2608.14638#S2.Thmtheorem2)fort=L​εt=L\\varepsilon, we obtain

ℙ​\(sup𝐮∈𝐒\+P𝐮−𝔼​sup𝐮∈𝐒\+P𝐮≥L​ε\)≤exp⁡\(−N​L​ε2​c2​σ2\),\\mathbb\{P\}\\Big\(\\sup\_\{\{\\bf u\}\\in\{\\bf S\}\_\{\+\}\}P\_\{\\bf u\}\-\\mathbb\{E\}\\sup\_\{\{\\bf u\}\\in\{\\bf S\}\_\{\+\}\}P\_\{\\bf u\}\\geq L\\varepsilon\\Big\)\\leq\\exp\\\!\\left\(\-\\frac\{NL\\varepsilon^\{2\}c\}\{2\\sigma^\{2\}\}\\right\),\(67\)which gives us \([65](https://arxiv.org/html/2608.14638#S2.E65)\) withc~=ε2​c2​σ2\\widetilde\{c\}=\\dfrac\{\\varepsilon^\{2\}c\}\{2\\sigma^\{2\}\}and completes the proof of Theorem[1\.2](https://arxiv.org/html/2608.14638#S1.Thmtheorem2)forα=0\\alpha=0\.

## 3Proof of Theorem[1\.1](https://arxiv.org/html/2608.14638#S1.Thmtheorem1)

Consider input\-output Jacobian𝐉L,σ2\(N\)​\(s\)\{\\bf J\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\)\([8](https://arxiv.org/html/2608.14638#S1.E8)\) of autoencoder𝐗L,σ2\(N\)​\(s\)\{\\bf X\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\(s\)with ReLU activation, Gaussian initialization, and asymptotically proportional layers:

nkN→ck∈\(c;C\)asn→∞,k=2,3,…,L\.\\dfrac\{n\_\{k\}\}\{N\}\\to c\_\{k\}\\in\(c;C\)\\quad\\text\{as $n\\to\\infty$, $k=2,3,\\ldots,L$\}\.\(68\)Sincen1=nL\+1=Nn\_\{1\}=n\_\{L\+1\}=N, we also denotec1=cL\+1=1c\_\{1\}=c\_\{L\+1\}=1for convenience\. Since random matrix𝐉L,σ2\(N\)​\(s\)\{\\bf J\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\)is generally non\-symmetric, its norm is determined by its singular values, i\.e\. by the eigenvalues of𝐌L,σ2\(N\)​\(s\)=\(𝐉L,σ2\(N\)​\(s\)\)T​𝐉L,σ2\(N\)​\(s\)\{\\bf M\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\)=\(\{\\bf J\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\)\)^\{T\}\{\\bf J\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\)\. Denoteμ\+\(N\)​\(s\)\\mu\_\{\+\}^\{\(N\)\}\(s\)the maximal eigenvalue of𝐌L,σ2\(N\)​\(s\)\{\\bf M\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\), then

‖𝐉L,σ2\(N\)​\(s\)‖=μ\+\(N\)​\(s\)\\\|\{\\bf J\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\)\\\|=\\sqrt\{\\mu\_\{\+\}^\{\(N\)\}\(s\)\}\(69\)We are interested in the behavior ofμ\+\(N\)​\(s\)\\mu\_\{\+\}^\{\(N\)\}\(s\)in the double limitlimL→∞limN→∞\\lim\\limits\_\{L\\to\\infty\}\\lim\\limits\_\{N\\to\\infty\}\. It was proved in\[[17](https://arxiv.org/html/2608.14638#bib.bib17)\]and\[[16](https://arxiv.org/html/2608.14638#bib.bib16)\]that for a given sequence of input vectorssN∈ℝNs\_\{N\}\\in\\mathbb\{R\}^\{N\}the empirical spectral distribution of𝐌L,σ2\(N\)​\(sN\)\{\\bf M\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\_\{N\}\)converges to a non\-random probability measureν𝐌L,σ2\\nu\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\.

Theorem[1\.1](https://arxiv.org/html/2608.14638#S1.Thmtheorem1)will follow from the following two facts:

###### Proposition 3\.1\(Right edge of limiting eigenvalue distribution\)

Letμ\+\(∞\)\\mu\_\{\+\}^\{\(\\infty\)\}be the right edge of the limiting eigenvalue distributionsupp​ν𝐌L,σ2\\text\{supp\}\\,\\nu\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}of𝐌L,σ2\(N\)​\(sN\)\{\\bf M\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\_\{N\}\)\. Then:

1. \(i\)μ\+\(∞\)\\mu\_\{\+\}^\{\(\\infty\)\}does not depend on the choice ofsNs\_\{N\}and μ\+\(∞\)=\(σ22\)L⋅m∗−1​\(m∗\+1\)⋅∏k=1L\(2​ck​m∗\+1\),where​m∗≍1L;\\mu\_\{\+\}^\{\(\\infty\)\}=\\left\(\\dfrac\{\\sigma^\{2\}\}\{2\}\\right\)^\{L\}\\cdot m\_\{\*\}^\{\-1\}\(m\_\{\*\}\+1\)\\cdot\\prod\\limits\_\{k=1\}^\{L\}\(2c\_\{k\}m\_\{\*\}\+1\),\\quad\\text\{where \}m\_\{\*\}\\asymp\\dfrac\{1\}\{L\};\(70\)
2. \(ii\)In the infinite\-width limit: limL→∞1L​log⁡μ\+\(∞\)=log⁡\(σ2/2\);\\lim\_\{L\\to\\infty\}\\dfrac\{1\}\{L\}\\log\\mu\_\{\+\}^\{\(\\infty\)\}=\\log\(\\sigma^\{2\}/2\);\(71\)
3. \(iii\)LetσL,crit2\\sigma\_\{L,\\mathrm\{crit\}\}^\{2\}be the value ofσ2\\sigma^\{2\}for whichμ\+\(∞\)=1\\mu\_\{\+\}^\{\(\\infty\)\}=1\. Then in the infinite\-depth limit: limL→∞σL,crit2=2\.\\lim\_\{L\\to\\infty\}\\sigma\_\{L,\\mathrm\{crit\}\}^\{2\}=2\.\(72\)

###### Proposition 3\.2\(Convergence of spectral norm\)

Let\{sN\}N=1∞\\\{s\_\{N\}\\\}\_\{N=1\}^\{\\infty\}be an arbitrary sequence of vectors such thatsN∈ℝNs\_\{N\}\\in\\mathbb\{R\}^\{N\}\. Then

‖𝐉L,σ2\(N\)​\(sN\)‖→μ\+\(∞\)a\.s\. asN→∞\.\\\|\{\\bf J\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\_\{N\}\)\\\|\\to\\sqrt\{\\mu\_\{\+\}^\{\(\\infty\)\}\}\\quad\\text\{a\.s\. as $N\\to\\infty$\}\.\(73\)

### 3\.1Proof of Proposition[3\.1](https://arxiv.org/html/2608.14638#S3.Thmtheorem1)

#### 3\.1\.1Outline of the proof

We will perform the proof in three steps:

- •Step 1:Obtain a functional equation on themoment generating functionof the eigenvalue distribution of𝐌L,σ2\(N\)​\(s\)\{\\bf M\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\)in infinite\-width limit\.
- •Step 2:Derive formula \([70](https://arxiv.org/html/2608.14638#S3.E70)\) right the edgeμ\+\(∞\)\\mu\_\{\+\}^\{\(\\infty\)\}of spectrum of𝐌L,σ2\(N\)​\(s\)\{\\bf M\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\)using the equation obtained inStep 1\.
- •Step 3:Obtain the limits \([71](https://arxiv.org/html/2608.14638#S3.E71)\) and \([72](https://arxiv.org/html/2608.14638#S3.E72)\) using \([70](https://arxiv.org/html/2608.14638#S3.E70)\)\.

The equation mentioned inStep 1was earlier obtained in\[[19](https://arxiv.org/html/2608.14638#bib.bib19)\]in the partial case of DNN with layers of the same width and under certain simplifications\. Namely, one can use chain rule to represent I/O Jacobian in a form

𝐉L,σ2\(N\)​\(s\)=D\(L\)​W\(L\)​…​D\(2\)​W\(2\)​D\(1\)​W\(1\),\{\\bf J\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\(s\)=D^\{\(L\)\}W^\{\(L\)\}\\ldots D^\{\(2\)\}W^\{\(2\)\}D^\{\(1\)\}W^\{\(1\)\},\(75\)whereD\(k\)D^\{\(k\)\}arenk\+1×nk\+1n\_\{k\+1\}\\times n\_\{k\+1\}diagonal matrices havingλ′​\(\(W\(k\)​s\(k−1\)\)j\+bj\(k\)\)\\lambda^\{\\prime\}\(\(W^\{\(k\)\}s^\{\(k\-1\)\}\)\_\{j\}\+b^\{\(k\)\}\_\{j\}\)on diagonals\. In\[[19](https://arxiv.org/html/2608.14638#bib.bib19)\]matricesW\(k\)W^\{\(k\)\}andD\(j\)D^\{\(j\)\}were treated as independent\. In the case of ReLU activation one can consider a matrix

𝐉~L,σ2\(N\)=D~\(L\)​W\(L\)​…​D~\(2\)​W\(2\)​D~\(1\)​W\(1\),\\widetilde\{\{\\bf J\}\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}=\\widetilde\{D\}^\{\(L\)\}W^\{\(L\)\}\\ldots\\widetilde\{D\}^\{\(2\)\}W^\{\(2\)\}\\widetilde\{D\}^\{\(1\)\}W^\{\(1\)\},\(76\)whereD~\(k\)\\widetilde\{D\}^\{\(k\)\}arenk\+1×nk\+1n\_\{k\+1\}\\times n\_\{k\+1\}diagonal matrices having i\.i\.d\.Bernoulli​\(12\)\\mathrm\{Bernoulli\}\\,\(\\frac\{1\}\{2\}\)variables independent ofW\(j\)W^\{\(j\)\}on diagonals\. The results of\[[17](https://arxiv.org/html/2608.14638#bib.bib17),[16](https://arxiv.org/html/2608.14638#bib.bib16)\]imply that in ReLU activation case, in the infinite\-width limit the densities of singular value distribution of𝐉L,σ2\(N\)​\(s\)\{\\bf J\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\(s\)and𝐉~L,σ2\(N\)\\widetilde\{\{\\bf J\}\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}are the same \(e\.g\.,\[[16](https://arxiv.org/html/2608.14638#bib.bib16), Theorem 2\.5\]\)\. In\[[11](https://arxiv.org/html/2608.14638#bib.bib11), Proposition 2\]it was proved that random matrices𝐌L,σ2\(N\)​\(s\)=\(𝐉L,σ2\(N\)​\(s\)\)T​𝐉L,σ2\(N\)​\(s\)\{\\bf M\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\(s\)=\(\{\\bf J\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\(s\)\)^\{T\}\{\\bf J\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\(s\)and𝐌~L,σ2\(N\)=\(𝐉~L,σ2\(N\)\)T​𝐉~L,σ2\(N\)\\widetilde\{\{\\bf M\}\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}=\(\\widetilde\{\{\\bf J\}\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\)^\{T\}\\widetilde\{\{\\bf J\}\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}are equal in distribution for anys∈ℝNs\\in\\mathbb\{R\}^\{N\}\. Both these results justify the independence assumption used in\[[19](https://arxiv.org/html/2608.14638#bib.bib19)\]\.

It was also mentioned in\[[16](https://arxiv.org/html/2608.14638#bib.bib16)\]that the results of the paper can be used to generalize the equation on moment generating function in\[[19](https://arxiv.org/html/2608.14638#bib.bib19)\]for the case of the layers of proportional widths\. Here inStep 1we finally derive generalized version of such equation in a closed form \(for proportional layers and ReLU activation\)\.

Step 2is the main technical step of the proof\. In case of ReLU networks, in\[[19](https://arxiv.org/html/2608.14638#bib.bib19)\]the first two moments of eigenvalue distribution of𝐌~L,σ2\(N\)\\widetilde\{\{\\bf M\}\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}were derived in order to study classical EoC for DNNs\. Dealing with the right edge of spectrum requires more subtle analysis\. In\[[20](https://arxiv.org/html/2608.14638#bib.bib20)\]an asymptotic formula forμ\+\(∞\)\\mu\_\{\+\}^\{\(\\infty\)\}\(asL→∞L\\to\\infty\) was derived in the case of layers of the same width\. Here in step 2 we obtain an asymptotic formula forμ\+\(∞\)\\mu\_\{\+\}^\{\(\\infty\)\}for arbitrary proportional widths of layers\. Moreover, we derive an exact expression forμ\+\(∞\)\\mu\_\{\+\}^\{\(\\infty\)\}for the layers of the same width \(see Remark[3\.3](https://arxiv.org/html/2608.14638#S3.Thmremark3)\)\. The main complication consists in the fact that the moment generating function of the eigenvalue distribution can be found only implicitly\.

InStep 3we use the derived formula for the right edgeμ\+\(∞\)\\mu\_\{\+\}^\{\(\\infty\)\}to obtain local EoC in the limitL→∞L\\to\\infty\.

#### 3\.1\.2Step 1: Equation on moment generating function\.

We start by introducing some standard notions from Random Matrix Theory concerning eigenvalue distributions\. Consider a sequence of randomN×NN\\times Nreal symmetric matrices𝐊\(N\)\{\\bf K\}^\{\(N\)\}\. Denote\{λk\(N\)\}k=1N\\\{\\lambda^\{\(N\)\}\_\{k\}\\\}\_\{k=1\}^\{N\}the eigenvalues of𝐊\(N\)\{\\bf K\}^\{\(N\)\}\.Normalized counting measure \(NCM\)of𝐊\(N\)\{\\bf K\}^\{\(N\)\}is a random measure defined by

ν𝐊\(N\)=1N​∑k=1Nδλk\(N\)\.\\nu\_\{\\scriptscriptstyle\{\\bf K\}^\{\(N\)\}\}=\\dfrac\{1\}\{N\}\\sum\_\{k=1\}^\{N\}\\delta\_\{\\scriptscriptstyle\\lambda^\{\(N\)\}\_\{k\}\}\.\(77\)Ifν𝐊\(N\)\\nu\_\{\\scriptscriptstyle\{\\bf K\}^\{\(N\)\}\}converges almost surely to a certain non\-random measureν𝐊\\nu\_\{\\scriptscriptstyle\\bf K\}in a weak sense, we callν𝐊\\nu\_\{\\scriptscriptstyle\\bf K\}limiting NCMof𝐊\(N\)\{\\bf K\}^\{\(N\)\}\.

Moment generating functionof𝐊\(N\)\{\\bf K\}^\{\(N\)\}is

m𝐊​\(z\)=∑k=1∞ak​zk,m\_\{\\scriptscriptstyle\{\\bf K\}\}\(z\)=\\sum\_\{k=1\}^\{\\infty\}a\_\{k\}z^\{k\},\(78\)whereaka\_\{k\}iskk\-th moment of the measureν𝐊\\nu\_\{\\scriptscriptstyle\\bf K\}:

ak=∫−∞∞λk​𝑑ν𝐊​\(λ\)\.a\_\{k\}=\\int\_\{\-\\infty\}^\{\\infty\}\\lambda^\{k\}\\,d\\nu\_\{\\scriptscriptstyle\\bf K\}\(\\lambda\)\.\(79\)Also denotez𝐊​\(m\)z\_\{\\scriptscriptstyle\{\\bf K\}\}\(m\)the functional inverse ofm𝐊​\(z\)m\_\{\\scriptscriptstyle\{\\bf K\}\}\(z\)\.

Recall that𝐌L,σ2\(N\)​\(s\)=\(𝐉L,σ2\(N\)​\(s\)\)T​𝐉L,σ2\(N\)​\(s\)\{\\bf M\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\(s\)=\(\{\\bf J\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\)\)^\{T\}\{\\bf J\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\(s\), where𝐉L,σ2\(N\)​\(s\)\{\\bf J\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\(s\)is I/O Jacobian of autoencoder functionXL,σ2\(N\)​\(s\)X\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\)\. It was shown in\[[17](https://arxiv.org/html/2608.14638#bib.bib17)\]that NCMν𝐌L,σ2\(N\)\\nu\_\{\\scriptscriptstyle\{\\bf M\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\}of𝐌L,σ2\(N\)​\(s\)\{\\bf M\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\(s\)converges almost surely to certain a non\-random measureν𝐌L,σ2\\nu\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}, i\.e\. the limiting NCMν𝐌L,σ2\\nu\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}of𝐌L,σ2\(N\)​\(s\)\{\\bf M\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\(s\)exists\.

The equation onm𝐌L,σ2m\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}when𝐗L,σ2\(N\)​\(s\)\{\\bf X\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\)is a DNN with square layer matrices \(n1=n2=…=nL\+1n\_\{1\}=n\_\{2\}=\\ldots=n\_\{L\+1\}\) was obtained in\[[19](https://arxiv.org/html/2608.14638#bib.bib19)\]under the conjecture of independence ofW\(k\)W^\{\(k\)\}andD\(j\)D^\{\(j\)\}\(see \([75](https://arxiv.org/html/2608.14638#S3.E75)\) and \([76](https://arxiv.org/html/2608.14638#S3.E76)\)\)\. It was later proven in\[[16](https://arxiv.org/html/2608.14638#bib.bib16)\]that we can consider𝐉L,σ2\(N\)\{\\bf J\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}defined as in \([75](https://arxiv.org/html/2608.14638#S3.E75)\) but with certainD\(j\)D^\{\(j\)\}independent ofW\(k\)W^\{\(k\)\}, and the eigenvalue distribution of𝐌L,σ2\(N\)​\(s\)=\(𝐉L,σ2\(N\)​\(s\)\)T​𝐉L,σ2\(N\)​\(s\)\{\\bf M\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\(s\)=\(\{\\bf J\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\)\)^\{T\}\{\\bf J\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\(s\)in infinite\-width limit remains the same, which justifies the result of\[[19](https://arxiv.org/html/2608.14638#bib.bib19)\]\. In the case of ReLU activation, after such replacement we obtain𝐌~L,σ2\(N\)=\(𝐉~L,σ2\(N\)\)T​𝐉~L,σ2\(N\)\\widetilde\{\{\\bf M\}\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}=\(\\widetilde\{\{\\bf J\}\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\)^\{T\}\\widetilde\{\{\\bf J\}\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}with𝐉~L,σ2\(N\)\\widetilde\{\{\\bf J\}\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}defined in \([76](https://arxiv.org/html/2608.14638#S3.E76)\), i\.e\.ν𝐌L,σ2=ν𝐌~L,σ2\\nu\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}=\\nu\_\{\\scriptscriptstyle\{\\bf\\widetilde\{M\}\}\_\{L,\\sigma^\{2\}\}\}\.

Moreover,\[[16](https://arxiv.org/html/2608.14638#bib.bib16)\]considered the case of DNNs with rectangular layer matrices of proportional sizes rather than square ones, withσ2=1\\sigma^\{2\}=1, and derived a recurrent relation onm𝐌L,σ2m\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\. We will use a slightly modified version of that relation for arbitrary value of weight variance parameterσ2\\sigma^\{2\}\. Let us formulate that relation in a shortest way\.

One can easily check that all matrices\(D~\(k\)\)2\(\\widetilde\{D\}^\{\(k\)\}\)^\{2\}from \([76](https://arxiv.org/html/2608.14638#S3.E76)\) have limiting NCM

ν𝐃~2=12​\(δ0\+δ1\),\\nu\_\{\\scriptscriptstyle\{\\bf\\widetilde\{D\}\}^\{2\}\}=\\dfrac\{1\}\{2\}\(\\delta\_\{0\}\+\\delta\_\{1\}\),\(80\)whereδx\\delta\_\{x\}is the delta\-measure atxx\. Letm𝐃~2​\(z\)m\_\{\\scriptscriptstyle\{\\bf\\widetilde\{D\}\}^\{2\}\}\(z\)be the moment generating function ofν𝐃~2\\nu\_\{\\scriptscriptstyle\{\\bf\\widetilde\{D\}\}^\{2\}\}\. Assume that the sizes of weight matrices satisfynknk\+1→dk∈\(0,\+∞\)\\dfrac\{n\_\{k\}\}\{n\_\{k\+1\}\}\\to d\_\{k\}\\in\(0,\+\\infty\)\. Then\[[16](https://arxiv.org/html/2608.14638#bib.bib16), Corollary 2\.3 and Remark 2\.4\]imply that forσ2=1\\sigma^\{2\}=1the functional inversez𝐌L,1​\(m\)z\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,1\}\}\(m\)ofm𝐌L,1​\(z\)m\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,1\}\}\(z\)satisfies

z𝐌L,1​\(m\)=z𝐌L−1,1​\(dL​m\)⋅z𝐃~2​\(dL​m\)⋅m−1\.z\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,1\}\}\(m\)=z\_\{\\scriptscriptstyle\{\\bf M\}\_\{L\-1,1\}\}\(d\_\{L\}m\)\\cdot z\_\{\\scriptscriptstyle\{\\bf\\widetilde\{D\}\}^\{2\}\}\(d\_\{L\}m\)\\cdot m^\{\-1\}\.\(81\)Observe that changing the variance of weights from1nk\\dfrac\{1\}\{n\_\{k\}\}toσ2nk\\dfrac\{\\sigma^\{2\}\}\{n\_\{k\}\}in𝐉~L,σ2\(N\)\\widetilde\{\{\\bf J\}\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}is equivalent to multiplying the matricesW\(k\)W^\{\(k\)\}byσ\\sigma, which means𝐌~L,σ2=\(σ2\)L​𝐌~L,1\\widetilde\{\{\\bf M\}\}\_\{L,\\sigma^\{2\}\}=\(\\sigma^\{2\}\)^\{L\}\\widetilde\{\{\\bf M\}\}\_\{L,1\}andz𝐌L,σ2​\(m\)=\(σ2\)L​z𝐌L,1​\(m\)z\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(m\)=\(\\sigma^\{2\}\)^\{L\}z\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,1\}\}\(m\)\. Thus, for arbitrary variance parameterσ2\\sigma^\{2\}relation \([81](https://arxiv.org/html/2608.14638#S3.E81)\) becomes

z𝐌L,σ2​\(m\)=z𝐌L−1,σ2​\(dL​m\)⋅z𝐃~2​\(dL​m\)⋅\(σ2​m\)−1\.z\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(m\)=z\_\{\\scriptscriptstyle\{\\bf M\}\_\{L\-1,\\sigma^\{2\}\}\}\(d\_\{L\}m\)\\cdot z\_\{\\scriptscriptstyle\{\\bf\\widetilde\{D\}\}^\{2\}\}\(d\_\{L\}m\)\\cdot\(\\sigma^\{2\}m\)^\{\-1\}\.\(82\)Direct computation shows that

m𝐃2​\(z\)=12​z1−z,z𝐃2​\(m\)=2​m2​m\+1\.m\_\{\\scriptscriptstyle\{\\bf D\}^\{2\}\}\(z\)=\\dfrac\{1\}\{2\}\\,\\dfrac\{z\}\{1\-z\},\\qquad z\_\{\\scriptscriptstyle\{\\bf D\}^\{2\}\}\(m\)=\\dfrac\{2m\}\{2m\+1\}\.\(83\)Substituting \([83](https://arxiv.org/html/2608.14638#S3.E83)\) into \([82](https://arxiv.org/html/2608.14638#S3.E82)\) gives us

z𝐌L,σ2​\(m\)=2σ2​z𝐌L−1,σ2​\(dL​m\)2​dL​m\+1z\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(m\)=\\dfrac\{2\}\{\\sigma^\{2\}\}\\,\\dfrac\{z\_\{\\scriptscriptstyle\{\\bf M\}\_\{L\-1,\\sigma^\{2\}\}\}\(d\_\{L\}m\)\}\{2d\_\{L\}m\+1\}\(84\)for ReLU activation\. Here we set𝐌0,σ2\(N\)=I\(N\)\{\\bf M\}\_\{0,\\sigma^\{2\}\}^\{\(N\)\}=I^\{\(N\)\}, which implies thatm𝐌0,σ2​\(z\)=z1−zm\_\{\\scriptscriptstyle\{\\bf M\}\_\{0,\\sigma^\{2\}\}\}\(z\)=\\dfrac\{z\}\{1\-z\},z𝐌0,σ2​\(m\)=mm\+1z\_\{\\scriptscriptstyle\{\\bf M\}\_\{0,\\sigma^\{2\}\}\}\(m\)=\\dfrac\{m\}\{m\+1\}\. Also recall that the autoencoder’s layer sizesn1=nL\+1=Nn\_\{1\}=n\_\{L\+1\}=Nandnk=ck​Nn\_\{k\}=c\_\{k\}N, thusdk=ckck\+1d\_\{k\}=\\dfrac\{c\_\{k\}\}\{c\_\{k\+1\}\}\(here and belowc1=cL\+1=1c\_\{1\}=c\_\{L\+1\}=1\)\. Putting everything together, we obtain an explicit formula forz𝐌L,σ2​\(m\)z\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(m\)for ReLU autoencoders:

z𝐌L,σ2​\(m\)=\(2σ2\)L​m\(m\+1\)⋅∏k=1L\(2​ck​m\+1\)\.z\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(m\)=\\left\(\\dfrac\{2\}\{\\sigma^\{2\}\}\\right\)^\{L\}\\dfrac\{m\}\{\(m\+1\)\\cdot\\prod\\limits\_\{k=1\}^\{L\}\(2c\_\{k\}m\+1\)\}\.\(85\)Finally, plugm=m𝐌L,σ2​\(z\)m=m\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(z\)into \([85](https://arxiv.org/html/2608.14638#S3.E85)\) and usez𝐌L,σ2​\(m𝐌L,σ2​\(z\)\)=zz\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(m\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(z\)\)=z, then we obtain an equation onm𝐌L,σ2​\(z\)m\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(z\):

z=\(2σ2\)L​m𝐌L,σ2​\(z\)\(m𝐌L,σ2​\(z\)\+1\)⋅∏k=1L\(2​ck​m𝐌L,σ2​\(z\)\+1\)z=\\left\(\\dfrac\{2\}\{\\sigma^\{2\}\}\\right\)^\{L\}\\dfrac\{m\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(z\)\}\{\(m\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(z\)\+1\)\\cdot\\prod\\limits\_\{k=1\}^\{L\}\(2c\_\{k\}m\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(z\)\+1\)\}\(86\)The derivation of \([86](https://arxiv.org/html/2608.14638#S3.E86)\) completesStep 1of the proof\.

#### 3\.1\.3Step 2: Formula for the right edge of spectrum

We start with rewriting equation \([86](https://arxiv.org/html/2608.14638#S3.E86)\) in the form

ℱ​\(m𝐌L,σ2​\(z\),z\)=0,\\mathcal\{F\}\(m\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(z\),z\)=0,\(87\)where

ℱ​\(m,z\)=2L​m−z​\(σ2\)L⋅\(m\+1\)⋅∏k=1L\(2​ck​m\+1\)\.\\mathcal\{F\}\(m,z\)=2^\{L\}m\-z\(\\sigma^\{2\}\)^\{L\}\\cdot\(m\+1\)\\cdot\\prod\\limits\_\{k=1\}^\{L\}\(2c\_\{k\}m\+1\)\.\(88\)Such equation cannot be solved with respect tom𝐌L,σ2​\(z\)m\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(z\)explicitly forL\>2L\>2\. That is the reason why we need to extract the information about the right edge of spectrum of𝐌L,σ2\(N\)\{\\bf M\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}from the equation \([87](https://arxiv.org/html/2608.14638#S3.E87)\) rather than from explicit formula form𝐌L,σ2​\(z\)m\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(z\)\.

Recall thatμ\+\(∞\)\\mu\_\{\+\}^\{\(\\infty\)\}is the right edge ofsupp​ν𝐌L,σ2\\text\{supp\}\\,\\nu\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\. Notice thatsupp​ν𝐌L,σ2⊂\[0;\+∞\)\\text\{supp\}\\,\\nu\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\\subset\[0;\+\\infty\)since𝐌L,σ2\(N\)\{\\bf M\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}is a nonnegative matrix\. A simple argument provides a formula for the right edge of compactly supported measure onℝ\+\\mathbb\{R\}\_\{\+\}in terms of its moments:

μ\+\(∞\)=lim supk→∞ak1/k,ak=∫−∞∞λk​𝑑ν𝐌L,σ2​\(λ\)\.\\mu\_\{\+\}^\{\(\\infty\)\}=\\limsup\_\{k\\to\\infty\}a\_\{k\}^\{1/k\},\\qquad a\_\{k\}=\\int\_\{\-\\infty\}^\{\\infty\}\\lambda^\{k\}\\,d\\nu\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(\\lambda\)\.\(89\)At the same time,aka\_\{k\}are the coefficients of power seriesm𝐌L,σ2​\(z\)=∑k=1∞ak​zkm\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(z\)=\\sum\\limits\_\{k=1\}^\{\\infty\}a\_\{k\}z^\{k\}which has radius of convergenceRRgiven by Cauchy\-Hadamard theorem:

1R=lim supk→∞ak1/k\.\\dfrac\{1\}\{R\}=\\limsup\_\{k\\to\\infty\}a\_\{k\}^\{1/k\}\.\(90\)As we see, \([89](https://arxiv.org/html/2608.14638#S3.E89)\) and \([90](https://arxiv.org/html/2608.14638#S3.E90)\) yield

μ\+\(∞\)=1R\.\\mu\_\{\+\}^\{\(\\infty\)\}=\\dfrac\{1\}\{R\}\.\(91\)We are left to find the radius of convergenceRRof the analytic functionm𝐌L,σ2​\(z\)m\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(z\)given implicitly by \([87](https://arxiv.org/html/2608.14638#S3.E87)\)\. A rather standard argument on singularities of implicitly given analytic function and their relation to the radius of convergence shows that there existm∗\>0m\_\{\*\}\>0such thatz=Rz=R,m=m∗m=m\_\{\*\}is a solution of the system

\{ℱ​\(m,z\)=0;∂ℱ∂m​\(m,z\)=0\.\\begin\{cases\}\\mathcal\{F\}\(m,z\)=0;\\\\ \\dfrac\{\\partial\\mathcal\{F\}\}\{\\partial m\}\(m,z\)=0\.\\end\{cases\}\(92\)A rigorous argument is placed in Appendix B\.

Observe that

∂ℱ∂m​\(m,z\)=2L−z​\(σ2\)L​\(m\+1\)​∏k=1L\(2​ck​m\+1\)⋅\(1m\+1\+∑k=1L2​ck2​ck​m\+1\)\.\\dfrac\{\\partial\\mathcal\{F\}\}\{\\partial m\}\(m,z\)=2^\{L\}\-z\(\\sigma^\{2\}\)^\{L\}\(m\+1\)\\prod\\limits\_\{k=1\}^\{L\}\(2c\_\{k\}m\+1\)\\cdot\\left\(\\dfrac\{1\}\{m\+1\}\+\\sum\_\{k=1\}^\{L\}\\dfrac\{2c\_\{k\}\}\{2c\_\{k\}m\+1\}\\right\)\.\(93\)Straightforward transformations show that system \([92](https://arxiv.org/html/2608.14638#S3.E92)\) implies

1m\+1\+∑k=1L2​ck2​ck​m\+1=1m,\\dfrac\{1\}\{m\+1\}\+\\sum\_\{k=1\}^\{L\}\\dfrac\{2c\_\{k\}\}\{2c\_\{k\}m\+1\}=\\dfrac\{1\}\{m\},\(94\)which is equivalent to

11\+1m\+∑k=1L11\+12​ck​m=1\.\\dfrac\{1\}\{1\+\\dfrac\{1\}\{m\}\}\+\\sum\_\{k=1\}^\{L\}\\dfrac\{1\}\{1\+\\dfrac\{1\}\{2c\_\{k\}m\}\}=1\.\(95\)Denotec~=min⁡\{1,2​c\}\\widetilde\{c\}=\\min\\\{1,2c\\\},C~=max⁡\{2,2​C\}\\widetilde\{C\}=\\max\\\{2,2C\\\}, thenc~​m<m<C~​m\\widetilde\{c\}m<m<\\widetilde\{C\}mandc~​m<2​ck​m<C~​m\\widetilde\{c\}m<2c\_\{k\}m<\\widetilde\{C\}mfork=1,2,…,Lk=1,2,\\ldots,L\. Using such estimations and monotonicity of \([95](https://arxiv.org/html/2608.14638#S3.E95)\) one can show that the positive solutionm∗m\_\{\*\}of \([95](https://arxiv.org/html/2608.14638#S3.E95)\) is unique and satisfies

1C~​L<m∗<1c~​L\.\\dfrac\{1\}\{\\widetilde\{C\}L\}<m\_\{\*\}<\\dfrac\{1\}\{\\widetilde\{c\}L\}\.\(96\)Substitutingm=m∗m=m\_\{\*\}into the first equation of \([92](https://arxiv.org/html/2608.14638#S3.E92)\), we obtain the solution\(m∗,R\)\(m\_\{\*\},R\)of \([92](https://arxiv.org/html/2608.14638#S3.E92)\) with

R=\(2σ2\)L​m∗\(m∗\+1\)⋅∏k=1L\(2​ck​m∗\+1\)\.R=\\left\(\\dfrac\{2\}\{\\sigma^\{2\}\}\\right\)^\{L\}\\dfrac\{m\_\{\*\}\}\{\(m\_\{\*\}\+1\)\\cdot\\prod\\limits\_\{k=1\}^\{L\}\(2c\_\{k\}m\_\{\*\}\+1\)\}\.\(97\)Using \([91](https://arxiv.org/html/2608.14638#S3.E91)\) we can conclude that

μ\+\(∞\)=\(σ22\)L⋅m∗−1​\(m∗\+1\)⋅∏k=1L\(2​ck​m∗\+1\),where​1C~​L<m∗<1c~​L,\\mu\_\{\+\}^\{\(\\infty\)\}=\\left\(\\dfrac\{\\sigma^\{2\}\}\{2\}\\right\)^\{L\}\\cdot m\_\{\*\}^\{\-1\}\(m\_\{\*\}\+1\)\\cdot\\prod\\limits\_\{k=1\}^\{L\}\(2c\_\{k\}m\_\{\*\}\+1\),\\quad\\text\{where \}\\dfrac\{1\}\{\\widetilde\{C\}L\}<m\_\{\*\}<\\dfrac\{1\}\{\\widetilde\{c\}L\},\(98\)which completes the proof of\(i\)andStep 2\.

#### 3\.1\.4Step 3: right edge andσc​r​i​t2​\(L\)\\sigma^\{2\}\_\{crit\}\(L\)in the infinite\-width limit

In order to obtain \([71](https://arxiv.org/html/2608.14638#S3.E71)\), rewrite \([98](https://arxiv.org/html/2608.14638#S3.E98)\) as

1L​log⁡μ\+\(∞\)=log⁡σ22−1L​log⁡m∗\+1L​log⁡\(1\+m∗\)\+1L​∑k=1Llog⁡\(1\+2​ck​m∗\)\.\\dfrac\{1\}\{L\}\\log\\mu\_\{\+\}^\{\(\\infty\)\}=\\log\\dfrac\{\\sigma^\{2\}\}\{2\}\-\\dfrac\{1\}\{L\}\\log m\_\{\*\}\+\\dfrac\{1\}\{L\}\\log\(1\+m\_\{\*\}\)\+\\dfrac\{1\}\{L\}\\sum\_\{k=1\}^\{L\}\\log\(1\+2c\_\{k\}m\_\{\*\}\)\.\(100\)Sincem∗≍1Lm\_\{\*\}\\asymp\\dfrac\{1\}\{L\}and

0<1L​∑k=1Llog⁡\(1\+2​ck​m∗\)≤log⁡\(1\+C~​m∗\),0<\\dfrac\{1\}\{L\}\\sum\_\{k=1\}^\{L\}\\log\(1\+2c\_\{k\}m\_\{\*\}\)\\leq\\log\(1\+\\widetilde\{C\}m\_\{\*\}\),\(101\)all the summands of the right hand side of \([100](https://arxiv.org/html/2608.14638#S3.E100)\) apart from the first one converge to0asL→∞L\\to\\infty, which gives us \([71](https://arxiv.org/html/2608.14638#S3.E71)\)\. And completes the proof of\(ii\)\.

In order to prove \([72](https://arxiv.org/html/2608.14638#S3.E72)\), we first use \([98](https://arxiv.org/html/2608.14638#S3.E98)\) to solveμ\+\(∞\)=1\\mu\_\{\+\}^\{\(\\infty\)\}=1with respect toσ2\\sigma^\{2\}:

σL,crit2=2⋅\(m∗\(m∗\+1\)⋅∏k=1L\(2​ck​m∗\+1\)\)1/L,where​1C~​L<m∗<1c~​L\.\\sigma\_\{L,\\mathrm\{crit\}\}^\{2\}=2\\cdot\\Biggl\(\\dfrac\{m\_\{\*\}\}\{\(m\_\{\*\}\+1\)\\cdot\\prod\\limits\_\{k=1\}^\{L\}\(2c\_\{k\}m\_\{\*\}\+1\)\}\\Biggr\)^\{1/L\},\\quad\\text\{where \}\\dfrac\{1\}\{\\widetilde\{C\}L\}<m\_\{\*\}<\\dfrac\{1\}\{\\widetilde\{c\}L\}\.\(102\)Rewrite the formula as

log⁡σc​r​i​t2​\(L\)2=1L​log⁡m∗−1L​log⁡\(1\+m∗\)−1L​∑k=1Llog⁡\(1\+2​ck​m∗\)\.\\log\\dfrac\{\\sigma^\{2\}\_\{crit\}\(L\)\}\{2\}=\\dfrac\{1\}\{L\}\\log m\_\{\*\}\-\\dfrac\{1\}\{L\}\\log\(1\+m\_\{\*\}\)\-\\dfrac\{1\}\{L\}\\sum\_\{k=1\}^\{L\}\\log\(1\+2c\_\{k\}m\_\{\*\}\)\.\(103\)As we already showed, the right hand side converges to zero asL→∞L\\to\\infty, which gives us \([72](https://arxiv.org/html/2608.14638#S3.E72)\) and completes the prooof of Proposition[3\.1](https://arxiv.org/html/2608.14638#S3.Thmtheorem1)\.

### 3\.2Proof of Proposition[3\.2](https://arxiv.org/html/2608.14638#S3.Thmtheorem2)

Recall that in\[[11](https://arxiv.org/html/2608.14638#bib.bib11), Proposition 2\]it was proved that random matrices𝐌L,σ2\(N\)​\(s\)=\(𝐉L,σ2\(N\)​\(s\)\)T​𝐉L,σ2\(N\)​\(s\)\{\\bf M\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\(s\)=\(\{\\bf J\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\(s\)\)^\{T\}\{\\bf J\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\(s\)and𝐌~L,σ2\(N\)=\(𝐉~L,σ2\(N\)\)T​𝐉~L,σ2\(N\)\\widetilde\{\{\\bf M\}\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}=\(\\widetilde\{\{\\bf J\}\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\)^\{T\}\\widetilde\{\{\\bf J\}\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}are equal in distribution for anys∈ℝNs\\in\\mathbb\{R\}^\{N\}, where𝐉L,σ2\(N\)​\(s\)\{\\bf J\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\(s\)and𝐉~L,σ2\(N\)\\widetilde\{\{\\bf J\}\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}are defined in \([75](https://arxiv.org/html/2608.14638#S3.E75)\) and \([76](https://arxiv.org/html/2608.14638#S3.E76)\)\. This implies that random variables‖𝐉L,σ2\(N\)​\(s\)‖\\\|\{\\bf J\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\(s\)\\\|and‖𝐉~L,σ2\(N\)‖\\\|\\widetilde\{\{\\bf J\}\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\\\|are equal in distribution, and thus we are left to prove the convergence

‖𝐉~L,σ2\(N\)‖→μ\+\(∞\)a\.s\. asN→∞\.\\\|\\widetilde\{\{\\bf J\}\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\\\|\\to\\sqrt\{\\mu\_\{\+\}^\{\(\\infty\)\}\}\\quad\\text\{a\.s\. as $N\\to\\infty$\}\.\(104\)
We can use the result of\[[18](https://arxiv.org/html/2608.14638#bib.bib18)\]inductively onLLto obtain that\(𝐉~L,σ2\(N\)\)T​𝐉~L,σ2\(N\)\(\\widetilde\{\{\\bf J\}\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}\)^\{T\}\\widetilde\{\{\\bf J\}\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}has no eigenvalues outside of the support of a measureFcn,An,BnF^\{c\_\{n\},A\_\{n\},B\_\{n\}\}defined in\[[18](https://arxiv.org/html/2608.14638#bib.bib18)\]\. Then we can use the results of\[[4](https://arxiv.org/html/2608.14638#bib.bib4)\]\(see Proposition 3\.1, Proposition 3\.2 and Subsection 3\.3\) to show that the right edge ofFcn,An,BnF^\{c\_\{n\},A\_\{n\},B\_\{n\}\}converges toμ\+\(∞\)\\sqrt\{\\mu\_\{\+\}^\{\(\\infty\)\}\}almost surely asn→∞n\\to\\infty\. This observation finishes the proof\.

## 4Appendix

### A: Proof of Theorem[1\.2](https://arxiv.org/html/2608.14638#S1.Thmtheorem2), LeakyReLU case0<α<10<\\alpha<1

We considerLL\-layer autoencoder functionXN,L​\(s\)X\_\{N,L\}\(s\)with no biases, LeakyReLU activation, and Gaussian initialization of weights:

𝐗L,σ2\(N\)​\(s\)=Φ\(L\)∘…∘Φ\(2\)∘Φ\(1\),Φ\(k\)​\(s\)=ϕα​\(W\(k\)​s\),\{\\bf X\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\)=\\Phi^\{\(L\)\}\\circ\\ldots\\circ\\Phi^\{\(2\)\}\\circ\\Phi^\{\(1\)\},\\quad\\Phi^\{\(k\)\}\(s\)=\\phi\_\{\\alpha\}\(W^\{\(k\)\}s\),\(105\)whereW\(k\)W^\{\(k\)\}is ank\+1×nkn\_\{k\+1\}\\times n\_\{k\}with i\.i\.d\. entries distributed asN​\(0,σ2nk\)N\(0,\\dfrac\{\\sigma^\{2\}\}\{n\_\{k\}\}\)andϕα\\phi\_\{\\alpha\}is the coordinatewise version of LeakyReLU with parameterα∈\(0;1\)\\alpha\\in\(0;1\):

ϕα​\(x\)=\{x,if​x≥0;α​x,if​x<0\.\\phi\_\{\\alpha\}\(x\)=\\begin\{cases\}x,\\text\{\\quad if \}x\\geq 0;\\\\ \\alpha x,\\text\{\\quad if \}x<0\.\\end\{cases\}\(106\)Define

‖𝐗L,σ2\(N\)‖:=sup‖s‖=1‖𝐗L,σ2\(N\)​\(s\)‖\.\\\|\{\\bf X\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\\\|:=\\sup\_\{\\\|s\\\|=1\}\\\|\{\\bf X\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\(s\)\\\|\.\(107\)The proof of Theorem[1\.2](https://arxiv.org/html/2608.14638#S1.Thmtheorem2)for the case0<α<10<\\alpha<1follows the exact same steps as the proof in the caseα=0\\alpha=0described in Section[2](https://arxiv.org/html/2608.14638#S2)\. Here we specify the main differences\.

Define neuron vectors and their norms:

s\(0\):=s∈ℝN,s\(k\):=Φ\(k\)​\(s\(k−1\)\)=ϕα​\(W\(k\)​s\(k−1\)\)∈ℝnk\+1,r\(k\):=‖s\(k\)‖\.s^\{\(0\)\}:=s\\in\\mathbb\{R\}^\{N\},\\quad s^\{\(k\)\}:=\\Phi^\{\(k\)\}\(s^\{\(k\-1\)\}\)=\\phi\_\{\\alpha\}\(W^\{\(k\)\}s^\{\(k\-1\)\}\)\\in\\mathbb\{R\}^\{n\_\{k\+1\}\},\\quad r^\{\(k\)\}:=\\\|s^\{\(k\)\}\\\|\.\(108\)Then

‖𝐗L,σ2\(N\)‖=sups∈ℝN,‖s‖=1∏k=1Lr\(k\)r\(k−1\)\.\\\|\{\\bf X\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\\\|=\\sup\_\{s\\in\\mathbb\{R\}^\{N\},\\ \\\|s\\\|=1\}\\prod\_\{k=1\}^\{L\}\\dfrac\{r^\{\(k\)\}\}\{r^\{\(k\-1\)\}\}\.\(109\)Also define normalized neuron vectorsuk:=s\(k\)r\(k\)∈𝕊nk\+1−1u\_\{k\}:=\\dfrac\{s^\{\(k\)\}\}\{r^\{\(k\)\}\}\\in\\mathbb\{S\}^\{n\_\{k\+1\}\-1\}\. Using positive homogeneity ofϕα\\phi\_\{\\alpha\}one can show that

r\(k\)r\(k−1\)=‖ϕα​\(W\(k\)​uk−1\)‖,\\dfrac\{r^\{\(k\)\}\}\{r^\{\(k\-1\)\}\}=\\\|\\phi\_\{\\alpha\}\(W^\{\(k\)\}u\_\{k\-1\}\)\\\|,\(110\)anduku\_\{k\}’s satisfy the recurrence relation

u0=s,uk=ϕα​\(W\(k\)​uk−1\)‖ϕα​\(W\(k\)​uk−1\)‖\.u\_\{0\}=s,\\qquad u\_\{k\}=\\dfrac\{\\phi\_\{\\alpha\}\(W^\{\(k\)\}u\_\{k\-1\}\)\}\{\\\|\\phi\_\{\\alpha\}\(W^\{\(k\)\}u\_\{k\-1\}\)\\\|\}\.\(111\)Also positive homogeneity ofϕα\\phi\_\{\\alpha\}implies

\(W\(k\)​uk−1,ϕα​\(uk\)\)=\(W\(k\)​uk−1,ϕα​\(ϕα​\(W\(k\)​uk−1\)‖ϕα​\(W\(k\)​uk−1\)‖\)\)==1‖ϕα​\(W\(k\)​uk−1\)‖​\(W\(k\)​uk−1,ϕα​\(ϕα​\(W\(k\)​uk−1\)\)\)\.\(W^\{\(k\)\}u\_\{k\-1\},\\phi\_\{\\alpha\}\(u\_\{k\}\)\)=\\left\(W^\{\(k\)\}u\_\{k\-1\},\\phi\_\{\\alpha\}\\Bigl\(\\dfrac\{\\phi\_\{\\alpha\}\(W^\{\(k\)\}u\_\{k\-1\}\)\}\{\\\|\\phi\_\{\\alpha\}\(W^\{\(k\)\}u\_\{k\-1\}\)\\\|\}\\Bigr\)\\right\)=\\\\ =\\dfrac\{1\}\{\\\|\\phi\_\{\\alpha\}\(W^\{\(k\)\}u\_\{k\-1\}\)\\\|\}\\bigl\(W^\{\(k\)\}u\_\{k\-1\},\\phi\_\{\\alpha\}\(\\phi\_\{\\alpha\}\(W^\{\(k\)\}u\_\{k\-1\}\)\)\\bigr\)\.\(112\)Coordinatewise computation shows that\(v,ϕα​\(ϕα​\(v\)\)\)=‖ϕα​\(v\)‖2\\Bigl\(v,\\phi\_\{\\alpha\}\(\\phi\_\{\\alpha\}\(v\)\)\\Bigr\)=\\\|\\phi\_\{\\alpha\}\(v\)\\\|^\{2\}, thus

\(W\(k\)​uk−1,ϕα​\(uk\)\)=‖ϕα​\(W\(k\)​uk−1\)‖=r\(k\)r\(k−1\),\(W^\{\(k\)\}u\_\{k\-1\},\\phi\_\{\\alpha\}\(u\_\{k\}\)\)=\\\|\\phi\_\{\\alpha\}\(W^\{\(k\)\}u\_\{k\-1\}\)\\\|=\\dfrac\{r^\{\(k\)\}\}\{r^\{\(k\-1\)\}\},\(113\)which gives a formula similar to \([42](https://arxiv.org/html/2608.14638#S2.E42)\) for‖𝐗L,σ2\(N\)‖\\\|\{\\bf X\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\\\|in Leaky ReLU case:

‖𝐗L,σ2\(N\)‖=sups∈𝕊N−1∏k=1L\(W\(k\)​uk−1,ϕα​\(uk\)\)\.\\\|\{\\bf X\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\\\|=\\sup\_\{s\\in\\mathbb\{S\}^\{N\-1\}\}\\prod\_\{k=1\}^\{L\}\\bigl\(W^\{\(k\)\}u\_\{k\-1\},\\phi\_\{\\alpha\}\(u\_\{k\}\)\\bigr\)\.\(114\)Using AM\-GM similarly to Subsection[2\.3](https://arxiv.org/html/2608.14638#S2.SS3), we obtain

‖𝐗L,σ2\(N\)‖≤\(1L​supuk∈𝕊nk\+1−1,k=0,1,…,L∑k=1L\(W\(k\)​uk−1,ϕα​\(uk\)\)\)L\.\\\|\{\\bf X\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\\\|\\leq\\Bigg\(\\dfrac\{1\}\{L\}\\sup\_\{\\begin\{subarray\}\{c\}u\_\{k\}\\in\\mathbb\{S\}^\{n\_\{k\+1\}\-1\},\\\\ k=0,1,\\ldots,L\\end\{subarray\}\}\\quad\\sum\\limits\_\{k=1\}^\{L\}\(W^\{\(k\)\}u\_\{k\-1\},\\phi\_\{\\alpha\}\(u\_\{k\}\)\)\\Bigg\)^\{L\}\.\(115\)
Denote𝐮:=\(u0,u1,…,uL\)\{\\bf u\}:=\(u\_\{0\},u\_\{1\},\\ldots,u\_\{L\}\)and𝐒:=𝕊n1−1×𝕊n2−1×…×𝕊nL\+1−1\{\\bf S\}:=\\mathbb\{S\}^\{n\_\{1\}\-1\}\\times\\mathbb\{S\}^\{n\_\{2\}\-1\}\\times\\ldots\\times\\mathbb\{S\}^\{n\_\{L\+1\}\-1\}\. We have an upper bound for‖𝐗L,σ2\(N\)‖\\\|\{\\bf X\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\\\|:

‖𝐗L,σ2\(N\)‖≤\(1L​sup𝐮∈𝐒P𝐮\)L,P𝐮=∑k=1L\(W\(k\)​uk−1,ϕα​\(uk\)\),\\\|\{\\bf X\}\_\{L,\\sigma^\{2\}\}^\{\(N\)\}\\\|\\leq\\Bigg\(\\dfrac\{1\}\{L\}\\sup\_\{\{\\bf u\}\\in\{\\bf S\}\}P\_\{\\bf u\}\\Bigg\)^\{L\},\\qquad P\_\{\\bf u\}=\\sum\\limits\_\{k=1\}^\{L\}\(W^\{\(k\)\}u\_\{k\-1\},\\phi\_\{\\alpha\}\(u\_\{k\}\)\),\(116\)where\{P𝐮\}𝐮∈𝐒\\\{P\_\{\\bf u\}\\\}\_\{\{\\bf u\}\\in\{\\bf S\}\}is a Gaussian process\. This derivation completesStep 1\.

InStep 2we need to find majorizing Gaussian processQ𝐮Q\_\{\\bf u\}forP𝐮P\_\{\\bf u\}\. Consider independent Gaussian vectorsg0,g1,…,gL−1,h1,h2,…,hLg\_\{0\},g\_\{1\},\\ldots,g\_\{L\-1\},h\_\{1\},h\_\{2\},\\ldots,h\_\{L\}such that eachgk,hk∈ℝnk\+1g\_\{k\},h\_\{k\}\\in\\mathbb\{R\}^\{n\_\{k\+1\}\}has i\.i\.d\. entries distributed asN​\(0,1\)N\(0,1\)\. Denote

Q𝐮:=σ2n1​\(g0,u0\)\+∑k=1L−1\(σ2nk\+1​\(gk,uk\)\+σ2nk​\(hk,ϕα​\(uk\)\)\)\+σ2nL​\(hL,ϕα​\(uL\)\)Q\_\{\\bf u\}:=\\sqrt\{\\frac\{\\sigma^\{2\}\}\{n\_\{1\}\}\}\(g\_\{0\},u\_\{0\}\)\+\\sum\_\{k=1\}^\{L\-1\}\\left\(\\sqrt\{\\frac\{\\sigma^\{2\}\}\{n\_\{k\+1\}\}\}\(g\_\{k\},u\_\{k\}\)\+\\sqrt\{\\frac\{\\sigma^\{2\}\}\{n\_\{k\}\}\}\(h\_\{k\},\\phi\_\{\\alpha\}\(u\_\{k\}\)\)\\right\)\+\\sqrt\{\\frac\{\\sigma^\{2\}\}\{n\_\{L\}\}\}\(h\_\{L\},\\phi\_\{\\alpha\}\(u\_\{L\}\)\)\(117\)Computations similar to \([50](https://arxiv.org/html/2608.14638#S2.E50)\) and \([52](https://arxiv.org/html/2608.14638#S2.E52)\) show that

𝔼​\{\(P𝐮−P𝐮′\)2\}==∑k=1Lσ2nk​\(‖uk−1‖2​‖ϕα​\(uk\)‖2−2​\(uk−1,uk−1′\)​\(ϕα​\(uk\),ϕα​\(uk′\)\)\+‖uk−1′‖2​‖ϕα​\(uk′\)‖2\)==∑k=1Lσ2nk​\(‖ϕα​\(uk\)‖2−2​\(uk−1,uk−1′\)​\(ϕα​\(uk\),ϕα​\(uk′\)\)\+‖ϕα​\(uk′\)‖2\)\\mathbb\{E\}\\\{\(P\_\{\\bf u\}\-P\_\{\\bf u^\{\\prime\}\}\)^\{2\}\\\}=\\\\ =\\sum\_\{k=1\}^\{L\}\\dfrac\{\\sigma^\{2\}\}\{n\_\{k\}\}\(\\\|u\_\{k\-1\}\\\|^\{2\}\\\|\\phi\_\{\\alpha\}\(u\_\{k\}\)\\\|^\{2\}\-2\(u\_\{k\-1\},u\_\{k\-1\}^\{\\prime\}\)\(\\phi\_\{\\alpha\}\(u\_\{k\}\),\\phi\_\{\\alpha\}\(u\_\{k\}^\{\\prime\}\)\)\+\\\|u\_\{k\-1\}^\{\\prime\}\\\|^\{2\}\\\|\\phi\_\{\\alpha\}\(u\_\{k\}^\{\\prime\}\)\\\|^\{2\}\)=\\\\ =\\sum\_\{k=1\}^\{L\}\\dfrac\{\\sigma^\{2\}\}\{n\_\{k\}\}\(\\\|\\phi\_\{\\alpha\}\(u\_\{k\}\)\\\|^\{2\}\-2\(u\_\{k\-1\},u\_\{k\-1\}^\{\\prime\}\)\(\\phi\_\{\\alpha\}\(u\_\{k\}\),\\phi\_\{\\alpha\}\(u\_\{k\}^\{\\prime\}\)\)\+\\\|\\phi\_\{\\alpha\}\(u\_\{k\}^\{\\prime\}\)\\\|^\{2\}\)\(118\)since‖uk‖=‖uk′‖=1\\\|u\_\{k\}\\\|=\\\|u\_\{k\}^\{\\prime\}\\\|=1\. Observe that‖ϕα​\(uk\)‖,‖ϕα​\(uk′\)‖≤1\\\|\\phi\_\{\\alpha\}\(u\_\{k\}\)\\\|,\\\|\\phi\_\{\\alpha\}\(u\_\{k\}^\{\\prime\}\)\\\|\\leq 1sinceα∈\(0,1\)\\alpha\\in\(0,1\), thus\(uk−1,uk−1′\)≤1\(u\_\{k\-1\},u\_\{k\-1\}^\{\\prime\}\)\\leq 1,\(ϕα​\(uk\),ϕα​\(uk′\)\)≤1\(\\phi\_\{\\alpha\}\(u\_\{k\}\),\\phi\_\{\\alpha\}\(u\_\{k\}^\{\\prime\}\)\)\\leq 1\. We can use

−x​y≤1−x−y​for​x,y≤1\-xy\\leq 1\-x\-y\\text\{\\quad for \}x,y\\leq 1\(119\)to obtain

−2​\(uk−1,uk−1′\)​\(ϕα​\(uk\),ϕα​\(uk′\)\)≤‖uk−1‖2\+‖uk−1′‖−2​\(uk−1,uk−1′\)−2​\(ϕα​\(uk\),ϕα​\(uk′\)\)\.\-2\(u\_\{k\-1\},u\_\{k\-1\}^\{\\prime\}\)\(\\phi\_\{\\alpha\}\(u\_\{k\}\),\\phi\_\{\\alpha\}\(u\_\{k\}^\{\\prime\}\)\)\\leq\\\|u\_\{k\-1\}\\\|^\{2\}\+\\\|u\_\{k\-1\}^\{\\prime\}\\\|\-2\(u\_\{k\-1\},u\_\{k\-1\}^\{\\prime\}\)\-2\(\\phi\_\{\\alpha\}\(u\_\{k\}\),\\phi\_\{\\alpha\}\(u\_\{k\}^\{\\prime\}\)\)\.\(120\)Thus

𝔼​\{\(P𝐮−P𝐮′\)2\}≤∑k=1Lσ2nl​\(‖uk−1−uk−1′‖2\+‖ϕα​\(uk\)−ϕα​\(uk′\)‖2\)\.\\mathbb\{E\}\\\{\(P\_\{\\bf u\}\-P\_\{\\bf u^\{\\prime\}\}\)^\{2\}\\\}\\leq\\sum\_\{k=1\}^\{L\}\\dfrac\{\\sigma^\{2\}\}\{n\_\{l\}\}\\Bigl\(\\\|u\_\{k\-1\}\-u\_\{k\-1\}^\{\\prime\}\\\|^\{2\}\+\\\|\\phi\_\{\\alpha\}\(u\_\{k\}\)\-\\phi\_\{\\alpha\}\(u\_\{k\}^\{\\prime\}\)\\\|^\{2\}\\Bigr\)\.\(121\)Similarly to \([58](https://arxiv.org/html/2608.14638#S2.E58)\), using independence ofgk,hkg\_\{k\},h\_\{k\}and the fact that𝔼​\{\(g,v\)2\}=‖v‖2\\mathbb\{E\}\\\{\(g,v\)^\{2\}\\\}=\\\|v\\\|^\{2\}for a Gaussian vectorggwith i\.i\.d\. entriesN​\(0,1\)N\(0,1\), we obtain

𝔼​\{\(Q𝐮−Q𝐮′\)2\}=∑k=1Lσ2nl​\(‖uk−1−uk−1′‖2\+‖ϕα​\(uk\)−ϕα​\(uk′\)‖2\)\.\\mathbb\{E\}\\\{\(Q\_\{\\bf u\}\-Q\_\{\\bf u^\{\\prime\}\}\)^\{2\}\\\}=\\sum\_\{k=1\}^\{L\}\\dfrac\{\\sigma^\{2\}\}\{n\_\{l\}\}\\Bigl\(\\\|u\_\{k\-1\}\-u\_\{k\-1\}^\{\\prime\}\\\|^\{2\}\+\\\|\\phi\_\{\\alpha\}\(u\_\{k\}\)\-\\phi\_\{\\alpha\}\(u\_\{k\}^\{\\prime\}\)\\\|^\{2\}\\Bigr\)\.\(122\)Relations \([121](https://arxiv.org/html/2608.14638#S4.E121)\) and \([122](https://arxiv.org/html/2608.14638#S4.E122)\) show that Gaussian processQ𝐮Q\_\{\\bf u\}majorizesP𝐮P\_\{\\bf u\}in terms of \([29](https://arxiv.org/html/2608.14638#S2.E29)\)\. This observation completesStep 2\.

InStep 3we need to obtain an upper bound on𝔼​\{sup𝐮∈𝐒Q𝐮\}\\mathbb\{E\}\\\{\\sup\\limits\_\{\{\\bf u\}\\in\{\\bf S\}\}Q\_\{\\bf u\}\\\}From the definition ofQ𝐮Q\_\{\\bf u\}we have

𝔼​\{sup𝐮∈𝐒Q𝐮\}=σ2n1​𝔼​\{supu0∈𝕊n1−1\(g0,u0\)\}\+\+∑k=1L−1σ2nk\+1​𝔼​\{supuk∈𝕊nk\+1−1\(\(gk,uk\)\+ck\+1ck​\(hk,ϕα​\(uk\)\)\)\}\+\+σ2nL​𝔼​\{supuL∈𝕊nL\+1−1\(hL,ϕα​\(uL\)\)\}\.\\mathbb\{E\}\\\{\\sup\\limits\_\{\{\\bf u\}\\in\{\\bf S\}\}Q\_\{\\bf u\}\\\}=\\sqrt\{\\frac\{\\sigma^\{2\}\}\{n\_\{1\}\}\}\\mathbb\{E\}\\left\\\{\\sup\_\{u\_\{0\}\\in\\mathbb\{S\}^\{n\_\{1\}\-1\}\}\(g\_\{0\},u\_\{0\}\)\\right\\\}\+\\\\ \+\\sum\_\{k=1\}^\{L\-1\}\\sqrt\{\\frac\{\\sigma^\{2\}\}\{n\_\{k\+1\}\}\}\\mathbb\{E\}\\left\\\{\\sup\_\{u\_\{k\}\\in\\mathbb\{S\}^\{n\_\{k\+1\}\-1\}\}\\left\(\(g\_\{k\},u\_\{k\}\)\+\\sqrt\{\\dfrac\{c\_\{k\+1\}\}\{c\_\{k\}\}\}\(h\_\{k\},\\phi\_\{\\alpha\}\(u\_\{k\}\)\)\\right\)\\right\\\}\+\\\\ \+\\sqrt\{\\frac\{\\sigma^\{2\}\}\{n\_\{L\}\}\}\\mathbb\{E\}\\left\\\{\\sup\_\{u\_\{L\}\\in\\mathbb\{S\}^\{n\_\{L\+1\}\-1\}\}\(h\_\{L\},\\phi\_\{\\alpha\}\(u\_\{L\}\)\)\\right\\\}\.\(123\)For the first summand we have

𝔼​\{supu0∈𝕊n1−1\(g0,u0\)\}=𝔼​\{‖g0‖\}≤\(𝔼​\{‖g0‖2\}\)1/2=n1\.\\mathbb\{E\}\\left\\\{\\sup\_\{u\_\{0\}\\in\\mathbb\{S\}^\{n\_\{1\}\-1\}\}\(g\_\{0\},u\_\{0\}\)\\right\\\}=\\mathbb\{E\}\\\{\\\|g\_\{0\}\\\|\\\}\\leq\\Bigl\(\\mathbb\{E\}\\\{\\\|g\_\{0\}\\\|^\{2\}\\\}\\Bigr\)^\{1/2\}=\\sqrt\{n\_\{1\}\}\.\(124\)For other summands the estimations come from the following fact\.

###### Lemma 4\.1

Letg,h∈ℝng,h\\in\\mathbb\{R\}^\{n\}be independent Gaussian vectors with i\.i\.d\. entriesN​\(0,1\)N\(0,1\),ρ≥0\\rho\\geq 0,μ≥0\\mu\\geq 0,α∈\(0,1\)\\alpha\\in\(0,1\)\. Then

𝔼​\{supu∈𝕊n−1\(ρ​\(g,u\)\+μ​\(h,ϕα​\(u\)\)\)\}≤ρ2\+μ2​1\+α22⋅n\.\\mathbb\{E\}\\left\\\{\\sup\_\{u\\in\\mathbb\{S\}^\{n\-1\}\}\\Bigl\(\\rho\\,\(g,u\)\+\\mu\(h,\\phi\_\{\\alpha\}\(u\)\)\\Bigr\)\\right\\\}\\leq\\sqrt\{\\rho^\{2\}\+\\mu^\{2\}\\dfrac\{1\+\\alpha^\{2\}\}\{2\}\}\\cdot\\sqrt\{n\}\.\(125\)

The proof of Lemma[4\.1](https://arxiv.org/html/2608.14638#S4.Thmtheorem1)is placed in Appendix C\.

Using Lemma[4\.1](https://arxiv.org/html/2608.14638#S4.Thmtheorem1)we obtain

𝔼​\{sup𝐮∈𝐒Q𝐮\}≤σ⋅\(1\+∑k=1L−11\+ck\+1ck⋅1\+α22\+1cL​1\+α22\)\.\\mathbb\{E\}\\\{\\sup\\limits\_\{\{\\bf u\}\\in\{\\bf S\}\}Q\_\{\\bf u\}\\\}\\leq\\sigma\\cdot\\left\(1\+\\sum\\limits\_\{k=1\}^\{L\-1\}\\sqrt\{1\+\\dfrac\{c\_\{k\+1\}\}\{c\_\{k\}\}\\cdot\\dfrac\{1\+\\alpha^\{2\}\}\{2\}\}\+\\sqrt\{\\dfrac\{1\}\{c\_\{L\}\}\}\\sqrt\{\\dfrac\{1\+\\alpha^\{2\}\}\{2\}\}\\right\)\.\(126\)Finally, application of Sudakov\-Fernique inequality toP𝐮P\_\{\\bf u\}andQ𝐮Q\_\{\\bf u\}gives us

1L​𝔼​\{sup𝐮∈𝐒P𝐮\}≤σβα\(L\)\.\\dfrac\{1\}\{L\}\\mathbb\{E\}\\left\\\{\\sup\_\{\{\\bf u\}\\in\{\\bf S\}\}P\_\{\\bf u\}\\right\\\}\\leq\\dfrac\{\\sigma\}\{\\beta\_\{\\alpha\}^\{\(L\)\}\}\.\(127\)and completesStep 3\. Finally,Step 4\(Gaussian concentration\) can be done completely similar to Subsection[2\.6](https://arxiv.org/html/2608.14638#S2.SS6)\.

### B: Singularities of implicit analytic function and radius of convergence

Recall that𝐌L,σ2\(N\)\{\\bf M\}^\{\(N\)\}\_\{L,\\sigma^\{2\}\}isN×NN\\times Nnonnegative matrix,ν𝐌L,σ2\\nu\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}is its limiting NCM,m𝐌L,σ2​\(z\)m\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(z\)is the moment generating function:

m𝐌L,σ2​\(z\)=∑k=1∞ak​zk,ak=∫−∞∞λk​𝑑ν𝐌L,σ2​\(λ\),m\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(z\)=\\sum\_\{k=1\}^\{\\infty\}a\_\{k\}z^\{k\},\\quad a\_\{k\}=\\int\_\{\-\\infty\}^\{\\infty\}\\lambda^\{k\}\\,d\\nu\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(\\lambda\),\(128\)RRis the radius of convergence ofm𝐌L,σ2​\(z\)m\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(z\)\. Let us explain how to findRRusing standard singularity analysis of implicitly given analytic function\.

Sincesupp​ν𝐌L,σ2⊂\[0;\+∞\)\\text\{supp\}\\,\\nu\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\\subset\[0;\+\\infty\), we haveak≥0a\_\{k\}\\geq 0\. Vivanti\-Pringsheim theorem then yields thatm𝐌L,σ2​\(z\)m\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(z\)has singularity atz=Rz=R\. Also observe thatm𝐌L,σ2​\(z\)m\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(z\)is monotone increasing on\[0,R\)\[0,R\)and satisfies equation

z=\(2σ2\)L​m𝐌L,σ2​\(z\)\(m𝐌L,σ2​\(z\)\+1\)⋅∏k=1L\(2​ck​m𝐌L,σ2​\(z\)\+1\),z=\\left\(\\dfrac\{2\}\{\\sigma^\{2\}\}\\right\)^\{L\}\\dfrac\{m\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(z\)\}\{\(m\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(z\)\+1\)\\cdot\\prod\\limits\_\{k=1\}^\{L\}\(2c\_\{k\}m\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(z\)\+1\)\},\(129\)thus there existm∗=limz→R−m𝐌L​\(z\)m\_\{\*\}=\\lim\\limits\_\{z\\to R^\{\-\}\}m\_\{\\scriptscriptstyle\{\\bf M\}\_\{L\}\}\(z\),m∗∈\(0;\+∞\)m\_\{\*\}\\in\(0;\+\\infty\)\. Passing to the limit asz→R−z\\to R^\{\-\}in \([87](https://arxiv.org/html/2608.14638#S3.E87)\), we obtain

ℱ​\(m∗,R\)=0\.\\mathcal\{F\}\(m\_\{\*\},R\)=0\.\(130\)If∂ℱ∂m​\(m∗,R\)≠0\\dfrac\{\\partial\\mathcal\{F\}\}\{\\partial m\}\(m\_\{\*\},R\)\\neq 0, then implicit function theorem implies that there exist unique analytic solutionm​\(z\)m\(z\)of \([87](https://arxiv.org/html/2608.14638#S3.E87)\) in the neighborhood ofz=Rz=Rsatisfyingm​\(R\)=m∗m\(R\)=m\_\{\*\}\. Clearly,m​\(z\)m\(z\)is analytic continuation ofm𝐌L,σ2​\(z\)m\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(z\)in the neighborhood ofz=Rz=R, which is an obvious contradiction to singularity ofz=Rz=Rform𝐌L,σ2​\(z\)m\_\{\\scriptscriptstyle\{\\bf M\}\_\{L,\\sigma^\{2\}\}\}\(z\)\. As a conclusion, we proved existence ofm∗\>0m\_\{\*\}\>0such thatm=m∗m=m\_\{\*\},z=Rz=Ris a solution of

\{ℱ​\(m,z\)=0;∂ℱ∂m​\(m,z\)=0\.\\begin\{cases\}\\mathcal\{F\}\(m,z\)=0;\\\\ \\dfrac\{\\partial\\mathcal\{F\}\}\{\\partial m\}\(m,z\)=0\.\\end\{cases\}\(131\)

### C: Proof of Lemma[4\.1](https://arxiv.org/html/2608.14638#S4.Thmtheorem1)

Observe that foru∈𝕊n−1u\\in\\mathbb\{S\}^\{n\-1\}we have

ρ​\(g,u\)\+μ​\(h,ϕα​\(u\)\)=∑uk≥0\(ρ​gk\+μ​hk\)​uk\+∑uk<0\(−ρ​gk−μ​α​hk\)​\|uk\|≤≤∑k=1nmax⁡\{ρ​gk\+μ​hk;−ρ​gk−μ​α​hk;0\}​\|uk\|≤\(∑k=1n\(max⁡\{ρ​gk\+μ​hk;−ρ​gk−μ​α​hk;0\}\)2\)1/2\.\\rho\\,\(g,u\)\+\\mu\(h,\\phi\_\{\\alpha\}\(u\)\)=\\sum\_\{u\_\{k\}\\geq 0\}\(\\rho\\,g\_\{k\}\+\\mu\\,h\_\{k\}\)u\_\{k\}\+\\sum\_\{u\_\{k\}<0\}\(\-\\rho\\,g\_\{k\}\-\\mu\\alpha\\,h\_\{k\}\)\|u\_\{k\}\|\\leq\\\\ \\leq\\sum\_\{k=1\}^\{n\}\\max\\\{\\rho\\,g\_\{k\}\+\\mu\\,h\_\{k\};\-\\rho\\,g\_\{k\}\-\\mu\\alpha\\,h\_\{k\};0\\\}\|u\_\{k\}\|\\leq\\left\(\\sum\_\{k=1\}^\{n\}\(\\max\\\{\\rho\\,g\_\{k\}\+\\mu\\,h\_\{k\};\-\\rho\\,g\_\{k\}\-\\mu\\alpha\\,h\_\{k\};0\\\}\)^\{2\}\\right\)^\{1/2\}\.\(132\)Denotegk′:=ρ​gk\+μ​hkg\_\{k\}^\{\\prime\}:=\\rho\\,g\_\{k\}\+\\mu\\,h\_\{k\},gk′′:=−ρ​gk−μ​α​hkg\_\{k\}^\{\\prime\\prime\}:=\-\\rho\\,g\_\{k\}\-\\mu\\alpha\\,h\_\{k\}, thengk′∼N​\(0,ρ2\+μ2\)g\_\{k\}^\{\\prime\}\\sim N\(0,\\rho^\{2\}\+\\mu^\{2\}\),gk′′∼N​\(0,ρ2\+μ2​α2\)g\_\{k\}^\{\\prime\\prime\}\\sim N\(0,\\rho^\{2\}\+\\mu^\{2\}\\alpha^\{2\}\)\. Application of Cauchy\-Schwarz for expectation provides us

𝔼​\{supu∈𝕊n−1\(ρ​\(g,u\)\+μ​\(h,ϕα​\(u\)\)\)\}≤\(∑k=1n𝔼​\{\(max⁡\{gk′;gk′′;0\}\)2\}\)1/2\.\\mathbb\{E\}\\left\\\{\\sup\_\{u\\in\\mathbb\{S\}^\{n\-1\}\}\\Bigl\(\\rho\\,\(g,u\)\+\\mu\(h,\\phi\_\{\\alpha\}\(u\)\)\\Bigr\)\\right\\\}\\leq\\left\(\\sum\_\{k=1\}^\{n\}\\mathbb\{E\}\\left\\\{\(\\max\\\{g\_\{k\}^\{\\prime\};g\_\{k\}^\{\\prime\\prime\};0\\\}\)^\{2\}\\right\\\}\\right\)^\{1/2\}\.\(133\)One can check that\(max⁡\{x,y,0\}\)2≤\(x\+\)2\+\(y\+\)2\(\\max\\\{x,y,0\\\}\)^\{2\}\\leq\(x\_\{\+\}\)^\{2\}\+\(y\_\{\+\}\)^\{2\}\. Since

𝔼​\{\(\(gk′\)\+\)2\}=12​𝔼​\{\(gk′\)2\}=12​\(ρ2\+μ2\),𝔼​\{\(\(gk′′\)\+\)2\}=12​\(ρ2\+μ2​α2\),\\mathbb\{E\}\\\{\(\(g\_\{k\}^\{\\prime\}\)\_\{\+\}\)^\{2\}\\\}=\\dfrac\{1\}\{2\}\\mathbb\{E\}\\\{\(g\_\{k\}^\{\\prime\}\)^\{2\}\\\}=\\dfrac\{1\}\{2\}\(\\rho^\{2\}\+\\mu^\{2\}\),\\qquad\\mathbb\{E\}\\\{\(\(g\_\{k\}^\{\\prime\\prime\}\)\_\{\+\}\)^\{2\}\\\}=\\dfrac\{1\}\{2\}\(\\rho^\{2\}\+\\mu^\{2\}\\alpha^\{2\}\),\(134\)we finally obtain

𝔼​\{supu∈𝕊n−1\(ρ​\(g,u\)\+μ​\(h,ϕα​\(u\)\)\)\}≤\(n⋅\(12​\(ρ2\+μ2\)\+12​\(ρ2\+μ2​α2\)\)\)1/2,\\mathbb\{E\}\\left\\\{\\sup\_\{u\\in\\mathbb\{S\}^\{n\-1\}\}\\Bigl\(\\rho\\,\(g,u\)\+\\mu\(h,\\phi\_\{\\alpha\}\(u\)\)\\Bigr\)\\right\\\}\\leq\\left\(n\\cdot\\Bigl\(\\dfrac\{1\}\{2\}\(\\rho^\{2\}\+\\mu^\{2\}\)\+\\dfrac\{1\}\{2\}\(\\rho^\{2\}\+\\mu^\{2\}\\alpha^\{2\}\)\\Bigr\)\\right\)^\{1/2\},\(135\)which completes the proof of Lemma[4\.1](https://arxiv.org/html/2608.14638#S4.Thmtheorem1)\.

## References

- \[1\]L\. Berlyand, P\.\-E\. Jabin, Mathematics of Deep Learning: An Introduction, Walter de Gruyter GmbH & Co KG, 132 pages \(2023\)\.
- \[2\]L\. Berlyand, O\. Krupchytskyi, V\. Slavin, Random weights of DNNs and emergence of fixed points, Networks and Heterogeneous Media21\(1\), 170\-182 \(2026\)\.
- \[3\]C\. Borell, The Brunn\-Minkowski inequality in Gauss space, Invent\. Math\.,30, 205–216 \(1975\)\.
- \[4\]R\. Couillet, W\. Hachem, Analysis of the limiting spectral measure of large random matrices of the separable covariance type\. Random Matrices: Theory and Applications, 03\(04\):1450016 \(2014\)\.
- \[5\]E\. Dent, J\. Tanner, How Controlling the Variance can Improve Training Stability of Sparsely Activated DNNs and CNNs, arXiv:2602\.05779 \(2026\)
- \[6\]X\. Fernique, Fonctions aléatoires gaussiennes vecteurs aléatoires gaussiens\. CRM, Montreal \(1997\)\.
- \[7\]F\.L\. Gewers, G\.R\. Ferreira, H\.F\. de Arruda, F\.N\. Silva, C\.H\. Comin, D\.R\. Amancio, L\. da F\. Costa, Principal Component Analysis: A Natural Approach to Data Exploration, ACM Computing Surveys \(CSUR\),54\(4\), 1\-34 \(2021\)\.
- \[8\]X\. Glorot Y\. Bengio, Understanding the difficulty of training deep feedforward neural networks, Proceedings of Machine Learning Research9249\-256 \(2010\)
- \[9\]I\. Goodfellow, Y\. Bengio and A\. Courville, Deep Learning, MIT Press \(2016\)
- \[10\]B\. Hanin, Which Neural Net Architectures Give Rise to Exploding and Vanishing Gradients? Neural Information Processing Systems \(2018\)\.
- \[11\]B\. Hanin, M\. Nica, Products of Many Large Random Matrices and Gradients in Deep Neural Networks, Commun\. Math\. Phys\.376, 287–322 \(2020\)\.
- \[12\]B\. Hanin, T\. Jiang, Global Universality of Singular Values in Products of Many Large Random Matrices, arXiv:2503\.07872 \(2025\)\.
- \[13\]J\. Hoffman, D\.A\. Roberts, Sh\. Yaida, Robust Learning with Jacobian Regularization, arXiv:1908\.02729v1 \(2019\)\.
- \[14\]T\. Nait\-Saada, J\. Tanner, On the Initialisation of Wide Low\-Rank Feedforward Neural Networks, arXiv:2301\.13710 \(2023\)\.
- \[15\]T\. Nait\-Saada, A\. Naderi, J\. Tanner, Beyond IID weights: sparse and low\-rank deep Neural Networks are also Gaussian Processes, International Conference on Learning Representation \(2024\)\.
- \[16\]L\. Pastur, V\. Slavin, On random matrices arising in deep neural networks: General iid case, Random Matrices: Theory and Applications12\(01\), 2250046 \(2023\)\.
- \[17\]L\. Pastur, On random matrices arising in deep neural networks: Gaussian case, Pure Appl\. Funct\. Anal\.5\(6\), 1395—1424 \(2020\)\.
- \[18\]D\. Paul, J\. W\. Silverstein, No eigenvalues outside the support of the limiting empirical spectral distribution of a separable covariance matrix, J\. Multivariate Anal\.,100, 1, 37\-57 \(2009\)\.
- \[19\]J\. Pennington, S\. S\. Schoenholz, S\. Ganguli, The Emergence of Spectral Universality in Deep Networks, Proceedings of the Twenty\-First International Conference on Artificial Intelligence and Statistics\), PMLR 84:1924\-1932 \(2018\)\.
- \[20\]J\. Pennington, S\. S\. Schoenholz, S\. Ganguli, Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice\. In Proceedings of the 31st International Conference on Neural Information Processing Systems \(NIPS’17\)\. Curran Associates Inc\., Red Hook, NY, USA, 4788–4798 \(2017\)\.
- \[21\]B\. Poole, S\. Lahiri, M\. Raghu, J\. Sohl\-Dickstein, S\. Ganguli\. Exponential expressivity in deep neural networks through transient chaos\. Neural Information Processing Systems \(2016\)\.
- \[22\]M\.J\.E\. Richardson, Linear and non\-linear integrate\-and\-fire neurons driven by synaptic shot noise with reversal potentials, Phys\. Rev\. E109, 024407 \(2024\)\.
- \[23\]L\. Scabini, B\. De Baets, O\.M\. Bruno, Improving deep neural network random initialization through neuronal rewiring, Neurocomputing599\(28\) 128130\(2024\)\.
- \[24\]E\.S\. Schaffer, S\. Ostojic, L\.F\. Abbott, A Complex\-Valued Firing\-Rate Model That Approximates the Dynamics of Spiking Networks, Plos Computational biology,9\(10\), e1003301 \(2013\)\.
- \[25\]J\. Schmidhuber, Deep Learning in Neural Networks: An Overview, Neural Networks61, 85\-117 \(2015\)\.
- \[26\]V\. Slavin, Autoencoder deep neural networks and their applications in solid\-state physics, Low Temperature Physics52\(6\), 660\-666 \(2026\)\.
- \[27\]V\.N\. Sudakov, Gaussian random processes and measures of solid angles in Hilbert space\. Dokl\. Akad\. Nauk\. SSR197, 4345\.; English translation in Soviet Math\. Dokl\.12, 412–415 \(1971\)\.
- \[28\]V\.N\. Sudakov, Geometric Problems in the Theory of Infinite\-Dimensional Probability Distributions\. Trud\. Mat\. Inst\. Steklov141\. English translation in Proc\. Steklov Inst\. Math2, Amer\. Math\. Soc\. \(1976\)\.
- \[29\]B\.S\. Tsirelson, I\.A\. Ibragimov, V\.N\. Sudakov, Norms of Gaussian sample functions, in Proceedings of the 3rd Japan\-USSR Symposium on Probability Theory \(Tashkent, 1975\), Lecture Notes in Mathematics, Vol\. 550, Springer\-Verlag, Berlin, 1976, 20–41\.
- \[30\]G\. Wainrib, J\. Touboul, Topological and Dynamical Complexity of Random Neural Networks, PRL110, 118101 \(2013\)\.
- \[31\]H\. Wang, Ch\. Zheng, Macroscopic dynamics of quadratic integrate\-and\-fire neurons subject to correlated noise, arXiv:2601\.10032 \(2026\)\.
- \[32\]V\. Zhu, R\. Rosenbaum, Learning fixed points of recurrent neural networks by reparameterizing the network model, arXiv:2307\.06732 \(2023\)\.