Sphere Retraction Normalizations

# Sphere Retraction Normalizations
Source: [https://arxiv.org/html/2608.02668](https://arxiv.org/html/2608.02668)
1\]Department of Computer Science and Information Engineering, National Central University, Taiwan 2\]Department of Applied Mathematics, National Yang Ming Chiao Tung University, Taiwan 3\]Department of Mathematics, National Central University, Taiwan\\contribution\[\*\]Equal contribution

\(August 2, 2026\)

###### Abstract

Residual connections are the de facto mechanism for training deep neural networks stably\. Geodesic Normalization \(GeoNorm\) recasts them on a Riemannian manifold, orthogonalizing each layer output against the current hidden state and applying the resulting update through the Riemannian exponential map\. Every hidden state thus keeps a constantℓ2\\ell\_\{2\}\-norm, confining the residual stream to a hypersphere\. The exponential map, however, is only one member of a broad family of retraction maps\. We show that on the hypersphere this entire family collapses to a single scalar design choice\. What distinguishes one retraction from another is only how the magnitude of an update is converted into a rotation angle within the plane spanned by the hidden state and the update\. This view places Euclidean residual connections and GeoNorm in one framework\. Instantiating it with the metric projection retraction and the Cayley retraction yieldsProj\-SpheretNormandCay\-SpheretNorm, which are exactly norm\-preserving yet require only algebraic operations\. Both prove to be members of a one\-parameter family of angular retractions,pp\-SpheretNorm, whose rotation angle saturates rather than growing without bound\. The two methods above are recovered exactly atp=1p=1andp=2p=2, while the identity map and GeoNorm arise only as limits at either end\. On nanoGPT, all three methods outperform existing lightweight deep connection schemes, and the best validation loss is attained at finitepp, indicating that the exponential map is not the preferred retraction for spherical residual streams but merely one end of a spectrum\.

††preprint:Preprint\\ncudata

\[Code\]\\codeurlhttps://github\.com/hazdzz/deep\_connection\\ncudata\[Correspondence\]Jie Zhang \(\), Cheng\-Fang Su \(\)

> “Geometry is not true, it is advantageous\.” — Henri Poincaré

## 1Introduction

Training neural networks in a deep and stable manner remains an open problem\. Prior research\(Li et al\.,[2025](https://arxiv.org/html/2608.02668#bib.bib20); Kim et al\.,[2025](https://arxiv.org/html/2608.02668#bib.bib19); Chen and Wei,[2026](https://arxiv.org/html/2608.02668#bib.bib6)\)primarily tackles this challenge by combining skip connections\(He et al\.,[2016b](https://arxiv.org/html/2608.02668#bib.bib15),[a](https://arxiv.org/html/2608.02668#bib.bib14)\), normalization methods\(Ba et al\.,[2016](https://arxiv.org/html/2608.02668#bib.bib3); Zhang and Sennrich,[2019](https://arxiv.org/html/2608.02668#bib.bib41)\), and gating mechanisms\(Hochreiter and Schmidhuber,[1997](https://arxiv.org/html/2608.02668#bib.bib16); Cho et al\.,[2014](https://arxiv.org/html/2608.02668#bib.bib8)\)\. We refer to this class of methodologies collectively as*deep connections*\. In the Transformer architecture\(Vaswani et al\.,[2017](https://arxiv.org/html/2608.02668#bib.bib35)\), Post\-LayerNorm \(Post\-LN\)\(Vaswani et al\.,[2017](https://arxiv.org/html/2608.02668#bib.bib35)\)and Pre\-LayerNorm \(Pre\-LN\)\(Wang et al\.,[2019](https://arxiv.org/html/2608.02668#bib.bib36); Xiong et al\.,[2020](https://arxiv.org/html/2608.02668#bib.bib39)\)have emerged as the two most common choices for the placement of deep connections\.

However, both Pre\-LN and Post\-LN exhibit inherent limitations\. Transformers with Pre\-LN suffer from the curse of depth\(Sun et al\.,[2025](https://arxiv.org/html/2608.02668#bib.bib33)\), where the contribution of deeper layers progressively diminishes\. Moreover, the unbounded residual stream in Pre\-LN allows massive activations to accumulate across layers, which has been linked to more pronounced attention sinks — a phenomenon in which attention scores concentrate excessively on the first token\(Xiao et al\.,[2024](https://arxiv.org/html/2608.02668#bib.bib38); Sun et al\.,[2024](https://arxiv.org/html/2608.02668#bib.bib32); Gu et al\.,[2025](https://arxiv.org/html/2608.02668#bib.bib13)\)\. By contrast, Transformers with Post\-LN, on the other hand, suffer from large gradients near the output layer at initialization\(Xiong et al\.,[2020](https://arxiv.org/html/2608.02668#bib.bib39)\), which destabilizes training and necessitates a carefully tuned learning\-rate warm\-up schedule that slows down optimization and adds hyperparameter tuning\.

![Refer to caption](https://arxiv.org/html/2608.02668v1/figures/spheretnorm.png)\(a\)The whole process of SpheretNorm\.ℱl​\(⋅\)\\mathcal\{F\}\_\{l\}\(\\cdot\)represents a \(non\-\)linear mapping, the GS process stands for the Gram–Schmidt process, andℛl​\(⋅\)\\mathcal\{R\}\_\{l\}\(\\cdot\)denotes the retraction map\.
![Refer to caption](https://arxiv.org/html/2608.02668v1/figures/retraction.png)\(b\)Illustration of the geometric meaning of SpheretNorm\. The retraction mapℛl​\(⋅\)\\mathcal\{R\}\_\{l\}\(\\cdot\)is defined on the hypersphere of radius∥𝐡l∥2\\lVert\{\\mathbf\{h\}\_\{l\}\}\\rVert\_\{2\}\.

Figure 1:Overview of SpheretNorm \(including GeoNorm, Proj\-SpheretNorm, Cay\-SpheretNorm, andpp\-SpheretNorm\)\. \(a\) The overall computational pipeline, and \(b\) its underlying geometric interpretation\.A recent line of work\(Li et al\.,[2025](https://arxiv.org/html/2608.02668#bib.bib20); Kim et al\.,[2025](https://arxiv.org/html/2608.02668#bib.bib19); Chen and Wei,[2026](https://arxiv.org/html/2608.02668#bib.bib6)\)seeks to reconcile the Post\-LN and Pre\-LN paradigms within Euclidean space\.Zheng et al\. \([2026](https://arxiv.org/html/2608.02668#bib.bib42)\)pursue a similar goal but take a distinct route: their Geodesic Normalization \(GeoNorm\) accomplishes the residual connection on a Riemannian manifold\. Let𝒉l\\bm\{h\}\_\{l\}denote thell\-th hidden state, and letℱl​\(⋅\)\\mathcal\{F\}\_\{l\}\(\\cdot\)denote the corresponding \(non\-\)linear mapping\. Specifically, GeoNorm first orthogonalizesℱl​\(𝒉l\)\\mathcal\{F\}\_\{l\}\(\\bm\{h\}\_\{l\}\)against𝒉l\\bm\{h\}\_\{l\}via a Gram\-Schmidt process to obtain an intermediate tensor𝒛l\\bm\{z\}\_\{l\}, and then applies an exponential map at𝒉l\\bm\{h\}\_\{l\}along𝒛l\\bm\{z\}\_\{l\}to produce𝒉l\+1\\bm\{h\}\_\{l\+1\}\. By keeping theℓ2\\ell\_\{2\}\-norm of every hidden state constant across layers, GeoNorm in effect confines the hidden states to a hypersphere, which stabilizes forward propagation\.

While GeoNorm offers a compelling geometric perspective, its update is tied to one specific choice: the exponential map\. From the viewpoint of Riemannian geometry, however, the exponential map is merely one member of a broad family of retraction maps\(Absil et al\.,[2007](https://arxiv.org/html/2608.02668#bib.bib2); Absil and Malick,[2012](https://arxiv.org/html/2608.02668#bib.bib1)\)\. Thus, a natural question arises:*which retraction should a deep connection use on the hypersphere, and how does this choice shape the behavior of a deep neural network?*

Rather than proposing another way to unify Post\-LN and Pre\-LN into a single normalization mechanism, we revisit retraction maps on the hypersphere and propose threeSphereRetractionNormalizations \(SpheretNorms\):Proj\-SpheretNorm,Cay\-SpheretNorm, and\\textbfpspp\-SpheretNorm\. We first unify residual connection and GeoNorm within a single framework, under which GeoNorm can be interpreted as a Riemannian residual connection on the hypersphere\. Building upon this perspective, we further exploit the metric projection retraction and the Cayley retraction on the hypersphere to derive Proj\-SpheretNorm and Cay\-SpheretNorm, respectively\.

We then observe that both Proj\-SpheretNorm and Cay\-SpheretNorm can be subsumed into a unified formulation based on the exponential map with different effective angles\. We refer to this generalized formulation aspp\-SpheretNorm with a positive parameterpp, which recovers Proj\-SpheretNorm atp=1p=1and Cay\-SpheretNorm atp=2p=2\. Moreover, asp→0\+p\\to 0^\{\+\},pp\-SpheretNorm reduces to the identity mapping, while asp→\+∞p\\to\+\\inftywithαl∈\(0,1\]\\alpha\_\{l\}\\in\(0,1\], it recovers GeoNorm\. Through a single tunable parameter,pp\-SpheretNorm provides a principled and flexible family of normalization layers that bridges existing methods\.

We evaluate three types of SpheretNorm using nanoGPT\(Karpathy,[2022](https://arxiv.org/html/2608.02668#bib.bib18)\)as the backbone on both pretraining and downstream tasks\. Proj\-SpheretNorm achieves the best results, attaining the lowest pretraining and validation losses on the medium\- and large\-size nanoGPT models\. Results on the downstream tasks also show that Proj\-SpheretNorm achieves high accuracy\. Taken together, these results indicate that Proj\-SpheretNorm is the strongest of the three SpheretNorm variants across both pretraining and downstream evaluations\.

##### Contributions\.

In conclusion, our key contributions are summarized as follows:

1. 1\.We unify residual connections and GeoNorm as retraction\-based updates, showing that GeoNorm amounts to a Riemannian residual connection driven by the exponential map on the hypersphere\.
2. 2\.We instantiate two alternative retractions on the hypersphere, the metric projection retraction and the Cayley retraction, which yield Proj\-SpheretNorm and Cay\-SpheretNorm\.
3. 3\.We generalize both instances intopp\-SpheretNorm, a single\-parameter family that reduces to the identity mapping asp→0\+p\\to 0^\{\+\}, recovers Proj\-SpheretNorm atp=1p=1and Cay\-SpheretNorm atp=2p=2, and recovers GeoNorm asp→\+∞p\\to\+\\infty\.
4. 4\.On nanoGPT at up to 36 layers and6\.556\.55B tokens, SpheretNorm attains the lowest pre\-training and validation losses at 24 and 36 layers and the highest average zero\-shot accuracy on both OpenWebText and FineWeb\-Edu, outperforming Pre\-LN, Pre\-DyT, Peri\-LN, Keel, and GeoNorm, while converging in every setting in which Pre\-LN or Keel diverges\.

## 2Related Work

##### Normalization Methods\.

LayerNorm\(Ba et al\.,[2016](https://arxiv.org/html/2608.02668#bib.bib3)\)has served as the foundation of Transformers\(Vaswani et al\.,[2017](https://arxiv.org/html/2608.02668#bib.bib35)\)since its inception\. Later, RMSNorm\(Zhang and Sennrich,[2019](https://arxiv.org/html/2608.02668#bib.bib41)\), a simplified variant of LayerNorm that drops the re\-centering operation, became the de facto standard normalization\. A closely related and contemporaneous normalization method, ScaleNorm\(Nguyen and Salazar,[2019](https://arxiv.org/html/2608.02668#bib.bib25)\), differs mainly in that it normalizes by theℓ2\\ell\_\{2\}\-norm rather than the root mean square, and replaces the per\-dimension gain vector with a single tied scalar, making it more parameter\-efficient\.

##### Normalization\-Free Normalization Functions\.

A noteworthy line of work replaces LayerNorm with a zero\-centered, bounded, center\-sensitive, monotonic element\-wise non\-linearity\.Zhu et al\. \([2025](https://arxiv.org/html/2608.02668#bib.bib43)\)propose the Dynamic Tanh \(DyT\) function,Stollenwerk \([2026](https://arxiv.org/html/2608.02668#bib.bib31)\)introduce the Dynamic Inverse Square Root Unit \(DyISRU\) function, andChen et al\. \([2026](https://arxiv.org/html/2608.02668#bib.bib7)\)develop the Dynamic erf \(Derf\) function\. By eliminating the reduction along the hidden dimension, these activation functions remove the synchronization bottleneck of LayerNorm and can be fused with the preceding linear layer, substantially reducing overhead while preserving the squashing behavior of LayerNorm\.

##### Skip Connection, and its Variants\.

ResNet\(He et al\.,[2016a](https://arxiv.org/html/2608.02668#bib.bib14),[b](https://arxiv.org/html/2608.02668#bib.bib15)\)ushered in the era of very deep neural networks\. Later,Bachlechner et al\. \([2021](https://arxiv.org/html/2608.02668#bib.bib4)\)proposed ReZero, a residual connection with a learnable step sizeαl\\alpha\_\{l\}initialized at zero\.Touvron et al\. \([2021](https://arxiv.org/html/2608.02668#bib.bib34)\)further enhanced ReZero by using a learnable vector instead of a scalar\. Recently,Zheng et al\. \([2026](https://arxiv.org/html/2608.02668#bib.bib42)\)proposed GeoNorm, a Riemannian residual connection that preserves theℓ2\\ell\_\{2\}\-norm of each hidden state throughout forward propagation\. Inspired by the property of GeoNorm that maintains a constantℓ2\\ell\_\{2\}\-norm across hidden layers, we revisit retraction maps on the hypersphere, and propose SpheretNorms\.

##### Post\-LN, Pre\-LN, and their Variants\.

With the advent of Transformers, applying a residual connection followed by LayerNorm, known as Post\-LN, became standard practice\. However, Transformers using Post\-LN typically rely on a carefully tuned learning\-rate warm\-up schedule\. In contrast, Pre\-LN applies LayerNorm before each sub\-layer and adds the residual connection afterward\. While more stable, Transformers with Pre\-LN suffer from the curse of depth\(Sun et al\.,[2025](https://arxiv.org/html/2608.02668#bib.bib33)\), where deeper layers contribute far less to learning and representation than earlier ones, effectively widening the network rather than deepening it\. LayerNorm Scaling \(LNS\)\(Sun et al\.,[2025](https://arxiv.org/html/2608.02668#bib.bib33)\)attenuates the normalized hidden state by a depth\-dependent factor before each sub\-layer, mitigating the curse of depth\. In addition to the conventional Pre\-LN and Post\-LN designs, Peri\-LN\(Kim et al\.,[2025](https://arxiv.org/html/2608.02668#bib.bib19)\)and Keel\(Chen and Wei,[2026](https://arxiv.org/html/2608.02668#bib.bib6)\)apply LayerNorm twice to reconcile the two paradigms\. Collectively, these works explore different placements and scalings of LayerNorm within the residual stream to stabilize and deepen Transformers, yet they all operate within the Euclidean geometry of the hidden representations\.

## 3Background and Preliminary

Let𝐡l\\mathbf\{h\}\_\{l\}denote thell\-th hidden state, with𝐡1\\mathbf\{h\}\_\{1\}being the input feature \(or the output of the embedding layer, for Transformers\), and letℱl​\(⋅\)\\mathcal\{F\}\_\{l\}\(\\cdot\)denote thell\-th \(non\-\)linear mapping in Euclidean space, such as multi\-head self\-attention\(Vaswani et al\.,[2017](https://arxiv.org/html/2608.02668#bib.bib35)\)or a multi\-layer perceptron\. Unless otherwise specified, for notational convenience, all bias terms are omitted throughout this work\. Moreover, we assume𝐡l∈ℝd\\mathbf\{h\}\_\{l\}\\in\\mathbb\{R\}^\{d\}and𝐡l≠𝟎\\mathbf\{h\}\_\{l\}\\neq\\mathbf\{0\}\. Forl=1,2,…,L−1l=1,\\,2,\\,\\ldots,\\,L\-1, we take two instances of deep connection methods for illustrating\.

##### Residual Connection \(RC\)\.

To facilitate the training of deep CNNs,He et al\. \([2016a](https://arxiv.org/html/2608.02668#bib.bib14),[b](https://arxiv.org/html/2608.02668#bib.bib15)\)propose ResNet, whose key innovation is the residual connection\. It can be formulated as

𝐡l\+1=𝐡l\+ℱl​\(𝐡l\)\.\\mathbf\{h\}\_\{l\+1\}=\\mathbf\{h\}\_\{l\}\+\\mathcal\{F\}\_\{l\}\(\\mathbf\{h\}\_\{l\}\)\.\(1\)

##### Geodesic Normalization \(GeoNorm\)\.

The exponential map in Riemannian geometry is defined as

exp𝐡⁡\(𝐯\)=cos⁡\(∥𝐯∥2∥𝐡∥2\)​𝐡\+∥𝐡∥2​sin⁡\(∥𝐯∥2∥𝐡∥2\)​𝐯∥𝐯∥2,\\exp\_\{\\mathbf\{h\}\}\(\\mathbf\{v\}\)=\\cos\(\\frac\{\\lVert\{\\mathbf\{v\}\}\\rVert\_\{2\}\}\{\\lVert\{\\mathbf\{h\}\}\\rVert\_\{2\}\}\)\\mathbf\{h\}\+\\lVert\{\\mathbf\{h\}\}\\rVert\_\{2\}\\sin\(\\frac\{\\lVert\{\\mathbf\{v\}\}\\rVert\_\{2\}\}\{\\lVert\{\\mathbf\{h\}\}\\rVert\_\{2\}\}\)\\frac\{\\mathbf\{v\}\}\{\\lVert\{\\mathbf\{v\}\}\\rVert\_\{2\}\},\(2\)with𝐡⊤​𝐯=0\\mathbf\{h\}^\{\\top\}\\mathbf\{v\}=0\. Building upon this exponential map,Zheng et al\. \([2026](https://arxiv.org/html/2608.02668#bib.bib42)\)propose GeoNorm, which is defined as

𝐳l=\(𝐈−𝐡l​𝐡l⊤∥𝐡l∥2\)​ℱl​\(𝐡l\),βl=αl​∥𝐳l∥2∥𝐡l∥2,𝐡l\+1=cos⁡\(βl\)​𝐡l\+∥𝐡l∥2​sin⁡\(βl\)​𝐳l∥𝐳l∥2\.\\begin\{split\}\\mathbf\{z\}\_\{l\}&=\\Bigl\(\\mathbf\{I\}\-\\frac\{\\mathbf\{h\}\_\{l\}\\mathbf\{h\}\_\{l\}^\{\\top\}\}\{\\lVert\{\\mathbf\{h\}\_\{l\}\}\\rVert^\{2\}\}\\Bigr\)\\mathcal\{F\}\_\{l\}\(\\mathbf\{h\}\_\{l\}\),\\quad\\beta\_\{l\}=\\alpha\_\{l\}\\frac\{\\lVert\{\\mathbf\{z\}\_\{l\}\}\\rVert\_\{2\}\}\{\\lVert\{\\mathbf\{h\}\_\{l\}\}\\rVert\_\{2\}\},\\\\ \\mathbf\{h\}\_\{l\+1\}&=\\cos\(\\beta\_\{l\}\)\\mathbf\{h\}\_\{l\}\+\\lVert\{\\mathbf\{h\}\_\{l\}\}\\rVert\_\{2\}\\sin\(\\beta\_\{l\}\)\\frac\{\\mathbf\{z\}\_\{l\}\}\{\\lVert\{\\mathbf\{z\}\_\{l\}\}\\rVert\_\{2\}\}\.\\end\{split\}\(3\)Here,αl∈\(0,1\]\\alpha\_\{l\}\\in\(0,1\]is a learnable parameter111The range ofαl\\alpha\_\{l\}is not explicitly specified in\(Zheng et al\.,[2026](https://arxiv.org/html/2608.02668#bib.bib42)\)\. Given thatαl\\alpha\_\{l\}acts as the step size, we deduce its range from this property\.\.

## 4Proposed Method

### 4\.1Sphere Retraction Normalizations

In this subsection, we begin by casting the standard residual connection and GeoNorm into a common framework that exposes their shared computational structure and distinguishes their geometric components\. We single out the retraction map for redesign and consider alternatives to the exponential map\. Instantiating the framework with the metric projection and Cayley retractions yields Proj\-SpheretNorm and Cay\-SpheretNorm, respectively\. We then introduce a unified one\-parameter construction, termedpp\-SpheretNorm, which recovers these two methods as exact special cases\.

#### 4\.1\.1Retraction Maps on the Hypersphere

In a deep neural network, thell\-th block maps𝐡l−1\\mathbf\{h\}\_\{l\-1\}to𝐡l\\mathbf\{h\}\_\{l\}, forl=1,…,Ll=1,\\ldots,L\. Given a manifoldℳ\\mathcal\{M\}, the tangent\-space projectionprojT𝐡​ℳ\\operatorname\{proj\}\_\{T\_\{\\mathbf\{h\}\}\\mathcal\{M\}\}, a step sizeαl\\alpha\_\{l\}, and a retractionℛ𝐡\\mathcal\{R\}\_\{\\mathbf\{h\}\}, a deep connection can be written abstractly as

𝐡l=ℛ𝐡l−1​\[αl⋅projT𝐡l−1​ℳ⁡\(ℱl​\(𝐡l−1\)\)\]\.\\mathbf\{h\}\_\{l\}=\\mathcal\{R\}\_\{\\mathbf\{h\}\_\{l\-1\}\}\\\!\\Bigl\[\\alpha\_\{l\}\\cdot\\operatorname\{proj\}\_\{T\_\{\\mathbf\{h\}\_\{l\-1\}\}\\mathcal\{M\}\}\\bigl\(\\mathcal\{F\}\_\{l\}\(\\mathbf\{h\}\_\{l\-1\}\)\\bigr\)\\Bigr\]\.\(4\)Within this framework, RC and GeoNorm share the same computational skeleton and differ only in how the individual components are instantiated\. In particular, GeoNorm corresponds to choosing the hypersphere as the manifold and the exponential map as the retraction, which makes it a Riemannian residual connection\. Table[1](https://arxiv.org/html/2608.02668#S4.T1)contrasts the two methods component by component\.

Table 1:Residual Connection vs\. Geodesic Normalization\.RCGeoNormℳ\\mathcal\{M\}ℝd\\mathbb\{R\}^\{d\}𝕊rd−1=\{𝐡∈ℝd:∥𝐡∥2=r\}\\mathbb\{S\}^\{d\-1\}\_\{r\}=\\\{\\mathbf\{h\}\\in\\mathbb\{R\}^\{d\}:\\lVert\\mathbf\{h\}\\rVert\_\{2\}=r\\\}T𝐡​ℳT\_\{\\mathbf\{h\}\}\\mathcal\{M\}ℝd\\mathbb\{R\}^\{d\}\{𝐯∈ℝd:𝐡⊤​𝐯=0\}\\\{\\mathbf\{v\}\\in\\mathbb\{R\}^\{d\}:\\mathbf\{h\}^\{\\top\}\\mathbf\{v\}=0\\\}projT𝐡​ℳ⁡\(𝐮\)\\operatorname\{proj\}\_\{T\_\{\\mathbf\{h\}\}\\mathcal\{M\}\}\(\\mathbf\{u\}\)𝐮\\mathbf\{u\}\(𝐈−𝐡𝐡⊤∥𝐡∥22\)​𝐮\\Bigl\(\\mathbf\{I\}\-\\dfrac\{\\mathbf\{h\}\\mathbf\{h\}^\{\\top\}\}\{\\lVert\\mathbf\{h\}\\rVert\_\{2\}^\{2\}\}\\Bigr\)\\mathbf\{u\}αl\\alpha\_\{l\}11αl∈\(0,1\]\\alpha\_\{l\}\\in\(0,1\]ℛ𝐡​\(𝐯\)\\mathcal\{R\}\_\{\\mathbf\{h\}\}\(\\mathbf\{v\}\)𝐡\+𝐯\\mathbf\{h\}\+\\mathbf\{v\}exp𝐡⁡\(𝐯\)\\exp\_\{\\mathbf\{h\}\}\(\\mathbf\{v\}\)Table[1](https://arxiv.org/html/2608.02668#S4.T1)shows that GeoNorm chooses the retraction map to be the exponential map on the hypersphere\. The exponential map, however, is merely one instance of a broader class of retractions\(Absil et al\.,[2007](https://arxiv.org/html/2608.02668#bib.bib2); Absil and Malick,[2012](https://arxiv.org/html/2608.02668#bib.bib1)\)\. We replace the exponential map with other retractions keeps the framework intact while enriching the design space\. We recall the definition of a retraction below\.

###### Definition 1\(Retraction\(Absil et al\.,[2007](https://arxiv.org/html/2608.02668#bib.bib2)\)\)\.

A*retraction*on a manifoldℳ\\mathcal\{M\}is a smooth mappingℛ:T​ℳ→ℳ,\(𝐡,𝐯\)↦ℛ𝐡​\(𝐯\)\\mathcal\{R\}:T\\mathcal\{M\}\\to\\mathcal\{M\},\\ \(\\mathbf\{h\},\\mathbf\{v\}\)\\mapsto\\mathcal\{R\}\_\{\\mathbf\{h\}\}\(\\mathbf\{v\}\), such that for every𝐡∈ℳ\\mathbf\{h\}\\in\\mathcal\{M\}and every𝐯∈T𝐡​ℳ\\mathbf\{v\}\\in T\_\{\\mathbf\{h\}\}\\mathcal\{M\},ℛ𝐡​\(𝟎\)=𝐡,dd​t​ℛ𝐡​\(t​𝐯\)\|t=0=𝐯\\mathcal\{R\}\_\{\\mathbf\{h\}\}\(\\mathbf\{0\}\)=\\mathbf\{h\},\\ \\frac\{\\mathop\{\}\\\!\\mathrm\{d\}\}\{\\mathop\{\}\\\!\\mathrm\{d\}\{t\}\}\\,\\mathcal\{R\}\_\{\\mathbf\{h\}\}\(t\\mathbf\{v\}\)\\bigg\|\_\{t=0\}=\\mathbf\{v\}\.

##### Sphere Metric Projection Retraction Normalization\.

We first choose the metric projection retraction on the hypersphere\(Absil et al\.,[2007](https://arxiv.org/html/2608.02668#bib.bib2)\), also known as the gnomonic retraction on the hypersphere\(Absil and Malick,[2012](https://arxiv.org/html/2608.02668#bib.bib1)\), as the foundation for proposing a novel normalization layer, which we name the Sphere Metric Projection Retraction Normalization \(Proj\-SpheretNorm\)\. The definition of the metric projection retraction on the hypersphere is described as blows\.

###### Definition 2\(Metric Projection Retraction\(Absil et al\.,[2007](https://arxiv.org/html/2608.02668#bib.bib2)\)\)\.

Letℳ=𝕊rn−1=\{𝐡∈ℝn:‖𝐡‖2=r\}\\mathcal\{M\}=\\mathbb\{S\}\_\{r\}^\{n\-1\}=\\\{\\mathbf\{h\}\\in\\mathbb\{R\}^\{n\}:\\\|\\mathbf\{h\}\\\|\_\{2\}=r\\\}be the hypersphere of radiusr\>0r\>0, endowed with the Riemannian metric induced from the Euclidean inner product\. For each𝐡∈ℳ\\mathbf\{h\}\\in\\mathcal\{M\}, the tangent space isT𝐡​ℳ=\{𝐯∈ℝn:⟨𝐡,𝐯⟩=0\}T\_\{\\mathbf\{h\}\}\\mathcal\{M\}=\\\{\\mathbf\{v\}\\in\\mathbb\{R\}^\{n\}:\\,\\langle\\mathbf\{h\},\\mathbf\{v\}\\rangle=0\\\}, and the projective retraction is defined by

ℛ𝐡Proj​\(𝐯\)=r⋅𝐡\+𝐯∥𝐡\+𝐯∥2,𝐯∈T𝐡​ℳ\.\\mathcal\{R\}^\{\\text\{Proj\}\}\_\{\\mathbf\{h\}\}\(\\mathbf\{v\}\)=r\\cdot\\frac\{\\mathbf\{h\}\+\\mathbf\{v\}\}\{\\lVert\{\\mathbf\{h\}\+\\mathbf\{v\}\}\\rVert\_\{2\}\},\\quad\\mathbf\{v\}\\in T\_\{\\mathbf\{h\}\}\\mathcal\{M\}\.\(5\)ThenℛProj\\mathcal\{R\}^\{\\text\{Proj\}\}is a second\-order retraction onℳ\\mathcal\{M\}; that is, for every𝐡∈ℳ\\mathbf\{h\}\\in\\mathcal\{M\}and every𝐯∈T𝐡​ℳ\\mathbf\{v\}\\in T\_\{\\mathbf\{h\}\}\\mathcal\{M\}, the curvec​\(t\):=ℛ𝐡​\(t​𝐯\)c\(t\):=\\mathcal\{R\}\_\{\\mathbf\{h\}\}\(t\\mathbf\{v\}\)satisfiesc​\(0\)=𝐡,c′​\(0\)=𝐯,𝐏𝐡​\(c′′​\(0\)\)=𝟎c\(0\)=\\mathbf\{h\},\\ c^\{\\prime\}\(0\)=\\mathbf\{v\},\\ \\mathbf\{P\}\_\{\\mathbf\{h\}\}\\left\(c^\{\\prime\\prime\}\(0\)\\right\)=\\mathbf\{0\}, where𝐏𝐡:ℝn→T𝐡​ℳ\\mathbf\{P\}\_\{\\mathbf\{h\}\}:\\mathbb\{R\}^\{n\}\\to T\_\{\\mathbf\{h\}\}\\mathcal\{M\}denotes the orthogonal projection onto the tangent space\.

Based on Eq\. \([4](https://arxiv.org/html/2608.02668#S4.E4)\) and Eq\. \([5](https://arxiv.org/html/2608.02668#S4.E5)\), we have

𝐳l=\(𝐈−𝐡l​𝐡l⊤∥𝐡l∥2\)​ℱl​\(𝐡l\),𝐡l\+1=∥𝐡l∥2⋅𝐡l\+αl​𝐳l∥𝐡l\+αl​𝐳l∥2\.\\begin\{split\}\\mathbf\{z\}\_\{l\}&=\\Bigl\(\\mathbf\{I\}\-\\frac\{\\mathbf\{h\}\_\{l\}\\mathbf\{h\}\_\{l\}^\{\\top\}\}\{\\lVert\{\\mathbf\{h\}\_\{l\}\}\\rVert^\{2\}\}\\Bigr\)\\mathcal\{F\}\_\{l\}\(\\mathbf\{h\}\_\{l\}\),\\\\ \\mathbf\{h\}\_\{l\+1\}&=\\lVert\\mathbf\{h\}\_\{l\}\\rVert\_\{2\}\\cdot\\frac\{\\mathbf\{h\}\_\{l\}\+\\alpha\_\{l\}\\mathbf\{z\}\_\{l\}\}\{\\lVert\{\\mathbf\{h\}\_\{l\}\+\\alpha\_\{l\}\\mathbf\{z\}\_\{l\}\}\\rVert\_\{2\}\}\.\\end\{split\}\(6\)

##### Sphere Cayley Retraction Normalization\.

We also consider the Cayley retraction on the hypersphere\(Absil et al\.,[2007](https://arxiv.org/html/2608.02668#bib.bib2)\), also known as the stereographic retraction on the hypersphere\(Absil and Malick,[2012](https://arxiv.org/html/2608.02668#bib.bib1)\), to propose Sphere Cayley Retraction Normalization \(Cay\-SpheretNorm\)\. For a skew\-symmetric matrix𝐖∈𝔰​𝔬​\(d\)\\mathbf\{W\}\\in\\mathfrak\{so\}\(d\), where𝔰​𝔬​\(d\)=\{𝑾∈ℝd×d:𝑾⊤=−𝑾\}\\mathfrak\{so\}\(d\)=\\left\\\{\\bm\{W\}\\in\\mathbb\{R\}^\{d\\times d\}:\\bm\{W\}^\{\\top\}=\-\\bm\{W\}\\right\\\}, the Cayley transform is defined by𝒞​\(𝐖\)=\(𝐈−12​𝐖\)−1​\(𝐈\+12​𝐖\)\\mathcal\{C\}\(\\mathbf\{W\}\)=\\left\(\\mathbf\{I\}\-\\frac\{1\}\{2\}\\mathbf\{W\}\\right\)^\{\-1\}\\left\(\\mathbf\{I\}\+\\frac\{1\}\{2\}\\mathbf\{W\}\\right\)\. On the hypersphere, a tangent vector determines a unique skew\-symmetric generator supported on the two\-dimensional subspace spanned by the current point and the tangent vector\. This yields the following closed\-form expression\.

###### Proposition 3\.

Let𝐡∈𝕊rd−1\\mathbf\{h\}\\in\\mathbb\{S\}\_\{r\}^\{d\-1\}and𝐯∈T𝐡​𝕊rd−1\\mathbf\{v\}\\in T\_\{\\mathbf\{h\}\}\\mathbb\{S\}\_\{r\}^\{d\-1\}\. There exists a unique matrix𝐖=𝐖​\(𝐡,𝐯\)∈𝔰​𝔬​\(d\)\\mathbf\{W\}=\\mathbf\{W\}\(\\mathbf\{h\},\\mathbf\{v\}\)\\in\\mathfrak\{so\}\(d\)satisfying𝐖𝐡=𝐯,range⁡\(𝐖\)⊆span⁡\{𝐡,𝐯\}\\mathbf\{W\}\\mathbf\{h\}=\\mathbf\{v\},\\ \\operatorname\{range\}\(\\mathbf\{W\}\)\\subseteq\\operatorname\{span\}\\\{\\mathbf\{h\},\\mathbf\{v\}\\\}, and it is given by

𝐖​\(𝐡,𝐯\)=𝐯𝐡⊤−𝐡𝐯⊤r2\.\\mathbf\{W\}\(\\mathbf\{h\},\\mathbf\{v\}\)=\\frac\{\\mathbf\{v\}\\mathbf\{h\}^\{\\top\}\-\\mathbf\{h\}\\mathbf\{v\}^\{\\top\}\}\{r^\{2\}\}\.\(7\)Moreover, defineρ=‖𝐯‖2r\\rho=\\frac\{\\\|\\mathbf\{v\}\\\|\_\{2\}\}\{r\}\. The corresponding Cayley retraction admits the closed form

ℛ𝐡Cay​\(𝐯\)=𝒞​\(𝐖​\(𝐡,𝐯\)\)​𝐡=4−ρ24\+ρ2​𝐡\+44\+ρ2​𝐯\.\\mathcal\{R\}\_\{\\mathbf\{h\}\}^\{\\text\{Cay\}\}\(\\mathbf\{v\}\)=\\mathcal\{C\}\\\!\\left\(\\mathbf\{W\}\(\\mathbf\{h\},\\mathbf\{v\}\)\\right\)\\mathbf\{h\}=\\frac\{4\-\\rho^\{2\}\}\{4\+\\rho^\{2\}\}\\mathbf\{h\}\+\\frac\{4\}\{4\+\\rho^\{2\}\}\\mathbf\{v\}\.\(8\)

The proof is provided in Appendix[D\.1](https://arxiv.org/html/2608.02668#A4.SS1)\. Letρ=βl\\rho=\\beta\_\{l\}, based on Eq\. \([4](https://arxiv.org/html/2608.02668#S4.E4)\) and Eq\. \([8](https://arxiv.org/html/2608.02668#S4.E8)\), the whole process of Cay\-SpheretNorm is defined as

𝐳l=\(𝐈−𝐡l​𝐡l⊤∥𝐡l∥2\)​ℱl​\(𝐡l\),𝐡l\+1=4−βl24\+βl2​𝐡l\+4​αl4\+βl2​𝐳l\.\\begin\{split\}\\mathbf\{z\}\_\{l\}&=\\Bigl\(\\mathbf\{I\}\-\\frac\{\\mathbf\{h\}\_\{l\}\\mathbf\{h\}\_\{l\}^\{\\top\}\}\{\\lVert\{\\mathbf\{h\}\_\{l\}\}\\rVert^\{2\}\}\\Bigr\)\\mathcal\{F\}\_\{l\}\(\\mathbf\{h\}\_\{l\}\),\\\\ \\mathbf\{h\}\_\{l\+1\}&=\\frac\{4\-\\beta\_\{l\}^\{2\}\}\{4\+\\beta\_\{l\}^\{2\}\}\\mathbf\{h\}\_\{l\}\+\\frac\{4\\alpha\_\{l\}\}\{4\+\\beta\_\{l\}^\{2\}\}\\mathbf\{z\}\_\{l\}\.\\end\{split\}\(9\)

##### Spherepp\-Angular Retraction Normalization\.

The metric projection and Cayley retractions share the same planar\-rotation structure: they update the hidden state within the two\-dimensional plane spanned by the current state and the projected tangent direction, differing only in how the normalized step magnitude is converted into a rotation angle\. Motivated by this common structure, we introduce a one\-parameter family of hyper\-spherical retraction normalizations, termedpp\-SpheretNorm\. The parameterp\>0p\>0controls the saturation of the finite\-step rotation angle while preserving the same local behavior near the origin\.

###### Definition 4\(pp\-SpheretNorm\)\.

Let parameterp∈\(0,\+∞\)p\\in\(0,\+\\infty\), and the rotation angleθl=p​arctan⁡\(βlp\)\\theta\_\{l\}=p\\arctan\\left\(\\frac\{\\beta\_\{l\}\}\{p\}\\right\), then thepp\-SpheretNorm is defined as

𝐳l=\(𝐈−𝐡l​𝐡l⊤∥𝐡l∥2\)​ℱl​\(𝐡l\),βl=αl​∥𝐳l∥2∥𝐡l∥2,θl=p​arctan⁡\(βlp\),𝐡l\+1=cos⁡\(θl\)​𝐡l\+∥𝐡l∥2​sin⁡\(θl\)​𝐳l∥𝐳l∥2\.\\begin\{split\}\\mathbf\{z\}\_\{l\}&=\\Bigl\(\\mathbf\{I\}\-\\frac\{\\mathbf\{h\}\_\{l\}\\mathbf\{h\}\_\{l\}^\{\\top\}\}\{\\lVert\{\\mathbf\{h\}\_\{l\}\}\\rVert^\{2\}\}\\Bigr\)\\mathcal\{F\}\_\{l\}\(\\mathbf\{h\}\_\{l\}\),\\\\ \\beta\_\{l\}&=\\alpha\_\{l\}\\frac\{\\lVert\{\\mathbf\{z\}\_\{l\}\}\\rVert\_\{2\}\}\{\\lVert\{\\mathbf\{h\}\_\{l\}\}\\rVert\_\{2\}\},\\quad\\theta\_\{l\}=p\\arctan\\left\(\\frac\{\\beta\_\{l\}\}\{p\}\\right\),\\\\ \\mathbf\{h\}\_\{l\+1\}&=\\cos\(\\theta\_\{l\}\)\\mathbf\{h\}\_\{l\}\+\\lVert\{\\mathbf\{h\}\_\{l\}\}\\rVert\_\{2\}\\sin\(\\theta\_\{l\}\)\\frac\{\\mathbf\{z\}\_\{l\}\}\{\\lVert\{\\mathbf\{z\}\_\{l\}\}\\rVert\_\{2\}\}\.\\end\{split\}\(10\)

###### Proposition 5\.

For every fixedp\>0p\>0, thepp\-angular map underlyingpp\-SpheretNorm defines a smooth second\-order retraction on the hypersphere\. Moreover, the following identities and limits hold:

1. 1\.Whenp=1p=1,pp\-SpheretNorm reduces exactly to Proj\-SpheretNorm\.
2. 2\.Whenp=2p=2,pp\-SpheretNorm reduces exactly to Cay\-SpheretNorm\.
3. 3\.Asp→\+∞p\\to\+\\infty, thepp\-angular update converges to the exponential\-map update used by GeoNorm\.
4. 4\.Asp→0\+p\\to 0^\{\+\}, thepp\-angular update converges point\-wise to the identity mapping\.

To make the local geometry explicit, under the local boundedness conditions stated in Appendix[D\.4](https://arxiv.org/html/2608.02668#A4.SS4), for every fixedp\>0p\>0, asαl→0\\alpha\_\{l\}\\to 0,

𝐡l\+1=𝐡l\+αl​𝐳l−αl2​‖𝐳l‖222​rl2​𝐡l\+𝒪​\(αl3\)\.\\mathbf\{h\}\_\{l\+1\}=\\mathbf\{h\}\_\{l\}\+\\alpha\_\{l\}\\mathbf\{z\}\_\{l\}\-\\frac\{\\alpha\_\{l\}^\{2\}\\\|\\mathbf\{z\}\_\{l\}\\\|\_\{2\}^\{2\}\}\{2r\_\{l\}^\{2\}\}\\mathbf\{h\}\_\{l\}\+\\mathcal\{O\}\(\\alpha\_\{l\}^\{3\}\)\.The quadratic term is normal to the hypersphere at𝐡l\\mathbf\{h\}\_\{l\}; hence its tangent projection vanishes, which is precisely the second\-order retraction condition\. In particular, since𝐳l=𝐏l​ℱl​\(𝐡l\)\\mathbf\{z\}\_\{l\}=\\mathbf\{P\}\_\{l\}\\mathcal\{F\}\_\{l\}\(\\mathbf\{h\}\_\{l\}\), we obtain

𝐡l\+1=𝐡l\+αl​𝐏l​ℱl​\(𝐡l\)\+𝒪​\(αl2\)\.\\mathbf\{h\}\_\{l\+1\}=\\mathbf\{h\}\_\{l\}\+\\alpha\_\{l\}\\mathbf\{P\}\_\{l\}\\mathcal\{F\}\_\{l\}\(\\mathbf\{h\}\_\{l\}\)\+\\mathcal\{O\}\(\\alpha\_\{l\}^\{2\}\)\.Thus, all members of thepp\-angular family share the same first\-order tangent dynamics and the same second\-order normal correction, while the dependence onppfirst appears at cubic order\. The complete proof, including smoothness at𝐳l=𝟎\\mathbf\{z\}\_\{l\}=\\mathbf\{0\}, is provided in Appendix[D\.4](https://arxiv.org/html/2608.02668#A4.SS4)\.

### 4\.2Implementation Details

For thepp\-SpheretNorm, the rotation angleθl\\theta\_\{l\}in each block is always bounded in\(−p​π2,p​π2\)\(\-\\frac\{p\\pi\}\{2\},\\,\\frac\{p\\pi\}\{2\}\)\.Zheng et al\. \([2026](https://arxiv.org/html/2608.02668#bib.bib42)\)suggest that the rotation angle in each block should be restricted to\(0,π2\)\(0,\\,\\frac\{\\pi\}\{2\}\), and that restricting it to\(0,π4\]\(0,\\,\\frac\{\\pi\}\{4\}\]produces better results than restricting it to\(0,π2\)\(0,\\,\\frac\{\\pi\}\{2\}\)\. Our experiments also support this view\. We find that the rotation angle in each block should be restricted to\(0,π4\]\(0,\\,\\frac\{\\pi\}\{4\}\]for Cay\-SpheretNorm and forpp\-SpheretNorm withp\>1p\>1; otherwise, training becomes unstable and may even suffer from exploding gradients\. Therefore, forpp\-SpheretNorm, we choosep=0\.5p=0\.5, in which case the rotation angle in each block is kept in\(−π4,π4\)\(\-\\frac\{\\pi\}\{4\},\\,\\frac\{\\pi\}\{4\}\)\.

For all three types of SpheretNorm, we further adopt two additional approaches that ensure training stability with fewer loss spikes\. First, we adopt a decay method similar to the one in\(Zheng et al\.,[2026](https://arxiv.org/html/2608.02668#bib.bib42)\): we letαl=λl⋅Softplus⁡\(al\)\\alpha\_\{l\}=\\lambda\_\{l\}\\cdot\\operatorname\{Softplus\}\(a\_\{l\}\), whereλl∈\(0,1\)\\lambda\_\{l\}\\in\(0,\\,1\)is a decay factor andala\_\{l\}is a learnable parameter\. We find that settingλl=1/l\\lambda\_\{l\}=1/\\sqrt\{l\}, initializingal=ln⁡\(e−1\)a\_\{l\}=\\ln\(\\mathrm\{e\}\\mkern 1\.0mu\-1\), and constrainingSoftplus⁡\(al\)∈\(0,1\]\\operatorname\{Softplus\}\(a\_\{l\}\)\\in\(0,1\]can ensure training stability with fewer loss spikes\. Second, we apply a modified ScaleNorm before the first \(non\-\)linear mapping, i\.e\.,

𝐡1←Softplus⁡\(γ\)⋅𝐡1∥𝐡1∥2,\\mathbf\{h\}\_\{1\}\\leftarrow\\operatorname\{Softplus\}\(\\gamma\)\\cdot\\frac\{\\mathbf\{h\}\_\{1\}\}\{\\lVert\{\\mathbf\{h\}\_\{1\}\}\\rVert\_\{2\}\},\(11\)whereγ∈ℝ\\gamma\\in\\mathbb\{R\}is a learnable parameter that is initialized asγ=d\\gamma=\\sqrt\{d\}whend≥20\\sqrt\{d\}\\geq 20, and asγ=ln⁡\(ed−1\)\\gamma=\\ln\(\\mathrm\{e\}\\mkern 1\.0mu^\{\\sqrt\{d\}\}\-1\)otherwise\. We use thetorch\.clampfunction to ensure thatSoftplus⁡\(γ\)∈\[1,d\]\\operatorname\{Softplus\}\(\\gamma\)\\in\[1,\\sqrt\{d\}\], which we find can accelerate model convergence while stabilizing training\.

### 4\.3Analysis

Our analysis of SpheretNorms focuses on four parts\. First, we show thatpp\-angular retraction is a second\-order retraction\. Second, we analyze the Jacobian to establish training stability\. Third, we prove that the modified ScaleNorm in the first layer preserves the hypersphere\. Fourth, we discuss the trade\-off in choosing different decay factors\. For analysis, we definerl=‖𝐡l‖2,𝐏l=I−𝐡l​𝐡l⊤rl2,𝐳l=𝐏l​ℱl​\(𝐡l\),sl=‖𝐳l‖2r\_\{l\}=\\\|\\mathbf\{h\}\_\{l\}\\\|\_\{2\},\\ \\mathbf\{P\}\_\{l\}=I\-\\frac\{\\mathbf\{h\}\_\{l\}\\mathbf\{h\}\_\{l\}^\{\\top\}\}\{r\_\{l\}^\{2\}\},\\ \\mathbf\{z\}\_\{l\}=\\mathbf\{P\}\_\{l\}\\mathcal\{F\}\_\{l\}\(\\mathbf\{h\}\_\{l\}\),\\ s\_\{l\}=\\\|\\mathbf\{z\}\_\{l\}\\\|\_\{2\}\.

##### pp\-SpheretNorm based on the second order retraction\.

Letηl=slrl,βl=αl​ηl,θl=p​arctan⁡\(βlp\)\\eta\_\{l\}=\\frac\{s\_\{l\}\}\{r\_\{l\}\},\\ \\beta\_\{l\}=\\alpha\_\{l\}\\eta\_\{l\},\\ \\theta\_\{l\}=p\\arctan\\left\(\\frac\{\\beta\_\{l\}\}\{p\}\\right\)\. For every fixedp\>0p\>0, thepp\-angular retraction is a second\-order retraction\. Under the local boundedness conditions stated in Appendix[D\.4](https://arxiv.org/html/2608.02668#A4.SS4), it also satisfies, asαl→0\\alpha\_\{l\}\\to 0,

𝐡l\+1=𝐡l\+αl​𝐏l​ℱl​\(𝐡l\)\+𝒪​\(αl2\)\.\\mathbf\{h\}\_\{l\+1\}=\\mathbf\{h\}\_\{l\}\+\\alpha\_\{l\}\\mathbf\{P\}\_\{l\}\\mathcal\{F\}\_\{l\}\(\\mathbf\{h\}\_\{l\}\)\+\\mathcal\{O\}\(\\alpha\_\{l\}^\{2\}\)\.The proof is provided in Appendix[D\.4](https://arxiv.org/html/2608.02668#A4.SS4)\. Thus, all members of thepp\-angular family share the same first\-order tangent dynamics and differ only through their higher\-order finite\-step angular responses\.

##### Jacobian conditioning from signal propagation\.

Exact norm preservation controls the activation scale, but does not by itself control perturbation or gradient propagation\. DefineAl=∂ℱl​\(𝐡l\)∂𝐡l,cl=𝐡l⊤​ℱl​\(𝐡l\)rl2,𝐁l=𝐀l−cl​𝐈A\_\{l\}=\\frac\{\\partial\\mathcal\{F\}\_\{l\}\(\\mathbf\{h\}\_\{l\}\)\}\{\\partial\\mathbf\{h\}\_\{l\}\},\\ c\_\{l\}=\\frac\{\\mathbf\{h\}\_\{l\}^\{\\top\}\\mathcal\{F\}\_\{l\}\(\\mathbf\{h\}\_\{l\}\)\}\{r\_\{l\}^\{2\}\},\\ \\mathbf\{B\}\_\{l\}=\\mathbf\{A\}\_\{l\}\-c\_\{l\}\\mathbf\{I\}, and let𝐉l=∂𝐡l\+1∂𝐡l\\mathbf\{J\}\_\{l\}=\\frac\{\\partial\\mathbf\{h\}\_\{l\+1\}\}\{\\partial\\mathbf\{h\}\_\{l\}\}\. Forsl\>0s\_\{l\}\>0, defineQl=cos⁡\(θl\)​𝐈\+sin⁡\(θl\)rl​sl​\(𝐳l​𝐡l⊤−𝐡l​𝐳l⊤\)Q\_\{l\}=\\cos\\left\(\\theta\_\{l\}\\right\)\\mathbf\{I\}\+\\frac\{\\sin\\left\(\\theta\_\{l\}\\right\)\}\{r\_\{l\}s\_\{l\}\}\\left\(\\mathbf\{z\}\_\{l\}\\mathbf\{h\}\_\{l\}^\{\\top\}\-\\mathbf\{h\}\_\{l\}\\mathbf\{z\}\_\{l\}^\{\\top\}\\right\)\.

###### Proposition 6\(Layer\-wise and depth\-wise Jacobian bounds\)\.

Forsl\>0s\_\{l\}\>0, the Jacobian of thell\-th SpheretNorm block admits the exact decomposition𝐉l=𝐐l\+𝐄l\\mathbf\{J\}\_\{l\}=\\mathbf\{Q\}\_\{l\}\+\\mathbf\{E\}\_\{l\}, where‖𝐐l‖2=1,σmin​\(𝐐l\)≥\|cos⁡\(θl\)\|\\\|\\mathbf\{Q\}\_\{l\}\\\|\_\{2\}=1,\\ \\sigma\_\{\\min\}\(\\mathbf\{Q\}\_\{l\}\)\\geq\\left\|\\cos\\left\(\\theta\_\{l\}\\right\)\\right\|\. DefineΔl=\|αl\|​\(2​‖𝐁l‖2\+ηl\)\\Delta\_\{l\}=\|\\alpha\_\{l\}\|\\left\(2\\\|\\mathbf\{B\}\_\{l\}\\\|\_\{2\}\+\\eta\_\{l\}\\right\),ql=\|cos⁡\(θl\)\|q\_\{l\}=\\left\|\\cos\\left\(\\theta\_\{l\}\\right\)\\right\|, andml=max⁡\{0,ql−Δl\}m\_\{l\}=\\max\\\{0,q\_\{l\}\-\\Delta\_\{l\}\\\}\. Then‖𝐄l‖2≤Δl\\\|\\mathbf\{E\}\_\{l\}\\\|\_\{2\}\\leq\\Delta\_\{l\}, and,

σmax​\(𝐉l\)≤1\+Δl,σmin​\(𝐉l\)≥ml\.\\sigma\_\{\\max\}\(\\mathbf\{J\}\_\{l\}\)\\leq 1\+\\Delta\_\{l\},\\quad\\sigma\_\{\\min\}\(\\mathbf\{J\}\_\{l\}\)\\geq m\_\{l\}\.Consequently, for𝐆L=∂𝐡L∂𝐡1=𝐉L​𝐉L−1​⋯​𝐉1\\mathbf\{G\}\_\{L\}=\\frac\{\\partial\\mathbf\{h\}\_\{L\}\}\{\\partial\\mathbf\{h\}\_\{1\}\}=\\mathbf\{J\}\_\{L\}\\mathbf\{J\}\_\{L\-1\}\\cdots\\mathbf\{J\}\_\{1\}, one has

σmax​\(𝐆L\)≤exp⁡\(∑l=1LΔl\),σmin​\(𝐆L\)≥∏l=1Lml\.\\sigma\_\{\\max\}\(\\mathbf\{G\}\_\{L\}\)\\leq\\exp\\left\(\\sum\_\{l=1\}^\{L\}\\Delta\_\{l\}\\right\),\\quad\\sigma\_\{\\min\}\(\\mathbf\{G\}\_\{L\}\)\\geq\\prod\_\{l=1\}^\{L\}m\_\{l\}\.

The full Jacobian calculation and the smoothsl=0s\_\{l\}=0extension are provided in Appendix[D\.7](https://arxiv.org/html/2608.02668#A4.SS7)\. The angleθl\\theta\_\{l\}controls the geometric component𝐐l\\mathbf\{Q\}\_\{l\}, whereas\|αl\|​‖𝐁l‖2\|\\alpha\_\{l\}\|\\\|\\mathbf\{B\}\_\{l\}\\\|\_\{2\}and\|βl\|\|\\beta\_\{l\}\|control the block\-induced perturbation\. Hence, a cap

\|θl\|≤θmax<π2\|\\theta\_\{l\}\|\\leq\\theta\_\{\\max\}<\\frac\{\\pi\}\{2\}keeps𝐐l\\mathbf\{Q\}\_\{l\}away from singularity but does not by itself control the full Jacobian; a counterexample is given in Appendix[D\.8](https://arxiv.org/html/2608.02668#A4.SS8)\. Moreover, ifml≥m∗\>0m\_\{l\}\\geq m\_\{\*\}\>0, then

∑l=1LΔl=𝒪​\(1\),∑l=1L\|θl\|2=𝒪​\(1\)\\sum\_\{l=1\}^\{L\}\\Delta\_\{l\}=\\mathcal\{O\}\(1\),\\quad\\sum\_\{l=1\}^\{L\}\\left\|\\theta\_\{l\}\\right\|^\{2\}=\\mathcal\{O\}\(1\)is a sufficient condition for depth\-independent conditioning; see Appendix[D\.5](https://arxiv.org/html/2608.02668#A4.SS5)\.

##### First\-layer modified ScaleNorm preserves hypersphere\.

###### Proposition 7\(Fixed\-radius spherical dynamics\)\.

Recall Eq\. \([11](https://arxiv.org/html/2608.02668#S4.E11)\)\. Forl=1,2,…,Ll=1,\\,2,\\,\\ldots,\\,L, suppose that𝐡l\+1\\mathbf\{h\}\_\{l\+1\}is generated from𝐡l\\mathbf\{h\}\_\{l\}by a SpheretNorm update\. Then∥𝐡l∥2=Softplus⁡\(γ\)\\lVert\{\\mathbf\{h\}\_\{l\}\}\\rVert\_\{2\}=\\operatorname\{Softplus\}\(\\gamma\)\. Consequently,∥𝐡L∥2≤Softplus⁡\(γ\)\\lVert\{\\mathbf\{h\}\_\{L\}\}\\rVert\_\{2\}\\leq\\operatorname\{Softplus\}\(\\gamma\)\.

The proof is provided in Appendix[D\.3](https://arxiv.org/html/2608.02668#A4.SS3)\. Thus, the first modified ScaleNorm selects the hypersphere𝕊Softplus⁡\(γ\)d−1=\{𝐡∈ℝd:‖𝐡‖2=Softplus⁡\(γ\)\}\\mathbb\{S\}^\{d\-1\}\_\{\\operatorname\{Softplus\}\(\\gamma\)\}=\\\{\\mathbf\{h\}\\in\\mathbb\{R\}^\{d\}:\\\|\\mathbf\{h\}\\\|\_\{2\}=\\operatorname\{Softplus\}\(\\gamma\)\\\}, while all subsequent SpheretNorm blocks preserve this hypersphere exactly\. For Proj\-SpheretNorm, this relation can be made more explicit\. By Proposition[7](https://arxiv.org/html/2608.02668#Thmtheorem7),∥𝐡l∥2=Softplus⁡\(γ\)\\lVert\{\\mathbf\{h\}\_\{l\}\}\\rVert\_\{2\}=\\operatorname\{Softplus\}\(\\gamma\), and therefore

𝐡l\+1\\displaystyle\\mathbf\{h\}\_\{l\+1\}=Softplus⁡\(γ\)⋅𝐡l\+αl​𝐳l‖𝐡l\+αl​𝐳l‖2\\displaystyle=\\operatorname\{Softplus\}\(\\gamma\)\\cdot\\frac\{\\mathbf\{h\}\_\{l\}\+\\alpha\_\{l\}\\mathbf\{z\}\_\{l\}\}\{\\\|\\mathbf\{h\}\_\{l\}\+\\alpha\_\{l\}\\mathbf\{z\}\_\{l\}\\\|\_\{2\}\}=ΠSoftplus⁡\(γ\)​\(𝐡l\+αl​𝐏l​ℱl​\(𝐡l\)\),\\displaystyle=\\Pi\_\{\\operatorname\{Softplus\}\(\\gamma\)\}\\bigl\(\\mathbf\{h\}\_\{l\}\+\\alpha\_\{l\}\\mathbf\{P\}\_\{l\}\\mathcal\{F\}\_\{l\}\(\\mathbf\{h\}\_\{l\}\)\\bigr\),whereΠSoftplus⁡\(γ\)​\(𝐮\)=Softplus⁡\(γ\)​𝐮‖𝐮‖2\\Pi\_\{\\operatorname\{Softplus\}\(\\gamma\)\}\(\\mathbf\{u\}\)=\\operatorname\{Softplus\}\(\\gamma\)\\frac\{\\mathbf\{u\}\}\{\\\|\\mathbf\{u\}\\\|\_\{2\}\}is the radial projection onto𝕊Softplus⁡\(γ\)d−1\\mathbb\{S\}^\{d\-1\}\_\{\\operatorname\{Softplus\}\(\\gamma\)\}\.

##### Training stability trade\-off from different decay factors\.

The prescribed factorλl\\lambda\_\{l\}remains fixed throughout training\. Since0<al≤10<a\_\{l\}\\leq 1, we have0<αl=λl​al≤λl0<\\alpha\_\{l\}=\\lambda\_\{l\}a\_\{l\}\\leq\\lambda\_\{l\}\. Letκl=2​‖𝐁l‖2\+ηl\\kappa\_\{l\}=2\\\|\\mathbf\{B\}\_\{l\}\\\|\_\{2\}\+\\eta\_\{l\},𝒮L=∑l=1LΔl=∑l=1Lλl​al​κl\\mathcal\{S\}\_\{L\}=\\sum\_\{l=1\}^\{L\}\\Delta\_\{l\}=\\sum\_\{l=1\}^\{L\}\\lambda\_\{l\}a\_\{l\}\\kappa\_\{l\}, and𝒜L=∑l=1L\|θl\|2\\mathcal\{A\}\_\{L\}=\\sum\_\{l=1\}^\{L\}\\left\|\\theta\_\{l\}\\right\|^\{2\}\.

###### Proposition 8\(Growth under depth schedules\)\.

Suppose that there exist constantsK,H\>0K,\\,H\>0, independent ofllandLL, such thatκl≤K\\kappa\_\{l\}\\leq K,al​ηl≤H\\ a\_\{l\}\\eta\_\{l\}\\leq H\. Then

𝒮L≤K​∑l=1Lλl,𝒜L≤H2​∑l=1Lλl2\.\\mathcal\{S\}\_\{L\}\\leq K\\sum\_\{l=1\}^\{L\}\\lambda\_\{l\},\\quad\\mathcal\{A\}\_\{L\}\\leq H^\{2\}\\sum\_\{l=1\}^\{L\}\\lambda\_\{l\}^\{2\}\.Consequently,

λl𝒮L𝒜L1𝒪​\(L\)𝒪​\(L\)1/l𝒪​\(L\)𝒪​\(log⁡L\)l𝒪​\(log⁡L\)𝒪​\(1\)\\begin\{array\}\[\]\{c\|cc\}\\lambda\_\{l\}&\\mathcal\{S\}\_\{L\}&\\mathcal\{A\}\_\{L\}\\\\ \\hline\\cr 1&\\mathcal\{O\}\(L\)&\\mathcal\{O\}\(L\)\\\\ 1/\\sqrt\{l\}&\\mathcal\{O\}\(\\sqrt\{L\}\)&\\mathcal\{O\}\(\\log L\)\\\\ \\sqrt\{l\}&\\mathcal\{O\}\(\\log L\)&\\mathcal\{O\}\(1\)\\end\{array\}

The proof is provided in Appendix[D\.6](https://arxiv.org/html/2608.02668#A4.SS6)\. Thus, the decay factorλl=1/l\\lambda\_\{l\}=1/lgives the more conservative global worst\-case bounds\. However, Proposition[8](https://arxiv.org/html/2608.02668#Thmtheorem8)only provides global upper bounds\. A finer dyadic analysis is given in Appendix[D\.9](https://arxiv.org/html/2608.02668#A4.SS9)\. At any fixed optimization iteration, under the bounded small\-step and non\-degeneracy assumptions stated there, the accumulated squared angular motion over each dyadic intervalImI\_\{m\}has the same asymptotic order as∑l∈Imλl2\\sum\_\{l\\in I\_\{m\}\}\\lambda\_\{l\}^\{2\}\. Hence, the square\-root schedule assigns an approximately constant squared angular budget to each logarithmic depth scale, whereas the the decay factorλl=1/l\\lambda\_\{l\}=1/lassigns a budget of orderm−1m^\{\-1\}\.

The empirically preferred the decay factorλl=1/l\\lambda\_\{l\}=1/\\sqrt\{l\}should be interpreted as a finite\-depth compromise: it provides a tighter Jacobian envelope than the non\-decaying schedule while preserving stronger transformations in deeper layers than the decay factorλl=1/l\\lambda\_\{l\}=1/l\. This mechanism is consistent with the empirical advantage of the decay factorλl=1/l\\lambda\_\{l\}=1/\\sqrt\{l\}, but does not imply uniformly better worst\-case conditioning or a guaranteed lower loss\.

Table 2:Loss of trained models\. We report training and validation loss at the end of training\. To mitigate stochastic fluctuations, training loss is computed as a moving average over the last 200 iterations\.DatasetFineWeb\-EduOpenWebTextModel ScaleSMLSMLTrainValTrainValTrainValTrainValTrainValTrainValPre\-LN3\.057\\mathbf\{3\.057\}3\.071\\mathbf\{3\.071\}2\.9752\.9752\.9762\.9762\.8422\.8422\.8662\.8663\.062\\mathbf\{3\.062\}3\.061\\mathbf\{3\.061\}4\.6974\.6974\.7164\.7162\.8582\.8582\.8792\.879Pre\-DyT3\.6263\.6263\.6193\.6193\.0253\.0253\.0193\.0192\.8452\.8452\.8702\.8703\.5393\.5393\.5563\.5563\.0123\.0123\.0353\.0352\.8432\.8432\.8662\.866Peri\-LN3\.0913\.0913\.0883\.0882\.9522\.9522\.9492\.9492\.8172\.8172\.8442\.8443\.0713\.0713\.0893\.0892\.9502\.9502\.9732\.9732\.8282\.8282\.8502\.850Keel7\.6457\.6457\.6417\.6413\.0063\.0063\.0013\.0012\.9412\.9412\.9592\.9593\.4063\.4063\.4223\.4222\.9972\.9973\.0183\.0182\.9452\.9452\.9642\.964GeoNorm3\.1763\.1763\.1853\.1853\.0963\.0963\.0883\.0882\.9402\.9402\.9582\.9583\.2093\.2093\.2063\.2063\.0183\.0183\.0393\.0393\.0013\.0013\.0173\.0170\.50\.5\-SpheretNorm3\.089¯\\underline\{3\.089\}3\.086¯\\underline\{3\.086\}2\.8312\.8312\.8282\.8282\.809¯\\underline\{2\.809\}2\.833¯\\underline\{2\.833\}3\.0673\.0673\.0853\.0852\.932¯\\underline\{2\.932\}2\.955¯\\underline\{2\.955\}2\.826¯\\underline\{2\.826\}2\.847¯\\underline\{2\.847\}11\-SpheretNorm3\.0983\.0983\.1093\.1092\.825\\mathbf\{2\.825\}2\.822\\mathbf\{2\.822\}2\.808\\mathbf\{2\.808\}2\.832\\mathbf\{2\.832\}3\.0643\.0643\.0823\.0822\.931\\mathbf\{2\.931\}2\.954\\mathbf\{2\.954\}2\.795\\mathbf\{2\.795\}2\.822\\mathbf\{2\.822\}22\-SpheretNorm3\.1193\.1193\.1143\.1142\.826¯\\underline\{2\.826\}2\.824¯\\underline\{2\.824\}2\.8092\.8092\.8342\.8343\.063¯\\underline\{3\.063\}3\.081¯\\underline\{3\.081\}2\.931\\mathbf\{2\.931\}2\.955¯\\underline\{2\.955\}2\.8412\.8412\.8612\.861Table 3:Perplexity of trained models\. We report validation perplexity at the end of training\.DatasetFineWeb\-EduOpenWebTextScaleSMLSMLPre\-LN21\.56\\mathbf\{21\.56\}19\.6119\.6117\.5717\.5721\.35\\mathbf\{21\.35\}111\.70111\.7017\.8017\.80Pre\-DyT37\.2937\.2920\.4820\.4817\.6417\.6435\.0335\.0320\.7920\.7917\.5617\.56Peri\-LN21\.9421\.9419\.0919\.0917\.1817\.1821\.9621\.9619\.5519\.5517\.2917\.29Keel2081208120\.1120\.1119\.2719\.2730\.6330\.6320\.4620\.4619\.3719\.37GeoNorm24\.1624\.1621\.9321\.9319\.2719\.2724\.6724\.6720\.8820\.8820\.4220\.42SpheretNorm \(ours\)0\.50\.521\.88¯\\underline\{21\.88\}16\.9116\.9116\.99¯\\underline\{16\.99\}21\.8621\.8619\.20¯\\underline\{19\.20\}17\.24¯\\underline\{17\.24\}1122\.4022\.4016\.81\\mathbf\{16\.81\}16\.98\\mathbf\{16\.98\}21\.8121\.8119\.18\\mathbf\{19\.18\}16\.81\\mathbf\{16\.81\}2222\.5222\.5216\.84¯\\underline\{16\.84\}17\.0117\.0121\.79¯\\underline\{21\.79\}19\.20¯\\underline\{19\.20\}17\.4817\.48

## 5Experiments

##### Experimental Settings\.

To evaluate SpheretNorms against other deep connection methods, including Pre\-LN\(Wang et al\.,[2019](https://arxiv.org/html/2608.02668#bib.bib36); Xiong et al\.,[2020](https://arxiv.org/html/2608.02668#bib.bib39)\), Pre\-DyT\(Zhu et al\.,[2025](https://arxiv.org/html/2608.02668#bib.bib43)\), Peri\-LN\(Kim et al\.,[2025](https://arxiv.org/html/2608.02668#bib.bib19)\), Keel\(Chen and Wei,[2026](https://arxiv.org/html/2608.02668#bib.bib6)\), and GeoNorm\(Zheng et al\.,[2026](https://arxiv.org/html/2608.02668#bib.bib42)\), we implement all baselines within the nanoGPT framework\(Karpathy,[2022](https://arxiv.org/html/2608.02668#bib.bib18)\)\. For pre\-training, we adopt two datasets: OpenWebText\(Gokaslan and Cohen,[2019](https://arxiv.org/html/2608.02668#bib.bib12)\)and FineWeb\-Edu\(Penedo et al\.,[2024](https://arxiv.org/html/2608.02668#bib.bib28)\)\. All models are optimized with the AdamW optimizer\(Loshchilov and Hutter,[2019](https://arxiv.org/html/2608.02668#bib.bib22)\)under a cosine annealing learning rate schedule with a linear warm\-up\. We employbfloat16mixed\-precision training together with gradient clipping\. We conducted all experiments on a single node equipped with two 56\-core Intel Xeon Platinum 8480\+ CPUs, 2 TB of RAM, and eight NVIDIA H100 SXM GPUs, each with 80 GB of memory\. The code is implemented in PyTorch\(Paszke et al\.,[2019](https://arxiv.org/html/2608.02668#bib.bib27)\)using its Distributed Data Parallel \(DDP\) with the NCCL backend\. We train each model for 50,000 iterations \(about6\.556\.55B tokens in total\)\. We fix the random seed to 1337 in all experiments\. Detailed hyperparameter configurations are reported in Appendix[B](https://arxiv.org/html/2608.02668#A2)\.

### 5\.1Pre\-Training Language Models

To assess whether SpheretNorm attains loss improvements comparable to existing residual connection designs, we compare the final training and validation losses in Table[2](https://arxiv.org/html/2608.02668#S4.T2)and the corresponding validation perplexities in Table[3](https://arxiv.org/html/2608.02668#S4.T3)\. At the small scale \(12 layers,∼0\.12\\sim\\\!0\.12B parameters\), Pre\-LN attains the lowest training and validation loss on both FineWeb\-Edu and OpenWebText; the best SpheretNorm variant stays within0\.020\.02nats of it \(21\.8821\.88vs\.21\.5621\.56validation perplexity on FineWeb\-Edu,21\.7921\.79vs\.21\.3521\.35on OpenWebText\) and is on par with Peri\-LN\. This ordering changes as depth increases\. At the medium scale \(24 layers,∼0\.35\\sim\\\!0\.35B\),11\-SpheretNorm attains the lowest training and validation loss on both datasets, improving validation perplexity over the strongest baseline, Peri\-LN, from19\.0919\.09to16\.8116\.81on FineWeb\-Edu and from19\.5519\.55to19\.1819\.18on OpenWebText\. The same holds at the large scale \(36 layers,∼0\.77\\sim\\\!0\.77B\), where Proj\-SpheretNorm reaches16\.9816\.98and16\.8116\.81validation perplexity on the two datasets, ahead of Peri\-LN at17\.1817\.18and17\.2917\.29\.22\-SpheretNorm and0\.50\.5\-SpheretNorm track11\-SpheretNorm closely in every setting, with all three variants falling within0\.040\.04nats of one another, indicating that the benefit is not sensitive to the particular choice ofpp\. The methods also differ in stability: training destabilizes for Pre\-LN on OpenWebText at the medium scale \(validation perplexity111\.70111\.70\) and for Keel on FineWeb\-Edu at the small scale \(2081\.502081\.50\), whereas all three SpheretNorm variants converge in all six settings\. We show the training and validation losses in FineWeb\-Edu in Figure[2](https://arxiv.org/html/2608.02668#S6.F2)\.

### 5\.2Downstream Evaluations

We evaluate zero\-shot performance using the lm\-evaluation\-harness library\(Gao et al\.,[2021](https://arxiv.org/html/2608.02668#bib.bib11)\)on eleven benchmarks: ARC\-Challenge, ARC\-Easy\(Clark et al\.,[2018](https://arxiv.org/html/2608.02668#bib.bib10)\), BoolQ\(Clark et al\.,[2019](https://arxiv.org/html/2608.02668#bib.bib9)\), HellaSwag\(Zellers et al\.,[2019](https://arxiv.org/html/2608.02668#bib.bib40)\), OpenBookQA\(Mihaylov et al\.,[2018](https://arxiv.org/html/2608.02668#bib.bib24)\), PIQA\(Bisk et al\.,[2020](https://arxiv.org/html/2608.02668#bib.bib5)\), SciQ\(Welbl et al\.,[2017](https://arxiv.org/html/2608.02668#bib.bib37)\), SocialIQa\(Sap et al\.,[2019](https://arxiv.org/html/2608.02668#bib.bib30)\), WinoGrande\(Sakaguchi et al\.,[2021](https://arxiv.org/html/2608.02668#bib.bib29)\), LAMBADA\(Paperno et al\.,[2016](https://arxiv.org/html/2608.02668#bib.bib26)\), and WikiText\-2\(Merity et al\.,[2017](https://arxiv.org/html/2608.02668#bib.bib23)\)\. Tables[4](https://arxiv.org/html/2608.02668#S5.T4)and[5](https://arxiv.org/html/2608.02668#S5.T5)report the large\-size models trained for 6\.55 B tokens on FineWeb\-Edu and OpenWebText, respectively\.

SpheretNorms attain the best score on 11 of the 12 metrics on FineWeb\-Edu and 10 of 12 on OpenWebText; the exceptions are SocialIQa on both corpora and HellaSwag on OpenWebText, where the leading baseline is ahead by at most0\.520\.52points\.11\-SpheretNorm is the strongest variant overall, though the three stay within about one point of each other on most tasks\. The gains are clearest in perplexity, the metric least affected by the noise of zero\-shot evaluation at this scale: on FineWeb\-Edu every variant improves over every baseline, with11\-SpheretNorm reaching46\.7446\.74on LAMBADA against59\.5559\.55for the best baseline \(21\.5%21\.5\\%relative\) and33\.7033\.70on WikiText\-2; on OpenWebText it reaches31\.1331\.13and30\.7930\.79\.

Two baselines warrant comment\. Keel, a Post\-LN design targeting gradient flow at extreme depth, brings no benefit in our shallow, short\-budget regime and is among the weakest configurations\. GeoNorm is the closest comparison, sharing our view of the hidden state as a point on a sphere and of each sublayer output as an update direction, but taking the exact geodesic step via the exponential map where SpheretNorm uses a retraction\. The retraction is the better choice here: our best variant improves over GeoNorm on all twelve metrics on both corpora, by margins far larger than those separating the other methods\. Since several benchmarks remain near chance at this scale \(e\.g\., WinoGrande, SocialIQa\), our conclusion rests on the perplexities and on the consistency across the two corpora rather than on individual sub\-point differences\.

Table 4:Zero\-shot evaluation results for large\-size models trained on FineWeb\-Edu for 6\.55 B tokens and evaluated with lm\-evaluation\-harness\. acc means accuracy, acc\_n means length\-normalized accuracy, and ppl means perplexity\.nanoGPT w/ARC\-CARC\-EBoolQHellaSwagLMB\.OBQAPIQASciQSocialIQaWinoGrandeLMB\.Wiki\.acc\_n↑\\uparrowacc\_n↑\\uparrowacc↑\\uparrowacc\_n↑\\uparrowacc↑\\uparrowacc\_n↑\\uparrowacc\_n↑\\uparrowacc↑\\uparrowacc↑\\uparrowacc↑\\uparrowppl↓\\downarrowppl↓\\downarrowPre\-LN26\.5426\.5451\.8551\.8555\.8155\.8136\.8136\.8129\.1129\.1133\.4033\.4065\.1865\.1881\.2081\.2037\.5637\.5652\.4952\.4966\.6966\.6936\.2336\.23Pre\-DyT27\.3927\.3951\.1451\.1460\.3460\.3436\.6436\.6429\.5229\.5231\.4031\.4065\.0265\.0282\.50¯\\underline\{82\.50\}38\.28\\mathbf\{38\.28\}50\.7550\.7559\.5559\.5535\.5235\.52Peri\-LN26\.6226\.6250\.9350\.9353\.2753\.2738\.09¯\\underline\{38\.09\}29\.0129\.0131\.2031\.2064\.8564\.8582\.60\\mathbf\{82\.60\}38\.13¯\\underline\{38\.13\}52\.3352\.3367\.0967\.0934\.7734\.77Keel26\.4526\.4549\.2449\.2461\.5961\.5934\.3934\.3926\.9726\.9730\.6030\.6064\.2564\.2580\.9080\.9038\.0838\.0848\.6248\.6281\.0681\.0639\.7339\.73GeoNorm27\.8227\.8248\.0248\.0262\.05¯\\underline\{62\.05\}34\.3534\.3527\.9827\.9830\.2030\.2063\.9363\.9380\.5080\.5036\.8036\.8051\.7851\.7880\.6480\.6439\.8839\.880\.50\.5\-SpheretNorm27\.90¯\\underline\{27\.90\}50\.9350\.9360\.8760\.8737\.7737\.7730\.8430\.8433\.80¯\\underline\{33\.80\}66\.27¯\\underline\{66\.27\}81\.6081\.6037\.6937\.6952\.88\\mathbf\{52\.88\}50\.00¯\\underline\{50\.00\}33\.74¯\\underline\{33\.74\}11\-SpheretNorm28\.67\\mathbf\{28\.67\}52\.78\\mathbf\{52\.78\}62\.70\\mathbf\{62\.70\}38\.21\\mathbf\{38\.21\}32\.93\\mathbf\{32\.93\}32\.6032\.6066\.0566\.0582\.1082\.1038\.13¯\\underline\{38\.13\}51\.3651\.3646\.74\\mathbf\{46\.74\}33\.70\\mathbf\{33\.70\}22\-SpheretNorm27\.90¯\\underline\{27\.90\}52\.53¯\\underline\{52\.53\}61\.3461\.3438\.0738\.0731\.19¯\\underline\{31\.19\}34\.40\\mathbf\{34\.40\}66\.70\\mathbf\{66\.70\}82\.60\\mathbf\{82\.60\}37\.2637\.2652\.72¯\\underline\{52\.72\}53\.4553\.4533\.8533\.85

Table 5:Zero\-shot evaluation results for large\-size models trained on OpenWebText for 6\.55 B tokens and evaluated with lm\-evaluation\-harness\. acc means accuracy, acc\_n means length\-normalized accuracy, and ppl means perplexity\.nanoGPT w/ARC\-CARC\-EBoolQHellaSwagLMB\.OBQAPIQASciQSocialIQaWinoGrandeLMB\.Wiki\.acc\_n↑\\uparrowacc\_n↑\\uparrowacc↑\\uparrowacc\_n↑\\uparrowacc↑\\uparrowacc\_n↑\\uparrowacc\_n↑\\uparrowacc↑\\uparrowacc↑\\uparrowacc↑\\uparrowppl↓\\downarrowppl↓\\downarrowPre\-LN23\.3823\.3840\.4940\.4959\.5459\.5432\.7332\.7332\.1932\.1926\.2026\.2062\.8962\.8969\.9069\.9038\.3338\.3351\.5451\.5439\.2439\.2433\.1933\.19Pre\-DyT25\.60¯\\underline\{25\.60\}40\.8740\.8754\.3154\.3132\.8932\.8933\.9233\.9226\.2026\.2062\.1362\.1371\.6071\.6038\.54¯\\underline\{38\.54\}51\.5451\.5433\.4033\.4034\.1134\.11Peri\-LN25\.4325\.4342\.0042\.0059\.1759\.1733\.95\\mathbf\{33\.95\}33\.7333\.7325\.8025\.8063\.8763\.8773\.2073\.2038\.64\\mathbf\{38\.64\}50\.0450\.0431\.4831\.4832\.9032\.90Keel23\.2123\.2138\.8038\.8057\.6157\.6131\.1731\.1730\.7030\.7027\.4027\.4061\.9761\.9771\.9071\.9038\.1338\.1351\.7051\.7044\.4344\.4338\.6438\.64GeoNorm24\.8724\.8737\.8437\.8460\.55¯\\underline\{60\.55\}29\.9729\.9730\.0030\.0026\.2026\.2061\.9261\.9269\.3069\.3037\.6737\.6751\.1451\.1451\.9151\.9142\.2442\.240\.50\.5\-SpheretNorm25\.1525\.1542\.21¯\\underline\{42\.21\}59\.8859\.8833\.43¯\\underline\{33\.43\}34\.60¯\\underline\{34\.60\}27\.4027\.4063\.4463\.4473\.2073\.2037\.9237\.9252\.96\\mathbf\{52\.96\}31\.38¯\\underline\{31\.38\}31\.02¯\\underline\{31\.02\}11\-SpheretNorm26\.46\\mathbf\{26\.46\}42\.26\\mathbf\{42\.26\}61\.04\\mathbf\{61\.04\}32\.9032\.9034\.99\\mathbf\{34\.99\}27\.80¯\\underline\{27\.80\}63\.98¯\\underline\{63\.98\}73\.60\\mathbf\{73\.60\}37\.3637\.3652\.85¯\\underline\{52\.85\}31\.13\\mathbf\{31\.13\}30\.79\\mathbf\{30\.79\}22\-SpheretNorm25\.3225\.3242\.0542\.0558\.7458\.7432\.8632\.8634\.3934\.3928\.60\\mathbf\{28\.60\}64\.20\\mathbf\{64\.20\}73\.40¯\\underline\{73\.40\}38\.4838\.4851\.1451\.1431\.5031\.5031\.5531\.55

## 6Limitations

While the proposed SpheretNorms are designed to stabilize the training of a broad range of deep neural networks, two limitations remain\. First, the proposed methods are not directly applicable to certain specialized architectures, such as Equivariant Neural Networks\(Lim and Nelson,[2022](https://arxiv.org/html/2608.02668#bib.bib21)\), for which the structural constraints are not naturally compatible with our formulation\. Since SpheretNorms rely onℓ2\\ell\_\{2\}\-norm and the associated projection, they commute with a group representation only when that representation preserves the norm, and they generally break the equivariance condition for non\-orthogonal representations such as theGL​\(n\)\\mathrm\{GL\}\(n\)and Lorentz groups\. Second, owing to limited research resources, we have not evaluated SpheretNorms on larger\-scale deep Transformer models\. We leave the extension to such architectures and the verification at larger scales to future work\.

![Refer to caption](https://arxiv.org/html/2608.02668v1/figures/train_loss_fwe.png)\(a\)Training loss\.
![Refer to caption](https://arxiv.org/html/2608.02668v1/figures/val_loss_fwe.png)\(b\)Validation loss\.

Figure 2:Training and validation losses in the FineWeb\-Edu\.
## 7Conclusion and Future Work

In this work, we revisit retraction maps on the hypersphere and introduce three types of SpheretNorms: Proj\-SpheretNorm, Cay\-SpheretNorm, andpp\-SpheretNorm\. We analyze the three SpheretNorms from both theoretical and practical perspectives\. We implement them within the nanoGPT framework and evaluate them on pre\-training and downstream tasks\. The experimental results show that all three SpheretNorms achieve lower loss and higher accuracy than the baseline methods\.

At a deeper level, designing deep connection mechanisms is fundamentally a mathematical problem at the intersection of graph theory, optimization, and dynamical systems, raising two intertwined questions: \(i\) which connection topologies enable non\-trivial information propagation across depth while keeping the singular values of the end\-to\-end Jacobian bounded by constants independent of the depthLL; and \(ii\) among such topologies, is there a unifying principle that identifies the optimal trade\-off between information propagation and Jacobian conditioning? We leave these questions open and regard them as promising directions for future work\.

## References

- Absil and Malick \(2012\)P\.\-A\. Absil and Jérôme Malick\.Projection\-like Retractions on Matrix Manifolds\.*SIAM Journal on Optimization*, 22\(1\):135–158, 2012\.
- Absil et al\. \(2007\)P\.\-A\. Absil, R\. Mahony, and R\. Sepulchre\.*Optimization Algorithms on Matrix Manifolds*\.Princeton University Press, 2007\.ISBN 0691132984\.
- Ba et al\. \(2016\)Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E\. Hinton\.Layer Normalization, 2016\.
- Bachlechner et al\. \(2021\)Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley\.ReZero is all you need: fast convergence at large depth\.In Cassio de Campos and Marloes H\. Maathuis, editors,*Proceedings of the Thirty\-Seventh Conference on Uncertainty in Artificial Intelligence*, volume 161 of*Proceedings of Machine Learning Research*, pages 1352–1361\. PMLR, 07 2021\.
- Bisk et al\. \(2020\)Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi\.PIQA: Reasoning about Physical Commonsense in Natural Language\.*Thirty\-Fourth AAAI Conference on Artificial Intelligence*, 34\(05\):7432–7439, 04 2020\.
- Chen and Wei \(2026\)Chen Chen and Lai Wei\.Post\-LayerNorm Is Back: Stable, Expressive, and Deep, 2026\.
- Chen et al\. \(2026\)Mingzhi Chen, Taiming Lu, Jiachen Zhu, Mingjie Sun, and Zhuang Liu\.Stronger Normalization\-Free Transformers\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 27418–27428, 06 2026\.
- Cho et al\. \(2014\)Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio\.Learning Phrase Representations using RNN Encoder – Decoder for Statistical Machine Translation\.In*Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, pages 1724–1734\. Association for Computational Linguistics, 10 2014\.
- Clark et al\. \(2019\)Christopher Clark, Kenton Lee, Ming\-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova\.BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions\.In Jill Burstein, Christy Doran, and Thamar Solorio, editors,*Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\)*, pages 2924–2936\. Association for Computational Linguistics, 06 2019\.
- Clark et al\. \(2018\)Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord\.Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge, 2018\.
- Gao et al\. \(2021\)Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou\.The Language Model Evaluation Harness, 2021\.[https://github\.com/EleutherAI/lm\-evaluation\-harness](https://github.com/EleutherAI/lm-evaluation-harness)\.
- Gokaslan and Cohen \(2019\)Aaron Gokaslan and Vanya Cohen\.OpenWebText Corpus, 2019\.[https://Skylion007\.github\.io/OpenWebTextCorpus](https://skylion007.github.io/OpenWebTextCorpus)\.
- Gu et al\. \(2025\)Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin\.When Attention Sink Emerges in Language Models: An Empirical View\.In*The Thirteenth International Conference on Learning Representations*, 2025\.
- He et al\. \(2016a\)Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun\.Identity Mappings in Deep Residual Networks\.In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors,*Computer Vision – ECCV 2016*, pages 630–645\. Springer International Publishing, 2016a\.
- He et al\. \(2016b\)Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun\.Deep Residual Learning for Image Recognition\.In*2016 IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 770–778, 2016b\.
- Hochreiter and Schmidhuber \(1997\)Sepp Hochreiter and Jürgen Schmidhuber\.Long Short\-Term Memory\.*Neural Computation*, 9\(8\):1735–1780, 11 1997\.
- Ioffe and Szegedy \(2015\)Sergey Ioffe and Christian Szegedy\.Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift\.In Francis Bach and David Blei, editors,*Proceedings of the 32nd International Conference on Machine Learning*, volume 37 of*Proceedings of Machine Learning Research*, pages 448–456\. PMLR, 07 2015\.
- Karpathy \(2022\)Andrej Karpathy\.nanoGPT, 2022\.[https://github\.com/karpathy/nanoGPT](https://github.com/karpathy/nanoGPT)\.
- Kim et al\. \(2025\)Jeonghoon Kim, Byeongchan Lee, Cheonbok Park, Yeontaek Oh, Beomjun Kim, Taehwan Yoo, Seongjin Shin, Dongyoon Han, Jinwoo Shin, and Kang Min Yoo\.Peri\-LN: Revisiting Normalization Layer in the Transformer Architecture\.In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste\-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors,*Proceedings of the 42nd International Conference on Machine Learning*, volume 267 of*Proceedings of Machine Learning Research*, pages 30400–30436\. PMLR, 07 2025\.
- Li et al\. \(2025\)Pengxiang Li, Lu Yin, and Shiwei Liu\.Mix\-LN: Unleashing the Power of Deeper Layers by Combining Pre\-LN and Post\-LN\.In*The Thirteenth International Conference on Learning Representations*, 2025\.
- Lim and Nelson \(2022\)Lek\-Heng Lim and Bradley J\. Nelson\.What is an equivariant neural network?, 2022\.
- Loshchilov and Hutter \(2019\)Ilya Loshchilov and Frank Hutter\.Decoupled Weight Decay Regularization\.In*The Seventh International Conference on Learning Representations*, 2019\.
- Merity et al\. \(2017\)Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher\.Pointer Sentinel Mixture Models\.In*International Conference on Learning Representations*, 2017\.
- Mihaylov et al\. \(2018\)Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal\.Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering\.In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors,*Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, pages 2381–2391\. Association for Computational Linguistics, 10 2018\.
- Nguyen and Salazar \(2019\)Toan Q\. Nguyen and Julian Salazar\.Transformers without Tears: Improving the Normalization of Self\-Attention\.In*Proceedings of the 16th International Conference on Spoken Language Translation*\. Association for Computational Linguistics, 11 2019\.
- Paperno et al\. \(2016\)Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández\.The LAMBADA dataset: Word prediction requiring a broad discourse context\.In Katrin Erk and Noah A\. Smith, editors,*Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 1525–1534\. Association for Computational Linguistics, 08 2016\.
- Paszke et al\. \(2019\)Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala\.PyTorch: An Imperative Style, High\-Performance Deep Learning Library\.In H\. Wallach, H\. Larochelle, A\. Beygelzimer, F\. d'Alché\-Buc, E\. Fox, and R\. Garnett, editors,*Advances in Neural Information Processing Systems*, volume 32, pages 8024–8035\. Curran Associates, Inc\., 2019\.
- Penedo et al\. \(2024\)Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf\.The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale\.In A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang, editors,*The Thirty\-eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track*, volume 37, pages 30811–30849\. Curran Associates, Inc\., 2024\.
- Sakaguchi et al\. \(2021\)Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi\.WinoGrande: an adversarial winograd schema challenge at scale\.*Communications of the ACM*, 64\(9\):99–106, 08 2021\.
- Sap et al\. \(2019\)Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi\.Social IQa: Commonsense Reasoning about Social Interactions\.In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors,*Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\)*, pages 4463–4473\. Association for Computational Linguistics, 11 2019\.
- Stollenwerk \(2026\)Felix Stollenwerk\.On the Mathematical Relationship Between Layer Normalization and Dynamic Activation Functions\.In Vera Demberg, Kentaro Inui, and Lluís Marquez, editors,*Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 2: Short Papers\)*, pages 674–681\. Association for Computational Linguistics, 03 2026\.
- Sun et al\. \(2024\)Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu\.Massive Activations in Large Language Models\.In*First Conference on Language Modeling*, 2024\.
- Sun et al\. \(2025\)Wenfang Sun, Xinyuan Song, Pengxiang Li, Lu Yin, Yefeng Zheng, and Shiwei Liu\.The Curse of Depth in Large Language Models\.In*The Thirty\-ninth Annual Conference on Neural Information Processing Systems*, 2025\.
- Touvron et al\. \(2021\)Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou\.Going Deeper With Image Transformers\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\)*, pages 32–42, 10 2021\.
- Vaswani et al\. \(2017\)Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin\.Attention Is All You Need\.In I\. Guyon, U\. Von Luxburg, S\. Bengio, H\. Wallach, R\. Fergus, S\. Vishwanathan, and R\. Garnett, editors,*Advances in Neural Information Processing Systems*, volume 30, pages 5998–6008\. Curran Associates, Inc\., 2017\.
- Wang et al\. \(2019\)Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F\. Wong, and Lidia S\. Chao\.Learning Deep Transformer Models for Machine Translation\.In*Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pages 1810–1822\. Association for Computational Linguistics, 07 2019\.
- Welbl et al\. \(2017\)Johannes Welbl, Nelson F\. Liu, and Matt Gardner\.Crowdsourcing Multiple Choice Science Questions\.In Leon Derczynski, Wei Xu, Alan Ritter, and Tim Baldwin, editors,*Proceedings of the 3rd Workshop on Noisy User\-generated Text*, pages 94–106\. Association for Computational Linguistics, 09 2017\.
- Xiao et al\. \(2024\)Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis\.Efficient Streaming Language Models with Attention Sinks\.In*The Twelfth International Conference on Learning Representations*, 2024\.
- Xiong et al\. \(2020\)Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu\.On Layer Normalization in the Transformer Architecture\.In Hal Daumé III and Aarti Singh, editors,*Proceedings of the 37th International Conference on Machine Learning*, volume 119 of*Proceedings of Machine Learning Research*, pages 10524–10533\. PMLR, 07 2020\.
- Zellers et al\. \(2019\)Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi\.HellaSwag: Can a Machine Really Finish Your Sentence?In Anna Korhonen, David Traum, and Lluís Màrquez, editors,*Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pages 4791–4800\. Association for Computational Linguistics, 07 2019\.
- Zhang and Sennrich \(2019\)Biao Zhang and Rico Sennrich\.Root Mean Square Layer Normalization\.In H\. Wallach, H\. Larochelle, A\. Beygelzimer, F\. d'Alché\-Buc, E\. Fox, and R\. Garnett, editors,*Advances in Neural Information Processing Systems*, volume 32, pages 12360–12371\. Curran Associates, Inc\., 2019\.
- Zheng et al\. \(2026\)Chuanyang Zheng, Jiankai Sun, Yihang Gao, Chi Wang, Yuehao Wang, Jing Xiong, Liliang Ren, Bo Peng, Qingmei Wang, Xiaoran Shang, Mac Schwager, Anderson Schneider, Yuriy Nevmyvaka, and Xiaodong Liu\.GeoNorm: Unify Pre\-Norm and Post\-Norm with Geodesic Optimization, 2026\.
- Zhu et al\. \(2025\)Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, and Zhuang Liu\.Transformers without Normalization\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 14901–14911, 06 2025\.

\\beginappendix

## Appendix ANormalization as Hypersphere Projection

Throughout this appendix we work with idealized normalization maps: we omit the numerical stabilizerε\\varepsilonin the denominators, and we set all additive bias terms to zero, since a translation does not affect the projection interpretation\. Forr\>0r\>0, let

𝕊rd−1:=\{𝐱∈ℝd:∥𝐱∥2=r\},\\mathbb\{S\}^\{d\-1\}\_\{r\}:=\\bigl\\\{\\mathbf\{x\}\\in\\mathbb\{R\}^\{d\}:\\,\\lVert\\mathbf\{x\}\\rVert\_\{2\}=r\\bigr\\\},and, for𝐱∈ℝd∖\{𝟎\}\\mathbf\{x\}\\in\\mathbb\{R\}^\{d\}\\setminus\\\{\\bm\{0\}\\\}, define the radial projection onto𝕊rd−1\\mathbb\{S\}^\{d\-1\}\_\{r\}by

Πr​\(𝐱\):=r⋅𝐱∥𝐱∥2\.\\Pi\_\{r\}\(\\mathbf\{x\}\):=r\\cdot\\frac\{\\mathbf\{x\}\}\{\\lVert\\mathbf\{x\}\\rVert\_\{2\}\}\.For every nonzero𝐱\\mathbf\{x\}, the pointΠr​\(𝐱\)\\Pi\_\{r\}\(\\mathbf\{x\}\)is also the unique Euclidean metric projection of𝐱\\mathbf\{x\}onto𝕊rd−1\\mathbb\{S\}^\{d\-1\}\_\{r\}\. More generally, for a linear subspace𝒱⊆ℝd\\mathcal\{V\}\\subseteq\\mathbb\{R\}^\{d\}we write

𝕊r​\(𝒱\):=\{𝐲∈𝒱:∥𝐲∥2=r\},\\mathbb\{S\}\_\{r\}\(\\mathcal\{V\}\):=\\bigl\\\{\\mathbf\{y\}\\in\\mathcal\{V\}:\\,\\lVert\\mathbf\{y\}\\rVert\_\{2\}=r\\bigr\\\},which is a\(dim𝒱−1\)\(\\dim\\mathcal\{V\}\-1\)\-dimensional sphere isometric to𝕊rdim𝒱−1\\mathbb\{S\}^\{\\dim\\mathcal\{V\}\-1\}\_\{r\}\.

##### LayerNorm\.

Inspired by BatchNorm\(Ioffe and Szegedy,[2015](https://arxiv.org/html/2608.02668#bib.bib17)\),Ba et al\. \([2016](https://arxiv.org/html/2608.02668#bib.bib3)\)proposed LayerNorm, which normalizes each input vector along its feature dimension\. Let𝟏=\(1,…,1\)⊤∈ℝd\\mathbf\{1\}=\(1,\\ldots,1\)^\{\\top\}\\in\\mathbb\{R\}^\{d\}and define

μ=1d​𝟏⊤​𝐱,𝐏=𝐈−1d​𝟏𝟏⊤\.\\mu=\\frac\{1\}\{d\}\\mathbf\{1\}^\{\\top\}\\mathbf\{x\},\\quad\\mathbf\{P\}=\\mathbf\{I\}\-\\frac\{1\}\{d\}\\mathbf\{1\}\\mathbf\{1\}^\{\\top\}\.Since𝐏=𝐏⊤=𝐏2\\mathbf\{P\}=\\mathbf\{P\}^\{\\top\}=\\mathbf\{P\}^\{2\}andker⁡𝐏=span⁡\{𝟏\}\\ker\\mathbf\{P\}=\\operatorname\{span\}\\\{\\mathbf\{1\}\\\}, the matrix𝐏\\mathbf\{P\}is the orthogonal projection onto the mean\-zero hyperplane

ℋ=\{𝐲∈ℝd:𝟏⊤​𝐲=0\},dimℋ=d−1,\\mathcal\{H\}=\\bigl\\\{\\mathbf\{y\}\\in\\mathbb\{R\}^\{d\}:\\mathbf\{1\}^\{\\top\}\\mathbf\{y\}=0\\bigr\\\},\\quad\\dim\\mathcal\{H\}=d\-1,and the centered input satisfies𝐱−μ​𝟏=𝐏𝐱\\mathbf\{x\}\-\\mu\\mathbf\{1\}=\\mathbf\{P\}\\mathbf\{x\}\. Consequently, the coordinate\-wise standard deviation of𝐱\\mathbf\{x\}satisfies

σ=1d​∑i=1d\(xi−μ\)2=∥𝐏𝐱∥2d,\\sigma=\\sqrt\{\\frac\{1\}\{d\}\\sum\_\{i=1\}^\{d\}\(x\_\{i\}\-\\mu\)^\{2\}\}=\\frac\{\\lVert\\mathbf\{P\}\\mathbf\{x\}\\rVert\_\{2\}\}\{\\sqrt\{d\}\},so thatσ\>0\\sigma\>0if and only if𝐏𝐱≠𝟎\\mathbf\{P\}\\mathbf\{x\}\\neq\\bm\{0\}, i\.e\.,𝐱∉span⁡\{𝟏\}\\mathbf\{x\}\\notin\\operatorname\{span\}\\\{\\mathbf\{1\}\\\}; we restrict attention to this case, on which LayerNorm is well defined\. Let𝜸∈ℝd\\bm\{\\gamma\}\\in\\mathbb\{R\}^\{d\}denote the learnable gain and let𝐃𝜸=diag⁡\(𝜸\)\\mathbf\{D\}\_\{\\bm\{\\gamma\}\}=\\operatorname\{diag\}\(\\bm\{\\gamma\}\)\. LayerNorm can be written as

LayerNorm⁡\(𝐱\)=𝜸⊙𝐱−μ​𝟏σ=𝐃𝜸​\(d⋅𝐏𝐱∥𝐏𝐱∥2\)=𝐃𝜸​\(Πd​\(𝐏𝐱\)\)\.\\begin\{split\}\\operatorname\{LayerNorm\}\(\\mathbf\{x\}\)&=\\bm\{\\gamma\}\\odot\\frac\{\\mathbf\{x\}\-\\mu\\mathbf\{1\}\}\{\\sigma\}\\\\ &=\\mathbf\{D\}\_\{\\bm\{\\gamma\}\}\\left\(\\sqrt\{d\}\\cdot\\frac\{\\mathbf\{P\}\\mathbf\{x\}\}\{\\lVert\\mathbf\{P\}\\mathbf\{x\}\\rVert\_\{2\}\}\\right\)\\\\ &=\\mathbf\{D\}\_\{\\bm\{\\gamma\}\}\\left\(\\Pi\_\{\\sqrt\{d\}\}\(\\mathbf\{P\}\\mathbf\{x\}\)\\right\)\.\\end\{split\}\(12\)
Equation \([12](https://arxiv.org/html/2608.02668#A1.E12)\) shows that, before the coordinate\-wise gain is applied, LayerNorm performs two successive projections: the orthogonal projection of𝐱\\mathbf\{x\}ontoℋ\\mathcal\{H\}, followed by the radial projection of the centered vector onto𝕊d​\(ℋ\)\\mathbb\{S\}\_\{\\sqrt\{d\}\}\(\\mathcal\{H\}\), a\(d−2\)\(d\-2\)\-dimensional sphere of radiusd\\sqrt\{d\}contained inℋ\\mathcal\{H\}\. The image of the complete map is therefore

𝐃𝜸​\(𝕊d​\(ℋ\)\)=\{𝐃𝜸​𝐲:𝐲∈𝕊d​\(ℋ\)\},\\mathbf\{D\}\_\{\\bm\{\\gamma\}\}\\bigl\(\\mathbb\{S\}\_\{\\sqrt\{d\}\}\(\\mathcal\{H\}\)\\bigr\)=\\bigl\\\{\\mathbf\{D\}\_\{\\bm\{\\gamma\}\}\\mathbf\{y\}:\\,\\mathbf\{y\}\\in\\mathbb\{S\}\_\{\\sqrt\{d\}\}\(\\mathcal\{H\}\)\\bigr\\\},which lies in the hyperplane𝐃𝜸​ℋ\\mathbf\{D\}\_\{\\bm\{\\gamma\}\}\\mathcal\{H\}and is in general an ellipsoid, not necessarily axis\-aligned, rather than a sphere\. Assumed≥3d\\geq 3andγi≠0\\gamma\_\{i\}\\neq 0for everyii\. The image is a round sphere if and only if𝐃𝜸\\mathbf\{D\}\_\{\\bm\{\\gamma\}\}acts conformally onℋ\\mathcal\{H\}, i\.e\.,𝐏𝐃𝜸2​𝐏=c2​𝐏\\mathbf\{P\}\\mathbf\{D\}\_\{\\bm\{\\gamma\}\}^\{2\}\\mathbf\{P\}=c^\{2\}\\mathbf\{P\}for somec\>0c\>0\. Testing this identity on the pairs𝐞i−𝐞j\\mathbf\{e\}\_\{i\}\-\\mathbf\{e\}\_\{j\}and𝐞i−𝐞k\\mathbf\{e\}\_\{i\}\-\\mathbf\{e\}\_\{k\}withi,j,ki,j,kdistinct, which is possible becaused≥3d\\geq 3, yieldsγi2=c2\\gamma\_\{i\}^\{2\}=c^\{2\}for everyii\. Hence the image is a sphere precisely when\|γ1\|=\|γ2\|=⋯=\|γd\|\\lvert\\gamma\_\{1\}\\rvert=\\lvert\\gamma\_\{2\}\\rvert=\\cdots=\\lvert\\gamma\_\{d\}\\rvert\. Since𝜸\\bm\{\\gamma\}is typically initialized at𝟏\\mathbf\{1\}, at initialization LayerNorm reduces to𝐱↦Πd​\(𝐏𝐱\)\\mathbf\{x\}\\mapsto\\Pi\_\{\\sqrt\{d\}\}\(\\mathbf\{P\}\\mathbf\{x\}\)and all outputs share the same normd\\sqrt\{d\}\. Once training drives𝜸\\bm\{\\gamma\}away from𝟏\\mathbf\{1\}, the outputs leave𝕊d​\(ℋ\)\\mathbb\{S\}\_\{\\sqrt\{d\}\}\(\\mathcal\{H\}\); they remain on some sphere only in the special case in which all\|γi\|\\lvert\\gamma\_\{i\}\\rvertcoincide, and are otherwise confined to the ellipsoid𝐃𝜸​\(𝕊d​\(ℋ\)\)\\mathbf\{D\}\_\{\\bm\{\\gamma\}\}\\left\(\\mathbb\{S\}\_\{\\sqrt\{d\}\}\(\\mathcal\{H\}\)\\right\)\.

##### RMSNorm\.

RMSNorm\(Zhang and Sennrich,[2019](https://arxiv.org/html/2608.02668#bib.bib41)\)dispenses with the mean\-centering step of LayerNorm and normalizes an input solely by its root mean square\. For𝐱∈ℝd∖\{𝟎\}\\mathbf\{x\}\\in\\mathbb\{R\}^\{d\}\\setminus\\\{\\bm\{0\}\\\}, define the root mean square \(RMS\) operation as

RMS⁡\(𝐱\)=1d​∑i=1dxi2=∥𝐱∥2d\.\\operatorname\{RMS\}\(\\mathbf\{x\}\)=\\sqrt\{\\frac\{1\}\{d\}\\sum\_\{i=1\}^\{d\}x\_\{i\}^\{2\}\}=\\frac\{\\lVert\\mathbf\{x\}\\rVert\_\{2\}\}\{\\sqrt\{d\}\}\.Let𝜸∈ℝd\\bm\{\\gamma\}\\in\\mathbb\{R\}^\{d\}denote the learnable coordinate\-wise gain and let𝐃𝜸=diag⁡\(𝜸\)\\mathbf\{D\}\_\{\\bm\{\\gamma\}\}=\\operatorname\{diag\}\(\\bm\{\\gamma\}\)\. RMSNorm is then given by

RMSNorm⁡\(𝐱\)=𝜸⊙𝐱RMS⁡\(𝐱\)=𝐃𝜸​\(d⋅𝐱∥𝐱∥2\)=𝐃𝜸​Πd​\(𝐱\)\.\\begin\{split\}\\operatorname\{RMSNorm\}\(\\mathbf\{x\}\)&=\\bm\{\\gamma\}\\odot\\frac\{\\mathbf\{x\}\}\{\\operatorname\{RMS\}\(\\mathbf\{x\}\)\}\\\\ &=\\mathbf\{D\}\_\{\\bm\{\\gamma\}\}\\left\(\\sqrt\{d\}\\cdot\\frac\{\\mathbf\{x\}\}\{\\lVert\\mathbf\{x\}\\rVert\_\{2\}\}\\right\)\\\\ &=\\mathbf\{D\}\_\{\\bm\{\\gamma\}\}\\Pi\_\{\\sqrt\{d\}\}\(\\mathbf\{x\}\)\.\\end\{split\}\(13\)The normalization stage of RMSNorm, prior to the coordinate\-wise gain, is therefore precisely the radial projection onto𝕊dd−1\\mathbb\{S\}^\{d\-1\}\_\{\\sqrt\{d\}\}\. The complete map, however, is in general no longer spherical: its image is𝐃𝜸​\(𝕊dd−1\)\\mathbf\{D\}\_\{\\bm\{\\gamma\}\}\\\!\\left\(\\mathbb\{S\}^\{d\-1\}\_\{\\sqrt\{d\}\}\\right\), which, whenγi≠0\\gamma\_\{i\}\\neq 0for everyii, is an axis\-aligned ellipsoid whose semi\-axes have lengthsd​\|γi\|\\sqrt\{d\}\\,\\lvert\\gamma\_\{i\}\\rvert; if someγi\\gamma\_\{i\}vanishes, the image degenerates to a lower\-dimensional set\. Ford≥2d\\geq 2, the spherical geometry is retained precisely when all entries of𝜸\\bm\{\\gamma\}share a common absolute valuec≠0c\\neq 0, in which case the image is the rescaled sphere𝕊c​dd−1\\mathbb\{S\}^\{d\-1\}\_\{c\\sqrt\{d\}\}\. Since𝜸\\bm\{\\gamma\}is typically initialized at𝟏\\mathbf\{1\}, RMSNorm coincides with the projection onto𝕊dd−1\\mathbb\{S\}^\{d\-1\}\_\{\\sqrt\{d\}\}at initialization; once training updates𝜸\\bm\{\\gamma\}, its outputs generally drift off the hypersphere\.

##### ScaleNorm\.

Among commonly used normalization methods, ScaleNorm\(Nguyen and Salazar,[2019](https://arxiv.org/html/2608.02668#bib.bib25)\)is the one most directly related to hypersphere projection\. For𝐱∈ℝd∖\{𝟎\}\\mathbf\{x\}\\in\\mathbb\{R\}^\{d\}\\setminus\\\{\\bm\{0\}\\\}it is defined by

ScaleNorm⁡\(𝐱\)=g​𝐱∥𝐱∥2,\\operatorname\{ScaleNorm\}\(\\mathbf\{x\}\)=g\\,\\frac\{\\mathbf\{x\}\}\{\\lVert\\mathbf\{x\}\\rVert\_\{2\}\},\(14\)whereg∈ℝg\\in\\mathbb\{R\}is a learnable scalar parameter\. Ifg\>0g\>0, then

ScaleNorm⁡\(𝐱\)=Πg​\(𝐱\),\\operatorname\{ScaleNorm\}\(\\mathbf\{x\}\)=\\Pi\_\{g\}\(\\mathbf\{x\}\),so ScaleNorm maps every nonzero input onto the hypersphere𝕊gd−1\\mathbb\{S\}^\{d\-1\}\_\{g\}\. More generally, whenggis unconstrained,

∥ScaleNorm⁡\(𝐱\)∥2=\|g\|\.\\lVert\\operatorname\{ScaleNorm\}\(\\mathbf\{x\}\)\\rVert\_\{2\}=\\lvert g\\rvert\.Thus, forg<0g<0the spherical interpretation still holds: the map is the composition of the radial projectionΠ\|g\|\\Pi\_\{\\lvert g\\rvert\}onto𝕊\|g\|d−1\\mathbb\{S\}^\{d\-1\}\_\{\\lvert g\\rvert\}with the antipodal map𝐱↦−𝐱\\mathbf\{x\}\\mapsto\-\\mathbf\{x\}, and only the degenerate caseg=0g=0collapses the image to the origin\. If an explicitly positive radius parameter is needed, one may use a positive reparameterization such as replacingggwithSoftplus⁡\(g\)\\operatorname\{Softplus\}\(g\), although this is not required for the geometric interpretation\.

##### Connecting ScaleNorm with Proj\-SpheretNorm\.

The relationship between ScaleNorm and Proj\-SpheretNorm can be made precise through their common radial\-projection structure\. Let𝐮∈ℝd∖\{𝟎\}\\mathbf\{u\}\\in\\mathbb\{R\}^\{d\}\\setminus\\\{\\bm\{0\}\\\}be the input to a generic Proj\-SpheretNorm update, letℱ\\mathcal\{F\}denote the associated sub\-layer map, and letα∈ℝ\\alpha\\in\\mathbb\{R\}be the residual coefficient\. Set

r=∥𝐮∥2,𝐏𝐮=𝐈−𝐮𝐮⊤r2,𝐳=𝐏𝐮​ℱ​\(𝐮\)\.r=\\lVert\\mathbf\{u\}\\rVert\_\{2\},\\qquad\\mathbf\{P\}\_\{\\mathbf\{u\}\}=\\mathbf\{I\}\-\\frac\{\\mathbf\{u\}\\mathbf\{u\}^\{\\top\}\}\{r^\{2\}\},\\qquad\\mathbf\{z\}=\\mathbf\{P\}\_\{\\mathbf\{u\}\}\\mathcal\{F\}\(\\mathbf\{u\}\)\.Here𝐏𝐮\\mathbf\{P\}\_\{\\mathbf\{u\}\}is the orthogonal projection onto𝐮⟂\\mathbf\{u\}^\{\\perp\}, which is the tangent space of𝕊rd−1\\mathbb\{S\}^\{d\-1\}\_\{r\}at𝐮\\mathbf\{u\}; in particular𝐮⊤​𝐳=0\\mathbf\{u\}^\{\\top\}\\mathbf\{z\}=0by construction\. The Proj\-SpheretNorm update can therefore be written as

𝐮\+=r​𝐮\+α​𝐳∥𝐮\+α​𝐳∥2=Πr​\(𝐮\+α​𝐳\)\.\\mathbf\{u\}^\{\+\}=r\\,\\frac\{\\mathbf\{u\}\+\\alpha\\mathbf\{z\}\}\{\\lVert\\mathbf\{u\}\+\\alpha\\mathbf\{z\}\\rVert\_\{2\}\}=\\Pi\_\{r\}\\bigl\(\\mathbf\{u\}\+\\alpha\\mathbf\{z\}\\bigr\)\.\(15\)Moreover, orthogonality gives

∥𝐮\+α​𝐳∥22=r2\+α2​∥𝐳∥22\>0,\\lVert\\mathbf\{u\}\+\\alpha\\mathbf\{z\}\\rVert\_\{2\}^\{2\}=r^\{2\}\+\\alpha^\{2\}\\lVert\\mathbf\{z\}\\rVert\_\{2\}^\{2\}\>0,so the radial projection is well defined whenever𝐮≠𝟎\\mathbf\{u\}\\neq\\bm\{0\}\. Equation \([15](https://arxiv.org/html/2608.02668#A1.E15)\) also gives∥𝐮\+∥2=∥𝐮∥2\\lVert\\mathbf\{u\}^\{\+\}\\rVert\_\{2\}=\\lVert\\mathbf\{u\}\\rVert\_\{2\}\.

Forl=1,2,…,L−1l=1,\\,2,\\,\\ldots,\\,L\-1, the\(l\+1\)\(l\+1\)\-th hidden state is given by

𝐡l\+1=SpheretNorm⁡\(𝐡l,ℱl​\(𝐡l\)\)\.\\mathbf\{h\}\_\{l\+1\}=\\operatorname\{SpheretNorm\}\\\!\\left\(\\mathbf\{h\}\_\{l\},\\,\\mathcal\{F\}\_\{l\}\(\\mathbf\{h\}\_\{l\}\)\\right\)\.When Proj\-SpheretNorm is used, the definition in Eq\. \([6](https://arxiv.org/html/2608.02668#S4.E6)\) gives

𝐡l\+1=Π∥𝐡l∥2​\(𝐡l\+αl​𝐳l\),𝐳l=𝐏𝐡l​ℱl​\(𝐡l\),\\mathbf\{h\}\_\{l\+1\}=\\Pi\_\{\\lVert\\mathbf\{h\}\_\{l\}\\rVert\_\{2\}\}\\left\(\\mathbf\{h\}\_\{l\}\+\\alpha\_\{l\}\\mathbf\{z\}\_\{l\}\\right\),\\qquad\\mathbf\{z\}\_\{l\}=\\mathbf\{P\}\_\{\\mathbf\{h\}\_\{l\}\}\\mathcal\{F\}\_\{l\}\(\\mathbf\{h\}\_\{l\}\),so that∥𝐡l\+1∥2=∥𝐡l∥2\\lVert\\mathbf\{h\}\_\{l\+1\}\\rVert\_\{2\}=\\lVert\\mathbf\{h\}\_\{l\}\\rVert\_\{2\}forl=1,2,…,L−1l=1,2,\\ldots,L\-1\. Consequently, the residual stream preserves the hidden\-state norm across layers, i\.e\.,∥𝐡1∥2=∥𝐡2∥2=⋯=∥𝐡L∥2\\lVert\\mathbf\{h\}\_\{1\}\\rVert\_\{2\}=\\lVert\\mathbf\{h\}\_\{2\}\\rVert\_\{2\}=\\cdots=\\lVert\\mathbf\{h\}\_\{L\}\\rVert\_\{2\}\.

At the beginning of the residual stream considered in our analysis, we take𝐡1\\mathbf\{h\}\_\{1\}to be the hidden state produced by the modified ScaleNorm, so that, by construction,

∥𝐡1∥2=Softplus⁡\(γ\),\\lVert\\mathbf\{h\}\_\{1\}\\rVert\_\{2\}=\\operatorname\{Softplus\}\(\\gamma\),which is strictly positive for everyγ∈ℝ\\gamma\\in\\mathbb\{R\}\. We do not introduce notation for the representations preceding𝐡1\\mathbf\{h\}\_\{1\}, and any operations applied after𝐡L\\mathbf\{h\}\_\{L\}lie outside the scope of this analysis\.

Equation \([15](https://arxiv.org/html/2608.02668#A1.E15)\) indicates that ScaleNorm and Proj\-SpheretNorm share the same radial\-projection operation but use it for different purposes\. ScaleNorm fixes the radius of the initial hidden state𝐡1\\mathbf\{h\}\_\{1\}, whereas each Proj\-SpheretNorm sub\-layer forms a tangent residual update and projects the resulting point back onto the sphere determined by the radius of its own input state\. Under the above fixed\-radius dynamics, every hidden state along the residual stream lies on the same hypersphere𝕊Softplus⁡\(γ\)d−1\\mathbb\{S\}^\{d\-1\}\_\{\\operatorname\{Softplus\}\(\\gamma\)\}\.

In summary, LayerNorm, RMSNorm, and ScaleNorm are all related to hypersphere projection, but in different ways\. The normalization stages of LayerNorm and RMSNorm, taken before the gain, are spherical maps: LayerNorm first projects onto the zero\-mean hyperplane and then performs a radial projection within that hyperplane, while RMSNorm performs a radial projection in the ambient space\. Their complete maps, however, generally include a coordinate\-wise gain and therefore need not have spherical images\.

ScaleNorm differs in that it uses a single scalar gain\. Its complete map therefore has a spherical image of radius\|g\|\\lvert g\\rvert, and coincides exactly with the radial projectionΠg\\Pi\_\{g\}wheng\>0g\>0\. This positive\-radius case provides the closest traditional counterpart to Proj\-SpheretNorm\. Specifically, Proj\-SpheretNorm replaces the globally learned radiusggwith the radius∥𝐡l∥2\\lVert\\mathbf\{h\}\_\{l\}\\rVert\_\{2\}carried by the current hidden state, and applies the radial projection to the point obtained after the tangent update\. It can thus be viewed as a state\-dependent, retraction\-based variant of the radial projection used in ScaleNorm, designed to preserve the hidden\-state norm exactly across the entire depth\.

## Appendix BHyperparameters

The shared hyperparameters used across all experiments are summarized in Table[6](https://arxiv.org/html/2608.02668#A2.T6)\. The architecture hyperparameters for the small and medium model sizes are listed in Table[7](https://arxiv.org/html/2608.02668#A2.T7)\. The training and validation losses in OpenWebText is illustrated in Figure[3](https://arxiv.org/html/2608.02668#A2.F3)\.

Table 6:Shared hyperparameters\. The effective batch size is held fixed across all runs, independent of the number of GPUs\.NameValueoptimizerAdamWβ1\\beta\_\{1\}0\.90\.9β2\\beta\_\{2\}0\.950\.95weight decay0\.10\.1gradient clip1\.01\.0dropout0\.00\.0biasenable\# of iterations50,00050\{,\}000\# of learning\-rate decay iterations50,00050\{,\}000\# of warm\-up iterations2,0002\{,\}000block size \(sequence length\)1,0241\{,\}024batch size \(per GPU\)1616effective \(global\) batch size128128sequencestokens per update131,072131\{,\}072total training tokens∼6\.55\\sim\\\!6\.55Bprecisionbfloat16torch\.compiledisablevocabulary size50,30450\{,\}304Table 7:Architecture hyperparameters\. Parameter counts include the tied token embedding and are reported for the Pre\-LN baseline\. All variants have identical parameter counts\.NameSML\# of layers121224243636\# of heads121216162020\# of params\.∼0\.12\\sim\\\!0\.12B∼0\.35\\sim\\\!0\.35B∼0\.77\\sim\\\!0\.77Bhidden dim\.7687681024102412801280lr6×10−46\\times 10^\{\-4\}3×10−43\\times 10^\{\-4\}2\.5×10−42\.5\\times 10^\{\-4\}min\. lr6×10−56\\times 10^\{\-5\}3×10−53\\times 10^\{\-5\}2\.5×10−52\.5\\times 10^\{\-5\}![Refer to caption](https://arxiv.org/html/2608.02668v1/figures/train_loss_owt.png)\(a\)Training loss\.
![Refer to caption](https://arxiv.org/html/2608.02668v1/figures/val_loss_owt.png)\(b\)Validation loss\.

Figure 3:Training and validation losses in the OpenWebText\.
## Appendix CPseudocode for SpheretNorms

In this section, we provide PyTorch\-style pseudocode for our SpheretNorms: Proj\-SpheretNorm, Cay\-SpheretNorm, andpp\-SpheretNorm\.

List of codeblocks 1PyTorch\-style pseudocode for GeoNorm\.1defgeonorm\(h\_in,Fh,decay\_method,layer\_idx,num\_layers,angle\_upper\_bound,alpha\):

2h\_in\_norm\_sq=\(h\_in\*h\_in\)\.sum\(dim=\-1,keepdim=True\)\.clamp\(min=1e\-12\)

3h\_in\_norm=h\_in\_norm\_sq\.sqrt\(\)

4h\_in\_trs\_dot\_Fh=\(h\_in\*Fh\)\.sum\(dim=\-1,keepdim=True\)

5

6z=Fh\-\(h\_in\_trs\_dot\_Fh/h\_in\_norm\_sq\)\*h\_in

7z\_norm\_sq=\(z\*z\)\.sum\(dim=\-1,keepdim=True\)\.clamp\(min=1e\-12\)

8z\_norm=z\_norm\_sq\.sqrt\(\)

9z\_hs=z/z\_norm

10

11ifdecay\_method=='harmonic':

12alpha=torch\.clamp\(F\.softplus\(alpha\),max=1\.0\)/layer\_idx

13elifdecay\_method=='linear':

14alpha=torch\.clamp\(F\.softplus\(alpha\),max=1\.0\)\*\(num\_layers\-layer\_idx\)/num\_layers

15elifdecay\_method=='sqrt':

16alpha=torch\.clamp\(F\.softplus\(alpha\),max=1\.0\)/math\.sqrt\(layer\_idx\)

17

18beta=\(alpha\*\(z\_norm/h\_in\_norm\)\)\.clamp\(max=angle\_upper\_bound\)

19h\_out=torch\.cos\(beta\)\*h\_in\+h\_in\_norm\*torch\.sin\(beta\)\*z\_hs

20

21returnh\_out

List of codeblocks 2PyTorch\-style pseudocode for Proj\-SpheretNorm\.1defproj\_spheretnorm\(h\_in,Fh,layer\_idx,alpha\):

2h\_in\_norm\_sq=\(h\_in\*h\_in\)\.sum\(dim=\-1,keepdim=True\)\.clamp\(min=1e\-12\)

3h\_in\_norm=h\_in\_norm\_sq\.sqrt\(\)

4h\_in\_trs\_dot\_Fh=\(h\_in\*Fh\)\.sum\(dim=\-1,keepdim=True\)

5

6z=Fh\-\(h\_in\_trs\_dot\_Fh/h\_in\_norm\_sq\)\*h\_in

7

8alpha=torch\.clamp\(F\.softplus\(alpha\),max=1\.0\)/math\.sqrt\(self\.layer\_idx\)

9

10h\_plus=h\_in\+alpha\*z

11h\_plus\_norm\_sq=\(h\_plus\*h\_plus\)\.sum\(dim=\-1,keepdim=True\)\.clamp\(min=1e\-12\)

12h\_plus\_norm=h\_plus\_norm\_sq\.sqrt\(\)

13h\_out=h\_in\_norm\*\(h\_plus/h\_plus\_norm\)

14

15returnh\_out

List of codeblocks 3PyTorch\-style pseudocode for Cay\-SpheretNorm\.1defcay\_spheretnorm\(h\_in,Fh,layer\_idx,angle\_upper\_bound,alpha\):

2h\_in\_norm\_sq=\(h\_in\*h\_in\)\.sum\(dim=\-1,keepdim=True\)\.clamp\(min=1e\-12\)

3h\_in\_norm=h\_in\_norm\_sq\.sqrt\(\)

4h\_in\_trs\_dot\_Fh=\(h\_in\*Fh\)\.sum\(dim=\-1,keepdim=True\)

5

6z=Fh\-\(h\_in\_trs\_dot\_Fh/h\_in\_norm\_sq\)\*h\_in

7z\_norm\_sq=\(z\*z\)\.sum\(dim=\-1,keepdim=True\)\.clamp\(min=1e\-12\)

8z\_norm=z\_norm\_sq\.sqrt\(\)

9

10alpha=torch\.clamp\(F\.softplus\(alpha\),max=1\.0\)/math\.sqrt\(layer\_idx\)

11

12beta\_max=math\.tan\(angle\_upper\_bound\*0\.5\)\*2\.0

13gamma=torch\.minimum\(alpha,beta\_max\*\(h\_in\_norm/z\_norm\)\)

14beta=gamma\*\(z\_norm/h\_in\_norm\)

15beta\_sq=beta\*beta

16h\_out=\(\(4\-beta\_sq\)\*h\_in\+4\*gamma\*z\)/\(4\+beta\_sq\)

17

18returnh\_out

List of codeblocks 4PyTorch\-style pseudocode forpp\-SpheretNorm\.1defp\_spheretnorm\(h\_in,Fh,layer\_idx,angle\_upper\_bound,alpha,p\):

2h\_in\_norm\_sq=\(h\_in\*h\_in\)\.sum\(dim=\-1,keepdim=True\)\.clamp\(min=1e\-12\)

3h\_in\_norm=h\_in\_norm\_sq\.sqrt\(\)

4h\_in\_trs\_dot\_Fh=\(h\_in\*Fh\)\.sum\(dim=\-1,keepdim=True\)

5

6z=Fh\-\(h\_in\_trs\_dot\_Fh/h\_in\_norm\_sq\)\*h\_in

7z\_norm\_sq=\(z\*z\)\.sum\(dim=\-1,keepdim=True\)\.clamp\(min=1e\-12\)

8z\_norm=z\_norm\_sq\.sqrt\(\)

9z\_hs=z/z\_norm

10

11alpha=torch\.clamp\(F\.softplus\(alpha\),max=1\.0\)/math\.sqrt\(self\.layer\_idx\)

12

13beta=alpha\*\(z\_norm/h\_in\_norm\)

14theta=p\*torch\.arctan\(beta/p\)

15ifp\>1\.0:

16theta=torch\.clamp\(theta,max=angle\_upper\_bound\)

17

18h\_out=torch\.cos\(theta\)\*h\_in\+\(h\_in\_norm\*torch\.sin\(theta\)\)\*z\_hs

19

20returnh\_out

## Appendix DTechnical Proofs for Section[4](https://arxiv.org/html/2608.02668#S4)

### D\.1Proof of Proposition[3](https://arxiv.org/html/2608.02668#Thmtheorem3)

Let𝒰=span⁡\{𝐡,𝐯\}\\mathcal\{U\}=\\operatorname\{span\}\\\{\\mathbf\{h\},\\mathbf\{v\}\\\}\. We first prove uniqueness\. Suppose thatW∈𝔰​𝔬​\(d\)W\\in\\mathfrak\{so\}\(d\)satisfies

𝐖𝐡=𝐯,range⁡\(𝐖\)⊆𝒰\.\\mathbf\{W\}\\mathbf\{h\}=\\mathbf\{v\},\\qquad\\operatorname\{range\}\(\\mathbf\{W\}\)\\subseteq\\mathcal\{U\}\.
For every𝐱∈𝒰⟂\\mathbf\{x\}\\in\\mathcal\{U\}^\{\\perp\}and𝐲∈𝒰\\mathbf\{y\}\\in\\mathcal\{U\}, skew\-symmetry gives

⟨𝐖𝐱,𝐲⟩=−⟨𝐱,𝐖𝐲⟩\.\\langle\\mathbf\{W\}\\mathbf\{x\},\\mathbf\{y\}\\rangle=\-\\langle\\mathbf\{x\},\\mathbf\{W\}\\mathbf\{y\}\\rangle\.Since𝐖𝐲∈𝒰\\mathbf\{W\}\\mathbf\{y\}\\in\\mathcal\{U\}and𝐱∈𝒰⟂\\mathbf\{x\}\\in\\mathcal\{U\}^\{\\perp\}, it follows that⟨𝐖𝐱,𝐲⟩=0\\langle\\mathbf\{W\}\\mathbf\{x\},\\mathbf\{y\}\\rangle=0for every𝐲∈𝒰\\mathbf\{y\}\\in\\mathcal\{U\}\. On the other hand,𝐖𝐱∈𝒰\\mathbf\{W\}\\mathbf\{x\}\\in\\mathcal\{U\}by the range condition\. Therefore,

𝐖𝐱=𝟎,𝐱∈𝒰⟂\.\\mathbf\{W\}\\mathbf\{x\}=\\bm\{0\},\\qquad\\mathbf\{x\}\\in\\mathcal\{U\}^\{\\perp\}\.Thus,𝐖\\mathbf\{W\}is completely determined by its restriction to𝒰\\mathcal\{U\}\.

If𝐯=𝟎\\mathbf\{v\}=\\bm\{0\}, then𝒰=span⁡\{𝐡\}\\mathcal\{U\}=\\operatorname\{span\}\\\{\\mathbf\{h\}\\\}and𝐖𝐡=𝟎\\mathbf\{W\}\\mathbf\{h\}=\\bm\{0\}\. Hence𝐖=𝟎\\mathbf\{W\}=\\bm\{0\}, which agrees with the claimed formula\.

Suppose now that𝐯≠𝟎\\mathbf\{v\}\\neq\\bm\{0\}, and define the orthonormal vectors

𝐞1=𝐡r,𝐞2=𝐯‖𝐯‖2\.\\mathbf\{e\}\_\{1\}=\\frac\{\\mathbf\{h\}\}\{r\},\\qquad\\mathbf\{e\}\_\{2\}=\\frac\{\\mathbf\{v\}\}\{\\\|\\mathbf\{v\}\\\|\_\{2\}\}\.Since𝐖𝐡=𝐯\\mathbf\{W\}\\mathbf\{h\}=\\mathbf\{v\}, we have

𝐖𝐞1=‖𝐯‖2r​𝐞2\.\\mathbf\{W\}\\mathbf\{e\}\_\{1\}=\\frac\{\\\|\\mathbf\{v\}\\\|\_\{2\}\}\{r\}\\mathbf\{e\}\_\{2\}\.Skew\-symmetry then uniquely determines the action on𝐞2\\mathbf\{e\}\_\{2\}:𝐖𝐞2=−‖𝐯‖2r​𝐞1\\mathbf\{W\}\\mathbf\{e\}\_\{2\}=\-\\frac\{\\\|\\mathbf\{v\}\\\|\_\{2\}\}\{r\}\\mathbf\{e\}\_\{1\}\. Together with𝐖=𝟎\\mathbf\{W\}=\\bm\{0\}on𝒰⟂\\mathcal\{U\}^\{\\perp\}, this proves that at most one such skew\-symmetric matrix exists\.

Now define

𝐖^=𝐯𝐡⊤−𝐡𝐯⊤r2\.\\widehat\{\\mathbf\{W\}\}=\\frac\{\\mathbf\{v\}\\mathbf\{h\}^\{\\top\}\-\\mathbf\{h\}\\mathbf\{v\}^\{\\top\}\}\{r^\{2\}\}\.Then𝐖^⊤=−𝐖^\\widehat\{\\mathbf\{W\}\}^\{\\top\}=\-\\widehat\{\\mathbf\{W\}\}, so𝐖^∈𝔰​𝔬​\(d\)\\widehat\{\\mathbf\{W\}\}\\in\\mathfrak\{so\}\(d\)\. Since𝐡⊤​𝐯=0\\mathbf\{h\}^\{\\top\}\\mathbf\{v\}=0and‖𝐡‖22=r2\\\|\\mathbf\{h\}\\\|\_\{2\}^\{2\}=r^\{2\},

𝐖^​𝐡=𝐯𝐡⊤​𝐡−𝐡𝐯⊤​𝐡r2=𝐯\.\\widehat\{\\mathbf\{W\}\}\\mathbf\{h\}=\\frac\{\\mathbf\{v\}\\mathbf\{h\}^\{\\top\}\\mathbf\{h\}\-\\mathbf\{h\}\\mathbf\{v\}^\{\\top\}\\mathbf\{h\}\}\{r^\{2\}\}=\\mathbf\{v\}\.Moreover,range⁡\(𝐖^\)⊆span⁡\{𝐡,𝐯\}\\operatorname\{range\}\(\\widehat\{\\mathbf\{W\}\}\)\\subseteq\\operatorname\{span\}\\\{\\mathbf\{h\},\\mathbf\{v\}\\\}\. Therefore, by uniqueness,

𝐖=𝐖^=𝐯𝐡⊤−𝐡𝐯⊤r2\.\\mathbf\{W\}=\\widehat\{\\mathbf\{W\}\}=\\frac\{\\mathbf\{v\}\\mathbf\{h\}^\{\\top\}\-\\mathbf\{h\}\\mathbf\{v\}^\{\\top\}\}\{r^\{2\}\}\.
We next derive the closed form of the Cayley update\. Defineρ=‖𝐯‖2r\\rho=\\frac\{\\\|\\mathbf\{v\}\\\|\_\{2\}\}\{r\}, we obtain

𝐖𝐯=𝐯𝐡⊤​𝐯−𝐡𝐯⊤​𝐯r2=−ρ2​𝐡\.\\mathbf\{W\}\\mathbf\{v\}=\\frac\{\\mathbf\{v\}\\mathbf\{h\}^\{\\top\}\\mathbf\{v\}\-\\mathbf\{h\}\\mathbf\{v\}^\{\\top\}\\mathbf\{v\}\}\{r^\{2\}\}=\-\\rho^\{2\}\\mathbf\{h\}\.
The matrix𝐈−12​𝐖\\mathbf\{I\}\-\\frac\{1\}\{2\}\\mathbf\{W\}is invertible\. Indeed, if\(𝐈−12​𝐖\)​𝐱=0\\left\(\\mathbf\{I\}\-\\frac\{1\}\{2\}\\mathbf\{W\}\\right\)\\mathbf\{x\}=0, then𝐱=12​𝐖𝐱\\mathbf\{x\}=\\frac\{1\}\{2\}\\mathbf\{W\}\\mathbf\{x\}\. Taking the inner product with𝐱\\mathbf\{x\}and using𝐱⊤​𝐖𝐱=𝟎\\mathbf\{x\}^\{\\top\}\\mathbf\{W\}\\mathbf\{x\}=\\bm\{0\}gives‖𝐱‖22=0\\\|\\mathbf\{x\}\\\|\_\{2\}^\{2\}=0, and hence𝐱=𝟎\\mathbf\{x\}=\\bm\{0\}\.

Seta=4−ρ24\+ρ2a=\\frac\{4\-\\rho^\{2\}\}\{4\+\\rho^\{2\}\}andb=44\+ρ2b=\\frac\{4\}\{4\+\\rho^\{2\}\}\. Using𝐖𝐡=𝐯\\mathbf\{W\}\\mathbf\{h\}=\\mathbf\{v\}and𝐖𝐯=−ρ2​𝐡\\mathbf\{W\}\\mathbf\{v\}=\-\\rho^\{2\}\\mathbf\{h\}, we have

\(𝐈−12​𝐖\)​\(a​𝐡\+b​𝐯\)\\displaystyle\\left\(\\mathbf\{I\}\-\\frac\{1\}\{2\}\\mathbf\{W\}\\right\)\(a\\mathbf\{h\}\+b\\mathbf\{v\}\)=a​𝐡\+b​𝐯−a2​𝐯\+b​ρ22​𝐡\\displaystyle=a\\mathbf\{h\}\+b\\mathbf\{v\}\-\\frac\{a\}\{2\}\\mathbf\{v\}\+\\frac\{b\\rho^\{2\}\}\{2\}\\mathbf\{h\}=\(a\+b​ρ22\)​𝐡\+\(b−a2\)​𝐯\.\\displaystyle=\\left\(a\+\\frac\{b\\rho^\{2\}\}\{2\}\\right\)\\mathbf\{h\}\+\\left\(b\-\\frac\{a\}\{2\}\\right\)\\mathbf\{v\}\.The definitions ofaaandbbgivea\+b​ρ22=1a\+\\frac\{b\\rho^\{2\}\}\{2\}=1andb−a2=12b\-\\frac\{a\}\{2\}=\\frac\{1\}\{2\}\. Therefore,

\(𝐈−12​𝐖\)​\(a​𝐡\+b​𝐯\)=𝐡\+12​𝐯=\(𝐈\+12​𝐖\)​𝐡\.\\left\(\\mathbf\{I\}\-\\frac\{1\}\{2\}\\mathbf\{W\}\\right\)\(a\\mathbf\{h\}\+b\\mathbf\{v\}\)=\\mathbf\{h\}\+\\frac\{1\}\{2\}\\mathbf\{v\}=\\left\(\\mathbf\{I\}\+\\frac\{1\}\{2\}\\mathbf\{W\}\\right\)\\mathbf\{h\}\.Since𝐈−12​𝐖\\mathbf\{I\}\-\\frac\{1\}\{2\}\\mathbf\{W\}is invertible,

𝒞​\(W\)​𝐡\\displaystyle\\mathcal\{C\}\(W\)\\mathbf\{h\}=\(𝐈−12​𝐖\)−1​\(𝐈\+12​𝐖\)​𝐡\\displaystyle=\\left\(\\mathbf\{I\}\-\\frac\{1\}\{2\}\\mathbf\{W\}\\right\)^\{\-1\}\\left\(\\mathbf\{I\}\+\\frac\{1\}\{2\}\\mathbf\{W\}\\right\)\\mathbf\{h\}=a​𝐡\+b​𝐯\\displaystyle=a\\mathbf\{h\}\+b\\mathbf\{v\}=4−ρ24\+ρ2​𝐡\+44\+ρ2​𝐯\.\\displaystyle=\\frac\{4\-\\rho^\{2\}\}\{4\+\\rho^\{2\}\}\\mathbf\{h\}\+\\frac\{4\}\{4\+\\rho^\{2\}\}\\mathbf\{v\}\.This proves the claimed closed\-form expression\.

### D\.2Proof of Proposition[5](https://arxiv.org/html/2608.02668#Thmtheorem5)

Forp=1p=1, we haveθl=arctan⁡\(βl\)\\theta\_\{l\}=\\arctan\(\\beta\_\{l\}\), and hence

cos⁡\(θl\)=11\+βl2,sin⁡\(θl\)=βl1\+βl2\.\\cos\\\!\\left\(\\theta\_\{l\}\\right\)=\\frac\{1\}\{\\sqrt\{1\+\\beta\_\{l\}^\{2\}\}\},\\qquad\\sin\\\!\\left\(\\theta\_\{l\}\\right\)=\\frac\{\\beta\_\{l\}\}\{\\sqrt\{1\+\\beta\_\{l\}^\{2\}\}\}\.Sinceβl=αl​sl/rl\\beta\_\{l\}=\\alpha\_\{l\}s\_\{l\}/r\_\{l\}and𝐡l⊤​𝐳l=0\\mathbf\{h\}\_\{l\}^\{\\top\}\\mathbf\{z\}\_\{l\}=0, it follows that

𝐡l\+1\(1\)=𝐡l\+αl​𝐳l1\+βl2=rl​𝐡l\+αl​𝐳l‖𝐡l\+αl​𝐳l‖2\.\\mathbf\{h\}\_\{l\+1\}^\{\(1\)\}=\\frac\{\\mathbf\{h\}\_\{l\}\+\\alpha\_\{l\}\\mathbf\{z\}\_\{l\}\}\{\\sqrt\{1\+\\beta\_\{l\}^\{2\}\}\}=r\_\{l\}\\frac\{\\mathbf\{h\}\_\{l\}\+\\alpha\_\{l\}\\mathbf\{z\}\_\{l\}\}\{\\\|\\mathbf\{h\}\_\{l\}\+\\alpha\_\{l\}\\mathbf\{z\}\_\{l\}\\\|\_\{2\}\}\.This is precisely the Proj\-SpheretNorm update\.

Forp=2p=2, the double\-angle identities give

cos⁡\(2​arctan⁡βl2\)=4−βl24\+βl2,sin⁡\(2​arctan⁡βl2\)=4​βl4\+βl2\.\\cos\\\!\\left\(2\\arctan\\frac\{\\beta\_\{l\}\}\{2\}\\right\)=\\frac\{4\-\\beta\_\{l\}^\{2\}\}\{4\+\\beta\_\{l\}^\{2\}\},\\,\\sin\\\!\\left\(2\\arctan\\frac\{\\beta\_\{l\}\}\{2\}\\right\)=\\frac\{4\\beta\_\{l\}\}\{4\+\\beta\_\{l\}^\{2\}\}\.Substitutingβl=αl​sl/rl\\beta\_\{l\}=\\alpha\_\{l\}s\_\{l\}/r\_\{l\}into Definition[4](https://arxiv.org/html/2608.02668#Thmtheorem4)yields

𝐡l\+1\(2\)=4−βl24\+βl2​𝐡l\+4​αl4\+βl2​𝐳l,\\mathbf\{h\}\_\{l\+1\}^\{\(2\)\}=\\frac\{4\-\\beta\_\{l\}^\{2\}\}\{4\+\\beta\_\{l\}^\{2\}\}\\mathbf\{h\}\_\{l\}\+\\frac\{4\\alpha\_\{l\}\}\{4\+\\beta\_\{l\}^\{2\}\}\\mathbf\{z\}\_\{l\},which is the Cay\-SpheretNorm update\.

For every fixedβl\\beta\_\{l\},

limp→\+∞p​arctan⁡\(βlp\)=βl\.\\lim\_\{p\\to\+\\infty\}p\\arctan\\left\(\\frac\{\\beta\_\{l\}\}\{p\}\\right\)=\\beta\_\{l\}\.The continuity of the sine and cosine functions therefore gives the exponential\-map limit\.

Finally,

\|p​arctan⁡\(βlp\)\|≤π​p2→0as​p→0\+\.\\left\|p\\arctan\\left\(\\frac\{\\beta\_\{l\}\}\{p\}\\right\)\\right\|\\leq\\frac\{\\pi p\}\{2\}\\to 0\\qquad\\text\{as \}p\\to 0^\{\+\}\.Thuscos⁡\(θl\)→1\\cos\(\\theta\_\{l\}\)\\to 1andsin⁡\(θl\)→0\\sin\(\\theta\_\{l\}\)\\to 0, proving convergence to the identity mapping\.

### D\.3Proof of Proposition[7](https://arxiv.org/html/2608.02668#Thmtheorem7)

Consistent with theLLretained hidden states𝐡1,…,𝐡L\\mathbf\{h\}\_\{1\},\\ldots,\\mathbf\{h\}\_\{L\}, we index the individual SpheretNorm updates byl=1,…,L−1l=1,\\ldots,L\-1\. Let

𝐡l\+1=SpheretNorm⁡\(𝐡l,ℱl​\(𝐡l\)\),\\mathbf\{h\}\_\{l\+1\}=\\operatorname\{SpheretNorm\}\\bigl\(\\mathbf\{h\}\_\{l\},\\mathcal\{F\}\_\{l\}\(\\mathbf\{h\}\_\{l\}\)\\bigr\),where the step\-size and retraction parameters are suppressed from the notation\. Definerl=‖𝐡l‖2r\_\{l\}=\\\|\\mathbf\{h\}\_\{l\}\\\|\_\{2\},𝐏l=𝐈−𝐡l​𝐡l⊤rl2\\mathbf\{P\}\_\{l\}=\\mathbf\{I\}\-\\frac\{\\mathbf\{h\}\_\{l\}\\mathbf\{h\}\_\{l\}^\{\\top\}\}\{r\_\{l\}^\{2\}\}, and𝐳l=𝐏l​ℱl​\(𝐡l\)\\mathbf\{z\}\_\{l\}=\\mathbf\{P\}\_\{l\}\\mathcal\{F\}\_\{l\}\(\\mathbf\{h\}\_\{l\}\)\. Then𝐡l⊤​𝐳l=0\\mathbf\{h\}\_\{l\}^\{\\top\}\\mathbf\{z\}\_\{l\}=0\. Suppose first that𝐳l≠0\\mathbf\{z\}\_\{l\}\\neq 0\. For an appropriate intrinsic angleθl\\theta\_\{l\}, every SpheretNorm variant considered in this work admits the representation

𝐡l\+1=cos⁡\(θl\)​𝐡l\+rl​sin⁡\(θl\)​𝐳l‖𝐳l‖2\.\\mathbf\{h\}\_\{l\+1\}=\\cos\(\\theta\_\{l\}\)\\mathbf\{h\}\_\{l\}\+r\_\{l\}\\sin\(\\theta\_\{l\}\)\\frac\{\\mathbf\{z\}\_\{l\}\}\{\\\|\\mathbf\{z\}\_\{l\}\\\|\_\{2\}\}\.Since𝐡l⊤​𝐳l=0\\mathbf\{h\}\_\{l\}^\{\\top\}\\mathbf\{z\}\_\{l\}=0, the two terms on the right\-hand side are orthogonal, and therefore

‖𝐡l\+1‖22\\displaystyle\\\|\\mathbf\{h\}\_\{l\+1\}\\\|\_\{2\}^\{2\}=cos2⁡\(θl\)​‖𝐡l‖22\+rl2​sin2⁡\(θl\)​‖𝐳l‖𝐳l‖2‖22=rl2\.\\displaystyle=\\cos^\{2\}\(\\theta\_\{l\}\)\\\|\\mathbf\{h\}\_\{l\}\\\|\_\{2\}^\{2\}\+r\_\{l\}^\{2\}\\sin^\{2\}\(\\theta\_\{l\}\)\\left\\\|\\frac\{\\mathbf\{z\}\_\{l\}\}\{\\\|\\mathbf\{z\}\_\{l\}\\\|\_\{2\}\}\\right\\\|\_\{2\}^\{2\}=r\_\{l\}^\{2\}\.Hence,‖𝐡l\+1‖2=‖𝐡l‖2\\\|\\mathbf\{h\}\_\{l\+1\}\\\|\_\{2\}=\\\|\\mathbf\{h\}\_\{l\}\\\|\_\{2\}\. When𝐳l=0\\mathbf\{z\}\_\{l\}=0, the continuously extended update satisfies𝐡l\+1=𝐡l\\mathbf\{h\}\_\{l\+1\}=\\mathbf\{h\}\_\{l\}, and the same identity holds\. Therefore, by induction,

‖𝐡l‖2=‖𝐡1‖2,l=1,…,L\.\\\|\\mathbf\{h\}\_\{l\}\\\|\_\{2\}=\\\|\\mathbf\{h\}\_\{1\}\\\|\_\{2\},\\qquad l=1,\\ldots,L\.
Since𝐡1\\mathbf\{h\}\_\{1\}is the output of the modified ScaleNorm operation,

‖𝐡1‖2=Softplus⁡\(γ\)\.\\\|\\mathbf\{h\}\_\{1\}\\\|\_\{2\}=\\operatorname\{Softplus\}\(\\gamma\)\.Consequently,

‖𝐡l‖2=Softplus⁡\(γ\),l=1,…,L\.\\\|\\mathbf\{h\}\_\{l\}\\\|\_\{2\}=\\operatorname\{Softplus\}\(\\gamma\),\\qquad l=1,\\ldots,L\.In particular,

‖𝐡L‖2=Softplus⁡\(γ\),\\\|\\mathbf\{h\}\_\{L\}\\\|\_\{2\}=\\operatorname\{Softplus\}\(\\gamma\),which is stronger than, and therefore implies, the stated upper bound

‖𝐡L‖2≤Softplus⁡\(γ\)\.\\\|\\mathbf\{h\}\_\{L\}\\\|\_\{2\}\\leq\\operatorname\{Softplus\}\(\\gamma\)\.

### D\.4Second\-Order Retraction and Local Consistency

We prove the smoothness, second\-order retraction, and local\-consistency properties used in Section 1\.3\. Fixp\>0p\>0, let𝐡∈𝕊rd−1\\mathbf\{h\}\\in\\mathbb\{S\}\_\{r\}^\{d\-1\},𝐯∈T𝐡​𝕊rd−1\\mathbf\{v\}\\in T\_\{\\mathbf\{h\}\}\\mathbb\{S\}\_\{r\}^\{d\-1\}, and defineρ=‖𝐯‖2r\\rho=\\frac\{\\\|\\mathbf\{v\}\\\|\_\{2\}\}\{r\},θp​\(ρ\)=p​arctan⁡\(ρp\)\\theta\_\{p\}\(\\rho\)=p\\arctan\\left\(\\frac\{\\rho\}\{p\}\\right\)\.

Forρ\>0\\rho\>0, define

ap​\(ρ\)=cos⁡\(θp​\(ρ\)\),bp​\(ρ\)=sin⁡\(θp​\(ρ\)\)ρ,a\_\{p\}\(\\rho\)=\\cos\\bigl\(\\theta\_\{p\}\(\\rho\)\\bigr\),\\qquad b\_\{p\}\(\\rho\)=\\frac\{\\sin\\bigl\(\\theta\_\{p\}\(\\rho\)\\bigr\)\}\{\\rho\},and setbp​\(0\)=1b\_\{p\}\(0\)=1\. Thepp\-angular map can then be written as

ℛ𝐡\(p\)​\(𝐯\)=ap​\(ρ\)​𝐡\+bp​\(ρ\)​𝐯\.\\mathcal\{R\}\_\{\\mathbf\{h\}\}^\{\(p\)\}\(\\mathbf\{v\}\)=a\_\{p\}\(\\rho\)\\mathbf\{h\}\+b\_\{p\}\(\\rho\)\\mathbf\{v\}\.
##### Smoothness at the zero section\.

The apparent singularity inbp​\(ρ\)b\_\{p\}\(\\rho\)atρ=0\\rho=0is removable\. To see this, consider the real\-analytic odd function

θ~p​\(t\)=p​arctan⁡\(tp\),t∈ℝ\.\\widetilde\{\\theta\}\_\{p\}\(t\)=p\\arctan\\left\(\\frac\{t\}\{p\}\\right\),\\qquad t\\in\\mathbb\{R\}\.DefineAp​\(t\)=cos⁡\(θ~p​\(t\)\)A\_\{p\}\(t\)=\\cos\\bigl\(\\widetilde\{\\theta\}\_\{p\}\(t\)\\bigr\)andBp​\(t\)=\{sin⁡\(θ~p​\(t\)\)t,t≠0,1,t=0\.B\_\{p\}\(t\)=\\begin\{cases\}\\displaystyle\\frac\{\\sin\\bigl\(\\widetilde\{\\theta\}\_\{p\}\(t\)\\bigr\)\}\{t\},&t\\neq 0,\\\\\[5\.16663pt\] 1,&t=0\.\\end\{cases\}\. Sinceθ~p\\widetilde\{\\theta\}\_\{p\}is odd, bothApA\_\{p\}andBpB\_\{p\}are even real\-analytic functions in a neighborhood of the origin\. Hence, there exist real\-analytic functionsA^p\\widehat\{A\}\_\{p\}andB^p\\widehat\{B\}\_\{p\}, defined near zero, such that

Ap​\(t\)=A^p​\(t2\),Bp​\(t\)=B^p​\(t2\)\.A\_\{p\}\(t\)=\\widehat\{A\}\_\{p\}\(t^\{2\}\),\\qquad B\_\{p\}\(t\)=\\widehat\{B\}\_\{p\}\(t^\{2\}\)\.Consequently,

ℛ𝐡\(p\)​\(𝐯\)=A^p​\(‖𝐯‖22r2\)​𝐡\+B^p​\(‖𝐯‖22r2\)​𝐯\.\\mathcal\{R\}\_\{\\mathbf\{h\}\}^\{\(p\)\}\(\\mathbf\{v\}\)=\\widehat\{A\}\_\{p\}\\left\(\\frac\{\\\|\\mathbf\{v\}\\\|\_\{2\}^\{2\}\}\{r^\{2\}\}\\right\)\\mathbf\{h\}\+\\widehat\{B\}\_\{p\}\\left\(\\frac\{\\\|\\mathbf\{v\}\\\|\_\{2\}^\{2\}\}\{r^\{2\}\}\\right\)\\mathbf\{v\}\.This representation is smooth at𝐯=𝟎\\mathbf\{v\}=\\mathbf\{0\}\. Sincer\>0r\>0is fixed on𝕊rd−1\\mathbb\{S\}\_\{r\}^\{d\-1\}, it also shows that\(𝐡,𝐯\)→ℛ𝐡\(p\)​\(𝐯\)\(\\mathbf\{h\},\\mathbf\{v\}\)\\to\\mathcal\{R\}\_\{\\mathbf\{h\}\}^\{\(p\)\}\(\\mathbf\{v\}\)is smooth on the tangent bundleT​𝕊rd−1T\\mathbb\{S\}\_\{r\}^\{d\-1\}near the zero section\. Away from𝐯=𝟎\\mathbf\{v\}=\\mathbf\{0\}, smoothness follows directly from the defining formula\.

##### Sphere\-valued property\.

Since𝐡⊤​𝐯=0\\mathbf\{h\}^\{\\top\}\\mathbf\{v\}=0,‖𝐡‖2=r\\\|\\mathbf\{h\}\\\|\_\{2\}=r, and‖𝐯‖2=r​ρ\\\|\\mathbf\{v\}\\\|\_\{2\}=r\\rho, we obtain, for𝐯≠𝟎\\mathbf\{v\}\\neq\\bm\{0\},

‖ℛ𝐡\(p\)​\(𝐯\)‖22\\displaystyle\\left\\\|\\mathcal\{R\}\_\{\\mathbf\{h\}\}^\{\(p\)\}\(\\mathbf\{v\}\)\\right\\\|\_\{2\}^\{2\}=ap​\(ρ\)2​‖𝐡‖22\+bp​\(ρ\)2​‖𝐯‖22\\displaystyle=a\_\{p\}\(\\rho\)^\{2\}\\\|\\mathbf\{h\}\\\|\_\{2\}^\{2\}\+b\_\{p\}\(\\rho\)^\{2\}\\\|\\mathbf\{v\}\\\|\_\{2\}^\{2\}=r2​cos2⁡\(θp​\(ρ\)\)\+r2​sin2⁡\(θp​\(ρ\)\)=r2\.\\displaystyle=r^\{2\}\\cos^\{2\}\\bigl\(\\theta\_\{p\}\(\\rho\)\\bigr\)\+r^\{2\}\\sin^\{2\}\\bigl\(\\theta\_\{p\}\(\\rho\)\\bigr\)=r^\{2\}\.The same conclusion is immediate at𝐯=𝟎\\mathbf\{v\}=\\mathbf\{0\}\. Therefore,ℛ𝐡\(p\)​\(𝐯\)∈𝕊rd−1\\mathcal\{R\}\_\{\\mathbf\{h\}\}^\{\(p\)\}\(\\mathbf\{v\}\)\\in\\mathbb\{S\}\_\{r\}^\{d\-1\}\.

##### Retraction and second\-order properties\.

Asρ→0\\rho\\to 0,

θp​\(ρ\)=ρ−ρ33​p2\+𝒪​\(ρ5\)\.\\theta\_\{p\}\(\\rho\)=\\rho\-\\frac\{\\rho^\{3\}\}\{3p^\{2\}\}\+\\mathcal\{O\}\(\\rho^\{5\}\)\.It follows that

ap​\(ρ\)=1−ρ22\+\(124\+13​p2\)​ρ4\+𝒪​\(ρ6\),a\_\{p\}\(\\rho\)=1\-\\frac\{\\rho^\{2\}\}\{2\}\+\\left\(\\frac\{1\}\{24\}\+\\frac\{1\}\{3p^\{2\}\}\\right\)\\rho^\{4\}\+\\mathcal\{O\}\(\\rho^\{6\}\),and

bp​\(ρ\)=1−\(16\+13​p2\)​ρ2\+𝒪​\(ρ4\)\.b\_\{p\}\(\\rho\)=1\-\\left\(\\frac\{1\}\{6\}\+\\frac\{1\}\{3p^\{2\}\}\\right\)\\rho^\{2\}\+\\mathcal\{O\}\(\\rho^\{4\}\)\.Equivalently, ass→0s\\to 0, it follows that

A^p​\(s\)=1−s2\+\(124\+13​p2\)​s2\+𝒪​\(s3\),\\widehat\{A\}\_\{p\}\(s\)=1\-\\frac\{s\}\{2\}\+\\left\(\\frac\{1\}\{24\}\+\\frac\{1\}\{3p^\{2\}\}\\right\)s^\{2\}\+\\mathcal\{O\}\(s^\{3\}\),and

B^p​\(s\)=1−\(16\+13​p2\)​s\+𝒪​\(s2\)\.\\widehat\{B\}\_\{p\}\(s\)=1\-\\left\(\\frac\{1\}\{6\}\+\\frac\{1\}\{3p^\{2\}\}\\right\)s\+\\mathcal\{O\}\(s^\{2\}\)\.
Fix𝐯∈T𝐡​𝕊rd−1\\mathbf\{v\}\\in T\_\{\\mathbf\{h\}\}\\mathbb\{S\}\_\{r\}^\{d\-1\}, setρ0=‖𝐯‖2r\\rho\_\{0\}=\\frac\{\\\|\\mathbf\{v\}\\\|\_\{2\}\}\{r\}, and consider the curve𝐜​\(t\)=ℛ𝐡\(p\)​\(t​𝐯\)\\mathbf\{c\}\(t\)=\\mathcal\{R\}\_\{\\mathbf\{h\}\}^\{\(p\)\}\(t\\mathbf\{v\}\)\. Using the even\-function representation above, we have

𝐜​\(t\)=A^p​\(t2​ρ02\)​𝐡\+t​B^p​\(t2​ρ02\)​𝐯\.\\mathbf\{c\}\(t\)=\\widehat\{A\}\_\{p\}\(t^\{2\}\\rho\_\{0\}^\{2\}\)\\mathbf\{h\}\+t\\widehat\{B\}\_\{p\}\(t^\{2\}\\rho\_\{0\}^\{2\}\)\\mathbf\{v\}\.Therefore,

𝐜​\(t\)=𝐡\+t​𝐯−t2​‖𝐯‖222​r2​𝐡\+𝒪​\(t3\)\.\\mathbf\{c\}\(t\)=\\mathbf\{h\}\+t\\mathbf\{v\}\-\\frac\{t^\{2\}\\\|\\mathbf\{v\}\\\|\_\{2\}^\{2\}\}\{2r^\{2\}\}\\mathbf\{h\}\+\\mathcal\{O\}\(t^\{3\}\)\.Hence,𝐜​\(0\)=𝐡\\mathbf\{c\}\(0\)=\\mathbf\{h\},𝐜′​\(0\)=𝐯\\mathbf\{c\}^\{\\prime\}\(0\)=\\mathbf\{v\}and𝐜′′​\(0\)=−‖𝐯‖22r2​𝐡\\mathbf\{c\}^\{\\prime\\prime\}\(0\)=\-\\frac\{\\\|\\mathbf\{v\}\\\|\_\{2\}^\{2\}\}\{r^\{2\}\}\\mathbf\{h\}\. The first two identities verify the defining conditions of a retraction\. Moreover, since

N𝐡​𝕊rd−1=span⁡\{𝐡\},N\_\{\\mathbf\{h\}\}\\mathbb\{S\}\_\{r\}^\{d\-1\}=\\operatorname\{span\}\\\{\\mathbf\{h\}\\\},the acceleration𝐜′′​\(0\)\\mathbf\{c\}^\{\\prime\\prime\}\(0\)is normal to the hypersphere\. Equivalently,

𝐏𝐡​𝐜′′​\(0\)=𝟎,𝐏𝐡=𝐈−𝐡𝐡⊤r2\.\\mathbf\{P\}\_\{\\mathbf\{h\}\}\\mathbf\{c\}^\{\\prime\\prime\}\(0\)=\\bm\{0\},\\qquad\\mathbf\{P\}\_\{\\mathbf\{h\}\}=\\mathbf\{I\}\-\\frac\{\\mathbf\{h\}\\mathbf\{h\}^\{\\top\}\}\{r^\{2\}\}\.Hence, the retraction curve has no tangential acceleration at the origin, which is precisely the second\-order retraction condition\. Therefore,ℛ\(p\)\\mathcal\{R\}^\{\(p\)\}is a second\-order retraction on𝕊rd−1\\mathbb\{S\}\_\{r\}^\{d\-1\}for every fixedp\>0p\>0\.

##### Local expansion of the SpheretNorm update\.

For thell\-th SpheretNorm update, letrl=‖𝐡l‖2r\_\{l\}=\\\|\\mathbf\{h\}\_\{l\}\\\|\_\{2\},𝐏l=𝐈−𝐡l​𝐡l⊤rl2\\mathbf\{P\}\_\{l\}=\\mathbf\{I\}\-\\frac\{\\mathbf\{h\}\_\{l\}\\mathbf\{h\}\_\{l\}^\{\\top\}\}\{r\_\{l\}^\{2\}\},𝐳l=𝐏l​ℱl​\(𝐡l\)\\mathbf\{z\}\_\{l\}=\\mathbf\{P\}\_\{l\}\\mathcal\{F\}\_\{l\}\(\\mathbf\{h\}\_\{l\}\), andsl=‖𝐳l‖2s\_\{l\}=\\\|\\mathbf\{z\}\_\{l\}\\\|\_\{2\}\. Applying the preceding expansion to𝐯=αl​𝐳l\\mathbf\{v\}=\\alpha\_\{l\}\\mathbf\{z\}\_\{l\}gives

𝐡l\+1=\\displaystyle\\mathbf\{h\}\_\{l\+1\}=\{\}𝐡l\+αl​𝐳l−αl2​‖𝐳l‖222​rl2​𝐡l\\displaystyle\\mathbf\{h\}\_\{l\}\+\\alpha\_\{l\}\\mathbf\{z\}\_\{l\}\-\\frac\{\\alpha\_\{l\}^\{2\}\\\|\\mathbf\{z\}\_\{l\}\\\|\_\{2\}^\{2\}\}\{2r\_\{l\}^\{2\}\}\\mathbf\{h\}\_\{l\}−\(16\+13​p2\)​αl3​‖𝐳l‖22rl2​𝐳l\+𝒪​\(αl4\)\.\\displaystyle\-\\left\(\\frac\{1\}\{6\}\+\\frac\{1\}\{3p^\{2\}\}\\right\)\\frac\{\\alpha\_\{l\}^\{3\}\\\|\\mathbf\{z\}\_\{l\}\\\|\_\{2\}^\{2\}\}\{r\_\{l\}^\{2\}\}\\mathbf\{z\}\_\{l\}\+\\mathcal\{O\}\(\\alpha\_\{l\}^\{4\}\)\.Here, the expansion is taken asαl→0\\alpha\_\{l\}\\to 0, with𝐡l\\mathbf\{h\}\_\{l\}and𝐳l\\mathbf\{z\}\_\{l\}fixed\. More generally, the remainder is uniform over any local regime in whichrlr\_\{l\}is bounded away from zero andηl=‖𝐳l‖2rl\\eta\_\{l\}=\\frac\{\\\|\\mathbf\{z\}\_\{l\}\\\|\_\{2\}\}\{r\_\{l\}\}remains bounded\. The linear termαl​𝐳l\\alpha\_\{l\}\\mathbf\{z\}\_\{l\}is tangent to the hypersphere, while the quadratic term−αl2​‖𝐳l‖222​rl2​𝐡l\-\\frac\{\\alpha\_\{l\}^\{2\}\\\|\\mathbf\{z\}\_\{l\}\\\|\_\{2\}^\{2\}\}\{2r\_\{l\}^\{2\}\}\\mathbf\{h\}\_\{l\}is normal to the hypersphere at𝐡l\\mathbf\{h\}\_\{l\}\. Moreover, the parameterppfirst appears in the cubic tangent correction

−\(16\+13​p2\)​αl3​‖𝐳l‖22rl2​𝐳l\.\-\\left\(\\frac\{1\}\{6\}\+\\frac\{1\}\{3p^\{2\}\}\\right\)\\frac\{\\alpha\_\{l\}^\{3\}\\\|\\mathbf\{z\}\_\{l\}\\\|\_\{2\}^\{2\}\}\{r\_\{l\}^\{2\}\}\\mathbf\{z\}\_\{l\}\.Thus, all members of thepp\-angular family have the same first\-order tangent dynamics and the same second\-order normal correction, while their finite\-step dependence onppfirst appears at cubic order\. Finally, since𝐳l=𝐏l​ℱl​\(𝐡l\)\\mathbf\{z\}\_\{l\}=\\mathbf\{P\}\_\{l\}\\mathcal\{F\}\_\{l\}\(\\mathbf\{h\}\_\{l\}\), we obtain the first\-order consistency relation

𝐡l\+1=𝐡l\+αl​𝐏l​ℱl​\(𝐡l\)\+𝒪​\(αl2\)\.\\mathbf\{h\}\_\{l\+1\}=\\mathbf\{h\}\_\{l\}\+\\alpha\_\{l\}\\mathbf\{P\}\_\{l\}\\mathcal\{F\}\_\{l\}\(\\mathbf\{h\}\_\{l\}\)\+\\mathcal\{O\}\(\\alpha\_\{l\}^\{2\}\)\.

### D\.5Depth\-Wise Conditioning Criterion

We first formulate the conditioning criterion for a generic ordered sequence of individual SpheretNorm updates\. LetMMdenote the number of such updates, and let

𝐉j=∂𝐱j\+1∂𝐱j,j=1,…,M\.\\mathbf\{J\}\_\{j\}=\\frac\{\\partial\\mathbf\{x\}\_\{j\+1\}\}\{\\partial\\mathbf\{x\}\_\{j\}\},\\qquad j=1,\\ldots,M\.For eachjj, defineqj=\{\|cos⁡\(θj\)\|,sj\>0,1,sj=0,q\_\{j\}=\\begin\{cases\}\|\\cos\(\\theta\_\{j\}\)\|,&s\_\{j\}\>0,\\\\ 1,&s\_\{j\}=0,\\end\{cases\}andmj=max⁡\{0,qj−Δj\}m\_\{j\}=\\max\\\{0,q\_\{j\}\-\\Delta\_\{j\}\\\}\. Assume that there exists a constantm∗\>0m\_\{\*\}\>0, independent ofjjandMM, such thatmj≥m∗m\_\{j\}\\geq m\_\{\*\},j=1,…,Mj=1,\\ldots,M\. Sincemj≤1m\_\{j\}\\leq 1, we have

−log⁡mj=∫mj1d​tt≤1−mjm∗\.\-\\log m\_\{j\}=\\int\_\{m\_\{j\}\}^\{1\}\\frac\{dt\}\{t\}\\leq\\frac\{1\-m\_\{j\}\}\{m\_\{\*\}\}\.Becausemj\>0m\_\{j\}\>0, the maximum in its definition is inactive, and hencemj=qj−Δjm\_\{j\}=q\_\{j\}\-\\Delta\_\{j\}\. Therefore,

1−mj=1−qj\+Δj\.1\-m\_\{j\}=1\-q\_\{j\}\+\\Delta\_\{j\}\.For every realθ\\theta,1−\|cos⁡θ\|≤1−cos⁡θ≤θ221\-\|\\cos\\theta\|\\leq 1\-\\cos\\theta\\leq\\frac\{\\theta^\{2\}\}\{2\}\. Consequently,1−mj≤\|θj\|22\+Δj1\-m\_\{j\}\\leq\\frac\{\|\\theta\_\{j\}\|^\{2\}\}\{2\}\+\\Delta\_\{j\}\. It follows that

−log⁡\(∏j=1Mmj\)\\displaystyle\-\\log\\left\(\\prod\_\{j=1\}^\{M\}m\_\{j\}\\right\)=−∑j=1Mlog⁡mj\\displaystyle=\-\\sum\_\{j=1\}^\{M\}\\log m\_\{j\}≤1m∗​\[12​∑j=1M\|θj\|2\+∑j=1MΔj\]\.\\displaystyle\\leq\\frac\{1\}\{m\_\{\*\}\}\\left\[\\frac\{1\}\{2\}\\sum\_\{j=1\}^\{M\}\|\\theta\_\{j\}\|^\{2\}\+\\sum\_\{j=1\}^\{M\}\\Delta\_\{j\}\\right\]\.Hence,

∏j=1Mmj≥exp⁡\{−1m∗​\[12​∑j=1M\|θj\|2\+∑j=1MΔj\]\}\.\\prod\_\{j=1\}^\{M\}m\_\{j\}\\geq\\exp\\left\\\{\-\\frac\{1\}\{m\_\{\*\}\}\\left\[\\frac\{1\}\{2\}\\sum\_\{j=1\}^\{M\}\|\\theta\_\{j\}\|^\{2\}\+\\sum\_\{j=1\}^\{M\}\\Delta\_\{j\}\\right\]\\right\\\}\.
Let𝐆M=𝐉M​𝐉M−1​⋯​𝐉1\\mathbf\{G\}\_\{M\}=\\mathbf\{J\}\_\{M\}\\mathbf\{J\}\_\{M\-1\}\\cdots\\mathbf\{J\}\_\{1\}\. Combining the preceding estimate with the layerwise bounds of Proposition[6](https://arxiv.org/html/2608.02668#Thmtheorem6)gives

σmax​\(𝐆M\)≤exp⁡\(∑j=1MΔj\)\\sigma\_\{\\max\}\(\\mathbf\{G\}\_\{M\}\)\\leq\\exp\\left\(\\sum\_\{j=1\}^\{M\}\\Delta\_\{j\}\\right\)and

σmin​\(𝐆M\)≥exp⁡\{−1m∗​\[12​∑j=1M\|θj\|2\+∑j=1MΔj\]\}\.\\sigma\_\{\\min\}\(\\mathbf\{G\}\_\{M\}\)\\geq\\exp\\left\\\{\-\\frac\{1\}\{m\_\{\*\}\}\\left\[\\frac\{1\}\{2\}\\sum\_\{j=1\}^\{M\}\|\\theta\_\{j\}\|^\{2\}\+\\sum\_\{j=1\}^\{M\}\\Delta\_\{j\}\\right\]\\right\\\}\.Therefore, if∑j=1MΔj=𝒪​\(1\)\\sum\_\{j=1\}^\{M\}\\Delta\_\{j\}=\\mathcal\{O\}\(1\)and∑j=1M\|θj\|2=𝒪​\(1\)\\sum\_\{j=1\}^\{M\}\|\\theta\_\{j\}\|^\{2\}=\\mathcal\{O\}\(1\), then bothσmax​\(𝐆M\)\\sigma\_\{\\max\}\(\\mathbf\{G\}\_\{M\}\)andσmin​\(𝐆M\)−1\\sigma\_\{\\min\}\(\\mathbf\{G\}\_\{M\}\)^\{\-1\}remain bounded independently of the numberMMof individual SpheretNorm updates\.

For the residual stream𝐡1,𝐡2,…,𝐡L\\mathbf\{h\}\_\{1\},\\mathbf\{h\}\_\{2\},\\ldots,\\mathbf\{h\}\_\{L\}considered in Appendix[A](https://arxiv.org/html/2608.02668#A1)and Appendix[D\.3](https://arxiv.org/html/2608.02668#A4.SS3), there areM=L−1M=L\-1individual SpheretNorm updates\. More precisely, setting

𝐱j=𝐡j,𝐉j=∂𝐡j\+1∂𝐡j,j=1,…,L−1,\\mathbf\{x\}\_\{j\}=\\mathbf\{h\}\_\{j\},\\qquad\\mathbf\{J\}\_\{j\}=\\frac\{\\partial\\mathbf\{h\}\_\{j\+1\}\}\{\\partial\\mathbf\{h\}\_\{j\}\},\\qquad j=1,\\ldots,L\-1,gives

𝐆M=𝐆L−1=∂𝐡L∂𝐡1=𝐉L−1​𝐉L−2​⋯​𝐉1\.\\mathbf\{G\}\_\{M\}=\\mathbf\{G\}\_\{L\-1\}=\\frac\{\\partial\\mathbf\{h\}\_\{L\}\}\{\\partial\\mathbf\{h\}\_\{1\}\}=\\mathbf\{J\}\_\{L\-1\}\\mathbf\{J\}\_\{L\-2\}\\cdots\\mathbf\{J\}\_\{1\}\.Therefore, the preceding depth\-wise conditioning criterion applies to the actual residual stream with

∑j=1L−1Δj=𝒪​\(1\),∑j=1L−1\|θj\|2=𝒪​\(1\)\.\\sum\_\{j=1\}^\{L\-1\}\\Delta\_\{j\}=\\mathcal\{O\}\(1\),\\qquad\\sum\_\{j=1\}^\{L\-1\}\|\\theta\_\{j\}\|^\{2\}=\\mathcal\{O\}\(1\)\.Proposition[6](https://arxiv.org/html/2608.02668#Thmtheorem6)usesLLas the generic number of individual SpheretNorm updates\. When the result is applied to the retained\-state indexing𝐡1,…,𝐡L\\mathbf\{h\}\_\{1\},\\ldots,\\mathbf\{h\}\_\{L\}, used in Appendix[A](https://arxiv.org/html/2608.02668#A1)and Appendix[D\.3](https://arxiv.org/html/2608.02668#A4.SS3), the corresponding update count isM=L−1M=L\-1\.

### D\.6Proof of the Depth\-Schedule Growth Proposition

The three schedules compared in Proposition[8](https://arxiv.org/html/2608.02668#Thmtheorem8)areλl=1\\lambda\_\{l\}=1,λl=l−1/2\\lambda\_\{l\}=l^\{\-1/2\}, andλl=l−1\\lambda\_\{l\}=l^\{\-1\}\. In particular, the third schedule is the harmonic scheduleλl=l−1\\lambda\_\{l\}=l^\{\-1\}, for which

∑l=1Lλl=𝒪​\(log⁡L\),∑l=1Lλl2=𝒪​\(1\)\.\\sum\_\{l=1\}^\{L\}\\lambda\_\{l\}=\\mathcal\{O\}\(\\log L\),\\qquad\\sum\_\{l=1\}^\{L\}\\lambda\_\{l\}^\{2\}=\\mathcal\{O\}\(1\)\.By definition,SL=∑l=1Lλl​al​κlS\_\{L\}=\\sum\_\{l=1\}^\{L\}\\lambda\_\{l\}a\_\{l\}\\kappa\_\{l\}\. Since0<al≤10<a\_\{l\}\\leq 1andκl≤K\\kappa\_\{l\}\\leq K, it follows thatSL≤K​∑l=1LλlS\_\{L\}\\leq K\\sum\_\{l=1\}^\{L\}\\lambda\_\{l\}\. Furthermore, since

\|p​arctan⁡\(βp\)\|≤\|β\|,β∈ℝ,\\left\|p\\arctan\\left\(\\frac\{\\beta\}\{p\}\\right\)\\right\|\\leq\|\\beta\|,\\qquad\\beta\\in\\mathbb\{R\},we have\|θl\|≤\|βl\|=λl​al​ηl\\left\|\\theta\_\{l\}\\right\|\\leq\|\\beta\_\{l\}\|=\\lambda\_\{l\}a\_\{l\}\\eta\_\{l\}\. Usingal​ηl≤Ha\_\{l\}\\eta\_\{l\}\\leq H, we obtain

AL=∑l=1L\|θl\|2≤∑l=1Lλl2​\(al​ηl\)2≤H2​∑l=1Lλl2\.A\_\{L\}=\\sum\_\{l=1\}^\{L\}\\left\|\\theta\_\{l\}\\right\|^\{2\}\\leq\\sum\_\{l=1\}^\{L\}\\lambda\_\{l\}^\{2\}\(a\_\{l\}\\eta\_\{l\}\)^\{2\}\\leq H^\{2\}\\sum\_\{l=1\}^\{L\}\\lambda\_\{l\}^\{2\}\.
For the constant scheduleλl=1\\lambda\_\{l\}=1, we have∑l=1Lλl=L\\sum\_\{l=1\}^\{L\}\\lambda\_\{l\}=Land∑l=1Lλl2=L\\sum\_\{l=1\}^\{L\}\\lambda\_\{l\}^\{2\}=L\. For the layerwise square\-root scheduleλl=l−1/2\\lambda\_\{l\}=l^\{\-1/2\},

∑l=1Lλl=∑l=1L1l≤2​L=𝒪​\(L\),\\sum\_\{l=1\}^\{L\}\\lambda\_\{l\}=\\sum\_\{l=1\}^\{L\}\\frac\{1\}\{\\sqrt\{l\}\}\\leq 2\\sqrt\{L\}=\\mathcal\{O\}\(\\sqrt\{L\}\),while

∑l=1Lλl2=∑l=1L1l≤1\+log⁡L=𝒪​\(log⁡L\)\.\\sum\_\{l=1\}^\{L\}\\lambda\_\{l\}^\{2\}=\\sum\_\{l=1\}^\{L\}\\frac\{1\}\{l\}\\leq 1\+\\log L=\\mathcal\{O\}\(\\log L\)\.
For the harmonic scheduleλl=l−1\\lambda\_\{l\}=l^\{\-1\},

∑l=1Lλl\\displaystyle\\sum\_\{l=1\}^\{L\}\\lambda\_\{l\}=∑l=1L1l=𝒪​\(log⁡L\)​and\\displaystyle=\\sum\_\{l=1\}^\{L\}\\frac\{1\}\{l\}=\\mathcal\{O\}\(\\log L\)\\,\\mbox\{ and\}∑l=1Lλl2\\displaystyle\\sum\_\{l=1\}^\{L\}\\lambda\_\{l\}^\{2\}=∑l=1L1l2≤π26=𝒪​\(1\)\.\\displaystyle=\\sum\_\{l=1\}^\{L\}\\frac\{1\}\{l^\{2\}\}\\leq\\frac\{\\pi^\{2\}\}\{6\}=\\mathcal\{O\}\(1\)\.The stated growth rates follow\.

##### Application to the retained\-state indexing\.

Proposition[8](https://arxiv.org/html/2608.02668#Thmtheorem8)usesLLas the generic number of individual SpheretNorm updates\. For the retained\-state sequence𝐡1,𝐡2,…,𝐡L\\mathbf\{h\}\_\{1\},\\mathbf\{h\}\_\{2\},\\ldots,\\mathbf\{h\}\_\{L\}, the corresponding number of updates isM=L−1M=L\-1\. Forl=1,…,Ml=1,\\ldots,M, letαl=λl​al\\alpha\_\{l\}=\\lambda\_\{l\}a\_\{l\}andΔl=λl​al​κl\\Delta\_\{l\}=\\lambda\_\{l\}a\_\{l\}\\kappa\_\{l\}, whereκl=2​‖𝐁l‖2\+ηl\\kappa\_\{l\}=2\\\|\\mathbf\{B\}\_\{l\}\\\|\_\{2\}\+\\eta\_\{l\}\. Suppose that

κl≤K,al​ηl≤H,l=1,…,M\.\\kappa\_\{l\}\\leq K,\\qquad a\_\{l\}\\eta\_\{l\}\\leq H,\\qquad l=1,\\ldots,M\.Define the accumulated sensitivity envelope and squared angular budget by

SMres=∑l=1MΔl,AMres=∑l=1M\|θl\|2\.S\_\{M\}^\{\\mathrm\{res\}\}=\\sum\_\{l=1\}^\{M\}\\Delta\_\{l\},\\qquad A\_\{M\}^\{\\mathrm\{res\}\}=\\sum\_\{l=1\}^\{M\}\|\\theta\_\{l\}\|^\{2\}\.Then the preceding estimates give

SMres≤K​∑l=1Mλl,AMres≤H2​∑l=1Mλl2\.S\_\{M\}^\{\\mathrm\{res\}\}\\leq K\\sum\_\{l=1\}^\{M\}\\lambda\_\{l\},\\qquad A\_\{M\}^\{\\mathrm\{res\}\}\\leq H^\{2\}\\sum\_\{l=1\}^\{M\}\\lambda\_\{l\}^\{2\}\.SinceM=L−1M=L\-1, these estimates are equivalently

λlSL−1resAL−1res1𝒪​\(L\)𝒪​\(L\)1/l𝒪​\(L\)𝒪​\(log⁡L\)1/l𝒪​\(log⁡L\)𝒪​\(1\)\.\\begin\{array\}\[\]\{c\|cc\}\\lambda\_\{l\}&S\_\{L\-1\}^\{\\mathrm\{res\}\}&A\_\{L\-1\}^\{\\mathrm\{res\}\}\\\\ \\hline\\cr 1&\\mathcal\{O\}\(L\)&\\mathcal\{O\}\(L\)\\\\ \{1\}/\{\\sqrt\{l\}\}&\\mathcal\{O\}\(\\sqrt\{L\}\)&\\mathcal\{O\}\(\\log L\)\\\\ \{1\}/\{l\}&\\mathcal\{O\}\(\\log L\)&\\mathcal\{O\}\(1\)\.\\end\{array\}Thus, specializing the generic update count in Proposition[8](https://arxiv.org/html/2608.02668#Thmtheorem8)changes only the endpoint of the sums and does not alter any of the stated asymptotic growth rates\.

### D\.7Full Jacobian and Layerwise Estimates

##### Indexing convention\.

Consistent with the retained\-state indexing used in Appendix[A](https://arxiv.org/html/2608.02668#A1)and Appendix[D\.3](https://arxiv.org/html/2608.02668#A4.SS3), we consider the sequence𝐡1,𝐡2,…,𝐡L\\mathbf\{h\}\_\{1\},\\mathbf\{h\}\_\{2\},\\ldots,\\mathbf\{h\}\_\{L\}, where, fork=1,…,L−1k=1,\\ldots,L\-1,

𝐡k\+1=SpheretNorm⁡\(𝐡k,ℱk​\(𝐡k\)\)\.\\mathbf\{h\}\_\{k\+1\}=\\operatorname\{SpheretNorm\}\\bigl\(\\mathbf\{h\}\_\{k\},\\mathcal\{F\}\_\{k\}\(\\mathbf\{h\}\_\{k\}\)\\bigr\)\.There are thereforeN=L−1N=L\-1individual SpheretNorm updates between𝐡1\\mathbf\{h\}\_\{1\}and𝐡L\\mathbf\{h\}\_\{L\}\.

For notational convenience in the Jacobian calculation, set𝐱k=𝐡k\\mathbf\{x\}\_\{k\}=\\mathbf\{h\}\_\{k\}fork=1,…,N\+1k=1,\\ldots,N\+1\. Then

𝐱1=𝐡1,𝐱N\+1=𝐡L,\\mathbf\{x\}\_\{1\}=\\mathbf\{h\}\_\{1\},\\qquad\\mathbf\{x\}\_\{N\+1\}=\\mathbf\{h\}\_\{L\},and thekk\-th individual SpheretNorm update is written as

𝐱k⟼𝐱k\+1,k=1,…,N\.\\mathbf\{x\}\_\{k\}\\longmapsto\\mathbf\{x\}\_\{k\+1\},\\qquad k=1,\\ldots,N\.Proposition[6](https://arxiv.org/html/2608.02668#Thmtheorem6)usesLLas a generic count of individual SpheretNorm updates\. To avoid conflict with the retained\-state countLLused here, we denote that generic update count byNN\. For the retained\-state sequence𝐡1,…,𝐡L\\mathbf\{h\}\_\{1\},\\ldots,\\mathbf\{h\}\_\{L\}, one hasN=L−1N=L\-1\.

##### Differential of a single SpheretNorm update\.

For eachk=1,…,Nk=1,\\ldots,N, letℱk\\mathcal\{F\}\_\{k\}denote the mapping associated with thekk\-th SpheretNorm update, and letαk\\alpha\_\{k\}denote its step size\. We assume thatℱk\\mathcal\{F\}\_\{k\}is continuously differentiable in a neighborhood of𝐱k\\mathbf\{x\}\_\{k\}\. Throughout this subsection, all derivatives are taken with respect to𝐱k\\mathbf\{x\}\_\{k\}, whileppandαk\\alpha\_\{k\}are treated as constants\.

Definerk=‖𝐱k‖2r\_\{k\}=\\\|\\mathbf\{x\}\_\{k\}\\\|\_\{2\},𝐏k=𝐈−𝐱k​𝐱k⊤rk2\\mathbf\{P\}\_\{k\}=\\mathbf\{I\}\-\\frac\{\\mathbf\{x\}\_\{k\}\\mathbf\{x\}\_\{k\}^\{\\top\}\}\{r\_\{k\}^\{2\}\},ck=𝐱k⊤​ℱk​\(𝐱k\)rk2c\_\{k\}=\\frac\{\\mathbf\{x\}\_\{k\}^\{\\top\}\\mathcal\{F\}\_\{k\}\(\\mathbf\{x\}\_\{k\}\)\}\{r\_\{k\}^\{2\}\},𝐳k=ℱk​\(𝐱k\)−ck​𝐱k=𝐏k​ℱk​\(𝐱k\)\\mathbf\{z\}\_\{k\}=\\mathcal\{F\}\_\{k\}\(\\mathbf\{x\}\_\{k\}\)\-c\_\{k\}\\mathbf\{x\}\_\{k\}=\\mathbf\{P\}\_\{k\}\\mathcal\{F\}\_\{k\}\(\\mathbf\{x\}\_\{k\}\),sk=‖𝐳k‖2s\_\{k\}=\\\|\\mathbf\{z\}\_\{k\}\\\|\_\{2\}, and

ηk=skrk,βk=αk​ηk,θk=p​arctan⁡\(βkp\)\.\\eta\_\{k\}=\\frac\{s\_\{k\}\}\{r\_\{k\}\},\\quad\\beta\_\{k\}=\\alpha\_\{k\}\\eta\_\{k\},\\quad\\theta\_\{k\}=p\\arctan\\left\(\\frac\{\\beta\_\{k\}\}\{p\}\\right\)\.Let𝐀k=D​ℱk​\(𝐱k\)=∂ℱk​\(𝐱k\)∂𝐱k\\mathbf\{A\}\_\{k\}=D\\mathcal\{F\}\_\{k\}\(\\mathbf\{x\}\_\{k\}\)=\\frac\{\\partial\\mathcal\{F\}\_\{k\}\(\\mathbf\{x\}\_\{k\}\)\}\{\\partial\\mathbf\{x\}\_\{k\}\},𝐁k=𝐀k−ck​𝐈\\mathbf\{B\}\_\{k\}=\\mathbf\{A\}\_\{k\}\-c\_\{k\}\\mathbf\{I\}, and define𝐉k=∂𝐱k\+1∂𝐱k\\mathbf\{J\}\_\{k\}=\\frac\{\\partial\\mathbf\{x\}\_\{k\+1\}\}\{\\partial\\mathbf\{x\}\_\{k\}\}\. For a perturbation𝝃∈ℝd\\bm\{\\xi\}\\in\\mathbb\{R\}^\{d\}, differentiation of𝐳k=ℱk​\(𝐱k\)−ck​𝐱k\\mathbf\{z\}\_\{k\}=\\mathcal\{F\}\_\{k\}\(\\mathbf\{x\}\_\{k\}\)\-c\_\{k\}\\mathbf\{x\}\_\{k\}gives

D​𝐳k​\[𝝃\]=𝐁k​𝝃−D​ck​\[𝝃\]​𝐱k\.D\\mathbf\{z\}\_\{k\}\[\\bm\{\\xi\}\]=\\mathbf\{B\}\_\{k\}\\bm\{\\xi\}\-Dc\_\{k\}\[\\bm\{\\xi\}\]\\mathbf\{x\}\_\{k\}\.Since𝐏k​𝐱k=𝟎\\mathbf\{P\}\_\{k\}\\mathbf\{x\}\_\{k\}=\\bm\{0\}, we immediately obtain𝐏k​D​𝐳k​\[𝝃\]=𝐏k​𝐁k​𝝃\\mathbf\{P\}\_\{k\}D\\mathbf\{z\}\_\{k\}\[\\bm\{\\xi\}\]=\\mathbf\{P\}\_\{k\}\\mathbf\{B\}\_\{k\}\\bm\{\\xi\}\. We next compute the derivative ofckc\_\{k\}\. Direct differentiation gives

D​ck​\[𝝃\]=𝝃⊤​ℱk​\(𝐱k\)\+𝐱k⊤​𝐀k​𝝃rk2−2​𝐱k⊤​ℱk​\(𝐱k\)rk4​𝐱k⊤​𝝃\.Dc\_\{k\}\[\\bm\{\\xi\}\]=\\frac\{\\bm\{\\xi\}^\{\\top\}\\mathcal\{F\}\_\{k\}\(\\mathbf\{x\}\_\{k\}\)\+\\mathbf\{x\}\_\{k\}^\{\\top\}\\mathbf\{A\}\_\{k\}\\bm\{\\xi\}\}\{r\_\{k\}^\{2\}\}\-2\\frac\{\\mathbf\{x\}\_\{k\}^\{\\top\}\\mathcal\{F\}\_\{k\}\(\\mathbf\{x\}\_\{k\}\)\}\{r\_\{k\}^\{4\}\}\\mathbf\{x\}\_\{k\}^\{\\top\}\\bm\{\\xi\}\.Usingℱk​\(𝐱k\)=𝐳k\+ck​𝐱k\\mathcal\{F\}\_\{k\}\(\\mathbf\{x\}\_\{k\}\)=\\mathbf\{z\}\_\{k\}\+c\_\{k\}\\mathbf\{x\}\_\{k\}and𝐀k=𝐁k\+ck​𝐈\\mathbf\{A\}\_\{k\}=\\mathbf\{B\}\_\{k\}\+c\_\{k\}\\mathbf\{I\}, the terms containingck​𝐱k⊤​𝝃c\_\{k\}\\mathbf\{x\}\_\{k\}^\{\\top\}\\bm\{\\xi\}cancel, yielding

D​ck​\[𝝃\]=𝐳k⊤​𝝃\+𝐱k⊤​𝐁k​𝝃rk2\.Dc\_\{k\}\[\\bm\{\\xi\}\]=\\frac\{\\mathbf\{z\}\_\{k\}^\{\\top\}\\bm\{\\xi\}\+\\mathbf\{x\}\_\{k\}^\{\\top\}\\mathbf\{B\}\_\{k\}\\bm\{\\xi\}\}\{r\_\{k\}^\{2\}\}\.
Suppose first thatsk\>0s\_\{k\}\>0\. Sincesk=‖𝐳k‖2s\_\{k\}=\\\|\\mathbf\{z\}\_\{k\}\\\|\_\{2\}andrk=‖𝐱k‖2r\_\{k\}=\\\|\\mathbf\{x\}\_\{k\}\\\|\_\{2\}, we have∇𝐱ksk=𝐁k⊤​𝐳ksk\\nabla\_\{\\mathbf\{x\}\_\{k\}\}s\_\{k\}=\\frac\{\\mathbf\{B\}\_\{k\}^\{\\top\}\\mathbf\{z\}\_\{k\}\}\{s\_\{k\}\}and∇𝐱krk=𝐱krk\\nabla\_\{\\mathbf\{x\}\_\{k\}\}r\_\{k\}=\\frac\{\\mathbf\{x\}\_\{k\}\}\{r\_\{k\}\}\. Therefore,

∇𝐱kβk=αk​\(𝐁k⊤​𝐳krk​sk−sk​𝐱krk3\)\.\\nabla\_\{\\mathbf\{x\}\_\{k\}\}\\beta\_\{k\}=\\alpha\_\{k\}\\left\(\\frac\{\\mathbf\{B\}\_\{k\}^\{\\top\}\\mathbf\{z\}\_\{k\}\}\{r\_\{k\}s\_\{k\}\}\-\\frac\{s\_\{k\}\\mathbf\{x\}\_\{k\}\}\{r\_\{k\}^\{3\}\}\\right\)\.Moreover,d​θkd​βk=p2p2\+βk2\\frac\{d\\theta\_\{k\}\}\{d\\beta\_\{k\}\}=\\frac\{p^\{2\}\}\{p^\{2\}\+\\beta\_\{k\}^\{2\}\}, and hence

∇𝐱kθk=αk​p2p2\+βk2​\(𝐁k⊤​𝐳krk​sk−sk​𝐱krk3\)\.\\nabla\_\{\\mathbf\{x\}\_\{k\}\}\\theta\_\{k\}=\\frac\{\\alpha\_\{k\}p^\{2\}\}\{p^\{2\}\+\\beta\_\{k\}^\{2\}\}\\left\(\\frac\{\\mathbf\{B\}\_\{k\}^\{\\top\}\\mathbf\{z\}\_\{k\}\}\{r\_\{k\}s\_\{k\}\}\-\\frac\{s\_\{k\}\\mathbf\{x\}\_\{k\}\}\{r\_\{k\}^\{3\}\}\\right\)\.
Forsk\>0s\_\{k\}\>0, thekk\-th SpheretNorm update is𝐱k\+1=cos⁡\(θk\)​𝐱k\+rk​sin⁡\(θk\)​𝐳ksk\\mathbf\{x\}\_\{k\+1\}=\\cos\(\\theta\_\{k\}\)\\mathbf\{x\}\_\{k\}\+r\_\{k\}\\sin\(\\theta\_\{k\}\)\\frac\{\\mathbf\{z\}\_\{k\}\}\{s\_\{k\}\}\. Differentiating this expression and substituting the formula forD​ck​\[𝝃\]Dc\_\{k\}\[\\bm\{\\xi\}\]yields

𝐉k=\\displaystyle\\mathbf\{J\}\_\{k\}=\{\}cos⁡\(θk\)​𝐈\+sin⁡\(θk\)rk​sk​\(𝐳k​𝐱k⊤−𝐱k​𝐳k⊤\)\\displaystyle\\cos\(\\theta\_\{k\}\)\\mathbf\{I\}\+\\frac\{\\sin\(\\theta\_\{k\}\)\}\{r\_\{k\}s\_\{k\}\}\\left\(\\mathbf\{z\}\_\{k\}\\mathbf\{x\}\_\{k\}^\{\\top\}\-\\mathbf\{x\}\_\{k\}\\mathbf\{z\}\_\{k\}^\{\\top\}\\right\)\+rk​sin⁡\(θk\)sk​\(𝐏k−𝐳k​𝐳k⊤sk2\)​𝐁k\\displaystyle\+\\frac\{r\_\{k\}\\sin\(\\theta\_\{k\}\)\}\{s\_\{k\}\}\\left\(\\mathbf\{P\}\_\{k\}\-\\frac\{\\mathbf\{z\}\_\{k\}\\mathbf\{z\}\_\{k\}^\{\\top\}\}\{s\_\{k\}^\{2\}\}\\right\)\\mathbf\{B\}\_\{k\}\+\(rk​cos⁡\(θk\)sk​𝐳k−sin⁡\(θk\)​𝐱k\)​\(∇𝐱kθk\)⊤\.\\displaystyle\+\\left\(\\frac\{r\_\{k\}\\cos\(\\theta\_\{k\}\)\}\{s\_\{k\}\}\\mathbf\{z\}\_\{k\}\-\\sin\(\\theta\_\{k\}\)\\mathbf\{x\}\_\{k\}\\right\)\\left\(\\nabla\_\{\\mathbf\{x\}\_\{k\}\}\\theta\_\{k\}\\right\)^\{\\top\}\.
Define𝐐k=cos⁡\(θk\)​𝐈\+sin⁡\(θk\)rk​sk​\(𝐳k​𝐱k⊤−𝐱k​𝐳k⊤\)\\mathbf\{Q\}\_\{k\}=\\cos\(\\theta\_\{k\}\)\\mathbf\{I\}\+\\frac\{\\sin\(\\theta\_\{k\}\)\}\{r\_\{k\}s\_\{k\}\}\\left\(\\mathbf\{z\}\_\{k\}\\mathbf\{x\}\_\{k\}^\{\\top\}\-\\mathbf\{x\}\_\{k\}\\mathbf\{z\}\_\{k\}^\{\\top\}\\right\)\. Then𝐉k=𝐐k\+𝐄k\\mathbf\{J\}\_\{k\}=\\mathbf\{Q\}\_\{k\}\+\\mathbf\{E\}\_\{k\}, where

𝐄k=\\displaystyle\\mathbf\{E\}\_\{k\}=\{\}rk​sin⁡\(θk\)sk​\(𝐏k−𝐳k​𝐳k⊤sk2\)​𝐁k\\displaystyle\\frac\{r\_\{k\}\\sin\(\\theta\_\{k\}\)\}\{s\_\{k\}\}\\left\(\\mathbf\{P\}\_\{k\}\-\\frac\{\\mathbf\{z\}\_\{k\}\\mathbf\{z\}\_\{k\}^\{\\top\}\}\{s\_\{k\}^\{2\}\}\\right\)\\mathbf\{B\}\_\{k\}\+\(rk​cos⁡\(θk\)sk​𝐳k−sin⁡\(θk\)​𝐱k\)​\(∇𝐱kθk\)⊤\.\\displaystyle\+\\left\(\\frac\{r\_\{k\}\\cos\(\\theta\_\{k\}\)\}\{s\_\{k\}\}\\mathbf\{z\}\_\{k\}\-\\sin\(\\theta\_\{k\}\)\\mathbf\{x\}\_\{k\}\\right\)\\left\(\\nabla\_\{\\mathbf\{x\}\_\{k\}\}\\theta\_\{k\}\\right\)^\{\\top\}\.

##### Estimate of the perturbation term\.

Since𝐏k−𝐳k​𝐳k⊤sk2\\mathbf\{P\}\_\{k\}\-\\frac\{\\mathbf\{z\}\_\{k\}\\mathbf\{z\}\_\{k\}^\{\\top\}\}\{s\_\{k\}^\{2\}\}is the orthogonal projection onto the subspace ofT𝐱k​𝕊rkd−1T\_\{\\mathbf\{x\}\_\{k\}\}\\mathbb\{S\}\_\{r\_\{k\}\}^\{d\-1\}orthogonal to𝐳k\\mathbf\{z\}\_\{k\}, its operator norm is at most one\. Therefore,

‖rk​sin⁡\(θk\)sk​\(𝐏k−𝐳k​𝐳k⊤sk2\)​𝐁k‖2≤rk​\|sin⁡\(θk\)\|sk​‖𝐁k‖2\.\\left\\\|\\frac\{r\_\{k\}\\sin\(\\theta\_\{k\}\)\}\{s\_\{k\}\}\\left\(\\mathbf\{P\}\_\{k\}\-\\frac\{\\mathbf\{z\}\_\{k\}\\mathbf\{z\}\_\{k\}^\{\\top\}\}\{s\_\{k\}^\{2\}\}\\right\)\\mathbf\{B\}\_\{k\}\\right\\\|\_\{2\}\\leq\\frac\{r\_\{k\}\|\\sin\(\\theta\_\{k\}\)\|\}\{s\_\{k\}\}\\\|\\mathbf\{B\}\_\{k\}\\\|\_\{2\}\.Since\|βk\|=\|αk\|​skrk\|\\beta\_\{k\}\|=\|\\alpha\_\{k\}\|\\frac\{s\_\{k\}\}\{r\_\{k\}\}, we obtain

‖rk​sin⁡\(θk\)sk​\(𝐏k−𝐳k​𝐳k⊤sk2\)​𝐁k‖2≤\|αk\|​\|sin⁡\(θk\)\|\|βk\|​‖𝐁k‖2\.\\left\\\|\\frac\{r\_\{k\}\\sin\(\\theta\_\{k\}\)\}\{s\_\{k\}\}\\left\(\\mathbf\{P\}\_\{k\}\-\\frac\{\\mathbf\{z\}\_\{k\}\\mathbf\{z\}\_\{k\}^\{\\top\}\}\{s\_\{k\}^\{2\}\}\\right\)\\mathbf\{B\}\_\{k\}\\right\\\|\_\{2\}\\leq\|\\alpha\_\{k\}\|\\frac\{\|\\sin\(\\theta\_\{k\}\)\|\}\{\|\\beta\_\{k\}\|\}\\\|\\mathbf\{B\}\_\{k\}\\\|\_\{2\}\.
For the second term in𝐄k\\mathbf\{E\}\_\{k\}, the orthogonality𝐱k⊤​𝐳k=0\\mathbf\{x\}\_\{k\}^\{\\top\}\\mathbf\{z\}\_\{k\}=0gives

‖rk​cos⁡\(θk\)sk​𝐳k−sin⁡\(θk\)​𝐱k‖22\\displaystyle\\left\\\|\\frac\{r\_\{k\}\\cos\(\\theta\_\{k\}\)\}\{s\_\{k\}\}\\mathbf\{z\}\_\{k\}\-\\sin\(\\theta\_\{k\}\)\\mathbf\{x\}\_\{k\}\\right\\\|\_\{2\}^\{2\}=rk2​cos2⁡\(θk\)\+rk2​sin2⁡\(θk\)=rk2\.\\displaystyle=r\_\{k\}^\{2\}\\cos^\{2\}\(\\theta\_\{k\}\)\+r\_\{k\}^\{2\}\\sin^\{2\}\(\\theta\_\{k\}\)=r\_\{k\}^\{2\}\.Hence,

‖rk​cos⁡\(θk\)sk​𝐳k−sin⁡\(θk\)​𝐱k‖2=rk\.\\left\\\|\\frac\{r\_\{k\}\\cos\(\\theta\_\{k\}\)\}\{s\_\{k\}\}\\mathbf\{z\}\_\{k\}\-\\sin\(\\theta\_\{k\}\)\\mathbf\{x\}\_\{k\}\\right\\\|\_\{2\}=r\_\{k\}\.Furthermore,

‖∇𝐱kθk‖2≤p2p2\+βk2​\|αk\|​‖𝐁k‖2\+\|βk\|rk\.\\left\\\|\\nabla\_\{\\mathbf\{x\}\_\{k\}\}\\theta\_\{k\}\\right\\\|\_\{2\}\\leq\\frac\{p^\{2\}\}\{p^\{2\}\+\\beta\_\{k\}^\{2\}\}\\frac\{\|\\alpha\_\{k\}\|\\\|\\mathbf\{B\}\_\{k\}\\\|\_\{2\}\+\|\\beta\_\{k\}\|\}\{r\_\{k\}\}\.It follows that‖𝐄k‖2≤δkexact\\\|\\mathbf\{E\}\_\{k\}\\\|\_\{2\}\\leq\\delta\_\{k\}^\{\\mathrm\{exact\}\}, where

δkexact=\|αk\|​\|sin⁡\(θk\)\|\|βk\|​‖𝐁k‖2\+p2p2\+βk2​\(\|αk\|​‖𝐁k‖2\+\|βk\|\)\.\\delta\_\{k\}^\{\\mathrm\{exact\}\}=\|\\alpha\_\{k\}\|\\frac\{\|\\sin\(\\theta\_\{k\}\)\|\}\{\|\\beta\_\{k\}\|\}\\\|\\mathbf\{B\}\_\{k\}\\\|\_\{2\}\+\\frac\{p^\{2\}\}\{p^\{2\}\+\\beta\_\{k\}^\{2\}\}\\left\(\|\\alpha\_\{k\}\|\\\|\\mathbf\{B\}\_\{k\}\\\|\_\{2\}\+\|\\beta\_\{k\}\|\\right\)\.The quotient\|sin⁡\(θk\)\|\|βk\|\\frac\{\|\\sin\(\\theta\_\{k\}\)\|\}\{\|\\beta\_\{k\}\|\}is understood through its continuous extension atβk=0\\beta\_\{k\}=0\.

Since\|sin⁡\(θk\)\|≤\|θk\|≤\|βk\|\|\\sin\(\\theta\_\{k\}\)\|\\leq\|\\theta\_\{k\}\|\\leq\|\\beta\_\{k\}\|,p2p2\+βk2≤1\\frac\{p^\{2\}\}\{p^\{2\}\+\\beta\_\{k\}^\{2\}\}\\leq 1, and\|βk\|=\|αk\|​ηk\|\\beta\_\{k\}\|=\|\\alpha\_\{k\}\|\\eta\_\{k\}, we obtain

δkexact≤\|αk\|​\(2​‖𝐁k‖2\+ηk\)\.\\delta\_\{k\}^\{\\mathrm\{exact\}\}\\leq\|\\alpha\_\{k\}\|\\left\(2\\\|\\mathbf\{B\}\_\{k\}\\\|\_\{2\}\+\\eta\_\{k\}\\right\)\.DefineΔk=\|αk\|​\(2​‖𝐁k‖2\+ηk\)\\Delta\_\{k\}=\|\\alpha\_\{k\}\|\\left\(2\\\|\\mathbf\{B\}\_\{k\}\\\|\_\{2\}\+\\eta\_\{k\}\\right\)\. Then‖𝐄k‖2≤Δk\\\|\\mathbf\{E\}\_\{k\}\\\|\_\{2\}\\leq\\Delta\_\{k\}\.

##### The zero\-tangent case\.

We now considersk=0s\_\{k\}=0\. At the point under consideration,𝐳k=𝟎\\mathbf\{z\}\_\{k\}=\\bm\{0\},βk=0\\beta\_\{k\}=0\. Direct differentiation ofβk=αk​‖𝐳k‖2rk\\beta\_\{k\}=\\alpha\_\{k\}\\frac\{\\\|\\mathbf\{z\}\_\{k\}\\\|\_\{2\}\}\{r\_\{k\}\}is not valid at𝐳k=𝟎\\mathbf\{z\}\_\{k\}=\\bm\{0\}, because the Euclidean norm is not differentiable at the origin\. We therefore use the even\-function representation established in Appendix[D\.4](https://arxiv.org/html/2608.02668#A4.SS4)\.

Letap​\(β\)=cos⁡\(p​arctan⁡\(βp\)\)a\_\{p\}\(\\beta\)=\\cos\\left\(p\\arctan\\left\(\\frac\{\\beta\}\{p\}\\right\)\\right\)and

bp​\(β\)=\{sin⁡\(p​arctan⁡\(β/p\)\)β,β≠0,1,β=0\.b\_\{p\}\(\\beta\)=\\begin\{cases\}\\displaystyle\\frac\{\\sin\\left\(p\\arctan\(\\beta/p\)\\right\)\}\{\\beta\},&\\beta\\neq 0,\\\\\[5\.16663pt\] 1,&\\beta=0\.\\end\{cases\}There exist real\-analytic functionsA^p\\widehat\{A\}\_\{p\}andB^p\\widehat\{B\}\_\{p\}, defined near zero, such that

ap​\(β\)=A^p​\(β2\),bp​\(β\)=B^p​\(β2\),a\_\{p\}\(\\beta\)=\\widehat\{A\}\_\{p\}\(\\beta^\{2\}\),\\qquad b\_\{p\}\(\\beta\)=\\widehat\{B\}\_\{p\}\(\\beta^\{2\}\),withA^p​\(0\)=B^p​\(0\)=1\\widehat\{A\}\_\{p\}\(0\)=\\widehat\{B\}\_\{p\}\(0\)=1\. For𝐱≠𝟎\\mathbf\{x\}\\neq\\mathbf\{0\}, define

𝐳k​\(𝐱\)=\(𝐈−𝐱𝐱⊤‖𝐱‖22\)​ℱk​\(𝐱\),ωk​\(𝐱\)=αk2​‖𝐳k​\(𝐱\)‖22‖𝐱‖22\.\\mathbf\{z\}\_\{k\}\(\\mathbf\{x\}\)=\\left\(\\mathbf\{I\}\-\\frac\{\\mathbf\{x\}\\mathbf\{x\}^\{\\top\}\}\{\\\|\\mathbf\{x\}\\\|\_\{2\}^\{2\}\}\\right\)\\mathcal\{F\}\_\{k\}\(\\mathbf\{x\}\),\\quad\\omega\_\{k\}\(\\mathbf\{x\}\)=\\alpha\_\{k\}^\{2\}\\frac\{\\\|\\mathbf\{z\}\_\{k\}\(\\mathbf\{x\}\)\\\|\_\{2\}^\{2\}\}\{\\\|\\mathbf\{x\}\\\|\_\{2\}^\{2\}\}\.The layer map can be written locally as

𝐓k​\(𝐱\)=A^p​\(ωk​\(𝐱\)\)​𝐱\+αk​B^p​\(ωk​\(𝐱\)\)​𝐳k​\(𝐱\)\.\\mathbf\{T\}\_\{k\}\(\\mathbf\{x\}\)=\\widehat\{A\}\_\{p\}\\bigl\(\\omega\_\{k\}\(\\mathbf\{x\}\)\\bigr\)\\mathbf\{x\}\+\\alpha\_\{k\}\\widehat\{B\}\_\{p\}\\bigl\(\\omega\_\{k\}\(\\mathbf\{x\}\)\\bigr\)\\mathbf\{z\}\_\{k\}\(\\mathbf\{x\}\)\.For a perturbation𝝃∈ℝd\\bm\{\\xi\}\\in\\mathbb\{R\}^\{d\},

D​ωk​\(𝐱k\)​\[𝝃\]=2​αk2rk2​𝐳k⊤​D​𝐳k​\[𝝃\]−2​αk2​sk2rk4​𝐱k⊤​𝝃\.D\\omega\_\{k\}\(\\mathbf\{x\}\_\{k\}\)\[\\bm\{\\xi\}\]=\\frac\{2\\alpha\_\{k\}^\{2\}\}\{r\_\{k\}^\{2\}\}\\mathbf\{z\}\_\{k\}^\{\\top\}D\\mathbf\{z\}\_\{k\}\[\\bm\{\\xi\}\]\-\\frac\{2\\alpha\_\{k\}^\{2\}s\_\{k\}^\{2\}\}\{r\_\{k\}^\{4\}\}\\mathbf\{x\}\_\{k\}^\{\\top\}\\bm\{\\xi\}\.Whensk=0s\_\{k\}=0, we haveωk​\(𝐱k\)=0\\omega\_\{k\}\(\\mathbf\{x\}\_\{k\}\)=0andD​ωk​\(𝐱k\)​\[𝝃\]=0D\\omega\_\{k\}\(\\mathbf\{x\}\_\{k\}\)\[\\bm\{\\xi\}\]=0\. Consequently,

D​𝐓k​\(𝐱k\)​\[𝝃\]=𝝃\+αk​D​𝐳k​\[𝝃\]\.D\\mathbf\{T\}\_\{k\}\(\\mathbf\{x\}\_\{k\}\)\[\\bm\{\\xi\}\]=\\bm\{\\xi\}\+\\alpha\_\{k\}D\\mathbf\{z\}\_\{k\}\[\\bm\{\\xi\}\]\.
Since𝐳k=𝟎\\mathbf\{z\}\_\{k\}=\\mathbf\{0\}, one hasℱk​\(𝐱k\)=ck​𝐱k\\mathcal\{F\}\_\{k\}\(\\mathbf\{x\}\_\{k\}\)=c\_\{k\}\\mathbf\{x\}\_\{k\}\. The derivative ofckc\_\{k\}therefore reduces to

D​ck​\[𝝃\]=𝐱k⊤​𝐁k​𝝃rk2\.Dc\_\{k\}\[\\bm\{\\xi\}\]=\\frac\{\\mathbf\{x\}\_\{k\}^\{\\top\}\\mathbf\{B\}\_\{k\}\\bm\{\\xi\}\}\{r\_\{k\}^\{2\}\}\.Hence,

D​𝐳k​\[𝝃\]\\displaystyle D\\mathbf\{z\}\_\{k\}\[\\bm\{\\xi\}\]=𝐁k​𝝃−D​ck​\[𝝃\]​𝐱k\\displaystyle=\\mathbf\{B\}\_\{k\}\\bm\{\\xi\}\-Dc\_\{k\}\[\\bm\{\\xi\}\]\\mathbf\{x\}\_\{k\}=𝐁k​𝝃−𝐱k​𝐱k⊤rk2​𝐁k​𝝃=𝐏k​𝐁k​𝝃\.\\displaystyle=\\mathbf\{B\}\_\{k\}\\bm\{\\xi\}\-\\frac\{\\mathbf\{x\}\_\{k\}\\mathbf\{x\}\_\{k\}^\{\\top\}\}\{r\_\{k\}^\{2\}\}\\mathbf\{B\}\_\{k\}\\bm\{\\xi\}=\\mathbf\{P\}\_\{k\}\\mathbf\{B\}\_\{k\}\\bm\{\\xi\}\.It follows that𝐉k=𝐈\+αk​𝐏k​𝐁k\\mathbf\{J\}\_\{k\}=\\mathbf\{I\}\+\\alpha\_\{k\}\\mathbf\{P\}\_\{k\}\\mathbf\{B\}\_\{k\}whensk=0s\_\{k\}=0\. Defineδk\(0\)=\|αk\|​‖𝐏k​𝐁k‖2\\delta\_\{k\}^\{\(0\)\}=\|\\alpha\_\{k\}\|\\\|\\mathbf\{P\}\_\{k\}\\mathbf\{B\}\_\{k\}\\\|\_\{2\}\. Weyl’s singular\-value perturbation inequality gives

σmax​\(𝐉k\)≤1\+δk\(0\)\\sigma\_\{\\max\}\(\\mathbf\{J\}\_\{k\}\)\\leq 1\+\\delta\_\{k\}^\{\(0\)\}and

σmin​\(𝐉k\)≥max⁡\{0,1−δk\(0\)\}\.\\sigma\_\{\\min\}\(\\mathbf\{J\}\_\{k\}\)\\geq\\max\\\{0,1\-\\delta\_\{k\}^\{\(0\)\}\\\}\.Moreover,δk\(0\)≤\|αk\|​‖𝐁k‖2≤Δk\\delta\_\{k\}^\{\(0\)\}\\leq\|\\alpha\_\{k\}\|\\\|\\mathbf\{B\}\_\{k\}\\\|\_\{2\}\\leq\\Delta\_\{k\}\.

##### Singular\-value bounds\.

Forsk\>0s\_\{k\}\>0, define𝐞1,k=𝐱krk\\mathbf\{e\}\_\{1,k\}=\\frac\{\\mathbf\{x\}\_\{k\}\}\{r\_\{k\}\},𝐞2,k=𝐳ksk\\mathbf\{e\}\_\{2,k\}=\\frac\{\\mathbf\{z\}\_\{k\}\}\{s\_\{k\}\}\. Onspan⁡\{𝐞1,k,𝐞2,k\}\\operatorname\{span\}\\\{\\mathbf\{e\}\_\{1,k\},\\mathbf\{e\}\_\{2,k\}\\\}, the matrix of𝐐k\\mathbf\{Q\}\_\{k\}, with respect to this orthonormal basis, is

\(cos⁡\(θk\)−sin⁡\(θk\)sin⁡\(θk\)cos⁡\(θk\)\)\.\\begin\{pmatrix\}\\cos\(\\theta\_\{k\}\)&\-\\sin\(\\theta\_\{k\}\)\\\\ \\sin\(\\theta\_\{k\}\)&\\cos\(\\theta\_\{k\}\)\\end\{pmatrix\}\.Thus,𝐐k\\mathbf\{Q\}\_\{k\}acts as an isometry on this two\-dimensional subspace\. On its orthogonal complement,𝐐k\\mathbf\{Q\}\_\{k\}acts as multiplication bycos⁡\(θk\)\\cos\(\\theta\_\{k\}\)\. Consequently, we obtain that‖𝐐k‖2=1\\\|\\mathbf\{Q\}\_\{k\}\\\|\_\{2\}=1andσmin​\(𝐐k\)≥\|cos⁡\(θk\)\|\\sigma\_\{\\min\}\(\\mathbf\{Q\}\_\{k\}\)\\geq\|\\cos\(\\theta\_\{k\}\)\|\. Forsk\>0s\_\{k\}\>0, Weyl’s inequality therefore gives

σmax​\(𝐉k\)≤1\+Δk\\sigma\_\{\\max\}\(\\mathbf\{J\}\_\{k\}\)\\leq 1\+\\Delta\_\{k\}and

σmin​\(𝐉k\)≥max⁡\{0,\|cos⁡\(θk\)\|−Δk\}\.\\sigma\_\{\\min\}\(\\mathbf\{J\}\_\{k\}\)\\geq\\max\\\{0,\|\\cos\(\\theta\_\{k\}\)\|\-\\Delta\_\{k\}\\\}\.
Define

qk=\{\|cos⁡\(θk\)\|,sk\>0,1,sk=0,q\_\{k\}=\\begin\{cases\}\|\\cos\(\\theta\_\{k\}\)\|,&s\_\{k\}\>0,\\\\ 1,&s\_\{k\}=0,\\end\{cases\}andmk=max⁡\{0,qk−Δk\}m\_\{k\}=\\max\\\{0,q\_\{k\}\-\\Delta\_\{k\}\\\}\. Combining the casessk\>0s\_\{k\}\>0andsk=0s\_\{k\}=0, we obtain

σmax​\(𝐉k\)≤1\+Δk,σmin​\(𝐉k\)≥mk,k=1,…,N\.\\sigma\_\{\\max\}\(\\mathbf\{J\}\_\{k\}\)\\leq 1\+\\Delta\_\{k\},\\qquad\\sigma\_\{\\min\}\(\\mathbf\{J\}\_\{k\}\)\\geq m\_\{k\},\\qquad k=1,\\ldots,N\.

##### Depth\-wise Jacobian\.

The end\-to\-end Jacobian over the retained residual stream is

𝐆=∂𝐡L∂𝐡1=∂𝐱N\+1∂𝐱1\.\\mathbf\{G\}=\\frac\{\\partial\\mathbf\{h\}\_\{L\}\}\{\\partial\\mathbf\{h\}\_\{1\}\}=\\frac\{\\partial\\mathbf\{x\}\_\{N\+1\}\}\{\\partial\\mathbf\{x\}\_\{1\}\}\.SinceN=L−1N=L\-1, the chain rule gives𝐆=𝐉N​𝐉N−1​⋯​𝐉1=𝐉L−1​𝐉L−2​⋯​𝐉1\\mathbf\{G\}=\\mathbf\{J\}\_\{N\}\\mathbf\{J\}\_\{N\-1\}\\cdots\\mathbf\{J\}\_\{1\}=\\mathbf\{J\}\_\{L\-1\}\\mathbf\{J\}\_\{L\-2\}\\cdots\\mathbf\{J\}\_\{1\}\. By submultiplicativity,

σmax​\(𝐆\)\\displaystyle\\sigma\_\{\\max\}\(\\mathbf\{G\}\)≤∏k=1Nσmax​\(𝐉k\)≤∏k=1N\(1\+Δk\)\\displaystyle\\leq\\prod\_\{k=1\}^\{N\}\\sigma\_\{\\max\}\(\\mathbf\{J\}\_\{k\}\)\\leq\\prod\_\{k=1\}^\{N\}\(1\+\\Delta\_\{k\}\)≤exp⁡\(∑k=1NΔk\)\.\\displaystyle\\leq\\exp\\left\(\\sum\_\{k=1\}^\{N\}\\Delta\_\{k\}\\right\)\.Equivalently,σmax​\(𝐆\)≤exp⁡\(∑k=1L−1Δk\)\\sigma\_\{\\max\}\(\\mathbf\{G\}\)\\leq\\exp\\left\(\\sum\_\{k=1\}^\{L\-1\}\\Delta\_\{k\}\\right\)\. Similarly,

σmin​\(𝐆\)≥∏k=1Nσmin​\(𝐉k\)≥∏k=1Nmk=∏k=1L−1mk\.\\sigma\_\{\\min\}\(\\mathbf\{G\}\)\\geq\\prod\_\{k=1\}^\{N\}\\sigma\_\{\\min\}\(\\mathbf\{J\}\_\{k\}\)\\geq\\prod\_\{k=1\}^\{N\}m\_\{k\}=\\prod\_\{k=1\}^\{L\-1\}m\_\{k\}\.

##### A consequence of exact norm preservation\.

Since‖𝐱k\+1‖22=‖𝐱k‖22\\\|\\mathbf\{x\}\_\{k\+1\}\\\|\_\{2\}^\{2\}=\\\|\\mathbf\{x\}\_\{k\}\\\|\_\{2\}^\{2\}for every admissible𝐱k\\mathbf\{x\}\_\{k\}, differentiation with respect to𝐱k\\mathbf\{x\}\_\{k\}yields𝐉k⊤​𝐱k\+1=𝐱k\\mathbf\{J\}\_\{k\}^\{\\top\}\\mathbf\{x\}\_\{k\+1\}=\\mathbf\{x\}\_\{k\}\. Therefore,

‖𝐉k‖2=‖𝐉k⊤‖2≥‖𝐉k⊤​𝐱k\+1‖2‖𝐱k\+1‖2=‖𝐱k‖2‖𝐱k\+1‖2=1\.\\\|\\mathbf\{J\}\_\{k\}\\\|\_\{2\}=\\\|\\mathbf\{J\}\_\{k\}^\{\\top\}\\\|\_\{2\}\\geq\\frac\{\\\|\\mathbf\{J\}\_\{k\}^\{\\top\}\\mathbf\{x\}\_\{k\+1\}\\\|\_\{2\}\}\{\\\|\\mathbf\{x\}\_\{k\+1\}\\\|\_\{2\}\}=\\frac\{\\\|\\mathbf\{x\}\_\{k\}\\\|\_\{2\}\}\{\\\|\\mathbf\{x\}\_\{k\+1\}\\\|\_\{2\}\}=1\.Thus, an individual SpheretNorm update Jacobian cannot be a strict contraction in every ambient direction, although it may still strongly contract particular tangential directions\.

### D\.8Angle Control Alone Is Insufficient

The intrinsic rotation angle controls the geometric component of the update Jacobian, but it does not control the differential sensitivity of the underlying mapping\. The following example shows that even zero angular displacement provides no uniform upper bound on the update\-Jacobian norm and does not guarantee nonsingularity\.

###### Proposition 9\.

For everyR\>1R\>1and everyα\>0\\alpha\>0, there exist a linear mappingℱ:ℝ2→ℝ2\\mathcal\{F\}:\\mathbb\{R\}^\{2\}\\to\\mathbb\{R\}^\{2\}and a hidden state𝐡∈𝕊1\\mathbf\{h\}\\in\\mathbb\{S\}^\{1\}such that the corresponding SpheretNorm update satisfiesβ=0\\beta=0andθ\(p\)=0\\theta^\{\(p\)\}=0, while its update Jacobian satisfies‖𝐉‖2=R\\\|\\mathbf\{J\}\\\|\_\{2\}=R\.

###### Proof\.

Let𝐡=𝐞1=\(10\)\\mathbf\{h\}=\\mathbf\{e\}\_\{1\}=\\begin\{pmatrix\}1\\\\ 0\\end\{pmatrix\}and defineℱ​\(𝐱\)=𝐀𝐱\\mathcal\{F\}\(\\mathbf\{x\}\)=\\mathbf\{A\}\\mathbf\{x\}and𝐀=\(μ00μ\+K\)\\mathbf\{A\}=\\begin\{pmatrix\}\\mu&0\\\\ 0&\\mu\+K\\end\{pmatrix\}, whereμ,K∈ℝ\\mu,K\\in\\mathbb\{R\}\. When𝐡=𝐞1\\mathbf\{h\}=\\mathbf\{e\}\_\{1\}, we haveℱ​\(𝐡\)=μ​𝐞1\\mathcal\{F\}\(\\mathbf\{h\}\)=\\mu\\mathbf\{e\}\_\{1\}andc=𝐡⊤​ℱ​\(𝐡\)=μc=\\mathbf\{h\}^\{\\top\}\\mathcal\{F\}\(\\mathbf\{h\}\)=\\mu\. Therefore,

𝐁=𝐀−c​𝐈=\(000K\)\.\\mathbf\{B\}=\\mathbf\{A\}\-c\\mathbf\{I\}=\\begin\{pmatrix\}0&0\\\\ 0&K\\end\{pmatrix\}\.
Moreover,𝐏=𝐈−𝐡𝐡⊤=\(0001\)\\mathbf\{P\}=\\mathbf\{I\}\-\\mathbf\{h\}\\mathbf\{h\}^\{\\top\}=\\begin\{pmatrix\}0&0\\\\ 0&1\\end\{pmatrix\}, and hence𝐳=𝐏​ℱ​\(𝐡\)=𝟎\\mathbf\{z\}=\\mathbf\{P\}\\mathcal\{F\}\(\\mathbf\{h\}\)=\\bm\{0\}\. Therefore,β=0\\beta=0andθ\(p\)=0\\theta^\{\(p\)\}=0for everyp\>0p\>0\. Since𝐳=𝟎\\mathbf\{z\}=\\bm\{0\}, the zero\-tangent formula established in Appendix[D\.7](https://arxiv.org/html/2608.02668#A4.SS7)gives

𝐉=𝐈\+α​𝐏𝐁=\(1001\+α​K\)\.\\mathbf\{J\}=\\mathbf\{I\}\+\\alpha\\mathbf\{P\}\\mathbf\{B\}=\\begin\{pmatrix\}1&0\\\\ 0&1\+\\alpha K\\end\{pmatrix\}\.ChoosingK=R−1αK=\\frac\{R\-1\}\{\\alpha\}gives𝐉=\(100R\)\\mathbf\{J\}=\\begin\{pmatrix\}1&0\\\\ 0&R\\end\{pmatrix\}, and therefore‖𝐉‖2=R\\\|\\mathbf\{J\}\\\|\_\{2\}=R\. ∎

The same construction also shows that zero angular displacement does not imply nonsingularity\. Indeed, choosingK=−1αK=\-\\frac\{1\}\{\\alpha\}gives𝐉=\(1000\)\\mathbf\{J\}=\\begin\{pmatrix\}1&0\\\\ 0&0\\end\{pmatrix\}, so thatσmin​\(𝐉\)=0\\sigma\_\{\\min\}\(\\mathbf\{J\}\)=0\. Therefore, angular control must be supplemented by control of the update\-sensitivity term\|α\|​‖𝐁‖2\|\\alpha\|\\\|\\mathbf\{B\}\\\|\_\{2\}\.

### D\.9Dyadic Angular Budget of the Depth Schedules

The global estimates in Proposition[8](https://arxiv.org/html/2608.02668#Thmtheorem8)compare the accumulated stability envelopes over the entire sequence of SpheretNorm updates\. We now examine how the available angular motion is distributed across different depth scales\.

Under the retained\-state indexing𝐡1,𝐡2,…,𝐡L\\mathbf\{h\}\_\{1\},\\mathbf\{h\}\_\{2\},\\ldots,\\mathbf\{h\}\_\{L\}, the individual SpheretNorm updates are indexed byl=1,…,L−1l=1,\\ldots,L\-1\. For an integerm≥1m\\geq 1satisfying2​m≤L2m\\leq L, define the dyadic update intervalIm=\{m,m\+1,…,2​m−1\}I\_\{m\}=\\\{m,m\+1,\\ldots,2m\-1\\\}\. The condition2​m≤L2m\\leq Lensures thatIm⊆\{1,…,L−1\}I\_\{m\}\\subseteq\\\{1,\\ldots,L\-1\\\}\.

###### Proposition 10\.

Fix an optimization iterationτ\\tauand letp\>0p\>0\. Suppose that there exist constants

0<H−≤H\+<∞0<H\_\{\-\}\\leq H\_\{\+\}<\\inftyandε\>0\\varepsilon\>0such that, for everyl∈Iml\\in I\_\{m\},

H−≤al\(τ\)​ηl\(τ\)≤H\+and\|βl\(τ\)\|≤ε\.H\_\{\-\}\\leq a\_\{l\}^\{\(\\tau\)\}\\eta\_\{l\}^\{\(\\tau\)\}\\leq H\_\{\+\}\\quad\\mbox\{ and \}\\quad\\left\|\\beta\_\{l\}^\{\(\\tau\)\}\\right\|\\leq\\varepsilon\.Then there exist constantsC−,C\+\>0C\_\{\-\},C\_\{\+\}\>0, independent ofmmandLL, such that

C−​∑l∈Imλl2≤∑l∈Im\|θl,τ\(p\)\|2≤C\+​∑l∈Imλl2\.C\_\{\-\}\\sum\_\{l\\in I\_\{m\}\}\\lambda\_\{l\}^\{2\}\\leq\\sum\_\{l\\in I\_\{m\}\}\\left\|\\theta\_\{l,\\tau\}^\{\(p\)\}\\right\|^\{2\}\\leq C\_\{\+\}\\sum\_\{l\\in I\_\{m\}\}\\lambda\_\{l\}^\{2\}\.Consequently,∑l∈Im\|θl,τ\(p\)\|2=Θ​\(1\)\\sum\_\{l\\in I\_\{m\}\}\\left\|\\theta\_\{l,\\tau\}^\{\(p\)\}\\right\|^\{2\}=\\Theta\(1\)for the layerwise square\-root scheduleλl=l−1/2\\lambda\_\{l\}=l^\{\-1/2\}, while∑l∈Im\|θl,τ\(p\)\|2=Θ​\(1m\)\\sum\_\{l\\in I\_\{m\}\}\\left\|\\theta\_\{l,\\tau\}^\{\(p\)\}\\right\|^\{2\}=\\Theta\(\\frac\{1\}\{m\}\)for the harmonic scheduleλl=l−1\\lambda\_\{l\}=l^\{\-1\}\.

###### Proof\.

Define

ϕp​\(t\)=\{p​arctan⁡\(t/p\)t,t\>0,1,t=0\.\\phi\_\{p\}\(t\)=\\begin\{cases\}\\displaystyle\\frac\{p\\arctan\(t/p\)\}\{t\},&t\>0,\\\\\[8\.53581pt\] 1,&t=0\.\\end\{cases\}The functionϕp\\phi\_\{p\}is continuous and strictly positive on the compact interval\[0,ε\]\[0,\\varepsilon\]\. Hence there exists a constantcp,ε=min0≤t≤ε⁡ϕp​\(t\)\>0c\_\{p,\\varepsilon\}=\\min\_\{0\\leq t\\leq\\varepsilon\}\\phi\_\{p\}\(t\)\>0such that

cp,ε​t≤p​arctan⁡\(tp\)≤t,0≤t≤ε\.c\_\{p,\\varepsilon\}t\\leq p\\arctan\\left\(\\frac\{t\}\{p\}\\right\)\\leq t,\\qquad 0\\leq t\\leq\\varepsilon\.Applying this estimate witht=\|βl\(τ\)\|t=\|\\beta\_\{l\}^\{\(\\tau\)\}\|givescp,ε​\|βl\(τ\)\|≤\|θl,τ\(p\)\|≤\|βl\(τ\)\|c\_\{p,\\varepsilon\}\\left\|\\beta\_\{l\}^\{\(\\tau\)\}\\right\|\\leq\\left\|\\theta\_\{l,\\tau\}^\{\(p\)\}\\right\|\\leq\\left\|\\beta\_\{l\}^\{\(\\tau\)\}\\right\|\. Since\|βl\(τ\)\|=λl​al\(τ\)​ηl\(τ\)\\left\|\\beta\_\{l\}^\{\(\\tau\)\}\\right\|=\\lambda\_\{l\}a\_\{l\}^\{\(\\tau\)\}\\eta\_\{l\}^\{\(\\tau\)\}, the assumptions implycp,ε​H−​λl≤\|θl,τ\(p\)\|≤H\+​λlc\_\{p,\\varepsilon\}H\_\{\-\}\\lambda\_\{l\}\\leq\\left\|\\theta\_\{l,\\tau\}^\{\(p\)\}\\right\|\\leq H\_\{\+\}\\lambda\_\{l\}\. Squaring and summing overImI\_\{m\}yields

cp,ε2​H−2​∑l∈Imλl2≤∑l∈Im\|θl,τ\(p\)\|2≤H\+2​∑l∈Imλl2\.c\_\{p,\\varepsilon\}^\{2\}H\_\{\-\}^\{2\}\\sum\_\{l\\in I\_\{m\}\}\\lambda\_\{l\}^\{2\}\\leq\\sum\_\{l\\in I\_\{m\}\}\\left\|\\theta\_\{l,\\tau\}^\{\(p\)\}\\right\|^\{2\}\\leq H\_\{\+\}^\{2\}\\sum\_\{l\\in I\_\{m\}\}\\lambda\_\{l\}^\{2\}\.Thus, the first assertion holds withC−=cp,ε2​H−2C\_\{\-\}=c\_\{p,\\varepsilon\}^\{2\}H\_\{\-\}^\{2\}andC\+=H\+2C\_\{\+\}=H\_\{\+\}^\{2\}\. For the layerwise square\-root scheduleλl=l−1/2\\lambda\_\{l\}=l^\{\-1/2\}, we have

∑l=m2​m−1λl2=∑l=m2​m−11l\.\\sum\_\{l=m\}^\{2m\-1\}\\lambda\_\{l\}^\{2\}=\\sum\_\{l=m\}^\{2m\-1\}\\frac\{1\}\{l\}\.Sincem≤l≤2​m−1m\\leq l\\leq 2m\-1for everyl∈Iml\\in I\_\{m\}, we have12​m−1≤1l≤1m\\frac\{1\}\{2m\-1\}\\leq\\frac\{1\}\{l\}\\leq\\frac\{1\}\{m\}\. SinceImI\_\{m\}contains exactlymmindices, it follows thatm2​m−1≤∑l=m2​m−11l≤1\\frac\{m\}\{2m\-1\}\\leq\\sum\_\{l=m\}^\{2m\-1\}\\frac\{1\}\{l\}\\leq 1\. Hence,

∑l=m2​m−1λl2=Θ​\(1\)\.\\sum\_\{l=m\}^\{2m\-1\}\\lambda\_\{l\}^\{2\}=\\Theta\(1\)\.
For the harmonic scheduleλl=l−1\\lambda\_\{l\}=l^\{\-1\}, we have∑l=m2​m−1λl2=∑l=m2​m−11l2\\sum\_\{l=m\}^\{2m\-1\}\\lambda\_\{l\}^\{2\}=\\sum\_\{l=m\}^\{2m\-1\}\\frac\{1\}\{l^\{2\}\}\. Using againm≤l≤2​m−1m\\leq l\\leq 2m\-1, we obtain1\(2​m−1\)2≤1l2≤1m2\\frac\{1\}\{\(2m\-1\)^\{2\}\}\\leq\\frac\{1\}\{l^\{2\}\}\\leq\\frac\{1\}\{m^\{2\}\}\. Therefore,

m\(2​m−1\)2≤∑l=m2​m−11l2≤1m\.\\frac\{m\}\{\(2m\-1\)^\{2\}\}\\leq\\sum\_\{l=m\}^\{2m\-1\}\\frac\{1\}\{l^\{2\}\}\\leq\\frac\{1\}\{m\}\.Sincem\(2​m−1\)2≥14​m\\frac\{m\}\{\(2m\-1\)^\{2\}\}\\geq\\frac\{1\}\{4m\}, we conclude that

14​m≤∑l=m2​m−1λl2≤1m\.\\frac\{1\}\{4m\}\\leq\\sum\_\{l=m\}^\{2m\-1\}\\lambda\_\{l\}^\{2\}\\leq\\frac\{1\}\{m\}\.Hence,

∑l=m2​m−1λl2=Θ​\(1m\)\.\\sum\_\{l=m\}^\{2m\-1\}\\lambda\_\{l\}^\{2\}=\\Theta\\left\(\\frac\{1\}\{m\}\\right\)\.The stated conclusions follow\. ∎

Proposition[10](https://arxiv.org/html/2608.02668#Thmtheorem10)distinguishes the distribution of squared angular motion from the global worst\-case stability envelope\. The harmonic schedule gives the smaller global accumulated\-sensitivity bound, but allocates progressively less squared angular motion to deeper dyadic update intervals\. By contrast, the layerwise square\-root schedule retains an approximately constant squared angular budget across logarithmic scales of update depth, allowing the transformations associated with deeper SpheretNorm updates to remain nontrivial\.