Non-asymptotic implicit bias of logistic regression at early-stage gradient descent dynamics

# Non-asymptotic implicit bias of logistic regression at early-stage gradient descent dynamics
Source: [https://arxiv.org/html/2608.04382](https://arxiv.org/html/2608.04382)
###### Abstract

Gradient descent has been of particular interest in modern machine learning beyond sole focus on optimization\. Implicit bias emerging from optimization, though not being encoded by the learning objective, often prevents from overfitting to spurious patterns\. A typical instance is the max\-margin implicit bias of a linear classifier, widely established for exponentially tailed loss functions\. Even after having a given dataset separated, the parameter vector continues to evolve towards the max\-margin direction asymptotically along the gradient descent dynamics\. This phenomenon corroborates a frequent empirical observation of “train longer, generalize better\.” However, the max\-margin convergence is an asymptotic phenomenon, and what is worse, this asymptotic convergence rate is significantly slower than pure convex optimization\. Even so, the parameter vector along gradient descent dynamics commonly correlates with the max\-margin direction positively \(though not exactly\) within considerably fewer iterations than the asymptotic rate\. By shedding another light on this classical problem, this work aims to understand the mechanism of this early\-stage alignment phenomenon\. Our theoretical results demonstrate that the parameter vector weakly aligns with the max\-margin direction withinO​\(exp⁡\(exp⁡\(−δ\)\)\)O\(\\exp\(\\exp\(\-\\delta\)\)\)iterations, whereδ\>0\\delta\>0is the permissible alignment error, which is shown to be tight\. By tracking the radial and tangential flows, our proof operates on the alignment dynamics directly with dataset geometry and gets rid of the asymptotic expansion, which is a key insight to establishing faster weak alignment\.

## 1Introduction

We consider gradient descent with a binary linear classifierf𝐰:𝐱↦⟨𝐰,𝐱⟩f\_\{\\mathbf\{w\}\}\\colon\\mathbf\{x\}\\mapsto\\left\\langle\{\\mathbf\{w\}\},\{\\mathbf\{x\}\}\\right\\rangle, defined by a parameter vector𝐰∈ℝd\\mathbf\{w\}\\in\\mathbb\{R\}^\{d\}\. Letℓ:ℝ→ℝ≥0\\ell:\\mathbb\{R\}\\to\\mathbb\{R\}\_\{\\geq 0\}be a convex nonincreasing loss function, typically the logistic or exponential losses, measuring the margin at a given input\. By introducing the training riskL​\(𝐰\)≔n−1​∑i=1nℓ​\(⟨𝐰,yi​𝐱i⟩\)L\(\\mathbf\{w\}\)\\coloneqq n^\{\-1\}\\sum\_\{i=1\}^\{n\}\\ell\(\\left\\langle\{\\mathbf\{w\}\},\{y\_\{i\}\\mathbf\{x\}\_\{i\}\}\\right\\rangle\)over a training dataset𝒮≔\{\(𝐱i,yi\)\}i=1n\\mathcal\{S\}\\coloneqq\\\{\(\\mathbf\{x\}\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{n\}, gradient descent recursively operates by

𝐰​\(t\+1\)≔𝐰​\(t\)−η​∇L​\(𝐰​\(t\)\),fort=0,1,2,…,\\mathbf\{w\}\(\{t\+1\}\)\\coloneqq\\mathbf\{w\}\(t\)\-\\eta\\nabla L\(\\mathbf\{w\}\(t\)\),\\quad\\text\{for $t=0,1,2,\\dots$,\}whereη\>0\\eta\>0is a fixed stepsize\. During the last decade, it has been shown that gradient descent asymptotically leads the parameter direction𝐰​\(t\)/‖𝐰​\(t\)‖\\mathbf\{w\}\(t\)/\\\|\\mathbf\{w\}\(t\)\\\|to the max\-margin direction𝐮∗∈𝕊d−1\\mathbf\{u\}\_\{\*\}\\in\\mathbb\{S\}^\{d\-1\}of the dataset𝒮\\mathcal\{S\}\. This is known as the*implicit bias*of gradient descent, which means that the max\-margin convergence takes place even though the risk minimization problem does not apparently encode it\. The max\-margin implicit bias has been proven bySoudry et al\. \[[28](https://arxiv.org/html/2608.04382#bib.bib28)\]under the linear separability condition, and extended to non\-separable scenarios\[[12](https://arxiv.org/html/2608.04382#bib.bib12)\]\. The max\-margin convergence has implications on test performance through margin\-based generalization bounds\[[15](https://arxiv.org/html/2608.04382#bib.bib15),[27](https://arxiv.org/html/2608.04382#bib.bib27),[26](https://arxiv.org/html/2608.04382#bib.bib26)\]and robustness to input perturbation\[[36](https://arxiv.org/html/2608.04382#bib.bib36)\], rendering it practically relevant\. However, the asymptotic alignment is notoriously slow, at a rate ofO~​\(1/log2⁡t\)\\widetilde\{O\}\(1/\\log^\{2\}t\), which is unsatisfactory when compared with the standardO​\(1/t\)O\(1/t\)optimization convergence rate of gradient descent on convex smooth functions\. This slow parameter alignment cannot be improved unless an aggressively large stepsize is used\[[20](https://arxiv.org/html/2608.04382#bib.bib20)\], approaching normalized gradient descent\. Yet, can we expect faster parameter alignment for gradient descent in any sense?

![Refer to caption](https://arxiv.org/html/2608.04382v1/x1.png)

![Refer to caption](https://arxiv.org/html/2608.04382v1/x2.png)

Figure 1:Simulation of gradient descent running with the exponential lossℓexp​\(m\)≔exp⁡\(−m\)\\ell\_\{\\text\{exp\}\}\(m\)\\coloneqq\\exp\(\-m\):\(Left\)a training dataset with larger margin;\(Right\)a training dataset with smaller margin\. Each shows time evolution of the training riskL​\(𝐰​\(t\)\)L\(\\mathbf\{w\}\(t\)\), the radial‖𝐰​\(t\)‖\\\|\\mathbf\{w\}\(t\)\\\|, the parameter alignment⟨𝐮​\(t\),𝐮∗⟩\\left\\langle\{\\mathbf\{u\}\(t\)\},\{\\mathbf\{u\}\_\{\*\}\}\\right\\rangle\(where𝐮​\(t\)≔𝐰​\(t\)/‖𝐰​\(t\)‖\\mathbf\{u\}\(t\)\\coloneqq\\mathbf\{w\}\(t\)/\\\|\\mathbf\{w\}\(t\)\\\|\), and the tangential trajectory \(i\.e\., the trajectory of the normalized parameter𝐮​\(t\)\\mathbf\{u\}\(t\)\) in the loss landscape\. In the loss landscapes, the dots are the tangential component𝐮​\(t\)\\mathbf\{u\}\(t\), and darker and lighter colors indicate earlier and latertt, respectively;𝐮​\(t\)\\mathbf\{u\}\(t\)are plotted every7070and5050iteration for the left and right cases, respectively; the black and yellow arrows indicate the max\-margin direction𝐮∗\\mathbf\{u\}\_\{\*\}and the last parameter𝐰​\(T\)\\mathbf\{w\}\(T\), respectively\. In Appendix[F](https://arxiv.org/html/2608.04382#A6), the unnormalized parameter trajectories𝐰​\(t\)\\mathbf\{w\}\(t\)are additionally shown, as well as the quantitative alignment time\. The detailed simulation setup is shown there, too\.Key observation:compared with the radial growth‖𝐰​\(t\)‖\\\|\\mathbf\{w\}\(t\)\\\|, the parameter alignment⟨𝐮​\(t\),𝐮∗⟩\\left\\langle\{\\mathbf\{u\}\(t\)\},\{\\mathbf\{u\}\_\{\*\}\}\\right\\ranglesaturates significantly faster, which can also be shown in the parameter trajectory since the tangential trajectory is swept within the early stage, depicted by the darker dots\.Consider the numerical simulation in Figure[1](https://arxiv.org/html/2608.04382#S1.F1), where gradient descent is run for two distinct training datasets, one of which has a large margin \(the left figure\) and the other has a smaller margin \(the right figure\)\. Denote𝐮​\(t\)≔𝐰​\(t\)/‖𝐰​\(t\)‖\\mathbf\{u\}\(t\)\\coloneqq\\mathbf\{w\}\(t\)/\\\|\\mathbf\{w\}\(t\)\\\|the tangential component of𝐰​\(t\)\\mathbf\{w\}\(t\)\. In either case, the radial component‖𝐰​\(t\)‖\\\|\\mathbf\{w\}\(t\)\\\|exhibits slow growth, yet the parameter alignment⟨𝐮​\(t\),𝐮∗⟩\\left\\langle\{\\mathbf\{u\}\(t\)\},\{\\mathbf\{u\}\_\{\*\}\}\\right\\ranglegrows much more rapidly\. This is additionally demonstrated by both parameter trajectories, where the tangential component𝐮​\(t\)\\mathbf\{u\}\(t\)is attracted to the max\-margin direction𝐮∗\\mathbf\{u\}\_\{\*\}\(the black arrow\) within a remarkably short period, shown by the darker blue dots\. Readers may also refer to the unnormalized parameter trajectories shown in Appendix[F](https://arxiv.org/html/2608.04382#A6)to clearly see this transient behavior\. Whereas the theoretical alignment rate is⟨𝐮​\(t\),𝐮∗⟩=1−O~​\(1/log2⁡t\)\\left\\langle\{\\mathbf\{u\}\(t\)\},\{\\mathbf\{u\}\_\{\*\}\}\\right\\rangle=1\-\\widetilde\{O\}\(1/\\log^\{2\}t\)\[[28](https://arxiv.org/html/2608.04382#bib.bib28)\], this asymptotic analysis may not capture well the early\-stage behavior\.

The aim of this paper is to characterize this early\-stage parameter dynamics, bringing forth a deeper understanding of the gradient descent path\. Specifically, our main result rigorously proves that it only takesO​\(exp⁡\(exp⁡\(−δ\)\)\)O\(\\exp\(\\exp\(\-\\delta\)\)\)time to achieve weak alignment⟨𝐮​\(t\),𝐮∗⟩≥1−δ\\left\\langle\{\\mathbf\{u\}\(t\)\},\{\\mathbf\{u\}\_\{\*\}\}\\right\\rangle\\geq 1\-\\delta\. This fast convergence is guaranteed only for moderately but not arbitrarily smallδ=O​\(1\)\\delta=O\(1\)—typically up toδ∼1−γ\\delta\\sim 1\-\\sqrt\{\\gamma\}depending on the dataset marginγ\\gamma, which is slightly and non\-trivially better than the classical perceptron guaranteeδ=1−γ\\delta=1\-\\gamma\[[21](https://arxiv.org/html/2608.04382#bib.bib21)\]\. In this sense, we call this*weak*alignment, contrasted with*perfect*alignment to guarantee the asymptotic closeness of𝐮​\(t\)\\mathbf\{u\}\(t\)to the max\-margin direction𝐮∗\\mathbf\{u\}\_\{\*\}\. We argue that weak alignment is the right notion and necessary compromise to characterize the fast early\-stage dynamics observed in Figure[1](https://arxiv.org/html/2608.04382#S1.F1), by showing the doubly\-exponential time is tight\.

### 1\.1Setup

We writeat≲bta\_\{t\}\\lesssim b\_\{t\}if there exists an absolute constantC\>0C\>0such thatat≤C​bta\_\{t\}\\leq Cb\_\{t\}for alltt\. In this paper,∥⋅∥\\\|\\cdot\\\|denotes theℓ2\\ell\_\{2\}\-norm unless otherwise noted\. The unit hypersphere embedded inℝd\\mathbb\{R\}^\{d\}is written as𝕊d−1\\mathbb\{S\}^\{d\-1\}\. The identity matrix is written asII\. The probability simplex is written asΔn−1≔\{𝐩∈ℝn∣⟨𝐩,𝟏⟩=1\}\\Delta^\{n\-1\}\\coloneqq\\\{\\mathbf\{p\}\\in\\mathbb\{R\}^\{n\}\\mid\\left\\langle\{\\mathbf\{p\}\},\{\\mathbf\{1\}\}\\right\\rangle=1\\\}, where𝟏≔\[1,1,…,1\]⊤∈ℝn\\mathbf\{1\}\\coloneqq\[1,1,\\dots,1\]^\{\\top\}\\in\\mathbb\{R\}^\{n\}\. DefineP𝐯⟂∈ℝd×dP\_\{\\mathbf\{v\}\}^\{\\perp\}\\in\\mathbb\{R\}^\{d\\times d\}the projection operator onto the orthogonal complement to the subspacespan⁡\(𝐯\)⊆ℝd\\operatorname\{\\mathrm\{span\}\}\(\\mathbf\{v\}\)\\subseteq\\mathbb\{R\}^\{d\}, namely,P𝐯⟂≔I−𝐯𝐯⊤P\_\{\\mathbf\{v\}\}^\{\\perp\}\\coloneqq I\-\\mathbf\{v\}\\mathbf\{v\}^\{\\top\}\. Similarly,P𝐯≔𝐯𝐯⊤P\_\{\\mathbf\{v\}\}\\coloneqq\\mathbf\{v\}\\mathbf\{v\}^\{\\top\}denotes the subspace projection ontospan⁡\(𝐯\)\\operatorname\{\\mathrm\{span\}\}\(\\mathbf\{v\}\)\. The training dataset𝒮≔\{\(𝐱i,yi\)\}i=1n⊆ℝd×\{±1\}\\mathcal\{S\}\\coloneqq\\\{\(\\mathbf\{x\}\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{n\}\\subseteq\\mathbb\{R\}^\{d\}\\times\\\{\\pm 1\\\}satisfies‖𝐱i‖≤1\\\|\\mathbf\{x\}\_\{i\}\\\|\\leq 1for eachi∈\[n\]i\\in\[n\]\. We consider linear classifiers trained with the unregularized empirical risk minimization:

min𝐰∈ℝd⁡L​\(𝐰\)≔1n​∑i=1nℓ​\(⟨𝐰,𝐳i⟩\),\\min\_\{\\mathbf\{w\}\\in\\mathbb\{R\}^\{d\}\}L\(\\mathbf\{w\}\)\\coloneqq\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\ell\(\\left\\langle\{\\mathbf\{w\}\},\{\\mathbf\{z\}\_\{i\}\}\\right\\rangle\),where𝐳i≔yi​𝐱i\\mathbf\{z\}\_\{i\}\\coloneqq y\_\{i\}\\mathbf\{x\}\_\{i\}\(satisfying‖𝐳i‖≤1\\\|\\mathbf\{z\}\_\{i\}\\\|\\leq 1\) andℓ:ℝ→ℝ≥0\\ell:\\mathbb\{R\}\\to\\mathbb\{R\}\_\{\\geq 0\}is a loss function\. The data points are represented by a matrixZ∈ℝd×nZ\\in\\mathbb\{R\}^\{d\\times n\}storing each𝐳i\\mathbf\{z\}\_\{i\}in the column\-wise order\. Throughout this paper, we focus on the exponential lossℓ​\(m\)≔e−m\\ell\(m\)\\coloneqq e^\{\-m\}to simplify the analysis; yet we believe that the extension to exponentially\-tailed loss functions is possible with more algebra\. The training risk is minimized via gradient descent𝐰​\(t\+1\)=𝐰​\(t\)−η​∇L​\(𝐰​\(t\)\)\\mathbf\{w\}\(t\+1\)=\\mathbf\{w\}\(t\)\-\\eta\\nabla L\(\\mathbf\{w\}\(t\)\)with a fixed stepsizeη\>0\\eta\>0throughout training\. If we fixttand there is no confusion for a moment, we occasionally drop thett\-dependency for notational simplicity\. For the dataset𝒮\\mathcal\{S\}, let us consider the max\-margin problem:

min𝐰∈ℝd⁡12​‖𝐰‖2​s\.t\.​⟨𝐰,𝐳i⟩≥1​fori∈\[n\],\\min\_\{\\mathbf\{w\}\\in\\mathbb\{R\}^\{d\}\}\\frac\{1\}\{2\}\\\|\\mathbf\{w\}\\\|^\{2\}\\text\{~~~s\.t\.~~~\}\\left\\langle\{\\mathbf\{w\}\},\{\\mathbf\{z\}\_\{i\}\}\\right\\rangle\\geq 1\\text\{~~~for $i\\in\[n\]$\},\(1\)and denote𝐰∗∈ℝd\\mathbf\{w\}\_\{\*\}\\in\\mathbb\{R\}^\{d\}the max\-margin solution to \([1](https://arxiv.org/html/2608.04382#S1.E1)\) and𝐮∗≔𝐰∗/‖𝐰∗‖∈𝕊d−1\\mathbf\{u\}\_\{\*\}\\coloneqq\\mathbf\{w\}\_\{\*\}/\\\|\\mathbf\{w\}\_\{\*\}\\\|\\in\\mathbb\{S\}^\{d\-1\}the max\-margin direction\. Correspondingly, we decompose the parameter𝐰​\(t\)=r​\(t\)⋅𝐮​\(t\)\\mathbf\{w\}\(t\)=r\(t\)\\cdot\\mathbf\{u\}\(t\)into the radial componentr​\(t\)≔‖𝐰​\(t\)‖r\(t\)\\coloneqq\\\|\\mathbf\{w\}\(t\)\\\|and tangential component𝐮​\(t\)≔𝐰​\(t\)/r​\(t\)∈𝕊d−1\\mathbf\{u\}\(t\)\\coloneqq\\mathbf\{w\}\(t\)/r\(t\)\\in\\mathbb\{S\}^\{d\-1\}\. We focus on the linearly separable case throughout the paper, where the max\-margin problem \([1](https://arxiv.org/html/2608.04382#S1.E1)\) has a feasible point and𝐮∗\\mathbf\{u\}\_\{\*\}gives its direction\.

###### Assumption 1\.

The dataset𝒮\\mathcal\{S\}is linearly separable with a positive marginγ\>0\\gamma\>0, namely, we have⟨𝐮∗,𝐳i⟩≥γ\\left\\langle\{\\mathbf\{u\}\_\{\*\}\},\{\\mathbf\{z\}\_\{i\}\}\\right\\rangle\\geq\\gammafor alli∈\[n\]i\\in\[n\]\.

Under the above setup, we focus on the tangential alignment:V​\(t\)≔1−⟨𝐮​\(t\),𝐮∗⟩∈\[0,2\]V\(t\)\\coloneqq 1\-\\left\\langle\{\\mathbf\{u\}\(t\)\},\{\\mathbf\{u\}\_\{\*\}\}\\right\\rangle\\in\[0,2\]\.

### 1\.2Related work

In the era of overparametrized models, classical statistical generalization is too pessimistic\[[37](https://arxiv.org/html/2608.04382#bib.bib37)\], and optimization and loss landscapes lead to better generalization by stabilizing learning trajectories in a long run\[[11](https://arxiv.org/html/2608.04382#bib.bib11),[17](https://arxiv.org/html/2608.04382#bib.bib17)\], which inspires the study of implicit bias, questioning which solution an optimizer eventually converges to\. Classically,Soudry et al\. \[[28](https://arxiv.org/html/2608.04382#bib.bib28)\]reveals under the same setup as Section[1\.1](https://arxiv.org/html/2608.04382#S1.SS1)that gradient descent dynamics asymptotically leads the linear parameter to the max\-margin direction induced by theℓ2\\ell\_\{2\}\-norm, which is a prototypical expression of implicit bias\. Indeed, a similar max\-margin implicit bias is classical for the regularization path\[[24](https://arxiv.org/html/2608.04382#bib.bib24)\]\. Yet the convergence rateV​\(t\)=O~​\(1/log2⁡t\)V\(t\)=\\widetilde\{O\}\(1/\\log^\{2\}t\)is shown to be tight, fundamentally limiting the max\-margin implicit bias as an asymptotic behavior\. This slow behavior motivates our work, and we review it in detail in Section[1\.3](https://arxiv.org/html/2608.04382#S1.SS3)\. The max\-margin implicit bias has been extended to non\-separable datasets\[[12](https://arxiv.org/html/2608.04382#bib.bib12)\], convolutional networks\[[10](https://arxiv.org/html/2608.04382#bib.bib10)\], general loss functions\[[9](https://arxiv.org/html/2608.04382#bib.bib9),[20](https://arxiv.org/html/2608.04382#bib.bib20),[14](https://arxiv.org/html/2608.04382#bib.bib14),[23](https://arxiv.org/html/2608.04382#bib.bib23)\], homogeneous networks\[[13](https://arxiv.org/html/2608.04382#bib.bib13),[18](https://arxiv.org/html/2608.04382#bib.bib18)\], and transformers\[[29](https://arxiv.org/html/2608.04382#bib.bib29)\]\. In contrast, implicit bias has been generalized for other optimizers, including Adam\[[38](https://arxiv.org/html/2608.04382#bib.bib38)\], AdamW\[[35](https://arxiv.org/html/2608.04382#bib.bib35)\], steepest descent\[[30](https://arxiv.org/html/2608.04382#bib.bib30)\], mirror descent\[[16](https://arxiv.org/html/2608.04382#bib.bib16)\], and spectral descent\[[7](https://arxiv.org/html/2608.04382#bib.bib7)\], which typically leads to the max\-margin direction with a different norm from∥⋅∥2\\\|\\cdot\\\|\_\{2\}\. In functional spaces, we can view implicit bias as linear spline interpolation from the variational perspective\[[25](https://arxiv.org/html/2608.04382#bib.bib25),[22](https://arxiv.org/html/2608.04382#bib.bib22),[2](https://arxiv.org/html/2608.04382#bib.bib2)\]\. Moreover, the implicit bias can be used to describe feature learning\[[33](https://arxiv.org/html/2608.04382#bib.bib33),[32](https://arxiv.org/html/2608.04382#bib.bib32)\], where different initialization leads to implicit regularization enhancing feature selection\.

The aforementioned studies on implicit bias are more or less “static” because their focus is an asymptotic solution of an optimizer\. To better control the training schedule, understanding of optimizer’s dynamical behaviors matters\. In feature\-learning scenarios, two\-stage dynamics are often observed: parameters seek signal directions in the first stage, and then the parameter norms grow to decrease the loss in the second stage\[[5](https://arxiv.org/html/2608.04382#bib.bib5),[4](https://arxiv.org/html/2608.04382#bib.bib4),[8](https://arxiv.org/html/2608.04382#bib.bib8)\]\. This line of papers carefully characterizes the second\-stage behavior to establish benign overfitting for two\-layer neural networks; nevertheless, training points are already perfectly classified in the initial alignment stage\. Less is known for linear classifiers, while we empirically observe a clear early\-stage behavior\. Recently, gradient descent dynamics of linear classifiers has been revisited with a large stepsize, where we observe a two\-stage behavior even in the linear case\[[34](https://arxiv.org/html/2608.04382#bib.bib34),[19](https://arxiv.org/html/2608.04382#bib.bib19),[31](https://arxiv.org/html/2608.04382#bib.bib31),[3](https://arxiv.org/html/2608.04382#bib.bib3)\]\. Therein the dynamics exhibits a two\-stage behavior, the initial oscillation stage seeking an approximate max\-margin direction, and the late stabilization stage to converge asymptotically\. Our analysis is partially inspired by the two\-stage behaviors of linear models with a large stepsize\. More discussion is deferred to Section[5](https://arxiv.org/html/2608.04382#S5)\.

### 1\.3Review of asymptotic gradient descent dynamics

We review the implicit bias result bySoudry et al\. \[[28](https://arxiv.org/html/2608.04382#bib.bib28)\]in this section to describe how the gradient descent dynamics exhibits an asymptotically slow motion\. Consider the setup as in Section[1\.1](https://arxiv.org/html/2608.04382#S1.SS1)with the linear separability \(Assumption[1](https://arxiv.org/html/2608.04382#Thmtheorem1)\)\.

###### Theorem 2\(Soudry et al\. \[[28](https://arxiv.org/html/2608.04382#bib.bib28)\]\)\.

For any stepsizeη<1/L​\(𝐰​\(0\)\)\\eta<1/L\(\\mathbf\{w\}\(0\)\)and initialization𝐰​\(0\)\\mathbf\{w\}\(0\), the tangential parameter component𝐮​\(t\)\\mathbf\{u\}\(t\)following the gradient descent dynamics converges to the max\-margin direction𝐮∗\\mathbf\{u\}\_\{\*\}defined via \([1](https://arxiv.org/html/2608.04382#S1.E1)\) asymptotically\. The asymptotic convergence rates are given as follows:

\{Risk convergence:L​\(𝐰​\(t\)\)=O​\(1/t\);Radial divergence:r​\(t\)=Θ​\(log⁡t\);Tangential convergence:V​\(t\)=O~​\(1/log2⁡t\)\.\\begin\{cases\}\\text\{Risk convergence:\}&L\(\\mathbf\{w\}\(t\)\)=O\(1/t\);\\\\ \\text\{Radial divergence:\}&r\(t\)=\\Theta\(\\log t\);\\\\ \\text\{Tangential convergence:\}&V\(t\)=\\widetilde\{O\}\(1/\\log^\{2\}t\)\.\\end\{cases\}

Whereas the risk converges reasonably inO​\(1/t\)O\(1/t\)as in typical convex smooth \(but not strongly convex\) optimization, the tangential component𝐮​\(t\)\\mathbf\{u\}\(t\)converges to the max\-margin direction𝐮∗\\mathbf\{u\}\_\{\*\}very slowly\. The polylogarithmic rate is translated into the asymptotic tangential convergence timet=T∞=O~​\(exp⁡\(δ−1/2\)\)t=T\_\{\\infty\}=\\widetilde\{O\}\(\\exp\(\\delta^\{\-1/2\}\)\)\.

Here we briefly illustrate the proof sketch of the convergence rates to highlight obstacles that we need to overcome\. The risk convergence and radial divergence are easy to see from our structural results in Lemma[4](https://arxiv.org/html/2608.04382#Thmtheorem4), which we show later in Section[2](https://arxiv.org/html/2608.04382#S2)\. Suppose that we have proven them and focus on the tangential convergence\. Given the asymptotic convergence𝐮​\(t\)→𝐮∗\\mathbf\{u\}\(t\)\\to\\mathbf\{u\}\_\{\*\}and the radial divergence rate, we can write𝐰​\(t\)=𝐮∗​log⁡t\+𝝃​\(t\)\\mathbf\{w\}\(t\)=\\mathbf\{u\}\_\{\*\}\\log t\+\\bm\{\\xi\}\(t\)with a vanishing remainder‖𝝃​\(t\)‖=o​\(log⁡t\)\\\|\\bm\{\\xi\}\(t\)\\\|=o\(\\log t\)\. Then the tangential component𝐮=𝐰/‖𝐰‖\\mathbf\{u\}=\\mathbf\{w\}/\\\|\\mathbf\{w\}\\\|admits the following asymptotic expansion:

𝐮=𝐰‖𝐰‖\\displaystyle\\mathbf\{u\}=\\frac\{\\mathbf\{w\}\}\{\\\|\\mathbf\{w\}\\\|\}=𝐮∗\+𝝃⋅\(log⁡t\)−11\+2​⟨𝐮∗,𝝃⟩⋅\(log⁡t\)−1\+‖𝝃‖2​\(log⁡t\)−2\\displaystyle=\\frac\{\\mathbf\{u\}\_\{\*\}\+\\bm\{\\xi\}\\cdot\(\\log t\)^\{\-1\}\}\{\\sqrt\{1\+2\\left\\langle\{\\mathbf\{u\}\_\{\*\}\},\{\\bm\{\\xi\}\}\\right\\rangle\\cdot\(\\log t\)^\{\-1\}\+\\\|\\bm\{\\xi\}\\\|^\{2\}\(\\log t\)^\{\-2\}\}\}=\(𝐮∗\+𝝃log⁡t\)​\[1−⟨𝐮∗,𝝃⟩log⁡t\+3​⟨𝐮∗,𝝃⟩2−‖𝝃‖22​log2⁡t−O​\(1log3⁡t\)\]\\displaystyle=\\left\(\\mathbf\{u\}\_\{\*\}\+\\frac\{\\bm\{\\xi\}\}\{\\log t\}\\right\)\\left\[1\-\\frac\{\\left\\langle\{\\mathbf\{u\}\_\{\*\}\},\{\\bm\{\\xi\}\}\\right\\rangle\}\{\\log t\}\+\\frac\{3\\left\\langle\{\\mathbf\{u\}\_\{\*\}\},\{\\bm\{\\xi\}\}\\right\\rangle^\{2\}\-\\\|\\bm\{\\xi\}\\\|^\{2\}\}\{2\\log^\{2\}t\}\-O\\left\(\\frac\{1\}\{\\log^\{3\}t\}\\right\)\\right\]=𝐮∗\+P𝐮∗⟂​𝝃⋅\(log⁡t\)−1−O​\(1/log2⁡t\),\\displaystyle=\\mathbf\{u\}\_\{\*\}\+P\_\{\\mathbf\{u\}\_\{\*\}\}^\{\\perp\}\\bm\{\\xi\}\\cdot\(\\log t\)^\{\-1\}\-O\(1/\\log^\{2\}t\),where we used the Taylor expansion11\+x=1−12​x\+38​x2\+O​\(x3\)\\frac\{1\}\{\\sqrt\{1\+x\}\}=1\-\\frac\{1\}\{2\}x\+\\frac\{3\}\{8\}x^\{2\}\+O\(x^\{3\}\)\. By noting⟨𝐮∗,P𝐮∗⟂​𝝃⟩=0\\langle\{\\mathbf\{u\}\_\{\*\}\},\{P\_\{\\mathbf\{u\}\_\{\*\}\}^\{\\perp\}\\bm\{\\xi\}\}\\rangle=0, we have⟨𝐮,𝐮∗⟩=1−O​\(1/log2⁡t\)\\left\\langle\{\\mathbf\{u\}\},\{\\mathbf\{u\}\_\{\*\}\}\\right\\rangle=1\-O\(1/\\log^\{2\}t\), where an additional polylogarithmic factor may arise under a pathological case\. Thus, we verify the tangential convergence rateV​\(t\)=O~​\(1/log2⁡t\)V\(t\)=\\widetilde\{O\}\(1/\\log^\{2\}t\)\.

While being tight in largett, a caveat of this analysis is the application of the asymptotic expansion, and thus non\-asymptotic factors are hidden in the remainderO​\(1/log2⁡t\)O\(1/\\log^\{2\}t\)\. Our subsequent analysis directly operates on the tangential alignmentV​\(t\)V\(t\), instead of asymptotically expanding it\.

## 2Structural properties on radial and tangential flows

First, we consider the gradient flow𝐰˙​\(t\)=−∇L​\(𝐰​\(t\)\)\\dot\{\\mathbf\{w\}\}\(t\)=\-\\nabla L\(\\mathbf\{w\}\(t\)\)to focus on the fundamental structure of the dynamics, where we use the samettto denote time instead of discrete steps with an abuse of notation\. The dynamics decomposition \(Lemma[3](https://arxiv.org/html/2608.04382#Thmtheorem3)\) and radial bounds \(Lemma[4](https://arxiv.org/html/2608.04382#Thmtheorem4)\) are not technically novel, but we show their concise proofs to let readers better understand the preliminary\. The discrete\-time dynamics is analyzed later in Section[4](https://arxiv.org/html/2608.04382#S4)by controlling an extra discretization error via the stepsize choice\. We initialize the parameter with‖𝐰​\(0\)‖=ρ\\\|\\mathbf\{w\}\(0\)\\\|=\\rhofor someρ\>γ−2​log⁡L​\(0\)\\rho\>\\gamma^\{\-2\}\\log L\(0\)\.111This condition onρ\\rhoensures that the polar decomposition exists:𝐮​\(0\)\\mathbf\{u\}\(0\)is well\-defined, andr​\(t\)\>0r\(t\)\>0holds acrosst\>0t\>0, which is indirectly shown in the proof of Theorem[10](https://arxiv.org/html/2608.04382#Thmtheorem10)—the radial flow satisfiesr​\(t\)\>r¯=r​\(T0\)r\(t\)\>\\underline\{r\}=r\(T\_\{0\}\)therein\.Let us decompose the parameter dynamics into the radial and tangential components\.

###### Lemma 3\(Radial/tangential flows\)\.

Fort≥0t\\geq 0, the gradient flow𝐰˙​\(t\)=−∇L​\(𝐰​\(t\)\)\\dot\{\\mathbf\{w\}\}\(t\)=\-\\nabla L\(\\mathbf\{w\}\(t\)\)can be decomposed as follows:

r˙​\(t\)=−⟨𝐮​\(t\),∇L​\(𝐰​\(t\)\)⟩,𝐮˙​\(t\)=−1r​\(t\)​P𝐮​\(t\)⟂​∇L​\(𝐰​\(t\)\)\.\\dot\{r\}\(t\)=\-\\left\\langle\{\\mathbf\{u\}\(t\)\},\{\\nabla L\(\\mathbf\{w\}\(t\)\)\}\\right\\rangle,\\qquad\\dot\{\\mathbf\{u\}\}\(t\)=\-\\frac\{1\}\{r\(t\)\}P\_\{\\mathbf\{u\}\(t\)\}^\{\\perp\}\\nabla L\(\\mathbf\{w\}\(t\)\)\.\(2\)

###### Proof\.\.

We have by the product rule that𝐰˙=r˙​𝐮\+r​𝐮˙\\dot\{\\mathbf\{w\}\}=\\dot\{r\}\\mathbf\{u\}\+r\\dot\{\\mathbf\{u\}\}\. Taking the inner product with𝐮\\mathbf\{u\}on both sides and using‖𝐮‖=1\\\|\\mathbf\{u\}\\\|=1and⟨𝐮,𝐮˙⟩=0\\left\\langle\{\\mathbf\{u\}\},\{\\dot\{\\mathbf\{u\}\}\}\\right\\rangle=0\(which follows from differentiating‖𝐮‖2=1\\\|\\mathbf\{u\}\\\|^\{2\}=1\), we obtain

⟨𝐮,𝐰˙⟩=r˙​⟨𝐮,𝐮⟩\+r​⟨𝐮,𝐮˙⟩=r˙\.\\left\\langle\{\\mathbf\{u\}\},\{\\dot\{\\mathbf\{w\}\}\}\\right\\rangle=\\dot\{r\}\\left\\langle\{\\mathbf\{u\}\},\{\\mathbf\{u\}\}\\right\\rangle\+r\\left\\langle\{\\mathbf\{u\}\},\{\\dot\{\\mathbf\{u\}\}\}\\right\\rangle=\\dot\{r\}\.Since𝐰˙=−∇L​\(𝐰\)\\dot\{\\mathbf\{w\}\}=\-\\nabla L\(\\mathbf\{w\}\), this yieldsr˙=−⟨𝐮,∇L​\(𝐰\)⟩\\dot\{r\}=\-\\left\\langle\{\\mathbf\{u\}\},\{\\nabla L\(\\mathbf\{w\}\)\}\\right\\rangle\.

For the tangential flow, we rearrange the product rule to get

𝐮˙=1r​\(𝐰˙−r˙​𝐮\)=1r​\[−∇L​\(𝐰\)\+⟨𝐮,∇L​\(𝐰\)⟩​𝐮\]=−1r​\(I−𝐮𝐮⊤\)​∇L​\(𝐰\)=−1r​P𝐮⟂​∇L​\(𝐰\),\\dot\{\\mathbf\{u\}\}=\\frac\{1\}\{r\}\\bigl\(\\dot\{\\mathbf\{w\}\}\-\\dot\{r\}\\mathbf\{u\}\\bigr\)=\\frac\{1\}\{r\}\\bigl\[\-\\nabla L\(\\mathbf\{w\}\)\+\\left\\langle\{\\mathbf\{u\}\},\{\\nabla L\(\\mathbf\{w\}\)\}\\right\\rangle\\mathbf\{u\}\\bigr\]=\-\\frac\{1\}\{r\}\\bigl\(I\-\\mathbf\{u\}\\mathbf\{u\}^\{\\top\}\\bigr\)\\nabla L\(\\mathbf\{w\}\)=\-\\frac\{1\}\{r\}P\_\{\\mathbf\{u\}\}^\{\\perp\}\\nabla L\(\\mathbf\{w\}\),where we used𝐮𝐮⊤​∇L​\(𝐰\)=⟨𝐮,∇L​\(𝐰\)⟩​𝐮\\mathbf\{u\}\\mathbf\{u\}^\{\\top\}\\nabla L\(\\mathbf\{w\}\)=\\left\\langle\{\\mathbf\{u\}\},\{\\nabla L\(\\mathbf\{w\}\)\}\\right\\rangle\\mathbf\{u\}\. ∎

Next, we provide upper and lower bounds onL​\(𝐰​\(t\)\)L\(\\mathbf\{w\}\(t\)\)andr​\(t\)r\(t\)\. Hereafter, we frequently use the exponential weighting𝜶​\(t\)∈Δn−1\\bm\{\\alpha\}\(t\)\\in\\Delta^\{n\-1\}, where each data pointi∈\[n\]i\\in\[n\]is weighted byαi​\(t\)≔e−⟨𝐰​\(t\),𝐳i⟩/∑je−⟨𝐰​\(t\),𝐳j⟩\\alpha\_\{i\}\(t\)\\coloneqq e^\{\-\\left\\langle\{\\mathbf\{w\}\(t\)\},\{\\mathbf\{z\}\_\{i\}\}\\right\\rangle\}/\\sum\_\{j\}e^\{\-\\left\\langle\{\\mathbf\{w\}\(t\)\},\{\\mathbf\{z\}\_\{j\}\}\\right\\rangle\}\.

###### Lemma 4\(Risk/radial bounds\)\.

Fort≥0t\\geq 0, we have the following bounds:

\{Risk estimates:1L​\(0\)−1\+t≤L​\(𝐰​\(t\)\)≤1L​\(0\)−1\+γ2​t,Radial estimates:ρ−1γ2​log⁡\(1\+L​\(0\)​γ2​t\)≤r​\(t\)≤ρ\+1γ2​log⁡\(1\+L​\(0\)​γ2​t\),Radial estimates \(refined\):log⁡\(L​\(0\)−1\+γ2​t\)≤r​\(t\)\.\\begin\{cases\}\\text\{Risk estimates:\}&\\frac\{1\}\{L\(0\)^\{\-1\}\+t\}\\leq L\(\\mathbf\{w\}\(t\)\)\\leq\\frac\{1\}\{L\(0\)^\{\-1\}\+\\gamma^\{2\}t\},\\\\ \\text\{Radial estimates:\}&\\rho\-\\frac\{1\}\{\\gamma^\{2\}\}\\log\(1\+L\(0\)\\gamma^\{2\}t\)\\leq r\(t\)\\leq\\rho\+\\frac\{1\}\{\\gamma^\{2\}\}\\log\(1\+L\(0\)\\gamma^\{2\}t\),\\\\ \\text\{Radial estimates \(refined\):\}&\\log\(L\(0\)^\{\-1\}\+\\gamma^\{2\}t\)\\leq r\(t\)\.\\end\{cases\}

###### Proof\.\.

We useL​\(t\)L\(t\)as the shorthand ofL​\(𝐰​\(t\)\)L\(\\mathbf\{w\}\(t\)\)\. First, we derive the upper bound ofL​\(t\)L\(t\)\. By expanding‖∇L​\(t\)‖\\\|\\nabla L\(t\)\\\|, we have

‖∇L​\(t\)‖=1n​‖∑i=1ne−⟨𝐰​\(t\),𝐳i⟩​𝐳i‖=L​\(t\)​‖∑i=1nαi​\(t\)​𝐳i‖=L​\(t\)​‖Z​𝜶​\(t\)‖∈\[γ​L​\(t\),L​\(t\)\],\\\|\\nabla L\(t\)\\\|=\\frac\{1\}\{n\}\\left\\\|\\sum\_\{i=1\}^\{n\}e^\{\-\\left\\langle\{\\mathbf\{w\}\(t\)\},\{\\mathbf\{z\}\_\{i\}\}\\right\\rangle\}\\mathbf\{z\}\_\{i\}\\right\\\|=L\(t\)\\left\\\|\\sum\_\{i=1\}^\{n\}\\alpha\_\{i\}\(t\)\\mathbf\{z\}\_\{i\}\\right\\\|=L\(t\)\\\|Z\\bm\{\\alpha\}\(t\)\\\|\\in\[\\gamma L\(t\),L\(t\)\],\(3\)whereγ≤\|⟨𝐮∗,Z​𝜶⟩\|≤‖Z​𝜶‖≤1\\gamma\\leq\|\\left\\langle\{\\mathbf\{u\}\_\{\*\}\},\{Z\\bm\{\\alpha\}\}\\right\\rangle\|\\leq\\\|Z\\bm\{\\alpha\}\\\|\\leq 1is used\. We often call the inequalityγ​L≤‖∇L‖≤L\\gamma L\\leq\\\|\\nabla L\\\|\\leq Lin \([3](https://arxiv.org/html/2608.04382#S2.E3)\) the*self\-bounding property*of a loss function\. Readers should keep this in mind since we repeatedly use it\. To derive the bounds ofL​\(t\)L\(t\), we evaluate its time derivative

L˙​\(t\)=−1n​∑i=1ne−⟨𝐰​\(t\),𝐳i⟩​⟨𝐰˙​\(t\),𝐳i⟩=−‖∇L​\(t\)‖2,\\dot\{L\}\(t\)=\-\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}e^\{\-\\left\\langle\{\\mathbf\{w\}\(t\)\},\{\\mathbf\{z\}\_\{i\}\}\\right\\rangle\}\\left\\langle\{\\dot\{\\mathbf\{w\}\}\(t\)\},\{\\mathbf\{z\}\_\{i\}\}\\right\\rangle=\-\\\|\\nabla L\(t\)\\\|^\{2\},which together with \([3](https://arxiv.org/html/2608.04382#S2.E3)\) yields the differential inequality−L​\(t\)2≤L˙​\(t\)≤−γ2​L​\(t\)2\-L\(t\)^\{2\}\\leq\\dot\{L\}\(t\)\\leq\-\\gamma^\{2\}L\(t\)^\{2\}\. After dividing all byL​\(t\)2L\(t\)^\{2\}and integrating from0tott, we obtain the bounds ofL​\(t\)L\(t\)\.

To derive the bounds ofr​\(t\)r\(t\), we expand the radial flow in \([2](https://arxiv.org/html/2608.04382#S2.E2)\) as follows:

r˙​\(t\)=−⟨𝐮​\(t\),∇L​\(t\)⟩=1n​∑i=1ne−⟨𝐰​\(t\),𝐳i⟩​⟨𝐮​\(t\),𝐳i⟩=L​\(t\)​⟨𝐮​\(t\),Z​𝜶​\(t\)⟩\.\\dot\{r\}\(t\)=\-\\left\\langle\{\\mathbf\{u\}\(t\)\},\{\\nabla L\(t\)\}\\right\\rangle=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}e^\{\-\\left\\langle\{\\mathbf\{w\}\(t\)\},\{\\mathbf\{z\}\_\{i\}\}\\right\\rangle\}\\left\\langle\{\\mathbf\{u\}\(t\)\},\{\\mathbf\{z\}\_\{i\}\}\\right\\rangle=L\(t\)\\left\\langle\{\\mathbf\{u\}\(t\)\},\{Z\\bm\{\\alpha\}\(t\)\}\\right\\rangle\.Then we have the differential inequality−L​\(t\)≤r˙​\(t\)≤L​\(t\)\-L\(t\)\\leq\\dot\{r\}\(t\)\\leq L\(t\), where the Cauchy–Schwarz inequality\|⟨𝐮,Z​𝜶⟩\|≤‖Z​𝜶‖≤1\|\\left\\langle\{\\mathbf\{u\}\},\{Z\\bm\{\\alpha\}\}\\right\\rangle\|\\leq\\\|Z\\bm\{\\alpha\}\\\|\\leq 1is used\. By pluggingL​\(t\)≤1L​\(0\)−1\+γ2​tL\(t\)\\leq\\frac\{1\}\{L\(0\)^\{\-1\}\+\\gamma^\{2\}t\}and integrating from0tott, we obtain the bounds ofr​\(t\)r\(t\)\.

Finally, we derive the last lower bound ofr​\(t\)r\(t\)\. Noting⟨𝐰​\(t\),𝐳i⟩=r​\(t\)​⟨𝐮​\(t\),𝐳i⟩≤r​\(t\)\\left\\langle\{\\mathbf\{w\}\(t\)\},\{\\mathbf\{z\}\_\{i\}\}\\right\\rangle=r\(t\)\\left\\langle\{\\mathbf\{u\}\(t\)\},\{\\mathbf\{z\}\_\{i\}\}\\right\\rangle\\leq r\(t\), we haveL​\(t\)≥e−r​\(t\)L\(t\)\\geq e^\{\-r\(t\)\}\. Combining withL​\(t\)≤1L​\(0\)−1\+γ2​tL\(t\)\\leq\\frac\{1\}\{L\(0\)^\{\-1\}\+\\gamma^\{2\}t\}, we haver​\(t\)≥log⁡\(L​\(0\)−1\+γ2​t\)r\(t\)\\geq\\log\(L\(0\)^\{\-1\}\+\\gamma^\{2\}t\)\. ∎

The radial estimater​\(t\)≥ρ−1γ2​log⁡\(1\+L​\(0\)​γ2​t\)r\(t\)\\geq\\rho\-\\frac\{1\}\{\\gamma^\{2\}\}\\log\(1\+L\(0\)\\gamma^\{2\}t\)is vacuous with largett, and the refined estimater​\(t\)≥log⁡\(L​\(0\)−1\+γ2​t\)r\(t\)\\geq\\log\(L\(0\)^\{\-1\}\+\\gamma^\{2\}t\)is then effective\. This is particularly used in the lower bound \(Theorem[10](https://arxiv.org/html/2608.04382#Thmtheorem10)\)\.

## 3Main result

To analyze the tangential alignment non\-asymptotically, let us derive the differential inequality:

V˙​\(t\)=1r​\(t\)​⟨∇L​\(𝐰​\(t\)\),P𝐮​\(t\)⟂​𝐮∗⟩=−L​\(𝐰​\(t\)\)r​\(t\)​⟨Z​𝜶​\(t\),P𝐮​\(t\)⟂​𝐮∗⟩,\\dot\{V\}\(t\)=\\frac\{1\}\{r\(t\)\}\\left\\langle\{\\nabla L\(\\mathbf\{w\}\(t\)\)\},\{P\_\{\\mathbf\{u\}\(t\)\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\}\\right\\rangle=\-\\frac\{L\(\\mathbf\{w\}\(t\)\)\}\{r\(t\)\}\\left\\langle\{Z\\bm\{\\alpha\}\(t\)\},\{P\_\{\\mathbf\{u\}\(t\)\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\}\\right\\rangle,\(4\)where we applied the tangential flow in \([2](https://arxiv.org/html/2608.04382#S2.E2)\) toV​\(t\)=1−⟨𝐮​\(t\),𝐮∗⟩V\(t\)=1\-\\left\\langle\{\\mathbf\{u\}\(t\)\},\{\\mathbf\{u\}\_\{\*\}\}\\right\\rangle\. On the right\-hand side of \([4](https://arxiv.org/html/2608.04382#S3.E4)\),L​\(𝐰​\(t\)\)L\(\\mathbf\{w\}\(t\)\)andr​\(t\)r\(t\)can be easily handled by using Lemma[4](https://arxiv.org/html/2608.04382#Thmtheorem4), yet the term⟨Z​𝜶,P𝐮⟂​𝐮∗⟩\\left\\langle\{Z\\bm\{\\alpha\}\},\{P\_\{\\mathbf\{u\}\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\}\\right\\ranglerequires more convoluted treatment\. Nonetheless,*we can make an ansatz⟨Z​𝛂,P𝐮⟂​𝐮∗⟩\>κ\\left\\langle\{Z\\bm\{\\alpha\}\},\{P\_\{\\mathbf\{u\}\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\}\\right\\rangle\>\\kappafor someκ\>0\\kappa\>0*, which we rigorously justify later in Lemma[6](https://arxiv.org/html/2608.04382#Thmtheorem6)\. The geometric intuition behind this ansatz is as follows—the exponentially\-weighted dataZ​𝜶Z\\bm\{\\alpha\}should positively correlate with𝐮∗\\mathbf\{u\}\_\{\*\}\. This correlation persists even withP𝐮⟂​𝐮∗P\_\{\\mathbf\{u\}\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}, the projected𝐮∗\\mathbf\{u\}\_\{\*\}ontospan\(𝐮\)⟂\\operatorname\{\\mathrm\{span\}\}\(\\mathbf\{u\}\)^\{\\perp\}, unless𝐮\\mathbf\{u\}and𝐮∗\\mathbf\{u\}\_\{\*\}are nearly antipodal\. With this ansatz, the non\-asymptotic alignment is relatively easy to establish\.

###### Lemma 5\(Weak alignment under ansatz\)\.

For an absolute constantκ\>0\\kappa\>0, assume that the following uniform bound is valid fort∈\[0,T1\]t\\in\[0,T\_\{1\}\]:

⟨Z​𝜶​\(t\),P𝐮​\(t\)⟂​𝐮∗⟩≥κ\.\\left\\langle\{Z\\bm\{\\alpha\}\(t\)\},\{P\_\{\\mathbf\{u\}\(t\)\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\}\\right\\rangle\\geq\\kappa\.\(5\)Then, fort∈\[0,T1\]t\\in\[0,T\_\{1\}\], we have

V​\(t\)≤V​\(0\)−κ​γ2\(1\+γ2\)​\(ρ​γ2\+log⁡2\)−κ​γ22​log⁡\(1\+log⁡\(L​\(0\)​γ2​t\)ρ​γ2\+log⁡2\)\.V\(t\)\\leq V\(0\)\-\\frac\{\\kappa\\gamma^\{2\}\}\{\(1\+\\gamma^\{2\}\)\(\\rho\\gamma^\{2\}\+\\log 2\)\}\-\\frac\{\\kappa\\gamma^\{2\}\}\{2\}\\log\\left\(1\+\\frac\{\\log\(L\(0\)\\gamma^\{2\}t\)\}\{\\rho\\gamma^\{2\}\+\\log 2\}\\right\)\.\(6\)In this case,V​\(t\)≤δV\(t\)\\leq\\deltais attained afterT1=O​\(exp⁡\(exp⁡\(−δ/κ​γ2\)\)\)T\_\{1\}=O\(\\exp\(\\exp\(\-\\delta/\\kappa\\gamma^\{2\}\)\)\)time\.

###### Proof\.\.

Plugging \([5](https://arxiv.org/html/2608.04382#S3.E5)\) together with Lemma[4](https://arxiv.org/html/2608.04382#Thmtheorem4)into the differential equation \([4](https://arxiv.org/html/2608.04382#S3.E4)\), we have

V˙​\(t\)≤−κ​γ2\(L​\(0\)−1\+t\)​\[ρ​γ2\+log⁡\(1\+γ2​L​\(0\)​t\)\]for0≤t≤T1\.\\dot\{V\}\(t\)\\leq\-\\frac\{\\kappa\\gamma^\{2\}\}\{\(L\(0\)^\{\-1\}\+t\)\[\\rho\\gamma^\{2\}\+\\log\(1\+\\gamma^\{2\}L\(0\)t\)\]\}\\quad\\text\{for $0\\leq t\\leq T\_\{1\}$\.\}This differential inequality is elementary and can be solved via Lemma[15](https://arxiv.org/html/2608.04382#Thmtheorem15)\(in Appendix[B](https://arxiv.org/html/2608.04382#A2)\) witha≔L​\(0\)−1a\\coloneqq L\(0\)^\{\-1\},b≔ρ​γ2b\\coloneqq\\rho\\gamma^\{2\}, andc≔γ2​L​\(0\)c\\coloneqq\\gamma^\{2\}L\(0\)\. Then, we have the desired upper bound \([6](https://arxiv.org/html/2608.04382#S3.E6)\)\. Solving\(upper bound of \([6](https://arxiv.org/html/2608.04382#S3.E6)\)\)≤δ\\text\{\(upper bound of \\eqref\{equation:v\_ub\}\)\}\\leq\\deltawith some relaxation by dropping the second term \(which is negative\) leads to

t≥1L​\(0\)​γ2​exp⁡\{\(ρ​γ2\+log⁡2\)​\[exp⁡\(2​\(V​\(0\)−δ\)κ​γ2\)−1\]\}\.t\\geq\\frac\{1\}\{L\(0\)\\gamma^\{2\}\}\\exp\\left\\\{\(\\rho\\gamma^\{2\}\+\\log 2\)\\left\[\\exp\\left\(\\frac\{2\(V\(0\)\-\\delta\)\}\{\\kappa\\gamma^\{2\}\}\\right\)\-1\\right\]\\right\\\}\.HenceT1=O​\(exp⁡\(exp⁡\(−δ/κ​γ2\)\)\)T\_\{1\}=O\(\\exp\(\\exp\(\-\\delta/\\kappa\\gamma^\{2\}\)\)\)is sufficient time for achieving weak alignmentV​\(t\)≤δV\(t\)\\leq\\delta\. ∎

Lemma[5](https://arxiv.org/html/2608.04382#Thmtheorem5)already exhibits the non\-asymptotic tangential convergenceV​\(t\)≲C−log⁡log⁡tV\(t\)\\lesssim C\-\\log\\log t\. Or equivalently,V​\(t\)≤δV\(t\)\\leq\\deltais finitely achieved fort=O​\(exp⁡\(exp⁡\(−δ/κ​γ2\)\)\)t=O\(\\exp\(\\exp\(\-\\delta/\\kappa\\gamma^\{2\}\)\)\)\. This doubly\-logarithmic exponential convergence is much faster than the asymptotic convergence rateO~​\(1/log2⁡t\)\\widetilde\{O\}\(1/\\log^\{2\}t\)in Theorem[2](https://arxiv.org/html/2608.04382#Thmtheorem2)\. The explicit rate depends on the absolute constantκ\\kappa\.

Subsequently, we need to verify the uniform bound \([5](https://arxiv.org/html/2608.04382#S3.E5)\)\. We can establish the following geometric lemma to show a lower bound of form \([5](https://arxiv.org/html/2608.04382#S3.E5)\) \(but conditioned onV≤1V\\leq 1\)\. Note that this uniform bound is purely geometric, so we can readily use it for both continuous\-time and discrete\-time cases\.

![Refer to caption](https://arxiv.org/html/2608.04382v1/x3.png)

![Refer to caption](https://arxiv.org/html/2608.04382v1/x4.png)

Figure 2:The geometry behind the proof of Lemma[6](https://arxiv.org/html/2608.04382#Thmtheorem6),⟨Z​𝜶,P𝐮⟂​𝐮∗⟩≥κ0​\(V\)\\left\\langle\{Z\\bm\{\\alpha\}\},\{P\_\{\\mathbf\{u\}\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\}\\right\\rangle\\geq\\kappa\_\{0\}\(V\)\.\(Left\)In the first step, we decomposeP𝐮⟂​𝐮∗=𝐮∗−P𝐮​𝐮∗P\_\{\\mathbf\{u\}\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}=\\mathbf\{u\}\_\{\*\}\-P\_\{\\mathbf\{u\}\}\\mathbf\{u\}\_\{\*\}and derive bounds for⟨𝐮∗,Z​𝜶⟩\\left\\langle\{\\mathbf\{u\}\_\{\*\}\},\{Z\\bm\{\\alpha\}\}\\right\\rangleand⟨P𝐮​𝐮∗,Z​𝜶⟩\\left\\langle\{P\_\{\\mathbf\{u\}\}\\mathbf\{u\}\_\{\*\}\},\{Z\\bm\{\\alpha\}\}\\right\\rangleindependently\.\(Right\)In the second step, we decompose𝐮=P𝐮∗​𝐮\+P𝐮∗⟂​𝐮\\mathbf\{u\}=P\_\{\\mathbf\{u\}\_\{\*\}\}\\mathbf\{u\}\+P\_\{\\mathbf\{u\}\_\{\*\}\}^\{\\perp\}\\mathbf\{u\}and derive bounds for⟨P𝐮∗​𝐮,𝐳¯⟩\\left\\langle\{P\_\{\\mathbf\{u\}\_\{\*\}\}\\mathbf\{u\}\},\{\\bar\{\\mathbf\{z\}\}\}\\right\\rangleand⟨P𝐮∗⟂​𝐮,𝐳¯⟩\\left\\langle\{P\_\{\\mathbf\{u\}\_\{\*\}\}^\{\\perp\}\\mathbf\{u\}\},\{\\bar\{\\mathbf\{z\}\}\}\\right\\rangleindependently\.###### Lemma 6\(Geometric lemma\)\.

Fort≥0t\\geq 0, ifV≤1V\\leq 1, we have the following bounds:

⟨Z​𝜶,P𝐮⟂​𝐮∗⟩≥γ−\(1−V\)​\[\(1−V\)​γ¯⏟bounds⟨P𝐮∗​𝐮,𝐳¯⟩\+\(1−γ¯2\)​\(2​V−V2\)⏟bounds⟨P𝐮∗⟂​𝐮,𝐳¯⟩\]⏞bound⟨P𝐮​𝐮∗,Z​𝜶⟩\(≕κ0​\(V\)\),\\left\\langle\{Z\\bm\{\\alpha\}\},\{P\_\{\\mathbf\{u\}\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\}\\right\\rangle\\geq\\gamma\-\\overbrace\{\(1\-V\)\[\\underbrace\{\(1\-V\)\\bar\{\\gamma\}\}\_\{\\text\{bounds $\\left\\langle\{P\_\{\\mathbf\{u\}\_\{\*\}\}\\mathbf\{u\}\},\{\\bar\{\\mathbf\{z\}\}\}\\right\\rangle$\}\}\+\\underbrace\{\\sqrt\{\(1\-\\bar\{\\gamma\}^\{2\}\)\(2V\-V^\{2\}\)\}\}\_\{\\text\{bounds $\\langle\{P\_\{\\mathbf\{u\}\_\{\*\}\}^\{\\perp\}\\mathbf\{u\}\},\{\\bar\{\\mathbf\{z\}\}\}\\rangle$\}\}\]\}^\{\\text\{bound $\\left\\langle\{P\_\{\\mathbf\{u\}\}\\mathbf\{u\}\_\{\*\}\},\{Z\\bm\{\\alpha\}\}\\right\\rangle$\}\}\\quad\(\\eqqcolon\\kappa\_\{0\}\(V\)\),\(7\)whereγ¯≔⟨𝐮∗,𝐳¯⟩\\bar\{\\gamma\}\\coloneqq\\left\\langle\{\\mathbf\{u\}\_\{\*\}\},\{\\bar\{\\mathbf\{z\}\}\}\\right\\rangleis the margin of the data mean𝐳¯≔1n​∑i=1n𝐳i\\bar\{\\mathbf\{z\}\}\\coloneqq\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\mathbf\{z\}\_\{i\}\.

###### Proof\.\.

Using⟨𝐮∗,Z​𝜶⟩≥γ\\left\\langle\{\\mathbf\{u\}\_\{\*\}\},\{Z\\bm\{\\alpha\}\}\\right\\rangle\\geq\\gamma, we decomposeP𝐮⟂​𝐮∗=𝐮∗−P𝐮​𝐮∗P\_\{\\mathbf\{u\}\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}=\\mathbf\{u\}\_\{\*\}\-P\_\{\\mathbf\{u\}\}\\mathbf\{u\}\_\{\*\}and bound⟨Z​𝜶,P𝐮⟂​𝐮∗⟩\\left\\langle\{Z\\bm\{\\alpha\}\},\{P\_\{\\mathbf\{u\}\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\}\\right\\rangleas follows \(see Figure[2](https://arxiv.org/html/2608.04382#S3.F2)\(Left\)for the geometric illustration\):

⟨Z​𝜶,P𝐮⟂​𝐮∗⟩=⟨Z​𝜶,𝐮∗−⟨𝐮,𝐮∗⟩​𝐮⟩≥γ−\(1−V\)​⟨𝐮,Z​𝜶⟩≥γ−\(1−V\)​⟨𝐮,𝐳¯⟩,\\langle Z\\bm\{\\alpha\},P\_\{\\mathbf\{u\}\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\\rangle=\\left\\langle\{Z\\bm\{\\alpha\}\},\{\\mathbf\{u\}\_\{\*\}\-\\left\\langle\{\\mathbf\{u\}\},\{\\mathbf\{u\}\_\{\*\}\}\\right\\rangle\\mathbf\{u\}\}\\right\\rangle\\geq\\gamma\-\(1\-V\)\\left\\langle\{\\mathbf\{u\}\},\{Z\\bm\{\\alpha\}\}\\right\\rangle\\geq\\gamma\-\(1\-V\)\\left\\langle\{\\mathbf\{u\}\},\{\\bar\{\\mathbf\{z\}\}\}\\right\\rangle,where we used⟨𝐮,Z​𝜶⟩≤⟨𝐮,𝐳¯⟩\\left\\langle\{\\mathbf\{u\}\},\{Z\\bm\{\\alpha\}\}\\right\\rangle\\leq\\left\\langle\{\\mathbf\{u\}\},\{\\bar\{\\mathbf\{z\}\}\}\\right\\rangleimplied by Lemma[16](https://arxiv.org/html/2608.04382#Thmtheorem16)\(in Appendix[B](https://arxiv.org/html/2608.04382#A2); this is the temperature monotonicity of the exponentially weighted average\) withmi=⟨𝐮,𝐳i⟩m\_\{i\}=\\left\\langle\{\\mathbf\{u\}\},\{\\mathbf\{z\}\_\{i\}\}\\right\\rangleandβ=r\\beta=r\. Then, orthogonally decompose𝐮=⟨𝐮,𝐮∗⟩​𝐮∗\+𝐮⟂\\mathbf\{u\}=\\left\\langle\{\\mathbf\{u\}\},\{\\mathbf\{u\}\_\{\*\}\}\\right\\rangle\\mathbf\{u\}\_\{\*\}\+\\mathbf\{u\}\_\{\\perp\}, where𝐮⟂≔P𝐮∗⟂​𝐮\\mathbf\{u\}\_\{\\perp\}\\coloneqq P\_\{\\mathbf\{u\}\_\{\*\}\}^\{\\perp\}\\mathbf\{u\}\. The orthogonal component has‖𝐮⟂‖2=1−⟨𝐮,𝐮∗⟩2=2​V−V2\\\|\\mathbf\{u\}\_\{\\perp\}\\\|^\{2\}=1\-\\left\\langle\{\\mathbf\{u\}\},\{\\mathbf\{u\}\_\{\*\}\}\\right\\rangle^\{2\}=2V\-V^\{2\}\. Using this, we further have⟨𝐮,𝐳¯⟩=\(1−V\)​γ¯\+⟨𝐮⟂,𝐳¯⟩\\left\\langle\{\\mathbf\{u\}\},\{\\bar\{\\mathbf\{z\}\}\}\\right\\rangle=\(1\-V\)\\bar\{\\gamma\}\+\\left\\langle\{\\mathbf\{u\}\_\{\\perp\}\},\{\\bar\{\\mathbf\{z\}\}\}\\right\\rangle—this decomposition is illustrated in Figure[2](https://arxiv.org/html/2608.04382#S3.F2)\(Right\)\. Lastly, we bound⟨𝐮⟂,𝐳¯⟩\\left\\langle\{\\mathbf\{u\}\_\{\\perp\}\},\{\\bar\{\\mathbf\{z\}\}\}\\right\\rangleas follows:

⟨𝐮⟂,𝐳¯⟩\\displaystyle\\left\\langle\{\\mathbf\{u\}\_\{\\perp\}\},\{\\bar\{\\mathbf\{z\}\}\}\\right\\rangle=⟨𝐮⟂,P𝐮∗⟂​𝐳¯⟩\\displaystyle=\\left\\langle\{\\mathbf\{u\}\_\{\\perp\}\},\{P\_\{\\mathbf\{u\}\_\{\*\}\}^\{\\perp\}\\bar\{\\mathbf\{z\}\}\}\\right\\rangle\(by𝐮⟂⟂𝐮∗\\mathbf\{u\}\_\{\\perp\}\\perp\\mathbf\{u\}\_\{\*\}\)≤‖𝐮⟂‖⋅‖P𝐮∗⟂​𝐳¯‖\\displaystyle\\leq\\\|\\mathbf\{u\}\_\{\\perp\}\\\|\\cdot\\\|P\_\{\\mathbf\{u\}\_\{\*\}\}^\{\\perp\}\\bar\{\\mathbf\{z\}\}\\\|\(Cauchy–Schwarz\)=‖𝐮⟂‖⋅‖𝐳¯−γ¯​𝐮∗‖=‖𝐮⟂‖​‖𝐳¯‖2−γ¯2\\displaystyle=\\\|\\mathbf\{u\}\_\{\\perp\}\\\|\\cdot\\\|\\bar\{\\mathbf\{z\}\}\-\\bar\{\\gamma\}\\mathbf\{u\}\_\{\*\}\\\|=\\\|\\mathbf\{u\}\_\{\\perp\}\\\|\\sqrt\{\\\|\\bar\{\\mathbf\{z\}\}\\\|^\{2\}\-\\bar\{\\gamma\}^\{2\}\}\(byP𝐮∗⟂=I−𝐮∗​𝐮∗⊤P\_\{\\mathbf\{u\}\_\{\*\}\}^\{\\perp\}=I\-\\mathbf\{u\}\_\{\*\}\\mathbf\{u\}\_\{\*\}^\{\\top\}\)≤‖𝐮⟂‖​1−γ¯2=\(1−γ¯2\)​\(2​V−V2\)\.\\displaystyle\\leq\\\|\\mathbf\{u\}\_\{\\perp\}\\\|\\sqrt\{1\-\\bar\{\\gamma\}^\{2\}\}=\\sqrt\{\(1\-\\bar\{\\gamma\}^\{2\}\)\(2V\-V^\{2\}\)\}\.\(by‖𝐳¯‖≤1\\\|\\bar\{\\mathbf\{z\}\}\\\|\\leq 1\)By aggregating altogether, we have the lower bound \([7](https://arxiv.org/html/2608.04382#S3.E7)\)\. ∎

We alleviate thett\-dependency of the lower bound \([7](https://arxiv.org/html/2608.04382#S3.E7)\) \(viaVV\) later; with that, Lemma[5](https://arxiv.org/html/2608.04382#Thmtheorem5)implies the early\-stage weak alignment\. Lemma[6](https://arxiv.org/html/2608.04382#Thmtheorem6)requires that the initialization is not too bad:V​\(0\)≤1V\(0\)\\leq 1\. IfV​\(0\)\>1V\(0\)\>1holds, the initial tangential component𝐮​\(0\)\\mathbf\{u\}\(0\)and the max\-margin direction𝐮∗\\mathbf\{u\}\_\{\*\}are not in the same hemisphere; still, the flow rapidly escapes from such a “bad” initialization, as we show below\.

###### Lemma 7\(Escape from poor initialization\)\.

LetW​\(t\)≔⟨𝐰​\(t\),𝐮∗⟩=r​\(t\)​\(1−V​\(t\)\)W\(t\)\\coloneqq\\left\\langle\{\\mathbf\{w\}\(t\)\},\{\\mathbf\{u\}\_\{\*\}\}\\right\\rangle=r\(t\)\(1\-V\(t\)\)be the parameter projection, andTescape≔L​\(0\)−1​\{exp⁡\(ρ​γ−1​\[V​\(0\)−1\]\)−1\}T\_\{\\text\{escape\}\}\\coloneqq L\(0\)^\{\-1\}\\left\\\{\\exp\(\\rho\\gamma^\{\-1\}\[V\(0\)\-1\]\)\-1\\right\\\}\. IfV​\(0\)\>1V\(0\)\>1, then the flow escapes the bad initialization, namely,V​\(t\)≤1V\(t\)\\leq 1, aftert≥Tescapet\\geq T\_\{\\text\{escape\}\}\.

###### Proof\.\.

With⟨𝐮∗,Z​𝜶⟩≥γ\\left\\langle\{\\mathbf\{u\}\_\{\*\}\},\{Z\\bm\{\\alpha\}\}\\right\\rangle\\geq\\gammaand Lemma[4](https://arxiv.org/html/2608.04382#Thmtheorem4), we derive the differential inequality ofW​\(t\)W\(t\)as follows:

W˙​\(t\)=⟨𝐰˙​\(t\),𝐮∗⟩=−⟨∇L​\(𝐰​\(t\)\),𝐮∗⟩=L​\(𝐰​\(t\)\)​⟨Z​𝜶​\(t\),𝐮∗⟩≥γL​\(0\)−1\+t\.\\dot\{W\}\(t\)=\\left\\langle\{\\dot\{\\mathbf\{w\}\}\(t\)\},\{\\mathbf\{u\}\_\{\*\}\}\\right\\rangle=\-\\left\\langle\{\\nabla L\(\\mathbf\{w\}\(t\)\)\},\{\\mathbf\{u\}\_\{\*\}\}\\right\\rangle=L\(\\mathbf\{w\}\(t\)\)\\left\\langle\{Z\\bm\{\\alpha\}\(t\)\},\{\\mathbf\{u\}\_\{\*\}\}\\right\\rangle\\geq\\frac\{\\gamma\}\{L\(0\)^\{\-1\}\+t\}\.\(8\)Integrating over\[0,t\]\[0,t\], we haveW​\(t\)≥W​\(0\)\+γ​log⁡\(1\+L​\(0\)​t\)W\(t\)\\geq W\(0\)\+\\gamma\\log\(1\+L\(0\)t\)\. Note thatW​\(t\)∈ℝW\(t\)\\in\\mathbb\{R\}in general\. To escape from poor initialization \(or to haveV​\(t\)≤1V\(t\)\\leq 1\), it is sufficient to haveW​\(t\)=r​\(t\)​\(1−V​\(t\)\)≥0W\(t\)=r\(t\)\(1\-V\(t\)\)\\geq 0\. Hence, by solvingW​\(0\)\+γ​log⁡\(1\+L​\(0\)​t\)≥0W\(0\)\+\\gamma\\log\(1\+L\(0\)t\)\\geq 0, we havet≥Tescapet\\geq T\_\{\\text\{escape\}\}\. ∎

We are now ready to show our main result, early\-stage tangential alignment, by combining Lemmas[5](https://arxiv.org/html/2608.04382#Thmtheorem5),[6](https://arxiv.org/html/2608.04382#Thmtheorem6), and[7](https://arxiv.org/html/2608.04382#Thmtheorem7)\. Here, the parameter first escape from a bad initialization \(Lemma[7](https://arxiv.org/html/2608.04382#Thmtheorem7); escape stage\), and then weakly aligns with the max\-margin direction \(Lemma[5](https://arxiv.org/html/2608.04382#Thmtheorem5); weak alignment stage\)\.

###### Theorem 8\(Early\-stage weak alignment\)\. Under the setup in Section[1\.1](https://arxiv.org/html/2608.04382#S1.SS1), consider the radial and tangential flows in \([2](https://arxiv.org/html/2608.04382#S2.E2)\)\. There exists a problem\-dependent thresholdδ¯\>0\\bar\{\\delta\}\>0such that the following holds: pick a fixed constantΔ\>δ¯\\Delta\>\\bar\{\\delta\}, then for anyδ\>Δ\\delta\>\\Delta, the tangential alignment achievesV​\(t\)≤δV\(t\)\\leq\\deltaaftert=O​\(exp⁡\(exp⁡\(−δ/κ0​\(Δ\)​γ2\)\)\)t=O\(\\exp\(\\exp\(\-\\delta/\\kappa\_\{0\}\(\\Delta\)\\gamma^\{2\}\)\)\)time\.

We compare our early\-stage alignment and perfect alignment of Theorem[2](https://arxiv.org/html/2608.04382#Thmtheorem2)in Figure[3](https://arxiv.org/html/2608.04382#S3.F3)\. The early\-stage alignment first undergoes the escaping stage, takingTescape=O​\(1\)T\_\{\\text\{escape\}\}=O\(1\)time—by regarding initialization and the dataset marginγ\\gammaas problem\-dependent constants\. Then it undergoes the weak alignment stage to reachV​\(t\)≤δV\(t\)\\leq\\delta\. To apply Lemma[5](https://arxiv.org/html/2608.04382#Thmtheorem5), we need a uniform lower boundκ\\kappaon⟨Z​𝜶,P𝐮⟂​𝐮∗⟩\\langle\{Z\\bm\{\\alpha\}\},\{P\_\{\\mathbf\{u\}\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\}\\ranglethroughout the stage whenV∈\[δ,1\]V\\in\[\\delta,1\]\. As we show in Section[3\.1](https://arxiv.org/html/2608.04382#S3.SS1), for any fixedΔ\>δ¯\\Delta\>\\bar\{\\delta\}, we can takeκ=κ0​\(Δ\)\>0\\kappa=\\kappa\_\{0\}\(\\Delta\)\>0, which is a positive constant independent ofttandδ\\delta\. With this choice, the weak alignment time isT1=O​\(exp⁡\(exp⁡\(−δ/κ0​\(Δ\)​γ2\)\)\)=O​\(exp⁡\(exp⁡\(−δ\)\)\)T\_\{1\}=O\(\\exp\(\\exp\(\-\\delta/\\kappa\_\{0\}\(\\Delta\)\\gamma^\{2\}\)\)\)=O\(\\exp\(\\exp\(\-\\delta\)\)\)for allδ\>Δ\\delta\>\\Delta, whereκ0​\(Δ\)\\kappa\_\{0\}\(\\Delta\)andγ\\gammaare absorbed into the absolute constant\. Therefore, the early\-stage alignment is basically dominated by the weak alignment stage timeT1T\_\{1\}, and this is substantially faster than the asymptotic perfect alignmentT∞=O~​\(exp⁡\(δ−1/2\)\)T\_\{\\infty\}=\\widetilde\{O\}\(\\exp\(\\delta^\{\-1/2\}\)\)\.

Nevertheless, we must note again that the early\-stage alignment can only achieve*weak*alignment, that is,V​\(t\)≤δV\(t\)\\leq\\deltacannot be achieved for an arbitrarily small errorδ\\delta\. From Theorem[8](https://arxiv.org/html/2608.04382#Thmtheorem8), we only achieve alignment up to some dataset\-dependentδ¯\\bar\{\\delta\}\. In Section[3\.1](https://arxiv.org/html/2608.04382#S3.SS1), we will have a tight approximation boundδ¯≤1−γ\\bar\{\\delta\}\\leq 1\-\\gamma\. Roughly, this indicates that⟨𝐮,𝐮∗⟩≥γ\\left\\langle\{\\mathbf\{u\}\},\{\\mathbf\{u\}\_\{\*\}\}\\right\\rangle\\geq\\gammais achieved through weak alignment—or it can be improved up to⟨𝐮,𝐮∗⟩≥γ\\left\\langle\{\\mathbf\{u\}\},\{\\mathbf\{u\}\_\{\*\}\}\\right\\rangle\\geq\\sqrt\{\\gamma\}for a dataset with the data\-mean marginγ¯=⟨𝐮∗,𝐳¯⟩≈1\\bar\{\\gamma\}=\\left\\langle\{\\mathbf\{u\}\_\{\*\}\},\{\\bar\{\\mathbf\{z\}\}\}\\right\\rangle\\approx 1\. See Appendix[A](https://arxiv.org/html/2608.04382#A1)for details\. This alignment is strictly weaker than the asymptotic tangential convergence, but still numerically non\-negligible\.222For example,γ≥0\.31\\sqrt\{\\gamma\}\\geq 0\.31whenγ=0\.1\\gamma=0\.1andγ≥0\.44\\sqrt\{\\gamma\}\\geq 0\.44whenγ=0\.2\\gamma=0\.2\. Even with such a small dataset marginγ\\gamma, the weak\-alignment thresholdγ\\sqrt\{\\gamma\}is non\-trivially large and cannot be achieved by merely the perceptron argument\[[21](https://arxiv.org/html/2608.04382#bib.bib21)\]\. See Table[1](https://arxiv.org/html/2608.04382#A6.T1)in Appendix[F](https://arxiv.org/html/2608.04382#A6)for numerical simulation to show necessary time attaining differentγ\\gammaandγ\\sqrt\{\\gamma\}\.

### 3\.1Proof of Theorem[8](https://arxiv.org/html/2608.04382#Thmtheorem8)

The escaping stage only takesTescape=O​\(1\)T\_\{\\text\{escape\}\}=O\(1\)at most, as seen in Lemma[7](https://arxiv.org/html/2608.04382#Thmtheorem7)\. Hereafter, we assume that our initialization satisfiesV​\(0\)<1V\(0\)<1to establish the weak\-alignment stage and “re\-index”tt\. Note thatV​\(t\)≤1V\(t\)\\leq 1persists thereafter becauseW​\(t\)=r​\(t\)​\(1−V​\(t\)\)W\(t\)=r\(t\)\(1\-V\(t\)\)is strictly increasing due to \([8](https://arxiv.org/html/2608.04382#S3.E8)\), which maintainsW​\(t\)≥0W\(t\)\\geq 0throughoutttafter the escape\.

![[Uncaptioned image]](https://arxiv.org/html/2608.04382v1/x5.png)

Figure 3:The two\-stage dynamics: theescapestage quickly moves away from poor initialization, followed by theweak alignmentstage achieving⟨𝐮,𝐮∗⟩≥1−δ\\left\\langle\{\\mathbf\{u\}\},\{\\mathbf\{u\}\_\{\*\}\}\\right\\rangle\\geq 1\-\\delta\.

![[Uncaptioned image]](https://arxiv.org/html/2608.04382v1/x6.png)

Figure 4:Plot ofκ0​\(V\)\\kappa\_\{0\}\(V\)forV∈\[0,1\]V\\in\[0,1\]with parametersγ=0\.8\\gamma=0\.8andγ¯=0\.9\\bar\{\\gamma\}=0\.9, which is strictly increasing onδ¯<V<1\\bar\{\\delta\}<V<1\.

The weak\-alignment stage is proven by concatenating Lemmas[5](https://arxiv.org/html/2608.04382#Thmtheorem5)and[6](https://arxiv.org/html/2608.04382#Thmtheorem6), but we still need to check that⟨Z​𝜶,P𝐮⟂​𝐮∗⟩\\left\\langle\{Z\\bm\{\\alpha\}\},\{P\_\{\\mathbf\{u\}\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\}\\right\\ranglehas a non\-degenerate lower bound to invoke Lemma[5](https://arxiv.org/html/2608.04382#Thmtheorem5)because the lower bound \([7](https://arxiv.org/html/2608.04382#S3.E7)\) provided by Lemma[6](https://arxiv.org/html/2608.04382#Thmtheorem6)is time\-dependent \(throughVV\), whose strict positivity is not obvious\. Recall

κ0​\(V\)≔γ−\(1−V\)​\[\(1−V\)​γ¯\+\(1−γ¯2\)​\(2​V−V2\)\]\.\\kappa\_\{0\}\(V\)\\coloneqq\\gamma\-\(1\-V\)\[\(1\-V\)\\bar\{\\gamma\}\+\\sqrt\{\(1\-\\bar\{\\gamma\}^\{2\}\)\(2V\-V^\{2\}\)\}\]\.Subsequently, we mainly focus on checkingκ0​\(V\)\>0\\kappa\_\{0\}\(V\)\>0in the range of our interest\. By rearrangingκ0​\(V\)\>0\\kappa\_\{0\}\(V\)\>0, we have the quadratic inequalityU2−\(1\+2​γ​γ¯−γ¯2\)​U\+γ2\>0U^\{2\}\-\(1\+2\\gamma\\bar\{\\gamma\}\-\\bar\{\\gamma\}^\{2\}\)U\+\\gamma^\{2\}\>0withU≔\(1−V\)2U\\coloneqq\(1\-V\)^\{2\}\. Solving this, we haveδ¯<V<1\\bar\{\\delta\}<V<1, where

δ¯≔1−1\+2​γ​γ¯−γ¯2−\(1\+2​γ​γ¯−γ¯2\)2−4​γ22\.\\bar\{\\delta\}\\coloneqq 1\-\\sqrt\{\\frac\{1\+2\\gamma\\bar\{\\gamma\}\-\\bar\{\\gamma\}^\{2\}\-\\sqrt\{\(1\+2\\gamma\\bar\{\\gamma\}\-\\bar\{\\gamma\}^\{2\}\)^\{2\}\-4\\gamma^\{2\}\}\}\{2\}\}\.\(9\)Actually, we can make this bound cleaner \(but slightly looser\) by\(δ¯≤\)1−γ<V<1\(\\bar\{\\delta\}\\leq\)1\-\\gamma<V<1\(cf\. Section[A](https://arxiv.org/html/2608.04382#A1)\), which is convenient for interpretation\. In addition, we can check thatκ0\\kappa\_\{0\}is strictly increasing onV∈\(δ¯,1\)V\\in\(\\bar\{\\delta\},1\)by calculating the derivativeκ0′\\kappa\_\{0\}^\{\\prime\}\. We visualizeκ0\\kappa\_\{0\}in Figure[4](https://arxiv.org/html/2608.04382#S3.F4)\. Now we are ready to apply Lemma[5](https://arxiv.org/html/2608.04382#Thmtheorem5)\. Fix anyΔ\>δ¯\\Delta\>\\bar\{\\delta\}\. Sinceκ0\\kappa\_\{0\}is strictly increasing on\(δ¯,1\)\(\\bar\{\\delta\},1\), for anyδ≥Δ\(\>δ¯\)\\delta\\geq\\Delta\(\>\\bar\{\\delta\}\)andV∈\[δ,1\]V\\in\[\\delta,1\], we have⟨Z​𝜶,P𝐮⟂​𝐮∗⟩≥κ0​\(V\)≥κ0​\(δ\)≥κ0​\(Δ\)\>0\\langle\{Z\\bm\{\\alpha\}\},\{P\_\{\\mathbf\{u\}\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\}\\rangle\\geq\\kappa\_\{0\}\(V\)\\geq\\kappa\_\{0\}\(\\delta\)\\geq\\kappa\_\{0\}\(\\Delta\)\>0\. Settingκ=κ0​\(Δ\)\\kappa=\\kappa\_\{0\}\(\\Delta\), which is a positive constant independent ofttandδ\\delta, Lemma[5](https://arxiv.org/html/2608.04382#Thmtheorem5)givesT1=O​\(exp⁡\(exp⁡\(−δ/κ0​\(Δ\)​γ2\)\)\)T\_\{1\}=O\(\\exp\(\\exp\(\-\\delta/\\kappa\_\{0\}\(\\Delta\)\\gamma^\{2\}\)\)\)for allδ\>Δ\\delta\>\\Delta\. The total time in the early stage isTescape\+T1=O​\(exp⁡\(exp⁡\(−δ\)\)\)T\_\{\\text\{escape\}\}\+T\_\{1\}=O\(\\exp\(\\exp\(\-\\delta\)\)\), concluding the proof\.

### 3\.2Lower bound

So far, we confirmed the fast weak alignment withinT1=O​\(exp⁡\(exp⁡\(−δ\)\)\)T\_\{1\}=O\(\\exp\(\\exp\(\-\\delta\)\)\)time in Theorem[8](https://arxiv.org/html/2608.04382#Thmtheorem8)\. We now establish a lower bound on the tangential alignmentV​\(t\)V\(t\)during the early stage, showing the tightness of our analysis\. Its proof largely shares the same idea as Lemma[5](https://arxiv.org/html/2608.04382#Thmtheorem5)\(early\-stage alignment\), to solve the differential inequality ofV​\(t\)V\(t\), but we need a better bound ofr​\(t\)r\(t\)to get the matching lower bound\. The refined radial estimate in Lemma[4](https://arxiv.org/html/2608.04382#Thmtheorem4)serves for this purpose\. After all, we can establish a matching boundV​\(t\)∼1−log⁡log⁡tV\(t\)\\sim 1\-\\log\\log tduring the early staget≤Θ​\(exp⁡\(exp⁡\(−δ\)\)\)t\\leq\\Theta\(\\exp\(\\exp\(\-\\delta\)\)\), ensuring the tightness of the analysis\. The complete proof is deferred to Appendix[C](https://arxiv.org/html/2608.04382#A3)\.

###### Theorem 10\. Fixδ\>0\\delta\>0and assumeL​\(0\)<eρ​γ2L\(0\)<e^\{\\rho\\gamma^\{2\}\}\. Under the setup in Section[1\.1](https://arxiv.org/html/2608.04382#S1.SS1), fort≥0t\\geq 0:V​\(t\)≥V​\(0\)−C0−2​γ−2​V​\(0\)​log⁡log⁡\(L​\(0\)−1\+γ2​t\),V\(t\)\\geq V\(0\)\-C\_\{0\}\-2\\gamma^\{\-2\}\\sqrt\{V\(0\)\}\\log\\log\(L\(0\)^\{\-1\}\+\\gamma^\{2\}t\),\(11\)whereC0C\_\{0\}is a problem\-dependent constant\. In addition, forδ\>0\\delta\>0,V​\(t\)≥δV\(t\)\\geq\\deltaholds duringt∈\[0,T1\]t\\in\[0,T\_\{1\}\]withT1=Ω​\(exp⁡\(exp⁡\(−δ\)\)\)T\_\{1\}=\\Omega\(\\exp\(\\exp\(\-\\delta\)\)\)\.

### 3\.3Case of logistic loss

Theorem[8](https://arxiv.org/html/2608.04382#Thmtheorem8)can be shown even if we use the logistic lossℓ​\(m\)≔log⁡\(1\+e−m\)\\ell\(m\)\\coloneqq\\log\(1\+e^\{\-m\}\)\. While the main proof structure remains the same, \(i\) the self\-bounding propertyγ​L≤‖∇L‖≤L\\gamma L\\leq\\\|\\nabla L\\\|\\leq Land \(ii\) the temperature monotonicity⟨𝐮,Z​𝜶⟩\\left\\langle\{\\mathbf\{u\}\},\{Z\\bm\{\\alpha\}\}\\right\\rangleused in Lemma[6](https://arxiv.org/html/2608.04382#Thmtheorem6)are specific to the exponential loss so far\. Here, we explain how these two can be extended for the logistic loss\. With the marginmi​\(t\)≔⟨𝐰​\(t\),𝐳i⟩m\_\{i\}\(t\)\\coloneqq\\left\\langle\{\\mathbf\{w\}\(t\)\},\{\\mathbf\{z\}\_\{i\}\}\\right\\rangle, we use the following notations:

qi​\(t\)≔−ℓ′​\(mi​\(t\)\),G​\(t\)≔1n​∑i=1nqi​\(t\),and​αi​\(t\)≔qi​\(t\)∑j=1nqj​\(t\)\.q\_\{i\}\(t\)\\coloneqq\-\\ell^\{\\prime\}\(m\_\{i\}\(t\)\),\\quad G\(t\)\\coloneqq\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}q\_\{i\}\(t\),\\text\{~~and~~\}\\alpha\_\{i\}\(t\)\\coloneqq\\frac\{q\_\{i\}\(t\)\}\{\\sum\_\{j=1\}^\{n\}q\_\{j\}\(t\)\}\.The tangential Lyapunov function obeysV˙=−⟨Z​𝜶,P𝐮⟂​𝐮∗⟩⋅G/r\\dot\{V\}=\-\\left\\langle\{Z\\bm\{\\alpha\}\},\{P\_\{\\mathbf\{u\}\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\}\\right\\rangle\\cdot G/rinstead of \([4](https://arxiv.org/html/2608.04382#S3.E4)\), with this new gradient weight𝜶∈Δn−1\\bm\{\\alpha\}\\in\\Delta^\{n\-1\}\.

###### Lemma 11\.

For the logistic loss, define the problem\-dependent constant

ρ¯0≔1−e−n​L​\(0\)n​L​\(0\)​and​γℓ≔γ​ρ¯0\.\\bar\{\\rho\}\_\{0\}\\coloneqq\\frac\{1\-e^\{\-nL\(0\)\}\}\{nL\(0\)\}\\text\{~~and~~\}\\gamma\_\{\\ell\}\\coloneqq\\gamma\\bar\{\\rho\}\_\{0\}\.Then for allt≥0t\\geq 0, we haveρ¯0​L​\(t\)≤G​\(t\)≤L​\(t\)\\bar\{\\rho\}\_\{0\}L\(t\)\\leq G\(t\)\\leq L\(t\)\. Hereγℓ\>0\\gamma\_\{\\ell\}\>0as long asL​\(0\)\>0L\(0\)\>0\. As a corollary, we haveγℓ​L​\(t\)≤‖∇L​\(t\)‖≤L​\(t\)\\gamma\_\{\\ell\}L\(t\)\\leq\\\|\\nabla L\(t\)\\\|\\leq L\(t\)\.

This modified self\-bounding inequality leads to the risk estimates1L​\(0\)−1\+t≤L​\(t\)≤1L​\(0\)−1\+γℓ2​t\\frac\{1\}\{L\(0\)^\{\-1\}\+t\}\\leq L\(t\)\\leq\\frac\{1\}\{L\(0\)^\{\-1\}\+\\gamma\_\{\\ell\}^\{2\}t\}and the radial estimates\|r​\(t\)−ρ\|≤1γℓ2​log⁡\(1\+L​\(0\)​γℓ2​t\)\|r\(t\)\-\\rho\|\\leq\\frac\{1\}\{\\gamma\_\{\\ell\}^\{2\}\}\\log\(1\+L\(0\)\\gamma\_\{\\ell\}^\{2\}t\), replacing Lemma[4](https://arxiv.org/html/2608.04382#Thmtheorem4)\. This fix directly leads to the early weak\-alignment rateT1=O​\(exp⁡\(exp⁡\(−δ/κ​ρ¯0​γℓ2\)\)\)T\_\{1\}=O\(\\exp\(\\exp\(\-\\delta/\\kappa\\bar\{\\rho\}\_\{0\}\\gamma\_\{\\ell\}^\{2\}\)\)\), under the ansatz \([5](https://arxiv.org/html/2608.04382#S3.E5)\), replacing Lemma[5](https://arxiv.org/html/2608.04382#Thmtheorem5)\. The ansatz needs the following modified temperature monotonicity to justify\.

###### Lemma 12\.

For the logistic loss, we have⟨𝐮​\(t\),Z​𝛂​\(t\)⟩≤\[⟨𝐮​\(t\),𝐳¯⟩\]\+\\left\\langle\{\\mathbf\{u\}\(t\)\},\{Z\\bm\{\\alpha\}\(t\)\}\\right\\rangle\\leq\[\\left\\langle\{\\mathbf\{u\}\(t\)\},\{\\bar\{\\mathbf\{z\}\}\}\\right\\rangle\]\_\{\+\}for allt≥0t\\geq 0\.

With the above fixes, we can show the same weak alignment rate for the logistic loss\. Its proof together with Lemmas[11](https://arxiv.org/html/2608.04382#Thmtheorem11)and[12](https://arxiv.org/html/2608.04382#Thmtheorem12)are shown in Section[D](https://arxiv.org/html/2608.04382#A4)\.

###### Corollary 13\. Under the same setup as Theorem[8](https://arxiv.org/html/2608.04382#Thmtheorem8)except that the logistic loss is used, there exists a problem\-dependent thresholdδ¯\>0\\bar\{\\delta\}\>0such that the following holds: pick a fixed constantΔ\>δ¯\\Delta\>\\bar\{\\delta\}, then for anyδ\>Δ\\delta\>\\Delta, the tangential alignment achievesV​\(t\)≤δV\(t\)\\leq\\deltaaftert=O​\(exp⁡\(exp⁡\(−δ/κ0​\(Δ\)​ρ¯0​γℓ2\)\)\)t=O\(\\exp\(\\exp\(\-\\delta/\\kappa\_\{0\}\(\\Delta\)\\bar\{\\rho\}\_\{0\}\\gamma\_\{\\ell\}^\{2\}\)\)\)time\.

## 4Discrete\-time analysis

We now extend our early\-stage flow analysis to the discrete\-time gradient descent setting with stepsizeη\>0\\eta\>0:𝐰​\(t\+1\)=𝐰​\(t\)−η​∇L​\(𝐰​\(t\)\)\\mathbf\{w\}\(t\+1\)=\\mathbf\{w\}\(t\)\-\\eta\\nabla L\(\\mathbf\{w\}\(t\)\)for everyt≥0t\\geq 0\. Similarly, decompose the parameter into the radial and tangential components,r​\(t\)≔‖𝐰​\(t\)‖r\(t\)\\coloneqq\\\|\\mathbf\{w\}\(t\)\\\|and𝐮​\(t\)≔𝐰​\(t\)/‖𝐰​\(t\)‖\\mathbf\{u\}\(t\)\\coloneqq\\mathbf\{w\}\(t\)/\\\|\\mathbf\{w\}\(t\)\\\|, respectively, and focus on the tangential alignmentV​\(t\)≔1−⟨𝐮​\(t\),𝐮∗⟩V\(t\)\\coloneqq 1\-\\left\\langle\{\\mathbf\{u\}\(t\)\},\{\\mathbf\{u\}\_\{\*\}\}\\right\\rangle\. The parameter is initialized withL​\(0\)L\(0\),V​\(0\)V\(0\), andρ≔r​\(0\)\\rho\\coloneqq r\(0\)\. Note that these notations remain the same as the continuous case but should not be confused\. Here we show the overview of the discrete\-time analysis and defer the full analysis to Appendix[E](https://arxiv.org/html/2608.04382#A5)\. In the discrete\-time case, we need to carefully track the discretization error of the continuous flows, and the stepsize choice plays an important role therein\.

###### Assumption 14\.

The stepsize satisfiesη≤c0​min⁡\{ρ​L​\(0\)−1,γ2\}\\eta\\leq c\_\{0\}\\min\\\{\\rho L\(0\)^\{\-1\},\\gamma^\{2\}\\\}for a sufficiently smallc0\>0c\_\{0\}\>0\.

With this choice, we can show the discrete analogs of the structural results in Section[2](https://arxiv.org/html/2608.04382#S2), namely, the radial/tangential flows \(Lemma[3](https://arxiv.org/html/2608.04382#Thmtheorem3)\) and risk/radial bounds \(Lemma[4](https://arxiv.org/html/2608.04382#Thmtheorem4)\)\. We sketch both ideas\.

#### Discrete radial/tangential flows\.

Fort=0,1,2,…t=0,1,2,\\dots, we keep tracking the one\-step incrementsΔ​r​\(t\)≔r​\(t\+1\)−r​\(t\)\\Delta r\(t\)\\coloneqq r\(t\+1\)\-r\(t\)andΔ​𝐮​\(t\)≔𝐮​\(t\+1\)−𝐮​\(t\)\\Delta\\mathbf\{u\}\(t\)\\coloneqq\\mathbf\{u\}\(t\+1\)\-\\mathbf\{u\}\(t\), for which we can derive the following discretized dynamics \(and the formal statement is presented in Lemma[18](https://arxiv.org/html/2608.04382#Thmtheorem18)\):

\{Δ​r​\(t\)=−η​⟨𝐮​\(t\),∇L​\(𝐰​\(t\)\)⟩−O​\(η2​L​\(𝐰​\(t\)\)2r​\(t\)\),Δ​𝐮​\(t\)=−ηr​\(t\)​P𝐮​\(t\)⟂​∇L​\(𝐰​\(t\)\)\+O​\(η2​L​\(𝐰​\(t\)2\)r​\(t\)2\)\.\\begin\{cases\}\\Delta r\(t\)&=\-\\eta\\left\\langle\{\\mathbf\{u\}\(t\)\},\{\\nabla L\(\\mathbf\{w\}\(t\)\)\}\\right\\rangle\-O\\left\(\\frac\{\\eta^\{2\}L\(\\mathbf\{w\}\(t\)\)^\{2\}\}\{r\(t\)\}\\right\),\\\\ \\Delta\\mathbf\{u\}\(t\)&=\-\\frac\{\\eta\}\{r\(t\)\}P\_\{\\mathbf\{u\}\(t\)\}^\{\\perp\}\\nabla L\(\\mathbf\{w\}\(t\)\)\+O\\left\(\\frac\{\\eta^\{2\}L\(\\mathbf\{w\}\(t\)^\{2\}\)\}\{r\(t\)^\{2\}\}\\right\)\.\\end\{cases\}Here,Δ​r​\(t\)/η\\Delta r\(t\)/\\etaandΔ​𝐮​\(t\)/η\\Delta\\mathbf\{u\}\(t\)/\\etaplay similar roles tor˙​\(t\)\\dot\{r\}\(t\)and𝐮˙​\(t\)\\dot\{\\mathbf\{u\}\}\(t\)in the continuous flows \([2](https://arxiv.org/html/2608.04382#S2.E2)\), and the respective dynamics expressions remain similar up to the remainder terms\. To make these residuals negligible, we need the local conditionη​L​\(𝐰​\(t\)\)/r​\(t\)=O​\(1\)\\eta L\(\\mathbf\{w\}\(t\)\)/r\(t\)=O\(1\)\. Actually, Assumption[14](https://arxiv.org/html/2608.04382#Thmtheorem14)satisfies this att=0t=0\. Fort≥1t\\geq 1, we can expect thatL​\(𝐰​\(t\)\)=Θ​\(t−1\)L\(\\mathbf\{w\}\(t\)\)=\\Theta\(t^\{\-1\}\)holds andr​\(t\)r\(t\)has a non\-trivial lower bound, which is similar to the continuous case shown in Lemma[4](https://arxiv.org/html/2608.04382#Thmtheorem4)—still, their discrete\-time extensions need to be addressed carefully\. Then, it is natural to conjecture that\(η​L​\(𝐰​\(t\)\)/r​\(t\)\)t∈ℤ≥0\(\\eta L\(\\mathbf\{w\}\(t\)\)/r\(t\)\)\_\{t\\in\\mathbb\{Z\}\_\{\\geq 0\}\}is a decreasing sequence, and thus the local condition persists throughoutt=1,2,…t=1,2,\\dots\. This can be indeed verified via induction; see Remark[20](https://arxiv.org/html/2608.04382#Thmtheorem20)in Appendix[E](https://arxiv.org/html/2608.04382#A5)\.

#### Discrete risk/radial bounds\.

As seen in Lemma[4](https://arxiv.org/html/2608.04382#Thmtheorem4), we have similar bounds for the discrete case:L​\(t\)=Θ​\(t−1\)L\(t\)=\\Theta\(t^\{\-1\}\)andr​\(t\)∼ρ\+log⁡\(1\+t\)r\(t\)\\sim\\rho\+\\log\(1\+t\)—these bounds in discrete time incur multiplicative discretization error up to the factor1−O​\(η​γ−2\)1\-O\(\\eta\\gamma^\{\-2\}\), which is negligible with the stepsize choiceη≤c0​γ2\\eta\\leq c\_\{0\}\\gamma^\{2\}\. This error control suffices when taking the telescoping sum ofΔ​r​\(t\)\\Delta r\(t\)andΔ​𝐮​\(t\)\\Delta\\mathbf\{u\}\(t\)\(which replaces the differential inequalities we heavily relied on in the continuous\-time analysis\)\. The full statement of the risk/radial bounds is in Lemma[19](https://arxiv.org/html/2608.04382#Thmtheorem19)in Appendix[E](https://arxiv.org/html/2608.04382#A5)\.

#### Early\-stage alignment in discrete time\.

Building upon the above, the remaining piece is not substantially complicated; the geometric lemma \(Lemma[6](https://arxiv.org/html/2608.04382#Thmtheorem6)\) persists in the discrete case\. As in Section[3](https://arxiv.org/html/2608.04382#S3), the discrete\-time dynamics also undergoes the escape stage to get out from poor initialization, and the weak alignment stage to achieve up to⟨𝐮,𝐮∗⟩≥1−δ\\left\\langle\{\\mathbf\{u\}\},\{\\mathbf\{u\}\_\{\*\}\}\\right\\rangle\\geq 1\-\\delta\. Instead of solving the differential inequalities, we take the telescoping sums over the discrete dynamics\. Now we summarize the results informally\.

- •Escape stage: afterO​\(1\)O\(1\)iterations, the parameter achievesV​\(t\)≤1V\(t\)\\leq 1\.
- •Weak alignment stage: afterO​\(η−1​exp⁡\(exp⁡\(−δ\)\)\)O\(\\eta^\{\-1\}\\\!\\exp\(\\exp\(\-\\delta\)\)\)more iterations, the parameter achievesV​\(t\)≤δV\(t\)\\leq\\delta\. The error toleranceδ\\deltais roughly up to1−γ1\-\\sqrt\{\\gamma\}, as in the continuous case\.
- •Lower bound: to achieveV​\(t\)≤δV\(t\)\\leq\\delta, the necessary iteration is no less thanΩ​\(η−1​exp⁡\(exp⁡\(−δ\)\)\)\\Omega\(\\eta^\{\-1\}\\\!\\exp\(\\exp\(\-\\delta\)\)\)\.

## 5Discussion and open question

#### Improvement upon error toleranceδ\\delta\.

Despite our tight upper and lower bounds for the tangential alignment, the established weak alignment remains to be up to⟨𝐮,𝐮∗⟩≥γ\\left\\langle\{\\mathbf\{u\}\},\{\\mathbf\{u\}\_\{\*\}\}\\right\\rangle\\geq\\gammain general, orγ\\sqrt\{\\gamma\}at best for some nice dataset\. Revisiting Figure[1](https://arxiv.org/html/2608.04382#S1.F1), the early\-stage alignment would be higher than the dataset marginγ\\gammawithin considerably short time\. Improving this alignment error tolerance is an important open question\. To overcome this, we need to eschew invoking the uniform bound of the form⟨Z​𝜶,P𝐮⟂​𝐮∗⟩≥κ\\langle Z\\bm\{\\alpha\},P\_\{\\mathbf\{u\}\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\\rangle\\geq\\kappain \([5](https://arxiv.org/html/2608.04382#S3.E5)\)\. Our current proof in Lemma[6](https://arxiv.org/html/2608.04382#Thmtheorem6)purely relies on a geometric argument of a dataset, but its improvement is apparently impossible\. Thus, we presumably need to rely on other arguments, such as an additional assumption on a dataset structure, or a probabilistic argument\.

#### Analogy to “edge\-of\-stability” analysis\.

The edge of stability \(EoS\) is a phenomenon where learning dynamics with an excessively large stepsize oscillates significantly at the early stage yet eventually stabilizes to a flat minimum in a loss landscape, observed across deep learning and simpler problems\[[6](https://arxiv.org/html/2608.04382#bib.bib6),[1](https://arxiv.org/html/2608.04382#bib.bib1)\]\. Indeed, this also happens in logistic regression, and the dynamics undergoes the first oscillatory stage before stable convergence\[[34](https://arxiv.org/html/2608.04382#bib.bib34),[3](https://arxiv.org/html/2608.04382#bib.bib3)\]\. Their analysis framework is close to Section[1\.1](https://arxiv.org/html/2608.04382#S1.SS1), assuming the linear separable case\. Although a large stepsize benefits the convergence rate of the optimization problem, training points are correctly classified once the early oscillatory stage terminates\. At this point, the parameter already correlates with the max\-margin direction positively to some extent\. This observation inspires our study\. In contrast to these “EoS” analyses, our analysis deals with a small stepsize \(Assumption[14](https://arxiv.org/html/2608.04382#Thmtheorem14)\), and our tightness result hinges on this stepsize choice, so it remains unclear what our analysis implies with a larger stepsize\. While intentionally controlled, discretization error could accelerate the parameter to decrease the tangential alignment error additionally—this can be indirectly supported via the one\-step update ofV​\(t\)V\(t\)in \([21](https://arxiv.org/html/2608.04382#A5.E21)\)\. To verify this, we need to go beyond this infinitesimal error control\.

## Acknowledgment

HB is supported by JST PRESTO JPMJPR24K6\.

## References

- Ahn et al\. \[2022\]K\. Ahn, J\. Zhang, and S\. Sra\.Understanding the unstable convergence of gradient descent\.In*Proceedings of the 39th International Conference on Machine Learning*, pages 247–257, 2022\.
- Ardeshir et al\. \[2023\]N\. Ardeshir, D\. J\. Hsu, and C\. H\. Sanford\.Intrinsic dimensionality and generalization properties of theℛ\\mathcal\{R\}\-norm inductive bias\.In*Proceedings of the 36th Conference on Learning Theory*, pages 3264–3303, 2023\.
- Bao et al\. \[2025\]H\. Bao, S\. Sakaue, and Y\. Takezawa\.Any\-stepsize gradient descent for separable data under Fenchel–Young losses\.*Advances in Neural Information Processing Systems*, 38, 2025\.
- Boursier et al\. \[2022\]E\. Boursier, L\. Pillaud\-Vivien, and N\. Flammarion\.Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs\.*Advances in Neural Information Processing Systems*, 35:20105–20118, 2022\.
- Cao et al\. \[2022\]Y\. Cao, Z\. Chen, M\. Belkin, and Q\. Gu\.Benign overfitting in two\-layer convolutional neural networks\.*Advances in Neural Information Processing Systems*, 35:25237–25250, 2022\.
- Cohen et al\. \[2021\]J\. Cohen, S\. Kaur, Y\. Li, J\. Z\. Kolter, and A\. Talwalkar\.Gradient descent on neural networks typically occurs at the edge of stability\.In*Proceedings of the 9th International Conference on Learning Representations*, 2021\.
- Fan et al\. \[2025\]C\. Fan, M\. Schmidt, and C\. Thrampoulidis\.Implicit bias of spectral descent and Muon on multiclass separable data\.*Advances in Neural Information Processing Systems*, 38, 2025\.
- Glasgow \[2024\]M\. Glasgow\.SGD finds then tunes features in two\-layer neural networks with near\-optimal sample complexity: A case study in the XOR problem\.In*Proceedings of the 12th International Conference on Learning Representations*, 2024\.
- Gunasekar et al\. \[2018a\]S\. Gunasekar, J\. Lee, D\. Soudry, and N\. Srebro\.Characterizing implicit bias in terms of optimization geometry\.In*Proceedings of the 35th International Conference on Machine Learning*, pages 1832–1841, 2018a\.
- Gunasekar et al\. \[2018b\]S\. Gunasekar, J\. D\. Lee, D\. Soudry, and N\. Srebro\.Implicit bias of gradient descent on linear convolutional networks\.*Advances in Neural Information Processing Systems*, 31:9461–9471, 2018b\.
- Hoffer et al\. \[2017\]E\. Hoffer, I\. Hubara, and D\. Soudry\.Train longer, generalize better: closing the generalization gap in large batch training of neural networks\.*Advances in Neural Information Processing Systems*, 30:1731–1741, 2017\.
- Ji and Telgarsky \[2019\]Z\. Ji and M\. Telgarsky\.The implicit bias of gradient descent on nonseparable data\.In*Proceedings of the 32nd Conference on Learning Theory*, pages 1772–1798, 2019\.
- Ji and Telgarsky \[2020\]Z\. Ji and M\. Telgarsky\.Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks\.In*Proceedings of the 8th International Conference on Learning Representations*, 2020\.
- Ji et al\. \[2020\]Z\. Ji, M\. Dudík, R\. E\. Schapire, and M\. Telgarsky\.Gradient descent follows the regularization path for general losses\.In*Proceedings ot the 33rd Conference on Learning Theory*, pages 2109–2136, 2020\.
- Koltchinskii and Panchenko \[2002\]V\. Koltchinskii and D\. Panchenko\.Empirical margin distributions and bounding the generalization error of combined classifiers\.*The Annals of Statistics*, 30\(1\):1–50, 2002\.
- Liang and Montúfar \[2025\]S\. Liang and G\. Montúfar\.Implicit bias of mirror flow for shallow neural networks in univariate regression\.In*Proceedings of the 13th International Conference on Learning Representations*, 2025\.
- Lv et al\. \[2017\]K\. Lv, S\. Jiang, and J\. Li\.Learning gradient descent: Better generalization and longer horizons\.In*Proceedings of the 34th International Conference on Machine Learning*, pages 2247–2255, 2017\.
- Lyu and Li \[2020\]K\. Lyu and J\. Li\.Gradient descent maximizes the margin of homogeneous neural networks\.In*Proceedings of the 8th International Conference on Learning Representations*, 2020\.
- Meng et al\. \[2024\]S\. Y\. Meng, A\. Orvieto, D\. Y\. Cao, and C\. De Sa\.Gradient descent on logistic regression with non\-separable data and large step sizes\.*arXiv preprint arXiv:2406\.05033*, 2024\.
- Nacson et al\. \[2019\]M\. S\. Nacson, J\. Lee, S\. Gunasekar, P\. H\. P\. Savarese, N\. Srebro, and D\. Soudry\.Convergence of gradient descent on separable data\.In*Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics*, pages 3420–3428, 2019\.
- Novikoff \[1962\]A\. B\. J\. Novikoff\.On convergence proofs for perceptrons\.In*Proceedings of the Symposium on Mathematical Theory of Automata*, pages 615–620, 1962\.
- Ongie et al\. \[2020\]G\. Ongie, R\. Willett, D\. Soudry, and N\. Srebro\.A function space view of bounded norm infinite width relu nets: The multivariate case\.In*Proceedings of the 8th International Conference on Learning Representations*, 2020\.
- Ravi et al\. \[2024\]H\. Ravi, C\. Scott, D\. Soudry, and Y\. Wang\.The implicit bias of gradient descent on separable multiclass data\.*Advanced in Neural Information Processing Systems*, 37:81324–81359, 2024\.
- Rosset et al\. \[2003\]S\. Rosset, J\. Zhu, and T\. Hastie\.Margin maximizing loss functions\.*Advances in Neural Information Processing Systems*, 16:1237–1244, 2003\.
- Savarese et al\. \[2019\]P\. Savarese, I\. Evron, D\. Soudry, and N\. Srebro\.How do infinite width bounded norm networks look in function space?In*Proceedings of the 32nd Conference on Learning Theory*, pages 2667–2690, 2019\.
- Schliserman and Koren \[2022\]M\. Schliserman and T\. Koren\.Stability vs implicit bias of gradient methods on separable data and beyond\.In*Proeedings of the 35th Conference on Learning Theory*, pages 3380–3394, 2022\.
- Shamir \[2021\]O\. Shamir\.Gradient methods never overfit on separable data\.*Journal of Machine Learning Research*, 22\(85\):1–20, 2021\.
- Soudry et al\. \[2018\]D\. Soudry, E\. Hoffer, M\. S\. Nacson, S\. Gunasekar, and N\. Srebro\.The implicit bias of gradient descent on separable data\.*Journal of Machine Learning Research*, 19\(70\):1–57, 2018\.
- Tarzanagh et al\. \[2023\]D\. A\. Tarzanagh, Y\. Li, X\. Zhang, and S\. Oymak\.Max\-margin token selection in attention mechanism\.*Advances in Neural Information Processing Systems*, 36:48314–48362, 2023\.
- Tsilivis et al\. \[2025\]N\. Tsilivis, G\. Vardi, and J\. Kempe\.Flavors of margin: Implicit bias of steepest descent in homogeneous neural networks\.In*Proceedings of the 13th International Conference on Learning Representations*, 2025\.
- Tyurin \[2025\]A\. Tyurin\.From logistic regression to the perceptron algorithm: Exploring gradient descent with large step sizes\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 39, pages 20938–20946, 2025\.
- Vasudeva et al\. \[2025\]B\. Vasudeva, J\. W\. Lee, V\. Sharan, and M\. Soltanolkotabi\.The rich and the simple: On the implicit bias of Adam and SGD\.*Advances in Neural Information Processing Systems*, 38, 2025\.
- Woodworth et al\. \[2020\]B\. Woodworth, S\. Gunasekar, J\. D\. Lee, E\. Moroshko, P\. Savarese, I\. Golan, D\. Soudry, and N\. Srebro\.Kernel and rich regimes in overparametrized models\.In*Proceedings of the 33rd Conference on Learning Theory*, pages 3635–3673, 2020\.
- Wu et al\. \[2024\]J\. Wu, P\. L\. Bartlett, M\. Telgarsky, and B\. Yu\.Large stepsize gradient descent for logistic loss: Non\-monotonicity of the loss improves optimization efficiency\.In*Proceedings of the 37th Conference on Learning Theory*, pages 5019–5073, 2024\.
- Xie and Li \[2024\]S\. Xie and Z\. Li\.Implicit bias of AdamW:ℓ∞\\ell\_\{\\infty\}\-norm constrained optimization\.In*Proceedings of the 41st International Conference on Machine Learning*, pages 54488–54510, 2024\.
- Xu et al\. \[2009\]H\. Xu, C\. Caramanis, and S\. Mannor\.Robustness and regularization of support vector machines\.*Journal of Machine Learning Research*, 10:1485–1510, 2009\.
- Zhang et al\. \[2017\]C\. Zhang, S\. Bengio, M\. Hardt, B\. Recht, and O\. Vinyals\.Understanding deep learning requires rethinking generalization\.In*Proceedings of the 5th International Conference on Learning Representations*, 2017\.
- Zhang et al\. \[2024\]C\. Zhang, D\. Zou, and Y\. Cao\.The implicit bias of Adam on separable data\.*Advances in Neural Information Processing Systems*, 37:23988–24021, 2024\.

## Appendix ATechnicalities in Section[3\.1](https://arxiv.org/html/2608.04382#S3.SS1)

#### Tangential errorδ¯≤1−γ\\bar\{\\delta\}\\leq 1\-\\gamma\.

Recall the quantityδ¯\\bar\{\\delta\}defined in \([9](https://arxiv.org/html/2608.04382#S3.E9)\):

δ¯≔1−1\+2​γ​γ¯−γ¯2−\(1\+2​γ​γ¯−γ¯2\)2−4​γ22,\\bar\{\\delta\}\\coloneqq 1\-\\sqrt\{\\frac\{1\+2\\gamma\\bar\{\\gamma\}\-\\bar\{\\gamma\}^\{2\}\-\\sqrt\{\(1\+2\\gamma\\bar\{\\gamma\}\-\\bar\{\\gamma\}^\{2\}\)^\{2\}\-4\\gamma^\{2\}\}\}\{2\}\},which is the lowest possible error tolerance of the tangential alignment in the early stage\. This satisfiesδ¯≤1−γ\\bar\{\\delta\}\\leq 1\-\\gamma\. To showδ¯≤1−γ\\bar\{\\delta\}\\leq 1\-\\gamma, it suffices to show that

B−B2−4​γ22≥γ,where​B≔1\+2​γ​γ¯−γ¯2\\sqrt\{\\frac\{B\-\\sqrt\{B^\{2\}\-4\\gamma^\{2\}\}\}\{2\}\}\\geq\\gamma,\\text\{~~where~~\}B\\coloneqq 1\+2\\gamma\\bar\{\\gamma\}\-\\bar\{\\gamma\}^\{2\}By squaring both sides and rearranging, this is equivalent to

B−2​γ2≥B2−4​γ2\.B\-2\\gamma^\{2\}\\geq\\sqrt\{B^\{2\}\-4\\gamma^\{2\}\}\.Squaring again \(noting thatB−2​γ2≥0B\-2\\gamma^\{2\}\\geq 0; can be confirmed viaγ≤γ¯≤1\\gamma\\leq\\bar\{\\gamma\}\\leq 1\), we obtain

\(B−2​γ2\)2≥B2−4​γ2\.\(B\-2\\gamma^\{2\}\)^\{2\}\\geq B^\{2\}\-4\\gamma^\{2\}\.Rearranging, we haveγ2−B\+1≥0\\gamma^\{2\}\-B\+1\\geq 0, for which we substituteB=1\+2​γ​γ¯−γ¯2B=1\+2\\gamma\\bar\{\\gamma\}\-\\bar\{\\gamma\}^\{2\}and have

γ2−\(1\+2​γ​γ¯−γ¯2\)\+1=\(γ−γ¯\)2≥0\.\\gamma^\{2\}\-\(1\+2\\gamma\\bar\{\\gamma\}\-\\bar\{\\gamma\}^\{2\}\)\+1=\(\\gamma\-\\bar\{\\gamma\}\)^\{2\}\\geq 0\.This inequality is always true, and thus we confirmedδ¯≤1−γ\\bar\{\\delta\}\\leq 1\-\\gamma\.

#### δ¯\\bar\{\\delta\}is monotonically decreasing inγ¯\\bar\{\\gamma\}\.

Using the same notationB=1\+2​γ​γ¯−γ¯2B=1\+2\\gamma\\bar\{\\gamma\}\-\\bar\{\\gamma\}^\{2\}, we can write

δ¯=1−B−B2−4​γ22\.\\bar\{\\delta\}=1\-\\sqrt\{\\frac\{B\-\\sqrt\{B^\{2\}\-4\\gamma^\{2\}\}\}\{2\}\}\.To showδ¯\\bar\{\\delta\}is decreasing inγ¯\\bar\{\\gamma\}, it suffices to show thatg​\(γ¯\)≔B−B2−4​γ22g\(\\bar\{\\gamma\}\)\\coloneqq\\frac\{B\-\\sqrt\{B^\{2\}\-4\\gamma^\{2\}\}\}\{2\}is increasing inγ¯\\bar\{\\gamma\}\.

First, note that∂B∂γ¯=2​γ−2​γ¯=2​\(γ−γ¯\)≤0\\frac\{\\partial B\}\{\\partial\\bar\{\\gamma\}\}=2\\gamma\-2\\bar\{\\gamma\}=2\(\\gamma\-\\bar\{\\gamma\}\)\\leq 0sinceγ≤γ¯\\gamma\\leq\\bar\{\\gamma\}\. Also,B\>0B\>0forγ≤γ¯≤1\\gamma\\leq\\bar\{\\gamma\}\\leq 1can be shown by completing the square inγ¯\\bar\{\\gamma\}\. Now we compute

∂g∂γ¯=12​\(∂B∂γ¯−BB2−4​γ2​∂B∂γ¯\)=12​∂B∂γ¯​\(1−BB2−4​γ2\)\.\\frac\{\\partial g\}\{\\partial\\bar\{\\gamma\}\}=\\frac\{1\}\{2\}\\left\(\\frac\{\\partial B\}\{\\partial\\bar\{\\gamma\}\}\-\\frac\{B\}\{\\sqrt\{B^\{2\}\-4\\gamma^\{2\}\}\}\\frac\{\\partial B\}\{\\partial\\bar\{\\gamma\}\}\\right\)=\\frac\{1\}\{2\}\\frac\{\\partial B\}\{\\partial\\bar\{\\gamma\}\}\\left\(1\-\\frac\{B\}\{\\sqrt\{B^\{2\}\-4\\gamma^\{2\}\}\}\\right\)\.SinceB\>0B\>0andB2−4​γ2<B2B^\{2\}\-4\\gamma^\{2\}<B^\{2\}, we haveB2−4​γ2<B\\sqrt\{B^\{2\}\-4\\gamma^\{2\}\}<B, which impliesBB2−4​γ2\>1\\frac\{B\}\{\\sqrt\{B^\{2\}\-4\\gamma^\{2\}\}\}\>1\. Thusggis increasing inγ¯\\bar\{\\gamma\}, which impliesδ¯=1−g\\bar\{\\delta\}=1\-\\sqrt\{g\}is decreasing inγ¯\\bar\{\\gamma\}\.

This fact indicates that the best tangential error that we can achieve in the early stage isδ¯=1−g​\(1\)=1−γ\\bar\{\\delta\}=1\-\\sqrt\{g\(1\)\}=1\-\\sqrt\{\\gamma\}if a given dataset has a large average dataset marginγ¯≈1\\bar\{\\gamma\}\\approx 1\. In this case, we can achieve up to⟨𝐮,𝐮∗⟩≥γ\\left\\langle\{\\mathbf\{u\}\},\{\\mathbf\{u\}\_\{\*\}\}\\right\\rangle\\geq\\sqrt\{\\gamma\}\.

## Appendix BAuxiliary lemmas

###### Lemma 15\.

Fora,b,c\>0a,b,c\>0, assumea​c≤1ac\\leq 1\. Then, we have

∫0td​s\(a\+s\)​\(b\+log⁡\(1\+c​s\)\)≥1\(1\+a​c\)​\(b\+log⁡2\)\+12​log⁡\(1\+log⁡c​tb\+log⁡2\)\.\\int\_\{0\}^\{t\}\\frac\{\\mathrm\{d\}s\}\{\(a\+s\)\(b\+\\log\(1\+cs\)\)\}\\geq\\frac\{1\}\{\(1\+ac\)\(b\+\\log 2\)\}\+\\frac\{1\}\{2\}\\log\\left\(1\+\\frac\{\\log ct\}\{b\+\\log 2\}\\right\)\.

###### Proof\.\.

We split the interval of integration intos∈\[0,1/c\]s\\in\[0,1/c\]ands∈\(1/c,t\]s\\in\(1/c,t\]\. For the first interval, we have

∫01/cd​s\(a\+s\)​\(b\+log⁡\(1\+c​s\)\)≥∫01/cd​s\(a\+1c\)​\(b\+log⁡2\)=1\(1\+a​c\)​\(b\+log⁡2\)\.\\int\_\{0\}^\{1/c\}\\frac\{\\mathrm\{d\}s\}\{\(a\+s\)\(b\+\\log\(1\+cs\)\)\}\\geq\\int\_\{0\}^\{1/c\}\\frac\{\\mathrm\{d\}s\}\{\(a\+\\frac\{1\}\{c\}\)\(b\+\\log 2\)\}=\\frac\{1\}\{\(1\+ac\)\(b\+\\log 2\)\}\.For the second interval, by notinga≤1/c≤sa\\leq 1/c\\leq s, we have

∫1/ctd​s\(a\+s\)​\(b\+log⁡\(1\+c​s\)\)\\displaystyle\\int\_\{1/c\}^\{t\}\\frac\{\\mathrm\{d\}s\}\{\(a\+s\)\(b\+\\log\(1\+cs\)\)\}≥∫1/ctd​s2​s​\(b\+log⁡\(2​c​s\)\)\\displaystyle\\geq\\int\_\{1/c\}^\{t\}\\frac\{\\mathrm\{d\}s\}\{2s\(b\+\\log\(2cs\)\)\}=12​∫1/ctd​ss​\(b¯\+log⁡s\)\\displaystyle=\\frac\{1\}\{2\}\\int\_\{1/c\}^\{t\}\\frac\{\\mathrm\{d\}s\}\{s\(\\bar\{b\}\+\\log s\)\}\(b¯≔b\+log⁡2​c\\bar\{b\}\\coloneqq b\+\\log 2c\)=12​∫b¯−log⁡cb¯\+log⁡td​ςς\\displaystyle=\\frac\{1\}\{2\}\\int\_\{\\bar\{b\}\-\\log c\}^\{\\bar\{b\}\+\\log t\}\\frac\{\\mathrm\{d\}\\varsigma\}\{\\varsigma\}=12​log⁡\(b¯\+log⁡tb¯−log⁡c\),\\displaystyle=\\frac\{1\}\{2\}\\log\\left\(\\frac\{\\bar\{b\}\+\\log t\}\{\\bar\{b\}\-\\log c\}\\right\),where we use the change of variableς≔b¯\+log⁡s\\varsigma\\coloneqq\\bar\{b\}\+\\log s\. Adding the two terms concludes the desired lower bound\. ∎

###### Lemma 16\.

Fix𝐦∈ℝn\\mathbf\{m\}\\in\\mathbb\{R\}^\{n\}and define𝛂∈Δn−1\\bm\{\\alpha\}\\in\\Delta^\{n\-1\}byαi≔e−β​mi/∑j=1ne−β​mj\\alpha\_\{i\}\\coloneqq e^\{\-\\beta m\_\{i\}\}/\\sum\_\{j=1\}^\{n\}e^\{\-\\beta m\_\{j\}\}\. Letf​\(β\)≔∑i=1nαi​mif\(\\beta\)\\coloneqq\\sum\_\{i=1\}^\{n\}\\alpha\_\{i\}m\_\{i\}forβ≥0\\beta\\geq 0\. Then,ffis monotonically nonincreasing\.

###### Proof\.\.

We computef′​\(β\)=∑i=1nmi⋅αi′​\(β\)f^\{\\prime\}\(\\beta\)=\\sum\_\{i=1\}^\{n\}m\_\{i\}\\cdot\\alpha\_\{i\}^\{\\prime\}\(\\beta\), whereαi′​\(β\)=d​αi/d​β\\alpha\_\{i\}^\{\\prime\}\(\\beta\)=\\mathrm\{d\}\{\\alpha\_\{i\}\}/\\mathrm\{d\}\{\\beta\}\. The derivative ofαi\\alpha\_\{i\}is

d​αid​β\\displaystyle\\frac\{\\mathrm\{d\}\{\\alpha\_\{i\}\}\}\{\\mathrm\{d\}\{\\beta\}\}=−mi​e−β​mi​∑j=1ne−β​mj−e−β​mi​∑j=1n\(−mj\)​e−β​mj\(∑j=1ne−β​mj\)2\\displaystyle=\\frac\{\-m\_\{i\}e^\{\-\\beta m\_\{i\}\}\\sum\_\{j=1\}^\{n\}e^\{\-\\beta m\_\{j\}\}\-e^\{\-\\beta m\_\{i\}\}\\sum\_\{j=1\}^\{n\}\(\-m\_\{j\}\)e^\{\-\\beta m\_\{j\}\}\}\{\(\\sum\_\{j=1\}^\{n\}e^\{\-\\beta m\_\{j\}\}\)^\{2\}\}=e−β​mi∑j=1ne−β​mj⋅−mi​∑j=1ne−β​mj\+∑j=1nmj​e−β​mj∑j=1ne−β​mj\\displaystyle=\\frac\{e^\{\-\\beta m\_\{i\}\}\}\{\\sum\_\{j=1\}^\{n\}e^\{\-\\beta m\_\{j\}\}\}\\cdot\\frac\{\-m\_\{i\}\\sum\_\{j=1\}^\{n\}e^\{\-\\beta m\_\{j\}\}\+\\sum\_\{j=1\}^\{n\}m\_\{j\}e^\{\-\\beta m\_\{j\}\}\}\{\\sum\_\{j=1\}^\{n\}e^\{\-\\beta m\_\{j\}\}\}=αi​\(−mi\+∑j=1nαj​mj\)\\displaystyle=\\alpha\_\{i\}\\Bigl\(\-m\_\{i\}\+\\sum\_\{j=1\}^\{n\}\\alpha\_\{j\}m\_\{j\}\\Bigr\)=αi​\(f​\(β\)−mi\)\.\\displaystyle=\\alpha\_\{i\}\(f\(\\beta\)\-m\_\{i\}\)\.Therefore,

f′​\(β\)=∑i=1nmi​αi​\(f​\(β\)−mi\)=−∑i=1nαi​mi2\+\(∑i=1nαi​mi\)2=−Var𝜶​\(mi\)≤0,\\displaystyle f^\{\\prime\}\(\\beta\)=\\sum\_\{i=1\}^\{n\}m\_\{i\}\\alpha\_\{i\}\(f\(\\beta\)\-m\_\{i\}\)=\-\\sum\_\{i=1\}^\{n\}\\alpha\_\{i\}m\_\{i\}^\{2\}\+\\Bigl\(\\sum\_\{i=1\}^\{n\}\\alpha\_\{i\}m\_\{i\}\\Bigr\)^\{2\}=\-\\mathrm\{Var\}\_\{\\bm\{\\alpha\}\}\(m\_\{i\}\)\\leq 0,whereVar𝜶​\(mi\)≔∑i=1nαi​mi2−\(∑i=1nαi​mi\)2\\mathrm\{Var\}\_\{\\bm\{\\alpha\}\}\(m\_\{i\}\)\\coloneqq\\sum\_\{i=1\}^\{n\}\\alpha\_\{i\}m\_\{i\}^\{2\}\-\(\\sum\_\{i=1\}^\{n\}\\alpha\_\{i\}m\_\{i\}\)^\{2\}denotes the variance ofmim\_\{i\}under the distribution𝜶∈Δn−1\\bm\{\\alpha\}\\in\\Delta^\{n\-1\}\. Sincef′​\(β\)≤0f^\{\\prime\}\(\\beta\)\\leq 0for allβ≥0\\beta\\geq 0, the functionffis monotonically nonincreasing\. ∎

## Appendix CContinuous\-time lower bound proof

Here we show the complete proof of Theorem[10](https://arxiv.org/html/2608.04382#Thmtheorem10)\. From \([4](https://arxiv.org/html/2608.04382#S3.E4)\), we can derive the following differential inequality:

V˙​\(t\)=−L​\(𝐰​\(t\)\)r​\(t\)​⟨Z​𝜶,P𝐮​\(t\)⟂​𝐮∗⟩≥−L​\(𝐰​\(t\)\)r​\(t\)​‖Z​𝜶‖​‖P𝐮​\(t\)⟂​𝐮∗‖≥−L​\(𝐰​\(t\)\)r​\(t\)​2​V​\(t\),\\dot\{V\}\(t\)=\-\\frac\{L\(\\mathbf\{w\}\(t\)\)\}\{r\(t\)\}\\left\\langle\{Z\\bm\{\\alpha\}\},\{P\_\{\\mathbf\{u\}\(t\)\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\}\\right\\rangle\\geq\-\\frac\{L\(\\mathbf\{w\}\(t\)\)\}\{r\(t\)\}\\\|Z\\bm\{\\alpha\}\\\|\\\|P\_\{\\mathbf\{u\}\(t\)\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\\\|\\geq\-\\frac\{L\(\\mathbf\{w\}\(t\)\)\}\{r\(t\)\}\\sqrt\{2V\(t\)\},where we used‖P𝐮​\(t\)⟂​𝐮∗‖2=‖𝐮∗‖2−⟨𝐮​\(t\),𝐮∗⟩2=1−\(1−V​\(t\)\)2=2​V​\(t\)−V​\(t\)2≤2​V​\(t\)\\\|P\_\{\\mathbf\{u\}\(t\)\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\\\|^\{2\}=\\\|\\mathbf\{u\}\_\{\*\}\\\|^\{2\}\-\\left\\langle\{\\mathbf\{u\}\(t\)\},\{\\mathbf\{u\}\_\{\*\}\}\\right\\rangle^\{2\}=1\-\(1\-V\(t\)\)^\{2\}=2V\(t\)\-V\(t\)^\{2\}\\leq 2V\(t\)\. Separating variables, forV\>0V\>0we have

d​VV≥−2​L​\(t\)r​\(t\)​d​t↝integrate from0totV​\(t\)≥V​\(0\)−12​∫0tL​\(s\)r​\(s\)​ds\.\\frac\{\\mathrm\{d\}V\}\{\\sqrt\{V\}\}\\geq\-\\sqrt\{2\}\\frac\{L\(t\)\}\{r\(t\)\}\\mathrm\{d\}t\\quad\\overset\{\\text\{integrate from $0$ to $t$\}\}\{\\leadsto\}\\quad\\sqrt\{V\(t\)\}\\geq\\sqrt\{V\(0\)\}\-\\frac\{1\}\{\\sqrt\{2\}\}\\int\_\{0\}^\{t\}\\frac\{L\(s\)\}\{r\(s\)\}\\mathrm\{d\}s\.We now bound the integral using the lower bounds onr​\(t\)r\(t\)from Lemma[4](https://arxiv.org/html/2608.04382#Thmtheorem4)\. By definingT0≔L\(0\)−1γ−2\[exp\(ρ\+log⁡L​\(0\)1\+γ−2\)−1\]T\_\{0\}\\coloneqq\{L\(0\)^\{\-1\}\\gamma^\{\-2\}\}\\bigr\[\\exp\\bigr\(\\frac\{\\rho\+\\log L\(0\)\}\{1\+\\gamma^\{\-2\}\}\\bigl\)\-1\\bigl\], we have

r​\(t\)≥\{ρ−γ−2​log⁡\(1\+L​\(0\)​γ2​t\)ift∈\[0,T0\],log⁡\(L​\(0\)−1\+γ2​t\)ift≥T0,r\(t\)\\geq\\begin\{cases\}\\rho\-\\gamma^\{\-2\}\\log\(1\+L\(0\)\\gamma^\{2\}t\)&\\text\{if $t\\in\[0,T\_\{0\}\]$,\}\\\\ \\log\(L\(0\)^\{\-1\}\+\\gamma^\{2\}t\)&\\text\{if $t\\geq T\_\{0\}$,\}\\end\{cases\}\(12\)Att=T0t=T\_\{0\}, we haver​\(T0\)≥ρ​γ2−log⁡L​\(0\)1\+γ2≕r¯\>0r\(T\_\{0\}\)\\geq\\frac\{\\rho\\gamma^\{2\}\-\\log L\(0\)\}\{1\+\\gamma^\{2\}\}\\eqqcolon\\underline\{r\}\>0, where the strict positivity holds from the assumptionL​\(0\)<eρ​γ2L\(0\)<e^\{\\rho\\gamma^\{2\}\}\. We split the integral into two intervalss∈\[0,T0\]∪\(T0,t\)s\\in\[0,T\_\{0\}\]\\cup\(T\_\{0\},t\)\. For the first intervals∈\[0,T0\]s\\in\[0,T\_\{0\}\], we combiner​\(s\)≥r¯r\(s\)\\geq\\underline\{r\}with the upper bound ofL​\(s\)L\(s\)from Lemma[4](https://arxiv.org/html/2608.04382#Thmtheorem4)to have

∫0T0L​\(s\)r​\(s\)​ds≤1r¯​∫0T0d​sL​\(0\)−1\+γ2​s=1r¯​γ2​log⁡\(L​\(0\)−1\+γ2​T0L​\(0\)−1\)=ρ\+log⁡L​\(0\)ρ​γ2−log⁡L​\(0\)\.\\int\_\{0\}^\{T\_\{0\}\}\\frac\{L\(s\)\}\{r\(s\)\}\\mathrm\{d\}s\\leq\\frac\{1\}\{\\underline\{r\}\}\\int\_\{0\}^\{T\_\{0\}\}\\frac\{\\mathrm\{d\}s\}\{L\(0\)^\{\-1\}\+\\gamma^\{2\}s\}=\\frac\{1\}\{\\underline\{r\}\\gamma^\{2\}\}\\log\\left\(\\frac\{L\(0\)^\{\-1\}\+\\gamma^\{2\}T\_\{0\}\}\{L\(0\)^\{\-1\}\}\\right\)=\\frac\{\\rho\+\\log L\(0\)\}\{\\rho\\gamma^\{2\}\-\\log L\(0\)\}\.For the second intervals∈\(T0,t\]s\\in\(T\_\{0\},t\], we combine \([12](https://arxiv.org/html/2608.04382#A3.E12)\) with the upper bound ofL​\(s\)L\(s\)to have

∫T0tL​\(s\)r​\(s\)​ds≤∫T0td​sh​\(s\)​log⁡h​\(s\)=1γ2​∫h​\(T0\)h​\(t\)d​ςς=1γ2​log⁡\(log⁡\(L​\(0\)−1\+γ2​t\)ρ​γ2−log⁡L​\(0\)1\+γ−2\),\\int\_\{T\_\{0\}\}^\{t\}\\frac\{L\(s\)\}\{r\(s\)\}\\mathrm\{d\}s\\leq\\int\_\{T\_\{0\}\}^\{t\}\\frac\{\\mathrm\{d\}s\}\{h\(s\)\\log h\(s\)\}=\\frac\{1\}\{\\gamma^\{2\}\}\\int\_\{h\(T\_\{0\}\)\}^\{h\(t\)\}\\frac\{\\mathrm\{d\}\\varsigma\}\{\\varsigma\}=\\frac\{1\}\{\\gamma^\{2\}\}\\log\\biggl\(\\frac\{\\log\(L\(0\)^\{\-1\}\+\\gamma^\{2\}t\)\}\{\\frac\{\\rho\\gamma^\{2\}\-\\log L\(0\)\}\{1\+\\gamma^\{\-2\}\}\}\\biggr\),where we use the change of variableς≔h​\(s\)≔log⁡\(L​\(0\)−1\+γ2​s\)\\varsigma\\coloneqq h\(s\)\\coloneqq\\log\(L\(0\)^\{\-1\}\+\\gamma^\{2\}s\)\. Plugging them back yields

V​\(t\)≥V​\(0\)−12​\[C¯0\+1γ2​log⁡\(log⁡\(L​\(0\)−1\+γ2​t\)\)\]whereC¯0≔ρ\+log⁡L​\(0\)ρ​γ2−log⁡L​\(0\)−1γ2​log⁡\(ρ​γ2−log⁡L​\(0\)1\+γ−2\)\.\\begin\{aligned\} \\sqrt\{V\(t\)\}&\\geq\\sqrt\{V\(0\)\}\-\\frac\{1\}\{\\sqrt\{2\}\}\\left\[\\bar\{C\}\_\{0\}\+\\frac\{1\}\{\\gamma^\{2\}\}\\log\(\\log\(L\(0\)^\{\-1\}\+\\gamma^\{2\}t\)\)\\right\]\\\\ \\text\{where\}\\quad&\\bar\{C\}\_\{0\}\\coloneqq\\frac\{\\rho\+\\log L\(0\)\}\{\\rho\\gamma^\{2\}\-\\log L\(0\)\}\-\\frac\{1\}\{\\gamma^\{2\}\}\\log\\left\(\\frac\{\\rho\\gamma^\{2\}\-\\log L\(0\)\}\{1\+\\gamma^\{\-2\}\}\\right\)\\end\{aligned\}\.Squaring and relaxing the bound by\(a−b\)2≥a2−2​a​b\(a\-b\)^\{2\}\\geq a^\{2\}\-2ab, we have the lower bound \([11](https://arxiv.org/html/2608.04382#S3.E11)\), whereC0≔2​V​\(0\)​C¯0C\_\{0\}\\coloneqq\\sqrt\{2V\(0\)\}\\bar\{C\}\_\{0\}\. Lastly, to ensureV​\(t\)≥δV\(t\)\\geq\\delta, we need\(lower bound\)≥δ\\text\{\(lower bound\)\}\\geq\\delta, which yields

t<1γ2​exp⁡\[exp⁡\(γ2​\(V​\(0\)−C0−δ\)2​V​\(0\)\)\]−1L​\(0\)​γ2\.t<\\frac\{1\}\{\\gamma^\{2\}\}\\exp\\biggl\[\\exp\\biggl\(\\frac\{\\gamma^\{2\}\(V\(0\)\-C\_\{0\}\-\\delta\)\}\{\\sqrt\{2V\(0\)\}\}\\biggr\)\\biggr\]\-\\frac\{1\}\{L\(0\)\\gamma^\{2\}\}\.

## Appendix DExtension to logistic loss

In this section, we show the extension of the weak alignment analysis to the logistic loss\. Specifically, we show Lemmas[11](https://arxiv.org/html/2608.04382#Thmtheorem11)and[12](https://arxiv.org/html/2608.04382#Thmtheorem12)\. Recall notations: with the marginmi​\(t\)≔⟨𝐰​\(t\),𝐳i⟩m\_\{i\}\(t\)\\coloneqq\\left\\langle\{\\mathbf\{w\}\(t\)\},\{\\mathbf\{z\}\_\{i\}\}\\right\\rangle, we write

qi​\(t\)≔−ℓ′​\(mi​\(t\)\),G​\(t\)≔1n​∑i=1nqi​\(t\),and​αi​\(t\)≔qi​\(t\)∑j=1nqj​\(t\)\.q\_\{i\}\(t\)\\coloneqq\-\\ell^\{\\prime\}\(m\_\{i\}\(t\)\),\\quad G\(t\)\\coloneqq\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}q\_\{i\}\(t\),\\text\{~~and~~\}\\alpha\_\{i\}\(t\)\\coloneqq\\frac\{q\_\{i\}\(t\)\}\{\\sum\_\{j=1\}^\{n\}q\_\{j\}\(t\)\}\.
See[11](https://arxiv.org/html/2608.04382#Thmtheorem11)

###### Proof\.\.

We first derive the upper bound of‖∇L​\(t\)‖\\\|\\nabla L\(t\)\\\|:

‖∇L​\(t\)‖=‖1n​∑i=1n\(−ℓ′​\(mi​\(t\)\)\)⋅𝐳i‖=G​\(t\)⋅‖∑i=1nqi​\(t\)n​G​\(t\)​𝐳i‖=G​\(t\)⋅‖Z​𝜶​\(t\)‖≤G​\(t\)\.\\\|\\nabla L\(t\)\\\|=\\left\\\|\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\(\-\\ell^\{\\prime\}\(m\_\{i\}\(t\)\)\)\\cdot\\mathbf\{z\}\_\{i\}\\right\\\|=G\(t\)\\cdot\\left\\\|\\sum\_\{i=1\}^\{n\}\\frac\{q\_\{i\}\(t\)\}\{nG\(t\)\}\\mathbf\{z\}\_\{i\}\\right\\\|=G\(t\)\\cdot\\\|Z\\bm\{\\alpha\}\(t\)\\\|\\leq G\(t\)\.Furthermore, we have

qi​\(t\)=e−mi​\(t\)1\+e−mi​\(t\)=eℓ​\(mi​\(t\)\)−1eℓ​\(mi​\(t\)\)=1−e−ℓ​\(mi​\(t\)\)≤ℓ​\(mi​\(t\)\),q\_\{i\}\(t\)=\\frac\{e^\{\-m\_\{i\}\(t\)\}\}\{1\+e^\{\-m\_\{i\}\(t\)\}\}=\\frac\{e^\{\\ell\(m\_\{i\}\(t\)\)\}\-1\}\{e^\{\\ell\(m\_\{i\}\(t\)\)\}\}=1\-e^\{\-\\ell\(m\_\{i\}\(t\)\)\}\\leq\\ell\(m\_\{i\}\(t\)\),which impliesG​\(t\)≤L​\(t\)G\(t\)\\leq L\(t\)\.

To derive the lower bound ofG​\(t\)G\(t\), we use the Weierstrass product inequality∏iai≥1−∑i\(1−ai\)\\prod\_\{i\}a\_\{i\}\\geq 1\-\\sum\_\{i\}\(1\-a\_\{i\}\)forai∈\[0,1\]a\_\{i\}\\in\[0,1\]: withai=e−ℓ​\(mi​\(t\)\)a\_\{i\}=e^\{\-\\ell\(m\_\{i\}\(t\)\)\}, we have

n​G​\(t\)=∑i=1n\(1−e−ℓ​\(mi​\(t\)\)\)≥1−∏i=1ne−ℓ​\(mi​\(t\)\)=1−e−n​L​\(t\),nG\(t\)=\\sum\_\{i=1\}^\{n\}\(1\-e^\{\-\\ell\(m\_\{i\}\(t\)\)\}\)\\geq 1\-\\prod\_\{i=1\}^\{n\}e^\{\-\\ell\(m\_\{i\}\(t\)\)\}=1\-e^\{\-nL\(t\)\},which implies

G​\(t\)L​\(t\)≥1−e−n​L​\(t\)n​L​\(t\)\.\\frac\{G\(t\)\}\{L\(t\)\}\\geq\\frac\{1\-e^\{\-nL\(t\)\}\}\{nL\(t\)\}\.Note that the mapx↦1−e−xxx\\mapsto\\frac\{1\-e^\{\-x\}\}\{x\}is decreasing on\(0,∞\)\(0,\\infty\)\. AsL​\(t\)L\(t\)is nonincreasing due toL˙=−‖∇L‖2≤0\\dot\{L\}=\-\\\|\\nabla L\\\|^\{2\}\\leq 0, we have

G​\(t\)L​\(t\)≥1−e−n​L​\(0\)n​L​\(0\)=ρ¯0\.\\frac\{G\(t\)\}\{L\(t\)\}\\geq\\frac\{1\-e^\{\-nL\(0\)\}\}\{nL\(0\)\}=\\bar\{\\rho\}\_\{0\}\.Finally, we haveG≥ρ¯0​LG\\geq\\bar\{\\rho\}\_\{0\}L\. The last corollary is immediate by noting that‖∇L‖=G​‖Z​𝜶‖≥γ​G≥γℓ​L\\\|\\nabla L\\\|=G\\\|Z\\bm\{\\alpha\}\\\|\\geq\\gamma G\\geq\\gamma\_\{\\ell\}L\. ∎

See[12](https://arxiv.org/html/2608.04382#Thmtheorem12)

###### Proof\.\.

Throughout the proof, we fixttand drop its dependency, e\.g\.,mim\_\{i\}indicatesmi​\(t\)=⟨𝐰​\(t\),𝐳i⟩m\_\{i\}\(t\)=\\left\\langle\{\\mathbf\{w\}\(t\)\},\{\\mathbf\{z\}\_\{i\}\}\\right\\rangle\. Fix a direction𝐮∈𝕊d−1\\mathbf\{u\}\\in\\mathbb\{S\}^\{d\-1\}and write the normalized marginMi≔⟨𝐮,𝐳i⟩M\_\{i\}\\coloneqq\\left\\langle\{\\mathbf\{u\}\},\{\\mathbf\{z\}\_\{i\}\}\\right\\rangleindependent ofrr, so thatmi=r​Mim\_\{i\}=rM\_\{i\}and

αi=qi∑jqj=σ​\(−r​Mi\)∑jσ​\(−r​Mj\),where​σ​\(m\)=11\+e−m\.\\alpha\_\{i\}=\\frac\{q\_\{i\}\}\{\\sum\_\{j\}q\_\{j\}\}=\\frac\{\\sigma\(\-rM\_\{i\}\)\}\{\\sum\_\{j\}\\sigma\(\-rM\_\{j\}\)\},\\text\{~~where~~\}\\sigma\(m\)=\\frac\{1\}\{1\+e^\{\-m\}\}\.Letφ​\(m\)≔ℓ′′​\(m\)/\(−ℓ′​\(m\)\)=σ​\(m\)\\varphi\(m\)\\coloneqq\\ell^\{\\prime\\prime\}\(m\)/\(\-\\ell^\{\\prime\}\(m\)\)=\\sigma\(m\), which is nonnegative and nondecreasing\. DefineM¯​\(r\)≔⟨𝐮,Z​𝜶⟩\\bar\{M\}\(r\)\\coloneqq\\left\\langle\{\\mathbf\{u\}\},\{Z\\bm\{\\alpha\}\}\\right\\rangle, which depends onrrthrough𝜶\\bm\{\\alpha\}\. The goal of this lemma is to showM¯​\(r\)≤\[M¯​\(0\)\]\+\\bar\{M\}\(r\)\\leq\[\\bar\{M\}\(0\)\]\_\{\+\}forr≥0r\\geq 0\. With an abuse of notation, we hereafter often write𝜶​\(r\)\\bm\{\\alpha\}\(r\)andqi​\(r\)q\_\{i\}\(r\)to denote the dependency onrr\.

By elementary algebra, we have

αi′​\(r\)\\displaystyle\\alpha\_\{i\}^\{\\prime\}\(r\)=qi′​\(r\)​∑jqj​\(r\)−qi​\(r\)​∑jqj′​\(r\)\(∑jqj​\(r\)\)2\\displaystyle=\\frac\{q\_\{i\}^\{\\prime\}\(r\)\\sum\_\{j\}q\_\{j\}\(r\)\-q\_\{i\}\(r\)\\sum\_\{j\}q\_\{j\}^\{\\prime\}\(r\)\}\{\(\\sum\_\{j\}q\_\{j\}\(r\)\)^\{2\}\}=αi​\(r\)​qi′​\(r\)qi​\(r\)−αi​\(r\)​∑jqj′​\(r\)∑jqj​\(r\)\\displaystyle=\\alpha\_\{i\}\(r\)\\frac\{q\_\{i\}^\{\\prime\}\(r\)\}\{q\_\{i\}\(r\)\}\-\\alpha\_\{i\}\(r\)\\frac\{\\sum\_\{j\}q\_\{j\}^\{\\prime\}\(r\)\}\{\\sum\_\{j\}q\_\{j\}\(r\)\}=−αi​\(r\)​φ​\(mi\)​Mi\+αi​\(r\)​∑j=1nαj​\(r\)​φ​\(mj\)​Mj\.\\displaystyle=\-\\alpha\_\{i\}\(r\)\\varphi\(m\_\{i\}\)M\_\{i\}\+\\alpha\_\{i\}\(r\)\\sum\_\{j=1\}^\{n\}\\alpha\_\{j\}\(r\)\\varphi\(m\_\{j\}\)M\_\{j\}\.Then, by notingM¯=∑iαi​Mi\\bar\{M\}=\\sum\_\{i\}\\alpha\_\{i\}M\_\{i\}, we have

M¯′​\(r\)\\displaystyle\\bar\{M\}^\{\\prime\}\(r\)=∑i=1nαi′​\(r\)​Mi\\displaystyle=\\sum\_\{i=1\}^\{n\}\\alpha\_\{i\}^\{\\prime\}\(r\)M\_\{i\}=−∑i=1nαi​\(r\)​φ​\(mi\)​Mi2\+M¯​\(r\)​∑i=1nαi​\(r\)​φ​\(mi\)​Mi\\displaystyle=\-\\sum\_\{i=1\}^\{n\}\\alpha\_\{i\}\(r\)\\varphi\(m\_\{i\}\)M\_\{i\}^\{2\}\+\\bar\{M\}\(r\)\\sum\_\{i=1\}^\{n\}\\alpha\_\{i\}\(r\)\\varphi\(m\_\{i\}\)M\_\{i\}=−∑i=1nαi​\(r\)​φ​\(mi\)​\(Mi−M¯​\(r\)\)2⏟≕A​\(r\)−M¯​\(r\)​∑i=1nαi​\(r\)​φ​\(mi\)​\(Mi−M¯​\(r\)\)⏟≕B​\(r\)\\displaystyle=\-\\underbrace\{\\sum\_\{i=1\}^\{n\}\\alpha\_\{i\}\(r\)\\varphi\(m\_\{i\}\)\(M\_\{i\}\-\\bar\{M\}\(r\)\)^\{2\}\}\_\{\\eqqcolon A\(r\)\}\-\\bar\{M\}\(r\)\\underbrace\{\\sum\_\{i=1\}^\{n\}\\alpha\_\{i\}\(r\)\\varphi\(m\_\{i\}\)\(M\_\{i\}\-\\bar\{M\}\(r\)\)\}\_\{\\eqqcolon B\(r\)\}=−A​\(r\)−M¯​\(r\)​B​\(r\)\.\\displaystyle=\-A\(r\)\-\\bar\{M\}\(r\)B\(r\)\.Here, we haveA​\(r\)≥0A\(r\)\\geq 0andB​\(r\)≥0B\(r\)\\geq 0, where the latter satisfiesB​\(r\)=Cov𝜶​\(φ​\(mi\),Mi\)≥0B\(r\)=\\mathrm\{Cov\}\_\{\\bm\{\\alpha\}\}\(\\varphi\(m\_\{i\}\),M\_\{i\}\)\\geq 0becauseφ​\(mi\)=σ​\(r​Mi\)\\varphi\(m\_\{i\}\)=\\sigma\(rM\_\{i\}\)is increasing inMiM\_\{i\}, namely,φ​\(mi\)\\varphi\(m\_\{i\}\)andMiM\_\{i\}positively correlates\. Then, we divide the cases as follows:

- •IfM¯​\(r\)\>0\\bar\{M\}\(r\)\>0, then we haveM¯′​\(r\)≤0\\bar\{M\}^\{\\prime\}\(r\)\\leq 0\.
- •IfM¯​\(r\)=0\\bar\{M\}\(r\)=0, then we haveM¯′​\(r\)≤0\\bar\{M\}^\{\\prime\}\(r\)\\leq 0\.
- •\(IfM¯​\(r\)<0\\bar\{M\}\(r\)<0, the sign ofM¯′​\(r\)\\bar\{M\}^\{\\prime\}\(r\)is undetermined\.\)

Thus, asrrincreases,M¯​\(r\)\\bar\{M\}\(r\)never becomes positive for anyr≥r0r\\geq r\_\{0\}once we haveM¯​\(r0\)=0\\bar\{M\}\(r\_\{0\}\)=0, due to the continuity ofM¯\\bar\{M\}\. Together with the nonincreasing\-ness ofM¯​\(r\)\\bar\{M\}\(r\)on\{r:M¯​\(r\)\>0\}\\\{r:\\bar\{M\}\(r\)\>0\\\}, the maximum ofM¯\\bar\{M\}is attained atr=0r=0whenM¯​\(0\)\>0\\bar\{M\}\(0\)\>0, and we havesupr≥0M¯​\(r\)≤\[M¯​\(0\)\]\+\\sup\_\{r\\geq 0\}\\bar\{M\}\(r\)\\leq\[\\bar\{M\}\(0\)\]\_\{\+\}\. Therefore, we arrive at the conclusion⟨𝐮,Z​𝜶⟩≤\[M¯​\(0\)\]\+=\[⟨𝐮,𝐳¯⟩\]\+\\left\\langle\{\\mathbf\{u\}\},\{Z\\bm\{\\alpha\}\}\\right\\rangle\\leq\[\\bar\{M\}\(0\)\]\_\{\+\}=\[\\left\\langle\{\\mathbf\{u\}\},\{\\bar\{\\mathbf\{z\}\}\}\\right\\rangle\]\_\{\+\}, where𝐳¯=1n​∑i𝐳i\\bar\{\\mathbf\{z\}\}=\\frac\{1\}\{n\}\\sum\_\{i\}\\mathbf\{z\}\_\{i\}\. ∎

### D\.1Weak alignment result \(Proof of Corollary[13](https://arxiv.org/html/2608.04382#Thmtheorem13)\)

Finally, we derive the weak alignment timet=O​\(exp⁡\(exp⁡\(−δ/κ0​\(Δ\)​ρ¯0​γℓ2\)\)\)t=O\(\\exp\(\\exp\(\-\\delta/\\kappa\_\{0\}\(\\Delta\)\\bar\{\\rho\}\_\{0\}\\gamma\_\{\\ell\}^\{2\}\)\)\)\. The overall flow remains the same as Theorem[8](https://arxiv.org/html/2608.04382#Thmtheorem8), so we highlight a few important points only\. We begin with the tangential Lyapunov flow:

V˙​\(t\)=1r​\(t\)​⟨∇L​\(𝐰​\(t\)\),P𝐮​\(t\)⟂​𝐮∗⟩=−G​\(t\)r​\(t\)​⟨Z​𝜶​\(t\),P𝐮​\(t\)⟂​𝐮∗⟩\.\\dot\{V\}\(t\)=\\frac\{1\}\{r\(t\)\}\\left\\langle\{\\nabla L\(\\mathbf\{w\}\(t\)\)\},\{P\_\{\\mathbf\{u\}\(t\)\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\}\\right\\rangle=\-\\frac\{G\(t\)\}\{r\(t\)\}\\left\\langle\{Z\\bm\{\\alpha\}\(t\)\},\{P\_\{\\mathbf\{u\}\(t\)\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\}\\right\\rangle\.By following Lemma[4](https://arxiv.org/html/2608.04382#Thmtheorem4), we can get the bounds ofG​\(t\)G\(t\)andr​\(t\)r\(t\)by using the modified self\-bounding inequalities \(Lemma[11](https://arxiv.org/html/2608.04382#Thmtheorem11)\):

G​\(t\)≥ρ¯0​L​\(t\)≥ρ¯0L​\(0\)−1\+t,r​\(t\)≤ρ\+1γℓ2​log⁡\(1\+L​\(0\)​γℓ2​t\)\.G\(t\)\\geq\\bar\{\\rho\}\_\{0\}L\(t\)\\geq\\frac\{\\bar\{\\rho\}\_\{0\}\}\{L\(0\)^\{\-1\}\+t\},\\qquad r\(t\)\\leq\\rho\+\\frac\{1\}\{\\gamma\_\{\\ell\}^\{2\}\}\\log\(1\+L\(0\)\\gamma\_\{\\ell\}^\{2\}t\)\.By plugging the ansatz⟨Z​𝜶,P𝐮⟂​𝐮∗⟩≥κ\\left\\langle\{Z\\bm\{\\alpha\}\},\{P\_\{\\mathbf\{u\}\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\}\\right\\rangle\\geq\\kappaand the above bounds intoVV\-flow, we have

V˙​\(t\)≤−κ​ρ¯0​γℓ2\[L​\(0\)−1\+t\]​\[ρ​γℓ2\+log⁡\(1\+L​\(0\)​γℓ2​t\)\]\.\\dot\{V\}\(t\)\\leq\-\\frac\{\\kappa\\bar\{\\rho\}\_\{0\}\\gamma\_\{\\ell\}^\{2\}\}\{\[L\(0\)^\{\-1\}\+t\]\[\\rho\\gamma\_\{\\ell\}^\{2\}\+\\log\(1\+L\(0\)\\gamma\_\{\\ell\}^\{2\}t\)\]\}\.Solving this, we get the weak alignment timet=O​\(exp⁡\(exp⁡\(−δ/κ​ρ¯0​γℓ2\)\)\)t=O\(\\exp\(\\exp\(\-\\delta/\\kappa\\bar\{\\rho\}\_\{0\}\\gamma\_\{\\ell\}^\{2\}\)\)\)\.

Finally, the ansatz⟨Z​𝜶,P𝐮⟂​𝐮∗⟩≥κ\\left\\langle\{Z\\bm\{\\alpha\}\},\{P\_\{\\mathbf\{u\}\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\}\\right\\rangle\\geq\\kappacan be justified in the same way as Lemma[6](https://arxiv.org/html/2608.04382#Thmtheorem6): by replacing the temperature monotonicity for the exponential loss \(Lemma[16](https://arxiv.org/html/2608.04382#Thmtheorem16)\) with the logistic loss \(Lemma[12](https://arxiv.org/html/2608.04382#Thmtheorem12)\), Lemma[6](https://arxiv.org/html/2608.04382#Thmtheorem6)is fixed minimally and we have

⟨Z​𝜶,P𝐮⟂​𝐮∗⟩\\displaystyle\\left\\langle\{Z\\bm\{\\alpha\}\},\{P\_\{\\mathbf\{u\}\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\}\\right\\rangle≥γ−\(1−V\)​\[\(1−V\)​γ¯\+\(1−γ¯2\)​\(2​V−V2\)\]\+\\displaystyle\\geq\\gamma\-\(1\-V\)\[\(1\-V\)\\bar\{\\gamma\}\+\\sqrt\{\(1\-\\bar\{\\gamma\}^\{2\}\)\(2V\-V^\{2\}\)\}\]\_\{\+\}=γ−\(1−V\)​\[\(1−V\)​γ¯\+\(1−γ¯2\)​\(2​V−V2\)\]\\displaystyle=\\gamma\-\(1\-V\)\[\(1\-V\)\\bar\{\\gamma\}\+\\sqrt\{\(1\-\\bar\{\\gamma\}^\{2\}\)\(2V\-V^\{2\}\)\}\]where the positive part\[⋅\]\+\[\\cdot\]\_\{\+\}can be dropped and the equality holds throughout forV≤1V\\leq 1\. Thus, the ansatz⟨Z​𝜶,P𝐮⟂​𝐮∗⟩≥κ\\left\\langle\{Z\\bm\{\\alpha\}\},\{P\_\{\\mathbf\{u\}\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\}\\right\\rangle\\geq\\kappais justified withκ=κ0​\(Δ\)\\kappa=\\kappa\_\{0\}\(\\Delta\), giving the weak alignment timet=O​\(exp⁡\(exp⁡\(−δ/κ0​\(Δ\)​ρ¯0​γℓ2\)\)\)t=O\(\\exp\(\\exp\(\-\\delta/\\kappa\_\{0\}\(\\Delta\)\\bar\{\\rho\}\_\{0\}\\gamma\_\{\\ell\}^\{2\}\)\)\)\.

## Appendix EDiscrete\-time analysis \(full version\)

We now extend our analysis to the discrete\-time gradient descent setting with stepsizeη\>0\\eta\>0\. Recall that the gradient descent iterates𝐰​\(t\+1\)=𝐰​\(t\)−η​∇L​\(𝐰​\(t\)\)\\mathbf\{w\}\(t\+1\)=\\mathbf\{w\}\(t\)\-\\eta\\nabla L\(\\mathbf\{w\}\(t\)\)fort=0,1,2,…t=0,1,2,\\dotsand we decompose the parameter intor​\(t\)≔‖𝐰​\(t\)‖r\(t\)\\coloneqq\\\|\\mathbf\{w\}\(t\)\\\|\(radial\) and𝐮​\(t\)≔𝐰​\(t\)/r​\(t\)\\mathbf\{u\}\(t\)\\coloneqq\\mathbf\{w\}\(t\)/r\(t\)\(tangential\), where we use the same notation𝐰​\(t\)\\mathbf\{w\}\(t\),r​\(t\)r\(t\), and𝐮​\(t\)\\mathbf\{u\}\(t\)across the continuous\-time and discrete\-time cases but it should not be confused\. The key challenge in discrete\-time analysis is to control the nonlinear terms arising from the normalization𝐮​\(t\)=𝐰​\(t\)/‖𝐰​\(t\)‖\\mathbf\{u\}\(t\)=\\mathbf\{w\}\(t\)/\\\|\\mathbf\{w\}\(t\)\\\|after each update\. We follow the direct analysis approach by explicitly bounding the discrete updates and showing that they match the continuous\-time dynamics up to controlled error terms\. To this end, we impose the stepsize choice in Assumption[14](https://arxiv.org/html/2608.04382#Thmtheorem14),η≤c0​min⁡\{ρ​L​\(0\)−1,γ2\}\\eta\\leq c\_\{0\}\\min\\\{\\rho L\(0\)^\{\-1\},\\gamma^\{2\}\\\}for a sufficiently smallc0\>0c\_\{0\}\>0\. See Remark[20](https://arxiv.org/html/2608.04382#Thmtheorem20)below about how this stepsize choice controls discretization error\.

Before deriving the discrete analog of Lemma[3](https://arxiv.org/html/2608.04382#Thmtheorem3), the discrete\-time dynamics are shown first\.

###### Lemma 17\(Discrete\-time dynamics\)\.

For the gradient descent𝐰​\(t\+1\)=𝐰​\(t\)−η​∇L​\(𝐰​\(t\)\)\\mathbf\{w\}\(t\+1\)=\\mathbf\{w\}\(t\)\-\\eta\\nabla L\(\\mathbf\{w\}\(t\)\), the radial and tangential components evolve as follows:

r​\(t\+1\)2\\displaystyle r\(t\+1\)^\{2\}=r​\(t\)2−2​η​r​\(t\)​⟨𝐮​\(t\),∇L​\(𝐰​\(t\)\)⟩\+η2​‖∇L​\(𝐰​\(t\)\)‖2,\\displaystyle=r\(t\)^\{2\}\-2\\eta r\(t\)\\left\\langle\{\\mathbf\{u\}\(t\)\},\{\\nabla L\(\\mathbf\{w\}\(t\)\)\}\\right\\rangle\+\\eta^\{2\}\\\|\\nabla L\(\\mathbf\{w\}\(t\)\)\\\|^\{2\},\(13\)𝐮​\(t\+1\)\\displaystyle\\mathbf\{u\}\(t\+1\)=r​\(t\)​𝐮​\(t\)−η​∇L​\(𝐰​\(t\)\)r​\(t\+1\)\.\\displaystyle=\\frac\{r\(t\)\\mathbf\{u\}\(t\)\-\\eta\\nabla L\(\\mathbf\{w\}\(t\)\)\}\{r\(t\+1\)\}\.\(14\)

###### Proof\.\.

From𝐰​\(t\+1\)=𝐰​\(t\)−η​∇L​\(𝐰​\(t\)\)\\mathbf\{w\}\(t\+1\)=\\mathbf\{w\}\(t\)\-\\eta\\nabla L\(\\mathbf\{w\}\(t\)\), we have

r​\(t\+1\)2=‖𝐰​\(t\)−η​∇L​\(𝐰​\(t\)\)‖2=‖𝐰​\(t\)‖2−2​η​⟨𝐰​\(t\),∇L​\(𝐰​\(t\)\)⟩\+η2​‖∇L​\(𝐰​\(t\)\)‖2,r\(t\+1\)^\{2\}=\\\|\\mathbf\{w\}\(t\)\-\\eta\\nabla L\(\\mathbf\{w\}\(t\)\)\\\|^\{2\}=\\\|\\mathbf\{w\}\(t\)\\\|^\{2\}\-2\\eta\\left\\langle\{\\mathbf\{w\}\(t\)\},\{\\nabla L\(\\mathbf\{w\}\(t\)\)\}\\right\\rangle\+\\eta^\{2\}\\\|\\nabla L\(\\mathbf\{w\}\(t\)\)\\\|^\{2\},which yields \([13](https://arxiv.org/html/2608.04382#A5.E13)\) by substituting𝐰​\(t\)=r​\(t\)​𝐮​\(t\)\\mathbf\{w\}\(t\)=r\(t\)\\mathbf\{u\}\(t\)\. The tangential update \([14](https://arxiv.org/html/2608.04382#A5.E14)\) immediately follows from𝐮​\(t\+1\)=𝐰​\(t\+1\)/r​\(t\+1\)\\mathbf\{u\}\(t\+1\)=\\mathbf\{w\}\(t\+1\)/r\(t\+1\)\. ∎

Instead of the time derivativesr˙​\(t\)\\dot\{r\}\(t\)and𝐮˙​\(t\)\\dot\{\\mathbf\{u\}\}\(t\)in the continuous case, we focus on the following discrete\-time increments:

\{Δ​r​\(t\)≔r​\(t\+1\)−r​\(t\),Δ​𝐮​\(t\)≔𝐮​\(t\+1\)−𝐮​\(t\)\.\\begin\{cases\}\\Delta r\(t\)&\\coloneqq r\(t\+1\)\-r\(t\),\\\\ \\Delta\\mathbf\{u\}\(t\)&\\coloneqq\\mathbf\{u\}\(t\+1\)\-\\mathbf\{u\}\(t\)\.\\end\{cases\}The following lemma provides useful bounds on these increments, which are parallel results to the continuous\-time dynamics shown in Lemma[3](https://arxiv.org/html/2608.04382#Thmtheorem3), by controlling the discretization error\.

###### Lemma 18\(Radial/tangential increments\)\.

Assume the stepsize satisfiesη​‖∇L​\(𝐰​\(t\)\)‖≤r​\(t\)5\\eta\\\|\\nabla L\(\\mathbf\{w\}\(t\)\)\\\|\\leq\\frac\{r\(t\)\}\{5\}\. Then, the following hold:

Δ​r​\(t\)\\displaystyle\\Delta r\(t\)=−η​⟨𝐮​\(t\),∇L​\(𝐰​\(t\)\)⟩−O​\(η2​‖∇L​\(𝐰​\(t\)\)‖2r​\(t\)\),\\displaystyle=\-\\eta\\left\\langle\{\\mathbf\{u\}\(t\)\},\{\\nabla L\(\\mathbf\{w\}\(t\)\)\}\\right\\rangle\-O\\left\(\\frac\{\\eta^\{2\}\\\|\\nabla L\(\\mathbf\{w\}\(t\)\)\\\|^\{2\}\}\{r\(t\)\}\\right\),\(15\)Δ​𝐮​\(t\)\\displaystyle\\Delta\\mathbf\{u\}\(t\)=−ηr​\(t\)​P𝐮​\(t\)⟂​∇L​\(𝐰​\(t\)\)\+O​\(η2​‖∇L​\(𝐰​\(t\)\)‖2r​\(t\)2\)\.\\displaystyle=\-\\frac\{\\eta\}\{r\(t\)\}P\_\{\\mathbf\{u\}\(t\)\}^\{\\perp\}\\nabla L\(\\mathbf\{w\}\(t\)\)\+O\\left\(\\frac\{\\eta^\{2\}\\\|\\nabla L\(\\mathbf\{w\}\(t\)\)\\\|^\{2\}\}\{r\(t\)^\{2\}\}\\right\)\.\(16\)

###### Proof\.\.

To evaluate the radial increment, we use the Taylor expansion1\+x=1\+x/2−x2/8\+O​\(x3\)\\sqrt\{1\+x\}=1\+x/2\-x^\{2\}/8\+O\(x^\{3\}\)for\|x\|≤1/2\|x\|\\leq 1/2\. From \([13](https://arxiv.org/html/2608.04382#A5.E13)\), we have

r​\(t\+1\)=r​\(t\)​1−2​η​⟨𝐮​\(t\),∇L​\(𝐰​\(t\)\)⟩r​\(t\)\+η2​‖∇L​\(𝐰​\(t\)\)‖2r​\(t\)2\.r\(t\+1\)=r\(t\)\\sqrt\{1\-\\frac\{2\\eta\\left\\langle\{\\mathbf\{u\}\(t\)\},\{\\nabla L\(\\mathbf\{w\}\(t\)\)\}\\right\\rangle\}\{r\(t\)\}\+\\frac\{\\eta^\{2\}\\\|\\nabla L\(\\mathbf\{w\}\(t\)\)\\\|^\{2\}\}\{r\(t\)^\{2\}\}\}\.Under the assumptionη​‖∇L​\(𝐰​\(t\)\)‖≤r​\(t\)/5\\eta\\\|\\nabla L\(\\mathbf\{w\}\(t\)\)\\\|\\leq r\(t\)/5, we have

\|−2​η​⟨𝐮​\(t\),∇L​\(𝐰​\(t\)\)⟩r​\(t\)\+η2​‖∇L​\(𝐰​\(t\)\)‖2r​\(t\)2\|≤2​η​‖∇L​\(𝐰​\(t\)\)‖r​\(t\)\+η2​‖∇L​\(𝐰​\(t\)\)‖2r​\(t\)2≤12,\\left\|\\frac\{\-2\\eta\\left\\langle\{\\mathbf\{u\}\(t\)\},\{\\nabla L\(\\mathbf\{w\}\(t\)\)\}\\right\\rangle\}\{r\(t\)\}\+\\frac\{\\eta^\{2\}\\\|\\nabla L\(\\mathbf\{w\}\(t\)\)\\\|^\{2\}\}\{r\(t\)^\{2\}\}\\right\|\\leq\\frac\{2\\eta\\\|\\nabla L\(\\mathbf\{w\}\(t\)\)\\\|\}\{r\(t\)\}\+\\frac\{\\eta^\{2\}\\\|\\nabla L\(\\mathbf\{w\}\(t\)\)\\\|^\{2\}\}\{r\(t\)^\{2\}\}\\leq\\frac\{1\}\{2\},which allows us to apply the Taylor expansion to obtain

r​\(t\+1\)\\displaystyle r\(t\+1\)=r​\(t\)​\[1−η​⟨𝐮​\(t\),∇L​\(𝐰​\(t\)\)⟩r​\(t\)\+η2​‖∇L​\(𝐰​\(t\)\)‖22​r​\(t\)2−O​\(η2​‖∇L​\(𝐰​\(t\)\)‖2r​\(t\)2\)\]\\displaystyle=r\(t\)\\left\[1\-\\frac\{\\eta\\left\\langle\{\\mathbf\{u\}\(t\)\},\{\\nabla L\(\\mathbf\{w\}\(t\)\)\}\\right\\rangle\}\{r\(t\)\}\+\\frac\{\\eta^\{2\}\\\|\\nabla L\(\\mathbf\{w\}\(t\)\)\\\|^\{2\}\}\{2r\(t\)^\{2\}\}\-O\\left\(\\frac\{\\eta^\{2\}\\\|\\nabla L\(\\mathbf\{w\}\(t\)\)\\\|^\{2\}\}\{r\(t\)^\{2\}\}\\right\)\\right\]=r​\(t\)−η​⟨𝐮​\(t\),∇L​\(𝐰​\(t\)\)⟩−O​\(η2​‖∇L​\(𝐰​\(t\)\)‖2r​\(t\)\),\\displaystyle=r\(t\)\-\\eta\\left\\langle\{\\mathbf\{u\}\(t\)\},\{\\nabla L\(\\mathbf\{w\}\(t\)\)\}\\right\\rangle\-O\\left\(\\frac\{\\eta^\{2\}\\\|\\nabla L\(\\mathbf\{w\}\(t\)\)\\\|^\{2\}\}\{r\(t\)\}\\right\),which yields \([15](https://arxiv.org/html/2608.04382#A5.E15)\)\.

To evaluate the tangential increment, we need to evaluate\(r​\(t\+1\)\)−1=\[r​\(t\)\+Δ​r​\(t\)\]−1\(r\(t\+1\)\)^\{\-1\}=\[r\(t\)\+\\Delta r\(t\)\]^\{\-1\}by the Taylor expansion11\+x=1−x\+O​\(x2\)\\frac\{1\}\{1\+x\}=1\-x\+O\(x^\{2\}\)as follows:

1r​\(t\)\+Δ​r​\(t\)=1r​\(t\)⋅11\+Δ​r​\(t\)r​\(t\)=r​\(t\)−1​\[1−Δ​r​\(t\)r​\(t\)\+O​\(Δ​r​\(t\)2r​\(t\)2\)\]\.\\frac\{1\}\{r\(t\)\+\\Delta r\(t\)\}=\\frac\{1\}\{r\(t\)\}\\cdot\\frac\{1\}\{1\+\\frac\{\\Delta r\(t\)\}\{r\(t\)\}\}=r\(t\)^\{\-1\}\\left\[1\-\\frac\{\\Delta r\(t\)\}\{r\(t\)\}\+O\\left\(\\frac\{\\Delta r\(t\)^\{2\}\}\{r\(t\)^\{2\}\}\\right\)\\right\]\.This expansion leads the tangential dynamics \([14](https://arxiv.org/html/2608.04382#A5.E14)\) to

𝐮​\(t\+1\)\\displaystyle\\mathbf\{u\}\(t\+1\)=r​\(t\)r​\(t\+1\)​𝐮​\(t\)−ηr​\(t\+1\)​∇L​\(𝐰​\(t\)\)\\displaystyle=\\frac\{r\(t\)\}\{r\(t\+1\)\}\\mathbf\{u\}\(t\)\-\\frac\{\\eta\}\{r\(t\+1\)\}\\nabla L\(\\mathbf\{w\}\(t\)\)=\[1−Δ​r​\(t\)r​\(t\)\+O​\(Δ​r​\(t\)2r​\(t\)2\)\]​𝐮​\(t\)−ηr​\(t\)​\[1−Δ​r​\(t\)r​\(t\)\+O​\(Δ​r​\(t\)2r​\(t\)2\)\]​∇L​\(𝐰​\(t\)\)\\displaystyle=\\left\[1\-\\frac\{\\Delta r\(t\)\}\{r\(t\)\}\+O\\left\(\\frac\{\\Delta r\(t\)^\{2\}\}\{r\(t\)^\{2\}\}\\right\)\\right\]\\mathbf\{u\}\(t\)\-\\frac\{\\eta\}\{r\(t\)\}\\left\[1\-\\frac\{\\Delta r\(t\)\}\{r\(t\)\}\+O\\left\(\\frac\{\\Delta r\(t\)^\{2\}\}\{r\(t\)^\{2\}\}\\right\)\\right\]\\nabla L\(\\mathbf\{w\}\(t\)\)=𝐮​\(t\)−ηr​\(t\)​∇L​\(𝐰​\(t\)\)−Δ​r​\(t\)r​\(t\)​𝐮​\(t\)\+O​\(η2​‖∇L​\(𝐰​\(t\)\)‖2r​\(t\)2\)\.\\displaystyle=\\mathbf\{u\}\(t\)\-\\frac\{\\eta\}\{r\(t\)\}\\nabla L\(\\mathbf\{w\}\(t\)\)\-\\frac\{\\Delta r\(t\)\}\{r\(t\)\}\\mathbf\{u\}\(t\)\+O\\left\(\\frac\{\\eta^\{2\}\\\|\\nabla L\(\\mathbf\{w\}\(t\)\)\\\|^\{2\}\}\{r\(t\)^\{2\}\}\\right\)\.Substituting the radial increment \([15](https://arxiv.org/html/2608.04382#A5.E15)\) into the third term, we obtain

Δ​r​\(t\)r​\(t\)​𝐮​\(t\)=−ηr​\(t\)​⟨𝐮​\(t\),∇L​\(𝐰​\(t\)\)⟩​𝐮​\(t\)−O​\(η2​‖∇L​\(𝐰​\(t\)\)‖2r​\(t\)2\)\.\\frac\{\\Delta r\(t\)\}\{r\(t\)\}\\mathbf\{u\}\(t\)=\-\\frac\{\\eta\}\{r\(t\)\}\\left\\langle\{\\mathbf\{u\}\(t\)\},\{\\nabla L\(\\mathbf\{w\}\(t\)\)\}\\right\\rangle\\mathbf\{u\}\(t\)\-O\\left\(\\frac\{\\eta^\{2\}\\\|\\nabla L\(\\mathbf\{w\}\(t\)\)\\\|^\{2\}\}\{r\(t\)^\{2\}\}\\right\)\.Plugging this back, we have

𝐮​\(t\+1\)\\displaystyle\\mathbf\{u\}\(t\+1\)=𝐮​\(t\)−ηr​\(t\)​\[∇L​\(𝐰​\(t\)\)−⟨𝐮​\(t\),∇L​\(𝐰​\(t\)\)⟩​𝐮​\(t\)\]\+O​\(η2​‖∇L​\(𝐰​\(t\)\)‖2r​\(t\)2\)\\displaystyle=\\mathbf\{u\}\(t\)\-\\frac\{\\eta\}\{r\(t\)\}\\left\[\\nabla L\(\\mathbf\{w\}\(t\)\)\-\\left\\langle\{\\mathbf\{u\}\(t\)\},\{\\nabla L\(\\mathbf\{w\}\(t\)\)\}\\right\\rangle\\mathbf\{u\}\(t\)\\right\]\+O\\left\(\\frac\{\\eta^\{2\}\\\|\\nabla L\(\\mathbf\{w\}\(t\)\)\\\|^\{2\}\}\{r\(t\)^\{2\}\}\\right\)=𝐮​\(t\)−ηr​\(t\)​P𝐮​\(t\)⟂​∇L​\(𝐰​\(t\)\)\+O​\(η2​‖∇L​\(𝐰​\(t\)\)‖2r​\(t\)2\),\\displaystyle=\\mathbf\{u\}\(t\)\-\\frac\{\\eta\}\{r\(t\)\}P\_\{\\mathbf\{u\}\(t\)\}^\{\\perp\}\\nabla L\(\\mathbf\{w\}\(t\)\)\+O\\left\(\\frac\{\\eta^\{2\}\\\|\\nabla L\(\\mathbf\{w\}\(t\)\)\\\|^\{2\}\}\{r\(t\)^\{2\}\}\\right\),which yields \([16](https://arxiv.org/html/2608.04382#A5.E16)\)\. ∎

### E\.1Discrete\-time risk and radial bounds

We now establish discrete\-time analogs of the risk and radial bounds in Lemma[4](https://arxiv.org/html/2608.04382#Thmtheorem4)\. These bounds control the decay of the risk and growth of the radial norm throughout the discrete gradient descent dynamics\.

###### Lemma 19\(Discrete\-time loss and radial bounds\)\.

Letρ≔r​\(0\)\\rho\\coloneqq r\(0\)and assumeL​\(0\)<eρ​γ2L\(0\)<e^\{\\rho\\gamma^\{2\}\}\. For discrete\-time gradient descent witht≥0t\\geq 0, we have

1L​\(0\)−1\+η​t≲L​\(𝐰​\(t\)\)≲1L​\(0\)−1\+η​γ2​t,\\displaystyle\\frac\{1\}\{L\(0\)^\{\-1\}\+\\eta t\}\\lesssim L\(\\mathbf\{w\}\(t\)\)\\lesssim\\frac\{1\}\{L\(0\)^\{\-1\}\+\\eta\\gamma^\{2\}t\},\(17\)ρ−1γ2​log⁡\(1\+L​\(0\)​η​γ2​t\)≲r​\(t\)≲ρ\+1γ2​log⁡\(1\+L​\(0\)​η​γ2​t\),\\displaystyle\\rho\-\\frac\{1\}\{\\gamma^\{2\}\}\\log\(1\+L\(0\)\\eta\\gamma^\{2\}t\)\\lesssim r\(t\)\\lesssim\\rho\+\\frac\{1\}\{\\gamma^\{2\}\}\\log\(1\+L\(0\)\\eta\\gamma^\{2\}t\),\(18\)r​\(t\)≳log⁡\(L​\(0\)−1\+η​γ2​t\),\\displaystyle r\(t\)\\gtrsim\\log\(L\(0\)^\{\-1\}\+\\eta\\gamma^\{2\}t\),\(19\)whereO​\(1\)O\(1\)\-multiplicative negligible error is hidden in “≲\\lesssim”\.

###### Proof\.\.

Upper bound onL​\(𝐰​\(t\)\)L\(\\mathbf\{w\}\(t\)\)\.By expanding the riskL​\(𝐰​\(t\)\)L\(\\mathbf\{w\}\(t\)\)along the gradient descent dynamics, we have

L​\(𝐰​\(t\+1\)\)\\displaystyle L\(\\mathbf\{w\}\(t\+1\)\)=1n​∑i=1ne−⟨𝐰​\(t\)−η​∇L​\(𝐰​\(t\)\),𝐳i⟩=1n​∑i=1ne−⟨𝐰​\(t\),𝐳i⟩​eη​⟨∇L​\(𝐰​\(t\)\),𝐳i⟩\\displaystyle=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}e^\{\-\\left\\langle\{\\mathbf\{w\}\(t\)\-\\eta\\nabla L\(\\mathbf\{w\}\(t\)\)\},\{\\mathbf\{z\}\_\{i\}\}\\right\\rangle\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}e^\{\-\\left\\langle\{\\mathbf\{w\}\(t\)\},\{\\mathbf\{z\}\_\{i\}\}\\right\\rangle\}e^\{\\eta\\left\\langle\{\\nabla L\(\\mathbf\{w\}\(t\)\)\},\{\\mathbf\{z\}\_\{i\}\}\\right\\rangle\}=1n​∑i=1ne−⟨𝐰​\(t\),𝐳i⟩​\[1\+η​⟨∇L​\(𝐰​\(t\)\),𝐳i⟩\+O​\(η2​‖∇L​\(𝐰​\(t\)\)‖2\)\]\\displaystyle=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}e^\{\-\\left\\langle\{\\mathbf\{w\}\(t\)\},\{\\mathbf\{z\}\_\{i\}\}\\right\\rangle\}\\left\[1\+\\eta\\left\\langle\{\\nabla L\(\\mathbf\{w\}\(t\)\)\},\{\\mathbf\{z\}\_\{i\}\}\\right\\rangle\+O\(\\eta^\{2\}\\\|\\nabla L\(\\mathbf\{w\}\(t\)\)\\\|^\{2\}\)\\right\]=L​\(𝐰​\(t\)\)−η​‖∇L​\(𝐰​\(t\)\)‖2\+O​\(η2​‖∇L​\(𝐰​\(t\)\)‖2\),\\displaystyle=L\(\\mathbf\{w\}\(t\)\)\-\\eta\\\|\\nabla L\(\\mathbf\{w\}\(t\)\)\\\|^\{2\}\+O\(\\eta^\{2\}\\\|\\nabla L\(\\mathbf\{w\}\(t\)\)\\\|^\{2\}\),where the Taylor expansionex=1\+x\+O​\(x2\)e^\{x\}=1\+x\+O\(x^\{2\}\)is used\. By the self\-bounding property \([3](https://arxiv.org/html/2608.04382#S2.E3)\), we haveγ​L​\(𝐰​\(t\)\)≤‖∇L​\(𝐰​\(t\)\)‖\\gamma L\(\\mathbf\{w\}\(t\)\)\\leq\\\|\\nabla L\(\\mathbf\{w\}\(t\)\)\\\|, which yields

L​\(𝐰​\(t\+1\)\)≤L​\(𝐰​\(t\)\)−η​γ2​L​\(𝐰​\(t\)\)2\+O​\(η2​L​\(𝐰​\(t\)\)2\)\.L\(\\mathbf\{w\}\(t\+1\)\)\\leq L\(\\mathbf\{w\}\(t\)\)\-\\eta\\gamma^\{2\}L\(\\mathbf\{w\}\(t\)\)^\{2\}\+O\(\\eta^\{2\}L\(\\mathbf\{w\}\(t\)\)^\{2\}\)\.\(20\)Now, we invert both sides and take the telescoping sum:

1L​\(𝐰​\(t\+1\)\)\\displaystyle\\frac\{1\}\{L\(\\mathbf\{w\}\(t\+1\)\)\}≥1L​\(𝐰​\(t\)\)​\[1−η​γ2​L​\(𝐰​\(t\)\)\+O​\(η2​L​\(𝐰​\(t\)\)\)\]\\displaystyle\\geq\\frac\{1\}\{L\(\\mathbf\{w\}\(t\)\)\[1\-\\eta\\gamma^\{2\}L\(\\mathbf\{w\}\(t\)\)\+O\(\\eta^\{2\}L\(\\mathbf\{w\}\(t\)\)\)\]\}≥1L​\(𝐰​\(t\)\)​\[1\+η​γ2​L​\(𝐰​\(t\)\)−O​\(η2​L​\(𝐰​\(t\)\)\)\]\\displaystyle\\geq\\frac\{1\}\{L\(\\mathbf\{w\}\(t\)\)\}\\left\[1\+\\eta\\gamma^\{2\}L\(\\mathbf\{w\}\(t\)\)\-O\(\\eta^\{2\}L\(\\mathbf\{w\}\(t\)\)\)\\right\]=1L​\(𝐰​\(t\)\)\+η​γ2⋅\(1−O​\(η​γ−2\)\)⏟O​\(1\)\-discretization error\\displaystyle=\\frac\{1\}\{L\(\\mathbf\{w\}\(t\)\)\}\+\\eta\\gamma^\{2\}\\cdot\\underbrace\{\(1\-O\(\\eta\\gamma^\{\-2\}\)\)\}\_\{\\text\{$O\(1\)$\-discretization error\}\}≳1L​\(𝐰​\(t\)\)\+η​γ2,\\displaystyle\\gtrsim\\frac\{1\}\{L\(\\mathbf\{w\}\(t\)\)\}\+\\eta\\gamma^\{2\},where we use11−x≥1\+x\\frac\{1\}\{1\-x\}\\geq 1\+x\(when0<x<1/20<x<1/2\) at the second inequality and the stepsize choiceη≤c0​γ2\\eta\\leq c\_\{0\}\\gamma^\{2\}at the last inequality\. Telescoping yields

1L​\(𝐰​\(t\)\)−1L​\(0\)≳η​γ2​t↝L​\(𝐰​\(t\)\)≲1L​\(0\)−1\+η​γ2​t,\\frac\{1\}\{L\(\\mathbf\{w\}\(t\)\)\}\-\\frac\{1\}\{L\(0\)\}\\gtrsim\\eta\\gamma^\{2\}t\\quad\\leadsto\\quad L\(\\mathbf\{w\}\(t\)\)\\lesssim\\frac\{1\}\{L\(0\)^\{\-1\}\+\\eta\\gamma^\{2\}t\},where we used11−x=1\+x−O​\(x2\)\\frac\{1\}\{1\-x\}=1\+x\-O\(x^\{2\}\)in the second argument\.

#### Lower bound onL​\(𝐰​\(t\)\)L\(\\mathbf\{w\}\(t\)\)\.

Similarly, the risk expansion withex≥1\+xe^\{x\}\\geq 1\+xleads to

L​\(𝐰​\(t\+1\)\)≥1n​∑i=1ne−⟨𝐰​\(t\),𝐳i⟩​\[1\+η​⟨∇L​\(𝐰​\(t\)\),𝐳i⟩\]≥L​\(𝐰​\(t\)\)−η​L​\(𝐰​\(t\)\)2,L\(\\mathbf\{w\}\(t\+1\)\)\\geq\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}e^\{\-\\left\\langle\{\\mathbf\{w\}\(t\)\},\{\\mathbf\{z\}\_\{i\}\}\\right\\rangle\}\\left\[1\+\\eta\\left\\langle\{\\nabla L\(\\mathbf\{w\}\(t\)\)\},\{\\mathbf\{z\}\_\{i\}\}\\right\\rangle\\right\]\\geq L\(\\mathbf\{w\}\(t\)\)\-\\eta L\(\\mathbf\{w\}\(t\)\)^\{2\},where the self\-bounding property‖∇L​\(𝐰\)‖≤L​\(𝐰​\(t\)\)\\\|\\nabla L\(\\mathbf\{w\}\)\\\|\\leq L\(\\mathbf\{w\}\(t\)\)\([3](https://arxiv.org/html/2608.04382#S2.E3)\) is used\. Taking the reciprocals with11−x=1\+x\+O​\(x2\)\\frac\{1\}\{1\-x\}=1\+x\+O\(x^\{2\}\)\(when0<x<1/20<x<1/2\), we have

1L​\(𝐰​\(t\+1\)\)\\displaystyle\\frac\{1\}\{L\(\\mathbf\{w\}\(t\+1\)\)\}≤1L​\(𝐰​\(t\)\)​\[1−η​L​\(𝐰​\(t\)\)\]\\displaystyle\\leq\\frac\{1\}\{L\(\\mathbf\{w\}\(t\)\)\[1\-\\eta L\(\\mathbf\{w\}\(t\)\)\]\}=1L​\(𝐰​\(t\)\)​\[1\+η​L​\(𝐰​\(t\)\)\+O​\(η2​L​\(𝐰​\(t\)\)2\)\]\\displaystyle=\\frac\{1\}\{L\(\\mathbf\{w\}\(t\)\)\}\\left\[1\+\\eta L\(\\mathbf\{w\}\(t\)\)\+O\(\\eta^\{2\}L\(\\mathbf\{w\}\(t\)\)^\{2\}\)\\right\]=1L​\(𝐰​\(t\)\)\+η\+O​\(η2​L​\(𝐰​\(t\)\)\)\.\\displaystyle=\\frac\{1\}\{L\(\\mathbf\{w\}\(t\)\)\}\+\\eta\+O\(\\eta^\{2\}L\(\\mathbf\{w\}\(t\)\)\)\.Taking the telescoping sum from0tot−1t\-1yields

1L​\(𝐰​\(t\)\)\\displaystyle\\frac\{1\}\{L\(\\mathbf\{w\}\(t\)\)\}≤1L​\(0\)\+η​t\+O​\(η2\)​∑s=0t−1L​\(𝐰​\(s\)\)\\displaystyle\\leq\\frac\{1\}\{L\(0\)\}\+\\eta t\+O\(\\eta^\{2\}\)\\sum\_\{s=0\}^\{t\-1\}L\(\\mathbf\{w\}\(s\)\)≲1L​\(0\)\+η​t\+O​\(η2\)⋅∫0td​sL​\(0\)−1\+η​γ2​s\\displaystyle\\lesssim\\frac\{1\}\{L\(0\)\}\+\\eta t\+O\(\\eta^\{2\}\)\\cdot\\int\_\{0\}^\{t\}\\frac\{\\mathrm\{d\}s\}\{L\(0\)^\{\-1\}\+\\eta\\gamma^\{2\}s\}=1L​\(0\)\+η​t\+O​\(η\)⋅1γ2​log⁡\(1\+L​\(0\)​η​γ2​t\)\\displaystyle=\\frac\{1\}\{L\(0\)\}\+\\eta t\+O\(\\eta\)\\cdot\\frac\{1\}\{\\gamma^\{2\}\}\\log\(1\+L\(0\)\\eta\\gamma^\{2\}t\)≲1L​\(0\)\+η​t,\\displaystyle\\lesssim\\frac\{1\}\{L\(0\)\}\+\\eta t,where we used the upper bound ofL​\(𝐰​\(s\)\)L\(\\mathbf\{w\}\(s\)\)already derived\. Lastly, taking the reciprocal gives

L​\(𝐰​\(t\)\)≳1L​\(0\)−1\+η​t=1L​\(0\)−1\+η​t\.L\(\\mathbf\{w\}\(t\)\)\\gtrsim\\frac\{1\}\{L\(0\)^\{\-1\}\+\\eta t\}=\\frac\{1\}\{L\(0\)^\{\-1\}\+\\eta t\}\.

#### Upper bound onr​\(t\)r\(t\)\.

By plugging⟨𝐮,∇L​\(𝐰\)⟩≥−‖∇L​\(𝐰\)‖≥−L​\(𝐰\)\\left\\langle\{\\mathbf\{u\}\},\{\\nabla L\(\\mathbf\{w\}\)\}\\right\\rangle\\geq\-\\\|\\nabla L\(\\mathbf\{w\}\)\\\|\\geq\-L\(\\mathbf\{w\}\)\(the second inequality follows from the self\-bounding property \([3](https://arxiv.org/html/2608.04382#S2.E3)\)\) into Lemma[18](https://arxiv.org/html/2608.04382#Thmtheorem18), we have

Δ​r​\(s\)≤η​L​\(𝐰​\(s\)\)−O​\(η2​‖∇L​\(𝐰​\(s\)\)‖2r​\(s\)\)≲η​L​\(𝐰​\(s\)\),\\Delta r\(s\)\\leq\\eta L\(\\mathbf\{w\}\(s\)\)\-O\\left\(\\frac\{\\eta^\{2\}\\\|\\nabla L\(\\mathbf\{w\}\(s\)\)\\\|^\{2\}\}\{r\(s\)\}\\right\)\\lesssim\\eta L\(\\mathbf\{w\}\(s\)\),where the second inequality uses the stepsize choice and self\-bounding‖∇L​\(𝐰\)‖≤L​\(𝐰\)\\\|\\nabla L\(\\mathbf\{w\}\)\\\|\\leq L\(\\mathbf\{w\}\)\. Taking the telescoping sum from0tot−1t\-1leads to

r​\(t\)\\displaystyle r\(t\)≲ρ\+η​∑s=0t−1L​\(𝐰​\(s\)\)\\displaystyle\\lesssim\\rho\+\\eta\\sum\_\{s=0\}^\{t\-1\}L\(\\mathbf\{w\}\(s\)\)≤ρ\+η​∑s=0t−11L​\(0\)−1\+η​γ2​s\\displaystyle\\leq\\rho\+\\eta\\sum\_\{s=0\}^\{t\-1\}\\frac\{1\}\{L\(0\)^\{\-1\}\+\\eta\\gamma^\{2\}s\}≤ρ\+η​∫0td​sL​\(0\)−1\+η​γ2​s\\displaystyle\\leq\\rho\+\\eta\\int\_\{0\}^\{t\}\\frac\{\\mathrm\{d\}s\}\{L\(0\)^\{\-1\}\+\\eta\\gamma^\{2\}s\}=ρ\+1γ2​log⁡\(1\+L​\(0\)​η​γ2​t\),\\displaystyle=\\rho\+\\frac\{1\}\{\\gamma^\{2\}\}\\log\(1\+L\(0\)\\eta\\gamma^\{2\}t\),where we used the upper bound ofL​\(𝐰​\(s\)\)L\(\\mathbf\{w\}\(s\)\)in \([17](https://arxiv.org/html/2608.04382#A5.E17)\)\.

#### Lower bounds onr​\(t\)r\(t\)\.

Similarly, by plugging⟨𝐮,∇L​\(𝐰\)⟩≤‖∇L​\(𝐰\)‖≤L​\(𝐰\)\\left\\langle\{\\mathbf\{u\}\},\{\\nabla L\(\\mathbf\{w\}\)\}\\right\\rangle\\leq\\\|\\nabla L\(\\mathbf\{w\}\)\\\|\\leq L\(\\mathbf\{w\}\)and the stepsize choice into Lemma[18](https://arxiv.org/html/2608.04382#Thmtheorem18), we have

Δ​r​\(s\)≥−η​L​\(𝐰​\(s\)\)−O​\(η2​‖∇L​\(𝐰​\(s\)\)‖2r​\(s\)\)≳−η​L​\(𝐰​\(s\)\)\.\\Delta r\(s\)\\geq\-\\eta L\(\\mathbf\{w\}\(s\)\)\-O\\left\(\\frac\{\\eta^\{2\}\\\|\\nabla L\(\\mathbf\{w\}\(s\)\)\\\|^\{2\}\}\{r\(s\)\}\\right\)\\gtrsim\-\\eta L\(\\mathbf\{w\}\(s\)\)\.Taking the telescoping sum from0tot−1t\-1leads to

r​\(t\)≳ρ−η​∑s=0t−1L​\(𝐰​\(s\)\)≥ρ−1γ2​log⁡\(1\+L​\(0\)​η​γ2​t\)\.r\(t\)\\gtrsim\\rho\-\\eta\\sum\_\{s=0\}^\{t\-1\}L\(\\mathbf\{w\}\(s\)\)\\geq\\rho\-\\frac\{1\}\{\\gamma^\{2\}\}\\log\(1\+L\(0\)\\eta\\gamma^\{2\}t\)\.The last bound \([19](https://arxiv.org/html/2608.04382#A5.E19)\) can be similarly shown to Lemma[4](https://arxiv.org/html/2608.04382#Thmtheorem4)and hence we omit the proof\. ∎

### E\.2Upper bound

To establish the early\-stage alignment, we show the weak alignment and escape stages, respectively, as in the continuous case\. First, we show the discrete analog of Lemma[5](https://arxiv.org/html/2608.04382#Thmtheorem5)\(weak alignment\)\.

###### Lemma 21\.

For an absolute constantκ\>0\\kappa\>0, assume the following uniform lower bound holds duringt=0,1,…,T1t=0,1,\\dots,T\_\{1\}:⟨Z​𝛂​\(t\),P𝐮​\(t\)⟂​𝐮∗⟩≥κ\\langle\{Z\\bm\{\\alpha\}\(t\)\},\{P\_\{\\mathbf\{u\}\(t\)\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\}\\rangle\\geq\\kappa\. Then, we have

V​\(t\)≤V​\(0\)−κ​γ22​log⁡\(1\+log⁡\(L​\(0\)​η​γ2​t\)ρ​γ2\+log⁡2\)V\(t\)\\leq V\(0\)\-\\frac\{\\kappa\\gamma^\{2\}\}\{2\}\\log\\left\(1\+\\frac\{\\log\(L\(0\)\\eta\\gamma^\{2\}t\)\}\{\\rho\\gamma^\{2\}\+\\log 2\}\\right\)fort=0,1,…,T1t=0,1,\\dots,T\_\{1\}\. Forδ\>0\\delta\>0, if the uniform bound holds duringt∈\[0,T1\]t\\in\[0,T\_\{1\}\]withT1=O​\(η−1​exp⁡\(exp⁡\(−δ/κ​γ2\)\)\)T\_\{1\}=O\(\\eta^\{\-1\}\\exp\(\\exp\(\-\\delta/\\kappa\\gamma^\{2\}\)\)\), we haveV​\(t\)≤δV\(t\)\\leq\\deltaafterT1T\_\{1\}iterations\.

###### Proof\.\.

By Lemma[18](https://arxiv.org/html/2608.04382#Thmtheorem18), the discrete increment of the tangential alignmentV​\(t\)V\(t\)can be written as

Δ​V​\(t\)\\displaystyle\\Delta V\(t\)≔V​\(t\+1\)−V​\(t\)=−⟨Δ​𝐮​\(t\),𝐮∗⟩\\displaystyle\\coloneqq V\(t\+1\)\-V\(t\)=\-\\left\\langle\{\\Delta\\mathbf\{u\}\(t\)\},\{\\mathbf\{u\}\_\{\*\}\}\\right\\rangle\(21\)=ηr​\(t\)​⟨∇L​\(𝐰​\(t\)\),P𝐮​\(t\)⟂​𝐮∗⟩−O​\(η2​‖∇L​\(𝐰​\(t\)\)‖2r​\(t\)2\)\\displaystyle=\\frac\{\\eta\}\{r\(t\)\}\\left\\langle\{\\nabla L\(\\mathbf\{w\}\(t\)\)\},\{P\_\{\\mathbf\{u\}\(t\)\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\}\\right\\rangle\-O\\left\(\\frac\{\\eta^\{2\}\\\|\\nabla L\(\\mathbf\{w\}\(t\)\)\\\|^\{2\}\}\{r\(t\)^\{2\}\}\\right\)=−η​L​\(𝐰​\(t\)\)r​\(t\)​⟨Z​𝜶​\(t\),P𝐮​\(t\)⟂​𝐮∗⟩−O​\(η2​‖∇L​\(𝐰​\(t\)\)‖2r​\(t\)2\)\\displaystyle=\-\\frac\{\\eta L\(\\mathbf\{w\}\(t\)\)\}\{r\(t\)\}\\left\\langle\{Z\\bm\{\\alpha\}\(t\)\},\{P\_\{\\mathbf\{u\}\(t\)\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\}\\right\\rangle\-O\\left\(\\frac\{\\eta^\{2\}\\\|\\nabla L\(\\mathbf\{w\}\(t\)\)\\\|^\{2\}\}\{r\(t\)^\{2\}\}\\right\)≲−κ​η​L​\(𝐰​\(t\)\)r​\(t\),\\displaystyle\\lesssim\-\\frac\{\\kappa\\eta L\(\\mathbf\{w\}\(t\)\)\}\{r\(t\)\},where the uniform bound⟨Z​𝜶,P𝐮⟂​𝐮∗⟩≥κ\\langle\{Z\\bm\{\\alpha\}\},\{P\_\{\\mathbf\{u\}\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\}\\rangle\\geq\\kappaand the stepsize choice are collectively used at the last inequality\. With Lemma[19](https://arxiv.org/html/2608.04382#Thmtheorem19), the telescoping sum is taken forL​\(𝐰​\(s\)\)/r​\(s\)L\(\\mathbf\{w\}\(s\)\)/r\(s\)as follows:

∑s=0t−1L​\(𝐰​\(s\)\)r​\(s\)\\displaystyle\\sum\_\{s=0\}^\{t\-1\}\\frac\{L\(\\mathbf\{w\}\(s\)\)\}\{r\(s\)\}≳∑s=0t−11\(L​\(0\)−1\+η​s\)​\[ρ\+γ−2​log⁡\(1\+L​\(0\)​η​γ2​s\)\]\\displaystyle\\gtrsim\\sum\_\{s=0\}^\{t\-1\}\\frac\{1\}\{\(L\(0\)^\{\-1\}\+\\eta s\)\[\\rho\+\\gamma^\{\-2\}\\log\(1\+L\(0\)\\eta\\gamma^\{2\}s\)\]\}≥γ2η​∫0td​s\(L​\(0\)−1​η−1\+s\)​\[ρ​γ2\+log⁡\(1\+L​\(0\)​η​γ2​s\)\]\\displaystyle\\geq\\frac\{\\gamma^\{2\}\}\{\\eta\}\\int\_\{0\}^\{t\}\\frac\{\\mathrm\{d\}s\}\{\(L\(0\)^\{\-1\}\\eta^\{\-1\}\+s\)\[\\rho\\gamma^\{2\}\+\\log\(1\+L\(0\)\\eta\\gamma^\{2\}s\)\]\}≥γ2η​12​\(ρ​γ2\+log⁡\(1\+γ2\)\)\+γ22​η​log⁡\(1\+log⁡\(L​\(0\)​η​γ2​t\)ρ​γ2\+log⁡2\),\\displaystyle\\geq\\frac\{\\gamma^\{2\}\}\{\\eta\}\\frac\{1\}\{2\(\\rho\\gamma^\{2\}\+\\log\(1\+\\gamma^\{2\}\)\)\}\+\\frac\{\\gamma^\{2\}\}\{2\\eta\}\\log\\left\(1\+\\frac\{\\log\(L\(0\)\\eta\\gamma^\{2\}t\)\}\{\\rho\\gamma^\{2\}\+\\log 2\}\\right\),where we use Lemma[15](https://arxiv.org/html/2608.04382#Thmtheorem15)witha≔L​\(0\)−1​η−1a\\coloneqq L\(0\)^\{\-1\}\\eta^\{\-1\},b≔ρ​γ2b\\coloneqq\\rho\\gamma^\{2\}, andc≔L​\(0\)​η​γ2c\\coloneqq L\(0\)\\eta\\gamma^\{2\}\. Therefore, we have

V​\(t\)\\displaystyle V\(t\)≲V​\(0\)−κ​γ22​\(ρ​γ2\+log⁡\(1\+γ2\)\)−κ​γ22​log⁡\(1\+log⁡\(L​\(0\)​η​γ2​t\)ρ​γ2\+log⁡2\)\\displaystyle\\lesssim V\(0\)\-\\frac\{\\kappa\\gamma^\{2\}\}\{2\(\\rho\\gamma^\{2\}\+\\log\(1\+\\gamma^\{2\}\)\)\}\-\\frac\{\\kappa\\gamma^\{2\}\}\{2\}\\log\\left\(1\+\\frac\{\\log\(L\(0\)\\eta\\gamma^\{2\}t\)\}\{\\rho\\gamma^\{2\}\+\\log 2\}\\right\)≤V​\(0\)−κ​γ22​log⁡\(1\+log⁡\(L​\(0\)​η​γ2​t\)ρ​γ2\+log⁡2\)\.\\displaystyle\\leq V\(0\)\-\\frac\{\\kappa\\gamma^\{2\}\}\{2\}\\log\\left\(1\+\\frac\{\\log\(L\(0\)\\eta\\gamma^\{2\}t\)\}\{\\rho\\gamma^\{2\}\+\\log 2\}\\right\)\.To achieveV​\(t\)≤δV\(t\)\\leq\\delta, we need\(upper bound\)≤δ\\text\{\(upper bound\)\}\\leq\\delta, which yields

t≳1L​\(0\)​η​γ2​exp⁡\{\(ρ​γ2\+log⁡2\)​\[exp⁡\(2​\(V​\(0\)−δ\)κ​γ2\)−1\]\}=O​\(exp⁡\(exp⁡\(−δ/κ​γ2\)\)η\)\.t\\gtrsim\\frac\{1\}\{L\(0\)\\eta\\gamma^\{2\}\}\\exp\\left\\\{\(\\rho\\gamma^\{2\}\+\\log 2\)\\left\[\\exp\\left\(\\frac\{2\(V\(0\)\-\\delta\)\}\{\\kappa\\gamma^\{2\}\}\\right\)\-1\\right\]\\right\\\}=O\\left\(\\frac\{\\exp\(\\exp\(\-\\delta/\\kappa\\gamma^\{2\}\)\)\}\{\\eta\}\\right\)\.∎

Next, we show the discrete analog of Lemma[7](https://arxiv.org/html/2608.04382#Thmtheorem7)\(escape from poor initialization\)\.

###### Lemma 22\.

LetW​\(t\)≔⟨𝐰​\(t\),𝐮∗⟩=r​\(t\)​\(1−V​\(t\)\)W\(t\)\\coloneqq\\left\\langle\{\\mathbf\{w\}\(t\)\},\{\\mathbf\{u\}\_\{\*\}\}\\right\\rangle=r\(t\)\(1\-V\(t\)\)be the parameter projection, andTescape=1η​L​\(0\)​\{exp⁡\(ρ​γ−1​\[V​\(0\)−1\]\)−1\}T\_\{\\text\{escape\}\}=\\frac\{1\}\{\\eta L\(0\)\}\\left\\\{\\exp\(\\rho\\gamma^\{\-1\}\[V\(0\)\-1\]\)\-1\\right\\\}\. IfV​\(0\)\>1V\(0\)\>1, then the parameter eventually escapes from the bad initialization, namely,V​\(t\)≤1V\(t\)\\leq 1holds aftert≥Tescapet\\geq T\_\{\\text\{escape\}\}iterations\.

###### Proof\.\.

With⟨𝐮∗,Z​𝜶⟩≥γ\\left\\langle\{\\mathbf\{u\}\_\{\*\}\},\{Z\\bm\{\\alpha\}\}\\right\\rangle\\geq\\gammaand Lemma[19](https://arxiv.org/html/2608.04382#Thmtheorem19), we derive the discrete increment ofW​\(t\)W\(t\)as follows\. From the gradient descent update𝐰​\(t\+1\)=𝐰​\(t\)−η​∇L​\(𝐰​\(t\)\)\\mathbf\{w\}\(t\+1\)=\\mathbf\{w\}\(t\)\-\\eta\\nabla L\(\\mathbf\{w\}\(t\)\), we have

W​\(t\+1\)−W​\(t\)=−η​⟨∇L​\(𝐰​\(t\)\),𝐮∗⟩=η​L​\(𝐰​\(t\)\)​⟨𝐮∗,Z​𝜶​\(t\)⟩≳η​γL​\(0\)−1\+η​t\.W\(t\+1\)\-W\(t\)=\-\\eta\\left\\langle\{\\nabla L\(\\mathbf\{w\}\(t\)\)\},\{\\mathbf\{u\}\_\{\*\}\}\\right\\rangle=\\eta L\(\\mathbf\{w\}\(t\)\)\\left\\langle\{\\mathbf\{u\}\_\{\*\}\},\{Z\\bm\{\\alpha\}\(t\)\}\\right\\rangle\\gtrsim\\frac\{\\eta\\gamma\}\{L\(0\)^\{\-1\}\+\\eta t\}\.Summing froms=0s=0tos=t−1s=t\-1, we have

W​\(t\)≳W​\(0\)\+η​γ​∑s=0t−11L​\(0\)−1\+η​s≥W​\(0\)\+η​γ​∫0td​sL​\(0\)−1\+η​s≥W​\(0\)\+γ​log⁡\(1\+η​L​\(0\)​t\)\.W\(t\)\\gtrsim W\(0\)\+\\eta\\gamma\\sum\_\{s=0\}^\{t\-1\}\\frac\{1\}\{L\(0\)^\{\-1\}\+\\eta s\}\\geq W\(0\)\+\\eta\\gamma\\int\_\{0\}^\{t\}\\frac\{\\mathrm\{d\}s\}\{L\(0\)^\{\-1\}\+\\eta s\}\\geq W\(0\)\+\\gamma\\log\(1\+\\eta L\(0\)t\)\.To escape from poor initialization \(or to haveV​\(t\)≤1V\(t\)\\leq 1\), it is sufficient to haveW​\(t\)=r​\(t\)​\(1−V​\(t\)\)≥0W\(t\)=r\(t\)\(1\-V\(t\)\)\\geq 0\. Hence, by solvingW​\(0\)\+γ​log⁡\(1\+η​L​\(0\)​t\)≥0W\(0\)\+\\gamma\\log\(1\+\\eta L\(0\)t\)\\geq 0, we havet≥Tescapet\\geq T\_\{\\text\{escape\}\}\. ∎

We now state the upper bound of the tangential alignment for discrete\-time gradient descent\. The parameter first escapes from a bad initialization \(Lemma[22](https://arxiv.org/html/2608.04382#Thmtheorem22); escape stage\), and then weakly aligns with the max\-margin direction \(Lemma[21](https://arxiv.org/html/2608.04382#Thmtheorem21); weak alignment stage\)\.

###### Theorem 23\.

Under Assumption[1](https://arxiv.org/html/2608.04382#Thmtheorem1), consider the discrete\-time gradient descent𝐰​\(t\+1\)=𝐰​\(t\)−η​∇L​\(𝐰​\(t\)\)\\mathbf\{w\}\(t\+1\)=\\mathbf\{w\}\(t\)\-\\eta\\nabla L\(\\mathbf\{w\}\(t\)\)\. There exists a problem\-dependent thresholdδ¯\>0\\bar\{\\delta\}\>0such that the following holds: pick a fixed constantΔ\>δ¯\\Delta\>\\bar\{\\delta\}, then for anyδ\>Δ\\delta\>\\Delta, the tangential alignment achievesV​\(t\)≤δV\(t\)\\leq\\deltaaftert=O​\(η−1​exp⁡\(exp⁡\(−δ/κ0​\(Δ\)​γ2\)\)\)t=O\(\\eta^\{\-1\}\\exp\(\\exp\(\-\\delta/\\kappa\_\{0\}\(\\Delta\)\\gamma^\{2\}\)\)\)iterations\.

###### Proof\.\.

The proof is the same as the continuous case, Theorem[8](https://arxiv.org/html/2608.04382#Thmtheorem8)\. Note that our geometric lemma \(Lemma[6](https://arxiv.org/html/2608.04382#Thmtheorem6)\) carries over to the discrete case\. Therefore, Lemma[22](https://arxiv.org/html/2608.04382#Thmtheorem22)establishes the escape stage, and the weak alignment stage follows by combining Lemmas[21](https://arxiv.org/html/2608.04382#Thmtheorem21)and[6](https://arxiv.org/html/2608.04382#Thmtheorem6)\. ∎

Theorem[23](https://arxiv.org/html/2608.04382#Thmtheorem23)confirms that the early\-stage weak alignment phenomenon observed in continuous time persists in the discrete\-time setting\. The two\-stage structure—escaping followed by weak alignment—is identical to the continuous\-time case\.

### E\.3Lower bound

We now establish a lower bound on the tangential alignmentV​\(t\)V\(t\)during the early stage for discrete\-time gradient descent\. Again we establish a matching lower bound, ensuring the tightness of the analysis in the discrete\-time setting\. In this lower bound, we show the least elapsed time necessary for achieving weak alignmentV​\(t\)≈1−γV\(t\)\\approx 1\-\\gamma\. This error tolerance is reasonably chosen for the early stage; see the discussion in Section[3\.1](https://arxiv.org/html/2608.04382#S3.SS1)\.

###### Theorem 24\.

Fixδ\>0\\delta\>0and assumeL​\(0\)<eρ​γ2L\(0\)<e^\{\\rho\\gamma^\{2\}\}\. Under Assumption[1](https://arxiv.org/html/2608.04382#Thmtheorem1), fort=0,1,…,T1t=0,1,\\dots,T\_\{1\}, the parameter following the discrete\-time gradient descent𝐰​\(t\+1\)=𝐰​\(t\)−η​∇L​\(𝐰​\(t\)\)\\mathbf\{w\}\(t\+1\)=\\mathbf\{w\}\(t\)\-\\eta\\nabla L\(\\mathbf\{w\}\(t\)\)satisfies

V​\(t\)≥V​\(0\)−C−2​γ−2​V​\(0\)​log⁡\(log⁡\(L​\(0\)−1\+η​γ2​t\)\),V\(t\)\\geq V\(0\)\-C\-2\\gamma^\{\-2\}\\sqrt\{V\(0\)\}\\log\(\\log\(L\(0\)^\{\-1\}\+\\eta\\gamma^\{2\}t\)\),\(22\)whereC\>0C\>0is a problem\-dependent constant\. In addition, forδ\>1−γ\\delta\>1\-\\gamma,V​\(t\)≥δV\(t\)\\geq\\deltaholds duringt∈\[0,T1\]t\\in\[0,T\_\{1\}\]withT1=Ω​\(η−1​exp⁡\(exp⁡\(−δ\)\)\)T\_\{1\}=\\Omega\(\\eta^\{\-1\}\\exp\(\\exp\(\-\\delta\)\)\)\.

###### Proof\.\.

From the discrete increment ofV​\(t\)V\(t\)in \([21](https://arxiv.org/html/2608.04382#A5.E21)\), we have

Δ​V​\(t\)\\displaystyle\\Delta V\(t\)=η​L​\(𝐰​\(t\)\)r​\(t\)​⟨Z​𝜶​\(t\),P𝐮​\(t\)⟂​𝐮∗⟩−O​\(η2​‖∇L​\(𝐰​\(t\)\)‖2r​\(t\)2\)\\displaystyle=\\frac\{\\eta L\(\\mathbf\{w\}\(t\)\)\}\{r\(t\)\}\\left\\langle\{Z\\bm\{\\alpha\}\(t\)\},\{P\_\{\\mathbf\{u\}\(t\)\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\}\\right\\rangle\-O\\left\(\\frac\{\\eta^\{2\}\\\|\\nabla L\(\\mathbf\{w\}\(t\)\)\\\|^\{2\}\}\{r\(t\)^\{2\}\}\\right\)\(23\)≳−η​L​\(𝐰​\(t\)\)r​\(t\)​2​V​\(t\),\\displaystyle\\gtrsim\-\\frac\{\\eta L\(\\mathbf\{w\}\(t\)\)\}\{r\(t\)\}\\sqrt\{2V\(t\)\},where the bound follows from‖Z​𝜶‖≤1\\\|Z\\bm\{\\alpha\}\\\|\\leq 1,‖P𝐮⟂​𝐮∗‖2≤2​V​\(t\)\\\|P\_\{\\mathbf\{u\}\}^\{\\perp\}\\mathbf\{u\}\_\{\*\}\\\|^\{2\}\\leq 2V\(t\)\(see the proof of Theorem[24](https://arxiv.org/html/2608.04382#Thmtheorem24)\),‖∇L​\(𝐰​\(t\)\)‖≤L​\(𝐰​\(t\)\)\\\|\\nabla L\(\\mathbf\{w\}\(t\)\)\\\|\\leq L\(\\mathbf\{w\}\(t\)\)\(the self\-bounding property \([3](https://arxiv.org/html/2608.04382#S2.E3)\)\), and the stepsize choice, collectively\. Rearranging, forV​\(t\)\>0V\(t\)\>0we have the discrete inequality

Δ​V​\(t\)V​\(t\)≳−2​η​L​\(𝐰​\(t\)\)r​\(t\)\.\\frac\{\\Delta V\(t\)\}\{\\sqrt\{V\(t\)\}\}\\gtrsim\-\\sqrt\{2\}\\frac\{\\eta L\(\\mathbf\{w\}\(t\)\)\}\{r\(t\)\}\.\(24\)By the Taylor expansionV\+Δ​V=V\+Δ​V2​V−O​\(\(Δ​V\)2V3/2\)\\sqrt\{V\+\\Delta V\}=\\sqrt\{V\}\+\\frac\{\\Delta V\}\{2\\sqrt\{V\}\}\-O\\left\(\\frac\{\(\\Delta V\)^\{2\}\}\{V^\{3/2\}\}\\right\), we have

Δ​V​\(s\)V​\(s\)\\displaystyle\\frac\{\\Delta V\(s\)\}\{\\sqrt\{V\(s\)\}\}=2​\[V​\(s\+1\)−V​\(s\)\]\+O​\(\(Δ​V​\(s\)\)2V​\(s\)3/2\)\\displaystyle=2\\left\[\\sqrt\{V\(s\+1\)\}\-\\sqrt\{V\(s\)\}\\right\]\+O\\left\(\\frac\{\(\\Delta V\(s\)\)^\{2\}\}\{V\(s\)^\{3/2\}\}\\right\)≤2​\[V​\(s\+1\)−V​\(s\)\]\+O​\(η2​L​\(𝐰​\(s\)\)2χ​r​\(s\)2\),\(χ≔\(γ​\(1−γ\)\)3/2\)\\displaystyle\\leq 2\\left\[\\sqrt\{V\(s\+1\)\}\-\\sqrt\{V\(s\)\}\\right\]\+O\\left\(\\frac\{\\eta^\{2\}L\(\\mathbf\{w\}\(s\)\)^\{2\}\}\{\\chi r\(s\)^\{2\}\}\\right\),\\quad\\text\{\($\\chi\\coloneqq\(\\gamma\(1\-\\gamma\)\)^\{3/2\}$\)\}where\|Δ​V​\(s\)\|≲η​L​\(𝐰​\(s\)\)/r​\(s\)\|\\Delta V\(s\)\|\\lesssim\\eta L\(\\mathbf\{w\}\(s\)\)/r\(s\)\(from \([23](https://arxiv.org/html/2608.04382#A5.E23)\)\) andV​\(s\)≥γ​\(1−γ\)V\(s\)\\geq\\gamma\(1\-\\gamma\)\(maintained before achievingV​\(t\)≤1−γV\(t\)\\leq 1\-\\gamma; see the discussion after Theorem[8](https://arxiv.org/html/2608.04382#Thmtheorem8)\) are used\. Summing froms=0s=0tos=t−1s=t\-1yields

∑s=0t−1Δ​V​\(s\)V​\(s\)\\displaystyle\\sum\_\{s=0\}^\{t\-1\}\\frac\{\\Delta V\(s\)\}\{\\sqrt\{V\(s\)\}\}≤2​\(V​\(t\)−V​\(0\)\)\+O​\(η2χ​∑s=0t−1L​\(𝐰​\(s\)\)2r​\(s\)2\)\\displaystyle\\leq 2\(\\sqrt\{V\(t\)\}\-\\sqrt\{V\(0\)\}\)\+O\\left\(\\frac\{\\eta^\{2\}\}\{\\chi\}\\sum\_\{s=0\}^\{t\-1\}\\frac\{L\(\\mathbf\{w\}\(s\)\)^\{2\}\}\{r\(s\)^\{2\}\}\\right\)≤2​\(V​\(t\)−V​\(0\)\)\+O​\(ηχ​∑s=0t−1L​\(𝐰​\(s\)\)r​\(s\)\),\\displaystyle\\leq 2\(\\sqrt\{V\(t\)\}\-\\sqrt\{V\(0\)\}\)\+O\\left\(\\frac\{\\eta\}\{\\chi\}\\sum\_\{s=0\}^\{t\-1\}\\frac\{L\(\\mathbf\{w\}\(s\)\)\}\{r\(s\)\}\\right\),where we used the stepsize choice in the last inequality\. Combining this with \([24](https://arxiv.org/html/2608.04382#A5.E24)\), we have

V​\(t\)−V​\(0\)≳−η​\(1\+O​\(χ\)\)2​∑s=0t−1L​\(𝐰​\(s\)\)r​\(s\)\.\\sqrt\{V\(t\)\}\-\\sqrt\{V\(0\)\}\\gtrsim\-\\frac\{\\eta\(1\+O\(\\chi\)\)\}\{\\sqrt\{2\}\}\\sum\_\{s=0\}^\{t\-1\}\\frac\{L\(\\mathbf\{w\}\(s\)\)\}\{r\(s\)\}\.\(25\)We now bound the sum using Lemma[19](https://arxiv.org/html/2608.04382#Thmtheorem19)\. DefiningT0≔1L​\(0\)​η​γ2\[exp\(ρ\+log⁡L​\(0\)1\+γ−2\)−1\]T\_\{0\}\\coloneqq\\frac\{1\}\{L\(0\)\\eta\\gamma^\{2\}\}\\bigr\[\\exp\\bigr\(\\frac\{\\rho\+\\log L\(0\)\}\{1\+\\gamma^\{\-2\}\}\\bigl\)\-1\\bigl\], we have

r​\(t\)≥\{ρ−γ−2​log⁡\(1\+L​\(0\)​η​γ2​t\)ift∈\[0,T0\],log⁡\(L​\(0\)−1\+η​γ2​t\)ift≥T0\.r\(t\)\\geq\\begin\{cases\}\\rho\-\\gamma^\{\-2\}\\log\(1\+L\(0\)\\eta\\gamma^\{2\}t\)&\\text\{if $t\\in\[0,T\_\{0\}\]$,\}\\\\ \\log\(L\(0\)^\{\-1\}\+\\eta\\gamma^\{2\}t\)&\\text\{if $t\\geq T\_\{0\}$\.\}\\end\{cases\}\(26\)Att=T0t=T\_\{0\}, we haver​\(T0\)≥ρ​γ2−log⁡L​\(0\)1\+γ2≕r¯\>0r\(T\_\{0\}\)\\geq\\frac\{\\rho\\gamma^\{2\}\-\\log L\(0\)\}\{1\+\\gamma^\{2\}\}\\eqqcolon\\underline\{r\}\>0, where the strict positivity holds from the assumptionL​\(0\)<eρ​γ2L\(0\)<e^\{\\rho\\gamma^\{2\}\}\. We split the sum into two intervalss∈\[0,T0\]∪\[T0\+1,t−1\]s\\in\[0,T\_\{0\}\]\\cup\[T\_\{0\}\+1,t\-1\]\. For the first intervals∈\[0,T0\]s\\in\[0,T\_\{0\}\], we combiner​\(s\)≥r¯r\(s\)\\geq\\underline\{r\}with the upper bound ofL​\(s\)L\(s\)from Lemma[19](https://arxiv.org/html/2608.04382#Thmtheorem19)to have

∑s=0T0L​\(𝐰​\(s\)\)r​\(s\)≤1r¯​\[L​\(0\)\+∫0T0d​sL​\(0\)−1\+η​γ2​s\]=1r¯​\[L​\(0\)\+ρ\+log⁡L​\(0\)η​\(1\+γ2\)\]\.\\sum\_\{s=0\}^\{T\_\{0\}\}\\frac\{L\(\\mathbf\{w\}\(s\)\)\}\{r\(s\)\}\\leq\\frac\{1\}\{\\underline\{r\}\}\\left\[L\(0\)\+\\int\_\{0\}^\{T\_\{0\}\}\\frac\{\\mathrm\{d\}s\}\{L\(0\)^\{\-1\}\+\\eta\\gamma^\{2\}s\}\\right\]=\\frac\{1\}\{\\underline\{r\}\}\\left\[L\(0\)\+\\frac\{\\rho\+\\log L\(0\)\}\{\\eta\(1\+\\gamma^\{2\}\)\}\\right\]\.For the second intervals∈\[T0\+1,t−1\]s\\in\[T\_\{0\}\+1,t\-1\], we combine \([26](https://arxiv.org/html/2608.04382#A5.E26)\) with the upper bound ofL​\(s\)L\(s\)to have

∑s=T0\+1t−1L​\(𝐰​\(s\)\)r​\(s\)≤∫T0td​sh​\(s\)​log⁡h​\(s\)=1η​γ2​∫h​\(T0\)h​\(t\)d​ςς=1η​γ2​log⁡\(log⁡\(L​\(0\)−1\+η​γ2​t\)ρ​γ2−log⁡L​\(0\)1\+γ−2\),\\sum\_\{s=T\_\{0\}\+1\}^\{t\-1\}\\frac\{L\(\\mathbf\{w\}\(s\)\)\}\{r\(s\)\}\\leq\\int\_\{T\_\{0\}\}^\{t\}\\frac\{\\mathrm\{d\}s\}\{h\(s\)\\log h\(s\)\}=\\frac\{1\}\{\\eta\\gamma^\{2\}\}\\int\_\{h\(T\_\{0\}\)\}^\{h\(t\)\}\\frac\{\\mathrm\{d\}\\varsigma\}\{\\varsigma\}=\\frac\{1\}\{\\eta\\gamma^\{2\}\}\\log\\left\(\\frac\{\\log\(L\(0\)^\{\-1\}\+\\eta\\gamma^\{2\}t\)\}\{\\frac\{\\rho\\gamma^\{2\}\-\\log L\(0\)\}\{1\+\\gamma^\{\-2\}\}\}\\right\),where we use the change of variableς≔h​\(s\)≔log⁡\(L​\(0\)−1\+η​γ2​s\)\\varsigma\\coloneqq h\(s\)\\coloneqq\\log\(L\(0\)^\{\-1\}\+\\eta\\gamma^\{2\}s\)\. Thus, we combine these two intervals to have

∑s=0t−1L​\(𝐰​\(s\)\)r​\(s\)≤\(1\+γ2\)​L​\(0\)ρ​γ2−log⁡L​\(0\)\+ρ\+log⁡L​\(0\)η​\(ρ​γ2−log⁡L​\(0\)\)\+1η​γ2​log⁡\(log⁡\(L​\(0\)−1\+η​γ2​t\)ρ​γ2−log⁡L​\(0\)1\+γ−2\)\.\\sum\_\{s=0\}^\{t\-1\}\\frac\{L\(\\mathbf\{w\}\(s\)\)\}\{r\(s\)\}\\leq\\frac\{\(1\+\\gamma^\{2\}\)L\(0\)\}\{\\rho\\gamma^\{2\}\-\\log L\(0\)\}\+\\frac\{\\rho\+\\log L\(0\)\}\{\\eta\(\\rho\\gamma^\{2\}\-\\log L\(0\)\)\}\+\\frac\{1\}\{\\eta\\gamma^\{2\}\}\\log\\left\(\\frac\{\\log\(L\(0\)^\{\-1\}\+\\eta\\gamma^\{2\}t\)\}\{\\frac\{\\rho\\gamma^\{2\}\-\\log L\(0\)\}\{1\+\\gamma^\{\-2\}\}\}\\right\)\.By plugging this back into \([25](https://arxiv.org/html/2608.04382#A5.E25)\), we have

V​\(t\)≳V​\(0\)\\displaystyle\\sqrt\{V\(t\)\}\\gtrsim\\sqrt\{V\(0\)\}−η​\(1\+O​\(χ\)\)2​\[\(1\+γ2\)​L​\(0\)ρ​γ2−log⁡L​\(0\)\+ρ\+log⁡L​\(0\)η​\(ρ​γ2−log⁡L​\(0\)\)\+1η​γ2​log⁡\(log⁡\(L​\(0\)−1\+η​γ2​t\)ρ​γ2−log⁡L​\(0\)1\+γ−2\)\]\.\\displaystyle\-\\frac\{\\eta\(1\+O\(\\chi\)\)\}\{\\sqrt\{2\}\}\\\!\\left\[\\frac\{\(1\+\\gamma^\{2\}\)L\(0\)\}\{\\rho\\gamma^\{2\}\-\\log L\(0\)\}\+\\frac\{\\rho\+\\log L\(0\)\}\{\\eta\(\\rho\\gamma^\{2\}\-\\log L\(0\)\)\}\+\\frac\{1\}\{\\eta\\gamma^\{2\}\}\\log\\left\(\\frac\{\\log\(L\(0\)^\{\-1\}\+\\eta\\gamma^\{2\}t\)\}\{\\frac\{\\rho\\gamma^\{2\}\-\\log L\(0\)\}\{1\+\\gamma^\{\-2\}\}\}\\right\)\\right\]\\\!\.By squaring both sides and relaxing the lower bound by\(a−b\)2≥a2−2​a​b\(a\-b\)^\{2\}\\geq a^\{2\}\-2ab, we have

V​\(t\)≳V​\(0\)−C0−2​V​\(0\)​\(1\+O​\(χ\)\)γ2⏟≕C1​log⁡\(log⁡\(L​\(0\)−1\+η​γ2​t\)\),V\(t\)\\gtrsim V\(0\)\-C\_\{0\}\-\\underbrace\{\\frac\{\\sqrt\{2V\(0\)\}\(1\+O\(\\chi\)\)\}\{\\gamma^\{2\}\}\}\_\{\\eqqcolon C\_\{1\}\}\\log\\left\(\\log\(L\(0\)^\{\-1\}\+\\eta\\gamma^\{2\}t\)\\right\),where we can take a problem\-dependent constantC0C\_\{0\}\. Lastly, to ensureV​\(t\)≥δV\(t\)\\geq\\delta, we need\(lower bound\)≥δ\\text\{\(lower bound\)\}\\geq\\delta, which yields

t<1η​γ2​exp⁡\[exp⁡\(V​\(0\)−C0−δC1\)\]−1L​\(0\)​η​γ2\.t<\\frac\{1\}\{\\eta\\gamma^\{2\}\}\\exp\\left\[\\exp\\left\(\\frac\{V\(0\)\-C\_\{0\}\-\\delta\}\{C\_\{1\}\}\\right\)\\right\]\-\\frac\{1\}\{L\(0\)\\eta\\gamma^\{2\}\}\.∎

![Refer to caption](https://arxiv.org/html/2608.04382v1/x7.png)

![Refer to caption](https://arxiv.org/html/2608.04382v1/x8.png)

Figure 5:Simulation results with the unnormalized parameter trajectory𝐰​\(t\)\\mathbf\{w\}\(t\)depicted in the loss landscapes, unlike the tangential trajectory𝐮​\(t\)\\mathbf\{u\}\(t\)depicted in Figure[1](https://arxiv.org/html/2608.04382#S1.F1)\. The simulation setup remains the same\. We call the*top*and*bottom*setups, respectively\.

## Appendix FSimulation

#### Expanded simulation results\.

We show simulation results under the same setup as in Figure[1](https://arxiv.org/html/2608.04382#S1.F1)but illustrate the unnormalized parameter trajectory𝐰​\(t\)\\mathbf\{w\}\(t\), instead of the tangential trajectory𝐮​\(t\)\\mathbf\{u\}\(t\)\. The results are shown in Figure[5](https://arxiv.org/html/2608.04382#A5.F5)\. Again, the darker blue color in the \(left\) risk/radial/alignment curves and the \(right\) parameter trajectories indicates the earlier iterationtt, and the lighter blue colors indicate the later iterationtt\. In the loss landscapes, the parameter𝐰​\(t\)\\mathbf\{w\}\(t\)is plotted every7070and5050epochs for each dataset, respectively\. Evidently, the parameter trajectories quickly get away from initialization \(corresponding to the “scattered” trajectory\), and spend significantly more time when approaching the max\-margin direction \(corresponding to the smooth cyan trajectory\)\. The simulation details are as follows:

- •Top setup: - –Initialization𝐰​\(0\)=\[−2\.5,−1\.5\]⊤\\mathbf\{w\}\(0\)=\[\-2\.5,\-1\.5\]^\{\\top\} - –IterationsT=216T=2^\{16\}; stepsizeη=2−10\\eta=2^\{\-10\} - –Max\-margin direction𝐮∗=\[1,1\]⊤/2\\mathbf\{u\}\_\{\*\}=\[1,1\]^\{\\top\}/\\sqrt\{2\} - –Dataset: generaten=50n=50points by sampling𝐳∼𝒩​\(0,\(0\.3\)2​I2\)\\mathbf\{z\}\\sim\\mathcal\{N\}\(0,\(0\.3\)^\{2\}I\_\{2\}\)and perturbing𝐳\\mathbf\{z\}with\+\(2\.5−⟨𝐮∗,𝐳⟩\)​𝐮∗\+\(2\.5\-\\left\\langle\{\\mathbf\{u\}\_\{\*\}\},\{\\mathbf\{z\}\}\\right\\rangle\)\\mathbf\{u\}\_\{\*\}; marginγ∼0\.95\\gamma\\sim 0\.95
- •Bottom setup: - –Initialization𝐰​\(0\)=\[−1\.5,−0\.5\]⊤\\mathbf\{w\}\(0\)=\[\-1\.5,\-0\.5\]^\{\\top\} - –IterationsT=214T=2^\{14\}; stepsizeη=2−10\\eta=2^\{\-10\} - –Max\-margin direction𝐮∗=\[0,1\]⊤\\mathbf\{u\}\_\{\*\}=\[0,1\]^\{\\top\} - –Dataset:𝐳1=\[1,0\.2\]⊤\\mathbf\{z\}\_\{1\}=\[1,0\.2\]^\{\\top\},𝐳2=\[−2,0\.2\]⊤\\mathbf\{z\}\_\{2\}=\[\-2,0\.2\]^\{\\top\},𝐳3=\[0\.2,1\]⊤\\mathbf\{z\}\_\{3\}=\[0\.2,1\]^\{\\top\},𝐳4=\[2,0\.4\]⊤\\mathbf\{z\}\_\{4\}=\[2,0\.4\]^\{\\top\}; marginγ=0\.2\\gamma=0\.2

#### Alignment time\.

We sweep marginγ∈\{0\.1,0\.2,0\.3,0\.4,0\.5\}\\gamma\\in\\\{0\.1,0\.2,0\.3,0\.4,0\.5\\\}over 2D datasets \(𝐰∗=\[1,0\]⊤\\mathbf\{w\}\_\{\*\}=\[1,0\]^\{\\top\},n=50n=50points\)\. Two support vectors are placed at exactly marginγ\\gamma, at\[γ,ε\]⊤\[\\gamma,\\varepsilon\]^\{\\top\}and\[−γ,−ε\]⊤\[\-\\gamma,\-\\varepsilon\]^\{\\top\}withε=0\.01\\varepsilon=0\.01, and the remaining points have margin strictly greater thanγ\\gamma; all points are normalized to‖𝐱i‖≤1\\\|\\mathbf\{x\}\_\{i\}\\\|\\leq 1\. The weights are initialized as𝐰​\(0\)∼0\.01⋅𝒩​\(𝟎,𝐈\)\\mathbf\{w\}\(0\)\\sim 0\.01\\cdot\\mathcal\{N\}\(\\mathbf\{0\},\\mathbf\{I\}\), independent of the data seed\. We use the exponential loss, stepsizeη=2−10\\eta=2^\{\-10\}, andT=214T=2^\{14\}iterations, and record the first iteration at which the alignment⟨𝐰​\(t\)/‖𝐰​\(t\)‖,𝐰∗⟩\\langle\\mathbf\{w\}\(t\)/\\\|\\mathbf\{w\}\(t\)\\\|,\\mathbf\{w\}\_\{\*\}\\ranglereaches each of the thresholdsγ\\gamma\(the standard margin threshold\),γ\\sqrt\{\\gamma\}\(weak yet non\-trivial margin threshold\), and0\.9990\.999\(the asymptotic convergence threshold\)\. Results are shown in Table[1](https://arxiv.org/html/2608.04382#A6.T1), telling that \(i\) the weak alignment time \(up toγ\\gammaandγ\\sqrt\{\\gamma\}\) is much faster than the near\-perfect alignment up to0\.9990\.999; \(ii\) the alignment up toγ\\sqrt\{\\gamma\}does not significantly slow down with smallerγ\\gamma\(hence more difficult datasets\)\.

Table 1:Alignment time \(iterations\) to reach each threshold, as a function of the dataset marginγ\\gamma\.