Vanilla SGD with Momentum Survives Heavy-Tailed Noise: Convergence Analysis without Gradient Clipping or Normalization

# Vanilla SGD with Momentum Survives Heavy-Tailed Noise: Convergence Analysis without Gradient Clipping or Normalization
Source: [https://arxiv.org/html/2607.08104](https://arxiv.org/html/2607.08104)
###### Abstract

Stochastic gradient descent \(SGD\) is a cornerstone of modern optimization\. While its performance under heavy\-tailed noise is often addressed through specialized modifications such as gradient clipping or normalization, we investigate a more fundamental question: how does vanilla SGD, particularly with momentum, perform in the presence of heavy\-tailed noise? In this paper, we refine existing convergence results for vanilla SGD and, more importantly, provide the first comprehensive convergence analysis of vanilla SGD with momentum for strongly convex, convex, and nonconvex objectives, without employing any gradient control mechanisms\. Our results demonstrate that the obtained convergence rates are inferior to the optimal rates achieved by clipped or normalized variants of SGD, thereby revealing inherent limitations of vanilla methods under heavy\-tailed noise\. The theoretical findings are supported by experiments on synthetic functions\.

## 1Introduction

This paper considers the following optimization problem:

min𝒙∈ℝd⁡f​\(𝒙\):=1n​∑i=1nfi​\(𝒙\),\\displaystyle\\min\_\{\\bm\{x\}\\in\\mathbb\{R\}^\{d\}\}f\(\\bm\{x\}\):=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}f\_\{i\}\(\\bm\{x\}\),where eachfif\_\{i\}\(i∈\[n\]i\\in\[n\]\) is differentiable\. We address three fundamental cases whereffis strongly convex, convex, or nonconvex\. Empirical risk minimization is a paramount objective that lies at the core of machine learning\. As primary solvers for this problem, stochastic gradient descent \(SGD\)\[Her1951Ast\]and its momentum\-based variant\[Polyak1964Som,Rumelhart1986Lea\]remain the gold standard in contemporary optimization\. A standard assumption in the convergence analysis of such stochastic algorithms is bounded variance of stochastic noise, specifically, that the second moment of the error between the stochastic and full gradients is bounded\. While numerous seminal results\[Nemirovski2009Rob,Rakhlin2012Mak,Liu2020AnI\]rely on this assumption, recent experimental evidence suggests that stochastic noise in modern deep learning\[Simsekli2019ATa,Battash2024Rev,Ahn2024Lin\]and reinforcement learning\[Garg2021OnP\]often exhibits heavy tails\. Consequently, research has shifted focus toward the convergence behavior of algorithms when the bounded variance assumption is violated, specifically, when stochastic noise possesses only bounded𝔭\\mathfrak\{p\}\-th moments for𝔭∈\(1,2\]\\mathfrak\{p\}\\in\(1,2\]\.

Under this bounded𝔭\\mathfrak\{p\}\-th moment assumption, the convergence of vanilla SGD is known to break down\[Zhang2020Why\]\. To mitigate this issue, clipped SGD, which scales the gradient magnitude to stay within a predefined threshold, has emerged as a robust alternative\. Clipped SGD and its variants have been proven to converge in both expectation\[Zhang2020Why\]and with high probability\[Cutkosky2021Hig,Liu2023Bre,Sadiev2023Hig,Nguyen2023Imp,Nguyen2023Hig,Liu2024Hig\]\. Notably, these studies often incorporate gradient normalization \(see Table[1](https://arxiv.org/html/2607.08104#S1.T1)\), and it has been widely considered essential to clip or normalize gradients to ensure convergence when noise variance is unbounded\.

Recently, several studies have begun to challenge this notion by exploring the convergence of SGD \(with momentum\) without clipping under bounded𝔭\\mathfrak\{p\}\-th moment assumptions\.Hubler2025Frodemonstrated that normalized SGD converges with high probability even without clipping\.Liu2025Nonwere the first to establish that normalized SGD with momentum converges in expectation, achieving rates of𝒪​\(T−𝔭−13​𝔭−2\)\\mathcal\{O\}\\left\(T^\{\-\\frac\{\\mathfrak\{p\}\-1\}\{3\\mathfrak\{p\}\-2\}\}\\right\)when the tail index𝔭\\mathfrak\{p\}is known, and𝒪​\(T−𝔭−12​𝔭\)\\mathcal\{O\}\\left\(T^\{\-\\frac\{\\mathfrak\{p\}\-1\}\{2\\mathfrak\{p\}\}\}\\right\)when it is unknown\. Most recently,Fatkhullin2025CanandHe2025Accproved that vanilla SGD without any normalization or clipping can indeed converge in expectation\. Inspired by these recent developments, we decided to investigate the convergence rate of vanilla SGD with momentum under heavy\-tailed noise\.

Table 1:Comparison of assumptions, algorithmic features, and convergence rates between the related and our work under heavy\-tailed noise\. The abbreviations "w\.o\. Clip\.", "w\.o\. Norm\.", and "w\. Mom\." denote "without gradient clipping," "without gradient normalization," and "with momentum," respectively\.ReferenceFunctionSmoothnessw\.o\. Clip\.w\.o\. Norm\.w\. Mom\.RatesZhang2020WhyS\.C\.LL\-smooth×\\times×\\times×\\times𝒪​\(T−2​\(𝔭−1\)𝔭\)\\mathcal\{O\}\(T^\{\-\\frac\{2\(\\mathfrak\{p\}\-1\)\}\{\\mathfrak\{p\}\}\}\)Sadiev2023HigS\.C\.LL\-smooth×\\times✓×\\times𝒪​\(T−2​\(𝔭−1\)𝔭\)\\mathcal\{O\}\(T^\{\-\\frac\{2\(\\mathfrak\{p\}\-1\)\}\{\\mathfrak\{p\}\}\}\)Fatkhullin2025CanS\.C\.Hölder\-smooth✓✓×\\times𝒪​\(T−\(𝔭−1\)\)\\mathcal\{O\}\(T^\{\-\(\\mathfrak\{p\}\-1\)\}\)Ours\(Theorem[3\.1](https://arxiv.org/html/2607.08104#S3.Thmthm1)\)S\.C\.Hölder\-smooth✓✓×\\times𝒪​\(T−\(𝔭−1\)\)\\mathcal\{O\}\\left\(T^\{\-\(\\mathfrak\{p\}\-1\)\}\\right\)Ours\(Theorem[3\.2](https://arxiv.org/html/2607.08104#S3.Thmthm2)\)S\.C\.Hölder\-smooth✓✓✓𝒪​\(\(1−η\)T\+η𝔭\)\\mathcal\{O\}\\Big\(\(1\-\\eta\)^\{T\}\+\\eta^\{\\mathfrak\{p\}\}\\Big\)Sadiev2023HigC\.LL\-smooth×\\times✓×\\times𝒪​\(T−𝔭−1𝔭\)\\mathcal\{O\}\(T^\{\-\\frac\{\\mathfrak\{p\}\-1\}\{\\mathfrak\{p\}\}\}\)Fatkhullin2025CanC\.Hölder\-smooth✓✓×\\times𝒪​\(T−𝔭−1𝔭\)\\mathcal\{O\}\(T^\{\-\\frac\{\\mathfrak\{p\}\-1\}\{\\mathfrak\{p\}\}\}\)Ours\(Theorem[3\.3](https://arxiv.org/html/2607.08104#S3.Thmthm3)\)C\.Hölder\-smooth✓✓×\\times𝒪​\(log⁡TT\(𝔭−1\)/𝔭\)\\mathcal\{O\}\\Big\(\\frac\{\\log T\}\{T^\{\(\\mathfrak\{p\}\-1\)/\\mathfrak\{p\}\}\}\\Big\)Ours\(Theorem[3\.4](https://arxiv.org/html/2607.08104#S3.Thmthm4)\)C\.Hölder\-smooth✓✓✓𝒪​\(1η​T\+η𝔭−1\)\\mathcal\{O\}\\left\(\\frac\{1\}\{\\eta T\}\+\\eta^\{\\mathfrak\{p\-1\}\}\\right\)Zhang2020WhyN\.C\.LL\-smooth×\\times×\\times×\\times𝒪​\(T−𝔭−13​𝔭−2\)\\mathcal\{O\}\(T^\{\-\\frac\{\\mathfrak\{p\}\-1\}\{3\\mathfrak\{p\}\-2\}\}\)Cutkosky2021HigN\.C\.LL\-smooth×\\times×\\times✓𝒪​\(T𝔭−13​𝔭−2\)\\mathcal\{O\}\(T^\{\\frac\{\\mathfrak\{p\}\-1\}\{3\\mathfrak\{p\}\-2\}\}\)Liu2023BreN\.C\.LL\-smooth×\\times×\\times✓𝒪​\(T−𝔭−13​𝔭−2\)\\mathcal\{O\}\(T^\{\-\\frac\{\\mathfrak\{p\}\-1\}\{3\\mathfrak\{p\}\-2\}\}\)Sadiev2023HigN\.C\.LL\-smooth×\\times✓×\\times𝒪​\(T−𝔭−1𝔭\)\\mathcal\{O\}\(T^\{\-\\frac\{\\mathfrak\{p\}\-1\}\{\\mathfrak\{p\}\}\}\)Nguyen2023HigN\.C\.LL\-smooth×\\times×\\times×\\times𝒪​\(T−𝔭−13​𝔭−2\)\\mathcal\{O\}\(T^\{\-\\frac\{\\mathfrak\{p\}\-1\}\{3\\mathfrak\{p\}\-2\}\}\)Hubler2025FroN\.C\.LL\-smooth✓×\\times×\\times𝒪​\(T−𝔭−13​𝔭−2\)\\mathcal\{O\}\(T^\{\-\\frac\{\\mathfrak\{p\}\-1\}\{3\\mathfrak\{p\}\-2\}\}\)Liu2025NonN\.C\.\(L0,L1\)\(L\_\{0\},L\_\{1\}\)\-smooth✓×\\times✓\\checkmark𝒪​\(T−𝔭−13​𝔭−2\)\\mathcal\{O\}\(T^\{\-\\frac\{\\mathfrak\{p\}\-1\}\{3\\mathfrak\{p\}\-2\}\}\)Fatkhullin2025CanN\.C\.Hölder\-smooth✓✓×\\times𝒪​\(T−𝔭−1𝔭\)\\mathcal\{O\}\(T^\{\-\\frac\{\\mathfrak\{p\}\-1\}\{\\mathfrak\{p\}\}\}\)Ours\(Theorem[3\.5](https://arxiv.org/html/2607.08104#S3.Thmthm5)\)N\.C\.Hölder\-smooth✓✓×\\times𝒪​\(log⁡TT\(𝔭−1\)/𝔭\)\\mathcal\{O\}\\Big\(\\frac\{\\log T\}\{T^\{\(\\mathfrak\{p\}\-1\)/\\mathfrak\{p\}\}\}\\Big\)Ours\(Theorem[3\.6](https://arxiv.org/html/2607.08104#S3.Thmthm6)\)N\.C\.Hölder\-smooth✓✓✓𝒪​\(T−𝔭−12​𝔭\)\\mathcal\{O\}\\Big\(T^\{\-\\frac\{\\mathfrak\{p\}\-1\}\{2\\mathfrak\{p\}\}\}\\Big\)

Our contribution can be summarized as follows:

1. 1\.We provide the first theoretical guarantee that vanilla SGD with momentum converges under heavy\-tailed noise for strongly convex, convex, and nonconvex functions\. Specifically, for nonconvex objectives, we establish a convergence rate of𝒪​\(T−𝔭−12​𝔭\)\\mathcal\{O\}\\left\(T^\{\-\\frac\{\\mathfrak\{p\}\-1\}\{2\\mathfrak\{p\}\}\}\\right\)when the tail index𝔭∈\(1,2\]\\mathfrak\{p\}\\in\(1,2\]is known, and𝒪​\(T−𝔭−14\)\\mathcal\{O\}\\left\(T^\{\-\\frac\{\\mathfrak\{p\}\-1\}\{4\}\}\\right\)when it is unknown \(see Theorem[3\.6](https://arxiv.org/html/2607.08104#S3.Thmthm6)\)\. While these rates are slightly inferior to those of normalized SGD with momentum\[Liu2025Non\], our findings reveal how well vanilla SGD with momentum performs under heavy\-tailed noise, serving as a fundamental baseline for future algorithmic improvements\.
2. 2\.We establish that vanilla SGD converges in expectation under heavy\-tailed noise across all three classes of objective functions\. In contrast to existing work\[Fatkhullin2025Can\], our results do not require the assumption of bounded gradients and hold under significantly weaker conditions, offering a more general and robust theoretical framework\.
3. 3\.Through experiments on synthetic functions, we demonstrate that the conditionν\+1≤𝔭\\nu\+1\\leq\\mathfrak\{p\}is crucial for stable convergence of vanilla SGD with momentum\. Here,ν∈\(0,1\]\\nu\\in\(0,1\]denotes the Hölder continuity parameter \(Assumption[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1)\) and𝔭∈\(1,2\]\\mathfrak\{p\}\\in\(1,2\]is the tail index \(Assumption[2\.2](https://arxiv.org/html/2607.08104#S2.Thmassum2)\)\. This condition highlights a vital alignment between our theory and empirical observations \(Section[4](https://arxiv.org/html/2607.08104#S4)\)\. Notably, since standardLL\-smoothness \(ν=1\\nu=1\) fails to satisfy this condition when noise is strictly heavy\-tailed \(𝔭<2\\mathfrak\{p\}<2\), our results suggest that Hölder smoothness is a more appropriate and versatile framework for analysis in such settings\.

## 2Preliminaries

#### Notation

Letℕ\\mathbb\{N\}be the set of non\-negative integers\. Form∈ℕ∖\{0\}m\\in\\mathbb\{N\}\\setminus\\\{0\\\}, define\[m\]:=\{1,2,…,m\}\[m\]:=\\\{1,2,\\ldots,m\\\}\. Letℝd\\mathbb\{R\}^\{d\}be add\-dimensional Euclidean space with inner product⟨⋅,⋅⟩\\langle\\cdot,\\cdot\\rangle, which induces the norm∥⋅∥\\\|\\cdot\\\|\. Let\(𝒙t\)t≥1⊂ℝd\(\\bm\{x\}\_\{t\}\)\_\{t\\geq 1\}\\subset\\mathbb\{R\}^\{d\}denote the sequence of iterates generated by the recursion in Algorithm[1](https://arxiv.org/html/2607.08104#alg1)\. Letξ\\xibe a random variable, and let𝔼ξ​\[X\]\\mathbb\{E\}\_\{\\xi\}\[X\]denote the expectation with respect toξ\\xiof a random variableXX\.ξt,i\\xi\_\{t,i\}is a random variable generated from theii\-th sampling at timett, and𝝃t:=\(ξt,1,ξt,2,…,ξt,b\)⊤\\bm\{\\xi\}\_\{t\}:=\(\\xi\_\{t,1\},\\xi\_\{t,2\},\\ldots,\\xi\_\{t,b\}\)^\{\\top\}is independent of\(𝒙k\)k=0t\(\\bm\{x\}\_\{k\}\)\_\{k=0\}^\{t\}, whereb∈\[n\]b\\in\[n\]is the batch size\. The independence of𝝃1,𝝃2,…\\bm\{\\xi\}\_\{1\},\\bm\{\\xi\}\_\{2\},\\ldotsallows us to define the total expectation𝔼\\mathbb\{E\}as𝔼=𝔼𝝃1​𝔼𝝃2​⋯​𝔼𝝃t\\mathbb\{E\}=\\mathbb\{E\}\_\{\\bm\{\\xi\}\_\{1\}\}\\mathbb\{E\}\_\{\\bm\{\\xi\}\_\{2\}\}\\cdots\\mathbb\{E\}\_\{\\bm\{\\xi\}\_\{t\}\}\. Let𝖦ξ​\(𝒙\)\\mathsf\{G\}\_\{\\xi\}\(\\bm\{x\}\)be the stochastic gradient off​\(⋅\)f\(\\cdot\)at𝒙∈ℝd\\bm\{x\}\\in\\mathbb\{R\}^\{d\}\. A mini\-batch𝒮t\\mathcal\{S\}\_\{t\}consists ofbbsamples at timett, and the mini\-batch stochastic gradient off​\(𝒙\)f\(\\bm\{x\}\)for𝒮t\\mathcal\{S\}\_\{t\}is defined as∇f𝒮t​\(𝒙\):=1b​∑i∈\[b\]𝖦ξt,i​\(𝒙\)=1b​∑i∈𝒮t∇fi​\(𝒙\)\.\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\):=\\frac\{1\}\{b\}\\sum\_\{i\\in\[b\]\}\\mathsf\{G\}\_\{\\xi\_\{t,i\}\}\(\\bm\{x\}\)=\\frac\{1\}\{b\}\\sum\_\{i\\in\\mathcal\{S\}\_\{t\}\}\\nabla f\_\{i\}\(\\bm\{x\}\)\.We denote𝒙⋆∈argmin𝒙∈ℝdf​\(𝒙\)\\bm\{x\}^\{\\star\}\\in\\operatorname\*\{argmin\}\_\{\\bm\{x\}\\in\\mathbb\{R\}^\{d\}\}f\(\\bm\{x\}\)andf⋆:=f​\(𝒙⋆\)f^\{\\star\}:=f\(\\bm\{x\}^\{\\star\}\)in the strongly convex and convex cases\.

We will refer to SGD with momentum that employs neither normalization nor clipping as vanilla SGD with momentum\. Furthermore, we will refer to vanilla SGD without momentum obtained by settingβ=0\\beta=0in Algorithm[1](https://arxiv.org/html/2607.08104#alg1)simply as vanilla SGD\.

Algorithm 1Vanilla SGD with Momentum0:

𝒙1∈ℝd,ηt\>0,b∈\[n\],β∈\[0,1\),𝒎0:=𝟎,T∈ℕ\\bm\{x\}\_\{1\}\\in\\mathbb\{R\}^\{d\},\\eta\_\{t\}\>0,b\\in\[n\],\\beta\\in\[0,1\),\\bm\{m\}\_\{0\}:=\\bm\{0\},T\\in\\mathbb\{N\}
for

t=1t=1to

TTdo

𝒎t:=β​𝒎t−1\+\(1−β\)​∇f𝒮t​\(𝒙t\)\\bm\{m\}\_\{t\}:=\\beta\\bm\{m\}\_\{t\-1\}\+\(1\-\\beta\)\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)

𝒙t\+1:=𝒙t−ηt​𝒎t\\bm\{x\}\_\{t\+1\}:=\\bm\{x\}\_\{t\}\-\\eta\_\{t\}\\bm\{m\}\_\{t\}

endfor

return

𝒙T\+1\\bm\{x\}\_\{T\+1\}

We make the following assumptions\.

###### Assumption 2\.1\.

∇f:ℝd→ℝd\\nabla f\\colon\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{d\}is Hölder continuous; i\.e\., there existν∈\(0,1\]\\nu\\in\(0,1\]andL\>0L\>0such that, for all𝐱,𝐲∈ℝd\\bm\{x\},\\bm\{y\}\\in\\mathbb\{R\}^\{d\},

‖∇f​\(𝒙\)−∇f​\(𝒚\)‖≤L​‖𝒙−𝒚‖ν\.\\displaystyle\\\|\\nabla f\(\\bm\{x\}\)\-\\nabla f\(\\bm\{y\}\)\\\|\\leq L\\\|\\bm\{x\}\-\\bm\{y\}\\\|^\{\\nu\}\.

###### Assumption 2\.2\.

\(i\) For all𝐱∈ℝd\\bm\{x\}\\in\\mathbb\{R\}^\{d\}that does not depend onξ\\xi,

𝔼ξ​\[𝖦ξ​\(𝒙\)\]=∇f​\(𝒙\)\.\\mathbb\{E\}\_\{\\xi\}\[\\mathsf\{G\}\_\{\\xi\}\(\\bm\{x\}\)\]=\\nabla f\(\\bm\{x\}\)\.\(ii\) There existσ≥0\\sigma\\geq 0and𝔭∈\(1,2\]\\mathfrak\{p\}\\in\(1,2\]such that, for all𝐱∈ℝd\\bm\{x\}\\in\\mathbb\{R\}^\{d\}that does not depend onξ\\xi,

𝔼ξ​\[‖𝖦ξ​\(𝒙\)−∇f​\(𝒙\)‖𝔭\]≤σ𝔭\.\\mathbb\{E\}\_\{\\xi\}\[\\\|\\mathsf\{G\}\_\{\\xi\}\(\\bm\{x\}\)\-\\nabla f\(\\bm\{x\}\)\\\|^\{\\mathfrak\{p\}\}\]\\leq\\sigma^\{\\mathfrak\{p\}\}\.

Assumption[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1)forms the core of our analysis\. Since it reduces to standardLL\-smoothness whenν=1\\nu=1, this assumption represents an extension ofLL\-smoothness\. Assumption[2\.2](https://arxiv.org/html/2607.08104#S2.Thmassum2)\(ii\) is a standard condition for analyzing optimizers under heavy\-tailed noise and is widely used in previous studies\[Zhang2020Why,Cutkosky2021Hig,Sadiev2023Hig,Nguyen2023Imp,Chezhegov2025Cli\]\. When𝔭=2\\mathfrak\{p\}=2, it reduces to the standard bounded variance assumption\[Nemirovski2009Rob,Ghadimi2012Opt,Ghadimi2013Sto\]; when𝔭<2\\mathfrak\{p\}<2, the stochastic gradient may possess unbounded variance\.

### 2\.1Tools for heavy\-tailed analysis

These lemmas are useful for analysis under heavy\-tailed noise\. For the sake of completeness, we include their proofs in Appendix[A](https://arxiv.org/html/2607.08104#A1)\.

###### Lemma 2\.1\.

Suppose that Assumption[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1)holds\. Then, for all𝐱,𝐲∈ℝd\\bm\{x\},\\bm\{y\}\\in\\mathbb\{R\}^\{d\}, the following inequality holds:

f​\(𝒚\)≤f​\(𝒙\)\+⟨∇f​\(𝒙\),𝒚−𝒙⟩\+Lν\+1​‖𝒚−𝒙‖ν\+1\.\\displaystyle f\(\\bm\{y\}\)\\leq f\(\\bm\{x\}\)\+\\langle\\nabla f\(\\bm\{x\}\),\\bm\{y\}\-\\bm\{x\}\\rangle\+\\frac\{L\}\{\\nu\+1\}\\\|\\bm\{y\}\-\\bm\{x\}\\\|^\{\\nu\+1\}\.

###### Lemma 2\.2\.

Letq∈\(1,2\]q\\in\(1,2\]\. For any𝐲∈ℝd\\bm\{y\}\\in\\mathbb\{R\}^\{d\}and any𝐱∈ℝd∖\{𝟎\}\\bm\{x\}\\in\\mathbb\{R\}^\{d\}\\setminus\\\{\\bm\{0\}\\\}, the following inequality holds:

‖𝒙±𝒚‖q≤‖𝒙‖q±q​‖𝒙‖q−2​⟨𝒙,𝒚⟩\+22−q​‖𝒚‖q\.\\displaystyle\\\|\\bm\{x\}\\pm\\bm\{y\}\\\|^\{q\}\\leq\\\|\\bm\{x\}\\\|^\{q\}\\pm q\\\|\\bm\{x\}\\\|^\{q\-2\}\\langle\\bm\{x\},\\bm\{y\}\\rangle\+2^\{2\-q\}\\\|\\bm\{y\}\\\|^\{q\}\.

###### Lemma 2\.3\.

Suppose that Assumption[2\.2](https://arxiv.org/html/2607.08104#S2.Thmassum2)holds\. Then, for allt∈ℕt\\in\\mathbb\{N\}and all𝐱∈ℝd\\bm\{x\}\\in\\mathbb\{R\}^\{d\}that does not depend on𝛏t\\bm\{\\xi\}\_\{t\}, the following inequality holds:

𝔼𝝃t​\[‖∇f𝒮t​\(𝒙\)−∇f​\(𝒙\)‖𝔭\]≤C𝔭​σ𝔭b𝔭−1,\\displaystyle\\mathbb\{E\}\_\{\{\\bm\{\\xi\}\}\_\{t\}\}\[\\\|\\nabla f\_\{\{\\mathcal\{S\}\}\_\{t\}\}\(\\bm\{x\}\)\-\\nabla f\(\\bm\{x\}\)\\\|^\{\\mathfrak\{p\}\}\\big\]\\leq\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\},whereC𝔭∈\[1,2\)C\_\{\\mathfrak\{p\}\}\\in\[1,2\)is the constant given in Lemma[A\.1](https://arxiv.org/html/2607.08104#A1.Thmlem1)\.

###### Lemma 2\.4\.

Let𝔭∈\(1,2\]\\mathfrak\{p\}\\in\(1,2\]andν∈\(0,1\]\\nu\\in\(0,1\], suppose thatν\+1≤𝔭\\nu\+1\\leq\\mathfrak\{p\}\. Then, for all random vectors𝐗∈ℝd\\bm\{X\}\\in\\mathbb\{R\}^\{d\}, the following inequality holds:

𝔼𝑿​\[‖𝑿‖ν\+1\]≤\(𝔼𝑿​\[‖𝑿‖𝔭\]\)ν\+1𝔭\.\\displaystyle\\mathbb\{E\}\_\{\\bm\{X\}\}\[\\\|\\bm\{X\}\\\|^\{\\nu\+1\}\]\\leq\(\\mathbb\{E\}\_\{\\bm\{X\}\}\[\\\|\\bm\{X\}\\\|^\{\\mathfrak\{p\}\}\]\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\.

Lemma[2\.1](https://arxiv.org/html/2607.08104#S2.Thmlem1)characterizes functions with Hölder continuous gradients\. As it recovers the classical descent lemma whenν=1\\nu=1, it can be viewed as a generalized descent lemma \(see, e\.g\.,\[Nesterov2015Uni,Maryam2016Ont\]\)\. Lemma[2\.2](https://arxiv.org/html/2607.08104#S2.Thmlem2)has been extensively discussed in functional analysis\[Beauzamy1982Int,Xu1991Cha,Chidume2009Geo,Rodomanov2020Smo\]; forq=2q=2, it yields the standard expansion of the squared Euclidean norm,‖𝒙±𝒚‖2=‖𝒙‖2±2​⟨𝒙,𝒚⟩\+‖𝒚‖2\\\|\\bm\{x\}\\pm\\bm\{y\}\\\|^\{2\}=\\\|\\bm\{x\}\\\|^\{2\}\\pm 2\\langle\\bm\{x\},\\bm\{y\}\\rangle\+\\\|\\bm\{y\}\\\|^\{2\}\. Lemma[2\.3](https://arxiv.org/html/2607.08104#S2.Thmlem3)is obtained using von Bahr\-Esseen’s inequality\[Bahr1965Ine\]\(Lemma[A\.1](https://arxiv.org/html/2607.08104#A1.Thmlem1)\)\. Notably, it reduces to the standard result𝔼𝝃t​\[‖∇f𝒮t​\(𝒙\)−∇f​\(𝒙\)‖2\]≤σ2b\\mathbb\{E\}\_\{\\bm\{\\xi\}\_\{t\}\}\[\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\)\-\\nabla f\(\\bm\{x\}\)\\\|^\{2\}\]\\leq\\frac\{\\sigma^\{2\}\}\{b\}when𝔭=2\\mathfrak\{p\}=2\. Lemma[2\.4](https://arxiv.org/html/2607.08104#S2.Thmlem4)is crucial for our main results; it dictates the key constraintν\+1≤𝔭\\nu\+1\\leq\\mathfrak\{p\}required for our convergence analysis\.

## 3Main Results

### 3\.1Incompatibility betweenLL\-smoothness and heavy\-tailed noise

LL\-smoothness is a standard assumption in the analysis of first\-order optimization algorithms \(𝒙t\+1:=𝒙t−ηt​𝒅t\\bm\{x\}\_\{t\+1\}:=\\bm\{x\}\_\{t\}\-\\eta\_\{t\}\\bm\{d\}\_\{t\}\)\. It allows the use of the classical descent lemma, which is a special case of Lemma[2\.1](https://arxiv.org/html/2607.08104#S2.Thmlem1)withν=1\\nu=1:

f​\(𝒙t\+1\)\\displaystyle f\(\\bm\{x\}\_\{t\+1\}\)≤f​\(𝒙t\)\+⟨∇f​\(𝒙t\),𝒙t\+1−𝒙t⟩\+L2​‖𝒙t\+1−𝒙t‖2\\displaystyle\\quad\\leq f\(\\bm\{x\}\_\{t\}\)\+\\langle\\nabla f\(\\bm\{x\}\_\{t\}\),\\bm\{x\}\_\{t\+1\}\-\\bm\{x\}\_\{t\}\\rangle\+\\frac\{L\}\{2\}\\\|\\bm\{x\}\_\{t\+1\}\-\\bm\{x\}\_\{t\}\\\|^\{2\}=f​\(𝒙t\)−ηt​⟨∇f​\(𝒙t\),𝒅t⟩\+L​ηt22​‖𝒅t‖2\.\\displaystyle\\quad=f\(\\bm\{x\}\_\{t\}\)\-\\eta\_\{t\}\\langle\\nabla f\(\\bm\{x\}\_\{t\}\),\\bm\{d\}\_\{t\}\\rangle\+\\frac\{L\\eta\_\{t\}^\{2\}\}\{2\}\\\|\\bm\{d\}\_\{t\}\\\|^\{2\}\.Whether analyzing SGD \(𝒅t:=∇f𝒮t​\(𝒙t\)\\bm\{d\}\_\{t\}:=\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\) or any other stochastic variant, one cannot avoid the need to upper\-bound𝔼​\[‖𝒅t‖2\]\\mathbb\{E\}\[\\\|\\bm\{d\}\_\{t\}\\\|^\{2\}\]\. Under the standard bounded variance assumption, this is straightforward:𝔼​\[‖∇f𝒮t​\(𝒙t\)‖2\]≤σ2b\+𝔼​\[‖∇f​\(𝒙t\)‖2\]\\mathbb\{E\}\\left\[\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]\\leq\\frac\{\\sigma^\{2\}\}\{b\}\+\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]\. However, when stochastic noise exhibits heavy\-tailed behavior \(Assumption[2\.2](https://arxiv.org/html/2607.08104#S2.Thmassum2)\) and the variance is unbounded,‖∇f𝒮t​\(𝒙t\)‖2\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}cannot be bounded, causing the standard analysis to collapse\. Consequently, convergence for vanilla SGD or SGD with momentum under heavy\-tailed noise has been considered intractable underLL\-smoothness, leading prior studies to rely on clipping or normalization \(see Section[1](https://arxiv.org/html/2607.08104#S1)\)\.

To circumvent this problem, we utilize the Hölder continuity of the gradient \(Assumption[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1)\)\. Applying Lemma[2\.1](https://arxiv.org/html/2607.08104#S2.Thmlem1)yields:

f​\(𝒙t\+1\)\\displaystyle f\(\\bm\{x\}\_\{t\+1\}\)≤f​\(𝒙t\)−ηt​⟨∇f​\(𝒙t\),𝒅t⟩\+L​ηtν\+1ν\+1​‖𝒅t‖ν\+1\.\\displaystyle\\leq f\(\\bm\{x\}\_\{t\}\)\-\\eta\_\{t\}\\langle\\nabla f\(\\bm\{x\}\_\{t\}\),\\bm\{d\}\_\{t\}\\rangle\+\\frac\{L\\eta\_\{t\}^\{\\nu\+1\}\}\{\\nu\+1\}\\\|\\bm\{d\}\_\{t\}\\\|^\{\\nu\+1\}\.Crucially, this shift of the framework requires an upper\-bound estimate for‖∇f𝒮t​\(𝒙t\)‖ν\+1\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\\|^\{\\nu\+1\}rather than the squared norm\. By leveraging Lemmas[2\.2](https://arxiv.org/html/2607.08104#S2.Thmlem2)–[2\.4](https://arxiv.org/html/2607.08104#S2.Thmlem4), we obtain the following key auxiliary lemma, which serves as the technical foundation for our main results\.

###### Lemma 3\.1\.

Suppose that Assumptions[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1)and[2\.2](https://arxiv.org/html/2607.08104#S2.Thmassum2)hold\. Then, for all𝐱∈ℝd\\bm\{x\}\\in\\mathbb\{R\}^\{d\}that does not depend on𝛏t\\bm\{\\xi\}\_\{t\}, and allt∈ℕt\\in\\mathbb\{N\}, the following holds:

𝔼𝝃t​\[‖∇f𝒮t​\(𝒙\)‖ν\+1\]\\displaystyle\\mathbb\{E\}\_\{\{\\bm\{\\xi\}\}\_\{t\}\}\\left\[\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\)\\\|^\{\\nu\+1\}\\right\]≤21−ν​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+‖∇f​\(𝒙\)‖ν\+1\\displaystyle\\quad\\leq 2^\{1\-\\nu\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+\\\|\\nabla f\(\\bm\{x\}\)\\\|^\{\\nu\+1\}≤21−ν​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+ν\+12​‖∇f​\(𝒙\)‖2\+1−ν2\.\\displaystyle\\quad\\leq 2^\{1\-\\nu\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+\\frac\{\\nu\+1\}\{2\}\\\|\\nabla f\(\\bm\{x\}\)\\\|^\{2\}\+\\frac\{1\-\\nu\}\{2\}\.

Notably, Lemma[3\.1](https://arxiv.org/html/2607.08104#S3.Thmlem1)recovers the standard bound𝔼𝝃t​\[‖∇f𝒮t​\(𝒙\)‖2\]≤σ2b\+‖∇f​\(𝒙\)‖2\\mathbb\{E\}\_\{\\bm\{\\xi\}\_\{t\}\}\\left\[\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\)\\\|^\{2\}\\right\]\\leq\\frac\{\\sigma^\{2\}\}\{b\}\+\\\|\\nabla f\(\\bm\{x\}\)\\\|^\{2\}whenν=1\\nu=1and𝔭=2\\mathfrak\{p\}=2\. This generalization demonstrates that existing analyses underLL\-smoothness and bounded variance can be extended to the heavy\-tailed regime by adopting the Hölder smooth framework and the tools presented in Section[2\.1](https://arxiv.org/html/2607.08104#S2.SS1)\.

Due to space constraints, we will define the following constant frequently appearing in our theorems:

E0:=21−ν​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+1−ν2\.\\displaystyle E\_\{0\}:=2^\{1\-\\nu\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+\\frac\{1\-\\nu\}\{2\}\.\(1\)

### 3\.2Strongly Convex Case

We introduce a key property that holds under Assumption[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1)andμ\\mu\-strong convexity\. Specifically, for all𝒙,𝒚\\bm\{x\},\\bm\{y\}in the domain whereffisμ\\mu\-strongly convex and Hölder smooth, we have:

μ​‖𝒙−𝒚‖2\\displaystyle\\mu\\\|\\bm\{x\}\-\\bm\{y\}\\\|^\{2\}≤⟨∇f​\(𝒙\)−∇f​\(𝒚\),𝒙−𝒚⟩\\displaystyle\\leq\\langle\\nabla f\(\\bm\{x\}\)\-\\nabla f\(\\bm\{y\}\),\\bm\{x\}\-\\bm\{y\}\\rangle≤‖∇f​\(𝒙\)−∇f​\(𝒚\)‖​‖𝒙−𝒚‖\\displaystyle\\leq\\\|\\nabla f\(\\bm\{x\}\)\-\\nabla f\(\\bm\{y\}\)\\\|\\\|\\bm\{x\}\-\\bm\{y\}\\\|≤L​‖𝒙−𝒚‖ν\+1\.\\displaystyle\\leq L\\\|\\bm\{x\}\-\\bm\{y\}\\\|^\{\\nu\+1\}\.That is, we have‖𝒙−𝒚‖1−ν≤Lμ\\\|\\bm\{x\}\-\\bm\{y\}\\\|^\{1\-\\nu\}\\leq\\frac\{L\}\{\\mu\}for all𝒙,𝒚\\bm\{x\},\\bm\{y\}in the domain whereffisμ\\mu\-strongly convex and Hölder smooth \(𝒙≠𝒚\\bm\{x\}\\neq\\bm\{y\}\)\. This property ensures that, forν<1\\nu<1, the coexistence ofμ\\mu\-strong convexity and Hölder smoothness forces the domain to have a bounded diameter of at most\(L/μ\)11−ν\(L/\\mu\)^\{\\frac\{1\}\{1\-\\nu\}\}\. In particular, no function defined on all ofℝd\\mathbb\{R\}^\{d\}can satisfy both properties with finiteLL\. We use this fact in the proof of Theorem[3\.2](https://arxiv.org/html/2607.08104#S3.Thmthm2)\(see Lemma[B\.1](https://arxiv.org/html/2607.08104#A2.Thmlem1)\)\. Whenν=1\\nu=1, this reduces to the standard conditionμ≤L\\mu\\leq L\.

###### Theorem 3\.1\(Convergence Analysis of Vanilla SGD for Strongly Convex Functions\)\.

Let the functionf:ℝd→ℝf\\colon\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}beμ\\mu\-strongly convex\. Suppose that Assumptions[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1)and[2\.2](https://arxiv.org/html/2607.08104#S2.Thmassum2)hold\. With learning rateηt:=2​νμ​t\\eta\_\{t\}:=\\frac\{2\\nu\}\{\\mu t\}, ifν\+1≤𝔭\\nu\+1\\leq\\mathfrak\{p\}andηtν≤1L\\eta\_\{t\}^\{\\nu\}\\leq\\frac\{1\}\{L\}for allt∈\[T\]t\\in\[T\], then:

𝔼​\[f​\(𝒙T\)−f⋆\]≤E1Tν=𝒪​\(1Tν\)\.\\displaystyle\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{T\}\)\-f^\{\\star\}\\right\]\\leq\\frac\{E\_\{1\}\}\{T^\{\\nu\}\}=\\mathcal\{O\}\\left\(\\frac\{1\}\{T^\{\\nu\}\}\\right\)\.In particular, whenν\+1=𝔭\\nu\+1=\\mathfrak\{p\},

𝔼​\[f​\(𝒙T\)−f⋆\]=𝒪​\(1T𝔭−1\),\\displaystyle\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{T\}\)\-f^\{\\star\}\\right\]=\\mathcal\{O\}\\left\(\\frac\{1\}\{T^\{\\mathfrak\{p\}\-1\}\}\\right\),whereE1:=max⁡\{f​\(𝐱1\)−f⋆,E0​Lν​\(ν\+1\)​\(2​νμ\)ν\+1\}E\_\{1\}:=\\max\\left\\\{f\(\\bm\{x\}\_\{1\}\)\-f^\{\\star\},\\frac\{E\_\{0\}L\}\{\\nu\(\\nu\+1\)\}\\left\(\\frac\{2\\nu\}\{\\mu\}\\right\)^\{\\nu\+1\}\\right\\\}andE0E\_\{0\}defined in Eq\. \([1](https://arxiv.org/html/2607.08104#S3.E1)\) are non\-negative constants\.

The proof of Theorem[3\.1](https://arxiv.org/html/2607.08104#S3.Thmthm1)is in Appendix[B\.2](https://arxiv.org/html/2607.08104#A2.SS2)\. Theorem[3\.1](https://arxiv.org/html/2607.08104#S3.Thmthm1)shows that the convergence rate deteriorates whenν\\nuor\(𝔭−1\)\(\\mathfrak\{p\}\-1\)is sufficiently small\. Forν=1\\nu=1and𝔭=2\\mathfrak\{p\}=2, our result recovers the standard𝒪​\(1/T\)\\mathcal\{O\}\(1/T\)rate for vanilla SGD\[Nemirovski2009Rob,Rakhlin2012Mak\]\. UnlikeFatkhullin2025Can, who require bounded gradients to achieve similar rates, our analysis holds without such restrictive assumptions\.

###### Theorem 3\.2\(Convergence Analysis of Vanilla SGD with Momentum for Strongly Convex Functions\)\.

Let the functionf:ℝd→ℝf:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}beμ\\mu\-strongly convex\. Suppose that Assumptions[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1)and[2\.2](https://arxiv.org/html/2607.08104#S2.Thmassum2)hold\. With learning rateηt:=η\\eta\_\{t\}:=\\eta, ifν\+1≤𝔭\\nu\+1\\leq\\mathfrak\{p\}andη≤min⁡\{2​\(1−β\)E2,\(18​L\)1ν,…\}\\eta\\leq\\min\\left\\\{\\frac\{2\(1\-\\beta\)\}\{E\_\{2\}\},\\left\(\\frac\{1\}\{8L\}\\right\)^\{\\frac\{1\}\{\\nu\}\},\\dots\\right\\\}, then:

𝔼​\[f​\(𝒙T\)−f⋆\]\\displaystyle\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{T\}\)\-f^\{\\star\}\\right\]≤f​\(𝒙1\)−f⋆2​\(1−η​E24\)T\+E31−β\\displaystyle\\leq\\frac\{f\(\\bm\{x\}\_\{1\}\)\-f^\{\\star\}\}\{2\}\\left\(1\-\\frac\{\\eta E\_\{2\}\}\{4\}\\right\)^\{T\}\+\\frac\{E\_\{3\}\}\{1\-\\beta\}=𝒪​\(\(1−η​E24\)T\+ην\+1\)\.\\displaystyle=\\mathcal\{O\}\\left\(\\left\(1\-\\frac\{\\eta E\_\{2\}\}\{4\}\\right\)^\{T\}\+\\eta^\{\\nu\+1\}\\right\)\.In particular, whenν\+1=𝔭\\nu\+1=\\mathfrak\{p\},

𝔼​\[f​\(𝒙T\)−f⋆\]\\displaystyle\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{T\}\)\-f^\{\\star\}\\right\]=𝒪​\(\(1−η​E24\)T\+η𝔭\),\\displaystyle=\\mathcal\{O\}\\left\(\\left\(1\-\\frac\{\\eta E\_\{2\}\}\{4\}\\right\)^\{T\}\+\\eta^\{\\mathfrak\{p\}\}\\right\),whereE2:=2​μ​\(ν\+1\)ν​\(1\+L1ν\)\+1E\_\{2\}:=\\frac\{2\\mu\(\\nu\+1\)\}\{\\nu\\left\(1\+L^\{\\frac\{1\}\{\\nu\}\}\\right\)\+1\},E3:=L​E0ν\+1\+\(1\+2​L\)​E0​E24​\(ν\+1\)​\(β1−β\)ν\+1​η\+2​L2​ν​\(1−β\)​E0ν\+1​\(β1−β\)2​ν​ηνE\_\{3\}:=\\frac\{LE\_\{0\}\}\{\\nu\+1\}\+\\frac\{\(1\+2L\)E\_\{0\}E\_\{2\}\}\{4\(\\nu\+1\)\}\\left\(\\frac\{\\beta\}\{1\-\\beta\}\\right\)^\{\\nu\+1\}\\eta\+\\frac\{2L^\{2\}\\nu\(1\-\\beta\)E\_\{0\}\}\{\\nu\+1\}\\left\(\\frac\{\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\\eta^\{\\nu\}, andE0E\_\{0\}defined in Eq\. \([1](https://arxiv.org/html/2607.08104#S3.E1)\) are non\-negative constants\.

The proof of Theorem[3\.2](https://arxiv.org/html/2607.08104#S3.Thmthm2)is in Appendix[B\.3](https://arxiv.org/html/2607.08104#A2.SS3)\. Under standard assumptions \(ν=1,𝔭=2\\nu=1,\\mathfrak\{p\}=2\), vanilla SGD with momentum is known to achieve an𝒪​\(\(1−μ​η\)t\+η\)\\mathcal\{O\}\(\(1\-\\mu\\eta\)^\{t\}\+\\eta\)convergence rate for strongly convex functions\[Liu2020AnI\]\. Theorem[3\.2](https://arxiv.org/html/2607.08104#S3.Thmthm2)recovers this classical result as a special case\. To the best of our knowledge, Theorem[3\.2](https://arxiv.org/html/2607.08104#S3.Thmthm2)provides the first convergence analysis in expectation for vanilla SGD with momentum under heavy\-tailed noise\.

### 3\.3Convex Case

For the convex setting, we introduce an additional assumption regarding the boundedness of the iterates and the auxiliary sequence𝒛t\\bm\{z\}\_\{t\}, where𝒛t:=𝒙t−η​β1−β​𝒎t−1\\bm\{z\}\_\{t\}:=\\bm\{x\}\_\{t\}\-\\frac\{\\eta\\beta\}\{1\-\\beta\}\\bm\{m\}\_\{t\-1\}is defined for the analysis of SGD with momentum\.

###### Assumption 3\.1\.

There exist constantsD1,D2\>0D\_\{1\},D\_\{2\}\>0such that, for allt∈ℕt\\in\\mathbb\{N\}: \(i\)‖𝐱t−𝐱⋆‖≤D1\\\|\\bm\{x\}\_\{t\}\-\\bm\{x\}^\{\\star\}\\\|\\leq D\_\{1\}, \(ii\)‖𝐳t−𝐱⋆‖≤D2\\\|\\bm\{z\}\_\{t\}\-\\bm\{x\}^\{\\star\}\\\|\\leq D\_\{2\}\.

While Assumption[3\.1](https://arxiv.org/html/2607.08104#S3.Thmassum1)is relatively strong, it is a standard way to derive performance upper bounds in both convex and nonconvex optimization\[Nemirovski2009Rob,Kingma2015Ada,Reddi2018Ont,Zhuang2020Ada\]\. Note that while we do not explicitly assume boundedness of the gradient even in the convex case, combining Assumption[3\.1](https://arxiv.org/html/2607.08104#S3.Thmassum1)\(i\) and Assumption[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1)inherently ensures that the gradient remains bounded\.

###### Theorem 3\.3\(Convergence Analysis of Vanilla SGD for Convex Functions\)\.

Let the functionf:ℝd→ℝf:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}be convex\. Suppose that Assumptions[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1),[2\.2](https://arxiv.org/html/2607.08104#S2.Thmassum2), and[3\.1](https://arxiv.org/html/2607.08104#S3.Thmassum1)\(i\) hold\. With learning rateηt:=ηmax​t−1ν\+1\\eta\_\{t\}:=\\eta\_\{\\max\}t^\{\-\\frac\{1\}\{\\nu\+1\}\}, ifν\+1≤𝔭\\nu\+1\\leq\\mathfrak\{p\}, then:

min1≤t≤T⁡𝔼​\[f​\(𝒙t\)−f⋆\]\\displaystyle\\min\_\{1\\leq t\\leq T\}\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\\right\]≤‖𝒙1−𝒙⋆‖ν\+1\(ν\+1\)​D1ν−1​1∑t=1Tηt\+E4​∑t=1Tηtν\+1∑t=1Tηt\\displaystyle\\quad\\leq\\frac\{\\\|\\bm\{x\}\_\{1\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\}\{\(\\nu\+1\)D\_\{1\}^\{\\nu\-1\}\}\\frac\{1\}\{\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}\}\+E\_\{4\}\\frac\{\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}^\{\\nu\+1\}\}\{\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}\}=𝒪​\(log⁡TTνν\+1\)\.\\displaystyle\\quad=\\mathcal\{O\}\\left\(\\frac\{\\log T\}\{T^\{\\frac\{\\nu\}\{\\nu\+1\}\}\}\\right\)\.In particular, whenν\+1=𝔭\\nu\+1=\\mathfrak\{p\},

min1≤t≤T⁡𝔼​\[f​\(𝒙t\)−f⋆\]=𝒪​\(log⁡TT𝔭−1𝔭\),\\displaystyle\\min\_\{1\\leq t\\leq T\}\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\\right\]=\\mathcal\{O\}\\left\(\\frac\{\\log T\}\{T^\{\\frac\{\\mathfrak\{p\}\-1\}\{\\mathfrak\{p\}\}\}\}\\right\),whereE4:=21−ν​\(E0\+ν\+12​L2​D12​ν\)E\_\{4\}:=2^\{1\-\\nu\}\\left\(E\_\{0\}\+\\frac\{\\nu\+1\}\{2\}L^\{2\}D\_\{1\}^\{2\\nu\}\\right\)andE0E\_\{0\}defined in Eq\. \([1](https://arxiv.org/html/2607.08104#S3.E1)\) are non\-negative constants\.

The proof of Theorem[3\.3](https://arxiv.org/html/2607.08104#S3.Thmthm3)is in Appendix[B\.4](https://arxiv.org/html/2607.08104#A2.SS4)\. Under conventional assumptions \(ν=1,𝔭=2\\nu=1,\\mathfrak\{p\}=2\), vanilla SGD is known to achieve a convergence rate of𝒪​\(\(∑ηt\)−1\+∑ηt2/∑ηt\)\\mathcal\{O\}\\big\(\(\\sum\\eta\_\{t\}\)^\{\-1\}\+\\sum\\eta\_\{t\}^\{2\}/\\sum\\eta\_\{t\}\\big\)for convex functions\[Nemirovski2009Rob,Gower2021SGD\]\. Theorem[3\.3](https://arxiv.org/html/2607.08104#S3.Thmthm3)recovers this standard result as a special case\. Under Assumptions[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1)and[2\.2](https://arxiv.org/html/2607.08104#S2.Thmassum2),\[Fatkhullin2025Can, Theorem 3\.2\]previously derived a similar convergence bound, so the novelty of Theorem[3\.3](https://arxiv.org/html/2607.08104#S3.Thmthm3)is incremental\.

###### Theorem 3\.4\(Convergence Analysis of Vanilla SGD with Momentum for Convex Functions\)\.

Let the functionf:ℝd→ℝf:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}be convex\. Suppose that Assumptions[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1),[2\.2](https://arxiv.org/html/2607.08104#S2.Thmassum2),[3\.1](https://arxiv.org/html/2607.08104#S3.Thmassum1)\(i\), and[3\.1](https://arxiv.org/html/2607.08104#S3.Thmassum1)\(ii\) hold\. With learning rateηt:=η\\eta\_\{t\}:=\\eta, ifν\+1≤𝔭\\nu\+1\\leq\\mathfrak\{p\}, then:

min1≤t≤T⁡𝔼​\[f​\(𝒙t\)−f⋆\]\\displaystyle\\min\_\{1\\leq t\\leq T\}\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\\right\]≤‖𝒙1−𝒙⋆‖ν\+1η​\(ν\+1\)​D2ν−1​T\+E6​η\+21−ν​E5\(ν\+1\)​D2ν−1​ην\\displaystyle\\quad\\leq\\frac\{\\\|\\bm\{x\}\_\{1\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\}\{\\eta\(\\nu\+1\)D\_\{2\}^\{\\nu\-1\}T\}\+E\_\{6\}\\eta\+\\frac\{2^\{1\-\\nu\}E\_\{5\}\}\{\(\\nu\+1\)D\_\{2\}^\{\\nu\-1\}\}\\eta^\{\\nu\}=𝒪​\(1η​T\+ην\)\.\\displaystyle\\quad=\\mathcal\{O\}\\left\(\\frac\{1\}\{\\eta T\}\+\\eta^\{\\nu\}\\right\)\.In particular, whenν\+1=𝔭\\nu\+1=\\mathfrak\{p\},

min1≤t≤T⁡𝔼​\[f​\(𝒙t\)−f⋆\]=𝒪​\(1η​T\+η𝔭−1\),\\displaystyle\\min\_\{1\\leq t\\leq T\}\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\\right\]=\\mathcal\{O\}\\left\(\\frac\{1\}\{\\eta T\}\+\\eta^\{\\mathfrak\{p\}\-1\}\\right\),whereE5:=21−ν​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+Lν\+1​D1ν​\(ν\+1\)E\_\{5\}:=2^\{1\-\\nu\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+L^\{\\nu\+1\}D\_\{1\}^\{\\nu\(\\nu\+1\)\},E6:=β​\(E5\+ν​Lν\+1ν​D1ν\+1\)\(1−β\)​\(ν\+1\)E\_\{6\}:=\\frac\{\\beta\\left\(E\_\{5\}\+\\nu L^\{\\frac\{\\nu\+1\}\{\\nu\}\}D\_\{1\}^\{\\nu\+1\}\\right\)\}\{\(1\-\\beta\)\(\\nu\+1\)\}are non\-negative constants\.

The proof of Theorem[3\.4](https://arxiv.org/html/2607.08104#S3.Thmthm4)is in Appendix[B\.5](https://arxiv.org/html/2607.08104#A2.SS5)\. Under standard assumptions \(ν=1,𝔭=2\\nu=1,\\mathfrak\{p\}=2\), vanilla SGD with momentum is known to achieve a convergence rate of𝒪​\(1/\(η​T\)\+η\)\\mathcal\{O\}\(1/\(\\eta T\)\+\\eta\)for convex functions\[Sebbouh2021Alm\]\. Theorem[3\.4](https://arxiv.org/html/2607.08104#S3.Thmthm4)recovers this conventional result as a special case\. To the best of our knowledge, Theorem[3\.4](https://arxiv.org/html/2607.08104#S3.Thmthm4)provides the first convergence analysis in expectation for vanilla SGD with momentum on convex functions under heavy\-tailed noise\.

### 3\.4Nonconvex Case

###### Theorem 3\.5\(Convergence Analysis of Vanilla SGD for Nonconvex Functions\)\.

Let the functionf:ℝd→ℝf:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}be nonconvex\. Suppose that Assumptions[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1)and[2\.2](https://arxiv.org/html/2607.08104#S2.Thmassum2)hold\. With learning rateηt:=ηmax​t−1ν\+1\\eta\_\{t\}:=\\eta\_\{\\max\}t^\{\-\\frac\{1\}\{\\nu\+1\}\}, ifν\+1≤𝔭\\nu\+1\\leq\\mathfrak\{p\}andηtν<2L\\eta\_\{t\}^\{\\nu\}<\\frac\{2\}\{L\}for allt∈\[T\]t\\in\[T\], then:

min1≤t≤T⁡𝔼​\[‖∇f​\(𝒙t\)‖2\]\\displaystyle\\min\_\{1\\leq t\\leq T\}\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]≤2​\(f​\(𝒙1\)−f⋆\)\(2−L​ηmaxν\)​1∑t=1Tηt\+E7​∑t=1Tηtν\+1∑t=1Tηt\\displaystyle\\quad\\leq\\frac\{2\(f\(\\bm\{x\}\_\{1\}\)\-f^\{\\star\}\)\}\{\\left\(2\-L\\eta\_\{\\max\}^\{\\nu\}\\right\)\}\\frac\{1\}\{\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}\}\+E\_\{7\}\\frac\{\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}^\{\\nu\+1\}\}\{\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}\}=𝒪​\(log⁡TTνν\+1\)\.\\displaystyle\\quad=\\mathcal\{O\}\\left\(\\frac\{\\log T\}\{T^\{\\frac\{\\nu\}\{\\nu\+1\}\}\}\\right\)\.In particular, whenν\+1=𝔭\\nu\+1=\\mathfrak\{p\},

min1≤t≤T⁡𝔼​\[‖∇f​\(𝒙t\)‖2\]=𝒪​\(log⁡TT𝔭−1𝔭\),\\displaystyle\\min\_\{1\\leq t\\leq T\}\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]=\\mathcal\{O\}\\left\(\\frac\{\\log T\}\{T^\{\\frac\{\\mathfrak\{p\}\-1\}\{\\mathfrak\{p\}\}\}\}\\right\),whereE7:=2​E0​L\(ν\+1\)​\(2−L​ηmaxν\)E\_\{7\}:=\\frac\{2E\_\{0\}L\}\{\(\\nu\+1\)\\left\(2\-L\\eta\_\{\\max\}^\{\\nu\}\\right\)\}andE0E\_\{0\}defined in Eq\. \([1](https://arxiv.org/html/2607.08104#S3.E1)\) are non\-negative constants\.

The proof of Theorem[3\.5](https://arxiv.org/html/2607.08104#S3.Thmthm5)is in Appendix[B\.6](https://arxiv.org/html/2607.08104#A2.SS6)\. Under standard assumptions \(ν=1,𝔭=2\\nu=1,\\mathfrak\{p\}=2\), vanilla SGD is known to achieve a convergence rate of𝒪​\(\(∑ηt\)−1\+∑ηt2/∑ηt\)\\mathcal\{O\}\\big\(\(\\sum\\eta\_\{t\}\)^\{\-1\}\+\\sum\\eta\_\{t\}^\{2\}/\\sum\\eta\_\{t\}\\big\)for nonconvex functions\[Sebbouh2021Alm,Khaled2023Bet\]\. Theorem[3\.5](https://arxiv.org/html/2607.08104#S3.Thmthm5)recovers this classical result as a special case\. WhileFatkhullin2025Canpreviously derived a similar bound under Assumptions[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1)and[2\.2](https://arxiv.org/html/2607.08104#S2.Thmassum2), their analysis necessitates assuming bounded gradients\. In contrast, our framework eliminates this requirement, providing a more general and robust convergence guarantee for the nonconvex setting\.

###### Theorem 3\.6\(Convergence Analysis of Vanilla SGD with Momentum for Nonconvex Functions\)\.

Let the functionf:ℝd→ℝf:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}be nonconvex\. Suppose that Assumptions[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1)and[2\.2](https://arxiv.org/html/2607.08104#S2.Thmassum2)hold\. With learning rateηt:=η\\eta\_\{t\}:=\\eta, ifν\+1≤𝔭\\nu\+1\\leq\\mathfrak\{p\}andη≤min⁡\{\(14​L\)1ν,1−ββ​\(14​ν​L2\)12​ν\}\\eta\\leq\\min\\left\\\{\\left\(\\frac\{1\}\{4L\}\\right\)^\{\\frac\{1\}\{\\nu\}\},\\frac\{1\-\\beta\}\{\\beta\}\\left\(\\frac\{1\}\{4\\nu L^\{2\}\}\\right\)^\{\\frac\{1\}\{2\\nu\}\}\\right\\\}, then:

min1≤t≤T⁡𝔼​\[‖∇f​\(𝒙t\)‖2\]\\displaystyle\\min\_\{1\\leq t\\leq T\}\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]≤4​\(f​\(𝒙1\)−f⋆\)η​T\+L​E0​ην4​\(ν\+1\)\+E8​η2​ν\\displaystyle\\quad\\leq\\frac\{4\(f\(\\bm\{x\}\_\{1\}\)\-f^\{\\star\}\)\}\{\\eta T\}\+\\frac\{LE\_\{0\}\\eta^\{\\nu\}\}\{4\(\\nu\+1\)\}\+E\_\{8\}\\eta^\{2\\nu\}=𝒪​\(1η​T\+ην\)\.\\displaystyle\\quad=\\mathcal\{O\}\\left\(\\frac\{1\}\{\\eta T\}\+\\eta^\{\\nu\}\\right\)\.In particular, whenν\+1=𝔭\\nu\+1=\\mathfrak\{p\},

min1≤t≤T⁡𝔼​\[‖∇f​\(𝒙t\)‖2\]=𝒪​\(1η​T\+η𝔭−1\),\\displaystyle\\min\_\{1\\leq t\\leq T\}\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]=\\mathcal\{O\}\\left\(\\frac\{1\}\{\\eta T\}\+\\eta^\{\\mathfrak\{p\}\-1\}\\right\),whereE8:=2​Lν\+1​\(β1−β\)2​ν​\{L​\(1−ν\)\+2​E0​ν\}E\_\{8\}:=\\frac\{2L\}\{\\nu\+1\}\\left\(\\frac\{\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\\left\\\{L\(1\-\\nu\)\+2E\_\{0\}\\nu\\right\\\}andE0E\_\{0\}defined in Eq\. \([1](https://arxiv.org/html/2607.08104#S3.E1)\) are non\-negative constants\.

The proof of Theorem[3\.6](https://arxiv.org/html/2607.08104#S3.Thmthm6)is in Appendix[B\.7](https://arxiv.org/html/2607.08104#A2.SS7)\. Under standard assumptions \(ν=1,𝔭=2\\nu=1,\\mathfrak\{p\}=2\), vanilla SGD with momentum is known to achieve a convergence rate of𝒪​\(1/\(η​T\)\+η\)\\mathcal\{O\}\(1/\(\\eta T\)\+\\eta\)for nonconvex functions\[Yan2018Uni,Mai2020Con,Liu2020AnI\]\. Our analytical framework builds upon the approach established byLiu2020AnI, and Theorem[3\.6](https://arxiv.org/html/2607.08104#S3.Thmthm6)recovers their results as a special case whenν=1\\nu=1or𝔭=2\\mathfrak\{p\}=2\. To the best of our knowledge, Theorem[3\.6](https://arxiv.org/html/2607.08104#S3.Thmthm6)represents the first convergence analysis in expectation for vanilla SGD with momentum on nonconvex functions under heavy\-tailed noise\.

#### Comparison with\[Liu2025Non\]

Liu2025Nonestablished that normalized SGD with momentum achieves the following convergence rates for nonconvex functions under heavy\-tailed noise:

min1≤t≤T⁡𝔼​\[‖∇f​\(𝒙t\)‖\]=𝒪​\(T−𝔭−13​𝔭−2\)​\(with known𝔭\),\\displaystyle\\min\_\{1\\leq t\\leq T\}\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|\\right\]=\\mathcal\{O\}\\left\(T^\{\-\\frac\{\\mathfrak\{p\}\-1\}\{3\\mathfrak\{p\}\-2\}\}\\right\)\\ \\text\{\(with known $\\mathfrak\{p\}$\)\},min1≤t≤T⁡𝔼​\[‖∇f​\(𝒙t\)‖\]=𝒪​\(T−𝔭−12​𝔭\)​\(with unknown𝔭\)\.\\displaystyle\\min\_\{1\\leq t\\leq T\}\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|\\right\]=\\mathcal\{O\}\\left\(T^\{\-\\frac\{\\mathfrak\{p\}\-1\}\{2\\mathfrak\{p\}\}\}\\right\)\\ \\text\{\(with unknown $\\mathfrak\{p\}$\)\}\.In contrast, our Theorem[3\.6](https://arxiv.org/html/2607.08104#S3.Thmthm6)implies that vanilla SGD with momentum achieves:

min1≤t≤T⁡𝔼​\[‖∇f​\(𝒙t\)‖\]=𝒪​\(T−𝔭−12​𝔭\)​\(withη=T−1/𝔭\),\\displaystyle\\min\_\{1\\leq t\\leq T\}\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|\\right\]=\\mathcal\{O\}\\left\(T^\{\-\\frac\{\\mathfrak\{p\}\-1\}\{2\\mathfrak\{p\}\}\}\\right\)\\ \\text\{\(with $\\eta=T^\{\-1/\\mathfrak\{p\}\}$\)\},min1≤t≤T⁡𝔼​\[‖∇f​\(𝒙t\)‖\]=𝒪​\(T−p−14\)​\(withη=T−1/2\)\.\\displaystyle\\min\_\{1\\leq t\\leq T\}\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|\\right\]=\\mathcal\{O\}\\left\(T^\{\-\\frac\{p\-1\}\{4\}\}\\right\)\\ \\text\{\(with $\\eta=T^\{\-1/2\}$\)\}\.Since𝔭−13​𝔭−2≥𝔭−12​𝔭≥𝔭−14\\frac\{\\mathfrak\{p\}\-1\}\{3\\mathfrak\{p\}\-2\}\\geq\\frac\{\\mathfrak\{p\}\-1\}\{2\\mathfrak\{p\}\}\\geq\\frac\{\\mathfrak\{p\}\-1\}\{4\}for𝔭∈\(1,2\]\\mathfrak\{p\}\\in\(1,2\], these results illustrate the efficiency of normalization in mitigating heavy\-tailed noise, especially considering that the rate𝒪​\(T−𝔭−13​𝔭−2\)\\mathcal\{O\}\\left\(T^\{\-\\frac\{\\mathfrak\{p\}\-1\}\{3\\mathfrak\{p\}\-2\}\}\\right\)is optimal\[Zhang2020Why\]\. While vanilla SGD with momentum exhibits sub\-optimal rates compared to its normalized counterparts, our work fills a critical gap by characterizing the fundamental convergence behavior of the most widely used momentum\-based optimizer without explicit gradient control\. Our findings reveal that vanilla SGD with momentum maintains sublinear convergence even without normalization or clipping, providing a necessary theoretical baseline that more complex algorithms should aim to surpass\.

#### Analytical framework

As demonstrated, our analysis extends classical results established underLL\-smoothness and bounded variance to more general settings\. These results are obtained by integrating existing analytical frameworks with the technical tools developed in Section[2\.1](https://arxiv.org/html/2607.08104#S2.SS1)\. Specifically, for vanilla SGD with momentum, we employ an auxiliary sequence and a Lyapunov function to facilitate the analysis, following established methodologies \(e\.g\.,\[Ghadimi2015Glo,Gadat2018Sto,Mai2020Con,Liu2020AnI,Defazio2021Mom,Qiu2024Con,Kondo2025Acc\]\)\. Furthermore, for convex and nonconvex settings, we adopt the weighting scheme introduced byStich2019Unito streamline the convergence proofs\. Notably, our framework is sufficiently general to potentially establish convergence guarantees under heavy\-tailed noise for other popular optimizers, such as Adam\[Kingma2015Ada\]and Muon\[Jordan2024Muo\]\.

#### Summary of theoretical findings

Our theoretical analysis provides three key insights\. First, vanilla SGD with momentum guarantees convergence in expectation without requiring explicit gradient control, such as clipping or normalization\. Second, while convergence is guaranteed, the rate is inherently slower than those achieved under classicalLL\-smoothness and bounded variance, or those obtained by heavy\-tailed\-specific methods \(e\.g\., clipping\-based algorithms\)\. Third, the conditionν\+1≤𝔭\\nu\+1\\leq\\mathfrak\{p\}is essential for ensuring these convergence guarantees\. In the following section, we provide numerical experiments to empirically validate these findings, with a particular focus on the necessity of this third condition\.

## 4Numerical Experiments

This section presents numerical experiments to empirically validate the theoretical findings established in Section[3](https://arxiv.org/html/2607.08104#S3)\. We evaluate the performance of vanilla SGD and SGD with momentum on a series of synthetic strongly convex, convex, and nonconvex objective functions to observe and compare their convergence behaviors under heavy\-tailed noise\.

![Refer to caption](https://arxiv.org/html/2607.08104v1/x1.png)Figure 1:Convergence performance across various\(ν,α\)\(\\nu,\\alpha\)pairs\. The heatmaps show the average final objective values \(for strongly convex and convex cases\) and gradient norms \(for nonconvex cases\) after 4,000 steps of vanilla SGD with momentum\. The top 35 performing combinations are indicated by red markers\. Below the red dotted line, the theoretical conditionν\+1≤α\\nu\+1\\leq\\alphais satisfied, ensuring convergence guarantees\.#### Problem setup

We prepared objective functions defined as follows:

fstrongly convex​\(𝒙\)=12​∑i=1dxi2\+∑i=1d1ν\+1​\|xi\|ν\+1,\\displaystyle f\_\{\\text\{strongly convex\}\}\(\\bm\{x\}\)=\\frac\{1\}\{2\}\\sum\_\{i=1\}^\{d\}x\_\{i\}^\{2\}\+\\sum\_\{i=1\}^\{d\}\\frac\{1\}\{\\nu\+1\}\|x\_\{i\}\|^\{\\nu\+1\},fconvex​\(𝒙\)=∑i=1d1ν\+1​\|xi\|ν\+1,\\displaystyle f\_\{\\text\{convex\}\}\(\\bm\{x\}\)=\\sum\_\{i=1\}^\{d\}\\frac\{1\}\{\\nu\+1\}\|x\_\{i\}\|^\{\\nu\+1\},\(2\)fnonconvex​\(𝒙\)=∑i=1d\(1ν\+1​\|xi\|ν\+1−12​cos⁡\(2​π​xi\)\+12\),\\displaystyle f\_\{\\text\{nonconvex\}\}\(\\bm\{x\}\)=\\sum\_\{i=1\}^\{d\}\\left\(\\frac\{1\}\{\\nu\+1\}\|x\_\{i\}\|^\{\\nu\+1\}\-\\frac\{1\}\{2\}\\cos\(2\\pi x\_\{i\}\)\+\\frac\{1\}\{2\}\\right\),where𝒙:=\(x1,x2,…,xd\)⊤∈ℝd\\bm\{x\}:=\(x\_\{1\},x\_\{2\},\\ldots,x\_\{d\}\)^\{\\top\}\\in\\mathbb\{R\}^\{d\}\. For anyν∈\(0,1\]\\nu\\in\(0,1\], each of these functions satisfies Assumption[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1)with a Hölder continuous gradient\. In all experiments, the dimension is set tod=500d=500\. The learning rate was set toηt:=t−1ν\+1\\eta\_\{t\}:=t^\{\-\\frac\{1\}\{\\nu\+1\}\}for convex and nonconvex cases, andηt:=2​νt\\eta\_\{t\}:=\\frac\{2\\nu\}\{t\}for strongly convex cases\.

#### Heavy\-tailed noise setup

The minibatch stochastic gradient∇f𝒮t​\(𝒙t\)\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)in Algorithm[1](https://arxiv.org/html/2607.08104#alg1)was simulated by injecting synthetic heavy\-tailed noise𝝎t∈ℝd\\bm\{\\omega\}\_\{t\}\\in\\mathbb\{R\}^\{d\}into the full gradient:∇f𝒮t​\(𝒙t\)=∇f​\(𝒙t\)\+λ​𝝎t\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)=\\nabla f\(\\bm\{x\}\_\{t\}\)\+\\lambda\\bm\{\\omega\}\_\{t\}, whereλ\>0\\lambda\>0scales the noise intensity\. FollowingFatkhullin2025Can, each component of the synthetic noise was defined as:

\(𝝎t\)i:=si​ui−1α,\\displaystyle\(\\bm\{\\omega\}\_\{t\}\)\_\{i\}:=s\_\{i\}u\_\{i\}^\{\-\\frac\{1\}\{\\alpha\}\},wheresi∼Unif​\(\{−1,1\}\),ui∼Unif​\(0,1\)s\_\{i\}\\sim\\text\{Unif\}\(\\\{\-1,1\\\}\),u\_\{i\}\\sim\\text\{Unif\}\(0,1\), andα∈\(1,2\)\\alpha\\in\(1,2\)is the tail index\. In this two\-sided Pareto distribution, the𝔭\\mathfrak\{p\}\-th moment is finite for all𝔭∈\(1,α\)\\mathfrak\{p\}\\in\(1,\\alpha\)and infinite for𝔭≥α\\mathfrak\{p\}\\geq\\alpha\. Consequently, Assumption[2\.2](https://arxiv.org/html/2607.08104#S2.Thmassum2)\(ii\) is satisfied for any𝔭<α\\mathfrak\{p\}<\\alpha\. Furthermore, as𝔼si​\[si\]=0\\mathbb\{E\}\_\{s\_\{i\}\}\[s\_\{i\}\]=0, it follows from the independence ofsis\_\{i\}anduiu\_\{i\}that𝔼\(si,ui\)​\[\(ωt\)i\]=𝔼si​\[si\]​𝔼ui​\[ui−1/α\]=0\\mathbb\{E\}\_\{\(s\_\{i\},u\_\{i\}\)\}\[\(\\omega\_\{t\}\)\_\{i\}\]=\\mathbb\{E\}\_\{s\_\{i\}\}\[s\_\{i\}\]\\mathbb\{E\}\_\{u\_\{i\}\}\[u\_\{i\}^\{\-1/\\alpha\}\]=0, ensuring that Assumption[2\.2](https://arxiv.org/html/2607.08104#S2.Thmassum2)\(i\) holds for anyα∈\(1,2\)\\alpha\\in\(1,2\)\. For the boundary caseα=2\\alpha=2, we set𝝎t\\bm\{\\omega\}\_\{t\}to standard Gaussian noise\. The noise scaleλ\\lambdawas set to0\.010\.01for strongly convex and convex functions, and0\.10\.1for the nonconvex setting\.

![Refer to caption](https://arxiv.org/html/2607.08104v1/x2.png)![Refer to caption](https://arxiv.org/html/2607.08104v1/x3.png)![Refer to caption](https://arxiv.org/html/2607.08104v1/x4.png)![Refer to caption](https://arxiv.org/html/2607.08104v1/x5.png)

Figure 2:Convergence trajectories of vanilla SGD with momentum\. The plots illustrate the evolution of objective values \(for strongly convex and convex cases\) and gradient norms \(for nonconvex cases\) over 1,000 steps for the functions defined in Eq\. \([4](https://arxiv.org/html/2607.08104#S4.Ex44)\)\. Solid lines represent the average of 20 independent trials, while the lightly shaded regions indicate the range between the minimum and maximum values\.
#### Dependence of convergence onν\\nuandα\\alpha

All theorems in Section[3](https://arxiv.org/html/2607.08104#S3)necessitate the conditionν\+1≤α\\nu\+1\\leq\\alphain order to leverage the moment bounds in Lemma[2\.4](https://arxiv.org/html/2607.08104#S2.Thmlem4)\. Consequently, our theory does not guarantee the convergence of vanilla SGD or SGD with momentum when this condition is violated\. To empirically investigate the interplay betweenν\\nuandα\\alpha, we conducted experiments across 100 combinations ofν∈\{0\.1,…,1\.0\}\\nu\\in\\\{0\.1,\\dots,1\.0\\\}andα∈\{1\.1,…,2\.0\}\\alpha\\in\\\{1\.1,\\dots,2\.0\\\}\. For each pair, SGD with momentum was executed for 4,000 steps, and we recorded the final objective values \(for strongly convex and convex\) or gradient norms \(for nonconvex\)\. Figure[1](https://arxiv.org/html/2607.08104#S4.F1)presents heatmaps averaged over 20 independent trials, where the top 35 performing combinations are highlighted with red markers\. Notably, the conditionν\+1≤α\\nu\+1\\leq\\alphacorresponds to the lower\-right triangular region of each map\. As shown in the figure, the high\-performance markers are predominantly concentrated within this region, empirically validating thatν\+1≤α\\nu\+1\\leq\\alphais a near\-necessary requirement for the stable convergence of vanilla SGD with momentum\. Furthermore, we observe that for small values ofν\\nuorα\\alpha, the optimization performance remains poor even when the convergence condition is satisfied\. This observation aligns with our theoretical rates; for instance, the nonconvex convergence rate𝒪​\(1/\(η​T\)\+ην\)\\mathcal\{O\}\\big\(1/\{\(\\eta T\}\)\+\\eta^\{\\nu\}\\big\)deteriorates whenν\\nuor\(α−1\)\(\\alpha\-1\)is sufficiently small\. Thus, these numerical results provide strong empirical support for our theoretical results\.

#### Detailed observation of convergence behavior

Figure[2](https://arxiv.org/html/2607.08104#S4.F2)illustrates the evolution of objective values and gradient norms for fixedν\\nuand varyingα\\alpha\. Solid lines represent the average of 20 independent trials, with shaded regions indicating the range between extrema\. In all cases, a smallerα\\alphanot only degrades average performance but also amplifies variance, visually confirming our theoretical insight: the divergence of lower\-order moments manifests as extreme instability in optimization trajectories\. Crucially, under highly heavy\-tailed noise \(smallα\\alpha\), largerν\\nuvalues lead to more erratic loss curves, whereas smallerν\\numaintains relative stability\. This aligns with the requirementν\+1≤α\\nu\+1\\leq\\alpha, as a smallerν\\nu, representing broader Hölder smoothness, widens the guaranteed convergence range ofα\\alpha\. These results underscore the inherent incompatibility between extreme heavy\-tailed noise \(α<2\\alpha<2\) and classicalLL\-smoothness \(ν=1\\nu=1\), justifying the adoption of more flexible smoothness classes, such as Hölder or\(L0,L1\)\(L\_\{0\},L\_\{1\}\)\-smoothness, in stochastic optimization theory\. \(See Appendix[C](https://arxiv.org/html/2607.08104#A3)for corresponding results for vanilla SGD\)\.

#### Experiments on WikiText\-2 dataset with Transformer\-XL

![Refer to caption](https://arxiv.org/html/2607.08104v1/x6.png)Figure 3:Loss function value for training versus the number of epochs in training Transformer\-XL on the WikiText\-2 dataset\. The solid lines represent the mean value, and the shaded areas represent the maximum and minimum values over three runs\.Finally, we conducted more realistic experiments on the WikiText\-2 dataset\[Merity2017Int\]with Transformer\-XL, where heavy\-tailed gradient noise has been observed\[Kunstner2023Noi\]\. We evaluated a 6\-layer, 8\-head Transformer\-XL model with approximately 26\.9M parameters\. The embedding, head, and inner dimensions were set to 256, 32, and 1024, respectively, with tied input/output embeddings\. Training was performed for 200 epochs with a batch size of 20, and a target sequence and memory length of 128\. We compared vanilla SGD with momentum and normalized SGD with momentum under constant learning rates\. For each optimizer, we searched learning rates over\{0\.001,0\.005,0\.01,0\.05,0\.1\}\\\{0\.001,0\.005,0\.01,0\.05,0\.1\\\}and selected the best\-performing one on the basis of the final training loss\. The best learning rates were 0\.1 for vanilla SGD with momentum and 0\.005 for normalized SGD with momentum\. The momentum parameter was fixed to 0\.9 for both optimizers\. Figure[3](https://arxiv.org/html/2607.08104#S4.F3)confirms that \(1\) vanilla SGD with momentum converges without gradient clipping or normalization and \(2\) normalized SGD with momentum achieves a lower loss faster, consistent with our theory\. We note that validatingν\+1≤𝔭\\nu\+1\\leq\\mathfrak\{p\}on real tasks would not be feasible sinceν\\nucannot be reliably estimated in practice\. This is why the verification of this condition was conducted on synthetic functions where bothν\\nuand𝔭\\mathfrak\{p\}can be fully controlled\.

## 5Conclusion

In this study, we established a comprehensive convergence analysis for vanilla SGD and SGD with momentum under heavy\-tailed noise, encompassing strongly convex, convex, and nonconvex functions\. A key departure from traditional theory is our shift from classicalLL\-smoothness and bounded variance to a more general framework of Hölder continuous gradients\. This approach naturally extends existing results while proving that convergence in expectation is guaranteed for SGD with momentum, even without explicit gradient control such as clipping or normalization\. Our framework identifies the critical conditionν\+1≤𝔭\\nu\+1\\leq\\mathfrak\{p\}, capturing the intrinsic interplay between function smoothness and the noise tail index\. Numerical experiments empirically validate this condition, demonstrating that Hölder continuity offers a more robust and realistic foundation for heavy\-tailed optimization than standardLL\-smoothness\.

###### Acknowledgements\.

We are sincerely grateful to Program Chairs, Area Chairs, and four anonymous reviewers for helping us improve the original manuscript\. This work was supported by the Japan Society for the Promotion of Science \(JSPS\) KAKENHI Grants, Numbers 24K14846 and 26KJ2026, awarded to Hideaki Iiduka and Naoki Sato, respectively\.

## References

Vanilla SGD with Momentum Survives Heavy\-Tailed Noise: Convergence Analysis without Gradient Clipping or Normalization \(Supplementary Material\)

## Contents

#### Notation\.

Throughout the appendix,ttandTTretain the same meaning as in Algorithm[1](https://arxiv.org/html/2607.08104#alg1):T∈ℕT\\in\\mathbb\{N\}denotes the total number of iterations andttindexes the corresponding iterates\(𝒙t\)t≥1\(\\bm\{x\}\_\{t\}\)\_\{t\\geq 1\}and\(𝒎t\)t≥0\(\\bm\{m\}\_\{t\}\)\_\{t\\geq 0\}generated by Algorithm[1](https://arxiv.org/html/2607.08104#alg1)\.

## Appendix AProofs of Lemmas in Section[2](https://arxiv.org/html/2607.08104#S2)

### A\.1Proof of Lemma[2\.1](https://arxiv.org/html/2607.08104#S2.Thmlem1)

###### Lemma[2\.1](https://arxiv.org/html/2607.08104#S2.Thmlem1)\(restated\)\.

Suppose that Assumption[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1)holds\. Then, for all𝐱,𝐲∈ℝd\\bm\{x\},\\bm\{y\}\\in\\mathbb\{R\}^\{d\}, the following inequality holds\.

f​\(𝒚\)≤f​\(𝒙\)\+⟨∇f​\(𝒙\),𝒚−𝒙⟩\+Lν\+1​‖𝒚−𝒙‖ν\+1\.\\displaystyle f\(\\bm\{y\}\)\\leq f\(\\bm\{x\}\)\+\\langle\\nabla f\(\\bm\{x\}\),\\bm\{y\}\-\\bm\{x\}\\rangle\+\\frac\{L\}\{\\nu\+1\}\\\|\\bm\{y\}\-\\bm\{x\}\\\|^\{\\nu\+1\}\.

###### Proof\.

For allt≥0t\\geq 0, let us define the functionh:ℝ→ℝh\\colon\\mathbb\{R\}\\to\\mathbb\{R\}ash​\(t\):=f​\(t​𝒚\+\(1−t\)​𝒙\)=f​\(𝒙\+t​\(𝒚−𝒙\)\)h\(t\):=f\(t\\bm\{y\}\+\(1\-t\)\\bm\{x\}\)=f\(\\bm\{x\}\+t\(\\bm\{y\}\-\\bm\{x\}\)\)\. Then, we have

h′​\(t\)=⟨∇f​\(𝒙\+t​\(𝒚−𝒙\)\),𝒚−𝒙⟩\.\\displaystyle h^\{\\prime\}\(t\)=\\langle\\nabla f\(\\bm\{x\}\+t\(\\bm\{y\}\-\\bm\{x\}\)\),\\bm\{y\}\-\\bm\{x\}\\rangle\.From the fundamental theorem of calculus,

f​\(𝒚\)−f​\(𝒙\)=h​\(1\)−h​\(0\)=∫01h′​\(t\)​dt=∫01⟨∇f​\(𝒙\+t​\(𝒚−𝒙\)\),𝒚−𝒙⟩​𝑑t,\\displaystyle f\(\\bm\{y\}\)\-f\(\\bm\{x\}\)=h\(1\)\-h\(0\)=\\int\_\{0\}^\{1\}h^\{\\prime\}\(t\)d\_\{t\}=\\int\_\{0\}^\{1\}\\langle\\nabla f\(\\bm\{x\}\+t\(\\bm\{y\}\-\\bm\{x\}\)\),\\bm\{y\}\-\\bm\{x\}\\rangle dt,which implies

f​\(𝒚\)−f​\(𝒙\)\\displaystyle f\(\\bm\{y\}\)\-f\(\\bm\{x\}\)=∫01⟨∇f​\(𝒙\+t​\(𝒚−𝒙\)\),𝒚−𝒙⟩​𝑑t\\displaystyle=\\int\_\{0\}^\{1\}\\langle\\nabla f\(\\bm\{x\}\+t\(\\bm\{y\}\-\\bm\{x\}\)\),\\bm\{y\}\-\\bm\{x\}\\rangle dt=⟨∇f​\(𝒙\),𝒚−𝒙⟩\+∫01⟨∇f​\(𝒙\+t​\(𝒚−𝒙\)\)−∇f​\(𝒙\),𝒚−𝒙⟩​𝑑t\.\\displaystyle=\\langle\\nabla f\(\\bm\{x\}\),\\bm\{y\}\-\\bm\{x\}\\rangle\+\\int\_\{0\}^\{1\}\\langle\\nabla f\(\\bm\{x\}\+t\(\\bm\{y\}\-\\bm\{x\}\)\)\-\\nabla f\(\\bm\{x\}\),\\bm\{y\}\-\\bm\{x\}\\rangle dt\.From the Cauchy\-Schwarz inequality and Assumption[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1),

f​\(𝒚\)−f​\(𝒙\)\\displaystyle f\(\\bm\{y\}\)\-f\(\\bm\{x\}\)≤⟨∇f​\(𝒙\),𝒚−𝒙⟩\+∫01‖∇f​\(𝒙\+t​\(𝒚−𝒙\)\)−∇f​\(𝒙\)‖​‖𝒚−𝒙‖​𝑑t\\displaystyle\\leq\\langle\\nabla f\(\\bm\{x\}\),\\bm\{y\}\-\\bm\{x\}\\rangle\+\\int\_\{0\}^\{1\}\\\|\\nabla f\(\\bm\{x\}\+t\(\\bm\{y\}\-\\bm\{x\}\)\)\-\\nabla f\(\\bm\{x\}\)\\\|\\\|\\bm\{y\}\-\\bm\{x\}\\\|dt≤⟨∇f​\(𝒙\),𝒚−𝒙⟩\+L​‖𝒚−𝒙‖ν\+1​∫01tν​𝑑t\.\\displaystyle\\leq\\langle\\nabla f\(\\bm\{x\}\),\\bm\{y\}\-\\bm\{x\}\\rangle\+L\\\|\\bm\{y\}\-\\bm\{x\}\\\|^\{\\nu\+1\}\\int\_\{0\}^\{1\}t^\{\\nu\}dt\.Here, we have

∫01tν​𝑑t=\[tν\+1ν\+1\]01=1ν\+1\.\\displaystyle\\int\_\{0\}^\{1\}t^\{\\nu\}dt=\\left\[\\frac\{t^\{\\nu\+1\}\}\{\\nu\+1\}\\right\]\_\{0\}^\{1\}=\\frac\{1\}\{\\nu\+1\}\.Therefore, we find that

f​\(𝒚\)−f​\(𝒙\)≤⟨∇f​\(𝒙\),𝒚−𝒙⟩\+Lν\+1​‖𝒚−𝒙‖ν\+1\.\\displaystyle f\(\\bm\{y\}\)\-f\(\\bm\{x\}\)\\leq\\langle\\nabla f\(\\bm\{x\}\),\\bm\{y\}\-\\bm\{x\}\\rangle\+\\frac\{L\}\{\\nu\+1\}\\\|\\bm\{y\}\-\\bm\{x\}\\\|^\{\\nu\+1\}\.This completes the proof\. ∎

### A\.2Proof of Lemma[2\.2](https://arxiv.org/html/2607.08104#S2.Thmlem2)

###### Lemma[2\.2](https://arxiv.org/html/2607.08104#S2.Thmlem2)\(restated\)\.

Letq∈\(1,2\]q\\in\(1,2\]\. For any𝐲∈ℝd\\bm\{y\}\\in\\mathbb\{R\}^\{d\}and any𝐱∈ℝd∖\{𝟎\}\\bm\{x\}\\in\\mathbb\{R\}^\{d\}\\setminus\\\{\\bm\{0\}\\\}, the following inequality holds:

‖𝒙±𝒚‖q≤‖𝒙‖q±q​‖𝒙‖q−2​⟨𝒙,𝒚⟩\+22−q​‖𝒚‖q\.\\displaystyle\\\|\\bm\{x\}\\pm\\bm\{y\}\\\|^\{q\}\\leq\\\|\\bm\{x\}\\\|^\{q\}\\pm q\\\|\\bm\{x\}\\\|^\{q\-2\}\\langle\\bm\{x\},\\bm\{y\}\\rangle\+2^\{2\-q\}\\\|\\bm\{y\}\\\|^\{q\}\.

###### Proof\.

The proof is trivial forq=2q=2\. Thus, consider the case ofq∈\(1,2\)q\\in\(1,2\)\. Let us define

F​\(𝒚\):=‖𝒙‖q\+q​‖𝒙‖q−2​⟨𝒙,𝒚⟩\+22−q​‖𝒚‖q−‖𝒙\+𝒚‖q\.\\displaystyle F\(\\bm\{y\}\):=\\\|\\bm\{x\}\\\|^\{q\}\+q\\\|\\bm\{x\}\\\|^\{q\-2\}\\langle\\bm\{x\},\\bm\{y\}\\rangle\+2^\{2\-q\}\\\|\\bm\{y\}\\\|^\{q\}\-\\\|\\bm\{x\}\+\\bm\{y\}\\\|^\{q\}\.Accordingly, we have

∇F​\(𝒚\)=q​‖𝒙‖q−2​𝒙\+22−q​q​‖𝒚‖q−2​𝒚−q​‖𝒙\+𝒚‖q−2​\(𝒙\+𝒚\)\.\\displaystyle\\nabla F\(\\bm\{y\}\)=q\\\|\\bm\{x\}\\\|^\{q\-2\}\\bm\{x\}\+2^\{2\-q\}q\\\|\\bm\{y\}\\\|^\{q\-2\}\\bm\{y\}\-q\\\|\\bm\{x\}\+\\bm\{y\}\\\|^\{q\-2\}\(\\bm\{x\}\+\\bm\{y\}\)\.When∇F​\(𝒚\)=𝟎\\nabla F\(\\bm\{y\}\)=\\bm\{0\}, the following holds:

\(‖𝒙‖q−2−‖𝒙\+𝒚‖q−2\)​𝒙\+\(22−q​‖𝒚‖q−2−‖𝒙\+𝒚‖q−2\)​𝒚=𝟎\.\\displaystyle\\left\(\\\|\\bm\{x\}\\\|^\{q\-2\}\-\\\|\\bm\{x\}\+\\bm\{y\}\\\|^\{q\-2\}\\right\)\\bm\{x\}\+\\left\(2^\{2\-q\}\\\|\\bm\{y\}\\\|^\{q\-2\}\-\\\|\\bm\{x\}\+\\bm\{y\}\\\|^\{q\-2\}\\right\)\\bm\{y\}=\\bm\{0\}\.Therefore, when∇F​\(𝒚\)=𝟎\\nabla F\(\\bm\{y\}\)=\\bm\{0\},𝒙\\bm\{x\}and𝒚\\bm\{y\}are parallel\. That is, it suffices to show thatF​\(𝒚\)≥0F\(\\bm\{y\}\)\\geq 0holds when𝒙\\bm\{x\}and𝒚\\bm\{y\}are parallel\. Then, let𝒚=t​𝒙\\bm\{y\}=t\\bm\{x\}\(t∈ℝ\)\(t\\in\\mathbb\{R\}\)\. In this case, the inequality to be shown is:

\|t\+1\|q≤1\+q​t\+22−q​\|t\|q\.\\displaystyle\|t\+1\|^\{q\}\\leq 1\+qt\+2^\{2\-q\}\|t\|^\{q\}\.Here, let us defineg​\(t\):=1\+q​t\+22−q​\|t\|q−\|t\+1\|qg\(t\):=1\+qt\+2^\{2\-q\}\|t\|^\{q\}\-\|t\+1\|^\{q\}\. We will show thatg​\(t\)≥0g\(t\)\\geq 0for allt∈ℝt\\in\\mathbb\{R\}\. Note thatg​\(0\)=0g\(0\)=0\.

#### Case \(i\):t≥0t\\geq 0i\.e\.,\|t\|=t\|t\|=tand\|t\+1\|=t\+1\|t\+1\|=t\+1\.

In this case, we have

g​\(t\)=1\+q​t\+22−q​tq−\(t\+1\)q\.\\displaystyle g\(t\)=1\+qt\+2^\{2\-q\}t^\{q\}\-\(t\+1\)^\{q\}\.Therefore,

g′​\(t\)=q​\{1\+22−q​tq−1−\(t\+1\)q−1\}\.\\displaystyle g^\{\\prime\}\(t\)=q\\left\\\{1\+2^\{2\-q\}t^\{q\-1\}\-\(t\+1\)^\{q\-1\}\\right\\\}\.Sincef​\(x\)=xq−1f\(x\)=x^\{q\-1\}is a concave function and satisfiesf​\(0\)≥0f\(0\)\\geq 0, and given that any such concave function satisfiesf​\(x\+y\)≤f​\(x\)\+f​\(y\)f\(x\+y\)\\leq f\(x\)\+f\(y\), we have

g′​\(t\)\\displaystyle g^\{\\prime\}\(t\)≥q​\{1\+22−q​tq−1−\(tq−1\+1\)\}\\displaystyle\\geq q\\left\\\{1\+2^\{2\-q\}t^\{q\-1\}\-\(t^\{q\-1\}\+1\)\\right\\\}=q​\(22−q−1\)​tq−1\\displaystyle=q\\left\(2^\{2\-q\}\-1\\right\)t^\{q\-1\}\>0,\\displaystyle\>0,where we have used22−q\>12^\{2\-q\}\>1in the last inequality\. Therefore,g​\(t\)g\(t\)is monotone increasing fort≥0t\\geq 0\. Sinceg​\(0\)=0g\(0\)=0, it follows thatg​\(t\)≥0g\(t\)\\geq 0fort≥0t\\geq 0\.

#### Case \(ii\):−1≤t<0\-1\\leq t<0i\.e\.,\|t\|=−t\|t\|=\-tand\|t\+1\|=t\+1\|t\+1\|=t\+1\.

In this case, we have

g​\(t\)=1\+q​t\+22−q​\(−t\)q−\(t\+1\)q\.\\displaystyle g\(t\)=1\+qt\+2^\{2\-q\}\(\-t\)^\{q\}\-\(t\+1\)^\{q\}\.Therefore,

g′​\(t\)=q​\{1−22−q​\(−t\)q−1−\(t\+1\)q−1\}\.\\displaystyle g^\{\\prime\}\(t\)=q\\left\\\{1\-2^\{2\-q\}\(\-t\)^\{q\-1\}\-\(t\+1\)^\{q\-1\}\\right\\\}\.Here, let us defineu:=−tu:=\-t\(u∈\(0,1\]\)\(u\\in\(0,1\]\)\. Then,

g′​\(t\)=q​\{1−\(22−q​uq−1\+\(1−u\)q−1\)⏟=⁣:ψ​\(u\)\}\.\\displaystyle g^\{\\prime\}\(t\)=q\\\{1\-\\underbrace\{\(2^\{2\-q\}u^\{q\-1\}\+\(1\-u\)^\{q\-1\}\)\}\_\{=:\\psi\(u\)\}\\\}\.Sinceψ​\(u\)\\psi\(u\)is the sum of concave functions,ψ​\(u\)\\psi\(u\)itself is also concave\. Therefore, the minimum point ofψ​\(u\)\\psi\(u\)on an interval is an endpoint\. Fromlimu→0ψ​\(u\)=1\\displaystyle\\lim\_\{u\\to 0\}\\psi\(u\)=1andψ​\(1\)=22−q\>1\\psi\(1\)=2^\{2\-q\}\>1, we haveψ​\(u\)\>1\\psi\(u\)\>1andg′​\(t\)<0g^\{\\prime\}\(t\)<0\. Therefore,g​\(t\)g\(t\)is monotone decreasing fort∈\[−1,0\)t\\in\[\-1,0\)\. Sinceg​\(−1\)=1−q\+22−q\>0g\(\-1\)=1\-q\+2^\{2\-q\}\>0andg​\(0\)=0g\(0\)=0, it follows thatg​\(t\)≥0g\(t\)\\geq 0fort∈\[−1,0\)t\\in\[\-1,0\)\.

#### Case \(iii\):t<−1t<\-1i\.e\.,\|t\|=−t\|t\|=\-tand\|t\+1\|=−\(t\+1\)\|t\+1\|=\-\(t\+1\)\.

In this case, we have

g​\(t\)=1\+q​t\+22−q​\(−t\)q−\(−t−1\)q\.\\displaystyle g\(t\)=1\+qt\+2^\{2\-q\}\(\-t\)^\{q\}\-\(\-t\-1\)^\{q\}\.Therefore,

g′​\(t\)=q​\{1−22−q​\(−t\)q−1\+\(−t−1\)q−1\}\.\\displaystyle g^\{\\prime\}\(t\)=q\\left\\\{1\-2^\{2\-q\}\(\-t\)^\{q\-1\}\+\(\-t\-1\)^\{q\-1\}\\right\\\}\.Let us definev:=−tv:=\-t\(v\>1\)\(v\>1\)\. Here, we have\(−t\)q−1=vq−1\(\-t\)^\{q\-1\}=v^\{q\-1\}and\(−t−1\)q−1=\(v−1\)q−1\(\-t\-1\)^\{q\-1\}=\(v\-1\)^\{q\-1\}\. Accordingly,

g′​\(t\)\\displaystyle g^\{\\prime\}\(t\)=q​\{1−22−q​vq−1\+\(v−1\)q−1\}\.\\displaystyle=q\\\{1\-2^\{2\-q\}v^\{q\-1\}\+\(v\-1\)^\{q\-1\}\\\}\.From0<q−1<10<q\-1<1and Jensen’s inequality,

1\+\(v−1\)q−12≤\{1\+\(v−1\)2\}q−1=vq−12q−1\.\\displaystyle\\frac\{1\+\(v\-1\)^\{q\-1\}\}\{2\}\\leq\\left\\\{\\frac\{1\+\(v\-1\)\}\{2\}\\right\\\}^\{q\-1\}=\\frac\{v^\{q\-1\}\}\{2^\{\\,q\-1\}\}\.Then, we have1\+\(v−1\)q−1≤22−q​vq−11\+\(v\-1\)^\{q\-1\}\\leq 2^\{2\-q\}v^\{q\-1\}, which impliesg′​\(t\)≤0g^\{\\prime\}\(t\)\\leq 0\. Therefore,g​\(t\)g\(t\)is monotone decreasing fort<−1t<\-1\. Sinceg​\(−1\)=1−q\+22−q\>0g\(\-1\)=1\-q\+2^\{2\-q\}\>0, it follows thatg​\(t\)≥0g\(t\)\\geq 0fort<−1t<\-1\.

Therefore,g​\(t\)≥0g\(t\)\\geq 0holds for anyt∈ℝt\\in\\mathbb\{R\}, and the proof is complete\. ∎

### A\.3Proof of Lemma[2\.3](https://arxiv.org/html/2607.08104#S2.Thmlem3)

###### Lemma A\.1\(Theorem 3\.1 in\[Pinelis2022Mul\]\)\.

Let𝔭∈\(1,2\]\\mathfrak\{p\}\\in\(1,2\], and let𝐗1,𝐗2,…,𝐗n∈ℝd\\bm\{X\}\_\{1\},\\bm\{X\}\_\{2\},\\ldots,\\bm\{X\}\_\{n\}\\in\\mathbb\{R\}^\{d\}be a sequence of independent random variables with𝔼𝐗i​\[𝐗i\]=𝟎\\mathbb\{E\}\_\{\{\\bm\{X\}\}\_\{i\}\}\\left\[\\bm\{X\}\_\{i\}\\right\]=\\bm\{0\}and𝔼𝐗i​\[‖𝐗i‖𝔭\]<∞\\mathbb\{E\}\_\{\{\\bm\{X\}\}\_\{i\}\}\\left\[\\\|\\bm\{X\}\_\{i\}\\\|^\{\\mathfrak\{p\}\}\\right\]<\\inftyfor alli∈\[n\]i\\in\[n\]\. Then,

𝔼\(𝑿1,…,𝑿n\)​\[‖∑i∈\[n\]𝑿i‖𝔭\]≤C𝔭​∑i∈\[n\]𝔼𝑿i​\[‖𝑿i‖𝔭\],\\displaystyle\\mathbb\{E\}\_\{\(\{\{\\bm\{X\}\}\_\{1\}\},\.\.\.,\{\{\\bm\{X\}\}\_\{n\}\}\)\}\\left\[\\left\\\|\\sum\_\{i\\in\[n\]\}\\bm\{X\}\_\{i\}\\right\\\|^\{\\mathfrak\{p\}\}\\right\]\\leq C\_\{\\mathfrak\{p\}\}\\sum\_\{i\\in\[n\]\}\\mathbb\{E\}\_\{\{\\bm\{X\}\}\_\{i\}\}\\left\[\\\|\\bm\{X\}\_\{i\}\\\|^\{\\mathfrak\{p\}\}\\right\],whereC𝔭C\_\{\\mathfrak\{p\}\}is a constant that depends only on𝔭\\mathfrak\{p\}and continuously and strictly decreasing in𝔭∈\(1,2\]\\mathfrak\{p\}\\in\(1,2\]fromlim𝔭→1C𝔭=2\\displaystyle\\lim\_\{\\mathfrak\{p\}\\to 1\}C\_\{\\mathfrak\{p\}\}=2toC2=1C\_\{2\}=1\.

###### Lemma[2\.3](https://arxiv.org/html/2607.08104#S2.Thmlem3)\(restated\)\.

Suppose that Assumption[2\.2](https://arxiv.org/html/2607.08104#S2.Thmassum2)holds\. Then, for all𝐱∈ℝd\\bm\{x\}\\in\\mathbb\{R\}^\{d\}that does not depend on𝛏t\\bm\{\\xi\}\_\{t\}, the following inequality holds:

𝔼𝝃t​\[‖∇f𝒮t​\(𝒙\)−∇f​\(𝒙\)‖𝔭\]≤C𝔭​σ𝔭b𝔭−1,\\displaystyle\\mathbb\{E\}\_\{\{\\bm\{\\xi\}\}\_\{t\}\}\[\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\)\-\\nabla f\(\\bm\{x\}\)\\\|^\{\\mathfrak\{p\}\}\\big\]\\leq\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\},whereC𝔭∈\[1,2\)C\_\{\\mathfrak\{p\}\}\\in\[1,2\)is the constant given in Lemma[A\.1](https://arxiv.org/html/2607.08104#A1.Thmlem1)\.

###### Proof\.

From Assumption[2\.2](https://arxiv.org/html/2607.08104#S2.Thmassum2)and Lemma[A\.1](https://arxiv.org/html/2607.08104#A1.Thmlem1), we have

𝔼𝝃t​\[‖∇fSt​\(𝒙\)−∇f​\(𝒙\)‖𝔭\]\\displaystyle\\mathbb\{E\}\_\{\\bm\{\\xi\}\_\{t\}\}\\left\[\\left\\\|\\nabla f\_\{S\_\{t\}\}\(\\bm\{x\}\)\-\\nabla f\(\\bm\{x\}\)\\right\\\|^\{\\mathfrak\{p\}\}\\right\]=𝔼𝝃t​\[‖1b​∑i∈\[b\]𝖦ξt,i​\(𝒙\)−∇f​\(𝒙\)‖𝔭\]\\displaystyle=\\mathbb\{E\}\_\{\\bm\{\\xi\}\_\{t\}\}\\left\[\\left\\\|\\frac\{1\}\{b\}\\sum\_\{i\\in\[b\]\}\\mathsf\{G\}\_\{\\xi\_\{t,i\}\}\(\\bm\{x\}\)\-\\nabla f\(\\bm\{x\}\)\\right\\\|^\{\\mathfrak\{p\}\}\\right\]=𝔼𝝃t​\[‖1b​∑i∈\[b\]\(𝖦ξt,i​\(𝒙\)−∇f​\(𝒙\)\)‖𝔭\]\\displaystyle=\\mathbb\{E\}\_\{\\bm\{\\xi\}\_\{t\}\}\\left\[\\left\\\|\\frac\{1\}\{b\}\\sum\_\{i\\in\[b\]\}\\bigl\(\\mathsf\{G\}\_\{\\xi\_\{t,i\}\}\(\\bm\{x\}\)\-\\nabla f\(\\bm\{x\}\)\\bigr\)\\right\\\|^\{\\mathfrak\{p\}\}\\right\]=1b𝔭​𝔼𝝃t​\[‖∑i∈\[b\]\(𝖦ξt,i​\(𝒙\)−∇f​\(𝒙\)\)‖𝔭\]\\displaystyle=\\frac\{1\}\{b^\{\\mathfrak\{p\}\}\}\\mathbb\{E\}\_\{\\bm\{\\xi\}\_\{t\}\}\\left\[\\left\\\|\\sum\_\{i\\in\[b\]\}\\bigl\(\\mathsf\{G\}\_\{\\xi\_\{t,i\}\}\(\\bm\{x\}\)\-\\nabla f\(\\bm\{x\}\)\\bigr\)\\right\\\|^\{\\mathfrak\{p\}\}\\right\]≤C𝔭b𝔭​∑i∈\[b\]𝔼𝝃t​\[‖𝖦ξt,i​\(𝒙\)−∇f​\(𝒙\)‖𝔭\]\\displaystyle\\leq\\frac\{C\_\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\}\}\\sum\_\{i\\in\[b\]\}\\mathbb\{E\}\_\{\\bm\{\\xi\}\_\{t\}\}\\left\[\\left\\\|\\mathsf\{G\}\_\{\\xi\_\{t,i\}\}\(\\bm\{x\}\)\-\\nabla f\(\\bm\{x\}\)\\right\\\|^\{\\mathfrak\{p\}\}\\right\]=C𝔭​σ𝔭b𝔭−1\.\\displaystyle=\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\.This completes the proof\. ∎

### A\.4Proof of Lemma[2\.4](https://arxiv.org/html/2607.08104#S2.Thmlem4)

###### Lemma[2\.4](https://arxiv.org/html/2607.08104#S2.Thmlem4)\(restated\)\.

Let𝔭∈\(1,2\]\\mathfrak\{p\}\\in\(1,2\]andν∈\(0,1\]\\nu\\in\(0,1\], suppose thatν\+1≤𝔭\\nu\+1\\leq\\mathfrak\{p\}\. Then, for all random vectors𝐗∈ℝd\\bm\{X\}\\in\\mathbb\{R\}^\{d\}, the following inequality holds:

𝔼𝑿​\[‖𝑿‖ν\+1\]≤\(𝔼𝑿​\[‖𝑿‖𝔭\]\)ν\+1𝔭\.\\displaystyle\\mathbb\{E\}\_\{\\bm\{X\}\}\[\\\|\\bm\{X\}\\\|^\{\\nu\+1\}\]\\leq\\left\(\\mathbb\{E\}\_\{\\bm\{X\}\}\[\\\|\\bm\{X\}\\\|^\{\\mathfrak\{p\}\}\]\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\.

###### Proof\.

Consider the functiong​\(x\)=xkg\(x\)=x^\{k\}defined forx≥0x\\geq 0, wherek=𝔭ν\+1k=\\frac\{\\mathfrak\{p\}\}\{\\nu\+1\}\. Since we assumedν\+1≤𝔭\\nu\+1\\leq\\mathfrak\{p\}, it follows thatk≥1k\\geq 1\.g​\(x\)g\(x\)is a convex function\. From Jensen’s inequalityg​\(𝔼Y​\[Y\]\)≤𝔼Y​\[g​\(Y\)\]g\(\\mathbb\{E\}\_\{Y\}\[Y\]\)\\leq\\mathbb\{E\}\_\{Y\}\[g\(Y\)\], we have

\(𝔼𝑿​\[‖𝑿‖ν\+1\]\)𝔭ν\+1≤𝔼𝑿​\[\(‖𝑿‖ν\+1\)𝔭ν\+1\]=𝔼𝑿​\[‖𝑿‖𝔭\],\\displaystyle\\left\(\\mathbb\{E\}\_\{\\bm\{X\}\}\[\\\|\\bm\{X\}\\\|^\{\\nu\+1\}\]\\right\)^\{\\frac\{\\mathfrak\{p\}\}\{\\nu\+1\}\}\\leq\\mathbb\{E\}\_\{\\bm\{X\}\}\\left\[\\left\(\\\|\\bm\{X\}\\\|^\{\\nu\+1\}\\right\)^\{\\frac\{\\mathfrak\{p\}\}\{\\nu\+1\}\}\\right\]=\\mathbb\{E\}\_\{\\bm\{X\}\}\\left\[\\\|\\bm\{X\}\\\|^\{\\mathfrak\{p\}\}\\right\],where we have chosenY=‖𝑿‖ν\+1Y=\\\|\\bm\{X\}\\\|^\{\\nu\+1\}\. Therefore, we have

𝔼𝑿​\[‖𝑿‖ν\+1\]≤\(𝔼𝑿​\[‖𝑿‖𝔭\]\)ν\+1𝔭\.\\displaystyle\\mathbb\{E\}\_\{\\bm\{X\}\}\[\\\|\\bm\{X\}\\\|^\{\\nu\+1\}\]\\leq\\left\(\\mathbb\{E\}\_\{\\bm\{X\}\}\\left\[\\\|\\bm\{X\}\\\|^\{\\mathfrak\{p\}\}\\right\]\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\.This completes the proof\. ∎

## Appendix BProofs of Lemmas and Theorems in Section[3](https://arxiv.org/html/2607.08104#S3)

### B\.1Proof of Lemma[3\.1](https://arxiv.org/html/2607.08104#S3.Thmlem1)

###### Lemma[3\.1](https://arxiv.org/html/2607.08104#S3.Thmlem1)\(restated\)\.

Suppose that Assumptions[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1)and[2\.2](https://arxiv.org/html/2607.08104#S2.Thmassum2)hold\. Then, for all𝐱∈ℝd\\bm\{x\}\\in\\mathbb\{R\}^\{d\}that does not depend on𝛏t\\bm\{\\xi\}\_\{t\}, the following holds:

𝔼𝝃t​\[‖∇f𝒮t​\(𝒙\)‖ν\+1\]\\displaystyle\\mathbb\{E\}\_\{\\bm\{\\xi\}\_\{t\}\}\\left\[\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\)\\\|^\{\\nu\+1\}\\right\]≤21−ν​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+‖∇f​\(𝒙\)‖ν\+1\\displaystyle\\leq 2^\{1\-\\nu\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+\\\|\\nabla f\(\\bm\{x\}\)\\\|^\{\\nu\+1\}≤21−ν​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+ν\+12​‖∇f​\(𝒙\)‖2\+1−ν2\.\\displaystyle\\leq 2^\{1\-\\nu\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+\\frac\{\\nu\+1\}\{2\}\\\|\\nabla f\(\\bm\{x\}\)\\\|^\{2\}\+\\frac\{1\-\\nu\}\{2\}\.

###### Proof\.

From Lemma[2\.2](https://arxiv.org/html/2607.08104#S2.Thmlem2), we have

‖∇f𝒮t​\(𝒙\)‖ν\+1\\displaystyle\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\)\\\|^\{\\nu\+1\}=‖∇f𝒮t​\(𝒙\)−∇f​\(𝒙\)\+∇f​\(𝒙\)‖ν\+1\\displaystyle=\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\)\-\\nabla f\(\\bm\{x\}\)\+\\nabla f\(\\bm\{x\}\)\\\|^\{\\nu\+1\}≤‖∇f​\(𝒙\)‖ν\+1−\(ν\+1\)​‖∇f​\(𝒙\)‖ν−1​⟨∇f𝒮t​\(𝒙\)−∇f​\(𝒙\),∇f​\(𝒙\)⟩\\displaystyle\\leq\\\|\\nabla f\(\\bm\{x\}\)\\\|^\{\\nu\+1\}\-\(\\nu\+1\)\\\|\\nabla f\(\\bm\{x\}\)\\\|^\{\\nu\-1\}\\langle\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\)\-\\nabla f\(\\bm\{x\}\),\\nabla f\(\\bm\{x\}\)\\rangle\+21−ν​‖∇f𝒮t​\(𝒙\)−∇f​\(𝒙\)‖ν\+1\.\\displaystyle\\quad\+2^\{1\-\\nu\}\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\)\-\\nabla f\(\\bm\{x\}\)\\\|^\{\\nu\+1\}\.By taking the expectation,

𝔼𝝃t​\[‖∇f𝒮t​\(𝒙\)‖ν\+1\]≤𝔼𝝃t​\[‖∇f​\(𝒙\)‖ν\+1\]\+21−ν​𝔼𝝃t​\[‖∇f𝒮t​\(𝒙\)−∇f​\(𝒙\)‖ν\+1\]\.\\displaystyle\\mathbb\{E\}\_\{\\bm\{\\xi\}\_\{t\}\}\\left\[\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\)\\\|^\{\\nu\+1\}\\right\]\\leq\\mathbb\{E\}\_\{\\bm\{\\xi\}\_\{t\}\}\\left\[\\\|\\nabla f\(\\bm\{x\}\)\\\|^\{\\nu\+1\}\\right\]\+2^\{1\-\\nu\}\\mathbb\{E\}\_\{\\bm\{\\xi\}\_\{t\}\}\\left\[\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\)\-\\nabla f\(\\bm\{x\}\)\\\|^\{\\nu\+1\}\\right\]\.Here, from Lemmas[2\.4](https://arxiv.org/html/2607.08104#S2.Thmlem4)and[2\.3](https://arxiv.org/html/2607.08104#S2.Thmlem3), we obtain

𝔼𝝃t​\[‖∇f𝒮t​\(𝒙\)−∇f​\(𝒙\)‖ν\+1\]\\displaystyle\\mathbb\{E\}\_\{\\bm\{\\xi\}\_\{t\}\}\\left\[\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\)\-\\nabla f\(\\bm\{x\}\)\\\|^\{\\nu\+1\}\\right\]≤\(𝔼𝝃t​\[‖∇f𝒮t​\(𝒙\)−∇f​\(𝒙\)‖𝔭\]\)ν\+1𝔭\\displaystyle\\leq\\left\(\\mathbb\{E\}\_\{\\bm\{\\xi\}\_\{t\}\}\\left\[\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\)\-\\nabla f\(\\bm\{x\}\)\\\|^\{\\mathfrak\{p\}\}\\right\]\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}≤\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\.\\displaystyle\\leq\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\.Therefore, we have

𝔼𝝃t​\[‖∇f𝒮t​\(𝒙\)‖ν\+1\]≤𝔼𝝃t​\[‖∇f​\(𝒙\)‖ν\+1\]\+21−ν​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\.\\displaystyle\\mathbb\{E\}\_\{\\bm\{\\xi\}\_\{t\}\}\\left\[\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\)\\\|^\{\\nu\+1\}\\right\]\\leq\\mathbb\{E\}\_\{\\bm\{\\xi\}\_\{t\}\}\\left\[\\\|\\nabla f\(\\bm\{x\}\)\\\|^\{\\nu\+1\}\\right\]\+2^\{1\-\\nu\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\.In addition, from Young’s inequality,

‖∇f​\(𝒙\)‖ν\+1⋅1≤ν\+12​‖∇f​\(𝒙\)‖2\+1−ν2\.\\displaystyle\\\|\\nabla f\(\\bm\{x\}\)\\\|^\{\\nu\+1\}\\cdot 1\\leq\\frac\{\\nu\+1\}\{2\}\\\|\\nabla f\(\\bm\{x\}\)\\\|^\{2\}\+\\frac\{1\-\\nu\}\{2\}\.Therefore,

𝔼𝝃t​\[‖∇f𝒮t​\(𝒙\)‖ν\+1\]≤21−ν​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+ν\+12​‖∇f​\(𝒙\)‖2\+1−ν2\.\\displaystyle\\mathbb\{E\}\_\{\\bm\{\\xi\}\_\{t\}\}\\left\[\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\)\\\|^\{\\nu\+1\}\\right\]\\leq 2^\{1\-\\nu\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+\\frac\{\\nu\+1\}\{2\}\\\|\\nabla f\(\\bm\{x\}\)\\\|^\{2\}\+\\frac\{1\-\\nu\}\{2\}\.This completes the proof\. ∎

### B\.2Proof of Theorem[3\.1](https://arxiv.org/html/2607.08104#S3.Thmthm1)

###### Theorem[3\.1](https://arxiv.org/html/2607.08104#S3.Thmthm1)\(restated\)\.

Let the functionf:ℝd→ℝf\\colon\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}beμ\\mu\-strongly convex\. Suppose that Assumptions[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1)and[2\.2](https://arxiv.org/html/2607.08104#S2.Thmassum2)hold\. With learning rateηt:=2​νμ​t\\eta\_\{t\}:=\\frac\{2\\nu\}\{\\mu t\}, ifν\+1≤𝔭\\nu\+1\\leq\\mathfrak\{p\}andηtν≤1L\\eta\_\{t\}^\{\\nu\}\\leq\\frac\{1\}\{L\}for allt∈\[T\]t\\in\[T\], then:

𝔼​\[f​\(𝒙T\)−f⋆\]≤E1Tν=𝒪​\(1Tν\)\.\\displaystyle\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{T\}\)\-f^\{\\star\}\\right\]\\leq\\frac\{E\_\{1\}\}\{T^\{\\nu\}\}=\\mathcal\{O\}\\left\(\\frac\{1\}\{T^\{\\nu\}\}\\right\)\.In particular, whenν\+1=𝔭\\nu\+1=\\mathfrak\{p\},

𝔼​\[f​\(𝒙T\)−f⋆\]=𝒪​\(1T𝔭−1\),\\displaystyle\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{T\}\)\-f^\{\\star\}\\right\]=\\mathcal\{O\}\\left\(\\frac\{1\}\{T^\{\\mathfrak\{p\}\-1\}\}\\right\),whereE1:=max⁡\{f​\(𝐱1\)−f⋆,E0​Lν​\(ν\+1\)​\(2​νμ\)ν\+1\}E\_\{1\}:=\\max\\left\\\{f\(\\bm\{x\}\_\{1\}\)\-f^\{\\star\},\\frac\{E\_\{0\}L\}\{\\nu\(\\nu\+1\)\}\\left\(\\frac\{2\\nu\}\{\\mu\}\\right\)^\{\\nu\+1\}\\right\\\}andE0E\_\{0\}defined in Eq\. \([1](https://arxiv.org/html/2607.08104#S3.E1)\) are non\-negative constants\.

###### Proof\.

From Lemma[2\.1](https://arxiv.org/html/2607.08104#S2.Thmlem1), we have

f​\(𝒙t\+1\)\\displaystyle f\(\\bm\{x\}\_\{t\+1\}\)≤f​\(𝒙t\)\+⟨∇f​\(𝒙t\),𝒙t\+1−𝒙t⟩\+Lν\+1​‖𝒙t\+1−𝒙t‖ν\+1\\displaystyle\\leq f\(\\bm\{x\}\_\{t\}\)\+\\langle\\nabla f\(\\bm\{x\}\_\{t\}\),\\bm\{x\}\_\{t\+1\}\-\\bm\{x\}\_\{t\}\\rangle\+\\frac\{L\}\{\\nu\+1\}\\\|\\bm\{x\}\_\{t\+1\}\-\\bm\{x\}\_\{t\}\\\|^\{\\nu\+1\}=f​\(𝒙t\)−ηt​⟨∇f​\(𝒙t\),∇f𝒮t​\(𝒙t\)⟩\+L​ηtν\+1ν\+1​‖∇f𝒮t​\(𝒙t\)‖ν\+1\.\\displaystyle=f\(\\bm\{x\}\_\{t\}\)\-\\eta\_\{t\}\\langle\\nabla f\(\\bm\{x\}\_\{t\}\),\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\rangle\+\\frac\{L\\eta\_\{t\}^\{\\nu\+1\}\}\{\\nu\+1\}\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\\|^\{\\nu\+1\}\.By taking the expectation, we have

𝔼​\[f​\(𝒙t\+1\)\]\\displaystyle\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\+1\}\)\\right\]≤𝔼​\[f​\(𝒙t\)\]−ηt​𝔼​\[‖∇f​\(𝒙t\)‖2\]\+L​ηtν\+1ν\+1​𝔼​\[‖∇f𝒮t​\(𝒙t\)‖ν\+1\]\.\\displaystyle\\leq\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\}\)\\right\]\-\\eta\_\{t\}\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]\+\\frac\{L\\eta\_\{t\}^\{\\nu\+1\}\}\{\\nu\+1\}\\mathbb\{E\}\\left\[\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\\|^\{\\nu\+1\}\\right\]\.Furthermore, from Lemma[3\.1](https://arxiv.org/html/2607.08104#S3.Thmlem1)andηtν≤1L\\eta\_\{t\}^\{\\nu\}\\leq\\frac\{1\}\{L\},

𝔼​\[f​\(𝒙t\+1\)\]\\displaystyle\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\+1\}\)\\right\]≤𝔼​\[f​\(𝒙t\)\]−ηt​𝔼​\[‖∇f​\(𝒙t\)‖2\]\+L​ηtν\+1ν\+1​\{21−ν​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+ν\+12​𝔼​\[‖∇f​\(𝒙t\)‖2\]\+1−ν2\}\\displaystyle\\leq\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\}\)\\right\]\-\\eta\_\{t\}\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]\+\\frac\{L\\eta\_\{t\}^\{\\nu\+1\}\}\{\\nu\+1\}\\left\\\{2^\{1\-\\nu\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+\\frac\{\\nu\+1\}\{2\}\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]\+\\frac\{1\-\\nu\}\{2\}\\right\\\}=𝔼​\[f​\(𝒙t\)\]−ηt​\(1−L​ηtν2\)​𝔼​\[‖∇f​\(𝒙t\)‖2\]\+L​ηtν\+1ν\+1​\{21−ν​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+1−ν2\}\\displaystyle=\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\}\)\\right\]\-\\eta\_\{t\}\\left\(1\-\\frac\{L\\eta\_\{t\}^\{\\nu\}\}\{2\}\\right\)\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]\+\\frac\{L\\eta\_\{t\}^\{\\nu\+1\}\}\{\\nu\+1\}\\left\\\{2^\{1\-\\nu\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+\\frac\{1\-\\nu\}\{2\}\\right\\\}≤𝔼​\[f​\(𝒙t\)\]−ηt2​𝔼​\[‖∇f​\(𝒙t\)‖2\]\+Lν\+1​\{21−ν​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+1−ν2\}⏟=⁣:E0​ηtν\+1\.\\displaystyle\\leq\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\}\)\\right\]\-\\frac\{\\eta\_\{t\}\}\{2\}\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]\+\\frac\{L\}\{\\nu\+1\}\\underbrace\{\\left\\\{2^\{1\-\\nu\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+\\frac\{1\-\\nu\}\{2\}\\right\\\}\}\_\{=:E\_\{0\}\}\\eta\_\{t\}^\{\\nu\+1\}\.Here, sinceffisμ\\mu\-strongly convex, for all𝒙∈ℝd\\bm\{x\}\\in\\mathbb\{R\}^\{d\}, we have

f​\(𝒙\)−f⋆\\displaystyle f\(\\bm\{x\}\)\-f^\{\\star\}≤⟨∇f​\(𝒙\),𝒙−𝒙⋆⟩−μ2​‖𝒙−𝒙⋆‖2\\displaystyle\\leq\\langle\\nabla f\(\\bm\{x\}\),\\bm\{x\}\-\\bm\{x\}^\{\\star\}\\rangle\-\\frac\{\\mu\}\{2\}\\\|\\bm\{x\}\-\\bm\{x\}^\{\\star\}\\\|^\{2\}≤12​μ​‖∇f​\(𝒙\)‖2\+μ2​‖𝒙−𝒙⋆‖2−μ2​‖𝒙−𝒙⋆‖2\\displaystyle\\leq\\frac\{1\}\{2\\mu\}\\\|\\nabla f\(\\bm\{x\}\)\\\|^\{2\}\+\\frac\{\\mu\}\{2\}\\\|\\bm\{x\}\-\\bm\{x\}^\{\\star\}\\\|^\{2\}\-\\frac\{\\mu\}\{2\}\\\|\\bm\{x\}\-\\bm\{x\}^\{\\star\}\\\|^\{2\}=12​μ​‖∇f​\(𝒙\)‖2,\\displaystyle=\\frac\{1\}\{2\\mu\}\\\|\\nabla f\(\\bm\{x\}\)\\\|^\{2\},where we have used Young’s inequality in the second inequality\. Therefore, we have

𝔼​\[f​\(𝒙t\+1\)\]≤𝔼​\[f​\(𝒙t\)\]−μ​ηt​𝔼​\[f​\(𝒙𝒕\)−f⋆\]\+E0​Lν\+1​ηtν\+1\.\\displaystyle\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\+1\}\)\\right\]\\leq\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\}\)\\right\]\-\\mu\\eta\_\{t\}\\mathbb\{E\}\\left\[f\(\\bm\{x\_\{t\}\}\)\-f^\{\\star\}\\right\]\+\\frac\{E\_\{0\}L\}\{\\nu\+1\}\\eta\_\{t\}^\{\\nu\+1\}\.Subtractingf⋆f^\{\\star\}from both sides yields

𝔼​\[f​\(𝒙t\+1\)\]−f⋆≤\(1−μ​ηt\)​𝔼​\[f​\(𝒙t\)−f⋆\]\+E0​Lν\+1​ηtν\+1\.\\displaystyle\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\+1\}\)\\right\]\-f^\{\\star\}\\leq\(1\-\\mu\\eta\_\{t\}\)\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\\right\]\+\\frac\{E\_\{0\}L\}\{\\nu\+1\}\\eta\_\{t\}^\{\\nu\+1\}\.Using this inequality, we prove by induction that

𝔼​\[f​\(𝒙t\)−f⋆\]≤E1tν,where​E1:=max⁡\{f​\(𝒙1\)−f⋆,E0​Lν​\(ν\+1\)​\(2​νμ\)ν\+1\}\.\\displaystyle\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\\right\]\\leq\\frac\{E\_\{1\}\}\{t^\{\\nu\}\},\\text\{ where \}\\ E\_\{1\}:=\\max\\left\\\{f\(\\bm\{x\}\_\{1\}\)\-f^\{\\star\},\\frac\{E\_\{0\}L\}\{\\nu\(\\nu\+1\)\}\\left\(\\frac\{2\\nu\}\{\\mu\}\\right\)^\{\\nu\+1\}\\right\\\}\.Whent=1t=1, from the definition ofE1E\_\{1\}, we have𝔼​\[f​\(𝒙1\)−f⋆\]≤E1\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{1\}\)\-f^\{\\star\}\\right\]\\leq E\_\{1\}\. Assume that𝔼​\[f​\(𝒙t\)−f⋆\]≤E1tν\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\\right\]\\leq\\frac\{E\_\{1\}\}\{t^\{\\nu\}\}holds for somet∈ℕt\\in\\mathbb\{N\}\. Then, fromηt:=2​νμ​t\\eta\_\{t\}:=\\frac\{2\\nu\}\{\\mu t\}, we have

𝔼​\[f​\(𝒙t\+1\)−f⋆\]\\displaystyle\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\+1\}\)\-f^\{\\star\}\\right\]≤\(1−μ​ηt\)​E1tν\+E0​Lν\+1​ηtν\+1\\displaystyle\\leq\(1\-\\mu\\eta\_\{t\}\)\\frac\{E\_\{1\}\}\{t^\{\\nu\}\}\+\\frac\{E\_\{0\}L\}\{\\nu\+1\}\\eta\_\{t\}^\{\\nu\+1\}=\(1−2​νt\)​E1tν\+E0​Lν\+1​\(2​νμ\)ν\+1​1tν\+1\.\\displaystyle=\\left\(1\-\\frac\{2\\nu\}\{t\}\\right\)\\frac\{E\_\{1\}\}\{t^\{\\nu\}\}\+\\frac\{E\_\{0\}L\}\{\\nu\+1\}\\left\(\\frac\{2\\nu\}\{\\mu\}\\right\)^\{\\nu\+1\}\\frac\{1\}\{t^\{\\nu\+1\}\}\.From the definition ofE1E\_\{1\}, we haveE0​Lν\+1​\(2​νμ\)ν\+1≤ν​E1\\frac\{E\_\{0\}L\}\{\\nu\+1\}\\left\(\\frac\{2\\nu\}\{\\mu\}\\right\)^\{\\nu\+1\}\\leq\\nu E\_\{1\}\. Then,

𝔼​\[f​\(𝒙t\+1\)−f⋆\]\\displaystyle\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\+1\}\)\-f^\{\\star\}\\right\]≤E1tν−2​ν​E1tν\+1\+ν​E1tν\+1\\displaystyle\\leq\\frac\{E\_\{1\}\}\{t^\{\\nu\}\}\-\\frac\{2\\nu E\_\{1\}\}\{t^\{\\nu\+1\}\}\+\\frac\{\\nu E\_\{1\}\}\{t^\{\\nu\+1\}\}=E1​\(1tν−νtν\+1\)\.\\displaystyle=E\_\{1\}\\left\(\\frac\{1\}\{t^\{\\nu\}\}\-\\frac\{\\nu\}\{t^\{\\nu\+1\}\}\\right\)\.Here, since the functiong​\(t\)=t−νg\(t\)=t^\{\-\\nu\}\(ν∈\(0,1\]\)\(\\nu\\in\(0,1\]\)is convex, we haveg​\(t\+1\)≥g​\(t\)\+g′​\(t\)g\(t\+1\)\\geq g\(t\)\+g^\{\\prime\}\(t\), i\.e\.,1\(t\+1\)ν≥1tν−νtν\+1\\frac\{1\}\{\(t\+1\)^\{\\nu\}\}\\geq\\frac\{1\}\{t^\{\\nu\}\}\-\\frac\{\\nu\}\{t^\{\\nu\+1\}\}\. Hence,

𝔼​\[f​\(𝒙t\+1\)−f⋆\]≤E1\(t\+1\)ν\.\\displaystyle\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\+1\}\)\-f^\{\\star\}\\right\]\\leq\\frac\{E\_\{1\}\}\{\(t\+1\)^\{\\nu\}\}\.Therefore, for allt∈ℕt\\in\\mathbb\{N\},𝔼​\[f​\(𝒙t\)−f⋆\]≤E1tν\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\\right\]\\leq\\frac\{E\_\{1\}\}\{t^\{\\nu\}\}holds\. This completes the proof\. ∎

### B\.3Proof of Theorem[3\.2](https://arxiv.org/html/2607.08104#S3.Thmthm2)

###### Lemma B\.1\.

Suppose that Assumptions[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1)and[2\.2](https://arxiv.org/html/2607.08104#S2.Thmassum2)hold andffisμ\\mu\-strongly convex\. LetΦt:=f​\(𝐳t\)−f⋆\+A​‖𝐦t−1‖ν\+1\\Phi\_\{t\}:=f\(\\bm\{z\}\_\{t\}\)\-f^\{\\star\}\+A\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\. Then, the following inequality holds:

‖∇f​\(𝒙t\)‖2≥E2​Φt−E2​A​‖𝒎t−1‖ν\+1−\(1\+2​L\)​E22​\(ν\+1\)​\(η​β1−β\)ν\+1​‖𝒎t−1‖ν\+1,\\displaystyle\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\geq E\_\{2\}\\Phi\_\{t\}\-E\_\{2\}A\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\-\\frac\{\(1\+2L\)E\_\{2\}\}\{2\(\\nu\+1\)\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{\\nu\+1\}\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\},whereE2:=2​μ​\(ν\+1\)ν​\(1\+L1ν\)\+1E\_\{2\}:=\\frac\{2\\mu\(\\nu\+1\)\}\{\\nu\\left\(1\+L^\{\\frac\{1\}\{\\nu\}\}\\right\)\+1\}is a non\-negative constant\.

###### Proof\.

From Lemma[2\.1](https://arxiv.org/html/2607.08104#S2.Thmlem1), we have

f​\(𝒛t\)≤f​\(𝒙t\)\+⟨∇f​\(𝒙t\),𝒛t−𝒙t⟩\+Lν\+1​‖𝒛t−𝒙t‖ν\+1\.\\displaystyle f\(\\bm\{z\}\_\{t\}\)\\leq f\(\\bm\{x\}\_\{t\}\)\+\\langle\\nabla f\(\\bm\{x\}\_\{t\}\),\\bm\{z\}\_\{t\}\-\\bm\{x\}\_\{t\}\\rangle\+\\frac\{L\}\{\\nu\+1\}\\\|\\bm\{z\}\_\{t\}\-\\bm\{x\}\_\{t\}\\\|^\{\\nu\+1\}\.Rearranging the terms and subtractingf⋆f^\{\\star\}from both sides gives

f​\(𝒙t\)−f⋆≥f​\(𝒛t\)−f⋆−⟨∇f​\(𝒙t\),𝒛t−𝒙t⟩−Lν\+1​‖𝒛t−𝒙t‖ν\+1\.\\displaystyle f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\\geq f\(\\bm\{z\}\_\{t\}\)\-f^\{\\star\}\-\\langle\\nabla f\(\\bm\{x\}\_\{t\}\),\\bm\{z\}\_\{t\}\-\\bm\{x\}\_\{t\}\\rangle\-\\frac\{L\}\{\\nu\+1\}\\\|\\bm\{z\}\_\{t\}\-\\bm\{x\}\_\{t\}\\\|^\{\\nu\+1\}\.Furthermore, multiplying both sides by2​μ2\\muyields

2​μ​\(f​\(𝒙t\)−f⋆\)≥2​μ​\(f​\(𝒛t\)−f⋆\)−2​μ​⟨∇f​\(𝒙t\),𝒛t−𝒙t⟩−2​μ​Lν\+1​‖𝒛t−𝒙t‖ν\+1\.\\displaystyle 2\\mu\(f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\)\\geq 2\\mu\(f\(\\bm\{z\}\_\{t\}\)\-f^\{\\star\}\)\-2\\mu\\langle\\nabla f\(\\bm\{x\}\_\{t\}\),\\bm\{z\}\_\{t\}\-\\bm\{x\}\_\{t\}\\rangle\-\\frac\{2\\mu L\}\{\\nu\+1\}\\\|\\bm\{z\}\_\{t\}\-\\bm\{x\}\_\{t\}\\\|^\{\\nu\+1\}\.Here, from theμ\\mu\-strong convexity offfand Assumption[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1), we have, for all𝒙,𝒚∈ℝd\\bm\{x\},\\bm\{y\}\\in\\mathbb\{R\}^\{d\},

μ​‖𝒙−𝒚‖2≤⟨∇f​\(𝒙\)−∇f​\(𝒚\),𝒙−𝒚⟩≤‖∇f​\(𝒙\)−∇f​\(𝒚\)‖​‖𝒙−𝒚‖≤L​‖𝒙−𝒚‖ν\+1\.\\displaystyle\\mu\\\|\\bm\{x\}\-\\bm\{y\}\\\|^\{2\}\\leq\\langle\\nabla f\(\\bm\{x\}\)\-\\nabla f\(\\bm\{y\}\),\\bm\{x\}\-\\bm\{y\}\\rangle\\leq\\\|\\nabla f\(\\bm\{x\}\)\-\\nabla f\(\\bm\{y\}\)\\\|\\\|\\bm\{x\}\-\\bm\{y\}\\\|\\leq L\\\|\\bm\{x\}\-\\bm\{y\}\\\|^\{\\nu\+1\}\.That is, we have‖𝒙−𝒚‖1−ν≤Lμ\\\|\\bm\{x\}\-\\bm\{y\}\\\|^\{1\-\\nu\}\\leq\\frac\{L\}\{\\mu\}for allν∈\(0,1\)\\nu\\in\(0,1\)\. Therefore, from Assumption[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1),

‖∇f​\(𝒙t\)‖ν\+1ν=‖∇f​\(𝒙t\)‖2​‖∇f​\(𝒙t\)‖1−νν≤‖∇f​\(𝒙t\)‖2​L1−νν​‖𝒙t−𝒙⋆‖1−ν≤L1νμ​‖∇f​\(𝒙t\)‖2\.\\displaystyle\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{\\frac\{\\nu\+1\}\{\\nu\}\}=\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{\\frac\{1\-\\nu\}\{\\nu\}\}\\leq\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}L^\{\\frac\{1\-\\nu\}\{\\nu\}\}\\\|\\bm\{x\}\_\{t\}\-\\bm\{x\}^\{\\star\}\\\|^\{1\-\\nu\}\\leq\\frac\{L^\{\\frac\{1\}\{\\nu\}\}\}\{\\mu\}\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\.Note that we arrive at the same result whenν=1\\nu=1, fromμ≤L\\mu\\leq L\. Hence, from the Cauchy\-Schwarz and Young’s inequality,

−2​μ​⟨∇f​\(𝒙t\),𝒛t−𝒙t⟩\\displaystyle\-2\\mu\\langle\\nabla f\(\\bm\{x\}\_\{t\}\),\\bm\{z\}\_\{t\}\-\\bm\{x\}\_\{t\}\\rangle≥−2​μ​‖∇f​\(𝒙t\)‖​‖𝒛t−𝒙t‖≥−μ​\(νν\+1​‖∇f​\(𝒙t\)‖ν\+1ν\+1ν\+1​‖𝒛t−𝒙t‖ν\+1\)\\displaystyle\\geq\-2\\mu\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|\\\|\\bm\{z\}\_\{t\}\-\\bm\{x\}\_\{t\}\\\|\\geq\-\\mu\\left\(\\frac\{\\nu\}\{\\nu\+1\}\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{\\frac\{\\nu\+1\}\{\\nu\}\}\+\\frac\{1\}\{\\nu\+1\}\\\|\\bm\{z\}\_\{t\}\-\\bm\{x\}\_\{t\}\\\|^\{\\nu\+1\}\\right\)≥−ν​L1νν\+1​‖∇f​\(𝒙t\)‖2−μν\+1​‖𝒛t−𝒙t‖ν\+1\.\\displaystyle\\geq\-\\frac\{\\nu L^\{\\frac\{1\}\{\\nu\}\}\}\{\\nu\+1\}\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\-\\frac\{\\mu\}\{\\nu\+1\}\\\|\\bm\{z\}\_\{t\}\-\\bm\{x\}\_\{t\}\\\|^\{\\nu\+1\}\.Substituting this into the above inequality, we find that

2​μ​\(f​\(𝒙t\)−f⋆\)≥2​μ​\(f​\(𝒛t\)−f⋆\)−ν​L1νν\+1​‖∇f​\(𝒙t\)‖2−μ​\(1\+2​L\)ν\+1​‖𝒛t−𝒙t‖ν\+1\.\\displaystyle 2\\mu\(f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\)\\geq 2\\mu\(f\(\\bm\{z\}\_\{t\}\)\-f^\{\\star\}\)\-\\frac\{\\nu L^\{\\frac\{1\}\{\\nu\}\}\}\{\\nu\+1\}\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\-\\frac\{\\mu\(1\+2L\)\}\{\\nu\+1\}\\\|\\bm\{z\}\_\{t\}\-\\bm\{x\}\_\{t\}\\\|^\{\\nu\+1\}\.Sinceffisμ\\mu\-strongly convex, the inequality‖∇f​\(𝒙t\)‖2≥2​μ​\(f​\(𝒙t\)−f⋆\)\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\geq 2\\mu\(f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\)holds\. Then, we have

‖∇f​\(𝒙t\)‖2≥2​μ​\(f​\(𝒛t\)−f⋆\)−ν​L1νν\+1​‖∇f​\(𝒙t\)‖2−μ​\(1\+2​L\)ν\+1​‖𝒛t−𝒙t‖ν\+1\.\\displaystyle\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\geq 2\\mu\(f\(\\bm\{z\}\_\{t\}\)\-f^\{\\star\}\)\-\\frac\{\\nu L^\{\\frac\{1\}\{\\nu\}\}\}\{\\nu\+1\}\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\-\\frac\{\\mu\(1\+2L\)\}\{\\nu\+1\}\\\|\\bm\{z\}\_\{t\}\-\\bm\{x\}\_\{t\}\\\|^\{\\nu\+1\}\.Isolating‖∇f​\(𝒙t\)‖2\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}and organizing the terms lead to

‖∇f​\(𝒙t\)‖2≥2​μ​\(ν\+1\)ν​\(1\+L1ν\)\+1​\(f​\(𝒛t\)−f⋆\)−μ​\(1\+2​L\)ν​\(1\+L1ν\)\+1​‖𝒛t−𝒙t‖ν\+1\.\\displaystyle\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\geq\\frac\{2\\mu\(\\nu\+1\)\}\{\\nu\\left\(1\+L^\{\\frac\{1\}\{\\nu\}\}\\right\)\+1\}\(f\(\\bm\{z\}\_\{t\}\)\-f^\{\\star\}\)\-\\frac\{\\mu\(1\+2L\)\}\{\\nu\\left\(1\+L^\{\\frac\{1\}\{\\nu\}\}\\right\)\+1\}\\\|\\bm\{z\}\_\{t\}\-\\bm\{x\}\_\{t\}\\\|^\{\\nu\+1\}\.From the definition ofΦt\\Phi\_\{t\}and𝒛t−𝒙t=−η​β1−β​𝒎t−1\\bm\{z\}\_\{t\}\-\\bm\{x\}\_\{t\}=\-\\frac\{\\eta\\beta\}\{1\-\\beta\}\\bm\{m\}\_\{t\-1\},

‖∇f​\(𝒙t\)‖2\\displaystyle\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}≥2​μ​\(ν\+1\)ν​\(1\+L1ν\)\+1⏟=⁣:E2​Φt−2​μ​\(ν\+1\)​Aν​\(1\+L1ν\)\+1​‖𝒎t−1‖ν\+1−μ​\(1\+2​L\)ν​\(1\+L1ν\)\+1​\(η​β1−β\)ν\+1​‖𝒎t−1‖ν\+1\\displaystyle\\geq\\underbrace\{\\frac\{2\\mu\(\\nu\+1\)\}\{\\nu\\left\(1\+L^\{\\frac\{1\}\{\\nu\}\}\\right\)\+1\}\}\_\{=:E\_\{2\}\}\\Phi\_\{t\}\-\\frac\{2\\mu\(\\nu\+1\)A\}\{\\nu\\left\(1\+L^\{\\frac\{1\}\{\\nu\}\}\\right\)\+1\}\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\-\\frac\{\\mu\(1\+2L\)\}\{\\nu\\left\(1\+L^\{\\frac\{1\}\{\\nu\}\}\\right\)\+1\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{\\nu\+1\}\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}≥E2​Φt−E2​A​‖𝒎t−1‖ν\+1−\(1\+2​L\)​E22​\(ν\+1\)​\(η​β1−β\)ν\+1​‖𝒎t−1‖ν\+1\.\\displaystyle\\geq E\_\{2\}\\Phi\_\{t\}\-E\_\{2\}A\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\-\\frac\{\(1\+2L\)E\_\{2\}\}\{2\(\\nu\+1\)\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{\\nu\+1\}\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\.This completes the proof\. ∎

###### Lemma B\.2\.

Suppose that Assumption[2\.2](https://arxiv.org/html/2607.08104#S2.Thmassum2)holds\. Then,

𝔼​\[‖𝒎t‖ν\+1\]−𝔼​\[‖𝒎t−1‖ν\+1\]\\displaystyle\\mathbb\{E\}\\left\[\\\|\\bm\{m\}\_\{t\}\\\|^\{\\nu\+1\}\\right\]\-\\mathbb\{E\}\\left\[\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\\right\]≤−\(1−β\)​𝔼​\[‖𝒎t−1‖ν\+1\]\+\(1−β\)​\(ν\+1\)2​𝔼​\[‖∇f​\(𝒙t\)‖2\]\+\(1−β\)​\(1−ν\)2\+21−ν​\(1−β\)ν\+1​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\.\\displaystyle\\quad\\leq\-\(1\-\\beta\)\\mathbb\{E\}\\left\[\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\\right\]\+\\frac\{\(1\-\\beta\)\(\\nu\+1\)\}\{2\}\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]\+\\frac\{\(1\-\\beta\)\(1\-\\nu\)\}\{2\}\+2^\{1\-\\nu\}\(1\-\\beta\)^\{\\nu\+1\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\.

###### Proof\.

From the definition of𝒎t\\bm\{m\}\_\{t\}and Lemma[2\.2](https://arxiv.org/html/2607.08104#S2.Thmlem2), we have

‖𝒎t‖ν\+1\\displaystyle\\\|\\bm\{m\}\_\{t\}\\\|^\{\\nu\+1\}=‖β​𝒎t−1\+\(1−β\)​∇f𝒮t​\(𝒙t\)‖ν\+1\\displaystyle=\\\|\\beta\\bm\{m\}\_\{t\-1\}\+\(1\-\\beta\)\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\\|^\{\\nu\+1\}=‖\(β​𝒎t−1\+\(1−β\)​∇f​\(𝒙t\)\)\+\(1−β\)​\(∇f𝒮t​\(𝒙t\)−∇f​\(𝒙t\)\)‖ν\+1\\displaystyle=\\\|\\left\(\\beta\\bm\{m\}\_\{t\-1\}\+\(1\-\\beta\)\\nabla f\(\\bm\{x\}\_\{t\}\)\\right\)\+\(1\-\\beta\)\(\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\-\\nabla f\(\\bm\{x\}\_\{t\}\)\)\\\|^\{\\nu\+1\}=‖β​𝒎t−1\+\(1−β\)​∇f​\(𝒙t\)‖ν\+1\+21−ν​\(1−β\)ν\+1​‖∇f𝒮t​\(𝒙t\)−∇f​\(𝒙t\)‖ν\+1\\displaystyle=\\\|\\beta\\bm\{m\}\_\{t\-1\}\+\(1\-\\beta\)\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{\\nu\+1\}\+2^\{1\-\\nu\}\(1\-\\beta\)^\{\\nu\+1\}\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\-\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{\\nu\+1\}\+\(ν\+1\)​\(1−β\)​‖β​𝒎t−1\+\(1−β\)​∇f​\(𝒙t\)‖ν−1​⟨β​𝒎t−1\+\(1−β\)​∇f​\(𝒙t\),∇f𝒮t​\(𝒙t\)−∇f​\(𝒙t\)⟩\.\\displaystyle\\quad\+\(\\nu\+1\)\(1\-\\beta\)\\\|\\beta\\bm\{m\}\_\{t\-1\}\+\(1\-\\beta\)\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{\\nu\-1\}\\langle\\beta\\bm\{m\}\_\{t\-1\}\+\(1\-\\beta\)\\nabla f\(\\bm\{x\}\_\{t\}\),\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\-\\nabla f\(\\bm\{x\}\_\{t\}\)\\rangle\.From Assumption[2\.2](https://arxiv.org/html/2607.08104#S2.Thmassum2), we find, by taking the expectation, that

𝔼​\[‖𝒎t‖ν\+1\]\\displaystyle\\mathbb\{E\}\\left\[\\\|\\bm\{m\}\_\{t\}\\\|^\{\\nu\+1\}\\right\]=𝔼​\[‖β​𝒎t−1\+\(1−β\)​∇f​\(𝒙t\)‖ν\+1\]\+21−ν​\(1−β\)ν\+1​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\\displaystyle=\\mathbb\{E\}\\left\[\\\|\\beta\\bm\{m\}\_\{t\-1\}\+\(1\-\\beta\)\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{\\nu\+1\}\\right\]\+2^\{1\-\\nu\}\(1\-\\beta\)^\{\\nu\+1\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}≤β​𝔼​\[‖𝒎t−1‖ν\+1\]\+\(1−β\)​𝔼​\[‖∇f​\(𝒙t\)‖ν\+1\]\+21−ν​\(1−β\)ν\+1​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\.\\displaystyle\\leq\\beta\\mathbb\{E\}\\left\[\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\\right\]\+\(1\-\\beta\)\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{\\nu\+1\}\\right\]\+2^\{1\-\\nu\}\(1\-\\beta\)^\{\\nu\+1\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\.Subtracting𝔼​\[‖𝒎t−1‖ν\+1\]\\mathbb\{E\}\\left\[\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\\right\]from both sides gives

𝔼​\[‖𝒎t‖ν\+1\]−𝔼​\[‖𝒎t−1‖ν\+1\]\\displaystyle\\mathbb\{E\}\\left\[\\\|\\bm\{m\}\_\{t\}\\\|^\{\\nu\+1\}\\right\]\-\\mathbb\{E\}\\left\[\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\\right\]≤−\(1−β\)​𝔼​\[‖𝒎t−1‖ν\+1\]\+\(1−β\)​𝔼​\[‖∇f​\(𝒙t\)‖ν\+1\]\+21−ν​\(1−β\)ν\+1​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\.\\displaystyle\\quad\\leq\-\(1\-\\beta\)\\mathbb\{E\}\\left\[\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\\right\]\+\(1\-\\beta\)\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{\\nu\+1\}\\right\]\+2^\{1\-\\nu\}\(1\-\\beta\)^\{\\nu\+1\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\.From Young’s inequality, we have

‖∇f​\(𝒙t\)‖ν\+1⋅1≤ν\+12​‖∇f​\(𝒙t\)‖2\+1−ν2\.\\displaystyle\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{\\nu\+1\}\\cdot 1\\leq\\frac\{\\nu\+1\}\{2\}\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\+\\frac\{1\-\\nu\}\{2\}\.Therefore, we have

𝔼​\[‖𝒎t‖ν\+1\]−𝔼​\[‖𝒎t−1‖ν\+1\]\\displaystyle\\mathbb\{E\}\\left\[\\\|\\bm\{m\}\_\{t\}\\\|^\{\\nu\+1\}\\right\]\-\\mathbb\{E\}\\left\[\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\\right\]≤−\(1−β\)​𝔼​\[‖𝒎t−1‖ν\+1\]\+\(1−β\)​\(ν\+1\)2​𝔼​\[‖∇f​\(𝒙t\)‖2\]\+\(1−β\)​\(1−ν\)2\+21−ν​\(1−β\)ν\+1​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\.\\displaystyle\\quad\\leq\-\(1\-\\beta\)\\mathbb\{E\}\\left\[\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\\right\]\+\\frac\{\(1\-\\beta\)\(\\nu\+1\)\}\{2\}\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]\+\\frac\{\(1\-\\beta\)\(1\-\\nu\)\}\{2\}\+2^\{1\-\\nu\}\(1\-\\beta\)^\{\\nu\+1\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\.This completes the proof\. ∎

###### Lemma B\.3\.

Let𝐳t:=𝐱t−β​η1−β​𝐦t−1\\bm\{z\}\_\{t\}:=\\bm\{x\}\_\{t\}\-\\frac\{\\beta\\eta\}\{1\-\\beta\}\\bm\{m\}\_\{t\-1\}\. Suppose that Assumptions[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1)and[2\.2](https://arxiv.org/html/2607.08104#S2.Thmassum2)hold\. Then,

𝔼​\[f​\(𝒛t\+1\)\]\\displaystyle\\mathbb\{E\}\[f\(\\bm\{z\}\_\{t\+1\}\)\]≤𝔼​\[f​\(𝒛t\)\]−\(η2−L​ην\+12\)​𝔼​\[‖∇f​\(𝒙t\)‖2\]\+η​L2​νν\+1​\(η​β1−β\)2​ν​𝔼​\[‖𝒎t−1‖ν\+1\]\\displaystyle\\leq\\mathbb\{E\}\[f\(\\bm\{z\}\_\{t\}\)\]\-\\left\(\\frac\{\\eta\}\{2\}\-\\frac\{L\\eta^\{\\nu\+1\}\}\{2\}\\right\)\\mathbb\{E\}\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\]\+\\frac\{\\eta L^\{2\}\\nu\}\{\\nu\+1\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\\mathbb\{E\}\\left\[\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\\right\]\+η​\(1−ν\)​L22​\(ν\+1\)​\(η​β1−β\)2​ν\+L​ην\+1ν\+1​\{21−ν​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+1−ν2\}\.\\displaystyle\\quad\+\\frac\{\\eta\(1\-\\nu\)L^\{2\}\}\{2\(\\nu\+1\)\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\+\\frac\{L\\eta^\{\\nu\+1\}\}\{\\nu\+1\}\\left\\\{2^\{1\-\\nu\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+\\frac\{1\-\\nu\}\{2\}\\right\\\}\.

###### Proof\.

From the definition of𝒛t\\bm\{z\}\_\{t\}, we have

𝒛t\+1\\displaystyle\\bm\{z\}\_\{t\+1\}=𝒙t\+1−η​β1−β​𝒎t=\(𝒙t−η​𝒎t\)−η​β1−β​𝒎t=𝒙t−η1−β​𝒎t\\displaystyle=\\bm\{x\}\_\{t\+1\}\-\\frac\{\\eta\\beta\}\{1\-\\beta\}\\bm\{m\}\_\{t\}=\(\\bm\{x\}\_\{t\}\-\\eta\\bm\{m\}\_\{t\}\)\-\\frac\{\\eta\\beta\}\{1\-\\beta\}\\bm\{m\}\_\{t\}=\\bm\{x\}\_\{t\}\-\\frac\{\\eta\}\{1\-\\beta\}\\bm\{m\}\_\{t\}=𝒙t−η1−β​\{β​𝒎t−1\+\(1−β\)​∇f𝒮t​\(𝒙t\)\}=𝒛t−η​∇f𝒮t​\(𝒙t\)\.\\displaystyle=\\bm\{x\}\_\{t\}\-\\frac\{\\eta\}\{1\-\\beta\}\\left\\\{\\beta\\bm\{m\}\_\{t\-1\}\+\(1\-\\beta\)\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\right\\\}=\\bm\{z\}\_\{t\}\-\\eta\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\.From Assumption[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1),

f​\(𝒛t\+1\)\\displaystyle f\(\\bm\{z\}\_\{t\+1\}\)≤f​\(𝒛t\)\+⟨∇f​\(𝒛t\),𝒛t\+1−𝒛t⟩\+Lν\+1​‖𝒛t\+1−𝒛t‖ν\+1\\displaystyle\\leq f\(\\bm\{z\}\_\{t\}\)\+\\langle\\nabla f\(\\bm\{z\}\_\{t\}\),\\bm\{z\}\_\{t\+1\}\-\\bm\{z\}\_\{t\}\\rangle\+\\frac\{L\}\{\\nu\+1\}\\\|\\bm\{z\}\_\{t\+1\}\-\\bm\{z\}\_\{t\}\\\|^\{\\nu\+1\}=f​\(𝒛t\)−η​⟨∇f​\(𝒛t\),∇f𝒮t​\(𝒙t\)⟩\+L​ην\+1ν\+1​‖∇f𝒮t​\(𝒙t\)‖ν\+1\.\\displaystyle=f\(\\bm\{z\}\_\{t\}\)\-\\eta\\langle\\nabla f\(\\bm\{z\}\_\{t\}\),\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\rangle\+\\frac\{L\\eta^\{\\nu\+1\}\}\{\\nu\+1\}\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\\|^\{\\nu\+1\}\.By taking the expectation, we have

𝔼​\[f​\(𝒛t\+1\)\]≤𝔼​\[f​\(𝒛t\)\]−η​𝔼​\[⟨∇f​\(𝒛t\),∇f​\(𝒙t\)⟩\]\+L​ην\+1ν\+1​𝔼​\[‖∇f𝒮t​\(𝒙t\)‖ν\+1\]\.\\displaystyle\\mathbb\{E\}\[f\(\\bm\{z\}\_\{t\+1\}\)\]\\leq\\mathbb\{E\}\[f\(\\bm\{z\}\_\{t\}\)\]\-\\eta\\mathbb\{E\}\[\\langle\\nabla f\(\\bm\{z\}\_\{t\}\),\\nabla f\(\\bm\{x\}\_\{t\}\)\\rangle\]\+\\frac\{L\\eta^\{\\nu\+1\}\}\{\\nu\+1\}\\mathbb\{E\}\[\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\\|^\{\\nu\+1\}\]\.From‖𝒙−𝒚‖2=‖𝒙‖2−2​⟨𝒙,𝒚⟩\+‖𝒚‖2\\\|\\bm\{x\}\-\\bm\{y\}\\\|^\{2\}=\\\|\\bm\{x\}\\\|^\{2\}\-2\\langle\\bm\{x\},\\bm\{y\}\\rangle\+\\\|\\bm\{y\}\\\|^\{2\}and Lemma[3\.1](https://arxiv.org/html/2607.08104#S3.Thmlem1),

𝔼​\[f​\(𝒛t\+1\)\]\\displaystyle\\mathbb\{E\}\[f\(\\bm\{z\}\_\{t\+1\}\)\]≤𝔼​\[f​\(𝒛t\)\]−η2​\(𝔼​\[‖∇f​\(𝒛t\)‖2\]\+𝔼​\[‖∇f​\(𝒙t\)‖2\]−𝔼​\[‖∇f​\(𝒛t\)−∇f​\(𝒙t\)‖2\]\)\\displaystyle\\leq\\mathbb\{E\}\[f\(\\bm\{z\}\_\{t\}\)\]\-\\frac\{\\eta\}\{2\}\(\\mathbb\{E\}\[\\\|\\nabla f\(\\bm\{z\}\_\{t\}\)\\\|^\{2\}\]\+\\mathbb\{E\}\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\]\-\\mathbb\{E\}\[\\\|\\nabla f\(\\bm\{z\}\_\{t\}\)\-\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\]\)\+L​ην\+1ν\+1​\(21−ν​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+ν\+12​𝔼​\[‖∇f​\(𝒙t\)‖2\]\+1−ν2\)\\displaystyle\\quad\+\\frac\{L\\eta^\{\\nu\+1\}\}\{\\nu\+1\}\\left\(2^\{1\-\\nu\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+\\frac\{\\nu\+1\}\{2\}\\mathbb\{E\}\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\]\+\\frac\{1\-\\nu\}\{2\}\\right\)≤𝔼​\[f​\(𝒛t\)\]−\(η2−L​ην\+12\)​𝔼​\[‖∇f​\(𝒙t\)‖2\]\+η2​𝔼​\[‖∇f​\(𝒛t\)−∇f​\(𝒙t\)‖2\]\\displaystyle\\leq\\mathbb\{E\}\[f\(\\bm\{z\}\_\{t\}\)\]\-\\left\(\\frac\{\\eta\}\{2\}\-\\frac\{L\\eta^\{\\nu\+1\}\}\{2\}\\right\)\\mathbb\{E\}\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\]\+\\frac\{\\eta\}\{2\}\\mathbb\{E\}\[\\\|\\nabla f\(\\bm\{z\}\_\{t\}\)\-\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\]\+L​ην\+1ν\+1​\{21−ν​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+1−ν2\}\.\\displaystyle\\quad\+\\frac\{L\\eta^\{\\nu\+1\}\}\{\\nu\+1\}\\left\\\{2^\{1\-\\nu\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+\\frac\{1\-\\nu\}\{2\}\\right\\\}\.Here, from Assumption[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1), the definition of𝒛t\\bm\{z\}\_\{t\}, and Young’s inequality, we have

η2​𝔼​\[‖∇f​\(𝒛t\)−∇f​\(𝒙t\)‖2\]\\displaystyle\\frac\{\\eta\}\{2\}\\mathbb\{E\}\[\\\|\\nabla f\(\\bm\{z\}\_\{t\}\)\-\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\]≤η2​L2​𝔼​\[‖𝒛t−𝒙t‖2​ν\]\\displaystyle\\leq\\frac\{\\eta\}\{2\}L^\{2\}\\mathbb\{E\}\[\\\|\\bm\{z\}\_\{t\}\-\\bm\{x\}\_\{t\}\\\|^\{2\\nu\}\]=η2​L2​\(η​β1−β\)2​ν​𝔼​\[‖𝒎t−1‖2​ν\]\\displaystyle=\\frac\{\\eta\}\{2\}L^\{2\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\\mathbb\{E\}\\left\[\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{2\\nu\}\\right\]≤η2​L2​\(η​β1−β\)2​ν​\{2​νν\+1​𝔼​\[‖𝒎t−1‖ν\+1\]\+1−νν\+1\}\\displaystyle\\leq\\frac\{\\eta\}\{2\}L^\{2\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\\left\\\{\\frac\{2\\nu\}\{\\nu\+1\}\\mathbb\{E\}\\left\[\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\\right\]\+\\frac\{1\-\\nu\}\{\\nu\+1\}\\right\\\}=η​L2​νν\+1​\(η​β1−β\)2​ν​𝔼​\[‖𝒎t−1‖ν\+1\]\+η​\(1−ν\)​L22​\(ν\+1\)​\(η​β1−β\)2​ν\.\\displaystyle=\\frac\{\\eta L^\{2\}\\nu\}\{\\nu\+1\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\\mathbb\{E\}\\left\[\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\\right\]\+\\frac\{\\eta\(1\-\\nu\)L^\{2\}\}\{2\(\\nu\+1\)\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\.Therefore,

𝔼​\[f​\(𝒛t\+1\)\]\\displaystyle\\mathbb\{E\}\[f\(\\bm\{z\}\_\{t\+1\}\)\]≤𝔼​\[f​\(𝒛t\)\]−\(η2−L​ην\+12\)​𝔼​\[‖∇f​\(𝒙t\)‖2\]\+η​L2​νν\+1​\(η​β1−β\)2​ν​𝔼​\[‖𝒎t−1‖ν\+1\]\\displaystyle\\leq\\mathbb\{E\}\[f\(\\bm\{z\}\_\{t\}\)\]\-\\left\(\\frac\{\\eta\}\{2\}\-\\frac\{L\\eta^\{\\nu\+1\}\}\{2\}\\right\)\\mathbb\{E\}\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\]\+\\frac\{\\eta L^\{2\}\\nu\}\{\\nu\+1\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\\mathbb\{E\}\\left\[\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\\right\]\+η​\(1−ν\)​L22​\(ν\+1\)​\(η​β1−β\)2​ν\+L​ην\+1ν\+1​\{21−ν​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+1−ν2\}\.\\displaystyle\\quad\+\\frac\{\\eta\(1\-\\nu\)L^\{2\}\}\{2\(\\nu\+1\)\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\+\\frac\{L\\eta^\{\\nu\+1\}\}\{\\nu\+1\}\\left\\\{2^\{1\-\\nu\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+\\frac\{1\-\\nu\}\{2\}\\right\\\}\.This completes the proof\. ∎

###### Theorem[3\.2](https://arxiv.org/html/2607.08104#S3.Thmthm2)\(restated\)\.

Let the functionf:ℝd→ℝf\\colon\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}beμ\\mu\-strongly convex\. Suppose that Assumptions[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1)and[2\.2](https://arxiv.org/html/2607.08104#S2.Thmassum2)hold\. With learning rateηt:=η\\eta\_\{t\}:=\\eta, ifν\+1≤𝔭\\nu\+1\\leq\\mathfrak\{p\}andη≤min⁡\{2​\(1−β\)E2,\(18​L\)1ν,\(\(1−β\)ν\+12​E2​\(1\+2​L\)​βν\+1\)1ν\+1,\(\(1−β\)2​ν2​L2​ν​β2​ν\)12​ν\}\\eta\\leq\\min\\left\\\{\\frac\{2\(1\-\\beta\)\}\{E\_\{2\}\},\\left\(\\frac\{1\}\{8L\}\\right\)^\{\\frac\{1\}\{\\nu\}\},\\left\(\\frac\{\(1\-\\beta\)^\{\\nu\+1\}\}\{2E\_\{2\}\(1\+2L\)\\beta^\{\\nu\+1\}\}\\right\)^\{\\frac\{1\}\{\\nu\+1\}\},\\left\(\\frac\{\(1\-\\beta\)^\{2\\nu\}\}\{2L^\{2\}\\nu\\beta^\{2\\nu\}\}\\right\)^\{\\frac\{1\}\{2\\nu\}\}\\right\\\}, then:

𝔼​\[f​\(𝒙T\)−f⋆\]\\displaystyle\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{T\}\)\-f^\{\\star\}\\right\]≤f​\(𝒙1\)−f⋆2​\(1−η​E24\)T\+E31−β=𝒪​\(\(1−η​E24\)T\+ην\+1\)\.\\displaystyle\\leq\\frac\{f\(\\bm\{x\}\_\{1\}\)\-f^\{\\star\}\}\{2\}\\left\(1\-\\frac\{\\eta E\_\{2\}\}\{4\}\\right\)^\{T\}\+\\frac\{E\_\{3\}\}\{1\-\\beta\}=\\mathcal\{O\}\\left\(\\left\(1\-\\frac\{\\eta E\_\{2\}\}\{4\}\\right\)^\{T\}\+\\eta^\{\\nu\+1\}\\right\)\.In particular, whenν\+1=𝔭\\nu\+1=\\mathfrak\{p\},

𝔼​\[f​\(𝒙T\)−f⋆\]\\displaystyle\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{T\}\)\-f^\{\\star\}\\right\]=𝒪​\(\(1−η​E24\)T\+η𝔭\),\\displaystyle=\\mathcal\{O\}\\left\(\\left\(1\-\\frac\{\\eta E\_\{2\}\}\{4\}\\right\)^\{T\}\+\\eta^\{\\mathfrak\{p\}\}\\right\),whereE2:=2​μ​\(ν\+1\)ν​\(1\+L1ν\)\+1E\_\{2\}:=\\frac\{2\\mu\(\\nu\+1\)\}\{\\nu\\left\(1\+L^\{\\frac\{1\}\{\\nu\}\}\\right\)\+1\},E3:=L​E0ν\+1\+\(1\+2​L\)​E0​E24​\(ν\+1\)​\(β1−β\)ν\+1​η\+2​L2​ν​\(1−β\)​E0ν\+1​\(β1−β\)2​ν​ηνE\_\{3\}:=\\frac\{LE\_\{0\}\}\{\\nu\+1\}\+\\frac\{\(1\+2L\)E\_\{0\}E\_\{2\}\}\{4\(\\nu\+1\)\}\\left\(\\frac\{\\beta\}\{1\-\\beta\}\\right\)^\{\\nu\+1\}\\eta\+\\frac\{2L^\{2\}\\nu\(1\-\\beta\)E\_\{0\}\}\{\\nu\+1\}\\left\(\\frac\{\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\\eta^\{\\nu\}, andE0E\_\{0\}defined in Eq\. \([1](https://arxiv.org/html/2607.08104#S3.E1)\) are non\-negative constants\.

###### Proof\.

For allt∈ℕt\\in\\mathbb\{N\}, letΦt:=f​\(𝒛t\)−f⋆\+A​‖mt−1‖ν\+1\\Phi\_\{t\}:=f\(\\bm\{z\}\_\{t\}\)\-f^\{\\star\}\+A\\\|m\_\{t\-1\}\\\|^\{\\nu\+1\}\. Then,

𝔼​\[Φt\+1\]−𝔼​\[Φt\]=𝔼​\[f​\(zt\+1\)\]−𝔼​\[f​\(𝒛t\)\]\+A​\(𝔼​\[‖mt‖ν\+1\]−𝔼​\[‖mt−1‖ν\+1\]\)\.\\displaystyle\\mathbb\{E\}\[\\Phi\_\{t\+1\}\]\-\\mathbb\{E\}\[\\Phi\_\{t\}\]=\\mathbb\{E\}\[f\(z\_\{t\+1\}\)\]\-\\mathbb\{E\}\[f\(\\bm\{z\}\_\{t\}\)\]\+A\(\\mathbb\{E\}\[\\\|m\_\{t\}\\\|^\{\\nu\+1\}\]\-\\mathbb\{E\}\[\\\|m\_\{t\-1\}\\\|^\{\\nu\+1\}\]\)\.From Lemma[B\.3](https://arxiv.org/html/2607.08104#A2.Thmlem3), we have

𝔼​\[f​\(𝒛t\+1\)\]−𝔼​\[f​\(𝒛t\)\]\\displaystyle\\mathbb\{E\}\[f\(\\bm\{z\}\_\{t\+1\}\)\]\-\\mathbb\{E\}\[f\(\\bm\{z\}\_\{t\}\)\]≤−\(η2−L​ην\+12\)​𝔼​\[‖∇f​\(𝒙t\)‖2\]\+η​L2​νν\+1​\(η​β1−β\)2​ν​𝔼​\[‖𝒎t−1‖ν\+1\]\\displaystyle\\leq\-\\left\(\\frac\{\\eta\}\{2\}\-\\frac\{L\\eta^\{\\nu\+1\}\}\{2\}\\right\)\\mathbb\{E\}\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\]\+\\frac\{\\eta L^\{2\}\\nu\}\{\\nu\+1\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\\mathbb\{E\}\\left\[\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\\right\]\+η​\(1−ν\)​L22​\(ν\+1\)​\(η​β1−β\)2​ν\+L​ην\+1ν\+1​\{21−ν​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+1−ν2\}⏟=⁣:E0\.\\displaystyle\\quad\+\\frac\{\\eta\(1\-\\nu\)L^\{2\}\}\{2\(\\nu\+1\)\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\+\\frac\{L\\eta^\{\\nu\+1\}\}\{\\nu\+1\}\\underbrace\{\\left\\\{2^\{1\-\\nu\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+\\frac\{1\-\\nu\}\{2\}\\right\\\}\}\_\{=:E\_\{0\}\}\.Furthermore, from Lemma[B\.2](https://arxiv.org/html/2607.08104#A2.Thmlem2)and Young’s inequality,

A​\(𝔼​\[‖mt‖ν\+1\]−𝔼​\[‖mt−1‖ν\+1\]\)≤\\displaystyle A\(\\mathbb\{E\}\[\\\|m\_\{t\}\\\|^\{\\nu\+1\}\]\-\\mathbb\{E\}\[\\\|m\_\{t\-1\}\\\|^\{\\nu\+1\}\]\)\\leq−A​\(1−β\)​𝔼​\[‖mt−1‖ν\+1\]\+A​\(1−β\)​\(ν\+1\)2​𝔼​\[‖∇f​\(𝒙t\)‖2\]\\displaystyle\-A\(1\-\\beta\)\\mathbb\{E\}\[\\\|m\_\{t\-1\}\\\|^\{\\nu\+1\}\]\+\\frac\{A\(1\-\\beta\)\(\\nu\+1\)\}\{2\}\\mathbb\{E\}\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\]\+A​\(1−β\)​\(1−ν\)2\+21−ν​\(1−β\)ν\+1​A​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\.\\displaystyle\+\\frac\{A\(1\-\\beta\)\(1\-\\nu\)\}\{2\}\+2^\{1\-\\nu\}\(1\-\\beta\)^\{\\nu\+1\}A\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\.Combining these results,

𝔼​\[Φt\+1\]−𝔼​\[Φt\]\\displaystyle\\mathbb\{E\}\[\\Phi\_\{t\+1\}\]\-\\mathbb\{E\}\[\\Phi\_\{t\}\]≤−\(η2−L​ην\+12\)​𝔼​\[‖∇f​\(𝒙t\)‖2\]\+η​L2​νν\+1​\(η​β1−β\)2​ν​𝔼​\[‖𝒎t−1‖ν\+1\]\\displaystyle\\leq\-\\left\(\\frac\{\\eta\}\{2\}\-\\frac\{L\\eta^\{\\nu\+1\}\}\{2\}\\right\)\\mathbb\{E\}\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\]\+\\frac\{\\eta L^\{2\}\\nu\}\{\\nu\+1\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\\mathbb\{E\}\\left\[\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\\right\]\+η​\(1−ν\)​L22​\(ν\+1\)​\(η​β1−β\)2​ν\+L​ην\+1ν\+1​E0\\displaystyle\\quad\+\\frac\{\\eta\(1\-\\nu\)L^\{2\}\}\{2\(\\nu\+1\)\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\+\\frac\{L\\eta^\{\\nu\+1\}\}\{\\nu\+1\}E\_\{0\}−A​\(1−β\)​𝔼​\[‖mt−1‖ν\+1\]\+A​\(1−β\)​\(ν\+1\)2​𝔼​\[‖∇f​\(𝒙t\)‖2\]\\displaystyle\\quad\-A\(1\-\\beta\)\\mathbb\{E\}\[\\\|m\_\{t\-1\}\\\|^\{\\nu\+1\}\]\+\\frac\{A\(1\-\\beta\)\(\\nu\+1\)\}\{2\}\\mathbb\{E\}\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\]\+A​\(1−β\)​\(1−ν\)2\+21−ν​\(1−β\)ν\+1​A​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\.\\displaystyle\\quad\+\\frac\{A\(1\-\\beta\)\(1\-\\nu\)\}\{2\}\+2^\{1\-\\nu\}\(1\-\\beta\)^\{\\nu\+1\}A\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\.Next, we split the gradient term−η2​𝔼​\[‖∇f​\(𝒙t\)‖2\]\-\\frac\{\\eta\}\{2\}\\mathbb\{E\}\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\]into two−η4\-\\frac\{\\eta\}\{4\}parts and apply the result of Lemma[B\.1](https://arxiv.org/html/2607.08104#A2.Thmlem1)to the first part,

𝔼​\[Φt\+1\]−𝔼​\[Φt\]\\displaystyle\\mathbb\{E\}\[\\Phi\_\{t\+1\}\]\-\\mathbb\{E\}\[\\Phi\_\{t\}\]≤−η4​\{E2​𝔼​\[Φt\]−E2​A​𝔼​\[‖𝒎t−1‖ν\+1\]−\(1\+2​L\)​E22​\(ν\+1\)​\(η​β1−β\)ν\+1​𝔼​\[‖𝒎t−1‖ν\+1\]\}\\displaystyle\\quad\\leq\-\\frac\{\\eta\}\{4\}\\left\\\{E\_\{2\}\\mathbb\{E\}\\left\[\\Phi\_\{t\}\\right\]\-E\_\{2\}A\\mathbb\{E\}\\left\[\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\\right\]\-\\frac\{\(1\+2L\)E\_\{2\}\}\{2\(\\nu\+1\)\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{\\nu\+1\}\\mathbb\{E\}\\left\[\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\\right\]\\right\\\}−\(η4−L​ην\+12−A​\(1−β\)​\(1\+ν\)2\)​𝔼​\[‖∇f​\(𝒙t\)‖2\]\+\{η​L2​νν\+1​\(η​β1−β\)2​ν−A​\(1−β\)\}​𝔼​\[‖mt−1‖ν\+1\]\\displaystyle\\quad\\quad\-\\left\(\\frac\{\\eta\}\{4\}\-\\frac\{L\\eta^\{\\nu\+1\}\}\{2\}\-\\frac\{A\(1\-\\beta\)\(1\+\\nu\)\}\{2\}\\right\)\\mathbb\{E\}\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\]\+\\left\\\{\\frac\{\\eta L^\{2\}\\nu\}\{\\nu\+1\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\-A\(1\-\\beta\)\\right\\\}\\mathbb\{E\}\[\\\|m\_\{t\-1\}\\\|^\{\\nu\+1\}\]\+L​ην\+1ν\+1​E0\+η​\(1−ν\)​L22​\(ν\+1\)​\(η​β1−β\)2​ν\+A​\{21−ν​\(1−β\)ν\+1​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+\(1−β\)​\(1−ν\)2\}\\displaystyle\\quad\\quad\+\\frac\{L\\eta^\{\\nu\+1\}\}\{\\nu\+1\}E\_\{0\}\+\\frac\{\\eta\(1\-\\nu\)L^\{2\}\}\{2\(\\nu\+1\)\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\+A\\left\\\{2^\{1\-\\nu\}\(1\-\\beta\)^\{\\nu\+1\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+\\frac\{\(1\-\\beta\)\(1\-\\nu\)\}\{2\}\\right\\\}≤−η​E24​𝔼​\[Φt\]\+\{η​E24​A\+η​\(1\+2​L\)​E28​\(ν\+1\)​\(η​β1−β\)ν\+1\+η​L2​νν\+1​\(η​β1−β\)2​ν−A​\(1−β\)\}​𝔼​\[‖mt−1‖ν\+1\]\\displaystyle\\quad\\leq\-\\frac\{\\eta E\_\{2\}\}\{4\}\\mathbb\{E\}\\left\[\\Phi\_\{t\}\\right\]\+\\left\\\{\\frac\{\\eta E\_\{2\}\}\{4\}A\+\\frac\{\\eta\(1\+2L\)E\_\{2\}\}\{8\(\\nu\+1\)\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{\\nu\+1\}\+\\frac\{\\eta L^\{2\}\\nu\}\{\\nu\+1\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\-A\(1\-\\beta\)\\right\\\}\\mathbb\{E\}\[\\\|m\_\{t\-1\}\\\|^\{\\nu\+1\}\]−\(η4−L​ην\+12−A​\(1−β\)​\(1\+ν\)2\)​𝔼​\[‖∇f​\(𝒙t\)‖2\]\+L​ην\+1ν\+1​E0\\displaystyle\\quad\\quad\-\\left\(\\frac\{\\eta\}\{4\}\-\\frac\{L\\eta^\{\\nu\+1\}\}\{2\}\-\\frac\{A\(1\-\\beta\)\(1\+\\nu\)\}\{2\}\\right\)\\mathbb\{E\}\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\]\+\\frac\{L\\eta^\{\\nu\+1\}\}\{\\nu\+1\}E\_\{0\}\+A​\{21−ν​\(1−β\)ν\+1​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+\(1−β\)​\(1−ν\)2\},\\displaystyle\\quad\\quad\+A\\left\\\{2^\{1\-\\nu\}\(1\-\\beta\)^\{\\nu\+1\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+\\frac\{\(1\-\\beta\)\(1\-\\nu\)\}\{2\}\\right\\\},whereE2:=2​μ​\(ν\+1\)ν​\(1\+L1ν\)\+1E\_\{2\}:=\\frac\{2\\mu\(\\nu\+1\)\}\{\\nu\\left\(1\+L^\{\\frac\{1\}\{\\nu\}\}\\right\)\+1\}\. To eliminate the terms involving𝔼​\[‖mt−1‖ν\+1\]\\mathbb\{E\}\[\\\|m\_\{t\-1\}\\\|^\{\\nu\+1\}\], we define the coefficientAAas follows:

A=η​\(1\+2​L\)​E28​\(ν\+1\)​\(η​β1−β\)ν\+1\+η​L2​νν\+1​\(η​β1−β\)2​ν\(1−β\)−η​E24\.\\displaystyle A=\\frac\{\\frac\{\\eta\(1\+2L\)E\_\{2\}\}\{8\(\\nu\+1\)\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{\\nu\+1\}\+\\frac\{\\eta L^\{2\}\\nu\}\{\\nu\+1\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\}\{\(1\-\\beta\)\-\\frac\{\\eta E\_\{2\}\}\{4\}\}\.Sinceη≤2​\(1−β\)E2\\eta\\leq\\frac\{2\(1\-\\beta\)\}\{E\_\{2\}\},A\>0A\>0always holds and we have

A≤η​\(1\+2​L\)​E24​\(1−β\)​\(ν\+1\)​\(η​β1−β\)ν\+1\+2​η​L2​νν\+1​\(η​β1−β\)2​ν\.\\displaystyle A\\leq\\frac\{\\eta\(1\+2L\)E\_\{2\}\}\{4\(1\-\\beta\)\(\\nu\+1\)\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{\\nu\+1\}\+\\frac\{2\\eta L^\{2\}\\nu\}\{\\nu\+1\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\.Let us show that the coefficient of𝔼​\[‖∇f​\(𝒙t\)‖2\]\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]is negative\.

η4−L​ην\+12−A​\(1−β\)​\(1\+ν\)2\\displaystyle\\frac\{\\eta\}\{4\}\-\\frac\{L\\eta^\{\\nu\+1\}\}\{2\}\-\\frac\{A\(1\-\\beta\)\(1\+\\nu\)\}\{2\}≥η4−L​ην\+12−η8​\(1\+2​L\)​E2​\(η​β1−β\)ν\+1−\(1−β\)​η​L2​ν​\(η​β1−β\)2​ν\\displaystyle\\geq\\frac\{\\eta\}\{4\}\-\\frac\{L\\eta^\{\\nu\+1\}\}\{2\}\-\\frac\{\\eta\}\{8\}\(1\+2L\)E\_\{2\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{\\nu\+1\}\-\(1\-\\beta\)\\eta L^\{2\}\\nu\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}=η8​\{2−4​L​ην−\(1\+2​L\)​E2​\(β1−β\)ν\+1​ην\+1−\(1−β\)​L2​ν​\(β1−β\)2​ν​η2​ν\}\.\\displaystyle=\\frac\{\\eta\}\{8\}\\left\\\{2\-4L\\eta^\{\\nu\}\-\(1\+2L\)E\_\{2\}\\left\(\\frac\{\\beta\}\{1\-\\beta\}\\right\)^\{\\nu\+1\}\\eta^\{\\nu\+1\}\-\(1\-\\beta\)L^\{2\}\\nu\\left\(\\frac\{\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\\eta^\{2\\nu\}\\right\\\}\.Sinceη≤min⁡\{\(18​L\)1ν,\[\(1−β\)ν\+12​E2​\(1\+2​L\)​βν\+1\]1ν\+1,\[\(1−β\)2​ν2​L2​ν​β2​ν\]12​ν\}\\eta\\leq\\min\\left\\\{\\left\(\\frac\{1\}\{8L\}\\right\)^\{\\frac\{1\}\{\\nu\}\},\\left\[\\frac\{\(1\-\\beta\)^\{\\nu\+1\}\}\{2E\_\{2\}\(1\+2L\)\\beta^\{\\nu\+1\}\}\\right\]^\{\\frac\{1\}\{\\nu\+1\}\},\\left\[\\frac\{\(1\-\\beta\)^\{2\\nu\}\}\{2L^\{2\}\\nu\\beta^\{2\\nu\}\}\\right\]^\{\\frac\{1\}\{2\\nu\}\}\\right\\\}, we have

ην≤18​L,ην\+1≤12​E2​\(1\+2​L\)​\(1−ββ\)ν\+1,and​η2​ν≤12​L2​ν​\(1−β\)​\(1−ββ\)2​ν\.\\displaystyle\\eta^\{\\nu\}\\leq\\frac\{1\}\{8L\},\\ \\eta^\{\\nu\+1\}\\leq\\frac\{1\}\{2E\_\{2\}\(1\+2L\)\}\\left\(\\frac\{1\-\\beta\}\{\\beta\}\\right\)^\{\\nu\+1\},\\ \\text\{and \}\\ \\eta^\{2\\nu\}\\leq\\frac\{1\}\{2L^\{2\}\\nu\(1\-\\beta\)\}\\left\(\\frac\{1\-\\beta\}\{\\beta\}\\right\)^\{2\\nu\}\.Hence,

η4−L​ην\+12−A​\(1−β\)​\(1\+ν\)2≥η8​\(2−12−12−12\)=η16\.\\displaystyle\\frac\{\\eta\}\{4\}\-\\frac\{L\\eta^\{\\nu\+1\}\}\{2\}\-\\frac\{A\(1\-\\beta\)\(1\+\\nu\)\}\{2\}\\geq\\frac\{\\eta\}\{8\}\\left\(2\-\\frac\{1\}\{2\}\-\\frac\{1\}\{2\}\-\\frac\{1\}\{2\}\\right\)=\\frac\{\\eta\}\{16\}\.Therefore, we have

𝔼​\[Φt\+1\]\\displaystyle\\mathbb\{E\}\[\\Phi\_\{t\+1\}\]≤\(1−η​E24\)​𝔼​\[Φt\]\+L​ην\+1ν\+1​E0\+A​\{21−ν​\(1−β\)ν\+1​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+\(1−β\)​\(1−ν\)2\}\\displaystyle\\leq\\left\(1\-\\frac\{\\eta\{E\_\{2\}\}\}\{4\}\\right\)\\mathbb\{E\}\[\\Phi\_\{t\}\]\+\\frac\{L\\eta^\{\\nu\+1\}\}\{\\nu\+1\}E\_\{0\}\+A\\left\\\{2^\{1\-\\nu\}\(1\-\\beta\)^\{\\nu\+1\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+\\frac\{\(1\-\\beta\)\(1\-\\nu\)\}\{2\}\\right\\\}≤\(1−η​E24\)​𝔼​\[Φt\]\+L​ην\+1ν\+1​E0\+A​\(1−β\)​E0\\displaystyle\\leq\\left\(1\-\\frac\{\\eta\{E\_\{2\}\}\}\{4\}\\right\)\\mathbb\{E\}\[\\Phi\_\{t\}\]\+\\frac\{L\\eta^\{\\nu\+1\}\}\{\\nu\+1\}E\_\{0\}\+A\(1\-\\beta\)E\_\{0\}≤\(1−η​E24\)⏟=⁣:ρ​𝔼​\[Φt\]\+\{L​E0ν\+1\+\(1\+2​L\)​E0​E24​\(ν\+1\)​\(β1−β\)ν\+1​η\+2​L2​ν​\(1−β\)​E0ν\+1​\(β1−β\)2​ν​ην\}⏟=⁣:E3​ην\+1\.\\displaystyle\\leq\\underbrace\{\\left\(1\-\\frac\{\\eta\{E\_\{2\}\}\}\{4\}\\right\)\}\_\{=:\\rho\}\\mathbb\{E\}\[\\Phi\_\{t\}\]\+\\underbrace\{\\left\\\{\\frac\{LE\_\{0\}\}\{\\nu\+1\}\+\\frac\{\(1\+2L\)E\_\{0\}E\_\{2\}\}\{4\(\\nu\+1\)\}\\left\(\\frac\{\\beta\}\{1\-\\beta\}\\right\)^\{\\nu\+1\}\\eta\+\\frac\{2L^\{2\}\\nu\(1\-\\beta\)E\_\{0\}\}\{\\nu\+1\}\\left\(\\frac\{\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\\eta^\{\\nu\}\\right\\\}\}\_\{=:E\_\{3\}\}\\eta^\{\\nu\+1\}\.Sinceη≤2​\(1−β\)E2\\eta\\leq\\frac\{2\(1\-\\beta\)\}\{E\_\{2\}\}, we have\|ρ\|<1\|\\rho\|<1\. Then,

𝔼​\[Φt\+1\]≤ρt​𝔼​\[Φ1\]\+E31−ρ=ρt\+1⋅f​\(𝒙1\)−f⋆ρ\+E3​ην\+11−ρ\.\\displaystyle\\mathbb\{E\}\\left\[\\Phi\_\{t\+1\}\\right\]\\leq\\rho^\{t\}\\mathbb\{E\}\\left\[\\Phi\_\{1\}\\right\]\+\\frac\{E\_\{3\}\}\{1\-\\rho\}=\\rho^\{t\+1\}\\cdot\\frac\{f\(\\bm\{x\}\_\{1\}\)\-f^\{\\star\}\}\{\\rho\}\+\\frac\{E\_\{3\}\\eta^\{\\nu\+1\}\}\{1\-\\rho\}\.Hence,

𝔼​\[f​\(𝒛t\)−f⋆\]≤ρt⋅f​\(𝒙1\)−f⋆ρ\+E3​ην\+11−ρ\.\\displaystyle\\mathbb\{E\}\\left\[f\(\\bm\{z\}\_\{t\}\)\-f^\{\\star\}\\right\]\\leq\\rho^\{t\}\\cdot\\frac\{f\(\\bm\{x\}\_\{1\}\)\-f^\{\\star\}\}\{\\rho\}\+\\frac\{E\_\{3\}\\eta^\{\\nu\+1\}\}\{1\-\\rho\}\.Here, we can expand𝒎t−1=\(1−β\)​∑k=1t−1βt−1−k​∇f𝒮k​\(𝒙k\)\\bm\{m\}\_\{t\-1\}=\(1\-\\beta\)\\sum\_\{k=1\}^\{t\-1\}\\beta^\{t\-1\-k\}\\nabla f\_\{\\mathcal\{S\}\_\{k\}\}\(\\bm\{x\}\_\{k\}\)\. Then, by the definition of𝒛t\\bm\{z\}\_\{t\},

𝒙t\\displaystyle\\bm\{x\}\_\{t\}=𝒛t\+η​β1−β​𝒎t−1=𝒛t\+η​∑k=1t−1βt−k​∇f𝒮k​\(𝒙k\)\.\\displaystyle=\\bm\{z\}\_\{t\}\+\\frac\{\\eta\\beta\}\{1\-\\beta\}\\bm\{m\}\_\{t\-1\}=\\bm\{z\}\_\{t\}\+\\eta\\sum\_\{k=1\}^\{t\-1\}\\beta^\{t\-k\}\\nabla f\_\{\\mathcal\{S\}\_\{k\}\}\(\\bm\{x\}\_\{k\}\)\.Substitutingη​∇f𝒮k​\(𝒙k\)=𝒛k−𝒛k\+1\\eta\\nabla f\_\{\\mathcal\{S\}\_\{k\}\}\(\\bm\{x\}\_\{k\}\)=\\bm\{z\}\_\{k\}\-\\bm\{z\}\_\{k\+1\}gives

𝒙t\\displaystyle\\bm\{x\}\_\{t\}=𝒛t\+∑k=1t−1βt−k​\(𝒛k−𝒛k\+1\)\\displaystyle=\\bm\{z\}\_\{t\}\+\\sum\_\{k=1\}^\{t\-1\}\\beta^\{t\-k\}\(\\bm\{z\}\_\{k\}\-\\bm\{z\}\_\{k\+1\}\)=𝒛t\+\(∑k=1t−1βt−k​𝒛k−∑k=1t−1βt−k​𝒛k\+1\)\\displaystyle=\\bm\{z\}\_\{t\}\+\\left\(\\sum\_\{k=1\}^\{t\-1\}\\beta^\{t\-k\}\\bm\{z\}\_\{k\}\-\\sum\_\{k=1\}^\{t\-1\}\\beta^\{t\-k\}\\bm\{z\}\_\{k\+1\}\\right\)=𝒛t\+βt−1​𝒛1\+∑k=2t−1βt−k​𝒛k−\(∑k=2t−1βt−k\+1​𝒛k\+β​𝒛t\)\\displaystyle=\\bm\{z\}\_\{t\}\+\\beta^\{t\-1\}\\bm\{z\}\_\{1\}\+\\sum\_\{k=2\}^\{t\-1\}\\beta^\{t\-k\}\\bm\{z\}\_\{k\}\-\\left\(\\sum\_\{k=2\}^\{t\-1\}\\beta^\{t\-k\+1\}\\bm\{z\}\_\{k\}\+\\beta\\bm\{z\}\_\{t\}\\right\)=\(1−β\)​𝒛t\+∑k=2t−1βt−k​\(1−β\)​𝒛k\+βt−1​𝒛1\.\\displaystyle=\(1\-\\beta\)\\bm\{z\}\_\{t\}\+\\sum\_\{k=2\}^\{t\-1\}\\beta^\{t\-k\}\(1\-\\beta\)\\bm\{z\}\_\{k\}\+\\beta^\{t\-1\}\\bm\{z\}\_\{1\}\.Finally, we have

\(1−β\)\+\(1−β\)​∑k=2t−1βt−k\+βt−1\\displaystyle\(1\-\\beta\)\+\(1\-\\beta\)\\sum\_\{k=2\}^\{t\-1\}\\beta^\{t\-k\}\+\\beta^\{t\-1\}=\(1−β\)\+\(1−β\)​β​\(1−βt−2\)1−β\+βt−1\\displaystyle=\(1\-\\beta\)\+\(1\-\\beta\)\\frac\{\\beta\(1\-\\beta^\{t\-2\}\)\}\{1\-\\beta\}\+\\beta^\{t\-1\}=1−β\+β−βt−1\+βt−1\\displaystyle=1\-\\beta\+\\beta\-\\beta^\{t\-1\}\+\\beta^\{t\-1\}=1\.\\displaystyle=1\.Sinceβ∈\[0,1\)\\beta\\in\[0,1\),𝒙t\\bm\{x\}\_\{t\}can be expressed as a convex combination with the auxiliary sequence\{𝒛k\}k=1t\\\{\\bm\{z\}\_\{k\}\\\}\_\{k=1\}^\{t\}\. From the convexity offf, we have

𝔼​\[f​\(𝒙t\)−f⋆\]=𝔼​\[f​\(∑k=1twk​𝒛k\)−f⋆\]≤∑k=1twk​𝔼​\[f​\(𝒛k\)−f⋆\]≤f​\(𝒙1\)−f⋆ρ​∑k=1twk​ρk\+E3​ην\+11−ρ,\\displaystyle\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\\right\]=\\mathbb\{E\}\\left\[f\\left\(\\sum\_\{k=1\}^\{t\}w\_\{k\}\\bm\{z\}\_\{k\}\\right\)\-f^\{\\star\}\\right\]\\leq\\sum\_\{k=1\}^\{t\}w\_\{k\}\\mathbb\{E\}\\left\[f\(\\bm\{z\}\_\{k\}\)\-f^\{\\star\}\\right\]\\leq\\frac\{f\(\\bm\{x\}\_\{1\}\)\-f^\{\\star\}\}\{\\rho\}\\sum\_\{k=1\}^\{t\}w\_\{k\}\\rho^\{k\}\+\\frac\{E\_\{3\}\\eta^\{\\nu\+1\}\}\{1\-\\rho\},wherew1:=βt−1,wt:=1−βw\_\{1\}:=\\beta^\{t\-1\},w\_\{t\}:=1\-\\beta, andwk:=\(1−β\)​βt−kw\_\{k\}:=\(1\-\\beta\)\\beta^\{t\-k\}\(1<k<t\(1<k<t\)\. Sinceη≤2​\(1−β\)E2\\eta\\leq\\frac\{2\(1\-\\beta\)\}\{E\_\{2\}\}, we haveβ<ρ\\beta<\\rho\. Then,

∑k=1twk​ρk=βt−1​ρ\+\(1−β\)​∑k=2tβt−k​ρk=βt−1​ρ\+\(1−β\)​ρt​∑j=0t−2\(βρ\)j≤1−βρ−β​ρt\+1\.\\displaystyle\\sum\_\{k=1\}^\{t\}w\_\{k\}\\rho^\{k\}=\\beta^\{t\-1\}\\rho\+\(1\-\\beta\)\\sum\_\{k=2\}^\{t\}\\beta^\{t\-k\}\\rho^\{k\}=\\beta^\{t\-1\}\\rho\+\(1\-\\beta\)\\rho^\{t\}\\sum\_\{j=0\}^\{t\-2\}\\left\(\\frac\{\\beta\}\{\\rho\}\\right\)^\{j\}\\leq\\frac\{1\-\\beta\}\{\\rho\-\\beta\}\\rho^\{t\+1\}\.Finally, we have

𝔼​\[f​\(𝒙t\)−f⋆\]\\displaystyle\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\\right\]≤\(1−β\)​\(f​\(𝒙1\)−f⋆\)ρ−β​ρt\+E3​ην\+11−β\\displaystyle\\leq\\frac\{\(1\-\\beta\)\(f\(\\bm\{x\}\_\{1\}\)\-f^\{\\star\}\)\}\{\\rho\-\\beta\}\\rho^\{t\}\+\\frac\{E\_\{3\}\\eta^\{\\nu\+1\}\}\{1\-\\beta\}≤f​\(𝒙1\)−f⋆2​\(1−η​E24\)t\+E3​ην\+11−β\\displaystyle\\leq\\frac\{f\(\\bm\{x\}\_\{1\}\)\-f^\{\\star\}\}\{2\}\\left\(1\-\\frac\{\\eta E\_\{2\}\}\{4\}\\right\)^\{t\}\+\\frac\{E\_\{3\}\\eta^\{\\nu\+1\}\}\{1\-\\beta\}=𝒪​\(\(1−η​E24\)t\+ην\+1\),\\displaystyle=\\mathcal\{O\}\\left\(\\left\(1\-\\frac\{\\eta E\_\{2\}\}\{4\}\\right\)^\{t\}\+\\eta^\{\\nu\+1\}\\right\),whereE2:=2​μ​\(ν\+1\)ν​\(1\+L1ν\)\+1E\_\{2\}:=\\frac\{2\\mu\(\\nu\+1\)\}\{\\nu\\left\(1\+L^\{\\frac\{1\}\{\\nu\}\}\\right\)\+1\}andE3:=L​E0ν\+1\+\(1\+2​L\)​E0​E24​\(ν\+1\)​\(β1−β\)ν\+1​η\+2​L2​ν​\(1−β\)​E0ν\+1​\(β1−β\)2​ν​ηνE\_\{3\}:=\\frac\{LE\_\{0\}\}\{\\nu\+1\}\+\\frac\{\(1\+2L\)E\_\{0\}E\_\{2\}\}\{4\(\\nu\+1\)\}\\left\(\\frac\{\\beta\}\{1\-\\beta\}\\right\)^\{\\nu\+1\}\\eta\+\\frac\{2L^\{2\}\\nu\(1\-\\beta\)E\_\{0\}\}\{\\nu\+1\}\\left\(\\frac\{\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\\eta^\{\\nu\}\. This completes the proof\. ∎

### B\.4Proof of Theorem[3\.3](https://arxiv.org/html/2607.08104#S3.Thmthm3)

###### Lemma B\.4\.

Letηt:=ηmax​t−1ν\+1\\eta\_\{t\}:=\\eta\_\{\\max\}t^\{\-\\frac\{1\}\{\\nu\+1\}\}be the learning rate schedule\. Then,

1∑t=1Tηt≤νν\+1ηmax​\{\(T\+1\)νν\+1−1\}=𝒪​\(1Tνν\+1\),\\displaystyle\\frac\{1\}\{\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}\}\\leq\\frac\{\\frac\{\\nu\}\{\\nu\+1\}\}\{\\eta\_\{\\max\}\\left\\\{\(T\+1\)^\{\\frac\{\\nu\}\{\\nu\+1\}\}\-1\\right\\\}\}=\\mathcal\{O\}\\left\(\\frac\{1\}\{T^\{\\frac\{\\nu\}\{\\nu\+1\}\}\}\\right\),and

∑t=1Tηtν\+1∑t=1Tηt≤νν\+1​\(1\+log⁡T\)\(T\+1\)νν\+1−1=𝒪​\(log⁡TTνν\+1\)\.\\displaystyle\\frac\{\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}^\{\\nu\+1\}\}\{\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}\}\\leq\\frac\{\\frac\{\\nu\}\{\\nu\+1\}\\left\(1\+\\log T\\right\)\}\{\(T\+1\)^\{\\frac\{\\nu\}\{\\nu\+1\}\}\-1\}=\\mathcal\{O\}\\left\(\\frac\{\\log T\}\{T^\{\\frac\{\\nu\}\{\\nu\+1\}\}\}\\right\)\.

###### Proof\.

Here, we have

∑t=1Tηt=ηmax​∑t=1Tt−1ν\+1≥ηmax​∫1T\+1t−1ν\+1=ηmax​\(T\+1\)1−1ν\+1−11−1ν\+1=ηmax​\(T\+1\)νν\+1−1νν\+1\.\\displaystyle\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}=\\eta\_\{\\max\}\\sum\_\{t=1\}^\{T\}t^\{\-\\frac\{1\}\{\\nu\+1\}\}\\geq\\eta\_\{\\max\}\\int\_\{1\}^\{T\+1\}t^\{\-\\frac\{1\}\{\\nu\+1\}\}=\\eta\_\{\\max\}\\frac\{\(T\+1\)^\{1\-\\frac\{1\}\{\\nu\+1\}\}\-1\}\{1\-\\frac\{1\}\{\\nu\+1\}\}=\\eta\_\{\\max\}\\frac\{\(T\+1\)^\{\\frac\{\\nu\}\{\\nu\+1\}\}\-1\}\{\\frac\{\\nu\}\{\\nu\+1\}\}\.Therefore,

1∑t=1Tηt≤νν\+1ηmax​\{\(T\+1\)νν\+1−1\}\.\\displaystyle\\frac\{1\}\{\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}\}\\leq\\frac\{\\frac\{\\nu\}\{\\nu\+1\}\}\{\\eta\_\{\\max\}\\left\\\{\(T\+1\)^\{\\frac\{\\nu\}\{\\nu\+1\}\}\-1\\right\\\}\}\.In addition,

∑t=1Tηtν\+1=ηmax​∑t=1Tt−1≤ηmax​\(1\+log⁡T\)\.\\displaystyle\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}^\{\\nu\+1\}=\\eta\_\{\\max\}\\sum\_\{t=1\}^\{T\}t^\{\-1\}\\leq\\eta\_\{\\max\}\\left\(1\+\\log T\\right\)\.This completes the proof\. ∎

###### Theorem[3\.3](https://arxiv.org/html/2607.08104#S3.Thmthm3)\(restated\)\.

Let the functionf:ℝd→ℝf\\colon\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}be convex\. Suppose that Assumptions[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1),[2\.2](https://arxiv.org/html/2607.08104#S2.Thmassum2), and[3\.1](https://arxiv.org/html/2607.08104#S3.Thmassum1)\(i\) hold\. With learning rateηt:=ηmax​t−1ν\+1\\eta\_\{t\}:=\\eta\_\{\\max\}t^\{\-\\frac\{1\}\{\\nu\+1\}\}, ifν\+1≤𝔭\\nu\+1\\leq\\mathfrak\{p\}, then:

min1≤t≤T⁡𝔼​\[f​\(𝒙t\)−f⋆\]≤‖𝒙1−𝒙⋆‖ν\+1\(ν\+1\)​D1ν−1​1∑t=1Tηt\+E4​∑t=1Tηtν\+1∑t=1Tηt=𝒪​\(log⁡TTνν\+1\)\.\\displaystyle\\min\_\{1\\leq t\\leq T\}\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\\right\]\\leq\\frac\{\\\|\\bm\{x\}\_\{1\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\}\{\(\\nu\+1\)D\_\{1\}^\{\\nu\-1\}\}\\frac\{1\}\{\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}\}\+E\_\{4\}\\frac\{\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}^\{\\nu\+1\}\}\{\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}\}=\\mathcal\{O\}\\left\(\\frac\{\\log T\}\{T^\{\\frac\{\\nu\}\{\\nu\+1\}\}\}\\right\)\.In particular, whenν\+1=𝔭\\nu\+1=\\mathfrak\{p\},

min1≤t≤T⁡𝔼​\[f​\(𝒙t\)−f⋆\]=𝒪​\(log⁡TT𝔭−1𝔭\),\\displaystyle\\min\_\{1\\leq t\\leq T\}\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\\right\]=\\mathcal\{O\}\\left\(\\frac\{\\log T\}\{T^\{\\frac\{\\mathfrak\{p\}\-1\}\{\\mathfrak\{p\}\}\}\}\\right\),whereE4:=21−ν​\(E0\+ν\+12​L2​D12​ν\)E\_\{4\}:=2^\{1\-\\nu\}\\left\(E\_\{0\}\+\\frac\{\\nu\+1\}\{2\}L^\{2\}D\_\{1\}^\{2\\nu\}\\right\)andE0E\_\{0\}defined in Eq\. \([1](https://arxiv.org/html/2607.08104#S3.E1)\) are non\-negative constants\.

###### Proof\.

From Lemma[2\.2](https://arxiv.org/html/2607.08104#S2.Thmlem2)and Assumption[3\.1](https://arxiv.org/html/2607.08104#S3.Thmassum1)\(i\), we have

‖𝒙t\+1−𝒙⋆‖ν\+1\\displaystyle\\\|\\bm\{x\}\_\{t\+1\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}=‖𝒙t−ηt​∇f𝒮t​\(𝒙t\)−𝒙⋆‖ν\+1\\displaystyle=\\\|\\bm\{x\}\_\{t\}\-\\eta\_\{t\}\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}≤‖𝒙t−𝒙⋆‖ν\+1−ηt​\(ν\+1\)​‖𝒙t−𝒙⋆‖ν−1​⟨𝒙t−𝒙⋆,∇f𝒮t​\(𝒙t\)⟩\+21−ν​‖∇f𝒮t​\(𝒙t\)‖ν\+1\.\\displaystyle\\leq\\\|\\bm\{x\}\_\{t\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\-\\eta\_\{t\}\(\\nu\+1\)\\\|\\bm\{x\}\_\{t\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\-1\}\\langle\\bm\{x\}\_\{t\}\-\\bm\{x\}^\{\\star\},\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\rangle\+2^\{1\-\\nu\}\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\\|^\{\\nu\+1\}\.Taking the conditional expectation of𝒙t\\bm\{x\}\_\{t\}, we have, from the convexity offfand Assumption[3\.1](https://arxiv.org/html/2607.08104#S3.Thmassum1)\(i\),

𝔼​\[‖𝒙t\+1−𝒙⋆‖ν\+1\|𝒙t\]\\displaystyle\\mathbb\{E\}\\left\[\\\|\\bm\{x\}\_\{t\+1\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\|\\bm\{x\}\_\{t\}\\right\]≤‖𝒙t−𝒙⋆‖ν\+1−ηt​\(ν\+1\)​‖𝒙t−𝒙⋆‖ν−1​⟨𝒙t−𝒙⋆,∇f​\(𝒙t\)⟩\+21−ν​𝔼​\[‖∇f𝒮t​\(𝒙t\)‖ν\+1\|𝒙t\]\\displaystyle\\leq\\\|\\bm\{x\}\_\{t\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\-\\eta\_\{t\}\(\\nu\+1\)\\\|\\bm\{x\}\_\{t\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\-1\}\\langle\\bm\{x\}\_\{t\}\-\\bm\{x\}^\{\\star\},\\nabla f\(\\bm\{x\}\_\{t\}\)\\rangle\+2^\{1\-\\nu\}\\mathbb\{E\}\\left\[\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\\|^\{\\nu\+1\}\|\\bm\{x\}\_\{t\}\\right\]≤‖𝒙t−𝒙⋆‖ν\+1−ηt​\(ν\+1\)​‖𝒙t−𝒙⋆‖ν−1​\(f​\(𝒙t\)−f⋆\)\+21−ν​𝔼​\[‖∇f𝒮t​\(𝒙t\)‖ν\+1\|𝒙t\]\\displaystyle\\leq\\\|\\bm\{x\}\_\{t\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\-\\eta\_\{t\}\(\\nu\+1\)\\\|\\bm\{x\}\_\{t\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\-1\}\(f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\)\+2^\{1\-\\nu\}\\mathbb\{E\}\\left\[\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\\|^\{\\nu\+1\}\|\\bm\{x\}\_\{t\}\\right\]≤‖𝒙t−𝒙⋆‖ν\+1−ηt​\(ν\+1\)​D1ν−1​\(f​\(𝒙t\)−f⋆\)\+21−ν​𝔼​\[‖∇f𝒮t​\(𝒙t\)‖ν\+1\|𝒙t\]\.\\displaystyle\\leq\\\|\\bm\{x\}\_\{t\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\-\\eta\_\{t\}\(\\nu\+1\)D\_\{1\}^\{\\nu\-1\}\(f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\)\+2^\{1\-\\nu\}\\mathbb\{E\}\\left\[\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\\|^\{\\nu\+1\}\|\\bm\{x\}\_\{t\}\\right\]\.
By taking the total expectation, we have

𝔼​\[‖𝒙t\+1−𝒙⋆‖ν\+1\]≤𝔼​\[‖𝒙t−𝒙⋆‖ν\+1\]−ηt​\(ν\+1\)​D1ν−1​\(f​\(𝒙t\)−f⋆\)\+21−ν​ηtν\+1​𝔼​\[‖∇f𝒮t​\(𝒙t\)‖ν\+1\]\.\\displaystyle\\mathbb\{E\}\\left\[\\\|\\bm\{x\}\_\{t\+1\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\\right\]\\leq\\mathbb\{E\}\\left\[\\\|\\bm\{x\}\_\{t\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\\right\]\-\\eta\_\{t\}\(\\nu\+1\)D\_\{1\}^\{\\nu\-1\}\(f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\)\+2^\{1\-\\nu\}\\eta\_\{t\}^\{\\nu\+1\}\\mathbb\{E\}\\left\[\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\\|^\{\\nu\+1\}\\right\]\.From Lemma[3\.1](https://arxiv.org/html/2607.08104#S3.Thmlem1), we obtain

𝔼​\[‖𝒙t\+1−𝒙⋆‖ν\+1\]\\displaystyle\\mathbb\{E\}\\left\[\\\|\\bm\{x\}\_\{t\+1\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\\right\]≤𝔼​\[‖𝒙t−𝒙⋆‖ν\+1\]−ηt​\(ν\+1\)​D1ν−1​𝔼​\[f​\(𝒙t\)−f⋆\]\\displaystyle\\leq\\mathbb\{E\}\\left\[\\\|\\bm\{x\}\_\{t\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\\right\]\-\\eta\_\{t\}\(\\nu\+1\)D\_\{1\}^\{\\nu\-1\}\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\\right\]\+21−ν​\{21−ν​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+ν\+12​𝔼​\[‖∇f​\(𝒙t\)‖2\]\+1−ν2\}​ηtν\+1\.\\displaystyle\\quad\+2^\{1\-\\nu\}\\left\\\{2^\{1\-\\nu\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+\\frac\{\\nu\+1\}\{2\}\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]\+\\frac\{1\-\\nu\}\{2\}\\right\\\}\\eta\_\{t\}^\{\\nu\+1\}\.Here, from Assumptions[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1)and[3\.1](https://arxiv.org/html/2607.08104#S3.Thmassum1)\(i\), we have‖∇f​\(𝒙t\)−∇f​\(𝒙⋆\)‖≤L​‖𝒙t−𝒙⋆‖ν=L​D1ν\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\-\\nabla f\(\\bm\{x\}^\{\\star\}\)\\\|\\leq L\\\|\\bm\{x\}\_\{t\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\}=LD\_\{1\}^\{\\nu\}, i\.e\.,‖∇f​\(𝒙t\)‖2≤L2​D12​ν\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\leq L^\{2\}D\_\{1\}^\{2\\nu\}\. Hence,

𝔼​\[‖𝒙t\+1−𝒙⋆‖ν\+1\]\\displaystyle\\mathbb\{E\}\\left\[\\\|\\bm\{x\}\_\{t\+1\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\\right\]≤𝔼​\[‖𝒙t−𝒙⋆‖ν\+1\]−ηt​\(ν\+1\)​D1ν−1​𝔼​\[f​\(𝒙t\)−f⋆\]\\displaystyle\\leq\\mathbb\{E\}\\left\[\\\|\\bm\{x\}\_\{t\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\\right\]\-\\eta\_\{t\}\(\\nu\+1\)D\_\{1\}^\{\\nu\-1\}\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\\right\]\+21−ν​\{21−ν​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+ν\+12​L2​D12​ν\+1−ν2\}⏟=⁣:E4​ηtν\+1\\displaystyle\\quad\+\\underbrace\{2^\{1\-\\nu\}\\left\\\{2^\{1\-\\nu\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+\\frac\{\\nu\+1\}\{2\}L^\{2\}D\_\{1\}^\{2\\nu\}\+\\frac\{1\-\\nu\}\{2\}\\right\\\}\}\_\{=:E\_\{4\}\}\\eta\_\{t\}^\{\\nu\+1\}≤𝔼​\[‖𝒙t−𝒙⋆‖ν\+1\]−ηt​\(ν\+1\)​D1ν−1​𝔼​\[f​\(𝒙t\)−f⋆\]\+E4​ηtν\+1\.\\displaystyle\\leq\\mathbb\{E\}\\left\[\\\|\\bm\{x\}\_\{t\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\\right\]\-\\eta\_\{t\}\(\\nu\+1\)D\_\{1\}^\{\\nu\-1\}\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\\right\]\+E\_\{4\}\\eta\_\{t\}^\{\\nu\+1\}\.Rearranging the terms,

ηt​𝔼​\[f​\(𝒙t\)−f⋆\]≤1\(ν\+1\)​D1ν−1​\(𝔼​\[‖𝒙t−𝒙⋆‖ν\+1\]−𝔼​\[‖𝒙t\+1−𝒙⋆‖ν\+1\]\)\+E4\(ν\+1\)​D1ν−1​ηtν\+1,\\displaystyle\\eta\_\{t\}\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\\right\]\\leq\\frac\{1\}\{\(\\nu\+1\)D\_\{1\}^\{\\nu\-1\}\}\\left\(\\mathbb\{E\}\\left\[\\\|\\bm\{x\}\_\{t\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\\right\]\-\\mathbb\{E\}\\left\[\\\|\\bm\{x\}\_\{t\+1\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\\right\]\\right\)\+\\frac\{E\_\{4\}\}\{\(\\nu\+1\)D\_\{1\}^\{\\nu\-1\}\}\\eta\_\{t\}^\{\\nu\+1\},and summing overt=1t=1tot=Tt=T, we have

∑t=1Tηt​𝔼​\[f​\(𝒙t\)−f⋆\]≤‖𝒙1−𝒙⋆‖ν\+1\(ν\+1\)​D1ν−1\+E4\(ν\+1\)​D1ν−1​∑t=1Tηtν\+1\.\\displaystyle\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\\right\]\\leq\\frac\{\\\|\\bm\{x\}\_\{1\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\}\{\(\\nu\+1\)D\_\{1\}^\{\\nu\-1\}\}\+\\frac\{E\_\{4\}\}\{\(\\nu\+1\)D\_\{1\}^\{\\nu\-1\}\}\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}^\{\\nu\+1\}\.Hence,

min1≤t≤T⁡𝔼​\[f​\(𝒙t\)−f⋆\]≤‖𝒙1−𝒙⋆‖ν\+1\(ν\+1\)​D1ν−1​1∑t=1Tηt\+E4\(ν\+1\)​D1ν−1​∑t=1Tηtν\+1∑t=1Tηt\.\\displaystyle\\min\_\{1\\leq t\\leq T\}\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\\right\]\\leq\\frac\{\\\|\\bm\{x\}\_\{1\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\}\{\(\\nu\+1\)D\_\{1\}^\{\\nu\-1\}\}\\frac\{1\}\{\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}\}\+\\frac\{E\_\{4\}\}\{\(\\nu\+1\)D\_\{1\}^\{\\nu\-1\}\}\\frac\{\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}^\{\\nu\+1\}\}\{\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}\}\.Sinceηt:=ηmax​t−1ν\+1\\eta\_\{t\}:=\\eta\_\{\\max\}t^\{\-\\frac\{1\}\{\\nu\+1\}\}, from Lemma[B\.4](https://arxiv.org/html/2607.08104#A2.Thmlem4), this completes the proof\. ∎

### B\.5Proof of Theorem[3\.4](https://arxiv.org/html/2607.08104#S3.Thmthm4)

###### Theorem[3\.4](https://arxiv.org/html/2607.08104#S3.Thmthm4)\(restated\)\.

Let the functionf:ℝd→ℝf\\colon\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}be convex\. Suppose that Assumptions[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1),[2\.2](https://arxiv.org/html/2607.08104#S2.Thmassum2),[3\.1](https://arxiv.org/html/2607.08104#S3.Thmassum1)\(i\), and[3\.1](https://arxiv.org/html/2607.08104#S3.Thmassum1)\(ii\) hold\. With learning rateηt:=η\\eta\_\{t\}:=\\eta, ifν\+1≤𝔭\\nu\+1\\leq\\mathfrak\{p\}, then:

min1≤t≤T⁡f​\(𝒙t\)−f⋆≤‖𝒙1−𝒙⋆‖ν\+1η​\(ν\+1\)​D2ν−1​T\+E6​η\+21−ν​E5\(ν\+1\)​D2ν−1​ην=𝒪​\(1η​T\+ην\)\.\\displaystyle\\min\_\{1\\leq t\\leq T\}f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\\leq\\frac\{\\\|\\bm\{x\}\_\{1\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\}\{\\eta\(\\nu\+1\)D\_\{2\}^\{\\nu\-1\}T\}\+E\_\{6\}\\eta\+\\frac\{2^\{1\-\\nu\}E\_\{5\}\}\{\(\\nu\+1\)D\_\{2\}^\{\\nu\-1\}\}\\eta^\{\\nu\}=\\mathcal\{O\}\\left\(\\frac\{1\}\{\\eta T\}\+\\eta^\{\\nu\}\\right\)\.In particular, whenν\+1=𝔭\\nu\+1=\\mathfrak\{p\},

min1≤t≤T⁡f​\(𝒙t\)−f⋆=𝒪​\(1η​T\+η𝔭−1\),\\displaystyle\\min\_\{1\\leq t\\leq T\}f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}=\\mathcal\{O\}\\left\(\\frac\{1\}\{\\eta T\}\+\\eta^\{\\mathfrak\{p\}\-1\}\\right\),whereE5:=21−ν​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+Lν\+1​D1ν​\(ν\+1\)E\_\{5\}:=2^\{1\-\\nu\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+L^\{\\nu\+1\}D\_\{1\}^\{\\nu\(\\nu\+1\)\},E6:=β​\(E5\+ν​Lν\+1ν​D1ν\+1\)\(1−β\)​\(ν\+1\)E\_\{6\}:=\\frac\{\\beta\\left\(E\_\{5\}\+\\nu L^\{\\frac\{\\nu\+1\}\{\\nu\}\}D\_\{1\}^\{\\nu\+1\}\\right\)\}\{\(1\-\\beta\)\(\\nu\+1\)\}are non\-negative constants\.

###### Proof\.

For allt∈ℕt\\in\\mathbb\{N\}, letwt:=‖𝒛t−𝒙⋆‖ν−1w\_\{t\}:=\\\|\\bm\{z\}\_\{t\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\-1\}\. From Lemma[2\.2](https://arxiv.org/html/2607.08104#S2.Thmlem2),

‖𝒛t\+1−𝒙⋆‖ν\+1\\displaystyle\\\|\\bm\{z\}\_\{t\+1\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}=‖𝒛t−𝒙⋆−η​∇f𝒮t​\(𝒙t\)‖ν\+1\\displaystyle=\\\|\\bm\{z\}\_\{t\}\-\\bm\{x\}^\{\\star\}\-\\eta\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\\|^\{\\nu\+1\}≤‖𝒛t−𝒙⋆‖ν\+1−η​\(ν\+1\)​wt​⟨𝒛t−𝒙⋆,∇f𝒮t​\(𝒙t\)⟩\+21−ν​ην\+1​‖∇f𝒮t​\(𝒙t\)‖ν\+1\\displaystyle\\leq\\\|\\bm\{z\}\_\{t\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\-\\eta\(\\nu\+1\)w\_\{t\}\\langle\\bm\{z\}\_\{t\}\-\\bm\{x\}^\{\\star\},\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\rangle\+2^\{1\-\\nu\}\\eta^\{\\nu\+1\}\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\\|^\{\\nu\+1\}=‖𝒛t−𝒙⋆‖ν\+1−η​\(ν\+1\)​wt​⟨𝒙t−𝒙⋆,∇f𝒮t​\(𝒙t\)⟩\\displaystyle=\\\|\\bm\{z\}\_\{t\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\-\\eta\(\\nu\+1\)w\_\{t\}\\langle\\bm\{x\}\_\{t\}\-\\bm\{x\}^\{\\star\},\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\rangle\+η​\(ν\+1\)​wt​⟨𝒙t−𝒛t,∇f𝒮t​\(𝒙t\)⟩\+21−ν​ην\+1​‖∇f𝒮t​\(𝒙t\)‖ν\+1\\displaystyle\\quad\+\\eta\(\\nu\+1\)w\_\{t\}\\langle\\bm\{x\}\_\{t\}\-\\bm\{z\}\_\{t\},\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\rangle\+2^\{1\-\\nu\}\\eta^\{\\nu\+1\}\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\\|^\{\\nu\+1\}=‖𝒛t−𝒙⋆‖ν\+1−η​\(ν\+1\)​wt​⟨𝒙t−𝒙⋆,∇f𝒮t​\(𝒙t\)⟩\\displaystyle=\\\|\\bm\{z\}\_\{t\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\-\\eta\(\\nu\+1\)w\_\{t\}\\langle\\bm\{x\}\_\{t\}\-\\bm\{x\}^\{\\star\},\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\rangle\+η2​β​\(ν\+1\)1−β​wt​⟨𝒎t−1,∇f𝒮t​\(𝒙t\)⟩\+21−ν​ην\+1​‖∇f𝒮t​\(𝒙t\)‖ν\+1\.\\displaystyle\\quad\+\\frac\{\\eta^\{2\}\\beta\(\\nu\+1\)\}\{1\-\\beta\}w\_\{t\}\\langle\\bm\{m\}\_\{t\-1\},\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\rangle\+2^\{1\-\\nu\}\\eta^\{\\nu\+1\}\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\\|^\{\\nu\+1\}\.Taking the conditional expectation of𝒙t\\bm\{x\}\_\{t\}, we have, from the convexity offf,

𝔼​\[‖𝒛t\+1−𝒙⋆‖ν\+1\|𝒙t\]\\displaystyle\\mathbb\{E\}\\left\[\\\|\\bm\{z\}\_\{t\+1\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\|\\bm\{x\}\_\{t\}\\right\]≤‖𝒛t−𝒙⋆‖ν\+1−η​\(ν\+1\)​wt​\(f​\(𝒙t\)−f⋆\)\\displaystyle\\leq\\\|\\bm\{z\}\_\{t\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\-\\eta\(\\nu\+1\)w\_\{t\}\(f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\)\+η2​β​\(ν\+1\)1−β​wt​⟨𝒎t−1,∇f​\(𝒙t\)⟩\+21−ν​ην\+1​𝔼​\[‖∇f𝒮t​\(𝒙t\)‖ν\+1\|𝒙t\]\.\\displaystyle\\quad\+\\frac\{\\eta^\{2\}\\beta\(\\nu\+1\)\}\{1\-\\beta\}w\_\{t\}\\langle\\bm\{m\}\_\{t\-1\},\\nabla f\(\\bm\{x\}\_\{t\}\)\\rangle\+2^\{1\-\\nu\}\\eta^\{\\nu\+1\}\\mathbb\{E\}\\left\[\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\\|^\{\\nu\+1\}\|\\bm\{x\}\_\{t\}\\right\]\.Moreover, from the Cauchy\-Schwarz inequality, Young’s inequality, and Assumptions[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1)and[3\.1](https://arxiv.org/html/2607.08104#S3.Thmassum1)\(i\),

⟨𝒎t−1,∇f​\(𝒙t\)⟩≤‖𝒎t−1‖​‖∇f​\(𝒙t\)‖≤‖𝒎t−1‖ν\+1ν\+1\+ν​‖∇f​\(𝒙t\)‖ν\+1νν\+1≤‖𝒎t−1‖ν\+1ν\+1\+ν​Lν\+1ν​D1ν\+1ν\+1\.\\displaystyle\\langle\\bm\{m\}\_\{t\-1\},\\nabla f\(\\bm\{x\}\_\{t\}\)\\rangle\\leq\\\|\\bm\{m\}\_\{t\-1\}\\\|\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|\\leq\\frac\{\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\}\{\\nu\+1\}\+\\frac\{\\nu\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{\\frac\{\\nu\+1\}\{\\nu\}\}\}\{\\nu\+1\}\\leq\\frac\{\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\}\{\\nu\+1\}\+\\frac\{\\nu L^\{\\frac\{\\nu\+1\}\{\\nu\}\}D\_\{1\}^\{\\nu\+1\}\}\{\\nu\+1\}\.In addition, from Lemma[3\.1](https://arxiv.org/html/2607.08104#S3.Thmlem1)and the convexity of∥⋅∥ν\+1\\\|\\cdot\\\|^\{\\nu\+1\},

𝔼​\[‖𝒎t−1‖ν\+1\]\\displaystyle\\mathbb\{E\}\\left\[\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\\right\]≤β​𝔼​\[‖𝒎t−1‖ν\+1\]\+\(1−β\)​𝔼​\[‖∇f𝒮t​\(𝒙t\)‖ν\+1\]\\displaystyle\\leq\\beta\\mathbb\{E\}\\left\[\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\\right\]\+\(1\-\\beta\)\\mathbb\{E\}\\left\[\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\\|^\{\\nu\+1\}\\right\]≤β​𝔼​\[‖𝒎t−1‖ν\+1\]\+\(1−β\)​\{21−ν​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+𝔼​\[‖∇f​\(𝒙t\)‖ν\+1\]\}\\displaystyle\\leq\\beta\\mathbb\{E\}\\left\[\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\\right\]\+\(1\-\\beta\)\\left\\\{2^\{1\-\\nu\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{\\nu\+1\}\\right\]\\right\\\}≤β​𝔼​\[‖𝒎t−1‖ν\+1\]\+\(1−β\)​\{21−ν​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+Lν\+1​D1ν​\(ν\+1\)\}\\displaystyle\\leq\\beta\\mathbb\{E\}\\left\[\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\\right\]\+\(1\-\\beta\)\\left\\\{2^\{1\-\\nu\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+L^\{\\nu\+1\}D\_\{1\}^\{\\nu\(\\nu\+1\)\}\\right\\\}≤21−ν​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+Lν\+1​D1ν​\(ν\+1\)⏟=⁣:E5\.\\displaystyle\\leq\\underbrace\{2^\{1\-\\nu\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+L^\{\\nu\+1\}D\_\{1\}^\{\\nu\(\\nu\+1\)\}\}\_\{=:E\_\{5\}\}\.Therefore, by taking the expectation, we have

𝔼​\[‖𝒛t\+1−𝒙⋆‖ν\+1\]≤𝔼​\[‖𝒛t−𝒙⋆‖ν\+1\]−η​\(ν\+1\)​wt​\(f​\(𝒙t\)−f⋆\)\+η2​β​wt1−β​\(E5\+ν​Lν\+1ν​D1ν\+1\)\+21−ν​ην\+1​E5\.\\displaystyle\\mathbb\{E\}\\left\[\\\|\\bm\{z\}\_\{t\+1\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\\right\]\\leq\\mathbb\{E\}\\left\[\\\|\\bm\{z\}\_\{t\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\\right\]\-\\eta\(\\nu\+1\)w\_\{t\}\(f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\)\+\\frac\{\\eta^\{2\}\\beta w\_\{t\}\}\{1\-\\beta\}\\left\(E\_\{5\}\+\\nu L^\{\\frac\{\\nu\+1\}\{\\nu\}\}D\_\{1\}^\{\\nu\+1\}\\right\)\+2^\{1\-\\nu\}\\eta^\{\\nu\+1\}E\_\{5\}\.Rearranging the terms,

wt​\(f​\(𝒙t\)−f⋆\)≤𝔼​\[‖𝒛t−𝒙⋆‖ν\+1\]−𝔼​\[‖𝒛t\+1−𝒙⋆‖ν\+1\]η​\(ν\+1\)\+β​\(E5\+ν​Lν\+1ν​D1ν\+1\)\(1−β\)​\(ν\+1\)​η​wt\+21−ν​E5ν\+1​ην,\\displaystyle w\_\{t\}\(f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\)\\leq\\frac\{\\mathbb\{E\}\\left\[\\\|\\bm\{z\}\_\{t\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\\right\]\-\\mathbb\{E\}\\left\[\\\|\\bm\{z\}\_\{t\+1\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\\right\]\}\{\\eta\(\\nu\+1\)\}\+\\frac\{\\beta\\left\(E\_\{5\}\+\\nu L^\{\\frac\{\\nu\+1\}\{\\nu\}\}D\_\{1\}^\{\\nu\+1\}\\right\)\}\{\(1\-\\beta\)\(\\nu\+1\)\}\\eta w\_\{t\}\+\\frac\{2^\{1\-\\nu\}E\_\{5\}\}\{\\nu\+1\}\\eta^\{\\nu\},and summing overt=1t=1tot=Tt=T, we obtain

∑t=1Twt​\(f​\(𝒙t\)−f⋆\)≤‖𝒙1−𝒙⋆‖ν\+1η​\(ν\+1\)\+β​\(E5\+ν​Lν\+1ν​D1ν\+1\)\(1−β\)​\(ν\+1\)​η​∑t=1Twt\+21−ν​E5ν\+1​ην​T\.\\displaystyle\\sum\_\{t=1\}^\{T\}w\_\{t\}\(f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\)\\leq\\frac\{\\\|\\bm\{x\}\_\{1\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\}\{\\eta\(\\nu\+1\)\}\+\\frac\{\\beta\\left\(E\_\{5\}\+\\nu L^\{\\frac\{\\nu\+1\}\{\\nu\}\}D\_\{1\}^\{\\nu\+1\}\\right\)\}\{\(1\-\\beta\)\(\\nu\+1\)\}\\eta\\sum\_\{t=1\}^\{T\}w\_\{t\}\+\\frac\{2^\{1\-\\nu\}E\_\{5\}\}\{\\nu\+1\}\\eta^\{\\nu\}T\.Hence,

min1≤t≤T⁡f​\(𝒙t\)−f⋆≤∑t=1Twt​\(f​\(𝒙t\)−f⋆\)∑t=1Twt≤‖𝒙1−𝒙⋆‖ν\+1η​\(ν\+1\)​∑t=1Twt\+β​\(E5\+ν​Lν\+1ν​D1ν\+1\)\(1−β\)​\(ν\+1\)​η\+21−ν​E5\(ν\+1\)​∑t=1Twt​ην​T\.\\displaystyle\\min\_\{1\\leq t\\leq T\}f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\\leq\\frac\{\\sum\_\{t=1\}^\{T\}w\_\{t\}\(f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}\)\}\{\\sum\_\{t=1\}^\{T\}w\_\{t\}\}\\leq\\frac\{\\\|\\bm\{x\}\_\{1\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\}\{\\eta\(\\nu\+1\)\\sum\_\{t=1\}^\{T\}w\_\{t\}\}\+\\frac\{\\beta\\left\(E\_\{5\}\+\\nu L^\{\\frac\{\\nu\+1\}\{\\nu\}\}D\_\{1\}^\{\\nu\+1\}\\right\)\}\{\(1\-\\beta\)\(\\nu\+1\)\}\\eta\+\\frac\{2^\{1\-\\nu\}E\_\{5\}\}\{\(\\nu\+1\)\\sum\_\{t=1\}^\{T\}w\_\{t\}\}\\eta^\{\\nu\}T\.Here, from Assumption[3\.1](https://arxiv.org/html/2607.08104#S3.Thmassum1)\(ii\),wt:=‖𝒛t−𝒙⋆‖ν−1≥D2ν−1w\_\{t\}:=\\\|\\bm\{z\}\_\{t\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\-1\}\\geq D\_\{2\}^\{\\nu\-1\}\. Therefore,

min1≤t≤T⁡f​\(𝒙t\)−f⋆\\displaystyle\\min\_\{1\\leq t\\leq T\}f\(\\bm\{x\}\_\{t\}\)\-f^\{\\star\}≤‖𝒙1−𝒙⋆‖ν\+1η​\(ν\+1\)​D2ν−1​T\+β​\(E5\+ν​Lν\+1ν​D1ν\+1\)\(1−β\)​\(ν\+1\)⏟=⁣:E6​η\+21−ν​E5\(ν\+1\)​D2ν−1​ην\\displaystyle\\leq\\frac\{\\\|\\bm\{x\}\_\{1\}\-\\bm\{x\}^\{\\star\}\\\|^\{\\nu\+1\}\}\{\\eta\(\\nu\+1\)D\_\{2\}^\{\\nu\-1\}T\}\+\\underbrace\{\\frac\{\\beta\\left\(E\_\{5\}\+\\nu L^\{\\frac\{\\nu\+1\}\{\\nu\}\}D\_\{1\}^\{\\nu\+1\}\\right\)\}\{\(1\-\\beta\)\(\\nu\+1\)\}\}\_\{=:E\_\{6\}\}\\eta\+\\frac\{2^\{1\-\\nu\}E\_\{5\}\}\{\(\\nu\+1\)D\_\{2\}^\{\\nu\-1\}\}\\eta^\{\\nu\}=𝒪​\(1η​T\+ην\)\.\\displaystyle=\\mathcal\{O\}\\left\(\\frac\{1\}\{\\eta T\}\+\\eta^\{\\nu\}\\right\)\.This completes the proof\. ∎

### B\.6Proof of Theorem[3\.5](https://arxiv.org/html/2607.08104#S3.Thmthm5)

###### Theorem[3\.5](https://arxiv.org/html/2607.08104#S3.Thmthm5)\(restated\)\.

Let the functionf:ℝd→ℝf\\colon\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}be nonconvex\. Suppose that Assumptions[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1)and[2\.2](https://arxiv.org/html/2607.08104#S2.Thmassum2)hold\. With learning rateηt:=ηmax​t−1ν\+1\\eta\_\{t\}:=\\eta\_\{\\max\}t^\{\-\\frac\{1\}\{\\nu\+1\}\}, ifν\+1≤𝔭\\nu\+1\\leq\\mathfrak\{p\}andηtν<2L\\eta\_\{t\}^\{\\nu\}<\\frac\{2\}\{L\}for allt∈\[T\]t\\in\[T\], then:

min1≤t≤T⁡𝔼​\[‖∇f​\(𝒙t\)‖2\]≤2​\(f​\(𝒙1\)−f⋆\)\(2−L​ηmaxν\)​1∑t=1Tηt\+E7​∑t=1Tηtν\+1∑t=1Tηt=𝒪​\(log⁡TTνν\+1\)\.\\displaystyle\\min\_\{1\\leq t\\leq T\}\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]\\leq\\frac\{2\(f\(\\bm\{x\}\_\{1\}\)\-f^\{\\star\}\)\}\{\\left\(2\-L\\eta\_\{\\max\}^\{\\nu\}\\right\)\}\\frac\{1\}\{\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}\}\+E\_\{7\}\\frac\{\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}^\{\\nu\+1\}\}\{\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}\}=\\mathcal\{O\}\\left\(\\frac\{\\log T\}\{T^\{\\frac\{\\nu\}\{\\nu\+1\}\}\}\\right\)\.In particular, whenν\+1=𝔭\\nu\+1=\\mathfrak\{p\},

min1≤t≤T⁡𝔼​\[‖∇f​\(𝒙t\)‖2\]=𝒪​\(log⁡TT𝔭−1𝔭\),\\displaystyle\\min\_\{1\\leq t\\leq T\}\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]=\\mathcal\{O\}\\left\(\\frac\{\\log T\}\{T^\{\\frac\{\\mathfrak\{p\}\-1\}\{\\mathfrak\{p\}\}\}\}\\right\),whereE7:=2​E0​L\(ν\+1\)​\(2−L​ηmaxν\)E\_\{7\}:=\\frac\{2E\_\{0\}L\}\{\(\\nu\+1\)\\left\(2\-L\\eta\_\{\\max\}^\{\\nu\}\\right\)\}andE0E\_\{0\}defined in Eq\. \([1](https://arxiv.org/html/2607.08104#S3.E1)\) are non\-negative constants\.

###### Proof\.

From Lemma[2\.1](https://arxiv.org/html/2607.08104#S2.Thmlem1), we have

f​\(𝒙t\+1\)\\displaystyle f\(\\bm\{x\}\_\{t\+1\}\)≤f​\(𝒙t\)\+⟨∇f​\(𝒙t\),𝒙t\+1−𝒙t⟩\+Lν\+1​‖𝒙t\+1−𝒙t‖ν\+1\\displaystyle\\leq f\(\\bm\{x\}\_\{t\}\)\+\\langle\\nabla f\(\\bm\{x\}\_\{t\}\),\\bm\{x\}\_\{t\+1\}\-\\bm\{x\}\_\{t\}\\rangle\+\\frac\{L\}\{\\nu\+1\}\\\|\\bm\{x\}\_\{t\+1\}\-\\bm\{x\}\_\{t\}\\\|^\{\\nu\+1\}=f​\(𝒙t\)−ηt​⟨∇f​\(𝒙t\),∇f𝒮t​\(𝒙t\)⟩\+L​ηtν\+1ν\+1​‖∇f𝒮t​\(𝒙t\)‖ν\+1\.\\displaystyle=f\(\\bm\{x\}\_\{t\}\)\-\\eta\_\{t\}\\langle\\nabla f\(\\bm\{x\}\_\{t\}\),\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\rangle\+\\frac\{L\\eta\_\{t\}^\{\\nu\+1\}\}\{\\nu\+1\}\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\\|^\{\\nu\+1\}\.By taking the expectation, we have

𝔼​\[f​\(𝒙t\+1\)\]\\displaystyle\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\+1\}\)\\right\]≤𝔼​\[f​\(𝒙t\)\]−ηt​𝔼​\[‖∇f​\(𝒙t\)‖2\]\+L​ηtν\+1ν\+1​𝔼​\[‖∇f𝒮t​\(𝒙t\)‖ν\+1\]\.\\displaystyle\\leq\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\}\)\\right\]\-\\eta\_\{t\}\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]\+\\frac\{L\\eta\_\{t\}^\{\\nu\+1\}\}\{\\nu\+1\}\\mathbb\{E\}\\left\[\\\|\\nabla f\_\{\\mathcal\{S\}\_\{t\}\}\(\\bm\{x\}\_\{t\}\)\\\|^\{\\nu\+1\}\\right\]\.From Lemma[3\.1](https://arxiv.org/html/2607.08104#S3.Thmlem1),

𝔼​\[f​\(𝒙t\+1\)\]\\displaystyle\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\+1\}\)\\right\]≤𝔼​\[f​\(𝒙t\)\]−ηt​𝔼​\[‖∇f​\(𝒙t\)‖2\]\+L​ηtν\+1ν\+1​\{21−ν​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+ν\+12​𝔼​\[‖∇f​\(𝒙t\)‖2\]\+1−ν2\}\\displaystyle\\leq\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\}\)\\right\]\-\\eta\_\{t\}\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]\+\\frac\{L\\eta\_\{t\}^\{\\nu\+1\}\}\{\\nu\+1\}\\left\\\{2^\{1\-\\nu\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+\\frac\{\\nu\+1\}\{2\}\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]\+\\frac\{1\-\\nu\}\{2\}\\right\\\}=𝔼​\[f​\(𝒙t\)\]−ηt​\(1−L​ηtν2\)​𝔼​\[‖∇f​\(𝒙t\)‖2\]\+L​ηtν\+1ν\+1​\{21−ν​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+1−ν2\}\\displaystyle=\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\}\)\\right\]\-\\eta\_\{t\}\\left\(1\-\\frac\{L\\eta\_\{t\}^\{\\nu\}\}\{2\}\\right\)\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]\+\\frac\{L\\eta\_\{t\}^\{\\nu\+1\}\}\{\\nu\+1\}\\left\\\{2^\{1\-\\nu\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+\\frac\{1\-\\nu\}\{2\}\\right\\\}≤𝔼​\[f​\(𝒙t\)\]−ηt​\(1−L​ηmaxν2\)​𝔼​\[‖∇f​\(𝒙t\)‖2\]\+L​ηtν\+1ν\+1​\{21−ν​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+1−ν2\}\.\\displaystyle\\leq\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\}\)\\right\]\-\\eta\_\{t\}\\left\(1\-\\frac\{L\\eta\_\{\\max\}^\{\\nu\}\}\{2\}\\right\)\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]\+\\frac\{L\\eta\_\{t\}^\{\\nu\+1\}\}\{\\nu\+1\}\\left\\\{2^\{1\-\\nu\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+\\frac\{1\-\\nu\}\{2\}\\right\\\}\.Rearranging the terms,

ηt​\(1−L​ηmaxν2\)​𝔼​\[‖∇f​\(𝒙t\)‖2\]\\displaystyle\\eta\_\{t\}\\left\(1\-\\frac\{L\\eta\_\{\\max\}^\{\\nu\}\}\{2\}\\right\)\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]≤𝔼​\[f​\(𝒙t\)\]−𝔼​\[f​\(𝒙t\+1\)\]\+L​ηtν\+1ν\+1​\{21−ν​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+1−ν2\},\\displaystyle\\leq\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\}\)\\right\]\-\\mathbb\{E\}\\left\[f\(\\bm\{x\}\_\{t\+1\}\)\\right\]\+\\frac\{L\\eta\_\{t\}^\{\\nu\+1\}\}\{\\nu\+1\}\\left\\\{2^\{1\-\\nu\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+\\frac\{1\-\\nu\}\{2\}\\right\\\},and summing overt=1t=1tot=Tt=T, we have

\(1−L​ηmaxν2\)​∑t=1Tηt​𝔼​\[‖∇f​\(𝒙t\)‖2\]≤f​\(𝒙1\)−f⋆\+Lν\+1​\{21−ν​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+1−ν2\}⏟=⁣:E0​∑t=1Tηtν\+1\.\\displaystyle\\left\(1\-\\frac\{L\\eta\_\{\\max\}^\{\\nu\}\}\{2\}\\right\)\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]\\leq f\(\\bm\{x\}\_\{1\}\)\-f^\{\\star\}\+\\frac\{L\}\{\\nu\+1\}\\underbrace\{\\left\\\{2^\{1\-\\nu\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+\\frac\{1\-\\nu\}\{2\}\\right\\\}\}\_\{=:E\_\{0\}\}\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}^\{\\nu\+1\}\.Therefore,

min1≤t≤T⁡𝔼​\[‖∇f​\(𝒙t\)‖2\]≤2​\(f​\(𝒙1\)−f⋆\)\(2−L​ηmaxν\)​1∑t=1Tηt\+2​E0​L\(ν\+1\)​\(2−L​ηmaxν\)⏟=⁣:E7​∑t=1Tηtν\+1∑t=1Tηt\.\\displaystyle\\min\_\{1\\leq t\\leq T\}\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]\\leq\\frac\{2\(f\(\\bm\{x\}\_\{1\}\)\-f^\{\\star\}\)\}\{\\left\(2\-L\\eta\_\{\\max\}^\{\\nu\}\\right\)\}\\frac\{1\}\{\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}\}\+\\underbrace\{\\frac\{2E\_\{0\}L\}\{\(\\nu\+1\)\\left\(2\-L\\eta\_\{\\max\}^\{\\nu\}\\right\)\}\}\_\{=:E\_\{7\}\}\\frac\{\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}^\{\\nu\+1\}\}\{\\sum\_\{t=1\}^\{T\}\\eta\_\{t\}\}\.Sinceηt:=ηmax​t−1ν\+1\\eta\_\{t\}:=\\eta\_\{\\max\}t^\{\-\\frac\{1\}\{\\nu\+1\}\}, we can use Lemma[B\.4](https://arxiv.org/html/2607.08104#A2.Thmlem4)to complete the proof\. ∎

### B\.7Proof of Theorem[3\.6](https://arxiv.org/html/2607.08104#S3.Thmthm6)

###### Theorem[3\.6](https://arxiv.org/html/2607.08104#S3.Thmthm6)\(restated\)\.

Let the functionf:ℝd→ℝf\\colon\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}be nonconvex\. Suppose that Assumptions[2\.1](https://arxiv.org/html/2607.08104#S2.Thmassum1)and[2\.2](https://arxiv.org/html/2607.08104#S2.Thmassum2)hold\. With learning rateηt:=η\\eta\_\{t\}:=\\eta, ifν\+1≤𝔭\\nu\+1\\leq\\mathfrak\{p\}andη≤min⁡\{\(14​L\)1ν,1−ββ​\(14​ν​L2\)12​ν\}\\eta\\leq\\min\\left\\\{\\left\(\\frac\{1\}\{4L\}\\right\)^\{\\frac\{1\}\{\\nu\}\},\\frac\{1\-\\beta\}\{\\beta\}\\left\(\\frac\{1\}\{4\\nu L^\{2\}\}\\right\)^\{\\frac\{1\}\{2\\nu\}\}\\right\\\}, then:

min1≤t≤T⁡𝔼​\[‖∇f​\(𝒙t\)‖2\]≤4​\(f​\(𝒙1\)−f⋆\)η​T\+L​E0​ην4​\(ν\+1\)\+E8​η2​ν=𝒪​\(1η​T\+ην\)\.\\displaystyle\\min\_\{1\\leq t\\leq T\}\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]\\leq\\frac\{4\(f\(\\bm\{x\}\_\{1\}\)\-f^\{\\star\}\)\}\{\\eta T\}\+\\frac\{LE\_\{0\}\\eta^\{\\nu\}\}\{4\(\\nu\+1\)\}\+E\_\{8\}\\eta^\{2\\nu\}=\\mathcal\{O\}\\left\(\\frac\{1\}\{\\eta T\}\+\\eta^\{\\nu\}\\right\)\.In particular, whenν\+1=𝔭\\nu\+1=\\mathfrak\{p\},

min1≤t≤T⁡𝔼​\[‖∇f​\(𝒙t\)‖2\]=𝒪​\(1η​T\+η𝔭−1\),\\displaystyle\\min\_\{1\\leq t\\leq T\}\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]=\\mathcal\{O\}\\left\(\\frac\{1\}\{\\eta T\}\+\\eta^\{\\mathfrak\{p\}\-1\}\\right\),whereE8:=2​Lν\+1​\(β1−β\)2​ν​\{L​\(1−ν\)\+2​E0​ν\}E\_\{8\}:=\\frac\{2L\}\{\\nu\+1\}\\left\(\\frac\{\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\\left\\\{L\(1\-\\nu\)\+2E\_\{0\}\\nu\\right\\\}andE0E\_\{0\}defined in Eq\. \([1](https://arxiv.org/html/2607.08104#S3.E1)\) are non\-negative constants\.

###### Proof\.

For allt∈ℕt\\in\\mathbb\{N\}, let us defineΦt:=f​\(𝒛t\)\+λ​‖𝒎t−1‖ν\+1\\Phi\_\{t\}:=f\(\\bm\{z\}\_\{t\}\)\+\\lambda\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\},λ:=4​η​L​ν\(1−β\)​\(ν\+1\)​\(η​β1−β\)2​ν\\lambda:=\\frac\{4\\eta L\\nu\}\{\(1\-\\beta\)\(\\nu\+1\)\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\. Then, we have

𝔼​\[Φt\+1\]−𝔼​\[Φt\]=𝔼​\[f​\(𝒛t\+1\)\]−𝔼​\[f​\(𝒛t\)\]\+λ​\(𝔼​\[‖𝒎t‖ν\+1\]−𝔼​\[‖𝒎t−1‖ν\+1\]\)\.\\displaystyle\\mathbb\{E\}\[\\Phi\_\{t\+1\}\]\-\\mathbb\{E\}\[\\Phi\_\{t\}\]=\\mathbb\{E\}\[f\(\\bm\{z\}\_\{t\+1\}\)\]\-\\mathbb\{E\}\[f\(\\bm\{z\}\_\{t\}\)\]\+\\lambda\\left\(\\mathbb\{E\}\[\\\|\\bm\{m\}\_\{t\}\\\|^\{\\nu\+1\}\]\-\\mathbb\{E\}\[\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\]\\right\)\.Therefore, from Lemmas[B\.2](https://arxiv.org/html/2607.08104#A2.Thmlem2)and[B\.3](https://arxiv.org/html/2607.08104#A2.Thmlem3),

𝔼​\[Φt\+1\]−𝔼​\[Φt\]\\displaystyle\\mathbb\{E\}\[\\Phi\_\{t\+1\}\]\-\\mathbb\{E\}\[\\Phi\_\{t\}\]≤𝔼​\[f​\(𝒛t\)\]−\(η2−L​ην\+12\)​𝔼​\[‖∇f​\(𝒙t\)‖2\]\+η​L2​νν\+1​\(η​β1−β\)2​ν​𝔼​\[‖𝒎t−1‖ν\+1\]\\displaystyle\\quad\\leq\\mathbb\{E\}\[f\(\\bm\{z\}\_\{t\}\)\]\-\\left\(\\frac\{\\eta\}\{2\}\-\\frac\{L\\eta^\{\\nu\+1\}\}\{2\}\\right\)\\mathbb\{E\}\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\]\+\\frac\{\\eta L^\{2\}\\nu\}\{\\nu\+1\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\\mathbb\{E\}\\left\[\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\\right\]\+η​\(1−ν\)​L22​\(ν\+1\)​\(η​β1−β\)2​ν\+L​ην\+1ν\+1​\{21−ν​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\+1−ν2\}⏟=⁣:E0\\displaystyle\\quad\\quad\+\\frac\{\\eta\(1\-\\nu\)L^\{2\}\}\{2\(\\nu\+1\)\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\+\\frac\{L\\eta^\{\\nu\+1\}\}\{\\nu\+1\}\\underbrace\{\\left\\\{2^\{1\-\\nu\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\+\\frac\{1\-\\nu\}\{2\}\\right\\\}\}\_\{=:E\_\{0\}\}\+λ​\{−\(1−β\)​𝔼​\[‖𝒎t−1‖ν\+1\]\+\(1−β\)​\(1\+ν\)2​𝔼​\[‖∇f​\(𝒙t\)‖2\]\+\(1−β\)​\(1−ν\)2\+21−ν​\(1−β\)ν\+1​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\}\\displaystyle\\quad\\quad\+\\lambda\\left\\\{\-\(1\-\\beta\)\\mathbb\{E\}\[\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\]\+\\frac\{\(1\-\\beta\)\(1\+\\nu\)\}\{2\}\\mathbb\{E\}\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\]\+\\frac\{\(1\-\\beta\)\(1\-\\nu\)\}\{2\}\+2^\{1\-\\nu\}\(1\-\\beta\)^\{\\nu\+1\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\\right\\\}=−η2​\(1−L​ην−L2​ν​\(η​β1−β\)2​ν\)​𝔼​\[‖∇f​\(𝒙t\)‖2\]\+\(η​L2​νν\+1​\(η​β1−β\)2​ν−λ​\(1−β\)\)​𝔼​\[‖𝒎t−1‖ν\+1\]\\displaystyle\\quad=\-\\frac\{\\eta\}\{2\}\\left\(1\-L\\eta^\{\\nu\}\-L^\{2\}\\nu\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\\right\)\\mathbb\{E\}\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\]\+\\left\(\\frac\{\\eta L^\{2\}\\nu\}\{\\nu\+1\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\-\\lambda\(1\-\\beta\)\\right\)\\mathbb\{E\}\[\\\|\\bm\{m\}\_\{t\-1\}\\\|^\{\\nu\+1\}\]\+L​E0​ην\+1ν\+1\+η​\(1−ν\)​L22​\(ν\+1\)​\(η​β1−β\)2​ν\+λ​\{\(1−β\)​\(1−ν\)2\+21−ν​\(1−β\)ν\+1​\(C𝔭​σ𝔭b𝔭−1\)ν\+1𝔭\}\\displaystyle\\quad\\quad\+\\frac\{LE\_\{0\}\\eta^\{\\nu\+1\}\}\{\\nu\+1\}\+\\frac\{\\eta\(1\-\\nu\)L^\{2\}\}\{2\(\\nu\+1\)\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\+\\lambda\\left\\\{\\frac\{\(1\-\\beta\)\(1\-\\nu\)\}\{2\}\+2^\{1\-\\nu\}\(1\-\\beta\)^\{\\nu\+1\}\\left\(\\frac\{C\_\{\\mathfrak\{p\}\}\\sigma^\{\\mathfrak\{p\}\}\}\{b^\{\\mathfrak\{p\}\-1\}\}\\right\)^\{\\frac\{\\nu\+1\}\{\\mathfrak\{p\}\}\}\\right\\\}≤−η2​\(1−L​ην−L2​ν​\(η​β1−β\)2​ν\)​𝔼​\[‖∇f​\(𝒙t\)‖2\]\+L​E0​ην\+1ν\+1\\displaystyle\\quad\\leq\-\\frac\{\\eta\}\{2\}\\left\(1\-L\\eta^\{\\nu\}\-L^\{2\}\\nu\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\\right\)\\mathbb\{E\}\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\]\+\\frac\{LE\_\{0\}\\eta^\{\\nu\+1\}\}\{\\nu\+1\}\+η​\(1−ν\)​L22​\(ν\+1\)​\(η​β1−β\)2​ν\+4​η​L​ν\(ν\+1\)​\(η​β1−β\)2​ν​E0,\\displaystyle\\quad\\quad\+\\frac\{\\eta\(1\-\\nu\)L^\{2\}\}\{2\(\\nu\+1\)\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\+\\frac\{4\\eta L\\nu\}\{\(\\nu\+1\)\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}E\_\{0\},where we have used the definition ofλ\\lambdaand\(1−β\)ν\+1<1−β\(1\-\\beta\)^\{\\nu\+1\}<1\-\\betain the last inequality\. Here, from the condition onη\\eta,

η2​\(1−L​ην−L2​ν​\(η​β1−β\)2​ν\)≥η2⋅12=η4\.\\displaystyle\\frac\{\\eta\}\{2\}\\left\(1\-L\\eta^\{\\nu\}\-L^\{2\}\\nu\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\\right\)\\geq\\frac\{\\eta\}\{2\}\\cdot\\frac\{1\}\{2\}=\\frac\{\\eta\}\{4\}\.Hence, we have

η4​𝔼​\[‖∇f​\(𝒙t\)‖2\]\\displaystyle\\frac\{\\eta\}\{4\}\\mathbb\{E\}\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\]≤𝔼​\[Φt\]−𝔼​\[Φt\+1\]\+L​E0​ην\+1ν\+1\+η​\(1−ν\)​L22​\(ν\+1\)​\(η​β1−β\)2​ν\+4​η​L​ν\(ν\+1\)​\(η​β1−β\)2​ν​E0\.\\displaystyle\\leq\\mathbb\{E\}\[\\Phi\_\{t\}\]\-\\mathbb\{E\}\[\\Phi\_\{t\+1\}\]\+\\frac\{LE\_\{0\}\\eta^\{\\nu\+1\}\}\{\\nu\+1\}\+\\frac\{\\eta\(1\-\\nu\)L^\{2\}\}\{2\(\\nu\+1\)\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\+\\frac\{4\\eta L\\nu\}\{\(\\nu\+1\)\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}E\_\{0\}\.Summing overt=1t=1tot=Tt=T, we have

∑t=1T𝔼​\[‖∇f​\(𝒙t\)‖2\]\\displaystyle\\sum\_\{t=1\}^\{T\}\\mathbb\{E\}\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\]≤4​\(f​\(𝒙1\)−f⋆\)η\+L​E0​ην4​\(ν\+1\)​T\+2​\(1−ν\)​L2\(ν\+1\)​\(η​β1−β\)2​ν​T\+4​L​ν\(ν\+1\)​\(η​β1−β\)2​ν​E0​T\.\\displaystyle\\leq\\frac\{4\(f\(\\bm\{x\}\_\{1\}\)\-f^\{\\star\}\)\}\{\\eta\}\+\\frac\{LE\_\{0\}\\eta^\{\\nu\}\}\{4\(\\nu\+1\)\}T\+\\frac\{2\(1\-\\nu\)L^\{2\}\}\{\(\\nu\+1\)\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}T\+\\frac\{4L\\nu\}\{\(\\nu\+1\)\}\\left\(\\frac\{\\eta\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}E\_\{0\}T\.Therefore,

1T​∑t=1T𝔼​\[‖∇f​\(𝒙t\)‖2\]\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\mathbb\{E\}\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\]≤4​\(f​\(𝒙1\)−f⋆\)η​T\+L​E04​\(ν\+1\)​ην\+2​\(1−ν\)​L2\(ν\+1\)​\(β1−β\)2​ν​η2​ν\+4​L​ν\(ν\+1\)​\(β1−β\)2​ν​E0​η2​ν\\displaystyle\\leq\\frac\{4\(f\(\\bm\{x\}\_\{1\}\)\-f^\{\\star\}\)\}\{\\eta T\}\+\\frac\{LE\_\{0\}\}\{4\(\\nu\+1\)\}\\eta^\{\\nu\}\+\\frac\{2\(1\-\\nu\)L^\{2\}\}\{\(\\nu\+1\)\}\\left\(\\frac\{\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\\eta^\{2\\nu\}\+\\frac\{4L\\nu\}\{\(\\nu\+1\)\}\\left\(\\frac\{\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}E\_\{0\}\\eta^\{2\\nu\}=4​\(f​\(𝒙1\)−f⋆\)η​T\+L​E04​\(ν\+1\)​ην\+2​Lν\+1​\(β1−β\)2​ν​\{L​\(1−ν\)\+2​E0​ν\}⏟=⁣:E8​η2​ν\\displaystyle=\\frac\{4\(f\(\\bm\{x\}\_\{1\}\)\-f^\{\\star\}\)\}\{\\eta T\}\+\\frac\{LE\_\{0\}\}\{4\(\\nu\+1\)\}\\eta^\{\\nu\}\+\\underbrace\{\\frac\{2L\}\{\\nu\+1\}\\left\(\\frac\{\\beta\}\{1\-\\beta\}\\right\)^\{2\\nu\}\\left\\\{L\(1\-\\nu\)\+2E\_\{0\}\\nu\\right\\\}\}\_\{=:E\_\{8\}\}\\eta^\{2\\nu\}=𝒪​\(1η​T\+ην\)\.\\displaystyle=\\mathcal\{O\}\\left\(\\frac\{1\}\{\\eta T\}\+\\eta^\{\\nu\}\\right\)\.Since the minimum value is smaller than the average value, we have

min1≤t≤T⁡𝔼​\[‖∇f​\(𝒙t\)‖2\]≤1T​∑t=1T𝔼​\[‖∇f​\(𝒙t\)‖2\]\.\\displaystyle\\min\_\{1\\leq t\\leq T\}\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]\\leq\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\mathbb\{E\}\\left\[\\\|\\nabla f\(\\bm\{x\}\_\{t\}\)\\\|^\{2\}\\right\]\.This completes the proof\. ∎

## Appendix CAdditional Experiments

In this section, we present numerical experimental results for vanilla SGD\. These results parallel the observations for vanilla SGD with momentum discussed in Section[4](https://arxiv.org/html/2607.08104#S4)\. All results presented in this work are fully reproducible via the source code available at:[https://github\.com/iiduka\-researches/vanillaSGDM\_uai2026](https://github.com/iiduka-researches/vanillaSGDM_uai2026)\. The code is designed to run directly on Google Colab for ease of use\.

Figure[4](https://arxiv.org/html/2607.08104#A3.F4)displays the convergence performance of vanilla SGD across various combinations ofν\\nuandα\\alpha\. Similar to the SGD with momentum case, the high\-performance configurations \(marked in red\) are concentrated in the lower\-right region where the theoretical conditionν\+1≤α\\nu\+1\\leq\\alphais satisfied\. This further confirms that our stability boundary is a fundamental requirement for the vanilla stochastic gradient descent family, regardless of the momentum parameter\.

Figure[5](https://arxiv.org/html/2607.08104#A3.F5)illustrates the convergence behavior of vanilla SGD over 1,000 steps for fixedν\\nu\. The plots exhibit the same characteristics evident in Figure[2](https://arxiv.org/html/2607.08104#S4.F2): a smallerα\\alphaleads to both degraded average performance and significantly higher variance across trials\. Consistent with our theory, trajectories for largerν\\nubecome increasingly erratic asα\\alphadecreases, whereas smallerν\\nuvalues provide a more robust optimization path\.

![Refer to caption](https://arxiv.org/html/2607.08104v1/x7.png)Figure 4:Convergence performance across various\(ν,α\)\(\\nu,\\alpha\)pairs\. The heatmaps show the average final objective values \(for strongly convex and convex cases\) and gradient norms \(for nonconvex cases\) after 4,000 steps of vanilla SGD\. The top 35 performing combinations are indicated by red markers\. Below the red dotted line, the theoretical conditionν\+1≤α\\nu\+1\\leq\\alphais satisfied, ensuring convergence\.![Refer to caption](https://arxiv.org/html/2607.08104v1/x8.png)![Refer to caption](https://arxiv.org/html/2607.08104v1/x9.png)![Refer to caption](https://arxiv.org/html/2607.08104v1/x10.png)![Refer to caption](https://arxiv.org/html/2607.08104v1/x11.png)

Figure 5:Convergence trajectories of vanilla SGD\. The plots illustrate the evolution of objective values \(for strongly convex and convex cases\) and gradient norms \(for nonconvex cases\) over 1,000 steps for the functions defined in Eq\. \([4](https://arxiv.org/html/2607.08104#S4.Ex44)\)\. Solid lines represent the average of 20 independent trials, while the lightly shaded regions indicate the range between the minimum and maximum values\.