A Unified Analysis of Importance-Corrected Momentum Actor-Critic over a Markovian Trajectory

# A Unified Analysis of Importance-Corrected Momentum Actor-Critic over a Markovian Trajectory
Source: [https://arxiv.org/html/2610.33477](https://arxiv.org/html/2610.33477)
Yanjie Dong ydong@xjtu\.edu\.cnAffiliation:Zhou Su zhousu@ieee\.orgAffiliation:School of Cyber\-Science and Engineering, Xi’an Jiaotong UniversityWei Wang w25wang@xjtu\.edu\.cnGang Wang gangwang@bit\.edu\.cnAffiliation:State Key Laboratory of Autonomous Intelligent Unmanned Systems, Beijing Institute of Technology

###### Abstract

We present a generic actor\-critic method that combines a ridge\-regularized pathwise least\-squares temporal\-difference \(LSTD\) critic with importance\-corrected momentum \(ICM\) actors\. The critic solves a ridge\-regularized system per frame by Newton\-Schulz inversion and needs no stepsize, no knowledge of the mixing time, and no rank condition\. Averaging stochastic policy gradients over frames ofTTconsecutive transitions, the ICM actors incur a frame\-dependent gradient bias of order1/T\\nicefrac\{\{1\}\}\{\{T\}\}\. To our knowledge, we establish the first finite\-time bounds for ICM actor\-critic methods with normalized and unnormalized steps along a single Markovian trajectory\. Up to a stationary\-to\-discounted distribution mismatch, the averaged gradient norm overKKupdates is bounded by𝒪\(K−1/3\+1/T\)\{\\cal O\}\(K^\{\-1/3\}\+\\nicefrac\{\{1\}\}\{\{T\}\}\)for normalized steps and by𝒪\(K−1/4\+1/T\)\{\\cal O\}\(K^\{\-1/4\}\+\\nicefrac\{\{1\}\}\{\{T\}\}\)for unnormalized steps\. We specialize these bounds to Muon, normalized stochastic gradient descent \(SGD\) with momentum, SGD with momentum, and AdaNorm, and compare them in the nuclear norm\. With a dimension\-free stepsize, Muon confines the factormin⁡\{m,n\}1/2\\min\\\{m,n\\\}^\{1/2\}associated with anm×nm\\times nactor parameter matrix to the noise and correction terms\. The comparison also characterizes when orthogonalization guarantees a larger noiseless ascent than normalization: exactly when the gradient’s nuclear rank exceeds the spectral\-to\-Frobenius curvature ratio\.

††shortheadings:Generic Importance\-Corrected Momentum Actor\-Critic / Dong, Su, Wang, and Wang††firstpage:1††editor:TBA###### keywords

Actor\-critic methods, importance\-corrected momentum, least\-squares temporal difference, Markovian sampling

## 1Introduction

Actor\-critic \(AC\) methods improve the policy along a policy\-gradient \(PG\) estimate where a learned critic replaces Monte Carlo returns to reduce gradient variance\([Konda and Tsitsiklis, 1999](https://arxiv.org/html/2610.33477#bib.bib30);[Sutton et al\., 1999](https://arxiv.org/html/2610.33477#bib.bib58);[Sutton and Barto, 2018](https://arxiv.org/html/2610.33477#bib.bib57)\)\. Convergence analyses of AC methods bound the propagation of critic error into actor update asymptotically\([Konda and Tsitsiklis, 1999](https://arxiv.org/html/2610.33477#bib.bib30);[Zhang et al\., 2020b](https://arxiv.org/html/2610.33477#bib.bib72)\)and non\-asymptotically for double\-loop\([Qiu et al\., 2021](https://arxiv.org/html/2610.33477#bib.bib48);[Kumar et al\., 2023](https://arxiv.org/html/2610.33477#bib.bib32);[Zhang et al\., 2021](https://arxiv.org/html/2610.33477#bib.bib71)\), two\-timescale\([Wu et al\., 2020](https://arxiv.org/html/2610.33477#bib.bib63);[Hong et al\., 2023](https://arxiv.org/html/2610.33477#bib.bib19);[Zeng et al\., 2024](https://arxiv.org/html/2610.33477#bib.bib69);[Zeng and Doan, 2024](https://arxiv.org/html/2610.33477#bib.bib68)\), and single\-timescale\([Olshevsky and Gharesifard, 2023](https://arxiv.org/html/2610.33477#bib.bib42);[Chen and Zhao, 2023](https://arxiv.org/html/2610.33477#bib.bib9);[Shen and Chen, 2022](https://arxiv.org/html/2610.33477#bib.bib53)\)settings\. The vanilla critic performssemi\-gradienttemporal\-difference \(TD\) policy evaluation\([Bhandari et al\., 2021](https://arxiv.org/html/2610.33477#bib.bib5)\)Moreover, the vanilla critic can be accelerated by adaptive stepsizes and momentum accelerate\([Sun et al\., 2022](https://arxiv.org/html/2610.33477#bib.bib55);[Dong et al\., 2024](https://arxiv.org/html/2610.33477#bib.bib15);[Kumar et al\., 2026](https://arxiv.org/html/2610.33477#bib.bib33)\), and freed from knowledge of the mixing time by multi\-level Monte Carlo PG estimators\([Dorfman and Levy, 2022](https://arxiv.org/html/2610.33477#bib.bib16);[Suttle et al\., 2023](https://arxiv.org/html/2610.33477#bib.bib56);[Patel et al\., 2024](https://arxiv.org/html/2610.33477#bib.bib44);[Ganesh et al\., 2025](https://arxiv.org/html/2610.33477#bib.bib18)\)

Research gaps\.While current convergence analyses of AC methods update the actor by unnormalized gradients\([Wu et al\., 2020](https://arxiv.org/html/2610.33477#bib.bib63);[Xu et al\., 2020a](https://arxiv.org/html/2610.33477#bib.bib65);[Xu et al\., 2020b](https://arxiv.org/html/2610.33477#bib.bib66);[Qiu et al\., 2021](https://arxiv.org/html/2610.33477#bib.bib48);[Kumar et al\., 2023](https://arxiv.org/html/2610.33477#bib.bib32);[Olshevsky and Gharesifard, 2023](https://arxiv.org/html/2610.33477#bib.bib42);[Chen and Zhao, 2023](https://arxiv.org/html/2610.33477#bib.bib9)\), recent actor designs depart from such steps in two ways\. First, an importance\-corrected momentum \(ICM\) estimator\([Huang et al\., 2020](https://arxiv.org/html/2610.33477#bib.bib21)\)can reduce momentum variance by reweighting the previous PG estimator on the current trajectory via importance sampling\. Second, the momentum could be normalized\([Cutkosky and Mehta, 2020](https://arxiv.org/html/2610.33477#bib.bib11)\)and orthogonalized\([Jordan et al\., 2024](https://arxiv.org/html/2610.33477#bib.bib22);[Chang et al\., 2025](https://arxiv.org/html/2610.33477#bib.bib8);[Lion et al\., 2026](https://arxiv.org/html/2610.33477#bib.bib38)\)\. For example, Muon\([Jordan et al\., 2024](https://arxiv.org/html/2610.33477#bib.bib22)\)applies the Newton\-Schulz orthogonalization \(NSO\)\([Kelley, 1995](https://arxiv.org/html/2610.33477#bib.bib25)\)to the momentum matrix and moves the model parameter along an approximation of momentum polar factor\. Moreover, a normalized step can bound the importance weights without further assumptions\([Barakat et al\., 2023](https://arxiv.org/html/2610.33477#bib.bib3)\)\. To our knowledge, such orthogonalization is absent from policy updates reshaped by Fisher preconditioning\([Kakade, 2001](https://arxiv.org/html/2610.33477#bib.bib23);[Peters and Schaal, 2008](https://arxiv.org/html/2610.33477#bib.bib45);[Agarwal et al\., 2021](https://arxiv.org/html/2610.33477#bib.bib1);[Khodadadian et al\., 2023](https://arxiv.org/html/2610.33477#bib.bib27);[Wu et al\., 2017](https://arxiv.org/html/2610.33477#bib.bib62)\), mirror descent\([Yang et al\., 2022](https://arxiv.org/html/2610.33477#bib.bib67);[Lan, 2023](https://arxiv.org/html/2610.33477#bib.bib35)\), trust\-region and proximal surrogates\([Schulman et al\., 2015](https://arxiv.org/html/2610.33477#bib.bib51);[Schulman et al\., 2017](https://arxiv.org/html/2610.33477#bib.bib52)\), momentum\([Huang et al\., 2020](https://arxiv.org/html/2610.33477#bib.bib21);[Chen et al\., 2024](https://arxiv.org/html/2610.33477#bib.bib10)\), and Adam\-type steps\([Xiong et al\., 2021](https://arxiv.org/html/2610.33477#bib.bib64)\)\. Besides,Muon\-basedreinforcement learning has only been investigated empirically\([Ruan et al\., 2026](https://arxiv.org/html/2610.33477#bib.bib50);[Fan et al\., 2026](https://arxiv.org/html/2610.33477#bib.bib17)\)\. The current finite\-time analyses of normalized and orthogonalized momentum methods assume unbiased independent and identically distributed \(i\.i\.d\.\) gradients, e\.g\., normalized stochastic gradient descent \(SGD\) with momentum\([Cutkosky and Mehta, 2020](https://arxiv.org/html/2610.33477#bib.bib11)\), Muon with finite NSO\([Kim and Oh, 2026](https://arxiv.org/html/2610.33477#bib.bib28)\)or variance reduction\([Chang et al\., 2025](https://arxiv.org/html/2610.33477#bib.bib8)\), and linear\-minimization\-oracle methods\([Pethick et al\., 2025](https://arxiv.org/html/2610.33477#bib.bib46);[Kovalev, 2025](https://arxiv.org/html/2610.33477#bib.bib31);[Shen et al\., 2025](https://arxiv.org/html/2610.33477#bib.bib54);[Riabinin et al\., 2026](https://arxiv.org/html/2610.33477#bib.bib49)\)\.

The requirement of unbiased i\.i\.d\. gradients cannot be met by reinforcement learning tasks along a single trajectory, whose PG estimators exhibit bias and correlated noise: the score\-weighted TD\-based PG estimator relies on a critic that is still being learned, and the sample trajectory is drawn from a Markov chain induced by the evolving actor\. Prior works leave the underlying three aspects unaddressed when normalized and orthogonalized ICM actors learn along a single trajectory that consists ofTTtransitions per consecutive frame\.

- i\.Analyses of normalized and orthogonalized momentum are based on unbiased gradients\([Cutkosky and Mehta, 2020](https://arxiv.org/html/2610.33477#bib.bib11);[Barakat et al\., 2023](https://arxiv.org/html/2610.33477#bib.bib3);[Kim and Oh, 2026](https://arxiv.org/html/2610.33477#bib.bib28);[Chang et al\., 2025](https://arxiv.org/html/2610.33477#bib.bib8);[Pethick et al\., 2025](https://arxiv.org/html/2610.33477#bib.bib46);[Kovalev, 2025](https://arxiv.org/html/2610.33477#bib.bib31);[Shen et al\., 2025](https://arxiv.org/html/2610.33477#bib.bib54);[Riabinin et al\., 2026](https://arxiv.org/html/2610.33477#bib.bib49)\), analyses of biased stochastic approximation and SGD control the bias only through its norm\([Karimi et al\., 2019](https://arxiv.org/html/2610.33477#bib.bib24);[Ajalloeian and Stich, 2020](https://arxiv.org/html/2610.33477#bib.bib2);[Demidovich et al\., 2023](https://arxiv.org/html/2610.33477#bib.bib13)\), and variance\-reduced methods with biased estimators assume a mean field that depends on the current iterate alone\([Hu et al\., 2020](https://arxiv.org/html/2610.33477#bib.bib20);[Khanduri et al\., 2021](https://arxiv.org/html/2610.33477#bib.bib26)\), whereas the frame estimates of an AC loop have a bias that depends on the iterate and on a start state generated by the previous frame\.
- ii\.Existing finite\-time AC analyses are confined to unnormalized steps, whose progress is controlled through second moments\([Wu et al\., 2020](https://arxiv.org/html/2610.33477#bib.bib63);[Olshevsky and Gharesifard, 2023](https://arxiv.org/html/2610.33477#bib.bib42);[Chen and Zhao, 2023](https://arxiv.org/html/2610.33477#bib.bib9)\)\. Normalized steps, in contrast, necessitate first\-moment control of the momentum’s tracking error, and ICM actors are not accommodated\.
- iii\.ICM has been analyzed for i\.i\.d\. samples\([Cutkosky and Orabona, 2019](https://arxiv.org/html/2610.33477#bib.bib12)\)and for restarted episodes\([Huang et al\., 2020](https://arxiv.org/html/2610.33477#bib.bib21);[Barakat et al\., 2023](https://arxiv.org/html/2610.33477#bib.bib3)\),whereas along a single trajectory the kernel evolves with the actor and the importance weight corrects the actions of a frame but not its start state, which couples consecutive frames\.

Our contributions\.Different from the AC framework of[Kumar et al\. \(2023\)](https://arxiv.org/html/2610.33477#bib.bib32)that combines a fixed SGD actor with several critic under i\.i\.d\. sampling, our AC framework averages the PG estimator over a frame attenuates the critic error by the frame length such that the ICM\-based actor update determines the convergence rate under a single Markovian trajectory\. Therefore, by fixing the critic, the PG estimator, and the ICM, we investigate how different actor updates govern the convergence rate and the dependence on the rank of the actor parameter\. The critic is a ridge\-regularized pathwise least\-squares TD \(LSTD\) critic, i\.e\., the pathwise LSTD critic of[Lazaric et al\. \(2012\)](https://arxiv.org/html/2610.33477#bib.bib36)with a ridge term\. By leveraging Newton\-Schulz inversion \(NSI\) that requires only general matrix\-matrix multiplications \(GEMMs\), we approximate the critic parameter while fully exploiting the parallel computing power of graphics processing units \(GPUs\)\. Our contributions are summarized as follows\.

- •*An inverse\-free LSTD critic\.*Our proposed inverse\-free critic can approximate the ridge\-regularized pathwise LSTD solution with GEMMs alone for every ridge parameterλ\>0\\lambda\>0and every frame lengthTT; therefore, does not require stepsize, knowledge of the mixing time, and rank condition on the actor parameter\. Theorem[4](https://arxiv.org/html/2610.33477#Thmtheorem4)quantifies both sources of critic error: the NSI error decays superlinearly in the numberP1P\_\{1\}of NSI recursions, and the mean\-square Markovian sampling error is of orderτmix/T\\nicefrac\{\{\\tau\_\{\\mathrm\{mix\}\}\}\}\{\{T\}\}\.
- •*Generic bounds over a single Markovian trajectory\.*Conditioned on the start state of each frame and the lagged critic, the noise of the PG estimators forms a martingale\-difference sequence across frames, and their bias has two parts: the stationary\-to\-discounted mismatch \(that enters the convergence rate only once due to the cancellation of importance correction between consecutive iterates\) and the frame bias \(which is of order1/T\\nicefrac\{\{1\}\}\{\{T\}\}and varies with the actor parameter at a rate of order1/T\\nicefrac\{\{1\}\}\{\{T\}\}\)\. With such decomposition, every actor update inherits a finite\-time convergence rate in the nuclear norm: an𝒪\(K−1/3\)\{\\cal O\}\(K^\{\-1/3\}\)transient for normalized actor updates and an𝒪\(K−1/4\)\{\\cal O\}\(K^\{\-1/4\}\)transient for unnormalized ones \(Theorems[11](https://arxiv.org/html/2610.33477#Thmtheorem11)and[12](https://arxiv.org/html/2610.33477#Thmtheorem12)\), plus a floor formed by the mismatch and the frame bias\.
- •*Instances and comparisons\.*We apply our theoretical results to normalized actor updates, such as Muon and normalized SGD with momentum \(NSGDM\), and to unnormalized ones, such as SGD with momentum \(SGDM\) and an adaptive\-stepsize actor \(AdaNorm\)\. We establish that the Muon actor update retains at least78%78\\%of the first\-order progress of the exact polar step along the momentum without a conditioning assumption after𝒪⁡\(log⁡r\)\{\\cal O\}\(\\log r\)NSO recursions with the actor parameter rankr≤min⁡\{m,n\}r\\leq\\min\\\{m,n\\\}\. Comparing the instances, we identify two effects: normalization alone explains the faster rate, and orthogonalization with a dimension\-free stepsize confines the factorr1/2r^\{1/2\}to the noise and correction terms\. We also characterize when orthogonalization guarantees a larger noiseless ascent than normalization: exactly when the nuclear rank of the gradient exceeds the ratio of the spectral and Frobenius curvature constants \(Section[5\.4](https://arxiv.org/html/2610.33477#S5.SS4)\)\.

To our knowledge, Theorems[11](https://arxiv.org/html/2610.33477#Thmtheorem11)and[12](https://arxiv.org/html/2610.33477#Thmtheorem12)and their instances are the first finite\-time convergence rates for ICM actors along a single Markovian trajectory with a bootstrapped critic\. The floor of our bounds isolates the stationary\-to\-discounted mismatch of uniformly weighted samples, a bias that practical AC implementations share\([Thomas, 2014](https://arxiv.org/html/2610.33477#bib.bib59);[Nota and Thomas, 2020](https://arxiv.org/html/2610.33477#bib.bib41);[Zhang et al\., 2022](https://arxiv.org/html/2610.33477#bib.bib73)\); every other floor term decays at least as1/T\\nicefrac\{\{1\}\}\{\{T\}\}\. Appendix[A](https://arxiv.org/html/2610.33477#A1)collects our notation and defines the stationarity criterion\.

## 2Problem Description

We consider a discounted Markov decision process\(𝒮,𝒜,ℙ,r,γ\)\(\{\\cal S\},\{\\cal A\},\\mathbb\{P\},r,\\gamma\)with a state space𝒮\{\\cal S\}, an action space𝒜\{\\cal A\}, an unknown transition kernel that maps each state\-action pair\(s,a\)\(s,a\)to a distributionℙ\(⋅\|s,a\)\\mathbb\{P\}\(\\cdot\|s,a\)over𝒮\{\\cal S\}, a bounded rewardr:𝒮×𝒜→\[−Rr,Rr\]r:\{\\cal S\}\\times\{\\cal A\}\\rightarrow\[\-R\_\{r\},R\_\{r\}\], and a discount factorγ∈\(0,1\)\\gamma\\in\(0,1\)\. A policyπW\\pi\_\{W\}, parameterized by a matrixW∈ℝm×nW\\in\\mathbb\{R\}^\{m\\times n\}, maps each statessto a distributionπW\(⋅\|s\)\\pi\_\{W\}\(\\cdot\|s\)over𝒜\{\\cal A\}\. Under parametric policyπW\\pi\_\{W\}, the state\-value function and the*Q*\-function are

VW​\(s\)=𝔼W​\[∑t=0∞γt​r​\(st,at\)\|s0=s\]QW​\(s,a\)=𝔼W\[∑t=0∞γtr\(st,at\)\|s0=s,a0=a\]=r\(s,a\)\+γ𝔼\[VW\(s′\)\]\\begin\{split\}V\_\{W\}\(s\)&=\\mathbb\{E\}\_\{W\}\\Big\[\\sum\\nolimits\_\{t=0\}^\{\\infty\}\\gamma^\{t\}r\(s\_\{t\},a\_\{t\}\)\\,\\Big\|\\,s\_\{0\}=s\\Big\]\\\\ Q\_\{W\}\(s,a\)&=\\mathbb\{E\}\_\{W\}\\Big\[\\sum\\nolimits\_\{t=0\}^\{\\infty\}\\gamma^\{t\}r\(s\_\{t\},a\_\{t\}\)\\,\\Big\|\\,s\_\{0\}=s,a\_\{0\}=a\\Big\]=r\(s,a\)\+\\gamma\\mathbb\{E\}\[V\_\{W\}\(s^\{\\prime\}\)\]\\end\{split\}\(1\)whereat∼πW\(⋅\|st\)a\_\{t\}\\sim\\pi\_\{W\}\(\\cdot\|s\_\{t\}\),st\+1∼ℙ\(⋅\|st,at\)s\_\{t\+1\}\\sim\\mathbb\{P\}\(\\cdot\|s\_\{t\},a\_\{t\}\), and the last equality is the Bellman equation withs′∼ℙ\(⋅\|s,a\)s^\{\\prime\}\\sim\\mathbb\{P\}\(\\cdot\|s,a\)\.

Policy gradient theorem\.Given an initial\-state distributionρ\\rho, our objective is to maximize the normalized expected discounted returnJ⁡\(W\):=\(1−γ\)​𝔼s∼ρ​\[VW​\(s\)\]J\(W\):=\(1\-\\gamma\)\\,\\mathbb\{E\}\_\{s\\sim\\rho\}\[V\_\{W\}\(s\)\], whose gradient is given by the policy gradient theorem\([Sutton and Barto, 2018](https://arxiv.org/html/2610.33477#bib.bib57)\)as

∇J​\(W\)=𝔼\[\[QW\(st,at\)−VW\(st\)\]∇logπW\(at\|st\)\]=𝔼\[∑l=0T−1γlδW,t\+l∇logπW\(at\|st\)\]\\begin\{split\}\\nabla J\(W\)&=\\mathbb\{E\}\\Big\[\[Q\_\{W\}\(s\_\{t\},a\_\{t\}\)\-V\_\{W\}\(s\_\{t\}\)\]\\nabla\\log\\pi\_\{W\}\(a\_\{t\}\|s\_\{t\}\)\\Big\]\\\\ &=\\mathbb\{E\}\\Big\[\\sum\\nolimits\_\{l=0\}^\{T\-1\}\\gamma^\{l\}\\delta\_\{W,t\+l\}\\nabla\\log\\pi\_\{W\}\(a\_\{t\}\|s\_\{t\}\)\\Big\]\\end\{split\}\(2\)wherests\_\{t\}is drawn from the normalized discounted state\-visitation distributionζW\(⋅\):=\(1−γ\)∑t≥0γtℙ\(st∈⋅\|s0∼ρ,πW\)\\zeta\_\{W\}\(\\cdot\):=\(1\-\\gamma\)\\sum\_\{t\\geq 0\}\\gamma^\{t\}\\mathbb\{P\}\(s\_\{t\}\\in\\cdot\|s\_\{0\}\\sim\\rho,\\pi\_\{W\}\), the subsequent actions and states are generated byπW⊗ℙ\\pi\_\{W\}\\otimes\\mathbb\{P\}, and the one\-step TD error isδW,t:=r⁡\(st,at\)\+γ​VW​\(st\+1\)−VW​\(st\)\\delta\_\{W,t\}:=r\(s\_\{t\},a\_\{t\}\)\+\\gamma V\_\{W\}\(s\_\{t\+1\}\)\-V\_\{W\}\(s\_\{t\}\)\.

The second equality in \([2](https://arxiv.org/html/2610.33477#S2.E2)\) follows from the three facts: \(i\)𝔼\[δW,t\|st,at\]=QW\(st,at\)−VW\(st\)\\mathbb\{E\}\[\\delta\_\{W,t\}\|s\_\{t\},a\_\{t\}\]=Q\_\{W\}\(s\_\{t\},a\_\{t\}\)\-V\_\{W\}\(s\_\{t\}\)by leveraging definitions in \([1](https://arxiv.org/html/2610.33477#S2.E1)\); \(ii\)𝔼⁡\[δW,t\+l\|st\+l\]=0\\mathbb\{E\}\[\\delta\_\{W,t\+l\}\|s\_\{t\+l\}\]=0and𝔼\[δW,t\+l\|st,at\]=0\\mathbb\{E\}\[\\delta\_\{W,t\+l\}\|s\_\{t\},a\_\{t\}\]=0for alll≥1l\\geq 1by Markov property; and \(iii\) the score∇log⁡πW​\(at\|st\)\\nabla\\log\\pi\_\{W\}\(a\_\{t\}\|s\_\{t\}\)is a function of state\-action pair\(st,at\)\(s\_\{t\},a\_\{t\}\)\. Note that the state\-visitation distributionζW\\zeta\_\{W\}differs in general from the stationary distributionμW\\mu\_\{W\}of theπW\\pi\_\{W\}\-induced Markov chain\.

Linear function approximation\.Since the advantage in \([2](https://arxiv.org/html/2610.33477#S2.E2)\) is unavailable in the model\-free setting, a linear critic is used to approximate the state\-value function as

Vv​\(s\)=ϕ⊤​\(s\)​v≈VW​\(s\)V\_\{v\}\(s\)=\\phi^\{\\top\}\(s\)v\\approx V\_\{W\}\(s\)\(3\)where the feature embeddingϕ:𝒮→ℝd\\phi:\{\\cal S\}\\rightarrow\\mathbb\{R\}^\{d\}satisfies‖ϕ⁡\(s\)‖2≤1\\\|\\phi\(s\)\\\|\_\{2\}\\leq 1\.

Taking the expectation of the last equality in \([1](https://arxiv.org/html/2610.33477#S2.E1)\) overa∼πW\(⋅\|s\)a\\sim\\pi\_\{W\}\(\\cdot\|s\)and replacingVWV\_\{W\}withVvV\_\{v\}, we obtain the Bellman equation of state value as

ϕ⊤​\(s\)​v=𝔼⁡\[r⁡\(s,a\)\+γ​ϕ⊤​\(s′\)​v\|s\]\\phi^\{\\top\}\(s\)v=\\mathbb\{E\}\[r\(s,a\)\+\\gamma\\phi^\{\\top\}\(s^\{\\prime\}\)v\|s\]\(4\)where the expectation is taken overπW⊗ℙ\\pi\_\{W\}\\otimes\\mathbb\{P\}\.

Note that the Bellman equation of state value \([4](https://arxiv.org/html/2610.33477#S2.E4)\) imposes one equation per state onddunknowns and has a solution if and only ifVWV\_\{W\}lies in the function class of the critic\. Hence, we fix the targetr⁡\(s,a\)\+γ​ϕ⊤​\(s′\)​v∗r\(s,a\)\+\\gamma\\phi^\{\\top\}\(s^\{\\prime\}\)v^\{\*\}and require the critic parameterv∗v^\{\*\}to minimize the ridge\-regularized mean squared error as

v∗=arg​minu∈ℝd⁡𝔼​\[\(ϕ⊤​\(s\)​u−r⁡\(s,a\)−γ​ϕ⊤​\(s′\)​v∗\)2\]\+λ​‖u‖22v^\{\*\}=\\argmin\_\{u\\in\\mathbb\{R\}^\{d\}\}\\;\\mathbb\{E\}\[\(\\phi^\{\\top\}\(s\)u\-r\(s,a\)\-\\gamma\\phi^\{\\top\}\(s^\{\\prime\}\)v^\{\*\}\)^\{2\}\]\+\\lambda\\\|u\\\|\_\{2\}^\{2\}\(5\)whereλ\>0\\lambda\>0is the ridge parameter and the observationo=\(s,a,s′\)∼μW⊗πW⊗ℙo=\(s,a,s^\{\\prime\}\)\\sim\\mu\_\{W\}\\otimes\\pi\_\{W\}\\otimes\\mathbb\{P\}\.

Since the targetr⁡\(s,a\)\+γ​ϕ⊤​\(s′\)​v∗r\(s,a\)\+\\gamma\\phi^\{\\top\}\(s^\{\\prime\}\)v^\{\*\}is independent ofuu, the objective in \([5](https://arxiv.org/html/2610.33477#S2.E5)\) is a quadratic function ofuuwith positive definite Hessian2​\(Σϕ\+λ​Id\)2\(\\Sigma\_\{\\phi\}\+\\lambda I\_\{d\}\)due to the positive semidefiniteness ofΣϕ:=𝔼⁡\[ϕ⁡\(s\)​ϕ⊤​\(s\)\]\\Sigma\_\{\\phi\}:=\\mathbb\{E\}\[\\phi\(s\)\\phi^\{\\top\}\(s\)\]\. For a fixed target, the objective is strongly convex inuuand has a unique minimizer\. Thus, the optimal criticv∗v^\{\*\}solves \([5](https://arxiv.org/html/2610.33477#S2.E5)\) if and only if it satisfies the first\-order optimality condition atu=v∗u=v^\{\*\}, i\.e\., the ridge\-regularized Bellman equation

𝔼⁡\[δv∗​\(o\)​ϕ​\(s\)\]−λ​v∗=b−\(A\+λ​Id\)​v∗=0\\mathbb\{E\}\[\\delta\_\{v^\{\*\}\}\(o\)\\phi\(s\)\]\-\\lambda v^\{\*\}=b\-\(A\+\\lambda I\_\{d\}\)v^\{\*\}=0\(6\)where the TD errorδv​\(o\):=r⁡\(s,a\)\+\[γ​ϕ​\(s′\)−ϕ⁡\(s\)\]⊤​v\\delta\_\{v\}\(o\):=r\(s,a\)\+\[\\gamma\\phi\(s^\{\\prime\}\)\-\\phi\(s\)\]^\{\\top\}v,A:=𝔼⁡\[ϕ⁡\(s\)​\(ϕ⁡\(s\)−γ​ϕ​\(s′\)\)⊤\]A:=\\mathbb\{E\}\[\\phi\(s\)\(\\phi\(s\)\-\\gamma\\phi\(s^\{\\prime\}\)\)^\{\\top\}\], andb:=𝔼⁡\[ϕ⁡\(s\)​r​\(s,a\)\]b:=\\mathbb\{E\}\[\\phi\(s\)r\(s,a\)\]\.

Note that \([6](https://arxiv.org/html/2610.33477#S2.E6)\) is the population ridge\-regularized pathwise LSTD system that will guide the design of the empirical LSTD system per each frame in Section[3\.1](https://arxiv.org/html/2610.33477#S3.SS1)\. It is worth stressing that \([6](https://arxiv.org/html/2610.33477#S2.E6)\) withλ=0\\lambda=0recovers the projected Bellman equation\([Tsitsiklis and Roy, 1997](https://arxiv.org/html/2610.33477#bib.bib60)\), whose empirical counterpart is the well\-known LSTD critic estimator\([Bradtke and Barto, 1996](https://arxiv.org/html/2610.33477#bib.bib7);[Lagoudakis and Parr, 2003](https://arxiv.org/html/2610.33477#bib.bib34)\)\. Since the successor states′s^\{\\prime\}shares the distributionμW\\mu\_\{W\}with the current statess, we obtainx⊤​\(A\+λ​Id\)​x≥\(1−γ\)​x⊤​Σϕ​x\+λ​‖x‖22\>0x^\{\\top\}\(A\+\\lambda I\_\{d\}\)x\\geq\(1\-\\gamma\)x^\{\\top\}\\Sigma\_\{\\phi\}x\+\\lambda\\\|x\\\|\_\{2\}^\{2\}\>0for allx∈ℝdx\\in\\mathbb\{R\}^\{d\}\([Bhandari et al\., 2021](https://arxiv.org/html/2610.33477#bib.bib5), Lemma 3\)\. Therefore, the ridge\-regularized Bellman equation \([6](https://arxiv.org/html/2610.33477#S2.E6)\) and the fixed\-point problem \([5](https://arxiv.org/html/2610.33477#S2.E5)\) have the unique solutionv∗​\(W\):=\(A\+λ​Id\)−1​bv^\{\*\}\(W\):=\(A\+\\lambda I\_\{d\}\)^\{\-1\}bwith‖v∗​\(W\)‖2≤‖b‖2/λ≤Rr/λ\\\|v^\{\*\}\(W\)\\\|\_\{2\}\\leq\\nicefrac\{\{\\\|b\\\|\_\{2\}\}\}\{\{\\lambda\}\}\\leq\\nicefrac\{\{R\_\{r\}\}\}\{\{\\lambda\}\}\.

## 3Generic ICM\-AC over a Single Markovian Trajectory

The critic and the actor learn from a single trajectory of the policy\-induced Markov chain that consists of consecutive frames of a fixed lengthT≥1T\\geq 1, one per update, forKKupdates\. More specifically, framekkis theTT\-step trajectoryOk=\[ok,t\]t=0T−1O\_\{k\}=\[o\_\{k,t\}\]\_\{t=0\}^\{T\-1\}with the observationsok,t=\(sk,t,ak,t,sk,t\+1\)o\_\{k,t\}=\(s\_\{k,t\},a\_\{k,t\},s\_\{k,t\+1\}\)that are generated by the current policyπk:=πWk\\pi\_\{k\}:=\\pi\_\{W\_\{k\}\}from the last state of the previous frame, i\.e\.,sk,0=sk−1,Ts\_\{k,0\}=s\_\{k\-1,T\}fork≥1k\\geq 1\. The warm\-up frameO0O\_\{0\}is generated byπ1\\pi\_\{1\}froms0,0∼ρs\_\{0,0\}\\sim\\rho\. Based on theTT\-step trajectoryOkO\_\{k\}, Sections[3\.1](https://arxiv.org/html/2610.33477#S3.SS1)–[3\.3](https://arxiv.org/html/2610.33477#S3.SS3)develop the inverse\-free ridge\-regularized pathwise LSTD critic, the PG estimator, and the ICM of the actor, and the generic ICM\-AC algorithm, respectively\.

### 3\.1Inverse\-Free Ridge\-Regularized Pathwise LSTD Critic

The empirical estimates ofAAandbbareA^k=Φk​Yk⊤\\hat\{A\}\_\{k\}=\\Phi\_\{k\}Y\_\{k\}^\{\\top\}andb^k=Φk​rk\\hat\{b\}\_\{k\}=\\Phi\_\{k\}r\_\{k\}withYk=Φk−γ​Φk′Y\_\{k\}=\\Phi\_\{k\}\-\\gamma\\Phi\_\{k\}^\{\\prime\},Φk=T−1/2\[ϕk,t\]t=0T−1∈ℝd×T\\Phi\_\{k\}=T^\{\-1/2\}\[\\phi\_\{k,t\}\]\_\{t=0\}^\{T\-1\}\\in\\mathbb\{R\}^\{d\\times T\},Φk′=T−1/2\[ϕk,t𝟏\{t<T\}\]t=1T\\Phi\_\{k\}^\{\\prime\}=T^\{\-1/2\}\[\\phi\_\{k,t\}\\mathbf\{1\}\\\{t<T\\\}\]\_\{t=1\}^\{T\}, andrk=T−1/2\[rk,t\]t=0T−1∈ℝTr\_\{k\}=T^\{\-1/2\}\[r\_\{k,t\}\]\_\{t=0\}^\{T\-1\}\\in\\mathbb\{R\}^\{T\}, whereϕk,t=ϕ⁡\(sk,t\)\\phi\_\{k,t\}=\\phi\(s\_\{k,t\}\)andrk,t=r⁡\(sk,t,ak,t\)r\_\{k,t\}=r\(s\_\{k,t\},a\_\{k,t\}\)\. Note that the last column ofΦk′\\Phi\_\{k\}^\{\\prime\}drops the bootstrap of statesk,Ts\_\{k,T\}of the frame such that the LSTD system involves only the featuresϕk,0,…,ϕk,T−1\\phi\_\{k,0\},\\ldots,\\phi\_\{k,T\-1\}\. Hence, for each positive ridge parameterλ\>0\\lambda\>0, the matricesA^k\+λ​Id\\hat\{A\}\_\{k\}\+\\lambda I\_\{d\}andHk:=Yk⊤​Φk\+λ​ITH\_\{k\}:=Y\_\{k\}^\{\\top\}\\Phi\_\{k\}\+\\lambda I\_\{T\}are non\-singular with proofs relegated to Appendix[B](https://arxiv.org/html/2610.33477#A2)\.

Replacing the population quantitiesAAandbbin \([6](https://arxiv.org/html/2610.33477#S2.E6)\) withA^k\\hat\{A\}\_\{k\}andb^k\\hat\{b\}\_\{k\}, we obtain the empirical ridge\-regularized pathwise LSTD solution as

v^k=\(A^k\+λ​Id\)−1​b^k=Φk​Hk−1​rk\\hat\{v\}\_\{k\}=\(\\hat\{A\}\_\{k\}\+\\lambda I\_\{d\}\)^\{\-1\}\\hat\{b\}\_\{k\}=\\Phi\_\{k\}H\_\{k\}^\{\-1\}r\_\{k\}\(7\)where the last equality is the push\-through identity\([Boyd and Vandenberghe, 2018](https://arxiv.org/html/2610.33477#bib.bib6), p\. 218\)with the non\-singularHkH\_\{k\}\.

Different from a semi\-gradient TD\(0\) with weight decayλ\\lambda, which targets the samev∗v^\{\*\}by taking only one stochastic\-approximation step per transition, the ridge\-regularized pathwise LSTD critic solves the empirical optimalv^k\\hat\{v\}\_\{k\}of a frame in closed form to avoid critic stepsize tuning and the slow transient of stochastic updates\. Hence, the reducedT×TT\\times Tsystem lowers the cost of forming and factoring the LSTD system from𝒪⁡\(d2​T\+d3\)\{\\cal O\}\(d^\{2\}T\+d^\{3\}\)to𝒪⁡\(d​T2\+T3\)\{\\cal O\}\(dT^\{2\}\+T^\{3\}\)operations which is a significantly saving whenT≪dT\\ll d\. Since factoringHkH\_\{k\}uses GPUs far less efficiently than GEMMs, we choose to approximateHk−1H\_\{k\}^\{\-1\}by the GEMM\-only NSI recursion, which runs close to the peak throughput of GPUs, as

Ek,q=IT−Hk​Xk,q,Xk,q\+1=Xk,q​∑i=0κ1−1Ek,qi,vk,q=Φk​Xk,q​rkE\_\{k,q\}=I\_\{T\}\-H\_\{k\}X\_\{k,q\},\\kern 5\.0ptX\_\{k,q\+1\}=X\_\{k,q\}\\sum\\nolimits\_\{i=0\}^\{\\kappa\_\{1\}\-1\}E\_\{k,q\}^\{i\},\\kern 5\.0ptv\_\{k,q\}=\\Phi\_\{k\}X\_\{k,q\}r\_\{k\}\(8\)whereκ1≥2\\kappa\_\{1\}\\geq 2is the NSI order,Xk,0=Hk⊤/ck2X\_\{k,0\}=\\nicefrac\{\{H\_\{k\}^\{\\top\}\}\}\{\{c\_\{k\}^\{2\}\}\}with the safe scalingck2:=‖Hk‖1​‖Hk‖∞c\_\{k\}^\{2\}:=\\\|H\_\{k\}\\\|\_\{1\}\\\|H\_\{k\}\\\|\_\{\\infty\}of[Pan and Schreiber \(1991\)](https://arxiv.org/html/2610.33477#bib.bib43), and∥⋅∥1\\\|\\cdot\\\|\_\{1\}and∥⋅∥∞\\\|\\cdot\\\|\_\{\\infty\}are the maximum absolute column and row sums\. AfterP1P\_\{1\}NSI recursions, we obtain the inverse\-free LSTD critic of framekkasvk:=vk,P1v\_\{k\}:=v\_\{k,P\_\{1\}\}\.

### 3\.2Policy Gradient and Importance\-Corrected Momentum

For a critic parametervv, the discounted sum of the one\-step TD errors from stepttto the end of framekktelescopes into the bootstrapped advantage estimate as

δ^k,t​\(v\):=∑l=0T−t−1γl​δv​\(ok,t\+l\)=∑l=0T−t−1γl​rk,t\+l\+γT−t​ϕk,T⊤​v−ϕk,t⊤​v\.\\hat\{\\delta\}\_\{k,t\}\(v\):=\\sum\\nolimits\_\{l=0\}^\{T\-t\-1\}\\gamma^\{l\}\\delta\_\{v\}\(o\_\{k,t\+l\}\)=\\sum\\nolimits\_\{l=0\}^\{T\-t\-1\}\\gamma^\{l\}r\_\{k,t\+l\}\+\\gamma^\{T\-t\}\\phi\_\{k,T\}^\{\\top\}v\-\\phi\_\{k,t\}^\{\\top\}v\.\(9\)
For an actor parameterWW, the PG estimator on trajectoryOkO\_\{k\}is

g^\(W,v;Ok\):=1T∑t=0T−1δ^k,t\(v\)∇logπW\(ak,t\|sk,t\)\.\\hat\{g\}\(W,v;O\_\{k\}\):=\\frac\{1\}\{T\}\\sum\\nolimits\_\{t=0\}^\{T\-1\}\\hat\{\\delta\}\_\{k,t\}\(v\)\\,\\nabla\\log\\pi\_\{W\}\(a\_\{k,t\}\|s\_\{k,t\}\)\.\(10\)
At framek∈\[K\]k\\in\[K\], the actor evaluatesOkO\_\{k\}atWkW\_\{k\}with the lagged criticvk−1v\_\{k\-1\}, i\.e\., the PG estimator of framekkisGk:=g^​\(Wk,vk−1,Ok\)G\_\{k\}:=\\hat\{g\}\(W\_\{k\},v\_\{k\-1\};O\_\{k\}\)\. The critic error entersGkG\_\{k\}in two ways: through the baseline terms−ϕk,t⊤​vk−1\-\\phi\_\{k,t\}^\{\\top\}v\_\{k\-1\}, which have zero conditional mean because the lagged criticvk−1v\_\{k\-1\}is fixed during framekkand the score has zero mean givensk,ts\_\{k,t\}, and through the bootstrapγT−t​ϕk,T⊤​vk−1\\gamma^\{T\-t\}\\phi\_\{k,T\}^\{\\top\}v\_\{k\-1\}at the end of the frame, whose weights average toγ⁡\(1−γT\)/\[\(1−γ\)​T\]\\nicefrac\{\{\\gamma\(1\-\\gamma^\{T\}\)\}\}\{\{\[\(1\-\\gamma\)T\]\}\}over the frame\.

A plain momentumMk=η​Mk−1\+\(1−η\)​GkM\_\{k\}=\\eta M\_\{k\-1\}\+\(1\-\\eta\)G\_\{k\}averagesG1,…,GkG\_\{1\},\\ldots,G\_\{k\}at the past iterates and thus lags behind∇J​\(Wk\)\\nabla J\(W\_\{k\}\)by a drift that accumulates the changes∇J​\(Wj\)−∇J​\(Wj−1\)\\nabla J\(W\_\{j\}\)\-\\nabla J\(W\_\{j\-1\}\)over its memory1/\(1−η\)\\nicefrac\{\{1\}\}\{\{\(1\-\\eta\)\}\}\. Following STORM\([Cutkosky and Orabona, 2019](https://arxiv.org/html/2610.33477#bib.bib12)\), we cancel this drift by adding to the momentum an estimate of the change of the estimator’s mean fromWk−1W\_\{k\-1\}toWkW\_\{k\}over the same trajectoryOkO\_\{k\}\. SinceOkO\_\{k\}is sampled fromπk\\pi\_\{k\}rather thanπk−1\\pi\_\{k\-1\}, we reweightg^​\(Wk−1,vk−1,Ok\)\\hat\{g\}\(W\_\{k\-1\},v\_\{k\-1\};O\_\{k\}\)by the importance weightwkw\_\{k\}\([Huang et al\., 2020](https://arxiv.org/html/2610.33477#bib.bib21)\)as

G~k:=wk​g^​\(Wk−1,vk−1,Ok\),wk:=∏t=0T−1πk−1​\(ak,t\|sk,t\)πk​\(ak,t\|sk,t\)\\widetilde\{G\}\_\{k\}:=w\_\{k\}\\,\\hat\{g\}\(W\_\{k\-1\},v\_\{k\-1\};O\_\{k\}\),\\kern 5\.0ptw\_\{k\}:=\\prod\\nolimits\_\{t=0\}^\{T\-1\}\\frac\{\\pi\_\{k\-1\}\(a\_\{k,t\}\|s\_\{k,t\}\)\}\{\\pi\_\{k\}\(a\_\{k,t\}\|s\_\{k,t\}\)\}\(11\)where the start state and the transition probabilities are common to both policies such thatwkw\_\{k\}requires no knowledge of transition kernelℙ\\mathbb\{P\}\. Moreover, since the actor parameter enters the PG estimator \([10](https://arxiv.org/html/2610.33477#S3.E10)\) only via the score, the reweighted PG estimatorG~k\\widetilde\{G\}\_\{k\}can reuse the advantage estimates ofGkG\_\{k\}\.

The ICM with momentum factorη∈\[0,1\)\\eta\\in\[0,1\)shifts the previous momentum by the importance correction before exponential averaging as

Mk=\{G1,k=1\(1−η\)​Gk\+η⁡\[Mk−1\+Gk−G~k\],k≥2M\_\{k\}=\\begin\{cases\}G\_\{1\},&k=1\\\\ \(1\-\\eta\)G\_\{k\}\+\\eta\[M\_\{k\-1\}\+G\_\{k\}\-\\widetilde\{G\}\_\{k\}\],&k\\geq 2\\end\{cases\}\(12\)whereη=0\\eta=0induces the vanilla SGD, and dropping the importance correction termGk−G~kG\_\{k\}\-\\widetilde\{G\}\_\{k\}yields the plain momentum\.

### 3\.3Normalized and Unnormalized ICM\-Based Actors

Letℱk:=σ⁡\(O0,O1,…,Ok\)\{\\cal F\}\_\{k\}:=\\sigma\(O\_\{0\},O\_\{1\},\\ldots,O\_\{k\}\)denote the history up to framekkwith trivialℱ−1\{\\cal F\}\_\{\-1\}\. Based on the ICMMkM\_\{k\}, we define the two classes of actor updates as follows\.

###### Definition 1\(Actor updates\)\.

Let∥⋅∥\\\|\\cdot\\\|be a norm onℝm×n\\mathbb\{R\}^\{m\\times n\}with∥⋅∥≥∥⋅∥2\\\|\\cdot\\\|\\geq\\\|\\cdot\\\|\_\{2\}, and let∥⋅∥†\\\|\\cdot\\\|\_\{\\dagger\}andκ†\\kappa\_\{\\dagger\}be its dual norm and the constant of Appendix[A](https://arxiv.org/html/2610.33477#A1)\. An actor updateWk\+1=Wk\+Step⁡\(Mk\)W\_\{k\+1\}=W\_\{k\}\+\\mathrm\{Step\}\(M\_\{k\}\)whose output isℱk\{\\cal F\}\_\{k\}\-measurable is

- \(i\)*normalized*in∥⋅∥\\\|\\cdot\\\|, with stepsizeβ\>0\\beta\>0and alignment factorχ≥1\\chi\\geq 1, ifStep⁡\(Mk\)=β​Nk\\mathrm\{Step\}\(M\_\{k\}\)=\\beta N\_\{k\}with‖Nk‖≤1\\\|N\_\{k\}\\\|\\leq 1and⟨Mk,Nk⟩F≥χ−1​‖Mk‖†\\langle M\_\{k\},N\_\{k\}\\rangle\_\{\\scriptscriptstyle\\textrm\{F\}\}\\geq\\chi^\{\-1\}\\\|M\_\{k\}\\\|\_\{\\dagger\}for allk∈\[K\]k\\in\[K\];
- \(ii\)*unnormalized*, with stepsize boundβ¯\>0\\bar\{\\beta\}\>0, rangeϑ≥1\\vartheta\\geq 1, and variationVu≥0V\_\{u\}\\geq 0, ifStep⁡\(Mk\)=βk​Mk\\mathrm\{Step\}\(M\_\{k\}\)=\\beta\_\{k\}M\_\{k\}withℱk\{\\cal F\}\_\{k\}\-measurableβk∈\[β¯/ϑ,β¯\]\\beta\_\{k\}\\in\[\\nicefrac\{\{\\bar\{\\beta\}\}\}\{\{\\sqrt\{\\vartheta\}\}\},\\bar\{\\beta\}\]for allk∈\[K\]k\\in\[K\]and∑k=2K\|βk−1−βk−1−1\|≤Vu/β¯\\sum\_\{k=2\}^\{K\}\|\\beta\_\{k\}^\{\-1\}\-\\beta\_\{k\-1\}^\{\-1\}\|\\leq\\nicefrac\{\{V\_\{u\}\}\}\{\{\\bar\{\\beta\}\}\}for every realization\.

Since‖Nk‖2≤‖Nk‖≤1\\\|N\_\{k\}\\\|\_\{2\}\\leq\\\|N\_\{k\}\\\|\\leq 1, a normalized actor update moves by at mostβ\\betain the spectral norm irrespective of the momentum, whereas an unnormalized actor update scales with the momentum\. Note that the alignment factor isχ=1\\chi=1for an exact steepest\-ascent directionNk∈arg​max‖N‖≤1⁡⟨Mk,N⟩FN\_\{k\}\\in\\argmax\_\{\\\|N\\\|\\leq 1\}\\langle M\_\{k\},N\\rangle\_\{\\scriptscriptstyle\\textrm\{F\}\}, i\.e\., the polar factor ofMkM\_\{k\}in the spectral norm, which keeps the singular vectors ofMkM\_\{k\}and sets all nonzero singular values to one, andMk/‖Mk‖F\\nicefrac\{\{M\_\{k\}\}\}\{\{\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\}\}in the Frobenius norm\.

Table 1:Actor updates of Algorithm[1](https://arxiv.org/html/2610.33477#alg1)as instances of Definition[1](https://arxiv.org/html/2610.33477#Thmtheorem1)\.Herer=min⁡\{m,n\}r=\\min\\\{m,n\\\},sP2s\_\{P\_\{2\}\}is given in Lemma[15](https://arxiv.org/html/2610.33477#Thmtheorem15),RgR\_\{g\}in Lemma[5](https://arxiv.org/html/2610.33477#Thmtheorem5), and the AdaNorm constants requireα≤1/\(2​K\)\\alpha\\leq\\nicefrac\{\{1\}\}\{\{\(2K\)\}\}\(Section[5\.3](https://arxiv.org/html/2610.33477#S5.SS3)\)\.

Table[1](https://arxiv.org/html/2610.33477#S3.T1)summarizes the four actor updates that are investigated in Section[5](https://arxiv.org/html/2610.33477#S5)\. More specifically, the Muon actor update\([Jordan et al\., 2024](https://arxiv.org/html/2610.33477#bib.bib22)\)appliesP2P\_\{2\}NSO recursions to the momentum as

Nk,0=Mk‖Mk‖F,Nk,p=Nk,p−1​ψ​\(Nk,p−1\),p∈\[P2\],Step⁡\(Mk\)=β​Nk,P2N\_\{k,0\}=\\frac\{M\_\{k\}\}\{\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\},\\qquad N\_\{k,p\}=N\_\{k,p\-1\}\\,\\psi\(N\_\{k,p\-1\}\),\\ p\\in\[P\_\{2\}\],\\qquad\\mathrm\{Step\}\(M\_\{k\}\)=\\beta N\_\{k,P\_\{2\}\}\(13\)whereψ⁡\(N\):=∑i=0κ2−14−i​\(2​ii\)​\(In−N⊤​N\)i\\psi\(N\):=\\sum\_\{i=0\}^\{\\kappa\_\{2\}\-1\}4^\{\-i\}\\binom\{2i\}\{i\}\(I\_\{n\}\-N^\{\\top\}N\)^\{i\}is theκ2\\kappa\_\{2\}th\-order NSO polynomial withκ2≥2\\kappa\_\{2\}\\geq 2, andNk,p=0N\_\{k,p\}=0for allppifMk=0M\_\{k\}=0\. The NSGDM stepβ​Mk/‖Mk‖F\\nicefrac\{\{\\beta M\_\{k\}\}\}\{\{\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\}\}normalizes the momentum in the Frobenius norm\([Cutkosky and Mehta, 2020](https://arxiv.org/html/2610.33477#bib.bib11)\)and is zero ifMk=0M\_\{k\}=0, and the SGDM step isβ​Mk\\beta M\_\{k\}\. Moreover, the AdaNorm step scales the momentum by the inverse square root of a running average of the squared gradient norms as

nk=\(1−α\)​nk−1\+α​‖Gk‖F2,Step⁡\(Mk\)=βnk​Mkn\_\{k\}=\(1\-\\alpha\)n\_\{k\-1\}\+\\alpha\\\|G\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\},\\qquad\\mathrm\{Step\}\(M\_\{k\}\)=\\frac\{\\beta\}\{\\sqrt\{n\_\{k\}\}\}M\_\{k\}\(14\)wheren0\>0n\_\{0\}\>0andα∈\[0,1\]\\alpha\\in\[0,1\]\.

Algorithm 1Generic ICM\-AC over a Single Markovian Trajectory1:Initialize:actor

W1W\_\{1\}, frame length

T≥1T\\geq 1, ridge parameter

λ\>0\\lambda\>0, number

P1P\_\{1\}and order

κ1≥2\\kappa\_\{1\}\\geq 2of NSI recursions, momentum factor

η∈\[0,1\)\\eta\\in\[0,1\), and actor update

Step⁡\(⋅\)\\mathrm\{Step\}\(\\cdot\)\(Definition[1](https://arxiv.org/html/2610.33477#Thmtheorem1)\)

2:Roll out the warm\-up frame

O0O\_\{0\}of

TTtransitions via

π1\\pi\_\{1\}from

s0,0∼ρs\_\{0,0\}\\sim\\rho
3:Run

P1P\_\{1\}NSI recursions \([8](https://arxiv.org/html/2610.33477#S3.E8)\) on

O0O\_\{0\}and set the critic

v0:=v0,P1v\_\{0\}:=v\_\{0,P\_\{1\}\}
4:for

k=1,2,…,Kk=1,2,\\ldots,Kdo

5:Roll out the frame

OkO\_\{k\}of

TTtransitions via

πk\\pi\_\{k\}from

sk,0=sk−1,Ts\_\{k,0\}=s\_\{k\-1,T\}
6:Compute

Gk=g^​\(Wk,vk−1,Ok\)G\_\{k\}=\\hat\{g\}\(W\_\{k\},v\_\{k\-1\};O\_\{k\}\)by \([10](https://arxiv.org/html/2610.33477#S3.E10)\)

7:if

k=1k=1then

8:Set

M1=G1M\_\{1\}=G\_\{1\}
9:else

10:Compute

wkw\_\{k\}and

G~k\\widetilde\{G\}\_\{k\}by \([11](https://arxiv.org/html/2610.33477#S3.E11)\) and

MkM\_\{k\}by \([12](https://arxiv.org/html/2610.33477#S3.E12)\)

11:endif

12:Update the actor

Wk\+1=Wk\+Step⁡\(Mk\)W\_\{k\+1\}=W\_\{k\}\+\\mathrm\{Step\}\(M\_\{k\}\)
13:Run

P1P\_\{1\}NSI recursions \([8](https://arxiv.org/html/2610.33477#S3.E8)\) on

OkO\_\{k\}and set the critic

vk:=vk,P1v\_\{k\}:=v\_\{k,P\_\{1\}\}
14:endfor

15:returncritic

vKv\_\{K\}and actor

WUW\_\{U\}with

UUdrawn uniformly from

\[K\]\[K\]\(in practice,

WK\+1W\_\{K\+1\}\)

Based on the frames of a single trajectory, Algorithm[1](https://arxiv.org/html/2610.33477#alg1)runs the inverse\-free LSTD critic \([8](https://arxiv.org/html/2610.33477#S3.E8)\), the PG estimator \([10](https://arxiv.org/html/2610.33477#S3.E10)\), and the ICM \([12](https://arxiv.org/html/2610.33477#S3.E12)\), and leaves the actor updateStep⁡\(Mk\)\\mathrm\{Step\}\(M\_\{k\}\)open\. The critic and the actor share the same sample trajectory such thatKKupdates consume\(K\+1\)​T\(K\+1\)Ttransitions\. Reservoir sampling implements the uniform selection of the returned iterateWUW\_\{U\}without storing the full sequence of iterates\. Since𝔼⁡\[‖∇J​\(WU\)‖†\]=1K​∑k=1K𝔼⁡\[‖∇J​\(Wk\)‖†\]\\mathbb\{E\}\[\\\|\\nabla J\(W\_\{U\}\)\\\|\_\{\\dagger\}\]=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\[\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\\dagger\}\], the bounds on the averaged gradient norm in Section[4](https://arxiv.org/html/2610.33477#S4)also hold for the expected gradient norm atWUW\_\{U\}\.

## 4Generic Finite\-Time Analysis

In this section, we establish the finite\-time analysis of Algorithm[1](https://arxiv.org/html/2610.33477#alg1)when all frames are consecutive segments of a single continuing trajectory such that the observations are correlated within and across frames\. More specifically, Sections[4\.1](https://arxiv.org/html/2610.33477#S4.SS1)–[4\.4](https://arxiv.org/html/2610.33477#S4.SS4)bound the critic error, split the bias of the PG estimator, bound the tracking error of the ICM, and turn the tracking error into the convergence rates of the actor, respectively\. Note that the actor parametersWkW\_\{k\}andWk−1W\_\{k\-1\}, the lagged criticvk−1v\_\{k\-1\}, and the start statesk,0s\_\{k,0\}areℱk−1\{\\cal F\}\_\{k\-1\}\-measurable; hence, conditionally onℱk−1\{\\cal F\}\_\{k\-1\},OkO\_\{k\}is aTT\-step trajectory ofπk\\pi\_\{k\}fromsk,0s\_\{k,0\}\.

###### Assumption 1

For each state\-action pair\(s,a\)∈𝒮×𝒜\(s,a\)\\in\{\\cal S\}\\times\{\\cal A\}, the policyπW\\pi\_\{W\}has a densityπW​\(a\|s\)\>0\\pi\_\{W\}\(a\|s\)\>0that is differentiable with respect to the actor parameterWW\. Moreover, the score function∇log⁡πW\\nabla\\log\\pi\_\{W\}satisfies

‖∇log⁡πW​\(a\|s\)‖∗≤Rπ\\displaystyle\\\|\\nabla\\log\\pi\_\{W\}\(a\|s\)\\\|\_\{\*\}\\leq R\_\{\\pi\}\(15a\)‖∇log⁡πW​\(a\|s\)−∇log⁡πW′​\(a\|s\)‖∗≤Lπ​‖W−W′‖2\\displaystyle\\\|\\nabla\\log\\pi\_\{W\}\(a\|s\)\\\!\-\\\!\\nabla\\log\\pi\_\{W^\{\\prime\}\}\(a\|s\)\\\|\_\{\*\}\\\!\\leq\\\!L\_\{\\pi\}\\\|W\-W^\{\\prime\}\\\|\_\{2\}\\\!\\\!\(15b\)whereWWandW′W^\{\\prime\}are arbitrary actor parameters\.

With the Frobenius norm in place of the nuclear and spectral norms in \([15](https://arxiv.org/html/2610.33477#S4.E15)\), Assumption[1](https://arxiv.org/html/2610.33477#Thmassump1)is standard in PG analyses\([Zou et al\., 2019](https://arxiv.org/html/2610.33477#bib.bib74);[Hong et al\., 2023](https://arxiv.org/html/2610.33477#bib.bib19);[Wang et al\., 2020](https://arxiv.org/html/2610.33477#bib.bib61);[Chen and Zhao, 2023](https://arxiv.org/html/2610.33477#bib.bib9);[Wu et al\., 2020](https://arxiv.org/html/2610.33477#bib.bib63);[Shen and Chen, 2022](https://arxiv.org/html/2610.33477#bib.bib53);[Olshevsky and Gharesifard, 2023](https://arxiv.org/html/2610.33477#bib.bib42);[Kumar et al\., 2023](https://arxiv.org/html/2610.33477#bib.bib32);[Bedi et al\., 2022](https://arxiv.org/html/2610.33477#bib.bib4);[Ganesh et al\., 2025](https://arxiv.org/html/2610.33477#bib.bib18), see, e\.g\.,\)and holds for, e\.g\., Cauchy, Boltzmann, and truncated Gaussian policies\([Konda and Borkar, 1999](https://arxiv.org/html/2610.33477#bib.bib29);[Bedi et al\., 2022](https://arxiv.org/html/2610.33477#bib.bib4);[Dong, Oct\. 2026](https://arxiv.org/html/2610.33477#bib.bib14)\)\. Assumption[1](https://arxiv.org/html/2610.33477#Thmassump1)guarantees that∇J\\nabla Jis Lipschitz continuous with proof relegated to Appendix[C](https://arxiv.org/html/2610.33477#A3)\.

###### Lemma 2\.

Under Assumption[1](https://arxiv.org/html/2610.33477#Thmassump1), the policy gradient∇J\\nabla Jis Lipschitz continuous as

‖∇J​\(W\)−∇J​\(W′\)‖∗≤L​‖W−W′‖2\\\|\\nabla J\(W\)\-\\nabla J\(W^\{\\prime\}\)\\\|\_\{\*\}\\leq L\\\|W\-W^\{\\prime\}\\\|\_\{2\}\(16\)whereWWandW′W^\{\\prime\}are arbitrary actor parameters withL:=Rr​Lπ/\(1−γ\)\+\(1\+γ\)​Rr​Rπ2/\(1−γ\)2L:=\\nicefrac\{\{R\_\{r\}L\_\{\\pi\}\}\}\{\{\(1\-\\gamma\)\}\}\+\\nicefrac\{\{\(1\+\\gamma\)R\_\{r\}R\_\{\\pi\}^\{2\}\}\}\{\{\(1\-\\gamma\)^\{2\}\}\}\.

Based on Lemma[2](https://arxiv.org/html/2610.33477#Thmtheorem2), applying the descent lemma\([Nesterov, 2018](https://arxiv.org/html/2610.33477#bib.bib40)\)to−J\-J, we obtain

⟨∇J​\(W\),W′−W⟩F≤J⁡\(W′\)−J⁡\(W\)\+12​L​‖W′−W‖22\.\\langle\\nabla J\(W\),W^\{\\prime\}\-W\\rangle\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq J\(W^\{\\prime\}\)\-J\(W\)\+\\frac\{1\}\{2\}L\\\|W^\{\\prime\}\-W\\\|\_\{2\}^\{2\}\.\(17\)
Note that \([17](https://arxiv.org/html/2610.33477#S4.E17)\) is the smoothness with respect to the spectral norm in the analyses of Muon, e\.g\.,[Kim and Oh \(2026\)](https://arxiv.org/html/2610.33477#bib.bib28)\. Since∥⋅∥2≤∥⋅∥F\\\|\\cdot\\\|\_\{2\}\\leq\\\|\\cdot\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}, \([17](https://arxiv.org/html/2610.33477#S4.E17)\) also holds withL/2​‖W′−W‖F2\\nicefrac\{\{L\}\}\{\{2\}\}\\\|W^\{\\prime\}\-W\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}in place ofL/2​‖W′−W‖22\\nicefrac\{\{L\}\}\{\{2\}\}\\\|W^\{\\prime\}\-W\\\|\_\{2\}^\{2\}\.

###### Assumption 2

The policy\-induced chains mix uniformly, i\.e\., the mixing time

τmix:=supWmin\{t≥1:sups∥ℙ\(st∈⋅\|s0=s,πW\)−μW∥TV≤1/4\}\\tau\_\{\\mathrm\{mix\}\}:=\\sup\\nolimits\_\{W\}\\min\\big\\\{t\\geq 1:\\sup\\nolimits\_\{s\}\\\|\\mathbb\{P\}\(s\_\{t\}\\in\\cdot\\,\|\\,s\_\{0\}=s,\\pi\_\{W\}\)\-\\mu\_\{W\}\\\|\_\{\\mathrm\{TV\}\}\\leq\\nicefrac\{\{1\}\}\{\{4\}\}\\big\\\}is finite, whereμW\\mu\_\{W\}is a stationary distribution of the chain induced byπW\\pi\_\{W\}\.

The uniform ergodicity in Assumption[2](https://arxiv.org/html/2610.33477#Thmassump2)is standard in finite\-time analyses of TD learning and AC methods under Markovian sampling\([Bhandari et al\., 2021](https://arxiv.org/html/2610.33477#bib.bib5);[Zou et al\., 2019](https://arxiv.org/html/2610.33477#bib.bib74);[Wu et al\., 2020](https://arxiv.org/html/2610.33477#bib.bib63)\)\. Since the mixing time is uniform in the start state, the bounds for a frame hold conditionally on the past although each frame starts from the last state of the previous frame\. Note that the mixing time is unknown to Algorithm[1](https://arxiv.org/html/2610.33477#alg1)and enters only the analysis\. Assumptions[1](https://arxiv.org/html/2610.33477#Thmassump1)and[2](https://arxiv.org/html/2610.33477#Thmassump2)also guarantee that the solution of \([6](https://arxiv.org/html/2610.33477#S2.E6)\) is Lipschitz continuous in the actor parameter with proof relegated to Appendix[E](https://arxiv.org/html/2610.33477#A5)\.

###### Lemma 3\.

Under Assumptions[1](https://arxiv.org/html/2610.33477#Thmassump1)and[2](https://arxiv.org/html/2610.33477#Thmassump2), the solutionv∗​\(W\):=\(AW\+λ​Id\)−1​bWv^\{\*\}\(W\):=\(A\_\{W\}\+\\lambda I\_\{d\}\)^\{\-1\}b\_\{W\}of \([6](https://arxiv.org/html/2610.33477#S2.E6)\) underπW\\pi\_\{W\}is Lipschitz continuous in the actor parameter as

‖v∗​\(W\)−v∗​\(W′\)‖2≤L∗​‖W−W′‖2\\\|v^\{\*\}\(W\)\-v^\{\*\}\(W^\{\\prime\}\)\\\|\_\{2\}\\leq L\_\{\*\}\\\|W\-W^\{\\prime\}\\\|\_\{2\}\(18\)whereWWandW′W^\{\\prime\}are arbitrary actor parameters withL∗:=\(4​τmix\+1\)​Rπ​Rr​\(λ\+1\+γ\)/λ2L\_\{\*\}:=\\nicefrac\{\{\(4\\tau\_\{\\mathrm\{mix\}\}\+1\)R\_\{\\pi\}R\_\{r\}\(\\lambda\+1\+\\gamma\)\}\}\{\{\\lambda^\{2\}\}\}\.

### 4\.1Critic Error and Target Drift

For framek≥1\\smash\[t\]\{k\\geq 1\}, letAk\\smash\[t\]\{A\_\{k\}\}andbk\\smash\[t\]\{b\_\{k\}\}denote the population quantities in \([6](https://arxiv.org/html/2610.33477#S2.E6)\) underμk⊗πk⊗ℙ\\smash\[t\]\{\\mu\_\{k\}\\otimes\\pi\_\{k\}\\otimes\\mathbb\{P\}\}withμk:=μWk\\mu\_\{k\}:=\\mu\_\{W\_\{k\}\}, letvk∗:=v∗​\(Wk\)=\(Ak\+λ​Id\)−1​bk\\smash\[t\]\{v\_\{k\}^\{\*\}:=v^\{\*\}\(W\_\{k\}\)=\(A\_\{k\}\+\\lambda I\_\{d\}\)^\{\-1\}b\_\{k\}\}, and letv0∗:=v1∗v\_\{0\}^\{\*\}:=v\_\{1\}^\{\*\}for the warm\-up frame\. By leveraging a second\-moment bound for averages along the policy\-induced chain \(Lemma[20](https://arxiv.org/html/2610.33477#Thmtheorem20)in Appendix[D](https://arxiv.org/html/2610.33477#A4); cf\.[Dorfman and Levy, 2022](https://arxiv.org/html/2610.33477#bib.bib16), Lemma A\.6\), we establish the error bound of the inverse\-free LSTD critic in the following theorem with proof relegated to Appendix[F](https://arxiv.org/html/2610.33477#A6)\.

###### Theorem 4\(Inverse\-Free LSTD critic\)\.

Under Assumptions[1](https://arxiv.org/html/2610.33477#Thmassump1)and[2](https://arxiv.org/html/2610.33477#Thmassump2), choose any ridge parameterλ\>0\\lambda\>0, which lower\-boundsσmin​\(A^k\+λ​Id\)\\sigma\_\{\\min\}\(\\hat\{A\}\_\{k\}\+\\lambda I\_\{d\}\)\(Appendix[B](https://arxiv.org/html/2610.33477#A2)\), and initialize NSI withXk,0=Hk⊤/ck2X\_\{k,0\}=\\nicefrac\{\{H\_\{k\}^\{\\top\}\}\}\{\{c\_\{k\}^\{2\}\}\}for anyck∈\[‖Hk‖2,λ\+1\+γ\]c\_\{k\}\\in\[\\\|H\_\{k\}\\\|\_\{2\},\\lambda\+1\+\\gamma\]\. Then, for everyk≥0k\\geq 0, afterP1P\_\{1\}NSI recursions \([8](https://arxiv.org/html/2610.33477#S3.E8)\), the criticvk:=vk,P1v\_\{k\}:=v\_\{k,P\_\{1\}\}satisfies‖vk‖2≤Rv\\\|v\_\{k\}\\\|\_\{2\}\\leq R\_\{v\}and

𝔼⁡\[‖vk−vk∗‖22\|ℱk−1\]≤ϵlstd:=Rv​exp⁡\(−CE​κ1P1\)\+2​Cv​1\+4​τmixT\\sqrt\{\\mathbb\{E\}\[\\\|v\_\{k\}\-v\_\{k\}^\{\*\}\\\|\_\{2\}^\{2\}\\,\|\\,\{\\cal F\}\_\{k\-1\}\]\}\\leq\\epsilon\_\{\\rm lstd\}:=R\_\{v\}\\exp\(\-C\_\{E\}\\kappa\_\{1\}^\{P\_\{1\}\}\)\+2C\_\{v\}\\sqrt\{\\frac\{1\+4\\tau\_\{\\mathrm\{mix\}\}\}\{T\}\}\(19\)whereCE:=log⁡1/ρE\>0C\_\{E\}:=\\log\\nicefrac\{\{1\}\}\{\{\\rho\_\{E\}\}\}\>0, and

Rv=Rrλ,Cv:=2​Rr​\(λ\+1\+3​γ/2\)λ2,ρE:=1−\(λλ\+1\+γ\)4∈\(0,1\)\.\\displaystyle R\_\{v\}=\\frac\{R\_\{r\}\}\{\\lambda\},\\kern 5\.0ptC\_\{v\}:=\\frac\{\\sqrt\{2\}R\_\{r\}\(\\lambda\+1\+\\nicefrac\{\{3\\gamma\}\}\{\{2\}\}\)\}\{\\lambda^\{2\}\},\\kern 5\.0pt\\rho\_\{E\}:=1\-\\Big\(\\frac\{\\lambda\}\{\\lambda\+1\+\\gamma\}\\Big\)^\{4\}\\in\(0,1\)\.In particular,𝔼⁡\[‖vk−vk∗‖22\|ℱk−1\]≤2​Rv2​exp⁡\(−2​CE​κ1P1\)\+8​\(1\+4​τmix\)​Cv2/T\\mathbb\{E\}\[\\\|v\_\{k\}\-v\_\{k\}^\{\*\}\\\|\_\{2\}^\{2\}\\,\|\\,\{\\cal F\}\_\{k\-1\}\]\\leq 2R\_\{v\}^\{2\}\\exp\(\-2C\_\{E\}\\kappa\_\{1\}^\{P\_\{1\}\}\)\+\\nicefrac\{\{8\(1\+4\\tau\_\{\\mathrm\{mix\}\}\)C\_\{v\}^\{2\}\}\}\{\{T\}\}\.

The two terms in \([19](https://arxiv.org/html/2610.33477#S4.E19)\) are the NSI error and the Markovian sampling error, respectively\. Since the residual satisfies‖Ek,q\+1‖2=‖Ek,q‖2κ1\\\|E\_\{k,q\+1\}\\\|\_\{2\}=\\\|E\_\{k,q\}\\\|\_\{2\}^\{\\kappa\_\{1\}\}, the NSI error is at mostδ<Rv\\delta<R\_\{v\}afterP1=max⁡\{0,⌈logκ1⁡\(log⁡\(Rv/δ\)/CE\)⌉\}P\_\{1\}=\\max\\\{0,\\lceil\\log\_\{\\kappa\_\{1\}\}\(\\nicefrac\{\{\\log\(\\nicefrac\{\{R\_\{v\}\}\}\{\{\\delta\}\}\)\}\}\{\{C\_\{E\}\}\}\)\\rceil\\\}NSI recursions, which is doubly logarithmic inRv/δ\\nicefrac\{\{R\_\{v\}\}\}\{\{\\delta\}\}and logarithmic in the conditioning ofHkH\_\{k\}\. Moreover, the realized residual bounds the NSI error byRv​‖Ek,P1‖FR\_\{v\}\\\|E\_\{k,P\_\{1\}\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}and can be monitored during training\. It is worth stressing that the bounds hold for every ridge parameterλ\>0\\lambda\>0with the constantsRvR\_\{v\},CvC\_\{v\}, andρE\\rho\_\{E\}independent of the frame lengthTTdue to the two facts: \(i\) the ridge\-regularized pathwise LSTD system is accretive for every frame length \(Appendix[B](https://arxiv.org/html/2610.33477#A2)\); and \(ii\) dropping the bootstrap of the last transition leaves the targetvk∗v\_\{k\}^\{\*\}unchanged and makesA^k\\hat\{A\}\_\{k\}differ from the full average ofϕ⁡\(s\)​\(ϕ⁡\(s\)−γ​ϕ​\(s′\)\)⊤\\phi\(s\)\(\\phi\(s\)\-\\gamma\\phi\(s^\{\\prime\}\)\)^\{\\top\}over the frame by at mostγ/T\\nicefrac\{\{\\gamma\}\}\{\{T\}\}in the Frobenius norm, which is absorbed byCvC\_\{v\}\. Since the lagged criticvk−1v\_\{k\-1\}used at framekkis fitted toπk−1\\pi\_\{k\-1\}, Lemma[3](https://arxiv.org/html/2610.33477#Thmtheorem3)bounds the resulting policy drift of the lagged critic byL∗​‖Wk−Wk−1‖2L\_\{\*\}\\\|W\_\{k\}\-W\_\{k\-1\}\\\|\_\{2\}\.

### 4\.2Bias, Noise, and Importance Correction of the PG Estimator

Throughout the actor analysis, the ridge parameter and the NSI initialization satisfy the conditions of Theorem[4](https://arxiv.org/html/2610.33477#Thmtheorem4)\. For aTT\-step trajectoryO=\[ot\]t=0T−1O=\[o\_\{t\}\]\_\{t=0\}^\{T\-1\}withot=\(st,at,st\+1\)o\_\{t\}=\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\)that is induced by the policyπW\\pi\_\{W\}froms0s\_\{0\}, we writeδ^t​\(v\)\\hat\{\\delta\}\_\{t\}\(v\)andg^​\(W,v,O\)\\hat\{g\}\(W,v;O\)for \([9](https://arxiv.org/html/2610.33477#S3.E9)\) and \([10](https://arxiv.org/html/2610.33477#S3.E10)\) evaluated onOOin place ofOkO\_\{k\}\. The following lemma bounds the critic, the advantage estimates, and the PG estimators for every realization with proof relegated to Appendix[G](https://arxiv.org/html/2610.33477#A7)\.

###### Lemma 5\.

Under Assumption[1](https://arxiv.org/html/2610.33477#Thmassump1)and the conditions of Theorem[4](https://arxiv.org/html/2610.33477#Thmtheorem4), call a critic parametervvadmissible ifsups\|ϕ⊤​\(s\)​v\|≤Rv\\sup\_\{s\}\|\\phi^\{\\top\}\(s\)v\|\\leq R\_\{v\}, and letRA:=\[Rr\+\(1\+γ\)​Rv\]/\(1−γ\)R\_\{A\}:=\\nicefrac\{\{\[R\_\{r\}\+\(1\+\\gamma\)R\_\{v\}\]\}\}\{\{\(1\-\\gamma\)\}\}andRg:=RA​RπR\_\{g\}:=R\_\{A\}R\_\{\\pi\}\. Then, the following hold for every realization\.

- \(i\)Every NSI iterate satisfies‖vk,q‖2≤Rv\\\|v\_\{k,q\}\\\|\_\{2\}\\leq R\_\{v\}; hence, since‖ϕ⁡\(s\)‖2≤1\\\|\\phi\(s\)\\\|\_\{2\}\\leq 1, the criticvk=vk,P1v\_\{k\}=v\_\{k,P\_\{1\}\}is admissible\.
- \(ii\)For every admissiblevv, every observationoo, and every stepttof aTT\-step frame, the TD error and the advantage estimate \([9](https://arxiv.org/html/2610.33477#S3.E9)\) satisfy\|δv​\(o\)\|≤Rr\+\(1\+γ\)​Rv\|\\delta\_\{v\}\(o\)\|\\leq R\_\{r\}\+\(1\+\\gamma\)R\_\{v\}and\|δ^t​\(v\)\|≤RA\|\\hat\{\\delta\}\_\{t\}\(v\)\|\\leq R\_\{A\}\.
- \(iii\)For every actor parameterWW, every admissiblevv, and everyTT\-step frameOO, the PG estimator \([10](https://arxiv.org/html/2610.33477#S3.E10)\) and the PG satisfy‖g^​\(W,v,O\)‖∗≤Rg\\\|\\hat\{g\}\(W,v;O\)\\\|\_\{\*\}\\leq R\_\{g\}and‖∇J​\(W\)‖∗≤Rr​Rπ/\(1−γ\)≤Rg\\\|\\nabla J\(W\)\\\|\_\{\*\}\\leq\\nicefrac\{\{R\_\{r\}R\_\{\\pi\}\}\}\{\{\(1\-\\gamma\)\}\}\\leq R\_\{g\}; in particular,‖Gk‖∗≤Rg\\\|G\_\{k\}\\\|\_\{\*\}\\leq R\_\{g\}for everykk, and, since∥⋅∥F≤∥⋅∥∗\\\|\\cdot\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq\\\|\\cdot\\\|\_\{\*\}, the same bounds hold in the Frobenius norm\.

For an admissible critic parametervvand a start states0s\_\{0\}, we callω=\(v,s0\)\\omega=\(v,s\_\{0\}\)an admissible frame context\. Since the uniform weights in \([10](https://arxiv.org/html/2610.33477#S3.E10)\) average over the observations whose state distribution approachesμW\\mu\_\{W\}rather thanζW\\zeta\_\{W\}, we compare the PG estimator with the stationary counterpartg⁡\(W\)g\(W\)of \([2](https://arxiv.org/html/2610.33477#S2.E2)\) and split its bias into a context\-free partB¯\\bar\{B\}and a context\-dependent partDDas

g⁡\(W\):=𝔼\(s,a\)∼μW⊗πW\[\(QW\(s,a\)−VW\(s\)\)∇logπW\(a\|s\)\],h⁡\(W,ω\):=𝔼\[g^\(W,v;O\)\|s0\],D\(W;ω\):=h\(W;ω\)−g\(W\),B⁡\(W,ω\):=h⁡\(W,ω\)−∇J​\(W\),B¯​\(W\):=g⁡\(W\)−∇J​\(W\)\\begin\{split\}g\(W\)&:=\\mathbb\{E\}\_\{\(s,a\)\\sim\\mu\_\{W\}\\otimes\\pi\_\{W\}\}\\big\[\(Q\_\{W\}\(s,a\)\-V\_\{W\}\(s\)\)\\nabla\\log\\pi\_\{W\}\(a\|s\)\\big\],\\\\ h\(W;\\omega\)&:=\\mathbb\{E\}\[\\hat\{g\}\(W,v;O\)\|s\_\{0\}\],\\qquad D\(W;\\omega\):=h\(W;\\omega\)\-g\(W\),\\\\ B\(W;\\omega\)&:=h\(W;\\omega\)\-\\nabla J\(W\),\\qquad\\bar\{B\}\(W\):=g\(W\)\-\\nabla J\(W\)\\end\{split\}\(20\)whereμW\\mu\_\{W\}is unique by Assumption[2](https://arxiv.org/html/2610.33477#Thmassump2), the context\-free partB¯\\bar\{B\}is the mismatch betweenμW\\mu\_\{W\}andζW\\zeta\_\{W\}, and the context\-dependent partDDcollects the transient froms0s\_\{0\}and the critic error\. By theℱk−1\{\\cal F\}\_\{k\-1\}\-measurability in Section[4](https://arxiv.org/html/2610.33477#S4), the actor at framekkoperates under the admissible andℱk−1\{\\cal F\}\_\{k\-1\}\-measurable contextωk:=\(vk−1,sk,0\)\\omega\_\{k\}:=\(v\_\{k\-1\},s\_\{k,0\}\)such that𝔼⁡\[Gk\|ℱk−1\]=h⁡\(Wk,ωk\)\\mathbb\{E\}\[G\_\{k\}\\,\|\\,\{\\cal F\}\_\{k\-1\}\]=h\(W\_\{k\};\\omega\_\{k\}\)\. The critic enters the PG estimator through its value error at the current policy as

ϵk​\(s\):=ϕ⊤​\(s\)​vk−1−VWk​\(s\),ϵcrit:=supk∈\[K\]𝔼⁡\[‖ϵk‖∞2\],ϵdist:=supW‖B¯​\(W\)‖∗\\epsilon\_\{k\}\(s\):=\\phi^\{\\top\}\(s\)v\_\{k\-1\}\-V\_\{W\_\{k\}\}\(s\),\\kern 5\.0pt\\epsilon\_\{\\rm crit\}:=\\sup\_\{k\\in\[K\]\}\\sqrt\{\\mathbb\{E\}\[\\\|\\epsilon\_\{k\}\\\|\_\{\\infty\}^\{2\}\]\},\\kern 5\.0pt\\epsilon\_\{\\rm dist\}:=\\sup\_\{W\}\\\|\\bar\{B\}\(W\)\\\|\_\{\*\}\(21\)and we letGk∘G\_\{k\}^\{\\circ\}denote the oracle estimator, i\.e\., \([10](https://arxiv.org/html/2610.33477#S3.E10)\) evaluated atWkW\_\{k\}andOkO\_\{k\}withVWkV\_\{W\_\{k\}\}in place ofϕ⊤​v\\phi^\{\\top\}vin \([9](https://arxiv.org/html/2610.33477#S3.E9)\)\. Moreover, we define the constants

CF:=RA\[Lπ\+\(exp\(1\)−1\)TRπ2\],DF:=τmix2​RA​\(46​Rπ2\+5​Lπ\)\(1−γ\)2​T,ϵbias:=ϵdist\+Rπ​\(4​Rr​τmix\+γ​ϵcrit\)\(1−γ\)​T,V∘:=Rr​Rπ1−γ\(41\+4​τmix\+2​γ1−γ\),σ¯:=V∘\+Rπ​ϵcritT\+γ​Rπ​ϵcrit\(1−γ\)​T\.\\begin\{split\}&C\_\{F\}:=R\_\{A\}\\big\[L\_\{\\pi\}\+\(\\exp\(1\)\-1\)\\,TR\_\{\\pi\}^\{2\}\\big\],\\quad D\_\{F\}:=\\frac\{\\tau\_\{\\mathrm\{mix\}\}^\{2\}R\_\{A\}\(46R\_\{\\pi\}^\{2\}\+5L\_\{\\pi\}\)\}\{\(1\-\\gamma\)^\{2\}T\},\\\\ &\\epsilon\_\{\\rm bias\}:=\\epsilon\_\{\\rm dist\}\+\\frac\{R\_\{\\pi\}\(4R\_\{r\}\\tau\_\{\\mathrm\{mix\}\}\+\\gamma\\,\\epsilon\_\{\\rm crit\}\)\}\{\(1\-\\gamma\)T\},\\\\ &V^\{\\circ\}:=\\frac\{R\_\{r\}R\_\{\\pi\}\}\{1\-\\gamma\}\\Big\(4\\sqrt\{1\+4\\tau\_\{\\mathrm\{mix\}\}\}\+\\frac\{\\sqrt\{2\}\\,\\gamma\}\{1\-\\gamma\}\\Big\),\\quad\\bar\{\\sigma\}:=\\frac\{V^\{\\circ\}\+R\_\{\\pi\}\\epsilon\_\{\\rm crit\}\}\{\\sqrt\{T\}\}\+\\frac\{\\gamma R\_\{\\pi\}\\epsilon\_\{\\rm crit\}\}\{\(1\-\\gamma\)T\}\.\\end\{split\}\(22\)
Note that the a priori boundsϵcrit≤Rv\+Rr/\(1−γ\)\\epsilon\_\{\\rm crit\}\\leq R\_\{v\}\+\\nicefrac\{\{R\_\{r\}\}\}\{\{\(1\-\\gamma\)\}\}andϵdist≤2​Rr​Rπ/\(1−γ\)​supW‖μW−ζW‖TV\\epsilon\_\{\\rm dist\}\\leq\\nicefrac\{\{2R\_\{r\}R\_\{\\pi\}\}\}\{\{\(1\-\\gamma\)\}\}\\sup\_\{W\}\\\|\\mu\_\{W\}\-\\zeta\_\{W\}\\\|\_\{\\mathrm\{TV\}\}hold such that every constant in \([22](https://arxiv.org/html/2610.33477#S4.E22)\) is finite and free ofrr\. Besides, the nuclear quantitiesϵdist\\epsilon\_\{\\rm dist\}andϵbias\\epsilon\_\{\\rm bias\}also bound the bias in the Frobenius norm, whereasσ¯\\bar\{\\sigma\}bounds the noise in the Frobenius norm in which the ICM averages the noise\. Since \([15a](https://arxiv.org/html/2610.33477#S4.E15.1)\) and \([15b](https://arxiv.org/html/2610.33477#S4.E15.2)\) pair the score with the spectral norm, the constantsCFC\_\{F\}andDFD\_\{F\}bound the importance correction and the change of the frame bias per unit spectral norm of the step in the nuclear norm, respectively\. The following two lemmas are established with proofs relegated to Appendices[H](https://arxiv.org/html/2610.33477#A8)and[I](https://arxiv.org/html/2610.33477#A9), respectively\.

###### Lemma 6\.

Under Assumption[1](https://arxiv.org/html/2610.33477#Thmassump1), the following hold\.

- \(i\)For actor parametersWWandW′W^\{\\prime\}withℓ2:=‖W−W′‖2\\ell\_\{2\}:=\\\|W\-W^\{\\prime\}\\\|\_\{2\}and a trajectoryOOofTTsteps withT​Rπ​ℓ2≤1TR\_\{\\pi\}\\ell\_\{2\}\\leq 1, the importance weightw:=∏t=0T−1πW′​\(at\|st\)/πW​\(at\|st\)w:=\\prod\_\{t=0\}^\{T\-1\}\\nicefrac\{\{\\pi\_\{W^\{\\prime\}\}\(a\_\{t\}\|s\_\{t\}\)\}\}\{\{\\pi\_\{W\}\(a\_\{t\}\|s\_\{t\}\)\}\}satisfies\|log⁡w\|≤T​Rπ​ℓ2≤1\|\\log w\|\\leq TR\_\{\\pi\}\\ell\_\{2\}\\leq 1and\|w−1\|≤\(exp⁡\(1\)−1\)​T​Rπ​ℓ2\|w\-1\|\\leq\(\\exp\(1\)\-1\)TR\_\{\\pi\}\\ell\_\{2\}\.
- \(ii\)For everyk≥2k\\geq 2, the reweighted estimate is conditionally unbiased, i\.e\.,𝔼⁡\[G~k\|ℱk−1\]=h⁡\(Wk−1,ωk\)\\mathbb\{E\}\[\\widetilde\{G\}\_\{k\}\\,\|\\,\{\\cal F\}\_\{k\-1\}\]=h\(W\_\{k\-1\};\\omega\_\{k\}\)\.
- \(iii\)IfT​Rπ​ℓ2≤1TR\_\{\\pi\}\\ell\_\{2\}\\leq 1, then, for every admissiblevvand everyTT\-step frameOO, with the importance weightwwof part \(i\), the importance correction is of the order of the step ‖g^​\(W,v,O\)−w​g^​\(W′,v,O\)‖∗≤CF​ℓ2;\\\|\\hat\{g\}\(W,v;O\)\-w\\,\\hat\{g\}\(W^\{\\prime\},v;O\)\\\|\_\{\*\}\\leq C\_\{F\}\\,\\ell\_\{2\};since∥⋅∥F≤∥⋅∥∗\\\|\\cdot\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq\\\|\\cdot\\\|\_\{\*\}, the same bound holds in the Frobenius norm\.

In particular, if‖Wk−Wk−1‖2≤ℓ¯2\\\|W\_\{k\}\-W\_\{k\-1\}\\\|\_\{2\}\\leq\\bar\{\\ell\}\_\{2\}withT​Rπ​ℓ¯2≤1TR\_\{\\pi\}\\bar\{\\ell\}\_\{2\}\\leq 1, then parts \(i\) and \(iii\) withW=WkW=W\_\{k\},W′=Wk−1W^\{\\prime\}=W\_\{k\-1\},O=OkO=O\_\{k\}, andv=vk−1v=v\_\{k\-1\}give\|log⁡wk\|≤1\|\\log w\_\{k\}\|\\leq 1and‖Gk−G~k‖∗≤CF​ℓ¯2\\\|G\_\{k\}\-\\widetilde\{G\}\_\{k\}\\\|\_\{\*\}\\leq C\_\{F\}\\,\\bar\{\\ell\}\_\{2\}for everyk≥2k\\geq 2\.

Since a normalized actor update satisfies‖Wk−Wk−1‖2≤β\\\|W\_\{k\}\-W\_\{k\-1\}\\\|\_\{2\}\\leq\\beta, Lemma[6](https://arxiv.org/html/2610.33477#Thmtheorem6)bounds its importance weights deterministically whenT​Rπ​β≤1TR\_\{\\pi\}\\beta\\leq 1without the coupling betweenβ\\betaandη\\eta, the weight clipping, or the bounded variance of the importance weights that[Huang et al\. \(2020, Assumption 3\)](https://arxiv.org/html/2610.33477#bib.bib21)impose for Euclidean steps; for softmax policies, normalized steps remove this assumption in the same way\([Barakat et al\., 2023](https://arxiv.org/html/2610.33477#bib.bib3)\)\. Since an unnormalized actor update scales withMkM\_\{k\}, its importance weights are controlled only when the momentum stays bounded for every realization \(Lemma[22](https://arxiv.org/html/2610.33477#Thmtheorem22)in Appendix[J](https://arxiv.org/html/2610.33477#A10)\)\. Moreover, the importance weight spans the whole frame such that the step condition involvesTTandCFC\_\{F\}grows linearly withTT, as the corresponding constant of[Huang et al\. \(2020\)](https://arxiv.org/html/2610.33477#bib.bib21)grows with the horizon; an analysis of the martingale structure oflog⁡wk\\log w\_\{k\}might reduce such growth\. Note that the per\-decision weights∏i=0t\+lπk−1​\(ak,i\|sk,i\)/πk​\(ak,i\|sk,i\)\\smash\[t\]\{\\prod\_\{i=0\}^\{t\+l\}\\nicefrac\{\{\\pi\_\{k\-1\}\(a\_\{k,i\}\|s\_\{k,i\}\)\}\}\{\{\\pi\_\{k\}\(a\_\{k,i\}\|s\_\{k,i\}\)\}\}\}on the summandδv\(ok,t\+l\)∇logπk−1\(ak,t\|sk,t\)\\delta\_\{v\}\(o\_\{k,t\+l\}\)\\nabla\\log\\pi\_\{k\-1\}\(a\_\{k,t\}\|s\_\{k,t\}\)keepG~k\\widetilde\{G\}\_\{k\}conditionally unbiased, satisfy the same bounds, and can reduce the variance ofG~k\\widetilde\{G\}\_\{k\}\([Precup et al\., 2000](https://arxiv.org/html/2610.33477#bib.bib47)\)\.

###### Lemma 7\.

Under Assumptions[1](https://arxiv.org/html/2610.33477#Thmassump1)and[2](https://arxiv.org/html/2610.33477#Thmassump2), consider an actor update whose iteratesWkW\_\{k\}areℱk−1\{\\cal F\}\_\{k\-1\}\-measurable, the following hold for everyk∈\[K\]k\\in\[K\]\.

- \(i\)For every admissible contextω=\(v,s0\)\\omega=\(v,s\_\{0\}\)and everyWW, the context\-dependent part of the bias satisfies‖D⁡\(W,ω\)‖∗≤Rπ​\(4​Rr​τmix\+γ​‖ϵv‖∞\)/\[\(1−γ\)​T\]\\\|D\(W;\\omega\)\\\|\_\{\*\}\\leq\\nicefrac\{\{R\_\{\\pi\}\(4R\_\{r\}\\tau\_\{\\mathrm\{mix\}\}\+\\gamma\\\|\\epsilon\_\{v\}\\\|\_\{\\infty\}\)\}\}\{\{\[\(1\-\\gamma\)T\]\}\}withϵv:=ϕ⊤​v−VW\\epsilon\_\{v\}:=\\phi^\{\\top\}v\-V\_\{W\}; in particular,𝔼⁡\[‖B⁡\(Wk,ωk\)‖∗2\]≤ϵbias\\sqrt\{\\mathbb\{E\}\[\\\|B\(W\_\{k\};\\omega\_\{k\}\)\\\|\_\{\*\}^\{2\}\]\}\\leq\\epsilon\_\{\\rm bias\}\.
- \(ii\)For any two actor parametersWWandW′W^\{\\prime\}and every admissible contextω\\omega, the context\-dependent part of the bias satisfies‖D⁡\(W,ω\)−D⁡\(W′,ω\)‖∗≤DF​‖W−W′‖2\\\|D\(W;\\omega\)\-D\(W^\{\\prime\};\\omega\)\\\|\_\{\*\}\\leq D\_\{F\}\\\|W\-W^\{\\prime\}\\\|\_\{2\}\.
- \(iii\)The noise of the PG estimator satisfies𝔼⁡\[‖Gk−𝔼⁡\[Gk\|ℱk−1\]‖F2\]≤σ¯\\sqrt\{\\mathbb\{E\}\[\\\|G\_\{k\}\-\\mathbb\{E\}\[G\_\{k\}\\,\|\\,\{\\cal F\}\_\{k\-1\}\]\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\]\}\\leq\\bar\{\\sigma\}, and that of the oracle estimator satisfies𝔼⁡\[‖Gk∘−𝔼⁡\[Gk∘\|ℱk−1\]‖F2\]≤V∘/T\\sqrt\{\\mathbb\{E\}\[\\\|G\_\{k\}^\{\\circ\}\-\\mathbb\{E\}\[G\_\{k\}^\{\\circ\}\\,\|\\,\{\\cal F\}\_\{k\-1\}\]\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\]\}\\leq\\nicefrac\{\{V^\{\\circ\}\}\}\{\{\\sqrt\{T\}\}\}\.
- \(iv\)If‖Wk−Wk−1‖2≤ℓ¯2\\\|W\_\{k\}\-W\_\{k\-1\}\\\|\_\{2\}\\leq\\bar\{\\ell\}\_\{2\}for allk≥2k\\geq 2and a deterministicℓ¯2≥0\\bar\{\\ell\}\_\{2\}\\geq 0, the critic error satisfies ϵcrit≤ϵlstd\+L∗​ℓ¯2\+ϵapp\\epsilon\_\{\\rm crit\}\\leq\\epsilon\_\{\\rm lstd\}\+L\_\{\*\}\\bar\{\\ell\}\_\{2\}\+\\epsilon\_\{\\rm app\}whereϵlstd\\epsilon\_\{\\rm lstd\}is given in Theorem[4](https://arxiv.org/html/2610.33477#Thmtheorem4),L∗L\_\{\*\}in Lemma[3](https://arxiv.org/html/2610.33477#Thmtheorem3), and the approximation error of the linear critic at the solutionv∗​\(W\)v^\{\*\}\(W\)that Theorem[4](https://arxiv.org/html/2610.33477#Thmtheorem4)targets, ϵapp:=supW,s\|ϕ⊤​\(s\)​v∗​\(W\)−VW​\(s\)\|\\epsilon\_\{\\rm app\}:=\\sup\\nolimits\_\{W,s\}\|\\phi^\{\\top\}\(s\)v^\{\*\}\(W\)\-V\_\{W\}\(s\)\|includes the ridge bias\.

Part \(i\) of Lemma[7](https://arxiv.org/html/2610.33477#Thmtheorem7)shows that, besidesϵdist\\epsilon\_\{\\rm dist\}, the bias contains a frame bias of order1/T\\nicefrac\{\{1\}\}\{\{T\}\}with two sources: the start\-state transient, since the average over the frame forgets its start state after𝒪⁡\(τmix\)\{\\cal O\}\(\\tau\_\{\\mathrm\{mix\}\}\)steps; and the critic error at the end of the frame, which is attenuated byγ/\[\(1−γ\)​T\]\\nicefrac\{\{\\gamma\}\}\{\{\[\(1\-\\gamma\)T\]\}\}since the lagged critic enters only through the bootstrap \(Section[3\.2](https://arxiv.org/html/2610.33477#S3.SS2)\)\. Since the exact bootstrap weightγ⁡\(1−γT\)/\[\(1−γ\)​T\]\\nicefrac\{\{\\gamma\(1\-\\gamma^\{T\}\)\}\}\{\{\[\(1\-\\gamma\)T\]\}\}is at mostmin⁡\{γ,γ/\[\(1−γ\)​T\]\}\\min\\\{\\gamma,\\nicefrac\{\{\\gamma\}\}\{\{\[\(1\-\\gamma\)T\]\}\}\\\}, every frame lengthT\>1/\(1−γ\)T\>\\nicefrac\{\{1\}\}\{\{\(1\-\\gamma\)\}\}improves on the weightγ\\gammaof a one\-step TD error\. Part \(ii\) shows thatDDvaries slowly with the actor parameter sinceDDinvolves only the first𝒪⁡\(τmix\)\{\\cal O\}\(\\tau\_\{\\mathrm\{mix\}\}\)and the last𝒪⁡\(1/\(1−γ\)\)\{\\cal O\}\(\\nicefrac\{\{1\}\}\{\{\(1\-\\gamma\)\}\}\)steps of the frame, each with weight1/T\\nicefrac\{\{1\}\}\{\{T\}\}; different from the change of measure over the whole frame behindCFC\_\{F\}, the proof perturbs the policy\-induced chain in the spirit of[Mitrophanov \(2005\)](https://arxiv.org/html/2610.33477#bib.bib39)\. Part \(iii\) quantifies the noise: the oracle part follows from Lemma[20](https://arxiv.org/html/2610.33477#Thmtheorem20)and a martingale decomposition of the multi\-step TD errors \(Lemma[21](https://arxiv.org/html/2610.33477#Thmtheorem21)in Appendix[I](https://arxiv.org/html/2610.33477#A9)\), and the critic part averages the baseline terms over the frame without the factorr1/2r^\{1/2\}that a nuclear\-norm bound would cost\. Both parts decay asT−1/2T^\{\-1/2\}, andσ¯\\bar\{\\sigma\}enters the bounds of Section[4\.4](https://arxiv.org/html/2610.33477#S4.SS4)only through their transients\. Part \(iv\) carries Theorem[4](https://arxiv.org/html/2610.33477#Thmtheorem4)into the actor analysis, whereas parts \(i\)–\(iii\) use the actor update only through theℱk−1\{\\cal F\}\_\{k\-1\}\-measurability ofWkW\_\{k\}\.

### 4\.3Momentum Tracking

The following lemma bounds the tracking error of the ICM for every actor update of bounded spectral norm and in every dual norm of Appendix[A](https://arxiv.org/html/2610.33477#A1)with proof relegated to Appendix[J](https://arxiv.org/html/2610.33477#A10)\.

###### Lemma 8\.

Under Assumptions[1](https://arxiv.org/html/2610.33477#Thmassump1)and[2](https://arxiv.org/html/2610.33477#Thmassump2), consider Algorithm[1](https://arxiv.org/html/2610.33477#alg1)withη∈\[0,1\)\\eta\\in\[0,1\)and an actor update whose iterateWkW\_\{k\}isℱk−1\{\\cal F\}\_\{k\-1\}\-measurable and satisfies‖Wk−Wk−1‖2≤ℓ¯2\\\|W\_\{k\}\-W\_\{k\-1\}\\\|\_\{2\}\\leq\\bar\{\\ell\}\_\{2\}for allk≥2k\\geq 2, whereℓ¯2≥0\\bar\{\\ell\}\_\{2\}\\geq 0is deterministic and, ifη\>0\\eta\>0,T​Rπ​ℓ¯2≤1TR\_\{\\pi\}\\bar\{\\ell\}\_\{2\}\\leq 1, and letσ¯\\bar\{\\sigma\}andϵbias\\epsilon\_\{\\rm bias\}be evaluated along its iterates\. Then, for allk∈\[K\]k\\in\[K\]and every norm∥⋅∥≥∥⋅∥2\\\|\\cdot\\\|\\geq\\\|\\cdot\\\|\_\{2\}with dual norm∥⋅∥†\\\|\\cdot\\\|\_\{\\dagger\}, the ICM \([12](https://arxiv.org/html/2610.33477#S3.E12)\) tracks the PG as

𝔼⁡\[‖Mk−∇J​\(Wk\)‖†\]≤κ†​\[\(ηk−1\+2​\(1−η\)\)​σ¯\+2​η​CF​ℓ¯21−η\]\+ϵbias\+η​DF​ℓ¯21−η\\mathbb\{E\}\[\\\|M\_\{k\}\-\\nabla J\(W\_\{k\}\)\\\|\_\{\\dagger\}\]\\leq\\kappa\_\{\\dagger\}\\Big\[\\big\(\\eta^\{k\-1\}\+\\sqrt\{2\(1\-\\eta\)\}\\big\)\\,\\bar\{\\sigma\}\+\\frac\{\\sqrt\{2\}\\,\\eta\\,C\_\{F\}\\bar\{\\ell\}\_\{2\}\}\{\\sqrt\{1\-\\eta\}\}\\Big\]\+\\epsilon\_\{\\rm bias\}\+\\frac\{\\eta D\_\{F\}\\bar\{\\ell\}\_\{2\}\}\{1\-\\eta\}\(23\)and

𝔼⁡\[‖Mk−∇J​\(Wk\)‖F2\]≤2​η2​\(k−1\)​σ¯2\+4​\(1−η\)​σ¯2\+4​η2​CF2​ℓ¯221−η\+2​\[ϵbias\+η​DF​ℓ¯21−η\]2\.\\mathbb\{E\}\[\\\|M\_\{k\}\-\\nabla J\(W\_\{k\}\)\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\]\\leq 2\\eta^\{2\(k\-1\)\}\\bar\{\\sigma\}^\{2\}\+4\(1\-\\eta\)\\,\\bar\{\\sigma\}^\{2\}\+\\frac\{4\\eta^\{2\}C\_\{F\}^\{2\}\\bar\{\\ell\}\_\{2\}^\{2\}\}\{1\-\\eta\}\+2\\Big\[\\epsilon\_\{\\rm bias\}\+\\frac\{\\eta D\_\{F\}\\bar\{\\ell\}\_\{2\}\}\{1\-\\eta\}\\Big\]^\{2\}\.\(24\)

By the proof of Lemma[8](https://arxiv.org/html/2610.33477#Thmtheorem8)in Appendix[J](https://arxiv.org/html/2610.33477#A10), the tracking error splits asMk−∇J​\(Wk\)=ekn\+ekbM\_\{k\}\-\\nabla J\(W\_\{k\}\)=e^\{\\rm n\}\_\{k\}\+e^\{\\rm b\}\_\{k\}, where the noise partekne^\{\\rm n\}\_\{k\}, i\.e\., a weighted sum of martingale differences, satisfies𝔼⁡\[‖ekn‖F\]≤\(ηk−1\+2​\(1−η\)\)​σ¯\+2​η​CF​ℓ¯2/1−η\\mathbb\{E\}\[\\\|e^\{\\rm n\}\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\]\\leq\(\\eta^\{k\-1\}\+\\sqrt\{2\(1\-\\eta\)\}\)\\,\\bar\{\\sigma\}\+\\nicefrac\{\{\\sqrt\{2\}\\,\\eta C\_\{F\}\\bar\{\\ell\}\_\{2\}\}\}\{\{\\sqrt\{1\-\\eta\}\}\}, and the bias partekbe^\{\\rm b\}\_\{k\}, i\.e\., a weighted sum of conditional means, satisfies𝔼⁡\[‖ekb‖∗\]≤ϵbias\+η​DF​ℓ¯2/\(1−η\)\\mathbb\{E\}\[\\\|e^\{\\rm b\}\_\{k\}\\\|\_\{\*\}\]\\leq\\epsilon\_\{\\rm bias\}\+\\nicefrac\{\{\\eta D\_\{F\}\\bar\{\\ell\}\_\{2\}\}\}\{\{\(1\-\\eta\)\}\}\. Hence, the ICM averages the noiseσ¯\\bar\{\\sigma\}, including the critic noise, over about1/\(1−η\)\\nicefrac\{\{1\}\}\{\{\(1\-\\eta\)\}\}frames at the price of the correction noise2​η​CF​ℓ¯2/1−η\\nicefrac\{\{\\sqrt\{2\}\\eta C\_\{F\}\\bar\{\\ell\}\_\{2\}\}\}\{\{\\sqrt\{1\-\\eta\}\}\}, which is small since the importance correction and the bounded step makeGk−G~kG\_\{k\}\-\\widetilde\{G\}\_\{k\}of the order ofℓ¯2\\bar\{\\ell\}\_\{2\}\. Since such averaging rests on the orthogonality of martingale differences in the Frobenius inner product, the noise part is converted to the dual norm at the cost ofκ†\\kappa\_\{\\dagger\}, whereas the bias part carries no such factor since Lemma[7](https://arxiv.org/html/2610.33477#Thmtheorem7)\(i\),\(ii\) hold in the nuclear norm\. Moreover, since the importance correction makes the ICM track the mean fieldhh, the context\-free partB¯\\bar\{B\}telescopes in the bias recursion \([48](https://arxiv.org/html/2610.33477#A10.E48)\) such that the drift stems fromDDalone\. Note that the second\-moment bound \([24](https://arxiv.org/html/2610.33477#S4.E24)\) is needed only for unnormalized actor updates\.

### 4\.4Convergence Rates of Generic ICM\-Based Actors

Based on Lemmas[6](https://arxiv.org/html/2610.33477#Thmtheorem6)–[8](https://arxiv.org/html/2610.33477#Thmtheorem8), we establish the convergence rates of the two classes of actor updates in Definition[1](https://arxiv.org/html/2610.33477#Thmtheorem1)with proofs relegated to Appendices[K](https://arxiv.org/html/2610.33477#A11)and[L](https://arxiv.org/html/2610.33477#A12), respectively\. LetΔ1:=J∗−J⁡\(W1\)\\Delta\_\{1\}:=J^\{\*\}\-J\(W\_\{1\}\)withJ∗:=supWJ⁡\(W\)J^\{\*\}:=\\sup\_\{W\}J\(W\)\.

###### Theorem 11\(Normalized actor updates\)\.

Under Assumptions[1](https://arxiv.org/html/2610.33477#Thmassump1)and[2](https://arxiv.org/html/2610.33477#Thmassump2), let the actor update be normalized in a norm∥⋅∥≥∥⋅∥2\\\|\\cdot\\\|\\geq\\\|\\cdot\\\|\_\{2\}with stepsizeβ\>0\\beta\>0and alignment factorχ\\chi\(Definition[1](https://arxiv.org/html/2610.33477#Thmtheorem1)\), letη∈\[0,1\)\\eta\\in\[0,1\)withT​Rπ​β≤1TR\_\{\\pi\}\\beta\\leq 1ifη\>0\\eta\>0, and letσ¯\\bar\{\\sigma\}andϵbias\\epsilon\_\{\\rm bias\}be evaluated along the iterates\. Then Algorithm[1](https://arxiv.org/html/2610.33477#alg1)satisfies

1K​∑k=1K𝔼⁡\[‖∇J​\(Wk\)‖†\]≤χ⁡\[Δ1β​K\+L​β2\]\+\(1\+χ\)​κ†​\[σ¯K⁡\(1−η\)\+2​\(1−η\)​σ¯\+2​η​CF​β1−η\]\+\(1\+χ\)​\(ϵbias\+η​DF​β1−η\)\.\\begin\{split\}&\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\[\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\\dagger\}\]\\leq\\chi\\Big\[\\frac\{\\Delta\_\{1\}\}\{\\beta K\}\+\\frac\{L\\beta\}\{2\}\\Big\]\\\\ &\\quad\+\(1\+\\chi\)\\,\\kappa\_\{\\dagger\}\\Big\[\\frac\{\\bar\{\\sigma\}\}\{K\(1\-\\eta\)\}\+\\sqrt\{2\(1\-\\eta\)\}\\,\\bar\{\\sigma\}\+\\frac\{\\sqrt\{2\}\\,\\eta\\,C\_\{F\}\\beta\}\{\\sqrt\{1\-\\eta\}\}\\Big\]\+\(1\+\\chi\)\\Big\(\\epsilon\_\{\\rm bias\}\+\\frac\{\\eta D\_\{F\}\\beta\}\{1\-\\eta\}\\Big\)\.\\end\{split\}\(25\)In particular, for anyc\>0c\>0and a prescribed horizonK≥\(c​T​Rπ\)3/2K\\geq\(cTR\_\{\\pi\}\)^\{3/2\}, the choice1−η=K−2/31\-\\eta=K^\{\-2/3\}andβ=cK−2/3\\beta=cK^\{\-2/3\}yields

1K​∑k=1K𝔼⁡\[‖∇J​\(Wk\)‖†\]≤χ⁡\[Δ1c​1K1/3\+c​L2​K2/3\]\+\(1\+χ\)​κ†​\[\(1\+2\)​σ¯\+2​c​CF\]​1K1/3\+\(1\+χ\)​\(ϵbias\+c​DF\)\.\\begin\{split\}&\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\[\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\\dagger\}\]\\leq\\chi\\Big\[\\frac\{\\Delta\_\{1\}\}\{c\}\\,\\frac\{1\}\{K^\{1/3\}\}\+\\frac\{cL\}\{2K^\{2/3\}\}\\Big\]\\\\ &\\quad\+\(1\+\\chi\)\\,\\kappa\_\{\\dagger\}\\big\[\(1\+\\sqrt\{2\}\)\\,\\bar\{\\sigma\}\+\\sqrt\{2\}\\,cC\_\{F\}\\big\]\\frac\{1\}\{K^\{1/3\}\}\+\(1\+\\chi\)\\big\(\\epsilon\_\{\\rm bias\}\+cD\_\{F\}\\big\)\.\\end\{split\}\(26\)

The proof uses the actor update only through theℱk\{\\cal F\}\_\{k\}\-measurability, the step bound‖Wk\+1−Wk‖2≤β\\\|W\_\{k\+1\}\-W\_\{k\}\\\|\_\{2\}\\leq\\beta, and the alignment, which turn \([17](https://arxiv.org/html/2610.33477#S4.E17)\) into the inequality‖∇J​\(Wk\)‖†≤χ​\[J⁡\(Wk\+1\)−J⁡\(Wk\)\]/β\+\(1\+χ\)​‖Mk−∇J​\(Wk\)‖†\+χ​L​β/2\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\\dagger\}\\leq\\chi\\nicefrac\{\{\[J\(W\_\{k\+1\}\)\-J\(W\_\{k\}\)\]\}\}\{\{\\beta\}\}\+\(1\+\\chi\)\\\|M\_\{k\}\-\\nabla J\(W\_\{k\}\)\\\|\_\{\\dagger\}\+\\nicefrac\{\{\\chi L\\beta\}\}\{\{2\}\}that is linear in the tracking error\. The exponent of the𝒪\(K−1/3\)\{\\cal O\}\(K^\{\-1/3\}\)transient matches those of the ICM actor\([Huang et al\., 2020](https://arxiv.org/html/2610.33477#bib.bib21)\)and the variance\-reduced Muon\([Chang et al\., 2025](https://arxiv.org/html/2610.33477#bib.bib8)\), which rely on unbiased estimates from restarted episodes or i\.i\.d\. samples, whereas \([26](https://arxiv.org/html/2610.33477#S4.E26)\) holds for biased and Markov\-correlated frame estimates \(Remark[14](https://arxiv.org/html/2610.33477#Thmtheorem14)\)\. Note that the noiseσ¯\\bar\{\\sigma\}does not enter the floor since the ICM averages the noise across frames\. Forη=0\\eta=0, the bound retains2\(1\+χ\)κ†σ¯=𝒪\(T−1/2\)\\sqrt\{2\}\(1\+\\chi\)\\kappa\_\{\\dagger\}\\bar\{\\sigma\}=\{\\cal O\}\(T^\{\-1/2\}\)for every stepsize since normalizing a noisy estimate does not average its noise; hence, only longer frames reduce the noise \(Remark[19](https://arxiv.org/html/2610.33477#Thmtheorem19)\)\.

###### Theorem 12\(Unnormalized actor updates\)\.

Under Assumptions[1](https://arxiv.org/html/2610.33477#Thmassump1)and[2](https://arxiv.org/html/2610.33477#Thmassump2), let the actor update be unnormalized with stepsize boundβ¯\\bar\{\\beta\}, rangeϑ\\vartheta, and variationVuV\_\{u\}\(Definition[1](https://arxiv.org/html/2610.33477#Thmtheorem1)\), letη∈\[0,1\)\\eta\\in\[0,1\)withL​β¯≤1L\\bar\{\\beta\}\\leq 1and, ifη\>0\\eta\>0,2​T​Rπ​Rg​β¯≤12TR\_\{\\pi\}R\_\{g\}\\bar\{\\beta\}\\leq 1and2​η​CF​β¯≤1−η2\\eta C\_\{F\}\\bar\{\\beta\}\\leq 1\-\\eta, letσ¯\\bar\{\\sigma\}andϵbias\\epsilon\_\{\\rm bias\}be evaluated along the iterates, and letΔmax:=supW\[J∗−J⁡\(W\)\]≤2​Rr\\Delta\_\{\\max\}:=\\sup\_\{W\}\[J^\{\*\}\-J\(W\)\]\\leq 2R\_\{r\}andΛ:=2​ϑ​Δ1\+2​Vu​Δmax\\Lambda:=2\\sqrt\{\\vartheta\}\\,\\Delta\_\{1\}\+2V\_\{u\}\\Delta\_\{\\max\}\. Then Algorithm[1](https://arxiv.org/html/2610.33477#alg1)satisfies

1K​∑k=1K𝔼⁡\[‖∇J​\(Wk\)‖F\]≤Λβ¯​K\+2​σ¯K⁡\(1−η2\)\+2​1−η​σ¯\+4​η​CF​β¯​Rg1−η\+2​\[ϵbias\+2​η​DF​β¯​Rg1−η\]\.\\begin\{split\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\[\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\]\\leq\\sqrt\{\\frac\{\\Lambda\}\{\\bar\{\\beta\}K\}\}&\+\\frac\{\\sqrt\{2\}\\,\\bar\{\\sigma\}\}\{\\sqrt\{K\(1\-\\eta^\{2\}\)\}\}\+2\\sqrt\{1\-\\eta\}\\,\\bar\{\\sigma\}\\\\ &\+\\frac\{4\\eta C\_\{F\}\\bar\{\\beta\}R\_\{g\}\}\{\\sqrt\{1\-\\eta\}\}\+\\sqrt\{2\}\\Big\[\\epsilon\_\{\\rm bias\}\+\\frac\{2\\eta D\_\{F\}\\bar\{\\beta\}R\_\{g\}\}\{1\-\\eta\}\\Big\]\.\\end\{split\}\(27\)In particular, for any0<c≤1/\(2​CF\)0<c\\leq\\nicefrac\{\{1\}\}\{\{\(2C\_\{F\}\)\}\}andK≥\(c​L\)2K\\geq\(cL\)^\{2\}, the choice1−η=K−1/21\-\\eta=K^\{\-1/2\}andβ¯=cK−1/2\\bar\{\\beta\}=cK^\{\-1/2\}yields

1K​∑k=1K𝔼⁡\[‖∇J​\(Wk\)‖F\]≤\[Λc\+\(2\+2\)​σ¯\+4​c​CF​Rg\]​1K1/4\+2​\(ϵbias\+2​c​DF​Rg\)\.\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\[\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\]\\leq\\Big\[\\sqrt\{\\frac\{\\Lambda\}\{c\}\}\+\(2\+\\sqrt\{2\}\)\\bar\{\\sigma\}\+4cC\_\{F\}R\_\{g\}\\Big\]\\frac\{1\}\{K^\{1/4\}\}\+\\sqrt\{2\}\\big\(\\epsilon\_\{\\rm bias\}\+2cD\_\{F\}R\_\{g\}\\big\)\.\(28\)

The proof in Appendix[L](https://arxiv.org/html/2610.33477#A12)first bounds the momentum by2​Rg2R\_\{g\}for every realization \(Lemma[22](https://arxiv.org/html/2610.33477#Thmtheorem22)\) such that Lemma[8](https://arxiv.org/html/2610.33477#Thmtheorem8)applies withℓ¯2=2​β¯​Rg\\bar\{\\ell\}\_\{2\}=2\\bar\{\\beta\}R\_\{g\}, and then sums the descent inequality, which is quadratic in the gradient norm, by parts to obtainΛ\\Lambda\. Note that the bound holds in the nuclear norm with the factorr1/2r^\{1/2\}, and the stepsize conditionc≤1/\(2​CF\)c\\leq\\nicefrac\{\{1\}\}\{\{\(2C\_\{F\}\)\}\}tightens as1/T\\nicefrac\{\{1\}\}\{\{T\}\}\. For both classes of actor updates, the following corollary counts the numbers of updates and transitions after which the returned iterate comes withinϵ\\epsilonof the floor\.

###### Corollary 13\(Finite\-time error guarantee\)\.

Letϵ¯crit:=ϵlstd\+ϵapp\\bar\{\\epsilon\}\_\{\\rm crit\}:=\\epsilon\_\{\\rm lstd\}\+\\epsilon\_\{\\rm app\}denote the part of the bound of Lemma[7](https://arxiv.org/html/2610.33477#Thmtheorem7)\(iv\) that does not depend onKK, and letϵ\>0\\epsilon\>0\.

- \(i\)Under the conditions and the schedule of Theorem[11](https://arxiv.org/html/2610.33477#Thmtheorem11), letσ^\\hat\{\\sigma\}denoteσ¯\\bar\{\\sigma\}of \([22](https://arxiv.org/html/2610.33477#S4.E22)\) withϵ^:=ϵ¯crit\+c​L∗\\hat\{\\epsilon\}:=\\bar\{\\epsilon\}\_\{\\rm crit\}\+cL\_\{\*\}in place ofϵcrit\\epsilon\_\{\\rm crit\}, and define ℰ:=\(1\+χ\)​\[ϵdist\+Rπ​\(4​Rr​τmix\+γ​ϵ¯crit\)\(1−γ\)​T\+c​DF\],Γ:=χ⁡\[Δ1c\+c​L2\]\+\(1\+χ\)​\[κ†​\(1\+2\)​σ^\+2​κ†​c​CF\+γ​Rπ​c​L∗\(1−γ\)​T\]\.\\begin\{split\}&\\mathcal\{E\}:=\(1\+\\chi\)\\Big\[\\epsilon\_\{\\rm dist\}\+\\frac\{R\_\{\\pi\}\(4R\_\{r\}\\tau\_\{\\mathrm\{mix\}\}\+\\gamma\\,\\bar\{\\epsilon\}\_\{\\rm crit\}\)\}\{\(1\-\\gamma\)T\}\+cD\_\{F\}\\Big\],\\\\ &\\Gamma:=\\chi\\Big\[\\frac\{\\Delta\_\{1\}\}\{c\}\+\\frac\{cL\}\{2\}\\Big\]\+\(1\+\\chi\)\\Big\[\\kappa\_\{\\dagger\}\(1\+\\sqrt\{2\}\)\\,\\hat\{\\sigma\}\+\\sqrt\{2\}\\,\\kappa\_\{\\dagger\}cC\_\{F\}\+\\frac\{\\gamma R\_\{\\pi\}cL\_\{\*\}\}\{\(1\-\\gamma\)T\}\\Big\]\.\\end\{split\}\(29\)Then the returned iterate satisfies𝔼⁡\[‖∇J​\(WU\)‖†\]≤ℰ\+ϵ\\mathbb\{E\}\[\\\|\\nabla J\(W\_\{U\}\)\\\|\_\{\\dagger\}\]\\leq\\mathcal\{E\}\+\\epsilonfor every horizonK≥max⁡\{\(Γ/ϵ\)3,\(c​T​Rπ\)3/2\}K\\geq\\max\\\{\(\\nicefrac\{\{\\Gamma\}\}\{\{\\epsilon\}\}\)^\{3\},\(cTR\_\{\\pi\}\)^\{3/2\}\\\}\.
- \(ii\)Under the conditions and the schedule of Theorem[12](https://arxiv.org/html/2610.33477#Thmtheorem12), letσ^u\\hat\{\\sigma\}\_\{u\}denoteσ¯\\bar\{\\sigma\}withϵ^u:=ϵ¯crit\+2​c​Rg​L∗\\hat\{\\epsilon\}\_\{u\}:=\\bar\{\\epsilon\}\_\{\\rm crit\}\+2cR\_\{g\}L\_\{\*\}in place ofϵcrit\\epsilon\_\{\\rm crit\}, and define ℰu:=2​\[ϵdist\+Rπ​\(4​Rr​τmix\+γ​ϵ¯crit\)\(1−γ\)​T\+2​c​DF​Rg\],Γu:=Λc\+\(2\+2\)​σ^u\+4​c​CF​Rg\+2​2​γ​Rπ​c​Rg​L∗\(1−γ\)​T\.\\begin\{split\}&\\mathcal\{E\}\_\{u\}:=\\sqrt\{2\}\\Big\[\\epsilon\_\{\\rm dist\}\+\\frac\{R\_\{\\pi\}\(4R\_\{r\}\\tau\_\{\\mathrm\{mix\}\}\+\\gamma\\,\\bar\{\\epsilon\}\_\{\\rm crit\}\)\}\{\(1\-\\gamma\)T\}\+2cD\_\{F\}R\_\{g\}\\Big\],\\\\ &\\Gamma\_\{u\}:=\\sqrt\{\\frac\{\\Lambda\}\{c\}\}\+\(2\+\\sqrt\{2\}\)\\,\\hat\{\\sigma\}\_\{u\}\+4cC\_\{F\}R\_\{g\}\+\\frac\{2\\sqrt\{2\}\\,\\gamma R\_\{\\pi\}cR\_\{g\}L\_\{\*\}\}\{\(1\-\\gamma\)T\}\.\\end\{split\}\(30\)Then the returned iterate satisfies𝔼⁡\[‖∇J​\(WU\)‖F\]≤ℰu\+ϵ\\mathbb\{E\}\[\\\|\\nabla J\(W\_\{U\}\)\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\]\\leq\\mathcal\{E\}\_\{u\}\+\\epsilonfor every horizonK≥max⁡\{\(Γu/ϵ\)4,\(c​L\)2\}K\\geq\\max\\\{\(\\nicefrac\{\{\\Gamma\_\{u\}\}\}\{\{\\epsilon\}\}\)^\{4\},\(cL\)^\{2\}\\\}\.

In both cases, Algorithm[1](https://arxiv.org/html/2610.33477#alg1)consumes\(K\+1\)​T\(K\+1\)Ttransitions, and the floorsℰ\\mathcal\{E\}andℰu\\mathcal\{E\}\_\{u\}do not depend onKK; Remark[14](https://arxiv.org/html/2610.33477#Thmtheorem14)gives the limit ofℰ\\mathcal\{E\}asT→∞T\\to\\infty\.

Proof\.\(i\) By Lemma[7](https://arxiv.org/html/2610.33477#Thmtheorem7)\(iv\) withℓ¯2=β=cK−2/3\\bar\{\\ell\}\_\{2\}=\\beta=cK^\{\-2/3\}, we haveϵcrit≤ϵ¯crit\+cL∗K−2/3≤ϵ^\\epsilon\_\{\\rm crit\}\\leq\\bar\{\\epsilon\}\_\{\\rm crit\}\+cL\_\{\*\}K^\{\-2/3\}\\leq\\hat\{\\epsilon\}such thatσ¯≤σ^\\bar\{\\sigma\}\\leq\\hat\{\\sigma\}sinceσ¯\\bar\{\\sigma\}is nondecreasing inϵcrit\\epsilon\_\{\\rm crit\}, and the floor term of \([26](https://arxiv.org/html/2610.33477#S4.E26)\) is at mostℰ\+\(1\+χ\)​γ​Rπ​c​L∗/\[\(1−γ\)​T​K2/3\]\\mathcal\{E\}\+\\nicefrac\{\{\(1\+\\chi\)\\gamma R\_\{\\pi\}cL\_\{\*\}\}\}\{\{\[\(1\-\\gamma\)TK^\{2/3\}\]\}\}\. Inserting both bounds into \([26](https://arxiv.org/html/2610.33477#S4.E26)\) and usingK−2/3≤K−1/3K^\{\-2/3\}\\leq K^\{\-1/3\}, we bound the transient byΓK−1/3≤ϵ\\Gamma K^\{\-1/3\}\\leq\\epsilon\.

\(ii\) By Lemma[22](https://arxiv.org/html/2610.33477#Thmtheorem22), every step satisfies‖Wk\+1−Wk‖2≤‖Wk\+1−Wk‖F≤2​β¯​Rg\\\|W\_\{k\+1\}\-W\_\{k\}\\\|\_\{2\}\\leq\\\|W\_\{k\+1\}\-W\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq 2\\bar\{\\beta\}R\_\{g\}\. Hence, Lemma[7](https://arxiv.org/html/2610.33477#Thmtheorem7)\(iv\) withℓ¯2=2β¯Rg=2cRgK−1/2\\bar\{\\ell\}\_\{2\}=2\\bar\{\\beta\}R\_\{g\}=2cR\_\{g\}K^\{\-1/2\}givesϵcrit≤ϵ¯crit\+2cRgL∗K−1/2≤ϵ^u\\epsilon\_\{\\rm crit\}\\leq\\bar\{\\epsilon\}\_\{\\rm crit\}\+2cR\_\{g\}L\_\{\*\}K^\{\-1/2\}\\leq\\hat\{\\epsilon\}\_\{u\}such thatσ¯≤σ^u\\bar\{\\sigma\}\\leq\\hat\{\\sigma\}\_\{u\}, and the floor term of \([28](https://arxiv.org/html/2610.33477#S4.E28)\) is at mostℰu\+2​2​γ​Rπ​c​Rg​L∗/\[\(1−γ\)​T​K1/2\]\\mathcal\{E\}\_\{u\}\+\\nicefrac\{\{2\\sqrt\{2\}\\gamma R\_\{\\pi\}cR\_\{g\}L\_\{\*\}\}\}\{\{\[\(1\-\\gamma\)TK^\{1/2\}\]\}\}\. Inserting both bounds into \([28](https://arxiv.org/html/2610.33477#S4.E28)\) and usingK−1/2≤K−1/4K^\{\-1/2\}\\leq K^\{\-1/4\}, we bound the transient byΓuK−1/4≤ϵ\\Gamma\_\{u\}K^\{\-1/4\}\\leq\\epsilon\.

In both cases, the identity𝔼⁡\[‖∇J​\(WU\)‖†\]=1K​∑k=1K𝔼⁡\[‖∇J​\(Wk\)‖†\]\\mathbb\{E\}\[\\\|\\nabla J\(W\_\{U\}\)\\\|\_\{\\dagger\}\]=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\[\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\\dagger\}\], which also holds in the Frobenius norm, completes the proof since every frame consists ofTTtransitions\.

## 5Instances

In this section, we apply Theorems[11](https://arxiv.org/html/2610.33477#Thmtheorem11)and[12](https://arxiv.org/html/2610.33477#Thmtheorem12)to the four actor updates of Table[1](https://arxiv.org/html/2610.33477#S3.T1)\. The detailed proofs can be found in Appendix[M](https://arxiv.org/html/2610.33477#A13)\. Since all instances share the inverse\-free LSTD critic, the PG estimator, and the ICM, we compare them in the nuclear norm\.

### 5\.1Muon Actor

A vanilla SGD actor moves alongGkG\_\{k\}, which can be dominated by a few large singular directions\([Jordan et al\., 2024](https://arxiv.org/html/2610.33477#bib.bib22)\)\. Different from the vanilla SGD actor, the Muon actor moves along an approximation of the polar factor ofMkM\_\{k\}by the NSO recursions \([13](https://arxiv.org/html/2610.33477#S3.E13)\), whoseκ2\\kappa\_\{2\}th\-order NSO polynomial keeps the nonzero singular values of every iterate in\(0,1\]\(0,1\]\. The following lemma establishes the convergence and alignment of finite NSO recursions with proof relegated to Appendix[M](https://arxiv.org/html/2610.33477#A13)\.

###### Lemma 15\(Convergence and alignment of finite NSO\)\.

Let

f⁡\(x\):=x​∑i=0κ2−14−i​\(2​ii\)​\(1−x2\)i,sP:=supx∈\[0,1\]x⁡\(1−fP​\(x\)\)f\(x\):=x\\sum\_\{i=0\}^\{\\kappa\_\{2\}\-1\}4^\{\-i\}\\binom\{2i\}\{i\}\(1\-x^\{2\}\)^\{i\},\\qquad s\_\{P\}:=\\sup\_\{x\\in\[0,1\]\}x\\,\\big\(1\-f^\{P\}\(x\)\\big\)wherefPf^\{P\}denotes thePP\-fold composition offf\. ThensP≤\(2eκ2P\)−1/2s\_\{P\}\\leq\(2e\\,\\kappa\_\{2\}^\{P\}\)^\{\-1/2\}, and, for everykkand every realization, the NSO iterates in \([13](https://arxiv.org/html/2610.33477#S3.E13)\) satisfy the following\.

- \(i\)IfMk≠0M\_\{k\}\\neq 0, every iterateNk,pN\_\{k,p\}keeps the singular vectors and the rank ofMkM\_\{k\}, has the singular valuesfp​\(σi​\(Mk\)/‖Mk‖F\)f^\{p\}\(\\sigma\_\{i\}\(M\_\{k\}\)/\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\), and satisfies‖Nk,p‖2≤1\\\|N\_\{k,p\}\\\|\_\{2\}\\leq 1, andNk,pN\_\{k,p\}converges to the polar factorΘk\\Theta\_\{k\}ofMkM\_\{k\}locally with orderκ2\\kappa\_\{2\}inpp\. Moreover, withak:=σmin\+​\(Mk\)/‖Mk‖Fa\_\{k\}:=\\sigma\_\{\\min\}^\{\+\}\(M\_\{k\}\)/\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}, the polar residualRk:=Nk,P2−ΘkR\_\{k\}:=N\_\{k,P\_\{2\}\}\-\\Theta\_\{k\}satisfies ‖Rk‖2=1−σmin\+​\(Nk,P2\)≤\(1−ak2\)κ2P2<1\.\\\|R\_\{k\}\\\|\_\{2\}=1\-\\sigma\_\{\\min\}^\{\+\}\(N\_\{k,P\_\{2\}\}\)\\leq\\big\(1\-a\_\{k\}^\{2\}\\big\)^\{\\kappa\_\{2\}^\{P\_\{2\}\}\}<1\.IfMk=0M\_\{k\}=0, thenNk,p=0N\_\{k,p\}=0, and we setΘk=Rk=0\\Theta\_\{k\}=R\_\{k\}=0\.
- \(ii\)Withr=min⁡\{m,n\}r=\\min\\\{m,n\\\}, the NSO output is aligned with the momentum as ⟨Mk,Nk,P2⟩F≥\(1−r​sP2\)​‖Mk‖∗\\langle M\_\{k\},N\_\{k,P\_\{2\}\}\\rangle\_\{\\scriptscriptstyle\\textrm\{F\}\}\\geq\(1\-r\\,s\_\{P\_\{2\}\}\)\\\|M\_\{k\}\\\|\_\{\*\}\(31\)whereκ2P2≥4​r2\\kappa\_\{2\}^\{P\_\{2\}\}\\geq 4r^\{2\}implies⟨Mk,Nk,P2⟩F≥0\.78​‖Mk‖∗\\langle M\_\{k\},N\_\{k,P\_\{2\}\}\\rangle\_\{\\scriptscriptstyle\\textrm\{F\}\}\\geq 0\.78\\,\\\|M\_\{k\}\\\|\_\{\*\}\.

By Lemma[15](https://arxiv.org/html/2610.33477#Thmtheorem15), the Muon actor update is normalized in the spectral norm with the alignment factorχ=1/\(1−r​sP2\)\\chi=\\nicefrac\{\{1\}\}\{\{\(1\-r\\,s\_\{P\_\{2\}\}\)\}\}wheneverr​sP2<1r\\,s\_\{P\_\{2\}\}<1, i\.e\., with the constants of Table[1](https://arxiv.org/html/2610.33477#S3.T1)\. Note that Lemma[15](https://arxiv.org/html/2610.33477#Thmtheorem15)requires no conditioning assumption on the momentum, which is imposed by the finite\-NSO analysis of[Kim and Oh \(2026\)](https://arxiv.org/html/2610.33477#bib.bib28)\.

###### Corollary 16\(Muon\)\.

Under Assumptions[1](https://arxiv.org/html/2610.33477#Thmassump1)and[2](https://arxiv.org/html/2610.33477#Thmassump2), letP2P\_\{2\}satisfyr​sP2<1r\\,s\_\{P\_\{2\}\}<1\(e\.g\.,κ2P2≥4​r2\\kappa\_\{2\}^\{P\_\{2\}\}\\geq 4r^\{2\}\) withsP2s\_\{P\_\{2\}\}of Lemma[15](https://arxiv.org/html/2610.33477#Thmtheorem15), and letχ:=1/\(1−r​sP2\)\\chi:=\\nicefrac\{\{1\}\}\{\{\(1\-r\\,s\_\{P\_\{2\}\}\)\}\}\(henceχ≤1\.28\\chi\\leq 1\.28ifκ2P2≥4​r2\\kappa\_\{2\}^\{P\_\{2\}\}\\geq 4r^\{2\}\)\. Then Algorithm[1](https://arxiv.org/html/2610.33477#alg1)with the Muon step \([13](https://arxiv.org/html/2610.33477#S3.E13)\) satisfies \([25](https://arxiv.org/html/2610.33477#S4.E25)\) with∥⋅∥†=∥⋅∥∗\\\|\\cdot\\\|\_\{\\dagger\}=\\\|\\cdot\\\|\_\{\*\}andκ†=r1/2\\kappa\_\{\\dagger\}=r^\{1/2\}under the conditions of Theorem[11](https://arxiv.org/html/2610.33477#Thmtheorem11); in particular, for anyc\>0c\>0andK≥\(c​T​Rπ\)3/2K\\geq\(cTR\_\{\\pi\}\)^\{3/2\}, the choice1−η=K−2/31\-\\eta=K^\{\-2/3\}andβ=cK−2/3\\beta=cK^\{\-2/3\}yields

1K​∑k=1K𝔼⁡\[‖∇J​\(Wk\)‖∗\]≤χ⁡\[Δ1c​1K1/3\+c​L2​K2/3\]\+\(1\+χ\)​r1/2​\[\(1\+2\)​σ¯\+2​c​CF\]​1K1/3\+\(1\+χ\)​\(ϵbias\+c​DF\)\\begin\{split\}&\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\[\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\*\}\]\\leq\\chi\\Big\[\\frac\{\\Delta\_\{1\}\}\{c\}\\,\\frac\{1\}\{K^\{1/3\}\}\+\\frac\{cL\}\{2K^\{2/3\}\}\\Big\]\\\\ &\\quad\+\(1\+\\chi\)\\,r^\{1/2\}\\big\[\(1\+\\sqrt\{2\}\)\\,\\bar\{\\sigma\}\+\\sqrt\{2\}\\,cC\_\{F\}\\big\]\\frac\{1\}\{K^\{1/3\}\}\+\(1\+\\chi\)\\big\(\\epsilon\_\{\\rm bias\}\+cD\_\{F\}\\big\)\\end\{split\}\(32\)and the guarantee of Corollary[13](https://arxiv.org/html/2610.33477#Thmtheorem13)\(i\) holds in the nuclear norm withκ†=r1/2\\kappa\_\{\\dagger\}=r^\{1/2\}\.

The NSI approximation in the critic enters only throughϵcrit\\epsilon\_\{\\rm crit\}, whereas the NSO approximation in the actor enters multiplicatively throughχ≤1\.28\\chi\\leq 1\.28onceκ2P2≥4​r2\\kappa\_\{2\}^\{P\_\{2\}\}\\geq 4r^\{2\}, i\.e\., after𝒪⁡\(log⁡r\)\{\\cal O\}\(\\log r\)GEMM\-only NSO recursions\. Moreover, ifσmin\+​\(Mk\)/‖Mk‖F\\nicefrac\{\{\\sigma\_\{\\min\}^\{\+\}\(M\_\{k\}\)\}\}\{\{\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\}\}is bounded away from zero, the gap between the realized alignment and one decays doubly exponentially inP2P\_\{2\}since⟨Mk,Nk,P2⟩F≥\(1−‖Rk‖2\)​‖Mk‖∗\\langle M\_\{k\},N\_\{k,P\_\{2\}\}\\rangle\_\{\\scriptscriptstyle\\textrm\{F\}\}\\geq\(1\-\\\|R\_\{k\}\\\|\_\{2\}\)\\\|M\_\{k\}\\\|\_\{\*\}\. In \([32](https://arxiv.org/html/2610.33477#S5.E32)\), only the noise and correction terms carry the factorr1/2r^\{1/2\}due to the three facts: \(i\) the ascent of the Muon step along the momentum is at leastβ​‖Mk‖∗/χ\\nicefrac\{\{\\beta\\\|M\_\{k\}\\\|\_\{\*\}\}\}\{\{\\chi\}\}, and \([15](https://arxiv.org/html/2610.33477#S4.E15)\) pairs the score and its change with the spectral norm of the step such that the optimization term and the constantsCFC\_\{F\},DFD\_\{F\}, andLLare free ofrr; \(ii\) the bias part of the tracking error, which is controlled in the nuclear norm, is paired with‖Nk,P2‖2≤1\\\|N\_\{k,P\_\{2\}\}\\\|\_\{2\}\\leq 1; and \(iii\) only the noise part, which is averaged in the Frobenius norm, is converted by‖ekn‖∗≤r1/2​‖ekn‖F\\\|e^\{\\rm n\}\_\{k\}\\\|\_\{\*\}\\leq r^\{1/2\}\\\|e^\{\\rm n\}\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\. Note that the mismatchϵdist\\epsilon\_\{\\rm dist\}can exceedsupW‖B¯​\(W\)‖F\\sup\_\{W\}\\\|\\bar\{B\}\(W\)\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}by up tor1/2r^\{1/2\}, whereas its a priori bound is free ofrr\. Hence, the noise term fixes the orderr1/2r^\{1/2\}of the transient for every stepsize constant, andcconly shiftsr1/2r^\{1/2\}between the optimization and correction terms\. More specifically, for a dimension\-free constantc0c\_\{0\}, the choicec=c0c=c\_\{0\}makes the optimization and second\-order terms, the floor, and the condition onKKfree ofrr, whereasc=c0/r1/2c=\\nicefrac\{\{c\_\{0\}\}\}\{\{r^\{1/2\}\}\}makes the correction term free ofrr; for both choices, the horizon of Corollary[13](https://arxiv.org/html/2610.33477#Thmtheorem13)\(i\) is of orderr3/2​ϵ−3r^\{3/2\}\\epsilon^\{\-3\}\.

### 5\.2NSGDM Actor

The NSGDM stepβ​Mk/‖Mk‖F\\nicefrac\{\{\\beta M\_\{k\}\}\}\{\{\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\}\}is the normalized momentum of[Cutkosky and Mehta \(2020\)](https://arxiv.org/html/2610.33477#bib.bib11)with our importance correction, and is closely related to the normalized variance\-reduced PG method of[Barakat et al\. \(2023\)](https://arxiv.org/html/2610.33477#bib.bib3)that uses Monte Carlo episodes restarted from the initial distribution without a critic\. Since⟨Mk,Mk/‖Mk‖F⟩F=‖Mk‖F\\langle M\_\{k\},\\nicefrac\{\{M\_\{k\}\}\}\{\{\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\}\}\\rangle\_\{\\scriptscriptstyle\\textrm\{F\}\}=\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}, the NSGDM actor update is normalized in the self\-dual Frobenius norm withχ=κ†=1\\chi=\\kappa\_\{\\dagger\}=1\.

###### Corollary 17\(NSGDM\)\.

Under Assumptions[1](https://arxiv.org/html/2610.33477#Thmassump1)and[2](https://arxiv.org/html/2610.33477#Thmassump2), Algorithm[1](https://arxiv.org/html/2610.33477#alg1)with the NSGDM step satisfies \([25](https://arxiv.org/html/2610.33477#S4.E25)\) with∥⋅∥†=∥⋅∥F\\\|\\cdot\\\|\_\{\\dagger\}=\\\|\\cdot\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}andχ=κ†=1\\chi=\\kappa\_\{\\dagger\}=1under the conditions of Theorem[11](https://arxiv.org/html/2610.33477#Thmtheorem11); in particular, for anyc\>0c\>0andK≥\(c​T​Rπ\)3/2K\\geq\(cTR\_\{\\pi\}\)^\{3/2\}, the choice1−η=K−2/31\-\\eta=K^\{\-2/3\}andβ=cK−2/3\\beta=cK^\{\-2/3\}yields

1K∑k=1K𝔼\[∥∇J\(Wk\)∥∗\]≤r1/2\[\(Δ1c\+2​\(1\+2\)​σ¯\+2​2​c​CF\)​1K1/3\+c​L2​K2/3\+2\(ϵbias\+cDF\)\]\\begin\{split\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\[\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\*\}\]\\leq r^\{1/2\}\\bigg\[&\\Big\(\\frac\{\\Delta\_\{1\}\}\{c\}\+2\(1\+\\sqrt\{2\}\)\\,\\bar\{\\sigma\}\+2\\sqrt\{2\}\\,cC\_\{F\}\\Big\)\\frac\{1\}\{K^\{1/3\}\}\\\\ &\+\\frac\{cL\}\{2K^\{2/3\}\}\+2\\big\(\\epsilon\_\{\\rm bias\}\+cD\_\{F\}\\big\)\\bigg\]\\end\{split\}\(33\)where the bracket bounds1K​∑k=1K𝔼⁡\[‖∇J​\(Wk\)‖F\]\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\[\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\]\.

### 5\.3SGDM and AdaNorm Actors

The SGDM stepβ​Mk\\beta M\_\{k\}is unnormalized withβ¯=β\\bar\{\\beta\}=\\beta,ϑ=1\\vartheta=1, andVu=0V\_\{u\}=0\. Since the accumulator of the AdaNorm step \([14](https://arxiv.org/html/2610.33477#S3.E14)\) withα≤1/\(2​K\)\\alpha\\leq\\nicefrac\{\{1\}\}\{\{\(2K\)\}\}stays in\[n0/2,n0\+Rg2/2\]\[\\nicefrac\{\{n\_\{0\}\}\}\{\{2\}\},n\_\{0\}\+\\nicefrac\{\{R\_\{g\}^\{2\}\}\}\{\{2\}\}\]and varies little, the AdaNorm step is unnormalized with the constants of Table[1](https://arxiv.org/html/2610.33477#S3.T1)\(Appendix[M](https://arxiv.org/html/2610.33477#A13)\)\.

###### Corollary 18\(SGDM and AdaNorm\)\.

Under Assumptions[1](https://arxiv.org/html/2610.33477#Thmassump1)and[2](https://arxiv.org/html/2610.33477#Thmassump2), letβ¯:=β\\bar\{\\beta\}:=\\betafor the SGDM step andβ¯:=β​2/n0\\bar\{\\beta\}:=\\beta\\sqrt\{\\nicefrac\{\{2\}\}\{\{n\_\{0\}\}\}\}for the AdaNorm step with aKK\-independentn0\>0n\_\{0\}\>0andα∈\[0,1/\(2​K\)\]\\alpha\\in\[0,\\nicefrac\{\{1\}\}\{\{\(2K\)\}\}\], and letΛ:=2​Δ1\\Lambda:=2\\Delta\_\{1\}for SGDM andΛ:=2​ϑ​Δ1\+\(ϑ−1\)​Δmax\\Lambda:=2\\sqrt\{\\vartheta\}\\,\\Delta\_\{1\}\+\(\\vartheta\-1\)\\Delta\_\{\\max\}withϑ:=2\+Rg2/n0\\vartheta:=2\+\\nicefrac\{\{R\_\{g\}^\{2\}\}\}\{\{n\_\{0\}\}\}for AdaNorm\. Then Algorithm[1](https://arxiv.org/html/2610.33477#alg1)with either step satisfies \([27](https://arxiv.org/html/2610.33477#S4.E27)\), multiplied byr1/2r^\{1/2\}, in the nuclear norm under the conditions of Theorem[12](https://arxiv.org/html/2610.33477#Thmtheorem12); in particular, for any0<c≤1/\(2​CF\)0<c\\leq\\nicefrac\{\{1\}\}\{\{\(2C\_\{F\}\)\}\}andK≥\(c​L\)2K\\geq\(cL\)^\{2\}, the choice1−η=K−1/21\-\\eta=K^\{\-1/2\}andβ¯=cK−1/2\\bar\{\\beta\}=cK^\{\-1/2\}, i\.e\.,β=cK−1/2\\beta=cK^\{\-1/2\}for SGDM andβ=cn0/2K−1/2\\beta=c\\sqrt\{\\nicefrac\{\{n\_\{0\}\}\}\{\{2\}\}\}\\,K^\{\-1/2\}for AdaNorm, yields

1K​∑k=1K𝔼⁡\[‖∇J​\(Wk\)‖∗\]≤r1/2​\[\(Λc\+\(2\+2\)​σ¯\+4​c​CF​Rg\)​1K1/4\+2​\(ϵbias\+2​c​DF​Rg\)\]\.\\begin\{split\}&\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\[\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\*\}\]\\\\ &\\leq r^\{1/2\}\\Big\[\\Big\(\\sqrt\{\\frac\{\\Lambda\}\{c\}\}\+\(2\+\\sqrt\{2\}\)\\bar\{\\sigma\}\+4cC\_\{F\}R\_\{g\}\\Big\)\\frac\{1\}\{K^\{1/4\}\}\+\\sqrt\{2\}\\big\(\\epsilon\_\{\\rm bias\}\+2cD\_\{F\}R\_\{g\}\\big\)\\Big\]\.\\end\{split\}\(34\)

The AdaNorm bound is the SGDM bound withΛ≥2​2​Δ1\+Δmax\\Lambda\\geq 2\\sqrt\{2\}\\,\\Delta\_\{1\}\+\\Delta\_\{\\max\}in place of2​Δ12\\Delta\_\{1\}, which accounts for the variation of the effective stepsizeβ/nk∈\[β¯/ϑ,β¯\]\\nicefrac\{\{\\beta\}\}\{\{\\sqrt\{n\_\{k\}\}\}\}\\in\[\\nicefrac\{\{\\bar\{\\beta\}\}\}\{\{\\sqrt\{\\vartheta\}\}\},\\bar\{\\beta\}\]through the summation by parts\. Note that our analysis covers AdaNorm in its stabilized regimeα≤1/\(2​K\)\\alpha\\leq\\nicefrac\{\{1\}\}\{\{\(2K\)\}\}, in which the effective stepsize varies within the factorϑ\\sqrt\{\\vartheta\}\. Otherwise, aKK\-independentα\\alphaallowsnkn\_\{k\}to decay as\(1−α\)k​n0\(1\-\\alpha\)^\{k\}n\_\{0\}such that the effective stepsizes, the momentum, and the importance weights admit no deterministic bound independent ofKK\.

### 5\.4Comparison

Table 2:Bounds on1K​∑k=1K𝔼⁡\[‖∇J​\(Wk\)‖∗\]\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\[\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\*\}\]term by term \(Corollaries[16](https://arxiv.org/html/2610.33477#Thmtheorem16)–[18](https://arxiv.org/html/2610.33477#Thmtheorem18)\)\.For fixedTTandλ\\lambdaand a dimension\-free stepsize constantcc, the optimization, noise, and importance\-correction entries multiply the rate, and the second\-order entries multiplyK−2/3K^\{\-2/3\}\.

aAnyc\>0c\>0withK≥\(c​T​Rπ\)3/2K\\geq\(cTR\_\{\\pi\}\)^\{3/2\}; for Muon,χ≤1\.28\\chi\\leq 1\.28onceκ2P2≥4​r2\\kappa\_\{2\}^\{P\_\{2\}\}\\geq 4r^\{2\}, i\.e\., after𝒪⁡\(log⁡r\)\{\\cal O\}\(\\log r\)NSO recursions per update\.

bWithc≤1/\(2​CF\)c\\leq\\nicefrac\{\{1\}\}\{\{\(2C\_\{F\}\)\}\}andK≥\(c​L\)2K\\geq\(cL\)^\{2\}, which absorbs the second\-order term\.

cAll floors are stated with the nuclear\-norm mismatchϵdist\\epsilon\_\{\\rm dist\}; the Frobenius analyses behind the last three rows could use the possibly smallersupW‖B¯​\(W\)‖F\\sup\_\{W\}\\\|\\bar\{B\}\(W\)\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}instead\.

Since the four actors share the critic, the frames, the PG estimator, and the ICM, the differences among the bounds in Table[2](https://arxiv.org/html/2610.33477#S5.T2)stem from the actor updates up to the evaluation ofσ¯\\bar\{\\sigma\}andϵbias\\epsilon\_\{\\rm bias\}along the respective iterates and the constant factors, e\.g\.,2\\sqrt\{2\}and1\+χ1\+\\chi, that reflect the proof technique\.

Rate\.The bounds decay as𝒪\(K−1/3\)\{\\cal O\}\(K^\{\-1/3\}\)for Muon and NSGDM and as𝒪\(K−1/4\)\{\\cal O\}\(K^\{\-1/4\}\)for SGDM and AdaNorm\. Since all bounds contain the driftη​DF​β/\(1−η\)\\nicefrac\{\{\\eta D\_\{F\}\\beta\}\}\{\{\(1\-\\eta\)\}\}of the frame bias \(cf\. \([25](https://arxiv.org/html/2610.33477#S4.E25)\)\), they requireβ=𝒪⁡\(1−η\)\\beta=\{\\cal O\}\(1\-\\eta\)\. Moreover, the SGDM and AdaNorm steps, which scale withMkM\_\{k\}, additionally require2​η​CF​β¯≤1−η2\\eta C\_\{F\}\\bar\{\\beta\}\\leq 1\-\\etaand2​T​Rπ​Rg​β¯≤12TR\_\{\\pi\}R\_\{g\}\\bar\{\\beta\}\\leq 1to keep‖Mk‖F≤2​Rg\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq 2R\_\{g\}and the importance weights bounded \(Lemma[22](https://arxiv.org/html/2610.33477#Thmtheorem22)\), whereas the Muon and NSGDM actors admit anyc\>0c\>0\. Under the proportional schedule, the descent inequalities of normalized actor updates, which are linear in the tracking error, balanceΔ1/\(β​K\)\\nicefrac\{\{\\Delta\_\{1\}\}\}\{\{\(\\beta K\)\}\}against1−η​σ¯\\sqrt\{1\-\\eta\}\\,\\bar\{\\sigma\}atK−1/3K^\{\-1/3\}, whereas those of unnormalized actor updates, which are quadratic in the gradient norm, balanceΔ1/\(β​K\)\\nicefrac\{\{\\Delta\_\{1\}\}\}\{\{\(\\beta K\)\}\}against\(1−η\)​σ¯2\(1\-\\eta\)\\bar\{\\sigma\}^\{2\}atK−1/2K^\{\-1/2\}for the squared norm\. Hence, the better exponent in these bounds is due to normalization, which the Muon actor update shares with NSGDM\. Note that the coupling1−η∝β21\-\\eta\\propto\\beta^\{2\}behind the complexity𝒪⁡\(ϵ−3\)\{\\cal O\}\(\\epsilon^\{\-3\}\)of the unnormalized ICM actor with unbiased estimates\([Huang et al\., 2020](https://arxiv.org/html/2610.33477#bib.bib21)\)is unavailable here since it lets the drift of the frame bias grow asDF/β\\nicefrac\{\{D\_\{F\}\}\}\{\{\\beta\}\}\. Besides, without the importance correction, both normalized bounds decay only as𝒪\(K−1/4\)\{\\cal O\}\(K^\{\-1/4\}\)\(Remark[9](https://arxiv.org/html/2610.33477#Thmtheorem9)\); therefore, the faster rate in these upper bounds requires both normalization and the importance correction\.

Dimension dependence\.All four transients carry the factorr1/2r^\{1/2\}through their noise terms, whereas the bounds separate term by term\. More specifically, the SGDM, AdaNorm, and NSGDM analyses bound the averaged Frobenius norm of the PG, in which every quantity other thanϵdist\\epsilon\_\{\\rm dist\}is free ofrr, and convert the result to the nuclear norm as a whole, whereas only the noise and correction terms of the Muon bound carryr1/2r^\{1/2\}with a dimension\-freeccandκ2P2≥4​r2\\kappa\_\{2\}^\{P\_\{2\}\}\\geq 4r^\{2\}\. It is worth stressing that the descent arguments that control the bias only through its norm cannot remove the factorr1/2r^\{1/2\}of the baseline floors: a tracking error of nuclear normbbcan cancel the first\-order progress of the SGDM, AdaNorm, and NSGDM steps although‖∇J​\(Wk\)‖∗\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\*\}is of the orderr1/2​br^\{1/2\}b, whereas it cannot cancel the progress of the polar factor once‖∇J​\(Wk\)‖∗\>2​b\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\*\}\>2b\. At the samecc, the NSGDM row is the Muon row withr1/2r^\{1/2\}in place ofχ\\chion the optimization and second\-order terms,22in place of1\+χ1\+\\chion the noise and correction terms, and2​r1/22r^\{1/2\}in place of1\+χ1\+\\chion the floor\. Hence, when NSGDM uses a dimension\-free constantc0c\_\{0\}, the Muon bound is no larger than the NSGDM bound term by term up to the factorχ\\chifor everyc∈\[c0/r1/2,c0\]c\\in\[\\nicefrac\{\{c\_\{0\}\}\}\{\{r^\{1/2\}\}\},c\_\{0\}\]: the noise terms coincide up to\(1\+χ\)/2\\nicefrac\{\{\(1\+\\chi\)\}\}\{\{2\}\}; atc=c0c=c\_\{0\}, the optimization and second\-order terms are smaller byr1/2r^\{1/2\}; atc=c0/r1/2c=\\nicefrac\{\{c\_\{0\}\}\}\{\{r^\{1/2\}\}\}, the correction and second\-order terms are smaller byr1/2r^\{1/2\}andrr, respectively; and the floor with the common mismatchϵdist\\epsilon\_\{\\rm dist\}is smaller byr1/2r^\{1/2\}\. With each constant tuned to minimize the sum of the optimization and correction terms, these terms scale asr1/4r^\{1/4\}for Muon and asr1/2r^\{1/2\}for the other actors\.

When orthogonalization helps\.The advantage of Muon is stated in the nuclear norm, i\.e\., the dual norm of the spectral norm in which the Muon actor update is normalized, and its size depends on the curvature constants\. In the Frobenius norm, the roles reverse: the bracket of \([33](https://arxiv.org/html/2610.33477#S5.E33)\) carries no factorr1/2r^\{1/2\}, whereas the Muon bound keepsr1/2r^\{1/2\}on its noise and correction terms such that the NSGDM bound is the smaller one\. Moreover, NSGDM needs only the Frobenius counterparts of \([15](https://arxiv.org/html/2610.33477#S4.E15)\) and \([17](https://arxiv.org/html/2610.33477#S4.E17)\), whose constants can be smaller; even for scores of rank one, the curvature ofJJ, which averages over states, can differ between the two norms\. LetL2L\_\{2\}andLFL\_\{F\}denote the smallest constants for which \([17](https://arxiv.org/html/2610.33477#S4.E17)\) holds withL2/2​‖W′−W‖22\\nicefrac\{\{L\_\{2\}\}\}\{\{2\}\}\\\|W^\{\\prime\}\-W\\\|\_\{2\}^\{2\}and withLF/2​‖W′−W‖F2\\nicefrac\{\{L\_\{F\}\}\}\{\{2\}\}\\\|W^\{\\prime\}\-W\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}, respectively, such thatLF≤L2≤min⁡\{L,r​LF\}L\_\{F\}\\leq L\_\{2\}\\leq\\min\\\{L,rL\_\{F\}\\\}\. Without noise and bias, an exact polar step with the best stepsize gains at least‖∇J​\(W\)‖∗2/\(2​L2\)\\nicefrac\{\{\\\|\\nabla J\(W\)\\\|\_\{\*\}^\{2\}\}\}\{\{\(2L\_\{2\}\)\}\}, and an exact NSGDM step gains at least‖∇J​\(W\)‖F2/\(2​LF\)\\nicefrac\{\{\\\|\\nabla J\(W\)\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\}\}\{\{\(2L\_\{F\}\)\}\}\. Therefore, orthogonalization guarantees the larger gain if and only if the nuclear rank‖∇J​\(W\)‖∗2/‖∇J​\(W\)‖F2\\nicefrac\{\{\\\|\\nabla J\(W\)\\\|\_\{\*\}^\{2\}\}\}\{\{\\\|\\nabla J\(W\)\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\}\}of the PG exceeds the curvature ratioL2/LF\\nicefrac\{\{L\_\{2\}\}\}\{\{L\_\{F\}\}\}, where both ratios lie in\[1,r\]\[1,r\]\. For example,W↦−12​tr​\(W​Σ​W⊤\)W\\mapsto\-\\frac\{1\}\{2\}\\mathrm\{tr\}\(W\\Sigma W^\{\\top\}\)with a positive semidefiniteΣ∈ℝn×n\\Sigma\\in\\mathbb\{R\}^\{n\\times n\}andm≥nm\\geq nhasL2=tr⁡\(Σ\)L\_\{2\}=\\mathrm\{tr\}\(\\Sigma\)andLF=λmax​\(Σ\)L\_\{F\}=\\lambda\_\{\\max\}\(\\Sigma\), whose ratio, i\.e\., the effective rank ofΣ\\Sigma, can be as large asrr\. Hence, orthogonalization pays off for the PGs whose nuclear rank exceeds the curvature ratio, and the above term\-by\-term advantage with a common stepsize constant holds for the upper bounds in the nuclear criterion\.

Computation and frame length\.The actor evaluateslog⁡π\\log\\piand∇log⁡π\\nabla\\log\\piatWkW\_\{k\}andWk−1W\_\{k\-1\}on the frame\. Orthogonalization costs the factorχ≤1\.28\\chi\\leq 1\.28and𝒪⁡\(log⁡r\)\{\\cal O\}\(\\log r\)NSO recursions per update, i\.e\.,𝒪⁡\(P2​κ2​m​n​r\)\{\\cal O\}\(P\_\{2\}\\kappa\_\{2\}mnr\)operations in total when the Gram matrix is formed on the smaller side, against𝒪⁡\(m​n\)\{\\cal O\}\(mn\)operations for the other actor updates\. Note that such an overhead is dominated by the cost of forming the frame estimates, which is at least of orderT​m​nTmnfor a dense layer, onceT≥P2​κ2​rT\\geq P\_\{2\}\\kappa\_\{2\}r\. All actors consume the same frames and share the critic of Section[3\.1](https://arxiv.org/html/2610.33477#S3.SS1), which costs𝒪⁡\(d​T2\+κ1​P1​T3\)\{\\cal O\}\(dT^\{2\}\+\\kappa\_\{1\}P\_\{1\}T^\{3\}\)operations per update, i\.e\.,𝒪⁡\(d​T2\)\{\\cal O\}\(dT^\{2\}\)operations to form theT×TT\\times TmatrixYk⊤​ΦkY\_\{k\}^\{\\top\}\\Phi\_\{k\}and𝒪⁡\(κ1​T3\)\{\\cal O\}\(\\kappa\_\{1\}T^\{3\}\)operations per NSI recursion\.

A longer frame reduces the context\-dependent bias as1/T\\nicefrac\{\{1\}\}\{\{T\}\}and the noise bound asT−1/2T^\{\-1/2\}, whereas it increasesCFC\_\{F\}and tightens the step conditionT​Rπ​β≤1TR\_\{\\pi\}\\beta\\leq 1\. Moreover, each update consumesTTtransitions\. Since the ridge\-regularized pathwise LSTD critic is well posed for everyλ\>0\\lambda\>0andT≥1T\\geq 1, the choice ofTTis governed by such a bias–cost tradeoff\.

ForΔ1\>0\\Delta\_\{1\}\>0, the sufficient horizon of Corollary[13](https://arxiv.org/html/2610.33477#Thmtheorem13)\(i\) grows at least asT3/2​ϵ−3T^\{3/2\}\\epsilon^\{\-3\}for everycc\(Appendix[N\.2](https://arxiv.org/html/2610.33477#A14.SS2)\), whereas the capc≤1/\(2​CF\)c\\leq\\nicefrac\{\{1\}\}\{\{\(2C\_\{F\}\)\}\}makes that of Corollary[13](https://arxiv.org/html/2610.33477#Thmtheorem13)\(ii\) grow at least asT2​ϵ−4T^\{2\}\\epsilon^\{\-4\}\. Therefore, normalized actor updates also scale better with the frame length\.

## 6Conclusions

In this work, we have established the first finite\-time convergence rates for ICM actors that learn along a single Markovian trajectory with an inverse\-free ridge\-regularized pathwise LSTD critic\. Up to the stationary\-to\-discounted mismatch and an𝒪⁡\(1/T\)\{\\cal O\}\(\\nicefrac\{\{1\}\}\{\{T\}\}\)frame bias, the averaged gradient norm decays as𝒪\(K−1/3\)\{\\cal O\}\(K^\{\-1/3\}\)for normalized actor updates and as𝒪\(K−1/4\)\{\\cal O\}\(K^\{\-1/4\}\)for unnormalized ones\. Among the instances, Muon with a dimension\-free stepsize confines the factorr1/2r^\{1/2\}to the noise and correction terms, and orthogonalization guarantees a larger noiseless ascent than normalization exactly when the nuclear rank of the PG exceeds the spectral\-to\-Frobenius curvature ratio\. Future work includes the sampling protocols that remove the mismatch, the multilevel frames\([Dorfman and Levy, 2022](https://arxiv.org/html/2610.33477#bib.bib16);[Suttle et al\., 2023](https://arxiv.org/html/2610.33477#bib.bib56)\)that lower the transition cost per update, and the empirical comparisons of the actor updates at equal transition budgets\.

## Appendix ANotation and Stationarity Criterion

Lowercase letters denote vectors and uppercase letters matrices, unless they denote scalars such asdd,mm,nn,TT,KK, andLL;IdI\_\{d\}is thed×dd\\times didentity,\[K\]:=\{1,…,K\}\[K\]:=\\\{1,\\ldots,K\\\},𝟏​\{⋅\}\\mathbf\{1\}\\\{\\cdot\\\}is the indicator of an event, and\(⋅\)⊤\(\\cdot\)^\{\\top\}is the transpose\. For a vector,∥⋅∥2\\\|\\cdot\\\|\_\{2\}is the Euclidean norm; for a matrix,∥⋅∥2\\\|\\cdot\\\|\_\{2\},∥⋅∥F\\\|\\cdot\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}, and∥⋅∥∗\\\|\\cdot\\\|\_\{\*\}are the spectral, Frobenius, and nuclear norms,⟨⋅,⋅⟩F\\langle\\cdot,\\cdot\\rangle\_\{\\scriptscriptstyle\\textrm\{F\}\}is the Frobenius inner product,σi​\(⋅\)\\sigma\_\{i\}\(\\cdot\),σmin​\(⋅\)\\sigma\_\{\\min\}\(\\cdot\), andσmax​\(⋅\)\\sigma\_\{\\max\}\(\\cdot\)are theiith largest, smallest, and largest singular values, andσmin\+​\(⋅\)\\sigma\_\{\\min\}^\{\+\}\(\\cdot\)is the smallest nonzero singular value of a nonzero matrix\. For a bounded functionffon𝒮\{\\cal S\},‖f‖∞:=sups\|f⁡\(s\)\|\\\|f\\\|\_\{\\infty\}:=\\sup\_\{s\}\|f\(s\)\|, and, for a finite signed measureν\\nu,‖ν‖TV:=supℬ\|ν⁡\(ℬ\)\|\\\|\\nu\\\|\_\{\\mathrm\{TV\}\}:=\\sup\_\{\\cal B\}\|\\nu\(\{\\cal B\}\)\|over measurable setsℬ\{\\cal B\}, which is half the total variation ofν\\nuifν\\nuhas total mass zero\. The symbols𝔼\\mathbb\{E\}andℙ\\mathbb\{P\}denote expectation and probability \(ℙ\\mathbb\{P\}also denotes the transition kernel\), and𝒪⁡\(⋅\)\{\\cal O\}\(\\cdot\)hides constant factors, which may depend on problem constants such asRrR\_\{r\},RπR\_\{\\pi\},γ\\gamma, andλ\\lambdabut not on the quantities displayed in its argument;𝒪~​\(⋅\)\\tilde\{\\cal O\}\(\\cdot\)additionally hides polylogarithmic factors\. The symbolrrdenotes both the reward function andr:=min⁡\{m,n\}r:=\\min\\\{m,n\\\}, the largest rank of anm×nm\\times nmatrix, such that‖X‖∗≤r1/2​‖X‖F\\\|X\\\|\_\{\*\}\\leq r^\{1/2\}\\\|X\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}; likewise,nndenotes the number of columns ofWWand, asnkn\_\{k\}, the accumulator of the AdaNorm actor \(Section[3\.3](https://arxiv.org/html/2610.33477#S3.SS3)\); the meaning is clear from context\.

For a norm∥⋅∥\\\|\\cdot\\\|onℝm×n\\mathbb\{R\}^\{m\\times n\}, the dual norm is‖Y‖†:=sup‖X‖≤1⟨Y,X⟩F\\\|Y\\\|\_\{\\dagger\}:=\\sup\_\{\\\|X\\\|\\leq 1\}\\langle Y,X\\rangle\_\{\\scriptscriptstyle\\textrm\{F\}\}, such that the spectral and nuclear norms are dual to each other and the Frobenius norm is self\-dual, andκ†:=supX≠0‖X‖†/‖X‖F\\kappa\_\{\\dagger\}:=\\sup\_\{X\\neq 0\}\\nicefrac\{\{\\\|X\\\|\_\{\\dagger\}\}\}\{\{\\\|X\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\}\}, which isr1/2r^\{1/2\}for the nuclear norm and11for the Frobenius norm\. If∥⋅∥≥∥⋅∥2\\\|\\cdot\\\|\\geq\\\|\\cdot\\\|\_\{2\}, its unit ball lies in that of the spectral norm, such that∥⋅∥†≤∥⋅∥∗\\\|\\cdot\\\|\_\{\\dagger\}\\leq\\\|\\cdot\\\|\_\{\*\}\. SinceJJis nonconcave, we measure the progress of an actor by1K​∑k=1K𝔼⁡\[‖∇J​\(Wk\)‖†\]\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\[\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\\dagger\}\]in the dual norm of its step \(Definition[1](https://arxiv.org/html/2610.33477#Thmtheorem1)\), and we compare actors in the nuclear norm, which bounds every such dual norm because∥⋅∥†≤∥⋅∥∗≤r1/2∥⋅∥F\\\|\\cdot\\\|\_\{\\dagger\}\\leq\\\|\\cdot\\\|\_\{\*\}\\leq r^\{1/2\}\\\|\\cdot\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\.

## Appendix BNon\-Singularity of the Ridge\-Regularized Pathwise LSTD

We show that everyλ\>0\\lambda\>0makes both ridge\-regularized pathwise LSTD systems invertible, for every frame length and without any rank condition onΦk\\Phi\_\{k\}orYkY\_\{k\}\.

Proof\.Per framekkwith ridge parameterλ≠0\\lambda\\neq 0, we havedet\(λ​Id\+Φk​Yk⊤\)=λd−T​det\(λ​IT\+Yk⊤​Φk\)\\det\(\\lambda I\_\{d\}\+\\Phi\_\{k\}Y\_\{k\}^\{\\top\}\)=\\lambda^\{d\-T\}\\det\(\\lambda I\_\{T\}\+Y\_\{k\}^\{\\top\}\\Phi\_\{k\}\)withA^k=Φk​Yk⊤\\hat\{A\}\_\{k\}=\\Phi\_\{k\}Y\_\{k\}^\{\\top\}andHk=λ​IT\+Yk⊤​ΦkH\_\{k\}=\\lambda I\_\{T\}\+Y\_\{k\}^\{\\top\}\\Phi\_\{k\}\. Therefore, we can show thatHkH\_\{k\}is non\-singular if and only ifA^k\+λ​Id\\hat\{A\}\_\{k\}\+\\lambda I\_\{d\}is\.

Forv∈ℝdv\\in\\mathbb\{R\}^\{d\}, we definezt:=v⊤​ϕk,tz\_\{t\}:=v^\{\\top\}\\phi\_\{k,t\}fort<Tt<TandzT:=0z\_\{T\}:=0such that the zero last column ofΦk′\\Phi\_\{k\}^\{\\prime\}inducesT​v⊤​A^k​v=∑t=0T−1\(zt2−γ​zt​zt\+1\)Tv^\{\\top\}\\hat\{A\}\_\{k\}v=\\sum\_\{t=0\}^\{T\-1\}\(z\_\{t\}^\{2\}\-\\gamma z\_\{t\}z\_\{t\+1\}\)\. Based on the factszt2−γ​zt​zt\+1=\(1−γ\)​zt2\+γ2​\(zt2−zt\+12\)\+γ2​\(zt−zt\+1\)2z\_\{t\}^\{2\}\-\\gamma z\_\{t\}z\_\{t\+1\}=\(1\-\\gamma\)z\_\{t\}^\{2\}\+\\frac\{\\gamma\}\{2\}\(z\_\{t\}^\{2\}\-z\_\{t\+1\}^\{2\}\)\+\\frac\{\\gamma\}\{2\}\(z\_\{t\}\-z\_\{t\+1\}\)^\{2\}andzT=0z\_\{T\}=0, we obtain

v⊤​\(A^k\+λ​Id\)​v=1T​∑t=0T−1\[\(1−γ\)​zt2\+γ2​\(zt−zt\+1\)2\]\+γ​z022​T\+λ​‖v‖22≥λ​‖v‖22\.v^\{\\top\}\(\\hat\{A\}\_\{k\}\+\\lambda I\_\{d\}\)v=\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\Big\[\(1\-\\gamma\)z\_\{t\}^\{2\}\+\\frac\{\\gamma\}\{2\}\(z\_\{t\}\-z\_\{t\+1\}\)^\{2\}\\Big\]\+\\frac\{\\gamma z\_\{0\}^\{2\}\}\{2T\}\+\\lambda\\\|v\\\|\_\{2\}^\{2\}\\geq\\lambda\\\|v\\\|\_\{2\}^\{2\}\.\(35\)
We observe that \([35](https://arxiv.org/html/2610.33477#A2.E35)\) secures the smallest singular value ofA^k\+λ​Id\\hat\{A\}\_\{k\}\+\\lambda I\_\{d\}to be positive ridge parameterλ\\lambda\. Therefore, we conclude that the two matricesλ​Id\+Φk​Yk⊤\\lambda I\_\{d\}\+\\Phi\_\{k\}Y\_\{k\}^\{\\top\}andλ​IT\+Yk⊤​Φk\\lambda I\_\{T\}\+Y\_\{k\}^\{\\top\}\\Phi\_\{k\}are non\-singular\.

## Appendix CProof of Lemma[2](https://arxiv.org/html/2610.33477#Thmtheorem2)

Proof\.We make the argument of[Zhang et al\. \(2020a, Lemma 3\.2\)](https://arxiv.org/html/2610.33477#bib.bib70)explicit for the normalized return\. Fix a start states0s\_\{0\}, letℋt:=\(s0,a0,s1,…,st,at\)\{\\cal H\}\_\{t\}:=\(s\_\{0\},a\_\{0\},s\_\{1\},\\ldots,s\_\{t\},a\_\{t\}\), and letptWp^\{W\}\_\{t\}denote its law underπW\\pi\_\{W\}froms0s\_\{0\}\. Only the policy factors ofptWp^\{W\}\_\{t\}depend onWW, and∇W​log​ptW​\(ℋt\)=∑i=0t∇log⁡πW​\(ai\|si\)\\nabla\_\{W\}\\log p^\{W\}\_\{t\}\(\{\\cal H\}\_\{t\}\)=\\sum\_\{i=0\}^\{t\}\\nabla\\log\\pi\_\{W\}\(a\_\{i\}\|s\_\{i\}\); hence, differentiatingVW​\(s0\)=∑t≥0γt​𝔼ptW​\[r⁡\(st,at\)\]V\_\{W\}\(s\_\{0\}\)=\\sum\_\{t\\geq 0\}\\gamma^\{t\}\\,\\mathbb\{E\}\_\{p^\{W\}\_\{t\}\}\[r\(s\_\{t\},a\_\{t\}\)\]term by term and under each expectation gives

∇VW​\(s0\)=∑t≥0γt​𝔼ptW​\[ΥtW\],ΥtW​\(ℋt\):=r⁡\(st,at\)​∑i=0t∇log⁡πW​\(ai\|si\)\.\\nabla V\_\{W\}\(s\_\{0\}\)=\\sum\_\{t\\geq 0\}\\gamma^\{t\}\\,\\mathbb\{E\}\_\{p^\{W\}\_\{t\}\}\\big\[\\Upsilon^\{W\}\_\{t\}\\big\],\\qquad\\Upsilon^\{W\}\_\{t\}\(\{\\cal H\}\_\{t\}\):=r\(s\_\{t\},a\_\{t\}\)\\sum\_\{i=0\}^\{t\}\\nabla\\log\\pi\_\{W\}\(a\_\{i\}\|s\_\{i\}\)\.
Both interchanges are justified by dominated convergence: for‖W′′−W‖2≤δ\\\|W^\{\\prime\\prime\}\-W\\\|\_\{2\}\\leq\\deltawithγ​eRπ​δ<1\\gamma e^\{R\_\{\\pi\}\\delta\}<1, Hölder’s inequality and \([15a](https://arxiv.org/html/2610.33477#S4.E15.1)\), integrated along the segment fromWWtoW′′W^\{\\prime\\prime\}, give\|log⁡πW′′​\(a\|s\)−log⁡πW​\(a\|s\)\|≤Rπ​δ\|\\log\\pi\_\{W^\{\\prime\\prime\}\}\(a\|s\)\-\\log\\pi\_\{W\}\(a\|s\)\|\\leq R\_\{\\pi\}\\delta, such that the density ofptW′′p^\{W^\{\\prime\\prime\}\}\_\{t\}is at moste\(t\+1\)​Rπ​δe^\{\(t\+1\)R\_\{\\pi\}\\delta\}times that ofptWp^\{W\}\_\{t\}, the gradient of thettth integrand atW′′W^\{\\prime\\prime\}is at most\(t\+1\)​Rr​Rπ​e\(t\+1\)​Rπ​δ\(t\+1\)R\_\{r\}R\_\{\\pi\}e^\{\(t\+1\)R\_\{\\pi\}\\delta\}times the density ofptWp^\{W\}\_\{t\}in the nuclear norm, and∑t≥0γt​\(t\+1\)​e\(t\+1\)​Rπ​δ<∞\\sum\_\{t\\geq 0\}\\gamma^\{t\}\(t\+1\)e^\{\(t\+1\)R\_\{\\pi\}\\delta\}<\\infty\. FixW,W′W,W^\{\\prime\}, letℓ2:=‖W−W′‖2\\ell\_\{2\}:=\\\|W\-W^\{\\prime\}\\\|\_\{2\}, and split∇VW​\(s0\)−∇VW′​\(s0\)=∑t≥0γt​𝔼ptW​\[ΥtW−ΥtW′\]\+∑t≥0γt​\(𝔼ptW−𝔼ptW′\)​\[ΥtW′\]\\nabla V\_\{W\}\(s\_\{0\}\)\-\\nabla V\_\{W^\{\\prime\}\}\(s\_\{0\}\)=\\sum\_\{t\\geq 0\}\\gamma^\{t\}\\,\\mathbb\{E\}\_\{p^\{W\}\_\{t\}\}\[\\Upsilon^\{W\}\_\{t\}\-\\Upsilon^\{W^\{\\prime\}\}\_\{t\}\]\+\\sum\_\{t\\geq 0\}\\gamma^\{t\}\(\\mathbb\{E\}\_\{p^\{W\}\_\{t\}\}\-\\mathbb\{E\}\_\{p^\{W^\{\\prime\}\}\_\{t\}\}\)\[\\Upsilon^\{W^\{\\prime\}\}\_\{t\}\]\.

*Change of the score\.*By \([15b](https://arxiv.org/html/2610.33477#S4.E15.2)\) and the triangle inequality,‖ΥtW−ΥtW′‖∗≤\(t\+1\)​Rr​Lπ​ℓ2\\\|\\Upsilon^\{W\}\_\{t\}\-\\Upsilon^\{W^\{\\prime\}\}\_\{t\}\\\|\_\{\*\}\\leq\(t\+1\)R\_\{r\}L\_\{\\pi\}\\ell\_\{2\}, such that∑t≥0\(t\+1\)​γt=\(1−γ\)−2\\sum\_\{t\\geq 0\}\(t\+1\)\\gamma^\{t\}=\(1\-\\gamma\)^\{\-2\}bounds the first sum byRr​Lπ​ℓ2/\(1−γ\)2\\nicefrac\{\{R\_\{r\}L\_\{\\pi\}\\ell\_\{2\}\}\}\{\{\(1\-\\gamma\)^\{2\}\}\}in the nuclear norm\.

*Change of the trajectory distribution\.*AlongWτ:=W′\+τ⁡\(W−W′\)W\_\{\\tau\}:=W^\{\\prime\}\+\\tau\(W\-W^\{\\prime\}\),τ∈\[0,1\]\\tau\\in\[0,1\], we havedd​τ​πWτ​\(a\|s\)=πWτ​\(a\|s\)​⟨∇log⁡πWτ​\(a\|s\),W−W′⟩F\\frac\{d\}\{d\\tau\}\\pi\_\{W\_\{\\tau\}\}\(a\|s\)=\\pi\_\{W\_\{\\tau\}\}\(a\|s\)\\langle\\nabla\\log\\pi\_\{W\_\{\\tau\}\}\(a\|s\),W\-W^\{\\prime\}\\rangle\_\{\\scriptscriptstyle\\textrm\{F\}\}, where Hölder’s inequality and \([15a](https://arxiv.org/html/2610.33477#S4.E15.1)\) bound the inner product byRπ​ℓ2R\_\{\\pi\}\\ell\_\{2\}; integrating overτ\\tauandaayieldssups∥πW\(⋅\|s\)−πW′\(⋅\|s\)∥TV≤Rπ​ℓ2/2\\sup\_\{s\}\\\|\\pi\_\{W\}\(\\cdot\|s\)\-\\pi\_\{W^\{\\prime\}\}\(\\cdot\|s\)\\\|\_\{\\mathrm\{TV\}\}\\leq\\nicefrac\{\{R\_\{\\pi\}\\ell\_\{2\}\}\}\{\{2\}\}\. SinceptWp^\{W\}\_\{t\}andptW′p^\{W^\{\\prime\}\}\_\{t\}differ only in thet\+1t\+1policy factors, replacing them one at a time gives‖ptW−ptW′‖TV≤\(t\+1\)​Rπ​ℓ2/2\\\|p^\{W\}\_\{t\}\-p^\{W^\{\\prime\}\}\_\{t\}\\\|\_\{\\mathrm\{TV\}\}\\leq\\nicefrac\{\{\(t\+1\)R\_\{\\pi\}\\ell\_\{2\}\}\}\{\{2\}\}, and‖ΥtW′‖∗≤\(t\+1\)​Rr​Rπ\\\|\\Upsilon^\{W^\{\\prime\}\}\_\{t\}\\\|\_\{\*\}\\leq\(t\+1\)R\_\{r\}R\_\{\\pi\}and the bound on‖∫f​𝑑ν‖\\\|\\int f\\,d\\nu\\\|of Appendix[D](https://arxiv.org/html/2610.33477#A4)bound thettth summand of the second sum by2​γt​\(t\+1\)​Rr​Rπ​‖ptW−ptW′‖TV≤γt​\(t\+1\)2​Rr​Rπ2​ℓ22\\gamma^\{t\}\(t\+1\)R\_\{r\}R\_\{\\pi\}\\\|p^\{W\}\_\{t\}\-p^\{W^\{\\prime\}\}\_\{t\}\\\|\_\{\\mathrm\{TV\}\}\\leq\\gamma^\{t\}\(t\+1\)^\{2\}R\_\{r\}R\_\{\\pi\}^\{2\}\\ell\_\{2\}; hence,∑t≥0\(t\+1\)2​γt=\(1\+γ\)/\(1−γ\)3\\sum\_\{t\\geq 0\}\(t\+1\)^\{2\}\\gamma^\{t\}=\\nicefrac\{\{\(1\+\\gamma\)\}\}\{\{\(1\-\\gamma\)^\{3\}\}\}bounds the second sum by\(1\+γ\)​Rr​Rπ2​ℓ2/\(1−γ\)3\\nicefrac\{\{\(1\+\\gamma\)R\_\{r\}R\_\{\\pi\}^\{2\}\\ell\_\{2\}\}\}\{\{\(1\-\\gamma\)^\{3\}\}\}\.

*Normalization\.*Averaging overs0∼ρs\_\{0\}\\sim\\rho, which commutes with the gradient by dominated convergence since‖∇VW′′​\(s0\)‖∗≤Rr​Rπ/\(1−γ\)2\\\|\\nabla V\_\{W^\{\\prime\\prime\}\}\(s\_\{0\}\)\\\|\_\{\*\}\\leq\\nicefrac\{\{R\_\{r\}R\_\{\\pi\}\}\}\{\{\(1\-\\gamma\)^\{2\}\}\}for allW′′W^\{\\prime\\prime\}ands0s\_\{0\}, and multiplying by1−γ1\-\\gamma, sinceJ⁡\(W\)=\(1−γ\)​𝔼s0∼ρ​\[VW​\(s0\)\]J\(W\)=\(1\-\\gamma\)\\mathbb\{E\}\_\{s\_\{0\}\\sim\\rho\}\[V\_\{W\}\(s\_\{0\}\)\], yield‖∇J​\(W\)−∇J​\(W′\)‖∗≤L​ℓ2\\\|\\nabla J\(W\)\-\\nabla J\(W^\{\\prime\}\)\\\|\_\{\*\}\\leq L\\ell\_\{2\}, which is \([16](https://arxiv.org/html/2610.33477#S4.E16)\)\. Finally, by Hölder’s inequality,J\(W′\)−J\(W\)−⟨∇J\(W\),W′−W⟩F=∫01⟨∇J\(W\+τ\(W′−W\)\)−∇J\(W\),W′−W⟩Fdτ≥−∫01τL∥W′−W∥22dτ=−L2∥W′−W∥22J\(W^\{\\prime\}\)\-J\(W\)\-\\langle\\nabla J\(W\),W^\{\\prime\}\-W\\rangle\_\{\\scriptscriptstyle\\textrm\{F\}\}=\\int\_\{0\}^\{1\}\\langle\\nabla J\(W\+\\tau\(W^\{\\prime\}\-W\)\)\-\\nabla J\(W\),W^\{\\prime\}\-W\\rangle\_\{\\scriptscriptstyle\\textrm\{F\}\}\\,d\\tau\\geq\-\\int\_\{0\}^\{1\}\\tau L\\\|W^\{\\prime\}\-W\\\|\_\{2\}^\{2\}\\,d\\tau=\-\\frac\{L\}\{2\}\\\|W^\{\\prime\}\-W\\\|\_\{2\}^\{2\}, which is \([17](https://arxiv.org/html/2610.33477#S4.E17)\)\.

## Appendix DPreliminaries and a Markovian Moment Bound

Throughout the appendices,PWtP\_\{W\}^\{t\}denotes thett\-step kernel of the chain induced byπW\\pi\_\{W\},dmix​\(t\):=supWsups‖PWt​\(s,⋅\)−μW‖TVd\_\{\\mathrm\{mix\}\}\(t\):=\\sup\_\{W\}\\sup\_\{s\}\\\|P^\{t\}\_\{W\}\(s,\\cdot\)\-\\mu\_\{W\}\\\|\_\{\\mathrm\{TV\}\}, and Hölder’s inequality refers to\|⟨X,Y⟩F\|≤‖X‖∗​‖Y‖2\|\\langle X,Y\\rangle\_\{\\scriptscriptstyle\\textrm\{F\}\}\|\\leq\\\|X\\\|\_\{\*\}\\\|Y\\\|\_\{2\}\. For every finite signed measureν\\nuon a measurable space𝒳\{\\cal X\}, with‖ν‖TV:=supℬ⊆𝒳\|ν⁡\(ℬ\)\|\\\|\\nu\\\|\_\{\\mathrm\{TV\}\}:=\\sup\_\{\{\\cal B\}\\subseteq\{\\cal X\}\}\|\\nu\(\{\\cal B\}\)\|, and every bounded measurableffon𝒳\{\\cal X\}with values in a finite\-dimensional normed space,‖∫f​𝑑ν‖≤sup‖f‖​\|ν\|​\(𝒳\)≤2​sup‖f‖​‖ν‖TV\\\|\\int f\\,d\\nu\\\|\\leq\\sup\\\|f\\\|\\,\|\\nu\|\(\{\\cal X\}\)\\leq 2\\sup\\\|f\\\|\\,\\\|\\nu\\\|\_\{\\mathrm\{TV\}\}; we use this bound for measures on states, actions, observations, and frames\. By\([Levin and Peres, 2017](https://arxiv.org/html/2610.33477#bib.bib37), Sections 4\.4–4\.5\), whose arguments do not use the finiteness of the state space,dmixd\_\{\\mathrm\{mix\}\}is nonincreasing and, by Assumption[2](https://arxiv.org/html/2610.33477#Thmassump2), satisfiesdmix​\(j​τmix\)≤2−jd\_\{\\mathrm\{mix\}\}\(j\\,\\tau\_\{\\mathrm\{mix\}\}\)\\leq 2^\{\-j\}for all integersj≥0j\\geq 0; hence,

∑t≥0dmix​\(t\)≤τmix​∑j≥02−j=2​τmix\.\\sum\\nolimits\_\{t\\geq 0\}d\_\{\\mathrm\{mix\}\}\(t\)\\leq\\tau\_\{\\mathrm\{mix\}\}\\sum\\nolimits\_\{j\\geq 0\}2^\{\-j\}=2\\tau\_\{\\mathrm\{mix\}\}\.\(36\)
The following second\-moment bound for averages along the chain replaces the concentration bounds of[Dorfman and Levy \(2022, Lemma A\.6\)](https://arxiv.org/html/2610.33477#bib.bib16)for finite\-state chains and holds for general state spaces\.

###### Lemma 20\.

Under Assumption[2](https://arxiv.org/html/2610.33477#Thmassump2), letffmap observationso=\(s,a,s′\)o=\(s,a,s^\{\\prime\}\)into a real inner\-product space of finite dimension with‖f‖≤B\\\|f\\\|\\leq B, fixWWand a start states0s\_\{0\}, letot=\(st,at,st\+1\)o\_\{t\}=\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\),t≥0t\\geq 0, be generated byπW\\pi\_\{W\}andℙ\\mathbb\{P\}froms0s\_\{0\}, and letm:=𝔼μW⊗πW⊗ℙ​\[f\]m:=\\mathbb\{E\}\_\{\\mu\_\{W\}\\otimes\\pi\_\{W\}\\otimes\\mathbb\{P\}\}\[f\]\. Then, for everyT≥1T\\geq 1,

𝔼​‖1T​∑t=0T−1f⁡\(ot\)−m‖2≤4​B2​\(1\+4​τmix\)T\.\\mathbb\{E\}\\Big\\\|\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}f\(o\_\{t\}\)\-m\\Big\\\|^\{2\}\\leq\\frac\{4B^\{2\}\(1\+4\\tau\_\{\\mathrm\{mix\}\}\)\}\{T\}\.

Proof\.LetXt:=f⁡\(ot\)−mX\_\{t\}:=f\(o\_\{t\}\)\-m, such that‖Xt‖≤2​B\\\|X\_\{t\}\\\|\\leq 2B, and letf¯​\(s\)\\bar\{f\}\(s\)denote the mean off⁡\(s,a,s′\)f\(s,a,s^\{\\prime\}\)overa∼πW\(⋅\|s\)a\\sim\\pi\_\{W\}\(\\cdot\|s\)ands′∼ℙ\(⋅\|s,a\)s^\{\\prime\}\\sim\\mathbb\{P\}\(\\cdot\|s,a\), such thatm=∫f¯​d​μWm=\\int\\bar\{f\}\\,d\\mu\_\{W\}\. Forj≥1j\\geq 1, the Markov property gives𝔼\[Xt\+j\|s0,a0,…,at,st\+1\]=∫f¯d\(PWj−1\(st\+1,⋅\)−μW\)\\mathbb\{E\}\[X\_\{t\+j\}\\,\|\\,s\_\{0\},a\_\{0\},\\ldots,a\_\{t\},s\_\{t\+1\}\]=\\int\\bar\{f\}\\,d\(P\_\{W\}^\{j\-1\}\(s\_\{t\+1\},\\cdot\)\-\\mu\_\{W\}\), whose norm is at most2​B​dmix​\(j−1\)2B\\,d\_\{\\mathrm\{mix\}\}\(j\-1\); sinceXtX\_\{t\}is a function of\(st,at,st\+1\)\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\), the tower property gives\|𝔼⁡⟨Xt,Xt\+j⟩\|≤4​B2​dmix​\(j−1\)\|\\mathbb\{E\}\\langle X\_\{t\},X\_\{t\+j\}\\rangle\|\\leq 4B^\{2\}d\_\{\\mathrm\{mix\}\}\(j\-1\)\. Expanding the square and usingT−j≤TT\-j\\leq Tand \([36](https://arxiv.org/html/2610.33477#A4.E36)\) yield

𝔼​‖1T​∑t=0T−1Xt‖2≤4​B2T\+8​B2T2​∑j=1T−1\(T−j\)​dmix​\(j−1\)≤4​B2​\(1\+4​τmix\)T\.\\mathbb\{E\}\\Big\\\|\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}X\_\{t\}\\Big\\\|^\{2\}\\leq\\frac\{4B^\{2\}\}\{T\}\+\\frac\{8B^\{2\}\}\{T^\{2\}\}\\sum\_\{j=1\}^\{T\-1\}\(T\-j\)\\,d\_\{\\mathrm\{mix\}\}\(j\-1\)\\leq\\frac\{4B^\{2\}\(1\+4\\tau\_\{\\mathrm\{mix\}\}\)\}\{T\}\.
We complete the proof\.

## Appendix EProof of Lemma[3](https://arxiv.org/html/2610.33477#Thmtheorem3)

Proof\.FixWWandW′W^\{\\prime\}, letℓ2:=‖W−W′‖2\\ell\_\{2\}:=\\\|W\-W^\{\\prime\}\\\|\_\{2\}andϵπ:=12​Rπ​ℓ2\\epsilon\_\{\\pi\}:=\\tfrac\{1\}\{2\}R\_\{\\pi\}\\ell\_\{2\}, recall that the ridge parameter satisfiesλ\>0\\lambda\>0, and letℙW:=μW⊗πW⊗ℙ\\mathbb\{P\}\_\{W\}:=\\mu\_\{W\}\\otimes\\pi\_\{W\}\\otimes\\mathbb\{P\}denote the law ofo=\(s,a,s′\)o=\(s,a,s^\{\\prime\}\)underπW\\pi\_\{W\}, such thatAW=𝔼ℙW​\[ϕ⁡\(s\)​\(ϕ⁡\(s\)−γ​ϕ​\(s′\)\)⊤\]A\_\{W\}=\\mathbb\{E\}\_\{\\mathbb\{P\}\_\{W\}\}\[\\phi\(s\)\(\\phi\(s\)\-\\gamma\\phi\(s^\{\\prime\}\)\)^\{\\top\}\]andbW=𝔼ℙW​\[ϕ⁡\(s\)​r​\(s,a\)\]b\_\{W\}=\\mathbb\{E\}\_\{\\mathbb\{P\}\_\{W\}\}\[\\phi\(s\)r\(s,a\)\]\.

*Change of the observation law\.*Step 1 of the proof of Lemma[7](https://arxiv.org/html/2610.33477#Thmtheorem7)\(ii\) in Appendix[I](https://arxiv.org/html/2610.33477#A9), which holds for any two parameters, gives \([43](https://arxiv.org/html/2610.33477#A9.E43)\) and \([44](https://arxiv.org/html/2610.33477#A9.E44)\), i\.e\.,sups∥πW\(⋅\|s\)−πW′\(⋅\|s\)∥TV≤ϵπ\\sup\_\{s\}\\\|\\pi\_\{W\}\(\\cdot\|s\)\-\\pi\_\{W^\{\\prime\}\}\(\\cdot\|s\)\\\|\_\{\\mathrm\{TV\}\}\\leq\\epsilon\_\{\\pi\}and‖μW−μW′‖TV≤4​τmix​ϵπ\\\|\\mu\_\{W\}\-\\mu\_\{W^\{\\prime\}\}\\\|\_\{\\mathrm\{TV\}\}\\leq 4\\tau\_\{\\mathrm\{mix\}\}\\epsilon\_\{\\pi\}\. For a measurable setℬ\{\\cal B\}of observations, letpℬ​\(s,a\):=ℙ⁡\(\{s′:\(s,a,s′\)∈ℬ\}\|s,a\)p\_\{\\cal B\}\(s,a\):=\\mathbb\{P\}\(\\\{s^\{\\prime\}:\(s,a,s^\{\\prime\}\)\\in\{\\cal B\}\\\}\\,\|\\,s,a\)andf⁡\(s\):=∫πW​\(𝑑a\|s\)​pℬ​\(s,a\)f\(s\):=\\int\\pi\_\{W\}\(da\|s\)\\,p\_\{\\cal B\}\(s,a\), both\[0,1\]\[0,1\]\-valued\. ThenℙW​\(ℬ\)−ℙW′​\(ℬ\)=∫f​d​\(μW−μW′\)\+∫μW′​\(𝑑s\)​∫pℬ​\(s,a\)​\(πW−πW′\)​\(𝑑a\|s\)\\mathbb\{P\}\_\{W\}\(\{\\cal B\}\)\-\\mathbb\{P\}\_\{W^\{\\prime\}\}\(\{\\cal B\}\)=\\int f\\,d\(\\mu\_\{W\}\-\\mu\_\{W^\{\\prime\}\}\)\+\\int\\mu\_\{W^\{\\prime\}\}\(ds\)\\int p\_\{\\cal B\}\(s,a\)\\,\(\\pi\_\{W\}\-\\pi\_\{W^\{\\prime\}\}\)\(da\|s\), and, since\|∫f​𝑑ν\|≤‖ν‖TV\|\\int f\\,d\\nu\|\\leq\\\|\\nu\\\|\_\{\\mathrm\{TV\}\}for every\[0,1\]\[0,1\]\-valuedffand every signed measureν\\nuwith total mass zero,

∥ℙW−ℙW′∥TV≤∥μW−μW′∥TV\+sups∥πW\(⋅\|s\)−πW′\(⋅\|s\)∥TV≤\(4τmix\+1\)ϵπ\.\\\|\\mathbb\{P\}\_\{W\}\-\\mathbb\{P\}\_\{W^\{\\prime\}\}\\\|\_\{\\mathrm\{TV\}\}\\leq\\\|\\mu\_\{W\}\-\\mu\_\{W^\{\\prime\}\}\\\|\_\{\\mathrm\{TV\}\}\+\\sup\\nolimits\_\{s\}\\\|\\pi\_\{W\}\(\\cdot\|s\)\-\\pi\_\{W^\{\\prime\}\}\(\\cdot\|s\)\\\|\_\{\\mathrm\{TV\}\}\\leq\(4\\tau\_\{\\mathrm\{mix\}\}\+1\)\\,\\epsilon\_\{\\pi\}\.\(37\)
*Change ofv∗v^\{\*\}\.*Since‖ϕ⁡\(s\)‖2≤1\\\|\\phi\(s\)\\\|\_\{2\}\\leq 1gives‖ϕ⁡\(s\)​r​\(s,a\)‖2≤Rr\\\|\\phi\(s\)r\(s,a\)\\\|\_\{2\}\\leq R\_\{r\}and‖ϕ⁡\(s\)​\(ϕ⁡\(s\)−γ​ϕ​\(s′\)\)⊤‖2≤1\+γ\\\|\\phi\(s\)\(\\phi\(s\)\-\\gamma\\phi\(s^\{\\prime\}\)\)^\{\\top\}\\\|\_\{2\}\\leq 1\+\\gamma, the bound on‖∫f​𝑑ν‖\\\|\\int f\\,d\\nu\\\|of Appendix[D](https://arxiv.org/html/2610.33477#A4), applied to measures on observations, gives‖bW−bW′‖2≤2​Rr​‖ℙW−ℙW′‖TV\\\|b\_\{W\}\-b\_\{W^\{\\prime\}\}\\\|\_\{2\}\\leq 2R\_\{r\}\\\|\\mathbb\{P\}\_\{W\}\-\\mathbb\{P\}\_\{W^\{\\prime\}\}\\\|\_\{\\mathrm\{TV\}\}and‖AW−AW′‖2≤2​\(1\+γ\)​‖ℙW−ℙW′‖TV\\\|A\_\{W\}\-A\_\{W^\{\\prime\}\}\\\|\_\{2\}\\leq 2\(1\+\\gamma\)\\\|\\mathbb\{P\}\_\{W\}\-\\mathbb\{P\}\_\{W^\{\\prime\}\}\\\|\_\{\\mathrm\{TV\}\}\. Moreover,‖bW′‖2≤Rr\\\|b\_\{W^\{\\prime\}\}\\\|\_\{2\}\\leq R\_\{r\}, andx⊤​AW​x≥0x^\{\\top\}A\_\{W\}x\\geq 0for allxx\(see below \([6](https://arxiv.org/html/2610.33477#S2.E6)\)\) givesλ​‖x‖22≤‖x‖2​‖\(AW\+λ​Id\)​x‖2\\lambda\\\|x\\\|\_\{2\}^\{2\}\\leq\\\|x\\\|\_\{2\}\\\|\(A\_\{W\}\+\\lambda I\_\{d\}\)x\\\|\_\{2\}, i\.e\.,‖\(AW\+λ​Id\)−1‖2≤1/λ\\\|\(A\_\{W\}\+\\lambda I\_\{d\}\)^\{\-1\}\\\|\_\{2\}\\leq\\nicefrac\{\{1\}\}\{\{\\lambda\}\}, and likewise forW′W^\{\\prime\}\. Hence, the resolvent identity\(AW\+λ​Id\)−1−\(AW′\+λ​Id\)−1=\(AW\+λ​Id\)−1​\(AW′−AW\)​\(AW′\+λ​Id\)−1\(A\_\{W\}\+\\lambda I\_\{d\}\)^\{\-1\}\-\(A\_\{W^\{\\prime\}\}\+\\lambda I\_\{d\}\)^\{\-1\}=\(A\_\{W\}\+\\lambda I\_\{d\}\)^\{\-1\}\(A\_\{W^\{\\prime\}\}\-A\_\{W\}\)\(A\_\{W^\{\\prime\}\}\+\\lambda I\_\{d\}\)^\{\-1\}gives

‖v∗​\(W\)−v∗​\(W′\)‖2≤‖bW−bW′‖2λ\+Rr​‖AW−AW′‖2λ2≤2​Rr​\(λ\+1\+γ\)λ2​‖ℙW−ℙW′‖TV\\\|v^\{\*\}\(W\)\-v^\{\*\}\(W^\{\\prime\}\)\\\|\_\{2\}\\leq\\frac\{\\\|b\_\{W\}\-b\_\{W^\{\\prime\}\}\\\|\_\{2\}\}\{\\lambda\}\+\\frac\{R\_\{r\}\\\|A\_\{W\}\-A\_\{W^\{\\prime\}\}\\\|\_\{2\}\}\{\\lambda^\{2\}\}\\leq\\frac\{2R\_\{r\}\(\\lambda\+1\+\\gamma\)\}\{\\lambda^\{2\}\}\\\|\\mathbb\{P\}\_\{W\}\-\\mathbb\{P\}\_\{W^\{\\prime\}\}\\\|\_\{\\mathrm\{TV\}\}and \([37](https://arxiv.org/html/2610.33477#A5.E37)\) withϵπ=12​Rπ​ℓ2\\epsilon\_\{\\pi\}=\\tfrac\{1\}\{2\}R\_\{\\pi\}\\ell\_\{2\}yields‖v∗​\(W\)−v∗​\(W′\)‖2≤L∗​ℓ2\\\|v^\{\*\}\(W\)\-v^\{\*\}\(W^\{\\prime\}\)\\\|\_\{2\}\\leq L\_\{\*\}\\ell\_\{2\}\.

## Appendix FProof of Theorem[4](https://arxiv.org/html/2610.33477#Thmtheorem4)

Proof\.Fixk≥0k\\geq 0\. We bound the NSI and the statistical errors and then combine them\.

*NSI error\.*The safe scalingck2=‖Hk‖1​‖Hk‖∞c\_\{k\}^\{2\}=\\\|H\_\{k\}\\\|\_\{1\}\\\|H\_\{k\}\\\|\_\{\\infty\}of Section[3\.1](https://arxiv.org/html/2610.33477#S3.SS1)satisfies‖Hk‖2≤ck≤λ\+1\+γ\\\|H\_\{k\}\\\|\_\{2\}\\leq c\_\{k\}\\leq\\lambda\+1\+\\gamma, as the theorem requires: the first inequality holds because‖H‖22≤‖H‖1​‖H‖∞\\\|H\\\|\_\{2\}^\{2\}\\leq\\\|H\\\|\_\{1\}\\\|H\\\|\_\{\\infty\}for every square matrixHH, and the second because every entry ofYk⊤​ΦkY\_\{k\}^\{\\top\}\\Phi\_\{k\}, the inner product of a column ofYkY\_\{k\}with norm at most\(1\+γ\)/T\\nicefrac\{\{\(1\+\\gamma\)\}\}\{\{\\sqrt\{T\}\}\}and a column ofΦk\\Phi\_\{k\}with norm at most1/T\\nicefrac\{\{1\}\}\{\{\\sqrt\{T\}\}\}, is at most\(1\+γ\)/T\\nicefrac\{\{\(1\+\\gamma\)\}\}\{\{T\}\}in absolute value, such that every absolute row and column sum ofHkH\_\{k\}is at mostλ\+1\+γ\\lambda\+1\+\\gamma\. By the NSI recursion \([8](https://arxiv.org/html/2610.33477#S3.E8)\),Ek,q\+1=IT−Hk​Xk,q​∑i=0κ1−1Ek,qi=IT−\(IT−Ek,q\)​∑i=0κ1−1Ek,qi=Ek,qκ1E\_\{k,q\+1\}=I\_\{T\}\-H\_\{k\}X\_\{k,q\}\\sum\_\{i=0\}^\{\\kappa\_\{1\}\-1\}E\_\{k,q\}^\{i\}=I\_\{T\}\-\(I\_\{T\}\-E\_\{k,q\}\)\\sum\_\{i=0\}^\{\\kappa\_\{1\}\-1\}E\_\{k,q\}^\{i\}=E\_\{k,q\}^\{\\kappa\_\{1\}\}, such thatXk,0=Hk⊤/ck2X\_\{k,0\}=\\nicefrac\{\{H\_\{k\}^\{\\top\}\}\}\{\{c\_\{k\}^\{2\}\}\}givesEk,q=\(IT−Hk​Hk⊤/ck2\)κ1qE\_\{k,q\}=\(I\_\{T\}\-\\nicefrac\{\{H\_\{k\}H\_\{k\}^\{\\top\}\}\}\{\{c\_\{k\}^\{2\}\}\}\)^\{\\kappa\_\{1\}^\{q\}\}, a symmetric matrix with the eigenvalues\(1−σi2​\(Hk\)/ck2\)κ1q∈\[0,1\)\(1\-\\nicefrac\{\{\\sigma\_\{i\}^\{2\}\(H\_\{k\}\)\}\}\{\{c\_\{k\}^\{2\}\}\}\)^\{\\kappa\_\{1\}^\{q\}\}\\in\[0,1\), sinceck≥‖Hk‖2c\_\{k\}\\geq\\\|H\_\{k\}\\\|\_\{2\}andHkH\_\{k\}is non\-singular \(Appendix[B](https://arxiv.org/html/2610.33477#A2)\)\. Hence,‖Ek,q‖2=ϵEκ1q\\\|E\_\{k,q\}\\\|\_\{2\}=\\epsilon\_\{E\}^\{\\kappa\_\{1\}^\{q\}\}withϵE:=1−σmin2​\(Hk\)/ck2∈\[0,1\)\\epsilon\_\{E\}:=1\-\\nicefrac\{\{\\sigma\_\{\\min\}^\{2\}\(H\_\{k\}\)\}\}\{\{c\_\{k\}^\{2\}\}\}\\in\[0,1\)\. SinceXk,q−Hk−1=−Hk−1​Ek,qX\_\{k,q\}\-H\_\{k\}^\{\-1\}=\-H\_\{k\}^\{\-1\}E\_\{k,q\}, \([8](https://arxiv.org/html/2610.33477#S3.E8)\) and \([7](https://arxiv.org/html/2610.33477#S3.E7)\) givevk,q−v^k=−Φk​Hk−1​Ek,q​rkv\_\{k,q\}\-\\hat\{v\}\_\{k\}=\-\\Phi\_\{k\}H\_\{k\}^\{\-1\}E\_\{k,q\}r\_\{k\}, and the push\-through identityΦk​Hk−1=\(A^k\+λ​Id\)−1​Φk\\Phi\_\{k\}H\_\{k\}^\{\-1\}=\(\\hat\{A\}\_\{k\}\+\\lambda I\_\{d\}\)^\{\-1\}\\Phi\_\{k\}in \([7](https://arxiv.org/html/2610.33477#S3.E7)\), \([35](https://arxiv.org/html/2610.33477#A2.E35)\), and‖Φk‖2≤‖Φk‖F≤1\\\|\\Phi\_\{k\}\\\|\_\{2\}\\leq\\\|\\Phi\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq 1give‖Φk​Hk−1‖2≤1/λ\\\|\\Phi\_\{k\}H\_\{k\}^\{\-1\}\\\|\_\{2\}\\leq\\nicefrac\{\{1\}\}\{\{\\lambda\}\}\. Together with‖rk‖2≤Rr\\\|r\_\{k\}\\\|\_\{2\}\\leq R\_\{r\}, we obtain, for every realization,

‖vk,q−v^k‖2≤Rv​ϵEκ1q≤Rv​exp⁡\(−CE​κ1q\)\\\|v\_\{k,q\}\-\\hat\{v\}\_\{k\}\\\|\_\{2\}\\leq R\_\{v\}\\,\\epsilon\_\{E\}^\{\\kappa\_\{1\}^\{q\}\}\\leq R\_\{v\}\\exp\(\-C\_\{E\}\\kappa\_\{1\}^\{q\}\)\(38\)where the last inequality holds becauseϵE≤ρE=exp⁡\(−CE\)\\epsilon\_\{E\}\\leq\\rho\_\{E\}=\\exp\(\-C\_\{E\}\): indeed, every column ofYkY\_\{k\}has norm at most\(1\+γ\)/T\\nicefrac\{\{\(1\+\\gamma\)\}\}\{\{\\sqrt\{T\}\}\}, such that‖Yk⊤​Φk‖2≤‖Yk‖F​‖Φk‖2≤1\+γ\\\|Y\_\{k\}^\{\\top\}\\Phi\_\{k\}\\\|\_\{2\}\\leq\\\|Y\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\\|\\Phi\_\{k\}\\\|\_\{2\}\\leq 1\+\\gammagivesσmax​\(Hk\)≤λ\+1\+γ\\sigma\_\{\\max\}\(H\_\{k\}\)\\leq\\lambda\+1\+\\gamma, such that the range ofckc\_\{k\}in Theorem[4](https://arxiv.org/html/2610.33477#Thmtheorem4)is nonempty, and the Woodbury identityHk−1=λ−1​\[IT−Yk⊤​\(A^k\+λ​Id\)−1​Φk\]H\_\{k\}^\{\-1\}=\\lambda^\{\-1\}\[I\_\{T\}\-Y\_\{k\}^\{\\top\}\(\\hat\{A\}\_\{k\}\+\\lambda I\_\{d\}\)^\{\-1\}\\Phi\_\{k\}\]and‖\(A^k\+λ​Id\)−1‖2≤1/λ\\\|\(\\hat\{A\}\_\{k\}\+\\lambda I\_\{d\}\)^\{\-1\}\\\|\_\{2\}\\leq\\nicefrac\{\{1\}\}\{\{\\lambda\}\}giveσmin​\(Hk\)≥λ2/\(λ\+1\+γ\)\\sigma\_\{\\min\}\(H\_\{k\}\)\\geq\\nicefrac\{\{\\lambda^\{2\}\}\}\{\{\(\\lambda\+1\+\\gamma\)\}\}, such thatck≤λ\+1\+γc\_\{k\}\\leq\\lambda\+1\+\\gammayieldsϵE≤1−σmin2​\(Hk\)/\(λ\+1\+γ\)2≤ρE\\epsilon\_\{E\}\\leq 1\-\\nicefrac\{\{\\sigma\_\{\\min\}^\{2\}\(H\_\{k\}\)\}\}\{\{\(\\lambda\+1\+\\gamma\)^\{2\}\}\}\\leq\\rho\_\{E\}\.

*Statistical error\.*Sincev^k=\(A^k\+λ​Id\)−1​b^k\\hat\{v\}\_\{k\}=\(\\hat\{A\}\_\{k\}\+\\lambda I\_\{d\}\)^\{\-1\}\\hat\{b\}\_\{k\}and\(Ak\+λ​Id\)​vk∗=bk\(A\_\{k\}\+\\lambda I\_\{d\}\)v\_\{k\}^\{\*\}=b\_\{k\}, we havev^k−vk∗=\(A^k\+λ​Id\)−1​\[\(b^k−bk\)−\(A^k−Ak\)​vk∗\]\\hat\{v\}\_\{k\}\-v\_\{k\}^\{\*\}=\(\\hat\{A\}\_\{k\}\+\\lambda I\_\{d\}\)^\{\-1\}\[\(\\hat\{b\}\_\{k\}\-b\_\{k\}\)\-\(\\hat\{A\}\_\{k\}\-A\_\{k\}\)v\_\{k\}^\{\*\}\], where‖\(A^k\+λ​Id\)−1‖2≤1/λ\\\|\(\\hat\{A\}\_\{k\}\+\\lambda I\_\{d\}\)^\{\-1\}\\\|\_\{2\}\\leq\\nicefrac\{\{1\}\}\{\{\\lambda\}\}and‖vk∗‖2≤Rr/λ\\\|v\_\{k\}^\{\*\}\\\|\_\{2\}\\leq\\nicefrac\{\{R\_\{r\}\}\}\{\{\\lambda\}\}\. Fork≥1k\\geq 1, conditionally onℱk−1\{\\cal F\}\_\{k\-1\}, the frameOkO\_\{k\}is generated byπWk\\pi\_\{W\_\{k\}\}fromsk,0s\_\{k,0\}, its law is measurable in\(Wk,sk,0\)\(W\_\{k\},s\_\{k,0\}\)by Assumption[1](https://arxiv.org/html/2610.33477#Thmassump1), andvk∗=v∗​\(Wk\)v\_\{k\}^\{\*\}=v^\{\*\}\(W\_\{k\}\)isℱk−1\{\\cal F\}\_\{k\-1\}\-measurable becausev∗v^\{\*\}is continuous \(Lemma[3](https://arxiv.org/html/2610.33477#Thmtheorem3)\)\. Lemma[20](https://arxiv.org/html/2610.33477#Thmtheorem20), applied conditionally onℱk−1\{\\cal F\}\_\{k\-1\}tofb​\(o\):=ϕ⁡\(s\)​r​\(s,a\)f\_\{b\}\(o\):=\\phi\(s\)r\(s,a\)withB=RrB=R\_\{r\}and tofA​\(o\):=ϕ⁡\(s\)​\(ϕ⁡\(s\)−γ​ϕ​\(s′\)\)⊤f\_\{A\}\(o\):=\\phi\(s\)\(\\phi\(s\)\-\\gamma\\phi\(s^\{\\prime\}\)\)^\{\\top\}in the Frobenius norm withB=1\+γB=1\+\\gamma, bounds the errors ofb^k\\hat\{b\}\_\{k\}and of the averageA¯k:=1T​∑t<TfA​\(ok,t\)\\bar\{A\}\_\{k\}:=\\frac\{1\}\{T\}\\sum\_\{t<T\}f\_\{A\}\(o\_\{k,t\}\)over the frame, which differs from the pathwise LSTD matrix only in the dropped bootstrap,A^k=A¯k\+γT​ϕk,T−1​ϕk,T⊤\\hat\{A\}\_\{k\}=\\bar\{A\}\_\{k\}\+\\frac\{\\gamma\}\{T\}\\phi\_\{k,T\-1\}\\phi\_\{k,T\}^\{\\top\}\. Hence,∥⋅∥2≤∥⋅∥F\\\|\\cdot\\\|\_\{2\}\\leq\\\|\\cdot\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}, Minkowski’s inequality, andγ/T≤γ​\(1\+4​τmix\)/T\\nicefrac\{\{\\gamma\}\}\{\{T\}\}\\leq\\gamma\\sqrt\{\\nicefrac\{\{\(1\+4\\tau\_\{\\mathrm\{mix\}\}\)\}\}\{\{T\}\}\}give

𝔼⁡\[‖b^k−bk‖22\|ℱk−1\]≤4​Rr2​\(1\+4​τmix\)T,𝔼⁡\[‖A^k−Ak‖22\|ℱk−1\]≤\(2\+3​γ\)2​\(1\+4​τmix\)T\\mathbb\{E\}\[\\\|\\hat\{b\}\_\{k\}\-b\_\{k\}\\\|\_\{2\}^\{2\}\\,\|\\,\{\\cal F\}\_\{k\-1\}\]\\leq\\frac\{4R\_\{r\}^\{2\}\(1\+4\\tau\_\{\\mathrm\{mix\}\}\)\}\{T\},\\quad\\mathbb\{E\}\[\\\|\\hat\{A\}\_\{k\}\-A\_\{k\}\\\|\_\{2\}^\{2\}\\,\|\\,\{\\cal F\}\_\{k\-1\}\]\\leq\\frac\{\(2\+3\\gamma\)^\{2\}\(1\+4\\tau\_\{\\mathrm\{mix\}\}\)\}\{T\}\(39\)for every frame, sinceτmix\\tau\_\{\\mathrm\{mix\}\}is uniform over the policies and the start states; for the warm\-up frame, the bounds hold for every start state and hence fors0,0∼ρs\_\{0,0\}\\sim\\rho\. Hence,\(x\+y\)2≤2​x2\+2​y2\(x\+y\)^\{2\}\\leq 2x^\{2\}\+2y^\{2\}and1\+\(1\+3​γ/2\)2/λ2≤\(λ\+1\+3​γ/2\)2/λ21\+\\nicefrac\{\{\(1\+\\nicefrac\{\{3\\gamma\}\}\{\{2\}\}\)^\{2\}\}\}\{\{\\lambda^\{2\}\}\}\\leq\\nicefrac\{\{\(\\lambda\+1\+\\nicefrac\{\{3\\gamma\}\}\{\{2\}\}\)^\{2\}\}\}\{\{\\lambda^\{2\}\}\}give

𝔼⁡\[‖v^k−vk∗‖22\|ℱk−1\]≤8​Rr2​\(1\+4​τmix\)λ2​T​\(1\+\(1\+3​γ/2\)2λ2\)≤4​\(1\+4​τmix\)​Cv2T\.\\mathbb\{E\}\[\\\|\\hat\{v\}\_\{k\}\-v\_\{k\}^\{\*\}\\\|\_\{2\}^\{2\}\\,\|\\,\{\\cal F\}\_\{k\-1\}\]\\leq\\frac\{8R\_\{r\}^\{2\}\(1\+4\\tau\_\{\\mathrm\{mix\}\}\)\}\{\\lambda^\{2\}T\}\\Big\(1\+\\frac\{\(1\+\\nicefrac\{\{3\\gamma\}\}\{\{2\}\}\)^\{2\}\}\{\\lambda^\{2\}\}\\Big\)\\leq\\frac\{4\(1\+4\\tau\_\{\\mathrm\{mix\}\}\)\\,C\_\{v\}^\{2\}\}\{T\}\.\(40\)
*Combination\.*Sincevk=vk,P1v\_\{k\}=v\_\{k,P\_\{1\}\}, \([38](https://arxiv.org/html/2610.33477#A6.E38)\) withq=P1q=P\_\{1\}, \([40](https://arxiv.org/html/2610.33477#A6.E40)\), and the conditional Minkowski inequality give𝔼⁡\[‖vk−vk∗‖22\|ℱk−1\]≤Rv​exp⁡\(−CE​κ1P1\)\+2​Cv​\(1\+4​τmix\)/T=ϵlstd\\sqrt\{\\mathbb\{E\}\[\\\|v\_\{k\}\-v\_\{k\}^\{\*\}\\\|\_\{2\}^\{2\}\\,\|\\,\{\\cal F\}\_\{k\-1\}\]\}\\leq R\_\{v\}\\exp\(\-C\_\{E\}\\kappa\_\{1\}^\{P\_\{1\}\}\)\+2C\_\{v\}\\sqrt\{\\nicefrac\{\{\(1\+4\\tau\_\{\\mathrm\{mix\}\}\)\}\}\{\{T\}\}\}=\\epsilon\_\{\\rm lstd\}, which is \([19](https://arxiv.org/html/2610.33477#S4.E19)\)\. The critic is an NSI iterate, whose norm is at mostRvR\_\{v\}by Lemma[5](https://arxiv.org/html/2610.33477#Thmtheorem5)\(i\); the mean\-square form follows from\(x\+y\)2≤2​x2\+2​y2\(x\+y\)^\{2\}\\leq 2x^\{2\}\+2y^\{2\}, and, fork=0k=0,ℱ−1\{\\cal F\}\_\{\-1\}is trivial and the bounds hold for every start state and hence fors0,0∼ρs\_\{0,0\}\\sim\\rho\.

## Appendix GProof of Lemma[5](https://arxiv.org/html/2610.33477#Thmtheorem5)

Proof\.*Part \(i\)\.*By the proof of Theorem[4](https://arxiv.org/html/2610.33477#Thmtheorem4),Ek,qE\_\{k,q\}is symmetric with eigenvalues in\[0,1\)\[0,1\), such that‖IT−Ek,q‖2≤1\\\|I\_\{T\}\-E\_\{k,q\}\\\|\_\{2\}\\leq 1, and‖Φk​Hk−1‖2≤1/λ\\\|\\Phi\_\{k\}H\_\{k\}^\{\-1\}\\\|\_\{2\}\\leq\\nicefrac\{\{1\}\}\{\{\\lambda\}\}\. Sincevk,q=Φk​Xk,q​rkv\_\{k,q\}=\\Phi\_\{k\}X\_\{k,q\}r\_\{k\}withXk,q=Hk−1​\(IT−Ek,q\)X\_\{k,q\}=H\_\{k\}^\{\-1\}\(I\_\{T\}\-E\_\{k,q\}\)and‖rk‖2≤Rr\\\|r\_\{k\}\\\|\_\{2\}\\leq R\_\{r\}, we obtain‖vk,q‖2≤Rr/λ=Rv\\\|v\_\{k,q\}\\\|\_\{2\}\\leq\\nicefrac\{\{R\_\{r\}\}\}\{\{\\lambda\}\}=R\_\{v\}\. The criticvkv\_\{k\}is the iteratevk,P1v\_\{k,P\_\{1\}\}, such that‖vk‖2≤Rv\\\|v\_\{k\}\\\|\_\{2\}\\leq R\_\{v\}, and the Cauchy\-Schwarz inequality with‖ϕ⁡\(s\)‖2≤1\\\|\\phi\(s\)\\\|\_\{2\}\\leq 1givessups\|ϕ⊤​\(s\)​vk\|≤Rv\\sup\_\{s\}\|\\phi^\{\\top\}\(s\)v\_\{k\}\|\\leq R\_\{v\}, i\.e\.,vkv\_\{k\}is admissible\.

*Part \(ii\)\.*For an admissiblevvand an observationo=\(s,a,s′\)o=\(s,a,s^\{\\prime\}\), the bounds\|r⁡\(s,a\)\|≤Rr\|r\(s,a\)\|\\leq R\_\{r\},\|ϕ⊤​\(s′\)​v\|≤Rv\|\\phi^\{\\top\}\(s^\{\\prime\}\)v\|\\leq R\_\{v\}, and\|ϕ⊤​\(s\)​v\|≤Rv\|\\phi^\{\\top\}\(s\)v\|\\leq R\_\{v\}give\|δv​\(o\)\|≤Rr\+\(1\+γ\)​Rv\|\\delta\_\{v\}\(o\)\|\\leq R\_\{r\}\+\(1\+\\gamma\)R\_\{v\}, such that \([9](https://arxiv.org/html/2610.33477#S3.E9)\) gives\|δ^t​\(v\)\|≤∑l≥0γl​\(Rr\+\(1\+γ\)​Rv\)=RA\|\\hat\{\\delta\}\_\{t\}\(v\)\|\\leq\\sum\_\{l\\geq 0\}\\gamma^\{l\}\(R\_\{r\}\+\(1\+\\gamma\)R\_\{v\}\)=R\_\{A\}\.

*Part \(iii\)\.*Since the weights1/T\\nicefrac\{\{1\}\}\{\{T\}\}in \([10](https://arxiv.org/html/2610.33477#S3.E10)\) sum to one, the triangle inequality, part \(ii\), and \([15a](https://arxiv.org/html/2610.33477#S4.E15.1)\) give‖g^​\(W,v,O\)‖∗≤RA​Rπ=Rg\\\|\\hat\{g\}\(W,v;O\)\\\|\_\{\*\}\\leq R\_\{A\}R\_\{\\pi\}=R\_\{g\}; since the criticvk−1v\_\{k\-1\}used inGkG\_\{k\}is admissible by part \(i\),‖Gk‖∗≤Rg\\\|G\_\{k\}\\\|\_\{\*\}\\leq R\_\{g\}\. For the PG, the score has zero mean by dominated convergence and \([15a](https://arxiv.org/html/2610.33477#S4.E15.1)\), such that the per\-state PG satisfies

gW\(s\):=𝔼a∼πW\[\(QW\(s,a\)−VW\(s\)\)∇logπW\(a\|s\)\]=𝔼a∼πW\[QW\(s,a\)∇logπW\(a\|s\)\]g\_\{W\}\(s\):=\\mathbb\{E\}\_\{a\\sim\\pi\_\{W\}\}\[\(Q\_\{W\}\(s,a\)\-V\_\{W\}\(s\)\)\\nabla\\log\\pi\_\{W\}\(a\|s\)\]=\\mathbb\{E\}\_\{a\\sim\\pi\_\{W\}\}\[Q\_\{W\}\(s,a\)\\nabla\\log\\pi\_\{W\}\(a\|s\)\]and\|QW\|≤Rr/\(1−γ\)\|Q\_\{W\}\|\\leq\\nicefrac\{\{R\_\{r\}\}\}\{\{\(1\-\\gamma\)\}\}and \([15a](https://arxiv.org/html/2610.33477#S4.E15.1)\) give‖gW​\(s\)‖∗≤Rr​Rπ/\(1−γ\)\\\|g\_\{W\}\(s\)\\\|\_\{\*\}\\leq\\nicefrac\{\{R\_\{r\}R\_\{\\pi\}\}\}\{\{\(1\-\\gamma\)\}\}\. Since \([2](https://arxiv.org/html/2610.33477#S2.E2)\) gives∇J​\(W\)=𝔼s∼ζW​\[gW​\(s\)\]\\nabla J\(W\)=\\mathbb\{E\}\_\{s\\sim\\zeta\_\{W\}\}\[g\_\{W\}\(s\)\]andζW\\zeta\_\{W\}is a probability measure, we obtain‖∇J​\(W\)‖∗≤Rr​Rπ/\(1−γ\)≤RA​Rπ=Rg\\\|\\nabla J\(W\)\\\|\_\{\*\}\\leq\\nicefrac\{\{R\_\{r\}R\_\{\\pi\}\}\}\{\{\(1\-\\gamma\)\}\}\\leq R\_\{A\}R\_\{\\pi\}=R\_\{g\}; the Frobenius bounds follow from∥⋅∥F≤∥⋅∥∗\\\|\\cdot\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq\\\|\\cdot\\\|\_\{\*\}\.

## Appendix HProof of Lemma[6](https://arxiv.org/html/2610.33477#Thmtheorem6)

Proof\.*Part \(i\)\.*WriteO=\{\(st,at\)\}t=0T−1∪\{sT\}O=\\\{\(s\_\{t\},a\_\{t\}\)\\\}\_\{t=0\}^\{T\-1\}\\cup\\\{s\_\{T\}\\\}, abbreviate∇W′′,t:=∇log⁡πW′′​\(at\|st\)\\nabla\_\{W^\{\\prime\\prime\},t\}:=\\nabla\\log\\pi\_\{W^\{\\prime\\prime\}\}\(a\_\{t\}\|s\_\{t\}\), and letWτ:=W\+τ⁡\(W′−W\)W\_\{\\tau\}:=W\+\\tau\(W^\{\\prime\}\-W\),τ∈\[0,1\]\\tau\\in\[0,1\]\. By Assumption[1](https://arxiv.org/html/2610.33477#Thmassump1),τ↦log⁡πWτ​\(at\|st\)\\tau\\mapsto\\log\\pi\_\{W\_\{\\tau\}\}\(a\_\{t\}\|s\_\{t\}\)is differentiable with derivative⟨∇Wτ,t,W′−W⟩F\\langle\\nabla\_\{W\_\{\\tau\},t\},W^\{\\prime\}\-W\\rangle\_\{\\scriptscriptstyle\\textrm\{F\}\}, such that Hölder’s inequality and \([15a](https://arxiv.org/html/2610.33477#S4.E15.1)\) give\|log⁡πW′​\(at\|st\)/πW​\(at\|st\)\|≤Rπ​ℓ2\|\\log\\nicefrac\{\{\\pi\_\{W^\{\\prime\}\}\(a\_\{t\}\|s\_\{t\}\)\}\}\{\{\\pi\_\{W\}\(a\_\{t\}\|s\_\{t\}\)\}\}\|\\leq R\_\{\\pi\}\\ell\_\{2\}; summing overt<Tt<Tyields\|log⁡w\|≤T​Rπ​ℓ2≤1\|\\log w\|\\leq TR\_\{\\pi\}\\ell\_\{2\}\\leq 1\. Sinceexp⁡\(x\)−1≤\(exp⁡\(1\)−1\)​x\\exp\(x\)\-1\\leq\(\\exp\(1\)\-1\)xforx∈\[0,1\]x\\in\[0,1\]by convexity and1−exp⁡\(−x\)≤x1\-\\exp\(\-x\)\\leq xforx≥0x\\geq 0,\|w−1\|≤\(exp⁡\(1\)−1\)​T​Rπ​ℓ2\|w\-1\|\\leq\(\\exp\(1\)\-1\)TR\_\{\\pi\}\\ell\_\{2\}\.

*Part \(ii\)\.*Fixk≥2k\\geq 2and abbreviatev:=vk−1v:=v\_\{k\-1\}\. Conditionally onℱk−1\{\\cal F\}\_\{k\-1\}, the law ofOkO\_\{k\}is the product of the point mass at theℱk−1\{\\cal F\}\_\{k\-1\}\-measurable start statesk,0s\_\{k,0\}, the policy kernelsπk\(⋅\|st\)\\pi\_\{k\}\(\\cdot\|s\_\{t\}\), and the transition kernelsℙ\(⋅\|st,at\)\\mathbb\{P\}\(\\cdot\|s\_\{t\},a\_\{t\}\),t<Tt<T, and each rewardrt=r⁡\(st,at\)r\_\{t\}=r\(s\_\{t\},a\_\{t\}\)is a function of the state\-action pair\. By the positivity clause of Assumption[1](https://arxiv.org/html/2610.33477#Thmassump1), the policy kernels ofπk−1\\pi\_\{k\-1\}andπk\\pi\_\{k\}are mutually absolutely continuous with density ratioπk−1​\(a\|s\)/πk​\(a\|s\)\\nicefrac\{\{\\pi\_\{k\-1\}\(a\|s\)\}\}\{\{\\pi\_\{k\}\(a\|s\)\}\}; since the start state and the transition kernels are common to both laws, the Radon–Nikodym derivative of the law of aTT\-step trajectory underπk−1\\pi\_\{k\-1\}with respect to that underπk\\pi\_\{k\}iswkw\_\{k\}\. SinceWk−1W\_\{k\-1\},WkW\_\{k\},vv, andsk,0s\_\{k,0\}areℱk−1\{\\cal F\}\_\{k\-1\}\-measurable, this change of measure gives𝔼⁡\[wk​f​\(Ok,θ\)\|ℱk−1\]=𝔼O∼πk−1​\[f⁡\(O,θ\)\]\\mathbb\{E\}\[w\_\{k\}f\(O\_\{k\};\\theta\)\\,\|\\,\{\\cal F\}\_\{k\-1\}\]=\\mathbb\{E\}\_\{O\\sim\\pi\_\{k\-1\}\}\[f\(O;\\theta\)\]for every boundedffthat is jointly measurable in the frame and anℱk−1\{\\cal F\}\_\{k\-1\}\-measurable parameterθ\\theta, whereO∼πk−1O\\sim\\pi\_\{k\-1\}denotes aTT\-step trajectory of the chain induced byπk−1\\pi\_\{k\-1\}froms0=sk,0s\_\{0\}=s\_\{k,0\}\. We takef⁡\(O,θ\)=g^​\(Wk−1,v,O\)f\(O;\\theta\)=\\hat\{g\}\(W\_\{k\-1\},v;O\)withθ=\(Wk−1,v\)\\theta=\(W\_\{k\-1\},v\), which is bounded byRgR\_\{g\}\(Lemma[5](https://arxiv.org/html/2610.33477#Thmtheorem5)\(iii\)\), and obtain𝔼⁡\[G~k\|ℱk−1\]=𝔼O∼πk−1​\[g^​\(Wk−1,v,O\)\]=h⁡\(Wk−1,ωk\)\\mathbb\{E\}\[\\widetilde\{G\}\_\{k\}\\,\|\\,\{\\cal F\}\_\{k\-1\}\]=\\mathbb\{E\}\_\{O\\sim\\pi\_\{k\-1\}\}\[\\hat\{g\}\(W\_\{k\-1\},v;O\)\]=h\(W\_\{k\-1\};\\omega\_\{k\}\)by \([20](https://arxiv.org/html/2610.33477#S4.E20)\)\.

*Part \(iii\)\.*Sinceδ^t​\(v\)\\hat\{\\delta\}\_\{t\}\(v\)does not depend on the actor parameter, \([10](https://arxiv.org/html/2610.33477#S3.E10)\) gives

g^​\(W,v,O\)−w​g^​\(W′,v,O\)=1T​∑t=0T−1δ^t​\(v\)​\[∇W,t−∇W′,t\]\+\(1−w\)​g^​\(W′,v,O\)\.\\hat\{g\}\(W,v;O\)\-w\\,\\hat\{g\}\(W^\{\\prime\},v;O\)=\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\hat\{\\delta\}\_\{t\}\(v\)\\big\[\\nabla\_\{W,t\}\-\\nabla\_\{W^\{\\prime\},t\}\\big\]\+\(1\-w\)\\,\\hat\{g\}\(W^\{\\prime\},v;O\)\.
By\|δ^t​\(v\)\|≤RA\|\\hat\{\\delta\}\_\{t\}\(v\)\|\\leq R\_\{A\}\(Lemma[5](https://arxiv.org/html/2610.33477#Thmtheorem5)\(ii\)\) and \([15b](https://arxiv.org/html/2610.33477#S4.E15.2)\), the first term is at mostRA​Lπ​ℓ2R\_\{A\}L\_\{\\pi\}\\ell\_\{2\}in the nuclear norm; by‖g^​\(W′,v,O\)‖∗≤RA​Rπ\\\|\\hat\{g\}\(W^\{\\prime\},v;O\)\\\|\_\{\*\}\\leq R\_\{A\}R\_\{\\pi\}\(Lemma[5](https://arxiv.org/html/2610.33477#Thmtheorem5)\(iii\)\) and part \(i\), the second term is at most\(exp⁡\(1\)−1\)​T​Rπ2​RA​ℓ2\(\\exp\(1\)\-1\)TR\_\{\\pi\}^\{2\}R\_\{A\}\\ell\_\{2\}, and the two bounds sum toCF​ℓ2C\_\{F\}\\ell\_\{2\}\. The Frobenius bound follows from∥⋅∥F≤∥⋅∥∗\\\|\\cdot\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq\\\|\\cdot\\\|\_\{\*\}\. Finally, withW=WkW=W\_\{k\},W′=Wk−1W^\{\\prime\}=W\_\{k\-1\},O=OkO=O\_\{k\}, andv=vk−1v=v\_\{k\-1\}, the weightwwiswkw\_\{k\}and, by \([10](https://arxiv.org/html/2610.33477#S3.E10)\) and \([11](https://arxiv.org/html/2610.33477#S3.E11)\), the difference in part \(iii\) isGk−G~kG\_\{k\}\-\\widetilde\{G\}\_\{k\}, such that parts \(i\) and \(iii\) withℓ2≤ℓ¯2\\ell\_\{2\}\\leq\\bar\{\\ell\}\_\{2\}give the last claim\.

## Appendix IProof of Lemma[7](https://arxiv.org/html/2610.33477#Thmtheorem7)

We use the notation of Appendix[D](https://arxiv.org/html/2610.33477#A4)\. For everyWWand every signed measureν\\nuon𝒮\{\\cal S\}withν⁡\(𝒮\)=0\\nu\(\{\\cal S\}\)=0, we writeν=‖ν‖TV​\(p−q\)\\nu=\\\|\\nu\\\|\_\{\\mathrm\{TV\}\}\(p\-q\)with probability measuresppandqq, and‖p​PWt−q​PWt‖TV≤sups,s′‖PWt​\(s,⋅\)−PWt​\(s′,⋅\)‖TV≤d~mix​\(t\):=min⁡\{1,2​dmix​\(t\)\}\\\|pP^\{t\}\_\{W\}\-qP^\{t\}\_\{W\}\\\|\_\{\\mathrm\{TV\}\}\\leq\\sup\_\{s,s^\{\\prime\}\}\\\|P^\{t\}\_\{W\}\(s,\\cdot\)\-P^\{t\}\_\{W\}\(s^\{\\prime\},\\cdot\)\\\|\_\{\\mathrm\{TV\}\}\\leq\\tilde\{d\}\_\{\\mathrm\{mix\}\}\(t\):=\\min\\\{1,2d\_\{\\mathrm\{mix\}\}\(t\)\\\}gives the contraction‖ν​PWt‖TV≤d~mix​\(t\)​‖ν‖TV\\\|\\nu P^\{t\}\_\{W\}\\\|\_\{\\mathrm\{TV\}\}\\leq\\tilde\{d\}\_\{\\mathrm\{mix\}\}\(t\)\\\|\\nu\\\|\_\{\\mathrm\{TV\}\}; moreover, \([36](https://arxiv.org/html/2610.33477#A4.E36)\) gives∑t≥0d~mix​\(t\)≤4​τmix\\sum\_\{t\\geq 0\}\\tilde\{d\}\_\{\\mathrm\{mix\}\}\(t\)\\leq 4\\tau\_\{\\mathrm\{mix\}\}\. We prove parts \(i\) and \(ii\) in the nuclear norm, which implies the Frobenius norm since∥⋅∥F≤∥⋅∥∗\\\|\\cdot\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq\\\|\\cdot\\\|\_\{\*\}\. For framekk, conditionally onℱk−1\{\\cal F\}\_\{k\-1\}, the frameOkO\_\{k\}is aTT\-step trajectory of the chain induced byπk\\pi\_\{k\}fromsk,0s\_\{k,0\}\.

*Part \(i\)\.*FixWWand an admissibleω=\(v,s0\)\\omega=\(v,s\_\{0\}\), letO=\{\(st,at\)\}t=0T−1∪\{sT\}O=\\\{\(s\_\{t\},a\_\{t\}\)\\\}\_\{t=0\}^\{T\-1\}\\cup\\\{s\_\{T\}\\\}be aTT\-step trajectory of the chain induced byπW\\pi\_\{W\}froms0s\_\{0\}, such that all expectations below are conditional ons0s\_\{0\}, abbreviate∇t:=∇log⁡πW​\(at\|st\)\\nabla\_\{t\}:=\\nabla\\log\\pi\_\{W\}\(a\_\{t\}\|s\_\{t\}\), and letδ^t∘\\hat\{\\delta\}^\{\\circ\}\_\{t\}denoteδ^t\\hat\{\\delta\}\_\{t\}in \([9](https://arxiv.org/html/2610.33477#S3.E9)\) withVWV\_\{W\}in place of the critic\. In the telescoped form \([9](https://arxiv.org/html/2610.33477#S3.E9)\), the rewards cancel, such thatδ^t​\(v\)−δ^t∘=γT−t​ϵv​\(sT\)−ϵv​\(st\)\\hat\{\\delta\}\_\{t\}\(v\)\-\\hat\{\\delta\}^\{\\circ\}\_\{t\}=\\gamma^\{T\-t\}\\epsilon\_\{v\}\(s\_\{T\}\)\-\\epsilon\_\{v\}\(s\_\{t\}\)\. Facts \(i\) and \(ii\) below \([2](https://arxiv.org/html/2610.33477#S2.E2)\) hold for any number of summands and, by the Markov property, conditionally onℋt:=\(s0,a0,…,st,at\)\{\\cal H\}\_\{t\}:=\(s\_\{0\},a\_\{0\},\\ldots,s\_\{t\},a\_\{t\}\), such that𝔼⁡\[δ^t∘\|ℋt\]=QW​\(st,at\)−VW​\(st\)\\mathbb\{E\}\[\\hat\{\\delta\}^\{\\circ\}\_\{t\}\\,\|\\,\{\\cal H\}\_\{t\}\]=Q\_\{W\}\(s\_\{t\},a\_\{t\}\)\-V\_\{W\}\(s\_\{t\}\); since∇t\\nabla\_\{t\}is a function of\(st,at\)\(s\_\{t\},a\_\{t\}\)andat∼πW\(⋅\|st\)a\_\{t\}\\sim\\pi\_\{W\}\(\\cdot\|s\_\{t\}\)givens0,a0,…,sts\_\{0\},a\_\{0\},\\ldots,s\_\{t\}, the tower property gives𝔼⁡\[δ^t∘​∇t\]=𝔼⁡\[gW​\(st\)\]\\mathbb\{E\}\[\\hat\{\\delta\}^\{\\circ\}\_\{t\}\\nabla\_\{t\}\]=\\mathbb\{E\}\[g\_\{W\}\(s\_\{t\}\)\]withgWg\_\{W\}of Appendix[G](https://arxiv.org/html/2610.33477#A7), and, sincevvis fixed and the score has zero mean,𝔼⁡\[ϵv​\(st\)​∇t\]=0\\mathbb\{E\}\[\\epsilon\_\{v\}\(s\_\{t\}\)\\nabla\_\{t\}\]=0\. Therefore, sinceg⁡\(W\)=𝔼s∼μW​\[gW​\(s\)\]g\(W\)=\\mathbb\{E\}\_\{s\\sim\\mu\_\{W\}\}\[g\_\{W\}\(s\)\], \([10](https://arxiv.org/html/2610.33477#S3.E10)\) and \([20](https://arxiv.org/html/2610.33477#S4.E20)\) splitDDinto a transient part and a critic part,

D\(W;ω\)=1T​∑t=0T−1\(𝔼⁡\[gW​\(st\)\]−𝔼s∼μW​\[gW​\(s\)\]\)⏟=:Dtr​\(W,s0\)\+1T​∑t=0T−1γT−t​𝔼​\[ϵv​\(sT\)​∇t\]⏟=:Dcr​\(W,ω\)\.D\(W;\\omega\)=\\underbrace\{\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\Big\(\\mathbb\{E\}\[g\_\{W\}\(s\_\{t\}\)\]\-\\mathbb\{E\}\_\{s\\sim\\mu\_\{W\}\}\[g\_\{W\}\(s\)\]\\Big\)\}\_\{=:\\,D^\{\\rm tr\}\(W;s\_\{0\}\)\}\+\\underbrace\{\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\gamma^\{T\-t\}\\,\\mathbb\{E\}\[\\epsilon\_\{v\}\(s\_\{T\}\)\\nabla\_\{t\}\]\}\_\{=:\\,D^\{\\rm cr\}\(W;\\omega\)\}\.\(41\)Since‖gW​\(s\)‖∗≤Rr​Rπ/\(1−γ\)\\\|g\_\{W\}\(s\)\\\|\_\{\*\}\\leq\\nicefrac\{\{R\_\{r\}R\_\{\\pi\}\}\}\{\{\(1\-\\gamma\)\}\}\(Appendix[G](https://arxiv.org/html/2610.33477#A7)\) and𝔼⁡\[gW​\(st\)\]=∫gW​d​PWt​\(s0,⋅\)\\mathbb\{E\}\[g\_\{W\}\(s\_\{t\}\)\]=\\int g\_\{W\}\\,dP^\{t\}\_\{W\}\(s\_\{0\},\\cdot\), \([36](https://arxiv.org/html/2610.33477#A4.E36)\) gives‖Dtr​\(W,s0\)‖∗≤2​Rr​Rπ\(1−γ\)​T​∑t<Tdmix​\(t\)≤4​Rr​Rπ​τmix/\[\(1−γ\)​T\]\\\|D^\{\\rm tr\}\(W;s\_\{0\}\)\\\|\_\{\*\}\\leq\\frac\{2R\_\{r\}R\_\{\\pi\}\}\{\(1\-\\gamma\)T\}\\sum\_\{t<T\}d\_\{\\mathrm\{mix\}\}\(t\)\\leq\\nicefrac\{\{4R\_\{r\}R\_\{\\pi\}\\tau\_\{\\mathrm\{mix\}\}\}\}\{\{\[\(1\-\\gamma\)T\]\}\}, and \([15a](https://arxiv.org/html/2610.33477#S4.E15.1)\) and∑t<TγT−t≤γ/\(1−γ\)\\sum\_\{t<T\}\\gamma^\{T\-t\}\\leq\\nicefrac\{\{\\gamma\}\}\{\{\(1\-\\gamma\)\}\}give‖Dcr​\(W,ω\)‖∗≤γ​Rπ​‖ϵv‖∞/\[\(1−γ\)​T\]\\\|D^\{\\rm cr\}\(W;\\omega\)\\\|\_\{\*\}\\leq\\nicefrac\{\{\\gamma R\_\{\\pi\}\\\|\\epsilon\_\{v\}\\\|\_\{\\infty\}\}\}\{\{\[\(1\-\\gamma\)T\]\}\}, which proves the first claim\. For framekk,𝔼⁡\[Gk\|ℱk−1\]=h⁡\(Wk,ωk\)\\mathbb\{E\}\[G\_\{k\}\\,\|\\,\{\\cal F\}\_\{k\-1\}\]=h\(W\_\{k\};\\omega\_\{k\}\)\(Section[4\.2](https://arxiv.org/html/2610.33477#S4.SS2)\), whereωk\\omega\_\{k\}is admissible by Lemma[5](https://arxiv.org/html/2610.33477#Thmtheorem5)\(i\) andϵvk−1=ϵk\\epsilon\_\{v\_\{k\-1\}\}=\\epsilon\_\{k\}atW=WkW=W\_\{k\}; hence,‖B⁡\(Wk,ωk\)‖∗≤‖B¯​\(Wk\)‖∗\+‖D⁡\(Wk,ωk\)‖∗\\\|B\(W\_\{k\};\\omega\_\{k\}\)\\\|\_\{\*\}\\leq\\\|\\bar\{B\}\(W\_\{k\}\)\\\|\_\{\*\}\+\\\|D\(W\_\{k\};\\omega\_\{k\}\)\\\|\_\{\*\}, the definition ofϵdist\\epsilon\_\{\\rm dist\}, and Minkowski’s inequality with𝔼⁡\[‖ϵk‖∞2\]≤ϵcrit\\sqrt\{\\mathbb\{E\}\[\\\|\\epsilon\_\{k\}\\\|\_\{\\infty\}^\{2\}\]\}\\leq\\epsilon\_\{\\rm crit\}yield𝔼⁡\[‖B⁡\(Wk,ωk\)‖∗2\]≤ϵbias\\sqrt\{\\mathbb\{E\}\[\\\|B\(W\_\{k\};\\omega\_\{k\}\)\\\|\_\{\*\}^\{2\}\]\}\\leq\\epsilon\_\{\\rm bias\}\.

*Part \(ii\)\.*FixWW,W′W^\{\\prime\}, and an admissibleω=\(v,s0\)\\omega=\(v,s\_\{0\}\), letℓ2:=‖W−W′‖2\\ell\_\{2\}:=\\\|W\-W^\{\\prime\}\\\|\_\{2\}, and letCDR:=τmix2​RA​Rπ2/\(1−γ\)2C\_\{D\}^\{R\}:=\\nicefrac\{\{\\tau\_\{\\mathrm\{mix\}\}^\{2\}R\_\{A\}R\_\{\\pi\}^\{2\}\}\}\{\{\(1\-\\gamma\)^\{2\}\}\}andCDS:=τmix2​RA​Lπ/\(1−γ\)2C\_\{D\}^\{S\}:=\\nicefrac\{\{\\tau\_\{\\mathrm\{mix\}\}^\{2\}R\_\{A\}L\_\{\\pi\}\}\}\{\{\(1\-\\gamma\)^\{2\}\}\}, such thatDF=\(46​CDR\+5​CDS\)/TD\_\{F\}=\\nicefrac\{\{\(46C\_\{D\}^\{R\}\+5C\_\{D\}^\{S\}\)\}\}\{\{T\}\}; recall thatτmix≥1\\tau\_\{\\mathrm\{mix\}\}\\geq 1,Rr/\(1−γ\)≤RA\\nicefrac\{\{R\_\{r\}\}\}\{\{\(1\-\\gamma\)\}\}\\leq R\_\{A\}, and‖ϕ⊤​v−VW′′‖∞≤ϵ¯:=Rv\+Rr/\(1−γ\)≤RA\\\|\\phi^\{\\top\}v\-V\_\{W^\{\\prime\\prime\}\}\\\|\_\{\\infty\}\\leq\\bar\{\\epsilon\}:=R\_\{v\}\+\\nicefrac\{\{R\_\{r\}\}\}\{\{\(1\-\\gamma\)\}\}\\leq R\_\{A\}for everyW′′W^\{\\prime\\prime\}\. By \([41](https://arxiv.org/html/2610.33477#A9.E41)\), it suffices to bound the changes ofDtrD^\{\\rm tr\}andDcrD^\{\\rm cr\}, for which we use, for bounded matrix\-valuedf,f′f,f^\{\\prime\}and finite signed measuresν,ν′\\nu,\\nu^\{\\prime\}with\(ν−ν′\)​\(𝒮\)=0\(\\nu\-\\nu^\{\\prime\}\)\(\{\\cal S\}\)=0,

‖∫f​𝑑ν−∫f′​d​ν′‖∗≤sups‖f⁡\(s\)−f′​\(s\)‖∗​\|ν\|​\(𝒮\)\+2​sups‖f′​\(s\)‖∗​‖ν−ν′‖TV\.\\\|\\int f\\,d\\nu\-\\int f^\{\\prime\}\\,d\\nu^\{\\prime\}\\\|\_\{\*\}\\leq\\sup\\nolimits\_\{s\}\\\|f\(s\)\-f^\{\\prime\}\(s\)\\\|\_\{\*\}\\,\|\\nu\|\(\{\\cal S\}\)\+2\\sup\\nolimits\_\{s\}\\\|f^\{\\prime\}\(s\)\\\|\_\{\*\}\\,\\\|\\nu\-\\nu^\{\\prime\}\\\|\_\{\\mathrm\{TV\}\}\.\(42\)
*Step 1: policy and chain\.*AlongWτ:=W′\+τ⁡\(W−W′\)W\_\{\\tau\}:=W^\{\\prime\}\+\\tau\(W\-W^\{\\prime\}\),τ∈\[0,1\]\\tau\\in\[0,1\], we havedd​τ​πWτ​\(a\|s\)=πWτ​\(a\|s\)​⟨∇log⁡πWτ​\(a\|s\),W−W′⟩F\\frac\{d\}\{d\\tau\}\\pi\_\{W\_\{\\tau\}\}\(a\|s\)=\\pi\_\{W\_\{\\tau\}\}\(a\|s\)\\langle\\nabla\\log\\pi\_\{W\_\{\\tau\}\}\(a\|s\),W\-W^\{\\prime\}\\rangle\_\{\\scriptscriptstyle\\textrm\{F\}\}, where Hölder’s inequality and \([15a](https://arxiv.org/html/2610.33477#S4.E15.1)\) bound the inner product byRπ​ℓ2R\_\{\\pi\}\\ell\_\{2\}; integrating overτ\\tauandaayields

sups∥πW\(⋅\|s\)−πW′\(⋅\|s\)∥TV≤ϵπ:=12Rπℓ2\\sup\\nolimits\_\{s\}\\\|\\pi\_\{W\}\(\\cdot\|s\)\-\\pi\_\{W^\{\\prime\}\}\(\\cdot\|s\)\\\|\_\{\\mathrm\{TV\}\}\\leq\\epsilon\_\{\\pi\}:=\\tfrac\{1\}\{2\}R\_\{\\pi\}\\ell\_\{2\}\(43\)and the same bound for the state kernelsPW\(s,⋅\):=∫πW\(da\|s\)ℙ\(⋅\|s,a\)P\_\{W\}\(s,\\cdot\):=\\int\\pi\_\{W\}\(da\|s\)\\,\\mathbb\{P\}\(\\cdot\|s,a\), such that‖p⁡\(PW−PW′\)‖TV≤ϵπ\\\|p\(P\_\{W\}\-P\_\{W^\{\\prime\}\}\)\\\|\_\{\\mathrm\{TV\}\}\\leq\\epsilon\_\{\\pi\}for every probability measureppand‖ν⁡\(PW−PW′\)‖TV≤2​ϵπ​‖ν‖TV\\\|\\nu\(P\_\{W\}\-P\_\{W^\{\\prime\}\}\)\\\|\_\{\\mathrm\{TV\}\}\\leq 2\\epsilon\_\{\\pi\}\\\|\\nu\\\|\_\{\\mathrm\{TV\}\}ifν⁡\(𝒮\)=0\\nu\(\{\\cal S\}\)=0\. The identityPWt−PW′t=∑i=0t−1PWi​\(PW−PW′\)​PW′t−1−iP^\{t\}\_\{W\}\-P^\{t\}\_\{W^\{\\prime\}\}=\\sum\_\{i=0\}^\{t\-1\}P^\{i\}\_\{W\}\(P\_\{W\}\-P\_\{W^\{\\prime\}\}\)P^\{t\-1\-i\}\_\{W^\{\\prime\}\}and the contraction give, for allssandtt,

‖PWt​\(s,⋅\)−PW′t​\(s,⋅\)‖TV≤ϵπ​∑j=0t−1d~mix​\(j\)≤4​τmix​ϵπ,‖μW−μW′‖TV≤4​τmix​ϵπ\\\|P^\{t\}\_\{W\}\(s,\\cdot\)\-P^\{t\}\_\{W^\{\\prime\}\}\(s,\\cdot\)\\\|\_\{\\mathrm\{TV\}\}\\leq\\epsilon\_\{\\pi\}\\sum\_\{j=0\}^\{t\-1\}\\tilde\{d\}\_\{\\mathrm\{mix\}\}\(j\)\\leq 4\\tau\_\{\\mathrm\{mix\}\}\\epsilon\_\{\\pi\},\\qquad\\\|\\mu\_\{W\}\-\\mu\_\{W^\{\\prime\}\}\\\|\_\{\\mathrm\{TV\}\}\\leq 4\\tau\_\{\\mathrm\{mix\}\}\\epsilon\_\{\\pi\}\(44\)where the second bound follows from the first, since, for everytt, the distance‖μW−μW′‖TV\\\|\\mu\_\{W\}\-\\mu\_\{W^\{\\prime\}\}\\\|\_\{\\mathrm\{TV\}\}is at most‖PWt​\(s,⋅\)−PW′t​\(s,⋅\)‖TV\+2​dmix​\(t\)\\\|P^\{t\}\_\{W\}\(s,\\cdot\)\-P^\{t\}\_\{W^\{\\prime\}\}\(s,\\cdot\)\\\|\_\{\\mathrm\{TV\}\}\+2d\_\{\\mathrm\{mix\}\}\(t\), anddmix​\(t\)→0d\_\{\\mathrm\{mix\}\}\(t\)\\to 0ast→∞t\\to\\infty; Step 1 holds for any two parametersWWandW′W^\{\\prime\}\.

*Step 2: value function and per\-state gradient\.*By the Bellman equation,VW−VW′=∑t≥0γt​PWt​uV\_\{W\}\-V\_\{W^\{\\prime\}\}=\\sum\_\{t\\geq 0\}\\gamma^\{t\}P^\{t\}\_\{W\}uwithu\(s\):=\(𝔼a∼πW\(⋅\|s\)−𝔼a∼πW′\(⋅\|s\)\)\[QW′\(s,a\)\]u\(s\):=\(\\mathbb\{E\}\_\{a\\sim\\pi\_\{W\}\(\\cdot\|s\)\}\-\\mathbb\{E\}\_\{a\\sim\\pi\_\{W^\{\\prime\}\}\(\\cdot\|s\)\}\)\[Q\_\{W^\{\\prime\}\}\(s,a\)\], where\|u⁡\(s\)\|≤2​ϵπ​Rr/\(1−γ\)\|u\(s\)\|\\leq\\nicefrac\{\{2\\epsilon\_\{\\pi\}R\_\{r\}\}\}\{\{\(1\-\\gamma\)\}\}by \([43](https://arxiv.org/html/2610.33477#A9.E43)\); hence,‖VW−VW′‖∞≤LV​ℓ2\\\|V\_\{W\}\-V\_\{W^\{\\prime\}\}\\\|\_\{\\infty\}\\leq L\_\{V\}\\ell\_\{2\}and\|QW−QW′\|≤γ​LV​ℓ2\|Q\_\{W\}\-Q\_\{W^\{\\prime\}\}\|\\leq\\gamma L\_\{V\}\\ell\_\{2\}withLV:=Rπ​Rr/\(1−γ\)2≤Rπ​RA/\(1−γ\)L\_\{V\}:=\\nicefrac\{\{R\_\{\\pi\}R\_\{r\}\}\}\{\{\(1\-\\gamma\)^\{2\}\}\}\\leq\\nicefrac\{\{R\_\{\\pi\}R\_\{A\}\}\}\{\{\(1\-\\gamma\)\}\}\. SplittinggW​\(s\)−gW′​\(s\)g\_\{W\}\(s\)\-g\_\{W^\{\\prime\}\}\(s\), withgW\(s\)=𝔼a∼πW\(⋅\|s\)\[QW\(s,a\)∇logπW\(a\|s\)\]g\_\{W\}\(s\)=\\mathbb\{E\}\_\{a\\sim\\pi\_\{W\}\(\\cdot\|s\)\}\[Q\_\{W\}\(s,a\)\\nabla\\log\\pi\_\{W\}\(a\|s\)\], into the changes of the policy, ofQQ, and of the score, \([43](https://arxiv.org/html/2610.33477#A9.E43)\), \([15a](https://arxiv.org/html/2610.33477#S4.E15.1)\), and \([15b](https://arxiv.org/html/2610.33477#S4.E15.2)\) give

sups‖gW​\(s\)−gW′​\(s\)‖∗≤Rr​\(Rπ2\+\(1−γ\)​Lπ\)\(1−γ\)2​ℓ2≤RA​\(Rπ2\+Lπ\)1−γ​ℓ2\.\\sup\\nolimits\_\{s\}\\\|g\_\{W\}\(s\)\-g\_\{W^\{\\prime\}\}\(s\)\\\|\_\{\*\}\\leq\\frac\{R\_\{r\}\\big\(R\_\{\\pi\}^\{2\}\+\(1\-\\gamma\)L\_\{\\pi\}\\big\)\}\{\(1\-\\gamma\)^\{2\}\}\\ell\_\{2\}\\leq\\frac\{R\_\{A\}\(R\_\{\\pi\}^\{2\}\+L\_\{\\pi\}\)\}\{1\-\\gamma\}\\ell\_\{2\}\.\(45\)
*Step 3: transient part\.*Letνt:=PWt​\(s0,⋅\)−μW\\nu\_\{t\}:=P^\{t\}\_\{W\}\(s\_\{0\},\\cdot\)\-\\mu\_\{W\}andνt′:=PW′t​\(s0,⋅\)−μW′\\nu^\{\\prime\}\_\{t\}:=P^\{t\}\_\{W^\{\\prime\}\}\(s\_\{0\},\\cdot\)\-\\mu\_\{W^\{\\prime\}\}, such thatDtr​\(W,s0\)=1T​∑t<T∫gW​d​νtD^\{\\rm tr\}\(W;s\_\{0\}\)=\\frac\{1\}\{T\}\\sum\_\{t<T\}\\int g\_\{W\}\\,d\\nu\_\{t\}and‖νt‖TV≤dmix​\(t\)\\\|\\nu\_\{t\}\\\|\_\{\\mathrm\{TV\}\}\\leq d\_\{\\mathrm\{mix\}\}\(t\)\. SinceμW\\mu\_\{W\}andμW′\\mu\_\{W^\{\\prime\}\}are invariant,νt\+1=νt​PW\\nu\_\{t\+1\}=\\nu\_\{t\}P\_\{W\}andνt\+1′=νt′​PW′\\nu^\{\\prime\}\_\{t\+1\}=\\nu^\{\\prime\}\_\{t\}P\_\{W^\{\\prime\}\}, such thatνt−νt′=\(μW′−μW\)​PW′t\+∑i=0t−1νi​\(PW−PW′\)​PW′t−1−i\\nu\_\{t\}\-\\nu^\{\\prime\}\_\{t\}=\(\\mu\_\{W^\{\\prime\}\}\-\\mu\_\{W\}\)P^\{t\}\_\{W^\{\\prime\}\}\+\\sum\_\{i=0\}^\{t\-1\}\\nu\_\{i\}\(P\_\{W\}\-P\_\{W^\{\\prime\}\}\)P^\{t\-1\-i\}\_\{W^\{\\prime\}\}, where every summand has zero total mass; hence, \([44](https://arxiv.org/html/2610.33477#A9.E44)\) and the contraction give‖νt−νt′‖TV≤ϵπ​\[4​τmix​d~mix​\(t\)\+∑i=0t−12​dmix​\(i\)​d~mix​\(t−1−i\)\]\\\|\\nu\_\{t\}\-\\nu^\{\\prime\}\_\{t\}\\\|\_\{\\mathrm\{TV\}\}\\leq\\epsilon\_\{\\pi\}\[4\\tau\_\{\\mathrm\{mix\}\}\\tilde\{d\}\_\{\\mathrm\{mix\}\}\(t\)\+\\sum\_\{i=0\}^\{t\-1\}2d\_\{\\mathrm\{mix\}\}\(i\)\\,\\tilde\{d\}\_\{\\mathrm\{mix\}\}\(t\-1\-i\)\], and \([36](https://arxiv.org/html/2610.33477#A4.E36)\) yields∑t≥0‖νt−νt′‖TV≤32​τmix2​ϵπ\\sum\_\{t\\geq 0\}\\\|\\nu\_\{t\}\-\\nu^\{\\prime\}\_\{t\}\\\|\_\{\\mathrm\{TV\}\}\\leq 32\\tau\_\{\\mathrm\{mix\}\}^\{2\}\\epsilon\_\{\\pi\}\. By \([42](https://arxiv.org/html/2610.33477#A9.E42)\) with\|νt\|​\(𝒮\)≤2​dmix​\(t\)\|\\nu\_\{t\}\|\(\{\\cal S\}\)\\leq 2d\_\{\\mathrm\{mix\}\}\(t\), \([45](https://arxiv.org/html/2610.33477#A9.E45)\), \([36](https://arxiv.org/html/2610.33477#A4.E36)\), and‖gW′​\(s\)‖∗≤Rr​Rπ/\(1−γ\)\\\|g\_\{W^\{\\prime\}\}\(s\)\\\|\_\{\*\}\\leq\\nicefrac\{\{R\_\{r\}R\_\{\\pi\}\}\}\{\{\(1\-\\gamma\)\}\},

‖Dtr​\(W,s0\)−Dtr​\(W′,s0\)‖∗≤4​τmix​RA​\(Rπ2\+Lπ\)​ℓ2\(1−γ\)​T\+32​τmix2​Rr​Rπ2​ℓ2\(1−γ\)​T≤\(36​CDR\+4​CDS\)​ℓ2T\.\\\|D^\{\\rm tr\}\(W;s\_\{0\}\)\-D^\{\\rm tr\}\(W^\{\\prime\};s\_\{0\}\)\\\|\_\{\*\}\\leq\\frac\{4\\tau\_\{\\mathrm\{mix\}\}R\_\{A\}\(R\_\{\\pi\}^\{2\}\+L\_\{\\pi\}\)\\ell\_\{2\}\}\{\(1\-\\gamma\)T\}\+\\frac\{32\\tau\_\{\\mathrm\{mix\}\}^\{2\}R\_\{r\}R\_\{\\pi\}^\{2\}\\ell\_\{2\}\}\{\(1\-\\gamma\)T\}\\leq\\frac\{\(36C\_\{D\}^\{R\}\+4C\_\{D\}^\{S\}\)\\ell\_\{2\}\}\{T\}\.
*Step 4: critic part\.*LetϵW:=ϕ⊤​v−VW\\epsilon\_\{W\}:=\\phi^\{\\top\}v\-V\_\{W\}denoteϵv\\epsilon\_\{v\}as a function ofWWfor the fixedvv, such thatDcr​\(W,ω\)=1T​∑t<TγT−t​𝔼​\[ϵW​\(sT\)​∇t\]D^\{\\rm cr\}\(W;\\omega\)=\\frac\{1\}\{T\}\\sum\_\{t<T\}\\gamma^\{T\-t\}\\mathbb\{E\}\[\\epsilon\_\{W\}\(s\_\{T\}\)\\nabla\_\{t\}\]with∇t\\nabla\_\{t\}and the expectation taken underπW\\pi\_\{W\}, and fixt<Tt<T\. By the Markov property,𝔼⁡\[ϵW​\(sT\)​∇t\]=∫ϖW​d​PWt​\(s0,⋅\)\\mathbb\{E\}\[\\epsilon\_\{W\}\(s\_\{T\}\)\\nabla\_\{t\}\]=\\int\\varpi\_\{W\}\\,dP^\{t\}\_\{W\}\(s\_\{0\},\\cdot\)withϖW\(s\):=𝔼a∼πW\(⋅\|s\)\[∇logπW\(a\|s\)FW\(s,a\)\]\\varpi\_\{W\}\(s\):=\\mathbb\{E\}\_\{a\\sim\\pi\_\{W\}\(\\cdot\|s\)\}\[\\nabla\\log\\pi\_\{W\}\(a\|s\)F\_\{W\}\(s,a\)\]andFW​\(s,a\):=∫ℙ⁡\(d​s′\|s,a\)​\(PWT−t−1​ϵW\)​\(s′\)F\_\{W\}\(s,a\):=\\int\\mathbb\{P\}\(ds^\{\\prime\}\|s,a\)\(P^\{T\-t\-1\}\_\{W\}\\epsilon\_\{W\}\)\(s^\{\\prime\}\), where\|FW\|≤ϵ¯\|F\_\{W\}\|\\leq\\bar\{\\epsilon\}and‖ϖW‖∗≤Rπ​ϵ¯\\\|\\varpi\_\{W\}\\\|\_\{\*\}\\leq R\_\{\\pi\}\\bar\{\\epsilon\}\. SinceϵW−ϵW′=VW′−VW\\epsilon\_\{W\}\-\\epsilon\_\{W^\{\\prime\}\}=V\_\{W^\{\\prime\}\}\-V\_\{W\}, Step 2 and \([44](https://arxiv.org/html/2610.33477#A9.E44)\) give\|FW−FW′\|≤\(LV\+4​τmix​Rπ​ϵ¯\)​ℓ2\|F\_\{W\}\-F\_\{W^\{\\prime\}\}\|\\leq\(L\_\{V\}\+4\\tau\_\{\\mathrm\{mix\}\}R\_\{\\pi\}\\bar\{\\epsilon\}\)\\ell\_\{2\}, and splittingϖW−ϖW′\\varpi\_\{W\}\-\\varpi\_\{W^\{\\prime\}\}into the changes of the policy, of the score, and ofFFboundssups‖ϖW​\(s\)−ϖW′​\(s\)‖∗\\sup\_\{s\}\\\|\\varpi\_\{W\}\(s\)\-\\varpi\_\{W^\{\\prime\}\}\(s\)\\\|\_\{\*\}by\[ϵ¯​\(Rπ2\+Lπ\)\+Rπ​LV\+4​τmix​Rπ2​ϵ¯\]​ℓ2\[\\bar\{\\epsilon\}\(R\_\{\\pi\}^\{2\}\+L\_\{\\pi\}\)\+R\_\{\\pi\}L\_\{V\}\+4\\tau\_\{\\mathrm\{mix\}\}R\_\{\\pi\}^\{2\}\\bar\{\\epsilon\}\]\\,\\ell\_\{2\}\. Hence, by \([42](https://arxiv.org/html/2610.33477#A9.E42)\) and \([44](https://arxiv.org/html/2610.33477#A9.E44)\), the change of𝔼⁡\[ϵW​\(sT\)​∇t\]\\mathbb\{E\}\[\\epsilon\_\{W\}\(s\_\{T\}\)\\nabla\_\{t\}\]is at most

\[ϵ¯​\(\(8​τmix\+1\)​Rπ2\+Lπ\)\+Rπ​LV\]​ℓ2≤\[9​τmix​RA​Rπ2\+Rπ2​RA1−γ\+RA​Lπ\]​ℓ2≤\(1−γ\)​\(10​CDR\+CDS\)​ℓ2\\begin\{split\}\\Big\[\\bar\{\\epsilon\}\\big\(\(8\\tau\_\{\\mathrm\{mix\}\}\+1\)R\_\{\\pi\}^\{2\}\+L\_\{\\pi\}\\big\)\+R\_\{\\pi\}L\_\{V\}\\Big\]\\ell\_\{2\}&\\leq\\Big\[9\\tau\_\{\\mathrm\{mix\}\}R\_\{A\}R\_\{\\pi\}^\{2\}\+\\frac\{R\_\{\\pi\}^\{2\}R\_\{A\}\}\{1\-\\gamma\}\+R\_\{A\}L\_\{\\pi\}\\Big\]\\ell\_\{2\}\\\\ &\\leq\(1\-\\gamma\)\\big\(10C\_\{D\}^\{R\}\+C\_\{D\}^\{S\}\\big\)\\ell\_\{2\}\\end\{split\}and∑t<TγT−t≤1/\(1−γ\)\\sum\_\{t<T\}\\gamma^\{T\-t\}\\leq\\nicefrac\{\{1\}\}\{\{\(1\-\\gamma\)\}\}yields‖Dcr​\(W,ω\)−Dcr​\(W′,ω\)‖∗≤\(10​CDR\+CDS\)​ℓ2/T\\\|D^\{\\rm cr\}\(W;\\omega\)\-D^\{\\rm cr\}\(W^\{\\prime\};\\omega\)\\\|\_\{\*\}\\leq\\nicefrac\{\{\(10C\_\{D\}^\{R\}\+C\_\{D\}^\{S\}\)\\ell\_\{2\}\}\}\{\{T\}\}\. Steps 3 and 4 give‖D⁡\(W,ω\)−D⁡\(W′,ω\)‖∗≤\(46​CDR\+5​CDS\)​ℓ2/T=DF​ℓ2\\\|D\(W;\\omega\)\-D\(W^\{\\prime\};\\omega\)\\\|\_\{\*\}\\leq\\nicefrac\{\{\(46C\_\{D\}^\{R\}\+5C\_\{D\}^\{S\}\)\\ell\_\{2\}\}\}\{\{T\}\}=D\_\{F\}\\ell\_\{2\}\.

*Part \(iii\)\.*We first bound the noise of the oracle estimator\.

###### Lemma 21\.

Under Assumptions[1](https://arxiv.org/html/2610.33477#Thmassump1)and[2](https://arxiv.org/html/2610.33477#Thmassump2), fixWWand a start states0s\_\{0\}, letOObe aTT\-step trajectory of the chain induced byπW\\pi\_\{W\}froms0s\_\{0\}, and letg^∘​\(W,O\)\\hat\{g\}^\{\\circ\}\(W;O\)denote \([10](https://arxiv.org/html/2610.33477#S3.E10)\) withVWV\_\{W\}in place ofϕ⊤​v\\phi^\{\\top\}vin \([9](https://arxiv.org/html/2610.33477#S3.E9)\)\. Then𝔼⁡\[‖g^∘​\(W,O\)−g⁡\(W\)‖F2\]≤\(V∘\)2/T\\mathbb\{E\}\[\\\|\\hat\{g\}^\{\\circ\}\(W;O\)\-g\(W\)\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\]\\leq\\nicefrac\{\{\(V^\{\\circ\}\)^\{2\}\}\}\{\{T\}\}\.

Proof\.AbbreviateQ:=QWQ:=Q\_\{W\},V:=VWV:=V\_\{W\}, and∇t:=∇log⁡πW​\(at\|st\)\\nabla\_\{t\}:=\\nabla\\log\\pi\_\{W\}\(a\_\{t\}\|s\_\{t\}\), and letAt:=Q⁡\(st,at\)−V⁡\(st\)A\_\{t\}:=Q\(s\_\{t\},a\_\{t\}\)\-V\(s\_\{t\}\)andξt\+1:=r⁡\(st,at\)\+γ​V​\(st\+1\)−Q⁡\(st,at\)=γ⁡\[V⁡\(st\+1\)−∫ℙ⁡\(d​s′\|st,at\)​V​\(s′\)\]\\xi\_\{t\+1\}:=r\(s\_\{t\},a\_\{t\}\)\+\\gamma V\(s\_\{t\+1\}\)\-Q\(s\_\{t\},a\_\{t\}\)=\\gamma\[V\(s\_\{t\+1\}\)\-\\int\\mathbb\{P\}\(ds^\{\\prime\}\|s\_\{t\},a\_\{t\}\)V\(s^\{\\prime\}\)\], such that the TD error ofVVisδt∘=At\+ξt\+1\\delta^\{\\circ\}\_\{t\}=A\_\{t\}\+\\xi\_\{t\+1\}\. The telescoped form \([9](https://arxiv.org/html/2610.33477#S3.E9)\) withVVin place ofϕ⊤​v\\phi^\{\\top\}visδ^t∘=∑i=tT−1γi−t​δi∘\\hat\{\\delta\}^\{\\circ\}\_\{t\}=\\sum\_\{i=t\}^\{T\-1\}\\gamma^\{i\-t\}\\delta^\{\\circ\}\_\{i\}, and rearranging the double sum by the index ofAiA\_\{i\}andξi\\xi\_\{i\}gives

Tg^∘\(W;O\)=∑t=0T−1At∇t\+∑i=1T−1Ai​Zi\+∑i=1Tξi​Zi′⏟=:S,Zi:=∑t<iγi−t∇t,Zi′:=∑t<iγi−1−t∇t\.T\\,\\hat\{g\}^\{\\circ\}\(W;O\)=\\sum\_\{t=0\}^\{T\-1\}A\_\{t\}\\nabla\_\{t\}\+\\underbrace\{\\sum\_\{i=1\}^\{T\-1\}A\_\{i\}Z\_\{i\}\+\\sum\_\{i=1\}^\{T\}\\xi\_\{i\}Z^\{\\prime\}\_\{i\}\}\_\{=:\\,S\},\\qquad Z\_\{i\}:=\\sum\_\{t<i\}\\gamma^\{i\-t\}\\nabla\_\{t\},\\kern 5\.0ptZ^\{\\prime\}\_\{i\}:=\\sum\_\{t<i\}\\gamma^\{i\-1\-t\}\\nabla\_\{t\}\.
*Average part\.*The functionf\(s,a\):=\(Q\(s,a\)−V\(s\)\)∇logπW\(a\|s\)f\(s,a\):=\(Q\(s,a\)\-V\(s\)\)\\nabla\\log\\pi\_\{W\}\(a\|s\)satisfies‖f‖F≤‖f‖∗≤2​Rr​Rπ/\(1−γ\)\\\|f\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq\\\|f\\\|\_\{\*\}\\leq\\nicefrac\{\{2R\_\{r\}R\_\{\\pi\}\}\}\{\{\(1\-\\gamma\)\}\}, since\|Q\|,\|V\|≤Rr/\(1−γ\)\|Q\|,\|V\|\\leq\\nicefrac\{\{R\_\{r\}\}\}\{\{\(1\-\\gamma\)\}\}, and has the meang⁡\(W\)g\(W\)underμW⊗πW\\mu\_\{W\}\\otimes\\pi\_\{W\}by \([20](https://arxiv.org/html/2610.33477#S4.E20)\); hence, Lemma[20](https://arxiv.org/html/2610.33477#Thmtheorem20)gives𝔼⁡\[‖1T​∑t<TAt​∇t−g⁡\(W\)‖F2\]≤16​Rr2​Rπ2​\(1\+4​τmix\)/\[\(1−γ\)2​T\]\\mathbb\{E\}\[\\\|\\frac\{1\}\{T\}\\sum\_\{t<T\}A\_\{t\}\\nabla\_\{t\}\-g\(W\)\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\]\\leq\\nicefrac\{\{16R\_\{r\}^\{2\}R\_\{\\pi\}^\{2\}\(1\+4\\tau\_\{\\mathrm\{mix\}\}\)\}\}\{\{\[\(1\-\\gamma\)^\{2\}T\]\}\}\.

*Martingale part\.*Let theσ\\sigma\-fields𝒢2​i:=σ⁡\(s0,a0,…,ai−1,si\)\{\\cal G\}\_\{2i\}:=\\sigma\(s\_\{0\},a\_\{0\},\\ldots,a\_\{i\-1\},s\_\{i\}\)and𝒢2​i\+1:=σ⁡\(𝒢2​i,ai\)\{\\cal G\}\_\{2i\+1\}:=\\sigma\(\{\\cal G\}\_\{2i\},a\_\{i\}\)reveal the states and the actions one at a time\. Then the matrixξi​Zi′\\xi\_\{i\}Z^\{\\prime\}\_\{i\}is𝒢2​i\{\\cal G\}\_\{2i\}\-measurable with𝔼\[ξiZi′\|𝒢2​i−1\]=Zi′𝔼\[ξi\|si−1,ai−1\]=0\\mathbb\{E\}\[\\xi\_\{i\}Z^\{\\prime\}\_\{i\}\\,\|\\,\{\\cal G\}\_\{2i\-1\}\]=Z^\{\\prime\}\_\{i\}\\,\\mathbb\{E\}\[\\xi\_\{i\}\\,\|\\,s\_\{i\-1\},a\_\{i\-1\}\]=0by the Markov property, sinceZi′Z^\{\\prime\}\_\{i\}is𝒢2​i−1\{\\cal G\}\_\{2i\-1\}\-measurable, andAi​ZiA\_\{i\}Z\_\{i\}is𝒢2​i\+1\{\\cal G\}\_\{2i\+1\}\-measurable with𝔼\[AiZi\|𝒢2​i\]=Zi𝔼a∼πW\(⋅\|si\)\[Q\(si,a\)−V\(si\)\]=0\\mathbb\{E\}\[A\_\{i\}Z\_\{i\}\\,\|\\,\{\\cal G\}\_\{2i\}\]=Z\_\{i\}\\,\\mathbb\{E\}\_\{a\\sim\\pi\_\{W\}\(\\cdot\|s\_\{i\}\)\}\[Q\(s\_\{i\},a\)\-V\(s\_\{i\}\)\]=0, sinceZiZ\_\{i\}is𝒢2​i\{\\cal G\}\_\{2i\}\-measurable\. Hence,SSis a sum of martingale differences, which are orthogonal in the Frobenius inner product, and

𝔼⁡\[‖S‖F2\]=∑i=1T−1𝔼⁡\[‖Zi‖F2​𝔼​\[Ai2\|𝒢2​i\]\]\+∑i=1T𝔼⁡\[‖Zi′‖F2​𝔼​\[ξi2\|𝒢2​i−1\]\]≤2​T​γ2​Rr2​Rπ2\(1−γ\)4\\mathbb\{E\}\[\\\|S\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\]=\\sum\_\{i=1\}^\{T\-1\}\\mathbb\{E\}\[\\\|Z\_\{i\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\\,\\mathbb\{E\}\[A\_\{i\}^\{2\}\\,\|\\,\{\\cal G\}\_\{2i\}\]\]\+\\sum\_\{i=1\}^\{T\}\\mathbb\{E\}\[\\\|Z^\{\\prime\}\_\{i\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\\,\\mathbb\{E\}\[\\xi\_\{i\}^\{2\}\\,\|\\,\{\\cal G\}\_\{2i\-1\}\]\]\\leq\\frac\{2T\\gamma^\{2\}R\_\{r\}^\{2\}R\_\{\\pi\}^\{2\}\}\{\(1\-\\gamma\)^\{4\}\}since𝔼⁡\[Ai2\|𝒢2​i\]\\mathbb\{E\}\[A\_\{i\}^\{2\}\\,\|\\,\{\\cal G\}\_\{2i\}\]is the variance ofQ⁡\(si,a\)Q\(s\_\{i\},a\)overa∼πW\(⋅\|si\)a\\sim\\pi\_\{W\}\(\\cdot\|s\_\{i\}\), which is at mostRr2/\(1−γ\)2\\nicefrac\{\{R\_\{r\}^\{2\}\}\}\{\{\(1\-\\gamma\)^\{2\}\}\},𝔼⁡\[ξi2\|𝒢2​i−1\]≤γ2​Rr2/\(1−γ\)2\\mathbb\{E\}\[\\xi\_\{i\}^\{2\}\\,\|\\,\{\\cal G\}\_\{2i\-1\}\]\\leq\\nicefrac\{\{\\gamma^\{2\}R\_\{r\}^\{2\}\}\}\{\{\(1\-\\gamma\)^\{2\}\}\}likewise,‖Zi‖F≤γ​Rπ/\(1−γ\)\\\|Z\_\{i\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq\\nicefrac\{\{\\gamma R\_\{\\pi\}\}\}\{\{\(1\-\\gamma\)\}\}, and‖Zi′‖F≤Rπ/\(1−γ\)\\\|Z^\{\\prime\}\_\{i\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq\\nicefrac\{\{R\_\{\\pi\}\}\}\{\{\(1\-\\gamma\)\}\}\. Minkowski’s inequality combines the two parts into

𝔼⁡\[‖g^∘​\(W,O\)−g⁡\(W\)‖F2\]≤4​Rr​Rπ​1\+4​τmix\(1−γ\)​T\+2​γ​Rr​Rπ\(1−γ\)2​T=V∘T\.\\sqrt\{\\mathbb\{E\}\[\\\|\\hat\{g\}^\{\\circ\}\(W;O\)\-g\(W\)\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\]\}\\leq\\frac\{4R\_\{r\}R\_\{\\pi\}\\sqrt\{1\+4\\tau\_\{\\mathrm\{mix\}\}\}\}\{\(1\-\\gamma\)\\sqrt\{T\}\}\+\\frac\{\\sqrt\{2\}\\,\\gamma R\_\{r\}R\_\{\\pi\}\}\{\(1\-\\gamma\)^\{2\}\\sqrt\{T\}\}=\\frac\{V^\{\\circ\}\}\{\\sqrt\{T\}\}\.
We complete the proof\.

*Oracle noise\.*Conditionally onℱk−1\{\\cal F\}\_\{k\-1\},OkO\_\{k\}is aTT\-step trajectory of the chain induced byπWk\\pi\_\{W\_\{k\}\}fromsk,0s\_\{k,0\}, such that Lemma[21](https://arxiv.org/html/2610.33477#Thmtheorem21)gives𝔼⁡\[‖Gk∘−g⁡\(Wk\)‖F2\|ℱk−1\]≤\(V∘\)2/T\\mathbb\{E\}\[\\\|G\_\{k\}^\{\\circ\}\-g\(W\_\{k\}\)\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\\,\|\\,\{\\cal F\}\_\{k\-1\}\]\\leq\\nicefrac\{\{\(V^\{\\circ\}\)^\{2\}\}\}\{\{T\}\}; sinceg⁡\(Wk\)g\(W\_\{k\}\)isℱk−1\{\\cal F\}\_\{k\-1\}\-measurable and the conditional mean minimizes the conditional mean\-square distance, this proves the second claim of part \(iii\)\.

*Critic noise\.*Let∇k,t:=∇log⁡πk​\(ak,t\|sk,t\)\\nabla\_\{k,t\}:=\\nabla\\log\\pi\_\{k\}\(a\_\{k,t\}\|s\_\{k,t\}\)\. By the telescoped form \([9](https://arxiv.org/html/2610.33477#S3.E9)\) withv=vk−1v=v\_\{k\-1\}andW=WkW=W\_\{k\},Gk−Gk∘=Δkb−ΔksG\_\{k\}\-G\_\{k\}^\{\\circ\}=\\Delta^\{\\rm b\}\_\{k\}\-\\Delta^\{\\rm s\}\_\{k\}with

Δkb:=1T​∑t=0T−1γT−t​ϵk​\(sk,T\)​∇k,t,Δks:=1T​∑t=0T−1ϵk​\(sk,t\)​∇k,t\.\\Delta\_\{k\}^\{\\rm b\}:=\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\gamma^\{T\-t\}\\epsilon\_\{k\}\(s\_\{k,T\}\)\\nabla\_\{k,t\},\\qquad\\Delta\_\{k\}^\{\\rm s\}:=\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\epsilon\_\{k\}\(s\_\{k,t\}\)\\nabla\_\{k,t\}\.For the baseline part, letℱk,t\{\\cal F\}\_\{k,t\}extendℱk−1\{\\cal F\}\_\{k\-1\}bysk,0,ak,0,…,ak,t−1,sk,ts\_\{k,0\},a\_\{k,0\},\\ldots,a\_\{k,t\-1\},s\_\{k,t\}andΞt:=ϵk​\(sk,t\)​∇k,t\\Xi\_\{t\}:=\\epsilon\_\{k\}\(s\_\{k,t\}\)\\nabla\_\{k,t\}; sinceϵk\\epsilon\_\{k\}isℱk−1\{\\cal F\}\_\{k\-1\}\-measurable andak,t∼πk\(⋅\|sk,t\)a\_\{k,t\}\\sim\\pi\_\{k\}\(\\cdot\|s\_\{k,t\}\)givenℱk,t\{\\cal F\}\_\{k,t\},𝔼⁡\[Ξt\|ℱk,t\]=0\\mathbb\{E\}\[\\Xi\_\{t\}\\,\|\\,\{\\cal F\}\_\{k,t\}\]=0, and, fort<t′t<t^\{\\prime\},Ξt\\Xi\_\{t\}isℱk,t′\{\\cal F\}\_\{k,t^\{\\prime\}\}\-measurable, such that𝔼⁡\[⟨Ξt,Ξt′⟩F\|ℱk−1\]=0\\mathbb\{E\}\[\\langle\\Xi\_\{t\},\\Xi\_\{t^\{\\prime\}\}\\rangle\_\{\\scriptscriptstyle\\textrm\{F\}\}\\,\|\\,\{\\cal F\}\_\{k\-1\}\]=0\. Hence,𝔼⁡\[Δks\|ℱk−1\]=0\\mathbb\{E\}\[\\Delta^\{\\rm s\}\_\{k\}\\,\|\\,\{\\cal F\}\_\{k\-1\}\]=0and𝔼⁡\[‖Δks‖F2\|ℱk−1\]=1T2​∑t<T𝔼⁡\[‖Ξt‖F2\|ℱk−1\]≤Rπ2​‖ϵk‖∞2/T\\mathbb\{E\}\[\\\|\\Delta\_\{k\}^\{\\rm s\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\\,\|\\,\{\\cal F\}\_\{k\-1\}\]=\\frac\{1\}\{T^\{2\}\}\\sum\_\{t<T\}\\mathbb\{E\}\[\\\|\\Xi\_\{t\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\\,\|\\,\{\\cal F\}\_\{k\-1\}\]\\leq\\nicefrac\{\{R\_\{\\pi\}^\{2\}\\\|\\epsilon\_\{k\}\\\|\_\{\\infty\}^\{2\}\}\}\{\{T\}\}\. For the bootstrap part, \([15a](https://arxiv.org/html/2610.33477#S4.E15.1)\) and∑t<TγT−t≤γ/\(1−γ\)\\sum\_\{t<T\}\\gamma^\{T\-t\}\\leq\\nicefrac\{\{\\gamma\}\}\{\{\(1\-\\gamma\)\}\}give‖Δkb‖F≤‖Δkb‖∗≤γ​Rπ​‖ϵk‖∞/\[\(1−γ\)​T\]\\\|\\Delta\_\{k\}^\{\\rm b\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq\\\|\\Delta\_\{k\}^\{\\rm b\}\\\|\_\{\*\}\\leq\\nicefrac\{\{\\gamma R\_\{\\pi\}\\\|\\epsilon\_\{k\}\\\|\_\{\\infty\}\}\}\{\{\[\(1\-\\gamma\)T\]\}\}for every realization, and, since centering does not increase the conditional second moment,𝔼⁡\[‖Δkb−𝔼⁡\[Δkb\|ℱk−1\]‖F2\|ℱk−1\]≤γ2​Rπ2​‖ϵk‖∞2/\[\(1−γ\)2​T2\]\\mathbb\{E\}\[\\\|\\Delta^\{\\rm b\}\_\{k\}\-\\mathbb\{E\}\[\\Delta^\{\\rm b\}\_\{k\}\\,\|\\,\{\\cal F\}\_\{k\-1\}\]\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\\,\|\\,\{\\cal F\}\_\{k\-1\}\]\\leq\\nicefrac\{\{\\gamma^\{2\}R\_\{\\pi\}^\{2\}\\\|\\epsilon\_\{k\}\\\|\_\{\\infty\}^\{2\}\}\}\{\{\[\(1\-\\gamma\)^\{2\}T^\{2\}\]\}\}\.

*Conclusion of part \(iii\)\.*The noise splits asGk−𝔼⁡\[Gk\|ℱk−1\]=\(Gk∘−𝔼⁡\[Gk∘\|ℱk−1\]\)\+\(Δkb−𝔼⁡\[Δkb\|ℱk−1\]\)−ΔksG\_\{k\}\-\\mathbb\{E\}\[G\_\{k\}\\,\|\\,\{\\cal F\}\_\{k\-1\}\]=\(G\_\{k\}^\{\\circ\}\-\\mathbb\{E\}\[G\_\{k\}^\{\\circ\}\\,\|\\,\{\\cal F\}\_\{k\-1\}\]\)\+\(\\Delta^\{\\rm b\}\_\{k\}\-\\mathbb\{E\}\[\\Delta^\{\\rm b\}\_\{k\}\\,\|\\,\{\\cal F\}\_\{k\-1\}\]\)\-\\Delta^\{\\rm s\}\_\{k\}, such that Minkowski’s inequality, the three bounds above, and𝔼⁡\[‖ϵk‖∞2\]≤ϵcrit\\sqrt\{\\mathbb\{E\}\[\\\|\\epsilon\_\{k\}\\\|\_\{\\infty\}^\{2\}\]\}\\leq\\epsilon\_\{\\rm crit\}yield

𝔼⁡\[‖Gk−𝔼⁡\[Gk\|ℱk−1\]‖F2\]≤V∘\+Rπ​ϵcritT\+γ​Rπ​ϵcrit\(1−γ\)​T=σ¯\.\\sqrt\{\\mathbb\{E\}\[\\\|G\_\{k\}\-\\mathbb\{E\}\[G\_\{k\}\\,\|\\,\{\\cal F\}\_\{k\-1\}\]\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\]\}\\leq\\frac\{V^\{\\circ\}\+R\_\{\\pi\}\\epsilon\_\{\\rm crit\}\}\{\\sqrt\{T\}\}\+\\frac\{\\gamma R\_\{\\pi\}\\epsilon\_\{\\rm crit\}\}\{\(1\-\\gamma\)T\}=\\bar\{\\sigma\}\.
*Part \(iv\)\.*Letvk−1∗v^\{\*\}\_\{k\-1\}denote the population solution of Theorem[4](https://arxiv.org/html/2610.33477#Thmtheorem4)for the policy under whichOk−1O\_\{k\-1\}is collected, i\.e\.,vk−1∗=v∗​\(Wk−1\)v^\{\*\}\_\{k\-1\}=v^\{\*\}\(W\_\{k\-1\}\)fork≥2k\\geq 2andv0∗=v∗​\(W1\)v^\{\*\}\_\{0\}=v^\{\*\}\(W\_\{1\}\)\. Splittingϵk​\(s\)=ϕ⊤​\(s\)​\(vk−1−vk−1∗\)\+ϕ⊤​\(s\)​\(vk−1∗−v∗​\(Wk\)\)\+\(ϕ⊤​\(s\)​v∗​\(Wk\)−VWk​\(s\)\)\\epsilon\_\{k\}\(s\)=\\phi^\{\\top\}\(s\)\(v\_\{k\-1\}\-v^\{\*\}\_\{k\-1\}\)\+\\phi^\{\\top\}\(s\)\(v^\{\*\}\_\{k\-1\}\-v^\{\*\}\(W\_\{k\}\)\)\+\(\\phi^\{\\top\}\(s\)v^\{\*\}\(W\_\{k\}\)\-V\_\{W\_\{k\}\}\(s\)\)and using‖ϕ⁡\(s\)‖2≤1\\\|\\phi\(s\)\\\|\_\{2\}\\leq 1, Lemma[3](https://arxiv.org/html/2610.33477#Thmtheorem3),‖Wk−1−Wk‖2≤ℓ¯2\\\|W\_\{k\-1\}\-W\_\{k\}\\\|\_\{2\}\\leq\\bar\{\\ell\}\_\{2\}\(the middle term vanishes fork=1k=1\), and the definition ofϵapp\\epsilon\_\{\\rm app\}, we obtain‖ϵk‖∞≤‖vk−1−vk−1∗‖2\+L∗​ℓ¯2\+ϵapp\\\|\\epsilon\_\{k\}\\\|\_\{\\infty\}\\leq\\\|v\_\{k\-1\}\-v^\{\*\}\_\{k\-1\}\\\|\_\{2\}\+L\_\{\*\}\\bar\{\\ell\}\_\{2\}\+\\epsilon\_\{\\rm app\}, such that Minkowski’s inequality gives𝔼⁡\[‖ϵk‖∞2\]≤𝔼⁡\[‖vk−1−vk−1∗‖22\]\+L∗​ℓ¯2\+ϵapp\\sqrt\{\\mathbb\{E\}\[\\\|\\epsilon\_\{k\}\\\|\_\{\\infty\}^\{2\}\]\}\\leq\\sqrt\{\\mathbb\{E\}\[\\\|v\_\{k\-1\}\-v^\{\*\}\_\{k\-1\}\\\|\_\{2\}^\{2\}\]\}\+L\_\{\*\}\\bar\{\\ell\}\_\{2\}\+\\epsilon\_\{\\rm app\}\. Theorem[4](https://arxiv.org/html/2610.33477#Thmtheorem4), applied to framek−1k\-1, which starts from theℱk−2\{\\cal F\}\_\{k\-2\}\-measurable statesk−1,0s\_\{k\-1,0\}\(fork=1k=1, to the warm\-up frameO0O\_\{0\}collected underπ1\\pi\_\{1\}froms0,0∼ρs\_\{0,0\}\\sim\\rho\), bounds the first term byϵlstd\\epsilon\_\{\\rm lstd\}conditionally onℱk−2\{\\cal F\}\_\{k\-2\}and hence, by the tower property, inL2L^\{2\}\.

## Appendix JProof of Lemma[8](https://arxiv.org/html/2610.33477#Thmtheorem8)and Bounded Momenta

Proof of Lemma[8](https://arxiv.org/html/2610.33477#Thmtheorem8)\.LetekM:=Mk−∇J​\(Wk\)e^\{M\}\_\{k\}:=M\_\{k\}\-\\nabla J\(W\_\{k\}\),ξk:=Gk−∇J​\(Wk\)\\xi\_\{k\}:=G\_\{k\}\-\\nabla J\(W\_\{k\}\), and, fork≥2k\\geq 2,ςk:=Gk−G~k−\[∇J​\(Wk\)−∇J​\(Wk−1\)\]\\varsigma\_\{k\}:=G\_\{k\}\-\\widetilde\{G\}\_\{k\}\-\[\\nabla J\(W\_\{k\}\)\-\\nabla J\(W\_\{k\-1\}\)\]\. SinceWkW\_\{k\}isℱk−1\{\\cal F\}\_\{k\-1\}\-measurable, Lemmas[6](https://arxiv.org/html/2610.33477#Thmtheorem6)\(ii\) and[7](https://arxiv.org/html/2610.33477#Thmtheorem7)\(i\),\(iii\) hold along the iterates, withϵcrit\\epsilon\_\{\\rm crit\},ϵbias\\epsilon\_\{\\rm bias\}, andσ¯\\bar\{\\sigma\}evaluated along them\. Subtracting∇J\(Wk\)=\(1−η\)∇J\(Wk\)\+η∇J\(Wk−1\)\+η\[∇J\(Wk\)−∇J\(Wk−1\)\]\\nabla J\(W\_\{k\}\)=\(1\-\\eta\)\\nabla J\(W\_\{k\}\)\+\\eta\\nabla J\(W\_\{k\-1\}\)\+\\eta\[\\nabla J\(W\_\{k\}\)\-\\nabla J\(W\_\{k\-1\}\)\]from \([12](https://arxiv.org/html/2610.33477#S3.E12)\) givese1M=ξ1e^\{M\}\_\{1\}=\\xi\_\{1\}and

ekM=η​ek−1M\+\(1−η\)​ξk\+η​ςk,k≥2\.e^\{M\}\_\{k\}=\\eta\\,e^\{M\}\_\{k\-1\}\+\(1\-\\eta\)\\,\\xi\_\{k\}\+\\eta\\,\\varsigma\_\{k\},\\quad k\\geq 2\.\(46\)By \([20](https://arxiv.org/html/2610.33477#S4.E20)\), Lemma[6](https://arxiv.org/html/2610.33477#Thmtheorem6)\(ii\),𝔼⁡\[Gj\|ℱj−1\]=h⁡\(Wj,ωj\)\\mathbb\{E\}\[G\_\{j\}\\,\|\\,\{\\cal F\}\_\{j\-1\}\]=h\(W\_\{j\};\\omega\_\{j\}\)\(Section[4\.2](https://arxiv.org/html/2610.33477#S4.SS2)\), and theℱj−1\{\\cal F\}\_\{j\-1\}\-measurability ofWjW\_\{j\},Wj−1W\_\{j\-1\}, andωj\\omega\_\{j\}, the conditional means areBj:=𝔼⁡\[ξj\|ℱj−1\]=B⁡\(Wj,ωj\)B\_\{j\}:=\\mathbb\{E\}\[\\xi\_\{j\}\\,\|\\,\{\\cal F\}\_\{j\-1\}\]=B\(W\_\{j\};\\omega\_\{j\}\)forj≥1j\\geq 1andς¯j:=𝔼⁡\[ςj\|ℱj−1\]=B⁡\(Wj,ωj\)−B⁡\(Wj−1,ωj\)\\bar\{\\varsigma\}\_\{j\}:=\\mathbb\{E\}\[\\varsigma\_\{j\}\\,\|\\,\{\\cal F\}\_\{j\-1\}\]=B\(W\_\{j\};\\omega\_\{j\}\)\-B\(W\_\{j\-1\};\\omega\_\{j\}\)forj≥2j\\geq 2\. Withu1:=ξ1−B1u\_\{1\}:=\\xi\_\{1\}\-B\_\{1\}anduj:=\(1−η\)​\(ξj−Bj\)\+η⁡\(ςj−ς¯j\)u\_\{j\}:=\(1\-\\eta\)\(\\xi\_\{j\}\-B\_\{j\}\)\+\\eta\(\\varsigma\_\{j\}\-\\bar\{\\varsigma\}\_\{j\}\)forj≥2j\\geq 2, unrolling \([46](https://arxiv.org/html/2610.33477#A10.E46)\) splits the tracking error asekM=ekn\+ekbe^\{M\}\_\{k\}=e\_\{k\}^\{\\rm n\}\+e\_\{k\}^\{\\rm b\}with

ekn:=ηk−1​u1\+∑j=2kηk−j​uj,ekb:=ηk−1​B1\+∑j=2kηk−j​\[\(1−η\)​Bj\+η​ς¯j\]\.e\_\{k\}^\{\\rm n\}:=\\eta^\{k\-1\}u\_\{1\}\+\\sum\_\{j=2\}^\{k\}\\eta^\{k\-j\}u\_\{j\},\\qquad e\_\{k\}^\{\\rm b\}:=\\eta^\{k\-1\}B\_\{1\}\+\\sum\_\{j=2\}^\{k\}\\eta^\{k\-j\}\\big\[\(1\-\\eta\)B\_\{j\}\+\\eta\\bar\{\\varsigma\}\_\{j\}\\big\]\.
*Noise part\.*Eachuju\_\{j\}isℱj\{\\cal F\}\_\{j\}\-measurable with𝔼⁡\[uj\|ℱj−1\]=0\\mathbb\{E\}\[u\_\{j\}\\,\|\\,\{\\cal F\}\_\{j\-1\}\]=0, such that𝔼⁡\[⟨ui,uj⟩F\]=0\\mathbb\{E\}\[\\langle u\_\{i\},u\_\{j\}\\rangle\_\{\\scriptscriptstyle\\textrm\{F\}\}\]=0fori<ji<jby the tower property\. Sinceξj−Bj=Gj−𝔼⁡\[Gj\|ℱj−1\]\\xi\_\{j\}\-B\_\{j\}=G\_\{j\}\-\\mathbb\{E\}\[G\_\{j\}\\,\|\\,\{\\cal F\}\_\{j\-1\}\], Lemma[7](https://arxiv.org/html/2610.33477#Thmtheorem7)\(iii\) gives𝔼⁡\[‖ξj−Bj‖F2\]≤σ¯2\\mathbb\{E\}\[\\\|\\xi\_\{j\}\-B\_\{j\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\]\\leq\\bar\{\\sigma\}^\{2\}\. Since∇J​\(Wj\)−∇J​\(Wj−1\)\\nabla J\(W\_\{j\}\)\-\\nabla J\(W\_\{j\-1\}\)isℱj−1\{\\cal F\}\_\{j\-1\}\-measurable,ςj−ς¯j=\(Gj−G~j\)−𝔼⁡\[Gj−G~j\|ℱj−1\]\\varsigma\_\{j\}\-\\bar\{\\varsigma\}\_\{j\}=\(G\_\{j\}\-\\widetilde\{G\}\_\{j\}\)\-\\mathbb\{E\}\[G\_\{j\}\-\\widetilde\{G\}\_\{j\}\\,\|\\,\{\\cal F\}\_\{j\-1\}\], such that, forη\>0\\eta\>0, centering and the last claim of Lemma[6](https://arxiv.org/html/2610.33477#Thmtheorem6)give𝔼⁡\[‖ςj−ς¯j‖F2\]≤𝔼⁡\[‖Gj−G~j‖F2\]≤CF2​ℓ¯22\\mathbb\{E\}\[\\\|\\varsigma\_\{j\}\-\\bar\{\\varsigma\}\_\{j\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\]\\leq\\mathbb\{E\}\[\\\|G\_\{j\}\-\\widetilde\{G\}\_\{j\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\]\\leq C\_\{F\}^\{2\}\\bar\{\\ell\}\_\{2\}^\{2\}; forη=0\\eta=0, this term does not enteruju\_\{j\}\. Together with‖X\+Y‖F2≤2​‖X‖F2\+2​‖Y‖F2\\\|X\+Y\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\\leq 2\\\|X\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\+2\\\|Y\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}, this yields𝔼⁡\[‖u1‖F2\]≤σ¯2\\mathbb\{E\}\[\\\|u\_\{1\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\]\\leq\\bar\{\\sigma\}^\{2\}and𝔼⁡\[‖uj‖F2\]≤2​\(1−η\)2​σ¯2\+2​η2​CF2​ℓ¯22\\mathbb\{E\}\[\\\|u\_\{j\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\]\\leq 2\(1\-\\eta\)^\{2\}\\bar\{\\sigma\}^\{2\}\+2\\eta^\{2\}C\_\{F\}^\{2\}\\bar\{\\ell\}\_\{2\}^\{2\}forj≥2j\\geq 2, such that∑j=2kη2​\(k−j\)≤1/\(1−η2\)\\sum\_\{j=2\}^\{k\}\\eta^\{2\(k\-j\)\}\\leq\\nicefrac\{\{1\}\}\{\{\(1\-\\eta^\{2\}\)\}\}and1−η2≥1−η1\-\\eta^\{2\}\\geq 1\-\\etagive

𝔼⁡\[‖ekn‖F2\]≤η2​\(k−1\)​σ¯2\+2​\(1−η\)2​σ¯2\+2​η2​CF2​ℓ¯221−η2≤η2​\(k−1\)​σ¯2\+2​\(1−η\)​σ¯2\+2​η2​CF2​ℓ¯221−η\\mathbb\{E\}\[\\\|e\_\{k\}^\{\\rm n\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\]\\leq\\eta^\{2\(k\-1\)\}\\bar\{\\sigma\}^\{2\}\+\\frac\{2\(1\-\\eta\)^\{2\}\\bar\{\\sigma\}^\{2\}\+2\\eta^\{2\}C\_\{F\}^\{2\}\\bar\{\\ell\}\_\{2\}^\{2\}\}\{1\-\\eta^\{2\}\}\\leq\\eta^\{2\(k\-1\)\}\\bar\{\\sigma\}^\{2\}\+2\(1\-\\eta\)\\,\\bar\{\\sigma\}^\{2\}\+\\frac\{2\\eta^\{2\}C\_\{F\}^\{2\}\\bar\{\\ell\}\_\{2\}^\{2\}\}\{1\-\\eta\}\(47\)and Jensen’s inequality andx\+y\+z≤x\+y\+z\\sqrt\{x\+y\+z\}\\leq\\sqrt\{x\}\+\\sqrt\{y\}\+\\sqrt\{z\}yield𝔼⁡\[‖ekn‖F\]≤\(ηk−1\+2​\(1−η\)\)​σ¯\+2​η​CF​ℓ¯2/1−η\\mathbb\{E\}\[\\\|e\_\{k\}^\{\\rm n\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\]\\leq\(\\eta^\{k\-1\}\+\\sqrt\{2\(1\-\\eta\)\}\)\\,\\bar\{\\sigma\}\+\\nicefrac\{\{\\sqrt\{2\}\\,\\eta\\,C\_\{F\}\\bar\{\\ell\}\_\{2\}\}\}\{\{\\sqrt\{1\-\\eta\}\}\}, while‖ekn‖†≤κ†​‖ekn‖F\\\|e\_\{k\}^\{\\rm n\}\\\|\_\{\\dagger\}\\leq\\kappa\_\{\\dagger\}\\\|e\_\{k\}^\{\\rm n\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}by the definition ofκ†\\kappa\_\{\\dagger\}\.

*Bias part\.*By the expressions ofBjB\_\{j\}andς¯j\\bar\{\\varsigma\}\_\{j\},\(1−η\)​Bj\+η​ς¯j=B⁡\(Wj,ωj\)−η​B​\(Wj−1,ωj\)\(1\-\\eta\)B\_\{j\}\+\\eta\\bar\{\\varsigma\}\_\{j\}=B\(W\_\{j\};\\omega\_\{j\}\)\-\\eta B\(W\_\{j\-1\};\\omega\_\{j\}\), and, with the splitB=B¯\+DB=\\bar\{B\}\+Dof \([20](https://arxiv.org/html/2610.33477#S4.E20)\), the context\-free part telescopes as

ηk−1​B¯​\(W1\)\+∑j=2kηk−j​\[B¯​\(Wj\)−η​B¯​\(Wj−1\)\]=∑j=1kηk−j​B¯​\(Wj\)−∑j=1k−1ηk−j​B¯​\(Wj\)=B¯​\(Wk\)\\begin\{split\}\\eta^\{k\-1\}\\bar\{B\}\(W\_\{1\}\)\+\\sum\_\{j=2\}^\{k\}\\eta^\{k\-j\}\\big\[\\bar\{B\}\(W\_\{j\}\)\-\\eta\\bar\{B\}\(W\_\{j\-1\}\)\\big\]&=\\sum\_\{j=1\}^\{k\}\\eta^\{k\-j\}\\bar\{B\}\(W\_\{j\}\)\-\\sum\_\{j=1\}^\{k\-1\}\\eta^\{k\-j\}\\bar\{B\}\(W\_\{j\}\)\\\\ &=\\bar\{B\}\(W\_\{k\}\)\\end\{split\}\(48\)such that

ekb=B¯​\(Wk\)\+ηk−1​D​\(W1,ω1\)\+∑j=2kηk−j​\[\(1−η\)​D​\(Wj,ωj\)\+η⁡\(D⁡\(Wj,ωj\)−D⁡\(Wj−1,ωj\)\)\]e\_\{k\}^\{\\rm b\}=\\bar\{B\}\(W\_\{k\}\)\+\\eta^\{k\-1\}D\(W\_\{1\};\\omega\_\{1\}\)\+\\sum\_\{j=2\}^\{k\}\\eta^\{k\-j\}\\Big\[\(1\-\\eta\)D\(W\_\{j\};\\omega\_\{j\}\)\+\\eta\\big\(D\(W\_\{j\};\\omega\_\{j\}\)\-D\(W\_\{j\-1\};\\omega\_\{j\}\)\\big\)\\Big\]where each differenceD⁡\(Wj,ωj\)−D⁡\(Wj−1,ωj\)D\(W\_\{j\};\\omega\_\{j\}\)\-D\(W\_\{j\-1\};\\omega\_\{j\}\)holds the contextωj\\omega\_\{j\}fixed, such that no bound on the change of the context between frames is needed\. First,‖B¯​\(Wk\)‖∗≤ϵdist\\\|\\bar\{B\}\(W\_\{k\}\)\\\|\_\{\*\}\\leq\\epsilon\_\{\\rm dist\}\. Second, Lemma[7](https://arxiv.org/html/2610.33477#Thmtheorem7)\(i\) and Jensen’s inequality with𝔼⁡\[‖ϵj‖∞\]≤ϵcrit\\mathbb\{E\}\[\\\|\\epsilon\_\{j\}\\\|\_\{\\infty\}\]\\leq\\epsilon\_\{\\rm crit\}give𝔼⁡\[‖D⁡\(Wj,ωj\)‖∗\]≤ϵbias−ϵdist\\mathbb\{E\}\[\\\|D\(W\_\{j\};\\omega\_\{j\}\)\\\|\_\{\*\}\]\\leq\\epsilon\_\{\\rm bias\}\-\\epsilon\_\{\\rm dist\}for alljj, andηk−1\+\(1−η\)​∑j=2kηk−j=1\\eta^\{k\-1\}\+\(1\-\\eta\)\\sum\_\{j=2\}^\{k\}\\eta^\{k\-j\}=1\. Third, Lemma[7](https://arxiv.org/html/2610.33477#Thmtheorem7)\(ii\) gives‖D⁡\(Wj,ωj\)−D⁡\(Wj−1,ωj\)‖∗≤DF​ℓ¯2\\\|D\(W\_\{j\};\\omega\_\{j\}\)\-D\(W\_\{j\-1\};\\omega\_\{j\}\)\\\|\_\{\*\}\\leq D\_\{F\}\\bar\{\\ell\}\_\{2\}forj≥2j\\geq 2, andη​∑j=2kηk−j≤η/\(1−η\)\\eta\\sum\_\{j=2\}^\{k\}\\eta^\{k\-j\}\\leq\\nicefrac\{\{\\eta\}\}\{\{\(1\-\\eta\)\}\}\. Hence,𝔼⁡\[‖ekb‖∗\]≤ϵbias\+η​DF​ℓ¯2/\(1−η\)\\mathbb\{E\}\[\\\|e\_\{k\}^\{\\rm b\}\\\|\_\{\*\}\]\\leq\\epsilon\_\{\\rm bias\}\+\\nicefrac\{\{\\eta D\_\{F\}\\bar\{\\ell\}\_\{2\}\}\}\{\{\(1\-\\eta\)\}\}, and, since∥⋅∥†≤∥⋅∥∗\\\|\\cdot\\\|\_\{\\dagger\}\\leq\\\|\\cdot\\\|\_\{\*\}for∥⋅∥≥∥⋅∥2\\\|\\cdot\\\|\\geq\\\|\\cdot\\\|\_\{2\}\(Appendix[A](https://arxiv.org/html/2610.33477#A1)\),𝔼⁡\[‖ekM‖†\]≤κ†​𝔼​\[‖ekn‖F\]\+𝔼⁡\[‖ekb‖∗\]\\mathbb\{E\}\[\\\|e^\{M\}\_\{k\}\\\|\_\{\\dagger\}\]\\leq\\kappa\_\{\\dagger\}\\mathbb\{E\}\[\\\|e\_\{k\}^\{\\rm n\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\]\+\\mathbb\{E\}\[\\\|e\_\{k\}^\{\\rm b\}\\\|\_\{\*\}\]yields \([23](https://arxiv.org/html/2610.33477#S4.E23)\); the bounds onekne\_\{k\}^\{\\rm n\}andekbe\_\{k\}^\{\\rm b\}above are those stated below Lemma[8](https://arxiv.org/html/2610.33477#Thmtheorem8)\.

*Second moment\.*Lemma[7](https://arxiv.org/html/2610.33477#Thmtheorem7)\(i\) and Minkowski’s inequality with𝔼⁡\[‖ϵj‖∞2\]≤ϵcrit\\sqrt\{\\mathbb\{E\}\[\\\|\\epsilon\_\{j\}\\\|\_\{\\infty\}^\{2\}\]\}\\leq\\epsilon\_\{\\rm crit\}give𝔼⁡\[‖D⁡\(Wj,ωj\)‖∗2\]≤ϵbias−ϵdist\\sqrt\{\\mathbb\{E\}\[\\\|D\(W\_\{j\};\\omega\_\{j\}\)\\\|\_\{\*\}^\{2\}\]\}\\leq\\epsilon\_\{\\rm bias\}\-\\epsilon\_\{\\rm dist\}, such that the same three steps, with Minkowski’s inequality inL2L^\{2\}, give𝔼⁡\[‖ekb‖F2\]≤𝔼⁡\[‖ekb‖∗2\]≤ϵbias\+η​DF​ℓ¯2/\(1−η\)\\sqrt\{\\mathbb\{E\}\[\\\|e\_\{k\}^\{\\rm b\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\]\}\\leq\\sqrt\{\\mathbb\{E\}\[\\\|e\_\{k\}^\{\\rm b\}\\\|\_\{\*\}^\{2\}\]\}\\leq\\epsilon\_\{\\rm bias\}\+\\nicefrac\{\{\\eta D\_\{F\}\\bar\{\\ell\}\_\{2\}\}\}\{\{\(1\-\\eta\)\}\}; together with \([47](https://arxiv.org/html/2610.33477#A10.E47)\) and‖X\+Y‖F2≤2​‖X‖F2\+2​‖Y‖F2\\\|X\+Y\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\\leq 2\\\|X\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\+2\\\|Y\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}, this yields \([24](https://arxiv.org/html/2610.33477#S4.E24)\)\. Forη=0\\eta=0, the importance weights are not used, the condition onℓ¯2\\bar\{\\ell\}\_\{2\}is not needed, andekM=ξke^\{M\}\_\{k\}=\\xi\_\{k\}gives directly, by Lemma[7](https://arxiv.org/html/2610.33477#Thmtheorem7)\(i\),\(iii\),𝔼⁡\[‖ekM‖†\]≤κ†​σ¯\+ϵbias\\mathbb\{E\}\[\\\|e^\{M\}\_\{k\}\\\|\_\{\\dagger\}\]\\leq\\kappa\_\{\\dagger\}\\bar\{\\sigma\}\+\\epsilon\_\{\\rm bias\}and𝔼⁡\[‖ekM‖F2\]≤2​σ¯2\+2​\(ϵbias\)2\\mathbb\{E\}\[\\\|e^\{M\}\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\]\\leq 2\\bar\{\\sigma\}^\{2\}\+2\(\\epsilon\_\{\\rm bias\}\)^\{2\}, which are at most the corresponding bounds\.

*Plain momentum \(Remark[9](https://arxiv.org/html/2610.33477#Thmtheorem9)\)\.*ForMk=η​Mk−1\+\(1−η\)​GkM\_\{k\}=\\eta M\_\{k\-1\}\+\(1\-\\eta\)G\_\{k\}withM1=G1M\_\{1\}=G\_\{1\}, the same subtraction givese1M=ξ1e^\{M\}\_\{1\}=\\xi\_\{1\}andekM=η​ek−1M\+\(1−η\)​ξk−η⁡\[∇J​\(Wk\)−∇J​\(Wk−1\)\]e^\{M\}\_\{k\}=\\eta e^\{M\}\_\{k\-1\}\+\(1\-\\eta\)\\xi\_\{k\}\-\\eta\[\\nabla J\(W\_\{k\}\)\-\\nabla J\(W\_\{k\-1\}\)\]fork≥2k\\geq 2\. By the same orthogonality, the noise partηk−1​\(ξ1−B1\)\+\(1−η\)​∑j=2kηk−j​\(ξj−Bj\)\\eta^\{k\-1\}\(\\xi\_\{1\}\-B\_\{1\}\)\+\(1\-\\eta\)\\sum\_\{j=2\}^\{k\}\\eta^\{k\-j\}\(\\xi\_\{j\}\-B\_\{j\}\)has second moment at mostη2​\(k−1\)​σ¯2\+\(1−η\)2​σ¯2/\(1−η2\)≤η2​\(k−1\)​σ¯2\+\(1−η\)​σ¯2\\eta^\{2\(k\-1\)\}\\bar\{\\sigma\}^\{2\}\+\\nicefrac\{\{\(1\-\\eta\)^\{2\}\\bar\{\\sigma\}^\{2\}\}\}\{\{\(1\-\\eta^\{2\}\)\}\}\\leq\\eta^\{2\(k\-1\)\}\\bar\{\\sigma\}^\{2\}\+\(1\-\\eta\)\\bar\{\\sigma\}^\{2\}in the Frobenius norm and hence expected dual norm at mostκ†​\(ηk−1\+1−η\)​σ¯\\kappa\_\{\\dagger\}\(\\eta^\{k\-1\}\+\\sqrt\{1\-\\eta\}\)\\,\\bar\{\\sigma\}, and the bias partηk−1​B1\+\(1−η\)​∑j=2kηk−j​Bj\\eta^\{k\-1\}B\_\{1\}\+\(1\-\\eta\)\\sum\_\{j=2\}^\{k\}\\eta^\{k\-j\}B\_\{j\}has expected nuclear norm at mostϵbias\\epsilon\_\{\\rm bias\}by Lemma[7](https://arxiv.org/html/2610.33477#Thmtheorem7)\(i\), sinceηk−1\+\(1−η\)​∑j=2kηk−j=1\\eta^\{k\-1\}\+\(1\-\\eta\)\\sum\_\{j=2\}^\{k\}\\eta^\{k\-j\}=1; without the importance correction, the bias does not telescope, and its change between iterates does not enter\. Lemma[2](https://arxiv.org/html/2610.33477#Thmtheorem2)and‖Wj−Wj−1‖2≤β\\\|W\_\{j\}\-W\_\{j\-1\}\\\|\_\{2\}\\leq\\beta, which hold for any normalized step with stepsizeβ\>0\\beta\>0, bound the drift∑j=2kηk−j\+1​\[∇J​\(Wj\)−∇J​\(Wj−1\)\]\\sum\_\{j=2\}^\{k\}\\eta^\{k\-j\+1\}\[\\nabla J\(W\_\{j\}\)\-\\nabla J\(W\_\{j\-1\}\)\]byη​L​β/\(1−η\)\\nicefrac\{\{\\eta L\\beta\}\}\{\{\(1\-\\eta\)\}\}in the nuclear norm and hence in the dual norm, which gives the bound of Remark[9](https://arxiv.org/html/2610.33477#Thmtheorem9)\. The proof of Theorem[11](https://arxiv.org/html/2610.33477#Thmtheorem11), which uses Lemma[8](https://arxiv.org/html/2610.33477#Thmtheorem8)only through \([23](https://arxiv.org/html/2610.33477#S4.E23)\) withℓ¯2=β\\bar\{\\ell\}\_\{2\}=\\beta, applies with this bound, i\.e\., withCF=0C\_\{F\}=0,1−η\\sqrt\{1\-\\eta\}in place of2​\(1−η\)\\sqrt\{2\(1\-\\eta\)\}, andLLin place ofDFD\_\{F\}, which gives the floor term\(1\+χ\)​c​L\(1\+\\chi\)cLof Remark[9](https://arxiv.org/html/2610.33477#Thmtheorem9)\.

The following lemma bounds the momentum of unnormalized actor updates for every realization, which controls their importance weights\.

###### Lemma 22\.

Under Assumptions[1](https://arxiv.org/html/2610.33477#Thmassump1)and[2](https://arxiv.org/html/2610.33477#Thmassump2), consider the ICM \([12](https://arxiv.org/html/2610.33477#S3.E12)\) together with an actor update whose stepWk\+1−WkW\_\{k\+1\}\-W\_\{k\}isℱk\{\\cal F\}\_\{k\}\-measurable and satisfies‖Wk\+1−Wk‖F≤β¯​‖Mk‖F\\\|W\_\{k\+1\}\-W\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq\\bar\{\\beta\}\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}for a deterministicβ¯\>0\\bar\{\\beta\}\>0with, ifη\>0\\eta\>0,2​T​Rπ​Rg​β¯≤12TR\_\{\\pi\}R\_\{g\}\\bar\{\\beta\}\\leq 1and2​η​CF​β¯≤1−η2\\eta C\_\{F\}\\bar\{\\beta\}\\leq 1\-\\eta\. Then‖Mk‖F≤2​Rg\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq 2R\_\{g\}for allk∈\[K\]k\\in\[K\], and Lemma[8](https://arxiv.org/html/2610.33477#Thmtheorem8)applies withℓ¯2=2​β¯​Rg\\bar\{\\ell\}\_\{2\}=2\\bar\{\\beta\}R\_\{g\}\.

Proof\.Forη=0\\eta=0,Mk=GkM\_\{k\}=G\_\{k\}, and Lemma[5](https://arxiv.org/html/2610.33477#Thmtheorem5)\(iii\) gives‖Mk‖F≤Rg\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq R\_\{g\}\. Letη\>0\\eta\>0; then‖M1‖F=‖G1‖F≤Rg\\\|M\_\{1\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}=\\\|G\_\{1\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq R\_\{g\}\. Fork≥2k\\geq 2with‖Mk−1‖F≤2​Rg\\\|M\_\{k\-1\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq 2R\_\{g\}, the spectral norm of the step is at most its Frobenius norm, such that‖Wk−Wk−1‖2≤β¯​‖Mk−1‖F≤2​β¯​Rg≤1/\(T​Rπ\)\\\|W\_\{k\}\-W\_\{k\-1\}\\\|\_\{2\}\\leq\\bar\{\\beta\}\\\|M\_\{k\-1\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq 2\\bar\{\\beta\}R\_\{g\}\\leq\\nicefrac\{\{1\}\}\{\{\(TR\_\{\\pi\}\)\}\}, such that Lemma[6](https://arxiv.org/html/2610.33477#Thmtheorem6)\(iii\) withW=WkW=W\_\{k\},W′=Wk−1W^\{\\prime\}=W\_\{k\-1\},O=OkO=O\_\{k\}, andv=vk−1v=v\_\{k\-1\}gives‖Gk−G~k‖F≤CF​‖Wk−Wk−1‖2≤2​CF​β¯​Rg\\\|G\_\{k\}\-\\widetilde\{G\}\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq C\_\{F\}\\\|W\_\{k\}\-W\_\{k\-1\}\\\|\_\{2\}\\leq 2C\_\{F\}\\bar\{\\beta\}R\_\{g\}, and \([12](https://arxiv.org/html/2610.33477#S3.E12)\), Lemma[5](https://arxiv.org/html/2610.33477#Thmtheorem5)\(iii\), and2​η​CF​β¯≤1−η2\\eta C\_\{F\}\\bar\{\\beta\}\\leq 1\-\\etagive

‖Mk‖F≤\(1−η\)​‖Gk‖F\+η​‖Mk−1‖F\+η​‖Gk−G~k‖F≤\(1−η\)​Rg\+2​η​Rg\+2​η​CF​β¯​Rg≤2​Rg;\\begin\{split\}\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}&\\leq\(1\-\\eta\)\\\|G\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\+\\eta\\\|M\_\{k\-1\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\+\\eta\\\|G\_\{k\}\-\\widetilde\{G\}\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\\\ &\\leq\(1\-\\eta\)R\_\{g\}\+2\\eta R\_\{g\}\+2\\eta C\_\{F\}\\bar\{\\beta\}R\_\{g\}\\leq 2R\_\{g\};\\end\{split\}the bound follows by induction\. In both cases,‖Wk\+1−Wk‖2≤β¯​‖Mk‖F≤2​β¯​Rg\\\|W\_\{k\+1\}\-W\_\{k\}\\\|\_\{2\}\\leq\\bar\{\\beta\}\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq 2\\bar\{\\beta\}R\_\{g\}for allkk, andWkW\_\{k\}isℱk−1\{\\cal F\}\_\{k\-1\}\-measurable by induction, such that Lemma[8](https://arxiv.org/html/2610.33477#Thmtheorem8)applies withℓ¯2=2​β¯​Rg\\bar\{\\ell\}\_\{2\}=2\\bar\{\\beta\}R\_\{g\}, whereT​Rπ​ℓ¯2≤1TR\_\{\\pi\}\\bar\{\\ell\}\_\{2\}\\leq 1ifη\>0\\eta\>0\.

## Appendix KProof of Theorem[11](https://arxiv.org/html/2610.33477#Thmtheorem11)

Proof\.LetekM:=Mk−∇J​\(Wk\)e^\{M\}\_\{k\}:=M\_\{k\}\-\\nabla J\(W\_\{k\}\)\. Since the step isℱk\{\\cal F\}\_\{k\}\-measurable, everyWkW\_\{k\}isℱk−1\{\\cal F\}\_\{k\-1\}\-measurable, and‖Wk−Wk−1‖2=β​‖Nk−1‖2≤β​‖Nk−1‖≤β\\\|W\_\{k\}\-W\_\{k\-1\}\\\|\_\{2\}=\\beta\\\|N\_\{k\-1\}\\\|\_\{2\}\\leq\\beta\\\|N\_\{k\-1\}\\\|\\leq\\beta, such that Lemma[8](https://arxiv.org/html/2610.33477#Thmtheorem8)applies withℓ¯2=β\\bar\{\\ell\}\_\{2\}=\\beta, whereT​Rπ​β≤1TR\_\{\\pi\}\\beta\\leq 1ifη\>0\\eta\>0\. SubstitutingW=WkW=W\_\{k\}andW′=Wk\+1=Wk\+β​NkW^\{\\prime\}=W\_\{k\+1\}=W\_\{k\}\+\\beta N\_\{k\}into \([17](https://arxiv.org/html/2610.33477#S4.E17)\) and using‖Nk‖2≤‖Nk‖≤1\\\|N\_\{k\}\\\|\_\{2\}\\leq\\\|N\_\{k\}\\\|\\leq 1giveβ​⟨∇J​\(Wk\),Nk⟩F≤J⁡\(Wk\+1\)−J⁡\(Wk\)\+L​β2/2\\beta\\langle\\nabla J\(W\_\{k\}\),N\_\{k\}\\rangle\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq J\(W\_\{k\+1\}\)\-J\(W\_\{k\}\)\+\\nicefrac\{\{L\\beta^\{2\}\}\}\{\{2\}\}\. By Definition[1](https://arxiv.org/html/2610.33477#Thmtheorem1)\(i\) and the triangle inequality,⟨Mk,Nk⟩F≥χ−1​‖Mk‖†≥χ−1​\(‖∇J​\(Wk\)‖†−‖ekM‖†\)\\langle M\_\{k\},N\_\{k\}\\rangle\_\{\\scriptscriptstyle\\textrm\{F\}\}\\geq\\chi^\{\-1\}\\\|M\_\{k\}\\\|\_\{\\dagger\}\\geq\\chi^\{\-1\}\(\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\\dagger\}\-\\\|e^\{M\}\_\{k\}\\\|\_\{\\dagger\}\), and the definition of the dual norm gives⟨ekM,Nk⟩F≤‖ekM‖†​‖Nk‖≤‖ekM‖†\\langle e^\{M\}\_\{k\},N\_\{k\}\\rangle\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq\\\|e^\{M\}\_\{k\}\\\|\_\{\\dagger\}\\\|N\_\{k\}\\\|\\leq\\\|e^\{M\}\_\{k\}\\\|\_\{\\dagger\}, such that⟨∇J​\(Wk\),Nk⟩F=⟨Mk,Nk⟩F−⟨ekM,Nk⟩F≥χ−1​‖∇J​\(Wk\)‖†−\(1\+χ−1\)​‖ekM‖†\\langle\\nabla J\(W\_\{k\}\),N\_\{k\}\\rangle\_\{\\scriptscriptstyle\\textrm\{F\}\}=\\langle M\_\{k\},N\_\{k\}\\rangle\_\{\\scriptscriptstyle\\textrm\{F\}\}\-\\langle e^\{M\}\_\{k\},N\_\{k\}\\rangle\_\{\\scriptscriptstyle\\textrm\{F\}\}\\geq\\chi^\{\-1\}\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\\dagger\}\-\(1\+\\chi^\{\-1\}\)\\\|e^\{M\}\_\{k\}\\\|\_\{\\dagger\}; hence,

‖∇J​\(Wk\)‖†≤χ​J⁡\(Wk\+1\)−J⁡\(Wk\)β\+\(1\+χ\)​‖ekM‖†\+χ​L​β2\.\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\\dagger\}\\leq\\chi\\,\\frac\{J\(W\_\{k\+1\}\)\-J\(W\_\{k\}\)\}\{\\beta\}\+\(1\+\\chi\)\\\|e^\{M\}\_\{k\}\\\|\_\{\\dagger\}\+\\frac\{\\chi L\\beta\}\{2\}\.\(49\)Summing \([49](https://arxiv.org/html/2610.33477#A11.E49)\) overk∈\[K\]k\\in\[K\], usingJ⁡\(WK\+1\)≤J∗J\(W\_\{K\+1\}\)\\leq J^\{\*\}, taking expectations, dividing byKK, and inserting \([23](https://arxiv.org/html/2610.33477#S4.E23)\) withℓ¯2=β\\bar\{\\ell\}\_\{2\}=\\betaand1K​∑k=1Kηk−1≤1K⁡\(1−η\)\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\eta^\{k\-1\}\\leq\\frac\{1\}\{K\(1\-\\eta\)\}yield \([25](https://arxiv.org/html/2610.33477#S4.E25)\)\. With1−η=K−2/31\-\\eta=K^\{\-2/3\}andβ=cK−2/3\\beta=cK^\{\-2/3\}, the conditionT​Rπ​β≤1TR\_\{\\pi\}\\beta\\leq 1is equivalent toK≥\(c​T​Rπ\)3/2K\\geq\(cTR\_\{\\pi\}\)^\{3/2\}, and substitutingΔ1/\(β​K\)=Δ1/cK−1/3\\nicefrac\{\{\\Delta\_\{1\}\}\}\{\{\(\\beta K\)\}\}=\\nicefrac\{\{\\Delta\_\{1\}\}\}\{\{c\}\}\\,K^\{\-1/3\},L​β/2=c​L/2K−2/3\\nicefrac\{\{L\\beta\}\}\{\{2\}\}=\\nicefrac\{\{cL\}\}\{\{2\}\}\\,K^\{\-2/3\},σ¯/\[K⁡\(1−η\)\]=σ¯K−1/3\\nicefrac\{\{\\bar\{\\sigma\}\}\}\{\{\[K\(1\-\\eta\)\]\}\}=\\bar\{\\sigma\}K^\{\-1/3\},2​\(1−η\)σ¯=2σ¯K−1/3\\sqrt\{2\(1\-\\eta\)\}\\,\\bar\{\\sigma\}=\\sqrt\{2\}\\,\\bar\{\\sigma\}K^\{\-1/3\},2​η​CF​β/1−η≤2cCFK−1/3\\nicefrac\{\{\\sqrt\{2\}\\eta C\_\{F\}\\beta\}\}\{\{\\sqrt\{1\-\\eta\}\}\}\\leq\\sqrt\{2\}\\,cC\_\{F\}K^\{\-1/3\}, andη​DF​β/\(1−η\)≤c​DF\\nicefrac\{\{\\eta D\_\{F\}\\beta\}\}\{\{\(1\-\\eta\)\}\}\\leq cD\_\{F\}into \([25](https://arxiv.org/html/2610.33477#S4.E25)\) yields \([26](https://arxiv.org/html/2610.33477#S4.E26)\)\.

## Appendix LProof of Theorem[12](https://arxiv.org/html/2610.33477#Thmtheorem12)

Proof\.Letuk:=1/βku\_\{k\}:=\\nicefrac\{\{1\}\}\{\{\\beta\_\{k\}\}\},ekM:=Mk−∇J​\(Wk\)e^\{M\}\_\{k\}:=M\_\{k\}\-\\nabla J\(W\_\{k\}\), andΔk:=J∗−J⁡\(Wk\)∈\[0,Δmax\]\\Delta\_\{k\}:=J^\{\*\}\-J\(W\_\{k\}\)\\in\[0,\\Delta\_\{\\max\}\], whereΔmax≤2​Rr\\Delta\_\{\\max\}\\leq 2R\_\{r\}because\|J⁡\(W\)\|≤Rr\|J\(W\)\|\\leq R\_\{r\}for everyWW\. The stepWk\+1−Wk=βk​MkW\_\{k\+1\}\-W\_\{k\}=\\beta\_\{k\}M\_\{k\}isℱk\{\\cal F\}\_\{k\}\-measurable with‖βk​Mk‖F≤β¯​‖Mk‖F\\\|\\beta\_\{k\}M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq\\bar\{\\beta\}\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}by Definition[1](https://arxiv.org/html/2610.33477#Thmtheorem1)\(ii\), such that Lemma[22](https://arxiv.org/html/2610.33477#Thmtheorem22)gives‖Mk‖F≤2​Rg\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq 2R\_\{g\}and Lemma[8](https://arxiv.org/html/2610.33477#Thmtheorem8)applies withℓ¯2=2​β¯​Rg\\bar\{\\ell\}\_\{2\}=2\\bar\{\\beta\}R\_\{g\}\. SubstitutingW=WkW=W\_\{k\}andW′=Wk\+βk​MkW^\{\\prime\}=W\_\{k\}\+\\beta\_\{k\}M\_\{k\}into \([17](https://arxiv.org/html/2610.33477#S4.E17)\), using‖Mk‖2≤‖Mk‖F\\\|M\_\{k\}\\\|\_\{2\}\\leq\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}, and inserting⟨∇J​\(Wk\),Mk⟩F=12​\(‖∇J​\(Wk\)‖F2\+‖Mk‖F2−‖ekM‖F2\)\\langle\\nabla J\(W\_\{k\}\),M\_\{k\}\\rangle\_\{\\scriptscriptstyle\\textrm\{F\}\}=\\frac\{1\}\{2\}\(\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\+\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\-\\\|e^\{M\}\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\)give

βk2​‖∇J​\(Wk\)‖F2\+βk​\(1−L​βk\)2​‖Mk‖F2≤J⁡\(Wk\+1\)−J⁡\(Wk\)\+βk2​‖ekM‖F2\\frac\{\\beta\_\{k\}\}\{2\}\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\+\\frac\{\\beta\_\{k\}\(1\-L\\beta\_\{k\}\)\}\{2\}\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\\leq J\(W\_\{k\+1\}\)\-J\(W\_\{k\}\)\+\\frac\{\\beta\_\{k\}\}\{2\}\\\|e^\{M\}\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}whereL​βk≤L​β¯≤1L\\beta\_\{k\}\\leq L\\bar\{\\beta\}\\leq 1, such that‖∇J​\(Wk\)‖F2≤2​uk​\(Δk−Δk\+1\)\+‖ekM‖F2\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\\leq 2u\_\{k\}\(\\Delta\_\{k\}\-\\Delta\_\{k\+1\}\)\+\\\|e^\{M\}\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\. Sinceuku\_\{k\}need not be monotone, we sum by parts and useu1≤ϑ/β¯u\_\{1\}\\leq\\nicefrac\{\{\\sqrt\{\\vartheta\}\}\}\{\{\\bar\{\\beta\}\}\},Δk∈\[0,Δmax\]\\Delta\_\{k\}\\in\[0,\\Delta\_\{\\max\}\], and the variation bound of Definition[1](https://arxiv.org/html/2610.33477#Thmtheorem1)\(ii\):

∑k=1Kuk​\(Δk−Δk\+1\)=u1​Δ1\+∑k=2K\(uk−uk−1\)​Δk−uK​ΔK\+1≤ϑ​Δ1\+Vu​Δmaxβ¯=Λ2​β¯\.\\sum\_\{k=1\}^\{K\}u\_\{k\}\(\\Delta\_\{k\}\-\\Delta\_\{k\+1\}\)=u\_\{1\}\\Delta\_\{1\}\+\\sum\_\{k=2\}^\{K\}\(u\_\{k\}\-u\_\{k\-1\}\)\\Delta\_\{k\}\-u\_\{K\}\\Delta\_\{K\+1\}\\leq\\frac\{\\sqrt\{\\vartheta\}\\,\\Delta\_\{1\}\+V\_\{u\}\\Delta\_\{\\max\}\}\{\\bar\{\\beta\}\}=\\frac\{\\Lambda\}\{2\\bar\{\\beta\}\}\.Summing overk∈\[K\]k\\in\[K\], taking expectations, dividing byKK, and inserting \([24](https://arxiv.org/html/2610.33477#S4.E24)\) withℓ¯2=2​β¯​Rg\\bar\{\\ell\}\_\{2\}=2\\bar\{\\beta\}R\_\{g\}and1K​∑k=1Kη2​\(k−1\)≤1K⁡\(1−η2\)\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\eta^\{2\(k\-1\)\}\\leq\\frac\{1\}\{K\(1\-\\eta^\{2\}\)\}yield

1K​∑k=1K𝔼⁡\[‖∇J​\(Wk\)‖F2\]≤Λβ¯​K\+2​σ¯2K⁡\(1−η2\)\+4​\(1−η\)​σ¯2\+16​η2​CF2​β¯2​Rg21−η\+2​\(ϵbias\+2​η​DF​β¯​Rg1−η\)2\\begin\{split\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\[\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\]&\\leq\\frac\{\\Lambda\}\{\\bar\{\\beta\}K\}\+\\frac\{2\\bar\{\\sigma\}^\{2\}\}\{K\(1\-\\eta^\{2\}\)\}\+4\(1\-\\eta\)\\,\\bar\{\\sigma\}^\{2\}\\\\ &\\quad\+\\frac\{16\\eta^\{2\}C\_\{F\}^\{2\}\\bar\{\\beta\}^\{2\}R\_\{g\}^\{2\}\}\{1\-\\eta\}\+2\\Big\(\\epsilon\_\{\\rm bias\}\+\\frac\{2\\eta D\_\{F\}\\bar\{\\beta\}R\_\{g\}\}\{1\-\\eta\}\\Big\)^\{2\}\\end\{split\}and Jensen’s inequality and∑ixi≤∑ixi\\sqrt\{\\sum\_\{i\}x\_\{i\}\}\\leq\\sum\_\{i\}\\sqrt\{x\_\{i\}\}yield \([27](https://arxiv.org/html/2610.33477#S4.E27)\)\. With1−η=K−1/21\-\\eta=K^\{\-1/2\}andβ¯=cK−1/2\\bar\{\\beta\}=cK^\{\-1/2\}, the conditionL​β¯≤1L\\bar\{\\beta\}\\leq 1follows fromK≥\(c​L\)2K\\geq\(cL\)^\{2\}, the condition2​η​CF​β¯≤1−η2\\eta C\_\{F\}\\bar\{\\beta\}\\leq 1\-\\etafromc≤1/\(2​CF\)c\\leq\\nicefrac\{\{1\}\}\{\{\(2C\_\{F\}\)\}\}, and the condition2​T​Rπ​Rg​β¯≤12TR\_\{\\pi\}R\_\{g\}\\bar\{\\beta\}\\leq 1fromc≤1/\(2​CF\)c\\leq\\nicefrac\{\{1\}\}\{\{\(2C\_\{F\}\)\}\}andβ¯≤c\\bar\{\\beta\}\\leq c, sinceCF≥\(exp⁡\(1\)−1\)​T​RA​Rπ2C\_\{F\}\\geq\(\\exp\(1\)\-1\)TR\_\{A\}R\_\{\\pi\}^\{2\}andRg=RA​RπR\_\{g\}=R\_\{A\}R\_\{\\pi\}give2​c​T​Rπ​Rg≤1/\(exp⁡\(1\)−1\)<12cTR\_\{\\pi\}R\_\{g\}\\leq\\nicefrac\{\{1\}\}\{\{\(\\exp\(1\)\-1\)\}\}<1\. SubstitutingΛ/\(β¯​K\)=Λ/cK−1/4\\sqrt\{\\nicefrac\{\{\\Lambda\}\}\{\{\(\\bar\{\\beta\}K\)\}\}\}=\\sqrt\{\\nicefrac\{\{\\Lambda\}\}\{\{c\}\}\}\\,K^\{\-1/4\},1/K⁡\(1−η2\)≤1/K⁡\(1−η\)=K−1/4\\nicefrac\{\{1\}\}\{\{\\sqrt\{K\(1\-\\eta^\{2\}\)\}\}\}\\leq\\nicefrac\{\{1\}\}\{\{\\sqrt\{K\(1\-\\eta\)\}\}\}=K^\{\-1/4\},1−η=K−1/4\\sqrt\{1\-\\eta\}=K^\{\-1/4\},η​β¯/1−η≤cK−1/4\\nicefrac\{\{\\eta\\bar\{\\beta\}\}\}\{\{\\sqrt\{1\-\\eta\}\}\}\\leq cK^\{\-1/4\}, andη​β¯/\(1−η\)≤c\\nicefrac\{\{\\eta\\bar\{\\beta\}\}\}\{\{\(1\-\\eta\)\}\}\\leq cinto \([27](https://arxiv.org/html/2610.33477#S4.E27)\) yields \([28](https://arxiv.org/html/2610.33477#S4.E28)\)\.

## Appendix MProofs for the Instances

### M\.1Proof of Lemma[15](https://arxiv.org/html/2610.33477#Thmtheorem15)

Proof\.*Part \(i\)\.*IfMk=0M\_\{k\}=0, the NSO iterates,Θk\\Theta\_\{k\}, andRkR\_\{k\}vanish by convention\. Otherwise, letMk=Uk​Σk​Vk⊤M\_\{k\}=U\_\{k\}\\Sigma\_\{k\}V\_\{k\}^\{\\top\}be the compact singular value decomposition withΣk∈ℝr¯k×r¯k\\Sigma\_\{k\}\\in\\mathbb\{R\}^\{\\bar\{r\}\_\{k\}\\times\\bar\{r\}\_\{k\}\},r¯k:=rank⁡\(Mk\)\\bar\{r\}\_\{k\}:=\\operatorname\{rank\}\(M\_\{k\}\), andUk⊤​Uk=Vk⊤​Vk=Ir¯kU\_\{k\}^\{\\top\}U\_\{k\}=V\_\{k\}^\{\\top\}V\_\{k\}=I\_\{\\bar\{r\}\_\{k\}\}, such thatΘk=Uk​Vk⊤\\Theta\_\{k\}=U\_\{k\}V\_\{k\}^\{\\top\}\. By induction over \([13](https://arxiv.org/html/2610.33477#S3.E13)\) fromNk,0=Mk/‖Mk‖FN\_\{k,0\}=\\nicefrac\{\{M\_\{k\}\}\}\{\{\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\}\},Nk,p=Uk​Σk,p​Vk⊤N\_\{k,p\}=U\_\{k\}\\Sigma\_\{k,p\}V\_\{k\}^\{\\top\}withΣk,0=Σk/‖Mk‖F\\Sigma\_\{k,0\}=\\nicefrac\{\{\\Sigma\_\{k\}\}\}\{\{\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\}\}: indeed,In−Nk,p⊤​Nk,p=\(In−Vk​Vk⊤\)\+Vk​\(Ir¯k−Σk,p2\)​Vk⊤I\_\{n\}\-N\_\{k,p\}^\{\\top\}N\_\{k,p\}=\(I\_\{n\}\-V\_\{k\}V\_\{k\}^\{\\top\}\)\+V\_\{k\}\(I\_\{\\bar\{r\}\_\{k\}\}\-\\Sigma\_\{k,p\}^\{2\}\)V\_\{k\}^\{\\top\}and\(In−Vk​Vk⊤\)​Vk=0\(I\_\{n\}\-V\_\{k\}V\_\{k\}^\{\\top\}\)V\_\{k\}=0giveNk,p​\(In−Nk,p⊤​Nk,p\)i=Uk​Σk,p​\(Ir¯k−Σk,p2\)i​Vk⊤N\_\{k,p\}\(I\_\{n\}\-N\_\{k,p\}^\{\\top\}N\_\{k,p\}\)^\{i\}=U\_\{k\}\\Sigma\_\{k,p\}\(I\_\{\\bar\{r\}\_\{k\}\}\-\\Sigma\_\{k,p\}^\{2\}\)^\{i\}V\_\{k\}^\{\\top\}for alli≥0i\\geq 0, such that \([13](https://arxiv.org/html/2610.33477#S3.E13)\) maps each singular valueσk,p\\sigma\_\{k,p\}ofNk,pN\_\{k,p\}to

σk,p\+1=f⁡\(σk,p\)=σk,p​∑i=0κ2−114i​\(2​ii\)​\(1−σk,p2\)i\.\\sigma\_\{k,p\+1\}=f\(\\sigma\_\{k,p\}\)=\\sigma\_\{k,p\}\\sum\_\{i=0\}^\{\\kappa\_\{2\}\-1\}\\frac\{1\}\{4^\{i\}\}\\binom\{2i\}\{i\}\(1\-\\sigma\_\{k,p\}^\{2\}\)^\{i\}\.\(50\)
Matching the coefficients of\(1−x2\)j\(1\-x^\{2\}\)^\{j\}givesf′​\(x\)=\(2​κ2−1\)​\(2​κ2−2κ2−1\)​41−κ2​\(1−x2\)κ2−1≥0f^\{\\prime\}\(x\)=\(2\\kappa\_\{2\}\-1\)\\binom\{2\\kappa\_\{2\}\-2\}\{\\kappa\_\{2\}\-1\}4^\{1\-\\kappa\_\{2\}\}\(1\-x^\{2\}\)^\{\\kappa\_\{2\}\-1\}\\geq 0on\[0,1\]\[0,1\], such thatfpf^\{p\}preserves the order of the singular values and theiith largest singular value ofNk,pN\_\{k,p\}isfp​\(σi​\(Mk\)/‖Mk‖F\)f^\{p\}\(\\nicefrac\{\{\\sigma\_\{i\}\(M\_\{k\}\)\}\}\{\{\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\}\}\)\.

*Monotone convergence and local order\.*Since‖Mk‖2≤‖Mk‖F\\\|M\_\{k\}\\\|\_\{2\}\\leq\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}, every nonzero singular value satisfiesσk,0∈\(0,1\]\\sigma\_\{k,0\}\\in\(0,1\], and the Maclaurin series\(1−x\)−1/2=∑i≥04−i\(2​ii\)xi\(1\-x\)^\{\-1/2\}=\\sum\_\{i\\geq 0\}4^\{\-i\}\\binom\{2i\}\{i\}x^\{i\},x∈\[0,1\)x\\in\[0,1\), withx=1−σk,p2x=1\-\\sigma\_\{k,p\}^\{2\}gives

σk,p​∑i=0∞14i​\(2​ii\)​\(1−σk,p2\)i=1\.\\sigma\_\{k,p\}\\sum\_\{i=0\}^\{\\infty\}\\frac\{1\}\{4^\{i\}\}\\binom\{2i\}\{i\}\(1\-\\sigma\_\{k,p\}^\{2\}\)^\{i\}=1\.\(51\)Since its summands are nonnegative, \([50](https://arxiv.org/html/2610.33477#A13.E50)\) givesσk,p\+1≤1\\sigma\_\{k,p\+1\}\\leq 1; since thei=0i=0term of the sum in \([50](https://arxiv.org/html/2610.33477#A13.E50)\) equals one and the others are positive forσk,p∈\(0,1\)\\sigma\_\{k,p\}\\in\(0,1\),σk,p\+1\>σk,p\\sigma\_\{k,p\+1\}\>\\sigma\_\{k,p\}there, andσ=1\\sigma=1is the unique fixed point in\(0,1\]\(0,1\]\. Hence,‖Nk,p‖2≤1\\\|N\_\{k,p\}\\\|\_\{2\}\\leq 1, eachσk,p\\sigma\_\{k,p\}increases monotonically to one, andNk,p→ΘkN\_\{k,p\}\\to\\Theta\_\{k\}\. Withek,p:=1−σk,pe\_\{k,p\}:=1\-\\sigma\_\{k,p\}, subtracting \([50](https://arxiv.org/html/2610.33477#A13.E50)\) from \([51](https://arxiv.org/html/2610.33477#A13.E51)\) gives, asek,p→0e\_\{k,p\}\\to 0,

ek,p\+1=\(1−ek,p\)​∑i=κ2∞14i​\(2​ii\)​\[ek,p​\(2−ek,p\)\]i=12κ2​\(2​κ2κ2\)​ek,pκ2\+𝒪⁡\(ek,pκ2\+1\)e\_\{k,p\+1\}=\(1\-e\_\{k,p\}\)\\sum\_\{i=\\kappa\_\{2\}\}^\{\\infty\}\\frac\{1\}\{4^\{i\}\}\\binom\{2i\}\{i\}\\big\[e\_\{k,p\}\(2\-e\_\{k,p\}\)\\big\]^\{i\}=\\frac\{1\}\{2^\{\\kappa\_\{2\}\}\}\\binom\{2\\kappa\_\{2\}\}\{\\kappa\_\{2\}\}e\_\{k,p\}^\{\\kappa\_\{2\}\}\+\{\\cal O\}\(e\_\{k,p\}^\{\\kappa\_\{2\}\+1\}\)such that, by[Kelley \(1995, Definition 4\.1\.1\)](https://arxiv.org/html/2610.33477#bib.bib25), each singular value converges to one locally with orderκ2\\kappa\_\{2\}, and so doesNk,pN\_\{k,p\}toΘk\\Theta\_\{k\}, since‖Nk,p−Θk‖2=‖Σk,p−Ir¯k‖2=maxi⁡ek,p,i\\\|N\_\{k,p\}\-\\Theta\_\{k\}\\\|\_\{2\}=\\\|\\Sigma\_\{k,p\}\-I\_\{\\bar\{r\}\_\{k\}\}\\\|\_\{2\}=\\max\_\{i\}e\_\{k,p,i\}with the errorek,p,ie\_\{k,p,i\}of theiith singular value\.

*Global contraction\.*Letδk,p:=1−σk,p2\\delta\_\{k,p\}:=1\-\\sigma\_\{k,p\}^\{2\}andψ¯​\(δ\):=∑i=0κ2−14−i​\(2​ii\)​δi\\bar\{\\psi\}\(\\delta\):=\\sum\_\{i=0\}^\{\\kappa\_\{2\}\-1\}4^\{\-i\}\\binom\{2i\}\{i\}\\delta^\{i\}\. Sinceψ¯\\bar\{\\psi\}truncates the series of\(1−δ\)−1/2\(1\-\\delta\)^\{\-1/2\}, whose square is∑j≥0δj\\sum\_\{j\\geq 0\}\\delta^\{j\}, the coefficients ofψ¯2\\bar\{\\psi\}^\{2\}equal one up to degreeκ2−1\\kappa\_\{2\}\-1and are nonnegative beyond; hence,ψ¯​\(δ\)2≥∑j=0κ2−1δj\\bar\{\\psi\}\(\\delta\)^\{2\}\\geq\\sum\_\{j=0\}^\{\\kappa\_\{2\}\-1\}\\delta^\{j\}on\[0,1\]\[0,1\], and \([50](https://arxiv.org/html/2610.33477#A13.E50)\) gives0≤δk,p\+1=1−\(1−δk,p\)​ψ¯​\(δk,p\)2≤δk,pκ20\\leq\\delta\_\{k,p\+1\}=1\-\(1\-\\delta\_\{k,p\}\)\\bar\{\\psi\}\(\\delta\_\{k,p\}\)^\{2\}\\leq\\delta\_\{k,p\}^\{\\kappa\_\{2\}\}\. Therefore,1−σk,P2≤1−\(1−\(1−σk,02\)κ2P2\)1/2≤\(1−σk,02\)κ2P21\-\\sigma\_\{k,P\_\{2\}\}\\leq 1\-\(1\-\(1\-\\sigma\_\{k,0\}^\{2\}\)^\{\\kappa\_\{2\}^\{P\_\{2\}\}\}\)^\{1/2\}\\leq\(1\-\\sigma\_\{k,0\}^\{2\}\)^\{\\kappa\_\{2\}^\{P\_\{2\}\}\}for every nonzero singular value, andσk,0≥ak\>0\\sigma\_\{k,0\}\\geq a\_\{k\}\>0gives‖Rk‖2=maxi⁡ek,P2,i=1−σmin\+​\(Nk,P2\)≤\(1−ak2\)κ2P2<1\\\|R\_\{k\}\\\|\_\{2\}=\\max\_\{i\}e\_\{k,P\_\{2\},i\}=1\-\\sigma\_\{\\min\}^\{\+\}\(N\_\{k,P\_\{2\}\}\)\\leq\(1\-a\_\{k\}^\{2\}\)^\{\\kappa\_\{2\}^\{P\_\{2\}\}\}<1\.

*Part \(ii\)\.*IfMk=0M\_\{k\}=0, both sides of \([31](https://arxiv.org/html/2610.33477#S5.E31)\) vanish\. Otherwise, with the nonzero singular valuesσ1,…,σr¯k\\sigma\_\{1\},\\ldots,\\sigma\_\{\\bar\{r\}\_\{k\}\}ofMkM\_\{k\}andxi:=σi/‖Mk‖F∈\(0,1\]x\_\{i\}:=\\nicefrac\{\{\\sigma\_\{i\}\}\}\{\{\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\}\}\\in\(0,1\], part \(i\) gives‖Mk‖∗−⟨Mk,Nk,P2⟩F=∑i=1r¯kσi​\(1−fP2​\(xi\)\)=‖Mk‖F​∑i=1r¯kxi​\(1−fP2​\(xi\)\)≤r¯k​sP2​‖Mk‖F≤r​sP2​‖Mk‖∗\\\|M\_\{k\}\\\|\_\{\*\}\-\\langle M\_\{k\},N\_\{k,P\_\{2\}\}\\rangle\_\{\\scriptscriptstyle\\textrm\{F\}\}=\\sum\_\{i=1\}^\{\\bar\{r\}\_\{k\}\}\\sigma\_\{i\}\(1\-f^\{P\_\{2\}\}\(x\_\{i\}\)\)=\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\sum\_\{i=1\}^\{\\bar\{r\}\_\{k\}\}x\_\{i\}\(1\-f^\{P\_\{2\}\}\(x\_\{i\}\)\)\\leq\\bar\{r\}\_\{k\}s\_\{P\_\{2\}\}\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq r\\,s\_\{P\_\{2\}\}\\\|M\_\{k\}\\\|\_\{\*\}, sincer¯k≤r\\bar\{r\}\_\{k\}\\leq rand‖Mk‖F≤‖Mk‖∗\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq\\\|M\_\{k\}\\\|\_\{\*\}\. Finally, the global contraction givesx\(1−fP\(x\)\)≤x\(1−x2\)κ2P≤xexp\(−κ2Px2\)≤\(2eκ2P\)−1/2x\(1\-f^\{P\}\(x\)\)\\leq x\(1\-x^\{2\}\)^\{\\kappa\_\{2\}^\{P\}\}\\leq x\\exp\(\-\\kappa\_\{2\}^\{P\}x^\{2\}\)\\leq\(2e\\kappa\_\{2\}^\{P\}\)^\{\-1/2\}forx∈\[0,1\]x\\in\[0,1\], where the last inequality maximizes overxx; hence,sP≤\(2eκ2P\)−1/2s\_\{P\}\\leq\(2e\\kappa\_\{2\}^\{P\}\)^\{\-1/2\}, andκ2P2≥4​r2\\kappa\_\{2\}^\{P\_\{2\}\}\\geq 4r^\{2\}givesrsP2≤\(8e\)−1/2<0\.22r\\,s\_\{P\_\{2\}\}\\leq\(8e\)^\{\-1/2\}<0\.22\.

### M\.2Proofs of Corollaries[16](https://arxiv.org/html/2610.33477#Thmtheorem16)–[18](https://arxiv.org/html/2610.33477#Thmtheorem18)

Hereinafter, we provide the proofs of Corollaries[16](https://arxiv.org/html/2610.33477#Thmtheorem16)–[18](https://arxiv.org/html/2610.33477#Thmtheorem18)\.

*Muon\.*The NSO outputNk,P2N\_\{k,P\_\{2\}\}is a function ofMkM\_\{k\}and henceℱk\{\\cal F\}\_\{k\}\-measurable, and Lemma[15](https://arxiv.org/html/2610.33477#Thmtheorem15)gives‖Nk,P2‖2≤1\\\|N\_\{k,P\_\{2\}\}\\\|\_\{2\}\\leq 1and⟨Mk,Nk,P2⟩F≥\(1−r​sP2\)​‖Mk‖∗=χ−1​‖Mk‖∗\\langle M\_\{k\},N\_\{k,P\_\{2\}\}\\rangle\_\{\\scriptscriptstyle\\textrm\{F\}\}\\geq\(1\-r\\,s\_\{P\_\{2\}\}\)\\\|M\_\{k\}\\\|\_\{\*\}=\\chi^\{\-1\}\\\|M\_\{k\}\\\|\_\{\*\}withχ≥1\\chi\\geq 1ifr​sP2<1r\\,s\_\{P\_\{2\}\}<1\. Hence, the Muon step is normalized in the spectral norm, whose dual norm is the nuclear norm withκ†=r1/2\\kappa\_\{\\dagger\}=r^\{1/2\}, and Theorem[11](https://arxiv.org/html/2610.33477#Thmtheorem11)gives \([25](https://arxiv.org/html/2610.33477#S4.E25)\) and \([26](https://arxiv.org/html/2610.33477#S4.E26)\) in the nuclear norm, i\.e\., \([32](https://arxiv.org/html/2610.33477#S5.E32)\); Corollary[13](https://arxiv.org/html/2610.33477#Thmtheorem13)\(i\) applies verbatim\. Ifκ2P2≥4​r2\\kappa\_\{2\}^\{P\_\{2\}\}\\geq 4r^\{2\}, thenrsP2≤\(8e\)−1/2<0\.215r\\,s\_\{P\_\{2\}\}\\leq\(8e\)^\{\-1/2\}<0\.215, such thatχ<1/0\.785<1\.28\\chi<\\nicefrac\{\{1\}\}\{\{0\.785\}\}<1\.28\.

*NSGDM\.*The directionNk:=Mk/‖Mk‖FN\_\{k\}:=\\nicefrac\{\{M\_\{k\}\}\}\{\{\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\}\}ifMk≠0M\_\{k\}\\neq 0andNk:=0N\_\{k\}:=0otherwise isℱk\{\\cal F\}\_\{k\}\-measurable with‖Nk‖F≤1\\\|N\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq 1and⟨Mk,Nk⟩F=‖Mk‖F\\langle M\_\{k\},N\_\{k\}\\rangle\_\{\\scriptscriptstyle\\textrm\{F\}\}=\\\|M\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\. Hence, the NSGDM step is normalized withχ=1\\chi=1in the Frobenius norm, which satisfies∥⋅∥F≥∥⋅∥2\\\|\\cdot\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\geq\\\|\\cdot\\\|\_\{2\}and is self\-dual withκ†=1\\kappa\_\{\\dagger\}=1, and Theorem[11](https://arxiv.org/html/2610.33477#Thmtheorem11)bounds1K​∑k=1K𝔼⁡\[‖∇J​\(Wk\)‖F\]\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\[\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\]by \([25](https://arxiv.org/html/2610.33477#S4.E25)\) and, with its schedule, by the bracket of \([33](https://arxiv.org/html/2610.33477#S5.E33)\), such that‖∇J​\(Wk\)‖∗≤r1/2​‖∇J​\(Wk\)‖F\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\*\}\\leq r^\{1/2\}\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}, which holds since∇J​\(Wk\)\\nabla J\(W\_\{k\}\)has rank at mostrr, yields \([33](https://arxiv.org/html/2610.33477#S5.E33)\)\. At the samecc, \([33](https://arxiv.org/html/2610.33477#S5.E33)\) is \([32](https://arxiv.org/html/2610.33477#S5.E32)\) withr1/2r^\{1/2\}in place ofχ\\chion the optimization and second\-order terms,22in place of1\+χ1\+\\chion the noise and correction terms, and2​r1/2​\(ϵbias\+c​DF\)2r^\{1/2\}\(\\epsilon\_\{\\rm bias\}\+cD\_\{F\}\)in place of the floor\(1\+χ\)​\(ϵbias\+c​DF\)\(1\+\\chi\)\(\\epsilon\_\{\\rm bias\}\+cD\_\{F\}\); together with the monotonicity of each term of \([32](https://arxiv.org/html/2610.33477#S5.E32)\) inccand\(1\+χ\)/2≤χ\\nicefrac\{\{\(1\+\\chi\)\}\}\{\{2\}\}\\leq\\chi, this proves the comparison in Section[5\.4](https://arxiv.org/html/2610.33477#S5.SS4)\.

*SGDM\.*The stepβk​Mk\\beta\_\{k\}M\_\{k\}with the deterministicβk=β\\beta\_\{k\}=\\betais unnormalized withβ¯=β\\bar\{\\beta\}=\\beta,ϑ=1\\vartheta=1, andVu=0V\_\{u\}=0, such thatΛ=2​Δ1\\Lambda=2\\Delta\_\{1\}, and Theorem[12](https://arxiv.org/html/2610.33477#Thmtheorem12)and‖∇J​\(Wk\)‖∗≤r1/2​‖∇J​\(Wk\)‖F\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\*\}\\leq r^\{1/2\}\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}yield the nuclear form of \([27](https://arxiv.org/html/2610.33477#S4.E27)\) and, withβ=cK−1/2\\beta=cK^\{\-1/2\}, \([34](https://arxiv.org/html/2610.33477#S5.E34)\)\.

*AdaNorm\.*Letβk:=β/nk\\beta\_\{k\}:=\\nicefrac\{\{\\beta\}\}\{\{\\sqrt\{n\_\{k\}\}\}\}anduk:=1/βku\_\{k\}:=\\nicefrac\{\{1\}\}\{\{\\beta\_\{k\}\}\}, and recallβ¯=β​2/n0\\bar\{\\beta\}=\\beta\\sqrt\{\\nicefrac\{\{2\}\}\{\{n\_\{0\}\}\}\}andϑ=2\+Rg2/n0\\vartheta=2\+\\nicefrac\{\{R\_\{g\}^\{2\}\}\}\{\{n\_\{0\}\}\}\. Unrolling \([14](https://arxiv.org/html/2610.33477#S3.E14)\) givesnk=\(1−α\)k​n0\+α​∑j=1k\(1−α\)k−j​‖Gj‖F2n\_\{k\}=\(1\-\\alpha\)^\{k\}n\_\{0\}\+\\alpha\\sum\_\{j=1\}^\{k\}\(1\-\\alpha\)^\{k\-j\}\\\|G\_\{j\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}, a convex combination ofn0n\_\{0\}and‖G1‖F2,…,‖Gk‖F2\\\|G\_\{1\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\},\\ldots,\\\|G\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}in which the squared gradient norms have the total weightαk:=1−\(1−α\)k≤k​α≤1/2\\alpha\_\{k\}:=1\-\(1\-\\alpha\)^\{k\}\\leq k\\alpha\\leq\\nicefrac\{\{1\}\}\{\{2\}\}fork∈\[K\]k\\in\[K\]by Bernoulli’s inequality andα​K≤1/2\\alpha K\\leq\\nicefrac\{\{1\}\}\{\{2\}\}; hence,‖Gj‖F≤Rg\\\|G\_\{j\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}\\leq R\_\{g\}\(Lemma[5](https://arxiv.org/html/2610.33477#Thmtheorem5)\(iii\)\) gives, fork∈\[K\]k\\in\[K\],

n02≤\(1−αk\)​n0≤nk≤\(1−αk\)​n0\+αk​Rg2≤n0\+Rg22,i\.e\.,β¯ϑ≤βk≤β¯\.\\frac\{n\_\{0\}\}\{2\}\\leq\(1\-\\alpha\_\{k\}\)\\,n\_\{0\}\\leq n\_\{k\}\\leq\(1\-\\alpha\_\{k\}\)\\,n\_\{0\}\+\\alpha\_\{k\}R\_\{g\}^\{2\}\\leq n\_\{0\}\+\\frac\{R\_\{g\}^\{2\}\}\{2\},\\quad\\mbox\{i\.e\.,\}\\quad\\frac\{\\bar\{\\beta\}\}\{\\sqrt\{\\vartheta\}\}\\leq\\beta\_\{k\}\\leq\\bar\{\\beta\}\.\(52\)
Moreover,\|nk−nk−1\|=α​\|‖Gk‖F2−nk−1\|≤α⁡\(n0\+Rg2\)\|n\_\{k\}\-n\_\{k\-1\}\|=\\alpha\|\\\|G\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}\-n\_\{k\-1\}\|\\leq\\alpha\(n\_\{0\}\+R\_\{g\}^\{2\}\), since both‖Gk‖F2\\\|G\_\{k\}\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}^\{2\}andnk−1n\_\{k\-1\}lie in\[0,n0\+Rg2\]\[0,n\_\{0\}\+R\_\{g\}^\{2\}\], andnk\+nk−1≥2​n0\\sqrt\{n\_\{k\}\}\+\\sqrt\{n\_\{k\-1\}\}\\geq\\sqrt\{2n\_\{0\}\}, such thatα​K≤1/2\\alpha K\\leq\\nicefrac\{\{1\}\}\{\{2\}\}gives

∑k=2K\|uk−uk−1\|=1β​∑k=2K\|nk−nk−1\|nk\+nk−1≤K​α​\(n0\+Rg2\)β​2​n0≤n0\+Rg22​β​2​n0=ϑ−12​β¯\.\\sum\_\{k=2\}^\{K\}\|u\_\{k\}\-u\_\{k\-1\}\|=\\frac\{1\}\{\\beta\}\\sum\_\{k=2\}^\{K\}\\frac\{\|n\_\{k\}\-n\_\{k\-1\}\|\}\{\\sqrt\{n\_\{k\}\}\+\\sqrt\{n\_\{k\-1\}\}\}\\leq\\frac\{K\\alpha\\,\(n\_\{0\}\+R\_\{g\}^\{2\}\)\}\{\\beta\\sqrt\{2n\_\{0\}\}\}\\leq\\frac\{n\_\{0\}\+R\_\{g\}^\{2\}\}\{2\\beta\\sqrt\{2n\_\{0\}\}\}=\\frac\{\\vartheta\-1\}\{2\\bar\{\\beta\}\}\.
Sincenkn\_\{k\}, and henceβk\\beta\_\{k\}, isℱk\{\\cal F\}\_\{k\}\-measurable, \([52](https://arxiv.org/html/2610.33477#A13.E52)\) and this variation bound show that the AdaNorm step is unnormalized withβ¯\\bar\{\\beta\},ϑ\\vartheta, andVu=\(ϑ−1\)/2V\_\{u\}=\\nicefrac\{\{\(\\vartheta\-1\)\}\}\{\{2\}\}, such thatΛ=2​ϑ​Δ1\+\(ϑ−1\)​Δmax\\Lambda=2\\sqrt\{\\vartheta\}\\,\\Delta\_\{1\}\+\(\\vartheta\-1\)\\Delta\_\{\\max\}, and Theorem[12](https://arxiv.org/html/2610.33477#Thmtheorem12)and‖∇J​\(Wk\)‖∗≤r1/2​‖∇J​\(Wk\)‖F\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\*\}\\leq r^\{1/2\}\\\|\\nabla J\(W\_\{k\}\)\\\|\_\{\\scriptscriptstyle\\textrm\{F\}\}yield the nuclear form of \([27](https://arxiv.org/html/2610.33477#S4.E27)\)\. Withβ=cn0/2K−1/2\\beta=c\\sqrt\{\\nicefrac\{\{n\_\{0\}\}\}\{\{2\}\}\}\\,K^\{\-1/2\}, we haveβ¯=cK−1/2\\bar\{\\beta\}=cK^\{\-1/2\}, and, for aKK\-independentn0n\_\{0\},ϑ\\varthetaandΛ\\Lambdado not depend onKK, such that the schedule of Theorem[12](https://arxiv.org/html/2610.33477#Thmtheorem12)yields \([34](https://arxiv.org/html/2610.33477#S5.E34)\)\.

## Appendix NConsequences of the Actor Bounds

### N\.1Residual Error

By Lemma[7](https://arxiv.org/html/2610.33477#Thmtheorem7)\(iv\) and the schedule of Theorem[11](https://arxiv.org/html/2610.33477#Thmtheorem11), the floor term of \([26](https://arxiv.org/html/2610.33477#S4.E26)\) is at most

\(1\+χ\)\[ϵdist\+4​Rr​Rπ​τmix\(1−γ\)​T\+c​DF\+γ​Rπ\(1−γ\)​T\(Rvexp\(−CEκ1P1\)\+2Cv1\+4​τmixT\+c​L∗K2/3\+ϵapp\)\]\.\\begin\{split\}\(1\+\\chi\)\\,\\Big\[&\\epsilon\_\{\\rm dist\}\+\\frac\{4R\_\{r\}R\_\{\\pi\}\\tau\_\{\\mathrm\{mix\}\}\}\{\(1\-\\gamma\)T\}\+cD\_\{F\}\\\\ &\+\\frac\{\\gamma R\_\{\\pi\}\}\{\(1\-\\gamma\)T\}\\Big\(R\_\{v\}\\exp\(\-C\_\{E\}\\kappa\_\{1\}^\{P\_\{1\}\}\)\+2C\_\{v\}\\sqrt\{\\frac\{1\+4\\tau\_\{\\mathrm\{mix\}\}\}\{T\}\}\+\\frac\{cL\_\{\*\}\}\{K^\{2/3\}\}\+\\epsilon\_\{\\rm app\}\\Big\)\\Big\]\.\\end\{split\}\(53\)
Note that \([53](https://arxiv.org/html/2610.33477#A14.E53)\) separates the stationary\-to\-discounted mismatch, the start\-state transient, the drift of the frame bias, and the critic bootstrap\. Within the bootstrap term, the NSI, sampling, and approximation errors are attenuated byγ​Rπ/\[\(1−γ\)​T\]\\nicefrac\{\{\\gamma R\_\{\\pi\}\}\}\{\{\[\(1\-\\gamma\)T\]\}\}, whereas the policy drift of the lagged critic belongs to the transient\. In addition to the1/T\\nicefrac\{\{1\}\}\{\{T\}\}decay in Remark[14](https://arxiv.org/html/2610.33477#Thmtheorem14), the sampling part of the critic term decays asT−3/2T^\{\-3/2\}for fixedλ\>0\\lambda\>0andcc\.

*Mismatch despite one\-step mixing\.*Let𝒮=𝒜=\{0,1\}\{\\cal S\}=\{\\cal A\}=\\\{0,1\\\},st\+1=ats\_\{t\+1\}=a\_\{t\},ρ=δ0\\rho=\\delta\_\{0\},r⁡\(s,a\)=\(s−c¯\)​ar\(s,a\)=\(s\-\\bar\{c\}\)a, andπw​\(1\|s\)=p:=\(1\+e−w\)−1\\pi\_\{w\}\(1\|s\)=p:=\(1\+e^\{\-w\}\)^\{\-1\}\. Although the chain mixes in one step, we have

J⁡\(w\)=γ​p2−c¯​p,J′​\(w\)=p⁡\(1−p\)​\(2​γ​p−c¯\),g⁡\(w\)=p⁡\(1−p\)​\(\(1\+γ\)​p−c¯\)\.J\(w\)=\\gamma p^\{2\}\-\\bar\{c\}p,\\qquad J^\{\\prime\}\(w\)=p\(1\-p\)\(2\\gamma p\-\\bar\{c\}\),\\qquad g\(w\)=p\(1\-p\)\(\(1\+\\gamma\)p\-\\bar\{c\}\)\.
Atγ=1/2\\gamma=1/2,c¯=3/4\\bar\{c\}=3/4, andp=1/2p=1/2, the stationary mean satisfiesg⁡\(w\)=0g\(w\)=0, whereasJ′\(w\)=−1/16J^\{\\prime\}\(w\)=\-1/16\. Therefore, fast mixing alone does not remove the stationary\-to\-discounted mismatch\.

### N\.2Frame Length and Transition Complexity

Reducing the𝒪⁡\(1/T\)\{\\cal O\}\(\\nicefrac\{\{1\}\}\{\{T\}\}\)part of the floor requires a largerTT, which also enlarges the correction constantCFC\_\{F\}\. For everyc\>0c\>0, the inequality of arithmetic and geometric means givesΓ≥2​\[2​χ​\(1\+χ\)​κ†​Δ1​CF\]1/2\\Gamma\\geq 2\\big\[\\sqrt\{2\}\\chi\(1\+\\chi\)\\kappa\_\{\\dagger\}\\Delta\_\{1\}C\_\{F\}\\big\]^\{1/2\}\. SinceCF≥\(exp⁡\(1\)−1\)​RA​Rπ2​TC\_\{F\}\\geq\(\\exp\(1\)\-1\)R\_\{A\}R\_\{\\pi\}^\{2\}T, the sufficient horizon\(Γ/ϵ\)3\(\\nicefrac\{\{\\Gamma\}\}\{\{\\epsilon\}\}\)^\{3\}of Corollary[13](https://arxiv.org/html/2610.33477#Thmtheorem13)\(i\) grows at least asT3/2T^\{3/2\}whenΔ1\>0\\Delta\_\{1\}\>0\. Note that such growth is a property of the sufficient bound rather than a lower bound on the number of updates required by Algorithm[1](https://arxiv.org/html/2610.33477#alg1)\.

For fixedλ\\lambdaand problem constants, we consider a targetϵ∈\(0,1\]\\epsilon\\in\(0,1\]for the part of \([25](https://arxiv.org/html/2610.33477#S4.E25)\) other than\(1\+χ\)​ϵdist\(1\+\\chi\)\\epsilon\_\{\\rm dist\}\. The𝒪⁡\(1/T\)\{\\cal O\}\(\\nicefrac\{\{1\}\}\{\{T\}\}\)part of the floor is of orderϵ\\epsilonforTTof order1/ϵ\\nicefrac\{\{1\}\}\{\{\\epsilon\}\}\. Moreover, in \([25](https://arxiv.org/html/2610.33477#S4.E25)\), the choices1−η1\-\\etaof orderϵ2/σ¯2\\nicefrac\{\{\\epsilon^\{2\}\}\}\{\{\\bar\{\\sigma\}^\{2\}\}\},β\\betaof orderϵ​\(1−η\)1/2/CF\\nicefrac\{\{\\epsilon\(1\-\\eta\)^\{1/2\}\}\}\{\{C\_\{F\}\}\}, andKKof order1/\(ϵ​β\)\\nicefrac\{\{1\}\}\{\{\(\\epsilon\\beta\)\}\}, i\.e\., of orderσ¯​CF/ϵ3\\nicefrac\{\{\\bar\{\\sigma\}C\_\{F\}\}\}\{\{\\epsilon^\{3\}\}\}, make every other term of orderϵ\\epsilon, including the driftη​DF​β/\(1−η\)\\nicefrac\{\{\\eta D\_\{F\}\\beta\}\}\{\{\(1\-\\eta\)\}\}that is of orderDF​σ¯/CF\\nicefrac\{\{D\_\{F\}\\bar\{\\sigma\}\}\}\{\{C\_\{F\}\}\}, and satisfyT​Rπ​β≤1TR\_\{\\pi\}\\beta\\leq 1for smallϵ\\epsilon\. Note that these choices are the schedule1−η∝K−2/31\-\\eta\\propto K^\{\-2/3\}andβ∝K−2/3\\beta\\propto K^\{\-2/3\}with constants that depend onσ¯\\bar\{\\sigma\}andCFC\_\{F\}\. Withσ¯=𝒪\(T−1/2\)\\bar\{\\sigma\}=\{\\cal O\}\(T^\{\-1/2\}\)andCF=𝒪⁡\(T\)C\_\{F\}=\{\\cal O\}\(T\), these choices give𝒪\(T1/2ϵ−3\)=𝒪\(ϵ−7/2\)\{\\cal O\}\(T^\{1/2\}\\epsilon^\{\-3\}\)=\{\\cal O\}\(\\epsilon^\{\-7/2\}\)updates and, atTTtransitions per update,𝒪\(ϵ−9/2\)\{\\cal O\}\(\\epsilon^\{\-9/2\}\)transitions\. With the fixed proportion1−η=K−2/31\-\\eta=K^\{\-2/3\}of Theorem[11](https://arxiv.org/html/2610.33477#Thmtheorem11)and the stepsize constantccof orderT−1/2T^\{\-1/2\}that minimizesΓ\\Gammaas shown above, the horizon is of orderT3/2​ϵ−3T^\{3/2\}\\epsilon^\{\-3\}, i\.e\.,𝒪\(ϵ−9/2\)\{\\cal O\}\(\\epsilon^\{\-9/2\}\)updates and𝒪\(ϵ−11/2\)\{\\cal O\}\(\\epsilon^\{\-11/2\}\)transitions\. Without momentum, i\.e\.,η=0\\eta=0, \([25](https://arxiv.org/html/2610.33477#S4.E25)\) needs neither the importance correction nor the step condition, and contains the noise term2\(1\+χ\)κ†σ¯=𝒪\(T−1/2\)\\sqrt\{2\}\(1\+\\chi\)\\kappa\_\{\\dagger\}\\bar\{\\sigma\}=\{\\cal O\}\(T^\{\-1/2\}\)instead of the correction noise and the drift; hence,TTof orderϵ−2\\epsilon^\{\-2\},β\\betaof orderϵ\\epsilon, andKKof orderϵ−2\\epsilon^\{\-2\}give𝒪⁡\(ϵ−2\)\{\\cal O\}\(\\epsilon^\{\-2\}\)updates and𝒪⁡\(ϵ−4\)\{\\cal O\}\(\\epsilon^\{\-4\}\)transitions\. Therefore, the ICM pays for the growth ofCFC\_\{F\}withTTin transition complexity, whereas the comparison at a fixed frame length concerns the reducible part of the floor, which decreases from𝒪\(T−1/2\)\{\\cal O\}\(T^\{\-1/2\}\)forη=0\\eta=0to𝒪⁡\(1/T\)\{\\cal O\}\(\\nicefrac\{\{1\}\}\{\{T\}\}\)\.

## References

- Agarwal et al\. \(2021\)A\. Agarwal, S\. M\. Kakade, J\. D\. Lee, and G\. Mahajan\.On the theory of policy gradient methods: Optimality, approximation, and distribution shift\.*Journal of Machine Learning Research*, 22\(98\):1–76, 2021\.
- Ajalloeian and Stich \(2020\)A\. Ajalloeian and S\. U\. Stich\.On the convergence of SGD with biased gradients\.arXiv:2008\.00051, 2020\.
- Barakat et al\. \(2023\)A\. Barakat, I\. Fatkhullin, and N\. He\.Reinforcement learning with general utilities: Simpler variance reduction and large state\-action space\.In*International Conference on Machine Learning*, volume 202, 2023\.
- Bedi et al\. \(2022\)A\. S\. Bedi, S\. Chakraborty, A\. Parayil, B\. M\. Sadler, P\. Tokekar, and A\. Koppel\.On the hidden biases of policy mirror ascent in continuous action spaces\.In*International Conference on Machine Learning*, pages 1716–1731, Baltimore, MD, USA, 2022\.
- Bhandari et al\. \(2021\)J\. Bhandari, D\. Russo, and R\. Singal\.A finite\-time analysis of temporal difference learning with linear function approximation\.*Operations Research*, 69\(3\):950–973, 2021\.doi:10\.1287/opre\.2020\.2024\.
- Boyd and Vandenberghe \(2018\)S\. Boyd and L\. Vandenberghe\.*Introduction to Applied Linear Algebra: Vectors, Matrices, and Least Squares*\.Cambridge University Press, 2018\.ISBN 9781316518960\.doi:10\.1017/9781108583664\.
- Bradtke and Barto \(1996\)S\. J\. Bradtke and A\. G\. Barto\.Linear least\-squares algorithms for temporal difference learning\.*Machine Learning*, 22\(1–3\):33–57, 1996\.
- Chang et al\. \(2025\)D\. Chang, Y\. Liu, H\. Zhou, and G\. Yuan\.On the convergence of Muon and beyond\.arXiv:2509\.15816, 2025\.
- Chen and Zhao \(2023\)X\. Chen and L\. Zhao\.Finite\-time analysis of single\-timescale actor\-critic\.In*Advances in Neural Information Processing Systems*, volume 36, pages 7017–7049, New Orleans, LA, USA, 2023\.
- Chen et al\. \(2024\)Y\.\-J\. Chen, N\.\-C\. Huang, C\.\-P\. Lee, and P\.\-C\. Hsieh\.Accelerated policy gradient: on the convergence rates of the nesterov momentum for reinforcement learning\.In*International Conference on Machine Learning*, pages 6881–6949, Vienna, Austria, 2024\.
- Cutkosky and Mehta \(2020\)A\. Cutkosky and H\. Mehta\.Momentum improves normalized SGD\.In*International Conference on Machine Learning*, volume 119, pages 2260–2268, 2020\.
- Cutkosky and Orabona \(2019\)A\. Cutkosky and F\. Orabona\.Momentum\-based variance reduction in non\-convex SGD\.In*Advances in Neural Information Processing Systems*, volume 32, 2019\.
- Demidovich et al\. \(2023\)Y\. Demidovich, G\. Malinovsky, I\. Sokolov, and P\. Richtárik\.A guide through the zoo of biased SGD\.In*Advances in Neural Information Processing Systems*, volume 36, 2023\.
- Dong \(Oct\. 2026\)Y\. Dong\.Notes on Uniform Policy Score Bounds and Lipschitz Continuity for Truncated Gaussian, Product Cauchy, Boltzmann, and Softmax Policies\.*arXiv preprint arXiv:2610\.15690*, Oct\. 2026\.
- Dong et al\. \(2024\)Y\. Dong, H\. Zhang, G\. Wang, S\. Cui, and X\. Hu\.Heavy\-ball momentum accelerated actor\-critic with function approximation\.*arXiv preprint arXiv:2408\.06945*, 2024\.
- Dorfman and Levy \(2022\)R\. Dorfman and K\. Y\. Levy\.Adapting to mixing time in stochastic optimization with Markovian data\.In*International Conference on Machine Learning*, pages 5429–5446, Baltimore, MD, USA, 2022\.
- Fan et al\. \(2026\)C\. Fan, G\. Liu, M\. Hong, R\. R\. Kompella, and S\. Liu\.Rethinking Muon beyond pretraining: Spectral failures and high\-pass remedies for VLA and RLVR\.arXiv:2605\.19282, 2026\.
- Ganesh et al\. \(2025\)S\. Ganesh, W\. U\. Mondal, and V\. Aggarwal\.A sharper global convergence analysis for average reward reinforcement learning via an actor\-critic approach\.In*International Conference on Machine Learning*, Vancouver, BC, Canada, 2025\.
- Hong et al\. \(2023\)M\. Hong, H\.\-T\. Wai, Z\. Wang, and Z\. Yang\.A two\-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor\-critic\.*SIAM Journal on Optimization*, 33\(1\):147–180, 2023\.
- Hu et al\. \(2020\)Y\. Hu, S\. Zhang, X\. Chen, and N\. He\.Biased stochastic first\-order methods for conditional stochastic optimization and applications in meta learning\.In*Advances in Neural Information Processing Systems*, volume 33, 2020\.
- Huang et al\. \(2020\)F\. Huang, S\. Gao, J\. Pei, and H\. Huang\.Momentum\-based policy gradient methods\.In*International Conference on Machine Learning*, volume 119, pages 4422–4433, Vienna, Austria, 2020\.
- Jordan et al\. \(2024\)K\. Jordan, Y\. Jin, V\. Boza, J\. You, F\. Cesista, L\. Newhouse, and J\. Bernstein\.Muon: An optimizer for hidden layers in neural networks\.[https://kellerjordan\.github\.io/posts/muon/](https://kellerjordan.github.io/posts/muon/), 2024\.
- Kakade \(2001\)S\. M\. Kakade\.A natural policy gradient\.In*Advances in Neural Information Processing Systems*, pages 1531–1538, Vancouver, BC, Canada, 2001\.
- Karimi et al\. \(2019\)B\. Karimi, B\. Miasojedow, E\. Moulines, and H\.\-T\. Wai\.Non\-asymptotic analysis of biased stochastic approximation scheme\.In*Conference on Learning Theory*, volume 99, pages 1944–1974, 2019\.
- Kelley \(1995\)C\. T\. Kelley\.*Iterative Methods for Linear and Nonlinear Equations*, volume 16 of*Frontiers in Applied Mathematics*\.Society for Industrial and Applied Mathematics, Philadelphia, PA, 1995\.
- Khanduri et al\. \(2021\)P\. Khanduri, S\. Zeng, M\. Hong, H\.\-T\. Wai, Z\. Wang, and Z\. Yang\.A near\-optimal algorithm for stochastic bilevel optimization via double\-momentum\.In*Advances in Neural Information Processing Systems*, volume 34, 2021\.
- Khodadadian et al\. \(2023\)S\. Khodadadian, T\. T\. Doan, J\. Romberg, and S\. T\. Maguluri\.Finite\-sample analysis of two\-time\-scale natural actor–critic algorithm\.*IEEE Transactions on Automatic Control*, 68\(6\):3273–3284, 2023\.
- Kim and Oh \(2026\)G\. Y\. Kim and M\.\-H\. Oh\.Convergence of muon with Newton\-Schulz\.In*International Conference on Learning Representations*, Rio de Janeiro, Brazil, 2026\.
- Konda and Borkar \(1999\)V\. R\. Konda and V\. S\. Borkar\.Actor\-critic–type learning algorithms for markov decision processes\.*SIAM Journal on Control and Optimization*, 38\(1\):94–123, 1999\.
- Konda and Tsitsiklis \(1999\)V\. R\. Konda and J\. N\. Tsitsiklis\.Actor\-critic algorithms\.In*Advances in Neural Information Processing Systems*, volume 12, Denver, CO, USA, 1999\.
- Kovalev \(2025\)D\. Kovalev\.Understanding gradient orthogonalization for deep learning via non\-Euclidean trust\-region optimization\.arXiv:2503\.12645, 2025\.
- Kumar et al\. \(2023\)H\. Kumar, A\. Koppel, and A\. Ribeiro\.On the sample complexity of actor\-critic method for reinforcement learning with function approximation\.*Machine Learning*, 112\(7\):2433–2467, 2023\.
- Kumar et al\. \(2026\)N\. Kumar, T\. Dahan, L\. Cohen, A\. Barua, G\. Ramponi, K\. Y\. Levy, and S\. Mannor\.Optimal sample complexity for single time\-scale actor\-critic with momentum\.arXiv:2602\.01505, 2026\.
- Lagoudakis and Parr \(2003\)M\. G\. Lagoudakis and R\. Parr\.Least\-squares policy iteration\.*Journal on Machine Learning Research*, 4:1107–1149, Dec\. 2003\.
- Lan \(2023\)G\. Lan\.Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes\.*Mathematical programming*, 198\(1\):1059–1106, 2023\.
- Lazaric et al\. \(2012\)A\. Lazaric, M\. Ghavamzadeh, and R\. Munos\.Finite\-sample analysis of least\-squares policy iteration\.*Journal of Machine Learning Research*, 13\(98\):3041–3074, 2012\.
- Levin and Peres \(2017\)D\. A\. Levin and Y\. Peres\.*Markov Chains and Mixing Times*, volume 107\.American Mathematical Soc\., 2017\.
- Lion et al\. \(2026\)K\. Lion, F\. Hübler, B\. Li, A\. Orvieto, and N\. He\.Muown: Row\-norm control for Muon optimization\.*arXiv preprint arXiv:2605\.10797*, 2026\.
- Mitrophanov \(2005\)A\. Y\. Mitrophanov\.Sensitivity and convergence of uniformly ergodic markov chains\.*Journal of Applied Probability*, 42\(4\):1003–1014, 2005\.
- Nesterov \(2018\)Y\. Nesterov\.*Lectures on Convex Optimization \(second edition\)*, volume 137\.Springer, Cham, Switzerland, 2018\.
- Nota and Thomas \(2020\)C\. Nota and P\. S\. Thomas\.Is the policy gradient a gradient?In*International Conference on Autonomous Agents and Multiagent Systems*, 2020\.
- Olshevsky and Gharesifard \(2023\)A\. Olshevsky and B\. Gharesifard\.A small gain analysis of single timescale actor critic\.*SIAM Journal on Control and Optimization*, 61\(2\):980–1007, 2023\.
- Pan and Schreiber \(1991\)V\. Pan and R\. Schreiber\.An improved Newton iteration for the generalized inverse of a matrix, with applications\.*SIAM Journal on Scientific and Statistical Computing*, 12\(5\):1109–1130, 1991\.doi:10\.1137/0912058\.
- Patel et al\. \(2024\)B\. Patel, W\. A\. Suttle, A\. Koppel, V\. Aggarwal, B\. M\. Sadler, D\. Manocha, and A\. S\. Bedi\.Towards global optimality for practical average reward reinforcement learning without mixing time oracles\.In*International Conference on Machine Learning*, pages 39889–39907, Vienna, Austria, 2024\.
- Peters and Schaal \(2008\)J\. Peters and S\. Schaal\.Natural actor\-critic\.*Neurocomputing*, 71\(7\):1180–1190, 2008\.
- Pethick et al\. \(2025\)T\. Pethick, W\. Xie, K\. Antonakopoulos, Z\. Zhu, A\. Silveti\-Falls, and V\. Cevher\.Training deep learning models with norm\-constrained LMOs\.In*International Conference on Machine Learning*, volume 267, pages 49069–49104, 2025\.
- Precup et al\. \(2000\)D\. Precup, R\. S\. Sutton, and S\. Singh\.Eligibility traces for off\-policy policy evaluation\.In*International Conference on Machine Learning*, pages 759–766, 2000\.
- Qiu et al\. \(2021\)S\. Qiu, Z\. Yang, J\. Ye, and Z\. Wang\.On finite\-time convergence of actor\-critic algorithm\.*IEEE Journal on Selected Areas in Information Theory*, 2\(2\):652–664, 2021\.
- Riabinin et al\. \(2026\)A\. Riabinin, E\. Shulgin, K\. Gruntkowska, and P\. Richtárik\.From muon to gluon: Bridging theory and practice of LMO\-based optimizers for LLMs\.In*International Conference on Machine Learning*, 2026\.
- Ruan et al\. \(2026\)K\. Ruan, J\. Lin, Z\. Huang, Z\. Zhou, Q\. Wei, X\. Wang, and H\. Sun\.When does Muon help agentic reinforcement learning?arXiv:2607\.16169, 2026\.
- Schulman et al\. \(2015\)J\. Schulman, S\. Levine, P\. Abbeel, M\. I\. Jordan, and P\. Moritz\.Trust region policy optimization\.In*International Conference on Machine Learning*, volume 37, pages 1889–1897, Lille, France, 2015\.
- Schulman et al\. \(2017\)J\. Schulman et al\.Proximal policy optimization algorithms\.*arXiv preprint arXiv:1707\.06347*, 2017\.
- Shen and Chen \(2022\)H\. Shen and T\. Chen\.A single\-timescale analysis for stochastic approximation with multiple coupled sequences\.In*Advances in Neural Information Processing Systems*, volume 35, pages 17415–17429, New Orleans, LA, USA, 2022\.
- Shen et al\. \(2025\)W\. Shen, R\. Huang, M\. Huang, C\. Shen, and J\. Zhang\.On the convergence analysis of Muon\.arXiv:2505\.23737, 2025\.
- Sun et al\. \(2022\)T\. Sun, H\. Shen, T\. Chen, and D\. Li\.Adaptive temporal difference learning with linear function approximation\.*IEEE Transactions on Pattern Analysis and Machine Intelligence*, 44\(12\):8812–8824, 2022\.
- Suttle et al\. \(2023\)W\. A\. Suttle, A\. S\. Bedi, B\. Patel, B\. M\. Sadler, A\. Koppel, and D\. Manocha\.Beyond exponentially fast mixing in average\-reward reinforcement learning via multi\-level monte carlo actor\-critic\.In*International Conference on Machine Learning*, pages 33240–33267, Honolulu, Hawaii, USA, 2023\.
- Sutton and Barto \(2018\)R\. S\. Sutton and A\. G\. Barto\.*Reinforcement learning: An introduction \(second edition\)*\.The MIT Press, Cambridge, MA, USA, 2018\.
- Sutton et al\. \(1999\)R\. S\. Sutton, D\. McAllester, S\. Singh, and Y\. Mansour\.Policy gradient methods for reinforcement learning with function approximation\.In*Advances in Neural Information Processing Systems*, volume 12, Denver, CO, USA, 1999\.
- Thomas \(2014\)P\. Thomas\.Bias in natural actor\-critic algorithms\.In*International Conference on Machine Learning*, volume 32, pages 441–448, 2014\.
- Tsitsiklis and Roy \(1997\)J\. N\. Tsitsiklis and B\. V\. Roy\.An analysis of temporal\-difference learning with function approximation\.*IEEE Transactions on Automatic Control*, 42\(5\):674–690, 1997\.doi:10\.1109/9\.580874\.
- Wang et al\. \(2020\)L\. Wang, Q\. Cai, Z\. Yang, and Z\. Wang\.Neural policy gradient methods: Global optimality and rates of convergence\.In*International Conference on Learning Representations*, Addis Ababa, Ethiopia, 2020\.
- Wu et al\. \(2017\)Y\. Wu, E\. Mansimov, S\. Liao, R\. Grosse, and J\. Ba\.Scalable trust\-region method for deep reinforcement learning using Kronecker\-factored approximation\.In*Advances in Neural Information Processing Systems*, volume 30, 2017\.
- Wu et al\. \(2020\)Y\. F\. Wu, W\. Zhang, P\. Xu, and Q\. Gu\.A finite\-time analysis of two time\-scale actor\-critic methods\.In*Advances in Neural Information Processing Systems*, volume 33, pages 17617–17628, Virtual, 2020\.
- Xiong et al\. \(2021\)H\. Xiong, T\. Xu, Y\. Liang, and W\. Zhang\.Non\-asymptotic convergence of adam\-type reinforcement learning algorithms under markovian sampling\.In*AAAI Conference on Artificial Intelligence*, volume 35, pages 10460–10468, May 2021\.
- Xu et al\. \(2020a\)T\. Xu, Z\. Wang, and Y\. Liang\.Improving sample complexity bounds for \(natural\) actor\-critic algorithms\.In*Advances in Neural Information Processing Systems*, volume 33, pages 4358–4369, Vancouver, BC, Canada, 2020a\.
- Xu et al\. \(2020b\)T\. Xu, Z\. Wang, and Y\. Liang\.Non\-asymptotic convergence analysis of two time\-scale \(natural\) actor\-critic algorithms\.*arXiv preprint arXiv:2005\.03557*, 2020b\.
- Yang et al\. \(2022\)L\. Yang, Y\. Zhang, G\. Zheng, Q\. Zheng, P\. Li, J\. Huang, and G\. Pan\.Policy optimization with stochastic mirror descent\.In*AAAI Conference on Artificial Intelligence*, volume 36, pages 8823–8831, Arlington, VA, USA, 2022\.
- Zeng and Doan \(2024\)S\. Zeng and T\. T\. Doan\.Fast two\-time\-scale stochastic gradient method with applications in reinforcement learning\.In*Conference on Learning Theory*, volume 196, pages 1–47, Edmonton, AB, Canada, 2024\.
- Zeng et al\. \(2024\)S\. Zeng, T\. T\. Doan, and J\. Romberg\.A two\-time\-scale stochastic optimization framework with applications in control and reinforcement learning\.*SIAM Journal on Optimization*, 34\(1\):946–976, 2024\.
- Zhang et al\. \(2020a\)K\. Zhang, A\. Koppel, H\. Zhu, and T\. Başar\.Global convergence of policy gradient methods to \(almost\) locally optimal policies\.*SIAM Journal on Control and Optimization*, 58\(6\):3586–3612, 2020a\.
- Zhang et al\. \(2021\)K\. Zhang, Z\. Yang, H\. Liu, T\. Zhang, and T\. Başar\.Finite\-sample analysis for decentralized batch multiagent reinforcement learning with networked agents\.*IEEE Transactions on Automatic Control*, 66\(12\):5925–5940, 2021\.
- Zhang et al\. \(2020b\)S\. Zhang, B\. Liu, H\. Yao, and S\. Whiteson\.Provably convergent two\-timescale off\-policy actor\-critic with function approximation\.In*International Conference on Machine Learning*, volume 119, pages 11204–11213, Vienna, Austria, 2020b\.
- Zhang et al\. \(2022\)S\. Zhang, R\. Laroche, H\. van Seijen, S\. Whiteson, and R\. Tachet des Combes\.A deeper look at discounting mismatch in actor\-critic algorithms\.In*International Conference on Autonomous Agents and Multiagent Systems*, pages 1491–1499, 2022\.
- Zou et al\. \(2019\)S\. Zou, T\. Xu, and Y\. Liang\.Finite\-sample analysis for SARSA with linear function approximation\.In*Advances in Neural Information Processing Systems*, volume 32, Vancouver, BC, Canada, 2019\.