Refined Analysis of Entropy-Regularized Actor-Critic

# Refined Analysis of Entropy-Regularized Actor-Critic
Source: [https://arxiv.org/html/2605.24357](https://arxiv.org/html/2605.24357)
###### Abstract

In this paper, we study the role of the critic in actor–critic for entropy\-regularized, finite, discounted environments\. We establish that, when the critic is exact, using the latter as a baseline is a variance\-reduction method in a strong sense\. In this case, actor–critic with stochastic gradients matches the sample complexity of deterministic policy gradient, reaching anϵ\\epsilon\-optimal regularized value withO~​\(log⁡\(1/ϵ\)\)\\tilde\{O\}\(\\log\(1/\\epsilon\)\)samples\. In practice, the critic is learned alongside the actor: the variance of the actor update is then influenced by the critic’s variance and bias\. Specifically, when the critic has a sufficiently small error, the variance reduction and rapid convergence are preserved\. This suggests to learn the critic first, keeping it up to date after each actor update, underscoring the crucial role of accurate critic estimation in actor–critic methods\.

Actor\-Critic, Sample Complexity, Entropy, Reinforcement Learning, ICML

## 1Introduction

Policy gradient methods are among the most widely used reinforcement learning \(RL\) algorithms\(Williams,[1992](https://arxiv.org/html/2605.24357#bib.bib31); Suttonet al\.,[1998](https://arxiv.org/html/2605.24357#bib.bib23)\)\. Due to their inherent flexibility and scalability, they have emerged as the dominant approach in modern RL\(Agarwalet al\.,[2021](https://arxiv.org/html/2605.24357#bib.bib21)\)\. Their widespread adoption has led to the development of numerous techniques and tricks to stabilize training and accelerate convergence, enabling faster discovery of high\-performing policies\.

A central technique in policy gradient methods is to introduce a*baseline*, which serves as a control variate that aims to reduce the variance of the gradient estimates\(Konda and Tsitsiklis,[1999](https://arxiv.org/html/2605.24357#bib.bib34)\)\. This comes from the observation that, when expressing the gradient, any term that does not change the expected gradient direction can be used to stabilize optimization\. In practice, there is a strong consensus on the fact that using the value function of the current policy strongly enhances performance\(Grondmanet al\.,[2012](https://arxiv.org/html/2605.24357#bib.bib32); Schulmanet al\.,[2015b](https://arxiv.org/html/2605.24357#bib.bib51)\)\. This effectively shifts the gradient’s focus from ’rewards’ to the so\-called ’advantage’ of one action over another\(Baird and Leemon,[1993](https://arxiv.org/html/2605.24357#bib.bib50)\), thereby providing a denser learning signal\. This observation is further supported by the dominance of*actor\-critic*methods \(AC,Bartoet al\.[1983](https://arxiv.org/html/2605.24357#bib.bib45); Konda and Tsitsiklis[1999](https://arxiv.org/html/2605.24357#bib.bib34)\), and most specifically algorithms like A2C\(Mnihet al\.,[2016](https://arxiv.org/html/2605.24357#bib.bib35)\), PPO\(Schulmanet al\.,[2017](https://arxiv.org/html/2605.24357#bib.bib36)\), TRPO\(Schulmanet al\.,[2015a](https://arxiv.org/html/2605.24357#bib.bib37)\), among many others\. Still, although this shift feels like a natural progression, the specific role of the value function as a stabilizer is poorly understood theoretically\.

Recently, the theory of RL has seen a surge of novel theoretical analyses, providing rigorous foundations for many fundamental methods\. While earlier work relied on asymptotic guarantees\(Williams,[1992](https://arxiv.org/html/2605.24357#bib.bib31); Greensmithet al\.,[2004](https://arxiv.org/html/2605.24357#bib.bib39)\), recent studies established non\-asymptotic, global convergence rates for policy gradient methods\(Meiet al\.,[2020b](https://arxiv.org/html/2605.24357#bib.bib3); Xiao,[2022](https://arxiv.org/html/2605.24357#bib.bib38); Labbiet al\.,[2026b](https://arxiv.org/html/2605.24357#bib.bib16)\)\. More recently, theory has been extended to AC, proving global convergence in finite\-time\(Kumaret al\.,[2024](https://arxiv.org/html/2605.24357#bib.bib1)\), following analyses of policy gradient by relying on a uniform bound on the gradients\.

Parallel to these theoretical advances, it has recently been observed empirically that improving the critic considerably improves the convergence of AC\(Wanget al\.,[2025](https://arxiv.org/html/2605.24357#bib.bib44)\)\. While this gives strong evidence that AC reduces gradient variance, it is not clear to what extent\. Specifically, we can distinguish between two types of variance reduction:

- •weak variance reduction: the variance of the updates is reduced by a multiplicative constant, akin to tail averaging\(Polyak and Juditsky,[1992](https://arxiv.org/html/2605.24357#bib.bib41)\)or moving average methods\(Morales\-Brotonset al\.,[2024](https://arxiv.org/html/2605.24357#bib.bib40)\);
- •strong variance reduction: the variance of the gradient estimator vanishes as iterates approach the optimum, giving linear convergence, like SVRG\(Johnson and Zhang,[2013](https://arxiv.org/html/2605.24357#bib.bib42)\), SAGA\(Defazioet al\.,[2014](https://arxiv.org/html/2605.24357#bib.bib43)\), and other methods\.

Naturally, strong variance\-reduction results in much faster convergence rates than its weak counterpart\. To our knowledge, it remains unknown whether AC methods achieve*weak*or*strong*variance reduction\.

Table 1:Comparison with related actor\-critic methods with unknown critic\. Our method is the first to achieve𝒪~​\(1/ϵ\)\\tilde\{\\mathcal\{O\}\}\(1/\\epsilon\)sample complexity\.- \(1\)\(1\)The results here are provided for the entropy\-regularized problem\.

In this paper, we answer positively:*yes, AC achieves strong variance reduction*, at least when the critic is known\. Our theory follows from the observation that, when the value function \(i\.e\., the critic\) is perfectly known, the variance of AC’s stochastic gradient can be bounded by the norm of the deterministic gradient up to a multiplicative factor\. In this case, we show that AC achieves𝒪​\(log⁡\(1/ϵ\)\)\\mathcal\{O\}\(\\log\(1/\\epsilon\)\)sample complexity, matching the deterministic iteration complexity\. Alternatively, if the critic is learned online, we establish that AC’s behavior is governed by the critic’s bias and variance\. Specifically, we show that if sufficiently many critic iterations are performed, AC attainsO~​\(1/ϵ\)\\tilde\{O\}\(1/\\epsilon\)sample complexity\. Our contributions can be summarized as follows:

- •We propose a novel theoretical analysis of actor\-critic\. To the best of our knowledge, we provide the first proof that AC with an exact critic*acts as a strong variance\-reduction method*, not merely reducing the residual variance of the updates, but entirely eliminating it\.
- •When the critic is inexact, it must be learned alongside the actor\. In that case, the critic’s bias and variance induce similar bias and variance on the actor\. Consequently, when properly setting up the algorithm, one can learn the actor at a rate similar to the rate of learning the critic: in some sense, the most important part of the training is the critic, and it pays off to spend time learning it\. Crucially, one needs to perform multiple updates of the critic in between actor updates, confirming recent empirical findings\(Wanget al\.,[2025](https://arxiv.org/html/2605.24357#bib.bib44)\)\.
- •We empirically confirm our findings on two environments, showing that the performance of the learned policy monotonically increases as the number of updates of the critic between consecutive actor updates increases\.

We give a comparison with the closest works in[Table1](https://arxiv.org/html/2605.24357#S1.T1), and discuss related work in[Section2](https://arxiv.org/html/2605.24357#S2)\.[Section3](https://arxiv.org/html/2605.24357#S3)introduces the background\. We analyze AC with exact critic in[Section4](https://arxiv.org/html/2605.24357#S4), and with inexact critic in[Section5](https://arxiv.org/html/2605.24357#S5)\. Empirical study is in[Section6](https://arxiv.org/html/2605.24357#S6), and we discuss perspectives in[Section7](https://arxiv.org/html/2605.24357#S7)\.

## 2Related Works

#### Entropy\-Regularized RL\.

Entropy\-regularized RL promotes stochastic policies by rewarding higher entropy, encouraging exploration, and improving stability in learning\(Williams and Peng,[1991](https://arxiv.org/html/2605.24357#bib.bib27); Mnihet al\.,[2016](https://arxiv.org/html/2605.24357#bib.bib35); Neuet al\.,[2017](https://arxiv.org/html/2605.24357#bib.bib59); Haarnojaet al\.,[2018](https://arxiv.org/html/2605.24357#bib.bib28)\)\. It underpins widely used deep RL methods such as Soft Actor–Critic and entropy\-regularized AC\(Haarnojaet al\.,[2018](https://arxiv.org/html/2605.24357#bib.bib28)\), which remain poorly understood in theory despite their impressive practical performance\. A line of work studies this objective both algorithmically and theoretically\(Nachumet al\.,[2017](https://arxiv.org/html/2605.24357#bib.bib19); Geistet al\.,[2019](https://arxiv.org/html/2605.24357#bib.bib49)\), relating entropy regularization to soft policy iteration and soft Q\-learning, making explicit the links between policy\-gradient and value\-based methods\.Another line of work\(Lan,[2023](https://arxiv.org/html/2605.24357#bib.bib18)\)studies Policy Mirror Descent for entropy\-regularized reinforcement learning and establishes convergence guarantees for the regularized objective\. However, Policy Mirror Descent operates directly in policy space, which makes extensions beyond the tabular setting less straightforward\. In contrast, actor–critic methods update parametrized policies directly, providing a more natural foundation for scalable extensions and motivating a dedicated analysis of actor–critic methods\.

#### Policy Gradient Methods\.

Policy gradient \(PG\) methods date back toWilliams \([1992](https://arxiv.org/html/2605.24357#bib.bib31)\); Suttonet al\.\([1999](https://arxiv.org/html/2605.24357#bib.bib22)\)\.Meiet al\.\([2020b](https://arxiv.org/html/2605.24357#bib.bib3)\); Zhanget al\.\([2020](https://arxiv.org/html/2605.24357#bib.bib24)\); Xiao \([2022](https://arxiv.org/html/2605.24357#bib.bib38)\)established global convergence for softmax policies with*exact*gradients by showing that the RL objective satisfies Łojasiewicz\-type inequalities, obtaining linear convergence rates in entropy\-regularized RL\. With stochastic gradients,Zhanget al\.\([2021a](https://arxiv.org/html/2605.24357#bib.bib25)\); Yuanet al\.\([2022](https://arxiv.org/html/2605.24357#bib.bib30)\)proved convergence to first\-order stationary points under Monte\-Carlo gradient estimates, and subsequent work showed that global guarantees can be recovered under additional structure, notably through regularization\(Zhanget al\.,[2021b](https://arxiv.org/html/2605.24357#bib.bib17); Dinget al\.,[2025](https://arxiv.org/html/2605.24357#bib.bib26); Labbiet al\.,[2026a](https://arxiv.org/html/2605.24357#bib.bib14),[b](https://arxiv.org/html/2605.24357#bib.bib16)\)\.However, vanilla stochastic PG remains brittle in practice due to high variance, motivating the use of more involved methods\. Several recent works establish convergence guarantees for variants of PG, relying for instance on Hessian\-based variance reduction\(Fatkhullinet al\.,[2023](https://arxiv.org/html/2605.24357#bib.bib46)\), momentum and importance sampling\(Barakatet al\.,[2023](https://arxiv.org/html/2605.24357#bib.bib48)\), or inverse\-Fisher preconditioning\(Mondal and Aggarwal,[2024](https://arxiv.org/html/2605.24357#bib.bib47)\)\. The approaches ofFatkhullinet al\.\([2023](https://arxiv.org/html/2605.24357#bib.bib46)\)andMondal and Aggarwal \([2024](https://arxiv.org/html/2605.24357#bib.bib47)\)typically require heavier computations, such as estimating second\-order quantities or inverting Fisher\-type matrices, while the variance\-reduction method ofBarakatet al\.\([2023](https://arxiv.org/html/2605.24357#bib.bib48)\)is less competitive in practice than actor–critic methods\. This motivates our focus on actor–critic methods\(Bartoet al\.,[1983](https://arxiv.org/html/2605.24357#bib.bib45); Konda and Tsitsiklis,[1999](https://arxiv.org/html/2605.24357#bib.bib34)\), which provide a scalable parameter\-space framework while enabling variance reduction through the critic\.

#### Convergence analysis of Actor–Critic\.

Early analyses of AC were asymptotic, using two\-timescale stochastic approximation\(Konda and Tsitsiklis,[1999](https://arxiv.org/html/2605.24357#bib.bib34)\), or ODE\-based arguments\(Bhatnagaret al\.,[2009](https://arxiv.org/html/2605.24357#bib.bib55); Castro and Meir,[2010](https://arxiv.org/html/2605.24357#bib.bib57)\)\. More recently, non\-asymptotic rates have been obtained in special control settings: for LQR,Yanget al\.\([2019](https://arxiv.org/html/2605.24357#bib.bib52)\)proves global linear convergence, but requires𝒪~​\(ϵ−5\)\\tilde\{\\mathcal\{O\}\}\(\\epsilon^\{\-5\}\)critic updates in order to maintain sufficient value\-estimation accuracy\. In the general RL setting, several works establish non\-asymptotic convergence to stationary points with finite\-sample complexity bounds\(Xuet al\.,[2020](https://arxiv.org/html/2605.24357#bib.bib53); Qiuet al\.,[2021](https://arxiv.org/html/2605.24357#bib.bib54); Kumaret al\.,[2023](https://arxiv.org/html/2605.24357#bib.bib56); Olshevsky and Gharesifard,[2023](https://arxiv.org/html/2605.24357#bib.bib10); Chen and Zhao,[2023](https://arxiv.org/html/2605.24357#bib.bib11)\)\. In the unregularized setting, global convergence guarantees are obtained by combining stability arguments with uniform gradient/exploration controls\(Kumaret al\.,[2024](https://arxiv.org/html/2605.24357#bib.bib1); Gauret al\.,[2024](https://arxiv.org/html/2605.24357#bib.bib7)\), achievingO~​\(ϵ−3\)\\tilde\{O\}\(\\epsilon^\{\-3\}\)sample complexity\. For the tabular entropy\-regularized objective,Cayciet al\.\([2024](https://arxiv.org/html/2605.24357#bib.bib12)\)also derives finite\-sample guarantees, albeit with even larger complexityO~​\(ϵ−5\)\\tilde\{O\}\(\\epsilon^\{\-5\}\)\.In continuous\-action settings, recent analyses\(Zorbaet al\.,[2026](https://arxiv.org/html/2605.24357#bib.bib13); Kerimkulovet al\.,[2025](https://arxiv.org/html/2605.24357#bib.bib15)\)establish global convergence guarantees for entropy\-regularized actor–critic methods, but only in deterministic regimes\.In contrast, we analyze entropy\-regularized AC in the stochastic setting and show that it is a*strong variance reduction method*when the critic is exact, reachingO~​\(log⁡\(1/ϵ\)\)\\tilde\{O\}\(\\log\(1/\\epsilon\)\)sample complexity; furthermore, we show that with inexact critic, entropy\-regularized AC still achievesO~​\(ϵ−1\)\\tilde\{O\}\(\\epsilon^\{\-1\}\)sample complexity\.

## 3Background

#### Markov Decision Process\.

We consider a discounted MDPℳ=\(𝒮,𝒜,γ,𝖯,𝗋,ρ\)\\mathcal\{M\}=\(\\mathcal\{S\},\\mathcal\{A\},\\gamma,\\mathsf\{P\},\\mathsf\{r\},\\rho\)with finite state and action spaces𝒮,𝒜\\mathcal\{S\},\\mathcal\{A\}, discount factorγ∈\(0,1\)\\gamma\\in\(0,1\), transition kernel𝖯​\(s′\|s,a\)\\mathsf\{P\}\(s^\{\\prime\}\|s,a\), reward𝗋​\(s,a\)∈\[0,1\]\\mathsf\{r\}\(s,a\)\\in\[0,1\], and initial distributionρ\\rho\. A stationary policyπ:𝒮→𝒫​\(𝒜\)\\pi\\colon\\mathcal\{S\}\\to\\mathcal\{P\}\(\\mathcal\{A\}\)induces𝖯π​\(s′\|s\)​=Δ​∑a𝖯​\(s′\|s,a\)​π​\(a\|s\)\\mathsf\{P\}\_\{\\pi\}\(s^\{\\prime\}\|s\)\\overset\{\\Delta\}\{=\}\\sum\_\{a\}\\mathsf\{P\}\(s^\{\\prime\}\|s,a\)\\pi\(a\|s\)\. The value function is

vπ​\(s\)​=Δ​𝔼sπ​\[∑t=0∞γt​𝗋​\(St,At\)\],\\displaystyle\\mathrm\{v\}\_\{\\hskip 0\.57367pt\\pi\}\(s\)\\overset\{\\Delta\}\{=\}\\textstyle\\mathbb\{E\}\_\{s\}^\{\\pi\}\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\mathsf\{r\}\(S\_\{t\},A\_\{t\}\)\\right\],\(1\)withS0=sS\_\{0\}=s,At∼π\(⋅\|St\)A\_\{t\}\\sim\\pi\(\\cdot\|S\_\{t\}\),St\+1∼𝖯\(⋅\|St,At\)S\_\{t\+1\}\\sim\\mathsf\{P\}\(\\cdot\|S\_\{t\},A\_\{t\}\)\. Forρ∈𝒫​\(𝒮\)\\rho\\in\\mathcal\{P\}\(\\mathcal\{S\}\), definevπ​\(ρ\)​=Δ​∑sρ​\(s\)​vπ​\(s\)\\smash\{\\mathrm\{v\}\_\{\\hskip 0\.57367pt\\pi\}\(\\rho\)\\overset\{\\Delta\}\{=\}\\sum\_\{s\}\\rho\(s\)\\mathrm\{v\}\_\{\\hskip 0\.57367pt\\pi\}\(s\)\}\. For a given policyπ\\pi, we define the occupancy measure

dρπ​\(s\)\\displaystyle d\_\{\\rho\}^\{\\hskip 0\.57367pt\\pi\}\(s\)=Δ​\(1−γ\)​∑t=0∞γt​ρ​𝖯πt​\(s\)\\displaystyle\\overset\{\\Delta\}\{=\}\\textstyle\(1\-\\gamma\)\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\rho\\mathsf\{P\}\_\{\\pi\}^\{t\}\(s\)\(2\)ofπ\\pi, measuring the discounted probability of visiting states along trajectories generated byπ\\pistarting fromρ\\rho\.

#### Entropy\-regularized RL\.

In entropy\-regularized RL, near\-deterministic policies are penalized by modifying the value of a policyπ\\pito

v~πλ​\(s\)\\displaystyle\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\pi\}^\{\\lambda\}\(s\)=Δ​𝔼sπ​\[∑t=0∞γt​𝗋~π​\(St,At\)\],\\displaystyle\\textstyle\\overset\{\\Delta\}\{=\}\\mathbb\{E\}\_\{s\}^\{\\pi\}\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\tilde\{\\mathsf\{r\}\}\_\{\\pi\}\(S\_\{t\},A\_\{t\}\)\\right\]\\kern 5\.0pt,\(3\)𝗋~π​\(s,a\)\\displaystyle\\tilde\{\\mathsf\{r\}\}\_\{\\pi\}\(s,a\)=Δ​𝗋​\(s,a\)−λ​log⁡\(π​\(a\|s\)\),\\displaystyle\\overset\{\\Delta\}\{=\}\\mathsf\{r\}\(s,a\)\-\\lambda\\log\(\\pi\(a\|s\)\)\\kern 5\.0pt,\(4\)whereλ\>0\\lambda\>0is the temperature, which determines the strength of the penalty\. We also define the regularized Q\-function and regularized advantage as

q~πλ​\(s,a\)\\displaystyle\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\pi\}^\{\\lambda\}\(s,a\)=Δ​𝗋​\(s,a\)\+γ​∑s′∈𝒮𝖯​\(s′\|s,a\)​v~πλ​\(s′\),\\displaystyle\\overset\{\\Delta\}\{=\}\\textstyle\\mathsf\{r\}\(s,a\)\+\\gamma\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}\\mathsf\{P\}\(s^\{\\prime\}\|s,a\)\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\pi\}^\{\\lambda\}\(s^\{\\prime\}\)\\kern 5\.0pt,\(5\)a~πλ​\(s,a\)\\displaystyle\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\\pi\}^\{\\lambda\}\(s,a\)=Δ​q~πλ​\(s,a\)−λ​log⁡\(π​\(a\|s\)\)−v~πλ​\(s\)\.\\displaystyle\\overset\{\\Delta\}\{=\}\\textstyle\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\pi\}^\{\\lambda\}\(s,a\)\-\\lambda\\log\(\\pi\(a\|s\)\)\-\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\pi\}^\{\\lambda\}\(s\)\\kern 5\.0pt\.\(6\)Importantly, this regularized value satisfies the following regularized Bellman equation

v~πλ​\(s\)=∑a∈𝒜π​\(a\|s\)​\(q~πλ​\(s,a\)−λ​log⁡π​\(a\|s\)\)\.\\displaystyle\\textstyle\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\pi\}^\{\\lambda\}\(s\)=\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\(a\|s\)\\big\(\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\pi\}^\{\\lambda\}\(s,a\)\-\\lambda\\log\\pi\(a\|s\)\\big\)\\kern 5\.0pt\.\(7\)The optimal regularized value function, defined byv~⋆λ​\(s\)​=Δ​maxπ∈Π⁡v~πλ​\(s\)\\smash\{\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\star\}^\{\\lambda\}\(s\)\\overset\{\\Delta\}\{=\}\\max\_\{\\pi\\in\\Pi\}\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\pi\}^\{\\lambda\}\(s\)\}, satisfies the following consistency equations\(Nachumet al\.,[2017](https://arxiv.org/html/2605.24357#bib.bib19); Geistet al\.,[2019](https://arxiv.org/html/2605.24357#bib.bib49)\):

v~⋆λ​\(s\)=λ​log⁡\(∑a∈𝒜exp⁡\(q~⋆λ​\(s,a\)/λ\)\)\\displaystyle\\textstyle\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\star\}^\{\\lambda\}\(s\)=\\lambda\\log\(\\sum\_\{a\\in\\mathcal\{A\}\}\\exp\(\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\star\}^\{\\lambda\}\(s,a\)/\\lambda\)\)\(8\)π⋆λ​\(a\|s\)=exp⁡\(\(q~⋆λ​\(s,a\)−v~⋆λ​\(s\)\)/λ\),\\displaystyle\\pi\_\{\\star\}^\{\\lambda\}\(a\|s\)=\\exp\(\(\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\star\}^\{\\lambda\}\(s,a\)\-\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\star\}^\{\\lambda\}\(s\)\)/\\lambda\)\\kern 5\.0pt,\(9\)which link the optimal values and policies together\. In this work, we consider*softmax policies*; that is, givenθ∈ℝ\|𝒮\|​\|𝒜\|\\theta\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}, we consider the policy defined asπθ​\(a\|s\)∝exp⁡\(θ​\(s,a\)\)\\pi\_\{\\theta\}\(a\|s\)\\propto\{\\exp\(\\theta\(s,a\)\)\}together with the proper normalization\. Given these definitions, we aim to optimize the regularized value function with softmax parametrization

maxθ∈ℝ\|𝒮\|​\|𝒜\|⁡\{J~λ​\(θ\)​=Δ​v~πθλ​\(ρ\)\},\\smash\{\\max\_\{\\theta\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}\}\\bigl\\\{\\tilde\{J\}\_\{\\lambda\}\(\\theta\)\\overset\{\\Delta\}\{=\}\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\pi\_\{\\theta\}\}^\{\\lambda\}\(\\rho\)\\bigr\\\}\\kern 5\.0pt,\}\(10\)where we defineJ~λ⋆≜maxθ∈ℝ\|𝒮\|​\|𝒜\|⁡J~λ​\(θ\)\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\\triangleq\\max\_\{\\theta\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}\}\\tilde\{J\}\_\{\\lambda\}\(\\theta\)\. With a slight abuse of notation, we also definedρθ≜dρπθd\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\\triangleq d\_\{\\rho\}^\{\\hskip 0\.57367pt\\pi\_\{\\theta\}\},q~θλ≜q~πθλ\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\\triangleq\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\pi\_\{\\theta\}\}^\{\\lambda\}, andv~θλ≜v~πθλ\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\\triangleq\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\pi\_\{\\theta\}\}^\{\\lambda\}\.

#### Properties of the regularized value\.

The gradient of the regularized value can be expressed as follows\.

###### Lemma 1\(Lemma 10 ofMeiet al\.[2020b](https://arxiv.org/html/2605.24357#bib.bib3)\)\.

For anyθ∈ℝ\|𝒮\|​\|𝒜\|\\theta\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}, and\(s,a\)∈𝒮×𝒜\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}, it holds that

∂v~πθλ​\(ρ\)∂θ​\(s,a\)=11−γ​dρπθ​\(s\)​πθ​\(a\|s\)​a~πθλ​\(s,a\)\.\\displaystyle\\tfrac\{\\partial\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.51534pt\\pi\_\{\\theta\}\}^\{\\lambda\}\(\\rho\)\}\{\\partial\\theta\(s,a\)\}=\\tfrac\{1\}\{1\-\\gamma\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\pi\_\{\\theta\}\}\(s\)\\pi\_\{\\theta\}\(a\|s\)\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\\pi\_\{\\theta\}\}^\{\\lambda\}\(s,a\)\\kern 5\.0pt\.

The regularized value function is also smooth with respect toθ\\theta\. Specifically, it satisfies the following lemma

###### Lemma 2\(Lemma 7 and 14 ofMeiet al\.[2020b](https://arxiv.org/html/2605.24357#bib.bib3)\)\.

The regularized valuev~πθλ​\(ρ\)\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\pi\_\{\\theta\}\}^\{\\lambda\}\(\\rho\)isLL\-smooth with

L≜\(8\+λ​\(4\+8​log⁡\(\|𝒜\|\)\)\)/\(1−γ\)3\.L\\triangleq\(8\+\\lambda\(4\+8\\log\(\|\\mathcal\{A\}\|\)\)\)/\(1\-\\gamma\)^\{3\}\\kern 5\.0pt\.

We now introduce the classical state exploration assumption\(Meiet al\.,[2020a](https://arxiv.org/html/2605.24357#bib.bib20),[b](https://arxiv.org/html/2605.24357#bib.bib3); Agarwalet al\.,[2021](https://arxiv.org/html/2605.24357#bib.bib21)\)\.

###### AssumptionAρ\\textbf\{A\}\_\{\\rho\}\.

​The smallest coefficientρmin​=Δ​mins∈𝒮⁡ρ​\(s\)\\rho\_\{\\min\}\\overset\{\\Delta\}\{=\}\\min\_\{s\\in\\mathcal\{S\}\}\\rho\(s\)of the initial distributionρ\\rhosatisfiesρmin\>0\\rho\_\{\\min\}\\\!\>0\.

Under the previous assumption, the regularized value satisfies a Non\-Uniform Łojasiewicz property\.

###### Lemma 3\(Lemma 15 ofMeiet al\.[2020b](https://arxiv.org/html/2605.24357#bib.bib3)\)\.

For anyθ∈ℝ\|𝒮\|​\|𝒜\|\\theta\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}, we have

∥∇θv~πθλ​\(ρ\)∥2\\displaystyle\\lVert\\nabla\_\{\\theta\}\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\pi\_\{\\theta\}\}^\{\\lambda\}\(\\rho\)\\rVert^\{2\}≥μ~λ​\(θ\)​\(v~⋆λ​\(ρ\)−v~πθλ​\(ρ\)\),\\displaystyle\\geq\\tilde\{\\mu\}\_\{\\lambda\}\(\\theta\)\\left\(\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\star\}^\{\\lambda\}\(\\rho\)\-\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\pi\_\{\\theta\}\}^\{\\lambda\}\(\\rho\)\\right\)\\kern 5\.0pt,μ~λ​\(θ\)\\displaystyle\\tilde\{\\mu\}\_\{\\lambda\}\(\\theta\)=Δ​λ​\(1−γ\)​ρmin2​min\(s,a\)∈𝒮×𝒜⁡πθ​\(a\|s\)2/\|𝒮\|,\\displaystyle\\overset\{\\Delta\}\{=\}\\lambda\(1\-\\gamma\)\\rho\_\{\\min\}^\{2\}\\min\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}\\pi\_\{\\theta\}\(a\|s\)^\{2\}/\|\\mathcal\{S\}\|\\kern 5\.0pt,and wherev~⋆λ=maxθ∈ℝ\|𝒮\|​\|𝒜\|⁡v~πθλ​\(ρ\)\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\star\}^\{\\lambda\}=\\max\_\{\\theta\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}\}\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\pi\_\{\\theta\}\}^\{\\lambda\}\(\\rho\)\.

Algorithm 1[Ent\-AC](https://arxiv.org/html/2605.24357#alg1): Entropy\-regularized Actor\-Critic1:Input:stepsizes

η𝖺,η𝖼\\eta\_\{\\mathsf\{a\}\},\\eta\_\{\\mathsf\{c\}\}; initial parameters

q^−1\\hat\{\\mathrm\{q\}\}\_\{\-1\}, and

θ0\\theta\_\{0\}; projection operator

𝒯\\mathcal\{T\}\.

2:for

k=0k=0to

K−1K\-1do

3:Set

q^k0=q^k−1\\hat\{\\mathrm\{q\}\}\_\{k\}^\{0\}=\\hat\{\\mathrm\{q\}\}\_\{k\-1\}and sample

Xk=\(Xkh\)h=1HX\_\{k\}=\(X\_\{k\}^\{h\}\)\_\{h=1\}^\{H\}where

Xkh=\(Skh,Akh,S~kh,A~kh\)X\_\{k\}^\{h\}=\(S\_\{k\}^\{h\},A\_\{k\}^\{h\},\\tilde\{S\}\_\{k\}^\{h\},\\tilde\{A\}\_\{k\}^\{h\}\)and

Xkh∼ν𝖼​\(θk,⋅\)X\_\{k\}^\{h\}\\sim\\nu^\{\\mathsf\{c\}\}\(\\theta\_\{k\},\\cdot\)\(defined in \([14](https://arxiv.org/html/2605.24357#S3.E14)\)\)\.

4:for

h=0h=0to

H−1H\-1do

5:Compute the stochastic gradient for the critic’s update

g𝖼Xkh\+1​\(θk,q^kh\)∈ℝ\|𝒮\|​\|𝒜\|\\mathrm\{g\}^\{X\_\{k\}^\{h\+1\}\}\_\{\\mathsf\{c\}\}\(\\theta\_\{k\},\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\)\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}defined with:

\[g𝖼Xkh\+1​\(θk,q^kh\)\]s,a\\displaystyle\\textstyle\[\\mathrm\{g\}^\{X\_\{k\}^\{h\+1\}\}\_\{\\mathsf\{c\}\}\(\\theta\_\{k\},\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\)\]\_\{s,a\}=𝟣\(s,a\)​\(Skh\+1,Akh\+1\)​δ​\(Xkh\+1,θk,q^kh\),where\\displaystyle=\\mathsf\{1\}\_\{\(s,a\)\}\(S\_\{k\}^\{h\+1\},A\_\{k\}^\{h\+1\}\)\\delta\(X\_\{k\}^\{h\+1\},\\theta\_\{k\},\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\)\\kern 5\.0pt,\\quad\\text\{ where \}\(11\)δ​\(Xkh\+1,θk,q^kh\)\\displaystyle\\delta\(X\_\{k\}^\{h\+1\},\\theta\_\{k\},\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\)=\[𝗋​\(Skh\+1,Akh\+1\)\+γ​\[q^kh​\(S~kh\+1,A~kh\+1\)−λ​log⁡\(πθ​\(A~kh\+1\|S~kh\+1\)\)\]−q^kh​\(Skh\+1,Akh\+1\)\]\\displaystyle=\\big\[\\mathsf\{r\}\(S\_\{k\}^\{h\+1\},A\_\{k\}^\{h\+1\}\)\+\\gamma\\,\[\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\(\\tilde\{S\}\_\{k\}^\{h\+1\},\\tilde\{A\}\_\{k\}^\{h\+1\}\)\-\\lambda\\log\(\\pi\_\{\\theta\}\(\\tilde\{A\}\_\{k\}^\{h\+1\}\|\\tilde\{S\}\_\{k\}^\{h\+1\}\)\)\]\-\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\(S\_\{k\}^\{h\+1\},A\_\{k\}^\{h\+1\}\)\\big\]
6:Update estimate:

q^kh\+1=q^kh\+η𝖼​g𝖼Xkh\+1​\(θk,q^kh\)\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\+1\}=\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\+\\eta\_\{\\mathsf\{c\}\}\\,\\mathrm\{g\}^\{X\_\{k\}^\{h\+1\}\}\_\{\\mathsf\{c\}\}\(\\theta\_\{k\},\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\)\{\# Update critic using Temporal Difference error\}

7:endfor

8:Set

q^k=q^kH\\hat\{\\mathrm\{q\}\}\_\{k\}=\\hat\{\\mathrm\{q\}\}\_\{k\}^\{H\}\. For all

\(s,a\)∈𝒮×𝒜\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}, compute the regularized values and advantages using \([13](https://arxiv.org/html/2605.24357#S3.E13)\)\.

9:Sample

Yk\+1=\(Sk\+1,Ak\+1\)∼ν𝖺​\(θk,⋅\)Y\_\{k\+1\}\\\!=\\\!\(S\_\{k\+1\},A\_\{k\+1\}\)\\\!\\sim\\\!\\nu^\{\\mathsf\{a\}\}\(\\theta\_\{k\},\\cdot\)\(see \([15](https://arxiv.org/html/2605.24357#S3.E15)\)\) and compute the stochastic gradient

g𝖺Yk\+1​\(a^k\)\\mathrm\{g\}^\{Y\_\{k\+1\}\}\_\{\\mathsf\{a\}\}\(\\hat\{\\mathrm\{a\}\}\_\{k\}\)defined with:

\[g𝖺Yk\+1​\(a^k\)\]s,a=𝟣\(s,a\)​\(Sk\+1,Ak\+1\)​\(1−γ\)−1​a^k​\(Sk\+1,Ak\+1\)\\displaystyle\[\\mathrm\{g\}^\{Y\_\{k\+1\}\}\_\{\\mathsf\{a\}\}\(\\hat\{\\mathrm\{a\}\}\_\{k\}\)\]\_\{s,a\}=\\mathsf\{1\}\_\{\(s,a\)\}\(S\_\{k\+1\},A\_\{k\+1\}\)\\,\(1\-\\gamma\)^\{\-1\}\\,\\hat\{\\mathrm\{a\}\}\_\{k\}\(S\_\{k\+1\},A\_\{k\+1\}\)\(12\)
10:Update the actor:

θk\+1=𝒯​\(θk\+η𝖺​g𝖺Yk\+1​\(a^k\)\)\\theta\_\{k\+1\}\\;=\\;\\mathcal\{T\}\\\!\\big\(\\theta\_\{k\}\+\\eta\_\{\\mathsf\{a\}\}\\,\\mathrm\{g\}^\{Y\_\{k\+1\}\}\_\{\\mathsf\{a\}\}\(\\hat\{\\mathrm\{a\}\}\_\{k\}\)\\big\)\.

11:endforI: CriticII: Actor

#### Entropy\-regularized AC\.

Actor critic with entropy regularization \([Ent\-AC](https://arxiv.org/html/2605.24357#alg1)\) alternates between actor and critic updates\. In this paper, we study the variant where at iterationkkthe criticq^k\\hat\{\\mathrm\{q\}\}\_\{k\}is updatedHHtimes, using the TD update \([11](https://arxiv.org/html/2605.24357#S3.E11)\)\. The regularized value and advantages are then updated as

v^k​\(s\)=∑a∈𝒜πθk​\(a\|s\)​\[q^k​\(s,a\)−λ​log⁡\(πθk​\(a\|s\)\)\],\\displaystyle\\hat\{\\mathrm\{v\}\}\_\{k\}\(s\)\\\!=\\\!\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\_\{\\theta\_\{k\}\}\(a\|s\)\\,\\\!\\big\[\\hat\{\\mathrm\{q\}\}\_\{k\}\(s,a\)\\\!\-\\\!\\lambda\\log\\\!\\big\(\\pi\_\{\\theta\_\{k\}\}\(a\|s\)\\big\)\\big\]\\\!\\,,\(13\)a^k​\(s,a\)=q^k​\(s,a\)−λ​log⁡\(πθk​\(a∣s\)\)−v^k​\(s\)\\displaystyle\\hat\{\\mathrm\{a\}\}\_\{k\}\(s,a\)\\\!=\\\!\\hat\{\\mathrm\{q\}\}\_\{k\}\(s,a\)\\\!\-\\\!\\lambda\\log\\\!\\big\(\\pi\_\{\\theta\_\{k\}\}\(a\\\!\\mid\\\!s\)\\big\)\\\!\-\\\!\\hat\{\\mathrm\{v\}\}\_\{k\}\(s\)\\,whereπθk\\pi\_\{\\theta\_\{k\}\}is the actor at iterationkk, using softmax parameterization with parameterθk\\theta\_\{k\}\. At the end of the iteration, the actor is updated using the gradient update \([12](https://arxiv.org/html/2605.24357#S3.E12)\)\.

We give the pseudo\-code of the procedure in[Algorithm1](https://arxiv.org/html/2605.24357#alg1)\. Given an initial distributionρ\\rhoover states, we define the two sampling distributionsν𝖼​\(θ;⋅\)\\nu^\{\\mathsf\{c\}\}\(\\theta;\\cdot\)andν𝖺​\(θ;⋅\)\\nu^\{\\mathsf\{a\}\}\(\\theta;\\cdot\)as, forx=\(s,a,s~,a~\)∈𝒮×𝒜×𝒮×𝒜x=\(s,a,\\tilde\{s\},\\tilde\{a\}\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\\times\\mathcal\{S\}\\times\\mathcal\{A\}, andy=\(s,a\)∈𝒮×𝒜y=\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\},

ν𝖼​\(θ;x\)\\displaystyle\\nu^\{\\mathsf\{c\}\}\(\\theta;x\)=Δ​dρθ​\(s\)​πθ​\(a\|s\)​𝖯​\(s~\|s,a\)​πθ​\(a~\|s~\),\\displaystyle\\overset\{\\Delta\}\{=\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)\\pi\_\{\\theta\}\(a\|s\)\\mathsf\{P\}\(\\tilde\{s\}\|s,a\)\\pi\_\{\\theta\}\(\\tilde\{a\}\|\\tilde\{s\}\)\\kern 5\.0pt,\(14\)ν𝖺​\(θ;y\)\\displaystyle\\nu^\{\\mathsf\{a\}\}\(\\theta;y\)=Δ​dρθ​\(s\)​πθ​\(a\|s\)\.\\displaystyle\\overset\{\\Delta\}\{=\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)\\pi\_\{\\theta\}\(a\|s\)\\kern 5\.0pt\.\(15\)These are respectively used to compute the TD update of the critic and stochastic actor gradients\.

## 4Analysis of[Ent\-AC](https://arxiv.org/html/2605.24357#alg1): Exact Critic Case

In this section, we show that, when used with a perfect critic, entropy\-regularized AC achieves*strong variance reduction*\. Formally, this consists in settingq^k≡q~θkλ\\hat\{\\mathrm\{q\}\}\_\{k\}\\equiv\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\theta\_\{k\}\}^\{\\lambda\}for allk≥0k\\geq 0\. To conduct our analysis, we first derive a bound onπmin\\pi\_\{\\min\}, the minimal value of the policy through the optimization\. We then useπmin\\pi\_\{\\min\}to link the variance of the stochastic actor gradient to the norm of its deterministic counterpart\.

#### Controlling𝝅𝐦𝐢𝐧\\boldsymbol\{\\pi\_\{\\min\}\}\.

To controlπmin\\pi\_\{\\min\}, we project the current policy onto a smaller subspace, followingZhanget al\.\([2021b](https://arxiv.org/html/2605.24357#bib.bib17)\); Labbiet al\.\([2026b](https://arxiv.org/html/2605.24357#bib.bib16)\)\. For any thresholdτ\>0\\tau\>0and policyπ\\pi, we introduce𝒜τπ​\(s\)​=Δ​\{a∈𝒜,π​\(a\|s\)≤τ\}\\mathcal\{A\}\_\{\\tau\}^\{\\pi\}\(s\)\\overset\{\\Delta\}\{=\}\\left\\\{a\\in\\mathcal\{A\},\\pi\(a\|s\)\\leq\\tau\\right\\\}, as well as the operator𝒰τ\\mathcal\{U\}\_\{\\tau\}which acts on\(s,a\)∈𝒮×𝒜\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}as

𝒰τ​\(π\)​\(a\|s\)​=Δ​\{τ,if​π​\(a\|s\)≤τ,π​\(a\|s\)−bπ​\(s\),if​a=amaxπ​\(s\),π​\(a\|s\),otherwise,\\textstyle\\\!\\mathcal\{U\}\_\{\\tau\}\(\\pi\)\(a\|s\)\\overset\{\\Delta\}\{=\}\\begin\{cases\}\\tau,&\\text\{if \}\\pi\(a\|s\)\\\!\\leq\\\!\\tau,\\\\ \\pi\(a\|s\)\-\\textstyle b\_\{\\pi\}\(s\),&\\text\{if \}a=a^\{\\pi\}\_\{\\max\}\(s\),\\\\ \\pi\(a\|s\),&\\text\{otherwise\},\\end\{cases\}wherebπ​\(s\)​=Δ​∑a∈𝒜τπ​\(s\)\(τ−π​\(a\|s\)\)\\smash\{b\_\{\\pi\}\(s\)\\overset\{\\Delta\}\{=\}\\sum\_\{a\\in\\mathcal\{A\}\_\{\\tau\}^\{\\pi\}\(s\)\}\(\\tau\-\\pi\(a\|s\)\)\}, andamaxπ​\(s\)​=Δ​arg​maxa∈𝒜⁡\{π​\(a\|s\)\}a\_\{\\max\}^\{\\pi\}\(s\)\\overset\{\\Delta\}\{=\}\\operatorname\*\{arg\\,max\}\_\{a\\in\\mathcal\{A\}\}\\\{\\pi\(a\|s\)\\\}, choosing at random in thearg​max\\operatorname\*\{arg\\,max\}in case of ties\. We denote𝒯τ\\mathcal\{T\}\_\{\\tau\}the corresponding operator in the logit space, i\.e\. for allθ\\theta,π𝒯τ​\(θ\)​=Δ​𝒰τ​\(πθ\)\\pi\_\{\\mathcal\{T\}\_\{\\tau\}\(\\theta\)\}\\overset\{\\Delta\}\{=\}\\mathcal\{U\}\_\{\\tau\}\(\\pi\_\{\\theta\}\)\.

This operator prevents policies from reaching policies with low entropy, i\.e\., from becoming too deterministic: for anys,a∈𝒮×𝒜s,a\\in\\mathcal\{S\}\\times\\mathcal\{A\}, ifπ​\(a\|s\)\\pi\(a\|s\)approaches zero,𝒰τ\\mathcal\{U\}\_\{\\tau\}raises it above aτ\\tau\-dependent threshold\. With a suitable choice ofτ\\tau,𝒯τ\\mathcal\{T\}\_\{\\tau\}yields logits with a higher regularized value\.

###### Lemma 4\.

Assume thatρ\\rhosatisfies[Aρ\\textbf\{A\}\_\{\\rho\}](https://arxiv.org/html/2605.24357#ThmassumMDPx1)\. Letτλ​=Δ​min⁡\(13​exp⁡\(−16\+8​γ​λ​log⁡\(\|𝒜\|\)λ​\(1−γ\)2​ρmin\),138​\|𝒜\|4\)\\tau\_\{\\lambda\}\\overset\{\\Delta\}\{=\}\\min\\left\(\\tfrac\{1\}\{3\}\\exp\\left\(\-\\tfrac\{16\+8\\gamma\\lambda\\log\(\|\\mathcal\{A\}\|\)\}\{\\lambda\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\}\\right\),\\frac\{1\}\{3^\{8\}\|\\mathcal\{A\}\|^\{4\}\}\\right\)\. Then, for anyθ∈ℝ\|𝒮\|​\|𝒜\|\\theta\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}and forθ~=𝒯τλ​\(θ\)\\widetilde\{\\theta\}=\\mathcal\{T\}\_\{\\tau\_\{\\lambda\}\}\(\\theta\), it holds thatv~θ~λ​\(ρ\)≥v~θλ​\(ρ\)\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\widetilde\{\\theta\}\}^\{\\lambda\}\(\\rho\)\\geq\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\(\\rho\)and that for any\(s,a\)∈𝒮×𝒜,πθ~​\(a\|s\)≥τλ\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\},\\pi\_\{\\widetilde\{\\theta\}\}\(a\|s\)\\geq\\tau\_\{\\lambda\}\.

Based on this lemma, we observe that for any parameter obtained by applying the improvement operator𝒯τλ\\mathcal\{T\}\_\{\\tau\_\{\\lambda\}\}, thenπmin\\pi\_\{\\min\}and the corresponding Łojasiewicz constant defined in[Lemma3](https://arxiv.org/html/2605.24357#Thmlemma3)are bounded from below by

πmin≥τλ,μ¯~λ​=Δ​λ​\(1−γ\)​ρmin2​τλ2/\|𝒮\|,\\displaystyle\\pi\_\{\\min\}\\geq\\tau\_\{\\lambda\}\\kern 5\.0pt,\\quad\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\\overset\{\\Delta\}\{=\}\\lambda\(1\-\\gamma\)\\,\\rho\_\{\\min\}^\{2\}\\tau\_\{\\lambda\}^\{2\}/\|\\mathcal\{S\}\|,\(16\)whereτλ\\tau\_\{\\lambda\}is defined in[Lemma4](https://arxiv.org/html/2605.24357#Thmlemma4)\.

#### Convergence of AC with exact critic\.

To establish the convergence of AC with exact critic, we show that its gradient is unbiased, but most importantly that its variance can be linked to the norm of the*deterministic gradient*\. This property is formalized in the following lemma\.

###### Lemma 5\.

Assume[Aρ\\textbf\{A\}\_\{\\rho\}](https://arxiv.org/html/2605.24357#ThmassumMDPx1)and assume that for all\(s,a\)∈𝒮×𝒜\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}, we haveπθ​\(a\|s\)≥πmin\>0\\pi\_\{\\theta\}\(a\|s\)\\geq\\pi\_\{\\min\}\>0\. For anyθ∈ℝ\|𝒮\|​\|𝒜\|\\theta\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}, it holds that

𝔼Y∼ν𝖺​\(θ\)​\[g𝖺Y​\(a~θλ\)\]\\displaystyle\\mathbb\{E\}\_\{Y\\sim\\nu^\{\\mathsf\{a\}\}\(\\theta\)\}\\left\[\\mathrm\{g\}^\{Y\}\_\{\\mathsf\{a\}\}\(\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\)\\right\]=∂J~λ​\(θ\)∂θ,\\displaystyle=\\frac\{\\partial\\tilde\{J\}\_\{\\lambda\}\(\\theta\)\}\{\\partial\\theta\}\\kern 5\.0pt,\\quad𝔼Y∼ν𝖺​\(θ\)​\[‖g𝖺Y​\(a~θλ\)−∂J~λ∂θ‖22\]\\displaystyle\\mathbb\{E\}\_\{Y\\sim\\nu^\{\\mathsf\{a\}\}\(\\theta\)\}\\left\[\\big\\\|\\mathrm\{g\}^\{Y\}\_\{\\mathsf\{a\}\}\(\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\)\-\\tfrac\{\\partial\\tilde\{J\}\_\{\\lambda\}\}\{\\partial\\theta\}\\big\\\|\_\{2\}^\{2\}\\right\]≤\(1−γ\)−1πmin​ρmin​‖∂J~λ​\(θ\)∂θ‖22\.\\displaystyle\\leq\\tfrac\{\(1\-\\gamma\)^\{\-1\}\}\{\\pi\_\{\\min\}\\rho\_\{\\min\}\}\\big\\\|\\tfrac\{\\partial\\tilde\{J\}\_\{\\lambda\}\(\\theta\)\}\{\\partial\\theta\}\\big\\\|\_\{2\}^\{2\}\\kern 5\.0pt\.

###### Proof sketch\.

The key observation is that the variance of the gradient can be expressed as

Var​\(g𝖺Y​\(a~θλ\)\)\\displaystyle\\text\{Var\}\(\\mathrm\{g\}^\{Y\}\_\{\\mathsf\{a\}\}\(\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\)\)=1\(1−γ\)2​∑\(s′,a′\)∈𝒮×𝒜dρθ​\(s′\)​πθ​\(a′\|s′\)​δs′,a′,\\displaystyle=\\frac\{1\}\{\(1\-\\gamma\)^\{2\}\}\\\!\\\!\\sum\_\{\(s^\{\\prime\},a^\{\\prime\}\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}\\\!\\\!\\\!d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s^\{\\prime\}\)\\pi\_\{\\theta\}\(a^\{\\prime\}\|s^\{\\prime\}\)\\delta\_\{s^\{\\prime\},a^\{\\prime\}\}\\kern 5\.0pt,whereδs′,a′=a~θλ​\(s′,a′\)2−dρθ​\(s′\)​πθ​\(a′\|s′\)​a~θλ​\(s′,a′\)2\\delta\_\{s^\{\\prime\},a^\{\\prime\}\}\\\!=\\\!\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\(s^\{\\prime\},a^\{\\prime\}\)^\{2\}\\\!\-\\\!d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s^\{\\prime\}\)\\pi\_\{\\theta\}\(a^\{\\prime\}\|s^\{\\prime\}\)\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\(s^\{\\prime\},a^\{\\prime\}\)^\{2\}\. Usingmin\(s,a\)∈𝒮×𝒜⁡πθ​\(a\|s\)≥πmin\\min\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}\\pi\_\{\\theta\}\(a\|s\)\\geq\\pi\_\{\\min\}to bound the termsdρθ​\(s′\)​πθ​\(a′\|s′\)≤\(dρθ​\(s′\)​πθ​\(a′\|s′\)\)2/\(\(1−γ\)​πmin​ρmin\)d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s^\{\\prime\}\)\\pi\_\{\\theta\}\(a^\{\\prime\}\|s^\{\\prime\}\)\\leq\{\(d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s^\{\\prime\}\)\\pi\_\{\\theta\}\(a^\{\\prime\}\|s^\{\\prime\}\)\)^\{2\}\}/\{\(\(1\-\\gamma\)\\pi\_\{\\min\}\\rho\_\{\\min\}\)\}and recognizing the expression of the gradient given in[Lemma1](https://arxiv.org/html/2605.24357#Thmlemma1)gives the result\. Full proof in[AppendixB](https://arxiv.org/html/2605.24357#A2)\. ∎

This result proves that, when using the critic as a baseline, which is the actual value function of the current policy, the AC stochastic gradient’s variance can be upper\-bounded by the corresponding deterministic gradient’s norm up to a multiplicative constant\. Remarkably, this means that when the algorithm has converged, and the deterministic gradient is zero, AC does not have any remaining variance at all\! This property can be leveraged to derive the following convergence result, which shows that when the critic is exact, AC converges to the optimal policy\.

###### Theorem 1\.

Assume that the initial distributionρ\\rhosatisfies[Aρ\\textbf\{A\}\_\{\\rho\}](https://arxiv.org/html/2605.24357#ThmassumMDPx1)\. Fixη𝖺≤\(1−γ\)​ρmin​πmin/L\\eta\_\{\\mathsf\{a\}\}\\leq\(1\-\\gamma\)\\rho\_\{\\min\}\\pi\_\{\\min\}/Land consider the iterates of[Ent\-AC](https://arxiv.org/html/2605.24357#alg1)with projection operator𝒯=𝒯τλ\\mathcal\{T\}=\\mathcal\{T\}\_\{\\tau\_\{\\lambda\}\}\. It holds thatmink≥0⁡min\(s,a\)∈𝒮×𝒜⁡πθk​\(a\|s\)≥τλ\\min\_\{k\\geq 0\}\\min\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}\\pi\_\{\\theta\_\{k\}\}\(a\|s\)\\geq\\tau\_\{\\lambda\}almost surely\. Additionally, for anyk≥0k\\geq 0we have that

𝔼​\[J~λ⋆−J~λ​\(θk\)\]≤\(1−η𝖺​μ¯~λ\)k​\(J~λ⋆−J~λ​\(θ0\)\)\.\\displaystyle\\mathbb\{E\}\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\]\\leq\\left\(1\-\\eta\_\{\\mathsf\{a\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\\right\)^\{k\}\\left\(\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{0\}\)\\right\)\\kern 5\.0pt\.

This theorem is a consequence of[Lemma5](https://arxiv.org/html/2605.24357#Thmlemma5)\. To our knowledge, this result is the first to show that the Actor\-Critic method*is a strong variance reduction method*\. When using a perfect baseline, it converges linearly towards the optimal policy, matching the rates of deterministic methods\.

###### Corollary 1\(Sample complexity\)\.

Under the same assumptions as Theorem[1](https://arxiv.org/html/2605.24357#Thmtheorem1), for anyϵ\>0\\epsilon\>0, it suffices to take

K≥Lμ¯~λ​1\(1−γ\)​ρmin​τλ​log⁡\(J~λ⋆−J~λ​\(θ0\)ϵ\)\\displaystyle K\\geq\\frac\{L\}\{\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\}\\frac\{1\}\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\\log\\Big\(\\frac\{\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{0\}\)\}\{\\epsilon\}\\Big\)iterations to guaranteeJ~λ⋆−𝔼​\[J~λ​\(θK\)\]≤ϵ\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\mathbb\{E\}\\left\[\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{K\}\)\\right\]\\leq\\epsilon\.

This is a direct consequence of[Theorem1](https://arxiv.org/html/2605.24357#Thmtheorem1)\. This shows that, akin to deterministic gradient methods, the sample complexity of AC is of orderlog⁡\(1/ϵ\)\\log\(1/\\epsilon\)\. Crucially, this shows that using the critic as a baseline does not diminish the scale of the stochastic gradient’s variance, but in fact completely removes its importance\.

## 5Analysis of[Ent\-AC](https://arxiv.org/html/2605.24357#alg1): General Case

In practice, no oracles for the critic are available, and one has to learn it alongside the actor\. In such a case, we show that*the variance of the actor does not matter*, and all the residual variance in the actor\-critic algorithm comes from*estimating the critic*\. Given a policyπ\\piand an estimated regularized q\-valueq\\mathrm\{q\}, we can construct an estimate of regularized advantage for any state action pair\(s,a\)\(s,a\)as

a​\(s,a\)\\displaystyle\\mathrm\{a\}\(s,a\)=Δ​q​\(s,a\)−λ​log⁡\(π​\(a∣s\)\)−v​\(s\),\\displaystyle\\overset\{\\Delta\}\{=\}\\mathrm\{q\}\(s,a\)\-\\lambda\\log\\big\(\\pi\(a\\mid s\)\\big\)\-\\mathrm\{v\}\(s\)\\kern 5\.0pt,v​\(s\)\\displaystyle\\mathrm\{v\}\(s\)=Δ​∑a∈𝒜π​\(a\|s\)​\(q​\(s,a\)−λ​log⁡\(π​\(a\|s\)\)\)\.\\displaystyle\\overset\{\\Delta\}\{=\}\\textstyle\\sum\\nolimits\_\{a\\in\\mathcal\{A\}\}\\pi\(a\|s\)\\left\(\\mathrm\{q\}\(s,a\)\-\\lambda\\log\(\\pi\(a\|s\)\)\\right\)\.Using the estimated advantagea\\mathrm\{a\}gives a gradient estimatorg𝖺Y​\(a\)\\mathrm\{g\}^\{Y\}\_\{\\mathsf\{a\}\}\(\\mathrm\{a\}\)\. Inexactness of the critic estimates has an impact on the bias and variance of this estimator, which we define as

b​\(θ,a\)\\displaystyle\\mathrm\{b\}\(\\theta,\\mathrm\{a\}\)=Δ​‖g¯𝖺​\(θ,a\)−∂J~λ​\(θ\)∂θ‖22,\\displaystyle\\overset\{\\Delta\}\{=\}\\\|\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{a\}\}\(\\theta,\\mathrm\{a\}\)\-\\frac\{\\partial\\tilde\{J\}\_\{\\lambda\}\(\\theta\)\}\{\\partial\\theta\}\\\|\_\{2\}^\{2\}\\kern 5\.0pt,\(17\)Var​\(θ,a\)\\displaystyle\\mathrm\{Var\}\(\\theta,\\mathrm\{a\}\)=Δ​𝔼Y∼ν𝖺​\(θ\)​\[‖g𝖺Y​\(a\)−g¯𝖺​\(θ,a\)‖22\],\\displaystyle\\overset\{\\Delta\}\{=\}\\mathbb\{E\}\_\{Y\\sim\\nu^\{\\mathsf\{a\}\}\(\\theta\)\}\\left\[\\\|\\mathrm\{g\}^\{Y\}\_\{\\mathsf\{a\}\}\(\\mathrm\{a\}\)\-\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{a\}\}\(\\theta,\\mathrm\{a\}\)\\\|\_\{2\}^\{2\}\\right\]\\kern 5\.0pt,\(18\)whereg¯𝖺​\(θ,a\)​=Δ​𝔼Y∼ν𝖺​\(θ\)​\[g𝖺Y​\(a\)\]\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{a\}\}\(\\theta,\\mathrm\{a\}\)\\overset\{\\Delta\}\{=\}\\smash\{\\mathbb\{E\}\_\{Y\\sim\\nu^\{\\mathsf\{a\}\}\(\\theta\)\}\[\\mathrm\{g\}^\{Y\}\_\{\\mathsf\{a\}\}\(\\mathrm\{a\}\)\]\}\. Next, we give a counterpart of[Lemma5](https://arxiv.org/html/2605.24357#Thmlemma5)with an inexact critic, bounding the bias and variance of the current estimator\.

###### Lemma 6\.

Fixθ∈ℝ\|𝒮\|​\|𝒜\|\\theta\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}anda∈ℝ\|𝒮\|​\|𝒜\|\\mathrm\{a\}\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}\. It holds that

b​\(θ,a\)=1\(1−γ\)2​∑\(s,a\)∈𝒮×𝒜dρθ​\(s\)2​πθ​\(a\|s\)2​\(a​\(s,a\)−a~θλ​\(s,a\)\)2,\\displaystyle\\mathrm\{b\}\(\\theta,\\\!\\mathrm\{a\}\)\\\!=\\\!\\frac\{1\}\{\(1\\\!\-\\\!\\gamma\)^\{2\}\}\\\!\\\!\\\!\\\!\\\!\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}\\\!\\\!\\\!\\\!\\\!\\\!\\\!\\\!d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)^\{2\}\\pi\_\{\\theta\}\(a\|s\)^\{2\}\(\\mathrm\{a\}\(s\\\!,\\\!a\)\\\!\-\\\!\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\(s\\\!,a\)\)^\{2\}\\\!\\\!,Var​\(θ,a\)≤1\(1−γ\)2​∑\(s,a\)∈𝒮×𝒜dρθ​\(s\)​πθ​\(a\|s\)​a​\(s,a\)2\.\\displaystyle\\hskip\-20\.0pt\\mathrm\{Var\}\(\\theta,\\mathrm\{a\}\)\\leq\\frac\{1\}\{\(1\\\!\-\\\!\\gamma\)^\{2\}\}\\\!\\\!\\\!\\\!\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}\\\!\\\!\\\!\\\!\\\!d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)\\pi\_\{\\theta\}\(a\|s\)\\mathrm\{a\}\(s,a\)^\{2\}\\kern 5\.0pt\.

We provide a proof in[AppendixC](https://arxiv.org/html/2605.24357#A3)\. While this lemma is very similar to its exact critic counterpart, it differs in a fundamental way: the advantages are replaced by the estimated advantages\. Consequently, having a biased critic ends up creating bias in the gradient estimator itself, directly depending on the difference between the estimated advantage and its true value\. However, the variance can still be bounded using the norm of the deterministic gradient and the bias directly when the minimal probability of the policy is uniformly lower\-bounded\. In order to guarantee that this coefficient remains uniformly lower bounded, we restrict the optimization to a smaller subspace, eliminating policies for which the regularization is too strong\. By proceeding in this way, we guarantee that the minimal probability stays uniformly lower\-bounded\.

Next, we derive the recursions for the actor’s and critic’s updates separately, then combine them to obtain the final convergence rate\. Before that, we define the filtration adapted to the iterates of the actor:

ℱk=Δσ\(Xℓ:ℓ∈\{0,…,k−1\},Yℓ:ℓ∈\{1,…,k\}\)\.\\displaystyle\\mathcal\{F\}\_\{k\}\\overset\{\\Delta\}\{=\}\\sigma\\Big\(X\_\{\\ell\}:\\ell\\in\\\{0,\\dots,k\-1\\\},Y\_\{\\ell\}:\\ell\\in\\\{1,\\dots,k\\\}\\Big\)\\kern 5\.0pt\.
#### Updating the Actor\.

Below, we derive a recursion on the actor updates with a full proof in[AppendixC](https://arxiv.org/html/2605.24357#A3)\.

###### Lemma 7\(Actor recursion\)\.

Assume[Aρ\\textbf\{A\}\_\{\\rho\}](https://arxiv.org/html/2605.24357#ThmassumMDPx1)and consider the iterates of[Ent\-AC](https://arxiv.org/html/2605.24357#alg1)with projection operator𝒯=𝒯τλ\\mathcal\{T\}=\\mathcal\{T\}\_\{\\tau\_\{\\lambda\}\}\. It holds thatmink≥0⁡min\(s,a\)∈𝒮×𝒜⁡πθk​\(a\|s\)\>τλ\\min\_\{k\\geq 0\}\\min\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}\\pi\_\{\\theta\_\{k\}\}\(a\|s\)\>\\tau\_\{\\lambda\}almost surely\. For anyk≥0k\\geq 0, we also have that

𝔼​\[J~λ⋆−J~λ​\(θk\+1\)\|ℱk\]\\displaystyle\\mathbb\{E\}\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\+1\}\)\\big\|\\mathcal\{F\}\_\{k\}\\right\]≤J~λ⋆−J~λ​\(θk\)−\(η𝖺2−2​L​η𝖺2\(1−γ\)​ρmin​τλ\)​‖∇J~λ​\(θk\)‖22\\displaystyle\\leq\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\-\\big\(\\tfrac\{\\eta\_\{\\mathsf\{a\}\}\}\{2\}\-\\tfrac\{2L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\\big\)\\\|\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\\|\_\{2\}^\{2\}\+η𝖺​2​b​\(θk,𝔼​\[a^k​\(s,a\)\|ℱk\]\)\(1−γ\)2\+L​η𝖺2​2​𝔼​\[Var​\(θk,a^k\)\|ℱk\]\(1−γ\)2,\\displaystyle\+\\eta\_\{\\mathsf\{a\}\}\\tfrac\{2\\mathrm\{b\}\(\\theta\_\{k\},\\\!\\mathbb\{E\}\[\\hat\{\\mathrm\{a\}\}\_\{k\}\(s,a\)\|\\mathcal\{F\}\_\{k\}\]\)\}\{\(1\-\\gamma\)^\{2\}\}\+L\\eta\_\{\\mathsf\{a\}\}^\{2\}\\tfrac\{2\\mathbb\{E\}\\left\[\\mathrm\{Var\}\(\\theta\_\{k\},\\hat\{\\mathrm\{a\}\}\_\{k\}\)\|\\mathcal\{F\}\_\{k\}\\right\]\}\{\(1\-\\gamma\)^\{2\}\}\\kern 5\.0pt,whereb​\(θ,a\)\\mathrm\{b\}\(\\theta,\\mathrm\{a\}\)andVar​\(θ,a\)\\mathrm\{Var\}\(\\theta,\\mathrm\{a\}\)are defined in \([17](https://arxiv.org/html/2605.24357#S5.E17)\) and \([18](https://arxiv.org/html/2605.24357#S5.E18)\) respectively\.

This recursion captures that the policy improvement at each actor step is controlled entirely by the critic’s*bias*\([17](https://arxiv.org/html/2605.24357#S5.E17)\) and*variance*\([18](https://arxiv.org/html/2605.24357#S5.E18)\)\. In particular, the stochasticity of the actor update is fully absorbed into the descent term‖∇J~λ​\(θk\)‖22\\\|\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\\|\_\{2\}^\{2\}\. Moreover, when the critic is learned exactly, both the bias and variance terms vanish, and we recover the linear convergence regime of[Theorem1](https://arxiv.org/html/2605.24357#Thmtheorem1)\. This stands in sharp contrast to recent actor–critic analyses, e\.g\.\(Kumaret al\.,[2024](https://arxiv.org/html/2605.24357#bib.bib1)\), which do not establish variance reduction and do not transfer the critic’s estimation error to the actor’s variance\. Thus, the central challenge is to obtain a sufficiently accurate critic\. Next, we bound the critic’s mean\-squared error\.

#### Updating the Critic\.

Before deriving a bound on the Mean Squared error of the critic, we need to:\(1\) establish sufficient state\-action exploration; \(2\) track how the switch of policy affects the switch of the critic target, which is what we do subsequently\. The state\-action exploration is often used as an assumption in prior analysis of actor\-critic, see e\.g\.\(Kumaret al\.,[2024](https://arxiv.org/html/2605.24357#bib.bib1); Chen and Zhao,[2023](https://arxiv.org/html/2605.24357#bib.bib11)\)\. Additionally, we emphasize that assuming such a property in the unregularized setting is contradictory, as it requires maintaining exploratory policies during the learning process while the goal is to learn a deterministic policy\. Thanks to the regularization and the projection operator, we relax this assumption\.Using only the sufficient state\-exploration, we establish the sufficient state\-action exploration condition\. The proof of the following lemma and all subsequent results are provided in[AppendixD](https://arxiv.org/html/2605.24357#A4)\.

###### Lemma 8\.

Assume[Aρ\\textbf\{A\}\_\{\\rho\}](https://arxiv.org/html/2605.24357#ThmassumMDPx1)\. Fork≥0k\\geq 0,v∈ℝ\|𝒮\|​\|𝒜\|v\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}, it holds

⟨𝖣θk​\(Id−γ​𝖯~θk\)​v,v⟩≥12​\(1−γ\)2​ρmin​τλ​‖v‖22,\\displaystyle\\langle\\mathsf\{D\}\_\{\\theta\_\{k\}\}\(\\mathrm\{Id\}\-\\gamma\\widetilde\{\\mathsf\{P\}\}\_\{\\theta\_\{k\}\}\)v,v\\rangle\\geq\\frac\{1\}\{2\}\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}\\left\\\|v\\right\\\|\_\{2\}^\{2\}\\kern 5\.0pt,where𝖣θ​=Δ​diag⁡\(\(dρθ​\(s\)​πθ​\(a\|s\)\)s,a\)\\mathsf\{D\}\_\{\\theta\}\\overset\{\\Delta\}\{=\}\\operatorname\{diag\}\(\\left\(d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)\\pi\_\{\\theta\}\(a\|s\)\\right\)\_\{s,a\}\)and𝖯~θk\\widetilde\{\\mathsf\{P\}\}\_\{\\theta\_\{k\}\}is a matrix of size\|𝒮\|​\|𝒜\|×\|𝒮\|​\|𝒜\|\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\\times\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|defined for\(s,a,s~,a~\)∈𝒮×𝒜×𝒮×𝒜\(s,a,\\tilde\{s\},\\tilde\{a\}\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\\times\\mathcal\{S\}\\times\\mathcal\{A\}, by𝖯~θ​\(s~,a~\|s,a\)​=Δ​𝖯​\(s~\|s,a\)​πθ​\(a~\|s~\)\\widetilde\{\\mathsf\{P\}\}\_\{\\theta\}\(\\tilde\{s\},\\tilde\{a\}\|s,a\)\\overset\{\\Delta\}\{=\}\\mathsf\{P\}\(\\tilde\{s\}\|s,a\)\\pi\_\{\\theta\}\(\\tilde\{a\}\|\\tilde\{s\}\)\.

Next, we bound the distance between the regularized q\-functions of two successive policies\.

###### Lemma 9\.

Assume[Aρ\\textbf\{A\}\_\{\\rho\}](https://arxiv.org/html/2605.24357#ThmassumMDPx1)\. It holds that

‖q~θk\+1λ−q~θkλ‖2≤C~λ​η𝖺​\|a^k​\(Sk\+1,Ak\+1\)\|,\\displaystyle\\left\\\|\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\+1\}\}\}^\{\\lambda\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}\\leq\\tilde\{C\}\_\{\\lambda\}\\eta\_\{\\mathsf\{a\}\}\\left\|\\hat\{\\mathrm\{a\}\}\_\{k\}\(S\_\{k\+1\},A\_\{k\+1\}\)\\right\|\\kern 5\.0pt,whereC~λ\\tilde\{C\}\_\{\\lambda\}is a coefficient that depends only on the problem parameters \(see[Corollary4](https://arxiv.org/html/2605.24357#Thmcorollary4)for the exact expression\)\.

Finally, we derive a bound on the MSE of the critic\.

###### Lemma 10\.

Assume[Aρ\\textbf\{A\}\_\{\\rho\}](https://arxiv.org/html/2605.24357#ThmassumMDPx1)and assume thatη𝖼≤\(1−γ\)2​ρmin​τλ/40\\eta\_\{\\mathsf\{c\}\}\\leq\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}/40andH≥2η𝖼​μ~𝖼​log⁡\(2\+4​C~λ2​η𝖺2\)H\\geq\\frac\{2\}\{\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\\log\(2\+4\\tilde\{C\}\_\{\\lambda\}^\{2\}\\eta\_\{\\mathsf\{a\}\}^\{2\}\)\. For anyk≥0k\\geq 0, it holds that

𝔼​\[‖q^k−q~θkλ‖22\]≲\(1−η𝖼​μ~𝖼\)H​\(k\+1\)/2​‖q^−1−q~θ0λ‖22\\displaystyle\\mathbb\{E\}\\left\[\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\\right\]\\lesssim\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\(k\+1\)/2\}\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{\-1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{0\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\+C~λ2η𝖺2\(1\+λ2log\(\|𝒜\|\)2\)\(1−γ\)2​11−\(1−η𝖼​μ~𝖼\)H/2\+η𝖼​σ𝖼2μ~𝖼,\\displaystyle\\qquad\\quad\+\\tfrac\{\\tilde\{C\}\_\{\\lambda\}^\{2\}\\eta\_\{\\mathsf\{a\}\}^\{2\}\(1\+\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\)\}\{\(1\-\\gamma\)^\{2\}\}\\tfrac\{1\}\{1\-\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H/2\}\}\+\\\!\\tfrac\{\\eta\_\{\\mathsf\{c\}\}\\sigma\_\{\\mathsf\{c\}\}^\{2\}\}\{\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\\kern 5\.0pt,whereμ~𝖼​=Δ​\(1−γ\)2​ρmin​τλ/2\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\\overset\{\\Delta\}\{=\}\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}/2, and the variance term is defined asσ𝖼2​=Δ​36\+4λ2\+36λ2log\(\|𝒜\|\)2\(1−γ\)2\\sigma\_\{\\mathsf\{c\}\}^\{2\}\\overset\{\\Delta\}\{=\}\\frac\{36\+4\\lambda^\{2\}\+36\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\}\{\(1\-\\gamma\)^\{2\}\}\.

The preceding lemma shows that, withHHTD steps per actor update, the critic tracks the moving targetq~θkλ\\tilde\{q\}^\{\\lambda\}\_\{\\theta\_\{k\}\}: the MSE𝔼​‖q^k−q~θkλ‖22\\mathbb\{E\}\\\|\\hat\{q\}\_\{k\}\-\\tilde\{q\}^\{\\lambda\}\_\{\\theta\_\{k\}\}\\\|\_\{2\}^\{2\}contracts geometrically from initialization up to a steady state\. The residual error decomposes into a drift termO​\(ηa2\)O\(\\eta\_\{a\}^\{2\}\)\(due to the actor moving the target\) and a stochastic TD floorO​\(ηc\)O\(\\eta\_\{c\}\); takingHHsufficiently large preserves accurate critic estimation along the actor trajectory\.

#### Convergence of Actor\-Critic\.

Combining the two previous recursions allows us to get the following convergence rate for[Ent\-AC](https://arxiv.org/html/2605.24357#alg1)\. The proof is provided in[AppendixE](https://arxiv.org/html/2605.24357#A5)\.

###### Theorem 2\.

Assume[Aρ\\textbf\{A\}\_\{\\rho\}](https://arxiv.org/html/2605.24357#ThmassumMDPx1)and assume thatη𝖼≤\(1−γ\)2​ρmin​τλ/40\\eta\_\{\\mathsf\{c\}\}\\leq\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}/40,H≥2η𝖼​μ~𝖼​log⁡\(2\+4​C~λ2​η𝖺2\)H\\geq\\frac\{2\}\{\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\\log\(2\+4\\tilde\{C\}\_\{\\lambda\}^\{2\}\\eta\_\{\\mathsf\{a\}\}^\{2\}\), and thatη𝖺≤\(1−γ\)​ρmin​τλ8​L\\eta\_\{\\mathsf\{a\}\}\\leq\\frac\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\{8L\}\. For anyK≥0K\\geq 0, it holds that

𝔼​\[J~λ⋆−J~λ​\(θK\)\]≲\(1−η𝖺​μ¯~λ8\)K​\[J~λ⋆−J~λ​\(θ0\)\]\\displaystyle\\mathbb\{E\}\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{K\}\)\\right\]\\lesssim\\left\(1\-\\tfrac\{\\eta\_\{\\mathsf\{a\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\}\{8\}\\right\)^\{K\}\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{0\}\)\\right\]\+L​η𝖺2​K\(1−γ\)2max\(1−η𝖺​μ¯~λ8,\(1−η𝖼μ~𝖼\)H/2\)K∥q^−1−q~θ0λ∥22\\displaystyle\+\\\!\\\!\\tfrac\{L\\eta\_\{\\mathsf\{a\}\}^\{2\}K\}\{\(1\-\\gamma\)^\{2\}\}\\max\\left\(1\\\!\-\\\!\\tfrac\{\\eta\_\{\\mathsf\{a\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\}\{8\},\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H/2\}\\right\)^\{K\}\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{\-1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{0\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\+\(1−η𝖼​μ~𝖼\)Hμ¯~λ​\(1−γ\)2​B\+η𝖺3​C~λ2L\(1\+λ2log\(\|𝒜\|\)2\)μ¯~λ​\(1−γ\)4\+L​η𝖺​η𝖼​σ𝖼2\(1−γ\)2​μ~𝖼​μ¯~λ,\\displaystyle\+\\tfrac\{\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}\}\{\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\(1\-\\gamma\)^\{2\}\}B\+\\eta\_\{\\mathsf\{a\}\}^\{3\}\\tfrac\{\\tilde\{C\}\_\{\\lambda\}^\{2\}L\(1\+\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\)\}\{\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\(1\-\\gamma\)^\{4\}\}\+\\tfrac\{L\\eta\_\{\\mathsf\{a\}\}\\eta\_\{\\mathsf\{c\}\}\\sigma\_\{\\mathsf\{c\}\}^\{2\}\}\{\(1\-\\gamma\)^\{2\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\}\\kern 5\.0pt,whereBBis a constant that depends only on the problem parameters, whose complete expression is provided in \([31](https://arxiv.org/html/2605.24357#A4.E31)\)\.

The previous bound clearly separates the contribution of four effects\. First, the first two terms quantify the*geometric forgetting*of the initialization error: the suboptimality contracts at a linear rate, and the influence of the initial critic mismatch is washed out at the slower of the actor and critic contraction factors\. Second, the term\(1−η𝖼​μ~𝖼\)Hμ¯~λ​\(1−γ\)2​B\\frac\{\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}\}\{\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\(1\-\\gamma\)^\{2\}\}\\,Bcaptures the*bias of the critic*, which arises when the inner TD loop is not run long enough to accurately track the current policy\. Third, the bound contains a*policy\-switch*contribution of orderO~​\(η𝖺3\)\\tilde\{O\}\(\\eta\_\{\\mathsf\{a\}\}^\{3\}\), which accounts for the higher\-order cost induced by changing the target that the critic must estimate\. Finally, the remaining term corresponds to the*variance of the critic*\(scaling likeη𝖺​η𝖼\\eta\_\{\\mathsf\{a\}\}\\eta\_\{\\mathsf\{c\}\}\), i\.e\., the stochastic TD noise floor propagated to the actor\.

A key takeaway is that the actor’s own sampling variance does not appear explicitly: it is fully absorbed by the descent term and is effectively replaced by the critic’s*bias*and*variance*\. Consequently, ifHHis too small, the multiplicative factor\(1−η𝖼​μ~𝖼\)H\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}remains sizable and the resulting bias term can prevent convergence to the optimal solution\. In contrast, whenHHis sufficiently large, the critic bias becomes negligible, and the limiting behavior is dominated by the variance floor\. This suggests choosingHHlarge enough to remove the bias, after which additional critic steps mainly improve the transient but do not change the asymptotic noise\-dominated regime\. Next, we derive the sample complexity of the entropy\-regularized problem

###### Corollary 2\.

Assume[Aρ\\textbf\{A\}\_\{\\rho\}](https://arxiv.org/html/2605.24357#ThmassumMDPx1)\. Letϵ\>0\\epsilon\>0, and set

η𝖺≲min⁡\(\(1−γ\)​ρmin​τλL,μ¯~λ1/3​\(1−γ\)4/3​ϵ1/3​C~λ−2/3L1/3​\(1\+λ​log⁡\(\|𝒜\|\)\)2/3,μ~𝖼​μ¯~λ​ϵL​σ𝖼2​ρmin​τλ\),\\displaystyle\\textstyle\\eta\_\{\\mathsf\{a\}\}\\lesssim\\min\\Bigl\(\\frac\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\{L\},\\frac\{\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}^\{1/3\}\(1\-\\gamma\)^\{4/3\}\\epsilon^\{1/3\}\\tilde\{C\}\_\{\\lambda\}^\{\-2/3\}\}\{L^\{1/3\}\(1\+\\lambda\\log\(\|\\mathcal\{A\}\|\)\)^\{2/3\}\},\\frac\{\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\\epsilon\}\{L\\sigma\_\{\\mathsf\{c\}\}^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\\Bigr\),as well asη𝖼≲\(1−γ\)2​ρmin​τλ\\eta\_\{\\mathsf\{c\}\}\\lesssim\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}\. In this case,[Ent\-AC](https://arxiv.org/html/2605.24357#alg1), achieves𝔼​\[J~λ⋆−J~λ​\(θK\)\]≤ϵ\\mathbb\{E\}\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{K\}\)\\right\]\\leq\\epsilon, with a number of critic updates per actor update of

H≳max⁡\{log⁡\(1\+C~λ2​\(1−γ\)2​ρmin2​τλ2L2\),log⁡\(Bμ¯~λ​\(1−γ\)2​ϵ\)\}\(1−γ\)2​ρmin​τλ​μ~𝖼,\\displaystyle H\\gtrsim\\frac\{\\max\\Bigl\\\{\\log\\Bigl\(1\+\\tfrac\{\\tilde\{C\}\_\{\\lambda\}^\{2\}\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}^\{2\}\\tau\_\{\\lambda\}^\{2\}\}\{L^\{2\}\}\\Bigr\),\\log\\\!\\left\(\\frac\{B\}\{\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\(1\-\\gamma\)^\{2\}\\epsilon\}\\right\)\\Bigr\\\}\}\{\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}\\,\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\},and a total number of actor updates of

K\\displaystyle K≳max⁡\(L​μ¯~λ−1​τλ−1\(1−γ\)​ρmin,C~λ2/3​L​\(1\+λ​log⁡\(\|𝒜\|\)\)2/3μ¯~λ5/3​\(1−γ\)4/3​ϵ1/3,L​σ𝖼2​ρmin​τλμ~𝖼​μ¯~λ2​ϵ\)\\displaystyle\\textstyle\\gtrsim\\max\\Bigl\(\\frac\{L\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}^\{\-1\}\\tau\_\{\\lambda\}^\{\-1\}\}\{\(1\-\\gamma\)\\rho\_\{\\min\}\},\\frac\{\\tilde\{C\}\_\{\\lambda\}^\{2/3\}L\\bigl\(1\+\\lambda\\log\(\|\\mathcal\{A\}\|\)\\bigr\)^\{2/3\}\}\{\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}^\{5/3\}\(1\-\\gamma\)^\{4/3\}\\epsilon^\{1/3\}\},\\frac\{L\\sigma\_\{\\mathsf\{c\}\}^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\{\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}^\{2\}\\epsilon\}\\Bigr\)×max⁡\{log⁡\(\(J~λ⋆−J~λ​\(θ0\)\)ϵ\),log⁡\(ρmin​τλ​‖q^−1−q~θ0λ‖22\(1−γ\)​μ¯~λ​ϵ\)\},\\displaystyle\\textstyle\\times\\max\\Bigl\\\{\\log\\\!\\left\(\\frac\{\(\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{0\}\)\)\}\{\\epsilon\}\\right\),\\ \\log\\\!\\left\(\\frac\{\\rho\_\{\\min\}\\tau\_\{\\lambda\}\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{\-1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.51534pt\{\\theta\_\{0\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\}\{\(1\-\\gamma\)\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\\epsilon\}\\right\)\\Bigr\\\},whereBBis a constant that depends only on the problem parameters, whose complete expression is provided in \([31](https://arxiv.org/html/2605.24357#A4.E31)\)\.

[Corollary2](https://arxiv.org/html/2605.24357#Thmcorollary2)shows that the*actor*\(outer\) loop enjoys the standard “linear\-with\-noise” complexity i\.e\., geometric contraction up to a variance\-limited1/ϵ1/\\epsilonregime driven by the*critic*noiseσc2\\sigma\_\{c\}^\{2\}\. Moreover, the*inner*critic loop needs onlyH=O~​\(log⁡\(1/ϵ\)\)H=\\tilde\{O\}\(\\log\(1/\\epsilon\)\)TD updates per actor step to reduce the critic bias belowϵ\\epsilon; beyond this, extra critic steps mainly improve transients without affecting the asymptotic rate\.

## 6Experiments

![Refer to caption](https://arxiv.org/html/2605.24357v1/x1.png)\(a\)Gridworld, size=2×2=2\\times 2
![Refer to caption](https://arxiv.org/html/2605.24357v1/x2.png)\(b\)Gridworld, size=3×3=3\\times 3
![Refer to caption](https://arxiv.org/html/2605.24357v1/x3.png)\(c\)Gridworld, size=3×4=3\\times 4
![Refer to caption](https://arxiv.org/html/2605.24357v1/x4.png)\(d\)Gridworld, size=4×4=4\\times 4
![Refer to caption](https://arxiv.org/html/2605.24357v1/x5.png)
![Refer to caption](https://arxiv.org/html/2605.24357v1/x6.png)\(e\)Synthetic,\|𝒮\|=4\|\\mathcal\{S\}\|=4
![Refer to caption](https://arxiv.org/html/2605.24357v1/x7.png)\(f\)Synthetic,\|𝒮\|=8\|\\mathcal\{S\}\|=8
![Refer to caption](https://arxiv.org/html/2605.24357v1/x8.png)\(g\)Synthetic,\|𝒮\|=12\|\\mathcal\{S\}\|=12
![Refer to caption](https://arxiv.org/html/2605.24357v1/x9.png)\(h\)Synthetic,\|𝒮\|=16\|\\mathcal\{S\}\|=16

Figure 1:Performance of[Ent\-AC](https://arxiv.org/html/2605.24357#alg1)across Gridworld and Synthetic environments\.We report the mean objective valueJ~λ​\(θk\)\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)as a function of actor iterations for varying critic update frequenciesH∈\{8,16,32,64\}H\\in\\\{8,16,32,64\\\}\. Panels \(a\)–\(d\) show results for tabular Gridworld layouts of increasing scale, while panels \(e\)–\(h\) illustrate performance on synthetic MDPs with varying state space sizes\|𝒮\|\|\\mathcal\{S\}\|and fixed action space of size\|𝒜\|=4\|\\mathcal\{A\}\|=4\. The ”Exact Critic” baseline \(dashed black line\) represents an oracle reference obtained by solving the critic to optimality\. Shaded regions denote one standard deviation across5050independent random seeds\. IncreasingHHconsistently reduces the approximation gap relative to the exact critic, which greatly enhances the performance of the algorithm\.In this section, we evaluate the empirical performance of[Ent\-AC](https://arxiv.org/html/2605.24357#alg1)across multiple tasks\. We focus on how the approximation of the regularized value function, controlled by the number of critic stepsHH, impacts the convergence and stability of the actor’s policy\. We compare our learned critic configurations against an “Exact Critic” oracle to establish a performance upper bound across varying environment complexities\. The code is available online at[https://github\.com/Labbi\-Safwan/Actor\-Critic](https://github.com/Labbi-Safwan/Actor-Critic)\. We describe below the two environments that we will use in the experiments\.

#### Experimental Setup\.

We start by describing the two environments we use, as well as the algorithmic setup\.

\(Synthetic\(Zhenget al\.,[2023](https://arxiv.org/html/2605.24357#bib.bib8)\)\.\)Thesyntheticenvironment is generated by sampling a dense tabular model\(𝖯,𝗋,ρ\)\(\\mathsf\{P\},\\mathsf\{r\},\\rho\)\. For each\(s,a\)∈𝒮×𝒜\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}, the transition kernel𝖯\(⋅\|s,a\)\\mathsf\{P\}\(\\cdot\|s,a\)is drawn uniformly at random from the\|𝒮\|\|\\mathcal\{S\}\|\-dimensional simplex, so that each action induces a distribution over all next states\. Rewards are sampled independently asR​\(s,a\)∼Unif​\[0,1\]R\(s,a\)\\sim\\mathrm\{Unif\}\[0,1\]and returned deterministically given\(s,a\)\(s,a\), and the initial distribution is set toρ​\(s\)=1/\|𝒮\|\\rho\(s\)=1/\|\\mathcal\{S\}\|\. We evaluate this environment in the discounted setting withγ=0\.99\\gamma=0\.99for four state\-space sizes\|𝒮\|∈\{4,8,12,16\}\|\\mathcal\{S\}\|\\in\\\{4,8,12,16\\\}, with a fixed action set of size\|𝒜\|=4\|\\mathcal\{A\}\|=4\.

\(Gridworld\(Domingueset al\.,[2021](https://arxiv.org/html/2605.24357#bib.bib9)\)\.\)We evaluate our method on a suite of tabular Gridworld layouts\(Domingueset al\.,[2021](https://arxiv.org/html/2605.24357#bib.bib9)\)with dimensionsM×N∈\{2×2,3×3,3×4,4×4\}M\\times N\\in\\\{2\\times 2,3\\times 3,3\\times 4,4\\times 4\\\}\. Each environment is modeled as a finite MDP where the state space𝒮\\mathcal\{S\}consists of discrete grid coordinates and the action space𝒜\\mathcal\{A\}comprises the four cardinal directions\. Transitions are deterministic; an actiona∈𝒜a\\in\\mathcal\{A\}moves the agent to the adjacent cell in the specified direction, or leaves the agent’s position unchanged if the move targets a boundary\. The agent is initialized at the bottom\-left coordinate and must navigate to a goal state at the top\-right of the grid\. The environment employs a sparse reward signal where the agent receivesR=1R=1only upon goal reaching, and receivesR=0R=0in any other case\.

\(Algorithmic Setup\.\)To ensure a rigorous comparison across different values ofH∈\{8,16,32,64\}H\\in\\\{8,16,32,64\\\}, we performed a grid search over actor and critic learning rates,η𝖺,η𝖼∈\{0\.003,0\.01,0\.03,0\.1\}\\eta\_\{\\mathsf\{a\}\},\\eta\_\{\\mathsf\{c\}\}\\in\\\{0\.003,0\.01,0\.03,0\.1\\\}\. The regularization parameter was fixed atλ=0\.05\\lambda=0\.05for all experiments\. In addition to evaluating[Ent\-AC](https://arxiv.org/html/2605.24357#alg1), we include an ”ideal critic” baseline, where the critic is set to the exact regularized value, to characterize the performance upper bound and highlight the impact of the critic’s bias and variance\. We report the mean objective valueJ~λ​\(θk\)\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)over50005000actor iterations, with shaded regions representing one standard deviation over5050independent runs\.

#### AC with exact critic converges fast\.

When the critic is exactly known, AC can leverage this strong baseline to consistently learn fast across all eight configurations of Gridworld \([Figures1\(a\)](https://arxiv.org/html/2605.24357#S6.F1.sf1),[1\(b\)](https://arxiv.org/html/2605.24357#S6.F1.sf2),[1\(c\)](https://arxiv.org/html/2605.24357#S6.F1.sf3)and[1\(d\)](https://arxiv.org/html/2605.24357#S6.F1.sf4)\) and Synthetic MDP \([Figures1\(e\)](https://arxiv.org/html/2605.24357#S6.F1.sf5),[1\(f\)](https://arxiv.org/html/2605.24357#S6.F1.sf6),[1\(g\)](https://arxiv.org/html/2605.24357#S6.F1.sf7)and[1\(h\)](https://arxiv.org/html/2605.24357#S6.F1.sf8)\)\. This is in line with our theory, which shows that AC enjoys performance comparable to using deterministic gradients when the critic is perfectly known\.

#### It pays off to learn the critic\.

Our experiments reveal a consistent trend: AC’s performance is strictly monotonic with respect to the number of critic stepsHH\. In all scenarios in[Figure1](https://arxiv.org/html/2605.24357#S6.F1), increasingHHfrom88to6464leads to faster convergence and higher objective values\. This phenomenon is particularly pronounced in the MDPs of smaller sizes, where lowerHHvalues \(e\.g\.,H=8H=8\) often result in a significant performance gap compared to the oracle\. This can be explained through the lens of critic bias and variance; whenHHis small, the critic’s estimate of the regularized q\-valueq~θkλ\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\theta\_\{k\}\}^\{\\lambda\}remains “cold” and fails to converge to the fixed point of the regularized Bellman operator\. In contrast, whenHHis larger, we obtain a more precise q\-value estimate, enabling more accurate policy updates\. Overall, while increasingHHincurs a higher computational cost per actor iteration, it consistently leads to superior policy performance:*learning the critic pays off*\.

## 7Conclusion

We established novel global convergence rates for actor\-critic in entropy\-regularized reinforcement learning, with a specific focus on the variance\-reduction phenomenon\. First, we proved that, when using a perfect critic as a baseline, AC achievesO​\(log⁡\(1/ϵ\)\)O\(\\log\(1/\\epsilon\)\)sample complexity\. To our knowledge, this is the first result proving that AC enjoys a*strong variance\-reduction*property, akin to methods like SVRG\(Johnson and Zhang,[2013](https://arxiv.org/html/2605.24357#bib.bib42)\)or SAGA\(Defazioet al\.,[2014](https://arxiv.org/html/2605.24357#bib.bib43)\)in stochastic optimization\. When no perfect critic is available, we show that most of the complexity of the algorithm amounts to learning the critic, and obtain the firstO​\(1/ϵ\)O\(1/\\epsilon\)rates for AC\. Our results shed new light on AC, confirming previous empirical evidence that estimation of the critic is crucial for good performance\. These results open new perspectives for AC, where it remains unknown whether its properties remain beyond the tabular case\. A promising research direction is to extend our results to the unregularized case and try to achieve faster rates by combining our approach with acceleration techniques for faster estimation of the critic and actor simultaneously\.

## Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning\. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here\.

## References

- A\. Agarwal, S\. M\. Kakade, J\. D\. Lee, and G\. Mahajan \(2021\)On the theory of policy gradient methods: optimality, approximation, and distribution shift\.Journal of Machine Learning Research22\(98\),pp\. 1–76\.Cited by:[§1](https://arxiv.org/html/2605.24357#S1.p1.1),[§3](https://arxiv.org/html/2605.24357#S3.SS0.SSS0.Px3.p3.1)\.
- I\. Baird and C\. Leemon \(1993\)Advantage updating\.Technical reportWRIGHT LAB WRIGHT\-PATTERSON AFB OH\.Cited by:[§1](https://arxiv.org/html/2605.24357#S1.p2.1)\.
- A\. Barakat, I\. Fatkhullin, and N\. He \(2023\)Reinforcement learning with general utilities: simpler variance reduction and large state\-action space\.InInternational Conference on Machine Learning,pp\. 1753–1800\.Cited by:[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px2.p1.1.2)\.
- A\. G\. Barto, R\. S\. Sutton, and C\. W\. Anderson \(1983\)Neuronlike adaptive elements that can solve difficult learning control problems\.IEEE Transactions on Systems, Man, and CyberneticsSMC\-13\(5\),pp\. 834–846\.External Links:[Document](https://dx.doi.org/10.1109/TSMC.1983.6313077)Cited by:[§1](https://arxiv.org/html/2605.24357#S1.p2.1),[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px2.p1.1.2)\.
- S\. Bhatnagar, R\. S\. Sutton, M\. Ghavamzadeh, and M\. Lee \(2009\)Natural actor–critic algorithms\.Automatica45\(11\),pp\. 2471–2482\.Cited by:[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px3.p1.5)\.
- D\. D\. Castro and R\. Meir \(2010\)A convergent online single time scale actor critic algorithm\.The Journal of Machine Learning Research11,pp\. 367–410\.Cited by:[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px3.p1.5)\.
- S\. Cayci, N\. He, and R\. Srikant \(2024\)Finite\-time analysis of entropy\-regularized neural natural actor\-critic algorithm\.Trans\. Mach\. Learn\. Res\.\.Cited by:[Table 1](https://arxiv.org/html/2605.24357#S1.T1.8.6.3.1.1),[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px3.p1.5)\.
- X\. Chen and L\. Zhao \(2023\)Finite\-time analysis of single\-timescale actor\-critic\.Advances in Neural Information Processing Systems36,pp\. 7017–7049\.Cited by:[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px3.p1.5),[§5](https://arxiv.org/html/2605.24357#S5.SS0.SSS0.Px2.p1.1)\.
- A\. Defazio, F\. Bach, and S\. Lacoste\-Julien \(2014\)SAGA: a fast incremental gradient method with support for non\-strongly convex composite objectives\.Advances in neural information processing systems27\.Cited by:[2nd item](https://arxiv.org/html/2605.24357#S1.I1.i2.p1.1),[§7](https://arxiv.org/html/2605.24357#S7.p1.2)\.
- Y\. Ding, J\. Zhang, H\. Lee, and J\. Lavaei \(2025\)Beyond exact gradients: convergence of stochastic soft\-max policy gradient methods with entropy regularization\.IEEE Transactions on Automatic Control\.Cited by:[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px2.p1.1)\.
- O\. D\. Domingues, Y\. Flet\-Berliac, E\. Leurent, P\. Ménard, X\. Shang, and M\. Valko \(2021\)rlberry \- A Reinforcement Learning Library for Research and Education\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.5544540),[Link](https://github.com/rlberry-py/rlberry)Cited by:[§6](https://arxiv.org/html/2605.24357#S6.SS0.SSS0.Px1.p3.6),[§6](https://arxiv.org/html/2605.24357#S6.SS0.SSS0.Px1.p3.6.1)\.
- I\. Fatkhullin, A\. Barakat, A\. Kireeva, and N\. He \(2023\)Stochastic policy gradient methods: improved sample complexity for fisher\-non\-degenerate policies\.InInternational Conference on Machine Learning,pp\. 9827–9869\.Cited by:[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px2.p1.1.2)\.
- M\. Gaur, A\. Bedi, D\. Wang, and V\. Aggarwal \(2024\)Closing the gap: achieving global convergence \(Last iterate\) of actor\-critic under Markovian sampling with neural network parametrization\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 15153–15179\.External Links:[Link](https://proceedings.mlr.press/v235/gaur24a.html)Cited by:[Table 1](https://arxiv.org/html/2605.24357#S1.T1.6.4.3.1.1),[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px3.p1.5)\.
- M\. Geist, B\. Scherrer, and O\. Pietquin \(2019\)A theory of regularized markov decision processes\.InInternational conference on machine learning,pp\. 2160–2169\.Cited by:[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2605.24357#S3.SS0.SSS0.Px2.p1.3)\.
- E\. Greensmith, P\. L\. Bartlett, and J\. Baxter \(2004\)Variance reduction techniques for gradient estimates in reinforcement learning\.Journal of Machine Learning Research5\(Nov\),pp\. 1471–1530\.Cited by:[§1](https://arxiv.org/html/2605.24357#S1.p3.1)\.
- I\. Grondman, L\. Busoniu, G\. A\. Lopes, and R\. Babuska \(2012\)A survey of actor\-critic reinforcement learning: standard and natural policy gradients\.IEEE Transactions on Systems, Man, and Cybernetics, part C \(applications and reviews\)42\(6\),pp\. 1291–1307\.Cited by:[§1](https://arxiv.org/html/2605.24357#S1.p2.1)\.
- T\. Haarnoja, A\. Zhou, P\. Abbeel, and S\. Levine \(2018\)Soft actor\-critic: off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.InInternational conference on machine learning,pp\. 1861–1870\.Cited by:[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px1.p1.1)\.
- R\. Johnson and T\. Zhang \(2013\)Accelerating stochastic gradient descent using predictive variance reduction\.Advances in neural information processing systems26\.Cited by:[2nd item](https://arxiv.org/html/2605.24357#S1.I1.i2.p1.1),[§7](https://arxiv.org/html/2605.24357#S7.p1.2)\.
- B\. Kerimkulov, J\. Leahy, D\. Siska, L\. Szpruch, and Y\. Zhang \(2025\)A fisher–rao gradient flow for entropy\-regularised markov decision processes in polish spaces: b\. kerimkulov et al\.\.Foundations of Computational Mathematics,pp\. 1–75\.Cited by:[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px3.p1.5.1)\.
- V\. Konda and J\. Tsitsiklis \(1999\)Actor\-critic algorithms\.Advances in neural information processing systems12\.Cited by:[§1](https://arxiv.org/html/2605.24357#S1.p2.1),[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px2.p1.1.2),[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px3.p1.5)\.
- H\. Kumar, A\. Koppel, and A\. Ribeiro \(2023\)On the sample complexity of actor\-critic method for reinforcement learning with function approximation\.Machine Learning112\(7\),pp\. 2433–2467\.Cited by:[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px3.p1.5)\.
- N\. Kumar, P\. Agrawal, G\. Ramponi, K\. Y\. Levy, and S\. Mannor \(2024\)On the convergence of single\-timescale actor\-critic\.arXiv preprint arXiv:2410\.08868\.Cited by:[Appendix D](https://arxiv.org/html/2605.24357#A4.p1.4),[Table 1](https://arxiv.org/html/2605.24357#S1.T1.5.3.3.1.1),[§1](https://arxiv.org/html/2605.24357#S1.p3.1),[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px3.p1.5),[§5](https://arxiv.org/html/2605.24357#S5.SS0.SSS0.Px1.p2.1),[§5](https://arxiv.org/html/2605.24357#S5.SS0.SSS0.Px2.p1.1)\.
- S\. Labbi, P\. Mangold, D\. Tiapkin, and E\. Moulines \(2026a\)On global convergence rates for federated softmax policy gradient under heterogeneous environments\.InThe 29th International Conference on Artificial Intelligence and Statistics,External Links:[Link](https://openreview.net/forum?id=aPAq9U7MFm)Cited by:[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Labbi, D\. Tiapkin, P\. Mangold, and E\. Moulines \(2026b\)Beyond softmax and entropy: convergence rates of policy gradients with f\-softargmax parameterization&\\&coupled regularization\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=O93c9H4SXc)Cited by:[§1](https://arxiv.org/html/2605.24357#S1.p3.1),[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px2.p1.1),[§4](https://arxiv.org/html/2605.24357#S4.SS0.SSS0.Px1.p1.6),[Remark 1](https://arxiv.org/html/2605.24357#Thmremark1.p1.3.3)\.
- G\. Lan \(2023\)Policy mirror descent for reinforcement learning: linear convergence, new sampling complexity, and generalized problem classes\.Mathematical programming198\(1\),pp\. 1059–1106\.Cited by:[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px1.p1.1.1)\.
- J\. Mei, C\. Xiao, B\. Dai, L\. Li, C\. Szepesvári, and D\. Schuurmans \(2020a\)Escaping the gravitational pull of softmax\.Advances in Neural Information Processing Systems33,pp\. 21130–21140\.Cited by:[§3](https://arxiv.org/html/2605.24357#S3.SS0.SSS0.Px3.p3.1)\.
- J\. Mei, C\. Xiao, C\. Szepesvari, and D\. Schuurmans \(2020b\)On the global convergence rates of softmax policy gradient methods\.InInternational conference on machine learning,pp\. 6820–6829\.Cited by:[§1](https://arxiv.org/html/2605.24357#S1.p3.1),[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2605.24357#S3.SS0.SSS0.Px3.p3.1),[Lemma 1](https://arxiv.org/html/2605.24357#Thmlemma1),[Lemma 2](https://arxiv.org/html/2605.24357#Thmlemma2),[Lemma 27](https://arxiv.org/html/2605.24357#Thmlemma27),[Lemma 3](https://arxiv.org/html/2605.24357#Thmlemma3),[Lemma 30](https://arxiv.org/html/2605.24357#Thmlemma30)\.
- V\. Mnih, A\. P\. Badia, M\. Mirza, A\. Graves, T\. Lillicrap, T\. Harley, D\. Silver, and K\. Kavukcuoglu \(2016\)Asynchronous methods for deep reinforcement learning\.InInternational conference on machine learning,pp\. 1928–1937\.Cited by:[§1](https://arxiv.org/html/2605.24357#S1.p2.1),[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px1.p1.1)\.
- W\. U\. Mondal and V\. Aggarwal \(2024\)Improved sample complexity analysis of natural policy gradient algorithm with general parameterization for infinite horizon discounted reward markov decision processes\.InInternational Conference on Artificial Intelligence and Statistics,pp\. 3097–3105\.Cited by:[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px2.p1.1.2)\.
- D\. Morales\-Brotons, T\. Vogels, and H\. Hendrikx \(2024\)Exponential moving average of weights in deep learning: dynamics and benefits\.arXiv preprint arXiv:2411\.18704\.Cited by:[1st item](https://arxiv.org/html/2605.24357#S1.I1.i1.p1.1)\.
- O\. Nachum, M\. Norouzi, K\. Xu, and D\. Schuurmans \(2017\)Bridging the gap between value and policy based reinforcement learning\.Advances in neural information processing systems30\.Cited by:[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2605.24357#S3.SS0.SSS0.Px2.p1.3)\.
- Y\. Nesterov \(2013\)Introductory lectures on convex optimization: a basic course\.Vol\.87,Springer Science & Business Media\.Cited by:[Lemma 23](https://arxiv.org/html/2605.24357#Thmlemma23)\.
- G\. Neu, A\. Jonsson, and V\. Gómez \(2017\)A unified view of entropy\-regularized markov decision processes\.arXiv preprint arXiv:1705\.07798\.Cited by:[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Olshevsky and B\. Gharesifard \(2023\)A small gain analysis of single timescale actor critic\.SIAM Journal on Control and Optimization61\(2\),pp\. 980–1007\.Cited by:[Table 1](https://arxiv.org/html/2605.24357#S1.T1.4.2.3.1.1),[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px3.p1.5)\.
- B\. T\. Polyak and A\. B\. Juditsky \(1992\)Acceleration of stochastic approximation by averaging\.SIAM journal on control and optimization30\(4\),pp\. 838–855\.Cited by:[1st item](https://arxiv.org/html/2605.24357#S1.I1.i1.p1.1)\.
- M\. L\. Puterman \(1994\)Discounted markov decision problems\.InMarkov Decision Processes,pp\. 142–276\.External Links:ISBN 9780470316887,[Document](https://dx.doi.org/https%3A//doi.org/10.1002/9780470316887.ch6),[Link](https://onlinelibrary.wiley.com/doi/abs/10.1002/9780470316887.ch6),https://onlinelibrary\.wiley\.com/doi/pdf/10\.1002/9780470316887\.ch6Cited by:[Appendix A](https://arxiv.org/html/2605.24357#A1.SS0.SSS0.Px1.p1.9),[Lemma 24](https://arxiv.org/html/2605.24357#Thmlemma24)\.
- S\. Qiu, Z\. Yang, J\. Ye, and Z\. Wang \(2021\)On finite\-time convergence of actor\-critic algorithm\.IEEE Journal on Selected Areas in Information Theory2\(2\),pp\. 652–664\.External Links:[Document](https://dx.doi.org/10.1109/JSAIT.2021.3078754)Cited by:[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px3.p1.5)\.
- J\. Schulman, S\. Levine, P\. Abbeel, M\. Jordan, and P\. Moritz \(2015a\)Trust region policy optimization\.InInternational conference on machine learning,pp\. 1889–1897\.Cited by:[§1](https://arxiv.org/html/2605.24357#S1.p2.1)\.
- J\. Schulman, P\. Moritz, S\. Levine, M\. Jordan, and P\. Abbeel \(2015b\)High\-dimensional continuous control using generalized advantage estimation\.arXiv preprint arXiv:1506\.02438\.Cited by:[§1](https://arxiv.org/html/2605.24357#S1.p2.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§1](https://arxiv.org/html/2605.24357#S1.p2.1)\.
- R\. S\. Sutton, A\. G\. Barto,et al\.\(1998\)Reinforcement learning: an introduction\.Vol\.1,MIT press Cambridge\.Cited by:[§1](https://arxiv.org/html/2605.24357#S1.p1.1)\.
- R\. S\. Sutton, D\. McAllester, S\. Singh, and Y\. Mansour \(1999\)Policy gradient methods for reinforcement learning with function approximation\.Advances in neural information processing systems12\.Cited by:[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px2.p1.1)\.
- T\. Wang, R\. Zhang, and S\. Gao \(2025\)Improving value estimation critically enhances vanilla policy gradient\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=Nq3oz7vn3j)Cited by:[2nd item](https://arxiv.org/html/2605.24357#S1.I3.i2.p1.1),[§1](https://arxiv.org/html/2605.24357#S1.p4.1)\.
- R\. J\. Williams and J\. Peng \(1991\)Function optimization using connectionist reinforcement learning algorithms\.Connection Science3\(3\),pp\. 241–268\.Cited by:[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px1.p1.1)\.
- R\. J\. Williams \(1992\)Simple statistical gradient\-following algorithms for connectionist reinforcement learning\.Machine learning8\(3\),pp\. 229–256\.Cited by:[§1](https://arxiv.org/html/2605.24357#S1.p1.1),[§1](https://arxiv.org/html/2605.24357#S1.p3.1),[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px2.p1.1)\.
- L\. Xiao \(2022\)On the convergence rates of policy gradient methods\.Journal of Machine Learning Research23\(282\),pp\. 1–36\.Cited by:[§1](https://arxiv.org/html/2605.24357#S1.p3.1),[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px2.p1.1)\.
- T\. Xu, Z\. Wang, and Y\. Liang \(2020\)Improving sample complexity bounds for \(natural\) actor\-critic algorithms\.Advances in Neural Information Processing Systems33,pp\. 4358–4369\.Cited by:[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px3.p1.5)\.
- Z\. Yang, Y\. Chen, M\. Hong, and Z\. Wang \(2019\)Provably global convergence of actor\-critic: a case for linear quadratic regulator with ergodic cost\.Advances in neural information processing systems32\.Cited by:[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px3.p1.5)\.
- R\. Yuan, R\. M\. Gower, and A\. Lazaric \(2022\)A general sample complexity analysis of vanilla policy gradient\.InInternational Conference on Artificial Intelligence and Statistics,pp\. 3332–3380\.Cited by:[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Zhang, A\. Koppel, A\. S\. Bedi, C\. Szepesvari, and M\. Wang \(2020\)Variational policy gradient method for reinforcement learning with general utilities\.Advances in Neural Information Processing Systems33,pp\. 4572–4583\.Cited by:[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Zhang, C\. Ni, C\. Szepesvari, M\. Wang,et al\.\(2021a\)On the convergence and sample efficiency of variance\-reduced policy gradient method\.Advances in Neural Information Processing Systems34,pp\. 2228–2240\.Cited by:[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Zhang, J\. Kim, B\. O’Donoghue, and S\. Boyd \(2021b\)Sample efficient reinforcement learning with reinforce\.InProceedings of the AAAI conference on artificial intelligence,Vol\.35,pp\. 10887–10895\.Cited by:[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px2.p1.1),[§4](https://arxiv.org/html/2605.24357#S4.SS0.SSS0.Px1.p1.6),[Remark 1](https://arxiv.org/html/2605.24357#Thmremark1.p1.3.3),[Remark 1](https://arxiv.org/html/2605.24357#Thmremark1.p1.9.6)\.
- Z\. Zheng, F\. Gao, L\. Xue, and J\. Yang \(2023\)Federated q\-learning: linear regret speedup with low communication cost\.InThe Twelfth International Conference on Learning Representations,Cited by:[§6](https://arxiv.org/html/2605.24357#S6.SS0.SSS0.Px1.p2.10.1)\.
- D\. Zorba, D\. Siska, and L\. Szpruch \(2026\)Convergence of an actor\-critic gradient flow for entropy regularised mdps in general spaces\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2605.24357#S2.SS0.SSS0.Px3.p1.5.1)\.

## Appendix ANotations

#### Distribution of the state\-action sequence\.

The state–action sequence\(St,At\)t≥0\(S\_\{t\},A\_\{t\}\)\_\{t\\geq 0\}defines a stochastic process on the canonical space\(𝒮×𝒜\)ℕ\(\\mathcal\{S\}\\times\\mathcal\{A\}\)^\{\\mathbb\{N\}\}\. For any initial states0∈𝒮s\_\{0\}\\in\\mathcal\{S\}, we denote byℙs0π\\mathbb\{P\}\_\{s\_\{0\}\}^\{\\pi\}the law of this process\. That is, for anyn∈ℕn\\in\\mathbb\{N\}and any subsetB⊂\(𝒮×𝒜\)nB\\subset\(\\mathcal\{S\}\\times\\mathcal\{A\}\)^\{n\},

ℙs0π​\(B\)=∑\(a0,…,an−1\)∈𝒜n∑\(s1,…,sn−1\)∈𝒮n−1𝟙B​\(\(s0,a0\),…,\(sn−1,an−1\)\)​∏i=0n−1π​\(ai∣si\)​𝖯​\(si\+1∣si,ai\),\\mathbb\{P\}\_\{s\_\{0\}\}^\{\\pi\}\(B\)=\\sum\_\{\(a\_\{0\},\\dots,a\_\{n\-1\}\)\\in\\mathcal\{A\}^\{n\}\}\\;\\sum\_\{\(s\_\{1\},\\dots,s\_\{n\-1\}\)\\in\\mathcal\{S\}^\{n\-1\}\}\\mathbbm\{1\}\_\{B\}\\\!\\bigl\(\(s\_\{0\},a\_\{0\}\),\\dots,\(s\_\{n\-1\},a\_\{n\-1\}\)\\bigr\)\\prod\_\{i=0\}^\{n\-1\}\\pi\(a\_\{i\}\\mid s\_\{i\}\)\\,\\mathsf\{P\}\(s\_\{i\+1\}\\mid s\_\{i\},a\_\{i\}\),with the conventions0s\_\{0\}is the given initial state\. We denote by𝔼s0π\\mathbb\{E\}\_\{s\_\{0\}\}^\{\\pi\}the corresponding expectation operator\. In particular, the state sequence\(st\)t≥0\(s\_\{t\}\)\_\{t\\geq 0\}defines a Markov reward process \(Section 2\.1\.6 in\(Puterman,[1994](https://arxiv.org/html/2605.24357#bib.bib2)\)\) with transition kernel

𝖯π​\(s′∣s\)=∑a∈𝒜𝖯​\(s′∣s,a\)​π​\(a∣s\)\.\\mathsf\{P\}\_\{\\pi\}\(s^\{\\prime\}\\mid s\)=\\sum\_\{a\\in\\mathcal\{A\}\}\\mathsf\{P\}\(s^\{\\prime\}\\mid s,a\)\\,\\pi\(a\\mid s\)\\kern 5\.0pt\.

#### Norms\.

Forx∈ℝdx\\in\\mathbb\{R\}^\{d\}, we define the norms

‖x‖∞=maxi∈\{1,…,d\}⁡\|xi\|,‖x‖1=∑i=1d\|xi\|,‖x‖2=\(∑i=1d\|xi\|2\)1/2\.\\displaystyle\\\|x\\\|\_\{\\infty\}=\\max\_\{i\\in\\\{1,\\dots,d\\\}\}\|x\_\{i\}\|\\kern 5\.0pt,\\quad\\\|x\\\|\_\{1\}=\\sum\_\{i=1\}^\{d\}\|x\_\{i\}\|\\kern 5\.0pt,\\quad\\\|x\\\|\_\{2\}=\\left\(\\sum\_\{i=1\}^\{d\}\|x\_\{i\}\|^\{2\}\\right\)^\{1/2\}\\kern 5\.0pt\.For ad×dd\\times dmatrixMM, we denote by‖M‖∞\\\|M\\\|\_\{\\infty\},‖M‖1\\\|M\\\|\_\{1\}, and‖M‖2\\\|M\\\|\_\{2\}respectively the max*row*sum, the max*column*sum, and the spectral norm:

‖M‖∞=supx≠0\{∥M​x∥∞/∥x∥∞\}=supi∈\{1,…,d\}∑j=1d\|Mi,j\|,‖M‖1=supx≠0\{∥M​x∥1/∥x∥1\}=supj∈\{1,…,d\}∑i=1d\|Mi,j\|,\\displaystyle\\\|M\\\|\_\{\\infty\}=\\sup\_\{x\\neq 0\}\\\{\\lVert Mx\\rVert\_\{\\infty\}/\\lVert x\\rVert\_\{\\infty\}\\\}=\\sup\_\{i\\in\\\{1,\\dots,d\\\}\}\\sum\_\{j=1\}^\{d\}\|M\_\{i,j\}\|\\kern 5\.0pt,\\quad\\\|M\\\|\_\{1\}=\\sup\_\{x\\neq 0\}\\\{\\lVert Mx\\rVert\_\{1\}/\\lVert x\\rVert\_\{1\}\\\}=\\sup\_\{j\\in\\\{1,\\dots,d\\\}\}\\sum\_\{i=1\}^\{d\}\|M\_\{i,j\}\|,\(19\)‖M‖2=supx≠0\{∥M​x∥2/∥x∥2\}\.\\displaystyle\\\|M\\\|\_\{2\}=\\sup\_\{x\\neq 0\}\\\{\\lVert Mx\\rVert\_\{2\}/\\lVert x\\rVert\_\{2\}\\\}\\kern 5\.0pt\.\(20\)Recall that, for anyx∈ℝdx\\in\\mathbb\{R\}^\{d\},‖M​x‖∞≤‖M‖∞​‖x‖∞\\\|Mx\\\|\_\{\\infty\}\\leq\\\|M\\\|\_\{\\infty\}\\\|x\\\|\_\{\\infty\}and‖M​x‖2≤‖M‖2​‖x‖2\\\|Mx\\\|\_\{2\}\\leq\\\|M\\\|\_\{2\}\\\|x\\\|\_\{2\}\.

#### KL divergence\.

For two discrete probability distributionsp=\(pi\)i=1np=\(p\_\{i\}\)\_\{i=1\}^\{n\}andq=\(qi\)i=1nq=\(q\_\{i\}\)\_\{i=1\}^\{n\}on a finite set\{1,…,n\}\\\{1,\\dots,n\\\}, the Kullback–Leibler \(KL\) divergence fromqqtoppis defined as

KL​\(p∥q\):=∑i=1npi​log⁡\(piqi\),\\mathrm\{KL\}\(p\\\|q\)\\;:=\\;\\sum\_\{i=1\}^\{n\}p\_\{i\}\\log\\\!\\left\(\\frac\{p\_\{i\}\}\{q\_\{i\}\}\\right\),with the convention that0​log⁡\(0/qi\)=00\\log\(0/q\_\{i\}\)=0andKL​\(p∥q\)=\+∞\\mathrm\{KL\}\(p\\\|q\)=\+\\inftyif there existsiisuch thatpi\>0p\_\{i\}\>0andqi=0q\_\{i\}=0\.

#### Functional and matrix forms\.

For notational convenience, we also view𝖯\\mathsf\{P\}as a\(\|𝒮\|⋅\|𝒜\|\)×\|𝒮\|\(\|\\mathcal\{S\}\|\\cdot\|\\mathcal\{A\}\|\)\\times\|\\mathcal\{S\}\|matrix with entries𝖯\(s,a\),s′=𝖯​\(s′∣s,a\)\\mathsf\{P\}\_\{\(s,a\),s^\{\\prime\}\}=\\mathsf\{P\}\(s^\{\\prime\}\\mid s,a\)\. Similarly,v~πλ\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\pi\}^\{\\lambda\}is a vector of size\|𝒮\|\|\\mathcal\{S\}\|andq~πλ\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\pi\}^\{\\lambda\}a vector of size\|𝒮\|×\|𝒜\|\|\\mathcal\{S\}\|\\times\|\\mathcal\{A\}\|\. Finally, we identify the parameterθ∈ℝ𝒮×𝒜\\theta\\in\\mathbb\{R\}^\{\\mathcal\{S\}\\times\\mathcal\{A\}\}with its vector representationθ∈ℝ\|𝒮\|​\|𝒜\|\\theta\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}, indexed by\(s,a\)∈𝒮×𝒜\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\. This slight abuse of notation allows us to conveniently switch between functional and matrix views\.

## Appendix BConvergence with exact critic

For any𝖺∈ℝ\|𝒮\|​\|𝒜\|\\mathsf\{a\}\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}andy=\(s,a\)∈𝒮×𝒜y=\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}, we define

\[g𝖺y​\(a\)\]\(x,b\)​=Δ​𝟣\(x,b\)​\(s,a\)​\(1−γ\)−1​a​\(s,a\)\\displaystyle\[\\mathrm\{g\}^\{y\}\_\{\\mathsf\{a\}\}\(a\)\]\_\{\(x,b\)\}\\overset\{\\Delta\}\{=\}\\mathsf\{1\}\_\{\(x,b\)\}\(s,a\)\\,\(1\-\\gamma\)^\{\-1\}\\,\\mathrm\{a\}\(s,a\)In the exact critic setting, the update of[Ent\-AC](https://arxiv.org/html/2605.24357#alg1)simplifies to:

θk\+1=𝒯​\(θk\+η𝖺​g𝖺Yk\+1​\(a~θ​kλ\)\),\\displaystyle\\theta\_\{k\+1\}\\;=\\;\\mathcal\{T\}\\\!\\big\(\\theta\_\{k\}\+\\eta\_\{\\mathsf\{a\}\}\\,\\mathrm\{g\}^\{Y\_\{k\+1\}\}\_\{\\mathsf\{a\}\}\(\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\{\\theta\{k\}\}\}^\{\\lambda\}\)\\big\)\\kern 5\.0pt,whereYk\+1∼ν𝖺​\(θ​k\+1\)Y\_\{k\+1\}\\sim\\nu^\{\\mathsf\{a\}\}\(\\theta\{k\+1\}\)\. We preface the proof of the convergence of[Ent\-AC](https://arxiv.org/html/2605.24357#alg1)in this setting, with the following key lemma:

###### Lemma 11\.

Assume[Aρ\\textbf\{A\}\_\{\\rho\}](https://arxiv.org/html/2605.24357#ThmassumMDPx1)and that for all\(s,a\)∈𝒮×𝒜\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}, we haveπθ​\(a\|s\)≥πmin\>0\\pi\_\{\\theta\}\(a\|s\)\\geq\\pi\_\{\\min\}\>0\. For anyθ∈ℝ\|𝒮\|​\|𝒜\|\\theta\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}, it holds that

𝔼Y∼ν𝖺​\(θ\)​\[g𝖺Y​\(a~θλ\)\]=∂J~λ​\(θ\)∂θ,𝔼Y∼ν𝖺​\(θ\)​\[‖g𝖺Y​\(a~θλ\)−∂J~λ​\(θ\)∂θ‖22\]≤1\(1−γ\)​πmin​ρmin​‖∂J~λ​\(θ\)∂θ‖22,\\displaystyle\\mathbb\{E\}\_\{Y\\sim\\nu^\{\\mathsf\{a\}\}\(\\theta\)\}\\left\[\\mathrm\{g\}^\{Y\}\_\{\\mathsf\{a\}\}\(\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\)\\right\]=\\frac\{\\partial\\tilde\{J\}\_\{\\lambda\}\(\\theta\)\}\{\\partial\\theta\}\\kern 5\.0pt,\\quad\\mathbb\{E\}\_\{Y\\sim\\nu^\{\\mathsf\{a\}\}\(\\theta\)\}\\left\[\\left\\\|\\mathrm\{g\}^\{Y\}\_\{\\mathsf\{a\}\}\(\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\)\-\\frac\{\\partial\\tilde\{J\}\_\{\\lambda\}\(\\theta\)\}\{\\partial\\theta\}\\right\\\|\_\{2\}^\{2\}\\right\]\\leq\\frac\{1\}\{\(1\-\\gamma\)\\pi\_\{\\min\}\\rho\_\{\\min\}\}\\left\\\|\\frac\{\\partial\\tilde\{J\}\_\{\\lambda\}\(\\theta\)\}\{\\partial\\theta\}\\right\\\|\_\{2\}^\{2\}\\kern 5\.0pt,whereν𝖺​\(θ\)\\nu^\{\\mathsf\{a\}\}\(\\theta\)is defined in \([15](https://arxiv.org/html/2605.24357#S3.E15)\)

###### Proof\.

Firstly, note that for any\(s,a\)∈𝒮×𝒜\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}, we have

\[𝔼Y∼ν𝖺​\(θ\)​\[g𝖺Y​\(a~θλ\)\]\]s,a=∑s′,a′dρθ​\(s′\)​πθ​\(a′\|s′\)​𝟣\(s,a\)​\(s′,a′\)​\(1−γ\)−1​a~θλ​\(s′,a′\)=∂J~λ​\(θ\)∂θ​\(s,a\),\\displaystyle\\left\[\\mathbb\{E\}\_\{Y\\sim\\nu^\{\\mathsf\{a\}\}\(\\theta\)\}\\left\[\\mathrm\{g\}^\{Y\}\_\{\\mathsf\{a\}\}\(\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\)\\right\]\\right\]\_\{s,a\}=\\sum\_\{s^\{\\prime\},a^\{\\prime\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s^\{\\prime\}\)\\pi\_\{\\theta\}\(a^\{\\prime\}\|s^\{\\prime\}\)\\mathsf\{1\}\_\{\(s,a\)\}\(s^\{\\prime\},a^\{\\prime\}\)\(1\-\\gamma\)^\{\-1\}\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\(s^\{\\prime\},a^\{\\prime\}\)=\\frac\{\\partial\\tilde\{J\}\_\{\\lambda\}\(\\theta\)\}\{\\partial\\theta\(s,a\)\}\\kern 5\.0pt,where the last equality follows from[Lemma1](https://arxiv.org/html/2605.24357#Thmlemma1)\. Next, we have

𝔼Y∼ν𝖺​\(θ\)​\[‖g𝖺Y​\(a~θλ\)−∂J~λ​\(θ\)∂θ‖22\]=𝔼Y∼ν𝖺​\(θ\)​\[‖g𝖺Y​\(a~θλ\)‖22\]−‖∂J~λ​\(θ\)∂θ‖22\\displaystyle\\mathbb\{E\}\_\{Y\\sim\\nu^\{\\mathsf\{a\}\}\(\\theta\)\}\\left\[\\left\\\|\\mathrm\{g\}^\{Y\}\_\{\\mathsf\{a\}\}\(\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\)\-\\frac\{\\partial\\tilde\{J\}\_\{\\lambda\}\(\\theta\)\}\{\\partial\\theta\}\\right\\\|\_\{2\}^\{2\}\\right\]=\\mathbb\{E\}\_\{Y\\sim\\nu^\{\\mathsf\{a\}\}\(\\theta\)\}\\left\[\\left\\\|\\mathrm\{g\}^\{Y\}\_\{\\mathsf\{a\}\}\(\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\)\\right\\\|\_\{2\}^\{2\}\\right\]\-\\left\\\|\\frac\{\\partial\\tilde\{J\}\_\{\\lambda\}\(\\theta\)\}\{\\partial\\theta\}\\right\\\|\_\{2\}^\{2\}=1\(1−γ\)2​∑\(s,a\)∈𝒮×𝒜dρθ​\(s\)​πθ​\(a\|s\)​\[a~θλ​\(s,a\)2−dρθ​\(s\)​πθ​\(a\|s\)​a~θλ​\(s,a\)2\]\.\\displaystyle=\\frac\{1\}\{\(1\-\\gamma\)^\{2\}\}\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)\\pi\_\{\\theta\}\(a\|s\)\\left\[\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\(s,a\)^\{2\}\-d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)\\pi\_\{\\theta\}\(a\|s\)\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\(s,a\)^\{2\}\\right\]\\kern 5\.0pt\.Using[Aρ\\textbf\{A\}\_\{\\rho\}](https://arxiv.org/html/2605.24357#ThmassumMDPx1)combined with the assumptionmin\(s,a\)∈𝒮×𝒜⁡πθ​\(a\|s\)≥πmin\\min\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}\\pi\_\{\\theta\}\(a\|s\)\\geq\\pi\_\{\\min\}, and[Lemma1](https://arxiv.org/html/2605.24357#Thmlemma1)concludes the proof\.∎

In the following, we define the filtration adapted to the iterates of[Ent\-AC](https://arxiv.org/html/2605.24357#alg1)as

ℱk=Δσ\(Yk:k∈\{1,…,K\}\)\.\\displaystyle\\mathcal\{F\}\_\{k\}\\overset\{\\Delta\}\{=\}\\sigma\\Big\(Y\_\{k\}:k\\in\\\{1,\\dots,K\\\}\\Big\)\\kern 5\.0pt\.
###### Theorem 3\.

Assume[Aρ\\textbf\{A\}\_\{\\rho\}](https://arxiv.org/html/2605.24357#ThmassumMDPx1)\. Fixη𝖺≤\(1−γ\)​ρmin​πmin/L\\eta\_\{\\mathsf\{a\}\}\\leq\(1\-\\gamma\)\\rho\_\{\\min\}\\pi\_\{\\min\}/Land consider the iterates of[Ent\-AC](https://arxiv.org/html/2605.24357#alg1)\. It holds thatmin\(s,a\)∈𝒮×𝒜⁡πθk​\(a\|s\)\>τλ\\min\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}\\pi\_\{\\theta\_\{k\}\}\(a\|s\)\>\\tau\_\{\\lambda\}almost surely\. Addtionnally, for anyk≥0k\\geq 0we have that

J~λ⋆−𝔼​\[J~λ​\(θk\)\]≤\[1−η𝖺​μ¯~λ\]k​\(J~λ⋆−J~λ​\(θ0\)\)\.\\displaystyle\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\mathbb\{E\}\\left\[\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\]\\leq\\left\[1\-\\eta\_\{\\mathsf\{a\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\\right\]^\{k\}\\left\(\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{0\}\)\\right\)\\kern 5\.0pt\.

###### Proof\.

Defineθ~k\+1=θk\+η𝖺​g𝖺Yk\+1​\(a~θkλ\)\\tilde\{\\theta\}\_\{k\+1\}=\\theta\_\{k\}\+\\eta\_\{\\mathsf\{a\}\}\\,\\mathrm\{g\}^\{Y\_\{k\+1\}\}\_\{\\mathsf\{a\}\}\(\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\)\. Using the monotone improvement property of𝒯\\mathcal\{T\}, provided in[Lemma21](https://arxiv.org/html/2605.24357#Thmlemma21), we have that

J~λ​\(θk\+1\)=J~λ​\(𝒯​\(θ~k\+1\)\)≥J~λ​\(θ~k\+1\)\.\\displaystyle\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\+1\}\)=\\tilde\{J\}\_\{\\lambda\}\(\\mathcal\{T\}\(\\tilde\{\\theta\}\_\{k\+1\}\)\)\\geq\\tilde\{J\}\_\{\\lambda\}\(\\tilde\{\\theta\}\_\{k\+1\}\)\\kern 5\.0pt\.Next, applying[Lemma2](https://arxiv.org/html/2605.24357#Thmlemma2), combined with[Lemma23](https://arxiv.org/html/2605.24357#Thmlemma23)yields

J~λ​\(θ~k\+1\)≥J~λ​\(θk\)\+2​η𝖺​⟨∇J~λ​\(θk\),g𝖺Yk\+1​\(a~θkλ\)⟩−η𝖺2​L2​‖g𝖺Yk\+1​\(a~θkλ\)‖22\.\\displaystyle\\tilde\{J\}\_\{\\lambda\}\(\\tilde\{\\theta\}\_\{k\+1\}\)\\geq\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\+2\\eta\_\{\\mathsf\{a\}\}\\langle\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\),\\mathrm\{g\}^\{Y\_\{k\+1\}\}\_\{\\mathsf\{a\}\}\(\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\)\\rangle\-\\frac\{\\eta\_\{\\mathsf\{a\}\}^\{2\}L\}\{2\}\\left\\\|\\mathrm\{g\}^\{Y\_\{k\+1\}\}\_\{\\mathsf\{a\}\}\(\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\)\\right\\\|\_\{2\}^\{2\}\\kern 5\.0pt\.Combining the two previous bounds, and taking the conditional expectation with respect toℱk\\mathcal\{F\}\_\{k\}gives

𝔼​\[J~λ​\(θk\+1\)\|ℱk\]\\displaystyle\\mathbb\{E\}\\left\[\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\+1\}\)\|\\mathcal\{F\}\_\{k\}\\right\]≥J~λ​\(θk\)\+2​η𝖺​‖∇J~λ​\(θk\)‖22−η𝖺2​L2​𝔼​\[‖g𝖺Yk\+1​\(a~θkλ\)‖22\|ℱk\]\\displaystyle\\geq\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\+2\\eta\_\{\\mathsf\{a\}\}\\left\\\|\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\-\\frac\{\\eta\_\{\\mathsf\{a\}\}^\{2\}L\}\{2\}\\mathbb\{E\}\\left\[\\left\\\|\\mathrm\{g\}^\{Y\_\{k\+1\}\}\_\{\\mathsf\{a\}\}\(\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\)\\right\\\|\_\{2\}^\{2\}\|\\mathcal\{F\}\_\{k\}\\right\]=J~λ​\(θk\)\+\(2​η𝖺−η𝖺2​L2\)​‖∇J~λ​\(θk\)‖22−η𝖺2​L2​𝔼​\[‖g𝖺Yk\+1​\(a~θkλ\)−∇J~λ​\(θk\)‖22\|ℱk\],\\displaystyle=\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\+\\left\(2\\eta\_\{\\mathsf\{a\}\}\-\\frac\{\\eta\_\{\\mathsf\{a\}\}^\{2\}L\}\{2\}\\right\)\\left\\\|\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\-\\frac\{\\eta\_\{\\mathsf\{a\}\}^\{2\}L\}\{2\}\\mathbb\{E\}\\left\[\\left\\\|\\mathrm\{g\}^\{Y\_\{k\+1\}\}\_\{\\mathsf\{a\}\}\(\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\)\-\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\|\\mathcal\{F\}\_\{k\}\\right\]\\kern 5\.0pt,where the last identity is obtained via the bias–variance decomposition of the estimatorg𝖺Yk\+1​\(a~θkλ\)\\mathrm\{g\}^\{Y\_\{k\+1\}\}\_\{\\mathsf\{a\}\}\(\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\)\. We then apply[Lemma11](https://arxiv.org/html/2605.24357#Thmlemma11)to obtain

𝔼​\[J~λ​\(θk\+1\)\|ℱk\]≥J~λ​\(θk\)\+\(2​η𝖺−η𝖺2​L2−η𝖺2​L2​\(1−γ\)​πmin​ρmin\)​‖∇J~λ​\(θk\)‖22\.\\displaystyle\\mathbb\{E\}\\left\[\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\+1\}\)\|\\mathcal\{F\}\_\{k\}\\right\]\\geq\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\+\\left\(2\\eta\_\{\\mathsf\{a\}\}\-\\frac\{\\eta\_\{\\mathsf\{a\}\}^\{2\}L\}\{2\}\-\\frac\{\\eta\_\{\\mathsf\{a\}\}^\{2\}L\}\{2\(1\-\\gamma\)\\pi\_\{\\min\}\\rho\_\{\\min\}\}\\right\)\\left\\\|\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\\kern 5\.0pt\.After subtractingJ~λ⋆\\tilde\{J\}^\{\\star\}\_\{\\lambda\}from both sides, we apply the non\-uniform PL inequality forJ~λ\\tilde\{J\}\_\{\\lambda\}recalled in[Lemma3](https://arxiv.org/html/2605.24357#Thmlemma3), along with the uniform lower bound on the PL coefficient in \([16](https://arxiv.org/html/2605.24357#S4.E16)\), to obtain

J~λ⋆−𝔼​\[J~λ​\(θk\+1\)\|ℱk\]\\displaystyle\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\mathbb\{E\}\\left\[\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\+1\}\)\|\\mathcal\{F\}\_\{k\}\\right\]≤\[1−μ¯~λ​\(2​η𝖺−η𝖺2​L2−η𝖺2​L2​\(1−γ\)​πmin​ρmin\)\]​\(J~λ⋆−J~λ​\(θk\)\)\\displaystyle\\leq\\left\[1\-\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\\left\(2\\eta\_\{\\mathsf\{a\}\}\-\\frac\{\\eta\_\{\\mathsf\{a\}\}^\{2\}L\}\{2\}\-\\frac\{\\eta\_\{\\mathsf\{a\}\}^\{2\}L\}\{2\(1\-\\gamma\)\\pi\_\{\\min\}\\rho\_\{\\min\}\}\\right\)\\right\]\(\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\)≤\[1−η𝖺​μ¯~λ\]​\(J~λ⋆−J~λ​\(θk\)\),\\displaystyle\\leq\\left\[1\-\\eta\_\{\\mathsf\{a\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\\right\]\(\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\)\\kern 5\.0pt,where the last inequality follows from the step\-size conditionη𝖺≤\(1−γ\)​ρmin​πminL\\eta\_\{\\mathsf\{a\}\}\\leq\\frac\{\(1\-\\gamma\)\\rho\_\{\\min\}\\pi\_\{\\min\}\}\{L\}\. Finally, taking expectation over all the randomness and unrolling the recursion gives the desired result\. ∎

Next, we derive the sample complexity of[Ent\-AC](https://arxiv.org/html/2605.24357#alg1), in case of this exact critic\.

###### Corollary 3\(Sample complexity\)\.

Under the same assumptions as Theorem[3](https://arxiv.org/html/2605.24357#Thmtheorem3), for anyϵ\>0\\epsilon\>0, settingη𝖺=\(1−γ\)​ρmin​πmin/L\\eta\_\{\\mathsf\{a\}\}=\(1\-\\gamma\)\\rho\_\{\\min\}\\pi\_\{\\min\}/L, and taking

k≥Lμ¯~λ​1\(1−γ\)​ρmin​τλ​log⁡\(J~λ⋆−J~λ​\(θ0\)ϵ\),\\displaystyle k\\geq\\frac\{L\}\{\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\}\\frac\{1\}\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\\log\\Big\(\\frac\{\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{0\}\)\}\{\\epsilon\}\\Big\)\\kern 5\.0pt,we getJ~λ⋆−𝔼​\[J~λ​\(θk\)\]≤ϵ\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\mathbb\{E\}\\left\[\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\]\\leq\\epsilon\.

## Appendix CGeneral Analysis of[Ent\-AC](https://arxiv.org/html/2605.24357#alg1): Actor Recursion

We define

g¯𝖺​\(θ,a\)​=Δ​𝔼Y∼ν𝖺​\(θ\)​\[g𝖺Y​\(a\)\]\.\\displaystyle\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{a\}\}\(\\theta,\\mathrm\{a\}\)\\overset\{\\Delta\}\{=\}\\mathbb\{E\}\_\{Y\\sim\\nu^\{\\mathsf\{a\}\}\(\\theta\)\}\\left\[\\mathrm\{g\}^\{Y\}\_\{\\mathsf\{a\}\}\(\\mathrm\{a\}\)\\right\]\\kern 5\.0pt\.We start with the following lemma that bounds the variance and the bias of the estimator in the presence of an inexact advantage used in the computation of the critic\.

###### Lemma 12\.

Fixθ∈ℝ\|𝒮\|​\|𝒜\|\\theta\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}anda∈ℝ\|𝒮\|​\|𝒜\|\\mathrm\{a\}\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}\. It holds that

‖g¯𝖺​\(θ,a\)−∂v~θλ∂θ‖22\\displaystyle\\left\\\|\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{a\}\}\(\\theta,\\mathrm\{a\}\)\-\\frac\{\\partial\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\}\{\\partial\\theta\}\\right\\\|\_\{2\}^\{2\}=1\(1−γ\)2​∑\(s,a\)∈𝒮×𝒜dρθ​\(s\)2​πθ​\(a\|s\)2​\(a​\(s,a\)−a~θλ​\(s,a\)\)2,\\displaystyle=\\frac\{1\}\{\(1\-\\gamma\)^\{2\}\}\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)^\{2\}\\pi\_\{\\theta\}\(a\|s\)^\{2\}\\left\(\\mathrm\{a\}\(s,a\)\-\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\(s,a\)\\right\)^\{2\}\\kern 5\.0pt,𝔼Y∼ν𝖺​\(θ\)​\[‖g𝖺Y​\(a\)−g¯𝖺​\(θ,a\)‖22\]\\displaystyle\\mathbb\{E\}\_\{Y\\sim\\nu^\{\\mathsf\{a\}\}\(\\theta\)\}\\left\[\\left\\\|\\mathrm\{g\}^\{Y\}\_\{\\mathsf\{a\}\}\(\\mathrm\{a\}\)\-\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{a\}\}\(\\theta,\\mathrm\{a\}\)\\right\\\|\_\{2\}^\{2\}\\right\]≤1\(1−γ\)2​∑\(s,a\)∈𝒮×𝒜dρθ​\(s\)​πθ​\(a\|s\)​a​\(s,a\)2\.\\displaystyle\\leq\\frac\{1\}\{\(1\-\\gamma\)^\{2\}\}\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)\\pi\_\{\\theta\}\(a\|s\)\\mathrm\{a\}\(s,a\)^\{2\}\\kern 5\.0pt\.

###### Proof\.

Firstly, note that for any\(s,a\)∈𝒮×𝒜\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}, we have

\[g¯𝖺​\(θ,a\)\]s,a\\displaystyle\[\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{a\}\}\(\\theta,\\mathrm\{a\}\)\]\_\{s,a\}=\[𝔼Y∼ν𝖺​\(θ\)​\[g𝖺Y​\(a\)\]\]s,a=∑s′,a′dρθ​\(s′\)​πθ​\(a′\|s′\)​𝟣\(s,a\)​\(s′,a′\)​\(1−γ\)−1​a​\(s′,a′\)\\displaystyle=\\left\[\\mathbb\{E\}\_\{Y\\sim\\nu^\{\\mathsf\{a\}\}\(\\theta\)\}\\left\[\\mathrm\{g\}^\{Y\}\_\{\\mathsf\{a\}\}\(\\mathrm\{a\}\)\\right\]\\right\]\_\{s,a\}=\\sum\_\{s^\{\\prime\},a^\{\\prime\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s^\{\\prime\}\)\\pi\_\{\\theta\}\(a^\{\\prime\}\|s^\{\\prime\}\)\\mathsf\{1\}\_\{\(s,a\)\}\(s^\{\\prime\},a^\{\\prime\}\)\(1\-\\gamma\)^\{\-1\}\\mathrm\{a\}\(s^\{\\prime\},a^\{\\prime\}\)=\(1−γ\)−1​dρθ​\(s\)​πθ​\(a\|s\)​a​\(s,a\)\.\\displaystyle=\(1\-\\gamma\)^\{\-1\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)\\pi\_\{\\theta\}\(a\|s\)\\mathrm\{a\}\(s,a\)\\kern 5\.0pt\.Combining the previous identity with[Lemma1](https://arxiv.org/html/2605.24357#Thmlemma1), yields the result\. Next, we have

𝔼Y∼ν𝖺​\(θ\)​\[‖g𝖺Y​\(a\)−g¯𝖺​\(θ,a\)‖22\]=𝔼Y∼ν𝖺​\(θ\)​\[‖g𝖺Y​\(a\)‖22\]−‖g¯𝖺​\(θ,a\)‖22\\displaystyle\\mathbb\{E\}\_\{Y\\sim\\nu^\{\\mathsf\{a\}\}\(\\theta\)\}\\left\[\\left\\\|\\mathrm\{g\}^\{Y\}\_\{\\mathsf\{a\}\}\(\\mathrm\{a\}\)\-\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{a\}\}\(\\theta,\\mathrm\{a\}\)\\right\\\|\_\{2\}^\{2\}\\right\]=\\mathbb\{E\}\_\{Y\\sim\\nu^\{\\mathsf\{a\}\}\(\\theta\)\}\\left\[\\left\\\|\\mathrm\{g\}^\{Y\}\_\{\\mathsf\{a\}\}\(\\mathrm\{a\}\)\\right\\\|\_\{2\}^\{2\}\\right\]\-\\left\\\|\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{a\}\}\(\\theta,\\mathrm\{a\}\)\\right\\\|\_\{2\}^\{2\}=1\(1−γ\)2​∑\(s,a\)∈𝒮×𝒜dρθ​\(s′\)​πθ​\(a′\|s′\)​\[a​\(s,a\)2−dρθ​\(s\)​πθ​\(a\|s\)​a​\(s,a\)2\],\\displaystyle=\\frac\{1\}\{\(1\-\\gamma\)^\{2\}\}\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s^\{\\prime\}\)\\pi\_\{\\theta\}\(a^\{\\prime\}\|s^\{\\prime\}\)\\left\[\\mathrm\{a\}\(s,a\)^\{2\}\-d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)\\pi\_\{\\theta\}\(a\|s\)\\mathrm\{a\}\(s,a\)^\{2\}\\right\]\\kern 5\.0pt,which concludes the proof\. ∎

Next, we define the two following filtrations

ℱk=Δσ\(Xℓ:ℓ∈\{0,…,k−1\},Yℓ:ℓ∈\{1,…,k\}\),𝒢k=Δσ\(Xℓ:ℓ∈\{1,…,k\},Yℓ:ℓ∈\{1,…,k\}\)\.\\displaystyle\\mathcal\{F\}\_\{k\}\\overset\{\\Delta\}\{=\}\\sigma\\Big\(X\_\{\\ell\}:\\ell\\in\\\{0,\\dots,k\-1\\\},Y\_\{\\ell\}:\\ell\\in\\\{1,\\dots,k\\\}\\Big\)\\kern 5\.0pt,\\quad\\mathcal\{G\}\_\{k\}\\overset\{\\Delta\}\{=\}\\sigma\\Big\(X\_\{\\ell\}:\\ell\\in\\\{1,\\dots,k\\\},Y\_\{\\ell\}:\\ell\\in\\\{1,\\dots,k\\\}\\Big\)\\kern 5\.0pt\.
In the latter, we derive a recursion on the error of the actor\.

###### Lemma 13\(Actor recursion\)\.

Assume[Aρ\\textbf\{A\}\_\{\\rho\}](https://arxiv.org/html/2605.24357#ThmassumMDPx1)\. For anyk≥0k\\geq 0and\(s,a\)∈𝒮×𝒜\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}, it holds that

πθk​\(a\|s\)≥τλ\.\\displaystyle\\pi\_\{\\theta\_\{k\}\}\(a\|s\)\\geq\\tau\_\{\\lambda\}\\kern 5\.0pt\.Additionally, it holds that

𝔼​\[J~λ⋆−J~λ​\(θk\+1\)\|ℱk\]\\displaystyle\\mathbb\{E\}\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\+1\}\)\\big\|\\mathcal\{F\}\_\{k\}\\right\]≤J~λ⋆−J~λ​\(θk\)−\(η𝖺2−2​L​η𝖺2\(1−γ\)​ρmin​τλ\)​‖∇J~λ​\(θk\)‖22\\displaystyle\\leq\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\-\\left\(\\frac\{\\eta\_\{\\mathsf\{a\}\}\}\{2\}\-\\frac\{2L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\\right\)\\left\\\|\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\+2​η𝖺\(1−γ\)2​∑\(s,a\)∈𝒮×𝒜dρθk​\(s\)2​πθk​\(a\|s\)2​\(𝔼​\[a^k​\(s,a\)\|ℱk\]−a~θkλ​\(s,a\)\)2\\displaystyle\+\\frac\{2\\eta\_\{\\mathsf\{a\}\}\}\{\(1\-\\gamma\)^\{2\}\}\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}\(s\)^\{2\}\\pi\_\{\\theta\_\{k\}\}\(a\|s\)^\{2\}\\left\(\\mathbb\{E\}\[\\hat\{\\mathrm\{a\}\}\_\{k\}\(s,a\)\|\\mathcal\{F\}\_\{k\}\]\-\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\(s,a\)\\right\)^\{2\}\+2​L​η𝖺2\(1−γ\)2​∑\(s,a\)∈𝒮×𝒜dρθk​\(s\)​πθk​\(a\|s\)​𝔼​\[\(a^k​\(s,a\)−a~θkλ​\(s,a\)\)2\|ℱk\]\.\\displaystyle\+\\frac\{2L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)^\{2\}\}\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}\(s\)\\pi\_\{\\theta\_\{k\}\}\(a\|s\)\\mathbb\{E\}\\left\[\(\\hat\{\\mathrm\{a\}\}\_\{k\}\(s,a\)\-\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\(s,a\)\)^\{2\}\|\\mathcal\{F\}\_\{k\}\\right\]\\kern 5\.0pt\.

###### Proof\.

First, the claimed lower bound on

mink≥0⁡min\(s,a\)⁡πθk​\(a∣s\)\\min\_\{k\\geq 0\}\\min\_\{\(s,a\)\}\\pi\_\{\\theta\_\{k\}\}\(a\\mid s\)is an immediate consequence of the update rule and[Lemma21](https://arxiv.org/html/2605.24357#Thmlemma21)\. In addition,[Lemma4](https://arxiv.org/html/2605.24357#Thmlemma4)gives

J~λ​\(θk\+1\)≥J~λ​\(θk\+η𝖺​g𝖺Yk\+1​\(a^k\)\)\.\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\+1\}\)\\geq\\tilde\{J\}\_\{\\lambda\}\\\!\\left\(\\theta\_\{k\}\+\\eta\_\{\\mathsf\{a\}\}\\mathrm\{g\}^\{Y\_\{k\+1\}\}\_\{\\mathsf\{a\}\}\(\\hat\{\\mathrm\{a\}\}\_\{k\}\)\\right\)\.SinceJ~λ\\tilde\{J\}\_\{\\lambda\}isLL\-smooth by[Lemma2](https://arxiv.org/html/2605.24357#Thmlemma2), we may then apply[Lemma23](https://arxiv.org/html/2605.24357#Thmlemma23)to obtain

J~λ​\(θk\+1\)≥J~λ​\(θk\)\+η𝖺​⟨∇J~λ​\(θk\),g𝖺Yk\+1​\(a^k\)⟩−L​η𝖺22​‖g𝖺Yk\+1​\(a^k\)‖22\.\\displaystyle\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\+1\}\)\\geq\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\+\\eta\_\{\\mathsf\{a\}\}\\langle\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\),\\mathrm\{g\}^\{Y\_\{k\+1\}\}\_\{\\mathsf\{a\}\}\(\\hat\{\\mathrm\{a\}\}\_\{k\}\)\\rangle\-\\frac\{L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{2\}\\left\\\|\\mathrm\{g\}^\{Y\_\{k\+1\}\}\_\{\\mathsf\{a\}\}\(\\hat\{\\mathrm\{a\}\}\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\\kern 5\.0pt\.SubtractingJ~λ⋆\\tilde\{J\}^\{\\star\}\_\{\\lambda\}and multiplying both sides by−1\-1, yields

J~λ⋆−J~λ​\(θk\+1\)≤J~λ⋆−J~λ​\(θk\)−η𝖺​⟨∇J~λ​\(θk\),g𝖺Yk\+1​\(a^k\)⟩\+L​η𝖺22​‖g𝖺Yk\+1​\(a^k\)‖22\.\\displaystyle\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\+1\}\)\\leq\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\-\\eta\_\{\\mathsf\{a\}\}\\langle\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\),\\mathrm\{g\}^\{Y\_\{k\+1\}\}\_\{\\mathsf\{a\}\}\(\\hat\{\\mathrm\{a\}\}\_\{k\}\)\\rangle\+\\frac\{L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{2\}\\left\\\|\\mathrm\{g\}^\{Y\_\{k\+1\}\}\_\{\\mathsf\{a\}\}\(\\hat\{\\mathrm\{a\}\}\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\\kern 5\.0pt\.Taking the conditionnal expectation with respect to𝒢k\\mathcal\{G\}\_\{k\}, yields

𝔼​\[J~λ⋆−J~λ​\(θk\+1\)\|𝒢k\]\\displaystyle\\mathbb\{E\}\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\+1\}\)\\big\|\\mathcal\{G\}\_\{k\}\\right\]≤J~λ⋆−J~λ​\(θk\)−η𝖺​⟨∇J~λ​\(θk\),g¯𝖺​\(θk,a^k\)⟩\+L​η𝖺22​𝔼​\[‖g𝖺Yk\+1​\(a^k\)‖22\|𝒢k\]\\displaystyle\\leq\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\-\\eta\_\{\\mathsf\{a\}\}\\langle\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\),\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{a\}\}\(\\theta\_\{k\},\\hat\{\\mathrm\{a\}\}\_\{k\}\)\\rangle\+\\frac\{L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{2\}\\mathbb\{E\}\\left\[\\left\\\|\\mathrm\{g\}^\{Y\_\{k\+1\}\}\_\{\\mathsf\{a\}\}\(\\hat\{\\mathrm\{a\}\}\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\\bigg\|\\mathcal\{G\}\_\{k\}\\right\]=𝔼​\[J~λ⋆−J~λ​\(θk\)\|𝒢k\]−η𝖺​⟨∇J~λ​\(θk\),g¯𝖺​\(θk,a^k\)⟩\\displaystyle=\\mathbb\{E\}\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\big\|\\mathcal\{G\}\_\{k\}\\right\]\-\\eta\_\{\\mathsf\{a\}\}\\langle\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\),\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{a\}\}\(\\theta\_\{k\},\\hat\{\\mathrm\{a\}\}\_\{k\}\)\\rangle\+L​η𝖺22​𝔼​\[‖g𝖺Yk\+1​\(a^k\)−g¯𝖺​\(θk,a^k\)‖22\|𝒢k\]\+L​η𝖺22​‖g¯𝖺​\(θk,a^k\)‖22\.\\displaystyle\+\\frac\{L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{2\}\\mathbb\{E\}\\left\[\\left\\\|\\mathrm\{g\}^\{Y\_\{k\+1\}\}\_\{\\mathsf\{a\}\}\(\\hat\{\\mathrm\{a\}\}\_\{k\}\)\-\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{a\}\}\(\\theta\_\{k\},\\hat\{\\mathrm\{a\}\}\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\\bigg\|\\mathcal\{G\}\_\{k\}\\right\]\+\\frac\{L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{2\}\\left\\\|\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{a\}\}\(\\theta\_\{k\},\\hat\{\\mathrm\{a\}\}\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\\kern 5\.0pt\.\(21\)Applying Young’s inequality gives

L​η𝖺22​‖g¯𝖺​\(θk,a^k\)‖22\\displaystyle\\frac\{L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{2\}\\left\\\|\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{a\}\}\(\\theta\_\{k\},\\hat\{\\mathrm\{a\}\}\_\{k\}\)\\right\\\|\_\{2\}^\{2\}≤L​η𝖺2​‖g¯𝖺​\(θk,a^k\)−∇J~λ​\(θk\)‖22\+L​η𝖺2​‖∇J~λ​\(θk\)‖22\.\\displaystyle\\leq L\\eta\_\{\\mathsf\{a\}\}^\{2\}\\left\\\|\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{a\}\}\(\\theta\_\{k\},\\hat\{\\mathrm\{a\}\}\_\{k\}\)\-\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\+L\\eta\_\{\\mathsf\{a\}\}^\{2\}\\left\\\|\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\\kern 5\.0pt\.Combining this with \([21](https://arxiv.org/html/2605.24357#A3.E21)\) and the variance formula of[Lemma12](https://arxiv.org/html/2605.24357#Thmlemma12)yields

𝔼​\[J~λ⋆−J~λ​\(θk\+1\)\|𝒢k\]\\displaystyle\\mathbb\{E\}\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\+1\}\)\\big\|\\mathcal\{G\}\_\{k\}\\right\]≤J~λ⋆−J~λ​\(θk\)−η𝖺​⟨∇J~λ​\(θk\),g¯𝖺​\(θk,a^k\)⟩\\displaystyle\\leq\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\-\\eta\_\{\\mathsf\{a\}\}\\langle\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\),\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{a\}\}\(\\theta\_\{k\},\\hat\{\\mathrm\{a\}\}\_\{k\}\)\\rangle\+L​η𝖺22​\(1−γ\)2​∑\(s,a\)∈𝒮×𝒜dρθk​\(s\)​πθk​\(a\|s\)​a^k​\(s,a\)2\\displaystyle\+\\frac\{L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{2\(1\-\\gamma\)^\{2\}\}\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}\(s\)\\pi\_\{\\theta\_\{k\}\}\(a\|s\)\\hat\{\\mathrm\{a\}\}\_\{k\}\(s,a\)^\{2\}\+L​η𝖺2​‖g¯𝖺​\(θk,a^k\)−∇J~λ​\(θk\)‖22\+L​η𝖺2​‖∇J~λ​\(θk\)‖22\.\\displaystyle\+L\\eta\_\{\\mathsf\{a\}\}^\{2\}\\left\\\|\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{a\}\}\(\\theta\_\{k\},\\hat\{\\mathrm\{a\}\}\_\{k\}\)\-\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\+L\\eta\_\{\\mathsf\{a\}\}^\{2\}\\left\\\|\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\\kern 5\.0pt\.Adding and subtracting∇J~λ​\(θk\)\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)in the scalar product term, combined with[Lemma12](https://arxiv.org/html/2605.24357#Thmlemma12)applied to the fourth term, gives

𝔼​\[J~λ⋆−J~λ​\(θk\+1\)\|𝒢k\]\\displaystyle\\mathbb\{E\}\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\+1\}\)\\big\|\\mathcal\{G\}\_\{k\}\\right\]≤J~λ⋆−J~λ​\(θk\)−η𝖺​‖∇J~λ​\(θk\)‖22−η𝖺​⟨∇J~λ​\(θk\),g¯𝖺​\(θk,a^k\)−∇J~λ​\(θk\)⟩\\displaystyle\\leq\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\-\\eta\_\{\\mathsf\{a\}\}\\left\\\|\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\-\\eta\_\{\\mathsf\{a\}\}\\langle\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\),\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{a\}\}\(\\theta\_\{k\},\\hat\{\\mathrm\{a\}\}\_\{k\}\)\-\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\rangle\+L​η𝖺22​\(1−γ\)2​∑\(s,a\)∈𝒮×𝒜dρθk​\(s\)​πθk​\(a\|s\)​a^k​\(s,a\)2\\displaystyle\+\\frac\{L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{2\(1\-\\gamma\)^\{2\}\}\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}\(s\)\\pi\_\{\\theta\_\{k\}\}\(a\|s\)\\hat\{\\mathrm\{a\}\}\_\{k\}\(s,a\)^\{2\}\+L​η𝖺2\(1−γ\)2​∑\(s,a\)∈𝒮×𝒜dρθk​\(s\)2​πθk​\(a\|s\)2​\(a^k​\(s,a\)−a~θkλ​\(s,a\)\)2\+L​η𝖺2​‖∇J~λ​\(θk\)‖22\.\\displaystyle\+\\frac\{L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)^\{2\}\}\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}\(s\)^\{2\}\\pi\_\{\\theta\_\{k\}\}\(a\|s\)^\{2\}\\left\(\\hat\{\\mathrm\{a\}\}\_\{k\}\(s,a\)\-\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\(s,a\)\\right\)^\{2\}\+L\\eta\_\{\\mathsf\{a\}\}^\{2\}\\left\\\|\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\\kern 5\.0pt\.Next, taking the conditional expectation, with respect toℱk\\mathcal\{F\}\_\{k\}, yields

𝔼​\[J~λ⋆−J~λ​\(θk\+1\)\|ℱk\]\\displaystyle\\mathbb\{E\}\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\+1\}\)\\big\|\\mathcal\{F\}\_\{k\}\\right\]≤J~λ⋆−J~λ​\(θk\)−\(η𝖺−L​η𝖺2\)​‖∇J~λ​\(θk\)‖22−η𝖺​⟨∇J~λ​\(θk\),𝔼​\[g¯𝖺​\(θk,a^k\)\|ℱk\]−∇J~λ​\(θk\)⟩⏟\(𝐏\)\\displaystyle\\leq\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\-\(\\eta\_\{\\mathsf\{a\}\}\-L\\eta\_\{\\mathsf\{a\}\}^\{2\}\)\\left\\\|\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\-\\eta\_\{\\mathsf\{a\}\}\\underbrace\{\\langle\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\),\\mathbb\{E\}\\left\[\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{a\}\}\(\\theta\_\{k\},\\hat\{\\mathrm\{a\}\}\_\{k\}\)\|\\mathcal\{F\}\_\{k\}\\right\]\-\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\rangle\}\_\{\\mathbf\{\(P\)\}\}\+L​η𝖺22​\(1−γ\)2​∑\(s,a\)∈𝒮×𝒜dρθk​\(s\)​πθk​\(a\|s\)​𝔼​\[a^k​\(s,a\)2\|ℱk\]\\displaystyle\+\\frac\{L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{2\(1\-\\gamma\)^\{2\}\}\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}\(s\)\\pi\_\{\\theta\_\{k\}\}\(a\|s\)\\mathbb\{E\}\[\\hat\{\\mathrm\{a\}\}\_\{k\}\(s,a\)^\{2\}\|\\mathcal\{F\}\_\{k\}\]\+L​η𝖺2\(1−γ\)2​∑\(s,a\)∈𝒮×𝒜dρθk​\(s\)2​πθk​\(a\|s\)2​𝔼​\[\(a^k​\(s,a\)−a~θkλ​\(s,a\)\)2\|ℱk\]\.\\displaystyle\+\\frac\{L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)^\{2\}\}\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}\(s\)^\{2\}\\pi\_\{\\theta\_\{k\}\}\(a\|s\)^\{2\}\\mathbb\{E\}\[\\left\(\\hat\{\\mathrm\{a\}\}\_\{k\}\(s,a\)\-\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\(s,a\)\\right\)^\{2\}\|\\mathcal\{F\}\_\{k\}\]\\kern 5\.0pt\.To bound\(𝐏\)\\mathbf\{\(P\)\}, we apply Cauchy\-Schwarz inequality, followed by Young’s inequality, which gives

\|P\|≤η𝖺2∥∇J~λ\(θk\)∥22\+2η𝖺∥𝔼\[g¯𝖺\(θk,a^k\)\|ℱk\]−∇J~λ\(θk\)∥22\.\\displaystyle\|\\textbf\{P\}\|\\leq\\frac\{\\eta\_\{\\mathsf\{a\}\}\}\{2\}\\left\\\|\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\+2\\eta\_\{\\mathsf\{a\}\}\\left\\\|\\mathbb\{E\}\[\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{a\}\}\(\\theta\_\{k\},\\hat\{\\mathrm\{a\}\}\_\{k\}\)\|\\mathcal\{F\}\_\{k\}\]\-\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\\kern 5\.0pt\.Plugging in the previous bound on\(𝐏\)\\mathbf\{\(P\)\}in the preceding inequality gives

𝔼​\[J~λ⋆−J~λ​\(θk\+1\)\|ℱk\]\\displaystyle\\mathbb\{E\}\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\+1\}\)\\big\|\\mathcal\{F\}\_\{k\}\\right\]≤J~λ⋆−J~λ\(θk\)−\(η𝖺2−Lη𝖺2\)∥∇J~λ\(θk\)∥22\+2η𝖺∥𝔼\[g¯𝖺\(θk,a^k\)\|ℱk\]−∇J~λ\(θk\)∥22\\displaystyle\\leq\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\-\\left\(\\frac\{\\eta\_\{\\mathsf\{a\}\}\}\{2\}\-L\\eta\_\{\\mathsf\{a\}\}^\{2\}\\right\)\\left\\\|\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\+2\\eta\_\{\\mathsf\{a\}\}\\left\\\|\\mathbb\{E\}\[\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{a\}\}\(\\theta\_\{k\},\\hat\{\\mathrm\{a\}\}\_\{k\}\)\|\\mathcal\{F\}\_\{k\}\]\-\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\+L​η𝖺22​\(1−γ\)2​∑\(s,a\)∈𝒮×𝒜dρθk​\(s\)​πθk​\(a\|s\)​𝔼​\[a^k​\(s,a\)2\|ℱk\]\\displaystyle\+\\frac\{L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{2\(1\-\\gamma\)^\{2\}\}\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}\(s\)\\pi\_\{\\theta\_\{k\}\}\(a\|s\)\\mathbb\{E\}\\left\[\\hat\{\\mathrm\{a\}\}\_\{k\}\(s,a\)^\{2\}\|\\mathcal\{F\}\_\{k\}\\right\]\+L​η𝖺2\(1−γ\)2​∑\(s,a\)∈𝒮×𝒜dρθk​\(s\)2​πθk​\(a\|s\)2​𝔼​\[\(a^k​\(s,a\)−a~θkλ​\(s,a\)\)2\|ℱk\]\.\\displaystyle\+\\frac\{L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)^\{2\}\}\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}\(s\)^\{2\}\\pi\_\{\\theta\_\{k\}\}\(a\|s\)^\{2\}\\mathbb\{E\}\\left\[\\left\(\\hat\{\\mathrm\{a\}\}\_\{k\}\(s,a\)\-\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\(s,a\)\\right\)^\{2\}\|\\mathcal\{F\}\_\{k\}\\right\]\\kern 5\.0pt\.\(22\)Observe that applying Young’s inequality, we have that

∑\(s,a\)∈𝒮×𝒜dρθk​\(s\)​πθk​\(a\|s\)​𝔼​\[a^k​\(s,a\)2\|ℱk\]\\displaystyle\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}\(s\)\\pi\_\{\\theta\_\{k\}\}\(a\|s\)\\mathbb\{E\}\\left\[\\hat\{\\mathrm\{a\}\}\_\{k\}\(s,a\)^\{2\}\|\\mathcal\{F\}\_\{k\}\\right\]≤2​∑\(s,a\)∈𝒮×𝒜dρθk​\(s\)​πθk​\(a\|s\)​𝔼​\[\(a^k​\(s,a\)−a~θkλ​\(s,a\)\)2\|ℱk\]\\displaystyle\\leq 2\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}\(s\)\\pi\_\{\\theta\_\{k\}\}\(a\|s\)\\mathbb\{E\}\\left\[\(\\hat\{\\mathrm\{a\}\}\_\{k\}\(s,a\)\-\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\(s,a\)\)^\{2\}\|\\mathcal\{F\}\_\{k\}\\right\]\+2​∑\(s,a\)∈𝒮×𝒜dρθk​\(s\)​πθk​\(a\|s\)​𝔼​\[a~θkλ​\(s,a\)2\|ℱk\]\.\\displaystyle\+2\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}\(s\)\\pi\_\{\\theta\_\{k\}\}\(a\|s\)\\mathbb\{E\}\\left\[\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\(s,a\)^\{2\}\|\\mathcal\{F\}\_\{k\}\\right\]\\kern 5\.0pt\.Plugging in the previous inequality in \([22](https://arxiv.org/html/2605.24357#A3.E22)\) yields

𝔼​\[J~λ⋆−J~λ​\(θk\+1\)\|ℱk\]\\displaystyle\\mathbb\{E\}\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\+1\}\)\\big\|\\mathcal\{F\}\_\{k\}\\right\]≤J~λ⋆−J~λ\(θk\)−\(η𝖺2−Lη𝖺2\)∥∇J~λ\(θk\)∥22\+2η𝖺∥𝔼\[g¯𝖺\(θk,a^k\)\|ℱk\]−∇J~λ\(θk\)∥22\\displaystyle\\leq\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\-\\left\(\\frac\{\\eta\_\{\\mathsf\{a\}\}\}\{2\}\-L\\eta\_\{\\mathsf\{a\}\}^\{2\}\\right\)\\left\\\|\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\+2\\eta\_\{\\mathsf\{a\}\}\\left\\\|\\mathbb\{E\}\[\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{a\}\}\(\\theta\_\{k\},\\hat\{\\mathrm\{a\}\}\_\{k\}\)\|\\mathcal\{F\}\_\{k\}\]\-\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\+L​η𝖺2\(1−γ\)2​∑\(s,a\)∈𝒮×𝒜dρθk​\(s\)​πθk​\(a\|s\)​a~θkλ​\(s,a\)2⏟\(𝐀\)\\displaystyle\+\\underbrace\{\\frac\{L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)^\{2\}\}\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}\(s\)\\pi\_\{\\theta\_\{k\}\}\(a\|s\)\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\(s,a\)^\{2\}\}\_\{\\mathbf\{\(A\)\}\}\+L​η𝖺2\(1−γ\)2​∑\(s,a\)∈𝒮×𝒜dρθk​\(s\)​πθk​\(a\|s\)​𝔼​\[\(a^k​\(s,a\)−a~θkλ​\(s,a\)\)2\|ℱk\]⏟\(𝐁\)\\displaystyle\+\\underbrace\{\\frac\{L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)^\{2\}\}\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}\(s\)\\pi\_\{\\theta\_\{k\}\}\(a\|s\)\\mathbb\{E\}\\left\[\\left\(\\hat\{\\mathrm\{a\}\}\_\{k\}\(s,a\)\-\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\(s,a\)\\right\)^\{2\}\|\\mathcal\{F\}\_\{k\}\\right\]\}\_\{\\mathbf\{\(B\)\}\}\+L​η𝖺2\(1−γ\)2​∑\(s,a\)∈𝒮×𝒜dρθk​\(s\)2​πθk​\(a\|s\)2​𝔼​\[\(a^k​\(s,a\)−a~θkλ​\(s,a\)\)2\|ℱk\]⏟\(𝐂\)\.\\displaystyle\+\\underbrace\{\\frac\{L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)^\{2\}\}\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}\(s\)^\{2\}\\pi\_\{\\theta\_\{k\}\}\(a\|s\)^\{2\}\\mathbb\{E\}\\left\[\\left\(\\hat\{\\mathrm\{a\}\}\_\{k\}\(s,a\)\-\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\(s,a\)\\right\)^\{2\}\|\\mathcal\{F\}\_\{k\}\\right\]\}\_\{\\mathbf\{\(C\)\}\}\\kern 5\.0pt\.

#### Bounding\(𝐀\)\\mathbf\{\(A\)\}\.

Using[Aρ\\textbf\{A\}\_\{\\rho\}](https://arxiv.org/html/2605.24357#ThmassumMDPx1)combined with the fact that for any\(s,a\)∈𝒮×𝒜\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}, we have thatπθk​\(a\|s\)≥τλ\\pi\_\{\\theta\_\{k\}\}\(a\|s\)\\geq\\tau\_\{\\lambda\}, we have that

\(𝐀\)≤1\(1−γ\)​ρmin​τλ​L​η𝖺2\(1−γ\)2​∑\(s,a\)∈𝒮×𝒜dρθk​\(s\)2​πθk​\(a\|s\)2​a~θkλ​\(s,a\)2≤L​η𝖺2\(1−γ\)​ρmin​τλ​‖∇J~λ​\(θk\)‖22\.\\displaystyle\\mathbf\{\(A\)\}\\leq\\frac\{1\}\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\\frac\{L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)^\{2\}\}\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}\(s\)^\{2\}\\pi\_\{\\theta\_\{k\}\}\(a\|s\)^\{2\}\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\(s,a\)^\{2\}\\leq\\frac\{L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\\left\\\|\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\\kern 5\.0pt\.

#### Bounding\(𝐁\)\\mathbf\{\(B\)\}\.

Using thatdρθk​\(s\)​πθk​\(a∣s\)≤1d\_\{\\rho\}^\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}\(s\)\\pi\_\{\\theta\_\{k\}\}\(a\\mid s\)\\leq 1, we have\(𝐁\)≤\(𝐂\)\\mathbf\{\(B\)\}\\leq\\mathbf\{\(C\)\}\.

Combining the previous bounds, we get

𝔼​\[J~λ⋆−J~λ​\(θk\+1\)\|ℱk\]\\displaystyle\\mathbb\{E\}\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\+1\}\)\\big\|\\mathcal\{F\}\_\{k\}\\right\]≤J~λ⋆−J~λ\(θk\)−\(η𝖺2−Lη𝖺2\)∥∇J~λ\(θk\)∥22\+2η𝖺∥𝔼\[g¯𝖺\(θk,a^k\)\|ℱk\]−∇J~λ\(θk\)∥22\\displaystyle\\leq\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\-\\left\(\\frac\{\\eta\_\{\\mathsf\{a\}\}\}\{2\}\-L\\eta\_\{\\mathsf\{a\}\}^\{2\}\\right\)\\left\\\|\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\+2\\eta\_\{\\mathsf\{a\}\}\\left\\\|\\mathbb\{E\}\[\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{a\}\}\(\\theta\_\{k\},\\hat\{\\mathrm\{a\}\}\_\{k\}\)\|\\mathcal\{F\}\_\{k\}\]\-\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\+L​η𝖺2\(1−γ\)​ρmin​τλ​‖∇J~λ​\(θk\)‖22\\displaystyle\+\\frac\{L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\\left\\\|\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\+2​L​η𝖺2\(1−γ\)2​∑\(s,a\)∈𝒮×𝒜dρθk​\(s\)​πθk​\(a\|s\)​𝔼​\[\(a^k​\(s,a\)−a~θkλ​\(s,a\)\)2\|ℱk\]\.\\displaystyle\+\\frac\{2L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)^\{2\}\}\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}\(s\)\\pi\_\{\\theta\_\{k\}\}\(a\|s\)\\mathbb\{E\}\\left\[\\left\(\\hat\{\\mathrm\{a\}\}\_\{k\}\(s,a\)\-\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\(s,a\)\\right\)^\{2\}\|\\mathcal\{F\}\_\{k\}\\right\]\\kern 5\.0pt\.Next, using that

L​η𝖺2≤L​η𝖺2\(1−γ\)​ρmin​τλ,\\displaystyle L\\eta\_\{\\mathsf\{a\}\}^\{2\}\\leq\\frac\{L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\\kern 5\.0pt,we get

𝔼​\[J~λ⋆−J~λ​\(θk\+1\)\|ℱk\]\\displaystyle\\mathbb\{E\}\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\+1\}\)\\big\|\\mathcal\{F\}\_\{k\}\\right\]≤J~λ⋆−J~λ\(θk\)−\(η𝖺2−2​L​η𝖺2\(1−γ\)​ρmin​τλ\)∥∇J~λ\(θk\)∥22\+2η𝖺∥𝔼\[g¯𝖺\(θk,a^k\)\|ℱk\]−∇J~λ\(θk\)∥22\\displaystyle\\leq\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\-\\left\(\\frac\{\\eta\_\{\\mathsf\{a\}\}\}\{2\}\-\\frac\{2L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\\right\)\\left\\\|\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\+2\\eta\_\{\\mathsf\{a\}\}\\left\\\|\\mathbb\{E\}\[\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{a\}\}\(\\theta\_\{k\},\\hat\{\\mathrm\{a\}\}\_\{k\}\)\|\\mathcal\{F\}\_\{k\}\]\-\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\+2​L​η𝖺2\(1−γ\)2​∑\(s,a\)∈𝒮×𝒜dρθk​\(s\)​πθk​\(a\|s\)​𝔼​\[\(a^k​\(s,a\)−a~θkλ​\(s,a\)\)2\|ℱk\]\.\\displaystyle\+\\frac\{2L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)^\{2\}\}\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}\(s\)\\pi\_\{\\theta\_\{k\}\}\(a\|s\)\\mathbb\{E\}\\left\[\(\\hat\{\\mathrm\{a\}\}\_\{k\}\(s,a\)\-\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\(s,a\)\)^\{2\}\|\\mathcal\{F\}\_\{k\}\\right\]\\kern 5\.0pt\.Finally, using that𝔼​\[g¯𝖺​\(θk,a^k\)\|ℱk\]=g¯𝖺​\(θk,𝔼​\[a^k\|ℱk\]\)\\mathbb\{E\}\[\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{a\}\}\(\\theta\_\{k\},\\hat\{\\mathrm\{a\}\}\_\{k\}\)\|\\mathcal\{F\}\_\{k\}\]=\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{a\}\}\(\\theta\_\{k\},\\mathbb\{E\}\[\\hat\{\\mathrm\{a\}\}\_\{k\}\|\\mathcal\{F\}\_\{k\}\]\), combined with[Lemma17](https://arxiv.org/html/2605.24357#Thmlemma17)concludes the proof\. ∎ Importantly, in the preceding lemma, we observe that if the critic is perfectly learned, we recover the linear convergence regime for constant step\-size observed in[Theorem3](https://arxiv.org/html/2605.24357#Thmtheorem3)\. The next lemma bounds the distance between two consecutive policies computed by[Ent\-AC](https://arxiv.org/html/2605.24357#alg1)\.

###### Lemma 14\.

Assume[Aρ\\textbf\{A\}\_\{\\rho\}](https://arxiv.org/html/2605.24357#ThmassumMDPx1)\. It holds that

‖q~θk\+1λ−q~θkλ‖∞≤Cλ​η𝖺​\|a^k​\(Sk\+1,Ak\+1\)\|,where​Cλ=2​γ1−γ​\(1\+λ​log⁡\(\|𝒜\|\)1−γ\+λ​log⁡\(1/τλ\)\+λ2​τλ\)\.\\displaystyle\\left\\\|\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\+1\}\}\}^\{\\lambda\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{\\infty\}\\leq C\_\{\\lambda\}\\eta\_\{\\mathsf\{a\}\}\\left\|\\hat\{\\mathrm\{a\}\}\_\{k\}\(S\_\{k\+1\},A\_\{k\+1\}\)\\right\|\\kern 5\.0pt,\\text\{ where \}C\_\{\\lambda\}=\\frac\{2\\gamma\}\{1\-\\gamma\}\\left\(\\frac\{1\+\\lambda\\log\(\|\\mathcal\{A\}\|\)\}\{1\-\\gamma\}\+\\lambda\\log\(1/\\tau\_\{\\lambda\}\)\+\\frac\{\\lambda\}\{2\\tau\_\{\\lambda\}\}\\right\)\\kern 5\.0pt\.

###### Proof\.

Denote byθ~k\+1=θk\+η𝖺​g𝖺Yk\+1​\(a^k\)\\tilde\{\\theta\}\_\{k\+1\}=\\theta\_\{k\}\+\\eta\_\{\\mathsf\{a\}\}\\,\\mathrm\{g\}^\{Y\_\{k\+1\}\}\_\{\\mathsf\{a\}\}\(\\hat\{\\mathrm\{a\}\}\_\{k\}\)\. By[Equation5](https://arxiv.org/html/2605.24357#S3.E5)\. For any\(s,a\)∈𝒮×𝒜\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}, we have

\|q~θk\+1λ​\(s,a\)−q~θkλ​\(s,a\)\|≤\|q~θk\+1λ​\(s,a\)−q~θ~k\+1λ​\(s,a\)\|\+\|q~θ~k\+1λ​\(s,a\)−q~θkλ​\(s,a\)\|\\displaystyle\\left\|\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\+1\}\}\}^\{\\lambda\}\(s,a\)\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\(s,a\)\\right\|\\leq\\left\|\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\+1\}\}\}^\{\\lambda\}\(s,a\)\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\tilde\{\\theta\}\_\{k\+1\}\}\}^\{\\lambda\}\(s,a\)\\right\|\+\\left\|\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\tilde\{\\theta\}\_\{k\+1\}\}\}^\{\\lambda\}\(s,a\)\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\(s,a\)\\right\|=γ\|∑s′∈𝒮𝖯\(s′\|s,a\)\[v~θk\+1λ\(s′\)−v~θ~k\+1λ\(s′\)\]\|⏟\(𝐀𝟏\)\+γ\|∑s′∈𝒮𝖯\(s′\|s,a\)\[v~θ~k\+1λ\(s′\)−v~θkλ\(s′\)\]\|⏟\(𝐀𝟐\),\\displaystyle=\\underbrace\{\\gamma\\left\|\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}\\mathsf\{P\}\(s^\{\\prime\}\|s,a\)\\left\[\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\+1\}\}\}^\{\\lambda\}\(s^\{\\prime\}\)\-\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\tilde\{\\theta\}\_\{k\+1\}\}^\{\\lambda\}\(s^\{\\prime\}\)\\right\]\\right\|\}\_\{\\mathbf\{\(A\_\{1\}\)\}\}\+\\underbrace\{\\gamma\\left\|\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}\\mathsf\{P\}\(s^\{\\prime\}\|s,a\)\\left\[\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\tilde\{\\theta\}\_\{k\+1\}\}^\{\\lambda\}\(s^\{\\prime\}\)\-\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\(s^\{\\prime\}\)\\right\]\\right\|\}\_\{\\mathbf\{\(A\_\{2\}\)\}\}\\kern 5\.0pt,where in the last equality, we used \([5](https://arxiv.org/html/2605.24357#S3.E5)\)\. Next, we bound each of these terms similarly\.

#### Bounding\(𝐀𝟏\)\\mathbf\{\(A\_\{1\}\)\}\.

Applying the soft performance\-difference lemma \([Lemma26](https://arxiv.org/html/2605.24357#Thmlemma26)\), we obtain

\(𝐀𝟏\)\\displaystyle\\mathbf\{\(A\_\{1\}\)\}≤\|∑s′∈𝒮𝖯\(s′\|s,a\)\[γ1−γ∑s′′∈𝒮ds′θ~k\+1\(s′′\)\[∑a′∈𝒜\(πθk\+1\(a′\|s′′\)−πθ~k\+1\(a′\|s′′\)\)\\displaystyle\\leq\\bigg\|\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}\\mathsf\{P\}\(s^\{\\prime\}\|s,a\)\\bigg\[\\frac\{\\gamma\}\{1\-\\gamma\}\\sum\_\{s^\{\\prime\\prime\}\\in\\mathcal\{S\}\}d\_\{s^\{\\prime\}\}^\{\\hskip 0\.57367pt\{\\tilde\{\\theta\}\_\{k\+1\}\}\}\(s^\{\\prime\\prime\}\)\\bigg\[\\sum\_\{a^\{\\prime\}\\in\\mathcal\{A\}\}\\left\(\\pi\_\{\\theta\_\{k\+1\}\}\(a^\{\\prime\}\|s^\{\\prime\\prime\}\)\-\\pi\_\{\\tilde\{\\theta\}\_\{k\+1\}\}\(a^\{\\prime\}\|s^\{\\prime\\prime\}\)\\right\)×\[q~θk\+1λ\(s′′,a′\)−λlog\(πθk\+1\(a′\|s′′\)\)\]\+λKL\(πθ~k\+1\(⋅\|s′′\)∥πθk\+1\(⋅\|s′′\)\)\]\]\|\.\\displaystyle\\hskip 80\.00012pt\\times\\left\[\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\+1\}\}\}^\{\\lambda\}\(s^\{\\prime\\prime\},a^\{\\prime\}\)\-\\lambda\\log\\big\(\\pi\_\{\\theta\_\{k\+1\}\}\(a^\{\\prime\}\|s^\{\\prime\\prime\}\)\\big\)\\right\]\+\\lambda\\,\\mathrm\{KL\}\\\!\\left\(\\pi\_\{\\tilde\{\\theta\}\_\{k\+1\}\}\(\\cdot\|s^\{\\prime\\prime\}\)\\,\\middle\\\|\\,\\pi\_\{\\theta\_\{k\+1\}\}\(\\cdot\|s^\{\\prime\\prime\}\)\\right\)\\bigg\]\\bigg\]\\bigg\|\\kern 5\.0pt\.
Next, since for all\(s,a\)∈𝒮×𝒜\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}we have

πθk​\(a\|s\)≥τλandπθk\+1​\(a\|s\)≥τλ,\\pi\_\{\\theta\_\{k\}\}\(a\|s\)\\geq\\tau\_\{\\lambda\}\\qquad\\text\{and\}\\qquad\\pi\_\{\\theta\_\{k\+1\}\}\(a\|s\)\\geq\\tau\_\{\\lambda\},[Lemma29](https://arxiv.org/html/2605.24357#Thmlemma29)implies that, for everys′′∈𝒮s^\{\\prime\\prime\}\\in\\mathcal\{S\},

KL\(πθ~k\+1\(⋅\|s′′\)∥πθk\+1\(⋅\|s′′\)\)≤12​τλ∥πθ~k\+1\(⋅\|s′′\)−πθk\+1\(⋅\|s′′\)∥1\.\\displaystyle\\mathrm\{KL\}\\\!\\left\(\\pi\_\{\\tilde\{\\theta\}\_\{k\+1\}\}\(\\cdot\|s^\{\\prime\\prime\}\)\\,\\middle\\\|\\,\\pi\_\{\\theta\_\{k\+1\}\}\(\\cdot\|s^\{\\prime\\prime\}\)\\right\)\\leq\\frac\{1\}\{2\\tau\_\{\\lambda\}\}\\left\\\|\\pi\_\{\\tilde\{\\theta\}\_\{k\+1\}\}\(\\cdot\|s^\{\\prime\\prime\}\)\-\\pi\_\{\\theta\_\{k\+1\}\}\(\\cdot\|s^\{\\prime\\prime\}\)\\right\\\|\_\{1\}\\kern 5\.0pt\.Therefore, using the triangle inequality together with

∑\(s′,s′′\)∈𝒮×𝒮𝖯​\(s′\|s,a\)​ds′θ~k\+1​\(s′′\)=1,\\displaystyle\\sum\_\{\(s^\{\\prime\},s^\{\\prime\\prime\}\)\\in\\mathcal\{S\}\\times\\mathcal\{S\}\}\\mathsf\{P\}\(s^\{\\prime\}\|s,a\)\\,d\_\{s^\{\\prime\}\}^\{\\hskip 0\.57367pt\{\\tilde\{\\theta\}\_\{k\+1\}\}\}\(s^\{\\prime\\prime\}\)=1,we arrive at

\(𝐀𝟏\)≤γ1−γmaxs′′∈𝒮\|\\displaystyle\\mathbf\{\(A\_\{1\}\)\}\\leq\\frac\{\\gamma\}\{1\-\\gamma\}\\max\_\{s^\{\\prime\\prime\}\\in\\mathcal\{S\}\}\\bigg\|∑a′∈𝒜\(πθk\+1\(a′\|s′′\)−πθ~k\+1\(a′\|s′′\)\)\[q~θk\+1λ\(s′′,a′\)−λlog\(πθk\+1\(a′\|s′′\)\)\]\|\\displaystyle\\sum\_\{a^\{\\prime\}\\in\\mathcal\{A\}\}\\left\(\\pi\_\{\\theta\_\{k\+1\}\}\(a^\{\\prime\}\|s^\{\\prime\\prime\}\)\-\\pi\_\{\\tilde\{\\theta\}\_\{k\+1\}\}\(a^\{\\prime\}\|s^\{\\prime\\prime\}\)\\right\)\\left\[\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\+1\}\}\}^\{\\lambda\}\(s^\{\\prime\\prime\},a^\{\\prime\}\)\-\\lambda\\log\\big\(\\pi\_\{\\theta\_\{k\+1\}\}\(a^\{\\prime\}\|s^\{\\prime\\prime\}\)\\big\)\\right\]\\bigg\|\+λ2​τλ∥πθ~k\+1\(⋅\|s′′\)−πθk\+1\(⋅\|s′′\)∥1\.\\displaystyle\\qquad\+\\frac\{\\lambda\}\{2\\tau\_\{\\lambda\}\}\\left\\\|\\pi\_\{\\tilde\{\\theta\}\_\{k\+1\}\}\(\\cdot\|s^\{\\prime\\prime\}\)\-\\pi\_\{\\theta\_\{k\+1\}\}\(\\cdot\|s^\{\\prime\\prime\}\)\\right\\\|\_\{1\}\\kern 5\.0pt\.
Now, using[Lemma27](https://arxiv.org/html/2605.24357#Thmlemma27), we deduce that

\(𝐀𝟏\)≤γ1−γmaxs′∈𝒮\{∥πθ~k\+1\(⋅\|s′\)−πθk\+1\(⋅\|s′\)∥1\(1\+λ​log⁡\(\|𝒜\|\)1−γ−λlog\(τλ\)\+λτλ\)\}\.\\displaystyle\\mathbf\{\(A\_\{1\}\)\}\\leq\\frac\{\\gamma\}\{1\-\\gamma\}\\max\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}\\bigg\\\{\\left\\\|\\pi\_\{\\tilde\{\\theta\}\_\{k\+1\}\}\(\\cdot\|s^\{\\prime\}\)\-\\pi\_\{\\theta\_\{k\+1\}\}\(\\cdot\|s^\{\\prime\}\)\\right\\\|\_\{1\}\\left\(\\frac\{1\+\\lambda\\log\(\|\\mathcal\{A\}\|\)\}\{1\-\\gamma\}\-\\lambda\\log\(\\tau\_\{\\lambda\}\)\+\\frac\{\\lambda\}\{\\tau\_\{\\lambda\}\}\\right\)\\bigg\\\}\\kern 5\.0pt\.
Finally, using[Lemma22](https://arxiv.org/html/2605.24357#Thmlemma22), the fact thatπθk\+1=𝒰τλ​\(πθ~k\+1\)\\pi\_\{\\theta\_\{k\+1\}\}=\\mathcal\{U\}\_\{\\tau\_\{\\lambda\}\}\(\\pi\_\{\\tilde\{\\theta\}\_\{k\+1\}\}\), andπθk∈Πτλ\\pi\_\{\\theta\_\{k\}\}\\in\\Pi\_\{\\tau\_\{\\lambda\}\}, ensures that

∥πθ~k\+1\(⋅\|s′\)−πθk\+1\(⋅\|s′\)∥1≤∥πθ~k\+1\(⋅\|s′\)−πθk\(⋅\|s′\)∥1\.\\displaystyle\\left\\\|\\pi\_\{\\tilde\{\\theta\}\_\{k\+1\}\}\(\\cdot\|s^\{\\prime\}\)\-\\pi\_\{\\theta\_\{k\+1\}\}\(\\cdot\|s^\{\\prime\}\)\\right\\\|\_\{1\}\\leq\\left\\\|\\pi\_\{\\tilde\{\\theta\}\_\{k\+1\}\}\(\\cdot\|s^\{\\prime\}\)\-\\pi\_\{\\theta\_\{k\}\}\(\\cdot\|s^\{\\prime\}\)\\right\\\|\_\{1\}\.Substituting this estimate into the previous display gives

\(𝐀𝟏\)≤γ1−γmaxs′∈𝒮\{∥πθ~k\+1\(⋅\|s′\)−πθk\(⋅\|s′\)∥1\(1\+λ​log⁡\(\|𝒜\|\)1−γ−λlog\(τλ\)\+λ2​τλ\)\}\.\\displaystyle\\mathbf\{\(A\_\{1\}\)\}\\leq\\frac\{\\gamma\}\{1\-\\gamma\}\\max\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}\\bigg\\\{\\left\\\|\\pi\_\{\\tilde\{\\theta\}\_\{k\+1\}\}\(\\cdot\|s^\{\\prime\}\)\-\\pi\_\{\\theta\_\{k\}\}\(\\cdot\|s^\{\\prime\}\)\\right\\\|\_\{1\}\\left\(\\frac\{1\+\\lambda\\log\(\|\\mathcal\{A\}\|\)\}\{1\-\\gamma\}\-\\lambda\\log\(\\tau\_\{\\lambda\}\)\+\\frac\{\\lambda\}\{2\\tau\_\{\\lambda\}\}\\right\)\\bigg\\\}\\kern 5\.0pt\.

#### Bounding\(𝐀𝟐\)\\mathbf\{\(A\_\{2\}\)\}\.

Applying again the soft performance\-difference lemma \([Lemma26](https://arxiv.org/html/2605.24357#Thmlemma26)\), we get

\(𝐀𝟐\)\\displaystyle\\mathbf\{\(A\_\{2\}\)\}≤\|∑s′∈𝒮𝖯\(s′\|s,a\)\[γ1−γ∑s′′∈𝒮ds′θ~k\+1\(s′′\)\[∑a′∈𝒜\(πθk\(a′\|s′′\)−πθ~k\+1\(a′\|s′′\)\)\\displaystyle\\leq\\bigg\|\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}\\mathsf\{P\}\(s^\{\\prime\}\|s,a\)\\bigg\[\\frac\{\\gamma\}\{1\-\\gamma\}\\sum\_\{s^\{\\prime\\prime\}\\in\\mathcal\{S\}\}d\_\{s^\{\\prime\}\}^\{\\hskip 0\.57367pt\\tilde\{\\theta\}\_\{k\+1\}\}\(s^\{\\prime\\prime\}\)\\bigg\[\\sum\_\{a^\{\\prime\}\\in\\mathcal\{A\}\}\\left\(\\pi\_\{\\theta\_\{k\}\}\(a^\{\\prime\}\|s^\{\\prime\\prime\}\)\-\\pi\_\{\\tilde\{\\theta\}\_\{k\+1\}\}\(a^\{\\prime\}\|s^\{\\prime\\prime\}\)\\right\)×\[q~θkλ\(s′′,a′\)−λlog\(πθk\(a′\|s′′\)\)\]\+λKL\(πθ~k\+1\(⋅\|s′′\)∥πθk\(⋅\|s′′\)\)\]\]\|\.\\displaystyle\\hskip 80\.00012pt\\times\\left\[\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\(s^\{\\prime\\prime\},a^\{\\prime\}\)\-\\lambda\\log\\big\(\\pi\_\{\\theta\_\{k\}\}\(a^\{\\prime\}\|s^\{\\prime\\prime\}\)\\big\)\\right\]\+\\lambda\\,\\mathrm\{KL\}\\\!\\left\(\\pi\_\{\\tilde\{\\theta\}\_\{k\+1\}\}\(\\cdot\|s^\{\\prime\\prime\}\)\\,\\middle\\\|\\,\\pi\_\{\\theta\_\{k\}\}\(\\cdot\|s^\{\\prime\\prime\}\)\\right\)\\bigg\]\\bigg\]\\bigg\|\\kern 5\.0pt\.
Using again the lower bound on the policy probabilities together with[Lemma29](https://arxiv.org/html/2605.24357#Thmlemma29), we obtain

KL\(πθ~k\+1\(⋅\|s′′\)∥πθk\(⋅\|s′′\)\)≤12​τλ∥πθ~k\+1\(⋅\|s′′\)−πθk\(⋅\|s′′\)∥1\.\\displaystyle\\mathrm\{KL\}\\\!\\left\(\\pi\_\{\\tilde\{\\theta\}\_\{k\+1\}\}\(\\cdot\|s^\{\\prime\\prime\}\)\\,\\middle\\\|\\,\\pi\_\{\\theta\_\{k\}\}\(\\cdot\|s^\{\\prime\\prime\}\)\\right\)\\leq\\frac\{1\}\{2\\tau\_\{\\lambda\}\}\\left\\\|\\pi\_\{\\tilde\{\\theta\}\_\{k\+1\}\}\(\\cdot\|s^\{\\prime\\prime\}\)\-\\pi\_\{\\theta\_\{k\}\}\(\\cdot\|s^\{\\prime\\prime\}\)\\right\\\|\_\{1\}\\kern 5\.0pt\.Proceeding exactly as in the bound on\(𝐀𝟏\)\\mathbf\{\(A\_\{1\}\)\}, and using both

∑\(s′,s′′\)∈𝒮×𝒮𝖯​\(s′\|s,a\)​ds′θ~k\+1​\(s′′\)=1\\displaystyle\\sum\_\{\(s^\{\\prime\},s^\{\\prime\\prime\}\)\\in\\mathcal\{S\}\\times\\mathcal\{S\}\}\\mathsf\{P\}\(s^\{\\prime\}\|s,a\)\\,d\_\{s^\{\\prime\}\}^\{\\hskip 0\.57367pt\{\\tilde\{\\theta\}\_\{k\+1\}\}\}\(s^\{\\prime\\prime\}\)=1and the bound on the regularizedQQ\-function, we obtain

\(𝐀𝟐\)≤γ1−γmaxs′∈𝒮\{∥πθ~k\+1\(⋅\|s′\)−πθk\(⋅\|s′\)∥1\(1\+λ​log⁡\(\|𝒜\|\)1−γ−λlog\(τλ\)\+λ2​τλ\)\}\.\\displaystyle\\mathbf\{\(A\_\{2\}\)\}\\leq\\frac\{\\gamma\}\{1\-\\gamma\}\\max\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}\\bigg\\\{\\left\\\|\\pi\_\{\\tilde\{\\theta\}\_\{k\+1\}\}\(\\cdot\|s^\{\\prime\}\)\-\\pi\_\{\\theta\_\{k\}\}\(\\cdot\|s^\{\\prime\}\)\\right\\\|\_\{1\}\\left\(\\frac\{1\+\\lambda\\log\(\|\\mathcal\{A\}\|\)\}\{1\-\\gamma\}\-\\lambda\\log\(\\tau\_\{\\lambda\}\)\+\\frac\{\\lambda\}\{2\\tau\_\{\\lambda\}\}\\right\)\\bigg\\\}\\kern 5\.0pt\.
Combining the bounds on\(𝐀𝟏\)\\mathbf\{\(A\_\{1\}\)\}and\(𝐀𝟐\)\\mathbf\{\(A\_\{2\}\)\}yields

\|q~θk\+1λ​\(s,a\)−q~θkλ​\(s,a\)\|\\displaystyle\\left\|\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\+1\}\}\}^\{\\lambda\}\(s,a\)\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\(s,a\)\\right\|≤2​γ1−γmaxs′∈𝒮\{∥πθ~k\+1\(⋅\|s′\)−πθk\(⋅\|s′\)∥1\\displaystyle\\leq\\frac\{2\\gamma\}\{1\-\\gamma\}\\max\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}\\bigg\\\{\\left\\\|\\pi\_\{\\tilde\{\\theta\}\_\{k\+1\}\}\(\\cdot\|s^\{\\prime\}\)\-\\pi\_\{\\theta\_\{k\}\}\(\\cdot\|s^\{\\prime\}\)\\right\\\|\_\{1\}\(23\)×\(1\+λ​log⁡\(\|𝒜\|\)1−γ−λlog\(τλ\)\+λ2​τλ\)\}\.\\displaystyle\\hskip 50\.00008pt\\times\\left\(\\frac\{1\+\\lambda\\log\(\|\\mathcal\{A\}\|\)\}\{1\-\\gamma\}\-\\lambda\\log\(\\tau\_\{\\lambda\}\)\+\\frac\{\\lambda\}\{2\\tau\_\{\\lambda\}\}\\right\)\\bigg\\\}\\kern 5\.0pt\.\(24\)
It remains to bound the distance∥πθ~k\+1\(⋅\|s′\)−πθk\(⋅\|s′\)∥1\.\\left\\\|\\pi\_\{\\tilde\{\\theta\}\_\{k\+1\}\}\(\\cdot\|s^\{\\prime\}\)\-\\pi\_\{\\theta\_\{k\}\}\(\\cdot\|s^\{\\prime\}\)\\right\\\|\_\{1\}\.To this end, we combine Pinsker’s inequality \([Lemma28](https://arxiv.org/html/2605.24357#Thmlemma28)\) with the KL\-logit inequality \([Lemma30](https://arxiv.org/html/2605.24357#Thmlemma30)\) to obtain

∥πθ~k\+1\(⋅\|s′\)−πθk\(⋅\|s′\)∥1\\displaystyle\\left\\\|\\pi\_\{\\tilde\{\\theta\}\_\{k\+1\}\}\(\\cdot\|s^\{\\prime\}\)\-\\pi\_\{\\theta\_\{k\}\}\(\\cdot\|s^\{\\prime\}\)\\right\\\|\_\{1\}≤2​KL\(πθ~k\+1\(⋅\|s′\)∥πθk\(⋅\|s′\)\)\\displaystyle\\leq\\sqrt\{2\}\\sqrt\{\\mathrm\{KL\}\\\!\\left\(\\pi\_\{\\tilde\{\\theta\}\_\{k\+1\}\}\(\\cdot\|s^\{\\prime\}\)\\,\\middle\\\|\\,\\pi\_\{\\theta\_\{k\}\}\(\\cdot\|s^\{\\prime\}\)\\right\)\}≤‖θ~k\+1​\(s′,⋅\)−θk​\(s′,⋅\)‖∞\\displaystyle\\leq\\left\\\|\\tilde\{\\theta\}\_\{k\+1\}\(s^\{\\prime\},\\cdot\)\-\\theta\_\{k\}\(s^\{\\prime\},\\cdot\)\\right\\\|\_\{\\infty\}≤η𝖺​\|a^k​\(Sk\+1,Ak\+1\)\|\.\\displaystyle\\leq\\eta\_\{\\mathsf\{a\}\}\\left\|\\hat\{\\mathrm\{a\}\}\_\{k\}\(S\_\{k\+1\},A\_\{k\+1\}\)\\right\|\\kern 5\.0pt\.Plugging this estimate into[Equation23](https://arxiv.org/html/2605.24357#A3.E23)concludes the proof\. ∎

###### Corollary 4\.

Assume[Aρ\\textbf\{A\}\_\{\\rho\}](https://arxiv.org/html/2605.24357#ThmassumMDPx1)\. It holds that

‖q~θk\+1λ−q~θkλ‖2≤C~λ​η𝖺​\|a^k​\(Sk\+1,Ak\+1\)\|,where​C~λ=2​γ​\|𝒮\|​\|𝒜\|1−γ​\(1\+λ​log⁡\(\|𝒜\|\)1−γ\+λ​log⁡\(1/τλ\)\+λ2​τλ\)\.\\displaystyle\\left\\\|\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\+1\}\}\}^\{\\lambda\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}\\leq\\tilde\{C\}\_\{\\lambda\}\\eta\_\{\\mathsf\{a\}\}\\left\|\\hat\{\\mathrm\{a\}\}\_\{k\}\(S\_\{k\+1\},A\_\{k\+1\}\)\\right\|\\kern 5\.0pt,\\text\{ where \}\\tilde\{C\}\_\{\\lambda\}=\\frac\{2\\gamma\\sqrt\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}\}\{1\-\\gamma\}\\left\(\\frac\{1\+\\lambda\\log\(\|\\mathcal\{A\}\|\)\}\{1\-\\gamma\}\+\\lambda\\log\(1/\\tau\_\{\\lambda\}\)\+\\frac\{\\lambda\}\{2\\tau\_\{\\lambda\}\}\\right\)\\kern 5\.0pt\.

## Appendix DGeneral Analysis of[Ent\-AC](https://arxiv.org/html/2605.24357#alg1): Critic Recursion

The critic step in[Ent\-AC](https://arxiv.org/html/2605.24357#alg1)consists of iteratingHHtimes astochasticBellman operator with a step\-sizeη𝖼\\eta\_\{\\mathsf\{c\}\}\. For completeness, we start by establishing the contractive property of this operator \(in contrast,\(Kumaret al\.,[2024](https://arxiv.org/html/2605.24357#bib.bib1)\)directly assumes its contractivity and does not prove it\)\. Afterwards, we give a bound on the variance of the gradient estimator used to build this stochasticstochasticBellman operator\. Next, we give control over the estimated q\-function obtained after iterating theHH\-critic steps\.

### D\.1Contractivity of the Bellman operator

Firstly, define the expected gradient of the critic and the regularized TD operator as

g¯𝖼​\(θ,q\)\\displaystyle\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{c\}\}\(\\theta,\\mathrm\{q\}\)=Δ​𝔼X∼ν𝖼​\(θ;⋅\)​\[g𝖼X​\(θ,q\)\],𝖳θη𝖼​q​=Δ​q\+η𝖼​𝖣θ​\[𝗋\+γ​𝖯~θ​\[q−λ​log⁡\(πθ\)\]−q\],\\displaystyle\\overset\{\\Delta\}\{=\}\\mathbb\{E\}\_\{X\\sim\\nu^\{\\mathsf\{c\}\}\(\\theta;\\cdot\)\}\\left\[\\mathrm\{g\}^\{X\}\_\{\\mathsf\{c\}\}\(\\theta,\\mathrm\{q\}\)\\right\],\\qquad\\mathsf\{T\}\_\{\\theta\}^\{\\eta\_\{\\mathsf\{c\}\}\}\\mathrm\{q\}\\overset\{\\Delta\}\{=\}\\mathrm\{q\}\+\\eta\_\{\\mathsf\{c\}\}\\mathsf\{D\}\_\{\\theta\}\\left\[\\mathsf\{r\}\+\\gamma\\widetilde\{\\mathsf\{P\}\}\_\{\\theta\}\\big\[\\mathrm\{q\}\-\\lambda\\log\(\\pi\_\{\\theta\}\)\\big\]\-\\mathrm\{q\}\\right\]\\kern 5\.0pt,\(25\)where the diagonal weight matrix𝖣θ\\mathsf\{D\}\_\{\\theta\}is given by

𝖣θ=diag⁡\(\(dρθ​\(s\)​πθ​\(a\|s\)\)\(s,a\)∈𝒮×𝒜\),\\mathsf\{D\}\_\{\\theta\}=\\operatorname\{diag\}\\\!\\left\(\\left\(d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)\\pi\_\{\\theta\}\(a\|s\)\\right\)\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}\\right\),the extended transition kernel𝖯~θ∈ℝ\|𝒮\|​\|𝒜\|×\|𝒮\|​\|𝒜\|\\widetilde\{\\mathsf\{P\}\}\_\{\\theta\}\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\\times\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}is defined by

𝖯~θ​\(s~,a~∣s,a\)=𝖯​\(s~∣s,a\)​πθ​\(a~∣s~\),∀\(s,a,s~,a~\)∈𝒮×𝒜×𝒮×𝒜,\\widetilde\{\\mathsf\{P\}\}\_\{\\theta\}\(\\tilde\{s\},\\tilde\{a\}\\mid s,a\)=\\mathsf\{P\}\(\\tilde\{s\}\\mid s,a\)\\,\\pi\_\{\\theta\}\(\\tilde\{a\}\\mid\\tilde\{s\}\),\\qquad\\forall\(s,a,\\tilde\{s\},\\tilde\{a\}\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\\times\\mathcal\{S\}\\times\\mathcal\{A\},andlog⁡\(πθ\)∈ℝ\|𝒮\|​\|𝒜\|\\log\(\\pi\_\{\\theta\}\)\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}denotes the vector whose coordinates satisfy

\[log⁡\(πθ\)\]\(s,a\)=log⁡\(πθ​\(a∣s\)\),∀\(s,a\)∈𝒮×𝒜\.\[\\log\(\\pi\_\{\\theta\}\)\]\_\{\(s,a\)\}=\\log\\big\(\\pi\_\{\\theta\}\(a\\mid s\)\\big\),\\qquad\\forall\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\.By construction,q~θλ\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}is a fixed point of the operator𝖳θη𝖼\\mathsf\{T\}\_\{\\theta\}^\{\\eta\_\{\\mathsf\{c\}\}\}\.

###### Lemma 15\.

Assume[Aρ\\textbf\{A\}\_\{\\rho\}](https://arxiv.org/html/2605.24357#ThmassumMDPx1)\. For anyθ∈ℝ\|𝒮\|​\|𝒜\|\\theta\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}such that for all\(s,a\)∈𝒮×𝒜\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}, we haveπθ​\(a\|s\)≥πmin\\pi\_\{\\theta\}\(a\|s\)\\geq\\pi\_\{\\min\}for someπmin\>0\\pi\_\{\\min\}\>0, andv∈ℝ\|𝒮\|​\|𝒜\|v\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}, it holds that

⟨𝖣θ​\(Id−γ​𝖯~θ\)​v,v⟩≥12​\(1−γ\)2​ρmin​πmin​‖v‖22\.\\displaystyle\\langle\\mathsf\{D\}\_\{\\theta\}\(\\mathrm\{Id\}\-\\gamma\\widetilde\{\\mathsf\{P\}\}\_\{\\theta\}\)v,v\\rangle\\geq\\frac\{1\}\{2\}\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\\pi\_\{\\min\}\\left\\\|v\\right\\\|\_\{2\}^\{2\}\\kern 5\.0pt\.

###### Proof\.

It holds that

v⊤​𝖣θ​\(Id−γ​𝖯~θ\)​v\\displaystyle v^\{\\top\}\\mathsf\{D\}\_\{\\theta\}\(\\mathrm\{Id\}\-\\gamma\\widetilde\{\\mathsf\{P\}\}\_\{\\theta\}\)v=∑s,adρθ​\(s\)​πθ​\(a\|s\)​v​\(s,a\)2−γ​∑\(s,a,s~,a~\)dρθ​\(s\)​πθ​\(a\|s\)​𝖯​\(s~\|s,a\)​πθ​\(a~\|s~\)​v​\(s,a\)​v​\(s~,a~\)\\displaystyle=\\sum\_\{s,a\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)\\pi\_\{\\theta\}\(a\|s\)v\(s,a\)^\{2\}\-\\gamma\\sum\_\{\(s,a,\\tilde\{s\},\\tilde\{a\}\)\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)\\pi\_\{\\theta\}\(a\|s\)\\mathsf\{P\}\(\\tilde\{s\}\|s,a\)\\pi\_\{\\theta\}\(\\tilde\{a\}\|\\tilde\{s\}\)v\(s,a\)v\(\\tilde\{s\},\\tilde\{a\}\)≥\(1−γ2\)​∑s,adρθ​\(s\)​πθ​\(a\|s\)​v​\(s,a\)2−γ2​∑\(s,a,s~,a~\)dρθ​\(s\)​πθ​\(a\|s\)​𝖯​\(s~\|s,a\)​πθ​\(a~\|s~\)​v​\(s~,a~\)2,\\displaystyle\\geq\(1\-\\frac\{\\gamma\}\{2\}\)\\sum\_\{s,a\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)\\pi\_\{\\theta\}\(a\|s\)v\(s,a\)^\{2\}\-\\frac\{\\gamma\}\{2\}\\sum\_\{\(s,a,\\tilde\{s\},\\tilde\{a\}\)\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)\\pi\_\{\\theta\}\(a\|s\)\\mathsf\{P\}\(\\tilde\{s\}\|s,a\)\\pi\_\{\\theta\}\(\\tilde\{a\}\|\\tilde\{s\}\)v\(\\tilde\{s\},\\tilde\{a\}\)^\{2\}\\kern 5\.0pt,\(26\)where in the last inequality, we used Young’s inequality\. Next, using the flow conservation constraints for occupancy measures, see[Lemma24](https://arxiv.org/html/2605.24357#Thmlemma24), for any\(s~,a~\)\(\\tilde\{s\},\\tilde\{a\}\), we have

dρθ​\(s~\)​πθ​\(a~\|s~\)\\displaystyle d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(\\tilde\{s\}\)\\pi\_\{\\theta\}\(\\tilde\{a\}\|\\tilde\{s\}\)=\(1−γ\)​ρ​\(s~\)​πθ​\(a~\|s~\)\+γ​∑\(s,a\)πθ​\(a~\|s~\)​𝖯​\(s~\|s,a\)​πθ​\(a\|s\)​dρθ​\(s\)\\displaystyle=\(1\-\\gamma\)\\rho\(\\tilde\{s\}\)\\pi\_\{\\theta\}\(\\tilde\{a\}\|\\tilde\{s\}\)\+\\gamma\\sum\_\{\(s,a\)\}\\pi\_\{\\theta\}\(\\tilde\{a\}\|\\tilde\{s\}\)\\mathsf\{P\}\(\\tilde\{s\}\|s,a\)\\pi\_\{\\theta\}\(a\|s\)d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)≥γ​∑\(s,a\)πθ​\(a~\|s~\)​𝖯​\(s~\|s,a\)​πθ​\(a\|s\)​dρθ​\(s\)\.\\displaystyle\\geq\\gamma\\sum\_\{\(s,a\)\}\\pi\_\{\\theta\}\(\\tilde\{a\}\|\\tilde\{s\}\)\\mathsf\{P\}\(\\tilde\{s\}\|s,a\)\\pi\_\{\\theta\}\(a\|s\)d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)\\kern 5\.0pt\.Multiplying both sides byv​\(s~,a~\)2v\(\\tilde\{s\},\\tilde\{a\}\)^\{2\}and summing over\(s~,a~\)∈𝒮×𝒜\(\\tilde\{s\},\\tilde\{a\}\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}, gives

∑\(s,a,s~,a~\)dρθ​\(s\)​πθ​\(a\|s\)​𝖯​\(s~\|s,a\)​πθ​\(a~\|s~\)​v​\(s~,a~\)2≤1γ​∑s~,a~dρθ​\(s~\)​πθ​\(a~\|s~\)​v​\(s~,a~\)2\.\\displaystyle\\sum\_\{\(s,a,\\tilde\{s\},\\tilde\{a\}\)\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)\\pi\_\{\\theta\}\(a\|s\)\\mathsf\{P\}\(\\tilde\{s\}\|s,a\)\\pi\_\{\\theta\}\(\\tilde\{a\}\|\\tilde\{s\}\)v\(\\tilde\{s\},\\tilde\{a\}\)^\{2\}\\leq\\frac\{1\}\{\\gamma\}\\sum\_\{\\tilde\{s\},\\tilde\{a\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(\\tilde\{s\}\)\\pi\_\{\\theta\}\(\\tilde\{a\}\|\\tilde\{s\}\)v\(\\tilde\{s\},\\tilde\{a\}\)^\{2\}\\kern 5\.0pt\.Plugging in the previous identity in \([26](https://arxiv.org/html/2605.24357#A4.E26)\), gives

v⊤​𝖣θ​\(Id−γ​𝖯~θ\)​v≥12​\(1−γ\)​∑s,adρθ​\(s\)​πθ​\(a\|s\)​v​\(s,a\)2≥12​\(1−γ\)2​ρmin​πmin​‖v‖22,\\displaystyle v^\{\\top\}\\mathsf\{D\}\_\{\\theta\}\(\\mathrm\{Id\}\-\\gamma\\widetilde\{\\mathsf\{P\}\}\_\{\\theta\}\)v\\geq\\frac\{1\}\{2\}\(1\-\\gamma\)\\sum\_\{s,a\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)\\pi\_\{\\theta\}\(a\|s\)v\(s,a\)^\{2\}\\geq\\frac\{1\}\{2\}\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\\pi\_\{\\min\}\\left\\\|v\\right\\\|\_\{2\}^\{2\}\\kern 5\.0pt,where the last inequality holds by[Aρ\\textbf\{A\}\_\{\\rho\}](https://arxiv.org/html/2605.24357#ThmassumMDPx1)combined with the fact that each entryπθ\\pi\_\{\\theta\}is larger thanπmin\\pi\_\{\\min\}\. ∎

###### Lemma 16\(Contractivity of𝖳θη𝖼\\mathsf\{T\}\_\{\\theta\}^\{\\eta\_\{\\mathsf\{c\}\}\}inL2L\_\{2\}norm\)\.

Assume[Aρ\\textbf\{A\}\_\{\\rho\}](https://arxiv.org/html/2605.24357#ThmassumMDPx1)\. Fixθ∈ℝ\|𝒮\|​\|𝒜\|\\theta\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}andq∈ℝ\|𝒮\|​\|𝒜\|\\mathrm\{q\}\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}\. Additionnally, assume that for all\(s,a\)∈𝒮×𝒜\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}, we haveπθ​\(a\|s\)≥τλ\\pi\_\{\\theta\}\(a\|s\)\\geq\\tau\_\{\\lambda\}\. It holds that

‖q~θλ−𝖳θη𝖼​q‖22≤\(1−η𝖼​\(1−γ\)2​ρmin​τλ\+η𝖼2​\(1\+γ\)2\)​‖q~θλ−q‖22\.\\displaystyle\\left\\\|\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\-\\mathsf\{T\}\_\{\\theta\}^\{\\eta\_\{\\mathsf\{c\}\}\}\\mathrm\{q\}\\right\\\|\_\{2\}^\{2\}\\leq\(1\-\\eta\_\{\\mathsf\{c\}\}\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}\+\\eta\_\{\\mathsf\{c\}\}^\{2\}\(1\+\\gamma\)^\{2\}\)\\left\\\|\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\-\\mathrm\{q\}\\right\\\|\_\{2\}^\{2\}\\kern 5\.0pt\.

###### Proof\.

Fixθ∈ℝ\|𝒮\|​\|𝒜\|\\theta\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}andq∈ℝ\|𝒮\|​\|𝒜\|\\mathrm\{q\}\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}\. Using thatq~θλ\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}is a fixed point of𝖳θη𝖼\\mathsf\{T\}\_\{\\theta\}^\{\\eta\_\{\\mathsf\{c\}\}\}implies that

‖q~θλ−𝖳θη𝖼​q‖22\\displaystyle\\left\\\|\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\-\\mathsf\{T\}\_\{\\theta\}^\{\\eta\_\{\\mathsf\{c\}\}\}\\mathrm\{q\}\\right\\\|\_\{2\}^\{2\}=‖𝖳θη𝖼​q~θλ−𝖳θη𝖼​q‖22\\displaystyle=\\left\\\|\\mathsf\{T\}\_\{\\theta\}^\{\\eta\_\{\\mathsf\{c\}\}\}\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\-\\mathsf\{T\}\_\{\\theta\}^\{\\eta\_\{\\mathsf\{c\}\}\}\\mathrm\{q\}\\right\\\|\_\{2\}^\{2\}=‖q~θλ\+η𝖼​𝖣θ​\[𝗋\+γ​𝖯~θ​\[q~θλ−λ​log⁡\(πθ\)\]−q~θλ\]−q−η𝖼​𝖣θ​\[𝗋\+γ​𝖯~θ​\[q−λ​log⁡\(πθ\)\]−q\]‖22\\displaystyle=\\left\\\|\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\+\\eta\_\{\\mathsf\{c\}\}\\mathsf\{D\}\_\{\\theta\}\\left\[\\mathsf\{r\}\+\\gamma\\widetilde\{\\mathsf\{P\}\}\_\{\\theta\}\[\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\-\\lambda\\log\(\\pi\_\{\\theta\}\)\]\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\\right\]\-\\mathrm\{q\}\-\\eta\_\{\\mathsf\{c\}\}\\mathsf\{D\}\_\{\\theta\}\\left\[\\mathsf\{r\}\+\\gamma\\widetilde\{\\mathsf\{P\}\}\_\{\\theta\}\[\\mathrm\{q\}\-\\lambda\\log\(\\pi\_\{\\theta\}\)\]\-\\mathrm\{q\}\\right\]\\right\\\|\_\{2\}^\{2\}=‖q~θλ\+η𝖼​𝖣θ​\[γ​𝖯~θ​q~θλ−q~θλ\]−q−η𝖼​𝖣θ​\[γ​𝖯~θ​q−q\]‖22\\displaystyle=\\left\\\|\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\+\\eta\_\{\\mathsf\{c\}\}\\mathsf\{D\}\_\{\\theta\}\\left\[\\gamma\\widetilde\{\\mathsf\{P\}\}\_\{\\theta\}\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\\right\]\-\\mathrm\{q\}\-\\eta\_\{\\mathsf\{c\}\}\\mathsf\{D\}\_\{\\theta\}\\left\[\\gamma\\widetilde\{\\mathsf\{P\}\}\_\{\\theta\}\\mathrm\{q\}\-\\mathrm\{q\}\\right\]\\right\\\|\_\{2\}^\{2\}=‖q~θλ−q‖22\+\(−2\)​η𝖼​⟨𝖣θ​\(Id−γ​𝖯~θ\)​\(q~θλ−q\),q~θλ−q⟩⏟\(𝐌\)\+η𝖼2​‖𝖣θ​\(Id−γ​𝖯~θ\)​\(q~θλ−q\)‖22⏟\(𝐍\)\.\\displaystyle=\\left\\\|\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\-\\mathrm\{q\}\\right\\\|\_\{2\}^\{2\}\+\\underbrace\{\(\-2\)\\eta\_\{\\mathsf\{c\}\}\\langle\\mathsf\{D\}\_\{\\theta\}\(\\mathrm\{Id\}\-\\gamma\\widetilde\{\\mathsf\{P\}\}\_\{\\theta\}\)\(\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\-\\mathrm\{q\}\),\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\-\\mathrm\{q\}\\rangle\}\_\{\\mathbf\{\(M\)\}\}\+\\underbrace\{\\eta\_\{\\mathsf\{c\}\}^\{2\}\\left\\\|\\mathsf\{D\}\_\{\\theta\}\(\\mathrm\{Id\}\-\\gamma\\widetilde\{\\mathsf\{P\}\}\_\{\\theta\}\)\(\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\-\\mathrm\{q\}\)\\right\\\|\_\{2\}^\{2\}\}\_\{\\mathbf\{\(N\)\}\}\\kern 5\.0pt\.

#### Bounding\(𝐌\)\\mathbf\{\(M\)\}:

To bound\(𝐌\)\\mathbf\{\(M\)\}, we apply[Lemma15](https://arxiv.org/html/2605.24357#Thmlemma15)which yields

\(𝐌\)≤−η𝖼​\(1−γ\)2​ρmin​τλ​‖q~θλ−q‖22\.\\displaystyle\\mathbf\{\(M\)\}\\leq\-\\eta\_\{\\mathsf\{c\}\}\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}\\left\\\|\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\-\\mathrm\{q\}\\right\\\|\_\{2\}^\{2\}\\kern 5\.0pt\.

#### Bounding\(𝐍\)\\mathbf\{\(N\)\}:

Define the matrixAθ=𝖣θ​\(Id−γ​𝖯~θ\)A\_\{\\theta\}=\\mathsf\{D\}\_\{\\theta\}\(\\mathrm\{Id\}\-\\gamma\\widetilde\{\\mathsf\{P\}\}\_\{\\theta\}\)\. We claim‖Aθ‖2≤1\+γ\\\|A\_\{\\theta\}\\\|\_\{2\}\\leq 1\+\\gamma, where‖Aθ‖2\\\|A\_\{\\theta\}\\\|\_\{2\}is the spectral norm norm ofAθA\_\{\\theta\}\(see[AppendixA](https://arxiv.org/html/2605.24357#A1)\) for the exact definition\)\. Indeed, sing that𝖣θ\\mathsf\{D\}\_\{\\theta\}is diagonal with nonnegative entries\(dρθ​\(s\)​πθ​\(a\|s\)\)\(s,a\)∈𝒮×𝒜\(d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)\\pi\_\{\\theta\}\(a\|s\)\)\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}that sum to11, we have for every row\(s,a\)\(s,a\):

‖Aθ‖∞\\displaystyle\\\|A\_\{\\theta\}\\\|\_\{\\infty\}=∑\(s~,a~\)\|\(Aθ\)\(s,a\),\(s~,a~\)\|=dρθ\(s\)πθ\(a\|s\)∑\(s~,a~\)∈𝒮×𝒜\|δ\(s,a\)\(s~,a~\)−γ𝖯~θ\(s~,a~∣s,a\)\|\\displaystyle=\\sum\_\{\(\\tilde\{s\},\\tilde\{a\}\)\}\|\(A\_\{\\theta\}\)\_\{\(s,a\),\(\\tilde\{s\},\\tilde\{a\}\)\}\|=d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)\\pi\_\{\\theta\}\(a\|s\)\\sum\_\{\(\\tilde\{s\},\\tilde\{a\}\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}\|\\delta\_\{\(s,a\)\}\(\\tilde\{s\},\\tilde\{a\}\)\-\\gamma\\widetilde\{\\mathsf\{P\}\}\_\{\\theta\}\(\\tilde\{s\},\\tilde\{a\}\\mid s,a\)\|≤dρθ\(s\)πθ\(a\|s\)∑\(s~,a~\)∈𝒮×𝒜δ\(s,a\)\(s~,a~\)\+γ𝖯\(s~∣s,a\)πθ\(a~∣s~\)\)=dρθ\(s\)πθ\(a\|s\)\(1\+γ\)≤1\+γ\.\\displaystyle\\leq d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)\\pi\_\{\\theta\}\(a\|s\)\\sum\_\{\(\\tilde\{s\},\\tilde\{a\}\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}\\delta\_\{\(s,a\)\}\(\\tilde\{s\},\\tilde\{a\}\)\+\\gamma\\mathsf\{P\}\(\\tilde\{s\}\\mid s,a\)\\,\\pi\_\{\\theta\}\(\\tilde\{a\}\\mid\\tilde\{s\}\)\)=d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)\\pi\_\{\\theta\}\(a\|s\)\(1\+\\gamma\)\\leq 1\+\\gamma\.Hence‖Aθ‖∞≤1\+γ\\\|A\_\{\\theta\}\\\|\_\{\\infty\}\\leq 1\+\\gamma\. Similarly for every column\(s~,a~\)\(\\tilde\{s\},\\tilde\{a\}\):

∑\(s,a\)∈𝒮×𝒜\|\(Aθ\)\(s,a\),\(s~,a~\)\|\\displaystyle\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}\|\(A\_\{\\theta\}\)\_\{\(s,a\),\(\\tilde\{s\},\\tilde\{a\}\)\}\|≤∑\(s,a\)∈𝒮×𝒜dρθ​\(s\)​πθ​\(a\|s\)​\(δ\(s,a\)​\(s~,a~\)\+γ​𝖯​\(s~∣s,a\)​πθ​\(a~∣s~\)\)\\displaystyle\\leq\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)\\pi\_\{\\theta\}\(a\|s\)\(\\delta\_\{\(s,a\)\}\(\\tilde\{s\},\\tilde\{a\}\)\+\\gamma\\mathsf\{P\}\(\\tilde\{s\}\\mid s,a\)\\,\\pi\_\{\\theta\}\(\\tilde\{a\}\\mid\\tilde\{s\}\)\)=dρθ​\(s~\)​πθ​\(a~\|s~\)\+γ​∑\(s,a\)∈𝒮×𝒜dρθ​\(s\)​πθ​\(a\|s\)​𝖯​\(s~∣s,a\)​πθ​\(a~∣s~\)\\displaystyle=d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(\\tilde\{s\}\)\\pi\_\{\\theta\}\(\\tilde\{a\}\|\\tilde\{s\}\)\+\\gamma\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)\\pi\_\{\\theta\}\(a\|s\)\\mathsf\{P\}\(\\tilde\{s\}\\mid s,a\)\\pi\_\{\\theta\}\(\\tilde\{a\}\\mid\\tilde\{s\}\)≤1\+γ,\\displaystyle\\leq 1\+\\gamma\\kern 5\.0pt,so‖Aθ‖1≤1\+γ\\\|A\_\{\\theta\}\\\|\_\{1\}\\leq 1\+\\gamma\. Using the standard inequality‖Aθ‖2≤‖Aθ‖1​‖Aθ‖∞\\\|A\_\{\\theta\}\\\|\_\{2\}\\leq\\sqrt\{\\\|A\_\{\\theta\}\\\|\_\{1\}\\\|A\_\{\\theta\}\\\|\_\{\\infty\}\}gives

‖Aθ‖2≤1\+γ\.\\\|A\_\{\\theta\}\\\|\_\{2\}\\leq 1\+\\gamma\.Therefore

\(𝐍\)≤η𝖼2​‖𝖣θ​\(Id−γ​𝖯~θ\)​\(q~θλ−q\)‖22≤\(1\+γ\)2​‖\(q~θλ−q\)‖22\.\\mathbf\{\(N\)\}\\leq\\eta\_\{\\mathsf\{c\}\}^\{2\}\\\|\\mathsf\{D\}\_\{\\theta\}\(\\mathrm\{Id\}\-\\gamma\\widetilde\{\\mathsf\{P\}\}\_\{\\theta\}\)\(\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\-\\mathrm\{q\}\)\\\|\_\{2\}^\{2\}\\leq\(1\+\\gamma\)^\{2\}\\\|\(\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\-\\mathrm\{q\}\)\\\|\_\{2\}^\{2\}\\kern 5\.0pt\.Combining the two previous bounds concludes the proof\. ∎

### D\.2Properties of the stochastic gradient

Next, we provide properties on the gradient estimator used in the critic’s update\.

###### Lemma 17\(Properties of the critic estimator\)\.

Fixθ∈ℝ\|𝒮\|​\|𝒜\|\\theta\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}andq∈ℝ\|𝒮\|​\|𝒜\|\\mathrm\{q\}\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}\. Then

g¯𝖼​\(θ,q\)=𝖳θη𝖼​q−qη𝖼,\\displaystyle\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{c\}\}\(\\theta,\\mathrm\{q\}\)=\\frac\{\\mathsf\{T\}\_\{\\theta\}^\{\\eta\_\{\\mathsf\{c\}\}\}\\mathrm\{q\}\-\\mathrm\{q\}\}\{\\eta\_\{\\mathsf\{c\}\}\},and

𝔼X∼ν𝖼​\(θ\)\[∥g𝖼X\(θ,q\)−g¯𝖼\(θ,q\)∥22\]≤8∥q∥∞2\+4\+4λ2\+4λ2log\(\|𝒜\|\)2\.\\displaystyle\\mathbb\{E\}\_\{X\\sim\\nu^\{\\mathsf\{c\}\}\(\\theta\)\}\\left\[\\left\\\|\\mathrm\{g\}^\{X\}\_\{\\mathsf\{c\}\}\(\\theta,\\mathrm\{q\}\)\-\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{c\}\}\(\\theta,\\mathrm\{q\}\)\\right\\\|\_\{2\}^\{2\}\\right\]\\leq 8\\\|\\mathrm\{q\}\\\|\_\{\\infty\}^\{2\}\+4\+4\\lambda^\{2\}\+4\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\.

###### Proof\.

Fixθ∈ℝ\|𝒮\|​\|𝒜\|\\theta\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}andq∈ℝ\|𝒮\|​\|𝒜\|\\mathrm\{q\}\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}\. We first compute the mean of the critic estimator\. For any\(s,a\)∈𝒮×𝒜\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}, we have

\[g¯𝖼​\(θ,q\)\]s,a\\displaystyle\[\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{c\}\}\(\\theta,\\mathrm\{q\}\)\]\_\{s,a\}=\[𝔼X∼ν𝖼​\(θ;⋅\)​\[g𝖼X​\(θ,q\)\]\]s,a\\displaystyle=\\left\[\\mathbb\{E\}\_\{X\\sim\\nu^\{\\mathsf\{c\}\}\(\\theta;\\cdot\)\}\\left\[\\mathrm\{g\}^\{X\}\_\{\\mathsf\{c\}\}\(\\theta,\\mathrm\{q\}\)\\right\]\\right\]\_\{s,a\}=∑\(s′,a′,s~′,a~′\)dρθ​\(s′\)​πθ​\(a′\|s′\)​𝖯​\(s~′\|s′,a′\)​πθ​\(a~′\|s~′\)​1\(s,a\)​\(s′,a′\)\\displaystyle=\\sum\_\{\(s^\{\\prime\},a^\{\\prime\},\\tilde\{s\}^\{\\prime\},\\tilde\{a\}^\{\\prime\}\)\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s^\{\\prime\}\)\\,\\pi\_\{\\theta\}\(a^\{\\prime\}\|s^\{\\prime\}\)\\,\\mathsf\{P\}\(\\tilde\{s\}^\{\\prime\}\|s^\{\\prime\},a^\{\\prime\}\)\\,\\pi\_\{\\theta\}\(\\tilde\{a\}^\{\\prime\}\|\\tilde\{s\}^\{\\prime\}\)\\,\\mathsf\{1\}\_\{\(s,a\)\}\(s^\{\\prime\},a^\{\\prime\}\)×\[𝗋​\(s′,a′\)\+γ​\(q​\(s~′,a~′\)−λ​log⁡πθ​\(a~′\|s~′\)\)−q​\(s′,a′\)\]\\displaystyle\\qquad\\qquad\\times\\left\[\\mathsf\{r\}\(s^\{\\prime\},a^\{\\prime\}\)\+\\gamma\\bigl\(\\mathrm\{q\}\(\\tilde\{s\}^\{\\prime\},\\tilde\{a\}^\{\\prime\}\)\-\\lambda\\log\\pi\_\{\\theta\}\(\\tilde\{a\}^\{\\prime\}\|\\tilde\{s\}^\{\\prime\}\)\\bigr\)\-\\mathrm\{q\}\(s^\{\\prime\},a^\{\\prime\}\)\\right\]=dρθ\(s\)πθ\(a\|s\)\[𝗋\(s,a\)\+γ∑\(s~,a~\)𝖯\(s~\|s,a\)πθ\(a~\|s~\)\\displaystyle=d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)\\,\\pi\_\{\\theta\}\(a\|s\)\\Bigg\[\\mathsf\{r\}\(s,a\)\+\\gamma\\sum\_\{\(\\tilde\{s\},\\tilde\{a\}\)\}\\mathsf\{P\}\(\\tilde\{s\}\|s,a\)\\,\\pi\_\{\\theta\}\(\\tilde\{a\}\|\\tilde\{s\}\)×\(q\(s~,a~\)−λlogπθ\(a~\|s~\)\)−q\(s,a\)\]\\displaystyle\\qquad\\qquad\\qquad\\qquad\\qquad\\qquad\\times\\bigl\(\\mathrm\{q\}\(\\tilde\{s\},\\tilde\{a\}\)\-\\lambda\\log\\pi\_\{\\theta\}\(\\tilde\{a\}\|\\tilde\{s\}\)\\bigr\)\-\\mathrm\{q\}\(s,a\)\\Bigg\]=\[𝖣θ​\(𝗋\+γ​𝖯~θ​\[q−λ​log⁡\(πθ\)\]−q\)\]s,a\.\\displaystyle=\\left\[\\mathsf\{D\}\_\{\\theta\}\\Bigl\(\\mathsf\{r\}\+\\gamma\\widetilde\{\\mathsf\{P\}\}\_\{\\theta\}\\bigl\[\\mathrm\{q\}\-\\lambda\\log\(\\pi\_\{\\theta\}\)\\bigr\]\-\\mathrm\{q\}\\Bigr\)\\right\]\_\{s,a\}\.By the definition of𝖳θη𝖼\\mathsf\{T\}\_\{\\theta\}^\{\\eta\_\{\\mathsf\{c\}\}\}, this proves that

g¯𝖼​\(θ,q\)=𝖳θη𝖼​q−qη𝖼\.\\displaystyle\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{c\}\}\(\\theta,\\mathrm\{q\}\)=\\frac\{\\mathsf\{T\}\_\{\\theta\}^\{\\eta\_\{\\mathsf\{c\}\}\}\\mathrm\{q\}\-\\mathrm\{q\}\}\{\\eta\_\{\\mathsf\{c\}\}\}\.
We now prove the second\-moment bound\. Using

𝔼​\[‖X−𝔼​\[X\]‖22\]≤𝔼​\[‖X‖22\],\\mathbb\{E\}\\bigl\[\\\|X\-\\mathbb\{E\}\[X\]\\\|\_\{2\}^\{2\}\\bigr\]\\leq\\mathbb\{E\}\[\\\|X\\\|\_\{2\}^\{2\}\],we obtain

𝔼X∼ν𝖼​\(θ\)​\[‖g𝖼X​\(θ,q\)−g¯𝖼​\(θ,q\)‖22\]\\displaystyle\\mathbb\{E\}\_\{X\\sim\\nu^\{\\mathsf\{c\}\}\(\\theta\)\}\\left\[\\left\\\|\\mathrm\{g\}^\{X\}\_\{\\mathsf\{c\}\}\(\\theta,\\mathrm\{q\}\)\-\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{c\}\}\(\\theta,\\mathrm\{q\}\)\\right\\\|\_\{2\}^\{2\}\\right\]≤𝔼X∼ν𝖼​\(θ\)​\[‖g𝖼X​\(θ,q\)‖22\]\.\\displaystyle\\leq\\mathbb\{E\}\_\{X\\sim\\nu^\{\\mathsf\{c\}\}\(\\theta\)\}\\left\[\\left\\\|\\mathrm\{g\}^\{X\}\_\{\\mathsf\{c\}\}\(\\theta,\\mathrm\{q\}\)\\right\\\|\_\{2\}^\{2\}\\right\]\.Expanding the last term gives

𝔼X∼ν𝖼​\(θ\)​\[‖g𝖼X​\(θ,q\)‖22\]\\displaystyle\\mathbb\{E\}\_\{X\\sim\\nu^\{\\mathsf\{c\}\}\(\\theta\)\}\\left\[\\left\\\|\\mathrm\{g\}^\{X\}\_\{\\mathsf\{c\}\}\(\\theta,\\mathrm\{q\}\)\\right\\\|\_\{2\}^\{2\}\\right\]=𝔼X∼ν𝖼​\(θ\)​\[∑\(s,a\)∈𝒮×𝒜\(𝟣\(s,a\)​\(S,A\)​\[𝗋​\(S,A\)\+γ​\(q​\(S~,A~\)−λ​log⁡πθ​\(A~\|S~\)\)−q​\(S,A\)\]\)2\]\\displaystyle=\\mathbb\{E\}\_\{X\\sim\\nu^\{\\mathsf\{c\}\}\(\\theta\)\}\\left\[\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}\\left\(\\mathsf\{1\}\_\{\(s,a\)\}\(S,A\)\\Bigl\[\\mathsf\{r\}\(S,A\)\+\\gamma\\bigl\(\\mathrm\{q\}\(\\tilde\{S\},\\tilde\{A\}\)\-\\lambda\\log\\pi\_\{\\theta\}\(\\tilde\{A\}\|\\tilde\{S\}\)\\bigr\)\-\\mathrm\{q\}\(S,A\)\\Bigr\]\\right\)^\{2\}\\right\]=∑\(s,a,s~,a~\)dρθ​\(s\)​πθ​\(a\|s\)​𝖯​\(s~\|s,a\)​πθ​\(a~\|s~\)\\displaystyle=\\sum\_\{\(s,a,\\tilde\{s\},\\tilde\{a\}\)\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)\\,\\pi\_\{\\theta\}\(a\|s\)\\,\\mathsf\{P\}\(\\tilde\{s\}\|s,a\)\\,\\pi\_\{\\theta\}\(\\tilde\{a\}\|\\tilde\{s\}\)×\[𝗋​\(s,a\)\+γ​\(q​\(s~,a~\)−λ​log⁡πθ​\(a~\|s~\)\)−q​\(s,a\)\]2\.\\displaystyle\\qquad\\qquad\\times\\left\[\\mathsf\{r\}\(s,a\)\+\\gamma\\bigl\(\\mathrm\{q\}\(\\tilde\{s\},\\tilde\{a\}\)\-\\lambda\\log\\pi\_\{\\theta\}\(\\tilde\{a\}\|\\tilde\{s\}\)\\bigr\)\-\\mathrm\{q\}\(s,a\)\\right\]^\{2\}\.Using

\(a\+b\+c\+d\)2≤4​\(a2\+b2\+c2\+d2\),\(a\+b\+c\+d\)^\{2\}\\leq 4\(a^\{2\}\+b^\{2\}\+c^\{2\}\+d^\{2\}\),we deduce

𝔼X∼ν𝖼​\(θ\)​\[‖g𝖼X​\(θ,q\)‖22\]\\displaystyle\\mathbb\{E\}\_\{X\\sim\\nu^\{\\mathsf\{c\}\}\(\\theta\)\}\\left\[\\left\\\|\\mathrm\{g\}^\{X\}\_\{\\mathsf\{c\}\}\(\\theta,\\mathrm\{q\}\)\\right\\\|\_\{2\}^\{2\}\\right\]≤4​∑\(s,a,s~,a~\)dρθ​\(s\)​πθ​\(a\|s\)​𝖯​\(s~\|s,a\)​πθ​\(a~\|s~\)\\displaystyle\\leq 4\\sum\_\{\(s,a,\\tilde\{s\},\\tilde\{a\}\)\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)\\,\\pi\_\{\\theta\}\(a\|s\)\\,\\mathsf\{P\}\(\\tilde\{s\}\|s,a\)\\,\\pi\_\{\\theta\}\(\\tilde\{a\}\|\\tilde\{s\}\)×\(𝗋​\(s,a\)2\+γ2​q​\(s~,a~\)2\+γ2​λ2​log⁡πθ​\(a~\|s~\)2\+q​\(s,a\)2\)\.\\displaystyle\\qquad\\qquad\\times\\Bigl\(\\mathsf\{r\}\(s,a\)^\{2\}\+\\gamma^\{2\}\\mathrm\{q\}\(\\tilde\{s\},\\tilde\{a\}\)^\{2\}\+\\gamma^\{2\}\\lambda^\{2\}\\log\\pi\_\{\\theta\}\(\\tilde\{a\}\|\\tilde\{s\}\)^\{2\}\+\\mathrm\{q\}\(s,a\)^\{2\}\\Bigr\)\.Since the reward is bounded by11andγ≤1\\gamma\\leq 1, this yields

𝔼X∼ν𝖼​\(θ\)​\[‖g𝖼X​\(θ,q\)‖22\]\\displaystyle\\mathbb\{E\}\_\{X\\sim\\nu^\{\\mathsf\{c\}\}\(\\theta\)\}\\left\[\\left\\\|\\mathrm\{g\}^\{X\}\_\{\\mathsf\{c\}\}\(\\theta,\\mathrm\{q\}\)\\right\\\|\_\{2\}^\{2\}\\right\]≤4​\(γ2\+1\)​‖q‖∞2\+4​∑s,adρθ​\(s\)​πθ​\(a\|s\)\+4​λ2​maxs~∈𝒮​∑a~∈𝒜πθ​\(a~\|s~\)​log⁡πθ​\(a~\|s~\)2\\displaystyle\\leq 4\(\\gamma^\{2\}\+1\)\\\|\\mathrm\{q\}\\\|\_\{\\infty\}^\{2\}\+4\\sum\_\{s,a\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\}\(s\)\\,\\pi\_\{\\theta\}\(a\|s\)\+4\\lambda^\{2\}\\max\_\{\\tilde\{s\}\\in\\mathcal\{S\}\}\\sum\_\{\\tilde\{a\}\\in\\mathcal\{A\}\}\\pi\_\{\\theta\}\(\\tilde\{a\}\|\\tilde\{s\}\)\\log\\pi\_\{\\theta\}\(\\tilde\{a\}\|\\tilde\{s\}\)^\{2\}≤8​‖q‖∞2\+4\+4​λ2​maxs~∈𝒮​∑a~∈𝒜πθ​\(a~\|s~\)​log⁡πθ​\(a~\|s~\)2\.\\displaystyle\\leq 8\\\|\\mathrm\{q\}\\\|\_\{\\infty\}^\{2\}\+4\+4\\lambda^\{2\}\\max\_\{\\tilde\{s\}\\in\\mathcal\{S\}\}\\sum\_\{\\tilde\{a\}\\in\\mathcal\{A\}\}\\pi\_\{\\theta\}\(\\tilde\{a\}\|\\tilde\{s\}\)\\log\\pi\_\{\\theta\}\(\\tilde\{a\}\|\\tilde\{s\}\)^\{2\}\.Finally, using that for any probability vectorp∈𝒫​\(𝒜\)p\\in\\mathcal\{P\}\(\\mathcal\{A\}\)\(see[Lemma31](https://arxiv.org/html/2605.24357#Thmlemma31)\), we have

∑a∈𝒜p\(a\)log\(p\(a\)\)2≤1\+log\(\|𝒜\|\)2,\\sum\_\{a\\in\\mathcal\{A\}\}p\(a\)\\log\(p\(a\)\)^\{2\}\\leq 1\+\\log\(\|\\mathcal\{A\}\|\)^\{2\}\\kern 5\.0pt,concludes the proof\. ∎

### D\.3Critic recursion

Define the filtration adapted to the local iterates of the critic

ℱkh=Δσ\(Xℓ,Yℓ:ℓ∈\{1,…,k−1\},Xkp:p∈\{1,…,h\},Yk\),\\displaystyle\\mathcal\{F\}\_\{k\}^\{h\}\\overset\{\\Delta\}\{=\}\\sigma\\Big\(X\_\{\\ell\},Y\_\{\\ell\}:\\ell\\in\\\{1,\\dots,k\-1\\\},X\_\{k\}^\{p\}:p\\in\\\{1,\\dots,h\\\},Y\_\{k\}\\Big\)\\kern 5\.0pt,Additionally, define:

μ~𝖼​=Δ​\(1−γ\)2​ρmin​τλ/2,andσ𝖼2​=Δ​36\+4λ2\+36λ2log\(\|𝒜\|\)2\(1−γ\)2\.\\displaystyle\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\\overset\{\\Delta\}\{=\}\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}/2\\kern 5\.0pt,\\quad\\text\{and\}\\quad\\sigma\_\{\\mathsf\{c\}\}^\{2\}\\overset\{\\Delta\}\{=\}\\frac\{36\+4\\lambda^\{2\}\+36\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\}\{\(1\-\\gamma\)^\{2\}\}\\kern 5\.0pt\.\(27\)The following lemma bounds the variance and the bias of the critic after performingHHduring iterationkk\.

###### Lemma 18\.

Assume[Aρ\\textbf\{A\}\_\{\\rho\}](https://arxiv.org/html/2605.24357#ThmassumMDPx1)and assume thatη𝖼≤\(1−γ\)2​ρmin​τλ/40\\eta\_\{\\mathsf\{c\}\}\\leq\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}/40\. It holds that

𝔼​\[‖q^kH−q~θkλ‖22\|ℱk\]≤\(1−η𝖼​μ~𝖼\)H​‖q^k0−q~θkλ‖22\+η𝖼​σ𝖼2μ~𝖼​\(1−\(1−η𝖼​μ~𝖼\)H\),\\displaystyle\\mathbb\{E\}\\left\[\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\}^\{H\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\|\\mathcal\{F\}\_\{k\}\\right\]\\leq\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\}^\{0\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\+\\frac\{\\eta\_\{\\mathsf\{c\}\}\\sigma\_\{\\mathsf\{c\}\}^\{2\}\}\{\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\(1\-\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}\)\\kern 5\.0pt,whereμ~𝖼\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}, andσ𝖼2\\sigma\_\{\\mathsf\{c\}\}^\{2\}are defined in \([27](https://arxiv.org/html/2605.24357#A4.E27)\)\. Additionally, we have that

∥𝔼\[q^kH\|ℱk\]−q~θkλ∥22≤\(1−η𝖼μ~𝖼\)H∥q^k0−q~θkλ∥22\.\\displaystyle\\left\\\|\\mathbb\{E\}\\left\[\\hat\{\\mathrm\{q\}\}\_\{k\}^\{H\}\|\\mathcal\{F\}\_\{k\}\\right\]\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\\leq\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\}^\{0\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\\kern 5\.0pt\.

###### Proof\.

Fixk∈\{0,…,K−1\}k\\in\\\{0,\\dots,K\-1\\\}andh∈\{0,…,H−1\}h\\in\\\{0,\\dots,H\-1\\\}\. It holds that

‖q^kh\+1−q~θkλ‖22\\displaystyle\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\+1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}=‖q^kh\+η𝖼​g𝖼Xkh\+1​\(θk,q^kh\)−q~θkλ‖22\\displaystyle=\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\+\\eta\_\{\\mathsf\{c\}\}\\,\\mathrm\{g\}^\{X\_\{k\}^\{h\+1\}\}\_\{\\mathsf\{c\}\}\(\\theta\_\{k\},\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\)\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}=‖q^kh−q~θkλ‖22\+2​η𝖼​⟨g𝖼Xkh\+1​\(θk,q^kh\),q^kh−q~θkλ⟩\+η𝖼2​‖g𝖼Xkh\+1​\(θk,q^kh\)‖22\.\\displaystyle=\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\+2\\eta\_\{\\mathsf\{c\}\}\\langle\\mathrm\{g\}^\{X\_\{k\}^\{h\+1\}\}\_\{\\mathsf\{c\}\}\(\\theta\_\{k\},\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\),\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\rangle\+\\eta\_\{\\mathsf\{c\}\}^\{2\}\\left\\\|\\mathrm\{g\}^\{X\_\{k\}^\{h\+1\}\}\_\{\\mathsf\{c\}\}\(\\theta\_\{k\},\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\)\\right\\\|\_\{2\}^\{2\}\\kern 5\.0pt\.Taking the conditional expectation with respect toℱkh\\mathcal\{F\}\_\{k\}^\{h\}, and using the fact thatg¯𝖼​\(θk,q^kh\)=\(𝖳θkη𝖼​q^kh−q^kh\)/η𝖼\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{c\}\}\(\\theta\_\{k\},\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\)=\(\\mathsf\{T\}\_\{\\theta\_\{k\}\}^\{\\eta\_\{\\mathsf\{c\}\}\}\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\-\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\)/\\eta\_\{\\mathsf\{c\}\}\([Lemma17](https://arxiv.org/html/2605.24357#Thmlemma17)\) yields

𝔼​\[‖q^kh\+1−q~θkλ‖22\|ℱkh\]\\displaystyle\\mathbb\{E\}\\left\[\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\+1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\|\\mathcal\{F\}\_\{k\}^\{h\}\\right\]=‖q^kh−q~θkλ‖22\+2​η𝖼​⟨g¯𝖼​\(θk,q^kh\),q^kh−q~θkλ⟩\+η𝖼2​𝔼​\[‖g𝖼Xkh\+1​\(θk,q^kh\)‖22\|ℱkh\]\\displaystyle=\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\+2\\eta\_\{\\mathsf\{c\}\}\\langle\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{c\}\}\(\\theta\_\{k\},\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\),\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\rangle\+\\eta\_\{\\mathsf\{c\}\}^\{2\}\\mathbb\{E\}\\left\[\\left\\\|\\mathrm\{g\}^\{X\_\{k\}^\{h\+1\}\}\_\{\\mathsf\{c\}\}\(\\theta\_\{k\},\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\)\\right\\\|\_\{2\}^\{2\}\\bigg\|\\mathcal\{F\}\_\{k\}^\{h\}\\right\]=‖𝖳θkη𝖼​q^kh−q~θkλ‖22\+η𝖼2​𝔼​\[‖g𝖼Xkh\+1​\(θk,q^kh\)−g¯𝖼​\(θk,q^kh\)‖22\|ℱkh\]\\displaystyle=\\left\\\|\\mathsf\{T\}\_\{\\theta\_\{k\}\}^\{\\eta\_\{\\mathsf\{c\}\}\}\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\+\\eta\_\{\\mathsf\{c\}\}^\{2\}\\mathbb\{E\}\\left\[\\left\\\|\\mathrm\{g\}^\{X\_\{k\}^\{h\+1\}\}\_\{\\mathsf\{c\}\}\(\\theta\_\{k\},\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\)\-\\bar\{\\mathrm\{g\}\}\_\{\\mathsf\{c\}\}\(\\theta\_\{k\},\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\)\\right\\\|\_\{2\}^\{2\}\\bigg\|\\mathcal\{F\}\_\{k\}^\{h\}\\right\]≤\(1−\(1−γ\)2η𝖼ρminτλ\+η𝖼2\(1\+γ\)2\)∥q^kh−q~θkλ∥22\+η𝖼2\(8∥q^kh∥∞2\+4\+4λ2\+4λ2log\(\|𝒜\|\)2\),\\displaystyle\\leq\(1\-\(1\-\\gamma\)^\{2\}\\eta\_\{\\mathsf\{c\}\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}\+\\eta\_\{\\mathsf\{c\}\}^\{2\}\(1\+\\gamma\)^\{2\}\)\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\+\\eta\_\{\\mathsf\{c\}\}^\{2\}\\left\(8\\\|\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\\\|\_\{\\infty\}^\{2\}\+4\+4\\lambda^\{2\}\+4\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\\right\)\\kern 5\.0pt,where in the last identity, we used the contractivity of𝖳θkη𝖼\\mathsf\{T\}\_\{\\theta\_\{k\}\}^\{\\eta\_\{\\mathsf\{c\}\}\}\([Lemma16](https://arxiv.org/html/2605.24357#Thmlemma16)\) combined with the bound on the variance of the critic \([Lemma17](https://arxiv.org/html/2605.24357#Thmlemma17)\)\. Next, by Young’s inequality, we have‖q^kh‖∞2≤2​‖q^kh−q~θkλ‖∞2\+2​‖q~θkλ‖∞2\\\|\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\\\|\_\{\\infty\}^\{2\}\\leq 2\\\|\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\\|\_\{\\infty\}^\{2\}\+2\\\|\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\\|\_\{\\infty\}^\{2\}, which implies that

𝔼\[∥q^kh\+1−q~θkλ∥22\|ℱkh\]≤\(1−\(1−γ\)2η𝖼ρminτλ\+20η𝖼2\)∥q^kh−q~θkλ∥22\+η𝖼2\(16∥q~θkλ∥∞2\+4\+4λ2\+4λ2log\(\|𝒜\|\)2\)\.\\displaystyle\\mathbb\{E\}\\left\[\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\+1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\|\\mathcal\{F\}\_\{k\}^\{h\}\\right\]\\leq\(1\-\(1\-\\gamma\)^\{2\}\\eta\_\{\\mathsf\{c\}\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}\+20\\eta\_\{\\mathsf\{c\}\}^\{2\}\)\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\+\\eta\_\{\\mathsf\{c\}\}^\{2\}\\left\(16\\\|\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\\|\_\{\\infty\}^\{2\}\+4\+4\\lambda^\{2\}\+4\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\\right\)\.Taking the conditional expectation, with respect toℱk\\mathcal\{F\}\_\{k\}, the fact thatη𝖼≤\(1−γ\)2​ρmin​τλ/40\\eta\_\{\\mathsf\{c\}\}\\leq\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}/40,∥q~θkλ∥∞2≤\(2\+2λ2log\(\|𝒜\|\)2\)/\(1−γ\)2\\\|\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\\|\_\{\\infty\}^\{2\}\\leq\(2\+2\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\)/\(1\-\\gamma\)^\{2\}\(which holds by a combination of[Lemma27](https://arxiv.org/html/2605.24357#Thmlemma27)and then Young’s inequality\) gives

𝔼\[∥q^kh\+1−q~θkλ∥22\|ℱk\]≤\(1−\(1−γ\)2η𝖼ρminτλ/2\)𝔼\[∥q^kh−q~θkλ∥22\|ℱk\]\+η𝖼2\(36\+4λ2\+36λ2log\(\|𝒜\|\)2\)/\(1−γ\)2\.\\displaystyle\\mathbb\{E\}\\left\[\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\+1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\|\\mathcal\{F\}\_\{k\}\\right\]\\leq\(1\-\(1\-\\gamma\)^\{2\}\\eta\_\{\\mathsf\{c\}\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}/2\)\\mathbb\{E\}\\left\[\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\}^\{h\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\|\\mathcal\{F\}\_\{k\}\\right\]\+\\eta\_\{\\mathsf\{c\}\}^\{2\}\\left\(36\+4\\lambda^\{2\}\+36\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\\right\)/\(1\-\\gamma\)^\{2\}\\kern 5\.0pt\.Finally, unrolling the recursion yields the result, and the second claim follows from the affinity of𝖳θkη𝖼\\mathsf\{T\}\_\{\\theta\_\{k\}\}^\{\\eta\_\{\\mathsf\{c\}\}\}\. ∎

The next lemma bounds the mean\-squared error of the critic\.

###### Lemma 19\.

Assume[Aρ\\textbf\{A\}\_\{\\rho\}](https://arxiv.org/html/2605.24357#ThmassumMDPx1)and

η𝖼≤\(1−γ\)2​ρmin​τλ/40,andH≥2η𝖼​μ~𝖼​log⁡\(2\+4​C~λ2​η𝖺2\)\.\\displaystyle\\eta\_\{\\mathsf\{c\}\}\\leq\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}/40\\kern 5\.0pt,\\quad\\text\{ and \}\\quad H\\geq\\frac\{2\}\{\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\\log\(2\+4\\tilde\{C\}\_\{\\lambda\}^\{2\}\\eta\_\{\\mathsf\{a\}\}^\{2\}\)\\kern 5\.0pt\.\(28\)For anyk≥0k\\geq 0, it holds that

𝔼​\[‖q^k−q~θkλ‖22\]≤\(1−η𝖼​μ~𝖼\)H​\(k\+1\)/2​‖q^−1−q~θ0λ‖22\+C~λ2η𝖺2\(16\+8λ2\+24λ2log\(\|𝒜\|\)2\)\(1−γ\)2⋅11−\(1−η𝖼​μ~𝖼\)H/2\+2​η𝖼​σ𝖼2μ~𝖼\.\\displaystyle\\mathbb\{E\}\\left\[\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\\right\]\\leq\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\(k\+1\)/2\}\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{\-1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{0\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\+\\frac\{\\tilde\{C\}\_\{\\lambda\}^\{2\}\\eta\_\{\\mathsf\{a\}\}^\{2\}\(16\+8\\lambda^\{2\}\+24\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\)\}\{\(1\-\\gamma\)^\{2\}\}\\cdot\\frac\{1\}\{1\-\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H/2\}\}\+\\\!\\frac\{2\\eta\_\{\\mathsf\{c\}\}\\sigma\_\{\\mathsf\{c\}\}^\{2\}\}\{\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\\kern 5\.0pt\.

###### Proof\.

Firstly, applying[Lemma18](https://arxiv.org/html/2605.24357#Thmlemma18)combined with the tower property of the conditional expectation, gives

𝔼​\[‖q^k−q~θkλ‖22\]≤\(1−η𝖼​μ~𝖼\)H​𝔼​\[‖q^k−1−q~θkλ‖22\]\+η𝖼​σ𝖼2μ~𝖼​\(1−\(1−η𝖼​μ~𝖼\)H\)\.\\displaystyle\\mathbb\{E\}\\left\[\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\\right\]\\leq\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}\\mathbb\{E\}\\left\[\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\-1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\\right\]\+\\frac\{\\eta\_\{\\mathsf\{c\}\}\\sigma\_\{\\mathsf\{c\}\}^\{2\}\}\{\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\(1\-\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}\)\\kern 5\.0pt\.Thus, the bound holds in the case wherek=0k=0\. Now consider the case wherek≥1k\\geq 1\. Adding and subtractingq~θk−1λ\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\-1\}\}\}^\{\\lambda\}inside the squared norm of the right\-hand side, and applying Jensen’s inequality gives

𝔼​\[‖q^k−q~θkλ‖22\]≤\(1−η𝖼​μ~𝖼\)H​\[2​𝔼​\[‖q^k−1−q~θk−1λ‖22\]\+2​𝔼​\[‖q~θk−1λ−q~θkλ‖22\]\]\+η𝖼​σ𝖼2μ~𝖼​\(1−\(1−η𝖼​μ~𝖼\)H\)\.\\displaystyle\\mathbb\{E\}\\left\[\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\\right\]\\leq\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}\\left\[2\\mathbb\{E\}\\left\[\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\-1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\-1\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\\right\]\+2\\mathbb\{E\}\\left\[\\left\\\|\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\-1\}\}\}^\{\\lambda\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\\right\]\\right\]\+\\frac\{\\eta\_\{\\mathsf\{c\}\}\\sigma\_\{\\mathsf\{c\}\}^\{2\}\}\{\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\(1\-\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}\)\\kern 5\.0pt\.Applying[Corollary4](https://arxiv.org/html/2605.24357#Thmcorollary4)yields

𝔼​\[‖q^k−q~θkλ‖22\]≤\(1−η𝖼​μ~𝖼\)H​\[2​𝔼​\[‖q^k−1−q~θk−1λ‖22\]\+2​C~λ2​η𝖺2​𝔼​\[a^k−1​\(Sk,Ak\)2\]\]\+η𝖼​σ𝖼2μ~𝖼​\(1−\(1−η𝖼​μ~𝖼\)H\)\.\\displaystyle\\\!\\\!\\\!\\mathbb\{E\}\\left\[\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\}\\\!\-\\\!\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\\right\]\\\!\\leq\\\!\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}\\left\[2\\mathbb\{E\}\\left\[\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\-1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\-1\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\\right\]\\\!\+\\\!2\\tilde\{C\}\_\{\\lambda\}^\{2\}\\eta\_\{\\mathsf\{a\}\}^\{2\}\\mathbb\{E\}\\left\[\\hat\{\\mathrm\{a\}\}\_\{k\-1\}\(S\_\{k\},A\_\{k\}\)^\{2\}\\right\]\\right\]\\\!\+\\\!\\frac\{\\eta\_\{\\mathsf\{c\}\}\\sigma\_\{\\mathsf\{c\}\}^\{2\}\}\{\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\(1\-\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}\)\\kern 5\.0pt\.\(29\)Next, using Jensen’s inequality combined with the definition of the true regularized advantage \([6](https://arxiv.org/html/2605.24357#S3.E6)\) and the estimated regularized advantage \([13](https://arxiv.org/html/2605.24357#S3.E13)\) gives

𝔼​\[a^k−1​\(Sk,Ak\)2\]\\displaystyle\\mathbb\{E\}\\left\[\\hat\{\\mathrm\{a\}\}\_\{k\-1\}\(S\_\{k\},A\_\{k\}\)^\{2\}\\right\]≤2​𝔼​\[\(a^k−1​\(Sk,Ak\)−a~θk−1λ​\(Sk,Ak\)\)2\]\+2​𝔼​\[a~θk−1λ​\(Sk,Ak\)2\]\\displaystyle\\leq 2\\mathbb\{E\}\\left\[\\left\(\\hat\{\\mathrm\{a\}\}\_\{k\-1\}\(S\_\{k\},A\_\{k\}\)\-\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\\theta\_\{k\-1\}\}^\{\\lambda\}\(S\_\{k\},A\_\{k\}\)\\right\)^\{2\}\\right\]\+2\\mathbb\{E\}\\left\[\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\\theta\_\{k\-1\}\}^\{\\lambda\}\(S\_\{k\},A\_\{k\}\)^\{2\}\\right\]≤2𝔼\[∑s∈𝒮∑a∈𝒜dρθk−1\(s\)πθk−1\(a\|s\)\(q^k−1\(s,a\)−λlog\(πθk−1\(a\|s\)\)−v^k−1\(s\)\\displaystyle\\leq 2\\mathbb\{E\}\\bigg\[\\sum\_\{s\\in\\mathcal\{S\}\}\\sum\_\{a\\in\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\_\{k\-1\}\}\(s\)\\pi\_\{\\theta\_\{k\-1\}\}\(a\|s\)\\bigg\(\\hat\{\\mathrm\{q\}\}\_\{k\-1\}\(s,a\)\-\\lambda\\log\(\\pi\_\{\\theta\_\{k\-1\}\}\(a\|s\)\)\-\\hat\{\\mathrm\{v\}\}\_\{k\-1\}\(s\)−\(q~θk−1λ\(s,a\)−λlog\(πθk−1\(a\|s\)\)−v~θk−1λ\(s\)\)\)2\]\\displaystyle\\hskip 150\.00023pt\-\(\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\theta\_\{k\-1\}\}^\{\\lambda\}\(s,a\)\-\\lambda\\log\(\\pi\_\{\\theta\_\{k\-1\}\}\(a\|s\)\)\-\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\theta\_\{k\-1\}\}^\{\\lambda\}\(s\)\)\\bigg\)^\{2\}\\bigg\]\+2​𝔼​\[∑s∈𝒮∑a∈𝒜dρθk−1​\(s\)​πθk−1​\(a\|s\)​\(q~θk−1λ​\(s,a\)−λ​log⁡\(πθk−1​\(a\|s\)\)−v~θk−1λ​\(s\)\)2\]\.\\displaystyle\+2\\mathbb\{E\}\\left\[\\sum\_\{s\\in\\mathcal\{S\}\}\\sum\_\{a\\in\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\_\{k\-1\}\}\(s\)\\pi\_\{\\theta\_\{k\-1\}\}\(a\|s\)\(\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\theta\_\{k\-1\}\}^\{\\lambda\}\(s,a\)\-\\lambda\\log\(\\pi\_\{\\theta\_\{k\-1\}\}\(a\|s\)\)\-\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\theta\_\{k\-1\}\}^\{\\lambda\}\(s\)\)^\{2\}\\right\]\\kern 5\.0pt\.Next, using that for anyp∈𝒫​\(𝒜\)p\\in\\mathcal\{P\}\(\\mathcal\{A\}\), andx1,…,x\|𝒜\|∈ℝx\_\{1\},\\dots,x\_\{\|\\mathcal\{A\}\|\}\\in\\mathbb\{R\}, we have∑i=1\|𝒜\|pi​\(xi−x¯\)2≤∑i=1\|𝒜\|pi​xi2\\sum\_\{i=1\}^\{\|\\mathcal\{A\}\|\}p\_\{i\}\(x\_\{i\}\-\\bar\{x\}\)^\{2\}\\leq\\sum\_\{i=1\}^\{\|\\mathcal\{A\}\|\}p\_\{i\}x\_\{i\}^\{2\}, wherex¯=∑i=1\|𝒜\|pi​xi\\bar\{x\}=\\sum\_\{i=1\}^\{\|\\mathcal\{A\}\|\}p\_\{i\}x\_\{i\}, gives

𝔼​\[a^k−1​\(Sk,Ak\)2\]\\displaystyle\\mathbb\{E\}\\left\[\\hat\{\\mathrm\{a\}\}\_\{k\-1\}\(S\_\{k\},A\_\{k\}\)^\{2\}\\right\]≤2​𝔼​\[∑s∈𝒮∑a∈𝒜dρθk−1​\(s\)​πθk−1​\(a\|s\)​\(q^k−1​\(s,a\)−q~θk−1λ​\(s,a\)\)2\]\\displaystyle\\leq 2\\mathbb\{E\}\\left\[\\sum\_\{s\\in\\mathcal\{S\}\}\\sum\_\{a\\in\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\_\{k\-1\}\}\(s\)\\pi\_\{\\theta\_\{k\-1\}\}\(a\|s\)\\left\(\\hat\{\\mathrm\{q\}\}\_\{k\-1\}\(s,a\)\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\theta\_\{k\-1\}\}^\{\\lambda\}\(s,a\)\\right\)^\{2\}\\right\]\+2​𝔼​\[∑s∈𝒮∑a∈𝒜dρθk−1​\(s\)​πθk−1​\(a\|s\)​\(q~θk−1λ​\(s,a\)−λ​log⁡\(πθk−1​\(a\|s\)\)\)2\]\\displaystyle\+2\\mathbb\{E\}\\left\[\\sum\_\{s\\in\\mathcal\{S\}\}\\sum\_\{a\\in\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\theta\_\{k\-1\}\}\(s\)\\pi\_\{\\theta\_\{k\-1\}\}\(a\|s\)\(\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\theta\_\{k\-1\}\}^\{\\lambda\}\(s,a\)\-\\lambda\\log\(\\pi\_\{\\theta\_\{k\-1\}\}\(a\|s\)\)\)^\{2\}\\right\]≤2𝔼\[∥q^k−1−q~θk−1λ∥22\]\+8\(1\+λ2log\(\|𝒜\|\)2\)\(1−γ\)2\+4λ2\+4λ2log\(\|𝒜\|\)2,\\displaystyle\\leq 2\\mathbb\{E\}\\left\[\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\-1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\-1\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\\right\]\+\\frac\{8\(1\+\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\)\}\{\(1\-\\gamma\)^\{2\}\}\+4\\lambda^\{2\}\+4\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\\kern 5\.0pt,\(30\)where in the last inequality, we used Young’s inequality combined with[Lemma27](https://arxiv.org/html/2605.24357#Thmlemma27)and[Lemma31](https://arxiv.org/html/2605.24357#Thmlemma31)\. Plugging in the previous inequality in[Equation29](https://arxiv.org/html/2605.24357#A4.E29)gives

𝔼​\[‖q^k−q~θkλ‖22\]\\displaystyle\\\!\\\!\\\!\\mathbb\{E\}\\left\[\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\}\\\!\-\\\!\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\\right\]\\\!≤\(1−η𝖼​μ~𝖼\)H​\[\(2\+4​C~λ2​η𝖺2\)​𝔼​\[‖q^k−1−q~θk−1λ‖22\]\+C~λ2η𝖺2\(16\+8λ2\+24λ2log\(\|𝒜\|\)2\)\(1−γ\)2\]\\displaystyle\\leq\\\!\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}\\left\[\(2\+4\\tilde\{C\}\_\{\\lambda\}^\{2\}\\eta\_\{\\mathsf\{a\}\}^\{2\}\)\\mathbb\{E\}\\left\[\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\-1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\-1\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\\right\]\\\!\+\\\!\\frac\{\\tilde\{C\}\_\{\\lambda\}^\{2\}\\eta\_\{\\mathsf\{a\}\}^\{2\}\(16\+8\\lambda^\{2\}\+24\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\)\}\{\(1\-\\gamma\)^\{2\}\}\\right\]\\\!\+η𝖼​σ𝖼2μ~𝖼​\(1−\(1−η𝖼​μ~𝖼\)H\)\.\\displaystyle\+\\\!\\frac\{\\eta\_\{\\mathsf\{c\}\}\\sigma\_\{\\mathsf\{c\}\}^\{2\}\}\{\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\(1\-\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}\)\\kern 5\.0pt\.Next, unrolling the previous recursion yields

𝔼​\[‖q^k−q~θkλ‖22\]\\displaystyle\\\!\\\!\\\!\\mathbb\{E\}\\left\[\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\}\\\!\-\\\!\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\\right\]\\\!≤\[\(1−η𝖼​μ~𝖼\)H⋅\(2\+4​C~λ2​η𝖺2\)\]k\+1​‖q^−1−q~θ0λ‖22\\displaystyle\\leq\\\!\\left\[\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}\\cdot\(2\+4\\tilde\{C\}\_\{\\lambda\}^\{2\}\\eta\_\{\\mathsf\{a\}\}^\{2\}\)\\right\]^\{k\+1\}\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{\-1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{0\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\+C~λ2η𝖺2\(16\+8λ2\+24λ2log\(\|𝒜\|\)2\)\(1−γ\)2​∑ℓ=0k\[\(1−η𝖼​μ~𝖼\)H⋅\(2\+4​C~λ2​η𝖺2\)\]ℓ\\displaystyle\+\\frac\{\\tilde\{C\}\_\{\\lambda\}^\{2\}\\eta\_\{\\mathsf\{a\}\}^\{2\}\(16\+8\\lambda^\{2\}\+24\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\)\}\{\(1\-\\gamma\)^\{2\}\}\\sum\_\{\\ell=0\}^\{k\}\\left\[\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}\\cdot\(2\+4\\tilde\{C\}\_\{\\lambda\}^\{2\}\\eta\_\{\\mathsf\{a\}\}^\{2\}\)\\right\]^\{\\ell\}\+η𝖼​σ𝖼2μ~𝖼​\(1−\(1−η𝖼​μ~𝖼\)H\)​∑ℓ=0k\[\(1−η𝖼​μ~𝖼\)H⋅\(2\+4​C~λ2​η𝖺2\)\]ℓ\.\\displaystyle\+\\\!\\frac\{\\eta\_\{\\mathsf\{c\}\}\\sigma\_\{\\mathsf\{c\}\}^\{2\}\}\{\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\(1\-\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}\)\\sum\_\{\\ell=0\}^\{k\}\\left\[\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}\\cdot\(2\+4\\tilde\{C\}\_\{\\lambda\}^\{2\}\\eta\_\{\\mathsf\{a\}\}^\{2\}\)\\right\]^\{\\ell\}\\kern 5\.0pt\.AsH≥2η𝖼​μ~𝖼​log⁡\(2\+4​C~λ2​η𝖺2\)H\\geq\\frac\{2\}\{\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\\log\(2\+4\\tilde\{C\}\_\{\\lambda\}^\{2\}\\eta\_\{\\mathsf\{a\}\}^\{2\}\), then\(1−η𝖼​μ~𝖼\)H⋅\(2\+4​C~λ2​η𝖺2\)≤\(1−η𝖼​μ~𝖼\)H/2\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}\\cdot\(2\+4\\tilde\{C\}\_\{\\lambda\}^\{2\}\\eta\_\{\\mathsf\{a\}\}^\{2\}\)\\leq\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H/2\}which implies

𝔼​\[‖q^k−q~θkλ‖22\]\\displaystyle\\\!\\\!\\\!\\mathbb\{E\}\\left\[\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\}\\\!\-\\\!\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\\right\]\\\!≤\(1−η𝖼​μ~𝖼\)H​\(k\+1\)/2​‖q^−1−q~θ0λ‖22\+C~λ2η𝖺2\(16\+8λ2\+24λ2log\(\|𝒜\|\)2\)\(1−γ\)2​∑ℓ=0k\(1−η𝖼​μ~𝖼\)H​ℓ/2\\displaystyle\\leq\\\!\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\(k\+1\)/2\}\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{\-1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{0\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\+\\frac\{\\tilde\{C\}\_\{\\lambda\}^\{2\}\\eta\_\{\\mathsf\{a\}\}^\{2\}\(16\+8\\lambda^\{2\}\+24\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\)\}\{\(1\-\\gamma\)^\{2\}\}\\sum\_\{\\ell=0\}^\{k\}\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\\ell/2\}\+η𝖼​σ𝖼2μ~𝖼​\(1−\(1−η𝖼​μ~𝖼\)H\)​∑ℓ=0k\(1−η𝖼​μ~𝖼\)H​ℓ/2\\displaystyle\+\\\!\\frac\{\\eta\_\{\\mathsf\{c\}\}\\sigma\_\{\\mathsf\{c\}\}^\{2\}\}\{\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\(1\-\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}\)\\sum\_\{\\ell=0\}^\{k\}\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\\ell/2\}≤\(1−η𝖼​μ~𝖼\)H​\(k\+1\)/2​‖q^−1−q~θ0λ‖22\+C~λ2η𝖺2\(16\+8λ2\+24λ2log\(\|𝒜\|\)2\)\(1−γ\)2⋅1−\(1−η𝖼​μ~𝖼\)H​\(k\+1\)/21−\(1−η𝖼​μ~𝖼\)H/2\\displaystyle\\leq\\\!\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\(k\+1\)/2\}\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{\-1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{0\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\+\\frac\{\\tilde\{C\}\_\{\\lambda\}^\{2\}\\eta\_\{\\mathsf\{a\}\}^\{2\}\(16\+8\\lambda^\{2\}\+24\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\)\}\{\(1\-\\gamma\)^\{2\}\}\\cdot\\frac\{1\-\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\(k\+1\)/2\}\}\{1\-\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H/2\}\}\+η𝖼​σ𝖼2μ~𝖼​\(1−\(1−η𝖼​μ~𝖼\)H\)⋅1−\(1−η𝖼​μ~𝖼\)H​\(k\+1\)/21−\(1−η𝖼​μ~𝖼\)H/2\.\\displaystyle\+\\\!\\frac\{\\eta\_\{\\mathsf\{c\}\}\\sigma\_\{\\mathsf\{c\}\}^\{2\}\}\{\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\(1\-\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}\)\\cdot\\frac\{1\-\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\(k\+1\)/2\}\}\{1\-\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H/2\}\}\\kern 5\.0pt\.Finally, using that1−\(1−η𝖼​μ~𝖼\)H1−\(1−η𝖼​μ~𝖼\)H/2≤2\\frac\{1\-\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}\}\{1\-\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H/2\}\}\\leq 2concludes the proof\. ∎

We define

B:=2​‖q^−1−q~θ0λ‖22\+2C~λ2ρmin2τλ2\(2\+λ2\+3λ2log\(\|𝒜\|\)2\)L2\+2​\(1−γ\)2​ρmin​τλ​σ𝖼220​μ~𝖼\+2\|𝒮\|\|𝒜\|\(1\+λ2log\(\|𝒜\|\)2\)\(1−γ\)2\.\\displaystyle B:=2\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{\-1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{0\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\+\\frac\{2\\tilde\{C\}\_\{\\lambda\}^\{2\}\\rho\_\{\\min\}^\{2\}\\tau\_\{\\lambda\}^\{2\}\(2\+\\lambda^\{2\}\+3\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\)\}\{L^\{2\}\}\+\\\!\\frac\{2\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}\\sigma\_\{\\mathsf\{c\}\}^\{2\}\}\{20\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\+\\frac\{2\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\(1\+\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\)\}\{\(1\-\\gamma\)^\{2\}\}\\kern 5\.0pt\.\(31\)
###### Corollary 5\.

Assume[Aρ\\textbf\{A\}\_\{\\rho\}](https://arxiv.org/html/2605.24357#ThmassumMDPx1),

η𝖺≤\(1−γ\)​ρmin​τλ8​L,\\eta\_\{\\mathsf\{a\}\}\\leq\\frac\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\{8L\}\\kern 5\.0pt,and thatη𝖼\\eta\_\{\\mathsf\{c\}\}andHHsatisfy the condition \([28](https://arxiv.org/html/2605.24357#A4.E28)\)\. Then, for anyk≥1k\\geq 1, it holds that

11−\(1−η𝖼​μ~𝖼\)H/2≤2,𝔼​\[‖q^k−1−q~θkλ‖22\]≤B\.\\displaystyle\\frac\{1\}\{1\-\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H/2\}\}\\leq 2,\\qquad\\mathbb\{E\}\\\!\\left\[\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\-1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\\right\]\\leq B\.

###### Proof\.

We bound the two terms separately\.

#### Bound on the first term\.

Using

H≥2η𝖼​μ~𝖼​log⁡\(2\+4​C~λ2​η𝖺2\)H\\geq\\frac\{2\}\{\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\\log\\\!\\Bigl\(2\+4\\tilde\{C\}\_\{\\lambda\}^\{2\}\\eta\_\{\\mathsf\{a\}\}^\{2\}\\Bigr\)and the inequality1−x≤e−x1\-x\\leq e^\{\-x\}forx≥0x\\geq 0, we obtain

\(1−η𝖼​μ~𝖼\)H/2≤exp⁡\(−η𝖼​μ~𝖼​H/2\)≤12\+4​C~λ2​η𝖺2≤12\.\\displaystyle\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H/2\}\\leq\\exp\\\!\\left\(\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}H/2\\right\)\\leq\\frac\{1\}\{2\+4\\tilde\{C\}\_\{\\lambda\}^\{2\}\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\\leq\\frac\{1\}\{2\}\\kern 5\.0pt\.Therefore,

11−\(1−η𝖼​μ~𝖼\)H/2≤2\.\\displaystyle\\frac\{1\}\{1\-\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H/2\}\}\\leq 2\.\(32\)

#### Bound on the second term\.

Applying Young’s inequality, it holds that

𝔼​\[‖q^k−1−q~θkλ‖22\]≤2​𝔼​\[‖q^k−1−q~θk−1λ‖22\]⏟\(𝐖\)\+2​𝔼​\[‖q~θk−1λ−q~θkλ‖22\]⏟\(𝐙\)\.\\displaystyle\\mathbb\{E\}\\\!\\left\[\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\-1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\\right\]\\leq 2\\underbrace\{\\mathbb\{E\}\\\!\\left\[\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\-1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\-1\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\\right\]\}\_\{\\mathbf\{\(W\)\}\}\+2\\underbrace\{\\mathbb\{E\}\\\!\\left\[\\left\\\|\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\-1\}\}\}^\{\\lambda\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\\right\]\}\_\{\\mathbf\{\(Z\)\}\}\\kern 5\.0pt\.Bounding\(𝐖\)\\mathbf\{\(W\)\}\.Applying[Lemma19](https://arxiv.org/html/2605.24357#Thmlemma19), yields

𝔼​\[‖q^k−1−q~θk−1λ‖22\]\\displaystyle\\mathbb\{E\}\\left\[\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\-1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\-1\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\\right\]≤\(1−η𝖼​μ~𝖼\)H​k/2​‖q^−1−q~θ0λ‖22\+C~λ2η𝖺2\(16\+8λ2\+24λ2log\(\|𝒜\|\)2\)\(1−γ\)2⋅11−\(1−η𝖼​μ~𝖼\)H/2\\displaystyle\\leq\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{Hk/2\}\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{\-1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{0\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\+\\frac\{\\tilde\{C\}\_\{\\lambda\}^\{2\}\\eta\_\{\\mathsf\{a\}\}^\{2\}\(16\+8\\lambda^\{2\}\+24\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\)\}\{\(1\-\\gamma\)^\{2\}\}\\cdot\\frac\{1\}\{1\-\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H/2\}\}\+2​η𝖼​σ𝖼2μ~𝖼\.\\displaystyle\+\\\!\\frac\{2\\eta\_\{\\mathsf\{c\}\}\\sigma\_\{\\mathsf\{c\}\}^\{2\}\}\{\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\\kern 5\.0pt\.Using[Equation32](https://arxiv.org/html/2605.24357#A4.E32), combined with the fact that1−η𝖼​μ~𝖼≤11\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\\leq 1yields

\(𝐖\)\\displaystyle\\mathbf\{\(W\)\}≤‖q^−1−q~θ0λ‖22\+C~λ2η𝖺2\(32\+16λ2\+48λ2log\(\|𝒜\|\)2\)\(1−γ\)2\+2​η𝖼​σ𝖼2μ~𝖼\.\\displaystyle\\leq\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{\-1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{0\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\+\\frac\{\\tilde\{C\}\_\{\\lambda\}^\{2\}\\eta\_\{\\mathsf\{a\}\}^\{2\}\(32\+16\\lambda^\{2\}\+48\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\)\}\{\(1\-\\gamma\)^\{2\}\}\+\\\!\\frac\{2\\eta\_\{\\mathsf\{c\}\}\\sigma\_\{\\mathsf\{c\}\}^\{2\}\}\{\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\\kern 5\.0pt\.Plugging in the conditions onη𝖺\\eta\_\{\\mathsf\{a\}\}andη𝖼\\eta\_\{\\mathsf\{c\}\}, we get

\(𝐖\)≤‖q^−1−q~θ0λ‖22\+C~λ2ρmin2τλ2\(2\+λ2\+3λ2log\(\|𝒜\|\)2\)L2\+\(1−γ\)2​ρmin​τλ​σ𝖼220​μ~𝖼\.\\displaystyle\\mathbf\{\(W\)\}\\leq\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{\-1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{0\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\+\\frac\{\\tilde\{C\}\_\{\\lambda\}^\{2\}\\rho\_\{\\min\}^\{2\}\\tau\_\{\\lambda\}^\{2\}\(2\+\\lambda^\{2\}\+3\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\)\}\{L^\{2\}\}\+\\\!\\frac\{\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}\\sigma\_\{\\mathsf\{c\}\}^\{2\}\}\{20\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\\kern 5\.0pt\.Bounding\(𝐙\)\\mathbf\{\(Z\)\}\.Using that‖q~θk−1λ−q~θkλ‖∞≤\(1\+λ​log⁡\(\|𝒜\|\)\)/\(1−γ\)\\left\\\|\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\-1\}\}\}^\{\\lambda\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{\\infty\}\\leq\(1\+\\lambda\\log\(\|\\mathcal\{A\}\|\)\)/\(1\-\\gamma\), combined with the fact that for anyx∈ℝ\|𝒮\|​\|𝒜\|x\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}, we have‖x‖∞≤\|𝒮\|​\|𝒜\|​‖x‖2\\left\\\|x\\right\\\|\_\{\\infty\}\\leq\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\\left\\\|x\\right\\\|\_\{2\}, gives

\(𝐙\)≤2\|𝒮\|\|𝒜\|\(1\+λ2log\(\|𝒜\|\)2\)/\(1−γ\)2\.\\displaystyle\\mathbf\{\(Z\)\}\\leq 2\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\(1\+\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\)/\(1\-\\gamma\)^\{2\}\\kern 5\.0pt\.Combining the two previous bounds concludes the proof ∎

## Appendix EGeneral Analysis of[Ent\-AC](https://arxiv.org/html/2605.24357#alg1): Combining the Two Recursions

Next, we derive the convergence rate of[Ent\-AC](https://arxiv.org/html/2605.24357#alg1)\. In this section, we prove the final theorem that combines both the actor recursion and the critic recursion\.

###### Theorem 4\.

Assume[Aρ\\textbf\{A\}\_\{\\rho\}](https://arxiv.org/html/2605.24357#ThmassumMDPx1)and assume thatη𝖼≤\(1−γ\)2​ρmin​τλ/40\\eta\_\{\\mathsf\{c\}\}\\leq\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}/40,H≥2η𝖼​μ~𝖼​log⁡\(2\+4​C~λ2​η𝖺2\)H\\geq\\frac\{2\}\{\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\\log\(2\+4\\tilde\{C\}\_\{\\lambda\}^\{2\}\\eta\_\{\\mathsf\{a\}\}^\{2\}\), and thatη𝖺≤\(1−γ\)​ρmin​τλ8​L\\eta\_\{\\mathsf\{a\}\}\\leq\\frac\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\{8L\}\. For anyk≥0k\\geq 0, it holds that

𝔼​\[J~λ⋆−J~λ​\(θK\)\]\\displaystyle\\mathbb\{E\}\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{K\}\)\\right\]≤\(1−η𝖺​μ¯~λ8\)K\[J~λ⋆−J~λ\(θ0\)\]\+2​L​η𝖺2\(1−γ\)2Kmax\(1−η𝖺​μ¯~λ8,\(1−η𝖼μ~𝖼\)H/2\)K∥q^−1−q~θ0λ∥22\\displaystyle\\leq\\left\(1\-\\frac\{\\eta\_\{\\mathsf\{a\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\}\{8\}\\right\)^\{K\}\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{0\}\)\\right\]\+\\frac\{2L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)^\{2\}\}K\\max\\left\(1\-\\frac\{\\eta\_\{\\mathsf\{a\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\}\{8\},\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H/2\}\\right\)^\{K\}\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{\-1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{0\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\+16​\(1−η𝖼​μ~𝖼\)Hμ¯~λ​\(1−γ\)2​B\+η𝖺3​256C~λ2L\(1\+λ2log\(\|𝒜\|\)2\)μ¯~λ​\(1−γ\)4\+32​L​η𝖺​η𝖼​σ𝖼2\(1−γ\)2​μ~𝖼​μ¯~λ\.\\displaystyle\+\\frac\{16\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}\}\{\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\(1\-\\gamma\)^\{2\}\}B\+\\eta\_\{\\mathsf\{a\}\}^\{3\}\\frac\{256\\tilde\{C\}\_\{\\lambda\}^\{2\}L\\bigl\(1\+\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\\bigr\)\}\{\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\(1\-\\gamma\)^\{4\}\}\+\\frac\{32L\\eta\_\{\\mathsf\{a\}\}\\eta\_\{\\mathsf\{c\}\}\\sigma\_\{\\mathsf\{c\}\}^\{2\}\}\{\(1\-\\gamma\)^\{2\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\}\\kern 5\.0pt\.

###### Proof\.

Using[Lemma13](https://arxiv.org/html/2605.24357#Thmlemma13), it holds that

𝔼​\[J~λ⋆−J~λ​\(θk\+1\)\|ℱk\]\\displaystyle\\mathbb\{E\}\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\+1\}\)\\big\|\\mathcal\{F\}\_\{k\}\\right\]≤J~λ⋆−J~λ​\(θk\)−\(η𝖺2−2​L​η𝖺2\(1−γ\)​ρmin​τλ\)​‖∇J~λ​\(θk\)‖22\\displaystyle\\leq\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\-\\left\(\\frac\{\\eta\_\{\\mathsf\{a\}\}\}\{2\}\-\\frac\{2L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\\right\)\\left\\\|\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\+2​η𝖺\(1−γ\)2​∑\(s,a\)∈𝒮×𝒜dρθk​\(s\)2​πθk​\(a\|s\)2​\(𝔼​\[a^k​\(s,a\)\|ℱk\]−a~θkλ​\(s,a\)\)2\\displaystyle\+\\frac\{2\\eta\_\{\\mathsf\{a\}\}\}\{\(1\-\\gamma\)^\{2\}\}\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}\(s\)^\{2\}\\pi\_\{\\theta\_\{k\}\}\(a\|s\)^\{2\}\\left\(\\mathbb\{E\}\[\\hat\{\\mathrm\{a\}\}\_\{k\}\(s,a\)\|\\mathcal\{F\}\_\{k\}\]\-\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\(s,a\)\\right\)^\{2\}\+2​L​η𝖺2\(1−γ\)2​∑\(s,a\)∈𝒮×𝒜dρθk​\(s\)​πθk​\(a\|s\)​𝔼​\[\(a^k​\(s,a\)−a~θkλ​\(s,a\)\)2\|ℱk\]\.\\displaystyle\+\\frac\{2L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)^\{2\}\}\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}\(s\)\\pi\_\{\\theta\_\{k\}\}\(a\|s\)\\mathbb\{E\}\\left\[\(\\hat\{\\mathrm\{a\}\}\_\{k\}\(s,a\)\-\\tilde\{\\mathrm\{a\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\(s,a\)\)^\{2\}\|\\mathcal\{F\}\_\{k\}\\right\]\\kern 5\.0pt\.Next, using thatdρθk​\(s\)​πθk​\(a\|s\)≤1d\_\{\\rho\}^\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}\(s\)\\pi\_\{\\theta\_\{k\}\}\(a\|s\)\\leq 1, combined with the fact that for anyp∈𝒫​\(𝒜\)p\\in\\mathcal\{P\}\(\\mathcal\{A\}\), andx1,…,x\|𝒜\|∈ℝx\_\{1\},\\dots,x\_\{\|\\mathcal\{A\}\|\}\\in\\mathbb\{R\}, we have∑i=1\|𝒜\|pi​\(xi−x¯\)2≤∑i=1\|𝒜\|pi​xi2\\sum\_\{i=1\}^\{\|\\mathcal\{A\}\|\}p\_\{i\}\(x\_\{i\}\-\\bar\{x\}\)^\{2\}\\leq\\sum\_\{i=1\}^\{\|\\mathcal\{A\}\|\}p\_\{i\}x\_\{i\}^\{2\}wherex¯\\bar\{x\}is thepp\-average of the\(xi\)i=1\|𝒜\|\(x\_\{i\}\)\_\{i=1\}^\{\|\\mathcal\{A\}\|\}gives

𝔼​\[J~λ⋆−J~λ​\(θk\+1\)\|ℱk\]\\displaystyle\\mathbb\{E\}\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\+1\}\)\\big\|\\mathcal\{F\}\_\{k\}\\right\]≤J~λ⋆−J~λ​\(θk\)−\(η𝖺2−2​L​η𝖺2\(1−γ\)​ρmin​τλ\)​‖∇J~λ​\(θk\)‖22\\displaystyle\\leq\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\-\\left\(\\frac\{\\eta\_\{\\mathsf\{a\}\}\}\{2\}\-\\frac\{2L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\\right\)\\left\\\|\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\+2​η𝖺\(1−γ\)2​∑\(s,a\)∈𝒮×𝒜dρθk​\(s\)​πθk​\(a\|s\)​\(𝔼​\[q^k​\(s,a\)\|ℱk\]−q~θkλ​\(s,a\)\)2\\displaystyle\+\\frac\{2\\eta\_\{\\mathsf\{a\}\}\}\{\(1\-\\gamma\)^\{2\}\}\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}\(s\)\\pi\_\{\\theta\_\{k\}\}\(a\|s\)\\left\(\\mathbb\{E\}\[\\hat\{\\mathrm\{q\}\}\_\{k\}\(s,a\)\|\\mathcal\{F\}\_\{k\}\]\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\(s,a\)\\right\)^\{2\}\+2​L​η𝖺2\(1−γ\)2​∑\(s,a\)∈𝒮×𝒜dρθk​\(s\)​πθk​\(a\|s\)​𝔼​\[\(q^k​\(s,a\)−q~θkλ​\(s,a\)\)2\|ℱk\]\\displaystyle\+\\frac\{2L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)^\{2\}\}\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}\(s\)\\pi\_\{\\theta\_\{k\}\}\(a\|s\)\\mathbb\{E\}\\left\[\(\\hat\{\\mathrm\{q\}\}\_\{k\}\(s,a\)\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\(s,a\)\)^\{2\}\|\\mathcal\{F\}\_\{k\}\\right\]≤J~λ⋆−J~λ​\(θk\)−\(η𝖺2−2​L​η𝖺2\(1−γ\)​ρmin​τλ\)​‖∇J~λ​\(θk\)‖22\\displaystyle\\leq\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\-\\left\(\\frac\{\\eta\_\{\\mathsf\{a\}\}\}\{2\}\-\\frac\{2L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\\right\)\\left\\\|\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\+2​η𝖺\(1−γ\)2∥𝔼\[q^k\|ℱk\]−q~θkλ∥22\+2​L​η𝖺2\(1−γ\)2𝔼\[∥q^k−q~θkλ∥22\|ℱk\]\.\\displaystyle\+\\frac\{2\\eta\_\{\\mathsf\{a\}\}\}\{\(1\-\\gamma\)^\{2\}\}\\left\\\|\\mathbb\{E\}\[\\hat\{\\mathrm\{q\}\}\_\{k\}\|\\mathcal\{F\}\_\{k\}\]\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\+\\frac\{2L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)^\{2\}\}\\mathbb\{E\}\\left\[\\\|\\hat\{\\mathrm\{q\}\}\_\{k\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\\|\_\{2\}^\{2\}\|\\mathcal\{F\}\_\{k\}\\right\]\\kern 5\.0pt\.Taking the expectation with respect to all the stochasticity, and using[Lemma18](https://arxiv.org/html/2605.24357#Thmlemma18), combined with[Lemma19](https://arxiv.org/html/2605.24357#Thmlemma19)gives

𝔼​\[J~λ⋆−J~λ​\(θk\+1\)\]≤𝔼​\[J~λ⋆−J~λ​\(θk\)\]−\(η𝖺2−2​L​η𝖺2\(1−γ\)​ρmin​τλ\)​𝔼​\[‖∇J~λ​\(θk\)‖22\]\\displaystyle\\mathbb\{E\}\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\+1\}\)\\right\]\\leq\\mathbb\{E\}\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\]\-\\left\(\\frac\{\\eta\_\{\\mathsf\{a\}\}\}\{2\}\-\\frac\{2L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\\right\)\\mathbb\{E\}\\left\[\\left\\\|\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\\right\]\+2​η𝖺\(1−γ\)2​\(1−η𝖼​μ~𝖼\)H​𝔼​\[‖q^k−1−q~θkλ‖22\]\\displaystyle\+\\frac\{2\\eta\_\{\\mathsf\{a\}\}\}\{\(1\-\\gamma\)^\{2\}\}\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}\\mathbb\{E\}\\left\[\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{k\-1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{k\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\\right\]\+2​L​η𝖺2\(1−γ\)2​\[\(1−η𝖼​μ~𝖼\)H​\(k\+1\)/2​‖q^−1−q~θ0λ‖22\+C~λ2η𝖺2\(16\+8λ2\+24λ2log\(\|𝒜\|\)2\)\(1−γ\)2⋅11−\(1−η𝖼​μ~𝖼\)H/2\+2​η𝖼​σ𝖼2μ~𝖼\]\.\\displaystyle\+\\frac\{2L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)^\{2\}\}\\left\[\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\(k\+1\)/2\}\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{\-1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{0\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\+\\frac\{\\tilde\{C\}\_\{\\lambda\}^\{2\}\\eta\_\{\\mathsf\{a\}\}^\{2\}\(16\+8\\lambda^\{2\}\+24\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\)\}\{\(1\-\\gamma\)^\{2\}\}\\cdot\\frac\{1\}\{1\-\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H/2\}\}\+\\\!\\frac\{2\\eta\_\{\\mathsf\{c\}\}\\sigma\_\{\\mathsf\{c\}\}^\{2\}\}\{\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\\right\]\\kern 5\.0pt\.Next, applying Jensen’s inequality, combined with[Lemma14](https://arxiv.org/html/2605.24357#Thmlemma14), yields

𝔼​\[J~λ⋆−J~λ​\(θk\+1\)\]≤𝔼​\[J~λ⋆−J~λ​\(θk\)\]−\(η𝖺2−2​L​η𝖺2\(1−γ\)​ρmin​τλ\)​𝔼​\[‖∇J~λ​\(θk\)‖22\]\+2​η𝖺\(1−γ\)2​\(1−η𝖼​μ~𝖼\)H​B\\displaystyle\\mathbb\{E\}\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\+1\}\)\\right\]\\leq\\mathbb\{E\}\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\]\-\\left\(\\frac\{\\eta\_\{\\mathsf\{a\}\}\}\{2\}\-\\frac\{2L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\\right\)\\mathbb\{E\}\\left\[\\left\\\|\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\\right\]\+\\frac\{2\\eta\_\{\\mathsf\{a\}\}\}\{\(1\-\\gamma\)^\{2\}\}\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}B\+2​L​η𝖺2\(1−γ\)2​\[\(1−η𝖼​μ~𝖼\)H​\(k\+1\)/2​‖q^−1−q~θ0λ‖22\+C~λ2η𝖺2\(16\+8λ2\+24λ2log\(\|𝒜\|\)2\)\(1−γ\)2⋅11−\(1−η𝖼​μ~𝖼\)H/2\+2​η𝖼​σ𝖼2μ~𝖼\]\.\\displaystyle\+\\frac\{2L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)^\{2\}\}\\left\[\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\(k\+1\)/2\}\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{\-1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{0\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\+\\frac\{\\tilde\{C\}\_\{\\lambda\}^\{2\}\\eta\_\{\\mathsf\{a\}\}^\{2\}\(16\+8\\lambda^\{2\}\+24\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\)\}\{\(1\-\\gamma\)^\{2\}\}\\cdot\\frac\{1\}\{1\-\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H/2\}\}\+\\\!\\frac\{2\\eta\_\{\\mathsf\{c\}\}\\sigma\_\{\\mathsf\{c\}\}^\{2\}\}\{\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\\right\]\\kern 5\.0pt\.Using the bound1/\(1−\(1−η𝖼​μ~𝖼\)H/2\)≤21/\(1\-\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H/2\}\)\\leq 2, established in[Corollary5](https://arxiv.org/html/2605.24357#Thmcorollary5), we obtain

𝔼​\[J~λ⋆−J~λ​\(θk\+1\)\]≤𝔼​\[J~λ⋆−J~λ​\(θk\)\]−\(η𝖺2−2​L​η𝖺2\(1−γ\)​ρmin​τλ\)​𝔼​\[‖∇J~λ​\(θk\)‖22\]\+2​η𝖺\(1−γ\)2​\(1−η𝖼​μ~𝖼\)H​B\\displaystyle\\mathbb\{E\}\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\+1\}\)\\right\]\\leq\\mathbb\{E\}\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\]\-\\left\(\\frac\{\\eta\_\{\\mathsf\{a\}\}\}\{2\}\-\\frac\{2L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\\right\)\\mathbb\{E\}\\left\[\\left\\\|\\nabla\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\\right\]\+\\frac\{2\\eta\_\{\\mathsf\{a\}\}\}\{\(1\-\\gamma\)^\{2\}\}\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}B\+2​L​η𝖺2\(1−γ\)2​\[\(1−η𝖼​μ~𝖼\)H​k/2​‖q^−1−q~θ0λ‖22\+C~λ2η𝖺2\(32\+16λ2\+48λ2log\(\|𝒜\|\)2\)\(1−γ\)2\+2​η𝖼​σ𝖼2μ~𝖼\]\.\\displaystyle\+\\frac\{2L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)^\{2\}\}\\left\[\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{Hk/2\}\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{\-1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{0\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\+\\frac\{\\tilde\{C\}\_\{\\lambda\}^\{2\}\\eta\_\{\\mathsf\{a\}\}^\{2\}\(32\+16\\lambda^\{2\}\+48\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\)\}\{\(1\-\\gamma\)^\{2\}\}\+\\\!\\frac\{2\\eta\_\{\\mathsf\{c\}\}\\sigma\_\{\\mathsf\{c\}\}^\{2\}\}\{\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\\right\]\\kern 5\.0pt\.Next, using[Lemma3](https://arxiv.org/html/2605.24357#Thmlemma3)combined with \([16](https://arxiv.org/html/2605.24357#S4.E16)\), and the fact thatη𝖺≤\(1−γ\)​ρmin​τλ/\(8​L\)\\eta\_\{\\mathsf\{a\}\}\\leq\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}/\(8L\)gives

𝔼​\[J~λ⋆−J~λ​\(θk\+1\)\]≤\(1−η𝖺​μ¯~λ8\)​𝔼​\[J~λ⋆−J~λ​\(θk\)\]\+2​η𝖺\(1−γ\)2​\(1−η𝖼​μ~𝖼\)H​B\\displaystyle\\mathbb\{E\}\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\+1\}\)\\right\]\\leq\(1\-\\frac\{\\eta\_\{\\mathsf\{a\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\}\{8\}\)\\mathbb\{E\}\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{k\}\)\\right\]\+\\frac\{2\\eta\_\{\\mathsf\{a\}\}\}\{\(1\-\\gamma\)^\{2\}\}\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}B\+2​L​η𝖺2\(1−γ\)2​\[\(1−η𝖼​μ~𝖼\)H​\(k\+1\)/2​‖q^−1−q~θ0λ‖22\+C~λ2η𝖺2\(32\+16λ2\+48λ2log\(\|𝒜\|\)2\)\(1−γ\)2\+2​η𝖼​σ𝖼2μ~𝖼\]\.\\displaystyle\+\\frac\{2L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)^\{2\}\}\\left\[\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\(k\+1\)/2\}\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{\-1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{0\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\+\\frac\{\\tilde\{C\}\_\{\\lambda\}^\{2\}\\eta\_\{\\mathsf\{a\}\}^\{2\}\(32\+16\\lambda^\{2\}\+48\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\)\}\{\(1\-\\gamma\)^\{2\}\}\+\\\!\\frac\{2\\eta\_\{\\mathsf\{c\}\}\\sigma\_\{\\mathsf\{c\}\}^\{2\}\}\{\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\\right\]\\kern 5\.0pt\.Finally, unrolling the recursion concludes the proof ∎

Finally, we derive the sample complexity of[Ent\-AC](https://arxiv.org/html/2605.24357#alg1)to find anϵ\\epsilon\-precision of the entropy\-regularized problem\.

###### Corollary 6\.

Assume[Aρ\\textbf\{A\}\_\{\\rho\}](https://arxiv.org/html/2605.24357#ThmassumMDPx1)\. Letϵ\>0\\epsilon\>0, and set

η𝖼=\(1−γ\)2​ρmin​τλ40\\displaystyle\\eta\_\{\\mathsf\{c\}\}=\\frac\{\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\{40\}and

η𝖺=min⁡\(\(1−γ\)​ρmin​τλ8​L,\[μ¯~λ​\(1−γ\)4​ϵ1280C~λ2L\(1\+λ2log\(\|𝒜\|\)2\)\]1/3,μ~𝖼​μ¯~λ​ϵ4​L​σ𝖼2​ρmin​τλ\)\.\\displaystyle\\eta\_\{\\mathsf\{a\}\}=\\min\\Biggl\(\\frac\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\{8L\},\\left\[\\frac\{\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\(1\-\\gamma\)^\{4\}\\epsilon\}\{1280\\tilde\{C\}\_\{\\lambda\}^\{2\}L\\bigl\(1\+\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\\bigr\)\}\\right\]^\{1/3\},\\frac\{\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\\epsilon\}\{4L\\sigma\_\{\\mathsf\{c\}\}^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\\Biggr\)\\kern 5\.0pt\.If, in addition,

H≥40\(1−γ\)2​ρmin​τλ​μ~𝖼​max⁡\{2​log⁡\(2\+4​C~λ2​\(\(1−γ\)​ρmin​τλ8​L\)2\),log⁡\(80​Bμ¯~λ​\(1−γ\)2​ϵ\)\},\\displaystyle H\\geq\\frac\{40\}\{\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}\\,\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\\max\\Biggl\\\{2\\log\\\!\\Biggl\(2\+4\\tilde\{C\}\_\{\\lambda\}^\{2\}\\Bigl\(\\tfrac\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\{8L\}\\Bigr\)^\{2\}\\Biggr\),\\log\\\!\\left\(\\frac\{80B\}\{\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\(1\-\\gamma\)^\{2\}\\epsilon\}\\right\)\\Biggr\\\},and

K\\displaystyle K≥16μ¯~λ​max⁡\(8​L\(1−γ\)​ρmin​τλ,\[1280C~λ2L\(1\+λ2log\(\|𝒜\|\)2\)μ¯~λ​\(1−γ\)4​ϵ\]1/3,4​L​σ𝖼2​ρmin​τλμ~𝖼​μ¯~λ​ϵ\)\\displaystyle\\geq\\frac\{16\}\{\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\}\\max\\Biggl\(\\frac\{8L\}\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\},\\left\[\\frac\{1280\\tilde\{C\}\_\{\\lambda\}^\{2\}L\\bigl\(1\+\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\\bigr\)\}\{\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\(1\-\\gamma\)^\{4\}\\epsilon\}\\right\]^\{1/3\},\\frac\{4L\\sigma\_\{\\mathsf\{c\}\}^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\{\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\\epsilon\}\\Biggr\)×max⁡\{log⁡\(5​\(J~λ⋆−J~λ​\(θ0\)\)ϵ\),log⁡\(20​ρmin​τλ​‖q^−1−q~θ0λ‖22e​\(1−γ\)​μ¯~λ​ϵ\)\},\\displaystyle\\hskip 50\.00008pt\\times\\max\\Biggl\\\{\\log\\\!\\left\(\\frac\{5\(\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{0\}\)\)\}\{\\epsilon\}\\right\),\\ \\log\\\!\\left\(\\frac\{20\\rho\_\{\\min\}\\tau\_\{\\lambda\}\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{\-1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{0\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\}\{e\(1\-\\gamma\)\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\\epsilon\}\\right\)\\Biggr\\\},then[Ent\-AC](https://arxiv.org/html/2605.24357#alg1)satisfies

𝔼​\[J~λ⋆−J~λ​\(θK\)\]≤ϵ\.\\displaystyle\\mathbb\{E\}\\\!\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{K\}\)\\right\]\\leq\\epsilon\\kern 5\.0pt\.

###### Proof\.

Let

a​=Δ​1−η𝖺​μ¯~λ8,b​=Δ​\(1−η𝖼​μ~𝖼\)H/2,Δ0​=Δ​J~λ⋆−J~λ​\(θ0\),E0​=Δ​‖q^−1−q~θ0λ‖22\.\\displaystyle a\\overset\{\\Delta\}\{=\}1\-\\frac\{\\eta\_\{\\mathsf\{a\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\}\{8\},\\qquad b\\overset\{\\Delta\}\{=\}\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H/2\},\\qquad\\Delta\_\{0\}\\overset\{\\Delta\}\{=\}\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{0\}\),\\qquad E\_\{0\}\\overset\{\\Delta\}\{=\}\\left\\\|\\hat\{\\mathrm\{q\}\}\_\{\-1\}\-\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\{\\theta\_\{0\}\}\}^\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\\kern 5\.0pt\.By[Theorem4](https://arxiv.org/html/2605.24357#Thmtheorem4), combined with[Corollary5](https://arxiv.org/html/2605.24357#Thmcorollary5), it holds that

𝔼​\[J~λ⋆−J~λ​\(θK\)\]\\displaystyle\\mathbb\{E\}\\\!\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{K\}\)\\right\]≤aKΔ0\+2​L​η𝖺2\(1−γ\)2Kmax\(a,b\)KE0\\displaystyle\\leq a^\{K\}\\Delta\_\{0\}\+\\frac\{2L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)^\{2\}\}K\\max\(a,b\)^\{K\}E\_\{0\}\+16​\(1−η𝖼​μ~𝖼\)Hμ¯~λ​\(1−γ\)2​B\+η𝖺3​256C~λ2L\(1\+λ2log\(\|𝒜\|\)2\)μ¯~λ​\(1−γ\)4\+32​L​η𝖺​η𝖼​σ𝖼2\(1−γ\)2​μ~𝖼​μ¯~λ\.\\displaystyle\\quad\+\\frac\{16\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}\}\{\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\(1\-\\gamma\)^\{2\}\}B\+\\eta\_\{\\mathsf\{a\}\}^\{3\}\\frac\{256\\tilde\{C\}\_\{\\lambda\}^\{2\}L\\bigl\(1\+\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\\bigr\)\}\{\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\(1\-\\gamma\)^\{4\}\}\+\\frac\{32L\\eta\_\{\\mathsf\{a\}\}\\eta\_\{\\mathsf\{c\}\}\\sigma\_\{\\mathsf\{c\}\}^\{2\}\}\{\(1\-\\gamma\)^\{2\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\}\\kern 5\.0pt\.
We now show that each of the five terms on the right\-hand side is at mostϵ/5\\epsilon/5\.

#### Step 1: control of the max\-term\.

We first prove thatb≤ab\\leq a\. Recall from the previous lemmas that

μ~𝖼=\(1−γ\)2​ρmin​τλ2,μ¯~λ=λ​\(1−γ\)​ρmin2​τλ2\|𝒮\|\.\\displaystyle\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}=\\frac\{\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\{2\},\\qquad\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}=\\frac\{\\lambda\(1\-\\gamma\)\\rho\_\{\\min\}^\{2\}\\tau\_\{\\lambda\}^\{2\}\}\{\|\\mathcal\{S\}\|\}\\kern 5\.0pt\.Since

η𝖺≤\(1−γ\)​ρmin​τλ8​L,\\displaystyle\\eta\_\{\\mathsf\{a\}\}\\leq\\frac\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\{8L\},we obtain

η𝖺​μ¯~λ8\\displaystyle\\frac\{\\eta\_\{\\mathsf\{a\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\}\{8\}≤\(1−γ\)​ρmin​τλ​μ¯~λ64​L\\displaystyle\\leq\\frac\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\}\{64L\}=λ​\(1−γ\)2​ρmin3​τλ364​L​\|𝒮\|\.\\displaystyle=\\frac\{\\lambda\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}^\{3\}\\tau\_\{\\lambda\}^\{3\}\}\{64L\|\\mathcal\{S\}\|\}\\kern 5\.0pt\.Hence, by monotonicity of the mapx↦log⁡\(11−x\)x\\mapsto\\log\\\!\\bigl\(\\frac\{1\}\{1\-x\}\\bigr\)on\(0,1\)\(0,1\),

log⁡\(11−η𝖺​μ¯~λ/8\)≤log⁡\(11−\(1−γ\)​ρmin​τλ​μ¯~λ64​L\)\.\\displaystyle\\log\\\!\\left\(\\frac\{1\}\{1\-\\eta\_\{\\mathsf\{a\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}/8\}\\right\)\\leq\\log\\\!\\Biggl\(\\frac\{1\}\{1\-\\frac\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\}\{64L\}\}\\Biggr\)\\kern 5\.0pt\.Now, if

H≥2η𝖼​μ~𝖼​log⁡\(11−\(1−γ\)​ρmin​τλ​μ¯~λ64​L\),\\displaystyle H\\geq\\frac\{2\}\{\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\\log\\\!\\Biggl\(\\frac\{1\}\{1\-\\frac\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\}\{64L\}\}\\Biggr\),then

H≥2η𝖼​μ~𝖼​log⁡\(11−η𝖺​μ¯~λ/8\)\.\\displaystyle H\\geq\\frac\{2\}\{\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\\log\\\!\\left\(\\frac\{1\}\{1\-\\eta\_\{\\mathsf\{a\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}/8\}\\right\)\\kern 5\.0pt\.Since

η𝖼=\(1−γ\)2​ρmin​τλ40,μ~𝖼=\(1−γ\)2​ρmin​τλ2,\\displaystyle\\eta\_\{\\mathsf\{c\}\}=\\frac\{\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\{40\},\\qquad\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}=\\frac\{\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\{2\},we have

2η𝖼​μ~𝖼=80\(1−γ\)4​ρmin2​τλ2=40\(1−γ\)2​ρmin​τλ​μ~𝖼\.\\displaystyle\\frac\{2\}\{\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}=\\frac\{80\}\{\(1\-\\gamma\)^\{4\}\\rho\_\{\\min\}^\{2\}\\tau\_\{\\lambda\}^\{2\}\}=\\frac\{40\}\{\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}\\,\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\\kern 5\.0pt\.Therefore, the lower bound

H≥40\(1−γ\)2​ρmin​τλ​μ~𝖼​2​log⁡\(11−\(1−γ\)​ρmin​τλ​μ¯~λ64​L\)\\displaystyle H\\geq\\frac\{40\}\{\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}\\,\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\\,2\\log\\\!\\Biggl\(\\frac\{1\}\{1\-\\frac\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\}\{64L\}\}\\Biggr\)implies

H≥2η𝖼​μ~𝖼​log⁡\(11−η𝖺​μ¯~λ/8\)\.\\displaystyle H\\geq\\frac\{2\}\{\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\\log\\\!\\left\(\\frac\{1\}\{1\-\\eta\_\{\\mathsf\{a\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}/8\}\\right\)\\kern 5\.0pt\.Consequently,

η𝖼​μ~𝖼​H2≥log⁡\(11−η𝖺​μ¯~λ/8\)=−log⁡\(a\)\.\\displaystyle\\frac\{\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\\,H\}\{2\}\\geq\\log\\\!\\left\(\\frac\{1\}\{1\-\\eta\_\{\\mathsf\{a\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}/8\}\\right\)=\-\\log\(a\)\\kern 5\.0pt\.Multiplying by−1\-1and exponentiating both sides yield

exp⁡\(−η𝖼​μ~𝖼​H2\)≤a\.\\displaystyle\\exp\\\!\\left\(\-\\frac\{\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\\,H\}\{2\}\\right\)\\leq a\\kern 5\.0pt\.Using1−x≤e−x1\-x\\leq e^\{\-x\}for allx≥0x\\geq 0, we obtain

b=\(1−η𝖼​μ~𝖼\)H/2≤exp⁡\(−η𝖼​μ~𝖼​H2\)≤a\.\\displaystyle b=\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H/2\}\\leq\\exp\\\!\\left\(\-\\frac\{\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\\,H\}\{2\}\\right\)\\leq a\\kern 5\.0pt\.Hence

max⁡\(a,b\)=a\.\\displaystyle\\max\(a,b\)=a\\kern 5\.0pt\.

#### Step 2: bound on the first term\.

To ensure thataK​Δ0≤ϵ/5a^\{K\}\\Delta\_\{0\}\\leq\\epsilon/5, it is enough that

K≥1log⁡\(1/a\)​log⁡\(5​Δ0ϵ\)\.\\displaystyle K\\geq\\frac\{1\}\{\\log\(1/a\)\}\\log\\\!\\left\(\\frac\{5\\Delta\_\{0\}\}\{\\epsilon\}\\right\)\\kern 5\.0pt\.Since−log⁡\(1−x\)≥x\-\\log\(1\-x\)\\geq xfor allx∈\(0,1\)x\\in\(0,1\), we have

log⁡\(1/a\)=−log⁡\(1−η𝖺​μ¯~λ8\)≥η𝖺​μ¯~λ8\.\\displaystyle\\log\(1/a\)=\-\\log\\\!\\left\(1\-\\frac\{\\eta\_\{\\mathsf\{a\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\}\{8\}\\right\)\\geq\\frac\{\\eta\_\{\\mathsf\{a\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\}\{8\}\\kern 5\.0pt\.Thus it is sufficient that

K≥8η𝖺​μ¯~λ​log⁡\(5​Δ0ϵ\)\.\\displaystyle K\\geq\\frac\{8\}\{\\eta\_\{\\mathsf\{a\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\}\\log\\\!\\left\(\\frac\{5\\Delta\_\{0\}\}\{\\epsilon\}\\right\)\\kern 5\.0pt\.

#### Step 3: bound on the second term\.

We use the inequality

t​xt≤2e​log⁡\(1/x\)​xt/2,x∈\(0,1\),t≥0,\\displaystyle tx^\{t\}\\leq\\frac\{2\}\{e\\log\(1/x\)\}x^\{t/2\},\\qquad x\\in\(0,1\),\\ t\\geq 0,which follows by settingu=t​log⁡\(1/x\)u=t\\log\(1/x\)and usingu​e−u/2≤2/eue^\{\-u/2\}\\leq 2/efor allu≥0u\\geq 0\. Applying this withx=ax=aandt=Kt=K, we obtain

K​aK≤2e​log⁡\(1/a\)​aK/2≤16e​η𝖺​μ¯~λ​aK/2,\\displaystyle Ka^\{K\}\\leq\\frac\{2\}\{e\\log\(1/a\)\}a^\{K/2\}\\leq\\frac\{16\}\{e\\eta\_\{\\mathsf\{a\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\}a^\{K/2\},where we used againlog⁡\(1/a\)≥η𝖺​μ¯~λ/8\\log\(1/a\)\\geq\\eta\_\{\\mathsf\{a\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}/8\. Hence

2​L​η𝖺2\(1−γ\)2​K​aK​E0≤32​L​η𝖺e​\(1−γ\)2​μ¯~λ​aK/2​E0\.\\displaystyle\\frac\{2L\\eta\_\{\\mathsf\{a\}\}^\{2\}\}\{\(1\-\\gamma\)^\{2\}\}Ka^\{K\}E\_\{0\}\\leq\\frac\{32L\\eta\_\{\\mathsf\{a\}\}\}\{e\(1\-\\gamma\)^\{2\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\}a^\{K/2\}E\_\{0\}\\kern 5\.0pt\.Therefore, it is sufficient to require

K≥16η𝖺​μ¯~λ​log⁡\(160​L​η𝖺​E0e​\(1−γ\)2​μ¯~λ​ϵ\)\\displaystyle K\\geq\\frac\{16\}\{\\eta\_\{\\mathsf\{a\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\}\\log\\\!\\left\(\\frac\{160L\\eta\_\{\\mathsf\{a\}\}E\_\{0\}\}\{e\(1\-\\gamma\)^\{2\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\\epsilon\}\\right\)to make the second term at mostϵ/5\\epsilon/5\. Now, since

η𝖺≤\(1−γ\)​ρmin​τλ8​L,\\displaystyle\\eta\_\{\\mathsf\{a\}\}\\leq\\frac\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\{8L\},we have

160​L​η𝖺​E0e​\(1−γ\)2​μ¯~λ​ϵ≤20​ρmin​τλ​E0e​\(1−γ\)​μ¯~λ​ϵ\.\\displaystyle\\frac\{160L\\eta\_\{\\mathsf\{a\}\}E\_\{0\}\}\{e\(1\-\\gamma\)^\{2\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\\epsilon\}\\leq\\frac\{20\\rho\_\{\\min\}\\tau\_\{\\lambda\}E\_\{0\}\}\{e\(1\-\\gamma\)\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\\epsilon\}\\kern 5\.0pt\.Hence a sufficient condition is

K≥16η𝖺​μ¯~λ​log⁡\(20​ρmin​τλ​E0e​\(1−γ\)​μ¯~λ​ϵ\)\.\\displaystyle K\\geq\\frac\{16\}\{\\eta\_\{\\mathsf\{a\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\}\\log\\\!\\left\(\\frac\{20\\rho\_\{\\min\}\\tau\_\{\\lambda\}E\_\{0\}\}\{e\(1\-\\gamma\)\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\\epsilon\}\\right\)\\kern 5\.0pt\.

#### Step 4: bound on the critic\-bias term\.

To ensure that

16​\(1−η𝖼​μ~𝖼\)Hμ¯~λ​\(1−γ\)2​B≤ϵ5,\\displaystyle\\frac\{16\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}\}\{\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\(1\-\\gamma\)^\{2\}\}B\\leq\\frac\{\\epsilon\}\{5\},it is enough that

\(1−η𝖼​μ~𝖼\)H≤μ¯~λ​\(1−γ\)2​ϵ80​B\.\\displaystyle\(1\-\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\)^\{H\}\\leq\\frac\{\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\(1\-\\gamma\)^\{2\}\\epsilon\}\{80B\}\\kern 5\.0pt\.Using\(1−x\)m≤e−x​m\(1\-x\)^\{m\}\\leq e^\{\-xm\}, a sufficient condition is

H≥1η𝖼​μ~𝖼​log⁡\(80​Bμ¯~λ​\(1−γ\)2​ϵ\)=40\(1−γ\)2​ρmin​τλ​μ~𝖼​log⁡\(80​Bμ¯~λ​\(1−γ\)2​ϵ\)\.\\displaystyle H\\geq\\frac\{1\}\{\\eta\_\{\\mathsf\{c\}\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\\log\\\!\\left\(\\frac\{80B\}\{\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\(1\-\\gamma\)^\{2\}\\epsilon\}\\right\)=\\frac\{40\}\{\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}\\,\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\\log\\\!\\left\(\\frac\{80B\}\{\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\(1\-\\gamma\)^\{2\}\\epsilon\}\\right\)\\kern 5\.0pt\.

#### Step 5: bound on the policy\-switch term\.

To ensure that

η𝖺3​256C~λ2L\(1\+λ2log\(\|𝒜\|\)2\)μ¯~λ​\(1−γ\)4≤ϵ5,\\displaystyle\\eta\_\{\\mathsf\{a\}\}^\{3\}\\frac\{256\\tilde\{C\}\_\{\\lambda\}^\{2\}L\\bigl\(1\+\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\\bigr\)\}\{\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\(1\-\\gamma\)^\{4\}\}\\leq\\frac\{\\epsilon\}\{5\},it is sufficient that

η𝖺≤\[μ¯~λ​\(1−γ\)4​ϵ1280C~λ2L\(1\+λ2log\(\|𝒜\|\)2\)\]1/3\.\\displaystyle\\eta\_\{\\mathsf\{a\}\}\\leq\\left\[\\frac\{\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\(1\-\\gamma\)^\{4\}\\epsilon\}\{1280\\tilde\{C\}\_\{\\lambda\}^\{2\}L\\bigl\(1\+\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\\bigr\)\}\\right\]^\{1/3\}\\kern 5\.0pt\.

#### Step 6: bound on the variance term\.

To ensure that

32​L​η𝖺​η𝖼​σ𝖼2\(1−γ\)2​μ~𝖼​μ¯~λ≤ϵ5,\\displaystyle\\frac\{32L\\eta\_\{\\mathsf\{a\}\}\\eta\_\{\\mathsf\{c\}\}\\sigma\_\{\\mathsf\{c\}\}^\{2\}\}\{\(1\-\\gamma\)^\{2\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\}\\leq\\frac\{\\epsilon\}\{5\},it is sufficient that

η𝖺≤\(1−γ\)2​μ~𝖼​μ¯~λ​ϵ160​L​σ𝖼2​η𝖼=μ~𝖼​μ¯~λ​ϵ4​L​σ𝖼2​ρmin​τλ\.\\displaystyle\\eta\_\{\\mathsf\{a\}\}\\leq\\frac\{\(1\-\\gamma\)^\{2\}\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\\epsilon\}\{160L\\sigma\_\{\\mathsf\{c\}\}^\{2\}\\eta\_\{\\mathsf\{c\}\}\}=\\frac\{\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\\epsilon\}\{4L\\sigma\_\{\\mathsf\{c\}\}^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\\kern 5\.0pt\.

#### Conclusion\.

Since

η𝖺=min⁡\(\(1−γ\)​ρmin​τλ8​L,\[μ¯~λ​\(1−γ\)4​ϵ1280C~λ2L\(1\+λ2log\(\|𝒜\|\)2\)\]1/3,μ~𝖼​μ¯~λ​ϵ4​L​σ𝖼2​ρmin​τλ\),\\displaystyle\\eta\_\{\\mathsf\{a\}\}=\\min\\Biggl\(\\frac\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\{8L\},\\left\[\\frac\{\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\(1\-\\gamma\)^\{4\}\\epsilon\}\{1280\\tilde\{C\}\_\{\\lambda\}^\{2\}L\\bigl\(1\+\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\\bigr\)\}\\right\]^\{1/3\},\\frac\{\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\\epsilon\}\{4L\\sigma\_\{\\mathsf\{c\}\}^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\\Biggr\),we have

1η𝖺=max⁡\(8​L\(1−γ\)​ρmin​τλ,\[1280C~λ2L\(1\+λ2log\(\|𝒜\|\)2\)μ¯~λ​\(1−γ\)4​ϵ\]1/3,4​L​σ𝖼2​ρmin​τλμ~𝖼​μ¯~λ​ϵ\)\.\\displaystyle\\frac\{1\}\{\\eta\_\{\\mathsf\{a\}\}\}=\\max\\Biggl\(\\frac\{8L\}\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\},\\left\[\\frac\{1280\\tilde\{C\}\_\{\\lambda\}^\{2\}L\\bigl\(1\+\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\\bigr\)\}\{\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\(1\-\\gamma\)^\{4\}\\epsilon\}\\right\]^\{1/3\},\\frac\{4L\\sigma\_\{\\mathsf\{c\}\}^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\{\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\\epsilon\}\\Biggr\)\\kern 5\.0pt\.Therefore, the condition

K≥16μ¯~λ​max⁡\(8​L\(1−γ\)​ρmin​τλ,\[1280C~λ2L\(1\+λ2log\(\|𝒜\|\)2\)μ¯~λ​\(1−γ\)4​ϵ\]1/3,4​L​σ𝖼2​ρmin​τλμ~𝖼​μ¯~λ​ϵ\)​max⁡\{log⁡\(5​Δ0ϵ\),log⁡\(20​ρmin​τλ​E0e​\(1−γ\)​μ¯~λ​ϵ\)\}\\displaystyle K\\geq\\frac\{16\}\{\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\}\\max\\Biggl\(\\frac\{8L\}\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\},\\left\[\\frac\{1280\\tilde\{C\}\_\{\\lambda\}^\{2\}L\\bigl\(1\+\\lambda^\{2\}\\log\(\|\\mathcal\{A\}\|\)^\{2\}\\bigr\)\}\{\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\(1\-\\gamma\)^\{4\}\\epsilon\}\\right\]^\{1/3\},\\frac\{4L\\sigma\_\{\\mathsf\{c\}\}^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\{\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\\epsilon\}\\Biggr\)\\max\\Biggl\\\{\\log\\\!\\left\(\\frac\{5\\Delta\_\{0\}\}\{\\epsilon\}\\right\),\\ \\log\\\!\\left\(\\frac\{20\\rho\_\{\\min\}\\tau\_\{\\lambda\}E\_\{0\}\}\{e\(1\-\\gamma\)\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\\epsilon\}\\right\)\\Biggr\\\}implies both bounds obtained in Steps 2 and 3\.

Similarly, since

η𝖺≤\(1−γ\)​ρmin​τλ8​L,\\displaystyle\\eta\_\{\\mathsf\{a\}\}\\leq\\frac\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\{8L\},we have

2​log⁡\(2\+4​C~λ2​η𝖺2\)\\displaystyle 2\\log\\\!\\Bigl\(2\+4\\tilde\{C\}\_\{\\lambda\}^\{2\}\\eta\_\{\\mathsf\{a\}\}^\{2\}\\Bigr\)≤2​log⁡\(2\+4​C~λ2​\(\(1−γ\)​ρmin​τλ8​L\)2\),\\displaystyle\\leq 2\\log\\\!\\Biggl\(2\+4\\tilde\{C\}\_\{\\lambda\}^\{2\}\\Bigl\(\\tfrac\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\{8L\}\\Bigr\)^\{2\}\\Biggr\),2​log⁡\(11−η𝖺​μ¯~λ/8\)\\displaystyle 2\\log\\\!\\left\(\\frac\{1\}\{1\-\\eta\_\{\\mathsf\{a\}\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}/8\}\\right\)≤2​log⁡\(11−\(1−γ\)​ρmin​τλ​μ¯~λ64​L\)\.\\displaystyle\\leq 2\\log\\\!\\Biggl\(\\frac\{1\}\{1\-\\frac\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\}\{64L\}\}\\Biggr\)\.Therefore, the lower bound

H≥40\(1−γ\)2​ρmin​τλ​μ~𝖼​max⁡\{2​log⁡\(2\+4​C~λ2​\(\(1−γ\)​ρmin​τλ8​L\)2\),2​log⁡\(11−\(1−γ\)​ρmin​τλ​μ¯~λ64​L\),log⁡\(80​Bμ¯~λ​\(1−γ\)2​ϵ\)\}\\displaystyle H\\geq\\frac\{40\}\{\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\\tau\_\{\\lambda\}\\,\\tilde\{\\mu\}\_\{\\mathsf\{c\}\}\}\\max\\Biggl\\\{2\\log\\\!\\Biggl\(2\+4\\tilde\{C\}\_\{\\lambda\}^\{2\}\\Bigl\(\\tfrac\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\}\{8L\}\\Bigr\)^\{2\}\\Biggr\),\\ 2\\log\\\!\\Biggl\(\\frac\{1\}\{1\-\\frac\{\(1\-\\gamma\)\\rho\_\{\\min\}\\tau\_\{\\lambda\}\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\}\{64L\}\}\\Biggr\),\\ \\log\\\!\\left\(\\frac\{80B\}\{\\underline\{\\tilde\{\\mu\}\}\_\{\\lambda\}\(1\-\\gamma\)^\{2\}\\epsilon\}\\right\)\\Biggr\\\}simultaneously guarantees the condition of[Corollary5](https://arxiv.org/html/2605.24357#Thmcorollary5), the dominationb≤ab\\leq a, and the bound on the critic\-bias term\. Observing that the first term of the maximum dominates the second allows to simplify the bound and to obtain the desired condition onHH\. Under all these conditions, each of the five terms above is at mostϵ/5\\epsilon/5, and therefore

𝔼​\[J~λ⋆−J~λ​\(θK\)\]≤ϵ\.\\displaystyle\\mathbb\{E\}\\\!\\left\[\\tilde\{J\}^\{\\star\}\_\{\\lambda\}\-\\tilde\{J\}\_\{\\lambda\}\(\\theta\_\{K\}\)\\right\]\\leq\\epsilon\\kern 5\.0pt\.This concludes the proof\. ∎

## Appendix FMontone Improvement operator

The goal of this section is therefore to show the existence of an operator𝒰τ:π→𝒰τ​\(π\)\\mathcal\{U\}\_\{\\tau\}\\colon\\pi\\to\\mathcal\{U\}\_\{\\tau\}\(\\pi\)with three crucial properties: \(i\) for any policy, applying this operator produces a new policy with a higher regularized value than the one achieved byπ\\pi; \(ii\) every policy generated by this operator assigns at least a fixed minimum probability to every action; \(iii\) th\. The main idea is to build the improvement operator such that it slightly augments the smallest probability weights, such that for any state action pair\(s,a\)∈𝒮×𝒜\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}the probabilityπ​\(a\|s\)\\pi\(a\|s\)stays above a certain threshold\. We will show below that this procedure improves the global objective while keeping the probabilities uniformly bounded away from0when the threshold is properly chosen\. For any policyπ\\pi, states∈𝒮s\\in\\mathcal\{S\},τ<1/\(2​\|𝒜\|2\)\\tau<1/\(2\|\\mathcal\{A\}\|^\{2\}\), we respectively define𝒜τπ​\(s\)\\mathcal\{A\}\_\{\\tau\}^\{\\pi\}\(s\), andamaxπ​\(s\)a\_\{\\max\}^\{\\pi\}\(s\)as

𝒜τπ​\(s\)​=Δ​\{a∈𝒜,π​\(a\|s\)≤τ\},amaxπ​\(s\)=arg​maxa∈𝒜⁡\{π​\(a\|s\)\},\\displaystyle\\mathcal\{A\}\_\{\\tau\}^\{\\pi\}\(s\)\\overset\{\\Delta\}\{=\}\\left\\\{a\\in\\mathcal\{A\},\\pi\(a\|s\)\\leq\\tau\\right\\\}\\kern 5\.0pt,\\quad a\_\{\\max\}^\{\\pi\}\(s\)=\\operatorname\*\{arg\\,max\}\_\{a\\in\\mathcal\{A\}\}\\\{\\pi\(a\|s\)\\\}\\kern 5\.0pt,where thearg​max\\operatorname\*\{arg\\,max\}is chosen at random in the case of ties\. Note that the definition ofτλ\\tau\_\{\\lambda\}ensures thatamaxπ​\(s\)a\_\{\\max\}^\{\\pi\}\(s\)does not belong to the set𝒜τπ​\(s\)\\mathcal\{A\}\_\{\\tau\}^\{\\pi\}\(s\)as

maxa∈𝒜⁡π​\(a\|s\)≥1/\|𝒜\|\.\\displaystyle\\max\_\{a\\in\\mathcal\{A\}\}\\pi\(a\|s\)\\geq 1/\|\\mathcal\{A\}\|\\kern 5\.0pt\.Finally, we define the improvement operator as follows:

𝒰τ:𝒫​\(𝒜\)𝒮\\displaystyle\\mathcal\{U\}\_\{\\tau\}:\\mathcal\{P\}\(\\mathcal\{A\}\)^\{\\mathcal\{S\}\}⟶𝒫​\(𝒜\)𝒮,\\displaystyle\\;\\longrightarrow\\;\\mathcal\{P\}\(\\mathcal\{A\}\)^\{\\mathcal\{S\}\},\(33\)π\\displaystyle\\pi⟼𝒰τ​\(π\),\\displaystyle\\;\\longmapsto\\;\\mathcal\{U\}\_\{\\tau\}\(\\pi\),where for every\(s,a\)∈𝒮×𝒜\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\},

𝒰τ​\(π\)​\(a\|s\)=\{τ,if​π​\(a\|s\)≤τ,π​\(a\|s\)−∑b∈𝒜τπ​\(s\)\(τ−π​\(b\|s\)\),if​a=amaxπ​\(s\),π​\(a\|s\),otherwise\.\\mathcal\{U\}\_\{\\tau\}\(\\pi\)\(a\|s\)\\;=\\;\\begin\{cases\}\\tau,&\\text\{if \}\\pi\(a\|s\)\\leq\\tau,\\\\\[6\.0pt\] \\pi\(a\|s\)\-\\displaystyle\\sum\_\{b\\in\\mathcal\{A\}\_\{\\tau\}^\{\\pi\}\(s\)\}\\bigl\(\\tau\-\\pi\(b\|s\)\\bigr\),&\\text\{if \}a=a\_\{\\max\}^\{\\pi\}\(s\),\\\\\[10\.0pt\] \\pi\(a\|s\),&\\text\{otherwise\}\.\\end\{cases\}The operator𝒰τ\\mathcal\{U\}\_\{\\tau\}builds𝒰τ​\(π\)​\(a\|s\)\\mathcal\{U\}\_\{\\tau\}\(\\pi\)\(a\|s\)by \(statewise\) raising eacha∈𝒜τπ​\(s\)a\\in\\mathcal\{A\}\_\{\\tau\}^\{\\pi\}\(s\)toτ\\tau, substracting the total added mass from the single actionamaxπ​\(s\)a\_\{\\max\}^\{\\pi\}\(s\), and leaving other actions unchanged\. If𝒜τπ​\(s\)=∅\\mathcal\{A\}\_\{\\tau\}^\{\\pi\}\(s\)=\\emptyset, for alls∈𝒮s\\in\\mathcal\{S\}, then𝒰τ​\(π\)=π\\mathcal\{U\}\_\{\\tau\}\(\\pi\)=\\pi\. Note that mass conservation is immediate from the definition and the fact thatτ<1/\(2​\|𝒜\|2\)\\tau<1/\(2\|\\mathcal\{A\}\|^\{2\}\)\. Non\-negativity of𝒰τ​\(π\)​\(amaxπ​\(s\)\|s\)\\mathcal\{U\}\_\{\\tau\}\(\\pi\)\(a\_\{\\max\}^\{\\pi\}\(s\)\|s\)follows because the removed mass is

∑a∈𝒜τπ​\(s\)\{τ−π​\(a\|s\)\}≤τ×\|𝒜\|≤12​\|𝒜\|\\sum\_\{a\\in\\mathcal\{A\}\_\{\\tau\}^\{\\pi\}\(s\)\}\\\{\\tau\-\\pi\(a\|s\)\\\}\\leq\\tau\\times\|\\mathcal\{A\}\|\\leq\\frac\{1\}\{2\|\\mathcal\{A\}\|\}Sinceπ​\(amaxπ​\(s\)\|s\)≥1/\|𝒜\|\\pi\(a\_\{\\max\}^\{\\pi\}\(s\)\|s\)\\geq 1/\|\\mathcal\{A\}\|, we get that𝒰​\(π\)​\(amaxπ​\(s\)\|s\)≥1/\(2​\|𝒜\|\)\\mathcal\{U\}\(\\pi\)\(a\_\{\\max\}^\{\\pi\}\(s\)\|s\)\\geq 1/\(2\|\\mathcal\{A\}\|\)\. This in particular shows that𝒰τ​\(π\)\\mathcal\{U\}\_\{\\tau\}\(\\pi\)is a policy\. Next, define

τλ​=Δ​min⁡\(13​exp⁡\(−16\+8​γ​λ​log⁡\(\|𝒜\|\)λ​\(1−γ\)2​ρmin\),138​\|𝒜\|4\)\.\\displaystyle\\tau\_\{\\lambda\}\\overset\{\\Delta\}\{=\}\\min\\left\(\\frac\{1\}\{3\}\\exp\\left\(\-\\frac\{16\+8\\gamma\\lambda\\log\(\|\\mathcal\{A\}\|\)\}\{\\lambda\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\}\\right\),\\frac\{1\}\{3^\{8\}\|\\mathcal\{A\}\|^\{4\}\}\\right\)\\kern 5\.0pt\.\(34\)The following lemma establishes the crucial improvement property whenτ=τλ\\tau=\\tau\_\{\\lambda\}\.

###### Lemma 20\.

Assume that the initial distributionρ\\rhosatisfies[Aρ\\textbf\{A\}\_\{\\rho\}](https://arxiv.org/html/2605.24357#ThmassumMDPx1)\. For any policyπ\\pi, it holds that

v~𝒰τλ​\(π\)λ​\(ρ\)≥v~πλ​\(ρ\)\.\\displaystyle\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\mathcal\{U\}\_\{\\tau\_\{\\lambda\}\}\(\\pi\)\}^\{\\lambda\}\(\\rho\)\\geq\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\pi\}^\{\\lambda\}\(\\rho\)\\kern 5\.0pt\.Additionally, for any policyπ\\pi, we have that

𝒰τλ​\(π\)​\(a\|s\)≥τλ\.\\displaystyle\\mathcal\{U\}\_\{\\tau\_\{\\lambda\}\}\(\\pi\)\(a\|s\)\\geq\\tau\_\{\\lambda\}\\kern 5\.0pt\.

###### Proof\.

Set an arbitrary policyπ\\pi\. For avoiding heavy notations, we will, through this proof, denote by𝒜τπ=𝒜τλπ\\mathcal\{A\}\_\{\\tau\}^\{\\pi\}=\\mathcal\{A\}\_\{\\tau\_\{\\lambda\}\}^\{\\pi\}\. We consider the case where there iss∈𝒮s\\in\\mathcal\{S\}such that𝒜τπ​\(s\)≠∅\\mathcal\{A\}\_\{\\tau\}^\{\\pi\}\(s\)\\neq\\emptyset\(alternatively𝒰τλ​\(π\)=π\\mathcal\{U\}\_\{\\tau\_\{\\lambda\}\}\(\\pi\)=\\pi, which makes the previous inequality immediately valid\)\. Defineπ~=𝒰τλ​\(π\)\\tilde\{\\pi\}=\\mathcal\{U\}\_\{\\tau\_\{\\lambda\}\}\(\\pi\)\. The following applies

v~π~λ​\(ρ\)−v~πλ​\(ρ\)\\displaystyle\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\tilde\{\\pi\}\}^\{\\lambda\}\(\\rho\)\-\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\pi\}^\{\\lambda\}\(\\rho\)=∑s∈𝒮dρπ~​\(s\)​∑a∈𝒜\[π~​\(a\|s\)​𝗋​\(s,a\)−λ​π~​\(a\|s\)​log⁡\(π~​\(a\|s\)\)\]\\displaystyle=\\sum\_\{s\\in\\mathcal\{S\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\tilde\{\\pi\}\}\(s\)\\sum\_\{a\\in\\mathcal\{A\}\}\\left\[\\tilde\{\\pi\}\(a\|s\)\\mathsf\{r\}\(s,a\)\-\\lambda\\tilde\{\\pi\}\(a\|s\)\\log\\left\(\\tilde\{\\pi\}\(a\|s\)\\right\)\\right\]−∑s∈𝒮dρπ​\(s\)​∑a∈𝒜\[π​\(a\|s\)​𝗋​\(s,a\)−λ​π​\(a\|s\)​log⁡\(π​\(a\|s\)\)\]\\displaystyle\\hskip 72\.0pt\-\\sum\_\{s\\in\\mathcal\{S\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\pi\}\(s\)\\sum\_\{a\\in\\mathcal\{A\}\}\\left\[\\pi\(a\|s\)\\mathsf\{r\}\(s,a\)\-\\lambda\\pi\(a\|s\)\\log\\left\(\\pi\(a\|s\)\\right\)\\right\]=∑s∈𝒮\(dρπ~​\(s\)−dρπ​\(s\)\)​∑a∈𝒜\[π~​\(a\|s\)​𝗋​\(s,a\)−λ​π~​\(a\|s\)​log⁡\(π~​\(a\|s\)\)\]⏟\(𝐈\)\\displaystyle=\\underbrace\{\\sum\_\{s\\in\\mathcal\{S\}\}\\left\(d\_\{\\rho\}^\{\\hskip 0\.57367pt\\tilde\{\\pi\}\}\(s\)\-d\_\{\\rho\}^\{\\hskip 0\.57367pt\\pi\}\(s\)\\right\)\\sum\_\{a\\in\\mathcal\{A\}\}\\left\[\\tilde\{\\pi\}\(a\|s\)\\mathsf\{r\}\(s,a\)\-\\lambda\\tilde\{\\pi\}\(a\|s\)\\log\\left\(\\tilde\{\\pi\}\(a\|s\)\\right\)\\right\]\}\_\{\\mathbf\{\(I\)\}\}\+∑s∈𝒮dρπ​\(s\)​∑a∈𝒜\(π~​\(a\|s\)−π​\(a\|s\)\)​𝗋​\(s,a\)⏟\(𝐈𝐈\)\\displaystyle\+\\underbrace\{\\sum\_\{s\\in\\mathcal\{S\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\pi\}\(s\)\\sum\_\{a\\in\\mathcal\{A\}\}\(\\tilde\{\\pi\}\(a\|s\)\-\\pi\(a\|s\)\)\\mathsf\{r\}\(s,a\)\}\_\{\\mathbf\{\(II\)\}\}\+λ​∑s∈𝒮dρπ​\(s\)​∑a∈𝒜\[π​\(a\|s\)​log⁡\(π​\(a\|s\)\)−π~​\(a\|s\)​log⁡\(π~​\(a\|s\)\)\]⏟\(𝐈𝐈𝐈\)\.\\displaystyle\+\\underbrace\{\\lambda\\sum\_\{s\\in\\mathcal\{S\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\pi\}\(s\)\\sum\_\{a\\in\\mathcal\{A\}\}\\left\[\\pi\(a\|s\)\\log\\left\(\\pi\(a\|s\)\\right\)\-\\tilde\{\\pi\}\(a\|s\)\\log\\left\(\\tilde\{\\pi\}\(a\|s\)\\right\)\\right\]\}\_\{\\mathbf\{\(III\)\}\}\\kern 5\.0pt\.We now lower\-bound each of the three terms separately\.

#### Bounding\(𝐈\)\\mathbf\{\(I\)\}\.

Using[Lemma25](https://arxiv.org/html/2605.24357#Thmlemma25), we have

\(𝐈\)\\displaystyle\\mathbf\{\(I\)\}≥−∥dρπ~−dρπ∥1maxs∈𝒮\|∑a∈𝒜\[π~\(a\|s\)𝗋\(s,a\)−λπ~\(a\|s\)log\(π~\(a\|s\)\)\]\|\\displaystyle\\geq\-\\lVert d\_\{\\rho\}^\{\\hskip 0\.57367pt\\tilde\{\\pi\}\}\-d\_\{\\rho\}^\{\\hskip 0\.57367pt\\pi\}\\rVert\_\{1\}\\max\_\{s\\in\\mathcal\{S\}\}\\left\|\\sum\_\{a\\in\\mathcal\{A\}\}\\left\[\\tilde\{\\pi\}\(a\|s\)\\mathsf\{r\}\(s,a\)\-\\lambda\\tilde\{\\pi\}\(a\|s\)\\log\\left\(\\tilde\{\\pi\}\(a\|s\)\\right\)\\right\]\\right\|≥−γ1−γsups∈𝒮∥π~\(⋅\|s\)−π\(⋅\|s\)∥1\(1\+λlog\(\|𝒜\|\)\)\.\\displaystyle\\geq\-\\frac\{\\gamma\}\{1\-\\gamma\}\\sup\_\{s\\in\\mathcal\{S\}\}\\lVert\\tilde\{\\pi\}\(\\cdot\|s\)\-\\pi\(\\cdot\|s\)\\rVert\_\{1\}\\left\(1\+\\lambda\\log\(\|\\mathcal\{A\}\|\)\\right\)\\kern 5\.0pt\.

#### Bounding\(𝐈𝐈\)\\mathbf\{\(II\)\}\.

Using the triangle inequality yields

\(𝐈𝐈\)≥−sups∈𝒮∥π~\(⋅\|s\)−π\(⋅\|s\)∥1\.\\displaystyle\\mathbf\{\(II\)\}\\geq\-\\sup\_\{s\\in\\mathcal\{S\}\}\\lVert\\tilde\{\\pi\}\(\\cdot\|s\)\-\\pi\(\\cdot\|s\)\\rVert\_\{1\}\\kern 5\.0pt\.

#### Bounding\(𝐈𝐈𝐈\)\\mathbf\{\(III\)\}\.

All the state\-action pairs on which the originalπ\\piallocates the same probability then the policyπ~\\tilde\{\\pi\}are equal to0in\(𝐈𝐈𝐈\)\\mathbf\{\(III\)\}allowing us to simplify this term

\(𝐈𝐈𝐈\)\\displaystyle\\mathbf\{\(III\)\}=λ​∑s∈𝒮dρπ​\(s\)​∑a∈𝒜\[π​\(a\|s\)​log⁡\(π​\(a\|s\)\)−π~​\(a\|s\)​log⁡\(π~​\(a\|s\)\)\]\\displaystyle=\\lambda\\sum\_\{s\\in\\mathcal\{S\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\pi\}\(s\)\\sum\_\{a\\in\\mathcal\{A\}\}\\left\[\\pi\(a\|s\)\\log\\left\(\\pi\(a\|s\)\\right\)\-\\tilde\{\\pi\}\(a\|s\)\\log\\left\(\\tilde\{\\pi\}\(a\|s\)\\right\)\\right\]=λ​∑s∈𝒮dρπ​\(s\)​∑a∈𝒜τπ​\(s\)\[π​\(a\|s\)​log⁡\(π​\(a\|s\)\)−π~​\(a\|s\)​log⁡\(π~​\(a\|s\)\)\]\\displaystyle=\\lambda\\sum\_\{s\\in\\mathcal\{S\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\pi\}\(s\)\\sum\_\{a\\in\\mathcal\{A\}\_\{\\tau\}^\{\\pi\}\(s\)\}\\left\[\\pi\(a\|s\)\\log\\left\(\\pi\(a\|s\)\\right\)\-\\tilde\{\\pi\}\(a\|s\)\\log\(\\tilde\{\\pi\}\(a\|s\)\)\\right\]\+λ​∑s∈𝒮𝟣​\(𝒜τπ​\(s\)≠∅\)​dρπ​\(s\)​\[π​\(amaxπ​\(s\)\|s\)​log⁡\(π​\(amaxπ​\(s\)\|s\)\)−π~​\(amaxπ​\(s\)\|s\)​log⁡\(π~​\(amaxπ​\(s\)\|s\)\)\]\.\\displaystyle\+\\lambda\\sum\_\{s\\in\\mathcal\{S\}\}\\mathsf\{1\}\(\\mathcal\{A\}\_\{\\tau\}^\{\\pi\}\(s\)\\neq\\emptyset\)d\_\{\\rho\}^\{\\hskip 0\.57367pt\\pi\}\(s\)\\left\[\\pi\(a\_\{\\max\}^\{\\pi\}\(s\)\|s\)\\log\\left\(\\pi\(a\_\{\\max\}^\{\\pi\}\(s\)\|s\)\\right\)\-\\tilde\{\\pi\}\(a\_\{\\max\}^\{\\pi\}\(s\)\|s\)\\log\\left\(\\tilde\{\\pi\}\(a\_\{\\max\}^\{\\pi\}\(s\)\|s\)\\right\)\\right\]\\kern 5\.0pt\.Sincex↦x​log⁡\(x\)x\\mapsto x\\log\(x\)is convex, for allu,v∈\[0;1\]u,v\\in\[0;1\],u​log⁡\(u\)−v​log⁡\(v\)≥\[log⁡\(v\)\+1\]​\(u−v\)u\\log\(u\)\-v\\log\(v\)\\geq\\left\[\\log\(v\)\+1\\right\]\(u\-v\), we have

\(𝐈𝐈𝐈\)\\displaystyle\\mathbf\{\(III\)\}≥λ​∑s∈𝒮dρπ​\(s\)​∑a∈𝒜τπ​\(s\)\(π​\(a\|s\)−π~​\(a\|s\)\)​\[log⁡\(τλ\)\+1\]\(sinceπ~​\(a\|s\)=τλ\)\\displaystyle\\geq\\lambda\\sum\_\{s\\in\\mathcal\{S\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\pi\}\(s\)\\sum\_\{a\\in\\mathcal\{A\}\_\{\\tau\}^\{\\pi\}\(s\)\}\(\\pi\(a\|s\)\-\\tilde\{\\pi\}\(a\|s\)\)\\left\[\\log\(\\tau\_\{\\lambda\}\)\+1\\right\]\\qquad\\text\{\(since $\\tilde\{\\pi\}\(a\|s\)=\\tau\_\{\\lambda\}$\)\}\+λ​∑s∈𝒮𝟣​\(𝒜τπ​\(s\)≠∅\)​dρπ​\(s\)​\[π​\(amaxπ​\(s\)\|s\)−π~​\(amaxπ​\(s\)\|s\)\]​\[log⁡\(π~​\(amaxπ​\(s\)\|s\)\)\+1\],\\displaystyle\+\\lambda\\sum\_\{s\\in\\mathcal\{S\}\}\\mathsf\{1\}\(\\mathcal\{A\}\_\{\\tau\}^\{\\pi\}\(s\)\\neq\\emptyset\)d\_\{\\rho\}^\{\\hskip 0\.57367pt\\pi\}\(s\)\\left\[\\pi\(a\_\{\\max\}^\{\\pi\}\(s\)\|s\)\-\\tilde\{\\pi\}\(a\_\{\\max\}^\{\\pi\}\(s\)\|s\)\\right\]\\left\[\\log\\left\(\\tilde\{\\pi\}\(a\_\{\\max\}^\{\\pi\}\(s\)\|s\)\\right\)\+1\\right\]\\kern 5\.0pt,Next, using that

π~​\(amaxπ​\(s\)\|s\)≥π​\(amaxπ​\(s\)\|s\)−\|𝒜\|​τλ≥1\|𝒜\|−12​\|𝒜\|=12​\|𝒜\|,\\displaystyle\\tilde\{\\pi\}\(a\_\{\\max\}^\{\\pi\}\(s\)\|s\)\\geq\\pi\(a\_\{\\max\}^\{\\pi\}\(s\)\|s\)\-\|\\mathcal\{A\}\|\\tau\_\{\\lambda\}\\geq\\frac\{1\}\{\|\\mathcal\{A\}\|\}\-\\frac\{1\}\{2\|\\mathcal\{A\}\|\}=\\frac\{1\}\{2\|\\mathcal\{A\}\|\}\\kern 5\.0pt,combined with the monotonicity ofx:log⁡\(x\)\+1x\\colon\\log\(x\)\+1and the fact thatπ​\(amaxπ​\(s\)\|s\)−π~​\(amaxπ​\(s\)\|s\)≥0\\pi\(a\_\{\\max\}^\{\\pi\}\(s\)\|s\)\-\\tilde\{\\pi\}\(a\_\{\\max\}^\{\\pi\}\(s\)\|s\)\\geq 0yields

\(𝐈𝐈𝐈\)\\displaystyle\\mathbf\{\(III\)\}≥λ​∑s∈𝒮dρ,π~​\(s\)​∑a∈𝒜τπ​\(s\)\(π​\(a\|s\)−π~​\(a\|s\)\)​\[log⁡\(τλ\)\+1\]\\displaystyle\\geq\\lambda\\sum\_\{s\\in\\mathcal\{S\}\}d\_\{\\rho,\\tilde\{\\pi\}\}\(s\)\\sum\_\{a\\in\\mathcal\{A\}\_\{\\tau\}^\{\\pi\}\(s\)\}\(\\pi\(a\|s\)\-\\tilde\{\\pi\}\(a\|s\)\)\\left\[\\log\(\\tau\_\{\\lambda\}\)\+1\\right\]\+λ​∑s∈𝒮𝟣​\(𝒜τπ​\(s\)≠∅\)​dρπ~​\(s\)​\[π​\(amaxπ​\(s\)\|s\)−π~​\(amaxπ​\(s\)\|s\)\]​\[log⁡\(12​\|𝒜\|\)\+1\],\\displaystyle\+\\lambda\\sum\_\{s\\in\\mathcal\{S\}\}\\mathsf\{1\}\(\\mathcal\{A\}\_\{\\tau\}^\{\\pi\}\(s\)\\neq\\emptyset\)d\_\{\\rho\}^\{\\hskip 0\.57367pt\\tilde\{\\pi\}\}\(s\)\\left\[\\pi\(a\_\{\\max\}^\{\\pi\}\(s\)\|s\)\-\\tilde\{\\pi\}\(a\_\{\\max\}^\{\\pi\}\(s\)\|s\)\\right\]\\left\[\\log\(\\frac\{1\}\{2\|\\mathcal\{A\}\|\}\)\+1\\right\]\\kern 5\.0pt,Additionally, since

0≤π\(amaxπ\(s\)\|s\)−π~\(amaxπ\(s\)\|s\)=∑a∈𝒜τπ​\(s\)\(π\(a\|s\)−π~\(a\|s\)\)=12∥π\(⋅\|s\)−π~\(⋅\|s\)∥1,\\displaystyle 0\\leq\\pi\(a\_\{\\max\}^\{\\pi\}\(s\)\|s\)\-\\tilde\{\\pi\}\(a\_\{\\max\}^\{\\pi\}\(s\)\|s\)=\\sum\_\{a\\in\\mathcal\{A\}\_\{\\tau\}^\{\\pi\}\(s\)\}\(\\pi\(a\|s\)\-\\tilde\{\\pi\}\(a\|s\)\)=\\frac\{1\}\{2\}\\left\\\|\\pi\(\\cdot\|s\)\-\\tilde\{\\pi\}\(\\cdot\|s\)\\right\\\|\_\{1\}\\kern 5\.0pt,implies

\(𝐈𝐈𝐈\)\\displaystyle\\mathbf\{\(III\)\}≥−λ2∑s∈𝒮dρπ~\(s\)𝟣\(𝒜τπ\(s\)≠∅\)∥π\(⋅\|s\)−π~\(⋅\|s\)∥1\[log\(τλ\)\+1\]\\displaystyle\\geq\-\\frac\{\\lambda\}\{2\}\\sum\_\{s\\in\\mathcal\{S\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\tilde\{\\pi\}\}\(s\)\\mathsf\{1\}\(\\mathcal\{A\}\_\{\\tau\}^\{\\pi\}\(s\)\\neq\\emptyset\)\\left\\\|\\pi\(\\cdot\|s\)\-\\tilde\{\\pi\}\(\\cdot\|s\)\\right\\\|\_\{1\}\\left\[\\log\(\\tau\_\{\\lambda\}\)\+1\\right\]−λ2∑s∈𝒮dρπ~\(s\)𝟣\(𝒜τπ\(s\)≠∅\)∥π\(⋅\|s\)−π~\(⋅\|s\)∥1\[log\(2\|𝒜\|\)\+1\],\\displaystyle\-\\frac\{\\lambda\}\{2\}\\sum\_\{s\\in\\mathcal\{S\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\tilde\{\\pi\}\}\(s\)\\mathsf\{1\}\(\\mathcal\{A\}\_\{\\tau\}^\{\\pi\}\(s\)\\neq\\emptyset\)\\left\\\|\\pi\(\\cdot\|s\)\-\\tilde\{\\pi\}\(\\cdot\|s\)\\right\\\|\_\{1\}\[\\log\(2\|\\mathcal\{A\}\|\)\+1\]\\kern 5\.0pt,≥−λ4∑s∈𝒮dρπ~\(s\)𝟣\(𝒜τπ\(s\)≠∅\)∥π\(⋅\|s\)−π~\(⋅\|s\)∥1\[log\(τλ\)\+1\],\\displaystyle\\geq\-\\frac\{\\lambda\}\{4\}\\sum\_\{s\\in\\mathcal\{S\}\}d\_\{\\rho\}^\{\\hskip 0\.57367pt\\tilde\{\\pi\}\}\(s\)\\mathsf\{1\}\(\\mathcal\{A\}\_\{\\tau\}^\{\\pi\}\(s\)\\neq\\emptyset\)\\left\\\|\\pi\(\\cdot\|s\)\-\\tilde\{\\pi\}\(\\cdot\|s\)\\right\\\|\_\{1\}\\left\[\\log\(\\tau\_\{\\lambda\}\)\+1\\right\]\\kern 5\.0pt,where in the last inequality, we used thatτλ≤138​\|𝒜\|4≤exp⁡\(−4​log⁡\(2​\|𝒜\|\)−5\)\\tau\_\{\\lambda\}\\leq\\frac\{1\}\{3^\{8\}\|\\mathcal\{A\}\|^\{4\}\}\\leq\\exp\(\-4\\log\(2\|\\mathcal\{A\}\|\)\-5\)\. Hence, by using[Aρ\\textbf\{A\}\_\{\\rho\}](https://arxiv.org/html/2605.24357#ThmassumMDPx1), we can lower bound this term as follows

\(𝐈𝐈𝐈\)≥−λ4\(1−γ\)ρminmaxs∈𝒮∥π\(⋅\|s\)−π~\(⋅\|s\)∥1\[log\(τλ\)\+1\]\.\\displaystyle\\mathbf\{\(III\)\}\\geq\-\\frac\{\\lambda\}\{4\}\(1\-\\gamma\)\\rho\_\{\\min\}\\max\_\{s\\in\\mathcal\{S\}\}\\left\\\|\\pi\(\\cdot\|s\)\-\\tilde\{\\pi\}\(\\cdot\|s\)\\right\\\|\_\{1\}\\left\[\\log\(\\tau\_\{\\lambda\}\)\+1\\right\]\\kern 5\.0pt\.Collecting these lower bounds and using that

\[log⁡\(τλ\)\+1\]≤−16\+8​γ​λ​log⁡\(\|𝒜\|\)λ​\(1−γ\)2​ρmin\\displaystyle\\left\[\\log\(\\tau\_\{\\lambda\}\)\+1\\right\]\\leq\-\\frac\{16\+8\\gamma\\lambda\\log\(\|\\mathcal\{A\}\|\)\}\{\\lambda\(1\-\\gamma\)^\{2\}\\rho\_\{\\min\}\}concludes the proof\. ∎ Finally, we define the operator that maps each policy to one corresponding parameter

ℒ:Π→ℝ\|𝒮\|​\|𝒜\|\\mathcal\{L\}:\\Pi\\;\\to\\;\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}by

ℒ​\(π\)​\(s,a\)​=Δ​log⁡\(π​\(a\|s\)\),for all​\(s,a\)∈𝒮×𝒜\.\\displaystyle\\mathcal\{L\}\(\\pi\)\(s,a\)\\;\\overset\{\\Delta\}\{=\}\\;\\log\(\\pi\(a\|s\)\),\\quad\\text\{for all\}\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\\,\.\(35\)Finally, we define the improvement operator on the logitspace as

𝒯τ​=Δ​ℒ∘𝒰τ\.\\displaystyle\\mathcal\{T\}\_\{\\tau\}\\overset\{\\Delta\}\{=\}\\mathcal\{L\}\\circ\\mathcal\{U\}\_\{\\tau\}\\kern 5\.0pt\.The following lemma shows thatℒτ\\mathcal\{L\}\_\{\\tau\}successfully recovers a parameter that gives the policy and that𝒯τ\\mathcal\{T\}\_\{\\tau\}improves the value of the objective whenτ=τλ\\tau=\\tau\_\{\\lambda\}\.

###### Lemma 21\.

Assume that the initial distributionρ\\rhosatisfies[Aρ\\textbf\{A\}\_\{\\rho\}](https://arxiv.org/html/2605.24357#ThmassumMDPx1)\. For any policyπ\\pi, it holds that

πℒ​\(π\)=π,\\displaystyle\\pi\_\{\\mathcal\{L\}\(\\pi\)\}=\\pi\\kern 5\.0pt,Additionally, for anyθ∈ℝ\|𝒮\|​\|𝒜\|\\theta\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\|\\mathcal\{A\}\|\}and\(s,a\)∈𝒮×𝒜\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}, we have that

v~𝒯τλ​\(θ\)λ​\(ρ\)≥v~θλ​\(ρ\),π𝒯τλ​\(θ\)≥τλ\.\\displaystyle\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\mathcal\{T\}\_\{\\tau\_\{\\lambda\}\}\(\\theta\)\}^\{\\lambda\}\(\\rho\)\\geq\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\theta\}^\{\\lambda\}\(\\rho\)\\kern 5\.0pt,\\quad\\pi\_\{\\mathcal\{T\}\_\{\\tau\_\{\\lambda\}\}\(\\theta\)\}\\geq\\tau\_\{\\lambda\}\\kern 5\.0pt\.

###### Proof\.

The proof follows immediately from the definition of the softmax policy, from \([35](https://arxiv.org/html/2605.24357#A6.E35)\), and[Lemma20](https://arxiv.org/html/2605.24357#Thmlemma20)\. ∎

Define the set

Πτ​=Δ​\{π∈𝒫​\(𝒜\)𝒮,such that for all​\(s,a\)∈𝒮×𝒜,π​\(a\|s\)≥τ\}\.\\displaystyle\\Pi\_\{\\tau\}\\overset\{\\Delta\}\{=\}\\left\\\{\\pi\\in\\mathcal\{P\}\(\\mathcal\{A\}\)^\{\\mathcal\{S\}\},\\text\{ such that for all \}\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\},\\pi\(a\|s\)\\geq\\tau\\right\\\}\\kern 5\.0pt\.The next lemma shows that𝒰τ\\mathcal\{U\}\_\{\\tau\}can be seen as a projection on this set\.

###### Lemma 22\.

Fix any policyπ1\\pi\_\{1\}and any policyπ2∈Πτ\\pi\_\{2\}\\in\\Pi\_\{\\tau\}\. Then

‖π1−𝒰τ​\(π1\)‖1≤‖π1−π2‖1\.\\left\\\|\\pi\_\{1\}\-\\mathcal\{U\}\_\{\\tau\}\(\\pi\_\{1\}\)\\right\\\|\_\{1\}\\leq\\left\\\|\\pi\_\{1\}\-\\pi\_\{2\}\\right\\\|\_\{1\}\.More precisely, for everys∈𝒮s\\in\\mathcal\{S\},

∥π1\(⋅∣s\)−𝒰τ\(π1\)\(⋅∣s\)∥1≤∥π1\(⋅∣s\)−π2\(⋅∣s\)∥1\.\\left\\\|\\pi\_\{1\}\(\\cdot\\mid s\)\-\\mathcal\{U\}\_\{\\tau\}\(\\pi\_\{1\}\)\(\\cdot\\mid s\)\\right\\\|\_\{1\}\\leq\\left\\\|\\pi\_\{1\}\(\\cdot\\mid s\)\-\\pi\_\{2\}\(\\cdot\\mid s\)\\right\\\|\_\{1\}\.

###### Proof\.

Since theℓ1\\ell\_\{1\}\-norm decomposes over the states, it is enough to prove the statewise inequality\. Fixs∈𝒮s\\in\\mathcal\{S\}and set

As:=𝒜τπ1​\(s\),Ds:=∑a∈As\(τ−π1​\(a∣s\)\)\.A\_\{s\}:=\\mathcal\{A\}\_\{\\tau\}^\{\\pi\_\{1\}\}\(s\),\\qquad D\_\{s\}:=\\sum\_\{a\\in A\_\{s\}\}\\bigl\(\\tau\-\\pi\_\{1\}\(a\\mid s\)\\bigr\)\.
IfAs=∅A\_\{s\}=\\emptyset, then𝒰τ\(π1\)\(⋅∣s\)=π1\(⋅∣s\)\\mathcal\{U\}\_\{\\tau\}\(\\pi\_\{1\}\)\(\\cdot\\mid s\)=\\pi\_\{1\}\(\\cdot\\mid s\), so the claim is immediate\. Assume now thatAs≠∅A\_\{s\}\\neq\\emptyset\. By definition of𝒰τ\\mathcal\{U\}\_\{\\tau\}, each action inAsA\_\{s\}is increased toτ\\tau, the actionamaxπ1​\(s\)a\_\{\\max\}^\{\\pi\_\{1\}\}\(s\)is decreased by exactly the total transferred massDsD\_\{s\}, and all other actions are unchanged\. Therefore,

∥π1\(⋅∣s\)−𝒰τ\(π1\)\(⋅∣s\)∥1=∑a∈As\(τ−π1\(a∣s\)\)\+Ds=2Ds\.\\left\\\|\\pi\_\{1\}\(\\cdot\\mid s\)\-\\mathcal\{U\}\_\{\\tau\}\(\\pi\_\{1\}\)\(\\cdot\\mid s\)\\right\\\|\_\{1\}=\\sum\_\{a\\in A\_\{s\}\}\\bigl\(\\tau\-\\pi\_\{1\}\(a\\mid s\)\\bigr\)\+D\_\{s\}=2D\_\{s\}\.
Now decompose

∥π1\(⋅∣s\)−π2\(⋅∣s\)∥1=∑a∈As\|π1\(a∣s\)−π2\(a∣s\)\|\+∑a∉As\|π1\(a∣s\)−π2\(a∣s\)\|\.\\left\\\|\\pi\_\{1\}\(\\cdot\\mid s\)\-\\pi\_\{2\}\(\\cdot\\mid s\)\\right\\\|\_\{1\}=\\sum\_\{a\\in A\_\{s\}\}\\left\|\\pi\_\{1\}\(a\\mid s\)\-\\pi\_\{2\}\(a\\mid s\)\\right\|\+\\sum\_\{a\\notin A\_\{s\}\}\\left\|\\pi\_\{1\}\(a\\mid s\)\-\\pi\_\{2\}\(a\\mid s\)\\right\|\.
Sinceπ2∈Πτ\\pi\_\{2\}\\in\\Pi\_\{\\tau\}, for everya∈Asa\\in A\_\{s\}we haveπ2​\(a∣s\)≥τ≥π1​\(a∣s\)\\pi\_\{2\}\(a\\mid s\)\\geq\\tau\\geq\\pi\_\{1\}\(a\\mid s\)\. Hence

∑a∈As\|π1\(a∣s\)−π2\(a∣s\)\|=∑a∈As\(π2\(a∣s\)−π1\(a∣s\)\)≥∑a∈As\(τ−π1\(a∣s\)\)=Ds\.\\sum\_\{a\\in A\_\{s\}\}\\left\|\\pi\_\{1\}\(a\\mid s\)\-\\pi\_\{2\}\(a\\mid s\)\\right\|=\\sum\_\{a\\in A\_\{s\}\}\\bigl\(\\pi\_\{2\}\(a\\mid s\)\-\\pi\_\{1\}\(a\\mid s\)\\bigr\)\\geq\\sum\_\{a\\in A\_\{s\}\}\\bigl\(\\tau\-\\pi\_\{1\}\(a\\mid s\)\\bigr\)=D\_\{s\}\.
Also, because bothπ1\(⋅∣s\)\\pi\_\{1\}\(\\cdot\\mid s\)andπ2\(⋅∣s\)\\pi\_\{2\}\(\\cdot\\mid s\)are probability distributions,

∑a∈As\(π2​\(a∣s\)−π1​\(a∣s\)\)=∑a∉As\(π1​\(a∣s\)−π2​\(a∣s\)\)\.\\sum\_\{a\\in A\_\{s\}\}\\bigl\(\\pi\_\{2\}\(a\\mid s\)\-\\pi\_\{1\}\(a\\mid s\)\\bigr\)=\\sum\_\{a\\notin A\_\{s\}\}\\bigl\(\\pi\_\{1\}\(a\\mid s\)\-\\pi\_\{2\}\(a\\mid s\)\\bigr\)\.Therefore, by the triangle inequality,

∑a∉As\|π1\(a∣s\)−π2\(a∣s\)\|≥\|∑a∉As\(π1\(a∣s\)−π2\(a∣s\)\)\|=∑a∈As\(π2\(a∣s\)−π1\(a∣s\)\)≥Ds\.\\sum\_\{a\\notin A\_\{s\}\}\\left\|\\pi\_\{1\}\(a\\mid s\)\-\\pi\_\{2\}\(a\\mid s\)\\right\|\\geq\\left\|\\sum\_\{a\\notin A\_\{s\}\}\\bigl\(\\pi\_\{1\}\(a\\mid s\)\-\\pi\_\{2\}\(a\\mid s\)\\bigr\)\\right\|=\\sum\_\{a\\in A\_\{s\}\}\\bigl\(\\pi\_\{2\}\(a\\mid s\)\-\\pi\_\{1\}\(a\\mid s\)\\bigr\)\\geq D\_\{s\}\.
Combining the two bounds gives

∥π1\(⋅∣s\)−π2\(⋅∣s\)∥1≥Ds\+Ds=2Ds=∥π1\(⋅∣s\)−𝒰τ\(π1\)\(⋅∣s\)∥1\.\\left\\\|\\pi\_\{1\}\(\\cdot\\mid s\)\-\\pi\_\{2\}\(\\cdot\\mid s\)\\right\\\|\_\{1\}\\geq D\_\{s\}\+D\_\{s\}=2D\_\{s\}=\\left\\\|\\pi\_\{1\}\(\\cdot\\mid s\)\-\\mathcal\{U\}\_\{\\tau\}\(\\pi\_\{1\}\)\(\\cdot\\mid s\)\\right\\\|\_\{1\}\.This proves the statewise inequality\. Summing overs∈𝒮s\\in\\mathcal\{S\}concludes the proof\. ∎

## Appendix GTechnical Lemmas

###### Lemma 23\(Lemma 1\.2\.3 in\(Nesterov,[2013](https://arxiv.org/html/2605.24357#bib.bib4)\)\)\.

Letf:ℝd→ℝf:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}be twice continuously differentiable\. Suppose there existsL≥0L\\geq 0such that for allx∈ℝdx\\in\\mathbb\{R\}^\{d\}andv∈ℝdv\\in\\mathbb\{R\}^\{d\},

\|v⊤​∇2f​\(x\)​v\|≤L​‖v‖2\.\|v^\{\\top\}\\nabla^\{2\}f\(x\)\\,v\|\\;\\leq\\;L\\\|v\\\|^\{2\}\.Thenffhas anLL\-Lipschitz continuous gradient \(i\.e\.,ffisLL\-smooth\); in particular,

‖∇f​\(y\)−∇f​\(x\)‖≤L​‖y−x‖,\\\|\\nabla f\(y\)\-\\nabla f\(x\)\\\|\\;\\leq\\;L\\\|y\-x\\\|,and

f​\(y\)≥f​\(x\)\+⟨∇f​\(x\),y−x⟩−L2​‖y−x‖2f\(y\)\\;\\geq\\;f\(x\)\+\\langle\\nabla f\(x\),y\-x\\rangle\-\\tfrac\{L\}\{2\}\\\|y\-x\\\|^\{2\}for allx,y∈ℝdx,y\\in\\mathbb\{R\}^\{d\}\.

###### Lemma 24\(Flow conservation constraints\(Puterman,[1994](https://arxiv.org/html/2605.24357#bib.bib2)\)\)\.

For anyπ∈Π\\pi\\in\\Pi, ands∈𝒮s\\in\\mathcal\{S\}, it holds that

dρπ​\(s\)=\(1−γ\)​ρ​\(s\)\+γ​∑\(s′,a′\)𝖯​\(s\|s′,a′\)​π​\(a′\|s′\)​dρπ​\(s′\)\.\\displaystyle d\_\{\\rho\}^\{\\hskip 0\.57367pt\\pi\}\(s\)=\(1\-\\gamma\)\\rho\(s\)\+\\gamma\\sum\_\{\(s^\{\\prime\},a^\{\\prime\}\)\}\\mathsf\{P\}\(s\|s^\{\\prime\},a^\{\\prime\}\)\\pi\(a^\{\\prime\}\|s^\{\\prime\}\)d\_\{\\rho\}^\{\\hskip 0\.57367pt\\pi\}\(s^\{\\prime\}\)\\kern 5\.0pt\.

###### Lemma 25\.

Consider any two policiesπi\\pi\_\{i\},i=1,2i=1,2\. It holds that

∥dρπ1−dρπ2∥1≤γ1−γsups∈𝒮∥π1\(⋅\|s\)−π2\(⋅\|s\)∥1\.\\displaystyle\\lVert d\_\{\\rho\}^\{\\hskip 0\.57367pt\\pi\_\{1\}\}\-d\_\{\\rho\}^\{\\hskip 0\.57367pt\\pi\_\{2\}\}\\rVert\_\{1\}\\leq\\frac\{\\gamma\}\{1\-\\gamma\}\\sup\_\{s\\in\\mathcal\{S\}\}\\lVert\\pi\_\{1\}\(\\cdot\|s\)\-\\pi\_\{2\}\(\\cdot\|s\)\\rVert\_\{1\}\\kern 5\.0pt\.

###### Proof\.

Let us start from the definition of flow conservation constraints for the discounted state occupancy[Lemma24](https://arxiv.org/html/2605.24357#Thmlemma24), fori∈\{1,2\}i\\in\\\{1,2\\\}, we have

dρπi​\(s\)=\(1−γ\)​ρ​\(s\)\+γ​∑s′𝖯πi​\(s\|s′\)​dρπi​\(s′\)\.\\displaystyle d\_\{\\rho\}^\{\\hskip 0\.57367pt\\pi\_\{i\}\}\(s\)=\(1\-\\gamma\)\\rho\(s\)\+\\gamma\\sum\_\{s^\{\\prime\}\}\\mathsf\{P\}\_\{\\pi\_\{i\}\}\(s\|s^\{\\prime\}\)d\_\{\\rho\}^\{\\hskip 0\.57367pt\\pi\_\{i\}\}\(s^\{\\prime\}\)\\kern 5\.0pt\.Then, we have

∑s∈𝒮\|dρπ2​\(s\)−dρπ1​\(s\)\|\\displaystyle\\sum\_\{s\\in\\mathcal\{S\}\}\|d\_\{\\rho\}^\{\\hskip 0\.57367pt\\pi\_\{2\}\}\(s\)\-d\_\{\\rho\}^\{\\hskip 0\.57367pt\\pi\_\{1\}\}\(s\)\|≤γ∑\(s′,s′\)∑s\|𝖯\(s\|s′,s′\)π2\(s′\|s′\)dρπ2\(s′\)−𝖯\(s\|s′,s′\)π1\(s′\|s′\)dρπ1\(s′\)\|\\displaystyle\\leq\\gamma\\sum\_\{\(s^\{\\prime\},s^\{\\prime\}\)\}\\sum\_\{s\}\\left\|\\mathsf\{P\}\(s\|s^\{\\prime\},s^\{\\prime\}\)\\pi\_\{2\}\(s^\{\\prime\}\|s^\{\\prime\}\)d\_\{\\rho\}^\{\\hskip 0\.57367pt\\pi\_\{2\}\}\(s^\{\\prime\}\)\-\\mathsf\{P\}\(s\|s^\{\\prime\},s^\{\\prime\}\)\\pi\_\{1\}\(s^\{\\prime\}\|s^\{\\prime\}\)d\_\{\\rho\}^\{\\hskip 0\.57367pt\\pi\_\{1\}\}\(s^\{\\prime\}\)\\right\|≤γ∑s′,s′∑s𝖯\(s\|s′,s′\)\|π2\(a′\|s′\)−π1\(a′\|s′\)\|dρπ2\(s′\)\\displaystyle\\leq\\gamma\\sum\_\{s^\{\\prime\},s^\{\\prime\}\}\\sum\_\{s\}\\mathsf\{P\}\(s\|s^\{\\prime\},s^\{\\prime\}\)\\left\|\\pi\_\{2\}\(a^\{\\prime\}\|s^\{\\prime\}\)\-\\pi\_\{1\}\(a^\{\\prime\}\|s^\{\\prime\}\)\\right\|d\_\{\\rho\}^\{\\hskip 0\.57367pt\\pi\_\{2\}\}\(s^\{\\prime\}\)\+γ​∑s′,a′∑s𝖯​\(s\|s′,a′\)​π1​\(a′\|s′\)​\|dρπ1​\(s′\)−dρπ2​\(s′\)\|\\displaystyle\+\\gamma\\sum\_\{s^\{\\prime\},a^\{\\prime\}\}\\sum\_\{s\}\\mathsf\{P\}\(s\|s^\{\\prime\},a^\{\\prime\}\)\\pi\_\{1\}\(a^\{\\prime\}\|s^\{\\prime\}\)\\left\|d\_\{\\rho\}^\{\\hskip 0\.57367pt\\pi\_\{1\}\}\(s^\{\\prime\}\)\-d\_\{\\rho\}^\{\\hskip 0\.57367pt\\pi\_\{2\}\}\(s^\{\\prime\}\)\\right\|≤γsups∈𝒮∥π1\(⋅\|s\)−π2\(⋅\|s\)∥1\+γ∑s′\|dρπ1\(s′\)−dρπ2\(s′\)\|,\\displaystyle\\leq\\gamma\\sup\_\{s\\in\\mathcal\{S\}\}\\lVert\\pi\_\{1\}\(\\cdot\|s\)\-\\pi\_\{2\}\(\\cdot\|s\)\\rVert\_\{1\}\+\\gamma\\sum\_\{s^\{\\prime\}\}\|d\_\{\\rho\}^\{\\hskip 0\.57367pt\\pi\_\{1\}\}\(s^\{\\prime\}\)\-d\_\{\\rho\}^\{\\hskip 0\.57367pt\\pi\_\{2\}\}\(s^\{\\prime\}\)\|\\kern 5\.0pt,which concludes the proof\. ∎

###### Lemma 26\(Soft\-Performance Difference Lemma\)\.

Consider any two policiesπi\\pi\_\{i\},i=1,2i=1,2\. For anys∈𝒮s\\in\\mathcal\{S\}, it holds that

v~π2λ\(s\)−v~π1λ\(s\)=11−γ∑s∈𝒮dsπ1\(s′\)\[∑a∈𝒜\(π2\(a\|s′\)−π1\(a\|s′\)\)\[q~π2λ\(s′,a\)−λlog\(π2\(a\|s′\)\)\+λKL\(π1\(⋅\|s′\)\|\|π2\(⋅\|s′\)\)\]\]\.\\displaystyle\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\pi\_\{2\}\}^\{\\lambda\}\(s\)\\\!\-\\\!\\tilde\{\\mathrm\{v\}\}\_\{\\hskip 0\.57367pt\\pi\_\{1\}\}^\{\\lambda\}\(s\)\\\!=\\\!\\frac\{1\}\{1\-\\gamma\}\\sum\_\{s\\in\\mathcal\{S\}\}d\_\{s\}^\{\\hskip 0\.57367pt\\pi\_\{1\}\}\(s^\{\\prime\}\)\\left\[\\sum\_\{a\\in\\mathcal\{A\}\}\\left\(\\pi\_\{2\}\(a\|s^\{\\prime\}\)\\\!\-\\\!\\pi\_\{1\}\(a\|s^\{\\prime\}\)\\right\)\\left\[\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\pi\_\{2\}\}^\{\\lambda\}\(s^\{\\prime\},a\)\\\!\-\\\!\\lambda\\log\(\\pi\_\{2\}\(a\|s^\{\\prime\}\)\)\\\!\+\\\!\\lambda\\mathrm\{KL\}\(\\pi\_\{1\}\(\\cdot\|s^\{\\prime\}\)\|\|\\pi\_\{2\}\(\\cdot\|s^\{\\prime\}\)\)\\right\]\\right\]\.
###### Lemma 27\(Bound on the entropy regularized Q\-value: Equation 86 of\(Meiet al\.,[2020b](https://arxiv.org/html/2605.24357#bib.bib3)\)\)\.

For any policyπ∈Π\\pi\\in\\Pi,λ≥0\\lambda\\geq 0, and0≤γ<10\\leq\\gamma<1, it holds that

‖q~πλ‖∞≤1\+λ​log⁡\(\|𝒜\|\)1−γ\.\\displaystyle\\left\\\|\\tilde\{\\mathrm\{q\}\}\_\{\\hskip 0\.57367pt\\pi\}^\{\\lambda\}\\right\\\|\_\{\\infty\}\\leq\\frac\{1\+\\lambda\\log\(\|\\mathcal\{A\}\|\)\}\{1\-\\gamma\}\\kern 5\.0pt\.

###### Lemma 28\(Pinsker’s inequality \(discrete version\)\)\.

Letp=\(pi\)i=1np=\(p\_\{i\}\)\_\{i=1\}^\{n\}andq=\(qi\)i=1nq=\(q\_\{i\}\)\_\{i=1\}^\{n\}be probability distributions on a finite set\{1,…,n\}\\\{1,\\dots,n\\\}\. Then

12​∑i=1n\|pi−qi\|≤12​KL​\(p∥q\)\.\\frac\{1\}\{2\}\\sum\_\{i=1\}^\{n\}\|p\_\{i\}\-q\_\{i\}\|\\;\\leq\\;\\sqrt\{\\tfrac\{1\}\{2\}\\,\\mathrm\{KL\}\(p\\\|q\)\}\.

###### Lemma 29\(KL upper bound\)\.

Letp=\(pi\)i=1np=\(p\_\{i\}\)\_\{i=1\}^\{n\}andq=\(qi\)i=1nq=\(q\_\{i\}\)\_\{i=1\}^\{n\}be probability distributions on a finite set\{1,…,n\}\\\{1,\\dots,n\\\}, and assume

qmin≔mini∈\[n\]⁡qi\>0\.q\_\{\\min\}\\;\\coloneqq\\;\\min\_\{i\\in\[n\]\}q\_\{i\}\\;\>\\;0\\kern 5\.0pt\.Then

KL​\(p∥q\)≤1qmin​‖p−q‖1\.\\mathrm\{KL\}\(p\\\|q\)\\;\\leq\\;\\frac\{1\}\{q\_\{\\min\}\}\\;\\\|p\-q\\\|\_\{1\}\\kern 5\.0pt\.

###### Proof\.

LetA≔\{i∈\[n\]:pi≥qi\}A\\coloneqq\\\{i\\in\[n\]:p\_\{i\}\\geq q\_\{i\}\\\}\. Sincelog⁡\(⋅\)\\log\(\\cdot\)is increasing, fori∉Ai\\notin Awe havelog⁡\(pi/qi\)≤0\\log\(p\_\{i\}/q\_\{i\}\)\\leq 0, hence

KL​\(p∥q\)=∑i=1npi​log⁡piqi≤∑i∈Api​log⁡piqi\.\\mathrm\{KL\}\(p\\\|q\)=\\sum\_\{i=1\}^\{n\}p\_\{i\}\\log\\frac\{p\_\{i\}\}\{q\_\{i\}\}\\;\\leq\\;\\sum\_\{i\\in A\}p\_\{i\}\\log\\frac\{p\_\{i\}\}\{q\_\{i\}\}\.Fori∈Ai\\in A, setui≔pi/qi≥1u\_\{i\}\\coloneqq p\_\{i\}/q\_\{i\}\\geq 1\. Moreover,pi≤1p\_\{i\}\\leq 1andqi≥qminq\_\{i\}\\geq q\_\{\\min\}implyui≤1/qminu\_\{i\}\\leq 1/q\_\{\\min\}\. We claim that for anym∈\(0,1\]m\\in\(0,1\]and anyu∈\[1,1/m\]u\\in\[1,1/m\],

u​log⁡u≤u−1m\.u\\log u\\;\\leq\\;\\frac\{u\-1\}\{m\}\.\(36\)To see this, defineh​\(u\)≔u−1m−u​log⁡uh\(u\)\\coloneqq\\frac\{u\-1\}\{m\}\-u\\log u\. Thenh′′​\(u\)=−1/u<0h^\{\\prime\\prime\}\(u\)=\-1/u<0, sohhis concave\. Alsoh​\(1\)=0h\(1\)=0\. Atu=1/mu=1/m,

h​\(1/m\)=1/m−1m−1m​log⁡1m=1m​\(1−mm−log⁡1m\)≥0,h\(1/m\)=\\frac\{1/m\-1\}\{m\}\-\\frac\{1\}\{m\}\\log\\frac\{1\}\{m\}=\\frac\{1\}\{m\}\\Big\(\\frac\{1\-m\}\{m\}\-\\log\\frac\{1\}\{m\}\\Big\)\\geq 0,where the last inequality follows fromlog⁡x≤x−1\\log x\\leq x\-1withx=1/mx=1/m\. By concavity,h​\(u\)≥0h\(u\)\\geq 0for allu∈\[1,1/m\]u\\in\[1,1/m\], proving \([36](https://arxiv.org/html/2605.24357#A7.E36)\)\.

Applying \([36](https://arxiv.org/html/2605.24357#A7.E36)\) withm=qminm=q\_\{\\min\}andu=uiu=u\_\{i\}yields

pi​log⁡piqi=qi​ui​log⁡ui≤qi⋅ui−1qmin=pi−qiqmin\.p\_\{i\}\\log\\frac\{p\_\{i\}\}\{q\_\{i\}\}=q\_\{i\}\\,u\_\{i\}\\log u\_\{i\}\\;\\leq\\;q\_\{i\}\\cdot\\frac\{u\_\{i\}\-1\}\{q\_\{\\min\}\}=\\frac\{p\_\{i\}\-q\_\{i\}\}\{q\_\{\\min\}\}\.Summing overi∈Ai\\in Agives

DKL​\(p∥q\)≤1qmin​∑i∈A\(pi−qi\)\.D\_\{\\mathrm\{KL\}\}\(p\\\|q\)\\;\\leq\\;\\frac\{1\}\{q\_\{\\min\}\}\\sum\_\{i\\in A\}\(p\_\{i\}\-q\_\{i\}\)\.which completes the proof\. ∎

###### Lemma 30\(KL\-logit inequality \(Lemma 27 of\(Meiet al\.,[2020b](https://arxiv.org/html/2605.24357#bib.bib3)\)\)\)\.

Forθ∈ℝ\|𝒜\|\\theta\\in\\mathbb\{R\}^\{\|\\mathcal\{A\}\|\}, definesoftmaxθ∈𝒫​\(𝒜\)\\operatorname\{softmax\}\_\{\\theta\}\\in\\mathcal\{P\}\(\\mathcal\{A\}\)such that

softmaxθ⁡\(a\)=exp⁡\(θ​\(a\)\)∑a′∈𝒜exp⁡\(θ​\(a′\)\)\.\\displaystyle\\operatorname\{softmax\}\_\{\\theta\}\(a\)=\\frac\{\\exp\(\\theta\(a\)\)\}\{\\sum\_\{a^\{\\prime\}\\in\\mathcal\{A\}\}\\exp\(\\theta\(a^\{\\prime\}\)\)\}\\kern 5\.0pt\.Fixθ,θ′∈ℝ\|𝒜\|\\theta,\\theta^\{\\prime\}\\in\\mathbb\{R\}^\{\|\\mathcal\{A\}\|\}\. Then for any constantc∈ℝc\\in\\mathbb\{R\}, we have

KL​\(softmaxθ∥softmaxθ′\)≤12​‖θ−θ′−c​𝟣\|𝒜\|‖∞2\.\\displaystyle\\mathrm\{KL\}\(\\operatorname\{softmax\}\_\{\\theta\}\\\|\\operatorname\{softmax\}\_\{\\theta^\{\\prime\}\}\)\\leq\\frac\{1\}\{2\}\\left\\\|\\theta\-\\theta^\{\\prime\}\-c\\mathsf\{1\}\_\{\|\\mathcal\{A\}\|\}\\right\\\|\_\{\\infty\}^\{2\}\\kern 5\.0pt\.

###### Lemma 31\.

Letp=\(pi\)i=1n∈Δnp=\(p\_\{i\}\)\_\{i=1\}^\{n\}\\in\\Delta\_\{n\}, where

Δn:=\{\(p1,…,pn\)∈\[0,1\]n:∑i=1npi=1\}\.\\Delta\_\{n\}:=\\left\\\{\(p\_\{1\},\\dots,p\_\{n\}\)\\in\[0,1\]^\{n\}:\\ \\sum\_\{i=1\}^\{n\}p\_\{i\}=1\\right\\\}\.Then

∑i=1npi​\(log⁡pi\)2≤1\+\(log⁡n\)2\.\\sum\_\{i=1\}^\{n\}p\_\{i\}\\bigl\(\\log p\_\{i\}\\bigr\)^\{2\}\\leq 1\+\(\\log n\)^\{2\}\.Here and below,log\\logdenotes the natural logarithm, and we use the convention0​\(log⁡0\)2:=00\(\\log 0\)^\{2\}:=0, which is justified by

limx↓0x​\(log⁡x\)2=0\.\\lim\_\{x\\downarrow 0\}x\(\\log x\)^\{2\}=0\.

###### Proof\.

Define

Fn​\(p\):=∑i=1npi​\(log⁡pi\)2,p∈Δn\.F\_\{n\}\(p\):=\\sum\_\{i=1\}^\{n\}p\_\{i\}\(\\log p\_\{i\}\)^\{2\},\\qquad p\\in\\Delta\_\{n\}\.We prove the claim by induction onnn\.

Step 1: the casesn=1n=1andn=2n=2\.

Ifn=1n=1, thenp1=1p\_\{1\}=1, so

F1​\(p\)=1⋅\(log⁡1\)2=0≤1\.F\_\{1\}\(p\)=1\\cdot\(\\log 1\)^\{2\}=0\\leq 1\.
Now letn=2n=2\. Consider the function

h​\(x\):=x​\(log⁡x\)2,x∈\[0,1\]\.h\(x\):=x\(\\log x\)^\{2\},\\qquad x\\in\[0,1\]\.A direct computation gives

h′​\(x\)=log⁡x​\(log⁡x\+2\)\.h^\{\\prime\}\(x\)=\\log x\\,\(\\log x\+2\)\.Hence the only critical point in\(0,1\)\(0,1\)isx=e−2x=e^\{\-2\}, and sinceh​\(0\)=h​\(1\)=0h\(0\)=h\(1\)=0, the maximum ofhhon\[0,1\]\[0,1\]is

maxx∈\[0,1\]⁡h​\(x\)=h​\(e−2\)=4e2\.\\max\_\{x\\in\[0,1\]\}h\(x\)=h\(e^\{\-2\}\)=\\frac\{4\}\{e^\{2\}\}\.Therefore, for every\(p1,p2\)∈Δ2\(p\_\{1\},p\_\{2\}\)\\in\\Delta\_\{2\},

F2​\(p\)=h​\(p1\)\+h​\(p2\)≤8e2\.F\_\{2\}\(p\)=h\(p\_\{1\}\)\+h\(p\_\{2\}\)\\leq\\frac\{8\}\{e^\{2\}\}\.Also,

e\>1\+1\+12\+16=83,e\>1\+1\+\\frac\{1\}\{2\}\+\\frac\{1\}\{6\}=\\frac\{8\}\{3\},so

8e2<8\(8/3\)2=98<54\.\\frac\{8\}\{e^\{2\}\}<\\frac\{8\}\{\(8/3\)^\{2\}\}=\\frac\{9\}\{8\}<\\frac\{5\}\{4\}\.On the other hand, sincee<4e<4, we havee<2\\sqrt\{e\}<2, hence

log⁡2\>12,\\log 2\>\\frac\{1\}\{2\},and therefore

1\+\(log⁡2\)2\>1\+14=54\.1\+\(\\log 2\)^\{2\}\>1\+\\frac\{1\}\{4\}=\\frac\{5\}\{4\}\.Combining the last two inequalities,

F2​\(p\)≤8e2<54<1\+\(log⁡2\)2\.F\_\{2\}\(p\)\\leq\\frac\{8\}\{e^\{2\}\}<\\frac\{5\}\{4\}<1\+\(\\log 2\)^\{2\}\.So the result holds forn=2n=2as well\.

Step 2: induction hypothesis\.

Assume now thatn≥3n\\geq 3, and that the statement has already been proved for all dimensions1,2,…,n−11,2,\\dots,n\-1\.

Letp⋆∈Δnp^\{\\star\}\\in\\Delta\_\{n\}be a maximizer ofFnF\_\{n\}onΔn\\Delta\_\{n\}\. Such a maximizer exists becauseΔn\\Delta\_\{n\}is compact andFnF\_\{n\}is continuous\.

We distinguish two cases\.

Case 1:p⋆p^\{\\star\}lies on the boundary ofΔn\\Delta\_\{n\}\.

Then at least one coordinate ofp⋆p^\{\\star\}is zero\. Letmmbe the number of positive coordinates ofp⋆p^\{\\star\}\. Then1≤m<n1\\leq m<n\. After removing the zero coordinates, we obtain a vectorq∈Δmq\\in\\Delta\_\{m\}such that

Fn​\(p⋆\)=Fm​\(q\),F\_\{n\}\(p^\{\\star\}\)=F\_\{m\}\(q\),because the zero coordinates contribute nothing to the sum\.

By the induction hypothesis,

Fn​\(p⋆\)=Fm​\(q\)≤1\+\(log⁡m\)2≤1\+\(log⁡n\)2\.F\_\{n\}\(p^\{\\star\}\)=F\_\{m\}\(q\)\\leq 1\+\(\\log m\)^\{2\}\\leq 1\+\(\\log n\)^\{2\}\.So the claim holds in this case\.

Case 2:p⋆p^\{\\star\}lies in the interior ofΔn\\Delta\_\{n\}\.

Then every coordinate ofp⋆p^\{\\star\}is strictly positive, so we may apply the method of Lagrange multipliers to maximizeFnF\_\{n\}under the constraint∑i=1npi=1\\sum\_\{i=1\}^\{n\}p\_\{i\}=1\. Thus there existsλ∈ℝ\\lambda\\in\\mathbb\{R\}such that for everyi=1,…,ni=1,\\dots,n,

∂∂pi​Fn​\(p⋆\)=λ\.\\frac\{\\partial\}\{\\partial p\_\{i\}\}F\_\{n\}\(p^\{\\star\}\)=\\lambda\.Since

dd​x​\[x​\(log⁡x\)2\]=\(log⁡x\)2\+2​log⁡x,\\frac\{d\}\{dx\}\\bigl\[x\(\\log x\)^\{2\}\\bigr\]=\(\\log x\)^\{2\}\+2\\log x,we get

\(log⁡pi⋆\)2\+2​log⁡pi⋆=λfor all​i=1,…,n\.\(\\log p\_\{i\}^\{\\star\}\)^\{2\}\+2\\log p\_\{i\}^\{\\star\}=\\lambda\\qquad\\text\{for all \}i=1,\\dots,n\.
Now suppose two positive numbersa,ba,bsatisfy

\(log⁡a\)2\+2​log⁡a=\(log⁡b\)2\+2​log⁡b\.\(\\log a\)^\{2\}\+2\\log a=\(\\log b\)^\{2\}\+2\\log b\.Subtracting the two sides gives

\(log⁡a−log⁡b\)​\(log⁡a\+log⁡b\+2\)=0\.\(\\log a\-\\log b\)\(\\log a\+\\log b\+2\)=0\.Hence either

or

log⁡a\+log⁡b\+2=0,i\.e\.a​b=e−2\.\\log a\+\\log b\+2=0,\\qquad\\text\{i\.e\.\}\\qquad ab=e^\{\-2\}\.
It follows that the coordinates ofp⋆p^\{\\star\}can take at most two distinct values\. So there are two possibilities:

*\(i\) All coordinates are equal\.*Then

pi⋆=1nfor all​i,p\_\{i\}^\{\\star\}=\\frac\{1\}\{n\}\\qquad\\text\{for all \}i,and therefore

Fn​\(p⋆\)=n⋅1n​\(log⁡\(1/n\)\)2=\(log⁡n\)2≤1\+\(log⁡n\)2\.F\_\{n\}\(p^\{\\star\}\)=n\\cdot\\frac\{1\}\{n\}\\bigl\(\\log\(1/n\)\\bigr\)^\{2\}=\(\\log n\)^\{2\}\\leq 1\+\(\\log n\)^\{2\}\.
*\(ii\) There are exactly two distinct values\.*Sayaaoccurskktimes andbboccursn−kn\-ktimes, where1≤k≤n−11\\leq k\\leq n\-1,a≠ba\\neq b, and necessarily

Since the coordinates sum to11,

k​a\+\(n−k\)​b=1\.ka\+\(n\-k\)b=1\.Substitutingb=e−2/ab=e^\{\-2\}/ainto this equation yields

k​a\+\(n−k\)​e−2a=1,ka\+\(n\-k\)\\frac\{e^\{\-2\}\}\{a\}=1,or equivalently

k​a2−a\+\(n−k\)​e−2=0\.ka^\{2\}\-a\+\(n\-k\)e^\{\-2\}=0\.This is a quadratic equation inaa, whose discriminant is

Δ=1−4​k​\(n−k\)​e−2\.\\Delta=1\-4k\(n\-k\)e^\{\-2\}\.But for1≤k≤n−11\\leq k\\leq n\-1we have

k​\(n−k\)≥n−1,k\(n\-k\)\\geq n\-1,so, sincen≥3n\\geq 3,

Δ≤1−4​\(n−1\)​e−2≤1−8​e−2<0\.\\Delta\\leq 1\-4\(n\-1\)e^\{\-2\}\\leq 1\-8e^\{\-2\}<0\.This is impossible\. Hence case \(ii\) cannot occur\.

Therefore the only interior maximizer is the uniform distribution, and in that case

Fn​\(p⋆\)=\(log⁡n\)2≤1\+\(log⁡n\)2\.F\_\{n\}\(p^\{\\star\}\)=\(\\log n\)^\{2\}\\leq 1\+\(\\log n\)^\{2\}\.
Since every maximizer satisfies the desired bound, we conclude that for everyp∈Δnp\\in\\Delta\_\{n\},

Fn​\(p\)=∑i=1npi​\(log⁡pi\)2≤1\+\(log⁡n\)2\.F\_\{n\}\(p\)=\\sum\_\{i=1\}^\{n\}p\_\{i\}\(\\log p\_\{i\}\)^\{2\}\\leq 1\+\(\\log n\)^\{2\}\.This completes the proof\. ∎