Regularized Emphatic Temporal-Difference Learning: Stability under Constant Stepsizes

# Regularized Emphatic Temporal-Difference Learning: Stability under Constant Stepsizes
Source: [https://arxiv.org/html/2609.19170](https://arxiv.org/html/2609.19170)
Xingguo Chen, Zhaohui Wu, Jinguo Ye, Chao Li, Shangdong Yang, Guang Yang, Skylar Liang, and Wenhao Wang

Xingguo Chen††thanks:Corresponding author: chenxg@njupt\.edu\.cn\.Affiliation:Jiangsu Key Laboratory of Big Data Security & Intelligent ProcessingNanjing University of Posts & Telecommunications, Nanjing 210023, ChinaEmail:[chenxg@njupt\.edu\.cn](mailto:[email protected])Zhaohui WuAffiliation:Jiangsu Key Laboratory of Big Data Security & Intelligent ProcessingNanjing University of Posts & Telecommunications, Nanjing 210023, ChinaEmail:[liangtianyu@microsoft\.com](mailto:[email protected])Jinguo YeAffiliation:Jiangsu Key Laboratory of Big Data Security & Intelligent ProcessingNanjing University of Posts & Telecommunications, Nanjing 210023, ChinaChao LiAffiliation:Jiangsu Key Laboratory of Big Data Security & Intelligent ProcessingNanjing University of Posts & Telecommunications, Nanjing 210023, ChinaShangdong YangAffiliation:Jiangsu Key Laboratory of Big Data Security & Intelligent ProcessingNanjing University of Posts & Telecommunications, Nanjing 210023, ChinaSkylar LiangAffiliation:Microsoft Corporation, One Microsoft Way, Redmond, WA 98052, USAWenhao WangAffiliation:College of Electronic Countermeasure, National University of Defense TechnologyHefei 230037, China

###### Abstract

Emphatic temporal\-difference learning \(ETD\) stabilizes the expected off\-policy TD update and changes its projection geometry, but neither property determines constant\-stepsize sampled dynamics\. We construct an ergodic two\-state counterexample in which the ETD mean map contracts while the sampled product has a positive top Lyapunov exponent\. Regenerative\-cycle analysis separates this sign from the infinite variance of the follow\-on trace\. We introduce regularized emphatic TD \(RETD\), a normalized first\-order post\-shock repair that leaves the trace and importance ratios unchanged, stores the emphatic TD signal in a leaky scalar state, and releases a delayed correction\. RETD’s raw equilibrium is an affine shift of the ETD equilibrium; single\- and two\-regularization readouts recover the ETD fixed point exactly\. We prove almost\-sure convergence for harmonic diminishing stepsizes and a conditional constant\-stepsize moment\-contraction result from a Markovian random\-product bound\. RETD has certified negative exponents on the two\-state construction and one Baird point, whereas the positive Baird ETD sign remains numerical\. Paired 10,000\-run experiments validate both separations, fixed\-point recovery, a nonmonotone stability region, and task dependence\. RETD changes post\-shock dynamics; it does not reduce the shared follow\-on\-trace variance\.

††heading:2026 1\-††shortheadings:Regularized Emphatic Temporal\-Difference Learning / Chen et al\.††firstpage:1###### keywords

off\-policy learning, emphatic temporal\-difference learning, constant stepsize, random matrix products, fixed\-point recovery

## 1Introduction

Off\-policy temporal\-difference learning separates at least two questions: whether the expected recursion is stable and whether its fixed point is a good target\-policy approximation\. Baird’s counterexample showed that ordinary off\-policy TD can fail at the mean\-update level\([Baird, 1995](https://arxiv.org/html/2609.19170#bib.bib1)\); Gradient\-TD methods instead stabilize projected objectives\([Sutton et al\., 2008](https://arxiv.org/html/2609.19170#bib.bib28);[Sutton et al\., 2009](https://arxiv.org/html/2609.19170#bib.bib29)\)\. Oblique\-projection analyses further show that convergence need not imply a good approximation\([Scherrer, 2010](https://arxiv.org/html/2609.19170#bib.bib26);[Kolter, 2011](https://arxiv.org/html/2609.19170#bib.bib18)\)\.

ETD addresses both issues through emphatic state weighting: its stationary expected update is positive stable and its projected Bellman equation uses emphatic weights\([Sutton et al\., 2016](https://arxiv.org/html/2609.19170#bib.bib30);[Hallak et al\., 2016](https://arxiv.org/html/2609.19170#bib.bib14)\)\. Diminishing\- stepsize convergence and constrained constant\-step weak\-convergence results are known\([Yu, 2015](https://arxiv.org/html/2609.19170#bib.bib24);[Yu, 2016](https://arxiv.org/html/2609.19170#bib.bib25)\), but these results do not settle the unconstrained sampled recursion at a fixed nonzero stepsize\. A finite\-mean, infinite\-variance follow\-on trace identifies a shock source, not the sign of the associated random\-matrix product\.

We study the gap in a fully specified two\-state example: the target action has behavior probability0\.30\.3, the ratios are10/310/3and zero,γ=0\.9\\gamma=0\.9, the scalar features are11and22, andα=0\.1\\alpha=0\.1\. The ETD mean multiplier is0\.6840\.684, while regenerative analysis proves a strictly positive scalar top Lyapunov exponent\. We call this the*mean\-to\-path stability gap*\.

The same construction identifies the repair mechanism\. A long target\-action run creates a large emphatic shock, but the following zero\-ratio ETD update is the identity\. RETD adds one leaky scalar state, so later samples can release a delayed correction even when the current ratio is zero; the trace and ratios themselves remain unchanged\.

Section[3](https://arxiv.org/html/2609.19170#S3)derives RETD from seven design constraints\. Positive leakage removes thec=0c=0conserved mode, while the equilibrium analysis gives single\-ccand two\-ccrecovery of the ETD prediction, including singular and ill\-conditioned cases\. We then prove the diminishing\-stepsize result and instantiate the constant\-stepsize random\-product theorem of[Durmus et al\. \(2021\)](https://arxiv.org/html/2609.19170#bib.bib22); explicit counterexample points are handled by regenerative or projective certificates\.

Our contributions are threefold\.*\(1\)*We give an exact two\-state mean\-to\-path separation: ETD is strictly mean stable but has a positive top Lyapunov exponent, while RETD on the same task and stepsize has a certified negative exponent\.*\(2\)*We derive RETD as a one\-scalar feedback modification of the post\-shock dynamics and characterize its equilibrium, including single\-ccand two\-ccrecovery of the ETD fixed point and the associated singularity and conditioning conditions\.*\(3\)*We prove diminishing\- and conditional constant\-stepsize stochastic stability results, give a rigorous RETD certificate and numerical ETD diagnosis on Baird’s counterexample, and evaluate mechanism, recovery, stability regions, baselines, cost, and adverse tasks\.

The paper follows this chain: Section[2](https://arxiv.org/html/2609.19170#S2)defines the stability notions and proves the ETD half of the two\-state separation; Section[3](https://arxiv.org/html/2609.19170#S3)derives RETD and its recovery identities; and Section[4](https://arxiv.org/html/2609.19170#S4)gives the two stochastic theorems\. Section[5](https://arxiv.org/html/2609.19170#S5)extends the mean/product analysis to Baird’s system, Section[6](https://arxiv.org/html/2609.19170#S6)reports the paired evaluations and adverse tasks, and the final sections give the literature position and limitations\.

## 2Preliminaries and a Constant\-Stepsize ETD Counterexample

We begin with the ordinary and emphatic projected systems and distinguish fixed\-point quality from three notions of dynamical stability\. A two\-state counterexample then shows why mean stability of ETD does not settle the constant\-stepsize sample\-path question\.

### 2\.1Off\-Policy Linear Prediction

Consider a finite Markov decision process with state space𝒮\\mathcal\{S\}, action space𝒜\\mathcal\{A\}, target policyπ\\pi, behavior policyμ\\mu, and a constant discountγ∈\[0,1\)\\gamma\\in\[0,1\)\. The behavior chain is irreducible and aperiodic with stationary distribution𝒅μ\\bm\{d\}\_\{\\mu\}, and the target policy is covered by the behavior policy\. At timett, defineρt=π⁡\(At∣St\)/μ⁡\(At∣St\)\\rho\_\{t\}=\\pi\(A\_\{t\}\\mid S\_\{t\}\)/\\mu\(A\_\{t\}\\mid S\_\{t\}\)andv𝜽​\(s\)=ϕ​\(s\)𝖳​𝜽v\_\{\\bm\{\\theta\}\}\(s\)=\\bm\{\\phi\}\(s\)^\{\\mathsf\{T\}\}\\bm\{\\theta\}, where the rows of𝚽\\bm\{\\Phi\}are the feature vectors\. The TD error isδt=Rt\+1\+γ​ϕ​\(St\+1\)𝖳​𝜽t−ϕ​\(St\)𝖳​𝜽t\\delta\_\{t\}=R\_\{t\+1\}\+\\gamma\\bm\{\\phi\}\(S\_\{t\+1\}\)^\{\\mathsf\{T\}\}\\bm\{\\theta\}\_\{t\}\-\\bm\{\\phi\}\(S\_\{t\}\)^\{\\mathsf\{T\}\}\\bm\{\\theta\}\_\{t\}\. Let𝑷π\\bm\{P\}\_\{\\pi\}and𝒓π\\bm\{r\}\_\{\\pi\}denote the target\-policy transition matrix and expected one\-step reward\. Then𝒗π=\(𝑰−γ​𝑷π\)−1​𝒓π\\bm\{v\}\_\{\\pi\}=\(\\bm\{I\}\-\\gamma\\bm\{P\}\_\{\\pi\}\)^\{\-1\}\\bm\{r\}\_\{\\pi\}\.

Ordinary importance\-sampled TD\(0\) has stationary mean system

𝑨μ=𝚽𝖳​𝑫μ​\(𝑰−γ​𝑷π\)​𝚽,𝒃μ=𝚽𝖳​𝑫μ​𝒓π,\\bm\{A\}\_\{\\mu\}=\\bm\{\\Phi\}^\{\\mathsf\{T\}\}\\bm\{D\}\_\{\\mu\}\(\\bm\{I\}\-\\gamma\\bm\{P\}\_\{\\pi\}\)\\bm\{\\Phi\},\\qquad\\bm\{b\}\_\{\\mu\}=\\bm\{\\Phi\}^\{\\mathsf\{T\}\}\\bm\{D\}\_\{\\mu\}\\bm\{r\}\_\{\\pi\},where𝑫μ=diag⁡\(𝒅μ\)\\bm\{D\}\_\{\\mu\}=\\diag\(\\bm\{d\}\_\{\\mu\}\)\. Baird’s counterexample shows that𝑨μ\\bm\{A\}\_\{\\mu\}need not be positive stable, so the expected recursion itself can diverge\([Baird, 1995](https://arxiv.org/html/2609.19170#bib.bib1)\)\.

### 2\.2The Fixed Point Is a Separate Question

Suppose an algorithm converges to𝜽μ=𝑨μ−1​𝒃μ\\bm\{\\theta\}\_\{\\mu\}=\\bm\{A\}\_\{\\mu\}^\{\-1\}\\bm\{b\}\_\{\\mu\}\. Its prediction is an oblique projection whose quality depends on the sampling and projection geometry; convergence therefore does not imply closeness to𝒗π\\bm\{v\}\_\{\\pi\}\([Scherrer, 2010](https://arxiv.org/html/2609.19170#bib.bib26);[Kolter, 2011](https://arxiv.org/html/2609.19170#bib.bib18)\)\.

Reachability and quality are therefore separate questions\. Gradient\-TD methods stabilize projected objectives, but do not automatically replace behavior weighting by the emphatic weighting relevant to ETD\.

ETD\(0\), with unit interest, uses the follow\-on traceFt\+1=1\+γ​ρt​FtF\_\{t\+1\}=1\+\\gamma\\rho\_\{t\}F\_\{t\}and update𝜽t\+1=𝜽t\+αt​Ft​ρt​δt​ϕ​\(St\)\\bm\{\\theta\}\_\{t\+1\}=\\bm\{\\theta\}\_\{t\}\+\\alpha\_\{t\}F\_\{t\}\\rho\_\{t\}\\delta\_\{t\}\\bm\{\\phi\}\(S\_\{t\}\)\. Its expected emphatic weights and linear system are

𝒇\\displaystyle\\bm\{f\}=\(𝑰−γ​𝑷π𝖳\)−1​𝒅μ,\\displaystyle=\(\\bm\{I\}\-\\gamma\\bm\{P\}\_\{\\pi\}^\{\\mathsf\{T\}\}\)^\{\-1\}\\bm\{d\}\_\{\\mu\},𝑭\\displaystyle\\bm\{F\}=diag⁡\(𝒇\),\\displaystyle=\\diag\(\\bm\{f\}\),𝑨\\displaystyle\\bm\{A\}=𝚽𝖳​𝑭​\(𝑰−γ​𝑷π\)​𝚽,\\displaystyle=\\bm\{\\Phi\}^\{\\mathsf\{T\}\}\\bm\{F\}\(\\bm\{I\}\-\\gamma\\bm\{P\}\_\{\\pi\}\)\\bm\{\\Phi\},𝒃\\displaystyle\\bm\{b\}=𝚽𝖳​𝑭​𝒓π\.\\displaystyle=\\bm\{\\Phi\}^\{\\mathsf\{T\}\}\\bm\{F\}\\bm\{r\}\_\{\\pi\}\.Under standard positive\-interest and feature conditions,𝑨\\bm\{A\}is positive stable and𝜽E=𝑨−1​𝒃\\bm\{\\theta\}\_\{E\}=\\bm\{A\}^\{\-1\}\\bm\{b\}solves the emphatically weighted projected Bellman equation\([Sutton et al\., 2016](https://arxiv.org/html/2609.19170#bib.bib30);[Hallak et al\., 2016](https://arxiv.org/html/2609.19170#bib.bib14)\)\. Thus a path\-stability modification must state how this fixed\-point information is preserved\.

### 2\.3Three Notions of Stability

The phrase “ETD is stable” is incomplete unless the asymptotic regime and random object are specified\. We distinguish three notions\.

*Mean Stability\.*With the trace process in stationarity and the parameter frozen, the expected ETD drift is𝜽˙=𝒃−𝑨​𝜽\\dot\{\\bm\{\\theta\}\}=\\bm\{b\}\-\\bm\{A\}\\bm\{\\theta\}\. The ODE is stable when−𝑨\-\\bm\{A\}is Hurwitz; for constantα\\alpha,ρ⁡\(𝑰−α​𝑨\)<1\\rho\(\\bm\{I\}\-\\alpha\\bm\{A\}\)<1is the corresponding mean\-map diagnostic\. Neither is an expectation of a sampled product\.

*Diminishing\-Stepsize Stochastic Stability\.*For a Robbins–Monro schedule withαt↓0\\alpha\_\{t\}\\downarrow 0, one asks whether the actual iterates remain bounded and converge almost surely\. This does not imply stability whenαt≡α\>0\\alpha\_\{t\}\\equiv\\alpha\>0\.

*Constant\-Stepsize Stochastic Stability\.*For the constant scheduleαt≡α\>0\\alpha\_\{t\}\\equiv\\alpha\>0, subtract an equilibrium and write the homogeneous error recursion as𝒆t\+1=𝑴t​\(α\)​𝒆t\\bm\{e\}\_\{t\+1\}=\\bm\{M\}\_\{t\}\(\\alpha\)\\bm\{e\}\_\{t\}\. The relevant long\-run object is the ordered random product𝑴t−1⋯𝑴0\\bm\{M\}\_\{t\-1\}\\cdots\\bm\{M\}\_\{0\}\. On the identifiable prediction space, define the upper top Lyapunov exponent

λ\(α\)=lim supt→∞1tlog∥𝑴t−1\(α\)⋯𝑴0\(α\)∥\.\\lambda\(\\alpha\)=\\limsup\_\{t\\to\\infty\}\\frac\{1\}\{t\}\\log\\\|\\bm\{M\}\_\{t\-1\}\(\\alpha\)\\cdots\\bm\{M\}\_\{0\}\(\\alpha\)\\\|\.\(1\)A negative exponent gives exponential forgetting of initial perturbations and, for an affine recursion, generally convergence to a stationary causal process rather than a deterministic point\. A positive exponent gives an expanding top direction; in the scalar counterexample every nonzero initial error grows exponentially\.

These notions are not linked by a simple expectation identity\. In particular,𝔼⁡\[𝑴t​𝒆t\]≠𝔼⁡\[𝑴t\]​𝔼​\[𝒆t\]\\mathbb\{E\}\[\\bm\{M\}\_\{t\}\\bm\{e\}\_\{t\}\]\\neq\\mathbb\{E\}\[\\bm\{M\}\_\{t\}\]\\mathbb\{E\}\[\\bm\{e\}\_\{t\}\]becauseFtF\_\{t\}, the state, and the current iterate are correlated\. Infinite variance ofFtF\_\{t\}warns that shocks are severe, but does not determine the sign of Equation[1](https://arxiv.org/html/2609.19170#S2.E1); that sign must be analyzed from the product\.

The matrices𝑨\\bm\{A\}and𝑮c\\bm\{G\}\_\{c\}used later are stationary mean matrices, whereas sampled factors generate ordered random products\. A spectral\-radius calculation for a mean matrix cannot replace a Lyapunov\-exponent calculation for its sampled product\.

### 2\.4A Two\-State Counterexample

Let𝒮=\{s1,s2\}\\mathcal\{S\}=\\\{s\_\{1\},s\_\{2\}\\\}and𝒜=\{a0,aπ\}\\mathcal\{A\}=\\\{a\_\{0\},a\_\{\\pi\}\\\}\. With rows denoting the current state and columns the next state, the action\-conditioned transition matrices are

𝑷a0=\[1010\],𝑷aπ=\[0101\]\.\\bm\{P\}^\{a\_\{0\}\}=\\begin\{bmatrix\}1&0\\\\ 1&0\\end\{bmatrix\},\\qquad\\bm\{P\}^\{a\_\{\\pi\}\}=\\begin\{bmatrix\}0&1\\\\ 0&1\\end\{bmatrix\}\.
s1s\_\{1\}ϕ⁡\(s1\)=1\\phi\(s\_\{1\}\)=1s2s\_\{2\}ϕ⁡\(s2\)=2\\phi\(s\_\{2\}\)=2aπa\_\{\\pi\}aπa\_\{\\pi\}a0a\_\{0\}a0a\_\{0\}π⁡\(aπ∣s\)=1\\pi\(a\_\{\\pi\}\\mid s\)=1;μ⁡\(aπ∣s\)=\.3\\mu\(a\_\{\\pi\}\\mid s\)=\.3,μ⁡\(a0∣s\)=\.7\\mu\(a\_\{0\}\\mid s\)=\.7Rt\+1=0R\_\{t\+1\}=0,γ=\.9\\gamma=\.9,α=\.1\\alpha=\.1Figure 1:The new two\-state counterexample\. Actionaπa\_\{\\pi\}sends either state tos2s\_\{2\}, whereasa0a\_\{0\}sends either state tos1s\_\{1\}\. The target policy always choosesaπa\_\{\\pi\}and the behavior policy choosesaπa\_\{\\pi\}with probability\.3\.3\. Solid blue and dashed gray arrows distinguish the two actions and remain legible in grayscale\.The target policy always selectsaπa\_\{\\pi\}\. In either state the behavior policy selectsaπa\_\{\\pi\}with probabilityp=3/10p=3/10anda0a\_\{0\}with probability7/107/10\. Therefore

𝑷μ=\[7/103/107/103/10\],𝒅μ=\[7/103/10\],\\bm\{P\}\_\{\\mu\}=\\begin\{bmatrix\}7/10&3/10\\\\ 7/10&3/10\\end\{bmatrix\},\\qquad\\bm\{d\}\_\{\\mu\}=\\begin\{bmatrix\}7/10\\\\ 3/10\\end\{bmatrix\},andρ⁡\(aπ\)=10/3\\rho\(a\_\{\\pi\}\)=10/3,ρ⁡\(a0\)=0\\rho\(a\_\{0\}\)=0\. We setRt\+1=0R\_\{t\+1\}=0,γ=9/10\\gamma=9/10,α=1/10\\alpha=1/10,ϕ⁡\(s1\)=1\\phi\(s\_\{1\}\)=1, andϕ⁡\(s2\)=2\\phi\(s\_\{2\}\)=2\.

The two states remain distinct: they have different transition destinations under the two actions and different feature values\. A scalar parameter merely restricts the two predictions to\(θ,2​θ\)\(\\theta,2\\theta\); it does not merge the states\. Zero reward makes the true value𝒗π=𝟎\\bm\{v\}\_\{\\pi\}=\\bm\{0\}, which is exactly represented byθ∗=0\\theta^\{\*\}=0\. The affine term then vanishes, so the homogeneous stability question can be studied without approximation error\.

### 2\.5The Trace Has a Finite Mean and Infinite Variance

LetKtK\_\{t\}be the number of consecutive preceding target actions\. Independence of behavior actions gives, in stationarity,ℙ⁡\(Kt=k\)=\(7/10\)​\(3/10\)k\\mathbb\{P\}\(K\_\{t\}=k\)=\(7/10\)\(3/10\)^\{k\},k=0,1,…k=0,1,\\ldots\. After ana0a\_\{0\}action the next trace is one\. Each target action multiplies the old trace byγ​ρ=\(9/10\)​\(10/3\)=3\\gamma\\rho=\(9/10\)\(10/3\)=3before adding one\. HenceFk=1\+3\+⋯\+3k=\(3k\+1−1\)/2F\_\{k\}=1\+3\+\\cdots\+3^\{k\}=\(3^\{k\+1\}\-1\)/2\. For everyr\>0r\>0, comparison with3k3^\{k\}gives𝔼⁡\[Ftr\]<∞\\mathbb\{E\}\[F\_\{t\}^\{r\}\]<\\inftyif and only if\(3/10\)​3r<1\(3/10\)3^\{r\}<1\. Since\(3/10\)​3=0\.9<1\(3/10\)3=0\.9<1but\(3/10\)​32=2\.7\>1\(3/10\)3^\{2\}=2\.7\>1, so𝔼⁡\[Ft\]<∞\\mathbb\{E\}\[F\_\{t\}\]<\\inftybut𝔼⁡\[Ft2\]=∞\\mathbb\{E\}\[F\_\{t\}^\{2\}\]=\\infty\. The first\-moment recursion closes because𝔼⁡\[γ​ρt\]=γ<1\\mathbb\{E\}\[\\gamma\\rho\_\{t\}\]=\\gamma<1, whereas the second\-moment coefficient is𝔼⁡\[\(γ​ρt\)2\]=γ2/p=2\.7\>1\\mathbb\{E\}\[\(\\gamma\\rho\_\{t\}\)^\{2\}\]=\\gamma^\{2\}/p=2\.7\>1\. This identifies the heavy\-tailed shock channel but does not prove product expansion\.

### 2\.6The ETD Mean Recursion Is Strictly Stable

Here𝑷π=𝑷aπ\\bm\{P\}\_\{\\pi\}=\\bm\{P\}^\{a\_\{\\pi\}\}\. Withϕ=\(1,2\)𝖳\\bm\{\\phi\}=\(1,2\)^\{\\mathsf\{T\}\}, the emphatic weight vector and scalar ETD coefficient are

𝒇\\displaystyle\\bm\{f\}=\(𝑰−γ​𝑷π𝖳\)−1​𝒅μ=\[7/1093/10\],\\displaystyle=\(\\bm\{I\}\-\\gamma\\bm\{P\}\_\{\\pi\}^\{\\mathsf\{T\}\}\)^\{\-1\}\\bm\{d\}\_\{\\mu\}=\\begin\{bmatrix\}7/10\\\\ 93/10\\end\{bmatrix\},A\\displaystyle A=ϕ𝖳​diag⁡\(𝒇\)​\(𝑰−γ​𝑷π\)​ϕ\\displaystyle=\\bm\{\\phi\}^\{\\mathsf\{T\}\}\\diag\(\\bm\{f\}\)\(\\bm\{I\}\-\\gamma\\bm\{P\}\_\{\\pi\}\)\\bm\{\\phi\}=1⋅710​\(−45\)\+2⋅9310​\(15\)=7925\.\\displaystyle=1\\cdot\\frac\{7\}\{10\}\\left\(\-\\frac\{4\}\{5\}\\right\)\+2\\cdot\\frac\{93\}\{10\}\\left\(\\frac\{1\}\{5\}\\right\)=\\frac\{79\}\{25\}\.The deterministic mean error multiplier is

1−α​A=1−110​7925=171250=0\.684\.1\-\\alpha A=1\-\\frac\{1\}\{10\}\\frac\{79\}\{25\}=\\frac\{171\}\{250\}=0\.684\.\(2\)Thus the mean recursion has a strict contraction margin\.

### 2\.7The Constant\-Stepsize ETD Product Nevertheless Expands

Use cycles ofLLtarget actions followed by onea0a\_\{0\}action\. Thenℙ⁡\(L=ℓ\)=\(7/10\)​\(3/10\)ℓ\\mathbb\{P\}\(L=\\ell\)=\(7/10\)\(3/10\)^\{\\ell\}and𝔼⁡\[L\+1\]=10/7\\mathbb\{E\}\[L\+1\]=10/7\. The first target action moves froms1s\_\{1\}tos2s\_\{2\}; every later target action remains ins2s\_\{2\}\. Their scalar error multipliers are

m0\\displaystyle m\_\{0\}=1\+α​F0​103​\(γ​ϕ​\(s2\)−ϕ⁡\(s1\)\)​ϕ​\(s1\)=1915,\\displaystyle=1\+\\alpha F\_\{0\}\\frac\{10\}\{3\}\(\\gamma\\phi\(s\_\{2\}\)\-\\phi\(s\_\{1\}\)\)\\phi\(s\_\{1\}\)=\\frac\{19\}\{15\},\(3\)mk\\displaystyle m\_\{k\}=1\+α​Fk​103​\(γ​ϕ​\(s2\)−ϕ⁡\(s2\)\)​ϕ​\(s2\)=1−2​Fk15=16−3k\+115,k≥1\.\\displaystyle=1\+\\alpha F\_\{k\}\\frac\{10\}\{3\}\(\\gamma\\phi\(s\_\{2\}\)\-\\phi\(s\_\{2\}\)\)\\phi\(s\_\{2\}\)=1\-\\frac\{2F\_\{k\}\}\{15\}=\\frac\{16\-3^\{k\+1\}\}\{15\},\\quad k\\geq 1\.\(4\)The finala0a\_\{0\}action hasρ=0\\rho=0, so ETD applies the identity\. The cycle factor isCL=∏k=0L−1mkC\_\{L\}=\\prod\_\{k=0\}^\{L\-1\}m\_\{k\}, withC0=1C\_\{0\}=1\. Its first four nonempty values areC1=19/15C\_\{1\}=19/15,C2=133/225C\_\{2\}=133/225andC3=−1463/3375C\_\{3\}=\-1463/3375,C4=19019/10125C\_\{4\}=19019/10125\. Direct outward\-rounded evaluation gives

∑L=14710​\(310\)L​log⁡\|CL\|\>0\.004295228922459683\.\\sum\_\{L=1\}^\{4\}\\frac\{7\}\{10\}\\left\(\\frac\{3\}\{10\}\\right\)^\{L\}\\log\|C\_\{L\}\|\>0\.004295228922459683\.Moreover\|C4\|\>1\|C\_\{4\}\|\>1and all factors appended afterm3m\_\{3\}have magnitude above one, so omitted terms withL≥5L\\geq 5are nonnegative\.

For the full\-time limit, there are constantsa0,a1<∞a\_\{0\},a\_\{1\}<\\inftysuch that\|log⁡\|mk\|\|≤a0\+a1​k\|\\log\|m\_\{k\}\|\|\\leq a\_\{0\}\+a\_\{1\}kfork≥0k\\geq 0, because none of the factors in Equations[3](https://arxiv.org/html/2609.19170#S2.E3)–[4](https://arxiv.org/html/2609.19170#S2.E4)vanishes and their logarithms grow at most linearly inkk\. Consequently,

\|log⁡\|CL\|\|≤a0​L\+a12​L​\(L−1\)\.\|\\log\|C\_\{L\}\|\|\\leq a\_\{0\}L\+\\frac\{a\_\{1\}\}\{2\}L\(L\-1\)\.\(5\)The geometric law ofLLtherefore gives𝔼\|log⁡\|CL\|\|<∞\\mathbb\{E\}\|\\log\|C\_\{L\}\|\|<\\infty\.

LetL1,L2,…L\_\{1\},L\_\{2\},\\ldotsbe the successive iid run lengths, putτn=Ln\+1\\tau\_\{n\}=L\_\{n\}\+1, and letTN=∑n=1NτnT\_\{N\}=\\sum\_\{n=1\}^\{N\}\\tau\_\{n\}be theNNth cycle endpoint\. For every nonzero initial error, the strong law gives

1TN​log⁡\|θTNθ0\|=NTN​1N​∑n=1Nlog⁡\|CLn\|⟶𝔼​log⁡\|CL\|𝔼⁡\[L\+1\]a\.s\.\\frac\{1\}\{T\_\{N\}\}\\log\\left\|\\frac\{\\theta\_\{T\_\{N\}\}\}\{\\theta\_\{0\}\}\\right\|=\\frac\{N\}\{T\_\{N\}\}\\frac\{1\}\{N\}\\sum\_\{n=1\}^\{N\}\\log\|C\_\{L\_\{n\}\}\|\\longrightarrow\\frac\{\\mathbb\{E\}\\log\|C\_\{L\}\|\}\{\\mathbb\{E\}\[L\+1\]\}\\quad\\text\{a\.s\.\}The endpoint limit is also the full\-time limit\. IfTN≤t<TN\+1T\_\{N\}\\leq t<T\_\{N\+1\}, the unfinished part is bounded by Equation[5](https://arxiv.org/html/2609.19170#S2.E5)withL=LN\+1L=L\_\{N\+1\}\. For everyε\>0\\varepsilon\>0,∑n=1∞ℙ⁡\(Ln2\>ε​n\)<∞\\sum\_\{n=1\}^\{\\infty\}\\mathbb\{P\}\(L\_\{n\}^\{2\}\>\\varepsilon n\)<\\infty; the geometric tail and Borel–Cantelli giveLn2/n→0L\_\{n\}^\{2\}/n\\to 0almost surely, whileTN/N→𝔼⁡\[L\+1\]\>0T\_\{N\}/N\\to\\mathbb\{E\}\[L\+1\]\>0\. The unfinished\-cycle contribution is thereforeo⁡\(t\)o\(t\), and the per\-transition exponent is

λE=𝔼​log⁡\|CL\|𝔼⁡\[L\+1\]\>0\.003006660245721778\>0\.\\lambda\_\{E\}=\\frac\{\\mathbb\{E\}\\log\|C\_\{L\}\|\}\{\\mathbb\{E\}\[L\+1\]\}\>0\.003006660245721778\>0\.\(6\)
Equations[2](https://arxiv.org/html/2609.19170#S2.E2)and[6](https://arxiv.org/html/2609.19170#S2.E6)give the separation: the mean multiplier is171/250171/250, yet every nonzero scalar ETD error has almost\-sure logarithmic growth above0\.0030066602457217780\.003006660245721778\. The interpolation above makes this a complete\-trajectory statement, not only a cycle\-endpoint statement\.

The trace\-moment calculation identifies the shock source, whereas the cycle calculation proves product expansion\. The finalρ=0\\rho=0identity cannot respond to the preceding target\-action run; Section[3](https://arxiv.org/html/2609.19170#S3)modifies this post\-shock response\.

## 3RETD: Delayed Shock Recovery and Solution Preservation

The two\-state analysis in Section[2\.4](https://arxiv.org/html/2609.19170#S2.SS4)identifies two facts that must be kept separate\. ETD’s emphatic weighting gives a stable and useful mean system, but the sampled constant\-stepsize recursion can accumulate an expanding ordered product\. The algorithm proposed here therefore changes the response to an emphatic shock without discarding the emphatic trace or its fixed\-point information\.

### 3\.1Design Principles and the Unit\-Loop\-Gain Choice

Write the scalar ETD signal asgt=Ft​ρt​δtg\_\{t\}=F\_\{t\}\\rho\_\{t\}\\delta\_\{t\}\. After a largegtg\_\{t\}, the next ratio may be zero, so ETD applies no correction to the history summarized byFtF\_\{t\}\. We require the modification to leaveFtF\_\{t\}andρt\\rho\_\{t\}unchanged, react after zero\-ratio samples, add and decay only one scalar state, preserve the solution along a recoverable direction, and retainO⁡\(d\)O\(d\)cost\. These requirements are met by the following first\-order scalar\-feedback family:

𝜽t\+1\\displaystyle\\bm\{\\theta\}\_\{t\+1\}=𝜽t\+αt​\(gt−κ​ωt\)​ϕ​\(St\),\\displaystyle=\\bm\{\\theta\}\_\{t\}\+\\alpha\_\{t\}\(g\_\{t\}\-\\kappa\\omega\_\{t\}\)\\bm\{\\phi\}\(S\_\{t\}\),\(7\)ωt\+1\\displaystyle\\omega\_\{t\+1\}=ωt\+αt​\{η​gt−β​ωt\},κ,η,β\>0\.\\displaystyle=\\omega\_\{t\}\+\\alpha\_\{t\}\\\{\\eta g\_\{t\}\-\\beta\\omega\_\{t\}\\\},\\qquad\\kappa,\\eta,\\beta\>0\.Ifω~t=κ​ωt\\widetilde\{\\omega\}\_\{t\}=\\kappa\\omega\_\{t\}, the updates become𝜽t\+1=𝜽t\+αt​\(gt−ω~t\)​ϕ​\(St\)\\bm\{\\theta\}\_\{t\+1\}=\\bm\{\\theta\}\_\{t\}\+\\alpha\_\{t\}\(g\_\{t\}\-\\widetilde\{\\omega\}\_\{t\}\)\\bm\{\\phi\}\(S\_\{t\}\)andω~t\+1=ω~t\+αt​\{ℓ​gt−β​ω~t\}\\widetilde\{\\omega\}\_\{t\+1\}=\\widetilde\{\\omega\}\_\{t\}\+\\alpha\_\{t\}\\\{\\ell g\_\{t\}\-\\beta\\widetilde\{\\omega\}\_\{t\}\\\}, where only the loop\-gain productℓ=κ​η\\ell=\\kappa\\etaremains\. The coordinate change sets the coefficient in the𝜽\\bm\{\\theta\}update to one but leavesℓ\\ellinvariant\. RETD choosesℓ=1\\ell=1, so that the shock enters both channels with the same gain\. We use the representativeκ=η=1\\kappa=\\eta=1and writeβ=1\+c\\beta=1\+c\. This defines the one\-parameter family; it is a design choice, not a uniqueness claim for the broader family in Equation[7](https://arxiv.org/html/2609.19170#S3.E7)\.

The resulting RETD recursion is

𝜽t\+1\\displaystyle\\bm\{\\theta\}\_\{t\+1\}=𝜽t\+αt​\(gt−ωt\)​ϕ​\(St\),\\displaystyle=\\bm\{\\theta\}\_\{t\}\+\\alpha\_\{t\}\(g\_\{t\}\-\\omega\_\{t\}\)\\bm\{\\phi\}\(S\_\{t\}\),\(8\)ωt\+1\\displaystyle\\omega\_\{t\+1\}=ωt\+αt\{gt−\(1\+c\)ωt\},c\>0\.\\displaystyle=\\omega\_\{t\}\+\\alpha\_\{t\}\\\{g\_\{t\}\-\(1\+c\)\\omega\_\{t\}\\\},\\qquad c\>0\.The follow\-on trace andρt\\rho\_\{t\}are unchanged, so RETD does not make the infinite trace variance finite\. It changes only the joint dynamics after the same shock, with one inner product and one scalar recursion per update\.

### 3\.2Impulse Response and the Necessity of Positive Leakage

For a constant stepsize, a sample withρt=0\\rho\_\{t\}=0has homogeneous matrix

𝑴tR=\[𝑰−α​ϕ​\(St\)𝟎𝖳1−α⁡\(1\+c\)\]\.\\bm\{M\}\_\{t\}^\{R\}=\\begin\{bmatrix\}\\bm\{I\}&\-\\alpha\\bm\{\\phi\}\(S\_\{t\}\)\\\\ \\bm\{0\}^\{\\mathsf\{T\}\}&1\-\\alpha\(1\+c\)\\end\{bmatrix\}\.\(9\)Unlike ETD’s identity update, this matrix retains and decays the stored shock while correcting the value parameter\. The auxiliary channel requires0<α⁡\(1\+c\)<20<\\alpha\(1\+c\)<2, hencec<2/α−1c<2/\\alpha\-1for fixedα\\alpha\.

###### Proposition 1\(Impulse response of the recovery channel\)\.

Fixα,c\\alpha,cwith0<α⁡\(1\+c\)<20<\\alpha\(1\+c\)<2and putqc=1−α⁡\(1\+c\)q\_\{c\}=1\-\\alpha\(1\+c\)\. Ifωτ=0\\omega\_\{\\tau\}=0,gτ=gg\_\{\\tau\}=g, andgτ\+j=0g\_\{\\tau\+j\}=0forj≥1j\\geq 1, then

ωτ\+j\\displaystyle\\omega\_\{\\tau\+j\}=α​g​qcj−1,\\displaystyle=\\alpha gq\_\{c\}^\{j\-1\},j≥1,\\displaystyle j\\geq 1,𝜽τ\+j\+1−𝜽τ\+j\\displaystyle\\bm\{\\theta\}\_\{\\tau\+j\+1\}\-\\bm\{\\theta\}\_\{\\tau\+j\}=−α2​g​qcj−1​ϕ​\(Sτ\+j\),\\displaystyle=\-\\alpha^\{2\}gq\_\{c\}^\{j\-1\}\\bm\{\\phi\}\(S\_\{\\tau\+j\}\),j≥1\.\\displaystyle j\\geq 1\.If the post\-shock features equalϕ¯\\bar\{\\bm\{\\phi\}\}, the total correction is−αgϕ¯/\(1\+c\)\-\\alpha g\\bar\{\\bm\{\\phi\}\}/\(1\+c\)\.

###### Proof\.

At the shockωτ\+1=α​g\\omega\_\{\\tau\+1\}=\\alpha g; thereafterωt\+1=qc​ωt\\omega\_\{t\+1\}=q\_\{c\}\\omega\_\{t\}\. Substitution into the first update and summing the geometric series \(using\|qc\|<1\|q\_\{c\}\|<1\) gives both identities\. ∎

A positive leakage removes a conserved mode atc=0c=0\. If the constant function lies in the feature span, chooseℓ\\bm\{\\ell\}withℓ𝖳​ϕ​\(s\)=1\\bm\{\\ell\}^\{\\mathsf\{T\}\}\\bm\{\\phi\}\(s\)=1\. In a zero\-reward task, Equation[8](https://arxiv.org/html/2609.19170#S3.E8)givesJt=ℓ𝖳​𝜽t−ωtJ\_\{t\}=\\bm\{\\ell\}^\{\\mathsf\{T\}\}\\bm\{\\theta\}\_\{t\}\-\\omega\_\{t\}, which is preserved sample by sample\. The joint product therefore has a noncontracting left direction\. Forc\>0c\>0,Jt\+1=Jt\+αt​c​ωtJ\_\{t\+1\}=J\_\{t\}\+\\alpha\_\{t\}c\\omega\_\{t\}, so strict contraction is no longer ruled out, although some positive values ofccmay still be unstable\.

### 3\.3Mean Equilibrium and the Quality of the RETD Solution

Let𝒒=𝚽𝖳​𝒅μ\\bm\{q\}=\\bm\{\\Phi\}^\{\\mathsf\{T\}\}\\bm\{d\}\_\{\\mu\}andβ=𝒇𝖳​𝒓π\\beta=\\bm\{f\}^\{\\mathsf\{T\}\}\\bm\{r\}\_\{\\pi\}\. In an identifiable full\-rank feature coordinate, the stationary RETD mean system is

𝑮c​\[𝜽cωc\]=𝒉,𝑮c=\[𝑨𝒒𝒒𝖳1\+c\],𝒉=\[𝒃β\]\.\\bm\{G\}\_\{c\}\\begin\{bmatrix\}\\bm\{\\theta\}\_\{c\}\\\\ \\omega\_\{c\}\\end\{bmatrix\}=\\bm\{h\},\\qquad\\bm\{G\}\_\{c\}=\\begin\{bmatrix\}\\bm\{A\}&\\bm\{q\}\\\\ \\bm\{q\}^\{\\mathsf\{T\}\}&1\+c\\end\{bmatrix\},\\qquad\\bm\{h\}=\\begin\{bmatrix\}\\bm\{b\}\\\\ \\beta\\end\{bmatrix\}\.If the originaln×dn\\times dfeature matrix has rankr<dr<d, let𝑩∈ℝd×r\\bm\{B\}\\in\\mathbb\{R\}^\{d\\times r\}have orthonormal columns spanningrow⁡\(𝚽\)\\operatorname\{row\}\(\\bm\{\\Phi\}\)and use

𝚽~=𝚽​𝑩,𝑨~=𝑩𝖳​𝑨​𝑩,𝒒~=𝑩𝖳​𝒒,𝒃~=𝑩𝖳​𝒃\.\\widetilde\{\\bm\{\\Phi\}\}=\\bm\{\\Phi\}\\bm\{B\},\\quad\\widetilde\{\\bm\{A\}\}=\\bm\{B\}^\{\\mathsf\{T\}\}\\bm\{A\}\\bm\{B\},\\quad\\widetilde\{\\bm\{q\}\}=\\bm\{B\}^\{\\mathsf\{T\}\}\\bm\{q\},\\quad\\widetilde\{\\bm\{b\}\}=\\bm\{B\}^\{\\mathsf\{T\}\}\\bm\{b\}\.\(10\)Then𝚽~\\widetilde\{\\bm\{\\Phi\}\}has full column rank and𝜽~=𝑩𝖳​𝜽\\widetilde\{\\bm\{\\theta\}\}=\\bm\{B\}^\{\\mathsf\{T\}\}\\bm\{\\theta\}is identifiable\. All inverses below are in these coordinates; the orthogonal parameter component is unchanged and prediction\-null\.

Define𝒖=𝑨−1​𝒒\\bm\{u\}=\\bm\{A\}^\{\-1\}\\bm\{q\},s=𝒒𝖳​𝑨−1​𝒒s=\\bm\{q\}^\{\\mathsf\{T\}\}\\bm\{A\}^\{\-1\}\\bm\{q\},mE=β−𝒒𝖳​𝜽Em\_\{E\}=\\beta\-\\bm\{q\}^\{\\mathsf\{T\}\}\\bm\{\\theta\}\_\{E\}, and𝜽E=𝑨−1​𝒃\\bm\{\\theta\}\_\{E\}=\\bm\{A\}^\{\-1\}\\bm\{b\}\.

###### Proposition 2\(Exact single\-ccrecovery\)\.

If1\+c−s≠01\+c\-s\\neq 0, the RETD equilibrium is unique and satisfies

ωc=mE1\+c−s,𝜽c=𝜽E−𝑨−1​𝒒​ωc\.\\omega\_\{c\}=\\frac\{m\_\{E\}\}\{1\+c\-s\},\\qquad\\bm\{\\theta\}\_\{c\}=\\bm\{\\theta\}\_\{E\}\-\\bm\{A\}^\{\-1\}\\bm\{q\}\\,\\omega\_\{c\}\.\(11\)Consequently,

𝜽E=𝜽c\+𝑨−1​𝒒​ωc\.\\boxed\{\\bm\{\\theta\}\_\{E\}=\\bm\{\\theta\}\_\{c\}\+\\bm\{A\}^\{\-1\}\\bm\{q\}\\,\\omega\_\{c\}\.\}

###### Proof\.

Subtracting𝑨​𝜽E=𝒃\\bm\{A\}\\bm\{\\theta\}\_\{E\}=\\bm\{b\}from the first block equation gives𝜽c−𝜽E=−𝑨−1​𝒒​ωc\\bm\{\\theta\}\_\{c\}\-\\bm\{\\theta\}\_\{E\}=\-\\bm\{A\}^\{\-1\}\\bm\{q\}\\omega\_\{c\}\. Substitution into the second gives\(1\+c−s\)​ωc=mE\(1\+c\-s\)\\omega\_\{c\}=m\_\{E\}and hence both formulas\. The same condition is equivalent todet𝑮c=det\(𝑨\)​\(1\+c−s\)≠0\\det\\bm\{G\}\_\{c\}=\\det\(\\bm\{A\}\)\(1\+c\-s\)\\neq 0\. ∎

Equation[11](https://arxiv.org/html/2609.19170#S3.E11)shows that the raw parameter differs from ETD only along𝒖\\bm\{u\}; in value space,𝚽​𝜽c=𝚽​𝜽E−\(𝚽​𝒖\)​ωc\\bm\{\\Phi\}\\bm\{\\theta\}\_\{c\}=\\bm\{\\Phi\}\\bm\{\\theta\}\_\{E\}\-\(\\bm\{\\Phi\}\\bm\{u\}\)\\omega\_\{c\}\. The recovered output cancels this direction and inherits the ETD fixed\-point guarantee under the same assumptions and norm\. This is an equilibrium identity, not equality of finite\-time trajectories, covariances, or action rankings\.

If1\+c−s=01\+c\-s=0andmE≠0m\_\{E\}\\neq 0, the mean equations are inconsistent; if both are zero, the equilibrium is not unique\. IfmE=0m\_\{E\}=0and1\+c−s≠01\+c\-s\\neq 0, thenωc=0\\omega\_\{c\}=0and raw RETD already equals ETD, as in the zero\-reward examples\.

###### Corollary 3\(Two\-ccmodel\-free readout\)\.

Let\(𝛉1,ω1\)\(\\bm\{\\theta\}\_\{1\},\\omega\_\{1\}\)and\(𝛉2,ω2\)\(\\bm\{\\theta\}\_\{2\},\\omega\_\{2\}\)be RETD equilibria for distinct regularizers\. Ifω1≠ω2\\omega\_\{1\}\\neq\\omega\_\{2\}, then

𝜽E=ω1​𝜽2−ω2​𝜽1ω1−ω2\.\\boxed\{\\bm\{\\theta\}\_\{E\}=\\frac\{\\omega\_\{1\}\\bm\{\\theta\}\_\{2\}\-\\omega\_\{2\}\\bm\{\\theta\}\_\{1\}\}\{\\omega\_\{1\}\-\\omega\_\{2\}\}\.\}\(12\)Moreover,\|ω1−ω2\|=\|mE\|​\|c2−c1\|/\(\|1\+c1−s\|​\|1\+c2−s\|\)\|\\omega\_\{1\}\-\\omega\_\{2\}\|=\|m\_\{E\}\|\\,\|c\_\{2\}\-c\_\{1\}\|/\(\|1\+c\_\{1\}\-s\|\\,\|1\+c\_\{2\}\-s\|\)\.

###### Proof\.

Use𝜽i=𝜽E−𝒖​ωi\\bm\{\\theta\}\_\{i\}=\\bm\{\\theta\}\_\{E\}\-\\bm\{u\}\\omega\_\{i\}from Proposition[2](https://arxiv.org/html/2609.19170#Thmtheorem2); eliminating𝒖\\bm\{u\}gives the readout, and subtractingωi=mE/\(1\+ci−s\)\\omega\_\{i\}=m\_\{E\}/\(1\+c\_\{i\}\-s\)gives its conditioning formula\. ∎

The two\-ccformula is asymptotic, not finite\-time ETD reproduction, and becomes ill conditioned whenmEm\_\{E\}is near zero, the regularizers are close, or estimation errors approach\|ω1−ω2\|\|\\omega\_\{1\}\-\\omega\_\{2\}\|\. In the zero\-reward examples both auxiliary equilibria are zero, giving0/00/0because no recovery is needed\.

*Which Output Is Used in Prediction\.*The raw online predictor is𝚽​𝜽t\\bm\{\\Phi\}\\bm\{\\theta\}\_\{t\}\. If the model quantities𝑨\\bm\{A\}and𝒒\\bm\{q\}are available, the single\-ccpredictor instead uses𝚽⁡\{𝜽t\+𝑨−1​𝒒​ωt\}\\bm\{\\Phi\}\\\{\\bm\{\\theta\}\_\{t\}\+\\bm\{A\}^\{\-1\}\\bm\{q\}\\,\\omega\_\{t\}\\\}\. This model\-based readout does not alter learner dynamics\. If𝑨−1​𝒒\\bm\{A\}^\{\-1\}\\bm\{q\}is unavailable, two RETD learners with distinctccvalues can share a trajectory and apply Equation[12](https://arxiv.org/html/2609.19170#S3.E12)after averaging; this doubles the parameter updates and must report\|ω¯1−ω¯2\|\|\\bar\{\\omega\}\_\{1\}\-\\bar\{\\omega\}\_\{2\}\|\. Section[6](https://arxiv.org/html/2609.19170#S6)evaluates both readouts on nonzero\-reward tasks\.

### 3\.4RETD Closes the Two\-State Sign Separation

Return to the counterexample in Section[2\.4](https://arxiv.org/html/2609.19170#S2.SS4)\. Hereq=𝔼⁡\[ϕ⁡\(St\)\]=13/10q=\\mathbb\{E\}\[\\phi\(S\_\{t\}\)\]=13/10\. Atc=4c=4, the RETD mean matrix is

𝑮4=\[79/2513/1013/105\],det\(𝑮4\)=1411100\>0\.\\bm\{G\}\_\{4\}=\\begin\{bmatrix\}79/25&13/10\\\\ 13/10&5\\end\{bmatrix\},\\qquad\\det\(\\bm\{G\}\_\{4\}\)=\\frac\{1411\}\{100\}\>0\.\(13\)It is symmetric positive definite, andρ⁡\(𝑰−0\.1​𝑮4\)≈0\.7513<1\\rho\(\\bm\{I\}\-0\.1\\bm\{G\}\_\{4\}\)\\approx 0\.7513<1\. Thus both ETD and RETD have stable mean recursions on the same counterexample\.

For the regenerative product, puthk=Fk​ρk​\{γ​ϕ​\(Sk\+1\)−ϕ⁡\(Sk\)\}h\_\{k\}=F\_\{k\}\\rho\_\{k\}\\\{\\gamma\\phi\(S\_\{k\+1\}\)\-\\phi\(S\_\{k\}\)\\\}\. Atc=4c=4, the auxiliary retention is1−α⁡\(1\+c\)=1/21\-\\alpha\(1\+c\)=1/2\. Thekkth target\-action matrix is

𝑻k=\[1\+α​hk​ϕk−α​ϕkα​hk1/2\],\\bm\{T\}\_\{k\}=\\begin\{bmatrix\}1\+\\alpha h\_\{k\}\\phi\_\{k\}&\-\\alpha\\phi\_\{k\}\\\\ \\alpha h\_\{k\}&1/2\\end\{bmatrix\},whereh0=8/3h\_\{0\}=8/3,ϕ0=1\\phi\_\{0\}=1, andhk=−2Fk/3h\_\{k\}=\-2F\_\{k\}/3,ϕk=2\\phi\_\{k\}=2fork≥1k\\geq 1\. The final off\-target action has matrix

𝑫1=\[1−1/1001/2\],𝑫2=\[1−1/501/2\],\\bm\{D\}\_\{1\}=\\begin\{bmatrix\}1&\-1/10\\\\ 0&1/2\\end\{bmatrix\},\\qquad\\bm\{D\}\_\{2\}=\\begin\{bmatrix\}1&\-1/5\\\\ 0&1/2\\end\{bmatrix\},depending on the current state\. The complete cycle matrices are

𝑴0=𝑫1,𝑴L=𝑫2𝑻L−1⋯𝑻0,L≥1\.\\bm\{M\}\_\{0\}=\\bm\{D\}\_\{1\},\\qquad\\bm\{M\}\_\{L\}=\\bm\{D\}\_\{2\}\\bm\{T\}\_\{L\-1\}\\cdots\\bm\{T\}\_\{0\},\\quad L\\geq 1\.The dashed/off\-target matrix is essential: omitting it would remove the recovery mechanism\.

Use the induced similarity norm

𝑺=\[4/3−1/303/4\],‖𝑴‖S=‖𝑺​𝑴​𝑺−1‖2\.\\bm\{S\}=\\begin\{bmatrix\}4/3&\-1/3\\\\ 0&3/4\\end\{bmatrix\},\\qquad\\\|\\bm\{M\}\\\|\_\{S\}=\\\|\\bm\{S\}\\bm\{M\}\\bm\{S\}^\{\-1\}\\\|\_\{2\}\.Exact rational cycle matrices and outward\-rounded spectral\-norm calculations give the certificate in Table[1](https://arxiv.org/html/2609.19170#S3.T1)\.

Table 1:Finite prefix of the two\-state RETD cycle certificate\. The cycle matrices are rational; only outward\-rounded logarithmic bounds are printed\.For everyL≥7L\\geq 7, the inequalities‖𝑺‖2​‖𝑺−1‖2<20/9\\\|\\bm\{S\}\\\|\_\{2\}\\\|\\bm\{S\}^\{\-1\}\\\|\_\{2\}<20/9,‖𝑫2‖2<11/10\\\|\\bm\{D\}\_\{2\}\\\|\_\{2\}<11/10,‖𝑻k‖2≤\(9/5\)​Fk\\\|\\bm\{T\}\_\{k\}\\\|\_\{2\}\\leq\(9/5\)F\_\{k\}, andFk<\(3/2\)​3kF\_\{k\}<\(3/2\)3^\{k\}give

log⁡‖𝑴L‖S≤log⁡229\+L​log⁡2710\+L⁡\(L−1\)2​log⁡3\.\\log\\\|\\bm\{M\}\_\{L\}\\\|\_\{S\}\\leq\\log\\frac\{22\}\{9\}\+L\\log\\frac\{27\}\{10\}\+\\frac\{L\(L\-1\)\}\{2\}\\log 3\.Multiplying by the geometric cycle probabilities and summing the tail yields

∑L=7∞710​\(310\)L​log⁡‖𝑴L‖S<0\.007619669942317829\.\\sum\_\{L=7\}^\{\\infty\}\\frac\{7\}\{10\}\\left\(\\frac\{3\}\{10\}\\right\)^\{L\}\\log\\\|\\bm\{M\}\_\{L\}\\\|\_\{S\}<0\.007619669942317829\.Together with the finite prefix,𝔼​log⁡‖𝑴L‖S<−0\.0010016772775798414\\mathbb\{E\}\\log\\\|\\bm\{M\}\_\{L\}\\\|\_\{S\}<\-0\.0010016772775798414\. Dividing by the mean cycle length gives the cycle\-endpoint bound\. For every transition, let𝚷N=𝑴LN⋯𝑴L1\\bm\{\\Pi\}\_\{N\}=\\bm\{M\}\_\{L\_\{N\}\}\\cdots\\bm\{M\}\_\{L\_\{1\}\}andTN=∑n=1N\(Ln\+1\)T\_\{N\}=\\sum\_\{n=1\}^\{N\}\(L\_\{n\}\+1\)\. Submultiplicativity and the strong law imply

lim supN→∞1TN​log⁡‖𝚷N‖S≤𝔼​log⁡‖𝑴L‖S𝔼⁡\[L\+1\]a\.s\.\\limsup\_\{N\\to\\infty\}\\frac\{1\}\{T\_\{N\}\}\\log\\\|\\bm\{\\Pi\}\_\{N\}\\\|\_\{S\}\\leq\\frac\{\\mathbb\{E\}\\log\\\|\\bm\{M\}\_\{L\}\\\|\_\{S\}\}\{\\mathbb\{E\}\[L\+1\]\}\\quad\\text\{a\.s\.\}\(14\)The same factor bounds give constantsd0,d1<∞d\_\{0\},d\_\{1\}<\\inftysuch that every partial product within a cycle of lengthLLsatisfiesmaxjlog\+∥𝑻j−1⋯𝑻0∥S≤d0\+d1L2\\max\_\{j\}\\log^\{\+\}\\\|\\bm\{T\}\_\{j\-1\}\\cdots\\bm\{T\}\_\{0\}\\\|\_\{S\}\\leq d\_\{0\}\+d\_\{1\}L^\{2\}, wherelog\+⁡x=max⁡\{0,log⁡x\}\\log^\{\+\}x=\\max\\\{0,\\log x\\\}\. The geometric cycle law and Borel–Cantelli giveLn2/n→0L\_\{n\}^\{2\}/n\\to 0, whereasTN/N→10/7T\_\{N\}/N\\to 10/7; the unfinished cycle is thereforeo⁡\(t\)o\(t\)\. Hence Equation[14](https://arxiv.org/html/2609.19170#S3.E14)holds for the complete product and proves

λR​\(α=0\.1,c=4\)≤−0\.0007011740943058889<0\.\\lambda\_\{R\}\(\\alpha=0\.1,c=4\)\\leq\-0\.0007011740943058889<0\.\(15\)
Together, Equations[6](https://arxiv.org/html/2609.19170#S2.E6),[13](https://arxiv.org/html/2609.19170#S3.E13), and[15](https://arxiv.org/html/2609.19170#S3.E15)give the sign separation\. Both mean maps contract, but atα=\.1\\alpha=\.1,λE\>0\.003006660245721778\\lambda\_\{E\}\>0\.003006660245721778whileλR≤−0\.0007011740943058889\\lambda\_\{R\}\\leq\-0\.0007011740943058889\. TheSS\-norm bound and the unfinished\-cycle estimate give almost\-sure contraction at every transition\. Because rewards are zero, RETD’s affine residual vanishes sample by sample\. The sign difference comes from the final zero\-ratio action: ETD applies the identity, whereas RETD’s𝑫i\\bm\{D\}\_\{i\}releases the delayed correction and decays the stored shock\. Section[4](https://arxiv.org/html/2609.19170#S4)treats stochastic stability beyond this example\.

## 4Stochastic Stability of RETD

The expected RETD matrix supplies the common geometric certificate for two stochastic regimes\. We first establish almost\-sure convergence with a diminishing stepsize and then establish contraction of the constant\-stepsize random product\. The first result is a stochastic\-approximation statement; the second addresses the mean\-to\-path question isolated by the counterexample\.

### 4\.1Common Geometry and the Explicit Role ofcc

Work in an identifiable, full\-column\-rank feature coordinate and let𝑯=Sym⁡\(𝑨\)=\(𝑨\+𝑨𝖳\)/2\\bm\{H\}=\\Sym\(\\bm\{A\}\)=\(\\bm\{A\}\+\\bm\{A\}^\{\\mathsf\{T\}\}\)/2\.

###### Assumption 1\(Finite off\-policy prediction model\)\.

The state and action spaces are finite\. The behavior chain is irreducible and aperiodic, the target policy is covered by the behavior policy, rewards are bounded functions of\(St,At,St\+1\)\(S\_\{t\},A\_\{t\},S\_\{t\+1\}\), interest is one,γ∈\[0,1\)\\gamma\\in\[0,1\)is constant, andλ=0\\lambda=0\. The identifiable feature matrix has full column rank, its rows are bounded, and𝐇≻0\\bm\{H\}\\succ 0\.

###### Assumption 2\(RETD regularization\)\.

The parameterccis chosen before the run and is required to satisfyc\>cmin:=max⁡\{0,𝐪𝖳​𝐇−1​𝐪−1\}c\>c\_\{\\min\}:=\\max\\\{0,\\bm\{q\}^\{\\mathsf\{T\}\}\\bm\{H\}^\{\-1\}\\bm\{q\}\-1\\\}\.

This is a direct hypothesis of both RETD theorems, rather than a choice made inside either proof\. Under Assumption[1](https://arxiv.org/html/2609.19170#Thmassumption1), it is equivalent to positive definiteness of the symmetric RETD mean matrix\. To see this, define

Δc=1\+c−𝒒𝖳​𝑯−1​𝒒\>0,𝑼=\[𝑰𝑯−1​𝒒𝟎𝖳1\]\.\\Delta\_\{c\}=1\+c\-\\bm\{q\}^\{\\mathsf\{T\}\}\\bm\{H\}^\{\-1\}\\bm\{q\}\>0,\\qquad\\bm\{U\}=\\begin\{bmatrix\}\\bm\{I\}&\\bm\{H\}^\{\-1\}\\bm\{q\}\\\\ \\bm\{0\}^\{\\mathsf\{T\}\}&1\\end\{bmatrix\}\.Then the exact congruence

Sym⁡\(𝑮c\)=\[𝑯𝒒𝒒𝖳1\+c\]=𝑼𝖳​\[𝑯𝟎𝟎𝖳Δc\]​𝑼≻0\\Sym\(\\bm\{G\}\_\{c\}\)=\\begin\{bmatrix\}\\bm\{H\}&\\bm\{q\}\\\\ \\bm\{q\}^\{\\mathsf\{T\}\}&1\+c\\end\{bmatrix\}=\\bm\{U\}^\{\\mathsf\{T\}\}\\begin\{bmatrix\}\\bm\{H\}&\\bm\{0\}\\\\ \\bm\{0\}^\{\\mathsf\{T\}\}&\\Delta\_\{c\}\\end\{bmatrix\}\\bm\{U\}\\succ 0\(16\)shows that−𝑮c\-\\bm\{G\}\_\{c\}is Hurwitz\. In particular,

m¯c=min⁡\{λmin​\(𝑯\),Δc\}‖𝑼−1‖22≤λmin​\{Sym⁡\(𝑮c\)\}\.\\underline\{m\}\_\{c\}=\\frac\{\\min\\\{\\lambda\_\{\\min\}\(\\bm\{H\}\),\\Delta\_\{c\}\\\}\}\{\\\|\\bm\{U\}^\{\-1\}\\\|\_\{2\}^\{2\}\}\\leq\\lambda\_\{\\min\}\\\{\\Sym\(\\bm\{G\}\_\{c\}\)\\\}\.\(17\)The bound follows from the RETD mean matrix and does not assume contraction of the sampled matrix product\.

### 4\.2Diminishing\-Stepsize Stochastic Stability

The follow\-on trace is unbounded, so an independent, bounded\-noise Robbins–Monro template is not sufficient\. We use the invariant\-law and pathwise averaging results for the ETD trace process in[Yu \(2015\)](https://arxiv.org/html/2609.19170#bib.bib24)and the general\-state\-space ODE theorem of[Liu et al\. \(2025\)](https://arxiv.org/html/2609.19170#bib.bib23)\. Both are external results; the proof below verifies their conditions for the actual RETD recursion\.

###### Theorem 4\(Diminishing\-stepsize RETD\)\.

Suppose the common conditions in Assumption[1](https://arxiv.org/html/2609.19170#Thmassumption1)and the explicitcccondition in Assumption[2](https://arxiv.org/html/2609.19170#Thmassumption2)hold\. Letαt=B1/\(t\+B2\)\\alpha\_\{t\}=B\_\{1\}/\(t\+B\_\{2\}\), whereB1,B2\>0B\_\{1\},B\_\{2\}\>0\. Then the joint RETD iterates𝐳t=\[𝛉t𝖳,ωt\]𝖳\\bm\{z\}\_\{t\}=\[\\bm\{\\theta\}\_\{t\}^\{\\mathsf\{T\}\},\\omega\_\{t\}\]^\{\\mathsf\{T\}\}are almost surely bounded and converge to the unique RETD mean equilibrium:

supt‖𝒛t‖2<∞a\.s\.,𝒛t⟶𝒛c∗=𝑮c−1​𝒉a\.s\.\\sup\_\{t\}\\\|\\bm\{z\}\_\{t\}\\\|\_\{2\}<\\infty\\quad\\text\{a\.s\.\},\\qquad\\bm\{z\}\_\{t\}\\longrightarrow\\bm\{z\}\_\{c\}^\{\*\}=\\bm\{G\}\_\{c\}^\{\-1\}\\bm\{h\}\\quad\\text\{a\.s\.\}\(18\)For a rank\-deficient original representation, the conclusion holds in the identifiable coordinate; the parameter\-null component is unchanged and the predictions converge to the corresponding RETD equilibrium prediction\.

###### Proof\.

We verify the Markovian ODE conditions in five steps\.

Step 1: exact Markov stochastic\-approximation form\.Let𝒙t\\bm\{x\}\_\{t\}be the feature in the identifiable coordinate,𝝍t=𝒙t−γ​𝒙t\+1\\bm\{\\psi\}\_\{t\}=\\bm\{x\}\_\{t\}\-\\gamma\\bm\{x\}\_\{t\+1\}, andXt=Ft​ρtX\_\{t\}=F\_\{t\}\\rho\_\{t\}\. Use the augmented driverYt\+1=\(St,At,St\+1,Ft,Rt\+1\)Y\_\{t\+1\}=\(S\_\{t\},A\_\{t\},S\_\{t\+1\},F\_\{t\},R\_\{t\+1\}\)\. Fory=\(s,a,s′,f,r\)y=\(s,a,s^\{\\prime\},f,r\)define

𝑮c​\(y\)\\displaystyle\\bm\{G\}\_\{c\}\(y\)=\[X​𝒙​𝝍𝖳𝒙X​𝝍𝖳1\+c\],\\displaystyle=\\begin\{bmatrix\}X\\bm\{x\}\\bm\{\\psi\}^\{\\mathsf\{T\}\}&\\bm\{x\}\\\\ X\\bm\{\\psi\}^\{\\mathsf\{T\}\}&1\+c\\end\{bmatrix\},𝒈⁡\(y\)\\displaystyle\\bm\{g\}\(y\)=X​r​\[𝒙1\],\\displaystyle=Xr\\begin\{bmatrix\}\\bm\{x\}\\\\ 1\\end\{bmatrix\},\(19\)whereX=f​π​\(a∣s\)/μ⁡\(a∣s\)X=f\\pi\(a\\mid s\)/\\mu\(a\\mid s\)\. Direct substitution into Equation[8](https://arxiv.org/html/2609.19170#S3.E8)gives𝒛t\+1=𝒛t\+αt​H​\(𝒛t,Yt\+1\)\\bm\{z\}\_\{t\+1\}=\\bm\{z\}\_\{t\}\+\\alpha\_\{t\}H\(\\bm\{z\}\_\{t\},Y\_\{t\+1\}\), whereH⁡\(𝒛,y\)=𝒈⁡\(y\)−𝑮c​\(y\)​𝒛H\(\\bm\{z\},y\)=\\bm\{g\}\(y\)\-\\bm\{G\}\_\{c\}\(y\)\\bm\{z\}\. No independence or square\-integrability ofFtF\_\{t\}has been inserted\.

Step 2: invariant law and one common averaging event\.Lemma 25 of[Liu et al\. \(2025\)](https://arxiv.org/html/2609.19170#bib.bib23), which packages Theorems 3\.2 and 3\.3 of[Yu \(2015\)](https://arxiv.org/html/2609.19170#bib.bib24), gives the ETD augmented chain a unique invariant probability measure and an almost\-sure law of large numbers for every finite\-dimensional function that is Lipschitz in the trace coordinate\. Ourλ=0\\lambda=0driver is the corresponding state–action–follow\-on\-trace chain\. The bounded deterministic reward in Assumption[1](https://arxiv.org/html/2609.19170#Thmassumption1)is a measurable graph of\(St,At,St\+1\)\(S\_\{t\},A\_\{t\},S\_\{t\+1\}\), so adjoining it neither changes the invariant marginal nor creates another invariant law\. Denote the resulting invariant law byζ\\zeta\.

Every entry of𝑮c​\(y\)\\bm\{G\}\_\{c\}\(y\)and𝒈⁡\(y\)\\bm\{g\}\(y\)is affine inff\. Moreover,Lc​\(y\)=1\+‖𝑮c​\(y\)‖2L\_\{c\}\(y\)=1\+\\\|\\bm\{G\}\_\{c\}\(y\)\\\|\_\{2\}is Lipschitz inff, because the reverse triangle inequality bounds the difference of two matrix norms by the norm of their difference\. Lemma 25 therefore gives finite invariant expectations and a law of large numbers for every scalar entry of𝑮c\\bm\{G\}\_\{c\}, every coordinate of𝒈\\bm\{g\}, andLcL\_\{c\}\. There are only finitely many such functions\. Intersecting their probability\- one events yields a single eventΩ0\\Omega\_\{0\}on which all these averages converge simultaneously\. In particular, onΩ0\\Omega\_\{0\},

𝔼ζ​\[𝑮c​\(Y\)\]=𝑮c,𝔼ζ​\[𝒈⁡\(Y\)\]=𝒉,H¯​\(𝒛\)=𝒉−𝑮c​𝒛\.\\mathbb\{E\}\_\{\\zeta\}\[\\bm\{G\}\_\{c\}\(Y\)\]=\\bm\{G\}\_\{c\},\\qquad\\mathbb\{E\}\_\{\\zeta\}\[\\bm\{g\}\(Y\)\]=\\bm\{h\},\\qquad\\bar\{H\}\(\\bm\{z\}\)=\\bm\{h\}\-\\bm\{G\}\_\{c\}\\bm\{z\}\.
Step 3: Liu Assumptions 2–4 and 6\.The harmonic schedule verifies Liu Assumption 2: it is positive and nonincreasing,∑tαt=∞\\sum\_\{t\}\\alpha\_\{t\}=\\infty,αt→0\\alpha\_\{t\}\\to 0, and\(αt−αt\+1\)/αt=\(t\+B2\+1\)−1=O⁡\(αt\)\(\\alpha\_\{t\}\-\\alpha\_\{t\+1\}\)/\\alpha\_\{t\}=\(t\+B\_\{2\}\+1\)^\{\-1\}=O\(\\alpha\_\{t\}\)\.

For a scaling variabler≥1r\\geq 1,

Hr​\(𝒛,y\)=H⁡\(r​𝒛,y\)r=−𝑮c​\(y\)​𝒛\+𝒈⁡\(y\)r\.H\_\{r\}\(\\bm\{z\},y\)=\\frac\{H\(r\\bm\{z\},y\)\}\{r\}=\-\\bm\{G\}\_\{c\}\(y\)\\bm\{z\}\+\\frac\{\\bm\{g\}\(y\)\}\{r\}\.\(20\)Thus Liu Assumption 3 holds withH∞​\(𝒛,y\)=−𝑮c​\(y\)​𝒛H\_\{\\infty\}\(\\bm\{z\},y\)=\-\\bm\{G\}\_\{c\}\(y\)\\bm\{z\},κ⁡\(r\)=r−1\\kappa\(r\)=r^\{\-1\},b⁡\(𝒛,y\)=𝒈⁡\(y\)b\(\\bm\{z\},y\)=\\bm\{g\}\(y\), andLb​\(y\)=0L\_\{b\}\(y\)=0\. BothHHandH∞H\_\{\\infty\}are Lipschitz in𝒛\\bm\{z\}with envelopeLc​\(y\)L\_\{c\}\(y\), whose invariant expectation is finite by Step 2\. This verifies Liu Assumption 4\.

It remains to verify Liu Assumption 6 without taking an uncountable intersection of null sets\. On the single eventΩ0\\Omega\_\{0\}from Step 2, entrywise convergence gives, simultaneously for every𝒛\\bm\{z\},

1n​∑k=1n\{H⁡\(𝒛,Yk\)−H¯​\(𝒛\)\}\\displaystyle\\frac\{1\}\{n\}\\sum\_\{k=1\}^\{n\}\\\{H\(\\bm\{z\},Y\_\{k\}\)\-\\bar\{H\}\(\\bm\{z\}\)\\\}=1n​∑k=1n\{𝒈⁡\(Yk\)−𝒉\}\\displaystyle=\\frac\{1\}\{n\}\\sum\_\{k=1\}^\{n\}\\\{\\bm\{g\}\(Y\_\{k\}\)\-\\bm\{h\}\\\}−\{1n​∑k=1n\[𝑮c​\(Yk\)−𝑮c\]\}​𝒛⟶𝟎\.\\displaystyle\\quad\-\\left\\\{\\frac\{1\}\{n\}\\sum\_\{k=1\}^\{n\}\[\\bm\{G\}\_\{c\}\(Y\_\{k\}\)\-\\bm\{G\}\_\{c\}\]\\right\\\}\\bm\{z\}\\longrightarrow\\bm\{0\}\.\(21\)The same event coversLb=0L\_\{b\}=0andLcL\_\{c\}\. Becauseαn=O⁡\(n−1\)\\alpha\_\{n\}=O\(n^\{\-1\}\), multiplying the centered partial sums in Equation[21](https://arxiv.org/html/2609.19170#S4.E21)byαn\\alpha\_\{n\}makes them converge to zero\. Liu Assumption 6 is therefore satisfied for all𝒛\\bm\{z\}on one probability\-one event; no square\-integrable martingale\-difference assumption has been inserted\.

Step 4: Liu Assumption 5 and the role ofcc\.Stationary averaging of Equation[20](https://arxiv.org/html/2609.19170#S4.E20)givesH¯r​\(𝒛\)=−𝑮c​𝒛\+𝒉/r→−𝑮c​𝒛\\bar\{H\}\_\{r\}\(\\bm\{z\}\)=\-\\bm\{G\}\_\{c\}\\bm\{z\}\+\\bm\{h\}/r\\to\-\\bm\{G\}\_\{c\}\\bm\{z\}uniformly on every compact set\. The limiting mean ODE is𝒛˙=−𝑮c​𝒛\\dot\{\\bm\{z\}\}=\-\\bm\{G\}\_\{c\}\\bm\{z\}\. Write𝒛=\[𝜽𝖳,ω\]𝖳\\bm\{z\}=\[\\bm\{\\theta\}^\{\\mathsf\{T\}\},\\omega\]^\{\\mathsf\{T\}\}\. Assumption[2](https://arxiv.org/html/2609.19170#Thmassumption2)givesΔc\>0\\Delta\_\{c\}\>0, and direct completion of the square yields

𝒛𝖳​Sym⁡\(𝑮c\)​𝒛\\displaystyle\\bm\{z\}^\{\\mathsf\{T\}\}\\Sym\(\\bm\{G\}\_\{c\}\)\\bm\{z\}=𝜽𝖳​𝑯​𝜽\+2​ω​𝒒𝖳​𝜽\+\(1\+c\)​ω2\\displaystyle=\\bm\{\\theta\}^\{\\mathsf\{T\}\}\\bm\{H\}\\bm\{\\theta\}\+2\\omega\\bm\{q\}^\{\\mathsf\{T\}\}\\bm\{\\theta\}\+\(1\+c\)\\omega^\{2\}=\(𝜽\+𝑯−1​𝒒​ω\)𝖳​𝑯​\(𝜽\+𝑯−1​𝒒​ω\)\+Δc​ω2\>0\\displaystyle=\\bigl\(\\bm\{\\theta\}\+\\bm\{H\}^\{\-1\}\\bm\{q\}\\,\\omega\\bigr\)^\{\\mathsf\{T\}\}\\bm\{H\}\\bigl\(\\bm\{\\theta\}\+\\bm\{H\}^\{\-1\}\\bm\{q\}\\,\\omega\\bigr\)\+\\Delta\_\{c\}\\omega^\{2\}\>0for every nonzero𝒛\\bm\{z\}\. HenceSym⁡\(𝑮c\)≻0\\Sym\(\\bm\{G\}\_\{c\}\)\\succ 0\. Ifλ\\lambdais an eigenvalue of𝑮c\\bm\{G\}\_\{c\}with right eigenvector𝒗≠0\\bm\{v\}\\neq 0, thenℜ⁡\(λ\)=𝒗∗​Sym⁡\(𝑮c\)​𝒗/\(𝒗∗​𝒗\)\>0\\Re\(\\lambda\)=\\bm\{v\}^\{\*\}\\Sym\(\\bm\{G\}\_\{c\}\)\\bm\{v\}/\(\\bm\{v\}^\{\*\}\\bm\{v\}\)\>0\. Thus𝑮c\\bm\{G\}\_\{c\}is positive stable and−𝑮c\-\\bm\{G\}\_\{c\}is Hurwitz\. Consequently, the finite matrix𝑷c=∫0∞e−𝑮c𝖳​s​e−𝑮c​s​𝑑s≻0\\bm\{P\}\_\{c\}=\\int\_\{0\}^\{\\infty\}e^\{\-\\bm\{G\}\_\{c\}^\{\\mathsf\{T\}\}s\}e^\{\-\\bm\{G\}\_\{c\}s\}\\,\\mathrm\{d\}s\\succ 0satisfies𝑮c𝖳​𝑷c\+𝑷c​𝑮c=𝑰\\bm\{G\}\_\{c\}^\{\\mathsf\{T\}\}\\bm\{P\}\_\{c\}\+\\bm\{P\}\_\{c\}\\bm\{G\}\_\{c\}=\\bm\{I\}\. Along every nonzero solution,

dd​t​\(𝒛𝖳​𝑷c​𝒛\)=−‖𝒛‖22≤−𝒛𝖳​𝑷c​𝒛λmax​\(𝑷c\)\.\\frac\{\\mathrm\{d\}\}\{\\mathrm\{d\}t\}\(\\bm\{z\}^\{\\mathsf\{T\}\}\\bm\{P\}\_\{c\}\\bm\{z\}\)=\-\\\|\\bm\{z\}\\\|\_\{2\}^\{2\}\\leq\-\\frac\{\\bm\{z\}^\{\\mathsf\{T\}\}\\bm\{P\}\_\{c\}\\bm\{z\}\}\{\\lambda\_\{\\max\}\(\\bm\{P\}\_\{c\}\)\}\.Thus the scaling\-limit origin is globally exponentially stable, which completes Liu Assumption 5\.

Step 5: boundedness and identification of the limit\.Steps 1–4 verify Liu Assumptions 1–6 in their stated order: Step 2 gives Assumption 1, the harmonic schedule gives Assumption 2, Step 3 gives Assumptions 3, 4, and 6, and Step 4 gives Assumption 5\. Theorem 7 of[Liu et al\. \(2025\)](https://arxiv.org/html/2609.19170#bib.bib23)therefore yields almost\-sure boundedness\. Corollary 8 of the same paper places the limit set inside a bounded invariant set of𝒛˙=𝒉−𝑮c​𝒛\\dot\{\\bm\{z\}\}=\\bm\{h\}\-\\bm\{G\}\_\{c\}\\bm\{z\}\. Translating by𝒛c∗=𝑮c−1​𝒉\\bm\{z\}\_\{c\}^\{\*\}=\\bm\{G\}\_\{c\}^\{\-1\}\\bm\{h\}reduces this ODE to the globally exponentially stable system in Step 4\. Its only such invariant limit set is the singleton\{𝒛c∗\}\\\{\\bm\{z\}\_\{c\}^\{\*\}\\\}, which proves Equation[18](https://arxiv.org/html/2609.19170#S4.E18)\.

In the original rank\-deficient coordinate, use the matrix𝑩\\bm\{B\}in Equation[10](https://arxiv.org/html/2609.19170#S3.E10)\. Every parameter update lies incol⁡\(𝑩\)=row⁡\(𝚽\)\\operatorname\{col\}\(\\bm\{B\}\)=\\operatorname\{row\}\(\\bm\{\\Phi\}\), so the orthogonal parameter\-null component is conserved; the scalar auxiliary coordinate is unchanged by this reduction\. Multiplication by𝚽\\bm\{\\Phi\}then gives the stated prediction\-space limit\. ∎

The shrinking stepsize is essential to this theorem\. Over any fixed late window of lengthKK,0≤1−αt\+j/αt=j/\(t\+B2\+j\)≤K/\(t\+B2\)0\\leq 1\-\\alpha\_\{t\+j\}/\\alpha\_\{t\}=j/\(t\+B\_\{2\}\+j\)\\leq K/\(t\+B\_\{2\}\)for0≤j<K0\\leq j<K, but local near\-constancy does not turn the limitαt→0\\alpha\_\{t\}\\to 0into a theorem for a constant positiveα\\alpha\.

### 4\.3Constant\-Stepsize Stochastic Stability

For a nonzero constant stepsize, the driver must control an unbounded matrix coefficient over arbitrarily long time intervals\. We state those additional conditions explicitly\. They concern the Markov trace process and matrix growth, not the sign of the RETD random product\.

###### Assumption 3\(Super\-Lyapunov driver condition\)\.

The augmented driverYtY\_\{t\}is irreducible and aperiodic\. There existb0,B0\>0b\_\{0\},B\_\{0\}\>0,δ∈\(1/2,1\]\\delta\\in\(1/2,1\], a measurableV:𝖸→\[e,∞\)V:\\mathsf\{Y\}\\to\[e,\\infty\),W=log⁡VW=\\log V, and a small setC0C\_\{0\}such that

P​V​\(y\)≤e−b0​W​\(y\)δ​V​\(y\)​𝟏C0c​\(y\)\+B0​𝟏C0​\(y\),PV\(y\)\\leq e^\{\-b\_\{0\}W\(y\)^\{\\delta\}\}V\(y\)\\bm\{1\}\_\{C\_\{0\}^\{c\}\}\(y\)\+B\_\{0\}\\bm\{1\}\_\{C\_\{0\}\}\(y\),and every finite sublevel set ofWWis small\.

###### Assumption 4\(Growth of the RETD sample matrix\)\.

For someCG,0<∞C\_\{G,0\}<\\infty,ϵ∈\(0,1\)\\epsilon\\in\(0,1\), andβG<min⁡\{2​δ−1,δ/\(1\+ϵ\)\}\\beta\_\{G\}<\\min\\\{2\\delta\-1,\\delta/\(1\+\\epsilon\)\\\}, every entry satisfies

\|\[𝑮0​\(y\)\]i​j\|\\displaystyle\|\[\\bm\{G\}\_\{0\}\(y\)\]\_\{ij\}\|≤CG,0​W​\(y\)βG,\\displaystyle\\leq C\_\{G,0\}W\(y\)^\{\\beta\_\{G\}\},\|\[𝑮c​\(y\)\]i​j\|\\displaystyle\|\[\\bm\{G\}\_\{c\}\(y\)\]\_\{ij\}\|≤CG​\(c\)​W​\(y\)βG,CG​\(c\)=CG,0\+c\.\\displaystyle\\leq C\_\{G\}\(c\)W\(y\)^\{\\beta\_\{G\}\},\\qquad C\_\{G\}\(c\)=C\_\{G,0\}\+c\.\(22\)

For the affine conclusion define𝒛c∗=𝑮c−1​𝒉\\bm\{z\}\_\{c\}^\{\*\}=\\bm\{G\}\_\{c\}^\{\-1\}\\bm\{h\}and𝝃tR=𝒈⁡\(Yt\)−𝑮c​\(Yt\)​𝒛c∗\\bm\{\\xi\}\_\{t\}^\{R\}=\\bm\{g\}\(Y\_\{t\}\)\-\\bm\{G\}\_\{c\}\(Y\_\{t\}\)\\bm\{z\}\_\{c\}^\{\*\}\.

###### Assumption 5\(Affine residual moment\)\.

For the requiredp≥1p\\geq 1,𝔼ζ​\[V​\(Y0\)1/2​‖𝛏0R‖2p\]<∞\\mathbb\{E\}\_\{\\zeta\}\[V\(Y\_\{0\}\)^\{1/2\}\\\|\\bm\{\\xi\}\_\{0\}^\{R\}\\\|\_\{2\}^\{p\}\]<\\infty, whereζ\\zetais the invariant law of the RETD driver\. We avoid the symbolπ\\pihere because it already denotes the target policy\.

### 4\.4Checkability and Coverage of the Constant\-Step Assumptions

Assumptions[3](https://arxiv.org/html/2609.19170#Thmassumption3)–[4](https://arxiv.org/html/2609.19170#Thmassumption4)are not automatic in a finite MDP\. The augmented state contains the unbounded follow\-on trace, so the driver is a general\-state\-space Markov chain even when𝒮\\mathcal\{S\}and𝒜\\mathcal\{A\}are finite\. A conservative directly checkable route is the uniformly contractive conditionγ​ρ​\(s,a\)≤r<1\\gamma\\rho\(s,a\)\\leq r<1for every covered\(s,a\)\(s,a\), combined with a geometric drift/minorization certificate for the resulting bounded augmented driver\. ThenFt≤max⁡\{F0,\(1−r\)−1\}F\_\{t\}\\leq\\max\\\{F\_\{0\},\(1\-r\)^\{\-1\}\\\}, the sample matrices are bounded, and the matrix\-growth part becomes immediate\. This sufficient condition does not cover the trace\-amplifying runs used by the two counterexamples\.

The heavy\-tailed two\-state and Baird instances are analyzed outside this small\-stepsize route because explicit super\-Lyapunov constantsb0,B0,δ,Vb\_\{0\},B\_\{0\},\\delta,Vare unavailable for their augmented drivers\. Their product signs instead follow from environment\-specific regenerative or projective certificates\. By contrast, a bounded\-trace finite MDP satisfying the uniformly contractive condition above is covered once its standard drift/minorization constants are supplied\. This separation prevents the general theorem from being claimed for the two stress tests without the required driver certificate\.

The external tool is Theorem 1 of[Durmus et al\. \(2021\)](https://arxiv.org/html/2609.19170#bib.bib22)\. Under their UE1 driver condition, A1 matrix\-growth condition, and the Hurwitz condition for the negative stationary mean matrix, that theorem gives, for eachp≥1p\\geq 1, a constructive positive sufficient thresholdα∞,p\\alpha\_\{\\infty,p\}such that

\(𝔼y‖∏k=m\+1n\(𝑰−αk𝒜\(Yk\)\)‖2p\)1/p≤Cpexp\[−a4∑k=m\+1nαk\]V\(y\)1/\(2​p\)\.\\left\(\\mathbb\{E\}\_\{y\}\\left\\\|\\prod\_\{k=m\+1\}^\{n\}\(\\bm\{I\}\-\\alpha\_\{k\}\\mathcal\{A\}\(Y\_\{k\}\)\)\\right\\\|\_\{2\}^\{p\}\\right\)^\{1/p\}\\leq C\_\{p\}\\exp\\left\[\-\\frac\{a\}\{4\}\\sum\_\{k=m\+1\}^\{n\}\\alpha\_\{k\}\\right\]V\(y\)^\{1/\(2p\)\}\.Here𝑸≻0\\bm\{Q\}\\succ 0solves𝒜¯𝖳​𝑸\+𝑸​𝒜¯=𝑰\\overline\{\\mathcal\{A\}\}^\{\\mathsf\{T\}\}\\bm\{Q\}\+\\bm\{Q\}\\overline\{\\mathcal\{A\}\}=\\bm\{I\}anda=\(2​‖𝑸‖2\)−1a=\(2\\\|\\bm\{Q\}\\\|\_\{2\}\)^\{\-1\}\. The proof below identifies the RETD driver, matrix family, stationary mean, Lyapunov matrix, and growth constant required by the quoted theorem\. In particular, the substitution is

The block lengthhhand the thresholdα∞,p\\alpha\_\{\\infty,p\}are then exactly those in Equations \(89\) and \(90\) of[Durmus et al\. \(2021\)](https://arxiv.org/html/2609.19170#bib.bib22)after this substitution\. This table is also the definition used below; no additional undefined threshold function is introduced\.

###### Theorem 5\(Constant\-stepsize RETD\)\.

Let Assumptions[1](https://arxiv.org/html/2609.19170#Thmassumption1)–[4](https://arxiv.org/html/2609.19170#Thmassumption4)hold\. Fixp≥1p\\geq 1, and let𝐐c≻0\\bm\{Q\}\_\{c\}\\succ 0be the unique solution of

𝑮c𝖳​𝑸c\+𝑸c​𝑮c=𝑰,κQc=λmax​\(𝑸c\)λmin​\(𝑸c\),ac=\(2​‖𝑸c‖2\)−1\.\\bm\{G\}\_\{c\}^\{\\mathsf\{T\}\}\\bm\{Q\}\_\{c\}\+\\bm\{Q\}\_\{c\}\\bm\{G\}\_\{c\}=\\bm\{I\},\\qquad\\kappa\_\{Q\_\{c\}\}=\\frac\{\\lambda\_\{\\max\}\(\\bm\{Q\}\_\{c\}\)\}\{\\lambda\_\{\\min\}\(\\bm\{Q\}\_\{c\}\)\},\\qquad a\_\{c\}=\(2\\\|\\bm\{Q\}\_\{c\}\\\|\_\{2\}\)^\{\-1\}\.\(23\)LetαD,pR​\(c\)\\alpha\_\{D,p\}^\{R\}\(c\)be the positive numberα∞,p\\alpha\_\{\\infty,p\}in Equation \(90\) of[Durmus et al\. \(2021\)](https://arxiv.org/html/2609.19170#bib.bib22), with the complete substitution displayed above and with its block lengthhhdefined by their Equation \(89\):

αD,pR​\(c\):=α∞,p\|Z=Y,𝒜¯=𝑮c​\(⋅\),𝒜=𝑮c,𝑸=𝑸c,CA=CG​\(c\)\.\\alpha\_\{D,p\}^\{R\}\(c\):=\\alpha\_\{\\infty,p\}\\big\|\_\{Z=Y,\\,\\overline\{\\mathcal\{A\}\}=\\bm\{G\}\_\{c\}\(\\cdot\),\\,\\mathcal\{A\}=\\bm\{G\}\_\{c\},\\,\\bm\{Q\}=\\bm\{Q\}\_\{c\},\\,C\_\{A\}=C\_\{G\}\(c\)\}\.The number is computable after the UE1 drift and minorization constants are supplied, is sufficient rather than sharp, and exists by the quoted theorem\. Define

α¯pR​\(c\)=min⁡\{αD,pR​\(c\),2​m¯c‖𝑮c‖22,21\+c\}\.\\overline\{\\alpha\}\_\{p\}^\{R\}\(c\)=\\min\\left\\\{\\alpha\_\{D,p\}^\{R\}\(c\),\\frac\{2\\underline\{m\}\_\{c\}\}\{\\\|\\bm\{G\}\_\{c\}\\\|\_\{2\}^\{2\}\},\\frac\{2\}\{1\+c\}\\right\\\}\.\(24\)The first term is the external random\-product threshold\. The second and third terms are additional transparent RETD restrictions: they make the deterministic mean step contractive and give the isolated auxiliary retention\|1−α⁡\(1\+c\)\|<1\|1\-\\alpha\(1\+c\)\|<1\. They are not assumptions of Theorem 1 of[Durmus et al\. \(2021\)](https://arxiv.org/html/2609.19170#bib.bib22)\. If0<α<α¯pR​\(c\)0<\\alpha<\\overline\{\\alpha\}\_\{p\}^\{R\}\(c\), put

𝑨t\(c\)=𝑰−α𝑮c\(Yt\),𝚽R\(t,s;c\)=𝑨t\(c\)⋯𝑨s\(c\),s≤t,𝚽R\(t,t\+1;c\)=𝑰\.\\bm\{A\}\_\{t\}\(c\)=\\bm\{I\}\-\\alpha\\bm\{G\}\_\{c\}\(Y\_\{t\}\),\\qquad\\bm\{\\Phi\}\_\{R\}\(t,s;c\)=\\bm\{A\}\_\{t\}\(c\)\\cdots\\bm\{A\}\_\{s\}\(c\),\\quad s\\leq t,\\qquad\\bm\{\\Phi\}\_\{R\}\(t,t\+1;c\)=\\bm\{I\}\.Then, with time increasing from right to left,

\(𝔼y​‖𝚽R​\(n,1,c\)‖2p\)1/p≤Cc,p​e−\(ac/4\)​α​n​V​\(y\)1/\(2​p\)\.\\left\(\\mathbb\{E\}\_\{y\}\\\|\\bm\{\\Phi\}\_\{R\}\(n,1;c\)\\\|\_\{2\}^\{p\}\\right\)^\{1/p\}\\leq C\_\{c,p\}e^\{\-\(a\_\{c\}/4\)\\alpha n\}V\(y\)^\{1/\(2p\)\}\.\(25\)Consequently,

λR​\(α,c\):=lim supn→∞1n​log⁡‖𝚽R​\(n,1,c\)‖2≤−ac​α4=−α8​‖𝑸c‖2<0a\.s\.\\lambda\_\{R\}\(\\alpha,c\):=\\limsup\_\{n\\to\\infty\}\\frac\{1\}\{n\}\\log\\\|\\bm\{\\Phi\}\_\{R\}\(n,1;c\)\\\|\_\{2\}\\leq\-\\frac\{a\_\{c\}\\alpha\}\{4\}=\-\\frac\{\\alpha\}\{8\\\|\\bm\{Q\}\_\{c\}\\\|\_\{2\}\}<0\\quad\\text\{a\.s\.\}\(26\)If Assumption[5](https://arxiv.org/html/2609.19170#Thmassumption5)also holds on a two\-sided stationary extension of the driver, the centered affine recursion has the unique finite\-ppth\-moment stationary causal solution

𝒆t∘=α​∑j=0∞𝚽R​\(t,t−j\+1,c\)​𝝃t−jR,\\bm\{e\}\_\{t\}^\{\\circ\}=\\alpha\\sum\_\{j=0\}^\{\\infty\}\\bm\{\\Phi\}\_\{R\}\(t,t\-j\+1;c\)\\bm\{\\xi\}\_\{t\-j\}^\{R\},\(27\)and every two solutions driven by the same samples forget their initial conditions exponentially\. If𝛏tR≡𝟎\\bm\{\\xi\}\_\{t\}^\{R\}\\equiv\\bm\{0\}sample by sample, the iterate itself converges exponentially to𝐳c∗\\bm\{z\}\_\{c\}^\{\*\}almost surely\.

###### Proof\.

Step 1: the regularizer changes each sampled product\.From Equation[19](https://arxiv.org/html/2609.19170#S4.E19),

𝑮c​\(y\)=𝑮0​\(y\)\+c​𝒆ω​𝒆ω𝖳\.\\bm\{G\}\_\{c\}\(y\)=\\bm\{G\}\_\{0\}\(y\)\+c\\bm\{e\}\_\{\\omega\}\\bm\{e\}\_\{\\omega\}^\{\\mathsf\{T\}\}\.\(28\)Thus the product in the theorem depends oncc\. On a zero\-importance\-ratio sample, its auxiliary diagonal entry is1−α⁡\(1\+c\)1\-\\alpha\(1\+c\)\. The final term in Equation[24](https://arxiv.org/html/2609.19170#S4.E24)ensures that this scalar mode has magnitude less than one\.

Step 2:cccreates a verified mean margin\.Assumption[2](https://arxiv.org/html/2609.19170#Thmassumption2)makesΔc\>0\\Delta\_\{c\}\>0\. The congruence in Equation[16](https://arxiv.org/html/2609.19170#S4.E16)therefore provesSym⁡\(𝑮c\)≻0\\Sym\(\\bm\{G\}\_\{c\}\)\\succ 0and hence that−𝑮c\-\\bm\{G\}\_\{c\}is Hurwitz\. Furthermore, Equation[17](https://arxiv.org/html/2609.19170#S4.E17)gives𝒚𝖳​Sym⁡\(𝑮c\)​𝒚≥m¯c​‖𝒚‖22\\bm\{y\}^\{\\mathsf\{T\}\}\\Sym\(\\bm\{G\}\_\{c\}\)\\bm\{y\}\\geq\\underline\{m\}\_\{c\}\\\|\\bm\{y\}\\\|\_\{2\}^\{2\}for every𝒚\\bm\{y\}\. Expanding a deterministic mean step yields

‖\(𝑰−α​𝑮c\)​𝒚‖22\\displaystyle\\\|\(\\bm\{I\}\-\\alpha\\bm\{G\}\_\{c\}\)\\bm\{y\}\\\|\_\{2\}^\{2\}=‖𝒚‖22−2​α​𝒚𝖳​Sym⁡\(𝑮c\)​𝒚\+α2​‖𝑮c​𝒚‖22\\displaystyle=\\\|\\bm\{y\}\\\|\_\{2\}^\{2\}\-2\\alpha\\bm\{y\}^\{\\mathsf\{T\}\}\\Sym\(\\bm\{G\}\_\{c\}\)\\bm\{y\}\+\\alpha^\{2\}\\\|\\bm\{G\}\_\{c\}\\bm\{y\}\\\|\_\{2\}^\{2\}≤\{1−2​α​m¯c\+α2​‖𝑮c‖22\}​‖𝒚‖22\.\\displaystyle\\leq\\\{1\-2\\alpha\\underline\{m\}\_\{c\}\+\\alpha^\{2\}\\\|\\bm\{G\}\_\{c\}\\\|\_\{2\}^\{2\}\\\}\\\|\\bm\{y\}\\\|\_\{2\}^\{2\}\.The second term in Equation[24](https://arxiv.org/html/2609.19170#S4.E24)makes this multiplier strictly less than one\. The same Hurwitz conclusion verifies condition A2 of the quoted random\-product theorem, without assuming contraction of the sampled product\.

Step 3:ccenters the rate and the stochastic threshold\.Since−𝑮c\-\\bm\{G\}\_\{c\}is Hurwitz,𝑸c=∫0∞e−𝑮c𝖳​s​e−𝑮c​s​𝑑s\\bm\{Q\}\_\{c\}=\\int\_\{0\}^\{\\infty\}e^\{\-\\bm\{G\}\_\{c\}^\{\\mathsf\{T\}\}s\}e^\{\-\\bm\{G\}\_\{c\}s\}\\,\\mathrm\{d\}sis the unique positive\-definite solution of Equation[23](https://arxiv.org/html/2609.19170#S4.E23)\. Henceccchanges𝑸c\\bm\{Q\}\_\{c\}and the decay constantaca\_\{c\}\. Independently, Equation[22](https://arxiv.org/html/2609.19170#S4.E22)shows thatccchanges the one\-step growth constantCG​\(c\)C\_\{G\}\(c\)\. Assumption[3](https://arxiv.org/html/2609.19170#Thmassumption3)verifies UE1, Assumption[4](https://arxiv.org/html/2609.19170#Thmassumption4)verifies A1, and Step 2 verifies A2 for the actual RETD matrix family\. The external theorem therefore applies at every constant stepsize belowαD,pR​\(c\)\\alpha\_\{D,p\}^\{R\}\(c\)and gives Equation[25](https://arxiv.org/html/2609.19170#S4.E25)\.

Step 4: moment contraction gives a negative top exponent\.Fix a rationalε∈\(0,ac​α/4\)\\varepsilon\\in\(0,a\_\{c\}\\alpha/4\)\. Markov’s inequality and Equation[25](https://arxiv.org/html/2609.19170#S4.E25)give

ℙy​\(‖𝚽R​\(n,1,c\)‖2\>e−\(ac​α/4−ε\)​n\)≤Cc,pp​V​\(y\)1/2​e−p​ε​n\.\\mathbb\{P\}\_\{y\}\\\!\\left\(\\\|\\bm\{\\Phi\}\_\{R\}\(n,1;c\)\\\|\_\{2\}\>e^\{\-\(a\_\{c\}\\alpha/4\-\\varepsilon\)n\}\\right\)\\leq C\_\{c,p\}^\{p\}V\(y\)^\{1/2\}e^\{\-p\\varepsilon n\}\.The right\-hand side is summable\. Borel–Cantelli implies that the displayed event occurs only finitely often\. Taking the intersection over the countable set of positive rationalε<ac​α/4\\varepsilon<a\_\{c\}\\alpha/4and then sendingε↓0\\varepsilon\\downarrow 0proves Equation[26](https://arxiv.org/html/2609.19170#S4.E26)on one probability\-one event\.

Step 5: affine stability and uniqueness\.Letχ=ac​α/4\\chi=a\_\{c\}\\alpha/4\. Forj≥1j\\geq 1, condition first onYt−jY\_\{t\-j\}\. The Markov property and Equation[25](https://arxiv.org/html/2609.19170#S4.E25)give

𝔼ζ​\[‖𝚽R​\(t,t−j\+1,c\)​𝝃t−jR‖2p\]\\displaystyle\\mathbb\{E\}\_\{\\zeta\}\\\!\\left\[\\\|\\bm\{\\Phi\}\_\{R\}\(t,t\-j\+1;c\)\\bm\{\\xi\}\_\{t\-j\}^\{R\}\\\|\_\{2\}^\{p\}\\right\]≤Cc,pp​e−p​χ​j​𝔼ζ​\[V​\(Yt−j\)1/2​‖𝝃t−jR‖2p\]\.\\displaystyle\\quad\\leq C\_\{c,p\}^\{p\}e^\{\-p\\chi j\}\\mathbb\{E\}\_\{\\zeta\}\\\!\\left\[V\(Y\_\{t\-j\}\)^\{1/2\}\\\|\\bm\{\\xi\}\_\{t\-j\}^\{R\}\\\|\_\{2\}^\{p\}\\right\]\.The expectation on the right is finite and independent oft,jt,jby Assumption[5](https://arxiv.org/html/2609.19170#Thmassumption5)and stationarity\. Thej=0j=0term is𝝃tR\\bm\{\\xi\}\_\{t\}^\{R\}because𝚽R​\(t,t\+1,c\)=𝑰\\bm\{\\Phi\}\_\{R\}\(t,t\+1;c\)=\\bm\{I\}, and it is also inLpL^\{p\}\. Hence∑j=0∞‖𝚽R​\(t,t−j\+1,c\)​𝝃t−jR‖Lp<∞\\sum\_\{j=0\}^\{\\infty\}\\\|\\bm\{\\Phi\}\_\{R\}\(t,t\-j\+1;c\)\\bm\{\\xi\}\_\{t\-j\}^\{R\}\\\|\_\{L^\{p\}\}<\\infty\. Minkowski’s inequality proves that Equation[27](https://arxiv.org/html/2609.19170#S4.E27)converges inLpL^\{p\}\. Sincep≥1p\\geq 1, the same summability also gives∑j𝔼​‖𝚽R​𝝃R‖2<∞\\sum\_\{j\}\\mathbb\{E\}\\\|\\bm\{\\Phi\}\_\{R\}\\bm\{\\xi\}^\{R\}\\\|\_\{2\}<\\infty; Tonelli’s theorem therefore gives absolute convergence almost surely\. The limit is causal and stationary\. Separating thej=0j=0term and using𝚽R​\(t,t−j\+1,c\)=𝑨t​\(c\)​𝚽R​\(t−1,t−j\+1,c\)\\bm\{\\Phi\}\_\{R\}\(t,t\-j\+1;c\)=\\bm\{A\}\_\{t\}\(c\)\\bm\{\\Phi\}\_\{R\}\(t\-1,t\-j\+1;c\)forj≥1j\\geq 1shows directly that𝒆t∘=𝑨t​\(c\)​𝒆t−1∘\+α​𝝃tR\\bm\{e\}\_\{t\}^\{\\circ\}=\\bm\{A\}\_\{t\}\(c\)\\bm\{e\}\_\{t\-1\}^\{\\circ\}\+\\alpha\\bm\{\\xi\}\_\{t\}^\{R\}, so it is a stationary solution of the centered affine recursion\.

For any two solutions driven by the same two\-sided sample path,𝒆t−𝒆~t=𝚽R​\(t,s\+1,c\)​\(𝒆s−𝒆~s\)\\bm\{e\}\_\{t\}\-\\widetilde\{\\bm\{e\}\}\_\{t\}=\\bm\{\\Phi\}\_\{R\}\(t,s\+1;c\)\(\\bm\{e\}\_\{s\}\-\\widetilde\{\\bm\{e\}\}\_\{s\}\)fort\>st\>s\. Step 4 makes the norm of this difference decay with exponent at most−χ\-\\chifor every finite initial difference\. If the difference is stationary, its distribution is independent oftt, while the displayed product makes it converge to zero in probability; its stationary distribution must therefore be the point mass at zero\. This proves uniqueness, while arbitrary finite initial differences are forgotten exponentially\. If𝝃tR≡𝟎\\bm\{\\xi\}\_\{t\}^\{R\}\\equiv\\bm\{0\}sample by sample, the stationary series is identically zero and the same homogeneous contraction gives𝒛t→𝒛c∗\\bm\{z\}\_\{t\}\\to\\bm\{z\}\_\{c\}^\{\*\}exponentially\.

Finally, increasingccenlargesΔc\\Delta\_\{c\}but also changes‖𝑮c‖2\\\|\\bm\{G\}\_\{c\}\\\|\_\{2\},𝑸c\\bm\{Q\}\_\{c\},CG​\(c\)C\_\{G\}\(c\), and the scalar factor1−α⁡\(1\+c\)1\-\\alpha\(1\+c\)\. The theorem therefore yields the feasible set𝒞α,pR=\{c\>cmin:α<α¯pR​\(c\)\}⊂\(cmin,2/α−1\)\\mathcal\{C\}\_\{\\alpha,p\}^\{R\}=\\\{c\>c\_\{\\min\}:\\alpha<\\overline\{\\alpha\}\_\{p\}^\{R\}\(c\)\\\}\\subset\(c\_\{\\min\},2/\\alpha\-1\), which need not expand monotonically withcc\. ∎

### 4\.5Comparison with ETD under the Constant\-Stepsize Theorem

The external theorem of[Durmus et al\. \(2021\)](https://arxiv.org/html/2609.19170#bib.bib22)is algorithm agnostic; Theorem[5](https://arxiv.org/html/2609.19170#Thmtheorem5)is not\. ETD and RETD supply different matrix families:

𝒜E​\(Yt\)=Xt​𝒙t​𝝍t𝖳,𝒜R​\(Yt,c\)=\[Xt​𝒙t​𝝍t𝖳𝒙tXt​𝝍t𝖳1\+c\]\.\\mathcal\{A\}\_\{E\}\(Y\_\{t\}\)=X\_\{t\}\\bm\{x\}\_\{t\}\\bm\{\\psi\}\_\{t\}^\{\\mathsf\{T\}\},\\qquad\\mathcal\{A\}\_\{R\}\(Y\_\{t\};c\)=\\begin\{bmatrix\}X\_\{t\}\\bm\{x\}\_\{t\}\\bm\{\\psi\}\_\{t\}^\{\\mathsf\{T\}\}&\\bm\{x\}\_\{t\}\\\\ X\_\{t\}\\bm\{\\psi\}\_\{t\}^\{\\mathsf\{T\}\}&1\+c\\end\{bmatrix\}\.\(29\)Consequently they have different dimensions, sample products, stationary mean matrices, Lyapunov equations, growth constants, and admissible stepsize thresholds\. ETD must therefore be analyzed with its own matrix family; it is not a principal submatrix of the RETD recursion\.

ETD satisfies the same*external analytical tool*on a particular task when its augmented trace driver verifies UE1, its entries satisfy the corresponding growth bound,−𝑨\-\\bm\{A\}is Hurwitz, andα\\alphais below the threshold obtained from𝒜E\\mathcal\{A\}\_\{E\}\. Under these conditions ETD has moment contraction and a negative top exponent, which covers sufficiently small stepsizes in suitable environments\.

There are three distinct reasons an ETD certificate can fail\. First,−𝑨\-\\bm\{A\}may not be Hurwitz\. Second, the unbounded trace driver may fail the required drift or growth condition\. Third, even when the first two requirements do hold, the specified stepsize may still lie outside ETD’s own product\-stability region\. The two\-state counterexample proves directly that the third requirement fails atα=0\.1\\alpha=0\.1: ETD’s exact scalar exponent is positive, so no valid negative\-exponent theorem can cover that matrix product at that stepsize\. The example also has𝔼​Ft2=∞\\mathbb\{E\}F\_\{t\}^\{2\}=\\infty, and we have not supplied UE1 or A1 constants for its augmented driver\. It would therefore be incorrect to claim that the first two external conditions have been verified there\. The Baird stress test has the same coverage boundary: its signs are established by the separate analysis in Section[5](https://arxiv.org/html/2609.19170#S5), not by Theorem[5](https://arxiv.org/html/2609.19170#Thmtheorem5)\. RETD does not use the ETD matrices; it uses the block matrices in Equation[29](https://arxiv.org/html/2609.19170#S4.E29), and atc=4c=4their complete cycle product has a negative exponent\.

Whenρt=0\\rho\_\{t\}=0, ETD supplies𝑰\\bm\{I\}, while RETD supplies Equation[9](https://arxiv.org/html/2609.19170#S3.E9)\. The parametercctherefore changes an actual post\-shock response, then propagates throughΔc\\Delta\_\{c\},𝑸c\\bm\{Q\}\_\{c\},CG​\(c\)C\_\{G\}\(c\), andα¯pR​\(c\)\\overline\{\\alpha\}\_\{p\}^\{R\}\(c\)\. These quantities determine separate stability regions for ETD and RETD\.

## 5Numerical Study of ETD and RETD on Baird’s Counterexample

We analyze Baird’s seven\-state geometry using the same quantities as in the two\-state counterexample: regenerative trace cycles, deterministic mean maps, sampled cycle matrices, and prediction\-relevant top Lyapunov exponents\.

### 5\.1The Counterexample and the Identifiable Prediction Space

There are six upper statesu1,…,u6u\_\{1\},\\ldots,u\_\{6\}and one lower stateℓ\\ell\. From every state, the solid action moves toℓ\\ell, whereas the dashed action moves uniformly to an upper state\. The target policy always selects solid\. The behavior policy selects solid with probability1/71/7and dashed with probability6/76/7\. Henceρt=7\\rho\_\{t\}=7on solid,ρt=0\\rho\_\{t\}=0on dashed, anddμ​\(s\)=1/7d\_\{\\mu\}\(s\)=1/7\.

u1u\_\{1\}u2u\_\{2\}u3u\_\{3\}u4u\_\{4\}u5u\_\{5\}u6u\_\{6\}ℓ\\ellsix upper statessolid: toℓ\\ellπ⁡\(solid\)=1\\pi\(\\mathrm\{solid\}\)=1dashed: uniform overuiu\_\{i\}μ⁡\(solid\)=1/7\\mu\(\\mathrm\{solid\}\)=1/7Figure 2:Baird’s seven\-state counterexample\. The behavior policy uses the blue solid transition with probability1/71/7and otherwise uses the dashed uniform\-to\-upper transition\. The dashed arcs denote the common kernel from every state\.We useγ=0\.99\\gamma=0\.99, zero rewards, and the standard featuresϕ⁡\(ui\)=2​𝒆i\+𝒆8\\bm\{\\phi\}\(u\_\{i\}\)=2\\bm\{e\}\_\{i\}\+\\bm\{e\}\_\{8\}andϕ⁡\(ℓ\)=𝒆7\+2​𝒆8\\bm\{\\phi\}\(\\ell\)=\\bm\{e\}\_\{7\}\+2\\bm\{e\}\_\{8\}\. The7×87\\times 8feature matrix has rank seven, with

ker⁡𝚽=span⁡\{\(−1,−1,−1,−1,−1,−1,−4,2\)𝖳\}\.\\ker\\bm\{\\Phi\}=\\operatorname\{span\}\\\{\(\-1,\-1,\-1,\-1,\-1,\-1,\-4,2\)^\{\\mathsf\{T\}\}\\\}\.Thus an eight\-dimensional parameter product contains one prediction\-null direction\. We remove it and work in the seven\-dimensional prediction coordinate𝒚=𝚽​𝜽\\bm\{y\}=\\bm\{\\Phi\}\\bm\{\\theta\}\. Because rewards are zero, the target value is exactly𝟎\\bm\{0\}and both error recursions are homogeneous\.

### 5\.2The Common Regenerative Cycle

A dashed action has zero importance ratio and resets the next follow\-on trace to one\. LetLLbe the number of consecutive solid actions before the next dashed action\. Successive cycles are independent, withPr⁡\(L=ℓ\)=\(6/7\)​\(1/7\)ℓ\\Pr\(L=\\ell\)=\(6/7\)\(1/7\)^\{\\ell\}and𝔼⁡\[L\+1\]=7/6\\mathbb\{E\}\[L\+1\]=7/6\. Within a solid run,

F0=1,Fk\+1=1\+6\.93​Fk,Fk=6\.93k\+1−15\.93\.F\_\{0\}=1,\\qquad F\_\{k\+1\}=1\+6\.93F\_\{k\},\\qquad F\_\{k\}=\\frac\{6\.93^\{k\+1\}\-1\}\{5\.93\}\.\(30\)Consequently,𝔼⁡\[Ftr\]<∞\\mathbb\{E\}\[F\_\{t\}^\{r\}\]<\\inftyif and only if6\.93r/7<16\.93^\{r\}/7<1\. The thresholdlog⁡7/log⁡6\.93≈1\.00519\\log 7/\\log 6\.93\\approx 1\.00519lies between one and two: the trace has a finite mean and infinite variance\. This reproduces the shock source in the two\-state counterexample, but it still does not establish the sign of either complete matrix product\.

The effect of a long run is explicit\. On a lower\-to\-lower solid transition, the scalar multiplier in the lower\-prediction direction isdk=1−α​ρ​Fk​\(1−γ\)​‖ϕ⁡\(ℓ\)‖2=1−0\.35​α​Fkd\_\{k\}=1\-\\alpha\\rho F\_\{k\}\(1\-\\gamma\)\\\|\\bm\{\\phi\}\(\\ell\)\\\|^\{2\}=1\-0\.35\\alpha F\_\{k\}\. SinceFk≍6\.93kF\_\{k\}\\asymp 6\.93^\{k\}, a length\-LLrun eventually accumulates∑k<Llog⁡\|dk\|=12​log⁡\(6\.93\)​L2\+O⁡\(L\)\\sum\_\{k<L\}\\log\|d\_\{k\}\|=\\tfrac\{1\}\{2\}\\log\(6\.93\)L^\{2\}\+O\(L\)\. Whether the algorithm contracts or expands depends on how the final dashed sample completes this cycle\.

### 5\.3ETD: Stable Mean Dynamics, Expanding Sampled Dynamics

The emphatic weights solve𝒇𝖳​\(𝑰−γ​𝑷π\)=𝒅μ𝖳\\bm\{f\}^\{\\mathsf\{T\}\}\(\\bm\{I\}\-\\gamma\\bm\{P\}\_\{\\pi\}\)=\\bm\{d\}\_\{\\mu\}^\{\\mathsf\{T\}\}and aref⁡\(ui\)=1/7f\(u\_\{i\}\)=1/7andf⁡\(ℓ\)=694/7f\(\\ell\)=694/7\. In the identifiable coordinate, the ETD mean matrix has five eigenvalues4/74/7and two eigenvalues\(1641±37​649\)/700=0\.99772471,3\.69084672\(1641\\pm 37\\sqrt\{649\}\)/700=0\.99772471,3\.69084672\. Therefore, atα=0\.01\\alpha=0\.01,ρ⁡\(𝑰−α​𝑨E\)=0\.99428571<1\\rho\(\\bm\{I\}\-\\alpha\\bm\{A\}\_\{E\}\)=0\.99428571<1\. The deterministic ETD mean map is strictly contractive\.

To analyze the samples, define𝑲=𝚽​𝚽𝖳\\bm\{K\}=\\bm\{\\Phi\}\\bm\{\\Phi\}^\{\\mathsf\{T\}\},𝒌s=𝑲​𝒆s\\bm\{k\}\_\{s\}=\\bm\{K\}\\bm\{e\}\_\{s\},𝒗s,s′=𝒆s−γ​𝒆s′\\bm\{v\}\_\{s,s^\{\\prime\}\}=\\bm\{e\}\_\{s\}\-\\gamma\\bm\{e\}\_\{s^\{\\prime\}\}, andXt=Ft​ρtX\_\{t\}=F\_\{t\}\\rho\_\{t\}\. The prediction error obeys

𝒚t\+1E=𝑴tE​𝒚tE,𝑴tE=𝑰−α​Xt​𝒌St​𝒗St,St\+1𝖳\.\\bm\{y\}\_\{t\+1\}^\{E\}=\\bm\{M\}\_\{t\}^\{E\}\\bm\{y\}\_\{t\}^\{E\},\\qquad\\bm\{M\}\_\{t\}^\{E\}=\\bm\{I\}\-\\alpha X\_\{t\}\\bm\{k\}\_\{S\_\{t\}\}\\bm\{v\}\_\{S\_\{t\},S\_\{t\+1\}\}^\{\\mathsf\{T\}\}\.Let𝑴uE​\(i,F0\)\\bm\{M\}\_\{u\}^\{E\}\(i,F\_\{0\}\)denote the first upper\-to\-lower solid factor and𝑴ℓE​\(Fk\)\\bm\{M\}\_\{\\ell\}^\{E\}\(F\_\{k\}\)a later lower\-to\-lower solid factor\. The final dashed ETD factor is the identity becauseXt=0X\_\{t\}=0\. Hence

𝑪E\(i,0\)=𝑰,𝑪E\(i,L\)=𝑴ℓE\(FL−1\)⋯𝑴ℓE\(F1\)𝑴uE\(i,F0\),L≥1\.\\bm\{C\}\_\{E\}\(i,0\)=\\bm\{I\},\\qquad\\bm\{C\}\_\{E\}\(i,L\)=\\bm\{M\}\_\{\\ell\}^\{E\}\(F\_\{L\-1\}\)\\cdots\\bm\{M\}\_\{\\ell\}^\{E\}\(F\_\{1\}\)\\bm\{M\}\_\{u\}^\{E\}\(i,F\_\{0\}\),\\quad L\\geq 1\.\(31\)The prediction\-relevant exponent is

λEpred=limN→∞log∥𝑪E,N⋯𝑪E,1∥∑n=1N\(Ln\+1\)\.\\lambda\_\{E\}^\{\\mathrm\{pred\}\}=\\lim\_\{N\\to\\infty\}\\frac\{\\log\\\|\\bm\{C\}\_\{E,N\}\\cdots\\bm\{C\}\_\{E,1\}\\\|\}\{\\sum\_\{n=1\}^\{N\}\(L\_\{n\}\+1\)\}\.The geometric cycle law above and the quadratic log\-growth bound above give the logarithmic moments required for this limit\.

Atα=0\.01\\alpha=0\.01, a10,00010\{,\}000\-transition burn\-in followed by200,000200\{,\}000retained transitions in each of ten independent runs givesλ^Epred=0\.00138528±0\.00008328\\widehat\{\\lambda\}\_\{E\}^\{\\mathrm\{pred\}\}=0\.00138528\\pm 0\.00008328, where the uncertainty is the sample standard deviation across runs; all ten estimates are positive\. This is an empirical estimate and provides no analytic lower bound\. The mechanism is the same as in Section[2\.7](https://arxiv.org/html/2609.19170#S2.SS7): long solid runs create large factors, and the subsequent zero\-ratio ETD factor is exactly𝑰\\bm\{I\}, so it cannot release a post\-shock correction\.

### 5\.4RETD: The Same Cycle Includes Recovery

For RETD the joint prediction error is\[\(𝒚tR\)𝖳,ωt\]𝖳\[\(\\bm\{y\}\_\{t\}^\{R\}\)^\{\\mathsf\{T\}\},\\omega\_\{t\}\]^\{\\mathsf\{T\}\}, with sample matrix

𝑴tR​\(c\)=𝑰−α​\[Xt​𝒌St​𝒗St,St\+1𝖳𝒌StXt​𝒗St,St\+1𝖳1\+c\]\.\\bm\{M\}\_\{t\}^\{R\}\(c\)=\\bm\{I\}\-\\alpha\\begin\{bmatrix\}X\_\{t\}\\bm\{k\}\_\{S\_\{t\}\}\\bm\{v\}\_\{S\_\{t\},S\_\{t\+1\}\}^\{\\mathsf\{T\}\}&\\bm\{k\}\_\{S\_\{t\}\}\\\\ X\_\{t\}\\bm\{v\}\_\{S\_\{t\},S\_\{t\+1\}\}^\{\\mathsf\{T\}\}&1\+c\\end\{bmatrix\}\.Its Baird mean matrix satisfiesSym⁡\(𝑮c\)≻0\\Sym\(\\bm\{G\}\_\{c\}\)\\succ 0if and only ifc\>29403/109397c\>29403/109397, soc=0\.5c=0\.5is inside the certified region\. At\(α,c\)=\(0\.01,0\.5\)\(\\alpha,c\)=\(0\.01,0\.5\),

min⁡Re​spec⁡\(𝑮c\)=0\.24794122,ρ⁡\(𝑰−α​𝑮c\)=0\.99752059<1\.\\min\\operatorname\{Re\}\\spec\(\\bm\{G\}\_\{c\}\)=0\.24794122,\\qquad\\rho\(\\bm\{I\}\-\\alpha\\bm\{G\}\_\{c\}\)=0\.99752059<1\.The RETD mean map is stable, but it is slower than the ETD mean map reported above\.

On the final dashed sample, RETD applies

𝑫s​\(c\)=𝑰−α​\[𝟎𝒌s𝟎𝖳1\+c\],\\bm\{D\}\_\{s\}\(c\)=\\bm\{I\}\-\\alpha\\begin\{bmatrix\}\\bm\{0\}&\\bm\{k\}\_\{s\}\\\\ \\bm\{0\}^\{\\mathsf\{T\}\}&1\+c\\end\{bmatrix\},which is not the identity\. Its complete cycles are

𝑪R​\(i,0,c\)\\displaystyle\\bm\{C\}\_\{R\}\(i,0;c\)=𝑫ui​\(c\),\\displaystyle=\\bm\{D\}\_\{u\_\{i\}\}\(c\),𝑪R​\(i,L,c\)\\displaystyle\\bm\{C\}\_\{R\}\(i,L;c\)=𝑫ℓ\(c\)𝑴ℓR\(FL−1;c\)⋯𝑴ℓR\(F1;c\)𝑴uR\(i,F0;c\),L≥1\.\\displaystyle=\\bm\{D\}\_\{\\ell\}\(c\)\\bm\{M\}\_\{\\ell\}^\{R\}\(F\_\{L\-1\};c\)\\cdots\\bm\{M\}\_\{\\ell\}^\{R\}\(F\_\{1\};c\)\\bm\{M\}\_\{u\}^\{R\}\(i,F\_\{0\};c\),\\quad L\\geq 1\.The only structural change relative to Equation[31](https://arxiv.org/html/2609.19170#S5.E31)is precisely the change identified in the two\-state counterexample: RETD uses the zero\-ratio sample to release the stored correction\.

The same numerical setting givesλ^Rpred=−0\.00143906±0\.00006106\\widehat\{\\lambda\}\_\{R\}^\{\\mathrm\{pred\}\}=\-0\.00143906\\pm 0\.00006106, and all ten estimates are negative\. Here the negative sign also admits a rigorous cycle certificate\.

###### Proposition 6\(Certified Baird RETD contraction\)\.

For Baird RETD in the identifiable joint prediction coordinate atα=0\.01\\alpha=0\.01andc=0\.5c=0\.5,

λRpred≤−10−4<0\.\\lambda\_\{R\}^\{\\mathrm\{pred\}\}\\leq\-10^\{\-4\}<0\.

###### Proof\.

The dashed times generate iid cycles with probabilitiespL=\(6/7\)​\(1/7\)Lp\_\{L\}=\(6/7\)\(1/7\)^\{L\}and mean length𝔼⁡\[L\+1\]=7/6\\mathbb\{E\}\[L\+1\]=7/6\. We first verify that no logarithm below is applied to zero\. The matrix determinant lemma gives

det𝑴tR​\(c\)=\(1−α​c\)​\(1−α​Xt​𝒗St,St\+1𝖳​𝒌St\)−α\.\\det\\bm\{M\}\_\{t\}^\{R\}\(c\)=\(1\-\\alpha c\)\\left\(1\-\\alpha X\_\{t\}\\bm\{v\}\_\{S\_\{t\},S\_\{t\+1\}\}^\{\\mathsf\{T\}\}\\bm\{k\}\_\{S\_\{t\}\}\\right\)\-\\alpha\.At\(α,c,γ\)=\(1/100,1/2,99/100\)\(\\alpha,c,\\gamma\)=\(1/100,1/2,99/100\), a dashed factor has determinant197/200197/200\. An upper\-to\-lower solid factor has determinant774657/1000000774657/1000000\. For a lower\-to\-lower solid factor,𝒗ℓ,ℓ𝖳​𝒌ℓ=1/20\\bm\{v\}\_\{\\ell,\\ell\}^\{\\mathsf\{T\}\}\\bm\{k\}\_\{\\ell\}=1/20andX=7​FkX=7F\_\{k\}, so

det𝑴ℓR​\(Fk,1/2\)=394000−1393​Fk400000\.\\det\\bm\{M\}\_\{\\ell\}^\{R\}\(F\_\{k\};1/2\)=\\frac\{394000\-1393F\_\{k\}\}\{400000\}\.It could vanish only ifFk=394000/1393F\_\{k\}=394000/1393\. The fraction is reduced and1393=7⋅1991393=7\\cdot 199, whereas the recurrence in Equation[30](https://arxiv.org/html/2609.19170#S5.E30)givesFk∈ℤ⁡\[1/10\]F\_\{k\}\\in\\mathbb\{Z\}\[1/10\], whose reduced denominator contains only the primes22and55\. Hence every sample and cycle matrix is nonsingular\.

We next control every projective direction, rather than a finite set of sampled initial vectors\. Use the following positive\-definite matrix𝑷\\bm\{P\}, invariant under permutations of the six upper states, whose independent entry classes are

Pi​i\\displaystyle P\_\{ii\}=0\.7473190564759538\(i≤6\),\\displaystyle=0\.7473190564759538\\quad\(i\\leq 6\),Pi​j\\displaystyle P\_\{ij\}=0\.13894231222775422\(i≠j≤6\),\\displaystyle=0\.13894231222775422\\quad\(i\\neq j\\leq 6\),Pi​ℓ\\displaystyle P\_\{i\\ell\}=−1\.4462721104545542,\\displaystyle=\-1\.4462721104545542,Pi​ω\\displaystyle P\_\{i\\omega\}=1\.3688386146118432,\\displaystyle=1\.3688386146118432,Pℓ​ℓ\\displaystyle P\_\{\\ell\\ell\}=9\.245683770562014,\\displaystyle=9\.245683770562014,Pℓ​ω\\displaystyle P\_\{\\ell\\omega\}=−9\.008824250903285,\\displaystyle=\-9\.008824250903285,Pω​ω\\displaystyle P\_\{\\omega\\omega\}=11\.073004906486352\.\\displaystyle=11\.073004906486352\.Every displayed decimal is interpreted as a rational number\. Exact symmetry\-reduced Sylvester tests give

350​𝑰≺𝑷≺21​𝑰\.\\frac\{3\}\{50\}\\bm\{I\}\\prec\\bm\{P\}\\prec 21\\bm\{I\}\.\(32\)For each run length define

𝑸¯L\\displaystyle\\overline\{\\bm\{Q\}\}\_\{L\}=16​∑i=16𝑪R​\(i,L,1/2\)𝖳​𝑷​𝑪R​\(i,L,1/2\),\\displaystyle=\\frac\{1\}\{6\}\\sum\_\{i=1\}^\{6\}\\bm\{C\}\_\{R\}\(i,L;1/2\)^\{\\mathsf\{T\}\}\\bm\{P\}\\bm\{C\}\_\{R\}\(i,L;1/2\),𝒗∗\\displaystyle\\bm\{v\}\_\{\*\}=\(0\.3772374​16𝖳,0\.38224657,0\.00626418\)𝖳,βL=𝒗∗𝖳​𝑸¯L​𝒗∗𝒗∗𝖳​𝑷​𝒗∗\.\\displaystyle=\\bigl\(0\.3772374\\,\\bm\{1\}\_\{6\}^\{\\mathsf\{T\}\},0\.38224657,0\.00626418\\bigr\)^\{\\mathsf\{T\}\},\\qquad\\beta\_\{L\}=\\frac\{\\bm\{v\}\_\{\*\}^\{\\mathsf\{T\}\}\\overline\{\\bm\{Q\}\}\_\{L\}\\bm\{v\}\_\{\*\}\}\{\\bm\{v\}\_\{\*\}^\{\\mathsf\{T\}\}\\bm\{P\}\\bm\{v\}\_\{\*\}\}\.All displayed decimals are again interpreted as exact rationals\. Put

m11=∑L=011pL=1384128720013841287201,𝑺11=∑L=011pL​𝑸¯LβL\.m\_\{11\}=\\sum\_\{L=0\}^\{11\}p\_\{L\}=\\frac\{13841287200\}\{13841287201\},\\qquad\\bm\{S\}\_\{11\}=\\sum\_\{L=0\}^\{11\}p\_\{L\}\\frac\{\\overline\{\\bm\{Q\}\}\_\{L\}\}\{\\beta\_\{L\}\}\.The read\-only verifier performs symmetry\-reduced exact Sylvester tests and certifies

\(m11\+10−10\)​𝑷−𝑺11≻0\.\\left\(m\_\{11\}\+10^\{\-10\}\\right\)\\bm\{P\}\-\\bm\{S\}\_\{11\}\\succ 0\.\(33\)
We spell out how this matrix inequality becomes a direction\-uniform log drift\. For a nonzero current direction𝒖\\bm\{u\}, first average over the next upper stateii, which is uniform and independent of the past\. Jensen’s inequality gives

16​∑i=16log⁡‖𝑪R​\(i,L,1/2\)​𝒖‖𝑷‖𝒖‖𝑷\\displaystyle\\frac\{1\}\{6\}\\sum\_\{i=1\}^\{6\}\\log\\frac\{\\\|\\bm\{C\}\_\{R\}\(i,L;1/2\)\\bm\{u\}\\\|\_\{\\bm\{P\}\}\}\{\\\|\\bm\{u\}\\\|\_\{\\bm\{P\}\}\}≤12​log⁡𝒖𝖳​𝑸¯L​𝒖𝒖𝖳​𝑷​𝒖,‖𝒖‖𝑷=\(𝒖𝖳​𝑷​𝒖\)1/2\.\\displaystyle\\qquad\\leq\\frac\{1\}\{2\}\\log\\frac\{\\bm\{u\}^\{\\mathsf\{T\}\}\\overline\{\\bm\{Q\}\}\_\{L\}\\bm\{u\}\}\{\\bm\{u\}^\{\\mathsf\{T\}\}\\bm\{P\}\\bm\{u\}\},\\qquad\\\|\\bm\{u\}\\\|\_\{\\bm\{P\}\}=\(\\bm\{u\}^\{\\mathsf\{T\}\}\\bm\{P\}\\bm\{u\}\)^\{1/2\}\.\(34\)Write the ratio on the right asβL​rL​\(𝒖\)\\beta\_\{L\}r\_\{L\}\(\\bm\{u\}\)and uselog⁡r≤r−1\\log r\\leq r\-1\. Summing over0≤L≤110\\leq L\\leq 11and applying Equation[33](https://arxiv.org/html/2609.19170#S5.E33)yields, uniformly in𝒖≠0\\bm\{u\}\\neq 0, the following bound, whereg0:11\(𝒖\)g\_\{0:11\}\(\\bm\{u\}\)denotes thepLp\_\{L\}\-weighted sum of the left\-hand side of Equation[34](https://arxiv.org/html/2609.19170#S5.E34):

g0:11\(𝒖\)≤12\{∑L=011pL\(log⁡βL¯−1\)\+m11\+10−10\},g\_\{0:11\}\(\\bm\{u\}\)\\leq\\frac\{1\}\{2\}\\left\\\{\\sum\_\{L=0\}^\{11\}p\_\{L\}\(\\overline\{\\log\\beta\_\{L\}\}\-1\)\+m\_\{11\}\+10^\{\-10\}\\right\\\},wherelog⁡βL¯\\overline\{\\log\\beta\_\{L\}\}is the outward rational upper bound obtained from the 48\-term logarithm enclosure used by the verifier\. Exact rational evaluation gives

g0:11:=sup𝒖≠0g0:11\(𝒖\)≤−0\.000211197432303078308\.g\_\{0:11\}:=\\sup\_\{\\bm\{u\}\\neq 0\}g\_\{0:11\}\(\\bm\{u\}\)\\leq\-0\.000211197432303078308\.\(35\)It remains to bound the unenumerated event explicitly\. PutN=12N=12\. The geometric law gives

Pr⁡\(L≥N\)\\displaystyle\\Pr\(L\\geq N\)=7−N,\\displaystyle=7^\{\-N\},∑L=N∞pL​L\\displaystyle\\sum\_\{L=N\}^\{\\infty\}p\_\{L\}L=7−N​\(N\+16\),\\displaystyle=7^\{\-N\}\\left\(N\+\\frac\{1\}\{6\}\\right\),∑L=N∞pL​L​\(L−1\)\\displaystyle\\sum\_\{L=N\}^\{\\infty\}p\_\{L\}L\(L\-1\)=7−N​\{N⁡\(N−1\)\+N3\+118\}\.\\displaystyle=7^\{\-N\}\\left\\\{N\(N\-1\)\+\\frac\{N\}\{3\}\+\\frac\{1\}\{18\}\\right\\\}\.\(36\)UsingFk≤\(693/593\)​\(693/100\)kF\_\{k\}\\leq\(693/593\)\(693/100\)^\{k\}, the factor norm bound‖𝑴kR‖2≤\(357/200\)​Fk\\\|\\bm\{M\}\_\{k\}^\{R\}\\\|\_\{2\}\\leq\(357/200\)F\_\{k\}, and Equation[32](https://arxiv.org/html/2609.19170#S5.E32), every direction has the tail majorant

g12:∞\(𝒖\)\\displaystyle g\_\{12:\\infty\}\(\\bm\{u\}\)≤∑L=N∞pL\{\(L\+1\)log357200\+Llog693593\\displaystyle\\leq\\sum\_\{L=N\}^\{\\infty\}p\_\{L\}\\left\\\{\(L\+1\)\\log\\frac\{357\}\{200\}\+L\\log\\frac\{693\}\{593\}\\right\.\+L⁡\(L−1\)2log693100\+12log350\}\.\\displaystyle\\hskip 96\.73936pt\\left\.\+\\frac\{L\(L\-1\)\}\{2\}\\log\\frac\{693\}\{100\}\+\\frac\{1\}\{2\}\\log 350\\right\\\}\.Substitution of the three exact tail sums in Equation[36](https://arxiv.org/html/2609.19170#S5.E36), followed by the same outward 48\-term logarithm enclosures, gives

g12:∞:=sup𝒖≠0g12:∞\(𝒖\)≤1\.04142204840942604×10−8\.g\_\{12:\\infty\}:=\\sup\_\{\\bm\{u\}\\neq 0\}g\_\{12:\\infty\}\(\\bm\{u\}\)\\leq 1\.04142204840942604\\times 10^\{\-8\}\.Withε=10−4\\varepsilon=10^\{\-4\}and𝔼⁡\[L\+1\]=7/6\\mathbb\{E\}\[L\+1\]=7/6,

g0:11\+g12:∞\+ε76≤−9\.4520351415927536×10−5<0\.g\_\{0:11\}\+g\_\{12:\\infty\}\+\\varepsilon\\frac\{7\}\{6\}\\leq\-9\.4520351415927536\\times 10^\{\-5\}<0\.\(37\)It remains to pass from a one\-cycle conditional drift to the top physical\-time exponent\. LetΔn\\Delta\_\{n\}be the log𝑷\\bm\{P\}\-norm gain of cyclennalong the direction produced by the preceding cycles, and letτn=Ln\+1\\tau\_\{n\}=L\_\{n\}\+1\. Equations[35](https://arxiv.org/html/2609.19170#S5.E35)–[37](https://arxiv.org/html/2609.19170#S5.E37)imply𝔼⁡\[Δn\+ε​τn∣ℱn−1\]≤−η\\mathbb\{E\}\[\\Delta\_\{n\}\+\\varepsilon\\tau\_\{n\}\\mid\\mathcal\{F\}\_\{n\-1\}\]\\leq\-\\etafor a constantη\>0\\eta\>0, uniformly over the incoming nonzero direction\. The determinant calculation above and the factor bounds give\|Δn\|≤a\+b​Ln2\|\\Delta\_\{n\}\|\\leq a\+bL\_\{n\}^\{2\}for finite constantsa,ba,b\. SinceLnL\_\{n\}is geometric, the associated martingale differences are square integrable, and the martingale strong law gives

lim supN→∞∑n=1NΔn∑n=1Nτn≤−εa\.s\.\\limsup\_\{N\\to\\infty\}\\frac\{\\sum\_\{n=1\}^\{N\}\\Delta\_\{n\}\}\{\\sum\_\{n=1\}^\{N\}\\tau\_\{n\}\}\\leq\-\\varepsilon\\qquad\\text\{a\.s\.\}for every fixed initial direction\.

Choose a fixed1/21/2\-net of the𝑷\\bm\{P\}\-unit sphere\. It is finite, so the preceding probability\-one events may be intersected, and the standard net bound makes the operator𝑷\\bm\{P\}\-norm at most twice the largest norm on the net\. Norm equivalence from Equation[32](https://arxiv.org/html/2609.19170#S5.E32)leaves the exponent unchanged\. Finally, a partial cycle has positive log norm at mosta′\+b′​Ln2a^\{\\prime\}\+b^\{\\prime\}L\_\{n\}^\{2\}; the geometric tail and Borel–Cantelli giveLn2/n→0L\_\{n\}^\{2\}/n\\to 0, so the unfinished cycle contributeso⁡\(t\)o\(t\)\. The endpoint bound therefore holds at every transition and provesλRpred≤−10−4\\lambda\_\{R\}^\{\\mathrm\{pred\}\}\\leq\-10^\{\-4\}\. ∎

### 5\.5The Sign Comparison

Table 2:Baird comparison atα=0\.01\\alpha=0\.01\. Mean radii are deterministic\. Exponent estimates are means plus/minus sample standard deviations over ten independent runs\. The RETD negative sign has the uniform cycle certificate in Proposition[6](https://arxiv.org/html/2609.19170#Thmtheorem6); the ETD positive sign remains numerical\.Both mean maps contract, and RETD’s mean map is the slower one\. The sampled sign reversal is caused by the random\-product dynamics; the mean dynamics do not explain it\. The difference comes from the final zero\-ratio factor: ETD applies the identity after a trace shock, whereas RETD applies𝑫s​\(c\)\\bm\{D\}\_\{s\}\(c\)and releases a delayed correction\. Section[2](https://arxiv.org/html/2609.19170#S2)supplies the exact positive\-ETD existence proof; this Baird analysis shows that the same mechanism persists in the canonical multidimensional counterexample, with the stated distinction between numerical ETD evidence and certified RETD contraction\.

## 6Experimental Study

The experiments evaluate four questions:*Q1*, whether the new two\-state and Baird counterexamples show the predicted finite\-horizon ETD expansion and RETD contraction while other off\-policy algorithms can learn on the same trajectories;*Q2*, how the RETD pathwise and mean\-stability regions change with leakagecc, behavior mismatch, and discount;*Q3*, whether the raw RETD equilibrium shift can be removed from finite\-sample outputs to recover the ETD fixed prediction; and*Q4*, how RETD behaves away from the counterexamples on Boyan’s chain and three random\-walk feature representations\.

For every learning curve in Q1, Q3, and Q4, the reported mean and sample standard deviation are computed from10,00010\{,\}000independent runs\. Within an environment and seed, all algorithms consume the same state–action trajectory\. Hyperparameters for the contextual baselines are selected on ten independent validation seeds and then frozen before the10,00010\{,\}000disjoint test seeds are evaluated\. Saved seed\-level outcomes, selected configurations, aggregate curves, and checksums accompany the implementation\.

Q2 estimates Lyapunov signs from long normalized random\-matrix products over parameter grids\. Its seed counts refer to independent long\-product estimators and are reported separately from the10,00010\{,\}000learning\-curve replications\.

Figure 3:Training curves on the two counterexamples over10,00010\{,\}000independent paired runs\. Curves are mean cumulative log\-amplificationGtG\_\{t\}; bands are one sample standard deviation\. The green half\-planeGt<0G\_\{t\}<0means contraction and the red half\-planeGt\>0G\_\{t\}\>0means expansion\. Both axes are linear\. ETD and RETD use the analyzed settings; the remaining methods use configurations selected on independent validation trajectories\.### 6\.1Evaluation Metrics

For the zero\-reward counterexamples, letEtE\_\{t\}denote RMS value error over all states\. We reportλ^t=t−1​log⁡\(Et/E0\)\\widehat\{\\lambda\}\_\{t\}=t^\{\-1\}\\log\(E\_\{t\}/E\_\{0\}\)andGt=t​λ^t=log⁡\(Et/E0\)G\_\{t\}=t\\widehat\{\\lambda\}\_\{t\}=\\log\(E\_\{t\}/E\_\{0\}\)\. ThusGt<0G\_\{t\}<0means contraction andGt\>0G\_\{t\}\>0means expansion; more negative is better\. PlottingGtG\_\{t\}preserves the sign and makes different contraction rates visible on linear axes\. For the nonzero\-reward tasks in Q3 and Q4, the metric is RMS value error, for which lower is better\. Every shaded band is one sample standard deviation across independent runs, and no temporal smoothing or logarithmic axis is used\.

### 6\.2Q1: Finite\-Horizon Counterexample Validation

*Settings and Sign Predictions\.*

The new two\-state experiment uses the construction in Sections[2\.4](https://arxiv.org/html/2609.19170#S2.SS4)and[3\.4](https://arxiv.org/html/2609.19170#S3.SS4):p=\.3p=\.3,γ=\.9\\gamma=\.9,ϕ⁡\(s1\)=1\\phi\(s\_\{1\}\)=1,ϕ⁡\(s2\)=2\\phi\(s\_\{2\}\)=2, zero reward, andθ0=1\\theta\_\{0\}=1\. The analyzed ETD point isα=\.1\\alpha=\.1; RETD uses the same stepsize,c=4c=4, andω0=0\\omega\_\{0\}=0\. Att=5,000t=5\{,\}000, the10,00010\{,\}000runs giveλ^5000ETD=\.011878±\.004505\\widehat\{\\lambda\}\_\{5000\}^\{\\rm ETD\}=\.011878\\pm\.004505andλ^5000RETD=−\.010495±\.004903\\widehat\{\\lambda\}\_\{5000\}^\{\\rm RETD\}=\-\.010495\\pm\.004903\. The exponent is positive in99\.90%99\.90\\%of ETD runs but only2\.23%2\.23\\%of RETD runs\. This is the finite\-horizon counterpart of the exact sign calculation, not an outlier\-driven difference\.

For Baird’s seven\-state construction in Section[5](https://arxiv.org/html/2609.19170#S5), ETD usesα=\.01\\alpha=\.01and RETD uses\(α,c\)=\(\.01,\.5\)\(\\alpha,c\)=\(\.01,\.5\)\. The corresponding estimates areλ^5000ETD=\.001682±\.000488\\widehat\{\\lambda\}\_\{5000\}^\{\\rm ETD\}=\.001682\\pm\.000488andλ^5000RETD=−\.001118±\.000435\\widehat\{\\lambda\}\_\{5000\}^\{\\rm RETD\}=\-\.001118\\pm\.000435, with positive signs in99\.99%99\.99\\%and1\.84%1\.84\\%of runs, respectively\. The negative RETD limiting sign is covered by the certificate in Section[5](https://arxiv.org/html/2609.19170#S5); the positive ETD sign remains a finite\-horizon numerical result\.

*Training\-Curve Comparison with Off\-Policy Baselines\.*

Figure[3](https://arxiv.org/html/2609.19170#S6.F3)adds GTD\(0\), GTD2, TDC, TDRC, and TETD\(4\) to the same paired test trajectories\. ETD and RETD retain the settings fixed by the counterexample analyses\. The other methods use configurations selected on independent validation trajectories; their update scales differ, so forcing a common numerical stepsize would not be a fair comparison\.

Att=5,000t=5\{,\}000on the new two\-state counterexample, meanGtG\_\{t\}is59\.3959\.39for ETD,−52\.47\-52\.47for RETD, and−41\.43\-41\.43for the next\-fastest TETD\(4\)\. On Baird’s counterexample, the corresponding values are8\.418\.41for ETD,−5\.59\-5\.59for RETD,−2\.77\-2\.77for TDRC, and2\.022\.02for TETD\(4\)\. RETD is therefore the fastest contracting method at the analyzed points in both environments\. The contraction of the other methods shows that both tasks are learnable\. ETD’s positive curves arise from its sampled update product\.

### 6\.3Q2: Sensitivity tocc, Behavior Mismatch, and Discount

Figure 4:Baird regularization path on linear axes\. Panel \(a\) reports the sample\-path exponent and panel \(b\) the deterministic mean\-map margin; negative values denote stability\.*WhyccCannot Be Chosen from Mean Stability Alone\.*

The first scan holds Baird’s environment andα=\.01\\alpha=\.01fixed\. The full numerical scan evaluates 23 values fromc=0c=0to150150using ten independent normalized matrix\-product estimates, each with 10,000 burn\-in and 200,000 retained matrices\. Figure[4](https://arxiv.org/html/2609.19170#S6.F4)displays the 16 values in\[0,3\]\[0,3\], where the contraction–expansion transition occurs; every evaluated point withc≥4c\\geq 4remains on the positive side of the pathwise boundary and is retained in the released result file\. The band is one sample standard deviation over the ten estimates, and the star marksc=\.5c=\.5, whose negative sign also has the rigorous cycle certificate\.

The upper panel falls from the neutralc=0c=0point to−\.001439±\.000061\-\.001439\\pm\.000061atc=\.5c=\.5, then crosses zero betweenc=1\.5c=1\.5andc=1\.75c=1\.75\. Hence neither zero leakage nor arbitrarily large leakage is appropriate\. The lower panel stays below zero over the displayed range\. The two diagnostics have different signs:ρ⁡\(I−α​Gc\)<1⇏λtop<0\\rho\(I\-\\alpha G\_\{c\}\)<1\\not\\Rightarrow\\lambda\_\{\\rm top\}<0\. Thusccchanges the sampled feedback dynamics even when the deterministic mean matrix remains positive stable\.

![Refer to caption](https://arxiv.org/html/2609.19170v1/baird_phase_map_explained_20260826.png)Figure 5:Finite\-time path\-exponent map over one common Baird\-family grid\. Blue cells have negative exponents and contract; red cells have positive exponents and expand\. Black curves mark the zero boundary, stars mark the canonical Baird point, and open squares are independent boundary checks\. The in\-panel count reports expanding cells out of 225\. All mean systems on this grid are positive stable, so the color changes show changes in the sample\-path exponent\.*Stability over a Family of Baird Environments\.*

The second scan varies the behavior solid\-action probabilityppand discount factorγ\\gammawhile holding the feature geometry, target policy, reward, reset mechanism, andα=\.01\\alpha=\.01fixed\. The common15×1515\\times 15grid coversp∈\[\.05,\.25\]p\\in\[\.05,\.25\]andγ∈\[\.90,\.99\]\\gamma\\in\[\.90,\.99\]\. Each cell is screened by ten independent long\-product estimates; open squares identify boundary cells rerun with 30 new long\-product estimates\. These counts characterize a random\-matrix diagnostic, not repeated learning curves\.

ETD expands in 127 of 225 cells\. RETD expands in 4, 7, and 45 cells atc=\.1,\.5,c=\.1,\.5,and22, respectively\. The increase from 7 expanding cells atc=\.5c=\.5to 45 atc=2c=2shows that the stable region is not monotone incc\. Only the canonical RETD\(\.5\)\(\.5\)point is covered by the rigorous certificate; the regional map is numerical and provides no uniform guarantee over the grid\.

### 6\.4Q3: Recovery of the ETD Fixed Prediction

Zero reward isolates homogeneous stability but cannot test the affine solution identities\.

Figure 6:Absolute prediction error after tail averaging over10,00010\{,\}000paired runs\. Points are means and bars are one sample standard deviation; lower is better\. Raw RETD retains the predictedcc\-dependent shift, whereas the single\-ccand two\-ccreadouts return to the ETD error scale\. The ordinate is linear\.We use a separate bounded\-trace, nonzero\-reward two\-state task solely to evaluate fixed\-point recovery\. Its transition matrices are

Pleft=\[1010\],Pright=\[0101\]\.P\_\{\\rm left\}=\\begin\{bmatrix\}1&0\\\\ 1&0\\end\{bmatrix\},\\qquad P\_\{\\rm right\}=\\begin\{bmatrix\}0&1\\\\ 0&1\\end\{bmatrix\}\.The behavior policy chooses right with probability3/43/4, the target always chooses right,γ=1/2\\gamma=1/2, andϕ=\(1,3/5\)⊤\\phi=\(1,3/5\)^\{\\top\}\. Reward is one precisely when the current state iss2s\_\{2\}and action right is selected, and zero otherwise\. Hencedμ=\(1/4,3/4\)⊤d\_\{\\mu\}=\(1/4,3/4\)^\{\\top\},ρright=4/3\\rho\_\{\\rm right\}=4/3,ρleft=0\\rho\_\{\\rm left\}=0, andγ​ρt≤2/3\\gamma\\rho\_\{t\}\\leq 2/3, so the follow\-on trace is bounded\.

Direct evaluation gives

A\\displaystyle A=49100,q=710,b=2120,β=74,\\displaystyle=\\frac\{49\}\{100\},\\quad q=\\frac\{7\}\{10\},\\quad b=\\frac\{21\}\{20\},\\quad\\beta=\\frac\{7\}\{4\},θE\\displaystyle\\theta\_\{E\}=157,u=A−1q=107,q⊤u=1,mE=14\.\\displaystyle=\\frac\{15\}\{7\},\\quad u=A^\{\-1\}q=\\frac\{10\}\{7\},\\quad q^\{\\top\}u=1,\\quad m\_\{E\}=\\frac\{1\}\{4\}\.The raw RETD equilibria are\(θ,ω\)=\(10/7,1/2\)\(\\theta,\\omega\)=\(10/7,1/2\)atc=\.5c=\.5and\(55/28,1/8\)\(55/28,1/8\)atc=2c=2\. Thus the raw shift is nonzero and the ETD target is known analytically\.

We run10,00010\{,\}000paired trajectories for 20,000 transitions from zero, usingαt=5/\(t\+100\)\\alpha\_\{t\}=5/\(t\+100\)and averaging the last 4,000 iterates\. The single\-ccoutput appliesθE=θ\+u​ω\\theta\_\{E\}=\\theta\+u\\omega; the two\-ccoutput combines two RETD learners using the identity in Section[3\.3](https://arxiv.org/html/2609.19170#S3.SS3)and does not useuu\.

The mean errors are\.61241±\.01827\.61241\\pm\.01827and\.14789±\.01608\.14789\\pm\.01608for raw RETD atc=\.5c=\.5andc=2c=2\. Single\-ccrecovery reduces them to\.02028±\.01526\.02028\\pm\.01526and\.01435±\.01082\.01435\\pm\.01082; two\-ccrecovery gives\.01305±\.00984\.01305\\pm\.00984, compared with\.01330±\.01013\.01330\\pm\.01013for ETD\. No two\-ccdenominator is invalid: its absolute value ranges from\.3539\.3539to\.4293\.4293\. The recovered errors agree with the ETD fixed\-point identity\. Two\-ccrecovery maintains two learners and becomes ill conditioned when their empirical auxiliary states are too close\.

### 6\.5Q4: Additional Prediction Tasks

Q4 uses Boyan’s chain and an off\-policy random walk with tabular, inverted, and linearly dependent features\. These tasks evaluate the dependence of finite\-sample performance on the task and feature representation outside the two instability examples\.

Figure 7:Learning curves on Boyan’s chain and three representations of the off\-policy random walk\. Curves are mean RMS value error over10,00010\{,\}000independent held\-out runs; bands are one sample standard deviation\. Lower is better, both axes are linear, and no temporal smoothing is applied\.The 13\-state Boyan chain\([Boyan, 1999](https://arxiv.org/html/2609.19170#bib.bib2)\)has states0,…,120,\\ldots,12\. From anys≥2s\\geq 2, the behavior process moves tos−2s\-2with probability\.4\.4and reward11, or tos−1s\-1with probability\.6\.6and reward22; the target probabilities are\.5\.5and\.5\.5, so the two importance ratios are1\.251\.25and5/65/6\. State11moves deterministically to state00with reward1\.51\.5\. From state00, a new state is sampled uniformly from all 13 states with reward1\.51\.5and ratio one\. We useγ=\.9\\gamma=\.9andθ0=𝟏4\\theta\_\{0\}=\\bm\{1\}\_\{4\}\. Leteje\_\{j\}be thejjth coordinate vector inℝ4\\mathbb\{R\}^\{4\}\. Fors=4​j\+rs=4j\+r,j∈\{0,1,2\}j\\in\\\{0,1,2\\\}andr∈\{0,1,2,3\}r\\in\\\{0,1,2,3\\\},ϕ⁡\(s\)=\(1−r/4\)​ej\+1\+\(r/4\)​ej\+2\\phi\(s\)=\(1\-r/4\)e\_\{j\+1\}\+\(r/4\)e\_\{j\+2\}andϕ⁡\(12\)=e4\\phi\(12\)=e\_\{4\}\.

The random walk has five nonterminal states0,…,40,\\ldots,4, starts each episode at state22, and moves left or right with behavior probabilities\(\.5,\.5\)\(\.5,\.5\)and target probabilities\(\.4,\.6\)\(\.4,\.6\)\. Thus the importance ratios are\.8\.8and1\.21\.2\. Crossing the left boundary gives reward zero, crossing the right boundary gives reward one, and either event restarts the episode at state22with the follow\-on trace reset to one\. Hereγ=\.9\\gamma=\.9andθ0=0\\theta\_\{0\}=0\. The three feature matrices, with states indexing rows, areΦtab=I5\\Phi\_\{\\rm tab\}=I\_\{5\},Φinv=12​\(𝟏𝟏⊤−I5\)\\Phi\_\{\\rm inv\}=\\tfrac\{1\}\{2\}\(\\bm\{1\}\\bm\{1\}^\{\\top\}\-I\_\{5\}\), andΦdep=\[1001/21/201/31/31/301/21/2001\]\\Phi\_\{\\rm dep\}=\\left\[\\begin\{smallmatrix\}1&0&0\\\\ 1/\\sqrt\{2\}&1/\\sqrt\{2\}&0\\\\ 1/\\sqrt\{3\}&1/\\sqrt\{3\}&1/\\sqrt\{3\}\\\\ 0&1/\\sqrt\{2\}&1/\\sqrt\{2\}\\\\ 0&0&1\\end\{smallmatrix\}\\right\], respectively\. Boyan’s chain is run for 5,000 transitions and each random\-walk variant for 10,000 transitions\.

We compare GTD2, TDC, TDRC, ETD, TETD\(4\), and RETD\. For each task, every method receives 24 candidate configurations evaluated on the same ten validation seeds\. The selected configuration is frozen and then evaluated on10,00010\{,\}000new integer seeds\. All methods share the same counter\-based random stream within a task and seed\. RETD is evaluated through its analytic single\-ccrecovered prediction, so any remaining difference is not the rawcc\-dependent fixed\-point shift\.

The terminal RMS errors depend on the feature representation\. On Boyan’s chain, TETD\(4\) is best at\.13884±\.04487\.13884\\pm\.04487, followed by ETD at\.15301±\.05294\.15301\\pm\.05294and RETD at\.19145±\.03998\.19145\\pm\.03998\. On the tabular random walk, TDRC is best at\.01690±\.00568\.01690\\pm\.00568, whereas ETD and RETD attain\.01797±\.00504\.01797\\pm\.00504and\.03239±\.01199\.03239\\pm\.01199\. With inverted features, GTD2 is best at\.01859±\.00633\.01859\\pm\.00633; ETD and RETD attain\.03069±\.00695\.03069\\pm\.00695and\.03515±\.00578\.03515\\pm\.00578\. With linearly dependent features, in contrast, RETD is best at\.11633±\.01162\.11633\\pm\.01162, compared with\.12337±\.01100\.12337\\pm\.01100for ETD and\.12868±\.01117\.12868\\pm\.01117for TETD\(4\)\. No method diverges on any of these four tasks\.

These tasks evaluate RETD outside the two instability examples\. After hyperparameter selection, its prediction error is lower on the linearly dependent random walk but higher on Boyan’s chain and the other two random\-walk representations\. The finite\-sample advantage therefore depends on the task and feature representation; the results do not establish uniform dominance over the baselines\.

## 7Related Work

We group related methods by the object they modify: the projected equation, the trace or ratios, the update dynamics, or the stability criterion\. Table[3](https://arxiv.org/html/2609.19170#S7.T3)summarizes the distinctions most relevant to RETD\.

Table 3:Related off\-policy methods grouped by the component they modify\.*The First Two Off\-Policy Problems\.*

Baird’s counterexample established that ordinary linear off\-policy TD can diverge even with bounded features and zero reward\([Baird, 1995](https://arxiv.org/html/2609.19170#bib.bib1)\)\. Gradient\-TD and TDRC stabilize projected or gradient\-correction systems\([Sutton et al\., 2008](https://arxiv.org/html/2609.19170#bib.bib28);[Sutton et al\., 2009](https://arxiv.org/html/2609.19170#bib.bib29);[Ghiassian et al\., 2020](https://arxiv.org/html/2609.19170#bib.bib9)\); RETD instead uses a scalarωt\\omega\_\{t\}to store and leak an emphatic TD shock\. It is therefore not a scalar GTD2/TDC variant: its recovered equilibrium is the ETD solution, whereas those methods target an ordinary MSPBE\-related system\. Oblique\-projection work further shows why this recovery matters: a stable projected recursion can have a poor target\-policy approximation\([Scherrer, 2010](https://arxiv.org/html/2609.19170#bib.bib26);[Kolter, 2011](https://arxiv.org/html/2609.19170#bib.bib18)\), while emphatic weighting changes that projection geometry\([Sutton et al\., 2016](https://arxiv.org/html/2609.19170#bib.bib30);[Hallak et al\., 2016](https://arxiv.org/html/2609.19170#bib.bib14)\)\.

*Modifications of Emphatic Traces\.*

The original ETD study identified high trace variance\([Sutton et al\., 2016](https://arxiv.org/html/2609.19170#bib.bib30)\); later work established diminishing\-stepsize convergence and constrained constant\-step weak convergence\([Yu, 2015](https://arxiv.org/html/2609.19170#bib.bib24);[Yu, 2016](https://arxiv.org/html/2609.19170#bib.bib25)\)\. These results concern different stability objects from an unconstrained homogeneous constant\-stepsize product\.

PER\-ETD, TETD, learned or deep emphatic methods, LC\-ETD, emphatic actor–critic methods, and V\-trace respectively estimate, truncate, relax, or replace parts of the emphasis or ratio pathway\([Guan et al\., 2022](https://arxiv.org/html/2609.19170#bib.bib11);[Zhang and Whiteson, 2022](https://arxiv.org/html/2609.19170#bib.bib31);[Jiang et al\., 2021](https://arxiv.org/html/2609.19170#bib.bib16);[Jiang et al\., 2022](https://arxiv.org/html/2609.19170#bib.bib17);[He et al\., 2023](https://arxiv.org/html/2609.19170#bib.bib15);[Graves et al\., 2023](https://arxiv.org/html/2609.19170#bib.bib10);[Espeholt et al\., 2018](https://arxiv.org/html/2609.19170#bib.bib7)\)\.

RETD occupies a different point: it leavesFt\+1=1\+γ​ρt​FtF\_\{t\+1\}=1\+\\gamma\\rho\_\{t\}F\_\{t\}andρt\\rho\_\{t\}unchanged, hence does not change the trace tail index\. The same shock instead entersθt\\theta\_\{t\}andωt\\omega\_\{t\}; later zero\-ratio transitions use the leaked scalar for recovery\. This is the algebraic distinction in Equation \([29](https://arxiv.org/html/2609.19170#S4.E29)\) and the dynamical distinction in the exact two\-state cycle products\.

*Random Products and Lyapunov Stability\.*

Top Lyapunov exponents and regenerative drift arguments are standard tools for random products and Markov chains\([Furstenberg and Kesten, 1960](https://arxiv.org/html/2609.19170#bib.bib19);[Kingman, 1968](https://arxiv.org/html/2609.19170#bib.bib20);[Oseledets, 1968](https://arxiv.org/html/2609.19170#bib.bib21);[Bougerol and Lacroix, 1985](https://arxiv.org/html/2609.19170#bib.bib3);[Meyn and Tweedie, 2009](https://arxiv.org/html/2609.19170#bib.bib4)\)\.

The external small\-stepsize random\-product result used in Theorem[5](https://arxiv.org/html/2609.19170#Thmtheorem5)is due to[Durmus et al\. \(2021\)](https://arxiv.org/html/2609.19170#bib.bib22)\. RETD requires an algorithm\-specific verification: its sample matrix contains the rank\-onecc\-term in Equation \([28](https://arxiv.org/html/2609.19170#S4.E28)\), the explicit condition onccmakes the mean matrix positive stable, and the resulting Lyapunov matrix and growth constant determine an admissible stepsize\. Recent TD analyses also use exponential stability\([Samsonov et al\., 2024](https://arxiv.org/html/2609.19170#bib.bib32);[Lee and Orabona, 2026](https://arxiv.org/html/2609.19170#bib.bib33)\), but they do not verify the augmented trace\-driven RETD matrices or compare their post\-shock dynamics with ETD\. Satisfying the RETD condition therefore proves no ETD result\.

The Baird certificate in Section[5](https://arxiv.org/html/2609.19170#S5)combines these tools with an environment\-specific projective drift computation and separates fitted quantities from rigorous interval verification\([Rump, 2010](https://arxiv.org/html/2609.19170#bib.bib5)\)\. The RETD point has a projective certificate; the positive Baird ETD exponent is only a finite\-horizon numerical estimate\. RETD also differs from optimization analyses of multiplicative noise and heavy tails\([Simsekli et al\., 2019](https://arxiv.org/html/2609.19170#bib.bib27);[Gurbuzbalaban et al\., 2021](https://arxiv.org/html/2609.19170#bib.bib12);[Hodgkinson and Mahoney, 2021](https://arxiv.org/html/2609.19170#bib.bib13);[Chemnitz and Engel, 2025](https://arxiv.org/html/2609.19170#bib.bib6)\): its matrices are trace\-driven and prediction\-relevant exponents require removal of feature\-null directions\.

*Relation to Bellman\-Error Centering\.*

Bellman\-error centering stores a shared TD\-error component\([Chen et al\., 2026](https://arxiv.org/html/2609.19170#bib.bib8)\), but RETD drives its scalar state byFt​ρt​δtF\_\{t\}\\rho\_\{t\}\\delta\_\{t\}, couples it to an emphatically weighted update, and targets a recoverable ETD fixed point\. Its leaky impulse response,c=0c=0conserved\-mode obstruction, two\-state sign separation, and Baird certificate are therefore distinct from centered TD’s objectives and stability mechanism\.

## 8Conclusion and Future Work

This paper separates mean stability, diminishing\-stepsize convergence, and constant\-stepsize sample\-path stability in off\-policy emphatic prediction\. A two\-state counterexample proves that ETD can have a strictly contractive mean map and a positive top Lyapunov exponent at the same stepsize\. RETD adds one leaky scalar state, reverses the product sign on that counterexample, and preserves the ETD solution through an exact affine readout\. The theory establishes almost\-sure convergence with a diminishing stepsize and conditional moment contraction with a sufficiently small constant stepsize; the Baird study and held\-out experiments show both the resulting stability gains and their task dependence\.

Future work will extend the analysis to RETD\(λ\\lambda\)\. A multi\-step extension must treat the eligibility trace, follow\-on trace, parameter vector, and recovery state as one coupled stochastic system; derive the corresponding mean matrix and fixed\-point recovery identity; and establish diminishing\- and constant\- stepsize results without assuming the random\-product contraction that the algorithm is intended to prove\.

###### acknowledgments\-disclosure\-of\-funding\.

This study was supported in part by the National Natural Science Foundation of China \(Nos\. 62276142, 62206133, 62202240, and 62506172\), the Fundamental Research Funds for the Central Universities \(No\. 2242025K30024\), the Basic Research Program of Jiangsu \(No\. BK20250658\), and the Natural Science Research Start\-up Foundation of Recruiting Talents of Nanjing University of Posts and Telecommunications \(No\. NY225026\)\.*Competing Interests\.*The authors declare that they have no competing interests relevant to this work\.

## References

- Baird \(1995\)L\. BairdResidual algorithms: reinforcement learning with function approximation\.InProceedings of the 12th International Conference on Machine Learning,pp\. 30–37\.Cited by:[§1](https://arxiv.org/html/2609.19170#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.19170#S2.SS1.p2.2),[§7](https://arxiv.org/html/2609.19170#S7.p3.1)\.
- Bougerol and Lacroix \(1985\)P\. Bougerol and J\. LacroixProducts of random matrices with applications to schr”odinger operators\.Progress in Probability and Statistics, Vol\.8,Birkh”auser,Boston\.Cited by:[§7](https://arxiv.org/html/2609.19170#S7.p9.1)\.
- Boyan \(1999\)J\. A\. BoyanLeast\-squares temporal difference learning\.InProceedings of the 16th International Conference on Machine Learning,pp\. 49–56\.Cited by:[§6\.5](https://arxiv.org/html/2609.19170#S6.SS5.p2.1)\.
- Chemnitz and Engel \(2025\)T\. Chemnitz and M\. EngelCharacterizing dynamical stability of stochastic gradient descent in overparameterized learning\.Journal of Machine Learning Research26\(134\),pp\. 1–46\.Cited by:[§7](https://arxiv.org/html/2609.19170#S7.p11.1)\.
- Chenet al\.\(2026\)X\. Chen, Y\. Gong, J\. Ye, C\. Li, S\. Yang, and W\. WangBellman error centering\.Neural Networks201,pp\. 108896\.Cited by:[§7](https://arxiv.org/html/2609.19170#S7.p13.1)\.
- Durmuset al\.\(2021\)A\. Durmus, E\. Moulines, A\. Naumov, S\. Samsonov, and H\. WaiOn the stability of random matrix product with Markovian noise: application to linear stochastic approximation and TD learning\.InProceedings of the 34th Conference on Learning Theory,Proceedings of Machine Learning Research, Vol\.134,pp\. 1711–1752\.Cited by:[§1](https://arxiv.org/html/2609.19170#S1.p5.1),[§4\.4](https://arxiv.org/html/2609.19170#S4.SS4.p3.1),[§4\.4](https://arxiv.org/html/2609.19170#S4.SS4.p4.1),[§4\.5](https://arxiv.org/html/2609.19170#S4.SS5.p1.1),[§7](https://arxiv.org/html/2609.19170#S7.p10.1),[Theorem 5](https://arxiv.org/html/2609.19170#Thmtheorem5.p1.2.1),[Theorem 5](https://arxiv.org/html/2609.19170#Thmtheorem5.p1.4.1)\.
- Espeholtet al\.\(2018\)L\. Espeholt, H\. Soyer, R\. Munos, K\. Simonyan, V\. Mnih, T\. Ward, Y\. Doron, V\. Firoiu, T\. Harley, I\. Dunning, S\. Legg, and K\. KavukcuogluIMPALA: scalable distributed deep\-RL with importance weighted actor\-learner architectures\.InProceedings of the 35th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.80,pp\. 1407–1416\.Cited by:[§7](https://arxiv.org/html/2609.19170#S7.p6.1)\.
- Furstenberg and Kesten \(1960\)H\. Furstenberg and H\. KestenProducts of random matrices\.The Annals of Mathematical Statistics31\(2\),pp\. 457–469\.Cited by:[§7](https://arxiv.org/html/2609.19170#S7.p9.1)\.
- Ghiassianet al\.\(2020\)S\. Ghiassian, A\. Patterson, S\. Garg, D\. Gupta, A\. White, and M\. WhiteGradient temporal\-difference learning with regularized corrections\.InProceedings of the 37th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.119,pp\. 3524–3534\.Cited by:[§7](https://arxiv.org/html/2609.19170#S7.p3.1)\.
- Graveset al\.\(2023\)E\. Graves, E\. Imani, R\. Kumaraswamy, and M\. WhiteOff\-policy actor\-critic with emphatic weightings\.Journal of Machine Learning Research24\(146\),pp\. 1–63\.Cited by:[§7](https://arxiv.org/html/2609.19170#S7.p6.1)\.
- Guanet al\.\(2022\)Z\. Guan, T\. Xu, and Y\. LiangPER\-ETD: a polynomially efficient emphatic temporal difference learning method\.InInternational Conference on Learning Representations,Cited by:[§7](https://arxiv.org/html/2609.19170#S7.p6.1)\.
- Gurbuzbalabanet al\.\(2021\)M\. Gurbuzbalaban, U\. Simsekli, and L\. ZhuThe heavy\-tail phenomenon in SGD\.InProceedings of the 38th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.139,pp\. 3964–3975\.Cited by:[§7](https://arxiv.org/html/2609.19170#S7.p11.1)\.
- Hallaket al\.\(2016\)A\. Hallak, A\. Tamar, R\. Munos, and S\. MannorGeneralized emphatic temporal difference learning: bias\-variance analysis\.InProceedings of the 30th AAAI Conference on Artificial Intelligence,pp\. 1631–1637\.Cited by:[§1](https://arxiv.org/html/2609.19170#S1.p2.1),[§2\.2](https://arxiv.org/html/2609.19170#S2.SS2.p3.2),[§7](https://arxiv.org/html/2609.19170#S7.p3.1)\.
- Heet al\.\(2023\)J\. He, F\. Che, Y\. Wan, and A\. R\. MahmoodLoosely consistent emphatic temporal\-difference learning\.InProceedings of the 39th Conference on Uncertainty in Artificial Intelligence,Proceedings of Machine Learning Research, Vol\.216,pp\. 849–859\.Cited by:[§7](https://arxiv.org/html/2609.19170#S7.p6.1)\.
- Hodgkinson and Mahoney \(2021\)L\. Hodgkinson and M\. W\. MahoneyMultiplicative noise and heavy tails in stochastic optimization\.InProceedings of the 38th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.139,pp\. 4262–4274\.Cited by:[§7](https://arxiv.org/html/2609.19170#S7.p11.1)\.
- Jianget al\.\(2021\)R\. Jiang, T\. Zahavy, Z\. Xu, A\. White, M\. Hessel, C\. Blundell, and H\. van HasseltEmphatic algorithms for deep reinforcement learning\.InProceedings of the 38th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.139,pp\. 5023–5033\.Cited by:[§7](https://arxiv.org/html/2609.19170#S7.p6.1)\.
- Jianget al\.\(2022\)R\. Jiang, S\. Zhang, V\. Chelu, A\. White, and H\. van HasseltLearning expected emphatic traces for deep reinforcement learning\.InProceedings of the 36th AAAI Conference on Artificial Intelligence,pp\. 7015–7023\.Cited by:[§7](https://arxiv.org/html/2609.19170#S7.p6.1)\.
- Kingman \(1968\)J\. F\. C\. KingmanThe ergodic theory of subadditive stochastic processes\.Journal of the Royal Statistical Society: Series B \(Methodological\)30\(3\),pp\. 499–510\.Cited by:[§7](https://arxiv.org/html/2609.19170#S7.p9.1)\.
- Kolter \(2011\)J\. Z\. KolterThe fixed points of off\-policy TD\.InAdvances in Neural Information Processing Systems 24,pp\. 2169–2177\.Cited by:[§1](https://arxiv.org/html/2609.19170#S1.p1.1),[§2\.2](https://arxiv.org/html/2609.19170#S2.SS2.p1.1),[§7](https://arxiv.org/html/2609.19170#S7.p3.1)\.
- Lee and Orabona \(2026\)W\. Lee and F\. OrabonaA single stepsize suffices for unprojected linear TD\(0\): simultaneous robust and fast rates via Polyak–Ruppert averaging\.InProceedings of the 39th Conference on Learning Theory,Proceedings of Machine Learning Research, Vol\.336,pp\. 4588–4634\.Cited by:[§7](https://arxiv.org/html/2609.19170#S7.p10.1)\.
- Liuet al\.\(2025\)S\. D\. Liu, S\. Chen, and S\. ZhangThe ODE method for stochastic approximation and reinforcement learning with Markovian noise\.Journal of Machine Learning Research26\(24\),pp\. 1–76\.Cited by:[§4\.2](https://arxiv.org/html/2609.19170#S4.SS2.p1.1),[§4\.2](https://arxiv.org/html/2609.19170#S4.SS2.p10.1.2),[§4\.2](https://arxiv.org/html/2609.19170#S4.SS2.p4.1.2)\.
- Meyn and Tweedie \(2009\)S\. P\. Meyn and R\. L\. TweedieMarkov chains and stochastic stability\.2 edition,Cambridge University Press\.Cited by:[§7](https://arxiv.org/html/2609.19170#S7.p9.1)\.
- Oseledets \(1968\)V\. I\. OseledetsA multiplicative ergodic theorem: characteristic Lyapunov exponents of dynamical systems\.Transactions of the Moscow Mathematical Society19,pp\. 197–231\.Cited by:[§7](https://arxiv.org/html/2609.19170#S7.p9.1)\.
- Rump \(2010\)S\. M\. RumpVerification methods: rigorous results using floating\-point arithmetic\.Acta Numerica19,pp\. 287–449\.Cited by:[§7](https://arxiv.org/html/2609.19170#S7.p11.1)\.
- Samsonovet al\.\(2024\)S\. Samsonov, D\. Tiapkin, A\. Naumov, and E\. MoulinesImproved high\-probability bounds for the temporal difference learning algorithm via exponential stability\.InProceedings of the 37th Conference on Learning Theory,Proceedings of Machine Learning Research, Vol\.247,pp\. 4511–4547\.Cited by:[§7](https://arxiv.org/html/2609.19170#S7.p10.1)\.
- Scherrer \(2010\)B\. ScherrerShould one compute the temporal difference fixed point or minimize the Bellman residual? the unified oblique projection view\.InProceedings of the 27th International Conference on Machine Learning,pp\. 959–966\.Cited by:[§1](https://arxiv.org/html/2609.19170#S1.p1.1),[§2\.2](https://arxiv.org/html/2609.19170#S2.SS2.p1.1),[§7](https://arxiv.org/html/2609.19170#S7.p3.1)\.
- Simsekliet al\.\(2019\)U\. Simsekli, L\. Sagun, and M\. GurbuzbalabanA tail\-index analysis of stochastic gradient noise in deep neural networks\.InProceedings of the 36th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.97,pp\. 5827–5837\.Cited by:[§7](https://arxiv.org/html/2609.19170#S7.p11.1)\.
- Suttonet al\.\(2009\)R\. S\. Sutton, H\. R\. Maei, D\. Precup, S\. Bhatnagar, D\. Silver, C\. Szepesvári, and E\. WiewioraFast gradient\-descent methods for temporal\-difference learning with linear function approximation\.InProceedings of the 26th International Conference on Machine Learning,pp\. 993–1000\.Cited by:[§1](https://arxiv.org/html/2609.19170#S1.p1.1),[§7](https://arxiv.org/html/2609.19170#S7.p3.1)\.
- Suttonet al\.\(2008\)R\. S\. Sutton, H\. R\. Maei, and C\. SzepesváriA convergentO⁡\(n\)O\(n\)temporal\-difference algorithm for off\-policy learning with linear function approximation\.InAdvances in Neural Information Processing Systems 21,pp\. 1609–1616\.Cited by:[§1](https://arxiv.org/html/2609.19170#S1.p1.1),[§7](https://arxiv.org/html/2609.19170#S7.p3.1)\.
- Suttonet al\.\(2016\)R\. S\. Sutton, A\. R\. Mahmood, and M\. WhiteAn emphatic approach to the problem of off\-policy temporal\-difference learning\.Journal of Machine Learning Research17\(73\),pp\. 1–29\.Cited by:[§1](https://arxiv.org/html/2609.19170#S1.p2.1),[§2\.2](https://arxiv.org/html/2609.19170#S2.SS2.p3.2),[§7](https://arxiv.org/html/2609.19170#S7.p3.1),[§7](https://arxiv.org/html/2609.19170#S7.p5.1)\.
- Yu \(2015\)H\. YuOn convergence of emphatic temporal\-difference learning\.InProceedings of the 28th Conference on Learning Theory,Proceedings of Machine Learning Research, Vol\.40,pp\. 1724–1751\.Cited by:[§1](https://arxiv.org/html/2609.19170#S1.p2.1),[§4\.2](https://arxiv.org/html/2609.19170#S4.SS2.p1.1),[§4\.2](https://arxiv.org/html/2609.19170#S4.SS2.p4.1.2),[§7](https://arxiv.org/html/2609.19170#S7.p5.1)\.
- Yu \(2016\)H\. YuWeak convergence properties of constrained emphatic temporal\-difference learning with constant and slowly diminishing stepsize\.Journal of Machine Learning Research17\(219\),pp\. 1–58\.Cited by:[§1](https://arxiv.org/html/2609.19170#S1.p2.1),[§7](https://arxiv.org/html/2609.19170#S7.p5.1)\.
- Zhang and Whiteson \(2022\)S\. Zhang and S\. WhitesonTruncated emphatic temporal difference methods for prediction and control\.Journal of Machine Learning Research23\(153\),pp\. 1–59\.Cited by:[§7](https://arxiv.org/html/2609.19170#S7.p6.1)\.