Robust Average-Reward Markov Decision Processes: Minimax-Optimal Learning via Plug-in Reductions

# Minimax-Optimal Learning via Plug-in Reductions
Source: [https://arxiv.org/html/2608.06545](https://arxiv.org/html/2608.06545)
## Robust Average\-Reward Markov Decision Processes: Minimax\-Optimal Learning via Plug\-in Reductions

Yuepeng Yang Yale & PennDepartment of Statistics and Data Science, Yale University\.Department of Statistics and Data Science, the Wharton School, University of Pennsylvania\.Yuejie Chi11footnotemark:1 Yale

\(August 5, 2026\)

###### Abstract

Distributionally robust Markov decision processes provide a principled framework for sequential decision making under model uncertainty\. We study how many samples are necessary and sufficient to learn anε\\varepsilon\-optimal robust policy under the average\-reward criterion\. A generative model provides samples from the nominal transition kernel, whereas policy performance is evaluated over\(s,a\)\(s,a\)\-rectangular total\-variation uncertainty sets of radius at mostσ\\sigma\.

LetH0H\_\{0\}andHσH\_\{\\sigma\}denote the nominal and robust optimal bias spans, respectively\. We identifyσ​H0\\sigma H\_\{0\}as the perturbation scale separating high\- and low\-tolerance regimes\. Our matching upper and lower bounds show that, up to logarithmic factors, the minimax total sample complexity is

N​S​A≍S​Aε2⋅\{min⁡\{H0,Hσ\},ε≳σ​H0min⁡\{H0,Hσ\}\+σ​Hσ2,ε≲σ​H0,NSA\\asymp\\frac\{SA\}\{\\varepsilon^\{2\}\}\\cdot\\begin\{cases\}\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\},&\\varepsilon\\gtrsim\\sigma H\_\{0\}\\\\\[2\.84526pt\] \\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\+\\sigma H\_\{\\sigma\}^\{2\},&\\varepsilon\\lesssim\\sigma H\_\{0\}\\end\{cases\},whereSSandAAare the numbers of states and actions, andNNis the number of samples per state\-action pair\. The sample complexity consists of a linear\-span term that resembles the nominal AMDP results, and a robustness\-specific term that appears only in the low\-tolerance regime\. We attain these rates using reduction\-based plug\-in procedures that select the reduction—nominal or robust—and its discount factor: a span\-informed procedure that makes these choices using known span parameters, and a span\-agnostic procedure that calibrates both choices from data\.

## 1Introduction

Reinforcement learning \(RL\)\(Sutton and Barto,[2018](https://arxiv.org/html/2608.06545#bib.bib88)\), as a paradigm for sequential decision making under uncertainty, enables agents to learn optimal behavior through interactions with an environment\. RL has found successful applications in domains such as robotics\(Mnihet al\.,[2015](https://arxiv.org/html/2608.06545#bib.bib24); Koberet al\.,[2013](https://arxiv.org/html/2608.06545#bib.bib23)\), game playing\(Silveret al\.,[2016](https://arxiv.org/html/2608.06545#bib.bib25)\), and generative AI\(Guoet al\.,[2025](https://arxiv.org/html/2608.06545#bib.bib45)\)\. One popular model underpinning RL is a Markov decision process \(MDP\), where the goal of the agent is to learn a policy that maximizes some form of aggregated expected reward in the environment\. Common aggregations include the total reward over a finite horizon and the sum of discounted rewards over an infinite horizon\. Despite their popularity, they may be less suitable for continuous learning tasks\(Naiket al\.,[2019](https://arxiv.org/html/2608.06545#bib.bib843)\)\. In this work, we focus on the long\-term average reward:

ρπ​\(s\):=limT→∞𝔼P0π​\[1T​∑t=0T−1r​\(st,at\)∣s0=s\],\\rho^\{\\pi\}\(s\):=\\lim\_\{T\\rightarrow\\infty\}\\mathbb\{E\}\_\{P^\{0\}\}^\{\\pi\}\\left\[\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}r\(s\_\{t\},a\_\{t\}\)\\mid s\_\{0\}=s\\right\],which characterizes the steady\-state performance of a policy by evaluating the expected reward gained per time step as the number of steps approaches infinity\. Here,r​\(st,at\)r\(s\_\{t\},a\_\{t\}\)is the instantaneous reward received at time stepttwhen the agent selects actionata\_\{t\}in statests\_\{t\}according to policyπ\\pi, and the expectation is taken over the randomness of the trajectory according to the transition kernelP0P^\{0\}and policyπ\\pi, given the initial states0=ss\_\{0\}=s\. Unlike discounted settings that prioritize earlier rewards, this metric seeks a policy that maximizes the consistent, long\-term gain across all states\.

A significant challenge in standard RL is the reliance on a fixed probability kernel\. The policy learned in one environment may not be effective in another environment even if the change is modest\(Rameshet al\.,[2024](https://arxiv.org/html/2608.06545#bib.bib26); Sinhaet al\.,[2020](https://arxiv.org/html/2608.06545#bib.bib27)\)\. A popular approach to address this issue is to consider the distributionally robust optimization \(DRO\) framework, where the probability distribution of the environment is allowed to vary within a prescribed uncertainty set instead of being fixed\(Mohajerin Esfahani and Kuhn,[2018](https://arxiv.org/html/2608.06545#bib.bib31); Wiesemannet al\.,[2014](https://arxiv.org/html/2608.06545#bib.bib32); Goh and Sim,[2010](https://arxiv.org/html/2608.06545#bib.bib33); Duchi and Namkoong,[2021](https://arxiv.org/html/2608.06545#bib.bib34)\)\. In the context of Markov decision processes, transition uncertainty has long been studied through robust and distributionally robust formulations\(Iyengar,[2005](https://arxiv.org/html/2608.06545#bib.bib887); Nilim and El Ghaoui,[2005](https://arxiv.org/html/2608.06545#bib.bib865); Xu and Mannor,[2012](https://arxiv.org/html/2608.06545#bib.bib22); Wiesemannet al\.,[2013](https://arxiv.org/html/2608.06545#bib.bib907)\)\. More directly relevant to our setting, distributionally robust average\-reward MDPs \(AMDPs\) model the transition kernel as belonging to a designated uncertainty set𝒫\\mathcal\{P\}\(Wanget al\.,[2023b](https://arxiv.org/html/2608.06545#bib.bib35),[a](https://arxiv.org/html/2608.06545#bib.bib36)\)\. Under this framework, we seek a policy that is effective under the worst\-case scenario within that set, defined as the robust average reward:

ρ𝒫π​\(s\):=minP∈𝒫​limT→∞𝔼Pπ​\[1T​∑t=0T−1r​\(st,at\)∣s0=s\]\.\\rho\_\{\\mathcal\{P\}\}^\{\\pi\}\(s\):=\\min\_\{P\\in\\mathcal\{P\}\}\\lim\_\{T\\rightarrow\\infty\}\\mathbb\{E\}\_\{P\}^\{\\pi\}\\left\[\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}r\(s\_\{t\},a\_\{t\}\)\\mid s\_\{0\}=s\\right\]\.By optimizing against this most pessimistic model, we help ensure that the agent’s performance remains reliable even when the environment dynamics are uncertain\.

This paper studies the statistical cost of distributional robustness in a generative\-model setting, in which an algorithm has access to a simulator that producesNNindependent samples from the nominal transition kernelP0P^\{0\}for every state\-action pair\. Meanwhile, the performance is evaluated through the robust average reward over a rectangular total\-variation uncertainty set aroundP0P^\{0\}\. Within this framework, a key statistical question is:

*How many samples are necessary and sufficient to*

*obtain a policy that isε\\varepsilon\-optimal in robust average reward?*

A key problem parameter for studying sample complexity in average\-reward MDPs is the optimal bias span\. It quantifies the dynamic range of an average\-reward MDP by measuring how much the transient reward relative to the long\-run average varies across initial states\. It is known that nominal average\-reward MDPs have minimax\-optimal sample complexityO~​\(S​A​H0​ε−2\)\\widetilde\{O\}\(SAH\_\{0\}\\varepsilon^\{\-2\}\)\(Wanget al\.,[2022](https://arxiv.org/html/2608.06545#bib.bib4); Zurek and Chen,[2024](https://arxiv.org/html/2608.06545#bib.bib5)\), whereH0H\_\{0\}is the nominal optimal bias span\.

In the robust setting, recent work has established upper bounds for this problem\.Rochet al\.\([2025](https://arxiv.org/html/2608.06545#bib.bib8)\)developed a reduction from robust AMDPs to robust discounted MDPs, whileRochet al\.\([2026](https://arxiv.org/html/2608.06545#bib.bib9)\)proposed a parameter\-free variant of robust Halpern iteration\. The guarantees depend quadratically on the respective robust span parametersHRochH\_\{\\mathrm\{Roch\}\}andHRHIH\_\{\\mathrm\{RHI\}\}defined in the corresponding papers, with sample complexitiesO~​\(S​A​HRoch2​ε−2\)\\widetilde\{O\}\(SAH\_\{\\mathrm\{Roch\}\}^\{2\}\\varepsilon^\{\-2\}\)andO~​\(S​A​HRHI2​ε−2\)\\widetilde\{O\}\(SAH\_\{\\mathrm\{RHI\}\}^\{2\}\\varepsilon^\{\-2\}\), respectively\. The contrast between the minimax theory for nominal AMDPs and the available robust upper bounds leaves our main statistical question largely unresolved\.

### 1\.1Our contributions

We characterize the minimax sample complexity and develop reduction\-based procedures that attain it\. For\(s,a\)\(s,a\)\-rectangular total\-variation uncertainty sets of radius at mostσ\\sigma, our matching upper and lower bounds characterize the minimax sample complexity, up to logarithmic factors, as

N​S​A≍S​Aε2​\{min⁡\{H0,Hσ\},ε≳σ​H0,min⁡\{H0,Hσ\}\+σ​Hσ2,ε≲σ​H0\.NSA\\asymp\\frac\{SA\}\{\\varepsilon^\{2\}\}\\begin\{cases\}\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\},&\\varepsilon\\gtrsim\\sigma H\_\{0\},\\\\\[2\.84526pt\] \\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\+\\sigma H\_\{\\sigma\}^\{2\},&\\varepsilon\\lesssim\\sigma H\_\{0\}\.\\end\{cases\}\(1\)Here,H0H\_\{0\}andHσH\_\{\\sigma\}are the nominal and robust span parameters, respectively\. Both parameters are at least 1, and neither controls the other in general: Proposition[1](https://arxiv.org/html/2608.06545#Thmproposition1)shows that, for any fixedσ\>0\\sigma\>0, any prescribed pair\(H0,Hσ\)\(H\_\{0\},H\_\{\\sigma\}\)can be realized\.

##### High\- and low\-tolerance regimes\.

In Proposition[2](https://arxiv.org/html/2608.06545#Thmproposition2), we show that the optimal robust average reward is at mostσ​H0\\sigma H\_\{0\}smaller than the nominal optimal average reward\. Comparing this perturbation scale with the target toleranceε\\varepsilonseparates two regimes of robust learning\. In the*high\-tolerance regime*σ​H0≲ε\\sigma H\_\{0\}\\lesssim\\varepsilon, solving the nominal AMDP achieves robustε\\varepsilon\-optimality at the standard rateO~​\(S​A​H0​ε−2\)\\widetilde\{O\}\(SAH\_\{0\}\\varepsilon^\{\-2\}\)\. Moreover, using the robust reduction whenHσ≤H0H\_\{\\sigma\}\\leq H\_\{0\}improves it toO~​\(S​A​Hσ​ε−2\)\\widetilde\{O\}\(SAH\_\{\\sigma\}\\varepsilon^\{\-2\}\)\. In the*low\-tolerance regime*ε≲σ​H0\\varepsilon\\lesssim\\sigma H\_\{0\}, however, the perturbation must be accounted for\. Table[1](https://arxiv.org/html/2608.06545#S1.T1)summarizes these regimes under the two orderings ofH0H\_\{0\}andHσH\_\{\\sigma\}\.

Table 1:Four\-regime summary of the minimax\-optimal sample complexity, up to logarithmic factors\. The dashed outline marks the regimes where the robust reduction is used\.Span orderingHigh toleranceσ​H0≲ε\\sigma H\_\{0\}\\lesssim\\varepsilonLow toleranceε≲σ​H0\\varepsilon\\lesssim\\sigma H\_\{0\}Hσ≤H0H\_\{\\sigma\}\\leq H\_\{0\}S​A​Hσε2\\dfrac\{SAH\_\{\\sigma\}\}\{\\varepsilon^\{2\}\}S​A​\(Hσ\+σ​Hσ2\)ε2\\dfrac\{SA\(H\_\{\\sigma\}\+\\sigma H\_\{\\sigma\}^\{2\}\)\}\{\\varepsilon^\{2\}\}H0<HσH\_\{0\}<H\_\{\\sigma\}S​A​H0ε2\\dfrac\{SAH\_\{0\}\}\{\\varepsilon^\{2\}\}S​A​\(H0\+σ​Hσ2\)ε2\\dfrac\{SA\(H\_\{0\}\+\\sigma H\_\{\\sigma\}^\{2\}\)\}\{\\varepsilon^\{2\}\}robust reduction

##### Minimax sample complexity lower bound\.

We prove a minimax lower bound matching the rate \([1](https://arxiv.org/html/2608.06545#S1.E1)\) in Theorem[1](https://arxiv.org/html/2608.06545#Thmtheorem1)\. To our knowledge, this is the first lower bound that reveals the fundamental statistical limit of distributionally robust AMDPs\. It consists of a linear termmin⁡\{H0,Hσ\}\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}that is reminiscent of the standard AMDP lower bound, and a robust\-specific termσ​Hσ2\\sigma H\_\{\\sigma\}^\{2\}that captures the additional cost of robustness in the low\-tolerance regimeε≲σ​H0\\varepsilon\\lesssim\\sigma H\_\{0\}\.

##### Span\-informed and span\-agnostic upper bounds\.

We develop two model\-based procedures based on reductions to discounted problems\. WhenH0H\_\{0\}andHσH\_\{\\sigma\}are known, Algorithm[1](https://arxiv.org/html/2608.06545#alg1)uses the nominal reduction whenH0<HσH\_\{0\}<H\_\{\\sigma\}andσ​H0≲ε\\sigma H\_\{0\}\\lesssim\\varepsilon, and the robust reduction otherwise\. When the spans are unknown, Algorithm[2](https://arxiv.org/html/2608.06545#alg2)uses the data to adaptively select a policy based on the preferable reduction and discount factor\. Both procedures attain the minimax rates \([1](https://arxiv.org/html/2608.06545#S1.E1)\) up to logarithmic factors\. This uniformly improves upon the previous span\-informed\(Rochet al\.,[2025](https://arxiv.org/html/2608.06545#bib.bib8)\)and span\-agnostic bounds\(Rochet al\.,[2026](https://arxiv.org/html/2608.06545#bib.bib9)\)\. Table[2](https://arxiv.org/html/2608.06545#S1.T2)compares our guarantees with existing results for standard and robust AMDPs\.

The upper bounds rely on three complementary ideas: comparing the robust discounted problem with a nominal reference solution, controlling discounted value functions through their spans, and using concentration bounds governed by these spans rather than by the full effective horizon\. Together, these ideas yield the full minimax rate in \([1](https://arxiv.org/html/2608.06545#S1.E1)\), including its sharp dependence on the span parameters, robustness radius, and target accuracy across both tolerance regimes\. For unknown spans, we convert the same bounds into data\-dependent certificates that guide the selection of the reduction and discount factor\.

##### Consequence under anHσH\_\{\\sigma\}\-only assumption\.

Our results also cover the setting where only the robust optimal bias spanHσH\_\{\\sigma\}is assumed bounded, with no analogous assumption onH0H\_\{0\}\. The minimax sample complexity in this setting isΘ~​\(S​A​\(Hσ\+σ​Hσ2\)​ε−2\)\\widetilde\{\\Theta\}\\\!\\left\(SA\(H\_\{\\sigma\}\+\\sigma H\_\{\\sigma\}^\{2\}\)\\varepsilon^\{\-2\}\\right\)\. We attain this rate using the robust reduction whenHσH\_\{\\sigma\}is known and a robust\-only variant of the span\-agnostic procedure when it is unknown\.

Table 2:Comparison of robust and standard AMDP sample\-complexity bounds under a generative model\. Here,HRochH\_\{\\mathrm\{Roch\}\}andHRHIH\_\{\\mathrm\{RHI\}\}denote the respective span parameters used in the two cited works; both are at leastHσH\_\{\\sigma\}\. Span knowledge refers to whether the algorithm uses the relevant span parameter\. Logarithmic factors are omitted\.SettingSpan KnowledgeSample ComplexityReferenceStandard AMDPYesS​A​H0​ε−2SAH\_\{0\}\\varepsilon^\{\-2\}Zurek and Chen \([2024](https://arxiv.org/html/2608.06545#bib.bib5)\)NoS​A​H0​ε−2SAH\_\{0\}\\varepsilon^\{\-2\}Zurek and Chen \([2025](https://arxiv.org/html/2608.06545#bib.bib16)\)Robust AMDPYesS​A​HRoch2​ε−2SAH\_\{\\mathrm\{Roch\}\}^\{2\}\\varepsilon^\{\-2\}Rochet al\.\([2025](https://arxiv.org/html/2608.06545#bib.bib8)\)NoS​A​HRHI2​ε−2SAH\_\{\\mathrm\{RHI\}\}^\{2\}\\varepsilon^\{\-2\}Rochet al\.\([2026](https://arxiv.org/html/2608.06545#bib.bib9)\)YesS​Aε2⋅\{min⁡\{H0,Hσ\},ε≳σ​H0min⁡\{H0,Hσ\}\+σ​Hσ2,ε≲σ​H0\\displaystyle\\frac\{SA\}\{\\varepsilon^\{2\}\}\\cdot\\begin\{cases\}\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\},&\\varepsilon\\gtrsim\\sigma H\_\{0\}\\\\\[2\.84526pt\] \\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\+\\sigma H\_\{\\sigma\}^\{2\},&\\varepsilon\\lesssim\\sigma H\_\{0\}\\end\{cases\}Theorem[2](https://arxiv.org/html/2608.06545#Thmtheorem2)NoTheorem[3](https://arxiv.org/html/2608.06545#Thmtheorem3)\(lower bound\)Theorem[1](https://arxiv.org/html/2608.06545#Thmtheorem1)

### 1\.2Related work

##### Average\-reward MDPs\.

Average\-reward MDPs provide a classical framework for sequential decision making under long\-run performance criteria\(Puterman,[1994](https://arxiv.org/html/2608.06545#bib.bib42)\)\. This criterion is well\-suited for continuing tasks\(Kumaret al\.,[2025](https://arxiv.org/html/2608.06545#bib.bib2)\)in which performance is measured by the steady\-state reward per period rather than by a finite horizon or a discounted sum\. The literature has studied dynamic programming and planning\(Puterman,[1994](https://arxiv.org/html/2608.06545#bib.bib42)\), regret and exploration\(Jakschet al\.,[2010](https://arxiv.org/html/2608.06545#bib.bib18); Bartlett and Tewari,[2009](https://arxiv.org/html/2608.06545#bib.bib6); Fruitet al\.,[2018](https://arxiv.org/html/2608.06545#bib.bib7)\), structural complexity measures such as diameter, mixing time, and bias span\(Jakschet al\.,[2010](https://arxiv.org/html/2608.06545#bib.bib18); Jin and Sidford,[2020](https://arxiv.org/html/2608.06545#bib.bib13),[2021](https://arxiv.org/html/2608.06545#bib.bib1); Wanget al\.,[2022](https://arxiv.org/html/2608.06545#bib.bib4); Zurek and Chen,[2024](https://arxiv.org/html/2608.06545#bib.bib5)\), and model\-free learning and function approximation\(Weiet al\.,[2020](https://arxiv.org/html/2608.06545#bib.bib19),[2021](https://arxiv.org/html/2608.06545#bib.bib20); Zhang and Xie,[2023](https://arxiv.org/html/2608.06545#bib.bib17); Jinet al\.,[2024](https://arxiv.org/html/2608.06545#bib.bib53); Leeet al\.,[2025](https://arxiv.org/html/2608.06545#bib.bib49); Jiaoet al\.,[2026](https://arxiv.org/html/2608.06545#bib.bib67)\)\.

##### Distributionally robust MDPs\.

Distributionally robust MDPs build on the broader robust optimization principle of optimizing against a worst\-case model in an ambiguity set\. Some foundational references on distributionally robust MDPs includeIyengar \([2005](https://arxiv.org/html/2608.06545#bib.bib887)\); Nilim and El Ghaoui \([2005](https://arxiv.org/html/2608.06545#bib.bib865)\); Xu and Mannor \([2012](https://arxiv.org/html/2608.06545#bib.bib22)\)\. In the discounted setting, a growing literature has established finite\-sample guarantees for robust and distributionally robust RL\. Under a generative model, such guarantees have been established for model\-based approaches\(Yanget al\.,[2022](https://arxiv.org/html/2608.06545#bib.bib37); Panaganti and Kalathil,[2022](https://arxiv.org/html/2608.06545#bib.bib29); Shiet al\.,[2026](https://arxiv.org/html/2608.06545#bib.bib3); Clavieret al\.,[2024](https://arxiv.org/html/2608.06545#bib.bib39)\)and model\-free Q\-learning\(Yanget al\.,[2022](https://arxiv.org/html/2608.06545#bib.bib37); Wanget al\.,[2024b](https://arxiv.org/html/2608.06545#bib.bib38)\)\. In the offline setting, robust policy learning from pre\-collected datasets has been studied\(Panagantiet al\.,[2022](https://arxiv.org/html/2608.06545#bib.bib28); Shi and Chi,[2024](https://arxiv.org/html/2608.06545#bib.bib40); Wanget al\.,[2024a](https://arxiv.org/html/2608.06545#bib.bib41)\)\.

##### Robust average\-reward MDPs\.

For the average\-reward criterion,Wanget al\.\([2023b](https://arxiv.org/html/2608.06545#bib.bib35)\)andWanget al\.\([2023a](https://arxiv.org/html/2608.06545#bib.bib36)\)developed robust Bellman equations, robust relative value iteration, and model\-free robust AMDP algorithms\.Chenet al\.\([2025](https://arxiv.org/html/2608.06545#bib.bib12)\)established mixing\-time\-based sample\-complexity guarantees for robust AMDPs under a uniform ergodicity condition over all transition kernels in the uncertainty set\. Recent work has also considered efficient model\-free robust average\-reward methods and non\-rectangular robust AMDPs\(Xuet al\.,[2026](https://arxiv.org/html/2608.06545#bib.bib21); Wang and Si,[2026](https://arxiv.org/html/2608.06545#bib.bib11)\)\.

##### Distributionally robust optimization\.

Our formulation is also connected to the broader DRO literature, which studies decision\-making rules with uniform performance over ambiguity sets\. Representative foundational references includeGoh and Sim \([2010](https://arxiv.org/html/2608.06545#bib.bib33)\),Wiesemannet al\.\([2014](https://arxiv.org/html/2608.06545#bib.bib32)\),Mohajerin Esfahani and Kuhn \([2018](https://arxiv.org/html/2608.06545#bib.bib31)\), andDuchi and Namkoong \([2021](https://arxiv.org/html/2608.06545#bib.bib34)\)\. The DRO viewpoint provides a principled way to trade nominal performance for reliability under sampling error, model misspecification, and distribution shift, while retaining an optimization problem whose conservatism is controlled explicitly by the ambiguity set\.

### 1\.3Paper organization and notation

The rest of the paper is organized as follows\. Section[2](https://arxiv.org/html/2608.06545#S2)formulates distributionally robust average\-reward MDPs and defines the structural assumptions and span parameters used throughout the paper\. Section[3](https://arxiv.org/html/2608.06545#S3)proves the minimax lower bound and explains why the robust optimal bias spanHσH\_\{\\sigma\}is the relevant complexity parameter\. Section[4](https://arxiv.org/html/2608.06545#S4)develops the reduction from robust average\-reward MDPs to robust discounted MDPs, presents both the span\-informed and span\-agnostic algorithms, and proves the corresponding sample complexity upper bounds\. Section[5](https://arxiv.org/html/2608.06545#S5)presents experiments illustrating our theoretical predictions\. Section[6](https://arxiv.org/html/2608.06545#S6)concludes with a discussion of future directions\. The appendices collect the proofs and supplementary experimental results\.

##### Notation\.

For any finite set𝒳\\mathcal\{X\},Δ​\(𝒳\)\\Delta\(\\mathcal\{X\}\)denotes the probability simplex over𝒳\\mathcal\{X\}\. For any functionh:𝒮→ℝh:\\mathcal\{S\}\\rightarrow\\mathbb\{R\}, its span seminorm is defined as‖h‖span:=maxs⁡h​\(s\)−mins⁡h​\(s\)\\\|h\\\|\_\{\\mathrm\{span\}\}:=\\max\_\{s\}h\(s\)\-\\min\_\{s\}h\(s\)\. For a stationary policyπ\\piand a transition kernelPP, we writePπP\_\{\\pi\}for the induced state\-to\-state transition matrix, so that\(Pπ​h\)​\(s\)=𝔼a∼π\(⋅∣s\)​∑s′∈𝒮P​\(s′∣s,a\)​h​\(s′\)\(P\_\{\\pi\}h\)\(s\)=\\mathbb\{E\}\_\{a\\sim\\pi\(\\cdot\\mid s\)\}\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}P\(s^\{\\prime\}\\mid s,a\)h\(s^\{\\prime\}\)\. The total variation distance between two probability distributionsPPandQQover a finite state space𝒮\\mathcal\{S\}is defined as‖P−Q‖TV=12​∑s∈𝒮\|P​\(s\)−Q​\(s\)\|\\\|P\-Q\\\|\_\{\\mathrm\{TV\}\}=\\frac\{1\}\{2\}\\sum\_\{s\\in\\mathcal\{S\}\}\|P\(s\)\-Q\(s\)\|\. We use standard asymptotic notation such asO~​\(⋅\)\\widetilde\{O\}\(\\cdot\),Θ~​\(⋅\)\\widetilde\{\\Theta\}\(\\cdot\), andΩ​\(⋅\)\\Omega\(\\cdot\), where the tilde indicates the suppression of logarithmic factors\. For nonnegativexxandyy, we writex≲yx\\lesssim yifx≤C​yx\\leq Cyfor a universal constantC\>0C\>0, andx≳yx\\gtrsim yifx≥C​yx\\geq Cyfor a universal constantC\>0C\>0\. We also writex≪yx\\ll yandx≫yx\\gg yto indicate a separation of scales\. In particular, we use1−O​\(δ\)1\-O\(\\delta\)to mean that an event occurs with probability at least1−C​δ1\-C\\deltafor some constantCC\. We use𝟏\\mathbf\{1\}to denote the all\-ones vector, with its dimension clear from context\. For any vectors𝒙,𝒚∈ℝd\\bm\{x\},\\bm\{y\}\\in\\mathbb\{R\}^\{d\}, we use𝒙≤𝒚\\bm\{x\}\\leq\\bm\{y\}to denotexi≤yix\_\{i\}\\leq y\_\{i\}for alli∈\{1,…,d\}i\\in\\\{1,\\ldots,d\\\}\. For a scalarxx, let\[x\]\+≔max⁡\{x,0\}\[x\]\_\{\+\}\\coloneqq\\max\\\{x,0\\\}; for a vector,\[⋅\]\+\[\\cdot\]\_\{\+\}is applied coordinatewise\. Let\[N\]\[N\]be\{1,…,N\}\\\{1,\\ldots,N\\\}\.

## 2Problem formulation

This section sets up the model of robust average\-reward MDPs and the span parameters that determine the sample complexity\. We begin with the basic definitions of the nominal and robust average\-reward MDPs\. We then state the structural assumptions and explain the split into high\- and low\-tolerance regimes\.

### 2\.1Robust average\-reward MDPs

##### Standard average\-reward MDP\.

We start by introducing the standard average\-reward Markov decision process \(AMDP\), which is specified byℳ0=\(𝒮,𝒜,P0,r\)\\mathcal\{M\}^\{0\}=\(\\mathcal\{S\},\\mathcal\{A\},P^\{0\},r\)\. Here,𝒮=\{1,…,S\}\\mathcal\{S\}=\\\{1,\\ldots,S\\\}is the state space,𝒜=\{1,…,A\}\\mathcal\{A\}=\\\{1,\\ldots,A\\\}is the action space,P0=\{Ps,a0\}\(s,a\)∈𝒮×𝒜P^\{0\}=\\\{P\_\{s,a\}^\{0\}\\\}\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}is the transition kernel, wherePs,a0P\_\{s,a\}^\{0\}is the next\-state distribution given the state\-action pair\(s,a\)\(s,a\), andr:𝒮×𝒜→\[0,1\]r:\\mathcal\{S\}\\times\\mathcal\{A\}\\to\[0,1\]is the reward function\. A stationary policyπ:𝒮→Δ​\(𝒜\)\\pi:\\mathcal\{S\}\\to\\Delta\(\\mathcal\{A\}\)specifies an action selection rule for a given states∈𝒮s\\in\\mathcal\{S\}, whereπ​\(s\)\\pi\(s\)is a probability distribution over the action space\.

Average reward measures the long\-run steady\-state value of a policy\. For a transition kernelP0P^\{0\}and a policyπ\\pi, the average reward from initial statessis

ρP0π​\(s\)≔limT→∞𝔼P0π​\[1T​∑t=0T−1r​\(st,at\)∣s0=s\],\\rho\_\{P^\{0\}\}^\{\\pi\}\(s\)\\coloneqq\\lim\_\{T\\rightarrow\\infty\}\\mathbb\{E\}\_\{P^\{0\}\}^\{\\pi\}\\left\[\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}r\(s\_\{t\},a\_\{t\}\)\\mid s\_\{0\}=s\\right\],whenever the limit exists\. The expectation is taken over the actionat∼π​\(st\)a\_\{t\}\\sim\\pi\(s\_\{t\}\)and the next statest\+1∼Pst,at0s\_\{t\+1\}\\sim P^\{0\}\_\{s\_\{t\},a\_\{t\}\}\. Under kernelP0P^\{0\}, denote the optimal average reward from initial statessby

ρP0⋆​\(s\)≔supπρP0π​\(s\)\.\\rho\_\{P^\{0\}\}^\{\\star\}\(s\)\\coloneqq\\sup\_\{\\pi\}\\rho\_\{P^\{0\}\}^\{\\pi\}\(s\)\.

##### Distributionally robust AMDP\.

Since the performance of a policy can be sensitive to perturbations of the transition kernel, the distributionally robust formulation evaluates policies by their worst\-case performance against all plausible transition kernels near a nominal oneP0P^\{0\}\. Specifically, a distributionally robust AMDP is written asℳ=\(𝒮,𝒜,P0,𝒰,r\)\\mathcal\{M\}=\(\\mathcal\{S\},\\mathcal\{A\},P^\{0\},\\mathcal\{U\},r\), where\(𝒮,𝒜,P0,r\)\(\\mathcal\{S\},\\mathcal\{A\},P^\{0\},r\)is the nominal AMDP described above and𝒰\\mathcal\{U\}describes the admissible transition perturbations by mapping a transition kernel to a set of transition kernels\. We focus on\(s,a\)\(s,a\)\-rectangular total\-variation \(TV\) uncertainty sets, meaning that the uncertainty for each state\-action pair is decoupled\. For each state\-action pair\(s,a\)\(s,a\), we are given a local radiusσs,a∈\[0,σ\]\\sigma\_\{s,a\}\\in\[0,\\sigma\]\. The uncertainty set nearP0P^\{0\}, denoted by𝒫≔𝒰​\(P0\)\\mathcal\{P\}\\coloneqq\\mathcal\{U\}\(P^\{0\}\), is defined as

𝒰​\(Q\)≔∏\(s,a\)∈𝒮×𝒜𝒰s,a​\(Qs,a\),𝒰s,a​\(Qs,a\)=\{Ps,a∈Δ​\(𝒮\):‖Ps,a−Qs,a‖TV≤σs,a\},\\mathcal\{U\}\(Q\)\\coloneqq\\prod\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}\\mathcal\{U\}\_\{s,a\}\(Q\_\{s,a\}\),\\quad\\mathcal\{U\}\_\{s,a\}\(Q\_\{s,a\}\)=\\left\\\{P\_\{s,a\}\\in\\Delta\(\\mathcal\{S\}\):\\\|P\_\{s,a\}\-Q\_\{s,a\}\\\|\_\{\\mathrm\{TV\}\}\\leq\\sigma\_\{s,a\}\\right\\\},for any transition kernelQ=\{Qs,a\}Q=\\\{Q\_\{s,a\}\\\}\. Furthermore, let𝒫s,a≔𝒰s,a​\(Ps,a0\)\\mathcal\{P\}\_\{s,a\}\\coloneqq\\mathcal\{U\}\_\{s,a\}\(P\_\{s,a\}^\{0\}\)be the uncertainty set for the transition vectorPs,a0P\_\{s,a\}^\{0\}at state\-action pair\(s,a\)\(s,a\)\. Here we use a broader class of uncertainty sets that allow local radiiσs,a\\sigma\_\{s,a\}to be smaller thanσ\\sigma, whileShiet al\.\([2026](https://arxiv.org/html/2608.06545#bib.bib3)\)assumesσs,a=σ\\sigma\_\{s,a\}=\\sigmafor all\(s,a\)\(s,a\)\.

For a policyπ\\pi, the robust average reward is the worst\-case average reward over𝒫\\mathcal\{P\}, and the robust optimal average reward is the best such worst\-case value:

ρπ,σ​\(s\)≔infP∈𝒫ρPπ​\(s\)\.\\rho^\{\\pi,\\sigma\}\(s\)\\coloneqq\\inf\_\{P\\in\\mathcal\{P\}\}\\rho\_\{P\}^\{\\pi\}\(s\)\.The optimal robust average reward from initial statessis

ρ⋆,σ​\(s\)≔supπρπ,σ​\(s\)\.\\rho^\{\\star,\\sigma\}\(s\)\\coloneqq\\sup\_\{\\pi\}\\rho^\{\\pi,\\sigma\}\(s\)\.

##### Sampling model and the goal\.

We assume that we have access to a generative model that samples from the nominal transition kernelP0P^\{0\}\. For each\(s,a\)∈𝒮×𝒜\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}, we generateNNindependent samples

si,s,a′∼Ps,a0,i=1,…,N\.s\_\{i,s,a\}^\{\\prime\}\\sim P^\{0\}\_\{s,a\},\\qquad i=1,\\ldots,N\.We also assume that the reward functionrris known\.

Our learning goal is to use as few samples as possible to compute a policyπ^\\widehat\{\\pi\}such that, for every initial statess,

ρ⋆,σ​\(s\)−ρπ^,σ​\(s\)≤ε,\\rho^\{\\star,\\sigma\}\(s\)\-\\rho^\{\\widehat\{\\pi\},\\sigma\}\(s\)\\leq\\varepsilon,for a target accuracyε\\varepsilon\.

### 2\.2Assumptions and key parameters

We first state a structural assumption used throughout our analysis and then introduce the nominal and robust optimal bias spans that govern the sample complexity\.

###### Assumption 1\(Unichain\)\.

For every stationary policyπ\\piand transition kernelP∈𝒫P\\in\\mathcal\{P\}, the induced Markov chain contains exactly one recurrent class\.

This assumption ensures thatρP0π​\(s\)\\rho\_\{P^\{0\}\}^\{\\pi\}\(s\)andρπ,σ​\(s\)\\rho^\{\\pi,\\sigma\}\(s\)are well defined and independent of the initial statessfor every stationary policyπ\\pi; seeWanget al\.\([2023b](https://arxiv.org/html/2608.06545#bib.bib35),[a](https://arxiv.org/html/2608.06545#bib.bib36)\)\. We therefore omit the state argument from now on and writeρπ\\rho^\{\\pi\},ρ⋆\\rho^\{\\star\},ρπ,σ\\rho^\{\\pi,\\sigma\}, andρ⋆,σ\\rho^\{\\star,\\sigma\}as scalars\. We also letπσ⋆∈arg⁡maxπ⁡ρπ,σ\\pi^\{\\star\}\_\{\\sigma\}\\in\\arg\\max\_\{\\pi\}\\rho^\{\\pi,\\sigma\}denote a robust optimal policy\.

##### Optimal bias functions\.

The average reward captures only the long\-run reward rate\. A bias function complements it by measuring differences in the transient reward accumulated from different initial states relative to this rate\. In this paper, we use nominal and robust optimal bias functions, characterized through their respective average\-reward Bellman optimality equations\. Forh∈ℝ𝒮h\\in\\mathbb\{R\}^\{\\mathcal\{S\}\}, define the corresponding Bellman operators by

\(𝒯0​h\)​\(s\)\\displaystyle\(\\mathcal\{T\}\_\{0\}h\)\(s\)≔maxa∈𝒜⁡\{r​\(s,a\)\+\(Ps,a0\)⊤​h\},\\displaystyle\\coloneqq\\max\_\{a\\in\\mathcal\{A\}\}\\left\\\{r\(s,a\)\+\(P^\{0\}\_\{s,a\}\)^\{\\top\}h\\right\\\},\(𝒯σ​h\)​\(s\)\\displaystyle\(\\mathcal\{T\}\_\{\\sigma\}h\)\(s\)≔maxa∈𝒜⁡\{r​\(s,a\)\+minPs,a∈𝒫s,a⁡Ps,a⊤​h\}\.\\displaystyle\\coloneqq\\max\_\{a\\in\\mathcal\{A\}\}\\left\\\{r\(s,a\)\+\\min\_\{P\_\{s,a\}\\in\\mathcal\{P\}\_\{s,a\}\}P\_\{s,a\}^\{\\top\}h\\right\\\}\.A nominal optimal bias is any vectorh∈ℝ𝒮h\\in\\mathbb\{R\}^\{\\mathcal\{S\}\}satisfying the first equation below, whereas a robust optimal bias is any vectorh∈ℝ𝒮h\\in\\mathbb\{R\}^\{\\mathcal\{S\}\}satisfying the second:

ρ⋆​𝟏\+h\\displaystyle\\rho^\{\\star\}\\mathbf\{1\}\+h=𝒯0​h,\\displaystyle=\\mathcal\{T\}\_\{0\}h,\(2a\)ρ⋆,σ​𝟏\+h\\displaystyle\\rho^\{\\star,\\sigma\}\\mathbf\{1\}\+h=𝒯σ​h\.\\displaystyle=\\mathcal\{T\}\_\{\\sigma\}h\.\(2b\)Both Bellman equations are invariant under adding a constant tohh\.

##### Optimal bias spans\.

For any vectorh∈ℝ𝒮h\\in\\mathbb\{R\}^\{\\mathcal\{S\}\}, define its span seminorm by

‖h‖span≔maxs∈𝒮⁡h​\(s\)−mins∈𝒮⁡h​\(s\)\.\\\|h\\\|\_\{\\mathrm\{span\}\}\\coloneqq\\max\_\{s\\in\\mathcal\{S\}\}h\(s\)\-\\min\_\{s\\in\\mathcal\{S\}\}h\(s\)\.Define the*nominal optimal bias span*and the*robust optimal bias span*by

H0\\displaystyle H\_\{0\}≔max⁡\{1,infh:ρ⋆​𝟏\+h=𝒯0​h‖h‖span\},\\displaystyle\\coloneqq\\max\\left\\\{1,\\inf\_\{h:\\,\\rho^\{\\star\}\\mathbf\{1\}\+h=\\mathcal\{T\}\_\{0\}h\}\\\|h\\\|\_\{\\mathrm\{span\}\}\\right\\\},\(3a\)Hσ\\displaystyle H\_\{\\sigma\}≔max⁡\{1,infh:ρ⋆,σ​𝟏\+h=𝒯σ​h‖h‖span\}\.\\displaystyle\\coloneqq\\max\\left\\\{1,\\inf\_\{h:\\,\\rho^\{\\star,\\sigma\}\\mathbf\{1\}\+h=\\mathcal\{T\}\_\{\\sigma\}h\}\\\|h\\\|\_\{\\mathrm\{span\}\}\\right\\\}\.\(3b\)The inner infima in \([3a](https://arxiv.org/html/2608.06545#S2.E3.1)\) and \([3b](https://arxiv.org/html/2608.06545#S2.E3.2)\) are attained; see Propositions[3](https://arxiv.org/html/2608.06545#Thmproposition3)and[4](https://arxiv.org/html/2608.06545#Thmproposition4)in Appendix[A\.2](https://arxiv.org/html/2608.06545#A1.SS2)\. We denote the corresponding minimum\-span solutions byhP0⋆h\_\{P^\{0\}\}^\{\\star\}andh⋆,σh^\{\\star,\\sigma\}, respectively\.

###### Proposition 1\(Independence of the nominal and robust optimal bias spans\)\.

Fixσ\>0\\sigma\>0and anyH0,Hσ≥1H\_\{0\},H\_\{\\sigma\}\\geq 1\. There exists a robust AMDP satisfying Assumption[1](https://arxiv.org/html/2608.06545#Thmassumption1), with local radii at mostσ\\sigma, whose nominal optimal bias span is exactlyH0H\_\{0\}and whose robust optimal bias span is exactlyHσH\_\{\\sigma\}\.

### 2\.3High\- and low\-tolerance regimes

The perturbation scaleσ​H0\\sigma H\_\{0\}separates the*high\-tolerance regime*σ​H0≲ε\\sigma H\_\{0\}\\lesssim\\varepsilon, where a suitable nominal optimal policy is guaranteed to beε\\varepsilon\-optimal for the robust problem, from the*low\-tolerance regime*ε≲σ​H0\\varepsilon\\lesssim\\sigma H\_\{0\}, where nominal optimization is no longer guaranteed to suffice\. The following proposition formalizes this by providing a perturbation bound on the optimal average reward\. Its proof is deferred to Appendix[B\.2](https://arxiv.org/html/2608.06545#A2.SS2)\.

###### Proposition 2\(Nominal\-to\-robust perturbation bound\)\.

There exists a nominal optimal policyπ\\pisuch that

ρ⋆−σ​H0≤ρπ,σ≤ρ⋆,σ≤ρ⋆\.\\rho^\{\\star\}\-\\sigma H\_\{0\}\\leq\\rho^\{\\pi,\\sigma\}\\leq\\rho^\{\\star,\\sigma\}\\leq\\rho^\{\\star\}\.

Proposition[2](https://arxiv.org/html/2608.06545#Thmproposition2)shows that the difference between the robust optimal average rewardρ⋆,σ\\rho^\{\\star,\\sigma\}and the nominal optimal average rewardρ⋆\\rho^\{\\star\}is at mostσ​H0\\sigma H\_\{0\}\. Moreover, there exists a nominal optimal policy that isσ​H0\\sigma H\_\{0\}\-optimal for the robust problem\. This observation suggests that in the high\-tolerance regime, we should be able to learn a robustε\\varepsilon\-optimal policy by solving the nominal problem\.

## 3Minimax sample complexity lower bound

In this section, we establish a minimax lower bound for robust AMDPs\. Consider a bounded class of robust AMDPs—denoted by𝔐​\(H0,Hσ,σ\)\\mathfrak\{M\}\(H\_\{0\},H\_\{\\sigma\},\\sigma\)—that satisfy Assumption[1](https://arxiv.org/html/2608.06545#Thmassumption1)and have local uncertainty radii at mostσ\\sigma,\|𝒮\|=S\|\\mathcal\{S\}\|=S,\|𝒜\|=A\|\\mathcal\{A\}\|=A, and span constraints

‖hP0⋆‖span≤H0,‖h⋆,σ‖span≤Hσ\.\\\|h\_\{P^\{0\}\}^\{\\star\}\\\|\_\{\\mathrm\{span\}\}\\leq H\_\{0\},\\qquad\\\|h^\{\\star,\\sigma\}\\\|\_\{\\mathrm\{span\}\}\\leq H\_\{\\sigma\}\.The following theorem characterizes the minimum sample complexity required to obtain a policy that isε\\varepsilon\-optimal in the robust average reward\. The proof is deferred to Appendix[C](https://arxiv.org/html/2608.06545#A3)\.

###### Theorem 1\.

Letε∈\(0,0\.01\]\\varepsilon\\in\(0,0\.01\]\. Assume thatS≥5S\\geq 5,A≥3A\\geq 3,min⁡\{H0,Hσ\}≥4\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\\geq 4, and0<σ≤1/20<\\sigma\\leq 1/2\. For some sufficiently small universal constantC\>0C\>0, suppose the number of samplesN​S​ANSAsatisfies \(a\)

N​S​A≤C​S​A​min⁡\{H0,Hσ\}ε2,NSA\\leq\\frac\{CSA\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\}\{\\varepsilon^\{2\}\},or \(b\)ε≤0\.01​σ​H0\\varepsilon\\leq 0\.01\\sigma H\_\{0\}, and

N​S​A≤C​S​A​\(min⁡\{H0,Hσ\}\+σ​Hσ2\)ε2\.NSA\\leq\\frac\{CSA\\left\(\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\+\\sigma H\_\{\\sigma\}^\{2\}\\right\)\}\{\\varepsilon^\{2\}\}\.Then

infπ^supℳ∈𝔐​\(H0,Hσ,σ\)ℙℳ​\{ρ⋆,σ−ρπ^,σ\>ε\}≥14\.\\inf\_\{\\widehat\{\\pi\}\}\\sup\_\{\\mathcal\{M\}\\in\\mathfrak\{M\}\(H\_\{0\},H\_\{\\sigma\},\\sigma\)\}\\mathbb\{P\}\_\{\\mathcal\{M\}\}\\\!\\left\\\{\\rho^\{\\star,\\sigma\}\-\\rho^\{\\widehat\{\\pi\},\\sigma\}\>\\varepsilon\\right\\\}\\geq\\frac\{1\}\{4\}\.

Theorem[1](https://arxiv.org/html/2608.06545#Thmtheorem1)shows that to achieve anε\\varepsilon\-optimal policy in the robust average reward, every algorithm requires at least

Ω​\(S​Aε2​\(min⁡\{H0,Hσ\}⏟linear min\-span term\+σ​Hσ2⏟robust\-specific term\)\)\\Omega\\Bigg\(\\frac\{SA\}\{\\varepsilon^\{2\}\}\\Big\(\\underbrace\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\}\_\{\\text\{linear min\-span term\}\}\+\\underbrace\{\\sigma H\_\{\\sigma\}^\{2\}\}\_\{\\text\{robust\-specific term\}\}\\Big\)\\Bigg\)samples, where the robust\-specific term is active whenε≤0\.01​σ​H0\\varepsilon\\leq 0\.01\\sigma H\_\{0\}\. This gives the lower\-bound side of the four\-regime summary in Table[1](https://arxiv.org/html/2608.06545#S1.T1)\. To the best of our knowledge, this is the first characterization of the minimax sample complexity of robust AMDPs\.

The lower bound consists of two components\. The linear min\-span termmin⁡\{H0,Hσ\}\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}is analogous to theH0H\_\{0\}term in the standard AMDP lower bound, but depends on the smaller of the two span parameters\. This reflects the fact that the MDP class𝔐​\(H0,Hσ,σ\)\\mathfrak\{M\}\(H\_\{0\},H\_\{\\sigma\},\\sigma\)is constrained by both span parameters\.

On the other hand, the robust\-specific termσ​Hσ2\\sigma H\_\{\\sigma\}^\{2\}captures the additional cost of distributional robustness\. For comparison,Shiet al\.\([2026](https://arxiv.org/html/2608.06545#bib.bib3)\)establish a robust discounted\-MDP lower bound when every local uncertainty radius is fixed atσ\\sigmaand find that robustness can require fewer samples than the standard problem\. This contrast emphasizes that the specification of the uncertainty sets affects sample complexity\.

##### Consequence under anHσH\_\{\\sigma\}\-only assumption\.

If only the robust optimal bias spanHσH\_\{\\sigma\}is assumed bounded, Proposition[1](https://arxiv.org/html/2608.06545#Thmproposition1)implies thatH0H\_\{0\}can be arbitrarily large\. Then Theorem[1](https://arxiv.org/html/2608.06545#Thmtheorem1)gives the minimax lower boundΩ​\(S​A​\(Hσ\+σ​Hσ2\)​ε−2\)\\Omega\\\!\\left\(SA\(H\_\{\\sigma\}\+\\sigma H\_\{\\sigma\}^\{2\}\)\\varepsilon^\{\-2\}\\right\)\.

## 4The plug\-in approach for robust AMDPs

### 4\.1Motivation for a reduction to DMDPs

Directly solving the robust average\-reward problem is difficult because the objective is defined through the long\-run robust gain, and its Bellman operator is not contractive\. The discounted MDP, on the other hand, offers a more tractable surrogate: if the discount factorγ\\gammais close enough to11, then the robust discounted value approximates the robust average reward after multiplying by1−γ1\-\\gamma\. Formally, for a fixed policyπ\\piand discount factorγ∈\(0,1\)\\gamma\\in\(0,1\), let

Vγπ,σ​\(s\)=infP∈𝒫𝔼Pπ​\[∑t=0∞γt​r​\(st,at\)∣s0=s\]V\_\{\\gamma\}^\{\\pi,\\sigma\}\(s\)=\\inf\_\{P\\in\\mathcal\{P\}\}\\mathbb\{E\}\_\{P\}^\{\\pi\}\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}r\(s\_\{t\},a\_\{t\}\)\\mid s\_\{0\}=s\\right\]be the robust discounted value function\. The robust optimal discounted value is defined componentwise by

Vγ⋆,σ​\(s\)≔supπVγπ,σ​\(s\),s∈𝒮\.V\_\{\\gamma\}^\{\\star,\\sigma\}\(s\)\\coloneqq\\sup\_\{\\pi\}V\_\{\\gamma\}^\{\\pi,\\sigma\}\(s\),\\qquad s\\in\\mathcal\{S\}\.Lemma[2](https://arxiv.org/html/2608.06545#Thmlemma2)shows that compactness and\(s,a\)\(s,a\)\-rectangularity identify these values with the unique fixed points of their respective robust discounted Bellman operators\. The fixed\-policy discounted value approximates the robust average rewardρπ,σ\\rho^\{\\pi,\\sigma\}, especially whenγ\\gammais close to11\.

This motivates a reduction\-based route: choose a large effective horizon\(1−γ\)−1\(1\-\\gamma\)^\{\-1\}and return the optimal policy for the corresponding robust discounted MDP as the solution to the robust AMDP problem\. The choice ofγ\\gammais subtle as it balances two competing requirements: when it is too far from 1, the discounted objective does not approximate the average\-reward objective well; when it is too close to 1, the statistical error of estimating the discounted problem becomes too large, since the sample complexity of DMDPs grows polynomially with the effective horizon\(1−γ\)−1\(1\-\\gamma\)^\{\-1\}\(Liet al\.,[2024](https://arxiv.org/html/2608.06545#bib.bib147)\)\. In the next two sections, we consider two algorithms based on this reduction principle: a span\-informed procedure for knownH0H\_\{0\}andHσH\_\{\\sigma\}, and a span\-agnostic procedure for unknown spans\.

### 4\.2The span\-informed reduction

For each state\-action pair\(s,a\)∈𝒮×𝒜\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}, we observeNNindependent next\-state samples from the nominal transition kernelPs,a0P\_\{s,a\}^\{0\}and form the empirical nominal kernel

P^s,a0​\(s′\)=1N​∑i=1N𝟏si′=s′,\(s,a,s′\)∈𝒮×𝒜×𝒮\.\\widehat\{P\}^\{0\}\_\{s,a\}\(s^\{\\prime\}\)=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbf\{1\}\_\{s\_\{i\}^\{\\prime\}=s^\{\\prime\}\},\\qquad\(s,a,s^\{\\prime\}\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\\times\\mathcal\{S\}\.\(4\)GivenP^0\\widehat\{P\}^\{0\}, we assume access to a distributionally robust DMDP solver that returns a policyπ^\\widehat\{\\pi\}satisfying

‖V^γ⋆,σ−V^γπ^,σ‖∞≤εopt,\\left\\\|\\widehat\{V\}\_\{\\gamma\}^\{\\star,\\sigma\}\-\\widehat\{V\}\_\{\\gamma\}^\{\\widehat\{\\pi\},\\sigma\}\\right\\\|\_\{\\infty\}\\leq\\varepsilon\_\{\\mathrm\{opt\}\},\(5\)where

V^γπ,σ​\(s\)=infP∈𝒰​\(P^0\)𝔼Pπ​\[∑t=0∞γt​r​\(st,at\)∣s0=s\],\\widehat\{V\}\_\{\\gamma\}^\{\\pi,\\sigma\}\(s\)=\\inf\_\{P\\in\\mathcal\{U\}\(\\widehat\{P\}^\{0\}\)\}\\mathbb\{E\}\_\{P\}^\{\\pi\}\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}r\(s\_\{t\},a\_\{t\}\)\\mid s\_\{0\}=s\\right\],andV^γ⋆,σ\\widehat\{V\}\_\{\\gamma\}^\{\\star,\\sigma\}is the corresponding optimal value\. We refer toεopt\\varepsilon\_\{\\mathrm\{opt\}\}as the solver tolerance\. This solver also serves as a nominal DMDP solver when𝒰​\(P^0\)\\mathcal\{U\}\(\\widehat\{P\}^\{0\}\)is the singleton\{P^0\}\\\{\\widehat\{P\}^\{0\}\\\}\. Under the\(s,a\)\(s,a\)\-rectangular TV uncertainty sets in Section[2](https://arxiv.org/html/2608.06545#S2), robust value iteration or policy iteration can solve this discounted problem to arbitrary accuracy\(Iyengar,[2005](https://arxiv.org/html/2608.06545#bib.bib887); Nilim and El Ghaoui,[2005](https://arxiv.org/html/2608.06545#bib.bib865); Hoet al\.,[2018](https://arxiv.org/html/2608.06545#bib.bib909)\)\.

As discussed in Section[2\.3](https://arxiv.org/html/2608.06545#S2.SS3), in the high\-tolerance regime, it is possible to find a nominally optimal policy that is robustlyε\\varepsilon\-optimal\. Moreover, intuitively, the nominal problem could be statistically easier to solve than the robust problem\. This suggests that we should consider both the nominal and robust discounted reductions and select the one that is more sample\-efficient\.

The span\-informed procedure treatsH0H\_\{0\}andHσH\_\{\\sigma\}as known\. It uses them to select between nominal and robust discounted reductions and pick a discount factorγ\\gammaof the right scale\.

Algorithm 1Span\-informed procedure for robust AMDPsInput\.For every\(s,a\)\(s,a\),NNnominal transition samples\{\(s,a,si′\)\}i=1N\\\{\(s,a,s\_\{i\}^\{\\prime\}\)\\\}\_\{i=1\}^\{N\}; the uncertainty set rule𝒰\\mathcal\{U\}; the nominal and robust optimal bias spansH0H\_\{0\}andHσH\_\{\\sigma\}; the desired accuracyε\\varepsilon; and the solver toleranceεopt\\varepsilon\_\{\\mathrm\{opt\}\}\.

1. 1\.Estimate the nominal model\.ConstructP^0\\widehat\{P\}^\{0\}from theNNsamples according to \([4](https://arxiv.org/html/2608.06545#S4.E4)\)\.
2. 2\.Nominal or robust reduction\. - •IfH0<HσH\_\{0\}<H\_\{\\sigma\}and7​σ​H0≤ε7\\sigma H\_\{0\}\\leq\\varepsilon, setγ=1−ε/\(20​H0\)\\gamma=1\-\\varepsilon/\(20H\_\{0\}\)and run a nominal DMDP solver on\(𝒮,𝒜,r,P^0,γ\)\(\\mathcal\{S\},\\mathcal\{A\},r,\\widehat\{P\}^\{0\},\\gamma\)\. - •Otherwise, setγ=1−ε/\(3​Hσ\)\\gamma=1\-\\varepsilon/\(3H\_\{\\sigma\}\)and run a robust DMDP solver on\(𝒮,𝒜,r,𝒰​\(P^0\),γ\)\(\\mathcal\{S\},\\mathcal\{A\},r,\\mathcal\{U\}\(\\widehat\{P\}^\{0\}\),\\gamma\)\.
3. 3\.Return the resulting policyπ^\\widehat\{\\pi\}\.

This span\-informed procedure satisfies the following sample\-complexity guarantee\.

###### Theorem 2\(Span\-informed upper bound\)\.

There exist a sufficiently large universal constantC\>0C\>0and a sufficiently small universal constantcopt\>0c\_\{\\mathrm\{opt\}\}\>0such that the following holds\. Suppose thatℳ\\mathcal\{M\}is a robust AMDP satisfying Assumption[1](https://arxiv.org/html/2608.06545#Thmassumption1)\. Letε∈\(0,1\]\\varepsilon\\in\(0,1\]be the target accuracy and letδ∈\(0,1/2\]\\delta\\in\(0,1/2\]\. Assume thatεopt≤copt​ε\\varepsilon\_\{\\mathrm\{opt\}\}\\leq c\_\{\\mathrm\{opt\}\}\\varepsilonand thatπ^\\widehat\{\\pi\}is the output of Algorithm[1](https://arxiv.org/html/2608.06545#alg1)\. Suppose that either of the following conditions holds:

1. \(a\)When7​σ​H0≤ε7\\sigma H\_\{0\}\\leq\\varepsilon, N​S​A≥C​S​A​log⁡\(Hσ​S​A​Nε​δ\)⋅min⁡\{H0,Hσ\}ε2;NSA\\geq CSA\\log\\\!\\left\(\\frac\{H\_\{\\sigma\}SAN\}\{\\varepsilon\\delta\}\\right\)\\cdot\\frac\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\}\{\\varepsilon^\{2\}\};
2. \(b\)when7​σ​H0\>ε7\\sigma H\_\{0\}\>\\varepsilon, N​S​A≥C​S​A​log⁡\(Hσ​S​A​Nε​δ\)⋅min⁡\{H0,Hσ\}\+σ​Hσ2ε2\.NSA\\geq CSA\\log\\\!\\left\(\\frac\{H\_\{\\sigma\}SAN\}\{\\varepsilon\\delta\}\\right\)\\cdot\\frac\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\+\\sigma H\_\{\\sigma\}^\{2\}\}\{\\varepsilon^\{2\}\}\.

Then, with probability at least1−O​\(δ\)1\-O\(\\delta\), the returned policyπ^\\widehat\{\\pi\}satisfiesρ⋆,σ−ρπ^,σ≤ε\\rho^\{\\star,\\sigma\}\-\\rho^\{\\widehat\{\\pi\},\\sigma\}\\leq\\varepsilon\.

The two displayed conditions give the high\- and low\-tolerance rates, respectively\. Comparing Theorem[2](https://arxiv.org/html/2608.06545#Thmtheorem2)with the lower bound in Theorem[1](https://arxiv.org/html/2608.06545#Thmtheorem1)shows that Algorithm[1](https://arxiv.org/html/2608.06545#alg1)is minimax\-optimal, up to logarithmic factors\. Notably, the high\-tolerance regime does not necessarily favor the nominal reduction: whenHσ≤H0H\_\{\\sigma\}\\leq H\_\{0\}, the robust reduction yields the smaller span dependence and sample complexity\.

##### Variant when onlyHσH\_\{\\sigma\}is known\.

IfHσH\_\{\\sigma\}is known butH0H\_\{0\}is not, always taking the robust reduction gives the sample\-complexity upper boundO~​\(S​A​\(Hσ\+σ​Hσ2\)​ε−2\)\\widetilde\{O\}\\\!\\left\(SA\(H\_\{\\sigma\}\+\\sigma H\_\{\\sigma\}^\{2\}\)\\varepsilon^\{\-2\}\\right\)\. Comparing with theHσH\_\{\\sigma\}\-only lower bound, this rate is minimax\-optimal up to logarithmic factors\.

##### Comparison with a prior reduction framework\.

Rochet al\.\([2025](https://arxiv.org/html/2608.06545#bib.bib8)\)gave a robust\-only reduction\-based procedure with sample complexityO~​\(S​A​HRoch2​ε−2\)\\widetilde\{O\}\(SAH\_\{\\mathrm\{Roch\}\}^\{2\}\\varepsilon^\{\-2\}\), whereHRochH\_\{\\mathrm\{Roch\}\}is a robust span parameter different fromHσH\_\{\\sigma\}\. We compare them in Appendix[A\.4](https://arxiv.org/html/2608.06545#A1.SS4)and prove thatHσ≤HRochH\_\{\\sigma\}\\leq H\_\{\\mathrm\{Roch\}\}\. Consequently, sinceσ≤1\\sigma\\leq 1and all span parameters are at least11, our span\-informed rate uniformly matches or improves upon their result\.

##### Analysis overview\.

We first focus on the robust branch of Algorithm[1](https://arxiv.org/html/2608.06545#alg1)\. The robust reduction sets

γ=1−ε3​Hσ\.\\gamma=1\-\\frac\{\\varepsilon\}\{3H\_\{\\sigma\}\}\.At this discount factor, it is enough to find a policy satisfying

‖Vγ⋆,σ−Vγπ^,σ‖∞≤Hσ\.\\left\\\|V\_\{\\gamma\}^\{\\star,\\sigma\}\-V\_\{\\gamma\}^\{\\widehat\{\\pi\},\\sigma\}\\right\\\|\_\{\\infty\}\\leq H\_\{\\sigma\}\.A generic robust discounted\-MDP bound at this horizon and accuracy would give the suboptimal sample complexityO~​\(S​A​Hσ​ε−3\)\\widetilde\{O\}\(SAH\_\{\\sigma\}\\varepsilon^\{\-3\}\)\. Theorem[2](https://arxiv.org/html/2608.06545#Thmtheorem2)obtains the sharper rate through three refinements\.

The first refinement compares the discounted problem with a nominal Bellman supersolution\. We call a pair\(ρ¯,h¯\)\(\\bar\{\\rho\},\\bar\{h\}\)a*nominal anchor*if

ρ¯\+h¯​\(s\)≥maxa∈𝒜⁡\{r​\(s,a\)\+Ps,a0​h¯\},s∈𝒮\.\\bar\{\\rho\}\+\\bar\{h\}\(s\)\\geq\\max\_\{a\\in\\mathcal\{A\}\}\\left\\\{r\(s,a\)\+P\_\{s,a\}^\{0\}\\bar\{h\}\\right\\\},\\qquad s\\in\\mathcal\{S\}\.\(6\)We choose the nominal optimal pair\(ρ⋆,hP0⋆\)\(\\rho^\{\\star\},h\_\{P^\{0\}\}^\{\\star\}\)as the anchor whenH0<HσH\_\{0\}<H\_\{\\sigma\}and the trivial pair\(1,0\)\(1,0\)otherwise\. Using this anchor as a reference in the variance analysis yieldsmin⁡\{H0,Hσ\}\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}, rather thanHσH\_\{\\sigma\}alone, in the leading statistical term\.

The second refinement proves that the true and empirical discounted value functions used in the analysis have spans of orderHσH\_\{\\sigma\}\. Their fluctuations can therefore be controlled byHσH\_\{\\sigma\}, instead of the worst\-case discounted scale\(1−γ\)−1\(1\-\\gamma\)^\{\-1\}\.

The third refinement replaces the blanket sample\-size requirementN≳\(1−γ\)−2N\\gtrsim\(1\-\\gamma\)^\{\-2\}from generic robust discounted analyses with refined localized requirements\. Together, these refinements show that, up to logarithmic factors, achieving discounted errorHσH\_\{\\sigma\}requires

N≳min⁡\{H0,Hσ\}\+σ​Hσ2\(1−γ\)2​Hσ2\+11−γ\.N\\gtrsim\\frac\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\+\\sigma H\_\{\\sigma\}^\{2\}\}\{\(1\-\\gamma\)^\{2\}H\_\{\\sigma\}^\{2\}\}\+\\frac\{1\}\{1\-\\gamma\}\.Substituting1−γ≍ε/Hσ1\-\\gamma\\asymp\\varepsilon/H\_\{\\sigma\}gives

N≳min⁡\{H0,Hσ\}\+σ​Hσ2ε2\+Hσε,N\\gtrsim\\frac\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\+\\sigma H\_\{\\sigma\}^\{2\}\}\{\\varepsilon^\{2\}\}\+\\frac\{H\_\{\\sigma\}\}\{\\varepsilon\},where the last term is absorbed when the robust reduction is used\.

The nominal branch of Algorithm[1](https://arxiv.org/html/2608.06545#alg1)applies the same discounted plug\-in reduction with degenerate uncertainty sets\. We use an additional argument to transfer its nominal performance guarantee to a robust performance guarantee\. The complete discounted theorem and the analysis of Theorem[2](https://arxiv.org/html/2608.06545#Thmtheorem2)are given in Appendix[D](https://arxiv.org/html/2608.06545#A4)\.

### 4\.3The span\-agnostic reduction

WhenH0H\_\{0\}andHσH\_\{\\sigma\}are unknown, neither the reduction nor the discount factor in Algorithm[1](https://arxiv.org/html/2608.06545#alg1)can be selected directly\. This motivates a span\-agnostic procedure that adaptively selects the appropriate discount factor and chooses between the nominal and robust discounted reductions\.

Here we describe the framework of the span\-agnostic reduction and present the main theoretical result\. For conciseness, we defer the full details of the algorithm to Appendix[E\.1](https://arxiv.org/html/2608.06545#A5.SS1)\. The key idea of this procedure is to construct a set of candidate policies from both nominal and robust discounted reductions and then select the best one based on a lower\-confidence bound on its robust average reward\.

##### Two independent data batches\.

We use two batches so that the anchor certificate and the nominal policy candidates are independent of the data used to assess robust performance\. For each state\-action pair, split theNNtransition samples into a*nominal batch*𝒟nom\\mathcal\{D\}\_\{\\mathrm\{nom\}\}of sizeNnomN\_\{\\mathrm\{nom\}\}and a*robust batch*𝒟rob\\mathcal\{D\}\_\{\\mathrm\{rob\}\}of sizeNrobN\_\{\\mathrm\{rob\}\}, where both sizes are constant fractions ofNN\. LetP^nom0\\widehat\{P\}\_\{\\mathrm\{nom\}\}^\{0\}andP^rob0\\widehat\{P\}\_\{\\mathrm\{rob\}\}^\{0\}be the two empirical nominal kernels, constructed as in \([4](https://arxiv.org/html/2608.06545#S4.E4)\)\. The nominal batch is used to compute the anchor certificate and the nominal policy candidates\. The robust batch is then used to compute the robust policy candidates and to evaluate the nominal candidates in the empirical robust MDP\.

##### Anchor certificate\.

Recall from the span\-informed analysis in Section[4\.2](https://arxiv.org/html/2608.06545#S4.SS2)that, whenH0<HσH\_\{0\}<H\_\{\\sigma\}, the nominal optimal pair\(ρ⋆,hP0⋆\)\(\\rho^\{\\star\},h\_\{P^\{0\}\}^\{\\star\}\)provides an analytical anchor satisfying \([6](https://arxiv.org/html/2608.06545#S4.E6)\)\. Its bias spanH0H\_\{0\}yields theH0H\_\{0\}\-dependent part of the upper bound in Theorem[2](https://arxiv.org/html/2608.06545#Thmtheorem2)\. In the span\-agnostic setting, however,H0H\_\{0\}is unknown and therefore cannot be used directly to calibrate the confidence penalty\. We instead use the nominal batch to compute an anchor certificate\(ρ^anc\+,H^anc\+\)\(\\widehat\{\\rho\}\_\{\\mathrm\{anc\}\}^\{\+\},\\widehat\{H\}\_\{\\mathrm\{anc\}\}^\{\+\}\)\. With probability at least1−O​\(δ\)1\-O\(\\delta\), this certificate is associated with a nominal anchor\(ρ¯,h¯\)\(\\bar\{\\rho\},\\bar\{h\}\)satisfying \([6](https://arxiv.org/html/2608.06545#S4.E6)\) and

ρ¯≤ρ^anc\+,max⁡\{1,‖h¯‖span\}≤H^anc\+\.\\bar\{\\rho\}\\leq\\widehat\{\\rho\}\_\{\\mathrm\{anc\}\}^\{\+\},\\qquad\\max\\\{1,\\\|\\bar\{h\}\\\|\_\{\\mathrm\{span\}\}\\\}\\leq\\widehat\{H\}\_\{\\mathrm\{anc\}\}^\{\+\}\.Moreover, the certificate satisfiesH^anc\+=O​\(H0\)\\widehat\{H\}\_\{\\mathrm\{anc\}\}^\{\+\}=O\(H\_\{0\}\), so it permits the confidence penalty to retain theH0H\_\{0\}scale without requiringH0H\_\{0\}as an input\. For conciseness, we defer the exact algorithm and the lemma confirming these properties to Appendix[E\.1](https://arxiv.org/html/2608.06545#A5.SS1)\.

##### Candidate policies\.

To adapt to the unknown effective horizon, we consider a dyadic gridΓN\\Gamma\_\{N\}consisting of discount factorsγ=1−2−k\\gamma=1\-2^\{\-k\},k=1,2,…k=1,2,\\ldots, whose exact cutoff is sample\-size\-dependent and given in Appendix[E\.1](https://arxiv.org/html/2608.06545#A5.SS1)\. For eachγ∈ΓN\\gamma\\in\\Gamma\_\{N\}, we compute two candidate policies\. First, we solve the empirical nominal discounted MDP with kernelP^nom0\\widehat\{P\}\_\{\\mathrm\{nom\}\}^\{0\}to obtainπ^γ0\\widehat\{\\pi\}\_\{\\gamma\}^\{0\}\. Then, in the empirical robust discounted MDP centered atP^rob0\\widehat\{P\}\_\{\\mathrm\{rob\}\}^\{0\}, we evaluateπ^γ0\\widehat\{\\pi\}\_\{\\gamma\}^\{0\}to obtainV^γπ^γ0,σ\\widehat\{V\}\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\}^\{0\},\\sigma\}and solve the robust discounted problem to obtainπ^γ\\widehat\{\\pi\}\_\{\\gamma\}andV^γπ^γ,σ\\widehat\{V\}\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\},\\sigma\}\.

##### Policy selection by lower\-confidence bounds\.

For eachγ∈ΓN\\gamma\\in\\Gamma\_\{N\}, we assign each of the two candidate policies a lower\-confidence bound on its robust average reward\. The superscriptsnom\\mathrm\{nom\}androb\\mathrm\{rob\}distinguish the nominal and robust candidate families, respectively\. Letpenγnom⁡\(π^γ0\)\\operatorname\{pen\}\_\{\\gamma\}^\{\\mathrm\{nom\}\}\(\\widehat\{\\pi\}\_\{\\gamma\}^\{0\}\)denote the confidence penalty for the nominal candidate\. Its lower\-confidence bound is

LCBγnom⁡\(π^γ0\)≔\(1−γ\)​mins⁡V^γπ^γ0,σ​\(s\)−penγnom⁡\(π^γ0\)\.\\operatorname\{LCB\}\_\{\\gamma\}^\{\\mathrm\{nom\}\}\(\\widehat\{\\pi\}\_\{\\gamma\}^\{0\}\)\\coloneqq\(1\-\\gamma\)\\min\_\{s\}\\widehat\{V\}\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\}^\{0\},\\sigma\}\(s\)\-\\operatorname\{pen\}\_\{\\gamma\}^\{\\mathrm\{nom\}\}\(\\widehat\{\\pi\}\_\{\\gamma\}^\{0\}\)\.\(7\)For the robust candidate, the anchor certificate and the trivial anchor\(ρ,h\)=\(1,0\)\(\\rho,h\)=\(1,0\)give two valid penalties,penγanc⁡\(π^γ\)\\operatorname\{pen\}\_\{\\gamma\}^\{\\mathrm\{anc\}\}\(\\widehat\{\\pi\}\_\{\\gamma\}\)andpenγtriv⁡\(π^γ\)\\operatorname\{pen\}\_\{\\gamma\}^\{\\mathrm\{triv\}\}\(\\widehat\{\\pi\}\_\{\\gamma\}\), respectively\. Since both induce valid lower\-confidence bounds, we subtract the smaller penalty and define

LCBγrob⁡\(π^γ\)≔\(1−γ\)​mins⁡V^γπ^γ,σ​\(s\)−min⁡\{penγanc⁡\(π^γ\),penγtriv⁡\(π^γ\)\}\.\\operatorname\{LCB\}\_\{\\gamma\}^\{\\mathrm\{rob\}\}\(\\widehat\{\\pi\}\_\{\\gamma\}\)\\coloneqq\(1\-\\gamma\)\\min\_\{s\}\\widehat\{V\}\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\},\\sigma\}\(s\)\-\\min\\left\\\{\\operatorname\{pen\}\_\{\\gamma\}^\{\\mathrm\{anc\}\}\(\\widehat\{\\pi\}\_\{\\gamma\}\),\\operatorname\{pen\}\_\{\\gamma\}^\{\\mathrm\{triv\}\}\(\\widehat\{\\pi\}\_\{\\gamma\}\)\\right\\\}\.\(8\)The complete data\-dependent definitions of the penaltiespenγnom\\operatorname\{pen\}\_\{\\gamma\}^\{\\mathrm\{nom\}\},penγanc\\operatorname\{pen\}\_\{\\gamma\}^\{\\mathrm\{anc\}\}, andpenγtriv\\operatorname\{pen\}\_\{\\gamma\}^\{\\mathrm\{triv\}\}are given in Appendix[E\.1](https://arxiv.org/html/2608.06545#A5.SS1)\. With probability at least1−O​\(δ\)1\-O\(\\delta\), simultaneously for everyγ∈ΓN\\gamma\\in\\Gamma\_\{N\},

LCBγnom⁡\(π^γ0\)≤ρπ^γ0,σ,LCBγrob⁡\(π^γ\)≤ρπ^γ,σ\.\\operatorname\{LCB\}\_\{\\gamma\}^\{\\mathrm\{nom\}\}\(\\widehat\{\\pi\}\_\{\\gamma\}^\{0\}\)\\leq\\rho^\{\\widehat\{\\pi\}\_\{\\gamma\}^\{0\},\\sigma\},\\qquad\\operatorname\{LCB\}\_\{\\gamma\}^\{\\mathrm\{rob\}\}\(\\widehat\{\\pi\}\_\{\\gamma\}\)\\leq\\rho^\{\\widehat\{\\pi\}\_\{\\gamma\},\\sigma\}\.We therefore return the candidate policy with the largest lower\-confidence bound\.

Algorithm[2](https://arxiv.org/html/2608.06545#alg2)summarizes the procedure; the full implementable specification is deferred to Appendix[E\.1](https://arxiv.org/html/2608.06545#A5.SS1)\.

Algorithm 2Span\-agnostic reduction with nominal and robust candidatesInput\.For every\(s,a\)\(s,a\),NNindependent nominal transition samples;ε\\varepsilon,δ\\delta,σ\\sigma, and𝒰\\mathcal\{U\}\.

1. 1\.Prepare the empirical models\.Split the samples into𝒟nom\\mathcal\{D\}\_\{\\mathrm\{nom\}\}and𝒟rob\\mathcal\{D\}\_\{\\mathrm\{rob\}\}, formP^nom0\\widehat\{P\}\_\{\\mathrm\{nom\}\}^\{0\}andP^rob0\\widehat\{P\}\_\{\\mathrm\{rob\}\}^\{0\}, constructΓN\\Gamma\_\{N\}, and compute the anchor certificate from𝒟nom\\mathcal\{D\}\_\{\\mathrm\{nom\}\}\.
2. 2\.Construct the candidate policies\.For everyγ∈ΓN\\gamma\\in\\Gamma\_\{N\}, compute the nominal candidateπ^γ0\\widehat\{\\pi\}\_\{\\gamma\}^\{0\}fromP^nom0\\widehat\{P\}\_\{\\mathrm\{nom\}\}^\{0\}, evaluate it in the empirical robust MDP centered atP^rob0\\widehat\{P\}\_\{\\mathrm\{rob\}\}^\{0\}, and compute the robust candidateπ^γ\\widehat\{\\pi\}\_\{\\gamma\}in the same empirical robust MDP\.
3. 3\.Select a policy\.Compute the lower\-confidence bounds in \([7](https://arxiv.org/html/2608.06545#S4.E7)\) and \([8](https://arxiv.org/html/2608.06545#S4.E8)\), and return a policy with the largest lower\-confidence bound\.

The following theorem gives the sample\-complexity guarantee for the resulting span\-agnostic procedure\.

###### Theorem 3\(Span\-agnostic robust policy learning\)\.

There exist a sufficiently large universal constantC\>0C\>0and sufficiently small universal constantsc,copt\>0c,c\_\{\\mathrm\{opt\}\}\>0such that the following holds\. Letℳ\\mathcal\{M\}be a robust AMDP satisfying Assumption[1](https://arxiv.org/html/2608.06545#Thmassumption1), and fixε∈\(0,1\]\\varepsilon\\in\(0,1\],δ∈\(0,1\)\\delta\\in\(0,1\), andN≥16N\\geq 16\. Letπ^\\widehat\{\\pi\}be the output of Algorithm[2](https://arxiv.org/html/2608.06545#alg2), and assumeεopt≤copt​ε\\varepsilon\_\{\\mathrm\{opt\}\}\\leq c\_\{\\mathrm\{opt\}\}\\varepsilon\. Suppose that either of the following conditions holds:

1. \(a\)Whenσ​H0≤c​ε\\sigma H\_\{0\}\\leq c\\varepsilon, N​S​A≥C​S​A​log⁡\(S​A​Nδ\)⋅min⁡\{H0,Hσ\}ε2;NSA\\geq CSA\\log\\\!\\left\(\\frac\{SAN\}\{\\delta\}\\right\)\\cdot\\frac\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\}\{\\varepsilon^\{2\}\};
2. \(b\)whenσ​H0\>c​ε\\sigma H\_\{0\}\>c\\varepsilon, N​S​A≥C​S​A​log⁡\(S​A​Nδ\)⋅min⁡\{H0,Hσ\}\+σ​Hσ2ε2\.NSA\\geq CSA\\log\\\!\\left\(\\frac\{SAN\}\{\\delta\}\\right\)\\cdot\\frac\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\+\\sigma H\_\{\\sigma\}^\{2\}\}\{\\varepsilon^\{2\}\}\.

Then, with probability at least1−O​\(δ\)1\-O\(\\delta\), the returned policyπ^\\widehat\{\\pi\}satisfies

ρ⋆,σ−ρπ^,σ≤ε\.\\rho^\{\\star,\\sigma\}\-\\rho^\{\\widehat\{\\pi\},\\sigma\}\\leq\\varepsilon\.

This guarantee is adaptive: Algorithm[2](https://arxiv.org/html/2608.06545#alg2)uses neitherH0H\_\{0\}norHσH\_\{\\sigma\}and is not told which regime holds\. By maximizing its lower\-confidence bounds over the discount\-factor grid and both candidate families, it automatically uses the data to select both the discount factor and whether to use the nominal or robust reduction\. Comparing with Theorem[1](https://arxiv.org/html/2608.06545#Thmtheorem1), Theorem[3](https://arxiv.org/html/2608.06545#Thmtheorem3)shows that Algorithm[2](https://arxiv.org/html/2608.06545#alg2)is minimax\-optimal across all regimes up to logarithmic factors:

- •In the high\-tolerance regime, theH0H\_\{0\}rate is certified by a nominal candidate whenH0<HσH\_\{0\}<H\_\{\\sigma\}, whereas theHσH\_\{\\sigma\}rate is certified by a robust candidate whenHσ≤H0H\_\{\\sigma\}\\leq H\_\{0\}\.
- •In the low\-tolerance regime, the robust candidate gives sample complexity O~​\(S​A​min⁡\{H0,Hσ\}\+σ​Hσ2ε2\)\.\\widetilde\{O\}\\\!\\left\(SA\\frac\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\+\\sigma H\_\{\\sigma\}^\{2\}\}\{\\varepsilon^\{2\}\}\\right\)\.The minimum in the leading span term comes from the two anchor choices: the calibrated nominal anchor suppliesH0H\_\{0\}whenH0<HσH\_\{0\}<H\_\{\\sigma\}, while the trivial anchor suppliesHσH\_\{\\sigma\}whenHσ≤H0H\_\{\\sigma\}\\leq H\_\{0\}\.

Both rates match the minimax lower bound in Theorem[1](https://arxiv.org/html/2608.06545#Thmtheorem1)up to logarithmic factors\.

##### Robust\-only variant\.

A simpler variant uses only the robust candidates and the trivial\-anchor penalty\. Its uniform sample complexity isO~​\(S​A​\(Hσ\+σ​Hσ2\)​ε−2\)\\widetilde\{O\}\\\!\\left\(SA\(H\_\{\\sigma\}\+\\sigma H\_\{\\sigma\}^\{2\}\)\\varepsilon^\{\-2\}\\right\)\. This robust\-only variant attains theHσH\_\{\\sigma\}\-only minimax rate, while the full span\-agnostic procedure can additionally exploit a smallerH0H\_\{0\}whenH0<HσH\_\{0\}<H\_\{\\sigma\}\.

##### Comparison with Roch et al\.

Rochet al\.\([2026](https://arxiv.org/html/2608.06545#bib.bib9)\)propose a model\-free, span\-agnostic approach with sample complexityO~​\(S​A​HRHI2​ε−2\)\\widetilde\{O\}\(SAH\_\{\\mathrm\{RHI\}\}^\{2\}\\varepsilon^\{\-2\}\), whereHRHIH\_\{\\mathrm\{RHI\}\}is a robust span parameter satisfyingHRHI≥HσH\_\{\\mathrm\{RHI\}\}\\geq H\_\{\\sigma\}\. See Appendix A\.4 for a detailed comparison\. Our sample\-complexity guarantee uniformly matches or improves upon theirs\.

##### Analysis overview\.

If we knewH0H\_\{0\},HσH\_\{\\sigma\}, and which reduction is more sample\-efficient, we could choose a single discount factor and use that reduction to obtain anO​\(ε\)O\(\\varepsilon\)\-optimal policy\. Algorithm[2](https://arxiv.org/html/2608.06545#alg2)does not know these quantities, but its dyadic grid contains a discount factor whose effective horizon is within a factor of two of the ideal horizon\. At this grid point, one of the two candidate policies isO​\(ε\)O\(\\varepsilon\)\-optimal, and its lower\-confidence bound is at leastρ⋆,σ−O​\(ε\)\\rho^\{\\star,\\sigma\}\-O\(\\varepsilon\)\. Because all the lower\-confidence bounds are valid and the algorithm selects the largest one,

ρπ^,σ\\displaystyle\\rho^\{\\widehat\{\\pi\},\\sigma\}≥maxγ∈ΓN⁡\{LCBγnom⁡\(π^γ0\),LCBγrob⁡\(π^γ\)\}≥ρ⋆,σ−O​\(ε\)\.\\displaystyle\\geq\\max\_\{\\gamma\\in\\Gamma\_\{N\}\}\\left\\\{\\operatorname\{LCB\}\_\{\\gamma\}^\{\\mathrm\{nom\}\}\(\\widehat\{\\pi\}\_\{\\gamma\}^\{0\}\),\\operatorname\{LCB\}\_\{\\gamma\}^\{\\mathrm\{rob\}\}\(\\widehat\{\\pi\}\_\{\\gamma\}\)\\right\\\}\\geq\\rho^\{\\star,\\sigma\}\-O\(\\varepsilon\)\.The complete proof is given in Appendix[E](https://arxiv.org/html/2608.06545#A5)\.

![Refer to caption](https://arxiv.org/html/2608.06545v1/x1.png)Figure 1:High\- and low\-tolerance sample\-complexity checks\. All four panels reportN95N\_\{95\}\. \(a\) High tolerance, withH0<HσH\_\{0\}<H\_\{\\sigma\}:N95N\_\{95\}againstH0H\_\{0\}\. \(b\) Low tolerance, withH0\>HσH\_\{0\}\>H\_\{\\sigma\}: the minimum\-span component\. \(c\)–\(d\) Low tolerance: dependence onσ\\sigmaandHσ2H\_\{\\sigma\}^\{2\}in the robustness\-specific component\. Bars are95%95\\%bootstrap intervals forN95N\_\{95\}\. Panels \(a\), \(b\), and \(d\) use log\-log axes; the dashed lines have the indicated slopes\. The toleranceε\\varepsilonis held fixed within each panel\.![Refer to caption](https://arxiv.org/html/2608.06545v1/x2.png)Figure 2:Span\-agnostic adaptation\. \(a\) Policy\-family selection across the transition from the nominal to the robust family\. \(b\) Selected effective horizon\(1−γ\)−1\(1\-\\gamma\)^\{\-1\}at the nominal\-family and robust\-family endpoints, compared withH0/εH\_\{0\}/\\varepsilonandHσ/εH\_\{\\sigma\}/\\varepsilon, respectively; the shaded region is the factor\-of\-two band\. \(c\) Robust performance of the selected and nominal policies\. \(d\) Comparison of span\-agnostic and span\-informedN95N\_\{95\}\.

## 5Experiments

We use controlled simulations to evaluate two parts of the theory\. We first verify the sample\-complexity rates in Theorem[2](https://arxiv.org/html/2608.06545#Thmtheorem2), including the uniform linear dependence onmin⁡\{H0,Hσ\}\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}and the robustness\-specific dependence onσ​Hσ2\\sigma H\_\{\\sigma\}^\{2\}in the low\-tolerance regime\. We then showcase the adaptivity of the span\-agnostic approach in Algorithm[2](https://arxiv.org/html/2608.06545#alg2)\.

The experiments use finite AMDPs with various parameters\. In each setting, the learner receivesNNnominal next\-state samples per state\-action pair\. We summarize sample cost byN95N\_\{95\}, the sample size at which the estimated probability of returning a policy that isε\\varepsilon\-optimal in robust average reward reaches0\.950\.95\. Because success should increase withNN, we estimateN95N\_\{95\}by fitting a nondecreasing success curve and interpolating its0\.950\.95crossing\. Appendix[F](https://arxiv.org/html/2608.06545#A6)gives further experimental details and additional figures\.

### 5\.1Empirical sample\-complexity rates of Algorithm[1](https://arxiv.org/html/2608.06545#alg1)

For per\-state\-action sample complexity, Theorem[2](https://arxiv.org/html/2608.06545#Thmtheorem2)predicts the high\-tolerance rate

min⁡\{H0,Hσ\}ε2\\frac\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\}\{\\varepsilon^\{2\}\}and the low\-tolerance rate

min⁡\{H0,Hσ\}\+σ​Hσ2ε2\.\\frac\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\+\\sigma H\_\{\\sigma\}^\{2\}\}\{\\varepsilon^\{2\}\}\.
We first test the high\-tolerance branch with7​σ​H0≤ε7\\sigma H\_\{0\}\\leq\\varepsilonin the orderingH0<HσH\_\{0\}<H\_\{\\sigma\}\. We varyH0H\_\{0\}while holdingε\\varepsilonandσ\\sigmafixed\. The construction givesHσ=H0/\(1−σ​H0\)\>H0H\_\{\\sigma\}=H\_\{0\}/\(1\-\\sigma H\_\{0\}\)\>H\_\{0\}, somin⁡\{H0,Hσ\}=H0\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}=H\_\{0\}\. Figure[1a](https://arxiv.org/html/2608.06545#S4.F1)shows the predicted linear increase ofN95N\_\{95\}withH0H\_\{0\}\.

For the low\-tolerance branch, we isolate its two rate components whenHσ≤H0H\_\{\\sigma\}\\leq H\_\{0\}\. Figure[1b](https://arxiv.org/html/2608.06545#S4.F1)variesHσH\_\{\\sigma\}below a fixedH0H\_\{0\}while keepingσ​Hσ2\\sigma H\_\{\\sigma\}^\{2\}small relative to the minimum\-span component\. It shows the expected linear dependence onHσH\_\{\\sigma\}\. The complementary orderingH0<HσH\_\{0\}<H\_\{\\sigma\}yields a similar result and is reported in Appendix[5](https://arxiv.org/html/2608.06545#A6.F5)\. Finally, Figures[1c](https://arxiv.org/html/2608.06545#S4.F1)and[1d](https://arxiv.org/html/2608.06545#S4.F1)isolate the robustness\-specific term\. The first variesσ\\sigmaat fixedHσH\_\{\\sigma\}and the second variesHσH\_\{\\sigma\}at fixedσ\\sigma; the resulting linear and quadratic trends are consistent with theσ​Hσ2\\sigma H\_\{\\sigma\}^\{2\}component in the low\-tolerance rate\.

Together, Figure[1](https://arxiv.org/html/2608.06545#S4.F1)shows that the sample\-complexity rates in Theorem[2](https://arxiv.org/html/2608.06545#Thmtheorem2)are consistent with the empirical behavior of Algorithm[1](https://arxiv.org/html/2608.06545#alg1)\.

### 5\.2Span\-agnostic adaptation of Algorithm[2](https://arxiv.org/html/2608.06545#alg2)

We next evaluate the adaptive behavior of Algorithm[2](https://arxiv.org/html/2608.06545#alg2)\. We test whether the span\-agnostic algorithm can choose between the nominal and robust policy families and select the discounted horizon without knowingH0H\_\{0\}orHσH\_\{\\sigma\}\. We vary the regime ratioσ​H0/ε\\sigma H\_\{0\}/\\varepsilonacross the nominal\-to\-robust transition and repeat the experiment over severalH0H\_\{0\}settings\.

Figures[2a](https://arxiv.org/html/2608.06545#S4.F2)and[2b](https://arxiv.org/html/2608.06545#S4.F2)make this adaptation visible\. As the regime ratio crosses the transition, Figure[2a](https://arxiv.org/html/2608.06545#S4.F2)shows that the algorithm shifts from the nominal family to the robust family\. The thin gray curves show the robust\-family selection probability separately for eachH0H\_\{0\}setting, while the blue curve pools trials across settings; their close agreement shows that the transition is driven by the regime ratio rather than by a particularH0H\_\{0\}setting\. Figure[2b](https://arxiv.org/html/2608.06545#S4.F2)examines horizon selection at the two endpoints of this transition\. Atσ​H0/ε=0\.2\\sigma H\_\{0\}/\\varepsilon=0\.2, the algorithm selects the nominal family, and we compare its selected horizon withH0/εH\_\{0\}/\\varepsilon\. Atσ​H0/ε=2\\sigma H\_\{0\}/\\varepsilon=2, it selects the robust family, and we compare its selected horizon withHσ/εH\_\{\\sigma\}/\\varepsilon\. In both cases, the selected effective horizon\(1−γ\)−1\(1\-\\gamma\)^\{\-1\}remains within a factor of two of the reference scale for the corresponding policy family\.

Figures[2c](https://arxiv.org/html/2608.06545#S4.F2)and[2d](https://arxiv.org/html/2608.06545#S4.F2)connect these adaptive choices to performance\. Figure[2c](https://arxiv.org/html/2608.06545#S4.F2)shows that the adaptively selected policies continue to meet the target robust accuracy as uncertainty grows, while the nominal policy deteriorates\. Figure[2d](https://arxiv.org/html/2608.06545#S4.F2)further illustrates that the span\-agnostic sample complexity thresholds remain within constant factors of the span\-informed benchmark\. Together, Figure[2](https://arxiv.org/html/2608.06545#S4.F2)shows that Algorithm[2](https://arxiv.org/html/2608.06545#alg2)adapts both its policy family and effective horizon without span information, while maintaining robust accuracy and sample complexity within constant factors of the span\-informed benchmark\.

## 6Discussion

In this paper, we develop a minimax theory for learning distributionally robust average\-reward MDPs from a generative model\. We provide span\-informed and span\-agnostic reduction\-based procedures that achieve minimax\-optimal sample complexity\. We identify the scaleσ​H0\\sigma H\_\{0\}that separates the high\- and low\-tolerance regimes\. In the high\-tolerance regime, the optimal sample complexity is determined byS​A​min⁡\{H0,Hσ\}​ε−2SA\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\\varepsilon^\{\-2\}, while an additional termS​A​σ​Hσ2​ε−2SA\\sigma H\_\{\\sigma\}^\{2\}\\varepsilon^\{\-2\}appears in the low\-tolerance regime\. We supplement our theoretical results with numerical experiments that support our findings\.

While this work provides a tight characterization of robust AMDPs in the generative model setting, several avenues remain open for future work\. We assume\(s,a\)\(s,a\)\-rectangular TV uncertainty sets\. Exploring more coupled uncertainty structures or other divergences could expand our understanding of the cost of distributional robustness\. We also focus on the generative\-model setting, where the agent has access to a simulator to obtain samples for each state\-action pair\. A natural next step is to investigate regret bounds or sample complexity in the online setting, where the agent must explore the environment without a simulator\.

## Acknowledgments

The work of Y\. Yang and Y\. Chi is supported in part by NSF under ECCS\-2537078, ECCS\-2537189, and CNS\-2148212\.

Y\. Chen is supported in part by the Alfred P\. Sloan Research Fellowship, the NSF grants IIS\-2218773 and CIF\-2221009, the ONR grant N00014\-25\-1\-2344, the AFOSR grant FA9550261B178, and the Wharton AI & Analytics Initiative’s AI Research Fund\. This work is also supported in part by the NSF under Cooperative Agreement No\. 2433450\.

## References

- P\. L\. Bartlett and A\. Tewari \(2009\)REGAL: a regularization based algorithm for reinforcement learning in weakly communicating MDPs\.InProceedings of the Twenty\-Fifth Conference on Uncertainty in Artificial Intelligence,pp\. 35–42\.External Links:[Link](https://arxiv.org/abs/1205.2661)Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px1.p1.1)\.
- Z\. Chen, S\. Wang, and N\. Si \(2025\)Sample complexity of distributionally robust average\-reward reinforcement learning\.Advances in Neural Information Processing Systems38\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2025/hash/7b75da9b61eda40fa35453ee5d077df6-Abstract-Conference.html)Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px3.p1.1)\.
- P\. Clavier, L\. Shi, E\. Le Pennec, E\. Mazumdar, A\. Wierman, and M\. Geist \(2024\)Near\-optimal distributionally robust reinforcement learning with generalLpL\_\{p\}norms\.Advances in Neural Information Processing Systems37,pp\. 1750–1810\.Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px2.p1.1)\.
- J\. C\. Duchi and H\. Namkoong \(2021\)Learning models with uniform performance via distributionally robust optimization\.The Annals of Statistics49\(3\),pp\. 1378–1406\.Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2608.06545#S1.p2.1)\.
- R\. Fruit, M\. Pirotta, A\. Lazaric, and R\. Ortner \(2018\)Efficient bias\-span\-constrained exploration\-exploitation in reinforcement learning\.InProceedings of the 35th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.80,pp\. 1578–1586\.External Links:[Link](https://proceedings.mlr.press/v80/fruit18a.html)Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px1.p1.1)\.
- J\. Goh and M\. Sim \(2010\)Distributionally robust optimization and its tractable approximations\.Operations Research58\(4\-part\-1\),pp\. 902–917\.Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2608.06545#S1.p2.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025\)DeepSeek\-R1 incentivizes reasoning in LLMs through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.Cited by:[§1](https://arxiv.org/html/2608.06545#S1.p1.9)\.
- C\. P\. Ho, M\. Petrik, and W\. Wiesemann \(2018\)Fast Bellman updates for robust MDPs\.InInternational Conference on Machine Learning,pp\. 1979–1988\.Cited by:[§4\.2](https://arxiv.org/html/2608.06545#S4.SS2.p1.10)\.
- G\. N\. Iyengar \(2005\)Robust dynamic programming\.Mathematics of Operations Research30\(2\),pp\. 257–280\.Cited by:[§A\.2](https://arxiv.org/html/2608.06545#A1.SS2.1.p1.2),[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2608.06545#S1.p2.1),[§4\.2](https://arxiv.org/html/2608.06545#S4.SS2.p1.10)\.
- T\. Jaksch, R\. Ortner, and P\. Auer \(2010\)Near\-optimal regret bounds for reinforcement learning\.Journal of Machine Learning Research11\(51\),pp\. 1563–1600\.External Links:[Link](https://www.jmlr.org/papers/v11/jaksch10a.html)Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px1.p1.1)\.
- Y\. Jiao, J\. Woo, G\. Li, G\. Joshi, and Y\. Chi \(2026\)Sample complexity of average\-reward Q\-learning: from single\-agent to federated reinforcement learning\.arXiv preprint arXiv:2601\.13642\.External Links:[Link](https://arxiv.org/abs/2601.13642)Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px1.p1.1)\.
- Y\. Jin, R\. Gummadi, Z\. Zhou, and J\. Blanchet \(2024\)Feasible Q\-learning for average reward reinforcement learning\.InInternational Conference on Artificial Intelligence and Statistics,pp\. 1630–1638\.Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px1.p1.1)\.
- Y\. Jin and A\. Sidford \(2020\)Efficiently solving MDPs with stochastic mirror descent\.InInternational Conference on Machine Learning,pp\. 4890–4900\.Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px1.p1.1)\.
- Y\. Jin and A\. Sidford \(2021\)Towards tight bounds on the sample complexity of average\-reward MDPs\.InInternational Conference on Machine Learning,pp\. 5055–5064\.Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px1.p1.1)\.
- J\. Kober, J\. A\. Bagnell, and J\. Peters \(2013\)Reinforcement learning in robotics: a survey\.The International Journal of Robotics Research32\(11\),pp\. 1238–1274\.Cited by:[§1](https://arxiv.org/html/2608.06545#S1.p1.9)\.
- S\. Kumar, H\. Marklund, A\. Rao, Y\. Zhu, H\. J\. Jeon, Y\. Liu, and B\. Van Roy \(2025\)Continual learning as computationally constrained reinforcement learning\.Foundations and Trends in Machine Learning18\(5\),pp\. 913–1053\.External Links:[Document](https://dx.doi.org/10.1561/2200000116)Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px1.p1.1)\.
- J\. Lee, M\. Bravo, and R\. Cominetti \(2025\)Near\-optimal sample complexity for MDPs via anchoring\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 32907–32929\.External Links:[Link](https://proceedings.mlr.press/v267/lee25c.html)Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px1.p1.1)\.
- G\. Li, Y\. Wei, Y\. Chi, and Y\. Chen \(2024\)Breaking the sample size barrier in model\-based reinforcement learning with a generative model\.Operations Research72\(1\),pp\. 203–221\.Cited by:[§4\.1](https://arxiv.org/html/2608.06545#S4.SS1.p2.5)\.
- V\. Mnih, K\. Kavukcuoglu, D\. Silver, A\. A\. Rusu, J\. Veness, M\. G\. Bellemare, A\. Graves, M\. Riedmiller, A\. K\. Fidjeland, G\. Ostrovski, S\. Petersen, C\. Beattie, A\. Sadik, I\. Antonoglou, H\. King, D\. Kumaran, D\. Wierstra, S\. Legg, and D\. Hassabis \(2015\)Human\-level control through deep reinforcement learning\.Nature518\(7540\),pp\. 529–533\.Cited by:[§1](https://arxiv.org/html/2608.06545#S1.p1.9)\.
- P\. Mohajerin Esfahani and D\. Kuhn \(2018\)Data\-driven distributionally robust optimization using the Wasserstein metric: performance guarantees and tractable reformulations\.Mathematical Programming171\(1\),pp\. 115–166\.Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2608.06545#S1.p2.1)\.
- A\. Naik, R\. Shariff, N\. Yasui, H\. Yao, and R\. S\. Sutton \(2019\)Discounted reinforcement learning is not an optimization problem\.arXiv preprint arXiv:1910\.02140\.Cited by:[§1](https://arxiv.org/html/2608.06545#S1.p1.9)\.
- A\. Nilim and L\. El Ghaoui \(2005\)Robust control of Markov decision processes with uncertain transition matrices\.Operations Research53\(5\),pp\. 780–798\.Cited by:[§A\.2](https://arxiv.org/html/2608.06545#A1.SS2.1.p1.2),[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2608.06545#S1.p2.1),[§4\.2](https://arxiv.org/html/2608.06545#S4.SS2.p1.10)\.
- K\. Panaganti and D\. Kalathil \(2022\)Sample complexity of robust reinforcement learning with a generative model\.InInternational Conference on Artificial Intelligence and Statistics,pp\. 9582–9602\.Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px2.p1.1),[Lemma21](https://arxiv.org/html/2608.06545#Thmlemma21)\.
- K\. Panaganti, Z\. Xu, D\. Kalathil, and M\. Ghavamzadeh \(2022\)Robust reinforcement learning using offline data\.Advances in Neural Information Processing Systems35,pp\. 32211–32224\.Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px2.p1.1)\.
- M\. L\. Puterman \(1994\)Markov decision processes: discrete stochastic dynamic programming\.Wiley Series in Probability and Statistics,John Wiley & Sons,New York\.External Links:ISBN 978\-0\-471\-61977\-2,[Document](https://dx.doi.org/10.1002/9780470316887)Cited by:[§A\.1](https://arxiv.org/html/2608.06545#A1.SS1.p1.5),[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px1.p1.1)\.
- S\. S\. Ramesh, P\. G\. Sessa, Y\. Hu, A\. Krause, and I\. Bogunovic \(2024\)Distributionally robust model\-based reinforcement learning with large state spaces\.InInternational Conference on Artificial Intelligence and Statistics,pp\. 100–108\.Cited by:[§1](https://arxiv.org/html/2608.06545#S1.p2.1)\.
- Z\. A\. Roch, G\. K\. Atia, and Y\. Wang \(2025\)A reduction framework for distributionally robust reinforcement learning under average reward\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 51809–51850\.External Links:[Link](https://proceedings.mlr.press/v267/roch25a.html)Cited by:[§A\.4](https://arxiv.org/html/2608.06545#A1.SS4.SSS0.Px1.p1.1),[§A\.4](https://arxiv.org/html/2608.06545#A1.SS4.p1.4),[§1\.1](https://arxiv.org/html/2608.06545#S1.SS1.SSS0.Px3.p1.4),[Table 2](https://arxiv.org/html/2608.06545#S1.T2.9.3.4.1.1),[§1](https://arxiv.org/html/2608.06545#S1.p5.4),[§4\.2](https://arxiv.org/html/2608.06545#S4.SS2.SSS0.Px2.p1.6),[Remark1](https://arxiv.org/html/2608.06545#Thmremark1.p1.2.2),[footnote 1](https://arxiv.org/html/2608.06545#footnote1)\.
- Z\. Roch, G\. Atia, and Y\. Wang \(2026\)Model\-free robust average\-reward reinforcement learning with sample complexity analysis\.arXiv preprint arXiv:2505\.12462\.Note:Accepted at ICML 2026External Links:[Link](https://arxiv.org/abs/2505.12462)Cited by:[§A\.4](https://arxiv.org/html/2608.06545#A1.SS4.SSS0.Px1.p2.1),[§A\.4](https://arxiv.org/html/2608.06545#A1.SS4.p1.4),[§1\.1](https://arxiv.org/html/2608.06545#S1.SS1.SSS0.Px3.p1.4),[Table 2](https://arxiv.org/html/2608.06545#S1.T2.10.4.4.1.1),[§1](https://arxiv.org/html/2608.06545#S1.p5.4),[§4\.3](https://arxiv.org/html/2608.06545#S4.SS3.SSS0.Px6.p1.3),[Remark1](https://arxiv.org/html/2608.06545#Thmremark1.p1.2.2),[footnote 1](https://arxiv.org/html/2608.06545#footnote1)\.
- L\. Shi and Y\. Chi \(2024\)Distributionally robust model\-based offline reinforcement learning with near\-optimal sample complexity\.Journal of Machine Learning Research25\(200\),pp\. 1–91\.Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px2.p1.1)\.
- L\. Shi, G\. Li, Y\. Wei, Y\. Chen, M\. Geist, and Y\. Chi \(2026\)The curious price of distributional robustness in reinforcement learning with a generative model\.Operations Research\.Note:Articles in AdvanceExternal Links:[Document](https://dx.doi.org/10.1287/opre.2025.2240)Cited by:[§D\.5\.11](https://arxiv.org/html/2608.06545#A4.SS5.SSS11.Px1.p1.6),[§D\.5\.11](https://arxiv.org/html/2608.06545#A4.SS5.SSS11.Px2.p1.4),[§D\.5\.3](https://arxiv.org/html/2608.06545#A4.SS5.SSS3.Px1.p2.1),[§D\.5\.5](https://arxiv.org/html/2608.06545#A4.SS5.SSS5.Px1.p1.4),[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px2.p1.1),[§2\.1](https://arxiv.org/html/2608.06545#S2.SS1.SSS0.Px2.p1.17),[§3](https://arxiv.org/html/2608.06545#S3.p4.2),[Remark2](https://arxiv.org/html/2608.06545#Thmremark2.p1.3.3)\.
- D\. Silver, A\. Huang, C\. J\. Maddison, A\. Guez, L\. Sifre, G\. Van Den Driessche, J\. Schrittwieser, I\. Antonoglou, V\. Panneershelvam, M\. Lanctot,et al\.\(2016\)Mastering the game of Go with deep neural networks and tree search\.Nature529\(7587\),pp\. 484–489\.Cited by:[§1](https://arxiv.org/html/2608.06545#S1.p1.9)\.
- A\. Sinha, M\. O’Kelly, H\. Zheng, R\. Mangharam, J\. Duchi, and R\. Tedrake \(2020\)FormulaZero: distributionally robust online adaptation via offline population synthesis\.InInternational Conference on Machine Learning,pp\. 8992–9004\.Cited by:[§1](https://arxiv.org/html/2608.06545#S1.p2.1)\.
- R\. S\. Sutton and A\. G\. Barto \(2018\)Reinforcement learning: an introduction\.2nd edition,MIT Press\.Cited by:[§1](https://arxiv.org/html/2608.06545#S1.p1.9)\.
- M\. J\. Wainwright \(2019\)High\-dimensional statistics: a non\-asymptotic viewpoint\.Cambridge Series in Statistical and Probabilistic Mathematics,Cambridge University Press\.External Links:[Document](https://dx.doi.org/10.1017/9781108627771)Cited by:[§C\.2\.1](https://arxiv.org/html/2608.06545#A3.SS2.SSS1.Px2.p1.4)\.
- H\. Wang, L\. Shi, and Y\. Chi \(2024a\)Sample complexity of offline distributionally robust linear Markov decision processes\.Reinforcement Learning Journal3,pp\. 1467–1510\.External Links:[Link](https://rlj.cs.umass.edu/2024/papers/Paper189.html)Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px2.p1.1)\.
- J\. Wang, M\. Wang, and L\. F\. Yang \(2022\)Near sample\-optimal reduction\-based policy learning for average reward MDP\.arXiv preprint arXiv:2212\.00603\.External Links:[Link](https://arxiv.org/abs/2212.00603)Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2608.06545#S1.p4.2)\.
- S\. Wang, N\. Si, J\. Blanchet, and Z\. Zhou \(2024b\)Sample complexity of variance\-reduced distributionally robust Q\-learning\.Journal of Machine Learning Research25\(341\),pp\. 1–77\.Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px2.p1.1)\.
- S\. Wang and N\. Si \(2026\)Non\-rectangular average\-reward robust MDPs: optimal policies and their transient values\.arXiv preprint arXiv:2603\.00945\.External Links:[Link](https://arxiv.org/abs/2603.00945)Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px3.p1.1)\.
- Y\. Wang, A\. Velasquez, G\. K\. Atia, A\. Prater\-Bennette, and S\. Zou \(2023a\)Model\-free robust average\-reward reinforcement learning\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 36431–36469\.External Links:[Link](https://proceedings.mlr.press/v202/wang23am.html)Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2608.06545#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.06545#S2.SS2.p2.9)\.
- Y\. Wang, A\. Velasquez, G\. Atia, A\. Prater\-Bennette, and S\. Zou \(2023b\)Robust average\-reward Markov decision processes\.Proceedings of the AAAI Conference on Artificial Intelligence37\(12\),pp\. 15215–15223\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v37i12.26775)Cited by:[§A\.2](https://arxiv.org/html/2608.06545#A1.SS2.4.p1.7),[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2608.06545#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.06545#S2.SS2.p2.9)\.
- C\. Wei, M\. J\. Jahromi, H\. Luo, and R\. Jain \(2021\)Learning infinite\-horizon average\-reward MDPs with linear function approximation\.InProceedings of The 24th International Conference on Artificial Intelligence and Statistics,Vol\.130,pp\. 3007–3015\.Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px1.p1.1)\.
- C\. Wei, M\. J\. Jahromi, H\. Luo, H\. Sharma, and R\. Jain \(2020\)Model\-free reinforcement learning in infinite\-horizon average\-reward Markov decision processes\.InProceedings of the 37th International Conference on Machine Learning,Vol\.119,pp\. 10170–10180\.Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px1.p1.1)\.
- W\. Wiesemann, D\. Kuhn, and B\. Rustem \(2013\)Robust Markov decision processes\.Mathematics of Operations Research38\(1\),pp\. 153–183\.Cited by:[§1](https://arxiv.org/html/2608.06545#S1.p2.1)\.
- W\. Wiesemann, D\. Kuhn, and M\. Sim \(2014\)Distributionally robust convex optimization\.Operations Research62\(6\),pp\. 1358–1376\.Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2608.06545#S1.p2.1)\.
- H\. Xu and S\. Mannor \(2012\)Distributionally robust Markov decision processes\.Mathematics of Operations Research37\(2\),pp\. 288–300\.Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2608.06545#S1.p2.1)\.
- Y\. Xu, S\. Ganesh, and V\. Aggarwal \(2026\)Efficient Q\-learning and actor\-critic methods for robust average\-reward reinforcement learning\.arXiv preprint arXiv:2506\.07040\.Note:Accepted at UAI 2026External Links:[Link](https://arxiv.org/abs/2506.07040)Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px3.p1.1)\.
- W\. Yang, L\. Zhang, and Z\. Zhang \(2022\)Toward theoretical understandings of robust Markov decision processes: sample complexity and asymptotics\.The Annals of Statistics50\(6\),pp\. 3223–3248\.External Links:[Document](https://dx.doi.org/10.1214/22-AOS2225)Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px2.p1.1)\.
- Z\. Zhang and Q\. Xie \(2023\)Sharper model\-free reinforcement learning for average\-reward Markov decision processes\.InThe Thirty Sixth Annual Conference on Learning Theory,pp\. 5476–5477\.Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px1.p1.1)\.
- M\. Zurek and Y\. Chen \(2024\)Span\-based optimal sample complexity for weakly communicating and general average reward MDPs\.Advances in Neural Information Processing Systems37,pp\. 33455–33504\.Cited by:[§1\.2](https://arxiv.org/html/2608.06545#S1.SS2.SSS0.Px1.p1.1),[Table 2](https://arxiv.org/html/2608.06545#S1.T2.7.1.4.1.1.1),[§1](https://arxiv.org/html/2608.06545#S1.p4.2)\.
- M\. Zurek and Y\. Chen \(2025\)Span\-agnostic optimal sample complexity and oracle inequalities for average\-reward RL\.InProceedings of Thirty Eighth Conference on Learning Theory,Proceedings of Machine Learning Research, Vol\.291,pp\. 6156–6209\.External Links:[Link](https://proceedings.mlr.press/v291/zurek25a.html)Cited by:[Table 2](https://arxiv.org/html/2608.06545#S1.T2.8.2.4.1.1.1)\.

## Appendix ARobust Bellman equations and optimal bias spans

This appendix develops the Bellman theory underlying the nominal and robust optimal bias spans\. It establishes existence and minimum\-span attainment of optimal Bellman solutions, shows thatHσH\_\{\\sigma\}controls the span of the robust discounted optimal value, and comparesHσH\_\{\\sigma\}with the uniform\-over\-kernels span used in prior work\.

### A\.1Fixed\-policy robust average\-reward verification

The following lemma is the fixed\-policy robust analogue of the average\-reward verification argument inPuterman \([1994](https://arxiv.org/html/2608.06545#bib.bib42), Section 8\)\. For a stationary policyπ\\pi, writerπ​\(s\)=∑a∈𝒜π​\(a∣s\)​r​\(s,a\)r^\{\\pi\}\(s\)=\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\(a\\mid s\)r\(s,a\)\. Recall thatPπP\_\{\\pi\}denotes the transition matrix induced byπ\\piand a transition kernelPP, so that

\(Pπ​h\)​\(s\)=∑a∈𝒜π​\(a∣s\)​∑s′∈𝒮Ps,a​\(s′\)​h​\(s′\)\.\(P\_\{\\pi\}h\)\(s\)=\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\(a\\mid s\)\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}P\_\{s,a\}\(s^\{\\prime\}\)h\(s^\{\\prime\}\)\.
###### Lemma 1\(Fixed\-policy robust average\-reward verification\)\.

Fix a stationary policyπ\\piand suppose Assumption[1](https://arxiv.org/html/2608.06545#Thmassumption1)holds\. Leth:𝒮→ℝh:\\mathcal\{S\}\\to\\mathbb\{R\}be bounded and letρ∈ℝ\\rho\\in\\mathbb\{R\}\. If, for alls∈𝒮s\\in\\mathcal\{S\},

ρ\+h​\(s\)≥rπ​\(s\)\+infP∈𝒫\(Pπ​h\)​\(s\),\\rho\+h\(s\)\\geq r^\{\\pi\}\(s\)\+\\inf\_\{P\\in\\mathcal\{P\}\}\(P\_\{\\pi\}h\)\(s\),thenρπ,σ≤ρ\\rho^\{\\pi,\\sigma\}\\leq\\rho\. Conversely, if, for alls∈𝒮s\\in\\mathcal\{S\},

ρ\+h​\(s\)≤rπ​\(s\)\+infP∈𝒫\(Pπ​h\)​\(s\),\\rho\+h\(s\)\\leq r^\{\\pi\}\(s\)\+\\inf\_\{P\\in\\mathcal\{P\}\}\(P\_\{\\pi\}h\)\(s\),thenρπ,σ≥ρ\\rho^\{\\pi,\\sigma\}\\geq\\rho\.

###### Proof\.

We prove the two directions separately\.

##### Upper bound\.

We first proveρπ,σ≤ρ\\rho^\{\\pi,\\sigma\}\\leq\\rho\. For every\(s,a\)\(s,a\), compactness of𝒫s,a\\mathcal\{P\}\_\{s,a\}allows us to choose

Ps,ah∈arg⁡minPs,a∈𝒫s,a⁡Ps,a⊤​h\.P^\{h\}\_\{s,a\}\\in\\arg\\min\_\{P\_\{s,a\}\\in\\mathcal\{P\}\_\{s,a\}\}P\_\{s,a\}^\{\\top\}h\.By rectangularity, these transition vectors define a kernelPh∈𝒫P^\{h\}\\in\\mathcal\{P\}\. Consequently,

\(Pπh​h\)​\(s\)=infP∈𝒫\(Pπ​h\)​\(s\),s∈𝒮\.\(P^\{h\}\_\{\\pi\}h\)\(s\)=\\inf\_\{P\\in\\mathcal\{P\}\}\(P\_\{\\pi\}h\)\(s\),\\qquad s\\in\\mathcal\{S\}\.The assumed inequality gives

ρ\+h​\(s\)≥rπ​\(s\)\+\(Pπh​h\)​\(s\),∀s∈𝒮\.\\rho\+h\(s\)\\geq r^\{\\pi\}\(s\)\+\(P^\{h\}\_\{\\pi\}h\)\(s\),\\qquad\\forall s\\in\\mathcal\{S\}\.By Assumption[1](https://arxiv.org/html/2608.06545#Thmassumption1), the Markov chain induced by\(π,Ph\)\(\\pi,P^\{h\}\)has a stationary distributionμh\\mu^\{h\}supported on its unique recurrent class, and its average reward is state independent\. Multiplying the preceding inequality by\(μh\)⊤\(\\mu^\{h\}\)^\{\\top\}gives

ρ\+\(μh\)⊤​h≥\(μh\)⊤​rπ\+\(μh\)⊤​Pπh​h\.\\rho\+\(\\mu^\{h\}\)^\{\\top\}h\\geq\(\\mu^\{h\}\)^\{\\top\}r^\{\\pi\}\+\(\\mu^\{h\}\)^\{\\top\}P^\{h\}\_\{\\pi\}h\.Since\(μh\)⊤​Pπh=\(μh\)⊤\(\\mu^\{h\}\)^\{\\top\}P^\{h\}\_\{\\pi\}=\(\\mu^\{h\}\)^\{\\top\}and the stationary average reward equalsρPhπ\\rho\_\{P^\{h\}\}^\{\\pi\}, we have

\(μh\)⊤​Pπh​h=\(μh\)⊤​h,\(μh\)⊤​rπ=ρPhπ\.\(\\mu^\{h\}\)^\{\\top\}P^\{h\}\_\{\\pi\}h=\(\\mu^\{h\}\)^\{\\top\}h,\\qquad\(\\mu^\{h\}\)^\{\\top\}r^\{\\pi\}=\\rho\_\{P^\{h\}\}^\{\\pi\}\.Consequently,

ρ\+\(μh\)⊤​h≥ρPhπ\+\(μh\)⊤​h\.\\rho\+\(\\mu^\{h\}\)^\{\\top\}h\\geq\\rho\_\{P^\{h\}\}^\{\\pi\}\+\(\\mu^\{h\}\)^\{\\top\}h\.ThusρPhπ≤ρ\\rho\_\{P^\{h\}\}^\{\\pi\}\\leq\\rho\. Sinceρπ,σ=infP∈𝒫ρPπ\\rho^\{\\pi,\\sigma\}=\\inf\_\{P\\in\\mathcal\{P\}\}\\rho\_\{P\}^\{\\pi\}, we obtainρπ,σ≤ρ\\rho^\{\\pi,\\sigma\}\\leq\\rho\.

##### Lower bound\.

We now proveρπ,σ≥ρ\\rho^\{\\pi,\\sigma\}\\geq\\rho\. Fix an arbitraryP∈𝒫P\\in\\mathcal\{P\}\. SinceinfP′∈𝒫\(Pπ′​h\)​\(s\)≤\(Pπ​h\)​\(s\)\\inf\_\{P^\{\\prime\}\\in\\mathcal\{P\}\}\(P^\{\\prime\}\_\{\\pi\}h\)\(s\)\\leq\(P\_\{\\pi\}h\)\(s\), the assumed inequality implies

ρ\+h​\(s\)≤rπ​\(s\)\+\(Pπ​h\)​\(s\),∀s∈𝒮\.\\rho\+h\(s\)\\leq r^\{\\pi\}\(s\)\+\(P\_\{\\pi\}h\)\(s\),\\qquad\\forall s\\in\\mathcal\{S\}\.Letμ\\mube a stationary distribution supported on the unique recurrent class induced by\(π,P\)\(\\pi,P\)\. Multiplying the preceding inequality byμ⊤\\mu^\{\\top\}gives

ρ\+μ⊤​h≤μ⊤​rπ\+μ⊤​Pπ​h\.\\rho\+\\mu^\{\\top\}h\\leq\\mu^\{\\top\}r^\{\\pi\}\+\\mu^\{\\top\}P\_\{\\pi\}h\.Sinceμ⊤​Pπ=μ⊤\\mu^\{\\top\}P\_\{\\pi\}=\\mu^\{\\top\}andμ⊤​rπ=ρPπ\\mu^\{\\top\}r^\{\\pi\}=\\rho\_\{P\}^\{\\pi\}, it follows that

ρ\+μ⊤​h≤ρPπ\+μ⊤​h\.\\rho\+\\mu^\{\\top\}h\\leq\\rho\_\{P\}^\{\\pi\}\+\\mu^\{\\top\}h\.ThereforeρPπ≥ρ\\rho\_\{P\}^\{\\pi\}\\geq\\rho\. Taking the infimum overP∈𝒫P\\in\\mathcal\{P\}givesρπ,σ≥ρ\\rho^\{\\pi,\\sigma\}\\geq\\rho\. ∎

### A\.2Existence and attainment of robust optimal biases

This subsection establishes the Bellman foundations needed for the robust optimal bias span\. It proves that the robust average\-reward Bellman equation has a solution and that a minimum\-span solution exists\. The same results hold for the nominal Bellman equation by applying the same arguments to the singleton uncertainty set𝒫=\{P0\}\\mathcal\{P\}=\\\{P^\{0\}\\\}\.

We begin by connecting the discounted value definition in Section[4](https://arxiv.org/html/2608.06545#S4)with the robust Bellman equations used below\. This connection uses compactness and\(s,a\)\(s,a\)\-rectangularity, but not the unichain assumption\.

Forγ∈\(0,1\)\\gamma\\in\(0,1\)and a stationary policyπ\\pi, define the fixed\-policy robust discounted Bellman operator and the robust discounted Bellman optimality operator by

\(𝒯γ,σπ​v\)​\(s\)\\displaystyle\(\\mathcal\{T\}\_\{\\gamma,\\sigma\}^\{\\pi\}v\)\(s\)≔∑a∈𝒜π​\(a∣s\)​\{r​\(s,a\)\+γ​minPs,a∈𝒫s,a⁡Ps,a⊤​v\},\\displaystyle\\coloneqq\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\(a\\mid s\)\\left\\\{r\(s,a\)\+\\gamma\\min\_\{P\_\{s,a\}\\in\\mathcal\{P\}\_\{s,a\}\}P\_\{s,a\}^\{\\top\}v\\right\\\},\(9a\)\(𝒯γ,σ​v\)​\(s\)\\displaystyle\(\\mathcal\{T\}\_\{\\gamma,\\sigma\}v\)\(s\)≔maxa∈𝒜⁡\{r​\(s,a\)\+γ​minPs,a∈𝒫s,a⁡Ps,a⊤​v\}\.\\displaystyle\\coloneqq\\max\_\{a\\in\\mathcal\{A\}\}\\left\\\{r\(s,a\)\+\\gamma\\min\_\{P\_\{s,a\}\\in\\mathcal\{P\}\_\{s,a\}\}P\_\{s,a\}^\{\\top\}v\\right\\\}\.\(9b\)
###### Lemma 2\(Bellman characterization of robust discounted values\)\.

Fixγ∈\(0,1\)\\gamma\\in\(0,1\)\. For every stationary policyπ\\pi, the unique fixed point of the operator in \([9a](https://arxiv.org/html/2608.06545#A1.E9.1)\) equals the robust discounted valueVγπ,σV\_\{\\gamma\}^\{\\pi,\\sigma\}defined in Section[4](https://arxiv.org/html/2608.06545#S4)\. The unique fixed point of the operator in \([9b](https://arxiv.org/html/2608.06545#A1.E9.2)\) isVγ⋆,σV\_\{\\gamma\}^\{\\star,\\sigma\}\.

###### Proof\.

These are the standard discounted Bellman characterizations for rectangular robust MDPs\(Iyengar,[2005](https://arxiv.org/html/2608.06545#bib.bib887); Nilim and El Ghaoui,[2005](https://arxiv.org/html/2608.06545#bib.bib865)\)\. We give the short argument that also identifies their fixed points with the stationary\-kernel definitions used in this paper\. For anyf,g∈ℝ𝒮f,g\\in\\mathbb\{R\}^\{\\mathcal\{S\}\}and\(s,a\)\(s,a\),

\|minPs,a∈𝒫s,a⁡Ps,a⊤​f−minPs,a∈𝒫s,a⁡Ps,a⊤​g\|≤∥f−g∥∞\.\\left\|\\min\_\{P\_\{s,a\}\\in\\mathcal\{P\}\_\{s,a\}\}P\_\{s,a\}^\{\\top\}f\-\\min\_\{P\_\{s,a\}\\in\\mathcal\{P\}\_\{s,a\}\}P\_\{s,a\}^\{\\top\}g\\right\|\\leq\\lVert f\-g\\rVert\_\{\\infty\}\.Hence both operators are monotoneγ\\gamma\-contractions and have unique fixed points\.

Letvvbe the fixed point of𝒯γ,σπ\\mathcal\{T\}\_\{\\gamma,\\sigma\}^\{\\pi\}\. Compactness and rectangularity give a kernelPv∈𝒫P^\{v\}\\in\\mathcal\{P\}whose rows attain the minima atvv, sov=rπ\+γ​Pπv​v=Vγ,Pvπv=r^\{\\pi\}\+\\gamma P^\{v\}\_\{\\pi\}v=V\_\{\\gamma,P^\{v\}\}^\{\\pi\}\. For everyP∈𝒫P\\in\\mathcal\{P\},

𝒯γ,σπ​f≤rπ\+γ​Pπ​f\.\\mathcal\{T\}\_\{\\gamma,\\sigma\}^\{\\pi\}f\\leq r^\{\\pi\}\+\\gamma P\_\{\\pi\}f\.Iterating the right\-hand side fromvvgivesv≤Vγ,Pπv\\leq V\_\{\\gamma,P\}^\{\\pi\}\. ThusPvP^\{v\}attains the componentwise infimum andv=Vγπ,σv=V\_\{\\gamma\}^\{\\pi,\\sigma\}\.

Finally, letVVbe the fixed point of𝒯γ,σ\\mathcal\{T\}\_\{\\gamma,\\sigma\}and choose a deterministic policyπV\\pi\_\{V\}greedy with respect toVV\. Then𝒯γ,σπV​V=V\\mathcal\{T\}\_\{\\gamma,\\sigma\}^\{\\pi\_\{V\}\}V=V, whereas𝒯γ,σπ​V≤V\\mathcal\{T\}\_\{\\gamma,\\sigma\}^\{\\pi\}V\\leq Vfor every stationary policyπ\\pi\. The preceding fixed\-policy characterization and monotone iteration therefore give

V=VγπV,σandVγπ,σ≤Vfor every​π\.V=V\_\{\\gamma\}^\{\\pi\_\{V\},\\sigma\}\\quad\\text\{and\}\\quad V\_\{\\gamma\}^\{\\pi,\\sigma\}\\leq V\\quad\\text\{for every \}\\pi\.HenceV=Vγ⋆,σV=V\_\{\\gamma\}^\{\\star,\\sigma\}\. ∎

The next proposition proves directly that the robust average\-reward Bellman equation used in \([3b](https://arxiv.org/html/2608.06545#S2.E3.2)\) has a solution for the paper’s uncertainty set\. More generally, the proof applies whenever each local uncertainty set is a nonempty compact subset of the probability simplex and the resulting product set satisfies Assumption[1](https://arxiv.org/html/2608.06545#Thmassumption1)\.

###### Proposition 3\(Existence of a robust optimal Bellman solution\)\.

There existsh∈ℝ𝒮h\\in\\mathbb\{R\}^\{\\mathcal\{S\}\}satisfying

ρ⋆,σ​𝟏\+h=𝒯σ​h\.\\rho^\{\\star,\\sigma\}\\mathbf\{1\}\+h=\\mathcal\{T\}\_\{\\sigma\}h\.

###### Proof\.

We first establish the uniform discounted span bound needed for a vanishing\-discount argument\. Fix a deterministic stationary policyπ\\piandP∈𝒫P\\in\\mathcal\{P\}\. By Assumption[1](https://arxiv.org/html/2608.06545#Thmassumption1),PπP\_\{\\pi\}has a unique stationary distributionμPπ\\mu\_\{P\}^\{\\pi\}, andρPπ=\(μPπ\)⊤​rπ\\rho\_\{P\}^\{\\pi\}=\(\\mu\_\{P\}^\{\\pi\}\)^\{\\top\}r^\{\\pi\}\. Select the normalized bias

hPπ=\(I−Pπ\+𝟏​\(μPπ\)⊤\)−1​\(rπ−ρPπ​𝟏\)\.h\_\{P\}^\{\\pi\}=\\left\(I\-P\_\{\\pi\}\+\\mathbf\{1\}\(\\mu\_\{P\}^\{\\pi\}\)^\{\\top\}\\right\)^\{\-1\}\\left\(r^\{\\pi\}\-\\rho\_\{P\}^\{\\pi\}\\mathbf\{1\}\\right\)\.\(10\)Equivalently,

\(I−Pπ\+𝟏​\(μPπ\)⊤\)​hPπ=rπ−ρPπ​𝟏\.\\left\(I\-P\_\{\\pi\}\+\\mathbf\{1\}\(\\mu\_\{P\}^\{\\pi\}\)^\{\\top\}\\right\)h\_\{P\}^\{\\pi\}=r^\{\\pi\}\-\\rho\_\{P\}^\{\\pi\}\\mathbf\{1\}\.\(11\)Standard finite\-state unichain theory gives\(μPπ\)⊤​hPπ=0\(\\mu\_\{P\}^\{\\pi\}\)^\{\\top\}h\_\{P\}^\{\\pi\}=0and the Poisson equation

ρPπ​𝟏\+hPπ=rπ\+Pπ​hPπ\.\\rho\_\{P\}^\{\\pi\}\\mathbf\{1\}\+h\_\{P\}^\{\\pi\}=r^\{\\pi\}\+P\_\{\\pi\}h\_\{P\}^\{\\pi\}\.\(12\)The vector in \([10](https://arxiv.org/html/2608.06545#A1.E10)\) is the corresponding deviation matrix applied torπr^\{\\pi\}\. That matrix is continuous and uniformly bounded over compact unichain families\(Wanget al\.,[2023b](https://arxiv.org/html/2608.06545#bib.bib35), Appendix, Lemma 2\)\. Since rewards are bounded and the deterministic stationary policies form a finite set,

B≔supP∈𝒫,π‖hPπ‖span<∞,B\\coloneqq\\sup\_\{\\begin\{subarray\}\{c\}P\\in\\mathcal\{P\},\\pi\\end\{subarray\}\}\\\|h\_\{P\}^\{\\pi\}\\\|\_\{\\mathrm\{span\}\}<\\infty,\(13\)where the supremum is over deterministic stationary policies\.

Fixγ∈\(0,1\)\\gamma\\in\(0,1\)\. Choose rowwise minimizers atVγ⋆,σV\_\{\\gamma\}^\{\\star,\\sigma\}and assemble them intoPγ∈𝒫P\_\{\\gamma\}\\in\\mathcal\{P\}, and letπγ\\pi\_\{\\gamma\}be a deterministic greedy policy\. Lemma[2](https://arxiv.org/html/2608.06545#Thmlemma2)yields

Vγ⋆,σ=rπγ\+γ​\(Pγ\)πγ​Vγ⋆,σ\.V\_\{\\gamma\}^\{\\star,\\sigma\}=r^\{\\pi\_\{\\gamma\}\}\+\\gamma\(P\_\{\\gamma\}\)\_\{\\pi\_\{\\gamma\}\}V\_\{\\gamma\}^\{\\star,\\sigma\}\.\(14\)WriteQ=\(Pγ\)πγQ=\(P\_\{\\gamma\}\)\_\{\\pi\_\{\\gamma\}\},ρ=ρPγπγ\\rho=\\rho\_\{P\_\{\\gamma\}\}^\{\\pi\_\{\\gamma\}\}, andh=hPγπγh=h\_\{P\_\{\\gamma\}\}^\{\\pi\_\{\\gamma\}\}\. Combining \([14](https://arxiv.org/html/2608.06545#A1.E14)\) with the Poisson equation gives the resolvent identity

Vγ⋆,σ=ρ1−γ​𝟏\+h−qγ,qγ=\(1−γ\)​\(I−γ​Q\)−1​Q​h=\(1−γ\)​∑t=0∞γt​Qt\+1​h\.V\_\{\\gamma\}^\{\\star,\\sigma\}=\\frac\{\\rho\}\{1\-\\gamma\}\\mathbf\{1\}\+h\-q\_\{\\gamma\},\\qquad q\_\{\\gamma\}=\(1\-\\gamma\)\(I\-\\gamma Q\)^\{\-1\}Qh=\(1\-\\gamma\)\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}Q^\{t\+1\}h\.Each coordinate ofqγq\_\{\\gamma\}lies between the minimum and maximum coordinates ofhh\. Hence‖qγ‖span≤‖h‖span\\\|q\_\{\\gamma\}\\\|\_\{\\mathrm\{span\}\}\\leq\\\|h\\\|\_\{\\mathrm\{span\}\}, and therefore

supγ∈\(0,1\)‖Vγ⋆,σ‖span≤2​B<∞\.\\sup\_\{\\gamma\\in\(0,1\)\}\\\|V\_\{\\gamma\}^\{\\star,\\sigma\}\\\|\_\{\\mathrm\{span\}\}\\leq 2B<\\infty\.\(15\)
We now apply the vanishing\-discount argument\. Chooseγn→1\\gamma\_\{n\}\\to 1, fixs0∈𝒮s\_\{0\}\\in\\mathcal\{S\}, and set

un=Vγn⋆,σ−Vγn⋆,σ​\(s0\)​𝟏\.u\_\{n\}=V\_\{\\gamma\_\{n\}\}^\{\\star,\\sigma\}\-V\_\{\\gamma\_\{n\}\}^\{\\star,\\sigma\}\(s\_\{0\}\)\\mathbf\{1\}\.By \([15](https://arxiv.org/html/2608.06545#A1.E15)\),\(un\)\(u\_\{n\}\)is bounded; moreover, rewards in\[0,1\]\[0,1\]give0≤\(1−γn\)​Vγn⋆,σ​\(s0\)≤10\\leq\(1\-\\gamma\_\{n\}\)V\_\{\\gamma\_\{n\}\}^\{\\star,\\sigma\}\(s\_\{0\}\)\\leq 1\. Passing to a subsequence, let

un→h,\(1−γn\)​Vγn⋆,σ​\(s0\)→ρ¯\.u\_\{n\}\\to h,\\qquad\(1\-\\gamma\_\{n\}\)V\_\{\\gamma\_\{n\}\}^\{\\star,\\sigma\}\(s\_\{0\}\)\\to\\bar\{\\rho\}\.Translation of the discounted Bellman equation gives

\(1−γn\)​Vγn⋆,σ​\(s0\)​𝟏\+un=𝒯γn,σ​un\.\(1\-\\gamma\_\{n\}\)V\_\{\\gamma\_\{n\}\}^\{\\star,\\sigma\}\(s\_\{0\}\)\\mathbf\{1\}\+u\_\{n\}=\\mathcal\{T\}\_\{\\gamma\_\{n\},\\sigma\}u\_\{n\}\.\(16\)Since

‖𝒯γn,σ​un−𝒯σ​h‖∞≤γn​‖un−h‖∞\+\(1−γn\)​‖h‖∞⟶0,\\\|\\mathcal\{T\}\_\{\\gamma\_\{n\},\\sigma\}u\_\{n\}\-\\mathcal\{T\}\_\{\\sigma\}h\\\|\_\{\\infty\}\\leq\\gamma\_\{n\}\\\|u\_\{n\}\-h\\\|\_\{\\infty\}\+\(1\-\\gamma\_\{n\}\)\\\|h\\\|\_\{\\infty\}\\longrightarrow 0,taking limits yields

ρ¯​𝟏\+h=𝒯σ​h\.\\bar\{\\rho\}\\mathbf\{1\}\+h=\\mathcal\{T\}\_\{\\sigma\}h\.\(17\)
For every stationary policyπ\\pi, rectangularity and \([17](https://arxiv.org/html/2608.06545#A1.E17)\) give

ρ¯​𝟏\+h≥rπ\+infP∈𝒫Pπ​h\.\\bar\{\\rho\}\\mathbf\{1\}\+h\\geq r^\{\\pi\}\+\\inf\_\{P\\in\\mathcal\{P\}\}P\_\{\\pi\}h\.Lemma[1](https://arxiv.org/html/2608.06545#Thmlemma1)therefore impliesρπ,σ≤ρ¯\\rho^\{\\pi,\\sigma\}\\leq\\bar\{\\rho\}, and henceρ⋆,σ≤ρ¯\\rho^\{\\star,\\sigma\}\\leq\\bar\{\\rho\}\. Conversely, a deterministic policyπ¯\\bar\{\\pi\}greedy with respect tohhmakes the preceding inequality an equality\. The reverse direction of the same lemma givesρ¯≤ρπ¯,σ≤ρ⋆,σ\\bar\{\\rho\}\\leq\\rho^\{\\bar\{\\pi\},\\sigma\}\\leq\\rho^\{\\star,\\sigma\}\. Thusρ¯=ρ⋆,σ\\bar\{\\rho\}=\\rho^\{\\star,\\sigma\}, and \([17](https://arxiv.org/html/2608.06545#A1.E17)\) proves the claim\.

∎

###### Proposition 4\(Existence of a minimum\-span robust Bellman solution\)\.

There existsh⋆,σ∈ℝ𝒮h^\{\\star,\\sigma\}\\in\\mathbb\{R\}^\{\\mathcal\{S\}\}satisfying

ρ⋆,σ​𝟏\+h⋆,σ=𝒯σ​h⋆,σ\\rho^\{\\star,\\sigma\}\\mathbf\{1\}\+h^\{\\star,\\sigma\}=\\mathcal\{T\}\_\{\\sigma\}h^\{\\star,\\sigma\}and

‖h⋆,σ‖span=infh:ρ⋆,σ​𝟏\+h=𝒯σ​h‖h‖span\.\\\|h^\{\\star,\\sigma\}\\\|\_\{\\mathrm\{span\}\}=\\inf\_\{h:\\,\\rho^\{\\star,\\sigma\}\\mathbf\{1\}\+h=\\mathcal\{T\}\_\{\\sigma\}h\}\\\|h\\\|\_\{\\mathrm\{span\}\}\.Thus the inner infimum in \([3b](https://arxiv.org/html/2608.06545#S2.E3.2)\) is attained\.

###### Proof\.

Proposition[3](https://arxiv.org/html/2608.06545#Thmproposition3)makes the Bellman\-solution set nonempty\. Fixs0∈𝒮s\_\{0\}\\in\\mathcal\{S\}and normalize its elements ats0s\_\{0\}:

ℋ0≔\{h:h​\(s0\)=0,ρ⋆,σ​𝟏\+h=𝒯σ​h\}\.\\mathcal\{H\}\_\{0\}\\coloneqq\\left\\\{h:h\(s\_\{0\}\)=0,\\quad\\rho^\{\\star,\\sigma\}\\mathbf\{1\}\+h=\\mathcal\{T\}\_\{\\sigma\}h\\right\\\}\.This normalization does not change the span because𝒯σ​\(h\+c​𝟏\)=𝒯σ​h\+c​𝟏\\mathcal\{T\}\_\{\\sigma\}\(h\+c\\mathbf\{1\}\)=\\mathcal\{T\}\_\{\\sigma\}h\+c\\mathbf\{1\}\. The operator𝒯σ\\mathcal\{T\}\_\{\\sigma\}is11\-Lipschitz in the supremum norm, soℋ0\\mathcal\{H\}\_\{0\}is closed\. Moreover, ifh∈ℋ0h\\in\\mathcal\{H\}\_\{0\}and‖h‖span≤C\\\|h\\\|\_\{\\mathrm\{span\}\}\\leq C, then‖h‖∞≤C\\\|h\\\|\_\{\\infty\}\\leq C\. Hence every bounded\-span sublevel set ofℋ0\\mathcal\{H\}\_\{0\}is compact\.

Let

m=infh∈ℋ0‖h‖spanm=\\inf\_\{h\\in\\mathcal\{H\}\_\{0\}\}\\\|h\\\|\_\{\\mathrm\{span\}\}and choosehn∈ℋ0h\_\{n\}\\in\\mathcal\{H\}\_\{0\}with‖hn‖span≤m\+1/n\\\|h\_\{n\}\\\|\_\{\\mathrm\{span\}\}\\leq m\+1/n\. The sequence lies in a compact bounded\-span sublevel set, so a subsequence converges to someh⋆,σ∈ℋ0h^\{\\star,\\sigma\}\\in\\mathcal\{H\}\_\{0\}\. Continuity of the span seminorm gives‖h⋆,σ‖span=m\\\|h^\{\\star,\\sigma\}\\\|\_\{\\mathrm\{span\}\}=m, proving the claim\. ∎

### A\.3Discounted\-value control by the robust bias span

The next lemma shows thatHσH\_\{\\sigma\}controls the span of the robust discounted optimal value\.

###### Lemma 3\(Robust discounted optimal\-value span\)\.

For everyγ∈\(0,1\)\\gamma\\in\(0,1\),

‖Vγ⋆,σ‖span≤2​Hσ\.\\\|V\_\{\\gamma\}^\{\\star,\\sigma\}\\\|\_\{\\mathrm\{span\}\}\\leq 2H\_\{\\sigma\}\.

###### Proof\.

Fixγ∈\(0,1\)\\gamma\\in\(0,1\)\. We first state the comparison used below\. Leth∈ℝ𝒮h\\in\\mathbb\{R\}^\{\\mathcal\{S\}\}satisfy

ρ⋆,σ​𝟏\+h=𝒯σ​h,\\rho^\{\\star,\\sigma\}\\mathbf\{1\}\+h=\\mathcal\{T\}\_\{\\sigma\}h,and suppose thatmins∈𝒮⁡h​\(s\)=0\\min\_\{s\\in\\mathcal\{S\}\}h\(s\)=0\. Then

h\+\(ρ⋆,σ1−γ−‖h‖span\)​𝟏≤Vγ⋆,σ≤h\+ρ⋆,σ1−γ​𝟏\.h\+\\left\(\\frac\{\\rho^\{\\star,\\sigma\}\}\{1\-\\gamma\}\-\\\|h\\\|\_\{\\mathrm\{span\}\}\\right\)\\mathbf\{1\}\\leq V\_\{\\gamma\}^\{\\star,\\sigma\}\\leq h\+\\frac\{\\rho^\{\\star,\\sigma\}\}\{1\-\\gamma\}\\mathbf\{1\}\.\(18\)
By Proposition[4](https://arxiv.org/html/2608.06545#Thmproposition4), the robust Bellman solutionh⋆,σh^\{\\star,\\sigma\}attaining the minimum in \([3b](https://arxiv.org/html/2608.06545#S2.E3.2)\) exists\. Shift it by a constant so thatmins⁡h⋆,σ​\(s\)=0\\min\_\{s\}h^\{\\star,\\sigma\}\(s\)=0\. For anys,t∈𝒮s,t\\in\\mathcal\{S\}, applying the upper comparison in \([18](https://arxiv.org/html/2608.06545#A1.E18)\) atssand the lower comparison atttgives

Vγ⋆,σ​\(s\)−Vγ⋆,σ​\(t\)≤h⋆,σ​\(s\)−h⋆,σ​\(t\)\+‖h⋆,σ‖span≤2​Hσ\.V\_\{\\gamma\}^\{\\star,\\sigma\}\(s\)\-V\_\{\\gamma\}^\{\\star,\\sigma\}\(t\)\\leq h^\{\\star,\\sigma\}\(s\)\-h^\{\\star,\\sigma\}\(t\)\+\\\|h^\{\\star,\\sigma\}\\\|\_\{\\mathrm\{span\}\}\\leq 2H\_\{\\sigma\}\.Taking the maximum overs,ts,tproves the claimed span bound\.

##### Proof of \([18](https://arxiv.org/html/2608.06545#A1.E18)\)\.

The conditionmins∈𝒮⁡h​\(s\)=0\\min\_\{s\\in\\mathcal\{S\}\}h\(s\)=0gives

0≤h≤‖h‖span​𝟏\.0\\leq h\\leq\\\|h\\\|\_\{\\mathrm\{span\}\}\\mathbf\{1\}\.Hence, for every\(s,a\)\(s,a\),

0≤minPs,a∈𝒫s,a⁡Ps,a⊤​h≤‖h‖span\.0\\leq\\min\_\{P\_\{s,a\}\\in\\mathcal\{P\}\_\{s,a\}\}P\_\{s,a\}^\{\\top\}h\\leq\\\|h\\\|\_\{\\mathrm\{span\}\}\.Comparing the definition of𝒯σ\\mathcal\{T\}\_\{\\sigma\}in Section[2](https://arxiv.org/html/2608.06545#S2)with the definition of𝒯γ,σ\\mathcal\{T\}\_\{\\gamma,\\sigma\}in \([9b](https://arxiv.org/html/2608.06545#A1.E9.2)\) gives

𝒯σ​h−\(1−γ\)​‖h‖span​𝟏≤𝒯γ,σ​h≤𝒯σ​h\.\\mathcal\{T\}\_\{\\sigma\}h\-\(1\-\\gamma\)\\\|h\\\|\_\{\\mathrm\{span\}\}\\mathbf\{1\}\\leq\\mathcal\{T\}\_\{\\gamma,\\sigma\}h\\leq\\mathcal\{T\}\_\{\\sigma\}h\.Using the robust average\-reward Bellman equation \([2b](https://arxiv.org/html/2608.06545#S2.E2.2)\),

ρ⋆,σ​𝟏\+h−\(1−γ\)​‖h‖span​𝟏≤𝒯γ,σ​h≤ρ⋆,σ​𝟏\+h\.\\rho^\{\\star,\\sigma\}\\mathbf\{1\}\+h\-\(1\-\\gamma\)\\\|h\\\|\_\{\\mathrm\{span\}\}\\mathbf\{1\}\\leq\\mathcal\{T\}\_\{\\gamma,\\sigma\}h\\leq\\rho^\{\\star,\\sigma\}\\mathbf\{1\}\+h\.\(19\)For everyv∈ℝ𝒮v\\in\\mathbb\{R\}^\{\\mathcal\{S\}\}andc∈ℝc\\in\\mathbb\{R\}, the discounted Bellman operator satisfies the shift identity

𝒯γ,σ​\(v\+c​𝟏\)=𝒯γ,σ​v\+γ​c​𝟏\.\\mathcal\{T\}\_\{\\gamma,\\sigma\}\(v\+c\\mathbf\{1\}\)=\\mathcal\{T\}\_\{\\gamma,\\sigma\}v\+\\gamma c\\mathbf\{1\}\.Applying the lower bound in \([19](https://arxiv.org/html/2608.06545#A1.E19)\) yields

𝒯γ,σ​\(h\+\(ρ⋆,σ1−γ−‖h‖span\)​𝟏\)\\displaystyle\\mathcal\{T\}\_\{\\gamma,\\sigma\}\\left\(h\+\\left\(\\frac\{\\rho^\{\\star,\\sigma\}\}\{1\-\\gamma\}\-\\\|h\\\|\_\{\\mathrm\{span\}\}\\right\)\\mathbf\{1\}\\right\)≥h\+\(ρ⋆,σ1−γ−‖h‖span\)​𝟏\.\\displaystyle\\qquad\\geq h\+\\left\(\\frac\{\\rho^\{\\star,\\sigma\}\}\{1\-\\gamma\}\-\\\|h\\\|\_\{\\mathrm\{span\}\}\\right\)\\mathbf\{1\}\.Thus the lower comparison vector in \([18](https://arxiv.org/html/2608.06545#A1.E18)\) is a subsolution\. Similarly, the upper bound in \([19](https://arxiv.org/html/2608.06545#A1.E19)\) yields

𝒯γ,σ​\(h\+ρ⋆,σ1−γ​𝟏\)≤h\+ρ⋆,σ1−γ​𝟏,\\mathcal\{T\}\_\{\\gamma,\\sigma\}\\left\(h\+\\frac\{\\rho^\{\\star,\\sigma\}\}\{1\-\\gamma\}\\mathbf\{1\}\\right\)\\leq h\+\\frac\{\\rho^\{\\star,\\sigma\}\}\{1\-\\gamma\}\\mathbf\{1\},so the upper comparison vector is a supersolution\. By Lemma[2](https://arxiv.org/html/2608.06545#Thmlemma2), the unique fixed point of𝒯γ,σ\\mathcal\{T\}\_\{\\gamma,\\sigma\}isVγ⋆,σV\_\{\\gamma\}^\{\\star,\\sigma\}\. Iterating this monotone contraction from the subsolution and the supersolution proves \([18](https://arxiv.org/html/2608.06545#A1.E18)\)\. ∎

### A\.4Comparison with other robust bias span parameters

Our bounds are expressed in terms of the minimum robust optimal bias spanHσH\_\{\\sigma\}, whereas the bounds ofRochet al\.\([2025](https://arxiv.org/html/2608.06545#bib.bib8)\)andRochet al\.\([2026](https://arxiv.org/html/2608.06545#bib.bib9)\)use different robust span parameters\. Comparing the rates therefore requires comparing the parameters themselves\. We show below thatHσH\_\{\\sigma\}is no larger than either of the two parameters and then explain whyHσH\_\{\\sigma\}suffices for our analysis\. Throughout this subsection, we truncate all span parameters from below at11\.111Rochet al\.\([2025](https://arxiv.org/html/2608.06545#bib.bib8)\)define their parameter without this truncation, although their reduction proof uses the corresponding normalization truncated below at11;Rochet al\.\([2026](https://arxiv.org/html/2608.06545#bib.bib9)\)assume that their parameter is at least11\.

##### Prior span parameters\.

We start with the span parameter used inRochet al\.\([2025](https://arxiv.org/html/2608.06545#bib.bib8)\)\. Let\(πRoch⋆,hRoch⋆\)\(\\pi^\{\\star\}\_\{\\mathrm\{Roch\}\},h^\{\\star\}\_\{\\mathrm\{Roch\}\}\)be a robust\-optimal policy–bias pair satisfying

ρ⋆,σ​𝟏S\+hRoch⋆=𝒯σ​hRoch⋆,\\rho^\{\\star,\\sigma\}\\bm\{1\}\_\{S\}\+h^\{\\star\}\_\{\\mathrm\{Roch\}\}=\\mathcal\{T\}\_\{\\sigma\}h^\{\\star\}\_\{\\mathrm\{Roch\}\},withπRoch⋆\\pi^\{\\star\}\_\{\\mathrm\{Roch\}\}attaining the statewise maximum in𝒯σ\\mathcal\{T\}\_\{\\sigma\}\. Their span parameter, in the form used by their proof, is

HRoch≔max⁡\{1,maxP∈𝒫⁡‖hPπRoch⋆‖span\},H\_\{\\mathrm\{Roch\}\}\\coloneqq\\max\\left\\\{1,\\max\_\{P\\in\\mathcal\{P\}\}\\\|h\_\{P\}^\{\\pi^\{\\star\}\_\{\\mathrm\{Roch\}\}\}\\\|\_\{\\mathrm\{span\}\}\\right\\\},wherehPπRoch⋆h\_\{P\}^\{\\pi^\{\\star\}\_\{\\mathrm\{Roch\}\}\}is the ordinary average\-reward bias ofπRoch⋆\\pi^\{\\star\}\_\{\\mathrm\{Roch\}\}under kernelPP\. Thus,HRochH\_\{\\mathrm\{Roch\}\}controls the bias of one robust\-optimal policy uniformly over every kernel in the uncertainty set\.

The robust Halpern iteration result ofRochet al\.\([2026](https://arxiv.org/html/2608.06545#bib.bib9)\)instead uses

HRHI≔max⁡\{1,maxπ:ρπ,σ=ρ⋆,σP∈𝒫:ρPπ=ρπ,σ⁡‖hPπ‖span\}\.H\_\{\\mathrm\{RHI\}\}\\coloneqq\\max\\left\\\{1,\\,\\max\_\{\\begin\{subarray\}\{c\}\\pi:\\,\\rho^\{\\pi,\\sigma\}=\\rho^\{\\star,\\sigma\}\\\\ P\\in\\mathcal\{P\}:\\,\\rho\_\{P\}^\{\\pi\}=\\rho^\{\\pi,\\sigma\}\\end\{subarray\}\}\\\|h\_\{P\}^\{\\pi\}\\\|\_\{\\mathrm\{span\}\}\\right\\\}\.This parameter controls the ordinary biases associated with all robust\-optimal policies and their worst\-case kernels\.

##### Comparison withHσH\_\{\\sigma\}\.

Our definition minimizes the span over robust Bellman solutions\. SincehRoch⋆h^\{\\star\}\_\{\\mathrm\{Roch\}\}is one such solution,

Hσ≤max⁡\{1,‖hRoch⋆‖span\}\.H\_\{\\sigma\}\\leq\\max\\left\\\{1,\\,\\\|h^\{\\star\}\_\{\\mathrm\{Roch\}\}\\\|\_\{\\mathrm\{span\}\}\\right\\\}\.To relatehRoch⋆h^\{\\star\}\_\{\\mathrm\{Roch\}\}to the ordinary bias functions appearing in the two prior parameters, we associate it with a kernel in the uncertainty set\. For every state, choose transition rows attaining the minima in the fixed\-policy robust Bellman operator\. Compactness guarantees that these minimizers exist, and rectangularity allows them to be assembled into a single kernelP†∈𝒫P^\{\\dagger\}\\in\\mathcal\{P\}satisfying

\(PπRoch⋆†​hRoch⋆\)​\(s\)=infP∈𝒫\(PπRoch⋆​hRoch⋆\)​\(s\),s∈𝒮\.\\bigl\(P^\{\\dagger\}\_\{\\pi^\{\\star\}\_\{\\mathrm\{Roch\}\}\}h^\{\\star\}\_\{\\mathrm\{Roch\}\}\\bigr\)\(s\)=\\inf\_\{P\\in\\mathcal\{P\}\}\\bigl\(P\_\{\\pi^\{\\star\}\_\{\\mathrm\{Roch\}\}\}h^\{\\star\}\_\{\\mathrm\{Roch\}\}\\bigr\)\(s\),\\qquad s\\in\\mathcal\{S\}\.BecauseπRoch⋆\\pi^\{\\star\}\_\{\\mathrm\{Roch\}\}attains the maximum in the robust Bellman equation, we therefore have

ρ⋆,σ​𝟏S\+hRoch⋆=rπRoch⋆\+infP∈𝒫PπRoch⋆​hRoch⋆=rπRoch⋆\+PπRoch⋆†​hRoch⋆\.\\rho^\{\\star,\\sigma\}\\bm\{1\}\_\{S\}\+h^\{\\star\}\_\{\\mathrm\{Roch\}\}=r^\{\\pi^\{\\star\}\_\{\\mathrm\{Roch\}\}\}\+\\inf\_\{P\\in\\mathcal\{P\}\}P\_\{\\pi^\{\\star\}\_\{\\mathrm\{Roch\}\}\}h^\{\\star\}\_\{\\mathrm\{Roch\}\}=r^\{\\pi^\{\\star\}\_\{\\mathrm\{Roch\}\}\}\+P^\{\\dagger\}\_\{\\pi^\{\\star\}\_\{\\mathrm\{Roch\}\}\}h^\{\\star\}\_\{\\mathrm\{Roch\}\}\.Applying both directions of Lemma[1](https://arxiv.org/html/2608.06545#Thmlemma1)to the first equality givesρπRoch⋆,σ=ρ⋆,σ\\rho^\{\\pi^\{\\star\}\_\{\\mathrm\{Roch\}\},\\sigma\}=\\rho^\{\\star,\\sigma\}\. The second equality, together with Assumption[1](https://arxiv.org/html/2608.06545#Thmassumption1), identifiesρ⋆,σ\\rho^\{\\star,\\sigma\}as the average reward ofπRoch⋆\\pi^\{\\star\}\_\{\\mathrm\{Roch\}\}underP†P^\{\\dagger\}\. HenceρP†πRoch⋆=ρ⋆,σ\\rho\_\{P^\{\\dagger\}\}^\{\\pi^\{\\star\}\_\{\\mathrm\{Roch\}\}\}=\\rho^\{\\star,\\sigma\}andhRoch⋆h^\{\\star\}\_\{\\mathrm\{Roch\}\}is, up to an additive constant, the ordinary biashP†πRoch⋆h\_\{P^\{\\dagger\}\}^\{\\pi^\{\\star\}\_\{\\mathrm\{Roch\}\}\}\. ThusP†P^\{\\dagger\}is a worst\-case kernel forπRoch⋆\\pi^\{\\star\}\_\{\\mathrm\{Roch\}\}\.

The definition ofHRochH\_\{\\mathrm\{Roch\}\}maximizes over all kernels in𝒫\\mathcal\{P\}for the fixed policyπRoch⋆\\pi^\{\\star\}\_\{\\mathrm\{Roch\}\}, so its maximization includesP†P^\{\\dagger\}\. Moreover,πRoch⋆\\pi^\{\\star\}\_\{\\mathrm\{Roch\}\}is robust optimal andP†P^\{\\dagger\}is worst\-case for this policy, so the pair\(πRoch⋆,P†\)\(\\pi^\{\\star\}\_\{\\mathrm\{Roch\}\},P^\{\\dagger\}\)is included in the maximization definingHRHIH\_\{\\mathrm\{RHI\}\}\. Therefore,

max⁡\{1,‖hRoch⋆‖span\}≤HRochandmax⁡\{1,‖hRoch⋆‖span\}≤HRHI\.\\max\\left\\\{1,\\\|h^\{\\star\}\_\{\\mathrm\{Roch\}\}\\\|\_\{\\mathrm\{span\}\}\\right\\\}\\leq H\_\{\\mathrm\{Roch\}\}\\qquad\\text\{and\}\\qquad\\max\\left\\\{1,\\\|h^\{\\star\}\_\{\\mathrm\{Roch\}\}\\\|\_\{\\mathrm\{span\}\}\\right\\\}\\leq H\_\{\\mathrm\{RHI\}\}\.Combining these inequalities with the preceding bound onHσH\_\{\\sigma\}gives

Hσ≤min⁡\{HRoch,HRHI\}\.H\_\{\\sigma\}\\leq\\min\\left\\\{H\_\{\\mathrm\{Roch\}\},H\_\{\\mathrm\{RHI\}\}\\right\\\}\.

##### WhyHσH\_\{\\sigma\}suffices\.

Our reduction only requires uniform control of the span of the robust discounted optimal value\. Lemma[3](https://arxiv.org/html/2608.06545#Thmlemma3)provides exactly this control: for everyγ∈\(0,1\)\\gamma\\in\(0,1\),

‖Vγ⋆,σ‖span≤2​Hσ\.\\\|V\_\{\\gamma\}^\{\\star,\\sigma\}\\\|\_\{\\mathrm\{span\}\}\\leq 2H\_\{\\sigma\}\.This bound is obtained directly from a minimum\-span robust Bellman solution\. Consequently, our analysis does not require uniform control over all kernels or over all robust\-optimal bias functions\.

## Appendix BProofs for Section[2](https://arxiv.org/html/2608.06545#S2)

This appendix contains the proofs of the propositions stated in Section[2](https://arxiv.org/html/2608.06545#S2)\.

### B\.1Proof of Proposition[1](https://arxiv.org/html/2608.06545#Thmproposition1)

Fixσ\>0\\sigma\>0andH0,Hσ≥1H\_\{0\},H\_\{\\sigma\}\\geq 1\. IfH0≠HσH\_\{0\}\\neq H\_\{\\sigma\}, choose

0<g≤min⁡\{1,σ\|H0−1−Hσ−1\|\}\.0<g\\leq\\min\\left\\\{1,\\frac\{\\sigma\}\{\\left\|H\_\{0\}^\{\-1\}\-H\_\{\\sigma\}^\{\-1\}\\right\|\}\\right\\\}\.IfH0=HσH\_\{0\}=H\_\{\\sigma\}, setg=1g=1\. Consider a two\-state AMDP with one action and rewards

r​\(1\)=g,r​\(2\)=0\.r\(1\)=g,\\qquad r\(2\)=0\.Because there is only one action, we suppress its action index below\. Denote the local TV radii at states11and22byσ1\\sigma\_\{1\}andσ2\\sigma\_\{2\}, respectively\. In both cases below, let the nominal transition vectors be

P10=\(1,0\),P20=\(gH0,1−gH0\)\.P^\{0\}\_\{1\}=\(1,0\),\\qquad P^\{0\}\_\{2\}=\\left\(\\frac\{g\}\{H\_\{0\}\},1\-\\frac\{g\}\{H\_\{0\}\}\\right\)\.The nominal Bellman equation is

ρ\+h​\(1\)=g\+h​\(1\),ρ\+h​\(2\)=gH0​h​\(1\)\+\(1−gH0\)​h​\(2\)\.\\rho\+h\(1\)=g\+h\(1\),\\qquad\\rho\+h\(2\)=\\frac\{g\}\{H\_\{0\}\}h\(1\)\+\\left\(1\-\\frac\{g\}\{H\_\{0\}\}\\right\)h\(2\)\.Thus every nominal Bellman solution satisfiesρ=g\\rho=gandh​\(1\)−h​\(2\)=H0h\(1\)\-h\(2\)=H\_\{0\}\. Its span is therefore exactlyH0H\_\{0\}\.

We now consider the two casesHσ≥H0H\_\{\\sigma\}\\geq H\_\{0\}andHσ<H0H\_\{\\sigma\}<H\_\{0\}separately\. In each case, we set the local TV radii so that the robust Bellman solution has span exactlyHσH\_\{\\sigma\}\.

##### CaseHσ≥H0H\_\{\\sigma\}\\geq H\_\{0\}\.

Set the local TV radii to

σ1=0,σ2=g​\(1H0−1Hσ\)\.\\sigma\_\{1\}=0,\\qquad\\sigma\_\{2\}=g\\left\(\\frac\{1\}\{H\_\{0\}\}\-\\frac\{1\}\{H\_\{\\sigma\}\}\\right\)\.SinceHσ≥H0H\_\{\\sigma\}\\geq H\_\{0\},σ2\\sigma\_\{2\}is nonnegative and at mostσ\\sigmaby the choice ofgg\. Every admissible transition vector at state22has the form

P2=\(p,1−p\),gHσ=gH0−σ2≤p≤min⁡\{1,gH0\+σ2\}\.P\_\{2\}=\(p,1\-p\),\\qquad\\frac\{g\}\{H\_\{\\sigma\}\}=\\frac\{g\}\{H\_\{0\}\}\-\\sigma\_\{2\}\\leq p\\leq\\min\\left\\\{1,\\frac\{g\}\{H\_\{0\}\}\+\\sigma\_\{2\}\\right\\\}\.Becauseσ1=0\\sigma\_\{1\}=0, state11is absorbing\. Moreover,p≥g/Hσ\>0p\\geq g/H\_\{\\sigma\}\>0, so from state22the chain reaches state11with positive probability at each step and state22is transient\. Hence every admissible kernel has the unique recurrent class\{1\}\\\{1\\\}\. Since there is only one action, this verifies Assumption[1](https://arxiv.org/html/2608.06545#Thmassumption1)\.

For a robust Bellman solution, the equation at state11is

ρ⋆,σ\+h​\(1\)=g\+h​\(1\),\\rho^\{\\star,\\sigma\}\+h\(1\)=g\+h\(1\),soρ⋆,σ=g\\rho^\{\\star,\\sigma\}=g\. The equation at state22is

ρ⋆,σ\+h​\(2\)=minP2∈𝒫2⁡\{P2​\(1\)​h​\(1\)\+P2​\(2\)​h​\(2\)\}\.\\rho^\{\\star,\\sigma\}\+h\(2\)=\\min\_\{P\_\{2\}\\in\\mathcal\{P\}\_\{2\}\}\\left\\\{P\_\{2\}\(1\)h\(1\)\+P\_\{2\}\(2\)h\(2\)\\right\\\}\.UsingP2​\(1\)\+P2​\(2\)=1P\_\{2\}\(1\)\+P\_\{2\}\(2\)=1andρ⋆,σ=g\\rho^\{\\star,\\sigma\}=g, this equation becomes

g=minP2∈𝒫2⁡P2​\(1\)​\(h​\(1\)−h​\(2\)\)\.g=\\min\_\{P\_\{2\}\\in\\mathcal\{P\}\_\{2\}\}P\_\{2\}\(1\)\\bigl\(h\(1\)\-h\(2\)\\bigr\)\.The right\-hand side cannot be positive ifh​\(1\)−h​\(2\)≤0h\(1\)\-h\(2\)\\leq 0\. Henceh​\(1\)−h​\(2\)\>0h\(1\)\-h\(2\)\>0, so the minimizing transition vector assigns the smallest possible probabilityg/Hσg/H\_\{\\sigma\}to state11\. It follows that every robust Bellman solution satisfies

h​\(1\)−h​\(2\)=Hσ\.h\(1\)\-h\(2\)=H\_\{\\sigma\}\.

##### CaseHσ<H0H\_\{\\sigma\}<H\_\{0\}\.

Set

σ1=g​\(1Hσ−1H0\),σ2=0\.\\sigma\_\{1\}=g\\left\(\\frac\{1\}\{H\_\{\\sigma\}\}\-\\frac\{1\}\{H\_\{0\}\}\\right\),\\qquad\\sigma\_\{2\}=0\.SinceHσ<H0H\_\{\\sigma\}<H\_\{0\}, we have0<σ1≤σ0<\\sigma\_\{1\}\\leq\\sigmaandσ1<1\\sigma\_\{1\}<1\. Every admissible kernel has transition vectors

P1=\(1−x,x\),0≤x≤σ1,P2=\(gH0,1−gH0\)\.P\_\{1\}=\(1\-x,x\),\\qquad 0\\leq x\\leq\\sigma\_\{1\},\\qquad P\_\{2\}=\\left\(\\frac\{g\}\{H\_\{0\}\},1\-\\frac\{g\}\{H\_\{0\}\}\\right\)\.Whenx=0x=0, state11is the unique recurrent class and state22is transient\. Whenx\>0x\>0, the chain is irreducible\. Thus every admissible kernel is unichain\.

For a robust Bellman solution, the equations at states11and22are

ρ⋆,σ\+h​\(1\)\\displaystyle\\rho^\{\\star,\\sigma\}\+h\(1\)=g\+min0≤x≤σ1⁡\{\(1−x\)​h​\(1\)\+x​h​\(2\)\},\\displaystyle=g\+\\min\_\{0\\leq x\\leq\\sigma\_\{1\}\}\\left\\\{\(1\-x\)h\(1\)\+xh\(2\)\\right\\\},ρ⋆,σ\+h​\(2\)\\displaystyle\\rho^\{\\star,\\sigma\}\+h\(2\)=gH0​h​\(1\)\+\(1−gH0\)​h​\(2\)\.\\displaystyle=\\frac\{g\}\{H\_\{0\}\}h\(1\)\+\\left\(1\-\\frac\{g\}\{H\_\{0\}\}\\right\)h\(2\)\.Ifh​\(1\)−h​\(2\)≤0h\(1\)\-h\(2\)\\leq 0, the minimum in the first equation equalsh​\(1\)h\(1\)\. The two equations would then give

ρ⋆,σ=g,ρ⋆,σ=gH0​\(h​\(1\)−h​\(2\)\)≤0,\\rho^\{\\star,\\sigma\}=g,\\qquad\\rho^\{\\star,\\sigma\}=\\frac\{g\}\{H\_\{0\}\}\\bigl\(h\(1\)\-h\(2\)\\bigr\)\\leq 0,which is impossible\. Henceh​\(1\)−h​\(2\)\>0h\(1\)\-h\(2\)\>0, and the minimum in the first equation is attained atx=σ1x=\\sigma\_\{1\}\. The two Bellman equations therefore give

ρ⋆,σ=g−σ1​\(h​\(1\)−h​\(2\)\),ρ⋆,σ=gH0​\(h​\(1\)−h​\(2\)\)\.\\rho^\{\\star,\\sigma\}=g\-\\sigma\_\{1\}\\bigl\(h\(1\)\-h\(2\)\\bigr\),\\qquad\\rho^\{\\star,\\sigma\}=\\frac\{g\}\{H\_\{0\}\}\\bigl\(h\(1\)\-h\(2\)\\bigr\)\.Equating these two expressions yields

g=\{gH0\+σ1\}​\(h​\(1\)−h​\(2\)\)=gHσ​\(h​\(1\)−h​\(2\)\)\.g=\\left\\\{\\frac\{g\}\{H\_\{0\}\}\+\\sigma\_\{1\}\\right\\\}\\bigl\(h\(1\)\-h\(2\)\\bigr\)=\\frac\{g\}\{H\_\{\\sigma\}\}\\bigl\(h\(1\)\-h\(2\)\\bigr\)\.Thus every robust Bellman solution again satisfiesh​\(1\)−h​\(2\)=Hσh\(1\)\-h\(2\)=H\_\{\\sigma\}\.

In either ordering, the nominal and robust Bellman solution sets therefore have spans exactlyH0H\_\{0\}andHσH\_\{\\sigma\}, respectively\. Since both prescribed spans are at least one, the maxima with one in \([3a](https://arxiv.org/html/2608.06545#S2.E3.1)\) and \([3b](https://arxiv.org/html/2608.06545#S2.E3.2)\) do not change these values\. This proves the proposition\.

### B\.2Proof of Proposition[2](https://arxiv.org/html/2608.06545#Thmproposition2)

We use a nominal optimal bias as a robust Bellman certificate and show that transition perturbations reduce this certificate by at mostσ​H0\\sigma H\_\{0\}\.

Leth0≔hP0⋆h\_\{0\}\\coloneqq h\_\{P^\{0\}\}^\{\\star\}\. For each statess, choose

a0​\(s\)∈argmaxa∈𝒜\{r​\(s,a\)\+∑s′∈𝒮Ps,a0​\(s′\)​h0​\(s′\)\},π0​\(a∣s\)≔𝟙\{a=a0​\(s\)\}\.a\_\{0\}\(s\)\\in\\operatorname\*\{argmax\}\_\{a\\in\\mathcal\{A\}\}\\left\\\{r\(s,a\)\+\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}P^\{0\}\_\{s,a\}\(s^\{\\prime\}\)h\_\{0\}\(s^\{\\prime\}\)\\right\\\},\\qquad\\pi\_\{0\}\(a\\mid s\)\\coloneqq\\mathds\{1\}\_\{\\\{a=a\_\{0\}\(s\)\\\}\}\.Thus,π0\\pi\_\{0\}is a deterministic policy attaining the maximum in the nominal Bellman equation at every state and is therefore nominally optimal\. Moreover, for every stationary policyπ\\pi, sinceP0∈𝒫P^\{0\}\\in\\mathcal\{P\},

ρπ,σ=infP∈𝒫ρPπ≤ρP0π\.\\rho^\{\\pi,\\sigma\}=\\inf\_\{P\\in\\mathcal\{P\}\}\\rho\_\{P\}^\{\\pi\}\\leq\\rho\_\{P^\{0\}\}^\{\\pi\}\.Taking the supremum overπ\\pigives

ρ⋆,σ≤ρ⋆\.\\rho^\{\\star,\\sigma\}\\leq\\rho^\{\\star\}\.
It remains to lower boundρ⋆,σ\\rho^\{\\star,\\sigma\}\. By the construction ofπ0\\pi\_\{0\}, for every statess,

ρ⋆\+h0​\(s\)=∑a∈𝒜π0​\(a∣s\)​\{r​\(s,a\)\+∑s′∈𝒮Ps,a0​\(s′\)​h0​\(s′\)\}\.\\rho^\{\\star\}\+h\_\{0\}\(s\)=\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\_\{0\}\(a\\mid s\)\\left\\\{r\(s,a\)\+\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}P\_\{s,a\}^\{0\}\(s^\{\\prime\}\)h\_\{0\}\(s^\{\\prime\}\)\\right\\\}\.\(20\)For any distributionsp,qp,qand any vectorhh,

\|\(p−q\)⊤​h\|≤‖p−q‖TV​‖h‖span\.\|\(p\-q\)^\{\\top\}h\|\\leq\\\|p\-q\\\|\_\{\\mathrm\{TV\}\}\\\|h\\\|\_\{\\mathrm\{span\}\}\.By the definition ofH0H\_\{0\},‖h0‖span≤H0\\\|h\_\{0\}\\\|\_\{\\mathrm\{span\}\}\\leq H\_\{0\}\. Hence, for anyP∈𝒫P\\in\\mathcal\{P\}and any state\-action pair\(s,a\)\(s,a\),

∑s′∈𝒮\(Ps,a​\(s′\)−Ps,a0​\(s′\)\)​h0​\(s′\)≥−‖Ps,a−Ps,a0‖TV​‖h0‖span≥−σ​H0\.\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}\\left\(P\_\{s,a\}\(s^\{\\prime\}\)\-P\_\{s,a\}^\{0\}\(s^\{\\prime\}\)\\right\)h\_\{0\}\(s^\{\\prime\}\)\\geq\-\\\|P\_\{s,a\}\-P\_\{s,a\}^\{0\}\\\|\_\{\\mathrm\{TV\}\}\\\|h\_\{0\}\\\|\_\{\\mathrm\{span\}\}\\geq\-\\sigma H\_\{0\}\.Because the policy weights sum to one, averaging this rowwise bound with weightsπ0​\(a∣s\)\\pi\_\{0\}\(a\\mid s\)does not enlarge the error\. Taking the infimum overP∈𝒫P\\in\\mathcal\{P\}therefore gives

infP∈𝒫∑a∈𝒜π0​\(a∣s\)​∑s′∈𝒮Ps,a​\(s′\)​h0​\(s′\)≥∑a∈𝒜π0​\(a∣s\)​∑s′∈𝒮Ps,a0​\(s′\)​h0​\(s′\)−σ​H0\.\\inf\_\{P\\in\\mathcal\{P\}\}\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\_\{0\}\(a\\mid s\)\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}P\_\{s,a\}\(s^\{\\prime\}\)h\_\{0\}\(s^\{\\prime\}\)\\geq\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\_\{0\}\(a\\mid s\)\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}P\_\{s,a\}^\{0\}\(s^\{\\prime\}\)h\_\{0\}\(s^\{\\prime\}\)\-\\sigma H\_\{0\}\.Substituting this into \([20](https://arxiv.org/html/2608.06545#A2.E20)\), we have

ρ⋆−σ​H0\+h0​\(s\)≤∑a∈𝒜π0​\(a∣s\)​r​\(s,a\)\+infP∈𝒫∑a∈𝒜π0​\(a∣s\)​∑s′∈𝒮Ps,a​\(s′\)​h0​\(s′\)\.\\rho^\{\\star\}\-\\sigma H\_\{0\}\+h\_\{0\}\(s\)\\leq\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\_\{0\}\(a\\mid s\)r\(s,a\)\+\\inf\_\{P\\in\\mathcal\{P\}\}\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\_\{0\}\(a\\mid s\)\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}P\_\{s,a\}\(s^\{\\prime\}\)h\_\{0\}\(s^\{\\prime\}\)\.By Lemma[1](https://arxiv.org/html/2608.06545#Thmlemma1), applied toπ0\\pi\_\{0\}withρ=ρ⋆−σ​H0\\rho=\\rho^\{\\star\}\-\\sigma H\_\{0\}andh=h0h=h\_\{0\}, this implies

ρπ0,σ≥ρ⋆−σ​H0\.\\rho^\{\\pi\_\{0\},\\sigma\}\\geq\\rho^\{\\star\}\-\\sigma H\_\{0\}\.
Sinceρπ0,σ≤ρ⋆,σ\\rho^\{\\pi\_\{0\},\\sigma\}\\leq\\rho^\{\\star,\\sigma\}andρ⋆,σ≤ρ⋆\\rho^\{\\star,\\sigma\}\\leq\\rho^\{\\star\}, we conclude that

ρ⋆−σ​H0≤ρπ0,σ≤ρ⋆,σ≤ρ⋆,\\rho^\{\\star\}\-\\sigma H\_\{0\}\\leq\\rho^\{\\pi\_\{0\},\\sigma\}\\leq\\rho^\{\\star,\\sigma\}\\leq\\rho^\{\\star\},which proves the proposition\.

## Appendix CProofs for the lower bound

This appendix proves Theorem[1](https://arxiv.org/html/2608.06545#Thmtheorem1)\. We first define the hard\-instance template and compute the quantities used in the proof\. We then convert policy estimation into a two\-point testing problem and verify the required inequalities for each parameter regime\.

### C\.1Instance template

We construct a pair of instances\{ℳϕ:ϕ∈\{2,3\}\}\\\{\\mathcal\{M\}\_\{\\phi\}:\\phi\\in\\\{2,3\\\}\\\}with the same states, actions, and rewards\. The indexϕ\\phidetermines which of the actions22and33has the larger transition probability to its reward state\.

##### Definition of the instances\.

Recall thatSSandAAdenote the numbers of states and actions and thatσ\\sigmais the global uncertainty level\. For eachϕ∈\{2,3\}\\phi\\in\\\{2,3\\\}, defineℳϕ\\mathcal\{M\}\_\{\\phi\}as follows\. We write its nominal kernel asP0P^\{0\}, suppressing its dependence onϕ\\phi\.

- •Parameters\.We use p0,p1,δ,r0,r1∈\[0,1\],q0,q1∈\(0,1\],σ0,σ1∈\[0,σ\]\.p\_\{0\},p\_\{1\},\\delta,r\_\{0\},r\_\{1\}\\in\[0,1\],\\qquad q\_\{0\},q\_\{1\}\\in\(0,1\],\\qquad\\sigma\_\{0\},\\sigma\_\{1\}\\in\[0,\\sigma\]\.The three parameter choices below set these quantities differently\. We always choose them so that σ0\+p0≤1,σ1\+p1\+δ≤1\.\\sigma\_\{0\}\+p\_\{0\}\\leq 1,\\qquad\\sigma\_\{1\}\+p\_\{1\}\+\\delta\\leq 1\.
- •States and actions\.The state space is𝒮=\{1,…,S\}\\mathcal\{S\}=\\\{1,\\ldots,S\\\}and the action space is𝒜=\{1,…,A\}\\mathcal\{A\}=\\\{1,\\ldots,A\\\}\. StateSSis the decision state, states1,2,31,2,3are reward states, state44is the delay state, and whenS≥5S\\geq 5, states5,…,S−15,\\ldots,S\-1are padding states\.
- •Rewards\.For any actionaa, r​\(s,a\)=\{r0if​s=1,r1if​s∈\{2,3\},0otherwise\.r\(s,a\)=\\begin\{cases\}r\_\{0\}&\\text\{if \}s=1,\\\\ r\_\{1\}&\\text\{if \}s\\in\\\{2,3\\\},\\\\ 0&\\text\{otherwise\}\\end\{cases\}\.
- •Nominal transition kernel\. - –Delay and padding states: fors∈\{4,…,S−1\}s\\in\\\{4,\\ldots,S\-1\\\}anda∈𝒜a\\in\\mathcal\{A\},Ps,a0​\(S\)=1P^\{0\}\_\{s,a\}\(S\)=1andPs,a0​\(s′\)=0P^\{0\}\_\{s,a\}\(s^\{\\prime\}\)=0for everys′≠Ss^\{\\prime\}\\neq S\. - –Decision state: fors=Ss=S, 1. 1\.Whena=1a=1, Ps,a0​\(s′\)=\{σ0\+p0if​s′=11−p0−σ0if​s′=S0otherwise\.P\_\{s,a\}^\{0\}\(s^\{\\prime\}\)=\\begin\{cases\}\\sigma\_\{0\}\+p\_\{0\}&\\text\{if \}s^\{\\prime\}=1\\\\ 1\-p\_\{0\}\-\\sigma\_\{0\}&\\text\{if \}s^\{\\prime\}=S\\\\ 0&\\text\{otherwise\}\\end\{cases\}\. 2. 2\.Whena∈\{2,3\}a\\in\\\{2,3\\\}, Ps,a0​\(s′\)=\{σ1\+p1\+δ​𝟙\{ϕ=a\}if​s′=a1−p1−σ1−δ​𝟙\{ϕ=a\}if​s′=S0otherwise\.P\_\{s,a\}^\{0\}\(s^\{\\prime\}\)=\\begin\{cases\}\\sigma\_\{1\}\+p\_\{1\}\+\\delta\\mathds\{1\}\_\{\\\{\\phi=a\\\}\}&\\text\{if \}s^\{\\prime\}=a\\\\ 1\-p\_\{1\}\-\\sigma\_\{1\}\-\\delta\\mathds\{1\}\_\{\\\{\\phi=a\\\}\}&\\text\{if \}s^\{\\prime\}=S\\\\ 0&\\text\{otherwise\}\\end\{cases\}\. 3. 3\.Whena∉\{1,2,3\}a\\notin\\\{1,2,3\\\},Ps,a0​\(S\)=1P^\{0\}\_\{s,a\}\(S\)=1\. - –Reward states: 1. 1\.At state11, P1,10​\(1\)=1−q0,P1,10​\(S\)=q0,P1,a0​\(S\)=1,a∈𝒜∖\{1\}\.P^\{0\}\_\{1,1\}\(1\)=1\-q\_\{0\},\\qquad P^\{0\}\_\{1,1\}\(S\)=q\_\{0\},\\qquad P^\{0\}\_\{1,a\}\(S\)=1,\\quad a\\in\\mathcal\{A\}\\setminus\\\{1\\\}\. 2. 2\.At each states∈\{2,3\}s\\in\\\{2,3\\\}, Ps,s0​\(s\)=1−q1,Ps,s0​\(S\)=q1,Ps,a0​\(S\)=1,a∈𝒜∖\{s\}\.P^\{0\}\_\{s,s\}\(s\)=1\-q\_\{1\},\\qquad P^\{0\}\_\{s,s\}\(S\)=q\_\{1\},\\qquad P^\{0\}\_\{s,a\}\(S\)=1,\\quad a\\in\\mathcal\{A\}\\setminus\\\{s\\\}\. All unspecified entries in these rows are zero\.
- •Uncertainty set\.The set𝒫\\mathcal\{P\}is the standard rectangular TV ball centered at the nominal kernel\. For the decision\-state pairs, set σS,1=σ0,σS,a=σ1,a∈\{2,3\},\\sigma\_\{S,1\}=\\sigma\_\{0\},\\qquad\\sigma\_\{S,a\}=\\sigma\_\{1\},\\quad a\\in\\\{2,3\\\},and for all other state\-action pairs setσs,a=0\\sigma\_\{s,a\}=0\. The local uncertainty sets are 𝒫s,a=\{Ps,a′∈Δ​\(𝒮\):‖Ps,a′−Ps,a0‖TV≤σs,a\}\.\\mathcal\{P\}\_\{s,a\}=\\left\\\{P^\{\\prime\}\_\{s,a\}\\in\\Delta\(\\mathcal\{S\}\):\\\|P^\{\\prime\}\_\{s,a\}\-P^\{0\}\_\{s,a\}\\\|\_\{\\mathrm\{TV\}\}\\leq\\sigma\_\{s,a\}\\right\\\}\.\(21\)Rectangularity means that 𝒫=∏\(s,a\)∈𝒮×𝒜𝒫s,a\.\\mathcal\{P\}=\\prod\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}\\mathcal\{P\}\_\{s,a\}\.Sinceσ0,σ1≤σ\\sigma\_\{0\},\\sigma\_\{1\}\\leq\\sigma, all local radii are bounded by the global uncertainty levelσ\\sigma\. For zero\-radius pairs, the uncertainty set is a singleton\.

The nominal transition structure is illustrated in Figure[3](https://arxiv.org/html/2608.06545#A3.F3)\.

padding states\(if any\)SSdecision44delayreward011rewardr0r\_\{0\}22rewardr1r\_\{1\}33rewardr1r\_\{1\}σ0\+p0\\sigma\_\{0\}\+p\_\{0\}σ1\+p1\+δ\\sigma\_\{1\}\+p\_\{1\}\+\\deltaσ1\+p1\\sigma\_\{1\}\+p\_\{1\}Action 1Action 2Action 3q0q\_\{0\}q1q\_\{1\}q1q\_\{1\}Figure 3:The nominal transition structure forϕ=2\\phi=2\. Self\-loops and transitions that return directly toSSare omitted\. Forϕ=3\\phi=3, the roles of actions22and33are reversed\.To analyze worst\-case performance, we single out an admissible kernelP¯\\bar\{P\}that moves the available uncertainty mass from the reward states to the zero\-reward delay state44\. We will show that this kernel attains the robust minimum for the comparison policies and use it to bound the robust loss of an arbitrary policy in the testing reduction\. At the decision state, define

P¯S,1ϕ​\(1\)\\displaystyle\\bar\{P\}\_\{S,1\}^\{\\phi\}\(1\)=p0,\\displaystyle=p\_\{0\},P¯S,1ϕ​\(4\)\\displaystyle\\bar\{P\}\_\{S,1\}^\{\\phi\}\(4\)=σ0,\\displaystyle=\\sigma\_\{0\},P¯S,1ϕ​\(S\)\\displaystyle\\bar\{P\}\_\{S,1\}^\{\\phi\}\(S\)=1−p0−σ0,\\displaystyle=1\-p\_\{0\}\-\\sigma\_\{0\},\(22\)P¯S,aϕ​\(a\)\\displaystyle\\bar\{P\}\_\{S,a\}^\{\\phi\}\(a\)=p1\+δ​𝟙\{ϕ=a\},\\displaystyle=p\_\{1\}\+\\delta\\mathds\{1\}\_\{\\\{\\phi=a\\\}\},P¯S,aϕ​\(4\)\\displaystyle\\bar\{P\}\_\{S,a\}^\{\\phi\}\(4\)=σ1,\\displaystyle=\\sigma\_\{1\},P¯S,aϕ​\(S\)\\displaystyle\\bar\{P\}\_\{S,a\}^\{\\phi\}\(S\)=1−p1−σ1−δ​𝟙\{ϕ=a\},a∈\{2,3\}\.\\displaystyle=1\-p\_\{1\}\-\\sigma\_\{1\}\-\\delta\\mathds\{1\}\_\{\\\{\\phi=a\\\}\},\\quad a\\in\\\{2,3\\\}\.All unspecified entries in these rows are zero, andP¯s,aϕ=Ps,a0\\bar\{P\}\_\{s,a\}^\{\\phi\}=P^\{0\}\_\{s,a\}at every other state\-action pair\. In particular,

‖P¯S,1ϕ−PS,10‖TV=σ0,‖P¯S,aϕ−PS,a0‖TV=σ1,a∈\{2,3\},\\left\\\|\\bar\{P\}\_\{S,1\}^\{\\phi\}\-P^\{0\}\_\{S,1\}\\right\\\|\_\{\\mathrm\{TV\}\}=\\sigma\_\{0\},\\qquad\\left\\\|\\bar\{P\}\_\{S,a\}^\{\\phi\}\-P^\{0\}\_\{S,a\}\\right\\\|\_\{\\mathrm\{TV\}\}=\\sigma\_\{1\},\\quad a\\in\\\{2,3\\\},while every other row has TV distance zero from its nominal counterpart\. HenceP¯ϕ∈𝒫\\bar\{P\}^\{\\phi\}\\in\\mathcal\{P\}by \([21](https://arxiv.org/html/2608.06545#A3.E21)\)\.

##### Action\-wise comparison policies\.

To compare the actions available atSS, we associate each action with a reference policy\. The robust average rewards and expected return times of these policies will provide the action\-wise benchmarks used in the testing reduction\. Specifically, for everya∈𝒜a\\in\\mathcal\{A\}, letπa\\pi\_\{a\}choose actionaaat every state:

πa​\(b∣s\)=𝟙\{b=a\},b∈𝒜,s∈𝒮\.\\pi\_\{a\}\(b\\mid s\)=\\mathds\{1\}\_\{\\\{b=a\\\}\},\\qquad b\\in\\mathcal\{A\},s\\in\\mathcal\{S\}\.\(23\)Fora∈\{1,2,3\}a\\in\\\{1,2,3\\\}, the policyπa\\pi\_\{a\}selects actionaaatSSand continues with actionaaat reward stateaa\. At either of the other reward states, it returns immediately toSS\. Its actions at the delay and padding states are immaterial because these states return deterministically toSS\.

##### Unichain property\.

Fix a stationary policy and an admissible kernel\. Every delay or padding state returns directly toSS\. At reward state11, action11returns toSSwith probabilityq0\>0q\_\{0\}\>0, while every other action returns toSSimmediately\. At each reward statei∈\{2,3\}i\\in\\\{2,3\\\}, actioniireturns toSSwith probabilityq1\>0q\_\{1\}\>0, while every other action returns immediately\. Consequently, every state reachesSSalmost surely, regardless of the policy’s randomization\. Every recurrent class must therefore containSS, so the induced Markov chain has exactly one recurrent class\. Hence the constructed MDP satisfies Assumption[1](https://arxiv.org/html/2608.06545#Thmassumption1)\.

##### Average rewards between successive returns toSS\.

Because the process returns to the decision stateSSunder every stationary policy and everyP∈𝒫P\\in\\mathcal\{P\}, we can analyze its average reward through the reward accumulated and time elapsed between successive visits toSS\. Starting froms0=Ss\_\{0\}=S, let

τS\+≔inf\{t≥1:st=S\}\\tau\_\{S\}^\{\+\}\\coloneqq\\inf\\\{t\\geq 1:s\_\{t\}=S\\\}denote the first return time toSS\. Fix a stationary policyπ\\piand a kernelP∈𝒫P\\in\\mathcal\{P\}\. The return timeτS\+\\tau\_\{S\}^\{\+\}has finite expectation\. Since successive visits toSSdivide the trajectory into identically distributed cycles, the average reward equals the expected reward in one cycle divided by its expected length:

ρPπ=𝔼Pπ\[∑t=0τS\+−1r\(st,at\)\|s0=S\]𝔼Pπ​\[τS\+∣s0=S\]\.\\rho\_\{P\}^\{\\pi\}=\\frac\{\\mathbb\{E\}\_\{P\}^\{\\pi\}\\\!\\left\[\\sum\_\{t=0\}^\{\\tau\_\{S\}^\{\+\}\-1\}r\(s\_\{t\},a\_\{t\}\)\\,\\middle\|\\,s\_\{0\}=S\\right\]\}\{\\mathbb\{E\}\_\{P\}^\{\\pi\}\[\\tau\_\{S\}^\{\+\}\\mid s\_\{0\}=S\]\}\.\(24\)For eacha∈𝒜a\\in\\mathcal\{A\}, define

ρa≔ρP0πa,ρaσ≔minP∈𝒫⁡ρPπa,Taσ≔𝔼P¯ϕπa​\[τS\+∣s0=S\]\.\\rho\_\{a\}\\coloneqq\\rho\_\{P^\{0\}\}^\{\\pi\_\{a\}\},\\qquad\\rho\_\{a\}^\{\\sigma\}\\coloneqq\\min\_\{P\\in\\mathcal\{P\}\}\\rho\_\{P\}^\{\\pi\_\{a\}\},\\qquad T\_\{a\}^\{\\sigma\}\\coloneqq\\mathbb\{E\}\_\{\\bar\{P\}^\{\\phi\}\}^\{\\pi\_\{a\}\}\[\\tau\_\{S\}^\{\+\}\\mid s\_\{0\}=S\]\.These quantities depend onϕ\\phi, but we leave this dependence implicit in the notation\. The following lemma computes these quantities and verifies thatP¯ϕ\\bar\{P\}^\{\\phi\}attains the robust minimum for the policies in \([23](https://arxiv.org/html/2608.06545#A3.E23)\)\. The proof is deferred to Appendix[C\.3\.1](https://arxiv.org/html/2608.06545#A3.SS3.SSS1)\.

###### Lemma 4\.

For the constructed instances,P¯ϕ\\bar\{P\}^\{\\phi\}attainsminP∈𝒫⁡ρPπa\\min\_\{P\\in\\mathcal\{P\}\}\\rho\_\{P\}^\{\\pi\_\{a\}\}for everya∈𝒜a\\in\\mathcal\{A\}\. The nominal and robust average rewards and the expected return times forπ1,πϕ\\pi\_\{1\},\\pi\_\{\\phi\}, andπ5−ϕ\\pi\_\{5\-\\phi\}are as follows\.

- •Forπ1\\pi\_\{1\}, ρ1=σ0\+p0σ0\+p0\+q0⋅r0,ρ1σ=p0p0\+q0​\(1\+σ0\)⋅r0,T1σ=1\+σ0\+p0q0\.\\rho\_\{1\}=\\frac\{\\sigma\_\{0\}\+p\_\{0\}\}\{\\sigma\_\{0\}\+p\_\{0\}\+q\_\{0\}\}\\cdot r\_\{0\},\\qquad\\rho\_\{1\}^\{\\sigma\}=\\frac\{p\_\{0\}\}\{p\_\{0\}\+q\_\{0\}\(1\+\\sigma\_\{0\}\)\}\\cdot r\_\{0\},\\qquad T\_\{1\}^\{\\sigma\}=1\+\\sigma\_\{0\}\+\\frac\{p\_\{0\}\}\{q\_\{0\}\}\.
- •Forπϕ\\pi\_\{\\phi\}, ρϕ=σ1\+p1\+δσ1\+p1\+q1\+δ⋅r1,ρϕσ=p1\+δp1\+δ\+q1​\(1\+σ1\)⋅r1,Tϕσ=1\+σ1\+p1\+δq1\.\\rho\_\{\\phi\}=\\frac\{\\sigma\_\{1\}\+p\_\{1\}\+\\delta\}\{\\sigma\_\{1\}\+p\_\{1\}\+q\_\{1\}\+\\delta\}\\cdot r\_\{1\},\\qquad\\rho\_\{\\phi\}^\{\\sigma\}=\\frac\{p\_\{1\}\+\\delta\}\{p\_\{1\}\+\\delta\+q\_\{1\}\(1\+\\sigma\_\{1\}\)\}\\cdot r\_\{1\},\\qquad T\_\{\\phi\}^\{\\sigma\}=1\+\\sigma\_\{1\}\+\\frac\{p\_\{1\}\+\\delta\}\{q\_\{1\}\}\.
- •Forπ5−ϕ\\pi\_\{5\-\\phi\}, ρ5−ϕ=σ1\+p1σ1\+p1\+q1⋅r1,ρ5−ϕσ=p1p1\+q1​\(1\+σ1\)⋅r1,T5−ϕσ=1\+σ1\+p1q1\.\\rho\_\{5\-\\phi\}=\\frac\{\\sigma\_\{1\}\+p\_\{1\}\}\{\\sigma\_\{1\}\+p\_\{1\}\+q\_\{1\}\}\\cdot r\_\{1\},\\qquad\\rho\_\{5\-\\phi\}^\{\\sigma\}=\\frac\{p\_\{1\}\}\{p\_\{1\}\+q\_\{1\}\(1\+\\sigma\_\{1\}\)\}\\cdot r\_\{1\},\\qquad T\_\{5\-\\phi\}^\{\\sigma\}=1\+\\sigma\_\{1\}\+\\frac\{p\_\{1\}\}\{q\_\{1\}\}\.

Fora∉\{1,2,3\}a\\notin\\\{1,2,3\\\}, the action returns immediately toSSwith no reward, soTaσ=1T\_\{a\}^\{\\sigma\}=1andρa=ρaσ=0\\rho\_\{a\}=\\rho\_\{a\}^\{\\sigma\}=0\.

### C\.2Proof of Theorem[1](https://arxiv.org/html/2608.06545#Thmtheorem1)

It suffices to verify the two sample\-size components in Theorem[1](https://arxiv.org/html/2608.06545#Thmtheorem1)separately\. We first establish a common testing argument and then apply it to three constructions\. The first construction proves the common minimax linear minimum\-span componentmin⁡\{H0,Hσ\}\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\.

Once all these components are established, we can combine them to obtain the desired lower bound\.

For theσ​Hσ2\\sigma H\_\{\\sigma\}^\{2\}component, it suffices to assume

ε≤0\.01​σ​min⁡\{H0,Hσ\}\.\\varepsilon\\leq 0\.01\\sigma\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\.\(25\)If this condition fails while condition \(b\) of Theorem[1](https://arxiv.org/html/2608.06545#Thmtheorem1)holds, namelyε≤0\.01​σ​H0\\varepsilon\\leq 0\.01\\sigma H\_\{0\}, thenHσ<H0H\_\{\\sigma\}<H\_\{0\}\. In this case,σ​Hσ<100​ε≤1\\sigma H\_\{\\sigma\}<100\\varepsilon\\leq 1and therefore

min⁡\{H0,Hσ\}\+σ​Hσ2≤2​Hσ=2​min⁡\{H0,Hσ\}\.\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\+\\sigma H\_\{\\sigma\}^\{2\}\\leq 2H\_\{\\sigma\}=2\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\.\(26\)Thus themin⁡\{H0,Hσ\}\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}component already covers the desired lower bound\. We proceed to prove theσ​Hσ2\\sigma H\_\{\\sigma\}^\{2\}component under this assumption with two different constructions, one forσ<4/min⁡\{H0,Hσ\}\\sigma<4/\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}and one forσ≥4/min⁡\{H0,Hσ\}\\sigma\\geq 4/\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\.

#### C\.2\.1A generic testing reduction

We start with a generic testing argument\. Suppose that a pair of instances\{ℳϕ:ϕ∈\{2,3\}\}\\\{\\mathcal\{M\}\_\{\\phi\}:\\phi\\in\\\{2,3\\\}\\\}satisfies

ρϕσ≤r0,ρϕσ≤r1,ϕ∈\{2,3\},\\rho\_\{\\phi\}^\{\\sigma\}\\leq r\_\{0\},\\qquad\\rho\_\{\\phi\}^\{\\sigma\}\\leq r\_\{1\},\\qquad\\phi\\in\\\{2,3\\\},\(27\)and

TaσTϕσ\+Taσ​\(ρϕσ−ρaσ\)\>ε,a∈𝒜,a≠ϕ,ϕ∈\{2,3\}\.\\frac\{T\_\{a\}^\{\\sigma\}\}\{T\_\{\\phi\}^\{\\sigma\}\+T\_\{a\}^\{\\sigma\}\}\\bigl\(\\rho\_\{\\phi\}^\{\\sigma\}\-\\rho\_\{a\}^\{\\sigma\}\\bigr\)\>\\varepsilon,\\qquad a\\in\\mathcal\{A\},\\ a\\neq\\phi,\\ \\phi\\in\\\{2,3\\\}\.\(28\)In particular, \([28](https://arxiv.org/html/2608.06545#A3.E28)\) impliesρϕσ\>ρaσ\\rho\_\{\\phi\}^\{\\sigma\}\>\\rho\_\{a\}^\{\\sigma\}for everya≠ϕa\\neq\\phi\.

Under \([27](https://arxiv.org/html/2608.06545#A3.E27)\) and \([28](https://arxiv.org/html/2608.06545#A3.E28)\), we derive the minimax lower bound in three steps\. First, we convert any policy estimator into a test ofϕ\\phi\. Next, we use Le Cam’s method to lower\-bound the resulting testing error in terms of KL divergence\. Finally, we compute the relevant KL divergence for the two instances\.

##### Step 1: From policy error to testing error\.

We begin by comparing the return time and reward obtained after each possible action atSSunder an arbitrary stationary policy\. Fix a stationary policyπ\\piand writewa=π​\(a∣S\)w\_\{a\}=\\pi\(a\\mid S\)\. Recall from \([22](https://arxiv.org/html/2608.06545#A3.E22)\) thatP¯ϕ\\bar\{P\}^\{\\phi\}moves the available uncertainty mass in the rows out ofSSfrom the reward states to the zero\-reward delay state44, while leaving all other rows unchanged\. By Lemma[4](https://arxiv.org/html/2608.06545#Thmlemma4), it attainsminP∈𝒫⁡ρPπa\\min\_\{P\\in\\mathcal\{P\}\}\\rho\_\{P\}^\{\\pi\_\{a\}\}for every comparison policyπa\\pi\_\{a\}\. For the arbitrary policyπ\\piconsidered here, we use only thatP¯ϕ∈𝒫\\bar\{P\}^\{\\phi\}\\in\\mathcal\{P\}: we evaluateπ\\piunderP¯ϕ\\bar\{P\}^\{\\phi\}and consider the path between successive returns toSS\.

First fixa∈\{1,2,3\}a\\in\\\{1,2,3\\\}\. In the row corresponding to actionaa, letppdenote the probability of entering reward stateaaunderP¯ϕ\\bar\{P\}^\{\\phi\}, letσ′\\sigma^\{\\prime\}denote the probability of entering state44, and letqqandrrdenote the return probability and reward at stateaa\. Ifβ\\betais the probability thatπ\\pichooses actionaawhenever stateaais visited, then, conditional on choosing actionaaatSS, the expected return timeT~aπ\\widetilde\{T\}\_\{a\}^\{\\pi\}and rewardR~aπ\\widetilde\{R\}\_\{a\}^\{\\pi\}are

T~aπ=1\+σ′\+p1−β​\(1−q\),R~aπ=p​r1−β​\(1−q\)\.\\widetilde\{T\}\_\{a\}^\{\\pi\}=1\+\\sigma^\{\\prime\}\+\\frac\{p\}\{1\-\\beta\(1\-q\)\},\\qquad\\widetilde\{R\}\_\{a\}^\{\\pi\}=\\frac\{pr\}\{1\-\\beta\(1\-q\)\}\.Whenβ=1\\beta=1, these expressions become

Taσ=1\+σ′\+pq,Taσ​ρaσ=p​rq\.T\_\{a\}^\{\\sigma\}=1\+\\sigma^\{\\prime\}\+\\frac\{p\}\{q\},\\qquad T\_\{a\}^\{\\sigma\}\\rho\_\{a\}^\{\\sigma\}=\\frac\{pr\}\{q\}\.Moreover,

1−β​\(1−q\)≥q,ρϕσ−r≤0,1\-\\beta\(1\-q\)\\geq q,\\qquad\\rho\_\{\\phi\}^\{\\sigma\}\-r\\leq 0,where the second inequality follows from \([27](https://arxiv.org/html/2608.06545#A3.E27)\)\. Therefore,

ρϕσ​T~aπ−R~aπ\\displaystyle\\rho\_\{\\phi\}^\{\\sigma\}\\widetilde\{T\}\_\{a\}^\{\\pi\}\-\\widetilde\{R\}\_\{a\}^\{\\pi\}=ρϕσ​\(1\+σ′\)\+p​\(ρϕσ−r\)1−β​\(1−q\)\\displaystyle=\\rho\_\{\\phi\}^\{\\sigma\}\(1\+\\sigma^\{\\prime\}\)\+\\frac\{p\(\\rho\_\{\\phi\}^\{\\sigma\}\-r\)\}\{1\-\\beta\(1\-q\)\}≥ρϕσ​\(1\+σ′\)\+p​\(ρϕσ−r\)q\\displaystyle\\geq\\rho\_\{\\phi\}^\{\\sigma\}\(1\+\\sigma^\{\\prime\}\)\+\\frac\{p\(\\rho\_\{\\phi\}^\{\\sigma\}\-r\)\}\{q\}=Taσ​\(ρϕσ−ρaσ\)\.\\displaystyle=T\_\{a\}^\{\\sigma\}\\bigl\(\\rho\_\{\\phi\}^\{\\sigma\}\-\\rho\_\{a\}^\{\\sigma\}\\bigr\)\.\(29\)We also haveT~aπ≤Taσ\\widetilde\{T\}\_\{a\}^\{\\pi\}\\leq T\_\{a\}^\{\\sigma\}\. Fora∉\{1,2,3\}a\\notin\\\{1,2,3\\\}, the return time is one and the reward is zero, so both conclusions remain valid\.

We next average these action\-wise comparisons according to the probabilitieswaw\_\{a\}to bound the robust loss ofπ\\pi\. Letρ¯π\\bar\{\\rho\}^\{\\pi\}be the average reward ofπ\\piunderP¯ϕ\\bar\{P\}^\{\\phi\}\. Since a return cycle begins with actionaaatSSwith probabilitywaw\_\{a\}, \([24](https://arxiv.org/html/2608.06545#A3.E24)\) gives

ρ¯π=∑a∈𝒜wa​R~aπ∑a∈𝒜wa​T~aπ\.\\bar\{\\rho\}^\{\\pi\}=\\frac\{\\sum\_\{a\\in\\mathcal\{A\}\}w\_\{a\}\\widetilde\{R\}\_\{a\}^\{\\pi\}\}\{\\sum\_\{a\\in\\mathcal\{A\}\}w\_\{a\}\\widetilde\{T\}\_\{a\}^\{\\pi\}\}\.Consequently,

ρϕσ−ρ¯π=∑a∈𝒜wa​\(ρϕσ​T~aπ−R~aπ\)∑a∈𝒜wa​T~aπ\.\\rho\_\{\\phi\}^\{\\sigma\}\-\\bar\{\\rho\}^\{\\pi\}=\\frac\{\\sum\_\{a\\in\\mathcal\{A\}\}w\_\{a\}\\bigl\(\\rho\_\{\\phi\}^\{\\sigma\}\\widetilde\{T\}\_\{a\}^\{\\pi\}\-\\widetilde\{R\}\_\{a\}^\{\\pi\}\\bigr\)\}\{\\sum\_\{a\\in\\mathcal\{A\}\}w\_\{a\}\\widetilde\{T\}\_\{a\}^\{\\pi\}\}\.The return\-time comparison above implies

∑a∈𝒜wa​T~aπ≤∑a∈𝒜wa​Taσ\.\\sum\_\{a\\in\\mathcal\{A\}\}w\_\{a\}\\widetilde\{T\}\_\{a\}^\{\\pi\}\\leq\\sum\_\{a\\in\\mathcal\{A\}\}w\_\{a\}T\_\{a\}^\{\\sigma\}\.On the other hand, applying \([29](https://arxiv.org/html/2608.06545#A3.E29)\) to each numerator term gives

∑a∈𝒜wa​\(ρϕσ​T~aπ−R~aπ\)≥∑a∈𝒜wa​Taσ​\(ρϕσ−ρaσ\)=∑a≠ϕwa​Taσ​\(ρϕσ−ρaσ\)\.\\sum\_\{a\\in\\mathcal\{A\}\}w\_\{a\}\\bigl\(\\rho\_\{\\phi\}^\{\\sigma\}\\widetilde\{T\}\_\{a\}^\{\\pi\}\-\\widetilde\{R\}\_\{a\}^\{\\pi\}\\bigr\)\\geq\\sum\_\{a\\in\\mathcal\{A\}\}w\_\{a\}T\_\{a\}^\{\\sigma\}\\bigl\(\\rho\_\{\\phi\}^\{\\sigma\}\-\\rho\_\{a\}^\{\\sigma\}\\bigr\)=\\sum\_\{a\\neq\\phi\}w\_\{a\}T\_\{a\}^\{\\sigma\}\\bigl\(\\rho\_\{\\phi\}^\{\\sigma\}\-\\rho\_\{a\}^\{\\sigma\}\\bigr\)\.The last sum is nonnegative by \([28](https://arxiv.org/html/2608.06545#A3.E28)\)\. We may therefore combine the numerator lower bound with the denominator upper bound to obtain

ρϕσ−ρ¯π≥∑a≠ϕwa​Taσ​\(ρϕσ−ρaσ\)∑a∈𝒜wa​Taσ\.\\rho\_\{\\phi\}^\{\\sigma\}\-\\bar\{\\rho\}^\{\\pi\}\\geq\\frac\{\\sum\_\{a\\neq\\phi\}w\_\{a\}T\_\{a\}^\{\\sigma\}\\bigl\(\\rho\_\{\\phi\}^\{\\sigma\}\-\\rho\_\{a\}^\{\\sigma\}\\bigr\)\}\{\\sum\_\{a\\in\\mathcal\{A\}\}w\_\{a\}T\_\{a\}^\{\\sigma\}\}\.\(30\)
By Lemma[4](https://arxiv.org/html/2608.06545#Thmlemma4), the robust average reward ofπϕ\\pi\_\{\\phi\}isρϕσ\\rho\_\{\\phi\}^\{\\sigma\}\. On the other hand, \([30](https://arxiv.org/html/2608.06545#A3.E30)\) shows thatP¯ϕ\\bar\{P\}^\{\\phi\}gives every stationary policy average reward at mostρϕσ\\rho\_\{\\phi\}^\{\\sigma\}\. Therefore

ρ⋆,σ=ρϕσ\.\\rho^\{\\star,\\sigma\}=\\rho\_\{\\phi\}^\{\\sigma\}\.BecauseP¯ϕ∈𝒫\\bar\{P\}^\{\\phi\}\\in\\mathcal\{P\}, we also haveρπ,σ≤ρ¯π\\rho^\{\\pi,\\sigma\}\\leq\\bar\{\\rho\}^\{\\pi\}\. Hence

ρ⋆,σ−ρπ,σ≥∑a≠ϕwa​Taσ​\(ρϕσ−ρaσ\)∑a∈𝒜wa​Taσ\.\\rho^\{\\star,\\sigma\}\-\\rho^\{\\pi,\\sigma\}\\geq\\frac\{\\sum\_\{a\\neq\\phi\}w\_\{a\}T\_\{a\}^\{\\sigma\}\\bigl\(\\rho\_\{\\phi\}^\{\\sigma\}\-\\rho\_\{a\}^\{\\sigma\}\\bigr\)\}\{\\sum\_\{a\\in\\mathcal\{A\}\}w\_\{a\}T\_\{a\}^\{\\sigma\}\}\.\(31\)
Multiplying \([28](https://arxiv.org/html/2608.06545#A3.E28)\) bywaw\_\{a\}and summing overa≠ϕa\\neq\\phigives

∑a≠ϕwa​Taσ​\(ρϕσ−ρaσ\)\>ε​\(\(1−wϕ\)​Tϕσ\+∑a≠ϕwa​Taσ\)\.\\sum\_\{a\\neq\\phi\}w\_\{a\}T\_\{a\}^\{\\sigma\}\\bigl\(\\rho\_\{\\phi\}^\{\\sigma\}\-\\rho\_\{a\}^\{\\sigma\}\\bigr\)\>\\varepsilon\\left\(\(1\-w\_\{\\phi\}\)T\_\{\\phi\}^\{\\sigma\}\+\\sum\_\{a\\neq\\phi\}w\_\{a\}T\_\{a\}^\{\\sigma\}\\right\)\.Ifwϕ≤1/2w\_\{\\phi\}\\leq 1/2, then1−wϕ≥wϕ1\-w\_\{\\phi\}\\geq w\_\{\\phi\}, so the right\-hand side is at least

ε​∑a∈𝒜wa​Taσ\.\\varepsilon\\sum\_\{a\\in\\mathcal\{A\}\}w\_\{a\}T\_\{a\}^\{\\sigma\}\.Together with \([31](https://arxiv.org/html/2608.06545#A3.E31)\), this proves

wϕ=π​\(ϕ∣S\)≤12⟹ρ⋆,σ−ρπ,σ\>ε\.w\_\{\\phi\}=\\pi\(\\phi\\mid S\)\\leq\\frac\{1\}\{2\}\\quad\\Longrightarrow\\quad\\rho^\{\\star,\\sigma\}\-\\rho^\{\\pi,\\sigma\}\>\\varepsilon\.Finally, we apply this pointwise implication to a policy estimator\. Letπ^\\widehat\{\\pi\}be any policy estimator and define its induced test by

ϕ^∈arg⁡maxi∈\{2,3\}⁡π^​\(i∣S\),\\widehat\{\\phi\}\\in\\arg\\max\_\{i\\in\\\{2,3\\\}\}\\widehat\{\\pi\}\(i\\mid S\),where ties are broken arbitrarily\. Ifϕ^≠ϕ\\widehat\{\\phi\}\\neq\\phi, thenπ^​\(ϕ∣S\)≤1/2\\widehat\{\\pi\}\(\\phi\\mid S\)\\leq 1/2\. Applying the preceding implication to each realization ofπ^\\widehat\{\\pi\}gives

ℙℳϕ​\{ϕ^≠ϕ\}≤ℙℳϕ​\{ρ⋆,σ−ρπ^,σ\>ε\}\.\\mathbb\{P\}\_\{\\mathcal\{M\}\_\{\\phi\}\}\\\{\\widehat\{\\phi\}\\neq\\phi\\\}\\leq\\mathbb\{P\}\_\{\\mathcal\{M\}\_\{\\phi\}\}\\left\\\{\\rho^\{\\star,\\sigma\}\-\\rho^\{\\widehat\{\\pi\},\\sigma\}\>\\varepsilon\\right\\\}\.\(32\)Taking the maximum overϕ∈\{2,3\}\\phi\\in\\\{2,3\\\}and then the infimum over policy estimators gives

infπ^maxϕ∈\{2,3\}⁡ℙℳϕ​\{ρ⋆,σ−ρπ^,σ\>ε\}≥infϕ~maxϕ∈\{2,3\}⁡ℙℳϕ​\{ϕ~≠ϕ\}\.\\inf\_\{\\widehat\{\\pi\}\}\\max\_\{\\phi\\in\\\{2,3\\\}\}\\mathbb\{P\}\_\{\\mathcal\{M\}\_\{\\phi\}\}\\left\\\{\\rho^\{\\star,\\sigma\}\-\\rho^\{\\widehat\{\\pi\},\\sigma\}\>\\varepsilon\\right\\\}\\geq\\inf\_\{\\widetilde\{\\phi\}\}\\max\_\{\\phi\\in\\\{2,3\\\}\}\\mathbb\{P\}\_\{\\mathcal\{M\}\_\{\\phi\}\}\\\{\\widetilde\{\\phi\}\\neq\\phi\\\}\.\(33\)Thus it remains to lower\-bound the testing error on the right\-hand side of \([33](https://arxiv.org/html/2608.06545#A3.E33)\)\.

##### Step 2: Le Cam’s two\-point bound\.

LetP0,ϕP^\{0,\\phi\}denote the nominal transition kernel ofℳϕ\\mathcal\{M\}\_\{\\phi\}, and let

ℚϕ\(N\)≔⨂\(s,a\)∈𝒮×𝒜\(Ps,a0,ϕ\)⊗N\\mathbb\{Q\}\_\{\\phi\}^\{\(N\)\}\\coloneqq\\bigotimes\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}\\left\(P\_\{s,a\}^\{0,\\phi\}\\right\)^\{\\otimes N\}\(34\)be the joint law of theNNtransition samples from every state\-action pair\. Additivity of KL divergence gives

KL​\(ℚ2\(N\)∥ℚ3\(N\)\)=N​∑\(s,a\)∈𝒮×𝒜KL​\(Ps,a0,2∥Ps,a0,3\)\.\\mathrm\{KL\}\\\!\\left\(\\mathbb\{Q\}\_\{2\}^\{\(N\)\}\\,\\\|\\,\\mathbb\{Q\}\_\{3\}^\{\(N\)\}\\right\)=N\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}\\mathrm\{KL\}\\\!\\left\(P\_\{s,a\}^\{0,2\}\\,\\\|\\,P\_\{s,a\}^\{0,3\}\\right\)\.\(35\)Le Cam’s two\-point method and Pinsker’s inequality\(Wainwright,[2019](https://arxiv.org/html/2608.06545#bib.bib10), see, e\.g\., \(15\.13\) and Lemma 15\.2\), followed by \([35](https://arxiv.org/html/2608.06545#A3.E35)\), yield

infϕ~maxϕ∈\{2,3\}⁡ℙℳϕ​\{ϕ~≠ϕ\}\\displaystyle\\inf\_\{\\widetilde\{\\phi\}\}\\max\_\{\\phi\\in\\\{2,3\\\}\}\\mathbb\{P\}\_\{\\mathcal\{M\}\_\{\\phi\}\}\\\{\\widetilde\{\\phi\}\\neq\\phi\\\}≥12−12​N2​∑\(s,a\)∈𝒮×𝒜KL​\(Ps,a0,2∥Ps,a0,3\)\.\\displaystyle\\geq\\frac\{1\}\{2\}\-\\frac\{1\}\{2\}\\sqrt\{\\frac\{N\}\{2\}\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}\\mathrm\{KL\}\\\!\\left\(P\_\{s,a\}^\{0,2\}\\,\\\|\\,P\_\{s,a\}^\{0,3\}\\right\)\}\.\(36\)Consequently, if

N​∑\(s,a\)∈𝒮×𝒜KL​\(Ps,a0,2∥Ps,a0,3\)≤14,N\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}\\mathrm\{KL\}\\\!\\left\(P\_\{s,a\}^\{0,2\}\\,\\\|\\,P\_\{s,a\}^\{0,3\}\\right\)\\leq\\frac\{1\}\{4\},\(37\)then the right\-hand side of \([36](https://arxiv.org/html/2608.06545#A3.E36)\) is at least

12−12​18\>14\.\\frac\{1\}\{2\}\-\\frac\{1\}\{2\}\\sqrt\{\\frac\{1\}\{8\}\}\>\\frac\{1\}\{4\}\.Combining this with \([33](https://arxiv.org/html/2608.06545#A3.E33)\) gives

infπ^maxϕ∈\{2,3\}⁡ℙℳϕ​\{ρ⋆,σ−ρπ^,σ\>ε\}≥14\.\\inf\_\{\\widehat\{\\pi\}\}\\max\_\{\\phi\\in\\\{2,3\\\}\}\\mathbb\{P\}\_\{\\mathcal\{M\}\_\{\\phi\}\}\\left\\\{\\rho^\{\\star,\\sigma\}\-\\rho^\{\\widehat\{\\pi\},\\sigma\}\>\\varepsilon\\right\\\}\\geq\\frac\{1\}\{4\}\.\(38\)To verify \([37](https://arxiv.org/html/2608.06545#A3.E37)\) for each construction, we next compute the KL divergence between the two nominal kernels\.

##### Step 3: KL divergence of the two changed rows\.

For each of the three parameter choices below, the nominal kernels differ only at\(S,2\)\(S,2\)and\(S,3\)\(S,3\)\. On these rows, the two instances exchange Bernoulli reward\-transition probabilitiesbbandb\+δb\+\\delta\. Therefore

∑\(s,a\)∈𝒮×𝒜KL​\(Ps,a0,2∥Ps,a0,3\)\\displaystyle\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}\\mathrm\{KL\}\\\!\\left\(P\_\{s,a\}^\{0,2\}\\,\\\|\\,P\_\{s,a\}^\{0,3\}\\right\)=KL​\(Bern​\(b\+δ\)∥Bern​\(b\)\)\+KL​\(Bern​\(b\)∥Bern​\(b\+δ\)\)\\displaystyle\\quad=\\mathrm\{KL\}\\bigl\(\\mathrm\{Bern\}\(b\+\\delta\)\\\|\\mathrm\{Bern\}\(b\)\\bigr\)\+\\mathrm\{KL\}\\bigl\(\\mathrm\{Bern\}\(b\)\\\|\\mathrm\{Bern\}\(b\+\\delta\)\\bigr\)=δ​log⁡\(\(b\+δ\)​\(1−b\)b​\(1−b−δ\)\)=δ​∫bb\+δ1t​\(1−t\)​dt\.\\displaystyle\\quad=\\delta\\log\\left\(\\frac\{\(b\+\\delta\)\(1\-b\)\}\{b\(1\-b\-\\delta\)\}\\right\)=\\delta\\int\_\{b\}^\{b\+\\delta\}\\frac\{1\}\{t\(1\-t\)\}\\,\\mathrm\{d\}t\.\(39\)This completes the common testing argument\.

#### C\.2\.2min⁡\{H0,Hσ\}\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}component

To establish themin⁡\{H0,Hσ\}\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}component, we specify an instance pair, verify the testing and span conditions, and then bound the KL divergence between the two instances\.

##### Parameter choice and resulting quantities\.

We use a zero\-radius subfamily of the instance template\. Choose

p0=1min⁡\{H0,Hσ\},q0=2min⁡\{H0,Hσ\},σ0=0,p\_\{0\}=\\frac\{1\}\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\},\\qquad q\_\{0\}=\\frac\{2\}\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\},\\qquad\\sigma\_\{0\}=0,p1=2min⁡\{H0,Hσ\},q1=2min⁡\{H0,Hσ\},σ1=0,p\_\{1\}=\\frac\{2\}\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\},\\qquad q\_\{1\}=\\frac\{2\}\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\},\\qquad\\sigma\_\{1\}=0,and

δ=40​εmin⁡\{H0,Hσ\},r0=r1=1\.\\delta=\\frac\{40\\varepsilon\}\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\},\\qquad r\_\{0\}=r\_\{1\}=1\.These parameters are valid becausemin⁡\{H0,Hσ\}≥4\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\\geq 4andε≤1/100\\varepsilon\\leq 1/100\. Indeed,

p0≤14,q0=p1=q1≤12,p1\+δ≤35\.p\_\{0\}\\leq\\frac\{1\}\{4\},\\qquad q\_\{0\}=p\_\{1\}=q\_\{1\}\\leq\\frac\{1\}\{2\},\\qquad p\_\{1\}\+\\delta\\leq\\frac\{3\}\{5\}\.All local uncertainty radii are zero, so the nominal and robust Bellman equations coincide\. The unichain argument in Appendix[C\.1](https://arxiv.org/html/2608.06545#A3.SS1)applies to every stationary policy\.

Substituting these parameters into Lemma[4](https://arxiv.org/html/2608.06545#Thmlemma4)gives

ρ1σ=13,ρ5−ϕσ=12,ρϕσ=2\+40​ε4\+40​ε,\\rho\_\{1\}^\{\\sigma\}=\\frac\{1\}\{3\},\\qquad\\rho\_\{5\-\\phi\}^\{\\sigma\}=\\frac\{1\}\{2\},\\qquad\\rho\_\{\\phi\}^\{\\sigma\}=\\frac\{2\+40\\varepsilon\}\{4\+40\\varepsilon\},and

T1σ=32,T5−ϕσ=2,Tϕσ=2\+20​ε\.T\_\{1\}^\{\\sigma\}=\\frac\{3\}\{2\},\\qquad T\_\{5\-\\phi\}^\{\\sigma\}=2,\\qquad T\_\{\\phi\}^\{\\sigma\}=2\+20\\varepsilon\.

##### Verification of the testing conditions\.

We now check the conditions needed to apply the common testing argument\. Becauser0=r1=1r\_\{0\}=r\_\{1\}=1, the reward\-dominance condition \([27](https://arxiv.org/html/2608.06545#A3.E27)\) holds\. For the gap to action5−ϕ5\-\\phi,

T5−ϕσTϕσ\+T5−ϕσ​\(ρϕσ−ρ5−ϕσ\)=5​ε\(1\+10​ε\)​\(2\+10​ε\)\>ε\.\\frac\{T\_\{5\-\\phi\}^\{\\sigma\}\}\{T\_\{\\phi\}^\{\\sigma\}\+T\_\{5\-\\phi\}^\{\\sigma\}\}\\bigl\(\\rho\_\{\\phi\}^\{\\sigma\}\-\\rho\_\{5\-\\phi\}^\{\\sigma\}\\bigr\)=\\frac\{5\\varepsilon\}\{\(1\+10\\varepsilon\)\(2\+10\\varepsilon\)\}\>\\varepsilon\.For action11,

T1σTϕσ\+T1σ​\(ρϕσ−ρ1σ\)≥574\>ε,\\frac\{T\_\{1\}^\{\\sigma\}\}\{T\_\{\\phi\}^\{\\sigma\}\+T\_\{1\}^\{\\sigma\}\}\\bigl\(\\rho\_\{\\phi\}^\{\\sigma\}\-\\rho\_\{1\}^\{\\sigma\}\\bigr\)\\geq\\frac\{5\}\{74\}\>\\varepsilon,and fora∉\{1,2,3\}a\\notin\\\{1,2,3\\\},

TaσTϕσ\+Taσ​\(ρϕσ−ρaσ\)≥532\>ε\.\\frac\{T\_\{a\}^\{\\sigma\}\}\{T\_\{\\phi\}^\{\\sigma\}\+T\_\{a\}^\{\\sigma\}\}\\bigl\(\\rho\_\{\\phi\}^\{\\sigma\}\-\\rho\_\{a\}^\{\\sigma\}\\bigr\)\\geq\\frac\{5\}\{32\}\>\\varepsilon\.Hence \([28](https://arxiv.org/html/2608.06545#A3.E28)\) also holds\.

##### Span constraints and class membership\.

We next verify that both instances belong to the class in the theorem\. Seth​\(S\)=0h\(S\)=0and let

h​\(1\)=h​\(2\)=h​\(3\)=min⁡\{H0,Hσ\}​\(1−ρϕσ\)2,h​\(s\)=−ρϕσ,s∈\{4,…,S−1\}\.h\(1\)=h\(2\)=h\(3\)=\\frac\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\(1\-\\rho\_\{\\phi\}^\{\\sigma\}\)\}\{2\},\\qquad h\(s\)=\-\\rho\_\{\\phi\}^\{\\sigma\},\\quad s\\in\\\{4,\\ldots,S\-1\\\}\.We now check thathhsatisfies both Bellman equations and has the required span\. At every reward statei∈\{1,2,3\}i\\in\\\{1,2,3\\\}, actiona=ia=isatisfies

ρϕσ\+h​\(i\)=1\+\(1−2min⁡\{H0,Hσ\}\)​h​\(i\)\.\\rho\_\{\\phi\}^\{\\sigma\}\+h\(i\)=1\+\\left\(1\-\\frac\{2\}\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\}\\right\)h\(i\)\.This action weakly dominates every action that returns immediately toSSbecauseh​\(i\)≥0h\(i\)\\geq 0\. At the decision state,

\(PS,ϕ0\)⊤​h=2\+40​εmin⁡\{H0,Hσ\}⋅min⁡\{H0,Hσ\}​\(1−ρϕσ\)2=ρϕσ\.\(P^\{0\}\_\{S,\\phi\}\)^\{\\top\}h=\\frac\{2\+40\\varepsilon\}\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\}\\cdot\\frac\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\(1\-\\rho\_\{\\phi\}^\{\\sigma\}\)\}\{2\}=\\rho\_\{\\phi\}^\{\\sigma\}\.The transition probabilities multiplying the same positive reward\-state coordinate are smaller for actions11and5−ϕ5\-\\phi, while every remaining action has value zero\. Thus actionϕ\\phimaximizes atSS\. Finally, fors∈\{4,…,S−1\}s\\in\\\{4,\\ldots,S\-1\\\},

ρϕσ\+h​\(s\)=0=\(Ps,a0\)⊤​h,a∈𝒜\.\\rho\_\{\\phi\}^\{\\sigma\}\+h\(s\)=0=\(P^\{0\}\_\{s,a\}\)^\{\\top\}h,\\qquad a\\in\\mathcal\{A\}\.Consequently,ρϕσ​𝟏\+h=𝒯0​h\\rho\_\{\\phi\}^\{\\sigma\}\\mathbf\{1\}\+h=\\mathcal\{T\}\_\{0\}h\. Since every uncertainty radius is zero, the same equation holds for𝒯σ\\mathcal\{T\}\_\{\\sigma\}\. Moreover,

‖h‖span\\displaystyle\\\|h\\\|\_\{\\mathrm\{span\}\}=min⁡\{H0,Hσ\}​\(1−ρϕσ\)2\+ρϕσ\\displaystyle=\\frac\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\(1\-\\rho\_\{\\phi\}^\{\\sigma\}\)\}\{2\}\+\\rho\_\{\\phi\}^\{\\sigma\}=min⁡\{H0,Hσ\}2−\(min⁡\{H0,Hσ\}2−1\)​ρϕσ\\displaystyle=\\frac\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\}\{2\}\-\\left\(\\frac\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\}\{2\}\-1\\right\)\\rho\_\{\\phi\}^\{\\sigma\}≤min⁡\{H0,Hσ\}2≤min⁡\{H0,Hσ\}\.\\displaystyle\\leq\\frac\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\}\{2\}\\leq\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\.Hence

ℳϕ∈𝔐​\(H0,Hσ,σ\),ϕ∈\{2,3\}\.\\mathcal\{M\}\_\{\\phi\}\\in\\mathfrak\{M\}\(H\_\{0\},H\_\{\\sigma\},\\sigma\),\\qquad\\phi\\in\\\{2,3\\\}\.\(40\)

##### KL divergence and testing conclusion\.

It remains to bound the statistical distance between the two instances and apply the common testing argument\. The two nominal kernels differ only at\(S,2\)\(S,2\)and\(S,3\)\(S,3\), where they exchange Bernoulli parameters2/min⁡\{H0,Hσ\}2/\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}and\(2\+40​ε\)/min⁡\{H0,Hσ\}\(2\+40\\varepsilon\)/\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\. Throughout the integration interval in \([39](https://arxiv.org/html/2608.06545#A3.E39)\),

t≥2min⁡\{H0,Hσ\},1−t≥25\.t\\geq\\frac\{2\}\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\},\\qquad 1\-t\\geq\\frac\{2\}\{5\}\.Consequently, \([39](https://arxiv.org/html/2608.06545#A3.E39)\) gives

∑\(s,a\)∈𝒮×𝒜KL​\(Ps,a0,2∥Ps,a0,3\)≤2000​ε2min⁡\{H0,Hσ\}\.\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}\\mathrm\{KL\}\\\!\\left\(P\_\{s,a\}^\{0,2\}\\,\\\|\\,P\_\{s,a\}^\{0,3\}\\right\)\\leq\\frac\{2000\\varepsilon^\{2\}\}\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\}\.\(41\)If

N≤min⁡\{H0,Hσ\}8000​ε2,N\\leq\\frac\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\}\{8000\\varepsilon^\{2\}\},then \([41](https://arxiv.org/html/2608.06545#A3.E41)\) implies \([37](https://arxiv.org/html/2608.06545#A3.E37)\), which then gives

infπ^maxϕ∈\{2,3\}⁡ℙℳϕ​\{ρ⋆,σ−ρπ^,σ\>ε\}≥14\.\\inf\_\{\\widehat\{\\pi\}\}\\max\_\{\\phi\\in\\\{2,3\\\}\}\\mathbb\{P\}\_\{\\mathcal\{M\}\_\{\\phi\}\}\\left\\\{\\rho^\{\\star,\\sigma\}\-\\rho^\{\\widehat\{\\pi\},\\sigma\}\>\\varepsilon\\right\\\}\\geq\\frac\{1\}\{4\}\.\(42\)This proves themin⁡\{H0,Hσ\}\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}component in Theorem[1](https://arxiv.org/html/2608.06545#Thmtheorem1)\.

#### C\.2\.3Caseσ<4/min⁡\{H0,Hσ\}\\sigma<4/\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}for theσ​Hσ2\\sigma H\_\{\\sigma\}^\{2\}component

For the caseσ<4/min⁡\{H0,Hσ\}\\sigma<4/\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}, we employ the generic testing argument to establish theσ​Hσ2\\sigma H\_\{\\sigma\}^\{2\}component of the lower bound\. We specify an instance pair, verify the testing and span conditions, and then bound the KL divergence between the two instances\.

##### Parameter choice and resulting quantities\.

Recall that we assume

σ≥100​εmin⁡\{H0,Hσ\}\.\\sigma\\geq\\frac\{100\\varepsilon\}\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\}\.Choose

p0=σ5,q0=2​σ5,σ0=σ,p\_\{0\}=\\frac\{\\sigma\}\{5\},\\qquad q\_\{0\}=\\frac\{2\\sigma\}\{5\},\\qquad\\sigma\_\{0\}=\\sigma,p1=q1=σ​min⁡\{H0,Hσ\}2​Hσ,σ1=σ4,p\_\{1\}=q\_\{1\}=\\frac\{\\sigma\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\}\{2H\_\{\\sigma\}\},\\qquad\\sigma\_\{1\}=\\frac\{\\sigma\}\{4\},and

δ=100​εHσ,r0=σ​min⁡\{H0,Hσ\}4,r1=3​σ​min⁡\{H0,Hσ\}16\.\\delta=\\frac\{100\\varepsilon\}\{H\_\{\\sigma\}\},\\qquad r\_\{0\}=\\frac\{\\sigma\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\}\{4\},\\qquad r\_\{1\}=\\frac\{3\\sigma\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\}\{16\}\.Sincemin⁡\{H0,Hσ\}≤Hσ\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\\leq H\_\{\\sigma\}and \([25](https://arxiv.org/html/2608.06545#A3.E25)\) holds,p1≤σ/2p\_\{1\}\\leq\\sigma/2andδ≤σ\\delta\\leq\\sigma\. Thusσ1\+p1\+δ≤σ/4\+σ/2\+σ≤7/8<1\\sigma\_\{1\}\+p\_\{1\}\+\\delta\\leq\\sigma/4\+\\sigma/2\+\\sigma\\leq 7/8<1; alsoσ0\+p0=6​σ/5≤3/5<1\\sigma\_\{0\}\+p\_\{0\}=6\\sigma/5\\leq 3/5<1\. In addition,r0<1r\_\{0\}<1andr1<3/4r\_\{1\}<3/4becauseσ​min⁡\{H0,Hσ\}<4\\sigma\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}<4\.

Substitution into Lemma[4](https://arxiv.org/html/2608.06545#Thmlemma4)gives

ρϕσ\>ρ5−ϕσ=3​σ​min⁡\{H0,Hσ\}16​\(2\+σ/4\)\>σ​min⁡\{H0,Hσ\}4​\(3\+2​σ\)=ρ1σ\.\\rho\_\{\\phi\}^\{\\sigma\}\>\\rho\_\{5\-\\phi\}^\{\\sigma\}=\\frac\{3\\sigma\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\}\{16\(2\+\\sigma/4\)\}\>\\frac\{\\sigma\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\}\{4\(3\+2\\sigma\)\}=\\rho\_\{1\}^\{\\sigma\}\.\(43\)After cross\-multiplication, the first inequality reduces toδ​q1​\(1\+σ1\)​r1\>0\\delta q\_\{1\}\(1\+\\sigma\_\{1\}\)r\_\{1\}\>0, and the second reduces to4\+20​σ\>04\+20\\sigma\>0\. Moreover,

ρϕσ<r1<r0\.\\rho\_\{\\phi\}^\{\\sigma\}<r\_\{1\}<r\_\{0\}\.\(44\)

##### Verification of the testing conditions\.

Equation \([44](https://arxiv.org/html/2608.06545#A3.E44)\) verifies the reward\-dominance condition \([27](https://arxiv.org/html/2608.06545#A3.E27)\)\. The following lemma verifies the gap condition \([28](https://arxiv.org/html/2608.06545#A3.E28)\); its proof is deferred to Appendix[C\.3\.2](https://arxiv.org/html/2608.06545#A3.SS3.SSS2)\.

###### Lemma 5\.

For the instances constructed in the caseσ<4/min⁡\{H0,Hσ\}\\sigma<4/\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}, the gap condition \([28](https://arxiv.org/html/2608.06545#A3.E28)\) holds\.

##### Span constraints and class membership\.

The following lemma provides the nominal and robust span bounds; its proof is deferred to Appendix[C\.3\.3](https://arxiv.org/html/2608.06545#A3.SS3.SSS3)\.

###### Lemma 6\.

For the instances constructed in the caseσ<4/min⁡\{H0,Hσ\}\\sigma<4/\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\},‖hP0⋆‖span≤H0\\\|h\_\{P^\{0\}\}^\{\\star\}\\\|\_\{\\mathrm\{span\}\}\\leq H\_\{0\}and‖h⋆,σ‖span≤Hσ\\\|h^\{\\star,\\sigma\}\\\|\_\{\\mathrm\{span\}\}\\leq H\_\{\\sigma\}\.

The parameter checks give bounded rewards and local TV radii at mostσ\\sigma, the unichain argument in Appendix[C\.1](https://arxiv.org/html/2608.06545#A3.SS1)verifies Assumption[1](https://arxiv.org/html/2608.06545#Thmassumption1), and Lemma[6](https://arxiv.org/html/2608.06545#Thmlemma6)gives the nominal and robust span bounds\. Hence

ℳϕ∈𝔐​\(H0,Hσ,σ\),ϕ∈\{2,3\}\.\\mathcal\{M\}\_\{\\phi\}\\in\\mathfrak\{M\}\(H\_\{0\},H\_\{\\sigma\},\\sigma\),\\qquad\\phi\\in\\\{2,3\\\}\.\(45\)

##### KL divergence and testing conclusion\.

It remains to bound the statistical distance between the two instances and apply the testing reduction in Appendix[C\.2\.1](https://arxiv.org/html/2608.06545#A3.SS2.SSS1)\. Here the lower Bernoulli parameter in \([39](https://arxiv.org/html/2608.06545#A3.E39)\) isσ/4\+p1\\sigma/4\+p\_\{1\}\. The parameter bounds give

2​\(σ4\+p1\)\+δ≤34\+14=1,2\\left\(\\frac\{\\sigma\}\{4\}\+p\_\{1\}\\right\)\+\\delta\\leq\\frac\{3\}\{4\}\+\\frac\{1\}\{4\}=1,where we usedp1≤σ/2p\_\{1\}\\leq\\sigma/2,σ≤1/2\\sigma\\leq 1/2, andδ≤1/Hσ≤1/4\\delta\\leq 1/H\_\{\\sigma\}\\leq 1/4\. Equivalently,

σ4\+p1\+δ≤1−σ4−p1\.\\frac\{\\sigma\}\{4\}\+p\_\{1\}\+\\delta\\leq 1\-\\frac\{\\sigma\}\{4\}\-p\_\{1\}\.Hence the integration interval in \([39](https://arxiv.org/html/2608.06545#A3.E39)\) is contained in\[σ/4\+p1,1−σ/4−p1\]\[\\sigma/4\+p\_\{1\},1\-\\sigma/4\-p\_\{1\}\]\. Thereforet​\(1−t\)t\(1\-t\)is at least\(σ/4\+p1\)​\(1−σ/4−p1\)\(\\sigma/4\+p\_\{1\}\)\(1\-\\sigma/4\-p\_\{1\}\)throughout the integration interval in \([39](https://arxiv.org/html/2608.06545#A3.E39)\)\. Consequently,

∑\(s,a\)∈𝒮×𝒜KL​\(Ps,a0,2∥Ps,a0,3\)≤δ2\(σ/4\+p1\)​\(1−σ/4−p1\)≤64000​ε2σ​Hσ2\.\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}\\mathrm\{KL\}\\\!\\left\(P\_\{s,a\}^\{0,2\}\\,\\\|\\,P\_\{s,a\}^\{0,3\}\\right\)\\leq\\frac\{\\delta^\{2\}\}\{\(\\sigma/4\+p\_\{1\}\)\(1\-\\sigma/4\-p\_\{1\}\)\}\\leq\\frac\{64000\\varepsilon^\{2\}\}\{\\sigma H\_\{\\sigma\}^\{2\}\}\.\(46\)The last inequality usesσ/4\+p1≥σ/4\\sigma/4\+p\_\{1\}\\geq\\sigma/4and1−σ/4−p1≥1−1/8−1/4=5/81\-\\sigma/4\-p\_\{1\}\\geq 1\-1/8\-1/4=5/8\.

By \([46](https://arxiv.org/html/2608.06545#A3.E46)\), the sample\-size condition \([37](https://arxiv.org/html/2608.06545#A3.E37)\) holds whenever

N≤σ​Hσ2256000​ε2\.N\\leq\\frac\{\\sigma H\_\{\\sigma\}^\{2\}\}\{256000\\varepsilon^\{2\}\}\.Under this sample\-size bound, \([38](https://arxiv.org/html/2608.06545#A3.E38)\) gives

infπ^maxϕ∈\{2,3\}⁡ℙℳϕ​\{ρ⋆,σ−ρπ^,σ\>ε\}≥14\.\\inf\_\{\\widehat\{\\pi\}\}\\max\_\{\\phi\\in\\\{2,3\\\}\}\\mathbb\{P\}\_\{\\mathcal\{M\}\_\{\\phi\}\}\\left\\\{\\rho^\{\\star,\\sigma\}\-\\rho^\{\\widehat\{\\pi\},\\sigma\}\>\\varepsilon\\right\\\}\\geq\\frac\{1\}\{4\}\.\(47\)This proves theσ​Hσ2\\sigma H\_\{\\sigma\}^\{2\}component whenσ<4/min⁡\{H0,Hσ\}\\sigma<4/\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\.

#### C\.2\.4Caseσ≥4/min⁡\{H0,Hσ\}\\sigma\\geq 4/\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}for theσ​Hσ2\\sigma H\_\{\\sigma\}^\{2\}component

For the caseσ≥4/min⁡\{H0,Hσ\}\\sigma\\geq 4/\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}, we employ the generic testing argument to establish theσ​Hσ2\\sigma H\_\{\\sigma\}^\{2\}component of the lower bound\. We specify an instance pair, verify the testing and span conditions, and then bound the KL divergence between the two instances\.

##### Parameter choice and resulting quantities\.

Choose

p0=1min⁡\{H0,Hσ\},q0=3min⁡\{H0,Hσ\},σ0=0,p\_\{0\}=\\frac\{1\}\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\},\\qquad q\_\{0\}=\\frac\{3\}\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\},\\qquad\\sigma\_\{0\}=0,p1=q1=2Hσ,σ1=σ,p\_\{1\}=q\_\{1\}=\\frac\{2\}\{H\_\{\\sigma\}\},\\qquad\\sigma\_\{1\}=\\sigma,and

δ=40​εHσ,r0=r1=1\.\\delta=\\frac\{40\\varepsilon\}\{H\_\{\\sigma\}\},\\qquad r\_\{0\}=r\_\{1\}=1\.These parameters are valid\. Indeed,min⁡\{H0,Hσ\}≥4\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\\geq 4, the case condition, and \([25](https://arxiv.org/html/2608.06545#A3.E25)\) give1/min⁡\{H0,Hσ\}≤1/41/\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\\leq 1/4,3/min⁡\{H0,Hσ\}≤3/43/\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\\leq 3/4,2/Hσ≤σ/22/H\_\{\\sigma\}\\leq\\sigma/2, and40​ε/Hσ≤0\.4​σ40\\varepsilon/H\_\{\\sigma\}\\leq 0\.4\\sigma, soσ1\+p1\+δ≤1\.9​σ≤0\.95<1\\sigma\_\{1\}\+p\_\{1\}\+\\delta\\leq 1\.9\\sigma\\leq 0\.95<1\.

Substituting these parameters into Lemma[4](https://arxiv.org/html/2608.06545#Thmlemma4)gives

ρϕσ=2\+40​ε4\+2​σ\+40​ε\>12\+σ=ρ5−ϕσ≥25\>14=ρ1σ\.\\rho\_\{\\phi\}^\{\\sigma\}=\\frac\{2\+40\\varepsilon\}\{4\+2\\sigma\+40\\varepsilon\}\>\\frac\{1\}\{2\+\\sigma\}=\\rho\_\{5\-\\phi\}^\{\\sigma\}\\geq\\frac\{2\}\{5\}\>\\frac\{1\}\{4\}=\\rho\_\{1\}^\{\\sigma\}\.\(48\)The strict inequality follows fromδ\>0\\delta\>0, and the remaining inequalities useσ≤1/2\\sigma\\leq 1/2\. In addition,

ρϕσ<1=r0=r1\.\\rho\_\{\\phi\}^\{\\sigma\}<1=r\_\{0\}=r\_\{1\}\.\(49\)

##### Verification of the testing conditions\.

Equation \([49](https://arxiv.org/html/2608.06545#A3.E49)\) verifies the reward\-dominance condition \([27](https://arxiv.org/html/2608.06545#A3.E27)\)\. The following lemma verifies the gap condition \([28](https://arxiv.org/html/2608.06545#A3.E28)\); its proof is deferred to Appendix[C\.3\.2](https://arxiv.org/html/2608.06545#A3.SS3.SSS2)\.

###### Lemma 7\.

For the instances constructed in the caseσ≥4/min⁡\{H0,Hσ\}\\sigma\\geq 4/\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}, the gap condition \([28](https://arxiv.org/html/2608.06545#A3.E28)\) holds\.

##### Span constraints and class membership\.

The following lemma provides the nominal and robust span bounds; its proof is deferred to Appendix[C\.3\.3](https://arxiv.org/html/2608.06545#A3.SS3.SSS3)\.

###### Lemma 8\.

For the instances constructed in the caseσ≥4/min⁡\{H0,Hσ\}\\sigma\\geq 4/\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\},‖hP0⋆‖span≤H0\\\|h\_\{P^\{0\}\}^\{\\star\}\\\|\_\{\\mathrm\{span\}\}\\leq H\_\{0\}and‖h⋆,σ‖span≤Hσ\\\|h^\{\\star,\\sigma\}\\\|\_\{\\mathrm\{span\}\}\\leq H\_\{\\sigma\}\.

The parameter checks give bounded rewards and local TV radii at mostσ\\sigma, the unichain argument in Appendix[C\.1](https://arxiv.org/html/2608.06545#A3.SS1)verifies Assumption[1](https://arxiv.org/html/2608.06545#Thmassumption1), and Lemma[8](https://arxiv.org/html/2608.06545#Thmlemma8)gives the nominal and robust span bounds\. Hence

ℳϕ∈𝔐​\(H0,Hσ,σ\),ϕ∈\{2,3\}\.\\mathcal\{M\}\_\{\\phi\}\\in\\mathfrak\{M\}\(H\_\{0\},H\_\{\\sigma\},\\sigma\),\\qquad\\phi\\in\\\{2,3\\\}\.\(50\)

##### KL divergence and testing conclusion\.

It remains to bound the statistical distance between the two instances and apply the testing reduction in Appendix[C\.2\.1](https://arxiv.org/html/2608.06545#A3.SS2.SSS1)\. Here the integration interval in \([39](https://arxiv.org/html/2608.06545#A3.E39)\) runs fromσ\+p1\\sigma\+p\_\{1\}toσ\+p1\+δ\\sigma\+p\_\{1\}\+\\delta\. By the case condition, \([25](https://arxiv.org/html/2608.06545#A3.E25)\), andσ≤1/2\\sigma\\leq 1/2, we have

2Hσ≤σ2,40​εHσ≤2​σ5\.\\frac\{2\}\{H\_\{\\sigma\}\}\\leq\\frac\{\\sigma\}\{2\},\\qquad\\frac\{40\\varepsilon\}\{H\_\{\\sigma\}\}\\leq\\frac\{2\\sigma\}\{5\}\.Therefore

t≥σ,1−t≥1−σ−p1−δ≥1−σ−σ2−2​σ5≥120t\\geq\\sigma,\\qquad 1\-t\\geq 1\-\\sigma\-p\_\{1\}\-\\delta\\geq 1\-\\sigma\-\\frac\{\\sigma\}\{2\}\-\\frac\{2\\sigma\}\{5\}\\geq\\frac\{1\}\{20\}throughout this interval\. Applying \([39](https://arxiv.org/html/2608.06545#A3.E39)\) directly therefore gives

∑\(s,a\)∈𝒮×𝒜KL​\(Ps,a0,2∥Ps,a0,3\)≤20​δ2σ≤32000​ε2σ​Hσ2\.\\sum\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}\\mathrm\{KL\}\\\!\\left\(P\_\{s,a\}^\{0,2\}\\,\\\|\\,P\_\{s,a\}^\{0,3\}\\right\)\\leq\\frac\{20\\delta^\{2\}\}\{\\sigma\}\\leq\\frac\{32000\\varepsilon^\{2\}\}\{\\sigma H\_\{\\sigma\}^\{2\}\}\.\(51\)
By \([51](https://arxiv.org/html/2608.06545#A3.E51)\), the sample\-size condition \([37](https://arxiv.org/html/2608.06545#A3.E37)\) holds whenever

N≤σ​Hσ2128000​ε2\.N\\leq\\frac\{\\sigma H\_\{\\sigma\}^\{2\}\}\{128000\\varepsilon^\{2\}\}\.Under this sample\-size bound, \([38](https://arxiv.org/html/2608.06545#A3.E38)\) gives

infπ^maxϕ∈\{2,3\}⁡ℙℳϕ​\{ρ⋆,σ−ρπ^,σ\>ε\}≥14\.\\inf\_\{\\widehat\{\\pi\}\}\\max\_\{\\phi\\in\\\{2,3\\\}\}\\mathbb\{P\}\_\{\\mathcal\{M\}\_\{\\phi\}\}\\left\\\{\\rho^\{\\star,\\sigma\}\-\\rho^\{\\widehat\{\\pi\},\\sigma\}\>\\varepsilon\\right\\\}\\geq\\frac\{1\}\{4\}\.\(52\)This proves theσ​Hσ2\\sigma H\_\{\\sigma\}^\{2\}component whenσ≥4/min⁡\{H0,Hσ\}\\sigma\\geq 4/\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\.

### C\.3Proofs of auxiliary lemmas

#### C\.3\.1Proof of Lemma[4](https://arxiv.org/html/2608.06545#Thmlemma4)

##### Policyπ1\\pi\_\{1\}\.

We first analyze the policyπ1\\pi\_\{1\}\. We calculate its nominal and robust average rewards and then its expected time to return toSSunderP¯ϕ\\bar\{P\}^\{\\phi\}\.

###### Nominal average reward\.

Under the nominal kernel, the stationary distributionμ0\\mu^\{0\}is supported on\{S,1\}\\\{S,1\\\}and satisfies

\(σ0\+p0\)​μ0​\(S\)=q0​μ0​\(1\),μ0​\(S\)\+μ0​\(1\)=1\.\(\\sigma\_\{0\}\+p\_\{0\}\)\\mu^\{0\}\(S\)=q\_\{0\}\\mu^\{0\}\(1\),\\qquad\\mu^\{0\}\(S\)\+\\mu^\{0\}\(1\)=1\.Solving these equations gives

ρ1=r0​μ0​\(1\)=σ0\+p0σ0\+p0\+q0​r0\.\\rho\_\{1\}=r\_\{0\}\\mu^\{0\}\(1\)=\\frac\{\\sigma\_\{0\}\+p\_\{0\}\}\{\\sigma\_\{0\}\+p\_\{0\}\+q\_\{0\}\}\\,r\_\{0\}\.

###### Robust average reward\.

We next verify the robust formula\. We claim that

ρ1σ=p0p0\+q0​\(1\+σ0\)​r0\.\\rho\_\{1\}^\{\\sigma\}=\\frac\{p\_\{0\}\}\{p\_\{0\}\+q\_\{0\}\(1\+\\sigma\_\{0\}\)\}\\,r\_\{0\}\.To verify this claim, we construct a fixed\-policy bias for which the robust Bellman equation holds andP¯ϕ\\bar\{P\}^\{\\phi\}attains the minimizing transition rows\. Normalize this bias byh​\(S\)=0h\(S\)=0and define

h​\(1\)=r0−ρ1σq0,h​\(2\)=h​\(3\)=r1−ρ1σ,h\(1\)=\\frac\{r\_\{0\}\-\\rho\_\{1\}^\{\\sigma\}\}\{q\_\{0\}\},\\qquad h\(2\)=h\(3\)=r\_\{1\}\-\\rho\_\{1\}^\{\\sigma\},and

h​\(s\)=−ρ1σ,s∈\{4,…,S−1\}\.h\(s\)=\-\\rho\_\{1\}^\{\\sigma\},\\qquad s\\in\\\{4,\\ldots,S\-1\\\}\.Sinceρ1σ≤r0\\rho\_\{1\}^\{\\sigma\}\\leq r\_\{0\}and all rewards are nonnegative,

h​\(1\)≥h​\(S\)=0,mins∈𝒮⁡h​\(s\)=h​\(4\)=−ρ1σ\.h\(1\)\\geq h\(S\)=0,\\qquad\\min\_\{s\\in\\mathcal\{S\}\}h\(s\)=h\(4\)=\-\\rho\_\{1\}^\{\\sigma\}\.The nominal rowPS,10P^\{0\}\_\{S,1\}assigns mass only to states11andSS, withPS,10​\(1\)=σ0\+p0≥σ0P^\{0\}\_\{S,1\}\(1\)=\\sigma\_\{0\}\+p\_\{0\}\\geq\\sigma\_\{0\}\. Among these two states,h​\(1\)≥h​\(S\)h\(1\)\\geq h\(S\), and state44has a minimum coordinate and zero nominal mass\. It is therefore feasible to transfer massσ0\\sigma\_\{0\}from state11to state44, and no perturbation of TV distance at mostσ0\\sigma\_\{0\}can decrease the expectation further\. Thus

minQ∈𝒫S,1⁡Q⊤​h=\(PS,10\)⊤​h−σ0​\(h​\(1\)−h​\(4\)\)=\(P¯S,1ϕ\)⊤​h\.\\min\_\{Q\\in\\mathcal\{P\}\_\{S,1\}\}Q^\{\\top\}h=\(P^\{0\}\_\{S,1\}\)^\{\\top\}h\-\\sigma\_\{0\}\\bigl\(h\(1\)\-h\(4\)\\bigr\)=\(\\bar\{P\}\_\{S,1\}^\{\\phi\}\)^\{\\top\}h\.For this minimizing row,

\(P¯S,1ϕ\)⊤​h=p0​r0−ρ1σq0−σ0​ρ1σ=p0​r0−\(p0\+q0​σ0\)​ρ1σq0=ρ1σ,\(\\bar\{P\}\_\{S,1\}^\{\\phi\}\)^\{\\top\}h=p\_\{0\}\\frac\{r\_\{0\}\-\\rho\_\{1\}^\{\\sigma\}\}\{q\_\{0\}\}\-\\sigma\_\{0\}\\rho\_\{1\}^\{\\sigma\}=\\frac\{p\_\{0\}r\_\{0\}\-\(p\_\{0\}\+q\_\{0\}\\sigma\_\{0\}\)\\rho\_\{1\}^\{\\sigma\}\}\{q\_\{0\}\}=\\rho\_\{1\}^\{\\sigma\},where the last equality usesp0​r0=ρ1σ​\[p0\+q0​\(1\+σ0\)\]p\_\{0\}r\_\{0\}=\\rho\_\{1\}^\{\\sigma\}\[p\_\{0\}\+q\_\{0\}\(1\+\\sigma\_\{0\}\)\]\. At state11,

r0\+\(1−q0\)​h​\(1\)=ρ1σ\+h​\(1\)\.r\_\{0\}\+\(1\-q\_\{0\}\)h\(1\)=\\rho\_\{1\}^\{\\sigma\}\+h\(1\)\.Atj∈\{2,3\}j\\in\\\{2,3\\\}, policyπ1\\pi\_\{1\}returns directly toSS, and

r1\+h​\(S\)=ρ1σ\+h​\(j\)\.r\_\{1\}\+h\(S\)=\\rho\_\{1\}^\{\\sigma\}\+h\(j\)\.At each delay or padding state,

0\+h​\(S\)=ρ1σ\+h​\(s\)\.0\+h\(S\)=\\rho\_\{1\}^\{\\sigma\}\+h\(s\)\.Thus the fixed\-policy robust Bellman equation holds at every state, and the minimizing rows are those ofP¯ϕ\\bar\{P\}^\{\\phi\}:

ρ1σ​𝟏\+h=rπ1\+P¯π1ϕ​h\.\\rho\_\{1\}^\{\\sigma\}\\mathbf\{1\}\+h=r^\{\\pi\_\{1\}\}\+\\bar\{P\}^\{\\phi\}\_\{\\pi\_\{1\}\}h\.Multiplying this equality by a stationary distribution of the Markov chain induced by\(π1,P¯ϕ\)\(\\pi\_\{1\},\\bar\{P\}^\{\\phi\}\)and canceling the bias terms gives

ρP¯ϕπ1=ρ1σ\.\\rho\_\{\\bar\{P\}^\{\\phi\}\}^\{\\pi\_\{1\}\}=\\rho\_\{1\}^\{\\sigma\}\.Both directions of Lemma[1](https://arxiv.org/html/2608.06545#Thmlemma1)also giveminP∈𝒫⁡ρPπ1=ρ1σ\\min\_\{P\\in\\mathcal\{P\}\}\\rho\_\{P\}^\{\\pi\_\{1\}\}=\\rho\_\{1\}^\{\\sigma\}\. HenceP¯ϕ\\bar\{P\}^\{\\phi\}attains this minimum\.

###### Expected return time\.

UnderP¯ϕ\\bar\{P\}^\{\\phi\}, the time between successive returns toSSincludes one step fromSS, one additional step with probabilityσ0\\sigma\_\{0\}, and an expected1/q01/q\_\{0\}steps at state11with probabilityp0p\_\{0\}\. Hence

T1σ=1\+σ0\+p0q0\.T\_\{1\}^\{\\sigma\}=1\+\\sigma\_\{0\}\+\\frac\{p\_\{0\}\}\{q\_\{0\}\}\.

##### Remaining policies\.

First fixi∈\{2,3\}i\\in\\\{2,3\\\}\. The same argument applies with

\(p0,q0,σ0,r0\)replaced by\(p1\+δ​𝟙\{ϕ=i\},q1,σ1,r1\)\.\(p\_\{0\},q\_\{0\},\\sigma\_\{0\},r\_\{0\}\)\\quad\\text\{replaced by\}\\quad\\left\(p\_\{1\}\+\\delta\\mathds\{1\}\_\{\\\{\\phi=i\\\}\},q\_\{1\},\\sigma\_\{1\},r\_\{1\}\\right\)\.At each other reward state, set the bias coordinate equal to its immediate reward minusρiσ\\rho\_\{i\}^\{\\sigma\}\. Becauseπi\\pi\_\{i\}returns immediately toSSthere, the corresponding fixed\-policy Bellman equation holds\. Consequently,P¯ϕ\\bar\{P\}^\{\\phi\}attains the robust minimum\. The resulting rewards are

ρi=σ1\+p1\+δ​𝟙\{ϕ=i\}σ1\+p1\+q1\+δ​𝟙\{ϕ=i\}​r1,ρiσ=p1\+δ​𝟙\{ϕ=i\}p1\+δ​𝟙\{ϕ=i\}\+q1​\(1\+σ1\)​r1,\\rho\_\{i\}=\\frac\{\\sigma\_\{1\}\+p\_\{1\}\+\\delta\\mathds\{1\}\_\{\\\{\\phi=i\\\}\}\}\{\\sigma\_\{1\}\+p\_\{1\}\+q\_\{1\}\+\\delta\\mathds\{1\}\_\{\\\{\\phi=i\\\}\}\}r\_\{1\},\\qquad\\rho\_\{i\}^\{\\sigma\}=\\frac\{p\_\{1\}\+\\delta\\mathds\{1\}\_\{\\\{\\phi=i\\\}\}\}\{p\_\{1\}\+\\delta\\mathds\{1\}\_\{\\\{\\phi=i\\\}\}\+q\_\{1\}\(1\+\\sigma\_\{1\}\)\}r\_\{1\},and the expected return time is

Tiσ=1\+σ1\+p1\+δ​𝟙\{ϕ=i\}q1\.T\_\{i\}^\{\\sigma\}=1\+\\sigma\_\{1\}\+\\frac\{p\_\{1\}\+\\delta\\mathds\{1\}\_\{\\\{\\phi=i\\\}\}\}\{q\_\{1\}\}\.Takingi=ϕi=\\phiandi=5−ϕi=5\-\\phiproves the two corresponding cases of the lemma\.

Finally, ifa∉\{1,2,3\}a\\notin\\\{1,2,3\\\}, the uncertainty radius at\(S,a\)\(S,a\)is zero\. Hence everyP∈𝒫P\\in\\mathcal\{P\}returns immediately fromSStoSSunderπa\\pi\_\{a\}, with zero reward\. Therefore

ρa=ρaσ=0,Taσ=1,\\rho\_\{a\}=\\rho\_\{a\}^\{\\sigma\}=0,\\qquad T\_\{a\}^\{\\sigma\}=1,which completes the proof of Lemma[4](https://arxiv.org/html/2608.06545#Thmlemma4)\.

#### C\.3\.2Proof of Lemmas[5](https://arxiv.org/html/2608.06545#Thmlemma5)and[7](https://arxiv.org/html/2608.06545#Thmlemma7)

WriteH¯:=min⁡\{H0,Hσ\}\\bar\{H\}:=\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\. We verify the return\-time and gap conditions in the two parameter regimes\. Throughout, the formulas forTaσT\_\{a\}^\{\\sigma\}andρaσ\\rho\_\{a\}^\{\\sigma\}come from Lemma[4](https://arxiv.org/html/2608.06545#Thmlemma4)\.

##### Caseσ<4/H¯\\sigma<4/\\bar\{H\}\.

Direct substitution, together withε≤0\.01​σ​H¯\\varepsilon\\leq 0\.01\\sigma\\bar\{H\}andσ≤1/2\\sigma\\leq 1/2, gives

Tϕσ=2\+σ4\+200​εσ​H¯≤338,T5−ϕσ=2\+σ4∈\[2,178\],T1σ=32\+σ∈\[32,2\],T\_\{\\phi\}^\{\\sigma\}=2\+\\frac\{\\sigma\}\{4\}\+\\frac\{200\\varepsilon\}\{\\sigma\\bar\{H\}\}\\leq\\frac\{33\}\{8\},\\quad T\_\{5\-\\phi\}^\{\\sigma\}=2\+\\frac\{\\sigma\}\{4\}\\in\\left\[2,\\frac\{17\}\{8\}\\right\],\\quad T\_\{1\}^\{\\sigma\}=\\frac\{3\}\{2\}\+\\sigma\\in\\left\[\\frac\{3\}\{2\},2\\right\],whileTaσ=1T\_\{a\}^\{\\sigma\}=1fora∉\{1,2,3\}a\\notin\\\{1,2,3\\\}\. For the only delicate reward\-gap calculation, subtracting the two robust reward formulas yields

ρϕσ−ρ5−ϕσ=3​σ​H¯16​100​εHσ​\(1\+σ/4\)\(σ​H¯2​Hσ​\(2\+σ/4\)\+100​εHσ\)​\(2\+σ/4\)\.\\rho\_\{\\phi\}^\{\\sigma\}\-\\rho\_\{5\-\\phi\}^\{\\sigma\}=\\frac\{3\\sigma\\bar\{H\}\}\{16\}\\frac\{\\frac\{100\\varepsilon\}\{H\_\{\\sigma\}\}\(1\+\\sigma/4\)\}\{\\left\(\\frac\{\\sigma\\bar\{H\}\}\{2H\_\{\\sigma\}\}\(2\+\\sigma/4\)\+\\frac\{100\\varepsilon\}\{H\_\{\\sigma\}\}\\right\)\(2\+\\sigma/4\)\}\.Since the first factor in parentheses in the denominator is at mostσ​H¯2​Hσ​\(4\+σ/4\)\\frac\{\\sigma\\bar\{H\}\}\{2H\_\{\\sigma\}\}\(4\+\\sigma/4\),

ρϕσ−ρ5−ϕσ≥600​ε​\(1\+σ/4\)16​\(4\+σ/4\)​\(2\+σ/4\)\>4​ε;\\rho\_\{\\phi\}^\{\\sigma\}\-\\rho\_\{5\-\\phi\}^\{\\sigma\}\\geq\\frac\{600\\varepsilon\(1\+\\sigma/4\)\}\{16\(4\+\\sigma/4\)\(2\+\\sigma/4\)\}\>4\\varepsilon;the last inequality is equivalent to88\+54​σ−4​σ2\>088\+54\\sigma\-4\\sigma^\{2\}\>0\. Moreover,

ρ5−ϕσ−ρ1σ=σ​H¯​\[316​\(2\+σ/4\)−14​\(3\+2​σ\)\]≥σ​H¯96\>ε,\\rho\_\{5\-\\phi\}^\{\\sigma\}\-\\rho\_\{1\}^\{\\sigma\}=\\sigma\\bar\{H\}\\left\[\\frac\{3\}\{16\(2\+\\sigma/4\)\}\-\\frac\{1\}\{4\(3\+2\\sigma\)\}\\right\]\\geq\\frac\{\\sigma\\bar\{H\}\}\{96\}\>\\varepsilon,because the bracket is at least1/961/96\. Thusρϕσ−ρ1σ\>5​ε\\rho\_\{\\phi\}^\{\\sigma\}\-\\rho\_\{1\}^\{\\sigma\}\>5\\varepsilon\. Finally,ρaσ=0\\rho\_\{a\}^\{\\sigma\}=0fora∉\{1,2,3\}a\\notin\\\{1,2,3\\\}andρϕσ≥ρ5−ϕσ=3​σ​H¯/\[16​\(2\+σ/4\)\]\\rho\_\{\\phi\}^\{\\sigma\}\\geq\\rho\_\{5\-\\phi\}^\{\\sigma\}=3\\sigma\\bar\{H\}/\[16\(2\+\\sigma/4\)\]\. The required weighted gaps are therefore summarized by

aTaσTϕσ\+Taσ​\(ρϕσ−ρaσ\)​is bounded below by5−ϕ233/8\+17/8​4​ε\>ε13/233/8\+2​5​ε\>εa∉\{1,2,3\}12697​σ​H¯\>ε\.\\begin\{array\}\[\]\{c\|c\}a&\\displaystyle\\frac\{T\_\{a\}^\{\\sigma\}\}\{T\_\{\\phi\}^\{\\sigma\}\+T\_\{a\}^\{\\sigma\}\}\(\\rho\_\{\\phi\}^\{\\sigma\}\-\\rho\_\{a\}^\{\\sigma\}\)\\text\{ is bounded below by\}\\\\ \\hline\\cr 5\-\\phi&\\displaystyle\\frac\{2\}\{33/8\+17/8\}\\,4\\varepsilon\>\\varepsilon\\\\ 1&\\displaystyle\\frac\{3/2\}\{33/8\+2\}\\,5\\varepsilon\>\\varepsilon\\\\ a\\notin\\\{1,2,3\\\}&\\displaystyle\\frac\{12\}\{697\}\\sigma\\bar\{H\}\>\\varepsilon\.\\end\{array\}This proves \([28](https://arxiv.org/html/2608.06545#A3.E28)\) in the first regime\.

##### Caseσ≥4/H¯\\sigma\\geq 4/\\bar\{H\}\.

Here

Tϕσ=2\+σ\+20​ε≤2710,T5−ϕσ=2\+σ∈\[2,52\],T1σ=43,T\_\{\\phi\}^\{\\sigma\}=2\+\\sigma\+20\\varepsilon\\leq\\frac\{27\}\{10\},\\quad T\_\{5\-\\phi\}^\{\\sigma\}=2\+\\sigma\\in\\left\[2,\\frac\{5\}\{2\}\\right\],\\quad T\_\{1\}^\{\\sigma\}=\\frac\{4\}\{3\},and againTaσ=1T\_\{a\}^\{\\sigma\}=1fora∉\{1,2,3\}a\\notin\\\{1,2,3\\\}\. In this regime,

ρϕσ−ρ5−ϕσ=80​ε​\(1\+σ\)\(4\+2​σ\+40​ε\)​\(4\+2​σ\)\>4​ε\.\\rho\_\{\\phi\}^\{\\sigma\}\-\\rho\_\{5\-\\phi\}^\{\\sigma\}=\\frac\{80\\varepsilon\(1\+\\sigma\)\}\{\(4\+2\\sigma\+40\\varepsilon\)\(4\+2\\sigma\)\}\>4\\varepsilon\.Indeed,40​ε≤0\.440\\varepsilon\\leq 0\.4and

20​\(1\+σ\)−\(4\+2​σ\+40​ε\)​\(4\+2​σ\)≥2\.4\+3\.2​σ−4​σ2\>0\.20\(1\+\\sigma\)\-\(4\+2\\sigma\+40\\varepsilon\)\(4\+2\\sigma\)\\geq 2\.4\+3\.2\\sigma\-4\\sigma^\{2\}\>0\.The robust reward ordering established above also givesρϕσ−ρ1σ\>2/5−1/4=3/20\\rho\_\{\\phi\}^\{\\sigma\}\-\\rho\_\{1\}^\{\\sigma\}\>2/5\-1/4=3/20andρϕσ−ρaσ\>2/5\\rho\_\{\\phi\}^\{\\sigma\}\-\\rho\_\{a\}^\{\\sigma\}\>2/5fora∉\{1,2,3\}a\\notin\\\{1,2,3\\\}\. Hence

aTaσTϕσ\+Taσ​\(ρϕσ−ρaσ\)​is bounded below by5−ϕ227/10\+5/2​4​ε\>ε14/327/10\+4/3​320=6121\>εa∉\{1,2,3\}127/10\+1​25=437\>ε\.\\begin\{array\}\[\]\{c\|c\}a&\\displaystyle\\frac\{T\_\{a\}^\{\\sigma\}\}\{T\_\{\\phi\}^\{\\sigma\}\+T\_\{a\}^\{\\sigma\}\}\(\\rho\_\{\\phi\}^\{\\sigma\}\-\\rho\_\{a\}^\{\\sigma\}\)\\text\{ is bounded below by\}\\\\ \\hline\\cr 5\-\\phi&\\displaystyle\\frac\{2\}\{27/10\+5/2\}\\,4\\varepsilon\>\\varepsilon\\\\ 1&\\displaystyle\\frac\{4/3\}\{27/10\+4/3\}\\,\\frac\{3\}\{20\}=\\frac\{6\}\{121\}\>\\varepsilon\\\\ a\\notin\\\{1,2,3\\\}&\\displaystyle\\frac\{1\}\{27/10\+1\}\\,\\frac\{2\}\{5\}=\\frac\{4\}\{37\}\>\\varepsilon\.\\end\{array\}This proves \([28](https://arxiv.org/html/2608.06545#A3.E28)\) in the second regime\.

#### C\.3\.3Proof of Lemmas[6](https://arxiv.org/html/2608.06545#Thmlemma6)and[8](https://arxiv.org/html/2608.06545#Thmlemma8)

We first give a robust Bellman certificate common to both parameter regimes\. Normalizeh​\(S\)=0h\(S\)=0and set

h​\(1\)=r0−ρϕσq0,h​\(2\)=h​\(3\)=r1−ρϕσq1,h​\(s\)=−ρϕσ,s∈\{4,…,S−1\}\.h\(1\)=\\frac\{r\_\{0\}\-\\rho\_\{\\phi\}^\{\\sigma\}\}\{q\_\{0\}\},\\qquad h\(2\)=h\(3\)=\\frac\{r\_\{1\}\-\\rho\_\{\\phi\}^\{\\sigma\}\}\{q\_\{1\}\},\\qquad h\(s\)=\-\\rho\_\{\\phi\}^\{\\sigma\},\\quad s\\in\\\{4,\\ldots,S\-1\\\}\.The reward comparisons established in the corresponding regimes implyh​\(1\),h​\(2\),h​\(3\)≥0h\(1\),h\(2\),h\(3\)\\geq 0\. At the reward states, the definition ofhhgives

r0\+\(1−q0\)​h​\(1\)=ρϕσ\+h​\(1\),r1\+\(1−q1\)​h​\(i\)=ρϕσ\+h​\(i\),i∈\{2,3\};r\_\{0\}\+\(1\-q\_\{0\}\)h\(1\)=\\rho\_\{\\phi\}^\{\\sigma\}\+h\(1\),\\qquad r\_\{1\}\+\(1\-q\_\{1\}\)h\(i\)=\\rho\_\{\\phi\}^\{\\sigma\}\+h\(i\),\\quad i\\in\\\{2,3\\\};the selected action therefore dominates an immediate return toSS\. The delay and padding states satisfyρϕσ\+h​\(s\)=0\\rho\_\{\\phi\}^\{\\sigma\}\+h\(s\)=0\. AtSS, a minimizing row transfers its full uncertainty budget from the reward state to a delay state; this is feasible because the nominal row assigns at least the uncertainty radius to the former and no mass to the latter\. The expression for actionϕ\\phithen equals

\(p1\+δ\)​h​\(ϕ\)−σ1​ρϕσ=ρϕσ\.\(p\_\{1\}\+\\delta\)h\(\\phi\)\-\\sigma\_\{1\}\\rho\_\{\\phi\}^\{\\sigma\}=\\rho\_\{\\phi\}^\{\\sigma\}\.For actions11and5−ϕ5\-\\phi, the same substitution shows that their expressions are at mostρϕσ\\rho\_\{\\phi\}^\{\\sigma\}precisely whenρ1σ≤ρϕσ\\rho\_\{1\}^\{\\sigma\}\\leq\\rho\_\{\\phi\}^\{\\sigma\}andρ5−ϕσ≤ρϕσ\\rho\_\{5\-\\phi\}^\{\\sigma\}\\leq\\rho\_\{\\phi\}^\{\\sigma\}, respectively; these are the robust reward orderings already proved\. All remaining actions contribute zero\. Consequently,

ρϕσ​𝟏\+h=𝒯σ​h\.\\rho\_\{\\phi\}^\{\\sigma\}\\mathbf\{1\}\+h=\\mathcal\{T\}\_\{\\sigma\}h\.Lemma[1](https://arxiv.org/html/2608.06545#Thmlemma1), applied first to an arbitrary policy and then to a maximizing selector of𝒯σ​h\\mathcal\{T\}\_\{\\sigma\}h, shows thatρ⋆,σ=ρϕσ\\rho^\{\\star,\\sigma\}=\\rho\_\{\\phi\}^\{\\sigma\}and thathhis a robust optimal Bellman solution\.

It remains to construct the nominal certificates and bound their spans\. Whenσ<4/H¯\\sigma<4/\\bar\{H\}, Lemma[4](https://arxiv.org/html/2608.06545#Thmlemma4)givesρ1=r1\\rho\_\{1\}=r\_\{1\}\. Define

g​\(S\)=0,g​\(1\)=r0−r1q0,g​\(2\)=g​\(3\)=0,g​\(s\)=−r1,s∈\{4,…,S−1\}\.g\(S\)=0,\\qquad g\(1\)=\\frac\{r\_\{0\}\-r\_\{1\}\}\{q\_\{0\}\},\\qquad g\(2\)=g\(3\)=0,\\qquad g\(s\)=\-r\_\{1\},\\quad s\\in\\\{4,\\ldots,S\-1\\\}\.The reward\-state and delay\-state equations follow directly from this definition, while atSS,

\(𝒯0​g\)​\(S\)=\(σ0\+p0\)​g​\(1\)=6​σ5​5​H¯32=r1\.\(\\mathcal\{T\}\_\{0\}g\)\(S\)=\(\\sigma\_\{0\}\+p\_\{0\}\)g\(1\)=\\frac\{6\\sigma\}\{5\}\\frac\{5\\bar\{H\}\}\{32\}=r\_\{1\}\.Thusr1​𝟏\+g=𝒯0​gr\_\{1\}\\mathbf\{1\}\+g=\\mathcal\{T\}\_\{0\}g\.

Whenσ≥4/H¯\\sigma\\geq 4/\\bar\{H\}, we haver0=r1=1r\_\{0\}=r\_\{1\}=1,q0=3/H¯q\_\{0\}=3/\\bar\{H\}, andq1=2/Hσq\_\{1\}=2/H\_\{\\sigma\}\. Moreover,

ρ1=14,ρϕ=σ\+p1\+δσ\+p1\+q1\+δ\>12,ρ5−ϕ<ρϕ,\\rho\_\{1\}=\\frac\{1\}\{4\},\\qquad\\rho\_\{\\phi\}=\\frac\{\\sigma\+p\_\{1\}\+\\delta\}\{\\sigma\+p\_\{1\}\+q\_\{1\}\+\\delta\}\>\\frac\{1\}\{2\},\\qquad\\rho\_\{5\-\\phi\}<\\rho\_\{\\phi\},whereq1≤σ/2<σ\+p1\+δq\_\{1\}\\leq\\sigma/2<\\sigma\+p\_\{1\}\+\\delta\. Defineg​\(S\)=0g\(S\)=0,g​\(1\)=\(1−ρϕ\)/q0g\(1\)=\(1\-\\rho\_\{\\phi\}\)/q\_\{0\},g​\(2\)=g​\(3\)=\(1−ρϕ\)/q1g\(2\)=g\(3\)=\(1\-\\rho\_\{\\phi\}\)/q\_\{1\}, andg​\(s\)=−ρϕg\(s\)=\-\\rho\_\{\\phi\}fors∈\{4,…,S−1\}s\\in\\\{4,\\ldots,S\-1\\\}\. Again the Bellman equations away fromSSfollow directly\. AtSS, actionϕ\\phihas value\(σ\+p1\+δ\)​g​\(ϕ\)=ρϕ\(\\sigma\+p\_\{1\}\+\\delta\)g\(\\phi\)=\\rho\_\{\\phi\}; action5−ϕ5\-\\phihas smaller value becauseδ\>0\\delta\>0, and action11has value at mostρϕ\\rho\_\{\\phi\}becauseρ1≤ρϕ\\rho\_\{1\}\\leq\\rho\_\{\\phi\}\. Henceρϕ​𝟏\+g=𝒯0​g\\rho\_\{\\phi\}\\mathbf\{1\}\+g=\\mathcal\{T\}\_\{0\}g\.

For completeness, the span calculations for these certificates are collected below\. In the second nominal case, the parameter bounds give

g​\(2\)\+ρϕ=1\+σ\+p1\+δσ\+p1\+q1\+δ≤2σ≤H¯2,g​\(1\)\+ρϕ≤1q0\+1≤7​H¯12\.g\(2\)\+\\rho\_\{\\phi\}=\\frac\{1\+\\sigma\+p\_\{1\}\+\\delta\}\{\\sigma\+p\_\{1\}\+q\_\{1\}\+\\delta\}\\leq\\frac\{2\}\{\\sigma\}\\leq\\frac\{\\bar\{H\}\}\{2\},\\qquad g\(1\)\+\\rho\_\{\\phi\}\\leq\\frac\{1\}\{q\_\{0\}\}\+1\\leq\\frac\{7\\bar\{H\}\}\{12\}\.The remaining bounds follow immediately from the displayed definitions:

regimecertificatespan calculationσ<4/H¯nominal​gmax⁡g=5​H¯/32,−min⁡g=3​σ​H¯/16,‖g‖span≤H¯/4σ<4/H¯robust​hmax⁡h≤5​Hσ/8,−min⁡h≤Hσ/8,‖h‖span≤3​Hσ/4σ≥4/H¯nominal​g‖g‖span=max⁡\{g​\(1\)\+ρϕ,g​\(2\)\+ρϕ\}≤7​H¯/12σ≥4/H¯robust​hmax⁡h≤Hσ/2,−min⁡h≤Hσ/4,‖h‖span≤3​Hσ/4\.\\begin\{array\}\[\]\{c@\{\\qquad\}c@\{\\qquad\}l\}\\text\{regime\}&\\text\{certificate\}&\\text\{span calculation\}\\\\ \\hline\\cr\\sigma<4/\\bar\{H\}&\\text\{nominal \}g&\\max g=5\\bar\{H\}/32,\\quad\-\\min g=3\\sigma\\bar\{H\}/16,\\quad\\\|g\\\|\_\{\\mathrm\{span\}\}\\leq\\bar\{H\}/4\\\\\[2\.58334pt\] \\sigma<4/\\bar\{H\}&\\text\{robust \}h&\\max h\\leq 5H\_\{\\sigma\}/8,\\quad\-\\min h\\leq H\_\{\\sigma\}/8,\\quad\\\|h\\\|\_\{\\mathrm\{span\}\}\\leq 3H\_\{\\sigma\}/4\\\\\[2\.58334pt\] \\sigma\\geq 4/\\bar\{H\}&\\text\{nominal \}g&\\\|g\\\|\_\{\\mathrm\{span\}\}=\\max\\\{g\(1\)\+\\rho\_\{\\phi\},g\(2\)\+\\rho\_\{\\phi\}\\\}\\leq 7\\bar\{H\}/12\\\\\[2\.58334pt\] \\sigma\\geq 4/\\bar\{H\}&\\text\{robust \}h&\\max h\\leq H\_\{\\sigma\}/2,\\quad\-\\min h\\leq H\_\{\\sigma\}/4,\\quad\\\|h\\\|\_\{\\mathrm\{span\}\}\\leq 3H\_\{\\sigma\}/4\.\\end\{array\}SincehP0⋆h\_\{P^\{0\}\}^\{\\star\}andh⋆,σh^\{\\star,\\sigma\}are minimum\-span Bellman solutions andH¯≤H0\\bar\{H\}\\leq H\_\{0\}, the first two rows prove Lemma[6](https://arxiv.org/html/2608.06545#Thmlemma6), and the last two prove Lemma[8](https://arxiv.org/html/2608.06545#Thmlemma8)\.

## Appendix DProofs for the reduction\-based upper bounds

This appendix contains the proofs for the reduction\-based upper bounds in Section[4](https://arxiv.org/html/2608.06545#S4)\. Appendix[D\.1](https://arxiv.org/html/2608.06545#A4.SS1)establishes the discounted\-to\-average reduction, while Appendix[D\.2](https://arxiv.org/html/2608.06545#A4.SS2)states the anchored robust DMDP theorem and its corollaries\. Appendix[D\.3](https://arxiv.org/html/2608.06545#A4.SS3)states the nominal\-branch guarantee and combines it with the robust reduction to prove Theorem[2](https://arxiv.org/html/2608.06545#Thmtheorem2); the proof of the nominal\-branch guarantee is deferred to Appendix[D\.5\.1](https://arxiv.org/html/2608.06545#A4.SS5.SSS1)\. Appendix[D\.4](https://arxiv.org/html/2608.06545#A4.SS4)proves Theorem[4](https://arxiv.org/html/2608.06545#Thmtheorem4)\. Finally, Appendix[D\.5](https://arxiv.org/html/2608.06545#A4.SS5)collects the proofs of the technical lemmas used above\.

Throughout this appendix,C\>0C\>0denotes a universal constant whose value may change from line to line\. Constants with subscripts, such asC0C\_\{0\}andC1C\_\{1\}, remain fixed once introduced\.

### D\.1Discounted\-to\-average reduction

In this subsection, we present a few results that convert discounted guarantees into average\-reward guarantees\.

The first lemma controls the difference between a robust discounted value and the corresponding robust average reward\.

###### Lemma 9\(Robust discounted\-to\-average comparison\)\.

Suppose Assumption[1](https://arxiv.org/html/2608.06545#Thmassumption1)holds\. Fix a stationary policyπ\\piand a discount factorγ∈\(0,1\)\\gamma\\in\(0,1\)\.

\(1−γ\)​mins⁡Vγπ,σ​\(s\)≤ρπ,σ≤\(1−γ\)​maxs⁡Vγπ,σ​\(s\)\.\(1\-\\gamma\)\\min\_\{s\}V\_\{\\gamma\}^\{\\pi,\\sigma\}\(s\)\\leq\\rho^\{\\pi,\\sigma\}\\leq\(1\-\\gamma\)\\max\_\{s\}V\_\{\\gamma\}^\{\\pi,\\sigma\}\(s\)\.\(53\)Equivalently, for alls∈𝒮s\\in\\mathcal\{S\},

−\(1−γ\)​‖Vγπ,σ‖span≤ρπ,σ−\(1−γ\)​Vγπ,σ​\(s\)≤\(1−γ\)​‖Vγπ,σ‖span\.\-\(1\-\\gamma\)\\\|V\_\{\\gamma\}^\{\\pi,\\sigma\}\\\|\_\{\\mathrm\{span\}\}\\leq\\rho^\{\\pi,\\sigma\}\-\(1\-\\gamma\)V\_\{\\gamma\}^\{\\pi,\\sigma\}\(s\)\\leq\(1\-\\gamma\)\\\|V\_\{\\gamma\}^\{\\pi,\\sigma\}\\\|\_\{\\mathrm\{span\}\}\.\(54\)Moreover,

ρ⋆,σ≤\(1−γ\)​mins⁡Vγ⋆,σ​\(s\)\+2​\(1−γ\)​Hσ\.\\rho^\{\\star,\\sigma\}\\leq\(1\-\\gamma\)\\min\_\{s\}V\_\{\\gamma\}^\{\\star,\\sigma\}\(s\)\+2\(1\-\\gamma\)H\_\{\\sigma\}\.\(55\)

The next lemma turns a discounted\-policy error bound into an average\-reward error bound\.

###### Lemma 10\(Direct robust discounted\-to\-average reduction\)\.

Letπ\\pibe a stationary policy satisfying, componentwise,

Vγ⋆,σ−Vγπ,σ≤εγ​𝟏\.V\_\{\\gamma\}^\{\\star,\\sigma\}\-V\_\{\\gamma\}^\{\\pi,\\sigma\}\\leq\\varepsilon\_\{\\gamma\}\\mathbf\{1\}\.Then

ρ⋆,σ−ρπ,σ≤\(1−γ\)​\(2​Hσ\+εγ\)\.\\rho^\{\\star,\\sigma\}\-\\rho^\{\\pi,\\sigma\}\\leq\(1\-\\gamma\)\\left\(2H\_\{\\sigma\}\+\\varepsilon\_\{\\gamma\}\\right\)\.

###### Proof\.

Equations \([55](https://arxiv.org/html/2608.06545#A4.E55)\) and \([53](https://arxiv.org/html/2608.06545#A4.E53)\) give

ρ⋆,σ−ρπ,σ\\displaystyle\\rho^\{\\star,\\sigma\}\-\\rho^\{\\pi,\\sigma\}≤\(1−γ\)​\(mins⁡Vγ⋆,σ​\(s\)−mins⁡Vγπ,σ​\(s\)\+2​Hσ\)\\displaystyle\\leq\(1\-\\gamma\)\\left\(\\min\_\{s\}V\_\{\\gamma\}^\{\\star,\\sigma\}\(s\)\-\\min\_\{s\}V\_\{\\gamma\}^\{\\pi,\\sigma\}\(s\)\+2H\_\{\\sigma\}\\right\)≤\(1−γ\)​\(2​Hσ\+εγ\)\.\\displaystyle\\leq\(1\-\\gamma\)\\left\(2H\_\{\\sigma\}\+\\varepsilon\_\{\\gamma\}\\right\)\.Here the second inequality follows from the componentwise assumption, which implies

mins⁡Vγπ,σ​\(s\)≥mins⁡Vγ⋆,σ​\(s\)−εγ\.\\min\_\{s\}V\_\{\\gamma\}^\{\\pi,\\sigma\}\(s\)\\geq\\min\_\{s\}V\_\{\\gamma\}^\{\\star,\\sigma\}\(s\)\-\\varepsilon\_\{\\gamma\}\.∎

Finally, the following corollary chooses the discount factor and discounted\-policy error bound so that the resulting policy isε\\varepsilon\-optimal for the robust average\-reward problem\.

###### Corollary 1\(Parameters for the span\-informed reduction\)\.

Suppose0<ε≤10<\\varepsilon\\leq 1and

γ=1−ε3​Hσ\.\\gamma=1\-\\frac\{\\varepsilon\}\{3H\_\{\\sigma\}\}\.If a stationary policyπ\\pisatisfies

Vγ⋆,σ−Vγπ,σ≤Hσ​𝟏,V\_\{\\gamma\}^\{\\star,\\sigma\}\-V\_\{\\gamma\}^\{\\pi,\\sigma\}\\leq H\_\{\\sigma\}\\mathbf\{1\},then

ρ⋆,σ−ρπ,σ≤ε\.\\rho^\{\\star,\\sigma\}\-\\rho^\{\\pi,\\sigma\}\\leq\\varepsilon\.

###### Proof\.

The definition in \([3b](https://arxiv.org/html/2608.06545#S2.E3.2)\) givesHσ≥1H\_\{\\sigma\}\\geq 1, so the displayed choice ofγ\\gammabelongs to\(0,1\)\(0,1\)\. Apply Lemma[10](https://arxiv.org/html/2608.06545#Thmlemma10)withεγ=Hσ\\varepsilon\_\{\\gamma\}=H\_\{\\sigma\}\. ∎

#### D\.1\.1Proof of Lemma[9](https://arxiv.org/html/2608.06545#Thmlemma9)

We first prove \([53](https://arxiv.org/html/2608.06545#A4.E53)\) in two directions, and then derive \([54](https://arxiv.org/html/2608.06545#A4.E54)\) and \([55](https://arxiv.org/html/2608.06545#A4.E55)\)\.

##### Lower bound in \([53](https://arxiv.org/html/2608.06545#A4.E53)\)\.

Fix an arbitrary transition kernelP∈𝒫P\\in\\mathcal\{P\}, and letρPπ\\rho\_\{P\}^\{\\pi\}andVγ,PπV\_\{\\gamma,P\}^\{\\pi\}denote the average reward and the discounted value ofπ\\piunderPP, respectively\. Under Assumption[1](https://arxiv.org/html/2608.06545#Thmassumption1), letμ\\mube a stationary distribution supported on the unique recurrent class induced by\(π,P\)\(\\pi,P\)\. The discounted Bellman equation

Vγ,Pπ=rπ\+γ​Pπ​Vγ,PπV\_\{\\gamma,P\}^\{\\pi\}=r^\{\\pi\}\+\\gamma P\_\{\\pi\}V\_\{\\gamma,P\}^\{\\pi\}andμ⊤​Pπ=μ⊤\\mu^\{\\top\}P\_\{\\pi\}=\\mu^\{\\top\}imply

ρPπ=μ⊤​rπ=\(1−γ\)​μ⊤​Vγ,Pπ\.\\rho\_\{P\}^\{\\pi\}=\\mu^\{\\top\}r^\{\\pi\}=\(1\-\\gamma\)\\mu^\{\\top\}V\_\{\\gamma,P\}^\{\\pi\}\.Therefore

\(1−γ\)​mins⁡Vγ,Pπ​\(s\)≤ρPπ≤\(1−γ\)​maxs⁡Vγ,Pπ​\(s\)\.\(1\-\\gamma\)\\min\_\{s\}V\_\{\\gamma,P\}^\{\\pi\}\(s\)\\leq\\rho\_\{P\}^\{\\pi\}\\leq\(1\-\\gamma\)\\max\_\{s\}V\_\{\\gamma,P\}^\{\\pi\}\(s\)\.SinceVγπ,σ​\(s\)≤Vγ,Pπ​\(s\)V\_\{\\gamma\}^\{\\pi,\\sigma\}\(s\)\\leq V\_\{\\gamma,P\}^\{\\pi\}\(s\)for everys∈𝒮s\\in\\mathcal\{S\}and everyP∈𝒫P\\in\\mathcal\{P\},

\(1−γ\)​mins⁡Vγπ,σ​\(s\)≤ρPπ\.\(1\-\\gamma\)\\min\_\{s\}V\_\{\\gamma\}^\{\\pi,\\sigma\}\(s\)\\leq\\rho\_\{P\}^\{\\pi\}\.Taking the infimum overP∈𝒫P\\in\\mathcal\{P\}gives the lower bound in \([53](https://arxiv.org/html/2608.06545#A4.E53)\)\.

##### Upper bound in \([53](https://arxiv.org/html/2608.06545#A4.E53)\)\.

For the fixed policyπ\\pi, the discounted robust Bellman equation gives

Vγπ,σ​\(s\)=rπ​\(s\)\+γ​infP∈𝒫\(Pπ​Vγπ,σ\)​\(s\)\.V\_\{\\gamma\}^\{\\pi,\\sigma\}\(s\)=r^\{\\pi\}\(s\)\+\\gamma\\inf\_\{P\\in\\mathcal\{P\}\}\(P\_\{\\pi\}V\_\{\\gamma\}^\{\\pi,\\sigma\}\)\(s\)\.Since

infP∈𝒫\(Pπ​Vγπ,σ\)​\(s\)≤maxs′⁡Vγπ,σ​\(s′\),\\inf\_\{P\\in\\mathcal\{P\}\}\(P\_\{\\pi\}V\_\{\\gamma\}^\{\\pi,\\sigma\}\)\(s\)\\leq\\max\_\{s^\{\\prime\}\}V\_\{\\gamma\}^\{\\pi,\\sigma\}\(s^\{\\prime\}\),we have

\(1−γ\)​maxs′⁡Vγπ,σ​\(s′\)\+Vγπ,σ​\(s\)\\displaystyle\(1\-\\gamma\)\\max\_\{s^\{\\prime\}\}V\_\{\\gamma\}^\{\\pi,\\sigma\}\(s^\{\\prime\}\)\+V\_\{\\gamma\}^\{\\pi,\\sigma\}\(s\)≥Vγπ,σ​\(s\)\+\(1−γ\)​infP∈𝒫\(Pπ​Vγπ,σ\)​\(s\)\\displaystyle\\qquad\\geq V\_\{\\gamma\}^\{\\pi,\\sigma\}\(s\)\+\(1\-\\gamma\)\\inf\_\{P\\in\\mathcal\{P\}\}\(P\_\{\\pi\}V\_\{\\gamma\}^\{\\pi,\\sigma\}\)\(s\)=rπ​\(s\)\+infP∈𝒫\(Pπ​Vγπ,σ\)​\(s\)\.\\displaystyle\\qquad=r^\{\\pi\}\(s\)\+\\inf\_\{P\\in\\mathcal\{P\}\}\(P\_\{\\pi\}V\_\{\\gamma\}^\{\\pi,\\sigma\}\)\(s\)\.Lemma[1](https://arxiv.org/html/2608.06545#Thmlemma1), applied withh=Vγπ,σh=V\_\{\\gamma\}^\{\\pi,\\sigma\}andρ=\(1−γ\)​maxs′⁡Vγπ,σ​\(s′\)\\rho=\(1\-\\gamma\)\\max\_\{s^\{\\prime\}\}V\_\{\\gamma\}^\{\\pi,\\sigma\}\(s^\{\\prime\}\), gives

ρπ,σ≤\(1−γ\)​maxs′⁡Vγπ,σ​\(s′\)\.\\rho^\{\\pi,\\sigma\}\\leq\(1\-\\gamma\)\\max\_\{s^\{\\prime\}\}V\_\{\\gamma\}^\{\\pi,\\sigma\}\(s^\{\\prime\}\)\.This proves the upper bound in \([53](https://arxiv.org/html/2608.06545#A4.E53)\)\.

##### Proof of \([54](https://arxiv.org/html/2608.06545#A4.E54)\)\.

Subtracting\(1−γ\)​Vγπ,σ​\(s\)\(1\-\\gamma\)V\_\{\\gamma\}^\{\\pi,\\sigma\}\(s\)from each side of \([53](https://arxiv.org/html/2608.06545#A4.E53)\) gives

\(1−γ\)​\(mins′⁡Vγπ,σ​\(s′\)−Vγπ,σ​\(s\)\)\\displaystyle\(1\-\\gamma\)\\left\(\\min\_\{s^\{\\prime\}\}V\_\{\\gamma\}^\{\\pi,\\sigma\}\(s^\{\\prime\}\)\-V\_\{\\gamma\}^\{\\pi,\\sigma\}\(s\)\\right\)≤ρπ,σ−\(1−γ\)​Vγπ,σ​\(s\)\\displaystyle\\qquad\\leq\\rho^\{\\pi,\\sigma\}\-\(1\-\\gamma\)V\_\{\\gamma\}^\{\\pi,\\sigma\}\(s\)≤\(1−γ\)​\(maxs′⁡Vγπ,σ​\(s′\)−Vγπ,σ​\(s\)\),\\displaystyle\\qquad\\leq\(1\-\\gamma\)\\left\(\\max\_\{s^\{\\prime\}\}V\_\{\\gamma\}^\{\\pi,\\sigma\}\(s^\{\\prime\}\)\-V\_\{\\gamma\}^\{\\pi,\\sigma\}\(s\)\\right\),which implies \([54](https://arxiv.org/html/2608.06545#A4.E54)\) by the definition of the span seminorm\.

##### Proof of \([55](https://arxiv.org/html/2608.06545#A4.E55)\)\.

For everys∈𝒮s\\in\\mathcal\{S\}anda∈𝒜a\\in\\mathcal\{A\},

r​\(s,a\)\+infPs,a∈𝒫s,aPs,a​Vγ⋆,σ≤r​\(s,a\)\+γ​infPs,a∈𝒫s,aPs,a​Vγ⋆,σ\+\(1−γ\)​maxs′⁡Vγ⋆,σ​\(s′\)\.r\(s,a\)\+\\inf\_\{P\_\{s,a\}\\in\\mathcal\{P\}\_\{s,a\}\}P\_\{s,a\}V\_\{\\gamma\}^\{\\star,\\sigma\}\\leq r\(s,a\)\+\\gamma\\inf\_\{P\_\{s,a\}\\in\\mathcal\{P\}\_\{s,a\}\}P\_\{s,a\}V\_\{\\gamma\}^\{\\star,\\sigma\}\+\(1\-\\gamma\)\\max\_\{s^\{\\prime\}\}V\_\{\\gamma\}^\{\\star,\\sigma\}\(s^\{\\prime\}\)\.Taking the maximum overaaand using the discounted robust Bellman equation forVγ⋆,σV\_\{\\gamma\}^\{\\star,\\sigma\}gives

maxa∈𝒜⁡\{r​\(s,a\)\+infPs,a∈𝒫s,aPs,a​Vγ⋆,σ\}≤Vγ⋆,σ​\(s\)\+\(1−γ\)​maxs′⁡Vγ⋆,σ​\(s′\)\.\\max\_\{a\\in\\mathcal\{A\}\}\\left\\\{r\(s,a\)\+\\inf\_\{P\_\{s,a\}\\in\\mathcal\{P\}\_\{s,a\}\}P\_\{s,a\}V\_\{\\gamma\}^\{\\star,\\sigma\}\\right\\\}\\leq V\_\{\\gamma\}^\{\\star,\\sigma\}\(s\)\+\(1\-\\gamma\)\\max\_\{s^\{\\prime\}\}V\_\{\\gamma\}^\{\\star,\\sigma\}\(s^\{\\prime\}\)\.By rectangularity of the uncertainty set, for every stationary policyπ\\piand every statess,

rπ​\(s\)\+infP∈𝒫\(Pπ​Vγ⋆,σ\)​\(s\)\\displaystyle r^\{\\pi\}\(s\)\+\\inf\_\{P\\in\\mathcal\{P\}\}\(P\_\{\\pi\}V\_\{\\gamma\}^\{\\star,\\sigma\}\)\(s\)=∑a∈𝒜π​\(a∣s\)​\{r​\(s,a\)\+infPs,a∈𝒫s,aPs,a​Vγ⋆,σ\}\\displaystyle\\qquad=\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\(a\\mid s\)\\left\\\{r\(s,a\)\+\\inf\_\{P\_\{s,a\}\\in\\mathcal\{P\}\_\{s,a\}\}P\_\{s,a\}V\_\{\\gamma\}^\{\\star,\\sigma\}\\right\\\}≤maxa∈𝒜⁡\{r​\(s,a\)\+infPs,a∈𝒫s,aPs,a​Vγ⋆,σ\}\.\\displaystyle\\qquad\\leq\\max\_\{a\\in\\mathcal\{A\}\}\\left\\\{r\(s,a\)\+\\inf\_\{P\_\{s,a\}\\in\\mathcal\{P\}\_\{s,a\}\}P\_\{s,a\}V\_\{\\gamma\}^\{\\star,\\sigma\}\\right\\\}\.Combining this inequality with the preceding display gives

rπ​\(s\)\+infP∈𝒫\(Pπ​Vγ⋆,σ\)​\(s\)≤Vγ⋆,σ​\(s\)\+\(1−γ\)​maxs′⁡Vγ⋆,σ​\(s′\),s∈𝒮\.r^\{\\pi\}\(s\)\+\\inf\_\{P\\in\\mathcal\{P\}\}\(P\_\{\\pi\}V\_\{\\gamma\}^\{\\star,\\sigma\}\)\(s\)\\leq V\_\{\\gamma\}^\{\\star,\\sigma\}\(s\)\+\(1\-\\gamma\)\\max\_\{s^\{\\prime\}\}V\_\{\\gamma\}^\{\\star,\\sigma\}\(s^\{\\prime\}\),\\qquad s\\in\\mathcal\{S\}\.Lemma[1](https://arxiv.org/html/2608.06545#Thmlemma1), applied to each fixed policy withh=Vγ⋆,σh=V\_\{\\gamma\}^\{\\star,\\sigma\}andρ=\(1−γ\)​maxs′⁡Vγ⋆,σ​\(s′\)\\rho=\(1\-\\gamma\)\\max\_\{s^\{\\prime\}\}V\_\{\\gamma\}^\{\\star,\\sigma\}\(s^\{\\prime\}\), gives

ρπ,σ≤\(1−γ\)​maxs′⁡Vγ⋆,σ​\(s′\)\\rho^\{\\pi,\\sigma\}\\leq\(1\-\\gamma\)\\max\_\{s^\{\\prime\}\}V\_\{\\gamma\}^\{\\star,\\sigma\}\(s^\{\\prime\}\)for every stationary policyπ\\pi\. Taking the supremum overπ\\piyields

ρ⋆,σ≤\(1−γ\)​maxs′⁡Vγ⋆,σ​\(s′\)\.\\rho^\{\\star,\\sigma\}\\leq\(1\-\\gamma\)\\max\_\{s^\{\\prime\}\}V\_\{\\gamma\}^\{\\star,\\sigma\}\(s^\{\\prime\}\)\.Here the supremum is over stationary policies\. Finally,

\(1−γ\)​maxs′⁡Vγ⋆,σ​\(s′\)\\displaystyle\(1\-\\gamma\)\\max\_\{s^\{\\prime\}\}V\_\{\\gamma\}^\{\\star,\\sigma\}\(s^\{\\prime\}\)≤\(1−γ\)​mins⁡Vγ⋆,σ​\(s\)\+\(1−γ\)​‖Vγ⋆,σ‖span\\displaystyle\\leq\(1\-\\gamma\)\\min\_\{s\}V\_\{\\gamma\}^\{\\star,\\sigma\}\(s\)\+\(1\-\\gamma\)\\\|V\_\{\\gamma\}^\{\\star,\\sigma\}\\\|\_\{\\mathrm\{span\}\}≤\(1−γ\)​mins⁡Vγ⋆,σ​\(s\)\+2​\(1−γ\)​Hσ,\\displaystyle\\leq\(1\-\\gamma\)\\min\_\{s\}V\_\{\\gamma\}^\{\\star,\\sigma\}\(s\)\+2\(1\-\\gamma\)H\_\{\\sigma\},where the last inequality uses Lemma[3](https://arxiv.org/html/2608.06545#Thmlemma3)\. Together with the preceding bound onρ⋆,σ\\rho^\{\\star,\\sigma\}, this proves \([55](https://arxiv.org/html/2608.06545#A4.E55)\)\.

### D\.2Anchored robust DMDP guarantees

Throughout the discounted analysis, fix a discount factorγ∈\[1/2,1\)\\gamma\\in\[1/2,1\)\.

This subsection first states an anchored discounted MDP error bound with a corresponding target\-accuracy guarantee\. We also state an anchor\-free guarantee obtained from the trivial anchor\.

Recall the nominal anchor condition \([6](https://arxiv.org/html/2608.06545#S4.E6)\)\. The anchor provides a reference average\-reward level whose associated span can sharpen the discounted bound\. For such an anchor pair, define its span scale

Hanc≔max⁡\{1,‖h¯‖span\}\.H\_\{\\mathrm\{anc\}\}\\coloneqq\\max\\left\\\{1,\\\|\\bar\{h\}\\\|\_\{\\mathrm\{span\}\}\\right\\\}\.\(56\)We useR0R\_\{0\}as a common upper bound on the span of the anchor and the true optimal discounted value\. Specifically, let

R0≥max⁡\{1,Hanc,‖Vγ⋆,σ‖span\}\.R\_\{0\}\\geq\\max\\left\\\{1,H\_\{\\mathrm\{anc\}\},\\\|V\_\{\\gamma\}^\{\\star,\\sigma\}\\\|\_\{\\mathrm\{span\}\}\\right\\\}\.\(57\)The anchor defect measures how far the anchor reward level lies above the discounted optimal baseline\. Define

β⋆≔\[ρ¯−\(1−γ\)​mins⁡Vγ⋆,σ​\(s\)\]\+\.\\beta\_\{\\star\}\\coloneqq\\left\[\\bar\{\\rho\}\-\(1\-\\gamma\)\\min\_\{s\}V\_\{\\gamma\}^\{\\star,\\sigma\}\(s\)\\right\]\_\{\+\}\.\(58\)
For a confidence parameterδ∈\(0,1\)\\delta\\in\(0,1\)and an integerN≥1N\\geq 1, define the logarithmic confidence factor

ι≔log⁡\(54​S​A​N2\(1−γ\)​δ\)\.\\iota\\coloneqq\\log\\left\(\\frac\{54SAN^\{2\}\}\{\(1\-\\gamma\)\\delta\}\\right\)\.With this notation in place, the following theorem gives the finite\-sample discounted plug\-in bounds\. Its proof is deferred to Appendix[D\.4](https://arxiv.org/html/2608.06545#A4.SS4)\.

###### Theorem 4\(Anchored robust DMDP plug\-in theorem\)\.

Assume thatℳ\\mathcal\{M\}is a robust discounted MDP described in Section[2](https://arxiv.org/html/2608.06545#S2)with discount factorγ≥1/2\\gamma\\geq 1/2\. Let the anchor quantities\(ρ¯,h¯,Hanc,R0,β⋆\)\(\\bar\{\\rho\},\\bar\{h\},H\_\{\\mathrm\{anc\}\},R\_\{0\},\\beta\_\{\\star\}\)be fixed independently of the transition samples used to construct the empirical nominal kernel\. The anchor may be constructed from an independent data batch, in which case the result applies conditionally on any realization satisfying the stated anchor conditions\. Let a policyπ^\\widehat\{\\pi\}with solver toleranceεopt≥0\\varepsilon\_\{\\mathrm\{opt\}\}\\geq 0satisfy \([5](https://arxiv.org/html/2608.06545#S4.E5)\), \([6](https://arxiv.org/html/2608.06545#S4.E6)\), \([56](https://arxiv.org/html/2608.06545#A4.E56)\), \([57](https://arxiv.org/html/2608.06545#A4.E57)\), and \([58](https://arxiv.org/html/2608.06545#A4.E58)\)\. If, for a sufficiently large universal constantCsamC\_\{\\mathrm\{sam\}\}and a sufficiently small universal constantcc,

N≥Csam​\[ι1−γ\+σ⋅ι\(1−γ\)2\+\(Hanc\+R0​β⋆\)⋅ι\(1−γ\)2​R02\]N\\geq C\_\{\\mathrm\{sam\}\}\\left\[\\frac\{\\iota\}\{1\-\\gamma\}\+\\frac\{\\sigma\\cdot\\iota\}\{\(1\-\\gamma\)^\{2\}\}\+\\frac\{\\left\(H\_\{\\mathrm\{anc\}\}\+R\_\{0\}\\beta\_\{\\star\}\\right\)\\cdot\\iota\}\{\(1\-\\gamma\)^\{2\}R\_\{0\}^\{2\}\}\\right\]\(59\)andεopt≤c​\(1−γ\)​R0\\varepsilon\_\{\\mathrm\{opt\}\}\\leq c\(1\-\\gamma\)R\_\{0\}, then, with probability at least1−O​\(δ\)1\-O\(\\delta\),

‖Vγ⋆,σ−Vγπ^,σ‖∞≤C​\(Hanc\+R0​β⋆\+σ​R02\+\(1−γ\)​R02\)⋅ιN​\(1−γ\)2\+C​R0⋅ιN​\(1−γ\)\+C​εopt1−γ\.\\\|V\_\{\\gamma\}^\{\\star,\\sigma\}\-V\_\{\\gamma\}^\{\\widehat\{\\pi\},\\sigma\}\\\|\_\{\\infty\}\\leq C\\sqrt\{\\frac\{\\left\(H\_\{\\mathrm\{anc\}\}\+R\_\{0\}\\beta\_\{\\star\}\+\\sigma R\_\{0\}^\{2\}\+\(1\-\\gamma\)R\_\{0\}^\{2\}\\right\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\+C\\frac\{R\_\{0\}\\cdot\\iota\}\{N\(1\-\\gamma\)\}\+C\\frac\{\\varepsilon\_\{\\mathrm\{opt\}\}\}\{1\-\\gamma\}\.‖V^γπ^,σ−Vγπ^,σ‖∞≤C​\(Hanc\+R0​β⋆\+σ​R02\+\(1−γ\)​R02\)⋅ιN​\(1−γ\)2\+C​R0⋅ιN​\(1−γ\)\+C​εopt1−γ\.\\\|\\widehat\{V\}\_\{\\gamma\}^\{\\widehat\{\\pi\},\\sigma\}\-V\_\{\\gamma\}^\{\\widehat\{\\pi\},\\sigma\}\\\|\_\{\\infty\}\\leq C\\sqrt\{\\frac\{\\left\(H\_\{\\mathrm\{anc\}\}\+R\_\{0\}\\beta\_\{\\star\}\+\\sigma R\_\{0\}^\{2\}\+\(1\-\\gamma\)R\_\{0\}^\{2\}\\right\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\+C\\frac\{R\_\{0\}\\cdot\\iota\}\{N\(1\-\\gamma\)\}\+C\\frac\{\\varepsilon\_\{\\mathrm\{opt\}\}\}\{1\-\\gamma\}\.

The first corollary converts the theorem’s error bound into a sample\-size condition for a prescribed discounted accuracy\.

###### Corollary 2\(Accuracy form of the anchored robust DMDP guarantee\)\.

Under the model conditions and anchor notation of Theorem[4](https://arxiv.org/html/2608.06545#Thmtheorem4), let0<εγ≤R00<\\varepsilon\_\{\\gamma\}\\leq R\_\{0\}\. For a sufficiently large universal constantCsamC\_\{\\mathrm\{sam\}\}and a sufficiently small universal constantcc, if

N≥Csam​\(Hanc\+R0​β⋆\+σ​R02\+\(1−γ\)​R02\)⋅ι\(1−γ\)2​εγ2\+Csam​R0⋅ι\(1−γ\)​εγN\\geq C\_\{\\mathrm\{sam\}\}\\frac\{\\left\(H\_\{\\mathrm\{anc\}\}\+R\_\{0\}\\beta\_\{\\star\}\+\\sigma R\_\{0\}^\{2\}\+\(1\-\\gamma\)R\_\{0\}^\{2\}\\right\)\\cdot\\iota\}\{\(1\-\\gamma\)^\{2\}\\varepsilon\_\{\\gamma\}^\{2\}\}\+C\_\{\\mathrm\{sam\}\}\\frac\{R\_\{0\}\\cdot\\iota\}\{\(1\-\\gamma\)\\varepsilon\_\{\\gamma\}\}\(60\)andεopt≤c​\(1−γ\)​εγ\\varepsilon\_\{\\mathrm\{opt\}\}\\leq c\(1\-\\gamma\)\\varepsilon\_\{\\gamma\}, then, with probability at least1−O​\(δ\)1\-O\(\\delta\),

‖Vγ⋆,σ−Vγπ^,σ‖∞≤εγ\.\\\|V\_\{\\gamma\}^\{\\star,\\sigma\}\-V\_\{\\gamma\}^\{\\widehat\{\\pi\},\\sigma\}\\\|\_\{\\infty\}\\leq\\varepsilon\_\{\\gamma\}\.

###### Proof\.

Sinceεγ≤R0\\varepsilon\_\{\\gamma\}\\leq R\_\{0\}, \([60](https://arxiv.org/html/2608.06545#A4.E60)\) implies the sample\-size condition \([59](https://arxiv.org/html/2608.06545#A4.E59)\) and

εopt≤c​\(1−γ\)​εγ≤c​\(1−γ\)​R0,\\varepsilon\_\{\\mathrm\{opt\}\}\\leq c\(1\-\\gamma\)\\varepsilon\_\{\\gamma\}\\leq c\(1\-\\gamma\)R\_\{0\},so the condition onεopt\\varepsilon\_\{\\mathrm\{opt\}\}in Theorem[4](https://arxiv.org/html/2608.06545#Thmtheorem4)is satisfied\. Substituting \([60](https://arxiv.org/html/2608.06545#A4.E60)\) into the remaining terms in the theorem’s error bound and then choosingCsamC\_\{\\mathrm\{sam\}\}sufficiently large andccsufficiently small proves the claim\. ∎

The following corollary yields a span\-only guarantee when no informative anchor is available\.

###### Corollary 3\(Span\-only robust DMDP guarantee\)\.

Letεγ\>0\\varepsilon\_\{\\gamma\}\>0be the target discounted accuracy\. Assume thatℳ\\mathcal\{M\}is a robust discounted MDP described in Section[2](https://arxiv.org/html/2608.06545#S2)with discount factorγ≥1/2\\gamma\\geq 1/2, and letπ^\\widehat\{\\pi\}satisfy the solver guarantee \([5](https://arxiv.org/html/2608.06545#S4.E5)\) with solver toleranceεopt≥0\\varepsilon\_\{\\mathrm\{opt\}\}\\geq 0\. ChooseR0≥max⁡\{1,‖Vγ⋆,σ‖span\}R\_\{0\}\\geq\\max\\\{1,\\\|V\_\{\\gamma\}^\{\\star,\\sigma\}\\\|\_\{\\mathrm\{span\}\}\\\}\. Ifεγ≤R0\\varepsilon\_\{\\gamma\}\\leq R\_\{0\}and, for a sufficiently large universal constantCsamC\_\{\\mathrm\{sam\}\}and a sufficiently small universal constantcc,

N≥Csam​\(R0\+σ​R02\+\(1−γ\)​R02\)⋅ι\(1−γ\)2​εγ2\+Csam​R0⋅ι\(1−γ\)​εγ,N\\geq C\_\{\\mathrm\{sam\}\}\\frac\{\\left\(R\_\{0\}\+\\sigma R\_\{0\}^\{2\}\+\(1\-\\gamma\)R\_\{0\}^\{2\}\\right\)\\cdot\\iota\}\{\(1\-\\gamma\)^\{2\}\\varepsilon\_\{\\gamma\}^\{2\}\}\+C\_\{\\mathrm\{sam\}\}\\frac\{R\_\{0\}\\cdot\\iota\}\{\(1\-\\gamma\)\\varepsilon\_\{\\gamma\}\},andεopt≤c​\(1−γ\)​εγ\\varepsilon\_\{\\mathrm\{opt\}\}\\leq c\(1\-\\gamma\)\\varepsilon\_\{\\gamma\}, then, with probability at least1−O​\(δ\)1\-O\(\\delta\),

‖Vγ⋆,σ−Vγπ^,σ‖∞≤εγ\.\\\|V\_\{\\gamma\}^\{\\star,\\sigma\}\-V\_\{\\gamma\}^\{\\widehat\{\\pi\},\\sigma\}\\\|\_\{\\infty\}\\leq\\varepsilon\_\{\\gamma\}\.

###### Proof\.

Apply Corollary[2](https://arxiv.org/html/2608.06545#Thmcorollary2)with the trivial anchorρ¯=1\\bar\{\\rho\}=1andh¯=0\\bar\{h\}=0\. Sincer​\(s,a\)∈\[0,1\]r\(s,a\)\\in\[0,1\], this pair satisfies the nominal anchor supersolution condition \([6](https://arxiv.org/html/2608.06545#S4.E6)\) and hasHanc=1H\_\{\\mathrm\{anc\}\}=1\. Moreover, nonnegativity of the rewards gives

β⋆=\[1−\(1−γ\)​mins⁡Vγ⋆,σ​\(s\)\]\+≤1\.\\beta\_\{\\star\}=\\left\[1\-\(1\-\\gamma\)\\min\_\{s\}V\_\{\\gamma\}^\{\\star,\\sigma\}\(s\)\\right\]\_\{\+\}\\leq 1\.Hence

Hanc\+R0​β⋆\+σ​R02\+\(1−γ\)​R02≤2​R0\+σ​R02\+\(1−γ\)​R02,H\_\{\\mathrm\{anc\}\}\+R\_\{0\}\\beta\_\{\\star\}\+\\sigma R\_\{0\}^\{2\}\+\(1\-\\gamma\)R\_\{0\}^\{2\}\\leq 2R\_\{0\}\+\\sigma R\_\{0\}^\{2\}\+\(1\-\\gamma\)R\_\{0\}^\{2\},whereR0≥1R\_\{0\}\\geq 1\. Absorbing the factor22intoCsamC\_\{\\mathrm\{sam\}\}in Corollary[2](https://arxiv.org/html/2608.06545#Thmcorollary2)proves the claim\. ∎

### D\.3Proof of Theorem[2](https://arxiv.org/html/2608.06545#Thmtheorem2)

The proof follows the two branches of Algorithm[1](https://arxiv.org/html/2608.06545#alg1)\. We first state a guarantee for the nominal branch, whose proof is deferred to Appendix[D\.5\.1](https://arxiv.org/html/2608.06545#A4.SS5.SSS1)\. We then handle the robust branch by combining the discounted guarantees in Appendix[D\.2](https://arxiv.org/html/2608.06545#A4.SS2)with the discounted\-to\-average reduction in Appendix[D\.1](https://arxiv.org/html/2608.06545#A4.SS1)\.

##### Step 1: Nominal branch\.

Suppose thatH0<HσH\_\{0\}<H\_\{\\sigma\}and7​σ​H0≤ε7\\sigma H\_\{0\}\\leq\\varepsilon, so Algorithm[1](https://arxiv.org/html/2608.06545#alg1)selects the nominal branch\. In this case, we give the following guarantee for the nominal branch, whose proof is deferred to Appendix[D\.5\.1](https://arxiv.org/html/2608.06545#A4.SS5.SSS1)\.

###### Lemma 11\(Nominal\-branch guarantee\)\.

There exist a sufficiently large universal constantC\>0C\>0and a sufficiently small universal constantcopt\>0c\_\{\\mathrm\{opt\}\}\>0such that the following holds\. Suppose Assumption[1](https://arxiv.org/html/2608.06545#Thmassumption1)holds andH0H\_\{0\}is known\. Letε∈\(0,1\]\\varepsilon\\in\(0,1\]andδ∈\(0,1/2\]\\delta\\in\(0,1/2\], and assume that7​σ​H0≤ε7\\sigma H\_\{0\}\\leq\\varepsilonandεopt≤copt​ε\\varepsilon\_\{\\mathrm\{opt\}\}\\leq c\_\{\\mathrm\{opt\}\}\\varepsilon\. If the total sample size obeys

N​S​A≥C​S​A​H0ε2​log⁡\(S​A​H0​Nε​δ\),NSA\\geq CSA\\frac\{H\_\{0\}\}\{\\varepsilon^\{2\}\}\\log\\\!\\left\(\\frac\{SAH\_\{0\}N\}\{\\varepsilon\\delta\}\\right\),then, with probability at least1−O​\(δ\)1\-O\(\\delta\), Algorithm[1](https://arxiv.org/html/2608.06545#alg1)returns a policyπ^\\widehat\{\\pi\}that satisfies

ρ⋆,σ−ρπ^,σ≤ε\.\\rho^\{\\star,\\sigma\}\-\\rho^\{\\widehat\{\\pi\},\\sigma\}\\leq\\varepsilon\.

Sincemin⁡\{H0,Hσ\}=H0\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}=H\_\{0\}in this case, condition \(a\) in Theorem[2](https://arxiv.org/html/2608.06545#Thmtheorem2)implies the requirement of Lemma[11](https://arxiv.org/html/2608.06545#Thmlemma11)\.

##### Step 2: Robust branch whenH0<HσH\_\{0\}<H\_\{\\sigma\}and7​σ​H0\>ε7\\sigma H\_\{0\}\>\\varepsilon\.

We use Corollary[2](https://arxiv.org/html/2608.06545#Thmcorollary2)whenH0<HσH\_\{0\}<H\_\{\\sigma\}and7​σ​H0\>ε7\\sigma H\_\{0\}\>\\varepsilon, and Corollary[3](https://arxiv.org/html/2608.06545#Thmcorollary3)whenHσ≤H0H\_\{\\sigma\}\\leq H\_\{0\}\. We first verify the parameter choices shared by Steps 2 and 3\. Recall that in the robust branch, Algorithm[1](https://arxiv.org/html/2608.06545#alg1)sets

γ=1−ε3​Hσ\.\\gamma=1\-\\frac\{\\varepsilon\}\{3H\_\{\\sigma\}\}\.Sinceε≤1\\varepsilon\\leq 1andHσ≥1H\_\{\\sigma\}\\geq 1, this choice satisfiesγ≥2/3\\gamma\\geq 2/3\. For Steps 2 and 3, we use the target discounted accuracyεγ=Hσ\\varepsilon\_\{\\gamma\}=H\_\{\\sigma\}\. The solver tolerance satisfies

εopt≤copt​ε=3​copt​\(1−γ\)​Hσ\.\\varepsilon\_\{\\mathrm\{opt\}\}\\leq c\_\{\\mathrm\{opt\}\}\\varepsilon=3c\_\{\\mathrm\{opt\}\}\(1\-\\gamma\)H\_\{\\sigma\}\.By the choice ofcoptc\_\{\\mathrm\{opt\}\}, this satisfies theεopt\\varepsilon\_\{\\mathrm\{opt\}\}condition in both corollaries\. Moreover, since1/\(1−γ\)=3​Hσ/ε1/\(1\-\\gamma\)=3H\_\{\\sigma\}/\\varepsilonandδ≤1/2\\delta\\leq 1/2, the definition ofι\\iotagives

ι≤C​log⁡\(Hσ​S​A​Nε​δ\)\.\\iota\\leq C\\log\\left\(\\frac\{H\_\{\\sigma\}SAN\}\{\\varepsilon\\delta\}\\right\)\.
We now assumeH0<HσH\_\{0\}<H\_\{\\sigma\}and7​σ​H0\>ε7\\sigma H\_\{0\}\>\\varepsilon\. In this case, we apply Corollary[2](https://arxiv.org/html/2608.06545#Thmcorollary2)\. Set the anchor pair to be the nominal optimal average\-reward pair

ρ¯=ρ⋆,h¯=hP0⋆\.\\bar\{\\rho\}=\\rho^\{\\star\},\\qquad\\bar\{h\}=h\_\{P^\{0\}\}^\{\\star\}\.By the nominal average\-reward Bellman optimality equation,

ρ⋆\+hP0⋆​\(s\)=maxa∈𝒜⁡\{r​\(s,a\)\+Ps,a0​hP0⋆\},s∈𝒮\.\\rho^\{\\star\}\+h\_\{P^\{0\}\}^\{\\star\}\(s\)=\\max\_\{a\\in\\mathcal\{A\}\}\\left\\\{r\(s,a\)\+P^\{0\}\_\{s,a\}h\_\{P^\{0\}\}^\{\\star\}\\right\\\},\\qquad s\\in\\mathcal\{S\}\.Thus the nominal anchor supersolution condition \([6](https://arxiv.org/html/2608.06545#S4.E6)\) holds\. SinceH0≥1H\_\{0\}\\geq 1by the standing convention in Section[2](https://arxiv.org/html/2608.06545#S2), this anchor hasHanc=H0H\_\{\\mathrm\{anc\}\}=H\_\{0\}\.

We take the reference radius

R0=2​Hσ\.R\_\{0\}=2H\_\{\\sigma\}\.By Lemma[3](https://arxiv.org/html/2608.06545#Thmlemma3),

‖Vγ⋆,σ‖span≤2​Hσ\.\\\|V\_\{\\gamma\}^\{\\star,\\sigma\}\\\|\_\{\\mathrm\{span\}\}\\leq 2H\_\{\\sigma\}\.SinceHσ\>H0≥1H\_\{\\sigma\}\>H\_\{0\}\\geq 1, this gives

R0≥max⁡\{1,Hanc,‖Vγ⋆,σ‖span\}\.R\_\{0\}\\geq\\max\\left\\\{1,H\_\{\\mathrm\{anc\}\},\\\|V\_\{\\gamma\}^\{\\star,\\sigma\}\\\|\_\{\\mathrm\{span\}\}\\right\\\}\.We next bound the anchor defectβ⋆\\beta\_\{\\star\}\. Sinceρ⋆−ρ⋆,σ≥0\\rho^\{\\star\}\-\\rho^\{\\star,\\sigma\}\\geq 0, the inequality\[x\+y\]\+≤x\+\[y\]\+\[x\+y\]\_\{\+\}\\leq x\+\[y\]\_\{\+\}forx≥0x\\geq 0gives

β⋆\\displaystyle\\beta\_\{\\star\}=\[ρ⋆−\(1−γ\)​mins⁡Vγ⋆,σ​\(s\)\]\+\\displaystyle=\\left\[\\rho^\{\\star\}\-\(1\-\\gamma\)\\min\_\{s\}V\_\{\\gamma\}^\{\\star,\\sigma\}\(s\)\\right\]\_\{\+\}≤ρ⋆−ρ⋆,σ\+\[ρ⋆,σ−\(1−γ\)​mins⁡Vγ⋆,σ​\(s\)\]\+\\displaystyle\\leq\\rho^\{\\star\}\-\\rho^\{\\star,\\sigma\}\+\\left\[\\rho^\{\\star,\\sigma\}\-\(1\-\\gamma\)\\min\_\{s\}V\_\{\\gamma\}^\{\\star,\\sigma\}\(s\)\\right\]\_\{\+\}≤C​\[σ​H0\+\(1−γ\)​Hσ\]\.\\displaystyle\\leq C\\left\[\\sigma H\_\{0\}\+\(1\-\\gamma\)H\_\{\\sigma\}\\right\]\.Here the last line uses Proposition[2](https://arxiv.org/html/2608.06545#Thmproposition2)and the robust discounted\-to\-average optimal comparison \([55](https://arxiv.org/html/2608.06545#A4.E55)\) in Lemma[9](https://arxiv.org/html/2608.06545#Thmlemma9)\.

UsingHanc=H0H\_\{\\mathrm\{anc\}\}=H\_\{0\},R0=2​HσR\_\{0\}=2H\_\{\\sigma\}, the preceding bound onβ⋆\\beta\_\{\\star\}, and1−γ=ε/\(3​Hσ\)1\-\\gamma=\\varepsilon/\(3H\_\{\\sigma\}\), we obtain

Hanc\+R0​β⋆\+σ​R02\+\(1−γ\)​R02≤C​\[H0\+σ​Hσ2\+ε​Hσ\]\.H\_\{\\mathrm\{anc\}\}\+R\_\{0\}\\beta\_\{\\star\}\+\\sigma R\_\{0\}^\{2\}\+\(1\-\\gamma\)R\_\{0\}^\{2\}\\leq C\\left\[H\_\{0\}\+\\sigma H\_\{\\sigma\}^\{2\}\+\\varepsilon H\_\{\\sigma\}\\right\]\.
We now check the sample\-size condition in Corollary[2](https://arxiv.org/html/2608.06545#Thmcorollary2)\. Withεγ=Hσ\\varepsilon\_\{\\gamma\}=H\_\{\\sigma\}andR0=2​HσR\_\{0\}=2H\_\{\\sigma\}, it is enough to have

N≥C​\[H0\+σ​Hσ2ε2\+Hσε\]⋅ι\.N\\geq C\\left\[\\frac\{H\_\{0\}\+\\sigma H\_\{\\sigma\}^\{2\}\}\{\\varepsilon^\{2\}\}\+\\frac\{H\_\{\\sigma\}\}\{\\varepsilon\}\\right\]\\cdot\\iota\.SinceH0<HσH\_\{0\}<H\_\{\\sigma\}and7​σ​H0\>ε7\\sigma H\_\{0\}\>\\varepsilon,

Hσε≤7​σ​H0​Hσε2≤7​σ​Hσ2ε2\.\\frac\{H\_\{\\sigma\}\}\{\\varepsilon\}\\leq 7\\frac\{\\sigma H\_\{0\}H\_\{\\sigma\}\}\{\\varepsilon^\{2\}\}\\leq 7\\frac\{\\sigma H\_\{\\sigma\}^\{2\}\}\{\\varepsilon^\{2\}\}\.Together with the preceding bound onι\\iota, condition \(b\) in Theorem[2](https://arxiv.org/html/2608.06545#Thmtheorem2)therefore implies the requirement of Corollary[2](https://arxiv.org/html/2608.06545#Thmcorollary2)\.

##### Step 3: Robust branch whenHσ≤H0H\_\{\\sigma\}\\leq H\_\{0\}\.

In this case, we apply Corollary[3](https://arxiv.org/html/2608.06545#Thmcorollary3)with

R0=2​Hσ\.R\_\{0\}=2H\_\{\\sigma\}\.By Lemma[3](https://arxiv.org/html/2608.06545#Thmlemma3), this choice satisfies the reference\-radius requirement\. Substitutingεγ=Hσ\\varepsilon\_\{\\gamma\}=H\_\{\\sigma\},min⁡\{H0,Hσ\}=Hσ\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}=H\_\{\\sigma\}, and1−γ=ε/\(3​Hσ\)1\-\\gamma=\\varepsilon/\(3H\_\{\\sigma\}\)into the sample\-size condition of Corollary[3](https://arxiv.org/html/2608.06545#Thmcorollary3), it is enough that

N≥C​\[\(Hσ\+σ​Hσ2\+ε​Hσ\)⋅ιε2\+Hσ⋅ιε\]\.N\\geq C\\left\[\\frac\{\\left\(H\_\{\\sigma\}\+\\sigma H\_\{\\sigma\}^\{2\}\+\\varepsilon H\_\{\\sigma\}\\right\)\\cdot\\iota\}\{\\varepsilon^\{2\}\}\+\\frac\{H\_\{\\sigma\}\\cdot\\iota\}\{\\varepsilon\}\\right\]\.Sinceε≤1\\varepsilon\\leq 1, we can further simplify it to

N≥C​\[Hσ\+σ​Hσ2ε2\]⋅ι\.N\\geq C\\left\[\\frac\{H\_\{\\sigma\}\+\\sigma H\_\{\\sigma\}^\{2\}\}\{\\varepsilon^\{2\}\}\\right\]\\cdot\\iota\.If7​σ​H0\>ε7\\sigma H\_\{0\}\>\\varepsilon, condition \(b\) in Theorem[2](https://arxiv.org/html/2608.06545#Thmtheorem2)implies this requirement\. If7​σ​H0≤ε7\\sigma H\_\{0\}\\leq\\varepsilon, then

σ​Hσ2≤σ​H0​Hσ≤ε​Hσ7≤Hσ7\.\\sigma H\_\{\\sigma\}^\{2\}\\leq\\sigma H\_\{0\}H\_\{\\sigma\}\\leq\\frac\{\\varepsilon H\_\{\\sigma\}\}\{7\}\\leq\\frac\{H\_\{\\sigma\}\}\{7\}\.Condition \(a\) in Theorem[2](https://arxiv.org/html/2608.06545#Thmtheorem2)therefore implies the same requirement\.

Thus, under either span ordering, the sample\-size condition in Theorem[2](https://arxiv.org/html/2608.06545#Thmtheorem2)implies the requirement of the relevant discounted corollary\. We conclude that with probability at least1−O​\(δ\)1\-O\(\\delta\),

Vγ⋆,σ−Vγπ^,σ≤Hσ⋅𝟏S\.V\_\{\\gamma\}^\{\\star,\\sigma\}\-V\_\{\\gamma\}^\{\\widehat\{\\pi\},\\sigma\}\\leq H\_\{\\sigma\}\\cdot\\bm\{1\}\_\{S\}\.
On the same probability event, invoke Corollary[1](https://arxiv.org/html/2608.06545#Thmcorollary1)\. Since the discounted suboptimality level isHσH\_\{\\sigma\}, the corollary gives

ρ⋆,σ−ρπ^,σ≤ε\.\\rho^\{\\star,\\sigma\}\-\\rho^\{\\widehat\{\\pi\},\\sigma\}\\leq\\varepsilon\.

### D\.4Proof of Theorem[4](https://arxiv.org/html/2608.06545#Thmtheorem4)

We prove Theorem[4](https://arxiv.org/html/2608.06545#Thmtheorem4)in this section\. We first collect the notation and introduce the localization idea used throughout the proof\.

#### D\.4\.1Preliminaries and notation

We begin with the standard value, kernel, and variance notation\. The proof\-specific localization radius and concentration budgets are introduced afterward\.

##### Value functions and transition kernels\.

While some of the following objects are introduced elsewhere, we restate them here for convenience\. For a policyπ\\pi,Vγπ,σV\_\{\\gamma\}^\{\\pi,\\sigma\}denotes its robust discounted value under the true uncertainty set𝒰​\(P0\)\\mathcal\{U\}\(P^\{0\}\), andV^γπ,σ\\widehat\{V\}\_\{\\gamma\}^\{\\pi,\\sigma\}denotes the corresponding robust discounted value under the empirical uncertainty set𝒰​\(P^0\)\\mathcal\{U\}\(\\widehat\{P\}^\{0\}\)\. We write

Vγ⋆,σ=supπVγπ,σ,V^γ⋆,σ=supπV^γπ,σ\.V\_\{\\gamma\}^\{\\star,\\sigma\}=\\sup\_\{\\pi\}V\_\{\\gamma\}^\{\\pi,\\sigma\},\\qquad\\widehat\{V\}\_\{\\gamma\}^\{\\star,\\sigma\}=\\sup\_\{\\pi\}\\widehat\{V\}\_\{\\gamma\}^\{\\pi,\\sigma\}\.Letπγ⋆\\pi\_\{\\gamma\}^\{\\star\}be a deterministic optimal policy for the true robust discounted MDP\. For the fixed\-policy and learned\-policy comparisons used below, define

U≔Vγ⋆,σ,U^≔V^γπγ⋆,σ,V^⋆≔V^γ⋆,σ,W≔Vγπ^,σ,W^≔V^γπ^,σ\.U\\coloneqq V\_\{\\gamma\}^\{\\star,\\sigma\},\\qquad\\widehat\{U\}\\coloneqq\\widehat\{V\}\_\{\\gamma\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\sigma\},\\qquad\\widehat\{V\}^\{\\star\}\\coloneqq\\widehat\{V\}\_\{\\gamma\}^\{\\star,\\sigma\},\\qquad W\\coloneqq V\_\{\\gamma\}^\{\\widehat\{\\pi\},\\sigma\},\\qquad\\widehat\{W\}\\coloneqq\\widehat\{V\}\_\{\\gamma\}^\{\\widehat\{\\pi\},\\sigma\}\.Thus,UUandU^\\widehat\{U\}are compared under the fixed true\-optimal policy, whereasWWandW^\\widehat\{W\}are compared under the learned policy\. The valueV^⋆\\widehat\{V\}^\{\\star\}is the optimal value of the empirical robust MDP and links these two comparisons through theεopt\\varepsilon\_\{\\mathrm\{opt\}\}condition in \([5](https://arxiv.org/html/2608.06545#S4.E5)\)\. Recall that the empirical nominal kernel is denoted byP^0\\widehat\{P\}^\{0\}and is constructed fromNNsamples per state\-action pair\. For a policyπ\\pi, define the policy\-induced reward vector

rπ​\(s\)≔∑a∈𝒜π​\(a∣s\)​r​\(s,a\)\.r^\{\\pi\}\(s\)\\coloneqq\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\(a\\mid s\)r\(s,a\)\.For vectorsx,yx,y,x∘yx\\circ ydenotes entrywise multiplication, andf​\(x\)f\(x\)is applied entrywise for any scalar functionff, e\.g\.,\|x\|\|x\|\.

For a transition matrixPP,VarP​\(V\)\\mathrm\{Var\}\_\{P\}\(V\)denotes the variance vector

VarP​\(V\)​\(s\)≔P​\(V∘V\)​\(s\)−\(P​V\)∘\(P​V\)​\(s\)\.\\mathrm\{Var\}\_\{P\}\(V\)\(s\)\\coloneqq P\(V\\circ V\)\(s\)\-\(PV\)\\circ\(PV\)\(s\)\.
For a value vectorVV, letPs,aVP\_\{s,a\}^\{V\}andP^s,aV\\widehat\{P\}\_\{s,a\}^\{V\}denote worst\-case transition distributions selected from the true and empirical uncertainty sets, respectively:

Ps,aV∈arg⁡minQ∈𝒰s,a​\(P0\)⁡Q​V,P^s,aV∈arg⁡minQ∈𝒰s,a​\(P^0\)⁡Q​V\.P\_\{s,a\}^\{V\}\\in\\arg\\min\_\{Q\\in\\mathcal\{U\}\_\{s,a\}\(P^\{0\}\)\}QV,\\qquad\\widehat\{P\}\_\{s,a\}^\{V\}\\in\\arg\\min\_\{Q\\in\\mathcal\{U\}\_\{s,a\}\(\\widehat\{P\}^\{0\}\)\}QV\.For a policyπ\\pi, we denote the induced robust transition matrices by

Pπ,V​\(s,s′\)≔∑a∈𝒜π​\(a∣s\)​Ps,aV​\(s′\),P^π,V​\(s,s′\)≔∑a∈𝒜π​\(a∣s\)​P^s,aV​\(s′\)\.P^\{\\pi,V\}\(s,s^\{\\prime\}\)\\coloneqq\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\(a\\mid s\)P\_\{s,a\}^\{V\}\(s^\{\\prime\}\),\\qquad\\widehat\{P\}^\{\\pi,V\}\(s,s^\{\\prime\}\)\\coloneqq\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\(a\\mid s\)\\widehat\{P\}\_\{s,a\}^\{V\}\(s^\{\\prime\}\)\.Similarly, the nominal policy\-induced transition matrices are

P0,π​\(s,s′\)≔∑a∈𝒜π​\(a∣s\)​Ps,a0​\(s′\),P^0,π​\(s,s′\)≔∑a∈𝒜π​\(a∣s\)​P^s,a0​\(s′\)\.P^\{0,\\pi\}\(s,s^\{\\prime\}\)\\coloneqq\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\(a\\mid s\)P^\{0\}\_\{s,a\}\(s^\{\\prime\}\),\\qquad\\widehat\{P\}^\{0,\\pi\}\(s,s^\{\\prime\}\)\\coloneqq\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\(a\\mid s\)\\widehat\{P\}^\{0\}\_\{s,a\}\(s^\{\\prime\}\)\.

##### Localization radius and radius grid\.

The theorem aims for a bound in terms of the reference radiusR0R\_\{0\}, which controls the true optimal span\. The spans of the empirical value functions, however, are random\. We therefore treatRRas a candidate upper bound on all relevant spans\. At any fixed radiusRR, the localized analysis assumes that all relevant spans are at mostRRand controls the corresponding estimation errors at a scale depending onRR\.

We do not know in advance which candidate radius is large enough\. We therefore consider the dyadic grid

Rj≔2j​R0,j=0,…,J,J≔⌈log2⁡81−γ⌉,ℛ≔\{Rj:0≤j≤J\},R\_\{j\}\\coloneqq 2^\{j\}R\_\{0\},\\qquad j=0,\\ldots,J,\\qquad J\\coloneqq\\left\\lceil\\log\_\{2\}\\frac\{8\}\{1\-\\gamma\}\\right\\rceil,\\qquad\\mathcal\{R\}\\coloneqq\\\{R\_\{j\}:0\\leq j\\leq J\\\},\(61\)whereR0R\_\{0\}is the reference radius defined in \([57](https://arxiv.org/html/2608.06545#A4.E57)\)\. The grid starts atR0R\_\{0\}and extends beyond the worst\-case discounted value\-function span bound of order\(1−γ\)−1\(1\-\\gamma\)^\{\-1\}\. At each grid pointRjR\_\{j\}, this localized analysis provides error bounds whenever the relevant spans are at mostRjR\_\{j\}\. These error bounds then control the relevant spans in turn\. We take the first grid point at which this cycle of bounds closes and yields valid error control\. Because neighboring grid points differ only by a factor of two, the selected radius remains close to the smallest validRR\.

##### Localized concentration budgets\.

The quantities entering the localized bounds depend on this radius\. We lay out the definitions here for convenience in later presentations\.

In addition to the true anchor defectβ⋆\\beta\_\{\\star\}from \([58](https://arxiv.org/html/2608.06545#A4.E58)\), define the corresponding defect for the empirical policy by

βemp≔\[ρ¯−\(1−γ\)​mins⁡W^​\(s\)\]\+\.\\beta\_\{\\mathrm\{emp\}\}\\coloneqq\\left\[\\bar\{\\rho\}\-\(1\-\\gamma\)\\min\_\{s\}\\widehat\{W\}\(s\)\\right\]\_\{\+\}\.\(62\)The variance bounds repeatedly involve four contributions: the anchor scaleHancH\_\{\\mathrm\{anc\}\}, the anchor defect multiplied by the span radius, the robustness contributionσ​R2\\sigma R^\{2\}, and the discounted contribution\(1−γ\)​R2\(1\-\\gamma\)R^\{2\}\. For a generic defectβ\\beta, collect them in

B​\(R,β\)≔Hanc\+R​β\+σ​R2\+\(1−γ\)​R2\.B\(R,\\beta\)\\coloneqq H\_\{\\mathrm\{anc\}\}\+R\\beta\+\\sigma R^\{2\}\+\(1\-\\gamma\)R^\{2\}\.\(63\)The budgets for the true optimal value, the empirical policy, and their combination are the specializations

B⋆​\(R\)≔B​\(R,β⋆\),Bemp​\(R\)≔B​\(R,βemp\),Bcom​\(R\)≔B​\(R,β⋆\+βemp\)\.B\_\{\\star\}\(R\)\\coloneqq B\(R,\\beta\_\{\\star\}\),\\qquad B\_\{\\mathrm\{emp\}\}\(R\)\\coloneqq B\(R,\\beta\_\{\\mathrm\{emp\}\}\),\\qquad B\_\{\\mathrm\{com\}\}\(R\)\\coloneqq B\(R,\\beta\_\{\\star\}\+\\beta\_\{\\mathrm\{emp\}\}\)\.\(64\)

##### High\-probability event\.

For any deterministic radiusR≥max⁡\{1,Hanc\}R\\geq\\max\\\{1,H\_\{\\mathrm\{anc\}\}\\\}, letℰ​\(R\)\\mathcal\{E\}\(R\)be the intersection of the events in Lemmas[15](https://arxiv.org/html/2608.06545#Thmlemma15),[16](https://arxiv.org/html/2608.06545#Thmlemma16),[21](https://arxiv.org/html/2608.06545#Thmlemma21), and[22](https://arxiv.org/html/2608.06545#Thmlemma22), where Lemma[21](https://arxiv.org/html/2608.06545#Thmlemma21)is applied to bothUUandh¯\\bar\{h\}, and Lemma[22](https://arxiv.org/html/2608.06545#Thmlemma22)is instantiated at radiusRR\. For the gridℛ\\mathcal\{R\}in \([61](https://arxiv.org/html/2608.06545#A4.E61)\), we set the confidence parameter of each event to beδ/\(J\+1\)\\delta/\(J\+1\)\.

Since only a constant number of concentration statements is used at each radius,

ℙ​\{ℰ​\(R\)c\}≤C​δJ\+1,R∈ℛ\.\\mathbb\{P\}\\\!\\left\\\{\\mathcal\{E\}\(R\)^\{c\}\\right\\\}\\leq\\frac\{C\\delta\}\{J\+1\},\\qquad R\\in\\mathcal\{R\}\.Define the global event

ℰ≔⋂R∈ℛℰ​\(R\)\.\\mathcal\{E\}\\coloneqq\\bigcap\_\{R\\in\\mathcal\{R\}\}\\mathcal\{E\}\(R\)\.\(65\)A union bound over theJ\+1J\+1grid points givesℙ​\{ℰ\}≥1−O​\(δ\)\\mathbb\{P\}\\\{\\mathcal\{E\}\\\}\\geq 1\-O\(\\delta\)\. Moreover, the definition ofJJin \([61](https://arxiv.org/html/2608.06545#A4.E61)\) gives

log⁡\(54​S​A​N2​\(J\+1\)\(1−γ\)​δ\)≤C⋅ι,\\log\\\!\\left\(\\frac\{54SAN^\{2\}\(J\+1\)\}\{\(1\-\\gamma\)\\delta\}\\right\)\\leq C\\cdot\\iota,so the additional factorJ\+1J\+1caused by this confidence allocation is absorbed intoι\\iotaafter adjusting the universal constants\.

#### D\.4\.2Proof of the main error bound

Recall that

U≔Vγ⋆,σ,U^≔V^γπγ⋆,σ,V^⋆≔V^γ⋆,σ,W≔Vγπ^,σ,W^≔V^γπ^,σ\.U\\coloneqq V\_\{\\gamma\}^\{\\star,\\sigma\},\\qquad\\widehat\{U\}\\coloneqq\\widehat\{V\}\_\{\\gamma\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\sigma\},\\qquad\\widehat\{V\}^\{\\star\}\\coloneqq\\widehat\{V\}\_\{\\gamma\}^\{\\star,\\sigma\},\\qquad W\\coloneqq V\_\{\\gamma\}^\{\\widehat\{\\pi\},\\sigma\},\\qquad\\widehat\{W\}\\coloneqq\\widehat\{V\}\_\{\\gamma\}^\{\\widehat\{\\pi\},\\sigma\}\.
We start the proof by decomposing the gap betweenWWandUU\. Observe that for anys∈𝒮s\\in\\mathcal\{S\},

U​\(s\)−W​\(s\)\\displaystyle U\(s\)\-W\(s\)=U​\(s\)−U^​\(s\)\+U^​\(s\)−V^⋆​\(s\)\+V^⋆​\(s\)−W^​\(s\)\+W^​\(s\)−W​\(s\)\\displaystyle=U\(s\)\-\\widehat\{U\}\(s\)\+\\widehat\{U\}\(s\)\-\\widehat\{V\}^\{\\star\}\(s\)\+\\widehat\{V\}^\{\\star\}\(s\)\-\\widehat\{W\}\(s\)\+\\widehat\{W\}\(s\)\-W\(s\)≤‖U^−U‖∞\+εopt\+‖W^−W‖∞\.\\displaystyle\\leq\\\|\\widehat\{U\}\-U\\\|\_\{\\infty\}\+\\varepsilon\_\{\\mathrm\{opt\}\}\+\\\|\\widehat\{W\}\-W\\\|\_\{\\infty\}\.\(66\)Here, the last line uses the solver guarantee \([5](https://arxiv.org/html/2608.06545#S4.E5)\), and the fact thatU^−V^⋆≤0\\widehat\{U\}\-\\widehat\{V\}^\{\\star\}\\leq 0sinceV^⋆\\widehat\{V\}^\{\\star\}is the optimal value function under the estimated kernel\.

Define the actual comparison error

Δ≔‖W^−W‖∞\+‖U^−U‖∞\.\\Delta\\coloneqq\\\|\\widehat\{W\}\-W\\\|\_\{\\infty\}\+\\\|\\widehat\{U\}\-U\\\|\_\{\\infty\}\.\(67\)The decomposition \([66](https://arxiv.org/html/2608.06545#A4.E66)\) and \([67](https://arxiv.org/html/2608.06545#A4.E67)\) allow us to characterize the span of the relevant value functions as in the following lemma\. The proof is deferred to Appendix[D\.5\.2](https://arxiv.org/html/2608.06545#A4.SS5.SSS2)\.

###### Lemma 12\.

Under \([57](https://arxiv.org/html/2608.06545#A4.E57)\) and the solver guarantee \([5](https://arxiv.org/html/2608.06545#S4.E5)\), letΔ\\Deltabe defined by \([67](https://arxiv.org/html/2608.06545#A4.E67)\)\. Then

‖W‖span,‖W^‖span,‖U^‖span,‖V^⋆‖span≤R0\+2​Δ\+2​εopt\.\\\|W\\\|\_\{\\mathrm\{span\}\},\\quad\\\|\\widehat\{W\}\\\|\_\{\\mathrm\{span\}\},\\quad\\\|\\widehat\{U\}\\\|\_\{\\mathrm\{span\}\},\\quad\\\|\\widehat\{V\}^\{\\star\}\\\|\_\{\\mathrm\{span\}\}\\leq R\_\{0\}\+2\\Delta\+2\\varepsilon\_\{\\mathrm\{opt\}\}\.Moreover,

βemp≤β⋆\+\(1−γ\)​Δ\+\(1−γ\)​εopt\.\\beta\_\{\\mathrm\{emp\}\}\\leq\\beta\_\{\\star\}\+\(1\-\\gamma\)\\Delta\+\(1\-\\gamma\)\\varepsilon\_\{\\mathrm\{opt\}\}\.\(68\)Consequently, for everyR≥R0R\\geq R\_\{0\},

Bemp​\(R\)−B⋆​\(R\)≤R​\(1−γ\)​\(Δ\+εopt\)\.B\_\{\\mathrm\{emp\}\}\(R\)\-B\_\{\\star\}\(R\)\\leq R\(1\-\\gamma\)\(\\Delta\+\\varepsilon\_\{\\mathrm\{opt\}\}\)\.\(69\)

We proceed to bound the two components ofΔ\\Delta\. We package recursive bounds for these components in two lemmas\. Recall from \([64](https://arxiv.org/html/2608.06545#A4.E64)\) the localized concentration budgets

B⋆​\(R\)=B​\(R,β⋆\),Bemp​\(R\)=B​\(R,βemp\)\.B\_\{\\star\}\(R\)=B\(R,\\beta\_\{\\star\}\),\\qquad B\_\{\\mathrm\{emp\}\}\(R\)=B\(R,\\beta\_\{\\mathrm\{emp\}\}\)\.Both lemmas are stated under the shared localization condition

max\{\\displaystyle\\max\\big\\\{∥U∥span,∥U^∥span,∥W∥span,∥W^∥span,∥V^⋆∥span\}≤R\.\\displaystyle\\\|U\\\|\_\{\\mathrm\{span\}\},\\\|\\widehat\{U\}\\\|\_\{\\mathrm\{span\}\},\\\|W\\\|\_\{\\mathrm\{span\}\},\\\|\\widehat\{W\}\\\|\_\{\\mathrm\{span\}\},\\\|\\widehat\{V\}^\{\\star\}\\\|\_\{\\mathrm\{span\}\}\\big\\\}\\leq R\.\(70\)Moreover, they require the following two sample\-size conditions:

N≥C​ι1−γ,N≥C​σ⋅ι\(1−γ\)2\.N\\geq C\\frac\{\\iota\}\{1\-\\gamma\},\\qquad N\\geq C\\frac\{\\sigma\\cdot\\iota\}\{\(1\-\\gamma\)^\{2\}\}\.\(71\)These are the two radius\-independent sample\-size requirements in \([59](https://arxiv.org/html/2608.06545#A4.E59)\)\.

We now state the two lemmas\. We note that the second part of Lemma[13](https://arxiv.org/html/2608.06545#Thmlemma13)is included for later use in the span\-agnostic analysis\. The proof of Lemma[13](https://arxiv.org/html/2608.06545#Thmlemma13)is deferred to Appendix[D\.5\.3](https://arxiv.org/html/2608.06545#A4.SS5.SSS3), and the proof of Lemma[14](https://arxiv.org/html/2608.06545#Thmlemma14)is deferred to Appendix[D\.5\.4](https://arxiv.org/html/2608.06545#A4.SS5.SSS4)\.

###### Lemma 13\.

LetR≥max⁡\{1,Hanc\}R\\geq\\max\\\{1,H\_\{\\mathrm\{anc\}\}\\\}be a deterministic radius and suppose \([70](https://arxiv.org/html/2608.06545#A4.E70)\) and \([71](https://arxiv.org/html/2608.06545#A4.E71)\) hold\. On the high\-probability eventℰ​\(R\)\\mathcal\{E\}\(R\), for a universal constantCC,

‖U^−U‖∞≤C​B⋆​\(R\)⋅ιN​\(1−γ\)2\+C​R⋅ιN​\(1−γ\)\+150​Δ\.\\\|\\widehat\{U\}\-U\\\|\_\{\\infty\}\\leq C\\sqrt\{\\frac\{B\_\{\\star\}\(R\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\+C\\frac\{R\\cdot\\iota\}\{N\(1\-\\gamma\)\}\+\\frac\{1\}\{50\}\\Delta\.More generally, letπ\\pibe any deterministic policy fixed independently of the empirical transition samples\. Under \([71](https://arxiv.org/html/2608.06545#A4.E71)\), if

R≥max⁡\{1,Hanc,‖Vγπ,σ‖span,‖V^γπ,σ‖span\},R\\geq\\max\\left\\\{1,H\_\{\\mathrm\{anc\}\},\\\|V\_\{\\gamma\}^\{\\pi,\\sigma\}\\\|\_\{\\mathrm\{span\}\},\\\|\\widehat\{V\}\_\{\\gamma\}^\{\\pi,\\sigma\}\\\|\_\{\\mathrm\{span\}\}\\right\\\},then the same argument gives, with probability at least1−O​\(δ\)1\-O\(\\delta\),

‖V^γπ,σ−Vγπ,σ‖∞\\displaystyle\\left\\\|\\widehat\{V\}\_\{\\gamma\}^\{\\pi,\\sigma\}\-V\_\{\\gamma\}^\{\\pi,\\sigma\}\\right\\\|\_\{\\infty\}≤C​\(Hanc\+R​\[ρ¯−\(1−γ\)​mins⁡Vγπ,σ​\(s\)\]\+\+σ​R2\+\(1−γ\)​R2\)⋅ιN​\(1−γ\)2\\displaystyle\\quad\\leq C\\sqrt\{\\frac\{\\left\(H\_\{\\mathrm\{anc\}\}\+R\\left\[\\bar\{\\rho\}\-\(1\-\\gamma\)\\min\_\{s\}V\_\{\\gamma\}^\{\\pi,\\sigma\}\(s\)\\right\]\_\{\+\}\+\\sigma R^\{2\}\+\(1\-\\gamma\)R^\{2\}\\right\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\+C​R⋅ιN​\(1−γ\)\+150​‖V^γπ,σ−Vγπ,σ‖∞\.\\displaystyle\\qquad\+C\\frac\{R\\cdot\\iota\}\{N\(1\-\\gamma\)\}\+\\frac\{1\}\{50\}\\left\\\|\\widehat\{V\}\_\{\\gamma\}^\{\\pi,\\sigma\}\-V\_\{\\gamma\}^\{\\pi,\\sigma\}\\right\\\|\_\{\\infty\}\.\(72\)

###### Lemma 14\.

LetR≥max⁡\{1,Hanc\}R\\geq\\max\\\{1,H\_\{\\mathrm\{anc\}\}\\\}be deterministic and suppose \([70](https://arxiv.org/html/2608.06545#A4.E70)\) and \([71](https://arxiv.org/html/2608.06545#A4.E71)\) hold\. On the high\-probability eventℰ​\(R\)\\mathcal\{E\}\(R\), for a universal constantCC,

‖W^−W‖∞≤C​Bemp​\(R\)⋅ιN​\(1−γ\)2\+C​R⋅ιN​\(1−γ\)\+120​Δ\+C​εopt1−γ\.\\\|\\widehat\{W\}\-W\\\|\_\{\\infty\}\\leq C\\sqrt\{\\frac\{B\_\{\\mathrm\{emp\}\}\(R\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\+C\\frac\{R\\cdot\\iota\}\{N\(1\-\\gamma\)\}\+\\frac\{1\}\{20\}\\Delta\+C\\frac\{\\varepsilon\_\{\\mathrm\{opt\}\}\}\{1\-\\gamma\}\.

We now condition on the eventℰ\\mathcal\{E\}from \([65](https://arxiv.org/html/2608.06545#A4.E65)\)\. Becauseℰ⊆ℰ​\(R\)\\mathcal\{E\}\\subseteq\\mathcal\{E\}\(R\)for everyR∈ℛR\\in\\mathcal\{R\}, the localized lemmas hold simultaneously at every grid radius\.

Sincer​\(s,a\)∈\[0,1\]r\(s,a\)\\in\[0,1\], for every policyπ\\pi,

0≤Vγπ,σ≤11−γ⋅𝟏S,0≤V^γπ,σ≤11−γ⋅𝟏S\.0\\leq V\_\{\\gamma\}^\{\\pi,\\sigma\}\\leq\\frac\{1\}\{1\-\\gamma\}\\cdot\\bm\{1\}\_\{S\},\\qquad 0\\leq\\widehat\{V\}\_\{\\gamma\}^\{\\pi,\\sigma\}\\leq\\frac\{1\}\{1\-\\gamma\}\\cdot\\bm\{1\}\_\{S\}\.\(73\)HenceΔ≤2/\(1−γ\)\\Delta\\leq 2/\(1\-\\gamma\)\. The assumptionεopt≤c​\(1−γ\)​R0\\varepsilon\_\{\\mathrm\{opt\}\}\\leq c\(1\-\\gamma\)R\_\{0\}therefore ensures that the grid contains a radius larger thanR0\+2​Δ\+2​εoptR\_\{0\}\+2\\Delta\+2\\varepsilon\_\{\\mathrm\{opt\}\}\. LetjΔj\_\{\\Delta\}be the smallest index such that

RjΔ≥R0\+2​Δ\+2​εopt\.R\_\{j\_\{\\Delta\}\}\\geq R\_\{0\}\+2\\Delta\+2\\varepsilon\_\{\\mathrm\{opt\}\}\.By minimality and the assumptionεopt≤c​\(1−γ\)​R0≤c​R0\\varepsilon\_\{\\mathrm\{opt\}\}\\leq c\(1\-\\gamma\)R\_\{0\}\\leq cR\_\{0\}in Theorem[4](https://arxiv.org/html/2608.06545#Thmtheorem4),

RjΔ≤2​\(R0\+2​Δ\+2​εopt\)≤C​\(R0\+Δ\)\.R\_\{j\_\{\\Delta\}\}\\leq 2\\left\(R\_\{0\}\+2\\Delta\+2\\varepsilon\_\{\\mathrm\{opt\}\}\\right\)\\leq C\(R\_\{0\}\+\\Delta\)\.\(74\)Lemma[12](https://arxiv.org/html/2608.06545#Thmlemma12), together with the reference\-radius bound \([57](https://arxiv.org/html/2608.06545#A4.E57)\), shows that \([70](https://arxiv.org/html/2608.06545#A4.E70)\) holds atR=RjΔR=R\_\{j\_\{\\Delta\}\}\.

At this radius, the budget comparison in Lemma[12](https://arxiv.org/html/2608.06545#Thmlemma12)gives

Bemp​\(RjΔ\)≤B⋆​\(RjΔ\)\+RjΔ​\(1−γ\)​\(Δ\+εopt\)\.B\_\{\\mathrm\{emp\}\}\(R\_\{j\_\{\\Delta\}\}\)\\leq B\_\{\\star\}\(R\_\{j\_\{\\Delta\}\}\)\+R\_\{j\_\{\\Delta\}\}\(1\-\\gamma\)\(\\Delta\+\\varepsilon\_\{\\mathrm\{opt\}\}\)\.\(75\)Also, by \([74](https://arxiv.org/html/2608.06545#A4.E74)\),

B⋆​\(RjΔ\)≤C​\[B⋆​\(R0\)\+β⋆​Δ\+σ​Δ2\+\(1−γ\)​Δ2\]\.B\_\{\\star\}\(R\_\{j\_\{\\Delta\}\}\)\\leq C\\left\[B\_\{\\star\}\(R\_\{0\}\)\+\\beta\_\{\\star\}\\Delta\+\\sigma\\Delta^\{2\}\+\(1\-\\gamma\)\\Delta^\{2\}\\right\]\.\(76\)Invoking Lemmas[13](https://arxiv.org/html/2608.06545#Thmlemma13)and[14](https://arxiv.org/html/2608.06545#Thmlemma14)atR=RjΔR=R\_\{j\_\{\\Delta\}\}and substituting the two lemma bounds into the definition ofΔ\\Deltain \([67](https://arxiv.org/html/2608.06545#A4.E67)\) gives

Δ\\displaystyle\\Delta≤C​B⋆​\(RjΔ\)⋅ιN​\(1−γ\)2\+C​Bemp​\(RjΔ\)⋅ιN​\(1−γ\)2\+C​RjΔ⋅ιN​\(1−γ\)\\displaystyle\\leq C\\sqrt\{\\frac\{B\_\{\\star\}\(R\_\{j\_\{\\Delta\}\}\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\+C\\sqrt\{\\frac\{B\_\{\\mathrm\{emp\}\}\(R\_\{j\_\{\\Delta\}\}\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\+C\\frac\{R\_\{j\_\{\\Delta\}\}\\cdot\\iota\}\{N\(1\-\\gamma\)\}\+7100​Δ\+C​εopt1−γ\.\\displaystyle\\quad\+\\frac\{7\}\{100\}\\Delta\+C\\frac\{\\varepsilon\_\{\\mathrm\{opt\}\}\}\{1\-\\gamma\}\.Substituting \([75](https://arxiv.org/html/2608.06545#A4.E75)\) into the second square\-root term above and usinga\+b≤a\+b\\sqrt\{a\+b\}\\leq\\sqrt\{a\}\+\\sqrt\{b\}produces the additional term on the left below\. By \([74](https://arxiv.org/html/2608.06545#A4.E74)\) andεopt≤c​\(1−γ\)​R0≤c​R0\\varepsilon\_\{\\mathrm\{opt\}\}\\leq c\(1\-\\gamma\)R\_\{0\}\\leq cR\_\{0\},

C​RjΔ​\(Δ\+εopt\)⋅ιN​\(1−γ\)\\displaystyle C\\sqrt\{\\frac\{R\_\{j\_\{\\Delta\}\}\(\\Delta\+\\varepsilon\_\{\\mathrm\{opt\}\}\)\\cdot\\iota\}\{N\(1\-\\gamma\)\}\}≤C​\(R0\+Δ\)​ιN​\(1−γ\)\\displaystyle\\leq C\(R\_\{0\}\+\\Delta\)\\sqrt\{\\frac\{\\iota\}\{N\(1\-\\gamma\)\}\}≤C​B⋆​\(R0\)⋅ιN​\(1−γ\)2\+C​ιN​\(1−γ\)​Δ,\\displaystyle\\leq C\\sqrt\{\\frac\{B\_\{\\star\}\(R\_\{0\}\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\+C\\sqrt\{\\frac\{\\iota\}\{N\(1\-\\gamma\)\}\}\\Delta,where the second inequality usesB⋆​\(R0\)≥\(1−γ\)​R02B\_\{\\star\}\(R\_\{0\}\)\\geq\(1\-\\gamma\)R\_\{0\}^\{2\}\. Thus the first term is absorbed into the leadingB⋆​\(R0\)B\_\{\\star\}\(R\_\{0\}\)square\-root term, while the second contributes to the coefficient ofΔ\\Delta\. Combining this estimate with \([76](https://arxiv.org/html/2608.06545#A4.E76)\) and \([74](https://arxiv.org/html/2608.06545#A4.E74)\) gives

Δ\\displaystyle\\Delta≤C​B⋆​\(R0\)⋅ιN​\(1−γ\)2\+C​β⋆​Δ⋅ιN​\(1−γ\)2\\displaystyle\\leq C\\sqrt\{\\frac\{B\_\{\\star\}\(R\_\{0\}\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\+C\\sqrt\{\\frac\{\\beta\_\{\\star\}\\Delta\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\(77\)\+\[7100\+C​σ⋅ιN​\(1−γ\)2\+C​ιN​\(1−γ\)\+C​ιN​\(1−γ\)\]​Δ\+C​R0⋅ιN​\(1−γ\)\+C​εopt1−γ\.\\displaystyle\\quad\+\\left\[\\frac\{7\}\{100\}\+C\\sqrt\{\\frac\{\\sigma\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\+C\\sqrt\{\\frac\{\\iota\}\{N\(1\-\\gamma\)\}\}\+C\\frac\{\\iota\}\{N\(1\-\\gamma\)\}\\right\]\\Delta\+C\\frac\{R\_\{0\}\\cdot\\iota\}\{N\(1\-\\gamma\)\}\+C\\frac\{\\varepsilon\_\{\\mathrm\{opt\}\}\}\{1\-\\gamma\}\.Young’s inequality gives

C​β⋆​Δ⋅ιN​\(1−γ\)2≤93400​Δ\+C​β⋆⋅ιN​\(1−γ\)2\.C\\sqrt\{\\frac\{\\beta\_\{\\star\}\\Delta\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\\leq\\frac\{93\}\{400\}\\Delta\+C\\frac\{\\beta\_\{\\star\}\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\.The sample\-size condition \([59](https://arxiv.org/html/2608.06545#A4.E59)\) implies

β⋆⋅ιN​\(1−γ\)2≤C​R0\.\\frac\{\\beta\_\{\\star\}\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\\leq CR\_\{0\}\.Since \([64](https://arxiv.org/html/2608.06545#A4.E64)\) givesR0​β⋆≤B⋆​\(R0\)R\_\{0\}\\beta\_\{\\star\}\\leq B\_\{\\star\}\(R\_\{0\}\), it follows that

β⋆⋅ιN​\(1−γ\)2≤C​B⋆​\(R0\)⋅ιN​\(1−γ\)2,\\frac\{\\beta\_\{\\star\}\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\\leq C\\sqrt\{\\frac\{B\_\{\\star\}\(R\_\{0\}\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\},so the last term in the Young bound is absorbed into the first square\-root term\. The two sample\-size conditions in \([71](https://arxiv.org/html/2608.06545#A4.E71)\) make the sum of the other coefficients ofΔ\\Deltaat most93/40093/400\. Combining this with the93​Δ/40093\\Delta/400term from Young’s inequality gives

Δ≤C​B⋆​\(R0\)⋅ιN​\(1−γ\)2\+93200​Δ\+C​R0⋅ιN​\(1−γ\)\+C​εopt1−γ\.\\Delta\\leq C\\sqrt\{\\frac\{B\_\{\\star\}\(R\_\{0\}\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\+\\frac\{93\}\{200\}\\Delta\+C\\frac\{R\_\{0\}\\cdot\\iota\}\{N\(1\-\\gamma\)\}\+C\\frac\{\\varepsilon\_\{\\mathrm\{opt\}\}\}\{1\-\\gamma\}\.Moving theΔ\\Deltaterm on the right\-hand side to the left\-hand side and absorbing the numerical factor intoCC, we obtain

Δ≤C​B⋆​\(R0\)⋅ιN​\(1−γ\)2\+C​R0⋅ιN​\(1−γ\)\+C​εopt1−γ\.\\Delta\\leq C\\sqrt\{\\frac\{B\_\{\\star\}\(R\_\{0\}\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\+C\\frac\{R\_\{0\}\\cdot\\iota\}\{N\(1\-\\gamma\)\}\+C\\frac\{\\varepsilon\_\{\\mathrm\{opt\}\}\}\{1\-\\gamma\}\.Thus, by the definition ofΔ\\Deltaand \([66](https://arxiv.org/html/2608.06545#A4.E66)\),

‖W^−W‖∞≤C​B⋆​\(R0\)⋅ιN​\(1−γ\)2\+C​R0⋅ιN​\(1−γ\)\+C​εopt1−γ,\\\|\\widehat\{W\}\-W\\\|\_\{\\infty\}\\leq C\\sqrt\{\\frac\{B\_\{\\star\}\(R\_\{0\}\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\+C\\frac\{R\_\{0\}\\cdot\\iota\}\{N\(1\-\\gamma\)\}\+C\\frac\{\\varepsilon\_\{\\mathrm\{opt\}\}\}\{1\-\\gamma\},and

‖U−W‖∞≤C​B⋆​\(R0\)⋅ιN​\(1−γ\)2\+C​R0⋅ιN​\(1−γ\)\+C​εopt1−γ\.\\\|U\-W\\\|\_\{\\infty\}\\leq C\\sqrt\{\\frac\{B\_\{\\star\}\(R\_\{0\}\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\+C\\frac\{R\_\{0\}\\cdot\\iota\}\{N\(1\-\\gamma\)\}\+C\\frac\{\\varepsilon\_\{\\mathrm\{opt\}\}\}\{1\-\\gamma\}\.ExpandingB⋆​\(R0\)B\_\{\\star\}\(R\_\{0\}\)using \([64](https://arxiv.org/html/2608.06545#A4.E64)\) shows that both bounds in Theorem[4](https://arxiv.org/html/2608.06545#Thmtheorem4)hold\. Sinceℙ​\{ℰ\}≥1−O​\(δ\)\\mathbb\{P\}\\\{\\mathcal\{E\}\\\}\\geq 1\-O\(\\delta\), this proves the theorem\.

### D\.5Proof of auxiliary lemmas

Except for the nominal\-branch proof in Appendix[D\.5\.1](https://arxiv.org/html/2608.06545#A4.SS5.SSS1), the auxiliary proofs below inherit the notation from Appendix[D\.4\.1](https://arxiv.org/html/2608.06545#A4.SS4.SSS1)\. In particular, recall that

U≔Vγ⋆,σ,U^≔V^γπγ⋆,σ,V^⋆≔V^γ⋆,σ,W≔Vγπ^,σ,W^≔V^γπ^,σ\.U\\coloneqq V\_\{\\gamma\}^\{\\star,\\sigma\},\\qquad\\widehat\{U\}\\coloneqq\\widehat\{V\}\_\{\\gamma\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\sigma\},\\qquad\\widehat\{V\}^\{\\star\}\\coloneqq\\widehat\{V\}\_\{\\gamma\}^\{\\star,\\sigma\},\\qquad W\\coloneqq V\_\{\\gamma\}^\{\\widehat\{\\pi\},\\sigma\},\\qquad\\widehat\{W\}\\coloneqq\\widehat\{V\}\_\{\\gamma\}^\{\\widehat\{\\pi\},\\sigma\}\.
#### D\.5\.1Proof of Lemma[11](https://arxiv.org/html/2608.06545#Thmlemma11)

The argument has three steps\. We first obtain a discounted nominal guarantee, then derive nominal average\-reward and span bounds for the returned policy, and finally transfer these bounds to the robust average reward\.

##### Step 1: Discounted nominal guarantee\.

We analyze the nominal branch of Algorithm[1](https://arxiv.org/html/2608.06545#alg1)\. This branch applies the discounted plug\-in reduction with span inputH0H\_\{0\}, the degenerate uncertainty rule𝒰​\(P\)=\{P\}\\mathcal\{U\}\(P\)=\\\{P\\\}, and solver toleranceεopt≤copt​ε\\varepsilon\_\{\\mathrm\{opt\}\}\\leq c\_\{\\mathrm\{opt\}\}\\varepsilon, wherecopt\>0c\_\{\\mathrm\{opt\}\}\>0is a sufficiently small universal constant\. Under the degenerate uncertainty rule, the empirical robust discounted problem is exactly the empirical nominal discounted MDP with kernelP^0\\widehat\{P\}^\{0\}, and Corollary[2](https://arxiv.org/html/2608.06545#Thmcorollary2)applies withσ=0\\sigma=0\.

We now check the conditions and relevant quantities in Corollary[2](https://arxiv.org/html/2608.06545#Thmcorollary2)\. Set

γ=1−ε20​H0\.\\gamma=1\-\\frac\{\\varepsilon\}\{20H\_\{0\}\}\.Sinceε≤1\\varepsilon\\leq 1andH0≥1H\_\{0\}\\geq 1, this choice givesγ≥19/20\\gamma\\geq 19/20\.

We use the nominal optimal pair as the anchor:

ρ¯=ρ⋆,0,h¯=hP0⋆\.\\bar\{\\rho\}=\\rho^\{\\star,0\},\\qquad\\bar\{h\}=h\_\{P^\{0\}\}^\{\\star\}\.The nominal average\-reward Bellman equation shows that this pair satisfies the supersolution condition \([6](https://arxiv.org/html/2608.06545#S4.E6)\)\. It is fixed independently of the transition samples, andHanc=H0H\_\{\\mathrm\{anc\}\}=H\_\{0\}\. Moreover, Lemma[3](https://arxiv.org/html/2608.06545#Thmlemma3), applied to the degenerate uncertainty rule, gives

‖Vγ⋆,0‖span≤2​H0\.\\left\\\|V\_\{\\gamma\}^\{\\star,0\}\\right\\\|\_\{\\mathrm\{span\}\}\\leq 2H\_\{0\}\.We may therefore takeR0=2​H0R\_\{0\}=2H\_\{0\}\. The optimal discounted\-to\-average comparison \([55](https://arxiv.org/html/2608.06545#A4.E55)\) in Lemma[9](https://arxiv.org/html/2608.06545#Thmlemma9), again applied to the degenerate uncertainty rule, yields

β⋆=\[ρ⋆,0−\(1−γ\)​mins⁡Vγ⋆,0​\(s\)\]\+≤2​\(1−γ\)​H0\.\\beta\_\{\\star\}=\\left\[\\rho^\{\\star,0\}\-\(1\-\\gamma\)\\min\_\{s\}V\_\{\\gamma\}^\{\\star,0\}\(s\)\\right\]\_\{\+\}\\leq 2\(1\-\\gamma\)H\_\{0\}\.Consequently,

Hanc\+R0​β⋆\+\(1−γ\)​R02≤H0\+8​\(1−γ\)​H02≤C​H0\.H\_\{\\mathrm\{anc\}\}\+R\_\{0\}\\beta\_\{\\star\}\+\(1\-\\gamma\)R\_\{0\}^\{2\}\\leq H\_\{0\}\+8\(1\-\\gamma\)H\_\{0\}^\{2\}\\leq CH\_\{0\}\.\(78\)
Apply Corollary[2](https://arxiv.org/html/2608.06545#Thmcorollary2)with discounted target accuracyεγ=H0\\varepsilon\_\{\\gamma\}=H\_\{0\}\. Since\(1−γ\)​εγ=ε/20\(1\-\\gamma\)\\varepsilon\_\{\\gamma\}=\\varepsilon/20, its solver\-tolerance condition holds as long ascoptc\_\{\\mathrm\{opt\}\}is sufficiently small\. Using \([78](https://arxiv.org/html/2608.06545#A4.E78)\) andR0=2​H0R\_\{0\}=2H\_\{0\}, the sample\-size requirement of this corollary reduces toN​S​A≥C​S​A​\(H0/ε2\+H0/ε\)​ιNSA\\geq CSA\(H\_\{0\}/\\varepsilon^\{2\}\+H\_\{0\}/\\varepsilon\)\\iota\. Sinceε≤1\\varepsilon\\leq 1, the termH0/εH\_\{0\}/\\varepsilonis absorbed byH0/ε2H\_\{0\}/\\varepsilon^\{2\}\. Moreover, the definition ofι\\iotaand the choice ofγ\\gammagiveι≤C​log⁡\(S​A​H0​N/\(ε​δ\)\)\\iota\\leq C\\log\(SAH\_\{0\}N/\(\\varepsilon\\delta\)\)\. It is therefore sufficient that

N​S​A≥C​S​A​H0ε2​log⁡\(S​A​H0​Nε​δ\)\.NSA\\geq CSA\\frac\{H\_\{0\}\}\{\\varepsilon^\{2\}\}\\log\\\!\\left\(\\frac\{SAH\_\{0\}N\}\{\\varepsilon\\delta\}\\right\)\.Thus, with probability at least1−O​\(δ\)1\-O\(\\delta\),

‖Vγ⋆,0−Vγπ^,0‖∞≤H0\.\\left\\\|V\_\{\\gamma\}^\{\\star,0\}\-V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\\right\\\|\_\{\\infty\}\\leq H\_\{0\}\.

##### Step 2: Nominal average\-reward and span bounds\.

Lemma[10](https://arxiv.org/html/2608.06545#Thmlemma10), applied to the degenerate uncertainty rule, now gives

ρ⋆,0−ρπ^,0≤3​\(1−γ\)​H0=3​ε20≤ε5\.\\rho^\{\\star,0\}\-\\rho^\{\\widehat\{\\pi\},0\}\\leq 3\(1\-\\gamma\)H\_\{0\}=\\frac\{3\\varepsilon\}\{20\}\\leq\\frac\{\\varepsilon\}\{5\}\.\(79\)The same discounted guarantee also controls the span of the returned policy:

‖Vγπ^,0‖span≤‖Vγ⋆,0‖span\+2​‖Vγ⋆,0−Vγπ^,0‖∞≤4​H0\.\\left\\\|V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\\right\\\|\_\{\\mathrm\{span\}\}\\leq\\left\\\|V\_\{\\gamma\}^\{\\star,0\}\\right\\\|\_\{\\mathrm\{span\}\}\+2\\left\\\|V\_\{\\gamma\}^\{\\star,0\}\-V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\\right\\\|\_\{\\infty\}\\leq 4H\_\{0\}\.\(80\)

##### Step 3: Transfer to the robust average reward\.

We now translate the nominal accuracy guarantee \([79](https://arxiv.org/html/2608.06545#A4.E79)\) into the desired robust accuracy bound

ρ⋆,σ−ρπ^,σ≤ε\.\\rho^\{\\star,\\sigma\}\-\\rho^\{\\widehat\{\\pi\},\\sigma\}\\leq\\varepsilon\.The span\-controlled nominal discounted valueVγπ^,0V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}provides the link between the nominal and robust average rewards\.

Recall that the nominal discounted value function ofπ^\\widehat\{\\pi\}satisfies

Vγπ^,0​\(s\)=rπ^​\(s\)\+γ​P0,π^​Vγπ^,0​\(s\)\.V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\(s\)=r^\{\\widehat\{\\pi\}\}\(s\)\+\\gamma P^\{0,\\widehat\{\\pi\}\}V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\(s\)\.By Lemma[1](https://arxiv.org/html/2608.06545#Thmlemma1)applied to the nominal kernel andh=γ​Vγπ^,0h=\\gamma V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}, we have

ρπ^,0≤\(1−γ\)​mins⁡Vγπ^,0​\(s\)\+\(1−γ\)​‖Vγπ^,0‖span\.\\rho^\{\\widehat\{\\pi\},0\}\\leq\(1\-\\gamma\)\\min\_\{s\}V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\(s\)\+\(1\-\\gamma\)\\left\\\|V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\\right\\\|\_\{\\mathrm\{span\}\}\.Consequently,

ρ⋆,0−\(1−γ\)​mins⁡Vγπ^,0​\(s\)≤ε5\+4​\(1−γ\)​H0\.\\rho^\{\\star,0\}\-\(1\-\\gamma\)\\min\_\{s\}V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\(s\)\\leq\\frac\{\\varepsilon\}\{5\}\+4\(1\-\\gamma\)H\_\{0\}\.\(81\)
For anyP∈𝒫P\\in\\mathcal\{P\}, the total\-variation perturbation bound gives

γ​Pπ^​Vγπ^,0​\(s\)≥γ​P0,π^​Vγπ^,0​\(s\)−γ​σ​‖Vγπ^,0‖span\.\\gamma P^\{\\widehat\{\\pi\}\}V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\(s\)\\geq\\gamma P^\{0,\\widehat\{\\pi\}\}V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\(s\)\-\\gamma\\sigma\\left\\\|V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\\right\\\|\_\{\\mathrm\{span\}\}\.Because the right\-hand side does not depend onPP, taking the infimum overP∈𝒫P\\in\\mathcal\{P\}and addingrπ^​\(s\)r^\{\\widehat\{\\pi\}\}\(s\)gives

rπ^​\(s\)\+infP∈𝒫γ​Pπ^​Vγπ^,0​\(s\)\\displaystyle r^\{\\widehat\{\\pi\}\}\(s\)\+\\inf\_\{P\\in\\mathcal\{P\}\}\\gamma P^\{\\widehat\{\\pi\}\}V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\(s\)≥rπ^​\(s\)\+γ​P0,π^​Vγπ^,0​\(s\)−γ​σ​‖Vγπ^,0‖span\\displaystyle\\geq r^\{\\widehat\{\\pi\}\}\(s\)\+\\gamma P^\{0,\\widehat\{\\pi\}\}V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\(s\)\-\\gamma\\sigma\\left\\\|V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\\right\\\|\_\{\\mathrm\{span\}\}=Vγπ^,0​\(s\)−γ​σ​‖Vγπ^,0‖span,\\displaystyle=V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\(s\)\-\\gamma\\sigma\\left\\\|V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\\right\\\|\_\{\\mathrm\{span\}\},where the equality uses the nominal discounted Bellman equation\.

Now seth=γ​Vγπ^,0h=\\gamma V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\. Since\(1−γ\)​minx∈𝒮⁡Vγπ^,0​\(x\)≤\(1−γ\)​Vγπ^,0​\(s\)\(1\-\\gamma\)\\min\_\{x\\in\\mathcal\{S\}\}V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\(x\)\\leq\(1\-\\gamma\)V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\(s\), the preceding display implies that, for every statess,

\[\(1−γ\)​minx∈𝒮⁡Vγπ^,0​\(x\)−γ​σ​‖Vγπ^,0‖span\]\+h​\(s\)\\displaystyle\\left\[\(1\-\\gamma\)\\min\_\{x\\in\\mathcal\{S\}\}V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\(x\)\-\\gamma\\sigma\\left\\\|V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\\right\\\|\_\{\\mathrm\{span\}\}\\right\]\+h\(s\)≤Vγπ^,0​\(s\)−γ​σ​‖Vγπ^,0‖span≤rπ^​\(s\)\+infP∈𝒫\(Pπ^​h\)​\(s\)\.\\displaystyle\\quad\\leq V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\(s\)\-\\gamma\\sigma\\left\\\|V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\\right\\\|\_\{\\mathrm\{span\}\}\\leq r^\{\\widehat\{\\pi\}\}\(s\)\+\\inf\_\{P\\in\\mathcal\{P\}\}\(P\_\{\\widehat\{\\pi\}\}h\)\(s\)\.The lower\-bound direction of Lemma[1](https://arxiv.org/html/2608.06545#Thmlemma1), applied with thishhand

ρ=\(1−γ\)​minx∈𝒮⁡Vγπ^,0​\(x\)−γ​σ​‖Vγπ^,0‖span,\\rho=\(1\-\\gamma\)\\min\_\{x\\in\\mathcal\{S\}\}V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\(x\)\-\\gamma\\sigma\\left\\\|V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\\right\\\|\_\{\\mathrm\{span\}\},therefore gives

ρπ^,σ≥\(1−γ\)​mins⁡Vγπ^,0​\(s\)−γ​σ​‖Vγπ^,0‖span\.\\rho^\{\\widehat\{\\pi\},\\sigma\}\\geq\(1\-\\gamma\)\\min\_\{s\}V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\(s\)\-\\gamma\\sigma\\left\\\|V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\\right\\\|\_\{\\mathrm\{span\}\}\.
SinceP0∈𝒫P^\{0\}\\in\\mathcal\{P\}, we haveρ⋆,σ≤ρ⋆,0\\rho^\{\\star,\\sigma\}\\leq\\rho^\{\\star,0\}\. Combining this with the preceding lower bound onρπ^,σ\\rho^\{\\widehat\{\\pi\},\\sigma\}gives

ρ⋆,σ−ρπ^,σ\\displaystyle\\rho^\{\\star,\\sigma\}\-\\rho^\{\\widehat\{\\pi\},\\sigma\}≤ρ⋆,0−ρπ^,σ\\displaystyle\\leq\\rho^\{\\star,0\}\-\\rho^\{\\widehat\{\\pi\},\\sigma\}≤ρ⋆,0−\(1−γ\)​mins∈𝒮⁡Vγπ^,0​\(s\)\+γ​σ​‖Vγπ^,0‖span\\displaystyle\\leq\\rho^\{\\star,0\}\-\(1\-\\gamma\)\\min\_\{s\\in\\mathcal\{S\}\}V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\(s\)\+\\gamma\\sigma\\left\\\|V\_\{\\gamma\}^\{\\widehat\{\\pi\},0\}\\right\\\|\_\{\\mathrm\{span\}\}≤ε5\+4​\(1−γ\)​H0\+4​γ​σ​H0\.\\displaystyle\\leq\\frac\{\\varepsilon\}\{5\}\+4\(1\-\\gamma\)H\_\{0\}\+4\\gamma\\sigma H\_\{0\}\.Here, the last inequality uses \([81](https://arxiv.org/html/2608.06545#A4.E81)\) and \([80](https://arxiv.org/html/2608.06545#A4.E80)\)\. Using1−γ=ε/\(20​H0\)1\-\\gamma=\\varepsilon/\(20H\_\{0\}\),γ≤1\\gamma\\leq 1, and7​σ​H0≤ε7\\sigma H\_\{0\}\\leq\\varepsilon, we obtain

ρ⋆,σ−ρπ^,σ≤\(15\+15\+47\)​ε=3435​ε≤ε,\\rho^\{\\star,\\sigma\}\-\\rho^\{\\widehat\{\\pi\},\\sigma\}\\leq\\left\(\\frac\{1\}\{5\}\+\\frac\{1\}\{5\}\+\\frac\{4\}\{7\}\\right\)\\varepsilon=\\frac\{34\}\{35\}\\varepsilon\\leq\\varepsilon,as desired\.

#### D\.5\.2Proof of Lemma[12](https://arxiv.org/html/2608.06545#Thmlemma12)

The proof has two steps\. We first compare each relevant value function withUUand derive the span bounds\. We then compare the two anchor defects and the corresponding concentration budgets\.

##### Step 1: Value comparisons and span bounds\.

RecallΔ\\Deltafrom \([67](https://arxiv.org/html/2608.06545#A4.E67)\)\. For any value functionsV1V\_\{1\}andV2V\_\{2\},

‖V1‖span≤‖V2‖span\+2​‖V1−V2‖∞\.\\\|V\_\{1\}\\\|\_\{\\mathrm\{span\}\}\\leq\\\|V\_\{2\}\\\|\_\{\\mathrm\{span\}\}\+2\\\|V\_\{1\}\-V\_\{2\}\\\|\_\{\\infty\}\.\(82\)We will apply \([82](https://arxiv.org/html/2608.06545#A4.E82)\) withV2=UV\_\{2\}=Uafter establishing the required sup\-norm comparisons\.

By optimality in the true and empirical robust MDPs,

W≤U,U^≤V^⋆,W^≤V^⋆,W\\leq U,\\qquad\\widehat\{U\}\\leq\\widehat\{V\}^\{\\star\},\\qquad\\widehat\{W\}\\leq\\widehat\{V\}^\{\\star\},where all inequalities are componentwise\. Moreover, optimality and the solver guarantee \([5](https://arxiv.org/html/2608.06545#S4.E5)\) give

0≤V^⋆−W^≤εopt​𝟏S\.0\\leq\\widehat\{V\}^\{\\star\}\-\\widehat\{W\}\\leq\\varepsilon\_\{\\mathrm\{opt\}\}\\bm\{1\}\_\{S\}\.We first compareV^⋆\\widehat\{V\}^\{\\star\}withUU\. For everys∈𝒮s\\in\\mathcal\{S\},

V^⋆​\(s\)−U​\(s\)\\displaystyle\\widehat\{V\}^\{\\star\}\(s\)\-U\(s\)=V^⋆​\(s\)−W^​\(s\)\+W^​\(s\)−W​\(s\)\+W​\(s\)−U​\(s\)\\displaystyle=\\widehat\{V\}^\{\\star\}\(s\)\-\\widehat\{W\}\(s\)\+\\widehat\{W\}\(s\)\-W\(s\)\+W\(s\)\-U\(s\)≤εopt\+‖W^−W‖∞,\\displaystyle\\leq\\varepsilon\_\{\\mathrm\{opt\}\}\+\\\|\\widehat\{W\}\-W\\\|\_\{\\infty\},U​\(s\)−V^⋆​\(s\)\\displaystyle U\(s\)\-\\widehat\{V\}^\{\\star\}\(s\)=U​\(s\)−U^​\(s\)\+U^​\(s\)−V^⋆​\(s\)\\displaystyle=U\(s\)\-\\widehat\{U\}\(s\)\+\\widehat\{U\}\(s\)\-\\widehat\{V\}^\{\\star\}\(s\)≤‖U^−U‖∞\.\\displaystyle\\leq\\\|\\widehat\{U\}\-U\\\|\_\{\\infty\}\.Hence, by \([67](https://arxiv.org/html/2608.06545#A4.E67)\),

‖V^⋆−U‖∞≤Δ\+εopt\.\\\|\\widehat\{V\}^\{\\star\}\-U\\\|\_\{\\infty\}\\leq\\Delta\+\\varepsilon\_\{\\mathrm\{opt\}\}\.\(83\)Similarly, for everys∈𝒮s\\in\\mathcal\{S\},

W^​\(s\)−U​\(s\)\\displaystyle\\widehat\{W\}\(s\)\-U\(s\)=W^​\(s\)−W​\(s\)\+W​\(s\)−U​\(s\)≤‖W^−W‖∞,\\displaystyle=\\widehat\{W\}\(s\)\-W\(s\)\+W\(s\)\-U\(s\)\\leq\\\|\\widehat\{W\}\-W\\\|\_\{\\infty\},U​\(s\)−W^​\(s\)\\displaystyle U\(s\)\-\\widehat\{W\}\(s\)=U​\(s\)−U^​\(s\)\+U^​\(s\)−V^⋆​\(s\)\+V^⋆​\(s\)−W^​\(s\)\\displaystyle=U\(s\)\-\\widehat\{U\}\(s\)\+\\widehat\{U\}\(s\)\-\\widehat\{V\}^\{\\star\}\(s\)\+\\widehat\{V\}^\{\\star\}\(s\)\-\\widehat\{W\}\(s\)≤‖U^−U‖∞\+εopt\.\\displaystyle\\leq\\\|\\widehat\{U\}\-U\\\|\_\{\\infty\}\+\\varepsilon\_\{\\mathrm\{opt\}\}\.Therefore,

‖W^−U‖∞≤Δ\+εopt\.\\\|\\widehat\{W\}\-U\\\|\_\{\\infty\}\\leq\\Delta\+\\varepsilon\_\{\\mathrm\{opt\}\}\.\(84\)SinceW≤UW\\leq U, the decomposition \([66](https://arxiv.org/html/2608.06545#A4.E66)\) also gives

‖W−U‖∞≤Δ\+εopt\.\\\|W\-U\\\|\_\{\\infty\}\\leq\\Delta\+\\varepsilon\_\{\\mathrm\{opt\}\}\.\(85\)Finally, \([67](https://arxiv.org/html/2608.06545#A4.E67)\) directly gives

‖U^−U‖∞≤Δ\.\\\|\\widehat\{U\}\-U\\\|\_\{\\infty\}\\leq\\Delta\.\(86\)Applying \([82](https://arxiv.org/html/2608.06545#A4.E82)\) to \([83](https://arxiv.org/html/2608.06545#A4.E83)\)–\([86](https://arxiv.org/html/2608.06545#A4.E86)\) and using \([57](https://arxiv.org/html/2608.06545#A4.E57)\), we conclude that

‖W‖span,‖W^‖span,‖U^‖span,‖V^⋆‖span≤R0\+2​Δ\+2​εopt\.\\\|W\\\|\_\{\\mathrm\{span\}\},\\quad\\\|\\widehat\{W\}\\\|\_\{\\mathrm\{span\}\},\\quad\\\|\\widehat\{U\}\\\|\_\{\\mathrm\{span\}\},\\quad\\\|\\widehat\{V\}^\{\\star\}\\\|\_\{\\mathrm\{span\}\}\\leq R\_\{0\}\+2\\Delta\+2\\varepsilon\_\{\\mathrm\{opt\}\}\.

##### Step 2: Anchor defects and concentration budgets\.

Empirical optimality and the solver guarantee \([5](https://arxiv.org/html/2608.06545#S4.E5)\) imply

W^≥V^⋆−εopt​𝟏S≥U^−εopt​𝟏S\.\\widehat\{W\}\\geq\\widehat\{V\}^\{\\star\}\-\\varepsilon\_\{\\mathrm\{opt\}\}\\bm\{1\}\_\{S\}\\geq\\widehat\{U\}\-\\varepsilon\_\{\\mathrm\{opt\}\}\\bm\{1\}\_\{S\}\.Consequently,

mins⁡W^​\(s\)≥mins⁡U​\(s\)−‖U^−U‖∞−εopt\.\\min\_\{s\}\\widehat\{W\}\(s\)\\geq\\min\_\{s\}U\(s\)\-\\left\\\|\\widehat\{U\}\-U\\right\\\|\_\{\\infty\}\-\\varepsilon\_\{\\mathrm\{opt\}\}\.Substituting this bound into \([62](https://arxiv.org/html/2608.06545#A4.E62)\) and using\[x\+y\]\+≤\[x\]\+\+y\[x\+y\]\_\{\+\}\\leq\[x\]\_\{\+\}\+yfory≥0y\\geq 0, we obtain

βemp\\displaystyle\\beta\_\{\\mathrm\{emp\}\}=\[ρ¯−\(1−γ\)​mins⁡W^​\(s\)\]\+\\displaystyle=\\left\[\\bar\{\\rho\}\-\(1\-\\gamma\)\\min\_\{s\}\\widehat\{W\}\(s\)\\right\]\_\{\+\}≤\[ρ¯−\(1−γ\)​mins⁡U​\(s\)\+\(1−γ\)​‖U^−U‖∞\+\(1−γ\)​εopt\]\+\\displaystyle\\leq\\left\[\\bar\{\\rho\}\-\(1\-\\gamma\)\\min\_\{s\}U\(s\)\+\(1\-\\gamma\)\\\|\\widehat\{U\}\-U\\\|\_\{\\infty\}\+\(1\-\\gamma\)\\varepsilon\_\{\\mathrm\{opt\}\}\\right\]\_\{\+\}≤β⋆\+\(1−γ\)​‖U^−U‖∞\+\(1−γ\)​εopt\\displaystyle\\leq\\beta\_\{\\star\}\+\(1\-\\gamma\)\\\|\\widehat\{U\}\-U\\\|\_\{\\infty\}\+\(1\-\\gamma\)\\varepsilon\_\{\\mathrm\{opt\}\}≤β⋆\+\(1−γ\)​Δ\+\(1−γ\)​εopt,\\displaystyle\\leq\\beta\_\{\\star\}\+\(1\-\\gamma\)\\Delta\+\(1\-\\gamma\)\\varepsilon\_\{\\mathrm\{opt\}\},which proves \([68](https://arxiv.org/html/2608.06545#A4.E68)\)\. Finally, by \([64](https://arxiv.org/html/2608.06545#A4.E64)\), for everyR≥R0R\\geq R\_\{0\},

Bemp​\(R\)−B⋆​\(R\)=R​\(βemp−β⋆\)≤R​\(1−γ\)​\(Δ\+εopt\),B\_\{\\mathrm\{emp\}\}\(R\)\-B\_\{\\star\}\(R\)=R\(\\beta\_\{\\mathrm\{emp\}\}\-\\beta\_\{\\star\}\)\\leq R\(1\-\\gamma\)\(\\Delta\+\\varepsilon\_\{\\mathrm\{opt\}\}\),which proves \([69](https://arxiv.org/html/2608.06545#A4.E69)\)\.

#### D\.5\.3Proof of Lemma[13](https://arxiv.org/html/2608.06545#Thmlemma13)

The proof has three steps\. We first decompose the error‖U^−U‖∞\\\|\\widehat\{U\}\-U\\\|\_\{\\infty\}into two branches and then bound the terms in both branches\. Finally, we combine these bounds to obtain the claimed estimate\.

##### Step 1: Error decomposition\.

In this step, we derive an error decomposition that upper\-bounds‖U^−U‖∞\\\|\\widehat\{U\}\-U\\\|\_\{\\infty\}by the maximum of two sums of more manageable terms\.

The following two lemmas control the fixed\-vector residuals\. These lemmas are used repeatedly for both the decomposition and the variance analysis\. The first treats the empirical robust kernel and is related to Lemma 8 inShiet al\.\([2026](https://arxiv.org/html/2608.06545#bib.bib3)\); our version explicitly incorporates the span to obtain a tighter bound under a weaker sample\-size condition\. The second treats the nominal kernel applied to the fixed anchor vector\. For any stochastic matrixLL, define the normalized discounted resolvent

𝒢L≔\(1−γ\)​\(I−γ​L\)−1\.\\mathcal\{G\}\_\{L\}\\coloneqq\(1\-\\gamma\)\(I\-\\gamma L\)^\{\-1\}\.\(87\)Their proofs are deferred to Appendices[D\.5\.5](https://arxiv.org/html/2608.06545#A4.SS5.SSS5)and[D\.5\.6](https://arxiv.org/html/2608.06545#A4.SS5.SSS6), respectively\.

###### Lemma 15\.

Fix a value vectorVVindependently of the empirical transition kernels\{P^s,a0\}\(s,a\)∈𝒮×𝒜\\\{\\widehat\{P\}^\{0\}\_\{s,a\}\\\}\_\{\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\}\. With probability at least1−O​\(δ\)1\-O\(\\delta\), for every policyπ\\pi,

\|\(P^π,V−Pπ,V\)​V\|≤2​V​a​rP0,π​\(V\)⋅ιN\+‖V‖span⋅ιN⋅𝟏S\.\\left\|\\left\(\\widehat\{P\}^\{\\pi,V\}\-P^\{\\pi,V\}\\right\)V\\right\|\\leq\\sqrt\{\\frac\{2\\mathrm\{Var\}\_\{P^\{0,\\pi\}\}\(V\)\\cdot\\iota\}\{N\}\}\+\\frac\{\\\|V\\\|\_\{\\mathrm\{span\}\}\\cdot\\iota\}\{N\}\\cdot\\bm\{1\}\_\{S\}\.\(88\)Moreover, for any possibly data\-dependent stochastic matrixLL,

‖𝒢L​\|\(P^π,V−Pπ,V\)​V\|‖∞≤C​ιN​‖𝒢L​VarP0,π​\(V\)‖∞\+C​‖V‖span⋅ιN\.\\left\\\|\\mathcal\{G\}\_\{L\}\\left\|\\left\(\\widehat\{P\}^\{\\pi,V\}\-P^\{\\pi,V\}\\right\)V\\right\|\\right\\\|\_\{\\infty\}\\leq C\\sqrt\{\\frac\{\\iota\}\{N\}\}\\left\\\|\\mathcal\{G\}\_\{L\}\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\pi\}\}\(V\)\}\\right\\\|\_\{\\infty\}\+C\\frac\{\\\|V\\\|\_\{\\mathrm\{span\}\}\\cdot\\iota\}\{N\}\.\(89\)

###### Lemma 16\(Fixed\-vector nominal\-kernel concentration\)\.

Fix a vectorggthat is independent of the empirical transition kernels\. With probability at least1−O​\(δ\)1\-O\(\\delta\), for every policyπ\\pi,

\|\(P^0,π−P0,π\)​g\|≤C​VarP0,π​\(g\)⋅ιN\+C​‖g‖span⋅ιN⋅𝟏S\.\\left\|\\left\(\\widehat\{P\}^\{0,\\pi\}\-P^\{0,\\pi\}\\right\)g\\right\|\\leq C\\sqrt\{\\frac\{\\mathrm\{Var\}\_\{P^\{0,\\pi\}\}\(g\)\\cdot\\iota\}\{N\}\}\+C\\frac\{\\\|g\\\|\_\{\\mathrm\{span\}\}\\cdot\\iota\}\{N\}\\cdot\\bm\{1\}\_\{S\}\.\(90\)Moreover, for any possibly data\-dependent stochastic matrixLL,

‖𝒢L​\|\(P^0,π−P0,π\)​g\|‖∞≤C​ιN​‖𝒢L​VarP0,π​\(g\)‖∞\+C​‖g‖span⋅ιN\.\\left\\\|\\mathcal\{G\}\_\{L\}\\left\|\\left\(\\widehat\{P\}^\{0,\\pi\}\-P^\{0,\\pi\}\\right\)g\\right\|\\right\\\|\_\{\\infty\}\\leq C\\sqrt\{\\frac\{\\iota\}\{N\}\}\\left\\\|\\mathcal\{G\}\_\{L\}\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\pi\}\}\(g\)\}\\right\\\|\_\{\\infty\}\+C\\frac\{\\\|g\\\|\_\{\\mathrm\{span\}\}\\cdot\\iota\}\{N\}\.\(91\)

We now describe the decomposition of the fixed\-policy errorU^−U\\widehat\{U\}\-U\. The following lemma bounds this error by the maximum of two resolvent\-perturbation terms\. The proof is deferred to Appendix[D\.5\.7](https://arxiv.org/html/2608.06545#A4.SS5.SSS7)\.

###### Lemma 17\.

ConsiderU^\\widehat\{U\}andUU\. We have that

‖U^−U‖∞\\displaystyle\\\|\\widehat\{U\}\-U\\\|\_\{\\infty\}≤max\{∥\(I−γP^πγ⋆,U\)−1\|\(P^πγ⋆,U−Pπγ⋆,U\)U\|∥∞,\\displaystyle\\leq\\max\\Bigg\\\{\\left\\\|\\left\(I\-\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)^\{\-1\}\\left\|\\left\(\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\-P^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)U\\right\|\\right\\\|\_\{\\infty\},\(92\)∥\(I−γP^πγ⋆,U^\)−1\|\(P^πγ⋆,U−Pπγ⋆,U\)U\|∥∞\}\.\\displaystyle\\qquad\\qquad\\left\\\|\\left\(I\-\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\\right\)^\{\-1\}\\left\|\\left\(\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\-P^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)U\\right\|\\right\\\|\_\{\\infty\}\\Bigg\\\}\.

For simplicity, we first focus on the first term of \([92](https://arxiv.org/html/2608.06545#A4.E92)\)\. Invoking Lemma[15](https://arxiv.org/html/2608.06545#Thmlemma15)withπ=πγ⋆\\pi=\\pi\_\{\\gamma\}^\{\\star\}andV=UV=Uand using the nonnegativity of the resolvent, we obtain

‖\(I−γ​P^πγ⋆,U\)−1​\|\(P^πγ⋆,U−Pπγ⋆,U\)​U\|‖∞\\displaystyle\\left\\\|\\left\(I\-\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)^\{\-1\}\\left\|\\left\(\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\-P^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)U\\right\|\\right\\\|\_\{\\infty\}\(93\)≤‖\(I−γ​P^πγ⋆,U\)−1​‖U‖span⋅ιN⋅𝟏S\+2⋅ιN​\(I−γ​P^πγ⋆,U\)−1​VarP0,πγ⋆​\(U\)‖∞\.\\displaystyle\\quad\\leq\\left\\\|\\left\(I\-\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)^\{\-1\}\\frac\{\\\|U\\\|\_\{\\mathrm\{span\}\}\\cdot\\iota\}\{N\}\\cdot\\bm\{1\}\_\{S\}\+\\sqrt\{\\frac\{2\\cdot\\iota\}\{N\}\}\\,\\left\(I\-\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)^\{\-1\}\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\}\(U\)\}\\right\\\|\_\{\\infty\}\.\(94\)We further decompose the variance term\. By elementary algebra, we have that

VarP0,πγ⋆​\(U\)≤\\displaystyle\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\}\(U\)\}\\leq\{\}VarP^0,πγ⋆​\(U\)\+\|VarP0,πγ⋆​\(U\)−VarP^0,πγ⋆​\(U\)\|,\\displaystyle\\sqrt\{\\mathrm\{Var\}\_\{\\widehat\{P\}^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\}\(U\)\}\+\\left\|\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\}\(U\)\}\-\\sqrt\{\\mathrm\{Var\}\_\{\\widehat\{P\}^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\}\(U\)\}\\right\|,and

VarP^0,πγ⋆​\(U\)≤\\displaystyle\\sqrt\{\\mathrm\{Var\}\_\{\\widehat\{P\}^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\}\(U\)\}\\leq\{\}VarP^πγ⋆,U​\(U\)\+\|VarP^0,πγ⋆​\(U\)−VarP^πγ⋆,U​\(U\)\|\.\\displaystyle\\sqrt\{\\mathrm\{Var\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\(U\)\}\+\\sqrt\{\\left\|\\mathrm\{Var\}\_\{\\widehat\{P\}^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\}\(U\)\-\\mathrm\{Var\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\(U\)\\right\|\}\.Substituting these into \([93](https://arxiv.org/html/2608.06545#A4.E93)\) gives

‖\(I−γ​P^πγ⋆,U\)−1​\|\(P^πγ⋆,U−Pπγ⋆,U\)​U\|‖∞≤‖∑i=14Ti‖∞\.\\left\\\|\\left\(I\-\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)^\{\-1\}\\left\|\\left\(\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\-P^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)U\\right\|\\right\\\|\_\{\\infty\}\\leq\\left\\\|\\sum\_\{i=1\}^\{4\}T\_\{i\}\\right\\\|\_\{\\infty\}\.whereT1,…,T4T\_\{1\},\\ldots,T\_\{4\}are defined by

T1\\displaystyle T\_\{1\}≔\(I−γ​P^πγ⋆,U\)−1​‖U‖span⋅ιN⋅𝟏S\\displaystyle\\coloneqq\\,\\left\(I\-\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)^\{\-1\}\\frac\{\\\|U\\\|\_\{\\mathrm\{span\}\}\\cdot\\iota\}\{N\}\\cdot\\bm\{1\}\_\{S\}\(95\)T2\\displaystyle T\_\{2\}≔2⋅ιN​\(I−γ​P^πγ⋆,U\)−1​VarP^πγ⋆,U​\(U\),\\displaystyle\\coloneqq\\sqrt\{\\frac\{2\\cdot\\iota\}\{N\}\}\\,\\left\(I\-\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)^\{\-1\}\\sqrt\{\\mathrm\{Var\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\(U\)\},T3\\displaystyle T\_\{3\}≔2⋅ιN​\(I−γ​P^πγ⋆,U\)−1​\|VarP^0,πγ⋆​\(U\)−VarP^πγ⋆,U​\(U\)\|,\\displaystyle\\coloneqq\\sqrt\{\\frac\{2\\cdot\\iota\}\{N\}\}\\,\\left\(I\-\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)^\{\-1\}\\sqrt\{\\left\|\\mathrm\{Var\}\_\{\\widehat\{P\}^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\}\(U\)\-\\mathrm\{Var\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\(U\)\\right\|\},T4\\displaystyle T\_\{4\}≔2⋅ιN​\(I−γ​P^πγ⋆,U\)−1​\|VarP0,πγ⋆​\(U\)−VarP^0,πγ⋆​\(U\)\|\.\\displaystyle\\coloneqq\\sqrt\{\\frac\{2\\cdot\\iota\}\{N\}\}\\,\\left\(I\-\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)^\{\-1\}\\left\|\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\}\(U\)\}\-\\sqrt\{\\mathrm\{Var\}\_\{\\widehat\{P\}^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\}\(U\)\}\\right\|\.Repeating the steps used for \([93](https://arxiv.org/html/2608.06545#A4.E93)\) for the second term in \([92](https://arxiv.org/html/2608.06545#A4.E92)\), we obtain

‖\(I−γ​P^πγ⋆,U^\)−1​\|\(P^πγ⋆,U−Pπγ⋆,U\)​U\|‖∞≤‖∑i=58Ti‖∞\.\\left\\\|\\left\(I\-\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\\right\)^\{\-1\}\\left\|\\left\(\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\-P^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)U\\right\|\\right\\\|\_\{\\infty\}\\leq\\left\\\|\\sum\_\{i=5\}^\{8\}T\_\{i\}\\right\\\|\_\{\\infty\}\.whereT5,…,T8T\_\{5\},\\ldots,T\_\{8\}are defined by

T5\\displaystyle T\_\{5\}≔\(I−γ​P^πγ⋆,U^\)−1​‖U‖span⋅ιN⋅𝟏S\\displaystyle\\coloneqq\\,\\left\(I\-\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\\right\)^\{\-1\}\\frac\{\\\|U\\\|\_\{\\mathrm\{span\}\}\\cdot\\iota\}\{N\}\\cdot\\bm\{1\}\_\{S\}\(96\)T6\\displaystyle T\_\{6\}≔2⋅ιN​\(I−γ​P^πγ⋆,U^\)−1​VarP^πγ⋆,U^​\(U\),\\displaystyle\\coloneqq\\sqrt\{\\frac\{2\\cdot\\iota\}\{N\}\}\\,\\left\(I\-\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\\right\)^\{\-1\}\\sqrt\{\\mathrm\{Var\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\}\(U\)\},T7\\displaystyle T\_\{7\}≔2⋅ιN​\(I−γ​P^πγ⋆,U^\)−1​\|VarP^0,πγ⋆​\(U\)−VarP^πγ⋆,U^​\(U\)\|,\\displaystyle\\coloneqq\\sqrt\{\\frac\{2\\cdot\\iota\}\{N\}\}\\,\\left\(I\-\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\\right\)^\{\-1\}\\sqrt\{\\left\|\\mathrm\{Var\}\_\{\\widehat\{P\}^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\}\(U\)\-\\mathrm\{Var\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\}\(U\)\\right\|\},T8\\displaystyle T\_\{8\}≔2⋅ιN​\(I−γ​P^πγ⋆,U^\)−1​\|VarP0,πγ⋆​\(U\)−VarP^0,πγ⋆​\(U\)\|\.\\displaystyle\\coloneqq\\sqrt\{\\frac\{2\\cdot\\iota\}\{N\}\}\\,\\left\(I\-\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\\right\)^\{\-1\}\\left\|\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\}\(U\)\}\-\\sqrt\{\\mathrm\{Var\}\_\{\\widehat\{P\}^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\}\(U\)\}\\right\|\.Putting the two parts together, we have that

‖U^−U‖∞≤max⁡\{‖∑i=14Ti‖∞,‖∑i=58Ti‖∞\}\.\\big\\\|\\widehat\{U\}\-U\\big\\\|\_\{\\infty\}\\leq\\max\\left\\\{\\left\\\|\\sum\_\{i=1\}^\{4\}T\_\{i\}\\right\\\|\_\{\\infty\},\\left\\\|\\sum\_\{i=5\}^\{8\}T\_\{i\}\\right\\\|\_\{\\infty\}\\right\\\}\.\(97\)
It remains to bound the terms in these two branches\.

##### Step 2: Bounding the error termsT1,…,T8T\_\{1\},\\ldots,T\_\{8\}\.

##### Bound onT1T\_\{1\}andT5T\_\{5\}\.

For any stochastic matrixPP, expanding the matrix inverse yields

\(I−γ​P\)−1​𝟏S≤∑t=0∞γt​Pt​𝟏S\.\\left\(I\-\\gamma P\\right\)^\{\-1\}\\bm\{1\}\_\{S\}\\leq\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}P^\{t\}\\bm\{1\}\_\{S\}\.SincePPis a stochastic matrix,Pt​𝟏S=𝟏SP^\{t\}\\bm\{1\}\_\{S\}=\\bm\{1\}\_\{S\}for allt≥0t\\geq 0\. Therefore,

\(I−γ​P\)−1​𝟏S≤11−γ⋅𝟏S\.\\left\(I\-\\gamma P\\right\)^\{\-1\}\\bm\{1\}\_\{S\}\\leq\\frac\{1\}\{1\-\\gamma\}\\cdot\\bm\{1\}\_\{S\}\.\(98\)Substituting this into the definition ofT1T\_\{1\}in \([95](https://arxiv.org/html/2608.06545#A4.E95)\) and using‖U‖span≤R\\\|U\\\|\_\{\\mathrm\{span\}\}\\leq R, we have that

T1≤R⋅ιN​\(1−γ\)⋅𝟏S\.T\_\{1\}\\leq\\frac\{R\\cdot\\iota\}\{N\(1\-\\gamma\)\}\\cdot\\bm\{1\}\_\{S\}\.\(99\)Applying \([98](https://arxiv.org/html/2608.06545#A4.E98)\) to the definition ofT5T\_\{5\}in \([96](https://arxiv.org/html/2608.06545#A4.E96)\) produces

T5≤R⋅ιN​\(1−γ\)⋅𝟏S\.T\_\{5\}\\leq\\frac\{R\\cdot\\iota\}\{N\(1\-\\gamma\)\}\\cdot\\bm\{1\}\_\{S\}\.\(100\)

##### Bound onT2T\_\{2\}andT6T\_\{6\}\.

The following lemma converts Bellman and anchor residuals into a resolvent\-variance bound\. It is the common tool behind the matched terms of the form\(𝑰−γ​P\)−1​VarP​\(V\)\(\\bm\{I\}\-\\gamma P\)^\{\-1\}\\sqrt\{\\mathrm\{Var\}\_\{P\}\(V\)\}, whereVVis a value function andPPis a stochastic kernel\. Its proof is deferred to Appendix[D\.5\.8](https://arxiv.org/html/2608.06545#A4.SS5.SSS8)\.

###### Lemma 18\(Anchored resolvent\-variance bound\)\.

Let\(ρ¯,h¯\)\(\\bar\{\\rho\},\\bar\{h\}\)satisfy \([6](https://arxiv.org/html/2608.06545#S4.E6)\), and letHancH\_\{\\mathrm\{anc\}\}be defined by \([56](https://arxiv.org/html/2608.06545#A4.E56)\)\. Fix a radiusR≥max⁡\{1,Hanc\}R\\geq\\max\\\{1,H\_\{\\mathrm\{anc\}\}\\\}\. LetPPbe a stochastic kernel selected from the true or empirical robust uncertainty set, and letbbbe a residual vector\. Suppose that

‖V‖span≤R,\(1−γ\)​mins⁡V​\(s\)≤1,V=rπ\+γ​P​V\+b\.\\\|V\\\|\_\{\\mathrm\{span\}\}\\leq R,\\qquad\(1\-\\gamma\)\\min\_\{s\}V\(s\)\\leq 1,\\qquad V=r^\{\\pi\}\+\\gamma PV\+b\.\(101\)Define the anchor defect associated withVVby

β≔\[ρ¯−\(1−γ\)​mins⁡V​\(s\)\]\+\.\\beta\\coloneqq\\left\[\\bar\{\\rho\}\-\(1\-\\gamma\)\\min\_\{s\}V\(s\)\\right\]\_\{\+\}\.\(102\)LetξV,ξh∈ℝS\\xi\_\{V\},\\xi\_\{h\}\\in\\mathbb\{R\}^\{S\}be residual envelopes such that, for a universal constantCresC\_\{\\mathrm\{res\}\},

\|b\|≤\|ξV\|,\|\(P−P0,π\)​h¯\|≤Cres​σ​Hanc⋅𝟏S\+\|ξh\|\.\|b\|\\leq\|\\xi\_\{V\}\|,\\qquad\|\(P\-P^\{0,\\pi\}\)\\bar\{h\}\|\\leq C\_\{\\mathrm\{res\}\}\\sigma H\_\{\\mathrm\{anc\}\}\\cdot\\bm\{1\}\_\{S\}\+\|\\xi\_\{h\}\|\.\(103\)Then, for a universal constantC\>0C\>0,

‖\(I−γ​P\)−1​VarP​\(V\)‖∞≤C1−γ​B​\(R,β\)\+R​‖𝒢P​\|ξV\|‖∞\+R​‖𝒢P​\|ξh\|‖∞,\\left\\\|\(I\-\\gamma P\)^\{\-1\}\\sqrt\{\\mathrm\{Var\}\_\{P\}\(V\)\}\\right\\\|\_\{\\infty\}\\leq\\frac\{C\}\{1\-\\gamma\}\\sqrt\{B\(R,\\beta\)\+R\\\|\\mathcal\{G\}\_\{P\}\|\\xi\_\{V\}\|\\\|\_\{\\infty\}\+R\\\|\\mathcal\{G\}\_\{P\}\|\\xi\_\{h\}\|\\\|\_\{\\infty\}\},whereB​\(R,β\)B\(R,\\beta\)and𝒢P\\mathcal\{G\}\_\{P\}are defined in \([63](https://arxiv.org/html/2608.06545#A4.E63)\) and \([87](https://arxiv.org/html/2608.06545#A4.E87)\), respectively\.

Recall from \([95](https://arxiv.org/html/2608.06545#A4.E95)\) that

T2=2⋅ιN​\(I−γ​P^πγ⋆,U\)−1​VarP^πγ⋆,U​\(U\)\.T\_\{2\}=\\sqrt\{\\frac\{2\\cdot\\iota\}\{N\}\}\\left\(I\-\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)^\{\-1\}\\sqrt\{\\mathrm\{Var\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\(U\)\}\.
The following lemma is an application of Lemma[18](https://arxiv.org/html/2608.06545#Thmlemma18)that boundsT2T\_\{2\}\. Its proof is deferred to Appendix[D\.5\.9](https://arxiv.org/html/2608.06545#A4.SS5.SSS9)\.

###### Lemma 19\(Matched anchored variance bound\)\.

LetR≥max⁡\{1,Hanc\}R\\geq\\max\\\{1,H\_\{\\mathrm\{anc\}\}\\\}be a deterministic radius and suppose \([70](https://arxiv.org/html/2608.06545#A4.E70)\) and \([71](https://arxiv.org/html/2608.06545#A4.E71)\) hold\. On the eventℰ​\(R\)\\mathcal\{E\}\(R\),

T2≤C​B⋆​\(R\)⋅ιN​\(1−γ\)2⋅𝟏S\+C​R⋅ιN​\(1−γ\)⋅𝟏S\.T\_\{2\}\\leq C\\sqrt\{\\frac\{B\_\{\\star\}\(R\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\\cdot\\bm\{1\}\_\{S\}\+C\\frac\{R\\cdot\\iota\}\{N\(1\-\\gamma\)\}\\cdot\\bm\{1\}\_\{S\}\.\(104\)

The resolvent kernel inT6T\_\{6\}is selected byU^\\widehat\{U\}, whereas its variance is evaluated atUU\. The following lemma is an application of Lemma[18](https://arxiv.org/html/2608.06545#Thmlemma18)that boundsT6T\_\{6\}; its proof is deferred to Appendix[D\.5\.10](https://arxiv.org/html/2608.06545#A4.SS5.SSS10)\.

###### Lemma 20\(Mismatched anchored variance bound forT6T\_\{6\}\)\.

LetR≥max⁡\{1,Hanc\}R\\geq\\max\\\{1,H\_\{\\mathrm\{anc\}\}\\\}be a deterministic radius and suppose \([70](https://arxiv.org/html/2608.06545#A4.E70)\) and \([71](https://arxiv.org/html/2608.06545#A4.E71)\) hold\. On the eventℰ​\(R\)\\mathcal\{E\}\(R\),

T6\\displaystyle T\_\{6\}≤C​B⋆​\(R\)⋅ιN​\(1−γ\)2⋅𝟏S\+C​R⋅ιN​\(1−γ\)⋅𝟏S\+150​Δ⋅𝟏S\.\\displaystyle\\leq C\\sqrt\{\\frac\{B\_\{\\star\}\(R\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\\cdot\\bm\{1\}\_\{S\}\+C\\frac\{R\\cdot\\iota\}\{N\(1\-\\gamma\)\}\\cdot\\bm\{1\}\_\{S\}\+\\frac\{1\}\{50\}\\Delta\\cdot\\bm\{1\}\_\{S\}\.\(105\)

##### Bound onT3T\_\{3\}andT7T\_\{7\}\(uncertainty\-induced TV penalty\)\.

We first focus onT3T\_\{3\}; the argument forT7T\_\{7\}is similar\. Recall from \([95](https://arxiv.org/html/2608.06545#A4.E95)\) that

T3=2⋅ιN​\(I−γ​P^πγ⋆,U\)−1​\|VarP^0,πγ⋆​\(U\)−VarP^πγ⋆,U​\(U\)\|\.T\_\{3\}=\\sqrt\{\\frac\{2\\cdot\\iota\}\{N\}\}\\,\\left\(I\-\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)^\{\-1\}\\sqrt\{\\left\|\\mathrm\{Var\}\_\{\\widehat\{P\}^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\}\(U\)\-\\mathrm\{Var\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\(U\)\\right\|\}\.LetU′≔U−mins⁡U​\(s\)⋅𝟏SU^\{\\prime\}\\coloneqq U\-\\min\_\{s\}U\(s\)\\cdot\\bm\{1\}\_\{S\}\. Since variance is invariant under shifts,

\|VarP^0,πγ⋆​\(U\)−VarP^πγ⋆,U​\(U\)\|=\|VarP^0,πγ⋆​\(U′\)−VarP^πγ⋆,U​\(U′\)\|\.\\left\|\\mathrm\{Var\}\_\{\\widehat\{P\}^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\}\(U\)\-\\mathrm\{Var\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\(U\)\\right\|=\\left\|\\mathrm\{Var\}\_\{\\widehat\{P\}^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\}\(U^\{\\prime\}\)\-\\mathrm\{Var\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\(U^\{\\prime\}\)\\right\|\.By Hölder’s inequality,

\|VarP^0,πγ⋆​\(U′\)−VarP^πγ⋆,U​\(U′\)\|≤‖P^0,πγ⋆−P^πγ⋆,U‖1​‖U′‖∞2≤2​σ​‖U′‖∞2=2​σ​‖U‖span2\.\\left\|\\mathrm\{Var\}\_\{\\widehat\{P\}^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\}\(U^\{\\prime\}\)\-\\mathrm\{Var\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\(U^\{\\prime\}\)\\right\|\\leq\\\|\\widehat\{P\}^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\-\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\\|\_\{1\}\\\|U^\{\\prime\}\\\|\_\{\\infty\}^\{2\}\\leq 2\\sigma\\\|U^\{\\prime\}\\\|\_\{\\infty\}^\{2\}=2\\sigma\\\|U\\\|\_\{\\mathrm\{span\}\}^\{2\}\.Substituting this into the definition ofT3T\_\{3\}in \([95](https://arxiv.org/html/2608.06545#A4.E95)\), using \([98](https://arxiv.org/html/2608.06545#A4.E98)\), and using‖U‖span≤R\\\|U\\\|\_\{\\mathrm\{span\}\}\\leq Rfrom \([70](https://arxiv.org/html/2608.06545#A4.E70)\), we have that

T3≤2⋅ιN​\(1−γ\)2⋅2​σ⋅R⋅𝟏S\.T\_\{3\}\\leq\\sqrt\{\\frac\{2\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\\cdot\\sqrt\{2\\sigma\}\\cdot R\\cdot\\bm\{1\}\_\{S\}\.Therefore,

T3≤C​B⋆​\(R\)⋅ιN​\(1−γ\)2⋅𝟏S\.T\_\{3\}\\leq C\\sqrt\{\\frac\{B\_\{\\star\}\(R\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\\cdot\\bm\{1\}\_\{S\}\.\(106\)Repeating this calculation forT7T\_\{7\}in \([96](https://arxiv.org/html/2608.06545#A4.E96)\) results in

T7≤C​B⋆​\(R\)⋅ιN​\(1−γ\)2⋅𝟏S\.T\_\{7\}\\leq C\\sqrt\{\\frac\{B\_\{\\star\}\(R\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\\cdot\\bm\{1\}\_\{S\}\.\(107\)

##### Bound onT4T\_\{4\}andT8T\_\{8\}\.

The termsT4T\_\{4\}andT8T\_\{8\}come from replacing the true nominal variance by the empirical nominal variance in the Bernstein part of Lemma[15](https://arxiv.org/html/2608.06545#Thmlemma15)\. The relevant object is a standard deviation, so the perturbation is of order‖U‖span​ι/N\\\|U\\\|\_\{\\mathrm\{span\}\}\\sqrt\{\\iota/N\}rather than a full variance perturbation\.

###### Lemma 21\(Empirical standard\-deviation perturbation; Lemma 11 ofPanaganti and Kalathil \([2022](https://arxiv.org/html/2608.06545#bib.bib29)\)\)\.

Fix a value vectorVVthat is independent of the empirical nominal transition kernel, and letπ\\pibe a fixed deterministic policy\. With probability at least1−δ1\-\\delta, for every statess,

\|VarP0,π​\(V\)​\(s\)−VarP^0,π​\(V\)​\(s\)\|≤2​‖V‖span​2​ιN\.\\left\|\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\pi\}\}\(V\)\(s\)\}\-\\sqrt\{\\mathrm\{Var\}\_\{\\widehat\{P\}^\{0,\\pi\}\}\(V\)\(s\)\}\\right\|\\leq 2\\\|V\\\|\_\{\\mathrm\{span\}\}\\sqrt\{2\\frac\{\\iota\}\{N\}\}\.

We apply Lemma[21](https://arxiv.org/html/2608.06545#Thmlemma21)withV=UV=Uandπ=πγ⋆\\pi=\\pi\_\{\\gamma\}^\{\\star\}\. Using‖U‖span≤R\\\|U\\\|\_\{\\mathrm\{span\}\}\\leq Rfrom \([70](https://arxiv.org/html/2608.06545#A4.E70)\) and \([98](https://arxiv.org/html/2608.06545#A4.E98)\), we have that

T4≤C​R⋅ιN​\(1−γ\)⋅𝟏S\.T\_\{4\}\\leq C\\frac\{R\\cdot\\iota\}\{N\(1\-\\gamma\)\}\\cdot\\bm\{1\}\_\{S\}\.\(108\)Applying Lemma[21](https://arxiv.org/html/2608.06545#Thmlemma21)and \([98](https://arxiv.org/html/2608.06545#A4.E98)\) toT8T\_\{8\}in \([96](https://arxiv.org/html/2608.06545#A4.E96)\) yields

T8≤C​R⋅ιN​\(1−γ\)⋅𝟏S\.T\_\{8\}\\leq C\\frac\{R\\cdot\\iota\}\{N\(1\-\\gamma\)\}\\cdot\\bm\{1\}\_\{S\}\.\(109\)With all eight terms controlled, it remains to combine the two branches\.

##### Step 3: Putting the bounds together\.

Combining \([99](https://arxiv.org/html/2608.06545#A4.E99)\), \([104](https://arxiv.org/html/2608.06545#A4.E104)\), \([106](https://arxiv.org/html/2608.06545#A4.E106)\), and \([108](https://arxiv.org/html/2608.06545#A4.E108)\), we obtain

∑i=14Ti\\displaystyle\\sum\_\{i=1\}^\{4\}T\_\{i\}≤C​B⋆​\(R\)⋅ιN​\(1−γ\)2⋅𝟏S\+C​R⋅ιN​\(1−γ\)⋅𝟏S\.\\displaystyle\\leq C\\sqrt\{\\frac\{B\_\{\\star\}\(R\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\\cdot\\bm\{1\}\_\{S\}\+C\\frac\{R\\cdot\\iota\}\{N\(1\-\\gamma\)\}\\cdot\\bm\{1\}\_\{S\}\.Similarly, using \([100](https://arxiv.org/html/2608.06545#A4.E100)\), \([105](https://arxiv.org/html/2608.06545#A4.E105)\), \([107](https://arxiv.org/html/2608.06545#A4.E107)\), and \([109](https://arxiv.org/html/2608.06545#A4.E109)\),

∑i=58Ti\\displaystyle\\sum\_\{i=5\}^\{8\}T\_\{i\}≤C​B⋆​\(R\)⋅ιN​\(1−γ\)2⋅𝟏S\+C​R⋅ιN​\(1−γ\)⋅𝟏S\+150​Δ⋅𝟏S\.\\displaystyle\\leq C\\sqrt\{\\frac\{B\_\{\\star\}\(R\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\\cdot\\bm\{1\}\_\{S\}\+C\\frac\{R\\cdot\\iota\}\{N\(1\-\\gamma\)\}\\cdot\\bm\{1\}\_\{S\}\+\\frac\{1\}\{50\}\\Delta\\cdot\\bm\{1\}\_\{S\}\.Combining the two branches in \([97](https://arxiv.org/html/2608.06545#A4.E97)\), we have that

‖U^−U‖∞\\displaystyle\\big\\\|\\widehat\{U\}\-U\\big\\\|\_\{\\infty\}≤C​B⋆​\(R\)⋅ιN​\(1−γ\)2\+C​R⋅ιN​\(1−γ\)\+150​Δ\.\\displaystyle\\leq C\\sqrt\{\\frac\{B\_\{\\star\}\(R\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\+C\\frac\{R\\cdot\\iota\}\{N\(1\-\\gamma\)\}\+\\frac\{1\}\{50\}\\Delta\.This proves the first assertion in Lemma[13](https://arxiv.org/html/2608.06545#Thmlemma13)\. The argument within this proof usesπγ⋆\\pi\_\{\\gamma\}^\{\\star\}only as a deterministic policy that is independent of the empirical transition samples\. Replacing it throughout by any such policyπ\\pi, replacingΔ\\Deltaby

‖V^γπ,σ−Vγπ,σ‖∞,\\left\\\|\\widehat\{V\}\_\{\\gamma\}^\{\\pi,\\sigma\}\-V\_\{\\gamma\}^\{\\pi,\\sigma\}\\right\\\|\_\{\\infty\},and repeating the argument gives \([72](https://arxiv.org/html/2608.06545#A4.E72)\)\. This completes the proof of Lemma[13](https://arxiv.org/html/2608.06545#Thmlemma13)\.

#### D\.5\.4Proof of Lemma[14](https://arxiv.org/html/2608.06545#Thmlemma14)

Recall that

W^≔V^γπ^,σ,W≔Vγπ^,σ,V^⋆≔V^γ⋆,σ\.\\widehat\{W\}\\coloneqq\\widehat\{V\}\_\{\\gamma\}^\{\\widehat\{\\pi\},\\sigma\},\\qquad W\\coloneqq V\_\{\\gamma\}^\{\\widehat\{\\pi\},\\sigma\},\\qquad\\widehat\{V\}^\{\\star\}\\coloneqq\\widehat\{V\}\_\{\\gamma\}^\{\\star,\\sigma\}\.The proof has three steps\. We first decompose the error‖W^−W‖∞\\\|\\widehat\{W\}\-W\\\|\_\{\\infty\}into two branches and then bound the terms in both branches\. Finally, we combine these bounds to obtain the claimed estimate\.

##### Step 1: Error decomposition\.

In this step, we derive an error decomposition that upper\-bounds‖W^−W‖∞\\\|\\widehat\{W\}\-W\\\|\_\{\\infty\}by the maximum of two sums of more manageable terms\.

The following lemma controls the empirical\-to\-true robust Bellman difference atW^\\widehat\{W\}\. The proof is deferred to Appendix[D\.5\.11](https://arxiv.org/html/2608.06545#A4.SS5.SSS11)\.

###### Lemma 22\.

LetR≥1R\\geq 1be deterministic\. For a sufficiently large numerical constantC0C\_\{0\}, supposeN≥C0⋅ι/\(1−γ\)N\\geq C\_\{0\}\\cdot\\iota/\(1\-\\gamma\)\. Then, with probability at least1−O​\(δ\)1\-O\(\\delta\), the following bound holds whenever the solver guarantee \([5](https://arxiv.org/html/2608.06545#S4.E5)\) holds and the span bounds

‖W^‖span≤R,‖V^⋆‖span≤R\\\|\\widehat\{W\}\\\|\_\{\\mathrm\{span\}\}\\leq R,\\qquad\\\|\\widehat\{V\}^\{\\star\}\\\|\_\{\\mathrm\{span\}\}\\leq Rare satisfied:

\|\(P^π^,W^−Pπ^,W^\)​W^\|≤C​ιN​VarP0,π^​\(V^⋆\)\+C​R⋅ιN⋅𝟏S\+C​εopt⋅𝟏S\.\\left\|\\left\(\\widehat\{P\}^\{\\widehat\{\\pi\},\\widehat\{W\}\}\-P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\\right\)\\widehat\{W\}\\right\|\\leq C\\sqrt\{\\frac\{\\iota\}\{N\}\}\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\widehat\{\\pi\}\}\}\(\\widehat\{V\}^\{\\star\}\)\}\+C\\frac\{R\\cdot\\iota\}\{N\}\\cdot\\bm\{1\}\_\{S\}\+C\\varepsilon\_\{\\mathrm\{opt\}\}\\cdot\\bm\{1\}\_\{S\}\.\(110\)

We now describe the decomposition of the learned\-policy errorW^−W\\widehat\{W\}\-W\. The following lemma bounds the error by the maximum of two resolvent\-perturbation terms\. The proof is deferred to Appendix[D\.5\.12](https://arxiv.org/html/2608.06545#A4.SS5.SSS12)\.

###### Lemma 23\.

ConsiderW^\\widehat\{W\}andWW\. We have that

‖W^−W‖∞\\displaystyle\\\|\\widehat\{W\}\-W\\\|\_\{\\infty\}≤max\{∥\(I−γPπ^,W^\)−1\|\(P^π^,W^−Pπ^,W^\)W^\|∥∞,\\displaystyle\\leq\\max\\Bigg\\\{\\left\\\|\\left\(I\-\\gamma P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\\right\)^\{\-1\}\\left\|\\left\(\\widehat\{P\}^\{\\widehat\{\\pi\},\\widehat\{W\}\}\-P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\\right\)\\widehat\{W\}\\right\|\\right\\\|\_\{\\infty\},∥\(I−γPπ^,W\)−1\|\(P^π^,W^−Pπ^,W^\)W^\|∥∞\}\.\\displaystyle\\qquad\\qquad\\left\\\|\\left\(I\-\\gamma P^\{\\widehat\{\\pi\},W\}\\right\)^\{\-1\}\\left\|\\left\(\\widehat\{P\}^\{\\widehat\{\\pi\},\\widehat\{W\}\}\-P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\\right\)\\widehat\{W\}\\right\|\\right\\\|\_\{\\infty\}\\Bigg\\\}\.\(111\)

We now follow the same termwise structure as in the proof of Lemma[13](https://arxiv.org/html/2608.06545#Thmlemma13)\. The triangle inequality for standard deviations implies

VarP0,π^​\(V^⋆\)\\displaystyle\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\widehat\{\\pi\}\}\}\(\\widehat\{V\}^\{\\star\}\)\}≤VarPπ^,W^​\(W^\)\+VarPπ^,W^​\(V^⋆−W^\)\\displaystyle\\leq\\sqrt\{\\mathrm\{Var\}\_\{P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\}\(\\widehat\{W\}\)\}\+\\sqrt\{\\mathrm\{Var\}\_\{P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\}\(\\widehat\{V\}^\{\\star\}\-\\widehat\{W\}\)\}\+\|VarP0,π^​\(V^⋆\)−VarPπ^,W^​\(V^⋆\)\|\.\\displaystyle\\quad\+\\left\|\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\widehat\{\\pi\}\}\}\(\\widehat\{V\}^\{\\star\}\)\}\-\\sqrt\{\\mathrm\{Var\}\_\{P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\}\(\\widehat\{V\}^\{\\star\}\)\}\\right\|\.Applying Lemma[22](https://arxiv.org/html/2608.06545#Thmlemma22)to the first branch in \([23](https://arxiv.org/html/2608.06545#A4.Ex301)\) and using the preceding display, we obtain

‖\(I−γ​Pπ^,W^\)−1​\|\(P^π^,W^−Pπ^,W^\)​W^\|‖∞≤‖∑i=14Ti′‖∞,\\left\\\|\\left\(I\-\\gamma P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\\right\)^\{\-1\}\\left\|\\left\(\\widehat\{P\}^\{\\widehat\{\\pi\},\\widehat\{W\}\}\-P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\\right\)\\widehat\{W\}\\right\|\\right\\\|\_\{\\infty\}\\leq\\left\\\|\\sum\_\{i=1\}^\{4\}T\_\{i\}^\{\\prime\}\\right\\\|\_\{\\infty\},where

T1′\\displaystyle T\_\{1\}^\{\\prime\}≔\(I−γ​Pπ^,W^\)−1​\(C​R⋅ιN\+C​εopt\)⋅𝟏S,\\displaystyle\\coloneqq\\left\(I\-\\gamma P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\\right\)^\{\-1\}\\left\(C\\frac\{R\\cdot\\iota\}\{N\}\+C\\varepsilon\_\{\\mathrm\{opt\}\}\\right\)\\cdot\\bm\{1\}\_\{S\},\(112\)T2′\\displaystyle T\_\{2\}^\{\\prime\}≔C​ιN​\(I−γ​Pπ^,W^\)−1​VarPπ^,W^​\(W^\),\\displaystyle\\coloneqq C\\sqrt\{\\frac\{\\iota\}\{N\}\}\\left\(I\-\\gamma P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\\right\)^\{\-1\}\\sqrt\{\\mathrm\{Var\}\_\{P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\}\(\\widehat\{W\}\)\},T3′\\displaystyle T\_\{3\}^\{\\prime\}≔C​ιN​\(I−γ​Pπ^,W^\)−1​VarPπ^,W^​\(V^⋆−W^\),\\displaystyle\\coloneqq C\\sqrt\{\\frac\{\\iota\}\{N\}\}\\left\(I\-\\gamma P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\\right\)^\{\-1\}\\sqrt\{\\mathrm\{Var\}\_\{P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\}\(\\widehat\{V\}^\{\\star\}\-\\widehat\{W\}\)\},T4′\\displaystyle T\_\{4\}^\{\\prime\}≔C​ιN​\(I−γ​Pπ^,W^\)−1​\|VarP0,π^​\(V^⋆\)−VarPπ^,W^​\(V^⋆\)\|\.\\displaystyle\\coloneqq C\\sqrt\{\\frac\{\\iota\}\{N\}\}\\left\(I\-\\gamma P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\\right\)^\{\-1\}\\left\|\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\widehat\{\\pi\}\}\}\(\\widehat\{V\}^\{\\star\}\)\}\-\\sqrt\{\\mathrm\{Var\}\_\{P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\}\(\\widehat\{V\}^\{\\star\}\)\}\\right\|\.The same argument for the second branch yields

‖\(I−γ​Pπ^,W\)−1​\|\(P^π^,W^−Pπ^,W^\)​W^\|‖∞≤‖∑i=58Ti′‖∞,\\left\\\|\\left\(I\-\\gamma P^\{\\widehat\{\\pi\},W\}\\right\)^\{\-1\}\\left\|\\left\(\\widehat\{P\}^\{\\widehat\{\\pi\},\\widehat\{W\}\}\-P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\\right\)\\widehat\{W\}\\right\|\\right\\\|\_\{\\infty\}\\leq\\left\\\|\\sum\_\{i=5\}^\{8\}T\_\{i\}^\{\\prime\}\\right\\\|\_\{\\infty\},where

T5′\\displaystyle T\_\{5\}^\{\\prime\}≔\(I−γ​Pπ^,W\)−1​\(C​R⋅ιN\+C​εopt\)⋅𝟏S,\\displaystyle\\coloneqq\\left\(I\-\\gamma P^\{\\widehat\{\\pi\},W\}\\right\)^\{\-1\}\\left\(C\\frac\{R\\cdot\\iota\}\{N\}\+C\\varepsilon\_\{\\mathrm\{opt\}\}\\right\)\\cdot\\bm\{1\}\_\{S\},\(113\)T6′\\displaystyle T\_\{6\}^\{\\prime\}≔C​ιN​\(I−γ​Pπ^,W\)−1​VarPπ^,W^​\(W^\),\\displaystyle\\coloneqq C\\sqrt\{\\frac\{\\iota\}\{N\}\}\\left\(I\-\\gamma P^\{\\widehat\{\\pi\},W\}\\right\)^\{\-1\}\\sqrt\{\\mathrm\{Var\}\_\{P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\}\(\\widehat\{W\}\)\},T7′\\displaystyle T\_\{7\}^\{\\prime\}≔C​ιN​\(I−γ​Pπ^,W\)−1​VarPπ^,W^​\(V^⋆−W^\),\\displaystyle\\coloneqq C\\sqrt\{\\frac\{\\iota\}\{N\}\}\\left\(I\-\\gamma P^\{\\widehat\{\\pi\},W\}\\right\)^\{\-1\}\\sqrt\{\\mathrm\{Var\}\_\{P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\}\(\\widehat\{V\}^\{\\star\}\-\\widehat\{W\}\)\},T8′\\displaystyle T\_\{8\}^\{\\prime\}≔C​ιN​\(I−γ​Pπ^,W\)−1​\|VarP0,π^​\(V^⋆\)−VarPπ^,W^​\(V^⋆\)\|\.\\displaystyle\\coloneqq C\\sqrt\{\\frac\{\\iota\}\{N\}\}\\left\(I\-\\gamma P^\{\\widehat\{\\pi\},W\}\\right\)^\{\-1\}\\left\|\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\widehat\{\\pi\}\}\}\(\\widehat\{V\}^\{\\star\}\)\}\-\\sqrt\{\\mathrm\{Var\}\_\{P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\}\(\\widehat\{V\}^\{\\star\}\)\}\\right\|\.Combining the two branches in \([23](https://arxiv.org/html/2608.06545#A4.Ex301)\), we obtain

‖W^−W‖∞≤max⁡\{‖∑i=14Ti′‖∞,‖∑i=58Ti′‖∞\}\.\\\|\\widehat\{W\}\-W\\\|\_\{\\infty\}\\leq\\max\\left\\\{\\left\\\|\\sum\_\{i=1\}^\{4\}T\_\{i\}^\{\\prime\}\\right\\\|\_\{\\infty\},\\left\\\|\\sum\_\{i=5\}^\{8\}T\_\{i\}^\{\\prime\}\\right\\\|\_\{\\infty\}\\right\\\}\.\(114\)It remains to bound the terms in these two branches\.

##### Step 2: Bounding the error termsT1′,…,T8′T\_\{1\}^\{\\prime\},\\ldots,T\_\{8\}^\{\\prime\}\.

##### Bound onT1′T\_\{1\}^\{\\prime\}andT5′T\_\{5\}^\{\\prime\}\.

Applying \([98](https://arxiv.org/html/2608.06545#A4.E98)\) to \([112](https://arxiv.org/html/2608.06545#A4.E112)\) and \([113](https://arxiv.org/html/2608.06545#A4.E113)\) yields

max⁡\{‖T1′‖∞,‖T5′‖∞\}≤C​R⋅ιN​\(1−γ\)\+C​εopt1−γ\.\\max\\left\\\{\\\|T\_\{1\}^\{\\prime\}\\\|\_\{\\infty\},\\\|T\_\{5\}^\{\\prime\}\\\|\_\{\\infty\}\\right\\\}\\leq C\\frac\{R\\cdot\\iota\}\{N\(1\-\\gamma\)\}\+C\\frac\{\\varepsilon\_\{\\mathrm\{opt\}\}\}\{1\-\\gamma\}\.\(115\)

##### Bound onT2′T\_\{2\}^\{\\prime\}andT6′T\_\{6\}^\{\\prime\}\.

These are the principal resolvent\-variance terms\. The first is matched because its resolvent kernel is selected atW^\\widehat\{W\}, while the second is mismatched because its resolvent kernel is selected atWW\. The next two lemmas provide the corresponding bounds; their proofs are deferred to Appendices[D\.5\.13](https://arxiv.org/html/2608.06545#A4.SS5.SSS13)and[D\.5\.14](https://arxiv.org/html/2608.06545#A4.SS5.SSS14)\.

###### Lemma 24\(Matched learned\-policy variance bound forT2′T\_\{2\}^\{\\prime\}\)\.

LetR≥max⁡\{1,Hanc\}R\\geq\\max\\\{1,H\_\{\\mathrm\{anc\}\}\\\}be a deterministic radius and suppose \([70](https://arxiv.org/html/2608.06545#A4.E70)\) and \([71](https://arxiv.org/html/2608.06545#A4.E71)\) hold\. On the eventℰ​\(R\)\\mathcal\{E\}\(R\),

‖T2′‖∞≤C​Bemp​\(R\)⋅ιN​\(1−γ\)2\+C​R⋅ιN​\(1−γ\)\+C​εopt1−γ\.\\\|T\_\{2\}^\{\\prime\}\\\|\_\{\\infty\}\\leq C\\sqrt\{\\frac\{B\_\{\\mathrm\{emp\}\}\(R\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\+C\\frac\{R\\cdot\\iota\}\{N\(1\-\\gamma\)\}\+C\\frac\{\\varepsilon\_\{\\mathrm\{opt\}\}\}\{1\-\\gamma\}\.\(116\)

###### Lemma 25\(Mismatched learned\-policy variance bound forT6′T\_\{6\}^\{\\prime\}\)\.

LetR≥max⁡\{1,Hanc\}R\\geq\\max\\\{1,H\_\{\\mathrm\{anc\}\}\\\}be a deterministic radius and suppose \([70](https://arxiv.org/html/2608.06545#A4.E70)\) and \([71](https://arxiv.org/html/2608.06545#A4.E71)\) hold\. On the eventℰ​\(R\)\\mathcal\{E\}\(R\),

‖T6′‖∞≤C​Bemp​\(R\)⋅ιN​\(1−γ\)2\+C​R⋅ιN​\(1−γ\)\+120​‖W^−W‖∞\+C​εopt1−γ\.\\\|T\_\{6\}^\{\\prime\}\\\|\_\{\\infty\}\\leq C\\sqrt\{\\frac\{B\_\{\\mathrm\{emp\}\}\(R\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\+C\\frac\{R\\cdot\\iota\}\{N\(1\-\\gamma\)\}\+\\frac\{1\}\{20\}\\\|\\widehat\{W\}\-W\\\|\_\{\\infty\}\+C\\frac\{\\varepsilon\_\{\\mathrm\{opt\}\}\}\{1\-\\gamma\}\.\(117\)

##### Bound onT3′T\_\{3\}^\{\\prime\}andT7′T\_\{7\}^\{\\prime\}\.

Optimality and the solver guarantee \([5](https://arxiv.org/html/2608.06545#S4.E5)\) ensure that

0≤V^⋆−W^≤εopt⋅𝟏S\.0\\leq\\widehat\{V\}^\{\\star\}\-\\widehat\{W\}\\leq\\varepsilon\_\{\\mathrm\{opt\}\}\\cdot\\bm\{1\}\_\{S\}\.Hence

VarPπ^,W^​\(V^⋆−W^\)≤εopt⋅𝟏S\.\\sqrt\{\\mathrm\{Var\}\_\{P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\}\(\\widehat\{V\}^\{\\star\}\-\\widehat\{W\}\)\}\\leq\\varepsilon\_\{\\mathrm\{opt\}\}\\cdot\\bm\{1\}\_\{S\}\.Using \([98](https://arxiv.org/html/2608.06545#A4.E98)\),N≥ιN\\geq\\iota, and the definitions in \([112](https://arxiv.org/html/2608.06545#A4.E112)\) and \([113](https://arxiv.org/html/2608.06545#A4.E113)\), we obtain

max⁡\{‖T3′‖∞,‖T7′‖∞\}≤C​εopt1−γ\.\\max\\left\\\{\\\|T\_\{3\}^\{\\prime\}\\\|\_\{\\infty\},\\\|T\_\{7\}^\{\\prime\}\\\|\_\{\\infty\}\\right\\\}\\leq C\\frac\{\\varepsilon\_\{\\mathrm\{opt\}\}\}\{1\-\\gamma\}\.\(118\)

##### Bound onT4′T\_\{4\}^\{\\prime\}andT8′T\_\{8\}^\{\\prime\}\.

We use the following deterministic comparison whenever two transition distributions are close in total variation\. Its proof is deferred to Appendix[D\.5\.15](https://arxiv.org/html/2608.06545#A4.SS5.SSS15)\.

###### Lemma 26\(Standard\-deviation perturbation under total variation\)\.

For any probability vectorsq,q′q,q^\{\\prime\}on the same finite space and any vectorffof the corresponding dimension,

\|Varq​\(f\)−Varq′​\(f\)\|≤C​‖f‖span​‖q−q′‖TV,\\left\|\\sqrt\{\\mathrm\{Var\}\_\{q\}\(f\)\}\-\\sqrt\{\\mathrm\{Var\}\_\{q^\{\\prime\}\}\(f\)\}\\right\|\\leq C\\\|f\\\|\_\{\\mathrm\{span\}\}\\sqrt\{\\\|q\-q^\{\\prime\}\\\|\_\{\\mathrm\{TV\}\}\},\(119\)whereC\>0C\>0is a universal constant\.

Convexity of total variation and the robust\-set constraint imply

‖Pπ^,W^​\(s,⋅\)−P0,π^​\(s,⋅\)‖TV≤σ,s∈𝒮\.\\left\\\|P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\(s,\\cdot\)\-P^\{0,\\widehat\{\\pi\}\}\(s,\\cdot\)\\right\\\|\_\{\\mathrm\{TV\}\}\\leq\\sigma,\\qquad s\\in\\mathcal\{S\}\.Lemma[26](https://arxiv.org/html/2608.06545#Thmlemma26), together with‖V^⋆‖span≤R\\\|\\widehat\{V\}^\{\\star\}\\\|\_\{\\mathrm\{span\}\}\\leq Rfrom \([70](https://arxiv.org/html/2608.06545#A4.E70)\), therefore implies

\|VarP0,π^​\(V^⋆\)−VarPπ^,W^​\(V^⋆\)\|≤C​R​σ⋅𝟏S\.\\left\|\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\widehat\{\\pi\}\}\}\(\\widehat\{V\}^\{\\star\}\)\}\-\\sqrt\{\\mathrm\{Var\}\_\{P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\}\(\\widehat\{V\}^\{\\star\}\)\}\\right\|\\leq CR\\sqrt\{\\sigma\}\\cdot\\bm\{1\}\_\{S\}\.Using \([98](https://arxiv.org/html/2608.06545#A4.E98)\) andBemp​\(R\)≥σ​R2B\_\{\\mathrm\{emp\}\}\(R\)\\geq\\sigma R^\{2\}, we obtain

max⁡\{‖T4′‖∞,‖T8′‖∞\}≤C​Bemp​\(R\)⋅ιN​\(1−γ\)2\.\\max\\left\\\{\\\|T\_\{4\}^\{\\prime\}\\\|\_\{\\infty\},\\\|T\_\{8\}^\{\\prime\}\\\|\_\{\\infty\}\\right\\\}\\leq C\\sqrt\{\\frac\{B\_\{\\mathrm\{emp\}\}\(R\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\.\(120\)With all eight terms controlled, it remains to combine the two branches\.

##### Step 3: Putting the bounds together\.

Combining \([115](https://arxiv.org/html/2608.06545#A4.E115)\), \([116](https://arxiv.org/html/2608.06545#A4.E116)\), \([118](https://arxiv.org/html/2608.06545#A4.E118)\), and \([120](https://arxiv.org/html/2608.06545#A4.E120)\) yields

‖∑i=14Ti′‖∞≤C​Bemp​\(R\)⋅ιN​\(1−γ\)2\+C​R⋅ιN​\(1−γ\)\+C​εopt1−γ,\\left\\\|\\sum\_\{i=1\}^\{4\}T\_\{i\}^\{\\prime\}\\right\\\|\_\{\\infty\}\\leq C\\sqrt\{\\frac\{B\_\{\\mathrm\{emp\}\}\(R\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\+C\\frac\{R\\cdot\\iota\}\{N\(1\-\\gamma\)\}\+C\\frac\{\\varepsilon\_\{\\mathrm\{opt\}\}\}\{1\-\\gamma\},whereas \([115](https://arxiv.org/html/2608.06545#A4.E115)\), \([117](https://arxiv.org/html/2608.06545#A4.E117)\), \([118](https://arxiv.org/html/2608.06545#A4.E118)\), and \([120](https://arxiv.org/html/2608.06545#A4.E120)\) lead to

‖∑i=58Ti′‖∞≤C​Bemp​\(R\)⋅ιN​\(1−γ\)2\+C​R⋅ιN​\(1−γ\)\+120​‖W^−W‖∞\+C​εopt1−γ\.\\left\\\|\\sum\_\{i=5\}^\{8\}T\_\{i\}^\{\\prime\}\\right\\\|\_\{\\infty\}\\leq C\\sqrt\{\\frac\{B\_\{\\mathrm\{emp\}\}\(R\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\+C\\frac\{R\\cdot\\iota\}\{N\(1\-\\gamma\)\}\+\\frac\{1\}\{20\}\\\|\\widehat\{W\}\-W\\\|\_\{\\infty\}\+C\\frac\{\\varepsilon\_\{\\mathrm\{opt\}\}\}\{1\-\\gamma\}\.Substituting these two estimates into \([114](https://arxiv.org/html/2608.06545#A4.E114)\) and using‖W^−W‖∞≤Δ\\\|\\widehat\{W\}\-W\\\|\_\{\\infty\}\\leq\\Deltafrom \([67](https://arxiv.org/html/2608.06545#A4.E67)\) completes the proof of Lemma[14](https://arxiv.org/html/2608.06545#Thmlemma14)\.

#### D\.5\.5Proof of Lemma[15](https://arxiv.org/html/2608.06545#Thmlemma15)

The proof has three steps\. We first use TV duality and discretization to reduce the robust\-kernel error to a finite family of nominal empirical processes\. We then establish a uniform Bernstein bound for this family\. Finally, we average the rowwise estimate over an arbitrary policy and derive the normalized\-resolvent bound\.

##### Step 1: Reduction and discretization\.

In this step, we reduce the robust\-kernel error for a fixed state\-action pair to a clipped nominal empirical process and discretize its clipping level\. Fix a value vectorVVthat is independent of the empirical transition kernels, and set

V∘≔V−mins⁡V​\(s\)⋅𝟏S\.V^\{\\circ\}\\coloneqq V\-\\min\_\{s\}V\(s\)\\cdot\\bm\{1\}\_\{S\}\.The robust expectation and variance are invariant under adding constants toVV, and

0≤V∘≤‖V‖span⋅𝟏S,‖V∘‖∞=‖V‖span\.0\\leq V^\{\\circ\}\\leq\\\|V\\\|\_\{\\mathrm\{span\}\}\\cdot\\bm\{1\}\_\{S\},\\qquad\\\|V^\{\\circ\}\\\|\_\{\\infty\}=\\\|V\\\|\_\{\\mathrm\{span\}\}\.Fix a state\-action pair\(s,a\)\(s,a\)\. Applying the TV strong\-duality formula inShiet al\.\([2026](https://arxiv.org/html/2608.06545#bib.bib3), Lemma 4\)toV∘V^\{\\circ\}yields

Ps,aV​V\\displaystyle P\_\{s,a\}^\{V\}V=mins⁡V​\(s\)\+maxα∈\[0,‖V‖span\]⁡\{Ps,a0​min⁡\{V∘,α\}−σs,a​α\},\\displaystyle=\\min\_\{s\}V\(s\)\+\\max\_\{\\alpha\\in\[0,\\\|V\\\|\_\{\\mathrm\{span\}\}\]\}\\left\\\{P^\{0\}\_\{s,a\}\\min\\\{V^\{\\circ\},\\alpha\\\}\-\\sigma\_\{s,a\}\\alpha\\right\\\},P^s,aV​V\\displaystyle\\widehat\{P\}\_\{s,a\}^\{V\}V=mins⁡V​\(s\)\+maxα∈\[0,‖V‖span\]⁡\{P^s,a0​min⁡\{V∘,α\}−σs,a​α\}\.\\displaystyle=\\min\_\{s\}V\(s\)\+\\max\_\{\\alpha\\in\[0,\\\|V\\\|\_\{\\mathrm\{span\}\}\]\}\\left\\\{\\widehat\{P\}^\{0\}\_\{s,a\}\\min\\\{V^\{\\circ\},\\alpha\\\}\-\\sigma\_\{s,a\}\\alpha\\right\\\}\.The termsmins⁡V​\(s\)\\min\_\{s\}V\(s\)and−σs,a​α\-\\sigma\_\{s,a\}\\alphaare common to the two expressions\. Since the difference of two maxima is at most the maximum pointwise difference, we obtain

\|P^s,aV​V−Ps,aV​V\|≤supα∈\[0,‖V‖span\]\|\(P^s,a0−Ps,a0\)​min⁡\{V∘,α\}\|,\\left\|\\widehat\{P\}\_\{s,a\}^\{V\}V\-P\_\{s,a\}^\{V\}V\\right\|\\leq\\sup\_\{\\alpha\\in\[0,\\\|V\\\|\_\{\\mathrm\{span\}\}\]\}\\left\|\(\\widehat\{P\}^\{0\}\_\{s,a\}\-P^\{0\}\_\{s,a\}\)\\min\\\{V^\{\\circ\},\\alpha\\\}\\right\|,where the minimum is taken coordinatewise\.

If‖V‖span=0\\\|V\\\|\_\{\\mathrm\{span\}\}=0, the displayed supremum is zero\. It therefore remains to consider‖V‖span\>0\\\|V\\\|\_\{\\mathrm\{span\}\}\>0\. Define the clipping grid

𝒩α≔\{k​‖V‖span6​N:0≤k≤6​N\}\.\\mathcal\{N\}\_\{\\alpha\}\\coloneqq\\left\\\{\\frac\{k\\\|V\\\|\_\{\\mathrm\{span\}\}\}\{6N\}:0\\leq k\\leq 6N\\right\\\}\.Then\|𝒩α\|=6​N\+1\|\\mathcal\{N\}\_\{\\alpha\}\|=6N\+1\. For everyα∈\[0,‖V‖span\]\\alpha\\in\[0,\\\|V\\\|\_\{\\mathrm\{span\}\}\], chooseα′∈𝒩α\\alpha^\{\\prime\}\\in\\mathcal\{N\}\_\{\\alpha\}with\|α′−α\|≤‖V‖span/\(6​N\)\|\\alpha^\{\\prime\}\-\\alpha\|\\leq\\\|V\\\|\_\{\\mathrm\{span\}\}/\(6N\)\. Moreover,

‖min⁡\{V∘,α\}−min⁡\{V∘,α′\}‖∞=sups′∈𝒮\|min⁡\{V∘​\(s′\),α\}−min⁡\{V∘​\(s′\),α′\}\|≤‖V‖span6​N,\\left\\\|\\min\\\{V^\{\\circ\},\\alpha\\\}\-\\min\\\{V^\{\\circ\},\\alpha^\{\\prime\}\\\}\\right\\\|\_\{\\infty\}=\\sup\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}\\left\|\\min\\\{V^\{\\circ\}\(s^\{\\prime\}\),\\alpha\\\}\-\\min\\\{V^\{\\circ\}\(s^\{\\prime\}\),\\alpha^\{\\prime\}\\\}\\right\|\\leq\\frac\{\\\|V\\\|\_\{\\mathrm\{span\}\}\}\{6N\},and hence

\|\(P^s,a0−Ps,a0\)​\(min⁡\{V∘,α\}−min⁡\{V∘,α′\}\)\|≤‖V‖span3​N\\left\|\(\\widehat\{P\}^\{0\}\_\{s,a\}\-P^\{0\}\_\{s,a\}\)\\left\(\\min\\\{V^\{\\circ\},\\alpha\\\}\-\\min\\\{V^\{\\circ\},\\alpha^\{\\prime\}\\\}\\right\)\\right\|\\leq\\frac\{\\\|V\\\|\_\{\\mathrm\{span\}\}\}\{3N\}and

supα∈\[0,‖V‖span\]\|\(P^s,a0−Ps,a0\)​min⁡\{V∘,α\}\|≤maxα∈𝒩α⁡\|\(P^s,a0−Ps,a0\)​min⁡\{V∘,α\}\|\+‖V‖span3​N\.\\sup\_\{\\alpha\\in\[0,\\\|V\\\|\_\{\\mathrm\{span\}\}\]\}\\left\|\(\\widehat\{P\}^\{0\}\_\{s,a\}\-P^\{0\}\_\{s,a\}\)\\min\\\{V^\{\\circ\},\\alpha\\\}\\right\|\\leq\\max\_\{\\alpha\\in\\mathcal\{N\}\_\{\\alpha\}\}\\left\|\(\\widehat\{P\}^\{0\}\_\{s,a\}\-P^\{0\}\_\{s,a\}\)\\min\\\{V^\{\\circ\},\\alpha\\\}\\right\|\+\\frac\{\\\|V\\\|\_\{\\mathrm\{span\}\}\}\{3N\}\.\(121\)Therefore, up to the interpolation error in \([121](https://arxiv.org/html/2608.06545#A4.E121)\), it remains to control the empirical process at the finitely many levels in𝒩α\\mathcal\{N\}\_\{\\alpha\}\.

##### Step 2: Uniform rowwise concentration\.

In this step, we apply Bernstein’s inequality on the clipping grid and obtain a bound that holds simultaneously for all state\-action pairs\. For a fixed grid levelα\\alpha, the vectormin⁡\{V∘,α\}\\min\\\{V^\{\\circ\},\\alpha\\\}is independent ofP^s,a0\\widehat\{P\}^\{0\}\_\{s,a\}and is bounded in\[0,‖V‖span\]\[0,\\\|V\\\|\_\{\\mathrm\{span\}\}\]\. Bernstein’s inequality shows that with probability at least1−δ′1\-\\delta^\{\\prime\},

\|\(P^s,a0−Ps,a0\)​min⁡\{V∘,α\}\|≤2​V​a​rPs,a0​\(min⁡\{V∘,α\}\)​log⁡\(2δ′\)N\+2​‖V‖span​log⁡\(2δ′\)3​N\.\\left\|\(\\widehat\{P\}^\{0\}\_\{s,a\}\-P^\{0\}\_\{s,a\}\)\\min\\\{V^\{\\circ\},\\alpha\\\}\\right\|\\leq\\sqrt\{\\frac\{2\\mathrm\{Var\}\_\{P^\{0\}\_\{s,a\}\}\(\\min\\\{V^\{\\circ\},\\alpha\\\}\)\\log\(\\frac\{2\}\{\\delta^\{\\prime\}\}\)\}\{N\}\}\+\\frac\{2\\\|V\\\|\_\{\\mathrm\{span\}\}\\log\(\\frac\{2\}\{\\delta^\{\\prime\}\}\)\}\{3N\}\.Since the scalar clipping mapx↦min⁡\{x,α\}x\\mapsto\\min\\\{x,\\alpha\\\}is 1\-Lipschitz, the independent\-copy representation of variance implies

VarPs,a0​\(min⁡\{V∘,α\}\)≤VarPs,a0​\(V∘\)=VarPs,a0​\(V\)\.\\mathrm\{Var\}\_\{P^\{0\}\_\{s,a\}\}\(\\min\\\{V^\{\\circ\},\\alpha\\\}\)\\leq\\mathrm\{Var\}\_\{P^\{0\}\_\{s,a\}\}\(V^\{\\circ\}\)=\\mathrm\{Var\}\_\{P^\{0\}\_\{s,a\}\}\(V\)\.Set

δ′≔δS​A​\(6​N\+1\)\.\\delta^\{\\prime\}\\coloneqq\\frac\{\\delta\}\{SA\(6N\+1\)\}\.A union bound over allα∈𝒩α\\alpha\\in\\mathcal\{N\}\_\{\\alpha\}and all state\-action pairs yields probability at least1−δ1\-\\delta\. Moreover,

log⁡\(2δ′\)=log⁡\(2​S​A​\(6​N\+1\)δ\)≤ι\.\\log\\left\(\\frac\{2\}\{\\delta^\{\\prime\}\}\\right\)=\\log\\left\(\\frac\{2SA\(6N\+1\)\}\{\\delta\}\\right\)\\leq\\iota\.Consequently, on this event, simultaneously for every\(s,a\)\(s,a\),

maxα∈𝒩α⁡\|\(P^s,a0−Ps,a0\)​min⁡\{V∘,α\}\|≤2​V​a​rPs,a0​\(V\)⋅ιN\+2​‖V‖span⋅ι3​N\.\\max\_\{\\alpha\\in\\mathcal\{N\}\_\{\\alpha\}\}\\left\|\(\\widehat\{P\}^\{0\}\_\{s,a\}\-P^\{0\}\_\{s,a\}\)\\min\\\{V^\{\\circ\},\\alpha\\\}\\right\|\\leq\\sqrt\{\\frac\{2\\mathrm\{Var\}\_\{P^\{0\}\_\{s,a\}\}\(V\)\\cdot\\iota\}\{N\}\}\+\\frac\{2\\\|V\\\|\_\{\\mathrm\{span\}\}\\cdot\\iota\}\{3N\}\.Combining this with \([121](https://arxiv.org/html/2608.06545#A4.E121)\) results in

supα∈\[0,‖V‖span\]\|\(P^s,a0−Ps,a0\)​min⁡\{V∘,α\}\|≤2​V​a​rPs,a0​\(V\)⋅ιN\+‖V‖span⋅ιN\.\\sup\_\{\\alpha\\in\[0,\\\|V\\\|\_\{\\mathrm\{span\}\}\]\}\\left\|\(\\widehat\{P\}^\{0\}\_\{s,a\}\-P^\{0\}\_\{s,a\}\)\\min\\\{V^\{\\circ\},\\alpha\\\}\\right\|\\leq\\sqrt\{\\frac\{2\\mathrm\{Var\}\_\{P^\{0\}\_\{s,a\}\}\(V\)\\cdot\\iota\}\{N\}\}\+\\frac\{\\\|V\\\|\_\{\\mathrm\{span\}\}\\cdot\\iota\}\{N\}\.Here we usedι≥1\\iota\\geq 1to combine the two linear terms\. Thus, simultaneously for all\(s,a\)\(s,a\),

\|P^s,aV​V−Ps,aV​V\|≤2​V​a​rPs,a0​\(V\)⋅ιN\+‖V‖span⋅ιN\.\\left\|\\widehat\{P\}\_\{s,a\}^\{V\}V\-P\_\{s,a\}^\{V\}V\\right\|\\leq\\sqrt\{\\frac\{2\\mathrm\{Var\}\_\{P^\{0\}\_\{s,a\}\}\(V\)\\cdot\\iota\}\{N\}\}\+\\frac\{\\\|V\\\|\_\{\\mathrm\{span\}\}\\cdot\\iota\}\{N\}\.\(122\)We have therefore established the desired robust\-kernel concentration bound simultaneously for every state\-action pair\.

##### Step 3: Policy averaging and the resolvent bound\.

In this final step, we average the rowwise estimate over an arbitrary policy, convert the averaged conditional variances intoVarP0,π​\(V\)\\mathrm\{Var\}\_\{P^\{0,\\pi\}\}\(V\), and then pass to the normalized\-resolvent form\. Because \([122](https://arxiv.org/html/2608.06545#A4.E122)\) holds simultaneously for all\(s,a\)\(s,a\), we may now fix any policyπ\\pi\. By the definitions ofPπ,VP^\{\\pi,V\}andP^π,V\\widehat\{P\}^\{\\pi,V\}, for every statess,

\|\(P^π,V−Pπ,V\)​V\|​\(s\)≤∑a∈𝒜π​\(a∣s\)​\|P^s,aV​V−Ps,aV​V\|\.\\left\|\\left\(\\widehat\{P\}^\{\\pi,V\}\-P^\{\\pi,V\}\\right\)V\\right\|\(s\)\\leq\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\(a\\mid s\)\\left\|\\widehat\{P\}\_\{s,a\}^\{V\}V\-P\_\{s,a\}^\{V\}V\\right\|\.Combining \([122](https://arxiv.org/html/2608.06545#A4.E122)\) with Jensen’s inequality leads to

\|\(P^π,V−Pπ,V\)​V\|​\(s\)≤2​∑a∈𝒜π​\(a∣s\)​VarPs,a0​\(V\)⋅ιN\+‖V‖span⋅ιN\.\\left\|\\left\(\\widehat\{P\}^\{\\pi,V\}\-P^\{\\pi,V\}\\right\)V\\right\|\(s\)\\leq\\sqrt\{\\frac\{2\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\(a\\mid s\)\\mathrm\{Var\}\_\{P^\{0\}\_\{s,a\}\}\(V\)\\cdot\\iota\}\{N\}\}\+\\frac\{\\\|V\\\|\_\{\\mathrm\{span\}\}\\cdot\\iota\}\{N\}\.The law of total variance implies

VarP0,π​\(V\)​\(s\)\\displaystyle\\mathrm\{Var\}\_\{P^\{0,\\pi\}\}\(V\)\(s\)=∑a∈𝒜π​\(a∣s\)​VarPs,a0​\(V\)\+∑a∈𝒜π​\(a∣s\)​\(Ps,a0​V−P0,π​V​\(s\)\)2\\displaystyle=\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\(a\\mid s\)\\mathrm\{Var\}\_\{P^\{0\}\_\{s,a\}\}\(V\)\+\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\(a\\mid s\)\\left\(P^\{0\}\_\{s,a\}V\-P^\{0,\\pi\}V\(s\)\\right\)^\{2\}≥∑a∈𝒜π​\(a∣s\)​VarPs,a0​\(V\)\.\\displaystyle\\geq\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\(a\\mid s\)\\mathrm\{Var\}\_\{P^\{0\}\_\{s,a\}\}\(V\)\.Substituting this inequality into the preceding policy\-averaged estimate proves \([88](https://arxiv.org/html/2608.06545#A4.E88)\)\.

For any stochastic matrixLL, the matrix\(I−γ​L\)−1\(I\-\\gamma L\)^\{\-1\}is nonnegative and𝒢L​𝟏S=𝟏S\\mathcal\{G\}\_\{L\}\\bm\{1\}\_\{S\}=\\bm\{1\}\_\{S\}\. Multiplying \([88](https://arxiv.org/html/2608.06545#A4.E88)\) by the nonnegative matrix\(1−γ\)​\(I−γ​L\)−1\(1\-\\gamma\)\(I\-\\gamma L\)^\{\-1\}and using \([87](https://arxiv.org/html/2608.06545#A4.E87)\) therefore proves \([89](https://arxiv.org/html/2608.06545#A4.E89)\), completing the proof\.

#### D\.5\.6Proof of Lemma[16](https://arxiv.org/html/2608.06545#Thmlemma16)

Consider a fixed vectorggthat is independent of the empirical transition kernels\. Setg∘≔g−mins⁡g​\(s\)⋅𝟏Sg^\{\\circ\}\\coloneqq g\-\\min\_\{s\}g\(s\)\\cdot\\bm\{1\}\_\{S\}\. Then\(P^0,π−P0,π\)​g=\(P^0,π−P0,π\)​g∘\(\\widehat\{P\}^\{0,\\pi\}\-P^\{0,\\pi\}\)g=\(\\widehat\{P\}^\{0,\\pi\}\-P^\{0,\\pi\}\)g^\{\\circ\}andVarP0,π​\(g\)=VarP0,π​\(g∘\)\\mathrm\{Var\}\_\{P^\{0,\\pi\}\}\(g\)=\\mathrm\{Var\}\_\{P^\{0,\\pi\}\}\(g^\{\\circ\}\)\. Also,0≤g∘≤‖g‖span⋅𝟏S0\\leq g^\{\\circ\}\\leq\\\|g\\\|\_\{\\mathrm\{span\}\}\\cdot\\bm\{1\}\_\{S\}\.

Fix a state\-action pair\(s,a\)\(s,a\)\. Bernstein’s inequality applied to the fixed bounded vectorg∘g^\{\\circ\}shows that, with probability at least1−δ′1\-\\delta^\{\\prime\},

\|\(P^s,a0−Ps,a0\)​g\|=\|\(P^s,a0−Ps,a0\)​g∘\|≤2​V​a​rPs,a0​\(g\)​log⁡\(2δ′\)N\+2​‖g‖span​log⁡\(2δ′\)3​N\.\\left\|\(\\widehat\{P\}^\{0\}\_\{s,a\}\-P^\{0\}\_\{s,a\}\)g\\right\|=\\left\|\(\\widehat\{P\}^\{0\}\_\{s,a\}\-P^\{0\}\_\{s,a\}\)g^\{\\circ\}\\right\|\\leq\\sqrt\{\\frac\{2\\mathrm\{Var\}\_\{P^\{0\}\_\{s,a\}\}\(g\)\\log\(\\frac\{2\}\{\\delta^\{\\prime\}\}\)\}\{N\}\}\+\\frac\{2\\\|g\\\|\_\{\\mathrm\{span\}\}\\log\(\\frac\{2\}\{\\delta^\{\\prime\}\}\)\}\{3N\}\.Setδ′=δ/\(S​A\)\\delta^\{\\prime\}=\\delta/\(SA\)and take a union bound over all state\-action pairs\. Then apply policy\-averaging and total\-variance arguments similar to those used in the proof of Lemma[15](https://arxiv.org/html/2608.06545#Thmlemma15)\. With probability at least1−δ1\-\\delta, the following bound holds simultaneously for every policyπ\\pi:

\|\(P^0,π−P0,π\)​g\|≤C​VarP0,π​\(g\)⋅ιN\+C​‖g‖span⋅ιN⋅𝟏S\.\\left\|\\left\(\\widehat\{P\}^\{0,\\pi\}\-P^\{0,\\pi\}\\right\)g\\right\|\\leq C\\sqrt\{\\frac\{\\mathrm\{Var\}\_\{P^\{0,\\pi\}\}\(g\)\\cdot\\iota\}\{N\}\}\+C\\frac\{\\\|g\\\|\_\{\\mathrm\{span\}\}\\cdot\\iota\}\{N\}\\cdot\\bm\{1\}\_\{S\}\.Hereι\\iotadominateslog⁡\(2​S​A/δ\)\\log\(2SA/\\delta\)\. This proves \([90](https://arxiv.org/html/2608.06545#A4.E90)\)\. For any stochastic matrixLL, the matrix\(I−γ​L\)−1\(I\-\\gamma L\)^\{\-1\}is nonnegative and𝒢L​𝟏S=𝟏S\\mathcal\{G\}\_\{L\}\\bm\{1\}\_\{S\}=\\bm\{1\}\_\{S\}\. Multiplying \([90](https://arxiv.org/html/2608.06545#A4.E90)\) by\(1−γ\)​\(I−γ​L\)−1\(1\-\\gamma\)\(I\-\\gamma L\)^\{\-1\}and using \([87](https://arxiv.org/html/2608.06545#A4.E87)\) proves \([91](https://arxiv.org/html/2608.06545#A4.E91)\)\.

#### D\.5\.7Proof of Lemma[17](https://arxiv.org/html/2608.06545#Thmlemma17)

Recall the fixed\-policy aliasesUUandU^\\widehat\{U\}from Appendix[D\.4\.1](https://arxiv.org/html/2608.06545#A4.SS4.SSS1)\. For a generic value vectorVV, letPπγ⋆,VP^\{\\pi\_\{\\gamma\}^\{\\star\},V\}andP^πγ⋆,V\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},V\}denote minimizers in the true and empirical robust Bellman operators for the fixed policyπγ⋆\\pi\_\{\\gamma\}^\{\\star\}atVV, respectively\. The true and empirical robust Bellman equations are

U=rπγ⋆\+γ​Pπγ⋆,U​U,U^=rπγ⋆\+γ​P^πγ⋆,U^​U^\.U=r^\{\\pi\_\{\\gamma\}^\{\\star\}\}\+\\gamma P^\{\\pi\_\{\\gamma\}^\{\\star\},U\}U,\\qquad\\widehat\{U\}=r^\{\\pi\_\{\\gamma\}^\{\\star\}\}\+\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\\widehat\{U\}\.
##### Step 1: BoundingU^−U\\widehat\{U\}\-U\.

BecauseP^πγ⋆,U^\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}minimizes the empirical robust expectation ofU^\\widehat\{U\}andP^πγ⋆,U\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}is feasible,

P^πγ⋆,U^​U^≤P^πγ⋆,U​U^\.\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\\widehat\{U\}\\leq\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\widehat\{U\}\.Using this and comparing the two Bellman equations yields

U^−U\\displaystyle\\widehat\{U\}\-U=γ​P^πγ⋆,U^​U^−γ​Pπγ⋆,U​U\\displaystyle=\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\\widehat\{U\}\-\\gamma P^\{\\pi\_\{\\gamma\}^\{\\star\},U\}U≤γ​P^πγ⋆,U​U^−γ​Pπγ⋆,U​U\\displaystyle\\leq\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\widehat\{U\}\-\\gamma P^\{\\pi\_\{\\gamma\}^\{\\star\},U\}U≤γ​P^πγ⋆,U​\(U^−U\)\+γ​\(P^πγ⋆,U−Pπγ⋆,U\)​U\.\\displaystyle\\leq\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\left\(\\widehat\{U\}\-U\\right\)\+\\gamma\\left\(\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\-P^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)U\.The inverse\(I−γ​P^πγ⋆,U\)−1\(I\-\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\)^\{\-1\}is nonnegative\. Hence, after rearranging, multiplying by this inverse preserves the coordinatewise inequality and yields

U^−U≤γ​\(I−γ​P^πγ⋆,U\)−1​\(P^πγ⋆,U−Pπγ⋆,U\)​U\.\\widehat\{U\}\-U\\leq\\gamma\\left\(I\-\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)^\{\-1\}\\left\(\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\-P^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)U\.\(123\)

##### Step 2: BoundingU−U^U\-\\widehat\{U\}\.

We have

U−U^\\displaystyle U\-\\widehat\{U\}=γ​P^πγ⋆,U^​\(U−U^\)\+γ​\(Pπγ⋆,U−P^πγ⋆,U^\)​U\.\\displaystyle=\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\\left\(U\-\\widehat\{U\}\\right\)\+\\gamma\\left\(P^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\-\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\\right\)U\.SinceP^πγ⋆,U\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}minimizes the empirical robust expectation ofUU,P^πγ⋆,U​U≤P^πγ⋆,U^​U\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}U\\leq\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}U\. Hence

\(Pπγ⋆,U−P^πγ⋆,U^\)​U≤\(Pπγ⋆,U−P^πγ⋆,U\)​U\.\\left\(P^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\-\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\\right\)U\\leq\\left\(P^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\-\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)U\.The inverse\(I−γ​P^πγ⋆,U^\)−1\(I\-\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\)^\{\-1\}is also nonnegative\. Repeating the order\-preserving rearrangement used for \([123](https://arxiv.org/html/2608.06545#A4.E123)\) yields

U−U^\\displaystyle U\-\\widehat\{U\}≤γ​\(I−γ​P^πγ⋆,U^\)−1​\(Pπγ⋆,U−P^πγ⋆,U\)​U\.\\displaystyle\\leq\\gamma\\left\(I\-\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\\right\)^\{\-1\}\\left\(P^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\-\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)U\.

##### Step 3: Combining the two one\-sided bounds\.

Combining the two directions results in

∥U^−U∥∞≤γmax\{\\displaystyle\\left\\\|\\widehat\{U\}\-U\\right\\\|\_\{\\infty\}\\leq\\gamma\\max\\Bigg\\\{‖\(I−γ​P^πγ⋆,U\)−1​\(P^πγ⋆,U−Pπγ⋆,U\)​U‖∞,\\displaystyle\\left\\\|\\left\(I\-\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)^\{\-1\}\\left\(\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\-P^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)U\\right\\\|\_\{\\infty\},∥\(I−γP^πγ⋆,U^\)−1\(P^πγ⋆,U−Pπγ⋆,U\)U∥∞\}\.\\displaystyle\\left\\\|\\left\(I\-\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\\right\)^\{\-1\}\\left\(\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\-P^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)U\\right\\\|\_\{\\infty\}\\Bigg\\\}\.Since each resolvent is nonnegative, it satisfies\|A​x\|≤A​\|x\|\|Ax\|\\leq A\|x\|\. Together withγ≤1\\gamma\\leq 1, this inequality implies

∥U^−U∥∞≤max\{\\displaystyle\\big\\\|\\widehat\{U\}\-U\\big\\\|\_\{\\infty\}\\leq\\max\\Bigg\\\{‖\(I−γ​P^πγ⋆,U\)−1​\|\(P^πγ⋆,U−Pπγ⋆,U\)​U\|‖∞,\\displaystyle\\left\\\|\\left\(I\-\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)^\{\-1\}\\left\|\\left\(\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\-P^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)U\\right\|\\right\\\|\_\{\\infty\},∥\(I−γP^πγ⋆,U^\)−1\|\(P^πγ⋆,U−Pπγ⋆,U\)U\|∥∞\}\.\\displaystyle\\left\\\|\\left\(I\-\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\\right\)^\{\-1\}\\left\|\\left\(\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\-P^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)U\\right\|\\right\\\|\_\{\\infty\}\\Bigg\\\}\.The last display is \([92](https://arxiv.org/html/2608.06545#A4.E92)\), completing the proof\.

#### D\.5\.8Proof of Lemma[18](https://arxiv.org/html/2608.06545#Thmlemma18)

The proof has three steps\. We first prove a general resolvent\-variance inequality\. We then apply it twice to bound two different terms and combine the resulting bounds to obtain the desired estimate\.

##### Step 1: A general resolvent\-variance bound\.

Fix an initial statess, letese\_\{s\}be the corresponding standard basis vector, and set

νs≔\(1−γ\)​es⊤​\(I−γ​P\)−1\.\\nu\_\{s\}\\coloneqq\(1\-\\gamma\)e\_\{s\}^\{\\top\}\(I\-\\gamma P\)^\{\-1\}\.\(124\)Equivalently,

νs=\(1−γ\)​∑t=0∞γt​es⊤​Pt\.\\nu\_\{s\}=\(1\-\\gamma\)\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}e\_\{s\}^\{\\top\}P^\{t\}\.The coefficients\(1−γ\)​γt\(1\-\\gamma\)\\gamma^\{t\}sum to one, soνs\\nu\_\{s\}is a probability distribution\. For every vectorff, we claim that

νs​VarP​\(f\)≤1−γγ​‖f‖span2\+2γ​‖f‖span​νs​\[f−P​f\]\+\.\\nu\_\{s\}\\mathrm\{Var\}\_\{P\}\(f\)\\leq\\frac\{1\-\\gamma\}\{\\gamma\}\\\|f\\\|\_\{\\mathrm\{span\}\}^\{2\}\+\\frac\{2\}\{\\gamma\}\\\|f\\\|\_\{\\mathrm\{span\}\}\\,\\nu\_\{s\}\[f\-Pf\]\_\{\+\}\.\(125\)We now prove \([125](https://arxiv.org/html/2608.06545#A4.E125)\)\. SinceVarP​\(f\)\\mathrm\{Var\}\_\{P\}\(f\)andf−P​ff\-Pfare translation invariant, we may shiftffso that0≤f≤‖f‖span⋅𝟏S0\\leq f\\leq\\\|f\\\|\_\{\\mathrm\{span\}\}\\cdot\\bm\{1\}\_\{S\}\. Rearranging \([124](https://arxiv.org/html/2608.06545#A4.E124)\) yields

νs​P=1γ​νs−1−γγ​es⊤\.\\nu\_\{s\}P=\\frac\{1\}\{\\gamma\}\\nu\_\{s\}\-\\frac\{1\-\\gamma\}\{\\gamma\}e\_\{s\}^\{\\top\}\.Therefore,

νs​VarP​\(f\)\\displaystyle\\nu\_\{s\}\\mathrm\{Var\}\_\{P\}\(f\)=νs​P​\(f∘f\)−νs​\(P​f∘P​f\)\\displaystyle=\\nu\_\{s\}P\(f\\circ f\)\-\\nu\_\{s\}\(Pf\\circ Pf\)≤1γ​νs​\(f∘f\)−νs​\(P​f∘P​f\)\\displaystyle\\leq\\frac\{1\}\{\\gamma\}\\nu\_\{s\}\(f\\circ f\)\-\\nu\_\{s\}\(Pf\\circ Pf\)=1γ​νs​\(f∘f−γ​P​f∘P​f\),\\displaystyle=\\frac\{1\}\{\\gamma\}\\nu\_\{s\}\\\!\\left\(f\\circ f\-\\gamma Pf\\circ Pf\\right\),where the inequality drops the nonpositive term−\(1−γ\)​es⊤​\(f∘f\)/γ\-\(1\-\\gamma\)e\_\{s\}^\{\\top\}\(f\\circ f\)/\\gamma\. SincePPis stochastic,0≤P​f≤‖f‖span⋅𝟏S0\\leq Pf\\leq\\\|f\\\|\_\{\\mathrm\{span\}\}\\cdot\\bm\{1\}\_\{S\}\. Hence, coordinatewise,

f​\(s\)2−\(P​f\)​\(s\)2=\(f​\(s\)−\(P​f\)​\(s\)\)​\(f​\(s\)\+\(P​f\)​\(s\)\)≤2​‖f‖span​\[f​\(s\)−\(P​f\)​\(s\)\]\+\.f\(s\)^\{2\}\-\(Pf\)\(s\)^\{2\}=\\bigl\(f\(s\)\-\(Pf\)\(s\)\\bigr\)\\bigl\(f\(s\)\+\(Pf\)\(s\)\\bigr\)\\leq 2\\\|f\\\|\_\{\\mathrm\{span\}\}\[f\(s\)\-\(Pf\)\(s\)\]\_\{\+\}\.Here\[⋅\]\+\[\\cdot\]\_\{\+\}is applied coordinatewise\. Consequently,

f∘f−γ​P​f∘P​f\\displaystyle f\\circ f\-\\gamma Pf\\circ Pf=\(f∘f−P​f∘P​f\)\+\(1−γ\)​P​f∘P​f\\displaystyle=\\bigl\(f\\circ f\-Pf\\circ Pf\\bigr\)\+\(1\-\\gamma\)Pf\\circ Pf≤2​‖f‖span​\[f−P​f\]\+\+\(1−γ\)​‖f‖span2⋅𝟏S\.\\displaystyle\\leq 2\\\|f\\\|\_\{\\mathrm\{span\}\}\[f\-Pf\]\_\{\+\}\+\(1\-\\gamma\)\\\|f\\\|\_\{\\mathrm\{span\}\}^\{2\}\\cdot\\bm\{1\}\_\{S\}\.Combining this coordinatewise inequality with the preceding bound and usingνs​𝟏S=1\\nu\_\{s\}\\bm\{1\}\_\{S\}=1, we obtain

νs​VarP​\(f\)\\displaystyle\\nu\_\{s\}\\mathrm\{Var\}\_\{P\}\(f\)≤1γ​νs​\(f∘f−γ​P​f∘P​f\)\\displaystyle\\leq\\frac\{1\}\{\\gamma\}\\nu\_\{s\}\\left\(f\\circ f\-\\gamma Pf\\circ Pf\\right\)≤2γ​‖f‖span​νs​\[f−P​f\]\+\+1−γγ​‖f‖span2​νs​𝟏S\\displaystyle\\leq\\frac\{2\}\{\\gamma\}\\\|f\\\|\_\{\\mathrm\{span\}\}\\,\\nu\_\{s\}\[f\-Pf\]\_\{\+\}\+\\frac\{1\-\\gamma\}\{\\gamma\}\\\|f\\\|\_\{\\mathrm\{span\}\}^\{2\}\\nu\_\{s\}\\bm\{1\}\_\{S\}=2γ​‖f‖span​νs​\[f−P​f\]\+\+1−γγ​‖f‖span2\.\\displaystyle=\\frac\{2\}\{\\gamma\}\\\|f\\\|\_\{\\mathrm\{span\}\}\\,\\nu\_\{s\}\[f\-Pf\]\_\{\+\}\+\\frac\{1\-\\gamma\}\{\\gamma\}\\\|f\\\|\_\{\\mathrm\{span\}\}^\{2\}\.This is \([125](https://arxiv.org/html/2608.06545#A4.E125)\)\.

##### Step 2: Boundingνs​VarP​\(h¯\)\\nu\_\{s\}\\mathrm\{Var\}\_\{P\}\(\\bar\{h\}\)\.

Let

g≔ρ¯\+h¯−rπ−P0,π​h¯\.g\\coloneqq\\bar\{\\rho\}\+\\bar\{h\}\-r^\{\\pi\}\-P^\{0,\\pi\}\\bar\{h\}\.By \([6](https://arxiv.org/html/2608.06545#S4.E6)\),g≥0g\\geq 0\. We also haveVarP​\(h¯\)=VarP​\(−h¯\)\\mathrm\{Var\}\_\{P\}\(\\bar\{h\}\)=\\mathrm\{Var\}\_\{P\}\(\-\\bar\{h\}\)and

P​h¯−h¯=ρ¯−rπ−g\+\(P−P0,π\)​h¯\.P\\bar\{h\}\-\\bar\{h\}=\\bar\{\\rho\}\-r^\{\\pi\}\-g\+\(P\-P^\{0,\\pi\}\)\\bar\{h\}\.\(126\)This identity, the bounds in \([103](https://arxiv.org/html/2608.06545#A4.E103)\), and the inequalitiesrπ≥0r^\{\\pi\}\\geq 0andg≥0g\\geq 0imply

νs​\[P​h¯−h¯\]\+≤ρ¯\+C​σ​Hanc\+νs​\|ξh\|\.\\nu\_\{s\}\[P\\bar\{h\}\-\\bar\{h\}\]\_\{\+\}\\leq\\bar\{\\rho\}\+C\\sigma H\_\{\\mathrm\{anc\}\}\+\\nu\_\{s\}\|\\xi\_\{h\}\|\.Equations \([102](https://arxiv.org/html/2608.06545#A4.E102)\) and \([101](https://arxiv.org/html/2608.06545#A4.E101)\) implyρ¯≤β\+1\\bar\{\\rho\}\\leq\\beta\+1\. By \([124](https://arxiv.org/html/2608.06545#A4.E124)\) and \([87](https://arxiv.org/html/2608.06545#A4.E87)\),νs​\|ξh\|≤‖𝒢P​\|ξh\|‖∞\\nu\_\{s\}\|\\xi\_\{h\}\|\\leq\\\|\\mathcal\{G\}\_\{P\}\|\\xi\_\{h\}\|\\\|\_\{\\infty\}\. Applying \([125](https://arxiv.org/html/2608.06545#A4.E125)\) withf=−h¯f=\-\\bar\{h\}and usingγ≥1/2\\gamma\\geq 1/2now yields

νs​VarP​\(h¯\)\\displaystyle\\nu\_\{s\}\\mathrm\{Var\}\_\{P\}\(\\bar\{h\}\)≤1−γγ​Hanc2\+2γ​Hanc​νs​\[P​h¯−h¯\]\+\\displaystyle\\leq\\frac\{1\-\\gamma\}\{\\gamma\}H\_\{\\mathrm\{anc\}\}^\{2\}\+\\frac\{2\}\{\\gamma\}H\_\{\\mathrm\{anc\}\}\\,\\nu\_\{s\}\[P\\bar\{h\}\-\\bar\{h\}\]\_\{\+\}≤C​\(\(1−γ\)​Hanc2\+Hanc​β\+Hanc\+σ​Hanc2\+Hanc​‖𝒢P​\|ξh\|‖∞\)\\displaystyle\\leq C\\left\(\(1\-\\gamma\)H\_\{\\mathrm\{anc\}\}^\{2\}\+H\_\{\\mathrm\{anc\}\}\\beta\+H\_\{\\mathrm\{anc\}\}\+\\sigma H\_\{\\mathrm\{anc\}\}^\{2\}\+H\_\{\\mathrm\{anc\}\}\\\|\\mathcal\{G\}\_\{P\}\|\\xi\_\{h\}\|\\\|\_\{\\infty\}\\right\)≤C​\(B​\(R,β\)\+R​‖𝒢P​\|ξh\|‖∞\)\.\\displaystyle\\leq C\\left\(B\(R,\\beta\)\+R\\\|\\mathcal\{G\}\_\{P\}\|\\xi\_\{h\}\|\\\|\_\{\\infty\}\\right\)\.\(127\)For the last inequality, we usedHanc≤RH\_\{\\mathrm\{anc\}\}\\leq Rand matched the first four terms with the four terms inB​\(R,β\)B\(R,\\beta\)from \([63](https://arxiv.org/html/2608.06545#A4.E63)\)\.

##### Step 3: Boundingνs​VarP​\(V−h¯\)\\nu\_\{s\}\\mathrm\{Var\}\_\{P\}\(V\-\\bar\{h\}\)and concluding\.

Setu≔V−h¯u\\coloneqq V\-\\bar\{h\}\. Then

‖u‖span≤‖V‖span\+‖h¯‖span≤2​R\.\\\|u\\\|\_\{\\mathrm\{span\}\}\\leq\\\|V\\\|\_\{\\mathrm\{span\}\}\+\\\|\\bar\{h\}\\\|\_\{\\mathrm\{span\}\}\\leq 2R\.Moreover, combiningV=rπ\+γ​P​V\+bV=r^\{\\pi\}\+\\gamma PV\+bfrom \([101](https://arxiv.org/html/2608.06545#A4.E101)\) with \([126](https://arxiv.org/html/2608.06545#A4.E126)\) results in

u−P​u\\displaystyle u\-Pu=V−P​V−h¯\+P​h¯\\displaystyle=V\-PV\-\\bar\{h\}\+P\\bar\{h\}=ρ¯−\(1−γ\)​mins⁡V​\(s\)−g−\(1−γ\)​P​\(V−mins⁡V⋅𝟏S\)\+b\+\(P−P0,π\)​h¯\.\\displaystyle=\\bar\{\\rho\}\-\(1\-\\gamma\)\\min\_\{s\}V\(s\)\-g\-\(1\-\\gamma\)P\\\!\\left\(V\-\\min\_\{s\}V\\cdot\\bm\{1\}\_\{S\}\\right\)\+b\+\(P\-P^\{0,\\pi\}\)\\bar\{h\}\.Dropping the nonpositive terms−g\-gand−\(1−γ\)​P​\(V−mins⁡V⋅𝟏S\)\-\(1\-\\gamma\)P\(V\-\\min\_\{s\}V\\cdot\\bm\{1\}\_\{S\}\), the assumed bounds in \([103](https://arxiv.org/html/2608.06545#A4.E103)\) and the definition ofβ\\betain \([102](https://arxiv.org/html/2608.06545#A4.E102)\) then imply

νs​\[u−P​u\]\+≤β\+C​σ​Hanc\+‖𝒢P​\|ξV\|‖∞\+‖𝒢P​\|ξh\|‖∞\.\\nu\_\{s\}\[u\-Pu\]\_\{\+\}\\leq\\beta\+C\\sigma H\_\{\\mathrm\{anc\}\}\+\\\|\\mathcal\{G\}\_\{P\}\|\\xi\_\{V\}\|\\\|\_\{\\infty\}\+\\\|\\mathcal\{G\}\_\{P\}\|\\xi\_\{h\}\|\\\|\_\{\\infty\}\.A second application of \([125](https://arxiv.org/html/2608.06545#A4.E125)\), now withf=uf=u, together withγ≥1/2\\gamma\\geq 1/2and‖u‖span≤2​R\\\|u\\\|\_\{\\mathrm\{span\}\}\\leq 2R, yields

νs​VarP​\(u\)\\displaystyle\\nu\_\{s\}\\mathrm\{Var\}\_\{P\}\(u\)≤C​\(1−γ\)​R2\+C​R​νs​\[u−P​u\]\+\\displaystyle\\leq C\(1\-\\gamma\)R^\{2\}\+CR\\,\\nu\_\{s\}\[u\-Pu\]\_\{\+\}≤C​\(\(1−γ\)​R2\+R​β\+σ​R​Hanc\+R​‖𝒢P​\|ξV\|‖∞\+R​‖𝒢P​\|ξh\|‖∞\)\.\\displaystyle\\leq C\\left\(\(1\-\\gamma\)R^\{2\}\+R\\beta\+\\sigma RH\_\{\\mathrm\{anc\}\}\+R\\\|\\mathcal\{G\}\_\{P\}\|\\xi\_\{V\}\|\\\|\_\{\\infty\}\+R\\\|\\mathcal\{G\}\_\{P\}\|\\xi\_\{h\}\|\\\|\_\{\\infty\}\\right\)\.UsingHanc≤RH\_\{\\mathrm\{anc\}\}\\leq Rand the definition ofB​\(R,β\)B\(R,\\beta\)in \([63](https://arxiv.org/html/2608.06545#A4.E63)\), we obtain

νs​VarP​\(u\)≤C​\(B​\(R,β\)\+R​‖𝒢P​\|ξV\|‖∞\+R​‖𝒢P​\|ξh\|‖∞\)\.\\nu\_\{s\}\\mathrm\{Var\}\_\{P\}\(u\)\\leq C\\left\(B\(R,\\beta\)\+R\\\|\\mathcal\{G\}\_\{P\}\|\\xi\_\{V\}\|\\\|\_\{\\infty\}\+R\\\|\\mathcal\{G\}\_\{P\}\|\\xi\_\{h\}\|\\\|\_\{\\infty\}\\right\)\.\(128\)
Since

VarP​\(V\)=VarP​\(h¯\+u\)≤2​V​a​rP​\(h¯\)\+2​V​a​rP​\(u\),\\mathrm\{Var\}\_\{P\}\(V\)=\\mathrm\{Var\}\_\{P\}\(\\bar\{h\}\+u\)\\leq 2\\mathrm\{Var\}\_\{P\}\(\\bar\{h\}\)\+2\\mathrm\{Var\}\_\{P\}\(u\),\([127](https://arxiv.org/html/2608.06545#A4.E127)\) and \([128](https://arxiv.org/html/2608.06545#A4.E128)\) imply

νs​VarP​\(V\)≤C​\(B​\(R,β\)\+R​‖𝒢P​\|ξV\|‖∞\+R​‖𝒢P​\|ξh\|‖∞\)\.\\nu\_\{s\}\\mathrm\{Var\}\_\{P\}\(V\)\\leq C\\left\(B\(R,\\beta\)\+R\\\|\\mathcal\{G\}\_\{P\}\|\\xi\_\{V\}\|\\\|\_\{\\infty\}\+R\\\|\\mathcal\{G\}\_\{P\}\|\\xi\_\{h\}\|\\\|\_\{\\infty\}\\right\)\.Finally, Jensen’s inequality for the probability measureνs\\nu\_\{s\}yields

es⊤​\(I−γ​P\)−1​VarP​\(V\)=11−γ​νs​VarP​\(V\)≤11−γ​νs​VarP​\(V\)\.e\_\{s\}^\{\\top\}\(I\-\\gamma P\)^\{\-1\}\\sqrt\{\\mathrm\{Var\}\_\{P\}\(V\)\}=\\frac\{1\}\{1\-\\gamma\}\\nu\_\{s\}\\sqrt\{\\mathrm\{Var\}\_\{P\}\(V\)\}\\leq\\frac\{1\}\{1\-\\gamma\}\\sqrt\{\\nu\_\{s\}\\mathrm\{Var\}\_\{P\}\(V\)\}\.Sincesswas arbitrary, this proves the lemma\.

#### D\.5\.9Proof of Lemma[19](https://arxiv.org/html/2608.06545#Thmlemma19)

Recall the fixed\-policy aliases from Appendix[D\.4\.1](https://arxiv.org/html/2608.06545#A4.SS4.SSS1):

U≔Vγ⋆,σ=Vγπγ⋆,σ,U^≔V^γπγ⋆,σ\.U\\coloneqq V\_\{\\gamma\}^\{\\star,\\sigma\}=V\_\{\\gamma\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\sigma\},\\qquad\\widehat\{U\}\\coloneqq\\widehat\{V\}\_\{\\gamma\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\sigma\}\.The proof has three steps\. We first apply Lemma[18](https://arxiv.org/html/2608.06545#Thmlemma18)to reduce the bound onT2T\_\{2\}to two fixed residuals\. We then bound those residuals\. Finally, we substitute the residual bound into the reduction from the first step\.

##### Step 1: Reduction ofT2T\_\{2\}to fixed residuals\.

In this step, we verify the premises of Lemma[18](https://arxiv.org/html/2608.06545#Thmlemma18)and apply it to isolate the two fixed residuals that remain to be bounded\. For this application, set

π=πγ⋆,V=U,P=P^πγ⋆,U,rπ=rπγ⋆\.\\pi=\\pi\_\{\\gamma\}^\{\\star\},\\qquad V=U,\\qquad P=\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\},\\qquad r^\{\\pi\}=r^\{\\pi\_\{\\gamma\}^\{\\star\}\}\.The span premise in \([101](https://arxiv.org/html/2608.06545#A4.E101)\) follows from \([70](https://arxiv.org/html/2608.06545#A4.E70)\), and the associated anchor defect isβ=β⋆\\beta=\\beta\_\{\\star\}by \([58](https://arxiv.org/html/2608.06545#A4.E58)\)\. Define the value and anchor sampling residuals by

ξV=\(P^πγ⋆,U−Pπγ⋆,U\)​U,ξh=\(P^0,πγ⋆−P0,πγ⋆\)​h¯\.\\xi\_\{V\}=\\left\(\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\-P^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)U,\\qquad\\xi\_\{h\}=\\left\(\\widehat\{P\}^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\-P^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\\right\)\\bar\{h\}\.The true Bellman equation can then be written as

U=rπγ⋆\+γ​P^πγ⋆,U​U−γ​ξV\.U=r^\{\\pi\_\{\\gamma\}^\{\\star\}\}\+\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}U\-\\gamma\\xi\_\{V\}\.SinceP^πγ⋆,U\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}belongs to the robust uncertainty set centered atP^0,πγ⋆\\widehat\{P\}^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}, the definition of that set and‖h¯‖span≤Hanc\\\|\\bar\{h\}\\\|\_\{\\mathrm\{span\}\}\\leq H\_\{\\mathrm\{anc\}\}imply

\|\(P^πγ⋆,U−P0,πγ⋆\)​h¯\|≤C​σ​Hanc⋅𝟏S\+\|ξh\|\.\\left\|\\left\(\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\-P^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\\right\)\\bar\{h\}\\right\|\\leq C\\sigma H\_\{\\mathrm\{anc\}\}\\cdot\\bm\{1\}\_\{S\}\+\|\\xi\_\{h\}\|\.Becauseγ≤1\\gamma\\leq 1,\|b\|≤\|ξV\|\|b\|\\leq\|\\xi\_\{V\}\|forb=−γ​ξVb=\-\\gamma\\xi\_\{V\}\. Applying Lemma[18](https://arxiv.org/html/2608.06545#Thmlemma18)to the definition ofT2T\_\{2\}therefore yields

‖T2‖∞≤C​ιN​\(1−γ\)2​B⋆​\(R\)\+R​‖𝒢P^πγ⋆,U​\|ξV\|‖∞\+R​‖𝒢P^πγ⋆,U​\|ξh\|‖∞\.\\\|T\_\{2\}\\\|\_\{\\infty\}\\leq C\\sqrt\{\\frac\{\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\\sqrt\{B\_\{\\star\}\(R\)\+R\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\|\\xi\_\{V\}\|\\right\\\|\_\{\\infty\}\+R\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\|\\xi\_\{h\}\|\\right\\\|\_\{\\infty\}\}\.\(129\)

##### Step 2: Bounding the fixed residuals\.

In this step, we bound the two residuals isolated in Step 1\. Lemma[15](https://arxiv.org/html/2608.06545#Thmlemma15)applied to the fixed valueUUestablishes

‖𝒢P^πγ⋆,U​\|ξV\|‖∞≤C​ιN​‖𝒢P^πγ⋆,U​VarP0,πγ⋆​\(U\)‖∞\+C​R⋅ιN,\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\|\\xi\_\{V\}\|\\right\\\|\_\{\\infty\}\\leq C\\sqrt\{\\frac\{\\iota\}\{N\}\}\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\}\(U\)\}\\right\\\|\_\{\\infty\}\+C\\frac\{R\\cdot\\iota\}\{N\},\(130\)where we used‖U‖span≤R\\\|U\\\|\_\{\\mathrm\{span\}\}\\leq Rfrom \([70](https://arxiv.org/html/2608.06545#A4.E70)\)\. Separately, Lemma[16](https://arxiv.org/html/2608.06545#Thmlemma16)applied to the fixed anchor vectorh¯\\bar\{h\}provides

‖𝒢P^πγ⋆,U​\|ξh\|‖∞≤C​ιN​‖𝒢P^πγ⋆,U​VarP0,πγ⋆​\(h¯\)‖∞\+C​R⋅ιN,\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\|\\xi\_\{h\}\|\\right\\\|\_\{\\infty\}\\leq C\\sqrt\{\\frac\{\\iota\}\{N\}\}\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\}\(\\bar\{h\}\)\}\\right\\\|\_\{\\infty\}\+C\\frac\{R\\cdot\\iota\}\{N\},\(131\)where \([56](https://arxiv.org/html/2608.06545#A4.E56)\) andR≥HancR\\geq H\_\{\\mathrm\{anc\}\}give‖h¯‖span≤Hanc≤R\\\|\\bar\{h\}\\\|\_\{\\mathrm\{span\}\}\\leq H\_\{\\mathrm\{anc\}\}\\leq R\. We first transfer the two nominal standard deviations to the empirical robust kernel\. For each state, the robust\-set constraint implies

‖P^πγ⋆,U​\(s,⋅\)−P^0,πγ⋆​\(s,⋅\)‖TV\\displaystyle\\left\\\|\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\(s,\\cdot\)\-\\widehat\{P\}^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\(s,\\cdot\)\\right\\\|\_\{\\mathrm\{TV\}\}≤∑a∈𝒜πγ⋆​\(a∣s\)​‖P^s,aU−P^s,a0‖TV≤σ,\\displaystyle\\leq\\sum\_\{a\\in\\mathcal\{A\}\}\\pi\_\{\\gamma\}^\{\\star\}\(a\\mid s\)\\left\\\|\\widehat\{P\}\_\{s,a\}^\{U\}\-\\widehat\{P\}^\{0\}\_\{s,a\}\\right\\\|\_\{\\mathrm\{TV\}\}\\leq\\sigma,where the first inequality follows from the convexity of total variation\. The first transfer compares the empirical nominal kernel with the empirical robust kernel\. Forf∈\{U,h¯\}f\\in\\\{U,\\bar\{h\}\\\}, Lemma[26](https://arxiv.org/html/2608.06545#Thmlemma26)and‖f‖span≤R\\\|f\\\|\_\{\\mathrm\{span\}\}\\leq Rimply

VarP^0,πγ⋆​\(f\)≤VarP^πγ⋆,U​\(f\)\+C​R​σ⋅𝟏S\.\\sqrt\{\\mathrm\{Var\}\_\{\\widehat\{P\}^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\}\(f\)\}\\leq\\sqrt\{\\mathrm\{Var\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\(f\)\}\+CR\\sqrt\{\\sigma\}\\cdot\\bm\{1\}\_\{S\}\.The second transfer compares the true and empirical nominal kernels\. For the same two fixed vectors, Lemma[21](https://arxiv.org/html/2608.06545#Thmlemma21)provides

VarP0,πγ⋆​\(f\)≤VarP^0,πγ⋆​\(f\)\+C​R​ιN⋅𝟏S\.\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\}\(f\)\}\\leq\\sqrt\{\\mathrm\{Var\}\_\{\\widehat\{P\}^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\}\(f\)\}\+CR\\sqrt\{\\frac\{\\iota\}\{N\}\}\\cdot\\bm\{1\}\_\{S\}\.Combining the two transfers results in

VarP0,πγ⋆​\(f\)≤VarP^πγ⋆,U​\(f\)\+C​R​\(σ\+ιN\)⋅𝟏S,f∈\{U,h¯\}\.\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\}\(f\)\}\\leq\\sqrt\{\\mathrm\{Var\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\(f\)\}\+CR\\left\(\\sqrt\{\\sigma\}\+\\sqrt\{\\frac\{\\iota\}\{N\}\}\\right\)\\cdot\\bm\{1\}\_\{S\},\\qquad f\\in\\\{U,\\bar\{h\}\\\}\.Because𝒢P^πγ⋆,U\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}is nonnegative and maps𝟏S\\bm\{1\}\_\{S\}to itself, summing the two resulting resolvent bounds leads to

‖𝒢P^πγ⋆,U​VarP0,πγ⋆​\(U\)‖∞\+‖𝒢P^πγ⋆,U​VarP0,πγ⋆​\(h¯\)‖∞\\displaystyle\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\}\(U\)\}\\right\\\|\_\{\\infty\}\+\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\}\(\\bar\{h\}\)\}\\right\\\|\_\{\\infty\}≤‖𝒢P^πγ⋆,U​VarP^πγ⋆,U​\(U\)‖∞\+‖𝒢P^πγ⋆,U​VarP^πγ⋆,U​\(h¯\)‖∞\+C​R​\(σ\+ιN\)\.\\displaystyle\\quad\\leq\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\\sqrt\{\\mathrm\{Var\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\(U\)\}\\right\\\|\_\{\\infty\}\+\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\\sqrt\{\\mathrm\{Var\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\(\\bar\{h\}\)\}\\right\\\|\_\{\\infty\}\+CR\\left\(\\sqrt\{\\sigma\}\+\\sqrt\{\\frac\{\\iota\}\{N\}\}\\right\)\.\(132\)
Lemma[18](https://arxiv.org/html/2608.06545#Thmlemma18)bounds the first empirical robust\-kernel term on the right\-hand side of \([132](https://arxiv.org/html/2608.06545#A4.E132)\)\. For the second term, applying Jensen’s inequality to \([127](https://arxiv.org/html/2608.06545#A4.E127)\), exactly as in the final step of the proof of Lemma[18](https://arxiv.org/html/2608.06545#Thmlemma18), yields

‖𝒢P^πγ⋆,U​VarP^πγ⋆,U​\(h¯\)‖∞≤C​B⋆​\(R\)\+R​‖𝒢P^πγ⋆,U​\|ξh\|‖∞\.\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\\sqrt\{\\mathrm\{Var\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\(\\bar\{h\}\)\}\\right\\\|\_\{\\infty\}\\leq C\\sqrt\{B\_\{\\star\}\(R\)\+R\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\|\\xi\_\{h\}\|\\right\\\|\_\{\\infty\}\}\.\(133\)Consequently, the sum of the first two terms on the right\-hand side of \([132](https://arxiv.org/html/2608.06545#A4.E132)\) is at most

C​B⋆​\(R\)\+R​‖𝒢P^πγ⋆,U​\|ξV\|‖∞\+R​‖𝒢P^πγ⋆,U​\|ξh\|‖∞\.C\\sqrt\{B\_\{\\star\}\(R\)\+R\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\|\\xi\_\{V\}\|\\right\\\|\_\{\\infty\}\+R\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\|\\xi\_\{h\}\|\\right\\\|\_\{\\infty\}\}\.Moreover,

R​σ≤B⋆​\(R\),R​ιN≤R​1−γ≤B⋆​\(R\),R\\sqrt\{\\sigma\}\\leq\\sqrt\{B\_\{\\star\}\(R\)\},\\qquad R\\sqrt\{\\frac\{\\iota\}\{N\}\}\\leq R\\sqrt\{1\-\\gamma\}\\leq\\sqrt\{B\_\{\\star\}\(R\)\},where the second inequality uses the first condition in \([71](https://arxiv.org/html/2608.06545#A4.E71)\)\. Substituting these estimates into \([132](https://arxiv.org/html/2608.06545#A4.E132)\), we obtain

‖𝒢P^πγ⋆,U​VarP0,πγ⋆​\(U\)‖∞\+‖𝒢P^πγ⋆,U​VarP0,πγ⋆​\(h¯\)‖∞\\displaystyle\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\}\(U\)\}\\right\\\|\_\{\\infty\}\+\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\}\(\\bar\{h\}\)\}\\right\\\|\_\{\\infty\}≤C​B⋆​\(R\)\+R​‖𝒢P^πγ⋆,U​\|ξV\|‖∞\+R​‖𝒢P^πγ⋆,U​\|ξh\|‖∞\.\\displaystyle\\quad\\leq C\\sqrt\{B\_\{\\star\}\(R\)\+R\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\|\\xi\_\{V\}\|\\right\\\|\_\{\\infty\}\+R\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\|\\xi\_\{h\}\|\\right\\\|\_\{\\infty\}\}\.\(134\)
Adding \([130](https://arxiv.org/html/2608.06545#A4.E130)\) and \([131](https://arxiv.org/html/2608.06545#A4.E131)\) and then applying \([134](https://arxiv.org/html/2608.06545#A4.E134)\) results in

‖𝒢P^πγ⋆,U​\|ξV\|‖∞\+‖𝒢P^πγ⋆,U​\|ξh\|‖∞\\displaystyle\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\|\\xi\_\{V\}\|\\right\\\|\_\{\\infty\}\+\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\|\\xi\_\{h\}\|\\right\\\|\_\{\\infty\}≤C​ιN​B⋆​\(R\)\+R​‖𝒢P^πγ⋆,U​\|ξV\|‖∞\+R​‖𝒢P^πγ⋆,U​\|ξh\|‖∞\+C​R⋅ιN\.\\displaystyle\\quad\\leq C\\sqrt\{\\frac\{\\iota\}\{N\}\}\\sqrt\{B\_\{\\star\}\(R\)\+R\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\|\\xi\_\{V\}\|\\right\\\|\_\{\\infty\}\+R\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\|\\xi\_\{h\}\|\\right\\\|\_\{\\infty\}\}\+C\\frac\{R\\cdot\\iota\}\{N\}\.Usingx\+y≤x\+y\\sqrt\{x\+y\}\\leq\\sqrt\{x\}\+\\sqrt\{y\}followed by Young’s inequality,

C​R⋅ιN​‖𝒢P^πγ⋆,U​\|ξV\|‖∞\+‖𝒢P^πγ⋆,U​\|ξh\|‖∞\\displaystyle C\\sqrt\{\\frac\{R\\cdot\\iota\}\{N\}\}\\sqrt\{\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\|\\xi\_\{V\}\|\\right\\\|\_\{\\infty\}\+\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\|\\xi\_\{h\}\|\\right\\\|\_\{\\infty\}\}≤12​\[‖𝒢P^πγ⋆,U​\|ξV\|‖∞\+‖𝒢P^πγ⋆,U​\|ξh\|‖∞\]\\displaystyle\\leq\\frac\{1\}\{2\}\\left\[\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\|\\xi\_\{V\}\|\\right\\\|\_\{\\infty\}\+\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\|\\xi\_\{h\}\|\\right\\\|\_\{\\infty\}\\right\]\+C​R⋅ιN\.\\displaystyle\\quad\+C\\frac\{R\\cdot\\iota\}\{N\}\.Combining the preceding two displays leads to

‖𝒢P^πγ⋆,U​\|ξV\|‖∞\+‖𝒢P^πγ⋆,U​\|ξh\|‖∞\\displaystyle\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\|\\xi\_\{V\}\|\\right\\\|\_\{\\infty\}\+\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\|\\xi\_\{h\}\|\\right\\\|\_\{\\infty\}≤C​B⋆​\(R\)⋅ιN\+12​\[‖𝒢P^πγ⋆,U​\|ξV\|‖∞\+‖𝒢P^πγ⋆,U​\|ξh\|‖∞\]\+C​R⋅ιN\.\\displaystyle\\quad\\leq C\\sqrt\{\\frac\{B\_\{\\star\}\(R\)\\cdot\\iota\}\{N\}\}\+\\frac\{1\}\{2\}\\left\[\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\|\\xi\_\{V\}\|\\right\\\|\_\{\\infty\}\+\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\|\\xi\_\{h\}\|\\right\\\|\_\{\\infty\}\\right\]\+C\\frac\{R\\cdot\\iota\}\{N\}\.Subtracting one half of the residual sum from both sides and adjusting the universal constantCCgives

‖𝒢P^πγ⋆,U​\|ξV\|‖∞\+‖𝒢P^πγ⋆,U​\|ξh\|‖∞≤C​B⋆​\(R\)⋅ιN\+C​R⋅ιN\.\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\|\\xi\_\{V\}\|\\right\\\|\_\{\\infty\}\+\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\}\|\\xi\_\{h\}\|\\right\\\|\_\{\\infty\}\\leq C\\sqrt\{\\frac\{B\_\{\\star\}\(R\)\\cdot\\iota\}\{N\}\}\+C\\frac\{R\\cdot\\iota\}\{N\}\.\(135\)This is the second checkpoint: the two fixed residuals in Step 1 are now bounded explicitly\.

##### Step 3: Final substitution\.

In this step, we first substitute \([135](https://arxiv.org/html/2608.06545#A4.E135)\) into \([129](https://arxiv.org/html/2608.06545#A4.E129)\) and then simplify the resulting expression\. The substitution produces

‖T2‖∞≤C​ιN​\(1−γ\)2​B⋆​\(R\)\+R​B⋆​\(R\)⋅ιN\+R2⋅ιN\.\\\|T\_\{2\}\\\|\_\{\\infty\}\\leq C\\sqrt\{\\frac\{\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\\sqrt\{B\_\{\\star\}\(R\)\+R\\sqrt\{\\frac\{B\_\{\\star\}\(R\)\\cdot\\iota\}\{N\}\}\+\\frac\{R^\{2\}\\cdot\\iota\}\{N\}\}\.\(136\)Young’s inequality bounds the middle term in \([136](https://arxiv.org/html/2608.06545#A4.E136)\) as

R​B⋆​\(R\)⋅ιN≤C​B⋆​\(R\)\+C​R2⋅ιN\.R\\sqrt\{\\frac\{B\_\{\\star\}\(R\)\\cdot\\iota\}\{N\}\}\\leq CB\_\{\\star\}\(R\)\+C\\frac\{R^\{2\}\\cdot\\iota\}\{N\}\.Using this bound in \([136](https://arxiv.org/html/2608.06545#A4.E136)\) and then applyingx\+y≤x\+y\\sqrt\{x\+y\}\\leq\\sqrt\{x\}\+\\sqrt\{y\}, we obtain

‖T2‖∞≤C​B⋆​\(R\)⋅ιN​\(1−γ\)2\+C​R⋅ιN​\(1−γ\)\.\\\|T\_\{2\}\\\|\_\{\\infty\}\\leq C\\sqrt\{\\frac\{B\_\{\\star\}\(R\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\+C\\frac\{R\\cdot\\iota\}\{N\(1\-\\gamma\)\}\.

#### D\.5\.10Proof of Lemma[20](https://arxiv.org/html/2608.06545#Thmlemma20)

Recall the fixed\-policy aliases

U≔Vγ⋆,σ=Vγπγ⋆,σ,U^≔V^γπγ⋆,σ\.U\\coloneqq V\_\{\\gamma\}^\{\\star,\\sigma\}=V\_\{\\gamma\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\sigma\},\\qquad\\widehat\{U\}\\coloneqq\\widehat\{V\}\_\{\\gamma\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\sigma\}\.Define the optimal\-policy component of \([67](https://arxiv.org/html/2608.06545#A4.E67)\) by

ΔU≔‖U−U^‖∞\.\\Delta\_\{U\}\\coloneqq\\left\\\|U\-\\widehat\{U\}\\right\\\|\_\{\\infty\}\.The proof has three steps\. We first control the selector mismatch and apply Lemma[18](https://arxiv.org/html/2608.06545#Thmlemma18)to reduce the bound onT6T\_\{6\}to two fixed residuals\. We then bound those residuals\. Finally, we substitute the residual bound and absorb the term involvingΔU\\Delta\_\{U\}\.

##### Step 1: Reduction ofT6T\_\{6\}to fixed residuals\.

In this step, we decomposeT6T\_\{6\}into a form suitable for Lemma[18](https://arxiv.org/html/2608.06545#Thmlemma18)and apply the lemma to isolate the two fixed empirical residuals and the selector\-mismatch cost\.

Set

π=πγ⋆,V=U,P=P^πγ⋆,U^,rπ=rπγ⋆\.\\pi=\\pi\_\{\\gamma\}^\{\\star\},\\qquad V=U,\\qquad P=\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\},\\qquad r^\{\\pi\}=r^\{\\pi\_\{\\gamma\}^\{\\star\}\}\.The span premise follows from \([70](https://arxiv.org/html/2608.06545#A4.E70)\), and the associated anchor defect is againβ⋆\\beta\_\{\\star\}\. Rewriting the true Bellman equation under the mismatched empirical kernel yields

U=rπγ⋆\+γ​P^πγ⋆,U^​U\+γ​\(Pπγ⋆,U−P^πγ⋆,U^\)​U\.U=r^\{\\pi\_\{\\gamma\}^\{\\star\}\}\+\\gamma\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}U\+\\gamma\\left\(P^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\-\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\\right\)U\.Thus the Bellman residual is

b=γ​\(Pπγ⋆,U−P^πγ⋆,U^\)​U\.b=\\gamma\\left\(P^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\-\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\\right\)U\.As in the proof of Lemma[19](https://arxiv.org/html/2608.06545#Thmlemma19), define the fixed\-UUsampling residual and the anchor sampling residual by

ξV=\(P^πγ⋆,U−Pπγ⋆,U\)​U,ξh=\(P^0,πγ⋆−P0,πγ⋆\)​h¯\.\\xi\_\{V\}=\\left\(\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\-P^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)U,\\qquad\\xi\_\{h\}=\\left\(\\widehat\{P\}^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\-P^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\\right\)\\bar\{h\}\.
The only new ingredient is that the empirical robust kernel inbbis selected byU^\\widehat\{U\}rather than byUU\. For each\(s,a\)\(s,a\), optimality of the two selectors implies

\(P^s,aU^−P^s,aU\)​U≥0,\(P^s,aU^−P^s,aU\)​U^≤0\.\\left\(\\widehat\{P\}\_\{s,a\}^\{\\widehat\{U\}\}\-\\widehat\{P\}\_\{s,a\}^\{U\}\\right\)U\\geq 0,\\qquad\\left\(\\widehat\{P\}\_\{s,a\}^\{\\widehat\{U\}\}\-\\widehat\{P\}\_\{s,a\}^\{U\}\\right\)\\widehat\{U\}\\leq 0\.Consequently,

0≤\(P^s,aU^−P^s,aU\)​U≤ΔU​‖P^s,aU^−P^s,aU‖1\.0\\leq\\left\(\\widehat\{P\}\_\{s,a\}^\{\\widehat\{U\}\}\-\\widehat\{P\}\_\{s,a\}^\{U\}\\right\)U\\leq\\Delta\_\{U\}\\left\\\|\\widehat\{P\}\_\{s,a\}^\{\\widehat\{U\}\}\-\\widehat\{P\}\_\{s,a\}^\{U\}\\right\\\|\_\{1\}\.The two selected transition distributionsP^s,aU\\widehat\{P\}\_\{s,a\}^\{U\}andP^s,aU^\\widehat\{P\}\_\{s,a\}^\{\\widehat\{U\}\}both belong to𝒰s,a​\(P^0\)\\mathcal\{U\}\_\{s,a\}\(\\widehat\{P\}^\{0\}\)\. Therefore,

‖P^s,aU^−P^s,aU‖1≤‖P^s,aU^−P^s,a0‖1\+‖P^s,aU−P^s,a0‖1≤4​σ\.\\left\\\|\\widehat\{P\}\_\{s,a\}^\{\\widehat\{U\}\}\-\\widehat\{P\}\_\{s,a\}^\{U\}\\right\\\|\_\{1\}\\leq\\left\\\|\\widehat\{P\}\_\{s,a\}^\{\\widehat\{U\}\}\-\\widehat\{P\}^\{0\}\_\{s,a\}\\right\\\|\_\{1\}\+\\left\\\|\\widehat\{P\}\_\{s,a\}^\{U\}\-\\widehat\{P\}^\{0\}\_\{s,a\}\\right\\\|\_\{1\}\\leq 4\\sigma\.Averaging overa∼πγ⋆\(⋅∣s\)a\\sim\\pi\_\{\\gamma\}^\{\\star\}\(\\cdot\\mid s\)therefore implies

‖\(P^πγ⋆,U^−P^πγ⋆,U\)​U‖∞≤4​σ​ΔU\.\\left\\\|\\left\(\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\-\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\\right\)U\\right\\\|\_\{\\infty\}\\leq 4\\sigma\\Delta\_\{U\}\.\(137\)By the definitions ofbbandξV\\xi\_\{V\},

b=−γ​ξV\+γ​\(P^πγ⋆,U−P^πγ⋆,U^\)​U\.b=\-\\gamma\\xi\_\{V\}\+\\gamma\\left\(\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\-\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\\right\)U\.Because the normalized resolvent𝒢P^πγ⋆,U^\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\}is nonnegative, the triangle inequality implies

𝒢P^πγ⋆,U^​\|b\|≤\\displaystyle\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\}\|b\|\\leq\{\}γ​𝒢P^πγ⋆,U^​\|ξV\|\+γ​𝒢P^πγ⋆,U^​\|\(P^πγ⋆,U−P^πγ⋆,U^\)​U\|\\displaystyle\\gamma\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\}\|\\xi\_\{V\}\|\+\\gamma\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\}\\left\|\\left\(\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},U\}\-\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\\right\)U\\right\|componentwise\. Applying \([137](https://arxiv.org/html/2608.06545#A4.E137)\), using𝒢P^πγ⋆,U^​𝟏S=𝟏S\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\}\\bm\{1\}\_\{S\}=\\bm\{1\}\_\{S\}, and then usingγ≤1\\gamma\\leq 1yields

‖𝒢P^πγ⋆,U^​\|b\|‖∞≤‖𝒢P^πγ⋆,U^​\|ξV\|‖∞\+C​σ​ΔU\.\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\}\|b\|\\right\\\|\_\{\\infty\}\\leq\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\}\|\\xi\_\{V\}\|\\right\\\|\_\{\\infty\}\+C\\sigma\\Delta\_\{U\}\.\(138\)For the anchor residual, decompose

\(P^πγ⋆,U^−P0,πγ⋆\)​h¯=\(P^πγ⋆,U^−P^0,πγ⋆\)​h¯\+ξh\.\\left\(\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\-P^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\\right\)\\bar\{h\}=\\left\(\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\-\\widehat\{P\}^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\\right\)\\bar\{h\}\+\\xi\_\{h\}\.The first term is the robust perturbation from the empirical nominal kernel\. The robust\-set constraint and‖h¯‖span≤Hanc\\\|\\bar\{h\}\\\|\_\{\\mathrm\{span\}\}\\leq H\_\{\\mathrm\{anc\}\}bound its absolute value byC​σ​Hanc⋅𝟏SC\\sigma H\_\{\\mathrm\{anc\}\}\\cdot\\bm\{1\}\_\{S\}\. Therefore,

\|\(P^πγ⋆,U^−P0,πγ⋆\)​h¯\|≤C​σ​Hanc⋅𝟏S\+\|ξh\|\.\\left\|\\left\(\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\-P^\{0,\\pi\_\{\\gamma\}^\{\\star\}\}\\right\)\\bar\{h\}\\right\|\\leq C\\sigma H\_\{\\mathrm\{anc\}\}\\cdot\\bm\{1\}\_\{S\}\+\|\\xi\_\{h\}\|\.Applying Lemma[18](https://arxiv.org/html/2608.06545#Thmlemma18)withbbitself as the value residual envelope, withξh\\xi\_\{h\}as defined above, and then using \([138](https://arxiv.org/html/2608.06545#A4.E138)\), we obtain

‖T6‖∞≤C​ιN​\(1−γ\)2​B⋆​\(R\)\+R​‖𝒢P^πγ⋆,U^​\|ξV\|‖∞\+R​‖𝒢P^πγ⋆,U^​\|ξh\|‖∞\+σ​R​ΔU\.\\\|T\_\{6\}\\\|\_\{\\infty\}\\leq C\\sqrt\{\\frac\{\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\\sqrt\{B\_\{\\star\}\(R\)\+R\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\}\|\\xi\_\{V\}\|\\right\\\|\_\{\\infty\}\+R\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\}\|\\xi\_\{h\}\|\\right\\\|\_\{\\infty\}\+\\sigma R\\Delta\_\{U\}\}\.\(139\)This is the first checkpoint: the direct analysis ofT6T\_\{6\}is complete, and it remains to control the same two fixed residuals as in the proof of Lemma[19](https://arxiv.org/html/2608.06545#Thmlemma19), now under the mismatched resolvent\.

##### Step 2: Bounding the fixed residuals\.

In this step, we obtain an explicit bound for the two residuals isolated in Step 1\. We claim that

‖𝒢P^πγ⋆,U^​\|ξV\|‖∞\+‖𝒢P^πγ⋆,U^​\|ξh\|‖∞≤C​\(B⋆​\(R\)\+σ​R​ΔU\)⋅ιN\+C​R⋅ιN\.\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\}\|\\xi\_\{V\}\|\\right\\\|\_\{\\infty\}\+\\left\\\|\\mathcal\{G\}\_\{\\widehat\{P\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\widehat\{U\}\}\}\|\\xi\_\{h\}\|\\right\\\|\_\{\\infty\}\\leq C\\sqrt\{\\frac\{\\left\(B\_\{\\star\}\(R\)\+\\sigma R\\Delta\_\{U\}\\right\)\\cdot\\iota\}\{N\}\}\+C\\frac\{R\\cdot\\iota\}\{N\}\.\(140\)
The proof of \([140](https://arxiv.org/html/2608.06545#A4.E140)\) follows Step 2 of the proof of Lemma[19](https://arxiv.org/html/2608.06545#Thmlemma19), with the additional termσ​R​ΔU\\sigma R\\Delta\_\{U\}arising from the selector\-mismatch termC​σ​ΔUC\\sigma\\Delta\_\{U\}in \([138](https://arxiv.org/html/2608.06545#A4.E138)\)\. We omit the details for brevity\.

##### Step 3: Final substitution and absorption\.

In this step, we substitute the fixed\-residual bound into the reduction from Step 1 and bound theΔU\\Delta\_\{U\}\-related term\. To substitute \([140](https://arxiv.org/html/2608.06545#A4.E140)\) into the main reduction, first note that Young’s inequality shows that

R​\(B⋆​\(R\)\+σ​R​ΔU\)⋅ιN≤C​\(B⋆​\(R\)\+σ​R​ΔU\)\+C​R2⋅ιN\.R\\sqrt\{\\frac\{\\left\(B\_\{\\star\}\(R\)\+\\sigma R\\Delta\_\{U\}\\right\)\\cdot\\iota\}\{N\}\}\\leq C\\left\(B\_\{\\star\}\(R\)\+\\sigma R\\Delta\_\{U\}\\right\)\+C\\frac\{R^\{2\}\\cdot\\iota\}\{N\}\.Substituting this inequality and \([140](https://arxiv.org/html/2608.06545#A4.E140)\) into \([139](https://arxiv.org/html/2608.06545#A4.E139)\) produces

‖T6‖∞≤C​B⋆​\(R\)⋅ιN​\(1−γ\)2\+C​R⋅ιN​\(1−γ\)\+C​σ​R​ΔU⋅ιN​\(1−γ\)2\.\\\|T\_\{6\}\\\|\_\{\\infty\}\\leq C\\sqrt\{\\frac\{B\_\{\\star\}\(R\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\+C\\frac\{R\\cdot\\iota\}\{N\(1\-\\gamma\)\}\+C\\sqrt\{\\frac\{\\sigma R\\Delta\_\{U\}\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\.\(141\)By Young’s inequality and the second condition in \([71](https://arxiv.org/html/2608.06545#A4.E71)\),

C​σ​R​ΔU⋅ιN​\(1−γ\)2≤150​ΔU\+C​σ​R⋅ιN​\(1−γ\)2≤150​ΔU\+C​B⋆​\(R\)⋅ιN​\(1−γ\)2\.C\\sqrt\{\\frac\{\\sigma R\\Delta\_\{U\}\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\\leq\\frac\{1\}\{50\}\\Delta\_\{U\}\+C\\frac\{\\sigma R\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\\leq\\frac\{1\}\{50\}\\Delta\_\{U\}\+C\\sqrt\{\\frac\{B\_\{\\star\}\(R\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\.SinceΔU≤Δ\\Delta\_\{U\}\\leq\\Delta, the last display and \([141](https://arxiv.org/html/2608.06545#A4.E141)\) imply the desired bound \([105](https://arxiv.org/html/2608.06545#A4.E105)\)\.

#### D\.5\.11Proof of Lemma[22](https://arxiv.org/html/2608.06545#Thmlemma22)

The proof has four steps\. We first reduce the desired bound to a centered empirical process involvingV^⋆\\widehat\{V\}^\{\\star\}\. We then construct leave\-one\-out value functions and discretize their auxiliary reward parameter\. Next, we establish uniform concentration and transfer it back toV^⋆\\widehat\{V\}^\{\\star\}\. Finally, we average the resulting rowwise bounds over the learned policy\.

##### Step 1: Reduction to a centered empirical\-optimal\-value process\.

In this step, we reduce the rowwise robust\-kernel error atW^\\widehat\{W\}to a centered empirical process involvingV^⋆\\widehat\{V\}^\{\\star\}\. Fix a state\-action pair\(s,a\)\(s,a\)and a value vectorVV, and writeV∘≔V−mins′∈𝒮⁡V​\(s′\)⋅𝟏SV^\{\\circ\}\\coloneqq V\-\\min\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}V\(s^\{\\prime\}\)\\cdot\\bm\{1\}\_\{S\}\. Then0≤V∘≤‖V‖span⋅𝟏S0\\leq V^\{\\circ\}\\leq\\\|V\\\|\_\{\\mathrm\{span\}\}\\cdot\\bm\{1\}\_\{S\}\. The TV strong\-duality formula inShiet al\.\([2026](https://arxiv.org/html/2608.06545#bib.bib3), Lemma 4\)yields

\|P^s,aV​V−Ps,aV​V\|≤supα∈\[0,‖V‖span\]\|\(P^s,a0−Ps,a0\)​\[V∘\]α\|,\\left\|\\widehat\{P\}\_\{s,a\}^\{V\}V\-P\_\{s,a\}^\{V\}V\\right\|\\leq\\sup\_\{\\alpha\\in\[0,\\\|V\\\|\_\{\\mathrm\{span\}\}\]\}\\left\|\(\\widehat\{P\}^\{0\}\_\{s,a\}\-P^\{0\}\_\{s,a\}\)\[V^\{\\circ\}\]\_\{\\alpha\}\\right\|,\(142\)where\[z\]α≔\(min⁡\{z​\(s′\),α\}\)s′∈𝒮\[z\]\_\{\\alpha\}\\coloneqq\(\\min\\\{z\(s^\{\\prime\}\),\\alpha\\\}\)\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}\. Applying \([142](https://arxiv.org/html/2608.06545#A4.E142)\) withV=W^V=\\widehat\{W\}and using‖W^‖span≤R\\\|\\widehat\{W\}\\\|\_\{\\mathrm\{span\}\}\\leq R, we obtain

\|P^s,aW^​W^−Ps,aW^​W^\|≤supα∈\[0,R\]\|\(P^s,a0−Ps,a0\)​\[W^∘\]α\|\.\\left\|\\widehat\{P\}\_\{s,a\}^\{\\widehat\{W\}\}\\widehat\{W\}\-P\_\{s,a\}^\{\\widehat\{W\}\}\\widehat\{W\}\\right\|\\leq\\sup\_\{\\alpha\\in\[0,R\]\}\\left\|\(\\widehat\{P\}^\{0\}\_\{s,a\}\-P^\{0\}\_\{s,a\}\)\[\\widehat\{W\}^\{\\circ\}\]\_\{\\alpha\}\\right\|\.On the other hand, the optimality ofV^⋆\\widehat\{V\}^\{\\star\}and the solver guarantee \([5](https://arxiv.org/html/2608.06545#S4.E5)\) imply

0≤V^⋆−W^≤εopt⋅𝟏S\.0\\leq\\widehat\{V\}^\{\\star\}\-\\widehat\{W\}\\leq\\varepsilon\_\{\\mathrm\{opt\}\}\\cdot\\bm\{1\}\_\{S\}\.Since

‖W^∘−\(V^⋆\)∘‖∞\\displaystyle\\left\\\|\\widehat\{W\}^\{\\circ\}\-\(\\widehat\{V\}^\{\\star\}\)^\{\\circ\}\\right\\\|\_\{\\infty\}≤‖W^−V^⋆‖∞\+\|mins⁡W^​\(s\)−mins⁡V^⋆​\(s\)\|\\displaystyle\\leq\\\|\\widehat\{W\}\-\\widehat\{V\}^\{\\star\}\\\|\_\{\\infty\}\+\\left\|\\min\_\{s\}\\widehat\{W\}\(s\)\-\\min\_\{s\}\\widehat\{V\}^\{\\star\}\(s\)\\right\|≤2​‖W^−V^⋆‖∞,\\displaystyle\\leq 2\\\|\\widehat\{W\}\-\\widehat\{V\}^\{\\star\}\\\|\_\{\\infty\},we obtain

‖W^∘−\(V^⋆\)∘‖∞≤2​εopt\.\\left\\\|\\widehat\{W\}^\{\\circ\}\-\(\\widehat\{V\}^\{\\star\}\)^\{\\circ\}\\right\\\|\_\{\\infty\}\\leq 2\\varepsilon\_\{\\mathrm\{opt\}\}\.Sincez↦\[z\]αz\\mapsto\[z\]\_\{\\alpha\}is 1\-Lipschitz and‖P^s,a0−Ps,a0‖1≤2\\\|\\widehat\{P\}^\{0\}\_\{s,a\}\-P^\{0\}\_\{s,a\}\\\|\_\{1\}\\leq 2, combining the preceding displays results in

\|P^s,aW^​W^−Ps,aW^​W^\|≤supα∈\[0,R\]\|\(P^s,a0−Ps,a0\)​\[\(V^⋆\)∘\]α\|\+C​εopt\.\\left\|\\widehat\{P\}\_\{s,a\}^\{\\widehat\{W\}\}\\widehat\{W\}\-P\_\{s,a\}^\{\\widehat\{W\}\}\\widehat\{W\}\\right\|\\leq\\sup\_\{\\alpha\\in\[0,R\]\}\\left\|\(\\widehat\{P\}^\{0\}\_\{s,a\}\-P^\{0\}\_\{s,a\}\)\[\(\\widehat\{V\}^\{\\star\}\)^\{\\circ\}\]\_\{\\alpha\}\\right\|\+C\\varepsilon\_\{\\mathrm\{opt\}\}\.\(143\)Therefore, up to the additiveC​εoptC\\varepsilon\_\{\\mathrm\{opt\}\}term, our goal reduces to controlling the centered empirical process given by the supremum term in \([143](https://arxiv.org/html/2608.06545#A4.E143)\) uniformly over all\(s,a\)\(s,a\)\.

##### Step 2: Leave\-one\-out construction and discretization\.

The empirical process isolated in Step 1 is data\-dependent, so in this step we construct a discretized family of leave\-one\-out value functions that is independent of each empirical transition row and approximatesV^⋆\\widehat\{V\}^\{\\star\}\. We adapt the auxiliary\-MDP leave\-one\-out construction ofShiet al\.\([2026](https://arxiv.org/html/2608.06545#bib.bib3), Appendix B\.3\.5\)to the centered, span\-localized empirical process in \([143](https://arxiv.org/html/2608.06545#A4.E143)\)\. For each source statessand scalaru≥0u\\geq 0, letℳ^s,u\\widehat\{\\mathcal\{M\}\}^\{s,u\}have the same uncertainty\-set rule and discount factor as the empirical robust MDP, but with nominal kernel and reward

P^s~,a0,s,u≔\{es,s~=s,P^s~,a0,s~≠s,rs,u​\(s~,a\)≔\{u,s~=s,r​\(s~,a\),s~≠s,\\widehat\{P\}^\{0,s,u\}\_\{\\widetilde\{s\},a\}\\coloneqq\\begin\{cases\}e\_\{s\},&\\widetilde\{s\}=s,\\\\ \\widehat\{P\}^\{0\}\_\{\\widetilde\{s\},a\},&\\widetilde\{s\}\\neq s,\\end\{cases\}\\qquad r^\{s,u\}\(\\widetilde\{s\},a\)\\coloneqq\\begin\{cases\}u,&\\widetilde\{s\}=s,\\\\ r\(\\widetilde\{s\},a\),&\\widetilde\{s\}\\neq s,\\end\{cases\}for every\(s~,a\)∈𝒮×𝒜\(\\widetilde\{s\},a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}, whereese\_\{s\}is thess\-th standard basis probability vector\. In words, the construction makesssabsorbing in the auxiliary nominal model and assigns rewarduuto every action atss, while leaving all other nominal transition rows and rewards unchanged\. LetV^s,u⋆\\widehat\{V\}\_\{s,u\}^\{\\star\}be the optimal robust value of this auxiliary empirical MDP\. Because the nominal centers of the rows out of statessare fixed atese\_\{s\},V^s,u⋆\\widehat\{V\}\_\{s,u\}^\{\\star\}is independent of the samples used to formP^s,a0\\widehat\{P\}^\{0\}\_\{s,a\}, for every actionaaat that source state\.

Define

u⋆≔V^⋆​\(s\)−γ​maxb∈𝒜​infQ∈𝒰s,b​\(es\)Q​V^⋆\.u\_\{\\star\}\\coloneqq\\widehat\{V\}^\{\\star\}\(s\)\-\\gamma\\max\_\{b\\in\\mathcal\{A\}\}\\inf\_\{Q\\in\\mathcal\{U\}\_\{s,b\}\(e\_\{s\}\)\}Q\\widehat\{V\}^\{\\star\}\.Sincees∈𝒰s,b​\(es\)e\_\{s\}\\in\\mathcal\{U\}\_\{s,b\}\(e\_\{s\}\)for everyb∈𝒜b\\in\\mathcal\{A\}and \([73](https://arxiv.org/html/2608.06545#A4.E73)\) applies toV^⋆\\widehat\{V\}^\{\\star\}, we have0≤u⋆≤\(1−γ\)−10\\leq u\_\{\\star\}\\leq\(1\-\\gamma\)^\{\-1\}\. With this choice,V^⋆\\widehat\{V\}^\{\\star\}is a fixed point of the auxiliary robust Bellman operator: its update at statessequalsV^⋆​\(s\)\\widehat\{V\}^\{\\star\}\(s\), while all other states have the same updates as in the original empirical robust MDP\. Since this operator is aγ\\gamma\-contraction, its fixed point is unique, and hence

V^s,u⋆⋆=V^⋆\.\\widehat\{V\}\_\{s,u\_\{\\star\}\}^\{\\star\}=\\widehat\{V\}^\{\\star\}\.Let𝒩u\\mathcal\{N\}\_\{u\}be a uniform grid of\[0,\(1−γ\)−1\]\[0,\(1\-\\gamma\)^\{\-1\}\]with spacing at most

ηu≔\(1−γ\)​R⋅ιC1​N\\eta\_\{u\}\\coloneqq\\frac\{\(1\-\\gamma\)R\\cdot\\iota\}\{C\_\{1\}N\}for a sufficiently large numerical constantC1C\_\{1\}\. Chooseu∈𝒩uu\\in\\mathcal\{N\}\_\{u\}with\|u−u⋆\|≤ηu\|u\-u\_\{\\star\}\|\\leq\\eta\_\{u\}\. The auxiliary empirical robust MDPsℳ^s,u\\widehat\{\\mathcal\{M\}\}^\{s,u\}andℳ^s,u⋆\\widehat\{\\mathcal\{M\}\}^\{s,u\_\{\\star\}\}differ only in their rewards at statess\.

Consequently, for any value vector, their robust Bellman updates agree at every state other thanss, while the updates atssdiffer byu−u⋆u\-u\_\{\\star\}\. Thus, the two update vectors differ in sup\-norm by\|u−u⋆\|≤ηu\|u\-u\_\{\\star\}\|\\leq\\eta\_\{u\}\. Since both robust Bellman operators areγ\\gamma\-contractions, their fixed points satisfy

‖V^s,u⋆−V^⋆‖∞≤ηu1−γ≤R⋅ιC1​N\.\\\|\\widehat\{V\}\_\{s,u\}^\{\\star\}\-\\widehat\{V\}^\{\\star\}\\\|\_\{\\infty\}\\leq\\frac\{\\eta\_\{u\}\}\{1\-\\gamma\}\\leq\\frac\{R\\cdot\\iota\}\{C\_\{1\}N\}\.\(144\)Using \([144](https://arxiv.org/html/2608.06545#A4.E144)\), we obtain

‖\(V^s,u⋆\)∘−\(V^⋆\)∘‖∞\\displaystyle\\left\\\|\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}\-\(\\widehat\{V\}^\{\\star\}\)^\{\\circ\}\\right\\\|\_\{\\infty\}≤‖V^s,u⋆−V^⋆‖∞\+\|miny⁡V^s,u⋆​\(y\)−miny⁡V^⋆​\(y\)\|\\displaystyle\\leq\\left\\\|\\widehat\{V\}\_\{s,u\}^\{\\star\}\-\\widehat\{V\}^\{\\star\}\\right\\\|\_\{\\infty\}\+\\left\|\\min\_\{y\}\\widehat\{V\}\_\{s,u\}^\{\\star\}\(y\)\-\\min\_\{y\}\\widehat\{V\}^\{\\star\}\(y\)\\right\|\(145\)≤2​‖V^s,u⋆−V^⋆‖∞≤2​R⋅ιC1​N\.\\displaystyle\\leq 2\\left\\\|\\widehat\{V\}\_\{s,u\}^\{\\star\}\-\\widehat\{V\}^\{\\star\}\\right\\\|\_\{\\infty\}\\leq\\frac\{2R\\cdot\\iota\}\{C\_\{1\}N\}\.Moreover, since‖V^⋆‖span≤R\\\|\\widehat\{V\}^\{\\star\}\\\|\_\{\\mathrm\{span\}\}\\leq R,

0≤\(V^s,u⋆\)∘\\displaystyle 0\\leq\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}≤‖V^s,u⋆‖span⋅𝟏S\\displaystyle\\leq\\\|\\widehat\{V\}\_\{s,u\}^\{\\star\}\\\|\_\{\\mathrm\{span\}\}\\cdot\\bm\{1\}\_\{S\}\(146\)≤\(‖V^⋆‖span\+2​‖V^s,u⋆−V^⋆‖∞\)​𝟏S≤2​R⋅𝟏S\.\\displaystyle\\leq\\left\(\\\|\\widehat\{V\}^\{\\star\}\\\|\_\{\\mathrm\{span\}\}\+2\\\|\\widehat\{V\}\_\{s,u\}^\{\\star\}\-\\widehat\{V\}^\{\\star\}\\\|\_\{\\infty\}\\right\)\\bm\{1\}\_\{S\}\\leq 2R\\cdot\\bm\{1\}\_\{S\}\.The final inequality uses the sample\-size condition and sufficiently large numerical constantsC0C\_\{0\}andC1C\_\{1\}\.

##### Step 3: Uniform concentration and transfer\.

In this step, we apply Bernstein’s inequality uniformly to the leave\-one\-out family from Step 2 and transfer the resulting bound to the empirical process from Step 1\. Fix a triple\(s,a,u\)\(s,a,u\)and condition on all samples other than those formingP^s,a0\\widehat\{P\}^\{0\}\_\{s,a\}\. By the leave\-one\-out construction,\(V^s,u⋆\)∘\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}is then fixed and independent ofP^s,a0\\widehat\{P\}^\{0\}\_\{s,a\}\. If‖\(V^s,u⋆\)∘‖∞=0\\\|\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}\\\|\_\{\\infty\}=0, set𝒩αs,u=\{0\}\\mathcal\{N\}\_\{\\alpha\}^\{s,u\}=\\\{0\\\}\. Otherwise, let𝒩αs,u\\mathcal\{N\}\_\{\\alpha\}^\{s,u\}be a uniform grid of\[0,‖\(V^s,u⋆\)∘‖∞\]\[0,\\\|\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}\\\|\_\{\\infty\}\]with spacing at most

‖\(V^s,u⋆\)∘‖∞⋅ιC2​N,\\frac\{\\\|\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}\\\|\_\{\\infty\}\\cdot\\iota\}\{C\_\{2\}N\},whereC2C\_\{2\}is a sufficiently large numerical constant\. Both definitions satisfy\|𝒩αs,u\|≤1\+C2​N/ι\|\\mathcal\{N\}\_\{\\alpha\}^\{s,u\}\|\\leq 1\+C\_\{2\}N/\\iota\.

For eachα∈𝒩αs,u\\alpha\\in\\mathcal\{N\}\_\{\\alpha\}^\{s,u\}, the clipped vector is fixed and independent ofP^s,a0\\widehat\{P\}^\{0\}\_\{s,a\}\. Moreover,

‖\[\(V^s,u⋆\)∘\]α‖∞≤‖\(V^s,u⋆\)∘‖∞,VarPs,a0​\(\[\(V^s,u⋆\)∘\]α\)≤VarPs,a0​\(\(V^s,u⋆\)∘\)\.\\left\\\|\[\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}\]\_\{\\alpha\}\\right\\\|\_\{\\infty\}\\leq\\\|\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}\\\|\_\{\\infty\},\\qquad\\mathrm\{Var\}\_\{P^\{0\}\_\{s,a\}\}\(\[\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}\]\_\{\\alpha\}\)\\leq\\mathrm\{Var\}\_\{P^\{0\}\_\{s,a\}\}\(\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}\)\.Therefore, under this conditioning, Bernstein’s inequality and a union bound over𝒩αs,u\\mathcal\{N\}\_\{\\alpha\}^\{s,u\}imply that, with probability at least1−2​\|𝒩αs,u\|​e−3​ι1\-2\|\\mathcal\{N\}\_\{\\alpha\}^\{s,u\}\|e^\{\-3\\iota\},

supα∈𝒩αs,u\|\(P^s,a0−Ps,a0\)​\[\(V^s,u⋆\)∘\]α\|≤C​ιN​VarPs,a0​\(\(V^s,u⋆\)∘\)\+C​‖\(V^s,u⋆\)∘‖∞⋅ιN\.\\sup\_\{\\alpha\\in\\mathcal\{N\}\_\{\\alpha\}^\{s,u\}\}\\left\|\(\\widehat\{P\}^\{0\}\_\{s,a\}\-P^\{0\}\_\{s,a\}\)\[\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}\]\_\{\\alpha\}\\right\|\\leq C\\sqrt\{\\frac\{\\iota\}\{N\}\}\\sqrt\{\\mathrm\{Var\}\_\{P^\{0\}\_\{s,a\}\}\(\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}\)\}\+C\\frac\{\\\|\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}\\\|\_\{\\infty\}\\cdot\\iota\}\{N\}\.
It remains to extend the bound from the grid to every clipping level\. Givenα∈\[0,‖\(V^s,u⋆\)∘‖∞\]\\alpha\\in\[0,\\\|\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}\\\|\_\{\\infty\}\], letα′∈𝒩αs,u\\alpha^\{\\prime\}\\in\\mathcal\{N\}\_\{\\alpha\}^\{s,u\}be a nearest grid point\. The grid spacing and the 1\-Lipschitz property of clipping imply

‖\[\(V^s,u⋆\)∘\]α−\[\(V^s,u⋆\)∘\]α′‖∞≤\|α−α′\|≤‖\(V^s,u⋆\)∘‖∞⋅ιC2​N\.\\left\\\|\[\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}\]\_\{\\alpha\}\-\[\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}\]\_\{\\alpha^\{\\prime\}\}\\right\\\|\_\{\\infty\}\\leq\|\\alpha\-\\alpha^\{\\prime\}\|\\leq\\frac\{\\\|\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}\\\|\_\{\\infty\}\\cdot\\iota\}\{C\_\{2\}N\}\.Together with‖P^s,a0−Ps,a0‖1≤2\\\|\\widehat\{P\}^\{0\}\_\{s,a\}\-P^\{0\}\_\{s,a\}\\\|\_\{1\}\\leq 2, this shows that replacingα\\alphabyα′\\alpha^\{\\prime\}changes the empirical process by at mostC​‖\(V^s,u⋆\)∘‖∞⋅ι/NC\\\|\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}\\\|\_\{\\infty\}\\cdot\\iota/N\. Thus, on the same conditional event, the following bound holds for this fixed triple\(s,a,u\)\(s,a,u\):

supα∈\[0,‖\(V^s,u⋆\)∘‖∞\]\|\(P^s,a0−Ps,a0\)​\[\(V^s,u⋆\)∘\]α\|≤C​ιN​VarPs,a0​\(\(V^s,u⋆\)∘\)\+C​‖\(V^s,u⋆\)∘‖∞⋅ιN\.\\sup\_\{\\alpha\\in\[0,\\\|\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}\\\|\_\{\\infty\}\]\}\\left\|\(\\widehat\{P\}^\{0\}\_\{s,a\}\-P^\{0\}\_\{s,a\}\)\[\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}\]\_\{\\alpha\}\\right\|\\leq C\\sqrt\{\\frac\{\\iota\}\{N\}\}\\sqrt\{\\mathrm\{Var\}\_\{P^\{0\}\_\{s,a\}\}\(\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}\)\}\+C\\frac\{\\\|\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}\\\|\_\{\\infty\}\\cdot\\iota\}\{N\}\.\(147\)
Finally, unconditioning and taking a union bound over all\(s,a\)\(s,a\)andu∈𝒩uu\\in\\mathcal\{N\}\_\{u\}show that \([147](https://arxiv.org/html/2608.06545#A4.E147)\) holds simultaneously for all\(s,a,u\)\(s,a,u\)with probability at least

1−2​S​A​\(1\+C​N3\)​\(1\+C2​Nι\)​e−3​ι≥1−C​S​A​N4​e−3​ι=1−O​\(δ\)\.1\-2SA\(1\+CN^\{3\}\)\\left\(1\+\\frac\{C\_\{2\}N\}\{\\iota\}\\right\)e^\{\-3\\iota\}\\geq 1\-CSAN^\{4\}e^\{\-3\\iota\}=1\-O\(\\delta\)\.Here the last equality follows from

C​S​A​N4​e−3​ι=C​S​A​N4​\(\(1−γ\)​δ54​S​A​N2\)3=O​\(δ\)\.CSAN^\{4\}e^\{\-3\\iota\}=CSAN^\{4\}\\left\(\\frac\{\(1\-\\gamma\)\\delta\}\{54SAN^\{2\}\}\\right\)^\{3\}=O\(\\delta\)\.For the grid pointu∈𝒩uu\\in\\mathcal\{N\}\_\{u\}satisfying\|u−u⋆\|≤ηu\|u\-u\_\{\\star\}\|\\leq\\eta\_\{u\}, \([146](https://arxiv.org/html/2608.06545#A4.E146)\) ensures that‖\(V^s,u⋆\)∘‖∞≤2​R\\\|\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}\\\|\_\{\\infty\}\\leq 2R, so the bounded\-difference term in \([147](https://arxiv.org/html/2608.06545#A4.E147)\) isO​\(R⋅ι/N\)O\(R\\cdot\\iota/N\)rather thanO​\(ι/\[N​\(1−γ\)\]\)O\(\\iota/\[N\(1\-\\gamma\)\]\)\.

Combining \([145](https://arxiv.org/html/2608.06545#A4.E145)\) with the Lipschitz property of clipping yields

supα∈\[0,R\]\|\(P^s,a0−Ps,a0\)​\[\(V^⋆\)∘\]α\|\\displaystyle\\sup\_\{\\alpha\\in\[0,R\]\}\\left\|\(\\widehat\{P\}^\{0\}\_\{s,a\}\-P^\{0\}\_\{s,a\}\)\[\(\\widehat\{V\}^\{\\star\}\)^\{\\circ\}\]\_\{\\alpha\}\\right\|≤supα∈\[0,2​R\]\|\(P^s,a0−Ps,a0\)​\[\(V^s,u⋆\)∘\]α\|\+C​R⋅ιN\.\\displaystyle\\leq\\sup\_\{\\alpha\\in\[0,2R\]\}\\left\|\(\\widehat\{P\}^\{0\}\_\{s,a\}\-P^\{0\}\_\{s,a\}\)\[\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}\]\_\{\\alpha\}\\right\|\+C\\frac\{R\\cdot\\iota\}\{N\}\.\(148\)The concentration event \([147](https://arxiv.org/html/2608.06545#A4.E147)\) applies to the supremum over\[0,2​R\]\[0,2R\]: if2​R\>‖\(V^s,u⋆\)∘‖∞2R\>\\\|\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}\\\|\_\{\\infty\}, then clipping above‖\(V^s,u⋆\)∘‖∞\\\|\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}\\\|\_\{\\infty\}leaves\(V^s,u⋆\)∘\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}unchanged, so the supremum over\[0,2​R\]\[0,2R\]is the same as the supremum over\[0,‖\(V^s,u⋆\)∘‖∞\]\[0,\\\|\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}\\\|\_\{\\infty\}\]\. Moreover, applying Minkowski’s inequality inL2​\(Ps,a0\)L\_\{2\}\(P^\{0\}\_\{s,a\}\)to the decomposition of\(V^s,u⋆\)∘\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}around\(V^⋆\)∘\(\\widehat\{V\}^\{\\star\}\)^\{\\circ\}and then using \([145](https://arxiv.org/html/2608.06545#A4.E145)\), we obtain

VarPs,a0​\(\(V^s,u⋆\)∘\)\\displaystyle\\sqrt\{\\mathrm\{Var\}\_\{P^\{0\}\_\{s,a\}\}\(\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}\)\}≤VarPs,a0​\(\(V^⋆\)∘\)\+VarPs,a0​\(\(V^s,u⋆\)∘−\(V^⋆\)∘\)\\displaystyle\\leq\\sqrt\{\\mathrm\{Var\}\_\{P^\{0\}\_\{s,a\}\}\(\(\\widehat\{V\}^\{\\star\}\)^\{\\circ\}\)\}\+\\sqrt\{\\mathrm\{Var\}\_\{P^\{0\}\_\{s,a\}\}\\left\(\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}\-\(\\widehat\{V\}^\{\\star\}\)^\{\\circ\}\\right\)\}≤VarPs,a0​\(V^⋆\)\+‖\(V^s,u⋆\)∘−\(V^⋆\)∘‖∞\\displaystyle\\leq\\sqrt\{\\mathrm\{Var\}\_\{P^\{0\}\_\{s,a\}\}\(\\widehat\{V\}^\{\\star\}\)\}\+\\left\\\|\(\\widehat\{V\}\_\{s,u\}^\{\\star\}\)^\{\\circ\}\-\(\\widehat\{V\}^\{\\star\}\)^\{\\circ\}\\right\\\|\_\{\\infty\}≤VarPs,a0​\(V^⋆\)\+C​R⋅ιN\.\\displaystyle\\leq\\sqrt\{\\mathrm\{Var\}\_\{P^\{0\}\_\{s,a\}\}\(\\widehat\{V\}^\{\\star\}\)\}\+C\\frac\{R\\cdot\\iota\}\{N\}\.\(149\)Substituting \([147](https://arxiv.org/html/2608.06545#A4.E147)\) and \([D\.5\.11](https://arxiv.org/html/2608.06545#A4.Ex417)\) into \([148](https://arxiv.org/html/2608.06545#A4.E148)\) yields

supα∈\[0,R\]\|\(P^s,a0−Ps,a0\)​\[\(V^⋆\)∘\]α\|≤C​ιN​VarPs,a0​\(V^⋆\)\+C​R⋅ιN\.\\sup\_\{\\alpha\\in\[0,R\]\}\\left\|\(\\widehat\{P\}^\{0\}\_\{s,a\}\-P^\{0\}\_\{s,a\}\)\[\(\\widehat\{V\}^\{\\star\}\)^\{\\circ\}\]\_\{\\alpha\}\\right\|\\leq C\\sqrt\{\\frac\{\\iota\}\{N\}\}\\sqrt\{\\mathrm\{Var\}\_\{P^\{0\}\_\{s,a\}\}\(\\widehat\{V\}^\{\\star\}\)\}\+C\\frac\{R\\cdot\\iota\}\{N\}\.\(150\)Combining \([143](https://arxiv.org/html/2608.06545#A4.E143)\) and \([150](https://arxiv.org/html/2608.06545#A4.E150)\) establishes the rowwise bound

\|P^s,aW^​W^−Ps,aW^​W^\|≤C​ιN​VarPs,a0​\(V^⋆\)\+C​R⋅ιN\+C​εopt\.\\left\|\\widehat\{P\}\_\{s,a\}^\{\\widehat\{W\}\}\\widehat\{W\}\-P\_\{s,a\}^\{\\widehat\{W\}\}\\widehat\{W\}\\right\|\\leq C\\sqrt\{\\frac\{\\iota\}\{N\}\}\\sqrt\{\\mathrm\{Var\}\_\{P^\{0\}\_\{s,a\}\}\(\\widehat\{V\}^\{\\star\}\)\}\+C\\frac\{R\\cdot\\iota\}\{N\}\+C\\varepsilon\_\{\\mathrm\{opt\}\}\.

##### Step 4: Averaging over the learned policy\.

In this final step, we average the rowwise estimate from Step 3 over the learned policy and bound the resulting average of the state\-action conditional standard deviations by the standard deviation underP0,π^P^\{0,\\widehat\{\\pi\}\}\. Because the rowwise estimate holds simultaneously for every\(s,a\)\(s,a\), it may be averaged using the data\-dependent policyπ^\\widehat\{\\pi\}\.

For every statess, the definitions of the policy\-induced robust kernels and the triangle inequality imply

\|\(\(P^π^,W^−Pπ^,W^\)​W^\)​\(s\)\|\\displaystyle\\left\|\\left\(\(\\widehat\{P\}^\{\\widehat\{\\pi\},\\widehat\{W\}\}\-P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\)\\widehat\{W\}\\right\)\(s\)\\right\|=\|∑a∈𝒜π^\(a∣s\)\(P^s,aW^W^−Ps,aW^W^\)\|\\displaystyle\\qquad=\\left\|\\sum\_\{a\\in\\mathcal\{A\}\}\\widehat\{\\pi\}\(a\\mid s\)\\left\(\\widehat\{P\}\_\{s,a\}^\{\\widehat\{W\}\}\\widehat\{W\}\-P\_\{s,a\}^\{\\widehat\{W\}\}\\widehat\{W\}\\right\)\\right\|≤∑a∈𝒜π^​\(a∣s\)​\|P^s,aW^​W^−Ps,aW^​W^\|\\displaystyle\\qquad\\leq\\sum\_\{a\\in\\mathcal\{A\}\}\\widehat\{\\pi\}\(a\\mid s\)\\left\|\\widehat\{P\}\_\{s,a\}^\{\\widehat\{W\}\}\\widehat\{W\}\-P\_\{s,a\}^\{\\widehat\{W\}\}\\widehat\{W\}\\right\|≤C​ιN​∑a∈𝒜π^​\(a∣s\)​VarPs,a0​\(V^⋆\)\+C​R⋅ιN\+C​εopt,\\displaystyle\\qquad\\leq C\\sqrt\{\\frac\{\\iota\}\{N\}\}\\sum\_\{a\\in\\mathcal\{A\}\}\\widehat\{\\pi\}\(a\\mid s\)\\sqrt\{\\mathrm\{Var\}\_\{P^\{0\}\_\{s,a\}\}\(\\widehat\{V\}^\{\\star\}\)\}\+C\\frac\{R\\cdot\\iota\}\{N\}\+C\\varepsilon\_\{\\mathrm\{opt\}\},where the last two terms remain unchanged because∑a∈𝒜π^​\(a∣s\)=1\\sum\_\{a\\in\\mathcal\{A\}\}\\widehat\{\\pi\}\(a\\mid s\)=1\.

By Jensen’s inequality,

∑a∈𝒜π^​\(a∣s\)​VarPs,a0​\(V^⋆\)≤∑a∈𝒜π^​\(a∣s\)​VarPs,a0​\(V^⋆\)\.\\sum\_\{a\\in\\mathcal\{A\}\}\\widehat\{\\pi\}\(a\\mid s\)\\sqrt\{\\mathrm\{Var\}\_\{P^\{0\}\_\{s,a\}\}\(\\widehat\{V\}^\{\\star\}\)\}\\leq\\sqrt\{\\sum\_\{a\\in\\mathcal\{A\}\}\\widehat\{\\pi\}\(a\\mid s\)\\mathrm\{Var\}\_\{P^\{0\}\_\{s,a\}\}\(\\widehat\{V\}^\{\\star\}\)\}\.Moreover, the law of total variance and the definition ofP0,π^P^\{0,\\widehat\{\\pi\}\}yield

VarP0,π^​\(V^⋆\)​\(s\)\\displaystyle\\mathrm\{Var\}\_\{P^\{0,\\widehat\{\\pi\}\}\}\(\\widehat\{V\}^\{\\star\}\)\(s\)=∑a∈𝒜π^​\(a∣s\)​VarPs,a0​\(V^⋆\)\\displaystyle=\{\}\\sum\_\{a\\in\\mathcal\{A\}\}\\widehat\{\\pi\}\(a\\mid s\)\\mathrm\{Var\}\_\{P^\{0\}\_\{s,a\}\}\(\\widehat\{V\}^\{\\star\}\)\+∑a∈𝒜π^​\(a∣s\)​\(Ps,a0​V^⋆−P0,π^​V^⋆​\(s\)\)2\\displaystyle\\quad\+\\sum\_\{a\\in\\mathcal\{A\}\}\\widehat\{\\pi\}\(a\\mid s\)\\left\(P^\{0\}\_\{s,a\}\\widehat\{V\}^\{\\star\}\-P^\{0,\\widehat\{\\pi\}\}\\widehat\{V\}^\{\\star\}\(s\)\\right\)^\{2\}≥∑a∈𝒜π^​\(a∣s\)​VarPs,a0​\(V^⋆\)\.\\displaystyle\\geq\\sum\_\{a\\in\\mathcal\{A\}\}\\widehat\{\\pi\}\(a\\mid s\)\\mathrm\{Var\}\_\{P^\{0\}\_\{s,a\}\}\(\\widehat\{V\}^\{\\star\}\)\.Combining the preceding two displays leads to

∑a∈𝒜π^​\(a∣s\)​VarPs,a0​\(V^⋆\)≤VarP0,π^​\(V^⋆\)​\(s\)\.\\sum\_\{a\\in\\mathcal\{A\}\}\\widehat\{\\pi\}\(a\\mid s\)\\sqrt\{\\mathrm\{Var\}\_\{P^\{0\}\_\{s,a\}\}\(\\widehat\{V\}^\{\\star\}\)\}\\leq\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\widehat\{\\pi\}\}\}\(\\widehat\{V\}^\{\\star\}\)\(s\)\}\.Substituting this inequality into the averaged rowwise estimate produces

\|\(\(P^π^,W^−Pπ^,W^\)​W^\)​\(s\)\|≤C​ιN​VarP0,π^​\(V^⋆\)​\(s\)\+C​R⋅ιN\+C​εopt\.\\left\|\\left\(\(\\widehat\{P\}^\{\\widehat\{\\pi\},\\widehat\{W\}\}\-P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\)\\widehat\{W\}\\right\)\(s\)\\right\|\\leq C\\sqrt\{\\frac\{\\iota\}\{N\}\}\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\widehat\{\\pi\}\}\}\(\\widehat\{V\}^\{\\star\}\)\(s\)\}\+C\\frac\{R\\cdot\\iota\}\{N\}\+C\\varepsilon\_\{\\mathrm\{opt\}\}\.This is \([110](https://arxiv.org/html/2608.06545#A4.E110)\), completing the proof\.

#### D\.5\.12Proof of Lemma[23](https://arxiv.org/html/2608.06545#Thmlemma23)

The proof of \([23](https://arxiv.org/html/2608.06545#A4.Ex301)\) follows the proof of Lemma[17](https://arxiv.org/html/2608.06545#Thmlemma17)after interchanging the true and empirical models and making the substitutionsU↦W^U\\mapsto\\widehat\{W\},U^↦W\\widehat\{U\}\\mapsto W, andπγ⋆↦π^\\pi\_\{\\gamma\}^\{\\star\}\\mapsto\\widehat\{\\pi\}\. The order\-preserving resolvent rearrangement is deterministic, so the data dependence ofπ^\\widehat\{\\pi\}is immaterial\. We omit the details for brevity\.

#### D\.5\.13Proof of Lemma[24](https://arxiv.org/html/2608.06545#Thmlemma24)

The proof has three steps\. We first apply Lemma[18](https://arxiv.org/html/2608.06545#Thmlemma18)to reduce the bound onT2′T\_\{2\}^\{\\prime\}to a transition\-kernel error\. We then bound this error using Lemma[22](https://arxiv.org/html/2608.06545#Thmlemma22)and solve the resulting self\-bounding inequality\. Finally, we substitute the resulting bound into the reduction from the first step\.

##### Step 1: Reduction ofT2′T\_\{2\}^\{\\prime\}to a transition\-kernel error\.

In this step, we verify the premises of Lemma[18](https://arxiv.org/html/2608.06545#Thmlemma18)and apply it to isolate the transition\-kernel error that remains to be bounded\. For this application, set

π=π^,V=W^,P=Pπ^,W^,rπ=rπ^\.\\pi=\\widehat\{\\pi\},\\qquad V=\\widehat\{W\},\\qquad P=P^\{\\widehat\{\\pi\},\\widehat\{W\}\},\\qquad r^\{\\pi\}=r^\{\\widehat\{\\pi\}\}\.Recall from \([87](https://arxiv.org/html/2608.06545#A4.E87)\) that𝒢Q≔\(1−γ\)​\(I−γ​Q\)−1\\mathcal\{G\}\_\{Q\}\\coloneqq\(1\-\\gamma\)\(I\-\\gamma Q\)^\{\-1\}for any stochastic kernelQQ\. Define

ξV\\displaystyle\\xi\_\{V\}≔\(P^π^,W^−Pπ^,W^\)​W^,\\displaystyle\\coloneqq\\left\(\\widehat\{P\}^\{\\widehat\{\\pi\},\\widehat\{W\}\}\-P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\\right\)\\widehat\{W\},\(151\)ζW^\\displaystyle\\zeta\_\{\\widehat\{W\}\}≔‖𝒢Pπ^,W^​\|ξV\|‖∞\.\\displaystyle\\coloneqq\\left\\\|\\mathcal\{G\}\_\{P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\}\|\\xi\_\{V\}\|\\right\\\|\_\{\\infty\}\.Rewriting the empirical Bellman equation under the true robust kernelPπ^,W^P^\{\\widehat\{\\pi\},\\widehat\{W\}\}yields

W^=rπ^\+γ​Pπ^,W^​W^\+γ​ξV\.\\widehat\{W\}=r^\{\\widehat\{\\pi\}\}\+\\gamma P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\\widehat\{W\}\+\\gamma\\xi\_\{V\}\.The span premise of Lemma[18](https://arxiv.org/html/2608.06545#Thmlemma18)follows from \([70](https://arxiv.org/html/2608.06545#A4.E70)\), while\(1−γ\)​mins⁡W^​\(s\)≤1\(1\-\\gamma\)\\min\_\{s\}\\widehat\{W\}\(s\)\\leq 1follows from \([73](https://arxiv.org/html/2608.06545#A4.E73)\)\. The associated anchor defect isβemp\\beta\_\{\\mathrm\{emp\}\}by \([62](https://arxiv.org/html/2608.06545#A4.E62)\)\.

Moreover, convexity of total variation and the true robust\-set constraint imply, for every statess,

‖Pπ^,W^​\(s,⋅\)−P0,π^​\(s,⋅\)‖TV≤∑a∈𝒜π^​\(a∣s\)​‖Ps,aW^−Ps,a0‖TV≤σ\.\\left\\\|P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\(s,\\cdot\)\-P^\{0,\\widehat\{\\pi\}\}\(s,\\cdot\)\\right\\\|\_\{\\mathrm\{TV\}\}\\leq\\sum\_\{a\\in\\mathcal\{A\}\}\\widehat\{\\pi\}\(a\\mid s\)\\left\\\|P\_\{s,a\}^\{\\widehat\{W\}\}\-P^\{0\}\_\{s,a\}\\right\\\|\_\{\\mathrm\{TV\}\}\\leq\\sigma\.Hence

\|\(Pπ^,W^−P0,π^\)​h¯\|≤C​σ​Hanc⋅𝟏S,\\left\|\\left\(P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\-P^\{0,\\widehat\{\\pi\}\}\\right\)\\bar\{h\}\\right\|\\leq C\\sigma H\_\{\\mathrm\{anc\}\}\\cdot\\bm\{1\}\_\{S\},so \([103](https://arxiv.org/html/2608.06545#A4.E103)\) holds withξh=0\\xi\_\{h\}=0\. Becauseγ≤1\\gamma\\leq 1,\|b\|≤\|ξV\|\|b\|\\leq\|\\xi\_\{V\}\|forb=γ​ξVb=\\gamma\\xi\_\{V\}\. Applying Lemma[18](https://arxiv.org/html/2608.06545#Thmlemma18)and using the definition ofζW^\\zeta\_\{\\widehat\{W\}\}, we obtain

‖\(I−γ​Pπ^,W^\)−1​VarPπ^,W^​\(W^\)‖∞≤C1−γ​Bemp​\(R\)\+R​ζW^\.\\left\\\|\\left\(I\-\\gamma P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\\right\)^\{\-1\}\\sqrt\{\\mathrm\{Var\}\_\{P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\}\(\\widehat\{W\}\)\}\\right\\\|\_\{\\infty\}\\leq\\frac\{C\}\{1\-\\gamma\}\\sqrt\{B\_\{\\mathrm\{emp\}\}\(R\)\+R\\zeta\_\{\\widehat\{W\}\}\}\.\(152\)Combining this bound with the definition ofT2′T\_\{2\}^\{\\prime\}in \([112](https://arxiv.org/html/2608.06545#A4.E112)\) yields

‖T2′‖∞≤C1−γ​ιN​Bemp​\(R\)\+R​ζW^\.\\\|T\_\{2\}^\{\\prime\}\\\|\_\{\\infty\}\\leq\\frac\{C\}\{1\-\\gamma\}\\sqrt\{\\frac\{\\iota\}\{N\}\}\\sqrt\{B\_\{\\mathrm\{emp\}\}\(R\)\+R\\zeta\_\{\\widehat\{W\}\}\}\.\(153\)Thus, it remains to controlζW^\\zeta\_\{\\widehat\{W\}\}\.

##### Step 2: BoundingζW^\\zeta\_\{\\widehat\{W\}\}\.

In this step, we use Lemma[22](https://arxiv.org/html/2608.06545#Thmlemma22)and the bound from Step 1 to derive and solve a self\-bounding inequality forζW^\\zeta\_\{\\widehat\{W\}\}\. Optimality and the solver guarantee \([5](https://arxiv.org/html/2608.06545#S4.E5)\) ensure that0≤V^⋆−W^≤εopt⋅𝟏S0\\leq\\widehat\{V\}^\{\\star\}\-\\widehat\{W\}\\leq\\varepsilon\_\{\\mathrm\{opt\}\}\\cdot\\bm\{1\}\_\{S\}\. Hence, for every statess, Minkowski’s inequality yields

VarP0,π^​\(V^⋆\)​\(s\)\\displaystyle\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\widehat\{\\pi\}\}\}\(\\widehat\{V\}^\{\\star\}\)\(s\)\}≤VarP0,π^​\(W^\)​\(s\)\+VarP0,π^​\(V^⋆−W^\)​\(s\)\\displaystyle\\leq\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\widehat\{\\pi\}\}\}\(\\widehat\{W\}\)\(s\)\}\+\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\widehat\{\\pi\}\}\}\(\\widehat\{V\}^\{\\star\}\-\\widehat\{W\}\)\(s\)\}\(154\)≤VarP0,π^​\(W^\)​\(s\)\+εopt\.\\displaystyle\\leq\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\widehat\{\\pi\}\}\}\(\\widehat\{W\}\)\(s\)\}\+\\varepsilon\_\{\\mathrm\{opt\}\}\.Apply the nonnegative matrix𝒢Pπ^,W^\\mathcal\{G\}\_\{P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\}to \([110](https://arxiv.org/html/2608.06545#A4.E110)\)\. Using the preceding display,𝒢Pπ^,W^​𝟏S=𝟏S\\mathcal\{G\}\_\{P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\}\\bm\{1\}\_\{S\}=\\bm\{1\}\_\{S\}, andN≥ιN\\geq\\iota, which follows from the first condition in \([71](https://arxiv.org/html/2608.06545#A4.E71)\), we obtain

ζW^≤C​ιN​‖𝒢Pπ^,W^​VarP0,π^​\(W^\)‖∞\+C​R⋅ιN\+C​εopt\.\\zeta\_\{\\widehat\{W\}\}\\leq C\\sqrt\{\\frac\{\\iota\}\{N\}\}\\left\\\|\\mathcal\{G\}\_\{P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\}\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\widehat\{\\pi\}\}\}\(\\widehat\{W\}\)\}\\right\\\|\_\{\\infty\}\+C\\frac\{R\\cdot\\iota\}\{N\}\+C\\varepsilon\_\{\\mathrm\{opt\}\}\.\(155\)
BecausePπ^,W^P^\{\\widehat\{\\pi\},\\widehat\{W\}\}lies in the true robust uncertainty set, Lemma[26](https://arxiv.org/html/2608.06545#Thmlemma26)and \([70](https://arxiv.org/html/2608.06545#A4.E70)\) imply

VarP0,π^​\(W^\)≤VarPπ^,W^​\(W^\)\+C​R​σ⋅𝟏S\.\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\widehat\{\\pi\}\}\}\(\\widehat\{W\}\)\}\\leq\\sqrt\{\\mathrm\{Var\}\_\{P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\}\(\\widehat\{W\}\)\}\+CR\\sqrt\{\\sigma\}\\cdot\\bm\{1\}\_\{S\}\.\(156\)This deterministic inequality applies even thoughW^\\widehat\{W\}is data\-dependent\. Applying𝒢Pπ^,W^\\mathcal\{G\}\_\{P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\}to \([156](https://arxiv.org/html/2608.06545#A4.E156)\) and using \([152](https://arxiv.org/html/2608.06545#A4.E152)\), we obtain

‖𝒢Pπ^,W^​VarP0,π^​\(W^\)‖∞≤C​Bemp​\(R\)\+R​ζW^\+C​R​σ\.\\displaystyle\\left\\\|\\mathcal\{G\}\_\{P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\}\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\widehat\{\\pi\}\}\}\(\\widehat\{W\}\)\}\\right\\\|\_\{\\infty\}\\leq C\\sqrt\{B\_\{\\mathrm\{emp\}\}\(R\)\+R\\zeta\_\{\\widehat\{W\}\}\}\+CR\\sqrt\{\\sigma\}\.SinceBemp​\(R\)≥σ​R2B\_\{\\mathrm\{emp\}\}\(R\)\\geq\\sigma R^\{2\}, the final term can be absorbed into the square root\. Therefore,

‖𝒢Pπ^,W^​VarP0,π^​\(W^\)‖∞≤C​Bemp​\(R\)\+R​ζW^\.\\left\\\|\\mathcal\{G\}\_\{P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\}\\sqrt\{\\mathrm\{Var\}\_\{P^\{0,\\widehat\{\\pi\}\}\}\(\\widehat\{W\}\)\}\\right\\\|\_\{\\infty\}\\leq C\\sqrt\{B\_\{\\mathrm\{emp\}\}\(R\)\+R\\zeta\_\{\\widehat\{W\}\}\}\.\(157\)Together, \([155](https://arxiv.org/html/2608.06545#A4.E155)\) and \([157](https://arxiv.org/html/2608.06545#A4.E157)\) imply

ζW^≤C​ιN​Bemp​\(R\)\+R​ζW^\+C​R⋅ιN\+C​εopt\.\\zeta\_\{\\widehat\{W\}\}\\leq C\\sqrt\{\\frac\{\\iota\}\{N\}\}\\sqrt\{B\_\{\\mathrm\{emp\}\}\(R\)\+R\\zeta\_\{\\widehat\{W\}\}\}\+C\\frac\{R\\cdot\\iota\}\{N\}\+C\\varepsilon\_\{\\mathrm\{opt\}\}\.\(158\)To solve \([158](https://arxiv.org/html/2608.06545#A4.E158)\),x\+y≤x\+y\\sqrt\{x\+y\}\\leq\\sqrt\{x\}\+\\sqrt\{y\}and Young’s inequality yield

C​ιN​Bemp​\(R\)\+R​ζW^\\displaystyle C\\sqrt\{\\frac\{\\iota\}\{N\}\}\\sqrt\{B\_\{\\mathrm\{emp\}\}\(R\)\+R\\zeta\_\{\\widehat\{W\}\}\}≤C​Bemp​\(R\)⋅ιN\+C​R⋅ιN​ζW^\\displaystyle\\leq C\\sqrt\{\\frac\{B\_\{\\mathrm\{emp\}\}\(R\)\\cdot\\iota\}\{N\}\}\+C\\sqrt\{\\frac\{R\\cdot\\iota\}\{N\}\}\\sqrt\{\\zeta\_\{\\widehat\{W\}\}\}\(159\)≤C​Bemp​\(R\)⋅ιN\+12​ζW^\+C​R⋅ιN\.\\displaystyle\\leq C\\sqrt\{\\frac\{B\_\{\\mathrm\{emp\}\}\(R\)\\cdot\\iota\}\{N\}\}\+\\frac\{1\}\{2\}\\zeta\_\{\\widehat\{W\}\}\+C\\frac\{R\\cdot\\iota\}\{N\}\.Substituting \([159](https://arxiv.org/html/2608.06545#A4.E159)\) into \([158](https://arxiv.org/html/2608.06545#A4.E158)\) results in

ζW^≤C​Bemp​\(R\)⋅ιN\+12​ζW^\+C​R⋅ιN\+C​εopt\.\\zeta\_\{\\widehat\{W\}\}\\leq C\\sqrt\{\\frac\{B\_\{\\mathrm\{emp\}\}\(R\)\\cdot\\iota\}\{N\}\}\+\\frac\{1\}\{2\}\\zeta\_\{\\widehat\{W\}\}\+C\\frac\{R\\cdot\\iota\}\{N\}\+C\\varepsilon\_\{\\mathrm\{opt\}\}\.Rearranging establishes

ζW^≤C​Bemp​\(R\)⋅ιN\+C​R⋅ιN\+C​εopt\.\\zeta\_\{\\widehat\{W\}\}\\leq C\\sqrt\{\\frac\{B\_\{\\mathrm\{emp\}\}\(R\)\\cdot\\iota\}\{N\}\}\+C\\frac\{R\\cdot\\iota\}\{N\}\+C\\varepsilon\_\{\\mathrm\{opt\}\}\.\(160\)

##### Step 3: Final substitution\.

Finally, we use the bound from Step 2 to simplify the right\-hand side of \([153](https://arxiv.org/html/2608.06545#A4.E153)\)\. Young’s inequality and \([160](https://arxiv.org/html/2608.06545#A4.E160)\) show that

ιN​Bemp​\(R\)\+R​ζW^\\displaystyle\\sqrt\{\\frac\{\\iota\}\{N\}\}\\sqrt\{B\_\{\\mathrm\{emp\}\}\(R\)\+R\\zeta\_\{\\widehat\{W\}\}\}≤Bemp​\(R\)⋅ιN\+12​ζW^\+C​R⋅ιN\\displaystyle\\leq\\sqrt\{\\frac\{B\_\{\\mathrm\{emp\}\}\(R\)\\cdot\\iota\}\{N\}\}\+\\frac\{1\}\{2\}\\zeta\_\{\\widehat\{W\}\}\+C\\frac\{R\\cdot\\iota\}\{N\}≤C​Bemp​\(R\)⋅ιN\+C​R⋅ιN\+C​εopt\.\\displaystyle\\leq C\\sqrt\{\\frac\{B\_\{\\mathrm\{emp\}\}\(R\)\\cdot\\iota\}\{N\}\}\+C\\frac\{R\\cdot\\iota\}\{N\}\+C\\varepsilon\_\{\\mathrm\{opt\}\}\.Substituting the last display into \([153](https://arxiv.org/html/2608.06545#A4.E153)\) establishes \([116](https://arxiv.org/html/2608.06545#A4.E116)\), completing the proof\.

#### D\.5\.14Proof of Lemma[25](https://arxiv.org/html/2608.06545#Thmlemma25)

The proof again has three steps\. We first apply the anchored resolvent\-variance bound under the mismatched kernelPπ^,WP^\{\\widehat\{\\pi\},W\}and isolate the selector\-mismatch cost\. We then close the empirical Bellman residual under this resolvent\. Finally, we transfer the variance from the kernel selected atWWto the kernel selected atW^\\widehat\{W\}and absorb the resulting mismatch term\.

##### Step 1: Reduction under the mismatched resolvent\.

In this step, we rewrite the empirical Bellman equation underPπ^,WP^\{\\widehat\{\\pi\},W\}and quantify the cost of replacing the selectorPπ^,W^P^\{\\widehat\{\\pi\},\\widehat\{W\}\}byPπ^,WP^\{\\widehat\{\\pi\},W\}\. Define

ζmis≔‖𝒢Pπ^,W​\|\(P^π^,W^−Pπ^,W^\)​W^\|‖∞\.\\zeta\_\{\\mathrm\{mis\}\}\\coloneqq\\left\\\|\\mathcal\{G\}\_\{P^\{\\widehat\{\\pi\},W\}\}\\left\|\\left\(\\widehat\{P\}^\{\\widehat\{\\pi\},\\widehat\{W\}\}\-P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\\right\)\\widehat\{W\}\\right\|\\right\\\|\_\{\\infty\}\.For each\(s,a\)\(s,a\), optimality of the two true robust selectors implies

\(Ps,aW^−Ps,aW\)​W^≤0,\(Ps,aW^−Ps,aW\)​W≥0\.\\left\(P\_\{s,a\}^\{\\widehat\{W\}\}\-P\_\{s,a\}^\{W\}\\right\)\\widehat\{W\}\\leq 0,\\qquad\\left\(P\_\{s,a\}^\{\\widehat\{W\}\}\-P\_\{s,a\}^\{W\}\\right\)W\\geq 0\.Consequently,

0\\displaystyle 0≤−\(Ps,aW^−Ps,aW\)​W^\\displaystyle\\leq\-\\left\(P\_\{s,a\}^\{\\widehat\{W\}\}\-P\_\{s,a\}^\{W\}\\right\)\\widehat\{W\}≤\|\(Ps,aW^−Ps,aW\)​\(W^−W\)\|\\displaystyle\\leq\\left\|\\left\(P\_\{s,a\}^\{\\widehat\{W\}\}\-P\_\{s,a\}^\{W\}\\right\)\(\\widehat\{W\}\-W\)\\right\|≤‖Ps,aW^−Ps,aW‖1​‖W^−W‖∞\.\\displaystyle\\leq\\left\\\|P\_\{s,a\}^\{\\widehat\{W\}\}\-P\_\{s,a\}^\{W\}\\right\\\|\_\{1\}\\\|\\widehat\{W\}\-W\\\|\_\{\\infty\}\.Both selected rows belong to𝒰s,a​\(P0\)\\mathcal\{U\}\_\{s,a\}\(P^\{0\}\), so theirℓ1\\ell\_\{1\}\-distance is at most4​σ4\\sigma\. Averaging overa∼π^\(⋅∣s\)a\\sim\\widehat\{\\pi\}\(\\cdot\\mid s\)therefore leads to

‖\(Pπ^,W^−Pπ^,W\)​W^‖∞≤C​σ​‖W^−W‖∞\.\\left\\\|\\left\(P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\-P^\{\\widehat\{\\pi\},W\}\\right\)\\widehat\{W\}\\right\\\|\_\{\\infty\}\\leq C\\sigma\\\|\\widehat\{W\}\-W\\\|\_\{\\infty\}\.\(161\)
The empirical Bellman equation can now be written as

W^\\displaystyle\\widehat\{W\}=rπ^\+γ​Pπ^,W​W^\\displaystyle=r^\{\\widehat\{\\pi\}\}\+\\gamma P^\{\\widehat\{\\pi\},W\}\\widehat\{W\}\+γ​\(P^π^,W^−Pπ^,W^\)​W^\+γ​\(Pπ^,W^−Pπ^,W\)​W^\.\\displaystyle\\quad\+\\gamma\\left\(\\widehat\{P\}^\{\\widehat\{\\pi\},\\widehat\{W\}\}\-P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\\right\)\\widehat\{W\}\+\\gamma\\left\(P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\-P^\{\\widehat\{\\pi\},W\}\\right\)\\widehat\{W\}\.As in Step 1 of the proof of Lemma[24](https://arxiv.org/html/2608.06545#Thmlemma24), the span, minimum\-value, and anchor defect premises of Lemma[18](https://arxiv.org/html/2608.06545#Thmlemma18)follow from \([70](https://arxiv.org/html/2608.06545#A4.E70)\), \([73](https://arxiv.org/html/2608.06545#A4.E73)\), and \([62](https://arxiv.org/html/2608.06545#A4.E62)\), respectively\. SincePπ^,WP^\{\\widehat\{\\pi\},W\}belongs to the true robust uncertainty set, \([103](https://arxiv.org/html/2608.06545#A4.E103)\) holds withξh=0\\xi\_\{h\}=0\. For the value residual in the displayed Bellman equation,γ≤1\\gamma\\leq 1allows us to take

ξV=\(P^π^,W^−Pπ^,W^\)​W^\+\(Pπ^,W^−Pπ^,W\)​W^\\xi\_\{V\}=\\left\(\\widehat\{P\}^\{\\widehat\{\\pi\},\\widehat\{W\}\}\-P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\\right\)\\widehat\{W\}\+\\left\(P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\-P^\{\\widehat\{\\pi\},W\}\\right\)\\widehat\{W\}in \([103](https://arxiv.org/html/2608.06545#A4.E103)\)\. The triangle inequality, \([161](https://arxiv.org/html/2608.06545#A4.E161)\), and𝒢Pπ^,W​𝟏S=𝟏S\\mathcal\{G\}\_\{P^\{\\widehat\{\\pi\},W\}\}\\bm\{1\}\_\{S\}=\\bm\{1\}\_\{S\}imply

‖𝒢Pπ^,W​\|ξV\|‖∞≤ζmis\+C​σ​‖W^−W‖∞\.\\left\\\|\\mathcal\{G\}\_\{P^\{\\widehat\{\\pi\},W\}\}\|\\xi\_\{V\}\|\\right\\\|\_\{\\infty\}\\leq\\zeta\_\{\\mathrm\{mis\}\}\+C\\sigma\\\|\\widehat\{W\}\-W\\\|\_\{\\infty\}\.Lemma[18](https://arxiv.org/html/2608.06545#Thmlemma18)therefore yields

‖\(I−γ​Pπ^,W\)−1​VarPπ^,W​\(W^\)‖∞\\displaystyle\\left\\\|\\left\(I\-\\gamma P^\{\\widehat\{\\pi\},W\}\\right\)^\{\-1\}\\sqrt\{\\mathrm\{Var\}\_\{P^\{\\widehat\{\\pi\},W\}\}\(\\widehat\{W\}\)\}\\right\\\|\_\{\\infty\}\(162\)≤C1−γ​Bemp​\(R\)\+R​ζmis\+σ​R​‖W^−W‖∞\.\\displaystyle\\qquad\\leq\\frac\{C\}\{1\-\\gamma\}\\sqrt\{B\_\{\\mathrm\{emp\}\}\(R\)\+R\\zeta\_\{\\mathrm\{mis\}\}\+\\sigma R\\\|\\widehat\{W\}\-W\\\|\_\{\\infty\}\}\.Thus, relative to the matched reduction \([152](https://arxiv.org/html/2608.06545#A4.E152)\), the only additional term isσ​R​‖W^−W‖∞\\sigma R\\\|\\widehat\{W\}\-W\\\|\_\{\\infty\}\.

##### Step 2: Closing the empirical Bellman residual\.

We have

ζmis≤C​\(Bemp​\(R\)\+σ​R​‖W^−W‖∞\)⋅ιN\+C​R⋅ιN\+C​εopt\.\\zeta\_\{\\mathrm\{mis\}\}\\leq C\\sqrt\{\\frac\{\\left\(B\_\{\\mathrm\{emp\}\}\(R\)\+\\sigma R\\\|\\widehat\{W\}\-W\\\|\_\{\\infty\}\\right\)\\cdot\\iota\}\{N\}\}\+C\\frac\{R\\cdot\\iota\}\{N\}\+C\\varepsilon\_\{\\mathrm\{opt\}\}\.\(163\)The proof of \([163](https://arxiv.org/html/2608.06545#A4.E163)\) follows Step 2 of the proof of Lemma[24](https://arxiv.org/html/2608.06545#Thmlemma24), with the additional termσ​R​‖W^−W‖∞\\sigma R\\\|\\widehat\{W\}\-W\\\|\_\{\\infty\}arising from the selector\-mismatch term in \([162](https://arxiv.org/html/2608.06545#A4.E162)\)\. We omit the details for brevity\.

##### Step 3: Substitution and absorption\.

In this step, we transfer the variance inT6′T\_\{6\}^\{\\prime\}to the kernel analyzed in Step 1, substitute the residual bound from Step 2, and absorb the remaining selector\-mismatch term\. BothPπ^,W^P^\{\\widehat\{\\pi\},\\widehat\{W\}\}andPπ^,WP^\{\\widehat\{\\pi\},W\}are within total variation distanceσ\\sigmaofP0,π^P^\{0,\\widehat\{\\pi\}\}, and hence

‖Pπ^,W^​\(s,⋅\)−Pπ^,W​\(s,⋅\)‖TV≤2​σ,s∈𝒮\.\\left\\\|P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\(s,\\cdot\)\-P^\{\\widehat\{\\pi\},W\}\(s,\\cdot\)\\right\\\|\_\{\\mathrm\{TV\}\}\\leq 2\\sigma,\\qquad s\\in\\mathcal\{S\}\.Therefore, Lemma[26](https://arxiv.org/html/2608.06545#Thmlemma26)and \([70](https://arxiv.org/html/2608.06545#A4.E70)\) imply

VarPπ^,W^​\(W^\)≤VarPπ^,W​\(W^\)\+C​R​σ⋅𝟏S\.\\sqrt\{\\mathrm\{Var\}\_\{P^\{\\widehat\{\\pi\},\\widehat\{W\}\}\}\(\\widehat\{W\}\)\}\\leq\\sqrt\{\\mathrm\{Var\}\_\{P^\{\\widehat\{\\pi\},W\}\}\(\\widehat\{W\}\)\}\+CR\\sqrt\{\\sigma\}\\cdot\\bm\{1\}\_\{S\}\.\(164\)Using \([164](https://arxiv.org/html/2608.06545#A4.E164)\) in the definition ofT6′T\_\{6\}^\{\\prime\}in \([113](https://arxiv.org/html/2608.06545#A4.E113)\) and then applying \([162](https://arxiv.org/html/2608.06545#A4.E162)\), we obtain

‖T6′‖∞\\displaystyle\\\|T\_\{6\}^\{\\prime\}\\\|\_\{\\infty\}≤C1−γ​ιN​Bemp​\(R\)\+R​ζmis\+σ​R​‖W^−W‖∞\\displaystyle\\leq\\frac\{C\}\{1\-\\gamma\}\\sqrt\{\\frac\{\\iota\}\{N\}\}\\sqrt\{B\_\{\\mathrm\{emp\}\}\(R\)\+R\\zeta\_\{\\mathrm\{mis\}\}\+\\sigma R\\\|\\widehat\{W\}\-W\\\|\_\{\\infty\}\}\(165\)\+C​R​σ⋅ι/N1−γ\.\\displaystyle\\quad\+C\\frac\{R\\sqrt\{\\sigma\\cdot\\iota/N\}\}\{1\-\\gamma\}\.SinceBemp​\(R\)≥σ​R2B\_\{\\mathrm\{emp\}\}\(R\)\\geq\\sigma R^\{2\}, the additive variance\-transfer term in \([165](https://arxiv.org/html/2608.06545#A4.E165)\) is absorbed into itsBemp​\(R\)B\_\{\\mathrm\{emp\}\}\(R\)contribution\. Splitting the remaining square root, applying Young’s inequality to itsR​ζmisR\\zeta\_\{\\mathrm\{mis\}\}contribution, and then using \([163](https://arxiv.org/html/2608.06545#A4.E163)\), we obtain

‖T6′‖∞\\displaystyle\\\|T\_\{6\}^\{\\prime\}\\\|\_\{\\infty\}≤C​Bemp​\(R\)⋅ιN​\(1−γ\)2\+C​R⋅ιN​\(1−γ\)\\displaystyle\\leq C\\sqrt\{\\frac\{B\_\{\\mathrm\{emp\}\}\(R\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\+C\\frac\{R\\cdot\\iota\}\{N\(1\-\\gamma\)\}\(166\)\+C​σ​R​‖W^−W‖∞⋅ιN​\(1−γ\)2\+C​εopt1−γ\.\\displaystyle\\quad\+C\\sqrt\{\\frac\{\\sigma R\\\|\\widehat\{W\}\-W\\\|\_\{\\infty\}\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\+C\\frac\{\\varepsilon\_\{\\mathrm\{opt\}\}\}\{1\-\\gamma\}\.Young’s inequality bounds the remaining selector\-mismatch contribution by

C​σ​R​‖W^−W‖∞⋅ιN​\(1−γ\)2≤120​‖W^−W‖∞\+C​σ​R⋅ιN​\(1−γ\)2\.C\\sqrt\{\\frac\{\\sigma R\\\|\\widehat\{W\}\-W\\\|\_\{\\infty\}\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\\leq\\frac\{1\}\{20\}\\\|\\widehat\{W\}\-W\\\|\_\{\\infty\}\+C\\frac\{\\sigma R\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\.\(167\)Moreover,

σ​R⋅ιN​\(1−γ\)2=σ⋅ι/N1−γ​σ​R2⋅ιN​\(1−γ\)2≤C​Bemp​\(R\)⋅ιN​\(1−γ\)2,\\frac\{\\sigma R\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}=\\frac\{\\sqrt\{\\sigma\\cdot\\iota/N\}\}\{1\-\\gamma\}\\sqrt\{\\frac\{\\sigma R^\{2\}\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\}\\leq C\\sqrt\{\\frac\{B\_\{\\mathrm\{emp\}\}\(R\)\\cdot\\iota\}\{N\(1\-\\gamma\)^\{2\}\}\},\(168\)where the last inequality uses the second condition in \([71](https://arxiv.org/html/2608.06545#A4.E71)\) andBemp​\(R\)≥σ​R2B\_\{\\mathrm\{emp\}\}\(R\)\\geq\\sigma R^\{2\}\. Substituting \([167](https://arxiv.org/html/2608.06545#A4.E167)\) and \([168](https://arxiv.org/html/2608.06545#A4.E168)\) into \([166](https://arxiv.org/html/2608.06545#A4.E166)\) establishes \([117](https://arxiv.org/html/2608.06545#A4.E117)\), completing the proof\.

#### D\.5\.15Proof of Lemma[26](https://arxiv.org/html/2608.06545#Thmlemma26)

Fix probability vectorsq,q′q,q^\{\\prime\}and a vectorff\. Since the variance is the minimum mean squared deviation from a constant, evaluating the variance underqqaround the meanq′​fq^\{\\prime\}fgives

Varq​\(f\)\\displaystyle\\mathrm\{Var\}\_\{q\}\(f\)=minc∈ℝ⁡q​\[\(f−c\)2\]\\displaystyle=\\min\_\{c\\in\\mathbb\{R\}\}q\\\!\\left\[\(f\-c\)^\{2\}\\right\]≤q​\[\(f−q′​f\)2\]\\displaystyle\\leq q\\\!\\left\[\(f\-q^\{\\prime\}f\)^\{2\}\\right\]=q′​\[\(f−q′​f\)2\]\+\(q−q′\)​\[\(f−q′​f\)2\]\\displaystyle=q^\{\\prime\}\\\!\\left\[\(f\-q^\{\\prime\}f\)^\{2\}\\right\]\+\(q\-q^\{\\prime\}\)\\\!\\left\[\(f\-q^\{\\prime\}f\)^\{2\}\\right\]=Varq′​\(f\)\+\(q−q′\)​\[\(f−q′​f\)2\]\.\\displaystyle=\\mathrm\{Var\}\_\{q^\{\\prime\}\}\(f\)\+\(q\-q^\{\\prime\}\)\\\!\\left\[\(f\-q^\{\\prime\}f\)^\{2\}\\right\]\.Becauseq′​fq^\{\\prime\}fis a convex combination of the coordinates offf, it lies between their minimum and maximum\. Hence

0≤\(f−q′​f\)2≤‖f‖span2⋅𝟏S\.0\\leq\(f\-q^\{\\prime\}f\)^\{2\}\\leq\\\|f\\\|\_\{\\mathrm\{span\}\}^\{2\}\\cdot\\bm\{1\}\_\{S\}\.Applying this bound to the previous inequality yields

Varq​\(f\)≤Varq′​\(f\)\+C​‖f‖span2​‖q−q′‖TV\.\\mathrm\{Var\}\_\{q\}\(f\)\\leq\\mathrm\{Var\}\_\{q^\{\\prime\}\}\(f\)\+C\\\|f\\\|\_\{\\mathrm\{span\}\}^\{2\}\\\|q\-q^\{\\prime\}\\\|\_\{\\mathrm\{TV\}\}\.\(169\)Finally, applyingx\+y≤x\+y\\sqrt\{x\+y\}\\leq\\sqrt\{x\}\+\\sqrt\{y\}to \([169](https://arxiv.org/html/2608.06545#A4.E169)\) gives

Varq​\(f\)−Varq′​\(f\)\\displaystyle\\sqrt\{\\mathrm\{Var\}\_\{q\}\(f\)\}\-\\sqrt\{\\mathrm\{Var\}\_\{q^\{\\prime\}\}\(f\)\}≤C​‖f‖span​‖q−q′‖TV\.\\displaystyle\\leq C\\\|f\\\|\_\{\\mathrm\{span\}\}\\sqrt\{\\\|q\-q^\{\\prime\}\\\|\_\{\\mathrm\{TV\}\}\}\.Interchangingqqandq′q^\{\\prime\}gives the reverse comparison, thus establishing \([119](https://arxiv.org/html/2608.06545#A4.E119)\)\.

## Appendix EProofs for span\-agnostic horizon calibration

This appendix gives the full specification of Algorithm[2](https://arxiv.org/html/2608.06545#alg2)and proves Theorem[3](https://arxiv.org/html/2608.06545#Thmtheorem3)\. Appendix[E\.1](https://arxiv.org/html/2608.06545#A5.SS1)supplies the implementation details omitted from the main text, including the horizon grids, solver guarantees, anchor\-calibration procedure, and lower\-confidence penalties\. Appendix[E\.2](https://arxiv.org/html/2608.06545#A5.SS2)states the supporting lemmas for the anchor certificate and the nominal and robust policy candidates\.

Appendix[E\.3](https://arxiv.org/html/2608.06545#A5.SS3)combines these lemmas to prove the theorem\. It shows that every reported lower\-confidence bound is valid for its policy’s robust average reward and that, under either condition in the theorem, at least one candidate has a lower\-confidence bound of at leastρ⋆,σ−O​\(ε\)\\rho^\{\\star,\\sigma\}\-O\(\\varepsilon\)\. The policy with the largest bound is thereforeO​\(ε\)O\(\\varepsilon\)\-optimal\. Finally, Appendix[E\.4](https://arxiv.org/html/2608.06545#A5.SS4)proves the supporting lemmas\.

### E\.1Full specification of Algorithm[2](https://arxiv.org/html/2608.06545#alg2)

This subsection gives the implementable specification summarized by Algorithm[2](https://arxiv.org/html/2608.06545#alg2)\. We first define the policy grid, which consists of the discount factors used to generate the candidate policies, and the required solver guarantees\. We then specify the anchor calibration, the robust planning and evaluation calls, and the lower\-confidence penalties\. The final displays introduce the expanded penalty notation used in the proofs\.

Recall that the independent batches𝒟nom\\mathcal\{D\}\_\{\\mathrm\{nom\}\}and𝒟rob\\mathcal\{D\}\_\{\\mathrm\{rob\}\}containNnomN\_\{\\mathrm\{nom\}\}andNrobN\_\{\\mathrm\{rob\}\}samples per state\-action pair, respectively\. Their empirical nominal kernels areP^nom0\\widehat\{P\}\_\{\\mathrm\{nom\}\}^\{0\}andP^rob0\\widehat\{P\}\_\{\\mathrm\{rob\}\}^\{0\}\. All nominal and robust discounted solver calls below are governed by the common solver toleranceεopt≤copt​ε\\varepsilon\_\{\\mathrm\{opt\}\}\\leq c\_\{\\mathrm\{opt\}\}\\varepsilon, wherecoptc\_\{\\mathrm\{opt\}\}is the universal constant in Theorem[3](https://arxiv.org/html/2608.06545#Thmtheorem3)\. Their call\-specific discounted\-value guarantees are stated below\.

##### Logarithmic factor and policy grid\.

For the remainder of this appendix, define

ιN≔log⁡\(S​A​Nδ\)\.\\iota\_\{N\}\\coloneqq\\log\\\!\\left\(\\frac\{SAN\}\{\\delta\}\\right\)\.\(170\)The policy gridΓN\\Gamma\_\{N\}consists of discount factorsγ\\gammawith dyadic effective horizons\(1−γ\)−1=2k\(1\-\\gamma\)^\{\-1\}=2^\{k\}; each of these discount factors is used to generate candidate policies\. The grid excludes horizons that are too long for the robust batch\. For a sufficiently large universal constantCΓC\_\{\\Gamma\}, define

ΓN≔\{1−2−k:k∈\{1,2,…\},2k≤1CΓ​min⁡\{Nrobσ⋅ιN,NrobιN\}\},\\Gamma\_\{N\}\\coloneqq\\left\\\{1\-2^\{\-k\}:k\\in\\\{1,2,\\ldots\\\},\\ 2^\{k\}\\leq\\frac\{1\}\{C\_\{\\Gamma\}\}\\min\\left\\\{\\sqrt\{\\frac\{N\_\{\\mathrm\{rob\}\}\}\{\\sigma\\cdot\\iota\_\{N\}\}\},\\frac\{N\_\{\\mathrm\{rob\}\}\}\{\\iota\_\{N\}\}\\right\\\}\\right\\\},\(171\)Whenσ=0\\sigma=0, the first cutoff inΓN\\Gamma\_\{N\}is interpreted as\+∞\+\\infty\.

We useγ\\gammafor discount factors that generate candidate policies andλ\\lambdafor discount factors that generate anchor certificates\. When a definition applies to a discount factor in either role, we denote it byη\\eta\.

##### Nominal solver\.

For eachη∈\(0,1\)\\eta\\in\(0,1\)at which we call the nominal solver, letV^η⋆,0\\widehat\{V\}\_\{\\eta\}^\{\\star,0\}denote the exact optimal discounted value of the empirical nominal MDP with kernelP^nom0\\widehat\{P\}\_\{\\mathrm\{nom\}\}^\{0\}\. The solver returns a deterministic policyπ^η0\\widehat\{\\pi\}\_\{\\eta\}^\{0\}, which we evaluate exactly underP^nom0\\widehat\{P\}\_\{\\mathrm\{nom\}\}^\{0\}\. We denote its discounted value byV^ηπ^η0,0\\widehat\{V\}\_\{\\eta\}^\{\\widehat\{\\pi\}\_\{\\eta\}^\{0\},0\}and require

‖V^η⋆,0−V^ηπ^η0,0‖∞≤min⁡\{εopt1−η,1Nnom\}\.\\left\\\|\\widehat\{V\}\_\{\\eta\}^\{\\star,0\}\-\\widehat\{V\}\_\{\\eta\}^\{\\widehat\{\\pi\}\_\{\\eta\}^\{0\},0\}\\right\\\|\_\{\\infty\}\\leq\\min\\left\\\{\\frac\{\\varepsilon\_\{\\mathrm\{opt\}\}\}\{1\-\\eta\},\\frac\{1\}\{N\_\{\\mathrm\{nom\}\}\}\\right\\\}\.\(172\)The1/Nnom1/N\_\{\\mathrm\{nom\}\}requirement is a computational accuracy condition and does not require additional samples\. The solver is applied at eachγ∈ΓN\\gamma\\in\\Gamma\_\{N\}to obtain the nominal policy candidates\. It is also applied at the discount factors in the anchor grid defined below\. When a discount factor belongs to both grids, we reuse the same nominal solve\.

##### Anchor calibration\.

The following procedure uses only𝒟nom\\mathcal\{D\}\_\{\\mathrm\{nom\}\}\. It constructs a certificate at each anchor discount factor and selects one certificate to use for every robust policy candidate\.

Algorithm 3Nominal anchor calibrationInput\.The nominal\-batch empirical kernelP^nom0\\widehat\{P\}\_\{\\mathrm\{nom\}\}^\{0\};NN,NnomN\_\{\\mathrm\{nom\}\},ε\\varepsilon,δ\\delta, and the solver toleranceεopt\\varepsilon\_\{\\mathrm\{opt\}\}\.

1. 1\.Construct the anchor grid ΛN≔\{1−2−j:j∈\{1,2,…\},2j≤Nnom\}\.\\Lambda\_\{N\}\\coloneqq\\left\\\{1\-2^\{\-j\}:j\\in\\\{1,2,\\ldots\\\},\\ 2^\{j\}\\leq N\_\{\\mathrm\{nom\}\}\\right\\\}\.\(173\)
2. 2\.For eachλ∈ΛN\\lambda\\in\\Lambda\_\{N\}, solve the empirical nominal discounted problem subject to \([172](https://arxiv.org/html/2608.06545#A5.E172)\)\. For a sufficiently large universal constantCancC\_\{\\mathrm\{anc\}\}, define H^λ0\\displaystyle\\widehat\{H\}\_\{\\lambda\}^\{0\}≔max⁡\{1,‖V^λπ^λ0,0‖span\},\\displaystyle\\coloneqq\\max\\left\\\{1,\\left\\\|\\widehat\{V\}\_\{\\lambda\}^\{\\widehat\{\\pi\}\_\{\\lambda\}^\{0\},0\}\\right\\\|\_\{\\mathrm\{span\}\}\\right\\\},\(174a\)ρ^λ\+\\displaystyle\\widehat\{\\rho\}\_\{\\lambda\}^\{\+\}≔\(1−λ\)​maxs⁡V^λπ^λ0,0​\(s\)\+Canc​\(H^λ0⋅ιNNnom\+ιNNnom​\(1−λ\)\+εopt\),\\displaystyle\\coloneqq\(1\-\\lambda\)\\max\_\{s\}\\widehat\{V\}\_\{\\lambda\}^\{\\widehat\{\\pi\}\_\{\\lambda\}^\{0\},0\}\(s\)\+C\_\{\\mathrm\{anc\}\}\\left\(\\sqrt\{\\frac\{\\widehat\{H\}\_\{\\lambda\}^\{0\}\\cdot\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\}\}\+\\frac\{\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\(1\-\\lambda\)\}\+\\varepsilon\_\{\\mathrm\{opt\}\}\\right\),\(174b\)H^λ\+\\displaystyle\\widehat\{H\}\_\{\\lambda\}^\{\+\}≔H^λ0\+Canc​\(11−λ​H^λ0⋅ιNNnom\+ιNNnom​\(1−λ\)2\+εopt1−λ\)\.\\displaystyle\\coloneqq\\widehat\{H\}\_\{\\lambda\}^\{0\}\+C\_\{\\mathrm\{anc\}\}\\left\(\\frac\{1\}\{1\-\\lambda\}\\sqrt\{\\frac\{\\widehat\{H\}\_\{\\lambda\}^\{0\}\\cdot\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\}\}\+\\frac\{\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\(1\-\\lambda\)^\{2\}\}\+\\frac\{\\varepsilon\_\{\\mathrm\{opt\}\}\}\{1\-\\lambda\}\\right\)\.\(174c\)The pair\(ρ^λ\+,H^λ\+\)\(\\widehat\{\\rho\}\_\{\\lambda\}^\{\+\},\\widehat\{H\}\_\{\\lambda\}^\{\+\}\)is the candidate anchor certificate atλ\\lambda\.
3. 3\.For eachλ∈ΛN\\lambda\\in\\Lambda\_\{N\}, compute Q^λ≔\(1−λ\)​H^λ\+\+2​\[ρ^λ\+−\(1−λ\)​maxs⁡V^λπ^λ0,0​\(s\)\]\.\\widehat\{Q\}\_\{\\lambda\}\\coloneqq\(1\-\\lambda\)\\widehat\{H\}\_\{\\lambda\}^\{\+\}\+2\\left\[\\widehat\{\\rho\}\_\{\\lambda\}^\{\+\}\-\(1\-\\lambda\)\\max\_\{s\}\\widehat\{V\}\_\{\\lambda\}^\{\\widehat\{\\pi\}\_\{\\lambda\}^\{0\},0\}\(s\)\\right\]\.\(175\)Select, using arbitrary deterministic tie\-breaking, λ^anc∈arg⁡minλ∈ΛN:Q^λ≤minμ∈ΛN⁡Q^μ\+ε⁡H^λ\+\.\\widehat\{\\lambda\}\_\{\\mathrm\{anc\}\}\\in\\arg\\min\_\{\\begin\{subarray\}\{c\}\\lambda\\in\\Lambda\_\{N\}:\\\\ \\widehat\{Q\}\_\{\\lambda\}\\leq\\min\_\{\\mu\\in\\Lambda\_\{N\}\}\\widehat\{Q\}\_\{\\mu\}\+\\varepsilon\\end\{subarray\}\}\\widehat\{H\}\_\{\\lambda\}^\{\+\}\.\(176\)
4. 4\.Return the anchor certificate ρ^anc\+≔ρ^λ^anc\+,H^anc\+≔H^λ^anc\+\.\\widehat\{\\rho\}\_\{\\mathrm\{anc\}\}^\{\+\}\\coloneqq\\widehat\{\\rho\}\_\{\\widehat\{\\lambda\}\_\{\\mathrm\{anc\}\}\}^\{\+\},\\qquad\\widehat\{H\}\_\{\\mathrm\{anc\}\}^\{\+\}\\coloneqq\\widehat\{H\}\_\{\\widehat\{\\lambda\}\_\{\\mathrm\{anc\}\}\}^\{\+\}\.\(177\)

The rule \([176](https://arxiv.org/html/2608.06545#A5.E176)\) first restricts attention to the discount factors whoseQ^λ\\widehat\{Q\}\_\{\\lambda\}values are withinε\\varepsilonof the minimum and then selects one with the smallest certified spanH^λ\+\\widehat\{H\}\_\{\\lambda\}^\{\+\}\. The resulting certificate \([177](https://arxiv.org/html/2608.06545#A5.E177)\) supplies the anchor reward\-level and span bounds used in the confidence penalty for every robust policy candidate\. Because Algorithm[3](https://arxiv.org/html/2608.06545#alg3)uses only𝒟nom\\mathcal\{D\}\_\{\\mathrm\{nom\}\}, the selected discount factor and its certificate are independent of𝒟rob\\mathcal\{D\}\_\{\\mathrm\{rob\}\}\.

##### Robust planning and evaluation\.

For eachγ∈ΓN\\gamma\\in\\Gamma\_\{N\}, we useP^rob0\\widehat\{P\}\_\{\\mathrm\{rob\}\}^\{0\}in two ways\. First, we evaluate the nominal candidateπ^γ0\\widehat\{\\pi\}\_\{\\gamma\}^\{0\}exactly in the empirical robust MDP and denote its value byV^γπ^γ0,σ\\widehat\{V\}\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\}^\{0\},\\sigma\}\. Second, we run the robust discounted solver to obtain a policyπ^γ\\widehat\{\\pi\}\_\{\\gamma\}satisfying

‖V^γ⋆,σ−V^γπ^γ,σ‖∞≤εopt\.\\left\\\|\\widehat\{V\}\_\{\\gamma\}^\{\\star,\\sigma\}\-\\widehat\{V\}\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\},\\sigma\}\\right\\\|\_\{\\infty\}\\leq\\varepsilon\_\{\\mathrm\{opt\}\}\.\(178\)All empirical robust values below are computed in the uncertainty set centered atP^rob0\\widehat\{P\}\_\{\\mathrm\{rob\}\}^\{0\}\. We assume exact fixed\-policy evaluation, so the nominal\-candidate penalty has no optimization\-error term; an approximate evaluation can instead be accommodated by adding its tolerance to that penalty\.

##### Penalties\.

Fix a discount factorη\\etaand an empirical robust value functionVV\. Suppose an anchor\(ρ,h\)\(\\rho,h\)is associated with computable boundsρ¯\\bar\{\\rho\}andH¯\\bar\{H\}satisfying

ρ≤ρ¯,max⁡\{1,‖h‖span\}≤H¯\.\\rho\\leq\\bar\{\\rho\},\\qquad\\max\\\{1,\\\|h\\\|\_\{\\mathrm\{span\}\}\\\}\\leq\\bar\{H\}\.From these quantities, we construct the penalty in four stages\. First, the following quantity bounds both the span of the empirical value and the certified span of the anchor:

ℛη​\(V,H¯\)≔max⁡\{1,‖V‖span,H¯\}\.\\mathcal\{R\}\_\{\\eta\}\(V,\\bar\{H\}\)\\coloneqq\\max\\\{1,\\\|V\\\|\_\{\\mathrm\{span\}\},\\bar\{H\}\\\}\.\(179\)Here\[x\]\+≔max⁡\{x,0\}\[x\]\_\{\+\}\\coloneqq\\max\\\{x,0\\\}\. Next,

βη​\(V,ρ¯\)≔\[ρ¯−\(1−η\)​mins⁡V​\(s\)\]\+\\beta\_\{\\eta\}\(V,\\bar\{\\rho\}\)\\coloneqq\\left\[\\bar\{\\rho\}\-\(1\-\\eta\)\\min\_\{s\}V\(s\)\\right\]\_\{\+\}\(180\)measures how far the anchor’s reward upper bound lies above the empirical lower estimate\(1−η\)​mins⁡V​\(s\)\(1\-\\eta\)\\min\_\{s\}V\(s\)\. These two quantities determine

ℬη​\(V,ρ¯,H¯\)≔H¯\+2​ℛη​\(V,H¯\)​βη​\(V,ρ¯\)\+σ​ℛη​\(V,H¯\)2\+\(1−η\)​ℛη​\(V,H¯\)2,\\mathcal\{B\}\_\{\\eta\}\(V,\\bar\{\\rho\},\\bar\{H\}\)\\coloneqq\\bar\{H\}\+2\\mathcal\{R\}\_\{\\eta\}\(V,\\bar\{H\}\)\\beta\_\{\\eta\}\(V,\\bar\{\\rho\}\)\+\\sigma\\mathcal\{R\}\_\{\\eta\}\(V,\\bar\{H\}\)^\{2\}\+\(1\-\\eta\)\\mathcal\{R\}\_\{\\eta\}\(V,\\bar\{H\}\)^\{2\},\(181\)which determines the variance\-dependent part of the penalty\. Finally, define

penη⁡\(V;ρ¯,H¯\)≔Cpen​\(ℬη​\(V,ρ¯,H¯\)⋅ιNNrob\+ℛη​\(V,H¯\)⋅ιNNrob\+βη​\(V,ρ¯\)⋅ιNNrob​\(1−η\)\)\.\\operatorname\{pen\}\_\{\\eta\}\(V;\\bar\{\\rho\},\\bar\{H\}\)\\coloneqq C\_\{\\mathrm\{pen\}\}\\left\(\\sqrt\{\\frac\{\\mathcal\{B\}\_\{\\eta\}\(V,\\bar\{\\rho\},\\bar\{H\}\)\\cdot\\iota\_\{N\}\}\{N\_\{\\mathrm\{rob\}\}\}\}\+\\frac\{\\mathcal\{R\}\_\{\\eta\}\(V,\\bar\{H\}\)\\cdot\\iota\_\{N\}\}\{N\_\{\\mathrm\{rob\}\}\}\+\\frac\{\\beta\_\{\\eta\}\(V,\\bar\{\\rho\}\)\\cdot\\iota\_\{N\}\}\{N\_\{\\mathrm\{rob\}\}\(1\-\\eta\)\}\\right\)\.\(182\)The constantCpenC\_\{\\mathrm\{pen\}\}is a sufficiently large universal constant\. The square\-root term gives the main statistical error\. The term involvingℛη\\mathcal\{R\}\_\{\\eta\}scales with the common span bound, whereas the term involvingβη/\(1−η\)\\beta\_\{\\eta\}/\(1\-\\eta\)scales the anchor reward\-level gap by the effective horizon\.

For a nominal candidate, we use the trivial anchor\(ρ,h\)=\(1,0\)\(\\rho,h\)=\(1,0\)and hence the bounds\(ρ¯,H¯\)=\(1,1\)\(\\bar\{\\rho\},\\bar\{H\}\)=\(1,1\)\. Its empirical robust fixed\-policy evaluation is exact, so its penalty contains only the statistical terms above\. For a robust candidate, the solver guarantee \([178](https://arxiv.org/html/2608.06545#A5.E178)\) contributes the additional termCpen​εoptC\_\{\\mathrm\{pen\}\}\\varepsilon\_\{\\mathrm\{opt\}\}\. We compute one robust\-candidate penalty from the selected anchor certificate and another from the trivial anchor\. Accordingly, for eachγ∈ΓN\\gamma\\in\\Gamma\_\{N\}, define

penγnom⁡\(π^γ0\)\\displaystyle\\operatorname\{pen\}\_\{\\gamma\}^\{\\mathrm\{nom\}\}\(\\widehat\{\\pi\}\_\{\\gamma\}^\{0\}\)≔penγ⁡\(V^γπ^γ0,σ;1,1\),\\displaystyle\\coloneqq\\operatorname\{pen\}\_\{\\gamma\}\\left\(\\widehat\{V\}\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\}^\{0\},\\sigma\};1,1\\right\),\(183a\)penγanc⁡\(π^γ\)\\displaystyle\\operatorname\{pen\}\_\{\\gamma\}^\{\\mathrm\{anc\}\}\(\\widehat\{\\pi\}\_\{\\gamma\}\)≔penγ⁡\(V^γπ^γ,σ;ρ^anc\+,H^anc\+\)\+Cpen​εopt,\\displaystyle\\coloneqq\\operatorname\{pen\}\_\{\\gamma\}\\left\(\\widehat\{V\}\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\},\\sigma\};\\widehat\{\\rho\}\_\{\\mathrm\{anc\}\}^\{\+\},\\widehat\{H\}\_\{\\mathrm\{anc\}\}^\{\+\}\\right\)\+C\_\{\\mathrm\{pen\}\}\\varepsilon\_\{\\mathrm\{opt\}\},\(183b\)penγtriv⁡\(π^γ\)\\displaystyle\\operatorname\{pen\}\_\{\\gamma\}^\{\\mathrm\{triv\}\}\(\\widehat\{\\pi\}\_\{\\gamma\}\)≔penγ⁡\(V^γπ^γ,σ;1,1\)\+Cpen​εopt\.\\displaystyle\\coloneqq\\operatorname\{pen\}\_\{\\gamma\}\\left\(\\widehat\{V\}\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\},\\sigma\};1,1\\right\)\+C\_\{\\mathrm\{pen\}\}\\varepsilon\_\{\\mathrm\{opt\}\}\.\(183c\)The resulting nominal\- and robust\-policy lower\-confidence bounds are defined in \([7](https://arxiv.org/html/2608.06545#S4.E7)\) and \([8](https://arxiv.org/html/2608.06545#S4.E8)\), respectively\. The latter uses the smaller of the selected\-anchor and trivial\-anchor penalties\.

##### Notation shorthand\.

For eachγ∈ΓN\\gamma\\in\\Gamma\_\{N\}, collect the three statistical components of a penalty as

𝒞γ​\(V;ρ¯,H¯\)≔\(ℛγ​\(V,H¯\),βγ​\(V,ρ¯\),ℬγ​\(V,ρ¯,H¯\)\)\.\\mathcal\{C\}\_\{\\gamma\}\(V;\\bar\{\\rho\},\\bar\{H\}\)\\coloneqq\\bigl\(\\mathcal\{R\}\_\{\\gamma\}\(V,\\bar\{H\}\),\\beta\_\{\\gamma\}\(V,\\bar\{\\rho\}\),\\mathcal\{B\}\_\{\\gamma\}\(V,\\bar\{\\rho\},\\bar\{H\}\)\\bigr\)\.\(184\)Abbreviatingλ^anc\\widehat\{\\lambda\}\_\{\\mathrm\{anc\}\}byλ^\\widehat\{\\lambda\}in the following display, the proof shorthand is

\(R^γnom,β^γnom,B^γnom\)=𝒞γ​\(V^γπ^γ0,σ;1,1\)\.\\bigl\(\\widehat\{R\}\_\{\\gamma\}^\{\\mathrm\{nom\}\},\\widehat\{\\beta\}\_\{\\gamma\}^\{\\mathrm\{nom\}\},\\widehat\{B\}\_\{\\gamma\}^\{\\mathrm\{nom\}\}\\bigr\)=\\mathcal\{C\}\_\{\\gamma\}\\left\(\\widehat\{V\}\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\}^\{0\},\\sigma\};1,1\\right\)\.\(185a\)\(R^γ,λ^anc,β^γ,λ^,B^γ,λ^\)=𝒞γ​\(V^γπ^γ,σ;ρ^λ^\+,H^λ^\+\)\.\\bigl\(\\widehat\{R\}\_\{\\gamma,\\widehat\{\\lambda\}\}^\{\\mathrm\{anc\}\},\\widehat\{\\beta\}\_\{\\gamma,\\widehat\{\\lambda\}\},\\widehat\{B\}\_\{\\gamma,\\widehat\{\\lambda\}\}\\bigr\)=\\mathcal\{C\}\_\{\\gamma\}\\\!\\left\(\\widehat\{V\}\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\},\\sigma\};\\widehat\{\\rho\}\_\{\\widehat\{\\lambda\}\}^\{\+\},\\widehat\{H\}\_\{\\widehat\{\\lambda\}\}^\{\+\}\\right\)\.\(185b\)\(R^γ,β^γ,triv,B^γ,triv\)=𝒞γ​\(V^γπ^γ,σ;1,1\)\.\\bigl\(\\widehat\{R\}\_\{\\gamma\},\\widehat\{\\beta\}\_\{\\gamma,\\mathrm\{triv\}\},\\widehat\{B\}\_\{\\gamma,\\mathrm\{triv\}\}\\bigr\)=\\mathcal\{C\}\_\{\\gamma\}\\left\(\\widehat\{V\}\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\},\\sigma\};1,1\\right\)\.\(185c\)ThusR^\\widehat\{R\},β^\\widehat\{\\beta\}, andB^\\widehat\{B\}denote, respectively, the common span bound, the anchor\-level defect, and the budget under the leading square root\. The superscriptnom\\mathrm\{nom\}identifies the nominal\-policy family, while the second subscript records the anchor used for a robust\-policy candidate\. The associated penalties are those in \([183a](https://arxiv.org/html/2608.06545#A5.E183.1)\)–\([183c](https://arxiv.org/html/2608.06545#A5.E183.3)\); the robust\-policy lower\-confidence bound uses the smaller of the selected\-anchor and trivial\-anchor penalties\.

### E\.2Supporting lemmas for the span\-agnostic algorithm

We continue with the notation of Appendix[E\.1](https://arxiv.org/html/2608.06545#A5.SS1)\. In particular,λ∈ΛN\\lambda\\in\\Lambda\_\{N\}indexes anchor candidates constructed from𝒟nom\\mathcal\{D\}\_\{\\mathrm\{nom\}\}, whereasγ∈ΓN\\gamma\\in\\Gamma\_\{N\}indexes policy candidates evaluated using𝒟rob\\mathcal\{D\}\_\{\\mathrm\{rob\}\}\. Accordingly,V^λπ^λ0,0\\widehat\{V\}\_\{\\lambda\}^\{\\widehat\{\\pi\}\_\{\\lambda\}^\{0\},0\}usesP^nom0\\widehat\{P\}\_\{\\mathrm\{nom\}\}^\{0\}, while every hatted robust value usesP^rob0\\widehat\{P\}\_\{\\mathrm\{rob\}\}^\{0\}\.

The supporting lemmas are organized in four stages\. First, Lemma[27](https://arxiv.org/html/2608.06545#Thmlemma27)bounds the errors of the empirical nominal optimal value and the returned nominal policy at every discount factor in the grid\. Lemmas[28](https://arxiv.org/html/2608.06545#Thmlemma28)and[29](https://arxiv.org/html/2608.06545#Thmlemma29)then construct and select an anchor with level withinO​\(ε\)O\(\\varepsilon\)ofρ⋆\\rho^\{\\star\}and spanO​\(H0\)O\(H\_\{0\}\)\. Second, Lemma[30](https://arxiv.org/html/2608.06545#Thmlemma30)makes the fixed\-policy and learned\-policy discounted value\-error bounds hold simultaneously over the dyadic span\-radius grid\. Third, Lemma[31](https://arxiv.org/html/2608.06545#Thmlemma31)locates a nominal candidate at a horizon of orderH0/εH\_\{0\}/\\varepsilon, validates every nominal lower\-confidence bound, and identifies a near\-optimal candidate in the high\-tolerance regime\. Finally, Lemma[32](https://arxiv.org/html/2608.06545#Thmlemma32)gives the corresponding validity and existence guarantees for the robust candidates\. This yields the high\-tolerance guarantee whenHσ≤H0H\_\{\\sigma\}\\leq H\_\{0\}and the low\-tolerance guarantee for either span ordering\. Appendix[E\.3](https://arxiv.org/html/2608.06545#A5.SS3)combines these ingredients to analyze the final selection\.

##### Fixed\-discount nominal comparisons\.

The next lemma gives simultaneous nominal discounted\-value estimation bounds over a finite set of discount factors\. Importantly, the lemma includes bounds expressed in terms of observable empirical value spans\. We use them both to certify the anchor candidates overΛN\\Lambda\_\{N\}and to analyze a nominal policy candidate fromΓN\\Gamma\_\{N\}\. Its proof, given in Appendix[E\.4\.1](https://arxiv.org/html/2608.06545#A5.SS4.SSS1), follows by specializing the argument for Theorem[4](https://arxiv.org/html/2608.06545#Thmtheorem4)toσ=0\\sigma=0\.

###### Lemma 27\(Fixed\-discount nominal plug\-in bounds\)\.

There exists a universal constantC\>0C\>0such that the following holds\. LetN≥16N\\geq 16and let𝒢N⊂\[1/2,1\)\\mathcal\{G\}\_\{N\}\\subset\[1/2,1\)be a deterministic set ofO​\(log⁡N\)O\(\\log N\)discount factors such that

11−η≤Nnomfor every​η∈𝒢N\.\\frac\{1\}\{1\-\\eta\}\\leq N\_\{\\mathrm\{nom\}\}\\qquad\\text\{for every \}\\eta\\in\\mathcal\{G\}\_\{N\}\.For everyη∈𝒢N\\eta\\in\\mathcal\{G\}\_\{N\}, letπ^η0\\widehat\{\\pi\}\_\{\\eta\}^\{0\}satisfy

‖V^η⋆,0−V^ηπ^η0,0‖∞≤1Nnom\.\\left\\\|\\widehat\{V\}\_\{\\eta\}^\{\\star,0\}\-\\widehat\{V\}\_\{\\eta\}^\{\\widehat\{\\pi\}\_\{\\eta\}^\{0\},0\}\\right\\\|\_\{\\infty\}\\leq\\frac\{1\}\{N\_\{\\mathrm\{nom\}\}\}\.Then, with probability at least1−O​\(δ\)1\-O\(\\delta\), the following bounds hold simultaneously for everyη∈𝒢N\\eta\\in\\mathcal\{G\}\_\{N\}:

‖Vη⋆,0−V^η⋆,0‖∞\\displaystyle\\left\\\|V\_\{\\eta\}^\{\\star,0\}\-\\widehat\{V\}\_\{\\eta\}^\{\\star,0\}\\right\\\|\_\{\\infty\}≤C1−η​\(‖V^η⋆,0‖span\+1\)⋅ιNNnom\+C​ιNNnom​\(1−η\)2,\\displaystyle\\leq\\frac\{C\}\{1\-\\eta\}\\sqrt\{\\frac\{\(\\\|\\widehat\{V\}\_\{\\eta\}^\{\\star,0\}\\\|\_\{\\mathrm\{span\}\}\+1\)\\cdot\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\}\}\+C\\frac\{\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\(1\-\\eta\)^\{2\}\},\(186a\)‖V^η⋆,0−Vη⋆,0‖∞\\displaystyle\\left\\\|\\widehat\{V\}\_\{\\eta\}^\{\\star,0\}\-V\_\{\\eta\}^\{\\star,0\}\\right\\\|\_\{\\infty\}≤C​ιNNnom​\(1−η\)2\+C1−η​\(‖Vη⋆,0‖span\+1\)⋅ιNNnom\+CNnom,\\displaystyle\\leq C\\frac\{\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\(1\-\\eta\)^\{2\}\}\+\\frac\{C\}\{1\-\\eta\}\\sqrt\{\\frac\{\(\\\|V\_\{\\eta\}^\{\\star,0\}\\\|\_\{\\mathrm\{span\}\}\+1\)\\cdot\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\}\}\+\\frac\{C\}\{N\_\{\\mathrm\{nom\}\}\},\(186b\)‖V^ηπ^η0,0−Vηπ^η0,0‖∞\\displaystyle\\left\\\|\\widehat\{V\}\_\{\\eta\}^\{\\widehat\{\\pi\}\_\{\\eta\}^\{0\},0\}\-V\_\{\\eta\}^\{\\widehat\{\\pi\}\_\{\\eta\}^\{0\},0\}\\right\\\|\_\{\\infty\}≤C1−η​\(‖V^ηπ^η0,0‖span\+1\)⋅ιNNnom\+C​ιNNnom​\(1−η\)2\.\\displaystyle\\leq\\frac\{C\}\{1\-\\eta\}\\sqrt\{\\frac\{\(\\\|\\widehat\{V\}\_\{\\eta\}^\{\\widehat\{\\pi\}\_\{\\eta\}^\{0\},0\}\\\|\_\{\\mathrm\{span\}\}\+1\)\\cdot\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\}\}\+C\\frac\{\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\(1\-\\eta\)^\{2\}\}\.\(186c\)

##### Anchor certificates\.

The next two lemmas connect the population anchors used in the analysis with the certificates computed from𝒟nom\\mathcal\{D\}\_\{\\mathrm\{nom\}\}\. The first associates each discount factor with a population anchor\. The second collects the validity, oracle, and selection properties of the empirical certificates in a single calibration result\.

We begin with the population anchors\. The following lemma identifies a valid anchor for each discount factorλ∈\(0,1\)\\lambda\\in\(0,1\)\. Its proof is deferred to Appendix[E\.4\.2](https://arxiv.org/html/2608.06545#A5.SS4.SSS2)\.

###### Lemma 28\(Population nominal anchors\)\.

Fixλ∈\(0,1\)\\lambda\\in\(0,1\)and let

hλ0=Vλ⋆,0−mins⁡Vλ⋆,0​\(s\),h\_\{\\lambda\}^\{0\}=V\_\{\\lambda\}^\{\\star,0\}\-\\min\_\{s\}V\_\{\\lambda\}^\{\\star,0\}\(s\),and

ρ¯λ0=\(1−λ\)​maxs⁡Vλ⋆,0​\(s\)\.\\bar\{\\rho\}\_\{\\lambda\}^\{0\}=\(1\-\\lambda\)\\max\_\{s\}V\_\{\\lambda\}^\{\\star,0\}\(s\)\.Then, for everys∈𝒮s\\in\\mathcal\{S\},

ρ¯λ0\+hλ0​\(s\)≥maxa∈𝒜⁡\{r​\(s,a\)\+Ps,a0​hλ0\}\.\\bar\{\\rho\}\_\{\\lambda\}^\{0\}\+h\_\{\\lambda\}^\{0\}\(s\)\\geq\\max\_\{a\\in\\mathcal\{A\}\}\\left\\\{r\(s,a\)\+P^\{0\}\_\{s,a\}h\_\{\\lambda\}^\{0\}\\right\\\}\.

Recall from \([174](https://arxiv.org/html/2608.06545#A5.E174)\) that\(ρ^λ\+,H^λ\+\)\(\\widehat\{\\rho\}\_\{\\lambda\}^\{\+\},\\widehat\{H\}\_\{\\lambda\}^\{\+\}\)is the candidate certificate computed atλ\\lambda\. The following lemma gives all properties of these certificates used below\. Its proof is deferred to Appendix[E\.4\.3](https://arxiv.org/html/2608.06545#A5.SS4.SSS3)\.

###### Lemma 29\(Anchor calibration guarantees\)\.

LetN≥16N\\geq 16and, for eachλ∈ΛN\\lambda\\in\\Lambda\_\{N\}, let

Hλ0=max⁡\{1,‖hλ0‖span\}\.H\_\{\\lambda\}^\{0\}=\\max\\\{1,\\\|h\_\{\\lambda\}^\{0\}\\\|\_\{\\mathrm\{span\}\}\\\}\.IfCancC\_\{\\mathrm\{anc\}\}in \([174](https://arxiv.org/html/2608.06545#A5.E174)\) is sufficiently large, then, with probability at least1−O​\(δ\)1\-O\(\\delta\), simultaneously for everyλ∈ΛN\\lambda\\in\\Lambda\_\{N\},

\|ρ¯λ0−\(1−λ\)​maxs⁡V^λπ^λ0,0​\(s\)\|\\displaystyle\\left\|\\bar\{\\rho\}\_\{\\lambda\}^\{0\}\-\(1\-\\lambda\)\\max\_\{s\}\\widehat\{V\}\_\{\\lambda\}^\{\\widehat\{\\pi\}\_\{\\lambda\}^\{0\},0\}\(s\)\\right\|≤ρ^λ\+−\(1−λ\)​maxs⁡V^λπ^λ0,0​\(s\),\\displaystyle\\leq\\widehat\{\\rho\}\_\{\\lambda\}^\{\+\}\-\(1\-\\lambda\)\\max\_\{s\}\\widehat\{V\}\_\{\\lambda\}^\{\\widehat\{\\pi\}\_\{\\lambda\}^\{0\},0\}\(s\),\(187a\)ρ¯λ0\\displaystyle\\bar\{\\rho\}\_\{\\lambda\}^\{0\}≤ρ^λ\+,\\displaystyle\\leq\\widehat\{\\rho\}\_\{\\lambda\}^\{\+\},\(187b\)Hλ0\\displaystyle H\_\{\\lambda\}^\{0\}≤H^λ\+\.\\displaystyle\\leq\\widehat\{H\}\_\{\\lambda\}^\{\+\}\.\(187c\)Moreover, there exist a sufficiently large universal constantC\>0C\>0and a sufficiently small universal constantcopt\>0c\_\{\\mathrm\{opt\}\}\>0such that, if

Nnom≥C​H0⋅ιNε2,εopt≤copt​ε,N\_\{\\mathrm\{nom\}\}\\geq C\\frac\{H\_\{0\}\\cdot\\iota\_\{N\}\}\{\\varepsilon^\{2\}\},\\qquad\\varepsilon\_\{\\mathrm\{opt\}\}\\leq c\_\{\\mathrm\{opt\}\}\\varepsilon,\(188\)then the following conclusions hold on the same event:

1. \(i\)there existsλ†∈ΛN\\lambda^\{\\dagger\}\\in\\Lambda\_\{N\}satisfying 11−λ†≍min⁡\{Nnom​H0ιN,H0εopt\},\\frac\{1\}\{1\-\\lambda^\{\\dagger\}\}\\asymp\\min\\left\\\{\\sqrt\{\\frac\{N\_\{\\mathrm\{nom\}\}H\_\{0\}\}\{\\iota\_\{N\}\}\},\\frac\{H\_\{0\}\}\{\\varepsilon\_\{\\mathrm\{opt\}\}\}\\right\\\},H^λ†\+≤C​H0,ρ^λ†\+≤ρ⋆\+C​H0⋅ιNNnom\+C​εopt;\\widehat\{H\}\_\{\\lambda^\{\\dagger\}\}^\{\+\}\\leq CH\_\{0\},\\qquad\\widehat\{\\rho\}\_\{\\lambda^\{\\dagger\}\}^\{\+\}\\leq\\rho^\{\\star\}\+C\\sqrt\{\\frac\{H\_\{0\}\\cdot\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\}\}\+C\\varepsilon\_\{\\mathrm\{opt\}\};
2. \(ii\)the selected certificate satisfies Q^λ^anc\\displaystyle\\widehat\{Q\}\_\{\\widehat\{\\lambda\}\_\{\\mathrm\{anc\}\}\}≤C​ε,\\displaystyle\\leq C\\varepsilon,\(189a\)H^λ^anc\+\\displaystyle\\widehat\{H\}\_\{\\widehat\{\\lambda\}\_\{\\mathrm\{anc\}\}\}^\{\+\}≤C​H0,\\displaystyle\\leq CH\_\{0\},\(189b\)ρ⋆≤ρ¯λ^anc0≤ρ^λ^anc\+\\displaystyle\\rho^\{\\star\}\\leq\\bar\{\\rho\}\_\{\\widehat\{\\lambda\}\_\{\\mathrm\{anc\}\}\}^\{0\}\\leq\\widehat\{\\rho\}\_\{\\widehat\{\\lambda\}\_\{\\mathrm\{anc\}\}\}^\{\+\}≤ρ⋆\+C​ε\.\\displaystyle\\leq\\rho^\{\\star\}\+C\\varepsilon\.\(189c\)

Whenεopt=0\\varepsilon\_\{\\mathrm\{opt\}\}=0, the second quantity in the minimum is interpreted as\+∞\+\\infty\.

##### Shared robust comparison\.

The nominal and robust policy candidates require different discounted\-value comparisons\. Conditional on the nominal batch, each nominal candidate is fixed independently of𝒟rob\\mathcal\{D\}\_\{\\mathrm\{rob\}\}\. The fixed\-policy extension of Lemma[13](https://arxiv.org/html/2608.06545#Thmlemma13)therefore controls

‖V^γπ,σ−Vγπ,σ‖∞\\left\\\|\\widehat\{V\}\_\{\\gamma\}^\{\\pi,\\sigma\}\-V\_\{\\gamma\}^\{\\pi,\\sigma\}\\right\\\|\_\{\\infty\}for these candidates\. By contrast,π^γ\\widehat\{\\pi\}\_\{\\gamma\}is computed from𝒟rob\\mathcal\{D\}\_\{\\mathrm\{rob\}\}itself, so the learned\-policy bound in Lemma[14](https://arxiv.org/html/2608.06545#Thmlemma14)is needed to control

‖V^γπ^γ,σ−Vγπ^γ,σ‖∞\.\\left\\\|\\widehat\{V\}\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\},\\sigma\}\-V\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\},\\sigma\}\\right\\\|\_\{\\infty\}\.The former is used with the trivial anchor, whereas the latter is used with both the population anchor indexed byλ^anc\\widehat\{\\lambda\}\_\{\\mathrm\{anc\}\}and the trivial anchor\.

Both bounds require a deterministic numberRRthat upper\-bounds the relevant value spans; see \([70](https://arxiv.org/html/2608.06545#A4.E70)\)\. The required radius depends on empirical robust values and therefore cannot be selected after observing𝒟rob\\mathcal\{D\}\_\{\\mathrm\{rob\}\}\. We instead fix a dyadic grid of candidate radii independently of𝒟rob\\mathcal\{D\}\_\{\\mathrm\{rob\}\}and make both bounds hold simultaneously at every grid point\. The following lemma formalizes this simultaneous comparison\. Its proof is deferred to Appendix[E\.4\.4](https://arxiv.org/html/2608.06545#A5.SS4.SSS4)\.

###### Lemma 30\(Simultaneous comparison over candidate span bounds\)\.

Fixγ∈ΓN\\gamma\\in\\Gamma\_\{N\}\. Independently of𝒟rob\\mathcal\{D\}\_\{\\mathrm\{rob\}\}, choose a deterministic policyπ\\pi, an anchor pair\(ρ¯,h¯\)\(\\bar\{\\rho\},\\bar\{h\}\)satisfying \([6](https://arxiv.org/html/2608.06545#S4.E6)\), and a dyadic grid

Rj=2j​Rmin,j=0,…,J,R\_\{j\}=2^\{j\}R\_\{\\min\},\\qquad j=0,\\ldots,J,where

Rmin≥max⁡\{1,‖h¯‖span\},J=O​\(log⁡N\)\.R\_\{\\min\}\\geq\\max\\\{1,\\\|\\bar\{h\}\\\|\_\{\\mathrm\{span\}\}\\\},\\qquad J=O\(\\log N\)\.Then, with probability at least1−O​\(δ/N3\)1\-O\(\\delta/N^\{3\}\), the following conclusions hold simultaneously for everyj=0,…,Jj=0,\\ldots,J\. Whenever

Rj≥max⁡\{1,‖h¯‖span,‖Vγπ,σ‖span,‖V^γπ,σ‖span\},R\_\{j\}\\geq\\max\\left\\\{1,\\\|\\bar\{h\}\\\|\_\{\\mathrm\{span\}\},\\\|V\_\{\\gamma\}^\{\\pi,\\sigma\}\\\|\_\{\\mathrm\{span\}\},\\\|\\widehat\{V\}\_\{\\gamma\}^\{\\pi,\\sigma\}\\\|\_\{\\mathrm\{span\}\}\\right\\\},the fixed\-policy bound \([72](https://arxiv.org/html/2608.06545#A4.E72)\) from Lemma[13](https://arxiv.org/html/2608.06545#Thmlemma13)holds forπ\\pi\. If, in addition,π=πγ⋆\\pi=\\pi\_\{\\gamma\}^\{\\star\}and

Rj≥max⁡\{‖Vγπ^γ,σ‖span,‖V^γπ^γ,σ‖span,‖V^γ⋆,σ‖span\},R\_\{j\}\\geq\\max\\left\\\{\\\|V\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\},\\sigma\}\\\|\_\{\\mathrm\{span\}\},\\\|\\widehat\{V\}\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\},\\sigma\}\\\|\_\{\\mathrm\{span\}\},\\\|\\widehat\{V\}\_\{\\gamma\}^\{\\star,\\sigma\}\\\|\_\{\\mathrm\{span\}\}\\right\\\},then the learned\-policy bound from Lemma[14](https://arxiv.org/html/2608.06545#Thmlemma14)also holds for the policyπ^γ\\widehat\{\\pi\}\_\{\\gamma\}returned by the empirical robust solver\. In both bounds,NrobN\_\{\\mathrm\{rob\}\}andιN\\iota\_\{N\}take the places ofNNandι\\iota\.

##### Nominal policy candidates\.

The following lemma collects the validity and existence properties of the nominal policy candidates\. Its proof is deferred to Appendix[E\.4\.5](https://arxiv.org/html/2608.06545#A5.SS4.SSS5)\.

###### Lemma 31\(Nominal\-policy candidate guarantees\)\.

With probability at least1−O​\(δ\)1\-O\(\\delta\), simultaneously for everyγ∈ΓN\\gamma\\in\\Gamma\_\{N\},

\(1−γ\)​‖V^γπ^γ0,σ−Vγπ^γ0,σ‖∞\\displaystyle\(1\-\\gamma\)\\left\\\|\\widehat\{V\}\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\}^\{0\},\\sigma\}\-V\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\}^\{0\},\\sigma\}\\right\\\|\_\{\\infty\}≤penγnom⁡\(π^γ0\),\\displaystyle\\leq\\operatorname\{pen\}\_\{\\gamma\}^\{\\mathrm\{nom\}\}\(\\widehat\{\\pi\}\_\{\\gamma\}^\{0\}\),\(190\)ρπ^γ0,σ\\displaystyle\\rho^\{\\widehat\{\\pi\}\_\{\\gamma\}^\{0\},\\sigma\}≥LCBγnom⁡\(π^γ0\)\.\\displaystyle\\geq\\operatorname\{LCB\}\_\{\\gamma\}^\{\\mathrm\{nom\}\}\(\\widehat\{\\pi\}\_\{\\gamma\}^\{0\}\)\.\(191\)Moreover, there exist sufficiently large universal constantsC,Cnom\>0C,C\_\{\\mathrm\{nom\}\}\>0and a sufficiently small universal constantc\>0c\>0such that, if

σ​H0≤c​ε,N≥C​H0⋅ιNε2,\\sigma H\_\{0\}\\leq c\\varepsilon,\\qquad N\\geq C\\frac\{H\_\{0\}\\cdot\\iota\_\{N\}\}\{\\varepsilon^\{2\}\},\(192\)andNnomN\_\{\\mathrm\{nom\}\}andNrobN\_\{\\mathrm\{rob\}\}are constant fractions ofNN, thenΓN\\Gamma\_\{N\}contains a discount factorγnom†\\gamma\_\{\\mathrm\{nom\}\}^\{\\dagger\}satisfying

Cnom​H0ε≤11−γnom†≤2​Cnom​H0ε\.\\frac\{C\_\{\\mathrm\{nom\}\}H\_\{0\}\}\{\\varepsilon\}\\leq\\frac\{1\}\{1\-\\gamma\_\{\\mathrm\{nom\}\}^\{\\dagger\}\}\\leq\\frac\{2C\_\{\\mathrm\{nom\}\}H\_\{0\}\}\{\\varepsilon\}\.\(193\)On the same event,

\(1−γnom†\)​mins⁡Vγnom†π^γnom†0,0​\(s\)\\displaystyle\(1\-\\gamma\_\{\\mathrm\{nom\}\}^\{\\dagger\}\)\\min\_\{s\}V\_\{\\gamma\_\{\\mathrm\{nom\}\}^\{\\dagger\}\}^\{\\widehat\{\\pi\}\_\{\\gamma\_\{\\mathrm\{nom\}\}^\{\\dagger\}\}^\{0\},0\}\(s\)≥ρ⋆−c​ε,\\displaystyle\\geq\\rho^\{\\star\}\-c\\varepsilon,\(194\)‖Vγnom†π^γnom†0,0‖span\\displaystyle\\left\\\|V\_\{\\gamma\_\{\\mathrm\{nom\}\}^\{\\dagger\}\}^\{\\widehat\{\\pi\}\_\{\\gamma\_\{\\mathrm\{nom\}\}^\{\\dagger\}\}^\{0\},0\}\\right\\\|\_\{\\mathrm\{span\}\}≤C​H0,\\displaystyle\\leq CH\_\{0\},\(195\)LCBγnom†nom⁡\(π^γnom†0\)\\displaystyle\\operatorname\{LCB\}\_\{\\gamma\_\{\\mathrm\{nom\}\}^\{\\dagger\}\}^\{\\mathrm\{nom\}\}\(\\widehat\{\\pi\}\_\{\\gamma\_\{\\mathrm\{nom\}\}^\{\\dagger\}\}^\{0\}\)≥ρ⋆,σ−ε\.\\displaystyle\\geq\\rho^\{\\star,\\sigma\}\-\\varepsilon\.\(196\)

##### Robust policy candidates\.

The following lemma collects the validity and existence properties of the policies learned from the robust batch\. Its proof is deferred to Appendix[E\.4\.6](https://arxiv.org/html/2608.06545#A5.SS4.SSS6)\.

###### Lemma 32\(Robust\-policy candidate guarantees\)\.

There exists a sufficiently large universal constantC\>0C\>0such that the following holds\. For eachγ∈ΓN\\gamma\\in\\Gamma\_\{N\}, letπγ⋆\\pi\_\{\\gamma\}^\{\\star\}be a deterministic discounted\-optimal policy for the true robust MDP and define

Δγ=‖V^γπ^γ,σ−Vγπ^γ,σ‖∞\+‖V^γπγ⋆,σ−Vγ⋆,σ‖∞\.\\Delta\_\{\\gamma\}=\\left\\\|\\widehat\{V\}\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\},\\sigma\}\-V\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\},\\sigma\}\\right\\\|\_\{\\infty\}\+\\\|\\widehat\{V\}\_\{\\gamma\}^\{\\pi\_\{\\gamma\}^\{\\star\},\\sigma\}\-V\_\{\\gamma\}^\{\\star,\\sigma\}\\\|\_\{\\infty\}\.\(197\)Assumeεopt≤1\\varepsilon\_\{\\mathrm\{opt\}\}\\leq 1\. Then, with probability at least1−O​\(δ\)1\-O\(\\delta\), the following two bounds hold simultaneously for everyγ∈ΓN\\gamma\\in\\Gamma\_\{N\}\. The selected anchor certificate gives

\(1−γ\)​Δγ≤C​B^γ,λ^anc⋅ιNNrob\+C​R^γ,λ^ancanc⋅ιNNrob\+C​β^γ,λ^anc⋅ιNNrob​\(1−γ\)\+C​εopt\.\(1\-\\gamma\)\\Delta\_\{\\gamma\}\\leq C\\sqrt\{\\frac\{\\widehat\{B\}\_\{\\gamma,\\widehat\{\\lambda\}\_\{\\mathrm\{anc\}\}\}\\cdot\\iota\_\{N\}\}\{N\_\{\\mathrm\{rob\}\}\}\}\+C\\frac\{\\widehat\{R\}\_\{\\gamma,\\widehat\{\\lambda\}\_\{\\mathrm\{anc\}\}\}^\{\\mathrm\{anc\}\}\\cdot\\iota\_\{N\}\}\{N\_\{\\mathrm\{rob\}\}\}\+C\\frac\{\\widehat\{\\beta\}\_\{\\gamma,\\widehat\{\\lambda\}\_\{\\mathrm\{anc\}\}\}\\cdot\\iota\_\{N\}\}\{N\_\{\\mathrm\{rob\}\}\(1\-\\gamma\)\}\+C\\varepsilon\_\{\\mathrm\{opt\}\}\.\(198\)The trivial anchor\(ρ,h\)=\(1,0\)\(\\rho,h\)=\(1,0\)gives

\(1−γ\)​Δγ≤C​B^γ,triv⋅ιNNrob\+C​R^γ⋅ιNNrob\+C​β^γ,triv⋅ιNNrob​\(1−γ\)\+C​εopt\.\(1\-\\gamma\)\\Delta\_\{\\gamma\}\\leq C\\sqrt\{\\frac\{\\widehat\{B\}\_\{\\gamma,\\mathrm\{triv\}\}\\cdot\\iota\_\{N\}\}\{N\_\{\\mathrm\{rob\}\}\}\}\+C\\frac\{\\widehat\{R\}\_\{\\gamma\}\\cdot\\iota\_\{N\}\}\{N\_\{\\mathrm\{rob\}\}\}\+C\\frac\{\\widehat\{\\beta\}\_\{\\gamma,\\mathrm\{triv\}\}\\cdot\\iota\_\{N\}\}\{N\_\{\\mathrm\{rob\}\}\(1\-\\gamma\)\}\+C\\varepsilon\_\{\\mathrm\{opt\}\}\.\(199\)On the same event, the following holds for everyγ∈ΓN\\gamma\\in\\Gamma\_\{N\}:

ρπ^γ,σ≥LCBγrob⁡\(π^γ\)\.\\rho^\{\\widehat\{\\pi\}\_\{\\gamma\},\\sigma\}\\geq\\operatorname\{LCB\}\_\{\\gamma\}^\{\\mathrm\{rob\}\}\(\\widehat\{\\pi\}\_\{\\gamma\}\)\.Moreover, there exist sufficiently large universal constantsC,Crob\>0C,C\_\{\\mathrm\{rob\}\}\>0and a sufficiently small universal constantcopt\>0c\_\{\\mathrm\{opt\}\}\>0such that the following holds\. LetN≥16N\\geq 16andε∈\(0,1\]\\varepsilon\\in\(0,1\]\. Suppose that

N≥C​\[min⁡\{H0,Hσ\}\+σ​Hσ2ε2\+Hσε\]⋅ιNN\\geq C\\left\[\\frac\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\+\\sigma H\_\{\\sigma\}^\{2\}\}\{\\varepsilon^\{2\}\}\+\\frac\{H\_\{\\sigma\}\}\{\\varepsilon\}\\right\]\\cdot\\iota\_\{N\}\(200\)and thatNnomN\_\{\\mathrm\{nom\}\}andNrobN\_\{\\mathrm\{rob\}\}are constant fractions ofNN\. Assume alsoεopt≤copt​ε\\varepsilon\_\{\\mathrm\{opt\}\}\\leq c\_\{\\mathrm\{opt\}\}\\varepsilon\. Then the policy gridΓN\\Gamma\_\{N\}contains a discount factorγ†\\gamma^\{\\dagger\}such that

Crob​Hσε≤11−γ†≤2​Crob​Hσε\.\\frac\{C\_\{\\mathrm\{rob\}\}H\_\{\\sigma\}\}\{\\varepsilon\}\\leq\\frac\{1\}\{1\-\\gamma^\{\\dagger\}\}\\leq\\frac\{2C\_\{\\mathrm\{rob\}\}H\_\{\\sigma\}\}\{\\varepsilon\}\.Moreover, with probability at least1−O​\(δ\)1\-O\(\\delta\),

LCBγ†rob⁡\(π^γ†\)≥ρ⋆,σ−ε\.\\operatorname\{LCB\}\_\{\\gamma^\{\\dagger\}\}^\{\\mathrm\{rob\}\}\(\\widehat\{\\pi\}\_\{\\gamma^\{\\dagger\}\}\)\\geq\\rho^\{\\star,\\sigma\}\-\\varepsilon\.\(201\)

### E\.3Proof of Theorem[3](https://arxiv.org/html/2608.06545#Thmtheorem3)

We prove Theorem[3](https://arxiv.org/html/2608.06545#Thmtheorem3)by combining the candidate guarantees established in Appendix[E\.2](https://arxiv.org/html/2608.06545#A5.SS2)\.

##### Sample\-complexity check in different regimes\.

Recall that the theorem’s sample bound is

N​S​A≥C​S​A​min⁡\{H0,Hσ\}ε2⋅ιNNSA\\geq CSA\\frac\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\}\{\\varepsilon^\{2\}\}\\cdot\\iota\_\{N\}in the high\-tolerance regime and

N​S​A≥C​S​A​min⁡\{H0,Hσ\}\+σ​Hσ2ε2⋅ιNNSA\\geq CSA\\frac\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\+\\sigma H\_\{\\sigma\}^\{2\}\}\{\\varepsilon^\{2\}\}\\cdot\\iota\_\{N\}in the low\-tolerance regime\.

We first verify the sample\-complexity conditions required by the nominal\- and robust\-policy candidate guarantees \(Lemmas[31](https://arxiv.org/html/2608.06545#Thmlemma31)and[32](https://arxiv.org/html/2608.06545#Thmlemma32)\)\.

Consider the low\-tolerance regime in whichσ​H0\>c​ε\\sigma H\_\{0\}\>c\\varepsilon\. We verify the sample\-complexity condition of Lemma[32](https://arxiv.org/html/2608.06545#Thmlemma32)in \([200](https://arxiv.org/html/2608.06545#A5.E200)\)\. After dividing the theorem’s total\-sample bound byS​ASA, its low\-tolerance bound already controls the first two terms in that requirement\. It remains to controlHσ/εH\_\{\\sigma\}/\\varepsilon\. IfHσ≤H0H\_\{\\sigma\}\\leq H\_\{0\}, then

Hσε≤min⁡\{H0,Hσ\}ε2,\\frac\{H\_\{\\sigma\}\}\{\\varepsilon\}\\leq\\frac\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\}\{\\varepsilon^\{2\}\},where we usedε≤1\\varepsilon\\leq 1\. If insteadH0<HσH\_\{0\}<H\_\{\\sigma\}, the conditionσ​H0\>c​ε\\sigma H\_\{0\}\>c\\varepsilongives

Hσε≤1c​σ​Hσ2ε2\.\\frac\{H\_\{\\sigma\}\}\{\\varepsilon\}\\leq\\frac\{1\}\{c\}\\frac\{\\sigma H\_\{\\sigma\}^\{2\}\}\{\\varepsilon^\{2\}\}\.Thus the theorem’s low\-tolerance sample bound implies \([200](https://arxiv.org/html/2608.06545#A5.E200)\)\.

Consider the high\-tolerance regime in whichσ​H0≤c​ε\\sigma H\_\{0\}\\leq c\\varepsilon\. We first look at the case withH0<HσH\_\{0\}<H\_\{\\sigma\}\. We verify the sample\-complexity condition of Lemma[31](https://arxiv.org/html/2608.06545#Thmlemma31)in \([192](https://arxiv.org/html/2608.06545#A5.E192)\)\. Its tolerance condition holds by assumption, andmin⁡\{H0,Hσ\}=H0\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}=H\_\{0\}\. Hence, after dividing the theorem’s total\-sample bound byS​ASA, its high\-tolerance bound gives

N≥C​H0​ιNε2,N\\geq C\\frac\{H\_\{0\}\\iota\_\{N\}\}\{\\varepsilon^\{2\}\},which verifies \([192](https://arxiv.org/html/2608.06545#A5.E192)\)\.

We now look at the case withHσ≤H0H\_\{\\sigma\}\\leq H\_\{0\}\. We verify the sample\-complexity condition of Lemma[32](https://arxiv.org/html/2608.06545#Thmlemma32)in \([200](https://arxiv.org/html/2608.06545#A5.E200)\)\. Sincemin⁡\{H0,Hσ\}=Hσ\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}=H\_\{\\sigma\}, the theorem’s high\-tolerance sample bound controlsHσ/ε2H\_\{\\sigma\}/\\varepsilon^\{2\}\. Moreover,

Hσε≤Hσε2,σ​Hσ2ε2≤c​Hσε≤c​Hσε2,\\frac\{H\_\{\\sigma\}\}\{\\varepsilon\}\\leq\\frac\{H\_\{\\sigma\}\}\{\\varepsilon^\{2\}\},\\qquad\\frac\{\\sigma H\_\{\\sigma\}^\{2\}\}\{\\varepsilon^\{2\}\}\\leq c\\frac\{H\_\{\\sigma\}\}\{\\varepsilon\}\\leq c\\frac\{H\_\{\\sigma\}\}\{\\varepsilon^\{2\}\},where the second inequality usesσ​Hσ≤σ​H0≤c​ε\\sigma H\_\{\\sigma\}\\leq\\sigma H\_\{0\}\\leq c\\varepsilon\. Thus the theorem’s high\-tolerance sample bound implies \([200](https://arxiv.org/html/2608.06545#A5.E200)\)\.

In addition,NnomN\_\{\\mathrm\{nom\}\}andNrobN\_\{\\mathrm\{rob\}\}are constant fractions ofNN, andεopt≤copt​ε≤1\\varepsilon\_\{\\mathrm\{opt\}\}\\leq c\_\{\\mathrm\{opt\}\}\\varepsilon\\leq 1\. Therefore all supporting lemmas invoked below apply\. Throughout the rest of the proof, we work on the intersection of the high\-probability events from Lemmas[31](https://arxiv.org/html/2608.06545#Thmlemma31)and[32](https://arxiv.org/html/2608.06545#Thmlemma32), using the nominal existence conclusion in the high\-tolerance caseH0<HσH\_\{0\}<H\_\{\\sigma\}and the robust existence conclusion otherwise\. In every case, this intersection has probability at least1−O​\(δ\)1\-O\(\\delta\)\.

##### Candidate validity and selection\.

On the event fixed above, the following two bounds hold for everyγ∈ΓN\\gamma\\in\\Gamma\_\{N\}\. Lemma[32](https://arxiv.org/html/2608.06545#Thmlemma32)gives

ρπ^γ,σ≥LCBγrob⁡\(π^γ\),\\rho^\{\\widehat\{\\pi\}\_\{\\gamma\},\\sigma\}\\geq\\operatorname\{LCB\}\_\{\\gamma\}^\{\\mathrm\{rob\}\}\(\\widehat\{\\pi\}\_\{\\gamma\}\),\(202\)and Lemma[31](https://arxiv.org/html/2608.06545#Thmlemma31)gives

ρπ^γ0,σ≥LCBγnom⁡\(π^γ0\)\.\\rho^\{\\widehat\{\\pi\}\_\{\\gamma\}^\{0\},\\sigma\}\\geq\\operatorname\{LCB\}\_\{\\gamma\}^\{\\mathrm\{nom\}\}\(\\widehat\{\\pi\}\_\{\\gamma\}^\{0\}\)\.\(203\)
The candidate\-existence lemma selected above ensures thatΓN\\Gamma\_\{N\}is nonempty\. The algorithm returns a candidate with the largest lower\-confidence bound\. Therefore

ρπ^,σ≥max⁡\{maxγ∈ΓN⁡LCBγrob⁡\(π^γ\),maxγ∈ΓN⁡LCBγnom⁡\(π^γ0\)\}\.\\rho^\{\\widehat\{\\pi\},\\sigma\}\\geq\\max\\left\\\{\\max\_\{\\gamma\\in\\Gamma\_\{N\}\}\\operatorname\{LCB\}\_\{\\gamma\}^\{\\mathrm\{rob\}\}\(\\widehat\{\\pi\}\_\{\\gamma\}\),\\max\_\{\\gamma\\in\\Gamma\_\{N\}\}\\operatorname\{LCB\}\_\{\\gamma\}^\{\\mathrm\{nom\}\}\(\\widehat\{\\pi\}\_\{\\gamma\}^\{0\}\)\\right\\\}\.\(204\)
To lower\-bound the maximum in \([204](https://arxiv.org/html/2608.06545#A5.E204)\), suppose first that either the low\-tolerance condition holds or the high\-tolerance condition holds together withHσ≤H0H\_\{\\sigma\}\\leq H\_\{0\}\. Lemma[32](https://arxiv.org/html/2608.06545#Thmlemma32)suppliesγ†∈ΓN\\gamma^\{\\dagger\}\\in\\Gamma\_\{N\}satisfying \([201](https://arxiv.org/html/2608.06545#A5.E201)\)\. Hence \([204](https://arxiv.org/html/2608.06545#A5.E204)\) and \([201](https://arxiv.org/html/2608.06545#A5.E201)\) give

ρπ^,σ≥LCBγ†rob⁡\(π^γ†\)≥ρ⋆,σ−ε\.\\rho^\{\\widehat\{\\pi\},\\sigma\}\\geq\\operatorname\{LCB\}\_\{\\gamma^\{\\dagger\}\}^\{\\mathrm\{rob\}\}\(\\widehat\{\\pi\}\_\{\\gamma^\{\\dagger\}\}\)\\geq\\rho^\{\\star,\\sigma\}\-\\varepsilon\.
Suppose instead that the high\-tolerance condition holds andH0<HσH\_\{0\}<H\_\{\\sigma\}\. Lemma[31](https://arxiv.org/html/2608.06545#Thmlemma31)suppliesγnom†∈ΓN\\gamma\_\{\\mathrm\{nom\}\}^\{\\dagger\}\\in\\Gamma\_\{N\}satisfying \([196](https://arxiv.org/html/2608.06545#A5.E196)\)\. Hence \([204](https://arxiv.org/html/2608.06545#A5.E204)\) and \([196](https://arxiv.org/html/2608.06545#A5.E196)\) give

ρπ^,σ≥LCBγnom†nom⁡\(π^γnom†0\)≥ρ⋆,σ−ε\.\\rho^\{\\widehat\{\\pi\},\\sigma\}\\geq\\operatorname\{LCB\}\_\{\\gamma\_\{\\mathrm\{nom\}\}^\{\\dagger\}\}^\{\\mathrm\{nom\}\}\(\\widehat\{\\pi\}\_\{\\gamma\_\{\\mathrm\{nom\}\}^\{\\dagger\}\}^\{0\}\)\\geq\\rho^\{\\star,\\sigma\}\-\\varepsilon\.Thus, in every case,

ρ⋆,σ−ρπ^,σ≤ε\.\\rho^\{\\star,\\sigma\}\-\\rho^\{\\widehat\{\\pi\},\\sigma\}\\leq\\varepsilon\.This completes the proof of Theorem[3](https://arxiv.org/html/2608.06545#Thmtheorem3)\.

### E\.4Proof of auxiliary lemmas

#### E\.4\.1Proof of Lemma[27](https://arxiv.org/html/2608.06545#Thmlemma27)

If𝒢N\\mathcal\{G\}\_\{N\}is empty, the conclusion is vacuous\. Otherwise, fixη∈𝒢N\\eta\\in\\mathcal\{G\}\_\{N\}\. The proof has three steps\. First, we construct a population nominal anchor and simultaneously bound the returned policy’s plug\-in evaluation error and nominal discounted suboptimality\. Second, we use these bounds to prove the population\- and empirical\-span forms \([186b](https://arxiv.org/html/2608.06545#A5.E186.2)\) and \([186a](https://arxiv.org/html/2608.06545#A5.E186.1)\)\. Third, we prove the fixed\-policy bound \([186c](https://arxiv.org/html/2608.06545#A5.E186.3)\) and take a union bound over the discount\-factor grid\.

##### Step 1: Nominal anchor and comparison bound\.

We specialize the proof of Theorem[4](https://arxiv.org/html/2608.06545#Thmtheorem4)toσ=0\\sigma=0\. Use the population nominal anchor

h¯=Vη⋆,0−mins⁡Vη⋆,0​\(s\)​𝟏S,ρ¯=\(1−η\)​maxs⁡Vη⋆,0​\(s\)\.\\bar\{h\}=V\_\{\\eta\}^\{\\star,0\}\-\\min\_\{s\}V\_\{\\eta\}^\{\\star,0\}\(s\)\\bm\{1\}\_\{S\},\\qquad\\bar\{\\rho\}=\(1\-\\eta\)\\max\_\{s\}V\_\{\\eta\}^\{\\star,0\}\(s\)\.For every\(s,a\)∈𝒮×𝒜\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}, the discounted Bellman equation gives

r​\(s,a\)\+Ps,a0​h¯\\displaystyle r\(s,a\)\+P^\{0\}\_\{s,a\}\\bar\{h\}=r​\(s,a\)\+Ps,a0​Vη⋆,0−minx⁡Vη⋆,0​\(x\)\\displaystyle=r\(s,a\)\+P^\{0\}\_\{s,a\}V\_\{\\eta\}^\{\\star,0\}\-\\min\_\{x\}V\_\{\\eta\}^\{\\star,0\}\(x\)≤Vη⋆,0​\(s\)\+\(1−η\)​Ps,a0​Vη⋆,0−minx⁡Vη⋆,0​\(x\)\\displaystyle\\leq V\_\{\\eta\}^\{\\star,0\}\(s\)\+\(1\-\\eta\)P^\{0\}\_\{s,a\}V\_\{\\eta\}^\{\\star,0\}\-\\min\_\{x\}V\_\{\\eta\}^\{\\star,0\}\(x\)≤h¯​\(s\)\+\(1−η\)​maxx⁡Vη⋆,0​\(x\)\\displaystyle\\leq\\bar\{h\}\(s\)\+\(1\-\\eta\)\\max\_\{x\}V\_\{\\eta\}^\{\\star,0\}\(x\)=ρ¯\+h¯​\(s\)\.\\displaystyle=\\bar\{\\rho\}\+\\bar\{h\}\(s\)\.Thus\(ρ¯,h¯\)\(\\bar\{\\rho\},\\bar\{h\}\)satisfies \([6](https://arxiv.org/html/2608.06545#S4.E6)\)\. In the notation of Theorem[4](https://arxiv.org/html/2608.06545#Thmtheorem4), take

Hanc=R0=max⁡\{1,‖Vη⋆,0‖span\}\.H\_\{\\mathrm\{anc\}\}=R\_\{0\}=\\max\\\{1,\\\|V\_\{\\eta\}^\{\\star,0\}\\\|\_\{\\mathrm\{span\}\}\\\}\.The anchor defect then satisfies

β⋆=\(1−η\)​‖Vη⋆,0‖span≤\(1−η\)​R0,\\beta\_\{\\star\}=\(1\-\\eta\)\\\|V\_\{\\eta\}^\{\\star,0\}\\\|\_\{\\mathrm\{span\}\}\\leq\(1\-\\eta\)R\_\{0\},\(205\)and the discounted\-value range givesR0≤\(1−η\)−1R\_\{0\}\\leq\(1\-\\eta\)^\{\-1\}\.

Allocate failure probabilityδ/\|𝒢N\|\\delta/\|\\mathcal\{G\}\_\{N\}\|to this discount factor\. After replacingNNbyNnomN\_\{\\mathrm\{nom\}\}andδ\\deltabyδ/\|𝒢N\|\\delta/\|\\mathcal\{G\}\_\{N\}\|in the proof of Theorem[4](https://arxiv.org/html/2608.06545#Thmtheorem4), its confidence factor becomes

log⁡\(54​S​A​Nnom2​\|𝒢N\|\(1−η\)​δ\)\.\\log\\\!\\left\(\\frac\{54SAN\_\{\\mathrm\{nom\}\}^\{2\}\|\\mathcal\{G\}\_\{N\}\|\}\{\(1\-\\eta\)\\delta\}\\right\)\.This factor already accounts for the union bound over the radius grid in \([65](https://arxiv.org/html/2608.06545#A4.E65)\)\. SinceNnom≤NN\_\{\\mathrm\{nom\}\}\\leq N,\(1−η\)−1≤Nnom\(1\-\\eta\)^\{\-1\}\\leq N\_\{\\mathrm\{nom\}\}, and\|𝒢N\|=O​\(log⁡N\)\|\\mathcal\{G\}\_\{N\}\|=O\(\\log N\), it is at mostC​ιNC\\iota\_\{N\}\.

Suppose first that

Nnom≥C​ιN1−η\.N\_\{\\mathrm\{nom\}\}\\geq C\\frac\{\\iota\_\{N\}\}\{1\-\\eta\}\.Then the recursive comparison conditions \([71](https://arxiv.org/html/2608.06545#A4.E71)\) hold withσ=0\\sigma=0\. Moreover, the empirical solver error1/Nnom1/N\_\{\\mathrm\{nom\}\}is at mostc​\(1−η\)​R0c\(1\-\\eta\)R\_\{0\}\. Apply \([77](https://arxiv.org/html/2608.06545#A4.E77)\) withσ=0\\sigma=0,γ=η\\gamma=\\eta,N=NnomN=N\_\{\\mathrm\{nom\}\}, andεopt=1/Nnom\\varepsilon\_\{\\mathrm\{opt\}\}=1/N\_\{\\mathrm\{nom\}\}\. Applying Young’s inequality to the square\-root term involving the comparison error and then absorbing all terms proportional to that error using \([71](https://arxiv.org/html/2608.06545#A4.E71)\) gives

max⁡\{‖V^ηπ^η0,0−Vηπ^η0,0‖∞,‖Vη⋆,0−Vηπ^η0,0‖∞\}\\displaystyle\\max\\Big\\\{\\\|\\widehat\{V\}\_\{\\eta\}^\{\\widehat\{\\pi\}\_\{\\eta\}^\{0\},0\}\-V\_\{\\eta\}^\{\\widehat\{\\pi\}\_\{\\eta\}^\{0\},0\}\\\|\_\{\\infty\},\\\|V\_\{\\eta\}^\{\\star,0\}\-V\_\{\\eta\}^\{\\widehat\{\\pi\}\_\{\\eta\}^\{0\},0\}\\\|\_\{\\infty\}\\Big\\\}≤C​B⋆​\(R0\)⋅ιNNnom​\(1−η\)2\+C​β⋆⋅ιNNnom​\(1−η\)2\\displaystyle\\qquad\\leq C\\sqrt\{\\frac\{B\_\{\\star\}\(R\_\{0\}\)\\cdot\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\(1\-\\eta\)^\{2\}\}\}\+C\\frac\{\\beta\_\{\\star\}\\cdot\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\(1\-\\eta\)^\{2\}\}\+C​R0⋅ιNNnom​\(1−η\)\+C​1Nnom​\(1−η\)\.\\displaystyle\\qquad\\quad\+C\\frac\{R\_\{0\}\\cdot\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\(1\-\\eta\)\}\+C\\frac\{1\}\{N\_\{\\mathrm\{nom\}\}\(1\-\\eta\)\}\.Here we retain the remainder

C​β⋆⋅ιNNnom​\(1−η\)2C\\frac\{\\beta\_\{\\star\}\\cdot\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\(1\-\\eta\)^\{2\}\}from Young’s inequality instead of invoking the additional sample condition used there to absorb it into the leading square\-root term\. By \([205](https://arxiv.org/html/2608.06545#A5.E205)\) andR0≤\(1−η\)−1R\_\{0\}\\leq\(1\-\\eta\)^\{\-1\}, this remainder and the linear\-radius term are both at most

C​ιNNnom​\(1−η\)2\.C\\frac\{\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\(1\-\\eta\)^\{2\}\}\.The final solver\-error term is absorbed into the same bound becauseιN≥1\\iota\_\{N\}\\geq 1and1−η≤11\-\\eta\\leq 1\. Also, the nominal specialization of the localized budget satisfies

B⋆​\(R0\)=R0\+R0​β⋆\+\(1−η\)​R02≤3​R0\.B\_\{\\star\}\(R\_\{0\}\)=R\_\{0\}\+R\_\{0\}\\beta\_\{\\star\}\+\(1\-\\eta\)R\_\{0\}^\{2\}\\leq 3R\_\{0\}\.Consequently,

max⁡\{‖V^ηπ^η0,0−Vηπ^η0,0‖∞,‖Vη⋆,0−Vηπ^η0,0‖∞\}\\displaystyle\\max\\Big\\\{\\\|\\widehat\{V\}\_\{\\eta\}^\{\\widehat\{\\pi\}\_\{\\eta\}^\{0\},0\}\-V\_\{\\eta\}^\{\\widehat\{\\pi\}\_\{\\eta\}^\{0\},0\}\\\|\_\{\\infty\},\\\|V\_\{\\eta\}^\{\\star,0\}\-V\_\{\\eta\}^\{\\widehat\{\\pi\}\_\{\\eta\}^\{0\},0\}\\\|\_\{\\infty\}\\Big\\\}\(206\)≤C1−η​R0⋅ιNNnom\+C​ιNNnom​\(1−η\)2\.\\displaystyle\\qquad\\leq\\frac\{C\}\{1\-\\eta\}\\sqrt\{\\frac\{R\_\{0\}\\cdot\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\}\}\+C\\frac\{\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\(1\-\\eta\)^\{2\}\}\.If insteadNnom<C​ιN/\(1−η\)N\_\{\\mathrm\{nom\}\}<C\\iota\_\{N\}/\(1\-\\eta\), the second term on the right\-hand side of \([206](https://arxiv.org/html/2608.06545#A5.E206)\) dominates\(1−η\)−1\(1\-\\eta\)^\{\-1\}\. Both norms on the left\-hand side are at most\(1−η\)−1\(1\-\\eta\)^\{\-1\}by the discounted\-value range\. Thus \([206](https://arxiv.org/html/2608.06545#A5.E206)\) holds in this case as well\.

##### Step 2: Optimal\-value error bounds\.

The assumed empirical near\-optimality and the triangle inequality now give

‖V^η⋆,0−Vη⋆,0‖∞\\displaystyle\\\|\\widehat\{V\}\_\{\\eta\}^\{\\star,0\}\-V\_\{\\eta\}^\{\\star,0\}\\\|\_\{\\infty\}≤‖V^η⋆,0−V^ηπ^η0,0‖∞\+‖V^ηπ^η0,0−Vηπ^η0,0‖∞\\displaystyle\\leq\\\|\\widehat\{V\}\_\{\\eta\}^\{\\star,0\}\-\\widehat\{V\}\_\{\\eta\}^\{\\widehat\{\\pi\}\_\{\\eta\}^\{0\},0\}\\\|\_\{\\infty\}\+\\\|\\widehat\{V\}\_\{\\eta\}^\{\\widehat\{\\pi\}\_\{\\eta\}^\{0\},0\}\-V\_\{\\eta\}^\{\\widehat\{\\pi\}\_\{\\eta\}^\{0\},0\}\\\|\_\{\\infty\}\+‖Vηπ^η0,0−Vη⋆,0‖∞\.\\displaystyle\\quad\+\\\|V\_\{\\eta\}^\{\\widehat\{\\pi\}\_\{\\eta\}^\{0\},0\}\-V\_\{\\eta\}^\{\\star,0\}\\\|\_\{\\infty\}\.Substituting \([206](https://arxiv.org/html/2608.06545#A5.E206)\) proves \([186b](https://arxiv.org/html/2608.06545#A5.E186.2)\); the additional1/Nnom1/N\_\{\\mathrm\{nom\}\}solver term is the final term displayed there\.

For the empirical\-span form, apply \([82](https://arxiv.org/html/2608.06545#A4.E82)\) withV1=Vη⋆,0V\_\{1\}=V\_\{\\eta\}^\{\\star,0\}andV2=V^η⋆,0V\_\{2\}=\\widehat\{V\}\_\{\\eta\}^\{\\star,0\}to obtain

R0≤‖V^η⋆,0‖span\+1\+2​‖V^η⋆,0−Vη⋆,0‖∞\.R\_\{0\}\\leq\\\|\\widehat\{V\}\_\{\\eta\}^\{\\star,0\}\\\|\_\{\\mathrm\{span\}\}\+1\+2\\\|\\widehat\{V\}\_\{\\eta\}^\{\\star,0\}\-V\_\{\\eta\}^\{\\star,0\}\\\|\_\{\\infty\}\.Substitute this inequality into \([186b](https://arxiv.org/html/2608.06545#A5.E186.2)\), usex\+y≤x\+y\\sqrt\{x\+y\}\\leq\\sqrt\{x\}\+\\sqrt\{y\}, and apply Young’s inequality to the term containing the square root of‖V^η⋆,0−Vη⋆,0‖∞\\\|\\widehat\{V\}\_\{\\eta\}^\{\\star,0\}\-V\_\{\\eta\}^\{\\star,0\}\\\|\_\{\\infty\}\. After moving one half of this error to the left\-hand side, we obtain \([186a](https://arxiv.org/html/2608.06545#A5.E186.1)\)\.

##### Step 3: Returned\-policy evaluation\.

Finally, two applications of \([82](https://arxiv.org/html/2608.06545#A4.E82)\), together with \([206](https://arxiv.org/html/2608.06545#A5.E206)\), give

R0≤‖V^ηπ^η0,0‖span\+1\+4​max⁡\{‖V^ηπ^η0,0−Vηπ^η0,0‖∞,‖Vη⋆,0−Vηπ^η0,0‖∞\}\.R\_\{0\}\\leq\\\|\\widehat\{V\}\_\{\\eta\}^\{\\widehat\{\\pi\}\_\{\\eta\}^\{0\},0\}\\\|\_\{\\mathrm\{span\}\}\+1\+4\\max\\Big\\\{\\\|\\widehat\{V\}\_\{\\eta\}^\{\\widehat\{\\pi\}\_\{\\eta\}^\{0\},0\}\-V\_\{\\eta\}^\{\\widehat\{\\pi\}\_\{\\eta\}^\{0\},0\}\\\|\_\{\\infty\},\\\|V\_\{\\eta\}^\{\\star,0\}\-V\_\{\\eta\}^\{\\widehat\{\\pi\}\_\{\\eta\}^\{0\},0\}\\\|\_\{\\infty\}\\Big\\\}\.Substituting this inequality back into \([206](https://arxiv.org/html/2608.06545#A5.E206)\) and applying the same Young\-inequality absorption proves \([186c](https://arxiv.org/html/2608.06545#A5.E186.3)\)\. A union bound overη∈𝒢N\\eta\\in\\mathcal\{G\}\_\{N\}completes the proof\.

#### E\.4\.2Proof of Lemma[28](https://arxiv.org/html/2608.06545#Thmlemma28)

By the discounted Bellman equation,

Vλ⋆,0​\(s\)=maxa∈𝒜⁡\{r​\(s,a\)\+λ​Ps,a0​Vλ⋆,0\}\.V\_\{\\lambda\}^\{\\star,0\}\(s\)=\\max\_\{a\\in\\mathcal\{A\}\}\\left\\\{r\(s,a\)\+\\lambda P^\{0\}\_\{s,a\}V\_\{\\lambda\}^\{\\star,0\}\\right\\\}\.By the definition ofhλ0h\_\{\\lambda\}^\{0\},Vλ⋆,0=hλ0\+\(mins′⁡Vλ⋆,0​\(s′\)\)​𝟏SV\_\{\\lambda\}^\{\\star,0\}=h\_\{\\lambda\}^\{0\}\+\(\\min\_\{s^\{\\prime\}\}V\_\{\\lambda\}^\{\\star,0\}\(s^\{\\prime\}\)\)\\bm\{1\}\_\{S\}, so substituting this identity into the preceding display and subtractingλ​mins′⁡Vλ⋆,0​\(s′\)\\lambda\\min\_\{s^\{\\prime\}\}V\_\{\\lambda\}^\{\\star,0\}\(s^\{\\prime\}\)from both sides gives

hλ0​\(s\)\+\(1−λ\)​mins′⁡Vλ⋆,0​\(s′\)=maxa∈𝒜⁡\{r​\(s,a\)\+λ​Ps,a0​hλ0\}\.h\_\{\\lambda\}^\{0\}\(s\)\+\(1\-\\lambda\)\\min\_\{s^\{\\prime\}\}V\_\{\\lambda\}^\{\\star,0\}\(s^\{\\prime\}\)=\\max\_\{a\\in\\mathcal\{A\}\}\\left\\\{r\(s,a\)\+\\lambda P^\{0\}\_\{s,a\}h\_\{\\lambda\}^\{0\}\\right\\\}\.Moreover, the definition ofhλ0h\_\{\\lambda\}^\{0\}implies

0≤hλ0≤‖hλ0‖span​𝟏S,maxs′⁡Vλ⋆,0​\(s′\)=mins′⁡Vλ⋆,0​\(s′\)\+‖hλ0‖span\.0\\leq h\_\{\\lambda\}^\{0\}\\leq\\\|h\_\{\\lambda\}^\{0\}\\\|\_\{\\mathrm\{span\}\}\\bm\{1\}\_\{S\},\\qquad\\max\_\{s^\{\\prime\}\}V\_\{\\lambda\}^\{\\star,0\}\(s^\{\\prime\}\)=\\min\_\{s^\{\\prime\}\}V\_\{\\lambda\}^\{\\star,0\}\(s^\{\\prime\}\)\+\\\|h\_\{\\lambda\}^\{0\}\\\|\_\{\\mathrm\{span\}\}\.Thus, for everys∈𝒮s\\in\\mathcal\{S\}anda∈𝒜a\\in\\mathcal\{A\},

r​\(s,a\)\+Ps,a0​hλ0=r​\(s,a\)\+λ​Ps,a0​hλ0\+\(1−λ\)​Ps,a0​hλ0≤r​\(s,a\)\+λ​Ps,a0​hλ0\+\(1−λ\)​‖hλ0‖span\.r\(s,a\)\+P^\{0\}\_\{s,a\}h\_\{\\lambda\}^\{0\}=r\(s,a\)\+\\lambda P^\{0\}\_\{s,a\}h\_\{\\lambda\}^\{0\}\+\(1\-\\lambda\)P^\{0\}\_\{s,a\}h\_\{\\lambda\}^\{0\}\\leq r\(s,a\)\+\\lambda P^\{0\}\_\{s,a\}h\_\{\\lambda\}^\{0\}\+\(1\-\\lambda\)\\\|h\_\{\\lambda\}^\{0\}\\\|\_\{\\mathrm\{span\}\}\.Taking the maximum overaaand using the preceding identities, we obtain

maxa∈𝒜⁡\{r​\(s,a\)\+Ps,a0​hλ0\}\\displaystyle\\max\_\{a\\in\\mathcal\{A\}\}\\left\\\{r\(s,a\)\+P^\{0\}\_\{s,a\}h\_\{\\lambda\}^\{0\}\\right\\\}≤maxa∈𝒜⁡\{r​\(s,a\)\+λ​Ps,a0​hλ0\}\+\(1−λ\)​‖hλ0‖span\\displaystyle\\leq\\max\_\{a\\in\\mathcal\{A\}\}\\left\\\{r\(s,a\)\+\\lambda P^\{0\}\_\{s,a\}h\_\{\\lambda\}^\{0\}\\right\\\}\+\(1\-\\lambda\)\\\|h\_\{\\lambda\}^\{0\}\\\|\_\{\\mathrm\{span\}\}=hλ0​\(s\)\+\(1−λ\)​mins′⁡Vλ⋆,0​\(s′\)\+\(1−λ\)​‖hλ0‖span\\displaystyle=h\_\{\\lambda\}^\{0\}\(s\)\+\(1\-\\lambda\)\\min\_\{s^\{\\prime\}\}V\_\{\\lambda\}^\{\\star,0\}\(s^\{\\prime\}\)\+\(1\-\\lambda\)\\\|h\_\{\\lambda\}^\{0\}\\\|\_\{\\mathrm\{span\}\}=hλ0​\(s\)\+\(1−λ\)​maxs′⁡Vλ⋆,0​\(s′\)\\displaystyle=h\_\{\\lambda\}^\{0\}\(s\)\+\(1\-\\lambda\)\\max\_\{s^\{\\prime\}\}V\_\{\\lambda\}^\{\\star,0\}\(s^\{\\prime\}\)=hλ0​\(s\)\+ρ¯λ0\.\\displaystyle=h\_\{\\lambda\}^\{0\}\(s\)\+\\bar\{\\rho\}\_\{\\lambda\}^\{0\}\.This is precisely the claimed nominal anchor supersolution inequality\.

#### E\.4\.3Proof of Lemma[29](https://arxiv.org/html/2608.06545#Thmlemma29)

Apply Lemma[27](https://arxiv.org/html/2608.06545#Thmlemma27)with𝒢N=ΛN\\mathcal\{G\}\_\{N\}=\\Lambda\_\{N\}\. The grid and solver conditions required there follow from \([173](https://arxiv.org/html/2608.06545#A5.E173)\) and \([172](https://arxiv.org/html/2608.06545#A5.E172)\)\. Hence, simultaneously for everyλ∈ΛN\\lambda\\in\\Lambda\_\{N\}, the span triangle inequality and Young’s inequality give

\|ρ¯λ0−\(1−λ\)​maxs⁡V^λπ^λ0,0​\(s\)\|≤Canc​\(H^λ0​ιNNnom\+ιNNnom​\(1−λ\)\+εopt\),\\displaystyle\\left\|\\bar\{\\rho\}\_\{\\lambda\}^\{0\}\-\(1\-\\lambda\)\\max\_\{s\}\\widehat\{V\}\_\{\\lambda\}^\{\\widehat\{\\pi\}\_\{\\lambda\}^\{0\},0\}\(s\)\\right\|\\leq C\_\{\\mathrm\{anc\}\}\\left\(\\sqrt\{\\frac\{\\widehat\{H\}\_\{\\lambda\}^\{0\}\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\}\}\+\\frac\{\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\(1\-\\lambda\)\}\+\\varepsilon\_\{\\mathrm\{opt\}\}\\right\),‖Vλ⋆,0−V^λπ^λ0,0‖∞≤Canc2​\(11−λ​H^λ0​ιNNnom\+ιNNnom​\(1−λ\)2\+εopt1−λ\)\.\\displaystyle\\left\\\|V\_\{\\lambda\}^\{\\star,0\}\-\\widehat\{V\}\_\{\\lambda\}^\{\\widehat\{\\pi\}\_\{\\lambda\}^\{0\},0\}\\right\\\|\_\{\\infty\}\\leq\\frac\{C\_\{\\mathrm\{anc\}\}\}\{2\}\\left\(\\frac\{1\}\{1\-\\lambda\}\\sqrt\{\\frac\{\\widehat\{H\}\_\{\\lambda\}^\{0\}\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\}\}\+\\frac\{\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\(1\-\\lambda\)^\{2\}\}\+\\frac\{\\varepsilon\_\{\\mathrm\{opt\}\}\}\{1\-\\lambda\}\\right\)\.Indeed, the solver error transfers the exact empirical bounds in Lemma[27](https://arxiv.org/html/2608.06545#Thmlemma27)to the returned value, while

‖V^λ⋆,0‖span\+1≤2​H^λ0\+2​εopt1−λ;\\\|\\widehat\{V\}\_\{\\lambda\}^\{\\star,0\}\\\|\_\{\\mathrm\{span\}\}\+1\\leq 2\\widehat\{H\}\_\{\\lambda\}^\{0\}\+\\frac\{2\\varepsilon\_\{\\mathrm\{opt\}\}\}\{1\-\\lambda\};the resulting mixed square\-root terms are absorbed by Young’s inequality\. The first display is exactly the baseline certificate after substituting \([174b](https://arxiv.org/html/2608.06545#A5.E174.2)\)\. It also implies the level certificate\. The second display and the span triangle inequality give

Hλ0≤H^λ0\+2​‖Vλ⋆,0−V^λπ^λ0,0‖∞≤H^λ\+\.H\_\{\\lambda\}^\{0\}\\leq\\widehat\{H\}\_\{\\lambda\}^\{0\}\+2\\left\\\|V\_\{\\lambda\}^\{\\star,0\}\-\\widehat\{V\}\_\{\\lambda\}^\{\\widehat\{\\pi\}\_\{\\lambda\}^\{0\},0\}\\right\\\|\_\{\\infty\}\\leq\\widehat\{H\}\_\{\\lambda\}^\{\+\}\.This proves \([187](https://arxiv.org/html/2608.06545#A5.E187)\)\.

We next construct an oracle certificate\. Whenεopt=0\\varepsilon\_\{\\mathrm\{opt\}\}=0, interpretH0/εopt=\+∞H\_\{0\}/\\varepsilon\_\{\\mathrm\{opt\}\}=\+\\infty\. The sample condition and the dyadic grid giveλ†∈ΛN\\lambda^\{\\dagger\}\\in\\Lambda\_\{N\}such that

11−λ†≍min⁡\{Nnom​H0ιN,H0εopt\}\.\\frac\{1\}\{1\-\\lambda^\{\\dagger\}\}\\asymp\\min\\left\\\{\\sqrt\{\\frac\{N\_\{\\mathrm\{nom\}\}H\_\{0\}\}\{\\iota\_\{N\}\}\},\\frac\{H\_\{0\}\}\{\\varepsilon\_\{\\mathrm\{opt\}\}\}\\right\\\}\.Consequently,

\(1−λ†\)​H0\\displaystyle\(1\-\\lambda^\{\\dagger\}\)H\_\{0\}≤C​\(H0​ιNNnom\+εopt\),\\displaystyle\\leq C\\left\(\\sqrt\{\\frac\{H\_\{0\}\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\}\}\+\\varepsilon\_\{\\mathrm\{opt\}\}\\right\),11−λ†​H0​ιNNnom\+ιNNnom​\(1−λ†\)2\+εopt1−λ†\\displaystyle\\frac\{1\}\{1\-\\lambda^\{\\dagger\}\}\\sqrt\{\\frac\{H\_\{0\}\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\}\}\+\\frac\{\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\(1\-\\lambda^\{\\dagger\}\)^\{2\}\}\+\\frac\{\\varepsilon\_\{\\mathrm\{opt\}\}\}\{1\-\\lambda^\{\\dagger\}\}≤C​H0,\\displaystyle\\leq CH\_\{0\},ιNNnom​\(1−λ†\)\\displaystyle\\frac\{\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\(1\-\\lambda^\{\\dagger\}\)\}≤C​H0​ιNNnom\.\\displaystyle\\leq C\\sqrt\{\\frac\{H\_\{0\}\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\}\}\.Lemma[3](https://arxiv.org/html/2608.06545#Thmlemma3), applied withσ=0\\sigma=0, gives‖Vλ†⋆,0‖span≤2​H0\\\|V\_\{\\lambda^\{\\dagger\}\}^\{\\star,0\}\\\|\_\{\\mathrm\{span\}\}\\leq 2H\_\{0\}\. Substituting the preceding bounds into the value comparison above and absorbingC​H0​H^λ†0C\\sqrt\{H\_\{0\}\\widehat\{H\}\_\{\\lambda^\{\\dagger\}\}^\{0\}\}yieldsH^λ†0≤C​H0\\widehat\{H\}\_\{\\lambda^\{\\dagger\}\}^\{0\}\\leq CH\_\{0\}, and henceH^λ†\+≤C​H0\\widehat\{H\}\_\{\\lambda^\{\\dagger\}\}^\{\+\}\\leq CH\_\{0\}\. Moreover, Lemma[9](https://arxiv.org/html/2608.06545#Thmlemma9)gives

ρ¯λ†0≤ρ⋆\+C​\(1−λ†\)​H0\.\\bar\{\\rho\}\_\{\\lambda^\{\\dagger\}\}^\{0\}\\leq\\rho^\{\\star\}\+C\(1\-\\lambda^\{\\dagger\}\)H\_\{0\}\.The baseline comparison and the last three displayed bounds therefore imply

ρ^λ†\+≤ρ⋆\+C​H0​ιNNnom\+C​εopt,\\widehat\{\\rho\}\_\{\\lambda^\{\\dagger\}\}^\{\+\}\\leq\\rho^\{\\star\}\+C\\sqrt\{\\frac\{H\_\{0\}\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\}\}\+C\\varepsilon\_\{\\mathrm\{opt\}\},which proves the oracle assertion\.

It remains to apply the selection rule\. For everyλ∈ΛN\\lambda\\in\\Lambda\_\{N\}, discounted optimality, Lemma[9](https://arxiv.org/html/2608.06545#Thmlemma9), and the three certificate inequalities give

ρ⋆≤ρ¯λ0≤ρ^λ\+,0≤ρ^λ\+−ρ⋆≤Q^λ\.\\rho^\{\\star\}\\leq\\bar\{\\rho\}\_\{\\lambda\}^\{0\}\\leq\\widehat\{\\rho\}\_\{\\lambda\}^\{\+\},\\qquad 0\\leq\\widehat\{\\rho\}\_\{\\lambda\}^\{\+\}\-\\rho^\{\\star\}\\leq\\widehat\{Q\}\_\{\\lambda\}\.Atλ†\\lambda^\{\\dagger\}, the oracle bounds and \([175](https://arxiv.org/html/2608.06545#A5.E175)\) show that

Q^λ†≤C​\(H0​ιNNnom\+εopt\)≤ε\\widehat\{Q\}\_\{\\lambda^\{\\dagger\}\}\\leq C\\left\(\\sqrt\{\\frac\{H\_\{0\}\\iota\_\{N\}\}\{N\_\{\\mathrm\{nom\}\}\}\}\+\\varepsilon\_\{\\mathrm\{opt\}\}\\right\)\\leq\\varepsilonafter enlarging the sample constant and reducingcoptc\_\{\\mathrm\{opt\}\}\. Thusλ†\\lambda^\{\\dagger\}is admissible in \([176](https://arxiv.org/html/2608.06545#A5.E176)\)\. Minimality of the selected span and itsε\\varepsilon\-approximate quality give

H^λ^anc\+≤C​H0,Q^λ^anc≤C​ε\.\\widehat\{H\}\_\{\\widehat\{\\lambda\}\_\{\\mathrm\{anc\}\}\}^\{\+\}\\leq CH\_\{0\},\\qquad\\widehat\{Q\}\_\{\\widehat\{\\lambda\}\_\{\\mathrm\{anc\}\}\}\\leq C\\varepsilon\.The preceding level and quality inequalities, evaluated atλ^anc\\widehat\{\\lambda\}\_\{\\mathrm\{anc\}\}, now give \([189c](https://arxiv.org/html/2608.06545#A5.E189.3)\) and complete the proof\.

#### E\.4\.4Proof of Lemma[30](https://arxiv.org/html/2608.06545#Thmlemma30)

Consider an arbitrary realization of the policyπ\\pi, the anchor, and the radius grid\. Since all three objects are constructed independently of𝒟rob\\mathcal\{D\}\_\{\\mathrm\{rob\}\}, fixing their realization does not change the distribution of the robust\-batch samples\. We first check the logarithmic and sample\-size requirements for the two comparison bounds\. SinceNrob≤NN\_\{\\mathrm\{rob\}\}\\leq Nand\(1−γ\)−1≤N\(1\-\\gamma\)^\{\-1\}\\leq Nforγ∈ΓN\\gamma\\in\\Gamma\_\{N\}, replacing their failure probability byδ/N5\\delta/N^\{5\}changes the logarithmic factor to at most

log⁡\(54​S​A​Nrob2\(1−γ\)​\(δ/N5\)\)≤log⁡\(54​S​A​N8δ\)≤C⋅ιN\.\\log\\left\(\\frac\{54SAN\_\{\\mathrm\{rob\}\}^\{2\}\}\{\(1\-\\gamma\)\(\\delta/N^\{5\}\)\}\\right\)\\leq\\log\\left\(\\frac\{54SAN^\{8\}\}\{\\delta\}\\right\)\\leq C\\cdot\\iota\_\{N\}\.The policy\-grid cutoff \([171](https://arxiv.org/html/2608.06545#A5.E171)\) also gives

Nrob≥CΓ2​σ⋅ιN\(1−γ\)2,Nrob≥CΓ​ιN1−γ\.N\_\{\\mathrm\{rob\}\}\\geq C\_\{\\Gamma\}^\{2\}\\frac\{\\sigma\\cdot\\iota\_\{N\}\}\{\(1\-\\gamma\)^\{2\}\},\\qquad N\_\{\\mathrm\{rob\}\}\\geq C\_\{\\Gamma\}\\frac\{\\iota\_\{N\}\}\{1\-\\gamma\}\.SinceCΓC\_\{\\Gamma\}is sufficiently large, the two conditions in \([71](https://arxiv.org/html/2608.06545#A4.E71)\) hold withNrobN\_\{\\mathrm\{rob\}\}in place ofNNand with the logarithmic factor associated with failure probabilityδ/N5\\delta/N^\{5\}in place ofι\\iota\.

Now fix a radiusRjR\_\{j\}that upper\-bounds11,‖h¯‖span\\\|\\bar\{h\}\\\|\_\{\\mathrm\{span\}\},‖Vγπ,σ‖span\\\|V\_\{\\gamma\}^\{\\pi,\\sigma\}\\\|\_\{\\mathrm\{span\}\}, and‖V^γπ,σ‖span\\\|\\widehat\{V\}\_\{\\gamma\}^\{\\pi,\\sigma\}\\\|\_\{\\mathrm\{span\}\}\. By \([56](https://arxiv.org/html/2608.06545#A4.E56)\),Hanc=max⁡\{1,‖h¯‖span\}H\_\{\\mathrm\{anc\}\}=\\max\\\{1,\\\|\\bar\{h\}\\\|\_\{\\mathrm\{span\}\}\\\}, soRjR\_\{j\}satisfies the radius requirement for the fixed\-policy bound \([72](https://arxiv.org/html/2608.06545#A4.E72)\) from Lemma[13](https://arxiv.org/html/2608.06545#Thmlemma13)\. Hence that bound holds forπ\\piwith failure probabilityO​\(δ/N5\)O\(\\delta/N^\{5\}\)\.

Suppose, in addition, thatπ=πγ⋆\\pi=\\pi\_\{\\gamma\}^\{\\star\}and thatRjR\_\{j\}upper\-bounds‖Vγπ^γ,σ‖span\\\|V\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\},\\sigma\}\\\|\_\{\\mathrm\{span\}\},‖V^γπ^γ,σ‖span\\\|\\widehat\{V\}\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\},\\sigma\}\\\|\_\{\\mathrm\{span\}\}, and‖V^γ⋆,σ‖span\\\|\\widehat\{V\}\_\{\\gamma\}^\{\\star,\\sigma\}\\\|\_\{\\mathrm\{span\}\}\. Thenπ\\piis the fixed optimal policy used in the definitions ofUUandU^\\widehat\{U\}under which Lemma[14](https://arxiv.org/html/2608.06545#Thmlemma14)is stated\. Together with the fixed\-policy radius condition, the displayed bounds ensure that all five spans in \([70](https://arxiv.org/html/2608.06545#A4.E70)\) are at mostRjR\_\{j\}\. Consequently, the learned\-policy bound from Lemma[14](https://arxiv.org/html/2608.06545#Thmlemma14)holds directly at this radius with failure probabilityO​\(δ/N5\)O\(\\delta/N^\{5\}\)\.

There are two comparison bounds at each of theJ\+1=O​\(log⁡N\)J\+1=O\(\\log N\)radii\. A union bound gives a total conditional failure probability of at most

C​\(J\+1\)​δN5≤C​δ​log⁡NN5=O​\(δN3\)\.C\(J\+1\)\\frac\{\\delta\}\{N^\{5\}\}\\leq C\\frac\{\\delta\\log N\}\{N^\{5\}\}=O\\left\(\\frac\{\\delta\}\{N^\{3\}\}\\right\)\.This estimate is uniform over the fixed policy, anchor, and radius grid\. Averaging over their distribution gives the same unconditional probability bound and completes the proof\.

#### E\.4\.5Proof of Lemma[31](https://arxiv.org/html/2608.06545#Thmlemma31)

We first establish the simultaneous evaluation guarantee\. Fixγ∈ΓN\\gamma\\in\\Gamma\_\{N\}and condition on the nominal batch\. Thenπ^γ0\\widehat\{\\pi\}\_\{\\gamma\}^\{0\}is independent of𝒟rob\\mathcal\{D\}\_\{\\mathrm\{rob\}\}\. Write

Dγ=‖V^γπ^γ0,σ−Vγπ^γ0,σ‖∞\.D\_\{\\gamma\}=\\left\\\|\\widehat\{V\}\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\}^\{0\},\\sigma\}\-V\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\}^\{0\},\\sigma\}\\right\\\|\_\{\\infty\}\.Apply Lemma[30](https://arxiv.org/html/2608.06545#Thmlemma30)with the trivial anchor\(ρ¯,h¯\)=\(1,0\)\(\\bar\{\\rho\},\\bar\{h\}\)=\(1,0\)and a dyadic radius grid starting at11and extending beyond8/\(1−γ\)8/\(1\-\\gamma\)\. LetRγR\_\{\\gamma\}be its first point satisfying

Rγ≥R^γnom\+2​Dγ\.R\_\{\\gamma\}\\geq\\widehat\{R\}\_\{\\gamma\}^\{\\mathrm\{nom\}\}\+2D\_\{\\gamma\}\.The span triangle inequality makes this a valid radius, and minimality givesRγ≤2​\(R^γnom\+2​Dγ\)R\_\{\\gamma\}\\leq 2\(\\widehat\{R\}\_\{\\gamma\}^\{\\mathrm\{nom\}\}\+2D\_\{\\gamma\}\)\. Moreover, the 1\-Lipschitz property of\[⋅\]\+\[\\,\\cdot\\,\]\_\{\+\}gives

\[1−\(1−γ\)​mins⁡Vγπ^γ0,σ​\(s\)\]\+≤β^γnom\+\(1−γ\)​Dγ\.\\left\[1\-\(1\-\\gamma\)\\min\_\{s\}V\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\}^\{0\},\\sigma\}\(s\)\\right\]\_\{\+\}\\leq\\widehat\{\\beta\}\_\{\\gamma\}^\{\\mathrm\{nom\}\}\+\(1\-\\gamma\)D\_\{\\gamma\}\.Substitution in the localized budget therefore yields

B​\(Rγ\)≤C​\[B^γnom\+β^γnom​Dγ\+\(σ\+1−γ\)​Dγ2\]\.B\(R\_\{\\gamma\}\)\\leq C\\left\[\\widehat\{B\}\_\{\\gamma\}^\{\\mathrm\{nom\}\}\+\\widehat\{\\beta\}\_\{\\gamma\}^\{\\mathrm\{nom\}\}D\_\{\\gamma\}\+\(\\sigma\+1\-\\gamma\)D\_\{\\gamma\}^\{2\}\\right\]\.After multiplying the localized comparison by1−γ1\-\\gamma, Young’s inequality and the two cutoffs in \([171](https://arxiv.org/html/2608.06545#A5.E171)\) absorb all terms involvingDγD\_\{\\gamma\}on the right\. Thus

\(1−γ\)​Dγ≤penγnom⁡\(π^γ0\),\(1\-\\gamma\)D\_\{\\gamma\}\\leq\\operatorname\{pen\}\_\{\\gamma\}^\{\\mathrm\{nom\}\}\(\\widehat\{\\pi\}\_\{\\gamma\}^\{0\}\),where the inequality follows by choosingCpenC\_\{\\mathrm\{pen\}\}sufficiently large\. The fixed\-policy discounted\-to\-average comparison then gives

ρπ^γ0,σ\\displaystyle\\rho^\{\\widehat\{\\pi\}\_\{\\gamma\}^\{0\},\\sigma\}≥\(1−γ\)​mins⁡Vγπ^γ0,σ​\(s\)\\displaystyle\\geq\(1\-\\gamma\)\\min\_\{s\}V\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\}^\{0\},\\sigma\}\(s\)≥\(1−γ\)​mins⁡V^γπ^γ0,σ​\(s\)−penγnom⁡\(π^γ0\)=LCBγnom⁡\(π^γ0\)\.\\displaystyle\\geq\(1\-\\gamma\)\\min\_\{s\}\\widehat\{V\}\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\}^\{0\},\\sigma\}\(s\)\-\\operatorname\{pen\}\_\{\\gamma\}^\{\\mathrm\{nom\}\}\(\\widehat\{\\pi\}\_\{\\gamma\}^\{0\}\)=\\operatorname\{LCB\}\_\{\\gamma\}^\{\\mathrm\{nom\}\}\(\\widehat\{\\pi\}\_\{\\gamma\}^\{0\}\)\.A union bound over\|ΓN\|=O​\(log⁡N\)\|\\Gamma\_\{N\}\|=O\(\\log N\)proves the simultaneous assertions \([190](https://arxiv.org/html/2608.06545#A5.E190)\) and \([191](https://arxiv.org/html/2608.06545#A5.E191)\)\.

We now prove the existence assertion\. Under \([192](https://arxiv.org/html/2608.06545#A5.E192)\), the two cutoffs definingΓN\\Gamma\_\{N\}exceed2​Cnom​H0/ε2C\_\{\\mathrm\{nom\}\}H\_\{0\}/\\varepsilon\. Hence the dyadic grid containsγnom†\\gamma\_\{\\mathrm\{nom\}\}^\{\\dagger\}satisfying \([193](https://arxiv.org/html/2608.06545#A5.E193)\)\. For brevity writeγ†=γnom†\\gamma^\{\\dagger\}=\\gamma\_\{\\mathrm\{nom\}\}^\{\\dagger\}andπ^†=π^γ†0\\widehat\{\\pi\}^\{\\dagger\}=\\widehat\{\\pi\}\_\{\\gamma^\{\\dagger\}\}^\{0\}\. Lemma[3](https://arxiv.org/html/2608.06545#Thmlemma3), withσ=0\\sigma=0, and Lemma[27](https://arxiv.org/html/2608.06545#Thmlemma27)give

‖Vγ†⋆,0‖span≤2​H0,‖Vγ†⋆,0−Vγ†π^†,0‖∞≤C​H0\.\\\|V\_\{\\gamma^\{\\dagger\}\}^\{\\star,0\}\\\|\_\{\\mathrm\{span\}\}\\leq 2H\_\{0\},\\qquad\\left\\\|V\_\{\\gamma^\{\\dagger\}\}^\{\\star,0\}\-V\_\{\\gamma^\{\\dagger\}\}^\{\\widehat\{\\pi\}^\{\\dagger\},0\}\\right\\\|\_\{\\infty\}\\leq CH\_\{0\}\.The second bound follows by inserting the horizon and sample conditions into the population and fixed\-policy comparisons of that lemma and adding the solver error\. The span triangle inequality and Lemma[9](https://arxiv.org/html/2608.06545#Thmlemma9)now give, after choosingCnomC\_\{\\mathrm\{nom\}\}sufficiently large,

‖Vγ†π^†,0‖span\\displaystyle\\left\\\|V\_\{\\gamma^\{\\dagger\}\}^\{\\widehat\{\\pi\}^\{\\dagger\},0\}\\right\\\|\_\{\\mathrm\{span\}\}≤C​H0,\\displaystyle\\leq CH\_\{0\},\(1−γ†\)​mins⁡Vγ†π^†,0​\(s\)\\displaystyle\(1\-\\gamma^\{\\dagger\}\)\\min\_\{s\}V\_\{\\gamma^\{\\dagger\}\}^\{\\widehat\{\\pi\}^\{\\dagger\},0\}\(s\)≥ρ⋆−c​ε\.\\displaystyle\\geq\\rho^\{\\star\}\-c\\varepsilon\.These are \([195](https://arxiv.org/html/2608.06545#A5.E195)\) and \([194](https://arxiv.org/html/2608.06545#A5.E194)\)\.

The total\-variation constraint, translation equivariance, and monotonicity of the fixed\-policy Bellman operator give

Vγ†π^†,0−γ†​σ1−γ†​‖Vγ†π^†,0‖span​𝟏S≤Vγ†π^†,σ≤Vγ†π^†,0\.V\_\{\\gamma^\{\\dagger\}\}^\{\\widehat\{\\pi\}^\{\\dagger\},0\}\-\\frac\{\\gamma^\{\\dagger\}\\sigma\}\{1\-\\gamma^\{\\dagger\}\}\\left\\\|V\_\{\\gamma^\{\\dagger\}\}^\{\\widehat\{\\pi\}^\{\\dagger\},0\}\\right\\\|\_\{\\mathrm\{span\}\}\\bm\{1\}\_\{S\}\\leq V\_\{\\gamma^\{\\dagger\}\}^\{\\widehat\{\\pi\}^\{\\dagger\},\\sigma\}\\leq V\_\{\\gamma^\{\\dagger\}\}^\{\\widehat\{\\pi\}^\{\\dagger\},0\}\.Together withσ​H0≤c​ε\\sigma H\_\{0\}\\leq c\\varepsilonand\(1−γ†\)−1≍H0/ε\(1\-\\gamma^\{\\dagger\}\)^\{\-1\}\\asymp H\_\{0\}/\\varepsilon, this implies

‖Vγ†π^†,σ‖span≤C​H0\.\\left\\\|V\_\{\\gamma^\{\\dagger\}\}^\{\\widehat\{\\pi\}^\{\\dagger\},\\sigma\}\\right\\\|\_\{\\mathrm\{span\}\}\\leq CH\_\{0\}\.The lower comparison also gives

\(1−γ†\)​mins⁡Vγ†π^†,σ​\(s\)\\displaystyle\(1\-\\gamma^\{\\dagger\}\)\\min\_\{s\}V\_\{\\gamma^\{\\dagger\}\}^\{\\widehat\{\\pi\}^\{\\dagger\},\\sigma\}\(s\)≥\(1−γ†\)​mins⁡Vγ†π^†,0​\(s\)−γ†​σ​‖Vγ†π^†,0‖span\\displaystyle\\geq\(1\-\\gamma^\{\\dagger\}\)\\min\_\{s\}V\_\{\\gamma^\{\\dagger\}\}^\{\\widehat\{\\pi\}^\{\\dagger\},0\}\(s\)\-\\gamma^\{\\dagger\}\\sigma\\left\\\|V\_\{\\gamma^\{\\dagger\}\}^\{\\widehat\{\\pi\}^\{\\dagger\},0\}\\right\\\|\_\{\\mathrm\{span\}\}≥ρ⋆−c​ε−C​σ​H0≥ρ⋆,σ−c​ε,\\displaystyle\\geq\\rho^\{\\star\}\-c\\varepsilon\-C\\sigma H\_\{0\}\\geq\\rho^\{\\star,\\sigma\}\-c\\varepsilon,where the final inequality usesP0∈𝒫P^\{0\}\\in\\mathcal\{P\}and absorbsC​σ​H0C\\sigma H\_\{0\}into the displayedc​εc\\varepsilonloss\.

Finally, setD†=‖V^γ†π^†,σ−Vγ†π^†,σ‖∞D^\{\\dagger\}=\\\|\\widehat\{V\}\_\{\\gamma^\{\\dagger\}\}^\{\\widehat\{\\pi\}^\{\\dagger\},\\sigma\}\-V\_\{\\gamma^\{\\dagger\}\}^\{\\widehat\{\\pi\}^\{\\dagger\},\\sigma\}\\\|\_\{\\infty\}\. The span bound just proved and the definitions of the observable radius, defect, and budget give

R^γ†nom\\displaystyle\\widehat\{R\}\_\{\\gamma^\{\\dagger\}\}^\{\\mathrm\{nom\}\}≤C​H0\+2​D†,\\displaystyle\\leq CH\_\{0\}\+2D^\{\\dagger\},β^γ†nom\\displaystyle\\widehat\{\\beta\}\_\{\\gamma^\{\\dagger\}\}^\{\\mathrm\{nom\}\}≤1,\\displaystyle\\leq 1,B^γ†nom\\displaystyle\\widehat\{B\}\_\{\\gamma^\{\\dagger\}\}^\{\\mathrm\{nom\}\}≤C​\[H0\+D†\+\(σ\+1−γ†\)​\(D†\)2\]\.\\displaystyle\\leq C\\left\[H\_\{0\}\+D^\{\\dagger\}\+\(\\sigma\+1\-\\gamma^\{\\dagger\}\)\(D^\{\\dagger\}\)^\{2\}\\right\]\.Substituting these inequalities into the simultaneous evaluation bound and using Young’s inequality and \([171](https://arxiv.org/html/2608.06545#A5.E171)\) once more gives

\(1−γ†\)​D†≤C​\(H0​ιNNrob\+H0​ιNNrob\+ιNNrob​\(1−γ†\)\)≤c​ε\.\(1\-\\gamma^\{\\dagger\}\)D^\{\\dagger\}\\leq C\\left\(\\sqrt\{\\frac\{H\_\{0\}\\iota\_\{N\}\}\{N\_\{\\mathrm\{rob\}\}\}\}\+\\frac\{H\_\{0\}\\iota\_\{N\}\}\{N\_\{\\mathrm\{rob\}\}\}\+\\frac\{\\iota\_\{N\}\}\{N\_\{\\mathrm\{rob\}\}\(1\-\\gamma^\{\\dagger\}\)\}\\right\)\\leq c\\varepsilon\.It follows in turn thatR^γ†nom≤C​H0\\widehat\{R\}\_\{\\gamma^\{\\dagger\}\}^\{\\mathrm\{nom\}\}\\leq CH\_\{0\},B^γ†nom≤C​H0\\widehat\{B\}\_\{\\gamma^\{\\dagger\}\}^\{\\mathrm\{nom\}\}\\leq CH\_\{0\}, and

penγ†nom⁡\(π^†\)≤c​ε\.\\operatorname\{pen\}\_\{\\gamma^\{\\dagger\}\}^\{\\mathrm\{nom\}\}\(\\widehat\{\\pi\}^\{\\dagger\}\)\\leq c\\varepsilon\.Combining the last two displays with the true robust discounted\-reward bound in the definition \([7](https://arxiv.org/html/2608.06545#S4.E7)\) proves \([196](https://arxiv.org/html/2608.06545#A5.E196)\) and completes the proof\.

#### E\.4\.6Proof of Lemma[32](https://arxiv.org/html/2608.06545#Thmlemma32)

We first prove both observable comparisons through one anchor\-indexed argument\. Fixγ∈ΓN\\gamma\\in\\Gamma\_\{N\}and condition on the nominal batch\. Leta∈\{anc,triv\}a\\in\\\{\\mathrm\{anc\},\\mathrm\{triv\}\\\}index the selected and trivial anchors, and define

a=anc:\(Ha,R^γ,a\)\\displaystyle a=\\mathrm\{anc\}:\\quad\(H\_\{a\},\\widehat\{R\}\_\{\\gamma,a\}\)=\(H^λ^anc\+,R^γ,λ^ancanc\),\\displaystyle=\\left\(\\widehat\{H\}\_\{\\widehat\{\\lambda\}\_\{\\mathrm\{anc\}\}\}^\{\+\},\\widehat\{R\}\_\{\\gamma,\\widehat\{\\lambda\}\_\{\\mathrm\{anc\}\}\}^\{\\mathrm\{anc\}\}\\right\),\(β^γ,a,B^γ,a\)\\displaystyle\(\\widehat\{\\beta\}\_\{\\gamma,a\},\\widehat\{B\}\_\{\\gamma,a\}\)=\(β^γ,λ^anc,B^γ,λ^anc\);\\displaystyle=\\left\(\\widehat\{\\beta\}\_\{\\gamma,\\widehat\{\\lambda\}\_\{\\mathrm\{anc\}\}\},\\widehat\{B\}\_\{\\gamma,\\widehat\{\\lambda\}\_\{\\mathrm\{anc\}\}\}\\right\);a=triv:\(Ha,R^γ,a\)\\displaystyle a=\\mathrm\{triv\}:\\quad\(H\_\{a\},\\widehat\{R\}\_\{\\gamma,a\}\)=\(1,R^γ\),\\displaystyle=\\left\(1,\\widehat\{R\}\_\{\\gamma\}\\right\),\(β^γ,a,B^γ,a\)\\displaystyle\(\\widehat\{\\beta\}\_\{\\gamma,a\},\\widehat\{B\}\_\{\\gamma,a\}\)=\(β^γ,triv,B^γ,triv\)\.\\displaystyle=\\left\(\\widehat\{\\beta\}\_\{\\gamma,\\mathrm\{triv\}\},\\widehat\{B\}\_\{\\gamma,\\mathrm\{triv\}\}\\right\)\.Fora=anca=\\mathrm\{anc\}, use the population anchor\(ρ^λ^anc\+,hλ^anc0\)\(\\widehat\{\\rho\}\_\{\\widehat\{\\lambda\}\_\{\\mathrm\{anc\}\}\}^\{\+\},h\_\{\\widehat\{\\lambda\}\_\{\\mathrm\{anc\}\}\}^\{0\}\)\. It is valid by Lemmas[28](https://arxiv.org/html/2608.06545#Thmlemma28)and[29](https://arxiv.org/html/2608.06545#Thmlemma29), and its span is at mostHaH\_\{a\}\. Fora=triva=\\mathrm\{triv\}, use\(1,0\)\(1,0\), which is valid becauser∈\[0,1\]r\\in\[0,1\]\.

For eitheraa, take a dyadic grid starting atHaH\_\{a\}and letRΔ,aR\_\{\\Delta,a\}be the first point satisfying

RΔ,a≥8​\(R^γ,a\+Δγ\+εopt\)\.R\_\{\\Delta,a\}\\geq 8\\bigl\(\\widehat\{R\}\_\{\\gamma,a\}\+\\Delta\_\{\\gamma\}\+\\varepsilon\_\{\\mathrm\{opt\}\}\\bigr\)\.The discounted\-value range ensures that this point exists on anO​\(log⁡N\)O\(\\log N\)grid, and minimality gives the reverse bound with the factor1616\. The span triangle inequality, the solver guarantee, and \([197](https://arxiv.org/html/2608.06545#A5.E197)\) show thatRΔ,aR\_\{\\Delta,a\}dominates all value spans required in Lemma[30](https://arxiv.org/html/2608.06545#Thmlemma30)\. The true anchor defect also satisfies

β⋆,a≤β^γ,a\+\(1−γ\)​Δγ\.\\beta\_\{\\star,a\}\\leq\\widehat\{\\beta\}\_\{\\gamma,a\}\+\(1\-\\gamma\)\\Delta\_\{\\gamma\}\.Substituting these two comparisons in the generic localized budgets gives, for both anchors,

B⋆,a​\(RΔ,a\)\+Bcom,a​\(RΔ,a\)≤C​\[B^γ,a\+β^γ,a​Δγ\+\(σ\+1−γ\)​Δγ2\]\.B\_\{\\star,a\}\(R\_\{\\Delta,a\}\)\+B\_\{\\mathrm\{com\},a\}\(R\_\{\\Delta,a\}\)\\leq C\\left\[\\widehat\{B\}\_\{\\gamma,a\}\+\\widehat\{\\beta\}\_\{\\gamma,a\}\\Delta\_\{\\gamma\}\+\(\\sigma\+1\-\\gamma\)\\Delta\_\{\\gamma\}^\{2\}\\right\]\.We may chooseRΔ,aR\_\{\\Delta,a\}after observing the robust batch because the event in Lemma[30](https://arxiv.org/html/2608.06545#Thmlemma30)is simultaneous over the radius grid\. Summing its fixed\-policy and learned\-policy comparisons and multiplying by1−γ1\-\\gammatherefore yields

\(1−γ\)​Δγ\\displaystyle\(1\-\\gamma\)\\Delta\_\{\\gamma\}≤C​B^γ,a​ιNNrob\+C​R^γ,a​ιNNrob\+C​εopt\\displaystyle\\leq C\\sqrt\{\\frac\{\\widehat\{B\}\_\{\\gamma,a\}\\iota\_\{N\}\}\{N\_\{\\mathrm\{rob\}\}\}\}\+C\\frac\{\\widehat\{R\}\_\{\\gamma,a\}\\iota\_\{N\}\}\{N\_\{\\mathrm\{rob\}\}\}\+C\\varepsilon\_\{\\mathrm\{opt\}\}\+C​β^γ,a​Δγ​ιNNrob\+C​Δγ​\(σ\+1−γ\)​ιNNrob\+C​Δγ​ιNNrob\+7100​\(1−γ\)​Δγ\.\\displaystyle\\quad\+C\\sqrt\{\\frac\{\\widehat\{\\beta\}\_\{\\gamma,a\}\\Delta\_\{\\gamma\}\\iota\_\{N\}\}\{N\_\{\\mathrm\{rob\}\}\}\}\+C\\Delta\_\{\\gamma\}\\sqrt\{\\frac\{\(\\sigma\+1\-\\gamma\)\\iota\_\{N\}\}\{N\_\{\\mathrm\{rob\}\}\}\}\+C\\frac\{\\Delta\_\{\\gamma\}\\iota\_\{N\}\}\{N\_\{\\mathrm\{rob\}\}\}\+\\frac\{7\}\{100\}\(1\-\\gamma\)\\Delta\_\{\\gamma\}\.Young’s inequality absorbs the first mixed term at the cost ofC​β^γ,a​ιN/\[Nrob​\(1−γ\)\]C\\widehat\{\\beta\}\_\{\\gamma,a\}\\iota\_\{N\}/\[N\_\{\\mathrm\{rob\}\}\(1\-\\gamma\)\]\. The two cutoffs in \([171](https://arxiv.org/html/2608.06545#A5.E171)\) absorb the remaining copies ofΔγ\\Delta\_\{\\gamma\}\. Thus, simultaneously for bothaaand everyγ∈ΓN\\gamma\\in\\Gamma\_\{N\},

\(1−γ\)​Δγ≤C​B^γ,a​ιNNrob\+C​R^γ,a​ιNNrob\+C​β^γ,a​ιNNrob​\(1−γ\)\+C​εopt\.\(1\-\\gamma\)\\Delta\_\{\\gamma\}\\leq C\\sqrt\{\\frac\{\\widehat\{B\}\_\{\\gamma,a\}\\iota\_\{N\}\}\{N\_\{\\mathrm\{rob\}\}\}\}\+C\\frac\{\\widehat\{R\}\_\{\\gamma,a\}\\iota\_\{N\}\}\{N\_\{\\mathrm\{rob\}\}\}\+C\\frac\{\\widehat\{\\beta\}\_\{\\gamma,a\}\\iota\_\{N\}\}\{N\_\{\\mathrm\{rob\}\}\(1\-\\gamma\)\}\+C\\varepsilon\_\{\\mathrm\{opt\}\}\.Takinga=anca=\\mathrm\{anc\}anda=triva=\\mathrm\{triv\}proves \([198](https://arxiv.org/html/2608.06545#A5.E198)\) and \([199](https://arxiv.org/html/2608.06545#A5.E199)\), respectively\.

ChooseCpenC\_\{\\mathrm\{pen\}\}larger than the preceding universal constant\. Both penalties then dominate\(1−γ\)​Δγ\(1\-\\gamma\)\\Delta\_\{\\gamma\}\. Hence Lemma[9](https://arxiv.org/html/2608.06545#Thmlemma9)gives

ρπ^γ,σ\\displaystyle\\rho^\{\\widehat\{\\pi\}\_\{\\gamma\},\\sigma\}≥\(1−γ\)​mins⁡V^γπ^γ,σ​\(s\)−\(1−γ\)​Δγ\\displaystyle\\geq\(1\-\\gamma\)\\min\_\{s\}\\widehat\{V\}\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\},\\sigma\}\(s\)\-\(1\-\\gamma\)\\Delta\_\{\\gamma\}≥\(1−γ\)​mins⁡V^γπ^γ,σ​\(s\)−min⁡\{penγanc⁡\(π^γ\),penγtriv⁡\(π^γ\)\}\\displaystyle\\geq\(1\-\\gamma\)\\min\_\{s\}\\widehat\{V\}\_\{\\gamma\}^\{\\widehat\{\\pi\}\_\{\\gamma\},\\sigma\}\(s\)\-\\min\\left\\\{\\operatorname\{pen\}\_\{\\gamma\}^\{\\mathrm\{anc\}\}\(\\widehat\{\\pi\}\_\{\\gamma\}\),\\operatorname\{pen\}\_\{\\gamma\}^\{\\mathrm\{triv\}\}\(\\widehat\{\\pi\}\_\{\\gamma\}\)\\right\\\}=LCBγrob⁡\(π^γ\)\.\\displaystyle=\\operatorname\{LCB\}\_\{\\gamma\}^\{\\mathrm\{rob\}\}\(\\widehat\{\\pi\}\_\{\\gamma\}\)\.
It remains to construct a good candidate\. The sample condition makes both cutoffs definingΓN\\Gamma\_\{N\}at least2​Crob​Hσ/ε2C\_\{\\mathrm\{rob\}\}H\_\{\\sigma\}/\\varepsilon, so the grid containsγ†\\gamma^\{\\dagger\}with

Crob​Hσε≤11−γ†≤2​Crob​Hσε\.\\frac\{C\_\{\\mathrm\{rob\}\}H\_\{\\sigma\}\}\{\\varepsilon\}\\leq\\frac\{1\}\{1\-\\gamma^\{\\dagger\}\}\\leq\\frac\{2C\_\{\\mathrm\{rob\}\}H\_\{\\sigma\}\}\{\\varepsilon\}\.Use the selected anchor whenH0<HσH\_\{0\}<H\_\{\\sigma\}and the trivial anchor whenHσ≤H0H\_\{\\sigma\}\\leq H\_\{0\}\. In the first case, Lemma[29](https://arxiv.org/html/2608.06545#Thmlemma29)gives

Hanc≤C​H0,β⋆≤C​\(σ​H0\+ε\),H\_\{\\mathrm\{anc\}\}\\leq CH\_\{0\},\\qquad\\beta\_\{\\star\}\\leq C\(\\sigma H\_\{0\}\+\\varepsilon\),whereas in the second caseHanc=1H\_\{\\mathrm\{anc\}\}=1andβ⋆≤1\\beta\_\{\\star\}\\leq 1\. In both cases, withR0=2​HσR\_\{0\}=2H\_\{\\sigma\},

Hanc\+R0​β⋆\+σ​R02\+\(1−γ†\)​R02≤C​\[min⁡\{H0,Hσ\}\+σ​Hσ2\+ε​Hσ\]\.H\_\{\\mathrm\{anc\}\}\+R\_\{0\}\\beta\_\{\\star\}\+\\sigma R\_\{0\}^\{2\}\+\(1\-\\gamma^\{\\dagger\}\)R\_\{0\}^\{2\}\\leq C\\left\[\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\+\\sigma H\_\{\\sigma\}^\{2\}\+\\varepsilon H\_\{\\sigma\}\\right\]\.The sample and optimization conditions therefore verify the hypotheses of Theorem[4](https://arxiv.org/html/2608.06545#Thmtheorem4)\. Together with Lemma[3](https://arxiv.org/html/2608.06545#Thmlemma3), that theorem gives, on an event of probability at least1−O​\(δ\)1\-O\(\\delta\),

‖V^γ†π^γ†,σ−Vγ†π^γ†,σ‖∞\+‖Vγ†⋆,σ−Vγ†π^γ†,σ‖∞≤c​Hσ\.\\left\\\|\\widehat\{V\}\_\{\\gamma^\{\\dagger\}\}^\{\\widehat\{\\pi\}\_\{\\gamma^\{\\dagger\}\},\\sigma\}\-V\_\{\\gamma^\{\\dagger\}\}^\{\\widehat\{\\pi\}\_\{\\gamma^\{\\dagger\}\},\\sigma\}\\right\\\|\_\{\\infty\}\+\\left\\\|V\_\{\\gamma^\{\\dagger\}\}^\{\\star,\\sigma\}\-V\_\{\\gamma^\{\\dagger\}\}^\{\\widehat\{\\pi\}\_\{\\gamma^\{\\dagger\}\},\\sigma\}\\right\\\|\_\{\\infty\}\\leq cH\_\{\\sigma\}\.The robust discounted\-to\-average comparison and the choice ofγ†\\gamma^\{\\dagger\}now imply

\(1−γ†\)​mins⁡V^γ†π^γ†,σ​\(s\)≥ρ⋆,σ−c​ε2\.\(1\-\\gamma^\{\\dagger\}\)\\min\_\{s\}\\widehat\{V\}\_\{\\gamma^\{\\dagger\}\}^\{\\widehat\{\\pi\}\_\{\\gamma^\{\\dagger\}\},\\sigma\}\(s\)\\geq\\rho^\{\\star,\\sigma\}\-\\frac\{c\\varepsilon\}\{2\}\.The same error bound and the span triangle inequality give the following observable bounds\. In the selected\-anchor case,

R^γ†,λ^ancanc\\displaystyle\\widehat\{R\}\_\{\\gamma^\{\\dagger\},\\widehat\{\\lambda\}\_\{\\mathrm\{anc\}\}\}^\{\\mathrm\{anc\}\}≤C​Hσ,\\displaystyle\\leq CH\_\{\\sigma\},β^γ†,λ^anc\\displaystyle\\widehat\{\\beta\}\_\{\\gamma^\{\\dagger\},\\widehat\{\\lambda\}\_\{\\mathrm\{anc\}\}\}≤C​\(σ​H0\+ε\),\\displaystyle\\leq C\(\\sigma H\_\{0\}\+\\varepsilon\),B^γ†,λ^anc\\displaystyle\\widehat\{B\}\_\{\\gamma^\{\\dagger\},\\widehat\{\\lambda\}\_\{\\mathrm\{anc\}\}\}≤C​\(H0\+σ​Hσ2\+ε​Hσ\),\\displaystyle\\leq C\(H\_\{0\}\+\\sigma H\_\{\\sigma\}^\{2\}\+\\varepsilon H\_\{\\sigma\}\),while in the trivial\-anchor case,

R^γ†≤C​Hσ,β^γ†,triv≤1,B^γ†,triv≤C​\(Hσ\+σ​Hσ2\)\.\\widehat\{R\}\_\{\\gamma^\{\\dagger\}\}\\leq CH\_\{\\sigma\},\\qquad\\widehat\{\\beta\}\_\{\\gamma^\{\\dagger\},\\mathrm\{triv\}\}\\leq 1,\\qquad\\widehat\{B\}\_\{\\gamma^\{\\dagger\},\\mathrm\{triv\}\}\\leq C\(H\_\{\\sigma\}\+\\sigma H\_\{\\sigma\}^\{2\}\)\.Substitution in the corresponding penalty, using \([200](https://arxiv.org/html/2608.06545#A5.E200)\), the constant\-fraction sample split, andεopt≤copt​ε\\varepsilon\_\{\\mathrm\{opt\}\}\\leq c\_\{\\mathrm\{opt\}\}\\varepsilon, gives

min⁡\{penγ†anc⁡\(π^γ†\),penγ†triv⁡\(π^γ†\)\}≤c​ε2\.\\min\\left\\\{\\operatorname\{pen\}\_\{\\gamma^\{\\dagger\}\}^\{\\mathrm\{anc\}\}\(\\widehat\{\\pi\}\_\{\\gamma^\{\\dagger\}\}\),\\operatorname\{pen\}\_\{\\gamma^\{\\dagger\}\}^\{\\mathrm\{triv\}\}\(\\widehat\{\\pi\}\_\{\\gamma^\{\\dagger\}\}\)\\right\\\}\\leq\\frac\{c\\varepsilon\}\{2\}\.Combining the last two displays in \([8](https://arxiv.org/html/2608.06545#S4.E8)\) proves \([201](https://arxiv.org/html/2608.06545#A5.E201)\) and completes the proof\.

## Appendix FAdditional experiments and simulation details

This appendix provides supplementary results and implementation details for the experiments in Section[5](https://arxiv.org/html/2608.06545#S5)\. Section[F\.1](https://arxiv.org/html/2608.06545#A6.SS1)describes the instances used to test the sample\-complexity rates and reports an additional minimum\-span experiment\. Section[F\.2](https://arxiv.org/html/2608.06545#A6.SS2)describes the span\-agnostic adaptation experiment\. The experiment repository documents the exact sample\-size grids and trial allocations, the code and data organization, and the reproduction instructions\.

For every instance, we compute the nominal and robust average rewards and bias functions from the underlying AMDP to a tolerance of10−1010^\{\-10\}\. The resulting bias spans are used in all sample\-size calculations, normalizations, and rate fits\. For readability, the figure labels report these spans rounded to the nearest integer\.

At each state\-action pair, the learner receivesNNindependent next\-state samples\. As in the main text,N95N\_\{95\}denotes the sample size at which the estimated probability of returning a robustlyε\\varepsilon\-optimal policy reaches0\.950\.95\. We estimate this crossing by fitting a success curve constrained to be nondecreasing inNN\. For the rate experiments, the95%95\\%intervals forN95N\_\{95\}use500500parametric bootstrap repetitions\. In each repetition, we draw new success counts from the observed success rate at every tested sample size, refit the curve, and recalculate its crossing\.

### F\.1Sample\-complexity experiments

Section[5](https://arxiv.org/html/2608.06545#S5)tests the high\-tolerance rate

min⁡\{H0,Hσ\}ε2\\frac\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\}\{\\varepsilon^\{2\}\}and separately tests the two components of the low\-tolerance rate,

min⁡\{H0,Hσ\}ε2andσ​Hσ2ε2\.\\frac\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\}\{\\varepsilon^\{2\}\}\\qquad\\text\{and\}\\qquad\\frac\{\\sigma H\_\{\\sigma\}^\{2\}\}\{\\varepsilon^\{2\}\}\.Each tested sample size uses at least10001000independent trials, increased to20002000near the estimatedN95N\_\{95\}crossing\.

##### Instance constructions\.

For the high\-tolerance experiment in Figure[1a](https://arxiv.org/html/2608.06545#S4.F1), we use a three\-state AMDP consisting of a decision state and two rewarding states\. The rewarding states have reward11and return to the decision state with probability1/\(2​H0\)1/\(2H\_\{0\}\)\. Of the two informative actions, the better action enters its rewarding state with probability1/\(2​H0\)1/\(2H\_\{0\}\)and has TV radiusσ\\sigmaat the decision state\. The other action has radius zero and is calibrated so that its robust average reward is1\.5​ε1\.5\\varepsilonbelow that of the better action\. A third action self\-loops at the decision state and has reward zero\. This givesH0H\_\{0\}exactly and

Hσ=H01−σ​H0\>H0\.H\_\{\\sigma\}=\\frac\{H\_\{0\}\}\{1\-\\sigma H\_\{0\}\}\>H\_\{0\}\.We fixε=0\.02\\varepsilon=0\.02andσ=2\.5×10−5\\sigma=2\.5\\times 10^\{\-5\}and vary

H0∈\{8,10,12,16,20,28,40,56,80\}\.H\_\{0\}\\in\\\{8,10,12,16,20,28,40,56,80\\\}\.The nominal and robust optimal actions agree in every setting, and7​σ​H0/ε7\\sigma H\_\{0\}/\\varepsilonis at most0\.70\.7, so all settings lie within the formal high\-tolerance condition\.

The low\-tolerance minimum\-span experiment in Figure[1b](https://arxiv.org/html/2608.06545#S4.F1)uses the four\-state layout shown in Figure[4](https://arxiv.org/html/2608.06545#A6.F4)\. This is the core of the lower\-bound instance in Figure[3](https://arxiv.org/html/2608.06545#A3.F3): three actions at a decision state lead to three different rewarding states\. The padding states used to extend the lower\-bound construction to larger state spaces are unnecessary for this comparison and are omitted\. Unlike the lower\-bound proof, where the instances are constructed in pairs, each simulation uses a single instance with a fixed optimal action\. We useε=0\.001\\varepsilon=0\.001for this family\.

SSdecision stateR1R\_\{1\}reward0\.040\.04TV radius0\.0020\.002R2R\_\{2\}reward0\.040\.04R3R\_\{3\}reward0\.040\.04a1:p1a\_\{1\}:p\_\{1\}q1q\_\{1\}a2:p2a\_\{2\}:p\_\{2\}q2q\_\{2\}a3:p3a\_\{3\}:p\_\{3\}q3q\_\{3\}

actionpip\_\{i\}qiq\_\{i\}nominal averagerewardTV radiusatSSTV radius atRiR\_\{i\}\(all actions\)a1a\_\{1\}0\.000300\.000300\.000200\.000200\.0240\.0240\.0020\.0020\.0020\.002a2a\_\{2\}0\.001150\.001150\.000850\.000850\.0230\.02300a3a\_\{3\}0\.001050\.001050\.000950\.000950\.0210\.02100

Figure 4:Representative minimum\-span instance forHσ≤H0H\_\{\\sigma\}\\leq H\_\{0\}, atε=0\.001\\varepsilon=0\.001andσ=0\.002\\sigma=0\.002\. Actionaia\_\{i\}entersRiR\_\{i\}with probabilitypip\_\{i\}, and every action atRiR\_\{i\}returns toSSwith probabilityqiq\_\{i\}; remaining probability is assigned to a self\-loop\. Dashed elements mark positive TV radius\. The spans areH0=80H\_\{0\}=80andHσ=20H\_\{\\sigma\}=20\.For the other settings withHσ≤H0H\_\{\\sigma\}\\leq H\_\{0\},p2p\_\{2\}andq2q\_\{2\}are0\.023/Hσ0\.023/H\_\{\\sigma\}and0\.017/Hσ0\.017/H\_\{\\sigma\}, whilep3p\_\{3\}andq3q\_\{3\}are0\.021/Hσ0\.021/H\_\{\\sigma\}and0\.019/Hσ0\.019/H\_\{\\sigma\}; all other displayed quantities remain fixed\.

To study the minimum\-span termmin⁡\{H0,Hσ\}\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}whenHσ<H0H\_\{\\sigma\}<H\_\{0\}, we fixH0=80H\_\{0\}=80and vary

Hσ∈\{8,10,12,14,16,20,24,28,32,36,40\}\.H\_\{\\sigma\}\\in\\\{8,10,12,14,16,20,24,28,32,36,40\\\}\.Actiona1a\_\{1\}is nominally best, whereas uncertainty makesa2a\_\{2\}robustly optimal\. The difference betweena2a\_\{2\}anda3a\_\{3\}in robust average reward is2​ε2\\varepsilon\. Across these settings,σ​Hσ2\\sigma H\_\{\\sigma\}^\{2\}is at most8\.1%8\.1\\%ofmin⁡\{H0,Hσ\}=Hσ\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}=H\_\{\\sigma\}\. Therefore the expected sample\-complexity dependence is dominated byHσH\_\{\\sigma\}\.

To study the robustness\-specific termσ​Hσ2\\sigma H\_\{\\sigma\}^\{2\}, we instead keepH0H\_\{0\}near55\. We varyHσH\_\{\\sigma\}at fixedσ=0\.1\\sigma=0\.1,

Hσ∈\{40,45,50,60,70,80,90,100,120,140\},H\_\{\\sigma\}\\in\\\{40,45,50,60,70,80,90,100,120,140\\\},and varyσ\\sigmaat fixedHσ=50H\_\{\\sigma\}=50,

σ∈\{0\.02,0\.025,0\.035,0\.05,0\.07,0\.1,0\.14,0\.18,0\.22\}\.\\sigma\\in\\\{0\.02,0\.025,0\.035,0\.05,0\.07,0\.1,0\.14,0\.18,0\.22\\\}\.Across both sets of settings,min⁡\{H0,Hσ\}\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}is at most10%10\\%ofσ​Hσ2\\sigma H\_\{\\sigma\}^\{2\}\. The robustness\-specific component therefore determines most of the predicted sample complexity in these instances\.

##### Normalized success curves\.

The low\-tolerance panels of Figure[1](https://arxiv.org/html/2608.06545#S4.F1)summarize each setting by the single crossingN95N\_\{95\}\. Figure[5](https://arxiv.org/html/2608.06545#A6.F5)shows the corresponding normalized success curves\. Figure[5a](https://arxiv.org/html/2608.06545#A6.F5)plots success againstN​ε2/HσN\\varepsilon^\{2\}/H\_\{\\sigma\}for the minimum\-span settings, while Figure[5b](https://arxiv.org/html/2608.06545#A6.F5)usesN​ε2/\(σ​Hσ2\)N\\varepsilon^\{2\}/\(\\sigma H\_\{\\sigma\}^\{2\}\)for the robustness\-specific settings\. The near alignment within each panel shows that these normalizations capture the main change in the required sample size, supporting the rate

min⁡\{H0,Hσ\}\+σ​Hσ2ε2\\frac\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\+\\sigma H\_\{\\sigma\}^\{2\}\}\{\\varepsilon^\{2\}\}predicted by the theory\.

![Refer to caption](https://arxiv.org/html/2608.06545v1/x3.png)Figure 5:Normalized success curves for the two rate components\. \(a\) The1111minimum\-span settings withHσ≤H0H\_\{\\sigma\}\\leq H\_\{0\}, plotted againstN​ε2/HσN\\varepsilon^\{2\}/H\_\{\\sigma\}\. \(b\) The1212robustness\-specific settings, plotted againstN​ε2/\(σ​Hσ2\)N\\varepsilon^\{2\}/\(\\sigma H\_\{\\sigma\}^\{2\}\)\. The dashed line marks the0\.950\.95target definingN95N\_\{95\}\.
##### Additional experiment in the caseH0<HσH\_\{0\}<H\_\{\\sigma\}\.

Figure[1b](https://arxiv.org/html/2608.06545#S4.F1)considers the caseHσ≤H0H\_\{\\sigma\}\\leq H\_\{0\}, in whichmin⁡\{H0,Hσ\}=Hσ\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}=H\_\{\\sigma\}\. To check this in the complementary scenario whereH0<HσH\_\{0\}<H\_\{\\sigma\}, we fixHσ=50H\_\{\\sigma\}=50and vary

H0∈\{20,23,26,29,32,35,38,41,44\}\.H\_\{0\}\\in\\\{20,23,26,29,32,35,38,41,44\\\}\.This check uses a simpler three\-state instance andε=2×10−5\\varepsilon=2\\times 10^\{\-5\}\. Figure[6](https://arxiv.org/html/2608.06545#A6.F6)shows the setting withH0=32H\_\{0\}=32\.

SSRRreward0\.010\.01TTreward0TV radius9/544009/54400a2:p2a\_\{2\}:p\_\{2\}a3:p3a\_\{3\}:p\_\{3\}qRq\_\{R\}qT0q\_\{T\}^\{0\}a1:1a\_\{1\}:1decision stateFigure 6:Representative complementary minimum\-span instance \(H0=32<Hσ=50H\_\{0\}=32<H\_\{\\sigma\}=50andε=2×10−5\\varepsilon=2\\times 10^\{\-5\}\)\. Actionsa2a\_\{2\}anda3a\_\{3\}enter the rewarding stateRR; the unreachable stateTTchanges the robust bias span without changing their comparison\. Remaining probability is assigned to self\-loops\. The full transition table is recorded in the experiment documentation\.In this setting, Figure[7a](https://arxiv.org/html/2608.06545#A6.F7)again shows an approximately linear increase inN95N\_\{95\}asH0H\_\{0\}increases\. Figure[7b](https://arxiv.org/html/2608.06545#A6.F7)gives the corresponding curve\-level check: after normalizing the sample size byH0​ε−2H\_\{0\}\\varepsilon^\{\-2\}, the success\-probability transitions in different settings nearly align\. Both theHσ≤H0H\_\{\\sigma\}\\leq H\_\{0\}case and theH0<HσH\_\{0\}<H\_\{\\sigma\}case are consistent with the predictedmin⁡\{H0,Hσ\}\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}component in the sample complexity\.

![Refer to caption](https://arxiv.org/html/2608.06545v1/x4.png)Figure 7:Complementary minimum\-span experiment for nine settings withH0<HσH\_\{0\}<H\_\{\\sigma\}\. \(a\)N95N\_\{95\}versusH0H\_\{0\}, with the proportional referenceN95∝H0N\_\{95\}\\propto H\_\{0\}\. \(b\) Success curves normalized byN​ε2/H0N\\varepsilon^\{2\}/H\_\{0\}; the dashed line marks the0\.950\.95target\.

### F\.2Span\-agnostic adaptation

Section[5\.2](https://arxiv.org/html/2608.06545#S5.SS2)illustrates how Algorithm[2](https://arxiv.org/html/2608.06545#alg2)adapts its policy family and effective horizon without knowingH0H\_\{0\}orHσH\_\{\\sigma\}\. This subsection describes the instance, implementation, and simulation design behind Figure[2](https://arxiv.org/html/2608.06545#S4.F2)\.

##### Instance for span\-agnostic adaptation\.

We use a four\-state instance in which actionaia\_\{i\}at the decision stateSSenters rewarding stateRiR\_\{i\}\. With common scaleτ\\tau, the entry probability, return probability, and reward fora1,a2,a3a\_\{1\},a\_\{2\},a\_\{3\}are respectively\(0\.2/τ,0\.04/τ,0\.992\)\(0\.2/\\tau,0\.04/\\tau,0\.992\),\(2/τ,2/τ,1\)\(2/\\tau,2/\\tau,1\), and\(0\.1/τ,0\.02/τ,0\.9916\)\(0\.1/\\tau,0\.02/\\tau,0\.9916\)\. The TV radius isσ\\sigmaat\(S,a1\)\(S,a\_\{1\}\)and at every state\-action pair inR1R\_\{1\}, and is zero elsewhere; unshown probability is assigned to a self\-loop\. Thus the three actions trade off nominal value, robustness, and effective horizon\.

##### Span\-agnostic algorithm implementation\.

We implement Algorithm[2](https://arxiv.org/html/2608.06545#alg2)as stated in Appendix[E](https://arxiv.org/html/2608.06545#A5)\. We divide the samples equally between the two policy families, settingNnom=Nrob=N/2N\_\{\\mathrm\{nom\}\}=N\_\{\\mathrm\{rob\}\}=N/2\.

The theorem leaves three universal numerical constants unspecified\. We use the fixed setting

Canc=2,CΓ=1,Cpen=6\.C\_\{\\mathrm\{anc\}\}=2,\\qquad C\_\{\\Gamma\}=1,\\qquad C\_\{\\mathrm\{pen\}\}=6\.This is an implementation convention, not an estimate of the best possible constants\. The experiment documentation records the complete per\-trial diagnostic schema\.

##### Main adaptation experiment\.

We evaluate this instance and implementation in the experiment behind Figure[2](https://arxiv.org/html/2608.06545#S4.F2)\. The experiment usesε=0\.002\\varepsilon=0\.002and nineH0H\_\{0\}settings obtained from

τ∈\{16,20,25,32,40,50,64,80,100\}\.\\tau\\in\\\{16,20,25,32,40,50,64,80,100\\\}\.For each setting, we test1515values ofσ​H0/ε\\sigma H\_\{0\}/\\varepsilonbetween0\.20\.2and22\. The sample budget per state\-action pair is ten times

min⁡\{H0,Hσ\}\+σ​Hσ2ε2\.\\frac\{\\min\\\{H\_\{0\},H\_\{\\sigma\}\\\}\+\\sigma H\_\{\\sigma\}^\{2\}\}\{\\varepsilon^\{2\}\}\.We use300300trials at ratios between0\.70\.7and1\.31\.3, where the policy\-family transition occurs, and100100trials elsewhere\. Across all26,10026\{,\}100trials, every numerical solver converges, every selected policy is robustlyε\\varepsilon\-optimal, and every selected candidate has a valid lower\-confidence bound\.

For panel[2d](https://arxiv.org/html/2608.06545#S4.F2), we use a separate1919\-point sample grid on five minimum\-span and five robustness\-specific settings\. Both methods receive the same empirical transition counts in each trial\. Each grid point has at least10001000paired trials, with20002000near the twoN95N\_\{95\}crossings\. The uncertainty intervals use10001000paired bootstrap repetitions\. Each repetition resamples the joint outcome at every sample size: both methods succeed, only the span\-agnostic method succeeds, only the span\-informed method succeeds, or neither succeeds\. We then recompute both crossings from the same resampled trials\. The comparison uses270,000270\{,\}000paired trials in total\.