Pure Exploration for a Good Policy in Reinforcement Learning with Bandit Feedback

# Pure Exploration for a Good Policy in Reinforcement Learning with Bandit Feedback
Source: [https://arxiv.org/html/2605.23182](https://arxiv.org/html/2605.23182)
LI ZITIAN Department of Industrial Systems Engineering & Management National University of Singapore Engineering Drive 2 Block E1A \#06\-25 Singapore 117576 lizitian@u\.nus\.edu &Cheung Wang Chi Department of Industrial Systems Engineering & Management Engineering Drive 2 Block E1A \#06\-25 Singapore 117576 isecwc@nus\.edu\.sg

###### Abstract

Pure exploration in episodic Reinforcement Learning has primarily focused on Best Policy Identification \(BPI\), which seeks to identify a \(near\)\-optimal policy with high confidence\. Motivated by practical settings where a “good enough” policy suffices, we study an alternate objective of Good Policy Identification \(GPI\)\. For a given reward thresholdμ0\\mu\_\{0\}, GPI only requires identifying a policy with expected reward in an episode at leastμ0\\mu\_\{0\}if such a policy exists \(positive instance\), or declaring None if no such policy exists \(negative instance\)\. We formalize GPI under the fixed\-confidence setting\. We require the output to be correct with probability≥1−δ\\geq 1\-\\delta, and seek to minimize the expected sample complexity, which is the expected number of episodes explored for the output\. We propose a novel algorithm BEE\-GPI, and derive theoretically\-grounded upper bounds on its sample complexity for positive and negative instances\. Notably, for positive instances, the coefficient oflog⁡1/δ\\log 1/\\deltain our upper bound isO​\(H2/\(V∗−μ0\)2\)O\(H^\{2\}/\(V^\{\*\}\-\\mu\_\{0\}\)^\{2\}\), whereHHis the episode length andV∗V^\{\*\}is the optimal expected reward in an episode\. The coefficient does not depend on the action and state space sizes otherwise, in sharp contrast to the sample complexity in BPI\. We further establish lower bound results to show the near\-optimality of BEE\-GPI and the necessity of the1/\(V∗−μ\)21/\(V^\{\*\}\-\\mu\)^\{2\}term\. Numerical experiments further validate the efficiency of our approach\.

## 1Introduction

Pure exploration is a fundamental objective in Reinforcement Learning \(RL\), where a learning agent seeks to identify a policy that satisfies specific performance criteria\. Unlike the regret minimization framework, pure exploration prioritizes the probability of correct identification and sample efficiency over total episodes\. To date, most research in this field has concentrated on Best Policy Identification \(BPI\) orϵ\\epsilon\-Best Policy Identification \(ϵ\\epsilon\-PI\), leaving the problem of Good Policy Identification \(GPI\) largely unexplored\. In an episodic Markov Decision Process \(MDP\) with unknown transition dynamics, GPI requires the agent to identify a policy whose value function exceeds a pre\-specified thresholdμ0\\mu\_\{0\}, if such a policy exists, or to returnNoneotherwise\.

Many real\-world applications necessitate finding a “good enough” policy rather than a nearly optimal one, particularly when precise pairwise comparisons are computationally or practically difficult\. For instance, in medical informatics, a new treatment protocol is typically evaluated against a knownStandard of Care\(with known utilityμ0\\mu\_\{0\}\)\. The objective is to identify a treatment sequence that ensures a patient recovery rate or “Years of Life Gained” whose utility is≥μ0\\geq\\mu\_\{0\}\. In such cases, identifying the absolute best treatment with a high confidence1−δ1\-\\deltais often unnecessary, especially when the performance differences among top\-tier policies are marginal\. Similarly, in cloud computing, service providers use RL for traffic routing and resource allocation\. Rather than seeking an absolutely optimal routing rule, providers often aim to ensure reliability and efficiency levels that exceed a specified Service Level Agreement\. Furthermore, institutional investors frequently evaluate whether a new trading strategy meets a minimum Sharpe ratio or Return\-on\-Investment \(ROI\) benchmark before deployment\. This benchmarkμ0\\mu\_\{0\}is often derived from established indices with extensive historical data\. In these scenarios, the primary concern is determining whether a new alternative is competitive relative to the benchmark, rather than finding the best strategy among all possibile ones\.

Main Contributions\.We provide three primary contributions\. First, we formalize the Good Policy Identification \(GPI\) problem in the fixed\-confidence setting\. Given a failure probability tolerance parameterδ∈\(0,1\)\\delta\\in\(0,1\), the learning agent must identify a policy with expected reward at least thresholdμ0\\mu\_\{0\}\(or correctly identify its non\-existence\) with probability≥1−δ\\geq 1\-\\delta\. Second, we develop BEE\-GPI, a novelδ\\delta\-PAC algorithm that utilizes a early\-stopping BPI sub\-routine as an oracle to achieve efficient sample complexity\. We provide a rigorous upper bound on its expected sample complexity, demonstrating that for positive instances, the coefficient oflog⁡\(1/δ\)\\log\(1/\\delta\)shrinks with the gap\(V∗−μ0\)\(V^\{\*\}\-\\mu\_\{0\}\), but is independent of the state and action space sizes\. Third, we establish lower bounds for the GPI problem\. The resulting gap between our upper and lower bounds is limited to a polynomial factor ofS,AS,A, andHH, indicating that BEE\-GPI is nearly optimal\.

Notation\.Denoteℕ,ℝ\\mathbb\{N\},\\mathbb\{R\}as the sets of positive integers and real numbers respectively\. Forn∈ℕn\\in\\mathbb\{N\}, denote\[n\]=\{1,…,n\}\[n\]=\\\{1,\\ldots,n\\\}\. We abbreviate a random variableRRfollowing the probability distributionppasR∼pR\\sim p\.

Problem Formulation\.An instance of the Good Policy Identification \(GPI\) problem is specified by the tupleν=\(𝒮,𝒜,μ0,δ,H,R,P,𝗉\)\\nu=\(\\mathcal\{S\},\\mathcal\{A\},\\mu\_\{0\},\\delta,H,R,P,\\mathsf\{p\}\)\. The sets𝒮\\mathcal\{S\}and𝒜\\mathcal\{A\}denote the finite state and action spaces, respectively\. We denoteS,AS,Aas the respective cardinalities of𝒮,𝒜\\mathcal\{S\},\\mathcal\{A\}\. The scalarμ0∈ℝ\\mu\_\{0\}\\in\\mathbb\{R\}is the reward threshold, which is the expected total reward in an episode that the learning agent aims to accrue\. The error parameterδ∈\(0,1\)\\delta\\in\(0,1\)is the tolerance level on the probability of an incorrect output\. The integerHHrepresents the episode horizon\. The transition kernels and reward function are respectively denoted byP=\{ph\(⋅\|s,a\)\}h∈\[H\],s∈𝒮,a∈𝒜P=\\\{p\_\{h\}\(\\cdot\|s,a\)\\\}\_\{h\\in\[H\],s\\in\\mathcal\{S\},a\\in\\mathcal\{A\}\}andR=\{rh​\(s,a\)\}h∈\[H\],s∈𝒮,a∈𝒜R=\\\{r\_\{h\}\(s,a\)\\\}\_\{h\\in\[H\],s\\in\\mathcal\{S\},a\\in\\mathcal\{A\}\}, respectively\. At roundh∈\[H\]h\\in\[H\]when the agent is at states∈𝒮s\\in\\mathcal\{S\}and takes actiona∈𝒜a\\in\\mathcal\{A\}, the agent earns a deterministic rewardrh​\(s,a\)∈\[0,1\]r\_\{h\}\(s,a\)\\in\[0,1\], and transits to statess′s^\{\\prime\}with probabilityph​\(s′\|s,a\)p\_\{h\}\(s^\{\\prime\}\|s,a\)\. The initial stateS1S\_\{1\}is distributed according to the initial distributionp\. We assume𝒮,𝒜,μ0,δ,H,R\\mathcal\{S\},\\mathcal\{A\},\\mu\_\{0\},\\delta,H,Rare known to the agent, whileP,𝗉P,\\mathsf\{p\}are not known\. To facilitate our discussion onP,𝗉P,\\mathsf\{p\}, we prepend each episode with a fictitious time round0\. Round0is associated with one fictitiouss0s\_\{0\}and the single fictitious actiona0a\_\{0\}, which carry zero reward and yield state transitionp0\(⋅\|s0,a0\)=p\(⋅\)p\_\{0\}\(\\cdot\|s\_\{0\},a\_\{0\}\)=\\textsf\{p\}\(\\cdot\)\.

A policyπ\\piis expressed as\(πh\)h=1H∈ΠH\(\\pi\_\{h\}\)^\{H\}\_\{h=1\}\\in\\Pi^\{H\}, whereΠ=\{f\|f:𝒮→𝒜\}\\Pi=\\\{f~\|~f:\\mathcal\{S\}\\rightarrow\\mathcal\{A\}\\\}consists of mapping from the current state to an action\. Thus,ΠH\\Pi^\{H\}is the space of all deterministic policies, andπh\\pi\_\{h\}determines the action in roundh∈\[H\]h\\in\[H\]\. The value function ofπ\\pifrom roundhhto the end of the episode is

Vhπ​\(sh\):=𝔼π​\[∑ℓ=hHrℓ​\(sℓ,πℓ​\(sℓ\)\)∣sh\],\\displaystyle V\_\{h\}^\{\\pi\}\(s\_\{h\}\):=\\mathbb\{E\}^\{\\pi\}\\left\[\\sum\_\{\\ell=h\}^\{H\}r\_\{\\ell\}\(s\_\{\\ell\},\\pi\_\{\\ell\}\(s\_\{\\ell\}\)\)\\mid s\_\{h\}\\right\],wheresℓ\+1∼pℓ\(⋅\|sℓ,πℓ\(sℓ\)\)s\_\{\\ell\+1\}\\sim p\_\{\\ell\}\(\\cdot\|s\_\{\\ell\},\\pi\_\{\\ell\}\(s\_\{\\ell\}\)\)\. Letπ∗∈ΠH\\pi^\{\*\}\\in\\Pi^\{H\}denote an optimal policy\. By Bellman optimality, we haveVhπ∗​\(s\)≥Vhπ​\(s\)V\_\{h\}^\{\\pi^\{\*\}\}\(s\)\\geq V\_\{h\}^\{\\pi\}\(s\)for any policyπ\\piands∈𝒮,h∈\[H\]s\\in\\mathcal\{S\},h\\in\[H\]\. We abbreviateV0∗​\(s0\)=V0π∗​\(s0\)V^\{\*\}\_\{0\}\(s\_\{0\}\)=V^\{\\pi^\{\*\}\}\_\{0\}\(s\_\{0\}\)as the optimal total expected reward in an episode, and occasionally append the instanceν\\nuto denoteV0∗​\(s0\|ν\)V^\{\*\}\_\{0\}\(s\_\{0\}~\|~\\nu\)to emphasize the dependence onν\\nu\.

Dynamics\.The agent’s algorithm is characterized by a sampling rule\{πt\}t=1∞\\\{\\pi^\{t\}\\\}\_\{t=1\}^\{\\infty\}, a stopping timeτ\\tau, and a recommendation ruleπ^\\hat\{\\pi\}\. When episodet∈ℕt\\in\\mathbb\{N\}begins, the algorithm selects a policyπt=\(πht\)h=1H∈ΠH\\pi^\{t\}=\(\\pi\_\{h\}^\{t\}\)\_\{h=1\}^\{H\}\\in\\Pi^\{H\}based on the collected data𝒟t−1\\mathcal\{D\}\_\{t\-1\}in episodes1,…,t−11,\\ldots,t\-1\. Executingπt\\pi^\{t\}on the instance generates the trajectoryzt=\(s0,a0,St,1,At,1,r1​\(St,1,At,1\),…,St,H,At,H,rH​\(St,H,At,H\)\)z\_\{t\}=\(s\_\{0\},a\_\{0\},S\_\{t,1\},A\_\{t,1\},r\_\{1\}\(S\_\{t,1\},A\_\{t,1\}\),\\dots,S\_\{t,H\},A\_\{t,H\},r\_\{H\}\(S\_\{t,H\},A\_\{t,H\}\)\), whereSt,h∼ph−1\(⋅\|St,h−1,πh−1t\(St,h−1\)\)S\_\{t,h\}\\sim p\_\{h\-1\}\(\\cdot\|S\_\{t,h\-1\},\\pi^\{t\}\_\{h\-1\}\(S\_\{t,h\-1\}\)\)\. This trajectory is added to the history, i\.e\.𝒟t=𝒟t−1∪\{zt\}\\mathcal\{D\}\_\{t\}=\\mathcal\{D\}\_\{t\-1\}\\cup\\\{z\_\{t\}\\\}, where we initialize𝒟0=∅\\mathcal\{D\}\_\{0\}=\\emptyset\. At the end of an episode, the agent may choose to terminate the algorithm\. We define the stopping timeτ\\tauas the index of the terminating episode\. Upon termination, the agent outputs an answerπ^∈ΠH∪\{None\}\\hat\{\\pi\}\\in\\Pi^\{H\}\\cup\\\{\\textsf\{None\}\\\}based on𝒟τ\\mathcal\{D\}\_\{\\tau\}\.

The agent aims to output a policyπ^∈ΠH\\hat\{\\pi\}\\in\\Pi^\{H\}satisfyingV0π^​\(s0\)≥μ0V\_\{0\}^\{\\hat\{\\pi\}\}\(s\_\{0\}\)\\geq\\mu\_\{0\}ifμ0\\mu\_\{0\}can be achieved, or to outputNoneif the agent concludes thatV0∗​\(s0\)<μ0V^\{\*\}\_\{0\}\(s\_\{0\}\)<\\mu\_\{0\}\. We formalize the aim in the fixed confidence setting with some definitions that aid our discussions:

###### Definition 1\(Positive and Negative Instances\)\.

A GPI instanceν\\nuis a positive instance ifV0∗​\(s0\|ν\)\>μ0V^\{\*\}\_\{0\}\(s\_\{0\}~\|~\\nu\)\>\\mu\_\{0\}, and is a negative instance ifV0∗​\(s0\|ν\)<μ0V^\{\*\}\_\{0\}\(s\_\{0\}~\|~\\nu\)<\\mu\_\{0\}\.

###### Definition 2\(Qualified Policy\)\.

For a positive instanceν\\nu, we say that a policyπ\\piis qualified forν\\nu, ifV0π​\(s0\|ν\)≥μ0V\_\{0\}^\{\\pi\}\(s\_\{0\}~\|~\\nu\)\\geq\\mu\_\{0\}\.

The correctness of the output is qualified by beingδ\\delta\-Probably Approximately Correct \(δ\\delta\-PAC\):

###### Definition 3\(δ\\delta\-PAC\)\.

An algorithm is calledδ\\delta\-PAC, if it can guarantee

\{Pr⁡\(V0π^τ​\(s0\|ν\)≥μ0,τ<\+∞\)≥1−δif​V0∗​\(s0\|ν\)\>μ0Pr⁡\(π^τ=None,τ<\+∞\)≥1−δif​V0∗​\(s0\|ν\)<μ0\\displaystyle\\begin\{cases\}\\Pr\(V\_\{0\}^\{\\hat\{\\pi\}\_\{\\tau\}\}\(s\_\{0\}~\|~\\nu\)\\geq\\mu\_\{0\},\\tau<\+\\infty\)\\geq 1\-\\delta&\\text\{ if \}V^\{\*\}\_\{0\}\(s\_\{0\}~\|~\\nu\)\>\\mu\_\{0\}\\\\ \\Pr\(\\hat\{\\pi\}\_\{\\tau\}=\\textsf\{None\},\\tau<\+\\infty\)\\geq 1\-\\delta&\\text\{ if \}V^\{\*\}\_\{0\}\(s\_\{0\}~\|~\\nu\)<\\mu\_\{0\}\\end\{cases\}holds for any positive or negative Good Policy Identification instanceν\\nu\.

Degenne and Koolen \([2019](https://arxiv.org/html/2605.23182#bib.bib31)\)have proved that, on an instanceν\\nuwithV0∗​\(s0\|ν\)=μ0V\_\{0\}^\{\*\}\(s\_\{0\}\|\\nu\)=\\mu\_\{0\}, anyδ\\delta\-PAC algorithm suffers from𝔼ν​\[τ\]=∞\\mathbb\{E\}\_\{\\nu\}\[\\tau\]=\\infty, even when we restrictν\\nuto the case of\|𝒮\|=H=1\|\\mathcal\{S\}\|=H=1\. Thus, we only consider positive or negative instances in this paper\.

Objective\.The goal of the agent is to design aδ\\delta\-PAC algorithm\(\{πt\}t=1∞,τ,π^\)\(\\\{\\pi^\{t\}\\\}\_\{t=1\}^\{\\infty\},\\tau,\\hat\{\\pi\}\)that minimizes the expected sampling complexity𝔼ν​\[τ\]\\mathbb\{E\}\_\{\\nu\}\[\\tau\]\.

## 2Literature Review

A popular objective of pure exploration in Reinforcement Learning \(RL\) is to identify an optimal orϵ\\epsilon\-optimal policy with high confidence\. This line of research, often termed Best Policy Identification \(BPI\), was pioneered byFiechter \([1994](https://arxiv.org/html/2605.23182#bib.bib23)\)\. Much of the subsequent work assumes the availability of a generative model, which allows the agent to interact with the environment by querying arbitrary state\-action pairsKearns and Singh \([1998](https://arxiv.org/html/2605.23182#bib.bib24)\); Gheshlaghi Azaret al\.\([2013](https://arxiv.org/html/2605.23182#bib.bib25)\); Sidfordet al\.\([2018](https://arxiv.org/html/2605.23182#bib.bib27),[2023](https://arxiv.org/html/2605.23182#bib.bib28)\); Agarwalet al\.\([2020](https://arxiv.org/html/2605.23182#bib.bib26)\); Al Marjaniet al\.\([2021](https://arxiv.org/html/2605.23182#bib.bib7)\); Liet al\.\([2024](https://arxiv.org/html/2605.23182#bib.bib29)\)\. While the generative setting is theoretically insightful, a more practical and challenging branch of BPI focuses on the online \(non\-generative\) setting, where the agent must navigate the MDP through sequential episodes to collect data\. In this context,Dann and Brunskill \([2015](https://arxiv.org/html/2605.23182#bib.bib14)\)proposed theUCFHalgorithm based on the principle of optimism under uncertainty\. This was followed byBPI\-UCBVIMénardet al\.\([2021](https://arxiv.org/html/2605.23182#bib.bib20)\), which improved the dependence on the state spaceSSby adapting minimax\-optimal regret algorithms\.Al Marjani and Proutiere \([2021](https://arxiv.org/html/2605.23182#bib.bib6)\)study BPI for RL in non\-episodic MDPs\. More recently, research has pivoted toward instance\-dependent sample complexity\. While algorithms like those inWagenmakeret al\.\([2022](https://arxiv.org/html/2605.23182#bib.bib8)\)andNaranget al\.\([2024](https://arxiv.org/html/2605.23182#bib.bib1)\)achieve state\-of\-the\-art non\-asymptotic bounds, they are often computationally intensive\. To bridge this gap, computationally efficient but theoretically suboptimal optimistic sampling rules have been analyzed byKaufmannet al\.\([2021](https://arxiv.org/html/2605.23182#bib.bib9)\); Ménardet al\.\([2021](https://arxiv.org/html/2605.23182#bib.bib20)\); Tirinzoniet al\.\([2023](https://arxiv.org/html/2605.23182#bib.bib22)\)\.

Despite the extensive literature on BPI andϵ\\epsilon\-PI, these methods cannot be directly applied to the Good Policy Identification \(GPI\) problem\. Recall that forϵ\\epsilon\-PI, the goal is to identify a policy whose expected reward is at leastV0∗​\(s0\)−ϵV^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\epsilon\. The difficulty lies in the fact thatV0∗​\(s0\)V^\{\*\}\_\{0\}\(s\_\{0\}\)is unknown; consequently, an agent cannot pre\-determine an appropriateϵ\\epsilonfor anϵ\\epsilon\-PI algorithm to ensure that the identified policy achieves the thresholdμ0\\mu\_\{0\}\. Furthermore, the lower bounds established forϵ\\epsilon\-PIDomingueset al\.\([2021](https://arxiv.org/html/2605.23182#bib.bib21)\)do not apply under the threshold\-based requirements of GPI\.

In the simplified case whereS=1S=1andH=1H=1, GPI reduces to the multi\-armed bandit problem, specifically the “1\-identification” or “any\-low\(high\)” problem studied inDegenne and Koolen \([2019](https://arxiv.org/html/2605.23182#bib.bib31)\); Katz\-Samuels and Jamieson \([2020](https://arxiv.org/html/2605.23182#bib.bib30)\); Li and Cheung \([2025](https://arxiv.org/html/2605.23182#bib.bib18),[2026](https://arxiv.org/html/2605.23182#bib.bib33)\), also covered but not mainly focused inKanoet al\.\([2017](https://arxiv.org/html/2605.23182#bib.bib32)\); Jourdanet al\.\([2026](https://arxiv.org/html/2605.23182#bib.bib34)\)\. While these works provide different strategies for the bandit setting, the transition to the full episodic MDP setting remains unexplored\.

## 3Algorithms and Sample Complexity Upper Bounds

We propose the algorithm dubbed Balanced Exploration\-Exploitation for Good Policy Identification \(BEE\-GPI\), displayed in Algorithm[1](https://arxiv.org/html/2605.23182#alg1)\. We elaborate on BEE\-GPI in Section[3\.1](https://arxiv.org/html/2605.23182#S3.SS1), and discuss its theoretical performance guarantees in Section[3\.2](https://arxiv.org/html/2605.23182#S3.SS2)\.

### 3\.1Algorithm BEE\-GPI

BEE\-GPI, displayed in Algorithm[1](https://arxiv.org/html/2605.23182#alg1), proceeds in phases\. Each phasek=1,2,…k=1,2,\\ldotsconsists of an adaptive exploration stage \(Line[3](https://arxiv.org/html/2605.23182#alg1.l3)\) that estimatesP,𝗉P,\\mathsf\{p\}, and potentially outputs a policyπ^k\\hat\{\\pi\}\_\{k\}\. The exploration stage is followed by a decision branching \(Line[4](https://arxiv.org/html/2605.23182#alg1.l4)to[10](https://arxiv.org/html/2605.23182#alg1.l10)\), which leads to three possibilities: \(1\) the termination of BEE\-GPI, or \(2\) an exploitation stage for policy evaluation \(Lines[10](https://arxiv.org/html/2605.23182#alg1.l10)\-[15](https://arxiv.org/html/2605.23182#alg1.l15)\), or \(3\) proceeding to the next phasek\+1k\+1\. We elaborate on them in the following three main steps, preceded by a preliminary set\-up step 0\.

Algorithm 1Balanced Exploration\-Exploitation for Good Policy Identification \(BEE\-GPI\)1:Initialize:For

k∈ℕk\\in\\mathbb\{N\},

ϵk=1/2k\\epsilon\_\{k\}=1/2^\{k\},

δk=1/3k\\delta\_\{k\}=1/3^\{k\}and

αk=5k\\alpha\_\{k\}=5^\{k\}\. Initialize

ℋee=ℋet=∅,C=1\.01\\mathcal\{H\}^\{\\text\{ee\}\}=\\mathcal\{H\}^\{\\text\{et\}\}=\\emptyset,C=1\.01\.

2:forPhase

k=1,2,…k=1,2,\\ldotsdo

3:Explore:Call Algorithm[2](https://arxiv.org/html/2605.23182#alg2)with

k←kk\\leftarrow k,

δ←δk\\delta\\leftarrow\\delta\_\{k\},

C←CC\\leftarrow C,

ℋee←\\mathcal\{H\}^\{\\text\{ee\}\}\\leftarrowcurrent exploration history

ℋee\\mathcal\{H\}^\{\\text\{ee\}\}, exploration budget

Tkee←\([5](https://arxiv.org/html/2605.23182#S3.E5)\)T\_\{k\}^\{\\text\{ee\}\}\\leftarrow\(\\ref\{eqn:def\-T\_k\-ee\}\), Return Updated exploration history

ℋee\\mathcal\{H\}^\{\\text\{ee\}\}, output

π^k∈ΠH∪\{None,Not Completed\}\\hat\{\\pi\}\_\{k\}\\in\\Pi^\{H\}\\cup\\\{\\textsf\{None\},\\textsf\{Not Completed\}\\\}
4:if

π^k=Not Completed\\hat\{\\pi\}\_\{k\}=\\textsf\{Not Completed\}then

5:Go to the next phase

k\+1k\+1
6:elseif

π^k=None\\hat\{\\pi\}\_\{k\}=\\textsf\{None\}and

δk≥δ/10\\delta\_\{k\}\\geq\\delta/10then

7:Go to the next phase

k\+1k\+1
8:elseif

π^k=None\\hat\{\\pi\}\_\{k\}=\\textsf\{None\}and

δk<δ/10\\delta\_\{k\}<\\delta/10then

9:ReturnNone, and declare the instance as negative\.

10:elseif

π^k∈ΠH\\hat\{\\pi\}\_\{k\}\\in\\Pi^\{H\}then

11:Exploit:Denote

\{Xik,et\}i=1\+∞\\\{X\_\{i\}^\{k,\\text\{et\}\}\\\}\_\{i=1\}^\{\+\\infty\}as i\.i\.d\. random variables, where the

Xik,etX\_\{i\}^\{k,\\text\{et\}\}is the total reward collected in executing policy

π^k\\hat\{\\pi\}\_\{k\}for an episode in the

ii\-th trial\.

\{Xik,et\}i=1\+∞\\\{X\_\{i\}^\{k,\\text\{et\}\}\\\}\_\{i=1\}^\{\+\\infty\}are independent of

ℋee,ℋet\\mathcal\{H\}^\{\\text\{ee\}\},\\mathcal\{H\}^\{\\text\{et\}\}, and denote

V^etπ^k,N=∑i=1NXik,etN\\hat\{V\}\_\{\\text\{et\}\}^\{\\hat\{\\pi\}\_\{k\},N\}=\\frac\{\\sum\_\{i=1\}^\{N\}X\_\{i\}^\{k,\\text\{et\}\}\}\{N\}\. Initialize

N=0N=0\.

12:while

N≤100⋅log⁡αkδ\+log⁡log⁡24​H2ϵkϵk−1N\\leq 100\\cdot\\frac\{\\log\\frac\{\\alpha\_\{k\}\}\{\\delta\}\+\\log\\log\\frac\{24H^\{2\}\}\{\\epsilon\_\{k\}\}\}\{\\epsilon\_\{k\}\}\-1do

13:Execute

π^k\\hat\{\\pi\}\_\{k\}for one episode, and observe the total reward

XN\+1k,etX\_\{N\+1\}^\{k,\\text\{et\}\},

N←N\+1N\\leftarrow N\+1\.

14:if

V^etπ^k,N−H2​log⁡\[2​αk​\(log2⁡2​N\)2/δ\]N≥μ0\\hat\{V\}\_\{\\text\{et\}\}^\{\\hat\{\\pi\}\_\{k\},N\}\-\\sqrt\{\\frac\{H^\{2\}\\log\[2\\alpha\_\{k\}\(\\log\_\{2\}2N\)^\{2\}/\\delta\]\}\{N\}\}\\geq\\mu\_\{0\}then

15:Output

π^=π^k\\hat\{\\pi\}=\\hat\{\\pi\}\_\{k\}, and declare the instance as positive\.

16:endif

17:endwhile

18:

ℋet←ℋet∪\{Xik,et\}i=1N\\mathcal\{H\}^\{\\text\{et\}\}\\leftarrow\\mathcal\{H\}^\{\\text\{et\}\}\\cup\\\{X\_\{i\}^\{k,\\text\{et\}\}\\\}\_\{i=1\}^\{N\}
19:endif

20:endfor

Step 0: Initialization\.BEE\-GPI starts by initializing the hyper\-parameters in Line[1](https://arxiv.org/html/2605.23182#alg1.l1)\. For the exploration stage in phasekk, the parameterϵk\\epsilon\_\{k\}upper\-bounds the optimality gap of the potentially output policyπ^k\\hat\{\\pi\}\_\{k\}, andδk\\delta\_\{k\}is the tolerance level on the probability of an incorrect output\. The parameterαk\\alpha\_\{k\}is for tuning the length of the potential exploration stage in phasekk\. The setℋee\\mathcal\{H\}^\{\\text\{ee\}\}is for accumulating the observed sample trajectories during the exploration stages\.ℋet\\mathcal\{H\}^\{\\text\{et\}\}is for collecting the total realized rewards by executing a candidate policyπ^k\\hat\{\\pi\}\_\{k\}, during the exploitation stages\. The absolute constantCCis for tuning the stopping condition of the exploration stage\.

Step 1: Exploration Stage\.The exploration stage involves invoking Algorithm[2](https://arxiv.org/html/2605.23182#alg2), which follows the optimistic sampling strategy in the BPI\-UCRL frameworkKaufmannet al\.\([2021](https://arxiv.org/html/2605.23182#bib.bib9)\)\. Different fromKaufmannet al\.\([2021](https://arxiv.org/html/2605.23182#bib.bib9)\), Algorithm[2](https://arxiv.org/html/2605.23182#alg2)involves a novel early\-stopping mechanism, which allows saving on the sample complexity for positive instances, and hence dubbed Early\-Stopping BPI\-UCRL \(ES\-BPI\-UCRL\)\.

We first describe the optimistic strategy, which is warm\-started with the accumulated exploration trajectory setℋee\\mathcal\{H\}^\{\\text\{ee\}\}\. To ease the notation, we denoteℋee=\{zq\}q=1t\\mathcal\{H\}^\{\\text\{ee\}\}=\\\{z\_\{q\}\\\}^\{t\}\_\{q=1\}, wherettis the total number of episodes in the previous exploration stages\. FollowingKaufmannet al\.\([2021](https://arxiv.org/html/2605.23182#bib.bib9)\), we optimistically estimate theVVfunction by the following sequence of construction\. For eachs∈𝒮,a∈𝒜s\\in\\mathcal\{S\},a\\in\\mathcal\{A\}, we setnht\(s,a,\)=∑q=1t𝟏\(Sq,h=s,Aq,h=a\)n\_\{h\}^\{t\}\(s,a,\)=\\sum^\{t\}\_\{q=1\}\\mathbf\{1\}\(S\_\{q,h\}=s,A\_\{q,h\}=a\)\. For eachh∈\[H\],s∈𝒮,a∈𝒜h\\in\[H\],s\\in\\mathcal\{S\},a\\in\\mathcal\{A\}, set the empirical transitional kernel afterttepisode asp^ht​\(s′\|s,a\)=1nht​\(s,a\)​∑q=1t𝟏​\(Sq,h=s,Aq,h,Sq,h\+1=s′\)\\hat\{p\}\_\{h\}^\{t\}\(s^\{\\prime\}\|s,a\)=\\frac\{1\}\{n^\{t\}\_\{h\}\(s,a\)\}\\sum^\{t\}\_\{q=1\}\\mathbf\{1\}\(S\_\{q,h\}=s,A\_\{q,h\},S\_\{q,h\+1\}=s^\{\\prime\}\)ifnht​\(s,a\)\>0n\_\{h\}^\{t\}\(s,a\)\>0, and setp^ht​\(s′\|s,a\)=1/\|𝒮\|\\hat\{p\}\_\{h\}^\{t\}\(s^\{\\prime\}\|s,a\)=1/\|\\mathcal\{S\}\|otherwise\. We define the exploration bonusβp​\(t,δ\)=log⁡2​S​A​Hδ\+\(S−1\)​log⁡\(e​\(1\+tS−1\)\)\\beta\_\{p\}\(t,\\delta\)=\\log\\frac\{2SAH\}\{\\delta\}\+\(S\-1\)\\log\\big\(e\(1\+\\frac\{t\}\{S\-1\}\)\\big\), followingKaufmannet al\.\([2021](https://arxiv.org/html/2605.23182#bib.bib9)\)\. We further define the confidence sets for the transition kernels as:𝒞ht\(s,a;δ\):=\{q\(⋅\|s,a\)∈Δ𝒮:KL\(p^ht\(⋅\|s,a\),q\(⋅\|s,a\)\)≤βp​\(nht​\(s,a\),δ\)nht​\(s,a\)\}\.\\mathcal\{C\}\_\{h\}^\{t\}\(s,a;\\delta\):=\\Big\\\{q\(\\cdot\|s,a\)\\in\\Delta^\{\\mathcal\{S\}\}:\\text\{KL\}\\big\(\\widehat\{p\}\_\{h\}^\{t\}\(\\cdot\|s,a\),q\(\\cdot\|s,a\)\\big\)\\leq\\frac\{\\beta\_\{p\}\(n\_\{h\}^\{t\}\(s,a\),\\delta\)\}\{n\_\{h\}^\{t\}\(s,a\)\}\\Big\\\}\.For each pair\(s,h\)∈𝒮×\[H\]\(s,h\)\\in\\mathcal\{S\}\\times\[H\],δ∈\(0,1\)\\delta\\in\(0,1\), we define the optimistic value functionV¯ht\\overline\{V\}^\{t\}\_\{h\}and the corresponding optimistic policyπ¯t​\(⋅;δ\)=\(π¯ht​\(⋅;δ\)\)h=1H\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\)=\(\\overline\{\\pi\}^\{t\}\_\{h\}\(\\cdot;\\delta\)\)^\{H\}\_\{h=1\}as:

Q¯ht​\(s,a;δ\):=\\displaystyle\\overline\{Q\}\_\{h\}^\{t\}\(s,a;\\delta\):=rh​\(s,a\)\+maxp¯h\(⋅\|s,a\)∈𝒞ht\(s,a;δ\)​∑s′p¯h​\(s′\|s,a\)​V¯h\+1t​\(s′;δ\)\\displaystyle r\_\{h\}\(s,a\)\+\\max\_\{\\bar\{p\}\_\{h\}\(\\cdot\|s,a\)\\in\\mathcal\{C\}\_\{h\}^\{t\}\(s,a;\\delta\)\}\\sum\_\{s^\{\\prime\}\}\\bar\{p\}\_\{h\}\(s^\{\\prime\}\|s,a\)\\overline\{V\}\_\{h\+1\}^\{t\}\(s^\{\\prime\};\\delta\)\(1\)V¯ht​\(s;δ\):=\\displaystyle\\overline\{V\}^\{t\}\_\{h\}\(s;\\delta\):=maxa∈𝒜⁡Q¯ht​\(s,a;δ\),π¯ht​\(s;δ\)∈argmaxa∈𝒜​\{Q¯ht​\(s,a;δ\)\}\\displaystyle\\max\_\{a\\in\\mathcal\{A\}\}\\overline\{Q\}\_\{h\}^\{t\}\(s,a;\\delta\),\\quad\\overline\{\\pi\}^\{t\}\_\{h\}\(s;\\delta\)\\in\\underset\{a\\in\\mathcal\{A\}\}\{\\text\{argmax\}\}\\left\\\{\\overline\{Q\}\_\{h\}^\{t\}\(s,a;\\delta\)\\right\\\}\(2\)whereQ¯H\+1t​\(s,a;δ\):=0,∀s∈𝒮,a∈𝒜,δ∈\(0,1\)\\overline\{Q\}^\{t\}\_\{H\+1\}\(s,a;\\delta\):=0,\\forall s\\in\\mathcal\{S\},a\\in\\mathcal\{A\},\\delta\\in\(0,1\)\. The optimisticQ¯ht\\overline\{Q\}\_\{h\}^\{t\}guides our adaptive exploration in each episode\. Upon the conclusion of an episode, the exploration historyℋee\\mathcal\{H\}^\{\\text\{ee\}\}and the episode indexttare updated\. Then, the oracle checks whether any of the three stopping criteria are met \(Line[3](https://arxiv.org/html/2605.23182#alg2.l3)\)\.

We next discuss the novel early\-stopping mechanism of ES\-BPI\-UCRL, Recallπ¯t​\(⋅;δ\)=\{π¯ht​\(⋅;δ\)\}h=1H\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\)=\\\{\\overline\{\\pi\}^\{t\}\_\{h\}\(\\cdot;\\delta\)\\\}^\{H\}\_\{h=1\}defined in \([2](https://arxiv.org/html/2605.23182#S3.E2)\)\. We set a lower confidence bound on the value function:

V¯ht,π¯t​\(⋅;δ\)​\(s;δ\):=\\displaystyle\\underline\{V\}\_\{h\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\)\}\(s;\\delta\):=rh​\(s,π¯ht​\(s;δ\)\)\+minp¯h∈𝒞ht​\(s,a;δ\)​∑s′p¯h​\(s′\|s,π¯ht​\(s;δ\)\)​V¯h\+1t,π¯t​\(⋅;δ\)​\(s′;δ\),\\displaystyle r\_\{h\}\(s,\\overline\{\\pi\}^\{t\}\_\{h\}\(s;\\delta\)\)\+\\min\_\{\\underline\{p\}\_\{h\}\\in\\mathcal\{C\}\_\{h\}^\{t\}\(s,a;\\delta\)\}\\sum\_\{s^\{\\prime\}\}\\underline\{p\}\_\{h\}\(s^\{\\prime\}\|s,\\overline\{\\pi\}^\{t\}\_\{h\}\(s;\\delta\)\)\\underline\{V\}\_\{h\+1\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\)\}\(s^\{\\prime\};\\delta\),where we initializeV¯H\+1t,π¯t​\(⋅;δ\)​\(s;δ\)=0\\underline\{V\}\_\{H\+1\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\)\}\(s;\\delta\)=0for allss\. These definitions allow us to specify the following two stopping criteria for the oracle routine \(Algorithm[2](https://arxiv.org/html/2605.23182#alg2)\) with tolerance levelδk\\delta\_\{k\}:

V¯0t,π¯t​\(⋅;δk\)​\(s0;δk\)−\(C−1\)​\(V¯0t​\(s0;δk\)−V¯0t,π¯t​\(⋅;δk\)​\(s0;δk\)\)\>\\displaystyle\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)\-\(C\-1\)\(\\overline\{V\}^\{t\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)\-\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)\)\>μ0,\\displaystyle\\mu\_\{0\},\(3\)V¯0t​\(s0;δk\)<\\displaystyle\\overline\{V\}^\{t\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)<μ0\.\\displaystyle\\mu\_\{0\}\.\(4\)The left hand side of \([3](https://arxiv.org/html/2605.23182#S3.E3)\) is a carefully crafted lower confidence bound of the value function of policyπ¯t​\(⋅;δ\)\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\), while the left hand side of \([4](https://arxiv.org/html/2605.23182#S3.E4)\) represents an upper confidence bound of the value function of policyπ¯t​\(⋅;δ\)\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\), which is also the upper confidence bound ofV0∗​\(s0\)V^\{\*\}\_\{0\}\(s\_\{0\}\)\. The stopping criteria also involve a budget constraintTkeeT^\{\\text\{ee\}\}\_\{k\}on the cumulative number of exploration episodes conducted so far:

Tkee=\(H\+1\)2​S​A​log⁡\(2​S​A​H/δk\)ϵk\+\(H\+1\)2​S2​Aϵk​log⁡\(\(H\+1\)2​S2​A​log⁡\(2​S​A​H/δk\)ϵk\)\.T\_\{k\}^\{\\text\{ee\}\}=\\frac\{\(H\+1\)^\{2\}SA\\log\(2SAH/\\delta\_\{k\}\)\}\{\{\\epsilon\_\{k\}\}\}\+\\frac\{\(H\+1\)^\{2\}S^\{2\}A\}\{\\epsilon\_\{k\}\}\\log\\left\(\\frac\{\(H\+1\)^\{2\}S^\{2\}A\\log\(2SAH/\\delta\_\{k\}\)\}\{\\epsilon\_\{k\}\}\\right\)\.\(5\)The oracle checks whether With probability1−δk1\-\\delta\_\{k\}: \(i\) condition \([3](https://arxiv.org/html/2605.23182#S3.E3)\) impliesπ¯tk​\(⋅;δk\)\\overline\{\\pi\}^\{t\_\{k\}\}\(\\cdot;\\delta\_\{k\}\)is a qualified policy \(π^k=π¯tk​\(⋅;δk\)\\hat\{\\pi\}\_\{k\}=\\overline\{\\pi\}^\{t\_\{k\}\}\(\\cdot;\\delta\_\{k\}\)\); \(ii\) condition \([4](https://arxiv.org/html/2605.23182#S3.E4)\) implies no qualified policy exists \(π^k=None\\hat\{\\pi\}\_\{k\}=\\textsf\{None\}\); and \(iii\) exceeding the budget \(t\>Tkee−1t\>T\_\{k\}^\{\\text\{ee\}\}\-1\) indicates that the current exploration stage was insufficient\.

Algorithm 2Exploration Oracle: Early\-Stopping BPI\-UCRL \(ES\-BPI\-UCRL\)1:Input:Phase index

kk, Error tolerance level

δk\\delta\_\{k\}, tuning parameter

C\>1C\>1, accumulated exploration stage trajectories

ℋee\\mathcal\{H\}^\{\\text\{ee\}\}, total budget

TkeeT\_\{k\}^\{\\text\{ee\}\}\.

2:Initialize:round index

t=\|ℋee\|t=\|\\mathcal\{H\}^\{\\text\{ee\}\}\|,

nht​\(s,a\)=∑q=1t𝟏​\(Sq,h=s,Aq,h=a\)n\_\{h\}^\{t\}\(s,a\)=\\sum\_\{q=1\}^\{t\}\\mathbf\{1\}\(S\_\{q,h\}=s,A\_\{q,h\}=a\),

\{Sq,h,Aq,h\}q∈\[t\],h∈\[H\]\\\{S\_\{q,h\},A\_\{q,h\}\\\}\_\{q\\in\[t\],h\\in\[H\]\}are the trajectories stored in

ℋee\\mathcal\{H\}^\{\\text\{ee\}\}\.

3:while

¬\\neg\([3](https://arxiv.org/html/2605.23182#S3.E3)\) and

¬\\neg\([4](https://arxiv.org/html/2605.23182#S3.E4)\) and

t≤Tkee−1t\\leq T\_\{k\}^\{\\text\{ee\}\}\-1do

4:forround

h=0,1,2​⋯,Hh=0,1,2\\cdots,Hdo

5:Execute

π¯ht​\(St\+1,h;δk\)\\overline\{\\pi\}^\{t\}\_\{h\}\(S\_\{t\+1,h\};\\delta\_\{k\}\)by taking action

At\+1,h=maxa⁡Q¯ht​\(St\+1,h,a;δk\)A\_\{t\+1,h\}=\\max\_\{a\}\\overline\{Q\}^\{t\}\_\{h\}\(S\_\{t\+1,h\},a;\\delta\_\{k\}\)⊳\\trianglerightNoteSt\+1,0=s0,At\+1,0=a0S\_\{t\+1,0\}=s\_\{0\},A\_\{t\+1,0\}=a\_\{0\}

6:Observe reward

rh​\(St\+1,h,At\+1,h\)r\_\{h\}\(S\_\{t\+1,h\},A\_\{t\+1,h\}\)and next state

St\+1,h\+1S\_\{t\+1,h\+1\}
7:endfor

8:Update

ℋee←ℋee∪\{\(s0,a0,\{St\+1,h,At\+1,h,rh​\(St\+1,h,At\+1,h\)\}h=1H\)\}\\mathcal\{H\}^\{\\text\{ee\}\}\\leftarrow\\mathcal\{H\}^\{\\text\{ee\}\}\\cup\\\{\(s\_\{0\},a\_\{0\},\\\{S\_\{t\+1,h\},A\_\{t\+1,h\},r\_\{h\}\(S\_\{t\+1,h\},A\_\{t\+1,h\}\)\\\}^\{H\}\_\{h=1\}\)\\\}
9:Update

t←t\+1t\\leftarrow t\+1\.

10:endwhile

11:Denote

tk←tt\_\{k\}\\leftarrow tfor analysis\.⊳\\trianglerightTo clarify the definition oftkt\_\{k\},tk=\|ℋee\|t\_\{k\}=\|\\mathcal\{H\}^\{\\text\{ee\}\}\|also holds

12:if\([3](https://arxiv.org/html/2605.23182#S3.E3)\) holdsthen

13:Output a policy

π^k=\(π¯ht\(⋅\|δk\)\)h=1H\\hat\{\\pi\}\_\{k\}=\(\\overline\{\\pi\}^\{t\}\_\{h\}\(\\cdot\|\\delta\_\{k\}\)\)^\{H\}\_\{h=1\}
14:elseif\([4](https://arxiv.org/html/2605.23182#S3.E4)\) holdsthen

15:Output

π^k=None\\hat\{\\pi\}\_\{k\}=\\textsf\{None\}
16:elseif

t\>Tkee−1t\>T\_\{k\}^\{\\text\{ee\}\}\-1then

17:Output

π^k=Not Completed\\hat\{\\pi\}\_\{k\}=\\textsf\{Not Completed\}
18:endif

Step 2: Decision Branching\.Based on the outputπ^k\\hat\{\\pi\}\_\{k\}and the currentδk\\delta\_\{k\}, Algorithm[1](https://arxiv.org/html/2605.23182#alg1)executes one of four logic branches\. \(i\) Ifπ^k=Not Completed\\hat\{\\pi\}\_\{k\}=\\textsf\{Not Completed\}\(Line[4](https://arxiv.org/html/2605.23182#alg1.l4)\), the phasekkexploration stage is inconclusive, and BEE\-GPI proceeds phasek\+1k\+1with an increased budgetTk\+1eeT^\{\\text\{ee\}\}\_\{k\+1\}and decreased tolerance levelδk\+1\\delta\_\{k\+1\}to continue exploration\. \(ii\) Ifπ^k=None\\hat\{\\pi\}\_\{k\}=\\textsf\{None\}andδk≥δ/10\\delta\_\{k\}\\geq\\delta/10, BEE\-GPI still advances to phasek\+1k\+1but not terminate with a result\. This ensures theδ\\delta\-PAC requirement is met, as BEE\-GPI only recommendsNonewhen the confidence is sufficiently high \(δk<δ/10\\delta\_\{k\}<\\delta/10, the third branch at Line[8](https://arxiv.org/html/2605.23182#alg1.l8)\)\. \(iv\) Ifπ^k\\hat\{\\pi\}\_\{k\}is a policy \(Line[10](https://arxiv.org/html/2605.23182#alg1.l10)\), BEE\-GPI proceeds to the exploitation phase\.

Step 3: Exploitation and Verification\.The exploitation phase validates whether the candidate policyπ^k\\hat\{\\pi\}\_\{k\}is indeed qualified\. Treating the total reward per episode as a random variable, the algorithm constructs a confidence interval with a coverage probability of1−π26​δαk1\-\\frac\{\\pi^\{2\}\}\{6\}\\frac\{\\delta\}\{\\alpha\_\{k\}\}\. By collectingNNindependent samples, the algorithm evaluates the lower confidence bound:V^etπ^k,N−H2​log⁡\(2​αk​\(log2⁡2​N\)2/δ\)/N\\hat\{V\}\_\{\\text\{et\}\}^\{\\hat\{\\pi\}\_\{k\},N\}\-\\sqrt\{H^\{2\}\\log\(2\\alpha\_\{k\}\(\\log\_\{2\}2N\)^\{2\}/\\delta\)/N\}\. To ensure efficient sample complexity, we require thatN≤Tket−1N\\leq T\_\{k\}^\{\\text\{et\}\}\-1, where

Tket=\\displaystyle T\_\{k\}^\{\\text\{et\}\}=100⋅log⁡\(αk/δ\)\+log⁡log⁡\(24​H2/ϵk\)ϵk\.\\displaystyle 100\\cdot\\frac\{\\log\(\\alpha\_\{k\}/\\delta\)\+\\log\\log\(24H^\{2\}/\\epsilon\_\{k\}\)\}\{\\epsilon\_\{k\}\}\.\(6\)If the lower confidence bound exceedsμ0\\mu\_\{0\}within the budgetN≤Tket−1N\\leq T\_\{k\}^\{\\text\{et\}\}\-1, BEE\-GPI recommendsπ^k\\hat\{\\pi\}\_\{k\}as a qualified policy\. Otherwise, it returns to the exploration phase in phasek\+1k\+1\.

### 3\.2Performance Guarantee of BEE\-GPI, Algorithm[1](https://arxiv.org/html/2605.23182#alg1)

We now establish the theoretical performance of Algorithm[1](https://arxiv.org/html/2605.23182#alg1)\. We first prove that our algorithm satisfies the standard PAC requirement\.

###### Theorem 1\.

Algorithm[1](https://arxiv.org/html/2605.23182#alg1)isδ\\delta\-PAC\.

The following theorems provide upper bounds on the expected sample complexity𝔼​\[τ\]\\mathbb\{E\}\[\\tau\]for both positive and negative instances\.

###### Theorem 2\.

On a positive instanceν\\nuwhereV0∗​\(s0\)\>μ0V^\{\*\}\_\{0\}\(s\_\{0\}\)\>\\mu\_\{0\}, BEE\-GPI \(Algorithm[1](https://arxiv.org/html/2605.23182#alg1)\) satisfies

𝔼​\[τ\]≤O​\(H2​log⁡1δ\(V0∗​\(s0\)−μ0\)2\)\+O​\(\(H\+1\)4​S​A​log⁡S​A​H\(V0∗​\(s0\)−μ0\)2\+\(H\+1\)4​S2​A​log⁡\(H​S​A\)\(V0∗​\(s0\)−μ0\)2\)\.\\displaystyle\\mathbb\{E\}\[\\tau\]\\leq O\\left\(\\frac\{H^\{2\}\\log\\frac\{1\}\{\\delta\}\}\{\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\+O\\left\(\\frac\{\(H\+1\)^\{4\}SA\\log\\frac\{SAH\}\{\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\+\(H\+1\)^\{4\}S^\{2\}A\\log\\left\(HSA\\right\)\}\{\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\.

###### Theorem 3\.

On a negative instanceν\\nuwhereV0∗​\(s0\)<μ0V^\{\*\}\_\{0\}\(s\_\{0\}\)<\\mu\_\{0\}, BEE\-GPI \(Algorithm[1](https://arxiv.org/html/2605.23182#alg1)\) satisfies

𝔼​\[τ\]≤O​\(\(H\+1\)4​\[S​A​log⁡1δ\+S2​A​log⁡log⁡1δ\]\(V0∗​\(s0\)−μ0\)2\)\\displaystyle\\mathbb\{E\}\[\\tau\]\\leq O\\left\(\\frac\{\(H\+1\)^\{4\}\[SA\\log\\frac\{1\}\{\\delta\}\+S^\{2\}A\\log\\log\\frac\{1\}\{\\delta\}\]\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\+\\displaystyle\+O​\(\(H\+1\)4​S​A​log⁡H2\(V0∗​\(s0\)−μ0\)2\(V0∗​\(s0\)−μ0\)2\+\(H\+1\)4​S2​A​log⁡log⁡\(H3​S​A\(V0∗​\(s0\)−μ0\)2\)\(V0∗​\(s0\)−μ0\)2\)\.\\displaystyle O\\left\(\\frac\{\(H\+1\)^\{4\}SA\\log\\frac\{H^\{2\}\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\+\\frac\{\(H\+1\)^\{4\}S^\{2\}A\\log\\log\\left\(\\frac\{H^\{3\}SA\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\.

Theorems[1](https://arxiv.org/html/2605.23182#Thmtheorem1),[2](https://arxiv.org/html/2605.23182#Thmtheorem2),[3](https://arxiv.org/html/2605.23182#Thmtheorem3)are proved in Appendix[B\.1](https://arxiv.org/html/2605.23182#A2.SS1), and we provide their sketch proof after their discussions\. The complexity results reveal an inherent asymmetry between positive and negative GPI instances, stemming from the mechanics of our exploration\-exploitation stages\. For positive instances, the sample complexity is decomposed into aδ\\delta\-dependent termO​\(H2​log⁡\(1/δ\)\(V0∗​\(s0\)−μ0\)2\)O\\left\(\\frac\{H^\{2\}\\log\(1/\\delta\)\}\{\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)and aδ\\delta\-independent termO​\(poly​\(S,A,H\)\(V0∗​\(s0\)−μ0\)2\)O\\left\(\\frac\{\\text\{poly\}\(S,A,H\)\}\{\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\. Intuitively, the exploration stage first identifies a candidate policyπ^k\\hat\{\\pi\}\_\{k\}that is “sufficiently good” \(i\.e\.,V0π^k​\(s0\)−μ0≥C−1C​\(V0∗​\(s0\)−μ0\)V^\{\\hat\{\\pi\}\_\{k\}\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\\geq\\frac\{C\-1\}\{C\}\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)\) at a low confidence level, guaranteed by the stopping rule \([3](https://arxiv.org/html/2605.23182#S3.E3)\)\. The effect of adapting the stopping rules is in Lemma[13](https://arxiv.org/html/2605.23182#Thmtheorem13)\. The exploitation stage then usesO​\(H2​log⁡\(1/δ\)\(V0∗​\(s0\)−μ0\)2\)O\\left\(\\frac\{H^\{2\}\\log\(1/\\delta\)\}\{\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)samples to verify this candidate with high confidence\. Notably, the coefficient oflog⁡\(1/δ\)\\log\(1/\\delta\)is independent ofSSandAA, which is a significant improvement over standardϵ\\epsilon\-optimal BPI reductions by takingϵ=V0∗​\(s0\)−μ0\\epsilon=V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}, where the coefficient oflog⁡\(1/δ\)\\log\(1/\\delta\)is typically polynomial inH,S,AH,S,Aand1/ϵ2=1/\(V0∗​\(s0\)−μ0\)21/\\epsilon^\{2\}=1/\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\.

For negative instances, the algorithm must certify thateverypolicy is unqualified\. This requires the oracle to globally explore the MDP to ensure the upper confidence bound of the optimal value function is belowμ0\\mu\_\{0\}\. Because this verification requires checking all state\-action pairs\(h,s,a\)\(h,s,a\), thelog⁡\(1/δ\)\\log\(1/\\delta\)term remains coupled with a polynomial ofSSandAA\.

Sketch Proof of Theorems[1](https://arxiv.org/html/2605.23182#Thmtheorem1),[2](https://arxiv.org/html/2605.23182#Thmtheorem2),[3](https://arxiv.org/html/2605.23182#Thmtheorem3)\.We defineℰcnt​\(δk\)\\mathcal\{E\}^\{\\text\{cnt\}\}\(\\delta\_\{k\}\)andℰ​\(δk\)\\mathcal\{E\}\(\\delta\_\{k\}\)as the concentration events respectively onnhtn^\{t\}\_\{h\},p^ht\\hat\{p\}^\{t\}\_\{h\}for thekk\-th exploration stage, andℰket\\mathcal\{E\}\_\{k\}^\{\\text\{et\}\}as the concentration event for the exploitation stage at phasekkon evaluating the objective value underπ^k\\hat\{\\pi\}^\{k\}\. For a formal definition of the concentration event, we refer the reader to Appendix[B\.1](https://arxiv.org/html/2605.23182#A2.SS1)\. A key feature of our algorithm is that sampling trajectories are shared across all exploration periods; consequently, these events are nested, i\.e\.,ℰcnt​\(δk\)⊂ℰcnt​\(δk\+1\)\\mathcal\{E\}^\{\\text\{cnt\}\}\(\\delta\_\{k\}\)\\subset\\mathcal\{E\}^\{\\text\{cnt\}\}\(\\delta\_\{k\+1\}\)andℰ​\(δk\)⊂ℰ​\(δk\+1\)\\mathcal\{E\}\(\\delta\_\{k\}\)\\subset\\mathcal\{E\}\(\\delta\_\{k\+1\}\)\.

To analyze the termination round of the algorithm, we introduce the random variableκ\\kappa, representing the first phase in which the exploration concentration events hold:

κ=min⁡\{k:ℰ​\(δk\)∩ℰcnt​\(δk\)​hold\}\.\\displaystyle\\kappa=\\min\\\{k:\\mathcal\{E\}\(\\delta\_\{k\}\)\\cap\\mathcal\{E\}^\{\\text\{cnt\}\}\(\\delta\_\{k\}\)\\text\{ hold\}\\\}\.Furthermore, we defineκendpos\\kappa\_\{\\text\{end\}\}^\{\\text\{pos\}\}andκendneg\\kappa\_\{\\text\{end\}\}^\{\\text\{neg\}\}as the indices of the phases by which the algorithm is guaranteed to terminate for positive and negative instances, respectively:

κendpos=\\displaystyle\\kappa\_\{\\text\{end\}\}^\{\\text\{pos\}\}=min⁡\{k:k≥max⁡\{κ,O​\(1\)​log2⁡H2\(V0∗​\(s0\)−μ0\)2\},ℰket​holds\},\\displaystyle\\min\\left\\\{k:k\\geq\\max\\left\\\{\\kappa,O\(1\)\\log\_\{2\}\\frac\{H^\{2\}\}\{\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\\\},\\mathcal\{E\}\_\{k\}^\{\\text\{et\}\}\\text\{ holds\}\\right\\\},κendneg=\\displaystyle\\kappa\_\{\\text\{end\}\}^\{\\text\{neg\}\}=max⁡\{κ,O​\(1\)​log2⁡H2\(V0∗​\(s0\)−μ0\)2,log3⁡O​\(1\)δ\}\.\\displaystyle\\max\\left\\\{\\kappa,O\(1\)\\log\_\{2\}\\frac\{H^\{2\}\}\{\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\},\\log\_\{3\}\\frac\{O\(1\)\}\{\\delta\}\\right\\\}\.
Lemma[7](https://arxiv.org/html/2605.23182#Thmtheorem7)\(Appendix[B\.2](https://arxiv.org/html/2605.23182#A2.SS2)\) establishes that Algorithm[1](https://arxiv.org/html/2605.23182#alg1)terminates no later than the conclusion of phaseκendpos\\kappa\_\{\\text\{end\}\}^\{\\text\{pos\}\}\(for positiveν\\nu\) orκendneg\\kappa\_\{\\text\{end\}\}^\{\\text\{neg\}\}\(for negativeν\\nu\)\. This lemma provides the foundation for our three main results\.

Correctness \(δ\\delta\-PAC Guarantee\)\.First, we establish that the algorithm terminates almost surely\. Specifically, the probabilityPrν⁡\(τ=\+∞\)\\Pr\_\{\\nu\}\(\\tau=\+\\infty\)is bounded by the probability that the stopping phasesκend\\kappa\_\{\\text\{end\}\}are infinite\. Since concentration theorems ensure thatPrν⁡\(κ≥k\)≤δk−1\\Pr\_\{\\nu\}\(\\kappa\\geq k\)\\leq\\delta\_\{k\-1\}and the conditional probabilityPrν⁡\(κendpos−κ≥k∣κ<\+∞\)≤π26​αk−1\\Pr\_\{\\nu\}\(\\kappa\_\{\\text\{end\}\}^\{\\text\{pos\}\}\-\\kappa\\geq k\\mid\\kappa<\+\\infty\)\\leq\\frac\{\\pi^\{2\}\}\{6\\alpha\_\{k\-1\}\}, we conclude thatPrν⁡\(τ<\+∞\)=1\\Pr\_\{\\nu\}\(\\tau<\+\\infty\)=1\. Givenτ<\+∞\\tau<\+\\inftyalmost surely, Theorem[1](https://arxiv.org/html/2605.23182#Thmtheorem1)follows by showing that the conditions for recommending a policy or outputtingNoneonly lead to wrong output when a concentration event is violated\. By applying a union bound over all phases, we ensure the total error probability is at mostδ\\delta\.

Sample Complexity\.We bound the expected sample complexity\. Now, Lemma[18](https://arxiv.org/html/2605.23182#Thmtheorem18)provides implicit upper bounds fortκendpos−tκ−1t\_\{\\kappa\_\{\\text\{end\}\}^\{\\text\{pos\}\}\}\-t\_\{\\kappa\-1\}andtκendneg−tκ−1t\_\{\\kappa\_\{\\text\{end\}\}^\{\\text\{neg\}\}\}\-t\_\{\\kappa\-1\}\. Combining this with the inequalitytκ−1≤Tκ−1eet\_\{\\kappa\-1\}\\leq T\_\{\\kappa\-1\}^\{\\text\{ee\}\}, we derive an upper bound for conditional expectation𝔼​\[τ\|κ\]\\mathbb\{E\}\[\\tau\|\\kappa\]\. This conditional bound is formulated aspoly​\(S,A,H,log⁡\(1/δ\),\(V0∗​\(s0\)−μ0\)−2\)⋅log⁡ακ​ϵκδκϵκ\\text\{poly\}\(S,A,H,\\log\(1/\\delta\),\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{\-2\}\)\\cdot\\frac\{\\log\\frac\{\\alpha\_\{\\kappa\}\\epsilon\_\{\\kappa\}\}\{\\delta\_\{\\kappa\}\}\}\{\\epsilon\_\{\\kappa\}\}, wherepoly​\(S,A,H,log⁡\(1/δ\),\(V0∗​\(s0\)−μ0\)−2\)\\text\{poly\}\(S,A,H,\\log\(1/\\delta\),\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{\-2\}\)is the right\-hand side of inequalities in Theorem[2](https://arxiv.org/html/2605.23182#Thmtheorem2),[3](https://arxiv.org/html/2605.23182#Thmtheorem3)\. Finally, we compute the total expectation using the law of total expectation:𝔼​\[τ\]=𝔼​\[𝔼​\[τ\|κ\]\]\\mathbb\{E\}\[\\tau\]=\\mathbb\{E\}\[\\mathbb\{E\}\[\\tau\|\\kappa\]\]\.Prν⁡\(κ≥k\)≤δk−1\\Pr\_\{\\nu\}\(\\kappa\\geq k\)\\leq\\delta\_\{k\-1\}implies

𝔼​\[log⁡ακ​ϵκδκϵκ\]≤∑k=1\+∞δk−1⋅log⁡αk​ϵkδkϵk=O​\(1\),\\displaystyle\\mathbb\{E\}\\left\[\\frac\{\\log\\frac\{\\alpha\_\{\\kappa\}\\epsilon\_\{\\kappa\}\}\{\\delta\_\{\\kappa\}\}\}\{\\epsilon\_\{\\kappa\}\}\\right\]\\leq\\sum\_\{k=1\}^\{\+\\infty\}\\delta\_\{k\-1\}\\cdot\\frac\{\\log\\frac\{\\alpha\_\{k\}\\epsilon\_\{k\}\}\{\\delta\_\{k\}\}\}\{\\epsilon\_\{k\}\}=O\(1\),which concludes Theorems[2](https://arxiv.org/html/2605.23182#Thmtheorem2)and[3](https://arxiv.org/html/2605.23182#Thmtheorem3)\.

## 4Sample Complexity Lower Bounds forδ\\delta\-PAC Algorithms

We establish lower bounds on the sample complexity ofδ\\delta\-PAC algorithms on positive and negative instances\. Due to space constraints, we present the core definitions and the sketch of the main ideas here, while deferred formal proofs and complete notations are provided in Appendix[C\.1](https://arxiv.org/html/2605.23182#A3.SS1)\.

### 4\.1Positive Instances

We construct a class of difficult MDP instances termed "Uniform Instances\."

###### Definition 4\(Sketch Definition of a Uniform Instance, specified by\(S,A,H,r,ϵ\)\(S,A,H,r,\\epsilon\)\)\.

Given even numberHH, integerS≥3S\\geq 3, integerA≥2A\\geq 2, positive real numberr,ϵr,\\epsilonsuch that14<r<r\+ϵ<34\\frac\{1\}\{4\}<r<r\+\\epsilon<\\frac\{3\}\{4\}\. We define a Uniform Instance with𝒮=\[S−2\]∪\{sgood,sbad\}\\mathcal\{S\}=\[S\-2\]\\cup\\\{s\_\{\\text\{good\}\},s\_\{\\text\{bad\}\}\\\},𝒜=\[A\]\\mathcal\{A\}=\[A\], reward functionrh​\(s,a\)=1​iff​s=sgood​else​0r\_\{h\}\(s,a\)=1\\text\{ iff \}s=s\_\{\\text\{good\}\}\\text\{ else \}0\. For allh,a∈\[H\]×𝒜h,a\\in\[H\]\\times\\mathcal\{A\},ph\(⋅\|sgood,a\)p\_\{h\}\(\\cdot\|s\_\{\\text\{good\}\},a\)ph\(⋅\|sbad,a\)p\_\{h\}\(\\cdot\|s\_\{\\text\{bad\}\},a\)are both uniform distribution on\[S−2\]\[S\-2\]\. For allh,s∈\[H\]×\[S−2\]h,s\\in\[H\]\\times\[S\-2\],ph\(⋅\|s,a\)p\_\{h\}\(\\cdot\|s,a\)is a two\-point distribution on\{sbad,sgood\}\\\{s\_\{\\text\{bad\}\},s\_\{\\text\{good\}\}\\\}withph​\(sgood\|s,a=1\)=r\+ϵp\_\{h\}\(s\_\{\\text\{good\}\}\|s,a=1\)=r\+\\epsilon,ph​\(sgood\|s,a≥2\)=rp\_\{h\}\(s\_\{\\text\{good\}\}\|s,a\\geq 2\)=r\. The initial distributionp0​\(⋅\)p\_\{0\}\(\\cdot\)is uniform on\[S−2\]\[S\-2\]\.

The construction ensures that at any odd roundhh, the stateSt,hS\_\{t,h\}follows a uniform distribution over the\[S−2\]\[S\-2\], regardless of the policy\. At even round, the agent is forced into eithersgoods\_\{\\text\{good\}\}orsbads\_\{\\text\{bad\}\}\. The definition of a Uniform Instance assumes an agent can only collect positive reward at the statesgoods\_\{\\text\{good\}\}\. To maximize reward, the agent must identify the "optimal" action \(action 1\) at each states∈\[S−2\]s\\in\[S\-2\]\. To characterize the difficulty of identifying this optimal action, we consider the set of all possible permutations of the action space\.

###### Definition 5\(Permutation of an Instanceν\\nu\)\.

Given an 1\-policy\-identification instanceν\\nuwith reward function\{rhν​\(s,a\)\}\(s,a,h\)∈𝒮×𝒜×\[H\]\\\{r\_\{h\}^\{\\nu\}\(s,a\)\\\}\_\{\(s,a,h\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\\times\[H\]\}and transition probability kernel\{phν​\(s′\|s,a\)\}\(h,s′​s,a\)∈\[H\]×𝒮×𝒮×𝒜\\\{p\_\{h\}^\{\\nu\}\(s^\{\\prime\}\|s,a\)\\\}\_\{\(h,s^\{\\prime\}s,a\)\\in\[H\]\\times\\mathcal\{S\}\\times\\mathcal\{S\}\\times\\mathcal\{A\}\}, and a set of permutationsσ=\{σs,h\}\(s,h\)∈𝒮×\[H\]\\sigma=\\\{\\sigma\_\{s,h\}\\\}\_\{\(s,h\)\\in\\mathcal\{S\}\\times\[H\]\}, eachσs,h\\sigma\_\{s,h\}is a bijection𝒜→𝒜\\mathcal\{A\}\\rightarrow\\mathcal\{A\}, we define a permutation ofν\\nuas instanceνσ\\nu\_\{\\sigma\}, with reward functionrhνσ​\(s,σs,h​\(a\)\)=rhν​\(s,a\)r\_\{h\}^\{\\nu\_\{\\sigma\}\}\\big\(s,\\sigma\_\{s,h\}\(a\)\\big\)=r\_\{h\}^\{\\nu\}\(s,a\),phνσ​\(s′\|s,σ​\(a\)\)=phν​\(s′\|s,a\)p\_\{h\}^\{\\nu\_\{\\sigma\}\}\(s^\{\\prime\}\|s,\\sigma\(a\)\)=p\_\{h\}^\{\\nu\}\(s^\{\\prime\}\|s,a\)\.

For a Uniform Instanceν\\nu, the optimal policy for its permutationνσ\\nu\_\{\\sigma\}isπh​\(s\)=σs,h​\(1\)\\pi\_\{h\}\(s\)=\\sigma\_\{s,h\}\(1\), with optimal valueH​\(r\+ϵ\)2\\frac\{H\(r\+\\epsilon\)\}\{2\}\. In addition, the value function of a policyπ\\pion instanceνσ\\nu\_\{\\sigma\}is

V0π​\(s0\|νσ\)=∑h​:odd∑s=1S−2𝟙​\(πh​\(s\)=σs,h​\(1\)\)​\(r\+ϵ\)\+𝟙​\(πh​\(s\)≠σs,h​\(1\)\)​rS−2\\displaystyle V\_\{0\}^\{\\pi\}\(s\_\{0\}~\|~\\nu\_\{\\sigma\}\)=\\frac\{\\sum\_\{h\\text\{:odd\}\}\\sum\_\{s=1\}^\{S\-2\}\\mathds\{1\}\(\\pi\_\{h\}\(s\)=\\sigma\_\{s,h\}\(1\)\)\(r\+\\epsilon\)\+\\mathds\{1\}\(\\pi\_\{h\}\(s\)\\neq\\sigma\_\{s,h\}\(1\)\)r\}\{S\-2\}It is evident to seeH​r2≤V0π​\(s0;νσ\)≤H​\(r\+ϵ\)2\\frac\{Hr\}\{2\}\\leq V\_\{0\}^\{\\pi\}\(s\_\{0\};\\nu\_\{\\sigma\}\)\\leq\\frac\{H\(r\+\\epsilon\)\}\{2\}\. To explore for a qualified policy, the agent has to guarantee sufficient exploration for the optimal actionσs,h​\(1\)\\sigma\_\{s,h\}\(1\)on a fraction of pair\{\(h,s\):h​:odd,s∈\[S−2\]\}\\\{\(h,s\):h\\text\{:odd\},s\\in\[S\-2\]\\\}, leading to the following Theorem\.

###### Theorem 4\.

Consider anyδ\\delta\-PAC alg and any Uniform Instanceν\\nuspecified by\(S,A,H,r,ϵ\)\(S,A,H,r,\\epsilon\), withμ0\\mu\_\{0\}satisfying3​H​\(r\+ϵ\)8\+H​r8\>μ0\>H​r2\\frac\{3H\(r\+\\epsilon\)\}\{8\}\+\\frac\{Hr\}\{8\}\>\\mu\_\{0\}\>\\frac\{Hr\}\{2\},S≥100,H≥10S\\geq 100,H\\geq 10,A≥100A\\geq 100,\(S−2\)​H\(S\-2\)His divisible by1616,0<ϵ<1200​50<\\epsilon<\\frac\{1\}\{200\\sqrt\{5\}\}\. DenoteΓ:=\{σ=\{σs,h\}\(s,h\)∈𝒮×H:∀s,h,σs,h​is a bijection​𝒜→𝒜\}\\Gamma:=\\\{\\sigma=\\\{\\sigma\_\{s,h\}\\\}\_\{\(s,h\)\\in\\mathcal\{S\}\\times H\}:\\forall s,h,\\sigma\_\{s,h\}\\text\{ is a bijection \}\\mathcal\{A\}\\rightarrow\\mathcal\{A\}\\\}, the set of all the possible permutations onν\\nu, it holds that

maxσ∈Γ⁡𝔼νσ,alg​\[τ\]≥Ω​\(H​log⁡1δ\(V0∗​\(s0\|νσ\)−μ0\)2\+H2​S​A\(V0∗​\(s0\|νσ\)−μ0\)2\)\.\\displaystyle\\max\_\{\\sigma\\in\\Gamma\}\\mathbb\{E\}\_\{\\nu\_\{\\sigma\},\\text\{alg\}\}\[\\tau\]\\geq\\Omega\\left\(\\frac\{H\\log\\frac\{1\}\{\\delta\}\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}~\|~\\nu\_\{\\sigma\}\)\-\\mu\_\{0\}\)^\{2\}\}\+\\frac\{H^\{2\}SA\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}~\|~\\nu\_\{\\sigma\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\.

Theorem[4](https://arxiv.org/html/2605.23182#Thmtheorem4)is proved in Appendix[C\.2](https://arxiv.org/html/2605.23182#A3.SS2)\. The lower bound suggests that the upper bound in Theorem[2](https://arxiv.org/html/2605.23182#Thmtheorem2)is near\-optimal from the minimax perspective\. Specifically, the gap in theδ\\delta\-dependent term is at most a factor ofHH, while theδ\\delta\-independent term remains within poly\(H,S,log⁡\(A\)H,S,\\log\(A\)\)\.

### 4\.2Negative Instances

We first introduce the sketch definition of Tree Instance, with full definition in the Appendix[C\.1](https://arxiv.org/html/2605.23182#A3.SS1)\.

###### Definition 6\(Sketch Definition of a Tree Instance, specified by\(S,A,H,r\)\(S,A,H,r\)\)\.

Consider integerS,A,HS,A,Hsuch thatS=2N\+1S=2^\{N\}\+1for some integerN≥1N\\geq 1, andA≥3A\\geq 3,H−1≥6​N=6​log2⁡\(S−1\)H\-1\\geq 6N=6\\log\_\{2\}\(S\-1\), and a real numberr∈\(38,58\)r\\in\(\\frac\{3\}\{8\},\\frac\{5\}\{8\}\)\. Take𝒮=\[2N−1\]∪\{sgood,sbad\}\\mathcal\{S\}=\[2^\{N\}\-1\]\\cup\\\{s\_\{\\text\{good\}\},s\_\{\\text\{bad\}\}\\\},𝒜=\[A\]\\mathcal\{A\}=\[A\], and consider a binary tree with nodes set\[2N−1\]\[2^\{N\}\-1\]\.sgood,sbads\_\{\\text\{good\}\},s\_\{\\text\{bad\}\}are absorbed states, while for states∈\[2N−1−1\]s\\in\[2^\{N\-1\}\-1\], action 1 will go to state2​s2s, action 2 will go to state2​s\+12s\+1, for actiona≥3a\\geq 3,ph​\(sgood\|s,a≥3\)=r,ph​\(sbad\|s,a≥3\)=1−rp\_\{h\}\(s\_\{\\text\{good\}\}\|s,a\\geq 3\)=r,p\_\{h\}\(s\_\{\\text\{bad\}\}\|s,a\\geq 3\)=1\-r\. For states∈\[2N−1,2N−1\]s\\in\[2^\{N\-1\},2^\{N\}\-1\],ph​\(sgood\|s,a≥1\)=rp\_\{h\}\(s\_\{\\text\{good\}\}\|s,a\\geq 1\)=r,ph​\(sbad\|s,a≥1\)=1−rp\_\{h\}\(s\_\{\\text\{bad\}\}\|s,a\\geq 1\)=1\-r\. The reward function isrh​\(s,a\)=1​iff​s=sgood​else​0r\_\{h\}\(s,a\)=1\\text\{ iff \}s=s\_\{\\text\{good\}\}\\text\{ else \}0\. The initial state is11, withp0​\(1\)=1,p0​\(s\)=0,∀s≠1p\_\{0\}\(1\)=1,p\_\{0\}\(s\)=0,\\forall s\\neq 1\.

In a Tree Instance, an agent only collects rewards upon reachingsgoods\_\{\\text\{good\}\}\. To maximize the total reward in an episode, the agent must determine the roundhhand statessat which to transition to the absorbing states\. For a thresholdμ0\>\(H−1\)​r\\mu\_\{0\}\>\(H\-1\)r, the instance is negative\. To correctly identify it as a negative instance, the agent is forced to explore all state\-action pairs\(h,s,a\)\(h,s,a\)fora≥3a\\geq 3to ensure that no hidden "high\-reward" transitions exist\. This leads to the following lower bound:

###### Theorem 5\.

Apply anyδ\\delta\-PAC algorithm to a Tree instanceν\\nuspecified by\(S,A,H,r\)\(S,A,H,r\)withS,A,H≥4S,A,H\\geq 4and34​\(H−log2⁡\(S−3\)\)\>μ0\>\(H−1\)​r\\frac\{3\}\{4\}\(H\-\\log\_\{2\}\(S\-3\)\)\>\\mu\_\{0\}\>\(H\-1\)r, we have

𝔼ν​τ≥Ω​\(H​S​A​log⁡1δ\(μ0−V0∗​\(s0\|ν\)\)2\)\.\\displaystyle\\mathbb\{E\}\_\{\\nu\}\\tau\\geq\\Omega\\left\(\\frac\{HSA\\log\\frac\{1\}\{\\delta\}\}\{\(\\mu\_\{0\}\-V^\{\*\}\_\{0\}\(s\_\{0\}~\|~\\nu\)\)^\{2\}\}\\right\)\.

Full proof is in Appendix[C\.3](https://arxiv.org/html/2605.23182#A3.SS3)\. Comparing Theorem[5](https://arxiv.org/html/2605.23182#Thmtheorem5)with our upper bound in Theorem[3](https://arxiv.org/html/2605.23182#Thmtheorem3), we observe that the gap is restricted to a factor ofH3​log⁡S​A​H\(μ0−V0∗\)2\+S​log⁡log⁡\(1/δ\)H^\{3\}\\log\\frac\{SAH\}\{\(\\mu\_\{0\}\-V^\{\*\}\_\{0\}\)^\{2\}\}\+S\\log\\log\(1/\\delta\)\. Sincelog⁡log⁡\(1/δ\)<log⁡\(1/δ\)\\log\\log\(1/\\delta\)<\\log\(1/\\delta\)for allδ∈\(0,1\)\\delta\\in\(0,1\), the gap between our upper and lower bounds is bounded by a polynomial inS,AS,A, andHH\. Thus, our proposed algorithm BEE\-GPI achieves a nearly optimal sample complexity\.

## 5Numerical Experiments

We provide numerical evidence to evaluate the empirical efficiency of our proposed BEE\-GPI\. We consider two classic environments: the Single Chain and Double Chain MDPsKaufmannet al\.\([2021](https://arxiv.org/html/2605.23182#bib.bib9)\)\. For both settings, we fixδ=0\.001\\delta=0\.001,H=8H=8,𝒮=\{0,1,2,3\}\\mathcal\{S\}=\\\{0,1,2,3\\\}, and𝒜=\{left,right\}\\mathcal\{A\}=\\\{\\text\{left\},\\text\{right\}\\\}\. The transition dynamics are defined such that the "intended" direction occurs with probability0\.90\.9and the "reverse" direction with probability0\.10\.1\. Specifically,ph​\(max⁡\{s−1,0\}∣s,left\)=0\.9p\_\{h\}\(\\max\\\{s\-1,0\\\}\\mid s,\\text\{left\}\)=0\.9andph​\(min⁡\{s\+1,3\}∣s,right\)=0\.9p\_\{h\}\(\\min\\\{s\+1,3\\\}\\mid s,\\text\{right\}\)=0\.9\. The two environments primarily differ in the initial state distribution: the agent always starts at state0in the Single Chain and at state11in the Double Chain\.

To evaluate the performance of BEE\-GPI, we select threshold valuesμ0∈\{1,1\.5,2\.0,2\.5,3\.0\}\\mu\_\{0\}\\in\\\{1,1\.5,2\.0,2\.5,3\.0\\\}, ensuring that all tested instances are positive \(V0∗\>μ0V^\{\*\}\_\{0\}\>\\mu\_\{0\}\)\. We compare our approach against theϵ\\epsilon\-optimal BPI algorithm, BPI\-UCRL inKaufmannet al\.\([2021](https://arxiv.org/html/2605.23182#bib.bib9)\)for its execution efficiency\. To adapt BPI\-UCRL for the GPI task, we provide it with the oracle parameterϵ=V0∗−μ0\\epsilon=V^\{\*\}\_\{0\}\-\\mu\_\{0\}, a significant prior\-knowledge advantage\. Both algorithms were implemented using the MOSEK optimization solver\. We run each experiment with 20 independent trials, and deploy them on an Intel i5\-14500 CPU over a total computation time of approximately 72 hours\.

![Refer to caption](https://arxiv.org/html/2605.23182v1/x1.png)Figure 1:Numerical comparison between BEE\-GPI and BPI\-UCRL \(the lower the better\)\.Figure[1](https://arxiv.org/html/2605.23182#S5.F1)presents the empirical stopping times averaged for each setting\. The error bars represent three standard deviations \(3​σ3\\sigma\), though they are notably small due to the low variance of termination round\. As illustrated in Figure[1](https://arxiv.org/html/2605.23182#S5.F1), BEE\-GPI consistently outperforms BPI\-UCRL by requiring significantly fewer episodes to identify a qualified policy across all tested thresholds\. These observations validate our theoretical contribution in Theorem[2](https://arxiv.org/html/2605.23182#Thmtheorem2): by decoupling thelog⁡\(1/δ\)\\log\(1/\\delta\)coefficient from the state and action space sizes \(S,AS,A\), BEE\-GPI achieves a tighter sample complexity than a standard reductions to the state\-of\-the\-artϵ\\epsilon\-optimal BPI algorithms\.

## 6Conclusion

We formalized the Good Policy Identification \(GPI\) problem and proposed the BEE\-GPI algorithm\. We establish upper and lower bounds on sample complexity, demonstrating that BEE\-GPI is minimax optimal up to a polynomial factor ofS,AS,A, andHH\. Critically, we showed that for positive instances, our algorithm decouples thelog⁡1/δ\\log 1/\\deltaterm from the state\-action space size, an efficiency gain validated by our numerical experiments\. Future research includes deriving tighter instance\-dependent bounds and extending this threshold\-based framework to large\-scale MDPs using function approximation\.

## References

- Model\-based reinforcement learning with a generative model is minimax optimal\.InProceedings of Thirty Third Conference on Learning Theory,J\. Abernethy and S\. Agarwal \(Eds\.\),Proceedings of Machine Learning Research, Vol\.125,pp\. 67–83\.External Links:[Link](https://proceedings.mlr.press/v125/agarwal20b.html)Cited by:[§2](https://arxiv.org/html/2605.23182#S2.p1.2)\.
- A\. Al Marjani, A\. Garivier, and A\. Proutiere \(2021\)Navigating to the best policy in markov decision processes\.Advances in Neural Information Processing Systems34,pp\. 25852–25864\.Cited by:[§2](https://arxiv.org/html/2605.23182#S2.p1.2)\.
- A\. Al Marjani and A\. Proutiere \(2021\)Adaptive sampling for best policy identification in markov decision processes\.InInternational Conference on Machine Learning,pp\. 7459–7468\.Cited by:[Appendix A](https://arxiv.org/html/2605.23182#A1.p2.2),[§B\.3](https://arxiv.org/html/2605.23182#A2.SS3.p1.3),[§2](https://arxiv.org/html/2605.23182#S2.p1.2)\.
- C\. Dann and E\. Brunskill \(2015\)Sample complexity of episodic fixed\-horizon reinforcement learning\.Advances in Neural Information Processing Systems28\.Cited by:[§2](https://arxiv.org/html/2605.23182#S2.p1.2)\.
- R\. Degenne and W\. M\. Koolen \(2019\)Pure exploration with multiple correct answers\.Advances in Neural Information Processing Systems32\.Cited by:[§1](https://arxiv.org/html/2605.23182#S1.p10.6),[§2](https://arxiv.org/html/2605.23182#S2.p3.2)\.
- O\. D\. Domingues, P\. Ménard, E\. Kaufmann, and M\. Valko \(2021\)Episodic reinforcement learning in finite mdps: minimax lower bounds revisited\.InProceedings of the 32nd International Conference on Algorithmic Learning Theory,V\. Feldman, K\. Ligett, and S\. Sabato \(Eds\.\),Proceedings of Machine Learning Research, Vol\.132,pp\. 578–598\.External Links:[Link](https://proceedings.mlr.press/v132/domingues21a.html)Cited by:[§2](https://arxiv.org/html/2605.23182#S2.p2.8)\.
- C\. Fiechter \(1994\)Efficient reinforcement learning\.InProceedings of the seventh annual conference on Computational learning theory,pp\. 88–97\.Cited by:[§2](https://arxiv.org/html/2605.23182#S2.p1.2)\.
- M\. Gheshlaghi Azar, R\. Munos, and H\. J\. Kappen \(2013\)Minimax pac bounds on the sample complexity of reinforcement learning with a generative model\.Machine learning91\(3\),pp\. 325–349\.Cited by:[§2](https://arxiv.org/html/2605.23182#S2.p1.2)\.
- T\. Jaksch, R\. Ortner, and P\. Auer \(2010\)Near\-optimal regret bounds for reinforcement learning\.Journal of Machine Learning Research11\(51\),pp\. 1563–1600\.External Links:[Link](http://jmlr.org/papers/v11/jaksch10a.html)Cited by:[Lemma 41](https://arxiv.org/html/2605.23182#Thmtheorem41)\.
- M\. Jourdan, A\. Delahaye\-Duriez, and C\. Réda \(2026\)An anytime algorithm for good arm identification\.Journal of Machine Learning Research27\(19\),pp\. 1–90\.Cited by:[§2](https://arxiv.org/html/2605.23182#S2.p3.2)\.
- H\. Kano, J\. Honda, K\. Sakamaki, K\. Matsuura, and M\. Sugiyama \(2017\)Good arm identification via bandit feedback\.Machine Learning\(2\)\.Cited by:[§2](https://arxiv.org/html/2605.23182#S2.p3.2)\.
- J\. Katz\-Samuels and K\. Jamieson \(2020\)The true sample complexity of identifying good arms\.InInternational Conference on Artificial Intelligence and Statistics,pp\. 1781–1791\.Cited by:[§2](https://arxiv.org/html/2605.23182#S2.p3.2)\.
- E\. Kaufmann, O\. Cappé, and A\. Garivier \(2016\)On the complexity of best\-arm identification in multi\-armed bandit models\.The Journal of Machine Learning Research17\(1\),pp\. 1–42\.Cited by:[§C\.2](https://arxiv.org/html/2605.23182#A3.SS2.6.p2.1),[§C\.3](https://arxiv.org/html/2605.23182#A3.SS3.4.p3.4),[§C\.4](https://arxiv.org/html/2605.23182#A3.SS4.12.p5.1),[§C\.4](https://arxiv.org/html/2605.23182#A3.SS4.4.p2.16)\.
- E\. Kaufmann, P\. Ménard, O\. D\. Domingues, A\. Jonsson, E\. Leurent, and M\. Valko \(2021\)Adaptive reward\-free exploration\.InAlgorithmic Learning Theory,pp\. 865–891\.Cited by:[§B\.3](https://arxiv.org/html/2605.23182#A2.SS3.1.p1.1),[§B\.3](https://arxiv.org/html/2605.23182#A2.SS3.p2.1),[§B\.3](https://arxiv.org/html/2605.23182#A2.SS3.p6.1),[§2](https://arxiv.org/html/2605.23182#S2.p1.2),[§3\.1](https://arxiv.org/html/2605.23182#S3.SS1.p3.1),[§3\.1](https://arxiv.org/html/2605.23182#S3.SS1.p4.17),[§5](https://arxiv.org/html/2605.23182#S5.p1.10),[§5](https://arxiv.org/html/2605.23182#S5.p2.4),[Lemma 19](https://arxiv.org/html/2605.23182#Thmtheorem19)\.
- M\. Kearns and S\. Singh \(1998\)Finite\-sample convergence rates for q\-learning and indirect algorithms\.Advances in neural information processing systems11\.Cited by:[§2](https://arxiv.org/html/2605.23182#S2.p1.2)\.
- G\. Li, Y\. Wei, Y\. Chi, and Y\. Chen \(2024\)Breaking the sample size barrier in model\-based reinforcement learning with a generative model\.Operations Research72\(1\),pp\. 203–221\.Cited by:[§2](https://arxiv.org/html/2605.23182#S2.p1.2)\.
- Z\. Li and W\. C\. Cheung \(2025\)Near optimal non\-asymptotic sample complexity of 1\-identification\.InProceedings of the 42nd International Conference on Machine Learning,A\. Singh, M\. Fazel, D\. Hsu, S\. Lacoste\-Julien, F\. Berkenkamp, T\. Maharaj, K\. Wagstaff, and J\. Zhu \(Eds\.\),Proceedings of Machine Learning Research, Vol\.267,pp\. 34144–34184\.External Links:[Link](https://proceedings.mlr.press/v267/li25f.html)Cited by:[Appendix D](https://arxiv.org/html/2605.23182#A4.p2.1),[§2](https://arxiv.org/html/2605.23182#S2.p3.2),[Lemma 42](https://arxiv.org/html/2605.23182#Thmtheorem42),[Lemma 43](https://arxiv.org/html/2605.23182#Thmtheorem43)\.
- Z\. Li and W\. C\. Cheung \(2026\)Closing the gap on the sample complexity of 1\-identification\.arXiv preprint arXiv:2601\.15620\.Cited by:[§2](https://arxiv.org/html/2605.23182#S2.p3.2)\.
- P\. Ménard, O\. D\. Domingues, A\. Jonsson, E\. Kaufmann, E\. Leurent, and M\. Valko \(2021\)Fast active learning for pure exploration in reinforcement learning\.InInternational Conference on Machine Learning,pp\. 7599–7608\.Cited by:[§2](https://arxiv.org/html/2605.23182#S2.p1.2)\.
- A\. Narang, A\. Wagenmaker, L\. Ratliff, and K\. G\. Jamieson \(2024\)Sample complexity reduction via policy difference estimation in tabular reinforcement learning\.Advances in Neural Information Processing Systems37,pp\. 22772–22826\.Cited by:[§2](https://arxiv.org/html/2605.23182#S2.p1.2)\.
- A\. Sidford, M\. Wang, X\. Wu, L\. Yang, and Y\. Ye \(2018\)Near\-optimal time and sample complexities for solving markov decision processes with a generative model\.Advances in Neural Information Processing Systems31\.Cited by:[§2](https://arxiv.org/html/2605.23182#S2.p1.2)\.
- A\. Sidford, M\. Wang, X\. Wu, and Y\. Ye \(2023\)Variance reduced value iteration and faster algorithms for solving markov decision processes\.Naval Research Logistics \(NRL\)70\(5\),pp\. 423–442\.Cited by:[§2](https://arxiv.org/html/2605.23182#S2.p1.2)\.
- A\. Tirinzoni, A\. Al\-Marjani, and E\. Kaufmann \(2023\)Optimistic pac reinforcement learning: the instance\-dependent view\.InInternational Conference on Algorithmic Learning Theory,pp\. 1460–1480\.Cited by:[§2](https://arxiv.org/html/2605.23182#S2.p1.2)\.
- A\. J\. Wagenmaker, M\. Simchowitz, and K\. Jamieson \(2022\)Beyond no regret: instance\-dependent pac reinforcement learning\.InConference on Learning Theory,pp\. 358–418\.Cited by:[§2](https://arxiv.org/html/2605.23182#S2.p1.2)\.

## Appendix ANotation

In this section, We list the notation used in Algorithms[1](https://arxiv.org/html/2605.23182#alg1),[2](https://arxiv.org/html/2605.23182#alg2), and we supplement the details on the notation in the analysis in Appendix[B](https://arxiv.org/html/2605.23182#A2)\.

- •rh​\(s,a\)r\_\{h\}\(s,a\): mean reward of taking actionaain statessat roundhh\.
- •ph\(⋅\|s,a\)p\_\{h\}\(\\cdot\|s,a\): true transition kernel when we take actionaain statessat roundhh\.
- •phπ​\(s,a\):p\_\{h\}^\{\\pi\}\(s,a\):Execute a given policyπ\\pion the MDP\. DenoteShS\_\{h\}as the random state the agent locate on roundhh, we definephπ​\(s,a\)=Pr⁡\(Sh=s,πh​\(s\)=a\)p\_\{h\}^\{\\pi\}\(s,a\)=\\Pr\(S\_\{h\}=s,\\pi\_\{h\}\(s\)=a\)\.
- •St,hS\_\{t,h\}: realized state roundhhatttth episode\.
- •βcnt​\(δ\)=log⁡2​S​A​Hδ\\beta^\{\\text\{cnt\}\}\(\\delta\)=\\log\\frac\{2SAH\}\{\\delta\},βp​\(t,δ\)=log⁡2​S​A​Hδ\+\(S−1\)​log⁡\(e​\(1\+tS−1\)\)\\beta\_\{p\}\(t,\\delta\)=\\log\\frac\{2SAH\}\{\\delta\}\+\(S\-1\)\\log\\Big\(e\(1\+\\frac\{t\}\{S\-1\}\)\\Big\)
- •p^ht\(⋅\|s,a\)\\hat\{p\}\_\{h\}^\{t\}\(\\cdot\|s,a\)as empirical transition kernel after executingttepisodes during the exploration stages\.p^ht​\(s′\|s,a\)=1nht​\(s,a\)​∑q=1t𝟏​\(Sq,h=s,Aq,h,Sq,h\+1=s′\)\\hat\{p\}\_\{h\}^\{t\}\(s^\{\\prime\}\|s,a\)=\\frac\{1\}\{n^\{t\}\_\{h\}\(s,a\)\}\\sum^\{t\}\_\{q=1\}\\mathbf\{1\}\(S\_\{q,h\}=s,A\_\{q,h\},S\_\{q,h\+1\}=s^\{\\prime\}\)ifnht​\(s,a\)\>0n\_\{h\}^\{t\}\(s,a\)\>0, and setp^ht​\(s′\|s,a\)=1/\|𝒮\|\\hat\{p\}\_\{h\}^\{t\}\(s^\{\\prime\}\|s,a\)=1/\|\\mathcal\{S\}\|otherwise\.
- •nht​\(s,a\):=∑q=1t𝟏​\(Sq,h=s,Aq,h=a\)n^\{t\}\_\{h\}\(s,a\):=\\sum^\{t\}\_\{q=1\}\\mathbf\{1\}\(S\_\{q,h\}=s,A\_\{q,h\}=a\), visiting times of pair\(s,a,h\)\(s,a,h\)after executingttepisode during the exploration stages\.nht​\(s′,s,a\):=∑q=1t𝟏​\(Sq,h\+1=s,Sq,h=s,Aq,h=a\)n^\{t\}\_\{h\}\(s^\{\\prime\},s,a\):=\\sum^\{t\}\_\{q=1\}\\mathbf\{1\}\(S\_\{q,h\+1\}=s,S\_\{q,h\}=s,A\_\{q,h\}=a\), visiting times of pair\(s,a,h\)\(s,a,h\)and the next state iss′s^\{\\prime\}after executingttepisodes\. Here we denoteℋee=\{Sq,h,Aq,h\}q∈\[t\],h∈\[H\]\\mathcal\{H\}^\{\\text\{ee\}\}=\\\{S\_\{q,h\},A\_\{q,h\}\\\}\_\{q\\in\[t\],h\\in\[H\]\}\.
- •n¯ht​\(s,a\):\\bar\{n\}\_\{h\}^\{t\}\(s,a\):Denotekqk\_\{q\}as the stage index atqq\-th sampled episode, andπ¯q:=π¯q​\(⋅;δkq\)\\bar\{\\pi\}^\{q\}:=\\overline\{\\pi\}^\{q\}\(\\cdot;\\delta\_\{k\_\{q\}\}\), defined in \([2](https://arxiv.org/html/2605.23182#S3.E2)\)\. We definen¯ht​\(s,a\):=∑q=0t−1phπ¯q​\(s,a\)\\bar\{n\}\_\{h\}^\{t\}\(s,a\):=\\sum\_\{q=0\}^\{t\-1\}p\_\{h\}^\{\\bar\{\\pi\}^\{q\}\}\(s,a\)\. We defineπ¯0\\overline\{\\pi\}^\{0\}by \([2](https://arxiv.org/html/2605.23182#S3.E2)\) with𝒞h0​\(s,a;δ1\):=Δ𝒮\\mathcal\{C\}\_\{h\}^\{0\}\(s,a;\\delta\_\{1\}\):=\\Delta^\{\\mathcal\{S\}\}\.
- •V0∗​\(s0\|ν\)V^\{\*\}\_\{0\}\(s\_\{0\}\|\\nu\)optimal value function of instanceν\\nu\. If we only mention one instanceν\\nu, we may useV0∗​\(s0\)V^\{\*\}\_\{0\}\(s\_\{0\}\)for simplicity\.

What’s more, we introduce the confidence interval for each pair\(s,a,h\)\(s,a,h\)

𝒞ht\(s,a;δ\):=\{q\(⋅\|s,a\)∈Δ𝒮:KL\(p^ht\(⋅\|s,a\),q\(⋅\|s,a\)\)≤βp​\(nht​\(s,a\),δ\)nht​\(s,a\)\}\.\\displaystyle\\mathcal\{C\}\_\{h\}^\{t\}\(s,a;\\delta\):=\\left\\\{q\(\\cdot\|s,a\)\\in\\Delta^\{\\mathcal\{S\}\}:\\text\{KL\}\\big\(\\widehat\{p\}\_\{h\}^\{t\}\(\\cdot\|s,a\),q\(\\cdot\|s,a\)\\big\)\\leq\\frac\{\\beta\_\{p\}\(n\_\{h\}^\{t\}\(s,a\),\\delta\)\}\{n\_\{h\}^\{t\}\(s,a\)\}\\right\\\}\.FollowingAl Marjani and Proutiere \[[2021](https://arxiv.org/html/2605.23182#bib.bib6)\], we define the same optimistic and pessimistic approximation value function, for each\(h,s,a\)\(h,s,a\)\.

Q¯ht​\(s,a;δ\):=\\displaystyle\\overline\{Q\}\_\{h\}^\{t\}\(s,a;\\delta\):=rh​\(s,a\)\+maxp¯h\(⋅\|s,a\)∈𝒞ht\(s,a;δ\)​∑s′p¯h​\(s′\|s,a\)​V¯h\+1t​\(s′;δ\)\\displaystyle r\_\{h\}\(s,a\)\+\\max\_\{\\bar\{p\}\_\{h\}\(\\cdot\|s,a\)\\in\\mathcal\{C\}\_\{h\}^\{t\}\(s,a;\\delta\)\}\\sum\_\{s^\{\\prime\}\}\\bar\{p\}\_\{h\}\(s^\{\\prime\}\|s,a\)\\overline\{V\}\_\{h\+1\}^\{t\}\(s^\{\\prime\};\\delta\)V¯ht​\(s;δ\):=\\displaystyle\\overline\{V\}^\{t\}\_\{h\}\(s;\\delta\):=maxa⁡Q¯ht​\(s,a;δ\)\\displaystyle\\max\_\{a\}\\overline\{Q\}\_\{h\}^\{t\}\\big\(s,a;\\delta\\big\)V¯H\+1t​\(s;δ\):=\\displaystyle\\overline\{V\}^\{t\}\_\{H\+1\}\(s;\\delta\):=0\\displaystyle 0p¯ht​\(s,a;δ\)∈\\displaystyle\\overline\{p\}\_\{h\}^\{t\}\(s,a;\\delta\)\\inarg⁡maxp¯h∈𝒞ht​\(s,a,δ\)​∑s′p¯h​\(s′\|s,a\)​V¯h\+1t​\(s′;δ\)\\displaystyle\\arg\\max\_\{\\bar\{p\}\_\{h\}\\in\\mathcal\{C\}\_\{h\}^\{t\}\(s,a,\\delta\)\}\\sum\_\{s^\{\\prime\}\}\\bar\{p\}\_\{h\}\(s^\{\\prime\}\|s,a\)\\overline\{V\}\_\{h\+1\}^\{t\}\(s^\{\\prime\};\\delta\)π¯ht​\(s;δ\)∈\\displaystyle\\overline\{\\pi\}^\{t\}\_\{h\}\(s;\\delta\)\\inarg⁡maxa⁡Q¯ht​\(s,a;δ\)\\displaystyle\\arg\\max\_\{a\}\\overline\{Q\}\_\{h\}^\{t\}\\big\(s,a;\\delta\\big\)V¯0t​\(s0;δ\)=\\displaystyle\\overline\{V\}\_\{0\}^\{t\}\(s\_\{0\};\\delta\)=maxp¯0∈𝒞0t​\(s0;δ\)​∑s′p¯0​\(s′\|s0,a0\)​V¯1t​\(s′;δ\)\\displaystyle\\max\_\{\\bar\{p\}\_\{0\}\\in\\mathcal\{C\}\_\{0\}^\{t\}\(s\_\{0\};\\delta\)\}\\sum\_\{s^\{\\prime\}\}\\bar\{p\}\_\{0\}\(s^\{\\prime\}\|s\_\{0\},a\_\{0\}\)\\overline\{V\}\_\{1\}^\{t\}\(s^\{\\prime\};\\delta\)Q¯ht​\(s,a;δ\):=\\displaystyle\\underline\{Q\}\_\{h\}^\{t\}\(s,a;\\delta\):=rh​\(s,a\)\+minp¯h∈𝒞ht​\(s,a;δ\)​∑s′p¯h​\(s′\|s,a\)​V¯h\+1t​\(s′;δ\)\\displaystyle r\_\{h\}\(s,a\)\+\\min\_\{\\underline\{p\}\_\{h\}\\in\\mathcal\{C\}\_\{h\}^\{t\}\(s,a;\\delta\)\}\\sum\_\{s^\{\\prime\}\}\\underline\{p\}\_\{h\}\(s^\{\\prime\}\|s,a\)\\underline\{V\}\_\{h\+1\}^\{t\}\(s^\{\\prime\};\\delta\)V¯ht​\(s;δ\):=\\displaystyle\\underline\{V\}\_\{h\}^\{t\}\(s;\\delta\):=maxa⁡Q¯ht​\(s,a;δ\)\\displaystyle\\max\_\{a\}\\underline\{Q\}\_\{h\}^\{t\}\(s,a;\\delta\)V¯H\+1t​\(s;δ\):=\\displaystyle\\underline\{V\}\_\{H\+1\}^\{t\}\(s;\\delta\):=0\\displaystyle 0p¯ht​\(s,a;δ\)\\displaystyle\\underline\{p\}\_\{h\}^\{t\}\(s,a;\\delta\)∈arg⁡minp¯∈𝒞ht​\(s,a;δ\)​∑s′p¯h​\(s′\|s,a\)​V¯h\+1t​\(s′;δ\)\\displaystyle\\in\\arg\\min\_\{\\underline\{p\}\\in\\mathcal\{C\}\_\{h\}^\{t\}\(s,a;\\delta\)\}\\sum\_\{s^\{\\prime\}\}\\underline\{p\}\_\{h\}\(s^\{\\prime\}\|s,a\)\\underline\{V\}\_\{h\+1\}^\{t\}\(s^\{\\prime\};\\delta\)π¯ht​\(s;δ\)∈\\displaystyle\\underline\{\\pi\}^\{t\}\_\{h\}\(s;\\delta\)\\inarg⁡maxa⁡Q¯ht​\(s,a;δ\)\\displaystyle\\arg\\max\_\{a\}\\underline\{Q\}\_\{h\}^\{t\}\\big\(s,a;\\delta\\big\)V¯0t​\(s0;δ\)=\\displaystyle\\underline\{V\}\_\{0\}^\{t\}\(s\_\{0\};\\delta\)=minp¯0∈𝒞0t​\(s0;δ\)​∑s′p¯0​\(s′\|s0,a0\)​V¯1t​\(s′;δ\)\\displaystyle\\min\_\{\\underline\{p\}\_\{0\}\\in\\mathcal\{C\}\_\{0\}^\{t\}\(s\_\{0\};\\delta\)\}\\sum\_\{s^\{\\prime\}\}\\bar\{p\}\_\{0\}\(s^\{\\prime\}\|s\_\{0\},a\_\{0\}\)\\underline\{V\}\_\{1\}^\{t\}\(s^\{\\prime\};\\delta\)Q¯ht,π​\(s,a;δ\):=\\displaystyle\\overline\{Q\}\_\{h\}^\{t,\\pi\}\(s,a;\\delta\):=rh​\(s,a\)\+maxp¯h∈𝒞ht​\(s,a;δ\)​∑s′p¯h​\(s′\|s,a\)​V¯h\+1t,π​\(s′;δ\)\\displaystyle r\_\{h\}\(s,a\)\+\\max\_\{\\bar\{p\}\_\{h\}\\in\\mathcal\{C\}\_\{h\}^\{t\}\(s,a;\\delta\)\}\\sum\_\{s^\{\\prime\}\}\\bar\{p\}\_\{h\}\(s^\{\\prime\}\|s,a\)\\overline\{V\}\_\{h\+1\}^\{t,\\pi\}\(s^\{\\prime\};\\delta\)V¯ht,π​\(s;δ\):=\\displaystyle\\overline\{V\}^\{t,\\pi\}\_\{h\}\(s;\\delta\):=Q¯ht,π​\(s,π​\(s\);δ\)\\displaystyle\\overline\{Q\}\_\{h\}^\{t,\\pi\}\\big\(s,\\pi\(s\);\\delta\\big\)V¯H\+1t,π​\(s;δ\):=\\displaystyle\\overline\{V\}^\{t,\\pi\}\_\{H\+1\}\(s;\\delta\):=0\\displaystyle 0p¯ht,π​\(s,a;δ\)∈\\displaystyle\\overline\{p\}\_\{h\}^\{t,\\pi\}\(s,a;\\delta\)\\inarg⁡maxp¯h∈𝒞ht​\(s,a,δ\)​∑s′p¯h​\(s′\|s,a\)​V¯h\+1t,π​\(s′;δ\)\\displaystyle\\arg\\max\_\{\\bar\{p\}\_\{h\}\\in\\mathcal\{C\}\_\{h\}^\{t\}\(s,a,\\delta\)\}\\sum\_\{s^\{\\prime\}\}\\bar\{p\}\_\{h\}\(s^\{\\prime\}\|s,a\)\\overline\{V\}\_\{h\+1\}^\{t,\\pi\}\(s^\{\\prime\};\\delta\)Q¯ht,π​\(s,a;δ\):=\\displaystyle\\underline\{Q\}\_\{h\}^\{t,\\pi\}\(s,a;\\delta\):=rh​\(s,a\)\+minp¯h∈𝒞ht​\(s,a;δ\)​∑s′p¯h​\(s′\|s,a\)​V¯h\+1t,π​\(s′;δ\)\\displaystyle r\_\{h\}\(s,a\)\+\\min\_\{\\underline\{p\}\_\{h\}\\in\\mathcal\{C\}\_\{h\}^\{t\}\(s,a;\\delta\)\}\\sum\_\{s^\{\\prime\}\}\\underline\{p\}\_\{h\}\(s^\{\\prime\}\|s,a\)\\underline\{V\}\_\{h\+1\}^\{t,\\pi\}\(s^\{\\prime\};\\delta\)V¯ht,π​\(s;δ\):=\\displaystyle\\underline\{V\}\_\{h\}^\{t,\\pi\}\(s;\\delta\):=Q¯ht,π​\(s,π​\(s\);δ\)\\displaystyle\\underline\{Q\}\_\{h\}^\{t,\\pi\}\(s,\\pi\(s\);\\delta\)V¯H\+1t,π​\(s;δ\):=\\displaystyle\\underline\{V\}\_\{H\+1\}^\{t,\\pi\}\(s;\\delta\):=0\\displaystyle 0p¯ht,π​\(s,a;δ\)\\displaystyle\\underline\{p\}\_\{h\}^\{t,\\pi\}\(s,a;\\delta\)∈arg⁡minp¯∈𝒞ht​\(s,a;δ\)​∑s′p¯h​\(s′\|s,a\)​V¯h\+1t,π​\(s′;δ\)\.\\displaystyle\\in\\arg\\min\_\{\\underline\{p\}\\in\\mathcal\{C\}\_\{h\}^\{t\}\(s,a;\\delta\)\}\\sum\_\{s^\{\\prime\}\}\\underline\{p\}\_\{h\}\(s^\{\\prime\}\|s,a\)\\underline\{V\}\_\{h\+1\}^\{t,\\pi\}\(s^\{\\prime\};\\delta\)\.Finally, we introduce the commonly used notationQQandVV, noted as the “True” value function\.

Qhπ​\(s,a\):=\\displaystyle Q\_\{h\}^\{\\pi\}\(s,a\):=rh​\(s,a\)\+∑s′ph​\(s′\|s,a\)​Vh\+1​\(s′\)\\displaystyle r\_\{h\}\(s,a\)\+\\sum\_\{s^\{\\prime\}\}p\_\{h\}\(s^\{\\prime\}\|s,a\)V\_\{h\+1\}\(s^\{\\prime\}\)Vhπ​\(s\):=\\displaystyle V\_\{h\}^\{\\pi\}\(s\):=Qhπ​\(s,π​\(s\)\)\\displaystyle Q\_\{h\}^\{\\pi\}\(s,\\pi\(s\)\)VH\+1π​\(s\):=\\displaystyle V\_\{H\+1\}^\{\\pi\}\(s\):=0\\displaystyle 0V0π​\(s0\)=\\displaystyle V\_\{0\}^\{\\pi\}\(s\_\{0\}\)=∑s′p0​\(s′\|s0,a0\)​V1π​\(s′\)\.\\displaystyle\\sum\_\{s^\{\\prime\}\}p\_\{0\}\(s^\{\\prime\}\|s\_\{0\},a\_\{0\}\)V\_\{1\}^\{\\pi\}\(s^\{\\prime\}\)\.

## Appendix BPerformance Guarantee of Algorithm[1](https://arxiv.org/html/2605.23182#alg1)

### B\.1Proof of Main Theorems

Recall we take default valueδk=13k\\delta\_\{k\}=\\frac\{1\}\{3^\{k\}\}in Algorithm[1](https://arxiv.org/html/2605.23182#alg1)\. We define

ℰcnt​\(δ\)=\\displaystyle\\mathcal\{E\}^\{\\text\{cnt\}\}\(\\delta\)=\{∀t∈ℕ∗,∀\(s,a,h\):nht​\(s,a\)≥12​n¯ht​\(s,a\)−βcnt​\(δ\)\}\\displaystyle\\left\\\{\\forall t\\in\\mathbb\{N\}^\{\*\},\\forall\(s,a,h\):n^\{t\}\_\{h\}\(s,a\)\\geq\\frac\{1\}\{2\}\\bar\{n\}\_\{h\}^\{t\}\(s,a\)\-\\beta^\{\\text\{cnt\}\}\(\\delta\)\\right\\\}\(7\)ℰ​\(δ\)=\\displaystyle\\mathcal\{E\}\(\\delta\)=\{∀t∈ℕ,∀h∈\[H\],∀\(s,a,h\),KL\(p^ht\(⋅\|s,a\),ph\(⋅\|s,a\)\)≤βp​\(nht​\(s,a\),δ\)nht​\(s,a\)\}\\displaystyle\\left\\\{\\forall t\\in\\mathbb\{N\},\\forall h\\in\[H\],\\forall\(s,a,h\),\\text\{KL\}\(\\hat\{p\}^\{t\}\_\{h\}\(\\cdot\|s,a\),p\_\{h\}\(\\cdot\|s,a\)\)\\leq\\frac\{\\beta\_\{p\}\(n\_\{h\}^\{t\}\(s,a\),\\delta\)\}\{n\_\{h\}^\{t\}\(s,a\)\}\\right\\\}\(8\)κ=\\displaystyle\\kappa=min⁡\{k:ℰ​\(δk\)∩ℰcnt​\(δk\)​hold\}\\displaystyle\\min\\\{k:\\mathcal\{E\}\(\\delta\_\{k\}\)\\cap\\mathcal\{E\}^\{\\text\{cnt\}\}\(\\delta\_\{k\}\)\\text\{ hold\}\\\}\(9\)ℰket=\\displaystyle\\mathcal\{E\}\_\{k\}^\{\\text\{et\}\}=\{∀N\>0,\|V^etπ^k,N−V0π^k\|≤4​H2​log⁡2​αk​\(log2⁡2​N\)2δN\},\\displaystyle\\left\\\{\\forall N\>0,\\left\|\\hat\{V\}\_\{\\text\{et\}\}^\{\\hat\{\\pi\}\_\{k\},N\}\-V^\{\\hat\{\\pi\}\_\{k\}\}\_\{0\}\\right\|\\leq\\sqrt\{\\frac\{4H^\{2\}\\log\\frac\{2\\alpha\_\{k\}\(\\log\_\{2\}2N\)^\{2\}\}\{\\delta\}\}\{N\}\}\\right\\\},\(10\)whereℰcnt​\(δ\),ℰ​\(δ\)\\mathcal\{E\}^\{\\text\{cnt\}\}\(\\delta\),\\mathcal\{E\}\(\\delta\)are only defined for the samples collected in the exploration period\. Recall A key feature of Algorithm[1](https://arxiv.org/html/2605.23182#alg1)is that sampling trajectories are shared across all exploration periods; consequently,\{ℰcnt​\(δk\),ℰ​\(δk\)\}k=1\+∞\\\{\\mathcal\{E\}^\{\\text\{cnt\}\}\(\\delta\_\{k\}\),\\mathcal\{E\}\(\\delta\_\{k\}\)\\\}\_\{k=1\}^\{\+\\infty\}are nested, i\.e\.,ℰcnt​\(δk\)⊂ℰcnt​\(δk\+1\)\\mathcal\{E\}^\{\\text\{cnt\}\}\(\\delta\_\{k\}\)\\subset\\mathcal\{E\}^\{\\text\{cnt\}\}\(\\delta\_\{k\+1\}\)andℰ​\(δk\)⊂ℰ​\(δk\+1\)\\mathcal\{E\}\(\\delta\_\{k\}\)\\subset\\mathcal\{E\}\(\\delta\_\{k\+1\}\)\.

We first prove the output correctness of Algorithm[1](https://arxiv.org/html/2605.23182#alg1)\.

###### Theorem\(Restatement of Theorem[1](https://arxiv.org/html/2605.23182#Thmtheorem1)\)\.

Algorithm[1](https://arxiv.org/html/2605.23182#alg1)isδ\\delta\-PAC\.

###### Proof of Theorem[1](https://arxiv.org/html/2605.23182#Thmtheorem1)\.

By the Lemma[8](https://arxiv.org/html/2605.23182#Thmtheorem8), we knowPrν⁡\(τ<\+∞\)=1\\Pr\_\{\\nu\}\(\\tau<\+\\infty\)=1holds no matterν\\nuis positive or negative\. The remaining work is to provePrν⁡\(π^≠None,V0π^​\(s0\)\>μ0\)≥1−δ\\Pr\_\{\\nu\}\(\\hat\{\\pi\}\\neq\\textsf\{None\},V^\{\\hat\{\\pi\}\}\_\{0\}\(s\_\{0\}\)\>\\mu\_\{0\}\)\\geq 1\-\\deltafor any positive instanceν\\nuandPrν⁡\(π^=None\)≥1−δ\\Pr\_\{\\nu\}\(\\hat\{\\pi\}=\\textsf\{None\}\)\\geq 1\-\\deltafor any negative instanceν\\nu\. Or equivalently,Prν⁡\(π^=Noneor​V0π^​\(s0\)<μ0\)<δ\\Pr\_\{\\nu\}\(\\hat\{\\pi\}=\\textsf\{None\}\\text\{ or \}V^\{\\hat\{\\pi\}\}\_\{0\}\(s\_\{0\}\)<\\mu\_\{0\}\)<\\deltaholds for any positive instanceν\\nu, andPrν⁡\(π^≠None\)<δ\\Pr\_\{\\nu\}\(\\hat\{\\pi\}\\neq\\textsf\{None\}\)<\\delta

Ifν\\nuis positive, we have

Prν⁡\(π^=Noneor​V0π^​\(s0\)<μ0\)≤Prν⁡\(π^=None\)\+Prν⁡\(π^≠None,V0π^​\(s0\)<μ0\)\.\\displaystyle\\Pr\_\{\\nu\}\(\\hat\{\\pi\}=\\textsf\{None\}\\text\{ or \}V^\{\\hat\{\\pi\}\}\_\{0\}\(s\_\{0\}\)<\\mu\_\{0\}\)\\leq\\Pr\_\{\\nu\}\(\\hat\{\\pi\}=\\textsf\{None\}\)\+\\Pr\_\{\\nu\}\(\\hat\{\\pi\}\\neq\\textsf\{None\},V^\{\\hat\{\\pi\}\}\_\{0\}\(s\_\{0\}\)<\\mu\_\{0\}\)\.\(11\)We are going to provePrν⁡\(π^=None\)≤\(1\+π23\)​3​δ10\\Pr\_\{\\nu\}\(\\hat\{\\pi\}=\\textsf\{None\}\)\\leq\(1\+\\frac\{\\pi^\{2\}\}\{3\}\)\\frac\{3\\delta\}\{10\}andPrν\(∃k∈ℕ,π^≠None,V0π^\(s0\)≤π2​δ24\\Pr\_\{\\nu\}\(\\exists k\\in\\mathbb\{N\},\\hat\{\\pi\}\\neq\\textsf\{None\},V^\{\\hat\{\\pi\}\}\_\{0\}\(s\_\{0\}\)\\leq\\frac\{\\pi^\{2\}\\delta\}\{24\}\.

By the Line \([8](https://arxiv.org/html/2605.23182#alg1.l8)\) in Algorithm[1](https://arxiv.org/html/2605.23182#alg1), we have

Prν⁡\(π^=None\)\\displaystyle\\Pr\_\{\\nu\}\(\\hat\{\\pi\}=\\textsf\{None\}\)≤\\displaystyle\\leqPrν⁡\(∃δk≤δ10,∃t∈ℕ,V¯0π¯t​\(⋅;δk\)​\(s0;δk\)<μ0\)\\displaystyle\\Pr\_\{\\nu\}\\left\(\\exists\\delta\_\{k\}\\leq\\frac\{\\delta\}\{10\},\\exists t\\in\\mathbb\{N\},\\overline\{V\}^\{\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)<\\mu\_\{0\}\\right\)≤\\displaystyle\\leqPrν⁡\(∃δk≤δ10,∃t∈ℕ,V¯0π¯t​\(⋅;δk\)​\(s0;δk\)<V0∗​\(s0\)\)\\displaystyle\\Pr\_\{\\nu\}\\left\(\\exists\\delta\_\{k\}\\leq\\frac\{\\delta\}\{10\},\\exists t\\in\\mathbb\{N\},\\overline\{V\}^\{\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)<V\_\{0\}^\{\*\}\(s\_\{0\}\)\\right\)≤\\displaystyle\\leqPrν⁡\(∃k≥⌈log3⁡10δ⌉,κ≥k\)\\displaystyle\\Pr\_\{\\nu\}\\left\(\\exists k\\geq\\lceil\\log\_\{3\}\\frac\{10\}\{\\delta\}\\rceil,\\kappa\\geq k\\right\)\(12\)≤\\displaystyle\\leq13⌈log3⁡10δ⌉−1\.\\displaystyle\\frac\{1\}\{3^\{\\lceil\\log\_\{3\}\\frac\{10\}\{\\delta\}\\rceil\-1\}\}\.\(13\)≤\\displaystyle\\leq3​δ10\.\\displaystyle\\frac\{3\\delta\}\{10\}\.\(14\)Step \([12](https://arxiv.org/html/2605.23182#A2.E12)\) is by the Proposition[12](https://arxiv.org/html/2605.23182#Thmtheorem12)and Step \([13](https://arxiv.org/html/2605.23182#A2.E13)\) by the Lemma[10](https://arxiv.org/html/2605.23182#Thmtheorem10)\.

By the Line \([14](https://arxiv.org/html/2605.23182#alg1.l14)\) in Algorithm[1](https://arxiv.org/html/2605.23182#alg1), we have

Prν⁡\(∃k∈ℕ,π^k​is a policy,V0π^k​\(s0\)<μ0,π^=π^k\)\\displaystyle\\Pr\_\{\\nu\}\(\\exists k\\in\\mathbb\{N\},\\hat\{\\pi\}\_\{k\}\\text\{ is a policy\},V^\{\\hat\{\\pi\}\_\{k\}\}\_\{0\}\(s\_\{0\}\)<\\mu\_\{0\},\\hat\{\\pi\}=\\hat\{\\pi\}\_\{k\}\)≤\\displaystyle\\leqPrν⁡\(∃k∈ℕ,∃N∈ℕ,V^etπ^k,N−4​H2​log⁡2​αk​\(log2⁡2​N\)2δN≥μ0\),\\displaystyle\\Pr\_\{\\nu\}\\left\(\\exists k\\in\\mathbb\{N\},\\exists N\\in\\mathbb\{N\},\\hat\{V\}\_\{\\text\{et\}\}^\{\\hat\{\\pi\}\_\{k\},N\}\-\\sqrt\{\\frac\{4H^\{2\}\\log\\frac\{2\\alpha\_\{k\}\(\\log\_\{2\}2N\)^\{2\}\}\{\\delta\}\}\{N\}\}\\geq\\mu\_\{0\}\\right\),whereV^etπ^k,N=∑i=1NXi\(k\)N\\hat\{V\}\_\{\\text\{et\}\}^\{\\hat\{\\pi\}\_\{k\},N\}=\\frac\{\\sum\_\{i=1\}^\{N\}X\_\{i\}^\{\(k\)\}\}\{N\},\{Xi\(k\)\}i=1\+∞\\\{X\_\{i\}^\{\(k\)\}\\\}\_\{i=1\}^\{\+\\infty\}are i\.i\.d random variables, denoting the rewards collected by executing policyπ^k\\hat\{\\pi\}\_\{k\}during a single episode\. The mean value for eachXi\(k\)X\_\{i\}^\{\(k\)\}isV0π^k​\(s0\)<μ0V\_\{0\}^\{\\hat\{\\pi\}\_\{k\}\}\(s\_\{0\}\)<\\mu\_\{0\}\. SinceXi\(k\)X\_\{i\}^\{\(k\)\}is bounded by the interval\[0,H\]\[0,H\], we can assertXi\(k\)−V0π^k​\(s0\)X\_\{i\}^\{\(k\)\}\-V\_\{0\}^\{\\hat\{\\pi\}\_\{k\}\}\(s\_\{0\}\)isH24\\frac\{H^\{2\}\}\{4\}\-subgaussin\. By the union bound, we have

Prν⁡\(∃k∈ℕ,π^≠None,V0π^​\(s0\)<μ0\)\\displaystyle\\Pr\_\{\\nu\}\(\\exists k\\in\\mathbb\{N\},\\hat\{\\pi\}\\neq\\textsf\{None\},V^\{\\hat\{\\pi\}\}\_\{0\}\(s\_\{0\}\)<\\mu\_\{0\}\)≤\\displaystyle\\leq∑k=1\+∞Prν⁡\(∃N∈ℕ,∑i=1NXi\(k\)N−H2​log⁡2​αk​\(log2⁡2​N\)2δN≥μ0\)\\displaystyle\\sum\_\{k=1\}^\{\+\\infty\}\\Pr\_\{\\nu\}\\left\(\\exists N\\in\\mathbb\{N\},\\frac\{\\sum\_\{i=1\}^\{N\}X\_\{i\}^\{\(k\)\}\}\{N\}\-\\sqrt\{\\frac\{H^\{2\}\\log\\frac\{2\\alpha\_\{k\}\(\\log\_\{2\}2N\)^\{2\}\}\{\\delta\}\}\{N\}\}\\geq\\mu\_\{0\}\\right\)≤\\displaystyle\\leq∑k=1\+∞Prν⁡\(∃N∈ℕ,∑i=1NXi\(k\)N−H2​log⁡2​αk​\(log2⁡2​N\)2δN≥V0π^k​\(s0\)\)\\displaystyle\\sum\_\{k=1\}^\{\+\\infty\}\\Pr\_\{\\nu\}\\left\(\\exists N\\in\\mathbb\{N\},\\frac\{\\sum\_\{i=1\}^\{N\}X\_\{i\}^\{\(k\)\}\}\{N\}\-\\sqrt\{\\frac\{H^\{2\}\\log\\frac\{2\\alpha\_\{k\}\(\\log\_\{2\}2N\)^\{2\}\}\{\\delta\}\}\{N\}\}\\geq V\_\{0\}^\{\\hat\{\\pi\}\_\{k\}\}\(s\_\{0\}\)\\right\)≤\\displaystyle\\leq∑k=1\+∞π26​δαk\\displaystyle\\sum\_\{k=1\}^\{\+\\infty\}\\frac\{\\pi^\{2\}\}\{6\}\\frac\{\\delta\}\{\\alpha\_\{k\}\}≤\\displaystyle\\leqπ2​δ6​14\.\\displaystyle\\frac\{\\pi^\{2\}\\delta\}\{6\}\\frac\{1\}\{4\}\.\(15\)The second last step is by the Lemma[23](https://arxiv.org/html/2605.23182#Thmtheorem23)and the last step is by the default settingαk=15k\\alpha\_\{k\}=\\frac\{1\}\{5^\{k\}\}\. Substitute \([15](https://arxiv.org/html/2605.23182#A2.E15)\) and \([14](https://arxiv.org/html/2605.23182#A2.E14)\) into \([11](https://arxiv.org/html/2605.23182#A2.E11)\), we have

Prν⁡\(π^=Noneor​V0π^​\(s0\)<μ0\)≤π2​δ6​14\+3​δ10<δ\.\\displaystyle\\Pr\_\{\\nu\}\(\\hat\{\\pi\}=\\textsf\{None\}\\text\{ or \}V^\{\\hat\{\\pi\}\}\_\{0\}\(s\_\{0\}\)<\\mu\_\{0\}\)\\leq\\frac\{\\pi^\{2\}\\delta\}\{6\}\\frac\{1\}\{4\}\+\\frac\{3\\delta\}\{10\}<\\delta\.
Ifν\\nuis negative, we can follow the calculation of \([15](https://arxiv.org/html/2605.23182#A2.E15)\), and conclude

Prν⁡\(π^≠None\)≤π2​δ6​14<δ\.\\displaystyle\\Pr\_\{\\nu\}\(\\hat\{\\pi\}\\neq\\textsf\{None\}\)\\leq\\frac\{\\pi^\{2\}\\delta\}\{6\}\\frac\{1\}\{4\}<\\delta\.∎

Then, we turn to prove the upper bound of𝔼​τ\\mathbb\{E\}\\tau, by applying Algorithm[1](https://arxiv.org/html/2605.23182#alg1)\.

###### Theorem\(Restatement of Theorem[2](https://arxiv.org/html/2605.23182#Thmtheorem2)\)\.

Apply Algorithm[1](https://arxiv.org/html/2605.23182#alg1)to a positive instanceν\\nuwithV0∗​\(s0\)\>μ0V^\{\*\}\_\{0\}\(s\_\{0\}\)\>\\mu\_\{0\}, we have

𝔼​τ≤O​\(H2​log⁡1δ\+\(H\+1\)4​S​A​log⁡S​A​H\(V0∗​\(s0\)−μ0\)2\+\(H\+1\)4​S2​A​log⁡\(H​S​A\)\(V0∗​\(s0\)−μ0\)2\)\\displaystyle\\mathbb\{E\}\\tau\\leq O\\left\(\\frac\{H^\{2\}\\log\\frac\{1\}\{\\delta\}\+\(H\+1\)^\{4\}SA\\log\\frac\{SAH\}\{\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\+\(H\+1\)^\{4\}S^\{2\}A\\log\\left\(HSA\\right\)\}\{\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)

###### Proof of Theorem[2](https://arxiv.org/html/2605.23182#Thmtheorem2)\.

DenoteLpos=max⁡\{κ,⌈log2⁡9600​\(H\+1\)2​C2\(C−1\)2​\(V0∗​\(s0\)−μ0\)2⌉\}L^\{\\text\{pos\}\}=\\max\\\{\\kappa,\\lceil\\log\_\{2\}\\frac\{9600\(H\+1\)^\{2\}C^\{2\}\}\{\(C\-1\)^\{2\}\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\rceil\\\}, and define

κend=min⁡\{k:k≥Lpos,ℰket​holds\}\.\\displaystyle\\kappa\_\{\\text\{end\}\}=\\min\\left\\\{k:k\\geq L^\{\\text\{pos\}\},\\mathcal\{E\}\_\{k\}^\{\\text\{et\}\}\\text\{ holds\}\\right\\\}\.By the Lemma[7](https://arxiv.org/html/2605.23182#Thmtheorem7), We know Algorithm[1](https://arxiv.org/html/2605.23182#alg1)will terminate no later than the end of phaseκend\\kappa\_\{\\text\{end\}\}\.

Givenκ<\+∞\\kappa<\+\\infty, and indexk≥1k\\geq 1, we know

Pr⁡\(κend≥Lpos\+k\|κ\)\\displaystyle\\Pr\(\\kappa\_\{\\text\{end\}\}\\geq L^\{\\text\{pos\}\}\+k\|\\kappa\)≤\\displaystyle\\leqPr⁡\(\(¬ℰLposet\)∩⋯∩\(¬ℰLpos\+k−1et\)\|κ\)\\displaystyle\\Pr\\big\(\(\\neg\\mathcal\{E\}\_\{L^\{\\text\{pos\}\}\}^\{\\text\{et\}\}\)\\cap\\cdots\\cap\(\\neg\\mathcal\{E\}\_\{L^\{\\text\{pos\}\}\+k\-1\}^\{\\text\{et\}\}\)\|\\kappa\\big\)≤\\displaystyle\\leqPr⁡\(¬ℰLpos\+k−1et\|κ\)\\displaystyle\\Pr\(\\neg\\mathcal\{E\}\_\{L^\{\\text\{pos\}\}\+k\-1\}^\{\\text\{et\}\}\|\\kappa\)≤\\displaystyle\\leqπ2​δ6​αLpos\+k−1\\displaystyle\\frac\{\\pi^\{2\}\\delta\}\{6\\alpha\_\{L^\{\\text\{pos\}\}\+k\-1\}\}\(16\)The last step is by the Lemma[23](https://arxiv.org/html/2605.23182#Thmtheorem23)\.

Denoteτee\\tau^\{\\text\{ee\}\},τet\\tau^\{\\text\{et\}\}as the total collected episodes by the exploration and exploitation oracle\. Theorem[20](https://arxiv.org/html/2605.23182#Thmtheorem20)suggests that

τee≤\\displaystyle\\tau^\{\\text\{ee\}\}\\leq101​\(H\+1\)4​S​A​log⁡2​S​A​Hδκendϵpos2\+Tκ−1ee\+\\displaystyle\\frac\{101\(H\+1\)^\{4\}SA\\log\\frac\{2SAH\}\{\\delta\_\{\\kappa\_\{\\text\{end\}\}\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\+T\_\{\\kappa\-1\}^\{\\text\{ee\}\}\+400​\(H\+1\)4​S2​Aϵpos2​log⁡\(402​\(H\+1\)4​S2​A​log⁡2​S​A​Hδκendϵpos2\)\+400​\(H\+1\)4​S2​Aϵpos2​log⁡\(2​Tκ−1ee\)\\displaystyle\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\log\\left\(\\frac\{402\(H\+1\)^\{4\}S^\{2\}A\\log\\frac\{2SAH\}\{\\delta\_\{\\kappa\_\{\\text\{end\}\}\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\right\)\+\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\log\\left\(2T\_\{\\kappa\-1\}^\{\\text\{ee\}\}\\right\)τet≤\\displaystyle\\tau^\{\\text\{et\}\}\\leq∑k=1κend100⋅log⁡αkδ\+log⁡log⁡24​H2ϵkϵk\\displaystyle\\sum\_\{k=1\}^\{\\kappa\_\{\\text\{end\}\}\}100\\cdot\\frac\{\\log\\frac\{\\alpha\_\{k\}\}\{\\delta\}\+\\log\\log\\frac\{24H^\{2\}\}\{\\epsilon\_\{k\}\}\}\{\\epsilon\_\{k\}\}whereϵpos=V0∗​\(s0\)−μ0C\\epsilon^\{\\text\{pos\}\}=\\frac\{V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\}\{C\}\. For simplicity, we denoteV0∗:=V0∗​\(s0\)V\_\{0\}^\{\*\}:=V\_\{0\}^\{\*\}\(s\_\{0\}\)in the following proof\. We first derive an upper bound for𝔼​τee\\mathbb\{E\}\\tau^\{\\text\{ee\}\}\. By \([16](https://arxiv.org/html/2605.23182#A2.E16)\), we knowPr⁡\(κend−Lpos\+k\|κ\)≤π2​δ6​αLpos\+k−1,k≥1\\Pr\(\\kappa\_\{\\text\{end\}\}\-L^\{\\text\{pos\}\}\+k\|\\kappa\)\\leq\\frac\{\\pi^\{2\}\\delta\}\{6\\alpha\_\{L^\{\\text\{pos\}\}\+k\-1\}\},k\\geq 1, thus, we can conclude

𝔼​\[τee\|κ\]\\displaystyle\\mathbb\{E\}\[\\tau^\{\\text\{ee\}\}\|\\kappa\]≤\\displaystyle\\leq101​\(H\+1\)4​S​A​log⁡2​S​A​HδLposϵpos2\+∑k=1\+∞π26​δ5k−1​101​\(H\+1\)4​S​A​log⁡2​S​A​HδLpos\+kϵpos2\\displaystyle\\frac\{101\(H\+1\)^\{4\}SA\\log\\frac\{2SAH\}\{\\delta\_\{L^\{\\text\{pos\}\}\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\+\\sum\_\{k=1\}^\{\+\\infty\}\\frac\{\\pi^\{2\}\}\{6\}\\frac\{\\delta\}\{5^\{k\-1\}\}\\frac\{101\(H\+1\)^\{4\}SA\\log\\frac\{2SAH\}\{\\delta\_\{L^\{\\text\{pos\}\}\+k\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}Tκ−1ee\+400​\(H\+1\)4​S2​Aϵpos2​log⁡\(402​\(H\+1\)4​S2​A​log⁡2​S​A​HδLposϵpos2\)\+\\displaystyle T\_\{\\kappa\-1\}^\{\\text\{ee\}\}\+\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\log\\left\(\\frac\{402\(H\+1\)^\{4\}S^\{2\}A\\log\\frac\{2SAH\}\{\\delta\_\{L^\{\\text\{pos\}\}\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\right\)\+∑k=1\+∞π26​δ5k−1​400​\(H\+1\)4​S2​Aϵpos2​log⁡\(402​\(H\+1\)4​S2​A​log⁡2​S​A​HδLpos\+kϵpos2\)\+\\displaystyle\\sum\_\{k=1\}^\{\+\\infty\}\\frac\{\\pi^\{2\}\}\{6\}\\frac\{\\delta\}\{5^\{k\-1\}\}\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\log\\left\(\\frac\{402\(H\+1\)^\{4\}S^\{2\}A\\log\\frac\{2SAH\}\{\\delta\_\{L^\{\\text\{pos\}\}\+k\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\right\)\+400​\(H\+1\)4​S2​Aϵpos2​log⁡\(2​Tκ−1ee\)\\displaystyle\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\log\\left\(2T\_\{\\kappa\-1\}^\{\\text\{ee\}\}\\right\)≤\\displaystyle\\leqO​\(1\)​\(H\+1\)4​S​A​log⁡2​S​A​HδLposϵpos2⏟†1\+O​\(1\)​\(H\+1\)4​S2​Aϵpos2​log⁡\(\(H\+1\)4​S2​A​log⁡2​S​A​HδLposϵpos2\)⏟†2\+\\displaystyle O\(1\)\\underbrace\{\\frac\{\(H\+1\)^\{4\}SA\\log\\frac\{2SAH\}\{\\delta\_\{L^\{\\text\{pos\}\}\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\}\_\{\\dagger 1\}\+O\(1\)\\underbrace\{\\frac\{\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\log\\left\(\\frac\{\(H\+1\)^\{4\}S^\{2\}A\\log\\frac\{2SAH\}\{\\delta\_\{L^\{\\text\{pos\}\}\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\right\)\}\_\{\\dagger 2\}\+400​\(H\+1\)4​S2​Aϵpos2​log⁡\(2​Tκ−1ee\)\+Tκ−1ee⏟†3\\displaystyle\\underbrace\{\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\log\\left\(2T\_\{\\kappa\-1\}^\{\\text\{ee\}\}\\right\)\+T\_\{\\kappa\-1\}^\{\\text\{ee\}\}\}\_\{\\dagger 3\}To move forward, we take expectation on†1\\dagger 1,†2\\dagger 2and†3\\dagger 3\. By the Lemma[9](https://arxiv.org/html/2605.23182#Thmtheorem9), we have

𝔼​\(H\+1\)4​S​A​log⁡2​S​A​HδLposϵpos2\\displaystyle\\mathbb\{E\}\\frac\{\(H\+1\)^\{4\}SA\\log\\frac\{2SAH\}\{\\delta\_\{L^\{\\text\{pos\}\}\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}≤\\displaystyle\\leq𝔼​\(\(H\+1\)4​S​A​log⁡2​S​A​Hδκϵpos2\+\(H\+1\)4​S​A​log⁡2​S​A​Hδ⌈log2⁡9600​H2​C2\(C−1\)2​\(V0∗​\(s0\)−μ0\)2⌉ϵpos2\)\\displaystyle\\mathbb\{E\}\\left\(\\frac\{\(H\+1\)^\{4\}SA\\log\\frac\{2SAH\}\{\\delta\_\{\\kappa\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\+\\frac\{\(H\+1\)^\{4\}SA\\log\\frac\{2SAH\}\{\\delta\_\{\\lceil\\log\_\{2\}\\frac\{9600H^\{2\}C^\{2\}\}\{\(C\-1\)^\{2\}\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\rceil\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\right\)≤\\displaystyle\\leq∑k=1\+∞13k−1⋅\(H\+1\)4​S​A​log⁡\(2​S​A​H⋅3k\)ϵpos2\+O​\(\(H\+1\)4​S​A​log⁡S​A​H​C2\(C−1\)2​\(V0∗​\(s0\)−μ0\)2\(V0∗−μ0\)2\)\\displaystyle\\sum\_\{k=1\}^\{\+\\infty\}\\frac\{1\}\{3^\{k\-1\}\}\\cdot\\frac\{\(H\+1\)^\{4\}SA\\log\(2SAH\\cdot 3^\{k\}\)\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\+O\\left\(\\frac\{\(H\+1\)^\{4\}SA\\log\\frac\{SAHC^\{2\}\}\{\(C\-1\)^\{2\}\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\}\{\(V\_\{0\}^\{\*\}\-\\mu\_\{0\}\)^\{2\}\}\\right\)≤\\displaystyle\\leqO​\(\(H\+1\)4​S​A​log⁡S​A​H​C2\(C−1\)2​\(V0∗​\(s0\)−μ0\)2\(V0∗​\(s0\)−μ0\)2\)\.\\displaystyle O\\left\(\\frac\{\(H\+1\)^\{4\}SA\\log\\frac\{SAHC^\{2\}\}\{\(C\-1\)^\{2\}\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\}\{\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\.\(17\)𝔼​\(H\+1\)4​S2​Aϵpos2​log⁡\(\(H\+1\)4​S2​A​log⁡2​S​A​HδLposϵpos2\)\\displaystyle\\mathbb\{E\}\\frac\{\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\log\\left\(\\frac\{\(H\+1\)^\{4\}S^\{2\}A\\log\\frac\{2SAH\}\{\\delta\_\{L^\{\\text\{pos\}\}\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\right\)≤\\displaystyle\\leq\(H\+1\)4​S2​Aϵpos2​log⁡\(\(H\+1\)4​S2​Aϵpos2\)\+𝔼​\(H\+1\)4​S2​A​log⁡log⁡2​S​A​HδLposϵpos2\\displaystyle\\frac\{\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\log\\left\(\\frac\{\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\right\)\+\\mathbb\{E\}\\frac\{\(H\+1\)^\{4\}S^\{2\}A\\log\\log\\frac\{2SAH\}\{\\delta\_\{L^\{\\text\{pos\}\}\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}≤\\displaystyle\\leq\(H\+1\)4​S2​Aϵpos2​log⁡\(\(H\+1\)4​S2​Aϵpos2\)\+∑k=1\+∞13k−1​\(H\+1\)4​S2​A​log⁡log⁡2​S​A​Hδkϵpos2\+\\displaystyle\\frac\{\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\log\\left\(\\frac\{\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\right\)\+\\sum\_\{k=1\}^\{\+\\infty\}\\frac\{1\}\{3^\{k\-1\}\}\\frac\{\(H\+1\)^\{4\}S^\{2\}A\\log\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\+\(H\+1\)4​S2​A​log⁡log⁡2​S​A​Hδ⌈log2⁡9600​\(H\+1\)2​C2\(C−1\)2​\(V0∗​\(s0\)−μ0\)2⌉ϵpos2\\displaystyle\\frac\{\(H\+1\)^\{4\}S^\{2\}A\\log\\log\\frac\{2SAH\}\{\\delta\_\{\\lceil\\log\_\{2\}\\frac\{9600\(H\+1\)^\{2\}C^\{2\}\}\{\(C\-1\)^\{2\}\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\rceil\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}≤\\displaystyle\\leqO​\(C2​\(H\+1\)4​S2​A\(V0∗​\(s0\)−μ0\)2​log⁡\(C2​\(H\+1\)4​S2​A​log⁡C2​\(H\+1\)2\(C−1\)2​\(V0∗​\(s0\)−μ0\)2\(V0∗​\(s0\)−μ0\)2\)\)\\displaystyle O\\left\(\\frac\{C^\{2\}\(H\+1\)^\{4\}S^\{2\}A\}\{\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\log\\left\(\\frac\{C^\{2\}\(H\+1\)^\{4\}S^\{2\}A\\log\\frac\{C^\{2\}\(H\+1\)^\{2\}\}\{\(C\-1\)^\{2\}\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\}\{\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\\right\)\(18\)𝔼​400​\(H\+1\)4​S2​Aϵpos2​log⁡\(2​Tκ−1ee\)\+Tκ−1ee\\displaystyle\\mathbb\{E\}\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\log\\left\(2T\_\{\\kappa\-1\}^\{\\text\{ee\}\}\\right\)\+T\_\{\\kappa\-1\}^\{\\text\{ee\}\}≤\\displaystyle\\leq∑k=1\+∞13k−1​\(\(H\+1\)2​S​A​log⁡2​S​A​Hδkϵk\+\(H\+1\)2​S2​Aϵk​log⁡\(\(H\+1\)2​S2​A​log⁡2​S​A​Hδkϵk\)\)\+\\displaystyle\\sum\_\{k=1\}^\{\+\\infty\}\\frac\{1\}\{3^\{k\-1\}\}\\left\(\\frac\{\(H\+1\)^\{2\}SA\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon\_\{k\}\}\}\+\\frac\{\(H\+1\)^\{2\}S^\{2\}A\}\{\\epsilon\_\{k\}\}\\log\\left\(\\frac\{\(H\+1\)^\{2\}S^\{2\}A\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\\epsilon\_\{k\}\}\\right\)\\right\)\+400​\(H\+1\)4​S2​A​log⁡2ϵpos2\+400​\(H\+1\)4​S2​Aϵpos2⋅\\displaystyle\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\\log 2\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\+\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\cdot∑k=1\+∞13k−1​log⁡\(\(H\+1\)2​S​A​log⁡2​S​A​Hδkϵk\+\(H\+1\)2​S2​Aϵk​log⁡\(\(H\+1\)2​S2​A​log⁡2​S​A​Hδkϵk\)\)\\displaystyle\\sum\_\{k=1\}^\{\+\\infty\}\\frac\{1\}\{3^\{k\-1\}\}\\log\\left\(\\frac\{\(H\+1\)^\{2\}SA\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon\_\{k\}\}\}\+\\frac\{\(H\+1\)^\{2\}S^\{2\}A\}\{\\epsilon\_\{k\}\}\\log\\left\(\\frac\{\(H\+1\)^\{2\}S^\{2\}A\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\\epsilon\_\{k\}\}\\right\)\\right\)≤\\displaystyle\\leqO​\(C2​\(H\+1\)4​S2​A​log⁡\(S​A​H\)\(V0∗​\(s0\)−μ0\)2\+\(H\+1\)2​S2​A​log⁡\(S​A​H\)\)\\displaystyle O\\Big\(\\frac\{C^\{2\}\(H\+1\)^\{4\}S^\{2\}A\\log\(SAH\)\}\{\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\+\(H\+1\)^\{2\}S^\{2\}A\\log\(SAH\)\\Big\)≤\\displaystyle\\leqO​\(C2​\(H\+1\)4​S2​A​log⁡\(S​A​H\)\(V0∗​\(s0\)−μ0\)2\)\.\\displaystyle O\\Big\(\\frac\{C^\{2\}\(H\+1\)^\{4\}S^\{2\}A\\log\(SAH\)\}\{\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\Big\)\.\(19\)The last step is by the fact that\(V0∗​\(s0\)−μ0\)2≤H2\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\\leq H^\{2\}\. Sum up \([17](https://arxiv.org/html/2605.23182#A2.E17)\), \([18](https://arxiv.org/html/2605.23182#A2.E18)\) \([19](https://arxiv.org/html/2605.23182#A2.E19)\) and plug inC=1\.01C=1\.01, we can conclude

𝔼​τee\\displaystyle\\mathbb\{E\}\\tau^\{\\text\{ee\}\}=\\displaystyle=𝔼​𝔼​\[τee\|κ\]\\displaystyle\\mathbb\{E\}\\mathbb\{E\}\[\\tau^\{\\text\{ee\}\}\|\\kappa\]≤\\displaystyle\\leqO​\(\(H\+1\)4​S​A​log⁡S​A​H\(V0∗​\(s0\)−μ0\)2\(V0∗​\(s0\)−μ0\)2\+\(H\+1\)4​S2​A​log⁡\(H​S​A\)\(V0∗​\(s0\)−μ0\)2\)\\displaystyle O\\left\(\\frac\{\(H\+1\)^\{4\}SA\\log\\frac\{SAH\}\{\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\}\{\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\+\\frac\{\(H\+1\)^\{4\}S^\{2\}A\\log\\left\(HSA\\right\)\}\{\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\(20\)
The remaining work is to derive an upper bound for𝔼​τet\\mathbb\{E\}\\tau^\{\\text\{et\}\}, we also first derive an upper bound for𝔼​\[τet\|κ\]\\mathbb\{E\}\[\\tau^\{\\text\{et\}\}\|\\kappa\], which is

𝔼​\[τet\|κ\]≤\\displaystyle\\mathbb\{E\}\[\\tau^\{\\text\{et\}\}\|\\kappa\]\\leq𝔼​\[∑k=1κend100⋅log⁡αkδ\+log⁡log⁡24​H2ϵkϵk\|κ\]\\displaystyle\\mathbb\{E\}\\left\[\\sum\_\{k=1\}^\{\\kappa\_\{\\text\{end\}\}\}100\\cdot\\frac\{\\log\\frac\{\\alpha\_\{k\}\}\{\\delta\}\+\\log\\log\\frac\{24H^\{2\}\}\{\\epsilon\_\{k\}\}\}\{\\epsilon\_\{k\}\}\|\\kappa\\right\]≤\\displaystyle\\leq𝔼​\[200⋅log⁡ακendδ\+log⁡log⁡24​H2ϵκendϵκend\|κ\]\\displaystyle\\mathbb\{E\}\\left\[200\\cdot\\frac\{\\log\\frac\{\\alpha\_\{\\kappa\_\{\\text\{end\}\}\}\}\{\\delta\}\+\\log\\log\\frac\{24H^\{2\}\}\{\\epsilon\_\{\\kappa\_\{\\text\{end\}\}\}\}\}\{\\epsilon\_\{\\kappa\_\{\\text\{end\}\}\}\}\|\\kappa\\right\]=\\displaystyle=𝔼​\[200⋅log⁡ακend−Lpos⋅αLposδ\+log⁡log⁡24​H2ϵκend−Lpos⋅ϵLposϵκend−Lpos⋅ϵLpos\|κ\]\\displaystyle\\mathbb\{E\}\\left\[200\\cdot\\frac\{\\log\\frac\{\\alpha\_\{\\kappa\_\{\\text\{end\}\}\-L^\{\\text\{pos\}\}\}\\cdot\\alpha\_\{L^\{\\text\{pos\}\}\}\}\{\\delta\}\+\\log\\log\\frac\{24H^\{2\}\}\{\\epsilon\_\{\\kappa\_\{\\text\{end\}\}\-L^\{\\text\{pos\}\}\}\\cdot\\epsilon\_\{L^\{\\text\{pos\}\}\}\}\}\{\\epsilon\_\{\\kappa\_\{\\text\{end\}\}\-L^\{\\text\{pos\}\}\}\\cdot\\epsilon\_\{L^\{\\text\{pos\}\}\}\}\|\\kappa\\right\]≤\\displaystyle\\leq200​log⁡αLposδ\+200​log⁡log⁡24​H2ϵLposϵLpos\+∑k=1\+∞200​π2​δ6​αk−1⋅log⁡αk⋅αLposδ\+log⁡log⁡24​H2ϵk⋅ϵLposϵk⋅ϵLpos\\displaystyle\\frac\{200\\log\\frac\{\\alpha\_\{L^\{\\text\{pos\}\}\}\}\{\\delta\}\+200\\log\\log\\frac\{24H^\{2\}\}\{\\epsilon\_\{L^\{\\text\{pos\}\}\}\}\}\{\\epsilon\_\{L^\{\\text\{pos\}\}\}\}\+\\sum\_\{k=1\}^\{\+\\infty\}\\frac\{200\\pi^\{2\}\\delta\}\{6\\alpha\_\{k\-1\}\}\\cdot\\frac\{\\log\\frac\{\\alpha\_\{k\}\\cdot\\alpha\_\{L^\{\\text\{pos\}\}\}\}\{\\delta\}\+\\log\\log\\frac\{24H^\{2\}\}\{\\epsilon\_\{k\}\\cdot\\epsilon\_\{L^\{\\text\{pos\}\}\}\}\}\{\\epsilon\_\{k\}\\cdot\\epsilon\_\{L^\{\\text\{pos\}\}\}\}\(21\)=\\displaystyle=200​log⁡αLposδ\+200​log⁡log⁡24​H2ϵLposϵLpos\+∑k=1\+∞200​π2​δ⋅2k6⋅5k−1⋅k​log⁡5\+log⁡αLposδ\+log⁡log⁡24​H2⋅2kϵLposϵLpos\\displaystyle\\frac\{200\\log\\frac\{\\alpha\_\{L^\{\\text\{pos\}\}\}\}\{\\delta\}\+200\\log\\log\\frac\{24H^\{2\}\}\{\\epsilon\_\{L^\{\\text\{pos\}\}\}\}\}\{\\epsilon\_\{L^\{\\text\{pos\}\}\}\}\+\\sum\_\{k=1\}^\{\+\\infty\}\\frac\{200\\pi^\{2\}\\delta\\cdot 2^\{k\}\}\{6\\cdot 5^\{k\-1\}\}\\cdot\\frac\{k\\log 5\+\\log\\frac\{\\alpha\_\{L^\{\\text\{pos\}\}\}\}\{\\delta\}\+\\log\\log\\frac\{24H^\{2\}\\cdot 2^\{k\}\}\{\\epsilon\_\{L^\{\\text\{pos\}\}\}\}\}\{\\epsilon\_\{L^\{\\text\{pos\}\}\}\}≤\\displaystyle\\leqO​\(log⁡αLposδ\+log⁡log⁡24​H2ϵLposϵLpos\)\.\\displaystyle O\\left\(\\frac\{\\log\\frac\{\\alpha\_\{L^\{\\text\{pos\}\}\}\}\{\\delta\}\+\\log\\log\\frac\{24H^\{2\}\}\{\\epsilon\_\{L^\{\\text\{pos\}\}\}\}\}\{\\epsilon\_\{L^\{\\text\{pos\}\}\}\}\\right\)\.Step \([21](https://arxiv.org/html/2605.23182#A2.E21)\) is from the \([16](https://arxiv.org/html/2605.23182#A2.E16)\)\. To derive an upper bound for𝔼​τet\\mathbb\{E\}\\tau^\{\\text\{et\}\}, we can conduct the following calculation

𝔼​τet\\displaystyle\\mathbb\{E\}\\tau^\{\\text\{et\}\}≤\\displaystyle\\leq𝔼​𝔼​\[τet\|κ\]\\displaystyle\\mathbb\{E\}\\mathbb\{E\}\[\\tau^\{\\text\{et\}\}\|\\kappa\]≤\\displaystyle\\leqO​\(𝔼​log⁡αLposδ\+log⁡log⁡H2ϵLposϵLpos\)\\displaystyle O\\left\(\\mathbb\{E\}\\frac\{\\log\\frac\{\\alpha\_\{L^\{\\text\{pos\}\}\}\}\{\\delta\}\+\\log\\log\\frac\{H^\{2\}\}\{\\epsilon\_\{L^\{\\text\{pos\}\}\}\}\}\{\\epsilon\_\{L^\{\\text\{pos\}\}\}\}\\right\)≤\\displaystyle\\leqO​\(𝔼​log⁡ακδ\+log⁡log⁡24​H2ϵκϵκ\)\+O​\(log⁡α⌈log2⁡9600​\(H\+1\)2​C2\(C−1\)2​\(V0∗​\(s0\)−μ0\)2⌉δ\+log⁡log⁡24​H2ϵ⌈log2⁡9600​\(H\+1\)2​C2\(C−1\)2​\(V0∗​\(s0\)−μ0\)2⌉ϵ⌈log2⁡9600​\(H\+1\)2​C2\(C−1\)2​\(V0∗​\(s0\)−μ0\)2⌉\)\\displaystyle O\\left\(\\mathbb\{E\}\\frac\{\\log\\frac\{\\alpha\_\{\\kappa\}\}\{\\delta\}\+\\log\\log\\frac\{24H^\{2\}\}\{\\epsilon\_\{\\kappa\}\}\}\{\\epsilon\_\{\\kappa\}\}\\right\)\+O\\left\(\\frac\{\\log\\frac\{\\alpha\_\{\\lceil\\log\_\{2\}\\frac\{9600\(H\+1\)^\{2\}C^\{2\}\}\{\(C\-1\)^\{2\}\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\rceil\}\}\{\\delta\}\+\\log\\log\\frac\{24H^\{2\}\}\{\\epsilon\_\{\\lceil\\log\_\{2\}\\frac\{9600\(H\+1\)^\{2\}C^\{2\}\}\{\(C\-1\)^\{2\}\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\rceil\}\}\}\{\\epsilon\_\{\\lceil\\log\_\{2\}\\frac\{9600\(H\+1\)^\{2\}C^\{2\}\}\{\(C\-1\)^\{2\}\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\rceil\}\}\\right\)≤\\displaystyle\\leqO​\(log⁡log⁡Hδ\+∑k=1\+∞δk−1​log⁡αkδ\+log⁡log⁡24​H2ϵkϵk\)\+\\displaystyle O\\left\(\\log\\frac\{\\log H\}\{\\delta\}\+\\sum\_\{k=1\}^\{\+\\infty\}\\delta\_\{k\-1\}\\frac\{\\log\\frac\{\\alpha\_\{k\}\}\{\\delta\}\+\\log\\log\\frac\{24H^\{2\}\}\{\\epsilon\_\{k\}\}\}\{\\epsilon\_\{k\}\}\\right\)\+O​\(\(H\+1\)2​C2\(C−1\)2⋅log⁡1δ\+log⁡\(H\+1\)2​C2\(C−1\)2​\(V0∗​\(s0\)−μ0\)2\(V0∗​\(s0\)−μ0\)2\)\\displaystyle O\\left\(\\frac\{\(H\+1\)^\{2\}C^\{2\}\}\{\(C\-1\)^\{2\}\}\\cdot\\frac\{\\log\\frac\{1\}\{\\delta\}\+\\log\\frac\{\(H\+1\)^\{2\}C^\{2\}\}\{\(C\-1\)^\{2\}\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\}\{\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)≤\\displaystyle\\leqO​\(\(H\+1\)2​C2\(C−1\)2⋅log⁡1δ\+log⁡\(H\+1\)2​C2\(C−1\)2​\(V0∗​\(s0\)−μ0\)2\(V0∗−μ0\)2\+log⁡log⁡Hδ\)\.\\displaystyle O\\left\(\\frac\{\(H\+1\)^\{2\}C^\{2\}\}\{\(C\-1\)^\{2\}\}\\cdot\\frac\{\\log\\frac\{1\}\{\\delta\}\+\\log\\frac\{\(H\+1\)^\{2\}C^\{2\}\}\{\(C\-1\)^\{2\}\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\}\{\(V\_\{0\}^\{\*\}\-\\mu\_\{0\}\)^\{2\}\}\+\\log\\frac\{\\log H\}\{\\delta\}\\right\)\.The second last step is by the Lemma[9](https://arxiv.org/html/2605.23182#Thmtheorem9)\. Plug inC=1\.01C=1\.01, we can derive

𝔼​τet≤O​\(H2​log⁡1δ\+H2​log⁡H2\(V0∗​\(s0\)−μ0\)2\(V0∗​\(s0\)−μ0\)2\+log⁡log⁡Hδ\)\\displaystyle\\mathbb\{E\}\\tau^\{\\text\{et\}\}\\leq O\\left\(\\frac\{H^\{2\}\\log\\frac\{1\}\{\\delta\}\+H^\{2\}\\log\\frac\{H^\{2\}\}\{\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\}\{\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\+\\log\\frac\{\\log H\}\{\\delta\}\\right\)\(22\)Summing up \([20](https://arxiv.org/html/2605.23182#A2.E20)\) and \([22](https://arxiv.org/html/2605.23182#A2.E22)\), by the fact that\(V0∗​\(s0\)−μ0\)2≤H2\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\\leq H^\{2\}, we have

𝔼​τ≤O​\(H2​log⁡1δ\+\(H\+1\)4​S​A​log⁡S​A​H\(V0∗​\(s0\)−μ0\)2\+\(H\+1\)4​S2​A​log⁡\(H​S​A\)\(V0∗​\(s0\)−μ0\)2\)\.\\displaystyle\\mathbb\{E\}\\tau\\leq O\\left\(\\frac\{H^\{2\}\\log\\frac\{1\}\{\\delta\}\+\(H\+1\)^\{4\}SA\\log\\frac\{SAH\}\{\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\+\(H\+1\)^\{4\}S^\{2\}A\\log\\left\(HSA\\right\)\}\{\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\.∎

###### Theorem\(Restatement of Theorem[3](https://arxiv.org/html/2605.23182#Thmtheorem3)\)\.

Apply algorithm[1](https://arxiv.org/html/2605.23182#alg1)to a negative instanceν\\nuwithV0∗​\(s0\)<μ0V^\{\*\}\_\{0\}\(s\_\{0\}\)<\\mu\_\{0\}, we have

𝔼​τ\\displaystyle\\mathbb\{E\}\\tau≤\\displaystyle\\leqO​\(\(H\+1\)4​S​A​\(log⁡1δ\+log⁡H2\(V0∗​\(s0\)−μ0\)2\)\(V0∗​\(s0\)−μ0\)2\)\+\\displaystyle O\\left\(\\frac\{\(H\+1\)^\{4\}SA\(\\log\\frac\{1\}\{\\delta\}\+\\log\\frac\{H^\{2\}\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\)\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\+O​\(\(H\+1\)4​S2​A​log⁡\(log⁡\(S​A​H\)\+log⁡1δ\+log⁡H2\(V0∗​\(s0\)−μ0\)2\)\(V0∗​\(s0\)−μ0\)2\)\.\\displaystyle O\\left\(\\frac\{\(H\+1\)^\{4\}S^\{2\}A\\log\\left\(\\log\(SAH\)\+\\log\\frac\{1\}\{\\delta\}\+\\log\\frac\{H^\{2\}\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\.

###### Proof of Theorem[3](https://arxiv.org/html/2605.23182#Thmtheorem3)\.

Denoteτee\\tau^\{\\text\{ee\}\},τet\\tau^\{\\text\{et\}\}as the total collected episodes by the exploration and exploitation oracle\. By the Lemma[7](https://arxiv.org/html/2605.23182#Thmtheorem7), we know Algorithm[1](https://arxiv.org/html/2605.23182#alg1)will terminate no later than the end of phasemax\{⌈log29600​\(H\+1\)2\(V0∗​\(s0\)−μ0\)2⌉,⌈log310δ⌉,κ\}=:Lneg\\max\\\{\\lceil\\log\_\{2\}\\frac\{9600\(H\+1\)^\{2\}\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\rceil,\\lceil\\log\_\{3\}\\frac\{10\}\{\\delta\}\\rceil,\\kappa\\\}=:L^\{\\text\{neg\}\}\. The remaining work is to prove upper bounds forτee\\tau^\{\\text\{ee\}\}andτet\\tau^\{\\text\{et\}\}, at the end of phaseLnegL^\{\\text\{neg\}\}\.

Forτet\\tau^\{\\text\{et\}\}, by the Theorem[20](https://arxiv.org/html/2605.23182#Thmtheorem20), we can conclude at the end of phase indexk≥κk\\geq\\kappa, the Exploration Oracle\(Line[3](https://arxiv.org/html/2605.23182#alg1.l3)in Algorithm[1](https://arxiv.org/html/2605.23182#alg1)\) will always outputNoneorNot Completed\. This observation implies Algorithm[1](https://arxiv.org/html/2605.23182#alg1)will not enter the exploitation period at Line[11](https://arxiv.org/html/2605.23182#alg1.l11)for phase indexk≥κk\\geq\\kappa, suggesting that

τet≤∑k=1κ−1100⋅log⁡αkδ\+log⁡log⁡24​H2ϵkϵk\.\\displaystyle\\tau^\{\\text\{et\}\}\\leq\\sum\_\{k=1\}^\{\\kappa\-1\}100\\cdot\\frac\{\\log\\frac\{\\alpha\_\{k\}\}\{\\delta\}\+\\log\\log\\frac\{24H^\{2\}\}\{\\epsilon\_\{k\}\}\}\{\\epsilon\_\{k\}\}\.\(23\)
Forτee\\tau^\{\\text\{ee\}\}, by the Theorem[20](https://arxiv.org/html/2605.23182#Thmtheorem20), we have

τee≤101​\(H\+1\)4​S​A​log⁡2​S​A​HδLnegϵneg2\+Tκ−1ee\+400​\(H\+1\)4​S2​Aϵneg2​log⁡\(402​\(H\+1\)4​S2​A​log⁡2​S​A​HδLnegϵneg2\)\+400​\(H\+1\)4​S2​Aϵneg2​log⁡\(2​Tκ−1ee\)\.\\begin\{split\}\\tau^\{\\text\{ee\}\}\\leq&\\frac\{101\(H\+1\)^\{4\}SA\\log\\frac\{2SAH\}\{\\delta\_\{L^\{\\text\{neg\}\}\}\}\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\+T\_\{\\kappa\-1\}^\{\\text\{ee\}\}\+\\\\ &\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\\log\\left\(\\frac\{402\(H\+1\)^\{4\}S^\{2\}A\\log\\frac\{2SAH\}\{\\delta\_\{L^\{\\text\{neg\}\}\}\}\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\\right\)\+\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\\log\\left\(2T\_\{\\kappa\-1\}^\{\\text\{ee\}\}\\right\)\.\\end\{split\}\(24\)The remaining is to take expectation on both sides of \([23](https://arxiv.org/html/2605.23182#A2.E23)\), \([24](https://arxiv.org/html/2605.23182#A2.E24)\) and derive upper bounds\. For \([23](https://arxiv.org/html/2605.23182#A2.E23)\), by the Lemma[6](https://arxiv.org/html/2605.23182#Thmtheorem6), we have

𝔼​τet≤\\displaystyle\\mathbb\{E\}\\tau^\{\\text\{et\}\}\\leq𝔼​∑k=1κ−1100⋅log⁡αkδ\+log⁡log⁡24​H2ϵkϵk\\displaystyle\\mathbb\{E\}\\sum\_\{k=1\}^\{\\kappa\-1\}100\\cdot\\frac\{\\log\\frac\{\\alpha\_\{k\}\}\{\\delta\}\+\\log\\log\\frac\{24H^\{2\}\}\{\\epsilon\_\{k\}\}\}\{\\epsilon\_\{k\}\}≤\\displaystyle\\leq𝔼​200⋅log⁡ακ−1δ\+log⁡log⁡24​H2ϵκ−1ϵκ−1\\displaystyle\\mathbb\{E\}200\\cdot\\frac\{\\log\\frac\{\\alpha\_\{\\kappa\-1\}\}\{\\delta\}\+\\log\\log\\frac\{24H^\{2\}\}\{\\epsilon\_\{\\kappa\-1\}\}\}\{\\epsilon\_\{\\kappa\-1\}\}≤\\displaystyle\\leq∑k=1\+∞2003k−1​log⁡αk−1δ\+log⁡log⁡24​H2ϵk−1ϵk−1\\displaystyle\\sum\_\{k=1\}^\{\+\\infty\}\\frac\{200\}\{3^\{k\-1\}\}\\frac\{\\log\\frac\{\\alpha\_\{k\-1\}\}\{\\delta\}\+\\log\\log\\frac\{24H^\{2\}\}\{\\epsilon\_\{k\-1\}\}\}\{\\epsilon\_\{k\-1\}\}≤\\displaystyle\\leqO​\(log⁡log⁡Hδ\)\.\\displaystyle O\\left\(\\log\\frac\{\\log H\}\{\\delta\}\\right\)\.\(25\)The second step is by takingϵk=12k\\epsilon\_\{k\}=\\frac\{1\}\{2^\{k\}\}and\(2⋅log⁡αk\+1δ\+log⁡log⁡24​H2ϵk\+1ϵk\+1\)/\(2⋅log⁡αkδ\+log⁡log⁡24​H2ϵkϵk\)≥2\\Big\(2\\cdot\\frac\{\\log\\frac\{\\alpha\_\{k\+1\}\}\{\\delta\}\+\\log\\log\\frac\{24H^\{2\}\}\{\\epsilon\_\{k\+1\}\}\}\{\\epsilon\_\{k\+1\}\}\\Big\)/\\Big\(2\\cdot\\frac\{\\log\\frac\{\\alpha\_\{k\}\}\{\\delta\}\+\\log\\log\\frac\{24H^\{2\}\}\{\\epsilon\_\{k\}\}\}\{\\epsilon\_\{k\}\}\\Big\)\\geq 2\. The second last step is by the Lemma[10](https://arxiv.org/html/2605.23182#Thmtheorem10)\.

For \([24](https://arxiv.org/html/2605.23182#A2.E24)\), we have

𝔼​τee\\displaystyle\\mathbb\{E\}\\tau^\{\\text\{ee\}\}≤\\displaystyle\\leq𝔼​\(101​\(H\+1\)4​S​A​log⁡\(1δLneg\)ϵneg2\+400​\(H\+1\)4​S2​Aϵneg2​log⁡\(log⁡2​S​A​HδLneg\)\)⏟†4\+\\displaystyle\\underbrace\{\\mathbb\{E\}\\left\(\\frac\{101\(H\+1\)^\{4\}SA\\log\(\\frac\{1\}\{\\delta\_\{L^\{\\text\{neg\}\}\}\}\)\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\+\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\\log\\left\(\\log\\frac\{2SAH\}\{\\delta\_\{L^\{\\text\{neg\}\}\}\}\\right\)\\right\)\}\_\{\\dagger 4\}\+𝔼​\(Tκ−1ee\+400​\(H\+1\)4​S2​Aϵneg2​log⁡\(2​Tκ−1ee\)\)⏟†5\+\\displaystyle\\underbrace\{\\mathbb\{E\}\\left\(T\_\{\\kappa\-1\}^\{\\text\{ee\}\}\+\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\\log\\left\(2T\_\{\\kappa\-1\}^\{\\text\{ee\}\}\\right\)\\right\)\}\_\{\\dagger 5\}\+101​\(H\+1\)4​S​A​log⁡\(2​S​A​H\)ϵneg2\+400​\(H\+1\)4​S2​Aϵneg2​log⁡\(402​\(H\+1\)4​S2​Aϵneg2\)\\displaystyle\\frac\{101\(H\+1\)^\{4\}SA\\log\(2SAH\)\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\+\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\\log\\left\(\\frac\{402\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\\right\)\(26\)For†4\\dagger 4,

𝔼​101​\(H\+1\)4​S​A​log⁡\(1δLneg\)ϵneg2\+400​\(H\+1\)4​S2​Aϵneg2​log⁡\(log⁡2​S​A​HδLneg\)\\displaystyle\\mathbb\{E\}\\frac\{101\(H\+1\)^\{4\}SA\\log\(\\frac\{1\}\{\\delta\_\{L^\{\\text\{neg\}\}\}\}\)\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\+\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\\log\\left\(\\log\\frac\{2SAH\}\{\\delta\_\{L^\{\\text\{neg\}\}\}\}\\right\)≤\\displaystyle\\leq101​\(H\+1\)4​S​A​log⁡\(1δ⌈log2⁡9600​\(H\+1\)2\(V0∗​\(s0\)−μ0\)2⌉\)ϵneg2\+400​\(H\+1\)4​S2​Aϵneg2​log⁡\(log⁡2​S​A​Hδ⌈log2⁡9600​\(H\+1\)2\(V0∗​\(s0\)−μ0\)2⌉\)\+\\displaystyle\\frac\{101\(H\+1\)^\{4\}SA\\log\(\\frac\{1\}\{\\delta\_\{\\lceil\\log\_\{2\}\\frac\{9600\(H\+1\)^\{2\}\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\rceil\}\}\)\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\+\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\\log\\left\(\\log\\frac\{2SAH\}\{\\delta\_\{\\lceil\\log\_\{2\}\\frac\{9600\(H\+1\)^\{2\}\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\rceil\}\}\\right\)\+101​\(H\+1\)4​S​A​log⁡\(1δ⌈log3⁡10δ⌉\)ϵneg2\+400​\(H\+1\)4​S2​Aϵneg2​log⁡\(log⁡2​S​A​Hδ⌈log3⁡10δ⌉\)\+\\displaystyle\\frac\{101\(H\+1\)^\{4\}SA\\log\(\\frac\{1\}\{\\delta\_\{\\lceil\\log\_\{3\}\\frac\{10\}\{\\delta\}\\rceil\}\}\)\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\+\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\\log\\left\(\\log\\frac\{2SAH\}\{\\delta\_\{\\lceil\\log\_\{3\}\\frac\{10\}\{\\delta\}\\rceil\}\}\\right\)\+𝔼​101​\(H\+1\)4​S​A​log⁡\(1δκ\)ϵneg2\+400​\(H\+1\)4​S2​Aϵneg2​log⁡\(log⁡2​S​A​Hδκ\)\.\\displaystyle\\mathbb\{E\}\\frac\{101\(H\+1\)^\{4\}SA\\log\(\\frac\{1\}\{\\delta\_\{\\kappa\}\}\)\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\+\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\\log\\left\(\\log\\frac\{2SAH\}\{\\delta\_\{\\kappa\}\}\\right\)\.\(27\)By Lemma[10](https://arxiv.org/html/2605.23182#Thmtheorem10), we can derive

𝔼​101​\(H\+1\)4​S​A​log⁡\(1δκ\)ϵneg2\+400​\(H\+1\)4​S2​Aϵneg2​log⁡\(log⁡2​S​A​Hδκ\)\\displaystyle\\mathbb\{E\}\\frac\{101\(H\+1\)^\{4\}SA\\log\(\\frac\{1\}\{\\delta\_\{\\kappa\}\}\)\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\+\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\\log\\left\(\\log\\frac\{2SAH\}\{\\delta\_\{\\kappa\}\}\\right\)≤\\displaystyle\\leq∑k=1\+∞δk−1​\(101​\(H\+1\)4​S​A​log⁡\(1δk\)ϵneg2\+400​\(H\+1\)4​S2​Aϵneg2​log⁡\(log⁡2​S​A​Hδk\)\)\\displaystyle\\sum\_\{k=1\}^\{\+\\infty\}\\delta\_\{k\-1\}\\left\(\\frac\{101\(H\+1\)^\{4\}SA\\log\(\\frac\{1\}\{\\delta\_\{k\}\}\)\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\+\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\\log\\left\(\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\\right\)\\right\)≤\\displaystyle\\leqO​\(\(H\+1\)4​S2​A​log⁡\(log⁡\(S​A​H\)\)ϵneg2\)\.\\displaystyle O\\left\(\\frac\{\(H\+1\)^\{4\}S^\{2\}A\\log\\left\(\\log\(SAH\)\\right\)\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\\right\)\.Following \([27](https://arxiv.org/html/2605.23182#A2.E27)\), we can derive

†4\\displaystyle\\dagger 4≤\\displaystyle\\leqO​\(\(H\+1\)4​S​A​log⁡1δ\(V0∗​\(s0\)−μ0\)2\+\(H\+1\)4​S2​A​log⁡\(log⁡\(S​A​H\)\+log⁡1δ\+⌈log⁡9600​\(H\+1\)2\(V0∗​\(s0\)−μ0\)2⌉\)\(V0∗​\(s0\)−μ0\)2\)\+\\displaystyle O\\left\(\\frac\{\(H\+1\)^\{4\}SA\\log\\frac\{1\}\{\\delta\}\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\+\\frac\{\(H\+1\)^\{4\}S^\{2\}A\\log\\left\(\\log\(SAH\)\+\\log\\frac\{1\}\{\\delta\}\+\\lceil\\log\\frac\{9600\(H\+1\)^\{2\}\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\rceil\\right\)\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\+O​\(\(H\+1\)4​S​A​⌈log⁡9600​\(H\+1\)2\(V0∗​\(s0\)−μ0\)2⌉\(V0∗​\(s0\)−μ0\)2\)\\displaystyle O\\left\(\\frac\{\(H\+1\)^\{4\}SA\\lceil\\log\\frac\{9600\(H\+1\)^\{2\}\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\rceil\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\(28\)
For†5\\dagger 5, by Lemma[10](https://arxiv.org/html/2605.23182#Thmtheorem10)

𝔼​\(Tκ−1ee\+400​\(H\+1\)4​S2​Aϵneg2​log⁡\(2​Tκ−1ee\)\)\\displaystyle\\mathbb\{E\}\\left\(T\_\{\\kappa\-1\}^\{\\text\{ee\}\}\+\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\\log\\left\(2T\_\{\\kappa\-1\}^\{\\text\{ee\}\}\\right\)\\right\)≤\\displaystyle\\leq∑k=1\+∞δk−1​\(\(H\+1\)2​S​A​log⁡2​S​A​Hδkϵk\+\(H\+1\)2​S2​Aϵk​log⁡\(\(H\+1\)2​S2​A​log⁡2​S​A​Hδkϵk\)\)\+\\displaystyle\\sum\_\{k=1\}^\{\+\\infty\}\\delta\_\{k\-1\}\\left\(\\frac\{\(H\+1\)^\{2\}SA\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon\_\{k\}\}\}\+\\frac\{\(H\+1\)^\{2\}S^\{2\}A\}\{\\epsilon\_\{k\}\}\\log\\left\(\\frac\{\(H\+1\)^\{2\}S^\{2\}A\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\\epsilon\_\{k\}\}\\right\)\\right\)\+400​\(H\+1\)4​S2​A​log⁡2ϵneg2\+400​\(H\+1\)4​S2​Aϵneg2⋅\\displaystyle\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\\log 2\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\+\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\\cdot∑k=1\+∞δk−1​log⁡\(\(H\+1\)2​S​A​log⁡2​S​A​Hδkϵk\+\(H\+1\)2​S2​Aϵk​log⁡\(\(H\+1\)2​S2​A​log⁡2​S​A​Hδkϵk\)\)\\displaystyle\\sum\_\{k=1\}^\{\+\\infty\}\\delta\_\{k\-1\}\\log\\left\(\\frac\{\(H\+1\)^\{2\}SA\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon\_\{k\}\}\}\+\\frac\{\(H\+1\)^\{2\}S^\{2\}A\}\{\\epsilon\_\{k\}\}\\log\\left\(\\frac\{\(H\+1\)^\{2\}S^\{2\}A\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\\epsilon\_\{k\}\}\\right\)\\right\)≤\\displaystyle\\leqO​\(\(H\+1\)4​S2​A​log⁡\(S​A​H\)\(V0∗​\(s0\)−μ0\)2\+\(H\+1\)2​S2​A​log⁡\(S​A​H\)\)\\displaystyle O\\left\(\\frac\{\(H\+1\)^\{4\}S^\{2\}A\\log\(SAH\)\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\+\(H\+1\)^\{2\}S^\{2\}A\\log\(SAH\)\\right\)≤\\displaystyle\\leqO​\(\(H\+1\)4​S2​A​log⁡\(S​A​H\)\(V0∗​\(s0\)−μ0\)2\)\.\\displaystyle O\\left\(\\frac\{\(H\+1\)^\{4\}S^\{2\}A\\log\(SAH\)\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\.\(29\)The last step is by the fact that\(V0∗​\(s0\)−μ0\)2≤H2\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\\leq H^\{2\}\.

Substitute†4\\dagger 4,†5\\dagger 5in \([26](https://arxiv.org/html/2605.23182#A2.E26)\) with \([28](https://arxiv.org/html/2605.23182#A2.E28)\) and \([29](https://arxiv.org/html/2605.23182#A2.E29)\), we prove

𝔼​τee≤\\displaystyle\\mathbb\{E\}\\tau^\{\\text\{ee\}\}\\leqO​\(\(H\+1\)4​S​A​\(log⁡1δ\+log⁡H2\(V0∗​\(s0\)−μ0\)2\)\(V0∗​\(s0\)−μ0\)2\)\+\\displaystyle O\\left\(\\frac\{\(H\+1\)^\{4\}SA\(\\log\\frac\{1\}\{\\delta\}\+\\log\\frac\{H^\{2\}\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\)\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\+O​\(\(H\+1\)4​S2​A​log⁡\(log⁡\(S​A​H\)\+log⁡1δ\+log⁡H2\(V0∗​\(s0\)−μ0\)2\)\(V0∗​\(s0\)−μ0\)2\)\.\\displaystyle O\\left\(\\frac\{\(H\+1\)^\{4\}S^\{2\}A\\log\\left\(\\log\(SAH\)\+\\log\\frac\{1\}\{\\delta\}\+\\log\\frac\{H^\{2\}\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\.\(30\)Combine \([25](https://arxiv.org/html/2605.23182#A2.E25)\) and \([30](https://arxiv.org/html/2605.23182#A2.E30)\), we have

𝔼​τ≤\\displaystyle\\mathbb\{E\}\\tau\\leqO​\(\(H\+1\)4​S​A​\(log⁡1δ\+log⁡H2\(V0∗​\(s0\)−μ0\)2\)\(V0∗​\(s0\)−μ0\)2\)\+\\displaystyle O\\left\(\\frac\{\(H\+1\)^\{4\}SA\(\\log\\frac\{1\}\{\\delta\}\+\\log\\frac\{H^\{2\}\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\)\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\+O​\(\(H\+1\)4​S2​A​log⁡\(log⁡\(S​A​H\)\+log⁡1δ\+log⁡H2\(V0∗​\(s0\)−μ0\)2\)\(V0∗​\(s0\)−μ0\)2\)\.\\displaystyle O\\left\(\\frac\{\(H\+1\)^\{4\}S^\{2\}A\\log\\left\(\\log\(SAH\)\+\\log\\frac\{1\}\{\\delta\}\+\\log\\frac\{H^\{2\}\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\.∎

### B\.2Auxiliary Lemmas

In this section, we provide the proof of the required Lemmas in section[B\.1](https://arxiv.org/html/2605.23182#A2.SS1)\. The first one is about the maximum episodes collected in exploration and exploitation periods, up to the end of phasekk\.

###### Lemma 6\.

Each time Algorithm[1](https://arxiv.org/html/2605.23182#alg1)enters Line[20](https://arxiv.org/html/2605.23182#alg1.l20), i\.e\. the end of phasekk, we have\|ℋee\|≤Tkee,\|ℋet\|≤∑ℓ=1k100⋅log⁡αℓδ\+log⁡log⁡24​H2ϵℓϵℓ\|\\mathcal\{H\}^\{\\text\{ee\}\}\|\\leq T\_\{k\}^\{\\text\{ee\}\},\|\\mathcal\{H\}^\{\\text\{et\}\}\|\\leq\\sum\_\{\\ell=1\}^\{k\}100\\cdot\\frac\{\\log\\frac\{\\alpha\_\{\\ell\}\}\{\\delta\}\+\\log\\log\\frac\{24H^\{2\}\}\{\\epsilon\_\{\\ell\}\}\}\{\\epsilon\_\{\\ell\}\}hold with certainty\.

###### Proof of Lemma[6](https://arxiv.org/html/2605.23182#Thmtheorem6)\.

\|ℋee\|≤Tkee\|\\mathcal\{H\}^\{\\text\{ee\}\}\|\\leq T\_\{k\}^\{\\text\{ee\}\}is a direct conclusion from the conditiont≤Tkee−1t\\leq T\_\{k\}^\{\\text\{ee\}\}\-1at the Line[3](https://arxiv.org/html/2605.23182#alg2.l3)of Algorithm[2](https://arxiv.org/html/2605.23182#alg2)\.

Regarding\|ℋet\|\|\\mathcal\{H\}^\{\\text\{et\}\}\|, its upper bound is from the conditionN≤100⋅log⁡αkδ\+log⁡log⁡24​H2ϵkϵkN\\leq 100\\cdot\\frac\{\\log\\frac\{\\alpha\_\{k\}\}\{\\delta\}\+\\log\\log\\frac\{24H^\{2\}\}\{\\epsilon\_\{k\}\}\}\{\\epsilon\_\{k\}\}at the Line[12](https://arxiv.org/html/2605.23182#alg1.l12)of Algorithm[1](https://arxiv.org/html/2605.23182#alg1)\. Since exploitation periodkkallows at most collecting100⋅log⁡αkδ\+log⁡log⁡24​H2ϵkϵk100\\cdot\\frac\{\\log\\frac\{\\alpha\_\{k\}\}\{\\delta\}\+\\log\\log\\frac\{24H^\{2\}\}\{\\epsilon\_\{k\}\}\}\{\\epsilon\_\{k\}\}episodes,\|ℋet\|≤∑ℓ=1k100⋅log⁡αℓδ\+log⁡log⁡24​H2ϵℓϵℓ\|\\mathcal\{H\}^\{\\text\{et\}\}\|\\leq\\sum\_\{\\ell=1\}^\{k\}100\\cdot\\frac\{\\log\\frac\{\\alpha\_\{\\ell\}\}\{\\delta\}\+\\log\\log\\frac\{24H^\{2\}\}\{\\epsilon\_\{\\ell\}\}\}\{\\epsilon\_\{\\ell\}\}must hold\. ∎

The following is a basis Lemma, asserting the Algorithm must terminate no later than the end of specific phases\.

###### Lemma 7\.

Recall the definitionℰket=\{∀N\>0,\|V^etπ^k,N−V0π^k\|≤4​H2​log⁡2​αk​\(log2⁡2​N\)2δN\}\\mathcal\{E\}\_\{k\}^\{\\text\{et\}\}=\\left\\\{\\forall N\>0,\\left\|\\hat\{V\}\_\{\\text\{et\}\}^\{\\hat\{\\pi\}\_\{k\},N\}\-V^\{\\hat\{\\pi\}\_\{k\}\}\_\{0\}\\right\|\\leq\\sqrt\{\\frac\{4H^\{2\}\\log\\frac\{2\\alpha\_\{k\}\(\\log\_\{2\}2N\)^\{2\}\}\{\\delta\}\}\{N\}\}\\right\\\}\.

Apply Algorithm[1](https://arxiv.org/html/2605.23182#alg1)to an instanceν\\nu

- •If the instanceν\\nuis positive, denoteLpos=max⁡\{κ,⌈log2⁡9600​\(H\+1\)2​C2\(C−1\)2​\(V0∗​\(s0\)−μ0\)2⌉\}L^\{\\text\{pos\}\}=\\max\\\{\\kappa,\\lceil\\log\_\{2\}\\frac\{9600\(H\+1\)^\{2\}C^\{2\}\}\{\(C\-1\)^\{2\}\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\rceil\\\}, and define κend=min⁡\{k:k≥Lpos,ℰket​holds\}\.\\displaystyle\\kappa\_\{\\text\{end\}\}=\\min\\left\\\{k:k\\geq L^\{\\text\{pos\}\},\\mathcal\{E\}\_\{k\}^\{\\text\{et\}\}\\text\{ holds\}\\right\\\}\.We assert the Algorithm[1](https://arxiv.org/html/2605.23182#alg1)must terminate no later than the end of phaseκend\\kappa\_\{\\text\{end\}\}\.
- •If the instanceν\\nuis negative, denoteLneg:=max⁡\{⌈log2⁡9600​\(H\+1\)2\(V0∗​\(s0\)−μ0\)2⌉,⌈log3⁡10δ⌉,κ\}L^\{\\text\{neg\}\}:=\\max\\\{\\lceil\\log\_\{2\}\\frac\{9600\(H\+1\)^\{2\}\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\rceil,\\lceil\\log\_\{3\}\\frac\{10\}\{\\delta\}\\rceil,\\kappa\\\}\.We assert the Algorithm[1](https://arxiv.org/html/2605.23182#alg1)must terminate no later than the end of phaseLnegL^\{\\text\{neg\}\}\.

###### Proof of Lemma[7](https://arxiv.org/html/2605.23182#Thmtheorem7)\.

We split the proof into two parts by discussingν\\nuis is either positive or negative\.

Ifν\\nuis positive, by the Theorem[20](https://arxiv.org/html/2605.23182#Thmtheorem20), we can conclude at the end of phase indexk≥max⁡\{⌈log2⁡9600​\(H\+1\)2​C2\(V0∗​\(s0\)−μ0\)2⌉,κ\}k\\geq\\max\\\{\\lceil\\log\_\{2\}\\frac\{9600\(H\+1\)^\{2\}C^\{2\}\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\rceil,\\kappa\\\}, the Exploration Oracle\(Line[3](https://arxiv.org/html/2605.23182#alg1.l3)in Algorithm[1](https://arxiv.org/html/2605.23182#alg1)\) will always outputπ^k\\hat\{\\pi\}\_\{k\}such thatV0π^k​\(s0\)−μ0≥C−1C​\(V0∗​\(s0\)−μ0\)V^\{\\hat\{\\pi\}\_\{k\}\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\\geq\\frac\{C\-1\}\{C\}\\Big\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\\Big\)\.

By the Lemma[22](https://arxiv.org/html/2605.23182#Thmtheorem22)and[24](https://arxiv.org/html/2605.23182#Thmtheorem24), for any phase indexk≥⌈log2⁡H2​C2\(C−1\)2​\(V0∗​\(s0\)−μ0\)2⌉k\\geq\\lceil\\log\_\{2\}\\frac\{H^\{2\}C^\{2\}\}\{\(C\-1\)^\{2\}\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\rceil, if the Exploration Oracle returns aπ^k\\hat\{\\pi\}\_\{k\}satisfying thatV0π^k​\(s0\)−μ0≥C−1C​\(V0∗−μ0\)V\_\{0\}^\{\\hat\{\\pi\}\_\{k\}\}\(s\_\{0\}\)\-\\mu\_\{0\}\\geq\\frac\{C\-1\}\{C\}\(V\_\{0\}^\{\*\}\-\\mu\_\{0\}\)and the concentration eventℰket\\mathcal\{E\}\_\{k\}^\{\\text\{et\}\}holds, Algorithm[1](https://arxiv.org/html/2605.23182#alg1)will enter Line[15](https://arxiv.org/html/2605.23182#alg1.l15)and takeπ^k\\hat\{\\pi\}\_\{k\}as the final output\. Recall the definitionκend=min⁡\{k:k≥Lpos,ℰket​holds\}\\kappa\_\{\\text\{end\}\}=\\min\\left\\\{k:k\\geq L^\{\\text\{pos\}\},\\mathcal\{E\}\_\{k\}^\{\\text\{et\}\}\\text\{ holds\}\\right\\\}, we know Algorithm[1](https://arxiv.org/html/2605.23182#alg1)will terminate at the end of phaseκend\\kappa\_\{\\text\{end\}\}\.

Ifν\\nuis negative, by the Theorem[20](https://arxiv.org/html/2605.23182#Thmtheorem20), we can conclude at the end of phase indexk≥max⁡\{⌈log2⁡9600​\(H\+1\)2\(V0∗​\(s0\)−μ0\)2⌉,κ\}k\\geq\\max\\\{\\lceil\\log\_\{2\}\\frac\{9600\(H\+1\)^\{2\}\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\rceil,\\kappa\\\}, the Exploration Oracle\(Line[3](https://arxiv.org/html/2605.23182#alg1.l3)in Algorithm[1](https://arxiv.org/html/2605.23182#alg1)\) will always outputNone\. Because of the condition of Line[8](https://arxiv.org/html/2605.23182#alg1.l8)in Algorithm[1](https://arxiv.org/html/2605.23182#alg1), we can assert the algorithm will terminate no later than the end of phasemax\{⌈log29600​\(H\+1\)2\(V0∗​\(s0\)−μ0\)2⌉,⌈log310δ⌉,κ\}=:Lneg\\max\\\{\\lceil\\log\_\{2\}\\frac\{9600\(H\+1\)^\{2\}\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\rceil,\\lceil\\log\_\{3\}\\frac\{10\}\{\\delta\}\\rceil,\\kappa\\\}=:L^\{\\text\{neg\}\}\. ∎

The following Lemma is a direct conclusion from Lemma[7](https://arxiv.org/html/2605.23182#Thmtheorem7), which guarantee Algorithm[1](https://arxiv.org/html/2605.23182#alg1)must terminate almost surely\.

###### Lemma 8\.

Apply Algorithm[1](https://arxiv.org/html/2605.23182#alg1)to an instanceν\\nu\. IfV0∗​\(s0\)≠μ0V^\{\*\}\_\{0\}\(s\_\{0\}\)\\neq\\mu\_\{0\}, which meansν\\nuis either positive or negative, we havePr⁡\(τ<\+∞\)=1\\Pr\(\\tau<\+\\infty\)=1\.

###### Proof of Lemma[8](https://arxiv.org/html/2605.23182#Thmtheorem8)\.

Similar to the Lemma[7](https://arxiv.org/html/2605.23182#Thmtheorem7), we split the proof into two parts by discussingν\\nuis either positive or negative\.

Ifν\\nuis positive, by the Lemma[7](https://arxiv.org/html/2605.23182#Thmtheorem7), we have

Pr⁡\(τ=\+∞\)≤\\displaystyle\\Pr\(\\tau=\+\\infty\)\\leqPr⁡\(κend=\+∞\)\\displaystyle\\Pr\(\\kappa\_\{\\text\{end\}\}=\+\\infty\)≤\\displaystyle\\leqPr⁡\(κ=\+∞\)\+Pr⁡\(κend−Lpos=\+∞\|κ<\+∞\),\\displaystyle\\Pr\(\\kappa=\+\\infty\)\+\\Pr\(\\kappa\_\{\\text\{end\}\}\-L^\{\\text\{pos\}\}=\+\\infty\|\\kappa<\+\\infty\),whereLpos=max⁡\{κ,⌈log2⁡9600​\(H\+1\)2​C2\(C−1\)2​\(V0∗​\(s0\)−μ0\)2⌉\}L^\{\\text\{pos\}\}=\\max\\\{\\kappa,\\lceil\\log\_\{2\}\\frac\{9600\(H\+1\)^\{2\}C^\{2\}\}\{\(C\-1\)^\{2\}\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\rceil\\\}\. By the Lemma[10](https://arxiv.org/html/2605.23182#Thmtheorem10), we havePr⁡\(κ=\+∞\)=0\\Pr\(\\kappa=\+\\infty\)=0\. In addition, Lemma[23](https://arxiv.org/html/2605.23182#Thmtheorem23)suggests that

Pr⁡\(κend≥Lpos\+k\|κ\)\\displaystyle\\Pr\(\\kappa\_\{\\text\{end\}\}\\geq L^\{\\text\{pos\}\}\+k\|\\kappa\)≤\\displaystyle\\leqPr⁡\(\(¬ℰLposet\)∩⋯∩\(¬ℰLpos\+k−1et\)\|κ\)\\displaystyle\\Pr\\big\(\(\\neg\\mathcal\{E\}\_\{L^\{\\text\{pos\}\}\}^\{\\text\{et\}\}\)\\cap\\cdots\\cap\(\\neg\\mathcal\{E\}\_\{L^\{\\text\{pos\}\}\+k\-1\}^\{\\text\{et\}\}\)\|\\kappa\\big\)≤\\displaystyle\\leqPr⁡\(¬ℰLpos\+k−1et\|κ\)\\displaystyle\\Pr\(\\neg\\mathcal\{E\}\_\{L^\{\\text\{pos\}\}\+k\-1\}^\{\\text\{et\}\}\|\\kappa\)≤\\displaystyle\\leqπ2​δ6​αLpos\+k−1\\displaystyle\\frac\{\\pi^\{2\}\\delta\}\{6\\alpha\_\{L^\{\\text\{pos\}\}\+k\-1\}\}\(31\)≤\\displaystyle\\leqπ2​δ6​αk−1\\displaystyle\\frac\{\\pi^\{2\}\\delta\}\{6\\alpha\_\{k\-1\}\}\(32\)By the \([32](https://arxiv.org/html/2605.23182#A2.E32)\), we havePr⁡\(κend−Lpos=\+∞\|κ<\+∞\)=0\\Pr\(\\kappa\_\{\\text\{end\}\}\-L^\{\\text\{pos\}\}=\+\\infty\|\\kappa<\+\\infty\)=0\. Thus, we havePr⁡\(τ=\+∞\)=0\\Pr\(\\tau=\+\\infty\)=0, which meansPr⁡\(τ<\+∞\)=1\\Pr\(\\tau<\+\\infty\)=1\.

Ifν\\nuis negative, by the Lemma[7](https://arxiv.org/html/2605.23182#Thmtheorem7)and[9](https://arxiv.org/html/2605.23182#Thmtheorem9), we have

Pr⁡\(τ=\+∞\)≤Pr⁡\(κ=\+∞\)=0,\\displaystyle\\Pr\(\\tau=\+\\infty\)\\leq\\Pr\(\\kappa=\+\\infty\)=0,which suggestsPr⁡\(τ<\+∞\)=1\\Pr\(\\tau<\+\\infty\)=1\. ∎

### B\.3Performance Guarantee of Algorithm[2](https://arxiv.org/html/2605.23182#alg2)

This section mainly follows the existing analysis ofAl Marjani and Proutiere \[[2021](https://arxiv.org/html/2605.23182#bib.bib6)\], which also provides required Lemmas for Appendix[B\.1](https://arxiv.org/html/2605.23182#A2.SS1)\. For self\-completeness, we still provide detailed proof\. In the following proof, we follow the convention that takeing∑n=n1n2an=0\\sum\_\{n=n\_\{1\}\}^\{n\_\{2\}\}a\_\{n\}=0for non\-negative\{an\}n=1\+∞\\\{a\_\{n\}\\\}\_\{n=1\}^\{\+\\infty\}andn1\>n2≥0;n1,n2∈ℤn\_\{1\}\>n\_\{2\}\\geq 0;n\_\{1\},n\_\{2\}\\in\\mathbb\{Z\}\.

By the same argument as the Lemma 10 inKaufmannet al\.\[[2021](https://arxiv.org/html/2605.23182#bib.bib9)\], we can conclude

###### Lemma 9\.

Pr⁡\(ℰ​\(δ\)∩ℰcnt​\(δ\)\)≥1−δ\\Pr\\big\(\\mathcal\{E\}\(\\delta\)\\cap\\mathcal\{E\}^\{\\text\{cnt\}\}\(\\delta\)\\big\)\\geq 1\-\\deltaholds for allδ∈\(0,1\)\\delta\\in\(0,1\), whereℰcnt​\(δ\)\\mathcal\{E\}^\{\\text\{cnt\}\}\(\\delta\),ℰ​\(δ\)\\mathcal\{E\}\(\\delta\)are defined in \([7](https://arxiv.org/html/2605.23182#A2.E7)\), \([8](https://arxiv.org/html/2605.23182#A2.E8)\)\.

###### Proof of Lemma[9](https://arxiv.org/html/2605.23182#Thmtheorem9)\.

Lemma 10 ofKaufmannet al\.\[[2021](https://arxiv.org/html/2605.23182#bib.bib9)\]asserts

Pr⁡\(ℰ​\(δ\)\)≥\\displaystyle\\Pr\\big\(\\mathcal\{E\}\(\\delta\)\\big\)\\geq1−12​δ\\displaystyle 1\-\\frac\{1\}\{2\}\\deltaPr⁡\(ℰcnt​\(δ\)\)≥\\displaystyle\\Pr\\big\(\\mathcal\{E\}^\{\\text\{cnt\}\}\(\\delta\)\\big\)\\geq1−12​δ\.\\displaystyle 1\-\\frac\{1\}\{2\}\\delta\.∎

Given Lemma[9](https://arxiv.org/html/2605.23182#Thmtheorem9), the following Lemma is obvious\.

###### Lemma 10\.

Pr⁡\(κ≥k\)≤δk−1\\Pr\(\\kappa\\geq k\)\\leq\\delta\_\{k\-1\}\.

###### Proof of Lemma[10](https://arxiv.org/html/2605.23182#Thmtheorem10)\.

Recall the definition ofκ:=min⁡\{k:ℰ​\(δk\)∩ℰcnt​\(δk\)​hold\}\\kappa:=\\min\\\{k:\\mathcal\{E\}\(\\delta\_\{k\}\)\\cap\\mathcal\{E\}^\{\\text\{cnt\}\}\(\\delta\_\{k\}\)\\text\{ hold\}\\\}\. Evident to seePr⁡\(κ≥k\)≤Pr⁡\(ℰ​\(δk−1\)∩ℰcnt​\(δk−1\)​don’t hold\)\\Pr\(\\kappa\\geq k\)\\leq\\Pr\(\\mathcal\{E\}\(\\delta\_\{k\-1\}\)\\cap\\mathcal\{E\}^\{\\text\{cnt\}\}\(\\delta\_\{k\-1\}\)\\text\{ don't hold\}\)\. Then Lemma[9](https://arxiv.org/html/2605.23182#Thmtheorem9)implies Lemma[10](https://arxiv.org/html/2605.23182#Thmtheorem10)\. ∎

The following lemmas shows the optimistic and pessimistic estimation defined in Appendix[A](https://arxiv.org/html/2605.23182#A1)indeed contains the value function or Q function\.

###### Lemma 11\.

For any roundttduring phasek≥κk\\geq\\kappa, and any policyπ\\pi, we have

Q¯ht,π​\(s,a;δk\)≤Qhπ​\(s,a\)\\displaystyle\\underline\{Q\}\_\{h\}^\{t,\\pi\}\(s,a;\\delta\_\{k\}\)\\leq Q\_\{h\}^\{\\pi\}\(s,a\)≤Q¯ht,π​\(s,a;δk\)\\displaystyle\\leq\\overline\{Q\}\_\{h\}^\{t,\\pi\}\(s,a;\\delta\_\{k\}\)Q¯ht​\(s,a;δk\)≤Qh∗​\(s,a\)\\displaystyle\\underline\{Q\}\_\{h\}^\{t\}\(s,a;\\delta\_\{k\}\)\\leq Q\_\{h\}^\{\*\}\(s,a\)≤Q¯ht​\(s,a;δk\)\.\\displaystyle\\leq\\overline\{Q\}\_\{h\}^\{t\}\(s,a;\\delta\_\{k\}\)\.

###### Proof of Lemma[11](https://arxiv.org/html/2605.23182#Thmtheorem11)\.

We prove Lemma[11](https://arxiv.org/html/2605.23182#Thmtheorem11)by Induction\. Evident to see

Q¯H\+1t,π​\(s,a;δ\)=QH\+1π​\(s,a\)=Q¯H\+1t,π​\(s,a;δ\)=0\\displaystyle\\underline\{Q\}\_\{H\+1\}^\{t,\\pi\}\(s,a;\\delta\)=Q\_\{H\+1\}^\{\\pi\}\(s,a\)=\\overline\{Q\}\_\{H\+1\}^\{t,\\pi\}\(s,a;\\delta\)=0Q¯H\+1t​\(s,a;δ\)=QH\+1∗​\(s,a\)=Q¯H\+1t​\(s,a;δ\)=0\.\\displaystyle\\underline\{Q\}\_\{H\+1\}^\{t\}\(s,a;\\delta\)=Q\_\{H\+1\}^\{\*\}\(s,a\)=\\overline\{Q\}\_\{H\+1\}^\{t\}\(s,a;\\delta\)=0\.Now, assume the conclusion holds forh\+1,h\+2,⋯,H\+1h\+1,h\+2,\\cdots,H\+1, we turn to work on the casehh\. To proveQ¯ht,π​\(s,a;δ\)≤Qhπ​\(s,a\)\\underline\{Q\}\_\{h\}^\{t,\\pi\}\(s,a;\\delta\)\\leq Q\_\{h\}^\{\\pi\}\(s,a\), we can derive

Q¯ht,π​\(s,a;δ\)=\\displaystyle\\underline\{Q\}\_\{h\}^\{t,\\pi\}\(s,a;\\delta\)=rh​\(s,a\)\+minp¯h∈𝒞ht​\(s,a;δ\)​∑s′p¯h​\(s′\|s,a\)​V¯h\+1t,π​\(s′;δ\)\\displaystyle r\_\{h\}\(s,a\)\+\\min\_\{\\underline\{p\}\_\{h\}\\in\\mathcal\{C\}\_\{h\}^\{t\}\(s,a;\\delta\)\}\\sum\_\{s^\{\\prime\}\}\\underline\{p\}\_\{h\}\(s^\{\\prime\}\|s,a\)\\underline\{V\}\_\{h\+1\}^\{t,\\pi\}\(s^\{\\prime\};\\delta\)≤\\displaystyle\\leqrh​\(s,a\)\+∑s′ph​\(s′\|s,a\)​V¯h\+1t,π​\(s′;δ\)\\displaystyle r\_\{h\}\(s,a\)\+\\sum\_\{s^\{\\prime\}\}p\_\{h\}\(s^\{\\prime\}\|s,a\)\\underline\{V\}\_\{h\+1\}^\{t,\\pi\}\(s^\{\\prime\};\\delta\)=\\displaystyle=rh​\(s,a\)\+∑s′ph​\(s′\|s,a\)​Q¯h\+1t,π​\(s′,π​\(s′\);δ\)\\displaystyle r\_\{h\}\(s,a\)\+\\sum\_\{s^\{\\prime\}\}p\_\{h\}\(s^\{\\prime\}\|s,a\)\\underline\{Q\}\_\{h\+1\}^\{t,\\pi\}\(s^\{\\prime\},\\pi\(s^\{\\prime\}\);\\delta\)≤Induction\\displaystyle\\stackrel\{\{\\scriptstyle\\text\{Induction\}\}\}\{\{\\leq\}\}rh​\(s,a\)\+∑s′ph​\(s′\|s,a\)​Qh\+1π​\(s′,π​\(s′\)\)\\displaystyle r\_\{h\}\(s,a\)\+\\sum\_\{s^\{\\prime\}\}p\_\{h\}\(s^\{\\prime\}\|s,a\)Q\_\{h\+1\}^\{\\pi\}\(s^\{\\prime\},\\pi\(s^\{\\prime\}\)\)=\\displaystyle=Qhπ​\(s,a\)\.\\displaystyle Q\_\{h\}^\{\\pi\}\(s,a\)\.To proveQ¯ht,π​\(s,a;δ\)≥Qhπ​\(s,a\)\\overline\{Q\}\_\{h\}^\{t,\\pi\}\(s,a;\\delta\)\\geq Q\_\{h\}^\{\\pi\}\(s,a\), we can derive

Q¯ht,π​\(s,a;δ\)=\\displaystyle\\overline\{Q\}\_\{h\}^\{t,\\pi\}\(s,a;\\delta\)=rh​\(s,a\)\+maxp¯h∈𝒞ht​\(s,a;δ\)​∑s′p¯h​\(s′\|s,a\)​V¯h\+1t,π​\(s′;δ\)\\displaystyle r\_\{h\}\(s,a\)\+\\max\_\{\\bar\{p\}\_\{h\}\\in\\mathcal\{C\}\_\{h\}^\{t\}\(s,a;\\delta\)\}\\sum\_\{s^\{\\prime\}\}\\bar\{p\}\_\{h\}\(s^\{\\prime\}\|s,a\)\\overline\{V\}\_\{h\+1\}^\{t,\\pi\}\(s^\{\\prime\};\\delta\)≥\\displaystyle\\geqrh​\(s,a\)\+∑s′ph​\(s′\|s,a\)​V¯h\+1t,π​\(s′;δ\)\\displaystyle r\_\{h\}\(s,a\)\+\\sum\_\{s^\{\\prime\}\}p\_\{h\}\(s^\{\\prime\}\|s,a\)\\overline\{V\}\_\{h\+1\}^\{t,\\pi\}\(s^\{\\prime\};\\delta\)=\\displaystyle=rh​\(s,a\)\+∑s′ph​\(s′\|s,a\)​Q¯h\+1t,π​\(s′,π​\(s′\);δ\)\\displaystyle r\_\{h\}\(s,a\)\+\\sum\_\{s^\{\\prime\}\}p\_\{h\}\(s^\{\\prime\}\|s,a\)\\overline\{Q\}\_\{h\+1\}^\{t,\\pi\}\(s^\{\\prime\},\\pi\(s^\{\\prime\}\);\\delta\)≥Induction\\displaystyle\\stackrel\{\{\\scriptstyle\\text\{Induction\}\}\}\{\{\\geq\}\}rh​\(s,a\)\+∑s′ph​\(s′\|s,a\)​Qh\+1π​\(s′,π​\(s′\)\)\\displaystyle r\_\{h\}\(s,a\)\+\\sum\_\{s^\{\\prime\}\}p\_\{h\}\(s^\{\\prime\}\|s,a\)Q\_\{h\+1\}^\{\\pi\}\(s^\{\\prime\},\\pi\(s^\{\\prime\}\)\)=\\displaystyle=Qhπ​\(s,a\)\.\\displaystyle Q\_\{h\}^\{\\pi\}\(s,a\)\.To proveQ¯ht​\(s,a;δ\)≤Qh∗​\(s,a\)\\underline\{Q\}\_\{h\}^\{t\}\(s,a;\\delta\)\\leq Q^\{\*\}\_\{h\}\(s,a\), we can derive

Q¯ht​\(s,a;δ\)=\\displaystyle\\underline\{Q\}\_\{h\}^\{t\}\(s,a;\\delta\)=rh​\(s,a\)\+minp¯h∈𝒞ht​\(s,a;δ\)​∑s′p¯h​\(s′\|s,a\)​V¯h\+1t​\(s′;δ\)\\displaystyle r\_\{h\}\(s,a\)\+\\min\_\{\\underline\{p\}\_\{h\}\\in\\mathcal\{C\}\_\{h\}^\{t\}\(s,a;\\delta\)\}\\sum\_\{s^\{\\prime\}\}\\underline\{p\}\_\{h\}\(s^\{\\prime\}\|s,a\)\\underline\{V\}\_\{h\+1\}^\{t\}\(s^\{\\prime\};\\delta\)≤\\displaystyle\\leqrh​\(s,a\)\+∑s′ph​\(s′\|s,a\)​V¯h\+1t​\(s′;δ\)\\displaystyle r\_\{h\}\(s,a\)\+\\sum\_\{s^\{\\prime\}\}p\_\{h\}\(s^\{\\prime\}\|s,a\)\\underline\{V\}\_\{h\+1\}^\{t\}\(s^\{\\prime\};\\delta\)=\\displaystyle=rh​\(s,a\)\+∑s′ph​\(s′\|s,a\)​maxa⁡Q¯h\+1t​\(s,a;δ\)\\displaystyle r\_\{h\}\(s,a\)\+\\sum\_\{s^\{\\prime\}\}p\_\{h\}\(s^\{\\prime\}\|s,a\)\\max\_\{a\}\\underline\{Q\}\_\{h\+1\}^\{t\}\(s,a;\\delta\)≤Induction\\displaystyle\\stackrel\{\{\\scriptstyle\\text\{Induction\}\}\}\{\{\\leq\}\}rh​\(s,a\)\+∑s′ph​\(s′\|s,a\)​maxa⁡Qh\+1∗​\(s,a\)\\displaystyle r\_\{h\}\(s,a\)\+\\sum\_\{s^\{\\prime\}\}p\_\{h\}\(s^\{\\prime\}\|s,a\)\\max\_\{a\}Q\_\{h\+1\}^\{\*\}\(s,a\)=\\displaystyle=rh​\(s,a\)\+∑s′ph​\(s′\|s,a\)​Vh\+1∗​\(s\)\\displaystyle r\_\{h\}\(s,a\)\+\\sum\_\{s^\{\\prime\}\}p\_\{h\}\(s^\{\\prime\}\|s,a\)V\_\{h\+1\}^\{\*\}\(s\)=\\displaystyle=Qh∗​\(s,a\)\.\\displaystyle Q^\{\*\}\_\{h\}\(s,a\)\.To proveQ¯ht​\(s,a;δ\)≥Qh∗​\(s,a\)\\overline\{Q\}\_\{h\}^\{t\}\(s,a;\\delta\)\\geq Q^\{\*\}\_\{h\}\(s,a\), we can derive

Q¯ht​\(s,a;δ\)=\\displaystyle\\overline\{Q\}\_\{h\}^\{t\}\(s,a;\\delta\)=rh​\(s,a\)\+maxp¯h∈𝒞ht​\(s,a;δ\)​∑s′p¯h​\(s′\|s,a\)​V¯h\+1t​\(s′;δ\)\\displaystyle r\_\{h\}\(s,a\)\+\\max\_\{\\bar\{p\}\_\{h\}\\in\\mathcal\{C\}\_\{h\}^\{t\}\(s,a;\\delta\)\}\\sum\_\{s^\{\\prime\}\}\\bar\{p\}\_\{h\}\(s^\{\\prime\}\|s,a\)\\overline\{V\}\_\{h\+1\}^\{t\}\(s^\{\\prime\};\\delta\)≥\\displaystyle\\geqrh​\(s,a\)\+∑s′ph​\(s′\|s,a\)​V¯h\+1t​\(s′;δ\)\\displaystyle r\_\{h\}\(s,a\)\+\\sum\_\{s^\{\\prime\}\}p\_\{h\}\(s^\{\\prime\}\|s,a\)\\overline\{V\}\_\{h\+1\}^\{t\}\(s^\{\\prime\};\\delta\)=\\displaystyle=rh​\(s,a\)\+∑s′ph​\(s′\|s,a\)​maxa⁡Q¯h\+1t​\(s′,a;δ\)\\displaystyle r\_\{h\}\(s,a\)\+\\sum\_\{s^\{\\prime\}\}p\_\{h\}\(s^\{\\prime\}\|s,a\)\\max\_\{a\}\\overline\{Q\}\_\{h\+1\}^\{t\}\(s^\{\\prime\},a;\\delta\)≥Induction\\displaystyle\\stackrel\{\{\\scriptstyle\\text\{Induction\}\}\}\{\{\\geq\}\}rh​\(s,a\)\+∑s′ph​\(s′\|s,a\)​maxa⁡Qh\+1​\(s′,a\)\\displaystyle r\_\{h\}\(s,a\)\+\\sum\_\{s^\{\\prime\}\}p\_\{h\}\(s^\{\\prime\}\|s,a\)\\max\_\{a\}Q\_\{h\+1\}\(s^\{\\prime\},a\)=\\displaystyle=rh​\(s,a\)\+∑s′ph​\(s′\|s,a\)​Vh\+1∗​\(s\)\\displaystyle r\_\{h\}\(s,a\)\+\\sum\_\{s^\{\\prime\}\}p\_\{h\}\(s^\{\\prime\}\|s,a\)V\_\{h\+1\}^\{\*\}\(s\)=\\displaystyle=Qh∗​\(s,a\)\.\\displaystyle Q^\{\*\}\_\{h\}\(s,a\)\.Then, we complete the induction\. ∎

Given Lemma[11](https://arxiv.org/html/2605.23182#Thmtheorem11), the following proposition is evident\.

###### Proposition 12\.

For any roundttduring phasek≥κk\\geq\\kappa, we have

maxa⁡Q¯ht​\(s,a;δk\)≤maxa⁡Qh∗​\(s,a\)≤maxa⁡Q¯ht​\(s,a;δk\),∀s,h\\displaystyle\\max\_\{a\}\\underline\{Q\}\_\{h\}^\{t\}\(s,a;\\delta\_\{k\}\)\\leq\\max\_\{a\}Q\_\{h\}^\{\*\}\(s,a\)\\leq\\max\_\{a\}\\overline\{Q\}\_\{h\}^\{t\}\(s,a;\\delta\_\{k\}\),\\forall s,hwhich is equivalent toV¯ht​\(s;δ\)≤Vh∗​\(s\)≤V¯ht​\(s;δ\)\\underline\{V\}\_\{h\}^\{t\}\(s;\\delta\)\\leq V^\{\*\}\_\{h\}\(s\)\\leq\\overline\{V\}\_\{h\}^\{t\}\(s;\\delta\),∀s,h\\forall s,h\. Further, for any policyπ\\pi,

Q¯ht,π​\(s,πh​\(s\);δk\)≤Qhπ​\(s,πh​\(s\)\)\\displaystyle\\underline\{Q\}\_\{h\}^\{t,\\pi\}\(s,\\pi\_\{h\}\(s\);\\delta\_\{k\}\)\\leq Q\_\{h\}^\{\\pi\}\(s,\\pi\_\{h\}\(s\)\)≤Q¯ht,π​\(s,πh​\(s\);δk\),∀s,h\\displaystyle\\leq\\overline\{Q\}\_\{h\}^\{t,\\pi\}\(s,\\pi\_\{h\}\(s\);\\delta\_\{k\}\),\\forall s,hwhich is equivalent toV¯ht,π​\(s;δ\)≤Vhπ​\(s\)≤V¯ht,π​\(s;δ\)\\underline\{V\}\_\{h\}^\{t,\\pi\}\(s;\\delta\)\\leq V^\{\\pi\}\_\{h\}\(s\)\\leq\\overline\{V\}\_\{h\}^\{t,\\pi\}\(s;\\delta\),∀s,h\\forall s,h\.

Following the Lemma 3 inKaufmannet al\.\[[2021](https://arxiv.org/html/2605.23182#bib.bib9)\], we have

###### Lemma 13\.

For any roundttduring phasek≥κk\\geq\\kappain Algorithm[1](https://arxiv.org/html/2605.23182#alg1), meaning thatℰ​\(δk\)∩ℰcnt​\(δk\)\\mathcal\{E\}\(\\delta\_\{k\}\)\\cap\\mathcal\{E\}^\{\\text\{cnt\}\}\(\\delta\_\{k\}\)holds, we have

- •if the instance is positive, the outputa^\\hat\{a\}from Algorithm[2](https://arxiv.org/html/2605.23182#alg2)is either policyπ¯t​\(⋅;δk\)\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\), satisfying that V0π¯t​\(s0\)−μ0≥C−1C​\(V0∗​\(s0\)−μ0\)\\displaystyle V^\{\\overline\{\\pi\}^\{t\}\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\\geq\\frac\{C\-1\}\{C\}\\Big\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\\Big\)or “Not Complete”
- •If the instance is negative, the outputa^\\hat\{a\}from Algorithm[2](https://arxiv.org/html/2605.23182#alg2)is either "Negative" orNot Completed\.

###### Proof of Lemma[13](https://arxiv.org/html/2605.23182#Thmtheorem13)\.

For simpliciy, we denoteV¯t:=V¯0t​\(s0;δk\)\\overline\{V\}^\{t\}:=\\overline\{V\}^\{t\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\),V¯t,π¯t:=V¯0t,π¯t​\(⋅;δk\)​\(s0;δk\)\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\}:=\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\),V∗:=V0∗​\(s0\)V^\{\*\}:=V^\{\*\}\_\{0\}\(s\_\{0\}\),Vπ¯t:=V0π¯t​\(⋅;δk\)​\(s0\)V^\{\\overline\{\\pi\}^\{t\}\}:=V^\{\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\_\{0\}\(s\_\{0\}\)in the following proof\.

If the instance is positive, Proposition[12](https://arxiv.org/html/2605.23182#Thmtheorem12)suggests thatV¯t≥V∗\>μ0\\overline\{V\}^\{t\}\\geq V^\{\*\}\>\\mu\_\{0\}, suggesting that Algorithm[2](https://arxiv.org/html/2605.23182#alg2)can only leave the loop at the Line[3](https://arxiv.org/html/2605.23182#alg2.l3)when \([3](https://arxiv.org/html/2605.23182#S3.E3)\) holds ort≥Tkeet\\geq T\_\{k\}^\{\\text\{ee\}\}\. Ift≥Tkeet\\geq T\_\{k\}^\{\\text\{ee\}\}holds,a^=Not Completed\\hat\{a\}=\\textsf\{Not Completed\}\. If \([3](https://arxiv.org/html/2605.23182#S3.E3)\) holds,a^\\hat\{a\}must beπ¯t\\overline\{\\pi\}^\{t\}, notice that

μ0<V¯t,π¯t−\(C−1\)​\(V¯t−V¯t,π¯t\)\\displaystyle\\mu\_\{0\}<\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\}\-\(C\-1\)\(\\overline\{V\}^\{t\}\-\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\}\)⇔\\displaystyle\\Leftrightarrowμ0<V¯t,π¯t\+\(V¯t−V¯t,π¯t\)−C​\(V¯t−V¯t,π¯t\)\\displaystyle\\mu\_\{0\}<\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\}\+\(\\overline\{V\}^\{t\}\-\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\}\)\-C\(\\overline\{V\}^\{t\}\-\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\}\)⇔\\displaystyle\\LeftrightarrowC​\(V¯t−V¯t,π¯t\)<V¯t−μ0\.\\displaystyle C\(\\overline\{V\}^\{t\}\-\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\}\)<\\overline\{V\}^\{t\}\-\\mu\_\{0\}\.\(33\)Meanwhile, Lemma[11](https://arxiv.org/html/2605.23182#Thmtheorem11)and Proposition[12](https://arxiv.org/html/2605.23182#Thmtheorem12)implyVπ¯t,V∗∈\[V¯t,π¯t,V¯t\]V^\{\\overline\{\\pi\}^\{t\}\},V^\{\*\}\\in\[\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\},\\overline\{V\}^\{t\}\], suggesting

\(V¯t−μ0\)≤\(V¯t−V¯t,π¯t\)\+\(Vπ¯t−μ0\)\\displaystyle\(\\overline\{V\}^\{t\}\-\\mu\_\{0\}\)\\leq\(\\overline\{V\}^\{t\}\-\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\}\)\+\(V^\{\\overline\{\\pi\}^\{t\}\}\-\\mu\_\{0\}\)\(34\)⇒\\displaystyle\\Rightarrow\(V¯t−μ0\)−\(Vπ¯t−μ0\)≤\(V¯t−V¯t,π¯t\)\.\\displaystyle\(\\overline\{V\}^\{t\}\-\\mu\_\{0\}\)\-\(V^\{\\overline\{\\pi\}^\{t\}\}\-\\mu\_\{0\}\)\\leq\(\\overline\{V\}^\{t\}\-\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\}\)\.\(35\)ReplaceV¯t−V¯t\\overline\{V\}^\{t\}\-\\underline\{V\}^\{t\}in \([33](https://arxiv.org/html/2605.23182#A2.E33)\) with \([35](https://arxiv.org/html/2605.23182#A2.E35)\), we can conclude

C​\(V¯t−V¯t\)<V¯t−μ0\\displaystyle C\(\\overline\{V\}^\{t\}\-\\underline\{V\}^\{t\}\)<\\overline\{V\}^\{t\}\-\\mu\_\{0\}⇒\\displaystyle\\RightarrowC​\(\(V¯t−μ0\)−\(Vπ¯t−μ0\)\)<V¯t−μ0\\displaystyle C\\Big\(\(\\overline\{V\}^\{t\}\-\\mu\_\{0\}\)\-\(V^\{\\overline\{\\pi\}^\{t\}\}\-\\mu\_\{0\}\)\\Big\)<\\overline\{V\}^\{t\}\-\\mu\_\{0\}⇔\\displaystyle\\Leftrightarrow\(C−1\)​\(V¯t−μ0\)<C​\(Vπ¯t−μ0\)\\displaystyle\(C\-1\)\(\\overline\{V\}^\{t\}\-\\mu\_\{0\}\)<C\(V^\{\\overline\{\\pi\}^\{t\}\}\-\\mu\_\{0\}\)⇒\\displaystyle\\Rightarrow\(C−1\)​\(V∗−μ0\)<C​\(Vπ¯t−μ0\)\\displaystyle\(C\-1\)\(V^\{\*\}\-\\mu\_\{0\}\)<C\(V^\{\\overline\{\\pi\}^\{t\}\}\-\\mu\_\{0\}\)⇔\\displaystyle\\LeftrightarrowC−1C​\(V∗−μ0\)<Vπ¯t−μ0\.\\displaystyle\\frac\{C\-1\}\{C\}\(V^\{\*\}\-\\mu\_\{0\}\)<V^\{\\overline\{\\pi\}^\{t\}\}\-\\mu\_\{0\}\.The last line must complete the proof for the case that the instance is positive\.

If the instance is negative, Proposition[12](https://arxiv.org/html/2605.23182#Thmtheorem12)suggests thatV¯t≤V∗<μ0\\underline\{V\}^\{t\}\\leq V^\{\*\}<\\mu\_\{0\}, suggesting that Algorithm[2](https://arxiv.org/html/2605.23182#alg2)can only leave the loop at the Line[3](https://arxiv.org/html/2605.23182#alg2.l3)when \([4](https://arxiv.org/html/2605.23182#S3.E4)\) holds ort≥Tkeet\\geq T\_\{k\}^\{\\text\{ee\}\}\. Ift≥Tkeet\\geq T\_\{k\}^\{\\text\{ee\}\},a^=Not Completed\\hat\{a\}=\\textsf\{Not Completed\}\. If \([4](https://arxiv.org/html/2605.23182#S3.E4)\) holds,a^=None\\hat\{a\}=\\textsf\{None\}, which completes the proof for negative case\. ∎

The following Lemma builds the suggests a necessary condition for Algorithm[2](https://arxiv.org/html/2605.23182#alg2)to continue sampling\.

###### Lemma 14\.

For any roundttduring phasek≥κk\\geq\\kappa, meaning thatℰ​\(δk\)∩ℰcnt​\(δk\)\\mathcal\{E\}\(\\delta\_\{k\}\)\\cap\\mathcal\{E\}^\{\\text\{cnt\}\}\(\\delta\_\{k\}\)holds, if Algorithm[2](https://arxiv.org/html/2605.23182#alg2)doesn’t terminate after samplingtt\-th episode,

- •if the instance is positive, we have V0∗​\(s0\)−μ0C≤V¯0t​\(s0;δk\)−V¯0t,π¯t​\(⋅;δk\)​\(s0;δk\)\\displaystyle\\frac\{V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\}\{C\}\\leq\\overline\{V\}^\{t\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)\-\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)
- •if the instance is negative, we have μ0−V0∗​\(s0\)≤V¯0t​\(s0;δk\)−V¯0t,π¯t​\(⋅;δk\)​\(s0;δk\)\\displaystyle\\mu\_\{0\}\-V^\{\*\}\_\{0\}\(s\_\{0\}\)\\leq\\overline\{V\}^\{t\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)\-\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)

###### Proof of Lemma[14](https://arxiv.org/html/2605.23182#Thmtheorem14)\.

If the instance is positive, Algorithm[2](https://arxiv.org/html/2605.23182#alg2)doesn’t terminate after samplingtt\-th episode means

V¯0t,π¯t​\(⋅;δk\)​\(s0;δk\)−\(C−1\)​\(V¯0t​\(s0;δk\)−V¯0t,π¯t​\(⋅;δk\)​\(s0;δk\)\)≤μ0\\displaystyle\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)\-\(C\-1\)\(\\overline\{V\}^\{t\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)\-\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)\)\\leq\\mu\_\{0\}⇔\\displaystyle\\LeftrightarrowC​\(V¯0t​\(s0;δk\)−V¯0t,π¯t​\(⋅;δk\)​\(s0;δk\)\)≥V¯0t​\(s0;δk\)−μ0\\displaystyle C\(\\overline\{V\}^\{t\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)\-\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)\)\\geq\\overline\{V\}^\{t\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)\-\\mu\_\{0\}⇒\\displaystyle\\RightarrowC​\(V¯0t​\(s0;δk\)−V¯0t,π¯t​\(⋅;δk\)​\(s0;δk\)\)≥V0∗​\(s0\)−μ0\\displaystyle C\(\\overline\{V\}^\{t\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)\-\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)\)\\geq V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}⇔\\displaystyle\\LeftrightarrowV¯0t​\(s0;δk\)−V¯0t,π¯t​\(⋅;δk\)​\(s0;δk\)≥V0∗​\(s0\)−μ0C\.\\displaystyle\\overline\{V\}^\{t\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)\-\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)\\geq\\frac\{V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\}\{C\}\.The second last line is byV¯0t​\(s0;δk\)≥V0∗​\(s0\)\\overline\{V\}^\{t\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)\\geq V^\{\*\}\_\{0\}\(s\_\{0\}\), guarantee by theℰ​\(δk\)∩ℰcnt​\(δk\)\\mathcal\{E\}\(\\delta\_\{k\}\)\\cap\\mathcal\{E\}^\{\\text\{cnt\}\}\(\\delta\_\{k\}\), Lemma[11](https://arxiv.org/html/2605.23182#Thmtheorem11)and Proposition[12](https://arxiv.org/html/2605.23182#Thmtheorem12)\.

If the instance is negative, Algorithm[2](https://arxiv.org/html/2605.23182#alg2)doesn’t terminate after samplingtt\-th episode meansV¯0t​\(s0;δk\)≥μ0\\overline\{V\}^\{t\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)\\geq\\mu\_\{0\}, which is equivalent to

V¯0t​\(s0;δk\)≥μ0\\displaystyle\\overline\{V\}^\{t\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)\\geq\\mu\_\{0\}⇔\\displaystyle\\LeftrightarrowV¯0t​\(s0;δk\)−V¯0t,π¯t​\(⋅;δk\)​\(s0;δk\)≥μ0−V¯0t,π¯t​\(⋅;δk\)​\(s0;δk\)\\displaystyle\\overline\{V\}^\{t\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)\-\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)\\geq\\mu\_\{0\}\-\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)⇒\\displaystyle\\RightarrowV¯0t​\(s0;δk\)−V¯0t,π¯t​\(⋅;δk\)​\(s0;δk\)≥μ0−V0∗​\(s0\)\.\\displaystyle\\overline\{V\}^\{t\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)\-\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)\\geq\\mu\_\{0\}\-V^\{\*\}\_\{0\}\(s\_\{0\}\)\.The last line is byV0∗​\(s0\)≥V0π¯t​\(⋅;δk\)​\(s0\)≥V¯0t,π¯t​\(⋅;δk\)​\(s0;δk\)V^\{\*\}\_\{0\}\(s\_\{0\}\)\\geq V^\{\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\_\{0\}\(s\_\{0\}\)\\geq\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\), guarantee by theℰ​\(δk\)∩ℰcnt​\(δk\)\\mathcal\{E\}\(\\delta\_\{k\}\)\\cap\\mathcal\{E\}^\{\\text\{cnt\}\}\(\\delta\_\{k\}\), Lemma[11](https://arxiv.org/html/2605.23182#Thmtheorem11)and Proposition[12](https://arxiv.org/html/2605.23182#Thmtheorem12)\. ∎

To utilize Lemma[14](https://arxiv.org/html/2605.23182#Thmtheorem14)to prove an implicit upper bound for the length of a phsek≥κk\\geq\\kappa, we introduce the following Lemma[15](https://arxiv.org/html/2605.23182#Thmtheorem15)\. Lemma[15](https://arxiv.org/html/2605.23182#Thmtheorem15)can be considered an intermediate step for the proof of implicit upper bound in Lemma[18](https://arxiv.org/html/2605.23182#Thmtheorem18)\.

###### Lemma 15\.

For any roundttduring phasek≥κk\\geq\\kappa,∀s,a,h\\forall s,a,h, we have

Q¯ht​\(s,a;δk\)−Q¯ht,π¯t​\(⋅;δk\)​\(s,a;δk\)\\displaystyle\\overline\{Q\}\_\{h\}^\{t\}\(s,a;\\delta\_\{k\}\)\-\\underline\{Q\}\_\{h\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s,a;\\delta\_\{k\}\)≤\\displaystyle\\leq4​\(H−h\)​min⁡\{βp​\(nht​\(s,a\),δk\)nht​\(s,a\),1\}\+∑s′ph​\(s′\|s,a\)​\(V¯h\+1t​\(s′;δk\)−V¯h\+1t,π¯t​\(⋅;δk\)​\(s′;δk\)\)\.\\displaystyle 4\(H\-h\)\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(n\_\{h\}^\{t\}\(s,a\),\\delta\_\{k\}\)\}\{n\_\{h\}^\{t\}\(s,a\)\}\},1\\right\\\}\+\\sum\_\{s^\{\\prime\}\}p\_\{h\}\(s^\{\\prime\}\|s,a\)\\big\(\\overline\{V\}^\{t\}\_\{h\+1\}\(s^\{\\prime\};\\delta\_\{k\}\)\-\\underline\{V\}\_\{h\+1\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s^\{\\prime\};\\delta\_\{k\}\)\\big\)\.

###### Proof of Lema[15](https://arxiv.org/html/2605.23182#Thmtheorem15)\.

By the definition ofQ¯ht​\(s,a\)\\overline\{Q\}\_\{h\}^\{t\}\(s,a\),Q¯ht,π¯t​\(⋅;δk\)​\(s,a\)\\underline\{Q\}\_\{h\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s,a\), we have

Q¯ht​\(s,a\)−Q¯ht,π¯t​\(⋅;δk\)​\(s,a\)\\displaystyle\\overline\{Q\}\_\{h\}^\{t\}\(s,a\)\-\\underline\{Q\}\_\{h\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s,a\)=\\displaystyle=∑s′p¯ht​\(s′\|s,a\)​V¯h\+1t​\(s′;δk\)−∑s′p¯ht,π¯t​\(s′\|s,a\)​V¯h\+1t,π¯t​\(⋅;δk\)​\(s′;δk\)\\displaystyle\\sum\_\{s^\{\\prime\}\}\\overline\{p\}\_\{h\}^\{t\}\(s^\{\\prime\}\|s,a\)\\overline\{V\}\_\{h\+1\}^\{t\}\(s^\{\\prime\};\\delta\_\{k\}\)\-\\sum\_\{s^\{\\prime\}\}\\underline\{p\}\_\{h\}^\{t,\\overline\{\\pi\}^\{t\}\}\(s^\{\\prime\}\|s,a\)\\underline\{V\}\_\{h\+1\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s^\{\\prime\};\\delta\_\{k\}\)=\\displaystyle=∑s′p¯ht​\(s′\|s,a\)​V¯h\+1t​\(s′;δk\)−∑s′ph​\(s′\|s,a\)​V¯h\+1t​\(s′;δk\)\+\\displaystyle\\sum\_\{s^\{\\prime\}\}\\overline\{p\}\_\{h\}^\{t\}\(s^\{\\prime\}\|s,a\)\\overline\{V\}\_\{h\+1\}^\{t\}\(s^\{\\prime\};\\delta\_\{k\}\)\-\\sum\_\{s^\{\\prime\}\}p\_\{h\}\(s^\{\\prime\}\|s,a\)\\overline\{V\}\_\{h\+1\}^\{t\}\(s^\{\\prime\};\\delta\_\{k\}\)\+∑s′ph​\(s′\|s,a\)​V¯h\+1t​\(s′;δk\)−∑s′ph​\(s′\|s,a\)​V¯h\+1t,π¯t​\(⋅;δk\)​\(s′;δk\)\+\\displaystyle\\sum\_\{s^\{\\prime\}\}p\_\{h\}\(s^\{\\prime\}\|s,a\)\\overline\{V\}\_\{h\+1\}^\{t\}\(s^\{\\prime\};\\delta\_\{k\}\)\-\\sum\_\{s^\{\\prime\}\}p\_\{h\}\(s^\{\\prime\}\|s,a\)\\underline\{V\}\_\{h\+1\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s^\{\\prime\};\\delta\_\{k\}\)\+∑s′ph​\(s′\|s,a\)​V¯h\+1t,π¯t​\(⋅;δk\)​\(s′;δk\)−∑s′p¯ht,π¯t​\(s′;δk\)​\(s′\|s,a\)​V¯h\+1t,π¯t​\(⋅;δk\)​\(s′;δk\)\\displaystyle\\sum\_\{s^\{\\prime\}\}p\_\{h\}\(s^\{\\prime\}\|s,a\)\\underline\{V\}\_\{h\+1\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s^\{\\prime\};\\delta\_\{k\}\)\-\\sum\_\{s^\{\\prime\}\}\\underline\{p\}\_\{h\}^\{t,\\overline\{\\pi\}^\{t\}\(s^\{\\prime\};\\delta\_\{k\}\)\}\(s^\{\\prime\}\|s,a\)\\underline\{V\}\_\{h\+1\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s^\{\\prime\};\\delta\_\{k\}\)≤\\displaystyle\\leq∥p¯ht\(⋅\|s,a\)−ph\(⋅\|s,a\)∥1\(H−h\)\+∑s′ph\(s′\|s,a\)\(V¯h\+1t\(s′;δk\)−V¯h\+1t,π¯t​\(⋅;δk\)\(s′;δk\)\)\+\\displaystyle\\\|\\overline\{p\}\_\{h\}^\{t\}\(\\cdot\|s,a\)\-p\_\{h\}\(\\cdot\|s,a\)\\\|\_\{1\}\(H\-h\)\+\\sum\_\{s^\{\\prime\}\}p\_\{h\}\(s^\{\\prime\}\|s,a\)\\big\(\\overline\{V\}\_\{h\+1\}^\{t\}\(s^\{\\prime\};\\delta\_\{k\}\)\-\\underline\{V\}\_\{h\+1\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s^\{\\prime\};\\delta\_\{k\}\)\\big\)\+∥ph\(⋅\|s,a\)−p¯ht,π¯t​\(s′;δk\)\(⋅\|s,a\)∥1\(H−h\)\\displaystyle\\\|p\_\{h\}\(\\cdot\|s,a\)\-\\underline\{p\}\_\{h\}^\{t,\\overline\{\\pi\}^\{t\}\(s^\{\\prime\};\\delta\_\{k\}\)\}\(\\cdot\|s,a\)\\\|\_\{1\}\(H\-h\)\(36\)≤\\displaystyle\\leq4​\(H−h\)​min⁡\{βp​\(nht​\(s,a\),δk\)nht​\(s,a\),1\}\+∑s′ph​\(s′\|s,a\)​\(V¯h\+1t​\(s′;δk\)−V¯h\+1t,π¯t​\(⋅;δk\)​\(s′;δk\)\)\\displaystyle 4\(H\-h\)\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(n\_\{h\}^\{t\}\(s,a\),\\delta\_\{k\}\)\}\{n\_\{h\}^\{t\}\(s,a\)\}\},1\\right\\\}\+\\sum\_\{s^\{\\prime\}\}p\_\{h\}\(s^\{\\prime\}\|s,a\)\\big\(\\overline\{V\}\_\{h\+1\}^\{t\}\(s^\{\\prime\};\\delta\_\{k\}\)\-\\underline\{V\}\_\{h\+1\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s^\{\\prime\};\\delta\_\{k\}\)\\big\)\(37\)Step \([36](https://arxiv.org/html/2605.23182#A2.E36)\) is by the fact thatV¯h\+1t​\(s′;δk\),V¯h\+1t,π¯t​\(⋅;δk\)​\(s′;δk\)≤H\\overline\{V\}\_\{h\+1\}^\{t\}\(s^\{\\prime\};\\delta\_\{k\}\),\\underline\{V\}\_\{h\+1\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s^\{\\prime\};\\delta\_\{k\}\)\\leq H\. Step \([37](https://arxiv.org/html/2605.23182#A2.E37)\) is by the fact that‖p​\(⋅\)−q​\(⋅\)‖1≤2\\\|p\(\\cdot\)\-q\(\\cdot\)\\\|\_\{1\}\\leq 2, and the Pinsker’s Inequality‖p​\(⋅\)−q​\(⋅\)‖1≤12​KL​\(p,q\)\\\|p\(\\cdot\)\-q\(\\cdot\)\\\|\_\{1\}\\leq\\sqrt\{\\frac\{1\}\{2\}\\text\{KL\}\(p,q\)\}, ask≥κk\\geq\\kappasuggests thatp¯ht\(⋅\|s,a\),p¯ht,π¯t\(⋅\|s,a\)∈𝒞ht\(s,a;δk\)\\overline\{p\}\_\{h\}^\{t\}\(\\cdot\|s,a\),\\underline\{p\}\_\{h\}^\{t,\\overline\{\\pi\}^\{t\}\}\(\\cdot\|s,a\)\\in\\mathcal\{C\}\_\{h\}^\{t\}\(s,a;\\delta\_\{k\}\)\. ∎

Given Lemma[15](https://arxiv.org/html/2605.23182#Thmtheorem15), we can immediately derive the following proposition by takinga=π¯ht​\(s;δk\)a=\\overline\{\\pi\}^\{t\}\_\{h\}\(s;\\delta\_\{k\}\)\.

###### Proposition 16\.

For any roundttduring phasek≥κk\\geq\\kappa,∀s,a,h\\forall s,a,h, we have

V¯ht​\(s;δk\)−V¯ht,π¯t​\(⋅;δk\)​\(s;δk\)\\displaystyle\\overline\{V\}\_\{h\}^\{t\}\(s;\\delta\_\{k\}\)\-\\underline\{V\}\_\{h\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s;\\delta\_\{k\}\)≤\\displaystyle\\leq4​\(H−h\)​min⁡\{βp​\(nht​\(s,π¯ht​\(s\)\),δk\)nht​\(s,π¯ht​\(s\)\),1\}\+\\displaystyle 4\(H\-h\)\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(n\_\{h\}^\{t\}\(s,\\overline\{\\pi\}^\{t\}\_\{h\}\(s\)\),\\delta\_\{k\}\)\}\{n\_\{h\}^\{t\}\(s,\\overline\{\\pi\}^\{t\}\_\{h\}\(s\)\)\}\},1\\right\\\}\+∑s′ph​\(s′\|s,π¯ht​\(s;δk\)\)​\(V¯h\+1t​\(s′;δk\)−V¯h\+1t,π¯t​\(⋅;δk\)​\(s′;δk\)\)\.\\displaystyle\\sum\_\{s^\{\\prime\}\}p\_\{h\}\(s^\{\\prime\}\|s,\\overline\{\\pi\}^\{t\}\_\{h\}\(s;\\delta\_\{k\}\)\)\\big\(\\overline\{V\}^\{t\}\_\{h\+1\}\(s^\{\\prime\};\\delta\_\{k\}\)\-\\underline\{V\}\_\{h\+1\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s^\{\\prime\};\\delta\_\{k\}\)\\big\)\.

By repeatedly apply Proposition[16](https://arxiv.org/html/2605.23182#Thmtheorem16), we can prove an upper bound for theV¯0t​\(s0;δk\)−V¯0t,π¯t​\(⋅;δk\)​\(s0;δk\)\\overline\{V\}^\{t\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)\-\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\), which is the following Lemma\.

###### Lemma 17\.

For any roundttduring phasek≥κk\\geq\\kappa, meaning thatℰ​\(δ\)∩ℰcnt​\(δ\)\\mathcal\{E\}\(\\delta\)\\cap\\mathcal\{E\}^\{\\text\{cnt\}\}\(\\delta\)holds, we have

V¯0t​\(s0;δk\)−V¯0t,π¯t​\(⋅;δk\)​\(s0;δk\)\\displaystyle\\overline\{V\}^\{t\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)\-\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)≤\\displaystyle\\leq4​H​min⁡\{βp​\(t,δk\)t,1\}\+4​∑h=1H\(H−h\)​∑s′,a′phπ¯t​\(s′,a′\)​min⁡\{βp​\(nht​\(s′,a′\),δk\)nht​\(s′,a′\),1\}\\displaystyle 4H\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(t,\\delta\_\{k\}\)\}\{t\}\},1\\right\\\}\+4\\sum\_\{h=1\}^\{H\}\(H\-h\)\\sum\_\{s^\{\\prime\},a^\{\\prime\}\}p\_\{h\}^\{\\overline\{\\pi\}^\{t\}\}\(s^\{\\prime\},a^\{\\prime\}\)\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(n\_\{h\}^\{t\}\(s^\{\\prime\},a^\{\\prime\}\),\\delta\_\{k\}\)\}\{n\_\{h\}^\{t\}\(s^\{\\prime\},a^\{\\prime\}\)\}\},1\\right\\\}

###### Proof of Lemma[17](https://arxiv.org/html/2605.23182#Thmtheorem17)\.

Givenπ¯t​\(⋅;δk\)\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)and the definition ofphπ¯t​\(⋅;δk\)​\(s′,a′\):=Pr⁡\(St,h=s,π¯ht​\(s;δk\)=a\)p\_\{h\}^\{\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s^\{\\prime\},a^\{\\prime\}\):=\\Pr\(S\_\{t,h\}=s,\\overline\{\\pi\}^\{t\}\_\{h\}\(s;\\delta\_\{k\}\)=a\)\.

We use induction to prove

V¯0t​\(s0;δk\)−V¯0t,π¯t​\(⋅;δk\)​\(s0;δk\)\\displaystyle\\overline\{V\}^\{t\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)\-\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)≤\\displaystyle\\leq4​H​min⁡\{βp​\(t,δk\)t,1\}\+4​∑h=1h′−1\(H−h\)​∑s′,a′phπ¯t​\(s′,a′\)​min⁡\{βp​\(nht​\(s′,a′\),δk\)nht​\(s′,a′\),1\}\+\\displaystyle 4H\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(t,\\delta\_\{k\}\)\}\{t\}\},1\\right\\\}\+4\\sum\_\{h=1\}^\{h^\{\\prime\}\-1\}\(H\-h\)\\sum\_\{s^\{\\prime\},a^\{\\prime\}\}p\_\{h\}^\{\\overline\{\\pi\}^\{t\}\}\(s^\{\\prime\},a^\{\\prime\}\)\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(n\_\{h\}^\{t\}\(s^\{\\prime\},a^\{\\prime\}\),\\delta\_\{k\}\)\}\{n\_\{h\}^\{t\}\(s^\{\\prime\},a^\{\\prime\}\)\}\},1\\right\\\}\+∑s′,a′ph′π¯t​\(s′,a′\)​\(V¯h′t​\(s′;δk\)−V¯h′t,π¯t​\(⋅;δk\)​\(s′;δk\)\)\\displaystyle\\sum\_\{s^\{\\prime\},a^\{\\prime\}\}p\_\{h^\{\\prime\}\}^\{\\overline\{\\pi\}^\{t\}\}\(s^\{\\prime\},a^\{\\prime\}\)\\big\(\\overline\{V\}^\{t\}\_\{h^\{\\prime\}\}\(s^\{\\prime\};\\delta\_\{k\}\)\-\\underline\{V\}\_\{h^\{\\prime\}\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s^\{\\prime\};\\delta\_\{k\}\)\\big\)\(38\)holds for allh′=1,⋯,H\+1h^\{\\prime\}=1,\\cdots,H\+1\.

Forh′=1h^\{\\prime\}=1, by the Proposition[16](https://arxiv.org/html/2605.23182#Thmtheorem16), we know

V¯0t​\(s0;δk\)−V¯0t,π¯t​\(⋅;δk\)​\(s0;δk\)\\displaystyle\\overline\{V\}\_\{0\}^\{t\}\(s\_\{0\};\\delta\_\{k\}\)\-\\underline\{V\}\_\{0\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s\_\{0\};\\delta\_\{k\}\)≤\\displaystyle\\leq4​H​min⁡\{βp​\(nht​\(s0,a0\),δk\)nht​\(s0,a0\),1\}\+∑s′p0​\(s′\)​\(V¯1t​\(s′;δk\)−V¯1t,π¯t​\(⋅;δk\)​\(s′;δk\)\)\\displaystyle 4H\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(n\_\{h\}^\{t\}\(s\_\{0\},a\_\{0\}\),\\delta\_\{k\}\)\}\{n\_\{h\}^\{t\}\(s\_\{0\},a\_\{0\}\)\}\},1\\right\\\}\+\\sum\_\{s^\{\\prime\}\}p\_\{0\}\(s^\{\\prime\}\)\\big\(\\overline\{V\}^\{t\}\_\{1\}\(s^\{\\prime\};\\delta\_\{k\}\)\-\\underline\{V\}\_\{1\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s^\{\\prime\};\\delta\_\{k\}\)\\big\)=\\displaystyle=4​H​min⁡\{βp​\(t,δk\)t,1\}\+∑s′p0​\(s′\)​\(V¯1t​\(s′;δk\)−V¯1t,π¯t​\(⋅;δk\)​\(s′;δk\)\)\\displaystyle 4H\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(t,\\delta\_\{k\}\)\}\{t\}\},1\\right\\\}\+\\sum\_\{s^\{\\prime\}\}p\_\{0\}\(s^\{\\prime\}\)\\big\(\\overline\{V\}^\{t\}\_\{1\}\(s^\{\\prime\};\\delta\_\{k\}\)\-\\underline\{V\}\_\{1\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s^\{\\prime\};\\delta\_\{k\}\)\\big\)=\\displaystyle=4​H​min⁡\{βp​\(t,δk\)t,1\}\+∑s′,a′p0​\(s′\)​𝟙​\(a′=π¯1t​\(s′\)\)​\(V¯1t​\(s′;δk\)−V¯1t,π¯t​\(⋅;δk\)​\(s′;δk\)\)\\displaystyle 4H\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(t,\\delta\_\{k\}\)\}\{t\}\},1\\right\\\}\+\\sum\_\{s^\{\\prime\},a^\{\\prime\}\}p\_\{0\}\(s^\{\\prime\}\)\\mathds\{1\}\(a^\{\\prime\}=\\overline\{\\pi\}^\{t\}\_\{1\}\(s^\{\\prime\}\)\)\\big\(\\overline\{V\}^\{t\}\_\{1\}\(s^\{\\prime\};\\delta\_\{k\}\)\-\\underline\{V\}\_\{1\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s^\{\\prime\};\\delta\_\{k\}\)\\big\)=\\displaystyle=4​H​min⁡\{βp​\(t,δk\)t,1\}\+∑s′,a′p1π¯t​\(s′,a′\)​\(V¯1t​\(s′;δk\)−V¯1t,π¯t​\(⋅;δk\)​\(s′;δk\)\)\.\\displaystyle 4H\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(t,\\delta\_\{k\}\)\}\{t\}\},1\\right\\\}\+\\sum\_\{s^\{\\prime\},a^\{\\prime\}\}p\_\{1\}^\{\\overline\{\\pi\}^\{t\}\}\(s^\{\\prime\},a^\{\\prime\}\)\\big\(\\overline\{V\}^\{t\}\_\{1\}\(s^\{\\prime\};\\delta\_\{k\}\)\-\\underline\{V\}\_\{1\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s^\{\\prime\};\\delta\_\{k\}\)\\big\)\.The third line is bynht​\(s0,a0\)=tn\_\{h\}^\{t\}\(s\_\{0\},a\_\{0\}\)=tholds for alltt, the last line is by the definitionp1π¯t​\(s,a\)=Pr⁡\(S1=s,π¯ht​\(s\)=a\)p\_\{1\}^\{\\overline\{\\pi\}^\{t\}\}\(s,a\)=\\Pr\(S\_\{1\}=s,\\overline\{\\pi\}^\{t\}\_\{h\}\(s\)=a\)\.

Given \([38](https://arxiv.org/html/2605.23182#A2.E38)\) holds forh′h^\{\\prime\}, apply the Proposition[16](https://arxiv.org/html/2605.23182#Thmtheorem16)again, we can derive

V¯0t​\(s0;δk\)−V¯0t,π¯t​\(⋅;δk\)​\(s0;δk\)\\displaystyle\\overline\{V\}^\{t\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)\-\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\_\{0\}\(s\_\{0\};\\delta\_\{k\}\)≤\\displaystyle\\leq4​H​min⁡\{βp​\(t,δk\)t,1\}\+4​∑h=1h′−1\(H−h\)​∑s′,a′phπ¯t​\(⋅;δk\)​\(s′,a′\)​min⁡\{βp​\(nht​\(s′,a′\),δk\)nht​\(s′,a′\),1\}\+\\displaystyle 4H\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(t,\\delta\_\{k\}\)\}\{t\}\},1\\right\\\}\+4\\sum\_\{h=1\}^\{h^\{\\prime\}\-1\}\(H\-h\)\\sum\_\{s^\{\\prime\},a^\{\\prime\}\}p\_\{h\}^\{\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s^\{\\prime\},a^\{\\prime\}\)\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(n\_\{h\}^\{t\}\(s^\{\\prime\},a^\{\\prime\}\),\\delta\_\{k\}\)\}\{n\_\{h\}^\{t\}\(s^\{\\prime\},a^\{\\prime\}\)\}\},1\\right\\\}\+∑s′,a′ph′π¯t​\(⋅;δk\)\(s′,a′\)\(4\(H−h′\)min\{βp​\(nh′t​\(s′,π¯h′t​\(s′;δk\)\),δk\)nh′t​\(s′,π¯h′t​\(s′;δk\)\),1\}\+\\displaystyle\\sum\_\{s^\{\\prime\},a^\{\\prime\}\}p\_\{h^\{\\prime\}\}^\{\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s^\{\\prime\},a^\{\\prime\}\)\\left\(4\(H\-h^\{\\prime\}\)\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(n\_\{h^\{\\prime\}\}^\{t\}\(s^\{\\prime\},\\overline\{\\pi\}^\{t\}\_\{h^\{\\prime\}\}\(s^\{\\prime\};\\delta\_\{k\}\)\),\\delta\_\{k\}\)\}\{n\_\{h^\{\\prime\}\}^\{t\}\(s^\{\\prime\},\\overline\{\\pi\}^\{t\}\_\{h^\{\\prime\}\}\(s^\{\\prime\};\\delta\_\{k\}\)\)\}\},1\\right\\\}\+\\right\.∑s′′ph′\(s′′\|s′,π¯h′t\(s′;δk\)\)\(V¯h′\+1t\(s′′;δk\)−V¯h′\+1t,π¯t​\(⋅;δk\)\(s′′;δk\)\)\)\\displaystyle\\left\.\\sum\_\{s^\{\\prime\\prime\}\}p\_\{h^\{\\prime\}\}\(s^\{\\prime\\prime\}\|s^\{\\prime\},\\overline\{\\pi\}^\{t\}\_\{h^\{\\prime\}\}\(s^\{\\prime\};\\delta\_\{k\}\)\)\\big\(\\overline\{V\}^\{t\}\_\{h^\{\\prime\}\+1\}\(s^\{\\prime\\prime\};\\delta\_\{k\}\)\-\\underline\{V\}\_\{h^\{\\prime\}\+1\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s^\{\\prime\\prime\};\\delta\_\{k\}\)\\big\)\\right\)=\\displaystyle=4​H​min⁡\{βp​\(t,δk\)t,1\}\+4​∑h=1h′\(H−h\)​∑s′,a′phπ¯t​\(⋅;δk\)​\(s′,a′\)​min⁡\{βp​\(nht​\(s′,a′\),δk\)nht​\(s′,a′\),1\}\+\\displaystyle 4H\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(t,\\delta\_\{k\}\)\}\{t\}\},1\\right\\\}\+4\\sum\_\{h=1\}^\{h^\{\\prime\}\}\(H\-h\)\\sum\_\{s^\{\\prime\},a^\{\\prime\}\}p\_\{h\}^\{\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s^\{\\prime\},a^\{\\prime\}\)\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(n\_\{h\}^\{t\}\(s^\{\\prime\},a^\{\\prime\}\),\\delta\_\{k\}\)\}\{n\_\{h\}^\{t\}\(s^\{\\prime\},a^\{\\prime\}\)\}\},1\\right\\\}\+∑s′,a′∑s′′ph′π¯t​\(⋅;δk\)​\(s′,a′\)​ph′​\(s′′\|s′,π¯h′t​\(s′;δk\)\)​\(V¯h′\+1t​\(s′′;δk\)−V¯h′\+1t,π¯t​\(s′′;δk\)\)\\displaystyle\\sum\_\{s^\{\\prime\},a^\{\\prime\}\}\\sum\_\{s^\{\\prime\\prime\}\}p\_\{h^\{\\prime\}\}^\{\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s^\{\\prime\},a^\{\\prime\}\)p\_\{h^\{\\prime\}\}\(s^\{\\prime\\prime\}\|s^\{\\prime\},\\overline\{\\pi\}^\{t\}\_\{h^\{\\prime\}\}\(s^\{\\prime\};\\delta\_\{k\}\)\)\\big\(\\overline\{V\}^\{t\}\_\{h^\{\\prime\}\+1\}\(s^\{\\prime\\prime\};\\delta\_\{k\}\)\-\\underline\{V\}\_\{h^\{\\prime\}\+1\}^\{t,\\overline\{\\pi\}^\{t\}\}\(s^\{\\prime\\prime\};\\delta\_\{k\}\)\\big\)=\\displaystyle=4​H​min⁡\{βp​\(t,δk\)t,1\}\+4​∑h=1h′\(H−h\)​∑s′,a′phπ¯t​\(s′,a′\)​min⁡\{βp​\(nht​\(s′,a′\),δk\)nht​\(s′,a′\),1\}\+\\displaystyle 4H\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(t,\\delta\_\{k\}\)\}\{t\}\},1\\right\\\}\+4\\sum\_\{h=1\}^\{h^\{\\prime\}\}\(H\-h\)\\sum\_\{s^\{\\prime\},a^\{\\prime\}\}p\_\{h\}^\{\\overline\{\\pi\}^\{t\}\}\(s^\{\\prime\},a^\{\\prime\}\)\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(n\_\{h\}^\{t\}\(s^\{\\prime\},a^\{\\prime\}\),\\delta\_\{k\}\)\}\{n\_\{h\}^\{t\}\(s^\{\\prime\},a^\{\\prime\}\)\}\},1\\right\\\}\+∑s′∑s′′ph′π¯t​\(⋅;δk\)​\(s′,π¯h′t​\(s′;δk\)\)​ph′​\(s′′\|s′,π¯h′t​\(s′;δk\)\)​\(V¯h′\+1t​\(s′′;δk\)−V¯h′\+1t,π¯t​\(s′′;δk\)\)\\displaystyle\\sum\_\{s^\{\\prime\}\}\\sum\_\{s^\{\\prime\\prime\}\}p\_\{h^\{\\prime\}\}^\{\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s^\{\\prime\},\\overline\{\\pi\}^\{t\}\_\{h^\{\\prime\}\}\(s^\{\\prime\};\\delta\_\{k\}\)\)p\_\{h^\{\\prime\}\}\(s^\{\\prime\\prime\}\|s^\{\\prime\},\\overline\{\\pi\}^\{t\}\_\{h^\{\\prime\}\}\(s^\{\\prime\};\\delta\_\{k\}\)\)\\big\(\\overline\{V\}^\{t\}\_\{h^\{\\prime\}\+1\}\(s^\{\\prime\\prime\};\\delta\_\{k\}\)\-\\underline\{V\}\_\{h^\{\\prime\}\+1\}^\{t,\\overline\{\\pi\}^\{t\}\}\(s^\{\\prime\\prime\};\\delta\_\{k\}\)\\big\)=\\displaystyle=4​H​min⁡\{βp​\(t,δk\)t,1\}\+4​∑h=1h′\(H−h\)​∑s′,a′phπ¯t​\(⋅;δk\)​\(s′,a′\)​min⁡\{βp​\(nht​\(s′,a′\),δk\)nht​\(s′,a′\),1\}\+\\displaystyle 4H\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(t,\\delta\_\{k\}\)\}\{t\}\},1\\right\\\}\+4\\sum\_\{h=1\}^\{h^\{\\prime\}\}\(H\-h\)\\sum\_\{s^\{\\prime\},a^\{\\prime\}\}p\_\{h\}^\{\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s^\{\\prime\},a^\{\\prime\}\)\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(n\_\{h\}^\{t\}\(s^\{\\prime\},a^\{\\prime\}\),\\delta\_\{k\}\)\}\{n\_\{h\}^\{t\}\(s^\{\\prime\},a^\{\\prime\}\)\}\},1\\right\\\}\+∑s′′Prπ¯t​\(⋅;δk\)⁡\(Sh′\+1=s′′\)​\(V¯h′\+1t​\(s′′;δk\)−V¯h′\+1t,π¯t​\(⋅;δk\)​\(s′′;δk\)\)\\displaystyle\\sum\_\{s^\{\\prime\\prime\}\}\\Pr\_\{\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(S\_\{h^\{\\prime\}\+1\}=s^\{\\prime\\prime\}\)\\big\(\\overline\{V\}^\{t\}\_\{h^\{\\prime\}\+1\}\(s^\{\\prime\\prime\};\\delta\_\{k\}\)\-\\underline\{V\}\_\{h^\{\\prime\}\+1\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s^\{\\prime\\prime\};\\delta\_\{k\}\)\\big\)=\\displaystyle=4​H​min⁡\{βp​\(t,δk\)t,1\}\+4​∑h=1h′\(H−h\)​∑s′,a′phπ¯t​\(⋅;δk\)​\(s′,a′\)​min⁡\{βp​\(nht​\(s′,a′\),δk\)nht​\(s′,a′\),1\}\+\\displaystyle 4H\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(t,\\delta\_\{k\}\)\}\{t\}\},1\\right\\\}\+4\\sum\_\{h=1\}^\{h^\{\\prime\}\}\(H\-h\)\\sum\_\{s^\{\\prime\},a^\{\\prime\}\}p\_\{h\}^\{\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s^\{\\prime\},a^\{\\prime\}\)\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(n\_\{h\}^\{t\}\(s^\{\\prime\},a^\{\\prime\}\),\\delta\_\{k\}\)\}\{n\_\{h\}^\{t\}\(s^\{\\prime\},a^\{\\prime\}\)\}\},1\\right\\\}\+∑s′,a′ph′\+1π¯t​\(⋅;δk\)​\(s′,a′\)​\(V¯h′\+1t​\(s′′;δk\)−V¯h′\+1t,π¯t​\(⋅;δk\)​\(s′′;δk\)\),\\displaystyle\\sum\_\{s^\{\\prime\},a^\{\\prime\}\}p^\{\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\_\{h^\{\\prime\}\+1\}\(s^\{\\prime\},a^\{\\prime\}\)\\big\(\\overline\{V\}^\{t\}\_\{h^\{\\prime\}\+1\}\(s^\{\\prime\\prime\};\\delta\_\{k\}\)\-\\underline\{V\}\_\{h^\{\\prime\}\+1\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s^\{\\prime\\prime\};\\delta\_\{k\}\)\\big\),which completed the induction and \([38](https://arxiv.org/html/2605.23182#A2.E38)\) holds for allh′=1,2,⋯,H\+1h^\{\\prime\}=1,2,\\cdots,H\+1\.

By takingh′=Hh^\{\\prime\}=Hand utilizing the fact thatV¯H\+1t​\(s;δk\)=0,V¯H\+1t,π¯t​\(⋅;δk\)​\(s;δk\)=0,∀s\\overline\{V\}^\{t\}\_\{H\+1\}\(s;\\delta\_\{k\}\)=0,\\underline\{V\}\_\{H\+1\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s;\\delta\_\{k\}\)=0,\\forall s, we complete the proof of Lemma[17](https://arxiv.org/html/2605.23182#Thmtheorem17)\. ∎

###### Definition 7\.

Denotetk−1t\_\{k\-1\}as the starting t index of phasekkfor phase indexk≥1k\\geq 1\. Fork≥2k\\geq 2,tk−1t\_\{k\-1\}is a random variable\. For consistency, we sett0=0t\_\{0\}=0\.

What’s more, we know¬\\neg\([3](https://arxiv.org/html/2605.23182#S3.E3)\) and¬\\neg\([4](https://arxiv.org/html/2605.23182#S3.E4)\) must hold for episode indext=tk−1,tk−1\+1,⋯,tk−1t=t\_\{k\-1\},t\_\{k\-1\}\+1,\\cdots,t\_\{k\}\-1\. It is indeed possible thattk−1=tkt\_\{k\-1\}=t\_\{k\}for some sampling path\{\{St,h,At,h\}h=1H\}t=1\+∞\\\{\\\{S\_\{t,h\},A\_\{t,h\}\\\}\_\{h=1\}^\{H\}\\\}\_\{t=1\}^\{\+\\infty\}and indexk∈ℕk\\in\\mathbb\{N\}\. In this case, the historyℋee\\mathcal\{H\}^\{\\text\{ee\}\}with length\|ℋee\|=tk−1\|\\mathcal\{H\}^\{\\text\{ee\}\}\|=t\_\{k\-1\}guarantees thatV¯0t,π¯t​\(⋅;δk\)​\(s0;δk\+1\)−\(C−1\)​\(V¯0t​\(s0;δk\+1\)−V¯0t,π¯t​\(⋅;δk\)​\(s0;δk\+1\)\)\>μ0\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\_\{0\}\(s\_\{0\};\\delta\_\{k\+1\}\)\-\(C\-1\)\(\\overline\{V\}^\{t\}\_\{0\}\(s\_\{0\};\\delta\_\{k\+1\}\)\-\\underline\{V\}^\{t,\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\_\{0\}\(s\_\{0\};\\delta\_\{k\+1\}\)\)\>\\mu\_\{0\}orV¯0t​\(s0;δk\+1\)<μ0\\overline\{V\}^\{t\}\_\{0\}\(s\_\{0\};\\delta\_\{k\+1\}\)<\\mu\_\{0\}\. Following this idea, we can combine Lemma[14](https://arxiv.org/html/2605.23182#Thmtheorem14)and[17](https://arxiv.org/html/2605.23182#Thmtheorem17)to derive upper bounds for the length of each phase\.

###### Lemma 18\.

Consider any phase indexk≥κk\\geq\\kappa\. If the instance is positive, we have

\(tk−tk−1\)​V0∗​\(s0\)−μC\\displaystyle\(t\_\{k\}\-t\_\{k\-1\}\)\\frac\{V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\}\{C\}≤\\displaystyle\\leq∑t=tk−1tk−1∑h=0H∑\(s,a\)phπ¯t​\(⋅;δk\)​\(s,a\)​4​\(H−h\)​min⁡\{βp​\(nht​\(s,a\),δk\)nht​\(s,a\),1\}\.\\displaystyle\\sum\_\{t=t\_\{k\-1\}\}^\{t\_\{k\}\-1\}\\sum\_\{h=0\}^\{H\}\\sum\_\{\(s,a\)\}p\_\{h\}^\{\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s,a\)4\(H\-h\)\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(n\_\{h\}^\{t\}\(s,a\),\\delta\_\{k\}\)\}\{n\_\{h\}^\{t\}\(s,a\)\}\},1\\right\\\}\.If the instance is negative, we have

\(tk−tk−1\)​\(μ−V0∗​\(s0\)\)\\displaystyle\(t\_\{k\}\-t\_\{k\-1\}\)\(\\mu\-V^\{\*\}\_\{0\}\(s\_\{0\}\)\)≤\\displaystyle\\leq∑t=tk−1tk−1∑h=0H∑\(s,a\)phπ¯t​\(⋅;δk\)​\(s,a\)​4​\(H−h\)​min⁡\{βp​\(nht​\(s,a\),δk\)nht​\(s,a\),1\}\.\\displaystyle\\sum\_\{t=t\_\{k\-1\}\}^\{t\_\{k\}\-1\}\\sum\_\{h=0\}^\{H\}\\sum\_\{\(s,a\)\}p\_\{h\}^\{\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s,a\)4\(H\-h\)\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(n\_\{h\}^\{t\}\(s,a\),\\delta\_\{k\}\)\}\{n\_\{h\}^\{t\}\(s,a\)\}\},1\\right\\\}\.

###### Proof of Lemma[18](https://arxiv.org/html/2605.23182#Thmtheorem18)\.

If the instance is positive, by Lemma[14](https://arxiv.org/html/2605.23182#Thmtheorem14)and[17](https://arxiv.org/html/2605.23182#Thmtheorem17), we can conclude

V0∗​\(s0\)−μ0C\\displaystyle\\frac\{V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\}\{C\}≤\\displaystyle\\leq4​H​min⁡\{βp​\(t,δk\)t,1\}\+4​∑h=1H\(H−h\)​∑s′,a′phπ¯t​\(⋅;δk\)​\(s′,a′\)​min⁡\{βp​\(nht​\(s′,a′\),δk\)nht​\(s′,a′\),1\}\\displaystyle 4H\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(t,\\delta\_\{k\}\)\}\{t\}\},1\\right\\\}\+4\\sum\_\{h=1\}^\{H\}\(H\-h\)\\sum\_\{s^\{\\prime\},a^\{\\prime\}\}p\_\{h\}^\{\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s^\{\\prime\},a^\{\\prime\}\)\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(n\_\{h\}^\{t\}\(s^\{\\prime\},a^\{\\prime\}\),\\delta\_\{k\}\)\}\{n\_\{h\}^\{t\}\(s^\{\\prime\},a^\{\\prime\}\)\}\},1\\right\\\}holds for allt=tk−1,tk−1\+1,⋯\.tk−1t=t\_\{k\-1\},t\_\{k\-1\}\+1,\\cdots\.t\_\{k\}\-1\. Summing up the above inequalities fort=tk−1,tk−1\+1,⋯​tk−1t=t\_\{k\-1\},t\_\{k\-1\}\+1,\\cdots t\_\{k\}\-1leads to the inequality for the positive case\.

If the instance is negative, by Lemma[14](https://arxiv.org/html/2605.23182#Thmtheorem14)and[17](https://arxiv.org/html/2605.23182#Thmtheorem17), we can conclude

μ0−V0∗​\(s0\)\\displaystyle\\mu\_\{0\}\-V^\{\*\}\_\{0\}\(s\_\{0\}\)≤\\displaystyle\\leq4​H​min⁡\{βp​\(t,δk\)t,1\}\+4​∑h=1H\(H−h\)​∑s′,a′phπ¯t​\(⋅;δk\)​\(s′,a′\)​min⁡\{βp​\(nht​\(s′,a′\),δk\)nht​\(s′,a′\),1\}\\displaystyle 4H\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(t,\\delta\_\{k\}\)\}\{t\}\},1\\right\\\}\+4\\sum\_\{h=1\}^\{H\}\(H\-h\)\\sum\_\{s^\{\\prime\},a^\{\\prime\}\}p\_\{h\}^\{\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{k\}\)\}\(s^\{\\prime\},a^\{\\prime\}\)\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(n\_\{h\}^\{t\}\(s^\{\\prime\},a^\{\\prime\}\),\\delta\_\{k\}\)\}\{n\_\{h\}^\{t\}\(s^\{\\prime\},a^\{\\prime\}\)\}\},1\\right\\\}holds for allt=tk−1,tk−1\+1,⋯\.tk−1t=t\_\{k\-1\},t\_\{k\-1\}\+1,\\cdots\.t\_\{k\}\-1\. Summing up the above inequalities fort=tk−1,tk−1\+1,⋯​tk−1t=t\_\{k\-1\},t\_\{k\-1\}\+1,\\cdots t\_\{k\}\-1leads to the inequality for the negative case\. ∎

To derive an upper bound fortkt\_\{k\}, we need are going to prove an upper bound for the right\-hand side of inequalities in Lemma[18](https://arxiv.org/html/2605.23182#Thmtheorem18)\. The following Lemma will be useful\.

###### Lemma 19\(Lemma 7 inKaufmannet al\.\[[2021](https://arxiv.org/html/2605.23182#bib.bib9)\]\)\.

For any roundttduring phasek≥κk\\geq\\kappa, any pair\(s,a,h\)\(s,a,h\)

min⁡\{βp​\(nht​\(s,a\),δk\)nht​\(s,a\),1\}≤2​βp​\(n¯ht​\(s,a\),δk\)max⁡\{n¯ht​\(s,a\),1\}\\displaystyle\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(n\_\{h\}^\{t\}\(s,a\),\\delta\_\{k\}\)\}\{n\_\{h\}^\{t\}\(s,a\)\}\},1\\right\\\}\\leq 2\\sqrt\{\\frac\{\\beta\_\{p\}\(\\bar\{n\}\_\{h\}^\{t\}\(s,a\),\\delta\_\{k\}\)\}\{\\max\\\{\\bar\{n\}\_\{h\}^\{t\}\(s,a\),1\\\}\}\}holds for allt<τt<\\tau\.

###### Proof of Lemma[19](https://arxiv.org/html/2605.23182#Thmtheorem19)\.

Recall the definitionβcnt​\(δ\)=log⁡2​S​A​Hδ\\beta^\{\\text\{cnt\}\}\(\\delta\)=\\log\\frac\{2SAH\}\{\\delta\},βp​\(t,δ\)=log⁡2​S​A​Hδ\+\(S−1\)​log⁡\(e​\(1\+tS−1\)\)\\beta\_\{p\}\(t,\\delta\)=\\log\\frac\{2SAH\}\{\\delta\}\+\(S\-1\)\\log\\Big\(e\(1\+\\frac\{t\}\{S\-1\}\)\\Big\)\.k≥κk\\geq\\kappasuggests thatnht​\(s,a\)≥12​n¯ht​\(s,a\)−βcnt​\(δk\)n^\{t\}\_\{h\}\(s,a\)\\geq\\frac\{1\}\{2\}\\bar\{n\}\_\{h\}^\{t\}\(s,a\)\-\\beta^\{\\text\{cnt\}\}\(\\delta\_\{k\}\)holds for alls,a,hs,a,h\.

We prove Lemma[19](https://arxiv.org/html/2605.23182#Thmtheorem19)in two cases,βc​n​t​\(δk\)≤14​n¯ht​\(s,a\)\\beta^\{cnt\}\(\\delta\_\{k\}\)\\leq\\frac\{1\}\{4\}\\bar\{n\}\_\{h\}^\{t\}\(s,a\)orβc​n​t​\(δk\)\>14​n¯ht​\(s,a\)\\beta^\{cnt\}\(\\delta\_\{k\}\)\>\\frac\{1\}\{4\}\\bar\{n\}\_\{h\}^\{t\}\(s,a\)\.

First, ifβc​n​t​\(δk\)≤14​n¯ht​\(s,a\)\\beta^\{cnt\}\(\\delta\_\{k\}\)\\leq\\frac\{1\}\{4\}\\bar\{n\}\_\{h\}^\{t\}\(s,a\), we havenht​\(s,a\)≥14​n¯ht​\(s,a\)n\_\{h\}^\{t\}\(s,a\)\\geq\\frac\{1\}\{4\}\\bar\{n\}\_\{h\}^\{t\}\(s,a\)\. Notice thatβp​\(t,δ\)=log⁡2​S​A​Hδ\+\(S−1\)​log⁡\(S−1\+t\)\+\(S−1\)​log⁡eS−1\\beta\_\{p\}\(t,\\delta\)=\\log\\frac\{2SAH\}\{\\delta\}\+\(S\-1\)\\log\\Big\(S\-1\+t\\Big\)\+\(S\-1\)\\log\\frac\{e\}\{S\-1\}, we can derive its partial derivative regardingtt,

∂βp​\(t,δ\)t∂t=\\displaystyle\\frac\{\\partial\\frac\{\\beta\_\{p\}\(t,\\delta\)\}\{t\}\}\{\\partial t\}=−log⁡2​S​A​Hδt2\+\(S−1\)​\(1S−1\+t−log⁡\(S−1\+t\)\)t2\\displaystyle\-\\frac\{\\log\\frac\{2SAH\}\{\\delta\}\}\{t^\{2\}\}\+\\frac\{\(S\-1\)\\left\(\\frac\{1\}\{S\-1\+t\}\-\\log\(S\-1\+t\)\\right\)\}\{t^\{2\}\}Fort≥1t\\geq 1,1S−1\+t−log⁡\(S−1\+t\)≤1S−log⁡S<0\\frac\{1\}\{S\-1\+t\}\-\\log\(S\-1\+t\)\\leq\\frac\{1\}\{S\}\-\\log S<0holds forS≥2S\\geq 2\. We can conclude∂βp​\(t,δ\)t∂t<0\\frac\{\\partial\\frac\{\\beta\_\{p\}\(t,\\delta\)\}\{t\}\}\{\\partial t\}<0fort≥1t\\geq 1\. Sinceβc​n​t​\(δk\)\>1\\beta^\{cnt\}\(\\delta\_\{k\}\)\>1holds for allkk, we can conclude14​n¯ht​\(s,a\)\>1\\frac\{1\}\{4\}\\bar\{n\}\_\{h\}^\{t\}\(s,a\)\>1\. Together withnht​\(s,a\)≥14​n¯ht​\(s,a\)n\_\{h\}^\{t\}\(s,a\)\\geq\\frac\{1\}\{4\}\\bar\{n\}\_\{h\}^\{t\}\(s,a\), we have

min⁡\{βp​\(nht​\(s,a\),δk\)nht​\(s,a\),1\}≤βp​\(nht​\(s,a\),δk\)nht​\(s,a\)≤βp​\(14​n¯ht​\(s,a\),δk\)14​n¯ht​\(s,a\)≤2​βp​\(n¯ht​\(s,a\),δk\)n¯ht​\(s,a\)\.\\displaystyle\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(n\_\{h\}^\{t\}\(s,a\),\\delta\_\{k\}\)\}\{n\_\{h\}^\{t\}\(s,a\)\}\},1\\right\\\}\\leq\\sqrt\{\\frac\{\\beta\_\{p\}\(n\_\{h\}^\{t\}\(s,a\),\\delta\_\{k\}\)\}\{n\_\{h\}^\{t\}\(s,a\)\}\}\\leq\\sqrt\{\\frac\{\\beta\_\{p\}\(\\frac\{1\}\{4\}\\bar\{n\}\_\{h\}^\{t\}\(s,a\),\\delta\_\{k\}\)\}\{\\frac\{1\}\{4\}\\bar\{n\}\_\{h\}^\{t\}\(s,a\)\}\}\\leq 2\\sqrt\{\\frac\{\\beta\_\{p\}\(\\bar\{n\}\_\{h\}^\{t\}\(s,a\),\\delta\_\{k\}\)\}\{\\bar\{n\}\_\{h\}^\{t\}\(s,a\)\}\}\.Again, by the fact that4​βp​\(n¯ht​\(s,a\),δk\)n¯ht​\(s,a\)=4​βp​\(n¯ht​\(s,a\),δk\)max⁡\{n¯ht​\(s,a\),1\}\\frac\{4\\beta\_\{p\}\(\\bar\{n\}\_\{h\}^\{t\}\(s,a\),\\delta\_\{k\}\)\}\{\\bar\{n\}\_\{h\}^\{t\}\(s,a\)\}=\\frac\{4\\beta\_\{p\}\(\\bar\{n\}\_\{h\}^\{t\}\(s,a\),\\delta\_\{k\}\)\}\{\\max\\\{\\bar\{n\}\_\{h\}^\{t\}\(s,a\),1\\\}\}, we can conclude

min⁡\{βp​\(nht​\(s,a\),δk\)nht​\(s,a\),1\}≤2​βp​\(n¯ht​\(s,a\),δk\)max⁡\{n¯ht​\(s,a\),1\},\\displaystyle\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(n\_\{h\}^\{t\}\(s,a\),\\delta\_\{k\}\)\}\{n\_\{h\}^\{t\}\(s,a\)\}\},1\\right\\\}\\leq 2\\sqrt\{\\frac\{\\beta\_\{p\}\(\\bar\{n\}\_\{h\}^\{t\}\(s,a\),\\delta\_\{k\}\)\}\{\\max\\\{\\bar\{n\}\_\{h\}^\{t\}\(s,a\),1\\\}\}\},which suggests the Lemma holds\.

Second, ifβc​n​t​\(δk\)\>14​n¯ht​\(s,a\)\\beta^\{cnt\}\(\\delta\_\{k\}\)\>\\frac\{1\}\{4\}\\bar\{n\}\_\{h\}^\{t\}\(s,a\), we have4​βc​n​t​\(δk\)n¯ht​\(s,a\)≥1\\frac\{4\\beta^\{cnt\}\(\\delta\_\{k\}\)\}\{\\bar\{n\}\_\{h\}^\{t\}\(s,a\)\}\\geq 1\. Meanwhile,βc​n​t​\(δk\)≥1\\beta^\{cnt\}\(\\delta\_\{k\}\)\\geq 1by the definition, suggesting that

min⁡\{βp​\(nht​\(s,a\),δk\)nht​\(s,a\),1\}≤1≤4​βc​n​t​\(δk\)max⁡\{n¯ht​\(s,a\),1\}=2​βc​n​t​\(δk\)max⁡\{n¯ht​\(s,a\),1\}\\displaystyle\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(n\_\{h\}^\{t\}\(s,a\),\\delta\_\{k\}\)\}\{n\_\{h\}^\{t\}\(s,a\)\}\},1\\right\\\}\\leq 1\\leq\\sqrt\{\\frac\{4\\beta^\{cnt\}\(\\delta\_\{k\}\)\}\{\\max\\\{\\bar\{n\}\_\{h\}^\{t\}\(s,a\),1\\\}\}\}=2\\sqrt\{\\frac\{\\beta^\{cnt\}\(\\delta\_\{k\}\)\}\{\\max\\\{\\bar\{n\}\_\{h\}^\{t\}\(s,a\),1\\\}\}\}\(39\)By the definition ofβp\\beta\_\{p\}andβcnt\\beta^\{\\text\{cnt\}\}, we knowβp​\(t,δk\)\>βcnt​\(δk\)\\beta\_\{p\}\(t,\\delta\_\{k\}\)\>\\beta^\{\\text\{cnt\}\}\(\\delta\_\{k\}\)holds for allt\>0t\>0\. Together with \([39](https://arxiv.org/html/2605.23182#A2.E39)\), we conclude

min⁡\{βp​\(nht​\(s,a\),δk\)nht​\(s,a\),1\}≤2​βc​n​t​\(δk\)max⁡\{n¯ht​\(s,a\),1\}<2​β​\(n¯ht​\(s,a\),δk\)max⁡\{n¯ht​\(s,a\),1\},\\displaystyle\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(n\_\{h\}^\{t\}\(s,a\),\\delta\_\{k\}\)\}\{n\_\{h\}^\{t\}\(s,a\)\}\},1\\right\\\}\\leq 2\\sqrt\{\\frac\{\\beta^\{cnt\}\(\\delta\_\{k\}\)\}\{\\max\\\{\\bar\{n\}\_\{h\}^\{t\}\(s,a\),1\\\}\}\}<2\\sqrt\{\\frac\{\\beta\(\\bar\{n\}\_\{h\}^\{t\}\(s,a\),\\delta\_\{k\}\)\}\{\\max\\\{\\bar\{n\}\_\{h\}^\{t\}\(s,a\),1\\\}\}\},which means the Lemma still holds\. ∎

Given all the above Lemmas, we are ready to prove a main theorem, showing that Algorithm[2](https://arxiv.org/html/2605.23182#alg2)is able to output the correct prediction for large enough phase indexkk, with an upper bound oftkt\_\{k\}\.

###### Theorem 20\.

Apply Algorithm[2](https://arxiv.org/html/2605.23182#alg2)on an instanceν\\nu\.

Ifν\\nuis positive, for any phase indexk≥max⁡\{κ,⌈log2⁡9600​\(H\+1\)2ϵpos2⌉\}k\\geq\\max\\\{\\kappa,\\lceil\\log\_\{2\}\\frac\{9600\(H\+1\)^\{2\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\rceil\\\}, Algorithm[2](https://arxiv.org/html/2605.23182#alg2)will output a policyπ^k\\hat\{\\pi\}\_\{k\}satisfying

V0π^k​\(s0\)−μ0≥C−1C​\(V0∗​\(s0\)−μ0\)\\displaystyle V^\{\\hat\{\\pi\}\_\{k\}\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\\geq\\frac\{C\-1\}\{C\}\\Big\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\\Big\)at the end of phasekk, with

tk≤101​\(H\+1\)4​S​A​log⁡2​S​A​Hδkϵpos2\+Tκ−1ee\+400​\(H\+1\)4​S2​Aϵpos2​log⁡\(402​\(H\+1\)4​S2​A​log⁡2​S​A​Hδkϵpos2\)\+400​\(H\+1\)4​S2​Aϵpos2​log⁡\(2​Tκ−1ee\)\.\\begin\{split\}t\_\{k\}\\leq&\\frac\{101\(H\+1\)^\{4\}SA\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\+T\_\{\\kappa\-1\}^\{\\text\{ee\}\}\+\\\\ &\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\log\\left\(\\frac\{402\(H\+1\)^\{4\}S^\{2\}A\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\right\)\+\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\log\\left\(2T\_\{\\kappa\-1\}^\{\\text\{ee\}\}\\right\)\.\\end\{split\}\(40\)
Ifν\\nuis negative, for any phase indexk≥max⁡\{κ,⌈log2⁡9600​\(H\+1\)2ϵneg2⌉\}k\\geq\\max\\\{\\kappa,\\lceil\\log\_\{2\}\\frac\{9600\(H\+1\)^\{2\}\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\\rceil\\\}, Algorithm[2](https://arxiv.org/html/2605.23182#alg2)will outputNoneat the end of phasekk, with

tk≤101​\(H\+1\)4​S​A​log⁡2​S​A​Hδkϵneg2\+Tκ−1ee\+400​\(H\+1\)4​S2​Aϵneg2​log⁡\(402​\(H\+1\)4​S2​A​log⁡2​S​A​Hδkϵneg2\)\+400​\(H\+1\)4​S2​Aϵneg2​log⁡\(2​Tκ−1ee\)\\begin\{split\}t\_\{k\}\\leq&\\frac\{101\(H\+1\)^\{4\}SA\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\+T\_\{\\kappa\-1\}^\{\\text\{ee\}\}\+\\\\ &\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\\log\\left\(\\frac\{402\(H\+1\)^\{4\}S^\{2\}A\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\\right\)\+\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{neg\}\}\}^\{2\}\}\\log\\left\(2T\_\{\\kappa\-1\}^\{\\text\{ee\}\}\\right\)\\end\{split\}\(41\)Hereϵpos=V0∗​\(s0\)−μ0C\\epsilon^\{\\text\{pos\}\}=\\frac\{V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\}\{C\}, andϵneg=μ0−V0∗​\(s0\)\\epsilon^\{\\text\{neg\}\}=\\mu\_\{0\}\-V^\{\*\}\_\{0\}\(s\_\{0\}\)\.

###### Proof of Theorem[20](https://arxiv.org/html/2605.23182#Thmtheorem20)\.

We temporarily focus on the positive case\. The analysis for the negative case is similar\.

From Lemma[18](https://arxiv.org/html/2605.23182#Thmtheorem18), we know

\(tk−tk−1\)​ϵpos≤\\displaystyle\(t\_\{k\}\-t\_\{k\-1\}\)\\epsilon^\{\\text\{pos\}\}\\leq∑t=tk−1tk−1∑h=0H∑\(s,a\)phπ¯t​\(s,a\)​4​\(H−h\)​min⁡\{βp​\(nht​\(s,a\),δk\)nht​\(s,a\),1\}\\displaystyle\\sum\_\{t=t\_\{k\-1\}\}^\{t\_\{k\}\-1\}\\sum\_\{h=0\}^\{H\}\\sum\_\{\(s,a\)\}p\_\{h\}^\{\\overline\{\\pi\}^\{t\}\}\(s,a\)4\(H\-h\)\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(n\_\{h\}^\{t\}\(s,a\),\\delta\_\{k\}\)\}\{n\_\{h\}^\{t\}\(s,a\)\}\},1\\right\\\}\(42\)holds for allk≥κk\\geq\\kappa\. Recall we denotetk−1t\_\{k\-1\}as the starting t index of stagekk, we first show \([40](https://arxiv.org/html/2605.23182#A2.E40)\) holds fork≥κk\\geq\\kappa\. In the following, we will consider consider multiple phase indexesℓ=κ,κ\+1,⋯,k\\ell=\\kappa,\\kappa\+1,\\cdots,k\. For simplicity, we will useπ¯t\\overline\{\\pi\}^\{t\}as the shorthand ofπ¯t​\(⋅;δℓ\)\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{\\ell\}\)in the following calculation\.

By summing up \([42](https://arxiv.org/html/2605.23182#A2.E42)\) for stage indexℓ=κ,κ\+1,⋯,k\\ell=\\kappa,\\kappa\+1,\\cdots,k, We have

\(tk−tκ−1\)​ϵpos\\displaystyle\(t\_\{k\}\-t\_\{\\kappa\-1\}\)\\epsilon^\{\\text\{pos\}\}≤\\displaystyle\\leq∑ℓ=κk∑t=tℓ−1tℓ−1∑h=0H∑\(s,a\)phπ¯t​\(⋅;δℓ\)​\(s,a\)​4​\(H−h\)​min⁡\{βp​\(nht​\(s,a\),δℓ\)nht​\(s,a\),1\}\\displaystyle\\sum\_\{\\ell=\\kappa\}^\{k\}\\sum\_\{t=t\_\{\\ell\-1\}\}^\{t\_\{\\ell\}\-1\}\\sum\_\{h=0\}^\{H\}\\sum\_\{\(s,a\)\}p\_\{h\}^\{\\overline\{\\pi\}^\{t\}\(\\cdot;\\delta\_\{\\ell\}\)\}\(s,a\)4\(H\-h\)\\min\\left\\\{\\sqrt\{\\frac\{\\beta\_\{p\}\(n\_\{h\}^\{t\}\(s,a\),\\delta\_\{\\ell\}\)\}\{n\_\{h\}^\{t\}\(s,a\)\}\},1\\right\\\}≤Lemma[19](https://arxiv.org/html/2605.23182#Thmtheorem19)\\displaystyle\\stackrel\{\{\\scriptstyle\\text\{Lemma \}\\ref\{lemma:lower\-bound\-on\-n\_h^t\(s,a\)\}\}\}\{\{\\leq\}\}∑ℓ=κk∑t=tℓ−1tℓ−1∑h=0H∑\(s,a\)phπ¯t​\(s,a\)​8​\(H−h\)​βp​\(n¯ht​\(s,a\),δℓ\)max⁡\{n¯ht​\(s,a\),1\}\\displaystyle\\sum\_\{\\ell=\\kappa\}^\{k\}\\sum\_\{t=t\_\{\\ell\-1\}\}^\{t\_\{\\ell\}\-1\}\\sum\_\{h=0\}^\{H\}\\sum\_\{\(s,a\)\}p\_\{h\}^\{\\overline\{\\pi\}^\{t\}\}\(s,a\)8\(H\-h\)\\sqrt\{\\frac\{\\beta\_\{p\}\(\\bar\{n\}\_\{h\}^\{t\}\(s,a\),\\delta\_\{\\ell\}\)\}\{\\max\\\{\\bar\{n\}\_\{h\}^\{t\}\(s,a\),1\\\}\}\}≤\\displaystyle\\leq8​βp​\(tk,δk\)​∑h=0H\(H−h\)​∑\(s,a\)∑ℓ=κk∑t=tℓ−1tℓ−1phπ¯t​\(s,a\)​1max⁡\{n¯ht​\(s,a\),1\}\\displaystyle 8\\sqrt\{\\beta\_\{p\}\(t\_\{k\},\\delta\_\{k\}\)\}\\sum\_\{h=0\}^\{H\}\(H\-h\)\\sum\_\{\(s,a\)\}\\sum\_\{\\ell=\\kappa\}^\{k\}\\sum\_\{t=t\_\{\\ell\-1\}\}^\{t\_\{\\ell\}\-1\}p\_\{h\}^\{\\overline\{\\pi\}^\{t\}\}\(s,a\)\\sqrt\{\\frac\{1\}\{\\max\\\{\\bar\{n\}\_\{h\}^\{t\}\(s,a\),1\\\}\}\}=\\displaystyle=8​βp​\(tk,δk\)​∑h=0H\(H−h\)​∑\(s,a\)∑ℓ=κk∑t=tℓ−1tℓ−1\\displaystyle 8\\sqrt\{\\beta\_\{p\}\(t\_\{k\},\\delta\_\{k\}\)\}\\sum\_\{h=0\}^\{H\}\(H\-h\)\\sum\_\{\(s,a\)\}\\sum\_\{\\ell=\\kappa\}^\{k\}\\sum\_\{t=t\_\{\\ell\-1\}\}^\{t\_\{\\ell\}\-1\}phπ¯t​\(s,a\)​1max⁡\{∑v=0tκ−1−1phπ¯v​\(s,a\)\+∑v=tκ−1t−1phπ¯v​\(s,a\),1\}\\displaystyle p\_\{h\}^\{\\overline\{\\pi\}^\{t\}\}\(s,a\)\\sqrt\{\\frac\{1\}\{\\max\\\{\\sum\_\{v=0\}^\{t\_\{\\kappa\-1\}\-1\}p\_\{h\}^\{\\overline\{\\pi\}^\{v\}\}\(s,a\)\+\\sum\_\{v=t\_\{\\kappa\-1\}\}^\{t\-1\}p\_\{h\}^\{\\overline\{\\pi\}^\{v\}\}\(s,a\),1\\\}\}\}=\\displaystyle=8​βp​\(tk,δk\)​∑h=0H\(H−h\)​∑\(s,a\)∑t=tκ−1tk−1\\displaystyle 8\\sqrt\{\\beta\_\{p\}\(t\_\{k\},\\delta\_\{k\}\)\}\\sum\_\{h=0\}^\{H\}\(H\-h\)\\sum\_\{\(s,a\)\}\\sum\_\{t=t\_\{\\kappa\-1\}\}^\{t\_\{k\}\-1\}phπ¯t​\(s,a\)​1max⁡\{∑v=0tκ−1−1phπ¯v​\(s,a\)\+∑v=tκ−1t−1phπ¯v​\(s,a\),1\}\\displaystyle p\_\{h\}^\{\\overline\{\\pi\}^\{t\}\}\(s,a\)\\sqrt\{\\frac\{1\}\{\\max\\\{\\sum\_\{v=0\}^\{t\_\{\\kappa\-1\}\-1\}p\_\{h\}^\{\\overline\{\\pi\}^\{v\}\}\(s,a\)\+\\sum\_\{v=t\_\{\\kappa\-1\}\}^\{t\-1\}p\_\{h\}^\{\\overline\{\\pi\}^\{v\}\}\(s,a\),1\\\}\}\}≤Lemma[41](https://arxiv.org/html/2605.23182#Thmtheorem41)\\displaystyle\\stackrel\{\{\\scriptstyle\\text\{Lemma \}\\ref\{lemma:summation\-of\-prob\}\}\}\{\{\\leq\}\}8​\(1\+2\)​βp​\(tk,δk\)​∑h=0H\(H−h\)​∑\(s,a\)∑v=tκ−1tk−1phπ¯v​\(s,a\)\.\\displaystyle 8\(1\+\\sqrt\{2\}\)\\sqrt\{\\beta\_\{p\}\(t\_\{k\},\\delta\_\{k\}\)\}\\sum\_\{h=0\}^\{H\}\(H\-h\)\\sum\_\{\(s,a\)\}\\sqrt\{\\sum\_\{v=t\_\{\\kappa\-1\}\}^\{t\_\{k\}\-1\}p\_\{h\}^\{\\overline\{\\pi\}^\{v\}\}\(s,a\)\}\.\(43\)To get the last step, we apply Lemma[41](https://arxiv.org/html/2605.23182#Thmtheorem41)by takingM=∑v=0tκ−1−1phv​\(s,a\)M=\\sum\_\{v=0\}^\{t\_\{\\kappa\-1\}\-1\}p\_\{h\}^\{v\}\(s,a\)\. By the Cauchy\-Schwarz Inequality,

∑\(s,a\)∑v=tκ−1tk−1phv​\(s,a\)≤\(∑s,a∑v=tκ−1tk−1phv​\(s,a\)\)​\(∑\(s,a\)1\)=S​A​\(tk−tκ−1\)\.\\displaystyle\\sum\_\{\(s,a\)\}\\sqrt\{\\sum\_\{v=t\_\{\\kappa\-1\}\}^\{t\_\{k\}\-1\}p\_\{h\}^\{v\}\(s,a\)\}\\leq\\sqrt\{\\left\(\\sum\_\{s,a\}\\sum\_\{v=t\_\{\\kappa\-1\}\}^\{t\_\{k\}\-1\}p\_\{h\}^\{v\}\(s,a\)\\right\)\(\\sum\_\{\(s,a\)\}1\)\}=\\sqrt\{SA\(t\_\{k\}\-t\_\{\\kappa\-1\}\)\}\.Following \([43](https://arxiv.org/html/2605.23182#A2.E43)\), we conclude

\(tk−tκ−1\)​ϵpos\\displaystyle\(t\_\{k\}\-t\_\{\\kappa\-1\}\)\\epsilon^\{\\text\{pos\}\}≤\\displaystyle\\leq8​\(1\+2\)​S​A​βp​\(tk,δk\)​∑h=0H\(H−h\)​tk−tκ−1\\displaystyle 8\(1\+\\sqrt\{2\}\)\\sqrt\{SA\\beta\_\{p\}\(t\_\{k\},\\delta\_\{k\}\)\}\\sum\_\{h=0\}^\{H\}\(H\-h\)\\sqrt\{t\_\{k\}\-t\_\{\\kappa\-1\}\}=\\displaystyle=4​\(1\+2\)​S​A​βp​\(tk,δk\)​H​\(H\+1\)​tk−tκ−1\.\\displaystyle 4\(1\+\\sqrt\{2\}\)\\sqrt\{SA\\beta\_\{p\}\(t\_\{k\},\\delta\_\{k\}\)\}H\(H\+1\)\\sqrt\{t\_\{k\}\-t\_\{\\kappa\-1\}\}\.\(44\)By the definitionβp​\(t,δ\)=log⁡2​S​A​Hδ\+\(S−1\)​log⁡\(e​\(1\+tS−1\)\)\\beta\_\{p\}\(t,\\delta\)=\\log\\frac\{2SAH\}\{\\delta\}\+\(S\-1\)\\log\\Big\(e\(1\+\\frac\{t\}\{S\-1\}\)\\Big\), \([44](https://arxiv.org/html/2605.23182#A2.E44)\) suggests that

\(tk−tκ−1\)​ϵpos≤4​\(1\+2\)​H​\(H\+1\)​log⁡2​S​A​Hδk\+S​log⁡\(e​tk\)​S​A​tk−tκ−1\\displaystyle\(t\_\{k\}\-t\_\{\\kappa\-1\}\)\\epsilon^\{\\text\{pos\}\}\\leq 4\(1\+\\sqrt\{2\}\)H\(H\+1\)\\sqrt\{\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\+S\\log\(et\_\{k\}\)\}\\sqrt\{SA\}\\sqrt\{t\_\{k\}\-t\_\{\\kappa\-1\}\}⇔\\displaystyle\\Leftrightarrow\(tk−tκ−1\)​ϵpos2≤100​\(H\+1\)4​S​A​log⁡2​S​A​Hδk\+100​\(H\+1\)4​S2​A​log⁡\(e​tk\)\\displaystyle\(t\_\{k\}\-t\_\{\\kappa\-1\}\)\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\\leq 100\(H\+1\)^\{4\}SA\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\+100\(H\+1\)^\{4\}S^\{2\}A\\log\(et\_\{k\}\)⇔\\displaystyle\\Leftrightarrowtk≤100​\(H\+1\)4​S​A​log⁡2​S​A​Hδkϵpos2\+tκ−1\+100​\(H\+1\)4​S2​Aϵpos2\+100​\(H\+1\)4​S2​A​log⁡tkϵpos2\\displaystyle t\_\{k\}\\leq\\frac\{100\(H\+1\)^\{4\}SA\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\+t\_\{\\kappa\-1\}\+\\frac\{100\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\+\\frac\{100\(H\+1\)^\{4\}S^\{2\}A\\log t\_\{k\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}⇒Lemma[38](https://arxiv.org/html/2605.23182#Thmtheorem38)\\displaystyle\\stackrel\{\{\\scriptstyle\\text\{Lemma \}\\ref\{lemma:inequality\-t\-logt\}\}\}\{\{\\Rightarrow\}\}tk≤100​\(H\+1\)4​S​A​log⁡2​S​A​Hδkϵpos2\+tκ−1\+100​\(H\+1\)4​S2​Aϵpos2\+\\displaystyle t\_\{k\}\\leq\\frac\{100\(H\+1\)^\{4\}SA\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\+t\_\{\\kappa\-1\}\+\\frac\{100\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\+300​\(H\+1\)4​S2​Aϵpos2​log⁡\(100​\(H\+1\)4​S​A​log⁡2​S​A​Hδkϵpos2\+tκ−1\+100​\(H\+1\)4​S2​Aϵpos2\)\\displaystyle\\frac\{300\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\log\\left\(\\frac\{100\(H\+1\)^\{4\}SA\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\+t\_\{\\kappa\-1\}\+\\frac\{100\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\right\)⇒\\displaystyle\\Rightarrowtk≤100​\(H\+1\)4​S​A​log⁡2​S​A​Hδkϵpos2\+tκ−1\+\\displaystyle t\_\{k\}\\leq\\frac\{100\(H\+1\)^\{4\}SA\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\+t\_\{\\kappa\-1\}\+400​\(H\+1\)4​S2​Aϵpos2​log⁡\(200​\(H\+1\)4​S2​A​log⁡2​S​A​Hδkϵpos2\+tκ−1\)\\displaystyle\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\log\\left\(\\frac\{200\(H\+1\)^\{4\}S^\{2\}A\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\+t\_\{\\kappa\-1\}\\right\)⇒Lemma[6](https://arxiv.org/html/2605.23182#Thmtheorem6)\\displaystyle\\stackrel\{\{\\scriptstyle\\text\{Lemma \}\\ref\{lemma:length\-of\-exploration\-exploitation\-period\}\}\}\{\{\\Rightarrow\}\}tk≤100​\(H\+1\)4​S​A​log⁡2​S​A​Hδkϵpos2\+Tκ−1ee\+1\\displaystyle t\_\{k\}\\leq\\frac\{100\(H\+1\)^\{4\}SA\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\+T\_\{\\kappa\-1\}^\{\\text\{ee\}\}\+1400​\(H\+1\)4​S2​Aϵpos2​log⁡\(200​\(H\+1\)4​S2​A​log⁡2​S​A​Hδkϵpos2\+Tκ−1ee\+1\)\\displaystyle\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\log\\left\(\\frac\{200\(H\+1\)^\{4\}S^\{2\}A\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\+T\_\{\\kappa\-1\}^\{\\text\{ee\}\}\+1\\right\)⇒Lemma[39](https://arxiv.org/html/2605.23182#Thmtheorem39)\\displaystyle\\stackrel\{\{\\scriptstyle\\text\{Lemma \}\\ref\{lemma:log\_a\_plus\_b\-upper\_bound\}\}\}\{\{\\Rightarrow\}\}tk≤101​\(H\+1\)4​S​A​log⁡2​S​A​Hδkϵpos2\+Tκ−1ee\+\\displaystyle t\_\{k\}\\leq\\frac\{101\(H\+1\)^\{4\}SA\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\+T\_\{\\kappa\-1\}^\{\\text\{ee\}\}\+400​\(H\+1\)4​S2​Aϵpos2​log⁡\(402​\(H\+1\)4​S2​A​log⁡2​S​A​Hδkϵpos2\)\+400​\(H\+1\)4​S2​Aϵpos2​log⁡\(2​Tκ−1ee\)\.\\displaystyle\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\log\\left\(\\frac\{402\(H\+1\)^\{4\}S^\{2\}A\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\right\)\+\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\log\\left\(2T\_\{\\kappa\-1\}^\{\\text\{ee\}\}\\right\)\.For phase indexk≥max⁡\{κ,⌈log2⁡9600​\(H\+1\)2ϵpos2⌉\}k\\geq\\max\\\{\\kappa,\\lceil\\log\_\{2\}\\frac\{9600\(H\+1\)^\{2\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\rceil\\\}, by the Lemma[21](https://arxiv.org/html/2605.23182#Thmtheorem21), we can conclude

Tkee\\displaystyle T\_\{k\}^\{\\text\{ee\}\}≥\\displaystyle\\geq101​\(H\+1\)4​S​A​log⁡2​S​A​Hδkϵpos2\+Tk−1ee\+\\displaystyle\\frac\{101\(H\+1\)^\{4\}SA\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\+T\_\{k\-1\}^\{\\text\{ee\}\}\+400​\(H\+1\)4​S2​Aϵpos2​log⁡\(402​\(H\+1\)4​S2​A​log⁡2​S​A​Hδkϵpos2\)\+400​\(H\+1\)4​S2​Aϵpos2​log⁡\(2​Tk−1ee\)\\displaystyle\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\log\\left\(\\frac\{402\(H\+1\)^\{4\}S^\{2\}A\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\right\)\+\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\log\\left\(2T\_\{k\-1\}^\{\\text\{ee\}\}\\right\)≥\\displaystyle\\geq101​\(H\+1\)4​S​A​log⁡2​S​A​Hδkϵpos2\+Tκ−1ee\+\\displaystyle\\frac\{101\(H\+1\)^\{4\}SA\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\+T\_\{\\kappa\-1\}^\{\\text\{ee\}\}\+400​\(H\+1\)4​S2​Aϵpos2​log⁡\(402​\(H\+1\)4​S2​A​log⁡2​S​A​Hδkϵpos2\)\+400​\(H\+1\)4​S2​Aϵpos2​log⁡\(2​Tκ−1ee\)\.\\displaystyle\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\log\\left\(\\frac\{402\(H\+1\)^\{4\}S^\{2\}A\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\right\)\+\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon^\{\\text\{pos\}\}\}^\{2\}\}\\log\\left\(2T\_\{\\kappa\-1\}^\{\\text\{ee\}\}\\right\)\.That means Algorithm[2](https://arxiv.org/html/2605.23182#alg2)will fulfill \([3](https://arxiv.org/html/2605.23182#S3.E3)\) and quit the loop at Line[3](https://arxiv.org/html/2605.23182#alg2.l3)\. By the Lemma[13](https://arxiv.org/html/2605.23182#Thmtheorem13), we can conclude Algorithm[2](https://arxiv.org/html/2605.23182#alg2)will output a policyπ^k\\hat\{\\pi\}\_\{k\}satisfyingV0π^k​\(s0\)−μ0≥C−1C​\(V0∗​\(s0\)−μ0\)V^\{\\hat\{\\pi\}\_\{k\}\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\\geq\\frac\{C\-1\}\{C\}\\Big\(V^\{\*\}\_\{0\}\(s\_\{0\}\)\-\\mu\_\{0\}\\Big\)at the end of phasekk\.

For the case of negative instance, the analysis is nearly the same\. We can replaceϵpos\\epsilon^\{\\text\{pos\}\}withϵneg=μ0−V0∗​\(s0\)\\epsilon^\{\\text\{neg\}\}=\\mu\_\{0\}\-V^\{\*\}\_\{0\}\(s\_\{0\}\)\. By the Lemma[13](https://arxiv.org/html/2605.23182#Thmtheorem13), we complete the proof\. ∎

The upcoming Lemma[21](https://arxiv.org/html/2605.23182#Thmtheorem21)shows the minimum phase index for sufficiently largeTkeeT\_\{k\}^\{\\text\{ee\}\}\.\.

###### Lemma 21\.

For anyϵ\>0\\epsilon\>0,k≥⌈log2⁡9600​\(H\+1\)2ϵ2⌉k\\geq\\lceil\\log\_\{2\}\\frac\{9600\(H\+1\)^\{2\}\}\{\\epsilon^\{2\}\}\\rceilimplies

Tkee\\displaystyle T\_\{k\}^\{\\text\{ee\}\}≥\\displaystyle\\geq101​\(H\+1\)4​S​A​log⁡2​S​A​Hδkϵ2\+Tk−1ee\+\\displaystyle\\frac\{101\(H\+1\)^\{4\}SA\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon\}^\{2\}\}\+T\_\{k\-1\}^\{\\text\{ee\}\}\+400​\(H\+1\)4​S2​Aϵ2​log⁡\(402​\(H\+1\)4​S2​A​log⁡2​S​A​Hδkϵ2\)\+400​\(H\+1\)4​S2​Aϵ2​log⁡\(2​Tk−1ee\)\.\\displaystyle\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon\}^\{2\}\}\\log\\left\(\\frac\{402\(H\+1\)^\{4\}S^\{2\}A\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon\}^\{2\}\}\\right\)\+\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon\}^\{2\}\}\\log\\left\(2T\_\{k\-1\}^\{\\text\{ee\}\}\\right\)\.

###### Proof\.

We can validate the lemma by conducting direct calculation\. Recall the definition ofTkeeT\_\{k\}^\{\\text\{ee\}\},

Tkee=\(H\+1\)2​S​A​log⁡2​S​A​Hδkϵk\+\(H\+1\)2​S2​Aϵk​log⁡\(\(H\+1\)2​S2​A​log⁡2​S​A​Hδkϵk\),\\displaystyle T\_\{k\}^\{\\text\{ee\}\}=\\frac\{\(H\+1\)^\{2\}SA\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon\_\{k\}\}\}\+\\frac\{\(H\+1\)^\{2\}S^\{2\}A\}\{\\epsilon\_\{k\}\}\\log\\left\(\\frac\{\(H\+1\)^\{2\}S^\{2\}A\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\\epsilon\_\{k\}\}\\right\),and we takeϵk=12k\\epsilon\_\{k\}=\\frac\{1\}\{2^\{k\}\}\. We can concludek≥⌈log2⁡9600​\(H\+1\)2ϵ2⌉k\\geq\\lceil\\log\_\{2\}\\frac\{9600\(H\+1\)^\{2\}\}\{\\epsilon^\{2\}\}\\rceilimplies

Tkee≥404​\(H\+1\)4​S​A​log⁡2​S​A​Hδkϵ2\+3200​\(H\+1\)4​S2​Aϵ2​log⁡\(9600​\(H\+1\)4​S2​A​log⁡2​S​A​Hδkϵ2\)\.\\displaystyle T\_\{k\}^\{\\text\{ee\}\}\\geq\\frac\{404\(H\+1\)^\{4\}SA\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon\}^\{2\}\}\+\\frac\{3200\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon\}^\{2\}\}\\log\\left\(\\frac\{9600\(H\+1\)^\{4\}S^\{2\}A\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon\}^\{2\}\}\\right\)\.We can further conclude

14​Tkee≥\\displaystyle\\frac\{1\}\{4\}T\_\{k\}^\{\\text\{ee\}\}\\geq101​\(H\+1\)4​S​A​log⁡2​S​A​Hδkϵ2\+400​\(H\+1\)4​S2​Aϵ2​log⁡\(402​\(H\+1\)4​S2​A​log⁡2​S​A​Hδkϵ2\)\\displaystyle\\frac\{101\(H\+1\)^\{4\}SA\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon\}^\{2\}\}\+\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon\}^\{2\}\}\\log\\left\(\\frac\{402\(H\+1\)^\{4\}S^\{2\}A\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon\}^\{2\}\}\\right\)\(45\)2​Tkee≥\\displaystyle 2T\_\{k\}^\{\\text\{ee\}\}\\geq2⋅3200​\(H\+1\)4​S2​Aϵ2​log⁡\(3⋅3200​\(H\+1\)4​S2​Aϵ2\)\\displaystyle\\frac\{2\\cdot 3200\(H\+1\)^\{4\}S^\{2\}A\}\{\\epsilon^\{2\}\}\\log\\left\(\\frac\{3\\cdot 3200\(H\+1\)^\{4\}S^\{2\}A\}\{\\epsilon^\{2\}\}\\right\)\(46\)By the Lemma[40](https://arxiv.org/html/2605.23182#Thmtheorem40), \([46](https://arxiv.org/html/2605.23182#A2.E46)\) implies

2​Tkee≥2⋅3200​\(H\+1\)4​S2​Aϵ2​log⁡\(3⋅3200​\(H\+1\)4​S2​Aϵ2\)\\displaystyle 2T\_\{k\}^\{\\text\{ee\}\}\\geq\\frac\{2\\cdot 3200\(H\+1\)^\{4\}S^\{2\}A\}\{\\epsilon^\{2\}\}\\log\\left\(\\frac\{3\\cdot 3200\(H\+1\)^\{4\}S^\{2\}A\}\{\\epsilon^\{2\}\}\\right\)⇒\\displaystyle\\Rightarrow2​Tkee≥3200​\(H\+1\)4​S2​Aϵ2​log⁡\(2​Tkee\)\\displaystyle 2T\_\{k\}^\{\\text\{ee\}\}\\geq\\frac\{3200\(H\+1\)^\{4\}S^\{2\}A\}\{\\epsilon^\{2\}\}\\log\\left\(2T\_\{k\}^\{\\text\{ee\}\}\\right\)⇒\\displaystyle\\Rightarrow14​Tkee≥400​\(H\+1\)4​S2​Aϵ2​log⁡\(2​Tk−1ee\)\.\\displaystyle\\frac\{1\}\{4\}T\_\{k\}^\{\\text\{ee\}\}\\geq\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon\}^\{2\}\}\\log\\left\(2T\_\{k\-1\}^\{\\text\{ee\}\}\\right\)\.\(47\)Meanwhile,ϵk=12k\\epsilon\_\{k\}=\\frac\{1\}\{2^\{k\}\}implies

12​Tkee≥Tk−1ee\.\\displaystyle\\frac\{1\}\{2\}T\_\{k\}^\{\\text\{ee\}\}\\geq T\_\{k\-1\}^\{\\text\{ee\}\}\.\(48\)Sum up \([45](https://arxiv.org/html/2605.23182#A2.E45)\),\([47](https://arxiv.org/html/2605.23182#A2.E47)\) and \([48](https://arxiv.org/html/2605.23182#A2.E48)\), we have

Tkee\\displaystyle T\_\{k\}^\{\\text\{ee\}\}≥\\displaystyle\\geq101​\(H\+1\)4​S​A​log⁡2​S​A​Hδkϵ2\+Tk−1ee\+\\displaystyle\\frac\{101\(H\+1\)^\{4\}SA\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon\}^\{2\}\}\+T\_\{k\-1\}^\{\\text\{ee\}\}\+400​\(H\+1\)4​S2​Aϵ2​log⁡\(402​\(H\+1\)4​S2​A​log⁡2​S​A​Hδkϵ2\)\+400​\(H\+1\)4​S2​Aϵ2​log⁡\(2​Tk−1ee\)\.\\displaystyle\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon\}^\{2\}\}\\log\\left\(\\frac\{402\(H\+1\)^\{4\}S^\{2\}A\\log\\frac\{2SAH\}\{\\delta\_\{k\}\}\}\{\{\\epsilon\}^\{2\}\}\\right\)\+\\frac\{400\(H\+1\)^\{4\}S^\{2\}A\}\{\{\\epsilon\}^\{2\}\}\\log\\left\(2T\_\{k\-1\}^\{\\text\{ee\}\}\\right\)\.∎

### B\.4Performance Guarantee of Exploitation Stage

In this subsection, we present the performance Guarantee of the exploitation stage, i\.e\. Line[11](https://arxiv.org/html/2605.23182#alg1.l11)to[17](https://arxiv.org/html/2605.23182#alg1.l17)in Algorithm[1](https://arxiv.org/html/2605.23182#alg1)\.

The following Lemma shows the minimum phase index for sufficiently largeTketT\_\{k\}^\{\\text\{et\}\}\.

###### Lemma 22\.

k≥⌈log2⁡H2ω2​\(V0∗​\(s0\)−μ0\)2⌉\\displaystyle k\\geq\\lceil\\log\_\{2\}\\frac\{H^\{2\}\}\{\\omega^\{2\}\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\rceil⇒\\displaystyle\\Rightarrow100⋅log⁡αkδ\+log⁡log⁡1ϵkϵk≥28​H2​log⁡αkδ\+16​H2​log⁡\(log⁡\(24​H2ω2​\(V0∗​\(s0\)−μ0\)2\)\)ω2​\(V0∗​\(s0\)−μ0\)2,\\displaystyle 100\\cdot\\frac\{\\log\\frac\{\\alpha\_\{k\}\}\{\\delta\}\+\\log\\log\\frac\{1\}\{\\epsilon\_\{k\}\}\}\{\\epsilon\_\{k\}\}\\geq\\frac\{28H^\{2\}\\log\\frac\{\\alpha\_\{k\}\}\{\\delta\}\+16H^\{2\}\\log\\left\(\\log\\left\(\\frac\{24H^\{2\}\}\{\\omega^\{2\}\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\\right\)\}\{\\omega^\{2\}\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\},whereω\\omegais an arbitrary constant in\(0,1\)\(0,1\)\.

###### Proof\.

We can validate Lemma[22](https://arxiv.org/html/2605.23182#Thmtheorem22)by straight forward calculation\.

k≥⌈log2⁡H2ω2​\(V0∗​\(s0\)−μ0\)2⌉\\displaystyle k\\geq\\lceil\\log\_\{2\}\\frac\{H^\{2\}\}\{\\omega^\{2\}\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\rceil⇒\\displaystyle\\Rightarrow1ϵk≥H2ω2​\(V0∗​\(s0\)−μ0\)2\\displaystyle\\frac\{1\}\{\\epsilon\_\{k\}\}\\geq\\frac\{H^\{2\}\}\{\\omega^\{2\}\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}⇒\\displaystyle\\Rightarrow100⋅log⁡αkδ\+log⁡log⁡1ϵkϵk\>28​H2​log⁡αkδ\+16​H2​log⁡\(log⁡\(24​H2ω2​\(V0∗​\(s0\)−μ0\)2\)\)ω2​\(V0∗​\(s0\)−μ0\)2\\displaystyle 100\\cdot\\frac\{\\log\\frac\{\\alpha\_\{k\}\}\{\\delta\}\+\\log\\log\\frac\{1\}\{\\epsilon\_\{k\}\}\}\{\\epsilon\_\{k\}\}\>\\frac\{28H^\{2\}\\log\\frac\{\\alpha\_\{k\}\}\{\\delta\}\+16H^\{2\}\\log\\left\(\\log\\left\(\\frac\{24H^\{2\}\}\{\\omega^\{2\}\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\\right\)\}\{\\omega^\{2\}\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}∎

The upcoming Lemma[23](https://arxiv.org/html/2605.23182#Thmtheorem23),[24](https://arxiv.org/html/2605.23182#Thmtheorem24)are to prove sufficient conditions that Algorithm[1](https://arxiv.org/html/2605.23182#alg1)will accept a candidate policyπ^k\\hat\{\\pi\}\_\{k\}\.

###### Lemma 23\.

Given a policyπ^\\hat\{\\pi\}, denoteV^0π^,N\\hat\{V\}\_\{0\}^\{\\hat\{\\pi\},N\}as the empirical estimation ofV0π^​\(s0\)V\_\{0\}^\{\\hat\{\\pi\}\}\(s\_\{0\}\)by executingπ^\\hat\{\\pi\}NNtimes, we have

Pr⁡\(∀N∈ℕ,\|V^0π^,N−V0π^​\(s0\)\|<H2​log⁡2​\(log2⁡2​N\)2δN\)≥1−π26​δ\\displaystyle\\Pr\\left\(\\forall N\\in\\mathbb\{N\},\|\\hat\{V\}\_\{0\}^\{\\hat\{\\pi\},N\}\-V\_\{0\}^\{\\hat\{\\pi\}\}\(s\_\{0\}\)\|<\\sqrt\{\\frac\{H^\{2\}\\log\\frac\{2\(\\log\_\{2\}2N\)^\{2\}\}\{\\delta\}\}\{N\}\}\\right\)\\geq 1\-\\frac\{\\pi^\{2\}\}\{6\}\\deltaholds for allδ\>0\\delta\>0\.

###### Proof\.

DenoteXiX\_\{i\}as the total random reward collected by executing the policyπ^\\hat\{\\pi\}iniith time\. We haveV^0π^,N=∑i=1NXiN\\hat\{V\}\_\{0\}^\{\\hat\{\\pi\},N\}=\\frac\{\\sum\_\{i=1\}^\{N\}X\_\{i\}\}\{N\}\. From the assumption that the reward is always bounded in\[0,1\]\[0,1\], we knowXiX\_\{i\}isH24\\frac\{H^\{2\}\}\{4\}\-subgaussian\. By the Lemma[43](https://arxiv.org/html/2605.23182#Thmtheorem43), we complete the proof\. ∎

###### Lemma 24\.

Consider a positive instance withV0∗​\(s0\)\>μ0V\_\{0\}^\{\*\}\(s\_\{0\}\)\>\\mu\_\{0\}\. Assume policyπ^k\\hat\{\\pi\}\_\{k\}satisfiesV0π^k​\(s0\)−μ0≥ω​\(V0∗​\(s0\)−μ0\)V\_\{0\}^\{\\hat\{\\pi\}\_\{k\}\}\(s\_\{0\}\)\-\\mu\_\{0\}\\geq\\omega\\Big\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\\Big\)for someω\>0\\omega\>0andk≥⌈log2⁡H2ω2​\(V0∗​\(s0\)−μ0\)2⌉k\\geq\\lceil\\log\_\{2\}\\frac\{H^\{2\}\}\{\\omega^\{2\}\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\rceil\. If the concentration event

ℰket=\{∀N∈ℕ,\|V^0π^k,N−V0π^k​\(s0\)\|<H2​log⁡2​αk​\(log2⁡2​N\)2δN\}\\displaystyle\\mathcal\{E\}\_\{k\}^\{\\text\{et\}\}=\\left\\\{\\forall N\\in\\mathbb\{N\},\|\\hat\{V\}\_\{0\}^\{\\hat\{\\pi\}\_\{k\},N\}\-V\_\{0\}^\{\\hat\{\\pi\}\_\{k\}\}\(s\_\{0\}\)\|<\\sqrt\{\\frac\{H^\{2\}\\log\\frac\{2\\alpha\_\{k\}\(\\log\_\{2\}2N\)^\{2\}\}\{\\delta\}\}\{N\}\}\\right\\\}holds, whereV^0π^k,N\\hat\{V\}\_\{0\}^\{\\hat\{\\pi\}\_\{k\},N\}is the empirical mean value ofV0π^k​\(s0\)V\_\{0\}^\{\\hat\{\\pi\}\_\{k\}\}\(s\_\{0\}\)by executingπ^k\\hat\{\\pi\}\_\{k\}NNtimes, Line[11](https://arxiv.org/html/2605.23182#alg1.l11)to[17](https://arxiv.org/html/2605.23182#alg1.l17)in Algorithm[1](https://arxiv.org/html/2605.23182#alg1)will enter Line[15](https://arxiv.org/html/2605.23182#alg1.l15)and outputπ^k\\hat\{\\pi\}\_\{k\}\.

###### Proof\.

Given the eventℰket\\mathcal\{E\}\_\{k\}^\{\\text\{et\}\}, we have

V^0π^k,N−H2​log⁡2​αk​\(log2⁡2​N\)2δN<V0π^k​\(s0\)<V^0π^k,N\+H2​log⁡2​αk​\(log2⁡2​N\)2δN\\displaystyle\\hat\{V\}\_\{0\}^\{\\hat\{\\pi\}\_\{k\},N\}\-\\sqrt\{\\frac\{H^\{2\}\\log\\frac\{2\\alpha\_\{k\}\(\\log\_\{2\}2N\)^\{2\}\}\{\\delta\}\}\{N\}\}<V\_\{0\}^\{\\hat\{\\pi\}\_\{k\}\}\(s\_\{0\}\)<\\hat\{V\}\_\{0\}^\{\\hat\{\\pi\}\_\{k\},N\}\+\\sqrt\{\\frac\{H^\{2\}\\log\\frac\{2\\alpha\_\{k\}\(\\log\_\{2\}2N\)^\{2\}\}\{\\delta\}\}\{N\}\}holds for allN∈ℕN\\in\\mathbb\{N\}\. Thus, we can conclude

⇒\\displaystyle\\RightarrowN≥28​H2​log⁡αkδ\+16​H2​log⁡\(log⁡\(24​H2ω2​\(V0∗​\(s0\)−μ0\)2\)\)ω2​\(V0∗​\(s0\)−μ0\)2\\displaystyle N\\geq\\frac\{28H^\{2\}\\log\\frac\{\\alpha\_\{k\}\}\{\\delta\}\+16H^\{2\}\\log\\left\(\\log\\left\(\\frac\{24H^\{2\}\}\{\\omega^\{2\}\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\\right\)\}\{\\omega^\{2\}\(V\_\{0\}^\{\*\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}⇒\\displaystyle\\RightarrowN≥28​H2​log⁡αkδ\+16​H2​log⁡\(log⁡\(24​H2\(V0π^k​\(s0\)−μ0\)2\)\)\(V0π^k​\(s0\)−μ0\)2\\displaystyle N\\geq\\frac\{28H^\{2\}\\log\\frac\{\\alpha\_\{k\}\}\{\\delta\}\+16H^\{2\}\\log\\left\(\\log\\left\(\\frac\{24H^\{2\}\}\{\(V\_\{0\}^\{\\hat\{\\pi\}\_\{k\}\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\\right\)\}\{\(V\_\{0\}^\{\\hat\{\\pi\}\_\{k\}\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}⇒\\displaystyle\\RightarrowV0π^k​\(s0\)−2​H2​log⁡2​αk​\(log2⁡2​N\)2δN\>μ0\\displaystyle V\_\{0\}^\{\\hat\{\\pi\}\_\{k\}\}\(s\_\{0\}\)\-2\\sqrt\{\\frac\{H^\{2\}\\log\\frac\{2\\alpha\_\{k\}\(\\log\_\{2\}2N\)^\{2\}\}\{\\delta\}\}\{N\}\}\>\\mu\_\{0\}⇒\\displaystyle\\RightarrowV^0π^k,N−H2​log⁡2​αk​\(log2⁡2​N\)2δN\>μ0\\displaystyle\\hat\{V\}\_\{0\}^\{\\hat\{\\pi\}\_\{k\},N\}\-\\sqrt\{\\frac\{H^\{2\}\\log\\frac\{2\\alpha\_\{k\}\(\\log\_\{2\}2N\)^\{2\}\}\{\\delta\}\}\{N\}\}\>\\mu\_\{0\}The second last step is by the Lemma[42](https://arxiv.org/html/2605.23182#Thmtheorem42)and the last step is by the eventℰket\\mathcal\{E\}\_\{k\}^\{\\text\{et\}\}\. Meanwhile, by the Lemma[22](https://arxiv.org/html/2605.23182#Thmtheorem22), we know100​H2⋅log⁡αkδ\+log⁡log⁡1ϵkϵk\>28​H2​log⁡αkδ\+16​H2​log⁡\(log⁡\(24​H2\(V0π^k​\(s0\)−μ0\)2\)\)\(V0π^k​\(s0\)−μ0\)2100H^\{2\}\\cdot\\frac\{\\log\\frac\{\\alpha\_\{k\}\}\{\\delta\}\+\\log\\log\\frac\{1\}\{\\epsilon\_\{k\}\}\}\{\\epsilon\_\{k\}\}\>\\frac\{28H^\{2\}\\log\\frac\{\\alpha\_\{k\}\}\{\\delta\}\+16H^\{2\}\\log\\left\(\\log\\left\(\\frac\{24H^\{2\}\}\{\(V\_\{0\}^\{\\hat\{\\pi\}\_\{k\}\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\\right\)\}\{\(V\_\{0\}^\{\\hat\{\\pi\}\_\{k\}\}\(s\_\{0\}\)\-\\mu\_\{0\}\)^\{2\}\}, that means the Algorithm[1](https://arxiv.org/html/2605.23182#alg1)will enter Line[15](https://arxiv.org/html/2605.23182#alg1.l15)and outputπ^k\\hat\{\\pi\}\_\{k\}\.\. ∎

## Appendix CProof of Lower Bound

### C\.1Full Definition of Instances

In this subsection, we present the full definition of Uniform Instance and Tree Instance\.

###### Definition 8\(Uniform Instance specified by\(S,A,H,r,ϵ\)\(S,A,H,r,\\epsilon\)\)\.

Given integersS,H≥10S,H\\geq 10,A\>100A\>100,HHis even,\(S−2\)​H\(S\-2\)His divisible by1616,0<ϵ<1200​50<\\epsilon<\\frac\{1\}\{200\\sqrt\{5\}\},14<r<r\+ϵ<34\\frac\{1\}\{4\}<r<r\+\\epsilon<\\frac\{3\}\{4\}, we define𝒮=\[S−2\]∪\{sgood,sbad\},𝒜=\[A\],ℋ=\[H\]\\mathcal\{S\}=\[S\-2\]\\cup\\\{s\_\{\\text\{good\},s\_\{\\text\{bad\}\}\}\\\},\\mathcal\{A\}=\[A\],\\mathcal\{H\}=\[H\]\. Define a Meaningless instanceν\\nuwith reward functionrh​\(s,a\)=0,∀\(h,s,a\)∈\[H\]×\[S−2\]×𝒜r\_\{h\}\(s,a\)=0,\\forall\(h,s,a\)\\in\[H\]\\times\[S\-2\]\\times\\mathcal\{A\},rh​\(sgood,a\)=1,∀\(h,a\)∈\[H\]×𝒜r\_\{h\}\(s\_\{\\text\{good\}\},a\)=1,\\forall\(h,a\)\\in\[H\]\\times\\mathcal\{A\},rh​\(sbad,a\)=0,∀\(h,a\)∈\[H\]×𝒜r\_\{h\}\(s\_\{\\text\{bad\}\},a\)=0,\\forall\(h,a\)\\in\[H\]\\times\\mathcal\{A\}; initial distributionp0​\(s\)=1S−2,∀s∈\[S−2\]p\_\{0\}\(s\)=\\frac\{1\}\{S\-2\},\\forall s\\in\[S\-2\],p0​\(s\)=0,∀s∈\{sgood,sbad\}p\_\{0\}\(s\)=0,\\forall s\\in\\\{s\_\{\\text\{good\}\},s\_\{\\text\{bad\}\}\\\}; transition probabilityph​\(s′\|s,1\)=\{r\+ϵs′=sgood1−r−ϵs′=sbad0e​l​s​e,∀\(h,s\)∈\[H\]×\[S−2\]p\_\{h\}\(s^\{\\prime\}\|s,1\)=\\begin\{cases\}r\+\\epsilon&s^\{\\prime\}=s\_\{\\text\{good\}\}\\\\ 1\-r\-\\epsilon&s^\{\\prime\}=s\_\{\\text\{bad\}\}\\\\ 0&else\\end\{cases\},\\forall\(h,s\)\\in\[H\]\\times\[S\-2\];ph​\(s′\|s,a\)=\{rs′=sgood1−rs′=sbad0e​l​s​e,∀\(h,s\)∈\[H\]×\[S−2\],a∈\{2,3,⋯,A\}p\_\{h\}\(s^\{\\prime\}\|s,a\)=\\begin\{cases\}r&s^\{\\prime\}=s\_\{\\text\{good\}\}\\\\ 1\-r&s^\{\\prime\}=s\_\{\\text\{bad\}\}\\\\ 0&else\\end\{cases\},\\forall\(h,s\)\\in\[H\]\\times\[S\-2\],a\\in\\\{2,3,\\cdots,A\\\};ph​\(s′\|s,a\)=\{1S−2s′∈\[S−2\]0e​l​s​e,∀h∈\[H\],s∈\{sgood,sbad\},a∈\{1,2,3,⋯,A\}p\_\{h\}\(s^\{\\prime\}\|s,a\)=\\begin\{cases\}\\frac\{1\}\{S\-2\}&s^\{\\prime\}\\in\[S\-2\]\\\\ 0&else\\end\{cases\},\\forall h\\in\[H\],s\\in\\\{s\_\{\\text\{good\},s\_\{\\text\{bad\}\}\}\\\},a\\in\\\{1,2,3,\\cdots,A\\\}

###### Definition 9\(Tree Instance specified by\(S,A,H,r\)\(S,A,H,r\)\)\.

Consider integerS,A,HS,A,Hsuch thatS−1=2NS\-1=2^\{N\}for some integerN≥1N\\geq 1, andA≥3A\\geq 3,H−1≥6​N=6​log2⁡\(S−1\)H\-1\\geq 6N=6\\log\_\{2\}\(S\-1\), and a real valuer∈\(38,58\)r\\in\(\\frac\{3\}\{8\},\\frac\{5\}\{8\}\)\. We consider a binary tree, with state space𝒮=\[2N−1\]∪\{sgood,sbad\}\\mathcal\{S\}=\[2^\{N\}\-1\]\\cup\\\{s\_\{\\text\{good\}\},s\_\{\\text\{bad\}\}\\\}\. The transition probability is

ph​\(2​s\|s,a=1\)=\\displaystyle p\_\{h\}\(2s\|s,a=1\)=1,∀s≤2N−1−1,∀h∈\[H\]\\displaystyle 1,\\forall s\\leq 2^\{N\-1\}\-1,\\forall h\\in\[H\]ph​\(2​s\+1\|s,a=2\)=\\displaystyle p\_\{h\}\(2s\+1\|s,a=2\)=1,∀s≤2N−1−1,∀h∈\[H\]\\displaystyle 1,\\forall s\\leq 2^\{N\-1\}\-1,\\forall h\\in\[H\]ph​\(sgood\|s,a≥3\)=\\displaystyle p\_\{h\}\(s\_\{\\text\{good\}\}\|s,a\\geq 3\)=r,∀s≤2N−1−1,∀h∈\[H\]\\displaystyle r,\\forall s\\leq 2^\{N\-1\}\-1,\\forall h\\in\[H\]ph​\(sbad\|s,a≥3\)=\\displaystyle p\_\{h\}\(s\_\{\\text\{bad\}\}\|s,a\\geq 3\)=1−r,∀s≤2N−1−1,∀h∈\[H\]\\displaystyle 1\-r,\\forall s\\leq 2^\{N\-1\}\-1,\\forall h\\in\[H\]ph​\(sgood\|s,a\)=\\displaystyle p\_\{h\}\(s\_\{\\text\{good\}\}\|s,a\)=r,∀2N−1≥s≥2N−1,∀a∈\[A\],∀h∈\[H\]\\displaystyle r,\\forall 2^\{N\}\-1\\geq s\\geq 2^\{N\-1\},\\forall a\\in\[A\],\\forall h\\in\[H\]ph​\(sbad\|s,a\)=\\displaystyle p\_\{h\}\(s\_\{\\text\{bad\}\}\|s,a\)=1−r,∀2N−1≥s≥2N−1,∀a∈\[A\],∀h∈\[H\]\\displaystyle 1\-r,\\forall 2^\{N\}\-1\\geq s\\geq 2^\{N\-1\},\\forall a\\in\[A\],\\forall h\\in\[H\]ph​\(sgood\|sgood,a\)=\\displaystyle p\_\{h\}\(s\_\{\\text\{good\}\}\|s\_\{\\text\{good\}\},a\)=1,∀h∈\[H\],∀a∈\[A\]\\displaystyle 1,\\forall h\\in\[H\],\\forall a\\in\[A\]ph​\(sbad\|sbad,a\)=\\displaystyle p\_\{h\}\(s\_\{\\text\{bad\}\}\|s\_\{\\text\{bad\}\},a\)=1,∀h∈\[H\],∀a∈\[A\]\\displaystyle 1,\\forall h\\in\[H\],\\forall a\\in\[A\]Assume we always start at the state 1, i\.e\.Prν⁡\(s1π=1\)=1\\Pr\_\{\\nu\}\(s\_\{1\}^\{\\pi\}=1\)=1holds for any policyπ\\pi\. The reward function isrh​\(s,a\)=0,∀s∈\[2N−1\]∪\{sbad\}r\_\{h\}\(s,a\)=0,\\forall s\\in\[2^\{N\}\-1\]\\cup\\\{s\_\{\\text\{bad\}\}\\\};rh​\(sgood,a\)=1,∀h∈\[H\],a∈\[A\]r\_\{h\}\(s\_\{\\text\{good\}\},a\)=1,\\forall h\\in\[H\],a\\in\[A\]\.

### C\.2Proof of Lower Bound, for positive instances

Before illustrating the proof of Theorem[4](https://arxiv.org/html/2605.23182#Thmtheorem4), we first introduce the definition of symmetric algorithm\.

###### Definition 10\(Symmetric Algorithm\)\.

Given a 1\-policy\-identification algorithm alg, an instanceν\\nu, two different permutationsσ′=\{σs,h′\}\(s,h\)∈𝒮×ℋ\\sigma^\{\\prime\}=\\\{\\sigma\_\{s,h\}^\{\\prime\}\\\}\_\{\(s,h\)\\in\\mathcal\{S\}\\times\\mathcal\{H\}\},σ′′=\{σs,h′′\}\(s,h\)∈𝒮×ℋ\\sigma^\{\\prime\\prime\}=\\\{\\sigma\_\{s,h\}^\{\\prime\\prime\}\\\}\_\{\(s,h\)\\in\\mathcal\{S\}\\times\\mathcal\{H\}\}\. We call the algorithm is symmetric, if for anyT∈ℕT\\in\\mathbb\{N\}, any realization path\{\(st,h,at,h\)h=1H\}t=1T\\\{\(s\_\{t,h\},a\_\{t,h\}\)\_\{h=1\}^\{H\}\\\}\_\{t=1\}^\{T\}up toTTepisodes, any output policy\{a​\(s,h\)\}\(s,h\)∈𝒮×ℋ\\\{a\(s,h\)\\\}\_\{\(s,h\)\\in\\mathcal\{S\}\\times\\mathcal\{H\}\}algorithm can fulfill

Pralg,νσ′⁡\(∀t∈\[T\],∀h∈\[H\],St,h=st,h,At,h=σst,h,h′​\(at,h\);∀s,h,τ=T,π^h​\(s\)=σs,h′​\(a​\(s,h\)\)\)\\displaystyle\\Pr\_\{\\text\{alg\},\\nu\_\{\\sigma^\{\\prime\}\}\}\\Big\(\\forall t\\in\[T\],\\forall h\\in\[H\],S\_\{t,h\}=s\_\{t,h\},A\_\{t,h\}=\\sigma\_\{s\_\{t,h\},h\}^\{\\prime\}\(a\_\{t,h\}\);\\forall s,h,\\tau=T,\\hat\{\\pi\}\_\{h\}\(s\)=\\sigma\_\{s,h\}^\{\\prime\}\(a\(s,h\)\)\\Big\)=\\displaystyle=Pralg,νσ′′⁡\(∀t∈\[T\],∀h∈\[H\],St,h=st,h,At,h=σst,h,h′′​\(at,h\);∀s,h,τ=T,π^h​\(s\)=σs,h′′​\(a​\(s,h\)\)\)\\displaystyle\\Pr\_\{\\text\{alg\},\\nu\_\{\\sigma^\{\\prime\\prime\}\}\}\\Big\(\\forall t\\in\[T\],\\forall h\\in\[H\],S\_\{t,h\}=s\_\{t,h\},A\_\{t,h\}=\\sigma\_\{s\_\{t,h\},h\}^\{\\prime\\prime\}\(a\_\{t,h\}\);\\forall s,h,\\tau=T,\\hat\{\\pi\}\_\{h\}\(s\)=\\sigma\_\{s,h\}^\{\\prime\\prime\}\(a\(s,h\)\)\\Big\)HereSt,h,At,hS\_\{t,h\},A\_\{t,h\}denote the state and action we take at roundhh, episodett\.

The following Theorem suggests that we suffice to consider symmetric algorithm for lower bounds

###### Theorem 25\.

DenoteΓ\\Gammaas the set of all the possible permutations onν\\nu\. For any instanceν\\nuand anyδ\\delta\-PAC alg, we can find aδ\\delta\-PAC and symmetricalg~\\tilde\{\\text\{alg\}\}, such thatmaxσ∈Γ⁡𝔼alg,νσ​τ≥𝔼alg~,ν​τ\\max\_\{\\sigma\\in\\Gamma\}\\mathbb\{E\}\_\{\\text\{alg\},\\nu\_\{\\sigma\}\}\\tau\\geq\\mathbb\{E\}\_\{\\tilde\{\\text\{alg\}\},\\nu\}\\tau\.

Given Theorem[25](https://arxiv.org/html/2605.23182#Thmtheorem25), we suffice to prove the following theorem to show Theorem[4](https://arxiv.org/html/2605.23182#Thmtheorem4)holds\.

###### Theorem 26\.

For anyδ\\delta\-PAC symmetric algorithm and any Uniform Instance specified by\(S,A,H,r,ϵ\)\(S,A,H,r,\\epsilon\), with3​H​\(r\+ϵ\)8\+H​r8\>μ0\>H​r2\\frac\{3H\(r\+\\epsilon\)\}\{8\}\+\\frac\{Hr\}\{8\}\>\\mu\_\{0\}\>\\frac\{Hr\}\{2\},3​H​\(r\+ϵ\)8\+H​r8\>μ0\>H​r2\\frac\{3H\(r\+\\epsilon\)\}\{8\}\+\\frac\{Hr\}\{8\}\>\\mu\_\{0\}\>\\frac\{Hr\}\{2\},S,H≥10S,H\\geq 10,A≥100A\\geq 100,\(S−2\)​H\(S\-2\)His divisible by1616,0<ϵ<1200​50<\\epsilon<\\frac\{1\}\{200\\sqrt\{5\}\}, it holds that

𝔼​τ≥Ω​\(H​log⁡1δ\(V0∗​\(s0\|ν\)−μ0\)2\+S​A​H2\(V0∗​\(s0\|ν\)−μ0\)2\)\\displaystyle\\mathbb\{E\}\\tau\\geq\\Omega\\left\(\\frac\{H\\log\\frac\{1\}\{\\delta\}\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\|\\nu\)\-\\mu\_\{0\}\)^\{2\}\}\+\\frac\{SAH^\{2\}\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\|\\nu\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)

We first present the proof of Theorem[25](https://arxiv.org/html/2605.23182#Thmtheorem25)\. After that, we split the proof of Theorem[26](https://arxiv.org/html/2605.23182#Thmtheorem26)into multiple steps\.

###### Proof of Theorem[25](https://arxiv.org/html/2605.23182#Thmtheorem25)\.

We define the symmetricalg~\\tilde\{\\text\{alg\}\}as follows\. We choose a permutationσ=\{σs,h\}\(s,h\)∈𝒮×ℋ\\sigma=\\\{\\sigma\_\{s,h\}\\\}\_\{\(s,h\)\\in\\mathcal\{S\}\\times\\mathcal\{H\}\}fromΓ\\Gammauniformly and randomly, before the sampling process starts\. Then apply the alg to the instanceνσ\\nu\_\{\\sigma\}to conduct the sampling process\.

We first validatealg~\\tilde\{\\text\{alg\}\}isδ−P​A​C\\delta\-PAC\. If the instanceν\\nuis positive, we have

Pralg~,ν⁡\(V0π^τ​\(s0\)≥μ0,τ<\+∞\)\\displaystyle\\Pr\_\{\\tilde\{\\text\{alg\}\},\\nu\}\(V\_\{0\}^\{\\hat\{\\pi\}\_\{\\tau\}\}\(s\_\{0\}\)\\geq\\mu\_\{0\},\\tau<\+\\infty\)=\\displaystyle=∑σ∈ΓPralg,νσ⁡\(V0π^τ​\(s0\)≥μ0,τ<\+∞\)\(A\!\)S​H\\displaystyle\\sum\_\{\\sigma\\in\\Gamma\}\\frac\{\\Pr\_\{\\text\{alg\},\\nu\_\{\\sigma\}\}\(V\_\{0\}^\{\\hat\{\\pi\}\_\{\\tau\}\}\(s\_\{0\}\)\\geq\\mu\_\{0\},\\tau<\+\\infty\)\}\{\(A\!\)^\{SH\}\}≥\\displaystyle\\geq∑σ∈Γ1−δ\(A\!\)S​H\\displaystyle\\sum\_\{\\sigma\\in\\Gamma\}\\frac\{1\-\\delta\}\{\(A\!\)^\{SH\}\}=\\displaystyle=1−δ\.\\displaystyle 1\-\\delta\.The idea applies to the case thatν\\nuis negative\.

Then we turn to showalg~\\tilde\{\\text\{alg\}\}is symmetric\. Given permutationσ′,σ′′\\sigma^\{\\prime\},\\sigma^\{\\prime\\prime\}, since we chooseσ∈Γ\\sigma\\in\\Gammain uniformly random manner, we know instances\(νσ′\)σ\{\(\\nu\_\{\\sigma^\{\\prime\}\}\)\}\_\{\\sigma\}and\(νσ′′\)σ\{\(\\nu\_\{\\sigma^\{\\prime\\prime\}\}\)\}\_\{\\sigma\}share the same distribution asνσ\\nu\_\{\\sigma\}\. Thus, we can conclude

Pralg,νσ′⁡\(∀t∈\[T\],∀h∈\[H\],St,h=st,h,At,h=σst,h,h′​\(at,h\);∀s,h,τ=T,π^h​\(s\)=σs,h′​\(a​\(s,h\)\)\)\\displaystyle\\Pr\_\{\\text\{alg\},\\nu\_\{\\sigma^\{\\prime\}\}\}\\Big\(\\forall t\\in\[T\],\\forall h\\in\[H\],S\_\{t,h\}=s\_\{t,h\},A\_\{t,h\}=\\sigma\_\{s\_\{t,h\},h\}^\{\\prime\}\(a\_\{t,h\}\);\\forall s,h,\\tau=T,\\hat\{\\pi\}\_\{h\}\(s\)=\\sigma\_\{s,h\}^\{\\prime\}\(a\(s,h\)\)\\Big\)=\\displaystyle=∑σ∈ΓPralg,νσ⁡\(∀t∈\[T\],∀h∈\[H\],St,h=st,h,At,h=σst,h,h​\(at,h\);∀s,h,τ=T,π^h​\(s\)=σs,h​\(a​\(s,h\)\)\)\(A\!\)S​H\\displaystyle\\frac\{\\sum\_\{\\sigma\\in\\Gamma\}\\Pr\_\{\\text\{alg\},\\nu\_\{\\sigma\}\}\\Big\(\\forall t\\in\[T\],\\forall h\\in\[H\],S\_\{t,h\}=s\_\{t,h\},A\_\{t,h\}=\\sigma\_\{s\_\{t,h\},h\}\(a\_\{t,h\}\);\\forall s,h,\\tau=T,\\hat\{\\pi\}\_\{h\}\(s\)=\\sigma\_\{s,h\}\(a\(s,h\)\)\\Big\)\}\{\(A\!\)^\{SH\}\}=\\displaystyle=Pralg,νσ′′⁡\(∀t∈\[T\],∀h∈\[H\],St,h=st,h,At,h=σst,h,h′′​\(at,h\);∀s,h,τ=T,π^h​\(s\)=σs,h′′​\(a​\(s,h\)\)\),\\displaystyle\\Pr\_\{\\text\{alg\},\\nu\_\{\\sigma^\{\\prime\\prime\}\}\}\\Big\(\\forall t\\in\[T\],\\forall h\\in\[H\],S\_\{t,h\}=s\_\{t,h\},A\_\{t,h\}=\\sigma\_\{s\_\{t,h\},h\}^\{\\prime\\prime\}\(a\_\{t,h\}\);\\forall s,h,\\tau=T,\\hat\{\\pi\}\_\{h\}\(s\)=\\sigma\_\{s,h\}^\{\\prime\\prime\}\(a\(s,h\)\)\\Big\),suggesting thatalg~\\tilde\{\\text\{alg\}\}is symmetric\.

The last step is to provemaxσ∈Γ⁡𝔼alg,νσ​τ≥𝔼alg~,ν​τ\\max\_\{\\sigma\\in\\Gamma\}\\mathbb\{E\}\_\{\\text\{alg\},\\nu\_\{\\sigma\}\}\\tau\\geq\\mathbb\{E\}\_\{\\tilde\{\\text\{alg\}\},\\nu\}\\tau\. From the definition of algorithmalg~\\tilde\{\\text\{alg\}\}, we know

𝔼alg~,ν​τ=∑σ∈Γ𝔼alg,νσ​τ\(A\!\)S​H≤maxσ∈Γ⁡𝔼alg,νσ​τ\.\\displaystyle\\mathbb\{E\}\_\{\\tilde\{\\text\{alg\}\},\\nu\}\\tau=\\frac\{\\sum\_\{\\sigma\\in\\Gamma\}\\mathbb\{E\}\_\{\\text\{alg\},\\nu\_\{\\sigma\}\}\\tau\}\{\(A\!\)^\{SH\}\}\\leq\\max\_\{\\sigma\\in\\Gamma\}\\mathbb\{E\}\_\{\\text\{alg\},\\nu\_\{\\sigma\}\}\\tau\.And we have completed the proof\. ∎

To prove Theorem[26](https://arxiv.org/html/2605.23182#Thmtheorem26), we present the following two theorems, which is required to prove Theorem[26](https://arxiv.org/html/2605.23182#Thmtheorem26)\.

###### Theorem 27\.

Given a Uniform Instanceν\\nuwithμ0\\mu\_\{0\}satisfying thatH​\(r\+ϵ\)2≥μ0\>H​r2\\frac\{H\(r\+\\epsilon\)\}\{2\}\\geq\\mu\_\{0\}\>\\frac\{Hr\}\{2\}, for anyδ\\delta\-PAC alg, we have

𝔼ν​τ≥Ω​\(H​log⁡12\.4​δ\(V0∗​\(s0\|ν\)−μ0\)2\)\.\\displaystyle\\mathbb\{E\}\_\{\\nu\}\\tau\\geq\\Omega\\left\(\\frac\{H\\log\\frac\{1\}\{2\.4\\delta\}\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\|\\nu\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\.

###### Theorem 28\.

Given a Uniform Instanceν\\nuwith\(S,A,H,r,ϵ\)\(S,A,H,r,\\epsilon\),μ0\\mu\_\{0\}satisfying thatH​\(r\+ϵ\)2\>μ0≥H​\(r\+ϵ\)4\+H​r4\\frac\{H\(r\+\\epsilon\)\}\{2\}\>\\mu\_\{0\}\\geq\\frac\{H\(r\+\\epsilon\)\}\{4\}\+\\frac\{Hr\}\{4\},0<δ<120<\\delta<\\frac\{1\}\{2\}, for anyδ\\delta\-PAC and symmetric algorithm, we have

𝔼​τ≥Ω​\(S​A\)ϵ2\.\\displaystyle\\mathbb\{E\}\\tau\\geq\\frac\{\\Omega\(SA\)\}\{\\epsilon^\{2\}\}\.

We first present the proof of Theorem[27](https://arxiv.org/html/2605.23182#Thmtheorem27)and[28](https://arxiv.org/html/2605.23182#Thmtheorem28), which require Lemmas in section[C\.4](https://arxiv.org/html/2605.23182#A3.SS4)\. At the end of this subsection, we present the proof of Theorem[26](https://arxiv.org/html/2605.23182#Thmtheorem26)\.

###### Proof of Theorem[27](https://arxiv.org/html/2605.23182#Thmtheorem27)\.

Definer′​\(η\)=μ0H−ηr^\{\\prime\}\(\\eta\)=\\frac\{\\mu\_\{0\}\}\{H\}\-\\etafor arbitraryμ0H−r\>η\>0\\frac\{\\mu\_\{0\}\}\{H\}\-r\>\\eta\>0\. We consider an alternative instanceν′​\(η\)\\nu^\{\\prime\}\(\\eta\)which is Uniform Instance with parameters\(S,A,H,r,2​μ0H−r−η\)\(S,A,H,r,\\frac\{2\\mu\_\{0\}\}\{H\}\-r\-\\eta\)\. It is evident to seeV0∗​\(s0\|ν′​\(η\)\)=μ0−H2​η<μ0V^\{\*\}\_\{0\}\(s\_\{0\}\|\\nu^\{\\prime\}\(\\eta\)\)=\\mu\_\{0\}\-\\frac\{H\}\{2\}\\eta<\\mu\_\{0\}, suggesting thatPralg,ν′​\(η\)⁡\(π^=None\)\>1−δ\\Pr\_\{\\text\{alg\},\\nu^\{\\prime\}\(\\eta\)\}\(\\hat\{\\pi\}=\\textsf\{None\}\)\>1\-\\delta\.

Notice thatPralg,ν⁡\(π^=None\)<δ\\Pr\_\{\\text\{alg\},\\nu\}\(\\hat\{\\pi\}=\\textsf\{None\}\)<\\delta, by the Lemma 1 inKaufmannet al\.\[[2016](https://arxiv.org/html/2605.23182#bib.bib4)\], we can conclude

∑s=1S−2∑h:oddkl​\(r\+ϵ,2​μ0H−η\)​𝔼alg,ν​nhτ​\(s,1\)≥kl​\(1−δ,δ\)\\displaystyle\\sum\_\{s=1\}^\{S\-2\}\\sum\_\{h:\\text\{odd\}\}\\text\{kl\}\(r\+\\epsilon,\\frac\{2\\mu\_\{0\}\}\{H\}\-\\eta\)\\mathbb\{E\}\_\{\\text\{alg\},\\nu\}n\_\{h\}^\{\\tau\}\(s,1\)\\geq\\text\{kl\}\(1\-\\delta,\\delta\)\(49\)wherekl​\(x,y\)=x​log⁡xy\+\(1−x\)​log⁡1−x1−y\\text\{kl\}\(x,y\)=x\\log\\frac\{x\}\{y\}\+\(1\-x\)\\log\\frac\{1\-x\}\{1\-y\}\. From the definition of Simple Requirement, we know14<r<2​μ0H−η<34\\frac\{1\}\{4\}<r<\\frac\{2\\mu\_\{0\}\}\{H\}\-\\eta<\\frac\{3\}\{4\}\. By the Lemma[36](https://arxiv.org/html/2605.23182#Thmtheorem36), we knowkl​\(r\+ϵ,2​μ0H−r−η\)≤3​\(r\+ϵ−2​μ0H\+η\)2\\text\{kl\}\(r\+\\epsilon,\\frac\{2\\mu\_\{0\}\}\{H\}\-r\-\\eta\)\\leq 3\(r\+\\epsilon\-\\frac\{2\\mu\_\{0\}\}\{H\}\+\\eta\)^\{2\}and \([49](https://arxiv.org/html/2605.23182#A3.E49)\) implies

∑s=1S−2∑h:odd𝔼alg,ν​nhτ​\(s,1\)≥kl​\(1−δ,δ\)3​\(r\+ϵ−2​μ0H\+η\)2\\displaystyle\\sum\_\{s=1\}^\{S\-2\}\\sum\_\{h:\\text\{odd\}\}\\mathbb\{E\}\_\{\\text\{alg\},\\nu\}n\_\{h\}^\{\\tau\}\(s,1\)\\geq\\frac\{\\text\{kl\}\(1\-\\delta,\\delta\)\}\{3\(r\+\\epsilon\-\\frac\{2\\mu\_\{0\}\}\{H\}\+\\eta\)^\{2\}\}⇒\\displaystyle\\Rightarrow1H​∑s=1S∑h=1H∑a=1A𝔼alg,ν​nhτ​\(s,a\)≥H​kl​\(1−δ,δ\)12​\(V0∗​\(s0\|ν\)−μ0\+H2​η\)2\\displaystyle\\frac\{1\}\{H\}\\sum\_\{s=1\}^\{S\}\\sum\_\{h=1\}^\{H\}\\sum\_\{a=1\}^\{A\}\\mathbb\{E\}\_\{\\text\{alg\},\\nu\}n\_\{h\}^\{\\tau\}\(s,a\)\\geq\\frac\{H\\text\{kl\}\(1\-\\delta,\\delta\)\}\{12\(V^\{\*\}\_\{0\}\(s\_\{0\}\|\\nu\)\-\\mu\_\{0\}\+\\frac\{H\}\{2\}\\eta\)^\{2\}\}By the fact that𝔼ν​τ=1H​∑s=1S∑h=1H∑a=1A𝔼alg,ν​nhτ​\(s,a\)\\mathbb\{E\}\_\{\\nu\}\\tau=\\frac\{1\}\{H\}\\sum\_\{s=1\}^\{S\}\\sum\_\{h=1\}^\{H\}\\sum\_\{a=1\}^\{A\}\\mathbb\{E\}\_\{\\text\{alg\},\\nu\}n\_\{h\}^\{\\tau\}\(s,a\), we can conclude

𝔼ν​τ≥H​log⁡12\.4​δ12​\(V0∗​\(s0\|ν\)−μ0\+H2​η\)2\\displaystyle\\mathbb\{E\}\_\{\\nu\}\\tau\\geq\\frac\{H\\log\\frac\{1\}\{2\.4\\delta\}\}\{12\(V^\{\*\}\_\{0\}\(s\_\{0\}\|\\nu\)\-\\mu\_\{0\}\+\\frac\{H\}\{2\}\\eta\)^\{2\}\}holds for allμ0H−r\>η\>0\\frac\{\\mu\_\{0\}\}\{H\}\-r\>\\eta\>0\. Letη→0\\eta\\rightarrow 0, we have completed the proof\. ∎

###### Proof of Theorem[28](https://arxiv.org/html/2605.23182#Thmtheorem28)\.

By the Lemma[31](https://arxiv.org/html/2605.23182#Thmtheorem31), we can find\{\(s\(i\),h\(i\)\)\}i=1\(S−2\)​H16\\\{\(s^\{\(i\)\},h^\{\(i\)\}\)\\\}\_\{i=1\}^\{\\frac\{\(S\-2\)H\}\{16\}\}, wheres\(i\)∈\[S−2\]s^\{\(i\)\}\\in\[S\-2\],h\(i\)∈\[H\]h^\{\(i\)\}\\in\[H\]is odd, such that

Pralg,ν⁡\(π^h\(i\)​\(s\(i\)\)=1,τ<\+∞\)≥17,∀i∈\{1,2,⋯,\(S−2\)​H16\}\.\\displaystyle\\Pr\_\{\\text\{alg\},\\nu\}\\big\(\\hat\{\\pi\}\_\{h^\{\(i\)\}\}\(s^\{\(i\)\}\)=1,\\tau<\+\\infty\\big\)\\geq\\frac\{1\}\{7\},\\forall i\\in\\\{1,2,\\cdots,\\frac\{\(S\-2\)H\}\{16\}\\\}\.By the Lemma[34](https://arxiv.org/html/2605.23182#Thmtheorem34), we can conclude

𝔼ν,alg~​nh\(i\)τ​\(s\(i\),a\)≥12⋅105​ϵ2,∀a∈\{2,⋯,A\},∀i∈\{1,2,⋯,\(S−2\)​H16\}\.\\displaystyle\\mathbb\{E\}\_\{\\nu,\\tilde\{\\text\{alg\}\}\}n\_\{h^\{\(i\)\}\}^\{\\tau\}\(s^\{\(i\)\},a\)\\geq\\frac\{1\}\{2\\cdot 10^\{5\}\\epsilon^\{2\}\},\\forall a\\in\\\{2,\\cdots,A\\\},\\forall i\\in\\\{1,2,\\cdots,\\frac\{\(S\-2\)H\}\{16\}\\\}\.\(50\)Then, we can conclude

𝔼ν,alg~​τ=\\displaystyle\\mathbb\{E\}\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\\tau=1H​∑a=1A∑s=1S∑h=1H𝔼ν,alg~​nh​\(s,a\)\\displaystyle\\frac\{1\}\{H\}\\sum\_\{a=1\}^\{A\}\\sum\_\{s=1\}^\{S\}\\sum\_\{h=1\}^\{H\}\\mathbb\{E\}\_\{\\nu,\\tilde\{\\text\{alg\}\}\}n\_\{h\}\(s,a\)≥\\displaystyle\\geq1H​∑a=2A∑i=1\(S−2\)​H16𝔼ν,alg~​nh\(i\)τ​\(s\(i\),a\)\\displaystyle\\frac\{1\}\{H\}\\sum\_\{a=2\}^\{A\}\\sum\_\{i=1\}^\{\\frac\{\(S\-2\)H\}\{16\}\}\\mathbb\{E\}\_\{\\nu,\\tilde\{\\text\{alg\}\}\}n\_\{h^\{\(i\)\}\}^\{\\tau\}\(s^\{\(i\)\},a\)≥\\displaystyle\\geq\(A−1\)​\(S−2\)32⋅105​ϵ2\.\\displaystyle\\frac\{\(A\-1\)\(S\-2\)\}\{32\\cdot 10^\{5\}\\epsilon^\{2\}\}\.The last line is by the \([50](https://arxiv.org/html/2605.23182#A3.E50)\), which completes the proof\. ∎

Given Theorem[27](https://arxiv.org/html/2605.23182#Thmtheorem27)and[28](https://arxiv.org/html/2605.23182#Thmtheorem28), we are ready to prove Theorem[26](https://arxiv.org/html/2605.23182#Thmtheorem26)\.

###### Proof of Theorem[26](https://arxiv.org/html/2605.23182#Thmtheorem26)\.

By Theorem[27](https://arxiv.org/html/2605.23182#Thmtheorem27)and[28](https://arxiv.org/html/2605.23182#Thmtheorem28), we can conclude

𝔼ν​τ≥Ω​\(H​log⁡12\.4​δ\(V0∗​\(s0\|ν\)−μ0\)2\+S​Aϵ2\)\\displaystyle\\mathbb\{E\}\_\{\\nu\}\\tau\\geq\\Omega\\left\(\\frac\{H\\log\\frac\{1\}\{2\.4\\delta\}\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\|\\nu\)\-\\mu\_\{0\}\)^\{2\}\}\+\\frac\{SA\}\{\\epsilon^\{2\}\}\\right\)By the construction of Uniform Instanceν\\nu, we knowV0∗​\(s0\|ν\)=H​\(r\+ϵ\)2V^\{\*\}\_\{0\}\(s\_\{0\}\|\\nu\)=\\frac\{H\(r\+\\epsilon\)\}\{2\}\. Since3​H​\(r\+ϵ\)8\+H​r8\>μ0\>H​r2\\frac\{3H\(r\+\\epsilon\)\}\{8\}\+\\frac\{Hr\}\{8\}\>\\mu\_\{0\}\>\\frac\{Hr\}\{2\}, we can conclude

\(V0∗​\(s0\|ν\)−μ0\)2≥\(H​\(r\+ϵ\)2−3​H​\(r\+ϵ\)8−H​r8\)2=\(H​ϵ8\)2=H2​ϵ264\.\\displaystyle\(V^\{\*\}\_\{0\}\(s\_\{0\}\|\\nu\)\-\\mu\_\{0\}\)^\{2\}\\geq\(\\frac\{H\(r\+\\epsilon\)\}\{2\}\-\\frac\{3H\(r\+\\epsilon\)\}\{8\}\-\\frac\{Hr\}\{8\}\)^\{2\}=\(\\frac\{H\\epsilon\}\{8\}\)^\{2\}=\\frac\{H^\{2\}\\epsilon^\{2\}\}\{64\}\.The last ste implies the lower bound

𝔼ν​τ≥Ω​\(H​log⁡12\.4​δ\(V0∗​\(s0\|ν\)−μ0\)2\+S​A​H2\(V0∗​\(s0\|ν\)−μ0\)2\)\.\\displaystyle\\mathbb\{E\}\_\{\\nu\}\\tau\\geq\\Omega\\left\(\\frac\{H\\log\\frac\{1\}\{2\.4\\delta\}\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\|\\nu\)\-\\mu\_\{0\}\)^\{2\}\}\+\\frac\{SAH^\{2\}\}\{\(V^\{\*\}\_\{0\}\(s\_\{0\}\|\\nu\)\-\\mu\_\{0\}\)^\{2\}\}\\right\)\.∎

### C\.3Proof of Lower Bound, for negative instances

Before delivering the proof of Theorem[5](https://arxiv.org/html/2605.23182#Thmtheorem5), we first present the following lemma to calculate the exact value ofV0∗​\(s0\|ν\)V^\{\*\}\_\{0\}\(s\_\{0\}\|\\nu\)for a Tree instanceν\\nu\.

###### Lemma 29\.

For a Tree Instanceν\\nuwith parameter\(S,A,H,r\)\(S,A,H,r\),V0∗​\(s0\|ν\)=\(H−1\)​rV^\{\*\}\_\{0\}\(s\_\{0\}\|\\nu\)=\(H\-1\)r\.

###### Proof\.

Given a policyπ\\pi, notice that\{\(s,a,h\):h:Prν,π⁡\(hitting​s​at round​h\)\>0,πh​\(s\)=a≥3\}\\\{\(s,a,h\):h:\\Pr\_\{\\nu,\\pi\}\(\\text\{hitting \}s\\text\{ at round \}h\)\>0,\\pi\_\{h\}\(s\)=a\\geq 3\\\}contains at most 1 element\. Denote the unique element as\(sπ,aπ,hπ\)\(s^\{\\pi\},a^\{\\pi\},h^\{\\pi\}\)\. We haveV0π​\(1\)=\(H−hπ\)​r≤\(H−1\)​rV^\{\\pi\}\_\{0\}\(1\)=\(H\-h^\{\\pi\}\)r\\leq\(H\-1\)r\. And the equality holds only whenπh=1​\(s=1\)=a\\pi\_\{h=1\}\(s=1\)=afora≥3a\\geq 3\. ∎

Then, we are ready to prove Theorem[5](https://arxiv.org/html/2605.23182#Thmtheorem5)\.

###### Theorem\(Restatement of Theorem[5](https://arxiv.org/html/2605.23182#Thmtheorem5)\)\.

Apply anyδ\\delta\-PAC algorithm to a Tree instance specified by\(S,A,H,r\)\(S,A,H,r\)withS,A,H≥4S,A,H\\geq 4and34​\(H−log2⁡\(S−3\)\)\>μ0\>\(H−1\)​r\\frac\{3\}\{4\}\(H\-\\log\_\{2\}\(S\-3\)\)\>\\mu\_\{0\}\>\(H\-1\)r, we have

𝔼ν​τ≥Ω​\(H​A​S​log⁡1δ\(μ0−V0∗​\(s0\|ν\)\)2\)\.\\displaystyle\\mathbb\{E\}\_\{\\nu\}\\tau\\geq\\Omega\\left\(\\frac\{HAS\\log\\frac\{1\}\{\\delta\}\}\{\(\\mu\_\{0\}\-V^\{\*\}\_\{0\}\(s\_\{0\}\|\\nu\)\)^\{2\}\}\\right\)\.

###### Proof of Theorem[5](https://arxiv.org/html/2605.23182#Thmtheorem5)\.

For any pair\(s0,a0\)\(s\_\{0\},a\_\{0\}\), where1≤s0≤2N−11\\leq s\_\{0\}\\leq 2^\{N\}\-1,A≥a0≥3A\\geq a\_\{0\}\\geq 3, we defineh​\(s\)=⌈log2⁡\(s0\+1\)⌉h\(s\)=\\lceil\\log\_\{2\}\(s\_\{0\}\+1\)\\rceiland an instanceνs0,a0​\(η\)\\nu\_\{s\_\{0\},a\_\{0\}\}\(\\eta\),1−μ0H−h​\(s\)\>η\>01\-\\frac\{\\mu\_\{0\}\}\{H\-h\(s\)\}\>\\eta\>0, such thatνs0,a0\\nu\_\{s\_\{0\},a\_\{0\}\}only differs inph​\(s0\)νs0,a0​\(sgood\|s0,a0\)=μ0H−h​\(s\)\+ηp\_\{h\(s\_\{0\}\)\}^\{\\nu\_\{s\_\{0\},a\_\{0\}\}\}\(s\_\{\\text\{good\}\}\|s\_\{0\},a\_\{0\}\)=\\frac\{\\mu\_\{0\}\}\{H\-h\(s\)\}\+\\etacompared toν\\nu\.

We first validate the instanceνs0,a0​\(η\)\\nu\_\{s\_\{0\},a\_\{0\}\}\(\\eta\)is positive for anyη\>0\\eta\>0, by figuring out a qualified policy\. From Lemma[30](https://arxiv.org/html/2605.23182#Thmtheorem30), we know there exists a policyπ\\pisuch thatPrν⁡\(sh​\(s0\)π=s0\)=1\\Pr\_\{\\nu\}\(s\_\{h\(s\_\{0\}\)\}^\{\\pi\}=s\_\{0\}\)=1\. Now, we consider a policyπ′\\pi^\{\\prime\}such thatπh′​\(s\)=\{a0s=s0,h=h​\(s0\)πh​\(s\)else\\pi\_\{h\}^\{\\prime\}\(s\)=\\begin\{cases\}a\_\{0\}&s=s\_\{0\},h=h\(s\_\{0\}\)\\\\ \\pi\_\{h\}\(s\)&\\text\{else\}\\end\{cases\}\. Evident to seeV0π′​\(s0;νs0,a0​\(η\)\)=\(H−h​\(s0\)\)​\(μ0H−h​\(s\)\+η\)=μ0\+η​\(H−h​\(s0\)\)\>μ0V^\{\\pi^\{\\prime\}\}\_\{0\}\(s\_\{0\};\\nu\_\{s\_\{0\},a\_\{0\}\}\(\\eta\)\)=\(H\-h\(s\_\{0\}\)\)\\left\(\\frac\{\\mu\_\{0\}\}\{H\-h\(s\)\}\+\\eta\\right\)=\\mu\_\{0\}\+\\eta\(H\-h\(s\_\{0\}\)\)\>\\mu\_\{0\}\.

Givenνs0,a0​\(η\)\\nu\_\{s\_\{0\},a\_\{0\}\}\(\\eta\)is positive andν\\nuis negative, we can concludePrν,alg⁡\(π^=None\)≥1−δ\\Pr\_\{\\nu,\\text\{alg\}\}\(\\hat\{\\pi\}=\\textsf\{None\}\)\\geq 1\-\\delta,Prνs0,a0​\(η\),alg⁡\(π^=None\)≤δ\\Pr\_\{\\nu\_\{s\_\{0\},a\_\{0\}\}\(\\eta\),\\text\{alg\}\}\(\\hat\{\\pi\}=\\textsf\{None\}\)\\leq\\delta\. Apply the Lemma 1 inKaufmannet al\.\[[2016](https://arxiv.org/html/2605.23182#bib.bib4)\], we have

𝔼ν,alg​nh​\(s0\)τ​\(s0,a0\)​kl​\(r,μ0H−h​\(s\)\+η\)≥log⁡12\.4​δ\.\\displaystyle\\mathbb\{E\}\_\{\\nu,\\text\{alg\}\}n\_\{h\(s\_\{0\}\)\}^\{\\tau\}\(s\_\{0\},a\_\{0\}\)\\text\{kl\}\(r,\\frac\{\\mu\_\{0\}\}\{H\-h\(s\)\}\+\\eta\)\\geq\\log\\frac\{1\}\{2\.4\\delta\}\.Since the above inequality holds for arbitraryη\\eta, lettingη→0\\eta\\rightarrow 0implies

𝔼ν,alg​nh​\(s0\)τ​\(s0,a0\)​kl​\(r,μ0H−h​\(s\)\)≥log⁡12\.4​δ\\displaystyle\\mathbb\{E\}\_\{\\nu,\\text\{alg\}\}n\_\{h\(s\_\{0\}\)\}^\{\\tau\}\(s\_\{0\},a\_\{0\}\)\\text\{kl\}\(r,\\frac\{\\mu\_\{0\}\}\{H\-h\(s\)\}\)\\geq\\log\\frac\{1\}\{2\.4\\delta\}⇒\\displaystyle\\Rightarrow𝔼ν,alg​nh​\(s0\)τ​\(s0,a0\)⋅3​\(μ0H−h​\(s\)−r\)2≥log⁡12\.4​δ\\displaystyle\\mathbb\{E\}\_\{\\nu,\\text\{alg\}\}n\_\{h\(s\_\{0\}\)\}^\{\\tau\}\(s\_\{0\},a\_\{0\}\)\\cdot 3\\big\(\\frac\{\\mu\_\{0\}\}\{H\-h\(s\)\}\-r\\big\)^\{2\}\\geq\\log\\frac\{1\}\{2\.4\\delta\}\(51\)⇒\\displaystyle\\Rightarrow𝔼ν,alg​nh​\(s0\)τ​\(s0,a0\)⋅3​\(μ0H−⌈log2⁡\(S−1\)⌉−r\)2≥log⁡12\.4​δ\\displaystyle\\mathbb\{E\}\_\{\\nu,\\text\{alg\}\}n\_\{h\(s\_\{0\}\)\}^\{\\tau\}\(s\_\{0\},a\_\{0\}\)\\cdot 3\\big\(\\frac\{\\mu\_\{0\}\}\{H\-\\lceil\\log\_\{2\}\(S\-1\)\\rceil\}\-r\\big\)^\{2\}\\geq\\log\\frac\{1\}\{2\.4\\delta\}\(52\)⇒\\displaystyle\\Rightarrow𝔼ν,alg​nh​\(s0\)τ​\(s0,a0\)⋅3​\(μ023​\(H−1\)−r\)2≥log⁡12\.4​δ\\displaystyle\\mathbb\{E\}\_\{\\nu,\\text\{alg\}\}n\_\{h\(s\_\{0\}\)\}^\{\\tau\}\(s\_\{0\},a\_\{0\}\)\\cdot 3\\big\(\\frac\{\\mu\_\{0\}\}\{\\frac\{2\}\{3\}\(H\-1\)\}\-r\\big\)^\{2\}\\geq\\log\\frac\{1\}\{2\.4\\delta\}\(53\)⇒\\displaystyle\\Rightarrow𝔼ν,alg​nh​\(s0\)τ​\(s0,a0\)≥49​\(H−1\)2​log⁡12\.4​δ3​\(μ0−\(H−1\)​r\)2\\displaystyle\\mathbb\{E\}\_\{\\nu,\\text\{alg\}\}n\_\{h\(s\_\{0\}\)\}^\{\\tau\}\(s\_\{0\},a\_\{0\}\)\\geq\\frac\{\\frac\{4\}\{9\}\(H\-1\)^\{2\}\\log\\frac\{1\}\{2\.4\\delta\}\}\{3\(\\mu\_\{0\}\-\(H\-1\)r\)^\{2\}\}\(54\)Step \([51](https://arxiv.org/html/2605.23182#A3.E51)\) is by the assumption34​\(H−log2⁡\(S−3\)\)\>μ0\>\(H−1\)​r\\frac\{3\}\{4\}\(H\-\\log\_\{2\}\(S\-3\)\)\>\\mu\_\{0\}\>\(H\-1\)r, which suggests thatr,μ0H−h​\(s\)∈\(14,34\)r,\\frac\{\\mu\_\{0\}\}\{H\-h\(s\)\}\\in\(\\frac\{1\}\{4\},\\frac\{3\}\{4\}\)\. Lemma[37](https://arxiv.org/html/2605.23182#Thmtheorem37)concludeskl​\(r,μ0H−h​\(s\)\)≤3​\(μ0H−h​\(s\)−r\)2\\text\{kl\}\(r,\\frac\{\\mu\_\{0\}\}\{H\-h\(s\)\}\)\\leq 3\(\\frac\{\\mu\_\{0\}\}\{H\-h\(s\)\}\-r\)^\{2\}\. Step \([52](https://arxiv.org/html/2605.23182#A3.E52)\) is bys0≤S−2s\_\{0\}\\leq S\-2and34​\(H−log2⁡\(S−3\)\)\>μ0\>\(H−1\)​r\\frac\{3\}\{4\}\(H\-\\log\_\{2\}\(S\-3\)\)\>\\mu\_\{0\}\>\(H\-1\)rimpliesμ0H−⌈log2⁡\(S−1\)⌉≥μ0H−h​\(s\)\>r\\frac\{\\mu\_\{0\}\}\{H\-\\lceil\\log\_\{2\}\(S\-1\)\\rceil\}\\geq\\frac\{\\mu\_\{0\}\}\{H\-h\(s\)\}\>r\. Step \([53](https://arxiv.org/html/2605.23182#A3.E53)\) is by the definition of Tree Instance, which requiresH≥6​N,2N=S−3,N≥1H\\geq 6N,2^\{N\}=S\-3,N\\geq 1, implying

⌈log2⁡\(S−1\)⌉\\displaystyle\\lceil\\log\_\{2\}\(S\-1\)\\rceil=\\displaystyle=⌈log2⁡\(2N\)⌉\\displaystyle\\lceil\\log\_\{2\}\(2^\{N\}\)\\rceil≤\\displaystyle\\leqN\\displaystyle N≤\\displaystyle\\leqH6\.\\displaystyle\\frac\{H\}\{6\}\.The last line meansH−⌈log2⁡\(S−1\)⌉≥2​\(H−1\)3H\-\\lceil\\log\_\{2\}\(S\-1\)\\rceil\\geq\\frac\{2\(H\-1\)\}\{3\}, which is step \([53](https://arxiv.org/html/2605.23182#A3.E53)\)\.

Summing up \([54](https://arxiv.org/html/2605.23182#A3.E54)\) fors0∈\[S−2\]s\_\{0\}\\in\[S\-2\]anda0∈\{3,⋯,A\}a\_\{0\}\\in\\\{3,\\cdots,A\\\}, we can conclude

∑s=1S−2∑a=3A𝔼ν,alg​nh​\(s\)τ​\(s,a\)≥4​\(S−2\)​\(A−1\)​\(H−1\)2​log⁡12\.4​δ27​\(μ0−\(H−1\)​r\)2\\displaystyle\\sum\_\{s=1\}^\{S\-2\}\\sum\_\{a=3\}^\{A\}\\mathbb\{E\}\_\{\\nu,\\text\{alg\}\}n\_\{h\(s\)\}^\{\\tau\}\(s,a\)\\geq\\frac\{4\(S\-2\)\(A\-1\)\(H\-1\)^\{2\}\\log\\frac\{1\}\{2\.4\\delta\}\}\{27\(\\mu\_\{0\}\-\(H\-1\)r\)^\{2\}\}⇒\\displaystyle\\Rightarrow𝔼ν,alg​τ=1H​∑h=1H∑s=1S∑a=1A𝔼ν,alg​nh​\(s\)τ​\(s,a\)≥Ω​\(H​A​S​log⁡1δ\(μ0−V0∗​\(s0\|ν\)\)2\)\.\\displaystyle\\mathbb\{E\}\_\{\\nu,\\text\{alg\}\}\\tau=\\frac\{1\}\{H\}\\sum\_\{h=1\}^\{H\}\\sum\_\{s=1\}^\{S\}\\sum\_\{a=1\}^\{A\}\\mathbb\{E\}\_\{\\nu,\\text\{alg\}\}n\_\{h\(s\)\}^\{\\tau\}\(s,a\)\\geq\\Omega\\left\(\\frac\{HAS\\log\\frac\{1\}\{\\delta\}\}\{\(\\mu\_\{0\}\-V^\{\*\}\_\{0\}\(s\_\{0\}\|\\nu\)\)^\{2\}\}\\right\)\.We have completed the proof\. ∎

The following is an auxiliary Lemma, showing that there exists a shorted path from state 1 tossfor anys∈\[S−2\]s\\in\[S\-2\]in the Tree Instance\.

###### Lemma 30\.

Consider a Tree Instanceν\\nuspecified by parameter\(S,A,H,r\)\(S,A,H,r\)\. For any states∈\[S−2\]s\\in\[S\-2\], there exists a policyπ\\pi, such thatPrν⁡\(s⌈log2⁡\(s\+1\)⌉π=s\)=1\\Pr\_\{\\nu\}\(s\_\{\\lceil\\log\_\{2\}\(s\+1\)\\rceil\}^\{\\pi\}=s\)=1\.

###### Proof of Lemma[30](https://arxiv.org/html/2605.23182#Thmtheorem30)\.

We prove the Lemma by induction\. Fors=1s=1, the lemma is true, as we always start at the state11\. Now, assume for alls∈\[2n,2n\+1−1\]∩ℤs\\in\[2^\{n\},2^\{n\+1\}\-1\]\\cap\\mathbb\{Z\}, the conclusion is true, we consider a states∈\[2n\+1,2n\+2−1\]∩ℤs\\in\[2^\{n\+1\},2^\{n\+2\}\-1\]\\cap\\mathbb\{Z\}\. Easy to see⌊s2⌋∈\[2n,2n\+1−1\]\\lfloor\\frac\{s\}\{2\}\\rfloor\\in\[2^\{n\},2^\{n\+1\}\-1\]\. Thus, there exists a policyπ\\pi, such thatPrν⁡\(s⌈log2⁡\(⌊s2⌋\+1\)⌉π=⌊s2⌋\)=1\\Pr\_\{\\nu\}\(s\_\{\\lceil\\log\_\{2\}\(\\lfloor\\frac\{s\}\{2\}\\rfloor\+1\)\\rceil\}^\{\\pi\}=\\lfloor\\frac\{s\}\{2\}\\rfloor\)=1\. Since there exists an actionas∈\{1,2\}a\_\{s\}\\in\\\{1,2\\\}in state⌊s2⌋\\lfloor\\frac\{s\}\{2\}\\rfloorsuch thatp⌈log2⁡\(⌊s2⌋\+1\)⌉​\(s\|⌊s2⌋,as\)=1p\_\{\\lceil\\log\_\{2\}\(\\lfloor\\frac\{s\}\{2\}\\rfloor\+1\)\\rceil\}\(s\|\\lfloor\\frac\{s\}\{2\}\\rfloor,a\_\{s\}\)=1, we can conclude there exists a policyπ′\\pi^\{\\prime\}such thatPrν⁡\(s⌈log2⁡\(⌊s2⌋\+1\)⌉\+1π′=s\)=1\\Pr\_\{\\nu\}\(s\_\{\\lceil\\log\_\{2\}\(\\lfloor\\frac\{s\}\{2\}\\rfloor\+1\)\\rceil\+1\}^\{\\pi^\{\\prime\}\}=s\)=1\.

Notice that2n\+1≤⌊s2⌋\+1≤2n\+12^\{n\}\+1\\leq\\lfloor\\frac\{s\}\{2\}\\rfloor\+1\\leq 2^\{n\+1\}, suggesting that

⌈log2⁡\(⌊s2⌋\+1\)⌉\+1\\displaystyle\\lceil\\log\_\{2\}\(\\lfloor\\frac\{s\}\{2\}\\rfloor\+1\)\\rceil\+1=\\displaystyle=n\+1\+1\\displaystyle n\+1\+1=\\displaystyle=n\+2\\displaystyle n\+2=\\displaystyle=⌈log2⁡\(s\+1\)⌉\.\\displaystyle\\lceil\\log\_\{2\}\(s\+1\)\\rceil\.The last line completes the induction, also the proof of Lemma[30](https://arxiv.org/html/2605.23182#Thmtheorem30)\. ∎

### C\.4Auxiliary Lemmas in the Proof of Lower Bound, for Uniform Instances

In this section, We present auxiliary lemmas to support Theorem[28](https://arxiv.org/html/2605.23182#Thmtheorem28)\. Without extra description, we assume3​H​\(r\+ϵ\)8\+H​r8\>μ0\>H​r2\\frac\{3H\(r\+\\epsilon\)\}\{8\}\+\\frac\{Hr\}\{8\}\>\\mu\_\{0\}\>\\frac\{Hr\}\{2\},S,H≥10S,H\\geq 10,A≥100A\\geq 100,\(S−2\)​H\(S\-2\)His divisible by1616,0<ϵ<1200​50<\\epsilon<\\frac\{1\}\{200\\sqrt\{5\}\}always hold in the remaining subsection\.

The first lemma is to show aδ\\delta\-PAC algorithm must output policy that identifies action 1 as the optimal action for “enough” pairs of\(s,h\)\(s,h\)\.

###### Lemma 31\.

Apply aδ\\delta\-PAC algorithm alg to a Uniform Instanceν\\nuwith\(S,A,H,r,ϵ\)\(S,A,H,r,\\epsilon\)\. Assumeμ0\\mu\_\{0\}satisfiesH​\(r\+ϵ\)2\>μ0≥H​\(r\+ϵ\)4\+H​r4\\frac\{H\(r\+\\epsilon\)\}\{2\}\>\\mu\_\{0\}\\geq\\frac\{H\(r\+\\epsilon\)\}\{4\}\+\\frac\{Hr\}\{4\},0<δ<120<\\delta<\\frac\{1\}\{2\}, we can find\{\(s\(i\),h\(i\)\)\}i=1\(S−2\)​H16\\\{\(s^\{\(i\)\},h^\{\(i\)\}\)\\\}\_\{i=1\}^\{\\frac\{\(S\-2\)H\}\{16\}\}, wheres\(i\)∈\[S−2\]s^\{\(i\)\}\\in\[S\-2\],h\(i\)∈\[H\]h^\{\(i\)\}\\in\[H\]is odd, such that

Pralg,ν⁡\(π^h\(i\)​\(s\(i\)\)=1,τ<\+∞\)≥17,∀i∈\{1,2,⋯,\(S−2\)​H16\}\\displaystyle\\Pr\_\{\\text\{alg\},\\nu\}\\big\(\\hat\{\\pi\}\_\{h^\{\(i\)\}\}\(s^\{\(i\)\}\)=1,\\tau<\+\\infty\\big\)\\geq\\frac\{1\}\{7\},\\forall i\\in\\\{1,2,\\cdots,\\frac\{\(S\-2\)H\}\{16\}\\\}

###### Proof of Lemma[31](https://arxiv.org/html/2605.23182#Thmtheorem31)\.

Given a policyπ\\pion the Uniform Instance, we know

V0π​\(s0\)=∑s∈\[S−2\],odd​h\(r\+ϵ\)​𝟙​\(πh​\(s\)=1\)\+∑s∈\[S−2\],odd​hr​𝟙​\(πh​\(s\)≠1\)S−2\.\\displaystyle V\_\{0\}^\{\\pi\}\(s\_\{0\}\)=\\frac\{\\sum\_\{s\\in\[S\-2\],\\text\{odd \}h\}\(r\+\\epsilon\)\\mathds\{1\}\(\\pi\_\{h\}\(s\)=1\)\+\\sum\_\{s\\in\[S\-2\],\\text\{odd \}h\}r\\mathds\{1\}\(\\pi\_\{h\}\(s\)\\neq 1\)\}\{S\-2\}\.By the assumption thatμ0≥H​\(r\+ϵ\)4\+H​r4\\mu\_\{0\}\\geq\\frac\{H\(r\+\\epsilon\)\}\{4\}\+\\frac\{Hr\}\{4\}and theδ\\delta\-PAC requirement, we know

Pralg,ν⁡\(∑s∈\[S−2\],odd​h𝟙​\(π^h​\(s\)=1\)≥H​\(S−2\)4,τ<\+∞\)≥1−δ\.\\displaystyle\\Pr\_\{\\text\{alg\},\\nu\}\\left\(\\sum\_\{s\\in\[S\-2\],\\text\{odd \}h\}\\mathds\{1\}\\big\(\\hat\{\\pi\}\_\{h\}\(s\)=1\\big\)\\geq\\frac\{H\(S\-2\)\}\{4\},\\tau<\+\\infty\\right\)\\geq 1\-\\delta\.Denote\{s\(i\),h\(i\)\}i=1S​H\\\{s^\{\(i\)\},h^\{\(i\)\}\\\}\_\{i=1\}^\{SH\}as a permutation of\[S−2\]×\{h∈\[H\]:h​is odd\}\[S\-2\]\\times\\\{h\\in\[H\]:h\\text\{ is odd\}\\\}, such that

Pralg,ν⁡\(π^h\(i\)​\(s\(i\)\)=1,τ<\+∞\)≥Pralg,ν⁡\(π^h\(i\+1\)​\(s\(i\+1\)\)=1,τ<\+∞\)\\displaystyle\\Pr\_\{\\text\{alg\},\\nu\}\\left\(\\hat\{\\pi\}\_\{h^\{\(i\)\}\}\(s^\{\(i\)\}\)=1,\\tau<\+\\infty\\right\)\\geq\\Pr\_\{\\text\{alg\},\\nu\}\\left\(\\hat\{\\pi\}\_\{h^\{\(i\+1\)\}\}\(s^\{\(i\+1\)\}\)=1,\\tau<\+\\infty\\right\)DefineΛ=\{C:C⊂\[S−2\]×\{h∈\[H\]:h​is odd\},\|C\|≥\(S−2\)​H4\}\\Lambda=\\\{C:C\\subset\[S\-2\]\\times\\\{h\\in\[H\]:h\\text\{ is odd\}\\\},\|C\|\\geq\\frac\{\(S\-2\)H\}\{4\}\\\},Λall=\{C:C⊂\[S−2\]×\{h∈\[H\]:h​is odd\}\}\\Lambda\_\{\\text\{all\}\}=\\\{C:C\\subset\[S\-2\]\\times\\\{h\\in\[H\]:h\\text\{ is odd\}\\\}\\\}\. Notice that

\{∑\(s,h\)𝟙​\(π^h​\(s\)=1\)≥H​\(S−2\)4\}\\displaystyle\\left\\\{\\sum\_\{\(s,h\)\}\\mathds\{1\}\\big\(\\hat\{\\pi\}\_\{h\}\(s\)=1\\big\)\\geq\\frac\{H\(S\-2\)\}\{4\}\\right\\\}=\\displaystyle=∪C∈Λ\{∀\(s,h\)∈C,π^h​\(s\)=1;∀\(s,h\)∉C,π^h​\(s\)≠1\}\\displaystyle\\cup\_\{C\\in\\Lambda\}\\\{\\forall\(s,h\)\\in C,\\hat\{\\pi\}\_\{h\}\(s\)=1;\\forall\(s,h\)\\notin C,\\hat\{\\pi\}\_\{h\}\(s\)\\neq 1\\\}Pralg,ν⁡\(∑\(s,h\)𝟙​\(π^h​\(s\)=1\)≥H​\(S−2\)4,τ<\+∞\)≥1−δ\\Pr\_\{\\text\{alg\},\\nu\}\\left\(\\sum\_\{\(s,h\)\}\\mathds\{1\}\\big\(\\hat\{\\pi\}\_\{h\}\(s\)=1\\big\)\\geq\\frac\{H\(S\-2\)\}\{4\},\\tau<\+\\infty\\right\)\\geq 1\-\\deltaimplies

∑C∈ΛPralg,ν⁡\(∀\(s,h\)∈C,π^h​\(s\)=1;∀\(s,h\)∉C,π^h​\(s\)≠1\)≥1−δ\.\\displaystyle\\sum\_\{C\\in\\Lambda\}\\Pr\_\{\\text\{alg\},\\nu\}\\left\(\\forall\(s,h\)\\in C,\\hat\{\\pi\}\_\{h\}\(s\)=1;\\forall\(s,h\)\\notin C,\\hat\{\\pi\}\_\{h\}\(s\)\\neq 1\\right\)\\geq 1\-\\delta\.\(55\)By straight forward calculation,

∑s∈\[S−2\],odd​hPralg,ν⁡\(π^h​\(s\)=1,τ<\+∞\)\\displaystyle\\sum\_\{s\\in\[S\-2\],\\text\{odd \}h\}\\Pr\_\{\\text\{alg\},\\nu\}\\left\(\\hat\{\\pi\}\_\{h\}\(s\)=1,\\tau<\+\\infty\\right\)=\\displaystyle=∑s∈\[S−2\],odd​h∑C:C∈Λall,\(s,h\)∈CPralg,ν⁡\(∀\(s′,h′\)∈C,π^h′​\(s′\)=1;∀\(s′,h′\)∉C,π^h′​\(s′\)≠1,τ<\+∞\)\\displaystyle\\sum\_\{s\\in\[S\-2\],\\text\{odd \}h\}\\sum\_\{C:C\\in\\Lambda\_\{\\text\{all\}\},\(s,h\)\\in C\}\\Pr\_\{\\text\{alg\},\\nu\}\\left\(\\forall\(s^\{\\prime\},h^\{\\prime\}\)\\in C,\\hat\{\\pi\}\_\{h^\{\\prime\}\}\(s^\{\\prime\}\)=1;\\forall\(s^\{\\prime\},h^\{\\prime\}\)\\notin C,\\hat\{\\pi\}\_\{h^\{\\prime\}\}\(s^\{\\prime\}\)\\neq 1,\\tau<\+\\infty\\right\)≥\\displaystyle\\geq∑s∈\[S−2\],odd​h∑C:C∈Λ,\(s,h\)∈CPralg,ν⁡\(∀\(s′,h′\)∈C,π^h′​\(s′\)=1;∀\(s′,h′\)∉C,π^h′​\(s′\)≠1,τ<\+∞\)\\displaystyle\\sum\_\{s\\in\[S\-2\],\\text\{odd \}h\}\\sum\_\{C:C\\in\\Lambda,\(s,h\)\\in C\}\\Pr\_\{\\text\{alg\},\\nu\}\\left\(\\forall\(s^\{\\prime\},h^\{\\prime\}\)\\in C,\\hat\{\\pi\}\_\{h^\{\\prime\}\}\(s^\{\\prime\}\)=1;\\forall\(s^\{\\prime\},h^\{\\prime\}\)\\notin C,\\hat\{\\pi\}\_\{h^\{\\prime\}\}\(s^\{\\prime\}\)\\neq 1,\\tau<\+\\infty\\right\)=\\displaystyle=∑s∈\[S−2\],odd​h∑C:C∈Λ𝟙​\(\(s,h\)∈C\)​Pralg,ν⁡\(∀\(s′,h′\)∈C,π^h′​\(s′\)=1;∀\(s′,h′\)∉C,π^h′​\(s′\)≠1,τ<\+∞\)\\displaystyle\\sum\_\{s\\in\[S\-2\],\\text\{odd \}h\}\\sum\_\{C:C\\in\\Lambda\}\\mathds\{1\}\\Big\(\(s,h\)\\in C\\Big\)\\Pr\_\{\\text\{alg\},\\nu\}\\left\(\\forall\(s^\{\\prime\},h^\{\\prime\}\)\\in C,\\hat\{\\pi\}\_\{h^\{\\prime\}\}\(s^\{\\prime\}\)=1;\\forall\(s^\{\\prime\},h^\{\\prime\}\)\\notin C,\\hat\{\\pi\}\_\{h^\{\\prime\}\}\(s^\{\\prime\}\)\\neq 1,\\tau<\+\\infty\\right\)=\\displaystyle=∑C:C∈Λ∑s∈\[S−2\],odd​h𝟙​\(\(s,h\)∈C\)​Pralg,ν⁡\(∀\(s′,h′\)∈C,π^h′​\(s′\)=1;∀\(s′,h′\)∉C,π^h′​\(s′\)≠1,τ<\+∞\)\\displaystyle\\sum\_\{C:C\\in\\Lambda\}\\sum\_\{s\\in\[S\-2\],\\text\{odd \}h\}\\mathds\{1\}\\Big\(\(s,h\)\\in C\\Big\)\\Pr\_\{\\text\{alg\},\\nu\}\\left\(\\forall\(s^\{\\prime\},h^\{\\prime\}\)\\in C,\\hat\{\\pi\}\_\{h^\{\\prime\}\}\(s^\{\\prime\}\)=1;\\forall\(s^\{\\prime\},h^\{\\prime\}\)\\notin C,\\hat\{\\pi\}\_\{h^\{\\prime\}\}\(s^\{\\prime\}\)\\neq 1,\\tau<\+\\infty\\right\)≥\\displaystyle\\geq∑C:C∈Λ\(S−2\)​H4​Pralg,ν⁡\(∀\(s′,h′\)∈C,π^h′​\(s′\)=1;∀\(s′,h′\)∉C,π^h′​\(s′\)≠1,τ<\+∞\)\\displaystyle\\sum\_\{C:C\\in\\Lambda\}\\frac\{\(S\-2\)H\}\{4\}\\Pr\_\{\\text\{alg\},\\nu\}\\left\(\\forall\(s^\{\\prime\},h^\{\\prime\}\)\\in C,\\hat\{\\pi\}\_\{h^\{\\prime\}\}\(s^\{\\prime\}\)=1;\\forall\(s^\{\\prime\},h^\{\\prime\}\)\\notin C,\\hat\{\\pi\}\_\{h^\{\\prime\}\}\(s^\{\\prime\}\)\\neq 1,\\tau<\+\\infty\\right\)≥\\displaystyle\\geq\(S−2\)​H4​\(1−δ\)\\displaystyle\\frac\{\(S\-2\)H\}\{4\}\(1\-\\delta\)≥\\displaystyle\\geq\(S−2\)​H8\.\\displaystyle\\frac\{\(S\-2\)H\}\{8\}\.The second last step is by \([55](https://arxiv.org/html/2605.23182#A3.E55)\) and the last step is byδ<12\\delta<\\frac\{1\}\{2\}\.

RecallPralg,ν⁡\(π^h\(i\)​\(s\(i\)\)=1,τ<\+∞\)\\Pr\_\{\\text\{alg\},\\nu\}\\left\(\\hat\{\\pi\}\_\{h^\{\(i\)\}\}\(s^\{\(i\)\}\)=1,\\tau<\+\\infty\\right\)is non\-decreasing regardingii, we can conclude

∑s,hPralg,ν⁡\(π^h​\(s\)=1,τ<\+∞\)\\displaystyle\\sum\_\{s,h\}\\Pr\_\{\\text\{alg\},\\nu\}\\left\(\\hat\{\\pi\}\_\{h\}\(s\)=1,\\tau<\+\\infty\\right\)=\\displaystyle=∑i=1\(S−2\)​H/2Pralg,ν⁡\(π^h\(i\)​\(s\(i\)\)=1,τ<\+∞\)\\displaystyle\\sum\_\{i=1\}^\{\(S\-2\)H/2\}\\Pr\_\{\\text\{alg\},\\nu\}\\left\(\\hat\{\\pi\}\_\{h^\{\(i\)\}\}\(s^\{\(i\)\}\)=1,\\tau<\+\\infty\\right\)=\\displaystyle=∑i=1\(S−2\)​H/16Pralg,ν⁡\(π^h\(i\)​\(s\(i\),τ<\+∞\)=1\)\+∑i=\(S−2\)​H/16\+1\(S−2\)​H/2Pralg,ν⁡\(π^h\(i\)​\(s\(i\)\)=1,τ<\+∞\)\\displaystyle\\sum\_\{i=1\}^\{\(S\-2\)H/16\}\\Pr\_\{\\text\{alg\},\\nu\}\\left\(\\hat\{\\pi\}\_\{h^\{\(i\)\}\}\(s^\{\(i\)\},\\tau<\+\\infty\)=1\\right\)\+\\sum\_\{i=\(S\-2\)H/16\+1\}^\{\(S\-2\)H/2\}\\Pr\_\{\\text\{alg\},\\nu\}\\left\(\\hat\{\\pi\}\_\{h^\{\(i\)\}\}\(s^\{\(i\)\}\)=1,\\tau<\+\\infty\\right\)≤\\displaystyle\\leq\(S−2\)​H/16\+7​\(S−2\)​H16​Pralg,ν⁡\(π^h\(\(S−2\)​H/16\+1\)​\(s\(\(S−2\)​H/16\+1\)\)=1,τ<\+∞\)\.\\displaystyle\(S\-2\)H/16\+\\frac\{7\(S\-2\)H\}\{16\}\\Pr\_\{\\text\{alg\},\\nu\}\\left\(\\hat\{\\pi\}\_\{h^\{\(\(S\-2\)H/16\+1\)\}\}\(s^\{\(\(S\-2\)H/16\+1\)\}\)=1,\\tau<\+\\infty\\right\)\.The last line suggests that

\(S−2\)​H8≤\(S−2\)​H/16\+7​\(S−2\)​H16​Pralg,ν⁡\(π^h\(\(S−2\)​H/16\+1\)​\(s\(\(S−2\)​H/16\+1\)\)=1,τ<\+∞\),\\displaystyle\\frac\{\(S\-2\)H\}\{8\}\\leq\(S\-2\)H/16\+\\frac\{7\(S\-2\)H\}\{16\}\\Pr\_\{\\text\{alg\},\\nu\}\\left\(\\hat\{\\pi\}\_\{h^\{\(\(S\-2\)H/16\+1\)\}\}\(s^\{\(\(S\-2\)H/16\+1\)\}\)=1,\\tau<\+\\infty\\right\),which means

17≤Pralg,ν⁡\(π^h\(\(S−2\)​H/16\+1\)​\(s\(\(S−2\)​H/16\+1\)\)=1,τ<\+∞\)\.\\displaystyle\\frac\{1\}\{7\}\\leq\\Pr\_\{\\text\{alg\},\\nu\}\\left\(\\hat\{\\pi\}\_\{h^\{\(\(S\-2\)H/16\+1\)\}\}\(s^\{\(\(S\-2\)H/16\+1\)\}\)=1,\\tau<\+\\infty\\right\)\.From the non\-decreasing order, we can conclude the Lemma\. ∎

Lemma[31](https://arxiv.org/html/2605.23182#Thmtheorem31)shows there exists at leastΩ​\(S​H\)\\Omega\(SH\)pairs of\(s,h\)\(s,h\)such that aδ\\delta\-PAC algorithm will identify its optimal action with probability1/71/7\. The following lemmas shows aδ\\delta\-PAC and symmetric algorithm must distinguish arm11and other arms by pulling themΩ​\(1ϵ2\)\\Omega\(\\frac\{1\}\{\\epsilon^\{2\}\}\)times in these pairs\(s,h\)\(s,h\)\.

###### Lemma 32\.

Apply aδ\\delta\-PAC symmetric algorithmalg~\\tilde\{\\text\{alg\}\}to a Uniform Instanceν\\nuwith\(S,A,H,r,ϵ\)\(S,A,H,r,\\epsilon\)\. If∃s0∈\[S−2\]\\exists s\_\{0\}\\in\[S\-2\]and evenh0h\_\{0\}such thatPrν,alg~⁡\(π^h0​\(s0\)=1,τ<\+∞\)≥17\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\hat\{\\pi\}\_\{h\_\{0\}\}\(s\_\{0\}\)=1,\\tau<\+\\infty\)\\geq\\frac\{1\}\{7\}, we have

Prν,alg~⁡\(π^h0​\(s0\)=1,nh0τ​\(s0,1\)\+nh0τ​\(s0,a\)≥1M​ϵ2,τ<\+∞\)≥114,∀a=2,⋯,A\.\\displaystyle\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\hat\{\\pi\}\_\{h\_\{0\}\}\(s\_\{0\}\)=1,n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},1\)\+n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},a\)\\geq\\frac\{1\}\{M\\epsilon^\{2\}\},\\tau<\+\\infty\)\\geq\\frac\{1\}\{14\},\\forall a=2,\\cdots,A\.whereM=2000M=2000\.

###### Proof of Lemma[32](https://arxiv.org/html/2605.23182#Thmtheorem32)\.

We first show fora′,a′′∈\{2,⋯,A\},a′≠a′′a^\{\\prime\},a^\{\\prime\\prime\}\\in\\\{2,\\cdots,A\\\},a^\{\\prime\}\\neq a^\{\\prime\\prime\}, we have

Prν,alg~⁡\(π^h0​\(s0\)=a′,τ<\+∞\)=Prν,alg~⁡\(π^h0​\(s0\)=a′′,τ<\+∞\)\\displaystyle\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\hat\{\\pi\}\_\{h\_\{0\}\}\(s\_\{0\}\)=a^\{\\prime\},\\tau<\+\\infty\)=\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\hat\{\\pi\}\_\{h\_\{0\}\}\(s\_\{0\}\)=a^\{\\prime\\prime\},\\tau<\+\\infty\)\(56\)Consider a permutationσ=\{σs,h\}i=1S​H\\sigma=\\\{\\sigma\_\{s,h\}\\\}\_\{i=1\}^\{SH\}such thatσs,h​\(a\)=\{a′s=s0,h=h0,a=a′′a′′s=s0,h=h0,a=a′aelse\\sigma\_\{s,h\}\(a\)=\\begin\{cases\}a^\{\\prime\}&s=s\_\{0\},h=h\_\{0\},a=a^\{\\prime\\prime\}\\\\ a^\{\\prime\\prime\}&s=s\_\{0\},h=h\_\{0\},a=a^\{\\prime\}\\\\ a&\\text\{else\}\\end\{cases\}\. From the definition of Uniform Instance, we knowph0​\(s′\|s0,a′\)=ph0​\(s′\|s0,a′′\)p\_\{h\_\{0\}\}\(s^\{\\prime\}\|s\_\{0\},a^\{\\prime\}\)=p\_\{h\_\{0\}\}\(s^\{\\prime\}\|s\_\{0\},a^\{\\prime\\prime\}\),rh0​\(s0,a′\)=rh0​\(s0,a′′\)r\_\{h\_\{0\}\}\(s\_\{0\},a^\{\\prime\}\)=r\_\{h\_\{0\}\}\(s\_\{0\},a^\{\\prime\\prime\}\), which meansνσ=ν\\nu\_\{\\sigma\}=\\nu\. Meanwhile, by the definition of symmetric algorithm, we know

Prν,alg~⁡\(π^h0​\(s0\)=a′,τ<\+∞\)=Prνσ,alg~⁡\(π^h0​\(s0\)=a′′,τ<\+∞\)\.\\displaystyle\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\hat\{\\pi\}\_\{h\_\{0\}\}\(s\_\{0\}\)=a^\{\\prime\},\\tau<\+\\infty\)=\\Pr\_\{\\nu\_\{\\sigma\},\\tilde\{\\text\{alg\}\}\}\(\\hat\{\\pi\}\_\{h\_\{0\}\}\(s\_\{0\}\)=a^\{\\prime\\prime\},\\tau<\+\\infty\)\.Together with the fact thatνσ=ν\\nu\_\{\\sigma\}=\\nu, we complete the proof\.

In the following, we assume1M​ϵ2\\frac\{1\}\{M\\epsilon^\{2\}\}is an integer for simplicity\. To derive Lemma[32](https://arxiv.org/html/2605.23182#Thmtheorem32), we prove it by contradiction\. Assume

Prν,alg~⁡\(π^h0​\(s0\)=1,nh0τ​\(s0,1\)\+nh0τ​\(s0,a\)≥1M​ϵ2,τ<\+∞\)<114\\displaystyle\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\hat\{\\pi\}\_\{h\_\{0\}\}\(s\_\{0\}\)=1,n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},1\)\+n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},a\)\\geq\\frac\{1\}\{M\\epsilon^\{2\}\},\\tau<\+\\infty\)<\\frac\{1\}\{14\}holds for somea∈\{2,3,⋯,A\}a\\in\\\{2,3,\\cdots,A\\\}\. From the assumption thatPrν,alg~⁡\(π^h0​\(s0\)=1,τ<\+∞\)≥17\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\hat\{\\pi\}\_\{h\_\{0\}\}\(s\_\{0\}\)=1,\\tau<\+\\infty\)\\geq\\frac\{1\}\{7\}, we know

Prν,alg~⁡\(π^h0​\(s0\)=1,nh0τ​\(s0,1\)\+nh0τ​\(s0,a\)<1M​ϵ2,τ<\+∞\)≥114\\displaystyle\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\hat\{\\pi\}\_\{h\_\{0\}\}\(s\_\{0\}\)=1,n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},1\)\+n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},a\)<\\frac\{1\}\{M\\epsilon^\{2\}\},\\tau<\+\\infty\)\\geq\\frac\{1\}\{14\}\(57\)Denote eventℰ\(1\)=\{π^h0​\(s0\)=1,τ<\+∞\}\\mathcal\{E\}^\{\(1\)\}=\\\{\\hat\{\\pi\}\_\{h\_\{0\}\}\(s\_\{0\}\)=1,\\tau<\+\\infty\\\},ℰ\(a\)=\{π^h0​\(s0\)=a,τ<\+∞\}\\mathcal\{E\}^\{\(a\)\}=\\\{\\hat\{\\pi\}\_\{h\_\{0\}\}\(s\_\{0\}\)=a,\\tau<\+\\infty\\\},ℰpull=\{nh0τ​\(s0,1\)\+nh0τ​\(s0,a\)<1M​ϵ2\}\\mathcal\{E\}\_\{\\text\{pull\}\}=\\\{n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},1\)\+n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},a\)<\\frac\{1\}\{M\\epsilon^\{2\}\}\\\}\. We consider a permutationσ\(a\)\\sigma^\{\(a\)\}withσs,h\(a\)​\(a~\)=\{1s=s0,h=h0,a~=aas=s0,h=h0,a~=1a~else\\sigma\_\{s,h\}^\{\(a\)\}\(\\tilde\{a\}\)=\\begin\{cases\}1&s=s\_\{0\},h=h\_\{0\},\\tilde\{a\}=a\\\\ a&s=s\_\{0\},h=h\_\{0\},\\tilde\{a\}=1\\\\ \\tilde\{a\}&\\text\{else\}\\end\{cases\}\. By the symmetry ofalg~\\tilde\{\\text\{alg\}\}and \([57](https://arxiv.org/html/2605.23182#A3.E57)\), we can conclude

Prνσ\(a\),alg~⁡\(ℰ\(a\)∩ℰpull\)≥114\.\\displaystyle\\Pr\_\{\\nu\_\{\\sigma^\{\(a\)\}\},\\tilde\{\\text\{alg\}\}\}\(\\mathcal\{E\}^\{\(a\)\}\\cap\\mathcal\{E\}\_\{\\text\{pull\}\}\)\\geq\\frac\{1\}\{14\}\.\(58\)Apply the Transportation Equality\(Lemma 18 inKaufmannet al\.\[[2016](https://arxiv.org/html/2605.23182#bib.bib4)\]\), which is based on the idea of Change of Measure, we have

Prν,alg~⁡\(ℰ\(a\)∩ℰpull\)=𝔼νσ\(a\),alg~​𝟙​\(ℰ\(a\)∩ℰpull\)​exp⁡\(−∑t=1nh0τ​\(s0,1\)KL^1,t−∑t=1nh0τ​\(s0,a\)KL^a,t\)\\displaystyle\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\mathcal\{E\}^\{\(a\)\}\\cap\\mathcal\{E\}\_\{\\text\{pull\}\}\)=\\mathbb\{E\}\_\{\\nu\_\{\\sigma^\{\(a\)\}\},\\tilde\{\\text\{alg\}\}\}\\mathds\{1\}\(\\mathcal\{E\}^\{\(a\)\}\\cap\\mathcal\{E\}\_\{\\text\{pull\}\}\)\\exp\\left\(\-\\sum\_\{t=1\}^\{n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},1\)\}\\hat\{\\text\{KL\}\}\_\{1,t\}\-\\sum\_\{t=1\}^\{n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},a\)\}\\hat\{\\text\{KL\}\}\_\{a,t\}\\right\)\(59\)whereKL^1,t=X1,t​log⁡r\+ϵr\+\(1−X1,t\)​log⁡1−r−ϵ1−r\\hat\{\\text\{KL\}\}\_\{1,t\}=X\_\{1,t\}\\log\\frac\{r\+\\epsilon\}\{r\}\+\(1\-X\_\{1,t\}\)\\log\\frac\{1\-r\-\\epsilon\}\{1\-r\},KL^a,t=Xa,t​log⁡rr\+ϵ\+\(1−X1,t\)​log⁡1−r1−r−ϵ\\hat\{\\text\{KL\}\}\_\{a,t\}=X\_\{a,t\}\\log\\frac\{r\}\{r\+\\epsilon\}\+\(1\-X\_\{1,t\}\)\\log\\frac\{1\-r\}\{1\-r\-\\epsilon\},X1,t\|\{X1,s\}s=1t−1∼Bern​\(r\+ϵ\)X\_\{1,t\}\|\\\{X\_\{1,s\}\\\}\_\{s=1\}^\{t\-1\}\\sim\\text\{Bern\}\(r\+\\epsilon\),Xa,t\|\{Xa,s\}s=1t−1∼Bern​\(r\)X\_\{a,t\}\|\\\{X\_\{a,s\}\\\}\_\{s=1\}^\{t\-1\}\\sim\\text\{Bern\}\(r\)\.\{X1,s\}s=1\+∞\\\{X\_\{1,s\}\\\}\_\{s=1\}^\{\+\\infty\}are independent with\{Xa,s\}s=1\+∞\\\{X\_\{a,s\}\\\}\_\{s=1\}^\{\+\\infty\}\.

Denote

kl​\(x,y\)=\\displaystyle\\text\{kl\}\(x,y\)=x​log⁡xy\+\(1−x\)​log⁡1−x1−y\\displaystyle x\\log\\frac\{x\}\{y\}\+\(1\-x\)\\log\\frac\{1\-x\}\{1\-y\}ℰkl\-cnt\(1\)=\\displaystyle\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(1\)\}=\{max1≤n≤1M​ϵ2​∑t=1n\(KL^1,t−kl​\(r\+ϵ,r\)\)≤14\}\\displaystyle\\left\\\{\\max\_\{1\\leq n\\leq\\frac\{1\}\{M\\epsilon^\{2\}\}\}\\sum\_\{t=1\}^\{n\}\(\\hat\{\\text\{KL\}\}\_\{1,t\}\-\\text\{kl\}\(r\+\\epsilon,r\)\)\\leq\\frac\{1\}\{4\}\\right\\\}ℰkl\-cnt\(a\)=\\displaystyle\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(a\)\}=\{max1≤n≤1M​ϵ2​∑t=1n\(KL^a,t−kl​\(r,r\+ϵ\)\)≤14\}\\displaystyle\\left\\\{\\max\_\{1\\leq n\\leq\\frac\{1\}\{M\\epsilon^\{2\}\}\}\\sum\_\{t=1\}^\{n\}\(\\hat\{\\text\{KL\}\}\_\{a,t\}\-\\text\{kl\}\(r,r\+\\epsilon\)\)\\leq\\frac\{1\}\{4\}\\right\\\}By the Lemma[36](https://arxiv.org/html/2605.23182#Thmtheorem36), we knowKL^1,t−kl​\(r\+ϵ,r\)\\hat\{\\text\{KL\}\}\_\{1,t\}\-\\text\{kl\}\(r\+\\epsilon,r\)andKL^a,t−kl​\(r,r\+ϵ\)\\hat\{\\text\{KL\}\}\_\{a,t\}\-\\text\{kl\}\(r,r\+\\epsilon\)are both8​ϵ28\\epsilon^\{2\}\-subgaussian\. By the Lemma[35](https://arxiv.org/html/2605.23182#Thmtheorem35), since2⋅⌊1M​ϵ2⌋⋅8​ϵ2≤16M2\\cdot\\lfloor\\frac\{1\}\{M\\epsilon^\{2\}\}\\rfloor\\cdot 8\\epsilon^\{2\}\\leq\\frac\{16\}\{M\}, we havePrνσ\(a\),alg~⁡\(ℰkl\-cnt\(1\)\)≥1−exp⁡\(−M16​116\)\\Pr\_\{\\nu\_\{\\sigma^\{\(a\)\}\},\\tilde\{\\text\{alg\}\}\}\(\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(1\)\}\)\\geq 1\-\\exp\(\-\\frac\{M\}\{16\}\\frac\{1\}\{16\}\),Prνσ\(a\),alg~⁡\(ℰkl\-cnt\(a\)\)≥1−exp⁡\(−M16​116\)\\Pr\_\{\\nu\_\{\\sigma^\{\(a\)\}\},\\tilde\{\\text\{alg\}\}\}\(\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(a\)\}\)\\geq 1\-\\exp\(\-\\frac\{M\}\{16\}\\frac\{1\}\{16\}\)\.

Following \([59](https://arxiv.org/html/2605.23182#A3.E59)\), we have

Prν,alg~⁡\(ℰ\(a\)∩ℰpull\)\\displaystyle\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\mathcal\{E\}^\{\(a\)\}\\cap\\mathcal\{E\}\_\{\\text\{pull\}\}\)≥\\displaystyle\\geq𝔼νσ\(a\),alg~​𝟙​\(ℰ\(a\)∩ℰpull∩ℰkl\-cnt\(1\)∩ℰkl\-cnt\(a\)\)​exp⁡\(−∑t=1nh0τ​\(s0,1\)KL^1,t−∑t=1nh0τ​\(s0,a\)KL^a,t\)\\displaystyle\\mathbb\{E\}\_\{\\nu\_\{\\sigma^\{\(a\)\}\},\\tilde\{\\text\{alg\}\}\}\\mathds\{1\}\(\\mathcal\{E\}^\{\(a\)\}\\cap\\mathcal\{E\}\_\{\\text\{pull\}\}\\cap\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(1\)\}\\cap\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(a\)\}\)\\exp\\left\(\-\\sum\_\{t=1\}^\{n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},1\)\}\\hat\{\\text\{KL\}\}\_\{1,t\}\-\\sum\_\{t=1\}^\{n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},a\)\}\\hat\{\\text\{KL\}\}\_\{a,t\}\\right\)=\\displaystyle=𝔼νσ\(a\),alg~​𝟙​\(ℰ\(a\)∩ℰpull∩ℰkl\-cnt\(1\)∩ℰkl\-cnt\(a\)\)​exp⁡\(−nh0τ​\(s0,1\)​kl​\(r\+ϵ,r\)−nh0τ​\(s0,a\)​kl​\(r,r\+ϵ\)\)\\displaystyle\\mathbb\{E\}\_\{\\nu\_\{\\sigma^\{\(a\)\}\},\\tilde\{\\text\{alg\}\}\}\\mathds\{1\}\(\\mathcal\{E\}^\{\(a\)\}\\cap\\mathcal\{E\}\_\{\\text\{pull\}\}\\cap\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(1\)\}\\cap\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(a\)\}\)\\exp\(\-n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},1\)\\text\{kl\}\(r\+\\epsilon,r\)\-n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},a\)\\text\{kl\}\(r,r\+\\epsilon\)\)exp⁡\(−∑t=1nh0τ​\(s0,1\)\(KL^1,t−kl​\(r\+ϵ,r\)\)−∑t=1nh0τ​\(s0,a\)\(KL^a,t−kl​\(r,r\+ϵ\)\)\)\\displaystyle\\exp\\left\(\-\\sum\_\{t=1\}^\{n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},1\)\}\\left\(\\hat\{\\text\{KL\}\}\_\{1,t\}\-\\text\{kl\}\(r\+\\epsilon,r\)\\right\)\-\\sum\_\{t=1\}^\{n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},a\)\}\\left\(\\hat\{\\text\{KL\}\}\_\{a,t\}\-\\text\{kl\}\(r,r\+\\epsilon\)\\right\)\\right\)≥\\displaystyle\\geq𝔼νσ\(a\),alg~​𝟙​\(ℰ\(a\)∩ℰpull∩ℰkl\-cnt\(1\)∩ℰkl\-cnt\(a\)\)​exp⁡\(−3​ϵ2M​ϵ2−3​ϵ2M​ϵ2\)​exp⁡\(−12\)\\displaystyle\\mathbb\{E\}\_\{\\nu\_\{\\sigma^\{\(a\)\}\},\\tilde\{\\text\{alg\}\}\}\\mathds\{1\}\(\\mathcal\{E\}^\{\(a\)\}\\cap\\mathcal\{E\}\_\{\\text\{pull\}\}\\cap\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(1\)\}\\cap\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(a\)\}\)\\exp\(\-\\frac\{3\\epsilon^\{2\}\}\{M\\epsilon^\{2\}\}\-\\frac\{3\\epsilon^\{2\}\}\{M\\epsilon^\{2\}\}\)\\exp\\left\(\-\\frac\{1\}\{2\}\\right\)\(60\)≥\\displaystyle\\geqexp⁡\(−6M−12\)​\(Prνσ\(a\),alg~⁡\(ℰ\(a\)∩ℰpull\)−Prνσ\(a\),alg~⁡\(¬ℰkl\-cnt\(1\)\)−Prνσ\(a\),alg~⁡\(¬ℰkl\-cnt\(a\)\)\)\\displaystyle\\exp\\left\(\-\\frac\{6\}\{M\}\-\\frac\{1\}\{2\}\\right\)\\left\(\\Pr\_\{\\nu\_\{\\sigma^\{\(a\)\}\},\\tilde\{\\text\{alg\}\}\}\\left\(\\mathcal\{E\}^\{\(a\)\}\\cap\\mathcal\{E\}\_\{\\text\{pull\}\}\\right\)\-\\Pr\_\{\\nu\_\{\\sigma^\{\(a\)\}\},\\tilde\{\\text\{alg\}\}\}\(\\neg\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(1\)\}\)\-\\Pr\_\{\\nu\_\{\\sigma^\{\(a\)\}\},\\tilde\{\\text\{alg\}\}\}\(\\neg\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(a\)\}\)\\right\)≥\\displaystyle\\geqexp⁡\(−6M−12\)​\(Prνσ\(a\),alg~⁡\(ℰ\(a\)∩ℰpull\)−2​exp⁡\(−M256\)\)\.\\displaystyle\\exp\\left\(\-\\frac\{6\}\{M\}\-\\frac\{1\}\{2\}\\right\)\\left\(\\Pr\_\{\\nu\_\{\\sigma^\{\(a\)\}\},\\tilde\{\\text\{alg\}\}\}\\left\(\\mathcal\{E\}^\{\(a\)\}\\cap\\mathcal\{E\}\_\{\\text\{pull\}\}\\right\)\-2\\exp\(\-\\frac\{M\}\{256\}\)\\right\)\.\(61\)Step \([60](https://arxiv.org/html/2605.23182#A3.E60)\) is by the fact that under eventℰpull\\mathcal\{E\}\_\{\\text\{pull\}\},nh0τ​\(s0,1\),nh0τ​\(s0,a\)≤1M​ϵ2n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},1\),n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},a\)\\leq\\frac\{1\}\{M\\epsilon^\{2\}\}\. Given this, eventℰkl\-cnt\(1\),ℰkl\-cnt\(a\)\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(1\)\},\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(a\)\}guarantee that

∑t=1nh0τ​\(s0,1\)\(KL^1,t−kl​\(r\+ϵ,r\)\)≤max1≤n≤1M​ϵ2​∑t=1n\(KL^1,t−kl​\(r\+ϵ,r\)\)≤14\\displaystyle\\sum\_\{t=1\}^\{n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},1\)\}\\left\(\\hat\{\\text\{KL\}\}\_\{1,t\}\-\\text\{kl\}\(r\+\\epsilon,r\)\\right\)\\leq\\max\_\{1\\leq n\\leq\\frac\{1\}\{M\\epsilon^\{2\}\}\}\\sum\_\{t=1\}^\{n\}\\left\(\\hat\{\\text\{KL\}\}\_\{1,t\}\-\\text\{kl\}\(r\+\\epsilon,r\)\\right\)\\leq\\frac\{1\}\{4\}∑t=1nh0τ​\(s0,a\)\(KL^a,t−kl​\(r,r\+ϵ\)\)≤max1≤n≤1M​ϵ2​∑t=1n\(KL^a,t−kl​\(r,r\+ϵ\)\)≤14\\displaystyle\\sum\_\{t=1\}^\{n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},a\)\}\\left\(\\hat\{\\text\{KL\}\}\_\{a,t\}\-\\text\{kl\}\(r,r\+\\epsilon\)\\right\)\\leq\\max\_\{1\\leq n\\leq\\frac\{1\}\{M\\epsilon^\{2\}\}\}\\sum\_\{t=1\}^\{n\}\\left\(\\hat\{\\text\{KL\}\}\_\{a,t\}\-\\text\{kl\}\(r,r\+\\epsilon\)\\right\)\\leq\\frac\{1\}\{4\}In addition, Lemma[37](https://arxiv.org/html/2605.23182#Thmtheorem37)suggests thatkl​\(r\+ϵ,r\),kl​\(r,r\+ϵ\)≤3​ϵ2\\text\{kl\}\(r\+\\epsilon,r\),\\text\{kl\}\(r,r\+\\epsilon\)\\leq 3\\epsilon^\{2\}, since we require14<r<r\+ϵ<34\\frac\{1\}\{4\}<r<r\+\\epsilon<\\frac\{3\}\{4\}in the definition of Uniform Instance\. Step \([61](https://arxiv.org/html/2605.23182#A3.E61)\) is by the inequalitiesPrνσ\(a\),alg~⁡\(ℰkl\-cnt\(1\)\)≥1−exp⁡\(−M256\)\\Pr\_\{\\nu\_\{\\sigma^\{\(a\)\}\},\\tilde\{\\text\{alg\}\}\}\(\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(1\)\}\)\\geq 1\-\\exp\(\-\\frac\{M\}\{256\}\),Prνσ\(a\),alg~⁡\(ℰkl\-cnt\(a\)\)≥1−exp⁡\(−M256\)\\Pr\_\{\\nu\_\{\\sigma^\{\(a\)\}\},\\tilde\{\\text\{alg\}\}\}\(\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(a\)\}\)\\geq 1\-\\exp\(\-\\frac\{M\}\{256\}\)\.

Combining \([58](https://arxiv.org/html/2605.23182#A3.E58)\) and \([61](https://arxiv.org/html/2605.23182#A3.E61)\), we have

Prν,alg~⁡\(ℰ\(a\)∩ℰpull\)≥exp⁡\(−6M−12\)​\(114−2​exp⁡\(−M256\)\)≥1100,\\displaystyle\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\mathcal\{E\}^\{\(a\)\}\\cap\\mathcal\{E\}\_\{\\text\{pull\}\}\)\\geq\\exp\\left\(\-\\frac\{6\}\{M\}\-\\frac\{1\}\{2\}\\right\)\\left\(\\frac\{1\}\{14\}\-2\\exp\(\-\\frac\{M\}\{256\}\)\\right\)\\geq\\frac\{1\}\{100\},which impliesPrν,alg~⁡\(π^h0​\(s0\)=a,τ<\+∞\)≥1100\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\hat\{\\pi\}\_\{h\_\{0\}\}\(s\_\{0\}\)=a,\\tau<\+\\infty\)\\geq\\frac\{1\}\{100\}\. \([56](https://arxiv.org/html/2605.23182#A3.E56)\) implies

Prν,alg~⁡\(π^h0​\(s0\)∈\{2,⋯,A\},τ<\+∞\)≥A100\.\\displaystyle\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\\big\(\\hat\{\\pi\}\_\{h\_\{0\}\}\(s\_\{0\}\)\\in\\\{2,\\cdots,A\\\},\\tau<\+\\infty\\big\)\\geq\\frac\{A\}\{100\}\.By the requirement ofA\>100A\>100in the definition of Uniform Instance, we have found a contradiction, suggesting we have proved Lemma[32](https://arxiv.org/html/2605.23182#Thmtheorem32)\. ∎

Based on Lemma[32](https://arxiv.org/html/2605.23182#Thmtheorem32), the following lemma claims that a symmetric algorithm is unable to distinguish action11andaawhen the total pulling times of action11andaais below12000​ϵ2\\frac\{1\}\{2000\\epsilon^\{2\}\}\.

###### Lemma 33\.

Apply aδ\\delta\-PAC symmetric algorithmalg~\\tilde\{\\text\{alg\}\}to a Uniform Instanceν\\nuwith\(S,A,H,r,ϵ\)\(S,A,H,r,\\epsilon\)\. If∃s0∈\[S−2\]\\exists s\_\{0\}\\in\[S\-2\]and evenh0h\_\{0\}such thatPrν,alg~⁡\(π^h0​\(s0\)=1,τ<\+∞\)≥17\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\hat\{\\pi\}\_\{h\_\{0\}\}\(s\_\{0\}\)=1,\\tau<\+\\infty\)\\geq\\frac\{1\}\{7\}, we have

Prν,alg~⁡\(nh0τ​\(s0,a\)≥12​M​ϵ2,τ<\+∞\)≥150,∀a=2,⋯,A\.\\displaystyle\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},a\)\\geq\\frac\{1\}\{2M\\epsilon^\{2\}\},\\tau<\+\\infty\)\\geq\\frac\{1\}\{50\},\\forall a=2,\\cdots,A\.whereM=2000M=2000\.

###### Proof of Lemma[33](https://arxiv.org/html/2605.23182#Thmtheorem33)\.

We first show fora′,a′′∈\{2,⋯,A\},a′≠a′′a^\{\\prime\},a^\{\\prime\\prime\}\\in\\\{2,\\cdots,A\\\},a^\{\\prime\}\\neq a^\{\\prime\\prime\}, we have

Prν,alg~⁡\(nh0τ​\(s0,a′\)≥12​M​ϵ2,τ<\+∞\)=Prν,alg~⁡\(nh0τ​\(s0,a′′\)≥12​M​ϵ2,τ<\+∞\)\.\\displaystyle\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},a^\{\\prime\}\)\\geq\\frac\{1\}\{2M\\epsilon^\{2\}\},\\tau<\+\\infty\)=\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},a^\{\\prime\\prime\}\)\\geq\\frac\{1\}\{2M\\epsilon^\{2\}\},\\tau<\+\\infty\)\.\(62\)Similar to the statement at the proof of Lemma[32](https://arxiv.org/html/2605.23182#Thmtheorem32), consider a permutationσ=\{σs,h\}i=1S​H\\sigma=\\\{\\sigma\_\{s,h\}\\\}\_\{i=1\}^\{SH\}such thatσs,h​\(a\)=\{a′s=s0,h=h0,a=a′′a′′s=s0,h=h0,a=a′aelse\\sigma\_\{s,h\}\(a\)=\\begin\{cases\}a^\{\\prime\}&s=s\_\{0\},h=h\_\{0\},a=a^\{\\prime\\prime\}\\\\ a^\{\\prime\\prime\}&s=s\_\{0\},h=h\_\{0\},a=a^\{\\prime\}\\\\ a&\\text\{else\}\\end\{cases\}\. From the definition of Uniform Instance, we knowph0​\(s′\|s0,a′\)=ph0​\(s′\|s0,a′′\)p\_\{h\_\{0\}\}\(s^\{\\prime\}\|s\_\{0\},a^\{\\prime\}\)=p\_\{h\_\{0\}\}\(s^\{\\prime\}\|s\_\{0\},a^\{\\prime\\prime\}\),rh0​\(s0,a′\)=rh0​\(s0,a′′\)r\_\{h\_\{0\}\}\(s\_\{0\},a^\{\\prime\}\)=r\_\{h\_\{0\}\}\(s\_\{0\},a^\{\\prime\\prime\}\), which meansνσ=ν\\nu\_\{\\sigma\}=\\nu\. Meanwhile, by the definition of symmetric algorithm, we know

Prν,alg~⁡\(nh0τ​\(s0,a′\)≥12​M​ϵ2,τ<\+∞\)=Prνσ,alg~⁡\(nh0τ​\(s0,a′′\)≥12​M​ϵ2,τ<\+∞\)\.\\displaystyle\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},a^\{\\prime\}\)\\geq\\frac\{1\}\{2M\\epsilon^\{2\}\},\\tau<\+\\infty\)=\\Pr\_\{\\nu\_\{\\sigma\},\\tilde\{\\text\{alg\}\}\}\(n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},a^\{\\prime\\prime\}\)\\geq\\frac\{1\}\{2M\\epsilon^\{2\}\},\\tau<\+\\infty\)\.Together with the fact thatνσ=ν\\nu\_\{\\sigma\}=\\nu, we complete the proof\.

\([62](https://arxiv.org/html/2605.23182#A3.E62)\) suggests that we suffice to provePrν,alg~⁡\(nh0τ​\(s0,a\)≥12​M​ϵ2,τ<\+∞\)≥150\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},a\)\\geq\\frac\{1\}\{2M\\epsilon^\{2\}\},\\tau<\+\\infty\)\\geq\\frac\{1\}\{50\}fora=2a=2\. In the following proof, we fixa=2a=2and we assume1M​ϵ2\\frac\{1\}\{M\\epsilon^\{2\}\}is an integer for simplicity\.

Defineτ~1,a=min\{t:nh0t\(s0,1\)\+nh0t\(s0,a\)=⌊1M​ϵ2\}⌋\\tilde\{\\tau\}\_\{1,a\}=\\min\\\{t:n^\{t\}\_\{h\_\{0\}\}\(s\_\{0\},1\)\+n^\{t\}\_\{h\_\{0\}\}\(s\_\{0\},a\)=\\lfloor\\frac\{1\}\{M\\epsilon^\{2\}\}\\\}\\rfloor\. From Lemma[32](https://arxiv.org/html/2605.23182#Thmtheorem32), we know

Prν,alg~⁡\(τ~1,a≤τ<\+∞\)\\displaystyle\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\tilde\{\\tau\}\_\{1,a\}\\leq\\tau<\+\\infty\)=\\displaystyle=Prν,alg~⁡\(nh0τ​\(s0,1\)\+nh0τ​\(s0,a\)≥1M​ϵ2,τ<\+∞\)\\displaystyle\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},1\)\+n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},a\)\\geq\\frac\{1\}\{M\\epsilon^\{2\}\},\\tau<\+\\infty\)≥\\displaystyle\\geqPrν,alg~⁡\(π^h0​\(s0\)=1,nh0τ​\(s0,1\)\+nh0τ​\(s0,a\)≥1M​ϵ2,τ<\+∞\)\\displaystyle\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\hat\{\\pi\}\_\{h\_\{0\}\}\(s\_\{0\}\)=1,n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},1\)\+n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},a\)\\geq\\frac\{1\}\{M\\epsilon^\{2\}\},\\tau<\+\\infty\)≥\\displaystyle\\geq114\.\\displaystyle\\frac\{1\}\{14\}\.Denoteℰ1,apull=\{τ~1,a≤τ<\+∞,nh0τ~1,a\(s0,1\)≤nh0τ~1,a\(s0,a\)\}\\mathcal\{E\}\_\{1,a\}^\{\\text\{pull\}\}=\\\{\\tilde\{\\tau\}\_\{1,a\}\\leq\\tau<\+\\infty,n^\{\\tilde\{\\tau\}\_\{1,a\}\}\_\{h\_\{0\}\}\(s\_\{0\},1\)\\leq n^\{\\tilde\{\\tau\}\_\{1,a\}\}\_\{h\_\{0\}\}\(s\_\{0\},a\)\\\}andℰa,1pull=\{τ~1,a≤τ<\+∞,nh0τ~1,a\(s0,a\)≤nh0τ~1,a\(s0,1\)\}\\mathcal\{E\}\_\{a,1\}^\{\\text\{pull\}\}=\\\{\\tilde\{\\tau\}\_\{1,a\}\\leq\\tau<\+\\infty,n^\{\\tilde\{\\tau\}\_\{1,a\}\}\_\{h\_\{0\}\}\(s\_\{0\},a\)\\leq n^\{\\tilde\{\\tau\}\_\{1,a\}\}\_\{h\_\{0\}\}\(s\_\{0\},1\)\\\}\. Evident to see

Prν,alg~⁡\(ℰ1,apull\)\+Prν,alg~⁡\(ℰa,1pull\)≥Prν,alg~⁡\(τ~1,a≤τ<\+∞\)≥114\.\\displaystyle\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\mathcal\{E\}\_\{1,a\}^\{\\text\{pull\}\}\)\+\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\mathcal\{E\}\_\{a,1\}^\{\\text\{pull\}\}\)\\geq\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\tilde\{\\tau\}\_\{1,a\}\\leq\\tau<\+\\infty\)\\geq\\frac\{1\}\{14\}\.\(63\)
We consider a permutationσ\(a\)\\sigma^\{\(a\)\}withσs,h\(a\)​\(a~\)=\{1s=s0,h=h0,a~=aas=s0,h=h0,a~=1a~else\\sigma\_\{s,h\}^\{\(a\)\}\(\\tilde\{a\}\)=\\begin\{cases\}1&s=s\_\{0\},h=h\_\{0\},\\tilde\{a\}=a\\\\ a&s=s\_\{0\},h=h\_\{0\},\\tilde\{a\}=1\\\\ \\tilde\{a\}&\\text\{else\}\\end\{cases\}\. By the symmetry ofalg~\\tilde\{\\text\{alg\}\}, we know

Prν,alg~⁡\(ℰ1,apull\)=Prνσ\(a\),alg~⁡\(ℰa,1pull\),Prν,alg~⁡\(ℰa,1pull\)=Prνσ\(a\),alg~⁡\(ℰ1,apull\)\.\\displaystyle\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\mathcal\{E\}\_\{1,a\}^\{\\text\{pull\}\}\)=\\Pr\_\{\\nu\_\{\\sigma^\{\(a\)\}\},\\tilde\{\\text\{alg\}\}\}\(\\mathcal\{E\}\_\{a,1\}^\{\\text\{pull\}\}\),\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\mathcal\{E\}\_\{a,1\}^\{\\text\{pull\}\}\)=\\Pr\_\{\\nu\_\{\\sigma^\{\(a\)\}\},\\tilde\{\\text\{alg\}\}\}\(\\mathcal\{E\}\_\{1,a\}^\{\\text\{pull\}\}\)\.\(64\)
Apply the Transportation Equality\(Lemma 18 inKaufmannet al\.\[[2016](https://arxiv.org/html/2605.23182#bib.bib4)\]\) on the eventℰ1,apull\\mathcal\{E\}\_\{1,a\}^\{\\text\{pull\}\}, we can derive

Prνσ\(a\),alg~⁡\(ℰa,1pull\)=𝔼ν,alg~​𝟙​\(ℰa,1pull\)​exp⁡\(−∑t=1nh0τ~1,a​\(s0,1\)KL^1,t−∑t=1nh0τ~1,a​\(s0,a\)KL^a,t\),\\displaystyle\\Pr\_\{\\nu\_\{\\sigma^\{\(a\)\}\},\\tilde\{\\text\{alg\}\}\}\(\\mathcal\{E\}\_\{a,1\}^\{\\text\{pull\}\}\)=\\mathbb\{E\}\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\\mathds\{1\}\(\\mathcal\{E\}\_\{a,1\}^\{\\text\{pull\}\}\)\\exp\\left\(\-\\sum\_\{t=1\}^\{n^\{\\tilde\{\\tau\}\_\{1,a\}\}\_\{h\_\{0\}\}\(s\_\{0\},1\)\}\\hat\{\\text\{KL\}\}\_\{1,t\}\-\\sum\_\{t=1\}^\{n^\{\\tilde\{\\tau\}\_\{1,a\}\}\_\{h\_\{0\}\}\(s\_\{0\},a\)\}\\hat\{\\text\{KL\}\}\_\{a,t\}\\right\),\(65\)whereKL^1,t=X1,t​log⁡r\+ϵr\+\(1−X1,t\)​log⁡1−r−ϵ1−r\\hat\{\\text\{KL\}\}\_\{1,t\}=X\_\{1,t\}\\log\\frac\{r\+\\epsilon\}\{r\}\+\(1\-X\_\{1,t\}\)\\log\\frac\{1\-r\-\\epsilon\}\{1\-r\},KL^a,t=Xa,t​log⁡rr\+ϵ\+\(1−X1,t\)​log⁡1−r1−r−ϵ\\hat\{\\text\{KL\}\}\_\{a,t\}=X\_\{a,t\}\\log\\frac\{r\}\{r\+\\epsilon\}\+\(1\-X\_\{1,t\}\)\\log\\frac\{1\-r\}\{1\-r\-\\epsilon\},X1,t\|\{X1,s\}s=1t−1∼Bern​\(r\+ϵ\)X\_\{1,t\}\|\\\{X\_\{1,s\}\\\}\_\{s=1\}^\{t\-1\}\\sim\\text\{Bern\}\(r\+\\epsilon\),Xa,t\|\{Xa,s\}s=1t−1∼Bern​\(r\)X\_\{a,t\}\|\\\{X\_\{a,s\}\\\}\_\{s=1\}^\{t\-1\}\\sim\\text\{Bern\}\(r\)\.\{X1,s\}s=1\+∞\\\{X\_\{1,s\}\\\}\_\{s=1\}^\{\+\\infty\}are independent with\{Xa,s\}s=1\+∞\\\{X\_\{a,s\}\\\}\_\{s=1\}^\{\+\\infty\}\. Similar to the proof of Lemma[32](https://arxiv.org/html/2605.23182#Thmtheorem32), we introduce

kl​\(x,y\)=\\displaystyle\\text\{kl\}\(x,y\)=x​log⁡xy\+\(1−x\)​log⁡1−x1−y\\displaystyle x\\log\\frac\{x\}\{y\}\+\(1\-x\)\\log\\frac\{1\-x\}\{1\-y\}ℰkl\-cnt\(1\)=\\displaystyle\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(1\)\}=\{max1≤n≤1M​ϵ2​∑t=1n\(KL^1,t−kl​\(r\+ϵ,r\)\)≤14\}\\displaystyle\\left\\\{\\max\_\{1\\leq n\\leq\\frac\{1\}\{M\\epsilon^\{2\}\}\}\\sum\_\{t=1\}^\{n\}\(\\hat\{\\text\{KL\}\}\_\{1,t\}\-\\text\{kl\}\(r\+\\epsilon,r\)\)\\leq\\frac\{1\}\{4\}\\right\\\}ℰkl\-cnt\(a\)=\\displaystyle\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(a\)\}=\{max1≤n≤1M​ϵ2​∑t=1n\(KL^a,t−kl​\(r,r\+ϵ\)\)≤14\}\\displaystyle\\left\\\{\\max\_\{1\\leq n\\leq\\frac\{1\}\{M\\epsilon^\{2\}\}\}\\sum\_\{t=1\}^\{n\}\(\\hat\{\\text\{KL\}\}\_\{a,t\}\-\\text\{kl\}\(r,r\+\\epsilon\)\)\\leq\\frac\{1\}\{4\}\\right\\\}Following \([65](https://arxiv.org/html/2605.23182#A3.E65)\) and \([64](https://arxiv.org/html/2605.23182#A3.E64)\), we have

Prν,alg~⁡\(ℰ1,apull\)\\displaystyle\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\mathcal\{E\}\_\{1,a\}^\{\\text\{pull\}\}\)≥\\displaystyle\\geq𝔼ν,alg~​𝟙​\(ℰa,1pull∩ℰkl\-cnt\(1\)∩ℰkl\-cnt\(a\)\)​exp⁡\(−∑t=1nh0τ~1,a​\(s0,1\)KL^1,t−∑t=1nh0τ~1,a​\(s0,a\)KL^a,t\)\\displaystyle\\mathbb\{E\}\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\\mathds\{1\}\(\\mathcal\{E\}\_\{a,1\}^\{\\text\{pull\}\}\\cap\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(1\)\}\\cap\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(a\)\}\)\\exp\\left\(\-\\sum\_\{t=1\}^\{n^\{\\tilde\{\\tau\}\_\{1,a\}\}\_\{h\_\{0\}\}\(s\_\{0\},1\)\}\\hat\{\\text\{KL\}\}\_\{1,t\}\-\\sum\_\{t=1\}^\{n^\{\\tilde\{\\tau\}\_\{1,a\}\}\_\{h\_\{0\}\}\(s\_\{0\},a\)\}\\hat\{\\text\{KL\}\}\_\{a,t\}\\right\)=\\displaystyle=𝔼ν,alg~​𝟙​\(ℰa,1pull∩ℰkl\-cnt\(1\)∩ℰkl\-cnt\(a\)\)​exp⁡\(−nh0τ~1,a​\(s0,1\)​kl​\(r\+ϵ,r\)−nh0τ~1,a​\(s0,a\)​kl​\(r,r\+ϵ\)\)\\displaystyle\\mathbb\{E\}\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\\mathds\{1\}\(\\mathcal\{E\}\_\{a,1\}^\{\\text\{pull\}\}\\cap\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(1\)\}\\cap\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(a\)\}\)\\exp\(\-n^\{\\tilde\{\\tau\}\_\{1,a\}\}\_\{h\_\{0\}\}\(s\_\{0\},1\)\\text\{kl\}\(r\+\\epsilon,r\)\-n^\{\\tilde\{\\tau\}\_\{1,a\}\}\_\{h\_\{0\}\}\(s\_\{0\},a\)\\text\{kl\}\(r,r\+\\epsilon\)\)exp⁡\(−∑t=1nh0τ~1,a​\(s0,1\)\(KL^1,t−kl​\(r\+ϵ,r\)\)−∑t=1nh0τ~1,a​\(s0,a\)\(KL^a,t−kl​\(r,r\+ϵ\)\)\)\\displaystyle\\exp\\left\(\-\\sum\_\{t=1\}^\{n^\{\\tilde\{\\tau\}\_\{1,a\}\}\_\{h\_\{0\}\}\(s\_\{0\},1\)\}\\big\(\\hat\{\\text\{KL\}\}\_\{1,t\}\-\\text\{kl\}\(r\+\\epsilon,r\)\\big\)\-\\sum\_\{t=1\}^\{n^\{\\tilde\{\\tau\}\_\{1,a\}\}\_\{h\_\{0\}\}\(s\_\{0\},a\)\}\\big\(\\hat\{\\text\{KL\}\}\_\{a,t\}\-\\text\{kl\}\(r,r\+\\epsilon\)\\big\)\\right\)≥\\displaystyle\\geq𝔼ν,alg~​𝟙​\(ℰa,1pull∩ℰkl\-cnt\(1\)∩ℰkl\-cnt\(a\)\)​exp⁡\(−3​ϵ2M​ϵ2−3​ϵ2M​ϵ2\)​exp⁡\(−12\)\\displaystyle\\mathbb\{E\}\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\\mathds\{1\}\(\\mathcal\{E\}\_\{a,1\}^\{\\text\{pull\}\}\\cap\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(1\)\}\\cap\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(a\)\}\)\\exp\(\-\\frac\{3\\epsilon^\{2\}\}\{M\\epsilon^\{2\}\}\-\\frac\{3\\epsilon^\{2\}\}\{M\\epsilon^\{2\}\}\)\\exp\(\-\\frac\{1\}\{2\}\)\(66\)≥\\displaystyle\\geqexp⁡\(−6M−12\)​\(Prν,alg~⁡\(ℰa,1pull\)−Prν,alg~⁡\(¬ℰkl\-cnt\(1\)\)−Prν,alg~⁡\(¬ℰkl\-cnt\(a\)\)\)\\displaystyle\\exp\(\-\\frac\{6\}\{M\}\-\\frac\{1\}\{2\}\)\\left\(\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\mathcal\{E\}\_\{a,1\}^\{\\text\{pull\}\}\)\-\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\neg\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(1\)\}\)\-\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\neg\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(a\)\}\)\\right\)≥\\displaystyle\\geqexp⁡\(−6M−12\)​\(Prν,alg~⁡\(ℰa,1pull\)−2​exp⁡\(−M256\)\)\\displaystyle\\exp\(\-\\frac\{6\}\{M\}\-\\frac\{1\}\{2\}\)\\left\(\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\mathcal\{E\}\_\{a,1\}^\{\\text\{pull\}\}\)\-2\\exp\(\-\\frac\{M\}\{256\}\)\\right\)\(67\)Step \([66](https://arxiv.org/html/2605.23182#A3.E66)\) is by the fact thatnh0τ~1,a​\(s0,1\),nh0τ~1,a​\(s0,a\)≤nh0τ~1,a​\(s0,1\)\+nh0τ~1,a​\(s0,a\)≤1M​ϵ2n^\{\\tilde\{\\tau\}\_\{1,a\}\}\_\{h\_\{0\}\}\(s\_\{0\},1\),n^\{\\tilde\{\\tau\}\_\{1,a\}\}\_\{h\_\{0\}\}\(s\_\{0\},a\)\\leq n^\{\\tilde\{\\tau\}\_\{1,a\}\}\_\{h\_\{0\}\}\(s\_\{0\},1\)\+n^\{\\tilde\{\\tau\}\_\{1,a\}\}\_\{h\_\{0\}\}\(s\_\{0\},a\)\\leq\\frac\{1\}\{M\\epsilon^\{2\}\}, which is guaranteed by the definition of stopping timeτ~1,a\\tilde\{\\tau\}\_\{1,a\}\. Andℰkl\-cnt\(1\),ℰkl\-cnt\(a\)\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(1\)\},\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(a\)\}guarantee

exp⁡\(−∑t=1nh0τ~1,a​\(s0,1\)\(KL^1,t−kl​\(r\+ϵ,r\)\)−∑t=1nh0τ~1,a​\(s0,a\)\(KL^a,t−kl​\(r,r\+ϵ\)\)\)≥exp⁡\(12\)\.\\displaystyle\\exp\\left\(\-\\sum\_\{t=1\}^\{n^\{\\tilde\{\\tau\}\_\{1,a\}\}\_\{h\_\{0\}\}\(s\_\{0\},1\)\}\\big\(\\hat\{\\text\{KL\}\}\_\{1,t\}\-\\text\{kl\}\(r\+\\epsilon,r\)\\big\)\-\\sum\_\{t=1\}^\{n^\{\\tilde\{\\tau\}\_\{1,a\}\}\_\{h\_\{0\}\}\(s\_\{0\},a\)\}\\big\(\\hat\{\\text\{KL\}\}\_\{a,t\}\-\\text\{kl\}\(r,r\+\\epsilon\)\\big\)\\right\)\\geq\\exp\(\\frac\{1\}\{2\}\)\.Step \([67](https://arxiv.org/html/2605.23182#A3.E67)\) is by the fact thatPrν,alg~\(¬ℰkl\-cnt\(1\)\),Prν,alg~\(¬ℰkl\-cnt\(a\)\),≤exp\(−M256\)\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\neg\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(1\)\}\),\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\neg\\mathcal\{E\}\_\{\\text\{kl\-cnt\}\}^\{\(a\)\}\),\\leq\\exp\(\-\\frac\{M\}\{256\}\), proved by Lemma[35](https://arxiv.org/html/2605.23182#Thmtheorem35)and[36](https://arxiv.org/html/2605.23182#Thmtheorem36)\.

Combining \([63](https://arxiv.org/html/2605.23182#A3.E63)\) and \([67](https://arxiv.org/html/2605.23182#A3.E67)\), we have

Prν,alg~⁡\(ℰ1,apull\)≥exp⁡\(−6M−12\)​\(114−Prν,alg~⁡\(ℰ1,apull\)−2​exp⁡\(−M256\)\)\\displaystyle\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\mathcal\{E\}\_\{1,a\}^\{\\text\{pull\}\}\)\\geq\\exp\(\-\\frac\{6\}\{M\}\-\\frac\{1\}\{2\}\)\\left\(\\frac\{1\}\{14\}\-\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\mathcal\{E\}\_\{1,a\}^\{\\text\{pull\}\}\)\-2\\exp\(\-\\frac\{M\}\{256\}\)\\right\)⇔\\displaystyle\\LeftrightarrowPrν,alg~⁡\(ℰ1,apull\)≥exp⁡\(−6M−12\)​\(114−2​exp⁡\(−M256\)\)1\+exp⁡\(−6M−12\)\\displaystyle\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\mathcal\{E\}\_\{1,a\}^\{\\text\{pull\}\}\)\\geq\\frac\{\\exp\(\-\\frac\{6\}\{M\}\-\\frac\{1\}\{2\}\)\\left\(\\frac\{1\}\{14\}\-2\\exp\(\-\\frac\{M\}\{256\}\)\\right\)\}\{1\+\\exp\(\-\\frac\{6\}\{M\}\-\\frac\{1\}\{2\}\)\}⇒\\displaystyle\\RightarrowPrν,alg~⁡\(ℰ1,apull\)≥150\\displaystyle\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\mathcal\{E\}\_\{1,a\}^\{\\text\{pull\}\}\)\\geq\\frac\{1\}\{50\}⇒\\displaystyle\\RightarrowPrν,alg~⁡\(τ~1,a≤τ<\+∞,nh0τ~1,a​\(s0,a\)\+nh0τ~1,a​\(s0,1\)=1M​ϵ2,nh0τ~1,a​\(s0,a\)≥nh0τ~1,a​\(s0,1\)\)≥150\\displaystyle\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\\left\(\\tilde\{\\tau\}\_\{1,a\}\\leq\\tau<\+\\infty,n^\{\\tilde\{\\tau\}\_\{1,a\}\}\_\{h\_\{0\}\}\(s\_\{0\},a\)\+n^\{\\tilde\{\\tau\}\_\{1,a\}\}\_\{h\_\{0\}\}\(s\_\{0\},1\)=\\frac\{1\}\{M\\epsilon^\{2\}\},n^\{\\tilde\{\\tau\}\_\{1,a\}\}\_\{h\_\{0\}\}\(s\_\{0\},a\)\\geq n^\{\\tilde\{\\tau\}\_\{1,a\}\}\_\{h\_\{0\}\}\(s\_\{0\},1\)\\right\)\\geq\\frac\{1\}\{50\}⇒\\displaystyle\\RightarrowPrν,alg~⁡\(nh0τ​\(s0,a\)≥12​M​ϵ2,τ<\+∞\)≥150\\displaystyle\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\\left\(n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},a\)\\geq\\frac\{1\}\{2M\\epsilon^\{2\}\},\\tau<\+\\infty\\right\)\\geq\\frac\{1\}\{50\}The last line suggests that we have proved the Lemma\. ∎

Given Lemma[33](https://arxiv.org/html/2605.23182#Thmtheorem33), we are ready to prove the lower bounds for expected pulling times\.

###### Lemma 34\.

Apply aδ\\delta\-PAC symmetric algorithmalg~\\tilde\{\\text\{alg\}\}to a Uniform Instanceν\\nuwith\(S,A,H,r,ϵ\)\(S,A,H,r,\\epsilon\)\. If∃s0∈\[S−2\]\\exists s\_\{0\}\\in\[S\-2\]and evenh0h\_\{0\}such thatPrν,alg~⁡\(π^h0​\(s0\)=1\)≥17\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\(\\hat\{\\pi\}\_\{h\_\{0\}\}\(s\_\{0\}\)=1\)\\geq\\frac\{1\}\{7\}, we have

𝔼ν,alg~​nh0τ​\(s0,a\)≥1100​M​ϵ2,∀a∈\{2,⋯,A\}\\displaystyle\\mathbb\{E\}\_\{\\nu,\\tilde\{\\text\{alg\}\}\}n\_\{h\_\{0\}\}^\{\\tau\}\(s\_\{0\},a\)\\geq\\frac\{1\}\{100M\\epsilon^\{2\}\},\\forall a\\in\\\{2,\\cdots,A\\\}whereM=2000M=2000\.

###### Proof of Lemma[34](https://arxiv.org/html/2605.23182#Thmtheorem34)\.

By the Lemma[33](https://arxiv.org/html/2605.23182#Thmtheorem33), we know

Prν,alg~⁡\(nh0τ​\(s0,a\)≥12​M​ϵ2\)≥150\\displaystyle\\Pr\_\{\\nu,\\tilde\{\\text\{alg\}\}\}\\left\(n^\{\\tau\}\_\{h\_\{0\}\}\(s\_\{0\},a\)\\geq\\frac\{1\}\{2M\\epsilon^\{2\}\}\\right\)\\geq\\frac\{1\}\{50\}holds for alla∈\{2,⋯,A\}a\\in\\\{2,\\cdots,A\\\}\. We can prove Lemma[34](https://arxiv.org/html/2605.23182#Thmtheorem34)by applying the Markov Inequality\. ∎

## Appendix DTechnical Inequality

In this section, we present technical inequalities used by the above proof for Lemma and Theorems, including multiple inequalities and concentration theorems\.

###### Lemma 35\.

Consider Random variable\{Xt\}t=1\+∞\\\{X\_\{t\}\\\}\_\{t=1\}^\{\+\\infty\}such thatXt\|\{Xs\}s=1t−1X\_\{t\}\|\\\{X\_\{s\}\\\}\_\{s=1\}^\{t\-1\}isσ2\\sigma^\{2\}\-subgaussian with mean value0\. Then for an integerNN, we have

Pr⁡\(max1≤n≤N​∑t=1nXt\>x\)≤exp⁡\(−x22​N​σ2\)\.\\displaystyle\\Pr\\left\(\\max\_\{1\\leq n\\leq N\}\\sum\_\{t=1\}^\{n\}X\_\{t\}\>x\\right\)\\leq\\exp\(\-\\frac\{x^\{2\}\}\{2N\\sigma^\{2\}\}\)\.

###### Proof of Lemma[35](https://arxiv.org/html/2605.23182#Thmtheorem35)\.

Takeλ=xN​σ2\\lambda=\\frac\{x\}\{N\\sigma^\{2\}\}\. We first validate\{exp⁡\(λ​∑t=1nXt\)\}n=1\+∞\\\{\\exp\\left\(\\lambda\\sum\_\{t=1\}^\{n\}X\_\{t\}\\right\)\\\}\_\{n=1\}^\{\+\\infty\}is a submartingale\. By Jensen’s Inequality,

𝔼​\[exp⁡\(λ​∑t=1nXt\)\|X1,⋯,Xn−1\]≥exp⁡\(λ​∑t=1n−1Xt\)​exp⁡\(𝔼​λ​Xn\)=exp⁡\(λ​∑t=1n−1Xt\)\.\\displaystyle\\mathbb\{E\}\\left\[\\exp\\left\(\\lambda\\sum\_\{t=1\}^\{n\}X\_\{t\}\\right\)\|X\_\{1\},\\cdots,X\_\{n\-1\}\\right\]\\geq\\exp\\left\(\\lambda\\sum\_\{t=1\}^\{n\-1\}X\_\{t\}\\right\)\\exp\(\\mathbb\{E\}\\lambda X\_\{n\}\)=\\exp\\left\(\\lambda\\sum\_\{t=1\}^\{n\-1\}X\_\{t\}\\right\)\.
By direct calculation, we have

Pr⁡\(max1≤n≤N​∑t=1nXt\>x\)\\displaystyle\\Pr\\left\(\\max\_\{1\\leq n\\leq N\}\\sum\_\{t=1\}^\{n\}X\_\{t\}\>x\\right\)=\\displaystyle=Pr⁡\(max1≤n≤N⁡exp⁡\(λ​∑t=1nXt\)\>exp⁡\(λ​x\)\)\\displaystyle\\Pr\\left\(\\max\_\{1\\leq n\\leq N\}\\exp\\left\(\\lambda\\sum\_\{t=1\}^\{n\}X\_\{t\}\\right\)\>\\exp\(\\lambda x\)\\right\)≤\\displaystyle\\leq𝔼​exp⁡\(λ​∑t=1NXt\)exp⁡\(λ​x\)\\displaystyle\\frac\{\\mathbb\{E\}\\exp\\left\(\\lambda\\sum\_\{t=1\}^\{N\}X\_\{t\}\\right\)\}\{\\exp\(\\lambda x\)\}\(68\)≤\\displaystyle\\leqexp⁡\(N​λ2​σ22\)exp⁡\(λ​x\)\\displaystyle\\frac\{\\exp\(\\frac\{N\\lambda^\{2\}\\sigma^\{2\}\}\{2\}\)\}\{\\exp\(\\lambda x\)\}=\\displaystyle=exp⁡\(−x22​N​σ2\)\.\\displaystyle\\exp\(\-\\frac\{x^\{2\}\}\{2N\\sigma^\{2\}\}\)\.\(69\)Step \([68](https://arxiv.org/html/2605.23182#A4.E68)\) is by the maximal inequality for the submartingale\. ∎

###### Lemma 36\.

Assumep,q∈\(14,34\)p,q\\in\(\\frac\{1\}\{4\},\\frac\{3\}\{4\}\), andX∼Bern​\(p\)X\\sim\\text\{Bern\}\(p\), let

Z:=X​log⁡pq\+\(1−X\)​log⁡1−p1−q−\(p​log⁡pq\+\(1−p\)​log⁡1−p1−q\),\\displaystyle Z:=X\\log\\frac\{p\}\{q\}\+\(1\-X\)\\log\\frac\{1\-p\}\{1\-q\}\-\(p\\log\\frac\{p\}\{q\}\+\(1\-p\)\\log\\frac\{1\-p\}\{1\-q\}\),ZZis8​\(p−q\)28\(p\-q\)^\{2\}\-subgaussian, i\.e\. for anyλ∈ℝ\\lambda\\in\\mathbb\{R\}, we have𝔼\[exp\(λZ\)≤exp\(4λ2\(p−q\)2\)\\mathbb\{E\}\[\\exp\(\\lambda Z\)\\leq\\exp\(4\\lambda^\{2\}\(p\-q\)^\{2\}\)

###### Proof of Lemma[36](https://arxiv.org/html/2605.23182#Thmtheorem36)\.

Not hard to see𝔼​Z=0\\mathbb\{E\}Z=0and

Z=\\displaystyle Z=\(X−p\)​\(log⁡pq−log⁡1−p1−q\)\\displaystyle\(X\-p\)\\left\(\\log\\frac\{p\}\{q\}\-\\log\\frac\{1\-p\}\{1\-q\}\\right\)=\\displaystyle=\(X−p\)​log⁡p1−pq1−q\\displaystyle\(X\-p\)\\log\\frac\{\\frac\{p\}\{1\-p\}\}\{\\frac\{q\}\{1\-q\}\}SinceX−pX\-pis a14\\frac\{1\}\{4\}\-subgaussian random variable, we suffice to show14​\(log⁡p1−pq1−q\)2≤8​\(p−q\)2,∀p,q∈\(14,34\)\\frac\{1\}\{4\}\\left\(\\log\\frac\{\\frac\{p\}\{1\-p\}\}\{\\frac\{q\}\{1\-q\}\}\\right\)^\{2\}\\leq 8\(p\-q\)^\{2\},\\forall p,q\\in\(\\frac\{1\}\{4\},\\frac\{3\}\{4\}\)\. Notice that

14​\(log⁡p1−pq1−q\)2≤8​\(p−q\)2\\displaystyle\\frac\{1\}\{4\}\\left\(\\log\\frac\{\\frac\{p\}\{1\-p\}\}\{\\frac\{q\}\{1\-q\}\}\\right\)^\{2\}\\leq 8\(p\-q\)^\{2\}⇔\\displaystyle\\Leftrightarrow\|log⁡p1−p−log⁡q1−qp−q\|≤2​8\\displaystyle\|\\frac\{\\log\\frac\{p\}\{1\-p\}\-\\log\\frac\{q\}\{1\-q\}\}\{p\-q\}\|\\leq 2\\sqrt\{8\}⇐\\displaystyle\\Leftarrowsup14≤x≤34\|d​log⁡x1−xd​x\|≤2​8\\displaystyle\\sup\_\{\\frac\{1\}\{4\}\\leq x\\leq\\frac\{3\}\{4\}\}\|\\frac\{d\\log\\frac\{x\}\{1\-x\}\}\{dx\}\|\\leq 2\\sqrt\{8\}⇔\\displaystyle\\Leftrightarrowsup14≤x≤34\|1x\+11−x\|≤2​8\\displaystyle\\sup\_\{\\frac\{1\}\{4\}\\leq x\\leq\\frac\{3\}\{4\}\}\|\\frac\{1\}\{x\}\+\\frac\{1\}\{1\-x\}\|\\leq 2\\sqrt\{8\}The second last step is by the median value theorem\. Sincesup14≤x≤34\|1x\+11−x\|=\|1x\+11−x\|x=14​or​34=163<2​8\\sup\_\{\\frac\{1\}\{4\}\\leq x\\leq\\frac\{3\}\{4\}\}\|\\frac\{1\}\{x\}\+\\frac\{1\}\{1\-x\}\|=\|\\frac\{1\}\{x\}\+\\frac\{1\}\{1\-x\}\|\_\{x=\\frac\{1\}\{4\}\\text\{ or \}\\frac\{3\}\{4\}\}=\\frac\{16\}\{3\}<2\\sqrt\{8\}, we have completed the proof\. ∎

###### Lemma 37\.

Definekl​\(x,y\)=x​log⁡xy\+\(1−x\)​log⁡1−x1−y\\text\{kl\}\(x,y\)=x\\log\\frac\{x\}\{y\}\+\(1\-x\)\\log\\frac\{1\-x\}\{1\-y\}\. Forx,yx,ysuch that14<x,y<34\\frac\{1\}\{4\}<x,y<\\frac\{3\}\{4\}, we have

kl​\(x,y\)≤3​\(x−y\)2\\displaystyle\\text\{kl\}\(x,y\)\\leq 3\(x\-y\)^\{2\}

###### Proof of Lemma[37](https://arxiv.org/html/2605.23182#Thmtheorem37)\.

Givenx∈\(14,43\)x\\in\(\\frac\{1\}\{4\},\\frac\{4\}\{3\}\), definef​\(y\)=x​log⁡xy\+\(1−x\)​log⁡1−x1−y−3​\(x−y\)2f\(y\)=x\\log\\frac\{x\}\{y\}\+\(1\-x\)\\log\\frac\{1\-x\}\{1\-y\}\-3\(x\-y\)^\{2\}\. We have

f′​\(y\)=\\displaystyle f^\{\\prime\}\(y\)=−xy\+1−x1−y−6​\(y−x\)\\displaystyle\-\\frac\{x\}\{y\}\+\\frac\{1\-x\}\{1\-y\}\-6\(y\-x\)=\\displaystyle=−x\+yy​\(1−y\)−6​\(y−x\)\\displaystyle\\frac\{\-x\+y\}\{y\(1\-y\)\}\-6\(y\-x\)=\\displaystyle=\(y−x\)​\(1y​\(1−y\)−6\)\\displaystyle\(y\-x\)\\left\(\\frac\{1\}\{y\(1\-y\)\}\-6\\right\)Notice that316≤y​\(1−y\)≤14\\frac\{3\}\{16\}\\leq y\(1\-y\)\\leq\\frac\{1\}\{4\}holds fory∈\(14,34\)y\\in\(\\frac\{1\}\{4\},\\frac\{3\}\{4\}\), which means1y​\(1−y\)−6≤0\\frac\{1\}\{y\(1\-y\)\}\-6\\leq 0holds fory∈\(14,34\)y\\in\(\\frac\{1\}\{4\},\\frac\{3\}\{4\}\)\. We can concludef′​\(y\)f^\{\\prime\}\(y\)is decreasing in the interval\(x,34\)\(x,\\frac\{3\}\{4\}\)and increasing in the interval\(14,x\)\(\\frac\{1\}\{4\},x\)\. That meansmax14<y<34⁡f​\(y\)=f​\(x\)=0\\max\_\{\\frac\{1\}\{4\}<y<\\frac\{3\}\{4\}\}f\(y\)=f\(x\)=0, which proves the conclusion\. ∎

###### Lemma 38\.

Given anyb≥a≥2b\\geq a\\geq 2\. Ifx\>0x\>0satisfiesx≤b\+a​log⁡xx\\leq b\+a\\log x, we can concludex<b\+3​a​log⁡bx<b\+3a\\log b\.

###### Proof\.

Prove by contradiction\. Assumex≥b\+3​a​log⁡bx\\geq b\+3a\\log b\. Notice thatd​\(x−a​log⁡x−b\)d​x=1−ax\\frac\{d\(x\-a\\log x\-b\)\}\{dx\}=1\-\\frac\{a\}\{x\}, which impliesx−a​log⁡x−bx\-a\\log x\-bincreases at the interval\(b\+3​a​log⁡b,\+∞\)\(b\+3a\\log b,\+\\infty\)\. In addition, we have

x−a​log⁡x−b\|x=b\+3​a​log⁡b\\displaystyle x\-a\\log x\-b\|\_\{x=b\+3a\\log b\}=\\displaystyle=b\+3​a​log⁡b−a​log⁡\(b\+3​a​log⁡b\)−b\\displaystyle b\+3a\\log b\-a\\log\(b\+3a\\log b\)\-b=\\displaystyle=3​a​log⁡b−a​log⁡\(b\+3​a​log⁡b\)\\displaystyle 3a\\log b\-a\\log\(b\+3a\\log b\)=\\displaystyle=a​log⁡b3b\+3​a​log⁡b\\displaystyle a\\log\\frac\{b^\{3\}\}\{b\+3a\\log b\}≥\\displaystyle\\geqa​log⁡b3b\+3​b​log⁡b\\displaystyle a\\log\\frac\{b^\{3\}\}\{b\+3b\\log b\}=\\displaystyle=a​log⁡b21\+3​log⁡b\.\\displaystyle a\\log\\frac\{b^\{2\}\}\{1\+3\\log b\}\.Asb2≥1\+3​log⁡bb^\{2\}\\geq 1\+3\\log bfor anyb≥2b\\geq 2, \(22≥1\+3​log⁡22^\{2\}\\geq 1\+3\\log 2\), we can concludex−a​log⁡x−b≥0x\-a\\log x\-b\\geq 0holds for allx≥b\+3​a​log⁡bx\\geq b\+3a\\log b, which implies a contradiction\. ∎

###### Lemma 39\.

log⁡\(a\+b\)≤log⁡\(4​a​b\)=log⁡\(2​a\)\+log⁡\(2​b\)\\log\(a\+b\)\\leq\\log\(4ab\)=\\log\(2a\)\+\\log\(2b\)fora,b≥1a,b\\geq 1

###### Proof\.

Notice that fora,b≥1a,b\\geq 1, we have4​a​b−a−b=a​\(2​b−1\)\+b​\(2​a−1\)\>04ab\-a\-b=a\(2b\-1\)\+b\(2a\-1\)\>0, we can concludelog⁡\(a\+b\)≤log⁡\(4​a​b\)\\log\(a\+b\)\\leq\\log\(4ab\)\. ∎

###### Lemma 40\.

Givena≥1a\\geq 1, we havex≥2​a​log⁡\(3​a\)⇒x≥a​log⁡xx\\geq 2a\\log\(3a\)\\Rightarrow x\\geq a\\log x

###### Proof\.

Definef​\(x\)=x−a​log⁡xf\(x\)=x\-a\\log x\. Byd​fd​x=1−ax\\frac\{df\}\{dx\}=1\-\\frac\{a\}\{x\}, we knowffis increasing in the interval\(a,\+∞\)\(a,\+\\infty\)\. By the fact thata≥1a\\geq 1, we havelog⁡\(3​a\)≥1\\log\(3a\)\\geq 1, further2​a​log⁡\(3​a\)≥a2a\\log\(3a\)\\geq a\. Meanwhile, not hard to see

a≥1\\displaystyle a\\geq 1⇒\\displaystyle\\Rightarrowlog⁡\(3​a\)≥log⁡\(2​a\),log⁡\(3​a\)≥log⁡\(log⁡\(3​a\)\)\\displaystyle\\log\(3a\)\\geq\\log\(2a\),\\log\(3a\)\\geq\\log\(\\log\(3a\)\)⇒\\displaystyle\\Rightarrow2​log⁡\(3​a\)≥log⁡\(2​a\)\+log⁡\(log⁡\(3​a\)\)\\displaystyle 2\\log\(3a\)\\geq\\log\(2a\)\+\\log\(\\log\(3a\)\)⇔\\displaystyle\\Leftrightarrow2​log⁡\(3​a\)≥log⁡\(2​a​log⁡\(3​a\)\)\\displaystyle 2\\log\(3a\)\\geq\\log\(2a\\log\(3a\)\)⇒\\displaystyle\\Rightarrow2​a​log⁡\(3​a\)−a​log⁡\(2​a​log⁡\(3​a\)\)≥0\.\\displaystyle 2a\\log\(3a\)\-a\\log\(2a\\log\(3a\)\)\\geq 0\.We can concludef​\(2​a​log⁡\(3​a\)\)≥0f\(2a\\log\(3a\)\)\\geq 0, which meansx≥2​a​log⁡\(3​a\)⇒x≥a​log⁡xx\\geq 2a\\log\(3a\)\\Rightarrow x\\geq a\\log x\. ∎

###### Lemma 41\(Lemma 19 inJakschet al\.\[[2010](https://arxiv.org/html/2605.23182#bib.bib12)\]\)\.

For0≤zk≤10\\leq z\_\{k\}\\leq 1,M≥0M\\geq 0, we have∑k=1nzkmax⁡\{M\+∑i=1k−1zi,1\}≤\(2\+1\)​∑i=1nzi\\sum\_\{k=1\}^\{n\}\\frac\{z\_\{k\}\}\{\\sqrt\{\\max\\\{M\+\\sum\_\{i=1\}^\{k\-1\}z\_\{i\},1\\\}\}\}\\leq\(\\sqrt\{2\}\+1\)\\sqrt\{\\sum\_\{i=1\}^\{n\}z\_\{i\}\}

###### Proof\.

DenoteZk−1:=max⁡\{1,∑i=1k−1zi\}Z\_\{k\-1\}:=\\max\\big\\\{1,\\sum\_\{i=1\}^\{k\-1\}z\_\{i\}\\big\\\}\. Trivial to seeZk−1≥zkZ\_\{k\-1\}\\geq z\_\{k\}\. We use induction overnn\. First, we show the lemma holds forn=1n=1\. The reason is

∑k=11zkmax⁡\{M\+∑i=1n−1zi,1\}=z1max⁡\{M,1\}≤z1≤z1<\(2\+1\)​z1\.\\displaystyle\\sum\_\{k=1\}^\{1\}\\frac\{z\_\{k\}\}\{\\sqrt\{\\max\\\{M\+\\sum\_\{i=1\}^\{n\-1\}z\_\{i\},1\\\}\}\}=\\frac\{z\_\{1\}\}\{\\max\\\{M,1\\\}\}\\leq z\_\{1\}\\leq\\sqrt\{z\_\{1\}\}<\(\\sqrt\{2\}\+1\)\\sqrt\{z\_\{1\}\}\.
Then, we turn to work on two cases for a generalnn, givenn−1n\-1is correct, we consider two cases∑k=1n−1zk≤1\\sum\_\{k=1\}^\{n\-1\}z\_\{k\}\\leq 1or∑k=1n−1zk\>1\\sum\_\{k=1\}^\{n\-1\}z\_\{k\}\>1\. If∑k=1n−1zk≤1\\sum\_\{k=1\}^\{n\-1\}z\_\{k\}\\leq 1, we have∑k=1nzk=zn\+∑k=1n−1zk≤2\\sum\_\{k=1\}^\{n\}z\_\{k\}=z\_\{n\}\+\\sum\_\{k=1\}^\{n\-1\}z\_\{k\}\\leq 2, which implies∑k=1nzk≤2​∑k=1nzk\\sum\_\{k=1\}^\{n\}z\_\{k\}\\leq\\sqrt\{2\}\\sqrt\{\\sum\_\{k=1\}^\{n\}z\_\{k\}\}\. It follows that

∑k=1nzkmax⁡\{M\+∑i=1k−1zi,1\}\\displaystyle\\sum\_\{k=1\}^\{n\}\\frac\{z\_\{k\}\}\{\\sqrt\{\\max\\\{M\+\\sum\_\{i=1\}^\{k\-1\}z\_\{i\},1\\\}\}\}≤\\displaystyle\\leq∑k=1n−1zk\+zn\\displaystyle\\sum\_\{k=1\}^\{n\-1\}z\_\{k\}\+z\_\{n\}≤\\displaystyle\\leq2​∑k=1nzk\\displaystyle\\sqrt\{2\}\\sqrt\{\\sum\_\{k=1\}^\{n\}z\_\{k\}\}<\\displaystyle<\(2\+1\)​∑k=1nzk\.\\displaystyle\(\\sqrt\{2\}\+1\)\\sqrt\{\\sum\_\{k=1\}^\{n\}z\_\{k\}\}\.Then, we show that the lemma also holds fornn, when∑k=1n−1zk\>1\\sum\_\{k=1\}^\{n\-1\}z\_\{k\}\>1\. By the induction hypothesis, we have

∑k=1nzkmax⁡\{M\+∑i=1k−1zi,1\}\\displaystyle\\sum\_\{k=1\}^\{n\}\\frac\{z\_\{k\}\}\{\\max\\\{M\+\\sum\_\{i=1\}^\{k\-1\}z\_\{i\},1\\\}\}≤\\displaystyle\\leq\(2\+1\)​∑k=1n−1zk\+znmax⁡\{∑i=1n−1zi,1\}\\displaystyle\(\\sqrt\{2\}\+1\)\\sqrt\{\\sum\_\{k=1\}^\{n\-1\}z\_\{k\}\}\+\\frac\{z\_\{n\}\}\{\\sqrt\{\\max\\\{\\sum\_\{i=1\}^\{n\-1\}z\_\{i\},1\\\}\}\}=\\displaystyle=\(2\+1\)​∑k=1n−1zk\+zn∑i=1n−1zi\\displaystyle\(\\sqrt\{2\}\+1\)\\sqrt\{\\sum\_\{k=1\}^\{n\-1\}z\_\{k\}\}\+\\frac\{z\_\{n\}\}\{\\sqrt\{\\sum\_\{i=1\}^\{n\-1\}z\_\{i\}\}\}=\\displaystyle=\(\(2\+1\)​∑k=1n−1zk\+zn∑i=1n−1zi\)2\\displaystyle\\sqrt\{\\left\(\(\\sqrt\{2\}\+1\)\\sqrt\{\\sum\_\{k=1\}^\{n\-1\}z\_\{k\}\}\+\\frac\{z\_\{n\}\}\{\\sqrt\{\\sum\_\{i=1\}^\{n\-1\}z\_\{i\}\}\}\\right\)^\{2\}\}=\\displaystyle=\(2\+1\)2​\(∑k=1n−1zk\)\+2​\(2\+1\)​zn\+zn2∑k=1n−1zk\\displaystyle\\sqrt\{\(\\sqrt\{2\}\+1\)^\{2\}\(\\sum\_\{k=1\}^\{n\-1\}z\_\{k\}\)\+2\(\\sqrt\{2\}\+1\)z\_\{n\}\+\\frac\{z\_\{n\}^\{2\}\}\{\\sum\_\{k=1\}^\{n\-1\}z\_\{k\}\}\}≤zn≤1,∑k=1n−1zk≥1\\displaystyle\\stackrel\{\{\\scriptstyle z\_\{n\}\\leq 1,\\sum\_\{k=1\}^\{n\-1\}z\_\{k\}\\geq 1\}\}\{\{\\leq\}\}\(2\+1\)2​\(∑k=1n−1zk\)\+2​\(2\+1\)​zn\+zn\\displaystyle\\sqrt\{\(\\sqrt\{2\}\+1\)^\{2\}\(\\sum\_\{k=1\}^\{n\-1\}z\_\{k\}\)\+2\(\\sqrt\{2\}\+1\)z\_\{n\}\+z\_\{n\}\}=\\displaystyle=\(2​2\+3\)​\(∑k=1n−1zk\)\+\(2​2\+3\)​zn\\displaystyle\\sqrt\{\(2\\sqrt\{2\}\+3\)\(\\sum\_\{k=1\}^\{n\-1\}z\_\{k\}\)\+\(2\\sqrt\{2\}\+3\)z\_\{n\}\}=\\displaystyle=\(2\+1\)​∑k=1nzk,\\displaystyle\(\\sqrt\{2\}\+1\)\\sqrt\{\\sum\_\{k=1\}^\{n\}z\_\{k\}\},which means the induction holds\. ∎

The remaining two lemmas are fromLi and Cheung \[[2025](https://arxiv.org/html/2605.23182#bib.bib18)\]\. We don’t repeat the proof as they are exactly the same\.

###### Lemma 42\(Lemma D\.2 inLi and Cheung \[[2025](https://arxiv.org/html/2605.23182#bib.bib18)\]\)\.

For anyΔ∈\(0,1\],K≥2,δ∈\(0,12\],C≥1\\Delta\\in\(0,1\],K\\geq 2,\\delta\\in\(0,\\frac\{1\}\{2\}\],C\\geq 1, we can conclude

t\>28​C2​log⁡2​KδΔ2\+16​C2​log⁡\(log⁡\(24​C2Δ2\)\)Δ2\\displaystyle t\>\\frac\{28C^\{2\}\\log\\frac\{2K\}\{\\delta\}\}\{\\Delta^\{2\}\}\+\\frac\{16C^\{2\}\\log\\left\(\\log\\left\(\\frac\{24C^\{2\}\}\{\\Delta^\{2\}\}\\right\)\\right\)\}\{\\Delta^\{2\}\}⇒\\displaystyle\\RightarrowC​4​log⁡2​K​\(log2⁡2​t\)2δt<Δ\\displaystyle C\\sqrt\{\\frac\{4\\log\\frac\{2K\(\\log\_\{2\}2t\)^\{2\}\}\{\\delta\}\}\{t\}\}<\\Delta

###### Lemma 43\(Lemma D\.3 inLi and Cheung \[[2025](https://arxiv.org/html/2605.23182#bib.bib18)\]\)\.

Assume\{Xi\}i=1\+∞\\\{X\_\{i\}\\\}\_\{i=1\}^\{\+\\infty\}are i\.i\.d random variables with mean valueμ\\mu\. Assume\{Xi−μ\}i=1\+∞\\\{X\_\{i\}\-\\mu\\\}\_\{i=1\}^\{\+\\infty\}areσ2\\sigma^\{2\}\-subgaussian\. We have

Pr⁡\(∀N∈ℕ,\|∑i=1NXiN−μ\|<4​σ2​log⁡2​\(log2⁡2​N\)2δN\)≥1−π26​δ\\displaystyle\\Pr\\left\(\\forall N\\in\\mathbb\{N\},\\left\|\\frac\{\\sum\_\{i=1\}^\{N\}X\_\{i\}\}\{N\}\-\\mu\\right\|<\\sqrt\{\\frac\{4\\sigma^\{2\}\\log\\frac\{2\(\\log\_\{2\}2N\)^\{2\}\}\{\\delta\}\}\{N\}\}\\right\)\\geq 1\-\\frac\{\\pi^\{2\}\}\{6\}\\deltaholds for allδ\>0\\delta\>0\.