Optimizing Digital Therapeutic Interventions: Online Learning under Endogenous Adherence

# 1 Introduction
Source: [https://arxiv.org/html/2605.24261](https://arxiv.org/html/2605.24261)
\\OneAndAHalfSpacedXII\\EquationsNumberedThrough\\TheoremsNumberedThrough\\ECRepeatTheorems\\MANUSCRIPTNO

MSOM\-0001\-2024\.00

\\RUNAUTHOR

\\RUNTITLE

Optimizing Digital Therapeutic Interventions

\\TITLE

Optimizing Digital Therapeutic Interventions: Online Learning under Endogenous Adherence

\\ARTICLEAUTHORS\\AUTHOR

Eric Pulick\\AFFDepartment of Industrial and Systems Engineering, University of Wisconsin\-Madison,\\EMAILpulick@wisc\.edu

\\AUTHOR

Stephanie Carpenter\\AFFCollege of Health Solutions, Arizona State University,\\EMAILstephanie\.m\.carpenter@asu\.edu

\\AUTHOR

Matthew Buman\\AFFCollege of Health Solutions, Arizona State University,\\EMAILmatthew\.buman@asu\.edu

\\AUTHOR

Yonatan Mintz\\AFFDepartment of Industrial and Systems Engineering, University of Wisconsin\-Madison,\\EMAILymintz@wisc\.edu

\\ABSTRACT

Problem definition:A critical challenge facing clinicians managing chronic disease interventions is sustaining long\-run patient health given limited information and resources\. Digital therapeutics \(DTs\) provide a cost\-effective way to manage interventions at scale through repeated interactions \(e\.g\. daily treatment recommendations\), but patient success is highly dependent on their adherence\. Behavioral psychology suggests that both treatment recommendations and past adherence affect future adherence, yet existing decision support frameworks for DTs model only recommendation effects or treat adherence as exogenous context, leaving a key gap in model and algorithm development\.Methodology/results:To address this gap, we present a DT decision support framework that captures both recommendation and adherence effects, allowing clinicians to better plan treatment recommendations\. We model a patient’s time\-varying capacity for engagement with treatment using a linear dynamical system \(LDS\) that captures both recommendation and adherence effects, endogenously connected to adherence behavior with a logit link\. We establish finite\-time identification guarantees for this model, extending LDS results to our setting\. Next, we propose an optimism\-based algorithm,UCB\-BOLD, for online treatment selection and prove that it achieves sublinear regret\. We evaluateUCB\-BOLDagainst benchmarks via ablation studies on a synthetic patient cohort generated using micro\-randomized trial data collected to develop a workplace sedentary behavior intervention\.Managerial implications:DT decision support tools can include dynamical models to enable decision makers to efficiently use the data in DT settings to improve patient health through effective resource allocation\. While myopic or heuristic approaches suffice for some patient types, the benefits of explicitly planning around recommendation and adherence effects are significant for others;UCB\-BOLDachieves 2\-3x lower conditional value\-at\-risk regret than the next\-best benchmark\.

Chronic disorders, including metabolic disease and addiction, affect millions of Americans \(34\.7% and 16\.8% of adults, respectively\)\(Palaniappanet al\.[2026](https://arxiv.org/html/2605.24261#bib.bib52), SAMHSA[2025](https://arxiv.org/html/2605.24261#bib.bib59)\)\. A cost\-effective approach to manage treatment of these disorders is through the use of digital therapeutics \(DTs\)\. DTs provide individualized behavioral interventions to address a wide range of conditions, such as lack of physical activity \(PA\)\(Springet al\.[2024](https://arxiv.org/html/2605.24261#bib.bib66)\), mental illness\(Ben\-Zeevet al\.[2018](https://arxiv.org/html/2605.24261#bib.bib12)\), substance use disorders \(SUDs\)\(Gustafsonet al\.[2014](https://arxiv.org/html/2605.24261#bib.bib29)\), and diabetes\(Sepahet al\.[2017](https://arxiv.org/html/2605.24261#bib.bib60)\)\. DTs interact repeatedly with patients but depend on patient adherence to be effective\. This creates a sequential decision\-making \(SDM\) problem for the DT provider; at each decision point, the provider gathers relevant patient data and chooses which \(if any\) treatment to recommend, while the patient chooses whether to adhere to the given recommendation\. Both the DT’s recommendation and the patient’s adherence decision can affect the patient’s future adherence behavior\. For instance, a recommendation may impose cognitive burden regardless of adherence, and adherence itself can be either fatiguing or habit\-forming\. Further, these effects may vary by individual or treatment, and must be learned online\. Existing DT frameworks model only recommendation effects or treat adherence as exogenous context\. Thus there is a critical need for frameworks that model both adherence and recommendation effects to help DTs better support long\-term patient health\.

In this paper, we present an SDM framework for DT decision support\. We model a patient’s capacity for engaging with treatment as a linear dynamical system \(LDS\) whose dynamics depend on previous capacity, treatment recommendations, and patient adherence, with capacity connected endogenously to adherence using a logit link\. This framework can capture both beneficial \(e\.g\., habit\-forming\) and detrimental \(e\.g\., burden or fatigue\) effects\. The model is deliberately parsimonious, designed to capture the relevant recommendation and adherence phenomena while remaining interpretable for clinicians and tractable for optimization\. We present a statistical analysis of this model and demonstrate finite\-time identification guarantees\. We also propose an online optimization algorithm for DT treatment recommendations,UCB\-BOLD, and prove that it achieves sublinear regret in this setting\. We conclude with a case study on a synthetic patient cohort, calibrated using data from a sedentary behavior micro\-randomized trial \(MRT\), where we show thatUCB\-BOLDoutperforms benchmark algorithms\. This framework is aimed particularly at long\-running behavioral interventions, where sustaining patient engagement over time is especially relevant\.

### 1\.1Problem Context: Digital Therapeutics and Adaptive Interventions

Sustained patient engagement is an important determinant of long\-term patient outcomes in conditions targeted by DT interventions\. Within SUD, for instance, patient engagement with months\-to\-years of continuing care \(i\.e\., lower intensity, but sustained monitoring and support\) is an important predictor of durable patient recovery\(McLellanet al\.[2013](https://arxiv.org/html/2605.24261#bib.bib46), Proctor and Herschman[2014](https://arxiv.org/html/2605.24261#bib.bib55)\)\. Similarly, long\-term patient engagement is needed to achieve the health benefits associated with increased PA or weight management\(Jakicicet al\.[2019](https://arxiv.org/html/2605.24261#bib.bib36)\), reflected in the length of recent studies like the1212\-month HeartSteps II trial\(Spruijt\-Metzet al\.[2022](https://arxiv.org/html/2605.24261#bib.bib67)\)\. Across these settings, however, patient engagement has been observed to decrease over time\(Baumelet al\.[2019](https://arxiv.org/html/2605.24261#bib.bib11)\), with diminished intervention effects following repeated prompt delivery\(Klasnjaet al\.[2019](https://arxiv.org/html/2605.24261#bib.bib38)\)\. This motivates DT approaches that better manage patient engagement to improve long\-run patient outcomes\.

The just\-in\-time\-adaptive\-intervention \(JITAI\) and mHealth literature has increasingly characterized engagement as a time\-varying rather than static construct\(Perskiet al\.[2017](https://arxiv.org/html/2605.24261#bib.bib53), Nahum\-Shaniet al\.[2022](https://arxiv.org/html/2605.24261#bib.bib48), Nahum\-Shani and Yoon[2024](https://arxiv.org/html/2605.24261#bib.bib51)\)\. Theoretical frameworks, such as the Cumulative Complexity model\(Shippeeet al\.[2012](https://arxiv.org/html/2605.24261#bib.bib63)\)and its extension to the digital health setting\(Cross and Alvarez\-Jimenez[2024](https://arxiv.org/html/2605.24261#bib.bib19)\), propose viewing patient adherence and engagement behavior as functions of an underlying, time\-varying patient capacity\. This capacity is affected by mechanisms like burden, fatigue, and habit formation that the JITAI literature has highlighted as important considerations for future intervention design\(Nahum\-Shani and Murphy[2026](https://arxiv.org/html/2605.24261#bib.bib50)\)\. These conceptual frameworks align with a broader shift in the psychology literature toward dynamical system models as tools to understand and approximate individual behavior\(Brigantiet al\.[2024](https://arxiv.org/html/2605.24261#bib.bib17), Perskiet al\.[2025](https://arxiv.org/html/2605.24261#bib.bib54)\), further motivating the integration of explicit generative models into DT decision support tools\.

To this point, though, few personalized JITAI \(pJITAI\) algorithms have incorporated formal dynamics models in their construction\(Perskiet al\.[2025](https://arxiv.org/html/2605.24261#bib.bib54)\)as a way to capture both recommendation and adherence effects\. Prior work has made significant methodological and deployment contributions \(see Section[2](https://arxiv.org/html/2605.24261#S2)\), but no prior approach combines a parametric dynamical model for mHealth patient engagement with system identification and algorithmic regret guarantees as we do here\.

### 1\.2Contribution

We develop a decision support framework for DT providers with four specific contributions:

1. 1\.We propose an LDS with stochastic, state\-dependent adherence to model a patient’s time\-varying engagement capacity\. This framework enables digital therapeutics to plan based on both recommendation and adherence effects\. It also frames time\-varying patient behavior in a clinically interpretable way, aligned with trends toward dynamic patient models in the psychology literature\.
2. 2\.We establish finite\-time system identification guarantees under a structured policy class that permits arbitrary policy behavior \(possibly adaptive or adversarial\) outside of exploration steps\. We show that dynamics and adherence parameters are identifiable at the typical𝒪~​\(1/T\)\\tilde\{\\mathcal\{O\}\}\(\\nicefrac\{\{1\}\}\{\{\\sqrt\{T\}\}\}\)rate, extending LDS guarantees to a setting with11\-sparse inputs and state\-dependent adherence\.
3. 3\.We develop an optimism\-based algorithm,UCB\-BOLD, to optimize DT treatment recommendations\. We demonstrate thatUCB\-BOLDachieves𝒪~​\(T\)\\tilde\{\\mathcal\{O\}\}\(\\sqrt\{T\}\)regret in this context, making it competitive with other reinforcement learning \(RL\) algorithms\.
4. 4\.We fit a Bayesian hierarchical model with data from a sedentary behavior MRT\. We use this model to generate synthetic patients and evaluateUCB\-BOLDagainst heuristic, myopic, and model\-free RL benchmark algorithms on simulated trajectories in ablation studies\. We find thatUCB\-BOLDachieves 2\-3x lower conditional value\-at\-risk regret than the next best benchmark\.

## 2Literature review

Our work draws from four main streams of literature: SDM in healthcare operations, JITAIs, mHealth\-motivated bandits, and model\-based control\.

### 2\.1SDM in healthcare operations

There is a rich body of work exploring patient\-specific SDM in healthcare operations\. For example, Markov decision process \(MDP\) and partially\-observed Markov decision process \(POMDP\) models have been used to analyze organ transplantation\(Alagozet al\.[2004](https://arxiv.org/html/2605.24261#bib.bib3)\), treatment initiation\(Shechteret al\.[2008](https://arxiv.org/html/2605.24261#bib.bib61)\), chronic disease management\(Masonet al\.[2012](https://arxiv.org/html/2605.24261#bib.bib45), Helmet al\.[2015](https://arxiv.org/html/2605.24261#bib.bib31)\), and disease screening\(Ayeret al\.[2012](https://arxiv.org/html/2605.24261#bib.bib7)\)\. Recent work has emphasized perspectives from machine learning\(Bertsimas and Kallus[2020](https://arxiv.org/html/2605.24261#bib.bib14)\), focusing on data\-driven guarantees for personalized healthcare settings\(Bastani and Bayati[2020](https://arxiv.org/html/2605.24261#bib.bib10)\)\. Our work contributes to this stream of literature, specifically focused on the setting of online learning of individual behavior models\(Aswaniet al\.[2019](https://arxiv.org/html/2605.24261#bib.bib5)\)\. In contrast to these works, we focus on explicit modeling of patient engagement and its impact on policy decisions\.

### 2\.2Just\-in\-time\-adaptive\-interventions

JITAIs are principled frameworks for delivering personalized behavioral interventions\(Nahum\-Shaniet al\.[2018](https://arxiv.org/html/2605.24261#bib.bib49)\), often optimized with data from MRTs\(Klasnjaet al\.[2015](https://arxiv.org/html/2605.24261#bib.bib39)\)\. JITAIs have been deployed effectively at scale, for instance in the HeartSteps program for physical activity\(Klasnjaet al\.[2019](https://arxiv.org/html/2605.24261#bib.bib38), Spruijt\-Metzet al\.[2022](https://arxiv.org/html/2605.24261#bib.bib67)\)and the Oralytics program for oral care\(Trellaet al\.[2025](https://arxiv.org/html/2605.24261#bib.bib69)\)\. Our work complements this literature, particularly for long\-running interventions such as SUD continuing care\(Proctor and Herschman[2014](https://arxiv.org/html/2605.24261#bib.bib55)\)\.

### 2\.3mHealth contextual and non\-stationary bandits

Recently deployed pJITAIs use Thompson sampling contextual bandits that handle temporal dynamics through hand\-crafted features, such as recent treatment dosage\(Liaoet al\.[2020](https://arxiv.org/html/2605.24261#bib.bib42)\)\. Extensions include partial pooling of information across patients to better leverage scarce samples\(Tomkinset al\.[2021](https://arxiv.org/html/2605.24261#bib.bib68), Abbottet al\.[2024](https://arxiv.org/html/2605.24261#bib.bib2)\)\. Importantly, these methods do not have explicit dynamics models, instead treating context as exogenous and tuning rewards to account for time\-varying processes\. Non\-stationary and restless bandit approaches are an alternative means of capturing time\-varying behavior in the bandit setting\(Whittle[1988](https://arxiv.org/html/2605.24261#bib.bib72)\), with extensions exploring various structural assumptions\(Garivier and Moulines[2008](https://arxiv.org/html/2605.24261#bib.bib27), Besbeset al\.[2014](https://arxiv.org/html/2605.24261#bib.bib15)\)\. mHealth\-motivated non\-stationary bandit work has emerged from both model\-free\(Baeket al\.[2025](https://arxiv.org/html/2605.24261#bib.bib9)\)and model\-based\(Mintzet al\.[2020](https://arxiv.org/html/2605.24261#bib.bib47)\)perspectives\.Mintzet al\.\([2020](https://arxiv.org/html/2605.24261#bib.bib47)\)and extensions\(He and Mintz[2024](https://arxiv.org/html/2605.24261#bib.bib34), Liet al\.[2025](https://arxiv.org/html/2605.24261#bib.bib44)\)are closest to our work, modeling the case where each arm has action\-dependent habituation and recovery dynamics\. Our approach differs in modeling how both controller actionsandendogenous patient adherence decisions affect future engagement\. We also model a common, time\-varying engagement state rather than per\-arm states\. Methodologically, our algorithm relies on optimism\(Lai and Robbins[1985](https://arxiv.org/html/2605.24261#bib.bib40), Aueret al\.[2008](https://arxiv.org/html/2605.24261#bib.bib6)\), adapting upper confidence bound \(UCB\) techniques deployed across linear\(Abbasi\-Yadkoriet al\.[2011](https://arxiv.org/html/2605.24261#bib.bib1)\)and logistic\(Fauryet al\.[2020](https://arxiv.org/html/2605.24261#bib.bib25)\)bandit settings\. Our proposed algorithm adapts the structure of LSVI\-UCB inJinet al\.\([2023](https://arxiv.org/html/2605.24261#bib.bib37)\), constructed for episodic linear MDPs, to the discounted, infinite horizon setting with hybrid linear\-logistic system dynamics\.

### 2\.4Model\-based control approaches

A related research area applies control\-system tools such as system identification and model predictive control \(MPC\) to behavioral interventions\(Riveraet al\.[2007](https://arxiv.org/html/2605.24261#bib.bib58), Hekleret al\.[2016](https://arxiv.org/html/2605.24261#bib.bib30), El Mistiriet al\.[2025](https://arxiv.org/html/2605.24261#bib.bib23)\)\. Closely related to our modeling focus on mHealth engagement areEl Mistiriet al\.\([2024](https://arxiv.org/html/2605.24261#bib.bib24)\)andDe La Torreet al\.\([2025](https://arxiv.org/html/2605.24261#bib.bib21)\), which model time\-varying engagement states with dynamics motivated by continuous time fluid processes\. This line of research shares our model\-based approach, but typically evaluates performance empirically in simulation rather than with formal regret analysis\. Closer to our work is a line of model\-based RL mHealth literature targeting provable guarantees, such asHe and Mintz \([2023](https://arxiv.org/html/2605.24261#bib.bib32)\)for heparin dosing,Liet al\.\([2026](https://arxiv.org/html/2605.24261#bib.bib41)\)for weight\-loss, andPulick and Mintz \([2025](https://arxiv.org/html/2605.24261#bib.bib56)\)for substance use disorders\. Similar to our work, these fit patient\-specific dynamics models, but their dynamics are deterministic with noise affecting an observation model, while in our formulation process noise affects the state dynamics directly\.

Our system identification result follows recent work establishing finite\-time identification and regret guarantees for specific control model classes\(Tsiamiset al\.[2023](https://arxiv.org/html/2605.24261#bib.bib70)\)\. We build on the block\-martingale small ball machinery for finite\-time LDS analysis fromSimchowitzet al\.\([2018](https://arxiv.org/html/2605.24261#bib.bib64)\), which has been extended to settings like the linear quadratic regulator\(Deanet al\.[2020](https://arxiv.org/html/2605.24261#bib.bib20)\)and LDS with non\-linear policies\(Liet al\.[2023](https://arxiv.org/html/2605.24261#bib.bib43)\)\. We extend this core machinery to our LDS setting with11\-sparse inputs and state\-dependent adherence under a flexible exploratory policy class\.

## 3Engagement state dynamics and adherence model

Our motivating setting is a DT where the decision\-maker selects a daily treatment recommendation to support long\-term patient health\. Treatment recommendations themselves can impact a patient’s future adherence \(e\.g\., recommendation burden\)\. Similarly, adherence itself can affect future patient adherence negatively \(e\.g\., fatigue\) or positively \(e\.g\., increased self\-efficacy\)\. Our model captures these effects with a scalar, time\-varying construct: the patient’s engagement state, representing their capacity for adherence\. We model patient adherence as stochastic decisions depending endogenously on the engagement state, with higher values yielding higher adherence probability\. The model can express empirically observed behavior, such as burden or fatigue reducing patient adherence, via the patient’s time\-varying engagement state\. This structure re\-frames the decision\-maker’s problem from myopic decision\-making \(i\.e\., selecting actions with the largest immediate reward\) to long\-term planning around relevant engagement state dynamics \(i\.e\., accounting for recommendation\- and adherence\-driven effects on a patient’s future engagement state\)\. A planning algorithm can strategically ask more or less of a patient over time to maintain engagement and accrue greater long\-run rewards than myopic approaches\. To reflect patient heterogeneity, each patient has an unknown set of dynamics and adherence parameters; the DT decision\-maker does not know these parameters*a priori*and must learn them online through repeated interaction with the patient\.

### 3\.1Model definition

We model the patient’s time\-varying engagement state and adherence behavior as follows:

xt\+1\\displaystyle x\_\{t\+1\}=a​xt\+b⊤​ut\+c⊤​dt\+wt,\\displaystyle=ax\_\{t\}\+b^\{\\top\}u\_\{t\}\+c^\{\\top\}d\_\{t\}\+w\_\{t\},dti\|xt,uti∼Ber​\(p^i​\(xt,uti\)\),p^i​\(x,ui\)=ui​σ​\(x\+μi\)\\displaystyle d^\{i\}\_\{t\}\|x\_\{t\},u^\{i\}\_\{t\}\\sim\\text\{Ber\}\\bigl\(\\hat\{p\}\_\{i\}\(x\_\{t\},u^\{i\}\_\{t\}\)\\bigr\),\\quad\\hat\{p\}\_\{i\}\(x,u^\{i\}\)=u^\{i\}\\sigma\(x\+\\mu\_\{i\}\)\(1\)Let there beMMavailable treatments and at most one treatment can be recommended per day, indexed byt=1,…,Tt=1,\\dots,T\. The patient’s engagement state is given byxt∈ℝx\_\{t\}\\in\\mathbb\{R\}\.ut∈𝒰:=\{v∈\{0,1\}M:‖v‖0≤1\}u\_\{t\}\\in\\mathcal\{U\}:=\\\{v\\in\\\{0,1\\\}^\{M\}:\\\|v\\\|\_\{0\}\\leq 1\\\}is the11\-sparse treatment recommendation, whereutiu^\{i\}\_\{t\}is11if recommending treatmentiiand0otherwise\.\|𝒰\|=M\+1\|\\mathcal\{U\}\|=M\+1as the decision maker may choose the null actionut=𝟎Mu\_\{t\}=\\bm\{0\}\_\{M\}\.dt∈𝒰d\_\{t\}\\in\\mathcal\{U\}is the patient’s11\-sparse adherence decision\.dtid\_\{t\}^\{i\}is11if the patient adheres to the recommendation and0otherwise\.σ\\sigmais the standard sigmoid\. The initial state,x1x\_\{1\}, has the same distribution as the noiseww\.\{assumption\}\(Full observation\)xtx\_\{t\},utu\_\{t\}, anddtd\_\{t\}are fully observed with no missingness\. This assumption comes from our DT setting, where it is common to gather self\-reported quantities from patients through daily check\-ins, called ecological momentary assessments \(EMAs\)\(Shiffmanet al\.[2008](https://arxiv.org/html/2605.24261#bib.bib62)\)\. Further, assuming treatments are hosted on the DT itself, the platform can reasonably monitor the patient’s adherence decisions in response to the recommended treatment\.

Parametera∈Θa:=\[0,a¯\]a\\in\\Theta\_\{a\}:=\[0,\\bar\{a\}\]captures the state’s persistence over time for knowna¯∈\(0,1\)\\bar\{a\}\\in\(0,1\)\.b∈Θb:=\{v∈ℝM:‖v‖∞≤b¯\}b\\in\\Theta\_\{b\}:=\\\{v\\in\\mathbb\{R\}^\{M\}:\\\|v\\\|\_\{\\infty\}\\leq\\bar\{b\}\\\}captures the effect of recommendation on the next state, for knownb¯∈ℝ\+\\bar\{b\}\\in\\mathbb\{R\}\_\{\+\}\.c∈Θc:=\{v∈ℝM:‖v‖∞≤c¯\}c\\in\\Theta\_\{c\}:=\\\{v\\in\\mathbb\{R\}^\{M\}:\\\|v\\\|\_\{\\infty\}\\leq\\bar\{c\}\\\}captures the effect of adherence on the next state, for knownc¯∈ℝ\+\\bar\{c\}\\in\\mathbb\{R\}\_\{\+\}\.μ∈Φ:=\[−μ¯,μ¯\]M\\mu\\in\\Phi:=\[\-\\bar\{\\mu\},\\bar\{\\mu\}\]^\{M\}contains the treatment\-specific shift parameters in the adherence model for knownμ¯∈ℝ\+\\bar\{\\mu\}\\in\\mathbb\{R\}\_\{\+\}\. Collectively,θ=\[a​b⊤​c⊤\]⊤\\theta=\[a\\ b^\{\\top\}\\ c^\{\\top\}\]^\{\\top\}andμ=\[μ1,…,μM\]⊤\\mu=\[\\mu\_\{1\},\\dots,\\mu\_\{M\}\]^\{\\top\}are fixed but unknown and must be learned online\. Last, we assume the following properties for the process noiseww:

\{assumption\}

\(Noise\)\{wt\}t=1T\\\{w\_\{t\}\\\}\_\{t=1\}^\{T\}are independent, identically distributed, zero\-symmetric,σs2\\sigma\_\{s\}^\{2\}\-subgaussian, bounded as\|wt\|≤w¯\|w\_\{t\}\|\\leq\\bar\{w\}\(w¯\>0\)\(\\bar\{w\}\>0\), have a log\-concave density, and known varianceσw2\>0\\sigma\_\{w\}^\{2\}\>0\. These assumptions are relatively mild and are satisfied by zero\-mean, symmetrically truncated Gaussian or uniform noise\. Similar analysis can be used without assuming boundedness at the expense of heavier technical machinery\. Last, we note two important properties for our system\. The construction of the system ensuresxtx\_\{t\}is bounded, and this boundedness sets upper and lower bounds for patient adherence\. Both properties play an important role in our downstream analysis\.

###### Proposition 3\.1

\(Bounded state\)Under Assumptions[3\.1](https://arxiv.org/html/2605.24261#S3.SS1)\-[3\.1](https://arxiv.org/html/2605.24261#S3.SS1),xt∈𝒳:=\[−Cx,Cx\]x\_\{t\}\\in\\mathcal\{X\}:=\[\-C\_\{x\},C\_\{x\}\]fort≥1t\\geq 1, withCx:=b¯\+c¯\+w¯1−a¯C\_\{x\}:=\\frac\{\\bar\{b\}\+\\bar\{c\}\+\\bar\{w\}\}\{1\-\\bar\{a\}\}\. Further,‖\[xt​ut⊤​dt⊤\]⊤‖22≤Cx2\+2\\\|\[x\_\{t\}\\ u\_\{t\}^\{\\top\}\\ d\_\{t\}^\{\\top\}\]^\{\\top\}\\\|^\{2\}\_\{2\}\\leq C\_\{x\}^\{2\}\+2\.

The proof \(Appendix[10\.1](https://arxiv.org/html/2605.24261#S10.SS1)\) follows from strictly stable dynamics and bounded inputs\.

## 4System identification

An important precondition for data\-driven DT algorithm design is that the patient model can be learned from observed trajectory data\. We characterize high\-probability guarantees for learningθ\\thetaandμ\\muthat hold for finite samples, bounding the rate at which estimation error decays with respect to trajectory length,TT, the number of treatments,MM, and other system\-specific parameters\. Extending these types of finite\-time guarantees to a variety of model classes has become an active focus of the system identification literature\(Tsiamiset al\.[2023](https://arxiv.org/html/2605.24261#bib.bib70)\)\. Our model departs from existing LDS work due to the presence of11\-sparse inputs and state\-dependent adherence decisions\. Our identification result, given as Theorem[4\.6](https://arxiv.org/html/2605.24261#S4.Thmtheorem6), demonstrates that we can recover the standard𝒪~​\(1/T\)\\tilde\{\\mathcal\{O\}\}\(\\nicefrac\{\{1\}\}\{\{\\sqrt\{T\}\}\}\)learning rate for LDS systems under a flexible exploratory policy class that we define\. This complements the Section[5](https://arxiv.org/html/2605.24261#S5)analysis of the separateUCB\-BOLDalgorithm, together establishing both identification guarantees for the patient model, and algorithmic regret guarantees for the proposed SDM setting\.

### 4\.1Estimators and exploratory policy class

We estimate the dynamics parameters,θ\\theta, with ordinary least squares \(OLS\), regressing the next statext\+1x\_\{t\+1\}on the feature vectorzt:=\[xt​ut⊤​dt⊤\]⊤z\_\{t\}:=\[x\_\{t\}\\ u\_\{t\}^\{\\top\}\\ d\_\{t\}^\{\\top\}\]^\{\\top\}:θ^T:=arg⁡minθ∈ℝ2​M\+1⁡1/2​∑t=1T\(xt\+1−θ⊤​zt\)2\\hat\{\\theta\}\_\{T\}:=\\arg\\min\_\{\\theta\\in\\mathbb\{R\}^\{2M\+1\}\}\\nicefrac\{\{1\}\}\{\{2\}\}\\sum\_\{t=1\}^\{T\}\(x\_\{t\+1\}\-\\theta^\{\\top\}z\_\{t\}\)^\{2\}\. We estimate the adherence parameters,μi\\mu\_\{i\}, by maximum likelihood estimation \(MLE\):μ^i,T:=arg⁡maxμi∈\[−μ¯,μ¯\]​∑t=1T𝟏\{ut=ei\}​\[dti​log⁡\(σ​\(xt\+μi\)\)\+\(1−dti\)​log⁡\(1−σ​\(xt\+μi\)\)\]\\hat\{\\mu\}\_\{i,T\}:=\\arg\\max\_\{\\mu\_\{i\}\\in\[\-\\bar\{\\mu\},\\bar\{\\mu\}\]\}\\sum\_\{t=1\}^\{T\}\\mathbf\{1\}\_\{\\\{u\_\{t\}=e\_\{i\}\\\}\}\\bigl\[d^\{i\}\_\{t\}\\log\(\\sigma\(x\_\{t\}\+\\mu\_\{i\}\)\)\+\(1\-d^\{i\}\_\{t\}\)\\log\(1\-\\sigma\(x\_\{t\}\+\\mu\_\{i\}\)\)\\bigr\]\. We assume that the decision\-maker acts per an\(r,k\)\(r,k\)\-exploratory policy\. This policy class adapts continuous white noise excitation commonly used in system identification to our discrete action set\.

###### Definition 4\.1

\(r,k\)\(r,k\)\-Exploratory Policy\. Letr∈\(0,1\]r\\in\(0,1\]andk∈ℕk\\in\\mathbb\{N\}\. A policy is an\(r,k\)\(r,k\)\-exploratory policy if for everykk\-block in a trajectory \(i\.e\.,t∈\{j\+1,…,j\+k\}t\\in\\\{j\+1,\\dots,j\+k\\\}for anyj≥0j\\geq 0\), the policy almost surely takes at least⌈r⋅k⌉\\lceil r\\cdot k\\rceilactions drawn uniformly from𝒰\\mathcal\{U\}independent ofℱt−1\\mathcal\{F\}\_\{t\-1\}\.

This definition permits the decision\-maker to choose which steps are exploratory based on history \(including the current statextx\_\{t\}\), so long as the block\-level exploration condition is satisfied\. The limiting cases ofr=1r=1ork=1k=1imply a purely random policy, but this framework can flexibly accommodate a randomized trial where the exploration level is fixed \(i\.e\., to roughlyrr\) and a separate policy is used in non\-exploratory time steps\. Further, our identification result does not rely on the policy for non\-exploratory actions, which can be adversarial or non\-stationary\.

### 4\.2Identification strategy

We refer to the patient’s true parameters asθ∗\\theta^\{\*\}andμ∗\\mu^\{\*\}\. Our identification result combines separate analysis of the OLS and MLE estimators\. The OLS\-portion of our system identification result builds on a result bySimchowitzet al\.\([2018](https://arxiv.org/html/2605.24261#bib.bib64)\), restated as Theorem[4\.2](https://arxiv.org/html/2605.24261#S4.Thmtheorem2)\. The analysis of the MLE estimators for the sigmoid shifts relies on standard concentration arguments for generalized linear models \(GLM\)\(Filippiet al\.[2010](https://arxiv.org/html/2605.24261#bib.bib26)\)\(see Appendix[11](https://arxiv.org/html/2605.24261#S11)\)\.

###### Theorem 4\.2

\(Simchowitzet al\.\([2018](https://arxiv.org/html/2605.24261#bib.bib64)\)Theorem 2\.4\) Fixδ∈\(0,1\)\\delta\\in\(0,1\),T∈ℕT\\in\\mathbb\{N\}, and0≺Γs​b⪯Γ¯0\\prec\\Gamma\_\{sb\}\\preceq\\bar\{\\Gamma\}\. Consider a random process\(Zt,Yt\)t≥1∈\(ℝd×ℝn\)\(Z\_\{t\},Y\_\{t\}\)\_\{t\\geq 1\}\\in\(\\mathbb\{R\}^\{d\}\\times\\mathbb\{R\}^\{n\}\)and a filtration\{ℱt\}t≥1\\\{\\mathcal\{F\}\_\{t\}\\\}\_\{t\\geq 1\}\. Suppose the following conditions hold: \(a\)Yt=θ∗​Zt\+ηtY\_\{t\}=\\mathbf\{\\theta\}^\{\*\}Z\_\{t\}\+\\eta\_\{t\}, whereηt\|ℱt\\eta\_\{t\}\|\\mathcal\{F\}\_\{t\}isσ2\\sigma^\{2\}\-sub\-Gaussian and mean zero, \(b\)\(Zt\)t≥1\(Z\_\{t\}\)\_\{t\\geq 1\}is an\{ℱt\}t≥1\\\{\\mathcal\{F\}\_\{t\}\\\}\_\{t\\geq 1\}\-adapted random process satisfying the\(k,Γs​b,p\)\(k,\\Gamma\_\{sb\},p\)\-small ball condition, and \(c\)𝐏​\[∑t=1TZt​Zt⊤⋠T​Γ¯\]≤δ\\mathbf\{P\}\[\\sum\_\{t=1\}^\{T\}Z\_\{t\}Z\_\{t\}^\{\\top\}\\not\\preceq T\\bar\{\\Gamma\}\]\\leq\\delta\. Then ifT≥10​k/p2​\(log⁡\(1/δ\)\+2​d​log⁡\(10/p\)\+log​det\(Γ¯​Γs​b−1\)\)T\\geq\\nicefrac\{\{10k\}\}\{\{p^\{2\}\}\}\(\\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)\+2d\\log\(\\nicefrac\{\{10\}\}\{\{p\}\}\)\+\\log\\det\(\\bar\{\\Gamma\}\\Gamma\_\{sb\}^\{\-1\}\)\), we have𝐏​\[‖θ^​\(T\)−θ∗‖op\>90​σp​n\+d​log⁡\(10/p\)\+log​detΓ¯​Γs​b−1\+log⁡\(1/δ\)T​λmin​\(Γs​b\)\]≤3​δ\\mathbf\{P\}\[\\\|\\hat\{\\mathbf\{\\theta\}\}\(T\)\-\\mathbf\{\\theta\}^\{\*\}\\\|\_\{\\text\{op\}\}\>\\frac\{90\\sigma\}\{p\}\\sqrt\{\\tfrac\{n\+d\\log\(\\nicefrac\{\{10\}\}\{\{p\}\}\)\+\\log\\det\\bar\{\\Gamma\}\\Gamma\_\{sb\}^\{\-1\}\+\\log\(\\nicefrac\{\{1\}\}\{\{\\delta\}\}\)\}\{T\\lambda\_\{\\text\{min\}\}\(\\Gamma\_\{sb\}\)\}\}\]\\leq 3\\delta\.

This result provides flexible machinery for finite\-time linear system analysis\. Intuitively, precondition \(b\) ensures that the system is sufficiently excited while precondition \(c\) ensures the empirical covariance is bounded such that self\-normalized martingale arguments can be used\. Precondition \(b\) relies on the following definition fromSimchowitzet al\.\([2018](https://arxiv.org/html/2605.24261#bib.bib64)\):

###### Definition 4\.3

Block Martingale Small Ball \(BMSB\)\. Let\(Zt\)t≥1\(Z\_\{t\}\)\_\{t\\geq 1\}be an\{ℱt\}t≥1\\\{\\mathcal\{F\}\_\{t\}\\\}\_\{t\\geq 1\}\-adapted random process taking values inℝd\\mathbb\{R\}^\{d\}\.\(Zt\)t≥1\(Z\_\{t\}\)\_\{t\\geq 1\}satisfies the\(k,Γs​b,p\)\(k,\\Gamma\_\{sb\},p\)\-BMSB condition for a positive integerkk, a positive definite matrixΓs​b\\Gamma\_\{sb\}, and0≤p≤10\\leq p\\leq 1, if for any fixedλ∈ℝd\\lambda\\in\\mathbb\{R\}^\{d\}with‖λ‖2=1\\\|\\lambda\\\|\_\{2\}=1, the process\(Zt\)t≥1\(Z\_\{t\}\)\_\{t\\geq 1\}satisfies1k​∑i=1k𝐏​\(\|λ⊤​Zj\+i\|≥λ⊤​Γs​b​λ\|ℱj\)≥p\\frac\{1\}\{k\}\\sum\_\{i=1\}^\{k\}\\mathbf\{P\}\(\|\\lambda^\{\\top\}Z\_\{j\+i\}\|\\geq\\sqrt\{\\lambda^\{\\top\}\\Gamma\_\{sb\}\\lambda\}\|\\mathcal\{F\}\_\{j\}\)\\geq palmost surely for anyj≥0j\\geq 0\.

Our primary contribution in this section is adapting the BMSB machinery to our system\. We provide our system\-specific preconditions \(b\) and \(c\) here\.

###### Proposition 4\.4

Informal precondition \(b\)\. Let Assumptions[3\.1](https://arxiv.org/html/2605.24261#S3.SS1)\-[3\.1](https://arxiv.org/html/2605.24261#S3.SS1)hold and actions come from an\(r,k\)\(r,k\)\-exploratory policy\. Then for an appropriateτ∈\(0,1\)\\tau\\in\(0,1\)\(see Lemma[10\.5](https://arxiv.org/html/2605.24261#S10.Thmtheorem5)in Appendix[10\.2](https://arxiv.org/html/2605.24261#S10.SS2)\), our system satisfies the\(k,1−τ28​M​I2​M\+1,p\)\(k,\\frac\{1\-\\tau^\{2\}\}\{8M\}I\_\{2M\+1\},p\)BMSB condition forppof orderrM\\frac\{r\}\{M\}\.

The proof partitions the unit sphere into three cases, showing that the excitation condition can always be satisfied by randomness from one of the process noise, random action selection, and random adherence outcomes\. The formal version is given as Proposition[10\.7](https://arxiv.org/html/2605.24261#S10.Thmtheorem7)in Appendix[10\.2](https://arxiv.org/html/2605.24261#S10.SS2)\.

###### Proposition 4\.5

Precondition \(c\)Let Assumptions[3\.1](https://arxiv.org/html/2605.24261#S3.SS1)\-[3\.1](https://arxiv.org/html/2605.24261#S3.SS1)hold\. Then forΓ¯:=\(Cx2\+2\)​𝐈2​M\+1\\bar\{\\Gamma\}:=\(C\_\{x\}^\{2\}\+2\)\\mathbf\{I\}\_\{2M\+1\},𝐏​\[∑t=1Tzt​zt⊤⋠T​Γ¯\]=0\\mathbf\{P\}\[\\sum\_\{t=1\}^\{T\}z\_\{t\}z\_\{t\}^\{\\top\}\\not\\preceq T\\bar\{\\Gamma\}\]=0, i\.e\.∑t=1Tzt​zt⊤⪯T​Γ¯\\sum\_\{t=1\}^\{T\}z\_\{t\}z\_\{t\}^\{\\top\}\\preceq T\\bar\{\\Gamma\}almost surely\.

Simchowitzet al\.\([2018](https://arxiv.org/html/2605.24261#bib.bib64)\)derive a high probability bound usingΓ¯\\bar\{\\Gamma\}from the controllability Gramian, as this matrix is available in closed form for their system\. This type of analysis is not possible for our system due to the sigmoidal adherence mechanism, but our system’s bounded covariates permit us to constructΓ¯\\bar\{\\Gamma\}such that our bound holds deterministically\.

### 4\.3Identification result

We synthesize theθ\\thetaandμ\\muconcentration arguments into an identification result for our system\.

###### Theorem 4\.6

Let Assumptions[3\.1](https://arxiv.org/html/2605.24261#S3.SS1)\-[3\.1](https://arxiv.org/html/2605.24261#S3.SS1)hold and actions come from an\(r,k\)\(r,k\)\-exploratory policy\. Fixδ∈\(0,1\)\\delta\\in\(0,1\),τ∈\(0,1\)\\tau\\in\(0,1\)per Lemma[10\.5](https://arxiv.org/html/2605.24261#S10.Thmtheorem5), andT∈ℕT\\in\\mathbb\{N\}\. Letp=min⁡\{p1​\(τ\),r​p¯M\+1\}p=\\min\\\{p\_\{1\}\(\\tau\),\\frac\{r\\underline\{p\}\}\{M\+1\}\\\}\(see Appendix[10\.2](https://arxiv.org/html/2605.24261#S10.SS2)\)\. IfT≥max⁡\{TOLS,TMLE,2​k\}T\\geq\\max\\\{T\_\{\\text\{OLS\}\},T\_\{\\text\{MLE\}\},2k\\\}forTOLS=10​kp2​\[log⁡\(6/δ\)\+\(2​M\+1\)​\(2​log⁡\(10/p\)\+log⁡\(8​M​\(Cx2\+2\)1−τ2\)\)\]T\_\{\\text\{OLS\}\}=\\tfrac\{10k\}\{p^\{2\}\}\[\\log\(\\nicefrac\{\{6\}\}\{\{\\delta\}\}\)\+\(2M\+1\)\(2\\log\(\\nicefrac\{\{10\}\}\{\{p\}\}\)\+\\log\(\\frac\{8M\(C\_\{x\}^\{2\}\+2\)\}\{1\-\\tau^\{2\}\}\)\)\],TMLE=k\+8/r​\(M\+1\)​log⁡\(4​M/δ\)T\_\{\\text\{MLE\}\}=k\+\\nicefrac\{\{8\}\}\{\{r\}\}\(M\+1\)\\log\(\\nicefrac\{\{4M\}\}\{\{\\delta\}\}\), then with probability at least1−δ1\-\\delta:

‖θ^​\(T\)−θ∗‖2≤90​σsp​8​MT​\(1−τ2\)​\[1\+log⁡\(6/δ\)\+\(2​M\+1\)​log⁡\(80​M​\(Cx2\+2\)p​\(1−τ2\)\)\],\\displaystyle\\\|\\hat\{\\theta\}\(T\)\-\\theta^\{\*\}\\\|\_\{2\}\\leq\\tfrac\{90\\sigma\_\{s\}\}\{p\}\\sqrt\{\\tfrac\{8M\}\{T\(1\-\\tau^\{2\}\)\}\[1\+\\log\(\\nicefrac\{\{6\}\}\{\{\\delta\}\}\)\+\(2M\+1\)\\log\(\\tfrac\{80M\(C\_\{x\}^\{2\}\+2\)\}\{p\(1\-\\tau^\{2\}\)\}\)\]\},\(2\)andmaxi∈\{1,…,M\}⁡\|μ^i,T−μi∗\|≤4​\(M\+1\)​2​log⁡\(8​M/δ\)p¯2​r​T\.\\displaystyle\\max\\nolimits\_\{i\\in\\\{1,\\dots,M\\\}\}\|\\hat\{\\mu\}\_\{i,T\}\-\\mu\_\{i\}^\{\*\}\|\\leq\\tfrac\{4\(M\+1\)\\sqrt\{2\\log\(\\nicefrac\{\{8M\}\}\{\{\\delta\}\}\)\}\}\{\\underline\{p\}^\{2\}r\\sqrt\{T\}\}\.\(3\)

The proof for Theorem[4\.6](https://arxiv.org/html/2605.24261#S4.Thmtheorem6)can be found in Appendix[12](https://arxiv.org/html/2605.24261#S12)\. Thus we demonstrate that theθ\\thetaandμ\\muparameters for the patient model proposed in Section[3](https://arxiv.org/html/2605.24261#S3)are identified at the standard parametric𝒪~​\(1/T\)\\tilde\{\\mathcal\{O\}\}\\left\(\\nicefrac\{\{1\}\}\{\{\\sqrt\{T\}\}\}\\right\)rate under\(r,k\)\(r,k\)\-policies\. We emphasize the flexibility of this policy class, allowing the decision\-maker to enact arbitrary policies \(i\.e\., non\-stationary or adversarial\) in non\-exploratory steps\. Further, the result only depends on the model and policy parameters, not the properties of any specific realized trajectory\. The primary identification bottleneck is the estimation of the dynamics parameters,𝒪~​\(M2r​T\)\\tilde\{\\mathcal\{O\}\}\\left\(\\frac\{M^\{2\}\}\{r\\sqrt\{T\}\}\\right\), as compared to the adherence shift parameters,𝒪~​\(Mr​T\)\\tilde\{\\mathcal\{O\}\}\\left\(\\frac\{M\}\{r\\sqrt\{T\}\}\\right\)\. This makes sense, as we use the bounded state space to lower bound per\-sample Fisher information forμi\\mu\_\{i\}in each adherence Bernoulli trial; this lower bound means that we do not need to account for the autocorrelation ofxtx\_\{t\}and thus reducesμ\\muestimation to Bernoulli concentration for each of theMMtreatments\. In contrast, we cannot sidestep the autocorrelation of the feature vector, which we handle with the heavier BMSB machinery\.

The BMSB machinery provides a natural framework for analyzing our system; while the state\-dependent adherence mechanism complicates the distribution of the covariates themselves, the regression problem is still linear inθ\\theta\(i\.e\., precisely the motivating setting of the BMSB machinery\)\. We show that this machinery can be adapted to handle the unique departures of our system from a standard LDS\. In particular, we show how the exploration induced by the\(r,k\)\(r,k\)\-policy ensures that the BMSB condition is satisfied and we exploit our bounded state space to bound the empirical covariance\. One consequence of the combination of the\(r,k\)\(r,k\)\-policy class,11\-sparse inputs, and this BMSB machinery is that the resulting rate𝒪~​\(M2r​T\)\\tilde\{\\mathcal\{O\}\}\\left\(\\frac\{M^\{2\}\}\{r\\sqrt\{T\}\}\\right\)is likely loose inMM\(i\.e\., theSimchowitzet al\.\([2018](https://arxiv.org/html/2605.24261#bib.bib64)\)LDS result is𝒪~​\(d\)\\tilde\{\\mathcal\{O\}\}\(\\sqrt\{d\}\)in the covariate dimensiondd, but assumes continuous noise that excites all directions simultaneously\)\. We suspect that an analysis that separates learning of the autoregressive parameteraafrom the bandit\-like parametersbbandccmay yield tighter rates, but this separation is non\-trivial, as the state\-dependent adherence mechanism couples analysis of the parameters\. Such analysis is a natural direction for future work\.

## 5Policy optimization

Section[4](https://arxiv.org/html/2605.24261#S4)demonstrated that our patient model is identifiable under the\(r,k\)\(r,k\)\-exploratory policy class\. This type of forced exploration is needed to identify all patient parameters to a given accuracy, but comes at the cost of linear regret rates\. While this makes sense in some settings, for instance randomized trials, a DT decision\-maker is typically motivated to learn patient parameters only to an accuracy sufficient to make good recommendation decisions\. To this end, we adopt the principle of optimism\(Lai and Robbins[1985](https://arxiv.org/html/2605.24261#bib.bib40)\)underpinning upper confidence bound \(UCB\) approaches\. In this paradigm, algorithms adapt their exploration based on current parameter uncertainty, only exploring where uncertainty is large enough that a better policy might still exist\. We propose an optimism\-based algorithm,UCB\-BOLD, that extends UCB approaches from bandit settings to the RL setting faced by DT decision\-makers, and prove that it achieves sublinear regret in this context\.

### 5\.1Problem formulation

In our DT setting, a decision\-maker is ultimately motivated to achieve long\-term patient health\. Patient adherence is needed to provide the clinical benefit of treatment, but this is conditional on the patient staying engaged with the DT and not dropping out\. To capture these separate motivations, we make the following assumption about the reward structure for our RL setting\.\{assumption\}\(Reward structure\)Time invariant rewardsr​\(x,u,d\):=−β​σ​\(β0−x\)\+∑i=1Mρi​dir\(x,u,d\):=\-\\beta\\sigma\(\\beta\_\{0\}\-x\)\+\\sum\_\{i=1\}^\{M\}\\rho\_\{i\}d\_\{i\}for knownρi∈\[0,ρ¯\]\\rho\_\{i\}\\in\[0,\\bar\{\\rho\}\],β∈ℝ\+\\beta\\in\\mathbb\{R\}\_\{\+\}, andβ0∈ℝ\\beta\_\{0\}\\in\\mathbb\{R\}\. Rewards are discounted by knownγ∈\(0,1\)\\gamma\\in\(0,1\)\. The decision maker receives a fixed, known reward,ρi\\rho\_\{i\}, when the patient adheres to theit​hi^\{th\}treatment recommendation, implying clinical guidance is available to assign values to the treatment set\. Since we do not explicitly model the possibility of a patient being completely removed from the DT, theβ\\beta\-scaled sigmoid enforces a tunable penalty for low state values\. A decision\-maker can tune the penalty and the discount factorγ\\gammato match their specific setting\. The decision\-maker must identify a policyπ\\pito maximize their discounted, infinite horizon return:J∗​\(x\)=maxπ∈Π⁡𝐄​\[∑t=1∞γt−1​r​\(xt,ut,dt\)\]J^\{\*\}\(x\)=\\max\_\{\\pi\\in\\Pi\}\\mathbf\{E\}\[\\sum\_\{t=1\}^\{\\infty\}\\gamma^\{t\-1\}r\(x\_\{t\},u\_\{t\},d\_\{t\}\)\], where the system structure and constraints are defined as per Section[3](https://arxiv.org/html/2605.24261#S3)andJ∗​\(x\)J^\{\*\}\(x\)is the optimal value function\. While we focus our analysis on sigmoidal penalties and rewards, our approach readily extends to any bounded and Lipschitz continuous stage cost functions\.

### 5\.2Optimism\-based algorithm

Here we present our epoch\-based algorithm for optimizing treatment recommendation decisions: Upper Confidence Bound with Binary Outcomes and Linear Dynamics \(UCB\-BOLD\), shown as Algorithm[1](https://arxiv.org/html/2605.24261#alg1)\. At the start of each epoch, indexed bykk,UCB\-BOLDuses observations up to the current time step to calculate parameter estimates and confidence sets, i\.e\.,θ^\(k\)=θ^t\\hat\{\\theta\}^\{\(k\)\}=\\hat\{\\theta\}\_\{t\}andμ^\(k\)=μ^t\\hat\{\\mu\}^\{\(k\)\}=\\hat\{\\mu\}\_\{t\}\(with the confidence sets calculated similarly\)\. These quantities define an optimistic surrogate system, identical to the true system except that it is parameterized byθ^\(k\)\\hat\{\\theta\}^\{\(k\)\}andμ^\(k\)\\hat\{\\mu\}^\{\(k\)\}and it uses an augmented, optimistic reward that encodes current uncertainty in the parameters via the confidence sets\. The algorithm solves for the optimal value function of the surrogate system using value iteration \(VI\) on a discretized grid, and implements the resulting stationary policy for the remainder of the epoch\. A new epoch begins when a triggering condition is met, based on either the determinant of the regularized second moment matrix or the number of times each action has been taken\. When a new epoch begins, the estimators and confidence sets are re\-calculated and the process repeats\. While the within\-epoch policy is deterministic, the algorithm is prompted to explore strategically based on parameter uncertainty via the surrogate system’s augmented reward\. Unlike the forced exploration of an\(r,k\)\(r,k\)\-policy, this optimism\-driven exploration only reduces parameter uncertainty to a point where actions can be determined to be suboptimal, leading to a sublinear regret rate\.

Algorithm 1Upper Confidence Bound with Binary Outcomes, Linear Dynamics \(UCB\-BOLD\)1:Confidence

δ∈\(0,1\)\\delta\\in\(0,1\), discount factor

γ∈\(0,1\)\\gamma\\in\(0,1\), regularizers

λ1,λ2\>0\\lambda\_\{1\},\\lambda\_\{2\}\>0, constants

Cd,CN\>0C\_\{d\},C\_\{N\}\>0\.

2:Set epoch count

k=1k=1and initialize action counts

Ni=0N\_\{i\}=0\(

i=1,…,Mi=1,\\dots,M\)\.

3:Set

V¯\(k\)=λ1​I2​M\+1\\bar\{V\}^\{\(k\)\}=\\lambda\_\{1\}I\_\{2M\+1\},

θ^\(k\)=θ^1\\hat\{\\theta\}^\{\(k\)\}=\\hat\{\\theta\}\_\{1\},

μ^\(k\)=μ^1\\hat\{\\mu\}^\{\(k\)\}=\\hat\{\\mu\}\_\{1\},

αθ,\(k\)=α1θ​\(δ/2\)\\alpha^\{\\theta,\(k\)\}=\\alpha^\{\\theta\}\_\{1\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\),

αiμ,\(k\)=αi,1μ​\(δ/2\)\\alpha\_\{i\}^\{\\mu,\(k\)\}=\\alpha^\{\\mu\}\_\{i,1\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\), action counts

Ni\(k\)=Ni​∀iN\_\{i\}^\{\(k\)\}=N\_\{i\}\\;\\forall i\.

4:Calculate optimistic

J~δ\(k\)​\(x\)\\tilde\{J\}\_\{\\delta\}^\{\(k\)\}\(x\)using VI with

𝒯~δ\(k\)\\tilde\{\\mathcal\{T\}\}\_\{\\delta\}^\{\(k\)\}and calculate stationary optimal policy

π\(k\)\\pi^\{\(k\)\}per

J~δ\(k\)\\tilde\{J\}\_\{\\delta\}^\{\(k\)\}
5:fort = 1,…,Tdo

6:Observe

dt−1d\_\{t\-1\},

xtx\_\{t\}, update

V¯t\\bar\{V\}\_\{t\}
7:if

det\(V¯t\)\>\(1\+Cd\)​det\(V¯\(k\)\)\\det\(\\bar\{V\}\_\{t\}\)\>\(1\+C\_\{d\}\)\\det\(\\bar\{V\}^\{\(k\)\}\)OR

Ni\>\(1\+CN\)​Ni\(k\)N\_\{i\}\>\(1\+C\_\{N\}\)N\_\{i\}^\{\(k\)\}for any

i=1,…,Mi=1,\\dots,Mthen

8:Set

k=k\+1k=k\+1,

V¯\(k\)=V¯t\\bar\{V\}^\{\(k\)\}=\\bar\{V\}\_\{t\},

θ^\(k\)=θ^t\\hat\{\\theta\}^\{\(k\)\}=\\hat\{\\theta\}\_\{t\},

μ^\(k\)=μ^t\\hat\{\\mu\}^\{\(k\)\}=\\hat\{\\mu\}\_\{t\},

αθ,\(k\)=αtθ​\(δ/2\)\\alpha^\{\\theta,\(k\)\}=\\alpha^\{\\theta\}\_\{t\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\),

αiμ,\(k\)=αi,tμ​\(δ/2\)\\alpha^\{\\mu,\(k\)\}\_\{i\}=\\alpha^\{\\mu\}\_\{i,t\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\),

Ni\(k\)=NiN\_\{i\}^\{\(k\)\}=N\_\{i\}for

i=1,…,Mi=1,\\dots,M
9:Calculate optimistic

J~δ\(k\)​\(x\)\\tilde\{J\}\_\{\\delta\}^\{\(k\)\}\(x\)using VI with

𝒯~δ\(k\)\\tilde\{\\mathcal\{T\}\}\_\{\\delta\}^\{\(k\)\}and calculate stationary optimal policy

π\(k\)\\pi^\{\(k\)\}per

J~δ\(k\)\\tilde\{J\}\_\{\\delta\}^\{\(k\)\}
10:endif

11:Take action

ut=π\(k\)​\(xt\)u\_\{t\}=\\pi^\{\(k\)\}\(x\_\{t\}\)
12:Set

Ni=Ni\+𝟏\{ut=ei\}N\_\{i\}=N\_\{i\}\+\\mathbf\{1\}\_\{\\\{u\_\{t\}=e\_\{i\}\\\}\},

i=1,…,Mi=1,\\dots,M
13:endfor

### 5\.3UCB\-BOLDcomponents

As in Section[4](https://arxiv.org/html/2605.24261#S4), we use separate estimation strategies for the dynamics \(θ\\theta\) and adherence \(μ\\mu\) parameters, as well as separate techniques for constructing their respective confidence sets\. These elements allow us to define the optimistic surrogate system necessary for our regret analysis\.

#### 5\.3\.1Estimator and confidence set construction

At each time step, we useℓ2\\ell\_\{2\}\-regularized regression to compute a regularized least squares estimator forθ\\theta, i\.e\., for someλ1\>0\\lambda\_\{1\}\>0:

θ^tR​L​S:=\\argminθ∈ℝ2​M\+1​12​\[∑s=1t−1\(xs\+1−θ⊤​zs\)2\+λ1​‖θ‖22\]\\displaystyle\\hat\{\\theta\}^\{RLS\}\_\{t\}:=\\argmin\\nolimits\_\{\\theta\\in\\mathbb\{R\}^\{2M\+1\}\}\\tfrac\{1\}\{2\}\[\\textstyle\\sum\\nolimits\_\{s=1\}^\{t\-1\}\(x\_\{s\+1\}\-\\theta^\{\\top\}z\_\{s\}\)^\{2\}\+\\lambda\_\{1\}\\\|\\theta\\\|\_\{2\}^\{2\}\]\(4\)θ^t\\hat\{\\theta\}\_\{t\}is the weighted projection ofθ^tR​L​S\\hat\{\\theta\}\_\{t\}^\{RLS\}onto the known parameter spaceΘ=Θa×Θb×Θc\\Theta=\\Theta\_\{a\}\\times\\Theta\_\{b\}\\times\\Theta\_\{c\}:θ^t:=arg⁡minθ∈Θ⁡‖θ−θ^tR​L​S‖V¯t2\\hat\{\\theta\}\_\{t\}:=\\arg\\min\_\{\\theta\\in\\Theta\}\\\|\\theta\-\\hat\{\\theta\}^\{RLS\}\_\{t\}\\\|\_\{\\bar\{V\}\_\{t\}\}^\{2\}, whereV¯t:=∑s=1t−1zs​zs⊤\+λ1​I2​M\+1\\bar\{V\}\_\{t\}:=\\sum\_\{s=1\}^\{t\-1\}z\_\{s\}z\_\{s\}^\{\\top\}\+\\lambda\_\{1\}I\_\{2M\+1\}is the regularized second moment matrix available at the beginning of time steptt\. AsΘ\\Thetais a convex set, we can computeθ^t\\hat\{\\theta\}\_\{t\}using the quadratic program:minθ∈Θ⁡1/2​θ⊤​V¯t​θ−\(V¯t​θ^tR​L​S\)⊤​θ\\min\_\{\\theta\\in\\Theta\}\\nicefrac\{\{1\}\}\{\{2\}\}\\theta^\{\\top\}\\bar\{V\}\_\{t\}\\theta\-\(\\bar\{V\}\_\{t\}\\hat\{\\theta\}\_\{t\}^\{RLS\}\)^\{\\top\}\\theta\. We form confidence sets forθ\\thetaby Proposition[5\.1](https://arxiv.org/html/2605.24261#S5.Thmtheorem1)\.

###### Proposition 5\.1

\(θ\\thetaconfidence set\)Let Assumptions[3\.1](https://arxiv.org/html/2605.24261#S3.SS1)\-[3\.1](https://arxiv.org/html/2605.24261#S3.SS1)hold\. Letδ∈\(0,1\)\\delta\\in\(0,1\)\. Then with probability at least1−δ1\-\\delta, the eventℰθ​\(δ\):=\{θ∗∈𝒞tθ​\(δ\),t≥1\}\\mathcal\{E\}\_\{\\theta\}\(\\delta\):=\\\{\\theta^\{\*\}\\in\\mathcal\{C\}^\{\\theta\}\_\{t\}\(\\delta\),\\;t\\geq 1\\\}holds, where:𝒞tθ​\(δ\)=\{θ∈ℝ2​M\+1:‖θ^t−θ‖V¯t≤αtθ​\(δ\)\}\\mathcal\{C\}^\{\\theta\}\_\{t\}\(\\delta\)=\\\{\\theta\\in\\mathbb\{R\}^\{2M\+1\}:\\\|\\hat\{\\theta\}\_\{t\}\-\\theta\\\|\_\{\\overline\{V\}\_\{t\}\}\\leq\\alpha^\{\\theta\}\_\{t\}\(\\delta\)\\\}, withαtθ​\(δ\):=σs​\(2​M\+1\)​log⁡\(1δ\+t​\(Cx2\+2\)δ​λ1\)\+λ1​\(a¯2\+M​b¯2\+M​c¯2\)\\alpha^\{\\theta\}\_\{t\}\(\\delta\):=\\sigma\_\{s\}\\sqrt\{\(2M\+1\)\\log\(\\frac\{1\}\{\\delta\}\+\\frac\{t\(C\_\{x\}^\{2\}\+2\)\}\{\\delta\\lambda\_\{1\}\}\)\}\+\\sqrt\{\\lambda\_\{1\}\(\\bar\{a\}^\{2\}\+M\\bar\{b\}^\{2\}\+M\\bar\{c\}^\{2\}\)\}\.

The proof is in Appendix[13\.1](https://arxiv.org/html/2605.24261#S13.SS1)\. The dynamics portion of our model maps to a result for the linear bandit setting given byAbbasi\-Yadkoriet al\.\([2011](https://arxiv.org/html/2605.24261#bib.bib1)\)\. The proof shows that our system satisfies the conditions of their result, and that the confidence sets can be re\-centered fromθ^tR​L​S\\hat\{\\theta\}\_\{t\}^\{RLS\}toθ^t\\hat\{\\theta\}\_\{t\}\. To estimate each sigmoid shiftμi\\mu\_\{i\}we use regularized MLE at each time step, i\.e\., for someλ2\>0\\lambda\_\{2\}\>0:

μ^i,tM​L​E=\\argmaxμi∈ℝ​∑s=1t−1𝟏\{us=ei\}​\[dsi​log⁡\(σ​\(xs\+μi\)\)\+\(1−dsi\)​log⁡\(1−σ​\(xs\+μi\)\)\]−λ22​μi2\\displaystyle\\hat\{\\mu\}\_\{i,t\}^\{MLE\}=\\argmax\\nolimits\_\{\\mu\_\{i\}\\in\\mathbb\{R\}\}\\textstyle\\sum\\nolimits\_\{s=1\}^\{t\-1\}\\mathbf\{1\}\_\{\\\{u\_\{s\}=e\_\{i\}\\\}\}\\left\[d^\{i\}\_\{s\}\\log\(\\sigma\(x\_\{s\}\+\\mu\_\{i\}\)\)\+\(1\-d^\{i\}\_\{s\}\)\\log\(1\-\\sigma\(x\_\{s\}\+\\mu\_\{i\}\)\)\\right\]\-\\tfrac\{\\lambda\_\{2\}\}\{2\}\\mu\_\{i\}^\{2\}\(5\)Our final estimatorμ^i,t\\hat\{\\mu\}\_\{i,t\}is the projection ofμ^i,tM​L​E\\hat\{\\mu\}\_\{i,t\}^\{MLE\}onto the known parameter space\[−μ¯,μ¯\]\[\-\\bar\{\\mu\},\\bar\{\\mu\}\]\. We form confidence sets forμ\\muper Proposition[5\.2](https://arxiv.org/html/2605.24261#S5.Thmtheorem2)\.

###### Proposition 5\.2

\(μ\\muconfidence set\)Let Assumptions[3\.1](https://arxiv.org/html/2605.24261#S3.SS1)\-[3\.1](https://arxiv.org/html/2605.24261#S3.SS1)hold\. Letδ∈\(0,1\)\\delta\\in\(0,1\)\. Then with probability at least1−δ1\-\\deltathe eventℰμ​\(δ\):=\{μ∗∈𝒞tμ​\(δ\),t≥1\}\\mathcal\{E\}\_\{\\mu\}\(\\delta\):=\\\{\\mu^\{\*\}\\in\\mathcal\{C\}^\{\\mu\}\_\{t\}\(\\delta\),t\\geq 1\\\}holds where:𝒞tμ​\(δ\)=\{μ∈ℝM:\|μ^i,t−μi\|≤αi,tμ​\(δ\),i=1,…,M\}\\mathcal\{C\}^\{\\mu\}\_\{t\}\(\\delta\)=\\\{\\mu\\in\\mathbb\{R\}^\{M\}:\\left\|\\hat\{\\mu\}\_\{i,t\}\-\\mu\_\{i\}\\right\|\\leq\\alpha^\{\\mu\}\_\{i,t\}\(\\delta\),\\;i=1,\\dots,M\\\}, forαi,tμ​\(δ\):=e3​μ¯Ht​\(μ^i,t\)​\[λ22\+2λ2​\(μ¯\+log⁡\(2​M​Ht​\(μ^i,t\)δ​λ2\)\)\]\+e2​μ¯​λ2​μ¯Ht​\(μ^i,t\)\\alpha^\{\\mu\}\_\{i,t\}\(\\delta\):=\\frac\{e^\{3\\bar\{\\mu\}\}\}\{\\sqrt\{H\_\{t\}\(\\hat\{\\mu\}\_\{i,t\}\)\}\}\\bigl\[\\frac\{\\sqrt\{\\lambda\_\{2\}\}\}\{2\}\+\\frac\{2\}\{\\sqrt\{\\lambda\_\{2\}\}\}\\bigl\(\\bar\{\\mu\}\+\\log\\bigl\(\\frac\{2M\\sqrt\{H\_\{t\}\(\\hat\{\\mu\}\_\{i,t\}\)\}\}\{\\delta\\sqrt\{\\lambda\_\{2\}\}\}\\bigr\)\\bigr\)\\bigr\]\+\\frac\{e^\{2\\bar\{\\mu\}\}\\lambda\_\{2\}\\bar\{\\mu\}\}\{H\_\{t\}\(\\hat\{\\mu\}\_\{i,t\}\)\}andHt​\(μ^i,t\):=λ2\+∑s=1t−1𝟏\{us=ei\}​σ′​\(xs\+μ^i,t\)H\_\{t\}\(\\hat\{\\mu\}\_\{i,t\}\):=\\lambda\_\{2\}\+\\sum\\nolimits\_\{s=1\}^\{t\-1\}\\mathbf\{1\}\_\{\\\{u\_\{s\}=e\_\{i\}\\\}\}\\sigma^\{\\prime\}\(x\_\{s\}\+\\hat\{\\mu\}\_\{i,t\}\)\.

The proof is provided in Appendix[13\.1](https://arxiv.org/html/2605.24261#S13.SS1)\. We build on a result fromFauryet al\.\([2020](https://arxiv.org/html/2605.24261#bib.bib25)\), which considers the adjacent case of estimating scale parameters in the GLM bandit setting\. We show that their result \(see Theorem[13\.5](https://arxiv.org/html/2605.24261#S13.Thmtheorem5)\) can be adapted to form confidence sets for our shift parametersμi\\mu\_\{i\}\.

#### 5\.3\.2Optimistic surrogate system

The surrogate system is structured identically to the true system, except that it is parameterized by the current estimatesθ^\\hat\{\\theta\}andμ^\\hat\{\\mu\}and adds an exploration bonus\(Azaret al\.[2017](https://arxiv.org/html/2605.24261#bib.bib8), Jinet al\.[2023](https://arxiv.org/html/2605.24261#bib.bib37)\)to the reward\. This bonus is constructed specifically so that the surrogate system’s optimal value function overestimatesJ∗J^\{\*\}and is given by:

bt,δ​\(x,u\):=\(ρ¯\+γ​L1​c¯\)​\[∑i=1Mui​κ​\(x,μ^i,t\)​αi,tμ​\(δ/2\)\]\+γ​L1​αtθ​\(δ/2\)​𝐄μ^t​\[‖z‖V¯t−1\|x,u\]\\displaystyle b\_\{t,\\delta\}\(x,u\):=\(\\bar\{\\rho\}\+\\gamma L\_\{1\}\\bar\{c\}\)\\bigl\[\\textstyle\\sum\\nolimits\_\{i=1\}^\{M\}u^\{i\}\\kappa\(x,\\hat\{\\mu\}\_\{i,t\}\)\\alpha\_\{i,t\}^\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\\bigr\]\+\\gamma L\_\{1\}\\alpha^\{\\theta\}\_\{t\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\\mathbf\{E\}\_\{\\hat\{\\mu\}\_\{t\}\}\\big\[\\\|z\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}\|x,u\\big\]\(6\)L1L\_\{1\}is a Lipschitz constant \(see Lemma[13\.10](https://arxiv.org/html/2605.24261#S13.Thmtheorem10)\),κ​\(x,μ^i,t\):=min⁡\(1/4,e2​μ¯​σ′​\(x\+μ^i,t\)\)\\kappa\(x,\\hat\{\\mu\}\_\{i,t\}\):=\\min\(\\nicefrac\{\{1\}\}\{\{4\}\},e^\{2\\bar\{\\mu\}\}\\sigma^\{\\prime\}\(x\+\\hat\{\\mu\}\_\{i,t\}\)\)is a bound on the derivative of the sigmoid, and theαtθ\\alpha^\{\\theta\}\_\{t\}andαi,tμ\\alpha^\{\\mu\}\_\{i,t\}terms come from the confidence sets in Proposition[5\.1](https://arxiv.org/html/2605.24261#S5.Thmtheorem1)and Proposition[5\.2](https://arxiv.org/html/2605.24261#S5.Thmtheorem2)\. The expectation is over the adherence outcome\. Using an additive bonus simplifies VI as parameter uncertainty is captured in the reward instead of the dynamics\. The bonus term encourages exploration by rewarding actions that reduce parameter uncertainty \(i\.e\., high parameter uncertainty manifests as a large bonus value\)\. This bonus allows us to define a modified Bellman operator for the optimistic surrogate system,𝒯~t,δ\\tilde\{\\mathcal\{T\}\}\_\{t,\\delta\}, parameterized byθ^t\\hat\{\\theta\}\_\{t\}andμ^t\\hat\{\\mu\}\_\{t\}forδ∈\(0,1\)\\delta\\in\(0,1\)\. Our analysis also uses the typical Bellman operator,𝒯\{\\mathcal\{T\}\}, and the policy Bellman operator,𝒯π\\mathcal\{T\}^\{\\pi\}, both parameterized byθ∗\\theta^\{\*\}andμ∗\\mu^\{\*\}, defined as follows:

𝒯~t,δ​J​\(x\)\\displaystyle\\tilde\{\\mathcal\{T\}\}\_\{t,\\delta\}J\(x\):=maxu∈𝒰⁡\{𝐄θ^t,μ^t​\[r​\(x,u,d\)\+bt,δ​\(x,u\)\+γ​J​\(f​\(x,u,d,w\)\)\]\}\\displaystyle:=\\max\_\{u\\in\\mathcal\{U\}\}\\big\\\{\\mathbf\{E\}\_\{\\hat\{\\theta\}\_\{t\},\\hat\{\\mu\}\_\{t\}\}\\big\[r\(x,u,d\)\+b\_\{t,\\delta\}\(x,u\)\+\\gamma J\(f\(x,u,d,w\)\)\\big\]\\big\\\}\(7\)𝒯​J​\(x\)\\displaystyle\\mathcal\{T\}J\(x\):=maxu∈𝒰⁡\{𝐄θ∗,μ∗​\[r​\(x,u,d\)\+γ​J​\(f​\(x,u,d,w\)\)\]\}\\displaystyle:=\\max\_\{u\\in\\mathcal\{U\}\}\\bigl\\\{\\mathbf\{E\}\_\{\\theta^\{\*\},\\mu^\{\*\}\}\\bigl\[r\(x,u,d\)\+\\gamma J\(f\(x,u,d,w\)\)\\bigr\]\\bigr\\\}\(8\)𝒯π​J​\(x\)\\displaystyle\\mathcal\{T\}^\{\\pi\}J\(x\):=𝐄θ∗,μ∗​\[r​\(x,π​\(x\),d\)\+γ​J​\(f​\(x,π​\(x\),d,w\)\)\]\\displaystyle:=\\mathbf\{E\}\_\{\\theta^\{\*\},\\mu^\{\*\}\}\\big\[r\(x,\\pi\(x\),d\)\+\\gamma J\(f\(x,\\pi\(x\),d,w\)\)\\big\]\(9\)
###### Lemma 5\.3

𝒯\\mathcal\{T\},𝒯π\\mathcal\{T\}^\{\\pi\}, and𝒯~t,δ\\tilde\{\\mathcal\{T\}\}\_\{t,\\delta\}are monotone and converge uniformly to unique fixed pointsJ∗​\(x\)J^\{\*\}\(x\),Jπ​\(x\)J^\{\\pi\}\(x\), andJ~t,δ​\(x\)\\tilde\{J\}\_\{t,\\delta\}\(x\)\(withJ∗​\(x\)J^\{\*\}\(x\),J~t,δ​\(x\)\\tilde\{J\}\_\{t,\\delta\}\(x\)optimal\)\.J∗​\(x\)J^\{\*\}\(x\)andJπ​\(x\)J^\{\\pi\}\(x\)are bounded byCJ:=β\+ρ¯1−γC\_\{J\}:=\\frac\{\\beta\+\\bar\{\\rho\}\}\{1\-\\gamma\}\.J~t,δ​\(x\)\\tilde\{J\}\_\{t,\\delta\}\(x\)is bounded fort≤Tt\\leq TbyCJ~,T,δ:=β\+ρ¯\+Cb​\(T,δ\)1−γC\_\{\\tilde\{J\},T,\\delta\}:=\\frac\{\\beta\+\\bar\{\\rho\}\+C\_\{b\}\(T,\\delta\)\}\{1\-\\gamma\}, whereCb​\(T,δ\)C\_\{b\}\(T,\\delta\)is the bonus bound \(Lemma[13\.8](https://arxiv.org/html/2605.24261#S13.Thmtheorem8)\)\.

These properties follow from the bounded and discounted setting \(proof in Appendix[13\.2](https://arxiv.org/html/2605.24261#S13.SS2)\), and allow us to show the following optimism result \(T\+1T\+1is needed for downstream regret analysis\)\.

###### Lemma 5\.4

\(Valid optimism\)Letℰθ​\(δ/2\)\\mathcal\{E\}\_\{\\theta\}\\left\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\\right\)andℰμ​\(δ/2\)\\mathcal\{E\}\_\{\\mu\}\\left\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\\right\)hold \(Propositions[5\.1](https://arxiv.org/html/2605.24261#S5.Thmtheorem1)\-[5\.2](https://arxiv.org/html/2605.24261#S5.Thmtheorem2)\) forδ∈\(0,1\)\\delta\\in\(0,1\)\. ThenJ~t,δ​\(x\)≥J∗​\(x\)\\tilde\{J\}\_\{t,\\delta\}\(x\)\\geq J^\{\*\}\(x\)for anyx∈𝒳x\\in\\mathcal\{X\}, andt∈\{1,…,T\+1\}t\\in\\\{1,\\dots,T\+1\\\}\.

The proof is provided in Appendix[13\.2](https://arxiv.org/html/2605.24261#S13.SS2)\. Broadly, it shows that a single application of𝒯~t,δ\\tilde\{\\mathcal\{T\}\}\_\{t,\\delta\}toJ∗J^\{\*\}is optimistic \(i\.e\.,𝒯~t,δ​J∗​\(x\)≥J∗​\(x\)\\tilde\{\\mathcal\{T\}\}\_\{t,\\delta\}J^\{\*\}\(x\)\\geq J^\{\*\}\(x\)\), then uses the monotonicity and uniform convergence properties of the operators to show that this impliesJ~t,δ​\(x\)≥J∗​\(x\)\\tilde\{J\}\_\{t,\\delta\}\(x\)\\geq J^\{\*\}\(x\)\. The core proof step is leveraging theL1L\_\{1\}\-Lipschitz continuity ofJ∗​\(x\)J^\{\*\}\(x\)\(see Lemma[13\.10](https://arxiv.org/html/2605.24261#S13.Thmtheorem10)\) to express value function differences as estimation errors \(i\.e\.,\|θ^t−θ∗\|\|\\hat\{\\theta\}\_\{t\}\-\\theta^\{\*\}\|\), which are bounded by the confidence set Propositions[5\.1](https://arxiv.org/html/2605.24261#S5.Thmtheorem1)\-[5\.2](https://arxiv.org/html/2605.24261#S5.Thmtheorem2)\. This implicitly extends toUCB\-BOLD’s epoch quantities, which are a subset of the time\-indexed quantities\.

### 5\.4Regret analysis

We use a notion of regret from the discounted MDP literature\(Heet al\.[2021](https://arxiv.org/html/2605.24261#bib.bib33)\)that measures the cumulative sub\-optimality gap of the enacted policy:RT=∑t=1TJ∗​\(xt\)−Jπt​\(xt\)R\_\{T\}=\\sum\_\{t=1\}^\{T\}J^\{\*\}\(x\_\{t\}\)\-J^\{\\pi\_\{t\}\}\(x\_\{t\}\)\. This is the natural choice for regret in our bounded and discounted setting, as it captures convergence to an optimal policy even when the value functions themselves are bounded\. From this definition, we construct our main regret result for the performance ofUCB\-BOLD\.

###### Theorem 5\.5

\(Main regret result\)Letδa​l​l∈\(0,1\)\\delta\_\{all\}\\in\(0,1\)and Assumptions[3\.1](https://arxiv.org/html/2605.24261#S3.SS1)\-[5\.1](https://arxiv.org/html/2605.24261#S5.SS1)hold\. LetCd,CN\>0C\_\{d\},C\_\{N\}\>0be given by Algorithm[1](https://arxiv.org/html/2605.24261#alg1)\. Then with probability at least1−δa​l​l1\-\\delta\_\{all\}it holds that:

RT\\displaystyle R\_\{T\}≤14​\(1−γ\)​\[\(2​ρ¯\+γ​c¯​\(L1\+L2\)\)​νT​\(δa​l​l/6\)​1\+CN​\[Mλ2\+2​M​Tq¯\]\]\\displaystyle\\leq\\tfrac\{1\}\{4\(1\-\\gamma\)\}\[\(2\\bar\{\\rho\}\+\\gamma\\bar\{c\}\(L\_\{1\}\+L\_\{2\}\)\)\\nu\_\{T\}\(\\nicefrac\{\{\\delta\_\{all\}\}\}\{\{6\}\}\)\\sqrt\{1\+C\_\{N\}\}\[\\tfrac\{M\}\{\\sqrt\{\\lambda\_\{2\}\}\}\+2\\sqrt\{\\tfrac\{MT\}\{\\underline\{q\}\}\}\]\]\+γ1−γ​\[CJ\+CJ~,T\+1,δa​l​l/3\+\(L1\+L2\)​αTθ​\(δa​l​l/6\)​\[C1\+C2\+C3\]\]\\displaystyle\\qquad\+\\tfrac\{\\gamma\}\{1\-\\gamma\}\[C\_\{J\}\+C\_\{\\tilde\{J\},T\+1,\\nicefrac\{\{\\delta\_\{all\}\}\}\{\{3\}\}\}\+\(L\_\{1\}\+L\_\{2\}\)\\alpha\_\{T\}^\{\\theta\}\(\\nicefrac\{\{\\delta\_\{all\}\}\}\{\{6\}\}\)\[C\_\{1\}\+C\_\{2\}\+C\_\{3\}\]\]\+γ1−γ​\[2​\(CJ~,T\+1,δa​l​l/3\+CJ\)​\(2​T​log⁡\(3/δa​l​l\)\+Kd\+KN\)\]\\displaystyle\\qquad\+\\tfrac\{\\gamma\}\{1\-\\gamma\}\[2\(C\_\{\\tilde\{J\},T\+1,\\nicefrac\{\{\\delta\_\{all\}\}\}\{\{3\}\}\}\+C\_\{J\}\)\(\\sqrt\{2T\\log\(\\nicefrac\{\{3\}\}\{\{\\delta\_\{all\}\}\}\)\}\+K\_\{d\}\+K\_\{N\}\)\]\(10\)withL1=β\+ρ¯4​\(1−γ​a¯\)​\(1\+2​γ1−γ\)L\_\{1\}=\\frac\{\\beta\+\\bar\{\\rho\}\}\{4\(1\-\\gamma\\bar\{a\}\)\}\\left\(1\+\\frac\{2\\gamma\}\{1\-\\gamma\}\\right\),L2=11−γ​a¯​\[\(β\+ρ¯\)/4\+\(e2​μ¯/4\)​\(ρ¯\+γ​L1​c¯\)​α¯μ​\(δa​l​l/6\)\+5​γ​L1​αTθ​\(δa​l​l/6\)4​λ1\+\(γ/2\)​CJ~,T,δa​l​l/3\]L\_\{2\}=\\frac\{1\}\{1\-\\gamma\\bar\{a\}\}\[\\nicefrac\{\{\(\\beta\+\\bar\{\\rho\}\)\}\}\{\{4\}\}\+\(\\nicefrac\{\{e^\{2\\bar\{\\mu\}\}\}\}\{\{4\}\}\)\(\\bar\{\\rho\}\+\\gamma L\_\{1\}\\bar\{c\}\)\\overline\{\\alpha\}^\{\\mu\}\(\\nicefrac\{\{\\delta\_\{all\}\}\}\{\{6\}\}\)\+\\frac\{5\\gamma L\_\{1\}\\alpha^\{\\theta\}\_\{T\}\(\\nicefrac\{\{\\delta\_\{all\}\}\}\{\{6\}\}\)\}\{4\\sqrt\{\\lambda\_\{1\}\}\}\+\(\\nicefrac\{\{\\gamma\}\}\{\{2\}\}\)C\_\{\\tilde\{J\},T,\\nicefrac\{\{\\delta\_\{all\}\}\}\{\{3\}\}\}\],νT​\(δ\)=e3​μ¯​\[λ22\+2​μ¯λ2\+2λ2​log⁡\(2​M​λ2\+T/4δ​λ2\)\+e−μ¯​λ2​μ¯\]\\nu\_\{T\}\(\\delta\)=e^\{3\\bar\{\\mu\}\}\[\\frac\{\\sqrt\{\\lambda\_\{2\}\}\}\{2\}\+\\frac\{2\\bar\{\\mu\}\}\{\\sqrt\{\\lambda\_\{2\}\}\}\+\\frac\{2\}\{\\sqrt\{\\lambda\_\{2\}\}\}\\log\(\\frac\{2M\\sqrt\{\\lambda\_\{2\}\+\\nicefrac\{\{T\}\}\{\{4\}\}\}\}\{\\delta\\sqrt\{\\lambda\_\{2\}\}\}\)\+e^\{\-\\bar\{\\mu\}\}\\sqrt\{\\lambda\_\{2\}\}\\bar\{\\mu\}\],C1=νT​\(δa​l​l/6\)​1\+CN4​λ1​\[M/λ2\+2​M​T/q¯\]C\_\{1\}=\\frac\{\\nu\_\{T\}\(\\nicefrac\{\{\\delta\_\{all\}\}\}\{\{6\}\}\)\\sqrt\{1\+C\_\{N\}\}\}\{4\\sqrt\{\\lambda\_\{1\}\}\}\[\\nicefrac\{\{M\}\}\{\{\\sqrt\{\\lambda\_\{2\}\}\}\}\+2\\sqrt\{\\nicefrac\{\{MT\}\}\{\{\\underline\{q\}\}\}\}\],C2=2​log⁡\(3δa​l​l\)​Tλ1C\_\{2\}=\\sqrt\{2\\log\(\\frac\{3\}\{\\delta\_\{all\}\}\)\\frac\{T\}\{\\lambda\_\{1\}\}\},C3=T​\(1\+Cd\)​\(2​M\+1\)​log⁡\(1\+T​\(Cx2\+2\)\(2​M\+1\)​λ1\)​\[2\+Cx2\+2log⁡\(2\)​λ1\]C\_\{3\}=\\sqrt\{T\(1\+C\_\{d\}\)\(2M\+1\)\\log\(1\+\\frac\{T\(C\_\{x\}^\{2\}\+2\)\}\{\(2M\+1\)\\lambda\_\{1\}\}\)\[2\+\\frac\{C\_\{x\}^\{2\}\+2\}\{\\log\(2\)\\lambda\_\{1\}\}\]\},Kd=\(2​M\+1\)log⁡\(1\+Cd\)​log⁡\(1\+T​\(Cx2\+2\)\(2​M\+1\)​λ1\)K\_\{d\}=\\frac\{\(2M\+1\)\}\{\\log\(1\+C\_\{d\}\)\}\\log\(1\+\\frac\{T\(C\_\{x\}^\{2\}\+2\)\}\{\(2M\+1\)\\lambda\_\{1\}\}\),KN=M​\(1\+log⁡\(T\)log⁡\(1\+CN\)\)K\_\{N\}=M\(1\+\\frac\{\\log\(T\)\}\{\\log\(1\+C\_\{N\}\)\}\),q¯=σ′​\(Cx\+μ¯\)\\underline\{q\}=\\sigma^\{\\prime\}\(C\_\{x\}\+\\bar\{\\mu\}\), andα¯μ​\(δ/2\)=e3​μ¯λ2​\[λ22\+2λ2​\(μ¯\+log⁡\(4​Mδ​λ2\)\)\]\+2​e3​μ¯−1λ2\+e2​μ¯​μ¯\\overline\{\\alpha\}^\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)=\\frac\{e^\{3\\bar\{\\mu\}\}\}\{\\sqrt\{\\lambda\_\{2\}\}\}\[\\frac\{\\sqrt\{\\lambda\_\{2\}\}\}\{2\}\+\\frac\{2\}\{\\sqrt\{\\lambda\_\{2\}\}\}\(\\bar\{\\mu\}\+\\log\(\\frac\{4M\}\{\\delta\\sqrt\{\\lambda\_\{2\}\}\}\)\)\]\+\\frac\{2e^\{3\\bar\{\\mu\}\-1\}\}\{\\sqrt\{\\lambda\_\{2\}\}\}\+e^\{2\\bar\{\\mu\}\}\\bar\{\\mu\}\.

###### Corollary 5\.6

Under the conditions of Theorem[5\.5](https://arxiv.org/html/2605.24261#S5.Thmtheorem5)andδa​l​l=1/T\\delta\_\{all\}=\\nicefrac\{\{1\}\}\{\{T\}\},𝐄​\[RT\]=𝒪~​\(M3/2​T\(1−γ\)3\)\\mathbf\{E\}\[R\_\{T\}\]=\\tilde\{\\mathcal\{O\}\}\\left\(\\frac\{M^\{\\nicefrac\{\{3\}\}\{\{2\}\}\}\\sqrt\{T\}\}\{\(1\-\\gamma\)^\{3\}\}\\right\)\.

Proofs of these results can be found in Appendix[13\.4](https://arxiv.org/html/2605.24261#S13.SS4)\. The proof of Theorem[5\.5](https://arxiv.org/html/2605.24261#S5.Thmtheorem5)relies on the Lipschitz continuity of the rewards, bonus, and the value functionsJ∗J^\{\*\}andJ~t,δ\\tilde\{J\}\_\{t,\\delta\}\. We add and subtract cross terms to generate pairs of quantities \(e\.g\., differences in the value\-to\-go\) under differentθ\\thetaandμ\\muparameterizations\. Similar to the approach used to demonstrate valid optimism for the surrogate system, we use Lipschitz continuity to express differences as estimation errors in eitherμ\\muorθ\\theta\. Propositions[5\.1](https://arxiv.org/html/2605.24261#S5.Thmtheorem1)and[5\.2](https://arxiv.org/html/2605.24261#S5.Thmtheorem2)then provide high probability bounds on these estimation errors\. We bound the remaining terms either by standard concentration arguments for martingale difference sequences or using the epoch\-based construction of the algorithm to show that the number of bounded terms grows sublinearly\. We then union bound these events to obtain the final result\.

ThusUCB\-BOLDachieves the𝒪~​\(T\)\\tilde\{\\mathcal\{O\}\}\(\\sqrt\{T\}\)regret rate typical for related settings, such as discounted tabular MDPs\(Heet al\.[2021](https://arxiv.org/html/2605.24261#bib.bib33)\)and linear mixture MDPs\(Zhouet al\.[2021](https://arxiv.org/html/2605.24261#bib.bib73)\), linear and logistic bandits\(Abbasi\-Yadkoriet al\.[2011](https://arxiv.org/html/2605.24261#bib.bib1), Fauryet al\.[2020](https://arxiv.org/html/2605.24261#bib.bib25)\)and the average reward linear quadratic regulator\(Simchowitz and Foster[2020](https://arxiv.org/html/2605.24261#bib.bib65)\)\. The leading order term in our regret bound comes from the\(L1\+L2\)​αTθ​\(δa​l​l/6\)​\[C1\+C2\+C3\]\(L\_\{1\}\+L\_\{2\}\)\\alpha\_\{T\}^\{\\theta\}\(\\nicefrac\{\{\\delta\_\{all\}\}\}\{\{6\}\}\)\\left\[C\_\{1\}\+C\_\{2\}\+C\_\{3\}\\right\]quantity; this term includes theγ1−γ\\frac\{\\gamma\}\{1\-\\gamma\}term from the telescoping decomposition, the𝒪~​\(M\(1−γ\)2\)\\tilde\{\\mathcal\{O\}\}\\left\(\\frac\{\\sqrt\{M\}\}\{\(1\-\\gamma\)^\{2\}\}\\right\)L2L\_\{2\}\-Lipschitz continuity ofJ~t,δ\\tilde\{J\}\_\{t,\\delta\}, the𝒪~​\(M\)\\tilde\{\\mathcal\{O\}\}\(\\sqrt\{M\}\)confidence set widthαTθ\\alpha^\{\\theta\}\_\{T\}, and the𝒪~​\(M​T\)\\tilde\{\\mathcal\{O\}\}\(\\sqrt\{MT\}\)sum ofC1C\_\{1\}andC3C\_\{3\}\.C1C\_\{1\}captures joint interactions betweenμ\\muuncertainty andθ\\theta\-based exploration whileC3C\_\{3\}captures typical dynamics exploration costs for learningθ\\thetain linear bandit or linear MDP settings\. A regret lower bound in our model class remains open\. For context,Zhouet al\.\([2021](https://arxiv.org/html/2605.24261#bib.bib73)\)established theΩ​\(d​T\(1−γ\)1\.5\)\\Omega\\left\(\\frac\{d\\sqrt\{T\}\}\{\(1\-\\gamma\)^\{1\.5\}\}\\right\)rate for the related linear mixture MDP setting, which assumes a transition kernel that is linear in add\-dimensional feature set\. Our model’s noise and state\-dependent adherence do not fit this modeling class, but their bound suggests that tighter analysis may be possible in our setting and serves as a motivating reference point for future work\.

## 6Case study and computational experiments

We evaluateUCB\-BOLDagainst benchmark algorithms on a synthetic patient cohort, tasking the algorithms with recommending daily treatments from a small \(M=3M=3\) treatment set\. Our experiments are derived from an MRT where digital prompts were used to promote non\-sedentary behavior in the workplace\. We use Bayesian inference to fit a hierarchical model \(HM\) to the MRT data and generate synthetic patients\. We evaluate algorithm performance on simulated patient trajectories, including extensive ablation studies to highlight algorithm performance across parameter regimes\. We find thatUCB\-BOLDconsistently outperforms the benchmarks across ablation conditions\.

### 6\.1Physical activity case study

We rely on MRT study data fromCarpenteret al\.\([2025](https://arxiv.org/html/2605.24261#bib.bib74)\)\. This study followed 16 participants for up to 4 work weeks\. Within each hour of the participant’s 8\-hour workday, participants were randomly assigned a message type: \(1\) no message, \(2\) a prompt to stand up, and \(3\) a prompt to move \(i\.e\., to take a brief walk\)\. The study set internal thresholds for standing and moving to convert measured physical activity to binary adherence outcomes for each prompt\.

### 6\.2Synthetic cohort generation

This MRT largely maps onto the model described in Section[3](https://arxiv.org/html/2605.24261#S3)\. One exception is the absence of a self\-reported engagement state; this is treated as a latent quantity during Bayesian HM fitting\. The other key difference is the MRT’s hourly, not daily, timescale\. Modeling both within\- and between\-day effects was impractical for a dataset of this size, so we chose to treat each day as a standalone trajectory\. Thus, we consider the following patient\-level model:xt\+1i,j=ai​xti,j\+bi⊤​uti,j\+ci⊤​dti,j\+wtjx\_\{t\+1\}^\{i,j\}=a^\{i\}x\_\{t\}^\{i,j\}\+\{b^\{i\}\}^\{\\top\}u^\{i,j\}\_\{t\}\+\{c^\{i\}\}^\{\\top\}d^\{i,j\}\_\{t\}\+w^\{j\}\_\{t\},x1i,j∼w0jx^\{i,j\}\_\{1\}\\sim w^\{j\}\_\{0\}, whereiiindexes the participant,jjindexes the trajectory for that participant, andttindexes the time step within the day\. Adherence decisionsdti,j\|xti,j,uti,jd^\{i,j\}\_\{t\}\|x^\{i,j\}\_\{t\},u^\{i,j\}\_\{t\}follow the model in Section[3](https://arxiv.org/html/2605.24261#S3)\. The HM models patient parameters as samples from a population distribution, i\.e\., lettingmmindex the non\-null action type,bmi∼𝒩​\(b¯m,σbm2\)b^\{i\}\_\{m\}\\sim\\mathcal\{N\}\(\\bar\{b\}\_\{m\},\\sigma\_\{b\_\{m\}\}^\{2\}\)\. The same approach is used foraa,cc, andμ\\mu, with a logit transformation to ensurea∈\(0,1\)a\\in\(0,1\)\. We emphasize that we do not use the MRT dataset to validate our particular model structure against alternatives; rather, we use the dataset as a principled way to generate a plausible patient parameter distribution for this model class\.

We use`Stan`\(Carpenteret al\.[2017](https://arxiv.org/html/2605.24261#bib.bib18)\)to perform Markov chain Monte Carlo Bayesian inference on the HM and MRT dataset described above\. We used weakly informative priors, resulting in a roughly uniform prior for the sampling distribution ofaaand zero\-mean Gaussian priors for the sampling distributions ofbb,cc, andμ\\mu\. Figure[6\.2](https://arxiv.org/html/2605.24261#S6.SS2)shows the sampling distribution for the synthetic patients under the weakly informative priors and under the posterior of the MRT\-fitted model\.

\\FIGURE![[Uncaptioned image]](https://arxiv.org/html/2605.24261v1/x1.png)

Synthetic patient population sampling distributions Plots display kernel density estimates, medians, and80%80\\%and99%99\\%credible intervals for each model parameter\.

While there is still meaningful uncertainty in the posterior, its shape agrees with intuition\. The uniform prior overaabecomes right skewed, suggesting low\-to\-moderate state persistence in this context\. The zero\-mean Gaussian priors forbbtighten and shift left \(i\.e\., indicating burden\), with the more demanding move request being more negative than the stand request\. The zero\-mean Gaussian priors forccshrink and shift right, suggesting adherence can increase engagement capacity in this context \(with standing having a stronger effect than moving\)\. Bothμ\\muquantities tighten and shift left, capturing that baseline adherence in this context is less than50%50\\%for both prompt types\.

### 6\.3Experiment description

All experiments simulated patient trajectories with once\-daily treatment recommendations\. We performed2525replications with different random seeds for each patient\. Whilexxwas a latent quantity for Bayesian inference using the MRT data, here it is fully\-observed, matching Section[3](https://arxiv.org/html/2605.24261#S3)\. Using the joint parameter posterior described previously, we sampled100100synthetic patients for evaluation\. The sampling process fixed most relevant patient parameters, i\.e\.,aia^\{i\},b1ib^\{i\}\_\{1\},b2ib^\{i\}\_\{2\},c1ic^\{i\}\_\{1\},c2ic^\{i\}\_\{2\},μ1i\\mu^\{i\}\_\{1\}, andμ2i\\mu^\{i\}\_\{2\}\. The process noise was zero\-mean truncated Gaussian noise with originalσw2=1\\sigma\_\{w\}^\{2\}=1, truncated usingw¯=2\.5\\bar\{w\}=2\.5\. Given the importance of reducing dropout in DT interventions, we tested the impact of including an additional, motivating action\. As a result, the experiments had33non\-null actions \(two from the MRT plus the motivating action\) and the null action\. We setρ3\\rho\_\{3\}andb3b\_\{3\}for the motivating action to0, meaning it carries no treatment reward but also no cost or benefit associated with recommendation\. We also fixedμ3\\mu\_\{3\}for this action to0, implying50%50\\%adherence atx=0x=0\. Ablation studies explored the effect of the remaining free parameters, namely the strength of the motivating action \(c3c\_\{3\}\), the relative scaling of rewards \(ρ1\\rho\_\{1\}vs\.ρ2\\rho\_\{2\}\), the trajectory length \(TT\), the discount factor \(γ\\gamma\), and the state\-based penalty \(viaβ\\betaandβ0\\beta\_\{0\}\)\.

Our experiments comparedUCB\-BOLDto several benchmark algorithms\.Optimalis the optimal policy, obtained by VI using the true system; its value function serves as the reference point for regret calculations\.Fixed1andFixed2are fixed action policies for the two treatment actions\.Randomselects actions uniformly at random from𝒰\\mathcal\{U\}\. We also included epsilon\-greedy Q\-learning algorithms, using linear function approximation \(LFA\-Q\) and tile\-coding \(TC\-Q\), given as Algorithms[2](https://arxiv.org/html/2605.24261#alg2)and[3](https://arxiv.org/html/2605.24261#alg3)in Appendix[9](https://arxiv.org/html/2605.24261#S9), to serve as model\-free baselines\. As a myopic benchmark, we includeGLM\-Bandit, given as Algorithm[4](https://arxiv.org/html/2605.24261#alg4)in Appendix[9](https://arxiv.org/html/2605.24261#S9), which estimates only the shift parameters \(i\.e\., Algorithm[1](https://arxiv.org/html/2605.24261#alg1)with no planning\)\. We performed grid\-search hyperparameter tuning to ensure the Q\-learners were configured to be fair model\-free baselines \(see Appendix[9](https://arxiv.org/html/2605.24261#S9)\)\.

#### 6\.3\.1Experiment 1: Reward scaling and motivation strength

This experiment fixedγ=0\.8\\gamma=0\.8,T=730T=730days, and explored a4×44\\times 4ablation grid, studying the impact of varied reward scaling between treatments and different values ofcc\(i\.e\., the strength\) for the motivating action\. For simplicity, we fixed the reward associated with the first treatment action toρ1=1\\rho\_\{1\}=1and varyρ2=\[0\.5,1\.0,1\.5,2\.0\]\\rho\_\{2\}=\[0\.5,1\.0,1\.5,2\.0\]\. For thecc\-dimension, we vary a patient\-scaled value ofc3i=\[0,1,2,3\]⋅\(1−ai\)c^\{i\}\_\{3\}=\[0,1,2,3\]\\cdot\(1\-a^\{i\}\)\. We chose this structure as it normalizesc3c\_\{3\}to have a more similar impulse response across patients\. For simplicity, this experiment assumesβ=0\\beta=0such that the state\-based penalty is absent; we would expect planning\-based algorithms to improve in the presence of a state\-based penalty \(compared to myopic approaches\), so removing the penalty represents a floor for their performance\.

#### 6\.3\.2Experiment 2: Shortened trajectory

This experiment tests the robustness of findings with respect to trajectory length, testing Experiment 1’s conditions for shorter,180180\-day trajectories\.

#### 6\.3\.3Experiment 3: Effect of discount factor

The discount factor is ultimately a clinical decision, tied to the particular condition of interest\. This experiment shows how regret performance varies under different discounting assumptions\. We tested a730730\-day horizon, fixing a plausible setting on the interior of the primary ablation grid: reward scaling of1\.51\.5and a patient\-scaledccvalue of22\. For this configuration, we tested discount factorsγ=\[0\.0,0\.5,0\.8,0\.9,0\.95,0\.98\]\\gamma=\[0\.0,0\.5,0\.8,0\.9,0\.95,0\.98\], representing effective discounting horizons of11,22,55,1010,2020, and5050days\.

#### 6\.3\.4Experiment 4: Effect of the state\-based penalty

DT\-targeted conditions may carry meaningfully different disengagement or dropout costs; our model allows providers to encode this motivation via the state penalty\. This experiment evaluated algorithm performance across a range of penalty types\. Similar to the prior experiment, we used a730730\-day horizon, reward scaling of1\.51\.5, and patient scaledccvalue of22\. With these conditions fixed, we tested a grid of penalty magnitudesβ=\[0,1,2,3\]\\beta=\[0,1,2,3\]and penalty shift parametersβ0=\[−2,−4\]\\beta\_\{0\}=\[\-2,\-4\]\(i\.e\., setting the midpoint of the penalty sigmoid roughly22or44standard deviations of process noise below the baseline state value\)\.

### 6\.4Results

To place experiments with different patients \(and thus different optimal value functions\) on a common scale, we normalized cumulative regret by the expected regret of theRandomalgorithm for the same patient and trajectory length\. We averaged replications for each patient\-algorithm\-ablation point combination, so plotted variation reflects performance differences across the population\.

#### 6\.4\.1Experiment 1: Reward scaling and motivation strength

Figure[6\.4\.1](https://arxiv.org/html/2605.24261#S6.SS4.SSS1)shows box plots for the normalized regret of the algorithms across the simulated patient population\.

\\FIGURE![[Uncaptioned image]](https://arxiv.org/html/2605.24261v1/x2.png)

Cumulative normalized regret after730730days over the population for the primary ablation grid The scaling of the reward for the second treatment compared to the first increases from0\.50\.5to2\.02\.0from left\-to\-right and the patient\-scaled effect of the motivating action increases from0\.00\.0to3\.03\.0from bottom\-to\-top\.

The impact of varied reward scaling is most visible in the performance reversal of the fixed action policies, as it does not significantly impact the learning algorithms\. The impact of the motivation strength ablation is most visible in the worsening tail performance of the fixed policies and the myopicGLM\-Bandit\. This makes sense as the presence of a stronger motivating action can increase the optimal value function by managing the engagement state \(i\.e\., selecting the motivating action, even though it carries no reward, to increase the engagement state and thus the likelihood of future adherence\)\.OptimalandRandomare included for reference, achieving normalized regret of0and roughly11, respectively\. Overall,UCB\-BOLDshows near\-uniform best performance among the algorithms across the grid, both in terms of median regret and regret variation across the population\. We see exceptions in the most extreme cases of the ablation grid\. For instance, the bottom right subplot shows a case where theFixed2policy and the myopicGLM\-Banditshow better median performance; here the reward scaling and lack of a strong motivating action make it optimal, for many patients, to recommend the second treatment at most \(or all\) engagement states\. Even in such cases,UCB\-BOLDshows much lower variation in performance across the population, indicating that while simpler policy structures may serve some patients well, they perform substantially worse for the remaining population \(i\.e\., yielding more extreme worst\-case outcomes\)\. The model\-free benchmarks perform similarly to one another, typically ranked betweenUCB\-BOLDandGLM\-Bandit, depending on the ablation point\.TC\-QandLFA\-Qare capable of planning with the engagement state, but suffer from sample complexity limitations compared to model\-based approaches in this setting\.

Figure[6\.4\.1](https://arxiv.org/html/2605.24261#S6.SS4.SSS1)provides empirical cumulative distribution functions \(ECDFs\), showing the proportion of the population for which the algorithms achieve a given cumulative normalized regret\. This type of plot enables a richer assessment of algorithm behavior over the population\.OptimalandRandomappear as vertical lines, incurring0and roughly11normalized regret for every patient\. We clearly see the cases where fixed or myopic policies serve portions \(typically30−40%30\-40\\%\) of the population well \(i\.e\., the curves vertically track on or close to that ofOptimalbefore shifting right\)\. We found that the upper tail contained individuals whereRandomsets better baseline performance \(i\.e\., because the optimal policy tends to involve the complete action set, not a trivial collapse to a single action\)\.UCB\-BOLDseparates itself in this part of the population, suggesting that algorithms that plan around engagement in a sample\-efficient way show significantly improved performance for a large portion of the population, even if such planning is not a critical concern for every patient\.

\\FIGURE![[Uncaptioned image]](https://arxiv.org/html/2605.24261v1/x3.png)

ECDFs for regret across the patient population for the primary ablation grid\. The scaling of the reward for the second treatment compared to the first increases from0\.50\.5to2\.02\.0from left\-to\-right and the patient\-scaled effect of the motivating action increases from0\.00\.0to3\.03\.0from bottom\-to\-top\.

Table[6\.4\.1](https://arxiv.org/html/2605.24261#S6.SS4.SSS1)shows the empirical conditional value\-at\-risk \(CVaR\) regret for various tail widths, averaged over the ablation grid; CVaR effectively measures the expectation of outcomes in the upper1−α1\-\\alphatail\.UCB\-BOLDoutperforms the next best algorithm by a factor of22\-3×3\\times, depending on the tail width, with the absolute gap between algorithms widening as the tail width decreases\.

\\TABLE

Experiment 1 cumulative normalized regret CVaR\. CVaR regret for varying tail widths \(1\-α\\alpha\), averaged over the ablation grid\. Values given as median \(Q1, Q3\)\. Lowest values bolded\.

#### 6\.4\.2Experiment 2: Shortened trajectory

\\FIGURE![[Uncaptioned image]](https://arxiv.org/html/2605.24261v1/x4.png)

Cumulative normalized regret after180180days over the population for the primary ablation grid The scaling of the reward for the second treatment compared to the first increases from0\.50\.5to2\.02\.0from left\-to\-right and the patient\-scaled effect of the motivating action increases from0\.00\.0to3\.03\.0from bottom\-to\-top\.

Figure[6\.4\.2](https://arxiv.org/html/2605.24261#S6.SS4.SSS2)shows box plots for the normalized regret performance of the algorithms for180180day trajectories, showing essentially the same performance trends as in the730730\-day trajectories of Figure[6\.4\.1](https://arxiv.org/html/2605.24261#S6.SS4.SSS1)\.UCB\-BOLDis a consistently strong performer across ablation conditions, withGLM\-Banditshowing competitive median performance for some ablation points but substantially larger variance over the population\. The Q\-learning algorithms perform worse, relative to their competition, in this shorter horizon setting, owing to the greater sample complexity demands of model\-free approaches\.

#### 6\.4\.3Experiment 3: Effect of discount factor

\\FIGURE![[Uncaptioned image]](https://arxiv.org/html/2605.24261v1/x5.png)

Cumulative normalized regret after730730days over the population for different discount valuesγ\\gammaDiscount factor increases from left\-to\-right\. Effective horizons,\(1/\(1−γ\)\)\\left\(\\nicefrac\{\{1\}\}\{\{\(1\-\\gamma\)\}\}\\right\), range from11day to5050days\. This sweep fixes the reward scaling value to1\.51\.5and the patient\-scaledccvalue for the motivating action to2\.02\.0\.

Figure[6\.4\.3](https://arxiv.org/html/2605.24261#S6.SS4.SSS3)shows box plots for730730\-day normalized regret performance with varying discount factors\. Myopic approaches such asGLM\-Banditperform well for lower discount factors, but as the discount factor increases,UCB\-BOLDseparates as the best performer\. In practice,γ\\gammais a clinical choice; these results suggest that planning algorithms are critical for settings focused on long\-term outcomes\. We see a separation betweenUCB\-BOLDand the Q\-learners, owing to a poorer sample complexity scaling of these methods inγ\\gammacompared to model\-based approaches\.

#### 6\.4\.4Experiment 4: Effect of state\-based penalty

\\FIGURE![[Uncaptioned image]](https://arxiv.org/html/2605.24261v1/x6.png)

Cumulative normalized regret after730730days over the population for various state penaltiesβ\\betavalues, denoting penalty strength, increase from0to33moving left\-to\-right\. The location parameterβ0\\beta\_\{0\}is−4\-4in the bottom row and−2\-2in the top row\. A more negative value means the sigmoidal penalty does not ‘turn on’ until the state goes more strongly negative\. This sweep fixes the reward scaling value to1\.51\.5and the patient\-scaledccvalue for the motivating action to2\.02\.0\.

Figure[6\.4\.4](https://arxiv.org/html/2605.24261#S6.SS4.SSS4)explores different state\-based penalties, varying both penalty function magnitude \(β\\beta\) and location \(β0\\beta\_\{0\}\)\. This penalty, designed to reflect a provider’s motivation to keep a patient from disengaging, increases the importance of engagement planning\. We observe thatUCB\-BOLDperforms uniformly well across this ablation study while myopic approaches likeGLM\-Banditsuffer with a stronger penalty, particularly in the upper tail of the population\. The performance of the model\-free Q\-learners degrades moderately with higher penalties, likely due to the fact that the fixed set of hyperparameters used across the ablation studies are not optimal in these separate system configurations and that they are not designed to exploit a \(partially\) known reward structure\. However, the purpose of this ablation is not to highlight differences between model\-free and model\-based approaches, but rather to highlight scenarios where engagement planning is especially important versus where myopic policies suffice\.

## 7Managerial implications and limitations

Our work suggests the following key implications:

1. 1\.The benefits of engagement planning \(i\.e\., selecting treatments based on recommendation and adherence effects on the patient’s future engagement state\) may differ meaningfully across a patient population\.Our computational experiments relied on a synthetic patient cohort, calibrated using MRT data, allowing examination of algorithm performance across a range of plausible patient types\. For some patient types \(up to roughly50%50\\%of the population under some ablation conditions\), myopic or heuristic policies sufficed due to a single action dominating the optimal policy for those patients\. Among the rest of the population, there were clear benefits to strategically planning around the patient’s time\-varying engagement dynamics \(i\.e\., higher state persistence or larger recommendation and adherence effects meant the optimal policy was not dominated by a single action\)\.UCB\-BOLDshowed drastically better performance for this sub\-cohort, showing 2\-3 times lower CVaR regret than the next\-best benchmark \(Table[6\.4\.1](https://arxiv.org/html/2605.24261#S6.SS4.SSS1)\)\. These trends were robust to trajectory length \(Figure[6\.4\.2](https://arxiv.org/html/2605.24261#S6.SS4.SSS2)\) and were even stronger in ablation settings that favor planning, for instance higher discount factors \(Figure[6\.4\.3](https://arxiv.org/html/2605.24261#S6.SS4.SSS3)\), a stronger motivating action \(Figure[6\.4\.1](https://arxiv.org/html/2605.24261#S6.SS4.SSS1)\), and a stronger state\-based penalty \(Figure[6\.4\.4](https://arxiv.org/html/2605.24261#S6.SS4.SSS4)\)\. Explicitly accounting for engagement dynamics may be an important way for DT providers to ensure quality treatment recommendation across a complete patient population\.
2. 2\.Dynamical modeling can bridge descriptive behavioral models with long\-run planning for mHealth interventions\.Personalized mHealth interventions operate in a sample\-scarce regime and incorporating explicit dynamics models allows decision\-makers to exploit problem structure and plan for concerns like burden and fatigue in ways that myopic approaches cannot\. However, doing so can introduce model misspecification concerns when models do not approximate reality well\. Our case study suggested that the benefits of engagement state planning may be meaningful for some, but not all, patients; decision\-makers will need to explicitly weigh the tradeoffs of imposing planning\-oriented structure in their intervention context\. Our work provides a blueprint for demonstrating relevant theoretical guarantees under the posited dynamics\. As the psychology community develops better\-grounded dynamical models for behavior\(Brigantiet al\.[2024](https://arxiv.org/html/2605.24261#bib.bib17), Perskiet al\.[2025](https://arxiv.org/html/2605.24261#bib.bib54)\), this same approach can be used to match methodological development to current behavioral science\.

### 7\.1Limitations and extensions

We focus on a purposefully parsimonious model class; there are many extensions not addressed here, for instance, hierarchical models, time\-varying dynamics, multi\-dimensional or latent states, or integration of context into the adherence model\. Further, treating engagement as synonymous with adherence behavior may be incomplete \(i\.e\., a patient can invest different amounts of energy into completing treatment\)\(Nahum\-Shaniet al\.[2022](https://arxiv.org/html/2605.24261#bib.bib48)\)\. Models capturing these distinctions offer rich theoretical and practical directions for future work\. Next, our MRT data have an hourly timescale, which we mapped to a daily timescale for our computational experiments\. It is not clear whether interventions operating at a daily timescale will involve more or less persistent dynamics; these effects may also vary significantly by intervention setting\. Additional exploratory and descriptive work is needed to refine the functional form of model\-based approaches, but existing work has shown that such approximations perform well empirically\(El Mistiriet al\.[2025](https://arxiv.org/html/2605.24261#bib.bib23), Pulicket al\.[2025](https://arxiv.org/html/2605.24261#bib.bib57)\)\.

## 8Conclusion

In this paper, we developed a framework modeling how DT treatment decisions affect patient adherence over time based on a time\-varying engagement state\. Our model operates on a daily timescale and is aimed at long\-running interventions, where sustaining engagement is especially important for providers\. We chose a parsimonious linear dynamical system model to capture the phenomena of interest, namely separate effects for recommending a treatment and for patient adherence\. We showed finite\-time system identification guarantees for this model under a flexible policy class\. We proposed an optimism\-based algorithm,UCB\-BOLD, and proved that it achieves sublinear regret\. Last, we used MRT data to generate synthetic patients and demonstrate thatUCB\-BOLDoutperformed benchmarks over a wide range of system conditions\. Our work offers a model\-based approach for JITAIs to incorporate dynamical systems models from behavioral psychology, and our framework serves as a foundation for natural model extensions in future work\.

## References

- Improved Algorithms for Linear Stochastic Bandits\.InAdv\. Neural Inf\. Process\. Syst\.,Vol\.24\.Cited by:[Theorem 13\.1](https://arxiv.org/html/2605.24261#S13.Thmtheorem1.p1.19.19),[Lemma 13\.33](https://arxiv.org/html/2605.24261#S13.Thmtheorem33.p1.8.8),[Lemma 13\.36](https://arxiv.org/html/2605.24261#S13.Thmtheorem36.p1.5.5),[§2\.3](https://arxiv.org/html/2605.24261#S2.SS3.p1.1),[§5\.3\.1](https://arxiv.org/html/2605.24261#S5.SS3.SSS1.p2.4),[§5\.4](https://arxiv.org/html/2605.24261#S5.SS4.p3.18)\.
- M\. Abbott, W\. Dempsey, J\. Golbus, E\. Huch, A\. Moreno, and J\. Shi \(2024\)RoME: A Robust Mixed\-Effects Bandit Algorithm for Optimizing Mobile Health Interventions\.InAdv\. Neural Inf\. Process\. Syst\. 37,pp\. 128280–128329\.Cited by:[§2\.3](https://arxiv.org/html/2605.24261#S2.SS3.p1.1)\.
- O\. Alagoz, L\. M\. Maillart, A\. J\. Schaefer, and M\. S\. Roberts \(2004\)The Optimal Timing of Living\-Donor Liver Transplantation\.Management Sci\.50\(10\),pp\. 1420–1430\.Cited by:[§2\.1](https://arxiv.org/html/2605.24261#S2.SS1.p1.1)\.
- T\. W\. Anderson \(1955\)The integral of a symmetric unimodal function over a symmetric convex set and some probability inequalities\.Proc\. Amer\. Math\. Soc\.6\(2\),pp\. 170–176\.Cited by:[Proof 10\.4](https://arxiv.org/html/2605.24261#S10.Thmtheorem4.p1.5.5)\.
- A\. Aswani, P\. Kaminsky, Y\. Mintz, E\. Flowers, and Y\. Fukuoka \(2019\)Behavioral modeling in weight loss interventions\.Eur\. J\. Oper\. Res\.272\(3\),pp\. 1058–1072\.Cited by:[§2\.1](https://arxiv.org/html/2605.24261#S2.SS1.p1.1)\.
- P\. Auer, T\. Jaksch, and R\. Ortner \(2008\)Near\-optimal Regret Bounds for Reinforcement Learning\.InAdv\. Neural Inf\. Process\. Syst\.,Vol\.21\.Cited by:[§2\.3](https://arxiv.org/html/2605.24261#S2.SS3.p1.1)\.
- T\. Ayer, O\. Alagoz, and N\. K\. Stout \(2012\)OR Forum—A POMDP Approach to Personalize Mammography Screening Decisions\.Oper\. Res\.60\(5\),pp\. 1019–1034\.Cited by:[§2\.1](https://arxiv.org/html/2605.24261#S2.SS1.p1.1)\.
- M\. G\. Azar, I\. Osband, and R\. Munos \(2017\)Minimax Regret Bounds for Reinforcement Learning\.InProc\. 34th Int\. Conf\. Mach\. Learn\.,pp\. 263–272\.Cited by:[§5\.3\.2](https://arxiv.org/html/2605.24261#S5.SS3.SSS2.p1.3)\.
- J\. Baek, J\. J\. Boutilier, V\. F\. Farias, J\. O\. Jónasson, and E\. Yoeli \(2025\)Policy Optimization for Personalized Interventions in Behavioral Health\.M&SOM27\(3\),pp\. 770–788\.Cited by:[§2\.3](https://arxiv.org/html/2605.24261#S2.SS3.p1.1)\.
- H\. Bastani and M\. Bayati \(2020\)Online Decision Making with High\-Dimensional Covariates\.Oper\. Res\.68\(1\),pp\. 276–294\.Cited by:[§2\.1](https://arxiv.org/html/2605.24261#S2.SS1.p1.1)\.
- A\. Baumel, F\. Muench, S\. Edan, and J\. M\. Kane \(2019\)Objective User Engagement With Mental Health Apps: Systematic Search and Panel\-Based Usage Analysis\.J Med Internet Res21\(9\),pp\. e14567\.Cited by:[§1\.1](https://arxiv.org/html/2605.24261#S1.SS1.p1.1)\.
- D\. Ben\-Zeev, R\. M\. Brian, G\. Jonathan, L\. Razzano, N\. Pashka, E\. Carpenter\-Song, R\. E\. Drake, and E\. A\. Scherer \(2018\)Mobile Health \(mHealth\) Versus Clinic\-Based Group Intervention for People With Serious Mental Illness: A Randomized Controlled Trial\.PS69\(9\),pp\. 978–985\.Cited by:[§1](https://arxiv.org/html/2605.24261#S1.p1.1)\.
- D\. P\. Bertsekas \(2012\)Dynamic programming and optimal control\.4th edition,Athena Scientific Optimization and Computation Series,Athena scientific,Nashua, NH\.External Links:LCCN 004\.015 1Cited by:[Proof 13\.7](https://arxiv.org/html/2605.24261#S13.Thmtheorem7.p1.11.11)\.
- D\. Bertsimas and N\. Kallus \(2020\)From Predictive to Prescriptive Analytics\.Management Sci\.66\(3\),pp\. 1025–1044\.Cited by:[§2\.1](https://arxiv.org/html/2605.24261#S2.SS1.p1.1)\.
- O\. Besbes, Y\. Gur, and A\. Zeevi \(2014\)Stochastic multi\-armed\-bandit problem with non\-stationary rewards\.Adv\. Neural Inf\. Process\. Syst\.27\.Cited by:[§2\.3](https://arxiv.org/html/2605.24261#S2.SS3.p1.1)\.
- S\. P\. Boyd and L\. Vandenberghe \(2004\)Convex optimization\.Cambridge university press\.Cited by:[Proof 10\.4](https://arxiv.org/html/2605.24261#S10.Thmtheorem4.p1.5.5)\.
- G\. Briganti, M\. Scutari, S\. Epskamp, D\. Borsboom, R\. H\. A\. Hoekstra, H\. F\. Golino, A\. P\. Christensen, Y\. Morvan, O\. V\. Ebrahimi, G\. Costantini, A\. Heeren, J\. D\. Ron, L\. F\. Bringmann, K\. Huth, J\. M\. B\. Haslbeck, A\. Isvoranu, M\. Marsman, T\. Blanken, A\. Gilbert, T\. R\. Henry, E\. I\. Fried, and R\. J\. McNally \(2024\)Network analysis: An overview for mental health research\.Int J Methods Psych Res33\(4\),pp\. e2034\.Cited by:[§1\.1](https://arxiv.org/html/2605.24261#S1.SS1.p2.1),[item 2](https://arxiv.org/html/2605.24261#S7.I1.i2.p1.1)\.
- B\. Carpenter, A\. Gelman, M\. D\. Hoffman, D\. Lee, B\. Goodrich, M\. Betancourt, M\. Brubaker, J\. Guo, P\. Li, and A\. Riddell \(2017\)*Stan*: A Probabilistic Programming Language\.J\. Stat\. Soft\.76\(1\)\.Cited by:[§6\.2](https://arxiv.org/html/2605.24261#S6.SS2.p2.4)\.
- S\.M\. Carpenter, S\.L\. Mullane, M\.J\.L\. Toledo, R\.G\.N\. Seltzer, and M\.P\. Buman \(2025\)Reducing sedentary time among office workers with sit\-stand workstations\. The Transition Pilot Microrandomized trial\.San Francisco California USA\.Cited by:[§6\.1](https://arxiv.org/html/2605.24261#S6.SS1.p1.1)\.
- S\. P\. Cross and M\. Alvarez\-Jimenez \(2024\)The digital cumulative complexity model: a framework for improving engagement in digital mental health interventions\.Front\. Psychiatry15,pp\. 1382726\.Cited by:[§1\.1](https://arxiv.org/html/2605.24261#S1.SS1.p2.1)\.
- S\. A\. De La Torre, M\. El Mistiri, K\. Tung, E\. Hekler, P\. Klasnja, M\. Pavel, D\. E\. Rivera, D\. Spruijt\-Metz, and B\. Marlin \(2025\)A dynamic Bayesian network approach to modeling engagement and walking behavior: insights from a yearlong micro\-randomized trial \(Heartsteps II\)\.Health Psychol\. Behav\. Med\.13\(1\),pp\. 2552479\.Cited by:[§2\.4](https://arxiv.org/html/2605.24261#S2.SS4.p1.1)\.
- S\. Dean, H\. Mania, N\. Matni, B\. Recht, and S\. Tu \(2020\)On the Sample Complexity of the Linear Quadratic Regulator\.Found Comput Math20\(4\),pp\. 633–679\.Cited by:[§2\.4](https://arxiv.org/html/2605.24261#S2.SS4.p2.1)\.
- J\. Ding and A\. Zhou \(2007\)Eigenvalues of rank\-one updated matrices with some applications\.Appl\. Math\. Lett\.20\(12\),pp\. 1223–1226\.Cited by:[Proof 13\.35](https://arxiv.org/html/2605.24261#S13.Thmtheorem35.p1.39.39)\.
- M\. El Mistiri, S\. De La Torre, K\. Tung, B\. M\. Marlin, M\. Pavel, P\. Klasnja, D\. Spruijt\-Metz, and D\. E\. Rivera \(2024\)System Identification of User Engagement in mHealth Behavioral Interventions\.IFAC\-PapersOnLine58\(15\),pp\. 508–513\.Cited by:[§2\.4](https://arxiv.org/html/2605.24261#S2.SS4.p1.1)\.
- M\. El Mistiri, D\. E\. Rivera, P\. Klasnja, J\. Park, and E\. Hekler \(2025\)Model Predictive Control in mHealth: A Decision Framework for Optimised Personalised Physical Activity Interventions\.Int J Control,pp\. 10\.1080/00207179\.2025\.2491109\.Cited by:[§2\.4](https://arxiv.org/html/2605.24261#S2.SS4.p1.1),[§7\.1](https://arxiv.org/html/2605.24261#S7.SS1.p1.1)\.
- L\. Faury, M\. Abeille, C\. Calauzenes, and O\. Fercoq \(2020\)Improved Optimistic Algorithms for Logistic Bandits\.InProc\. 37th Int\. Conf\. Mach\. Learn\.,pp\. 3052–3060\.Cited by:[Proof 13\.14](https://arxiv.org/html/2605.24261#S13.Thmtheorem14.p1.31.17),[Theorem 13\.5](https://arxiv.org/html/2605.24261#S13.Thmtheorem5.p1.17.17),[Proof 13\.6](https://arxiv.org/html/2605.24261#S13.Thmtheorem6.p1.31.31),[Proof 13\.6](https://arxiv.org/html/2605.24261#S13.Thmtheorem6.p1.58.27),[§2\.3](https://arxiv.org/html/2605.24261#S2.SS3.p1.1),[§5\.3\.1](https://arxiv.org/html/2605.24261#S5.SS3.SSS1.p3.1),[§5\.4](https://arxiv.org/html/2605.24261#S5.SS4.p3.18)\.
- S\. Filippi, O\. Cappe, A\. Garivier, and C\. Szepesvári \(2010\)Parametric Bandits: The Generalized Linear Case\.InAdv\. Neural Inf\. Process\. Syst\.,Vol\.23\.Cited by:[§4\.2](https://arxiv.org/html/2605.24261#S4.SS2.p1.2)\.
- A\. Garivier and E\. Moulines \(2008\)On upper\-confidence bound policies for non\-stationary bandit problems\.ArXiv Prepr\. ArXiv08053415\.External Links:0805\.3415Cited by:[§2\.3](https://arxiv.org/html/2605.24261#S2.SS3.p1.1)\.
- A\. V\. Gerbessiotis \(2025\)A survey of Chernoff and Hoeffding bounds\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.2506.15612)Cited by:[Proof 11\.8](https://arxiv.org/html/2605.24261#S11.Thmtheorem8.p1.16.16)\.
- D\. H\. Gustafson, F\. M\. McTavish, M\. Chih, A\. K\. Atwood, R\. A\. Johnson, M\. G\. Boyle, M\. S\. Levy, H\. Driscoll, S\. M\. Chisholm, L\. Dillenburg, A\. Isham, and D\. Shah \(2014\)A Smartphone Application to Support Recovery From Alcoholism: A Randomized Clinical Trial\.JAMA Psychiatry71\(5\),pp\. 566\.Cited by:[§1](https://arxiv.org/html/2605.24261#S1.p1.1)\.
- J\. He, D\. Zhou, and Q\. Gu \(2021\)Nearly Minimax Optimal Reinforcement Learning for Discounted MDPs\.InAdv\. Neural Inf\. Process\. Syst\.,Vol\.34,pp\. 22288–22300\.Cited by:[§5\.4](https://arxiv.org/html/2605.24261#S5.SS4.p1.1),[§5\.4](https://arxiv.org/html/2605.24261#S5.SS4.p3.18)\.
- Q\. He and Y\. Mintz \(2023\)Model Based Reinforcement Learning for Personalized Heparin Dosing\.arXiv\.External Links:2304\.10000,[Document](https://dx.doi.org/10.48550/arXiv.2304.10000)Cited by:[§2\.4](https://arxiv.org/html/2605.24261#S2.SS4.p1.1)\.
- Q\. He and Y\. Mintz \(2024\)Non\-stationary Bandits with Habituation and Recovery Dynamics and Knapsack Constraints\.In2024 IEEE 63rd Conf\. Decis\. Control CDC,Milan, Italy,pp\. 1167–1172\.Cited by:[§2\.3](https://arxiv.org/html/2605.24261#S2.SS3.p1.1)\.
- E\. B\. Hekler, S\. Michie, M\. Pavel, D\. E\. Rivera, L\. M\. Collins, H\. B\. Jimison, C\. Garnett, S\. Parral, and D\. Spruijt\-Metz \(2016\)Advancing Models and Theories for Digital Behavior Change Interventions\.Am\. J\. Prev\. Med\.51\(5\),pp\. 825–832\.Cited by:[§2\.4](https://arxiv.org/html/2605.24261#S2.SS4.p1.1)\.
- J\. E\. Helm, M\. S\. Lavieri, M\. P\. Van Oyen, J\. D\. Stein, and D\. C\. Musch \(2015\)Dynamic Forecasting and Control Algorithms of Glaucoma Progression for Clinician Decision Support\.Oper\. Res\.63\(5\),pp\. 979–999\.Cited by:[§2\.1](https://arxiv.org/html/2605.24261#S2.SS1.p1.1)\.
- R\. A\. Horn and C\. R\. Johnson \(1985\)Matrix Analysis\.1 edition,Cambridge University Press\.Cited by:[Proof 13\.35](https://arxiv.org/html/2605.24261#S13.Thmtheorem35.p1.39.39)\.
- J\. M\. Jakicic, K\. E\. Powell, W\. W\. Campbell, L\. Dipietro, R\. R\. Pate, L\. S\. Pescatello, K\. A\. Collins, B\. Bloodgood, and K\. L\. Piercy \(2019\)Physical Activity and the Prevention of Weight Gain in Adults: A Systematic Review\.Med\. Sci\. Sports Exerc\.51\(6\),pp\. 1262–1269\.Cited by:[§1\.1](https://arxiv.org/html/2605.24261#S1.SS1.p1.1)\.
- C\. Jin, Z\. Yang, Z\. Wang, and M\. I\. Jordan \(2023\)Provably Efficient Reinforcement Learning with Linear Function Approximation\.Mathematics of OR48\(3\),pp\. 1496–1521\.Cited by:[§2\.3](https://arxiv.org/html/2605.24261#S2.SS3.p1.1),[§5\.3\.2](https://arxiv.org/html/2605.24261#S5.SS3.SSS2.p1.3)\.
- P\. Klasnja, E\. B\. Hekler, S\. Shiffman, A\. Boruvka, D\. Almirall, A\. Tewari, and S\. A\. Murphy \(2015\)Microrandomized trials: An experimental design for developing just\-in\-time adaptive interventions\.\.Health Psychology34\(Suppl\),pp\. 1220–1228\.Cited by:[§2\.2](https://arxiv.org/html/2605.24261#S2.SS2.p1.1)\.
- P\. Klasnja, S\. Smith, N\. J\. Seewald, A\. Lee, K\. Hall, B\. Luers, E\. B\. Hekler, and S\. A\. Murphy \(2019\)Efficacy of Contextually Tailored Suggestions for Physical Activity: A Micro\-randomized Optimization Trial of HeartSteps\.Ann\. Behav\. Med\.53\(6\),pp\. 573–582\.Cited by:[§1\.1](https://arxiv.org/html/2605.24261#S1.SS1.p1.1),[§2\.2](https://arxiv.org/html/2605.24261#S2.SS2.p1.1)\.
- T\. Lai and H\. Robbins \(1985\)Asymptotically efficient adaptive allocation rules\.Advances in Applied Mathematics6\(1\),pp\. 4–22\.Cited by:[§2\.3](https://arxiv.org/html/2605.24261#S2.SS3.p1.1),[§5](https://arxiv.org/html/2605.24261#S5.p1.1)\.
- F\. Li, S\. M\. Carpenter, M\. P\. Buman, and Y\. Mintz \(2025\)Power Constrained Nonstationary Bandits with Habituation and Recovery Dynamics\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.2511.02944)Cited by:[§2\.3](https://arxiv.org/html/2605.24261#S2.SS3.p1.1)\.
- Q\. Li, K\. L\. Gavin, C\. I\. Voils, and Y\. Mintz \(2026\)An Adaptive Optimization Approach to Personalized Financial Incentives In Behavioral Interventions\.Production Oper\. Management35\(2\),pp\. 546–565\.Cited by:[§2\.4](https://arxiv.org/html/2605.24261#S2.SS4.p1.1)\.
- Y\. Li, T\. Zhang, S\. Das, J\. Shamma, and N\. Li \(2023\)Non\-asymptotic System Identification for Linear Systems with Nonlinear Policies\.IFAC\-Pap\.56\(2\),pp\. 1672–1679\.Cited by:[§2\.4](https://arxiv.org/html/2605.24261#S2.SS4.p2.1)\.
- P\. Liao, K\. Greenewald, P\. Klasnja, and S\. Murphy \(2020\)Personalized HeartSteps: A Reinforcement Learning Algorithm for Optimizing Physical Activity\.Proc\. ACM Interact\. Mob\. Wearable Ubiquitous Technol\.4\(1\),pp\. 1–22\.Cited by:[§2\.3](https://arxiv.org/html/2605.24261#S2.SS3.p1.1)\.
- J\. E\. Mason, D\. A\. England, B\. T\. Denton, S\. A\. Smith, M\. Kurt, and N\. D\. Shah \(2012\)Optimizing Statin Treatment Decisions for Diabetes Patients in the Presence of Uncertain Future Adherence\.Med Decis Making32\(1\),pp\. 154–166\.Cited by:[§2\.1](https://arxiv.org/html/2605.24261#S2.SS1.p1.1)\.
- A\. T\. McLellan, J\. L\. Starrels, B\. Tai, A\. J\. Gordon, R\. Brown, U\. Ghitza, M\. Gourevitch, J\. Stein, M\. Oros, T\. Horton, R\. Lindblad, and J\. McNeely \(2013\)Can Substance Use Disorders be Managed Using the Chronic Care Model? Review and Recommendations from a NIDA Consensus Group\.Public Health Rev35\(2\),pp\. 8\.Cited by:[§1\.1](https://arxiv.org/html/2605.24261#S1.SS1.p1.1)\.
- Y\. Mintz, A\. Aswani, P\. Kaminsky, E\. Flowers, and Y\. Fukuoka \(2020\)Nonstationary Bandits with Habituation and Recovery Dynamics\.Oper\. Res\.68\(5\),pp\. 1493–1516\.Cited by:[§2\.3](https://arxiv.org/html/2605.24261#S2.SS3.p1.1)\.
- I\. Nahum\-Shani and S\. A\. Murphy \(2026\)Just\-in\-Time Adaptive Interventions: Where Are We Now and What Is Next?\.Annu\. Rev\. Psychol\.77\(1\),pp\. 679–703\.Cited by:[§1\.1](https://arxiv.org/html/2605.24261#S1.SS1.p2.1)\.
- I\. Nahum\-Shani, S\. D\. Shaw, S\. M\. Carpenter, S\. A\. Murphy, and C\. Yoon \(2022\)Engagement in digital interventions\.\.American Psychologist77\(7\),pp\. 836–852\.Cited by:[§1\.1](https://arxiv.org/html/2605.24261#S1.SS1.p2.1),[§7\.1](https://arxiv.org/html/2605.24261#S7.SS1.p1.1)\.
- I\. Nahum\-Shani, S\. N\. Smith, B\. J\. Spring, L\. M\. Collins, K\. Witkiewitz, A\. Tewari, and S\. A\. Murphy \(2018\)Just\-in\-Time Adaptive Interventions \(JITAIs\) in Mobile Health: Key Components and Design Principles for Ongoing Health Behavior Support\.Ann\. Behav\. Med\.52\(6\),pp\. 446–462\.Cited by:[§2\.2](https://arxiv.org/html/2605.24261#S2.SS2.p1.1)\.
- I\. Nahum\-Shani and C\. Yoon \(2024\)Toward the Science of Engagement With Digital Interventions\.Curr Dir Psychol Sci33\(4\),pp\. 239–246\.Cited by:[§1\.1](https://arxiv.org/html/2605.24261#S1.SS1.p2.1)\.
- L\. P\. Palaniappan, N\. B\. Allen, Z\. I\. Almarzooq, C\. A\. Anderson, P\. Arora, C\. L\. Avery, C\. M\. Baker\-Smith, N\. Bansal, M\. E\. Currie, R\. S\. Earlie,et al\.\(2026\)2026 heart disease and stroke statistics: a report of US and global data from the american heart association\.Circulation\.Cited by:[§1](https://arxiv.org/html/2605.24261#S1.p1.1)\.
- O\. Perski, A\. Blandford, R\. West, and S\. Michie \(2017\)Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis\.Behav\. Med\. Pract\. Policy Res\.7\(2\),pp\. 254–267\.Cited by:[§1\.1](https://arxiv.org/html/2605.24261#S1.SS1.p2.1)\.
- O\. Perski, A\. Copeland, J\. Allen, M\. Pavel, D\. E\. Rivera, E\. Hekler, N\. Hankonen, and G\. Chevance \(2025\)The iterative development and refinement of health psychology theories through formal, dynamical systems modelling: a scoping review and initial expert\-derived ‘best practice’ recommendations\.Health Psych\. Rev\.19\(1\),pp\. 1–44\.Cited by:[§1\.1](https://arxiv.org/html/2605.24261#S1.SS1.p2.1),[§1\.1](https://arxiv.org/html/2605.24261#S1.SS1.p3.1),[item 2](https://arxiv.org/html/2605.24261#S7.I1.i2.p1.1)\.
- S\. L\. Proctor and P\. L\. Herschman \(2014\)The Continuing Care Model of Substance Use Treatment: What Works, and When Is “Enough,” “Enough?”\.Psychiatry Journal2014,pp\. 1–16\.Cited by:[§1\.1](https://arxiv.org/html/2605.24261#S1.SS1.p1.1),[§2\.2](https://arxiv.org/html/2605.24261#S2.SS2.p1.1)\.
- E\. Pulick, J\. Curtin, and Y\. Mintz \(2025\)Idiographic Lapse Prediction With State Space Modeling: Algorithm Development and Validation Study\.JMIR Form Res9,pp\. e73265\.Cited by:[§7\.1](https://arxiv.org/html/2605.24261#S7.SS1.p1.1)\.
- E\. Pulick and Y\. Mintz \(2025\)An Adaptive Control Approach to Treatment Selection for Substance Use Disorders\.In2025 IEEE 64th Conf\. Decis\. Control CDC,Rio de Janeiro, Brazil,pp\. 5713–5719\.Cited by:[§2\.4](https://arxiv.org/html/2605.24261#S2.SS4.p1.1)\.
- D\. E\. Rivera, M\. D\. Pew, and L\. M\. Collins \(2007\)Using engineering control principles to inform the design of adaptive interventions: A conceptual introduction\.Drug and Alcohol Dependence88,pp\. S31–S40\.Cited by:[§2\.4](https://arxiv.org/html/2605.24261#S2.SS4.p1.1)\.
- SAMHSA \(2025\)Key substance use and mental health indicators in the united states: Results from the 2024 national survey on drug use and health\.Technical reportTechnical Reportrpt56287,HHS, SAMHSA,Rockville, MD\.Cited by:[§1](https://arxiv.org/html/2605.24261#S1.p1.1)\.
- S\. C\. Sepah, L\. Jiang, R\. J\. Ellis, K\. McDermott, and A\. L\. Peters \(2017\)Engagement and outcomes in a digital Diabetes Prevention Program: 3\-year update\.BMJ Open Diab Res Care5\(1\),pp\. e000422\.Cited by:[§1](https://arxiv.org/html/2605.24261#S1.p1.1)\.
- S\. M\. Shechter, M\. D\. Bailey, A\. J\. Schaefer, and M\. S\. Roberts \(2008\)The Optimal Time to Initiate HIV Therapy Under Ordered Health States\.Oper\. Res\.56\(1\),pp\. 20–33\.Cited by:[§2\.1](https://arxiv.org/html/2605.24261#S2.SS1.p1.1)\.
- S\. Shiffman, A\. A\. Stone, and M\. R\. Hufford \(2008\)Ecological Momentary Assessment\.Annu\. Rev\. Clin\. Psychol\.4\(1\),pp\. 1–32\.Cited by:[§3\.1](https://arxiv.org/html/2605.24261#S3.SS1.p1.22)\.
- N\. D\. Shippee, N\. D\. Shah, C\. R\. May, F\. S\. Mair, and V\. M\. Montori \(2012\)Cumulative complexity: a functional, patient\-centered model of patient complexity can improve research and practice\.J\. Clin\. Epidemiol\.65\(10\),pp\. 1041–1051\.Cited by:[§1\.1](https://arxiv.org/html/2605.24261#S1.SS1.p2.1)\.
- M\. Simchowitz and D\. Foster \(2020\)Naive Exploration is Optimal for Online LQR\.InProc\. 37th Int\. Conf\. Mach\. Learn\.,pp\. 8937–8948\.Cited by:[§5\.4](https://arxiv.org/html/2605.24261#S5.SS4.p3.18)\.
- M\. Simchowitz, H\. Mania, S\. Tu, M\. I\. Jordan, and B\. Recht \(2018\)Learning Without Mixing: Towards A Sharp Analysis of Linear System Identification\.InProc\. 31st Conf\. Learn\. Theory,pp\. 439–473\.Cited by:[§2\.4](https://arxiv.org/html/2605.24261#S2.SS4.p2.1),[§4\.2](https://arxiv.org/html/2605.24261#S4.SS2.p1.2),[§4\.2](https://arxiv.org/html/2605.24261#S4.SS2.p2.1),[§4\.2](https://arxiv.org/html/2605.24261#S4.SS2.p5.2),[§4\.3](https://arxiv.org/html/2605.24261#S4.SS3.p3.11),[Theorem 4\.2](https://arxiv.org/html/2605.24261#S4.Thmtheorem2.p1.14.14)\.
- B\. Spring, A\. F\. Pfammatter, L\. Scanlan, E\. Daly, J\. Reading, S\. Battalio, H\. G\. McFadden, D\. Hedeker, J\. Siddique, and I\. Nahum\-Shani \(2024\)An Adaptive Behavioral Intervention for Weight Loss Management: A Randomized Clinical Trial\.JAMA332\(1\),pp\. 21\.Cited by:[§1](https://arxiv.org/html/2605.24261#S1.p1.1)\.
- D\. Spruijt\-Metz, B\. M\. Marlin, M\. Pavel, D\. E\. Rivera, E\. Hekler, S\. De La Torre, M\. El Mistiri, N\. M\. Golaszweski, C\. Li, R\. Braga De Braganca, K\. Tung, R\. Kha, and P\. Klasnja \(2022\)Advancing Behavioral Intervention and Theory Development for Mobile Health: The HeartSteps II Protocol\.IJERPH19\(4\),pp\. 2267\.Cited by:[§1\.1](https://arxiv.org/html/2605.24261#S1.SS1.p1.1),[§2\.2](https://arxiv.org/html/2605.24261#S2.SS2.p1.1)\.
- S\. Tomkins, P\. Liao, P\. Klasnja, and S\. Murphy \(2021\)IntelligentPooling: practical Thompson sampling for mHealth\.Mach Learn110\(9\),pp\. 2685–2727\.Cited by:[§2\.3](https://arxiv.org/html/2605.24261#S2.SS3.p1.1)\.
- A\. L\. Trella, K\. W\. Zhang, H\. Jajal, I\. Nahum\-Shani, V\. Shetty, F\. Doshi\-Velez, and S\. A\. Murphy \(2025\)A Deployed Online Reinforcement Learning Algorithm in an Oral Health Clinical Trial\.AAAI39\(28\),pp\. 28792–28800\.Cited by:[§2\.2](https://arxiv.org/html/2605.24261#S2.SS2.p1.1)\.
- A\. Tsiamis, I\. Ziemann, N\. Matni, and G\. J\. Pappas \(2023\)Statistical Learning Theory for Control: A Finite\-Sample Perspective\.IEEE Control Syst\.43\(6\),pp\. 67–97\.Cited by:[§2\.4](https://arxiv.org/html/2605.24261#S2.SS4.p2.1),[§4](https://arxiv.org/html/2605.24261#S4.p1.6)\.
- M\. J\. Wainwright \(2019\)High\-dimensional statistics: a non\-asymptotic viewpoint\.Cambridge University Press,Cambridge\.Cited by:[Proof 11\.2](https://arxiv.org/html/2605.24261#S11.Thmtheorem2.p1.15.15),[Proof 13\.32](https://arxiv.org/html/2605.24261#S13.Thmtheorem32.p1.36.23),[Proof 13\.40](https://arxiv.org/html/2605.24261#S13.Thmtheorem40.p1.12.12)\.
- P\. Whittle \(1988\)Restless bandits: activity allocation in a changing world\.J\. Appl\. Probab\.25\(A\),pp\. 287–298\.Cited by:[§2\.3](https://arxiv.org/html/2605.24261#S2.SS3.p1.1)\.
- D\. Zhou, J\. He, and Q\. Gu \(2021\)Provably Efficient Reinforcement Learning for Discounted MDPs with Feature Mapping\.InProc\. 38th Int\. Conf\. Mach\. Learn\.,pp\. 12793–12802\.Cited by:[§5\.4](https://arxiv.org/html/2605.24261#S5.SS4.p3.18)\.

\{APPENDICES\}

## 9Computational experiments

We provide pseudocode forLFA\-Q,TC\-Q, andGLM\-Banditas Algorithms[2](https://arxiv.org/html/2605.24261#alg2)\-[4](https://arxiv.org/html/2605.24261#alg4)\.

Algorithm 2Linear Function Approximation Q\-Learner \(LFA\-Q\)1:Learning rate

α∈\(0,1\)\\alpha\\in\(0,1\), decay

ϵd∈\(0,1\)\\epsilon\_\{d\}\\in\(0,1\), radial basis function \(RBF\) count

nn, discount factor

γ∈\(0,1\)\\gamma\\in\(0,1\), grid bound

CgC\_\{g\}
2:Place RBF centers

\{cj\}j=1n\\\{c\_\{j\}\\\}\_\{j=1\}^\{n\}uniformly on

\[−1,1\]\[\-1,1\], set width

σ=0\.8⋅\(c2−c1\)\\sigma=0\.8\\cdot\(c\_\{2\}\-c\_\{1\}\)
3:Define features

ϕj​\(x\)=exp⁡\(−\(x~−cj\)2/\(2​σ2\)\)\\phi\_\{j\}\(x\)=\\exp\\big\(\-\(\\tilde\{x\}\-c\_\{j\}\)^\{2\}/\(2\\sigma^\{2\}\)\\big\), where

x~\\tilde\{x\}rescales

xxto

\[−1,1\]\[\-1,1\]using

CgC\_\{g\}
4:Initialize weights

W∈ℝ\(M\+1\)×nW\\in\\mathbb\{R\}^\{\(M\+1\)\\times n\}
5:fort = 1,…,Tdo

6:Observe state

xtx\_\{t\}, set exploration level

ϵt=1/tϵd\\epsilon\_\{t\}=\\nicefrac\{\{1\}\}\{\{t^\{\\epsilon\_\{d\}\}\}\}
7:With prob\.

ϵt\\epsilon\_\{t\}, select

utu\_\{t\}randomly from

𝒰\\mathcal\{U\}, otherwise

ut=arg⁡maxi⁡Wi⊤​ϕ​\(xt\)u\_\{t\}=\\arg\\max\_\{i\}W\_\{i\}^\{\\top\}\\phi\(x\_\{t\}\)
8:Observe

r​\(xt,ut,dt\)r\(x\_\{t\},u\_\{t\},d\_\{t\}\)and

xt\+1x\_\{t\+1\}
9:Set TD error

δt=r​\(xt,ut,dt\)\+γ​maxi⁡Wi⊤​ϕ​\(xt\+1\)−Wut⊤​ϕ​\(xt\)\\delta\_\{t\}=r\(x\_\{t\},u\_\{t\},d\_\{t\}\)\+\\gamma\\max\_\{i\}W\_\{i\}^\{\\top\}\\phi\(x\_\{t\+1\}\)\-W\_\{u\_\{t\}\}^\{\\top\}\\phi\(x\_\{t\}\)
10:Update

Wut=Wut\+α​δt​ϕ​\(xt\)W\_\{u\_\{t\}\}=W\_\{u\_\{t\}\}\+\\alpha\\delta\_\{t\}\\phi\(x\_\{t\}\)
11:endfor

Algorithm 3Tile\-coded Q\-Learner \(TC\-Q\)1:Learning rate

α∈\(0,1\)\\alpha\\in\(0,1\), decay

ϵd∈\(0,1\)\\epsilon\_\{d\}\\in\(0,1\), bin width

ww, tiling count

ntn\_\{t\}, discount factor

γ∈\(0,1\)\\gamma\\in\(0,1\), grid bound

CgC\_\{g\}
2:Construct

ntn\_\{t\}tilings of the state space, each with bin width

ww, offset by

k​w/nt\\nicefrac\{\{kw\}\}\{\{n\_\{t\}\}\}for

k=0,…,nt−1k=0,\\dots,n\_\{t\}\-1
3:Let

bk​\(x\)b\_\{k\}\(x\)denote the bin index for state

xxunder tiling

kk
4:Initialize

Qk​\(b,u\)Q\_\{k\}\(b,u\)for all tilings

kk, bin indices

bb, and actions

uu, set

Q​\(x,u\)=∑kQk​\(bk​\(x\),u\)Q\(x,u\)=\\sum\_\{k\}Q\_\{k\}\(b\_\{k\}\(x\),u\)
5:fort = 1,…,Tdo

6:Observe state

xtx\_\{t\}, set exploration level

ϵt=1/tϵd\\epsilon\_\{t\}=\\nicefrac\{\{1\}\}\{\{t^\{\\epsilon\_\{d\}\}\}\}
7:With prob\.

ϵt\\epsilon\_\{t\}, select

utu\_\{t\}randomly from

𝒰\\mathcal\{U\}, otherwise

ut=arg⁡maxu⁡Q​\(xt,u\)u\_\{t\}=\\arg\\max\_\{u\}Q\(x\_\{t\},u\)
8:Observe

r​\(xt,ut,dt\)r\(x\_\{t\},u\_\{t\},d\_\{t\}\)and

xt\+1x\_\{t\+1\}
9:Set TD error

δt=r​\(xt,ut,dt\)\+γ​maxu⁡Q​\(xt\+1,u\)−Q​\(xt,ut\)\\delta\_\{t\}=r\(x\_\{t\},u\_\{t\},d\_\{t\}\)\+\\gamma\\max\_\{u\}Q\(x\_\{t\+1\},u\)\-Q\(x\_\{t\},u\_\{t\}\)
10:Update

Qk​\(bk​\(xt\),ut\)=Qk​\(bk​\(xt\),ut\)\+α/nt​δtQ\_\{k\}\(b\_\{k\}\(x\_\{t\}\),u\_\{t\}\)=Q\_\{k\}\(b\_\{k\}\(x\_\{t\}\),u\_\{t\}\)\+\\nicefrac\{\{\\alpha\}\}\{\{n\_\{t\}\}\}\\delta\_\{t\}for

k=0,…,nt−1k=0,\\dots,n\_\{t\}\-1
11:endfor

Algorithm 4GLM\-Bandit1:Confidence

δ∈\(0,1\)\\delta\\in\(0,1\),

CN\>0C\_\{N\}\>0, epoch count

k=1k=1, action counts

Ni=0N\_\{i\}=0\(

i=1,…,Mi=1,\\dots,M\)\.

2:Estimate

μ^1\\hat\{\\mu\}\_\{1\}, set

αiμ,\(k\)=αi,1μ​\(δ/2\)\\alpha\_\{i\}^\{\\mu,\(k\)\}=\\alpha^\{\\mu\}\_\{i,1\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\),

μ~i\(k\)=Π\[−μ¯,μ¯\]​\[μ^i,1\+αi,1μ\]\\tilde\{\\mu\}\_\{i\}^\{\(k\)\}=\\Pi\_\{\[\-\\bar\{\\mu\},\\bar\{\\mu\}\]\}\\big\[\\hat\{\\mu\}\_\{i,1\}\+\\alpha^\{\\mu\}\_\{i,1\}\\big\], action counts

Ni\(k\)=NiN\_\{i\}^\{\(k\)\}=N\_\{i\}\(

i=1,…,M\)i=1,\\dots,M\)\.

3:fort = 1,…,Tdo

4:Observe

dt−1d\_\{t\-1\},

xtx\_\{t\}
5:if

Ni\>\(1\+CN\)​Ni\(k\)N\_\{i\}\>\(1\+C\_\{N\}\)N\_\{i\}^\{\(k\)\}for any

i=1,…,Mi=1,\\dots,Mthen

6:Set

k=k\+1k=k\+1,

αiμ,\(k\)=αi,tμ​\(δ/2\)\\alpha^\{\\mu,\(k\)\}\_\{i\}=\\alpha^\{\\mu\}\_\{i,t\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\),

μ~i\(k\)=Π\[−μ¯,μ¯\]​\[μ^i,t\+αi,tμ\]\\tilde\{\\mu\}\_\{i\}^\{\(k\)\}=\\Pi\_\{\[\-\\bar\{\\mu\},\\bar\{\\mu\}\]\}\\big\[\\hat\{\\mu\}\_\{i,t\}\+\\alpha^\{\\mu\}\_\{i,t\}\\big\],

Ni\(k\)=NiN\_\{i\}^\{\(k\)\}=N\_\{i\}for

i=1,…,Mi=1,\\dots,M
7:endif

8:Take action

ut=arg⁡maxu∈𝒰⁡𝐄μ~\(k\)​\[r​\(xt,ut,dt\)\]u\_\{t\}=\\arg\\max\_\{u\\in\\mathcal\{U\}\}\\mathbf\{E\}\_\{\\tilde\{\\mu\}^\{\(k\)\}\}\\big\[r\(x\_\{t\},u\_\{t\},d\_\{t\}\)\\big\]
9:Set

Ni=Ni\+𝟏\{ut=ei\}N\_\{i\}=N\_\{i\}\+\\mathbf\{1\}\_\{\\\{u\_\{t\}=e\_\{i\}\\\}\},

i=1,…,Mi=1,\\dots,M
10:endfor

We ran experiments on a university computing cluster, using parameters:a¯=0\.85\\bar\{a\}=0\.85,b¯=3\.75\\bar\{b\}=3\.75,c¯=2\.75\\bar\{c\}=2\.75,w¯=2\.5\\bar\{w\}=2\.5, andμ¯=2\.5\\bar\{\\mu\}=2\.5\. Where needed, we discretized the statexxto a grid with resolution0\.10\.1\. WhileCx=b¯\+c¯\+w¯1−a¯C\_\{x\}=\\frac\{\\bar\{b\}\+\\bar\{c\}\+\\bar\{w\}\}\{1\-\\bar\{a\}\}bounds the state, we used the tighter range\[−Cx/3,Cx/3\]\[\\nicefrac\{\{\-C\_\{x\}\}\}\{\{3\}\},\\nicefrac\{\{C\_\{x\}\}\}\{\{3\}\}\]sinceCxC\_\{x\}is loose in practice\. For hyperparameter tuning, we generated a separate5050\-person cohort from our HM posterior and ran coarse and fine grid searches\. We selected the tuning configurations with the lowest end\-of\-trajectory cumulative regret, averaged over1010replications, the4×44\\times 4ablation grid from Experiment 1, and the population\. For computational efficiency during tuning, we evaluated algorithm value functions every2020time steps for regret calculations\. During coarse tuning we found that both Q\-learners performed well with fastϵ\\epsilondecay rates \(i\.e\., greater than11\), due to the built\-in exploration from optimistic initialization\. We found a negligible difference between fast\-decay cases and the no exploration case, so we fixed theϵ\\epsilondecay parameter \(1\.51\.5\) to shrink the tuning grids\. The final grid search forTC\-Qexplored bin widths of\[20,𝟒𝟎\]\[20,\\mathbf\{40\}\], tiling counts of\[𝟔𝟒,96,128,160,192\]\[\\mathbf\{64\},96,128,160,192\], and learning rates of\[0\.3,0\.4,0\.5,0\.6\]\[0\.3,0\.4,\\mathbf\{0\.5\},0\.6\]\(final values bolded\)\. AsTC\-Qconstructs bins to be the given value or finer, these bin widths capture when the grid is divided into either22or11interior bins; the algorithm performed best with very few bins and a high tiling count\. Tile counts of6464and192192performed similarly, so we chose the simpler model\. The final grid search forLFA\-Qtested RBF center counts of\[4,5,𝟔,7\]\[4,5,\\mathbf\{6\},7\]and learning rates of\[0\.2,0\.3,0\.4,0\.5\]\[0\.2,\\mathbf\{0\.3\},0\.4,0\.5\]\. To reduce computation, Experiments 2\-4 used the same hyperparameters as Experiment 1\. We testedCd,CN=\[0\.5,0\.75,1\.0\]C\_\{d\},C\_\{N\}=\[0\.5,0\.75,1\.0\]forUCB\-BOLDandGLM\-Banditand found marginal improvements with lower values, so we selected0\.50\.5for each to balance performance and computational efficiency\. Both algorithms used regularization values of11\. As the theoretically derived constants forUCB\-BOLDandGLM\-Banditare loose we found the algorithms performed well when theθ\\theta\- andμ\\mu\-derived bonus terms were rescaled so that the maximum initial values over the grid wereρ¯/2\\nicefrac\{\{\\bar\{\\rho\}\}\}\{\{2\}\}\.

## 10Supporting proofs

### 10\.1Supporting analysis for model definition \(Section[3](https://arxiv.org/html/2605.24261#S3)\)

Define filtrationsℱtd:=σ​\(x0:t,u0:t,d0:t−1,w0:t−1\)\\mathcal\{F\}^\{d\}\_\{t\}:=\\sigma\\left\(x\_\{0:t\},u\_\{0:t\},d\_\{0:t\-1\},w\_\{0:t\-1\}\\right\),ℱtw:=σ​\(x0:t,u0:t,d0:t,w0:t−1\)\\mathcal\{F\}^\{w\}\_\{t\}:=\\sigma\(x\_\{0:t\},u\_\{0:t\},d\_\{0:t\},w\_\{0:t\-1\}\), andℱt:=σ​\(x0:t,u0:t,d0:t,w0:t\)\\mathcal\{F\}\_\{t\}:=\\sigma\\left\(x\_\{0:t\},u\_\{0:t\},d\_\{0:t\},w\_\{0:t\}\\right\)for separate parts of the following analysis \(we usex0:=0x\_\{0\}:=0,u0,d0:=𝟎Mu\_\{0\},d\_\{0\}:=\\bm\{0\}\_\{M\},w0∼ww\_\{0\}\\sim wfor notational convenience\)\.

###### Proof 10\.1

Proof of Proposition[3\.1](https://arxiv.org/html/2605.24261#S3.Thmtheorem1)x1∼wx\_\{1\}\\sim wand\|w\|≤w¯≤Cx\|w\|\\leq\\bar\{w\}\\leq C\_\{x\}sox1∈𝒳x\_\{1\}\\in\\mathcal\{X\}by construction and we proceed by induction ontt\. Assumext∈𝒳x\_\{t\}\\in\\mathcal\{X\}fort≥1t\\geq 1\. Then\|xt\+1\|≤\|a​xt\|\+\|b⊤​ut\|\+\|c⊤​dt\|\+\|wt\|≤a¯​Cx\+b¯\+c¯\+w¯=b¯\+c¯\+w¯1−a¯=Cx\|x\_\{t\+1\}\|\\leq\|ax\_\{t\}\|\+\|b^\{\\top\}u\_\{t\}\|\+\|c^\{\\top\}d\_\{t\}\|\+\|w\_\{t\}\|\\leq\\bar\{a\}C\_\{x\}\+\\bar\{b\}\+\\bar\{c\}\+\\bar\{w\}=\\frac\{\\bar\{b\}\+\\bar\{c\}\+\\bar\{w\}\}\{1\-\\bar\{a\}\}=C\_\{x\}implyingxt\+1∈𝒳x\_\{t\+1\}\\in\\mathcal\{X\}\. The bound on the feature vector follows by the11\-sparsity ofutu\_\{t\}anddtd\_\{t\}under the assumed model structure\.\\halmos

### 10\.2OLS identification supporting proofs \(Section[4](https://arxiv.org/html/2605.24261#S4)\)

###### Corollary 10\.3

\(Corollary to Anderson’s Inequality\) For log\-concave and zero\-symmetric RVX∈ℝX\\in\\mathbb\{R\},ν\>0\\nu\>0, andy∈ℝy\\in\\mathbb\{R\}, then𝐏​\(\|X\+y\|≥ν\)≥𝐏​\(\|X\|≥ν\)\\mathbf\{P\}\(\|X\+y\|\\geq\\nu\)\\geq\\mathbf\{P\}\(\|X\|\\geq\\nu\)\. If RVY∈ℝY\\in\\mathbb\{R\}is independent ofXXthen𝐏​\(\|X\+Y\|≥ν\)≥𝐏​\(\|X\|≥ν\)\\mathbf\{P\}\(\|X\+Y\|\\geq\\nu\)\\geq\\mathbf\{P\}\(\|X\|\\geq\\nu\)\.

###### Proof 10\.4

Proof of Corollary[10\.3](https://arxiv.org/html/2605.24261#S10.Thmtheorem3)LetE:=\{x∈ℝ:−ν<x<ν\}E:=\\\{x\\in\\mathbb\{R\}:\-\\nu<x<\\nu\\\}\. We havef​\(x\)=f​\(−x\)f\(x\)=f\(\-x\)by assumption\. Note that log\-concavity of a distribution implies quasiconcavity, meaning that the superlevel sets forf​\(x\)f\(x\)are convex\(Boyd and Vandenberghe[2004](https://arxiv.org/html/2605.24261#bib.bib16)\)\. Then by Anderson’s Inequality \(Anderson \([1955](https://arxiv.org/html/2605.24261#bib.bib4)\), Corollary 2 and Theorem 2\) withk=0k=0we have𝐏​\(\|X\|<ν\)≥𝐏​\(\|X\+y\|<ν\)⇒𝐏​\(\|X\+y\|≥ν\)≥𝐏​\(\|X\|≥ν\)\\mathbf\{P\}\(\|X\|<\\nu\)\\geq\\mathbf\{P\}\(\|X\+y\|<\\nu\)\\Rightarrow\\mathbf\{P\}\(\|X\+y\|\\geq\\nu\)\\geq\\mathbf\{P\}\(\|X\|\\geq\\nu\)\. The random variable case follows by an identical argument\.\\halmos

The following assume Assumptions[3\.1](https://arxiv.org/html/2605.24261#S3.SS1)\-[3\.1](https://arxiv.org/html/2605.24261#S3.SS1)hold and an\(r,k\)\(r,k\)\-exploratory policy\. Lemma[10\.5](https://arxiv.org/html/2605.24261#S10.Thmtheorem5)demonstrates that there exists an appropriate choice ofτ\\taufor Proposition[10\.7](https://arxiv.org/html/2605.24261#S10.Thmtheorem7)\.τ\\tauis a proof tool that comes from the decomposition of the unit sphere into three cases and the optimal choice ofτ\\taudepends on the properties of the chosen noise distribution\.

###### Lemma 10\.5

There existsτ∈\(0,1\)\\tau\\in\(0,1\)with𝐏​\(\|w\|≥\(1−τ2\)τ​1\+4​M2​2​M\)\>0\\mathbf\{P\}\\big\(\|w\|\\geq\\frac\{\\sqrt\{\(1\-\\tau^\{2\}\)\}\}\{\\tau\}\\frac\{1\+4\\sqrt\{M\}\}\{2\\sqrt\{2M\}\}\\big\)\>0\.

###### Proof 10\.6

Proof of Lemma[10\.5](https://arxiv.org/html/2605.24261#S10.Thmtheorem5)By Assumption[3\.1](https://arxiv.org/html/2605.24261#S3.SS1),wwmust be fully supported on\[−C,C\]\[\-C,C\]forC\>0C\>0\. Letg​\(τ\)=\(1−τ2\)τ​1\+4​M2​2​Mg\(\\tau\)=\\frac\{\\sqrt\{\(1\-\\tau^\{2\}\)\}\}\{\\tau\}\\frac\{1\+4\\sqrt\{M\}\}\{2\\sqrt\{2M\}\}\.g​\(τ\)g\(\\tau\)is continuous forτ∈\(0,1\)\\tau\\in\(0,1\)andlimτ→1−g​\(τ\)=0\\lim\_\{\\tau\\to 1^\{\-\}\}g\(\\tau\)=0\. Thus∃τ\\exists\\tausuch thatg​\(τ\)<Cg\(\\tau\)<Cand𝐏​\(\|w\|≥g​\(τ\)\)\>0\\mathbf\{P\}\(\|w\|\\geq g\(\\tau\)\)\>0\.\\halmos

###### Proposition 10\.7

Chooseτ∈\(0,1\)\\tau\\in\(0,1\)such thatp1​\(τ\):=𝐏​\(\|wt−1\|≥\(1−τ2\)τ​1\+4​M2​2​M\)\>0p\_\{1\}\(\\tau\):=\\mathbf\{P\}\\big\(\|w\_\{t\-1\}\|\\geq\\frac\{\\sqrt\{\(1\-\\tau^\{2\}\)\}\}\{\\tau\}\\frac\{1\+4\\sqrt\{M\}\}\{2\\sqrt\{2M\}\}\\big\)\>0\. Letp:=min⁡\{p1​\(τ\),r​p¯M\+1\}p:=\\min\\\{p\_\{1\}\(\\tau\),\\frac\{r\\underline\{p\}\}\{M\+1\}\\\}\. Then this system satisfies the\(k,1−τ28​M​I2​M\+1,p\)\(k,\\frac\{1\-\\tau^\{2\}\}\{8M\}I\_\{2M\+1\},p\)BMSB condition\.

###### Proof 10\.8

Proof of Proposition[10\.7](https://arxiv.org/html/2605.24261#S10.Thmtheorem7)Note that\(zt\)t≥1\(z\_\{t\}\)\_\{t\\geq 1\}isℱtw\\mathcal\{F\}^\{w\}\_\{t\}\-adapted\. We consider a block ofkktime steps following somej≥0j\\geq 0\. Letλ=\[λ1,λ2⊤,λ3⊤\]⊤\\lambda=\[\\lambda\_\{1\},\\lambda\_\{2\}^\{\\top\},\\lambda\_\{3\}^\{\\top\}\]^\{\\top\}withλ1∈ℝ\\lambda\_\{1\}\\in\\mathbb\{R\},λ2∈ℝM\\lambda\_\{2\}\\in\\mathbb\{R\}^\{M\},λ3∈ℝM\\lambda\_\{3\}\\in\\mathbb\{R\}^\{M\}\. We must show that for anyλ\\lambdawith‖λ‖2=1\\\|\\lambda\\\|\_\{2\}=1,1k​∑i=1k𝐏​\(\|λ⊤​zj\+i\|≥λ⊤​\(1−τ28​M​I2​M\+1\)​λ\|ℱjw\)≥p\\frac\{1\}\{k\}\\sum\_\{i=1\}^\{k\}\\mathbf\{P\}\(\|\\lambda^\{\\top\}z\_\{j\+i\}\|\\geq\\sqrt\{\\lambda^\{\\top\}\(\\frac\{1\-\\tau^\{2\}\}\{8M\}I\_\{2M\+1\}\)\\lambda\}\|\\mathcal\{F\}^\{w\}\_\{j\}\)\\geq pholds, or equivalently1k​∑i=1k𝐏​\(\|λ⊤​zj\+i\|≥ν\|ℱjw\)≥p\\frac\{1\}\{k\}\\sum\_\{i=1\}^\{k\}\\mathbf\{P\}\(\|\\lambda^\{\\top\}z\_\{j\+i\}\|\\geq\\nu\|\\mathcal\{F\}^\{w\}\_\{j\}\)\\geq pforν:=12​1−τ22​M\\nu:=\\frac\{1\}\{2\}\\sqrt\{\\frac\{1\-\\tau^\{2\}\}\{2M\}\}\. We prove this condition for three cases ofλ\\lambda:\|λ1\|≥τ\|\\lambda\_\{1\}\|\\geq\\tau,\|λ1\|<τ\|\\lambda\_\{1\}\|<\\tauwith‖λ3‖2≥1−τ22\\\|\\lambda\_\{3\}\\\|\_\{2\}\\geq\\sqrt\{\\frac\{1\-\\tau^\{2\}\}\{2\}\}, and\|λ1\|<τ\|\\lambda\_\{1\}\|<\\tauwith‖λ3‖2<1−τ22\\\|\\lambda\_\{3\}\\\|\_\{2\}<\\sqrt\{\\frac\{1\-\\tau^\{2\}\}\{2\}\}\. Cases 2 and 3 represent when at least half of the remainder magnitude \(1−τ21\-\\tau^\{2\}\) comes fromλ3\\lambda\_\{3\}orλ2\\lambda\_\{2\}, respectively\.

Case 1:Let\|λ1\|≥τ\|\\lambda\_\{1\}\|\\geq\\tau\. Consider arbitrary time stept\>jt\>jin thekk\-block\. If\|λ1\|≥τ\|\\lambda\_\{1\}\|\\geq\\tau, then‖\[λ2⊤,λ3⊤\]‖2≤1−τ2\\\|\[\\lambda\_\{2\}^\{\\top\},\\lambda\_\{3\}^\{\\top\}\]\\\|\_\{2\}\\leq\\sqrt\{1\-\\tau^\{2\}\}\. Asutu\_\{t\}anddtd\_\{t\}are11\-sparse, the reverse triangle inequality and Cauchy\-Schwarz give\|λ⊤​zt\|=\|λ1​xt\+λ2⊤​ut\+λ3⊤​dt\|≥\|\|λ1​xt\|−\|λ2⊤​ut\+λ3⊤​dt\|\|≥\|λ1​xt\|−2​\(1−τ2\)\\lvert\\lambda^\{\\top\}z\_\{t\}\\rvert=\\lvert\\lambda\_\{1\}x\_\{t\}\+\\lambda\_\{2\}^\{\\top\}u\_\{t\}\+\\lambda\_\{3\}^\{\\top\}d\_\{t\}\\rvert\\geq\\left\\lvert\|\\lambda\_\{1\}x\_\{t\}\|\-\|\\lambda\_\{2\}^\{\\top\}u\_\{t\}\+\\lambda\_\{3\}^\{\\top\}d\_\{t\}\|\\right\\rvert\\geq\\lvert\\lambda\_\{1\}x\_\{t\}\\rvert\-\\sqrt\{2\(1\-\\tau^\{2\}\)\}, so it suffices to lower bound𝐏​\(\|λ1​xt\|≥ν\+2​\(1−τ2\)\|ℱjw\)\\mathbf\{P\}\(\|\\lambda\_\{1\}x\_\{t\}\|\\geq\\nu\+\\sqrt\{2\(1\-\\tau^\{2\}\)\}\|\\mathcal\{F\}^\{w\}\_\{j\}\)\. Per the system dynamics:λ1​xt=λ1​wt−1\+\[λ1​a​xt−1\+λ1​b⊤​ut−1\+λ1​c⊤​dt−1\]\\lambda\_\{1\}x\_\{t\}=\\lambda\_\{1\}w\_\{t\-1\}\+\[\\lambda\_\{1\}ax\_\{t\-1\}\+\\lambda\_\{1\}b^\{\\top\}u\_\{t\-1\}\+\\lambda\_\{1\}c^\{\\top\}d\_\{t\-1\}\]\. LetQt−1Q\_\{t\-1\}denote the bracketed term\. Observe thatQt−1\|ℱjwQ\_\{t\-1\}\|\\mathcal\{F\}^\{w\}\_\{j\}is fixed fort=j\+1t=j\+1and a random variable, independent ofwt−1w\_\{t\-1\}, ift\>j\+1t\>j\+1\. Thus by Corollary[10\.3](https://arxiv.org/html/2605.24261#S10.Thmtheorem3):

𝐏​\(\|λ1​xt\|≥ν\+2​\(1−τ2\)\|ℱjw\)=𝐏​\(\|λ1​wt−1\+Qt−1\|≥ν\+2​\(1−τ2\)\|ℱjw\)\\displaystyle\\mathbf\{P\}\(\|\\lambda\_\{1\}x\_\{t\}\|\\geq\\nu\+\\sqrt\{2\(1\-\\tau^\{2\}\)\}\|\\mathcal\{F\}^\{w\}\_\{j\}\)=\\mathbf\{P\}\(\|\\lambda\_\{1\}w\_\{t\-1\}\+Q\_\{t\-1\}\|\\geq\\nu\+\\sqrt\{2\(1\-\\tau^\{2\}\)\}\|\\mathcal\{F\}^\{w\}\_\{j\}\)\(11\)≥𝐏​\(\|λ1​wt−1\|≥ν\+2​\(1−τ2\)\)≥𝐏​\(\|wt−1\|≥1τ​\[12​\(1−τ2\)/2​M\+2​\(1−τ2\)\]\)=p1​\(τ\)\\displaystyle\\quad\\geq\\mathbf\{P\}\(\|\\lambda\_\{1\}w\_\{t\-1\}\|\\geq\\nu\+\\sqrt\{2\(1\-\\tau^\{2\}\)\}\)\\geq\\mathbf\{P\}\(\|w\_\{t\-1\}\|\\geq\\tfrac\{1\}\{\\tau\}\[\\tfrac\{1\}\{2\}\\sqrt\{\\nicefrac\{\{\(1\-\\tau^\{2\}\)\}\}\{\{2M\}\}\}\+\\sqrt\{2\(1\-\\tau^\{2\}\)\}\]\)=p\_\{1\}\(\\tau\)\(12\)For thekk\-block we have1k​∑i=1k𝐏​\(\|λ⊤​zj\+i\|≥ν\|ℱjw\)≥1k​∑i=1kp1​\(τ\)=p1​\(τ\)\\frac\{1\}\{k\}\\sum\_\{i=1\}^\{k\}\\mathbf\{P\}\(\|\\lambda^\{\\top\}z\_\{j\+i\}\|\\geq\\nu\|\\mathcal\{F\}^\{w\}\_\{j\}\)\\geq\\frac\{1\}\{k\}\\sum\_\{i=1\}^\{k\}p\_\{1\}\(\\tau\)=p\_\{1\}\(\\tau\)\.

Case 2:Let\|λ1\|<τ\|\\lambda\_\{1\}\|<\\tauand‖λ3‖2≥1−τ22\\\|\\lambda\_\{3\}\\\|\_\{2\}\\geq\\sqrt\{\\frac\{1\-\\tau^\{2\}\}\{2\}\}\. LetEtE\_\{t\}be the \(ℱt−1\\mathcal\{F\}\_\{t\-1\}\-measurable\) event that an exploratory action is taken in steptt\. We considert\>jt\>jin thekk\-block with𝟏​\(Et\)=1\\mathbf\{1\}\(E\_\{t\}\)=1and focus on the action indexi∗∈\{1,…,M\}i^\{\*\}\\in\\\{1,\\dots,M\\\}where\|λ3i\|\|\\lambda\_\{3\}^\{i\}\|is largest\. Observe that\|λ3i∗\|\|\\lambda\_\{3\}^\{i^\{\*\}\}\|is smallest when all components ofλ3\\lambda\_\{3\}have equal magnitude\. Since‖λ3‖2≥1−τ22\\\|\\lambda\_\{3\}\\\|\_\{2\}\\geq\\sqrt\{\\frac\{1\-\\tau^\{2\}\}\{2\}\}, it follows that\|λ3i∗\|≥1−τ22​1M=2​ν\|\\lambda\_\{3\}^\{i^\{\*\}\}\|\\geq\\sqrt\{\\frac\{1\-\\tau^\{2\}\}\{2\}\}\\frac\{1\}\{\\sqrt\{M\}\}=2\\nu\. By definition of the\(r,k\)\(r,k\)\-policy,𝐏​\(ut=ei∗\|ℱt−1,Et\)=1M\+1\\mathbf\{P\}\(u\_\{t\}=e\_\{i^\{\*\}\}\|\\mathcal\{F\}\_\{t\-1\},E\_\{t\}\)=\\frac\{1\}\{M\+1\}\. We show that for any state,xt∈ℝx\_\{t\}\\in\\mathbb\{R\}, the system is sufficiently excited by at least one outcome; we define two sets corresponding to the outcomesdti∗=1d^\{i^\{\*\}\}\_\{t\}=1anddti∗=0d^\{i^\{\*\}\}\_\{t\}=0\. LetA=\{xt∈ℝ:\|λ1​xt\+λ2i∗\+λ3i∗\|≥ν\}A=\\\{x\_\{t\}\\in\\mathbb\{R\}:\|\\lambda\_\{1\}x\_\{t\}\+\\lambda^\{i^\{\*\}\}\_\{2\}\+\\lambda^\{i^\{\*\}\}\_\{3\}\|\\geq\\nu\\\}andB=\{xt∈ℝ:\|λ1​xt\+λ2i∗\|≥ν\}B=\\\{x\_\{t\}\\in\\mathbb\{R\}:\|\\lambda\_\{1\}x\_\{t\}\+\\lambda^\{i^\{\*\}\}\_\{2\}\|\\geq\\nu\\\}\. We claimAC∩BC=∅A^\{C\}\\cap B^\{C\}=\\emptyset, i\.e\., anyxt∈ℝx\_\{t\}\\in\\mathbb\{R\}belongs to at least one set\. By contradiction, let∃x∈ℝ\\exists x\\in\\mathbb\{R\}withx∈\(AC∩BC\)x\\in\(A^\{C\}\\cap B^\{C\}\)\. Above we established\|λ3i∗\|≥2​ν\|\\lambda^\{i^\{\*\}\}\_\{3\}\|\\geq 2\\nu, so the triangle inequality and the set properties produce the desired contradiction:\|λ3i∗\|=\|\(λ1​x\+λ2i∗\+λ3i∗\)−\(λ1​x\+λ2i∗\)\|≤\|λ1​x\+λ2i∗\+λ3i∗\|\+\|λ1​x\+λ2i∗\|<2​ν\|\\lambda^\{i^\{\*\}\}\_\{3\}\|=\|\(\\lambda\_\{1\}x\+\\lambda^\{i^\{\*\}\}\_\{2\}\+\\lambda^\{i^\{\*\}\}\_\{3\}\)\-\(\\lambda\_\{1\}x\+\\lambda^\{i^\{\*\}\}\_\{2\}\)\|\\leq\|\\lambda\_\{1\}x\+\\lambda^\{i^\{\*\}\}\_\{2\}\+\\lambda^\{i^\{\*\}\}\_\{3\}\|\+\|\\lambda\_\{1\}x\+\\lambda^\{i^\{\*\}\}\_\{2\}\|<2\\nu\. This implies that for anyxt∈ℝx\_\{t\}\\in\\mathbb\{R\},𝟏A​\(xt\)\+𝟏B​\(xt\)≥1\\mathbf\{1\}\_\{A\}\(x\_\{t\}\)\+\\mathbf\{1\}\_\{B\}\(x\_\{t\}\)\\geq 1\. Recall thatxtx\_\{t\}isℱt−1\\mathcal\{F\}\_\{t\-1\}\-measurable\. We lower bound with theut=ei∗u\_\{t\}=e\_\{i^\{\*\}\}case:

𝐏​\(\|λ⊤​zt\|≥ν\|ℱt−1,Et\)≥𝐏​\(ut=ei∗\|ℱt−1,Et\)​𝐏​\(\|λ⊤​zt\|≥ν\|ℱt−1,Et,ut=ei∗\)\\displaystyle\\mathbf\{P\}\(\|\\lambda^\{\\top\}z\_\{t\}\|\\geq\\nu\|\\mathcal\{F\}\_\{t\-1\},E\_\{t\}\)\\geq\\mathbf\{P\}\(u\_\{t\}=e\_\{i^\{\*\}\}\|\\mathcal\{F\}\_\{t\-1\},E\_\{t\}\)\\mathbf\{P\}\(\|\\lambda^\{\\top\}z\_\{t\}\|\\geq\\nu\|\\mathcal\{F\}\_\{t\-1\},E\_\{t\},u\_\{t\}=e\_\{i^\{\*\}\}\)\(13\)=\(a\)1M\+1\[𝟏A\(xt\)p^i∗\(xt,1\)\+𝟏B\(xt\)\(1−p^i∗\(xt,1\)\)\]≥\(b\)min\{p¯M\+1,\(1−p¯\)M\+1\}=:p2\\displaystyle\\quad\\overset\{\(a\)\}\{=\}\\tfrac\{1\}\{M\+1\}\[\\mathbf\{1\}\_\{A\}\(x\_\{t\}\)\\hat\{p\}\_\{i^\{\*\}\}\(x\_\{t\},1\)\+\\mathbf\{1\}\_\{B\}\(x\_\{t\}\)\(1\-\\hat\{p\}\_\{i^\{\*\}\}\(x\_\{t\},1\)\)\]\\overset\{\(b\)\}\{\\geq\}\\min\\\{\\tfrac\{\\underline\{p\}\}\{M\+1\},\\tfrac\{\(1\-\\overline\{p\}\)\}\{M\+1\}\\\}=:p\_\{2\}\(14\)\(a\)\(a\)substitutes the action probability, expands adherence outcomes, and uses the set definitions\.\(b\)\(b\)substitutesp^\\hat\{p\}bounds by Remark[3\.2](https://arxiv.org/html/2605.24261#S3.Thmtheorem2), holding∀xt∈𝒳\\forall x\_\{t\}\\in\\mathcal\{X\}, and the set conclusion above\. SinceEtE\_\{t\}isℱt−1\\mathcal\{F\}\_\{t\-1\}measurable𝐏​\(Et\|ℱt−1\)=𝟏​\(Et\)\\mathbf\{P\}\(E\_\{t\}\|\\mathcal\{F\}\_\{t\-1\}\)=\\mathbf\{1\}\(E\_\{t\}\)\. We have𝐏​\(\|λ⊤​zt\|≥ν\|ℱt−1\)≥𝐏​\(\|λ⊤​zt\|≥ν,Et\|ℱt−1\)=𝐏​\(Et\|ℱt−1\)​𝐏​\(\|λ⊤​zt\|≥ν\|ℱt−1,Et\)≥𝟏​\(Et\)​p2\\mathbf\{P\}\(\|\\lambda^\{\\top\}z\_\{t\}\|\\geq\\nu\|\\mathcal\{F\}\_\{t\-1\}\)\\geq\\mathbf\{P\}\(\|\\lambda^\{\\top\}z\_\{t\}\|\\geq\\nu,E\_\{t\}\|\\mathcal\{F\}\_\{t\-1\}\)=\\mathbf\{P\}\(E\_\{t\}\|\\mathcal\{F\}\_\{t\-1\}\)\\mathbf\{P\}\(\|\\lambda^\{\\top\}z\_\{t\}\|\\geq\\nu\|\\mathcal\{F\}\_\{t\-1\},E\_\{t\}\)\\geq\\mathbf\{1\}\(E\_\{t\}\)p\_\{2\}\. Thus:

1k​∑i=1k𝐏​\(\|λ⊤​zj\+i\|≥ν\|ℱjw\)​=\(c\)​1k​∑i=1k𝐄​\[𝐏​\(\|λ⊤​zj\+i\|≥ν\|ℱj\+i−1\)\|ℱjw\]\\displaystyle\\tfrac\{1\}\{k\}\\textstyle\\sum\\nolimits\_\{i=1\}^\{k\}\\mathbf\{P\}\(\|\\lambda^\{\\top\}z\_\{j\+i\}\|\\geq\\nu\|\\mathcal\{F\}^\{w\}\_\{j\}\)\\overset\{\(c\)\}\{=\}\\tfrac\{1\}\{k\}\\textstyle\\sum\\nolimits\_\{i=1\}^\{k\}\\mathbf\{E\}\[\\mathbf\{P\}\(\|\\lambda^\{\\top\}z\_\{j\+i\}\|\\geq\\nu\|\\mathcal\{F\}\_\{j\+i\-1\}\)\|\\mathcal\{F\}^\{w\}\_\{j\}\]\(15\)≥\(d\)​1k​∑i=1k𝐄​\[𝟏​\(Ej\+i\)​p2\|ℱjw\]​=\(e\)​p2k​𝐄​\[∑i=1k𝟏​\(Ej\+i\)\|ℱjw\]​≥\(f\)​p2​⌈r​k⌉k≥r​p2\\displaystyle\\qquad\\overset\{\(d\)\}\{\\geq\}\\tfrac\{1\}\{k\}\\textstyle\\sum\\nolimits\_\{i=1\}^\{k\}\\mathbf\{E\}\[\\mathbf\{1\}\(E\_\{j\+i\}\)p\_\{2\}\|\\mathcal\{F\}^\{w\}\_\{j\}\]\\overset\{\(e\)\}\{=\}\\tfrac\{p\_\{2\}\}\{k\}\\mathbf\{E\}\[\\textstyle\\sum\\nolimits\_\{i=1\}^\{k\}\\mathbf\{1\}\(E\_\{j\+i\}\)\|\\mathcal\{F\}^\{w\}\_\{j\}\]\\overset\{\(f\)\}\{\\geq\}\\tfrac\{p\_\{2\}\\lceil rk\\rceil\}\{k\}\\geq rp\_\{2\}\(16\)\(c\)\(c\)uses the tower property asℱjw⊆ℱj\+i−1\\mathcal\{F\}\_\{j\}^\{w\}\\subseteq\\mathcal\{F\}\_\{j\+i\-1\}fori≥1i\\geq 1\.\(d\)\(d\)uses the per\-step bound above\.\(e\)\(e\)uses the linearity of expectation and\(f\)\(f\)uses the guaranteed exploratory steps per block of\(r,k\)\(r,k\)\-policies\.

Case 3:Let\|λ1\|<τ\|\\lambda\_\{1\}\|<\\tauand‖λ3‖2<1−τ22\\\|\\lambda\_\{3\}\\\|\_\{2\}<\\sqrt\{\\frac\{1\-\\tau^\{2\}\}\{2\}\}\.‖λ‖2=1\\\|\\lambda\\\|\_\{2\}=1implies that‖λ2‖22=1−\|λ1\|2−‖λ3‖22≥1−τ22\\\|\\lambda\_\{2\}\\\|\_\{2\}^\{2\}=1\-\|\\lambda\_\{1\}\|^\{2\}\-\\\|\\lambda\_\{3\}\\\|\_\{2\}^\{2\}\\geq\\frac\{1\-\\tau^\{2\}\}\{2\}\. Similar to Case 2, this implies some maximizing indexi∗∈\{1,…,M\}i^\{\*\}\\in\\\{1,\\dots,M\\\}with\|λ2i∗\|≥1−τ22​1M=2​ν\|\\lambda\_\{2\}^\{i^\{\*\}\}\|\\geq\\sqrt\{\\frac\{1\-\\tau^\{2\}\}\{2\}\}\\frac\{1\}\{\\sqrt\{M\}\}=2\\nu\. DefineEtE\_\{t\}as in Case 2 and considert\>jt\>jwith𝟏​\(Et\)=1\\mathbf\{1\}\(E\_\{t\}\)=1\. We make a similar argument to Case 2, but consider two actions:ei∗e\_\{i^\{\*\}\}and𝟎M\\bm\{0\}\_\{M\}\. As before, the\(r,k\)\(r,k\)\-policy definition gives𝐏​\(ut=ei∗\|ℱt−1,Et\)=1M\+1\\mathbf\{P\}\(u\_\{t\}=e\_\{i^\{\*\}\}\|\\mathcal\{F\}\_\{t\-1\},E\_\{t\}\)=\\frac\{1\}\{M\+1\}\. We define two relevant sets,A=\{xt∈ℝ:\|λ1​xt\+λ2i∗\|≥ν\}A=\\\{x\_\{t\}\\in\\mathbb\{R\}:\|\\lambda\_\{1\}x\_\{t\}\+\\lambda\_\{2\}^\{i^\{\*\}\}\|\\geq\\nu\\\}andB=\{xt∈ℝ:\|λ1​xt\|≥ν\}B=\\\{x\_\{t\}\\in\\mathbb\{R\}:\|\\lambda\_\{1\}x\_\{t\}\|\\geq\\nu\\\}\. These correspond to the casesuti∗=1,dti∗=0u^\{i^\{\*\}\}\_\{t\}=1,d^\{i^\{\*\}\}\_\{t\}=0andut=𝟎M,dt=𝟎Mu\_\{t\}=\\bm\{0\}\_\{M\},d\_\{t\}=\\bm\{0\}\_\{M\}, respectively\. We claimAC∩BC=∅A^\{C\}\\cap B^\{C\}=\\emptyset, i\.e\., anyxt∈ℝx\_\{t\}\\in\\mathbb\{R\}belongs to at least one set\. For a contradiction, assume∃x∈ℝ\\exists x\\in\\mathbb\{R\}withx∈\(AC∩BC\)x\\in\\left\(A^\{C\}\\cap B^\{C\}\\right\)\. Earlier we established that\|λ2i∗\|≥2​ν\|\\lambda^\{i^\{\*\}\}\_\{2\}\|\\geq 2\\nu, so the triangle inequality and the set properties produce the desired contradiction:\|λ2i∗\|=\|\(λ1​x\+λ2i∗\)−\(λ1​x\)\|≤\|λ1​x\+λ2i∗\|\+\|λ1​x\|<2​ν\|\\lambda\_\{2\}^\{i^\{\*\}\}\|=\|\(\\lambda\_\{1\}x\+\\lambda\_\{2\}^\{i^\{\*\}\}\)\-\(\\lambda\_\{1\}x\)\|\\leq\|\\lambda\_\{1\}x\+\\lambda\_\{2\}^\{i^\{\*\}\}\|\+\|\\lambda\_\{1\}x\|<2\\nu\. This implies that for anyxt∈ℝx\_\{t\}\\in\\mathbb\{R\},𝟏A​\(xt\)\+𝟏B​\(xt\)≥1\\mathbf\{1\}\_\{A\}\(x\_\{t\}\)\+\\mathbf\{1\}\_\{B\}\(x\_\{t\}\)\\geq 1\. We lower bound with theut=𝟎Mu\_\{t\}=\\bm\{0\}\_\{M\}andut=ei∗u\_\{t\}=e\_\{i^\{\*\}\}cases:

𝐏​\(\|λ⊤​zt\|≥ν\|ℱt−1,Et\)\\displaystyle\\mathbf\{P\}\(\|\\lambda^\{\\top\}z\_\{t\}\|\\geq\\nu\|\\mathcal\{F\}\_\{t\-1\},E\_\{t\}\)≥𝐏​\(ut=𝟎M\|ℱt−1,Et\)​𝐏​\(\|λ⊤​zt\|≥ν\|ℱt−1,Et,ut=𝟎M\)\\displaystyle\\geq\\mathbf\{P\}\(u\_\{t\}=\\bm\{0\}\_\{M\}\|\\mathcal\{F\}\_\{t\-1\},E\_\{t\}\)\\mathbf\{P\}\(\|\\lambda^\{\\top\}z\_\{t\}\|\\geq\\nu\|\\mathcal\{F\}\_\{t\-1\},E\_\{t\},u\_\{t\}=\\bm\{0\}\_\{M\}\)\+𝐏​\(ut=ei∗\|ℱt−1,Et\)​𝐏​\(\|λ⊤​zt\|≥ν\|ℱt−1,Et,ut=ei∗\)\\displaystyle\\qquad\+\\mathbf\{P\}\(u\_\{t\}=e\_\{i^\{\*\}\}\|\\mathcal\{F\}\_\{t\-1\},E\_\{t\}\)\\mathbf\{P\}\(\|\\lambda^\{\\top\}z\_\{t\}\|\\geq\\nu\|\\mathcal\{F\}\_\{t\-1\},E\_\{t\},u\_\{t\}=e\_\{i^\{\*\}\}\)\(17\)≥\(a\)1M\+1\[𝟏B\(xt\)\+\(1−p^i∗\(xt,1\)\)𝟏A\(xt\)\]≥\(b\)min\{1M\+1,1−p¯M\+1\}=1−p¯M\+1=:p3\\displaystyle\\overset\{\(a\)\}\{\\geq\}\\tfrac\{1\}\{M\+1\}\[\\mathbf\{1\}\_\{B\}\(x\_\{t\}\)\+\(1\-\\hat\{p\}\_\{i^\{\*\}\}\(x\_\{t\},1\)\)\\mathbf\{1\}\_\{A\}\(x\_\{t\}\)\]\\overset\{\(b\)\}\{\\geq\}\\min\\\{\\tfrac\{1\}\{M\+1\},\\tfrac\{1\-\\overline\{p\}\}\{M\+1\}\\\}=\\tfrac\{1\-\\overline\{p\}\}\{M\+1\}=:p\_\{3\}\(18\)\(a\)\(a\)substitutes the action probabilities, expands the Bernoulli adherence outcomes \(adherent case is dropped\), and uses the set definitions\.\(b\)\(b\)uses the set conclusion above and substitutesp^\\hat\{p\}bounds by Remark[3\.2](https://arxiv.org/html/2605.24261#S3.Thmtheorem2), holding∀xt∈𝒳\\forall x\_\{t\}\\in\\mathcal\{X\}\. Following an identical tower property argument to Case 2, we have for thekk\-block:1k​∑i=1k𝐏​\(\|λ⊤​zj\+i\|≥ν\|ℱjw\)≥r​p3\\frac\{1\}\{k\}\\sum\_\{i=1\}^\{k\}\\mathbf\{P\}\(\|\\lambda^\{\\top\}z\_\{j\+i\}\|\\geq\\nu\|\\mathcal\{F\}^\{w\}\_\{j\}\)\\geq rp\_\{3\}\. Thus the BMSB condition holds for all cases withp=min⁡\{p1​\(τ\),r​p2,r​p3\}=min⁡\{p1​\(τ\),r​p¯M\+1\}p=\\min\\\{p\_\{1\}\(\\tau\),rp\_\{2\},rp\_\{3\}\\\}=\\min\\\{p\_\{1\}\(\\tau\),\\frac\{r\\underline\{p\}\}\{M\+1\}\\\}asp¯=1−p¯\\underline\{p\}=1\-\\overline\{p\}by the symmetry of the sigmoid\.\\halmos

###### Proof 10\.9

Proof of Proposition[4\.5](https://arxiv.org/html/2605.24261#S4.Thmtheorem5)By Proposition[3\.1](https://arxiv.org/html/2605.24261#S3.Thmtheorem1),‖zt‖22≤Cx2\+2\\\|z\_\{t\}\\\|^\{2\}\_\{2\}\\leq C\_\{x\}^\{2\}\+2fort≥1t\\geq 1almost surely\. Letu∈ℝ2​M\+1u\\in\\mathbb\{R\}^\{2M\+1\}and observe by Cauchy\-Schwarzu⊤​zt​zt⊤​u=\(zt⊤​u\)2≤‖zt‖22​‖u‖22≤u⊤​\(\(Cx2\+2\)​𝐈2​M\+1\)​uu^\{\\top\}z\_\{t\}z\_\{t\}^\{\\top\}u=\(z\_\{t\}^\{\\top\}u\)^\{2\}\\leq\\\|z\_\{t\}\\\|^\{2\}\_\{2\}\\\|u\\\|^\{2\}\_\{2\}\\leq u^\{\\top\}\(\(C\_\{x\}^\{2\}\+2\)\\mathbf\{I\}\_\{2M\+1\}\)u\. Thuszt​zt⊤⪯\(Cx2\+2\)​𝐈2​M\+1z\_\{t\}z\_\{t\}^\{\\top\}\\preceq\(C\_\{x\}^\{2\}\+2\)\\mathbf\{I\}\_\{2M\+1\}almost surely\. The proposition follows by summing over time steps\.\\halmos

\\ECSwitch\\ECHead

Electronic Companion This EC includes proofs and supporting results for system identification \(Section[4](https://arxiv.org/html/2605.24261#S4)\) and regret analysis forUCB\-BOLD\(Section[5](https://arxiv.org/html/2605.24261#S5)\)\.

## 11MLE\-related system identification supporting results

The following results support the MLE\-portion of our identification result\. Letpi,t=σ​\(xt\+μi\)p\_\{i,t\}=\\sigma\(x\_\{t\}\+\\mu\_\{i\}\)andpi,t′=∂pi,t∂μi=pi,t​\(1−pi,t\)p\_\{i,t\}^\{\\prime\}=\\frac\{\\partial p\_\{i,t\}\}\{\\partial\\mu\_\{i\}\}=p\_\{i,t\}\(1\-p\_\{i,t\}\)\. Define the per\-step log\-likelihoodℓi,t​\(μi\):=𝟏\{ut=ei\}​\[dti​log⁡\(pi,t\)\+\(1−dti\)​log⁡\(1−pi,t\)\]\\ell\_\{i,t\}\(\\mu\_\{i\}\):=\\mathbf\{1\}\_\{\\\{u\_\{t\}=e\_\{i\}\\\}\}\[d^\{i\}\_\{t\}\\log\(p\_\{i,t\}\)\+\(1\-d^\{i\}\_\{t\}\)\\log\(1\-p\_\{i,t\}\)\], the per\-step score functionsi,t​\(μi\):=∂∂μi​ℓi,t​\(μi\)=𝟏\{ut=ei\}​\[dti−pi,t\]s\_\{i,t\}\(\\mu\_\{i\}\):=\\frac\{\\partial\}\{\\partial\\mu\_\{i\}\}\\ell\_\{i,t\}\(\\mu\_\{i\}\)=\\mathbf\{1\}\_\{\\\{u\_\{t\}=e\_\{i\}\\\}\}\[d^\{i\}\_\{t\}\-p\_\{i,t\}\], and the per\-step second derivative of the log\-likelihoodhi,t​\(μi\):=∂2∂μi2​ℓi,t​\(μi\)=−𝟏\{ut=ei\}​pi,t​\(1−pi,t\)h\_\{i,t\}\(\\mu\_\{i\}\):=\\frac\{\\partial^\{2\}\}\{\\partial\\mu\_\{i\}^\{2\}\}\\ell\_\{i,t\}\(\\mu\_\{i\}\)=\-\\mathbf\{1\}\_\{\\\{u\_\{t\}=e\_\{i\}\\\}\}p\_\{i,t\}\(1\-p\_\{i,t\}\)\.Si​\(μi\):=∑t=1Tsi,t​\(μi\)S\_\{i\}\(\\mu\_\{i\}\):=\\sum\_\{t=1\}^\{T\}s\_\{i,t\}\(\\mu\_\{i\}\),Hi​\(μi\)=∑t=1Thi,t​\(μi\)H\_\{i\}\(\\mu\_\{i\}\)=\\sum\_\{t=1\}^\{T\}h\_\{i,t\}\(\\mu\_\{i\}\), andℓi​\(μi\)=∑t=1Tℓi,t​\(μi\)\\ell\_\{i\}\(\\mu\_\{i\}\)=\\sum\_\{t=1\}^\{T\}\\ell\_\{i,t\}\(\\mu\_\{i\}\)are the trajectory\-summed quantities\. Recall the MLE estimatorμ^i=arg⁡maxμi∈\[−μ¯,μ¯\]⁡ℓi​\(μi\)\\hat\{\\mu\}\_\{i\}=\\arg\\max\_\{\\mu\_\{i\}\\in\[\-\\bar\{\\mu\},\\bar\{\\mu\}\]\}\\ell\_\{i\}\(\\mu\_\{i\}\)\.Ni:=∑t=1T𝟏\{ut=ei\}N\_\{i\}:=\\sum\_\{t=1\}^\{T\}\\mathbf\{1\}\_\{\\\{u\_\{t\}=e\_\{i\}\\\}\}\. Lemmas[11\.1](https://arxiv.org/html/2605.24261#S11.Thmtheorem1)\-[11\.7](https://arxiv.org/html/2605.24261#S11.Thmtheorem7)assume Assumptions[3\.1](https://arxiv.org/html/2605.24261#S3.SS1)\-[3\.1](https://arxiv.org/html/2605.24261#S3.SS1)hold and actions come from an\(r,k\)\(r,k\)\-exploratory policy\.

###### Lemma 11\.1

\(Score function bound\)Letδ∈\(0,1\)\\delta\\in\(0,1\), then𝐏​\(\|Si​\(μi∗\)\|≤2​T​log⁡\(2/δ\)\)≥1−δ\\mathbf\{P\}\(\|S\_\{i\}\(\\mu\_\{i\}^\{\*\}\)\|\\leq\\sqrt\{2T\\log\\bigl\(\\nicefrac\{\{2\}\}\{\{\\delta\}\}\\bigr\)\}\)\\geq 1\-\\delta\.

###### Proof 11\.2

Proof of Lemma[11\.1](https://arxiv.org/html/2605.24261#S11.Thmtheorem1)Considerμi∈\[−μ¯,μ¯\]\\mu\_\{i\}\\in\[\-\\bar\{\\mu\},\\bar\{\\mu\}\]\. Letpi,t∗=σ​\(xt\+μi∗\)p\_\{i,t\}^\{\*\}=\\sigma\(x\_\{t\}\+\\mu\_\{i\}^\{\*\}\)\. Defineξt=si,t​\(μi∗\)\\xi\_\{t\}=s\_\{i,t\}\(\\mu\_\{i\}^\{\*\}\)and observe that\{ξt\}t=1T\\\{\\xi\_\{t\}\\\}\_\{t=1\}^\{T\}is anℱt\+1d\\mathcal\{F\}^\{d\}\_\{t\+1\}\-adapted martingale difference sequence sincext,ut,dti∈ℱt\+1dx\_\{t\},u\_\{t\},d^\{i\}\_\{t\}\\in\\mathcal\{F\}^\{d\}\_\{t\+1\}and𝐄​\[ξt\|ℱtd\]=0\\mathbf\{E\}\[\\xi\_\{t\}\|\\mathcal\{F\}^\{d\}\_\{t\}\]=0by \([1](https://arxiv.org/html/2605.24261#S3.E1)\) asξt\\xi\_\{t\}is defined usingμi∗\\mu\_\{i\}^\{\*\}\. Note thatξt∈\[−pi,t∗,1−pi,t∗\]\\xi\_\{t\}\\in\[\-p^\{\*\}\_\{i,t\},1\-p^\{\*\}\_\{i,t\}\]\(implying\|ξt\|≤1\|\\xi\_\{t\}\|\\leq 1\) and∑t=1Tξt=Si​\(μi∗\)\\sum\_\{t=1\}^\{T\}\\xi\_\{t\}=S\_\{i\}\(\\mu\_\{i\}^\{\*\}\)\. We apply Azuma\-Hoeffding\(Wainwright[2019](https://arxiv.org/html/2605.24261#bib.bib71)\):𝐏​\(\|Si​\(μi∗\)\|≥q\)≤2​exp⁡\(−q2/\(2​∑t=1T12\)\)\\mathbf\{P\}\(\|S\_\{i\}\(\\mu\_\{i\}^\{\*\}\)\|\\geq q\)\\leq 2\\exp\\left\(\-\\nicefrac\{\{q^\{2\}\}\}\{\{\(2\\sum\_\{t=1\}^\{T\}1^\{2\}\)\}\}\\right\)\. We set the RHS toδ\\delta, solve forqq, and the result follows\.\\halmos

###### Lemma 11\.3

\(Second derivative bound\)\|Hi​\(μi\)\|≥Ni​p¯2\|H\_\{i\}\(\\mu\_\{i\}\)\|\\geq N\_\{i\}\\underline\{p\}^\{2\}for allμi∈\[−μ¯,μ¯\]\\mu\_\{i\}\\in\[\-\\bar\{\\mu\},\\bar\{\\mu\}\]\.

###### Proof 11\.4

Proof of Lemma[11\.3](https://arxiv.org/html/2605.24261#S11.Thmtheorem3)\|Hi​\(μi\)\|=\|∑t=1Thi,t​\(μi\)\|=∑t=1T𝟏\{ut=ei\}​pi,t​\(1−pi,t\)\|H\_\{i\}\(\\mu\_\{i\}\)\|=\|\\sum\_\{t=1\}^\{T\}h\_\{i,t\}\(\\mu\_\{i\}\)\|=\\sum\_\{t=1\}^\{T\}\\mathbf\{1\}\_\{\\\{u\_\{t\}=e\_\{i\}\\\}\}p\_\{i,t\}\(1\-p\_\{i,t\}\)\. Remark[3\.2](https://arxiv.org/html/2605.24261#S3.Thmtheorem2)boundspi,tp\_\{i,t\}and1−pi,t1\-p\_\{i,t\}uniformly forμi∈\[−μ¯,μ¯\]\\mu\_\{i\}\\in\[\-\\bar\{\\mu\},\\bar\{\\mu\}\], giving\|Hi​\(μi\)\|≥Ni​p¯​\(1−p¯\)=Ni​p¯2\|H\_\{i\}\(\\mu\_\{i\}\)\|\\geq N\_\{i\}\\underline\{p\}\(1\-\\overline\{p\}\)=N\_\{i\}\\underline\{p\}^\{2\}\.\\halmos

###### Lemma 11\.5

\(μi\\mu\_\{i\}bound\)Letδ∈\(0,1\)\\delta\\in\(0,1\)\. Then𝐏​\(\|μ^i−μi∗\|≤2​T​log⁡\(2δ\)Ni​p¯2\)≥1−δ\\mathbf\{P\}\\bigl\(\|\\hat\{\\mu\}\_\{i\}\-\\mu\_\{i\}^\{\*\}\|\\leq\\frac\{\\sqrt\{2T\\log\(\\frac\{2\}\{\\delta\}\)\}\}\{N\_\{i\}\\underline\{p\}^\{2\}\}\\bigr\)\\geq 1\-\\delta\.

###### Proof 11\.6

Proof of Lemma[11\.5](https://arxiv.org/html/2605.24261#S11.Thmtheorem5)Si​\(μi\)S\_\{i\}\(\\mu\_\{i\}\)is a smooth function ofμi∈\[−μ¯,μ¯\]\\mu\_\{i\}\\in\[\-\\bar\{\\mu\},\\bar\{\\mu\}\], so by the mean value theorem, there exists someμ~\\tilde\{\\mu\}betweenμ^i\\hat\{\\mu\}\_\{i\}andμi∗\\mu^\{\*\}\_\{i\}such thatHi​\(μ~\)​\(μ^i−μi∗\)=Si​\(μ^i\)−Si​\(μi∗\)H\_\{i\}\(\\tilde\{\\mu\}\)\(\\hat\{\\mu\}\_\{i\}\-\\mu\_\{i\}^\{\*\}\)=S\_\{i\}\(\\hat\{\\mu\}\_\{i\}\)\-S\_\{i\}\(\\mu^\{\*\}\_\{i\}\)\. The first order optimality condition forμ^i∈\[−μ¯,μ¯\]\\hat\{\\mu\}\_\{i\}\\in\[\-\\bar\{\\mu\},\\bar\{\\mu\}\]givesSi​\(μ^i\)​\(μ^i−μi∗\)≥0S\_\{i\}\(\\hat\{\\mu\}\_\{i\}\)\(\\hat\{\\mu\}\_\{i\}\-\\mu\_\{i\}^\{\*\}\)\\geq 0\(i\.e\., the inequality is tight forμ^i∈\(−μ¯,μ¯\)\\hat\{\\mu\}\_\{i\}\\in\(\-\\bar\{\\mu\},\\bar\{\\mu\}\)and the terms have the same signs at the boundaries\)\. Multiplying the MVT expression by\(μ^i−μi∗\)\(\\hat\{\\mu\}\_\{i\}\-\\mu\_\{i\}^\{\*\}\)and droppingSi​\(μ^i\)​\(μ^i−μi∗\)≥0S\_\{i\}\(\\hat\{\\mu\}\_\{i\}\)\(\\hat\{\\mu\}\_\{i\}\-\\mu\_\{i\}^\{\*\}\)\\geq 0givesHi​\(μ~\)​\(μ^i−μi∗\)2≥−Si​\(μi∗\)​\(μ^i−μi∗\)H\_\{i\}\(\\tilde\{\\mu\}\)\(\\hat\{\\mu\}\_\{i\}\-\\mu\_\{i\}^\{\*\}\)^\{2\}\\geq\-S\_\{i\}\(\\mu\_\{i\}^\{\*\}\)\(\\hat\{\\mu\}\_\{i\}\-\\mu\_\{i\}^\{\*\}\)\.Hi≤0H\_\{i\}\\leq 0by definition so−Hi​\(μ~\)=\|Hi​\(μ~\)\|\-H\_\{i\}\(\\tilde\{\\mu\}\)=\|H\_\{i\}\(\\tilde\{\\mu\}\)\|, implying\|Hi​\(μ~\)\|​\|μ^i−μi∗\|2≤\|Si​\(μi∗\)\|​\|μ^i−μi∗\|\|H\_\{i\}\(\\tilde\{\\mu\}\)\|\|\\hat\{\\mu\}\_\{i\}\-\\mu\_\{i\}^\{\*\}\|^\{2\}\\leq\|S\_\{i\}\(\\mu\_\{i\}^\{\*\}\)\|\|\\hat\{\\mu\}\_\{i\}\-\\mu\_\{i\}^\{\*\}\|\. Ifμ^i≠μi∗\\hat\{\\mu\}\_\{i\}\\neq\\mu\_\{i\}^\{\*\}, then simplifying gives\|μ^i−μi∗\|≤\|Si​\(μi∗\)\|\|Hi​\(μ~\)\|\|\\hat\{\\mu\}\_\{i\}\-\\mu^\{\*\}\_\{i\}\|\\leq\\frac\{\|S\_\{i\}\(\\mu\_\{i\}^\{\*\}\)\|\}\{\|H\_\{i\}\(\\tilde\{\\mu\}\)\|\}\(the same holds trivially forμ^i=μi∗\\hat\{\\mu\}\_\{i\}=\\mu\_\{i\}^\{\*\}as the RHS is non\-negative\)\. The numerator is upper bounded by Lemma[11\.1](https://arxiv.org/html/2605.24261#S11.Thmtheorem1)with probability at least1−δ1\-\\deltaand the denominator is lower bounded by Lemma[11\.3](https://arxiv.org/html/2605.24261#S11.Thmtheorem3), so the lemma follows by substitution\.\\halmos

###### Lemma 11\.7

\(ℰN\\mathcal\{E\}\_\{N\}bound\)LetℰN​\(r,T,k\)\\mathcal\{E\}\_\{N\}\(r,T,k\)be the event that\{Ni\>r​\(T−k\)2​\(M\+1\)​for all​i=1,…,M\}\\\{N\_\{i\}\>\\frac\{r\(T\-k\)\}\{2\(M\+1\)\}\\text\{ for all \}i=1,\\dots,M\\\}\. Then forδ∈\(0,1\)\\delta\\in\(0,1\):𝐏​\(ℰN​\(r,T​\(δ\),k\)\)≥1−δ\\mathbf\{P\}\(\\mathcal\{E\}\_\{N\}\(r,T\(\\delta\),k\)\)\\geq 1\-\\deltawhenT​\(δ\)≥k\+8/r​\(M\+1\)​log⁡\(M/δ\)T\(\\delta\)\\geq k\+\\nicefrac\{\{8\}\}\{\{r\}\}\(M\+1\)\\log\(\\nicefrac\{\{M\}\}\{\{\\delta\}\}\)\.

###### Proof 11\.8

Proof of Lemma[11\.7](https://arxiv.org/html/2605.24261#S11.Thmtheorem7)Consideri∈\{1,…,M\}i\\in\\\{1,\\dots,M\\\}\. The\(r,k\)\(r,k\)policy a\.s\. takes at leastN:=⌊Tk⌋​⌈r​k⌉≥r​\(T−k\)N:=\\lfloor\\frac\{T\}\{k\}\\rfloor\\lceil rk\\rceil\\geq r\(T\-k\)exploratory steps in the trajectory\. LetN~i\\tilde\{N\}\_\{i\}be the count of steps whereut=eiu\_\{t\}=e\_\{i\}in the firstNNexploratory steps\. By the\(r,k\)\(r,k\)policy’s uniformly random exploration,N~i∼Bin​\(N,1M\+1\)\\tilde\{N\}\_\{i\}\\sim\\text\{Bin\}\(N,\\frac\{1\}\{M\+1\}\), so by the multiplicative form of the Chernoff bound for sums of Bernoulli random variables\(Gerbessiotis[2025](https://arxiv.org/html/2605.24261#bib.bib28)\), we have𝐏​\(N~i≤N2​\(M\+1\)\)≤exp⁡\{−N8​\(M\+1\)\}≤exp⁡\{−r​\(T−k\)8​\(M\+1\)\}\\mathbf\{P\}\(\\tilde\{N\}\_\{i\}\\leq\\frac\{N\}\{2\(M\+1\)\}\)\\leq\\exp\\\{\-\\frac\{N\}\{8\(M\+1\)\}\\\}\\leq\\exp\\\{\-\\frac\{r\(T\-k\)\}\{8\(M\+1\)\}\\\}\. AsNi≥N~iN\_\{i\}\\geq\\tilde\{N\}\_\{i\}by construction andN≥r​\(T−k\)N\\geq r\(T\-k\)a\.s\., this implies\{Ni≤r​\(T−k\)2​\(M\+1\)\}⊆\{N~i≤N2​\(M\+1\)\}\\\{N\_\{i\}\\leq\\frac\{r\(T\-k\)\}\{2\(M\+1\)\}\\\}\\subseteq\\\{\\tilde\{N\}\_\{i\}\\leq\\frac\{N\}\{2\(M\+1\)\}\\\}and thus𝐏​\(Ni≤r​\(T−k\)2​\(M\+1\)\)≤exp⁡\{−r​\(T−k\)8​\(M\+1\)\}\\mathbf\{P\}\(N\_\{i\}\\leq\\frac\{r\(T\-k\)\}\{2\(M\+1\)\}\)\\leq\\exp\\\{\-\\frac\{r\(T\-k\)\}\{8\(M\+1\)\}\\\}\. Applying the union bound yields𝐏​\(ℰN​\(r,T,k\)\)≥1−M​exp⁡\{−r​\(T−k\)8​\(M\+1\)\}\\mathbf\{P\}\(\\mathcal\{E\}\_\{N\}\(r,T,k\)\)\\geq 1\-M\\exp\\\{\-\\frac\{r\(T\-k\)\}\{8\(M\+1\)\}\\\}\. Setting the RHS equal toδ\\deltaand solving forT​\(δ\)T\(\\delta\)completes the proof\.\\halmos

## 12System identification result proof

###### Proof 12\.1

Proof of Theorem[4\.6](https://arxiv.org/html/2605.24261#S4.Thmtheorem6)We analyze the OLS and MLE results separately \(allocatingδ/2\\nicefrac\{\{\\delta\}\}\{\{2\}\}to each\) and then apply a union bound\. Letδ1=δ/6\\delta\_\{1\}=\\nicefrac\{\{\\delta\}\}\{\{6\}\}\(used for the OLS portion\),δ2=δ/4\\delta\_\{2\}=\\nicefrac\{\{\\delta\}\}\{\{4\}\}\(used forℰN\\mathcal\{E\}\_\{N\}\), andδ3=δ/\(4​M\)\\delta\_\{3\}=\\nicefrac\{\{\\delta\}\}\{\{\(4M\)\}\}\(used for the MLE portion\)\. We evaluate the preconditions in Theorem[4\.2](https://arxiv.org/html/2605.24261#S4.Thmtheorem2)using\(Zt,Yt\)t≥1=\(zt,xt\+1\)t≥1\(Z\_\{t\},Y\_\{t\}\)\_\{t\\geq 1\}=\(z\_\{t\},x\_\{t\+1\}\)\_\{t\\geq 1\}, filtration\{ℱtw\}t≥0\\\{\\mathcal\{F\}^\{w\}\_\{t\}\\\}\_\{t\\geq 0\}, andηt=wt\\eta\_\{t\}=w\_\{t\}\. Precondition \(a\) is satisfied trivially under Assumption[3\.1](https://arxiv.org/html/2605.24261#S3.SS1)forxt\+1=θ∗⊤​zt\+wtx\_\{t\+1\}=\\theta^\{\*\\top\}z\_\{t\}\+w\_\{t\}\. By Proposition[10\.7](https://arxiv.org/html/2605.24261#S10.Thmtheorem7), for the given\(r,k,τ\)\(r,k,\\tau\), the system satisfies the\(k,1−τ28​M​I2​M\+1,min⁡\{p1​\(τ\),r​p¯M\+1\}\)\(k,\\frac\{1\-\\tau^\{2\}\}\{8M\}I\_\{2M\+1\},\\min\\\{p\_\{1\}\(\\tau\),\\frac\{r\\underline\{p\}\}\{M\+1\}\\\}\)\-BMSB condition, establishing precondition \(b\)\. The system satisfies precondition \(c\) by Proposition[4\.5](https://arxiv.org/html/2605.24261#S4.Thmtheorem5)withΓ¯=\(Cx2\+2\)​I2​M\+1\\bar\{\\Gamma\}=\(C\_\{x\}^\{2\}\+2\)I\_\{2M\+1\}\. The OLS result then follows by substituting the system\-specific quantities into Theorem[4\.2](https://arxiv.org/html/2605.24261#S4.Thmtheorem2)\(usingδ1\\delta\_\{1\}gives an OLS\-related failure probability of no more thanδ/2\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\. Theorem[4\.2](https://arxiv.org/html/2605.24261#S4.Thmtheorem2)uses a factor of3​δ3\\delta, but as our precondition \(c\) is deterministic this could be refined to2​δ2\\delta\(omitted for brevity\)\. For the MLE result, we first have that𝐏​\(ℰN​\(r,T​\(δ2\),k\)\)≥1−δ2\\mathbf\{P\}\(\\mathcal\{E\}\_\{N\}\(r,T\(\\delta\_\{2\}\),k\)\)\\geq 1\-\\delta\_\{2\}whenT≥k\+8​\(M\+1\)/r​log⁡\(4​M/δ\)T\\geq k\+\\nicefrac\{\{8\(M\+1\)\}\}\{\{r\}\}\\log\(\\nicefrac\{\{4M\}\}\{\{\\delta\}\}\)by Lemma[11\.7](https://arxiv.org/html/2605.24261#S11.Thmtheorem7)\. By Lemma[11\.5](https://arxiv.org/html/2605.24261#S11.Thmtheorem5), for indexi∈\{1,…,M\}i\\in\\\{1,\\dots,M\\\}andδ3\\delta\_\{3\}, we have𝐏​\(\|μ^i−μi∗\|≤2​T​log⁡\(8​M/δ\)Ni​p¯2\)≥1−δ3\\mathbf\{P\}\(\|\\hat\{\\mu\}\_\{i\}\-\\mu\_\{i\}^\{\*\}\|\\leq\\frac\{\\sqrt\{2T\\log\(\\nicefrac\{\{8M\}\}\{\{\\delta\}\}\)\}\}\{N\_\{i\}\\underline\{p\}^\{2\}\}\)\\geq 1\-\\delta\_\{3\}\. OnℰN​\(r,T​\(δ2\),k\)\\mathcal\{E\}\_\{N\}\(r,T\(\\delta\_\{2\}\),k\),Ni\>r​\(T−k\)2​\(M\+1\)N\_\{i\}\>\\frac\{r\(T\-k\)\}\{2\(M\+1\)\}for alli=1,…,Mi=1,\\dots,M, meaning\|μ^i−μi∗\|≤2​\(M\+1\)p¯2​2​T​log⁡\(8​M/δ\)r​\(T−k\)\|\\hat\{\\mu\}\_\{i\}\-\\mu\_\{i\}^\{\*\}\|\\leq\\frac\{2\(M\+1\)\}\{\\underline\{p\}^\{2\}\}\\frac\{\\sqrt\{2T\\log\(\\nicefrac\{\{8M\}\}\{\{\\delta\}\}\)\}\}\{r\(T\-k\)\}\. Applying the union bound ensures this holds for alli=1,…,Mi=1,\\dots,Mwith probability at least1−δ/41\-\\nicefrac\{\{\\delta\}\}\{\{4\}\}\. For bound clarity, the assumedT≥2​kT\\geq 2kallows lower boundingT−kT\-kbyT/2\\nicefrac\{\{T\}\}\{\{2\}\}, giving the upper bound4​\(M\+1\)p¯2​2​log⁡\(8​M/δ\)r​T\\frac\{4\(M\+1\)\}\{\\underline\{p\}^\{2\}\}\\frac\{\\sqrt\{2\\log\(\\nicefrac\{\{8M\}\}\{\{\\delta\}\}\)\}\}\{r\\sqrt\{T\}\}used in the result\. A union bound ensures the MLE result holds with probability at least1−δ/21\-\\nicefrac\{\{\\delta\}\}\{\{2\}\}\. The complete result then holds by an additional union bound and taking the maximum of the burn\-in times\.\\halmos

## 13Regret analysis supporting results and proofs

### 13\.1Confidence set results

###### Theorem 13\.1

\(Abbasi\-Yadkoriet al\.\([2011](https://arxiv.org/html/2605.24261#bib.bib1)\)Theorem 2\) Let\{ℱt\}t=0∞\\\{\\mathcal\{F\}\_\{t\}\\\}\_\{t=0\}^\{\\infty\}be a filtration\. Let\{ηt\}t=1∞\\\{\\eta\_\{t\}\\\}\_\{t=1\}^\{\\infty\}be anℱt\\mathcal\{F\}\_\{t\}\-adapted random process taking value inℝ\\mathbb\{R\}, withηt\|ℱt−1\\eta\_\{t\}\|\\mathcal\{F\}\_\{t\-1\}RR\-subgaussian\. Let\{zt\}t=1∞\\\{z\_\{t\}\\\}\_\{t=1\}^\{\\infty\}be anℝd\\mathbb\{R\}^\{d\}\-valued stochastic process such thatztz\_\{t\}isℱt−1\\mathcal\{F\}\_\{t\-1\}measurable\. Letλ\>0\\lambda\>0and defineyt=⟨zt,θ∗⟩\+ηty\_\{t\}=\\langle z\_\{t\},\\theta\_\{\*\}\\rangle\+\\eta\_\{t\}\. If‖θ∗‖2≤S\\\|\\theta\_\{\*\}\\\|\_\{2\}\\leq Sand, for allt≥1t\\geq 1,‖zt‖2≤Z¯\\\|z\_\{t\}\\\|\_\{2\}\\leq\\bar\{Z\}, then with probability at least1−δ1\-\\deltafor allt≥0t\\geq 0,θ∗\\theta^\{\*\}lies in the following set𝒞t​\(δ\)=\{θ∈ℝd:‖θ^tR​L​S−θ‖V¯t≤R​d​log⁡\(1\+t​Z¯2/λδ\)\+λ​S\}\\mathcal\{C\}\_\{t\}\(\\delta\)=\\\{\\theta\\in\\mathbb\{R\}^\{d\}:\\\|\\hat\{\\theta\}^\{RLS\}\_\{t\}\-\\theta\\\|\_\{\\overline\{V\}\_\{t\}\}\\leq R\\sqrt\{d\\log\(\\frac\{1\+t\\bar\{Z\}^\{2\}/\\lambda\}\{\\delta\}\)\}\+\\sqrt\{\\lambda\}S\\\}\.

###### Proof 13\.2

Proof of Proposition[5\.1](https://arxiv.org/html/2605.24261#S5.Thmtheorem1)We construct a confidence set centered onθ^tR​L​S\\hat\{\\theta\}\_\{t\}^\{RLS\}then re\-center toθ^t\\hat\{\\theta\}\_\{t\}\. Letyt=xty\_\{t\}=x\_\{t\},zt=\[xt−1​ut−1⊤​dt−1⊤\]⊤z\_\{t\}=\[x\_\{t\-1\}\\ u\_\{t\-1\}^\{\\top\}\\ d\_\{t\-1\}^\{\\top\}\]^\{\\top\}, andηt=wt−1\\eta\_\{t\}=w\_\{t\-1\}, matchingyt=⟨zt,θ∗⟩\+ηty\_\{t\}=\\langle z\_\{t\},\\theta^\{\*\}\\rangle\+\\eta\_\{t\}in Theorem[13\.1](https://arxiv.org/html/2605.24261#S13.Thmtheorem1)\. Recall the filtrationℱtw\\mathcal\{F\}^\{w\}\_\{t\}, containingxt,ut,dtx\_\{t\},u\_\{t\},d\_\{t\}but notwtw\_\{t\}\. Thusηt\\eta\_\{t\}isℱtw\\mathcal\{F\}^\{w\}\_\{t\}\-measurable and isℱt−1w\\mathcal\{F\}^\{w\}\_\{t\-1\}conditionallyσs2\\sigma\_\{s\}^\{2\}\-subgaussian by Assumption[3\.1](https://arxiv.org/html/2605.24261#S3.SS1)\.\{zt\}t=1∞\\\{z\_\{t\}\\\}^\{\\infty\}\_\{t=1\}is aℱt−1w\\mathcal\{F\}^\{w\}\_\{t\-1\}\-adaptedℝ2​M\+1\\mathbb\{R\}^\{2M\+1\}valued stochastic process\.‖zt‖2≤Cx2\+2\\\|z\_\{t\}\\\|\_\{2\}\\leq\\sqrt\{C\_\{x\}^\{2\}\+2\}fort∈ℕt\\in\\mathbb\{N\}\(Prop\.[3\.1](https://arxiv.org/html/2605.24261#S3.Thmtheorem1)\) andmaxθ∈Θ⁡‖θ‖2≤a¯2\+M​b¯2\+M​c¯2\\max\_\{\\theta\\in\\Theta\}\\\|\\theta\\\|\_\{2\}\\leq\\sqrt\{\\bar\{a\}^\{2\}\+M\\bar\{b\}^\{2\}\+M\\bar\{c\}^\{2\}\}\(Remark[10\.2](https://arxiv.org/html/2605.24261#S10.Thmtheorem2)\)\. Thus our system satisfies the conditions of Theorem[13\.1](https://arxiv.org/html/2605.24261#S13.Thmtheorem1)\. Letδ∈\(0,1\)\\delta\\in\(0,1\)\. Substituting system\-specific quantities implies𝐏​\(ℰθR​L​S​\(δ\)\)≥1−δ\\mathbf\{P\}\(\\mathcal\{E\}\_\{\\theta\}^\{RLS\}\(\\delta\)\)\\geq 1\-\\deltaforℰθR​L​S​\(δ\):=\{θ∗∈𝒞tθ,R​L​S​\(δ\),t≥0\}\\mathcal\{E\}\_\{\\theta\}^\{RLS\}\(\\delta\):=\\\{\\theta^\{\*\}\\in\\mathcal\{C\}\_\{t\}^\{\\theta,RLS\}\(\\delta\),\\;t\\geq 0\\\}and𝒞tθ,R​L​S​\(δ\):=\{θ∈ℝ2​M\+1:‖θ^tR​L​S−θ‖V¯t≤αtθ​\(δ\)\}\\mathcal\{C\}\_\{t\}^\{\\theta,RLS\}\(\\delta\):=\\big\\\{\\theta\\in\\mathbb\{R\}^\{2M\+1\}:\\\|\\hat\{\\theta\}^\{RLS\}\_\{t\}\-\\theta\\\|\_\{\\overline\{V\}\_\{t\}\}\\leq\\alpha^\{\\theta\}\_\{t\}\(\\delta\)\\big\\\}\. By Lemma[13\.3](https://arxiv.org/html/2605.24261#S13.Thmtheorem3), ifℰθR​L​S​\(δ\)\\mathcal\{E\}\_\{\\theta\}^\{RLS\}\(\\delta\)holds then‖θ^t−θ∗‖V¯t≤αtθ​\(δ\)\\\|\\hat\{\\theta\}\_\{t\}\-\\theta^\{\*\}\\\|\_\{\\bar\{V\}\_\{t\}\}\\leq\\alpha\_\{t\}^\{\\theta\}\(\\delta\)fort≥1t\\geq 1\. Thus the proposition follows for the confidence set centered atθ^t\\hat\{\\theta\}\_\{t\}\.\\halmos

###### Lemma 13\.3

Let eventℰθR​L​S​\(δ\)\\mathcal\{E\}^\{RLS\}\_\{\\theta\}\(\\delta\)hold\. Then‖θ^t−θ∗‖V¯t≤‖θ^tR​L​S−θ∗‖V¯t≤αtθ​\(δ\)\\\|\\hat\{\\theta\}\_\{t\}\-\\theta^\{\*\}\\\|\_\{\\bar\{V\}\_\{t\}\}\\leq\\\|\\hat\{\\theta\}\_\{t\}^\{RLS\}\-\\theta^\{\*\}\\\|\_\{\\bar\{V\}\_\{t\}\}\\leq\\alpha\_\{t\}^\{\\theta\}\(\\delta\)fort≥1t\\geq 1\.

###### Proof 13\.4

Proof of Lemma[13\.3](https://arxiv.org/html/2605.24261#S13.Thmtheorem3)Lett≥1t\\geq 1, define⟨a,b⟩V¯t:=a⊤​V¯t​b\\langle a,b\\rangle\_\{\\bar\{V\}\_\{t\}\}:=a^\{\\top\}\\bar\{V\}\_\{t\}band note‖a‖V¯t2=⟨a,a⟩V¯t\\\|a\\\|\_\{\\bar\{V\}\_\{t\}\}^\{2\}=\\langle a,a\\rangle\_\{\\bar\{V\}\_\{t\}\}\.θ^t\\hat\{\\theta\}\_\{t\}andθ∗∈Θ\\theta^\{\*\}\\in\\Thetaby construction\. We consider their convex combinationθ~:=θ^t\+q​\(θ∗−θ^t\)\\tilde\{\\theta\}:=\\hat\{\\theta\}\_\{t\}\+q\(\\theta^\{\*\}\-\\hat\{\\theta\}\_\{t\}\)forq∈\(0,1\]q\\in\(0,1\]\.Θ\\Thetais a convex set soθ~∈Θ\\tilde\{\\theta\}\\in\\Theta\. Sinceθ^t=arg⁡minθ∈Θ⁡‖θ−θ^tR​L​S‖V¯t2\\hat\{\\theta\}\_\{t\}=\\arg\\min\_\{\\theta\\in\\Theta\}\\\|\\theta\-\\hat\{\\theta\}^\{RLS\}\_\{t\}\\\|\_\{\\bar\{V\}\_\{t\}\}^\{2\}, we have that‖θ~−θ^tR​L​S‖V¯t2≥‖θ^t−θ^tR​L​S‖V¯t2\\\|\\tilde\{\\theta\}\-\\hat\{\\theta\}\_\{t\}^\{RLS\}\\\|\_\{\\bar\{V\}\_\{t\}\}^\{2\}\\geq\\\|\\hat\{\\theta\}\_\{t\}\-\\hat\{\\theta\}\_\{t\}^\{RLS\}\\\|\_\{\\bar\{V\}\_\{t\}\}^\{2\}\. By expanding the LHS norm we can simplify this inequality toq2​‖θ∗−θ^t‖V¯t2\+2​q​⟨θ^t−θ^tR​L​S,θ∗−θ^t⟩V¯t≥0q^\{2\}\\\|\\theta^\{\*\}\-\\hat\{\\theta\}\_\{t\}\\\|^\{2\}\_\{\\bar\{V\}\_\{t\}\}\+2q\\langle\\hat\{\\theta\}\_\{t\}\-\\hat\{\\theta\}\_\{t\}^\{RLS\},\\theta^\{\*\}\-\\hat\{\\theta\}\_\{t\}\\rangle\_\{\\bar\{V\}\_\{t\}\}\\geq 0\. Dividing byqqand takinglimq→0\+\\lim\_\{q\\rightarrow 0^\{\+\}\}we obtain the inequality⟨θ^tR​L​S−θ^t,θ^t−θ∗⟩V¯t≥0\\langle\\hat\{\\theta\}\_\{t\}^\{RLS\}\-\\hat\{\\theta\}\_\{t\},\\hat\{\\theta\}\_\{t\}\-\\theta^\{\*\}\\rangle\_\{\\bar\{V\}\_\{t\}\}\\geq 0\. Next we expand the quantity‖θ^tR​L​S−θ∗‖V¯t2=‖\(θ^tR​L​S−θ^t\)\+\(θ^t−θ∗\)‖V¯t2=‖θ^tR​L​S−θ^t‖V¯t2\+‖θ^t−θ∗‖V¯t2\+2​⟨θ^tR​L​S−θ^t,θ^t−θ∗⟩V¯t\\\|\\hat\{\\theta\}\_\{t\}^\{RLS\}\-\\theta^\{\*\}\\\|\_\{\\bar\{V\}\_\{t\}\}^\{2\}=\\\|\(\\hat\{\\theta\}\_\{t\}^\{RLS\}\-\\hat\{\\theta\}\_\{t\}\)\+\(\\hat\{\\theta\}\_\{t\}\-\\theta^\{\*\}\)\\\|\_\{\\bar\{V\}\_\{t\}\}^\{2\}=\\\|\\hat\{\\theta\}\_\{t\}^\{RLS\}\-\\hat\{\\theta\}\_\{t\}\\\|\_\{\\bar\{V\}\_\{t\}\}^\{2\}\+\\\|\\hat\{\\theta\}\_\{t\}\-\\theta^\{\*\}\\\|\_\{\\bar\{V\}\_\{t\}\}^\{2\}\+2\\langle\\hat\{\\theta\}\_\{t\}^\{RLS\}\-\\hat\{\\theta\}\_\{t\},\\hat\{\\theta\}\_\{t\}\-\\theta^\{\*\}\\rangle\_\{\\bar\{V\}\_\{t\}\}\. The first and third terms are nonnegative, implying‖θ^t−θ∗‖V¯t2≤‖θ^tR​L​S−θ∗‖V¯t2\\\|\\hat\{\\theta\}\_\{t\}\-\\theta^\{\*\}\\\|^\{2\}\_\{\\bar\{V\}\_\{t\}\}\\leq\\\|\\hat\{\\theta\}\_\{t\}^\{RLS\}\-\\theta^\{\*\}\\\|^\{2\}\_\{\\bar\{V\}\_\{t\}\}\. We assumeℰθR​L​S​\(δ\)\\mathcal\{E\}\_\{\\theta\}^\{RLS\}\(\\delta\)holds, so‖θ^t−θ∗‖V¯t≤‖θ^tR​L​S−θ∗‖V¯t≤αtθ​\(δ\)\\\|\\hat\{\\theta\}\_\{t\}\-\\theta^\{\*\}\\\|\_\{\\bar\{V\}\_\{t\}\}\\leq\\\|\\hat\{\\theta\}\_\{t\}^\{RLS\}\-\\theta^\{\*\}\\\|\_\{\\bar\{V\}\_\{t\}\}\\leq\\alpha^\{\\theta\}\_\{t\}\(\\delta\)\.\\halmos

###### Theorem 13\.5

\(Scalar form ofFauryet al\.\([2020](https://arxiv.org/html/2605.24261#bib.bib25)\)Theorem 1\)Let\{ℱt\}t=1∞\\\{\\mathcal\{F\}\_\{t\}\\\}\_\{t=1\}^\{\\infty\}be a filtration\. Let\{xt\}t=1∞\\\{x\_\{t\}\\\}\_\{t=1\}^\{\\infty\}be a stochastic process in\[−1,1\]\[\-1,1\]such thatxtx\_\{t\}isℱt\\mathcal\{F\}\_\{t\}measurable\. Let\{ϵt\}t=1∞\\\{\\epsilon\_\{t\}\\\}\_\{t=1\}^\{\\infty\}be a martingale difference sequence such thatϵt\\epsilon\_\{t\}isℱt\+1\\mathcal\{F\}\_\{t\+1\}measurable\. Assume that conditionally onℱt\\mathcal\{F\}\_\{t\}we have\|ϵt\|≤1\|\\epsilon\_\{t\}\|\\leq 1almost surely, and noteσt2:=𝐄​\[ϵt2\|ℱt\]\\sigma^\{2\}\_\{t\}:=\\mathbf\{E\}\[\\epsilon\_\{t\}^\{2\}\|\\mathcal\{F\}\_\{t\}\]\. Letλ\>0\\lambda\>0and for anyt≥1t\\geq 1defineHt:=∑s=1t−1σs2​xs2\+λH\_\{t\}:=\\sum\_\{s=1\}^\{t\-1\}\\sigma\_\{s\}^\{2\}x\_\{s\}^\{2\}\+\\lambdaandSt:=∑s=1t−1ϵs​xsS\_\{t\}:=\\sum\_\{s=1\}^\{t\-1\}\\epsilon\_\{s\}x\_\{s\}\. Then for anyδ∈\(0,1\]\\delta\\in\(0,1\]:𝐏​\(∃t≥1,\|St\|Ht≥λ2\+2λ​log⁡\(2​Htδ​λ\)\)≤δ\\mathbf\{P\}\(\\exists t\\geq 1,\\;\\frac\{\|S\_\{t\}\|\}\{\\sqrt\{H\_\{t\}\}\}\\geq\\frac\{\\sqrt\{\\lambda\}\}\{2\}\+\\frac\{2\}\{\\sqrt\{\\lambda\}\}\\log\(\\frac\{2\\sqrt\{H\_\{t\}\}\}\{\\delta\\sqrt\{\\lambda\}\}\)\)\\leq\\delta\.

###### Proof 13\.6

Proof of Proposition[5\.2](https://arxiv.org/html/2605.24261#S5.Thmtheorem2)We focus on arbitrary actioniiand then apply the union bound\. First, we adapt Theorem[13\.5](https://arxiv.org/html/2605.24261#S13.Thmtheorem5)into a bound in terms of the regularized score functiongi,t​\(μi\):=∂ℓi,t​\(μi\)∂μi=−λ2​μi\+∑s=1t−1𝟏\{us=ei\}​\[dsi−σ​\(xs\+μi\)\]g\_\{i,t\}\(\\mu\_\{i\}\):=\\frac\{\\partial\\ell\_\{i,t\}\(\\mu\_\{i\}\)\}\{\\partial\\mu\_\{i\}\}=\-\\lambda\_\{2\}\\mu\_\{i\}\+\\sum\_\{s=1\}^\{t\-1\}\\mathbf\{1\}\_\{\\\{u\_\{s\}=e\_\{i\}\\\}\}\[d^\{i\}\_\{s\}\-\\sigma\(x\_\{s\}\+\\mu\_\{i\}\)\], whereℓi,t​\(μi\)\\ell\_\{i,t\}\(\\mu\_\{i\}\)is the regularized log likelihood forμi\\mu\_\{i\}on the trajectory1,…,t−11,\\dots,t\-1, as given in \([5](https://arxiv.org/html/2605.24261#S5.E5)\)\. Then we convert this result into a confidence set forμi\\mu\_\{i\}\. We show how our estimation problem maps to Theorem[13\.5](https://arxiv.org/html/2605.24261#S13.Thmtheorem5)\. Recall the filtrationℱtd\\mathcal\{F\}^\{d\}\_\{t\}\(containingxt,utx\_\{t\},u\_\{t\}but notdtd\_\{t\}\)\. Define a stochastic process\{qt\}t=1∞\\\{q\_\{t\}\\\}\_\{t=1\}^\{\\infty\}withqt=1​∀tq\_\{t\}=1\\;\\forall t\.qt∈\[−1,1\]q\_\{t\}\\in\[\-1,1\]and isℱtd\\mathcal\{F\}^\{d\}\_\{t\}\-measurable as it is deterministic\.qtq\_\{t\}isxtx\_\{t\}in Theorem[13\.5](https://arxiv.org/html/2605.24261#S13.Thmtheorem5)’s notation, which we set to11as we are estimating a shift parameter, not a scale parameter\. Define\{ϵt\}t=1∞\\\{\\epsilon\_\{t\}\\\}\_\{t=1\}^\{\\infty\}whereϵt:=𝟏\{ut=ei\}​\[dti−σ​\(xt\+μi∗⋅qt\)\]\\epsilon\_\{t\}:=\\mathbf\{1\}\_\{\\\{u\_\{t\}=e\_\{i\}\\\}\}\[d^\{i\}\_\{t\}\-\\sigma\(x\_\{t\}\+\\mu^\{\*\}\_\{i\}\\cdot q\_\{t\}\)\]\. Note that this is a martingale difference sequence that isℱt\+1d\\mathcal\{F\}^\{d\}\_\{t\+1\}\-measurable\.\|ϵt\|≤1\|\\epsilon\_\{t\}\|\\leq 1by the model assumptions\. Note that𝐄​\[ϵt2\|ℱtd\]=𝟏\{ut=ei\}​\[σ′​\(xt\+μi∗\)\]\\mathbf\{E\}\[\\epsilon\_\{t\}^\{2\}\|\\mathcal\{F\}^\{d\}\_\{t\}\]=\\mathbf\{1\}\_\{\\\{u\_\{t\}=e\_\{i\}\\\}\}\[\\sigma^\{\\prime\}\(x\_\{t\}\+\\mu\_\{i\}^\{\*\}\)\]\. FollowingFauryet al\.\([2020](https://arxiv.org/html/2605.24261#bib.bib25)\), letSt:=∑s=1t−1ϵs​qs=∑s=1t−1𝟏\{us=ei\}​\[dsi−σ​\(xs\+μi∗\)\]S\_\{t\}:=\\sum\_\{s=1\}^\{t\-1\}\\epsilon\_\{s\}q\_\{s\}=\\sum\_\{s=1\}^\{t\-1\}\\mathbf\{1\}\_\{\\\{u\_\{s\}=e\_\{i\}\\\}\}\[d^\{i\}\_\{s\}\-\\sigma\(x\_\{s\}\+\\mu\_\{i\}^\{\*\}\)\]andHt:=λ2\+∑s=1t−1𝐄​\[ϵs2\|ℱsd\]=λ2\+∑s=1t−1𝟏\{us=ei\}​σ′​\(xs\+μi∗\)H\_\{t\}:=\\lambda\_\{2\}\+\\sum\_\{s=1\}^\{t\-1\}\\mathbf\{E\}\[\\epsilon^\{2\}\_\{s\}\|\\mathcal\{F\}^\{d\}\_\{s\}\]=\\lambda\_\{2\}\+\\sum\_\{s=1\}^\{t\-1\}\\mathbf\{1\}\_\{\\\{u\_\{s\}=e\_\{i\}\\\}\}\\sigma^\{\\prime\}\(x\_\{s\}\+\\mu\_\{i\}^\{\*\}\)\. These are constructed to connect to our system asSt=gi,t​\(μi∗\)\+λ2​μi∗S\_\{t\}=g\_\{i,t\}\(\\mu\_\{i\}^\{\*\}\)\+\\lambda\_\{2\}\\mu\_\{i\}^\{\*\}andHt=−∂gi,t​\(μi\)∂μi\|μi=μi∗H\_\{t\}=\-\\frac\{\\partial g\_\{i,t\}\(\\mu\_\{i\}\)\}\{\\partial\\mu\_\{i\}\}\|\_\{\\mu\_\{i\}=\\mu\_\{i\}^\{\*\}\}\. We first consider the unconstrained MLEμ^i,tM​L​E\\hat\{\\mu\}\_\{i,t\}^\{MLE\}, for whichgi,t​\(μ^i,tM​L​E\)=0g\_\{i,t\}\(\\hat\{\\mu\}\_\{i,t\}^\{MLE\}\)=0\. We want to form a quantity to bound by Theorem[13\.5](https://arxiv.org/html/2605.24261#S13.Thmtheorem5)\. Considergi,t​\(μ^i,tM​L​E\)−gi,t​\(μi∗\)=−St\+λ2​μi∗g\_\{i,t\}\(\\hat\{\\mu\}\_\{i,t\}^\{MLE\}\)\-g\_\{i,t\}\(\\mu\_\{i\}^\{\*\}\)=\-S\_\{t\}\+\\lambda\_\{2\}\\mu\_\{i\}^\{\*\}\. Using the triangle inequality, dividing byHt\\sqrt\{H\_\{t\}\}, and applying Theorem[13\.5](https://arxiv.org/html/2605.24261#S13.Thmtheorem5)to\|St\|Ht\\frac\{\|S\_\{t\}\|\}\{\\sqrt\{H\_\{t\}\}\}, we have with probability at least1−δ/M1\-\\nicefrac\{\{\\delta\}\}\{\{M\}\}:

\|gi,t​\(μ^i,tM​L​E\)−gi,t​\(μi∗\)\|Ht\\displaystyle\\tfrac\{\|g\_\{i,t\}\(\\hat\{\\mu\}\_\{i,t\}^\{MLE\}\)\-g\_\{i,t\}\(\\mu\_\{i\}^\{\*\}\)\|\}\{\\sqrt\{H\_\{t\}\}\}≤\|St\|Ht\+\|λ2​μi∗\|Ht≤\|St\|Ht\+λ2​μ¯Ht≤λ22\+2λ2​log⁡\(2​M​Htδ​λ2\)\+λ2​μ¯Ht\\displaystyle\\leq\\tfrac\{\|S\_\{t\}\|\}\{\\sqrt\{H\_\{t\}\}\}\+\\tfrac\{\|\\lambda\_\{2\}\\mu\_\{i\}^\{\*\}\|\}\{\\sqrt\{H\_\{t\}\}\}\\leq\\tfrac\{\|S\_\{t\}\|\}\{\\sqrt\{H\_\{t\}\}\}\+\\tfrac\{\\lambda\_\{2\}\\bar\{\\mu\}\}\{\\sqrt\{H\_\{t\}\}\}\\leq\\tfrac\{\\sqrt\{\\lambda\_\{2\}\}\}\{2\}\+\\tfrac\{2\}\{\\sqrt\{\\lambda\_\{2\}\}\}\\log\(\\tfrac\{2M\\sqrt\{H\_\{t\}\}\}\{\\delta\\sqrt\{\\lambda\_\{2\}\}\}\)\+\\tfrac\{\\lambda\_\{2\}\\bar\{\\mu\}\}\{\\sqrt\{H\_\{t\}\}\}\(19\)Next,gi,t​\(μi\)g\_\{i,t\}\(\\mu\_\{i\}\)is a smooth function ofμi\\mu\_\{i\}, so the mean value theorem implies there existsμ~\\tilde\{\\mu\}betweenμi∗\\mu\_\{i\}^\{\*\}and the projected estimatorμ^i,t∈\[−μ¯,μ¯\]\\hat\{\\mu\}\_\{i,t\}\\in\[\-\\bar\{\\mu\},\\bar\{\\mu\}\]such that:∂gi,t​\(μi\)∂μi\|μi=μ~​\(μ^i,t−μi∗\)=gi,t​\(μ^i,t\)−gi,t​\(μi∗\)\\frac\{\\partial g\_\{i,t\}\(\\mu\_\{i\}\)\}\{\\partial\\mu\_\{i\}\}\|\_\{\\mu\_\{i\}=\\tilde\{\\mu\}\}\(\\hat\{\\mu\}\_\{i,t\}\-\\mu\_\{i\}^\{\*\}\)=g\_\{i,t\}\(\\hat\{\\mu\}\_\{i,t\}\)\-g\_\{i,t\}\(\\mu\_\{i\}^\{\*\}\)\. For brevity, letHt​\(μ\):=−∂gi,t​\(μi\)∂μi=λ2\+∑s=1t−1𝟏\{us=ei\}​σ′​\(xs\+μ\)H\_\{t\}\(\\mu\):=\\frac\{\-\\partial g\_\{i,t\}\(\\mu\_\{i\}\)\}\{\\partial\\mu\_\{i\}\}=\\lambda\_\{2\}\+\\sum\_\{s=1\}^\{t\-1\}\\mathbf\{1\}\_\{\\\{u\_\{s\}=e\_\{i\}\\\}\}\\sigma^\{\\prime\}\(x\_\{s\}\+\\mu\), so taking absolute values and rearranging \(noteHt​\(μ\)\>0H\_\{t\}\(\\mu\)\>0\) gives\|μ^i,t−μi∗\|=\|gi,t​\(μ^i,t\)−gi,t​\(μi∗\)\|\|Ht​\(μ~\)\|\|\\hat\{\\mu\}\_\{i,t\}\-\\mu\_\{i\}^\{\*\}\|=\\frac\{\|g\_\{i,t\}\(\\hat\{\\mu\}\_\{i,t\}\)\-g\_\{i,t\}\(\\mu\_\{i\}^\{\*\}\)\|\}\{\\left\|H\_\{t\}\(\\tilde\{\\mu\}\)\\right\|\}\. We upper bound the numerator to allow use of the bound in \([19](https://arxiv.org/html/2605.24261#S13.E19)\) and lower bound the denominator in terms of known quantities\. For the numerator, observe thatgi,t′​\(μi\)=−λ2−∑s=1t−1𝟏\{us=ei\}​σ′​\(xs\+μi\)<0g^\{\\prime\}\_\{i,t\}\(\\mu\_\{i\}\)=\-\\lambda\_\{2\}\-\\sum\_\{s=1\}^\{t\-1\}\\mathbf\{1\}\_\{\\\{u\_\{s\}=e\_\{i\}\\\}\}\\sigma^\{\\prime\}\(x\_\{s\}\+\\mu\_\{i\}\)<0, implying thatgi,t​\(μi\)g\_\{i,t\}\(\\mu\_\{i\}\)is decreasing inμi\\mu\_\{i\}\. Ifμ^i,tM​L​E∈\[−μ¯,μ¯\]\\hat\{\\mu\}\_\{i,t\}^\{MLE\}\\in\[\-\\bar\{\\mu\},\\bar\{\\mu\}\]then the projected estimator is triviallyμ^i,t=μ^i,tM​L​E\\hat\{\\mu\}\_\{i,t\}=\\hat\{\\mu\}\_\{i,t\}^\{MLE\}\. Next ifμ^i,tM​L​E<−μ¯\\hat\{\\mu\}\_\{i,t\}^\{MLE\}<\-\\bar\{\\mu\}, thenμ^i,t=−μ¯\\hat\{\\mu\}\_\{i,t\}=\-\\bar\{\\mu\}\. Sincegi,t​\(μi\)g\_\{i,t\}\(\\mu\_\{i\}\)is decreasing,gi,t​\(μ^i,tM​L​E\)\>gi,t​\(μ^i,t\)≥gi,t​\(μi∗\)g\_\{i,t\}\(\\hat\{\\mu\}\_\{i,t\}^\{MLE\}\)\>g\_\{i,t\}\(\\hat\{\\mu\}\_\{i,t\}\)\\geq g\_\{i,t\}\(\\mu\_\{i\}^\{\*\}\), implying\|gi,t​\(μ^i,t\)−gi,t​\(μi∗\)\|≤\|gi,t​\(μ^i,tM​L​E\)−gi,t​\(μi∗\)\|\|g\_\{i,t\}\(\\hat\{\\mu\}\_\{i,t\}\)\-g\_\{i,t\}\(\\mu\_\{i\}^\{\*\}\)\|\\leq\|g\_\{i,t\}\(\\hat\{\\mu\}\_\{i,t\}^\{MLE\}\)\-g\_\{i,t\}\(\\mu\_\{i\}^\{\*\}\)\|\. By a symmetrical argument the same holds whenμ^i,tM​L​E\>μ¯\\hat\{\\mu\}\_\{i,t\}^\{MLE\}\>\\bar\{\\mu\}, meaning the numerator can be upper bounded using \([19](https://arxiv.org/html/2605.24261#S13.E19)\)\. Whileμ~\\tilde\{\\mu\}in the denominator is unknown, sinceμ^i,t,μi∗∈\[−μ¯,μ¯\]\\hat\{\\mu\}\_\{i,t\},\\mu\_\{i\}^\{\*\}\\in\[\-\\bar\{\\mu\},\\bar\{\\mu\}\]it follows that\|μ~−μ^i,t\|≤2​μ¯\|\\tilde\{\\mu\}\-\\hat\{\\mu\}\_\{i,t\}\|\\leq 2\\bar\{\\mu\}\. By the sigmoid’s self\-concordance property \(Fauryet al\.\([2020](https://arxiv.org/html/2605.24261#bib.bib25)\)Lemma 9\),σ′​\(x\+μ^i,t\)​e−2​μ¯≤σ′​\(x\+μ~\)≤σ′​\(x\+μ^i,t\)​e2​μ¯\\sigma^\{\\prime\}\(x\+\\hat\{\\mu\}\_\{i,t\}\)e^\{\-2\\bar\{\\mu\}\}\\leq\\sigma^\{\\prime\}\(x\+\\tilde\{\\mu\}\)\\leq\\sigma^\{\\prime\}\(x\+\\hat\{\\mu\}\_\{i,t\}\)e^\{2\\bar\{\\mu\}\}\. The first inequality implies\|Ht\(μ~\)\|=Ht\(μ~\)≥λ2\+e−2​μ¯∑s=1t−1𝟏\{us=ei\}σ′\(xs\+μ^i,t\)=:Ht¯\(μ^i,t\)\|H\_\{t\}\(\\tilde\{\\mu\}\)\|=H\_\{t\}\(\\tilde\{\\mu\}\)\\geq\\lambda\_\{2\}\+e^\{\-2\\bar\{\\mu\}\}\\sum\_\{s=1\}^\{t\-1\}\\mathbf\{1\}\_\{\\\{u\_\{s\}=e\_\{i\}\\\}\}\\sigma^\{\\prime\}\(x\_\{s\}\+\\hat\{\\mu\}\_\{i,t\}\)=:\\underline\{H\_\{t\}\}\(\\hat\{\\mu\}\_\{i,t\}\)\. The second givesHt≤λ2\+e2​μ¯∑s=1t−1𝟏\{us=ei\}σ′\(xs\+μ^i,t\)=:Ht¯\(μ^i,t\)H\_\{t\}\\leq\\lambda\_\{2\}\+e^\{2\\bar\{\\mu\}\}\\sum\_\{s=1\}^\{t\-1\}\\mathbf\{1\}\_\{\\\{u\_\{s\}=e\_\{i\}\\\}\}\\sigma^\{\\prime\}\(x\_\{s\}\+\\hat\{\\mu\}\_\{i,t\}\)=:\\overline\{H\_\{t\}\}\(\\hat\{\\mu\}\_\{i,t\}\)\. Thus with probability at least1−δ/M1\-\\nicefrac\{\{\\delta\}\}\{\{M\}\}:

\|μ^i,t−μi∗\|\\displaystyle\|\\hat\{\\mu\}\_\{i,t\}\-\\mu\_\{i\}^\{\*\}\|≤\(a\)​Ht\|Ht​\(μ~\)\|​\|gi,t​\(μ^i,tM​L​E\)−gi,t​\(μi∗\)\|Ht​≤\(b\)​Ht¯​\(μ^i,t\)Ht¯​\(μ^i,t\)​\[λ22\+2λ2​log⁡\(2​M​Ht¯​\(μ^i,t\)δ​λ2\)\]\+λ2​μ¯Ht¯​\(μ^i,t\)\\displaystyle\\overset\{\(a\)\}\{\\leq\}\\tfrac\{\\sqrt\{H\_\{t\}\}\}\{\|H\_\{t\}\(\\tilde\{\\mu\}\)\|\}\\tfrac\{\|g\_\{i,t\}\(\\hat\{\\mu\}\_\{i,t\}^\{MLE\}\)\-g\_\{i,t\}\(\\mu\_\{i\}^\{\*\}\)\|\}\{\\sqrt\{H\_\{t\}\}\}\\overset\{\(b\)\}\{\\leq\}\\tfrac\{\\sqrt\{\\overline\{H\_\{t\}\}\(\\hat\{\\mu\}\_\{i,t\}\)\}\}\{\\underline\{H\_\{t\}\}\(\\hat\{\\mu\}\_\{i,t\}\)\}\[\\tfrac\{\\sqrt\{\\lambda\_\{2\}\}\}\{2\}\+\\tfrac\{2\}\{\\sqrt\{\\lambda\_\{2\}\}\}\\log\(\\tfrac\{2M\\sqrt\{\\overline\{H\_\{t\}\}\(\\hat\{\\mu\}\_\{i,t\}\)\}\}\{\\delta\\sqrt\{\\lambda\_\{2\}\}\}\)\]\+\\tfrac\{\\lambda\_\{2\}\\bar\{\\mu\}\}\{\\underline\{H\_\{t\}\}\(\\hat\{\\mu\}\_\{i,t\}\)\}\(20\)≤\(c\)​e3​μ¯Ht​\(μ^i,t\)​\[λ22\+2​μ¯λ2\+2λ2​log⁡\(2​M​Ht​\(μ^i,t\)δ​λ2\)\]\+e2​μ¯​λ2​μ¯Ht​\(μ^i,t\)\\displaystyle\\overset\{\(c\)\}\{\\leq\}\\tfrac\{e^\{3\\bar\{\\mu\}\}\}\{\\sqrt\{H\_\{t\}\(\\hat\{\\mu\}\_\{i,t\}\)\}\}\[\\tfrac\{\\sqrt\{\\lambda\_\{2\}\}\}\{2\}\+\\tfrac\{2\\bar\{\\mu\}\}\{\\sqrt\{\\lambda\_\{2\}\}\}\+\\tfrac\{2\}\{\\sqrt\{\\lambda\_\{2\}\}\}\\log\(\\tfrac\{2M\\sqrt\{H\_\{t\}\(\\hat\{\\mu\}\_\{i,t\}\)\}\}\{\\delta\\sqrt\{\\lambda\_\{2\}\}\}\)\]\+\\tfrac\{e^\{2\\bar\{\\mu\}\}\\lambda\_\{2\}\\bar\{\\mu\}\}\{H\_\{t\}\(\\hat\{\\mu\}\_\{i,t\}\)\}\(21\)\(a\)\(a\)multiplies byHt/Ht\\nicefrac\{\{\\sqrt\{H\_\{t\}\}\}\}\{\{\\sqrt\{H\_\{t\}\}\}\}so that\(b\)\(b\)can substituteHtH\_\{t\}bounds and \([19](https://arxiv.org/html/2605.24261#S13.E19)\)\.\(c\)\(c\)loosens the bound for clarity, specifically usingHt¯≥e−2​μ¯​\[λ2\+∑s=1t−1𝟏\{us=ei\}​σ′​\(xs\+μ^i,t\)\]=e−2​μ¯​Ht​\(μ^i,t\)\\underline\{H\_\{t\}\}\\geq e^\{\-2\\bar\{\\mu\}\}\[\\lambda\_\{2\}\+\\sum\_\{s=1\}^\{t\-1\}\\mathbf\{1\}\_\{\\\{u\_\{s\}=e\_\{i\}\\\}\}\\sigma^\{\\prime\}\(x\_\{s\}\+\\hat\{\\mu\}\_\{i,t\}\)\]=e^\{\-2\\bar\{\\mu\}\}H\_\{t\}\(\\hat\{\\mu\}\_\{i,t\}\),Ht¯≤e2​μ¯​\[λ2\+∑s=1t−1𝟏\{us=ei\}​σ′​\(xs\+μ^i,t\)\]=e2​μ¯​Ht​\(μ^i,t\)\\overline\{H\_\{t\}\}\\leq e^\{2\\bar\{\\mu\}\}\[\\lambda\_\{2\}\+\\sum\_\{s=1\}^\{t\-1\}\\mathbf\{1\}\_\{\\\{u\_\{s\}=e\_\{i\}\\\}\}\\sigma^\{\\prime\}\(x\_\{s\}\+\\hat\{\\mu\}\_\{i,t\}\)\]=e^\{2\\bar\{\\mu\}\}H\_\{t\}\(\\hat\{\\mu\}\_\{i,t\}\), and pullingeμ¯e^\{\\bar\{\\mu\}\}\(from substituting theHt¯\\overline\{H\_\{t\}\}\) out of the log\. The result follows by a union bound over the actions\.\\halmos

### 13\.2Value function supporting analysis

Let Assumptions[3\.1](https://arxiv.org/html/2605.24261#S3.SS1)\-[5\.1](https://arxiv.org/html/2605.24261#S5.SS1)hold for the following supporting results\. Here we provide proofs for properties of the relevant operators \(Lemma[5\.3](https://arxiv.org/html/2605.24261#S5.Thmtheorem3)\), boundedness of the bonus \(Lemma[13\.8](https://arxiv.org/html/2605.24261#S13.Thmtheorem8)\),J∗​\(x\)J^\{\*\}\(x\)andJ~t,δ​\(x\)\\tilde\{J\}\_\{t,\\delta\}\(x\)Lipschitz continuity \(Lemmas[13\.10](https://arxiv.org/html/2605.24261#S13.Thmtheorem10)\-[13\.12](https://arxiv.org/html/2605.24261#S13.Thmtheorem12)\), and valid optimism \(Lemma[5\.4](https://arxiv.org/html/2605.24261#S5.Thmtheorem4)\)\.

###### Proof 13\.7

Proof of Lemma[5\.3](https://arxiv.org/html/2605.24261#S5.Thmtheorem3)r​\(x,u,d\)r\(x,u,d\)is bounded by construction,bt,δ​\(x,u\)b\_\{t,\\delta\}\(x,u\)is bounded byCb​\(T,δ\)C\_\{b\}\(T,\\delta\)\(Lemma[13\.8](https://arxiv.org/html/2605.24261#S13.Thmtheorem8)\) fort≤Tt\\leq T, and rewards are discounted \(Assumption[5\.1](https://arxiv.org/html/2605.24261#S5.SS1)\), so contraction Assumption 1\.6\.1 inBertsekas \([2012](https://arxiv.org/html/2605.24261#bib.bib13)\)is satisfied and repeated applications of the operator converge uniformly to unique fixed pointsJ∗J^\{\*\}andJ~t,δ\\tilde\{J\}\_\{t,\\delta\}of these operators, respectively, by Proposition 1\.6\.1\(Bertsekas[2012](https://arxiv.org/html/2605.24261#bib.bib13)\)\. These operators also satisfy monotonicity Assumption 1\.6\.2 \(by Lemma 1\.1\.1\), implying that they are the optimal value functions by Proposition 1\.6\.2\(Bertsekas[2012](https://arxiv.org/html/2605.24261#bib.bib13)\)\.J∗​\(x\)J^\{\*\}\(x\)is clearly bounded by\(ρ¯\+β\)/\(1−γ\)\\nicefrac\{\{\(\\bar\{\\rho\}\+\\beta\)\}\}\{\{\(1\-\\gamma\}\}\)by the bounded construction ofr​\(x,u,d\)r\(x,u,d\)\(meaningJπJ^\{\\pi\}is identically bounded\)\.J~t,δ​\(x\)\\tilde\{J\}\_\{t,\\delta\}\(x\)is bounded by an identical argument using the boundedness of the bonus\.\\halmos

###### Lemma 13\.8

‖bt,δ​\(x,u\)‖∞≤\(ρ¯\+γ​L1​c¯\)​α¯μ​\(δ/2\)4\+γ​L1​αTθ​\(δ/2\)​Cx2\+2λ1\\\|b\_\{t,\\delta\}\(x,u\)\\\|\_\{\\infty\}\\leq\\frac\{\(\\bar\{\\rho\}\+\\gamma L\_\{1\}\\bar\{c\}\)\\overline\{\\alpha\}^\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\}\{4\}\+\\frac\{\\gamma L\_\{1\}\\alpha^\{\\theta\}\_\{T\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\\sqrt\{C\_\{x\}^\{2\}\+2\}\}\{\\sqrt\{\\lambda\_\{1\}\}\}fort∈\{1,…,T\}t\\in\\\{1,\\dots,T\\\},δ∈\(0,1\)\\delta\\in\(0,1\)\.

###### Proof 13\.9

Proof of Lemma[13\.8](https://arxiv.org/html/2605.24261#S13.Thmtheorem8)The bonus is non\-negative, so we show an upper bound\. Letx∈𝒳x\\in\\mathcal\{X\},u∈𝒰u\\in\\mathcal\{U\},δ∈\(0,1\)\\delta\\in\(0,1\),t∈\{1,…,T\}t\\in\\\{1,\\dots,T\\\}\. Asαtθ\\alpha^\{\\theta\}\_\{t\}is increasing intt,αtθ​\(δ/2\)≤αTθ​\(δ/2\)\\alpha^\{\\theta\}\_\{t\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\\leq\\alpha^\{\\theta\}\_\{T\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\.V¯0−1⪰V¯t−1\\bar\{V\}\_\{0\}^\{\-1\}\\succeq\\bar\{V\}\_\{t\}^\{\-1\}by construction\. Recallαi,tμ​\(δ/2\):=e3​μ¯Ht​\(μ^i,t\)​\[λ22\+2λ2​\(μ¯\+log⁡\(4​M​Ht​\(μ^i,t\)δ​λ2\)\)\]\+e2​μ¯​λ2​μ¯Ht​\(μ^i,t\)\\alpha^\{\\mu\}\_\{i,t\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\):=\\frac\{e^\{3\\bar\{\\mu\}\}\}\{\\sqrt\{H\_\{t\}\(\\hat\{\\mu\}\_\{i,t\}\)\}\}\[\\frac\{\\sqrt\{\\lambda\_\{2\}\}\}\{2\}\+\\frac\{2\}\{\\sqrt\{\\lambda\_\{2\}\}\}\(\\bar\{\\mu\}\+\\log\(\\frac\{4M\\sqrt\{H\_\{t\}\(\\hat\{\\mu\}\_\{i,t\}\)\}\}\{\\delta\\sqrt\{\\lambda\_\{2\}\}\}\)\)\]\+\\frac\{e^\{2\\bar\{\\mu\}\}\\lambda\_\{2\}\\bar\{\\mu\}\}\{H\_\{t\}\(\\hat\{\\mu\}\_\{i,t\}\)\}forHt​\(μ^i,t\)=λ2\+∑s=1t−1𝟏\{us=ei\}​σ′​\(xs\+μ^i,t\)H\_\{t\}\(\\hat\{\\mu\}\_\{i,t\}\)=\\lambda\_\{2\}\+\\sum\_\{s=1\}^\{t\-1\}\\mathbf\{1\}\_\{\\\{u\_\{s\}=e\_\{i\}\\\}\}\\sigma^\{\\prime\}\(x\_\{s\}\+\\hat\{\\mu\}\_\{i,t\}\)\. Separating the log term we equivalently haveαi,tμ​\(δ/2\)=e3​μ¯Ht​\(μ^i,t\)​\[λ22\+2λ2​\(μ¯\+log⁡\(4​Mδ​λ2\)\)\]\+2​e3​μ¯​log⁡\(Ht​\(μ^i,t\)\)λ2​Ht​\(μ^i,t\)\+e2​μ¯​λ2​μ¯Ht​\(μ^i,t\)\\alpha\_\{i,t\}^\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)=\\frac\{e^\{3\\bar\{\\mu\}\}\}\{\\sqrt\{H\_\{t\}\(\\hat\{\\mu\}\_\{i,t\}\)\}\}\[\\frac\{\\sqrt\{\\lambda\_\{2\}\}\}\{2\}\+\\frac\{2\}\{\\sqrt\{\\lambda\_\{2\}\}\}\(\\bar\{\\mu\}\+\\log\(\\frac\{4M\}\{\\delta\\sqrt\{\\lambda\_\{2\}\}\}\)\)\]\+\\frac\{2e^\{3\\bar\{\\mu\}\}\\log\(\\sqrt\{H\_\{t\}\(\\hat\{\\mu\}\_\{i,t\}\)\}\)\}\{\\sqrt\{\\lambda\_\{2\}\}\\sqrt\{H\_\{t\}\(\\hat\{\\mu\}\_\{i,t\}\)\}\}\+\\frac\{e^\{2\\bar\{\\mu\}\}\\lambda\_\{2\}\\bar\{\\mu\}\}\{H\_\{t\}\(\\hat\{\\mu\}\_\{i,t\}\)\}\. Using the boundsHt​\(μ^i,t\)≥λ2H\_\{t\}\(\\hat\{\\mu\}\_\{i,t\}\)\\geq\\lambda\_\{2\}andlog⁡xx≤e−1\\frac\{\\log x\}\{x\}\\leq e^\{\-1\}we haveαi,tμ\(δ/2\)≤e3​μ¯λ2\[λ22\+2λ2\(μ¯\+log\(4​Mδ​λ2\)\)\]\+2​e3​μ¯−1λ2\+e2​μ¯μ¯=:α¯μ\(δ/2\)\\alpha\_\{i,t\}^\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\\leq\\frac\{e^\{3\\bar\{\\mu\}\}\}\{\\sqrt\{\\lambda\_\{2\}\}\}\[\\frac\{\\sqrt\{\\lambda\_\{2\}\}\}\{2\}\+\\frac\{2\}\{\\sqrt\{\\lambda\_\{2\}\}\}\(\\bar\{\\mu\}\+\\log\(\\frac\{4M\}\{\\delta\\sqrt\{\\lambda\_\{2\}\}\}\)\)\]\+\\frac\{2e^\{3\\bar\{\\mu\}\-1\}\}\{\\sqrt\{\\lambda\_\{2\}\}\}\+e^\{2\\bar\{\\mu\}\}\\bar\{\\mu\}=:\\overline\{\\alpha\}^\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\. Thus:

bt,δ​\(x,u\)\\displaystyle b\_\{t,\\delta\}\(x,u\):=\(ρ¯\+γ​L1​c¯\)​∑i=1Mui​κ​\(x,μ^i,t\)​αi,tμ​\(δ/2\)\+γ​L1​αtθ​\(δ/2\)​𝐄μ^t​\[‖z‖V¯t−1\|x,u\]\\displaystyle:=\(\\bar\{\\rho\}\+\\gamma L\_\{1\}\\bar\{c\}\)\\textstyle\\sum\\nolimits\_\{i=1\}^\{M\}u^\{i\}\\kappa\(x,\\hat\{\\mu\}\_\{i,t\}\)\\alpha\_\{i,t\}^\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\+\\gamma L\_\{1\}\\alpha^\{\\theta\}\_\{t\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\\mathbf\{E\}\_\{\\hat\{\\mu\}\_\{t\}\}\[\\\|z\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}\|x,u\]\(22\)≤\(ρ¯\+γL1c¯\)\(1/4\)α¯μ\(δ/2\)\+γL1αTθ\(δ/2\)\(Cx2\+2/λ1\)=:Cb\(T,δ\)\\Halmos\\displaystyle\\leq\(\\bar\{\\rho\}\+\\gamma L\_\{1\}\\bar\{c\}\)\(\\nicefrac\{\{1\}\}\{\{4\}\}\)\\overline\{\\alpha\}^\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\+\\gamma L\_\{1\}\\alpha^\{\\theta\}\_\{T\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\(\\nicefrac\{\{\\sqrt\{C\_\{x\}^\{2\}\+2\}\}\}\{\{\\sqrt\{\\lambda\_\{1\}\}\}\}\)=:C\_\{b\}\(T,\\delta\)\\Halmos\(23\)

###### Lemma 13\.10

J∗​\(x\)J^\{\*\}\(x\)isL1L\_\{1\}\-Lipschitz continuous inxxon𝒳\\mathcal\{X\}withL1=β\+ρ¯4​\(1−γ​a¯\)​\(1\+2​γ1−γ\)L\_\{1\}=\\frac\{\\beta\+\\bar\{\\rho\}\}\{4\(1\-\\gamma\\bar\{a\}\)\}\(1\+\\frac\{2\\gamma\}\{1\-\\gamma\}\)\.

###### Proof 13\.11

Proof of Lemma[13\.10](https://arxiv.org/html/2605.24261#S13.Thmtheorem10)By definition,𝒯​J​\(x\)=maxu∈𝒰⁡\{𝐄θ∗,μ∗​\[r​\(x,u,d\)\+γ​J​\(f​\(x,u,d,w\)\)\]\}\\mathcal\{T\}J\(x\)=\\max\_\{u\\in\\mathcal\{U\}\}\\\{\\mathbf\{E\}\_\{\\theta^\{\*\},\\mu^\{\*\}\}\[r\(x,u,d\)\+\\gamma J\(f\(x,u,d,w\)\)\]\\\}\. By Lemma[13\.15](https://arxiv.org/html/2605.24261#S13.Thmtheorem15),𝐄θ∗,μ∗​\[r​\(x,u,d\)\]\\mathbf\{E\}\_\{\\theta^\{\*\},\\mu^\{\*\}\}\[r\(x,u,d\)\]is uniformly Lipschitz continuous inxxwith constantLr=\(β\+ρ¯\)/4L\_\{r\}=\\nicefrac\{\{\(\\beta\+\\bar\{\\rho\}\)\}\}\{\{4\}\}\. By Lemma[13\.17](https://arxiv.org/html/2605.24261#S13.Thmtheorem17), if a functionJ​\(x\)J\(x\)isLL\-Lipschitz continuous inxxand bounded byJ¯\\bar\{J\}thenγ𝐄θ∗,μ∗\[J\(f\(x,u,d,w\)\]\\gamma\\mathbf\{E\}\_\{\\theta^\{\*\},\\mu^\{\*\}\}\[J\(f\(x,u,d,w\)\]is uniformly Lipschitz continuous inxxwith constantLv=γ​\(L​a¯\+J¯/2\)L\_\{v\}=\\gamma\(L\\bar\{a\}\+\\nicefrac\{\{\\bar\{J\}\}\}\{\{2\}\}\)\. By Lemma[5\.3](https://arxiv.org/html/2605.24261#S5.Thmtheorem3),J¯=β\+ρ¯1−γ\\bar\{J\}=\\frac\{\\beta\+\\bar\{\\rho\}\}\{1\-\\gamma\}\. This implies that𝒯​J​\(x\)\\mathcal\{T\}J\(x\)isL′L^\{\\prime\}\-Lipschitz withL′=\(β\+ρ¯\)/4\+γ​L​a¯\+γ​β\+ρ¯2​\(1−γ\)L^\{\\prime\}=\\nicefrac\{\{\(\\beta\+\\bar\{\\rho\}\)\}\}\{\{4\}\}\+\\gamma L\\bar\{a\}\+\\gamma\\frac\{\\beta\+\\bar\{\\rho\}\}\{2\(1\-\\gamma\)\}\. Solving for the smallestL′L^\{\\prime\}such thatL′≥LL^\{\\prime\}\\geq LgivesL1=β\+ρ¯4​\(1\+2​γ1−γ\)1−γ​a¯L\_\{1\}=\\frac\{\\frac\{\\beta\+\\bar\{\\rho\}\}\{4\}\(1\+\\frac\{2\\gamma\}\{1\-\\gamma\}\)\}\{1\-\\gamma\\bar\{a\}\}\. Note that this denominator is strictly positive asγ∈\(0,1\)\\gamma\\in\(0,1\)anda¯<1\\bar\{a\}<1by assumption\. LetJ0​\(x\)J\_\{0\}\(x\)be the zero function, which is triviallyL1L\_\{1\}\-Lipschitz\. By induction onnn, we observe thatJn=𝒯n​J0J\_\{n\}=\\mathcal\{T\}^\{n\}J\_\{0\}is alsoL1L\_\{1\}\-Lipschitz for alln∈ℕn\\in\\mathbb\{N\}\. By Lemma[5\.3](https://arxiv.org/html/2605.24261#S5.Thmtheorem3),JnJ\_\{n\}converges uniformly toJ∗J^\{\*\}\. Thus for everyϵ\>0\\epsilon\>0there existsN∈ℕN\\in\\mathbb\{N\}such that\|J∗​\(x\)−Jn​\(x\)\|<ϵ/2\|J^\{\*\}\(x\)\-J\_\{n\}\(x\)\|<\\nicefrac\{\{\\epsilon\}\}\{\{2\}\}for allx∈𝒳x\\in\\mathcal\{X\}andn≥Nn\\geq N\. Forx1,x2∈𝒳x\_\{1\},x\_\{2\}\\in\\mathcal\{X\}:\|J∗​\(x1\)−J∗​\(x2\)\|≤\|J∗​\(x1\)−JN​\(x1\)\|\+\|JN​\(x1\)−JN​\(x2\)\|\+\|JN​\(x2\)−J∗​\(x2\)\|≤ϵ\+L1​\|x1−x2\|\\left\|J^\{\*\}\(x\_\{1\}\)\-J^\{\*\}\(x\_\{2\}\)\\right\|\\leq\\left\|J^\{\*\}\(x\_\{1\}\)\-J\_\{N\}\(x\_\{1\}\)\\right\|\+\\left\|J\_\{N\}\(x\_\{1\}\)\-J\_\{N\}\(x\_\{2\}\)\\right\|\+\\left\|J\_\{N\}\(x\_\{2\}\)\-J^\{\*\}\(x\_\{2\}\)\\right\|\\leq\\epsilon\+L\_\{1\}\\left\|x\_\{1\}\-x\_\{2\}\\right\|\. This holds for anyϵ\>0\\epsilon\>0, implying\|J∗​\(x1\)−J∗​\(x2\)\|≤L1​\|x1−x2\|\\left\|J^\{\*\}\(x\_\{1\}\)\-J^\{\*\}\(x\_\{2\}\)\\right\|\\leq L\_\{1\}\\left\|x\_\{1\}\-x\_\{2\}\\right\|⇔\\LeftrightarrowJ∗J^\{\*\}isL1L\_\{1\}\-Lipschitz\.\\halmos

###### Lemma 13\.12

J~t,δ\\tilde\{J\}\_\{t,\\delta\}isL2L\_\{2\}\-Lipschitz continuous inxxon𝒳\\mathcal\{X\}fort∈\{1,…,T\}t\\in\\\{1,\\dots,T\\\}withL2=11−γ​a¯​\[\(β\+ρ¯\)/4\+e2​μ¯/4​\(ρ¯\+γ​L1​c¯\)​α¯μ​\(δ/2\)\+5​γ​L1​αTθ​\(δ/2\)4​λ1\+γ/2​CJ~,T,δ\]L\_\{2\}=\\frac\{1\}\{1\-\\gamma\\bar\{a\}\}\\big\[\\nicefrac\{\{\(\\beta\+\\bar\{\\rho\}\)\}\}\{\{4\}\}\+\\nicefrac\{\{e^\{2\\bar\{\\mu\}\}\}\}\{\{4\}\}\(\\bar\{\\rho\}\+\\gamma L\_\{1\}\\bar\{c\}\)\\overline\{\\alpha\}^\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\+\\frac\{5\\gamma L\_\{1\}\\alpha^\{\\theta\}\_\{T\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\}\{4\\sqrt\{\\lambda\_\{1\}\}\}\+\\nicefrac\{\{\\gamma\}\}\{\{2\}\}C\_\{\\tilde\{J\},T,\\delta\}\\big\]\.

###### Proof 13\.13

Proof of Lemma[13\.12](https://arxiv.org/html/2605.24261#S13.Thmtheorem12)We follow a similar approach to Lemma[13\.10](https://arxiv.org/html/2605.24261#S13.Thmtheorem10)\. Lett∈\{1,…,T\}t\\in\\\{1,\\dots,T\\\},θ^t∈Θ\\hat\{\\theta\}\_\{t\}\\in\\Thetaandμ^t∈Φ\\hat\{\\mu\}\_\{t\}\\in\\Phi\. Recall𝒯~t,δ​J​\(x\):=maxu∈𝒰⁡\{𝐄θ^t,μ^t​\[r​\(x,u,d\)\+bt,δ​\(x,u\)\+γ​J​\(f​\(x,u,d,w\)\)\]\}\\tilde\{\\mathcal\{T\}\}\_\{t,\\delta\}J\(x\):=\\max\_\{u\\in\\mathcal\{U\}\}\\\{\\mathbf\{E\}\_\{\\hat\{\\theta\}\_\{t\},\\hat\{\\mu\}\_\{t\}\}\[r\(x,u,d\)\+b\_\{t,\\delta\}\(x,u\)\+\\gamma J\(f\(x,u,d,w\)\)\]\\\}\. By Lemmas[13\.15](https://arxiv.org/html/2605.24261#S13.Thmtheorem15)and[13\.19](https://arxiv.org/html/2605.24261#S13.Thmtheorem19),𝐄θ^t,μ^t​\[r​\(x,u,d\)\]\\mathbf\{E\}\_\{\\hat\{\\theta\}\_\{t\},\\hat\{\\mu\}\_\{t\}\}\[r\(x,u,d\)\]and𝐄θ^t,μ^t​\[bt,δ​\(x,u\)\]=bt,δ​\(x,u\)\\mathbf\{E\}\_\{\\hat\{\\theta\}\_\{t\},\\hat\{\\mu\}\_\{t\}\}\[b\_\{t,\\delta\}\(x,u\)\]=b\_\{t,\\delta\}\(x,u\)are uniformlyLrL\_\{r\}\- andLbL\_\{b\}\-Lipschitz continuous inxx\. By Lemma[13\.17](https://arxiv.org/html/2605.24261#S13.Thmtheorem17), ifJJisLL\-Lipschitz continuous inxxand bounded byJ¯\\bar\{J\}, thenγ𝐄θ^t,μ^t\[J\(f\(x,u,d,w\)\]\\gamma\\mathbf\{E\}\_\{\\hat\{\\theta\}\_\{t\},\\hat\{\\mu\}\_\{t\}\}\[J\(f\(x,u,d,w\)\]is uniformlyLvL\_\{v\}\-Lipschitz withLv=γ​L​a¯\+γ​J¯/2L\_\{v\}=\\gamma L\\bar\{a\}\+\\nicefrac\{\{\\gamma\\bar\{J\}\}\}\{\{2\}\}\(here‖J~t,δ‖∞≤CJ~,T,δ\\\|\\tilde\{J\}\_\{t,\\delta\}\\\|\_\{\\infty\}\\leq C\_\{\\tilde\{J\},T,\\delta\}by Lemma[5\.3](https://arxiv.org/html/2605.24261#S5.Thmtheorem3)\)\. The maximum over actions is \(Lr\+Lb\+Lv\)L\_\{r\}\+L\_\{b\}\+L\_\{v\}\)\-Lipschitz\. Thus ifJJisLL\-Lipschitz,𝒯~t,δ​J\\tilde\{\\mathcal\{T\}\}\_\{t,\\delta\}JisL′L^\{\\prime\}\-Lipschitz withL′=Lr\+Lb\+γ​L​a¯\+γ2​CJ~,T,δL^\{\\prime\}=L\_\{r\}\+L\_\{b\}\+\\gamma L\\bar\{a\}\+\\tfrac\{\\gamma\}\{2\}C\_\{\\tilde\{J\},T,\\delta\}\. Finding the smallestL′≥LL^\{\\prime\}\\geq LgivesL2=11−γ​a¯​\[Lr\+Lb\+γ2​CJ~,T,δ\]L\_\{2\}=\\frac\{1\}\{1\-\\gamma\\bar\{a\}\}\[L\_\{r\}\+L\_\{b\}\+\\frac\{\\gamma\}\{2\}C\_\{\\tilde\{J\},T,\\delta\}\]\. LetJ0​\(x\)J\_\{0\}\(x\)be the zero function, which is triviallyL2L\_\{2\}\-Lipschitz and bounded byCJ~,T,δC\_\{\\tilde\{J\},T,\\delta\}\. Thus, by an identical inductive argument to that presented in Lemma[13\.10](https://arxiv.org/html/2605.24261#S13.Thmtheorem10)it follows thatJ~t,δ\\tilde\{J\}\_\{t,\\delta\}isL2L\_\{2\}\-Lipschitz\. The result follows by substitutingLrL\_\{r\}andLbL\_\{b\}\.\\halmos

###### Proof 13\.14

Proof of Lemma[5\.4](https://arxiv.org/html/2605.24261#S5.Thmtheorem4)Lett∈\{1,…,T\+1\}t\\in\\\{1,\\dots,T\+1\\\},δ∈\(0,1\)\\delta\\in\(0,1\),x∈𝒳x\\in\\mathcal\{X\},ℰθ​\(δ/2\)\\mathcal\{E\}\_\{\\theta\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\), andℰμ​\(δ/2\)\\mathcal\{E\}\_\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)hold\. By Lemma[5\.3](https://arxiv.org/html/2605.24261#S5.Thmtheorem3),𝒯~t,δ\\tilde\{\\mathcal\{T\}\}\_\{t,\\delta\}is monotone and converges uniformly to its fixed point, so showing that𝒯~t,δ​J∗​\(x\)≥J∗​\(x\)\\tilde\{\\mathcal\{T\}\}\_\{t,\\delta\}J^\{\*\}\(x\)\\geq J^\{\*\}\(x\)impliesJ~t,δ​\(x\)≥J∗​\(x\)\\tilde\{J\}\_\{t,\\delta\}\(x\)\\geq J^\{\*\}\(x\)\. RecallJ∗​\(x\)=maxu∈𝒰⁡\{𝐄θ∗,μ∗​\[r​\(x,u,d\)\+γ​J∗​\(f​\(x,u,d,w\)\)\]\}J^\{\*\}\(x\)=\\max\_\{u\\in\\mathcal\{U\}\}\\\{\\mathbf\{E\}\_\{\\theta^\{\*\},\\mu^\{\*\}\}\[r\(x,u,d\)\+\\gamma J^\{\*\}\(f\(x,u,d,w\)\)\]\\\}\. Letux∗u^\{\*\}\_\{x\}be the optimal action to the true system atxx\.ux∗u^\{\*\}\_\{x\}is feasible for the surrogate system, so𝒯~t,δ​J∗​\(x\)≥𝐄θ^t,μ^t​\[r​\(x,ux∗,d\)\+bt,δ​\(x,ux∗\)\+γ​J∗​\(f​\(x,ux∗,d,w\)\)\]\\tilde\{\\mathcal\{T\}\}\_\{t,\\delta\}J^\{\*\}\(x\)\\geq\\mathbf\{E\}\_\{\\hat\{\\theta\}\_\{t\},\\hat\{\\mu\}\_\{t\}\}\[r\(x,u^\{\*\}\_\{x\},d\)\+b\_\{t,\\delta\}\(x,u^\{\*\}\_\{x\}\)\+\\gamma J^\{\*\}\(f\(x,u^\{\*\}\_\{x\},d,w\)\)\]\. LetΔJ​\(x\):=𝒯~t,δ​J∗​\(x\)−J∗​\(x\)\\Delta\_\{J\}\(x\):=\\tilde\{\\mathcal\{T\}\}\_\{t,\\delta\}J^\{\*\}\(x\)\-J^\{\*\}\(x\)\. We have:

ΔJ​\(x\)≥𝐄μ^t​\[bt,δ​\(x,ux∗\)\]\+𝐄μ^t​\[r​\(x,ux∗,d\)\]−𝐄μ∗​\[r​\(x,ux∗,d\)\]\+γ​\[𝐄μ^t​\[J∗​\(z⊤​θ^t\+w\)\|x,ux∗\]−𝐄μ∗​\[J∗​\(z⊤​θ∗\+w\)\|x,ux∗\]\]\\Delta\_\{J\}\(x\)\\geq\\mathbf\{E\}\_\{\\hat\{\\mu\}\_\{t\}\}\[b\_\{t,\\delta\}\(x,u^\{\*\}\_\{x\}\)\]\+\\mathbf\{E\}\_\{\\hat\{\\mu\}\_\{t\}\}\[r\(x,u^\{\*\}\_\{x\},d\)\]\-\\mathbf\{E\}\_\{\\mu^\{\*\}\}\[r\(x,u^\{\*\}\_\{x\},d\)\]\\\\ \+\\gamma\[\\mathbf\{E\}\_\{\\hat\{\\mu\}\_\{t\}\}\[J^\{\*\}\(z^\{\\top\}\\hat\{\\theta\}\_\{t\}\+w\)\|x,u^\{\*\}\_\{x\}\]\-\\mathbf\{E\}\_\{\\mu^\{\*\}\}\[J^\{\*\}\(z^\{\\top\}\\theta^\{\*\}\+w\)\|x,u^\{\*\}\_\{x\}\]\]\(24\)We first assume thatux∗=eiu^\{\*\}\_\{x\}=e\_\{i\}for somei=1,…,Mi=1,\\dots,Mand address theux∗=𝟎Mu^\{\*\}\_\{x\}=\\bm\{0\}\_\{M\}case at the end\. In the reward difference, theβ\\beta\-sigmoid terms cancel leaving the adherence terms:𝐄μ^t​\[r​\(x,ux∗,d\)\]−𝐄μ∗​\[r​\(x,ux∗,d\)\]=∑i=1M𝟏\{ux∗=ei\}​ρi​\(σ​\(x\+μ^i,t\)−σ​\(x\+μi∗\)\)≥−ρ¯​\|σ​\(x\+μ^i,t\)−σ​\(x\+μi∗\)\|\\mathbf\{E\}\_\{\\hat\{\\mu\}\_\{t\}\}\[r\(x,u^\{\*\}\_\{x\},d\)\]\-\\mathbf\{E\}\_\{\\mu^\{\*\}\}\[r\(x,u^\{\*\}\_\{x\},d\)\]=\\sum\_\{i=1\}^\{M\}\\mathbf\{1\}\_\{\\\{u^\{\*\}\_\{x\}=e\_\{i\}\\\}\}\\rho\_\{i\}\(\\sigma\(x\+\\hat\{\\mu\}\_\{i,t\}\)\-\\sigma\(x\+\\mu^\{\*\}\_\{i\}\)\)\\geq\-\\bar\{\\rho\}\|\\sigma\(x\+\\hat\{\\mu\}\_\{i,t\}\)\-\\sigma\(x\+\\mu^\{\*\}\_\{i\}\)\|\. We bound the RHS with the sigmoid’s1/4\\nicefrac\{\{1\}\}\{\{4\}\}\-Lipschitz continuity and a MVT/self\-concordance argument\. By the MVT there existsμ~\\tilde\{\\mu\}betweenμ^i,t\\hat\{\\mu\}\_\{i,t\}andμi∗\\mu\_\{i\}^\{\*\}such thatσ′​\(x\+μ~\)​\(μ^i,t−μi∗\)=σ​\(x\+μ^i,t\)−σ​\(x\+μi∗\)\\sigma^\{\\prime\}\(x\+\\tilde\{\\mu\}\)\(\\hat\{\\mu\}\_\{i,t\}\-\\mu\_\{i\}^\{\*\}\)=\\sigma\(x\+\\hat\{\\mu\}\_\{i,t\}\)\-\\sigma\(x\+\\mu\_\{i\}^\{\*\}\)and thus\|σ′​\(x\+μ~\)\|​\|\(μ^i,t−μi∗\)\|=\|σ​\(x\+μ^i,t\)−σ​\(x\+μi∗\)\|\|\\sigma^\{\\prime\}\(x\+\\tilde\{\\mu\}\)\|\|\(\\hat\{\\mu\}\_\{i,t\}\-\\mu\_\{i\}^\{\*\}\)\|=\|\\sigma\(x\+\\hat\{\\mu\}\_\{i,t\}\)\-\\sigma\(x\+\\mu\_\{i\}^\{\*\}\)\|\. Asμ^i,t,μi∗∈\[−μ¯,μ¯\]\\hat\{\\mu\}\_\{i,t\},\\mu\_\{i\}^\{\*\}\\in\[\-\\bar\{\\mu\},\\bar\{\\mu\}\], we have\|μ~−μ^i,t\|≤2​μ¯\|\\tilde\{\\mu\}\-\\hat\{\\mu\}\_\{i,t\}\|\\leq 2\\bar\{\\mu\}and by the sigmoid’s self\-concordance \(Fauryet al\.\([2020](https://arxiv.org/html/2605.24261#bib.bib25)\)Lemma 9\):σ′​\(x\+μ^i,t\)​e−2​μ¯≤σ′​\(x\+μ~\)≤σ′​\(x\+μ^i,t\)​e2​μ¯\\sigma^\{\\prime\}\(x\+\\hat\{\\mu\}\_\{i,t\}\)e^\{\-2\\bar\{\\mu\}\}\\leq\\sigma^\{\\prime\}\(x\+\\tilde\{\\mu\}\)\\leq\\sigma^\{\\prime\}\(x\+\\hat\{\\mu\}\_\{i,t\}\)e^\{2\\bar\{\\mu\}\}\. Thus,−ρ¯​\|σ​\(x\+μ^i,t\)−σ​\(x\+μi∗\)\|≥−ρ¯​\|μ^i,t−μi∗\|​min⁡\(1/4,e2​μ¯​σ′​\(x\+μ^i,t\)\)\-\\bar\{\\rho\}\|\\sigma\(x\+\\hat\{\\mu\}\_\{i,t\}\)\-\\sigma\(x\+\\mu^\{\*\}\_\{i\}\)\|\\geq\-\\bar\{\\rho\}\|\\hat\{\\mu\}\_\{i,t\}\-\\mu\_\{i\}^\{\*\}\|\\min\(\\nicefrac\{\{1\}\}\{\{4\}\},e^\{2\\bar\{\\mu\}\}\\sigma^\{\\prime\}\(x\+\\hat\{\\mu\}\_\{i,t\}\)\), where the arguments of the minimum come from the Lipschitz continuity and self\-concordance approaches\. For brevity, letκ​\(x,μ^i,t\)=min⁡\(1/4,e2​μ¯​σ′​\(x\+μ^i,t\)\)\\kappa\(x,\\hat\{\\mu\}\_\{i,t\}\)=\\min\(\\nicefrac\{\{1\}\}\{\{4\}\},e^\{2\\bar\{\\mu\}\}\\sigma^\{\\prime\}\(x\+\\hat\{\\mu\}\_\{i,t\}\)\)\. Adding and subtracting the cross termγ​𝐄μ^t​\[J∗​\(z⊤​θ∗\+w\)\]\\gamma\\mathbf\{E\}\_\{\\hat\{\\mu\}\_\{t\}\}\\left\[J^\{\*\}\(z^\{\\top\}\\theta^\{\*\}\+w\)\\right\], we have:

ΔJ​\(x\)\\displaystyle\\Delta\_\{J\}\(x\)≥𝐄μ^t​\[bt,δ​\(x,ux∗\)\]−ρ¯​κ​\(x,μ^i,t\)​\|μ^i,t−μi∗\|\+γ​𝐄μ^t​\[J∗​\(z⊤​θ^t\+w\)−J∗​\(z⊤​θ∗\+w\)\|x,ux∗\]\\displaystyle\\geq\\mathbf\{E\}\_\{\\hat\{\\mu\}\_\{t\}\}\[b\_\{t,\\delta\}\(x,u^\{\*\}\_\{x\}\)\]\-\\bar\{\\rho\}\\kappa\(x,\\hat\{\\mu\}\_\{i,t\}\)\|\\hat\{\\mu\}\_\{i,t\}\-\\mu^\{\*\}\_\{i\}\|\+\\gamma\\mathbf\{E\}\_\{\\hat\{\\mu\}\_\{t\}\}\[J^\{\*\}\(z^\{\\top\}\\hat\{\\theta\}\_\{t\}\+w\)\-J^\{\*\}\(z^\{\\top\}\\theta^\{\*\}\+w\)\|x,u^\{\*\}\_\{x\}\]\+γ​𝐄μ^t​\[J∗​\(z⊤​θ∗\+w\)\|x,ux∗\]−γ​𝐄μ∗​\[J∗​\(z⊤​θ∗\+w\)\|x,ux∗\]\\displaystyle\\qquad\+\\gamma\\mathbf\{E\}\_\{\\hat\{\\mu\}\_\{t\}\}\[J^\{\*\}\(z^\{\\top\}\\theta^\{\*\}\+w\)\|x,u^\{\*\}\_\{x\}\]\-\\gamma\\mathbf\{E\}\_\{\\mu^\{\*\}\}\[J^\{\*\}\(z^\{\\top\}\\theta^\{\*\}\+w\)\|x,u^\{\*\}\_\{x\}\]\(25\)≥𝐄μ^t​\[bt,δ​\(x,ux∗\)\]−ρ¯​κ​\(x,μ^i,t\)​\|μ^i,t−μi∗\|−γ​𝐄μ^t​\[\|J∗​\(z⊤​θ^t\+w\)−J∗​\(z⊤​θ∗\+w\)\|\|x,ux∗\]\\displaystyle\\geq\\mathbf\{E\}\_\{\\hat\{\\mu\}\_\{t\}\}\[b\_\{t,\\delta\}\(x,u^\{\*\}\_\{x\}\)\]\-\\bar\{\\rho\}\\kappa\(x,\\hat\{\\mu\}\_\{i,t\}\)\|\\hat\{\\mu\}\_\{i,t\}\-\\mu^\{\*\}\_\{i\}\|\-\\gamma\\mathbf\{E\}\_\{\\hat\{\\mu\}\_\{t\}\}\[\|J^\{\*\}\(z^\{\\top\}\\hat\{\\theta\}\_\{t\}\+w\)\-J^\{\*\}\(z^\{\\top\}\\theta^\{\*\}\+w\)\|\\big\|x,u^\{\*\}\_\{x\}\]\+γ​\(σ​\(x\+μ^i,t\)−σ​\(x\+μi∗\)\)​𝐄​\[J∗​\(z1⊤​θ∗\+w\)−J∗​\(z0⊤​θ∗\+w\)\]\\displaystyle\\qquad\+\\gamma\(\\sigma\(x\+\\hat\{\\mu\}\_\{i,t\}\)\-\\sigma\(x\+\\mu\_\{i\}^\{\*\}\)\)\\mathbf\{E\}\[J^\{\*\}\(z\_\{1\}^\{\\top\}\\theta^\{\*\}\+w\)\-J^\{\*\}\(z\_\{0\}^\{\\top\}\\theta^\{\*\}\+w\)\]\(26\)wherez1=\[x​ei⊤​ei⊤\]⊤z\_\{1\}=\[x\\ e\_\{i\}^\{\\top\}\\ e\_\{i\}^\{\\top\}\]^\{\\top\}andz0=\[x​ei⊤​0M⊤\]⊤z\_\{0\}=\[x\\ e\_\{i\}^\{\\top\}\\ \\bm\{0\}\_\{M\}^\{\\top\}\]^\{\\top\}denote state vectors for the adherence outcomes\. We further simplify using theL1L\_\{1\}\-Lipschitz continuity ofJ∗J^\{\*\}\(Lemma[13\.10](https://arxiv.org/html/2605.24261#S13.Thmtheorem10)\), expanding the Bernoulli outcomes, the confidence set bounds forθ^\\hat\{\\theta\}andμ^\\hat\{\\mu\}\(Propositions[5\.1](https://arxiv.org/html/2605.24261#S5.Thmtheorem1)and[5\.2](https://arxiv.org/html/2605.24261#S5.Thmtheorem2)\) under eventsℰθ​\(δ/2\)\\mathcal\{E\}\_\{\\theta\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)andℰμ​\(δ/2\)\\mathcal\{E\}\_\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\), and Jensen’s inequality:

ΔJ​\(x\)\\displaystyle\\Delta\_\{J\}\(x\)≥𝐄μ^t​\[bt,δ​\(x,ux∗\)\]−ρ¯​κ​\(x,μ^i,t\)​\|μ^i,t−μi∗\|−γ​L1​‖θ^t−θ∗‖V¯t​𝐄μ^t​\[‖z‖V¯t−1\|x,ux∗\]\\displaystyle\\geq\\mathbf\{E\}\_\{\\hat\{\\mu\}\_\{t\}\}\[b\_\{t,\\delta\}\(x,u^\{\*\}\_\{x\}\)\]\-\\bar\{\\rho\}\\kappa\(x,\\hat\{\\mu\}\_\{i,t\}\)\|\\hat\{\\mu\}\_\{i,t\}\-\\mu^\{\*\}\_\{i\}\|\-\\gamma L\_\{1\}\\\|\\hat\{\\theta\}\_\{t\}\-\\theta^\{\*\}\\\|\_\{\\bar\{V\}\_\{t\}\}\\mathbf\{E\}\_\{\\hat\{\\mu\}\_\{t\}\}\[\\\|z\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}\|x,u^\{\*\}\_\{x\}\]−γ​\|σ​\(x\+μ^i,t\)−σ​\(x\+μi∗\)\|​𝐄​\[L1​\|\(z1−z0\)⊤​θ∗\|\|x,ux∗\]\\displaystyle\\qquad\-\\gamma\|\\sigma\(x\+\\hat\{\\mu\}\_\{i,t\}\)\-\\sigma\(x\+\\mu\_\{i\}^\{\*\}\)\|\\mathbf\{E\}\[L\_\{1\}\|\(z\_\{1\}\-z\_\{0\}\)^\{\\top\}\\theta^\{\*\}\|\|x,u^\{\*\}\_\{x\}\]\(27\)≥bt,δ​\(x,ux∗\)−ρ¯​κ​\(x,μ^i,t\)​αi,tμ​\(δ/2\)−γ​L1​\(αtθ​\(δ/2\)​𝐄μ^t​\[‖z‖V¯t−1\|x,ux∗\]\+c¯​κ​\(x,μ^i,t\)​αi,tμ​\(δ/2\)\)\\displaystyle\\geq b\_\{t,\\delta\}\(x,u^\{\*\}\_\{x\}\)\-\\bar\{\\rho\}\\kappa\(x,\\hat\{\\mu\}\_\{i,t\}\)\\alpha\_\{i,t\}^\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\-\\gamma L\_\{1\}\(\\alpha\_\{t\}^\{\\theta\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\\mathbf\{E\}\_\{\\hat\{\\mu\}\_\{t\}\}\[\\\|z\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}\|x,u^\{\*\}\_\{x\}\]\+\\bar\{c\}\\kappa\(x,\\hat\{\\mu\}\_\{i,t\}\)\\alpha\_\{i,t\}^\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\)\(28\)The bonus is constructed such that the RHS exactly cancels to0\. Ifux∗=𝟎Mu^\{\*\}\_\{x\}=\\bm\{0\}\_\{M\}, the expression simplifies significantly \(rewards and terms from the differentμ\\muparameterizations cancel\) toΔJ​\(x\)≥bt,δ​\(x,ux∗\)−γ​L1​αtθ​\(δ/2\)​𝐄μ^t​\[‖z‖V¯t−1\|x,ux∗\]≥0\\Delta\_\{J\}\(x\)\\geq b\_\{t,\\delta\}\(x,u^\{\*\}\_\{x\}\)\-\\gamma L\_\{1\}\\alpha\_\{t\}^\{\\theta\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\\mathbf\{E\}\_\{\\hat\{\\mu\}\_\{t\}\}\[\\\|z\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}\|x,u^\{\*\}\_\{x\}\]\\geq 0\. Thus we have that𝒯~t,δ​J∗​\(x\)≥J∗​\(x\)\\tilde\{\\mathcal\{T\}\}\_\{t,\\delta\}J^\{\*\}\(x\)\\geq J^\{\*\}\(x\)for arbitraryx∈𝒳x\\in\\mathcal\{X\}and time steptt\.𝒯~t,δ\\tilde\{\\mathcal\{T\}\}\_\{t,\\delta\}is monotone \(Lemma[5\.3](https://arxiv.org/html/2605.24261#S5.Thmtheorem3)\), so we have by induction onnnthat𝒯~t,δn​J∗​\(x\)≥J∗​\(x\)\\tilde\{\\mathcal\{T\}\}\_\{t,\\delta\}^\{n\}J^\{\*\}\(x\)\\geq J^\{\*\}\(x\)for alln∈ℕn\\in\\mathbb\{N\}\. Suppose for a contradiction that∃x′∈𝒳\\exists x^\{\\prime\}\\in\\mathcal\{X\}withJ~t,δ​\(x′\)<J∗​\(x′\)\\tilde\{J\}\_\{t,\\delta\}\(x^\{\\prime\}\)<J^\{\*\}\(x^\{\\prime\}\)\. Letϵ:=J∗​\(x′\)−J~t,δ​\(x′\)\>0\\epsilon:=J^\{\*\}\(x^\{\\prime\}\)\-\\tilde\{J\}\_\{t,\\delta\}\(x^\{\\prime\}\)\>0\. Per Lemma[5\.3](https://arxiv.org/html/2605.24261#S5.Thmtheorem3),𝒯~t,δ​J\\tilde\{\\mathcal\{T\}\}\_\{t,\\delta\}Jconverges uniformly toJ~t,δ\\tilde\{J\}\_\{t,\\delta\}\. Thus∃N∈ℕ\\exists N\\in\\mathbb\{N\}such that\|𝒯~t,δn​J∗​\(x′\)−J~t,δ​\(x′\)\|<ϵ\|\\tilde\{\\mathcal\{T\}\}\_\{t,\\delta\}^\{n\}J^\{\*\}\(x^\{\\prime\}\)\-\\tilde\{J\}\_\{t,\\delta\}\(x^\{\\prime\}\)\|<\\epsilonforn≥Nn\\geq N\. Thus we have𝒯~t,δN​J∗​\(x′\)<J~t,δ​\(x′\)\+ϵ=J~t,δ​\(x′\)\+J∗​\(x′\)−J~t,δ​\(x′\)=J∗​\(x′\)\\tilde\{\\mathcal\{T\}\}^\{N\}\_\{t,\\delta\}J^\{\*\}\(x^\{\\prime\}\)<\\tilde\{J\}\_\{t,\\delta\}\(x^\{\\prime\}\)\+\\epsilon=\\tilde\{J\}\_\{t,\\delta\}\(x^\{\\prime\}\)\+J^\{\*\}\(x^\{\\prime\}\)\-\\tilde\{J\}\_\{t,\\delta\}\(x^\{\\prime\}\)=J^\{\*\}\(x^\{\\prime\}\), producing a contradiction and the lemma follows\.\\halmos

### 13\.3Lipschitz constants

For the supporting lemmas described here, we assume that Assumptions[3\.1](https://arxiv.org/html/2605.24261#S3.SS1)\-[3\.1](https://arxiv.org/html/2605.24261#S3.SS1)hold\.

###### Lemma 13\.15

\(Reward Lipschitz\)Forθ∈Θ\\theta\\in\\Theta,μ∈Φ\\mu\\in\\Phi,𝐄θ,μ​\[r​\(x,u,d\)\]\\mathbf\{E\}\_\{\\theta,\\mu\}\[r\(x,u,d\)\]is uniformlyLrL\_\{r\}\-Lipschitz continuous inxxover𝒳\\mathcal\{X\}, withLr=\(β\+ρ¯\)/4L\_\{r\}=\\nicefrac\{\{\(\\beta\+\\bar\{\\rho\}\)\}\}\{\{4\}\}\.

###### Proof 13\.16

Proof of Lemma[13\.15](https://arxiv.org/html/2605.24261#S13.Thmtheorem15)Letx1,x2∈𝒳x\_\{1\},x\_\{2\}\\in\\mathcal\{X\},u∈𝒰u\\in\\mathcal\{U\},θ∈Θ\\theta\\in\\Theta,μ∈Φ\\mu\\in\\Phi,Δ​\(u,x1,x2\):=𝐄θ,μ​\[r​\(x1,u,d\)−r​\(x2,u,d\)\]\\Delta\(u,x\_\{1\},x\_\{2\}\):=\\mathbf\{E\}\_\{\\theta,\\mu\}\[r\(x\_\{1\},u,d\)\-r\(x\_\{2\},u,d\)\]\. ThenΔ​\(u,x1,x2\)=−β​\(σ​\(β0−x1\)−σ​\(β0−x2\)\)\+∑i=1Mui​ρi​\(σ​\(x1\+μi\)−σ​\(x2\+μi\)\)\\Delta\(u,x\_\{1\},x\_\{2\}\)=\-\\beta\\big\(\\sigma\(\\beta\_\{0\}\-x\_\{1\}\)\-\\sigma\(\\beta\_\{0\}\-x\_\{2\}\)\\big\)\+\\sum\_\{i=1\}^\{M\}u^\{i\}\\rho\_\{i\}\\big\(\\sigma\(x\_\{1\}\+\\mu\_\{i\}\)\-\\sigma\(x\_\{2\}\+\\mu\_\{i\}\)\\big\)\. Applying the triangle inequality and the sigmoid’s1/4\\nicefrac\{\{1\}\}\{\{4\}\}\-Lipschitz continuity givesΔ​\(u,x1,x2\)≤β\+ρ¯4​\|x1−x2\|\\Delta\(u,x\_\{1\},x\_\{2\}\)\\leq\\frac\{\\beta\+\\bar\{\\rho\}\}\{4\}\|x\_\{1\}\-x\_\{2\}\|\. The Lipschitz result follows by a symmetrical argument \(swapx1x\_\{1\}andx2x\_\{2\}\)\.\\halmos

###### Lemma 13\.17

\(Value\-to\-go Lipschitz\)For anyθ∈Θ\\theta\\in\\Thetaandμ∈Φ\\mu\\in\\Phi, if a functionJJisLL\-Lipschitz and‖J‖∞≤J¯\\\|J\\\|\_\{\\infty\}\\leq\\bar\{J\}, then𝐄θ,μ​\[J​\(f​\(x,u,d,w\)\)\]\\mathbf\{E\}\_\{\\theta,\\mu\}\\left\[J\(f\(x,u,d,w\)\)\\right\]isLvL\_\{v\}\-Lipschitz withLv=L​a¯\+J¯/2L\_\{v\}=L\\bar\{a\}\+\\nicefrac\{\{\\bar\{J\}\}\}\{\{2\}\}\.

###### Proof 13\.18

Proof of Lemma[13\.17](https://arxiv.org/html/2605.24261#S13.Thmtheorem17)Letx1,x2∈𝒳x\_\{1\},x\_\{2\}\\in\\mathcal\{X\},u=ei∈𝒰∖𝟎Mu=e\_\{i\}\\in\\mathcal\{U\}\\setminus\\bm\{0\}\_\{M\},θ∈Θ\\theta\\in\\Theta, andμ∈Φ\\mu\\in\\Phi\. LetΔ​\(u,x1,x2\):=𝐄θ,μ​\[J​\(f​\(x1,ei,d,w\)\)\]−𝐄θ,μ​\[J​\(f​\(x2,ei,d,w\)\)\]\\Delta\(u,x\_\{1\},x\_\{2\}\):=\\mathbf\{E\}\_\{\\theta,\\mu\}\\left\[J\(f\(x\_\{1\},e\_\{i\},d,w\)\)\\right\]\-\\mathbf\{E\}\_\{\\theta,\\mu\}\\left\[J\(f\(x\_\{2\},e\_\{i\},d,w\)\)\\right\]\. LetJ1​\(x\):=J​\(a​x\+bi\+ci\+w\)J\_\{1\}\(x\):=J\(ax\+b\_\{i\}\+c\_\{i\}\+w\), andJ0​\(x\):=J​\(a​x\+bi\+w\)J\_\{0\}\(x\):=J\(ax\+b\_\{i\}\+w\)\. Expanding the expectation with respect to adherence and simplifying, we have:

Δ​\(u,x1,x2\)=𝐄​\[J0​\(x1\)\+p^i​\(x1,1\)​\(J1​\(x1\)−J0​\(x1\)\)−J0​\(x2\)−p^i​\(x2,1\)​\(J1​\(x2\)−J0​\(x2\)\)\]\\Delta\(u,x\_\{1\},x\_\{2\}\)=\\mathbf\{E\}\[J\_\{0\}\(x\_\{1\}\)\+\\hat\{p\}\_\{i\}\(x\_\{1\},1\)\(J\_\{1\}\(x\_\{1\}\)\-J\_\{0\}\(x\_\{1\}\)\)\-J\_\{0\}\(x\_\{2\}\)\-\\hat\{p\}\_\{i\}\(x\_\{2\},1\)\(J\_\{1\}\(x\_\{2\}\)\-J\_\{0\}\(x\_\{2\}\)\)\]\(29\)Add and subtract𝐄​\[p^i​\(x2,1\)​\[J1​\(x1\)−J0​\(x1\)\]\]\\mathbf\{E\}\[\\hat\{p\}\_\{i\}\(x\_\{2\},1\)\[J\_\{1\}\(x\_\{1\}\)\-J\_\{0\}\(x\_\{1\}\)\]\], rearrange, and apply the triangle inequality:

\([29](https://arxiv.org/html/2605.24261#S13.E29)\)≤𝐄\[\|\(1−p^i​\(x2,1\)\)​\[J0​\(x1\)−J0​\(x2\)\]\|⏟\(i\)\+\|\(p^i​\(x1,1\)−p^i​\(x2,1\)\)​\[J1​\(x1\)−J0​\(x1\)\]\|⏟\(i​i\)\+\|p^i​\(x2,1\)​\[J1​\(x1\)−J1​\(x2\)\]\|⏟\(i​i​i\)\]\\eqref\{eq:delta\_dff\_ec\}\\leq\\mathbf\{E\}\[\\underbrace\{\|\(1\-\\hat\{p\}\_\{i\}\(x\_\{2\},1\)\)\[J\_\{0\}\(x\_\{1\}\)\-J\_\{0\}\(x\_\{2\}\)\]\|\}\_\{\(i\)\}\+\\underbrace\{\|\(\\hat\{p\}\_\{i\}\(x\_\{1\},1\)\-\\hat\{p\}\_\{i\}\(x\_\{2\},1\)\)\[J\_\{1\}\(x\_\{1\}\)\-J\_\{0\}\(x\_\{1\}\)\]\|\}\_\{\(ii\)\}\\\\ \+\\underbrace\{\|\\hat\{p\}\_\{i\}\(x\_\{2\},1\)\[J\_\{1\}\(x\_\{1\}\)\-J\_\{1\}\(x\_\{2\}\)\]\|\}\_\{\(iii\)\}\]\(30\)JJisLL\-Lipschitz by assumption, so\(i\)≤\(1−p^i​\(x2,1\)\)​L​\|a\|​\|x1−x2\|\(i\)\\leq\(1\-\\hat\{p\}\_\{i\}\(x\_\{2\},1\)\)L\|a\|\|x\_\{1\}\-x\_\{2\}\|and\(i​i​i\)≤p^i​\(x2,1\)​L​\|a\|​\|x1−x2\|\(iii\)\\leq\\hat\{p\}\_\{i\}\(x\_\{2\},1\)L\|a\|\|x\_\{1\}\-x\_\{2\}\|\. The sigmoid is1/4\\nicefrac\{\{1\}\}\{\{4\}\}\-Lipschitz and‖J‖∞≤J¯\\\|J\\\|\_\{\\infty\}\\leq\\bar\{J\}, so\(i​i\)≤2​J¯/4​\|x1−x2\|\(ii\)\\leq\\nicefrac\{\{2\\bar\{J\}\}\}\{\{4\}\}\|x\_\{1\}\-x\_\{2\}\|\. Recall that\|a\|≤a¯\|a\|\\leq\\bar\{a\}asθ∈Θ\\theta\\in\\Theta\. The expression no longer depends onwwso the expectation drops and we haveΔ​\(u,x1,x2\)≤\[L​a¯\+J¯/2\]​\|x1−x2\|\\Delta\(u,x\_\{1\},x\_\{2\}\)\\leq\[L\\bar\{a\}\+\\nicefrac\{\{\\bar\{J\}\}\}\{\{2\}\}\]\|x\_\{1\}\-x\_\{2\}\|\. This constant also holds for theu=𝟎Mu=\\bm\{0\}\_\{M\}case\. The Lipschitz result follows by a symmetrical argument withx1x\_\{1\}andx2x\_\{2\}swapped\.\\halmos

###### Lemma 13\.19

\(Bonus Lipschitz\)bt,δ​\(x,u\)b\_\{t,\\delta\}\(x,u\)is uniformlyLbL\_\{b\}\-Lipschitz continuous inxxon𝒳\\mathcal\{X\}withLb=e2​μ¯/4​\(ρ¯\+γ​L1​c¯\)​α¯μ​\(δ/2\)\+5​γ​L1​αTθ​\(δ/2\)4​λ1L\_\{b\}=\\nicefrac\{\{e^\{2\\bar\{\\mu\}\}\}\}\{\{4\}\}\(\\bar\{\\rho\}\+\\gamma L\_\{1\}\\bar\{c\}\)\\overline\{\\alpha\}^\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\+\\frac\{5\\gamma L\_\{1\}\\alpha^\{\\theta\}\_\{T\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\}\{4\\sqrt\{\\lambda\_\{1\}\}\}for anyt∈\{1,…,T\}t\\in\\\{1,\\dots,T\\\}, andδ∈\(0,1\)\\delta\\in\(0,1\)\.

###### Proof 13\.20

Proof of Lemma[13\.19](https://arxiv.org/html/2605.24261#S13.Thmtheorem19)Considerx1,x2∈𝒳x\_\{1\},x\_\{2\}\\in\\mathcal\{X\},u=ei∈𝒰∖𝟎Mu=e\_\{i\}\\in\\mathcal\{U\}\\setminus\\bm\{0\}\_\{M\},δ∈\(0,1\)\\delta\\in\(0,1\), andt∈\{1,…,T\}t\\in\\\{1,\\dots,T\\\}\. DefineΔt​\(u,x1,x2\):=bt,δ​\(x1,u\)−bt,δ​\(x2,u\)\\Delta\_\{t\}\(u,x\_\{1\},x\_\{2\}\):=b\_\{t,\\delta\}\(x\_\{1\},u\)\-b\_\{t,\\delta\}\(x\_\{2\},u\)\. Consider the difference of theμ\\mu\-derived bonus term:Δtμ​\(u,x1,x2\):=\(ρ¯\+γ​L1​c¯\)​αi,tμ​\(δ/2\)​\[κ​\(x1,μ^i,t\)−κ​\(x2,μ^i,t\)\]\\Delta\_\{t\}^\{\\mu\}\(u,x\_\{1\},x\_\{2\}\):=\(\\bar\{\\rho\}\+\\gamma L\_\{1\}\\bar\{c\}\)\\alpha\_\{i,t\}^\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\[\\kappa\(x\_\{1\},\\hat\{\\mu\}\_\{i,t\}\)\-\\kappa\(x\_\{2\},\\hat\{\\mu\}\_\{i,t\}\)\]\. Recall thatκ​\(x,μ\):=min⁡\(1/4,e2​μ¯​σ′​\(x\+μ\)\)\\kappa\(x,\\mu\):=\\min\(\\nicefrac\{\{1\}\}\{\{4\}\},e^\{2\\bar\{\\mu\}\}\\sigma^\{\\prime\}\(x\+\\mu\)\)\. Note the sigmoid’s1/4\\nicefrac\{\{1\}\}\{\{4\}\}\-Lipschitz continuity and\|σ′′​\(x\+μ\)\|=\|σ′​\(x\+μ\)​\(1−2​σ​\(x\+μ\)\)\|≤1/4⋅1\|\\sigma^\{\\prime\\prime\}\(x\+\\mu\)\|=\|\\sigma^\{\\prime\}\(x\+\\mu\)\(1\-2\\sigma\(x\+\\mu\)\)\|\\leq\\nicefrac\{\{1\}\}\{\{4\}\}\\cdot 1, implyinge2​μ¯​σ′​\(x\+μ\)e^\{2\\bar\{\\mu\}\}\\sigma^\{\\prime\}\(x\+\\mu\)ise2​μ¯/4\\nicefrac\{\{e^\{2\\bar\{\\mu\}\}\}\}\{\{4\}\}\-Lipschitz\. Since the minimum with a constant is non\-expansive, the composition \(i\.e\.,κ​\(x,μ\)\\kappa\(x,\\mu\)\) ise2​μ¯/4\\nicefrac\{\{e^\{2\\bar\{\\mu\}\}\}\}\{\{4\}\}\-Lipschitz\. ThusΔtμ​\(u,x1,x2\)≤e2​μ¯/4​\(ρ¯\+γ​L1​c¯\)​αi,tμ​\(δ/2\)​\|x1−x2\|\\Delta^\{\\mu\}\_\{t\}\(u,x\_\{1\},x\_\{2\}\)\\leq\\nicefrac\{\{e^\{2\\bar\{\\mu\}\}\}\}\{\{4\}\}\(\\bar\{\\rho\}\+\\gamma L\_\{1\}\\bar\{c\}\)\\alpha\_\{i,t\}^\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\|x\_\{1\}\-x\_\{2\}\|\. Per Lemma[13\.8](https://arxiv.org/html/2605.24261#S13.Thmtheorem8), we boundαi,tμ​\(δ/2\)≤α¯μ​\(δ/2\)\\alpha\_\{i,t\}^\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\\leq\\overline\{\\alpha\}^\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\. Next consider theθ\\theta\-derived bonus term \(i\.e\.,Δt​\(u,x1,x2\)=Δtμ​\(u,x1,x2\)\+Δtθ​\(u,x1,x2\)\\Delta\_\{t\}\(u,x\_\{1\},x\_\{2\}\)=\\Delta\_\{t\}^\{\\mu\}\(u,x\_\{1\},x\_\{2\}\)\+\\Delta\_\{t\}^\{\\theta\}\(u,x\_\{1\},x\_\{2\}\)\), lettingp^​\(x\):=σ​\(x\+μ^i\)\\hat\{p\}\(x\):=\\sigma\(x\+\\hat\{\\mu\}\_\{i\}\):

Δtθ​\(u,x1,x2\)\\displaystyle\\Delta^\{\\theta\}\_\{t\}\(u,x\_\{1\},x\_\{2\}\)=γ​L1​αtθ​\(δ/2\)​\[𝐄μ^t​\[‖z‖V¯t−1\|x1,u=ei\]−𝐄μ^t​\[‖z‖V¯t−1\|x2,u=ei\]\]\\displaystyle=\\gamma L\_\{1\}\\alpha^\{\\theta\}\_\{t\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\[\\mathbf\{E\}\_\{\\hat\{\\mu\}\_\{t\}\}\[\\\|z\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}\|x\_\{1\},u=e\_\{i\}\]\-\\mathbf\{E\}\_\{\\hat\{\\mu\}\_\{t\}\}\[\\\|z\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}\|x\_\{2\},u=e\_\{i\}\]\]\(31\)=γL1αtθ\(δ/2\)\[p^\(x1\)\(∥z1,1∥V¯t−1−∥z1,0∥V¯t−1\)−p^\(x2\)\(∥z2,1∥V¯t−1−∥z2,0∥V¯t−1\)\\displaystyle=\\gamma L\_\{1\}\\alpha^\{\\theta\}\_\{t\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\[\\hat\{p\}\(x\_\{1\}\)\(\\\|z\_\{1,1\}\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}\-\\\|z\_\{1,0\}\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}\)\-\\hat\{p\}\(x\_\{2\}\)\(\\\|z\_\{2,1\}\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}\-\\\|z\_\{2,0\}\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}\)\+\(∥z1,0∥V¯t−1−∥z2,0∥V¯t−1\)±p^\(x1\)\(∥z2,1∥V¯t−1−∥z2,0∥V¯t−1\)\]\\displaystyle\\qquad\+\(\\\|z\_\{1,0\}\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}\-\\\|z\_\{2,0\}\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}\)\\pm\\hat\{p\}\(x\_\{1\}\)\(\\\|z\_\{2,1\}\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}\-\\\|z\_\{2,0\}\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}\)\]\(32\)wherezj,k=\[xj​u⊤​dki⊤\]⊤z\_\{j,k\}=\[x\_\{j\}\\ u^\{\\top\}\\ \{d^\{i\}\_\{k\}\}^\{\\top\}\]^\{\\top\}anddki=k​eid^\{i\}\_\{k\}=ke\_\{i\}\. Thus we have:

Δtθ​\(u,x1,x2\)\\displaystyle\\Delta^\{\\theta\}\_\{t\}\(u,x\_\{1\},x\_\{2\}\)=γL1αtθ\(δ/2\)\[p^\(x1\)\(∥z1,1∥V¯t−1−∥z2,1∥V¯t−1\)\+\(1−p^\(x1\)\)\(∥z1,0∥V¯t−1−∥z2,0∥V¯t−1\)\\displaystyle=\\gamma L\_\{1\}\\alpha^\{\\theta\}\_\{t\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\[\\hat\{p\}\(x\_\{1\}\)\(\\\|z\_\{1,1\}\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}\-\\\|z\_\{2,1\}\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}\)\+\(1\-\\hat\{p\}\(x\_\{1\}\)\)\(\\\|z\_\{1,0\}\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}\-\\\|z\_\{2,0\}\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}\)\+\(p^\(x1\)−p^\(x2\)\)\(∥z2,1∥V¯t−1−∥z2,0∥V¯t−1\)\]\\displaystyle\\qquad\+\(\\hat\{p\}\(x\_\{1\}\)\-\\hat\{p\}\(x\_\{2\}\)\)\(\\\|z\_\{2,1\}\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}\-\\\|z\_\{2,0\}\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}\)\]\(33\)≤\(a\)​γ​L1​αtθ​\(δ/2\)​\[‖z1,1−z2,1‖V¯t−1\+1/4​\|x1−x2\|​‖z2,1−z2,0‖V¯t−1\]\\displaystyle\\overset\{\(a\)\}\{\\leq\}\\gamma L\_\{1\}\\alpha^\{\\theta\}\_\{t\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\[\\\|z\_\{1,1\}\-z\_\{2,1\}\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}\+\\nicefrac\{\{1\}\}\{\{4\}\}\|x\_\{1\}\-x\_\{2\}\|\\\|z\_\{2,1\}\-z\_\{2,0\}\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}\]\(34\)≤\(b\)​γ​L1​αtθ​\(δ/2\)​\|x1−x2\|​1/λ1​\[1\+1/4\]\\displaystyle\\overset\{\(b\)\}\{\\leq\}\\gamma L\_\{1\}\\alpha^\{\\theta\}\_\{t\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\|x\_\{1\}\-x\_\{2\}\|\\nicefrac\{\{1\}\}\{\{\\sqrt\{\\lambda\_\{1\}\}\}\}\[1\+\\nicefrac\{\{1\}\}\{\{4\}\}\]\(35\)\(a\)\(a\)comes from applying the reverse triangle inequality to the norm differences, the fact thatz1,1−z2,1=z1,0−z2,0z\_\{1,1\}\-z\_\{2,1\}=z\_\{1,0\}\-z\_\{2,0\}to combinep^​\(x1\)\\hat\{p\}\(x\_\{1\}\)terms, and the sigmoid’s1/4\\nicefrac\{\{1\}\}\{\{4\}\}\-Lipschitz continuity\.\(b\)\(b\)follows from evaluating the norms and because the diagonal elements ofV¯t−1\\bar\{V\}\_\{t\}^\{\-1\}are upper bounded by1/λ1\\nicefrac\{\{1\}\}\{\{\\lambda\_\{1\}\}\}\.αtθ​\(δ/2\)\\alpha^\{\\theta\}\_\{t\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)is increasing intt, soαtθ​\(δ/2\)≤αTθ​\(δ/2\)\\alpha^\{\\theta\}\_\{t\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\\leq\\alpha^\{\\theta\}\_\{T\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)for aTT\-length trajectory\. Substituting theΔtμ\\Delta^\{\\mu\}\_\{t\}andΔtθ\\Delta^\{\\theta\}\_\{t\}results we obtainΔt​\(u,x1,x2\)≤\[e2​μ¯/4​\(ρ¯\+γ​L1​c¯\)​α¯μ​\(δ/2\)\+5​γ​L1​αTθ​\(δ/2\)4​λ1\]​\|x1−x2\|\\Delta\_\{t\}\(u,x\_\{1\},x\_\{2\}\)\\leq\[\\nicefrac\{\{e^\{2\\bar\{\\mu\}\}\}\}\{\{4\}\}\(\\bar\{\\rho\}\+\\gamma L\_\{1\}\\bar\{c\}\)\\overline\{\\alpha\}^\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\+\\frac\{5\\gamma L\_\{1\}\\alpha^\{\\theta\}\_\{T\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\}\{4\\sqrt\{\\lambda\_\{1\}\}\}\]\|x\_\{1\}\-x\_\{2\}\|\. The same holds ifu=𝟎Mu=\\bm\{0\}\_\{M\}\. The Lipschitz result follows by a symmetrical argument withx1x\_\{1\}andx2x\_\{2\}swapped\.\\halmos

### 13\.4Regret proof and supporting lemmas

###### Proof 13\.21

Proof of Theorem[5\.5](https://arxiv.org/html/2605.24261#S5.Thmtheorem5)We divideδa​l​l\\delta\_\{all\}into thirds such that the proposition holds after a union bound\. Letδ=δa​l​l/3\\delta=\\nicefrac\{\{\\delta\_\{all\}\}\}\{\{3\}\}be used to establish the optimistic surrogate system, i\.e\., with probability at least1−δa​l​l/31\-\\nicefrac\{\{\\delta\_\{all\}\}\}\{\{3\}\},ℰθ​\(δ/2\)\\mathcal\{E\}\_\{\\theta\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\(Proposition[5\.1](https://arxiv.org/html/2605.24261#S5.Thmtheorem1)\) andℰμ​\(δ/2\)\\mathcal\{E\}\_\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\(Proposition[5\.2](https://arxiv.org/html/2605.24261#S5.Thmtheorem2)\) hold\. The remaining2​δa​l​l/3\\nicefrac\{\{2\\delta\_\{all\}\}\}\{\{3\}\}is used for two martingale concentration arguments in the regret bound, each usingδa​l​l/3\\nicefrac\{\{\\delta\_\{all\}\}\}\{\{3\}\}\. We upper bound the regret using optimistic value functionsJ~t,δ\\tilde\{J\}\_\{t,\\delta\}\. This proof relies on a telescoping\-style simplification, so it is helpful to index on time stepsttrather than epochs\. We add the∘superscript to denote the quantity the algorithm is using at timett\. We denote the sequence of optimistic Bellman operators\{𝒯~t,δ∘\}t=1T\+1\\\{\\tilde\{\\mathcal\{T\}\}^\{\\circ\}\_\{t,\\delta\}\\\}\_\{t=1\}^\{T\+1\}to be defined using the sequence of bonus functions\{bt,δ∘​\(x,u\)\}t=1T\+1\\\{b^\{\\circ\}\_\{t,\\delta\}\(x,u\)\\\}\_\{t=1\}^\{T\+1\}, estimators\{\(θ^t∘,μ^t∘\)\}t=1T\+1\\\{\(\\hat\{\\theta\}^\{\\circ\}\_\{t\},\\hat\{\\mu\}^\{\\circ\}\_\{t\}\)\\\}\_\{t=1\}^\{T\+1\}, confidence set radii\{\(αtθ,∘,αtμ,∘\)\}t=1T\+1\\\{\(\\alpha^\{\\theta,\\circ\}\_\{t\},\\alpha^\{\\mu,\\circ\}\_\{t\}\)\\\}\_\{t=1\}^\{T\+1\}, and regularized empirical covariances matrices\{V¯t∘\}t=1T\+1\\\{\\bar\{V\}^\{\\circ\}\_\{t\}\\\}\_\{t=1\}^\{T\+1\}specifically used by Algorithm[1](https://arxiv.org/html/2605.24261#alg1), i\.e\.𝒯~t,δ∘​J:=maxu∈𝒰⁡\{𝐄θ^t∘,μ^t∘​\[r​\(x,u,d\)\+bt,δ∘​\(x,u\)\+γ​J​\(f​\(x,u,d,w\)\)\]\}\\tilde\{\\mathcal\{T\}\}^\{\\circ\}\_\{t,\\delta\}J:=\\max\_\{u\\in\\mathcal\{U\}\}\\\{\\mathbf\{E\}\_\{\\hat\{\\theta\}\_\{t\}^\{\\circ\},\\hat\{\\mu\}\_\{t\}^\{\\circ\}\}\[r\(x,u,d\)\+b^\{\\circ\}\_\{t,\\delta\}\(x,u\)\+\\gamma J\(f\(x,u,d,w\)\)\]\\\}\.\{J~t,δ∘\}t=1T\+1\\\{\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\\\}\_\{t=1\}^\{T\+1\}is the sequence of fixed points associated with this sequence of operators \(Lemma[5\.3](https://arxiv.org/html/2605.24261#S5.Thmtheorem3)\)\. As an example, for somet∈\{1,…,T\+1\}t\\in\\\{1,\\dots,T\+1\\\}we have that𝒯~t,δ∘\\tilde\{\\mathcal\{T\}\}^\{\\circ\}\_\{t,\\delta\}is equivalent to𝒯~τk,δ\\tilde\{\\mathcal\{T\}\}\_\{\\tau\_\{k\},\\delta\}, whereτk\\tau\_\{k\}is the time index of the start of epochkkactive at steptt\. By Lemma[5\.4](https://arxiv.org/html/2605.24261#S5.Thmtheorem4), whenℰθ​\(δ/2\)\\mathcal\{E\}\_\{\\theta\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)andℰμ​\(δ/2\)\\mathcal\{E\}\_\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)hold, the sequence of value functions\{J~t,δ∘\}t=1T\+1\\\{\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\\\}\_\{t=1\}^\{T\+1\}overestimate the true value functionJ∗J^\{\*\}fort=1,…,T\+1t=1,\\dots,T\+1\. Substituting into the regret definition:RT=∑t=1TJ∗​\(xt\)−Jπt​\(xt\)≤∑t=1TJ~t,δ∘​\(xt\)−Jπt​\(xt\)R\_\{T\}=\\sum\_\{t=1\}^\{T\}J^\{\*\}\(x\_\{t\}\)\-J^\{\\pi\_\{t\}\}\(x\_\{t\}\)\\leq\\sum\_\{t=1\}^\{T\}\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(x\_\{t\}\)\-J^\{\\pi\_\{t\}\}\(x\_\{t\}\)\. For brevity, defineR~t:=J~t,δ∘​\(xt\)−Jπt​\(xt\)\\tilde\{R\}\_\{t\}:=\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(x\_\{t\}\)\-J^\{\\pi\_\{t\}\}\(x\_\{t\}\), i\.e\.,RT≤∑t=1TR~tR\_\{T\}\\leq\\sum\_\{t=1\}^\{T\}\\tilde\{R\}\_\{t\}\. We expandR~t\\tilde\{R\}\_\{t\}\(a single time step\) then substitute back into the summation for a telescoping simplification:

R~t\\displaystyle\\tilde\{R\}\_\{t\}=maxu∈𝒰⁡\{𝐄μ^t∘,θ^t∘​\[r​\(xt,u,d\)\+bt,δ∘​\(xt,u\)\+γ​J~t,δ∘​\(f​\(xt,u,d,w\)\)\]\}\\displaystyle=\\max\_\{u\\in\\mathcal\{U\}\}\\\{\\mathbf\{E\}\_\{\\hat\{\\mu\}^\{\\circ\}\_\{t\},\\hat\{\\theta\}^\{\\circ\}\_\{t\}\}\[r\(x\_\{t\},u,d\)\+b^\{\\circ\}\_\{t,\\delta\}\(x\_\{t\},u\)\+\\gamma\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(f\(x\_\{t\},u,d,w\)\)\]\\\}−𝐄μ∗,θ∗​\[r​\(xt,πt​\(xt\),d\)\+γ​Jπt​\(f​\(xt,πt​\(xt\),d,w\)\)\]\\displaystyle\\qquad\-\\mathbf\{E\}\_\{\\mu^\{\*\},\\theta^\{\*\}\}\[r\(x\_\{t\},\\pi\_\{t\}\(x\_\{t\}\),d\)\+\\gamma J^\{\\pi\_\{t\}\}\(f\(x\_\{t\},\\pi\_\{t\}\(x\_\{t\}\),d,w\)\)\]\(36\)Per Algorithm[1](https://arxiv.org/html/2605.24261#alg1),πt\\pi\_\{t\}is derived fromJ~t,δ∘\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}, so we can express both usingut:=πt​\(xt\)u\_\{t\}:=\\pi\_\{t\}\(x\_\{t\}\)\. We add and subtract the following:γ​𝐄θ∗,μ^t∘​\[J~t,δ∘​\(f​\(xt,ut,d,w\)\)\]\\gamma\\mathbf\{E\}\_\{\\theta^\{\*\},\\hat\{\\mu\}^\{\\circ\}\_\{t\}\}\[\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(f\(x\_\{t\},u\_\{t\},d,w\)\)\],γ​𝐄θ∗,μ∗​\[J~t,δ∘​\(f​\(xt,ut,d,w\)\)\]\\gamma\\mathbf\{E\}\_\{\\theta^\{\*\},\\mu^\{\*\}\}\[\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(f\(x\_\{t\},u\_\{t\},d,w\)\)\],γ​\[J~t,δ∘​\(xt\+1\)−Jπt​\(xt\+1\)\]\\gamma\[\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(x\_\{t\+1\}\)\-J^\{\\pi\_\{t\}\}\(x\_\{t\+1\}\)\],γ​J~t\+1,δ∘​\(xt\+1\)\\gamma\\tilde\{J\}^\{\\circ\}\_\{t\+1,\\delta\}\(x\_\{t\+1\}\), andγ​Jπt\+1​\(xt\+1\)\\gamma J^\{\\pi\_\{t\+1\}\}\(x\_\{t\+1\}\), wherext\+1x\_\{t\+1\}is the observed state att\+1t\+1\. Rearranging and usingf​\(⋅\)=f​\(xt,ut,d,w\)f\(\\cdot\)=f\(x\_\{t\},u\_\{t\},d,w\)for brevity we have:

R~t\\displaystyle\\tilde\{R\}\_\{t\}=𝐄μ^t∘​\[r​\(xt,ut,d\)\]−𝐄μ∗​\[r​\(xt,ut,d\)\]\+𝐄μ^t∘​\[bt,δ∘​\(xt,ut\)\]\\displaystyle=\\mathbf\{E\}\_\{\\hat\{\\mu\}^\{\\circ\}\_\{t\}\}\[r\(x\_\{t\},u\_\{t\},d\)\]\-\\mathbf\{E\}\_\{\\mu^\{\*\}\}\[r\(x\_\{t\},u\_\{t\},d\)\]\+\\mathbf\{E\}\_\{\\hat\{\\mu\}\_\{t\}^\{\\circ\}\}\[b^\{\\circ\}\_\{t,\\delta\}\(x\_\{t\},u\_\{t\}\)\]\(37\)\+γ​\[𝐄θ∗,μ^t∘​\[J~t,δ∘​\(f​\(⋅\)\)\]−𝐄θ∗,μ∗​\[J~t,δ∘​\(f​\(⋅\)\)\]\]\+γ​\[𝐄θ^t∘,μ^t∘​\[J~t,δ∘​\(f​\(⋅\)\)\]−𝐄θ∗,μ^t∘​\[J~t,δ∘​\(f​\(⋅\)\)\]\]\\displaystyle\\quad\+\\gamma\[\\mathbf\{E\}\_\{\\theta^\{\*\},\\hat\{\\mu\}\_\{t\}^\{\\circ\}\}\[\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(f\(\\cdot\)\)\]\-\\mathbf\{E\}\_\{\\theta^\{\*\},\\mu^\{\*\}\}\[\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(f\(\\cdot\)\)\]\]\+\\gamma\[\\mathbf\{E\}\_\{\\hat\{\\theta\}\_\{t\}^\{\\circ\},\\hat\{\\mu\}\_\{t\}^\{\\circ\}\}\[\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(f\(\\cdot\)\)\]\-\\mathbf\{E\}\_\{\\theta^\{\*\},\\hat\{\\mu\}\_\{t\}^\{\\circ\}\}\[\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(f\(\\cdot\)\)\]\]\+γ​\[𝐄θ∗,μ∗​\[J~t,δ∘​\(f​\(⋅\)\)−Jπt​\(f​\(⋅\)\)\]−\(J~t,δ∘​\(xt\+1\)−Jπt​\(xt\+1\)\)\]\\displaystyle\\quad\+\\gamma\[\\mathbf\{E\}\_\{\\theta^\{\*\},\\mu^\{\*\}\}\[\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(f\(\\cdot\)\)\-J^\{\\pi\_\{t\}\}\(f\(\\cdot\)\)\]\-\(\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(x\_\{t\+1\}\)\-J^\{\\pi\_\{t\}\}\(x\_\{t\+1\}\)\)\]\+γ​\[J~t\+1,δ∘​\(xt\+1\)−Jπt\+1​\(xt\+1\)\]⏟=γ​R~t\+1\+γ​\[J~t,δ∘​\(xt\+1\)−J~t\+1,δ∘​\(xt\+1\)\+Jπt\+1​\(xt\+1\)−Jπt​\(xt\+1\)\]\\displaystyle\\quad\+\\underbrace\{\\gamma\[\\tilde\{J\}^\{\\circ\}\_\{t\+1,\\delta\}\(x\_\{t\+1\}\)\-J^\{\\pi\_\{t\+1\}\}\(x\_\{t\+1\}\)\]\}\_\{=\\gamma\\tilde\{R\}\_\{t\+1\}\}\+\\gamma\[\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(x\_\{t\+1\}\)\-\\tilde\{J\}^\{\\circ\}\_\{t\+1,\\delta\}\(x\_\{t\+1\}\)\+J^\{\\pi\_\{t\+1\}\}\(x\_\{t\+1\}\)\-J^\{\\pi\_\{t\}\}\(x\_\{t\+1\}\)\]
Thus we recover theγ​R~t\+1\\gamma\\tilde\{R\}\_\{t\+1\}term needed for a telescoping\-style simplification\. Let\(∗\)t\(\*\)\_\{t\}denote the non\-braced terms in Eq\.[37](https://arxiv.org/html/2605.24261#S13.E37)\. Substituting \([37](https://arxiv.org/html/2605.24261#S13.E37)\) into the regret summation givesRT≤∑t=1TR~t=∑t=1T\[\(∗\)t\+γ​R~t\+1\]=γ​R~T\+1\+∑t=1T\(∗\)t\+γ​∑t=2TR~tR\_\{T\}\\leq\\sum\_\{t=1\}^\{T\}\\tilde\{R\}\_\{t\}=\\sum\_\{t=1\}^\{T\}\[\(\*\)\_\{t\}\+\\gamma\\tilde\{R\}\_\{t\+1\}\]=\\gamma\\tilde\{R\}\_\{T\+1\}\+\\sum\_\{t=1\}^\{T\}\(\*\)\_\{t\}\+\\gamma\\sum\_\{t=2\}^\{T\}\\tilde\{R\}\_\{t\}\. Observe that∑t=1TR~t=\(1−γ\)​R~1\+γ​R~1\+∑t=2TR~t\\sum\_\{t=1\}^\{T\}\\tilde\{R\}\_\{t\}=\(1\-\\gamma\)\\tilde\{R\}\_\{1\}\+\\gamma\\tilde\{R\}\_\{1\}\+\\sum\_\{t=2\}^\{T\}\\tilde\{R\}\_\{t\}\. Rearranging and dividing by1−γ1\-\\gammagivesRT≤∑t=1TR~t≤11−γ​\[γ​\(R~T\+1−R~1\)\+∑t=1T\(∗\)t\]R\_\{T\}\\leq\\sum\_\{t=1\}^\{T\}\\tilde\{R\}\_\{t\}\\leq\\frac\{1\}\{1\-\\gamma\}\[\\gamma\(\\tilde\{R\}\_\{T\+1\}\-\\tilde\{R\}\_\{1\}\)\+\\sum\_\{t=1\}^\{T\}\(\*\)\_\{t\}\]\. Expanding the\(∗\)t\(\*\)\_\{t\}terms \(including splitting the bonus\) and usingκ​\(x,μ\)≤1/4\\kappa\(x,\\mu\)\\leq\\nicefrac\{\{1\}\}\{\{4\}\}we have:

RT\\displaystyle R\_\{T\}≤11−γ​∑t=1T\(𝐄μ^t∘​\[r​\(xt,ut,d\)\]−𝐄μ∗​\[r​\(xt,ut,d\)\]\+\(ρ¯\+γ​L1​c¯\)4​∑i=1Muti​αi,tμ,∘​\(δ/2\)\)⏟\(i\)\\displaystyle\\leq\\tfrac\{1\}\{1\-\\gamma\}\\underbrace\{\\textstyle\\sum\\nolimits\_\{t=1\}^\{T\}\(\\mathbf\{E\}\_\{\\hat\{\\mu\}^\{\\circ\}\_\{t\}\}\[r\(x\_\{t\},u\_\{t\},d\)\]\-\\mathbf\{E\}\_\{\\mu^\{\*\}\}\[r\(x\_\{t\},u\_\{t\},d\)\]\+\\tfrac\{\(\\bar\{\\rho\}\+\\gamma L\_\{1\}\\bar\{c\}\)\}\{4\}\\textstyle\\sum\\nolimits\_\{i=1\}^\{M\}u\_\{t\}^\{i\}\\alpha\_\{i,t\}^\{\\mu,\\circ\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\)\}\_\{\(i\)\}\+γ1−γ​\[R~T\+1−R~1\]⏟\(i​i\)\+γ1−γ​∑t=1T\(𝐄θ∗,μ^t∘​\[J~t,δ∘​\(f​\(⋅\)\)\]−𝐄θ∗,μ∗​\[J~t,δ∘​\(f​\(⋅\)\)\]\)⏟\(i​i​i\)\\displaystyle\\qquad\+\\tfrac\{\\gamma\}\{1\-\\gamma\}\\underbrace\{\[\\tilde\{R\}\_\{T\+1\}\-\\tilde\{R\}\_\{1\}\]\}\_\{\(ii\)\}\+\\tfrac\{\\gamma\}\{1\-\\gamma\}\\underbrace\{\\textstyle\\sum\\nolimits\_\{t=1\}^\{T\}\(\\mathbf\{E\}\_\{\\theta^\{\*\},\\hat\{\\mu\}\_\{t\}^\{\\circ\}\}\[\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(f\(\\cdot\)\)\]\-\\mathbf\{E\}\_\{\\theta^\{\*\},\\mu^\{\*\}\}\[\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(f\(\\cdot\)\)\]\)\}\_\{\(iii\)\}\+γ1−γ​∑t=1T\(𝐄θ^t∘,μ^t∘​\[J~t,δ∘​\(f​\(⋅\)\)\]−𝐄θ∗,μ^t∘​\[J~t,δ∘​\(f​\(⋅\)\)\]\+L1​αtθ,∘​\(δ/2\)​𝐄μ^t∘​\[‖z‖\(V¯t∘\)−1\|xt,ut\]\)⏟\(i​v\)\\displaystyle\\qquad\+\\tfrac\{\\gamma\}\{1\-\\gamma\}\\underbrace\{\\textstyle\\sum\\nolimits\_\{t=1\}^\{T\}\(\\mathbf\{E\}\_\{\\hat\{\\theta\}\_\{t\}^\{\\circ\},\\hat\{\\mu\}\_\{t\}^\{\\circ\}\}\[\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(f\(\\cdot\)\)\]\-\\mathbf\{E\}\_\{\\theta^\{\*\},\\hat\{\\mu\}\_\{t\}^\{\\circ\}\}\[\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(f\(\\cdot\)\)\]\+L\_\{1\}\\alpha\_\{t\}^\{\\theta,\\circ\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\\mathbf\{E\}\_\{\\hat\{\\mu\}\_\{t\}^\{\\circ\}\}\[\\\|z\\\|\_\{\(\\bar\{V\}\_\{t\}^\{\\circ\}\)^\{\-1\}\}\|x\_\{t\},u\_\{t\}\]\)\}\_\{\(iv\)\}\+γ1−γ​∑t=1T\(𝐄θ∗,μ∗​\[J~t,δ∘​\(f​\(⋅\)\)−Jπt​\(f​\(⋅\)\)\]−\(J~t,δ∘​\(xt\+1\)−Jπt​\(xt\+1\)\)\)⏟\(v\)\\displaystyle\\qquad\+\\tfrac\{\\gamma\}\{1\-\\gamma\}\\underbrace\{\\textstyle\\sum\\nolimits\_\{t=1\}^\{T\}\(\\mathbf\{E\}\_\{\\theta^\{\*\},\\mu^\{\*\}\}\[\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(f\(\\cdot\)\)\-J^\{\\pi\_\{t\}\}\(f\(\\cdot\)\)\]\-\(\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(x\_\{t\+1\}\)\-J^\{\\pi\_\{t\}\}\(x\_\{t\+1\}\)\)\)\}\_\{\(v\)\}\+γ1−γ​∑t=1T\[J~t,δ∘​\(xt\+1\)−J~t\+1,δ∘​\(xt\+1\)\+Jπt\+1​\(xt\+1\)−Jπt​\(xt\+1\)\]⏟\(v​i\)\\displaystyle\\qquad\+\\tfrac\{\\gamma\}\{1\-\\gamma\}\\underbrace\{\\textstyle\\sum\\nolimits\_\{t=1\}^\{T\}\[\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(x\_\{t\+1\}\)\-\\tilde\{J\}^\{\\circ\}\_\{t\+1,\\delta\}\(x\_\{t\+1\}\)\+J^\{\\pi\_\{t\+1\}\}\(x\_\{t\+1\}\)\-J^\{\\pi\_\{t\}\}\(x\_\{t\+1\}\)\]\}\_\{\(vi\)\}\(38\)We apply Lemma[13\.23](https://arxiv.org/html/2605.24261#S13.Thmtheorem23)for the givenCNC\_\{N\}to bound\(i\)\(i\)asℰμ​\(δ/2\)\\mathcal\{E\}\_\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)holds by assumption\. We apply Lemma[13\.27](https://arxiv.org/html/2605.24261#S13.Thmtheorem27)to bound\(i​i\)\(ii\)asℰθ​\(δ/2\)\\mathcal\{E\}\_\{\\theta\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)andℰμ​\(δ/2\)\\mathcal\{E\}\_\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)hold by assumption\. We apply Lemma[13\.29](https://arxiv.org/html/2605.24261#S13.Thmtheorem29)to bound\(i​i​i\)\(iii\)asℰμ​\(δ/2\)\\mathcal\{E\}\_\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)holds by assumption\. We apply Lemma[13\.31](https://arxiv.org/html/2605.24261#S13.Thmtheorem31)to bound\(i​v\)\(iv\)asℰθ​\(δ/2\)\\mathcal\{E\}\_\{\\theta\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)andℰμ​\(δ/2\)\\mathcal\{E\}\_\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)hold by assumption withδ′=δa​l​l/3\\delta^\{\\prime\}=\\nicefrac\{\{\\delta\_\{all\}\}\}\{\{3\}\}\. We apply Lemma[13\.39](https://arxiv.org/html/2605.24261#S13.Thmtheorem39)to bound\(v\)\(v\)withδ′=δa​l​l/3\\delta^\{\\prime\}=\\nicefrac\{\{\\delta\_\{all\}\}\}\{\{3\}\}\. We apply Lemma[13\.41](https://arxiv.org/html/2605.24261#S13.Thmtheorem41)to bound\(v​i\)\(vi\)\. The result follows by the union bound\.\\halmos

###### Proof 13\.22

Proof of Corollary[5\.6](https://arxiv.org/html/2605.24261#S5.Thmtheorem6)Per Theorem[5\.5](https://arxiv.org/html/2605.24261#S5.Thmtheorem5), with probability at least1−δa​l​l1\-\\delta\_\{all\}, the given regret bound holds, denotedB​\(T,δa​l​l\)B\(T,\\delta\_\{all\}\)for brevity\. Note thatB​\(T,δa​l​l\)=𝒪~​\(M3/2​T\(1−γ\)3\)B\(T,\\delta\_\{all\}\)=\\tilde\{\\mathcal\{O\}\}\\left\(\\frac\{M^\{\\nicefrac\{\{3\}\}\{\{2\}\}\}\\sqrt\{T\}\}\{\(1\-\\gamma\)^\{3\}\}\\right\), with\(L1\+L2\)​αTθ​\(δa​l​l/6\)​\[C1\+C2\+C3\]\(L\_\{1\}\+L\_\{2\}\)\\alpha\_\{T\}^\{\\theta\}\(\\nicefrac\{\{\\delta\_\{all\}\}\}\{\{6\}\}\)\\left\[C\_\{1\}\+C\_\{2\}\+C\_\{3\}\\right\]being the leading regret term\. With probability at mostδa​l​l\\delta\_\{all\}, the necessary events for Theorem[5\.5](https://arxiv.org/html/2605.24261#S5.Thmtheorem5)do not hold\. In this case, the per\-step regret is bounded byJ∗​\(xt\)−Jπt​\(xt\)≤2​CJJ^\{\*\}\(x\_\{t\}\)\-J^\{\\pi\_\{t\}\}\(x\_\{t\}\)\\leq 2C\_\{J\}\(see Lemma[5\.3](https://arxiv.org/html/2605.24261#S5.Thmtheorem3)\)\. Thus,𝐄​\[RT\]≤\(1−δa​l​l\)​B​\(T,δa​l​l\)\+δa​l​l​T​\(2​CJ\)≤B​\(T,δa​l​l\)\+2​T​δa​l​l​\(β\+ρ¯1−γ\)\\mathbf\{E\}\[R\_\{T\}\]\\leq\(1\-\\delta\_\{all\}\)B\(T,\\delta\_\{all\}\)\+\\delta\_\{all\}T\(2C\_\{J\}\)\\leq B\(T,\\delta\_\{all\}\)\+2T\\delta\_\{all\}\\left\(\\frac\{\\beta\+\\bar\{\\rho\}\}\{1\-\\gamma\}\\right\)\. Choosingδa​l​l=1/T\\delta\_\{all\}=\\nicefrac\{\{1\}\}\{\{T\}\}means the regret rate is dominated by theB​\(T,δa​l​l\)B\(T,\\delta\_\{all\}\)term and this choice ofδa​l​l\\delta\_\{all\}adds only logarithmic factors ofTTtoB​\(T,δa​l​l\)B\(T,\\delta\_\{all\}\), so the corollary follows\.\\halmos

#### 13\.4\.1Supporting lemmas

The following lemmas assume that Assumptions[3\.1](https://arxiv.org/html/2605.24261#S3.SS1)\-[5\.1](https://arxiv.org/html/2605.24261#S5.SS1)hold andCN,Cd\>0C\_\{N\},C\_\{d\}\>0are given by Algorithm[1](https://arxiv.org/html/2605.24261#alg1)\. We define a set of common terms for brevity:CJ=β\+ρ¯1−γC\_\{J\}=\\frac\{\\beta\+\\bar\{\\rho\}\}\{1\-\\gamma\},CJ~,t,δ=β\+ρ¯\+Cb​\(t,δ\)1−γC\_\{\\tilde\{J\},t,\\delta\}=\\frac\{\\beta\+\\bar\{\\rho\}\+C\_\{b\}\(t,\\delta\)\}\{1\-\\gamma\},νT​\(δ\):=e3​μ¯​\[λ22\+2​μ¯λ2\+2λ2​log⁡\(2​M​λ2\+T/4δ​λ2\)\+e−μ¯​λ2​μ¯\]\\nu\_\{T\}\(\\delta\):=e^\{3\\bar\{\\mu\}\}\[\\frac\{\\sqrt\{\\lambda\_\{2\}\}\}\{2\}\+\\frac\{2\\bar\{\\mu\}\}\{\\sqrt\{\\lambda\_\{2\}\}\}\+\\frac\{2\}\{\\sqrt\{\\lambda\_\{2\}\}\}\\log\(\\frac\{2M\\sqrt\{\\lambda\_\{2\}\+\\nicefrac\{\{T\}\}\{\{4\}\}\}\}\{\\delta\\sqrt\{\\lambda\_\{2\}\}\}\)\+e^\{\-\\bar\{\\mu\}\}\\sqrt\{\\lambda\_\{2\}\}\\bar\{\\mu\}\],C1=νT​\(δ/2\)​1\+CN4​λ1​\[M/λ2\+2​M​T/q¯\]C\_\{1\}=\\frac\{\\nu\_\{T\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\\sqrt\{1\+C\_\{N\}\}\}\{4\\sqrt\{\\lambda\_\{1\}\}\}\[\\nicefrac\{\{M\}\}\{\{\\sqrt\{\\lambda\_\{2\}\}\}\}\+2\\sqrt\{\\nicefrac\{\{MT\}\}\{\{\\underline\{q\}\}\}\}\],C2=2​log⁡\(1/δ′\)​T/λ1C\_\{2\}=\\sqrt\{2\\log\(\\nicefrac\{\{1\}\}\{\{\\delta^\{\\prime\}\}\}\)\\nicefrac\{\{T\}\}\{\{\\lambda\_\{1\}\}\}\},C3=T​\(1\+Cd\)​\(2​M\+1\)​log⁡\(1\+T​\(Cx2\+2\)\(2​M\+1\)​λ1\)​\[2\+Cx2\+2log⁡\(2\)​λ1\]C\_\{3\}=\\sqrt\{T\(1\+C\_\{d\}\)\(2M\+1\)\\log\(1\+\\frac\{T\(C\_\{x\}^\{2\}\+2\)\}\{\(2M\+1\)\\lambda\_\{1\}\}\)\[2\+\\frac\{C\_\{x\}^\{2\}\+2\}\{\\log\(2\)\\lambda\_\{1\}\}\]\},Kd=\(2​M\+1\)log⁡\(1\+Cd\)​log⁡\(1\+T​\(Cx2\+2\)\(2​M\+1\)​λ1\)K\_\{d\}=\\frac\{\(2M\+1\)\}\{\\log\(1\+C\_\{d\}\)\}\\log\(1\+\\frac\{T\(C\_\{x\}^\{2\}\+2\)\}\{\(2M\+1\)\\lambda\_\{1\}\}\),KN=M​\(1\+log⁡\(T\)log⁡\(1\+CN\)\)K\_\{N\}=M\(1\+\\frac\{\\log\(T\)\}\{\\log\(1\+C\_\{N\}\)\}\)\.

###### Lemma 13\.23

Letℰμ​\(δ/2\)\\mathcal\{E\}\_\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)hold\. For any trajectoryt=1,…,Tt=1,\\dots,T,∑t=1T\(𝐄μ^t∘​\[r​\(xt,ut,d\)\]−𝐄μ∗​\[r​\(xt,ut,d\)\]\+\(ρ¯\+γ​L1​c¯\)4​∑i=1Muti​αi,tμ,∘​\(δ/2\)\)≤\(2​ρ¯\+γ​L1​c¯\)​νT​\(δ/2\)​1\+CN4​\[M/λ2\+2​M​T/q¯\]\\sum\_\{t=1\}^\{T\}\(\\mathbf\{E\}\_\{\\hat\{\\mu\}^\{\\circ\}\_\{t\}\}\[r\(x\_\{t\},u\_\{t\},d\)\]\-\\mathbf\{E\}\_\{\\mu^\{\*\}\}\[r\(x\_\{t\},u\_\{t\},d\)\]\+\\frac\{\(\\bar\{\\rho\}\+\\gamma L\_\{1\}\\bar\{c\}\)\}\{4\}\\sum\_\{i=1\}^\{M\}u\_\{t\}^\{i\}\\alpha\_\{i,t\}^\{\\mu,\\circ\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\)\\leq\\frac\{\(2\\bar\{\\rho\}\+\\gamma L\_\{1\}\\bar\{c\}\)\\nu\_\{T\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\\sqrt\{1\+C\_\{N\}\}\}\{4\}\[\\nicefrac\{\{M\}\}\{\{\\sqrt\{\\lambda\_\{2\}\}\}\}\+2\\sqrt\{\\nicefrac\{\{MT\}\}\{\{\\underline\{q\}\}\}\}\]\.

###### Proof 13\.24

Proof of Lemma[13\.23](https://arxiv.org/html/2605.24261#S13.Thmtheorem23)LetΔr:=∑t=1T\(𝐄μ^t∘​\[r​\(xt,ut,d\)\]−𝐄μ∗​\[r​\(xt,ut,d\)\]\)\\Delta\_\{r\}:=\\sum\_\{t=1\}^\{T\}\\big\(\\mathbf\{E\}\_\{\\hat\{\\mu\}^\{\\circ\}\_\{t\}\}\\big\[r\(x\_\{t\},u\_\{t\},d\)\\big\]\-\\mathbf\{E\}\_\{\\mu^\{\*\}\}\\big\[r\(x\_\{t\},u\_\{t\},d\)\\big\]\\big\)\. Note that theβ\\beta\-sigmoid portion of the reward cancels between each pair of terms:Δr≤∑t=1T∑i=1M\|uti​ρi​\(σ​\(xt\+μ^i,t∘\)−σ​\(xt\+μi∗\)\)\|≤∑t=1T∑i=1Muti​ρ¯4​\|μ^i,t∘−μi∗\|​≤\(a\)​ρ¯4​∑t=1T∑i=1Muti​αi,tμ,∘​\(δ/2\)\\Delta\_\{r\}\\leq\\sum\_\{t=1\}^\{T\}\\sum\_\{i=1\}^\{M\}\|u\_\{t\}^\{i\}\\rho\_\{i\}\(\\sigma\(x\_\{t\}\+\\hat\{\\mu\}\_\{i,t\}^\{\\circ\}\)\-\\sigma\(x\_\{t\}\+\\mu^\{\*\}\_\{i\}\)\)\|\\leq\\sum\_\{t=1\}^\{T\}\\sum\_\{i=1\}^\{M\}u\_\{t\}^\{i\}\\frac\{\\bar\{\\rho\}\}\{4\}\|\\hat\{\\mu\}\_\{i,t\}^\{\\circ\}\-\\mu^\{\*\}\_\{i\}\|\\overset\{\(a\)\}\{\\leq\}\\frac\{\\bar\{\\rho\}\}\{4\}\\sum\_\{t=1\}^\{T\}\\sum\_\{i=1\}^\{M\}u^\{i\}\_\{t\}\\alpha\_\{i,t\}^\{\\mu,\\circ\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\.\(a\)\(a\)holds by Proposition[5\.2](https://arxiv.org/html/2605.24261#S5.Thmtheorem2)asℰμ​\(δ/2\)\\mathcal\{E\}\_\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)holds by assumption\. The lemma follows by applying Lemma[13\.25](https://arxiv.org/html/2605.24261#S13.Thmtheorem25)for the givenCNC\_\{N\}andδ/2\\nicefrac\{\{\\delta\}\}\{\{2\}\}to the commonαtμ,∘\\alpha^\{\\mu,\\circ\}\_\{t\}summation in the reward difference bound and the bonus terms\.\\halmos

###### Lemma 13\.25

For any realized trajectoryt=1,…,Tt=1,\\dots,Tfollowing Algorithm[1](https://arxiv.org/html/2605.24261#alg1),∑t=1T∑i=1Muti​αi,tμ,∘​\(δ\)≤νT​\(δ\)​1\+CN​\[M/λ2\+2​M​T/q¯\]\\sum\_\{t=1\}^\{T\}\\sum\_\{i=1\}^\{M\}u^\{i\}\_\{t\}\\alpha\_\{i,t\}^\{\\mu,\\circ\}\(\\delta\)\\leq\\nu\_\{T\}\(\\delta\)\\sqrt\{1\+C\_\{N\}\}\[\\nicefrac\{\{M\}\}\{\{\\sqrt\{\\lambda\_\{2\}\}\}\}\+2\\sqrt\{\\nicefrac\{\{MT\}\}\{\{\\underline\{q\}\}\}\}\]\.

###### Proof 13\.26

Proof of Lemma[13\.25](https://arxiv.org/html/2605.24261#S13.Thmtheorem25)First consider terms not indexed by\(⋅\)∘\(\{\\cdot\}\)^\{\\circ\}\. By Proposition[5\.2](https://arxiv.org/html/2605.24261#S5.Thmtheorem2),αi,tμ​\(δ\):=e3​μ¯Ht​\(μ^i,t\)​\[λ22\+2​μ¯λ2\+2λ2​log⁡\(2​M​Ht​\(μ^i,t\)δ​λ2\)\]\+e2​μ¯​λ2​μ¯Ht​\(μ^i,t\)\\alpha^\{\\mu\}\_\{i,t\}\(\\delta\):=\\frac\{e^\{3\\bar\{\\mu\}\}\}\{\\sqrt\{H\_\{t\}\(\\hat\{\\mu\}\_\{i,t\}\)\}\}\[\\frac\{\\sqrt\{\\lambda\_\{2\}\}\}\{2\}\+\\frac\{2\\bar\{\\mu\}\}\{\\sqrt\{\\lambda\_\{2\}\}\}\+\\frac\{2\}\{\\sqrt\{\\lambda\_\{2\}\}\}\\log\(\\frac\{2M\\sqrt\{H\_\{t\}\(\\hat\{\\mu\}\_\{i,t\}\)\}\}\{\\delta\\sqrt\{\\lambda\_\{2\}\}\}\)\]\+\\frac\{e^\{2\\bar\{\\mu\}\}\\lambda\_\{2\}\\bar\{\\mu\}\}\{H\_\{t\}\(\\hat\{\\mu\}\_\{i,t\}\)\}andHt​\(μ^i,t\):=λ2\+∑s=1t−1𝟏\{us=ei\}​σ′​\(xs\+μ^i,t\)H\_\{t\}\(\\hat\{\\mu\}\_\{i,t\}\):=\\lambda\_\{2\}\+\\sum\_\{s=1\}^\{t\-1\}\\mathbf\{1\}\_\{\\\{u\_\{s\}=e\_\{i\}\\\}\}\\sigma^\{\\prime\}\(x\_\{s\}\+\\hat\{\\mu\}\_\{i,t\}\)\. Note the loose boundsHt​\(μ\)≥λ2H\_\{t\}\(\\mu\)\\geq\\lambda\_\{2\}\(asσ′​\(z\)≥0\\sigma^\{\\prime\}\(z\)\\geq 0\) andHt​\(μ\)≤λ2\+T/4H\_\{t\}\(\\mu\)\\leq\\lambda\_\{2\}\+\\nicefrac\{\{T\}\}\{\{4\}\}\(asσ′​\(z\)≤1/4\\sigma^\{\\prime\}\(z\)\\leq\\nicefrac\{\{1\}\}\{\{4\}\}\), giving the boundαi,tμ​\(δ\)≤e3​μ¯Ht​\(μ^i,t\)​\[λ22\+2​μ¯λ2\+2λ2​log⁡\(2​M​λ2\+t/4δ​λ2\)\+e−μ¯​λ2​μ¯\]\\alpha\_\{i,t\}^\{\\mu\}\(\\delta\)\\leq\\frac\{e^\{3\\bar\{\\mu\}\}\}\{\\sqrt\{H\_\{t\}\(\\hat\{\\mu\}\_\{i,t\}\)\}\}\[\\frac\{\\sqrt\{\\lambda\_\{2\}\}\}\{2\}\+\\frac\{2\\bar\{\\mu\}\}\{\\sqrt\{\\lambda\_\{2\}\}\}\+\\frac\{2\}\{\\sqrt\{\\lambda\_\{2\}\}\}\\log\(\\frac\{2M\\sqrt\{\\lambda\_\{2\}\+\\nicefrac\{\{t\}\}\{\{4\}\}\}\}\{\\delta\\sqrt\{\\lambda\_\{2\}\}\}\)\+e^\{\-\\bar\{\\mu\}\}\\sqrt\{\\lambda\_\{2\}\}\\bar\{\\mu\}\]\. Defineνt​\(δ\):=e3​μ¯​\[λ22\+2​μ¯λ2\+2λ2​log⁡\(2​M​λ2\+t/4δ​λ2\)\+e−μ¯​λ2​μ¯\]\\nu\_\{t\}\(\\delta\):=e^\{3\\bar\{\\mu\}\}\[\\frac\{\\sqrt\{\\lambda\_\{2\}\}\}\{2\}\+\\frac\{2\\bar\{\\mu\}\}\{\\sqrt\{\\lambda\_\{2\}\}\}\+\\frac\{2\}\{\\sqrt\{\\lambda\_\{2\}\}\}\\log\(\\frac\{2M\\sqrt\{\\lambda\_\{2\}\+\\nicefrac\{\{t\}\}\{\{4\}\}\}\}\{\\delta\\sqrt\{\\lambda\_\{2\}\}\}\)\+e^\{\-\\bar\{\\mu\}\}\\sqrt\{\\lambda\_\{2\}\}\\bar\{\\mu\}\]\. Thus,αi,tμ​\(δ\)≤νt​\(δ\)Ht​\(μ^i,t\)≤νT​\(δ\)Ht​\(μ^i,t\)\\alpha\_\{i,t\}^\{\\mu\}\(\\delta\)\\leq\\frac\{\\nu\_\{t\}\(\\delta\)\}\{\\sqrt\{H\_\{t\}\(\\hat\{\\mu\}\_\{i,t\}\)\}\}\\leq\\frac\{\\nu\_\{T\}\(\\delta\)\}\{\\sqrt\{H\_\{t\}\(\\hat\{\\mu\}\_\{i,t\}\)\}\}\(νt​\(δ\)\\nu\_\{t\}\(\\delta\)increasing intt\)\. Next observe that for our systemminx∈𝒳,μi∈\[−μ¯,μ¯\]σ′\(x\+μi\)=σ′\(Cx\+μ¯\)=:q¯\\min\_\{x\\in\\mathcal\{X\},\\mu\_\{i\}\\in\[\-\\bar\{\\mu\},\\bar\{\\mu\}\]\}\\sigma^\{\\prime\}\(x\+\\mu\_\{i\}\)=\\sigma^\{\\prime\}\(C\_\{x\}\+\\bar\{\\mu\}\)=:\\underline\{q\}\. Thisq¯\\underline\{q\}is a uniform worst case information gain for a non\-null action\. LetNi,t:=∑s=1t−1𝟏\{us=ei\}N\_\{i,t\}:=\\sum\_\{s=1\}^\{t\-1\}\\mathbf\{1\}\_\{\\\{u\_\{s\}=e\_\{i\}\\\}\}be the number of times actioneie\_\{i\}has been taken by the beginning of steptt\. This implies the uniform lower boundHt​\(μi\)≥λ2\+q¯​Ni,tH\_\{t\}\(\\mu\_\{i\}\)\\geq\\lambda\_\{2\}\+\\underline\{q\}N\_\{i,t\}and further thatαi,tμ​\(δ\)≤νT​\(δ\)λ2\+q¯​Ni,t\\alpha\_\{i,t\}^\{\\mu\}\(\\delta\)\\leq\\frac\{\\nu\_\{T\}\(\\delta\)\}\{\\sqrt\{\\lambda\_\{2\}\+\\underline\{q\}N\_\{i,t\}\}\}\. Returning to\(⋅\)∘\(\{\\cdot\}\)^\{\\circ\}indexed terms, we haveαi,tμ,∘​\(δ\)≤νT​\(δ\)Ht∘​\(μ^i,t∘\)\\alpha^\{\\mu,\\circ\}\_\{i,t\}\(\\delta\)\\leq\\frac\{\\nu\_\{T\}\(\\delta\)\}\{\\sqrt\{H\_\{t\}^\{\\circ\}\(\\hat\{\\mu\}^\{\\circ\}\_\{i,t\}\)\}\}\. For time steps in thekt​hk^\{th\}epoch \(beginning at stepτk\\tau\_\{k\}\),Ht∘​\(μ^i,t∘\)≥λ2\+q¯​Ni,τkH\_\{t\}^\{\\circ\}\(\\hat\{\\mu\}^\{\\circ\}\_\{i,t\}\)\\geq\\lambda\_\{2\}\+\\underline\{q\}N\_\{i,\\tau\_\{k\}\}lower bounds the information available for treatmentii\(i\.e\., per the action count at the start of the epoch\)\. We connect\(⋅\)t∘\(\{\\cdot\}\)^\{\\circ\}\_\{t\}values to\(⋅\)t\(\{\\cdot\}\)\_\{t\}values by noting that at the end of steptt, Algorithm[1](https://arxiv.org/html/2605.24261#alg1)triggers a new epoch beginning att\+1t\+1\(i\.e\.,τk\+1=t\+1\\tau\_\{k\+1\}=t\+1\) ifNi,t\+1\>\(1\+CN\)​Ni,τkN\_\{i,t\+1\}\>\(1\+C\_\{N\}\)N\_\{i,\\tau\_\{k\}\}for anyi=1,…,Mi=1,\\dots,M\. Thus for time stepsτk≤t<τk\+1\\tau\_\{k\}\\leq t<\\tau\_\{k\+1\}within epochkkthe trigger has not fired, soNi,t≤\(1\+CN\)​Ni,τkN\_\{i,t\}\\leq\(1\+C\_\{N\}\)N\_\{i,\\tau\_\{k\}\}holds\. This implies \(after using loosening for bound clarity withλ2​\(1\+CN\)≥λ2\\lambda\_\{2\}\(1\+C\_\{N\}\)\\geq\\lambda\_\{2\}\):αi,tμ,∘​\(δ\)≤νT​\(δ\)​1\+CNλ2\+q¯​Ni,t\\alpha\_\{i,t\}^\{\\mu,\\circ\}\(\\delta\)\\leq\\frac\{\\nu\_\{T\}\(\\delta\)\\sqrt\{1\+C\_\{N\}\}\}\{\\sqrt\{\\lambda\_\{2\}\+\\underline\{q\}N\_\{i,t\}\}\}\. Now we introduce a count based indexing\. Lettijt\_\{i\}^\{j\}be the time step where actioniiis taken for thejt​hj^\{th\}time\. Observe thatNi,tij=j−1N\_\{i,t\_\{i\}^\{j\}\}=j\-1\(i\.e\., the action has previously been takenj−1j\-1times if it is taken thejt​hj^\{th\}time in the current time step\)\. Thus we can re\-index withjjrather thanttas follows:

∑t=1T∑i=1Muti​αi,tμ,∘​\(δ\)\\displaystyle\\textstyle\\sum\\nolimits\_\{t=1\}^\{T\}\\sum\\nolimits\_\{i=1\}^\{M\}u^\{i\}\_\{t\}\\alpha\_\{i,t\}^\{\\mu,\\circ\}\(\\delta\)≤∑i=1M∑t=1Tuti​νT​\(δ\)​1\+CNλ2\+q¯​Ni,t​=\(a\)​∑i=1M∑j=1Ni,T\+1νT​\(δ\)​1\+CNλ2\+q¯​\(j−1\)\\displaystyle\\leq\\textstyle\\sum\\nolimits\_\{i=1\}^\{M\}\\sum\\nolimits\_\{t=1\}^\{T\}u\_\{t\}^\{i\}\\tfrac\{\\nu\_\{T\}\(\\delta\)\\sqrt\{1\+C\_\{N\}\}\}\{\\sqrt\{\\lambda\_\{2\}\+\\underline\{q\}N\_\{i,t\}\}\}\\overset\{\(a\)\}\{=\}\\textstyle\\sum\\nolimits\_\{i=1\}^\{M\}\\sum\\nolimits\_\{j=1\}^\{N\_\{i,T\+1\}\}\\tfrac\{\\nu\_\{T\}\(\\delta\)\\sqrt\{1\+C\_\{N\}\}\}\{\\sqrt\{\\lambda\_\{2\}\+\\underline\{q\}\(j\-1\)\}\}\(39\)where\(a\)\(a\)uses the fact thatuti∈\{0,1\}u^\{i\}\_\{t\}\\in\\\{0,1\\\}ensures only theNi,T\+1N\_\{i,T\+1\}terms given byti1,…,tiNi,T\+1t\_\{i\}^\{1\},\\dots,t^\{N\_\{i,T\+1\}\}\_\{i\}are included in the summation andNi,tij=j−1N\_\{i,t\_\{i\}^\{j\}\}=j\-1\. SinceνT​\(δ\)​\(1\+CN\)\\nu\_\{T\}\(\\delta\)\(\\sqrt\{1\+C\_\{N\}\}\)factors out of the summation and summands\(λ2\+q¯​\(j−1\)\)−1/2\(\\lambda\_\{2\}\+\\underline\{q\}\(j\-1\)\)^\{\-\\nicefrac\{\{1\}\}\{\{2\}\}\}are decreasing injj, we have the following integral upper bound:

∑j=1Ni,T\+11λ2\+q¯​\(j−1\)≤1λ2\+∫1Ni,T\+11λ2\+q¯​\(j−1\)​𝑑j=1λ2\+2q¯​\[λ2\+q¯​\(Ni,T\+1−1\)−λ2\]\\displaystyle\\textstyle\\sum\\nolimits\_\{j=1\}^\{N\_\{i,T\+1\}\}\\tfrac\{1\}\{\\sqrt\{\\lambda\_\{2\}\+\\underline\{q\}\(j\-1\)\}\}\\leq\\tfrac\{1\}\{\\sqrt\{\\lambda\_\{2\}\}\}\+\\int\\nolimits\_\{1\}^\{N\_\{i,T\+1\}\}\\tfrac\{1\}\{\\sqrt\{\\lambda\_\{2\}\+\\underline\{q\}\(j\-1\)\}\}dj=\\tfrac\{1\}\{\\sqrt\{\\lambda\_\{2\}\}\}\+\\tfrac\{2\}\{\\underline\{q\}\}\[\\sqrt\{\\lambda\_\{2\}\+\\underline\{q\}\(N\_\{i,T\+1\}\-1\)\}\-\\sqrt\{\\lambda\_\{2\}\}\]\(40\)Using the subadditivity of square roots andNi,T\+1−1≤Ni,T\+1N\_\{i,T\+1\}\-1\\leq N\_\{i,T\+1\}we simplify to the more compact upper bound1λ2\+2​Ni,T\+1q¯\\frac\{1\}\{\\sqrt\{\\lambda\_\{2\}\}\}\+\\frac\{2\\sqrt\{N\_\{i,T\+1\}\}\}\{\\sqrt\{\\underline\{q\}\}\}\. Substituting back and applying Cauchy\-Schwarz we have:

∑i=1M\[1λ2\+2​Ni,T\+1q¯\]≤Mλ2\+∑i=1M\(1\)2​∑i=1M\(2​Ni,T\+1q¯\)2≤Mλ2\+2​M​Tq¯\\displaystyle\\textstyle\\sum\\nolimits\_\{i=1\}^\{M\}\[\\tfrac\{1\}\{\\sqrt\{\\lambda\_\{2\}\}\}\+\\tfrac\{2\\sqrt\{N\_\{i,T\+1\}\}\}\{\\sqrt\{\\underline\{q\}\}\}\]\\leq\\tfrac\{M\}\{\\sqrt\{\\lambda\_\{2\}\}\}\+\\sqrt\{\\sum\\nolimits\_\{i=1\}^\{M\}\(1\)^\{2\}\}\\sqrt\{\\sum\\nolimits\_\{i=1\}^\{M\}\(\\tfrac\{2\\sqrt\{N\_\{i,T\+1\}\}\}\{\\sqrt\{\\underline\{q\}\}\}\)^\{2\}\}\\leq\\tfrac\{M\}\{\\sqrt\{\\lambda\_\{2\}\}\}\+2\\sqrt\{\\tfrac\{MT\}\{\\underline\{q\}\}\}\(41\)The lemma follows by substitution\. Asq¯\\underline\{q\}is a uniform lower bound and only∑i=1MNi,T\+1≤T\\sum\\nolimits\_\{i=1\}^\{M\}N\_\{i,T\+1\}\\leq Twas used, the bound holds for any realized trajectory\.\\halmos

###### Lemma 13\.27

Letℰθ​\(δ/2\)\\mathcal\{E\}\_\{\\theta\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)andℰμ​\(δ/2\)\\mathcal\{E\}\_\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)hold\. ThenR~T\+1−R~1≤CJ\+CJ~,T\+1,δ\\tilde\{R\}\_\{T\+1\}\-\\tilde\{R\}\_\{1\}\\leq C\_\{J\}\+C\_\{\\tilde\{J\},T\+1,\\delta\}\.

###### Proof 13\.28

Proof of Lemma[13\.27](https://arxiv.org/html/2605.24261#S13.Thmtheorem27)AsJ∗​\(x\)≥Jπt​\(x\)J^\{\*\}\(x\)\\geq J^\{\\pi\_\{t\}\}\(x\)by the optimality ofJ∗J^\{\*\}, theR~\\tilde\{R\}terms are positive by chaining this inequality with optimism Lemma[5\.4](https://arxiv.org/html/2605.24261#S5.Thmtheorem4)\(ℰθ​\(δ/2\)\\mathcal\{E\}\_\{\\theta\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)andℰμ​\(δ/2\)\\mathcal\{E\}\_\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)hold by assumption\), so it is sufficient to boundR~T\+1\\tilde\{R\}\_\{T\+1\}\. By definition,R~T\+1=J~T\+1,δ∘​\(xT\+1\)−JπT\+1​\(xT\+1\)\\tilde\{R\}\_\{T\+1\}=\\tilde\{J\}^\{\\circ\}\_\{T\+1,\\delta\}\(x\_\{T\+1\}\)\-J^\{\\pi\_\{T\+1\}\}\(x\_\{T\+1\}\)\. Both value functions are bounded by Lemma[5\.3](https://arxiv.org/html/2605.24261#S5.Thmtheorem3), i\.e\.‖JπT\+1‖∞≤CJ\\\|J^\{\\pi\_\{T\+1\}\}\\\|\_\{\\infty\}\\leq C\_\{J\}and‖J~T\+1,δ∘‖∞≤CJ~,T\+1,δ\\\|\\tilde\{J\}^\{\\circ\}\_\{T\+1,\\delta\}\\\|\_\{\\infty\}\\leq C\_\{\\tilde\{J\},T\+1,\\delta\}\. ThusR~T\+1−R~1≤R~T\+1≤maxx∈𝒳⁡J~T\+1,δ∘​\(x\)−minx∈𝒳⁡JπT\+1​\(x\)≤CJ\+CJ~,T\+1,δ\\tilde\{R\}\_\{T\+1\}\-\\tilde\{R\}\_\{1\}\\leq\\tilde\{R\}\_\{T\+1\}\\leq\\max\_\{x\\in\\mathcal\{X\}\}\\tilde\{J\}^\{\\circ\}\_\{T\+1,\\delta\}\(x\)\-\\min\_\{x\\in\\mathcal\{X\}\}J^\{\\pi\_\{T\+1\}\}\(x\)\\leq C\_\{J\}\+C\_\{\\tilde\{J\},T\+1,\\delta\}\.\\halmos

###### Lemma 13\.29

Letℰμ​\(δ/2\)\\mathcal\{E\}\_\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)hold\. Then∑t=1T\(𝐄θ∗,μ^t∘\[J~t,δ∘\(f\(xt,ut,d,w\)\]−𝐄θ∗,μ∗\[J~t,δ∘\(f\(xt,ut,d,w\)\)\]\)≤1/4L2c¯νT\(δ/2\)1\+CN\[M/λ2\+2M​T/q¯\]\\sum\_\{t=1\}^\{T\}\(\\mathbf\{E\}\_\{\\theta^\{\*\},\\hat\{\\mu\}\_\{t\}^\{\\circ\}\}\[\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(f\(x\_\{t\},u\_\{t\},d,w\)\]\-\\mathbf\{E\}\_\{\\theta^\{\*\},\\mu^\{\*\}\}\[\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(f\(x\_\{t\},u\_\{t\},d,w\)\)\]\)\\leq\\nicefrac\{\{1\}\}\{\{4\}\}L\_\{2\}\\bar\{c\}\\nu\_\{T\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\\sqrt\{1\+C\_\{N\}\}\[\\nicefrac\{\{M\}\}\{\{\\sqrt\{\\lambda\_\{2\}\}\}\}\+2\\sqrt\{\\nicefrac\{\{MT\}\}\{\{\\underline\{q\}\}\}\}\]\.

###### Proof 13\.30

Proof of Lemma[13\.29](https://arxiv.org/html/2605.24261#S13.Thmtheorem29)Letℰμ​\(δ/2\)\\mathcal\{E\}\_\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)hold\. We use a similar simplification to Lemma[5\.4](https://arxiv.org/html/2605.24261#S5.Thmtheorem4)\. Letz1:=\[xt​ut⊤​ut⊤\]⊤z\_\{1\}:=\[x\_\{t\}\\ u\_\{t\}^\{\\top\}\\ u\_\{t\}^\{\\top\}\]^\{\\top\}andz0:=\[xt​ut⊤​0M⊤\]⊤z\_\{0\}:=\[x\_\{t\}\\ u\_\{t\}^\{\\top\}\\ \\bm\{0\}\_\{M\}^\{\\top\}\]^\{\\top\}denote the possible state vectors under adherence and non\-adherence\. Consider a single time step and letut=eiu\_\{t\}=e\_\{i\}\(the difference is trivially0for any time step withut=𝟎Mu\_\{t\}=\\bm\{0\}\_\{M\}\), withΔ:=𝐄θ∗,μ^t∘\[J~t,δ∘\(f\(xt,ut,d,w\)\]−𝐄θ∗,μ∗\[J~t,δ∘\(f\(xt,ut,d,w\)\)\]\\Delta:=\\mathbf\{E\}\_\{\\theta^\{\*\},\\hat\{\\mu\}\_\{t\}^\{\\circ\}\}\[\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(f\(x\_\{t\},u\_\{t\},d,w\)\]\-\\mathbf\{E\}\_\{\\theta^\{\*\},\\mu^\{\*\}\}\[\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(f\(x\_\{t\},u\_\{t\},d,w\)\)\]\. We have:

Δ\\displaystyle\\Delta=𝐄μ^t∘​\[J~t,δ∘​\(z⊤​θ∗\+w\)\|xt,ut\]−𝐄μ∗​\[J~t,δ∘​\(z⊤​θ∗\+w\)\|xt,ut\]\\displaystyle=\\mathbf\{E\}\_\{\\hat\{\\mu\}^\{\\circ\}\_\{t\}\}\[\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(z^\{\\top\}\\theta^\{\*\}\+w\)\|x\_\{t\},u\_\{t\}\]\-\\mathbf\{E\}\_\{\\mu^\{\*\}\}\[\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(z^\{\\top\}\\theta^\{\*\}\+w\)\|x\_\{t\},u\_\{t\}\]\(42\)≤\(a\)​\|σ​\(xt\+μ^i,t∘\)−σ​\(xt\+μi∗\)\|⋅𝐄​\[\|J~t,δ∘​\(z1⊤​θ∗\+w\)−J~t,δ∘​\(z0⊤​θ∗\+w\)\|\]\\displaystyle\\overset\{\(a\)\}\{\\leq\}\|\\sigma\(x\_\{t\}\+\\hat\{\\mu\}\_\{i,t\}^\{\\circ\}\)\-\\sigma\(x\_\{t\}\+\\mu\_\{i\}^\{\*\}\)\|\\cdot\\mathbf\{E\}\[\|\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(z\_\{1\}^\{\\top\}\\theta^\{\*\}\+w\)\-\\tilde\{J\}\_\{t,\\delta\}^\{\\circ\}\(z\_\{0\}^\{\\top\}\\theta^\{\*\}\+w\)\|\]\(43\)≤\(b\)​1/4​\|μ^i,t∘−μi∗\|⋅L2​\|\(z1−z0\)⊤​θ∗\|​≤\(c\)​1/4​αi,tμ,∘​\(δ/2\)⋅L2​c¯\\displaystyle\\overset\{\(b\)\}\{\\leq\}\\nicefrac\{\{1\}\}\{\{4\}\}\|\\hat\{\\mu\}\_\{i,t\}^\{\\circ\}\-\\mu\_\{i\}^\{\*\}\|\\cdot L\_\{2\}\|\(z\_\{1\}\-z\_\{0\}\)^\{\\top\}\\theta^\{\*\}\|\\overset\{\(c\)\}\{\\leq\}\\nicefrac\{\{1\}\}\{\{4\}\}\\alpha\_\{i,t\}^\{\\mu,\\circ\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\\cdot L\_\{2\}\\bar\{c\}\(44\)\(a\)\(a\)upper bounds with absolute values, uses Jensen’s inequality, expands adherence outcomes, and simplifies\.\(b\)\(b\)uses Lipschitz continuity of the sigmoid andJ~t,δ\\tilde\{J\}\_\{t,\\delta\}\(Lemma[13\.12](https://arxiv.org/html/2605.24261#S13.Thmtheorem12)\)\.\(c\)\(c\)uses Proposition[5\.2](https://arxiv.org/html/2605.24261#S5.Thmtheorem2)andci≤c¯c\_\{i\}\\leq\\bar\{c\}asθ∗∈Θ\\theta^\{\*\}\\in\\Theta\. Thus the full summation is upper bounded byL2​c¯4​∑t=1T∑i=1Muti​αi,tμ,∘​\(δ/2\)\\frac\{L\_\{2\}\\bar\{c\}\}\{4\}\\sum\_\{t=1\}^\{T\}\\sum\_\{i=1\}^\{M\}u\_\{t\}^\{i\}\\alpha^\{\\mu,\\circ\}\_\{i,t\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\. Asℰμ​\(δ/2\)\\mathcal\{E\}\_\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)holds, we apply Lemma[13\.25](https://arxiv.org/html/2605.24261#S13.Thmtheorem25)for the givenCNC\_\{N\}and the lemma follows\.\\halmos

###### Lemma 13\.31

Letδ′∈\(0,1\)\\delta^\{\\prime\}\\in\(0,1\)andℰθ​\(δ/2\)\\mathcal\{E\}\_\{\\theta\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\),ℰμ​\(δ/2\)\\mathcal\{E\}\_\{\\mu\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)hold\. Then𝐏​\(∑t=1T\(𝐄θ^t∘,μ^t∘​\[J~t,δ∘​\(f​\(⋅\)\)\]−𝐄θ∗,μ^t∘​\[J~t,δ∘​\(f​\(⋅\)\)\]\+L1​αtθ,∘​\(δ/2\)​𝐄μ^t∘​\[‖z‖\(V¯t∘\)−1\|xt,ut\]\)≤\(L1\+L2\)​αTθ​\(δ/2\)​\[C1\+C2\+C3\]\)≥1−δ′\\mathbf\{P\}\(\\sum\_\{t=1\}^\{T\}\\big\(\\mathbf\{E\}\_\{\\hat\{\\theta\}\_\{t\}^\{\\circ\},\\hat\{\\mu\}\_\{t\}^\{\\circ\}\}\\big\[\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(f\(\\cdot\)\)\\big\]\-\\mathbf\{E\}\_\{\\theta^\{\*\},\\hat\{\\mu\}\_\{t\}^\{\\circ\}\}\\big\[\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(f\(\\cdot\)\)\\big\]\+L\_\{1\}\\alpha\_\{t\}^\{\\theta,\\circ\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\\mathbf\{E\}\_\{\\hat\{\\mu\}\_\{t\}^\{\\circ\}\}\\big\[\\\|z\\\|\_\{\(\\bar\{V\}^\{\\circ\}\_\{t\}\)^\{\-1\}\}\|x\_\{t\},u\_\{t\}\\big\]\\big\)\\leq\(L\_\{1\}\+L\_\{2\}\)\\alpha\_\{T\}^\{\\theta\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\\left\[C\_\{1\}\+C\_\{2\}\+C\_\{3\}\\right\]\)\\geq 1\-\\delta^\{\\prime\}\.

###### Proof 13\.32

Proof of Lemma[13\.31](https://arxiv.org/html/2605.24261#S13.Thmtheorem31)Let\(i​v\)1=∑t=1T\(𝐄θ^t∘,μ^t∘​\[J~t,δ∘​\(f​\(xt,ut,d,w\)\)\]−𝐄θ∗,μ^t∘​\[J~t,δ∘​\(f​\(xt,ut,d,w\)\)\]\)\(iv\)\_\{1\}=\\sum\_\{t=1\}^\{T\}\(\\mathbf\{E\}\_\{\\hat\{\\theta\}\_\{t\}^\{\\circ\},\\hat\{\\mu\}\_\{t\}^\{\\circ\}\}\[\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(f\(x\_\{t\},u\_\{t\},d,w\)\)\]\-\\mathbf\{E\}\_\{\\theta^\{\*\},\\hat\{\\mu\}\_\{t\}^\{\\circ\}\}\[\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(f\(x\_\{t\},u\_\{t\},d,w\)\)\]\)and\(i​v\)2\(iv\)\_\{2\}denote the remaining summation\. We have:

\(i​v\)1\\displaystyle\(iv\)\_\{1\}≤\(a\)​∑t=1T𝐄μ^t∘​\[L2​\|θ^t∘⊤​zt\+w−θ∗⊤​zt−w\|\|xt,ut\]=L2​∑t=1T𝐄μ^t∘​\[\|\(θ^t∘−θ∗\)⊤​zt\|\|xt,ut\]\\displaystyle\\overset\{\(a\)\}\{\\leq\}\\textstyle\\sum\_\{t=1\}^\{T\}\\mathbf\{E\}\_\{\\hat\{\\mu\}\_\{t\}^\{\\circ\}\}\[L\_\{2\}\|\\hat\{\\theta\}^\{\\circ\\top\}\_\{t\}z\_\{t\}\+w\-\\theta^\{\*\\top\}z\_\{t\}\-w\|\|x\_\{t\},u\_\{t\}\]=L\_\{2\}\\textstyle\\sum\_\{t=1\}^\{T\}\\mathbf\{E\}\_\{\\hat\{\\mu\}^\{\\circ\}\_\{t\}\}\[\|\(\\hat\{\\theta\}^\{\\circ\}\_\{t\}\-\\theta^\{\*\}\)^\{\\top\}z\_\{t\}\|\|x\_\{t\},u\_\{t\}\]\(45\)≤\(b\)​L2​∑t=1T𝐄μ^t∘​\[‖θ^t∘−θ∗‖V¯t∘​‖zt‖\(V¯t∘\)−1\|xt,ut\]​≤\(c\)​L2​∑t=1Tαtθ,∘​\(δ/2\)​𝐄μ^t∘​\[‖zt‖\(V¯t∘\)−1\|xt,ut\]\\displaystyle\\overset\{\(b\)\}\{\\leq\}L\_\{2\}\\textstyle\\sum\\limits\_\{t=1\}^\{T\}\\mathbf\{E\}\_\{\\hat\{\\mu\}\_\{t\}^\{\\circ\}\}\[\\\|\\hat\{\\theta\}^\{\\circ\}\_\{t\}\-\\theta^\{\*\}\\\|\_\{\\bar\{V\}^\{\\circ\}\_\{t\}\}\\\|z\_\{t\}\\\|\_\{\(\\bar\{V\}^\{\\circ\}\_\{t\}\)^\{\-1\}\}\|x\_\{t\},u\_\{t\}\]\\overset\{\(c\)\}\{\\leq\}L\_\{2\}\\textstyle\\sum\\limits\_\{t=1\}^\{T\}\\alpha^\{\\theta,\\circ\}\_\{t\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\\mathbf\{E\}\_\{\\hat\{\\mu\}\_\{t\}^\{\\circ\}\}\[\\\|z\_\{t\}\\\|\_\{\(\\bar\{V\}^\{\\circ\}\_\{t\}\)^\{\-1\}\}\|x\_\{t\},u\_\{t\}\]\(46\)\(a\)\(a\)expandsf​\(⋅\)f\(\\cdot\)to removeθ\\thetafrom the expectation and uses the Lipschitz continuity ofJ~t,δ\\tilde\{J\}\_\{t,\\delta\}\(Lemma[13\.12](https://arxiv.org/html/2605.24261#S13.Thmtheorem12)\)\.\(b\)\(b\)uses Cauchy\-Schwarz and\(c\)\(c\)uses Proposition[5\.1](https://arxiv.org/html/2605.24261#S5.Thmtheorem1)\(ℰθ​\(δ/2\)\\mathcal\{E\}\_\{\\theta\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)holds by assumption\)\. Note thatαtθ,∘​\(⋅\)≤αTθ​\(⋅\)\\alpha^\{\\theta,\\circ\}\_\{t\}\(\\cdot\)\\leq\\alpha^\{\\theta\}\_\{T\}\(\\cdot\)\. Thus\(i​v\):=\(i​v\)1\+\(i​v\)2≤\(L1\+L2\)​αTθ​\(δ/2\)​∑t=1T𝐄μ^t∘​\[‖zt‖\(V¯t∘\)−1\|xt,ut\]\(iv\):=\(iv\)\_\{1\}\+\(iv\)\_\{2\}\\leq\(L\_\{1\}\+L\_\{2\}\)\\alpha\_\{T\}^\{\\theta\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\\sum\_\{t=1\}^\{T\}\\mathbf\{E\}\_\{\\hat\{\\mu\}\_\{t\}^\{\\circ\}\}\[\\\|z\_\{t\}\\\|\_\{\(\\bar\{V\}^\{\\circ\}\_\{t\}\)^\{\-1\}\}\|x\_\{t\},u\_\{t\}\]\. Defineξt:=𝐄μ∗​\[‖zt‖\(V¯t∘\)−1\|xt,ut\]−‖zt‖\(V¯t∘\)−1\\xi\_\{t\}:=\\mathbf\{E\}\_\{\\mu^\{\*\}\}\\big\[\\\|z\_\{t\}\\\|\_\{\(\\bar\{V\}^\{\\circ\}\_\{t\}\)^\{\-1\}\}\|x\_\{t\},u\_\{t\}\\big\]\-\\\|z\_\{t\}\\\|\_\{\(\\bar\{V\}^\{\\circ\}\_\{t\}\)^\{\-1\}\}\. We add and subtract\(L1\+L2\)​αTθ​\(δ/2\)​∑t=1Tξt\(L\_\{1\}\+L\_\{2\}\)\\alpha\_\{T\}^\{\\theta\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\\sum\_\{t=1\}^\{T\}\\xi\_\{t\}:

\(i​v\)\\displaystyle\(iv\)≤\(L1\+L2\)αTθ\(δ/2\)∑t=1T\[𝐄μ^t∘​\[‖zt‖\(V¯t∘\)−1\|xt,ut\]−𝐄μ∗​\[‖zt‖\(V¯t∘\)−1\|xt,ut\]⏟\(a\)t\\displaystyle\\leq\(L\_\{1\}\+L\_\{2\}\)\\alpha^\{\\theta\}\_\{T\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\\textstyle\\sum\\nolimits\_\{t=1\}^\{T\}\[\\underbrace\{\\mathbf\{E\}\_\{\\hat\{\\mu\}^\{\\circ\}\_\{t\}\}\[\\\|z\_\{t\}\\\|\_\{\(\\bar\{V\}^\{\\circ\}\_\{t\}\)^\{\-1\}\}\|x\_\{t\},u\_\{t\}\]\-\\mathbf\{E\}\_\{\\mu^\{\*\}\}\[\\\|z\_\{t\}\\\|\_\{\(\\bar\{V\}^\{\\circ\}\_\{t\}\)^\{\-1\}\}\|x\_\{t\},u\_\{t\}\]\}\_\{\(a\)\_\{t\}\}\+𝐄μ∗​\[‖zt‖\(V¯t∘\)−1\|xt,ut\]−‖zt‖\(V¯t∘\)−1⏟\(b\)t\+‖zt‖\(V¯t∘\)−1⏟\(c\)t\]\\displaystyle\\qquad\+\\underbrace\{\\mathbf\{E\}\_\{\\mu^\{\*\}\}\[\\\|z\_\{t\}\\\|\_\{\(\\bar\{V\}^\{\\circ\}\_\{t\}\)^\{\-1\}\}\|x\_\{t\},u\_\{t\}\]\-\\\|z\_\{t\}\\\|\_\{\(\\bar\{V\}^\{\\circ\}\_\{t\}\)^\{\-1\}\}\}\_\{\(b\)\_\{t\}\}\+\\underbrace\{\\\|z\_\{t\}\\\|\_\{\(\\bar\{V\}^\{\\circ\}\_\{t\}\)^\{\-1\}\}\}\_\{\(c\)\_\{t\}\}\]\(47\)Note that\|\(a\)t\|\|\(a\)\_\{t\}\|is trivially0ifut=𝟎Mu\_\{t\}=\\bm\{0\}\_\{M\}, so assumeut=eiu\_\{t\}=e\_\{i\}\. Letz1:=\[xt​ut⊤​ut⊤\]⊤z\_\{1\}:=\[x\_\{t\}\\ u\_\{t\}^\{\\top\}\\ u\_\{t\}^\{\\top\}\]^\{\\top\}andz0:=\[xt​ut⊤​0M⊤\]⊤z\_\{0\}:=\[x\_\{t\}\\ u\_\{t\}^\{\\top\}\\ \\bm\{0\}\_\{M\}^\{\\top\}\]^\{\\top\}denote the possible state vectors under adherence and non\-adherence\. Then by expanding the expectations and simplifying we have\|\(a\)t\|≤\|σ​\(xt\+μ^i,t∘\)−σ​\(xt\+μi∗\)\|⋅\|‖z1‖\(V¯t∘\)−1−‖z0‖\(V¯t∘\)−1\|≤1/4​\|μ^i,t∘−μi∗\|⋅‖z1−z0‖\(V¯t∘\)−1\|\(a\)\_\{t\}\|\\leq\|\\sigma\(x\_\{t\}\+\\hat\{\\mu\}\_\{i,t\}^\{\\circ\}\)\-\\sigma\(x\_\{t\}\+\\mu\_\{i\}^\{\*\}\)\|\\cdot\\lvert\\\|z\_\{1\}\\\|\_\{\(\\bar\{V\}^\{\\circ\}\_\{t\}\)^\{\-1\}\}\-\\\|z\_\{0\}\\\|\_\{\(\\bar\{V\}^\{\\circ\}\_\{t\}\)^\{\-1\}\}\\rvert\\leq\\nicefrac\{\{1\}\}\{\{4\}\}\|\\hat\{\\mu\}\_\{i,t\}^\{\\circ\}\-\\mu\_\{i\}^\{\*\}\|\\cdot\\\|z\_\{1\}\-z\_\{0\}\\\|\_\{\(\\bar\{V\}^\{\\circ\}\_\{t\}\)^\{\-1\}\}\(using the Lipschitz continuity of the sigmoid and the reverse triangle inequality\)\. Note thatz1−z0=\[0​0M⊤​ei⊤\]⊤z\_\{1\}\-z\_\{0\}=\[0\\ \\bm\{0\}\_\{M\}^\{\\top\}\\ e\_\{i\}^\{\\top\}\]^\{\\top\}and\(V¯t∘\)−1⪯V¯0−1=\(1/λ1\)​I2​M\+1\(\\bar\{V\}^\{\\circ\}\_\{t\}\)^\{\-1\}\\preceq\\bar\{V\}\_\{0\}^\{\-1\}=\(\\nicefrac\{\{1\}\}\{\{\\lambda\_\{1\}\}\}\)I\_\{2M\+1\}, so‖z1−z0‖\(V¯t∘\)−1≤1/λ1\\\|z\_\{1\}\-z\_\{0\}\\\|\_\{\(\\bar\{V\}^\{\\circ\}\_\{t\}\)^\{\-1\}\}\\leq\\nicefrac\{\{1\}\}\{\{\\sqrt\{\\lambda\_\{1\}\}\}\}\. Thus by Proposition[5\.2](https://arxiv.org/html/2605.24261#S5.Thmtheorem2),\|\(a\)t\|≤14​λ1​αi,tμ,∘​\(δ/2\)\|\(a\)\_\{t\}\|\\leq\\frac\{1\}\{4\\sqrt\{\\lambda\_\{1\}\}\}\\alpha\_\{i,t\}^\{\\mu,\\circ\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\. Thus\|∑t=1T\(a\)t\|≤14​λ1​∑t=1T∑i=1Muti​αi,tμ,∘​\(δ/2\)\|\\sum\_\{t=1\}^\{T\}\(a\)\_\{t\}\|\\leq\\frac\{1\}\{4\\sqrt\{\\lambda\_\{1\}\}\}\\sum\_\{t=1\}^\{T\}\\sum\_\{i=1\}^\{M\}u\_\{t\}^\{i\}\\alpha\_\{i,t\}^\{\\mu,\\circ\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\. Applying Lemma[13\.25](https://arxiv.org/html/2605.24261#S13.Thmtheorem25)with the givenCNC\_\{N\}gives\|∑t=1T\(a\)t\|≤νT​\(δ/2\)​1\+CN4​λ1​\[M/λ2\+2​M​T/q¯\]\|\\sum\_\{t=1\}^\{T\}\(a\)\_\{t\}\|\\leq\\frac\{\\nu\_\{T\}\(\\nicefrac\{\{\\delta\}\}\{\{2\}\}\)\\sqrt\{1\+C\_\{N\}\}\}\{4\\sqrt\{\\lambda\_\{1\}\}\}\[\\nicefrac\{\{M\}\}\{\{\\sqrt\{\\lambda\_\{2\}\}\}\}\+2\\sqrt\{\\nicefrac\{\{MT\}\}\{\{\\underline\{q\}\}\}\}\]\. Next, observe that\(b\)t=ξt\(b\)\_\{t\}=\\xi\_\{t\}and that\{ξt\}\\\{\\xi\_\{t\}\\\}is a\{ℱt\}\\\{\\mathcal\{F\}\_\{t\}\\\}\-adapted martingale difference sequence \(with\|ξt\|≤1/λ1\|\\xi\_\{t\}\|\\leq\\nicefrac\{\{1\}\}\{\{\\sqrt\{\\lambda\_\{1\}\}\}\}by the same logic used to bound‖z1−z0‖\(V¯t∘\)−1\\\|z\_\{1\}\-z\_\{0\}\\\|\_\{\(\\bar\{V\}^\{\\circ\}\_\{t\}\)^\{\-1\}\}\)\. We apply Azuma\-Hoeffding\(Wainwright[2019](https://arxiv.org/html/2605.24261#bib.bib71)\)to∑t=1Tξt\\sum\_\{t=1\}^\{T\}\\xi\_\{t\}\. For someδ′∈\(0,1\)\\delta^\{\\prime\}\\in\(0,1\)we have that𝐏​\(∑t=1Tξt<2​log⁡\(1/δ′\)​Tλ1\)≥1−δ′\\mathbf\{P\}\(\\sum\_\{t=1\}^\{T\}\\xi\_\{t\}<\\sqrt\{2\\log\(\\nicefrac\{\{1\}\}\{\{\\delta^\{\\prime\}\}\}\)\\frac\{T\}\{\\lambda\_\{1\}\}\}\)\\geq 1\-\\delta^\{\\prime\}\. For\(c\)t\(c\)\_\{t\}:

∑t=1T\(c\)t\\displaystyle\\textstyle\\sum\\nolimits\_\{t=1\}^\{T\}\(c\)\_\{t\}≤\(i\)​∑t=1T12​∑t=1T‖zt‖\(V¯t∘\)−12​≤\(i​i\)​T​\(1\+Cd\)​∑t=1T‖zt‖\(V¯t\)−12\\displaystyle\\overset\{\(i\)\}\{\\leq\}\\sqrt\{\\textstyle\\sum\\nolimits\_\{t=1\}^\{T\}1^\{2\}\}\\sqrt\{\\textstyle\\sum\\nolimits\_\{t=1\}^\{T\}\\\|z\_\{t\}\\\|^\{2\}\_\{\(\\bar\{V\}\_\{t\}^\{\\circ\}\)^\{\-1\}\}\}\\overset\{\(ii\)\}\{\\leq\}\\sqrt\{T\}\\sqrt\{\(1\+C\_\{d\}\)\\textstyle\\sum\\nolimits\_\{t=1\}^\{T\}\\\|z\_\{t\}\\\|^\{2\}\_\{\(\\bar\{V\}\_\{t\}\)^\{\-1\}\}\}\(48\)\(i\)\(i\)uses Cauchy\-Schwarz\.\(i​i\)\(ii\)uses Lemma[13\.37](https://arxiv.org/html/2605.24261#S13.Thmtheorem37)to convert\(V¯t∘\)−1\(\{\\bar\{V\}\}^\{\\circ\}\_\{t\}\)^\{\-1\}to\(V¯t\)−1\(\{\\bar\{V\}\}\_\{t\}\)^\{\-1\}perCdC\_\{d\}\. Lemma[13\.34](https://arxiv.org/html/2605.24261#S13.Thmtheorem34)bounds‖zt‖\(V¯t\)−12\\\|z\_\{t\}\\\|^\{2\}\_\{\(\\bar\{V\}\_\{t\}\)^\{\-1\}\}and the lemma follows by substituting the\(a\)t\(a\)\_\{t\},\(b\)t\(b\)\_\{t\},\(c\)t\(c\)\_\{t\}summation bounds\.\\halmos

###### Lemma 13\.33

\(Abbasi\-Yadkoriet al\.\([2011](https://arxiv.org/html/2605.24261#bib.bib1)\)Lemma 11\)Let\{Xt\}t=1∞\\\{X\_\{t\}\\\}\_\{t=1\}^\{\\infty\}be a sequence inℝd\\mathbb\{R\}^\{d\},VVbe ad×dd\\times dpositive definite matrix, andV¯t:=V\+∑s=1tXs​Xs⊤\\bar\{V\}\_\{t\}:=V\+\\sum\_\{s=1\}^\{t\}X\_\{s\}X\_\{s\}^\{\\top\}\. If‖Xt‖2≤L\\\|X\_\{t\}\\\|\_\{2\}\\leq Lfor allttthen∑t=1nmin⁡\{1,‖Xt‖V¯t−1−12\}≤2​log⁡\(det\(V¯n\)det\(V\)\)≤2​\(d​log⁡\(trace⁡\(V\)\+n​L2d\)−log​det\(V\)\)\\sum\_\{t=1\}^\{n\}\\min\\\{1,\\\|X\_\{t\}\\\|\_\{\\bar\{V\}\_\{t\-1\}^\{\-1\}\}^\{2\}\\\}\\leq 2\\log\(\\frac\{\\det\(\\bar\{V\}\_\{n\}\)\}\{\\det\(V\)\}\)\\leq 2\(d\\log\(\\frac\{\\operatorname\{trace\}\(V\)\+nL^\{2\}\}\{d\}\)\-\\log\\det\(V\)\)\.

###### Lemma 13\.34

∑t=1T‖zt‖V¯t−12≤\(2​M\+1\)​log⁡\(1\+T​\(Cx2\+2\)\(2​M\+1\)​λ1\)​\[2\+Cx2\+2log⁡\(2\)​λ1\]\\sum\_\{t=1\}^\{T\}\\\|z\_\{t\}\\\|^\{2\}\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}\\leq\(2M\+1\)\\log\(1\+\\frac\{T\(C\_\{x\}^\{2\}\+2\)\}\{\(2M\+1\)\\lambda\_\{1\}\}\)\[2\+\\frac\{C\_\{x\}^\{2\}\+2\}\{\\log\(2\)\\lambda\_\{1\}\}\]\.

###### Proof 13\.35

Proof of Lemma[13\.34](https://arxiv.org/html/2605.24261#S13.Thmtheorem34)We define index setsℐg:=\{t=1,…,T:‖zt‖V¯t−12<1\}\\mathcal\{I\}\_\{g\}:=\\\{t=1,\\dots,T:\\\|z\_\{t\}\\\|^\{2\}\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}<1\\\}andℐb:=\{t=1,…,T:‖zt‖V¯t−12≥1\}\\mathcal\{I\}\_\{b\}:=\\\{t=1,\\dots,T:\\\|z\_\{t\}\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}^\{2\}\\geq 1\\\}withℐg\+ℐb=\{1,…,T\}\\mathcal\{I\}\_\{g\}\+\\mathcal\{I\}\_\{b\}=\\\{1,\\dots,T\\\}and∑t=1T‖zt‖V¯t−12=∑t∈ℐg‖zt‖V¯t−12\+∑t∈ℐb‖zt‖V¯t−12\\sum\_\{t=1\}^\{T\}\\\|z\_\{t\}\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}^\{2\}=\\sum\_\{t\\in\\mathcal\{I\}\_\{g\}\}\\\|z\_\{t\}\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}^\{2\}\+\\sum\_\{t\\in\\mathcal\{I\}\_\{b\}\}\\\|z\_\{t\}\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}^\{2\}\. For theℐg\\mathcal\{I\}\_\{g\}summation we directly apply Lemma[13\.33](https://arxiv.org/html/2605.24261#S13.Thmtheorem33)as fort∈ℐgt\\in\\mathcal\{I\}\_\{g\}we have‖zt‖V¯t−12≤min⁡\{1,‖zt‖V¯t−12\}\\\|z\_\{t\}\\\|^\{2\}\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}\\leq\\min\\\{1,\\\|z\_\{t\}\\\|^\{2\}\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}\\\}by definition\. Note that ourV¯t\\bar\{V\}\_\{t\}is equivalent to Lemma[13\.33](https://arxiv.org/html/2605.24261#S13.Thmtheorem33)’s definition ofV¯t−1\\bar\{V\}\_\{t\-1\},ztz\_\{t\}takes the place ofXtX\_\{t\},d=2​M\+1d=2M\+1, andL=Cx2\+2L=\\sqrt\{C\_\{x\}^\{2\}\+2\}\. This gives∑t∈ℐg‖zt‖V¯t−12=∑t∈ℐgmin⁡\{1,‖zt‖V¯t−12\}≤∑t=1Tmin⁡\{1,‖zt‖V¯t−12\}≤2​\(2​M\+1\)​log⁡\(1\+T​\(Cx2\+2\)\(2​M\+1\)​λ1\)\\sum\_\{t\\in\\mathcal\{I\}\_\{g\}\}\\\|z\_\{t\}\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}^\{2\}=\\sum\_\{t\\in\\mathcal\{I\}\_\{g\}\}\\min\\\{1,\\\|z\_\{t\}\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}^\{2\}\\\}\\leq\\sum\_\{t=1\}^\{T\}\\min\\\{1,\\\|z\_\{t\}\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}^\{2\}\\\}\\leq 2\(2M\+1\)\\log\(1\+\\frac\{T\(C\_\{x\}^\{2\}\+2\)\}\{\(2M\+1\)\\lambda\_\{1\}\}\)\. For theℐb\\mathcal\{I\}\_\{b\}summation we upper bound the summands and show that the number of terms in the summation grows logarithmically inTT\. We first note that fort∈ℐbt\\in\\mathcal\{I\}\_\{b\}we havedet\(V¯t\+1\)≥2​det\(V¯t\)\\det\(\\bar\{V\}\_\{t\+1\}\)\\geq 2\\det\(\\bar\{V\}\_\{t\}\), which can be seen as follows\. By definition,V¯t\+1=V\+∑s=1tzs​zs⊤\\bar\{V\}\_\{t\+1\}=V\+\\sum\_\{s=1\}^\{t\}z\_\{s\}z\_\{s\}^\{\\top\}, so we equivalently haveV¯t\+1=V¯t\+zt​zt⊤=V¯t​\(I2​M\+1\+V¯t−1​zt​zt⊤\)\\bar\{V\}\_\{t\+1\}=\\bar\{V\}\_\{t\}\+z\_\{t\}z\_\{t\}^\{\\top\}=\\bar\{V\}\_\{t\}\(I\_\{2M\+1\}\+\\bar\{V\}\_\{t\}^\{\-1\}z\_\{t\}z\_\{t\}^\{\\top\}\)\. Taking the determinant of both sides and applying the matrix determinant lemma \(Ding and Zhou \([2007](https://arxiv.org/html/2605.24261#bib.bib22)\)Lemma 1\.1\) we havedet\(V¯t\+1\)=det\(V¯t\)​\(1\+‖zt‖V¯t−12\)≥2​det\(V¯t\)\\det\(\\bar\{V\}\_\{t\+1\}\)=\\det\(\\bar\{V\}\_\{t\}\)\(1\+\\\|z\_\{t\}\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}^\{2\}\)\\geq 2\\det\(\\bar\{V\}\_\{t\}\)\. SinceV¯t\\bar\{V\}\_\{t\}is constructed from the sum of a PD matrixVVand PSD matrices,det\(V¯t\+1\)≥det\(V¯t\)\\det\(\\bar\{V\}\_\{t\+1\}\)\\geq\\det\(\\bar\{V\}\_\{t\}\)during the time steps included inℐg\\mathcal\{I\}\_\{g\}\. Thus at the end of time stepTTit must follow thatdet\(V¯T\+1\)≥2\|ℐb\|​det\(V\)\\det\(\\bar\{V\}\_\{T\+1\}\)\\geq 2^\{\|\\mathcal\{I\}\_\{b\}\|\}\\det\(V\), further implying that\|ℐb\|≤log2⁡\(det\(V¯T\+1\)det\(V\)\)=1log⁡\(2\)​log⁡\(det\(V¯T\+1\)det\(V\)\)\|\\mathcal\{I\}\_\{b\}\|\\leq\\log\_\{2\}\(\\frac\{\\det\(\\bar\{V\}\_\{T\+1\}\)\}\{\\det\(V\)\}\)=\\frac\{1\}\{\\log\(2\)\}\\log\(\\frac\{\\det\(\\bar\{V\}\_\{T\+1\}\)\}\{\\det\(V\)\}\)\. Note that this quantity can be bounded directly by Lemma[13\.33](https://arxiv.org/html/2605.24261#S13.Thmtheorem33), i\.e\.:\|ℐb\|≤2​M\+1log⁡\(2\)​log⁡\(1\+T​\(Cx2\+2\)\(2​M\+1\)​λ1\)\|\\mathcal\{I\}\_\{b\}\|\\leq\\frac\{2M\+1\}\{\\log\(2\)\}\\log\(1\+\\frac\{T\(C\_\{x\}^\{2\}\+2\)\}\{\(2M\+1\)\\lambda\_\{1\}\}\)\. Thus we can complete the bounding of theℐb\\mathcal\{I\}\_\{b\}summation by upper bounding‖zt‖V¯t−12=zt⊤​V¯t−1​zt\\\|z\_\{t\}\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}^\{2\}=z\_\{t\}^\{\\top\}\\bar\{V\}\_\{t\}^\{\-1\}z\_\{t\}\. Note thatzt≠0z\_\{t\}\\neq 0fort∈ℐbt\\in\\mathcal\{I\}\_\{b\}as this would imply‖zt‖V¯t−12=0\\\|z\_\{t\}\\\|^\{2\}\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}=0\(and thust∈ℐgt\\in\\mathcal\{I\}\_\{g\}\)\. By the upper bound of the Rayleigh quotient \(Horn and Johnson \([1985](https://arxiv.org/html/2605.24261#bib.bib35)\)Theorem 4\.2\.2\) we have‖zt‖V¯t−12≤‖zt‖22​λmax​\(V¯t−1\)=‖zt‖22​1λmin​\(V¯t\)≤‖zt‖22​1λ1≤Cx2\+2λ1\\\|z\_\{t\}\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}^\{2\}\\leq\\\|z\_\{t\}\\\|\_\{2\}^\{2\}\\lambda\_\{\\text\{max\}\}\(\\bar\{V\}\_\{t\}^\{\-1\}\)=\\\|z\_\{t\}\\\|\_\{2\}^\{2\}\\frac\{1\}\{\\lambda\_\{\\text\{min\}\}\(\\bar\{V\}\_\{t\}\)\}\\leq\\\|z\_\{t\}\\\|\_\{2\}^\{2\}\\frac\{1\}\{\\lambda\_\{1\}\}\\leq\\frac\{C\_\{x\}^\{2\}\+2\}\{\\lambda\_\{1\}\}\. This holds uniformly fort=1,…,Tt=1,\\dots,T\. Thus we have∑t∈ℐb‖zt‖V¯t−12≤\|ℐb\|​\(Cx2\+2\)/λ1\\sum\_\{t\\in\\mathcal\{I\}\_\{b\}\}\\left\\\|z\_\{t\}\\right\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}^\{2\}\\leq\|\\mathcal\{I\}\_\{b\}\|\\nicefrac\{\{\(C\_\{x\}^\{2\}\+2\)\}\}\{\{\\lambda\_\{1\}\}\}\. The lemma follows by substituting each summation’s bound:∑t=1T‖zt‖V¯t−12=∑t∈ℐg‖zt‖V¯t−12\+∑t∈ℐb‖zt‖V¯t−12\\sum\_\{t=1\}^\{T\}\\left\\\|z\_\{t\}\\right\\\|^\{2\}\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}=\\sum\_\{t\\in\\mathcal\{I\}\_\{g\}\}\\left\\\|z\_\{t\}\\right\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}^\{2\}\+\\sum\_\{t\\in\\mathcal\{I\}\_\{b\}\}\\left\\\|z\_\{t\}\\right\\\|\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}^\{2\}\.\\halmos

###### Lemma 13\.36

\(ExcerptedAbbasi\-Yadkoriet al\.\([2011](https://arxiv.org/html/2605.24261#bib.bib1)\)Lemma 12\)LetAA,BB, andCCbe positive semidefinite matrices such thatA=B\+CA=B\+C\. Thensupx≠0x⊤​A​xx⊤​B​x≤det\(A\)det\(B\)\\sup\_\{x\\neq 0\}\\frac\{x^\{\\top\}Ax\}\{x^\{\\top\}Bx\}\\leq\\frac\{\\det\(A\)\}\{\\det\(B\)\}\.

###### Lemma 13\.37

‖zt‖\(V¯t∘\)−12≤\(1\+Cd\)​‖zt‖V¯t−12\\left\\\|z\_\{t\}\\right\\\|^\{2\}\_\{\(\\bar\{V\}^\{\\circ\}\_\{t\}\)^\{\-1\}\}\\leq\(1\+C\_\{d\}\)\\left\\\|z\_\{t\}\\right\\\|^\{2\}\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}fort=1,…,Tt=1,\\dots,T\.

###### Proof 13\.38

Proof of Lemma[13\.37](https://arxiv.org/html/2605.24261#S13.Thmtheorem37)Ifzt=𝟎2​M\+1z\_\{t\}=\\bm\{0\}\_\{2M\+1\}then the lemma follows trivially, so we proceed forzt≠𝟎2​M\+1z\_\{t\}\\neq\\bm\{0\}\_\{2M\+1\}\.V¯t∘\\bar\{V\}^\{\\circ\}\_\{t\}is fixed at the beginning of the current epochkk, implying thatV¯t=V¯t∘\+∑s=τkt−1zs​zs⊤\\bar\{V\}\_\{t\}=\\bar\{V\}^\{\\circ\}\_\{t\}\+\\sum\_\{s=\\tau\_\{k\}\}^\{t\-1\}z\_\{s\}z\_\{s\}^\{\\top\}\. SinceV¯t\\bar\{V\}\_\{t\}andV¯t∘\\bar\{V\}^\{\\circ\}\_\{t\}are PD by construction andV¯t⪰V¯t∘\\bar\{V\}\_\{t\}\\succeq\\bar\{V\}\_\{t\}^\{\\circ\}, PD matrices\(V¯t\)−1\(\\bar\{V\}\_\{t\}\)^\{\-1\}and\(V¯t∘\)−1\(\\bar\{V\}^\{\\circ\}\_\{t\}\)^\{\-1\}exist with\(V¯t∘\)−1⪰\(V¯t\)−1\(\\bar\{V\}^\{\\circ\}\_\{t\}\)^\{\-1\}\\succeq\(\\bar\{V\}\_\{t\}\)^\{\-1\}\. Thus the difference\(V¯t∘\)−1−\(V¯t\)−1\(\\bar\{V\}^\{\\circ\}\_\{t\}\)^\{\-1\}\-\(\\bar\{V\}\_\{t\}\)^\{\-1\}is PSD\. Trivially we have\(V¯t∘\)−1=\(V¯t\)−1\+\[\(V¯t∘\)−1−\(V¯t\)−1\]\(\\bar\{V\}^\{\\circ\}\_\{t\}\)^\{\-1\}=\(\\bar\{V\}\_\{t\}\)^\{\-1\}\+\[\(\\bar\{V\}^\{\\circ\}\_\{t\}\)^\{\-1\}\-\(\\bar\{V\}\_\{t\}\)^\{\-1\}\]\. Applying Lemma[13\.36](https://arxiv.org/html/2605.24261#S13.Thmtheorem36)with𝐀=\(V¯t∘\)−1\\mathbf\{A\}=\(\\bar\{V\}^\{\\circ\}\_\{t\}\)^\{\-1\},𝐁=\(V¯t\)−1\\mathbf\{B\}=\(\\bar\{V\}\_\{t\}\)^\{\-1\}and𝐂=\(V¯t∘\)−1−\(V¯t\)−1\\mathbf\{C\}=\(\\bar\{V\}^\{\\circ\}\_\{t\}\)^\{\-1\}\-\(\\bar\{V\}\_\{t\}\)^\{\-1\}implies for anyt=1,…,Tt=1,\\dots,T,‖zt‖\(V¯t∘\)−12‖zt‖V¯t−12≤supzt≠0zt⊤​\(V¯t∘\)−1​ztzt⊤​\(V¯t\)−1​zt≤det\(\(V¯t∘\)−1\)det\(\(V¯t\)−1\)=det\(V¯t\)det\(V¯t∘\)\\frac\{\\left\\\|z\_\{t\}\\right\\\|^\{2\}\_\{\(\\bar\{V\}^\{\\circ\}\_\{t\}\)^\{\-1\}\}\}\{\\left\\\|z\_\{t\}\\right\\\|^\{2\}\_\{\\bar\{V\}\_\{t\}^\{\-1\}\}\}\\leq\\sup\_\{z\_\{t\}\\neq 0\}\\frac\{z\_\{t\}^\{\\top\}\(\\bar\{V\}^\{\\circ\}\_\{t\}\)^\{\-1\}z\_\{t\}\}\{z\_\{t\}^\{\\top\}\(\\bar\{V\}\_\{t\}\)^\{\-1\}z\_\{t\}\}\\leq\\frac\{\\det\\left\(\(\\bar\{V\}^\{\\circ\}\_\{t\}\)^\{\-1\}\\right\)\}\{\\det\\left\(\(\\bar\{V\}\_\{t\}\)^\{\-1\}\\right\)\}=\\frac\{\\det\\left\(\\bar\{V\}\_\{t\}\\right\)\}\{\\det\\left\(\\bar\{V\}^\{\\circ\}\_\{t\}\\right\)\}\. Algorithm[1](https://arxiv.org/html/2605.24261#alg1)triggers new epochs such thatdet\(V¯t\)≤\(1\+Cd\)​det\(V¯t∘\)\\det\\left\(\\bar\{V\}\_\{t\}\\right\)\\leq\(1\+C\_\{d\}\)\\det\\left\(\\bar\{V\}^\{\\circ\}\_\{t\}\\right\)holds fort=1,…,Tt=1,\\dots,T\. so the RHS is upper bounded by\(1\+Cd\)\(1\+C\_\{d\}\)and the lemma follows\.\\halmos

###### Lemma 13\.39

Letδ′∈\(0,1\)\\delta^\{\\prime\}\\in\(0,1\)and defineζt:=𝐄θ∗,μ∗​\[J~t,δ∘​\(f​\(⋅\)\)−Jπt​\(f​\(⋅\)\)\]−\(J~t,δ∘​\(xt\+1\)−Jπt​\(xt\+1\)\)\\zeta\_\{t\}:=\\mathbf\{E\}\_\{\\theta^\{\*\},\\mu^\{\*\}\}\\big\[\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(f\(\\cdot\)\)\-J^\{\\pi\_\{t\}\}\(f\(\\cdot\)\)\\big\]\-\\big\(\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(x\_\{t\+1\}\)\-J^\{\\pi\_\{t\}\}\(x\_\{t\+1\}\)\\big\)\. With probability at least1−δ′1\-\\delta^\{\\prime\},∑t=1Tζt≤8​T​\(CJ~,T\+1,δ\+CJ\)2​log⁡\(1/δ′\)\\sum\_\{t=1\}^\{T\}\\zeta\_\{t\}\\leq\\sqrt\{8T\(C\_\{\\tilde\{J\},T\+1,\\delta\}\+C\_\{J\}\)^\{2\}\\log\(\\nicefrac\{\{1\}\}\{\{\\delta^\{\\prime\}\}\}\)\}\.

###### Proof 13\.40

Proof of Lemma[13\.39](https://arxiv.org/html/2605.24261#S13.Thmtheorem39)By Lemma[5\.3](https://arxiv.org/html/2605.24261#S5.Thmtheorem3), fort≤Tt\\leq Tandδ∈\(0,1\)\\delta\\in\(0,1\)‖J~t,δ∘‖∞≤CJ~,T,δ≤CJ~,T\+1,δ\\\|\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\\\|\_\{\\infty\}\\leq C\_\{\\tilde\{J\},T,\\delta\}\\leq C\_\{\\tilde\{J\},T\+1,\\delta\}and‖Jπt‖∞≤CJ\\\|J^\{\\pi\_\{t\}\}\\\|\_\{\\infty\}\\leq C\_\{J\}, so‖Jt,δ∘−Jπt‖∞≤CJ~,T\+1,δ\+CJ\\\|J^\{\\circ\}\_\{t,\\delta\}\-J^\{\\pi\_\{t\}\}\\\|\_\{\\infty\}\\leq C\_\{\\tilde\{J\},T\+1,\\delta\}\+C\_\{J\}\. Observe that\{ζt\}\\\{\\zeta\_\{t\}\\\}is a\{ℱt\}\\\{\\mathcal\{F\}\_\{t\}\\\}\-adapted martingale difference sequence \(as the expectation is with respect to the true parameters\) with\|ζt\|≤2​\(CJ~,T\+1,δ\+CJ\)\|\\zeta\_\{t\}\|\\leq 2\(C\_\{\\tilde\{J\},T\+1,\\delta\}\+C\_\{J\}\)\. We apply Azuma\-Hoeffding\(Wainwright[2019](https://arxiv.org/html/2605.24261#bib.bib71)\)forδ′∈\(0,1\)\\delta^\{\\prime\}\\in\(0,1\)\. Thus𝐏​\(∑t=1Tζt≥q\)≤exp⁡\(−q28​∑t=1T\(CJ~,T\+1,δ\+CJ\)2\)\\mathbf\{P\}\\big\(\\sum\_\{t=1\}^\{T\}\\zeta\_\{t\}\\geq q\\big\)\\leq\\exp\{\(\-\\frac\{q^\{2\}\}\{8\\sum\_\{t=1\}^\{T\}\(C\_\{\\tilde\{J\},T\+1,\\delta\}\+C\_\{J\}\)^\{2\}\}\)\}\. The lemma follows by solving forqqso the RHS equalsδ′\\delta^\{\\prime\}\.\\halmos

###### Lemma 13\.41

∑t=1T\[J~t,δ∘​\(xt\+1\)−J~t\+1,δ∘​\(xt\+1\)\+Jπt\+1​\(xt\+1\)−Jπt​\(xt\+1\)\]≤2​\(Kd\+KN\)​\(CJ~,T\+1,δ\+CJ\)\\sum\\limits\_\{t=1\}^\{T\}\[\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(x\_\{t\+1\}\)\-\\tilde\{J\}^\{\\circ\}\_\{t\+1,\\delta\}\(x\_\{t\+1\}\)\+J^\{\\pi\_\{t\+1\}\}\(x\_\{t\+1\}\)\-J^\{\\pi\_\{t\}\}\(x\_\{t\+1\}\)\]\\leq 2\(K\_\{d\}\+K\_\{N\}\)\(C\_\{\\tilde\{J\},T\+1,\\delta\}\+C\_\{J\}\)\.

###### Proof 13\.42

Proof of Lemma[13\.41](https://arxiv.org/html/2605.24261#S13.Thmtheorem41)For Algorithm[1](https://arxiv.org/html/2605.24261#alg1),J~t,δ∘=J~t\+1,δ∘\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}=\\tilde\{J\}^\{\\circ\}\_\{t\+1,\\delta\}andJπt=Jπt\+1J^\{\\pi\_\{t\}\}=J^\{\\pi\_\{t\+1\}\}for every time step except the final time step of each epoch\. We define the index setℐc\\mathcal\{I\}\_\{c\}to contain such steps, i\.e\.,\{t∈\{1,…,T\}:t=τk−1,k=2,…,K\+1\}\\\{t\\in\\\{1,\\dots,T\\\}:t=\\tau\_\{k\}\-1,\\;k=2,\\dots,K\+1\\\}\. The lemma summands are nonzero only for time indices inℐc\\mathcal\{I\}\_\{c\}\. By Lemma[5\.3](https://arxiv.org/html/2605.24261#S5.Thmtheorem3),‖J~t,δ‖∞≤CJ~,T,δ≤CJ~,T\+1,δ\\\|\\tilde\{J\}\_\{t,\\delta\}\\\|\_\{\\infty\}\\leq C\_\{\\tilde\{J\},T,\\delta\}\\leq C\_\{\\tilde\{J\},T\+1,\\delta\}and‖Jπt‖∞≤CJ\\\|J^\{\\pi\_\{t\}\}\\\|\_\{\\infty\}\\leq C\_\{J\}fort=1,…,T\+1t=1,\\dots,T\+1, thus the differences between these value function are also bounded\. Specifically,J~t,δ∘​\(xt\+1\)−J~t\+1,δ∘​\(xt\+1\)≤2​CJ~,T\+1,δ\\tilde\{J\}^\{\\circ\}\_\{t,\\delta\}\(x\_\{t\+1\}\)\-\\tilde\{J\}^\{\\circ\}\_\{t\+1,\\delta\}\(x\_\{t\+1\}\)\\leq 2C\_\{\\tilde\{J\},T\+1,\\delta\}andJπt\+1​\(xt\+1\)−Jπt​\(xt\+1\)≤2​CJJ^\{\\pi\_\{t\+1\}\}\(x\_\{t\+1\}\)\-J^\{\\pi\_\{t\}\}\(x\_\{t\+1\}\)\\leq 2C\_\{J\}\. We next bound the number of new epoch triggers\|ℐc\|\|\\mathcal\{I\}\_\{c\}\|\. First consider the determinant trigger\. Denotendn\_\{d\}as the number of times the determinant trigger condition in Algorithm[1](https://arxiv.org/html/2605.24261#alg1),det\(V¯t\+1\)\>\(1\+Cd\)​det\(V¯t∘\)\\det\(\\bar\{V\}\_\{t\+1\}\)\>\(1\+C\_\{d\}\)\\det\(\\bar\{V\}^\{\\circ\}\_\{t\}\), is met\. Further note that, as a second moment matrix, time steps where this condition onV¯t\+1\\bar\{V\}\_\{t\+1\}is not met cannot decrease the determinant\. Thus for a trajectoryt=1,…,Tt=1,\\dots,T, we havedet\(V¯T\+1\)≥\(1\+Cd\)nd​det\(V¯1∘\)=\(1\+Cd\)nd​det\(V\)\\det\(\\bar\{V\}\_\{T\+1\}\)\\geq\(1\+C\_\{d\}\)^\{n\_\{d\}\}\\det\(\\bar\{V\}^\{\\circ\}\_\{1\}\)=\(1\+C\_\{d\}\)^\{n\_\{d\}\}\\det\(V\)\. Consequently,nd≤log1\+Cd⁡\(det\(V¯T\+1\)det\(V\)\)=1log⁡\(1\+Cd\)​log⁡\(det\(V¯T\+1\)det\(V\)\)n\_\{d\}\\leq\\log\_\{1\+C\_\{d\}\}\(\\frac\{\\det\(\\bar\{V\}\_\{T\+1\}\)\}\{\\det\(V\)\}\)=\\frac\{1\}\{\\log\(1\+C\_\{d\}\)\}\\log\(\\frac\{\\det\(\\bar\{V\}\_\{T\+1\}\)\}\{\\det\(V\)\}\)\. Using Lemma[13\.33](https://arxiv.org/html/2605.24261#S13.Thmtheorem33)to bound thelog​det\\log\\detratio, we then havend≤\(2​M\+1\)log⁡\(1\+Cd\)​log⁡\(1\+T​\(Cx2\+2\)\(2​M\+1\)​λ1\)n\_\{d\}\\leq\\frac\{\(2M\+1\)\}\{\\log\(1\+C\_\{d\}\)\}\\log\(1\+\\frac\{T\(C\_\{x\}^\{2\}\+2\)\}\{\(2M\+1\)\\lambda\_\{1\}\}\)\. Next, we consider the action count trigger, withNi,t:=∑s=1t−1𝟏\{us=ei\}N\_\{i,t\}:=\\sum\_\{s=1\}^\{t\-1\}\\mathbf\{1\}\_\{\\\{u\_\{s\}=e\_\{i\}\\\}\}\. LetnN,in\_\{N,i\}denote the count of triggering time steps, i\.e\., whenNi,t\+1\>\(1\+CN\)​Ni,τkN\_\{i,t\+1\}\>\(1\+C\_\{N\}\)N\_\{i,\\tau\_\{k\}\}, fori∈\{1,…,M\}i\\in\\\{1,\\dots,M\\\}\.Ni,T\+1≤TN\_\{i,T\+1\}\\leq T, implyingnN,i≤1\+log1\+CN⁡\(T\)=1\+log⁡\(T\)log⁡\(1\+CN\)n\_\{N,i\}\\leq 1\+\\log\_\{1\+C\_\{N\}\}\(T\)=1\+\\frac\{\\log\(T\)\}\{\\log\(1\+C\_\{N\}\)\}\. Thus it follows that\|ℐc\|≤nd\+∑i=1MnN,i≤\(2​M\+1\)log⁡\(1\+Cd\)​log⁡\(1\+T​\(Cx2\+2\)\(2​M\+1\)​λ1\)\+M​\(1\+log⁡\(T\)log⁡\(1\+CN\)\)\|\\mathcal\{I\}\_\{c\}\|\\leq n\_\{d\}\+\\sum\_\{i=1\}^\{M\}n\_\{N,i\}\\leq\\frac\{\(2M\+1\)\}\{\\log\(1\+C\_\{d\}\)\}\\log\(1\+\\frac\{T\(C\_\{x\}^\{2\}\+2\)\}\{\(2M\+1\)\\lambda\_\{1\}\}\)\+M\(1\+\\frac\{\\log\(T\)\}\{\\log\(1\+C\_\{N\}\)\}\)\. The lemma follows by bounding the number of nonzero summation indices and the maximum contribution of each nonzero summand, i\.e\.,\|ℐc\|⋅2​\(CJ~,T\+1,δ\+CJ\)\|\\mathcal\{I\}\_\{c\}\|\\cdot 2\(C\_\{\\tilde\{J\},T\+1,\\delta\}\+C\_\{J\}\)\.\\halmos