Reinforcement Learning for Continuous-Time Jump Markov Decision Processes with Applications to Network Dynamic Pricing

# Reinforcement Learning for Continuous-Time Jump Markov Decision Processes with Applications to Network Dynamic Pricing
Source: [https://arxiv.org/html/2608.20680](https://arxiv.org/html/2608.20680)
Huiling MengNote:Department of Systems Engineering and Engineering Management, The Chinese University of Hong Kong, Hong Kong, China\. Email:hlmeng@link\.cuhk\.edu\.hkNingyuan ChenNote:Rotman School of Management, University of Toronto, Toronto, Canada, Email: ningyuan\.chen@utoronto\.caXuefeng GaoNote:Department of Systems Engineering and Engineering Management, The Chinese University of Hong Kong, Hong Kong, China\. Email:xfgao@se\.cuhk\.edu\.hk

August 21, 2026

###### Abstract

We study reinforcement learning \(RL\) in Continuous\-Time Jump Markov Decision Processes \(CTJMDPs\) featuring general discrete state spaces \(which need not possess a vector space structure\) and continuous/discrete action spaces\. The setup covers many well\-known applications in operations such as multi\-product dynamic pricing with capacitated resources\([11](https://arxiv.org/html/2608.20680#bib.bib26)\)\. To model the exploration\-exploitation tradeoff, we formulate an entropy\-regularized continuous\-time control problem with stochastic policies\. Recent continuous\-time RL techniques such asqq\-learning for controlled diffusions in\([20](https://arxiv.org/html/2608.20680#bib.bib13)\)focus on continuous state spacesℝd\\mathbb\{R\}^\{d\}and rely heavily on semimartingale theory inℝd\\mathbb\{R\}^\{d\}for their theoretical analysis\. Consequently, their methods cannot be directly applied to CTJMDPs with general discrete state spaces, which may lack the algebraic addition and subtraction structures inherent to Euclidean spaces\. To bridge this gap, we establish the theoretical foundations ofqq\-learning for CTJMDPs and develop model\-freeqq\-learning algorithms\. Compared to naïve time discretization and approximating CTJMDPs using discrete\-time MDPs, our approach has several conceptual and empirical benefits\. Numerical experiments in network dynamic pricing\([11](https://arxiv.org/html/2608.20680#bib.bib26)\)show that our proposed RL algorithm reliably learns near\-optimal policies and consistently outperforms standard benchmark methods, demonstrating superior solution quality and effective scalability to large\-scale network instances\.

## 1Introduction

Reinforcement Learning \(RL\) is a powerful approach that enables agents to learn optimal policies through interaction with their environment\([32](https://arxiv.org/html/2608.20680#bib.bib2)\)\. Despite the vast literature on RL, most existing work is grounded in discrete\-time Markov Decision Processes \(MDPs\), which provide a mathematical framework for modeling sequential decision\-making\. However, many real\-world physical systems, such as autonomous driving, high\-frequency trading, and robot navigation, operate in continuous time, requiring real\-time monitoring and decision\-making\. In particular, decisions are not necessarily made at fixed intervals, rendering discrete\-time models inadequate\. For continuous\-time decision problems, one can discretize time uniformly upfront and apply existing RL algorithms developed for discrete\-time MDPs\. However, this approach could be highly sensitive to the choice of discretization step and performs poorly with small time steps \(see e\.g\.[28](https://arxiv.org/html/2608.20680#bib.bib15),[29](https://arxiv.org/html/2608.20680#bib.bib21),[33](https://arxiv.org/html/2608.20680#bib.bib3)\)\. These limitations, coupled with the rich analytical tools in the continuous\-time setting, have sparked a surge of interest in continuous\-time RL in recent years\([7](https://arxiv.org/html/2608.20680#bib.bib25),[12](https://arxiv.org/html/2608.20680#bib.bib17),[16](https://arxiv.org/html/2608.20680#bib.bib24),[20](https://arxiv.org/html/2608.20680#bib.bib13),[35](https://arxiv.org/html/2608.20680#bib.bib18),[36](https://arxiv.org/html/2608.20680#bib.bib23),[39](https://arxiv.org/html/2608.20680#bib.bib22)\)\. Existing studies in this line of research predominantly focus on controlled systems withcontinuousstate spaces inℝd\\mathbb\{R\}^\{d\}, where system dynamics are governed by stochastic differential equations \(SDEs\)\.

In contrast, this paper focuses on continuous\-time RL indiscretestate spaces\. We consider the Continuous\-Time Jump Markov Decision Process \(CTJMDP\)\([9](https://arxiv.org/html/2608.20680#bib.bib11)\)with a denumerable state space as our mathematical decision model, which is a natural continuous\-time extension of discrete\-time MDPs\. The CTJMDPs we study have the following features: 1\) the system is observed continuously and \(deterministic\) actions can be made at any point in time; 2\) the state space is denumerable and need not possess a vector space structure such asℝd\\mathbb\{R\}^\{d\}; 3\) the action space may be discrete or continuous; 4\) the transition rates may be time\-dependent; and 5\) the reward structure includes both running reward rates and jump rewards, both of which may be time\-dependent and unbounded\. These features preclude the direct application of existing RL theories and necessitate novel methodological tools\. Critically, CTJMDPs differ from Semi\-Markov Decision Processes \(SMDP\)\([30](https://arxiv.org/html/2608.20680#bib.bib4), Chapter 11\), in which decisions are restricted to state\-transition epochs and cannot be updated continuously\. Consequently, CTJMDPs provide an ideal framework for modeling pure\-jump stochastic systems subject to real\-time control, with broad applications in operations research such as queueing control\([4](https://arxiv.org/html/2608.20680#bib.bib14)\), population management\([15](https://arxiv.org/html/2608.20680#bib.bib7)\), and dynamic pricing\([10](https://arxiv.org/html/2608.20680#bib.bib9)\)\. Despite their modeling flexibility and rich theoretical foundation\([27](https://arxiv.org/html/2608.20680#bib.bib8),[9](https://arxiv.org/html/2608.20680#bib.bib11),[15](https://arxiv.org/html/2608.20680#bib.bib7)\), model\-free RL algorithms for CTJMDPs, where underlying system dynamics are unknown, remain largely unexplored\. We address this fundamental gap by developing interpretable and scalable RL algorithms for CTJMDPs\.

### 1\.1Contributions

In this study, we focus on the finite\-horizon episodic RL setting, where an agent repeatedly interacts with the environment across multiple episodes, aiming to learn an optimal policy of a CTJMDP\. The main contributions of the paper are summarized below\.

- •First, we develop a comprehensive and principled framework for model\-free RL in CTJMDPs, encompassing both theoretical foundations and algorithm design\. Our framework extends the continuous\-timeqq\-learning theory, originally developed by[20](https://arxiv.org/html/2608.20680#bib.bib13)for controlled SDEs as a continuous\-time analogue of discrete\-timeQQ\-learning, to discrete\-state jump systems\. Unlike theℝd\\mathbb\{R\}^\{d\}\-valued diffusions in[20](https://arxiv.org/html/2608.20680#bib.bib13)or theℝd\\mathbb\{R\}^\{d\}\-valued jump\-diffusions in[12](https://arxiv.org/html/2608.20680#bib.bib17), RL in CTJMDPs pose unique theoretical challenges: their discrete state spaces generally lack vector\-space operations \(e\.g\., addition, subtraction\) and do not support differential calculus\. As a result, the continuous\-time RL analysis in\([20](https://arxiv.org/html/2608.20680#bib.bib13),[12](https://arxiv.org/html/2608.20680#bib.bib17)\), which relies heavily on semimartingales inℝd\\mathbb\{R\}^\{d\}and Itô’s formula, fails in our setting\. We overcome these challenges by establishing a Dynkin’s formula \(Theorem[1](https://arxiv.org/html/2608.20680#Thmtheorem1)\) tailored to the observable state process of a CTJMDP under randomized Markov policies\. This result enables a martingale characterization of the optimal value function andqq\-function using sample trajectories \(Theorem[2](https://arxiv.org/html/2608.20680#Thmtheorem2)\)\. Crucially, our derivation \(of Theorem[2](https://arxiv.org/html/2608.20680#Thmtheorem2)\) departs from the corresponding results in\([20](https://arxiv.org/html/2608.20680#bib.bib13),[12](https://arxiv.org/html/2608.20680#bib.bib17)\)by accommodating the jump rewards inherent to CTJMDPs under milder conditions\. This martingale characterization yields interpretableqq\-learning algorithms for CTJMDPs\. Moreover, unlike SDEs, sample paths of CTJMDPs are piecewise constant, with jumps occurring at discrete points in time\. This unique feature allows for more accurate approximations of the integrals that arise in value functions and it is incorporated in our development ofqq\-learning algorithms for CTJMDPs\.
- •Second, we apply the framework to the classical network dynamic pricing problem of[11](https://arxiv.org/html/2608.20680#bib.bib26)in an airline network\. The problem combines a finite horizon, a discrete state, a continuous price action, and unknown demand functions; shared flight legs make both the state and the pricing decision high\-dimensional and coupled\. In the small network, the learned policy achieves revenue within2\.39%2\.39\\%of the time\-discretized DP benchmark and improves on the two heuristic policies in[11](https://arxiv.org/html/2608.20680#bib.bib26)\. In the large network, which has approximately5\.88×10135\.88\\times 10^\{13\}states and an 18\-dimensional continuous action space, the learned policy slightly outperforms both heuristics even though they use the known demand functions\. These results show that the model\-freeqq\-learning algorithm can learn a near\-optimal policy in the small network and scale to the large network, for which direct dynamic programming is infeasible\. Appendix[A](https://arxiv.org/html/2608.20680#A1)applies the same continuous\-time RL framework to a finite\-horizon dynamic server allocation problem, demonstrating its broad applicability to queueing control\.

### 1\.2Related Work

Our work is most closely related to two research streams, including RL for continuous\-time jump decision processes, and network revenue management\. Below, we highlight the key distinctions between our work and existing studies\.

#### RL for continuous\-time jump decision models\.

Early foundational work introduced continuous time RL by adapting algorithms such asQQ\-learning to infinite\-horizon SMDPs\([3](https://arxiv.org/html/2608.20680#bib.bib5),[8](https://arxiv.org/html/2608.20680#bib.bib6)\)\. A standard approach for infinite\-horizon SMDPs is to apply uniformization, converting the system into an equivalent discrete\-time MDP to leverage discrete\-time RL techniques\([6](https://arxiv.org/html/2608.20680#bib.bib19), e\.g\.,\)\. However, uniformization fails for the finite\-horizon CTJMDPs considered in this paper\. Because optimal policies in finite\-horizon settings are non\-stationary and depend explicitly on the continuous time indextt, the system cannot be mapped to an equivalent discrete\-time MDP\. From a theoretical standpoint,[13](https://arxiv.org/html/2608.20680#bib.bib10)and[14](https://arxiv.org/html/2608.20680#bib.bib20)recently established regret bounds for tabular continuous\-time MDPs under infinite\-horizon average\-reward and finite\-horizon episodic settings, respectively\. Their decision models belong to the class of exponential SMDPs\([9](https://arxiv.org/html/2608.20680#bib.bib11)\), which differ fundamentally from our CTJMDP formulation: their framework restricts decision epochs to jump times, whereas our framework permits continuous\-time action adjustments between jump events\. More recently,[25](https://arxiv.org/html/2608.20680#bib.bib16)developed a policy\-gradient algorithm specifically for continuous\-time intensity control in choice\-based network revenue management\. In contrast, our work establishes a unified, model\-free RL framework for general CTJMDPs; the intensity control setting in their work can be viewed as a special case within our broader framework\.

As this work was being finalized, a concurrent pre\-print by[38](https://arxiv.org/html/2608.20680#bib.bib30)appeared on arXiv\. While they also explore reinforcement learning for controlled continuous\-time Markov chains in discrete state spaces, specifically proposing continuous\-time variants of proximal policy optimization to fine\-tune discrete generative diffusion models, our work was developed independently and differs fundamentally in its theoretical results, algorithmic design, and target applications\.

#### Network revenue management\.

The classical network dynamic pricing model of[11](https://arxiv.org/html/2608.20680#bib.bib26)considers multiple products that consume finite, shared resources\. Customers arrive in continuous time, and the firm continuously adjusts a vector of prices based on the remaining capacities\. The model is a natural CTJMDP: remaining capacities form a discrete state, sales generate state jumps, and prices form a continuous action space\. Related work studies the same continuous\-time and continuous\-price formulation\([1](https://arxiv.org/html/2608.20680#bib.bib39), e\.g\.,\)Since then, there are many variants of the problem being proposed and studied\. Notably, the action space can become discrete \(e\.g\., an accept/reject decision in[34](https://arxiv.org/html/2608.20680#bib.bib31)or an assortment decision in[23](https://arxiv.org/html/2608.20680#bib.bib32)\) and the horizon can become discrete, e\.g\., in[24](https://arxiv.org/html/2608.20680#bib.bib33),[37](https://arxiv.org/html/2608.20680#bib.bib37),[19](https://arxiv.org/html/2608.20680#bib.bib38)\. These changes make the problem more tractable and allow discrete\-time reinforcement learning approaches to be applied, e\.g\.,[22](https://arxiv.org/html/2608.20680#bib.bib29)\. However, the discretization of the horizon comes with approximation errors, which are typically hard to analyze and may induce numerical issues when the grid is too fine\. In this study, we retain the continuous\-time and continuous\-price formulation of[11](https://arxiv.org/html/2608.20680#bib.bib26), and develop a principled RL approach to solve the network dynamic pricing problem with arbitrary unknown demand functions\.

Related, there is extensive literature on demand learning for network revenue management\. They study the setting in[11](https://arxiv.org/html/2608.20680#bib.bib26)but the demand function is unknown initially\.[2](https://arxiv.org/html/2608.20680#bib.bib34)is one of the earliest studies and uses the continuous\-time setup\. Many of the studies use the discrete\-time and continuous\-action setup,\([5](https://arxiv.org/html/2608.20680#bib.bib36),[26](https://arxiv.org/html/2608.20680#bib.bib35), e\.g\.,\)\. This body of literature primarily focuses on designing efficient online learning algorithms with regret guarantees to balance the learning–earning tradeoff under demand uncertainty\. Unlike these studies, our focus is to develop computational model\-free RL methods that can handle high dimensionality and function approximation in massive state spaces\. Several such studies have appeared in the literature\. For instance,[31](https://arxiv.org/html/2608.20680#bib.bib28)applies online tabularQQ\-learning to a multi\-product setting where products interact through cross\-price demand effects with separate capacities, rather than competing for shared network resources\. Meanwhile,[18](https://arxiv.org/html/2608.20680#bib.bib27)adapts the offline actor\-critic method to learn network dynamic pricing policies from historical sales data\. These studies typically formulate the problem as a discrete\-time MDP with a discrete\-time and discrete\-action setup\. Hence, they are fundamentally different from our work\.

Notations\.For any given functionw:𝒳↦\(0,∞\)w:\\mathcal\{X\}\\mapsto\(0,\\infty\), we introduce some notation associated withwwas follows\. A functionφ:\[0,T\]×𝒳↦ℝ\\varphi:\[0,T\]\\times\\mathcal\{X\}\\mapsto\\mathbb\{R\}is calledww\-bounded if theww\-weighted norm ofφ\\varphi,‖φ‖w≜sup\(t,x\)∈\[0,T\]×𝒳\|φ⁡\(t,x\)\|w⁡\(x\)\\\|\\varphi\\\|\_\{w\}\\triangleq\\sup\_\{\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\}\}\\frac\{\|\\varphi\(t,x\)\|\}\{w\(x\)\}, is finite\. LetBw​\(\[0,T\]×𝒳\)B\_\{w\}\(\[0,T\]\\times\\mathcal\{X\}\)be the set of allww\-bounded Borel measurable functions on\[0,T\]×𝒳\[0,T\]\\times\\mathcal\{X\}; then it is a Banach space when endowed with the norm∥⋅∥w\\\|\\cdot\\\|\_\{w\}\. Moreover, a functionφ:\[0,T\]×𝒳↦ℝ\\varphi:\[0,T\]\\times\\mathcal\{X\}\\mapsto\\mathbb\{R\}is called essentiallyww\-bounded if there exists a Lebesgue null setZ0⊂\[0,T\]Z\_\{0\}\\subset\[0,T\]such that‖φ‖wes≜sup\(t,x\)∈\[0,T\]\\Z0×𝒳\|φ⁡\(t,x\)\|w⁡\(x\)<∞\\\|\\varphi\\\|\_\{w\}^\{\\mathrm\{es\}\}\\triangleq\\sup\_\{\(t,x\)\\in\[0,T\]\\backslash Z\_\{0\}\\times\\mathcal\{X\}\}\\frac\{\|\\varphi\(t,x\)\|\}\{w\(x\)\}<\\infty\. We denoteCw1,0​\(\[0,T\]×𝒳\)C\_\{w\}^\{1,0\}\(\[0,T\]\\times\\mathcal\{X\}\)the set of functionsφ∈Bw​\(\[0,T\]×𝒳\)\\varphi\\in B\_\{w\}\(\[0,T\]\\times\\mathcal\{X\}\)that satisfy: \(i\)φ⁡\(t,x\)\\varphi\(t,x\)is absolutely continuous int∈\[0,T\]t\\in\[0,T\]for each fixedx∈𝒳x\\in\\mathcal\{X\}; \(ii\)∂φ∂t​\(t,x\)\\frac\{\\partial\\varphi\}\{\\partial t\}\(t,x\)is universally measurable and essentiallyww\-bounded on\[0,T\]×𝒳\[0,T\]\\times\\mathcal\{X\}\.

## 2Classical Finite\-Horizon CTJMDPs

We consider a finite\-horizon CTJMDP defined by a tuple

\{T,𝒳,𝒜,λ⁡\(y∣t,x,a\),r⁡\(t,x,a\),ρ⁡\(t,x,a,y\),h⁡\(x\)\},\\displaystyle\\\{T,\\,\\mathcal\{X\},\\,\\mathcal\{A\},\\,\\lambda\(y\\mid t,x,a\),\\,r\(t,x,a\),\\,\\rho\(t,x,a,y\),\\,h\(x\)\\\},which are interpreted as follows:

- \(i\)T\>0T\>0denotes the finite planning horizon;
- \(ii\)𝒳\\mathcal\{X\}is a denumerable \(or countable\) state space, which may not possess a vector space structure;
- \(iii\)𝒜⊂ℝn\\mathcal\{A\}\\subset\\mathbb\{R\}^\{n\}is an action space, which can be either discrete or continuous;
- \(iv\)For each\(t,x,a\)∈𝕄≜\[0,T\]×𝒳×𝒜\(t,x,a\)\\in\\mathbb\{M\}\\triangleq\[0,T\]\\times\\mathcal\{X\}\\times\\mathcal\{A\}andy≠xy\\neq x,λ⁡\(y∣t,x,a\)≥0\\lambda\(y\\mid t,x,a\)\\geq 0defines the transition rate from statexxto stateyy, andλ\(x∣t,x,a\)≜−∑y≠xλ\(y∣t,x,a\)\\lambda\(x\\mid t,x,a\)\\triangleq\-\\sum\_\{y\\neq x\}\\lambda\(y\\mid t,x,a\)\. For eachy∈𝒳y\\in\\mathcal\{X\},λ⁡\(y∣t,x,a\)\\lambda\(y\\mid t,x,a\)is Borel measurable on𝕄\\mathbb\{M\}\. Moreover, the transition rates are stable in the sense that λ∗​\(x\)≜supt∈\[0,T\],a∈𝒜λ⁡\(t,x,a\)<∞,x∈𝒳\.\\displaystyle\\lambda^\{\*\}\(x\)\\triangleq\\sup\_\{t\\in\[0,T\],\\,a\\in\\mathcal\{A\}\}\\lambda\(t,x,a\)<\\infty,\\qquad x\\in\\mathcal\{X\}\.whereλ⁡\(t,x,a\)≜−λ⁡\(x∣t,x,a\)≥0\\lambda\(t,x,a\)\\triangleq\-\\lambda\(x\\mid t,x,a\)\\geq 0for all\(t,x,a\)∈𝕄\(t,x,a\)\\in\\mathbb\{M\}\.
- \(v\)r⁡\(t,x,a\)r\(t,x,a\)is a Borel measurable function defined on𝕄\\mathbb\{M\}, which represents the running reward rate at timettwhen the system is in statexxunder actionaa;
- \(vi\)ρ⁡\(t,x,a,y\)\\rho\(t,x,a,y\)is a Borel measurable function defined on𝕄\\mathbb\{M\}, which represents the jump reward received upon a jump from statexxto stateyyat timettunder actionaa;
- \(vii\)h⁡\(x\)h\(x\)is a real\-value function on𝒳\\mathcal\{X\}, which denotes the terminal reward incurred at the end of the planning horizonTT;

A deterministic admissible control𝒖\\boldsymbol\{u\}refers to a measurable function𝒖:\[0,T\]×𝒳↦𝒜\\boldsymbol\{u\}:\[0,T\]\\times\\mathcal\{X\}\\mapsto\\mathcal\{A\}, and we denote the set of all admissible controls by𝒰\\mathcal\{U\}\. For any given initial time\-state pair\(t,x\)∈\[0,T\]×𝒳\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\}, letX𝒖=\{Xs𝒖:s∈\[t,T\]\}X^\{\\boldsymbol\{u\}\}=\\\{X^\{\\boldsymbol\{u\}\}\_\{s\}:s\\in\[t,T\]\\\}withXt𝒖=xX\_\{t\}^\{\\boldsymbol\{u\}\}=xbe the controlled state process under an admissible control𝒖∈𝒰\\boldsymbol\{u\}\\in\\mathcal\{U\}\. Specifically, the state processX𝒖X^\{\\boldsymbol\{u\}\}evolves as follows\. Suppose the system is in stateXs𝒖=yX\_\{s\}^\{\\boldsymbol\{u\}\}=yat timess\. While the system remains in stateyy, rewards accrue continuously at rater⁡\(s,y,𝒖⁡\(s,y\)\)r\(s,y,\\boldsymbol\{u\}\(s,y\)\)\. Letτ\\taudenote the remaining sojourn time in stateyy\. GivenXs𝒖=yX\_\{s\}^\{\\boldsymbol\{u\}\}=y, its survival function is

ℙ\(τ\>v∣Xs𝒖=y\)=e−∫0vλ\(s\+s′,y,𝒖\(s\+s′,y\)\)ds′,v≥0\.\\displaystyle\\mathbb\{P\}\(\\tau\>v\\mid X\_\{s\}^\{\\boldsymbol\{u\}\}=y\)=e^\{\-\\int\_\{0\}^\{v\}\\lambda\(s\+s^\{\\prime\},y,\\boldsymbol\{u\}\(s\+s^\{\\prime\},y\)\)\\,\\mathrm\{d\}s^\{\\prime\}\},\\qquad v\\geq 0\.At times\+τs\+\\tau, the system jumps to a statez≠yz\\neq ywith probabilityλ⁡\(z∣s\+τ,y,𝒖⁡\(s\+τ,y\)\)λ⁡\(s\+τ,y,𝒖⁡\(s\+τ,y\)\)\\frac\{\\lambda\(z\\mid s\+\\tau,y,\\boldsymbol\{u\}\(s\+\\tau,y\)\)\}\{\\lambda\(s\+\\tau,y,\\boldsymbol\{u\}\(s\+\\tau,y\)\)\}, and receives the jump rewardρ⁡\(s\+τ,y,𝒖⁡\(s\+τ,y\),z\)\\rho\(s\+\\tau,y,\\boldsymbol\{u\}\(s\+\\tau,y\),z\)\.

Given an initial time\-state pair\(t,x\)∈\[0,T\]×𝒳\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\}and𝒖∈𝒰\\boldsymbol\{u\}\\in\\mathcal\{U\}, for anyy,z∈𝒳y,z\\in\\mathcal\{X\}withy≠zy\\neq z, letNy​z𝒖​\(s\)N\_\{yz\}^\{\\boldsymbol\{u\}\}\(s\),s∈\[t,T\]s\\in\[t,T\], be the counting process that records the number of jumps ofX𝒖X^\{\\boldsymbol\{u\}\}fromyytozzover\(t,s\]\(t,s\]\. The value function is given by

V⁡\(t,x,𝒖\)=𝔼⁡\[∫tTr⁡\(s,Xs−𝒖,𝒖⁡\(s,Xs−𝒖\)\)​𝑑s\+∑y∈𝒳∑z≠y∫\(t,T\]ρ⁡\(s,y,𝒖⁡\(s,y\),z\)​d​Ny​z𝒖​\(s\)\+h⁡\(XT𝒖\)∣Xt𝒖=x\]\.\\displaystyle V\(t,x;\\boldsymbol\{u\}\)=\\mathbb\{E\}\\bigg\[\\int\_\{t\}^\{T\}r\\left\(s,X\_\{s\-\}^\{\\boldsymbol\{u\}\},\\boldsymbol\{u\}\(s,X\_\{s\-\}^\{\\boldsymbol\{u\}\}\)\\right\)\\mathrm\{d\}s\+\\sum\_\{y\\in\\mathcal\{X\}\}\\sum\_\{z\\neq y\}\\int\_\{\(t,T\]\}\\rho\\bigl\(s,y,\\boldsymbol\{u\}\(s,y\),z\\bigr\)\\,\\mathrm\{d\}N\_\{yz\}^\{\\boldsymbol\{u\}\}\(s\)\+h\(X\_\{T\}^\{\\boldsymbol\{u\}\}\)\\mid X\_\{t\}^\{\\boldsymbol\{u\}\}=x\\bigg\]\.\(1\)The objective of the control problem is to find an optimal admissible control𝒖∗∈𝒰\\boldsymbol\{u\}^\{\*\}\\in\\mathcal\{U\}that maximizes the value function

V∗​\(t,x\)=sup𝒖∈𝒰V⁡\(t,x,𝒖\),∀\(t,x\)∈\[0,T\]×𝒳\.\\displaystyle V^\{\*\}\(t,x\)=\\sup\_\{\\boldsymbol\{u\}\\in\\mathcal\{U\}\}V\(t,x;\\boldsymbol\{u\}\),\\quad\\forall\\,\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\}\.\(2\)
We make the following assumptions regarding the transition rates, the running reward rates, the jump rewards, and the terminal reward throughout this paper\.

###### Assumption 1\.

Suppose thatsupx∈𝒳λ∗​\(x\)<∞\\sup\_\{x\\in\\mathcal\{X\}\}\\lambda^\{\*\}\(x\)<\\infty\. Moreover, there exists a functionw:𝒳↦\[1,∞\)w:\\mathcal\{X\}\\mapsto\[1,\\infty\)and constantsc\>0c\>0,b≥0b\\geq 0, andM1\>0M\_\{1\}\>0, such that the following conditions hold for all\(t,x,a\)∈𝕄\(t,x,a\)\\in\\mathbb\{M\}:

1. \(i\)∑y∈𝒳w⁡\(y\)​λ​\(y∣t,x,a\)≤c​w​\(x\)\+b\\sum\_\{y\\in\\mathcal\{X\}\}w\(y\)\\lambda\(y\\mid t,x,a\)\\leq cw\(x\)\+b\.
2. \(ii\)\|r⁡\(t,x,a\)\|≤M1​w​\(x\)\|r\(t,x,a\)\|\\leq M\_\{1\}w\(x\)and\|h⁡\(x\)\|≤M1​w​\(x\)\.\|h\(x\)\|\\leq M\_\{1\}w\(x\)\.
3. \(iii\)∑y≠x\|ρ⁡\(t,x,a,y\)\|​λ​\(y∣t,x,a\)≤M1​w​\(x\)\.\\sum\_\{y\\neq x\}\|\\rho\(t,x,a,y\)\|\\lambda\(y\\mid t,x,a\)\\leq M\_\{1\}w\(x\)\.

Assumption[1](https://arxiv.org/html/2608.20680#Thmassumption1)allows the reward ratesr⁡\(t,x,a\)r\(t,x,a\)and the terminal rewardh⁡\(x\)h\(x\)to potentially grow withxx\. Here, we focus on CTJMDPs with bounded transition rates \(i\.e\.supx∈𝒳λ∗​\(x\)<∞\\sup\_\{x\\in\\mathcal\{X\}\}\\lambda^\{\*\}\(x\)<\\infty\) to simplify the theoretical analysis\. Our results can be readily extended to CTJMDPs with unbounded transition rates by imposing conditions such as Assumptions 3\.1 and 3\.2 of[15](https://arxiv.org/html/2608.20680#bib.bib7)\.

Indeed, it is useful to relate our formulation to the classical finite\-horizon CTJMDP model of[15](https://arxiv.org/html/2608.20680#bib.bib7), which incorporates a running reward rate and a terminal reward but no jump rewards\. Define the augmented running reward rate

R⁡\(t,x,a\)≜r⁡\(t,x,a\)\+∑y≠xρ⁡\(t,x,a,y\)​λ​\(y∣t,x,a\)\.\\displaystyle R\(t,x,a\)\\triangleq r\(t,x,a\)\+\\sum\_\{y\\neq x\}\\rho\(t,x,a,y\)\\lambda\(y\\mid t,x,a\)\.\(3\)By Assumption[1](https://arxiv.org/html/2608.20680#Thmassumption1), one can readily verify that the compensated counting processes\{Ny​z𝒖\(s\)−∫ts𝟏\{Xr−𝒖=y\}λ\(z∣r,y,𝒖\(r,y\)\)dr:s∈\[t,T\]\}\\\{N\_\{yz\}^\{\\boldsymbol\{u\}\}\(s\)\-\\int\_\{t\}^\{s\}\\mathbf\{1\}\_\{\\\{X\_\{r\-\}^\{\\boldsymbol\{u\}\}=y\\\}\}\\lambda\\\!\\left\(z\\mid r,y,\\boldsymbol\{u\}\(r,y\)\\right\)\\,\\mathrm\{d\}r:s\\in\[t,T\]\\\}is a martingale fory,z∈𝒳y,\\,z\\in\\mathcal\{X\}withy≠zy\\neq z\. It follows that the value function \([1](https://arxiv.org/html/2608.20680#S2.E1)\) can be equivalently written as

V⁡\(t,x,𝒖\)=𝔼⁡\[∫tTR⁡\(s,Xs−𝒖,𝒖⁡\(s,Xs−𝒖\)\)​𝑑s\+h⁡\(XT𝒖\)∣Xt𝒖=x\]\.\\displaystyle V\(t,x;\\boldsymbol\{u\}\)=\\mathbb\{E\}\\bigg\[\\int\_\{t\}^\{T\}R\\bigl\(s,X\_\{s\-\}^\{\\boldsymbol\{u\}\},\\boldsymbol\{u\}\(s,X\_\{s\-\}^\{\\boldsymbol\{u\}\}\)\\bigr\)\\,\\mathrm\{d\}s\+h\(X\_\{T\}^\{\\boldsymbol\{u\}\}\)\\mid X\_\{t\}^\{\\boldsymbol\{u\}\}=x\\bigg\]\.where\|R⁡\(t,x,a\)\|≤2​M1​w​\(x\)\|R\(t,x,a\)\|\\leq 2M\_\{1\}w\(x\)\.

Under Assumption[1](https://arxiv.org/html/2608.20680#Thmassumption1), the optimal value functionV∗​\(t,x\)V^\{\*\}\(t,x\)defined in \([2](https://arxiv.org/html/2608.20680#S2.E2)\) satisfies the following Hamilton\-Jacobi\-Bellman \(HJB\) equation \(see Proposition 4\.1 in[15](https://arxiv.org/html/2608.20680#bib.bib7)\):

∂V∗∂t​\(t,x\)\+supa∈𝒜H⁡\(t,x,a,V∗​\(⋅,⋅\)\)\\displaystyle\\frac\{\\partial V^\{\*\}\}\{\\partial t\}\(t,x\)\+\\sup\_\{a\\in\\mathcal\{A\}\}H\(t,x,a,V^\{\*\}\(\\cdot,\\cdot\)\)=0,\(t,x\)∈\[0,T\)×𝒳,\\displaystyle=0,\\quad\(t,x\)\\in\[0,T\)\\times\\mathcal\{X\},V∗​\(T,x\)\\displaystyle V^\{\*\}\(T,x\)=h⁡\(x\),x∈𝒳,\\displaystyle=h\(x\),\\quad x\\in\\mathcal\{X\},where the associated HamiltonianH:𝕄×Cw1,0​\(\[0,T\]×𝒳\)↦ℝH:\\mathbb\{M\}\\times C^\{1,0\}\_\{w\}\(\[0,T\]\\times\\mathcal\{X\}\)\\mapsto\\mathbb\{R\}is defined as

H⁡\(t,x,a,v⁡\(⋅,⋅\)\)=R⁡\(t,x,a\)\+∑y∈𝒳v⁡\(t,y\)​λ​\(y∣t,x,a\),\(t,x,a\)∈𝕄,v∈Cw1,0​\(\[0,T\]×𝒳\)\.\\displaystyle H\(t,x,a,v\(\\cdot,\\cdot\)\)=R\(t,x,a\)\+\\sum\_\{y\\in\\mathcal\{X\}\}v\(t,y\)\\lambda\(y\\mid t,x,a\),\\quad\(t,x,a\)\\in\\mathbb\{M\},\\,v\\in C^\{1,0\}\_\{w\}\(\[0,T\]\\times\\mathcal\{X\}\)\.\(4\)Moreover, the optimal value is actually attained by an admissible control𝒖∗∈𝒰\\boldsymbol\{u\}^\{\*\}\\in\\mathcal\{U\}, which is given as

𝒖∗​\(t,x\)=arg​maxa∈𝒜⁡H​\(t,x,a,V∗​\(t,x\)\),for​\(t,x\)∈\[0,T\]×𝒳\.\\displaystyle\\boldsymbol\{u\}^\{\*\}\(t,x\)=\\argmax\_\{a\\in\\mathcal\{A\}\}H\(t,x,a,V^\{\*\}\(t,x\)\),\\quad\\text\{ for \}\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\}\.

## 3Reinforcement Learning Formulation for CTJMDPs

In this section, we discuss the formulation of continuous\-time RL for CTJMDPs and consider randomized policies \(for explorations\) as in standard discrete\-time RL\. To proceed, we introduce the definition of randomized Markov policies as follows\. Let the action space𝒜\\mathcal\{A\}be equipped with its Borelσ\\sigma\-algebraℬ⁡\(𝒜\)\\mathcal\{B\}\(\\mathcal\{A\}\)\.

###### Definition 1\.

A randomized Markov policy is a stochastic kernel𝛑⁡\(d​a∣t,x\)\\boldsymbol\{\\pi\}\(\\mathrm\{d\}a\\mid t,x\)on𝒜\\mathcal\{A\}given\[0,T\]×𝒳\[0,T\]\\times\\mathcal\{X\}; that is,

- \(i\)for each\(t,x\)∈\[0,T\]×𝒳\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\},𝝅\(⋅∣t,x\)\\boldsymbol\{\\pi\}\(\\cdot\\mid t,x\)is a probability measure on\(𝒜,ℬ⁡\(𝒜\)\)\(\\mathcal\{A\},\\,\\mathcal\{B\}\(\\mathcal\{A\}\)\);
- \(ii\)for eachx∈𝒳x\\in\\mathcal\{X\}andS∈ℬ⁡\(𝒜\)S\\in\\mathcal\{B\}\(\\mathcal\{A\}\), the mappingt↦𝝅⁡\(S∣t,x\)t\\mapsto\\boldsymbol\{\\pi\}\(S\\mid t,x\)is measurable on\[0,T\]\[0,T\]\.

Moreover, a randomized Markov policy is called deterministic if there exists a measurable decision rule𝐮:\[0,T\]×𝒳→𝒜\\boldsymbol\{u\}:\[0,T\]\\times\\mathcal\{X\}\\to\\mathcal\{A\}such that𝛑⁡\(d​a∣t,x\)=δ𝐮⁡\(t,x\)​\(d​a\)\\boldsymbol\{\\pi\}\(\\mathrm\{d\}a\\mid t,x\)=\\delta\_\{\\boldsymbol\{u\}\(t,x\)\}\(\\mathrm\{d\}a\)for all\(t,x\)∈\[0,T\]×𝒳\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\}\.

Next, we investigate the dynamics of the CTJMDP process under a randomized Markov policy𝝅\\boldsymbol\{\\pi\}\. It should be noted that the CTJMDP model, serving as a continuous\-time analogue of the classical discrete\-time MDP, enables actions to be altered at any time\. However, the construction of the CTJMDP under a randomized Markov policy may encounter measurability issues\. Suppose we want to select actions independently at each timet∈\[0,T\]t\\in\[0,T\]according to the non\-degenerate distribution𝝅\(⋅∣t,xt−\)\\boldsymbol\{\\pi\}\(\\cdot\\mid t,x\_\{t\-\}\); then, the action processata\_\{t\}may fail to be measurable, one can refer to Example 1\.2\.5 in[21](https://arxiv.org/html/2608.20680#bib.bib12)for a concrete illustration\. Since the reward rate and transition rate are defined for the randomized action process, the absence of measurability makes it impossible to properly define the associated integrals, which in turn hinders the construction of the corresponding CTJMDP\.

To avoid these obstacles, we follow the definition of CTJMDP under randomized Markov policies as presented in Section 7 of[9](https://arxiv.org/html/2608.20680#bib.bib11)\. Specifically, for any randomized Markov policy𝝅\\boldsymbol\{\\pi\}, we consider a Markov processX~𝝅=\{X~s𝝅:s∈\[t,T\]\}\\tilde\{X\}^\{\\boldsymbol\{\\pi\}\}=\\\{\\tilde\{X\}\_\{s\}^\{\\boldsymbol\{\\pi\}\}:s\\in\[t,T\]\\\}starting fromX~t𝝅=x\\tilde\{X\}\_\{t\}^\{\\boldsymbol\{\\pi\}\}=x, with transition rate

λ𝝅​\(y∣t,x\)≜∫𝒜λ⁡\(y∣t,x,a\)​𝝅​\(𝑑a∣t,x\),for​\(t,x\)∈\[0,T\]×𝒳,y∈𝒳,\\displaystyle\\lambda^\{\\boldsymbol\{\\pi\}\}\(y\\mid t,x\)\\triangleq\\int\_\{\\mathcal\{A\}\}\\lambda\(y\\mid t,x,a\)\\boldsymbol\{\\pi\}\(\\mathrm\{d\}a\\mid t,x\),\\quad\\text\{ for \}\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\},\\,y\\in\\mathcal\{X\},\(5\)andλ𝝅​\(t,x\)≜−λ𝝅​\(x∣t,x\)\\lambda^\{\\boldsymbol\{\\pi\}\}\(t,x\)\\triangleq\-\\lambda^\{\\boldsymbol\{\\pi\}\}\(x\\mid t,x\)for\(t,x\)∈\[0,T\]×𝒳\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\}\. We refer toX~𝝅\\tilde\{X\}^\{\\boldsymbol\{\\pi\}\}as the exploratory state process, which will be used for theoretical analysis only as it is not observable in practice\. The reward rate function associated withX~𝝅\\tilde\{X\}^\{\\boldsymbol\{\\pi\}\}is defined as

R𝝅​\(t,x\)≜∫𝒜R⁡\(t,x,a\)​𝝅​\(𝑑a∣t,x\),for​\(t,x\)∈\[0,T\]×𝒳,\\displaystyle R^\{\\boldsymbol\{\\pi\}\}\(t,x\)\\triangleq\\int\_\{\\mathcal\{A\}\}R\(t,x,a\)\\boldsymbol\{\\pi\}\(\\mathrm\{d\}a\\mid t,x\),\\quad\\text\{ for \}\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\},whereR⁡\(t,x,a\)R\(t,x,a\)is defined in \([3](https://arxiv.org/html/2608.20680#S2.E3)\)\. Suppose the exploratory state process is in stateX~s𝝅=y\\tilde\{X\}\_\{s\}^\{\\boldsymbol\{\\pi\}\}=yat timess\. While it remains in stateyy, running rewards accrue continuously at rateR𝝅​\(s,y\)R^\{\\boldsymbol\{\\pi\}\}\(s,y\)\. Letτ\\taudenote the remaining sojourn time in stateyy\. Its conditional survival function isℙ\(τ\>v∣X~s𝝅=y\)=e−∫0vλ𝝅\(s\+s′,y\)ds′\\mathbb\{P\}\(\\tau\>v\\mid\\tilde\{X\}\_\{s\}^\{\\boldsymbol\{\\pi\}\}=y\)=e^\{\-\\int\_\{0\}^\{v\}\\lambda^\{\\boldsymbol\{\\pi\}\}\(s\+s^\{\\prime\},y\)\\,\\mathrm\{d\}s^\{\\prime\}\}forv≥0v\\geq 0\. Note that the above dynamics does not involve the particular randomized actions, it depends only on the policy measure itself, and can be viewed as the average of the dynamics over the randomization of actions\. Then, the value function of policy𝝅\\boldsymbol\{\\pi\}is defined in terms of the exploratory state processX~𝝅\\tilde\{X\}^\{\\boldsymbol\{\\pi\}\}\. Specifically, for each\(t,x\)∈\[0,T\]×𝒳\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\},

V⁡\(t,x,𝝅\)≜𝔼⁡\[∫tTR𝝅​\(s,X~s−𝝅\)​𝑑s\+h⁡\(X~T𝝅\)∣X~t𝝅=x\]\.\\displaystyle V\(t,x;\\boldsymbol\{\\pi\}\)\\triangleq\\mathbb\{E\}\\bigg\[\\int\_\{t\}^\{T\}R^\{\\boldsymbol\{\\pi\}\}\(s,\\tilde\{X\}\_\{s\-\}^\{\\boldsymbol\{\\pi\}\}\)\\mathrm\{d\}s\+h\(\\tilde\{X\}\_\{T\}^\{\\boldsymbol\{\\pi\}\}\)\\mid\\tilde\{X\}\_\{t\}^\{\\boldsymbol\{\\pi\}\}=x\\bigg\]\.\(6\)
However, merely allowing randomized Markov policies does not ensure exploration needed for RL, because under mild conditions the finite\-horizon CTJMDP admits an optimal deterministic policy even when optimization is over all randomized Markov policies\([15](https://arxiv.org/html/2608.20680#bib.bib7), Theorem 4\.1\)\. Following entropy\-regularized formulations in discrete\-time RL\([17](https://arxiv.org/html/2608.20680#bib.bib1)\)and continuous\-time RL for diffusions\([20](https://arxiv.org/html/2608.20680#bib.bib13)\), we introduce an entropy bonus to encourage exploration\. To provide a unified definition of entropy for discrete and continuous action spaces inℝn\\mathbb\{R\}^\{n\}, fix a nonzeroσ\\sigma\-finite reference measureμ\\muon\(𝒜,ℬ⁡\(𝒜\)\)\(\\mathcal\{A\},\\mathcal\{B\}\(\\mathcal\{A\}\)\)\. For example, consider a pricing problem withnnproducts\. If the admissible price vectors form a finite menu𝒜⊂ℝ\+n\\mathcal\{A\}\\subset\\mathbb\{R\}\_\{\+\}^\{n\}, thenμ\\mumay be taken as the counting measure\. If each product can be priced at any nonnegative level, then𝒜=ℝ\+n\\mathcal\{A\}=\\mathbb\{R\}\_\{\+\}^\{n\}andμ\\mumay be taken as thenn\-dimensional Lebesgue measure restricted toℝ\+n\\mathbb\{R\}\_\{\+\}^\{n\}\.

For a randomized Markov policy𝝅\\boldsymbol\{\\pi\}that is absolutely continuous with respect toμ\\mu, written𝝅\(⋅∣t,x\)≪μ\\boldsymbol\{\\pi\}\(\\cdot\\mid t,x\)\\ll\\mu, we use𝝅⁡\(a∣t,x\)\\boldsymbol\{\\pi\}\(a\\mid t,x\)to denote its Radon\-Nikodym density:

𝝅⁡\(d​a∣t,x\)=𝝅⁡\(a∣t,x\)​μ​\(d​a\)\.\\boldsymbol\{\\pi\}\(\\mathrm\{d\}a\\mid t,x\)=\\boldsymbol\{\\pi\}\(a\\mid t,x\)\\mu\(\\mathrm\{d\}a\)\.The entropy of𝝅\(⋅∣t,x\)\\boldsymbol\{\\pi\}\(\\cdot\\mid t,x\)relative toμ\\muis defined by

ℋμ\(𝝅\(⋅∣t,x\)\)≜−∫𝒜log𝝅\(a∣t,x\)𝝅\(da∣t,x\)\.\\mathcal\{H\}\_\{\\mu\}\\bigl\(\\boldsymbol\{\\pi\}\(\\cdot\\mid t,x\)\\bigr\)\\triangleq\-\\int\_\{\\mathcal\{A\}\}\\log\\boldsymbol\{\\pi\}\(a\\mid t,x\)\\,\\boldsymbol\{\\pi\}\(\\mathrm\{d\}a\\mid t,x\)\.
Forγ\>0\\gamma\>0, the entropy\-regularized value function for a CTJMDP associated with a randomized Markov policy𝝅\\boldsymbol\{\\pi\}is defined as

J⁡\(t,x,𝝅\)\\displaystyle J\(t,x;\\boldsymbol\{\\pi\}\)≜V\(t,x;𝝅\)−γ𝔼\[∫tT∫𝒜log𝝅\(a∣s,X~s−𝝅\)\]𝝅\(da∣s,X~s−𝝅\)ds∣X~t𝝅=x\]\\displaystyle\\triangleq V\(t,x;\\boldsymbol\{\\pi\}\)\-\\gamma\\,\\mathbb\{E\}\\bigg\[\\int\_\{t\}^\{T\}\\int\_\{\\mathcal\{A\}\}\\log\\boldsymbol\{\\pi\}\(a\\mid s,\\tilde\{X\}\_\{s\-\}^\{\\boldsymbol\{\\pi\}\}\)\]\\boldsymbol\{\\pi\}\(\\mathrm\{d\}a\\mid s,\\tilde\{X\}\_\{s\-\}^\{\\boldsymbol\{\\pi\}\}\)\\mathrm\{d\}s\\mid\\tilde\{X\}\_\{t\}^\{\\boldsymbol\{\\pi\}\}=x\\bigg\]\(7\)=𝔼⁡\[∫tT∫𝒜\[R⁡\(s,X~s−𝝅,a\)−γ​log⁡𝝅⁡\(a∣s,X~s−𝝅\)\]​𝝅​\(𝑑a∣s,X~s−𝝅\)​𝑑s\+h⁡\(X~T𝝅\)∣X~t𝝅=x\],\\displaystyle=\\mathbb\{E\}\\bigg\[\\int\_\{t\}^\{T\}\\int\_\{\\mathcal\{A\}\}\[R\(s,\\tilde\{X\}\_\{s\-\}^\{\\boldsymbol\{\\pi\}\},a\)\-\\gamma\\log\\boldsymbol\{\\pi\}\(a\\mid s,\\tilde\{X\}\_\{s\-\}^\{\\boldsymbol\{\\pi\}\}\)\]\\boldsymbol\{\\pi\}\(\\mathrm\{d\}a\\mid s,\\tilde\{X\}\_\{s\-\}^\{\\boldsymbol\{\\pi\}\}\)\\mathrm\{d\}s\+h\(\\tilde\{X\}\_\{T\}^\{\\boldsymbol\{\\pi\}\}\)\\mid\\tilde\{X\}\_\{t\}^\{\\boldsymbol\{\\pi\}\}=x\\bigg\],Here,γ\\gammais the temperature parameter controlling the exploration weight, typically chosen as a small tuned constant in RL\. Asγ→0\\gamma\\rightarrow 0, we haveJ⁡\(t,x,𝝅\)→V⁡\(t,x,𝝅\)J\(t,x;\\boldsymbol\{\\pi\}\)\\rightarrow V\(t,x;\\boldsymbol\{\\pi\}\)given in \([6](https://arxiv.org/html/2608.20680#S3.E6)\)\. The optimal \(entropy\-regularized\) value function is defined by

J∗​\(t,x\)=sup𝝅∈𝚷J⁡\(t,x,𝝅\),\(t,x\)∈\[0,T\]×𝒳,\\displaystyle J^\{\*\}\(t,x\)=\\sup\_\{\\boldsymbol\{\\pi\}\\in\\boldsymbol\{\\Pi\}\}J\(t,x;\\boldsymbol\{\\pi\}\),\\qquad\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\},\(8\)where𝚷\\boldsymbol\{\\Pi\}denotes the class of admissible randomized Markov policies specified below\.

###### Definition 2\.

A randomized Markov policy𝛑\\boldsymbol\{\\pi\}is called admissible if:

- \(i\)For each\(t,x\)∈\[0,T\]×𝒳\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\},𝝅\(⋅∣t,x\)∼μ\\boldsymbol\{\\pi\}\(\\cdot\\mid t,x\)\\sim\\mu; that is, 𝝅\(⋅∣t,x\)≪μandμ≪𝝅\(⋅∣t,x\)\.\\boldsymbol\{\\pi\}\(\\cdot\\mid t,x\)\\ll\\mu\\quad\\text\{ and \}\\quad\\mu\\ll\\boldsymbol\{\\pi\}\(\\cdot\\mid t,x\)\.
- \(ii\)There exists a constantM2\>0M\_\{2\}\>0, such that \|ℋμ\(𝝅\(⋅∣t,x\)\)\|≤M2w\(x\),\(t,x\)∈\[0,T\]×𝒳,\\displaystyle\|\\mathcal\{H\}\_\{\\mu\}\(\\boldsymbol\{\\pi\}\(\\cdot\\mid t,x\)\)\|\\leq M\_\{2\}w\(x\),\\qquad\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\},where the functionw⁡\(⋅\)w\(\\cdot\)is as specified in Assumption[1](https://arxiv.org/html/2608.20680#Thmassumption1)\.

Condition \(i\) serves two purposes\. The relation𝝅\(⋅∣t,x\)≪μ\\boldsymbol\{\\pi\}\(\\cdot\\mid t,x\)\\ll\\mumakes the entropy term well defined, whereasμ≪𝝅\(⋅∣t,x\)\\mu\\ll\\boldsymbol\{\\pi\}\(\\cdot\\mid t,x\)ensures action coverage: every measurable action set with positiveμ\\mu\-measure is assigned positive probability\. Condition \(ii\) controls the growth of the entropy term and ensures the finiteness of the regularized value function\.

The following proposition establishes the corresponding Hamilton\-Jacobi\-Bellman characterization of the optimal value functionJ∗J^\{\*\}\. It further provides an explicit characterization of the optimal stochastic policy, which takes a soft\-max form determined by the Hamiltonian \(see \([4](https://arxiv.org/html/2608.20680#S2.E4)\)\) associated withJ∗J^\{\*\}\.

###### Proposition 1\.

Suppose Assumption[1](https://arxiv.org/html/2608.20680#Thmassumption1)holds with functionww\. Then there exists a unique solutionφ∈Cw1,0​\(\[0,T\]×𝒳\)\\varphi\\in C^\{1,0\}\_\{w\}\(\[0,T\]\\times\\mathcal\{X\}\)satisfying the terminal conditionφ⁡\(T,x\)=h⁡\(x\)\\varphi\(T,x\)=h\(x\)for allx∈𝒳x\\in\\mathcal\{X\}, and for Lebesgue\-a\.e\.t∈\[0,T\)t\\in\[0,T\)andx∈𝒳x\\in\\mathcal\{X\},

∂φ∂t​\(t,x\)\+sup𝝅∈𝚷∫𝒜\{H⁡\(t,x,a,φ⁡\(⋅,⋅\)\)−γ​log⁡𝝅⁡\(a∣t,x\)\}​𝝅​\(𝑑a∣t,x\)=0\.\\displaystyle\\frac\{\\partial\\varphi\}\{\\partial t\}\(t,x\)\+\\sup\_\{\\boldsymbol\{\\pi\}\\in\\boldsymbol\{\\Pi\}\}\\int\_\{\\mathcal\{A\}\}\\\{H\(t,x,a,\\varphi\(\\cdot,\\cdot\)\)\-\\gamma\\log\\boldsymbol\{\\pi\}\(a\\mid t,x\)\\\}\\boldsymbol\{\\pi\}\(\\mathrm\{d\}a\\mid t,x\)=0\.\(9\)Moreover, the unique solutionφ∈Cw1,0​\(\[0,T\)×𝒳\)\\varphi\\in C^\{1,0\}\_\{w\}\(\[0,T\)\\times\\mathcal\{X\}\)is the optimal value function as defined in \([8](https://arxiv.org/html/2608.20680#S3.E8)\), i\.e\.,φ⁡\(t,x\)=J∗​\(t,x\)\\varphi\(t,x\)=J^\{\*\}\(t,x\)for all\(t,x\)∈\[0,T\]×𝒳\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\}\. If the policy𝛑∗\\boldsymbol\{\\pi\}^\{\*\}defined by

𝝅∗​\(a∣t,x\)=exp⁡\{1γ​H​\(t,x,a,J∗​\(⋅,⋅\)\)\}∫𝒜exp⁡\{1γ​H​\(t,x,a′,J∗​\(⋅,⋅\)\)\}​μ​\(d​a′\),\(t,x,a\)∈𝕄\.\\displaystyle\\boldsymbol\{\\pi\}^\{\*\}\(a\\mid t,x\)=\\frac\{\\exp\\\{\\frac\{1\}\{\\gamma\}H\(t,x,a,J^\{\*\}\(\\cdot,\\cdot\)\)\\\}\}\{\\int\_\{\\mathcal\{A\}\}\\exp\\\{\\frac\{1\}\{\\gamma\}H\(t,x,a^\{\\prime\},J^\{\*\}\(\\cdot,\\cdot\)\)\\\}\\mu\(\\mathrm\{d\}a^\{\\prime\}\)\},\\qquad\(t,x,a\)\\in\\mathbb\{M\}\.\(10\)is admissible, then it is the optimal policy for \([8](https://arxiv.org/html/2608.20680#S3.E8)\)\.

## 4qq\-Learning Theory and Algorithm for CTJMDPs

The Hamilton\-Jacobi\-Bellman characterization in Proposition[1](https://arxiv.org/html/2608.20680#Thmproposition1)is model\-based, as evaluating the Hamiltonian \([4](https://arxiv.org/html/2608.20680#S2.E4)\) requires knowledge of the reward and transition rates\. This section develops a continuous\-timeqq\-learning framework that directly learns the optimal value functionJ∗J^\{\*\}and the so\-called optimalqq\-function from observed trajectories of CTJMDPs, without estimating model parameters, and recovers the optimal policy from the learnedqq\-function\.

### 4\.1The Optimalqq\-Function

The continuous\-timeqq\-learning framework was introduced by[20](https://arxiv.org/html/2608.20680#bib.bib13)for entropy\-regularized diffusion control problems\. They observe that the conventionalQQ\-function collapses to the value function as the duration of the initial action vanishes and introduce theqq\-function \(as opposed toQQ\-function\) to capture the first\-order, action\-dependent term\. The continuous\-timeqq\-function can be interpreted as the advantage rate function, an analogue of the advantage function commonly used in discrete\-time RL\. We adapt the construction ofqq\-function to finite\-horizon CTJMDPs and focus directly on its optimal counterpart\.

Define the optimalqq\-function for the entropy\-regularized CTJMDP as

q∗​\(t,x,a\)≜∂J∗∂t​\(t,x\)\+H⁡\(t,x,a,J∗​\(⋅,⋅\)\),\(t,x,a\)∈𝕄\.\\displaystyle q^\{\*\}\(t,x,a\)\\triangleq\\frac\{\\partial J^\{\*\}\}\{\\partial t\}\(t,x\)\+H\\bigl\(t,x,a,J^\{\*\}\(\\cdot,\\cdot\)\\bigr\),\\qquad\\quad\(t,x,a\)\\in\\mathbb\{M\}\.From Proposition[1](https://arxiv.org/html/2608.20680#Thmproposition1), one can show that the optimalqq\-function satisfies \(see the proof of Proposition 8 in[20](https://arxiv.org/html/2608.20680#bib.bib13)for an analogous argument\):

∫𝒜exp⁡\{1γ​q∗​\(t,x,a\)\}​μ​\(𝑑a\)=1\.\\displaystyle\\int\_\{\\mathcal\{A\}\}\\exp\\\{\\frac\{1\}\{\\gamma\}q^\{\*\}\(t,x,a\)\\\}\\mu\(\\mathrm\{d\}a\)=1\.Thus,exp\{1γq∗\(t,x,⋅\)\}\\exp\\\{\\frac\{1\}\{\\gamma\}q^\{\*\}\(t,x,\\cdot\)\\\}defines a probability density with respect toμ\\mu\. By the optimal\-policy characterization in Proposition[1](https://arxiv.org/html/2608.20680#Thmproposition1), this density is precisely

𝝅∗​\(a∣t,x\)=exp⁡\{1γ​q∗​\(t,x,a\)\}\.\\displaystyle\\boldsymbol\{\\pi\}^\{\*\}\(a\\mid t,x\)=\\exp\\\{\\frac\{1\}\{\\gamma\}q^\{\*\}\(t,x,a\)\\\}\.Therefore, learning the optimalqq\-function directly recovers the optimal policy𝝅\\boldsymbol\{\\pi\}\. This suggests usingq∗q^\{\*\}, together withJ∗J^\{\*\}, as our learning targets, similar as learning the conventional policy\-value pair in discrete\-time actor–critic RL methods\. Note that\(J∗,q∗\)\(J^\{\*\},q^\{\*\}\)are continuous\-time\-based targets and they are independent of time discretizations\.

### 4\.2Martingale Characterization and Main Theoretical Results

We now develop a martingale characterization that jointly identifies\(J∗,q∗\)\(J^\{\*\},q^\{\*\}\)from observed trajectories\. This characterization provides the theoretical foundation for our learning algorithm\. To formulate this characterization, we first specify how randomized policies are implemented to generate trajectories\. As discussed in Section[3](https://arxiv.org/html/2608.20680#S3), although a randomized Markov policy specifies an action distribution at every time\-state pair, independently sampling an action at each instant need not yield a measurable action process and is not operationally implementable\. We therefore implement an admissible randomized policy on a decision grid: actions are sampled only at the grid points and held fixed between consecutive points, while the state process continues to evolve in continuous time\.

Formally, given an initial time\-state pair\(t,x\)∈\[0,T\]×𝒳\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\}, consider a time grid on\[t,T\]\[t,T\]:

𝕊≜\{t=t0<t1<⋯<tK=T\}\.\\mathbb\{S\}\\triangleq\\\{t=t\_\{0\}<t\_\{1\}<\\cdots<t\_\{K\}=T\\\}\.Let\{Uk\}k=0K−1\\\{U\_\{k\}\\\}\_\{k=0\}^\{K\-1\}be independent uniform random variables on\[0,1\]n\[0,1\]^\{n\}used for action randomization \(nnis the dimension of the action space\)\. Given a randomized policy𝝅,\\boldsymbol\{\\pi\},there always exists some measurable functionΦ:\[0,T\]×𝒳×\[0,1\]n↦𝒜\\Phi:\[0,T\]\\times\\mathcal\{X\}\\times\[0,1\]^\{n\}\\mapsto\\mathcal\{A\}such thatΦ\(t,x,U\)∼𝝅\(⋅∣t,x\)\\Phi\(t,x,U\)\\sim\\boldsymbol\{\\pi\}\(\\cdot\\mid t,x\)for all\(t,x\)∈\[0,T\]×𝒳\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\}andU∼U⁡\(\[0,1\]n\)U\\sim U\(\[0,1\]^\{n\}\)\. We next implement𝝅\\boldsymbol\{\\pi\}on the time grid𝕊\\mathbb\{S\}\. LetX𝝅,𝕊=\{Xs𝝅,𝕊:s∈\[t,T\]\}X^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}=\\\{X\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}:s\\in\[t,T\]\\\}denote the resulting \(observable\) state process withXt𝝅,𝕊=xX\_\{t\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}=x, which we refer to as the grid sample state process\. For eachk=0,…,K−1k=0,\\ldots,K\-1, letAk≜Φ⁡\(tk,Xtk𝝅,𝕊,Uk\)A\_\{k\}\\triangleq\\Phi\\bigl\(t\_\{k\},X\_\{t\_\{k\}\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},U\_\{k\}\\bigr\)denote the action sampled at timetkt\_\{k\}\. The action processa𝝅,𝕊=\{as𝝅,𝕊:s∈\[t,T\]\}a^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}=\\\{a\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}:s\\in\[t,T\]\\\}is defined as

as𝝅,𝕊≜∑k=0K−1Ak​𝟏\(tk,tk\+1\]​\(s\),s∈\(t,T\]\.\\displaystyle a\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\\triangleq\\sum\_\{k=0\}^\{K\-1\}A\_\{k\}\\boldsymbol\{1\}\_\{\(t\_\{k\},t\_\{k\+1\}\]\}\(s\),\\qquad s\\in\(t,T\]\.Let\{ℱs𝕊\}s∈\[t,T\]\\\{\\mathcal\{F\}\_\{s\}^\{\\mathbb\{S\}\}\\\}\_\{s\\in\[t,T\]\}denote the filtration generated jointly by the state process and the action\-randomization variables, where

ℱs𝕊≜σ⁡\{Xu𝝅,𝕊:t≤u≤s\}∨σ⁡\{Uk:tk≤s\},s∈\[t,T\]\.\\mathcal\{F\}\_\{s\}^\{\\mathbb\{S\}\}\\triangleq\\sigma\\\{X\_\{u\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}:t\\leq u\\leq s\\\}\\vee\\sigma\\\{U\_\{k\}:t\_\{k\}\\leq s\\\},\\qquad s\\in\[t,T\]\.The \(observable\) state processX𝝅,𝕊X^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}and the action processa𝝅,𝕊a^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}are adapted to this filtration, anda𝝅,𝕊a^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}is predictable\. For later use, let\{τℓ\}ℓ≥1\\\{\\tau\_\{\\ell\}\\\}\_\{\\ell\\geq 1\}denote the successive jump times ofX𝝅,𝕊X^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}on\(t,T\]\(t,T\]\. For anyy,z∈𝒳y,\\,z\\in\\mathcal\{X\}, letNy​z𝝅,𝕊​\(s\)N\_\{yz\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\(s\),s∈\[t,T\]s\\in\[t,T\], denote the counting process that records the number of jumps ofX𝝅,𝕊X^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}fromyytozzover\(t,T\]\(t,T\]\. In what follows, expectations involvingX𝝅,𝕊X^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}ora𝝅,𝕊a^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}are taken over both the state\-transition randomness and the action randomization\. With a slight abuse of notation, we continue to denote such expectations by𝔼\\mathbb\{E\}\.

We are now ready to state the first main theoretical result of this paper\. The following Dynkin’s formula for the grid sample state processX𝝅,𝕊X^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}is crucial for the subsequent martingale characterization of\(J∗,q∗\)\(J^\{\*\},q^\{\*\}\)\. This result is non\-trivial: while the standard Dynkin’s formula applies to Markov processes, the grid sample state processX𝝅,𝕊X^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}is non\-Markovian on\[0,T\]\[0,T\]\. Its future evolution depends not only on the current state, but also on the action selected at the preceding grid point\.

###### Theorem 1\(Dynkin’s formula\)\.

Suppose Assumption[1](https://arxiv.org/html/2608.20680#Thmassumption1)holds with functionww\. Then, for any initial time\-state pair\(t,x\)∈\[0,T\]×𝒳\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\}and time grid𝕊\\mathbb\{S\}on\[t,T\]\[t,T\], the grid sample state process\{Xs𝛑,𝕊:s∈\[t,T\]\}\\\{X\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}:s\\in\[t,T\]\\\}satisfies: for eachφ∈Cw1,0​\(\[0,T\]×𝒳\)\\varphi\\in C^\{1,0\}\_\{w\}\(\[0,T\]\\times\\mathcal\{X\}\)ands∈\[t,T\]s\\in\[t,T\],

𝔼⁡\[∫sT\(∂φ∂u​\(u,Xu−𝝅,𝕊\)\+∑y∈𝒳φ⁡\(u,y\)​λ​\(y∣u,Xu−𝝅,𝕊,au𝝅,𝕊\)\)​𝑑u∣ℱs𝕊\]=𝔼⁡\[φ⁡\(T,XT𝝅,𝕊\)∣ℱs𝕊\]−φ⁡\(s,Xs𝝅,𝕊\)\.\\displaystyle\\mathbb\{E\}\\bigg\[\\int\_\{s\}^\{T\}\\bigg\(\\frac\{\\partial\\varphi\}\{\\partial u\}\(u,X\_\{u\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\+\\sum\_\{y\\in\\mathcal\{X\}\}\\varphi\(u,y\)\\lambda\(y\\mid u,X\_\{u\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},a\_\{u\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\\bigg\)\\mathrm\{d\}\{u\}\\mid\\mathcal\{F\}\_\{s\}^\{\\mathbb\{S\}\}\\bigg\]=\\mathbb\{E\}\[\\varphi\(T,X\_\{T\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\\mid\\mathcal\{F\}\_\{s\}^\{\\mathbb\{S\}\}\]\-\\varphi\(s,X\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\.

The proof of Theorem[1](https://arxiv.org/html/2608.20680#Thmtheorem1)differs from the semimartingale argument used by[12](https://arxiv.org/html/2608.20680#bib.bib17), which generalizes[20](https://arxiv.org/html/2608.20680#bib.bib13)to RL for jump\-diffusions\. In their jump\-diffusion setting, the grid sample state process is governed by an SDE driven by Brownian motion and Poisson random measures, which directly provides its semimartingale representation and permits an application of the Itô’s formula for their theoretical analysis\. A CTJMDP, by contrast, is specified through its transition\-rate kernel on a general denumerable state space, which need not be a subset ofℝd\\mathbb\{R\}^\{d\}\. Consequently, the grid sample state process of a CTJMDP may not possess a semimartingale structure\. Theorem[1](https://arxiv.org/html/2608.20680#Thmtheorem1)establishes the Dynkin’s formula forX𝝅,𝕊X^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}directly from the transition rates\.

We next state the second main theoretical result of this paper\. The following theorem shows that, subject to the terminal and normalization conditions, the martingale property along trajectories of the grid sample state process uniquely identifies the pair of optimal value function and optimalqq\-function\. This result provides the theoretical foundation for our proposed RL algorithm\.

###### Theorem 2\.

Suppose Assumption[1](https://arxiv.org/html/2608.20680#Thmassumption1)holds with functionww\. LetJ∗^∈Cw1,0​\(\[0,T\]×𝒳\)\\widehat\{J^\{\*\}\}\\in C\_\{w\}^\{1,0\}\(\[0,T\]\\times\\mathcal\{X\}\)andq∗^:𝕄↦ℝ\\widehat\{q^\{\*\}\}:\\mathbb\{M\}\\mapsto\\mathbb\{R\}be measurable, satisfying

J∗^​\(T,x\)=h⁡\(x\),∫𝒜exp⁡\{1γ​q∗^​\(t,x,a\)\}​μ​\(𝑑a\)=1,∀\(t,x\)∈\[0,T\]×𝒳\.\\displaystyle\\widehat\{J^\{\*\}\}\(T,x\)=h\(x\),\\quad\\int\_\{\\mathcal\{A\}\}\\exp\\\{\\frac\{1\}\{\\gamma\}\\widehat\{q^\{\*\}\}\(t,x,a\)\\\}\\mu\(\\mathrm\{d\}a\)=1,\\quad\\forall\\,\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\}\.\(11\)Then,

- \(i\)IfJ∗^\\widehat\{J^\{\*\}\}andq∗^\\widehat\{q^\{\*\}\}are respectively the optimal value function and the optimalqq\-function, then given any𝝅∈𝚷\\boldsymbol\{\\pi\}\\in\\boldsymbol\{\\Pi\}, for all initial time\-state pair\(t,x\)∈\[0,T\]×𝒳\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\}and any time grid𝕊\\mathbb\{S\}on\[t,T\]\[t,T\], the following process J∗^​\(s,Xs𝝅,𝕊\)\+∫ts∑y∈𝒳∑z≠yρ⁡\(u,y,au𝝅ψ,𝕊,z\)​d​Ny​z𝝅,𝕊​\(u\)\+∫ts\[r⁡\(u,Xu−𝝅,𝕊,au𝝅,𝕊\)−q∗^​\(u,Xu−𝝅,𝕊,au𝝅,𝕊\)\]​𝑑u\\displaystyle\\widehat\{J^\{\*\}\}\(s,X\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\+\\int\_\{t\}^\{s\}\\sum\_\{y\\in\\mathcal\{X\}\}\\sum\_\{z\\neq y\}\\rho\\bigl\(u,y,a\_\{u\}^\{\\boldsymbol\{\\pi\}^\{\\psi\},\\mathbb\{S\}\},z\\bigr\)\\,\\mathrm\{d\}N\_\{yz\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\(u\)\+\\int\_\{t\}^\{s\}\[r\(u,X\_\{u\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},a\_\{u\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\-\\widehat\{q^\{\*\}\}\(u,X\_\{u\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},a\_\{u\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\]\\mathrm\{d\}uis an\{ℱs𝕊\}s∈\[t,T\]\\\{\\mathcal\{F\}\_\{s\}^\{\\mathbb\{S\}\}\\\}\_\{s\\in\[t,T\]\}\-martingale\.
- \(ii\)If there exists one𝝅∈𝚷\\boldsymbol\{\\pi\}\\in\\boldsymbol\{\\Pi\}and a time grid𝕊=\{0=t0<t1<⋯<tK=T\}\\mathbb\{S\}=\\\{0=t\_\{0\}<t\_\{1\}<\\cdots<t\_\{K\}=T\\\}on\[0,T\]\[0,T\], such that for all initial statex∈𝒳x\\in\\mathcal\{X\}, the process J∗^​\(t,Xt𝝅,𝕊\)\+∫0t∑y∈𝒳∑z≠yρ⁡\(s,y,as𝝅ψ,𝕊,z\)​d​Ny​z𝝅,𝕊​\(s\)\+∫0t\[r⁡\(s,Xs−𝝅,𝕊,as𝝅,𝕊\)−q∗^​\(s,Xs−𝝅,𝕊,as𝝅,𝕊\)\]​𝑑s\\displaystyle\\widehat\{J^\{\*\}\}\(t,X\_\{t\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\+\\int\_\{0\}^\{t\}\\sum\_\{y\\in\\mathcal\{X\}\}\\sum\_\{z\\neq y\}\\rho\\bigl\(s,y,a\_\{s\}^\{\\boldsymbol\{\\pi\}^\{\\psi\},\\mathbb\{S\}\},z\\bigr\)\\,\\mathrm\{d\}N\_\{yz\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\(s\)\+\\int\_\{0\}^\{t\}\[r\(s,X\_\{s\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},a\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\-\\widehat\{q^\{\*\}\}\(s,X\_\{s\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},a\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\]\\mathrm\{d\}s\(12\)is an\{ℱt𝕊\}t∈\[0,T\]\\\{\\mathcal\{F\}\_\{t\}^\{\\mathbb\{S\}\}\\\}\_\{t\\in\[0,T\]\}\-martingale, thenJ∗^\\widehat\{J^\{\*\}\}andq∗^\\widehat\{q^\{\*\}\}are respectively the optimal value function and the optimalqq\-function\.

Note that the augmented running reward rateRR, introduced in \([3](https://arxiv.org/html/2608.20680#S2.E3)\) for theoretical analysis, cannot be evaluated from sample trajectories because its jump\-reward component depends on the unknown transition rates\. Accordingly, for algorithm design, the martingale condition in Theorem[2](https://arxiv.org/html/2608.20680#Thmtheorem2)is formulated in terms of the cumulative running and jump rewards, which are directly observable along sample trajectories\.

The two parts of Theorem[2](https://arxiv.org/html/2608.20680#Thmtheorem2)provide the necessity and sufficiency underlying our learning procedure\. Part \(i\) shows that\(J∗,q∗\)\(J^\{\*\},q^\{\*\}\)satisfies the martingale condition, which provides the foundation for designingqq\-learning algorithm as we will see later\. Part \(ii\) shows that a single policy and a fixed decision grid on the entire horizon\[0,T\]\[0,T\]suffices to uniquely identify\(J∗,q∗\)\(J^\{\*\},q^\{\*\}\), provided that the martingale condition holds for every initial state\.

Theorem[2](https://arxiv.org/html/2608.20680#Thmtheorem2)differs from existing results established in diffusion and jump\-diffusion settings\([20](https://arxiv.org/html/2608.20680#bib.bib13),[12](https://arxiv.org/html/2608.20680#bib.bib17)\)in two main aspects\. First, our result incorporates the jump rewards inherent to CTJMDPs \(such as the network dynamic pricing application considered later\), which are not considered in these prior studies\. Second, Part \(ii\) requires significantly less grid coverage than the corresponding results in\([20](https://arxiv.org/html/2608.20680#bib.bib13),[12](https://arxiv.org/html/2608.20680#bib.bib17)\)\. Their formulation requires the martingale condition to hold for every initial time\-state pair\(t,x\)\(t,x\)and every possible grid on\[t,T\]\[t,T\]\. In contrast, our condition is far less restrictive, requiring only one fixed grid on\[0,T\]\[0,T\]across all initial states\. This distinction stems directly from the path properties of the underlying processes\. The grid sample state process of a CTJMDP remains constant between jump events, whereas the sample state of a diffusion process evolves continuously\. In our framework, under the stability conditionλ∗​\(x\)<∞\\lambda^\{\*\}\(x\)<\\infty, the pure\-jump grid sample state process initialized atxxhas a strictly positive probability of remaining in statexxup to any arbitrary time in\[0,T\]\[0,T\]\. This property is the key mechanism that allows us to establish Part \(ii\) under relaxed grid conditions\.

### 4\.3qq\-learning Algorithm for CTJMDPs

We next develop theqq\-learning algorithm for the entropy\-regularized RL problem \([7](https://arxiv.org/html/2608.20680#S3.E7)\)–\([8](https://arxiv.org/html/2608.20680#S3.E8)\)\. The martingale characterization in Theorem[2](https://arxiv.org/html/2608.20680#Thmtheorem2)motivates the simultaneous approximation of the optimal value function andqq\-function\. We consider parametric families\{Jθ:θ∈Θ\}\\\{J^\{\\theta\}:\\theta\\in\\Theta\\\}and\{qψ:ψ∈Ψ\}\\\{q^\{\\psi\}:\\psi\\in\\Psi\\\}satisfying

Jθ​\(T,x\)\\displaystyle J^\{\\theta\}\(T,x\)=h⁡\(x\),∫𝒜exp⁡\{1γ​qψ​\(t,x,a\)\}​μ​\(𝑑a\)=1,\(t,x\)∈\[0,T\]×𝒳\.\\displaystyle=h\(x\),\\qquad\\int\_\{\\mathcal\{A\}\}\\exp\\\{\\frac\{1\}\{\\gamma\}q^\{\\psi\}\(t,x,a\)\\\}\\mu\(\\mathrm\{d\}a\)=1,\\qquad\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\}\.\(13\)The normalization condition in \([13](https://arxiv.org/html/2608.20680#S4.E13)\) ensures that𝝅ψ​\(a∣t,x\)≜exp⁡\{1γ​qψ​\(t,x,a\)\}\\boldsymbol\{\\pi\}^\{\\psi\}\(a\\mid t,x\)\\triangleq\\exp\\\{\\frac\{1\}\{\\gamma\}q^\{\\psi\}\(t,x,a\)\\\}defines a policy density with respect toμ\\mu\. If the state space is not a subset ofℝd\\mathbb\{R\}^\{d\}, we can mapx∈𝒳x\\in\\mathcal\{X\}tox′∈ℝdx^\{\\prime\}\\in\\mathbb\{R\}^\{d\}before passing it to the parametric functions \(e\.g\. neural nets\)JθJ^\{\\theta\}andqψq^\{\\psi\}\. For implementation, we fixΔ​t\>0\\Delta t\>0, letK≜⌈T/Δ​t⌉K\\triangleq\\lceil T/\\Delta t\\rceilandtk≜k​Δ​tt\_\{k\}\\triangleq k\\Delta tfork=0,…,K−1k=0,\\ldots,K\-1, and define𝕊≜\{0=t0<t1<⋯<tK=T\}\\mathbb\{S\}\\triangleq\\\{0=t\_\{0\}<t\_\{1\}<\\cdots<t\_\{K\}=T\\\}\. Note that one can also consider a non\-uniform time grid\.

To design the updating rules forθ\\thetaandψ\\psibased on Theorem[2](https://arxiv.org/html/2608.20680#Thmtheorem2), we utilize the martingale orthogonality condition, which states that a processMMis a \(square\-integrable\)\{ℱt\}t∈\[0,T\]\\\{\\mathcal\{F\}\_\{t\}\\\}\_\{t\\in\[0,T\]\}\-martingale if and only if𝔼⁡\[∫0THt​d​Mt\]=0\\mathbb\{E\}\[\\int\_\{0\}^\{T\}H\_\{t\}\\,\\mathrm\{d\}M\_\{t\}\]=0holds for any predictable processHH\(called a test function\)\. A popular approach is to choose

ξt=∇θJθ​\(t,Xt−𝝅ψ,𝕊\),ζt=∇ψqψ​\(t,Xt−𝝅ψ,𝕊,at𝝅ψ,𝕊\)\\displaystyle\\xi\_\{t\}=\\nabla\_\{\\theta\}J^\{\\theta\}\(t,X\_\{t\-\}^\{\\boldsymbol\{\\pi\}^\{\\psi\},\\mathbb\{S\}\}\),\\qquad\\zeta\_\{t\}=\\nabla\_\{\\psi\}q^\{\\psi\}\(t,X\_\{t\-\}^\{\\boldsymbol\{\\pi\}^\{\\psi\},\\mathbb\{S\}\},a\_\{t\}^\{\\boldsymbol\{\\pi\}^\{\\psi\},\\mathbb\{S\}\}\)\(14\)as two sets of test functions\. The martingale orthogonality conditions then reduce to

𝔼⁡\[∫0Tξt​d​Mtθ,ψ\]=0,𝔼⁡\[∫0Tζt​d​Mtθ,ψ\]=0,\\displaystyle\\mathbb\{E\}\\left\[\\int\_\{0\}^\{T\}\\xi\_\{t\}\\,\\mathrm\{d\}M\_\{t\}^\{\\theta,\\psi\}\\right\]=0,\\qquad\\mathbb\{E\}\\left\[\\int\_\{0\}^\{T\}\\zeta\_\{t\}\\,\\mathrm\{d\}M\_\{t\}^\{\\theta,\\psi\}\\right\]=0,\(15\)where

d​Mtθ,ψ≜d​Jθ​\(t,Xt𝝅ψ,𝕊\)\\displaystyle\\mathrm\{d\}M\_\{t\}^\{\\theta,\\psi\}\\triangleq\\mathrm\{d\}J^\{\\theta\}\(t,X\_\{t\}^\{\\boldsymbol\{\\pi\}^\{\\psi\},\\mathbb\{S\}\}\)\+∑y∈𝒳∑z≠yρ\(t,y,at𝝅ψ,𝕊,z\)dNy​z𝝅ψ,𝕊\(t\)\\displaystyle\+\\sum\_\{y\\in\\mathcal\{X\}\}\\sum\_\{z\\neq y\}\\rho\\bigl\(t,y,a\_\{t\}^\{\\boldsymbol\{\\pi\}^\{\\psi\},\\mathbb\{S\}\},z\\bigr\)\\,\\mathrm\{d\}N\_\{yz\}^\{\\boldsymbol\{\\pi\}^\{\\psi\},\\mathbb\{S\}\}\(t\)\+\[r⁡\(t,Xt−𝝅ψ,𝕊,at𝝅ψ,𝕊\)−qψ​\(t,Xt−𝝅ψ,𝕊,at𝝅ψ,𝕊\)\]​d​t\.\\displaystyle\+\[r\(t,X\_\{t\-\}^\{\\boldsymbol\{\\pi\}^\{\\psi\},\\mathbb\{S\}\},a\_\{t\}^\{\\boldsymbol\{\\pi\}^\{\\psi\},\\mathbb\{S\}\}\)\-q^\{\\psi\}\(t,X\_\{t\-\}^\{\\boldsymbol\{\\pi\}^\{\\psi\},\\mathbb\{S\}\},a\_\{t\}^\{\\boldsymbol\{\\pi\}^\{\\psi\},\\mathbb\{S\}\}\)\]\\mathrm\{d\}t\.In \([14](https://arxiv.org/html/2608.20680#S4.E14)\),ξt\\xi\_\{t\}is chosen so that the update ofθ\\thetacorresponds to TD\(0\) in discrete\-time RL, whileζt\\zeta\_\{t\}is chosen because the identity𝝅ψ​\(a∣t,x\)=exp⁡\{1γ​qψ​\(t,x,a\)\}\\boldsymbol\{\\pi\}^\{\\psi\}\(a\\mid t,x\)=\\exp\\\{\\frac\{1\}\{\\gamma\}q^\{\\psi\}\(t,x,a\)\\\}yieldsζt=γ​∇ψ​log⁡𝝅ψ​\(at𝝅ψ,𝕊∣t,Xt−𝝅ψ,𝕊\)\\zeta\_\{t\}=\\gamma\\nabla\_\{\\psi\}\\log\\boldsymbol\{\\pi\}^\{\\psi\}\(a\_\{t\}^\{\\boldsymbol\{\\pi\}^\{\\psi\},\\mathbb\{S\}\}\\mid t,X\_\{t\-\}^\{\\boldsymbol\{\\pi\}^\{\\psi\},\\mathbb\{S\}\}\), connecting the update ofψ\\psito policy\-gradient methods\. Note that the choices ofξt\\xi\_\{t\}andζt\\zeta\_\{t\}are not restricted to those specified in \([14](https://arxiv.org/html/2608.20680#S4.E14)\)\. Many alternative predictable test processes can be used, with different choices generally leading to different learning algorithms; See[20](https://arxiv.org/html/2608.20680#bib.bib13)for details\. We solve the system of equations in \([15](https://arxiv.org/html/2608.20680#S4.E15)\) by stochastic approximation and the update rules forθ\\thetaandψ\\psiare given by

θ←θ\+αθ​∫0Tξt​d​Mtθ,ψ,ψ←ψ\+αψ​∫0Tζt​d​Mtθ,ψ,\\displaystyle\\theta\\leftarrow\\theta\+\\alpha\_\{\\theta\}\\int\_\{0\}^\{T\}\\xi\_\{t\}\\,\\mathrm\{d\}M\_\{t\}^\{\\theta,\\psi\},\\qquad\\psi\\leftarrow\\psi\+\\alpha\_\{\\psi\}\\int\_\{0\}^\{T\}\\zeta\_\{t\}\\,\\mathrm\{d\}M\_\{t\}^\{\\theta,\\psi\},\(16\)whereαθ\\alpha\_\{\\theta\}andαψ\\alpha\_\{\\psi\}are the corresponding learning rates\.

We next discuss the computation of the integral in the update \([16](https://arxiv.org/html/2608.20680#S4.E16)\)\. Unlike the controlled diffusion processes in[20](https://arxiv.org/html/2608.20680#bib.bib13), we exploit the piecewise\-constant sample paths of the state and action processes in our CTJMDP setting\. Fix a realization\{\(xt,at\):t∈\[0,T\]\}\\\{\(x\_\{t\},a\_\{t\}\):t\\in\[0,T\]\\\}with state jump times0<τ1<⋯<τL≤T0<\\tau\_\{1\}<\\cdots<\\tau\_\{L\}\\leq T\. For eachℓ=1,…,L\\ell=1,\\ldots,L, letρℓ\\rho\_\{\\ell\}be the jump reward observed at timeτℓ\\tau\_\{\\ell\}\. We augment the decision grid with these realized jump times and write

𝕊∪\{τ1,…,τL\}=\{0=s0<s1<⋯<sM=T\},\\displaystyle\\mathbb\{S\}\\cup\\\{\\tau\_\{1\},\\ldots,\\tau\_\{L\}\\\}=\\\{0=s\_\{0\}<s\_\{1\}<\\cdots<s\_\{M\}=T\\\},where duplicate points are removed\. For eachm=0,…,M−1m=0,\\ldots,M\-1, letxmx\_\{m\}andama\_\{m\}denote the constant values of the state and action processes on\(sm,sm\+1\)\(s\_\{m\},s\_\{m\+1\}\), respectively\. LetΔ​Cm\\Delta C\_\{m\}denote the running reward accumulated over this interval, and setΔm≜sm\+1−sm\\Delta\_\{m\}\\triangleq s\_\{m\+1\}\-s\_\{m\}\. Define

Hθ​\(t,x,a\)\\displaystyle H^\{\\theta\}\(t,x,a\)≜∇θJθ​\(t,x\),Hψ​\(t,x,a\)≜∇ψqψ​\(t,x,a\)\.\\displaystyle\\triangleq\\nabla\_\{\\theta\}J^\{\\theta\}\(t,x\),\\qquad H^\{\\psi\}\(t,x,a\)\\triangleq\\nabla\_\{\\psi\}q^\{\\psi\}\(t,x,a\)\.Then, forv∈\{θ,ψ\}v\\in\\\{\\theta,\\psi\\\}, the corresponding pathwise increment in \([16](https://arxiv.org/html/2608.20680#S4.E16)\) is approximated by

ℐv\\displaystyle\\mathcal\{I\}\_\{v\}≜∫0THv​\(t,xt−,at\)​d​Jθ​\(t,xt\)\+∑y∈𝒳∑z≠y∫0THv​\(t,xt−,at\)​ρ​\(t,y,at,z\)​d​Ny​z​\(t\)\\displaystyle\\triangleq\\int\_\{0\}^\{T\}H^\{v\}\(t,x\_\{t\-\},a\_\{t\}\)\\mathrm\{d\}J^\{\\theta\}\(t,x\_\{t\}\)\+\\sum\_\{y\\in\\mathcal\{X\}\}\\sum\_\{z\\neq y\}\\int\_\{0\}^\{T\}H^\{v\}\(t,x\_\{t\-\},a\_\{t\}\)\\rho\(t,y,a\_\{t\},z\)\\,\\mathrm\{d\}N\_\{yz\}\(t\)\(17\)\+∫0THv\(t,xt−,at\)\[r\(t,xt−,at\)−qψ\(t,xt−,at\)\]dt\\displaystyle\\hskip 142\.26378pt\+\\int\_\{0\}^\{T\}H^\{v\}\(t,x\_\{t\-\},a\_\{t\}\)\[r\(t,x\_\{t\-\},a\_\{t\}\)\-q^\{\\psi\}\(t,x\_\{t\-\},a\_\{t\}\)\]\\mathrm\{d\}t≈∑ℓ=1LHv​\(τℓ,xτℓ−,aτℓ\)​\[Jθ​\(τℓ,xτℓ\)−Jθ​\(τℓ,xτℓ−\)\+ρτl\]\\displaystyle\\approx\\sum\_\{\\ell=1\}^\{L\}H^\{v\}\(\\tau\_\{\\ell\},x\_\{\\tau\_\{\\ell\}\-\},a\_\{\\tau\_\{\\ell\}\}\)\[J^\{\\theta\}\(\\tau\_\{\\ell\},x\_\{\\tau\_\{\\ell\}\}\)\-J^\{\\theta\}\(\\tau\_\{\\ell\},x\_\{\\tau\_\{\\ell\}\-\}\)\+\\rho\_\{\\tau\_\{l\}\}\]\+∑m=0M−1Hv\(sm,xm,am\)\[Jθ\(sm\+1,xm\)−Jθ\(sm,xm\)\+ΔCm−qψ\(sm,xm,am\)Δm\]\.\\displaystyle\\hskip 10\.00002pt\+\\sum\_\{m=0\}^\{M\-1\}H^\{v\}\(s\_\{m\},x\_\{m\},a\_\{m\}\)\[J^\{\\theta\}\(s\_\{m\+1\},x\_\{m\}\)\-J^\{\\theta\}\(s\_\{m\},x\_\{m\}\)\+\\Delta C\_\{m\}\-q^\{\\psi\}\(s\_\{m\},x\_\{m\},a\_\{m\}\)\\Delta\_\{m\}\]\.\(18\)This construction incorporates every realized state jump and grid\-based action change and uses the observed cumulative reward without temporal quadrature\. Consequently, the remaining approximation error arises only from the time variation of the integrands within each interval\. With all the above in place, we present ourqq\-learning algorithm in Algorithm[1](https://arxiv.org/html/2608.20680#alg1)\.

Algorithm 1Episodicqq\-Learning Algorithm1:Initial state

x0x\_\{0\}, horizon

TT, decision grid

𝕊=\{0=t0<t1<⋯<tK=T\}\\mathbb\{S\}=\\\{0=t\_\{0\}<t\_\{1\}<\\cdots<t\_\{K\}=T\\\}, number of episodes

NN, temperature parameter

γ\\gamma, and learning rates

αθ\\alpha\_\{\\theta\}and

αψ\\alpha\_\{\\psi\}\. Approximators

JθJ^\{\\theta\}and

qψq^\{\\psi\}satisfying \([13](https://arxiv.org/html/2608.20680#S4.E13)\)\. Policy

𝝅ψ​\(a∣t,x\)≜exp⁡\{1γ​qψ​\(t,x,a\)\}\\boldsymbol\{\\pi\}^\{\\psi\}\(a\\mid t,x\)\\triangleq\\exp\\\{\\frac\{1\}\{\\gamma\}q^\{\\psi\}\(t,x,a\)\\\}
2:Simulator:Given

\(tk,tk\+1,Xtk,Ak\)\(t\_\{k\},t\_\{k\+1\},X\_\{t\_\{k\}\},A\_\{k\}\), simulate the system over

\(tk,tk\+1\]\(t\_\{k\},t\_\{k\+1\}\]and return the state at

tk\+1t\_\{k\+1\}, all state jumps and the corresponding jump rewards, and the cumulative running reward between successive grid or jump times

3:Initialize

θ1\\theta\_\{1\}and

ψ1\\psi\_\{1\}
4:for

j=1,…,Nj=1,\\ldots,Ndo

5:Set

X0←x0X\_\{0\}\\leftarrow x\_\{0\},

C0←0C\_\{0\}\\leftarrow 0, and

𝒯J←∅\\mathcal\{T\}\_\{J\}\\leftarrow\\varnothing
6:for

k=0,…,K−1k=0,\\ldots,K\-1do

7:Sample

Ak∼𝝅ψj\(⋅∣tk,Xtk\)A\_\{k\}\\sim\\boldsymbol\{\\pi\}^\{\\psi\_\{j\}\}\(\\cdot\\mid t\_\{k\},X\_\{t\_\{k\}\}\)and set

At←AkA\_\{t\}\\leftarrow A\_\{k\}for

t∈\(tk,tk\+1\]t\\in\(t\_\{k\},t\_\{k\+1\}\]
8:Simulate the system over

\(tk,tk\+1\]\(t\_\{k\},t\_\{k\+1\}\]and observe

Xtk\+1X\_\{t\_\{k\+1\}\}
9:Record all state jump times

τℓ\\tau\_\{\\ell\}and post\-jump states in

\(tk,tk\+1\]\(t\_\{k\},t\_\{k\+1\}\], and add the jump times to

𝒯J\\mathcal\{T\}\_\{J\}
10:Record the cumulative running reward

Δ​Cm\\Delta C\_\{m\}between successive grid or jump times in

\[tk,tk\+1\]\[t\_\{k\},t\_\{k\+1\}\]
11:Record the jump reward

ρℓ\\rho\_\{\\ell\}observed at each jump time

τℓ\\tau\_\{\\ell\}in

\(tk,tk\+1\]\(t\_\{k\},t\_\{k\+1\}\]
12:endfor

13:Let

𝒯J=\{τ1<⋯<τL\}\\mathcal\{T\}\_\{J\}=\\\{\\tau\_\{1\}<\\cdots<\\tau\_\{L\}\\\}and construct

𝕊∪𝒯J=\{0=s0<s1<⋯<sM=T\},\\mathbb\{S\}\\cup\\mathcal\{T\}\_\{J\}=\\\{0=s\_\{0\}<s\_\{1\}<\\cdots<s\_\{M\}=T\\\},with duplicate points removed

14:For each

m=0,…,M−1m=0,\\ldots,M\-1, identify the constant state and action

\(xm,am\)\(x\_\{m\},a\_\{m\}\)on

\(sm,sm\+1\)\(s\_\{m\},s\_\{m\+1\}\)and set

Δm←sm\+1−sm\\Delta\_\{m\}\\leftarrow s\_\{m\+1\}\-s\_\{m\}
15:Evaluate

ℐθ\(j\)\\mathcal\{I\}\_\{\\theta\}^\{\(j\)\}and

ℐψ\(j\)\\mathcal\{I\}\_\{\\psi\}^\{\(j\)\}from \([18](https://arxiv.org/html/2608.20680#S4.E18)\) using

\(θj,ψj\)\(\\theta\_\{j\},\\psi\_\{j\}\)and the recorded trajectory

16:Update

θj\+1←θj\+αθ⋅ℐθ\(j\),ψj\+1←ψj\+αψ⋅ℐψ\(j\)\\theta\_\{j\+1\}\\leftarrow\\theta\_\{j\}\+\\alpha\_\{\\theta\}\\cdot\\mathcal\{I\}\_\{\\theta\}^\{\(j\)\},\\qquad\\psi\_\{j\+1\}\\leftarrow\\psi\_\{j\}\+\\alpha\_\{\\psi\}\\cdot\\mathcal\{I\}\_\{\\psi\}^\{\(j\)\}
17:endfor

## 5Case Study: Network Dynamic Pricing

In this section, we illustrate the application of our proposed RL framework for CTJMDPs\. We consider the multi\-product network dynamic pricing problem studied in[11](https://arxiv.org/html/2608.20680#bib.bib26)in the context of an airline network\. Letℐ≜\{1,2,…,m\}\\mathcal\{I\}\\triangleq\\\{1,2,\\ldots,m\\\}denote the set of flight legs and𝒥≜\{1,2,…,n\}\\mathcal\{J\}\\triangleq\\\{1,2,\\ldots,n\\\}the set of origin\-destination itineraries offered over a finite booking horizon\[0,T\]\[0,T\]\. Each flight legi∈ℐi\\in\\mathcal\{I\}has an initial capacitycic\_\{i\}, and each itineraryj∈𝒥j\\in\\mathcal\{J\}follows a fixed path and consumes one unit of capacity on every leg along that path\. At any time, the decision maker observes the remaining leg capacities and sets the itinerary prices\. Lower prices generate demand more rapidly but consume capacity that may be valuable for other itineraries, whereas higher prices preserve capacity at the cost of reduced demand\. Because itineraries share flight legs, their pricing decisions are coupled through the remaining capacities\.

LetA=\[ai​j\]m×nA=\[a\_\{ij\}\]\_\{m\\times n\}be the leg\-itinerary incidence matrix, whereai​j=1a\_\{ij\}=1if itineraryjjuses legiiandai​j=0a\_\{ij\}=0otherwise\. We model the remaining leg capacities as the statex=\(x1,x2,…,xm\)⊤x=\(x\_\{1\},x\_\{2\},\\ldots,x\_\{m\}\)^\{\\top\}, with state space𝒳=∏i=1m\{0,1,…,ci\}\.\\mathcal\{X\}=\\prod\_\{i=1\}^\{m\}\\\{0,1,\\ldots,c\_\{i\}\\\}\.The action is a price vector𝒑=\(p1,p2,…,pn\)⊤∈𝒜\\boldsymbol\{p\}=\(p\_\{1\},p\_\{2\},\\ldots,p\_\{n\}\)^\{\\top\}\\in\\mathcal\{A\}, where𝒜⊆ℝ\+n\\mathcal\{A\}\\subseteq\\mathbb\{R\}\_\{\+\}^\{n\}denotes the set of allowable prices\. Requests for itineraryjjarrive according to a Poisson process with rateλj​\(𝒑\)\\lambda\_\{j\}\(\\boldsymbol\{p\}\)\. LetAjA\_\{j\}denote columnjjofAA\. A sale of itineraryjjchanges the state fromxxtox−Ajx\-A\_\{j\}and can occur only if every leg on the itinerary has positive remaining capacity\. Thus, the transition rates are

λ\(x−Aj∣t,x,𝒑\)=λj\(𝒑\)𝟏\{x≥Aj\},j∈𝒥,\\displaystyle\\lambda\(x\-A\_\{j\}\\mid t,x,\\boldsymbol\{p\}\)=\\lambda\_\{j\}\(\\boldsymbol\{p\}\)\\boldsymbol\{1\}\_\{\\\{x\\geq A\_\{j\}\\\}\},\\qquad j\\in\\mathcal\{J\},andλ\(x∣t,x,𝒑\)=−∑j=1nλj\(𝒑\)𝟏\{x≥Aj\}\\lambda\(x\\mid t,x,\\boldsymbol\{p\}\)=\-\\sum\_\{j=1\}^\{n\}\\lambda\_\{j\}\(\\boldsymbol\{p\}\)\\boldsymbol\{1\}\_\{\\\{x\\geq A\_\{j\}\\\}\}\. Since revenue is earned only upon a sale, the running reward rate and the terminal reward are zero\. The jump reward function is given by

ρ⁡\(t,x,𝒑,x−Aj\)=pj,j∈𝒥,x≥Aj\.\\rho\(t,x,\\boldsymbol\{p\},x\-A\_\{j\}\)=p\_\{j\},\\qquad j\\in\\mathcal\{J\},\\quad x\\geq A\_\{j\}\.In the RL setting, the decision maker does not know the demand functions but observes the realized sales, the corresponding revenues, and the resulting changes in the remaining leg capacities\.

For both numerical instances considered below, the booking horizon is normalized toT=1T=1\. Requests for different itineraries arrive according to independent Poisson processes, and demand is time homogeneous and separable across itineraries\. Specifically, the demand function for itineraryjjis given by

λj​\(𝒑\)≡λj​\(pj\)=λj0​exp⁡\{−ϵj0​\(pjpj0−1\)\},\\displaystyle\\lambda\_\{j\}\(\\boldsymbol\{p\}\)\\equiv\\lambda\_\{j\}\(p\_\{j\}\)=\\lambda\_\{j\}^\{0\}\\exp\\\{\-\\epsilon\_\{j\}^\{0\}\(\\frac\{p\_\{j\}\}\{p\_\{j\}^\{0\}\}\-1\)\\\},\(19\)whereλj0\\lambda\_\{j\}^\{0\},pj0p\_\{j\}^\{0\}, andϵj0\\epsilon\_\{j\}^\{0\}denote the reference demand rate, reference price, and price elasticity, respectively\. Under the above formulation, one can readily check that Assumption[1](https://arxiv.org/html/2608.20680#Thmassumption1)is satisfied\. In particular, since the transition rates are conservative in the sense that∑y∈𝒳λ⁡\(y∣t,x,a\)≡0\\sum\_\{y\\in\\mathcal\{X\}\}\\lambda\(y\\mid t,x,a\)\\equiv 0, and the running reward rates and terminal reward are zero, one may takew⁡\(x\)≡1w\(x\)\\equiv 1,c\>0c\>0,b=0b=0, andM1=∑j=1nλj0​pj0​eϵj0−1/ϵj0M\_\{1\}=\\sum\_\{j=1\}^\{n\}\\lambda\_\{j\}^\{0\}p\_\{j\}^\{0\}e^\{\\epsilon\_\{j\}^\{0\}\-1\}/\\epsilon\_\{j\}^\{0\}\.

#### Benchmarks\.

We consider the following benchmarks for comparison\.

- •Dynamic programming \(DP\)\.The DP benchmark is obtained from a time\-discretized approximation of the continuous\-time pricing model\. Specifically, givenΔ​t′\>0\\Delta t^\{\\prime\}\>0, letK′≜⌈TΔ​t′⌉K^\{\\prime\}\\triangleq\\lceil\\frac\{T\}\{\\Delta t^\{\\prime\}\}\\rceilandtk′=k​Δ​t′t\_\{k\}^\{\\prime\}=k\\Delta t^\{\\prime\}\. In each interval\[tk′,tk\+1′\)\[t\_\{k\}^\{\\prime\},t\_\{k\+1\}^\{\\prime\}\), the price vector𝒑\\boldsymbol\{p\}is held fixed\. Conditional on the state beingxxattk′t\_\{k\}^\{\\prime\}, a sale of itineraryjjoccurs with probabilityλj\(pj\)𝟏\{x≥Aj\}Δs\\lambda\_\{j\}\(p\_\{j\}\)\\boldsymbol\{1\}\_\{\\\{x\\geq A\_\{j\}\\\}\}\\Delta s, and no sale occurs with the remaining probability\. The optimal value functionVΔ​t′∗V\_\{\\Delta t^\{\\prime\}\}^\{\*\}of the resulting discrete\-time model satisfies VΔ​t′∗​\(tk′,x\)\\displaystyle V\_\{\\Delta t^\{\\prime\}\}^\{\*\}\(t\_\{k\}^\{\\prime\},x\)=\\displaystyle=VΔ​t′∗\(tk\+1′,x\)\+Δt′max𝒑∈ℝ\+n∑j=1nλj\(pj\)𝟏\{x≥Aj\}\[pj\+VΔ​t′∗\(tk\+1′,x−Aj\)−VΔ​t′∗\(tk\+1′,x\)\],\\displaystyle V\_\{\\Delta t^\{\\prime\}\}^\{\*\}\(t\_\{k\+1\}^\{\\prime\},x\)\+\\Delta t^\{\\prime\}\\max\_\{\\boldsymbol\{p\}\\in\\mathbb\{R\}\_\{\+\}^\{n\}\}\\sum\_\{j=1\}^\{n\}\\lambda\_\{j\}\(p\_\{j\}\)\\boldsymbol\{1\}\_\{\\\{x\\geq A\_\{j\}\\\}\}\\big\[p\_\{j\}\+V\_\{\\Delta t^\{\\prime\}\}^\{\*\}\(t\_\{k\+1\}^\{\\prime\},x\-A\_\{j\}\)\-V\_\{\\Delta t^\{\\prime\}\}^\{\*\}\(t\_\{k\+1\}^\{\\prime\},x\)\\big\],VΔ​t′∗​\(T,x\)\\displaystyle V\_\{\\Delta t^\{\\prime\}\}^\{\*\}\(T,x\)=\\displaystyle=0\.\\displaystyle 0\.For separable exponential demand functions of the form in \([19](https://arxiv.org/html/2608.20680#S5.E19)\), the optimal price is given by pj,k∗​\(x\)=VΔ​t′∗​\(tk\+1′,x\)−VΔ​t′∗​\(tk\+1′,x−Aj\)\+pj0ϵj0\.p\_\{j,k\}^\{\*\}\(x\)=V\_\{\\Delta t^\{\\prime\}\}^\{\*\}\(t\_\{k\+1\}^\{\\prime\},x\)\-V\_\{\\Delta t^\{\\prime\}\}^\{\*\}\(t\_\{k\+1\}^\{\\prime\},x\-A\_\{j\}\)\+\\frac\{p\_\{j\}^\{0\}\}\{\\epsilon\_\{j\}^\{0\}\}\.Hence, no discretization of the price space is required\. For sufficiently smallΔ​t′\\Delta t^\{\\prime\},VΔ​t′∗​\(0,c\)V\_\{\\Delta t^\{\\prime\}\}^\{\*\}\(0,c\)provides a numerical approximation toV∗​\(0,c\)V^\{\*\}\(0,c\), the optimal value of the original continuous\-time pricing problem\.
- •Deterministic upper bound \(UB\)\.The deterministic problem replaces stochastic sales with their expected rates and solves sup𝒑⁡\(⋅\)∫0T𝒑​\(t\)⊤​𝝀​\(𝒑⁡\(t\)\)​𝑑ts\.t\.∫0TA​𝝀​\(𝒑⁡\(t\)\)​𝑑t≤c,\\displaystyle\\sup\_\{\\boldsymbol\{p\}\(\\cdot\)\}\\quad\\int\_\{0\}^\{T\}\\boldsymbol\{p\}\(t\)^\{\\top\}\\boldsymbol\{\\lambda\}\\bigl\(\\boldsymbol\{p\}\(t\)\\bigr\)\\,\\mathrm\{d\}t\\qquad\\text\{s\.t\.\}\\quad\\int\_\{0\}^\{T\}A\\boldsymbol\{\\lambda\}\(\\boldsymbol\{p\}\(t\)\)\\,\\mathrm\{d\}t\\leq c,where𝝀⁡\(𝒑\)=\(λ1​\(p1\),…,λn​\(pn\)\)⊤\.\\boldsymbol\{\\lambda\}\(\\boldsymbol\{p\}\)=\\bigl\(\\lambda\_\{1\}\(p\_\{1\}\),\\ldots,\\lambda\_\{n\}\(p\_\{n\}\)\\bigr\)^\{\\top\}\.Its optimal value provides an upper bound on the optimal expected revenue of the stochastic problem\.
- •Fluid pricing \(FP\)\.The FP policy follows the optimal price path obtained from the deterministic problem\. An itinerary is made unavailable once any of its required legs no longer has sufficient remaining capacity\.
- •Fluid pricing with booking limits \(FP\-BL\)\.The FP\-BL policy follows the same deterministic price path as the FP policy but imposes a booking limit for each itinerary\. Specifically, under an optimal deterministic price path𝒑d​\(t\)\\boldsymbol\{p\}^\{d\}\(t\), the booking limit for itineraryjjis set to⌊∫0Tλj​\(pjd​\(t\)\)​𝑑t⌋\\lfloor\\int\_\{0\}^\{T\}\\lambda\_\{j\}\\bigl\(p\_\{j\}^\{d\}\(t\)\\bigr\)\\,\\mathrm\{d\}t\\rfloor\. Itineraryjjis made unavailable once its booking limit is reached\.

The benchmarks above require knowledge of the demand functions, whereas Algorithm[1](https://arxiv.org/html/2608.20680#alg1)uses only observed state transitions and revenues\. The FP and FP\-BL policies are heuristics constructed from the optimal solution of the deterministic problem\.[11](https://arxiv.org/html/2608.20680#bib.bib26)shows that both policies are asymptotically optimal when the initial capacities and demand rates increase proportionally, and their revenues accordingly approach the deterministic upper bound as the scale increases\.

#### Implementation of Algorithm[1](https://arxiv.org/html/2608.20680#alg1)\.

For both instances, we employ two separate fully connected neural networks,fcriticθf\_\{\\mathrm\{critic\}\}^\{\\theta\}andfactorψf\_\{\\mathrm\{actor\}\}^\{\\psi\}, to construct the approximatorsJθJ^\{\\theta\}andqψq^\{\\psi\}, respectively\. Both networks use ReLU activations in the hidden layers, whose number and width are specified for each instance below\. The parameters of each network are updated using a separate Adam optimizer\. Both networks take the normalized\(m\+1\)\(m\+1\)\-dimensional time\-state vector\(t~,x~\)=\(1−tT,x1c1,…,xmcm\)\(\\tilde\{t\},\\tilde\{x\}\)=\(1\-\\frac\{t\}\{T\},\\frac\{x\_\{1\}\}\{c\_\{1\}\},\\ldots,\\frac\{x\_\{m\}\}\{c\_\{m\}\}\)as input\. The critic network outputs a scalar, and the value\-function approximator is defined by

Jθ​\(t,x\)=\(1−tT\)⋅fcriticθ​\(t~,x~\)\.\\displaystyle J^\{\\theta\}\(t,x\)=\(1\-\\frac\{t\}\{T\}\)\\cdot f\_\{\\mathrm\{critic\}\}^\{\\theta\}\(\\tilde\{t\},\\tilde\{x\}\)\.This construction ensures thatJθ​\(T,x\)=0=h⁡\(x\)J^\{\\theta\}\(T,x\)=0=h\(x\)for everyx∈𝒳x\\in\\mathcal\{X\}\. The actor network outputs two positivenn\-dimensional vectors,μψ​\(t~,x~\)\\mu^\{\\psi\}\(\\tilde\{t\},\\tilde\{x\}\)andkψ​\(t~,x~\)k^\{\\psi\}\(\\tilde\{t\},\\tilde\{x\}\)\. Conditional on\(t,x\)\(t,x\), the itinerary prices are sampled independently according to

pj∼Gamma\(kjψ​\(t~,x~\)γ,kjψ​\(t~,x~\)γ​μjψ​\(t~,x~\)\),j=1,…,n,\\displaystyle p\_\{j\}\\sim\\operatorname\{Gamma\}\\left\(\\frac\{k\_\{j\}^\{\\psi\}\(\\tilde\{t\},\\tilde\{x\}\)\}\{\\gamma\},\\frac\{k\_\{j\}^\{\\psi\}\(\\tilde\{t\},\\tilde\{x\}\)\}\{\\gamma\\mu\_\{j\}^\{\\psi\}\(\\tilde\{t\},\\tilde\{x\}\)\}\\right\),\\qquad j=1,\\ldots,n,\(20\)where the Gamma distribution is parameterized by its shape and rate\. We choose the Gamma distribution because its support on\[0,∞\)\[0,\\infty\)ensures that sampled prices are nonnegative, while its two parameters allow flexible control over the mean and relative dispersion\. Under \([20](https://arxiv.org/html/2608.20680#S5.E20)\), the mean and variance are given byμjψ\\mu\_\{j\}^\{\\psi\}andγ​\(μjψ\)2/kjψ\\gamma\(\\mu\_\{j\}^\{\\psi\}\)^\{2\}/k\_\{j\}^\{\\psi\}, respectively\. Thus, a smaller temperature parameterγ\\gammayields a policy that is more concentrated around its mean, consistent with the behavior of the optimal policy characterized in \([10](https://arxiv.org/html/2608.20680#S3.E10)\)\. Let𝝅jψ\(⋅∣t,x\)\\boldsymbol\{\\pi\}\_\{j\}^\{\\psi\}\(\\cdot\\mid t,x\)denote the resulting marginal price density for itineraryjj\. The joint policy density andqq\-function approximator are defined by

𝝅ψ​\(𝒑∣t,x\)=∏j=1n𝝅jψ​\(pj∣t,x\),qψ​\(t,x,𝒑\)=γ​∑j=1nlog⁡𝝅jψ​\(pj∣t,x\)\.\\displaystyle\\boldsymbol\{\\pi\}^\{\\psi\}\(\\boldsymbol\{p\}\\mid t,x\)=\\prod\_\{j=1\}^\{n\}\\boldsymbol\{\\pi\}\_\{j\}^\{\\psi\}\(p\_\{j\}\\mid t,x\),\\qquad q^\{\\psi\}\(t,x,\\boldsymbol\{p\}\)=\\gamma\\sum\_\{j=1\}^\{n\}\\log\\boldsymbol\{\\pi\}\_\{j\}^\{\\psi\}\(p\_\{j\}\\mid t,x\)\.This construction ensures thatqψq^\{\\psi\}satisfies the normalization condition in \([13](https://arxiv.org/html/2608.20680#S4.E13)\)\.

### 5\.1A Small\-Network Example

We first consider a two\-leg network with three itineraries\. The first two itineraries each use a single leg, whereas the third uses both legs\. The initial capacity vector and the leg–itinerary incidence matrix are given byc=\(12,10\)⊤c=\(12,10\)^\{\\top\}andA=\[101011\]A=\\begin\{bmatrix\}1&0&1\\\\ 0&1&1\\end\{bmatrix\}\. The demand parameters are reported in Table[1](https://arxiv.org/html/2608.20680#S5.T1)\.

Table 1:Itinerary and demand parameters for the example in Section[5\.1](https://arxiv.org/html/2608.20680#S5.SS1)To implement Algorithm[1](https://arxiv.org/html/2608.20680#alg1)for this instance, we configure both the actor and critic networks with two hidden layers, each of width 32\. We set the initial state tox0=cx\_\{0\}=cand the time\-grid spacing toΔ​t=0\.01\\Delta t=0\.01\. The actor and critic learning rates areαψ=3×10−5\\alpha\_\{\\psi\}=3\\times 10^\{\-5\}andαθ=5×10−4\\alpha\_\{\\theta\}=5\\times 10^\{\-4\}, respectively, and the temperature parameter isγ=1×10−2\\gamma=1\\times 10^\{\-2\}\.

During training, we periodically evaluate the current learned policy by averaging the total revenue over 10,000 independent simulation runs\. Figure[1](https://arxiv.org/html/2608.20680#S5.F1)reports the average revenues of the policies generated by Algorithm[1](https://arxiv.org/html/2608.20680#alg1)throughout the learning process, together with reference lines for the deterministic upper bound, the DP optimal value, and the revenues of FP and FP\-BL\. Here, the DP optimal value is computed with a time\-discretization step ofΔ​s=10−4\\Delta s=10^\{\-4\}, yieldingV0\.0001∗​\(0,c\)=709\.883V\_\{0\.0001\}^\{\*\}\(0,c\)=709\.883\. Table[2](https://arxiv.org/html/2608.20680#S5.T2)reports the performance of Algorithm[1](https://arxiv.org/html/2608.20680#alg1), FP, and FP\-BL relative to the DP optimal value\. The learned policy outperforms FP and FP\-BL by8\.87%8\.87\\%and18\.91%18\.91\\%, respectively, while attaining performance within2\.39%2\.39\\%of the DP optimal value\. These results demonstrate that the proposed algorithm can learn a near\-optimal pricing policy\. Note that although the FP and FP\-BL policies are asymptotically optimal, they exhibit performance gaps of10\.33%10\.33\\%and17\.90%17\.90\\%, respectively, relative to the DP optimal value in this instance\. These gaps arise from the small capacities and resource coupling in this instance, which make the opportunity costs of capacity sensitive to realized demand\. FP retains fixed prices, whereas FP\-BL additionally imposes fixed itinerary\-level allocations, so neither adapts these decisions to the evolving system state\. In contrast, our algorithm learns a state\-dependent pricing policy that responds to the remaining time and capacities, thereby achieving performance close to the DP optimal value\.

Figure 1:Average revenue of Algorithm[1](https://arxiv.org/html/2608.20680#alg1)over episodes for the example in Section[5\.1](https://arxiv.org/html/2608.20680#S5.SS1)0246810×105\\times 10^\{5\}500500582\.763582\.763636\.519636\.519709\.883709\.883741\.549741\.549EpisodeAverage RevenueUBDPFPFP\-BLAlgorithm[1](https://arxiv.org/html/2608.20680#alg1)Table 2:Numerical results for the example in Section[5\.1](https://arxiv.org/html/2608.20680#S5.SS1)
### 5\.2A Large\-Network Example

Next, we evaluate Algorithm[1](https://arxiv.org/html/2608.20680#alg1)on an airline pricing problem with a large state space and a high\-dimensional continuous action space\. The instance is based on Example 1 of[11](https://arxiv.org/html/2608.20680#bib.bib26), which comprises a six\-node network with 11 flight legs and 18 itineraries\. Figure[2](https://arxiv.org/html/2608.20680#S5.F2)depicts the network and the initial leg capacities\. The reference demand rate is set toλj0=30\\lambda\_\{j\}^\{0\}=30for every itinerary, and the remaining demand parameters and itinerary paths are reported in Table[3](https://arxiv.org/html/2608.20680#S5.T3)\. This instance corresponds to the scale\-0\.10\.1version of Example 1 in[11](https://arxiv.org/html/2608.20680#bib.bib26), where scale\-κ\\kappameans that both the initial leg capacities and reference demand rates are set toκ\\kappatimes their values in the base instance, with all other model parameters unchanged\. We focus on the scale\-0\.1 instance because the FP\-BL and FP policies are asymptotically optimal as the scaleκ\\kappagrows, so we use a small\-κ\\kappainstance to demonstrate the advantages of our algorithm in a non\-asymptotic regime\.

Figure 2:Airline network for the example in Section[5\.2](https://arxiv.org/html/2608.20680#S5.SS2)1234563010102020402010101030Table 3:Itinerary and demand parameters for example in Section[5\.2](https://arxiv.org/html/2608.20680#S5.SS2)For this instance, we implement Algorithm[1](https://arxiv.org/html/2608.20680#alg1)as follows\. Both the actor and critic networks have two hidden layers of width 128\. We initialize the state atx0=cx\_\{0\}=cand set the time\-grid spacing toΔ​t=0\.01\\Delta t=0\.01\. The learning rates areαθ=3×10−6\\alpha\_\{\\theta\}=3\\times 10^\{\-6\}for the actor andαϕ=3×10−4\\alpha\_\{\\phi\}=3\\times 10^\{\-4\}for the critic, and the temperature parameter is set toγ=1×10−2\\gamma=1\\times 10^\{\-2\}\.

Figure 3:Average revenue of Algorithm[1](https://arxiv.org/html/2608.20680#alg1)over episodes for the example in Section[5\.2](https://arxiv.org/html/2608.20680#S5.SS2)0246810×106\\times 10^\{6\}30,00030,00040,00040,00050,00050,00054,51054,51058,00158,00166,12066,120EpisodeAverage RevenueUBFPFP\-BLAlgorithm[1](https://arxiv.org/html/2608.20680#alg1)The state space for this instance contains approximately5\.88×10135\.88\\times 10^\{13\}states, rendering direct dynamic programming computationally infeasible\. Figure[3](https://arxiv.org/html/2608.20680#S5.F3)reports the average revenue of the policies generated by Algorithm[1](https://arxiv.org/html/2608.20680#alg1)throughout the learning process, together with the deterministic upper bound and the FP and FP\-BL benchmarks\. It shows that the final policy learned by our algorithm achieves of58,81858\{,\}818, exceeding the FP\-BL revenue by7\.9%7\.9\\%\. Moreover, despite operating without knowledge of the demand functions, our algorithm delivers performance comparable to that of the FP policy, whose construction requires a known demand model\.

## 6Conclusion and Future Work

In this paper, we establish the theoretical foundations for reinforcement learning in CTJMDPs with general discrete state spaces and propose efficientqq\-learning algorithms\. We apply this general framework to network dynamic pricing and queueing control, where numerical experiments demonstrate the strong performance of our approach\.

This work opens several avenues for future research\. From a theoretical perspective, critical directions include establishing convergence analysis for the proposedqq\-learning algorithm and studying the underlying stochastic approximation procedures\. From an applied perspective, it would be interesting to apply our general framework to other high\-dimensional operations research applications\.

## References

- R\. Atar and M\. I\. ReimanAsymptotically optimal dynamic pricing for network revenue management\.Stochastic Systems2\(2\),pp\. 232–276\.Cited by:[§1\.2](https://arxiv.org/html/2608.20680#S1.SS2.SSS0.Px2.p1.1)\.
- Besbes and Zeevi \(2012\)O\. Besbes and A\. ZeeviBlind network revenue management\.Operations research60\(6\),pp\. 1537–1550\.Cited by:[§1\.2](https://arxiv.org/html/2608.20680#S1.SS2.SSS0.Px2.p2.1)\.
- Bradtke and Duff \(1995\)S\. J\. Bradtke and M\. O\. DuffReinforcement learning methods for continuous\-time markov decision\.Advances in Neural Information Processing Systems 77,pp\. 393\.Cited by:[§1\.2](https://arxiv.org/html/2608.20680#S1.SS2.SSS0.Px1.p1.1)\.
- Brémaud \(1981\)P\. BrémaudPoint processes and queues: martingale dynamics\.Springer\-Verlag,New York\.Cited by:[Appendix A](https://arxiv.org/html/2608.20680#A1.p1.1),[§1](https://arxiv.org/html/2608.20680#S1.p2.1)\.
- Chen and Shi \(2023\)Y\. Chen and C\. ShiNetwork revenue management with online inverse batch gradient descent method\.Production and Operations Management32\(7\),pp\. 2123–2137\.Cited by:[§1\.2](https://arxiv.org/html/2608.20680#S1.SS2.SSS0.Px2.p2.1)\.
- Dai and Gluzman \(2022\)J\. G\. Dai and M\. GluzmanQueueing network controls via deep reinforcement learning\.Stochastic Systems12\(1\),pp\. 30–67\.Cited by:[§1\.2](https://arxiv.org/html/2608.20680#S1.SS2.SSS0.Px1.p1.1)\.
- Daiet al\.\(2023\)M\. Dai, Y\. Dong, and Y\. JiaLearning equilibrium mean\-variance strategy\.Mathematical Finance33\(4\),pp\. 1166–1212\.Cited by:[§1](https://arxiv.org/html/2608.20680#S1.p1.1)\.
- Daset al\.\(1999\)T\. K\. Das, A\. Gosavi, S\. Mahadevan, and N\. MarchalleckSolving semi\-markov decision problems using average reward reinforcement learning\.Management Science45\(4\),pp\. 560–574\.Cited by:[§1\.2](https://arxiv.org/html/2608.20680#S1.SS2.SSS0.Px1.p1.1)\.
- Feinberg \(2004\)E\. A\. FeinbergContinuous time discounted jump markov decision processes: a discrete\-event approach\.Mathematics of Operations Research29\(3\),pp\. 492–524\.Cited by:[§1\.2](https://arxiv.org/html/2608.20680#S1.SS2.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2608.20680#S1.p2.1),[§3](https://arxiv.org/html/2608.20680#S3.p3.1)\.
- Gallego and van Ryzin \(1994\)G\. Gallego and G\. van RyzinOptimal dynamic pricing of inventories with stochastic demand over finite horizons\.Management Science40\(8\),pp\. 999–1020\.Cited by:[§1](https://arxiv.org/html/2608.20680#S1.p2.1)\.
- Gallego and van Ryzin \(1997\)G\. Gallego and G\. van RyzinA multiproduct dynamic pricing problem and its applications to network yield management\.Operations research45\(1\),pp\. 24–41\.Cited by:[2nd item](https://arxiv.org/html/2608.20680#S1.I1.i2.p1.1),[§1\.2](https://arxiv.org/html/2608.20680#S1.SS2.SSS0.Px2.p1.1),[§1\.2](https://arxiv.org/html/2608.20680#S1.SS2.SSS0.Px2.p2.1),[§5](https://arxiv.org/html/2608.20680#S5.SS0.SSS0.Px1.p1.2),[§5\.2](https://arxiv.org/html/2608.20680#S5.SS2.p1.1),[§5](https://arxiv.org/html/2608.20680#S5.p1.1),[Abstract](https://arxiv.org/html/2608.20680#abstract1.1)\.
- Gaoet al\.\(2026\)X\. Gao, L\. Li, and X\. Y\. ZhouReinforcement learning for jump\-diffusions, with financial applications\.Mathematical Finance\.Cited by:[1st item](https://arxiv.org/html/2608.20680#S1.I1.i1.p2.1),[§1](https://arxiv.org/html/2608.20680#S1.p1.1),[§4\.2](https://arxiv.org/html/2608.20680#S4.SS2.p4.1),[§4\.2](https://arxiv.org/html/2608.20680#S4.SS2.p8.1)\.
- Gao and Zhou \(2024\)X\. Gao and X\. Y\. ZhouLogarithmic regret bounds for continuous\-time average\-reward markov decision processes\.SIAM Journal on Control and Optimization62\(5\),pp\. 2529–2556\.Cited by:[§1\.2](https://arxiv.org/html/2608.20680#S1.SS2.SSS0.Px1.p1.1)\.
- Gao and Zhou \(2025\)X\. Gao and X\. ZhouSquare\-root regret bounds for continuous\-time episodic markov decision processes\.Mathematics of Operations Research\.Cited by:[§1\.2](https://arxiv.org/html/2608.20680#S1.SS2.SSS0.Px1.p1.1)\.
- Guoet al\.\(2015\)X\. Guo, X\. Huang, and Y\. HuangFinite\-horizon optimality for continuous\-time markov decision processes with unbounded transition rates\.Advances in Applied Probability47\(4\),pp\. 1064–1087\.Cited by:[§B\.1](https://arxiv.org/html/2608.20680#A2.SS1.p3.2),[§B\.2](https://arxiv.org/html/2608.20680#A2.SS2.p2.2),[§1](https://arxiv.org/html/2608.20680#S1.p2.1),[§2](https://arxiv.org/html/2608.20680#S2.p5.1),[§2](https://arxiv.org/html/2608.20680#S2.p6.1),[§2](https://arxiv.org/html/2608.20680#S2.p7.1),[§3](https://arxiv.org/html/2608.20680#S3.p4.1)\.
- Guoet al\.\(2022\)X\. Guo, R\. Xu, and T\. ZariphopoulouEntropy regularization for mean field games with learning\.Mathematics of Operations research47\(4\),pp\. 3239–3260\.Cited by:[§1](https://arxiv.org/html/2608.20680#S1.p1.1)\.
- Haarnojaet al\.\(2018\)T\. Haarnoja, A\. Zhou, K\. Hartikainen, G\. Tucker, S\. Ha, J\. Tan, V\. Kumar, H\. Zhu, A\. Gupta, P\. Abbeel,et al\.Soft actor\-critic algorithms and applications\.arXiv preprint arXiv:1812\.05905\.Cited by:[§3](https://arxiv.org/html/2608.20680#S3.p4.1)\.
- Hausenblaset al\.\(2025\)P\. Hausenblas, D\. Eichhorn, A\. Brieden, M\. Soppert, and C\. SteinhardtImproving network dynamic pricing policies through offline reinforcement learning\.OR Spectrum47\(4\),pp\. 1217–1266\.Cited by:[§1\.2](https://arxiv.org/html/2608.20680#S1.SS2.SSS0.Px2.p2.1)\.
- Jasin \(2014\)S\. JasinReoptimization and self\-adjusting price control for network revenue management\.Operations Research62\(5\),pp\. 1168–1178\.Cited by:[§1\.2](https://arxiv.org/html/2608.20680#S1.SS2.SSS0.Px2.p1.1)\.
- Jia and Zhou \(2023\)Y\. Jia and X\. Y\. ZhouQ\-learning in continuous time\.Journal of Machine Learning Research24\(161\),pp\. 1–61\.External Links:[Link](http://jmlr.org/papers/v24/22-0755.html)Cited by:[Appendix B](https://arxiv.org/html/2608.20680#A2.p2.1.1),[1st item](https://arxiv.org/html/2608.20680#S1.I1.i1.p2.1),[§1](https://arxiv.org/html/2608.20680#S1.p1.1),[§3](https://arxiv.org/html/2608.20680#S3.p4.1),[§4\.1](https://arxiv.org/html/2608.20680#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2608.20680#S4.SS1.p2.2),[§4\.2](https://arxiv.org/html/2608.20680#S4.SS2.p4.1),[§4\.2](https://arxiv.org/html/2608.20680#S4.SS2.p8.1),[§4\.3](https://arxiv.org/html/2608.20680#S4.SS3.p2.4),[§4\.3](https://arxiv.org/html/2608.20680#S4.SS3.p3.1),[Abstract](https://arxiv.org/html/2608.20680#abstract1.1)\.
- Kallianpur \(2013\)G\. KallianpurStochastic filtering theory\.Vol\.13,Springer Science & Business Media\.Cited by:[§3](https://arxiv.org/html/2608.20680#S3.p2.1)\.
- Liet al\.\(2026\)T\. Li, C\. Wang, Y\. Wang, S\. Tang, and N\. ChenDeep reinforcement learning for online assortment customization: a data\-driven approach\.Production and Operations Management35\(2\),pp\. 665–684\.Cited by:[§1\.2](https://arxiv.org/html/2608.20680#S1.SS2.SSS0.Px2.p1.1)\.
- Liu and Van Ryzin \(2008\)Q\. Liu and G\. Van RyzinOn the choice\-based linear programming model for network revenue management\.Manufacturing & Service Operations Management10\(2\),pp\. 288–310\.Cited by:[§1\.2](https://arxiv.org/html/2608.20680#S1.SS2.SSS0.Px2.p1.1)\.
- Maglaras and Meissner \(2006\)C\. Maglaras and J\. MeissnerDynamic pricing strategies for multiproduct revenue management problems\.Manufacturing & Service Operations Management8\(2\),pp\. 136–148\.Cited by:[§1\.2](https://arxiv.org/html/2608.20680#S1.SS2.SSS0.Px2.p1.1)\.
- Menget al\.\(2024\)H\. Meng, N\. Chen, and X\. GaoReinforcement learning for intensity control: an application to choice\-based network revenue management\.Operations Research\. Available at arXiv:2406\.05358\.Cited by:[§1\.2](https://arxiv.org/html/2608.20680#S1.SS2.SSS0.Px1.p1.1)\.
- Miao and Wang \(2025\)S\. Miao and Y\. WangNetwork revenue management with nonparametric demand learning: t\-regret and polynomial dimension dependency\.Mathematics of Operations Research\.Cited by:[§1\.2](https://arxiv.org/html/2608.20680#S1.SS2.SSS0.Px2.p2.1)\.
- Miller \(1968\)B\. L\. MillerFinite state continuous time markov decision processes with a finite planning horizon\.SIAM Journal on Control6\(2\),pp\. 266–280\.Cited by:[§1](https://arxiv.org/html/2608.20680#S1.p2.1)\.
- Munos \(2006\)R\. MunosPolicy gradient in continuous time\.Journal of Machine Learning Research7,pp\. 771–791\.Cited by:[§1](https://arxiv.org/html/2608.20680#S1.p1.1)\.
- Parket al\.\(2021\)S\. Park, J\. Kim, and G\. KimTime discretization\-invariant safe action repetition for policy gradient methods\.Advances in Neural Information Processing Systems34,pp\. 267–279\.Cited by:[§1](https://arxiv.org/html/2608.20680#S1.p1.1)\.
- Puterman \(2014\)M\. L\. PutermanMarkov decision processes: discrete stochastic dynamic programming\.John Wiley & Sons\.Cited by:[§1](https://arxiv.org/html/2608.20680#S1.p2.1)\.
- Rana and Oliveira \(2015\)R\. Rana and F\. S\. OliveiraDynamic pricing policies for interdependent perishable products or services using reinforcement learning\.Expert systems with applications42\(1\),pp\. 426–436\.Cited by:[§1\.2](https://arxiv.org/html/2608.20680#S1.SS2.SSS0.Px2.p2.1)\.
- Sutton and Barto \(2018\)R\. S\. Sutton and A\. G\. BartoReinforcement learning: an introduction\.MIT press\.Cited by:[§1](https://arxiv.org/html/2608.20680#S1.p1.1)\.
- Tallecet al\.\(2019\)C\. Tallec, L\. Blier, and Y\. OllivierMaking deep q\-learning methods robust to time discretization\.arXiv preprint arXiv:1901\.09732\.Cited by:[§1](https://arxiv.org/html/2608.20680#S1.p1.1)\.
- Talluri and Van Ryzin \(1998\)K\. Talluri and G\. Van RyzinAn analysis of bid\-price controls for network revenue management\.Management science44\(11\-part\-1\),pp\. 1577–1593\.Cited by:[§1\.2](https://arxiv.org/html/2608.20680#S1.SS2.SSS0.Px2.p1.1)\.
- Wanget al\.\(2025\)B\. Wang, X\. Gao, and L\. LiReinforcement learning for continuous\-time optimal execution: actor\-critic algorithm and error analysis\.Finance and Stochastics\.Cited by:[§1](https://arxiv.org/html/2608.20680#S1.p1.1)\.
- Weiet al\.\(2024\)X\. Wei, X\. Yu, and F\. YuanUnified continuous\-time q\-learning for mean\-field game and mean\-field control problems\.arXiv preprint arXiv:2407\.04521\.Cited by:[§1](https://arxiv.org/html/2608.20680#S1.p1.1)\.
- Zhang and Lu \(2013\)D\. Zhang and Z\. LuAssessing the value of dynamic pricing in network revenue management\.INFORMS Journal on Computing25\(1\),pp\. 102–115\.Cited by:[§1\.2](https://arxiv.org/html/2608.20680#S1.SS2.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2026\)Z\. Zhang, J\. Sheng, D\. D\. Yao, and W\. TangA continuous\-time reinforcement learning framework for fine\-tuning discrete diffusion models\.arXiv preprint arXiv:2607\.14522\.Cited by:[§1\.2](https://arxiv.org/html/2608.20680#S1.SS2.SSS0.Px1.p2.1)\.
- Zhaoet al\.\(2023\)H\. Zhao, W\. Tang, and D\. YaoPolicy optimization for continuous reinforcement learning\.Advances in Neural Information Processing Systems36,pp\. 13637–13663\.Cited by:[§1](https://arxiv.org/html/2608.20680#S1.p1.1)\.

## Appendix ACase Study: Dynamic Server Allocation for Queue Control

In this section, we apply our RL framework to a classic finite\-horizon continuous\-time queueing control problem\([4](https://arxiv.org/html/2608.20680#bib.bib14), Chapter VII\), which can be formulated as a finite\-horizon CTJMDP\.

Spefically, we consider a service system operated over a finite horizon\[0,T\]\[0,T\], with capacityCCandSShomogeneous servers\. Letxxdenote the queue length, defined as the total number of customers in the system\. At any timett, the decision maker observesxxand chooses the numberaaof active servers\. Accordingly, the state and action spaces are𝒳=\{0,1,…,C\}\\mathcal\{X\}=\\\{0,1,\\ldots,C\\\}and𝒜=\{0,1,…,S\}\\mathcal\{A\}=\\\{0,1,\\ldots,S\\\}, respectively\. Activating more servers reduces congestion but increases operating costs, and the optimal choice depends on both the queue length and the remaining time\. The number of active servers may be changed at any time, including periods with no customer arrival or service completion\. This is in contrast to SMDPs where actions are restricted to event epochs or jump times\.

Customers arrive according to a nonhomogeneous Poisson process with rateλ⁡\(t\)\\lambda\(t\), and each busy server completes service at rateμ⁡\(t\)\\mu\(t\)\. Thus, the transition rates are given by

λ⁡\(x\+1∣t,x,a\)\\displaystyle\\lambda\(x\+1\\mid t,x,a\)=λ\(t\)𝟏\{x<C\},λ\(x−1∣t,x,a\)=\(x∧a\)μ\(t\)𝟏\{x\>0\},\\displaystyle=\\lambda\(t\)\\boldsymbol\{1\}\_\{\\\{x<C\\\}\},\\qquad\\lambda\(x\-1\\mid t,x,a\)=\(x\\wedge a\)\\mu\(t\)\\boldsymbol\{1\}\_\{\\\{x\>0\\\}\},andλ\(x∣t,x,a\)=−λ\(t\)𝟏\{x<C\}−\(x∧a\)μ\(t\)𝟏\{x\>0\}\\lambda\(x\\mid t,x,a\)=\-\\lambda\(t\)\\boldsymbol\{1\}\_\{\\\{x<C\\\}\}\-\(x\\wedge a\)\\mu\(t\)\\boldsymbol\{1\}\_\{\\\{x\>0\\\}\}\. Arrivals that find the system at capacity are blocked\. LetK1,K2,K3\>0K\_\{1\},\\,K\_\{2\},\\,K\_\{3\}\>0denote the operating cost per active server per unit time, the holding cost per customer per unit time, and the terminal cost per customer remaining at timeTT, respectively\. That is, the running reward rate and the terminal reward are given as follows:

r⁡\(t,x,a\)=−K1​a−K2​x,h⁡\(x\)=−K3​x\.\\displaystyle r\(t,x,a\)=\-K\_\{1\}a\-K\_\{2\}x,\\qquad h\(x\)=\-K\_\{3\}x\.The jump reward is assume to be zero\. These dynamics define a finite\-horizon CTJMDP\. In the RL setting, the decision maker knows the cost coefficientsK1K\_\{1\},K2K\_\{2\}, andK3K\_\{3\}, but not the arrival rateλ⁡\(t\)\\lambda\(t\)or the service rateμ⁡\(t\)\\mu\(t\)\. This finite\-horizon queueing control problem, featuring non\-stationary optimal policies that depend on both system states and time, has not been explored from a model\-free RL perspective to the best of our knowledge\. We apply our proposed RL algorithm to learn the optimal policy\.

To demonstrate the applicability of Algorithm[1](https://arxiv.org/html/2608.20680#alg1)to this problem, we consider a small instance withT=10T=10andC=S=10C=S=10\. The arrival and service rates are specified asλ⁡\(t\)=1\.5\+0\.5​sin⁡\(2​π​tT\)\\lambda\(t\)=1\.5\+0\.5\\sin\(\\frac\{2\\pi t\}\{T\}\)andμ⁡\(t\)=0\.5\+0\.1​tT\\mu\(t\)=0\.5\+0\.1\\frac\{t\}\{T\}\. We set the cost coefficients toK1=K2=K3=1K\_\{1\}=K\_\{2\}=K\_\{3\}=1\.

For benchmarking purposes, we construct a discrete\-time approximation of the continuous\-time control problem and solve the resulting dynamic programming problem\. Specifically, given a smallΔ​t′\>0\\Delta t^\{\\prime\}\>0, letK′≜⌈TΔ​t′⌉K^\{\\prime\}\\triangleq\\lceil\\frac\{T\}\{\\Delta t^\{\\prime\}\}\\rceil,tk′≜k​Δ​t′t\_\{k\}^\{\\prime\}\\triangleq k\\Delta t^\{\\prime\}fork=0,…,K′−1k=0,\\ldots,K^\{\\prime\}\-1, andtK′≜Tt\_\{K^\{\\prime\}\}\\triangleq T\. In each period\[tk′,tk\+1′\)\[t\_\{k\}^\{\\prime\},t\_\{k\+1\}^\{\\prime\}\), there is a probability ofλ\(tk\+1′\)𝟏\{x≤C−1\}Δs\\lambda\(t\_\{k\+1\}^\{\\prime\}\)\\boldsymbol\{1\}\_\{\\\{x\\leq C\-1\\\}\}\\Delta sthat a customer arrivals, and a probability of\(x∧a\)μ\(tk\+1′\)𝟏\{x≥1\}Δs\(x\\wedge a\)\\mu\(t\_\{k\+1\}^\{\\prime\}\)\\boldsymbol\{1\}\_\{\\\{x\\geq 1\\\}\}\\Delta sthat a customer departs, given the queue length attk′t\_\{k\}^\{\\prime\}isxxand the number of active servers in this period isaa\. With the remaining probability, the system state does not change\. The dynamic programming \(DP\) problem corresponding to the above discrete\-time model is given by

VΔ​t′∗​\(tk′,x\)\\displaystyle V\_\{\\Delta t^\{\\prime\}\}^\{\*\}\(t\_\{k\}^\{\\prime\},x\)=\\displaystyle=VΔ​t′∗\(tk\+1′,x\)\+λ\(tk\+1′\)𝟏\{x≤C−1\}Δt′\(VΔ​t′∗\(tk\+1′,x\+1\)−VΔ​t′∗\(tk\+1′,x\)\)−K2xΔt′\\displaystyle V\_\{\\Delta t^\{\\prime\}\}^\{\*\}\(t\_\{k\+1\}^\{\\prime\},x\)\+\\lambda\(t\_\{k\+1\}^\{\\prime\}\)\\boldsymbol\{1\}\_\{\\\{x\\leq C\-1\\\}\}\\Delta t^\{\\prime\}\(V\_\{\\Delta t^\{\\prime\}\}^\{\*\}\(t\_\{k\+1\}^\{\\prime\},x\+1\)\-V\_\{\\Delta t^\{\\prime\}\}^\{\*\}\(t\_\{k\+1\}^\{\\prime\},x\)\)\-K\_\{2\}x\\Delta t^\{\\prime\}\+maxa∈𝒜\{−K1aΔt′\+\(x∧a\)μ\(tk\+1′\)𝟏\{x≥1\}Δt′\(VΔ​t′∗\(tk\+1′,x−1\)−VΔ​t′∗\(tk\+1′,x\)\)\},\\displaystyle\+\\max\_\{a\\in\\mathcal\{A\}\}\\\{\-K\_\{1\}a\\Delta t^\{\\prime\}\+\(x\\wedge a\)\\mu\(t\_\{k\+1\}^\{\\prime\}\)\\boldsymbol\{1\}\_\{\\\{x\\geq 1\\\}\}\\Delta t^\{\\prime\}\(V\_\{\\Delta t^\{\\prime\}\}^\{\*\}\(t\_\{k\+1\}^\{\\prime\},x\-1\)\-V\_\{\\Delta t^\{\\prime\}\}^\{\*\}\(t\_\{k\+1\}^\{\\prime\},x\)\)\\\},VΔ​t′∗​\(T,x\)\\displaystyle V\_\{\\Delta t^\{\\prime\}\}^\{\*\}\(T,x\)=\\displaystyle=−K3​x\.\\displaystyle\-K\_\{3\}x\.\(21\)ForΔ​t′\\Delta t^\{\\prime\}sufficiently small, the valueVΔ​t′∗​\(0,0\)V\_\{\\Delta t^\{\\prime\}\}^\{\*\}\(0,0\)obtained from solving the DP problem \([A](https://arxiv.org/html/2608.20680#A1.Ex41)\) is expected to provide a reliable approximation to the optimal valueV∗​\(0,0\)V^\{\*\}\(0,0\)under the continuous time model\. In this example, withΔ​t′\\Delta t^\{\\prime\}set to0\.00010\.0001, we obtainV0\.0001∗​\(0,0\)=−30\.796V^\{\*\}\_\{0\.0001\}\(0,0\)=\-30\.796\.

Figure 4:Average reward of Algorithm[1](https://arxiv.org/html/2608.20680#alg1)over episodes for queue control0246810×104\\times 10^\{4\}−65\-65−50\-50−30\.796\-30\.796EpisodeAverage RewardDPAlgorithm[1](https://arxiv.org/html/2608.20680#alg1)To implement Algorithm[1](https://arxiv.org/html/2608.20680#alg1)for this instance, we employ two separate fully connected neural networks,fcriticθf\_\{\\mathrm\{critic\}\}^\{\\theta\}andfactorψf\_\{\\mathrm\{actor\}\}^\{\\psi\}, to construct the approximatorsJθJ^\{\\theta\}andqψq^\{\\psi\}, respectively\. Both the critic and actor networks are configured with two hidden layers of width88and use the ReLU activation function\. The parameters of each network are updated using a separate Adam optimizer\. Both networks take a two\-dimensional time\-state vector as input\. The critic network outputs a scalar, and the value\-function approximator is defined by

Jθ​\(t,x\)\\displaystyle J^\{\\theta\}\(t,x\)=fcriticθ\(t,x\)𝟏\{t<T\}−K3x𝟏\{t=T\}\.\\displaystyle=f\_\{\\mathrm\{critic\}\}^\{\\theta\}\(t,x\)\\boldsymbol\{1\}\_\{\\\{t<T\\\}\}\-K\_\{3\}x\\boldsymbol\{1\}\_\{\\\{t=T\\\}\}\.This construction ensures thatJθ​\(T,x\)=−K3​x=h⁡\(x\)J^\{\\theta\}\(T,x\)=\-K\_\{3\}x=h\(x\)for everyx∈𝒳x\\in\\mathcal\{X\}\. The actor network outputs an\(S\+1\)\(S\+1\)\-dimensional vector, with itsaa\-th component denoted byfactorψ​\(t,x\)​\[a\]f\_\{\\mathrm\{actor\}\}^\{\\psi\}\(t,x\)\[a\]\. The policy andqq\-function approximator are defined by

𝝅ψ​\(a∣t,x\)\\displaystyle\\boldsymbol\{\\pi\}^\{\\psi\}\(a\\mid t,x\)=exp⁡\{factorψ​\(t,x\)​\[a\+1\]/γ\}∑a′=0Sexp⁡\{factorψ​\(t,x\)​\[a′\+1\]/γ\},qψ​\(t,x,a\)=γ​log⁡𝝅ψ​\(a∣t,x\)\.\\displaystyle=\\frac\{\\exp\\\{f\_\{\\mathrm\{actor\}\}^\{\\psi\}\(t,x\)\[a\+1\]/\\gamma\\\}\}\{\\sum\_\{a^\{\\prime\}=0\}^\{S\}\\exp\\\{f\_\{\\mathrm\{actor\}\}^\{\\psi\}\(t,x\)\[a^\{\\prime\}\+1\]/\\gamma\\\}\},\\qquad q^\{\\psi\}\(t,x,a\)=\\gamma\\log\\boldsymbol\{\\pi\}^\{\\psi\}\(a\\mid t,x\)\.This construction ensures thatqψq^\{\\psi\}satisfies the normalization condition in \([13](https://arxiv.org/html/2608.20680#S4.E13)\)\.

BesidesJθJ^\{\\theta\}andqψq^\{\\psi\}, the remaining experimental parameters are configured as: initial statex0=0x\_\{0\}=0, time stepΔ​t=0\.01\\Delta t=0\.01, learning ratesαθ=1×10−3\\alpha\_\{\\theta\}=1\\times 10^\{\-3\},αϕ=2×10−6\\alpha\_\{\\phi\}=2\\times 10^\{\-6\}and temperature parameterγ=1×10−2\\gamma=1\\times 10^\{\-2\}\.

During training, we periodically evaluate the current policy𝝅\\boldsymbol\{\\pi\}by estimatingV⁡\(0,0,𝝅\)V\(0,0;\\boldsymbol\{\\pi\}\)from the average reward over10,00010,000independently simulated sample paths initialized atx0=0x\_\{0\}=0\. Figure[4](https://arxiv.org/html/2608.20680#A1.F4)reports the estimated policy values over the learning process and shows the benchmarkV0\.0001∗​\(0,0\)=−30\.796V\_\{0\.0001\}^\{\*\}\(0,0\)=\-30\.796as a dashed line\. The policy values increase and stabilize near the benchmark, with the final learned policy achieving an average reward of−31\.272\-31\.272\. This corresponds to a small performance gap of0\.4760\.476, indicating that the learned policy closely approximates the optimal policy\.

## Appendix BProofs of Statements

This section provides proofs of the main theoretical results\. We first state a lemma on entropy\-regularized maximization, which is needed for the proofs of Proposition[1](https://arxiv.org/html/2608.20680#Thmproposition1)and Theorem[2](https://arxiv.org/html/2608.20680#Thmtheorem2)\. We then proceed to prove Proposition[1](https://arxiv.org/html/2608.20680#Thmproposition1), Theorem[1](https://arxiv.org/html/2608.20680#Thmtheorem1), and Theorem[2](https://arxiv.org/html/2608.20680#Thmtheorem2)in turn\.

###### Lemma 1\.

Letγ\>0\\gamma\>0, and a measurable functionq^:𝕄↦ℝ\\hat\{q\}:\\mathbb\{M\}\\mapsto\\mathbb\{R\}with∫𝒜exp⁡\{1γ​q^​\(t,x,a\)\}​μ​\(𝑑a\)<∞\\int\_\{\\mathcal\{A\}\}\\exp\{\\\{\\frac\{1\}\{\\gamma\}\\hat\{q\}\(t,x,a\)\\\}\\mu\(\\mathrm\{d\}a\)\}<\\inftyfor each\(t,x\)∈\[0,T\]×𝒳\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\}\. Define the policy𝛑∗∈𝚷\\boldsymbol\{\\pi\}^\{\*\}\\in\\boldsymbol\{\\Pi\}by

𝝅∗​\(a∣t,x\)≜exp⁡\{1γ​q^​\(t,x,a\)\}∫𝒜exp⁡\{1γ​q^​\(t,x,a′\)\}​μ​\(d​a′\),for​\(t,x,a\)∈𝕄\.\\displaystyle\\boldsymbol\{\\pi\}^\{\*\}\(a\\mid t,x\)\\triangleq\\frac\{\\exp\\\{\\frac\{1\}\{\\gamma\}\\hat\{q\}\(t,x,a\)\\\}\}\{\\int\_\{\\mathcal\{A\}\}\\exp\\\{\\frac\{1\}\{\\gamma\}\\hat\{q\}\(t,x,a^\{\\prime\}\)\\\}\\mu\(\\mathrm\{d\}a^\{\\prime\}\)\},\\quad\\text\{ for \}\(t,x,a\)\\in\\mathbb\{M\}\.Then,𝛑∗\\boldsymbol\{\\pi\}^\{\*\}is the unique maximizer of the following problem

max⁡∫𝒜𝝅∈𝚷⁡\[q^​\(t,x,a\)−γ​log⁡𝝅⁡\(a∣t,x\)\]​𝝅​\(a∣t,x\)​μ​\(𝑑a\)\.\\displaystyle\\max\_\{\\boldsymbol\{\\pi\}\\in\\boldsymbol\{\\Pi\}\}\\int\_\{\\mathcal\{A\}\}\[\\hat\{q\}\(t,x,a\)\-\\gamma\\log\\boldsymbol\{\\pi\}\(a\\mid t,x\)\]\\boldsymbol\{\\pi\}\(a\\mid t,x\)\\mu\(\\mathrm\{d\}a\)\.

###### Proof\.

The result follows from the same argument as Lemma 13 in[20](https://arxiv.org/html/2608.20680#bib.bib13), with Lebesgue measure replaced by the reference measureμ\\mu\. ∎

### B\.1Proof of Proposition[1](https://arxiv.org/html/2608.20680#Thmproposition1)

Letρ≜2​L\+b\+c\+1\\rho\\triangleq 2L\+b\+c\+1withL≜supx∈𝒳λ∗​\(x\)L\\triangleq\\sup\_\{x\\in\\mathcal\{X\}\}\\lambda^\{\*\}\(x\)andbb,ccas in Assumption[1](https://arxiv.org/html/2608.20680#Thmassumption1)\. We define an operator𝒢\\mathcal\{G\}onBw​\(\[0,T\]×𝒳\)B\_\{w\}\(\[0,T\]\\times\\mathcal\{X\}\)as follows: for eachψ∈Bw​\(\[0,T\]×𝒳\)\\psi\\in B\_\{w\}\(\[0,T\]\\times\\mathcal\{X\}\)and\(t,x\)∈\[0,T\]×𝒳\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\},

𝒢​ψ​\(t,x\)≜\\displaystyle\\mathcal\{G\}\\psi\(t,x\)\\triangleq\{\}eρ​t​∫tTsup𝝅∈𝚷∫𝒜\{R⁡\(s,x,a\)\+e−ρ​s​∑y∈𝒳ψ⁡\(s,y\)​λ​\(y∣s,x,a\)−γ​log⁡𝝅⁡\(a∣s,x\)\}​𝝅​\(𝑑a∣s,x\)​𝑑s\\displaystyle e^\{\\rho t\}\\int\_\{t\}^\{T\}\\sup\_\{\\boldsymbol\{\\pi\}\\in\\boldsymbol\{\\Pi\}\}\\int\_\{\\mathcal\{A\}\}\\\{R\(s,x,a\)\+e^\{\-\\rho s\}\\sum\_\{y\\in\\mathcal\{X\}\}\\psi\(s,y\)\\lambda\(y\\mid s,x,a\)\-\\gamma\\log\\boldsymbol\{\\pi\}\(a\\mid s,x\)\\\}\\boldsymbol\{\\pi\}\(\\mathrm\{d\}a\\mid s,x\)\\mathrm\{d\}s\+eρ​t​h​\(x\)\.\\displaystyle\+e^\{\\rho t\}h\(x\)\.\(22\)We need to verify the operator𝒢\\mathcal\{G\}is well\-defined\. Indeed, by Assumption[1](https://arxiv.org/html/2608.20680#Thmassumption1)and Definition[2](https://arxiv.org/html/2608.20680#Thmdefinition2)\(ii\), we have

\|∫𝒜\{R⁡\(s,x,a\)\+e−ρ​s​∑y∈𝒳ψ⁡\(s,y\)​λ​\(y∣s,x,a\)−γ​log⁡𝝅⁡\(a∣s,x\)\}​𝝅​\(𝑑a∣s,x\)\|​𝑑s\\displaystyle\\bigg\|\\int\_\{\\mathcal\{A\}\}\\\{R\(s,x,a\)\+e^\{\-\\rho s\}\\sum\_\{y\\in\\mathcal\{X\}\}\\psi\(s,y\)\\lambda\(y\\mid s,x,a\)\-\\gamma\\log\\boldsymbol\{\\pi\}\(a\\mid s,x\)\\\}\\boldsymbol\{\\pi\}\(\\mathrm\{d\}a\\mid s,x\)\\bigg\|\\mathrm\{d\}s≤\\displaystyle\\leq\{\}\(∫𝒜\{\|R\(s,x,a\)\|\+∑y∈𝒳\|ψ\(s,y\)\|\|λ\(y∣s,x,a\)\|\}𝝅\(da∣s,x\)\+γ\|∫𝒜−log𝝅\(a∣s,x\)𝝅\(da∣s,x\)\|\)ds\\displaystyle\\bigg\(\\int\_\{\\mathcal\{A\}\}\\\{\|R\(s,x,a\)\|\+\\sum\_\{y\\in\\mathcal\{X\}\}\|\\psi\(s,y\)\|\|\\lambda\(y\\mid s,x,a\)\|\\\}\\boldsymbol\{\\pi\}\(\\mathrm\{d\}a\\mid s,x\)\+\\gamma\\bigg\|\\int\_\{\\mathcal\{A\}\}\-\\log\\boldsymbol\{\\pi\}\(a\\mid s,x\)\\boldsymbol\{\\pi\}\(\\mathrm\{d\}a\\mid s,x\)\\bigg\|\\bigg\)\\mathrm\{d\}s≤\\displaystyle\\leq\{\}\[M1\+‖ψ‖w​\(c\+b\+2​L\)\+γ​M2\]​w​\(x\),∀\(t,x\)∈\[0,T\]×𝒳,\\displaystyle\[M\_\{1\}\+\\\|\\psi\\\|\_\{w\}\(c\+b\+2L\)\+\\gamma M\_\{2\}\]w\(x\),\\quad\\forall\\,\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\},\(23\)which implies that the integrand on the right\-hand side of \([B\.1](https://arxiv.org/html/2608.20680#A2.Ex47)\) is Lebesgue integrable with respect tott, and thus𝒢​ψ​\(t,x\)\\mathcal\{G\}\\psi\(t,x\)defined in \([B\.1](https://arxiv.org/html/2608.20680#A2.Ex47)\) is absolutely continuous and Borel measurable\. Moreover, by combining \([B\.1](https://arxiv.org/html/2608.20680#A2.Ex47)\), \([23](https://arxiv.org/html/2608.20680#A2.E23)\) and the fact that\|h⁡\(x\)\|≤M1​w​\(x\)\|h\(x\)\|\\leq M\_\{1\}w\(x\)for allx∈𝒳x\\in\\mathcal\{X\}, we have

\|𝒢​ψ​\(t,x\)\|≤eρ​T​\[M1\+M1​T\+‖ψ‖w​T​\(c\+b\+2​L\)\+γ​M2​T\]​w​\(x\),∀\(t,x\)∈\[0,T\]×𝒳,\\displaystyle\|\\mathcal\{G\}\\psi\(t,x\)\|\\leq e^\{\\rho T\}\[M\_\{1\}\+M\_\{1\}T\+\\\|\\psi\\\|\_\{w\}T\(c\+b\+2L\)\+\\gamma M\_\{2\}T\]w\(x\),\\quad\\forall\\,\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\},which implies that‖𝒢​ψ‖w<∞\\\|\\mathcal\{G\}\\psi\\\|\_\{w\}<\\infty\. Hence,𝒢​ψ∈Bw​\(\[0,T\]×𝒳\)\\mathcal\{G\}\\psi\\in B\_\{w\}\(\[0,T\]\\times\\mathcal\{X\}\), and𝒢\\mathcal\{G\}is a well\-defined operator onBw​\(\[0,T\]×𝒳\)B\_\{w\}\(\[0,T\]\\times\\mathcal\{X\}\)\.

Next, we establish that𝒢\\mathcal\{G\}is a contraction operator\. For anyψ1,ψ2∈Bw​\(\[0,T\]×𝒳\)\\psi\_\{1\},\\psi\_\{2\}\\in B\_\{w\}\(\[0,T\]\\times\\mathcal\{X\}\), it follows from \([B\.1](https://arxiv.org/html/2608.20680#A2.Ex47)\) and∑y∈𝒳λ⁡\(y∣t,x,a\)=0\\sum\_\{y\\in\\mathcal\{X\}\}\\lambda\(y\\mid t,x,a\)=0that

\|𝒢​ψ1​\(t,x\)−𝒢​ψ2​\(t,x\)\|\\displaystyle\|\\mathcal\{G\}\\psi\_\{1\}\(t,x\)\-\\mathcal\{G\}\\psi\_\{2\}\(t,x\)\|≤\\displaystyle\\leq\{\}eρ​t​∫tTe−ρ​s⋅sup𝝅∈Π∫𝒜\(∑y∈𝒳\|ψ1​\(s,y\)−ψ2​\(t,y\)\|​\|λ⁡\(y∣s,x,a\)\|\)​𝝅​\(𝑑a∣s,x\)​𝑑s\\displaystyle e^\{\\rho t\}\\int\_\{t\}^\{T\}e^\{\-\\rho s\}\\cdot\\sup\_\{\\boldsymbol\{\\pi\}\\in\\Pi\}\\int\_\{\\mathcal\{A\}\}\\bigg\(\\sum\_\{y\\in\\mathcal\{X\}\}\|\\psi\_\{1\}\(s,y\)\-\\psi\_\{2\}\(t,y\)\|\|\\lambda\(y\\mid s,x,a\)\|\\bigg\)\\boldsymbol\{\\pi\}\(\\mathrm\{d\}a\\mid s,x\)\\mathrm\{d\}s≤\\displaystyle\\leq\{\}eρ​t​∫tTe−ρ​s​‖ψ1−ψ2‖w⋅sup𝝅∈Π∫𝒜\(∑y∈𝒳w⁡\(y\)​λ​\(y∣s,x,a\)\+2​w​\(x\)​λ​\(s,x,a\)\)​𝝅​\(𝑑a∣s,x\)​𝑑s\\displaystyle e^\{\\rho t\}\\int\_\{t\}^\{T\}e^\{\-\\rho s\}\\\|\\psi\_\{1\}\-\\psi\_\{2\}\\\|\_\{w\}\\cdot\\sup\_\{\\boldsymbol\{\\pi\}\\in\\Pi\}\\int\_\{\\mathcal\{A\}\}\\bigg\(\\sum\_\{y\\in\\mathcal\{X\}\}w\(y\)\\lambda\(y\\mid s,x,a\)\+2w\(x\)\\lambda\(s,x,a\)\\bigg\)\\boldsymbol\{\\pi\}\(\\mathrm\{d\}a\\mid s,x\)\\mathrm\{d\}s≤\\displaystyle\\leq\{\}eρ​t​∫tTe−ρ​s​‖ψ1−ψ2‖w​\[c​w​\(x\)\+b\+2​L​w​\(x\)\]​𝑑s\\displaystyle e^\{\\rho t\}\\int\_\{t\}^\{T\}e^\{\-\\rho s\}\\\|\\psi\_\{1\}\-\\psi\_\{2\}\\\|\_\{w\}\[cw\(x\)\+b\+2Lw\(x\)\]\\mathrm\{d\}s≤\\displaystyle\\leq\{\}2​L\+b\+cρ​\[1−e−β⁡\(T−t\)\]​‖ψ1−ψ2‖w​w​\(x\)\\displaystyle\\frac\{2L\+b\+c\}\{\\rho\}\[1\-e^\{\-\\beta\(T\-t\)\}\]\\\|\\psi\_\{1\}\-\\psi\_\{2\}\\\|\_\{w\}w\(x\)≤\\displaystyle\\leq\{\}2​L\+b\+cρ​‖ψ1−ψ2‖w​w​\(x\)\.\\displaystyle\\frac\{2L\+b\+c\}\{\\rho\}\\\|\\psi\_\{1\}\-\\psi\_\{2\}\\\|\_\{w\}w\(x\)\.Hence, we obtain

‖𝒢​ψ1−𝒢​ψ2‖w≤2​L\+b\+c2​L\+b\+c\+1​‖ψ1−ψ2‖w\.\\displaystyle\\\|\\mathcal\{G\}\\psi\_\{1\}\-\\mathcal\{G\}\\psi\_\{2\}\\\|\_\{w\}\\leq\\frac\{2L\+b\+c\}\{2L\+b\+c\+1\}\\\|\\psi\_\{1\}\-\\psi\_\{2\}\\\|\_\{w\}\.Since2​L\+b\+c2​L\+b\+c\+1<1\\frac\{2L\+b\+c\}\{2L\+b\+c\+1\}<1,𝒢\\mathcal\{G\}is a contraction operator on the Banach spaceBw​\(\[0,T\]×𝒳\)B\_\{w\}\(\[0,T\]\\times\\mathcal\{X\}\)\. Letψ∗∈Bw​\(\[0,T\]×𝒳\)\\psi^\{\*\}\\in B\_\{w\}\(\[0,T\]\\times\\mathcal\{X\}\)denote the unique fixed point of𝒢\\mathcal\{G\}\. That is,ψ∗\\psi^\{\*\}satisfies

ψ∗​\(t,x\)=\\displaystyle\\psi^\{\*\}\(t,x\)=\{\}eρ​t​∫tTsup𝝅∈Π∫𝒜\{R⁡\(s,x,a\)\+e−ρ​s​∑y∈𝒳ψ∗​\(s,y\)​λ​\(y∣s,x,a\)−γ​log⁡𝝅⁡\(a∣s,x\)\}​𝝅​\(𝑑a∣s,x\)​𝑑s\\displaystyle e^\{\\rho t\}\\int\_\{t\}^\{T\}\\sup\_\{\\boldsymbol\{\\pi\}\\in\\Pi\}\\int\_\{\\mathcal\{A\}\}\\\{R\(s,x,a\)\+e^\{\-\\rho s\}\\sum\_\{y\\in\\mathcal\{X\}\}\\psi^\{\*\}\(s,y\)\\lambda\(y\\mid s,x,a\)\-\\gamma\\log\\boldsymbol\{\\pi\}\(a\\mid s,x\)\\\}\\boldsymbol\{\\pi\}\(\\mathrm\{d\}a\\mid s,x\)\\mathrm\{d\}s\+eρ​t​h​\(x\)\.\\displaystyle\+e^\{\\rho t\}h\(x\)\.\(24\)Defineφ⁡\(t,x\)≜e−ρ​t​ψ∗​\(t,x\)\\varphi\(t,x\)\\triangleq e^\{\-\\rho t\}\\psi^\{\*\}\(t,x\)for all\(t,x\)∈\[0,T\]×𝒳\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\}\. By construction ofφ\\varphiand \([B\.1](https://arxiv.org/html/2608.20680#A2.Ex58)\), we see thatφ∈Bw​\(\[0,T\]×𝒳\)\\varphi\\in B\_\{w\}\(\[0,T\]\\times\\mathcal\{X\}\), andφ⁡\(t,x\)\\varphi\(t,x\)is differentiable inttalmost everywhere for each fixedx∈𝒳x\\in\\mathcal\{X\}and satisfies the differential equation \([9](https://arxiv.org/html/2608.20680#S3.E9)\)\. Then, it follows from \([9](https://arxiv.org/html/2608.20680#S3.E9)\) and \([23](https://arxiv.org/html/2608.20680#A2.E23)\) that∂φ∂t​\(t,x\)\\frac\{\\partial\\varphi\}\{\\partial t\}\(t,x\)is universally measurable andww\-bounded on\[0,T\]×𝒳\[0,T\]\\times\\mathcal\{X\}\. Thusφ∈Cw1,0​\(\[0,T\]×𝒳\)\\varphi\\in C^\{1,0\}\_\{w\}\(\[0,T\]\\times\\mathcal\{X\}\)\. Hence, we have established the existence of a solution to \([9](https://arxiv.org/html/2608.20680#S3.E9)\)\.

On the other hand, for any𝝅∈𝚷\\boldsymbol\{\\pi\}\\in\\boldsymbol\{\\Pi\}and any solutionφ∈Cw1,0​\(\[0,T\]×𝒳\)\\varphi\\in C^\{1,0\}\_\{w\}\(\[0,T\]\\times\\mathcal\{X\}\)to \([9](https://arxiv.org/html/2608.20680#S3.E9)\), we first show thatJ⁡\(t,x,𝝅\)≤φ⁡\(t,x\)J\(t,x;\\boldsymbol\{\\pi\}\)\\leq\\varphi\(t,x\)\. Sinceφ\\varphisatisfies Equation \([9](https://arxiv.org/html/2608.20680#S3.E9)\), we have

∂φ∂t​\(t,x\)\+∫𝒜\{H⁡\(t,x,a,φ⁡\(⋅,⋅\)\)−γ​log⁡𝝅⁡\(a∣t,x\)\}​𝝅​\(𝑑a∣t,x\)≤0\.\\displaystyle\\frac\{\\partial\\varphi\}\{\\partial t\}\(t,x\)\+\\int\_\{\\mathcal\{A\}\}\\\{H\(t,x,a,\\varphi\(\\cdot,\\cdot\)\)\-\\gamma\\log\\boldsymbol\{\\pi\}\(a\\mid t,x\)\\\}\\boldsymbol\{\\pi\}\(\\mathrm\{d\}a\\mid t,x\)\\leq 0\.\(25\)Then, by combining the definition ofJ⁡\(t,x,𝝅\)J\(t,x;\\boldsymbol\{\\pi\}\)in \([7](https://arxiv.org/html/2608.20680#S3.E7)\), inequality \([25](https://arxiv.org/html/2608.20680#A2.E25)\) and Theorem 3\.1 in[15](https://arxiv.org/html/2608.20680#bib.bib7), we derive that

J⁡\(t,x,𝝅\)\\displaystyle J\(t,x;\\boldsymbol\{\\pi\}\)=𝔼⁡\[∫tT∫𝒜\[R⁡\(s,X~s−𝝅,a\)−γ​log⁡𝝅⁡\(a∣s,X~s−𝝅\)\]​𝝅​\(𝑑a∣s,X~s−𝝅\)​𝑑s\+h⁡\(X~T𝝅\)∣X~t𝝅=x\]\\displaystyle=\\mathbb\{E\}\\bigg\[\\int\_\{t\}^\{T\}\\int\_\{\\mathcal\{A\}\}\[R\(s,\\tilde\{X\}\_\{s\-\}^\{\\boldsymbol\{\\pi\}\},a\)\-\\gamma\\log\\boldsymbol\{\\pi\}\(a\\mid s,\\tilde\{X\}\_\{s\-\}^\{\\boldsymbol\{\\pi\}\}\)\]\\boldsymbol\{\\pi\}\(\\mathrm\{d\}a\\mid s,\\tilde\{X\}\_\{s\-\}^\{\\boldsymbol\{\\pi\}\}\)\\mathrm\{d\}s\+h\(\\tilde\{X\}\_\{T\}^\{\\boldsymbol\{\\pi\}\}\)\\mid\\tilde\{X\}\_\{t\}^\{\\boldsymbol\{\\pi\}\}=x\\bigg\]≤𝔼⁡\[∫tT\(−∂φ∂s​\(s,X~s−𝝅\)−∑y∈𝒳φ⁡\(s,y\)​λ𝝅​\(y∣s,X~s−𝝅\)\)​𝑑s\+h⁡\(X~T𝝅\)∣X~t𝝅=x\]\\displaystyle\\leq\\mathbb\{E\}\\bigg\[\\int\_\{t\}^\{T\}\\bigg\(\-\\frac\{\\partial\\varphi\}\{\\partial s\}\(s,\\tilde\{X\}\_\{s\-\}^\{\\boldsymbol\{\\pi\}\}\)\-\\sum\_\{y\\in\\mathcal\{X\}\}\\varphi\(s,y\)\\lambda^\{\\boldsymbol\{\\pi\}\}\(y\\mid s,\\tilde\{X\}\_\{s\-\}^\{\\boldsymbol\{\\pi\}\}\)\\bigg\)\\mathrm\{d\}s\+h\(\\tilde\{X\}\_\{T\}^\{\\boldsymbol\{\\pi\}\}\)\\mid\\tilde\{X\}\_\{t\}^\{\\boldsymbol\{\\pi\}\}=x\\bigg\]=φ⁡\(t,x\),\\displaystyle=\\varphi\(t,x\),whereλ𝝅​\(y∣t,x\)\\lambda^\{\\boldsymbol\{\\pi\}\}\(y\\mid t,x\)is as defined in \([5](https://arxiv.org/html/2608.20680#S3.E5)\)\. Thus, we haveJ∗​\(t,x\)≜sup𝝅∈𝚷J⁡\(t,x,𝝅\)≤φ⁡\(t,x\)J^\{\*\}\(t,x\)\\triangleq\\sup\_\{\\boldsymbol\{\\pi\}\\in\\boldsymbol\{\\Pi\}\}J\(t,x;\\boldsymbol\{\\pi\}\)\\leq\\varphi\(t,x\)for all\(t,x\)∈\[0,T\]×𝒳\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\}\. Moreover, it follows from Lemma[1](https://arxiv.org/html/2608.20680#Thmlemma1)that the equality in \([25](https://arxiv.org/html/2608.20680#A2.E25)\) holds for policy𝝅∗\\boldsymbol\{\\pi\}^\{\*\}defined by

𝝅∗​\(a∣t,x\)=exp⁡\{1γ​H​\(t,x,a,φ⁡\(⋅,⋅\)\)\}∫𝒜exp⁡\{1γ​H​\(t,x,a′,φ⁡\(⋅,⋅\)\)\}​μ​\(d​a′\),for​\(t,x,a\)∈𝕄\.\\displaystyle\\boldsymbol\{\\pi\}^\{\*\}\(a\\mid t,x\)=\\frac\{\\exp\\\{\\frac\{1\}\{\\gamma\}H\(t,x,a,\\varphi\(\\cdot,\\cdot\)\)\\\}\}\{\\int\_\{\\mathcal\{A\}\}\\exp\\\{\\frac\{1\}\{\\gamma\}H\(t,x,a^\{\\prime\},\\varphi\(\\cdot,\\cdot\)\)\\\}\\mu\(\\mathrm\{d\}a^\{\\prime\}\)\},\\quad\\text\{ for \}\(t,x,a\)\\in\\mathbb\{M\}\.Hence we conclude thatJ∗​\(t,x\)=J⁡\(t,x,𝝅∗\)=φ⁡\(t,x\)J^\{\*\}\(t,x\)=J\(t,x;\\boldsymbol\{\\pi\}^\{\*\}\)=\\varphi\(t,x\)for all\(t,x\)∈\[0,T\]×𝒳\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\}\. Then, the optimal policy𝝅∗\\boldsymbol\{\\pi\}^\{\*\}admits the form given in \([10](https://arxiv.org/html/2608.20680#S3.E10)\)\.

### B\.2Proof of Theorem[1](https://arxiv.org/html/2608.20680#Thmtheorem1)

Let𝕊=\{t=t0<t1<⋯<tK=T\}\\mathbb\{S\}=\\\{t=t\_\{0\}<t\_\{1\}<\\cdots<t\_\{K\}=T\\\}be a time grid on\[t,T\]\[t,T\], and letAkA\_\{k\}denote the action selected attkt\_\{k\}fork=0,…,K−1k=0,\\ldots,K\-1\.

For anytk≤r1≤r2≤tk\+1t\_\{k\}\\leq r\_\{1\}\\leq r\_\{2\}\\leq t\_\{k\+1\},AkA\_\{k\}remains fixed on\[u,v\]\[u,v\]\. Therefore, conditional onXu𝝅,𝕊=zX\_\{u\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}=zandAk=aA\_\{k\}=a, the process\{Xs𝝅,𝕊:s∈\[u,v\]\}\\\{X\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}:s\\in\[u,v\]\\\}has the same law as the exploratory state process\{X~sδa:s∈\[u,v\]\}\\\{\\tilde\{X\}\_\{s\}^\{\\delta\_\{a\}\}:s\\in\[u,v\]\\\}under the deterministic Markov policyδa\\delta\_\{a\}, started fromX~uδa=z\\tilde\{X\}\_\{u\}^\{\\delta\_\{a\}\}=z\. Indeed, for anya∈𝒜a\\in\\mathcal\{A\}, the transition rates of\{X~sδa:s∈\[u,v\]\}\\\{\\tilde\{X\}\_\{s\}^\{\\delta\_\{a\}\}:s\\in\[u,v\]\\\}satisfy

λδa​\(y∣s,z\)=∫𝒜λ⁡\(y∣s,z,a′\)​δa​\(d​a′\)=λ⁡\(y∣s,z,a\),s∈\[u,v\]\.\\displaystyle\\lambda^\{\\delta\_\{a\}\}\(y\\mid s,z\)=\\int\_\{\\mathcal\{A\}\}\\lambda\(y\\mid s,z,a^\{\\prime\}\)\\,\\delta\_\{a\}\(\\mathrm\{d\}\{a^\{\\prime\}\}\)=\\lambda\(y\\mid s,z,a\),\\qquad s\\in\[u,v\]\.Applying Theorem 3\.1 of[15](https://arxiv.org/html/2608.20680#bib.bib7)to the process\{X~sδa:s∈\[u,v\]\}\\\{\\tilde\{X\}\_\{s\}^\{\\delta\_\{a\}\}:s\\in\[u,v\]\\\}with initial stateX~uδa=z\\tilde\{X\}\_\{u\}^\{\\delta\_\{a\}\}=zyields

𝔼u,zδa\[\\displaystyle\\mathbb\{E\}\_\{u,z\}^\{\\delta\_\{a\}\}\\bigg\[φ\(v,X~vδa\)−φ\(u,z\)−∫uvℒsaφ\(s,X~s−δa\)ds\]=0,\\displaystyle\\varphi\(v,\\tilde\{X\}\_\{v\}^\{\\delta\_\{a\}\}\)\-\\varphi\(u,z\)\-\\int\_\{u\}^\{v\}\\mathcal\{L\}\_\{s\}^\{a\}\\varphi\(s,\\tilde\{X\}\_\{s\-\}^\{\\delta\_\{a\}\}\)\\mathrm\{d\}s\\bigg\]=0,whereℒsa​φ​\(s,x\)≜∂φ∂s​\(s,x\)\+∑y∈𝒳φ⁡\(s,x\)​λ​\(y∣s,x,a\)\\mathcal\{L\}\_\{s\}^\{a\}\\varphi\(s,x\)\\triangleq\\frac\{\\partial\\varphi\}\{\\partial s\}\(s,x\)\+\\sum\_\{y\\in\\mathcal\{X\}\}\\varphi\(s,x\)\\lambda\(y\\mid s,x,a\)for alls∈\[u,v\]s\\in\[u,v\],x∈𝒳x\\in\\mathcal\{X\}, anda∈𝒜a\\in\\mathcal\{A\}\. By the conditional equality in law stated above, we have, for\{Xs𝝅,𝕊:s∈\[u,v\]\}\\\{X\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}:s\\in\[u,v\]\\\}that,

𝔼\[\\displaystyle\\mathbb\{E\}\\bigg\[φ\(v,Xv𝝅,𝕊\)−φ\(u,z\)−∫uvℒsaφ\(s,Xs−𝝅,𝕊\)ds∣Xu𝝅,𝕊=z,Ak=a\]=0\.\\displaystyle\\varphi\(v,X\_\{v\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\-\\varphi\(u,z\)\-\\int\_\{u\}^\{v\}\\mathcal\{L\}\_\{s\}^\{a\}\\varphi\(s,X\_\{s\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\\mathrm\{d\}s\\mid X\_\{u\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}=z,A\_\{k\}=a\\bigg\]=0\.Moreover, the conditional dynamics of\{Xs𝝅,𝕊:s∈\[u,v\]\}\\\{X\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}:s\\in\[u,v\]\\\}givenℱu𝕊\\mathcal\{F\}\_\{u\}^\{\\mathbb\{S\}\}, are completely determined by the current stateXu𝝅,𝕊X\_\{u\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}and the actionAkA\_\{k\}\. It follows that

𝔼\[\\displaystyle\\mathbb\{E\}\\bigg\[φ\(v,Xv𝝅,𝕊\)−φ\(u,Xu𝝅,𝕊\)−∫uvℒsAkφ\(s,Xs−𝝅,𝕊\)ds∣ℱu𝕊\]=0\.\\displaystyle\\varphi\(v,X\_\{v\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\-\\varphi\(u,X\_\{u\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\-\\int\_\{u\}^\{v\}\\mathcal\{L\}\_\{s\}^\{A\_\{k\}\}\\varphi\(s,X\_\{s\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\\mathrm\{d\}s\\mid\\mathcal\{F\}\_\{u\}^\{\\mathbb\{S\}\}\\bigg\]=0\.\(26\)Define

Msφ≜φ⁡\(s,Xs𝝅,𝕊\)−φ⁡\(t,x\)−∫tsℒrar𝝅,𝕊​φ​\(r,Xr−𝝅,𝕊\)​𝑑r,s∈\[t,T\]\.\\displaystyle M\_\{s\}^\{\\varphi\}\\triangleq\\varphi\(s,X\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\-\\varphi\(t,x\)\-\\int\_\{t\}^\{s\}\\mathcal\{L\}\_\{r\}^\{a\_\{r\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\}\\varphi\(r,X\_\{r\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\\,\\mathrm\{d\}r,\\qquad s\\in\[t,T\]\.\(27\)Under Assumption[1](https://arxiv.org/html/2608.20680#Thmassumption1)andφ∈Cw1,0​\(\[0,T\]×𝒳\)\\varphi\\in C\_\{w\}^\{1,0\}\(\[0,T\]\\times\\mathcal\{X\}\), the processMφM^\{\\varphi\}is integrable\. Sinceas𝝅,𝕊=Aka\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}=A\_\{k\}on\(tk,tk\+1\]\(t\_\{k\},t\_\{k\+1\}\], \([26](https://arxiv.org/html/2608.20680#A2.E26)\) implies that

𝔼⁡\[Mvφ−Muφ∣ℱu𝕊\]=0,\\displaystyle\\mathbb\{E\}\[M\_\{v\}^\{\\varphi\}\-M\_\{u\}^\{\\varphi\}\\mid\\mathcal\{F\}\_\{u\}^\{\\mathbb\{S\}\}\]=0,\(28\)whenevertk≤u≤v≤tk\+1t\_\{k\}\\leq u\\leq v\\leq t\_\{k\+1\}for somek∈\{0,…,K−1\}k\\in\\\{0,\\ldots,K\-1\\\}\.

Next, for anyt≤u≤v≤Tt\\leq u\\leq v\\leq T, insert all grid points in𝕊\\mathbb\{S\}that lie strictly betweenuuandvvand denote the resulting partition of\[u,v\]\[u,v\]by

u=σ0<σ1<⋯<σN=v\.u=\\sigma\_\{0\}<\\sigma\_\{1\}<\\cdots<\\sigma\_\{N\}=v\.Then each subinterval\[σn,σn\+1\]\[\\sigma\_\{n\},\\sigma\_\{n\+1\}\]is contained in some grid interval\[tk,tk\+1\]\[t\_\{k\},t\_\{k\+1\}\]\. By \([28](https://arxiv.org/html/2608.20680#A2.E28)\) and the tower property,

𝔼⁡\[Mvφ−Muφ∣ℱu𝕊\]=∑n=0N−1𝔼⁡\[𝔼⁡\[Mσn\+1φ−Mσnφ∣ℱσn𝕊\]∣ℱu𝕊\]=0\.\\displaystyle\\mathbb\{E\}\[M\_\{v\}^\{\\varphi\}\-M\_\{u\}^\{\\varphi\}\\mid\\mathcal\{F\}\_\{u\}^\{\\mathbb\{S\}\}\]=\\sum\_\{n=0\}^\{N\-1\}\\mathbb\{E\}\\left\[\\mathbb\{E\}\[M\_\{\\sigma\_\{n\+1\}\}^\{\\varphi\}\-M\_\{\\sigma\_\{n\}\}^\{\\varphi\}\\mid\\mathcal\{F\}\_\{\\sigma\_\{n\}\}^\{\\mathbb\{S\}\}\]\\mid\\mathcal\{F\}\_\{u\}^\{\\mathbb\{S\}\}\\right\]=0\.Hence,\{Msφ:s∈\[t,T\]\}\\\{M\_\{s\}^\{\\varphi\}:s\\in\[t,T\]\\\}is an\{ℱs𝕊\}s∈\[t,T\]\\\{\\mathcal\{F\}\_\{s\}^\{\\mathbb\{S\}\}\\\}\_\{s\\in\[t,T\]\}\-martingale\. In particular, for anys∈\[t,T\]s\\in\[t,T\],

0\\displaystyle 0=𝔼⁡\[MTφ−Msφ∣ℱs𝕊\]\\displaystyle=\\mathbb\{E\}\[M\_\{T\}^\{\\varphi\}\-M\_\{s\}^\{\\varphi\}\\mid\\mathcal\{F\}\_\{s\}^\{\\mathbb\{S\}\}\]=𝔼⁡\[φ⁡\(T,XT𝝅,𝕊\)−φ⁡\(s,Xs𝝅,𝕊\)−∫sTℒuau𝝅,𝕊​φ​\(u,Xu−𝝅,𝕊\)​𝑑u∣ℱs𝕊\]\.\\displaystyle=\\mathbb\{E\}\\left\[\\varphi\(T,X\_\{T\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\-\\varphi\(s,X\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\-\\int\_\{s\}^\{T\}\\mathcal\{L\}\_\{u\}^\{a\_\{u\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\}\\varphi\(u,X\_\{u\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\\mathrm\{d\}u\\mid\\mathcal\{F\}\_\{s\}^\{\\mathbb\{S\}\}\\right\]\.This yields the desired result\.

### B\.3Proof of Theorem[2](https://arxiv.org/html/2608.20680#Thmtheorem2)

To prove Theorem[2](https://arxiv.org/html/2608.20680#Thmtheorem2), we first establish the following auxiliary lemma\.

###### Lemma 2\.

Suppose Assumption[1](https://arxiv.org/html/2608.20680#Thmassumption1)holds with functionww\. Then, for any𝛑∈𝚷\\boldsymbol\{\\pi\}\\in\\boldsymbol\{\\Pi\}, initial time\-state pair\(t,x\)∈\[0,T\]×𝒳\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\}, and time grid𝕊=\{t=t0<⋯<tK=T\}\\mathbb\{S\}=\\\{t=t\_\{0\}<\\cdots<t\_\{K\}=T\\\}on\[t,T\]\[t,T\], the process

Msρ≜∑y∈𝒳∑z≠y∫\(t,s\]ρ⁡\(u,y,au𝝅,𝕊,z\)​d​Ny​z𝝅,𝕊​\(u\)−∫ts∑z≠Xu−𝝅,𝕊ρ⁡\(u,Xu−𝝅,𝕊,au𝝅,𝕊,z\)​λ​\(z∣u,Xu−𝝅,𝕊,au𝝅,𝕊\)​𝑑u,s∈\[t,T\]\.\\displaystyle M\_\{s\}^\{\\rho\}\\triangleq\\sum\_\{y\\in\\mathcal\{X\}\}\\sum\_\{z\\neq y\}\\int\_\{\(t,s\]\}\\rho\(u,y,a\_\{u\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},z\)\\,\\mathrm\{d\}N\_\{yz\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\(u\)\-\\int\_\{t\}^\{s\}\\sum\_\{z\\neq X\_\{u\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\}\\rho\(u,X\_\{u\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},a\_\{u\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},z\)\\lambda\(z\\mid u,X\_\{u\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},a\_\{u\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\\mathrm\{d\}u,\\quad s\\in\[t,T\]\.is an\{ℱs𝕊\}s∈\[t,T\]\\\{\\mathcal\{F\}\_\{s\}^\{\\mathbb\{S\}\}\\\}\_\{s\\in\[t,T\]\}\-martingale\.

###### Proof of Lemma[2](https://arxiv.org/html/2608.20680#Thmlemma2)\.

Regardingwwas the time\-independent function\(s,x\)↦w⁡\(x\)\(s,x\)\\mapsto w\(x\), we havew∈Cw1,0​\(\[0,T\]×𝒳\)w\\in C\_\{w\}^\{1,0\}\(\[0,T\]\\times\\mathcal\{X\}\)\. It follows from Theorem[1](https://arxiv.org/html/2608.20680#Thmtheorem1)that

OPENw⁡\(Xs𝝅,𝕊\)−∫ts∑z∈𝒳w⁡\(z\)​λ​\(z∣u,Xu−𝝅,𝕊,au𝝅,𝕊\)\)​𝑑u,s∈\[t,T\],\\displaystyle w\(X\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\-\\int\_\{t\}^\{s\}\\sum\_\{z\\in\\mathcal\{X\}\}w\(z\)\\lambda\(z\\mid u,X\_\{u\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},a\_\{u\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\)\\mathrm\{d\}u,\\qquad s\\in\[t,T\],is an\{ℱs𝕊\}s∈\[t,T\]\\\{\\mathcal\{F\}\_\{s\}^\{\\mathbb\{S\}\}\\\}\_\{s\\in\[t,T\]\}\-martingale\. Hence, for everys∈\[t,T\]s\\in\[t,T\],

𝔼⁡\[w⁡\(Xs𝝅,𝕊\)∣Xt𝝅,𝕊=x\]=\\displaystyle\\mathbb\{E\}\[w\(X\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\\mid X\_\{t\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}=x\]=\{\}w⁡\(x\)\+𝔼⁡\[∫ts∑z∈𝒳w⁡\(z\)​λ​\(z∣u,Xu−𝝅,𝕊,au𝝅,𝕊\)​𝑑u∣Xt𝝅,𝕊=x\]\.\\displaystyle w\(x\)\+\\mathbb\{E\}\\bigg\[\\int\_\{t\}^\{s\}\\sum\_\{z\\in\\mathcal\{X\}\}w\(z\)\\lambda\(z\\mid u,X\_\{u\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},a\_\{u\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\\,\\mathrm\{d\}u\\mid X\_\{t\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}=x\\bigg\]\.From Assumption[1](https://arxiv.org/html/2608.20680#Thmassumption1)\(i\), it follows that

𝔼⁡\[w⁡\(Xs𝝅,𝕊\)∣Xt𝝅,𝕊=x\]≤w⁡\(x\)\+∫ts\(c​𝔼​\[w⁡\(Xu−𝝅,𝕊\)∣Xt𝝅,𝕊=x\]\+b\)​𝑑u\.\\displaystyle\\mathbb\{E\}\[w\(X\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\\mid X\_\{t\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}=x\]\\leq w\(x\)\+\\int\_\{t\}^\{s\}\\bigl\(c\\,\\mathbb\{E\}\[w\(X\_\{u\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\\mid X\_\{t\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}=x\]\+b\\bigr\)\\mathrm\{d\}u\.Applying Gronwall’s inequality yields

𝔼⁡\[w⁡\(Xs𝝅,𝕊\)∣Xt𝝅,𝕊=x\]≤ec⁡\(s−t\)​w​\(x\)\+bc​\[ec⁡\(s−t\)−1\],s∈\[t,T\]\.\\displaystyle\\mathbb\{E\}\[w\(X\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\\mid X\_\{t\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}=x\]\\leq e^\{c\(s\-t\)\}w\(x\)\+\\frac\{b\}\{c\}\[e^\{c\(s\-t\)\}\-1\],\\qquad s\\in\[t,T\]\.\(29\)Combing \([29](https://arxiv.org/html/2608.20680#A2.E29)\) with Assumption[1](https://arxiv.org/html/2608.20680#Thmassumption1)\(iii\), we obtain the following integrability condition:

𝔼⁡\[∫tT∑z≠Xs−𝝅,𝕊\|ρ⁡\(s,Xs−𝝅,𝕊,as𝝅,𝕊,z\)\|​λ​\(z∣s,Xs−𝝅,𝕊,as𝝅,𝕊\)​𝑑s\|Xt𝝅,𝕊=x\]\\displaystyle\\mathbb\{E\}\\bigg\[\\int\_\{t\}^\{T\}\\sum\_\{z\\neq X\_\{s\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\}\|\\rho\(s,X\_\{s\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},a\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},z\)\|\\,\\lambda\(z\\mid s,X\_\{s\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},a\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\\mathrm\{d\}s\\,\\Big\|\\,X\_\{t\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}=x\\bigg\]≤\\displaystyle\\leq\{\}M1​∫tT𝔼⁡\[w⁡\(Xs−𝝅,𝕊\)∣Xt𝝅,𝕊=x\]​𝑑s<∞\.\\displaystyle M\_\{1\}\\int\_\{t\}^\{T\}\\mathbb\{E\}\[w\(X\_\{s\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\\mid X\_\{t\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}=x\]\\mathrm\{d\}s<\\infty\.\(30\)
For anyk∈\{0,…,K−1\}k\\in\\\{0,\\ldots,K\-1\\\},AkA\_\{k\}isℱtk𝕊\\mathcal\{F\}\_\{t\_\{k\}\}^\{\\mathbb\{S\}\}\-measurable and remains fixed on\(tk,tk\+1\]\(t\_\{k\},t\_\{k\+1\}\]\. Therefore, giventk≤u≤v≤tk\+1t\_\{k\}\\leq u\\leq v\\leq t\_\{k\+1\}, conditional onℱu𝕊\\mathcal\{F\}\_\{u\}^\{\\mathbb\{S\}\}, the process\{Xs𝝅,𝕊:s∈\[u,v\]\}\\\{X\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}:s\\in\[u,v\]\\\}is a jump Markov process starting fromXu𝝅,𝕊X\_\{u\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}with transition ratesλ⁡\(y∣s,Xs−𝝅,𝕊,Ak\)\\lambda\(y\\mid s,X\_\{s\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},A\_\{k\}\)\. By the integrability condition in \([B\.3](https://arxiv.org/html/2608.20680#A2.Ex74)\) and the martingale property of stochastic integrals with respect to compensated counting processes\{Ny​z𝝅,𝕊\(s\)−∫ts𝟏\{Xr−𝝅,𝕊=y\}λ\(z∣r,y,Ak\)dr:s∈\[u,v\]\}\\\{N\_\{yz\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\(s\)\-\\int\_\{t\}^\{s\}\\mathbf\{1\}\_\{\\\{X\_\{r\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}=y\\\}\}\\lambda\(z\\mid r,y,A\_\{k\}\)\\,\\mathrm\{d\}r:s\\in\[u,v\]\\\}fory,z∈𝒳y,\\,z\\in\\mathcal\{X\}withy≠zy\\neq z, we obtain

𝔼\[\\displaystyle\\mathbb\{E\}\\bigg\[∑y∈𝒳∑z≠y∫\(u,v\]ρ⁡\(s,y,Ak,z\)​d​Ny​z𝝅,𝕊​\(s\)\\displaystyle\\sum\_\{y\\in\\mathcal\{X\}\}\\sum\_\{z\\neq y\}\\int\_\{\(u,v\]\}\\rho\\bigl\(s,y,A\_\{k\},z\\bigr\)\\,\\mathrm\{d\}N\_\{yz\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\(s\)−∫uv∑z≠Xs−𝝅,𝕊ρ\(s,Xs−𝝅,𝕊,Ak,z\)λ\(z∣s,Xs−𝝅,𝕊,Ak\)ds\|ℱu𝕊\]=0\.\\displaystyle\-\\int\_\{u\}^\{v\}\\sum\_\{z\\neq X\_\{s\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\}\\rho\(s,X\_\{s\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},A\_\{k\},z\)\\,\\lambda\(z\\mid s,X\_\{s\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},A\_\{k\}\)\\,\\mathrm\{d\}s\\Bigm\|\\mathcal\{F\}\_\{u\}^\{\\mathbb\{S\}\}\\bigg\]=0\.\(31\)Sinceas𝝅,𝕊=Aka\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}=A\_\{k\}on\(tk,tk\+1\]\(t\_\{k\},t\_\{k\+1\}\], \([B\.3](https://arxiv.org/html/2608.20680#A2.Ex75)\) implies that

𝔼⁡\[Mvρ−Muρ∣ℱu𝕊\]=0\.\\displaystyle\\mathbb\{E\}\[M\_\{v\}^\{\\rho\}\-M\_\{u\}^\{\\rho\}\\mid\\mathcal\{F\}\_\{u\}^\{\\mathbb\{S\}\}\]=0\.Then, using the same argument as in the proof of Theorem[1](https://arxiv.org/html/2608.20680#Thmtheorem1), we conclude that\{Msρ:s∈\[t,T\]\}\\\{M\_\{s\}^\{\\rho\}:s\\in\[t,T\]\\\}is an\{ℱs𝕊\}s∈\[t,T\]\\\{\\mathcal\{F\}\_\{s\}^\{\\mathbb\{S\}\}\\\}\_\{s\\in\[t,T\]\}\-martingale\. ∎

With Lemma[2](https://arxiv.org/html/2608.20680#Thmlemma2)in hand, we now prove Theorem[2](https://arxiv.org/html/2608.20680#Thmtheorem2)\.

- \(i\)LetJ∗J^\{\*\}andq∗q^\{\*\}be the optimal value function and the optimalqq\-function, respectively\. For any𝝅∈𝚷\\boldsymbol\{\\pi\}\\in\\boldsymbol\{\\Pi\}, initial time\-state pair\(t,x\)∈\[0,T\]×𝒳\(t,x\)\\in\[0,T\]\\times\\mathcal\{X\}, and time grid𝕊\\mathbb\{S\}on\[t,T\]\[t,T\], we consider the process Ms∗≜J∗​\(s,Xs𝝅,𝕊\)\+∫ts\[R⁡\(u,Xu−𝝅,𝕊,au𝝅,𝕊\)−q∗​\(u,Xu−𝝅,𝕊,au𝝅,𝕊\)\]​𝑑u,s∈\[t,T\]\.\\displaystyle M\_\{s\}^\{\*\}\\triangleq J^\{\*\}\(s,X\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\+\\int\_\{t\}^\{s\}\[R\(u,X\_\{u\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},a\_\{u\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\-q^\{\*\}\(u,X\_\{u\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},a\_\{u\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\]\\mathrm\{d\}u,\\quad s\\in\[t,T\]\.where theR⁡\(t,x,a\)R\(t,x,a\)is the augmented running reward rate defined in \([3](https://arxiv.org/html/2608.20680#S2.E3)\)\. For anys∈\[t,T\]s\\in\[t,T\], we have 𝔼⁡\[MT∗−Ms∗∣ℱs𝕊\]\\displaystyle\\mathbb\{E\}\[M\_\{T\}^\{\*\}\-M\_\{s\}^\{\*\}\\mid\\mathcal\{F\}\_\{s\}^\{\\mathbb\{S\}\}\]=\\displaystyle=\{\}𝔼⁡\[J∗​\(T,XT𝝅,𝕊\)−J∗​\(s,Xs𝝅,𝕊\)−∫sT\[R⁡\(u,Xu−𝝅,𝕊,au𝝅,𝕊\)−q∗​\(u,Xu−𝝅,𝕊,au𝝅,𝕊\)\]​𝑑u∣ℱs𝕊\]\\displaystyle\\mathbb\{E\}\\bigg\[J^\{\*\}\(T,X\_\{T\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\-J^\{\*\}\(s,X\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\-\\int\_\{s\}^\{T\}\[R\(u,X\_\{u\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},a\_\{u\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\-q^\{\*\}\(u,X\_\{u\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},a\_\{u\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\]\\mathrm\{d\}u\\mid\\mathcal\{F\}\_\{s\}^\{\\mathbb\{S\}\}\\bigg\]=\\displaystyle=\{\}𝔼⁡\[J∗​\(T,XT𝝅,𝕊\)−J∗​\(s,Xs𝝅,𝕊\)−∫sT\(∂J∗∂u​\(u,Xu−𝝅,𝕊\)\+∑y∈𝒳J∗​\(u,y\)​λ​\(y∣u,Xu−𝝅,𝕊,au𝝅,𝕊\)\)​𝑑u∣ℱs𝕊\]\\displaystyle\\mathbb\{E\}\\bigg\[J^\{\*\}\(T,X\_\{T\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\-J^\{\*\}\(s,X\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\-\\int\_\{s\}^\{T\}\\bigg\(\\frac\{\\partial J^\{\*\}\}\{\\partial u\}\(u,X\_\{u\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\+\\sum\_\{y\\in\\mathcal\{X\}\}J^\{\*\}\(u,y\)\\lambda\(y\\mid u,X\_\{u\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},a\_\{u\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\\bigg\)\\mathrm\{d\}u\\mid\\mathcal\{F\}\_\{s\}^\{\\mathbb\{S\}\}\\bigg\]=\\displaystyle=\{\}0,\\displaystyle 0,where the last equality follows directly from Theorem[1](https://arxiv.org/html/2608.20680#Thmtheorem1)\. Hence,\{Ms∗:s∈\[t,T\]\}\\\{M\_\{s\}^\{\*\}:s\\in\[t,T\]\\\}is an\{ℱs𝕊\}s∈\[t,T\]\\\{\\mathcal\{F\}\_\{s\}^\{\\mathbb\{S\}\}\\\}\_\{s\\in\[t,T\]\}\-martingale\. Combining this with Lemma[2](https://arxiv.org/html/2608.20680#Thmlemma2), we conclude that\{Ms∗\+Msρ:s∈\[t,T\]\}\\\{M\_\{s\}^\{\*\}\+M\_\{s\}^\{\\rho\}:s\\in\[t,T\]\\\}is an\{ℱs𝕊\}s∈\[t,T\]\\\{\\mathcal\{F\}\_\{s\}^\{\\mathbb\{S\}\}\\\}\_\{s\\in\[t,T\]\}\-martingale\.
- \(ii\)For any givenx∈𝒳x\\in\\mathcal\{X\}, let\{Xt𝝅,𝕊:t∈\[0,T\]\}\\\{X\_\{t\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}:t\\in\[0,T\]\\\}and\{at𝝅,𝕊:t∈\[0,T\]\}\\\{a\_\{t\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}:t\\in\[0,T\]\\\}denote the grid sample state and action processes generated under𝝅\\boldsymbol\{\\pi\}and𝕊\\mathbb\{S\}, withX0𝝅,𝕊=xX\_\{0\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}=x\. Denote byℙx\\mathbb\{P\}\_\{x\}the corresponding probability law\. By the assumption in statement, the process \([12](https://arxiv.org/html/2608.20680#S4.E12)\) is an\{ℱt𝕊\}t∈\[0,T\]\\\{\\mathcal\{F\}\_\{t\}^\{\\mathbb\{S\}\}\\\}\_\{t\\in\[0,T\]\}\-martingale\. Theorem[1](https://arxiv.org/html/2608.20680#Thmtheorem1)and Lemma[2](https://arxiv.org/html/2608.20680#Thmlemma2)imply that the following two processes are\{ℱt𝕊\}t∈\[0,T\]\\\{\\mathcal\{F\}\_\{t\}^\{\\mathbb\{S\}\}\\\}\_\{t\\in\[0,T\]\}\-martingales: J∗^​\(t,Xt𝝅,𝕊\)−∫0t\(∂J∗^∂s​\(s,Xs−𝝅,𝕊\)\+∑y∈𝒳J∗^​\(s,y\)​λ​\(y∣s,Xs−𝝅,𝕊,as𝝅,𝕊\)\)​𝑑s,t∈\[0,T\],\\displaystyle\\widehat\{J^\{\*\}\}\(t,X\_\{t\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\-\\int\_\{0\}^\{t\}\\bigg\(\\frac\{\\partial\\widehat\{J^\{\*\}\}\}\{\\partial s\}\(s,X\_\{s\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\+\\sum\_\{y\\in\\mathcal\{X\}\}\\widehat\{J^\{\*\}\}\(s,y\)\\lambda\(y\\mid s,X\_\{s\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},a\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\\bigg\)\\mathrm\{d\}s,\\quad t\\in\[0,T\],∑y∈𝒳∑z≠y∫\(0,t\]ρ⁡\(s,y,as𝝅,𝕊,z\)​d​Ny​z𝝅,𝕊​\(s\)−∫0t∑z≠Xs−𝝅,𝕊ρ⁡\(s,Xs−𝝅,𝕊,as𝝅,𝕊,z\)​λ​\(z∣s,Xs−𝝅,𝕊,as𝝅,𝕊\)​𝑑s,t∈\[0,T\]\.\\displaystyle\\sum\_\{y\\in\\mathcal\{X\}\}\\sum\_\{z\\neq y\}\\int\_\{\(0,t\]\}\\rho\(s,y,a\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},z\)\\,\\mathrm\{d\}N\_\{yz\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\(s\)\-\\int\_\{0\}^\{t\}\\sum\_\{z\\neq X\_\{s\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\}\\rho\(s,X\_\{s\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},a\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},z\)\\lambda\(z\\mid s,X\_\{s\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},a\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\\mathrm\{d\}s,\\quad t\\in\[0,T\]\.Taking appropriate sum and difference of the three martingales above, we derive that the process\{∫0tD⁡\(s,Xs−𝝅,𝕊,as𝝅,𝕊\)​𝑑s:t∈\[0,T\]\}\\\{\\int\_\{0\}^\{t\}D\(s,X\_\{s\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},a\_\{s\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\\mathrm\{d\}s:t\\in\[0,T\]\\\}is a continuous finite\-variation martingale, where D⁡\(t,x,a\)≜∂J∗^∂t​\(t,x\)\+∑y∈𝒳J∗^​\(t,y\)​λ​\(y∣t,x,a\)\+R⁡\(t,x,a\)−q∗^​\(t,x,a\),\(t,x,a\)∈𝕄\.D\(t,x,a\)\\triangleq\\frac\{\\partial\\widehat\{J^\{\*\}\}\}\{\\partial t\}\(t,x\)\+\\sum\_\{y\\in\\mathcal\{X\}\}\\widehat\{J^\{\*\}\}\(t,y\)\\lambda\(y\\mid t,x,a\)\+R\(t,x,a\)\-\\widehat\{q^\{\*\}\}\(t,x,a\),\\qquad\(t,x,a\)\\in\\mathbb\{M\}\.Therefore, D⁡\(t,Xt−𝝅,𝕊,at𝝅,𝕊\)=0d​t⊗ℙx​\-a\.e\.\.\\displaystyle D\(t,X\_\{t\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},a\_\{t\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)=0\\qquad\\mathrm\{d\}t\\otimes\\mathbb\{P\}\_\{x\}\\text\{\-a\.e\.\}\.Tonelli’s theorem gives ∫tktk\+1𝔼x​\[\|D⁡\(t,Xt−𝝅,𝕊,at𝝅,𝕊\)\|\]​𝑑t=0,∀x∈𝒳\.\\displaystyle\\int\_\{t\_\{k\}\}^\{t\_\{k\+1\}\}\\mathbb\{E\}\_\{x\}\\bigl\[\|D\(t,X\_\{t\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},a\_\{t\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\|\\bigr\]\\mathrm\{d\}t=0,\\qquad\\forall\\ x\\in\\mathcal\{X\}\.\(32\)Next, we show that, for everyx∈𝒳x\\in\\mathcal\{X\}, D⁡\(t,x,a\)=0,d​t⊗μ⁡\(d​a\)​\-a\.e\. on​\[0,T\]×𝒜\.\\displaystyle D\(t,x,a\)=0,\\qquad\\mathrm\{d\}t\\otimes\\mu\(\\mathrm\{d\}a\)\\text\{\-a\.e\. on \}\[0,T\]\\times\\mathcal\{A\}\.\(33\)We prove this by contradiction\. Suppose \([33](https://arxiv.org/html/2608.20680#A2.E33)\) does not hold for somey∈𝒳y\\in\\mathcal\{X\}\. Then there existsk∈\{0,…,K−1\}k\\in\\\{0,\\ldots,K\-1\\\}such that the set Sky≜\{\(t,a\)∈\(tk,tk\+1\]×𝒜:\|D⁡\(t,y,a\)\|\>0\}\\displaystyle S\_\{k\}^\{y\}\\triangleq\\left\\\{\(t,a\)\\in\(t\_\{k\},t\_\{k\+1\}\]\\times\\mathcal\{A\}:\|D\(t,y,a\)\|\>0\\right\\\}\(34\)has positived​t⊗μ⁡\(d​a\)\\mathrm\{d\}t\\otimes\\mu\(\\mathrm\{d\}a\)\-measure\. We consider the grid sample state processX𝝅,𝕊X^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}initialized atX0𝝅,𝕊=yX\_\{0\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}=y, and define Eky≜\{Xt𝝅,𝕊=yfor allt∈\[0,tk\]\}\.E\_\{k\}^\{y\}\\triangleq\\\{X\_\{t\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}=y\\text\{ for all \}t\\in\[0,t\_\{k\}\]\\\}\.Sinceλ∗​\(y\)=supt∈\[0,T\],a∈𝒜λ⁡\(t,y,a\)<∞\\lambda^\{\*\}\(y\)=\\sup\_\{t\\in\[0,T\],\\,a\\in\\mathcal\{A\}\}\\lambda\(t,y,a\)<\\infty, it follows that ℙy​\(Eky\)≥exp⁡\{−λ∗​\(y\)​tk\}\>0\.\\displaystyle\\mathbb\{P\}\_\{y\}\(E\_\{k\}^\{y\}\)\\geq\\exp\\\{\-\\lambda^\{\*\}\(y\)t\_\{k\}\\\}\>0\.Next, letτky≜inf\{t\>tk:Xt𝝅,𝕊≠y\}\\tau\_\{k\}^\{y\}\\triangleq\\inf\\\{t\>t\_\{k\}:X\_\{t\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\\neq y\\\}, conditional onEkyE\_\{k\}^\{y\}and the actionAk=aA\_\{k\}=aselected attkt\_\{k\}, we have Pky\(a\)≜ℙy\(τky\>tk\+1∣Eky,Ak=a\)=exp\{−∫tktk\+1λ\(t,y,a\)dt\}≥e−λ∗​\(y\)​\(tk\+1−tk\)\>0\.P\_\{k\}^\{y\}\(a\)\\triangleq\\mathbb\{P\}\_\{y\}\(\\tau\_\{k\}^\{y\}\>t\_\{k\+1\}\\mid E\_\{k\}^\{y\},\\,A\_\{k\}=a\)=\\exp\\left\\\{\-\\int\_\{t\_\{k\}\}^\{t\_\{k\+1\}\}\\lambda\(t,y,a\)\\,\\mathrm\{d\}t\\right\\\}\\geq e^\{\-\\lambda^\{\*\}\(y\)\(t\_\{k\+1\}\-t\_\{k\}\)\}\>0\.Since conditional onEkyE\_\{k\}^\{y\},AkA\_\{k\}is sampled from𝝅\(⋅∣tk,y\)\\boldsymbol\{\\pi\}\(\\cdot\\mid t\_\{k\},y\), andat𝝅,𝕊=Aka\_\{t\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}=A\_\{k\}fort∈\(tk,tk\+1\]t\\in\(t\_\{k\},t\_\{k\+1\}\], it follows that ∫tktk\+1𝔼y​\[\|D⁡\(t,Xt−𝝅,𝕊,at𝝅,𝕊\)\|\]​𝑑t\\displaystyle\\int\_\{t\_\{k\}\}^\{t\_\{k\+1\}\}\\mathbb\{E\}\_\{y\}\\bigl\[\|D\(t,X\_\{t\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},a\_\{t\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\|\\bigr\]\\,\\mathrm\{d\}t≥\\displaystyle\\geq\{\}ℙ⁡\(Eky\)⋅∫tktk\+1𝔼y​\[\|D⁡\(t,Xt−𝝅,𝕊,Ak\)\|\|Eky\]​𝑑t\\displaystyle\\mathbb\{P\}\(E\_\{k\}^\{y\}\)\\cdot\\int\_\{t\_\{k\}\}^\{t\_\{k\+1\}\}\\mathbb\{E\}\_\{y\}\\Bigl\[\|D\(t,X\_\{t\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},A\_\{k\}\)\|\\,\\big\|\\,E\_\{k\}^\{y\}\\Bigr\]\\,\\mathrm\{d\}t=\\displaystyle=\{\}ℙ\(Eky\)⋅∫tktk\+1∫𝒜𝔼y\[\|D\(t,Xt−𝝅,𝕊,a\)\|\|Eky,Ak=a\]𝝅\(da∣tk,y\)dt\\displaystyle\\mathbb\{P\}\(E\_\{k\}^\{y\}\)\\cdot\\int\_\{t\_\{k\}\}^\{t\_\{k\+1\}\}\\int\_\{\\mathcal\{A\}\}\\mathbb\{E\}\_\{y\}\\Bigl\[\|D\(t,X\_\{t\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},a\)\|\\,\\big\|\\,E\_\{k\}^\{y\},A\_\{k\}=a\\Bigr\]\\boldsymbol\{\\pi\}\(\\mathrm\{d\}a\\mid t\_\{k\},y\)\\,\\mathrm\{d\}t≥\\displaystyle\\geq\{\}ℙ\(Eky\)⋅∫tktk\+1∫𝒜𝔼y\[\|D\(t,Xt−𝝅,𝕊,a\)\|𝟏\{τky\>tk\+1\}\|Eky,Ak=a\]𝝅\(da∣tk,y\)dt\\displaystyle\\mathbb\{P\}\(E\_\{k\}^\{y\}\)\\cdot\\int\_\{t\_\{k\}\}^\{t\_\{k\+1\}\}\\int\_\{\\mathcal\{A\}\}\\mathbb\{E\}\_\{y\}\\Bigl\[\|D\(t,X\_\{t\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},a\)\|\\boldsymbol\{1\}\_\{\\\{\\tau\_\{k\}^\{y\}\>t\_\{k\+1\}\\\}\}\\,\\big\|\\,E\_\{k\}^\{y\},A\_\{k\}=a\\Bigr\]\\boldsymbol\{\\pi\}\(\\mathrm\{d\}a\\mid t\_\{k\},y\)\\,\\mathrm\{d\}t=\\displaystyle=\{\}ℙ⁡\(Eky\)⋅∫tktk\+1∫𝒜\|D⁡\(t,y,a\)\|​Pky​\(a\)​𝝅​\(a∣tk,y\)​μ​\(𝑑a\)​𝑑t\.\\displaystyle\\mathbb\{P\}\(E\_\{k\}^\{y\}\)\\cdot\\int\_\{t\_\{k\}\}^\{t\_\{k\+1\}\}\\int\_\{\\mathcal\{A\}\}\|D\(t,y,a\)\|\\,P\_\{k\}^\{y\}\(a\)\\,\\boldsymbol\{\\pi\}\(a\\mid t\_\{k\},y\)\\,\\mu\(\\mathrm\{d\}a\)\\mathrm\{d\}t\.By the definition ofSkyS\_\{k\}^\{y\}in \([34](https://arxiv.org/html/2608.20680#A2.E34)\), we haveSky⊂\(tk,tk\+1\]×𝒜S\_\{k\}^\{y\}\\subset\(t\_\{k\},t\_\{k\+1\}\]\\times\\mathcal\{A\}, and hence ∫tktk\+1𝔼y​\[\|D⁡\(t,Xt−𝝅,𝕊,at𝝅,𝕊\)\|\]​𝑑t≥ℙ⁡\(Eky\)⋅∫Sky\|D⁡\(t,y,a\)\|​Pky​\(a\)​𝝅​\(a∣tk,y\)​μ​\(𝑑a\)​𝑑t\.\\displaystyle\\int\_\{t\_\{k\}\}^\{t\_\{k\+1\}\}\\mathbb\{E\}\_\{y\}\\bigl\[\|D\(t,X\_\{t\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},a\_\{t\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\|\\bigr\]\\,\\mathrm\{d\}t\\geq\\mathbb\{P\}\(E\_\{k\}^\{y\}\)\\cdot\\int\_\{S\_\{k\}^\{y\}\}\|D\(t,y,a\)\|\\,P\_\{k\}^\{y\}\(a\)\\,\\boldsymbol\{\\pi\}\(a\\mid t\_\{k\},y\)\\,\\mu\(\\mathrm\{d\}a\)\\mathrm\{d\}t\.Since𝝅\(⋅∣tk,y\)∼μ\\boldsymbol\{\\pi\}\(\\cdot\\mid t\_\{k\},y\)\\sim\\mufor every𝝅∈𝚷\\boldsymbol\{\\pi\}\\in\\boldsymbol\{\\Pi\}, we have the density𝝅⁡\(a∣tk,y\)\>0\\boldsymbol\{\\pi\}\(a\\mid t\_\{k\},y\)\>0forμ\\mu\-a\.e\. on𝒜\\mathcal\{A\}\. Moreover,SkyS\_\{k\}^\{y\}has positive\(d​t⊗d​μ\)\(\\mathrm\{d\}t\\otimes\\mathrm\{d\}\\mu\)\-measure, and\|D⁡\(t,y,a\)\|\>0\|D\(t,y,a\)\|\>0onSkyS\_\{k\}^\{y\},Pky​\(a\)\>0P\_\{k\}^\{y\}\(a\)\>0for alla∈𝒜a\\in\\mathcal\{A\}, andℙ⁡\(Eky\)\>0\\mathbb\{P\}\(E\_\{k\}^\{y\}\)\>0, it follows that ∫tktk\+1𝔼y​\[\|D⁡\(t,Xt−𝝅,𝕊,at𝝅,𝕊\)\|\]​𝑑t\>0\.\\displaystyle\\int\_\{t\_\{k\}\}^\{t\_\{k\+1\}\}\\mathbb\{E\}\_\{y\}\\bigl\[\|D\(t,X\_\{t\-\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\},a\_\{t\}^\{\\boldsymbol\{\\pi\},\\mathbb\{S\}\}\)\|\\bigr\]\\,\\mathrm\{d\}t\>0\.This contradicts the result derived in \([32](https://arxiv.org/html/2608.20680#A2.E32)\), and hence the desired result \([33](https://arxiv.org/html/2608.20680#A2.E33)\) holds\. From \([33](https://arxiv.org/html/2608.20680#A2.E33)\), we have q∗^​\(t,x,a\)=∂J∗^∂t​\(t,x\)\+H⁡\(t,x,a,J∗^​\(⋅,⋅\)\),d​t⊗μ⁡\(d​a\)​\-a\.e\. on​\[0,T\]×𝒜\.\\displaystyle\\widehat\{q^\{\*\}\}\(t,x,a\)=\\frac\{\\partial\\widehat\{J^\{\*\}\}\}\{\\partial t\}\(t,x\)\+H\(t,x,a,\\widehat\{J^\{\*\}\}\(\\cdot,\\cdot\)\),\\qquad\\mathrm\{d\}t\\otimes\\mu\(\\mathrm\{d\}a\)\\text\{\-a\.e\. on \}\[0,T\]\\times\\mathcal\{A\}\.\(35\)Sinceq∗^\\widehat\{q^\{\*\}\}satisfies the normalization condition in \([11](https://arxiv.org/html/2608.20680#S4.E11)\), we have 1=∫𝒜exp⁡\{1γ​q∗^​\(t,x,a\)\}​μ​\(𝑑a\)=exp⁡\{1γ​∂J^∗∂t​\(t,x\)\}⋅∫𝒜exp⁡\{1γ​H​\(t,x,a,J^∗​\(⋅,⋅\)\)\}​μ​\(𝑑a\)\.\\displaystyle 1=\\int\_\{\\mathcal\{A\}\}\\exp\\\{\\frac\{1\}\{\\gamma\}\\widehat\{q^\{\*\}\}\(t,x,a\)\\\}\\mu\(\\mathrm\{d\}a\)=\\exp\\\{\\frac\{1\}\{\\gamma\}\\frac\{\\partial\\widehat\{J\}^\{\*\}\}\{\\partial t\}\(t,x\)\\\}\\cdot\\int\_\{\\mathcal\{A\}\}\\exp\\\{\\frac\{1\}\{\\gamma\}H\(t,x,a,\\widehat\{J\}^\{\*\}\(\\cdot,\\cdot\)\)\\\}\\mu\(\\mathrm\{d\}a\)\.Taking logarithms on both sides yields ∂J∗^∂t​\(t,x\)\+γ​log⁡\[∫𝒜exp⁡\{1γ​H​\(t,x,a,J∗^​\(⋅,⋅\)\)\}​μ​\(𝑑a\)\]=0\.\\displaystyle\\frac\{\\partial\\widehat\{J^\{\*\}\}\}\{\\partial t\}\(t,x\)\+\\gamma\\log\\bigg\[\\int\_\{\\mathcal\{A\}\}\\exp\\\{\\frac\{1\}\{\\gamma\}H\(t,x,a,\\widehat\{J^\{\*\}\}\(\\cdot,\\cdot\)\)\\\}\\mu\(\\mathrm\{d\}a\)\\bigg\]=0\.\(36\)Now, we define a policy𝝅∗^∈𝚷\\widehat\{\\boldsymbol\{\\pi\}^\{\*\}\}\\in\\boldsymbol\{\\Pi\}as follows 𝝅∗^​\(a∣t,x\)≜exp⁡\{1γ​H​\(t,x,a,J∗^​\(⋅,⋅\)\)\}∫𝒜exp⁡\{1γ​H​\(t,x,a,J∗^​\(⋅,⋅\)\)\}​μ​\(𝑑a\),\(t,x,a\)∈𝕄\.\\displaystyle\\widehat\{\\boldsymbol\{\\pi\}^\{\*\}\}\(a\\mid t,x\)\\triangleq\\frac\{\\exp\\\{\\frac\{1\}\{\\gamma\}H\(t,x,a,\\widehat\{J^\{\*\}\}\(\\cdot,\\cdot\)\)\\\}\}\{\\int\_\{\\mathcal\{A\}\}\\exp\\\{\\frac\{1\}\{\\gamma\}H\(t,x,a,\\widehat\{J^\{\*\}\}\(\\cdot,\\cdot\)\)\\\}\\mu\(\\mathrm\{d\}a\)\},\\qquad\(t,x,a\)\\in\\mathbb\{M\}\.It follows from Lemma[1](https://arxiv.org/html/2608.20680#Thmlemma1)that sup𝝅∈𝚷∫𝒜\{H⁡\(t,x,a,J∗^​\(⋅,⋅\)\)−γ​log⁡𝝅⁡\(a∣t,x\)\}​𝝅​\(𝑑a∣t,x\)\\displaystyle\\sup\_\{\\boldsymbol\{\\pi\}\\in\\boldsymbol\{\\Pi\}\}\\int\_\{\\mathcal\{A\}\}\\\{H\(t,x,a,\\widehat\{J^\{\*\}\}\(\\cdot,\\cdot\)\)\-\\gamma\\log\\boldsymbol\{\\pi\}\(a\\mid t,x\)\\\}\\boldsymbol\{\\pi\}\(\\mathrm\{d\}a\\mid t,x\)=\\displaystyle=\{\}∫𝒜\{H⁡\(t,x,a,J∗^​\(⋅,⋅\)\)−γ​log⁡𝝅∗^​\(a∣t,x\)\}​𝝅∗^​\(𝑑a∣t,x\)\\displaystyle\\int\_\{\\mathcal\{A\}\}\\\{H\(t,x,a,\\widehat\{J^\{\*\}\}\(\\cdot,\\cdot\)\)\-\\gamma\\log\\widehat\{\\boldsymbol\{\\pi\}^\{\*\}\}\(a\\mid t,x\)\\\}\\widehat\{\\boldsymbol\{\\pi\}^\{\*\}\}\(\\mathrm\{d\}a\\mid t,x\)=\\displaystyle=\{\}γ​log⁡\[∫𝒜exp⁡\{1γ​H​\(t,x,a,J∗^​\(⋅,⋅\)\)\}​μ​\(𝑑a\)\]\\displaystyle\\gamma\\log\\bigg\[\\int\_\{\\mathcal\{A\}\}\\exp\\\{\\frac\{1\}\{\\gamma\}H\(t,x,a,\\widehat\{J^\{\*\}\}\(\\cdot,\\cdot\)\)\\\}\\mu\(\\mathrm\{d\}a\)\\bigg\]=\\displaystyle=\{\}−∂J∗^∂t​\(t,x\),\\displaystyle\-\\frac\{\\partial\\widehat\{J^\{\*\}\}\}\{\\partial t\}\(t,x\),\(37\)where the last equality is due to \([36](https://arxiv.org/html/2608.20680#A2.E36)\)\. SinceJ∗^\\widehat\{J^\{\*\}\}satisfies Equation \([\(ii\)](https://arxiv.org/html/2608.20680#A2.Ex98)\) and the terminal conditionJ∗^​\(T,x\)=h​\(x\)\\widehat\{J^\{\*\}\}\(T,x\)=h\(x\)in \([11](https://arxiv.org/html/2608.20680#S4.E11)\), and in light of Proposition[1](https://arxiv.org/html/2608.20680#Thmproposition1), we conclude thatJ∗^\\widehat\{J^\{\*\}\}is the optimal value function\. Moreover, from \([35](https://arxiv.org/html/2608.20680#A2.E35)\) and the definition of the optimalqq\-function, we conclude thatq∗^\\widehat\{q^\{\*\}\}is the optimalqq\-function\.