Deep Reinforcement Learning with Buffered Quantile Objectives

# Deep Reinforcement Learning with Buffered Quantile Objectives
Source: [https://arxiv.org/html/2609.21327](https://arxiv.org/html/2609.21327)
Sajad Khodadadian††thanks:Corresponding AuthorAffiliation:Virginia TechEmail:[sajadk@vt\.edu](mailto:)

###### Abstract

Quantile\-based reinforcement learning provides an interpretable approach to risk\-sensitive decision making by optimizing a prescribed quantile of the cumulative\-return distribution\. Despite this appeal, learning under a point quantile objective is challenging: quantiles can change abruptly under small perturbations of the return distribution, and exact quantile\-sensitive planning requires computationally demanding distributional optimization\. Lower\-buffered quantiles alleviate the former difficulty by averaging neighboring quantiles immediately below the target level, providing a smoother surrogate while preserving the underlying point\-quantile objective\. Existing methods based on this principle, however, remain model based and rely on explicit return\-law planning, limiting their applicability beyond small tabular problems\. We developDeep–BQRL, a model\-free distributional reinforcement\-learning framework that extends buffered\-quantile learning to neural function approximation\. The method learns conditional return quantiles directly from sampled transitions, constructs buffered action scores from the relevant region of the learned quantile function, and uses ensemble disagreement to guide exploration\. An augmented input representation allows the learned policy to respond to trajectory information without explicitly reproducing the quantile\-state recursion required by exact planning\. Experiments on an asset\-selling optimal\-stopping problem and slippery FrozenLake compareDeep–BQRLwith model\-basedUCB–BQRLand tabular PPO and TRPO implementations\. In asset selling,Deep–BQRLattains smaller mean cumulative point\-quantile policy gaps than PPO and TRPO at the reported target levels, whileUCB–BQRLretains the smallest gaps\. The learned stopping decisions also vary with the target quantile, providing an interpretable illustration of the method’s risk\-sensitive behavior\.

## 1Introduction

Reinforcement learning \(RL\) is commonly formulated to maximize expected cumulative reward\([Sutton et al\., 1998](https://arxiv.org/html/2609.21327#bib.bib19)\)\. In many applications, however, the expectation alone does not adequately characterize performance: two policies with similar mean returns may have substantially different probabilities of poor outcomes\([Li et al\., 2022](https://arxiv.org/html/2609.21327#bib.bib13)\)\. This motivates*risk\-sensitive*RL, where decisions depend on the distribution of cumulative return rather than only its expectation\([Howard & Matheson, 1972](https://arxiv.org/html/2609.21327#bib.bib11);[Garcıa & Fernández, 2015](https://arxiv.org/html/2609.21327#bib.bib9);[Gottesman et al\., 2019](https://arxiv.org/html/2609.21327#bib.bib10)\)\. Quantile objectives provide a particularly interpretable criterion by directly targeting a specified region of the return distribution\([Filar et al\., 1995](https://arxiv.org/html/2609.21327#bib.bib7);[Delage & Mannor, 2010](https://arxiv.org/html/2609.21327#bib.bib5);[Li et al\., 2022](https://arxiv.org/html/2609.21327#bib.bib13)\)\. In RL, maximizing theτ\\tau\-quantile of cumulative reward seeks a policy that makes the return at theτ\\tau\-th percentile of its cumulative\-return distribution as large as possible\.

Optimizing return quantiles nevertheless poses important statistical and computational challenges\. Point quantiles can be sensitive to small perturbations of the underlying return distribution, making them difficult to estimate reliably when the environment is learned from finite data\([Alipour\-Vaezi et al\., 2026b](https://arxiv.org/html/2609.21327#bib.bib2)\)\. To mitigate this instability,[Alipour\-Vaezi et al\. \(2026a\)](https://arxiv.org/html/2609.21327#bib.bib1)introduced the lower\-buffered quantile within theUCB–BQRLframework, which averages quantiles over a neighborhood immediately below the target level\. This criterion retains the distributional interpretation of a quantile while smoothing its dependence on the return law\. It is closely related to Range Value\-at\-Risk and interpolates between a point quantile and a lower\-tail average as the buffer width varies\([Fissler & Ziegel, 2021](https://arxiv.org/html/2609.21327#bib.bib8);[Embrechts et al\., 2018](https://arxiv.org/html/2609.21327#bib.bib6)\)\.

The remaining difficulty is scalability\. Exact quantile planning is fundamentally different from ordinary Bellman optimization because the optimal decision may depend on the realized trajectory through a dynamically evolving quantile level\([Li et al\., 2022](https://arxiv.org/html/2609.21327#bib.bib13)\)\. The model\-basedUCB–BQRLframework provides a principled tabular solution for learning under buffered\-quantile objectives, but its planner requires explicit manipulation of return distributions and optimization over transition confidence sets\. Moreover, exact evaluation of point and buffered quantiles is computationally hard in general\([Alipour\-Vaezi et al\., 2026a](https://arxiv.org/html/2609.21327#bib.bib1)\)\. These operations become increasingly costly as the state and action spaces grow and motivate a model\-free neural approximation that avoids explicit return\-law planning during training\.

We developDeep–BQRL, a distributional deep RL framework for risk\-sensitive control under lower\-buffered quantile objectives\. Rather than estimating a transition model and repeatedly solving a quantile\-planning problem,Deep–BQRLlearns a fixed collection of conditional return quantiles directly using distributional critics\([Bellemare et al\., 2017](https://arxiv.org/html/2609.21327#bib.bib3);[Dabney et al\., 2018](https://arxiv.org/html/2609.21327#bib.bib4)\)\. The buffered action score is then obtained by aggregating the learned quantiles over\[τ−β,τ\]\[\\tau\-\\beta,\\tau\], allowing policy improvement to depend on a local region of the return distribution rather than on a single estimated quantile\. To account for the history dependence of quantile\-sensitive decisions\([Li et al\., 2022](https://arxiv.org/html/2609.21327#bib.bib13)\), the policy and critic operate on an augmented state representation that incorporates trajectory information\. We further maintain an ensemble of distributional critics and use disagreement in their buffered action scores to encourage exploration directly in return\-value space\.

Our contributions are threefold\. First, we developDeep–BQRL, a model\-free deep RL method for quantile\-sensitive decision making that avoids learning an explicit transition model and repeatedly solving an exact quantile\-planning problem\. Second, we use learned return quantiles to construct a lower\-buffered criterion for action selection, providing a practical way to use the smoother objective with neural function approximation\. Third, we use disagreement among multiple learned critics to encourage exploration in parts of the return distribution that remain uncertain\. The main experiment uses an asset\-selling \(optimal\-stopping\) problem\([Seierstad, 1992](https://arxiv.org/html/2609.21327#bib.bib18);[Rosenfield et al\., 1983](https://arxiv.org/html/2609.21327#bib.bib15)\)as well as slippery FrozenLake to compare learning performance and examine how the target quantile affects the learned stopping decisions\.

## 2Preliminaries

### 2\.1Finite\-Horizon Reinforcement Learning

We consider a finite\-horizon Markov decision processℳ=\(𝒮,𝒜,H,P,r\)\\mathcal\{M\}=\(\\mathcal\{S\},\\mathcal\{A\},H,P,r\), where𝒮\\mathcal\{S\}and𝒜\\mathcal\{A\}are the state and action spaces,HHis the horizon,P=\{Ph\}h=0H−1P=\\\{P\_\{h\}\\\}\_\{h=0\}^\{H\-1\}is the collection of stage\-dependent transition kernels, andr=\{rh\}h=0H−1r=\\\{r\_\{h\}\\\}\_\{h=0\}^\{H\-1\}is the collection of bounded one\-step reward functions\. At stagehh, the agent observes stateShS\_\{h\}, selects actionAhA\_\{h\}, transitions according toSh\+1∼Ph\(⋅∣Sh,Ah\)S\_\{h\+1\}\\sim P\_\{h\}\(\\cdot\\mid S\_\{h\},A\_\{h\}\), and receivesRh=rh​\(Sh,Ah,Sh\+1\)R\_\{h\}=r\_\{h\}\(S\_\{h\},A\_\{h\},S\_\{h\+1\}\)\. Thus,rh​\(s,a,s′\)r\_\{h\}\(s,a,s^\{\\prime\}\)is deterministic conditional on a realized transition\(s,a,s′\)\(s,a,s^\{\\prime\}\), whileRhR\_\{h\}may be random through the random next stateSh\+1S\_\{h\+1\}\. We assume a fixed initial\-state distribution and suppress it from the notation\. A policyπ=\{πh\}h=0H−1\\pi=\\\{\\pi\_\{h\}\\\}\_\{h=0\}^\{H\-1\}may in general depend on the observed trajectory\. Its total episodic return isRπ≔∑h=0H−1RhR^\{\\pi\}\\coloneqq\\sum\_\{h=0\}^\{H\-1\}R\_\{h\}\.

### 2\.2Quantile Objective and Lower\-Buffered Quantile

For a bounded real\-valued random variableWWandq∈\(0,1\]q\\in\(0,1\], itsqq\-quantile isQq​\(W\)≔inf\{w∈ℝ:Pr⁡\(W≤w\)≥q\}Q\_\{q\}\(W\)\\coloneqq\\inf\\\{w\\in\\mathbb\{R\}:\\Pr\(W\\leq w\)\\geq q\\\}\. For completeness, we useQ0​\(W\)≔infsupp⁡\(W\)Q\_\{0\}\(W\)\\coloneqq\\inf\\operatorname\{supp\}\(W\)\. For a target levelτ∈\(0,1\)\\tau\\in\(0,1\), the risk\-sensitive objective of interest ismaxπ⁡Qτ​\(Rπ\)\\max\_\{\\pi\}Q\_\{\\tau\}\(R^\{\\pi\}\)\. Thus, the goal is to find a policy whose cumulative return is as large as possible at the prescribed quantile level\. Smaller values ofτ\\tauemphasize lower\-tail outcomes, whereas larger values place greater emphasis on upper\-tail realizations\([Li et al\., 2022](https://arxiv.org/html/2609.21327#bib.bib13)\)\. Direct optimization of this objective is challenging because point quantiles can change abruptly under small perturbations of the return distribution\. To obtain a smoother criterion, following[Alipour\-Vaezi et al\. \(2026a\)](https://arxiv.org/html/2609.21327#bib.bib1), for a buffer widthβ∈\(0,τ\]\\beta\\in\(0,\\tau\]we define the lower\-buffered quantileQτβ​\(W\)≔1β​∫τ−βτQu​\(W\)​𝑑uQ\_\{\\tau\}^\{\\beta\}\(W\)\\coloneqq\\frac\{1\}\{\\beta\}\\int\_\{\\tau\-\\beta\}^\{\\tau\}Q\_\{u\}\(W\)\\,du\. Rather than relying on a single point of the quantile function,Qτβ​\(W\)Q\_\{\\tau\}^\{\\beta\}\(W\)averages quantiles over the interval immediately belowτ\\tau\. Asβ\\betadecreases, the buffered criterion approaches the point quantile, while larger values ofβ\\betaincorporate more information from neighboring lower quantiles\([Alipour\-Vaezi et al\., 2026a](https://arxiv.org/html/2609.21327#bib.bib1)\)\.

## 3Deep–BQRL

We now introduceDeep–BQRL, a model\-free distributional learning method for finite action spaces\. A key distinction between expected\-return and quantile\-sensitive control is that the optimal policy of a quantile\-sensitive decision is history\-dependent, i\.e\., it depends on the entire trajectory and not only on the current state\. Exact QMDP planning handles this dependence by updating an auxiliary quantile level after each observed transition\([Li et al\., 2022](https://arxiv.org/html/2609.21327#bib.bib13)\)\. Exact lower\-buffered planning similarly relies on the notion of return distributions, which captures history dependence of the sampling policy\([Alipour\-Vaezi et al\., 2026a](https://arxiv.org/html/2609.21327#bib.bib1)\)\. Both operations become computationally demanding beyond small tabular problems\. Therefore,Deep–BQRLuses three approximations\. First, Approximation I compresses trajectory information into the current stage, state, and cumulative reward\. Second, Approximation II represents each conditional remaining\-return distribution by a finite collection of learned quantiles\. Third, Approximation III replaces exact quantile planning with greedy policy improvement by its approximation, denoted as buffered action scores\.

#### Approximation I: Augmented Trajectory Representation

The first approximation replaces the trajectory information with a fixed\-dimensional trajectory summary\. At stagehh, letCh≔∑k=0h−1RkC\_\{h\}\\coloneqq\\sum\_\{k=0\}^\{h\-1\}R\_\{k\}, denote the cumulative reward collected before actionAhA\_\{h\}is selected\. We define the augmented inputXh≔\(h,Sh,Ch\)X\_\{h\}\\coloneqq\(h,S\_\{h\},C\_\{h\}\),h=0,…,Hh=0,\\ldots,H, and writex=\(h,s,c\)x=\(h,s,c\)for a realization ofXhX\_\{h\}\. The augmented input is an implemented trajectory summary\. Forx=\(h,s,c\)x=\(h,s,c\)anda∈𝒜a\\in\\mathcal\{A\}, letZπ​\(x,a\)Z^\{\\pi\}\(x,a\)denote a random variable with the same law as the remaining reward∑k=hH−1Rk\\sum\_\{k=h\}^\{H\-1\}R\_\{k\}when actionaais taken at augmented inputxx, and policyπ\\piis followed thereafter\. Atxx, the previously collected rewardccis already known, so the corresponding total return has the same law asc\+Zπ​\(x,a\)c\+Z^\{\\pi\}\(x,a\)\. The critic estimates the distribution of the remaining returnZπ​\(x,a\)Z^\{\\pi\}\(x,a\), takingccas part of the input\. This is because a history\-dependent policy may condition its future actions on the reward accumulated so far\.

#### Approximation II: Quantile Representation of Remaining Returns

The second approximation estimates the law ofZπ​\(x,a\)Z^\{\\pi\}\(x,a\), by a finite collection of learned quantiles\.Deep–BQRLmaintainsM≥2M\\geq 2separately initialized critics\. For the fixed levelsηj≔\(j−12\)/K\\eta\_\{j\}\\coloneqq\(j\-\\frac\{1\}\{2\}\)/K,j=1,…,Kj=1,\\ldots,K, criticmmestimateszθm,j​\(x,a\)≈Qηj​\(Zπ​\(x,a\)\)z\_\{\\theta\_\{m\},j\}\(x,a\)\\approx Q\_\{\\eta\_\{j\}\}\\\!\\left\(Z^\{\\pi\}\(x,a\)\\right\)\. Quantile\-regression distributional RL provides a practical mechanism for learning these quantiles from sampled transitions\([Bellemare et al\., 2017](https://arxiv.org/html/2609.21327#bib.bib3);[Dabney et al\., 2018](https://arxiv.org/html/2609.21327#bib.bib4)\)\. Because the critic can occasionally predict a larger return for a lower quantile level than for a higher one, we sort theKKpredicted returns for each state–action pair\. Letz~θm,1​\(x,a\)≤⋯≤z~θm,K​\(x,a\)\\widetilde\{z\}\_\{\\theta\_\{m\},1\}\(x,a\)\\leq\\cdots\\leq\\widetilde\{z\}\_\{\\theta\_\{m\},K\}\(x,a\)denote these sorted outputs\. The sorted values are used only in subsequent decision\-making and learning targets\. When training the critic, the original outputzθm,j​\(x,a\)z\_\{\\theta\_\{m\},j\}\(x,a\)remains associated with its fixed quantile levelηj\\eta\_\{j\}\.

#### Approximation III: Buffered Policy Improvement

For a given augmented inputxxand actionaa, the lower\-buffered value of the remaining return isQτβ​\(Zπ​\(x,a\)\)=1β​∫τ−βτQu​\(Zπ​\(x,a\)\)​𝑑uQ\_\{\\tau\}^\{\\beta\}\\\!\\left\(Z^\{\\pi\}\(x,a\)\\right\)=\\frac\{1\}\{\\beta\}\\int\_\{\\tau\-\\beta\}^\{\\tau\}Q\_\{u\}\\\!\\left\(Z^\{\\pi\}\(x,a\)\\right\)\\,du\. This quantity averages the return quantiles betweenτ−β\\tau\-\\betaandτ\\tau\. Ideally, actions would be compared usingQτβ​\(Zπ​\(x,a\)\)Q\_\{\\tau\}^\{\\beta\}\\\!\\left\(Z^\{\\pi\}\(x,a\)\\right\)\. InDeep–BQRL, however, the full quantile functionQu​\(Zπ​\(x,a\)\)Q\_\{u\}\(Z^\{\\pi\}\(x,a\)\)is not available\. Approximation II provides onlyKKpredicted quantiles\. Each levelηj\\eta\_\{j\}corresponds to the interval\[\(j−1\)/K,j/K\]\[\(j\-1\)/K,j/K\]\. We therefore use the predicted quantile atηj\\eta\_\{j\}to represent the portion of this interval that lies inside\[τ−β,τ\]\[\\tau\-\\beta,\\tau\]\. The length of this overlap is

wjτ,β≔\[min\{τ,jK\}−max\{τ−β,j−1K\}\]\+,j=1,…,K,w\_\{j\}^\{\\tau,\\beta\}\\coloneqq\\left\[\\min\\left\\\{\\tau,\\frac\{j\}\{K\}\\right\\\}\-\\max\\left\\\{\\tau\-\\beta,\\frac\{j\-1\}\{K\}\\right\\\}\\right\]\_\{\+\},\\qquad j=1,\\ldots,K,\(1\)where\[v\]\+≔max⁡\{v,0\}\[v\]\_\{\+\}\\coloneqq\\max\\\{v,0\\\}\. Using the sorted predicted returns from criticmm, we approximateQτβ​\(Zπ​\(x,a\)\)Q\_\{\\tau\}^\{\\beta\}\\\!\\left\(Z^\{\\pi\}\(x,a\)\\right\)byB^m​\(x,a\)≔1β​∑j=1Kwjτ,β​z~θm,j​\(x,a\)\\widehat\{B\}\_\{m\}\(x,a\)\\coloneqq\\frac\{1\}\{\\beta\}\\sum\_\{j=1\}^\{K\}w\_\{j\}^\{\\tau,\\beta\}\\widetilde\{z\}\_\{\\theta\_\{m\},j\}\(x,a\)\. Thus,B^m​\(x,a\)\\widehat\{B\}\_\{m\}\(x,a\)is the buffered score assigned to actionaaby criticmm\. BecauseDeep–BQRLmaintainsMMcritics, we average their scores:μB​\(x,a\)≔1M​∑m=1MB^m​\(x,a\)\\mu\_\{B\}\(x,a\)\\coloneqq\\frac\{1\}\{M\}\\sum\_\{m=1\}^\{M\}\\widehat\{B\}\_\{m\}\(x,a\)\. After training, the policy selects the action with the largest ensemble\-mean buffered score:

π^​\(x\)∈arg⁡maxa∈𝒜​μB​\(x,a\)\.\\widehat\{\\pi\}\(x\)\\in\\arg\\max\_\{a\\in\\mathcal\{A\}\}\\mu\_\{B\}\(x,a\)\.\(2\)

### 3\.1Critic Learning and Exploration

The agent stores observed transitions in a replay buffer𝒟\\mathcal\{D\}\. A replay sample is denoted by\(x,a,r,x′,d\)\(x,a,r,x^\{\\prime\},d\), wherexxandx′x^\{\\prime\}are consecutive augmented inputs,aais the selected action,rris the observed reward, andd∈\{0,1\}d\\in\\\{0,1\\\}\. We setd=1d=1when the environment terminates or when the transition reaches the finite\-horizon boundary; otherwise,d=0d=0\. To stabilize training, for each criticzθmz\_\{\\theta\_\{m\}\}we maintain a slowly updated copyzθ¯mz\_\{\\bar\{\\theta\}\_\{m\}\}, called the target critic\. The target critic is used to evaluate the next augmented input when constructing the training targets\.

For each criticmm, we first evaluate its target criticzθ¯mz\_\{\\bar\{\\theta\}\_\{m\}\}at the next augmented inputx′x^\{\\prime\}and use these predictions to select the next action\. After sorting the target\-critic outputs, defineB¯m​\(x′,a\)≔1β​∑j=1Kwjτ,β​z~θ¯m,j​\(x′,a\)\\overline\{B\}\_\{m\}\(x^\{\\prime\},a\)\\coloneqq\\frac\{1\}\{\\beta\}\\sum\_\{j=1\}^\{K\}w\_\{j\}^\{\\tau,\\beta\}\\widetilde\{z\}\_\{\\bar\{\\theta\}\_\{m\},j\}\(x^\{\\prime\},a\)\. For a nonterminal next inputx′x^\{\\prime\}, criticmmselects

am\+​\(x′\)∈arg⁡maxa∈𝒜​B¯m​\(x′,a\)\.a\_\{m\}^\{\+\}\(x^\{\\prime\}\)\\in\\arg\\max\_\{a\\in\\mathcal\{A\}\}\\overline\{B\}\_\{m\}\(x^\{\\prime\},a\)\.\(3\)The corresponding target return values areym,i≔r\+\(1−d\)​z~θ¯m,i​\(x′,am\+​\(x′\)\)y\_\{m,i\}\\coloneqq r\+\(1\-d\)\\widetilde\{z\}\_\{\\bar\{\\theta\}\_\{m\},i\}\\\!\\left\(x^\{\\prime\},a\_\{m\}^\{\+\}\(x^\{\\prime\}\)\\right\),i=1,…,Ki=1,\\ldots,K\. Whend=1d=1, all target values reduce toym,i=ry\_\{m,i\}=r, and the continuation action need not be evaluated\.

Following the Huber loss of[Huber \(1992\)](https://arxiv.org/html/2609.21327#bib.bib12)and its quantile\-regression formulation in distributional RL\([Dabney et al\., 2018](https://arxiv.org/html/2609.21327#bib.bib4)\), we train each critic using the quantile Huber loss\. For a scalar residualδ\\deltaand thresholdκ\>0\\kappa\>0, define the Huber loss

hκ​\(δ\)≔\{12​δ2,\|δ\|≤κ,κ⁡\(\|δ\|−12​κ\),\|δ\|\>κ\.h\_\{\\kappa\}\(\\delta\)\\coloneqq\\begin\{cases\}\\frac\{1\}\{2\}\\delta^\{2\},&\|\\delta\|\\leq\\kappa,\\\\\[2\.84526pt\] \\kappa\\left\(\|\\delta\|\-\\frac\{1\}\{2\}\\kappa\\right\),&\|\\delta\|\>\\kappa\.\\end\{cases\}The corresponding quantile Huber loss isρηjκ\(δ\)≔\|ηj−𝟙\{δ<0\}\|hκ​\(δ\)κ\\rho\_\{\\eta\_\{j\}\}^\{\\kappa\}\(\\delta\)\\coloneqq\\left\|\\eta\_\{j\}\-\\mathbbm\{1\}\\\{\\delta<0\\\}\\right\|\\frac\{h\_\{\\kappa\}\(\\delta\)\}\{\\kappa\}\. The Huber loss is quadratic for small residuals and linear for large residuals, while the weight\|ηj−𝟙\{δ<0\}\|\\left\|\\eta\_\{j\}\-\\mathbbm\{1\}\\\{\\delta<0\\\}\\right\|in quantile Huber loss directs the prediction toward theηj\\eta\_\{j\}\-quantile of the target return distribution\.

For a minibatchℬ⊂𝒟\\mathcal\{B\}\\subset\\mathcal\{D\}, criticmmminimizes

𝒥m​\(θm\)≔1\|ℬ\|​K2​∑\(x,a,r,x′,d\)∈ℬ∑i=1K∑j=1Kρηjκ​\(ym,i−zθm,j​\(x,a\)\)\.\\mathcal\{J\}\_\{m\}\(\\theta\_\{m\}\)\\coloneqq\\frac\{1\}\{\|\\mathcal\{B\}\|K^\{2\}\}\\sum\_\{\(x,a,r,x^\{\\prime\},d\)\\in\\mathcal\{B\}\}\\sum\_\{i=1\}^\{K\}\\sum\_\{j=1\}^\{K\}\\rho\_\{\\eta\_\{j\}\}^\{\\kappa\}\\left\(y\_\{m,i\}\-z\_\{\\theta\_\{m\},j\}\(x,a\)\\right\)\.\(4\)Each ensemble member is updated by minimizing its corresponding quantile\-regression loss𝒥m​\(θm\)\\mathcal\{J\}\_\{m\}\(\\theta\_\{m\}\)\. This is the quantile\-regression distributional Bellman loss\([Dabney et al\., 2018](https://arxiv.org/html/2609.21327#bib.bib4)\), applied to the augmented trajectory representation\. Minimizing this loss updates the critic parameters so thatzθm,j​\(x,a\)z\_\{\\theta\_\{m\},j\}\(x,a\)approximates theηj\\eta\_\{j\}\-quantile of the conditional remaining\-return distribution\. The parameters\(τ,β\)\(\\tau,\\beta\)do not appear explicitly in the quantile Huber loss; they affect training through the buffered scores used to select actions and construct the target values\.

After each gradient update, the target parameters are updated using a soft target\-network update\([Lillicrap et al\., 2015](https://arxiv.org/html/2609.21327#bib.bib14)\),

θ¯m←\(1−ζ\)θ¯m\+ζθm,m=1,…,M,\\bar\{\\theta\}\_\{m\}\\leftarrow\(1\-\\zeta\)\\bar\{\\theta\}\_\{m\}\+\\zeta\\theta\_\{m\},\\qquad m=1,\\ldots,M,\(5\)whereζ∈\(0,1\]\\zeta\\in\(0,1\]controls the update rate\.

During data collection,Deep–BQRLencourages exploration by adding a bonus to actions for which the critics disagree more strongly about the buffered score\. DefineσB​\(x,a\)≔\[1M−1​∑m=1M\(B^m​\(x,a\)−μB​\(x,a\)\)2\]1/2\\sigma\_\{B\}\(x,a\)\\coloneqq\[\\frac\{1\}\{M\-1\}\\sum\_\{m=1\}^\{M\}\(\\widehat\{B\}\_\{m\}\(x,a\)\-\\mu\_\{B\}\(x,a\)\)^\{2\}\]^\{1/2\}\. During episodett, the behavior policy selectsAh∈arg⁡maxa∈𝒜​\{μB​\(Xh,a\)\+λt​σB​\(Xh,a\)\}A\_\{h\}\\in\\arg\\max\_\{a\\in\\mathcal\{A\}\}\\left\\\{\\mu\_\{B\}\(X\_\{h\},a\)\+\\lambda\_\{t\}\\sigma\_\{B\}\(X\_\{h\},a\)\\right\\\}, whereλt≔λ01\+t/λdecay\\lambda\_\{t\}\\coloneqq\\frac\{\\lambda\_\{0\}\}\{1\+t/\\lambda\_\{\\mathrm\{decay\}\}\}is the exploration coefficient\.

Algorithm[3\.1](https://arxiv.org/html/2609.21327#S3.SS1)combines the augmented trajectory representation, the quantile representation of remaining returns, and buffered policy improvement\. Quantile regression trains the critics, while ensemble disagreement guides data collection\.

Algorithm 1Deep–BQRLfor discrete\-action buffered\-quantile control

1:Input:target level

τ∈\(0,1\)\\tau\\in\(0,1\); buffer

β∈\(0,τ\]\\beta\\in\(0,\\tau\]; number of quantiles

KK; number of critics

M≥2M\\geq 2; Huber threshold

κ\\kappa; target\-update rate

ζ\\zeta; exploration parameters

λ0\\lambda\_\{0\}and

λdecay\\lambda\_\{\\mathrm\{decay\}\}\.

2:Initialize replay buffer

𝒟\\mathcal\{D\}, critics

\{zθm\}m=1M\\\{z\_\{\\theta\_\{m\}\}\\\}\_\{m=1\}^\{M\}, and target critics

\{zθ¯m\}m=1M\\\{z\_\{\\bar\{\\theta\}\_\{m\}\}\\\}\_\{m=1\}^\{M\}with

θ¯m←θm\\bar\{\\theta\}\_\{m\}\\leftarrow\\theta\_\{m\}\.

3:Compute the weights

\{wjτ,β\}j=1K\\\{w\_\{j\}^\{\\tau,\\beta\}\\\}\_\{j=1\}^\{K\}using equation[1](https://arxiv.org/html/2609.21327#S3.E1)\.

4:foreach training episode

ttdo

5:Observe the initial state

ssand set

x←\(0,s,0\)x\\leftarrow\(0,s,0\)\.

6:for

h=0,…,H−1h=0,\\ldots,H\-1do

7:Compute

B^m​\(x,a′\)\\widehat\{B\}\_\{m\}\(x,a^\{\\prime\}\)∀m∈\{1,…,M\}\\forall m\\\!\\in\\\!\\\{\\\!1,\\\!\\ldots\\\!,\\\!M\\\!\\\}and

a′∈𝒜a^\{\\prime\}\\\!\\in\\\!\\mathcal\{A\}\. Then compute

μB​\(x,a′\)\\mu\_\{B\}\(x,a^\{\\prime\}\)and

σB​\(x,a′\)\\sigma\_\{B\}\(x,a^\{\\prime\}\)\.

8:Select

a∈arg⁡maxa′∈𝒜​\{μB​\(x,a′\)\+λt​σB​\(x,a′\)\}a\\in\\arg\\max\_\{a^\{\\prime\}\\in\\mathcal\{A\}\}\\left\\\{\\mu\_\{B\}\(x,a^\{\\prime\}\)\+\\lambda\_\{t\}\\sigma\_\{B\}\(x,a^\{\\prime\}\)\\right\\\}\.

9:Execute

aaand observe reward

rrand next state

s′s^\{\\prime\}\.

10:Set

d=1d=1if the episode terminates or

h=H−1h=H\-1; otherwise set

d=0d=0\.

11:If

x=\(h,s,c\)x=\(h,s,c\), set

x′←\(h\+1,s′,c\+r\)x^\{\\prime\}\\leftarrow\(h\+1,s^\{\\prime\},c\+r\)\.

12:Store

\(x,a,r,x′,d\)\(x,a,r,x^\{\\prime\},d\)in

𝒟\\mathcal\{D\}\.

13:if

𝒟\\mathcal\{D\}contains enough samples for a minibatchthen

14:Sample

ℬ⊂𝒟\\mathcal\{B\}\\subset\\mathcal\{D\}\.

15:for

m=1,…,Mm=1,\\ldots,Mdo

16:For each

\(x,a,r,x′,d\)∈ℬ\(x,a,r,x^\{\\prime\},d\)\\in\\mathcal\{B\}, evaluate and sort the target\-critic outputs at

x′x^\{\\prime\}\.

18:Compute

B¯m​\(x′,a′\)\\overline\{B\}\_\{m\}\(x^\{\\prime\},a^\{\\prime\}\)∀a′∈𝒜\\forall a^\{\\prime\}\\in\\mathcal\{A\}and select

am\+​\(x′\)a\_\{m\}^\{\+\}\(x^\{\\prime\}\)using equation[3](https://arxiv.org/html/2609.21327#S3.E3)\.

19:Construct

\{ym,i\}i=1K\\\{y\_\{m,i\}\\\}\_\{i=1\}^\{K\}using the target values defined above\.

20:else

21:Set

ym,i←ry\_\{m,i\}\\leftarrow rfor

i=1,…,Ki=1,\\ldots,K\.

22:endif

23:Update

θm\\theta\_\{m\}by minimizing

𝒥m​\(θm\)\\mathcal\{J\}\_\{m\}\(\\theta\_\{m\}\)in equation[4](https://arxiv.org/html/2609.21327#S3.E4)\.

24:Update

θ¯m\\bar\{\\theta\}\_\{m\}using equation[5](https://arxiv.org/html/2609.21327#S3.E5)\.

25:endfor

26:endif

28:break

29:else

30:

x←x′x\\leftarrow x^\{\\prime\}
31:endif

32:endfor

33:endfor

34:Output:

π^\\widehat\{\\pi\}defined in equation[2](https://arxiv.org/html/2609.21327#S3.E2)\.

## 4Results

We evaluateDeep–BQRLon the finite\-horizon asset\-selling problem\([Seierstad, 1992](https://arxiv.org/html/2609.21327#bib.bib18);[Rosenfield et al\., 1983](https://arxiv.org/html/2609.21327#bib.bib15)\)studied by[Alipour\-Vaezi et al\. \(2026a\)](https://arxiv.org/html/2609.21327#bib.bib1)\. At each decision period, a seller observes an offer and chooses whether to accept it or wait for another\. Accepting yields a known reward, while waiting offers the possibility of a higher reward but also the risk of receiving less\. This makes the problem useful for evaluating both learning performance and how different target quantiles affect decisions\. We also evaluateDeep–BQRLon slippery FrozenLake, a stochastic navigation problem with sparse terminal rewards, to test the method under a different transition structure\.

We compareDeep–BQRLwith the model\-basedUCB–BQRLmethod\([Alipour\-Vaezi et al\., 2026a](https://arxiv.org/html/2609.21327#bib.bib1)\)and the PPO and TRPO implementations used in our experiments\([Schulman et al\., 2017](https://arxiv.org/html/2609.21327#bib.bib17);[Schulman et al\., 2015](https://arxiv.org/html/2609.21327#bib.bib16)\)\. The state space in both environments are small enough for the model\-based method to remain computationally feasible, providing a reference for evaluating the model\-free approach\. For both environments, hyperparameters are selected independently of the held\-out evaluation runs\. Following[Alipour\-Vaezi et al\. \(2026a\)](https://arxiv.org/html/2609.21327#bib.bib1), structural choices are fixed, and one dominant hyperparameter per method is varied over a1616\-point logarithmic grid\. We use successive halving with reduction factorη=2\\eta=2and increasing training budgets\. Validation uses seeds10001000,10011001, and10021002; configurations are ranked primarily by cumulativeτ\\tau\-quantile policy performance, with late\-training episodic return as a tie\-breaker\. Final results use the disjoint seeds4242,1004210042, and2004220042\.

### 4\.1Asset\-Selling

The environment has2525offer states,s∈\{0,…,24\}s\\in\\\{0,\\ldots,24\\\}, and an absorbing post\-sale state\. Each episode starts atS0=5S\_\{0\}=5and allows at mostH=10H=10decisions\. At each state, the action set is𝒜=\{Stop,Continue\}\\mathcal\{A\}=\\\{\\mathrm\{Stop\},\\mathrm\{Continue\}\\\}\. ChoosingStop\\mathrm\{Stop\}accepts the current offer, yields rewards/24s/24, and ends the episode; division by2424normalizes the sale reward to\[0,1\]\[0,1\]\. ChoosingContinue\\mathrm\{Continue\}rejects the offer, yields zero immediate reward, and draws a new offer independently and uniformly from\{0,…,24\}\\\{0,\\ldots,24\\\}\. If no offer is accepted within the horizon, the episode ends with zero return\. Thus, the total episodic return is the normalized accepted offer, or zero if no sale occurs\. The decision trades off accepting the current offer against waiting for an uncertain future offer\. A lower target quantile emphasizes protection against low sale rewards, whereas a higher target quantile emphasizes more favorable outcomes\. The learned policy determines how this preference affects stopping decisions across offers and decision periods\. These experiments use2,0002\{,\}000training episodes per run and three independent seeds\. We report learning curves forτ∈\{0\.1,0\.9\}\\tau\\in\\\{0\.1,0\.9\\\}and selected final policies forτ∈\{0\.1,0\.5,0\.9\}\\tau\\in\\\{0\.1,0\.5,0\.9\\\}\. Curves show the mean across the three runs, with shaded pointwise95%95\\%Student\-ttintervals\. Given only three runs, these intervals summarize variability rather than provide evidence of statistically significant differences\.

We report two cumulative performance measures\. LetGtG\_\{t\}denote the realized return in training episodett\. The empirical cumulative expected\-return regret isReg^E​\(k\)≔∑t=1k\(maxπ⁡𝔼⁡\[Rπ\]−Gt\)\\widehat\{\\operatorname\{Reg\}\}\_\{E\}\(k\)\\coloneqq\\sum\_\{t=1\}^\{k\}\\left\(\\max\_\{\\pi\}\\mathbb\{E\}\[R^\{\\pi\}\]\-G\_\{t\}\\right\), which compares rewards collected by the behavior policy, including exploration, with the optimal expected episodic return under the true environment\. To evaluate the learned policy without the exploration bonus, letπ¯t\\bar\{\\pi\}\_\{t\}denote the most recently evaluated greedy policy available at episodett\. We report the cumulativeτ\\tau\-quantile policy gap𝒢^τ​\(k\)≔∑t=1k\[Qτ⋆−Qτ​\(Rπ¯t\)\]\+\\widehat\{\\mathcal\{G\}\}\_\{\\tau\}\(k\)\\coloneqq\\sum\_\{t=1\}^\{k\}\\left\[Q\_\{\\tau\}^\{\\star\}\-Q\_\{\\tau\}\(R^\{\\bar\{\\pi\}\_\{t\}\}\)\\right\]\_\{\+\}, whereQτ⋆Q\_\{\\tau\}^\{\\star\}is the common numerical point\-quantile reference computed from the true environment model\. Policy evaluation is periodic, and the latest evaluated value is retained between evaluations\. The two measures capture different aspects of learning: expected\-return regret reflects rewards collected during training, while the quantile policy gap reflects the quality of the evaluated greedy policy at the prescribed quantile level, so their magnitudes should not be compared directly\. AlthoughDeep–BQRLselects actions using the lower\-buffered criterionQτβQ\_\{\\tau\}^\{\\beta\}, the policy\-gap measure evaluates the target point quantileQτQ\_\{\\tau\}\.

Figure[1](https://arxiv.org/html/2609.21327#S4.F1)compares the cumulative measures forτ=0\.1\\tau=0\.1andτ=0\.9\\tau=0\.9\. At episode20002000, the mean cumulative point\-quantile gap ofDeep–BQRLis538\.33538\.33forτ=0\.1\\tau=0\.1, compared with583\.63583\.63for PPO and855\.22855\.22for TRPO; forτ=0\.9\\tau=0\.9, the corresponding values are92\.7892\.78,173\.51173\.51, and221\.36221\.36\. Thus,Deep–BQRLhas smaller gaps than both policy\-gradient baselines at both target levels, while model\-basedUCB–BQRLremains substantially stronger, with gaps of28\.1428\.14and0\.790\.79\. Expected\-return regret shows a different pattern\. Forτ=0\.1\\tau=0\.1,Deep–BQRLreaches1025\.441025\.44, compared with229\.00229\.00forUCB–BQRL,719\.58719\.58for PPO, and685\.96685\.96for TRPO\. Forτ=0\.9\\tau=0\.9,Deep–BQRLhas the smallest regret,193\.68193\.68, compared with324\.65324\.65,719\.58719\.58, and685\.96685\.96, respectively\. Thus, a smaller point\-quantile gap does not necessarily imply greater reward during training:Deep–BQRLimproves on PPO and TRPO under the point\-quantile measure, while its expected\-return performance depends on the target level\.

Figure 1:Learning performance on the asset\-selling environment forτ∈\{0\.1,0\.9\}\\tau\\in\\\{0\.1,0\.9\\\}\. The top row reports the cumulativeτ\\tau\-quantile policy gap, while the bottom row reports cumulative regret under the expected\-return criterion\. Curves show the mean over three independent seeds; shaded regions denote pointwise95%95\\%Student\-ttintervals\. Lower values indicate better performance\.Figure[2](https://arxiv.org/html/2609.21327#S4.F2)shows selected finalDeep–BQRLpolicies, where blue denotesContinue\\mathrm\{Continue\}and gray denotesStop\\mathrm\{Stop\}\. Each panel shows one run, with seeds1004210042,4242, and2004220042forτ=0\.1\\tau=0\.1,0\.50\.5, and0\.90\.9, respectively\. Overall, smaller target quantiles lead to accepting lower offers, while larger target quantiles favor waiting for higher offers\. For example, in the second decision period, theτ=0\.1\\tau=0\.1,0\.50\.5, and0\.90\.9policies accept offers of at least1616,2121, and2222, respectively, illustrating the shift from protecting against low returns to pursuing more favorable outcomes\. The willingness to wait generally decreases toward the horizon, although the learned boundaries are not perfectly regular\. Some positive final\-period offers are still rejected despite no remaining sale opportunity \(e\.g\., ath=10h=10, offers11–99,11–44, and11–55forτ=0\.1\\tau=0\.1,0\.50\.5, and0\.90\.9, respectively\)\. Thus, the figure shows learned approximations rather than exact optimal stopping rules\.

Figure 2:Selected finalDeep–BQRLpolicies in the asset\-selling environment forτ∈\{0\.1,0\.5,0\.9\}\\tau\\in\\\{0\.1,0\.5,0\.9\\\}, evaluated atCh=0C\_\{h\}=0\. Blue cells indicateContinue\\mathrm\{Continue\}and gray cells indicateStop\\mathrm\{Stop\}\. The panels show individual runs with seeds1004210042,4242, and2004220042, respectively, rather than policies averaged across runs\. The absorbing post\-sale state is omitted\.
### 4\.2Slippery FrozenLake

The standard4×44\\times 4slippery FrozenLake environment has1616states and actions𝒜=\{Left,Down,Right,Up\}\\mathcal\{A\}=\\\{\\textsc\{Left\},\\textsc\{Down\},\\textsc\{Right\},\\textsc\{Up\}\\\}\. Each episode begins at a fixed start state and lasts at mostH=100H=100steps\. Frozen cells are safe, entering a hole ends the episode with zero reward, and reaching the goal ends it with reward one; all other rewards are zero\. Movement is stochastic under the slippery dynamics\. Thus,Rπ∈\{0,1\}R^\{\\pi\}\\in\\\{0,1\\\}, and the expected episodic return equals the probability of reaching the goal before entering a hole or reaching the horizon\. We report5,0005\{,\}000training episodes per run forτ∈\{0\.1,0\.5,0\.9\}\\tau\\in\\\{0\.1,0\.5,0\.9\\\}\. Curves show means across the three held\-out seeds, with shaded pointwise95%95\\%Student\-ttintervals interpreted as descriptive summaries of variability\. We use the two cumulative measures defined in Section[4\.1](https://arxiv.org/html/2609.21327#S4.SS1)and additionally report a5050\-episode moving average ofGtG\_\{t\}, which summarizes the observed frequency of reaching the goal during training\.

Figure[3](https://arxiv.org/html/2609.21327#S4.F3)shows faster initial learning forUCB–BQRL, whileDeep–BQRLimproves more gradually\. At episode50005000, the mean5050\-episode moving\-average rewards ofDeep–BQRLare0\.6270\.627,0\.6870\.687, and0\.6730\.673forτ=0\.1\\tau=0\.1,0\.50\.5, and0\.90\.9, respectively, compared with0\.7130\.713,0\.6400\.640, and0\.6800\.680forUCB–BQRL\. PPO reaches approximately0\.510\.51–0\.530\.53, whereas TRPO remains near zero\. Thus,Deep–BQRLachieves episodic reward close to the model\-based reference, with a numerically higher terminal mean atτ=0\.5\\tau=0\.5\.

Figure 3:Episodic reward on the slippery FrozenLake environment forτ∈\{0\.1,0\.5,0\.9\}\\tau\\in\\\{0\.1,0\.5,0\.9\\\}\. Curves report the 50\-episode moving\-average reward averaged over three independent seeds; shaded regions denote pointwise 95% Student\-ttconfidence intervals\.Figure 4:Learning performance on the slippery FrozenLake environment forτ=0\.5\\tau=0\.5\. The top panel reports the cumulativeτ\\tau\-quantile policy gap, while the bottom panel reports cumulative regret under the expected\-return criterion\. Lower values indicate better performance\.The binary return limits what point\-quantile comparisons can reveal\. For a policy with success probabilitypπp\_\{\\pi\},Qτ​\(Rπ\)=0Q\_\{\\tau\}\(R^\{\\pi\}\)=0whenτ≤1−pπ\\tau\\leq 1\-p\_\{\\pi\}and11otherwise\. Point\-quantile performance therefore changes only when the success probability crosses1−τ1\-\\tau\. Atτ=0\.1\\tau=0\.1, the cumulative point\-quantile gaps are zero for all evaluated policies and do not distinguish the methods\. We therefore useτ=0\.5\\tau=0\.5for the cumulative comparisons in Figure[4](https://arxiv.org/html/2609.21327#S4.F4)\. At episode50005000, the mean cumulative point\-quantile gaps are9\.679\.67forUCB–BQRL,400\.0400\.0forDeep–BQRL,3313\.333313\.33for PPO, and5000\.05000\.0for TRPO\. TheDeep–BQRLgap shows essentially no further increase after approximately10001000episodes\. The corresponding mean cumulative expected\-return regrets are404\.28404\.28,666\.28666\.28,2426\.622426\.62, and3613\.283613\.28, respectively\. Thus,Deep–BQRLranks second to the model\-based reference under both cumulative measures, improving on the tested policy\-gradient baselines while maintaining competitive episodic reward\. These results evaluate the complete method rather than isolate the effects of neural function approximation, buffered action selection, or the exploration bonus\.

## 5Discussion

The results support the feasibility of learning risk\-sensitive decisions from return quantiles without explicit transition estimation or return\-law planning\. In asset selling,Deep–BQRLachieves smaller mean cumulative point\-quantile policy gaps than PPO and TRPO at both reported target levels, althoughUCB–BQRLremains substantially stronger\. FrozenLake provides complementary evidence:Deep–BQRLapproaches the model\-based reference in episodic reward and outperforms the policy\-gradient baselines in the median point\-quantile comparison\. Overall, these findings supportDeep–BQRLas a practical quantile\-sensitive approach, but not as a uniformly better alternative to model\-based planning\.

The asset\-selling results also show that risk\-sensitive performance and training rewards should be evaluated separately\. Atτ=0\.1\\tau=0\.1,Deep–BQRLhas a smaller mean point\-quantile policy gap than PPO and TRPO but a larger expected\-return regret; atτ=0\.9\\tau=0\.9, it has the smallest mean expected\-return regret\. Because the point\-quantile gap evaluates the greedy policy while expected\-return regret includes exploration, these differences may reflect both the learned policy and data collection rather than the risk preference alone\. Evaluating both expectation and point quantile for the same final policy would better separate these effects\.

FrozenLake provides a different comparison because its binary return is either zero or one\. A better policy therefore has a higher probability of reaching the goal, which improves expected return and cannot worsen the point or lower\-buffered quantile\. Thus, FrozenLake tests learning under stochastic transitions and sparse rewards, but different target quantiles need not produce different optimal policies\. Moreover, once the goal\-reaching probability exceeds0\.50\.5, the median return is one, so further improvements are not captured by the median\-gap measure\. Hence, a flat median\-gap curve does not necessarily indicate the highest attainable success probability\.

A methodological strength ofDeep–BQRLis the direct link between the predicted return distribution and action selection\. Each critic estimates fixed return quantiles, and the buffered score combines those immediately below the target level, incorporating the target quantile directly into decisions learned from sampled transitions\. The method requires neither transition\-model estimation nor construction of achievable return distributions during training\. The controlled benchmarks also allow the learned policies to be evaluated under the true environment model rather than only through the critics’ predictions\. The difference between the learning criterion and evaluation measure is important\.Deep–BQRLselects actions usingQτβQ\_\{\\tau\}^\{\\beta\}, while the reported policy gap evaluatesQτQ\_\{\\tau\}\. With a nonzero buffer, these criteria may rank policies differently, and choosing actions by conditional buffered scores does not guarantee an optimal episodic point\-quantile policy\. The results therefore support the usefulness of the approximation empirically, rather than guarantee point\-quantile optimality\. The choices ofβ\\betaandKKalso require further study:β\\betadetermines which part of the return distribution is used for action selection, whileKKcontrols how finely that region is represented\. Sensitivity analysis could clarify their effects and whether buffering improves learning over using a single estimated quantile\.

A key direction for future work is to evaluateDeep–BQRLin larger, higher\-dimensional environments\. The current experiments use small state spaces whereUCB–BQRLremains computationally feasible, so they do not establish scalability\. Experiments on Atari games and other high\-dimensional discrete\-control benchmarks would test whetherDeep–BQRLretains its quantile\-sensitive behavior when neural function approximation is necessary\. Future studies should also consider longer horizons, larger action spaces, computational cost, and sensitivity to the buffer width, number of quantiles, and ensemble size\.

## References

- Alipour\-Vaezi et al\. \(2026a\)Mohammad Alipour\-Vaezi, Huaiyang Zhong, and Sajad Khodadadian\.Risk\-sensitive reinforcement learning with smoothed quantile objectives\.*arXiv preprint arXiv:2608\.22227*, 2026a\.
- Alipour\-Vaezi et al\. \(2026b\)Mohammad Alipour\-Vaezi, Huaiyang Zhong, Kwok leung Tsui, and Sajad Khodadadian\.Optimistic reinforcement learning with quantile objectives\.In*The 29th International Conference on Artificial Intelligence and Statistics*, 2026b\.
- Bellemare et al\. \(2017\)Marc G Bellemare, Will Dabney, and Rémi Munos\.A distributional perspective on reinforcement learning\.In*International conference on machine learning*, pp\. 449–458\. PMLR, 2017\.
- Dabney et al\. \(2018\)Will Dabney, Mark Rowland, Marc Bellemare, and Rémi Munos\.Distributional reinforcement learning with quantile regression\.In*Proceedings of the AAAI conference on artificial intelligence*, volume 32, 2018\.
- Delage & Mannor \(2010\)Erick Delage and Shie Mannor\.Percentile optimization for markov decision processes with parameter uncertainty\.*Operations research*, 58\(1\):203–213, 2010\.
- Embrechts et al\. \(2018\)Paul Embrechts, Haiyan Liu, and Ruodu Wang\.Quantile\-based risk sharing\.*Operations Research*, 66\(4\):936–949, 2018\.
- Filar et al\. \(1995\)Jerzy A Filar, Dmitry Krass, and Keith W Ross\.Percentile performance criteria for limiting average markov decision processes\.*IEEE Transactions on Automatic Control*, 40\(1\):2–10, 1995\.
- Fissler & Ziegel \(2021\)Tobias Fissler and Johanna F Ziegel\.On the elicitability of range value at risk\.*Statistics & risk modeling*, 38\(1\-2\):25–46, 2021\.
- Garcıa & Fernández \(2015\)Javier Garcıa and Fernando Fernández\.A comprehensive survey on safe reinforcement learning\.*Journal of Machine Learning Research*, 16\(1\):1437–1480, 2015\.
- Gottesman et al\. \(2019\)Omer Gottesman, Fredrik Johansson, Matthieu Komorowski, Aldo Faisal, David Sontag, Finale Doshi\-Velez, and Leo Anthony Celi\.Guidelines for reinforcement learning in healthcare\.*Nature medicine*, 25\(1\):16–18, 2019\.
- Howard & Matheson \(1972\)Ronald A Howard and James E Matheson\.Risk\-sensitive markov decision processes\.*Management science*, 18\(7\):356–369, 1972\.
- Huber \(1992\)Peter J Huber\.Robust estimation of a location parameter\.In*Breakthroughs in statistics: Methodology and distribution*, pp\. 492–518\. Springer, 1992\.
- Li et al\. \(2022\)Xiaocheng Li, Huaiyang Zhong, and Margaret L Brandeau\.Quantile markov decision processes\.*Operations research*, 70\(3\):1428–1447, 2022\.
- Lillicrap et al\. \(2015\)Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra\.Continuous control with deep reinforcement learning\.*arXiv preprint arXiv:1509\.02971*, 2015\.
- Rosenfield et al\. \(1983\)Donald B Rosenfield, Roy D Shapiro, and David A Butler\.Optimal strategies for selling an asset\.*Management science*, 29\(9\):1051–1061, 1983\.
- Schulman et al\. \(2015\)John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz\.Trust region policy optimization\.In*International conference on machine learning*, pp\. 1889–1897\. Pmlr, 2015\.
- Schulman et al\. \(2017\)John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov\.Proximal policy optimization algorithms\.*arXiv preprint arXiv:1707\.06347*, 2017\.
- Seierstad \(1992\)Atle Seierstad\.Reservation prices in optimal stopping\.*Operations research*, 40\(2\):409–415, 1992\.
- Sutton et al\. \(1998\)Richard S Sutton, Andrew G Barto, et al\.*Reinforcement learning: An introduction*, volume 1\.MIT press Cambridge, 1998\.