Adaptive Finite-Budget Training for CVaR Risk-Aware Q-Learning

# Adaptive Finite-Budget Training for CVaR Risk-Aware Q-Learning
Source: [https://arxiv.org/html/2608.04305](https://arxiv.org/html/2608.04305)
Yifan Wu[wuyifan0648@163\.com](https://arxiv.org/html/2608.04305v1/mailto:[email protected])Department of Data and Systems EngineeringThe University of Hong KongHong KongHong Kong SARJunjie Lei[junjie01@connect\.hku\.hk](https://arxiv.org/html/2608.04305v1/mailto:[email protected])Department of Data and Systems EngineeringThe University of Hong KongHong KongHong Kong SARandWenjie Huang[huangwj@hku\.hk](https://arxiv.org/html/2608.04305v1/mailto:[email protected])Department of Data and Systems Engineering & Musketeers Foundation Institute of Data ScienceThe University of Hong KongHong KongHong Kong SAR

###### Abstract\.

Risk\-aware Q\-learning \(RaQL\) provides a model\-free, two\-timescale estimator for dynamic risk objectives, but its finite\-budget behavior remains fragile: fixed inner\-loop hyperparameters can produce unstable value estimates, persistent Bellman residuals, and inefficient sample reuse\. This paper proposes an adaptive training controller for Conditional Value\-at\-Risk \(CVaR\) RaQL and evaluates it on a daily Bitcoin trading task\. The controller preserves the original CVaR estimator and Bellman fixed point; instead, it redesigns the training procedure through six coordinated mechanisms: per\-cell inner\-step sizing, outer\-rate\-matched decay synchronization, a short early correction for the VaR\-like inner variable, a coverage\-first\-then\-greedy sample allocation rule, progressive suffix aggregation of mature inner estimates, and data\-driven calibration of key scales from online\-observable quantities\. Across 20 random seeds and 856,000 inner\-transition samples, the controller reduces the mean empirical CVaR Bellman residual by approximately 85% relative to the fixed\-parameter baseline \(MeanBEQ: 1\.2202 to 0\.1854; MeanBEV: 1\.1624 to 0\.0535\) and maintains stability across CVaR levels, discount factors, and training budgets\. On the chronological out\-of\-sample test set, the learned policy attains a Sharpe ratio of 0\.9281 with a maximum drawdown of 6\.46% after transaction costs\. Although buy\-and\-hold yields a higher cumulative return \(35\.43% vs\. 23\.61%\), the adaptive policy achieves far lower volatility \(9\.57% vs\. 47\.93%\), drawdown, and CVaR loss\. These results demonstrate that adaptive finite\-budget training design, applied solely to the training procedure without altering the risk objective, can materially improve the reliability and risk\-adjusted performance of risk\-aware Q\-learning in financial applications\.

risk\-aware reinforcement learning, CVaR, Q\-learning, cryptocurrency trading, finite\-budget training

## 1\.Introduction

### 1\.1\.Risk\-Sensitive Reinforcement Learning in Finance and RaQL

Financial systems are often evaluated by downside exposure and tail losses rather than mean return alone\. This has motivated risk\-sensitive reinforcement\-learning \(RL\) applications in algorithmic trading, foreign exchange, hedging, and portfolio allocation\. Shen et al\.\(Shenet al\.,[2014](https://arxiv.org/html/2608.04305#bib.bib31)\)study risk\-averse algorithmic trading using high\-frequency NASDAQ data, while Bisi et al\.\(Bisiet al\.,[2020](https://arxiv.org/html/2608.04305#bib.bib32)\)combine fitted Q\-iteration with a risk–return objective for foreign\-exchange trading\. Vittori et al\.\(Vittoriet al\.,[2020](https://arxiv.org/html/2608.04305#bib.bib33)\)consider option hedging under transaction costs, and Vadori et al\.\(Vadoriet al\.,[2020](https://arxiv.org/html/2608.04305#bib.bib34)\), Coache et al\.\(Coacheet al\.,[2023](https://arxiv.org/html/2608.04305#bib.bib37)\), and Enkhsaikhan and Jo\(Enkhsaikhan and Jo,[2024](https://arxiv.org/html/2608.04305#bib.bib38)\)develop risk\-aware formulations for portfolio optimization and statistical arbitrage\. These studies introduce risk sensitivity through the reward objective, constraints, uncertainty representation, or policy architecture\.

Among the available risk measures, Conditional Value\-at\-Risk \(CVaR\) is particularly relevant because it averages losses in the worst1−α1\-\\alphatail beyond the Value\-at\-Risk threshold\. CVaR is coherent\(Artzneret al\.,[1999](https://arxiv.org/html/2608.04305#bib.bib17); Acerbi,[2002](https://arxiv.org/html/2608.04305#bib.bib18)\)and admits a tractable variational representation\(Rockafellar and Uryasev,[2000](https://arxiv.org/html/2608.04305#bib.bib2)\)\. In financial RL, CVaR has been incorporated into direct and deep trading methods\(Alameer and Alshehri,[2022](https://arxiv.org/html/2608.04305#bib.bib35); Cuiet al\.,[2023](https://arxiv.org/html/2608.04305#bib.bib36)\), while policy\-gradient and actor–critic methods have incorporated CVaR as an objective or constraint\(Chow and Ghavamzadeh,[2014](https://arxiv.org/html/2608.04305#bib.bib4); Prashanth,[2014](https://arxiv.org/html/2608.04305#bib.bib19)\)\. Distributional RL instead estimates return quantiles and extracts risk\-sensitive policies from the learned return distribution\(Dabneyet al\.,[2018](https://arxiv.org/html/2608.04305#bib.bib20); Lim and Malik,[2022](https://arxiv.org/html/2608.04305#bib.bib21); Chenet al\.,[2024](https://arxiv.org/html/2608.04305#bib.bib22)\)\.

Our starting point is the workHuang and Haskell \([2021](https://arxiv.org/html/2608.04305#bib.bib1)\), which develops risk\-aware Q\-learning \(RaQL\) under dynamic coherent risk measures\. RaQL replaces the conditional expectation in the Bellman recursion with a risk functional, estimates that functional through a stochastic approximation inner loop, and updates Q\-values in an outer loop\. For CVaR, the inner recursion tracks a VaR\-like minimizer\. As summarized byWang and Chapman \([2022](https://arxiv.org/html/2608.04305#bib.bib30)\),Huang and Haskell \([2021](https://arxiv.org/html/2608.04305#bib.bib1)\)are the first to establish almost\-sure convergence and an explicit non\-asymptotic rate for this model\-free two\-loop algorithm, building on classical Q\-learning and two\-timescale stochastic approximation\(Watkins and Dayan,[1992](https://arxiv.org/html/2608.04305#bib.bib5); Borkar and Meyn,[2000](https://arxiv.org/html/2608.04305#bib.bib7); Konda and Tsitsiklis,[1999](https://arxiv.org/html/2608.04305#bib.bib23)\)\.

The theory specifies schedules under which the coupled recursions converge, but it does not prescribe how to allocate a fixed sampling budget among inner risk estimation, outer Q\-updates, and state–action cells\. The allocation will affect the risk estimation error and Q\-updates error, and the two errors will be coupled during training\. Section 1\.2 and 1\.3 make this mechanism precise and motivate changing the training controller without changing the CVaR estimator or Bellman fixed point\.

*Adaptive training*has been studied extensively for risk\-neutral and single\-timescale learning agents\. It means that some part of the learning procedure is adjusted using the explored state–action pairs rather than fixed once before training\. Existing work illustrates several forms of adaptation\. Learning\-rate and sample\-efficiency analyses characterize how update schedules and exploration govern Q\-learning efficiency\(Even\-Dar and Mansour,[2003](https://arxiv.org/html/2608.04305#bib.bib8); Jinet al\.,[2018](https://arxiv.org/html/2608.04305#bib.bib24)\)\. Adaptive subgradient methods rescale coordinate\-wise updates using accumulated gradient information\(Duchiet al\.,[2011](https://arxiv.org/html/2608.04305#bib.bib25)\), while adaptive value\- and return\-normalization methods update the scale of learning targets from running statistics\(van Hasseltet al\.,[2016](https://arxiv.org/html/2608.04305#bib.bib26); Schaulet al\.,[2021](https://arxiv.org/html/2608.04305#bib.bib27)\)\. In exploration, count\-based methods reward rarely visited states through visitation\-based bonuses\(Bellemareet al\.,[2016](https://arxiv.org/html/2608.04305#bib.bib10)\), whereas value\-driven methods adjust the exploration probability according to observed value differences\(Tokic,[2010](https://arxiv.org/html/2608.04305#bib.bib11)\)\. Prioritized computation focuses updates on states expected to change most or on transitions with large temporal\-difference errors\(Moore and Atkeson,[1993](https://arxiv.org/html/2608.04305#bib.bib28); Schaulet al\.,[2016](https://arxiv.org/html/2608.04305#bib.bib29)\)\. Iterate averaging combines successive Q\-updates to reduce stochastic variance\(Liet al\.,[2023](https://arxiv.org/html/2608.04305#bib.bib12)\), and nested\-simulation theory allocates a finite simulation budget between outer scenarios and inner estimation\(Gordy and Juneja,[2010](https://arxiv.org/html/2608.04305#bib.bib9)\)\. These mechanisms explain concretely what may be adapted: the step\-size, target scale, exploration rule, update priority, estimator aggregation, or sample allocation\. However, none directly resolves the two\-timescale coupling in CVaR RaQL under a fixed budget\. We therefore adapt the RaQL training controller while retaining its estimator and fixed point\.

### 1\.2\.CVaR Recursion and the RaQL Baseline

This section represents the definition of CVaR and structure of RaQL Let𝒮\\mathcal\{S\}and𝒜\\mathcal\{A\}be the finite state and action sets, withS:=\|𝒮\|S:=\|\\mathcal\{S\}\|andA:=\|𝒜\|A:=\|\\mathcal\{A\}\|, and letQ:𝒮×𝒜→ℝQ:\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\mathbb\{R\}be the cost\-valued action\-value table\. LetXXdenote a random future loss\. The upper\-tail CVaR at levelα∈\(0,1\)\\alpha\\in\(0,1\)can be written in optimized\-certainty\-equivalent form as\(Rockafellar and Uryasev,[2000](https://arxiv.org/html/2608.04305#bib.bib2); Ben\-Tal and Teboulle,[2007](https://arxiv.org/html/2608.04305#bib.bib3)\)

\(1\)ρα​\(X\)=miny∈ℝ⁡\{y\+11−α​𝔼​\[\(X−y\)\+\]\},\\rho\_\{\\alpha\}\(X\)=\\min\_\{y\\in\\mathbb\{R\}\}\\left\\\{y\+\\frac\{1\}\{1\-\\alpha\}\\mathbb\{E\}\\left\[\(X\-y\)\_\{\+\}\\right\]\\right\\\},where\(u\)\+:=max⁡\{u,0\}\(u\)\_\{\+\}:=\\max\\\{u,0\\\}\. For CVaR, the optimizery⋆y^\{\\star\}is a Value\-at\-Risk threshold\. For a state–action pair\(s,a\)\(s,a\)and sampled next states′s^\{\\prime\}, define the state–action\-dependent one\-step loss target

\(2\)xs​a​\(s′;Q\)=ℓ​\(s,a,s′\)\+γ​VQ​\(s′\),VQ​\(s\)=mina∈𝒜⁡Q​\(s,a\),x\_\{sa\}\(s^\{\\prime\};Q\)=\\ell\(s,a,s^\{\\prime\}\)\+\\gamma V\_\{Q\}\(s^\{\\prime\}\),\\quad V\_\{Q\}\(s\)=\\min\_\{a\\in\\mathcal\{A\}\}Q\(s,a\),whereℓ​\(s,a,s′\)\\ell\(s,a,s^\{\\prime\}\)is the immediate loss,γ\\gammais the discount factor, ands′s^\{\\prime\}is sampled from the unknown transition kernel\. The algorithms abbreviate the realizedxs​a​\(s′;Q\)x\_\{sa\}\(s^\{\\prime\};Q\)asxxand write the cell\-specific threshold estimate asy​\(s,a\)y\(s,a\)\. For generic scalar argumentsxxandyy, the sample\-level CVaR target and its subgradient are

\(3\)Gα​\(x,y\)\\displaystyle G\_\{\\alpha\}\(x,y\)=y\+11−α​\(x−y\)\+,\\displaystyle=y\+\\frac\{1\}\{1\-\\alpha\}\(x\-y\)\_\{\+\},\(4\)gy​\(x,y\)\\displaystyle g\_\{y\}\(x,y\)=1−11−α​𝟏​\{x\>y\}\.\\displaystyle=1\-\\frac\{1\}\{1\-\\alpha\}\\mathbf\{1\}\\\{x\>y\\\}\.Accordingly,xs​a​\(s′;Q\)x\_\{sa\}\(s^\{\\prime\};Q\)andy​\(s,a\)y\(s,a\)are explicitly state–action dependent\. The shorter symbolsxxandyyare used only inside a sampled update\. Algorithm[1](https://arxiv.org/html/2608.04305#alg1)gives the fixed\-parameter RaQL implementation used as the reference baseline\. The total sampling budgetBBis the number of inner\-loop transition samples available for training: each sampled transition in the inner loop consumes one unit ofBB\. The inner\-loop depthLLis the maximum number of such transitions processed before the next outer Q\-update\. At the start of each outer iteration,Q−←QQ^\{\-\}\\leftarrow Qcreates a frozen copy of the current Q\-table, andV−​\(s\):=mina⁡Q−​\(s,a\)V^\{\-\}\(s\):=\\min\_\{a\}Q^\{\-\}\(s,a\)is the next\-state value computed from that copy\. Both are held fixed during the inner loop\. In addition to this within\-iteration freezing, the inner\-loop depthLLand the inner risk scalehyh\_\{y\}are both fixed throughout training, applies the within\-loop decay factorj−pj^\{\-p\}, and retains only the final CVaR target for each visited cell\. The counterns​an\_\{sa\}records the number of earlier outer Q\-updates to cell\(s,a\)\(s,a\)\. Within an inner loop,𝒴s​a\\mathcal\{Y\}\_\{sa\}stores the iterates ofy​\(s,a\)y\(s,a\),y¯s​a\\bar\{y\}\_\{sa\}is the mean of the latter half of that queue, andq^s​a\\widehat\{q\}\_\{sa\}is the most recently retained CVaR target\. The mapΠ𝒴\\Pi\_\{\\mathcal\{Y\}\}denotes projection onto the admissible interval𝒴\\mathcal\{Y\}\. These fixed choices define the reference baseline, but they cannot adjust as training progresses\. The next subsection explains why this can be problematic under a finite sampling budget\.

Algorithm 1Fixed\-parameter CVaR RaQL baseline0:Total inner\-transition sample budget

BB; inner\-loop depth

LL; base scale

hyh\_\{y\}; CVaR level

α\\alpha; discount

γ\\gamma; inner and outer exponents

p,ηp,\\eta; admissible interval

𝒴\\mathcal\{Y\}for

yy
1:Initialize

Q​\(s,a\)←0Q\(s,a\)\\leftarrow 0,

y​\(s,a\)←0y\(s,a\)\\leftarrow 0,

ns​a←1n\_\{sa\}\\leftarrow 1, and consumed\-sample count

b←0b\\leftarrow 0
2:while

b<Bb<Bdo

3:Freeze

Q−←QQ^\{\-\}\\leftarrow Qand set

V−​\(s\)←mina⁡Q−​\(s,a\)V^\{\-\}\(s\)\\leftarrow\\min\_\{a\}Q^\{\-\}\(s,a\)
4:Observe

ssand choose action

aaby exploration

5:Reset per\-cell queues

𝒴s​a\\mathcal\{Y\}\_\{sa\}and retained targets

q^s​a\\widehat\{q\}\_\{sa\}
6:for

j=1,…,min⁡\{L,B−b\}j=1,\\ldots,\\min\\\{L,B\-b\\\}do

7:Observe loss

ℓ\\elland next state

s′s^\{\\prime\}; set

b←b\+1b\\leftarrow b\+1
8:Append

y​\(s,a\)y\(s,a\)to

𝒴s​a\\mathcal\{Y\}\_\{sa\}and set

y¯s​a←HalfWindowMean⁡\(𝒴s​a\)\\bar\{y\}\_\{sa\}\\leftarrow\\operatorname\{HalfWindowMean\}\(\\mathcal\{Y\}\_\{sa\}\)
9:

x←ℓ\+γ​V−​\(s′\)x\\leftarrow\\ell\+\\gamma V^\{\-\}\(s^\{\\prime\}\),

q←Gα​\(x,y¯s​a\)q\\leftarrow G\_\{\\alpha\}\(x,\\bar\{y\}\_\{sa\}\), and

q^s​a←q\\widehat\{q\}\_\{sa\}\\leftarrow q
10:

y​\(s,a\)←Π𝒴​\[y​\(s,a\)−j−p​hy​gy​\(x,y¯s​a\)\]y\(s,a\)\\leftarrow\\Pi\_\{\\mathcal\{Y\}\}\[y\(s,a\)\-j^\{\-p\}h\_\{y\}g\_\{y\}\(x,\\bar\{y\}\_\{sa\}\)\]
11:

s←s′s\\leftarrow s^\{\\prime\}
12:endfor

13:for all

\(s,a\)\(s,a\)visited in this outer iterationdo

14:

Q​\(s,a\)←\(1−ns​a−η\)​Q−​\(s,a\)\+ns​a−η​q^s​aQ\(s,a\)\\leftarrow\(1\-n\_\{sa\}^\{\-\\eta\}\)Q^\{\-\}\(s,a\)\+n\_\{sa\}^\{\-\\eta\}\\widehat\{q\}\_\{sa\}
15:

ns​a←ns​a\+1n\_\{sa\}\\leftarrow n\_\{sa\}\+1
16:endfor

17:endwhile

18:return

QQand its greedy policy

### 1\.3\.Finite\-Budget Failure Mechanism and Research Gap

The convergence guarantee of RaQL specifies conditions under which the risk estimates and Q\-values approach the optimal solutions as the sampling continues, but it does not give a practical rule for allocating a fixed number of samples\. The allocation may need to change across training and across state–action cells\. This issue is especially important for CVaR RaQL because, wheny<xy<x,

\(5\)Gα​\(x,y\)=x1−α−α​y1−α\.G\_\{\\alpha\}\(x,y\)=\\frac\{x\}\{1\-\\alpha\}\-\\frac\{\\alpha y\}\{1\-\\alpha\}\.Atα=0\.8\\alpha=0\.8, for example,Gα​\(x,y\)=5​x−4​yG\_\{\\alpha\}\(x,y\)=5x\-4y\. Thus, ifyyremains below the relevant VaR threshold, its error is amplified in the target passed to the outer Q\-update rather than remaining confined to the inner recursion\.

For a frozen tableQ−Q^\{\-\}, defineXs​a​\(Q−\)=ℓ​\(s,a,S′\)\+γ​VQ−​\(S′\)X\_\{sa\}\(Q^\{\-\}\)=\\ell\(s,a,S^\{\\prime\}\)\+\\gamma V\_\{Q^\{\-\}\}\(S^\{\\prime\}\), whereS′S^\{\\prime\}is the next random state, and the corresponding CVaR optimizer

ys​a⋆​\(Q−\)∈arg⁡miny⁡𝔼​\[y\+\(Xs​a​\(Q−\)−y\)\+1−α∣s,a\]\.y^\{\\star\}\_\{sa\}\(Q^\{\-\}\)\\in\\arg\\min\_\{y\}\\mathbb\{E\}\\\!\\left\[y\+\\frac\{\(X\_\{sa\}\(Q^\{\-\}\)\-y\)\_\{\+\}\}\{1\-\\alpha\}\\mid s,a\\right\]\.This optimizer is a conditionalα\\alpha\-VaR threshold\. A*lagging*inner variable means thaty​\(s,a\)y\(s,a\)remains materially below the current optimizerys​a⋆​\(Q−\)y^\{\\star\}\_\{sa\}\(Q^\{\-\}\); it does not mean thaty​\(s,a\)y\(s,a\)should match an individual sampled target\.

With a finite budget, the baseline fixes the inner\-loop length and update scale before training\. Inner estimation error then affects the outer update, while changes in the Q\-table alter the target that the inner loop must subsequently estimate\. A single fixed setting may therefore be unstable early in training or leave persistent residual error later\. Existing RaQL theory characterizes the asymptotic requirements and links inner accuracy to outer convergence\(Huang and Haskell,[2021](https://arxiv.org/html/2608.04305#bib.bib1), Theorem 4\.7 and Remark 4\.8\), but it does not prescribe an online finite\-budget strategy for coordinating the two recursions\.

This paper asks:*how can CVaR RaQL use a fixed transition\-sample budget to reduce empirical Bellman residuals and mitigate the instability caused by fixed inner\-loop settings\.*We address this question with an adaptive training controller built around the original CVaR risk objective\. The controller uses observed training quantities to adjust the inner step sizes, allocate transition samples across state–action cells, aggregate the sampled CVaR targets, and calibrate the outer learning\-rate exponent, inner risk scale, and inner\-loop depth\.

Our contributions are:

- •We identify a CVaR\-specific error\-amplification mechanism: inaccurate tracking of the moving VaR estimate can magnify errors in the target used by the outer Q\-update\.
- •We develop a three\-module adaptive training controller that stabilizes inner risk estimation, reallocates samples as training progresses, and calibrates key training parameters during training process\.
- •We use cumulative ablations to measure the effect of each controller mechanism to assess sensitivity to the CVaR level, discount factor, and sampling budget\. The adaptive controller substantially reduces empirical Bellman residuals relative to the fixed\-parameter baseline\.
- •We demonstrate the adaptive controller in a chronological Bitcoin trading study using market and sentiment states and accounting for turnover costs, showing better risk\-adjusted performance and lower downside\-risk measures than the fixed\-parameter baseline\.

## 2\.Adaptive Finite\-Budget Controller

We keep the CVaR RaQL estimator fixed and modify only the training controller\. The controller is organized into three modules and evaluated through six schemes\.

### 2\.1\.Adaptive CVaR RaQL

The procedure of our final adaptive CVaR RaQL algorithm is presented in Algorithm[2](https://arxiv.org/html/2608.04305#alg2)\. Its main mechanisms are described in detail later\.

Algorithm 2Adaptive finite\-budget CVaR RaQL0:Training samples

𝒟\\mathcal\{D\};

BB;

α,γ\\alpha,\\gamma;

kw,κh,kT,pk\_\{w\},\\kappa\_\{h\},k\_\{T\},p
1:Set

Q​\(s,a\)←0Q\(s,a\)\\leftarrow 0,

y​\(s,a\)←0y\(s,a\)\\leftarrow 0,

ns​a←1n\_\{sa\}\\leftarrow 1, and

b←0b\\leftarrow 0
2:Use one epoch to estimate

ℓavg\\ell\_\{\\mathrm\{avg\}\},

ℓmin\\ell\_\{\\min\}, and

ℓmax\\ell\_\{\\max\}; add its samples to

bb
3:

ymin←xmin←ℓmin/\(1−γ\)y\_\{\\min\}\\leftarrow x\_\{\\min\}\\leftarrow\\ell\_\{\\min\}/\(1\-\\gamma\)and

ymax←xmax←ℓmax/\(1−γ\)y\_\{\\max\}\\leftarrow x\_\{\\max\}\\leftarrow\\ell\_\{\\max\}/\(1\-\\gamma\)
4:Set

η\\eta,

hyh\_\{y\}, and

LLusing Equations \([11](https://arxiv.org/html/2608.04305#S2.E11)\), \([12](https://arxiv.org/html/2608.04305#S2.E12)\), and \([14](https://arxiv.org/html/2608.04305#S2.E14)\)

5:while

b<Bb<Bdo

6:Freeze

Q−​\(s,a\)←Q​\(s,a\)Q^\{\-\}\(s,a\)\\leftarrow Q\(s,a\)and set

V−​\(s\)←mina⁡Q−​\(s,a\)V^\{\-\}\(s\)\\leftarrow\\min\_\{a\}Q^\{\-\}\(s,a\)
7:Reset

ks​a←0k\_\{sa\}\\leftarrow 0and the queues

𝒴s​a,ℋs​a←∅\\mathcal\{Y\}\_\{sa\},\\mathcal\{H\}\_\{sa\}\\leftarrow\\varnothing
8:for

j=1,…,min⁡\{L,B−b\}j=1,\\ldots,\\min\\\{L,B\-b\\\}do

9:Observe

ssand set

T←b/BT\\leftarrow b/B
10:Select

aausing Equation \([9](https://arxiv.org/html/2608.04305#S2.E9)\)

11:Observe loss

ℓ\\elland next state

s′s^\{\\prime\}; set

b←b\+1b\\leftarrow b\+1and

ks​a←ks​a\+1k\_\{sa\}\\leftarrow k\_\{sa\}\+1
12:Append

y​\(s,a\)y\(s,a\)to

𝒴s​a\\mathcal\{Y\}\_\{sa\}
13:Calculate

y¯s​a\\bar\{y\}\_\{sa\}by averaging the latter half of

𝒴s​a\\mathcal\{Y\}\_\{sa\}
14:

x←ℓ\+γ​V−​\(s′\)x\\leftarrow\\ell\+\\gamma V^\{\-\}\(s^\{\\prime\}\)and

q^s​a←Gα​\(x,y¯s​a\)\\widehat\{q\}\_\{sa\}\\leftarrow G\_\{\\alpha\}\(x,\\bar\{y\}\_\{sa\}\)
15:Calculate

λs​a\\lambda\_\{sa\}by Equation \([7](https://arxiv.org/html/2608.04305#S2.E7)\)

16:

y​\(s,a\)←clip⁡\(y​\(s,a\)−λs​a​hy​gy​\(x,y¯s​a\),ymin,ymax\)y\(s,a\)\\leftarrow\\operatorname\{clip\}\\\!\\left\(y\(s,a\)\-\\lambda\_\{sa\}h\_\{y\}g\_\{y\}\(x,\\bar\{y\}\_\{sa\}\),y\_\{\\min\},y\_\{\\max\}\\right\)
17:if

T≤0\.05T\\leq 0\.05then

18:Correct

y​\(s,a\)y\(s,a\)by Equation \([8](https://arxiv.org/html/2608.04305#S2.E8)\)

19:endif

20:Append

q^s​a\\widehat\{q\}\_\{sa\}to

ℋs​a\\mathcal\{H\}\_\{sa\}
21:endfor

22:for all

\(s,a\)\(s,a\)visited in this outer iterationdo

23:Calculate

q^agg,s​a\\widehat\{q\}\_\{\\mathrm\{agg\},sa\}using Equation \([10](https://arxiv.org/html/2608.04305#S2.E10)\)

24:

Q​\(s,a\)←\(1−ns​a−η\)​Q−​\(s,a\)\+ns​a−η​q^agg,s​aQ\(s,a\)\\leftarrow\(1\-n\_\{sa\}^\{\-\\eta\}\)Q^\{\-\}\(s,a\)\+n\_\{sa\}^\{\-\\eta\}\\widehat\{q\}\_\{\\mathrm\{agg\},sa\}
25:

ns​a←ns​a\+1n\_\{sa\}\\leftarrow n\_\{sa\}\+1
26:endfor

27:endwhile

28:return

QQand its greedy policy

In Algorithm[2](https://arxiv.org/html/2608.04305#alg2),BBdenotes the training budget,bbis the number of training samples already used, andT=b/B∈\[0,1\]T=b/B\\in\[0,1\]denotes the current training progress\. The parameterskwk\_\{w\},κh\\kappa\_\{h\}, andkTk\_\{T\}are calibration hyperparameters introduced in Section[2\.4](https://arxiv.org/html/2608.04305#S2.SS4)\. The parametersppandη\\etaare two decay exponents\. The parameterhyh\_\{y\}denotes the inner risk scale\. It converts the inner subgradientgyg\_\{y\}into an appropriate step size for theyy\-update\. The parameterLLdenotes the inner\-loop depth, whilejjdenotes the number of inner updates in the current outer iteration\. For each state–action cell\(s,a\)\(s,a\),λs​a\\lambda\_\{sa\}denotes the step size,ns​an\_\{sa\}denotes the number of times that cell has been updated in the outer loop, andks​ak\_\{sa\}denotes the number of times it has been visited in the inner loop of the current outer iteration\. The queues𝒴s​a\\mathcal\{Y\}\_\{sa\}andℋs​a\\mathcal\{H\}\_\{sa\}store the sequences ofy​\(s,a\)y\(s,a\)estimates andq^s​a\\widehat\{q\}\_\{sa\}estimates, respectively, in the current outer iteration\.

### 2\.2\.Tracking\-Stability Module

#### Inner\-Loop Decay\.

The baseline decays the step size by a global inner\-loop counterjj\. However, the per\-cell inner\-update counterks​ak\_\{sa\}, rather than the global inner\-update counterjj, better reflects the progress in estimating the minimizer for each state–action cell\. In the baseline, if a state–action cell is sampled for the first time whenjjis large, the step size used for its first update has already decayed significantly\. This can destabilize training\. We instead use

\(6\)λs​a=ks​a−p,\\lambda\_\{sa\}=k\_\{sa\}^\{\-p\},This can better match the step size to the progress in estimating the minimizer for each state–action cell\.

#### Outer\-Loop Decay\.

The outer loop estimates the Q\-table, while the inner loop tracks the minimizer of the risk subproblem\. Each Q\-table update shifts this minimizer\. The inner loop must then track its new location\. As the outer learning rate decays, later Q\-updates become smaller and the minimizer changes more slowly\. The inner step sizeλs​a\\lambda\_\{sa\}should therefore decay with the outer\-update count at the same rate as the Q\-update step size\. The baseline has no such outer\-scale decay\. For the same inner visit count \(ks​ak\_\{sa\}\), a late\-stage update \(largens​an\_\{sa\}\) uses the same step size as an early\-stage update \(smallns​an\_\{sa\}\)\. This can push an accurate estimate away from the current minimizer and continually introduce noise\. We therefore use

\(7\)λs​a=ks​a−p​ns​a−η,\\lambda\_\{sa\}=k\_\{sa\}^\{\-p\}n\_\{sa\}^\{\-\\eta\},whereη\\etais matched to the outer Q\-learning exponent\. This couples the inner step size to the Q\-update rate\.

#### EarlyyyCorrection\.

As shown in Equation \([5](https://arxiv.org/html/2608.04305#S1.E5)\), using ayyestimate that is far below its minimizery⋆y^\{\\star\}in the Q update can amplify estimation errors and cause training to diverge\. Early in training, an inappropriate inner risk scalehyh\_\{y\}, an unsuitable step sizeλs​a\\lambda\_\{sa\}, an insufficient inner\-loop depthLL, or limited updates for a hard\-to\-reach state–action cell can leaveyyfar below its current minimizery⋆y^\{\\star\}\. At this stage, the Q\-table is initialized at zero and changes rapidly, causingy⋆y^\{\\star\}to shift rapidly as well\. Consequently,yymay fail to track the moving minimizery⋆y^\{\\star\}\. For CVaR,y⋆y^\{\\star\}is the VaR ofX​\(s,a\)X\(s,a\)\. Thus,y⋆y^\{\\star\}and the sampled targetx​\(s,a\)x\(s,a\)are on the same scale\. We use the clippedx​\(s,a\)x\(s,a\)only as a reference and pull the early\-stageyyestimate toward it:

\(8\)y←y\+1ns​a\+2​0\.5ks​a−1​\(clip⁡\(x,ymin,ymax\)−y\)\.y\\leftarrow y\+\\frac\{1\}\{n\_\{sa\}\+2\}0\.5^\{k\_\{sa\}\-1\}\\left\(\\operatorname\{clip\}\(x,y\_\{\\min\},y\_\{\\max\}\)\-y\\right\)\.Because the clipped target is not an accurate estimate ofy⋆y^\{\\star\}and the risk of divergence is greatest in early training, we apply the correction only during the first5%5\\%of the budget\. Its strength also decays with both outer and inner visits \(ns​a,ks​an\_\{sa\},k\_\{sa\}\)\. The correction therefore weakens as the estimate becomes more accurate and does not affect late\-stage convergence\.

### 2\.3\.Sampling\-Efficiency Module

#### Two\-Phase Action Selection\.

As Equation \([2](https://arxiv.org/html/2608.04305#S1.E2)\) shows, the estimate of a cell\(s,a\)\(s,a\)depends on the valueVQ​\(s′\)V\_\{Q\}\(s^\{\\prime\}\)\. Early in training, the greedy action that determinesVQ​\(s′\)V\_\{Q\}\(s^\{\\prime\}\)is not yet stable\. Any action could eventually become the greedy action\. Accurate estimation therefore requires sufficient training across a broad range of Q\-table cells\. Concentrating updates on certain cells in the early stage may also deteriorate the training performance\. The outer visit countsns​an\_\{sa\}of these cells can grow rapidly while the corresponding next\-state valuesVQ​\(s′\)V\_\{Q\}\(s^\{\\prime\}\)remain inaccurate\. Their step sizes may then decay excessively\. As the rest of the Q\- table becomes more accurate and the targets change, the Q\-value update and minimizer estimates for these cells become difficult\. In the late stage, the greedy actions are generally stable\. At this stage, non\-greedy cells do not propagate errors or determine state values\. Their importance therefore decreases\. The budget should then shift toward the greedy cells that determineVQV\_\{Q\}\. These observations motivate the following two\-phase action selector:

ainner​\(s\)=\\displaystyle a\_\{\\mathrm\{inner\}\}\(s\)=\(9\)\{arg⁡mina∈𝒜⁡ns​a,T<0\.60,ϵ​\-greedy with greedy probability​0\.90,T≥0\.60\.\\displaystyle\\begin\{cases\}\\arg\\min\_\{a\\in\\mathcal\{A\}\}n\_\{sa\},&T<0\.60,\\\\ \\epsilon\\text\{\-greedy with greedy probability \}0\.90,&T\\geq 0\.60\.\\end\{cases\}

#### Progressive Suffix Aggregation\.

Letq^1,s​a,…,q^ks​a,s​a\\widehat\{q\}\_\{1,sa\},\\ldots,\\widehat\{q\}\_\{k\_\{sa\},sa\}be the sequence of inner CVaR targets for a given\(s,a\)\(s,a\)in one outer iteration\. The baseline uses onlyq^ks​a,s​a\\widehat\{q\}\_\{k\_\{sa\},sa\}\. We instead use a progress\-dependent suffix average:

\(10\)q^agg,s​a=1m​∑i=ks​a−m\+1ks​aq^i,s​a,m=⌈ω​\(T\)​ks​a⌉,\\widehat\{q\}\_\{\\mathrm\{agg\},sa\}=\\frac\{1\}\{m\}\\sum\_\{i=k\_\{sa\}\-m\+1\}^\{k\_\{sa\}\}\\widehat\{q\}\_\{i,sa\},\\qquad m=\\lceil\\omega\(T\)k\_\{sa\}\\rceil,whereω​\(T\)=0\.1\+0\.1​min⁡\{9,⌊10​T⌋\}\\omega\(T\)=0\.1\+0\.1\\min\\\{9,\\lfloor 10T\\rfloor\\\}\. Early in training, the inner variables are biased and only the tail of the sequence is trusted; later, more estimates are averaged to reduce target variance\. As training progresses and the inner variable starts closer to its current minimizer, the expanding window reuses more of the targets already computed\. This reduces target variance without allowing strongly transient early iterates to dominate the outer update, motivated by the variance\-reduction principle of iterate averaging\(Liet al\.,[2023](https://arxiv.org/html/2608.04305#bib.bib12)\)\.

### 2\.4\.Observable\-Calibration Module

To reduce manual tuning across different settings, the controller calibrates the learning\-rate exponent\(η\\eta\), inner risk scale, and inner\-loop depth separately\.

#### Discount\-aware outer learning\-rate decay\.

The outer Q\-learning step\-size isns​a−ηn\_\{sa\}^\{\-\\eta\}\. A largerγ\\gammaextends the effective horizon, so losses occurring further in the future retain greater weight in the current value estimate\. The outer Q\-learning step\-size should therefore remain responsive for longer and decay more slowly asγ\\gammaincreases\. RaQL requires the exponent to lie in\(1/2,1\]\(1/2,1\]\(Huang and Haskell,[2021](https://arxiv.org/html/2608.04305#bib.bib1)\)\. We set

\(11\)η=clip⁡\(0\.5\+kw​\(1−γ\),0\.5\+ε,1\.0\)\.\\eta=\\operatorname\{clip\}\\left\(0\.5\+k\_\{w\}\(1\-\\gamma\),\\,0\.5\+\\varepsilon,\\,1\.0\\right\)\.Asγ\\gammaincreases,η\\etadecreases and the outer Q\-learning rate decays more slowly\. The clipping keepsη\\etawithin the admissible RaQL range\.

#### Loss\-aware inner risk scale\.

Early in training,yymust track the moving minimizery⋆y^\{\\star\}\. We set the expected unattenuated one\-step movement,hy​𝔼​\[\|gy\|\]h\_\{y\}\\mathbb\{E\}\[\|g\_\{y\}\|\], to a fixed fraction of the scale ofyy\. This scale is approximatelyℓavg/\(1−γ\)\\ell\_\{\\mathrm\{avg\}\}/\(1\-\\gamma\)\. Since𝔼​\[\|gy\|\]=2​α\\mathbb\{E\}\[\|g\_\{y\}\|\]=2\\alphaaty⋆y^\{\\star\}, we set

\(12\)hy=κh​ℓavgα​\(1−γ\)\.h\_\{y\}=\\kappa\_\{h\}\\frac\{\\ell\_\{\\mathrm\{avg\}\}\}\{\\alpha\(1\-\\gamma\)\}\.ForΔ​y=−λs​a​hy​gy\\Delta y=\-\\lambda\_\{sa\}h\_\{y\}g\_\{y\}, the resulting update also satisfies

\(13\)\{𝔼​\[Δ​y∣y\]\>0,y<y⋆,𝔼​\[Δ​y∣y\]=0,y=y⋆,𝔼​\[Δ​y∣y\]<0,y\>y⋆\.\\begin\{cases\}\\mathbb\{E\}\[\\Delta y\\mid y\]\>0,&y<y^\{\\star\},\\\\ \\mathbb\{E\}\[\\Delta y\\mid y\]=0,&y=y^\{\\star\},\\\\ \\mathbb\{E\}\[\\Delta y\\mid y\]<0,&y\>y^\{\\star\}\.\\end\{cases\}The rule therefore improves early tracking without changing the convergence pointy⋆y^\{\\star\}\. To estimateℓavg\\ell\_\{\\mathrm\{avg\}\}, we use a short warmup before training\. Because the trading loss is signed, we computeℓavg\\ell\_\{\\mathrm\{avg\}\}as the mean absolute loss over the warmup epoch, rather than as the signed sample mean, so that it provides a positive magnitude scale\. The identity𝔼​\[\|gy\|\]=2​α\\mathbb\{E\}\[\|g\_\{y\}\|\]=2\\alphais exact aty⋆y^\{\\star\}when the target distribution is continuous at the quantile andℙ​\(X\>y⋆\)=1−α\\mathbb\{P\}\(X\>y^\{\\star\}\)=1\-\\alpha\. Here, it is used only for scale calibration, and the resulting rule is evaluated across the reportedα\\alphasweep\.

#### Budget\-aware inner\-loop depth\.

Under a fixed budget,LLbalances inner\-target accuracy against the number of outer Q\-learning updates\. Too few inner transitions leave an inaccurate CVaR target, while too many reduce the available outer updates\. Motivated by the cube\-root budget\-allocation rule for nested simulation in Gordy and Juneja\(Gordy and Juneja,[2010](https://arxiv.org/html/2608.04305#bib.bib9)\), we use the average per\-cell budgetB/\(S​A\)B/\(SA\)and set

\(14\)L=round⁡\(kT​\(BS​A\)1/3\)\.L=\\operatorname\{round\}\\left\(k\_\{T\}\\left\(\\frac\{B\}\{SA\}\\right\)^\{1/3\}\\right\)\.Because the nested\-simulation allocation rule is not derived specifically for RaQL, we use the cube\-root expression as an observable budget\-scaling heuristic rather than claim that it is theoretically optimal for this setting\. Its practical behavior is evaluated in the reported budget sweep\. Together, these three rules form the observable\-calibration module\. They set the learning\-rate exponent, inner risk scale, and inner\-loop depth from known or observed quantities\. The coefficientskwk\_\{w\},κh\\kappa\_\{h\}, andkTk\_\{T\}are selected once and kept fixed across the reported parameter sweeps\.

## 3\.Cryptocurrency Trading Experiment

### 3\.1\.Data and Market Environment

We construct a daily cryptocurrency dataset by aligning Bitcoin market data with sentiment information\. The Crypto Fear and Greed Index is retrieved from the Alternative\.me public API\(Alternative\.me,[n\.d\.](https://arxiv.org/html/2608.04305#bib.bib15)\); it ranges from 0 to 100 and is accompanied by a categorical label such as Extreme Fear, Fear, Neutral, Greed, or Extreme Greed\. Daily BTCUSDT closing prices are retrieved from the Binance public market\-data API\(Binance,[n\.d\.](https://arxiv.org/html/2608.04305#bib.bib16)\)\. From the closing\-price series, we compute daily return asrt=Pt/Pt−1−1r\_\{t\}=P\_\{t\}/P\_\{t\-1\}\-1, wherePtP\_\{t\}is the BTCUSDT close on daytt\. We also construct a sentiment\-momentum feature, defined as the seven\-day change in the Fear and Greed Index\. After aligning both sources by UTC calendar date and removing missing feature values, the dataset contains 3,059 daily observations from February 8, 2018 to June 28, 2026\.

We use a chronological split: the first70%70\\%of observations form the training set and the remaining30%30\\%form the out\-of\-sample test set\. The state space is obtained by discretizing market sentiment, sentiment momentum, and recent return into33=273^\{3\}=27discrete states\. The action set consists of six portfolio positions,

𝒜=\{−1,−0\.6,−0\.2,0\.2,0\.6,1\},\\mathcal\{A\}=\\\{\-1,\-0\.6,\-0\.2,0\.2,0\.6,1\\\},wherew​\(a\)w\(a\)denotes the BTC exposure associated with actionaa\.

### 3\.2\.Training Protocol

During training, the generic immediate lossℓ​\(st,a,st\+1\)\\ell\(s\_\{t\},a,s\_\{t\+1\}\)is instantiated as

ℓt​\(a\)=−100​w​\(a\)​rt\+1\.\\ell\_\{t\}\(a\)=\-100\\,w\(a\)r\_\{t\+1\}\.The factor 100 rescales daily returns to a numerically convenient range\. Transaction costs are not included in the tabular training loss because they depend on the previous position; including them would require augmenting the Markov state with the previous action\. In the out\-of\-sample trading evaluation, however, we deduct turnover costs:

\(15\)Rtnet=w​\(at\)​rt\+1−ctc​\|w​\(at\)−w​\(at−1\)\|\.R^\{\\mathrm\{net\}\}\_\{t\}=w\(a\_\{t\}\)r\_\{t\+1\}\-c\_\{\\mathrm\{tc\}\}\|w\(a\_\{t\}\)\-w\(a\_\{t\-1\}\)\|\.This makes the trading evaluation conservative relative to the training objective\.

All main ablation experiments use 20 random seeds, 400 epochs, and a total training sample budget ofBB=856,000 inner\-transition samples\. One epoch consists of one pass through the 2,140 adjacent transitions in the training set\. Thus, 400 epochs correspond toBB=856,000 inner\-transition samples, including the warmup samples\. Multiple seeds reduce the risk that an apparent improvement is caused by training randomness or a favorable run\(Hendersonet al\.,[2018](https://arxiv.org/html/2608.04305#bib.bib13); Agarwalet al\.,[2021](https://arxiv.org/html/2608.04305#bib.bib14)\); paired ablations additionally isolate the incremental effect of each controller component\. The baseline, Scheme 0, is the fixed\-parameter two\-timescale CVaR RaQL implementation in Algorithm[1](https://arxiv.org/html/2608.04305#alg1)\. It uses an inner\-loop depth ofL=80L=80and an inner risk scale ofhy=10h\_\{y\}=10, both of which remain fixed throughout training\.

Schemes 1–6 form a cumulative ablation sequence\. Schemerrretains all mechanisms included in Schemer−1r\-1and adds only the new mechanism listed in rowrrof Table[1](https://arxiv.org/html/2608.04305#S4.T1)\. Every later scheme contains all mechanisms introduced in the earlier schemes\.

### 3\.3\.Evaluation Protocol and Metrics

We evaluate fixed\-point accuracy using a sample\-based empirical CVaR Bellman residual on the training set\. We do not estimate or use an explicit transition matrix for evaluation\. For a learned tableQQ, define

\(16\)\(𝒯^ρ​Q\)​\(s,a\)=ρ^α​\(\{ℓt​\(a\)\+γ​VQ​\(st\+1\):t∈ℐ​\(s\)\}\),\(\\widehat\{\\mathcal\{T\}\}\_\{\\rho\}Q\)\(s,a\)=\\widehat\{\\rho\}\_\{\\alpha\}\\left\(\\left\\\{\\ell\_\{t\}\(a\)\+\\gamma V\_\{Q\}\(s\_\{t\+1\}\):t\\in\\mathcal\{I\}\(s\)\\right\\\}\\right\),whereℐ​\(s\)\\mathcal\{I\}\(s\)is the set of training indices with statessandρ^α\\widehat\{\\rho\}\_\{\\alpha\}is the empirical CVaR operator

\(17\)ρ^α​\(\{xi\}i=1m\)=miny∈ℝ⁡\{y\+1\(1−α\)​m​∑i=1m\(xi−y\)\+\}\.\\widehat\{\\rho\}\_\{\\alpha\}\(\\\{x\_\{i\}\\\}\_\{i=1\}^\{m\}\)=\\min\_\{y\\in\\mathbb\{R\}\}\\left\\\{y\+\\frac\{1\}\{\(1\-\\alpha\)m\}\\sum\_\{i=1\}^\{m\}\(x\_\{i\}\-y\)\_\{\+\}\\right\\\}\.We report Q\-level and V\-level residuals,

\(18\)eQ​\(s,a\)\\displaystyle e\_\{Q\}\(s,a\)=\|\(𝒯^ρ​Q\)​\(s,a\)−Q​\(s,a\)\|,\\displaystyle=\\left\|\(\\widehat\{\\mathcal\{T\}\}\_\{\\rho\}Q\)\(s,a\)\-Q\(s,a\)\\right\|,\(19\)eV​\(s\)\\displaystyle e\_\{V\}\(s\)=\|mina⁡\(𝒯^ρ​Q\)​\(s,a\)−VQ​\(s\)\|\.\\displaystyle=\\left\|\\min\_\{a\}\(\\widehat\{\\mathcal\{T\}\}\_\{\\rho\}Q\)\(s,a\)\-V\_\{Q\}\(s\)\\right\|\.The tables focus on mean Bellman residuals, with maximum residuals used as a robustness diagnostic\. We summarize these quantities using the mean empirical Q\-level Bellman residual \(MeanBEQ\), maximum empirical Q\-level Bellman residual \(MaxBEQ\), mean empirical value\-level Bellman residual \(MeanBEV\), and maximum empirical value\-level Bellman residual \(MaxBEV\)\.

For out\-of\-sample financial performance, the learned table induces the deterministic policy

\(20\)at=arg⁡mina∈𝒜⁡Q​\(st,a\)\.a\_\{t\}=\\arg\\min\_\{a\\in\\mathcal\{A\}\}Q\(s\_\{t\},a\)\.The policy is evaluated once on the 918 observations in the chronological test set, with no Q\-updates on test observations\. We use a transaction cost ofctc=0\.0005c\_\{\\mathrm\{tc\}\}=0\.0005\(5 basis points\) and set the position immediately before the test period to zero\. For each of the 20 trained tables, we report cumulative return, annualized return, annualized volatility, the Sharpe ratio with zero risk\-free rate, maximum drawdown, average turnover, and empirical CVaR of daily loss\. Scheme results are summarized by their mean and sample standard deviation across seeds\. These standard deviations measure variation due to training randomness on the same market path\. Buy\-and\-hold and cash/no\-trade are deterministic baselines\. We also include fixed\-exposure policies withw=0\.2w=0\.2andw=−0\.2w=\-0\.2as deterministic baselines\.

For daily net returnRtnetR\_\{t\}^\{\\mathrm\{net\}\}, cumulative and annualized returns are

\(21\)CumRet\\displaystyle\\mathrm\{CumRet\}=∏t=1N\(1\+Rtnet\)−1,\\displaystyle=\\prod\_\{t=1\}^\{N\}\(1\+R\_\{t\}^\{\\mathrm\{net\}\}\)\-1,\(22\)AnnRet\\displaystyle\\mathrm\{AnnRet\}=\[∏t=1N\(1\+Rtnet\)\]365/N−1\.\\displaystyle=\\left\[\\prod\_\{t=1\}^\{N\}\(1\+R\_\{t\}^\{\\mathrm\{net\}\}\)\\right\]^\{365\}/N\-1\.IfR¯\\bar\{R\}andsRs\_\{R\}are the sample mean and standard deviation of daily net returns, then

\(23\)AnnVol=365​sR,Sharpe=365​R¯sR\.\\mathrm\{AnnVol\}=\\sqrt\{365\}\\,s\_\{R\},\\qquad\\mathrm\{Sharpe\}=\\sqrt\{365\}\\,\\frac\{\\bar\{R\}\}\{s\_\{R\}\}\.Thus, the reported annualized metrics use a 365\-observation financial\-market convention\.Because Bitcoin trades continuously, this annualization uses 365 calendar\-day observations\. Maximum drawdown is computed from the net\-return wealth path, and average turnover is

\(24\)Turnover=1N​∑t=1N\|wt−wt−1\|\.\\mathrm\{Turnover\}=\\frac\{1\}\{N\}\\sum\_\{t=1\}^\{N\}\|w\_\{t\}\-w\_\{t\-1\}\|\.wherewt:=w​\(at\)w\_\{t\}:=w\(a\_\{t\}\)denote the BTC exposure selected on daytt\. Finally, test CVaR is the empirical upper\-tail CVaR of daily lossLt=−RtnetL\_\{t\}=\-R\_\{t\}^\{\\mathrm\{net\}\}atα=0\.6\\alpha=0\.6; lower values indicate less severe tail loss\.

## 4\.Results

### 4\.1\.Ablation of Controller Components

Table[1](https://arxiv.org/html/2608.04305#S4.T1)reports the ablation results at the end of training\. The final adaptive controller, Scheme 6, reduces MeanBEQ from 1\.2202 to 0\.1854 and MeanBEV from 1\.1624 to 0\.0535 relative to the fixed\-parameter baseline\.

Table 1\.Ablation results at the 100% checkpoint\. Lower is better\.The gains do not come from a single mechanism\. The[Inner\-Loop Decay](https://arxiv.org/html/2608.04305#S2.SS2.SSS0.Px1)alone reduces the effective decay relative to the global\-counter baseline\. This leaves larger steps late in training and increases the terminal residuals\. Introducing[Outer\-Loop Decay](https://arxiv.org/html/2608.04305#S2.SS2.SSS0.Px2)substantially reduces all four residual metrics\.[Two\-Phase Action Selection](https://arxiv.org/html/2608.04305#S2.SS3.SSS0.Px1)and[Progressive Suffix Aggregation](https://arxiv.org/html/2608.04305#S2.SS3.SSS0.Px2)then improve sample efficiency and further reduce the residuals\. The[Observable\-Calibration Module](https://arxiv.org/html/2608.04305#S2.SS4)preserves this accuracy with fewer hand\-set constants\.

### 4\.2\.Mechanism\-Specific Evaluation

[Scheme 3](https://arxiv.org/html/2608.04305#S2.SS2.SSS0.Px3)introduces the earlyyycorrection to improve training stability and reduce failures\. A training is considered failed if its MeanBEQ diverges\. Figure[1](https://arxiv.org/html/2608.04305#S4.F1)presents the stress\-test results for[Scheme 2](https://arxiv.org/html/2608.04305#S2.SS2.SSS0.Px2)and[Scheme 3](https://arxiv.org/html/2608.04305#S2.SS2.SSS0.Px3)across a range of small inner risk scales \(hyh\_\{y\}\)\. Athy=0\.03h\_\{y\}=0\.03and0\.040\.04, all[Scheme 2](https://arxiv.org/html/2608.04305#S2.SS2.SSS0.Px2)seeds fail\.[Scheme 3](https://arxiv.org/html/2608.04305#S2.SS2.SSS0.Px3)succeeds for all seeds at every tested scale and keeps the 5%\-budget MeanBEQ between 0\.89 and 0\.91\. The correction therefore improves both training stability and early estimation accuracy, with clearer benefits under stressful settings, particularly whenhyh\_\{y\}is small\.

![Refer to caption](https://arxiv.org/html/2608.04305v1/x1.png)Figure 1\.Stability across inner risk scales\.Two line charts compare Schemes 2 and 3 as the inner risk scale $h\_\{y\}$ varies from 0\.03 to 0\.08\. The left panel reports MeanBEQ at 5 percent of the training budget\. The right panel reports terminal MeanBEQ; Scheme 2 fails at 0\.03 and 0\.04, while Scheme 3 succeeds at all six tested scales\.[Scheme 4](https://arxiv.org/html/2608.04305#S2.SS3.SSS0.Px1)and[Scheme 5](https://arxiv.org/html/2608.04305#S2.SS3.SSS0.Px2)improve sample efficiency\. Figure[2](https://arxiv.org/html/2608.04305#S4.F2)presents their MeanBEQ trajectories together with[Scheme 3](https://arxiv.org/html/2608.04305#S2.SS2.SSS0.Px3)\. In[Scheme 4](https://arxiv.org/html/2608.04305#S2.SS3.SSS0.Px1), balanced coverage during the first 60% builds broad Q\-table accuracy and avoids excessive step\-size decay in individual cells\. Once action selection shifts toward greedy actions, MeanBEQ drops from 0\.69 at 60% to 0\.46 at 65%\.[Scheme 5](https://arxiv.org/html/2608.04305#S2.SS3.SSS0.Px2)uses progressive suffix aggregation to reduce sampling variance\. It holds the lowest MeanBEQ at nearly every checkpoint and achieves the highest sample efficiency\.

![Refer to caption](https://arxiv.org/html/2608.04305v1/x2.png)Figure 2\.Mean Q residual during training\.A line chart compares the MeanBEQ trajectories of Schemes 3, 4, and 5 over the training budget\.[Scheme 6](https://arxiv.org/html/2608.04305#S2.SS4)is designed to adapt its training parameters when the risk levelα\\alpha, discount factorγ\\gamma, or training budget changes\. Table[2](https://arxiv.org/html/2608.04305#S4.T2)compares[Scheme 5](https://arxiv.org/html/2608.04305#S2.SS3.SSS0.Px2)and[Scheme 6](https://arxiv.org/html/2608.04305#S2.SS4)across the completeα\\alphaandγ\\gammasweeps\.[Scheme 6](https://arxiv.org/html/2608.04305#S2.SS4)has lower MeanBEQ and MeanBEV at every tested setting\. It is also less sensitive to changes inα\\alpha, while both schemes remain stable acrossγ\\gamma\. Figure[3](https://arxiv.org/html/2608.04305#S4.F3)reports the budget sweep\.[Scheme 6](https://arxiv.org/html/2608.04305#S2.SS4)achieves lower MeanBEV than[Scheme 5](https://arxiv.org/html/2608.04305#S2.SS3.SSS0.Px2)at every tested budget\. As the training budget decreases,[Scheme 6](https://arxiv.org/html/2608.04305#S2.SS4)remains more stable than[Scheme 5](https://arxiv.org/html/2608.04305#S2.SS3.SSS0.Px2)\. Overall,[Scheme 6](https://arxiv.org/html/2608.04305#S2.SS4)performs better across all tested settings\.

Table 2\.Mean residual ranges for Schemes 5 and 6 across theα\\alphaandγ\\gammasweeps\.![Refer to caption](https://arxiv.org/html/2608.04305v1/x3.png)Figure 3\.Mean residuals across training budgets\.Two line charts compare Schemes 5 and 6 over training budgets from 100 to 800 epochs\. The left panel reports MeanBEQ and the right panel reports MeanBEV\. Shaded bands are 95 percent confidence intervals over 10 seeds\.
### 4\.3\.Out\-of\-Sample Trading Performance

Table[3](https://arxiv.org/html/2608.04305#S4.T3)reports performance on the chronological test set after transaction costs\. Scheme 6 achieves the highest risk\-adjusted performance among all evaluated policies\. Its mean Sharpe ratio is 0\.9281, compared with 0\.5628 for Scheme 0 and 0\.4902 for buy\-and\-hold\. It also lowers annualized volatility from 14\.53% for Scheme 0 to 9\.57%, maximum drawdown from 17\.77% to 6\.46%, and daily CVaR loss from 0\.0050 to 0\.0041\. Scheme 6 also has much lower volatility, drawdown, and CVaR loss than buy\-and\-hold\.

Table 3\.Out\-of\-sample trading performance after transaction costs\. Scheme entries are mean \(sample standard deviation\) across 20 seeds; values are rounded to four decimals\.Scheme 6 earns a lower cumulative return than buy\-and\-hold \(23\.61% versus 35\.43%\), so the result does not indicate dominance in raw return\. Instead, it shows a different risk\-return profile: Scheme 6 accepts less market exposure and gives up part of the upside while reducing annualized volatility from 47\.93% to 9\.57% and maximum drawdown from 52\.21% to 6\.46%\. Compared with the fixedw=0\.2w=0\.2baseline, Scheme 6 achieves a higher cumulative return \(23\.61% versus 11\.25%\)\. It also has a lower maximum drawdown \(6\.46% versus 12\.61%\)\. These results show that maintaining a low market exposure alone does not reproduce Scheme 6’s improved risk–return profile\. Scheme 6 is also more stable across seeds than Scheme 0\. The standard deviations of its cumulative return, Sharpe ratio, and maximum drawdown are 0\.0118, 0\.0401, and 0\.0038, respectively, compared with 0\.1121, 0\.2281, and 0\.0648 for Scheme 0\. Overall, the adaptive controller achieves smaller Bellman residuals, reduces variation in out\-of\-sample performance across training seeds, and lowers volatility, maximum drawdown, and CVaR loss\.

## 5\.Discussion, Limitations, and Conclusion

This paper studies a finite\-budget instability in CVaR RaQL caused by three interacting factors: \(i\) the asymptotic convergence theory prescribes no rule for allocating a fixed sampling budget between inner and outer recursions, or across state–action cells with unequal visitation; \(ii\) inner and outer estimation errors are coupled through bootstrapping and can amplify under fixed hyperparameters; and \(iii\) uniform inner\-loop configurations ignore per\-cell visitation asymmetry, leaving some cells over\-trained and others under\-trained\. The proposed adaptive controller addresses all three without changing the CVaR estimator or Bellman fixed point: per\-cell step sizing and rate\-matched decay counter the coupling, coverage\-first\-then\-greedy allocation counters the asymmetry, and observable calibration replaces hand\-tuned constants for portability\. Under 856,000 inner\-transition samples, the controller reduces MeanBEQ from 1\.2202 to 0\.1854 and MeanBEV from 1\.1624 to 0\.0535; the cumulative ablation confirms that no single mechanism dominates, consistent with the three\-factor diagnosis\. On the chronological test set, the Sharpe ratio rises from 0\.5628 to 0\.9281 and maximum drawdown drops from 17\.77% to 6\.46%; the return remains below buy\-and\-hold, confirming improved risk\-adjusted rather than raw\-return performance\.

The main limitations of this work are: the controller lacks convergence guarantees for the coupled adaptive recursions, and the evaluation is confined to one asset with tabular states\. Future work should develop this theory, extend to multi\-asset and function\-approximation settings, and examine whether the factor decomposition generalizes to other risk\-aware RL algorithms\.

## References

- C\. Acerbi \(2002\)Spectral measures of risk: a coherent representation of subjective risk aversion\.Journal of Banking & Finance26\(7\),pp\. 1505–1518\.Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p2.1)\.
- R\. Agarwal, M\. Schwarzer, P\. S\. Castro, A\. Courville, and M\. G\. Bellemare \(2021\)Deep reinforcement learning at the edge of the statistical precipice\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 29304–29320\.Cited by:[§3\.2](https://arxiv.org/html/2608.04305#S3.SS2.p2.4)\.
- A\. Alameer and K\. Alshehri \(2022\)Conditional value\-at\-risk for quantitative trading: a direct reinforcement learning approach\.In2022 IEEE Conference on Control Technology and Applications \(CCTA\),pp\. 1208–1213\.External Links:[Document](https://dx.doi.org/10.1109/CCTA49430.2022.9966017)Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p2.1)\.
- Alternative\.me \(n\.d\.\)Crypto fear & greed index\.Note:[https://alternative\.me/crypto/fear\-and\-greed\-index/](https://alternative.me/crypto/fear-and-greed-index/)Accessed 26 July 2026Cited by:[§3\.1](https://arxiv.org/html/2608.04305#S3.SS1.p1.3)\.
- P\. Artzner, F\. Delbaen, J\. Eber, and D\. Heath \(1999\)Coherent measures of risk\.Mathematical Finance9\(3\),pp\. 203–228\.Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p2.1)\.
- M\. G\. Bellemare, S\. Srinivasan, G\. Ostrovski, T\. Schaul, D\. Saxton, and R\. Munos \(2016\)Unifying count\-based exploration and intrinsic motivation\.InAdvances in Neural Information Processing Systems,Vol\.29\.Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p5.1)\.
- A\. Ben\-Tal and M\. Teboulle \(2007\)An old\-new concept of convex risk measures: the optimized certainty equivalent\.Mathematical Finance17\(3\),pp\. 449–476\.Cited by:[§1\.2](https://arxiv.org/html/2608.04305#S1.SS2.p1.7)\.
- Binance \(n\.d\.\)Binance spot rest api: kline/candlestick data\.Note:[https://developers\.binance\.com/en/docs/catalog/core\-trading\-spot\-trading/api/rest\-api/market\#klinecandlestick\-data](https://developers.binance.com/en/docs/catalog/core-trading-spot-trading/api/rest-api/market#klinecandlestick-data)Accessed 26 July 2026Cited by:[§3\.1](https://arxiv.org/html/2608.04305#S3.SS1.p1.3)\.
- L\. Bisi, P\. Liotet, L\. Sabbioni, G\. Reho, N\. Montali, M\. Restelli, and C\. Corno \(2020\)Foreign exchange trading: a risk\-averse batch reinforcement learning approach\.InProceedings of the First ACM International Conference on AI in Finance,External Links:[Document](https://dx.doi.org/10.1145/3383455.3422571)Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p1.1)\.
- V\. S\. Borkar and S\. P\. Meyn \(2000\)The O\.D\.E\. method for convergence of stochastic approximation and reinforcement learning\.SIAM Journal on Control and Optimization38\(2\),pp\. 447–469\.Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p3.1)\.
- Y\. Chen, X\. Zhang, S\. Wang, and L\. Huang \(2024\)Provable risk\-sensitive distributional reinforcement learning with general function approximation\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 7748–7791\.Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p2.1)\.
- Y\. Chow and M\. Ghavamzadeh \(2014\)Algorithms for CVaR optimization in MDPs\.InAdvances in Neural Information Processing Systems,Vol\.27\.Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p2.1)\.
- A\. Coache, S\. Jaimungal, and Á\. Cartea \(2023\)Conditionally elicitable dynamic risk measures for deep reinforcement learning\.SIAM Journal on Financial Mathematics14\(4\),pp\. 1249–1289\.External Links:[Document](https://dx.doi.org/10.1137/22M1527209)Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p1.1)\.
- T\. Cui, S\. Ding, H\. Jin, and Y\. Zhang \(2023\)Portfolio constructions in cryptocurrency market: a CVaR\-based deep reinforcement learning approach\.Economic Modelling119,pp\. 106078\.External Links:[Document](https://dx.doi.org/10.1016/j.econmod.2022.106078)Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p2.1)\.
- W\. Dabney, M\. Rowland, M\. G\. Bellemare, and R\. Munos \(2018\)Distributional reinforcement learning with quantile regression\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.32\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v32i1.11791)Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p2.1)\.
- J\. Duchi, E\. Hazan, and Y\. Singer \(2011\)Adaptive subgradient methods for online learning and stochastic optimization\.Journal of Machine Learning Research12,pp\. 2121–2159\.Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p5.1)\.
- B\. Enkhsaikhan and O\. Jo \(2024\)Risk\-averse reinforcement learning for portfolio optimization\.ICT Express10\(4\),pp\. 857–862\.External Links:[Document](https://dx.doi.org/10.1016/j.icte.2024.04.010)Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p1.1)\.
- E\. Even\-Dar and Y\. Mansour \(2003\)Learning rates for Q\-learning\.Journal of Machine Learning Research5,pp\. 1–25\.Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p5.1)\.
- M\. B\. Gordy and S\. Juneja \(2010\)Nested simulation in portfolio risk measurement\.Management Science56\(10\),pp\. 1833–1848\.Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p5.1),[§2\.4](https://arxiv.org/html/2608.04305#S2.SS4.SSS0.Px3.p1.2)\.
- P\. Henderson, R\. Islam, P\. Bachman, J\. Pineau, D\. Precup, and D\. Meger \(2018\)Deep reinforcement learning that matters\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.32\.Cited by:[§3\.2](https://arxiv.org/html/2608.04305#S3.SS2.p2.4)\.
- W\. Huang and W\. B\. Haskell \(2021\)Stochastic approximation for risk\-aware markov decision processes\.IEEE Transactions on Automatic Control66\(3\),pp\. 1314–1320\.Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p3.1),[§1\.3](https://arxiv.org/html/2608.04305#S1.SS3.p3.1),[§2\.4](https://arxiv.org/html/2608.04305#S2.SS4.SSS0.Px1.p1.4)\.
- C\. Jin, Z\. Allen\-Zhu, S\. Bubeck, and M\. I\. Jordan \(2018\)Is Q\-learning provably efficient?\.InAdvances in Neural Information Processing Systems,Vol\.31\.Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p5.1)\.
- V\. R\. Konda and J\. N\. Tsitsiklis \(1999\)Actor\-critic algorithms\.InAdvances in Neural Information Processing Systems,Vol\.12,pp\. 1008–1014\.Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p3.1)\.
- X\. Li, W\. Yang, J\. Liang, Z\. Zhang, and M\. I\. Jordan \(2023\)A statistical analysis of polyak–ruppert averaged Q\-learning\.InProceedings of the 26th International Conference on Artificial Intelligence and Statistics,Proceedings of Machine Learning Research, Vol\.206,pp\. 2207–2261\.Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p5.1),[§2\.3](https://arxiv.org/html/2608.04305#S2.SS3.SSS0.Px2.p1.4)\.
- S\. H\. Lim and I\. Malik \(2022\)Distributional reinforcement learning for risk\-sensitive policies\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p2.1)\.
- A\. W\. Moore and C\. G\. Atkeson \(1993\)Prioritized sweeping: reinforcement learning with less data and less time\.Machine Learning13\(1\),pp\. 103–130\.Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p5.1)\.
- L\. A\. Prashanth \(2014\)Policy gradients for CVaR\-constrained MDPs\.InAlgorithmic Learning Theory,Lecture Notes in Computer Science, Vol\.8776,pp\. 155–169\.Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p2.1)\.
- R\. T\. Rockafellar and S\. Uryasev \(2000\)Optimization of conditional value\-at\-risk\.Journal of Risk2\(3\),pp\. 21–41\.External Links:[Document](https://dx.doi.org/10.21314/JOR.2000.038)Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p2.1),[§1\.2](https://arxiv.org/html/2608.04305#S1.SS2.p1.7)\.
- T\. Schaul, G\. Ostrovski, I\. Kemaev, and D\. Borsa \(2021\)Return\-based scaling: yet another normalisation trick for deep RL\.Note:arXiv preprint arXiv:2105\.05347Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p5.1)\.
- T\. Schaul, J\. Quan, I\. Antonoglou, and D\. Silver \(2016\)Prioritized experience replay\.InInternational Conference on Learning Representations,Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p5.1)\.
- Y\. Shen, R\. Huang, C\. Yan, and K\. Obermayer \(2014\)Risk\-averse reinforcement learning for algorithmic trading\.In2014 IEEE Conference on Computational Intelligence for Financial Engineering & Economics \(CIFEr\),pp\. 391–398\.External Links:[Document](https://dx.doi.org/10.1109/CIFEr.2014.6924100)Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p1.1)\.
- M\. Tokic \(2010\)Adaptiveϵ\\epsilon\-greedy exploration in reinforcement learning based on value differences\.InKI 2010: Advances in Artificial Intelligence,Lecture Notes in Computer Science, Vol\.6359,pp\. 203–210\.Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p5.1)\.
- N\. Vadori, S\. Ganesh, P\. P\. Reddy, and M\. Veloso \(2020\)Risk\-sensitive reinforcement learning: a martingale approach to reward uncertainty\.InProceedings of the First ACM International Conference on AI in Finance,External Links:[Document](https://dx.doi.org/10.1145/3383455.3422519)Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p1.1)\.
- H\. van Hasselt, A\. Guez, M\. Hessel, V\. Mnih, and D\. Silver \(2016\)Learning values across many orders of magnitude\.InAdvances in Neural Information Processing Systems,Vol\.29\.Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p5.1)\.
- E\. Vittori, M\. Trapletti, and M\. Restelli \(2020\)Option hedging with risk averse reinforcement learning\.InProceedings of the First ACM International Conference on AI in Finance,External Links:[Document](https://dx.doi.org/10.1145/3383455.3422532)Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p1.1)\.
- Y\. Wang and M\. P\. Chapman \(2022\)Risk\-averse autonomous systems: a brief history and recent developments from the perspective of optimal control\.Artificial Intelligence311,pp\. 103743\.Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p3.1)\.
- C\. J\. C\. H\. Watkins and P\. Dayan \(1992\)Q\-learning\.Machine Learning8\(3–4\),pp\. 279–292\.Cited by:[§1\.1](https://arxiv.org/html/2608.04305#S1.SS1.p3.1)\.