Distributional Reinforcement Learning via the Cram\'er Distance

# Distributional Reinforcement Learning via the Cramér Distance
Source: [https://arxiv.org/html/2605.08104](https://arxiv.org/html/2605.08104)
I\. NowakHAW Hamburg,ivo\.nowak@haw\-hamburg\.deE\.M\.T\. HendrixUniversidad de Málaga,eligius@uma\.es

###### Abstract

This paper explores the application of the Soft Actor\-Critic \(SAC\) algorithm within a Distributional Reinforcement Learning setting and introduces an implementation of such algorithm named Cramér\-based Distributional Soft Actor\-Critic \(C\-DSAC\)\. The novel approach employs distributional reinforcement learning to represent state\-action values, and minimizes the squared Cramér distance for learning the distribution\. Empirical results across various robotic benchmarks indicate that our algorithm surpasses the performance of baseline SAC and contemporary distributional methods, with the performance advantage becoming increasingly pronounced in high\-complexity environments\. To explain the efficiency of the new approach, we conduct an analysis showing that its superior performance is partly due toconfidence\-drivenQ\-value updates: High\-variance target distributions \(low confidence in target\) lead to more conservative model updates, thereby attenuating the impact of overestimated values\. This work deepens the understanding of distributional reinforcement learning, offering insights into the algorithmic mechanisms governing convergence and value estimation\.

## 1Introduction

In science and technology, deep reinforcement learning algorithms have been successfully applied to various problems\. A notable example is AlphaTensor\(Fawziet al\.,[2022](https://arxiv.org/html/2605.08104#bib.bib20)\); a framework/agent exhibiting superhuman performance for automating algorithmic discoveries in matrix multiplication\. Reinforcement learning\(Sutton and Barto,[2018](https://arxiv.org/html/2605.08104#bib.bib78)\)is used \(among other techniques\) in contemporary large language models for alignment\(Ouyanget al\.,[2022](https://arxiv.org/html/2605.08104#bib.bib151)\)and improving model reasoning\(Zhanget al\.,[2025](https://arxiv.org/html/2605.08104#bib.bib162)\)\. In robotic tasks, this approach has been explored for data\-driven generation of optimal control\(Guet al\.,[2017](https://arxiv.org/html/2605.08104#bib.bib100)\)\. Nevertheless, applying reinforcement learning presents significant challenges, primarily due to the high sample complexity and the limited amount of data available in real\-world scenarios\(Paduraruet al\.,[2021](https://arxiv.org/html/2605.08104#bib.bib48)\)\. What has begun as a promising approach to learn discrete control policies\(Mnihet al\.,[2013](https://arxiv.org/html/2605.08104#bib.bib149)\), and has since evolved to address general problems and mitigate various limitations\(Hasseltet al\.,[2016](https://arxiv.org/html/2605.08104#bib.bib102); Lillicrapet al\.,[2019](https://arxiv.org/html/2605.08104#bib.bib146); Schulmanet al\.,[2017a](https://arxiv.org/html/2605.08104#bib.bib153),[b](https://arxiv.org/html/2605.08104#bib.bib154); Fujimotoet al\.,[2018](https://arxiv.org/html/2605.08104#bib.bib96); Haarnojaet al\.,[2018](https://arxiv.org/html/2605.08104#bib.bib101)\), appears to have reached a developmental standstill in model\-free, sample\-efficient real\-world control\.

A more recent approach byBellemareet al\.\([2017a](https://arxiv.org/html/2605.08104#bib.bib86),[2023](https://arxiv.org/html/2605.08104#bib.bib72)\), which expands and formalizes the distributional perspective on reinforcement learning, has revitalized interest in the field, including among researchers from other disciplines\(Mulleret al\.,[2021](https://arxiv.org/html/2605.08104#bib.bib41)\)\.

In this paper, we investigate the distributional approach to reinforcement learning and introduce a novel method for variance\-inverse gradient weighting using the squared Cramér distance\. We present Cramér\-based Distributional Soft Actor\-Critic \(C\-DSAC\), a framework that integrates maximum\-entropy objectives into a distributional reinforcement learning setting\. Moreover, we derive corresponding formulas for implementation with neural networks and present experimental data demonstrating state\-of\-the\-art performance across tested environments\. In order to explain the efficiency of the new approach, a thorough analysis of the algorithm’s dynamics is conducted\. The focus is on the algorithm’s behavior in presence of approximation errors and system noise and equations for its value gradients are derived showing that the adaptation to Q\-values is slower at state action pairs where the variance of the distributional Q\-model is large, i\.e\. confidence in its value is low\. Furthermore, we emphasize how these equations reveal the algorithm’s inherent mechanism for mitigating overestimation bias\.

This paper is organized as follows\. Section[2](https://arxiv.org/html/2605.08104#S2)reviews the relevant literature\. Section[3](https://arxiv.org/html/2605.08104#S3)introduces basic concepts and notation\. Section[4](https://arxiv.org/html/2605.08104#S4)describes the theoretical basis of the C\-DSAC algorithm and its properties\. In Section[5](https://arxiv.org/html/2605.08104#S5), formulas for implementation with neural networks are derived\. Section[6](https://arxiv.org/html/2605.08104#S6)provides a numerical illustration to showcase the performance of C\-DSAC in comparison with the SAC algorithm\. Section[7](https://arxiv.org/html/2605.08104#S7)summarizes our findings and outlines potential future research directions\.

## 2Related Work

Bellemare et al\.Bellemareet al\.\([2017a](https://arxiv.org/html/2605.08104#bib.bib86)\)provided the first formal analysis of the distributional approach in reinforcement learning, in which the objective is to learn the return distributions rather than their expectations\. They employed the Wasserstein metric to measure distances between return distributions, established convergence guarantees for policy evaluation, and analyzed distributional policy improvement\. It was shown that under mild assumptions, distributional policy evaluation and improvement converged to the optimal policy\. This work culminated in the C51 algorithm, a critic\-only method for discrete action spaces\. However, C51 relies on a heuristic projection step that lacks theoretical alignment with the distributional Bellman operator\. InDuanet al\.\([2022](https://arxiv.org/html/2605.08104#bib.bib18)\)andMaet al\.\([2020](https://arxiv.org/html/2605.08104#bib.bib148)\), the concepts of SAC and Distributional Reinforcement Learning were applied simultaneously for the first time\. The former method relies on the Kullback\-Leibler divergence as a probability metric\. This conflicts with the properties required for distributional policy evaluation, as the distance becomes infinite if the distributions have dissimilar supports\. To address this issue, the authors applied aggressive clipping, which negatively impacts performance\. Moreover, the authors of this work were unable to reproduce the results using their provided software\. The work inMaet al\.\([2020](https://arxiv.org/html/2605.08104#bib.bib148)\)extended the SAC framework by incorporating quantile\-regression\-based distributional reinforcement learning\(Dabneyet al\.,[2018](https://arxiv.org/html/2605.08104#bib.bib137)\)\. This integration spans several quantile\-based approaches, ranging from standard QR\-DQN\(Dabneyet al\.,[2017](https://arxiv.org/html/2605.08104#bib.bib89)\)to more complex architectures like IQN\(Dabneyet al\.,[2018](https://arxiv.org/html/2605.08104#bib.bib137)\)and "Fully Parameterized Quantile Function"\(Yanget al\.,[2019](https://arxiv.org/html/2605.08104#bib.bib81)\)\. The authors reported superior performance compared to common reinforcement learning algorithms in the MuJoCo benchmark environments with the IQN configuration\. While the reliance on IQN entails an expanded parameter set via quantile embedding, it is shown that such complexity is redundant for the novel approach presented in this work\. The results indicate that C\-DSAC’s inherent risk\-aversion enables superior performance through a more streamlined architecture\. The work ofLhéritier and Bondoux \([2022](https://arxiv.org/html/2605.08104#bib.bib145)\)focuses on the Cramér distance "in the setting of fixed quantile levels" and does not present an algorithm for continuous control\. Nam et al\.Namet al\.\([2021](https://arxiv.org/html/2605.08104#bib.bib111)\)similarly utilize the squared Cramér distance within a Distributional Reinforcement Learning framework, primarily to enhance the stability of on\-policy algorithms\. Their core contribution, the Sample\-Replacement \(S​R​\(λ\)SR\(\\lambda\)\) algorithm, establishes a principled distributional generalization of the multi\-stepλ\\lambda\-return\. In contrast, the present work pivots from on\-policy stability to the distinct challenge of overestimation bias inherent in off\-policyT​D​\(0\)TD\(0\)learning\.

## 3Background

Our work builds on maximum\-entropy and distributional reinforcement learning\. In the following sections, we provide an overview of these topics and introduce our notation\.

### 3\.1Reinforcement Learning

In reinforcement learning, the interaction of an agent with its environment is modelled as a MDPℳ=\(𝒮,𝒜,r,P,γ\)\\mathcal\{M\}=\(\\mathcal\{S\},\\mathcal\{A\},r,P,\\gamma\)\. In this context,𝒮\\mathcal\{S\}is the finite state space,𝒜\\mathcal\{A\}denotes the finite action space,r:𝒮×𝒜→\[rm​i​n,rm​a​x\]r:\\mathcal\{S\}\\times\\mathcal\{A\}\\rightarrow\[r\_\{min\},r\_\{max\}\]is the bounded reward function,P:𝒮×𝒜→P​r​\(𝒮\)P:\\mathcal\{S\}\\times\\mathcal\{A\}\\rightarrow Pr\(\\mathcal\{S\}\)defines the transition probability dynamics andγ∈\(0,1\)\\gamma\\in\(0,1\)denotes the discount factor\. At each time steptt, an agent executes an actionat∈𝒜a\_\{t\}\\in\\mathcal\{A\}from statest∈𝒮s\_\{t\}\\in\\mathcal\{S\}according to a policyπ\\piand receives a rewardr​\(st,at\)r\(s\_\{t\},a\_\{t\}\), transitioning into the next statest\+1∼P\(⋅∣st,at\)s\_\{t\+1\}\\sim P\(\\cdot\\mid s\_\{t\},a\_\{t\}\)\. An episodic MDP considers to reach a terminal state atTTwithin finite time\. In this work, the policy is treated as a stochastic quantity, modeled as a probability distribution over the action spaceπ:𝒮→P​r​\(𝒜\)\\pi:\\mathcal\{S\}\\rightarrow Pr\(\\mathcal\{A\}\)andΠ\\Pidenotes the set of such policies\. The action is sampled from the policy, i\.e\.at∼π\(⋅∣st\)a\_\{t\}\\sim\\pi\(\\cdot\\mid s\_\{t\}\)\. The general goal is to find an optimal policyπ∗\\pi^\{\*\}that maximizes the expected discounted return along a trajectory of interactions,

J​\(π\)\\displaystyle J\(\\pi\):=𝔼s0∼d0at∼π\(⋅∣st\)∀t=0,…,T−1,st\+1∼P\(⋅∣st,at\)∀t=0,…,T−1​\[∑t=0T−1γt​r​\(st,at\)\]\\displaystyle=\\mathbb\{E\}\_\{\\begin\{subarray\}\{c\}s\_\{0\}\\sim d\_\{0\}\\\\ a\_\{t\}\\sim\\pi\(\\cdot\\mid s\_\{t\}\)\\forall t=0,\\ldots,T\-1,\\\\ s\_\{t\+1\}\\sim P\(\\cdot\\mid s\_\{t\},a\_\{t\}\)\\forall t=0,\\ldots,T\-1\\end\{subarray\}\}\\left\[\\sum\_\{t=0\}^\{T\-1\}\{\\gamma^\{t\}r\(s\_\{t\},a\_\{t\}\)\}\\right\]\(1\):=𝔼π,P​\[∑t=0T−1γt​r​\(st,at\)\],\\displaystyle=\\mathbb\{E\}\_\{\\pi,P\}\\left\[\\sum\_\{t=0\}^\{T\-1\}\{\\gamma^\{t\}r\(s\_\{t\},a\_\{t\}\)\}\\right\],wheresTs\_\{T\}is an absorbing state\.

The initial state is obtained by drawing from the distributions0∼d0s\_\{0\}\\sim d\_\{0\}\. Assuming discrete spaces, equation \([1](https://arxiv.org/html/2605.08104#S3.E1)\) can be written more explicitly as

J​\(π\)=∑s∈𝒮∑a∈𝒜∑t=0T−1Pr⁡\(st=s∣π,P,s0∼d0\)​π​\(a∣s\)​γt​r​\(s,a\)J\(\\pi\)=\\sum\_\{s\\in\\mathcal\{S\}\}\{\\sum\_\{a\\in\\mathcal\{A\}\}\{\\sum\_\{t=0\}^\{T\-1\}\}\{\\Pr\(s\_\{t\}=s\\mid\\pi,P,s\_\{0\}\\sim d\_\{0\}\)\\pi\(a\\mid s\)\\gamma^\{t\}r\(s,a\)\}\}\(2\)In this setting, the episode has a length of T and thusJJoptimizes a discounted return in a finite MDP\.

The optimal policies can be approximated by policy iteration, a process that incorporates policy evaluation and improvement using Q\-values\. A Q\-value induced by a policyπ\\piis defined as

Qπ​\(sk,ak\)\\displaystyle Q^\{\\pi\}\(s\_\{k\},a\_\{k\}\):=𝔼at∼π\(⋅∣st\)∀t=k\+1,…,T−1st\+1∼P\(⋅∣st,at\)∀t=k,…,T−1​\[∑t=kT−1γt−k​r​\(st,at\)∣sk,ak\]\\displaystyle=\\mathbb\{E\}\_\{\\begin\{subarray\}\{c\}a\_\{t\}\\sim\\pi\(\\cdot\\mid s\_\{t\}\)\\forall t=k\+1,\\ldots,T\-1\\\\ s\_\{t\+1\}\\sim P\(\\cdot\\mid s\_\{t\},a\_\{t\}\)\\forall t=k,\\ldots,T\-1\\end\{subarray\}\}\\left\[\\sum\_\{t=k\}^\{T\-1\}\{\\gamma^\{t\-k\}r\(s\_\{t\},a\_\{t\}\)\}\\mid s\_\{k\},a\_\{k\}\\right\]\(3\):=𝔼π,P​\[∑t=kT−1γt−k​r​\(st,at\)∣sk,ak\],\\displaystyle=\\mathbb\{E\}\_\{\\pi,P\}\\left\[\\sum\_\{t=k\}^\{T\-1\}\{\\gamma^\{t\-k\}r\(s\_\{t\},a\_\{t\}\)\}\\mid s\_\{k\},a\_\{k\}\\right\],and satisfies the recursive Bellman equation

Qπ​\(st,at\)=r​\(st,at\)\+γ​𝔼at\+1∼π\(⋅∣st\+1\),st\+1∼P\(⋅∣st,at\)​\[Qπ​\(st\+1,at\+1\)\]\.Q^\{\\pi\}\(s\_\{t\},a\_\{t\}\)=r\(s\_\{t\},a\_\{t\}\)\+\\gamma\\mathbb\{E\}\_\{\\begin\{subarray\}\{c\}a\_\{t\+1\}\\sim\\pi\(\\cdot\\mid s\_\{t\+1\}\),\\\\ s\_\{t\+1\}\\sim P\(\\cdot\\mid s\_\{t\},a\_\{t\}\)\\end\{subarray\}\}\[Q^\{\\pi\}\(s\_\{t\+1\},a\_\{t\+1\}\)\]\.\(4\)Based on \([4](https://arxiv.org/html/2605.08104#S3.E4)\), we define the Bellman operator𝒯π\\mathcal\{T^\{\\pi\}\},

𝒯π​Q​\(st,at\):=r​\(st,at\)\+γ​𝔼st\+1∼P\(⋅∣st,at\),at\+1∼π\(⋅∣st\+1\)​\[Q​\(st\+1,at\+1\)\]\.\\mathcal\{T\}^\{\\pi\}Q\(s\_\{t\},a\_\{t\}\):=r\(s\_\{t\},a\_\{t\}\)\+\\gamma\\mathbb\{E\}\_\{\\begin\{subarray\}\{c\}s\_\{t\+1\}\\sim P\(\\cdot\\mid s\_\{t\},a\_\{t\}\),\\\\ a\_\{t\+1\}\\sim\\pi\(\\cdot\\mid s\_\{t\+1\}\)\\end\{subarray\}\}\\left\[Q\(s\_\{t\+1\},a\_\{t\+1\}\)\\right\]\.\(5\)For policy evaluation, it can be shown that the sequence of value functionsQk\+1:=𝒯π​QkQ\_\{k\+1\}:=\\mathcal\{T\}^\{\\pi\}Q\_\{k\}, starting from some arbitraryQ0Q\_\{0\}, will exponentially quickly converge toQπQ^\{\\pi\}askkincreases, see Lemma[A\.1](https://arxiv.org/html/2605.08104#A1.Thmtheorem1)\.

Policy improvement, Lemma[A\.2](https://arxiv.org/html/2605.08104#A1.Thmtheorem2), involves exploitation of knowledge about the environment and updating towards higher values, i\.e\.

πk\+1​\(st\)=argmaxπ∈Π𝔼at∼π\(⋅∣st\)​Qπk​\(st,at\)\.\\pi\_\{k\+1\}\(s\_\{t\}\)=\\operatorname\*\{argmax\}\_\{\\pi\\in\\Pi\}\\mathbb\{E\}\_\{a\_\{t\}\\sim\\pi\(\\cdot\\mid s\_\{t\}\)\}\{Q^\{\\pi\_\{k\}\}\(s\_\{t\},a\_\{t\}\)\}\.\(6\)
In that manner, the policy will be non\-decreasing\. The estimation of the state\-action value can be improved by better exploration of the state\-space\.

Convergence towards the optimal policyπ∗\\pi^\{\*\}is guaranteed under policy iteration, see Lemma[A\.3](https://arxiv.org/html/2605.08104#A1.Thmtheorem3)\.

### 3\.2Maximum Entropy Reinforcement Learning

In this setting, the standard objective in Eq\.\([1](https://arxiv.org/html/2605.08104#S3.E1)\) is augmented with policy entropy to improve exploration

Jh​\(π\)\\displaystyle J\_\{h\}\(\\pi\):=𝔼s0∼d0at∼π\(⋅∣st\)∀t=0,…,T−1,st\+1∼P\(⋅∣st,at\)∀t=0,…,T−1\[∑t=0T−1γt\(r\(st,at\)\+αℋ\(π\(⋅∣st\)\)\]\]\\displaystyle=\\mathbb\{E\}\_\{\\begin\{subarray\}\{c\}s\_\{0\}\\sim d\_\{0\}\\\\ a\_\{t\}\\sim\\pi\(\\cdot\\mid s\_\{t\}\)\\forall t=0,\\ldots,T\-1,\\\\ s\_\{t\+1\}\\sim P\(\\cdot\\mid s\_\{t\},a\_\{t\}\)\\forall t=0,\\ldots,T\-1\\end\{subarray\}\}\\left\[\\sum\_\{t=0\}^\{T\-1\}\{\\gamma^\{t\}\\left\(r\(s\_\{t\},a\_\{t\}\)\+\\alpha\\mathcal\{H\}\(\\pi\(\\cdot\\mid\\ s\_\{t\}\)\)\\right\]\}\\right\]\(7\):=𝔼π,P\[∑t=0T−1γt\(r\(st,at\)\+αℋ\(π\(⋅∣st\)\)\)\],\\displaystyle=\\mathbb\{E\}\_\{\\pi,P\}\\left\[\\sum\_\{t=0\}^\{T\-1\}\{\\gamma^\{t\}\(r\(s\_\{t\},a\_\{t\}\)\+\\alpha\\mathcal\{H\}\(\\pi\(\\cdot\\mid\\ s\_\{t\}\)\)\)\}\\right\],whereℋ\(π\(⋅\|st\)\):=𝔼at∼π\(⋅\|st\)\[−logπ\(at\|st\)\]\\mathcal\{H\}\(\\pi\(\\cdot\|s\_\{t\}\)\):=\\mathbb\{E\}\_\{a\_\{t\}\\sim\\pi\(\\cdot\|s\_\{t\}\)\}\[\-\\log\\pi\(a\_\{t\}\|s\_\{t\}\)\]\(Haarnojaet al\.,[2018](https://arxiv.org/html/2605.08104#bib.bib101)\)\.

Soft policy improvement involves the information projection and update towards the exponential of the new Q\-value,

πk\+1\(⋅∣st\)=argminπ\(⋅∣st\)dk​l\(π\(⋅\|st\)\|\|exp⁡\(1α​Qπk​\(st,⋅\)\)Wπk​\(st\)\),\\pi\_\{k\+1\}\(\\cdot\\mid s\_\{t\}\)=\\operatorname\*\{argmin\}\_\{\\pi\(\\cdot\\mid s\_\{t\}\)\}d\_\{kl\}\\left\(\\pi\(\\cdot\\operatorname\{\\,\|\\,\}s\_\{t\}\)\\mathrel\{\\Big\|\\Big\|\}\\frac\{\\exp\(\\frac\{1\}\{\\alpha\}Q^\{\\pi\_\{k\}\}\(s\_\{t\},\\cdot\)\)\}\{W^\{\\pi\_\{k\}\}\(s\_\{t\}\)\}\\right\),\(8\)whereα\>0\\alpha\>0,dk​ld\_\{kl\}is the Kullback\-Leibler divergence function andWπk​\(st\)=∑i\|𝒜\|exp⁡\(1α​Qπk​\(st,ai\)\)W^\{\\pi\_\{k\}\}\(s\_\{t\}\)=\\sum\_\{i\}^\{\|\\mathcal\{A\}\|\}\{\\exp\(\{\\frac\{1\}\{\\alpha\}Q^\{\\pi\_\{k\}\}\(s\_\{t\},a\_\{i\}\)\}\)\}denotes the partition function normalizing the distribution\.Wπk​\(st\)W^\{\\pi\_\{k\}\}\(s\_\{t\}\)can be ignored since it does not contribute to the new policy\. The parameterα\\alpharepresents the impact of policy stochasticity on the state\-action value\.

Analogous to the definition of the Bellman operator for standard reinforcement learning, a modified Bellman operator𝒯hπ\\mathcal\{T\}^\{\\pi\}\_\{h\}is defined as

𝒯hπQ\(st,at\):=r\(st,at\)\+γ𝔼st\+1∼P\(⋅∣st,at\)at\+1∼π\(⋅∣st\+1\)\[Q\(st\+1,at\+1\)\+αℋ\(π\(⋅∣st\+1\)\)\]\.\\mathcal\{T\}\_\{h\}^\{\\pi\}Q\(s\_\{t\},a\_\{t\}\):=r\(s\_\{t\},a\_\{t\}\)\+\\gamma\\mathbb\{E\}\_\{\\begin\{subarray\}\{c\}s\_\{t\+1\}\\sim P\(\\cdot\\mid s\_\{t\},a\_\{t\}\)\\\\ a\_\{t\+1\}\\sim\\pi\(\\cdot\\mid s\_\{t\+1\}\)\\end\{subarray\}\}\[Q\(s\_\{t\+1\},a\_\{t\+1\}\)\+\\alpha\\mathcal\{H\}\(\\pi\(\\cdot\\mid s\_\{t\+1\}\)\)\]\.\(9\)The soft state\-action value is given by

Vhπ\(st\):=𝔼at∼π\(⋅∣st\)\[Qπ\(st,at\)\+αℋ\(π\(⋅∣st\)\)\]\.V\_\{h\}^\{\\pi\}\(s\_\{t\}\):=\\mathbb\{E\}\_\{a\_\{t\}\\sim\\pi\(\\cdot\\mid s\_\{t\}\)\}\[Q^\{\\pi\}\(s\_\{t\},a\_\{t\}\)\+\\alpha\\mathcal\{H\}\(\\pi\(\\cdot\\mid s\_\{t\}\)\)\]\.As shown in the Lemma[B\.1](https://arxiv.org/html/2605.08104#A2.Thmtheorem1), the operator𝒯hπ\\mathcal\{T\}\_\{h\}^\{\\pi\}is aγ\\gamma\-contraction, analogous to𝒯π\\mathcal\{T\}^\{\\pi\}\.

Soft policy iteration is guaranteed to converge to the optimal policyπ∗\\pi^\{\*\}\(Lemma[B\.4](https://arxiv.org/html/2605.08104#A2.Thmtheorem4)\), proceeding through alternating steps of soft policy evaluation under the soft Bellman operator𝒯hπ\\mathcal\{T\}\_\{h\}^\{\\pi\}and soft policy improvement \(Lemmas[B\.2](https://arxiv.org/html/2605.08104#A2.Thmtheorem2)and[B\.3](https://arxiv.org/html/2605.08104#A2.Thmtheorem3)\)\.

### 3\.3Distributional Reinforcement Learning

Distributional reinforcement learning \(DistRL\) shifts the focus from modeling expected returns, as in traditional value\-based methods, to capturing the entire distribution of returns\. Therefore, the state\-action value function is treated as random variable\. The distribution over the return is defined as

Zπ​\(sk,ak\)\\displaystyle Z^\{\\pi\}\(s\_\{k\},a\_\{k\}\):=∑t=kT−1γt−kr\(st,at\),\(at∼π\(⋅∣st\)∀t=k\+1…T−1,\\displaystyle=\\sum\_\{t=k\}^\{T\-1\}\{\\gamma^\{t\-k\}r\(s\_\{t\},a\_\{t\}\)\},\(a\_\{t\}\\sim\\pi\(\\cdot\\mid s\_\{t\}\)\\forall t=k\+1\\ldots T\-1,\(10\)st\+1∼P\(⋅∣st,at\)∀t=k…T−1\)∣sk,ak\.\\displaystyle\\quad\\quad\\quad s\_\{t\+1\}\\sim P\(\\cdot\\mid s\_\{t\},a\_\{t\}\)\\forall t=k\\ldots T\-1\)\\mid s\_\{k\},a\_\{k\}\.If the reward is treated as a deterministic quantity, then stochasticity in the return distribution arises from the transition kernel and the stochastic policy\. Although DistRL was devised originally for aleatoric uncertainty\(Bellemareet al\.,[2017a](https://arxiv.org/html/2605.08104#bib.bib86)\), i\.e\. the reward is explicitly treated as a random variable, experiments show that distributional variants outperform their scalar counterparts even in environments with deterministic rewards, see Section[6](https://arxiv.org/html/2605.08104#S6)\. The expectation recovers the state\-action value,

Qπ​\(sk,ak\)=𝔼​\[Zπ​\(sk,ak\)\]\.Q^\{\\pi\}\(s\_\{k\},a\_\{k\}\)=\\mathbb\{E\}\[Z^\{\\pi\}\(s\_\{k\},a\_\{k\}\)\]\.\(11\)Despite prior investigations into the mechanisms explaining accelerated learning\(Bellemareet al\.,[2017a](https://arxiv.org/html/2605.08104#bib.bib86); Wanget al\.,[2023](https://arxiv.org/html/2605.08104#bib.bib161),[2024](https://arxiv.org/html/2605.08104#bib.bib127)\), a comprehensive understanding remains elusive\. We contribute to this discussion by introducing confidence\-driven model updates in Section[5\.2](https://arxiv.org/html/2605.08104#S5.SS2)\. The distributional counterpart of the Bellman operator in \([5](https://arxiv.org/html/2605.08104#S3.E5)\) is used for distributional evaluation\. We define it under the Wasserstein metric as

𝒯DπZ\(st,at\):=dr\(st,at\)\+γZ\(st\+1,at\+1\)\|st\+1∼P\(⋅∣st,at\),at\+1∼π\(⋅∣st\+1\),\\mathcal\{T\}^\{\\pi\}\_\{D\}Z\(s\_\{t\},a\_\{t\}\):\\operatorname\*\{=\}^\{d\}r\(s\_\{t\},a\_\{t\}\)\+\\gamma Z\(s\_\{t\+1\},a\_\{t\+1\}\)\\operatorname\{\\,\|\\,\}s\_\{t\+1\}\\sim P\(\\cdot\\mid s\_\{t\},a\_\{t\}\),a\_\{t\+1\}\\sim\\pi\(\\cdot\\mid s\_\{t\+1\}\),\(12\)where=d\\operatorname\*\{=\}^\{d\}indicates that the random variable on both sides are distributed according to the same law\. It can be shown that𝒯Dπ\\mathcal\{T\}\_\{D\}^\{\\pi\}is aγ\\gamma\-contraction\(Bellemareet al\.,[2017a](https://arxiv.org/html/2605.08104#bib.bib86)\), a property needed for distributional policy evaluation\. Theorem[4\.1](https://arxiv.org/html/2605.08104#S4.Thmtheorem1)establishes that theγ\\gamma\-contraction property of the distributional Bellman operator extends to the squared Cramér distance\.

Bellemare et al\.Bellemareet al\.\([2017a](https://arxiv.org/html/2605.08104#bib.bib86)\)analyze the distributional Bellman optimality operator, demonstrating that convergence to the optimal distribution holds only under the condition of a unique optimal policy or a total ordering on the set of optimal policies\.

### 3\.4The Squared Cramér Distance

Choosing an appropriate metric to quantify the distance between random variablesUUandVVis critical for policy evaluation and improvement in distributional reinforcement learning\.

Bellemare et al\.\(Bellemareet al\.,[2017b](https://arxiv.org/html/2605.08104#bib.bib134)\)noted that a metric must be both "ideal" according to Zolotarev et al\.Zolotarev \([1976](https://arxiv.org/html/2605.08104#bib.bib67)\)and suitable for gradient descent methods to be effective in machine learning with distributions\. The squaredL2L\_\{2\}Cramér distance possesses such ideal properties and is suited for machine learning\.

To formalize this, consider the properL2L\_\{2\}Cramér metric\. For two random variablesUUandVVit is defined as

dc​\(U,V\):=\(∫−∞∞\(FU​\(x\)−FV​\(x\)\)2​𝑑x\)12,d\_\{c\}\(U,V\):=\\left\(\\int\_\{\-\\infty\}^\{\\infty\}\{\(F\_\{U\}\(x\)\-F\_\{V\}\(x\)\)^\{2\}dx\}\\right\)^\{\\frac\{1\}\{2\}\},\(13\)whereFUF\_\{U\}andFVF\_\{V\}are the cumulative distribution functions ofUUandVVrespectively\. Note that the distance is formally between two laws of the random variables, dc​\(ℒ​\(U\),ℒ​\(V\)\)d\_\{c\}\(\\mathcal\{L\}\(U\),\\mathcal\{L\}\(V\)\), but the common shorthanddc​\(U,V\)d\_\{c\}\(U,V\)is used to avoid convoluted notation\. Ideal metrics require to be sum invariant and scale sensitive\. The Cramér metric is sum invariant, therefore it holds

dc​\(A\+U,A\+V\)≤dc​\(U,V\),d\_\{c\}\(A\+U,A\+V\)\\leq d\_\{c\}\(U,V\),\(C1\)whereAAis a random variable independent ofUUandVV\. TreatingAAas a degenerate random variable withP​\(A=a\)=1P\(A=a\)=1, this property can be also stated in terms of a constant

dc​\(a\+U,a\+V\)≤dc​\(U,V\)\.d\_\{c\}\(a\+U,a\+V\)\\leq d\_\{c\}\(U,V\)\.\(14\)Scale sensitivity, the second required property, ensures that fork\>0k\>0,

dc​\(k​U,k​V\)=\|k\|12​dc​\(U,V\)\.d\_\{c\}\(kU,kV\)=\|k\|^\{\\frac\{1\}\{2\}\}d\_\{c\}\(U,V\)\.\(C2\)
###### Lemma 3\.1\.

The Cramér metricdcd\_\{c\}satisfies properties \([C1](https://arxiv.org/html/2605.08104#S3.Ex2)\) and \([C2](https://arxiv.org/html/2605.08104#S3.Ex3)\)\.

###### Proof\.

See Theorem 2 ofBellemareet al\.\([2017b](https://arxiv.org/html/2605.08104#bib.bib134)\)\. ∎

A third property is needed to guarantee unbiased sample gradients\. Within the class ofLpL\_\{p\}Cramér metrics, only the squared Cramér distance \(p=2p=2\), also called the energy distance, satisfies this requirement\. The energy distance is expressed as

de​\(U,V\):=dc2=∫−∞∞\(FU​\(x\)−FV​\(x\)\)2​𝑑x,d\_\{e\}\(U,V\):=d\_\{c\}^\{2\}=\\int\_\{\-\\infty\}^\{\\infty\}\{\(F\_\{U\}\(x\)\-F\_\{V\}\(x\)\)^\{2\}dx\},\(15\)and has the unbiased sample gradient property

𝔼𝐗𝐦∼Γ​∇θde​\(Γ^m,Zθ\)=∇θde​\(Γ,Zθ\),\\mathbb\{E\}\_\{\\mathbf\{X\_\{m\}\}\\sim\\Gamma\}\\nabla\_\{\\theta\}d\_\{e\}\(\\hat\{\\Gamma\}\_\{m\},Z\_\{\\theta\}\)=\\nabla\_\{\\theta\}d\_\{e\}\(\\Gamma,Z\_\{\\theta\}\),\(C3\)where𝐗m\\mathbf\{X\}\_\{m\}is a vector with elementsX1,…,XmX\_\{1\},\\ldots,X\_\{m\}drawn from a Bernoulli distributionΓ\\Gamma, andΓ^m:=1m​∑i=1mδXi\\hat\{\\Gamma\}\_\{m\}:=\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}\\delta\_\{X\_\{i\}\}is an approximate distribution formed by the samples withδ\\deltabeing Dirac functions at valuesXiX\_\{i\}\.

###### Lemma 3\.2\.

The energy distanceded\_\{e\}satisfies properties \([C1](https://arxiv.org/html/2605.08104#S3.Ex2)\),\([C3](https://arxiv.org/html/2605.08104#S3.Ex4)\), and

de​\(k​U,k​V\)=k​de​\(U,V\),d\_\{e\}\(kU,kV\)=kd\_\{e\}\(U,V\),\(C4\)withk\>0k\>0\.

###### Proof\.

See Appendix A3 ofBellemareet al\.\([2017b](https://arxiv.org/html/2605.08104#bib.bib134)\)\. ∎

By parity of reasoning with the derivation of Equation \([14](https://arxiv.org/html/2605.08104#S3.E14)\), this identity extends to the case of constant values\.

The following Lemma provides the last property necessary to build the algorithm’s theoretical framework\.

###### Lemma 3\.3\.

Letded\_\{e\}be the energy distance as defined in \([15](https://arxiv.org/html/2605.08104#S3.E15)\)\. For any collection of distributions\{μi,νi\}i=1k\\\{\\mu\_\{i\},\\nu\_\{i\}\\\}\_\{i=1\}^\{k\}and non\-negative weights\{wi\}\\\{w\_\{i\}\\\}such thatμ=∑iwi​μi\\mu=\\sum\_\{i\}\{w\_\{i\}\\mu\_\{i\}\},ν=∑iwi​νi\\nu=\\sum\_\{i\}\{w\_\{i\}\\nu\_\{i\}\}and∑iwi=1\\sum\_\{i\}\{w\_\{i\}\}=1, the following inequality holds, where for notational consistency the energy distance is written in terms of variablesU∼μU\\sim\\mu,V∼νV\\sim\\nuandUi∼μiU\_\{i\}\\sim\\mu\_\{i\},Vi∼νiV\_\{i\}\\sim\\nu\_\{i\}, respectively

de​\(U,V\)≤∑i=1kwi​de​\(Ui,Vi\)\.d\_\{e\}\(U,V\)\\leq\\sum\_\{i=1\}^\{k\}\{w\_\{i\}d\_\{e\}\(U\_\{i\},V\_\{i\}\)\}\.\(C5\)

###### Proof\.

By Theorem 22 of Sejdinovic et al\.Sejdinovicet al\.\([2013](https://arxiv.org/html/2605.08104#bib.bib55)\), the energy distance equals a constant multiple of the RKHS norm of kernel mean embeddings\. Using linearity and standard results in the RKHS yields \([C5](https://arxiv.org/html/2605.08104#S3.Ex6)\)\. ∎

## 4The C\-DSAC Framework

Building on the principles of distributional and maximum\-entropy reinforcement learning, we introduce the theoretical framework of our proposed algorithm and offer a comprehensive theoretical justification for its approach in this section\.

#### Distributional Maximum Entropy Reinforcement Learning

The C\-DSAC objective is equivalent to \([7](https://arxiv.org/html/2605.08104#S3.E7)\), with the distributional soft state\-action value made explicit,

Jh​\(π\)=𝔼s0∼d0​\[Vhπ​\(s0\)\],J\_\{h\}\(\\pi\)=\\mathbb\{E\}\_\{\\begin\{subarray\}\{c\}s\_\{0\}\\sim d\_\{0\}\\end\{subarray\}\}\\left\[V\_\{h\}^\{\\pi\}\(s\_\{0\}\)\\right\],\(16\)where

Vhπ​\(st\):=𝔼at∼π\(⋅∣st\)​\[𝔼​\[Zhπ​\(st,at\)\]−α​log⁡π​\(at∣st\)\]V\_\{h\}^\{\\pi\}\(s\_\{t\}\):=\\mathbb\{E\}\_\{a\_\{t\}\\sim\\pi\(\\cdot\\mid s\_\{t\}\)\}\[\\mathbb\{E\}\[Z\_\{h\}^\{\\pi\}\(s\_\{t\},a\_\{t\}\)\]\-\\alpha\\log\{\\pi\(a\_\{t\}\\mid s\_\{t\}\)\}\]and

Zhπ​\(st,at\)\\displaystyle Z\_\{h\}^\{\\pi\}\(s\_\{t\},a\_\{t\}\)=r​\(st,at\)\+γ​\(Zhπ​\(st\+1,at\+1\)−α​log⁡π​\(at\+1∣st\+1\)\)\\displaystyle=r\(s\_\{t\},a\_\{t\}\)\+\\gamma\(Z\_\{h\}^\{\\pi\}\(s\_\{t\+1\},a\_\{t\+1\}\)\-\\alpha\\log\{\\pi\(a\_\{t\+1\}\\mid s\_\{t\+1\}\)\}\)∣st\+1∼P\(⋅∣st,at\),at\+1∼π\(⋅∣st\+1\)\.\\displaystyle\\mid s\_\{t\+1\}\\sim P\(\\cdot\\mid s\_\{t\},a\_\{t\}\),a\_\{t\+1\}\\sim\\pi\(\\cdot\\mid s\_\{t\+1\}\)\.The aim is to find a policyπ∗=argmaxπ∈ΠJH​\(π\)\\pi^\{\*\}=\\operatorname\*\{argmax\}\_\{\\pi\\in\\Pi\}J\_\{H\}\(\\pi\)\. We chose the negative log probability to quantify entropy\.

#### Policy Evaluation

Consider a distributional reinforcement learning setting under the squaredL2L\_\{2\}Cramér distance where the distribution over returnsZZis utilized\. Then the soft distributional Bellman operator is defined as

𝒯Hπ​Z​\(st,at\):=d\\displaystyle\\mathcal\{T\}\_\{H\}^\{\\pi\}Z\(s\_\{t\},a\_\{t\}\):\\operatorname\*\{=\}^\{d\}r​\(st,at\)\+γ​\(Z​\(st\+1,at\+1\)−α​log⁡π​\(at\+1∣st\+1\)\)\\displaystyle r\(s\_\{t\},a\_\{t\}\)\+\\gamma\(Z\(s\_\{t\+1\},a\_\{t\+1\}\)\-\\alpha\\log\{\\pi\(a\_\{t\+1\}\\mid s\_\{t\+1\}\)\}\)\(17\)\|st\+1∼P\(⋅∣st,at\),at\+1∼π\(⋅∣st\+1\)\.\\displaystyle\\operatorname\{\\,\|\\,\}s\_\{t\+1\}\\sim P\(\\cdot\\mid s\_\{t\},a\_\{t\}\),a\_\{t\+1\}\\sim\\pi\(\\cdot\\mid s\_\{t\+1\}\)\.
###### Theorem 4\.1\.

Fix1≤p<∞1\\leq p<\\inftyand\(st,at\)∈𝒮×𝒜\(s\_\{t\},a\_\{t\}\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\. Letd¯e​\(Z1,Z2\):=sups,ade​\(Z1​\(s,a\),Z2​\(s,a\)\)\\bar\{d\}\_\{e\}\(Z\_\{1\},Z\_\{2\}\):=\\sup\_\{s,a\}d\_\{e\}\(Z\_\{1\}\(s,a\),Z\_\{2\}\(s,a\)\)for two state\-action value distributionsZ1Z\_\{1\}andZ2Z\_\{2\}\. Assume that for each\(s,a\)\(s,a\), the distributions have finite first moments to attain completeness of the metric space\. Further, assume\|𝒜\|<∞\|\\mathcal\{A\}\|<\\infty,\|𝒮\|<∞\|\\mathcal\{S\}\|<\\infty,0≤γ<10\\leq\\gamma<1andα≥0\\alpha\\geq 0\. Then𝒯Hπ\\mathcal\{T\}\_\{H\}^\{\\pi\}is aγ\\gamma\-contraction in the energy distance distance, i\.e\.

d¯e​\(𝒯Hπ​Z1,𝒯Hπ​Z2\)≤γ​d¯e​\(Z1,Z2\)\.\\bar\{d\}\_\{e\}\(\\mathcal\{T\}\_\{H\}^\{\\pi\}Z\_\{1\},\\mathcal\{T\}\_\{H\}^\{\\pi\}Z\_\{2\}\)\\leq\\gamma\\bar\{d\}\_\{e\}\(Z\_\{1\},Z\_\{2\}\)\.\(18\)

###### Proof\.

The reward can be augmented with the entropy term, making it a random variable\. Let

Rh\(st,at\):=r\(st,at\)−γαlogπ\(at\+1∣st\+1\)\|st\+1∼P\(st,at\),at\+1∼π\(⋅∣st\+1\),R\_\{h\}\(s\_\{t\},a\_\{t\}\):=r\(s\_\{t\},a\_\{t\}\)\-\\gamma\\alpha\\log\{\\pi\(a\_\{t\+1\}\\mid s\_\{t\+1\}\)\}\\operatorname\{\\,\|\\,\}s\_\{t\+1\}\\sim P\(s\_\{t\},a\_\{t\}\),a\_\{t\+1\}\\sim\\pi\(\\cdot\\mid s\_\{t\+1\}\),then

de\(\\displaystyle d\_\{e\}\(𝒯HπZ1,𝒯HπZ2\)\\displaystyle\\mathcal\{T\}\_\{H\}^\{\\pi\}Z\_\{1\},\\mathcal\{T\}\_\{H\}^\{\\pi\}Z\_\{2\}\)=de​\(Rh​\(st,at\)\+γ​Z1​\(st\+1,at\+1\),Rh​\(st,at\)\+γ​Z2​\(st\+1,at\+1\)\)\\displaystyle=d\_\{e\}\\left\(R\_\{h\}\(s\_\{t\},a\_\{t\}\)\+\\gamma Z\_\{1\}\(s\_\{t\+1\},a\_\{t\+1\}\),R\_\{h\}\(s\_\{t\},a\_\{t\}\)\+\\gamma Z\_\{2\}\(s\_\{t\+1\},a\_\{t\+1\}\)\\right\)wherest\+1∼P\(⋅∣st,at\),at\+1∼π\(⋅∣st\+1\)\.\\displaystyle\\quad\\quad\\text\{where \\ \}s\_\{t\+1\}\\sim P\(\\cdot\\mid s\_\{t\},a\_\{t\}\),a\_\{t\+1\}\\sim\\pi\(\\cdot\\mid s\_\{t\+1\}\)\.By Property \([C5](https://arxiv.org/html/2605.08104#S3.Ex6)\) withwt\+1:=P​\(st\+1∣st,at\)​π​\(at\+1​\|⁡st\+1\)w\_\{t\+1\}:=P\(s\_\{t\+1\}\\mid s\_\{t\},a\_\{t\}\)\\pi\(a\_\{t\+1\}\\operatorname\{\\,\|\\,\}s\_\{t\+1\}\)≤∑st\+1,at\+1wt\+1​de​\(Rh​\(st,at\)\+γ​Z1​\(st\+1,at\+1\),Rh​\(st,at\)\+γ​Z2​\(st\+1,at\+1\)\)\.\\displaystyle\\leq\\sum\_\{s\_\{t\+1\},a\_\{t\+1\}\}\{w\_\{t\+1\}d\_\{e\}\\left\(R\_\{h\}\(s\_\{t\},a\_\{t\}\)\+\\gamma Z\_\{1\}\(s\_\{t\+1\},a\_\{t\+1\}\),R\_\{h\}\(s\_\{t\},a\_\{t\}\)\+\\gamma Z\_\{2\}\(s\_\{t\+1\},a\_\{t\+1\}\)\\right\)\}\.Property \([C1](https://arxiv.org/html/2605.08104#S3.Ex2)\) holds forded\_\{e\}\(Lemma[3\.2](https://arxiv.org/html/2605.08104#S3.Thmtheorem2)\) and each state\-action pair is fixed≤∑st\+1,at\+1wt\+1​de​\(γ​Z1​\(st\+1,at\+1\),γ​Z2​\(st\+1,at\+1\)\)\.\\displaystyle\\leq\\sum\_\{s\_\{t\+1\},a\_\{t\+1\}\}\{w\_\{t\+1\}d\_\{e\}\\left\(\\gamma Z\_\{1\}\(s\_\{t\+1\},a\_\{t\+1\}\),\\gamma Z\_\{2\}\(s\_\{t\+1\},a\_\{t\+1\}\)\\right\)\}\.By Property \([C4](https://arxiv.org/html/2605.08104#S3.Ex5)\)=γ​∑st\+1,at\+1wt\+1​de​\(Z1​\(st\+1,at\+1\),Z2​\(st\+1,at\+1\)\)\.\\displaystyle=\\gamma\\sum\_\{s\_\{t\+1\},a\_\{t\+1\}\}\{w\_\{t\+1\}d\_\{e\}\\left\(Z\_\{1\}\(s\_\{t\+1\},a\_\{t\+1\}\),Z\_\{2\}\(s\_\{t\+1\},a\_\{t\+1\}\)\\right\)\}\.Since∑iwi=1\\sum\_\{i\}\{w\_\{i\}\}=1,≤γ​sups,ade​\(Z1​\(s,a\),Z2​\(s,a\)\)\\displaystyle\\leq\\gamma\\sup\_\{s,a\}d\_\{e\}\(Z\_\{1\}\(s,a\),Z\_\{2\}\(s,a\)\)=γ​d¯e​\(Z1,Z2\)\.\\displaystyle=\\gamma\\bar\{d\}\_\{e\}\(Z\_\{1\},Z\_\{2\}\)\.
Taking the supremum over\(s,a\)∈𝒮×𝒜\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}on both sides yields the contraction

d¯e​\(𝒯Hπ​Z1,𝒯Hπ​Z2\)≤γ​d¯e​\(Z1,Z2\)\.\\bar\{d\}\_\{e\}\(\\mathcal\{T\}\_\{H\}^\{\\pi\}Z\_\{1\},\\mathcal\{T\}\_\{H\}^\{\\pi\}Z\_\{2\}\)\\leq\\gamma\\bar\{d\}\_\{e\}\(Z\_\{1\},Z\_\{2\}\)\.
∎

Since𝒯Hπ\\mathcal\{T\}\_\{H\}^\{\\pi\}is aγ\\gamma\-contraction in maximal form, it follows from the Banach fixed point theorem:

###### Corollary 4\.2\.

GivenZk\+1:=d𝒯Hπ​ZkZ\_\{k\+1\}:\\operatorname\*\{=\}^\{d\}\\mathcal\{T\}\_\{H\}^\{\\pi\}Z\_\{k\}, the series\{Zk\}\\\{Z\_\{k\}\\\}converges to the fixed point \(distribution\)ZhπZ\_\{h\}^\{\\pi\}in the energy distance, i\.e\.,

limk→∞d¯e​\(Zk,Zhπ\)=0with𝒯Hπ​Zhπ​=dZhπ\.\\lim\_\{k\\to\\infty\}\\bar\{d\}\_\{e\}\(Z\_\{k\},Z\_\{h\}^\{\\pi\}\)=0\\quad\\text\{ with \}\\quad\\mathcal\{T\}\_\{H\}^\{\\pi\}Z\_\{h\}^\{\\pi\}\\operatorname\*\{=\}^\{d\}Z\_\{h\}^\{\\pi\}\.\(19\)

#### Policy Improvement

Soft policy improvement is achieved via information projection, where the policy is updated toward the Boltzmann distribution defined by the expectation of the distributional soft state\-action functionZhπZ\_\{h\}^\{\\pi\},

πk\+1\(⋅∣st\)=argminπ\(⋅∣st\)dk​l\(π\(⋅\|st\)\|\|exp⁡\(1α​𝔼​\[Zhπk​\(st,⋅\)\]\)Wπk​\(st\)\)\.\\pi\_\{k\+1\}\(\\cdot\\mid s\_\{t\}\)=\\operatorname\*\{argmin\}\_\{\\pi\(\\cdot\\mid s\_\{t\}\)\}d\_\{kl\}\\left\(\\pi\(\\cdot\\operatorname\{\\,\|\\,\}s\_\{t\}\)\\ \{\|\|\}\\ \\frac\{\\exp\(\\frac\{1\}\{\\alpha\}\\mathbb\{E\}\[Z\_\{h\}^\{\\pi\_\{k\}\}\(s\_\{t\},\\cdot\)\]\)\}\{W^\{\\pi\_\{k\}\}\(s\_\{t\}\)\}\\right\)\.\(20\)The proof follows analogously to the soft policy improvement established in Lemmas[B\.2](https://arxiv.org/html/2605.08104#A2.Thmtheorem2)and[B\.3](https://arxiv.org/html/2605.08104#A2.Thmtheorem3), with the distinction that the update is performed with respect to the expectation over the distribution\.

#### Policy Iteration

The proof parallels the classical reinforcement learning argument, adjusting for the stochastic policy update via an expectation over the action distribution\.

###### Theorem 4\.3\.

Let the reward be bounded,\|𝒜\|<∞\|\\mathcal\{A\}\|<\\infty,\|𝒮\|<∞\|\\mathcal\{S\}\|<\\infty,0≤γ<10\\leq\\gamma<1andα\>0\\alpha\>0\. LetΠ\\Pibe a set of all stationary, stochastic policies\. Alternating between exact distributional soft policy evaluation and global soft policy improvement from some initial policyπ0∈Π\\pi\_\{0\}\\in\\Piand soft state\-action function𝔼​\[Zh\(0\)\]\\mathbb\{E\}\[Z\_\{h\}^\{\(0\)\}\], the process converges \(in the limit\) to the optimal policyπ∗\\pi^\{\*\}, satisfying𝔼​\[Zhπ∗\]=𝔼​\[Zh∗\]\\mathbb\{E\}\[Z\_\{h\}^\{\\pi^\{\*\}\}\]=\\mathbb\{E\}\[Z\_\{h\}^\{\*\}\]\.

###### Proof\.

At each iterationkk, the policyπk\\pi\_\{k\}is fixed during evaluation\. By Theorem[4\.1](https://arxiv.org/html/2605.08104#S4.Thmtheorem1)and its Corollary,𝒯Hπk\\mathcal\{T\}\_\{H\}^\{\\pi\_\{k\}\}is aγ\\gamma\-contraction in the energy distance and the evaluation sequence converges to the fixed pointZhπkZ\_\{h\}^\{\\pi\_\{k\}\}\. Therefore,𝔼​\[Zhπk\]\\mathbb\{E\}\[Z\_\{h\}^\{\\pi\_\{k\}\}\]is well defined at each iteration\. By the policy improvement argument, the sequence\{𝔼​\[Zhπk​\(st,at\)\]\}\\\{\\mathbb\{E\}\[Z\_\{h\}^\{\\pi\_\{k\}\}\(s\_\{t\},a\_\{t\}\)\]\\\}is non\-decreasing for each pair\(st,at\)\(s\_\{t\},a\_\{t\}\)\. Because the reward is bounded and\|𝒮\|<∞\|\\mathcal\{S\}\|<\\infty,𝒜<∞\\mathcal\{A\}<\\infty, the sequence converges to some point\-wise limitL¯\\bar\{L\}\. Policies are updated by thesoftmaxof the current𝔼​\[Zh\]\\mathbb\{E\}\[Z\_\{h\}\]and softmax is continuous, therefore the limit pointπ∗\\pi^\{\*\}exists and is the softmax ofL¯\\bar\{L\}, i\.e\.π∗=softmax​\(L¯\)\\pi^\{\*\}=\\text\{softmax\}\(\\bar\{L\}\)\. Sinceπ∗∈Π\\pi^\{\*\}\\in\\Pi, Theorem[4\.1](https://arxiv.org/html/2605.08104#S4.Thmtheorem1)equally guarantees well\-definedness in the limit point𝔼​\[Zhπ∗\]\\mathbb\{E\}\[Z\_\{h\}^\{\\pi^\{\*\}\}\]\. The mapπ↦𝔼​\[Zhπ\]\\pi\\mapsto\\mathbb\{E\}\[Z\_\{h\}^\{\\pi\}\]is continuous, henceπk→π∗\\pi\_\{k\}\\rightarrow\\pi^\{\*\}implies𝔼​\[Zhπk\]→𝔼​\[Zhπ∗\]\\mathbb\{E\}\[Z\_\{h\}^\{\\pi\_\{k\}\}\]\\rightarrow\\mathbb\{E\}\[Z\_\{h\}^\{\\pi^\{\*\}\}\]and thereforeL¯=𝔼​\[Zhπ∗\]\\bar\{L\}=\\mathbb\{E\}\[Z\_\{h\}^\{\\pi^\{\*\}\}\]\. It followsπ∗=softmax​\(𝔼​\[Zhπ∗\]\)\\pi^\{\*\}=\\text\{softmax\}\(\\mathbb\{E\}\[Z\_\{h\}^\{\\pi^\{\*\}\}\]\)and therefore𝔼​\[Zhπ∗\]\\mathbb\{E\}\[Z\_\{h\}^\{\\pi^\{\*\}\}\]satisfies the Bellman optimality equation, thus𝔼​\[Zhπ∗\]=𝔼​\[Zh∗\]\\mathbb\{E\}\[Z\_\{h\}^\{\\pi^\{\*\}\}\]=\\mathbb\{E\}\[Z\_\{h\}^\{\*\}\]\. ∎

The theorems above establish the theoretical foundation for our Cramér\-based Distributional Soft Actor\-Critic \(C\-DSAC\) approach\. In Section[5](https://arxiv.org/html/2605.08104#S5)we use these insights to construct the C\-DSAC algorithm\.

## 5Implementation and Analysis

In this section, the theoretical results in the previous sections are utilized to derive parameterized objective functions suitable for implementation\. To handle the large state and action spaces in complex tasks, neural networks are used as parameterized function approximators; the aim is to optimize their parameters w\.r\.t the objective\. The parameters for the random variable and policy will be denoted byθ\\thetaandϕ\\phi, respectively\. C\-DSAC is implemented as a fixed\-moment algorithm, i\.e\. the underlying distribution is fully characterized by its first two moments\. Each return distribution is obtained by shifting and rescaling a single standardized density\. It is assumed that the random variable obeys a Gaussian distribution with expectationQθ1Q\_\{\\theta\_\{1\}\}and standard deviationσθ2\\sigma\_\{\\theta\_\{2\}\}\. By constraining the return distribution to this simplified representation, fixed\-moment distributional reinforcement learning algorithms operate under strong assumptions about the underlying true distributions of returns\. The fixed\-moment approach offers computational simplicity, but it comes at the potential cost of accuracy\. The rigid assumptions can \(and generally will\) lead to biased approximations of the true return distribution\. However, this error introduces an intriguing tradeoff between variance and bias in learning\. By reducing the complexity of the distribution \- requiring fewer parameters for approximation \- fixed\-moment algorithms can yield lower gradient variance during optimization\. The reduction in variance can enhance learning stability and efficiency, making these algorithms appealing in scenarios where computational resources are limited\.

To understand the factors contributing to the state\-of\-the\-art performance of C\-DSAC, we analyze the impact of the energy distance loss on its training dynamics\. As discussed in the following sections, C\-DSAC has an inherent mechanism to counteract overestimation bias\.

### 5\.1Objective Functions and Algorithm

Formulas are derived for implementation with parameterized function approximators, utilizing neural networks for their high generalization capabilities\. Previously observed states and actions are stored in a replay bufferℬ\\mathcal\{B\}\. FollowingFujimotoet al\.\([2018](https://arxiv.org/html/2605.08104#bib.bib96)\), a critic target network with parametersθ¯\\bar\{\\theta\}is employed for the critic to further reduce approximation errors\.

#### Evaluation

The neural network parameters ofZθZ\_\{\\theta\}are optimized by minimizing the energy distance relative to the soft distributional Bellman target

JZ​\(θ\):=𝔼st∼dμγat∼μ\(⋅∣st\)st\+1∼P\(⋅∣st,at\)​\[de​\(𝒯Hπ​Z​\(st,at\),Zθ​\(st,at\)\)\],J\_\{Z\}\(\\theta\):=\\mathbb\{E\}\_\{\\begin\{subarray\}\{c\}s\_\{t\}\\sim d\_\{\\mu\}^\{\\gamma\}\\\\ a\_\{t\}\\sim\\mu\(\\cdot\\mid s\_\{t\}\)\\\\ s\_\{t\+1\}\\sim P\(\\cdot\\mid s\_\{t\},a\_\{t\}\)\\end\{subarray\}\}\[d\_\{e\}\(\\mathcal\{T\}\_\{H\}^\{\\pi\}Z\(s\_\{t\},a\_\{t\}\),Z\_\{\\theta\}\(s\_\{t\},a\_\{t\}\)\)\],\(21\)wheredμγ=1−γ1−γT​∑t=0T−1γt​P​r​\(st=s∣μ\)d\_\{\\mu\}^\{\\gamma\}=\\frac\{1\-\\gamma\}\{1\-\\gamma^\{T\}\}\\sum\_\{t=0\}^\{T\-1\}\{\\gamma^\{t\}Pr\(s\_\{t\}=s\\mid\\mu\)\}is the state\-visitation distribution induced by the stochastic behavior policyμ\\mu\.

In practice, gradients are estimated using a mini\-batchℬ^\\hat\{\\mathcal\{B\}\}sampled uniformly from the replay buffer

∇^θ​JZ​\(θ\)=1\|ℬ^\|​∑\(st,at,rt,st\+1\)∈ℬ^\[∇θde​\(𝒯^Hπ​Zθ¯​\(st,at\),Zθ​\(st,at\)\)\],\\hat\{\\nabla\}\_\{\\theta\}J\_\{Z\}\(\\theta\)=\\frac\{1\}\{\|\\hat\{\\mathcal\{B\}\}\|\}\\sum\_\{\(s\_\{t\},a\_\{t\},r\_\{t\},s\_\{t\+1\}\)\\in\\hat\{\\mathcal\{B\}\}\}\[\\nabla\_\{\\theta\}d\_\{e\}\(\\hat\{\\mathcal\{T\}\}\_\{H\}^\{\\pi\}Z\_\{\\bar\{\\theta\}\}\(s\_\{t\},a\_\{t\}\),Z\_\{\\theta\}\(s\_\{t\},a\_\{t\}\)\)\],\(22\)where the empirical soft distributional Bellman operator for a fixedst\+1s\_\{t\+1\}is defined as

𝒯^HπZθ¯\(st,at\):=r\(st,at\)\+γ\(Zθ¯\(st\+1,at\+1\)−αlogπ\(at\+1∣st\+1\)\)∣at\+1∼π\(⋅∣st\+1\)\\hat\{\\mathcal\{T\}\}\_\{H\}^\{\\pi\}Z\_\{\\bar\{\\theta\}\}\(s\_\{t\},a\_\{t\}\):=r\(s\_\{t\},a\_\{t\}\)\+\\gamma\(Z\_\{\\overline\{\\theta\}\}\(s\_\{t\+1\},a\_\{t\+1\}\)\-\\alpha\\log\{\\pi\(a\_\{t\+1\}\\mid s\_\{t\+1\}\)\}\)\\mid a\_\{t\+1\}\\sim\\pi\(\\cdot\\mid s\_\{t\+1\}\)andθ¯\\bar\{\\theta\}are frozen parameters \(for clarity,θ¯\\bar\{\\theta\}will be omitted in the subsequent sections\)\. The target network parameters are not updated via gradient descent; instead, it tracks the online network through Polyak averaging\.

The probability density function is Gaussian denoted by

φQ,σ​\(x\)=1σ​2​π​e−12​\(\(x−Q\)σ\)2\\varphi\_\{Q,\\sigma\}\(x\)=\\frac\{1\}\{\\sigma\\sqrt\{2\\pi\}\}e^\{\-\\frac\{1\}\{2\}\\left\(\\frac\{\(x\-Q\)\}\{\\sigma\}\\right\)^\{2\}\}regarding a mean valueQ∈\[Q¯,Q¯\]⊂ℝQ\\in\[\\underline\{Q\},\\overline\{Q\}\]\\subset\\mathbb\{R\}and a varianceσ∈\[σ¯,σ¯\]⊂ℝ\+\\sigma\\in\[\\underline\{\\sigma\},\\overline\{\\sigma\}\]\\subset\\mathbb\{R\}\_\{\+\}\. For terminal states, the standard deviation of the target is set to0\. The cumulative distribution function ofφQ,σ\\varphi\_\{Q,\\sigma\}is denoted by

FQ,σ​\(x\)=∫−∞xφQ,σ​\(t\)​𝑑t\.F\_\{Q,\\sigma\}\(x\)=\\int\_\{\-\\infty\}^\{x\}\\varphi\_\{Q,\\sigma\}\(t\)dt\.LetQθ1​\(st,at\)Q\_\{\\theta\_\{1\}\}\(s\_\{t\},a\_\{t\}\)andσθ2​\(st,at\)\\sigma\_\{\\theta\_\{2\}\}\(s\_\{t\},a\_\{t\}\)be neural network models ofQQandσ\\sigmaover𝒮×𝒜\{\\cal S\}\\times\{\\cal A\}respectively\. ThenZθ​\(st,at\)Z\_\{\\theta\}\(s\_\{t\},a\_\{t\}\)denotes a parameterized Gaussian value distribution with parametersθ=\(θ1,θ2\)\\theta=\(\\theta\_\{1\},\\theta\_\{2\}\), defined by the distribution function

fZθ​\(st,at\)=φQθ1​\(st,at\),σθ2​\(st,at\)\.f\_\{Z\_\{\\theta\}\(s\_\{t\},a\_\{t\}\)\}=\\varphi\_\{Q\_\{\\theta\_\{1\}\}\(s\_\{t\},a\_\{t\}\),\\sigma\_\{\\theta\_\{2\}\}\(s\_\{t\},a\_\{t\}\)\}\.

#### Improvement

The policy improvement follows from the information projection equation \([20](https://arxiv.org/html/2605.08104#S4.E20)\)\. Similar toHaarnojaet al\.\([2018](https://arxiv.org/html/2605.08104#bib.bib101)\)andKingma and Welling \([2022](https://arxiv.org/html/2605.08104#bib.bib143)\), a reparameterized policy is used to obtain the actions,at=gϕ​\(st;ξt\)a\_\{t\}=g\_\{\\phi\}\(s\_\{t\};\\xi\_\{t\}\), whereξt∼𝒩​\(0,1\)\\xi\_\{t\}\\sim\\mathcal\{N\}\(0,1\)and𝒩​\(0,1\)\\mathcal\{N\}\(0,1\)is the standard normal distribution\. The policyπϕ\(⋅∣st\)\\pi\_\{\\phi\}\(\\cdot\\mid s\_\{t\}\)is defined as the pushforward ofgϕ​\(st;⋅\)g\_\{\\phi\}\(s\_\{t\};\\cdot\)under the law ofξt\\xi\_\{t\}\. The policy loss can then be defined as

Jπ​\(ϕ\):=𝔼st∼dμγ,ξt∼𝒩​\(0,1\)at=gϕ​\(st;ξt\)​\[α​log⁡πϕ​\(at∣st\)−𝔼​\[Zθ​\(st,at\)\]\],J\_\{\\pi\}\(\\phi\):=\\mathbb\{E\}\_\{\\begin\{subarray\}\{c\}s\_\{t\}\\sim d\_\{\\mu\}^\{\\gamma\},\\\\ \\xi\_\{t\}\\sim\\mathcal\{N\}\(0,1\)\\\\ a\_\{t\}=g\_\{\\phi\}\(s\_\{t\};\\xi\_\{t\}\)\\end\{subarray\}\}\[\\alpha\\log\{\\pi\_\{\\phi\}\(a\_\{t\}\\mid s\_\{t\}\)\}\-\\mathbb\{E\}\[Z\_\{\\theta\}\(s\_\{t\},a\_\{t\}\)\]\],\(23\)wheredμγ=1−γ1−γT​∑t=0T−1γt​P​r​\(st=s∣μ\)d\_\{\\mu\}^\{\\gamma\}=\\frac\{1\-\\gamma\}\{1\-\\gamma^\{T\}\}\\sum\_\{t=0\}^\{T\-1\}\{\\gamma^\{t\}Pr\(s\_\{t\}=s\\mid\\mu\)\}is the state\-visitation distribution induced by the stochastic behavior policyμ\\mu\. The gradient of the empirical loss regarding a mini\-batchℬ^⊂ℬ\\hat\{\\mathcal\{B\}\}\\subset\\mathcal\{B\}is thus

∇^ϕ​Jπ​\(ϕ\)=\\displaystyle\\hat\{\\nabla\}\_\{\\phi\}J\_\{\\pi\}\(\\phi\)=1\|ℬ\|^​∑st∈ℬ^,ξ∼𝒩​\(0,1\)α​∂ϕlog⁡πϕ​\(at​\|⁡st\)\|at\+\\displaystyle\\frac\{1\}\{\|\\hat\{\\mathcal\{B\}\|\}\}\\sum\_\{\\begin\{subarray\}\{c\}s\_\{t\}\\in\\hat\{\\mathcal\{B\}\},\\\\ \\xi\\sim\\mathcal\{N\}\(0,1\)\\end\{subarray\}\}\\alpha\\partial\_\{\\phi\}\\log\{\\pi\_\{\\phi\}\(a\_\{t\}\\operatorname\{\\,\|\\,\}s\_\{t\}\)\}\|\_\{a\_\{t\}\}\+\(24\)\(α​∇atlog⁡πϕ​\(at​\|⁡st\)−∇at𝔼​\[Zθ​\(st,at\)\]\)​∇ϕgϕ​\(st;ξ\)\.\\displaystyle\(\\alpha\\nabla\_\{a\_\{t\}\}\\log\{\\pi\_\{\\phi\}\(a\_\{t\}\\operatorname\{\\,\|\\,\}s\_\{t\}\)\}\-\\nabla\_\{a\_\{t\}\}\\mathbb\{E\}\[Z\_\{\\theta\}\(s\_\{t\},a\_\{t\}\)\]\)\\nabla\_\{\\phi\}g\_\{\\phi\}\(s\_\{t\};\\xi\)\.Similar to the distributional value function, the action distribution is modeled as a Gaussian\. Additionally, the law ofξ\\xifor each action component is set to be a standard Gauss distribution\.

#### Algorithm

The C\-DSAC algorithm is presented in Algorithm[1](https://arxiv.org/html/2605.08104#alg1), with the Cramér gradient update described in line[13](https://arxiv.org/html/2605.08104#alg1.l13)\.

Algorithm 1C\-DSAC \(Cramér\-based Distributional Soft Actor\-Critic\)1:Initialize parameters

θ\\theta,

θ¯\\bar\{\\theta\},

ϕ\\phi
2:Initialize learning rates

βZ\\beta\_\{Z\},

βπ\\beta\_\{\\pi\}
3:

ℬ←∅\\mathcal\{B\}\\leftarrow\\emptyset
4:repeat

5:Receive initial observation state

s0s\_\{0\}
6:foreachenvironment stepdo

7:Select action

at∼πϕ\(⋅\|st\)a\_\{t\}\\sim\\pi\_\{\\phi\}\(\\cdot\\operatorname\{\\,\|\\,\}s\_\{t\}\)
8:Observe transition

st\+1∼P\(⋅∣st,at\)s\_\{t\+1\}\\sim P\(\\cdot\\mid s\_\{t\},a\_\{t\}\)
9:Observe reward

rtr\_\{t\}
10:

ℬ←ℬ∪\{\(st,at,rt,st\+1\)\}\\mathcal\{B\}\\leftarrow\\mathcal\{B\}\\cup\\\{\(s\_\{t\},a\_\{t\},r\_\{t\},s\_\{t\+1\}\)\\\}
11:foreachgradient stepdo

12:Uniformly sample a mini\-batch of

NNtransitions

\(si,ai,ri,si\+1\)\(s\_\{i\},a\_\{i\},r\_\{i\},s\_\{i\+1\}\)
13:

θ←θ\+βZ​∇^θ​JZ​\(θ\)\\theta\\leftarrow\\theta\+\\beta\_\{Z\}\\hat\{\\nabla\}\_\{\\theta\}J\_\{Z\}\(\\theta\), \(Eq\. \([22](https://arxiv.org/html/2605.08104#S5.E22)\)\)

14:

ϕ←ϕ\+βπ​∇^ϕ​Jπ​\(ϕ\)\\phi\\leftarrow\\phi\+\\beta\_\{\\pi\}\\hat\{\\nabla\}\_\{\\phi\}J\_\{\\pi\}\(\\phi\), \(Eq\. \([24](https://arxiv.org/html/2605.08104#S5.E24)\)\)

15:Update target network:

θ¯←τ​θ\+\(1−τ\)​θ¯\\bar\{\\theta\}\\leftarrow\\tau\\theta\+\(1\-\\tau\)\\bar\{\\theta\}
16:untilStopping criterion

17:return

Zθ,πϕZ\_\{\\theta\},\\pi\_\{\\phi\}

### 5\.2Confidence\-Driven Value Update

Fix a target distributionφQ′,σ′\\varphi\_\{Q^\{\\prime\},\\sigma^\{\\prime\}\}, whereQ′Q^\{\\prime\}andσ′\\sigma^\{\\prime\}represent the Gaussian parameters\. The cost functionC​\(Q,σ\)C\(Q,\\sigma\)measures the energy distance between a candidate distribution and this target

C​\(Q,σ\):=de​\(φQ,σ,φQ′,σ′\)=∫−∞∞\(FQ,σ​\(x\)−FQ′,σ′​\(x\)\)2​𝑑x\.C\(Q,\\sigma\):=d\_\{e\}\(\\varphi\_\{Q,\\sigma\},\\varphi\_\{Q^\{\\prime\},\\sigma^\{\\prime\}\}\)=\\int\_\{\-\\infty\}^\{\\infty\}\{\\left\(F\_\{Q,\\sigma\}\(x\)\-F\_\{Q^\{\\prime\},\\sigma^\{\\prime\}\}\(x\)\\right\)^\{2\}\}dx\.
###### Lemma 5\.1\.

LetφQ,σ\\varphi\_\{Q,\\sigma\}andφQ′,σ′\\varphi\_\{Q^\{\\prime\},\\sigma^\{\\prime\}\}be a current and a target distribution, respectively\. Then

∂∂Q​C​\(Q,σ\)=−2σ​B​\(Q,σ\),\\frac\{\\partial\}\{\\partial Q\}C\(Q,\\sigma\)=\-\\frac\{2\}\{\\sigma\}B\(Q,\\sigma\),\(25\)where

B​\(Q,σ\):=∫−∞∞\(FQ,σ​\(x\)−FQ′,σ′​\(x\)\)​φQ,σ​\(x\)​𝑑x\.B\(Q,\\sigma\):=\\int\_\{\-\\infty\}^\{\\infty\}\{\(F\_\{Q,\\sigma\}\(x\)\-F\_\{Q^\{\\prime\},\\sigma^\{\\prime\}\}\(x\)\)\\varphi\_\{Q,\\sigma\}\(x\)dx\}\.\(26\)

###### Proof\.

Letf​\(x\)=φ\(0,1\)f\(x\)=\\varphi\_\{\(0,1\)\}be the standard normal density function andΦ=F0,1\\Phi=F\_\{0,1\}the standard normal cumulative distribution function\. Applying the chain rule yields

∂∂Q​FQ,σ=∂∂Q​Φ​\(x−Qσ\)=−1σ​f​\(x−Qσ\)=−φQ,σσ\.\\frac\{\\partial\}\{\\partial Q\}F\_\{Q,\\sigma\}=\\frac\{\\partial\}\{\\partial Q\}\\Phi\\left\(\\frac\{x\-Q\}\{\\sigma\}\\right\)=\-\\frac\{1\}\{\\sigma\}f\\left\(\\frac\{x\-Q\}\{\\sigma\}\\right\)=\-\\frac\{\\varphi\_\{Q,\\sigma\}\}\{\\sigma\}\.Hence,

∂∂Q​C​\(Q,σ\)\\displaystyle\\frac\{\\partial\}\{\\partial Q\}C\(Q,\\sigma\)=\\displaystyle=∫−∞∞2​\(FQ,σ​\(x\)−FQ′,σ′​\(x\)\)​∂∂Q​FQ,σ​\(x\)​𝑑x\\displaystyle\\int\_\{\-\\infty\}^\{\\infty\}\{2\(F\_\{Q,\\sigma\}\(x\)\-F\_\{Q^\{\\prime\},\\sigma^\{\\prime\}\}\(x\)\)\\frac\{\\partial\}\{\\partial Q\}F\_\{Q,\\sigma\}\(x\)dx\}=\\displaystyle=∫−∞∞−2​\(FQ,σ​\(x\)−FQ′,σ′​\(x\)\)​φQ,σ​\(x\)σ​d​x\\displaystyle\\int\_\{\-\\infty\}^\{\\infty\}\{\-2\(F\_\{Q,\\sigma\}\(x\)\-F\_\{Q^\{\\prime\},\\sigma^\{\\prime\}\}\(x\)\)\\frac\{\\varphi\_\{Q,\\sigma\}\(x\)\}\{\\sigma\}dx\}=\\displaystyle=−2σ​∫−∞∞\(FQ,σ​\(x\)−FQ′,σ′​\(x\)\)​φQ,σ​\(x\)​𝑑x\.\\displaystyle\-\\frac\{2\}\{\\sigma\}\\int\_\{\-\\infty\}^\{\\infty\}\{\(F\_\{Q,\\sigma\}\(x\)\-F\_\{Q^\{\\prime\},\\sigma^\{\\prime\}\}\(x\)\)\\varphi\_\{Q,\\sigma\}\(x\)dx\}\.∎

![Refer to caption](https://arxiv.org/html/2605.08104v1/Images/VaryingCurrStd_dCdQ.png)Figure 1:Value of∂∂Q​C​\(Q,σ\)\\frac\{\\partial\}\{\\partial Q\}C\(Q,\\sigma\)for varyingσ∈\[σ¯,σ¯\]\\sigma\\in\[\\underline\{\\sigma\},\\overline\{\\sigma\}\]The following Lemma and Proposition establish the core result of the analysis\.

###### Lemma 5\.2\.

LetφQ,σ\\varphi\_\{Q,\\sigma\}andφQ′,σ′\\varphi\_\{Q^\{\\prime\},\\sigma^\{\\prime\}\}be a current and a target distribution, respectively\. Then

limσ→∞∂∂Q​C​\(Q,σ\)=0\.\\lim\_\{\\sigma\\to\\infty\}\\frac\{\\partial\}\{\\partial Q\}C\(Q,\\sigma\)=0\.

###### Proof\.

SinceFQ,σ​\(x\)∈\[0,1\]F\_\{Q,\\sigma\}\(x\)\\in\[0,1\]and‖φQ,σ‖1=1\\\|\\varphi\_\{Q,\\sigma\}\\\|\_\{1\}=1, it holds from the Hölder inequality

\|B​\(Q,σ\)\|≤‖\(FQ,σ−FQ′,σ′\)⋅φQ,σ‖1≤‖FQ,σ−FQ′,σ′‖∞⋅‖φQ,σ‖1≤1\.\|B\(Q,\\sigma\)\|\\leq\\\|\(F\_\{Q,\\sigma\}\-F\_\{Q^\{\\prime\},\\sigma^\{\\prime\}\}\)\\cdot\\varphi\_\{Q,\\sigma\}\\\|\_\{1\}\\leq\\\|F\_\{Q,\\sigma\}\-F\_\{Q^\{\\prime\},\\sigma^\{\\prime\}\}\\\|\_\{\\infty\}\\cdot\\\|\\varphi\_\{Q,\\sigma\}\\\|\_\{1\}\\leq 1\.This gives

limσ→∞\|∂∂Q​C​\(Q,σ\)\|=limσ→∞2σ⋅\|B​\(Q,σ\)\|≤limσ→∞2σ=0\.\\lim\_\{\\sigma\\to\\infty\}\\left\|\\frac\{\\partial\}\{\\partial Q\}C\(Q,\\sigma\)\\right\|=\\lim\_\{\\sigma\\to\\infty\}\\frac\{2\}\{\\sigma\}\\cdot\|B\(Q,\\sigma\)\|\\leq\\lim\_\{\\sigma\\to\\infty\}\\frac\{2\}\{\\sigma\}=0\.∎

###### Proposition 5\.3\.

Denote byΨθ​\(st,at\):=∂∂Qθ1​\(st,at\)​de​\(𝒯^Hπ​Z​\(st,at\),Zθ​\(st,at\)\)\\Psi\_\{\\theta\}\(s\_\{t\},a\_\{t\}\):=\\frac\{\\partial\}\{\\partial Q\_\{\\theta\_\{1\}\}\(s\_\{t\},a\_\{t\}\)\}d\_\{e\}\(\\hat\{\\mathcal\{T\}\}^\{\\pi\}\_\{H\}Z\(s\_\{t\},a\_\{t\}\),Z\_\{\\theta\}\(s\_\{t\},a\_\{t\}\)\)a gradient weight of Eq\. \([22](https://arxiv.org/html/2605.08104#S5.E22)\) such that ∇^θ1​JZ​\(θ\)=1\|ℬ^\|​∑ℬ^\[Ψθ​\(st,at\)​∇θ1Qθ1​\(st,at\)\]\\hat\{\\nabla\}\_\{\\theta\_\{1\}\}J\_\{Z\}\(\\theta\)=\\frac\{1\}\{\|\\hat\{\\mathcal\{B\}\}\|\}\\sum\_\{\\hat\{\\mathcal\{B\}\}\}\[\\Psi\_\{\\theta\}\(s\_\{t\},a\_\{t\}\)\\nabla\_\{\\theta\_\{1\}\}Q\_\{\\theta\_\{1\}\}\(s\_\{t\},a\_\{t\}\)\]\. Then

limσθ2​\(st,at\)→∞Ψθ​\(st,at\)=0\.\\lim\_\{\\sigma\_\{\\theta\_\{2\}\}\(s\_\{t\},a\_\{t\}\)\\to\\infty\}\\Psi\_\{\\theta\}\(s\_\{t\},a\_\{t\}\)=0\.\(27\)

###### Proof\.

From Lemma[5\.1](https://arxiv.org/html/2605.08104#S5.Thmtheorem1)it follows

Ψθ​\(st,at\)=∂∂Qθ1​\(st,at\)​de​\(𝒯^Hπ​Z​\(st,at\),Zθ​\(st,at\)\)=−2​B​\(Qθ1​\(st,at\),σθ2​\(st,at\)\)σθ2​\(st,at\)\.\\Psi\_\{\\theta\}\(s\_\{t\},a\_\{t\}\)=\\frac\{\\partial\}\{\\partial Q\_\{\\theta\_\{1\}\}\(s\_\{t\},a\_\{t\}\)\}d\_\{e\}\(\\hat\{\\mathcal\{T\}\}\_\{H\}^\{\\pi\}Z\(s\_\{t\},a\_\{t\}\),Z\_\{\\theta\}\(s\_\{t\},a\_\{t\}\)\)=\-\\frac\{2B\(Q\_\{\\theta\_\{1\}\}\(s\_\{t\},a\_\{t\}\),\\sigma\_\{\\theta\_\{2\}\}\(s\_\{t\},a\_\{t\}\)\)\}\{\\sigma\_\{\\theta\_\{2\}\}\(s\_\{t\},a\_\{t\}\)\}\.The statement follows from Lemma[5\.2](https://arxiv.org/html/2605.08104#S5.Thmtheorem2)\. ∎

Figure[1](https://arxiv.org/html/2605.08104#S5.F1)shows that the gradient weightΨθ​\(st,at\)\\Psi\_\{\\theta\}\(s\_\{t\},a\_\{t\}\)is quickly reducing, if the varianceσθ2​\(st,at\)\\sigma\_\{\\theta\_\{2\}\}\(s\_\{t\},a\_\{t\}\)is increased\. The analysis reveals that for a particular pair\(st,at\)\(s\_\{t\},a\_\{t\}\), the magnitude of theQθ1Q\_\{\\theta\_\{1\}\}update is reversely scaled by the varianceσθ2\\sigma\_\{\\theta\_\{2\}\}, such that the update vanishes asσθ2→∞\\sigma\_\{\\theta\_\{2\}\}\\rightarrow\\infty\. Hence,Qθ1​\(st,at\)Q\_\{\\theta\_\{1\}\}\(s\_\{t\},a\_\{t\}\)is updated more aggressively at state\-action pairs\(st,at\)\(s\_\{t\},a\_\{t\}\)whereσθ2​\(st,at\)\\sigma\_\{\\theta\_\{2\}\}\(s\_\{t\},a\_\{t\}\)is small, reflecting temporal stability of the target\. This update type is therefore calledconfidence\-driven\.

### 5\.3Effect on Overestimation Bias

A pervasive challenge inQQ\-based reinforcement learning is the overestimation bias that arises from function approximation in noisy environments\(Thrun and Schwartz,[1993](https://arxiv.org/html/2605.08104#bib.bib124)\)\. In this section, it is shown that the influence of the overestimation error at\(st,at\)\(s\_\{t\},a\_\{t\}\)on the gradient of the energy loss function \([22](https://arxiv.org/html/2605.08104#S5.E22)\) is small, if the varianceσθ2​\(st,at\)\\sigma\_\{\\theta\_\{2\}\}\(s\_\{t\},a\_\{t\}\)ofZθ​\(st,at\)Z\_\{\\theta\}\(s\_\{t\},a\_\{t\}\)is large\. Denote byQ~\\tilde\{Q\}andQ∗Q^\{\*\}a noisy and an exact mean \(Q\-value\), respectively\. Consider a noisy target

Y~​\(st,at\):=r​\(st,at\)\+γ​Z​\(st\+1,at\+1\),at\+1=argmaxa∈𝒜Q~​\(st\+1,a\)\\tilde\{Y\}\(s\_\{t\},a\_\{t\}\):=r\(s\_\{t\},a\_\{t\}\)\+\\gamma Z\(s\_\{t\+1\},a\_\{t\+1\}\),\\quad a\_\{t\+1\}=\\operatorname\*\{argmax\}\_\{a\\in\{\\cal A\}\}\\tilde\{Q\}\(s\_\{t\+1\},a\)and an exact target

Y∗​\(st,at\):=r​\(st,at\)\+γ​Z​\(st\+1,at\+1\),at\+1=argmaxa∈𝒜Q∗​\(st\+1,a\)\.Y^\{\*\}\(s\_\{t\},a\_\{t\}\):=r\(s\_\{t\},a\_\{t\}\)\+\\gamma Z\(s\_\{t\+1\},a\_\{t\+1\}\),\\quad a\_\{t\+1\}=\\operatorname\*\{argmax\}\_\{a\\in\{\\cal A\}\}Q^\{\*\}\(s\_\{t\+1\},a\)\.
###### Proposition 5\.4\.

LetΔ​Ψθ​\(st,at\):=\|Ψ~​\(st,at\)−Ψ∗​\(st,at\)\|\\Delta\\Psi\_\{\\theta\}\(s\_\{t\},a\_\{t\}\):=\|\\tilde\{\\Psi\}\(s\_\{t\},a\_\{t\}\)\-\\Psi^\{\*\}\(s\_\{t\},a\_\{t\}\)\|be the gradient weight error of the energy distance \([22](https://arxiv.org/html/2605.08104#S5.E22)\) regarding noisy and exact gradient weights, where

Ψ~​\(st,at\):=∂∂Qθ1​dc​\(Zθ​\(st,at\),Y~​\(st,at\)\),Ψ∗​\(st,at\):=∂∂Qθ1​dc​\(Zθ​\(st,at\),Y∗​\(st,at\)\)\.\\tilde\{\\Psi\}\(s\_\{t\},a\_\{t\}\):=\\frac\{\\partial\}\{\\partial Q\_\{\\theta\_\{1\}\}\}d\_\{c\}\(Z\_\{\\theta\}\(s\_\{t\},a\_\{t\}\),\\tilde\{Y\}\(s\_\{t\},a\_\{t\}\)\),\\quad\\Psi^\{\*\}\(s\_\{t\},a\_\{t\}\):=\\frac\{\\partial\}\{\\partial Q\_\{\\theta\_\{1\}\}\}d\_\{c\}\(Z\_\{\\theta\}\(s\_\{t\},a\_\{t\}\),Y^\{\*\}\(s\_\{t\},a\_\{t\}\)\)\.Then

limσθ2​\(st,at\)→∞Δ​Ψθ​\(st,at\)=0\.\\lim\_\{\\sigma\_\{\\theta\_\{2\}\}\(s\_\{t\},a\_\{t\}\)\\to\\infty\}\\Delta\\Psi\_\{\\theta\}\(s\_\{t\},a\_\{t\}\)=0\.\(28\)

###### Proof\.

DefineB∗B^\{\*\}andB~\\tilde\{B\}via \([26](https://arxiv.org/html/2605.08104#S5.E26)\) usingY∗Y^\{\*\}andY~\\tilde\{Y\}, respectively\. \. From Lemma[5\.1](https://arxiv.org/html/2605.08104#S5.Thmtheorem1)it follows

\|Ψ~​\(st,at\)−Ψ∗​\(st,at\)\|=2​\|B~θ​\(st,at\)σθ2​\(st,at\)−Bθ∗​\(st,at\)σθ2​\(st,at\)\|≤2σθ2​\(st,at\),\\left\|\\tilde\{\\Psi\}\(s\_\{t\},a\_\{t\}\)\-\\Psi^\{\*\}\(s\_\{t\},a\_\{t\}\)\\right\|=2\\left\|\\frac\{\\tilde\{B\}\_\{\\theta\}\(s\_\{t\},a\_\{t\}\)\}\{\\sigma\_\{\\theta\_\{2\}\}\(s\_\{t\},a\_\{t\}\)\}\-\\frac\{B^\{\*\}\_\{\\theta\}\(s\_\{t\},a\_\{t\}\)\}\{\\sigma\_\{\\theta\_\{2\}\}\(s\_\{t\},a\_\{t\}\)\}\\right\|\\leq\\frac\{2\}\{\\sigma\_\{\\theta\_\{2\}\}\(s\_\{t\},a\_\{t\}\)\},since\|Bθ∗​\(st,at\)\|≤1\|B^\{\*\}\_\{\\theta\}\(s\_\{t\},a\_\{t\}\)\|\\leq 1and\|B~θ​\(st,at\)\|≤1\|\\tilde\{B\}\_\{\\theta\}\(s\_\{t\},a\_\{t\}\)\|\\leq 1\. This proves \([28](https://arxiv.org/html/2605.08104#S5.E28)\)\. ∎

Figure[2](https://arxiv.org/html/2605.08104#S5.F2)shows that the gradient weight errorΔ​Ψθ​\(st,at\)\\Delta\\Psi\_\{\\theta\}\(s\_\{t\},a\_\{t\}\)is quickly reducing if the varianceσθ2​\(st,at\)\\sigma\_\{\\theta\_\{2\}\}\(s\_\{t\},a\_\{t\}\)is increased\. The variance is elevated at state\-action pairs exhibiting high return stochasticity, coinciding with the regions most susceptible to value overestimationLanet al\.\([2020](https://arxiv.org/html/2605.08104#bib.bib107)\)\.

![Refer to caption](https://arxiv.org/html/2605.08104v1/Images/VaryingCurrStd_Delta.png)Figure 2:Value ofΔ​Ψ\\Delta\\Psifor varyingσ∈\[σ¯,σ¯\]\\sigma\\in\[\\underline\{\\sigma\},\\overline\{\\sigma\}\]

## 6Numerical Experiments

The experiments are conducted with the same hyperparameter values as inHaarnojaet al\.\([2018](https://arxiv.org/html/2605.08104#bib.bib101)\), except for C\-DSAC\-specific ones \(e\.g, limits on standard deviation, … etc\. \), see Table[2](https://arxiv.org/html/2605.08104#A3.T2)\. Implementation\-level nuances and low\-level optimizations exert a substantial influence on an algorithm’s empirical performance\(Engstromet al\.,[2020](https://arxiv.org/html/2605.08104#bib.bib140)\)\. It is assumed that the original implementation used for the baseline results inHaarnojaet al\.\([2018](https://arxiv.org/html/2605.08104#bib.bib101)\)did not employ additional optimization techniques such as value function clipping, orthogonal initialization, layer scaling, learning rate annealing, reward/observation clipping, or observation and reward normalization\. To be consistent with these assumptions, these code\-level optimization techniques is not employed in the C\-DSAC implementation\. Using the same network architecture as the SAC experiments would grant C\-DSAC a disproportionate advantage, owing to its dual\-output critic\. Therefore, the last hidden layer is reduced by one neuron\. Consequently, the C\-DSAC critic has slightly fewer parameters than those reported in the original SAC configuration, see Appendix[C](https://arxiv.org/html/2605.08104#A3)\. AlthoughMaet al\.\([2020](https://arxiv.org/html/2605.08104#bib.bib148)\)andDuanet al\.\([2022](https://arxiv.org/html/2605.08104#bib.bib18)\)claim superior performance over SAC baselines, it is important to note that the former uses more complex neural network architectures with batch normalization for the value approximator\.

### 6\.1Comparative Evaluation

The goal of the experiments is to evaluate the impact of the energy distance on the entropy\-augmented actor\-critic framework across diverse and complex environmentsBrockmanet al\.\([2016](https://arxiv.org/html/2605.08104#bib.bib11)\), see Figure[3](https://arxiv.org/html/2605.08104#S6.F3): Hopper\-v4,\(𝒮×𝒜\)∈ℝ11×ℝ3\(\\mathcal\{S\}\\times\\mathcal\{A\}\)\\in\\mathbb\{R\}^\{11\}\\times\\mathbb\{R\}^\{3\}; Ant\-v4\(𝒮×𝒜\)∈ℝ111×ℝ8\(\\mathcal\{S\}\\times\\mathcal\{A\}\)\\in\\mathbb\{R\}^\{111\}\\times\\mathbb\{R\}^\{8\}; Humanoid\-v4\(𝒮×𝒜\)∈ℝ376×ℝ17\(\\mathcal\{S\}\\times\\mathcal\{A\}\)\\in\\mathbb\{R\}^\{376\}\\times\\mathbb\{R\}^\{17\}; HalfCheetah\-v4\(𝒮×𝒜\)∈ℝ17×ℝ6\(\\mathcal\{S\}\\times\\mathcal\{A\}\)\\in\\mathbb\{R\}^\{17\}\\times\\mathbb\{R\}^\{6\}; Walker2d\-v4\(𝒮×𝒜\)∈ℝ17×ℝ6\(\\mathcal\{S\}\\times\\mathcal\{A\}\)\\in\\mathbb\{R\}^\{17\}\\times\\mathbb\{R\}^\{6\}\. The entropy coefficient is constant for each environment, following the configuration inHaarnojaet al\.\([2018](https://arxiv.org/html/2605.08104#bib.bib101)\)\. Environment\-specific values forα\\alphaare given in Table[3](https://arxiv.org/html/2605.08104#A3.T3)\. Following the recommendations ofHendersonet al\.\([2017](https://arxiv.org/html/2605.08104#bib.bib103)\), evaluation rollouts are performed every10310^\{3\}iterations \. During evaluation, agents follow a deterministic policy derived from the distribution mean\. The agent is trained for one million iterations in each environment\. The results are averaged over five random seeds\. The graphs in Figure[5](https://arxiv.org/html/2605.08104#S6.F5)showcase \(among others\) the evaluation performances of C\-DSAC and SAC in the benchmark environments\. Especially in the most difficult environment, Humanoid\-v4, C\-DSAC clearly outperforms SAC and reaches an average reward of about 5715, a value that is achieved by SAC only after approximately four million iterations\. Notably, C\-DSAC performance has not yet reached a plateau by the conclusion of training, suggesting further potential for improvement with extended computation\. Although C\-DSAC and SAC reach the same average rollout in the Hopper\-v4 environment, C\-DSAC does so considerably faster\. The instability after4×1054\\times 10^\{5\}iterations can be due to the occurrence of catastrophic forgetting in one of the runs\. However, it manages to recover\. In the Walker2d\-v4 environment, C\-DSAC attains higher mean rewards, but the training profile reveals greater stochasticity relative to the baseline\. The least stable training occurs in the Ant\-v4 environment, where high stochasticity can be observed\. Still, C\-DSAC outperforms SAC in Ant\-v4\. In HalfCheetah\-v4, C\-DSAC obtains an average reward of less than10,00010,000in the end of training and is therefore outperformed by SAC in this environment\.

To provide more details about the performance, Table[1](https://arxiv.org/html/2605.08104#S6.T1)presents the best model performances across all environments, reporting average episodic rewards over 100 runs\.

![Refer to caption](https://arxiv.org/html/2605.08104v1/Images/Hopper-v4.png)\(a\)Hopper\-v4
![Refer to caption](https://arxiv.org/html/2605.08104v1/Images/Ant-v4.png)\(b\)Ant\-v4
![Refer to caption](https://arxiv.org/html/2605.08104v1/Images/Humanoid-v4.png)\(c\)Humanoid\-v4
![Refer to caption](https://arxiv.org/html/2605.08104v1/Images/HalfCheetah-v4.png)\(d\)HalfCheetah\-v4
![Refer to caption](https://arxiv.org/html/2605.08104v1/Images/Walker2d-v4.png)\(e\)Walker2d\-v4

Figure 3:Testing environments![Refer to caption](https://arxiv.org/html/2605.08104v1/Images/Hopper-v4_Rollout_Averaged.png)

![Refer to caption](https://arxiv.org/html/2605.08104v1/Images/Hopper_SAC.png)

\(a\)Hopper
![Refer to caption](https://arxiv.org/html/2605.08104v1/Images/Ant-v4_Rollout_Averaged.png)

![Refer to caption](https://arxiv.org/html/2605.08104v1/Images/Ant_SAC.png)

\(b\)Ant \(left one million, right three million steps\)
![Refer to caption](https://arxiv.org/html/2605.08104v1/Images/Humanoid-v4_Rollout_Averaged.png)

![Refer to caption](https://arxiv.org/html/2605.08104v1/Images/Humanoid_SAC.png)

\(c\)Humanoid \(left one million, right ten million steps\)
![Refer to caption](https://arxiv.org/html/2605.08104v1/Images/HalfCheetah-v4_Rollout_Averaged.png)

![Refer to caption](https://arxiv.org/html/2605.08104v1/Images/HalfCheetah_SAC.png)

\(d\)HalfCheetah \(left one million, right three million steps\)

![Refer to caption](https://arxiv.org/html/2605.08104v1/Images/Walker2d-v4_Rollout_Averaged.png)

![Refer to caption](https://arxiv.org/html/2605.08104v1/Images/Walker_SAC.png)

\(a\)WalkerFigure 5:Orange curves represent C\-DSAC \(left\) and SAC \(right, from\(Haarnojaet al\.,[2018](https://arxiv.org/html/2605.08104#bib.bib101)\)\)\. Other curves show performances of DDPG \(green\), PPO \(brown\), SQL \(pink\), and TD3 \(violet\)\.Table 1:Average performance over 100 runs of the best models in the training runs
### 6\.2Comparative Implementation Evaluation against a Standard Baseline

To assess the quality of the C\-DSAC implementation \- and therefore its effect on the performance, its code is modified to revert it to SAC by adjusting the loss function and critic network architecture\. The experiments are repeated on HalfCheetah\-v4 with five runs, Figure[6](https://arxiv.org/html/2605.08104#S6.F6)\. The SAC hyperparameter are set to values given in Appendix[C](https://arxiv.org/html/2605.08104#A3)\. The custom SAC implementation based on the C\-DSAC code achieved an average reward of slightly above 3000, compared to over 10000 in the original SAC implementation after one million iterations\. The observed performance gap potentially stems from unoptimized implementation details, suggesting that C\-DSAC has yet to reach its theoretical performance ceiling\.

![Refer to caption](https://arxiv.org/html/2605.08104v1/Images/HalfCheetah-v4_SACBase_Rollout_Averaged.png)Figure 6:Performance of SAC based on C\-DSAC implementation on HalfCheetah\-v4

## 7Conclusion

This work presented C\-DSAC, a distributional maximum\-entropy deep reinforcement learning algorithm\. The objective function incorporates an energy distance\-based metric to estimate the expected Q\-values and their associated variances\. It has been formally established that distributional soft policy iteration, utilizing the energy distance metric for evaluation alongside standard policy improvement, converges to an optimal policy\. Numerical experiments show that C\-DSAC outperforms SAC in most benchmark environments despite a sub\-optimal implementation\. The performance seems to be superior especially for complex environments, like the Humanoid\-v4 environment\. The presented analysis suggests that the superior performance of C\-DSAC may among others be attributed to a confidence\-driven update of Q\-values\. It was shown that gradient weights of the critic loss function are small at state\-action pairs with high variance\. This results in a more conservative update for high\-variance targets, reducing adaptation to potentially overestimated values\. In this context, high variance indicates low confidence in the target, which improves only after repeated observations\. As a consequence, C\-DSAC learns Q\-values in a two\-phase process: First, building confidence in the target, then adapting to its expected value\. While this confidence\-building phase may imply slow and unstable learning, the numerical results indicate that the reduction in the effect of overestimation outweighs the additional cost\. The C\-DSAC framework and the current implementation remain open to various optimizations\. Experiments with different initialization values can be carried out, particularly for the critic’s variance head\. For a fair comparison with SAC baselines, automatic temperature adjustment was omitted, but could be integrated to enhance performance\. Additionally, an exploration strategy based on the value distribution can be developed\. C\-DSAC models the return with a fixed\-moment distribution\. Efforts can be made to efficiently apply GMMs with complex neural network approximators to allow for a more expressive return distribution\. Another promising direction for research is the utilization of task and reward decomposition based on the moments of the value distribution\.

## Acknowledgements

This work has been funded by grant LFF FV 90 of the State Research Funding Hamburg \(Landesforschungsförderung Hamburg\), and by PID2021\-123278OB\-I00 from the Spanish Ministry of Science and Innovation financed by “ERDF A way of making Europe"\.

## Appendix AStandard Reinforcement Learning

The proofs provided below rely on established results\(Sutton and Barto,[2018](https://arxiv.org/html/2605.08104#bib.bib78); Haarnojaet al\.,[2019](https://arxiv.org/html/2605.08104#bib.bib129)\)\. They are included to ensure the self\-containment of this work and to clarify the foundations upon which our Theorems and Lemmas are built\.

### A\.1Policy Evaluation

###### Lemma A\.1\.

LetQ1​\(s,a\)Q\_\{1\}\(s,a\)andQ2​\(s,a\)Q\_\{2\}\(s,a\)be two bounded state\-action value functions\. Assume\|𝒜\|<∞\|\\mathcal\{A\}\|<\\infty,\|𝒮\|<∞\|\\mathcal\{S\}\|<\\inftyand0≤γ<10\\leq\\gamma<1\. Then𝒯π\\mathcal\{T\}^\{\\pi\}according to \([5](https://arxiv.org/html/2605.08104#S3.E5)\) is aγ\\gamma\-contraction, i\.e\.

‖𝒯π​Q1−𝒯π​Q2‖∞≤γ​‖Q1−Q2‖∞\.\\\|\\mathcal\{T\}^\{\\pi\}Q\_\{1\}\-\\mathcal\{T\}^\{\\pi\}Q\_\{2\}\\\|\_\{\\infty\}\\leq\\gamma\\\|Q\_\{1\}\-Q\_\{2\}\\\|\_\{\\infty\}\.\(29\)

###### Proof\.

To show that \([29](https://arxiv.org/html/2605.08104#A1.E29)\) holds, the left\-hand side is expanded\. Set𝔼P​\[⋅\]:=𝔼st\+1∼P\(⋅∣st,at\)​\[⋅\]\\mathbb\{E\}\_\{P\}\[\\cdot\]:=\\mathbb\{E\}\_\{s\_\{t\+1\}\\sim P\(\\cdot\\mid s\_\{t\},a\_\{t\}\)\}\[\\cdot\]and𝔼at∼π\(⋅∣st\)\[Q\(st,at\)\]:=Q\(st,π\(⋅∣st\)\)\\mathbb\{E\}\_\{a\_\{t\}\\sim\\pi\(\\cdot\\mid s\_\{t\}\)\}\[Q\(s\_\{t\},a\_\{t\}\)\]:=Q\(s\_\{t\},\\pi\(\\cdot\\mid s\_\{t\}\)\), then

‖𝒯π​Q1−𝒯π​Q2‖∞\\displaystyle\\\|\\mathcal\{T\}^\{\\pi\}Q\_\{1\}\-\\mathcal\{T\}^\{\\pi\}Q\_\{2\}\\\|\_\{\\infty\}=∥\(r\(st,at\)\+γ𝔼P\[Q1\(st\+1,π\(⋅∣st\+1\)\)\]\)−\(r\(st,at\)\+γ𝔼P\[Q2\(st\+1,π\(⋅∣st\+1\)\)\]\)∥∞\\displaystyle=\\\|\\left\(r\(s\_\{t\},a\_\{t\}\)\+\\gamma\\mathbb\{E\}\_\{P\}\[Q\_\{1\}\(s\_\{t\+1\},\\pi\(\\cdot\\mid s\_\{t\+1\}\)\)\]\\right\)\-\\left\(r\(s\_\{t\},a\_\{t\}\)\+\\gamma\\mathbb\{E\}\_\{P\}\[Q\_\{2\}\(s\_\{t\+1\},\\pi\(\\cdot\\mid s\_\{t\+1\}\)\)\]\\right\)\\\|\_\{\\infty\}=supst∈𝒮,at∈𝒜\|γ𝔼P\[Q1\(st\+1,π\(⋅∣st\+1\)\)\]−γ𝔼P\[Q2\(st\+1,π\(⋅∣st\+1\)\)\]\|\\displaystyle=\\sup\_\{s\_\{t\}\\in\\mathcal\{S\},a\_\{t\}\\in\\mathcal\{A\}\}\|\\gamma\\mathbb\{E\}\_\{P\}\[Q\_\{1\}\(s\_\{t\+1\},\\pi\(\\cdot\\mid s\_\{t\+1\}\)\)\]\-\\gamma\\mathbb\{E\}\_\{P\}\[Q\_\{2\}\(s\_\{t\+1\},\\pi\(\\cdot\\mid s\_\{t\+1\}\)\)\]\|=γsupst∈𝒮,at∈𝒜\|𝔼P\[\(Q1\(st\+1,π\(⋅∣st\+1\)\)−Q2\(st\+1,π\(⋅∣st\+1\)\)\)\]\|\\displaystyle=\\gamma\\sup\_\{s\_\{t\}\\in\\mathcal\{S\},a\_\{t\}\\in\\mathcal\{A\}\}\|\\mathbb\{E\}\_\{P\}\[\(Q\_\{1\}\(s\_\{t\+1\},\\pi\(\\cdot\\mid s\_\{t\+1\}\)\)\-Q\_\{2\}\(s\_\{t\+1\},\\pi\(\\cdot\\mid s\_\{t\+1\}\)\)\)\]\|≤γsupst∈𝒮,at∈𝒜𝔼P\[\|Q1\(st\+1,π\(⋅∣st\+1\)\)−Q2\(st\+1,π\(⋅∣st\+1\)\)\|\]\\displaystyle\\leq\\gamma\\sup\_\{s\_\{t\}\\in\\mathcal\{S\},a\_\{t\}\\in\\mathcal\{A\}\}\\mathbb\{E\}\_\{P\}\[\|Q\_\{1\}\(s\_\{t\+1\},\\pi\(\\cdot\\mid s\_\{t\+1\}\)\)\-Q\_\{2\}\(s\_\{t\+1\},\\pi\(\\cdot\\mid s\_\{t\+1\}\)\)\|\]≤γsups∈𝒮\|Q1\(s,π\(⋅∣s\)\)−Q2\(s,π\(⋅∣s\)\)\|\\displaystyle\\leq\\gamma\\sup\_\{s\\in\\mathcal\{S\}\}\|Q\_\{1\}\(s,\\pi\(\\cdot\\mid s\)\)\-Q\_\{2\}\(s,\\pi\(\\cdot\\mid s\)\)\|≤γ​sups∈𝒮,a∈𝒜\|Q1​\(s,a\)−Q2​\(s,a\)\|=γ​‖Q1−Q2‖∞\.\\displaystyle\\leq\\gamma\\sup\_\{s\\in\\mathcal\{S\},a\\in\\mathcal\{A\}\}\|Q\_\{1\}\(s,a\)\-Q\_\{2\}\(s,a\)\|=\\gamma\\\|Q\_\{1\}\-Q\_\{2\}\\\|\_\{\\infty\}\.Since𝒯π\\mathcal\{T\}^\{\\pi\}is aγ\\gamma\-contraction, it follows from the Banach theorem that the series\{Qk\}\\\{Q\_\{k\}\\\}withQk\+1:=𝒯π​QkQ\_\{k\+1\}:=\\mathcal\{T\}^\{\\pi\}Q\_\{k\}converges to a unique fixed pointQπQ^\{\\pi\}, i\.e\.Qπ=𝒯π​QπQ^\{\\pi\}=\\mathcal\{T\}^\{\\pi\}Q^\{\\pi\}and‖Qk−Qπ‖∞→0\\\|Q\_\{k\}\-Q^\{\\pi\}\\\|\_\{\\infty\}\\rightarrow 0ask→∞k\\rightarrow\\infty\. ∎

### A\.2Policy Improvement

###### Lemma A\.2\.

Given the policy update rule in \([6](https://arxiv.org/html/2605.08104#S3.E6)\) and assuming bounded state\-action value functions,\|𝒜\|<∞\|\\mathcal\{A\}\|<\\infty,0≤γ<10\\leq\\gamma<1then

𝔼at∼πk\(⋅∣st\)​\[Qπk​\(st,at\)\]≤𝔼at∼πk\+1\(⋅∣st\)​\[Qπk\+1​\(st,at\)\]∀st∈𝒮\\mathbb\{E\}\_\{a\_\{t\}\\sim\\pi\_\{k\}\(\\cdot\\mid s\_\{t\}\)\}\[Q^\{\\pi\_\{k\}\}\(s\_\{t\},a\_\{t\}\)\]\\leq\\mathbb\{E\}\_\{a\_\{t\}\\sim\\pi\_\{k\+1\}\(\\cdot\\mid s\_\{t\}\)\}\[Q^\{\\pi\_\{k\+1\}\}\(s\_\{t\},a\_\{t\}\)\]\\quad\\forall s\_\{t\}\\in\\mathcal\{S\}\.

###### Proof\.

The proof relies on expanding the Q\-function in the Bellman equation repeatedly with actions of the improved policy\. For convenience, the notationπ′:=πk\+1\\pi^\{\\prime\}:=\\pi\_\{k\+1\}andπ:=πk\\pi:=\\pi\_\{k\}and𝔼π′,P,k\[⋅∣st\]:=𝔼at:k−1∼π′\(⋅∣st:k−1\),st\+1:k∼P\(⋅∣st:k−1,at:k−1\)\[⋅∣st\]\\mathbb\{E\}\_\{\\pi^\{\\prime\},P,k\}\[\\cdot\\mid s\_\{t\}\]:=\\mathbb\{E\}\_\{a\_\{t:k\-1\}\\sim\\pi^\{\\prime\}\(\\cdot\\mid s\_\{t:k\-1\}\),s\_\{t\+1:k\}\\sim P\(\\cdot\\mid s\_\{t:k\-1\},a\_\{t:k\-1\}\)\}\[\\cdot\\mid s\_\{t\}\]is used\. LetQπ\(st,π\(⋅∣st\)\):=𝔼at∼π\(⋅∣st\)\[Qπ\(st,at\)\]Q^\{\\pi\}\(s\_\{t\},\\pi\(\\cdot\\mid s\_\{t\}\)\):=\\mathbb\{E\}\_\{a\_\{t\}\\sim\\pi\(\\cdot\\mid s\_\{t\}\)\}\[Q^\{\\pi\}\(s\_\{t\},a\_\{t\}\)\], then

Qπ\(st,π\(⋅∣st\)\)\\displaystyle Q^\{\\pi\}\(s\_\{t\},\\pi\(\\cdot\\mid s\_\{t\}\)\)≤Qπ\(st,π′\(⋅∣st\)\)\\displaystyle\\leq Q^\{\\pi\}\(s\_\{t\},\\pi^\{\\prime\}\(\\cdot\\mid s\_\{t\}\)\)=𝔼π′,P,t\+1\[r\(st,at\)\+γQπ\(st\+1,π\(⋅∣st\+1\)\)∣st\]\\displaystyle=\\mathbb\{E\}\_\{\\pi^\{\\prime\},P,t\+1\}\[r\(s\_\{t\},a\_\{t\}\)\+\\gamma Q^\{\\pi\}\(s\_\{t\+1\},\\pi\(\\cdot\\mid s\_\{t\+1\}\)\)\\mid s\_\{t\}\]≤𝔼π′,P,t\+1\[r\(st,at\)\+γQπ\(st\+1,π′\(⋅∣st\+1\)\)∣st\]\\displaystyle\\leq\\mathbb\{E\}\_\{\\pi^\{\\prime\},P,t\+1\}\[r\(s\_\{t\},a\_\{t\}\)\+\\gamma Q^\{\\pi\}\(s\_\{t\+1\},\\pi^\{\\prime\}\(\\cdot\\mid s\_\{t\+1\}\)\)\\mid s\_\{t\}\]=𝔼π′,P,t\+2\[r\(st,at\)\+γ\(r\(st\+1,at\+1\)\+γQπ\(st\+2,π\(⋅∣st\+2\)\)\)∣st\]\\displaystyle=\\mathbb\{E\}\_\{\\pi^\{\\prime\},P,t\+2\}\[r\(s\_\{t\},a\_\{t\}\)\+\\gamma\(r\(s\_\{t\+1\},a\_\{t\+1\}\)\+\\gamma Q^\{\\pi\}\(s\_\{t\+2\},\\pi\(\\cdot\\mid s\_\{t\+2\}\)\)\)\\mid s\_\{t\}\]=𝔼π′,P,t\+2\[r\(st,at\)\+γr\(st\+1,at\+1\)\+γ2Qπ\(st\+2,π\(⋅∣st\+2\)\)∣st\]\\displaystyle=\\mathbb\{E\}\_\{\\pi^\{\\prime\},P,t\+2\}\[r\(s\_\{t\},a\_\{t\}\)\+\\gamma r\(s\_\{t\+1\},a\_\{t\+1\}\)\+\\gamma^\{2\}Q^\{\\pi\}\(s\_\{t\+2\},\\pi\(\\cdot\\mid s\_\{t\+2\}\)\)\\mid s\_\{t\}\]≤𝔼π′,P,t\+3\[r\(st,at\)\+γr\(st\+1,at\+1\)\+γ2r\(st\+2,at\+2\)\+γ3Qπ\(st\+3,π\(⋅∣st\+3\)\)∣st\]\\displaystyle\\leq\\mathbb\{E\}\_\{\\pi^\{\\prime\},P,t\+3\}\[r\(s\_\{t\},a\_\{t\}\)\+\\gamma r\(s\_\{t\+1\},a\_\{t\+1\}\)\+\\gamma^\{2\}r\(s\_\{t\+2\},a\_\{t\+2\}\)\+\\gamma^\{3\}Q^\{\\pi\}\(s\_\{t\+3\},\\pi\(\\cdot\\mid s\_\{t\+3\}\)\)\\mid s\_\{t\}\]⋮\\displaystyle\\vdots≤𝔼π′,P,T−1​\[r​\(st,at\)\+γ​r​\(st\+1,at\+1\)\+γ2​r​\(st\+2,at\+2\)\+…\+γT−1​r​\(sT−1,aT−1\)∣st\]\\displaystyle\\leq\\mathbb\{E\}\_\{\\pi^\{\\prime\},P,T\-1\}\[r\(s\_\{t\},a\_\{t\}\)\+\\gamma r\(s\_\{t\+1\},a\_\{t\+1\}\)\+\\gamma^\{2\}r\(s\_\{t\+2\},a\_\{t\+2\}\)\+\\ldots\+\\gamma^\{T\-1\}r\(s\_\{T\-1\},a\_\{T\-1\}\)\\mid s\_\{t\}\]=Qπ′\(st,π′\(⋅∣st\)\)\.\\displaystyle=Q^\{\\pi^\{\\prime\}\}\(s\_\{t\},\\pi^\{\\prime\}\(\\cdot\\mid s\_\{t\}\)\)\.Thus for anysts\_\{t\},

𝔼at∼πk\(⋅∣st\)\[Qπk\(st,at\)\]=Qπ\(st,π\(⋅∣st\)\)≤Qπ′\(st,π′\(⋅∣st\)\)=𝔼at∼πk\+1\(⋅∣st\)\[Qπk\+1\(st,at\)\]\.\\mathbb\{E\}\_\{a\_\{t\}\\sim\\pi\_\{k\}\(\\cdot\\mid s\_\{t\}\)\}\[Q^\{\\pi\_\{k\}\}\(s\_\{t\},a\_\{t\}\)\]=Q^\{\\pi\}\(s\_\{t\},\\pi\(\\cdot\\mid s\_\{t\}\)\)\\leq Q^\{\\pi^\{\\prime\}\}\(s\_\{t\},\\pi^\{\\prime\}\(\\cdot\\mid s\_\{t\}\)\)=\\mathbb\{E\}\_\{a\_\{t\}\\sim\\pi\_\{k\+1\}\(\\cdot\\mid s\_\{t\}\)\}\[Q^\{\\pi\_\{k\+1\}\}\(s\_\{t\},a\_\{t\}\)\]\.∎

### A\.3Policy Iteration

###### Lemma A\.3\.

Let the reward be bounded,\|𝒜\|<∞\|\\mathcal\{A\}\|<\\infty,\|𝒮\|<∞\|\\mathcal\{S\}\|<\\infty, and0≤γ<10\\leq\\gamma<1\. LetΠ\\Pibe a the set of all stationary \(possibly stochastic\) policies\. In particular,Π\\Picontains the point\-mass policies\. Alternating between exact policy evaluation and policy improvement from some initial policyπ0∈Π\\pi\_\{0\}\\in\\Piand state\-action functionQ\(0\)Q^\{\(0\)\}, the process terminates after finitely many improvement steps at the optimal policyπ∗\\pi^\{\*\}satisfyingQπ∗=Q∗Q^\{\\pi^\{\*\}\}=Q^\{\*\}\.

###### Proof\.

Fixkk\. By Lemma[A\.1](https://arxiv.org/html/2605.08104#A1.Thmtheorem1), the inner evaluation sequence\{Qm\}\\\{Q\_\{m\}\\\}forQm\+1:=𝒯πk​QmQ\_\{m\+1\}:=\\mathcal\{T\}^\{\\pi\_\{k\}\}Q\_\{m\}approachesQπkQ^\{\\pi\_\{k\}\}, i\.e\.‖Qm−Qπk‖∞→0\|\|Q\_\{m\}\-Q^\{\\pi\_\{k\}\}\|\|\_\{\\infty\}\\rightarrow 0asm→∞m\\rightarrow\\infty\. It is established by Lemma[A\.2](https://arxiv.org/html/2605.08104#A1.Thmtheorem2)that an improvement of policyπk\\pi\_\{k\}by Eq\. \([6](https://arxiv.org/html/2605.08104#S3.E6)\), yieldingπk\+1\\pi\_\{k\+1\}, satisfies Vπk​\(st\)=𝔼at∼πk\(⋅∣st\)​\[Qπk​\(st,at\)\]≤𝔼at∼πk\+1\(⋅∣st\)​\[Qπk\+1​\(st,at\)\]=Vπk\+1​\(st\)​∀st∈𝒮V^\{\\pi\_\{k\}\}\(s\_\{t\}\)=\\mathbb\{E\}\_\{a\_\{t\}\\sim\\pi\_\{k\}\(\\cdot\\mid s\_\{t\}\)\}\[Q^\{\\pi\_\{k\}\}\(s\_\{t\},a\_\{t\}\)\]\\leq\\mathbb\{E\}\_\{a\_\{t\}\\sim\\pi\_\{k\+1\}\(\\cdot\\mid s\_\{t\}\)\}\[Q^\{\\pi\_\{k\+1\}\}\(s\_\{t\},a\_\{t\}\)\]=V^\{\\pi\_\{k\+1\}\}\(s\_\{t\}\)\\forall s\_\{t\}\\in\\mathcal\{S\}\. Although policies are treated as stochastic mappings, theargmax\\operatorname\*\{argmax\}in Eq\. \([6](https://arxiv.org/html/2605.08104#S3.E6)\) implies that actions are taken deterministically, provided the action set is finite\. With finite states and actions and ties being broken deterministically per state \(e\.g\. lexicographic smallest action\), the set of greedy policies is also finite\. Under these circumstances, a greedy policy for a fixedQπkQ^\{\\pi\_\{k\}\}is unique, soπk\+1≠πk\\pi\_\{k\+1\}\\neq\\pi\_\{k\}implies strict increase in the valueVVfor at least one state\. Eventuallyπk\+1=πk\\pi\_\{k\+1\}=\\pi\_\{k\}in finite time, implying convergence to a fixed pointπk\+1=πk=π∗\\pi\_\{k\+1\}=\\pi\_\{k\}=\\pi^\{\*\}\. The terminating policyπ∗\\pi^\{\*\}is greedy w\.r\.t\. its ownQQ, therefore satisfies the Bellman optimality equation and thusQπ∗=Q∗Q^\{\\pi^\{\*\}\}=Q^\{\*\}\. ∎

## Appendix BMaximum\-Entropy Reinforcement Learning

### B\.1Soft Evaluation

###### Lemma B\.1\.

LetQ1​\(s,a\)Q\_\{1\}\(s,a\)andQ2​\(s,a\)Q\_\{2\}\(s,a\)be two bounded state\-action value functions\. Assume\|𝒜\|<∞\|\\mathcal\{A\}\|<\\infty,\|𝒮\|<∞\|\\mathcal\{S\}\|<\\infty,0≤γ<10\\leq\\gamma<1andα≥0\\alpha\\geq 0\. Then the soft Bellman operator𝒯hπ\\mathcal\{T\}\_\{h\}^\{\\pi\}according to \([9](https://arxiv.org/html/2605.08104#S3.E9)\) is aγ\\gamma\-contraction

‖𝒯hπ​Q1−𝒯hπ​Q2‖∞≤γ​‖Q1−Q2‖∞\.\\\|\\mathcal\{T\}\_\{h\}^\{\\pi\}Q\_\{1\}\-\\mathcal\{T\}\_\{h\}^\{\\pi\}Q\_\{2\}\\\|\_\{\\infty\}\\leq\\gamma\\\|Q\_\{1\}\-Q\_\{2\}\\\|\_\{\\infty\}\.\(30\)

###### Proof\.

By linearity of expectation, the entropy term can be separated from the Q\-function\. Define

rh\(st,at\):=r\(st,at\)\+γ𝔼st\+1∼P\(⋅∣st,at\)\[αℋ\(π\(⋅∣st\+1\)\)\]\.r\_\{h\}\(s\_\{t\},a\_\{t\}\):=r\(s\_\{t\},a\_\{t\}\)\+\\gamma\\mathbb\{E\}\_\{s\_\{t\+1\}\\sim P\(\\cdot\\mid s\_\{t\},a\_\{t\}\)\}\[\\alpha\\mathcal\{H\}\(\\pi\(\\cdot\\mid s\_\{t\+1\}\)\)\]\.Note that

𝒯hπ​Q​\(st,at\)=rh​\(st,at\)\+γ​𝔼st\+1∼P\(⋅∣st,at\),at\+1∼π\(⋅∣st\+1\)​\[Q​\(st\+1,at\+1\)\]\\mathcal\{T\}\_\{h\}^\{\\pi\}Q\(s\_\{t\},a\_\{t\}\)=r\_\{h\}\(s\_\{t\},a\_\{t\}\)\+\\gamma\\mathbb\{E\}\_\{\\begin\{subarray\}\{c\}s\_\{t\+1\}\\sim P\(\\cdot\\mid s\_\{t\},a\_\{t\}\),\\\\ a\_\{t\+1\}\\sim\\pi\(\\cdot\\mid s\_\{t\+1\}\)\\end\{subarray\}\}\[Q\(s\_\{t\+1\},a\_\{t\+1\}\)\]and since entropy is bounded for discrete actions,𝒯hπ\\mathcal\{T\}\_\{h\}^\{\\pi\}acts on a Banach space\. Then the proof of Lemma[A\.1](https://arxiv.org/html/2605.08104#A1.Thmtheorem1)can be applied such that

‖𝒯hπ​Q1−𝒯hπ​Q2‖∞≤γ​‖Q1−Q2‖∞\.\\\|\\mathcal\{T\}\_\{h\}^\{\\pi\}Q\_\{1\}\-\\mathcal\{T\}\_\{h\}^\{\\pi\}Q\_\{2\}\\\|\_\{\\infty\}\\leq\\gamma\\\|Q\_\{1\}\-Q\_\{2\}\\\|\_\{\\infty\}\.\(31\)Thus ifQk\+1:=𝒯hπ​QkQ\_\{k\+1\}:=\\mathcal\{T\}\_\{h\}^\{\\pi\}Q\_\{k\}, the series\{Qk\}\\\{Q\_\{k\}\\\}converges toQhπQ\_\{h\}^\{\\pi\}, i\.e\.‖Qk−Qhπ‖∞→0\\\|Q\_\{k\}\-Q\_\{h\}^\{\\pi\}\\\|\_\{\\infty\}\\rightarrow 0ask→∞k\\rightarrow\\infty\. ∎

### B\.2Soft Policy Improvement

It must be shown that \([8](https://arxiv.org/html/2605.08104#S3.E8)\) improves the policy\.

###### Lemma B\.2\.

\(One\-step soft improvement\)\. Letsts\_\{t\}be fixed and

Qhπ​\(st,at\):=r​\(st,at\)\+γ​𝔼st\+1∼P\(⋅∣st,at\)at\+1∼π\(⋅∣st\+1\)​\[Qhπ​\(st\+1,at\+1\)−α​log⁡π​\(at\+1∣st\+1\)\]\.Q\_\{h\}^\{\\pi\}\(s\_\{t\},a\_\{t\}\):=r\(s\_\{t\},a\_\{t\}\)\+\\gamma\\mathbb\{E\}\_\{\\begin\{subarray\}\{c\}s\_\{t\+1\}\\sim P\(\\cdot\\mid s\_\{t\},a\_\{t\}\)\\\\ a\_\{t\+1\}\\sim\\pi\(\\cdot\\mid s\_\{t\+1\}\)\\end\{subarray\}\}\[Q\_\{h\}^\{\\pi\}\(s\_\{t\+1\},a\_\{t\+1\}\)\-\\alpha\\log\{\\pi\(a\_\{t\+1\}\\mid s\_\{t\+1\}\)\}\]\.If the policy is updated according to \([8](https://arxiv.org/html/2605.08104#S3.E8)\),QhQ\_\{h\}is bounded,α\>0\\alpha\>0,0≤γ<10\\leq\\gamma<1and\|𝒜\|<∞\|\\mathcal\{A\}\|<\\infty, then

𝔼at∼πk\+1\(⋅∣st\)​\[Qhπk​\(st,at\)−α​log⁡πk\+1​\(at​\|⁡st\)\]≥𝔼at∼πk\(⋅∣st\)​\[Qhπk​\(st,at\)−α​log⁡πk​\(at​\|⁡st\)\]\.\\mathbb\{E\}\_\{a\_\{t\}\\sim\\pi\_\{k\+1\}\(\\cdot\\mid s\_\{t\}\)\}\[Q\_\{h\}^\{\\pi\_\{k\}\}\(s\_\{t\},a\_\{t\}\)\-\\alpha\\log\{\\pi\_\{k\+1\}\(a\_\{t\}\\operatorname\{\\,\|\\,\}s\_\{t\}\)\}\]\\geq\\mathbb\{E\}\_\{a\_\{t\}\\sim\\pi\_\{k\}\(\\cdot\\mid s\_\{t\}\)\}\[Q\_\{h\}^\{\\pi\_\{k\}\}\(s\_\{t\},a\_\{t\}\)\-\\alpha\\log\{\\pi\_\{k\}\(a\_\{t\}\\operatorname\{\\,\|\\,\}s\_\{t\}\)\}\]\.

###### Proof\.

πk\+1\(⋅∣st\)\\displaystyle\\pi\_\{k\+1\}\(\\cdot\\mid s\_\{t\}\)=\\displaystyle=argminπ∈ΠDK​L\(π\(⋅\|st\)\|\|exp⁡\(1α​Qhπk​\(st,⋅\)\)Wπk​\(st\)\)\\displaystyle\\operatorname\*\{argmin\}\_\{\\pi\\in\\Pi\}\{D\_\{KL\}\\left\(\\pi\(\\cdot\\operatorname\{\\,\|\\,\}s\_\{t\}\)\\ \\mathrel\{\\Big\|\\Big\|\}\\ \\frac\{\\exp\{\(\\frac\{1\}\{\\alpha\}Q\_\{h\}^\{\\pi\_\{k\}\}\(s\_\{t\},\\cdot\)\)\}\}\{W^\{\\pi\_\{k\}\}\(s\_\{t\}\)\}\\right\)\}=\\displaystyle=argminπ∈Π𝔼at∼π\(⋅∣st\)​\[log⁡\(π​\(at​\|⁡st\)​Wπk​\(st\)exp⁡1α​Qhπk​\(st,at\)\)\]\\displaystyle\\operatorname\*\{argmin\}\_\{\\pi\\in\\Pi\}\\mathbb\{E\}\_\{a\_\{t\}\\sim\\pi\(\\cdot\\mid s\_\{t\}\)\}\\left\[\\log\\left\(\\frac\{\\pi\(a\_\{t\}\\operatorname\{\\,\|\\,\}s\_\{t\}\)W^\{\\pi\_\{k\}\}\(s\_\{t\}\)\}\{\\exp\{\\frac\{1\}\{\\alpha\}Q\_\{h\}^\{\\pi\_\{k\}\}\(s\_\{t\},a\_\{t\}\)\}\}\\right\)\\right\]=\\displaystyle=argminπ∈Π𝔼at∼π\(⋅∣st\)​\[log⁡π​\(at​\|⁡st\)−1α​Qhπk​\(st,at\)\+log⁡Wπk​\(st\)\]\.\\displaystyle\\operatorname\*\{argmin\}\_\{\\pi\\in\\Pi\}\\mathbb\{E\}\_\{a\_\{t\}\\sim\\pi\(\\cdot\\mid s\_\{t\}\)\}\\left\[\\log\{\\pi\(a\_\{t\}\\operatorname\{\\,\|\\,\}s\_\{t\}\)\}\-\\frac\{1\}\{\\alpha\}Q\_\{h\}^\{\\pi\_\{k\}\}\(s\_\{t\},a\_\{t\}\)\+\\log\{W^\{\\pi\_\{k\}\}\(s\_\{t\}\)\}\\right\]\.Due to minimization, it holds𝔼at∼πk\+1\(⋅∣st\)​\[log⁡πk\+1​\(at​\|⁡st\)−1α​Qhπk​\(st,at\)\+log⁡Wπk​\(st\)\]≤𝔼at∼πk\(⋅∣st\)​\[log⁡πk​\(at​\|⁡st\)−1α​Qhπk​\(st,at\)\+log⁡Wπk​\(st\)\]\\mathbb\{E\}\_\{a\_\{t\}\\sim\\pi\_\{k\+1\}\(\\cdot\\mid s\_\{t\}\)\}\\left\[\\log\{\\pi\_\{k\+1\}\(a\_\{t\}\\operatorname\{\\,\|\\,\}s\_\{t\}\)\}\-\\frac\{1\}\{\\alpha\}Q\_\{h\}^\{\\pi\_\{k\}\}\(s\_\{t\},a\_\{t\}\)\+\\log\{W^\{\\pi\_\{k\}\}\(s\_\{t\}\)\}\\right\]\\leq\\mathbb\{E\}\_\{a\_\{t\}\\sim\\pi\_\{k\}\(\\cdot\\mid s\_\{t\}\)\}\\left\[\\log\{\\pi\_\{k\}\(a\_\{t\}\\operatorname\{\\,\|\\,\}s\_\{t\}\)\}\-\\frac\{1\}\{\\alpha\}Q\_\{h\}^\{\\pi\_\{k\}\}\(s\_\{t\},a\_\{t\}\)\+\\log\{W^\{\\pi\_\{k\}\}\(s\_\{t\}\)\}\\right\]and since the partition function does not depend on the action, it follows

𝔼at∼πk\+1\(⋅∣st\)​\[Qhπk​\(st,at\)−α​log⁡πk\+1​\(at​\|⁡st\)\]≥𝔼at∼πk\(⋅∣st\)​\[Qhπk​\(st,at\)−α​log⁡πk​\(at​\|⁡st\)\]\.\\mathbb\{E\}\_\{a\_\{t\}\\sim\\pi\_\{k\+1\}\(\\cdot\\mid s\_\{t\}\)\}\[Q\_\{h\}^\{\\pi\_\{k\}\}\(s\_\{t\},a\_\{t\}\)\-\\alpha\\log\{\\pi\_\{k\+1\}\(a\_\{t\}\\operatorname\{\\,\|\\,\}s\_\{t\}\)\}\]\\geq\\mathbb\{E\}\_\{a\_\{t\}\\sim\\pi\_\{k\}\(\\cdot\\mid s\_\{t\}\)\}\[Q\_\{h\}^\{\\pi\_\{k\}\}\(s\_\{t\},a\_\{t\}\)\-\\alpha\\log\{\\pi\_\{k\}\(a\_\{t\}\\operatorname\{\\,\|\\,\}s\_\{t\}\)\}\]\.∎

The soft Q\-value is expanded to show thatπk\+1\\pi\_\{k\+1\}is indeed an improvement onπk\\pi\_\{k\}in the following Lemma\.

###### Lemma B\.3\.

\(Global soft policy improvement\) Assume\|𝒜\|<∞\|\\mathcal\{A\}\|<\\infty,α\>0\\alpha\>0,0≤γ≤10\\leq\\gamma\\leq 1and boundedQhπQ\_\{h\}^\{\\pi\}\. Ifπk\+1\\pi\_\{k\+1\}is obtained according to the soft policy improvement in \([8](https://arxiv.org/html/2605.08104#S3.E8)\), then

Qhπk​\(st,at\)≤Qhπk\+1​\(st,at\)∀\(st,at\)∈𝒮×𝒜,Q\_\{h\}^\{\\pi\_\{k\}\}\(s\_\{t\},a\_\{t\}\)\\leq Q\_\{h\}^\{\\pi\_\{k\+1\}\}\(s\_\{t\},a\_\{t\}\)\\quad\\forall\(s\_\{t\},a\_\{t\}\)\\in\\mathcal\{S\}\\times\\mathcal\{A\},when applying Lemma[B\.2](https://arxiv.org/html/2605.08104#A2.Thmtheorem2)pointwise at each successor state\.

###### Proof\.

Analogous to Lemma[A\.2](https://arxiv.org/html/2605.08104#A1.Thmtheorem2), the proof relies on expanding the soft Q\-function in the Bellman equation repeatedly with actions of the improved policy\. For convenience, defineπ′:=πk\+1\\pi^\{\\prime\}:=\\pi\_\{k\+1\},π:=πk\\pi:=\\pi\_\{k\}and𝔼π′,P,k\[⋅∣st\]:=𝔼at:k−1∼π′\(⋅∣st:k−1\),st\+1:k∼P\(⋅∣st:k−1,at:k−1\)\[⋅∣st\]\\mathbb\{E\}\_\{\\pi^\{\\prime\},P,k\}\[\\cdot\\mid s\_\{t\}\]:=\\mathbb\{E\}\_\{a\_\{t:k\-1\}\\sim\\pi^\{\\prime\}\(\\cdot\\mid s\_\{t:k\-1\}\),s\_\{t\+1:k\}\\sim P\(\\cdot\\mid s\_\{t:k\-1\},a\_\{t:k\-1\}\)\}\[\\cdot\\mid s\_\{t\}\]\. Applying the one\-step improvement from Lemma[B\.2](https://arxiv.org/html/2605.08104#A2.Thmtheorem2)\) at each successor state yields

Vhπ​\(st\)\\displaystyle V\_\{h\}^\{\\pi\}\(s\_\{t\}\)≤𝔼π′,P,t\+1​\[Qhπ​\(st,at\)−α​log⁡π′​\(at∣st\)∣st\]\\displaystyle\\leq\\mathbb\{E\}\_\{\\pi^\{\\prime\},P,t\+1\}\[Q\_\{h\}^\{\\pi\}\(s\_\{t\},a\_\{t\}\)\-\\alpha\\log\{\\pi^\{\\prime\}\(a\_\{t\}\\mid s\_\{t\}\)\}\\mid s\_\{t\}\]=𝔼π′,P,t\+1​\[r​\(st,at\)−α​log⁡π′​\(at∣st\)\+γ​Vhπ​\(st\+1\)∣st\]\\displaystyle=\\mathbb\{E\}\_\{\\pi^\{\\prime\},P,t\+1\}\[r\(s\_\{t\},a\_\{t\}\)\-\\alpha\\log\\pi^\{\\prime\}\(a\_\{t\}\\mid s\_\{t\}\)\+\\gamma V\_\{h\}^\{\\pi\}\(s\_\{t\+1\}\)\\mid s\_\{t\}\]≤𝔼π′,P,t\+2​\[r​\(st,at\)−α​log⁡π′​\(at∣st\)\+γ​\(Qhπ​\(st\+1,at\+1\)−α​log⁡π′​\(at\+1∣st\+1\)\)∣st\]\\displaystyle\\leq\\mathbb\{E\}\_\{\\pi^\{\\prime\},P,t\+2\}\[r\(s\_\{t\},a\_\{t\}\)\-\\alpha\\log\\pi^\{\\prime\}\(a\_\{t\}\\mid s\_\{t\}\)\+\\gamma\(Q\_\{h\}^\{\\pi\}\(s\_\{t\+1\},a\_\{t\+1\}\)\-\\alpha\\log\{\\pi^\{\\prime\}\(a\_\{t\+1\}\\mid s\_\{t\+1\}\)\)\}\\mid s\_\{t\}\]=𝔼π′,P,t\+2\[r\(st,at\)−αlogπ′\(at∣st\)\+γ\(r\(st\+1,at\+1\)−αlogπ′\(at\+1∣st\+1\)\)\+\\displaystyle=\\mathbb\{E\}\_\{\\pi^\{\\prime\},P,t\+2\}\[r\(s\_\{t\},a\_\{t\}\)\-\\alpha\\log\\pi^\{\\prime\}\(a\_\{t\}\\mid s\_\{t\}\)\+\\gamma\(r\(s\_\{t\+1\},a\_\{t\+1\}\)\-\\alpha\\log\{\\pi^\{\\prime\}\(a\_\{t\+1\}\\mid s\_\{t\+1\}\)\}\)\+γ2Vhπ\(st\+2\)∣st\]\\displaystyle\\quad\\quad\\quad\\quad\\gamma^\{2\}V\_\{h\}^\{\\pi\}\(s\_\{t\+2\}\)\\mid s\_\{t\}\]⋮\\displaystyle\\vdots≤𝔼π′,P,T−1\[r\(st,at\)−αlogπ′\(at∣st\)\+γ\(r\(st\+1,at\+1\)−αlogπ′\(at\+1\|st\+1\)\)\+…\\displaystyle\\leq\\mathbb\{E\}\_\{\\pi^\{\\prime\},P,T\-1\}\[r\(s\_\{t\},a\_\{t\}\)\-\\alpha\\log\{\\pi^\{\\prime\}\(a\_\{t\}\\mid s\_\{t\}\)\}\+\\gamma\(r\(s\_\{t\+1\},a\_\{t\+1\}\)\-\\alpha\\log\{\\pi^\{\\prime\}\(a\_\{t\+1\}\\operatorname\{\\,\|\\,\}s\_\{t\+1\}\)\}\)\+\\ldots…\+γT−1\(r\(sT−1,aT−1\)−αlogπ′\(aT−1\|sT−1\)\)\|st=s\]\\displaystyle\\quad\\quad\\quad\\quad\\quad\\quad\\ldots\+\\gamma^\{T\-1\}\(r\(s\_\{T\-1\},a\_\{T\-1\}\)\-\\alpha\\log\{\\pi^\{\\prime\}\(a\_\{T\-1\}\\operatorname\{\\,\|\\,\}s\_\{T\-1\}\)\}\)\\operatorname\{\\,\|\\,\}s\_\{t\}=s\]=Vhπ′​\(st\)\.\\displaystyle=V\_\{h\}^\{\\pi^\{\\prime\}\}\(s\_\{t\}\)\.
Thus for anysts\_\{t\}

Qhπk​\(st,at\)\\displaystyle Q\_\{h\}^\{\\pi\_\{k\}\}\(s\_\{t\},a\_\{t\}\)=𝔼st\+1∼P\(⋅∣st,at\)​\[r​\(st,at\)\+γ​Vhπk​\(st\+1\)\]\\displaystyle=\\mathbb\{E\}\_\{s\_\{t\+1\}\\sim P\(\\cdot\\mid s\_\{t\},a\_\{t\}\)\}\[r\(s\_\{t\},a\_\{t\}\)\+\\gamma V\_\{h\}^\{\\pi\_\{k\}\}\(s\_\{t\+1\}\)\]≤𝔼st\+1∼P\(⋅∣st,at\)​\[r​\(st,at\)\+γ​Vhπk\+1​\(st\+1\)\]=Qhπk\+1​\(st,at\)\.\\displaystyle\\leq\\mathbb\{E\}\_\{s\_\{t\+1\}\\sim P\(\\cdot\\mid s\_\{t\},a\_\{t\}\)\}\[r\(s\_\{t\},a\_\{t\}\)\+\\gamma V\_\{h\}^\{\\pi\_\{k\+1\}\}\(s\_\{t\+1\}\)\]=Q\_\{h\}^\{\\pi\_\{k\+1\}\}\(s\_\{t\},a\_\{t\}\)\.∎

### B\.3Iteration

###### Lemma B\.4\.

Let the reward be bounded,\|𝒜\|<∞\|\\mathcal\{A\}\|<\\infty,\|𝒮\|<∞\|\\mathcal\{S\}\|<\\infty,0≤γ<10\\leq\\gamma<1andα\>0\\alpha\>0\. LetΠ\\Pibe a set of all stationary, stochastic policies\. Alternating between exact soft policy evaluation and global soft policy improvement from some initial policyπ0∈Π\\pi\_\{0\}\\in\\Piand soft state\-action functionQh\(0\)Q\_\{h\}^\{\(0\)\}, the process converges \(in the limit\) to the optimal policyπ∗\\pi^\{\*\}, satisfyingQhπ∗=Qh∗Q\_\{h\}^\{\\pi^\{\*\}\}=Q\_\{h\}^\{\*\}\.

###### Proof\.

At each iterationkk, the policyπk\\pi\_\{k\}is fixed during evaluation\. By Lemma[B\.1](https://arxiv.org/html/2605.08104#A2.Thmtheorem1),𝒯hπk\\mathcal\{T\}\_\{h\}^\{\\pi\_\{k\}\}is aγ\\gamma\-contraction and the evaluation sequence converges to the fixed pointQhπkQ\_\{h\}^\{\\pi\_\{k\}\}\. Therefore,QhπkQ\_\{h\}^\{\\pi\_\{k\}\}is well defined at each iteration\. By Lemma[B\.3](https://arxiv.org/html/2605.08104#A2.Thmtheorem3), the sequence\{Qhπk​\(st,at\)\}\\\{Q\_\{h\}^\{\\pi\_\{k\}\}\(s\_\{t\},a\_\{t\}\)\\\}is non\-decreasing for each pair\(st,at\)\(s\_\{t\},a\_\{t\}\)\. Because the reward is bounded and\|𝒮\|<∞\|\\mathcal\{S\}\|<\\infty,𝒜<∞\\mathcal\{A\}<\\infty, the sequence converges to some point\-wise limitQ¯\\bar\{Q\}\. Policies are updated by the softmax of the currentQhQ\_\{h\}and softmax is continuous, therefore the limit pointπ∗\\pi^\{\*\}exists and is the softmax ofQ¯\\bar\{Q\}, i\.e\.π∗=softmax​\(Q¯\)\\pi^\{\*\}=\\text\{softmax\}\(\\bar\{Q\}\)\. Sinceπ∗∈Π\\pi^\{\*\}\\in\\Pi, Lemma[B\.1](https://arxiv.org/html/2605.08104#A2.Thmtheorem1)equally guarantees well\-definedness in the limit pointQhπ∗Q\_\{h\}^\{\\pi^\{\*\}\}\. The mapπ↦Qhπ\\pi\\mapsto Q\_\{h\}^\{\\pi\}is continuous, henceπk→π∗\\pi\_\{k\}\\rightarrow\\pi^\{\*\}impliesQhπk→Qhπ∗Q\_\{h\}^\{\\pi\_\{k\}\}\\rightarrow Q\_\{h\}^\{\\pi^\{\*\}\}and thereforeQ¯=Qhπ∗\\bar\{Q\}=Q\_\{h\}^\{\\pi^\{\*\}\}\. It followsπ∗=softmax​\(Qhπ∗\)\\pi^\{\*\}=\\text\{softmax\}\(Q\_\{h\}^\{\\pi^\{\*\}\}\)\(the policy is soft\-greedy w\.r\.t\.Qhπ∗Q\_\{h\}^\{\\pi^\{\*\}\}\) and thereforeQhπ∗Q\_\{h\}^\{\\pi^\{\*\}\}satisfies the Bellman optimality equation, thusQhπ∗=Qh∗Q\_\{h\}^\{\\pi^\{\*\}\}=Q\_\{h\}^\{\*\}\. ∎

## Appendix CHyperparameters

Hyperparameters for the C\-DSAC experiments were systematically tuned to maintain parity with the SAC baselines, ensuring an equitable comparison\.

Table 2:C\-DSAC HyperparametersTable 3:Environment\-specific Entropy
## References

- \[1\]M\. G\. Bellemare, W\. Dabney, and R\. Munos\(2017\)A distributional perspective on reinforcement learning\.InProceedings of the 34th International Conference on Machine Learning \- Volume 70,ICML’17,pp\. 449–458\.External Links:[Link](https://proceedings.mlr.press/v70/bellemare17a/bellemare17a.pdf)Cited by:[§1](https://arxiv.org/html/2605.08104#S1.p2.1),[§2](https://arxiv.org/html/2605.08104#S2.p1.3),[§3\.3](https://arxiv.org/html/2605.08104#S3.SS3.p1.4),[§3\.3](https://arxiv.org/html/2605.08104#S3.SS3.p1.6),[§3\.3](https://arxiv.org/html/2605.08104#S3.SS3.p1.7),[§3\.3](https://arxiv.org/html/2605.08104#S3.SS3.p2.1)\.
- \[2\]M\. G\. Bellemare, W\. Dabney, and M\. Rowland\(2023\)Distributional reinforcement learning\.MIT Press\.Note:[http://www\.distributional\-rl\.org](http://www.distributional-rl.org/), Accessed: 2023\-09\-15Cited by:[§1](https://arxiv.org/html/2605.08104#S1.p2.1)\.
- \[3\]M\. G\. Bellemare, I\. Danihelka, W\. Dabney, S\. Mohamed, B\. Lakshminarayanan, S\. Hoyer, and R\. Munos\(2017\)The Cramer distance as a solution to biased Wasserstein gradients\.External Links:1705\.10743,[Link](https://arxiv.org/abs/1705.10743)Cited by:[§3\.4](https://arxiv.org/html/2605.08104#S3.SS4.1.p1.1),[§3\.4](https://arxiv.org/html/2605.08104#S3.SS4.2.p1.1),[§3\.4](https://arxiv.org/html/2605.08104#S3.SS4.p2.1)\.
- \[4\]G\. Brockman, V\. Cheung, L\. Pettersson, J\. Schneider, J\. Schulman, J\. Tang, and W\. Zaremba\(2016\)Openai gym\.arXiv preprint arXiv:1606\.01540\.Cited by:[§6\.1](https://arxiv.org/html/2605.08104#S6.SS1.p1.9)\.
- \[5\]W\. Dabney, G\. Ostrovski, D\. Silver, and R\. Munos\(2018\)Implicit quantile networks for distributional reinforcement learning\.External Links:1806\.06923,[Link](https://arxiv.org/abs/1806.06923)Cited by:[§2](https://arxiv.org/html/2605.08104#S2.p1.3)\.
- \[6\]W\. Dabney, M\. Rowland, M\. G\. Bellemare, and R\. Munos\(2017\)Distributional reinforcement learning with quantile regression\.InAAAI Conference on Artificial Intelligence,External Links:[Link](https://ojs.aaai.org/index.php/AAAI/article/view/11791),[Document](https://dx.doi.org/10.1609/aaai.v32i1.11791)Cited by:[§2](https://arxiv.org/html/2605.08104#S2.p1.3)\.
- \[7\]J\. Duan, Y\. Guan, S\. E\. Li, Y\. Ren, Q\. Sun, and B\. CHENG\(2022\-11\)Distributional soft actor\-critic: off\-policy reinforcement learning for addressing value estimation errors\.IEEE Transactions on Neural Networks and Learning Systems33\(11\),pp\. 6584–6598\.External Links:[Link](https://arxiv.org/abs/2001.02811),[Document](https://dx.doi.org/10.1109/tnnls.2021.3082568)Cited by:[§2](https://arxiv.org/html/2605.08104#S2.p1.3),[§6](https://arxiv.org/html/2605.08104#S6.p1.1)\.
- \[8\]L\. Engstrom, A\. Ilyas, S\. Santurkar, D\. Tsipras, F\. Janoos, L\. Rudolph, and A\. Madry\(2020\)Implementation matters in deep policy gradients: a case study on PPO and TRPO\.External Links:2005\.12729,[Link](https://arxiv.org/abs/2005.12729)Cited by:[§6](https://arxiv.org/html/2605.08104#S6.p1.1)\.
- \[9\]A\. Fawzi, M\. Balog, A\. Huang, T\. Hubert, B\. Romera\-Paredes, M\. Barekatain, A\. Novikov, F\. J\. R\. Ruiz, J\. Schrittwieser, G\. Swirszcz, D\. Silver, D\. Hassabis, and P\. Kohli\(2022\)Discovering faster matrix multiplication algorithms with reinforcement learning\.Nature610,pp\. 47 – 53\.External Links:[Link](https://doi.org/10.1038/s41586-022-05172-4)Cited by:[§1](https://arxiv.org/html/2605.08104#S1.p1.1)\.
- \[10\]S\. Fujimoto, H\. van Hoof, and D\. Meger\(2018\)Addressing function approximation error in actor\-critic methods\.InProceedings of the 35th International Conference on Machine Learning,J\. Dy and A\. Krause \(Eds\.\),Proceedings of Machine Learning Research, Vol\.80,pp\. 1587–1596\.External Links:[Link](https://proceedings.mlr.press/v80/fujimoto18a.html)Cited by:[§1](https://arxiv.org/html/2605.08104#S1.p1.1),[§5\.1](https://arxiv.org/html/2605.08104#S5.SS1.p1.2)\.
- \[11\]S\. S\. Gu, E\. Holly, T\. P\. Lillicrap, and S\. Levine\(2017\-05\)Deep reinforcement learning for robotic manipulation with asynchronous off\-policy updates\.InProceedings of the IEEE International Conference on Robotics and Automation \(ICRA\),Piscataway, NJ, USA,pp\. 3389–3396\.External Links:[Document](https://dx.doi.org/10.1109/ICRA.2017.7989385),[Link](https://ieeexplore.ieee.org/document/7989385)Cited by:[§1](https://arxiv.org/html/2605.08104#S1.p1.1)\.
- \[12\]T\. Haarnoja, A\. Zhou, P\. Abbeel, and S\. Levine\(2018\)Soft actor\-critic: off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.InProceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10\-15, 2018,J\. G\. Dy and A\. Krause \(Eds\.\),Proceedings of Machine Learning Research, Vol\.80,pp\. 1856–1865\.External Links:[Link](http://proceedings.mlr.press/v80/haarnoja18b.html)Cited by:[§1](https://arxiv.org/html/2605.08104#S1.p1.1),[§3\.2](https://arxiv.org/html/2605.08104#S3.SS2.p1.1),[§5\.1](https://arxiv.org/html/2605.08104#S5.SS1.SSS0.Px2.p1.6),[Figure 5](https://arxiv.org/html/2605.08104#S6.F5),[Figure 5](https://arxiv.org/html/2605.08104#S6.F5.3.2),[§6\.1](https://arxiv.org/html/2605.08104#S6.SS1.p1.9),[§6](https://arxiv.org/html/2605.08104#S6.p1.1)\.
- \[13\]T\. Haarnoja, A\. Zhou, K\. Hartikainen, G\. Tucker, S\. Ha, J\. Tan, V\. Kumar, H\. Zhu, A\. Gupta, P\. Abbeel, and S\. Levine\(2019\)Soft actor\-critic algorithms and applications\.External Links:1812\.05905,[Link](https://arxiv.org/abs/1812.05905)Cited by:[Appendix A](https://arxiv.org/html/2605.08104#A1.p1.1)\.
- \[14\]H\. v\. Hasselt, A\. Guez, and D\. Silver\(2016\)Deep reinforcement learning with double Q\-learning\.InProceedings of the Thirtieth AAAI Conference on Artificial Intelligence,AAAI’16,pp\. 2094–2100\.Cited by:[§1](https://arxiv.org/html/2605.08104#S1.p1.1)\.
- \[15\]P\. Henderson, R\. Islam, P\. Bachman, J\. Pineau, D\. Precup, and D\. Meger\(2017\)Deep reinforcement learning that matters\.InAAAI Conference on Artificial Intelligence,External Links:[Link](https://doi.org/10.48550/arXiv.1709.06560)Cited by:[§6\.1](https://arxiv.org/html/2605.08104#S6.SS1.p1.9)\.
- \[16\]D\. Kingma and J\. Ba\(2015\)Adam: a method for stochastic optimization\.InInternational Conference on Learning Representations \(ICLR\),San Diega, CA, USA\.External Links:[Link](https://api.semanticscholar.org/CorpusID:6628106)Cited by:[Table 2](https://arxiv.org/html/2605.08104#A3.T2.5.8.3.2)\.
- \[17\]D\. P\. Kingma and M\. Welling\(2022\)Auto\-Encoding Variational Bayes\.External Links:1312\.6114,[Link](https://arxiv.org/abs/1312.6114)Cited by:[§5\.1](https://arxiv.org/html/2605.08104#S5.SS1.SSS0.Px2.p1.6)\.
- \[18\]Q\. Lan, Y\. Pan, A\. Fyshe, and M\. White\(2020\)Maxmin q\-learning: controlling the estimation bias of q\-learning\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Bkg0u3Etwr)Cited by:[§5\.3](https://arxiv.org/html/2605.08104#S5.SS3.p2.2)\.
- \[19\]A\. Lhéritier and N\. Bondoux\(2022\)A Cramér distance perspective on quantile regression based distributional reinforcement learning\.External Links:2110\.00535,[Link](https://arxiv.org/abs/2110.00535)Cited by:[§2](https://arxiv.org/html/2605.08104#S2.p1.3)\.
- \[20\]T\. P\. Lillicrap, J\. J\. Hunt, A\. Pritzel, N\. Heess, T\. Erez, Y\. Tassa, D\. Silver, and D\. Wierstra\(2019\)Continuous control with deep reinforcement learning\.External Links:1509\.02971,[Link](https://arxiv.org/abs/1509.02971)Cited by:[§1](https://arxiv.org/html/2605.08104#S1.p1.1)\.
- \[21\]X\. Ma, L\. Xia, Z\. Zhou, J\. Yang, and Q\. Zhao\(2020\)DSAC: distributional soft actor critic for risk\-sensitive reinforcement learning\.External Links:2004\.14547,[Link](https://arxiv.org/abs/2004.14547)Cited by:[§2](https://arxiv.org/html/2605.08104#S2.p1.3),[§6](https://arxiv.org/html/2605.08104#S6.p1.1)\.
- \[22\]V\. Mnih, K\. Kavukcuoglu, D\. Silver, A\. Graves, I\. Antonoglou, D\. Wierstra, and M\. Riedmiller\(2013\)Playing atari with deep reinforcement learning\.External Links:1312\.5602,[Link](https://arxiv.org/abs/1312.5602)Cited by:[§1](https://arxiv.org/html/2605.08104#S1.p1.1)\.
- \[23\]T\. H\. Muller, J\. L\. Butler, S\. Veselic, B\. Miranda, T\. E\. J\. Behrens, Z\. Kurth\-Nelsfon, and S\. W\. Kennerley\(2021\)Distributional reinforcement learning in prefrontal cortex\.Nature Neuroscience27,pp\. 403 – 408\.External Links:[Link](https://pubmed.ncbi.nlm.nih.gov/38200183/)Cited by:[§1](https://arxiv.org/html/2605.08104#S1.p2.1)\.
- \[24\]D\. W\. Nam, Y\. Kim, and C\. Y\. Park\(2021\-18–24 Jul\)GMAC: a distributional perspective on actor\-critic framework\.InProceedings of the 38th International Conference on Machine Learning,M\. Meila and T\. Zhang \(Eds\.\),Proceedings of Machine Learning Research, Vol\.139,pp\. 7930–7939\.External Links:[Link](https://proceedings.mlr.press/v139/nam21a.html)Cited by:[§2](https://arxiv.org/html/2605.08104#S2.p1.3)\.
- \[25\]L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. Christiano, J\. Leike, and R\. Lowe\(2022\)Training language models to follow instructions with human feedback\.External Links:2203\.02155,[Link](https://arxiv.org/abs/2203.02155)Cited by:[§1](https://arxiv.org/html/2605.08104#S1.p1.1)\.
- \[26\]C\. Paduraru, D\. J\. Mankowitz, G\. Dulac\-Arnold, J\. Li, N\. Levine, S\. Gowal, and T\. Hester\(2021\)Challenges of real\-world reinforcement learning: definitions, benchmarks and analysis\.Machine Learning110,pp\. 2419 – 2468\.External Links:[Link](https://link.springer.com/article/10.1007/s10994-021-05961-4)Cited by:[§1](https://arxiv.org/html/2605.08104#S1.p1.1)\.
- \[27\]J\. Schulman, S\. Levine, P\. Moritz, M\. I\. Jordan, and P\. Abbeel\(2017\)Trust region policy optimization\.External Links:1502\.05477,[Link](https://arxiv.org/abs/1502.05477)Cited by:[§1](https://arxiv.org/html/2605.08104#S1.p1.1)\.
- \[28\]J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov\(2017\)Proximal policy optimization algorithms\.External Links:1707\.06347,[Link](https://arxiv.org/abs/1707.06347)Cited by:[§1](https://arxiv.org/html/2605.08104#S1.p1.1)\.
- \[29\]D\. Sejdinovic, B\. Sriperumbudur, A\. Gretton, and K\. Fukumizu\(2013\)EQUIVALENCE of distance\-based and rkhs\-based statistics in hypothesis testing\.The Annals of Statistics41\(5\),pp\. 2263–2291\.External Links:ISSN 00905364, 21688966,[Link](http://www.jstor.org/stable/23566550)Cited by:[§3\.4](https://arxiv.org/html/2605.08104#S3.SS4.3.p1.1)\.
- \[30\]R\. S\. Sutton and A\. G\. Barto\(2018\)Reinforcement learning: an introduction\.Second edition,The MIT Press\.External Links:[Link](http://incompleteideas.net/book/the-book-2nd.html)Cited by:[Appendix A](https://arxiv.org/html/2605.08104#A1.p1.1),[§1](https://arxiv.org/html/2605.08104#S1.p1.1)\.
- \[31\]S\. Thrun and A\. Schwartz\(1993\)Issues in using function approximation for reinforcement learning\.InProceedings of the 1993 Connectionist Models Summer School,M\. Mozer, P\. Smolensky, D\. Touretzky, J\. Elman, and A\. Weigend \(Eds\.\),pp\. 255–263\.External Links:[Link](http://www.ri.cmu.edu/pub%5C_files/pub1/thrun%5C_sebastian%5C_1993%5C_1/thrun%5C_sebastian%5C_1993%5C_1.pdf)Cited by:[§5\.3](https://arxiv.org/html/2605.08104#S5.SS3.p1.6)\.
- \[32\]K\. Wang, O\. Oertell, A\. Agarwal, N\. Kallus, and W\. Sun\(2024\)More benefits of being distributional: second\-order bounds for reinforcement learning\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 51192–51213\.Cited by:[§3\.3](https://arxiv.org/html/2605.08104#S3.SS3.p1.7)\.
- \[33\]K\. Wang, K\. Zhou, R\. Wu, N\. Kallus, and W\. Sun\(2023\)The benefits of being distributional: small\-loss bounds for reinforcement learning\.External Links:2305\.15703Cited by:[§3\.3](https://arxiv.org/html/2605.08104#S3.SS3.p1.7)\.
- \[34\]D\. Yang, L\. Zhao, Z\. Lin, T\. Qin, J\. Bian, and T\. Liu\(2019\)Fully parameterized quantile function for distributional reinforcement learning\.InProceedings of the 33rd International Conference on Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2605.08104#S2.p1.3)\.
- \[35\]K\. Zhang, Y\. Zuo, B\. He, Y\. Sun, R\. Liu, C\. Jiang, Y\. Fan, K\. Tian, G\. Jia, P\. Li, Y\. Fu, X\. Lv, Y\. Zhang, S\. Zeng, S\. Qu, H\. Li, S\. Wang, Y\. Wang, X\. Long, F\. Liu, X\. Xu, J\. Ma, X\. Zhu, E\. Hua, Y\. Liu, Z\. Li, H\. Chen, X\. Qu, Y\. Li, W\. Chen, Z\. Yuan, J\. Gao, D\. Li, Z\. Ma, G\. Cui, Z\. Liu, B\. Qi, N\. Ding, and B\. Zhou\(2025\)A survey of reinforcement learning for large reasoning models\.External Links:2509\.08827,[Link](https://arxiv.org/abs/2509.08827)Cited by:[§1](https://arxiv.org/html/2605.08104#S1.p1.1)\.
- \[36\]V\. M\. Zolotarev\(1976\-04\)METRIC distances in spaces of random variables and their distributions\.Mathematics of the USSR\-Sbornik30\(3\),pp\. 373\.External Links:[Document](https://dx.doi.org/10.1070/SM1976v030n03ABEH002280),[Link](https://dx.doi.org/10.1070/SM1976v030n03ABEH002280)Cited by:[§3\.4](https://arxiv.org/html/2605.08104#S3.SS4.p2.1)\.