RS-Diffuser: Risk-Sensitive Diffusion Planning with Distributional Value Guidance

# RS-Diffuser: Risk-Sensitive Diffusion Planning with Distributional Value Guidance
Source: [https://arxiv.org/html/2606.27766](https://arxiv.org/html/2606.27766)
11institutetext:Northwestern Polytechnical University
11email:gongshiqiang@mail\.nwpu\.edu\.cn###### Abstract

Offline reinforcement learning enables policy learning from fixed datasets without additional environment interaction, making it appealing for safety\-critical applications where online exploration is costly or unsafe\. Diffusion\-based decision\-making methods have recently achieved strong performance in offline RL by modeling rich, multimodal trajectory distributions\. However, existing diffusion planners are typically risk\-neutral and therefore may overlook rare but catastrophic outcomes that are crucial in real\-world deployment\. In this work, we proposeRS\-Diffuser, a risk\-sensitive offline diffusion planning framework that combines diffusion\-based trajectory generation with distributional value critics\. RS\-Diffuser learns a diffusion planner over future state trajectories, a separate inverse dynamics model for action decoding, and a Monte Carlo distributional critic that estimates the full return distribution of candidate plans through quantile regression\. At sampling time, we incorporate a risk\-sensitive guidance signal into the denoising process, using gradients computed from tail\-aware objectives such as Conditional Value at Risk to steer generation toward desired risk profiles\. As a result, a single trained model can flexibly produce risk\-averse, risk\-neutral, or risk\-seeking behaviors by changing only the inference\-time risk parameter\. Extensive experiments on risk\-sensitive D4RL and risky robot navigation benchmarks demonstrate that RS\-Diffuser achieves state\-of\-the\-art performance, improving both overall return and worst\-case robustness while reducing safety violations\.

## 1Introduction

Offline reinforcement learning \(RL\) seeks to learn effective decision\-making policies from a fixed dataset without further interaction with the environment\[[20](https://arxiv.org/html/2606.27766#bib.bib27)\]\. This setting is particularly attractive in safety\-critical domains such as healthcare, autonomous driving, and robotics, where online exploration is costly, unsafe, or impractical\[[27](https://arxiv.org/html/2606.27766#bib.bib22)\]\. However, offline RL remains fundamentally challenging due to distributional shift: the learned policy must act beyond the support of the behavior data, often leading to unreliable value estimates and poor generalization\. In addition, many existing offline RL methods rely on deterministic or simple Gaussian policy classes, which can be insufficient for modeling the complex and multimodal behaviors commonly present in real\-world datasets\[[41](https://arxiv.org/html/2606.27766#bib.bib7)\]\.

Recently, diffusion models have emerged as a powerful paradigm for offline decision making by casting control as conditional generative modeling over actions or trajectories\[[9](https://arxiv.org/html/2606.27766#bib.bib28),[41](https://arxiv.org/html/2606.27766#bib.bib7)\]\. Existing approaches can be broadly divided into two families\. Planning\-based methods, such as Diffuser\[[10](https://arxiv.org/html/2606.27766#bib.bib1)\], generate trajectories through iterative denoising and perform decision making by trajectory optimization\. Policy\-based methods, such as Diffusion\-QL\[[28](https://arxiv.org/html/2606.27766#bib.bib29)\], instead use diffusion models as expressive policy classes for direct action generation\. By modeling rich multimodal distributions and long\-horizon dependencies, diffusion methods have achieved strong performance across a range of offline RL benchmarks\.

Despite these advances, most existing diffusion\-based decision\-making methods remain fundamentally risk\-neutral: they optimize expected return and do not explicitly model the tail behavior of returns\. This limitation is particularly problematic in safety\-critical settings, where two candidate behaviors may have similar expected returns but drastically different downside risk\[[22](https://arxiv.org/html/2606.27766#bib.bib31)\]\. Risk\-sensitive RL addresses this issue by optimizing risk\-aware objectives rather than expectation alone\[[31](https://arxiv.org/html/2606.27766#bib.bib15),[19](https://arxiv.org/html/2606.27766#bib.bib32)\]\. Among these approaches, distributional RL provides a principled foundation by explicitly modeling the full return distribution, thereby enabling tail\-aware criteria such as Value\-at\-Risk and Conditional Value\-at\-Risk \(CVaR\)\[[2](https://arxiv.org/html/2606.27766#bib.bib18),[3](https://arxiv.org/html/2606.27766#bib.bib44),[6](https://arxiv.org/html/2606.27766#bib.bib3)\]\. However, existing risk\-sensitive offline RL methods are predominantly built on actor\-critic or policy\-optimization pipelines, and relatively few works have explored how to endow diffusion\-based planners with explicit and controllable risk sensitivity\[[4](https://arxiv.org/html/2606.27766#bib.bib5),[18](https://arxiv.org/html/2606.27766#bib.bib9),[32](https://arxiv.org/html/2606.27766#bib.bib10)\]\.

In this paper, we proposeRS\-Diffuser, a risk\-sensitive offline diffusion planning framework that combines diffusion\-based trajectory generation with distributional value learning\. RS\-Diffuser learns a diffusion planner over future state trajectories and a separate inverse dynamics model for action recovery\. Its key component is a*distributional value critic*, trained with quantile regression to predict the full return distribution of candidate plans rather than only their expected value\. At inference time, instead of retraining the planner for a fixed risk preference, we inject a risk\-sensitive guidance signal into the diffusion denoising process, where gradients derived from tail\-aware objectives such as CVaR steer generation toward desired risk profiles\. As a result, a single trained model can realize risk\-averse, risk\-neutral, or risk\-seeking behaviors by changing only the inference\-time risk parameter\.

Our work makes three main contributions:

- ∙\\bulletWe introduce a diffusion planning framework for offline RL that explicitly incorporates distributional value modeling for tail\-aware decision making\.
- ∙\\bulletWe show how a distributional critic can be integrated into guided diffusion sampling, yielding flexible inference\-time control over risk preferences\.
- ∙\\bulletWe demonstrate on two risk\-sensitive benchmarks that RS\-Diffuser outperforms existing state\-of\-the\-art models\.

## 2Related Works

#### 2\.0\.1Diffusion Models for Decision Making\.

The application of diffusion models for decision making has gained significant traction due to their ability to represent complex, multi\-modal policy distributions\[[41](https://arxiv.org/html/2606.27766#bib.bib7)\]\. Diffusion planning represents a major branch of this research, where the model generates entire sequences of future states or actions\[[10](https://arxiv.org/html/2606.27766#bib.bib1)\]\. For example, Diffuser\[[10](https://arxiv.org/html/2606.27766#bib.bib1)\]utilizes a U\-Net architecture to iteratively denoise trajectories guided by a reward function\. Building on this, Decision Diffuser\[[1](https://arxiv.org/html/2606.27766#bib.bib2)\]frames decision\-making as a pure conditional generative task, removing the need for explicit reward gradients during sampling\. Recent studies have further optimized these architectures, exploring the benefits of Transformer\-based denoising networks and specialized sampling techniques\[[14](https://arxiv.org/html/2606.27766#bib.bib11),[5](https://arxiv.org/html/2606.27766#bib.bib20),[17](https://arxiv.org/html/2606.27766#bib.bib16)\]\. The applications of diffusion models in decision\-making span diverse domains, including autonomous driving\[[40](https://arxiv.org/html/2606.27766#bib.bib40),[37](https://arxiv.org/html/2606.27766#bib.bib35),[36](https://arxiv.org/html/2606.27766#bib.bib34),[11](https://arxiv.org/html/2606.27766#bib.bib36)\]and sports decision\-making\[[29](https://arxiv.org/html/2606.27766#bib.bib41)\]\. While these methods excel at capturing the behavior distribution, they are inherently risk\-neutral\. They prioritize the most frequent or highest\-reward paths in the training set without a mechanism to mitigate the risk of the generated plans\.

#### 2\.0\.2Risk\-sensitive Reinforcement Learning\.

Risk\-sensitive RL aims to learn policies that account for the uncertainty and variability of returns\[[19](https://arxiv.org/html/2606.27766#bib.bib32),[15](https://arxiv.org/html/2606.27766#bib.bib6)\]\. A cornerstone of this field is Distributional RL, which models the entire distribution of the random return variable instead of just its expectation\[[6](https://arxiv.org/html/2606.27766#bib.bib3),[2](https://arxiv.org/html/2606.27766#bib.bib18)\]\. Bellemare et al\.\[[2](https://arxiv.org/html/2606.27766#bib.bib18)\]introduced the categorical distribution approach, while Dabney et al\.\[[6](https://arxiv.org/html/2606.27766#bib.bib3)\]enhanced this with Quantile Regression DQN \(QR\-DQN\), allowing for more flexible and accurate distribution modeling\. To achieve risk\-aversion, researchers often optimize metrics such as the Conditional Value\-at\-Risk \(CVaR\), which focuses on the lower tail of the return distribution\[[33](https://arxiv.org/html/2606.27766#bib.bib45),[16](https://arxiv.org/html/2606.27766#bib.bib49)\]\. In the offline setting, risk\-aversion is particularly crucial as it provides a natural defense against the epistemic uncertainty arising from limited data\[[4](https://arxiv.org/html/2606.27766#bib.bib5)\]\. However, traditional risk\-sensitive offline RL methods are often built upon standard actor\-critic architectures, which can suffer from limited policy expressivity and training instabilities in complex environments\. It is worth noting that another line of research focuses on constrained learning from demonstrations, namely Inverse Constrained Reinforcement Learning \(ICRL\)\[[31](https://arxiv.org/html/2606.27766#bib.bib15),[30](https://arxiv.org/html/2606.27766#bib.bib39),[13](https://arxiv.org/html/2606.27766#bib.bib38),[39](https://arxiv.org/html/2606.27766#bib.bib33),[35](https://arxiv.org/html/2606.27766#bib.bib51),[34](https://arxiv.org/html/2606.27766#bib.bib37)\], which aims to infer latent safety constraints from expert demonstrations and incorporate them into policy optimization\. Our work departs from these by embedding distributional risk\-awareness directly into the generative diffusion process, leveraging the expressive power of diffusion models\.

## 3Problem Formulation

#### 3\.0\.1Markov Decision Process \(MDP\)\.

The agent optimizes the control policy under a Markov Decision Process \(MDP\), defined by a tuple\(𝒮,𝒜,P,r,γ\)\(\\mathcal\{S\},\\mathcal\{A\},P,r,\\gamma\), where𝒮\\mathcal\{S\}and𝒜\\mathcal\{A\}denote the state and action spaces,P​\(s′∣s,a\)P\(s^\{\\prime\}\\mid s,a\)is the transition dynamics,r​\(s,a\)r\(s,a\)is the reward function, andγ∈\(0,1\)\\gamma\\in\(0,1\)is the discount factor\. At each time step, the agent selects an action according to a policyπ​\(a∣s\)\\pi\(a\\mid s\), receives a reward, and transitions to the next state\. The objective is to learn a policy that maximizes the expected discounted return\. In this work, we consider the offline RL setting, where the policy is learned solely from a fixed dataset without further interaction with the environment\.

#### 3\.0\.2Diffusion Planner\.

Let𝐱0=\(s0,a0,s1,a1,…,sH−1,aH−1,sH\)\\mathbf\{x\}\_\{0\}=\(s\_\{0\},a\_\{0\},s\_\{1\},a\_\{1\},\\dots,s\_\{H\-1\},a\_\{H\-1\},s\_\{H\}\)denote a trajectory segment of horizonHHsampled from an offline dataset, wherest∈𝒮s\_\{t\}\\in\\mathcal\{S\}andat∈𝒜a\_\{t\}\\in\\mathcal\{A\}\. A diffusion planner\[[10](https://arxiv.org/html/2606.27766#bib.bib1)\]models a conditional distribution over trajectories asp​\(𝐱0∣𝐜\)p\(\\mathbf\{x\}\_\{0\}\\mid\\mathbf\{c\}\), where𝐜\\mathbf\{c\}denotes the planning condition, such as the current state, a goal, or a target return\. Following Denoising Diffusion Probabilistic Models \(DDPM\)\[[9](https://arxiv.org/html/2606.27766#bib.bib28)\], the clean trajectory𝐱0\\mathbf\{x\}\_\{0\}is gradually corrupted into a noisy version𝐱k\\mathbf\{x\}^\{k\}over diffusion stepsk=1,…,Kk=1,\\dots,K, and a denoising network is trained to reverse this process111In this work, we use superscripts \(k∈\{0,1,…,K\}k\\in\\\{0,1,\.\.\.,K\\\}\) to denote diffusion timesteps and subscripts \(t∈\{0,1,…,T\}t\\in\\\{0,1,\.\.\.,T\\\}\) to denote trajectory timesteps\. Specifically, the learned reverse dynamics are parameterized aspθ​\(𝐱k−1∣𝐱k,𝐜\)=𝒩​\(μθ​\(𝐱k,k,𝐜\),Σk\)p\_\{\\theta\}\(\\mathbf\{x\}^\{k\-1\}\\mid\\mathbf\{x\}^\{k\},\\mathbf\{c\}\)=\\mathcal\{N\}\\\!\\big\(\\mu\_\{\\theta\}\(\\mathbf\{x\}^\{k\},k,\\mathbf\{c\}\),\\Sigma\_\{k\}\\big\), whereμθ\\mu\_\{\\theta\}is a denoising neural network andΣk\\Sigma\_\{k\}is a predefined covariance\. Training minimizes the standard denoising objective\[[9](https://arxiv.org/html/2606.27766#bib.bib28)\]:

ℒDDPM​\(θ\)=𝔼𝐱0,ϵ,k​\[‖ϵ−ϵθ​\(𝐱k,k,𝐜\)‖22\],\\mathcal\{L\}\_\{\\mathrm\{DDPM\}\}\(\\theta\)=\\mathbb\{E\}\_\{\\mathbf\{x\}^\{0\},\\epsilon,k\}\\left\[\\left\\\|\\epsilon\-\\epsilon\_\{\\theta\}\(\\mathbf\{x\}^\{k\},k,\\mathbf\{c\}\)\\right\\\|\_\{2\}^\{2\}\\right\],\(1\)where𝐱k=α¯k​𝐱0\+1−α¯k​ϵ\\mathbf\{x\}^\{k\}=\\sqrt\{\\bar\{\\alpha\}\_\{k\}\}\\mathbf\{x\}^\{0\}\+\\sqrt\{1\-\\bar\{\\alpha\}\_\{k\}\}\\epsilondenotes the corrupted data andϵ∼𝒩​\(0,𝐈\)\\epsilon\\sim\\mathcal\{N\}\(0,\\mathbf\{I\}\)denotes the sampled noise\. At inference time, the planner starts from Gaussian noise and iteratively applies the learned reverse process viaμθ\\mu\_\{\\theta\}to recover a clean trajectory respecting𝐜\\mathbf\{c\}\.

#### 3\.0\.3Guided Sampling\.

To bias generation toward a task objectiveyy, we introduce a differentiable guidance model defined on the noisy trajectory, written aspϕ​\(y∣𝐱k,𝐜,k\)p\_\{\\phi\}\(y\\mid\\mathbf\{x\}^\{k\},\\mathbf\{c\},k\)\. Using the standard guidance approximation\[[23](https://arxiv.org/html/2606.27766#bib.bib14),[40](https://arxiv.org/html/2606.27766#bib.bib40),[29](https://arxiv.org/html/2606.27766#bib.bib41)\], the reverse generation step can be steered toward the desired objective as follows:

pθ,ϕ​\(𝐱k−1∣𝐱k,𝐜,y\)≈𝒩​\(μθ​\(𝐱k,k,𝐜\)\+w​Σk​gk,Σk\),p\_\{\\theta,\\phi\}\(\\mathbf\{x\}^\{k\-1\}\\mid\\mathbf\{x\}^\{k\},\\mathbf\{c\},y\)\\approx\\mathcal\{N\}\\\!\\left\(\\mu\_\{\\theta\}\(\\mathbf\{x\}^\{k\},k,\\mathbf\{c\}\)\+w\\,\\Sigma\_\{k\}\\,g\_\{k\},\\,\\Sigma\_\{k\}\\right\),\(2\)wherew\>0w\>0controls the guidance strength andgk=∇𝐱klog⁡pϕ​\(y∣𝐱k,𝐜,k\)g\_\{k\}=\\nabla\_\{\\mathbf\{x\}^\{k\}\}\\log p\_\{\\phi\}\(y\\mid\\mathbf\{x\}^\{k\},\\mathbf\{c\},k\)represents the gradient of the classifier regarding the noisy sample𝐱k\\mathbf\{x\}^\{k\}\. By training the diffusion modelμθ\\mu\_\{\\theta\}and the guidance modelpϕp\_\{\\phi\}separately, the diffusion model learns a trajectory prior from offline data, while the guidance term steers the denoising process toward trajectories that better satisfy the desired objective\.

#### 3\.0\.4Distributional Value Modeling\.

In distributional RL\[[2](https://arxiv.org/html/2606.27766#bib.bib18),[3](https://arxiv.org/html/2606.27766#bib.bib44)\], the value is modeled as a random return rather than only its expectation\. For a trajectory𝐱\\mathbf\{x\}, letZ​\(𝐱\)Z\(\\mathbf\{x\}\)denote the random discounted return associated with𝐱\\mathbf\{x\}\. Rather than predicting only the expectation as previous offline RL methods\[[12](https://arxiv.org/html/2606.27766#bib.bib4),[28](https://arxiv.org/html/2606.27766#bib.bib29),[38](https://arxiv.org/html/2606.27766#bib.bib13)\], we model the full return distribution, which is essential for risk\-sensitive decision making\.

Following quantile\-based distributional RL\[[6](https://arxiv.org/html/2606.27766#bib.bib3),[31](https://arxiv.org/html/2606.27766#bib.bib15)\], we approximate the return distributionZ​\(𝐱\)Z\(\\mathbf\{x\}\)byNNquantile values\{zi​\(𝐱\)\}i=1N\\\{z\_\{i\}\(\\mathbf\{x\}\)\\\}\_\{i=1\}^\{N\}, associated with quantile fractions\{τi\}i=1N⊂\(0,1\)\\\{\\tau\_\{i\}\\\}\_\{i=1\}^\{N\}\\subset\(0,1\)\. Eachτi\\tau\_\{i\}specifies a probability level, and the corresponding quantile valuezi​\(𝐱\)z\_\{i\}\(\\mathbf\{x\}\)approximates the return threshold below which a fractionτi\\tau\_\{i\}of outcomes lie\. The resulting distribution is represented as a uniform mixture of Dirac masses located at these quantile valuesZ​\(𝐱\)≈1N​∑i=1Nδzi​\(𝐱\)Z\(\\mathbf\{x\}\)\\approx\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\delta\_\{z\_\{i\}\(\\mathbf\{x\}\)\}, whereδzi​\(𝐱\)\\delta\_\{z\_\{i\}\(\\mathbf\{x\}\)\}denotes a Dirac delta distribution centered atzi​\(𝐱\)z\_\{i\}\(\\mathbf\{x\}\)\. In practice, the quantile fractions\{τi\}\\\{\\tau\_\{i\}\\\}are typically chosen uniformly in\(0,1\)\(0,1\)\. This distributional critic enables us to compute tail aware risk measures, such as VaR\[[8](https://arxiv.org/html/2606.27766#bib.bib47)\]and CVaR\[[21](https://arxiv.org/html/2606.27766#bib.bib48)\], which can be used to guide diffusion sampling\.

## 4Method

### 4\.1Overview

Given an offline trajectory dataset𝒟\\mathcal\{D\}that many consists of state\-action\-reward pairs, our goal is to learn a diffusion planner that can generate high\-quality future plans while allowing explicit control over risk preferences at inference time\. To this end, RS\-Diffuser consists of three components: 1\) a diffusion planner that generates future state trajectories conditioned on the current state \(Section[4\.2](https://arxiv.org/html/2606.27766#S4.SS2)\), 2\) an inverse dynamics model that converts planned state transitions into executable actions \(Section[4\.3](https://arxiv.org/html/2606.27766#S4.SS3)\), and 3\) a distributional value critic that predicts the return distribution of candidate generated trajectories \(Section[4\.4](https://arxiv.org/html/2606.27766#S4.SS4)\)\.

At test time, the distributional critic provides a differentiable risk\-sensitive objective whose gradients guide the diffusion denoising process, allowing a single diffusion model to produce risk\-averse, risk\-neutral, or risk\-seeking plans by adjusting the risk parameter \(Section[4\.5](https://arxiv.org/html/2606.27766#S4.SS5)\)\.

### 4\.2State Trajectory Diffusion Planner

While the generic formulation in Sec\.[3](https://arxiv.org/html/2606.27766#S3)allows state\-action trajectories, our method instantiates the diffusion variable as a*future state plan*by following\[[1](https://arxiv.org/html/2606.27766#bib.bib2)\]\. Specifically, at decision timett, we define𝐱0=\(st\+1,st\+2,…,st\+H\)\\mathbf\{x\}^\{0\}=\(s\_\{t\+1\},s\_\{t\+2\},\\dots,s\_\{t\+H\}\), which denotes a future state sequence of planning horizonHH, conditioned on the current statests\_\{t\}\. The planner therefore modelspθ​\(𝐱0∣st\)p\_\{\\theta\}\(\\mathbf\{x\}^\{0\}\\mid s\_\{t\}\)\.

This state\-only formulation decouples long\-horizon planning from low\-level action generation, allowing the diffusion model to focus on predicting feasible future state evolutions while delegating action recovery to a separate inverse dynamics model\. By adapting Equation[1](https://arxiv.org/html/2606.27766#S3.E1)to the state\-only setting, the plannerμθ\\mu\_\{\\theta\}is trained with the following denoising objective:

ℒplan​\(θ\)=𝔼𝐱0,ϵ,k​\[‖ϵ−ϵθ​\(𝐱k,k,st\)‖22\],\\mathcal\{L\}\_\{\\mathrm\{plan\}\}\(\\theta\)=\\mathbb\{E\}\_\{\\mathbf\{x\}^\{0\},\\epsilon,k\}\\left\[\\left\\\|\\epsilon\-\\epsilon\_\{\\theta\}\(\\mathbf\{x\}^\{k\},k,s\_\{t\}\)\\right\\\|\_\{2\}^\{2\}\\right\],\(3\)At inference time, the planner starts from Gaussian noise and iteratively denoises to produce a future state plan𝐱^0=\(s^t\+1,…,s^t\+H\)\\hat\{\\mathbf\{x\}\}^\{0\}=\(\\hat\{s\}\_\{t\+1\},\\dots,\\hat\{s\}\_\{t\+H\}\)\.

### 4\.3Inverse Dynamics Action Decoder

Because the planner operates in state space, we use a separate inverse dynamics model to recover executable actions from planned state transitions\. Specifically, we learn a deterministic decoderfψ​\(st,st\+1\)f\_\{\\psi\}\(s\_\{t\},s\_\{t\+1\}\)for continuous control\. In practice, given the current statests\_\{t\}and the predicted next states^t\+1\\hat\{s\}\_\{t\+1\}from the diffusion planner, the executed action is obtained asa^t=fψ​\(st,s^t\+1\)\\hat\{a\}\_\{t\}=f\_\{\\psi\}\(s\_\{t\},\\hat\{s\}\_\{t\+1\}\)\. The inverse dynamics model is trained on offline transitions using supervised regression\[[1](https://arxiv.org/html/2606.27766#bib.bib2)\]:

ℒinv​\(ψ\)=𝔼\(st,at,st\+1\)∼𝒟​\[‖at−fψ​\(st,st\+1\)‖22\]\.\\mathcal\{L\}\_\{\\mathrm\{inv\}\}\(\\psi\)=\\mathbb\{E\}\_\{\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\)\\sim\\mathcal\{D\}\}\\left\[\\\|a\_\{t\}\-f\_\{\\psi\}\(s\_\{t\},s\_\{t\+1\}\)\\\|\_\{2\}^\{2\}\\right\]\.\(4\)This design separates*what*future states should be reached from*how*to realize them, which is particularly suitable for diffusion planning over long horizons\.

### 4\.4Monte Carlo Distributional Value Critic

The core of RS\-Diffuser is a distributional value critic that estimates the return\-to\-go distribution of candidate plans\. Unlike standard risk\-neutral value guidance, which predicts only the expected return, our critic models the full return distribution and therefore supports tail\-aware risk measures\.

For each trajectory segment starting at time steptt, we compute the Monte Carlo discounted return\-to\-go as:

Rt=∑ℓ=0Te−t−1γℓ​rt\+ℓ,R\_\{t\}=\\sum\_\{\\ell=0\}^\{T\_\{e\}\-t\-1\}\\gamma^\{\\ell\}r\_\{t\+\\ell\},\(5\)whereγ∈\(0,1\]\\gamma\\in\(0,1\]is the discount factor andTeT\_\{e\}denotes the terminal index\.

To make the critic compatible with denoising\-time guidance, we condition it on the noisy plan𝐱k\\mathbf\{x\}^\{k\}, the current statests\_\{t\}, and the diffusion stepkk\. The distributional critic then predictsNNquantiles𝐳ϕ​\(𝐱k,st,k\)=\{ziϕ​\(𝐱k,st,k\)\}i=1N\\mathbf\{z\}\_\{\\phi\}\(\\mathbf\{x\}^\{k\},s\_\{t\},k\)=\\left\\\{z\_\{i\}^\{\\phi\}\(\\mathbf\{x\}^\{k\},s\_\{t\},k\)\\right\\\}\_\{i=1\}^\{N\}, corresponding to quantile fractions\{τi\}i=1N⊂\(0,1\)\\\{\{\\tau\}\_\{i\}\\\}\_\{i=1\}^\{N\}\\subset\(0,1\)\. Importantly, we adopt Monte Carlo supervision rather than Bellman bootstrapping, following the value\-learning paradigm in\[[14](https://arxiv.org/html/2606.27766#bib.bib11)\]\. Specifically, we train the critic by quantile regression using the Monte Carlo return sampleRtR\_\{t\}:

ℒdist​\(ϕ\)=𝔼\(𝐱0,st,Rt\)∼𝒟,ϵ,k​\[1N​∑i=1Nρτiκ​\(Rt−ziϕ​\(𝐱k,st,k\)\)\],\\mathcal\{L\}\_\{\\mathrm\{dist\}\}\(\\phi\)=\\mathbb\{E\}\_\{\(\\mathbf\{x\}^\{0\},s\_\{t\},R\_\{t\}\)\\sim\\mathcal\{D\},\\,\\epsilon,\\,k\}\\left\[\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\rho\_\{\{\\tau\}\_\{i\}\}^\{\\kappa\}\\\!\\left\(R\_\{t\}\-z\_\{i\}^\{\\phi\}\(\\mathbf\{x\}^\{k\},s\_\{t\},k\)\\right\)\\right\],\(6\)where𝒟\\mathcal\{D\}denotes the offline dataset andρτκ​\(⋅\)\\rho\_\{\\tau\}^\{\\kappa\}\(\\cdot\)is the quantile Huber loss\[[6](https://arxiv.org/html/2606.27766#bib.bib3)\]:

ρτκ​\(u\)=\|τ−𝕀​\(u<0\)\|​Lκ​\(u\),Lκ​\(u\)=\{12​u2/κ,\|u\|≤κ,\|u\|−12​κ,otherwise\.\\rho\_\{\\tau\}^\{\\kappa\}\(u\)=\\left\|\\tau\-\\mathbb\{I\}\(u<0\)\\right\|L\_\{\\kappa\}\(u\),\\quad L\_\{\\kappa\}\(u\)=\\begin\{cases\}\\frac\{1\}\{2\}u^\{2\}/\\kappa,&\|u\|\\leq\\kappa,\\\\\[5\.69054pt\] \|u\|\-\\frac\{1\}\{2\}\\kappa,&\\text\{otherwise\.\}\\end\{cases\}\(7\)
By learning the full return distribution instead of only its mean, the critic provides a natural basis for risk\-sensitive guidance during inference\.

### 4\.5Practical Implementation of RS\-Diffuser

#### 4\.5\.1Risk Sensitive Objective from Quantiles\.

Once the return distribution is available, different risk preferences can be expressed through different functionals of the predicted quantiles\. For a risk\-averse objective, we use the lower\-tail CVaR, approximated by averaging quantiles up to levelα\\alpha:

ραlow​\(𝐱k,st,k\)=CVaRα​\(𝐱k,st,k\)≈1\|ℐα\|​∑i∈ℐαziϕ​\(𝐱k,st,k\),\\rho\_\{\\alpha\}^\{\\mathrm\{low\}\}\(\\mathbf\{x\}^\{k\},s\_\{t\},k\)=\\mathrm\{CVaR\}\_\{\\alpha\}\(\\mathbf\{x\}^\{k\},s\_\{t\},k\)\\approx\\frac\{1\}\{\|\\mathcal\{I\}\_\{\\alpha\}\|\}\\sum\_\{i\\in\\mathcal\{I\}\_\{\\alpha\}\}z\_\{i\}^\{\\phi\}\(\\mathbf\{x\}^\{k\},s\_\{t\},k\),\(8\)whereℐα=\{i:τi≤α\}\\mathcal\{I\}\_\{\\alpha\}=\\\{i:\\tau\_\{i\}\\leq\\alpha\\\}\. Maximizing this objective encourages plans with favorable worst\-case outcomes\. For risk\-seeking behavior we use an upper\-tail objective:

ραup​\(𝐱k,st,k\)=1\|𝒥α\|​∑i∈𝒥αziϕ​\(𝐱k,st,k\),𝒥α=\{i:τi≥1−α\}\.\\rho\_\{\\alpha\}^\{\\mathrm\{up\}\}\(\\mathbf\{x\}\_\{k\},s\_\{t\},k\)=\\frac\{1\}\{\|\\mathcal\{J\}\_\{\\alpha\}\|\}\\sum\_\{i\\in\\mathcal\{J\}\_\{\\alpha\}\}z\_\{i\}^\{\\phi\}\(\\mathbf\{x\}^\{k\},s\_\{t\},k\),\\qquad\\mathcal\{J\}\_\{\\alpha\}=\\\{i:\\tau\_\{i\}\\geq 1\-\\alpha\\\}\.\(9\)

#### 4\.5\.2Risk Guided Diffusion Sampling

We now instantiate the generic guidance mechanism in Sec\.[3](https://arxiv.org/html/2606.27766#S3)with the distributional value critic\. Instead of using a conventional classifier, we treat the risk functionalρα​\(𝐱k,st,k\)\\rho\_\{\\alpha\}\(\\mathbf\{x\}^\{k\},s\_\{t\},k\)as a differentiable guidance objective defined on the noisy plan\. Its gradient with respect to the current denoising state is:

gk=∇𝐱kρα​\(𝐱k,st,k\),g\_\{k\}=\\nabla\_\{\\mathbf\{x\}^\{k\}\}\\rho\_\{\\alpha\}\(\\mathbf\{x\}^\{k\},s\_\{t\},k\),\(10\)whereρα\\rho\_\{\\alpha\}denotes the used risk measure on the distributional critic\. The guided reverse process can then be calculated by Eq\.[2](https://arxiv.org/html/2606.27766#S3.E2)\. Intuitively, the critic\-derived gradient biases denoising toward trajectories that match the desired risk profile\.

Algorithm 1RS\-Diffuser0:Offline dataset

𝒟\\mathcal\{D\}, planning horizon

HH, diffusion steps

KK, quantile numbers

NN, risk objective

ρα\\rho\_\{\\alpha\}
1:Calculate accumulated discounted returns for every step

ttvia Equation[5](https://arxiv.org/html/2606.27766#S4.E5)\.

2:Offline training

3:foreach training iterationdo

4:Sample

\(st,𝐱0,Rt\)\(s\_\{t\},\\mathbf\{x\}^\{0\},R\_\{t\}\)from

𝒟\\mathcal\{D\}, where

𝐱0=\(st\+1,…,st\+H\)\\mathbf\{x\}^\{0\}=\(s\_\{t\+1\},\\dots,s\_\{t\+H\}\)
5:Update diffusion planner

ϵθ\\epsilon\_\{\\theta\}with denoising loss

ℒplan\\mathcal\{L\}\_\{\\mathrm\{plan\}\}via Equation[1](https://arxiv.org/html/2606.27766#S3.E1)

6:Update inverse dynamics model

fψf\_\{\\psi\}with loss

ℒinv\\mathcal\{L\}\_\{\\mathrm\{inv\}\}via Equation[4](https://arxiv.org/html/2606.27766#S4.E4)

7:Update distributional critic

zϕz\_\{\\phi\}with quantile loss

ℒdist\\mathcal\{L\}\_\{\\mathrm\{dist\}\}via Equation[6](https://arxiv.org/html/2606.27766#S4.E6)

8:endfor

9:Risk\-guided planning at inference

10:foreach decision stepdo

11:Observe current state

sts\_\{t\}and initialize

𝐱K∼𝒩​\(0,𝐈\)\\mathbf\{x\}^\{K\}\\sim\\mathcal\{N\}\(0,\\mathbf\{I\}\)
12:for

k=K,K−1,…,1k=K,K\-1,\\dots,1do

13:Compute risk guidance

gk=∇𝐱kρα​\(𝐱k,st,k\)g\_\{k\}=\\nabla\_\{\\mathbf\{x\}^\{k\}\}\\rho\_\{\\alpha\}\(\\mathbf\{x\}^\{k\},s\_\{t\},k\)using

zϕz\_\{\\phi\}via Equation[10](https://arxiv.org/html/2606.27766#S4.E10)

14:Perform guided reverse diffusion to obtain

𝐱k−1\\mathbf\{x\}^\{k\-1\}
15:endfor

16:Obtain planned states

𝐱^0=\(s^t\+1,…,s^t\+H\)\\hat\{\\mathbf\{x\}\}^\{0\}=\(\\hat\{s\}\_\{t\+1\},\\dots,\\hat\{s\}\_\{t\+H\}\)
17:Decode action

a^t=fψ​\(st,s^t\+1\)\\hat\{a\}\_\{t\}=f\_\{\\psi\}\(s\_\{t\},\\hat\{s\}\_\{t\+1\}\)and execute it

18:endfor

#### 4\.5\.3Remarks\.

It is worth noting that this design is a key advantage of RS\-Diffuser: the planner is trained only once, while risk preference is entirely controlled at inference time through the choice ofρα\\rho\_\{\\alpha\}applied to the distributional value critic\. As a result, a single model can seamlessly switch between conservative, neutral, and aggressive behaviors according to different requirements without retraining\. Algorithm[1](https://arxiv.org/html/2606.27766#alg1)summarizes the overall pipeline of RS\-Diffuser, including offline training of the planner, inverse dynamics model, and distributional critic, as well as risk\-guided planning at inference time\.

## 5Experiments

In this section, we empirically evaluate the proposed RS\-Diffuser against state\-of\-the\-art methods on risk\-sensitive benchmarks, including risk\-sensitive D4RL \(Section[5\.2](https://arxiv.org/html/2606.27766#S5.SS2)\) and risky robot navigation \(Section[5\.3](https://arxiv.org/html/2606.27766#S5.SS3)\)\. To further assess robustness, we conduct ablation studies on key hyperparameters \(Section[5\.4](https://arxiv.org/html/2606.27766#S5.SS4)\)\.

### 5\.1Experimental Setup

#### 5\.1\.1Model Configurations

We implement our method using CleanDiffuser\[[7](https://arxiv.org/html/2606.27766#bib.bib8)\]\. The planner uses a 2\-block Transformer \(hidden size 256\), and the inverse dynamics model is an MLP \(hidden size 256\)\. We adopt DDIM\[[24](https://arxiv.org/html/2606.27766#bib.bib42)\]with 20 steps for planning and DDPM\[[9](https://arxiv.org/html/2606.27766#bib.bib28)\]with 10 steps for inverse dynamics\. Models are trained for 1M steps with batch size 256\. The planning horizon is 32 with discount factor 0\.99\. We use Adam optimizer with a learning rate 3e\-4\.

#### 5\.1\.2Baseline Methods

We compare with risk neutral baselines, Conservative Q Learning \(CQL\[[12](https://arxiv.org/html/2606.27766#bib.bib4)\]\) and Diffusion QL\[[28](https://arxiv.org/html/2606.27766#bib.bib29)\], as well as risk sensitive methods: 1\) OWCPG\[[25](https://arxiv.org/html/2606.27766#bib.bib19)\], optimizing worst case \(e\.g\., CVaR\) returns; 2\) ORAAC\[[26](https://arxiv.org/html/2606.27766#bib.bib50)\], a distributional critic with an imitation based policy; 3\) CODAC\[[16](https://arxiv.org/html/2606.27766#bib.bib49)\], conservative offline RL with a distributional critic; 4\) UDAC\[[4](https://arxiv.org/html/2606.27766#bib.bib5)\], an uncertainty aware actor critic with a diffusion policy\.

#### 5\.1\.3Evaluation Metric

Each method is evaluated over 100 test episodes\. We report episodic return \(cumulative reward per episode\) as the main metric, including both the mean and CVaR0\.1\(average of the worst 10 episodes\)\. For risky navigation tasks, we also report episodic violations, defined as the total number of time steps spent in risky regions\. Results are averaged over five random seeds and reported as mean±\\pmstandard deviation\.

### 5\.2Risk\-sensitive D4RL

#### 5\.2\.1Task Description

The risk\-sensitive D4RL benchmark\[[26](https://arxiv.org/html/2606.27766#bib.bib50)\]introduces stochastic penalties in MuJoCo to model safety\-critical failures\. Rewards are reduced by large penalties when agents enter undesirable states, such as exceeding velocity thresholds in Half\-Cheetah or violating stability constraints \(e\.g\., large pitch angles\) in Walker2D and Hopper\. This creates rare but severe negative outcomes, requiring policies to balance performance and safety\. The maximum episode length is 500 steps\.

Table 1:Episodic return on risk\-sensitive D4RL tasks\. Higher is better for both CVaR0\.1and Mean metrics\.
#### 5\.2\.2Results Analysis

Table[1](https://arxiv.org/html/2606.27766#S5.T1)shows the results\. We find that RS\-Diffuser achieves the best overall performance on the risk\-sensitive D4RL benchmark, outperforming both risk\-neutral and risk\-sensitive baselines across most settings\. This indicates not only stronger average performance but also improved robustness under worst\-case scenarios\. In contrast, risk\-neutral methods such as CQL perform poorly, as optimizing only the expected return fails to capture tail risks in safety\-critical environments\. Diffusion\-QL achieves moderately better performance, benefiting from the expressive generative capacity of diffusion models, but still lacks explicit risk awareness\. Risk\-sensitive baselines such as UDAC demonstrate relatively strong performance\. However, they are typically trained to optimize a fixed risk objective, limiting their adaptability across different scenarios\. In contrast, RS\-Diffuser enables flexible control over risk preferences at inference time, allowing a single model to adapt to different risk measures and consistently achieve strong performance\. Overall, these results demonstrate that guiding diffusion planning with a distributional critic is an effective approach for improving tail\-aware decision making in offline RL\.

### 5\.3Risky Robot Navigation

#### 5\.3\.1Task Description

Risky robot navigation benchmark was introduced in\[[16](https://arxiv.org/html/2606.27766#bib.bib49)\]\. It was evaluated on two challenging navigation tasks: Risky PointMass and Risky Ant\. In both environments, the agent starts from a randomly initialized state and aims to reach a target location efficiently\.

![Refer to caption](https://arxiv.org/html/2606.27766v1/riskyant.png)Figure 1:Risky Ant\.However, the direct path to the goal passes through a hazardous region that imposes significant penalties\. While risk\-neutral policies tend to take the shorter but hazardous route, risk\-sensitive policies are expected to avoid the hazardous area and instead favor safer trajectories\. Figure[1](https://arxiv.org/html/2606.27766#S5.F1)shows an illustrative example of Risky Ant\.

Table 2:Episodic return and violations on risky robot navigation tasks\. For return metrics \(Mean, Median, and CVaR0\.1\), higher is better\. For violations, lower is better\.
#### 5\.3\.2Results Analysis

Table[2](https://arxiv.org/html/2606.27766#S5.T2)shows the results\. In the Risky PointMass environment, CODAC, UDAC, and the proposed RS\-Diffuser all achieve zero violations while maintaining high returns, reflecting the relative simplicity of controlling a point mass system\. Despite the absence of violations across these methods, RS\-Diffuser consistently achieves the highest performance in terms of Mean, Median, and CVaR0\.1, indicating superior overall return and stronger tail performance\. In the more challenging Risky Ant environment, RS\-Diffuser attains the lowest number of violations while simultaneously achieving the highest returns\. This demonstrates its ability to effectively balance safety and performance\. The results highlight the advantage of combining a distributional value critic for risk\-aware evaluation with the expressive power of diffusion models for modeling complex transitions\.

### 5\.4Ablation Study on Hyperparameters

#### 5\.4\.1Task Description

To further evaluate the robustness of the proposed method, we conduct an ablation study on the Half\-Cheetah environment by varying the guidance measureρα\\rho\_\{\\alpha\}\. Specifically, we consider both VaR and CVaR as risk measures and systematically vary the parameterα\\alphato model different risk preferences\. Smaller values ofα\\alphacorrespond to more risk\-averse behavior, while larger values encourage more risk\-seeking behavior\.

Table 3:Ablation on risk\-sensitive Half\-Cheetah with varying risk measuresρα\\rho\_\{\\alpha\}at inference time\. CVaR0\.1is used as the default in the main experiments\.
#### 5\.4\.2Results Analysis

Table[3](https://arxiv.org/html/2606.27766#S5.T3)studies the effect of different inference\-time risk measures on Half\-Cheetah\. As expected, smallerα\\alphaleads to more risk\-averse behavior, yielding higher CVaR0\.1but lower mean return, while largerα\\alphaencourages more risk\-seeking behavior, improving the mean at the cost of weaker tail performance\. Comparing VaR and CVaR, we observe that CVaR\-based guidance generally provides stronger worst\-case performance, whereas VaR\-based guidance is slightly more aggressive and can achieve higher mean returns for largerα\\alpha\. Among all settings, CVaR0\.1offers the best overall trade\-off between average return and robustness, which justifies its use as the default choice\.

## 6Conclusion

We presentedRS\-Diffuser, a risk\-sensitive offline diffusion planning framework that integrates diffusion\-based trajectory generation with distributional value critics\. By guiding the denoising process with gradients derived from tail\-aware risk measures such as CVaR, our method enables a single trained model to flexibly realize risk\-averse, risk\-neutral, and risk\-seeking behaviors at inference time\. Experiments on risk\-sensitive D4RL and risky robot navigation benchmarks demonstrate that RS\-Diffuser achieves strong performance, improved robustness, and fewer safety violations compared to prior methods\. These results highlight the effectiveness of combining diffusion planning with distributional value guidance for risk\-sensitive decision making\. A promising direction for future work is to extend RS\-Diffuser to more challenging settings, including higher\-dimensional real\-world robotics tasks, richer uncertainty\-aware critics, and multi\-objective safety constraints\. We hope this work provides a useful step toward flexible and reliable risk\-sensitive decision\-making with diffusion models\.

## References

- \[1\]A\. Ajay, Y\. Du, A\. Gupta, J\. Tenenbaum, T\. Jaakkola, and P\. Agrawal\(2022\)Is conditional generative modeling all you need for decision\-making?\.arXiv preprint arXiv:2211\.15657\.Cited by:[§2\.0\.1](https://arxiv.org/html/2606.27766#S2.SS0.SSS1.p1.1),[§4\.2](https://arxiv.org/html/2606.27766#S4.SS2.p1.5),[§4\.3](https://arxiv.org/html/2606.27766#S4.SS3.p1.4)\.
- \[2\]M\. G\. Bellemare, W\. Dabney, and R\. Munos\(2017\)A distributional perspective on reinforcement learning\.InInternational conference on machine learning,pp\. 449–458\.Cited by:[§1](https://arxiv.org/html/2606.27766#S1.p3.1),[§2\.0\.2](https://arxiv.org/html/2606.27766#S2.SS0.SSS2.p1.1),[§3\.0\.4](https://arxiv.org/html/2606.27766#S3.SS0.SSS4.p1.3)\.
- \[3\]M\. G\. Bellemare, W\. Dabney, and M\. Rowland\(2023\)Distributional reinforcement learning\.MIT Press\.Cited by:[§1](https://arxiv.org/html/2606.27766#S1.p3.1),[§3\.0\.4](https://arxiv.org/html/2606.27766#S3.SS0.SSS4.p1.3)\.
- \[4\]X\. Chen, S\. Wang, T\. Yu, and L\. Yao\(2025\)Diffusion policies for risk\-averse behavior modeling in offline reinforcement learning\.In2025 IEEE/RSJ International Conference on Intelligent Robots and Systems \(IROS\),pp\. 567–574\.Cited by:[§1](https://arxiv.org/html/2606.27766#S1.p3.1),[§2\.0\.2](https://arxiv.org/html/2606.27766#S2.SS0.SSS2.p1.1),[§5\.1\.2](https://arxiv.org/html/2606.27766#S5.SS1.SSS2.p1.1)\.
- \[5\]Z\. Chen, X\. Long, L\. Zhang, and W\. Cai\(2025\)Toward diffusion\-based deep reinforcement learning for discrete decision\-making: methods and evaluations\.IEEE Transactions on Network Science and Engineering\.Cited by:[§2\.0\.1](https://arxiv.org/html/2606.27766#S2.SS0.SSS1.p1.1)\.
- \[6\]W\. Dabney, M\. Rowland, M\. Bellemare, and R\. Munos\(2018\)Distributional reinforcement learning with quantile regression\.InProceedings of the AAAI conference on artificial intelligence,Vol\.32\.Cited by:[§1](https://arxiv.org/html/2606.27766#S1.p3.1),[§2\.0\.2](https://arxiv.org/html/2606.27766#S2.SS0.SSS2.p1.1),[§3\.0\.4](https://arxiv.org/html/2606.27766#S3.SS0.SSS4.p2.12),[§4\.4](https://arxiv.org/html/2606.27766#S4.SS4.p3.9)\.
- \[7\]Z\. Dong, Y\. Yuan, J\. Hao, F\. Ni, Y\. Ma, P\. Li, and Y\. Zheng\(2024\)Cleandiffuser: an easy\-to\-use modularized library for diffusion models in decision making\.Advances in Neural Information Processing Systems37,pp\. 86899–86926\.Cited by:[§5\.1\.1](https://arxiv.org/html/2606.27766#S5.SS1.SSS1.p1.1)\.
- \[8\]D\. Duffie, J\. Pan,et al\.\(1997\)An overview of value at risk\.Journal of derivatives4\(3\),pp\. 7–49\.Cited by:[§3\.0\.4](https://arxiv.org/html/2606.27766#S3.SS0.SSS4.p2.12)\.
- \[9\]J\. Ho, A\. Jain, and P\. Abbeel\(2020\)Denoising diffusion probabilistic models\.Advances in neural information processing systems33,pp\. 6840–6851\.Cited by:[§1](https://arxiv.org/html/2606.27766#S1.p2.1),[§3\.0\.2](https://arxiv.org/html/2606.27766#S3.SS0.SSS2.p1.12),[§5\.1\.1](https://arxiv.org/html/2606.27766#S5.SS1.SSS1.p1.1)\.
- \[10\]M\. Janner, Y\. Du, J\. B\. Tenenbaum, and S\. Levine\(2022\)Planning with diffusion for flexible behavior synthesis\.arXiv preprint arXiv:2205\.09991\.Cited by:[§1](https://arxiv.org/html/2606.27766#S1.p2.1),[§2\.0\.1](https://arxiv.org/html/2606.27766#S2.SS0.SSS1.p1.1),[§3\.0\.2](https://arxiv.org/html/2606.27766#S3.SS0.SSS2.p1.12)\.
- \[11\]R\. Jiao, J\. Zhang, C\. Li, and L\. Hu\(2026\)Large\-kernel spatially parallel feature fusion for monocular 3d perception in autonomous driving\.Knowledge\-Based Systems343,pp\. 115998\.Cited by:[§2\.0\.1](https://arxiv.org/html/2606.27766#S2.SS0.SSS1.p1.1)\.
- \[12\]A\. Kumar, A\. Zhou, G\. Tucker, and S\. Levine\(2020\)Conservative q\-learning for offline reinforcement learning\.Advances in neural information processing systems33,pp\. 1179–1191\.Cited by:[§3\.0\.4](https://arxiv.org/html/2606.27766#S3.SS0.SSS4.p1.3),[§5\.1\.2](https://arxiv.org/html/2606.27766#S5.SS1.SSS2.p1.1)\.
- \[13\]G\. Liu, S\. Xu, S\. Liu, A\. Gaurav, S\. G\. Subramanian, and P\. Poupart\(2024\)A comprehensive survey on inverse constrained reinforcement learning: definitions, progress and challenges\.arXiv preprint arXiv:2409\.07569\.Cited by:[§2\.0\.2](https://arxiv.org/html/2606.27766#S2.SS0.SSS2.p1.1)\.
- \[14\]H\. Lu, D\. Han, Y\. Shen, and D\. Li\(2025\)What makes a good diffusion planner for decision making?\.InInternational conference on learning representations,Cited by:[§2\.0\.1](https://arxiv.org/html/2606.27766#S2.SS0.SSS1.p1.1),[§4\.4](https://arxiv.org/html/2606.27766#S4.SS4.p3.7)\.
- \[15\]X\. Ma, J\. Chen, L\. Xia, J\. Yang, Q\. Zhao, and Z\. Zhou\(2025\)Dsac: distributional soft actor\-critic for risk\-sensitive reinforcement learning\.Journal of Artificial Intelligence Research83\.Cited by:[§2\.0\.2](https://arxiv.org/html/2606.27766#S2.SS0.SSS2.p1.1)\.
- \[16\]Y\. Ma, D\. Jayaraman, and O\. Bastani\(2021\)Conservative offline distributional reinforcement learning\.Advances in neural information processing systems34,pp\. 19235–19247\.Cited by:[§2\.0\.2](https://arxiv.org/html/2606.27766#S2.SS0.SSS2.p1.1),[§5\.1\.2](https://arxiv.org/html/2606.27766#S5.SS1.SSS2.p1.1),[§5\.3\.1](https://arxiv.org/html/2606.27766#S5.SS3.SSS1.p1.1)\.
- \[17\]M\. A\. Mejia, L\. H\. Macedo, T\. Pinto, and J\. F\. Franco\(2025\)Integrating a spatio\-temporal diffusion model with a multi\-criteria decision\-making approach for optimal planning of electric vehicle charging infrastructure\.Applied Energy395,pp\. 126160\.Cited by:[§2\.0\.1](https://arxiv.org/html/2606.27766#S2.SS0.SSS1.p1.1)\.
- \[18\]R\. Miao, S\. Xu, R\. Zhao, W\. K\. V\. Chan, and G\. Liu\(2025\)Uncertainty\-aware preference alignment for diffusion policies\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2606.27766#S1.p3.1)\.
- \[19\]E\. Noorani, C\. N\. Mavridis, and J\. S\. Baras\(2025\)Risk\-sensitive reinforcement learning with exponential criteria\.IEEE Transactions on Cybernetics\.Cited by:[§1](https://arxiv.org/html/2606.27766#S1.p3.1),[§2\.0\.2](https://arxiv.org/html/2606.27766#S2.SS0.SSS2.p1.1)\.
- \[20\]R\. F\. Prudencio, M\. R\. Maximo, and E\. L\. Colombini\(2023\)A survey on offline reinforcement learning: taxonomy, review, and open problems\.IEEE transactions on neural networks and learning systems35\(8\),pp\. 10237–10257\.Cited by:[§1](https://arxiv.org/html/2606.27766#S1.p1.1)\.
- \[21\]R\. T\. Rockafellar, S\. Uryasev,et al\.\(2000\)Optimization of conditional value\-at\-risk\.Journal of risk2,pp\. 21–42\.Cited by:[§3\.0\.4](https://arxiv.org/html/2606.27766#S3.SS0.SSS4.p2.12)\.
- \[22\]Y\. Shen, M\. J\. Tobia, T\. Sommer, and K\. Obermayer\(2014\)Risk\-sensitive reinforcement learning\.Neural computation26\(7\),pp\. 1298–1328\.Cited by:[§1](https://arxiv.org/html/2606.27766#S1.p3.1)\.
- \[23\]J\. Sohl\-Dickstein, E\. Weiss, N\. Maheswaranathan, and S\. Ganguli\(2015\)Deep unsupervised learning using nonequilibrium thermodynamics\.InInternational conference on machine learning,pp\. 2256–2265\.Cited by:[§3\.0\.3](https://arxiv.org/html/2606.27766#S3.SS0.SSS3.p1.2)\.
- \[24\]J\. Song, C\. Meng, and S\. Ermon\(2020\)Denoising diffusion implicit models\.arXiv preprint arXiv:2010\.02502\.Cited by:[§5\.1\.1](https://arxiv.org/html/2606.27766#S5.SS1.SSS1.p1.1)\.
- \[25\]Y\. C\. Tang, J\. Zhang, and R\. Salakhutdinov\(2019\)Worst cases policy gradients\.arXiv preprint arXiv:1911\.03618\.Cited by:[§5\.1\.2](https://arxiv.org/html/2606.27766#S5.SS1.SSS2.p1.1)\.
- \[26\]N\. A\. Urpí, S\. Curi, and A\. Krause\(2021\)Risk\-averse offline reinforcement learning\.arXiv preprint arXiv:2102\.05371\.Cited by:[§5\.1\.2](https://arxiv.org/html/2606.27766#S5.SS1.SSS2.p1.1),[§5\.2\.1](https://arxiv.org/html/2606.27766#S5.SS2.SSS1.p1.1)\.
- \[27\]X\. Wang, S\. Wang, X\. Liang, D\. Zhao, J\. Huang, X\. Xu, B\. Dai, and Q\. Miao\(2022\)Deep reinforcement learning: a survey\.IEEE Transactions on Neural Networks and Learning Systems35\(4\),pp\. 5064–5078\.Cited by:[§1](https://arxiv.org/html/2606.27766#S1.p1.1)\.
- \[28\]Z\. Wang, J\. J\. Hunt, and M\. Zhou\(2022\)Diffusion policies as an expressive policy class for offline reinforcement learning\.arXiv preprint arXiv:2208\.06193\.Cited by:[§1](https://arxiv.org/html/2606.27766#S1.p2.1),[§3\.0\.4](https://arxiv.org/html/2606.27766#S3.SS0.SSS4.p1.3),[§5\.1\.2](https://arxiv.org/html/2606.27766#S5.SS1.SSS2.p1.1)\.
- \[29\]S\. Xu, G\. Liu, T\. Kharrat, Y\. Luo, M\. Aloulou, J\. L\. Peña, K\. Sofeikov, A\. Reid, P\. Roberts, S\. Spencer,et al\.\(2026\)TacticGen: grounding adaptable and scalable generation of football tactics\.arXiv preprint arXiv:2604\.18210\.Cited by:[§2\.0\.1](https://arxiv.org/html/2606.27766#S2.SS0.SSS1.p1.1),[§3\.0\.3](https://arxiv.org/html/2606.27766#S3.SS0.SSS3.p1.2)\.
- \[30\]S\. Xu and G\. Liu\(2024\)Robust inverse constrained reinforcement learning under model misspecification\.\.InICML,Cited by:[§2\.0\.2](https://arxiv.org/html/2606.27766#S2.SS0.SSS2.p1.1)\.
- \[31\]S\. Xu and G\. Liu\(2024\)Uncertainty\-aware constraint inference in inverse constrained reinforcement learning\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 17792–17816\.Cited by:[§1](https://arxiv.org/html/2606.27766#S1.p3.1),[§2\.0\.2](https://arxiv.org/html/2606.27766#S2.SS0.SSS2.p1.1),[§3\.0\.4](https://arxiv.org/html/2606.27766#S3.SS0.SSS4.p2.12)\.
- \[32\]S\. Xu, B\. Yue, H\. Zha, and G\. Liu\(2025\)A distributional approach to uncertainty\-aware preference alignment using offline demonstrations\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 3024–3049\.Cited by:[§1](https://arxiv.org/html/2606.27766#S1.p3.1)\.
- \[33\]C\. Ying, X\. Zhou, H\. Su, D\. Yan, N\. Chen, and J\. Zhu\(2022\)Towards safe reinforcement learning via constraining conditional value\-at\-risk\.arXiv preprint arXiv:2206\.04436\.Cited by:[§2\.0\.2](https://arxiv.org/html/2606.27766#S2.SS0.SSS2.p1.1)\.
- \[34\]B\. Yue, J\. Li, and G\. Liu\(2024\)Provably efficient exploration in inverse constrained reinforcement learning\.arXiv preprint arXiv:2409\.15963\.Cited by:[§2\.0\.2](https://arxiv.org/html/2606.27766#S2.SS0.SSS2.p1.1)\.
- \[35\]B\. Yue, S\. Wang, A\. Gaurav, J\. Li, P\. Poupart, and G\. Liu\(2025\)Understanding constraint inference in safety\-critical inverse reinforcement learning\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§2\.0\.2](https://arxiv.org/html/2606.27766#S2.SS0.SSS2.p1.1)\.
- \[36\]J\. Zhang, X\. Song, Y\. Li, D\. Liang, Z\. Zhang, and J\. Cai\(2026\)Adaptive dual cross\-attention network for multispectral object detection in autonomous driving\.Expert Systems with Applications,pp\. 132012\.Cited by:[§2\.0\.1](https://arxiv.org/html/2606.27766#S2.SS0.SSS1.p1.1)\.
- \[37\]J\. Zhang, M\. Xiang, Y\. Hu, W\. Hao, L\. Lei, and K\. Yi\(2026\)Multivariate feature learning and associative spatial information enhancement for snow object detection in autonomous driving\.Engineering Applications of Artificial Intelligence175,pp\. 114672\.Cited by:[§2\.0\.1](https://arxiv.org/html/2606.27766#S2.SS0.SSS1.p1.1)\.
- \[38\]S\. Zhang, W\. Zhang, and Q\. Gu\(2025\)Energy\-weighted flow matching for offline reinforcement learning\.InInternational conference on learning representations,Cited by:[§3\.0\.4](https://arxiv.org/html/2606.27766#S3.SS0.SSS4.p1.3)\.
- \[39\]R\. Zhao, S\. Xu, B\. Yue, and G\. Liu\(2025\)Toward exploratory inverse constraint inference with generative diffusion verifiers\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§2\.0\.2](https://arxiv.org/html/2606.27766#S2.SS0.SSS2.p1.1)\.
- \[40\]Y\. Zheng, R\. Liang, K\. Zheng, J\. Zheng, L\. Mao, J\. Li, W\. Gu, R\. Ai, S\. Li, X\. Zhan,et al\.\(2025\)Diffusion\-based planning for autonomous driving with flexible guidance\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 37207–37227\.Cited by:[§2\.0\.1](https://arxiv.org/html/2606.27766#S2.SS0.SSS1.p1.1),[§3\.0\.3](https://arxiv.org/html/2606.27766#S3.SS0.SSS3.p1.2)\.
- \[41\]Z\. Zhu, H\. Zhao, H\. He, Y\. Zhong, S\. Zhang, H\. Guo, T\. Chen, and W\. Zhang\(2023\)Diffusion models for reinforcement learning: a survey\.arXiv preprint arXiv:2311\.01223\.Cited by:[§1](https://arxiv.org/html/2606.27766#S1.p1.1),[§1](https://arxiv.org/html/2606.27766#S1.p2.1),[§2\.0\.1](https://arxiv.org/html/2606.27766#S2.SS0.SSS1.p1.1)\.