DiDrive: A Risk-Aware Hierarchical Diffusion Framework for Safe Offline Reinforcement Learning in Autonomous Driving

# DiDrive: A Risk-Aware Hierarchical Diffusion Framework for Safe Offline Reinforcement Learning in Autonomous Driving
Source: [https://arxiv.org/html/2609.01609](https://arxiv.org/html/2609.01609)
\\tnotetext

\[1\]This work was supported by the Natural Science Foundation of Fujian Province, China under Grant 2026J001253\.

1\]organization=College of Computer and Data Science, Fuzhou University, addressline=, city=Fuzhou, postcode=350116, state=Fujian, country=China

2\]organization=Maynooth International College of Engineering, Fuzhou University, addressline=, city=Fuzhou, postcode=350116, state=Fujian, country=China

3\]organization=Institute of Automation, Chinese Academy of Sciences, addressline=, city=Beijing, postcode=100190, state=, country=China\\cortext\[1\]Corresponding author

Jingtang ChenZhilin ChenPei XuMingjian Fusinceway@fzu\.edu\.cnWenxi LiuYuanlong Yu\[\[\[

###### Abstract

Autonomous driving requires safe and reliable sequential decision\-making in highly dynamic traffic environments\. Offline reinforcement learning \(RL\) has emerged as a promising paradigm by learning directly from static datasets, thereby mitigating online exploration hazards\. However, applying offline RL faces significant bottlenecks\. While diffusion models effectively capture multimodal behavioral priors, they are susceptible to distribution shift and heavy\-tailed risk signals, which can distort the reverse denoising direction and generate out\-of\-distribution \(OOD\) or hazardous actions\. This vulnerability is further compounded in complex traffic scenarios, where high\-dimensional state inputs often burden agents with redundant environmental information, masking essential local risks and misguiding action generation\.

To address these intertwined challenges, we propose DiDrive, a distribution\-guided offline diffusion framework for safe RL\. DiDrive is underpinned by two closely integrated core components: the Risk\-Aware Hierarchical Diffusion \(RHDif\) architecture and the Distribution Correction Estimation with Diffusion for Driving \(3DICE\) policy optimization paradigm\. In the state space, RHDif manages input complexity; it employs a low\-level risk\-gated encoder to enhance local danger features and a high\-level contextual modulator to align global semantics\. The fused features from these modules are injected into successive reverse denoising steps as a joint condition, guiding the model to filter redundancy and focus on core threats\. In the action space, 3DICE builds a four\-stage mechanism with internalized safety\. Building upon the refined representations extracted by RHDif, it further mitigates OOD overestimation and gradient oscillation through in\-sample calibrated guidance via density\-ratio weighting, spatiotemporal collaborative optimization, and ensemble\-based candidate ranking\. Working in tandem, RHDif refines environmental perception while 3DICE constrains action uncertainty, enhancing the overall safety utility of the hierarchical design\.

Experiments on the CARLA simulation benchmark demonstrate DiDrive’s stability across varying traffic densities\. Compared to baselines including IQL, CQL, and Diffusion\-QL, DiDrive exhibits improved overall performance in success rate, average reward, and lane deviation\. Notably, in complex high\-density scenarios with 60 background vehicles, DiDrive achieved the highest success rate of 85% and the highest average reward of 4295\.68\. By effectively integrating risk\-aware representation with distribution correction optimization, this study provides a robust and promising pathway for safe autonomous driving decision\-making in highly complex environments\.

###### keywords:

Autonomous driving\\sepReinforcement learning\\sepDiffusion model\\sepDICE

## 1Introduction

Safe and accurate sequential decision\-making in high\-dimensional, dynamic, and uncertain traffic environments is the bedrock of autonomous driving\[Wen\_2024\_ICLR\_DiLu,Hu\_2023\_CVPR\_UniAD,Zheng\_2024\_ECCV\_GenAD\]\. While data\-driven approaches like imitation learning and online reinforcement learning have driven significant progress\[10602544,9904958\], they face critical bottlenecks in building highly reliable closed\-loop systems\. Imitation learning often fails to cover long\-tailed risks, whereas online reinforcement learning introduces unacceptable safety hazards during trial\-and\-error exploration\. As a result, offline reinforcement learning has emerged as a vital alternative, learning effective policies directly from static datasets to circumvent the risks and costs of online interaction\[levine2020offline\]\.

Despite its promise, offline reinforcement learning is fundamentally bottlenecked by distribution shift\[10078377\]\. Current solutions generally fall into three categories, each with distinct limitations in the context of autonomous driving\. The first technical route relies on policy constraints and value regularization\[Ran2023PolicyRW,Kostrikov2021OfflineRL,kumar2019bear,NEURIPS2020\_0d2b2061\]\. Although these methods reduce distributional mismatch, their unimodal Gaussian assumptions struggle to capture the complex, multimodal nature of driving behaviors, such as yielding and overtaking\. The second technical route leverages highly expressive generative models, notably diffusion models\[NEURIPS2023\_3b3889d3,NEURIPS2023\_d45e0bfb,zheng2025diffusionbased,11094573,11093059,chi2025diffusionpolicy\]\. However, because the state\-action space in autonomous driving is high dimensional and continuous, diffusion\-based policies are prone to sampling actions outside the training data support\. Relying on value functions to filter these out\-of\-distribution candidate actions often leads to overestimation, potentially misguiding the agent into catastrophic decisions\. The third technical route employs distribution correction estimation, bypassing explicit out\-of\-distribution queries through in\-sample importance weighting\. Yet, in driving scenarios dominated by extreme, heavy\-tailed risk signals like collision penalties\[pmlr\-v238\-zhu24a\], traditional correction methods suffer from severe numerical instability, causing gradient oscillation and distorted policy guidance\.

Beyond challenges in the action space, high\-dimensional state representation poses another significant hurdle\. Complex traffic environments consist of road backgrounds, static obstacles, dynamic participants, and potential conflict zones\[Jiang2023VADVS\]\. Existing methods typically process these elements without adequate differentiation\. This uniform treatment diminishes the model’s ability to efficiently leverage safety\-critical information and delays the identification of high\-risk targets during sudden, hazardous events\[Jia\_2023\_ICCV\_DriveAdapter,Jia\_2023\_CVPR\_ThinkTwice\]\.

To overcome these intertwined challenges, policy optimization for autonomous driving must first restrict action queries to supported data regions\. Furthermore, it requires strict risk prioritization when processing complex environmental inputs\. To this end, models should maximally exploit the state\-action samples already present in the static dataset\. By assigning greater weight to high\-quality decision samples during training, the diffusion denoising process is naturally steered toward a superior action distribution within the data support\. Simultaneously, the agent must filter out high\-dimensional background redundancy to accurately isolate critical safety cues\. To seamlessly integrate support\-aware policy optimization with risk\-aware state representation, we propose DiDrive, a distribution\-guided diffusion framework for safe offline reinforcement learning\.

At the core of DiDrive is the Distribution Correction Estimation with Diffusion for Driving \(3DICE\) paradigm, which structures policy optimization into four stages: representation, guidance, optimization, and selection\. Under this paradigm, rather than explicitly querying unseen actions, the diffusion model adjusts its reverse denoising trajectory using in\-sample guidance derived from historical experience weights\. Coupled with progressive parameter integration \(PPI\), 3DICE securely shifts the behavior policy toward high\-quality, in\-support actions, effectively mitigating value overestimation and the instability caused by heavy\-tailed rewards\.

To combat information overload and the lack of risk differentiation in the state space, we design the Risk\-Aware Hierarchical Diffusion \(RHDif\) architecture\. At the lower level, a Risk\-Gated Spatiotemporal Encoder \(RGSE\) actively amplifies the feature responses of critical threats via bottom\-up local risk perception when safety margins shrink\. Concurrently, the high\-level module utilizes a Cross\-Modal Contextual Modulator \(CMCM\) to execute top\-down contextual filtering, effectively suppressing irrelevant background interference\.

The main contributions of this work are threefold:

- •First, we propose the 3DICE paradigm, formulating a novel optimization framework that integrates diffusion priors, in\-sample guidance, spatiotemporal collaborative optimization, and internalized support\-set constraints\. This design effectively avoids value overestimation from out\-of\-distribution queries and stabilizes training against heavy\-tailed risk signals\.
- •Second, we design the RHDif architecture to decouple local risk perception from high\-dimensional semantic denoising\. By synergistically combining the RGSE with the CMCM, the model efficiently filters redundant noise to focus computational resources on core threats\.
- •Third, extensive evaluations in the high\-fidelity CARLA simulator demonstrate that DiDrive significantly outperforms state\-of\-the\-art baselines across long\-tailed, high\-density, and safety\-critical scenarios, proving its robust generalization and decision reliability\.

## 2Related Work

### 2\.1Offline Reinforcement Learning

Offline reinforcement learning aims to learn decision\-making policies directly from static datasets without additional environment interaction\. This paradigm is particularly valuable in applications where data collection is costly or safety requirements are stringent\. Existing studies related to this work can be broadly categorized into three progressive directions: conservative value and policy learning, expressive policy modeling, and distribution correction\.

Representative conservative learning methods include behavior\-constrained algorithms like BCQ\[Fujimoto2018OffPolicyDR\]and TD3\-BC\[Fujimoto2021TD3BC\], as well as value regularization methods like CQL\[NEURIPS2020\_0d2b2061\]and implicit learning via IQL\[Kostrikov2021OfflineRL\]\. BCQ mitigates extrapolation error by constraining the learned policy to select actions close to the behavior policy, while CQL imposes conservative constraints on value estimates of OOD actions to reduce overestimation\. The empirical study by the Waymo team\[GuillenPerez2024OfflineRL\]further validates the effectiveness of CQL in autonomous driving, showing that it can leverage suboptimal data to improve policy performance and outperform traditional behavior cloning on medium\-scale datasets\.

While these existing offline RL methods effectively mitigate OOD overestimation, they typically rely on relatively restrictive policy parameterizations, such as unimodal Gaussian distributions\. This substantially limits their ability to model the highly multimodal nature of human driving behaviors\. To overcome this limitation, more expressive policy representations have been explored\. For instance, DT\[NEURIPS2021\_7f489f64\]formulates offline RL as return\-conditioned sequence modeling with a Transformer architecture, while Diffusion\-QL\[Wang2022DiffusionPA\]and QGPO\[10\.5555/3618408\.3619356\]employ diffusion models to enhance the multimodal expressiveness of policy distributions\.

Another line of work focuses on distribution correction in offline reinforcement learning\. Its theoretical foundation can be traced back to the DICE family of methods, which estimate stationary distribution ratios for off\-policy evaluation and optimization\. DualDICE estimates the discounted stationary occupancy ratio between the target policy and the behavior distribution\[nachum2019dualdice\], while GenDICE extends this idea to general stationary distribution correction through variational divergence minimization\[zhang2020gendice\]\. Compared with methods that directly search for high\-value actions in continuous action spaces, distribution correction methods reduce reliance on explicit value estimates of unseen actions, making them theoretically suitable for alleviating OOD queries\.

Although these diverse offline RL methods have shown promise in standard benchmarks, their direct application to autonomous driving reveals critical gaps in policy optimization\. Specifically, standard value regularization and distribution correction techniques are highly sensitive to the heavy\-tailed reward distributions typical of driving datasets, such as severe collision penalties\. When processing these heavy\-tailed signals, they often suffer from pronounced numerical instability and gradient oscillation, which distorts policy guidance\. Consequently, there is a pressing need for a robust optimization mechanism that can stabilize in\-sample value estimation against extreme safety risks, thereby laying a reliable foundation for integrating more expressive generative policies\.

### 2\.2Application of Diffusion Models in Autonomous Driving

As an emerging generative modeling approach, diffusion models can effectively capture multimodal driving behavior distributions through progressive denoising\[Ho2020DDPM,Song2021ScoreSDE,Dhariwal2021GuidedDiffusion,zheng2025diffusionbased,Zheng\_2024\_ECCV\_GenAD\], providing a promising technical pathway for handling uncertainty in autonomous driving\. Recent studies have demonstrated their potential in autonomous driving behavior planning\[11094573,11093059\]\. Compared with traditional generative models, diffusion models are more expressive in representing complex probability distributions and can generate diverse, reasonable, and smooth driving trajectories or behavioral sequences, thereby addressing multimodality and uncertainty in complex traffic scenarios\.

In behavior planning, diffusion models have achieved notable progress\. The pioneering work PlanningWD by Janner et al\.\[Janner2022PlanningWD\]validated the effectiveness of diffusion models in sequential decision\-making, and subsequent studies have extended this idea to prediction, simulation, and end\-to\-end planning for autonomous driving\[Hu\_2023\_CVPR\_UniAD\]\. At the multi\-agent interaction level, MotionDiffuser by Jiang et al\.\[10203706\]replaces traditional regression\-based prediction with diffusion\-based trajectory distribution modeling, achieving controllable generation with interaction consistency and multimodal diversity through a permutation\-invariant Transformer architecture\. Diffusion\-ES further combines trajectory denoising with gradient\-free evolutionary search for closed\-loop autonomous\-driving planning and instruction\-guided trajectory optimization\[Yang2024DiffusionES\]\. For real\-time end\-to\-end driving, DiffusionDrive by Liao et al\.\[11094573\]introduces a Truncated Diffusion Policy, which uses predefined multimodal anchors as priors and reduces the denoising process to only two steps, achieving 45 FPS inference while preserving trajectory diversity\. In city\-level simulation, SceneDiffuser\+\+ by Jeon et al\.\[Tan2025SceneDiffuserCT\]proposes CitySim based on a generative world model to jointly model dynamic agent generation, removal, and traffic light logic\. DriveDreamer\-2 further explores LLM\-enhanced world modeling for diverse driving\-video generation, showing the potential of generative world models for synthesizing uncommon and safety\-critical driving scenarios\[Zhao2025DriveDreamer2\]\. To further reduce planning latency, DiffuserLite by Dong et al\.\[NEURIPS2024\_dd6a47bc\]adopts a coarse\-to\-fine Progressive Refinement Strategy \(PRP\) to filter distant redundant information, increasing the decision frequency to 122 Hz and demonstrating the deployment potential of diffusion models on embedded vehicle platforms\.

However, existing diffusion architectures still face limitations in autonomous driving\. First, they may struggle to fully capture the multimodal state characteristics of complex traffic environments\. Second, the diffusion sampling process can generate actions or trajectory segments outside the support of the training data distribution\[NEURIPS2024\_a9f3457f\]\. If value functions are then used to filter these candidates, out\-of\-distribution value estimation errors may be further amplified\.

Furthermore, in terms of risk perception and representation learning, complex traffic scenes contain substantial background information that is irrelevant to the current decision\. Existing standard diffusion architectures often process high\-dimensional environmental inputs uniformly, lacking fine\-grained spatiotemporal attention mechanisms to dynamically quantify risk indicators\. Consequently, these methods fail to effectively address the inherent contradiction between amplifying local safety\-critical risk signals, such as vehicle cut\-ins or emergency braking, and suppressing global pseudo\-risks, such as distant static obstacles or non\-interacting vehicles in opposite lanes\[RACP\_2024,DQ\_GAT\_2022\]\. This deficiency in risk\-differentiated representation causes the modeling capacity to be dispersed across irrelevant inputs, frequently leading to overly conservative driving behaviors or erratic decision\-making\. Therefore, for safety\-critical events, the model must enhance its responsiveness to key local risk factors while relying on global semantic consistency to filter out pseudo\-risk interference\.

## 3Methodology

![[Uncaptioned image]](https://arxiv.org/html/2609.01609v1/DiDrive.png)

Figure 1:Overall architecture of DiDrive\.
This section presents the proposed DiDrive \(Diffusion\-based Driver\) framework, as illustrated in Fig\.[1](https://arxiv.org/html/2609.01609#S3.F1)\. Designed for safety\-critical domains such as autonomous driving, DiDrive tackles the challenges posed by high\-dimensional state spaces, complex temporal dependencies, and highly imbalanced long\-tailed data distributions\. Under such conditions, traditional offline RL methods often struggle to capture multimodal driving behaviors and are susceptible to value overestimation on OOD actions\. To overcome these bottlenecks, DiDrive synergistically couples the RHDif architecture with stabilized distribution correction mechanisms, thereby refining the offline policy optimization paradigm\.

The core architecture of DiDrive is driven by two closely synergistic components: the RHDif model and the 3DICE policy optimization paradigm\. Rather than operating in isolation, these two modules collaborate across the perception and decision\-making stages\. Specifically, RHDif serves as the foundational behavior learning module; it filters high\-dimensional environmental redundancy to extract risk\-aware state representations, constructing a representative multimodal behavior prior\. Building upon this structural prior, 3DICE acts as the distribution correction paradigm\. It leverages the representations and score functions provided by RHDif to perform in\-sample guidance and stabilized optimization in the action space\. Through this synergy, RHDif enables the model to effectively perceive essential local risks, while 3DICE constrains the policy updates within the data support, jointly enhancing the reliability of the autonomous driving policy without explicitly querying hazardous unseen actions\. To elaborate on this framework, we first formalize the driving problem and the inherent heavy\-tailed reward design in Section[3\.1](https://arxiv.org/html/2609.01609#S3.SS1)\. Building upon this formulation, we then detail the 3DICE paradigm and the RHDif architecture in the subsequent subsections\.

### 3\.1Problem Formulation and Reward Design

We model the autonomous driving decision\-making process as a standard Markov Decision Process \(MDP\), defined by the tupleℳ=⟨𝒮,𝒜,𝒫,ℛ,γ⟩\\mathcal\{M\}=\\langle\\mathcal\{S\},\\mathcal\{A\},\\mathcal\{P\},\\mathcal\{R\},\\gamma\\rangle\. Here,𝒮\\mathcal\{S\}denotes the continuous state space,𝒜\\mathcal\{A\}represents the continuous action space,𝒫​\(s′\|s,a\)\\mathcal\{P\}\(s^\{\\prime\}\|s,a\)is the unknown transition probability of the traffic environment,ℛ​\(s,a\)\\mathcal\{R\}\(s,a\)defines the reward function, andγ∈\[0,1\)\\gamma\\in\[0,1\)is the discount factor\. In our offline reinforcement learning paradigm, the agent cannot interact with the environment to collect new experiences\. Instead, it must learn an optimal policy solely from a static dataset𝒟=\{\(si,ai,ri,si′\)\}i=1N\\mathcal\{D\}=\\\{\(s\_\{i\},a\_\{i\},r\_\{i\},s^\{\\prime\}\_\{i\}\)\\\}\_\{i=1\}^\{N\}collected by behavior policies\.

In this driving MDP, the states∈𝒮s\\in\\mathcal\{S\}is constructed as a high\-dimensional multimodal vector, encompassing the ego\-vehicle kinematics, lane geometry, LiDAR point clouds, surrounding vehicle states, and navigation waypoints\. The actiona∈𝒜a\\in\\mathcal\{A\}comprises continuous longitudinal and lateral control commands\.

To effectively guide the offline policy optimization and ensure the safety of the autonomous driving agent, we design a comprehensive reward functionℛ​\(s,a\)\\mathcal\{R\}\(s,a\)that jointly encourages efficient driving, lane keeping, smooth control, and safety\. Specifically, the total reward is defined as:

r=rs​p​e​e​d\+rl​a​n​e\+rs​m​o​o​t​h\+rs​t​o​p\+rc​o​l​l​i​s​i​o​n\+roffroad,r=r\_\{speed\}\+r\_\{lane\}\+r\_\{smooth\}\+r\_\{stop\}\+r\_\{collision\}\+r\_\{\\textit\{offroad\}\},\(1\)where the desired\-speed tracking reward is computed as:

rs​p​e​e​d=\{v,v≤vd​e​s,−\(v−vd​e​s\),v\>vd​e​s,r\_\{speed\}=\\begin\{cases\}v,&v\\leq v\_\{des\},\\\\ \-\(v\-v\_\{des\}\),&v\>v\_\{des\},\\end\{cases\}\(2\)wherevvdenotes the ego\-vehicle speed andvd​e​sv\_\{des\}denotes the desired speed\. The lane\-keeping and smoothness terms are defined as:

rl​a​n​e=−\|Δ​dl​a​t\|,rs​m​o​o​t​h=−0\.5​\|al​a​t\|,r\_\{lane\}=\-\|\\Delta d\_\{lat\}\|,\\qquad r\_\{smooth\}=\-0\.5\|a\_\{lat\}\|,\(3\)whereΔ​dl​a​t\\Delta d\_\{lat\}denotes the lateral offset from the lane center andal​a​ta\_\{lat\}denotes the lateral acceleration\. To discourage unnecessary stopping in free\-flow conditions, a stopping penalty is applied when the front space is sufficient but the ego vehicle remains nearly stationary:

rs​t​o​p=\{−1,df​r​o​n​t\>10\.0​and​v<0\.1,0,otherwise,r\_\{stop\}=\\begin\{cases\}\-1,&d\_\{front\}\>10\.0\\ \\text\{and\}\\ v<0\.1,\\\\ 0,&\\text\{otherwise\},\\end\{cases\}\(4\)wheredf​r​o​n​td\_\{front\}denotes the front\-vehicle distance\. In addition, hard penalties are imposed on collision and off\-road events:

rc​o​l​l​i​s​i​o​n\\displaystyle r\_\{collision\}=\{−100,collision occurs,0,otherwise\.\\displaystyle=\\begin\{cases\}\-100,&\\text\{collision occurs\},\\\\ 0,&\\text\{otherwise\}\.\\end\{cases\}\(5\)roffroad\\displaystyle r\_\{\\textit\{offroad\}\}=\{−100,off\-road occurs,0,otherwise\.\\displaystyle=\\begin\{cases\}\-100,&\\text\{off\-road occurs\},\\\\ 0,&\\text\{otherwise\}\.\\end\{cases\}\(6\)
This reward design physically aligns with the safety\-critical nature of autonomous driving: while normal driving behaviors yield small, dense, and smooth rewards such asrs​p​e​e​dr\_\{speed\}andrl​a​n​er\_\{lane\}, it is essential to impose severe penalties on critical failures, settingrc​o​l​l​i​s​i​o​n=−100r\_\{collision\}=\-100for instance, to prevent reward hacking and ensure a reliable safety margin\. Consequently, this requirement naturally introduces a highly imbalanced, heavy\-tailed reward distribution\. Directly applying standard offline RL or diffusion policies to such imbalanced datasets often leads to value overestimation and gradient oscillation\. It is this inherent and realistic MDP property that motivates our proposed 3DICE optimization paradigm and RHDif architecture, which we detail in the following subsections\.

### 3\.23DICE

We propose the 3DICE paradigm\. To address the overestimation problem caused by value functions assigning excessively high values to unseen or out\-of\-distribution actions, we construct a four\-stage mechanism of "Representation\-Guidance\-Optimization\-Selection\." These four stages correspond to behavior prior modeling, in\-sample policy guidance, numerically stable optimization, and candidate action selection, respectively\. The goal is to reduce the influence of value estimation for out\-of\-distribution actions on policy updates and to improve the training stability of diffusion policies on heavy\-tailed driving data\.

Representation: In this stage, we use the RHDif architecture to construct a driving behavior prior\. Standard diffusion policies typically perform unified conditional modeling over input states, making them susceptible to interference from background vehicles, weakly relevant road elements, and sensor noise in high\-density traffic scenarios\. To this end, RHDif introduces local risk awareness and global semantic filtering into diffusion\-based behavior modeling: the low\-level module enhances state representations of potential risk interactions through RGSE, while the high\-level module suppresses background redundancy and pseudo\-risk responses through CMCM\. Through this risk\-aware representation, RHDif can capture multimodal driving behaviors such as overtaking, yielding, car\-following, and obstacle avoidance, while allowing the generation process to rely more heavily on safety\-related state information\. This diffusion prior further provides the action\-space score function∇alog⁡π𝒟​\(a\|s\)\\nabla\_\{a\}\\log\\pi^\{\\mathcal\{D\}\}\(a\|s\)of the behavior policy, offering a fundamental direction for subsequent in\-sample guidance\. As a result, the subsequent reverse denoising is firmly grounded in a high\-fidelity, risk\-filtered behavioral manifold\.

Guidance: We introduce the idea of distribution correction estimation and perform in\-sample reweighting of the behavior policy through a density ratio\. In distribution correction theory, the stationary distribution correction ratio between the target policy and the behavior distribution has been used to alleviate distribution shift and value overestimation in offline policy optimization\[nachum2019dualdice,zhang2020gendice,lee2021optidice\]\. The density ratio between the optimal stationary distribution and the behavior distribution,w∗​\(s,a\)=d∗​\(s,a\)d𝒟​\(s,a\)w^\{\*\}\(s,a\)=\\frac\{d^\{\*\}\(s,a\)\}\{d^\{\\mathcal\{D\}\}\(s,a\)\}, characterizes the relative importance of data samples under the target policy\. From the normalization relationship between the state\-action stationary distribution and the conditional policy, the target policy can be expressed as a weighted form of the behavior policy, namelyπ∗​\(a\|s\)∝w∗​\(s,a\)​π𝒟​\(a\|s\)\\pi^\{\*\}\(a\|s\)\\propto w^\{\*\}\(s,a\)\\pi^\{\\mathcal\{D\}\}\(a\|s\)\.

This relationship indicates that policy improvement can be achieved by increasing the generation probability of actions corresponding to high\-weight samples, without directly searching for optimal actions in unsupported regions of the action space\. Therefore, we first use a diffusion model to learn the behavior priorπ𝒟\\pi^\{\\mathcal\{D\}\}, and then introduce a guidance term induced by the density ratio during the reverse sampling process\. Specifically, the score functions of the target policy and the behavior policy at diffusion timestepttsatisfy the following relationship:

∇atlog⁡πt∗​\(at\|s\)=∇atlog⁡πt𝒟​\(at\|s\)\+∇atlog⁡𝔼a0∼π𝒟​\(a0\|at,s\)​\[w∗​\(s,a0\)\]\.\\begin\{split\}\\nabla\_\{a\_\{t\}\}\\log\\pi\_\{t\}^\{\*\}\(a\_\{t\}\|s\)=&\\,\\nabla\_\{a\_\{t\}\}\\log\\pi\_\{t\}^\{\\mathcal\{D\}\}\(a\_\{t\}\|s\)\\\\ &\+\\nabla\_\{a\_\{t\}\}\\log\\mathbb\{E\}\_\{a\_\{0\}\\sim\\pi^\{\\mathcal\{D\}\}\(a\_\{0\}\|a\_\{t\},s\)\}\\left\[w^\{\*\}\(s,a\_\{0\}\)\\right\]\.\\end\{split\}\(7\)
Here, the first term corresponds to the diffusion score of the behavior prior, while the second term represents the policy correction direction induced by the density ratio\. Since the second term contains a conditional expectation, it is difficult to compute directly\. Therefore, we introduce a neural networkgθ​\(s,at,t\)g\_\{\\theta\}\(s,a\_\{t\},t\)to approximatelog⁡𝔼a0∼π𝒟​\(a0\|at,s\)​\[w∗​\(s,a0\)\]\\log\\mathbb\{E\}\_\{a\_\{0\}\\sim\\pi^\{\\mathcal\{D\}\}\(a\_\{0\}\|a\_\{t\},s\)\}\[w^\{\*\}\(s,a\_\{0\}\)\], and propose the following In\-sample Guidance Learning \(IGL\) objective:

minθ⁡𝔼t∼𝒰​\(0,T\)𝔼a0∼π𝒟​\(a0\|s\)​𝔼at∼p​\(at\|a0\)⋅\[w∗​\(s,a0\)​e−gθ​\(s,at,t\)\+gθ​\(s,at,t\)\]\.\\begin\{split\}\\min\_\{\\theta\}\\mathbb\{E\}\_\{t\\sim\\mathcal\{U\}\(0,T\)\}&\\,\\mathbb\{E\}\_\{a\_\{0\}\\sim\\pi^\{\\mathcal\{D\}\}\(a\_\{0\}\|s\)\}\\mathbb\{E\}\_\{a\_\{t\}\\sim p\(a\_\{t\}\|a\_\{0\}\)\}\\\\ &\\cdot\\left\[w^\{\*\}\(s,a\_\{0\}\)e^\{\-g\_\{\\theta\}\(s,a\_\{t\},t\)\}\+g\_\{\\theta\}\(s,a\_\{t\},t\)\\right\]\.\\end\{split\}\(8\)
Under this objective,∇atgθ​\(s,at,t\)\\nabla\_\{a\_\{t\}\}g\_\{\\theta\}\(s,a\_\{t\},t\)can be used to approximate the policy correction term in Eq\. \([7](https://arxiv.org/html/2609.01609#S3.E7)\)\. Since the training samples are drawn from offline data and its diffusion forward perturbation process, IGL does not require the additional generation or evaluation of arbitrary actions, seamlessly constraining the policy optimization strictly within the data support\.

In the dual formulation of the DICE objective, the optimal density ratio is derived from the advantage function\. Consequently, when implementing IGL, it is crucial to ensure that the mappingw∗​\(s,a\)=\(f′\)−1​\(Q∗​\(s,a\)−V∗​\(s\)\)w^\{\*\}\(s,a\)=\(f^\{\\prime\}\)^\{\-1\}\(Q^\{\*\}\(s,a\)\-V^\{\*\}\(s\)\)remains strictly non\-negative\. To this end, we adopt a piecewiseff\-divergence: a quadratic constraint is used when the weight falls into the high\-weight region, while an entropy\-based constraint is applied when the weight falls into the low\-weight region\. This design ensures that the weight mapping\(f′\)−1​\(⋅\)\(f^\{\\prime\}\)^\{\-1\}\(\\cdot\)remains non\-negative across different intervals, thereby satisfying the basic requirement of IGL for sample weights\. Meanwhile, the corresponding convex conjugate functionf∗f^\{\*\}grows at most quadratically in the positive interval\. Compared with exponential growth, this weakens the amplification effect of extreme advantage values on DICE value updates, thereby improving optimization stability in heavy\-tailed reward scenarios\.

Optimization: With the model architecture and guidance direction established in the preceding stages, this third stage focuses on stabilizing the training process against data imbalance\. In autonomous driving, safety\-critical edge cases are extremely sparse and usually appear as heavy\-tailed reward signals\. Most samples correspond to normal driving with mild penalties, whereas a small number of collision\-related samples may produce large negative returns\. Directly optimizing diffusion policies on such imbalanced data can destabilize value estimation, amplify estimation bias, and trigger gradient oscillation, leading to distorted guidance signals and unstable control outputs\.

To stabilize training, we propose a spatiotemporal collaborative optimization mechanism\. In the spatial dimension, we regularize value estimation and the guidance field to suppress extreme estimation errors\. Specifically, for the value network, we introduce an error clipping operationclamp​\(⋅,0,cclip\)\\text\{clamp\}\(\\cdot,0,c\_\{\\text\{clip\}\}\)to bound the influence of extreme samples\. We further add a value regularization termλv​V~​\(s\)2\\lambda\_\{v\}\\tilde\{V\}\(s\)^\{2\}to constrain extrapolated values for states outside the data support, preventing abnormal overestimation\.

For the energy functiongθg\_\{\\theta\}used to guide diffusion sampling, numerical oscillations may destabilize the generated actions\. Therefore, we introduce statistical regularization to smooth the guidance field\. A variance regularization termλvar​Var​\(g~θ\)\\lambda\_\{\\text\{var\}\}\\mathrm\{Var\}\(\\tilde\{g\}\_\{\\theta\}\)suppresses high\-frequency fluctuations in the energy surface, while a mean regularization termλmean​\|𝔼​\[g~θ\]\|\\lambda\_\{\\text\{mean\}\}\|\\mathbb\{E\}\[\\tilde\{g\}\_\{\\theta\}\]\|anchors the guidance signal around zero and prevents systematic numerical drift\.

In the temporal dimension, mini\-batch sampling may introduce high\-frequency noise across training iterations\. To smooth the parameter update trajectory, we introduce Progressive Parameter Integration \(PPI\), which acts as a temporal low\-pass filter during optimization\. By computing an exponential moving average of historical network states, PPI mitigates the disproportionate impact of individual extreme mini\-batches, thereby dampening high\-frequency gradient oscillations\. Rather than relying solely on instantaneous estimates, PPI maintains smoothly evolving target estimates:

V~\(k\)​\(s\)\\displaystyle\\tilde\{V\}^\{\(k\)\}\(s\)=η⋅V~\(k−1\)​\(s\)\+\(1−η\)⋅Vθ\(k\)​\(s\),\\displaystyle=\\eta\\cdot\\tilde\{V\}^\{\(k\-1\)\}\(s\)\+\(1\-\\eta\)\\cdot V\_\{\\theta\}^\{\(k\)\}\(s\),\(9\)g~θ\(k\)​\(s,aτ,τ\)\\displaystyle\\tilde\{g\}\_\{\\theta\}^\{\(k\)\}\(s,a\_\{\\tau\},\\tau\)=η⋅g~θ\(k−1\)​\(s,aτ,τ\)\\displaystyle=\\eta\\cdot\\tilde\{g\}\_\{\\theta\}^\{\(k\-1\)\}\(s,a\_\{\\tau\},\\tau\)\+\(1−η\)⋅gθ\(k\)​\(s,aτ,τ\),\\displaystyle\\quad\+\(1\-\\eta\)\\cdot g\_\{\\theta\}^\{\(k\)\}\(s,a\_\{\\tau\},\\tau\),\(10\)wherekkdenotes the training iteration,τ\\taudenotes the diffusion timestep,η∈\[0,1\)\\eta\\in\[0,1\)is the decay coefficient, andVθ\(k\)V\_\{\\theta\}^\{\(k\)\}andgθ\(k\)g\_\{\\theta\}^\{\(k\)\}denote the instantaneous state\-value estimate and guidance\-term estimate of the current network, respectively\.

Based on the above mechanism,the value\-side objectiveis defined as:

ℒvalue​\(θ\)=\\displaystyle\\mathcal\{L\}\_\{\\text\{value\}\}\(\\theta\)=α​𝔼d𝒟​\[clamp​\(f∗​\(Q~−Vθα\),0,cclip\)\]\\displaystyle\\alpha\\mathbb\{E\}\_\{d^\{\\mathcal\{D\}\}\}\\left\[\\text\{clamp\}\\left\(f^\{\*\}\\left\(\\frac\{\\tilde\{Q\}\-V\_\{\\theta\}\}\{\\alpha\}\\right\),0,c\_\{\\text\{clip\}\}\\right\)\\right\]\(11\)\+𝔼d𝒟​\[\(1−γ\)​Vθ​\(s\)\]\+λv​𝔼d𝒟​\[Vθ​\(s\)2\],\\displaystyle\+\\mathbb\{E\}\_\{d^\{\\mathcal\{D\}\}\}\\left\[\(1\-\\gamma\)V\_\{\\theta\}\(s\)\\right\]\+\\lambda\_\{v\}\\mathbb\{E\}\_\{d^\{\\mathcal\{D\}\}\}\\left\[V\_\{\\theta\}\(s\)^\{2\}\\right\],whered𝒟d^\{\\mathcal\{D\}\}denotes the empirical distribution of the offline dataset,γ\\gammais the discount factor,α\\alphacontrols the strength of the dual constraint,cclipc\_\{\\text\{clip\}\}is the clipping threshold, andλv\\lambda\_\{v\}is the value regularization coefficient\.

Similarly,the energy\-side objectiveis defined as:

ℒdual=𝔼π𝒟,p​\(aτ\|a0\)​\[w⋅e−g~\+g~\]\+λvar​Var​\(g~\)\+λmean​\|𝔼​\[g~\]\|,\\mathcal\{L\}\_\{\\text\{dual\}\}=\\mathbb\{E\}\_\{\\pi^\{\\mathcal\{D\}\},p\(a\_\{\\tau\}\|a\_\{0\}\)\}\\left\[w\\cdot e^\{\-\\tilde\{g\}\}\+\\tilde\{g\}\\right\]\+\\lambda\_\{\\text\{var\}\}\\mathrm\{Var\}\(\\tilde\{g\}\)\+\\lambda\_\{\\text\{mean\}\}\|\\mathbb\{E\}\[\\tilde\{g\}\]\|,\(12\)where𝔼π𝒟,p​\(aτ\|a0\)\\mathbb\{E\}\_\{\\pi^\{\\mathcal\{D\}\},p\(a\_\{\\tau\}\|a\_\{0\}\)\}denotes the joint expectation over the behavior policy and the diffusion transition probability,wwis the importance weight derived from value estimation, andλvar\\lambda\_\{\\text\{var\}\}andλmean\\lambda\_\{\\text\{mean\}\}control the variance and magnitude of the guidance signal, respectively\.

Selection: As the final stage of the RGOS mechanism, this step identifies a high\-quality action from the diffusion\-generated candidates while ensuring the final decision remains tightly anchored to the behavior\-supported action region\.

First, we use the diffusion policyπθ\\pi\_\{\\theta\}trained in the representation stage as the generator\. Given the current statess, we sample in parallel a set of candidate actions𝒜r​a​w=\{ai\}i=1K\\mathcal\{A\}\_\{raw\}=\\\{a^\{i\}\\\}\_\{i=1\}^\{K\}that are consistent in intention but differ in detail, whereKKdenotes the number of parallel samples\. To avoid the high\-variance risk caused by a single value estimate, we introduce the critics\{Qϕj∗\}j=1M\\\{Q\_\{\\phi\_\{j\}\}^\{\*\}\\\}\_\{j=1\}^\{M\}, which are stably trained through the DICE objective in the Optimization stage, as evaluators\. Unlike traditional Softmax sampling, which inherently assigns non\-zero probabilities to sub\-optimal actions and may expose the agent to stochastic risks, we adopt an ensemble\-based rejection sampling strategy\. This approach effectively filters out lower\-confidence candidates and selects the action with the highest consensus value, providing a tighter safety margin for the final decision:

as​t​a​t∗=arg⁡maxai∈𝒜r​a​w⁡\(1M​∑j=1MQϕj∗​\(s,ai\)\),a\_\{stat\}^\{\*\}=\\arg\\max\_\{a^\{i\}\\in\\mathcal\{A\}\_\{raw\}\}\\left\(\\frac\{1\}\{M\}\\sum\_\{j=1\}^\{M\}Q\_\{\\phi\_\{j\}\}^\{\*\}\(s,a^\{i\}\)\\right\),\(13\)whereas​t​a​t∗a\_\{stat\}^\{\*\}denotes the optimal action,𝒜r​a​w\\mathcal\{A\}\_\{raw\}represents the sampled set containing multiple candidate actions, andMMdenotes the number of ensemble Q\-networks\.

Since the candidate actions are generated by a diffusion policy fitted to behavioral data, they are generally closer to the high\-probability regions covered by the behavior data than actions obtained through direct optimization in the continuous action space\[wu2022spot\]\. The critics perform relative ranking within this candidate set rather than conducting a global search over arbitrary unsupported actions, thereby effectively reducing the influence of out\-of\-distribution value overestimation on action selection\.

### 3\.3Risk\-Aware Hierarchical Diffusion

![[Uncaptioned image]](https://arxiv.org/html/2609.01609v1/RHDif.png)

Figure 2:Overview of the RHDif architecture\. RGSE enhances local risk\-aware features, while CMCM calibrates risk signals with global semantic context\.
To address environmental information redundancy and the difficulty of distinguishing local risk factors in complex traffic scenarios, this paper proposes RHDif, a risk\-aware spatiotemporally decoupled hierarchical diffusion architecture, as shown in Fig\.[2](https://arxiv.org/html/2609.01609#S3.F2)\. This architecture performs hierarchical feature selection in the latent space to balance rapid hazard perception with global scene comprehension\. At the low level, the RGSE enhances local safety\-related features to prevent critical threats from being diluted by background noise\. At the high level, the CMCM incorporates global semantic information to calibrate these local risk responses, effectively preventing overly conservative or erratic behaviors triggered by pseudo\-risks\.

Specifically, RGSE generates risk\-related weights based on kinematic priors—such as the relative positions of neighboring vehicles, longitudinal distance, lateral offset, and relative velocity—enabling the model to assign stronger feature responses to regions with elevated potential risk\. Concurrently, CMCM models the cross\-modal relationship between trajectory intentions and environmental context during the diffusion denoising process, suppressing redundant background features through channel and spatial gating\. Through this hierarchical mechanism, proceeding from local risk enhancement to global semantic calibration, RHDif precisely allocates its representational capacity to the most safety\-critical interactions, thereby significantly reducing the interference of high\-dimensional redundant information in action generation\.

#### 3\.3\.1Risk\-Gated Spatiotemporal Encoder Module

In complex traffic flows, surrounding vehicles, lane structures, and sensor observations introduce substantial redundant information\. If all neighboring objects are assigned uniform importance, the safety\-critical cues can be easily overwhelmed by background noise\. To address this issue, RGSE first employs local risk awareness based on kinematic priors to model the local interactions between surrounding vehicles and the ego vehicle\. Let the raw observation of theii\-th surrounding vehicle beoi∈ℝ4o\_\{i\}\\in\\mathbb\{R\}^\{4\}\. Its geometric importance weight is defined as:

ℳg​e​o\(i\)=σ​\(τl​o​n​g−x\(i\)κ1\)⋅σ​\(τl​a​t−\|y\(i\)\|κ2\)⋅𝕀​\(x\(i\)\>0\),\\mathcal\{M\}\_\{geo\}^\{\(i\)\}=\\sigma\\left\(\\frac\{\\tau\_\{long\}\-x^\{\(i\)\}\}\{\\kappa\_\{1\}\}\\right\)\\cdot\\sigma\\left\(\\frac\{\\tau\_\{lat\}\-\|y^\{\(i\)\}\|\}\{\\kappa\_\{2\}\}\\right\)\\cdot\\mathbb\{I\}\(x^\{\(i\)\}\>0\),\(14\)whereℳg​e​o\(i\)∈\(0,1\)\\mathcal\{M\}\_\{geo\}^\{\(i\)\}\\in\(0,1\)denotes the spatial importance of theii\-th surrounding vehicle,τl​o​n​g\\tau\_\{long\}andτl​a​t\\tau\_\{lat\}denote the longitudinal perception range and lateral attention range, respectively,x\(i\)x^\{\(i\)\}andy\(i\)y^\{\(i\)\}are the local coordinates of the surrounding vehicle relative to the ego vehicle,κ1\\kappa\_\{1\}andκ2\\kappa\_\{2\}control the smoothness of weight variation, and𝕀​\(⋅\)\\mathbb\{I\}\(\\cdot\)is the indicator function used to retain the region in front of the ego vehicle, wherex\(i\)\>0x^\{\(i\)\}\>0\. This weight is then applied to the projected features of surrounding vehicles, and the features of all surrounding vehicles are weighted and aggregated to obtain the local vehicle interaction representation𝐡v​e​h\\mathbf\{h\}\_\{veh\}\.

After multimodal information fusion, we concatenate the ego state, lane information, LiDAR features, and surrounding\-vehicle interaction representation, and add learnable type embeddings to form the multimodal context tensor𝐙c​t​x=Concat​\(𝐡e​g​o,𝐡l​a​n​e,𝐡l​i​d​a​r,𝐡v​e​h\)\+𝐄t​y​p​e\\mathbf\{Z\}\_\{ctx\}=\\text\{Concat\}\(\\mathbf\{h\}\_\{ego\},\\mathbf\{h\}\_\{lane\},\\mathbf\{h\}\_\{lidar\},\\mathbf\{h\}\_\{veh\}\)\+\\mathbf\{E\}\_\{type\}\. To further enhance the model’s sensitivity to local risk variations, RGSE introduces a risk\-gating mechanism that transforms spatiotemporal risk priors into channel\-level modulation weights\. Specifically, we construct the risk input vector𝐫r​i​s​k=\[v,\|Δ​dl​a​t\|,df​r​o​n​t,Δ​v\]\\mathbf\{r\}\_\{risk\}=\[v,\|\\Delta d\_\{lat\}\|,d\_\{front\},\\Delta v\], and generate dynamic risk weights through a multilayer perceptron:

𝐖r​i​s​k=σ​\(ℱm​l​p​\(𝐫r​i​s​k\)\)\+cb​a​s​e,\\mathbf\{W\}\_\{risk\}=\\sigma\(\\mathcal\{F\}\_\{mlp\}\(\\mathbf\{r\}\_\{risk\}\)\)\+c\_\{base\},\(15\)wherecb​a​s​ec\_\{base\}denotes the base activation level, which prevents the risk weights from becoming too small\. Then,𝐖r​i​s​k\\mathbf\{W\}\_\{risk\}is applied to the multimodal context features as a channel\-level multiplicative modulation factor, yielding𝐙~c​t​x=𝐙c​t​x⊙𝐖r​i​s​k\\tilde\{\\mathbf\{Z\}\}\_\{ctx\}=\\mathbf\{Z\}\_\{ctx\}\\odot\\mathbf\{W\}\_\{risk\}\. When the relative velocity increases, the longitudinal distance decreases, or the lateral offset increases, the risk\-gating mechanism increases the response weights of the relevant modalities and channels, enabling the diffusion policy to more fully utilize safety\-related state information during reverse denoising\. Compared with fixed\-threshold rules, this mechanism modulates feature responses in a continuous and differentiable manner, helping alleviate the discontinuities caused by hard thresholds\.

#### 3\.3\.2Cross\-Modal Contextual Modulator Module

In dynamic traffic scenarios, local risk signals are not always equivalent to real hazards\. For example, vehicles in the opposite lane, distant static obstacles, or transient sensor noise may produce strong responses at the local feature level, but they do not necessarily require conservative actions\. Therefore, relying solely on low\-level risk enhancement may lead to misjudgment or overly conservative behavior\. To alleviate this issue, CMCM introduces cross\-modal contextual modulation in the high\-level latent space, using global semantic information to filter and calibrate local risk responses\.

First, we formulate the risk\-modulated context tensor𝐙~c​t​x\\tilde\{\\mathbf\{Z\}\}\_\{ctx\}from the RGSE module as a sequence of condition tokensC∈ℝB×N×DC\\in\\mathbb\{R\}^\{B\\times N\\times D\}\. A multi\-head self\-attention mechanism is then applied to model global dependencies among these tokens, yielding the contextualized representationC′C^\{\\prime\}\. Subsequently, letYYdenote the noisy action latent at the current diffusion timestep\. For both the latent featureYYand the context featureC′C^\{\\prime\}, CMCM employs a unified channel\-spatial gating operator:

Φ​\(X\)=X⊙σ​\(ℳc​h​a​n​n​e​l​\(X\)\)⊙σ​\(ℳs​p​a​t​i​a​l​\(Pool​\(X\)\)\),\\Phi\(X\)=X\\odot\\sigma\(\\mathcal\{M\}\_\{channel\}\(X\)\)\\odot\\sigma\(\\mathcal\{M\}\_\{spatial\}\(\\text\{Pool\}\(X\)\)\),\(16\)whereXXmay denote either the latent featureYYor the context featureC′C^\{\\prime\}, andℳc​h​a​n​n​e​l\\mathcal\{M\}\_\{channel\}andℳs​p​a​t​i​a​l\\mathcal\{M\}\_\{spatial\}denote the channel attention block and the spatial attention block, respectively\. This gating operator enhances features related to interaction\-aware decision\-making while suppressing redundant channel responses and background regions\.

Finally, the modulated context featureΦ​\(C′\)\\Phi\(C^\{\\prime\}\)and latent state featureΦ​\(Y\)\\Phi\(Y\)are injected into the diffusion denoising backbone through residual fusion:

Zo​u​t=𝒲f​u​s​e​\(Φ​\(C′\)\)\+𝒲s​e​l​f​\(Φ​\(Y\)\)\+Yr​e​s​i​d​u​a​l,Z\_\{out\}=\\mathcal\{W\}\_\{fuse\}\(\\Phi\(C^\{\\prime\}\)\)\+\\mathcal\{W\}\_\{self\}\(\\Phi\(Y\)\)\+Y\_\{residual\},\(17\)where𝒲f​u​s​e\\mathcal\{W\}\_\{fuse\}and𝒲s​e​l​f\\mathcal\{W\}\_\{self\}denote linear projection layers\. Through this cross\-modal contextual modulation mechanism, CMCM incorporates global scene constraints while preserving local risk responses, thereby reducing the interference of irrelevant high\-response features\. Coupled with the RGSE, this hierarchical gating directly injects risk\-calibrated, high\-fidelity priors into the diffusion denoising backbone\.

## 4Experiments

### 4\.1Experimental Setup

To validate the effectiveness of the proposed algorithm under high\-dimensional states and dynamic interactions, we construct a closed\-loop experimental platform using the high\-fidelity autonomous driving simulator CARLA\[dosovitskiy2017carla\]\. Specifically, we select Town03, a complex urban environment featuring a multi\-lane roundabout, tunnels, and varied irregular junctions, for model training and in\-domain evaluation\. To assess zero\-shot cross\-town generalization, we introduce Town05, an unseen map characterized by grid\-like multi\-lane networks and highway intersections\. The layouts of both maps are illustrated in Fig\.[3](https://arxiv.org/html/2609.01609#S4.F3)\. Furthermore, to comprehensively evaluate the representational robustness of the model, we set up simple, medium, and complex scenarios with 20, 40, and 60 background vehicles, respectively, on both maps\.

![[Uncaptioned image]](https://arxiv.org/html/2609.01609v1/Town03.jpg)

\(a\) Town03

![[Uncaptioned image]](https://arxiv.org/html/2609.01609v1/Town05.jpg)

\(b\) Town05

Figure 3:Top\-view maps of CARLA Town03 and Town05\.
The experiments use a large\-scale offline dataset stored in HDF5 format\. This dataset is collected in Town03 and contains more than 7,000 trajectories and 1\.1 million timesteps\. The data distribution consists of a mixture of expert policies and random exploration policies at a ratio of8:28:2\. The model input is defined as a 307\-dimensional observation vector, including the ego vehicle state \(99dimensions\), lane information \(22dimensions\), LiDAR point\-cloud features \(240240dimensions\), local surrounding\-vehicle features \(2020dimensions\), and navigation waypoints \(3636dimensions\)\.

The training process adopts a two\-stage strategy\. In the first stage, the backbone of the RHDif architecture is pretrained with a learning rate of3×10−43\\times 10^\{\-4\}and a batch size of10241024to extract a high\-fidelity behavior prior\. In the second stage, the 3DICE paradigm is fine\-tuned for distribution correction\. In the experiments, the number of denoising iterations of the diffusion modelKKis set to 15, the inference sampling numberinference\_sampleis set to 32, and the number of ensemble estimation samplesMMis set to 16\. Additionally, the learning rate of the value network is set to4×10−54\\times 10^\{\-5\}, and the discount factorγ\\gammais set to 0\.99 to maintain the stability of long\-horizon optimization\.

During evaluation, we conduct 100 online closed\-loop test episodes for each configuration, using Success Rate \(SR\), Average Reward, and Average Distance as the core evaluation metrics\[Jia\_2024\_NeurIPS\_Bench2Drive,Caesar2020nuScenes,Ettinger2021WOMD\]\. SR is defined as the proportion of test episodes in which the vehicle neither collides nor drives off the road within 1000 timesteps\. Average Reward represents the cumulative task returns obtained per episode, averaged across all test runs, which comprehensively reflects the overall optimality of the policy\. Average Distance measures the mean longitudinal distance traveled by the ego vehicle before an episode terminates due to success, failure, or timeout, serving as a quantitative indicator of driving progress and survival capability\.

For a fair comparison, all baseline methods—including Implicit Q\-Learning \(IQL\)\[Kostrikov2021OfflineRL\], Conservative Q\-Learning \(CQL\)\[NEURIPS2020\_0d2b2061\], OptiDICE \(ODICE\)\[lee2021optidice\], TD3\-BC\[Fujimoto2021TD3BC\], Q\-Guided Policy Optimization \(QGPO\)\[10\.5555/3618408\.3619356\], and Diffusion\-QL\[Wang2022DiffusionPA\]—are trained and evaluated under an identical experimental protocol\. Specifically, they share the same offline dataset, state\-action space representations, reward function, and closed\-loop evaluation configurations\. Furthermore, diffusion\-based baselines \(QGPO and Diffusion\-QL\) adopt the exact same inference parameters, such as the number of denoising steps and sampled candidates, as DiDrive\. By strictly controlling these variables, we ensure that any performance disparities stem directly from the underlying policy learning and optimization mechanisms\.

### 4\.2Comparative Experiments

#### 4\.2\.1Training Analysis

To evaluate the learning efficiency and stability of our proposed DiDrive framework, we first analyze the convergence dynamics of different baseline methods during the training phase\.

![[Uncaptioned image]](https://arxiv.org/html/2609.01609v1/Comparison_TotalReward.png)

Figure 4:Learning curves of total reward for different methods during training\.
![[Uncaptioned image]](https://arxiv.org/html/2609.01609v1/Comparison_LaneDeviation.png)

Figure 5:Learning curves of lane deviation for different methods during training\.
Figs\.[4](https://arxiv.org/html/2609.01609#S4.F4)and[5](https://arxiv.org/html/2609.01609#S4.F5)illustrate the overall convergence dynamics of DiDrive and the baseline methods during training\. Overall, DiDrive shows superior performance and stability, while the baseline methods exhibit different limitations when handling high\-dimensional complex states\. The diffusion\-based baseline QGPO shows long\-term stagnation in total reward, while its lane deviation remains above 1\.0 m for most of the training process\. This indicates that, without sufficiently reliable value guidance, standard diffusion policies are prone to extrapolation errors in large state spaces and struggle to maintain lateral control constraints\. Although TD3\-BC keeps the deviation at a relatively low level, its low reward suggests that the policy suffers from unstable speed control in dynamic interactions\. Specifically, it exhibits severe frequent oscillations, alternating between excessive acceleration and extremely low\-speed stagnation, which limits its overall driving efficiency and long\-horizon stability\.

The strongly oscillating curve of ODICE reflects the numerical instability of traditional DICE\-based methods when facing high\-dimensional features and heavy\-tailed rewards\. The reward degradation and deviation increase in the later stage of training further indicate potential value divergence caused by unstable gradients\. In comparison, IQL and CQL show a certain degree of competitiveness\. However, the relatively large performance fluctuations of IQL suggest that its expectile regression mechanism may lack sufficient consistency in long\-horizon decision\-making, while the conservative nature of CQL limits its ability to approach the optimal performance boundary\. In contrast, benefiting from the high\-fidelity prior features provided by the hierarchical diffusion architecture and the smooth optimization introduced by the PPI mechanism in the 3DICE paradigm, DiDrive rapidly converges to the highest reward range \(\>5200\>5200\) within a small number of training steps, while maintaining the lane deviation at an extremely low level of approximately 0\.1 m throughout training\. This smooth and efficient convergence trend demonstrates that DiDrive effectively mitigates the interference of feature redundancy and environmental noise, achieving a favorable balance between traffic efficiency and control precision\.

#### 4\.2\.2Testing Analysis

Following the training phase, we comprehensively evaluate the closed\-loop driving performance and cross\-domain generalization capabilities of all fully trained models under varying traffic densities\.

Table 1:Performance comparison on Town03 under different traffic densities\. We report Success Rate, Average Reward, and Average Driving Distance over 100 closed\-loop evaluation episodes\.ScenarioMetricMethodIQL\[Kostrikov2021OfflineRL\]CQL\[NEURIPS2020\_0d2b2061\]ODICE\[lee2021optidice\]TD3\-BC\[Fujimoto2021TD3BC\]QGPO\[10\.5555/3618408\.3619356\]Diffusion\-QL\[Wang2022DiffusionPA\]DiDrive \(Ours\)Simple20 VehiclesSuccess Rate↑\\uparrow77%86%21%87%16%90%92%Avg\. Reward↑\\uparrow3943\.994004\.431174\.92955\.721157\.623978\.194902\.94Avg\. Distance \(m\)↑\\uparrow306247171327125241286Medium40 VehiclesSuccess Rate↑\\uparrow56%83%20%83%14%85%88%Avg\. Reward↑\\uparrow3488\.713922\.531063\.85787\.721148\.523823\.844713\.84Avg\. Distance \(m\)↑\\uparrow269280125276122236276Complex60 VehiclesSuccess Rate↑\\uparrow33%74%17%72%9%77%85%Avg\. Reward↑\\uparrow3275\.793819\.051017\.21539\.95927\.713425\.914295\.68Avg\. Distance \(m\)↑\\uparrow237245113238103222254

Table[1](https://arxiv.org/html/2609.01609#S4.T1)quantifies the closed\-loop performance of different methods under varying traffic densities\. In the complex scenario with 60 background vehicles, traditional baselines reveal clear performance bottlenecks\. CQL maintains a success rate of 74\.00% through conservative value constraints, but its average reward is only 3819\.05, which is lower than DiDrive’s 4295\.68\. This suggests that excessive conservatism, while helping avoid some high\-risk actions, may also limit driving efficiency and introduce unnecessary low\-speed behaviors, which are penalized by the desired\-speed tracking and stopping terms in the reward function\. IQL remains somewhat competitive in simple scenarios, but its success rate drops to 33\.00% in complex scenarios, indicating that its implicit policy improvement mechanism lacks stability under high\-density traffic flows and long\-horizon interactions\. Although TD3\-BC with behavior cloning regularization achieves the longest driving distance in the simple scenario \(327 m\), its speed control exhibits severe frequent oscillations in dynamic interactions, alternating between excessive acceleration and near\-stationary behavior\. This control instability substantially reduces its average reward to 539\.95 in the complex scenario, because excessive acceleration, near\-stationary behavior, and unstable lateral motion are directly penalized by the desired\-speed tracking, stopping, and lateral\-acceleration terms in the reward function\.

Notably, both the purely generative policy QGPO and the distribution correction method ODICE show significant performance degradation\. Across the three traffic densities, the highest success rates of QGPO and ODICE are only 16\.00% and 21\.00%, respectively\. This indicates that, without stable support\-set constraints or reliable density\-ratio estimation, the policy is vulnerable to the combined effects of high\-dimensional observations, accumulated closed\-loop errors, and heavy\-tailed rewards\. As a representative value\-guided diffusion policy, Diffusion\-QL demonstrates strong multimodal expressiveness, achieving success rates of 90\.00%, 85\.00%, and 77\.00% in simple, medium, and complex scenarios, respectively\. However, its average reward in the complex scenario is 3425\.91, which remains significantly lower than DiDrive’s 4295\.68\. This suggests that relying solely on the generative capability of diffusion policies and value guidance is still insufficient to fully address risk identification, background redundancy, and value estimation near the boundary of the data support in high\-density traffic\.

In contrast, DiDrive achieves the highest success rate and average reward across all three scenarios\. In the complex scenario with 60 vehicles, DiDrive still maintains a success rate of 85\.00%, an average reward of 4295\.68, and the longest average driving distance of 254 m\. These results show that the advantage of DiDrive does not stem from a single metric, but is reflected in a more stable balance among safety, traffic efficiency, and trajectory quality\. On the one hand, 3DICE reduces the influence of out\-of\-distribution value overestimation and heavy\-tailed sample perturbations on policy updates through in\-sample guidance, stabilized optimization, and support\-set candidate selection\. On the other hand, RHDif enables the model to focus more on key interactions that affect safety boundaries in high\-dimensional traffic inputs through risk\-aware hierarchical representation\. By combining these two components, DiDrive maintains strong closed\-loop robustness even as traffic density increases\.

Table 2:Generalization performance on Town05 under different traffic densities\. We report Success Rate, Average Reward, and Average Driving Distance over 100 closed\-loop evaluation episodes\.ScenarioMetricMethodIQLCQLODICETD3\-BCDiffusion\-QLQGPODiDrive \(Ours\)Simple20 VehiclesSuccess Rate↑\\uparrow72%85%8%81%82%72%89%Avg\. Reward↑\\uparrow2877\.132183\.611116\.173971\.173393\.62\-22\.704397\.61Avg\. Distance \(m\)↑\\uparrow29918810233621855306Medium40 VehiclesSuccess Rate↑\\uparrow68%81%7%65%77%63%86%Avg\. Reward↑\\uparrow2869\.622078\.35882\.483570\.023311\.9988\.804257\.76Avg\. Distance \(m\)↑\\uparrow3041798830121661299Complex60 VehiclesSuccess Rate↑\\uparrow58%70%7%35%66%49%81%Avg\. Reward↑\\uparrow2690\.991611\.80502\.232715\.143011\.24103\.224085\.39Avg\. Distance \(m\)↑\\uparrow2641457825119958283

To further evaluate the cross\-town generalization ability of the model, we conduct additional closed\-loop tests on Town05\. Compared to Town03, Town05 exhibits significantly different road layouts\. Therefore, testing on this map rigorously verifies whether the policy can effectively transfer to new road layouts and interaction patterns, rather than merely overfitting to the training environment\[Li\_2024\_CVPR\_EgoStatus\]\. As shown in Table[2](https://arxiv.org/html/2609.01609#S4.T2), DiDrive achieves success rates of 89\.00%, 86\.00%, and 81\.00% under simple, medium, and complex traffic densities, respectively, with corresponding average rewards of 4397\.61, 4257\.76, and 4085\.39\. These results indicate that DiDrive maintains strong closed\-loop performance under unseen town environments\.

As traffic density increases, the baseline methods exhibit varying degrees of performance degradation on Town05\. TD3\-BC shows certain competitiveness in the simple scenario, but its success rate drops to 35\.00% in the complex scenario\. This indicates that policies relying on behavior cloning constraints are prone to insufficient generalization when road environments and traffic interactions change\. CQL still maintains a relatively high success rate on Town05, reaching 70\.00% in the complex scenario, but its average reward is only 1611\.80\. This suggests that the inherent conservatism of CQL is further exacerbated in unseen environments\. While safe, the policy becomes overly timid and struggles to maintain driving efficiency when faced with unfamiliar road topologies\.

Diffusion\-QL achieves a success rate of 66\.00% and an average reward of 3011\.24 in the complex scenario, outperforming ODICE and QGPO overall\. This indicates that the multimodal expressiveness of diffusion policies provides certain benefits for cross\-town generalization\. However, improving generative capability alone remains insufficient to fully address risk identification and action selection in new towns\. ODICE achieves success rates below 10% across all three traffic densities, indicating that traditional distribution correction methods may suffer from unstable density\-ratio estimation when facing new state distributions\. QGPO obtains low average rewards, even reaching \-22\.70 in the simple scenario, suggesting that purely generative policies without stable data\-support constraints are vulnerable to distribution shift and accumulated closed\-loop errors\.

Overall, the Town05 generalization experiments show that the performance advantage of DiDrive is not limited to Town03\. Compared with baseline methods that rely on conservative value constraints, behavior cloning constraints, or pure generative expressiveness, DiDrive maintains high success rates and high average rewards under new road environments and traffic distributions\. This result supports the central claim of this paper: offline reinforcement learning for autonomous driving must not only address out\-of\-distribution value overestimation in the action space, but also improve the model’s adaptability to key traffic interactions in new scenarios through risk\-aware representation\.

### 4\.3Ablation Study

#### 4\.3\.1Training Analysis

To understand the individual contributions of our proposed modules to training stability, we compare the learning curves of various ablated variants\.

![[Uncaptioned image]](https://arxiv.org/html/2609.01609v1/Ablation_TotalReward.png)

Figure 6:Total reward analysis under different ablation settings\.
![[Uncaptioned image]](https://arxiv.org/html/2609.01609v1/Ablation_LaneDeviation.png)

Figure 7:Lane deviation analysis under different ablation settings\.
Figs\.[6](https://arxiv.org/html/2609.01609#S4.F6)and[7](https://arxiv.org/html/2609.01609#S4.F7)further reveal the synergistic mechanism among the components of DiDrive\. The full model, represented by the blue curve, exhibits the most stable training trend: its reward rapidly converges to around 5200 in the early training stage, while the lane deviation remains stable at approximately 0\.10 m\. In sharp contrast, the variant without the 3DICE paradigm, represented by the yellow curve, suffers the most significant performance degradation\. Its reward exhibits severe sawtooth\-like oscillations, and the vehicle deviation frequently exceeds 0\.60 m\. This indicates that, without distribution correction guidance and stabilized optimization, the policy is easily affected by out\-of\-distribution action overestimation and heavy\-tailed sample perturbations, making it difficult to maintain a stable driving trajectory\.

The partial ablation of the representation modules further highlights the synergistic design of RHDif\. When the CMCM module is removed, as depicted by the green curve, the policy exhibits pronounced lateral oscillations and severe reward fluctuations\. This instability suggests that without high\-level contextual filtering, the model’s raw local risk responses are easily hijacked by background noise or pseudo\-risks, leading to overreactive and erratic evasive maneuvers\. Conversely, as shown by the orange curve, removing the RGSE module yields a relatively smooth lane\-keeping trajectory, yet the total reward noticeably underperforms the full model\. This divergence indicates a lack of sensitivity to critical local interactions; without RGSE’s bottom\-up feature enhancement, the agent struggles to swiftly identify and react to sudden, close\-range threats like vehicle cut\-ins, ultimately penalizing its overall driving efficiency and safety score\. Finally, the basic variant lacking both RGSE and CMCM, represented by the purple curve, manages to maintain baseline lateral stability on straightforward road segments, but its total reward permanently stagnates at around 3000\. This stark performance drop demonstrates that a standard diffusion backbone, stripped of hierarchical risk awareness, is fundamentally ill\-equipped to handle fine\-grained multi\-vehicle interactions and complex, long\-tailed traffic scenarios\.

Ultimately, these results confirm that representation filtering and policy optimization are fundamentally interdependent\. Only when RHDif’s risk\-calibrated perception is coupled with 3DICE’s stable optimization can the model achieve a favorable balance between driving efficiency and safe control in complex traffic scenarios\.

#### 4\.3\.2Testing Analysis

Beyond training dynamics, we further quantify how each decoupled component specifically contributes to the final closed\-loop safety and driving efficiency\.

Table 3:Ablation study of different components in the proposed framework\.ComparisonComponentsSimple Scenario \(20 Vehicles\)Medium Scenario \(40 Vehicles\)Complex Scenario \(60 Vehicles\)3DICERGSECMCMAvg\. Dist \(m\)↑\\uparrowAvg\. Reward↑\\uparrowSR↑\\uparrowAvg\. Dist \(m\)↑\\uparrowAvg\. Reward↑\\uparrowSR↑\\uparrowAvg\. Dist \(m\)↑\\uparrowAvg\. Reward↑\\uparrowSR↑\\uparrowA✓✓2543913\.0173%2393645\.5066%2133313\.0444%B✓2843736\.1286%2603629\.9783%2413385\.7978%C✓✓2623866\.4677%2543852\.5376%2353594\.6767%D✓✓2884407\.9390%2724211\.1184%2453754\.9080%E✓✓✓2864902\.9492%2994257\.7686%2544295\.6885%

By constructing five variants, we further evaluate the individual contributions of 3DICE, RGSE, and CMCM in DiDrive\. The quantitative results under three traffic densities are summarized in Table[3](https://arxiv.org/html/2609.01609#S4.T3)\. Overall, the performance improvement of DiDrive does not come from a single component, but relies on the coordination between the policy optimization mechanism and the risk\-aware representation module\.

First, the 3DICE paradigm, as indicated by the comparison between Row A and Row E, serves as the core foundation for stable system optimization\. After removing this module, the performance drops significantly across all densities: for instance, the success rate falls from 86% to 66% in the medium scenario, and from 85% to 44% in the complex scenario, where the average reward also decreases from 4295\.68 to 3313\.04\. This indicates that relying solely on the diffusion backbone is insufficient to handle long\-tail risks and distribution shifts in high\-dimensional driving decisions\. Without in\-sample guidance and support\-set constraints, the generated actions are more susceptible to out\-of\-distribution value overestimation, making the policy unstable in complex interactive environments\.

Second, RGSE and CMCM play different roles in the spatiotemporally decoupled hierarchical diffusion architecture: RGSE focuses on amplifying local risk responses, while CMCM provides global semantic filtering to calibrate these signals\. With 3DICE retained, introducing only RGSE, as shown in Row C, yields success rates of 76% and 67% in the medium and complex scenarios, with average driving distances of 254 m and 235 m, respectively\. These metrics fall noticeably short of the 83% and 78% success rates, alongside the 260 m and 241 m average driving distances achieved by the baseline without any representation module in Row B\. This result demonstrates that enhancing local risk responses alone does not necessarily lead to better performance\. Without the global semantic filtering provided by CMCM, RGSE may overly amplify the local responses of non\-critical vehicles or background objects, leading to behaviors such as excessive avoidance, frequent sudden braking, or conservative stagnation\.

In contrast, the variant with only CMCM added, as shown in Row D, performs more stably across varying densities\. It achieves an average reward of 4211\.11 in the medium scenario and a success rate of 80% with a 3754\.90 reward in the complex scenario, generally outperforming both Row B and Row C\. This suggests that cross\-modal contextual modulation can suppress background interference and filter local pseudo\-risk responses\. However, without the low\-level risk enhancement provided by RGSE, the model’s sensitivity to sudden dangerous interactions remains limited, and its performance is therefore still lower than that of the full model\.

After simultaneously introducing RGSE and CMCM, the full model in Row E achieves the highest performance across all metrics\. Notably, it reaches an 86% success rate and a 299 m driving distance in the medium scenario, while improving the success rate in the complex scenario to 85% and achieving the highest average reward of 4295\.68\. This result indicates that RGSE provides low\-level enhancement for local risk interactions, while CMCM performs global semantic correction on local high\-response features\. By combining the two, the model can focus on key objects that may compress the safety boundary while reducing interference from background redundancy and pseudo\-risk signals, thereby maintaining more stable decision\-making in complex vehicle interactions and dense traffic environments\.

#### 4\.3\.3Risk\-Aware Module Analysis

To further examine the functional roles of the proposed risk\-aware modules, we introduce additional diagnostic metrics beyond standard closed\-loop evaluation\. While success rate and collision rate reflect final driving outcomes, they do not directly reveal how each module contributes to risk perception and feature organization\. Therefore, we first analyze the internal behavior of the CMCM module, focusing on whether the local risk signals amplified by RGSE can be effectively organized, filtered, and aligned with the diagnostic reference risk distribution under global contextual semantics\. To this end, we adopt four evaluation metrics: Risk Gate Activation and Risk Attention Alignment as internal diagnostic metrics, alongside Collision Rate and Success Rate as closed\-loop indicators\.

Risk Gate Activation:This metric measures the average intensity of the risk\-gated feature response, defined as:

Risk Gate Activation=𝔼​\[risk\_gate−0\.5\],\\text\{Risk Gate Activation\}=\\mathbb\{E\}\[\\text\{risk\\\_gate\}\-0\.5\],\(18\)whererisk\_gate=σ​\(MLP​\(𝐫r​i​s​k\)\)\+0\.5\\text\{risk\\\_gate\}=\\sigma\(\\text\{MLP\}\(\\mathbf\{r\}\_\{risk\}\)\)\+0\.5\. Here,𝐫r​i​s​k\\mathbf\{r\}\_\{risk\}denotes the risk\-related input vector, including ego speed, lane deviation, front\-vehicle distance, and relative velocity\. A higher value indicates stronger activation of risk\-related modalities or channels\. Since both the full model and the w/o CMCM variant contain RGSE, this metric reveals whether the risk features amplified by RGSE can be effectively preserved after contextual modulation\.

Risk Attention Alignment:This metric further measures whether the learned risk\-aware attention is consistent with a reference risk distribution derived from driving\-state indicators\. It is computed as the cosine similarity between the learned modal risk weights and the reference risk distribution:

Risk Attention Alignment=cos⁡\(𝐰l​e​a​r​n​e​d,𝐫t​r​u​e\)\.\\text\{Risk Attention Alignment\}=\\cos\(\\mathbf\{w\}\_\{learned\},\\mathbf\{r\}\_\{true\}\)\.\(19\)Here,𝐰l​e​a​r​n​e​d\\mathbf\{w\}\_\{learned\}is obtained from the combination of the RGSE risk activation and the CMCM spatial gate\. The reference vector𝐫t​r​u​e\\mathbf\{r\}\_\{true\}is constructed only for diagnostic evaluation and is not used for model training or action selection\.

Specifically,𝐫t​r​u​e\\mathbf\{r\}\_\{true\}is computed from normalized rule\-based risk indicators associated with different input modalities:

𝐫t​r​u​e=Normalize​\[re​g​o,rl​a​n​e,rl​i​d​a​r,rv​e​h,rw​p\],\\mathbf\{r\}\_\{true\}=\\text\{Normalize\}\\left\[r\_\{ego\},r\_\{lane\},r\_\{lidar\},r\_\{veh\},r\_\{wp\}\\right\],\(20\)wherere​g​or\_\{ego\}denotes the ego\-state risk derived from excessive speed or acceleration,rl​a​n​er\_\{lane\}denotes the lane\-deviation risk derived from the absolute lateral offset,rl​i​d​a​rr\_\{lidar\}denotes the obstacle\-proximity risk derived from the minimum LiDAR distance,rv​e​hr\_\{veh\}denotes the surrounding\-vehicle interaction risk derived from front\-vehicle distance and relative velocity, andrw​pr\_\{wp\}denotes the waypoint\-tracking risk derived from the deviation between the current motion direction and the navigation waypoint direction\. The normalization operation rescales these indicators into a comparable distribution across modalities:

Normalize​\(𝐱\)=𝐱∑mxm\+ϵ,\\text\{Normalize\}\(\\mathbf\{x\}\)=\\frac\{\\mathbf\{x\}\}\{\\sum\_\{m\}x\_\{m\}\+\\epsilon\},\(21\)whereϵ\\epsilonis a small constant for numerical stability\. A higher alignment score indicates that the learned attention focuses more consistently on the modalities associated with higher reference risk rather than irrelevant background responses\. Therefore, this metric is used only to analyze the internal behavior of RGSE and CMCM, rather than serving as an optimization objective\.

![[Uncaptioned image]](https://arxiv.org/html/2609.01609v1/fullmodel_wo_CMCM_risk_gate_activation.png)

\(a\) Risk Gate Activation

![[Uncaptioned image]](https://arxiv.org/html/2609.01609v1/fullmodel_wo_CMCM_risk_attention_alignment.png)

\(b\) Risk Attention Alignment

![[Uncaptioned image]](https://arxiv.org/html/2609.01609v1/fullmodel_wo_CMCM_collision_rate.png)

\(c\) Collision Rate

![[Uncaptioned image]](https://arxiv.org/html/2609.01609v1/fullmodel_wo_CMCM_success_rate.png)

\(d\) Success Rate

Figure 8:Effectiveness analysis of CMCM in complex scenarios\.
As shown in Fig\.[8](https://arxiv.org/html/2609.01609#S4.F8), the comparison between the full model and the w/o CMCM variant reveals the role of CMCM in risk\-aware feature calibration\. Since RGSE is retained in both models, the difference lies in whether the amplified local risk features are further processed by cross\-modal contextual modulation\. Risk Gate Activation reflects the strength of risk\-aware responses, but higher activation alone does not necessarily imply better safety, as misaligned activation may correspond to false risk responses\. Therefore, it must be interpreted alongside Risk Attention Alignment\.

The improvement in Risk Attention Alignment indicates that CMCM successfully aligns learned attention with the diagnostic reference risk distribution\. Rather than indiscriminately amplifying all risk\-related features, CMCM selectively filters background interference to preserve semantically meaningful risk cues\. This top\-down contextual calibration enables the policy to distinguish genuine threats—such as close leading vehicles or high\-risk surrounding agents—from irrelevant background noise\. Consequently, the closed\-loop metrics further validate this role: the integration of CMCM yields a higher success rate and a lower collision rate, confirming that improved risk\-attention alignment translates into safer driving behavior\.

Next, we examine the RGSE module to verify whether the model can perceive local safety\-critical interactions and respond properly under high\-risk conditions\. While CMCM focuses on global calibration, RGSE is designed to enhance bottom\-up local risk perception\. Thus, the RGSE ablation focuses on whether removing this module weakens the policy’s capacity to identify near\-collision states and suppress unsafe acceleration\. We adopt four evaluation metrics: TTC<2\.5<2\.5s Rate and High\-risk Throttle Mean as internal diagnostic metrics, alongside Collision Rate and Success Rate as closed\-loop indicators\.

TTC<2\.5<2\.5s Rate:The proportion of evaluation timesteps where the Time\-To\-Collision \(TTC\) falls below2\.52\.5seconds\. TTC estimates the time remaining before a potential rear\-end collision at the current relative velocity\. We adopt the 2\.5\-second threshold because it aligns with standard human perception\-reaction times in traffic engineering\. Falling below this margin effectively marks a high\-risk, near\-collision state; therefore, a lower rate indicates safer temporal margins\.

High\-risk Throttle Mean:The average throttle output during high\-risk states\. In our evaluation, a high\-risk state is triggered either when the temporal safety margin is critically low \(TTC<2\.5​s\\text\{TTC\}<2\.5\\text\{ s\}\) or when a composite environmental risk score \(integrating factors like front\-vehicle distance, lane deviation, and LiDAR obstacles\) exceeds0\.50\.5\. A lower mean throttle indicates that the policy has learned to effectively suppress acceleration when potential danger is detected, reflecting a highly risk\-sensitive control strategy integrated into the training paradigm\.

![[Uncaptioned image]](https://arxiv.org/html/2609.01609v1/fullmodel_wo_rgse_ttc_below_2p5_rate.png)

\(a\) TTC<2\.5<2\.5s Rate

![[Uncaptioned image]](https://arxiv.org/html/2609.01609v1/fullmodel_wo_rgse_high_risk_throttle_mean.png)

\(b\) High\-risk Throttle Mean

![[Uncaptioned image]](https://arxiv.org/html/2609.01609v1/fullmodel_wo_rgse_collision_rate.png)

\(c\) Collision Rate

![[Uncaptioned image]](https://arxiv.org/html/2609.01609v1/fullmodel_wo_rgse_success_rate.png)

\(d\) Success Rate

Figure 9:Effectiveness analysis of RGSE in complex scenarios\.
As shown in Fig\.[9](https://arxiv.org/html/2609.01609#S4.F9), removing RGSE leads to degradation in both diagnostic risk\-aware metrics and closed\-loop safety performance\. Compared with the full model, the w/o RGSE variant shows a higher TTC<2\.5<2\.5s Rate, increasing from 30\.0% to 33\.0%\. This indicates that the ego vehicle enters near\-collision states more frequently when RGSE is removed\. Since TTC directly reflects the temporal margin before a potential rear\-end collision, this result suggests that CMCM alone is insufficient to capture local interaction risks such as close leading vehicles, sudden cut\-ins, and short\-distance traffic conflicts\.

The High\-risk Throttle Mean further reveals the control\-level effect of removing RGSE\. The w/o RGSE variant produces a higher throttle output under high\-risk states, increasing from 0\.390 to 0\.406\. Although the numerical difference is moderate, it indicates that the policy without RGSE is less capable of suppressing acceleration when potential hazards are present\. In dense traffic scenarios, even a small increase in throttle under risky conditions may reduce the available reaction time and increase the probability of entering near\-collision states\. This result shows that RGSE contributes not only to risk feature extraction but also to safer control behavior by encouraging the policy to reduce acceleration under dangerous conditions\.

The closed\-loop metrics provide further evidence for the effectiveness of RGSE\. Without RGSE, the collision rate increases from 13\.0% to 18\.0%, while the success rate decreases from 85\.0% to 78\.0%\. These results are consistent with the diagnostic metrics: weaker local risk perception leads to more frequent near\-collision exposure and less effective throttle suppression, which eventually results in more collisions and fewer successful episodes\. Therefore, RGSE plays a crucial role in enhancing the policy’s sensitivity to local safety\-critical interactions\.

Overall, these results demonstrate that RGSE is responsible for bottom\-up local risk amplification in DiDrive\. By dynamically enhancing spatiotemporal features related to surrounding vehicles, front\-distance risks, lane deviations, and obstacle proximity, RGSE enables the policy to identify hazardous situations earlier and suppress unsafe acceleration more effectively\. In contrast, removing RGSE weakens the model’s ability to perceive local interaction risks, making the policy more likely to enter near\-collision states and increasing the probability of collision\. Together with the CMCM analysis, these results show that RGSE and CMCM play complementary roles: RGSE enhances sensitivity to local safety\-critical cues, while CMCM calibrates these risk responses through global contextual semantics\. Their combination enables DiDrive to achieve safer and more robust decision\-making in complex traffic scenarios\.

## 5Conclusion

This paper addresses training instability in offline reinforcement learning for autonomous driving caused by distribution shift, high\-dimensional traffic redundancy, and heavy\-tailed risk signals\. To tackle these challenges, we proposed DiDrive, a risk\-aware hierarchical diffusion framework\. By synergizing the 3DICE policy optimization paradigm with the RHDif representational architecture, DiDrive performs risk\-aware policy redistribution within the support of real driving data, substantially decreasing the reliance on extrapolated value estimation for out\-of\-distribution actions\.

Extensive closed\-loop evaluations in the high\-fidelity CARLA simulator demonstrate the superiority of our approach\. In high\-density complex traffic scenarios involving 60 background vehicles, DiDrive achieved the highest success rate of 85% and a peak average reward of 4295\.68, significantly outperforming state\-of\-the\-art baselines such as Diffusion\-QL and CQL, which achieved success rates of 77% and 74% respectively\. Furthermore, in zero\-shot cross\-town generalization tests, DiDrive maintained a robust 81% success rate in complex configurations\. These quantitative results prove that our risk\-aware hierarchical representation successfully extracts generalized safety\-critical features rather than overfitting to specific map structures\.

Despite these advancements, this work has certain limitations\. First, while the framework stabilizes training, the multi\-step reverse denoising of diffusion models inherently introduces inference latency, posing challenges for high\-frequency real\-time deployment on vehicle\-embedded platforms\. Second, our current evaluations are conducted within a simulator, leaving the simulation\-to\-reality domain gap unexplored\. Future work will focus on integrating accelerated sampling techniques, such as consistency models, to reduce inference latency\. Additionally, we plan to validate the framework on real\-world autonomous driving datasets and explore continuous simulation\-to\-reality transfer strategies for physical vehicle deployment\.

\\printcredits

## References