Threat-guided Policy-aware Scene Perturbation for Safe Autonomous Driving with Online Reinforcement Learning

# Threat-guided Policy-aware Scene Perturbation for Safe Autonomous Driving with Online Reinforcement Learning
Source: [https://arxiv.org/html/2608.10403](https://arxiv.org/html/2608.10403)
Xincong Hu1, Lei Ou111footnotemark:1, Maosen Li2, Jingtao Zhang2, Liguo Hou2, Zongzhang Zhang1This work was done during the internship at Yinwang Intelligent Technology Co\., Ltd\.Corresponding author: zzzhang@nju\.edu\.cn

###### Abstract

Reinforcement learning \(RL\) has shown promising performance in autonomous driving, yet ensuring the safety of online RL policies remains challenging due to insufficient exposure to safety\-critical driving scenes\. The long\-tailed nature of real\-world traffic situations makes dangerous and rare interactions difficult to encounter through conventional sampling, limiting the ability of RL policies to learn robust safety behaviors\. Existing methods improve training diversity by synthesizing challenging scenes or adversarial situations\. However, these approaches typically optimize scene generation objectives separately from the evolving policy, without explicitly modeling how generated perturbations relate to the current policy’s weaknesses and learning needs\. In this paper, we propose Threat\-guided Policy\-aware Scene Perturbation \(TPSP\) for safe autonomous driving with online RL\. TPSP introduces a policy\-aware scene encoder to capture the interaction between policy behaviors and surrounding environments, enabling scene perturbation aligned with the current policy\. Based on this representation, TPSP selectively perturbs critical objects rather than applying uniform modifications across the scene\. Furthermore, we develop a threat\-guided optimization strategy that evaluates perturbed scenes through threat\-level differences between policy rollouts on original and perturbed scenes, guiding the generation of safety\-critical scenes with higher training value\. Comprehensive experiments demonstrate that TPSP improves safety learning efficiency, achieving strong safety performance on NAVSIM v2 with approximately 4 million kilometers of simulated driving data\. Ablation studies verify that policy\-aware targeted perturbations provide more informative safety\-critical experiences than random or policy\-unaware strategies, enabling safer driving under limited interaction budgets\.

## Introduction

Reinforcement learning \(RL\) has emerged as a promising approach for autonomous driving by enabling policies to learn complex decision\-making strategies through environment interaction\(Kiranet al\.[2022](https://arxiv.org/html/2608.10403#bib.bib29); Cusumano\-Towneret al\.[2025](https://arxiv.org/html/2608.10403#bib.bib20); Kazemkhaniet al\.[2025](https://arxiv.org/html/2608.10403#bib.bib2)\)\. Unlike rule\-based systems relying on handcrafted objectives\(Padenet al\.[2016](https://arxiv.org/html/2608.10403#bib.bib27)\)and supervised approaches requiring expert demonstrations\(Codevillaet al\.[2018](https://arxiv.org/html/2608.10403#bib.bib28); Bansalet al\.[2019](https://arxiv.org/html/2608.10403#bib.bib30)\), RL can adapt driving behaviors through large\-scale experience collection\. However, achieving safe and robust driving remains challenging due to the scarcity of safety\-critical long\-tail scenes, where rare events such as collisions provide insufficient training signals for learning reliable safety behaviors\.

To improve the exposure of RL policies to challenging driving scenes, prior studies explore scene augmentation and perturbation strategies to enrich training experiences\. Representative approaches, including prioritized replay\(Schaulet al\.[2016](https://arxiv.org/html/2608.10403#bib.bib7)\), curriculum learning\(Bengioet al\.[2009](https://arxiv.org/html/2608.10403#bib.bib8)\), and adversarial scene generation\(Xuet al\.[2022](https://arxiv.org/html/2608.10403#bib.bib6)\), focus on selecting informative samples or constructing failure\-inducing scenes\. However, existing methods typically characterize scene difficulty based on environment\-level objectives, such as collision probability, adversarial objectives, or predefined safety violations\(Dinget al\.[2020](https://arxiv.org/html/2608.10403#bib.bib32); Wanget al\.[2021](https://arxiv.org/html/2608.10403#bib.bib5); Xuet al\.[2022](https://arxiv.org/html/2608.10403#bib.bib6); Chenet al\.[2025](https://arxiv.org/html/2608.10403#bib.bib23)\)\. Consequently, the generated scenes are often independent of the evolving policy: a specific event, such as a cut\-in or sudden braking, may be trivial for a mature policy but challenging for an immature one\. This limitation prevents existing difficulty\-oriented strategies from effectively targeting the current policy weaknesses during online RL training\.

![Refer to caption](https://arxiv.org/html/2608.10403v1/x1.png)Figure 1:Overview of the proposed framework\.Motivated by this observation, we propose that an effective scene perturbation should consider the current policy behavior instead of relying on predefined scene modifications\. Since different policies may fail under different traffic conditions, policy\-agnostic perturbations often provide limited training benefits\. Therefore, a more effective perturbation strategy should adapt to current policy characteristics and identify the safety\-critical situations that are most relevant to its limitations\. Based on this insight, we propose TPSP, a Threat\-guided Policy\-aware Scene Perturbation framework for safe autonomous driving with online RL\. TPSP identifies and perturbs policy\-relevant challenging scenes, enabling more effective safety\-oriented policy optimization\.

As illustrated in Figure[1](https://arxiv.org/html/2608.10403#Sx1.F1), TPSP consists of three key components: a policy\-aware scene encoder, a targeted scene perturbation module, and a threat\-guided scene perturbation optimization method\. First, the policy\-aware scene encoder integrates ego states, surrounding objects, road context, scene risk information, and the current policy’s action distribution to construct a policy\-aware scene representation\. This representation provides both scene\-level and policy\-level information for perturbation generation\. With this representation, targeted scene perturbation selectively modifies critical objects instead of applying uniform perturbations across the scene\. Finally, TPSP evaluates perturbed scenes by computing the threat\-level differences between the policy rollouts on perturbed and original scenes\. The resulting threat signal optimizes the scene perturbation network, guiding it toward discovering safety\-critical scenes with higher training value for autonomous driving policy improvement\.

Extensive experiments on autonomous driving benchmarks demonstrate that TPSP improves safety learning efficiency under limited online RL interaction budgets\. With roughly 4 million kilometers of simulated driving data, TPSP effectively discovers policy\-relevant safety\-critical experiences and enhances policy robustness\. On the NAVSIM v2navhard\_two\_stagebenchmark\(Caoet al\.[2025](https://arxiv.org/html/2608.10403#bib.bib17)\), TPSP achieves 99\.8% NC and 99\.6% TTC on Stage 1 and maintains strong performance in the more challenging Stage 2 setting with 96\.7% NC and 94\.6% TTC, outperforming previous best results by 2\.2% and 1\.8%, respectively\. These results demonstrate that policy\-aware exploration enables more efficient utilization of limited interaction data for learning safer driving behaviors\. In this work, our contributions are summarized as follows:

- •We propose TPSP, a novel threat\-guided perturbation framework designed to generalize safety\-critical scenes for safe autonomous driving\. By targeting the current policy, TPSP generates high\-risk interaction cases to robustly train safe driving behaviors\.
- •Within TPSP, we quantify the threat by comparing pre\- and post\-perturbation rollouts to identify high\-risk samples\. The scene perturbation module is then trained with the objective of maximizing these threat scores\.
- •Extensive experiments demonstrate that TPSP significantly outperforms baseline methods in both driving safety and data efficiency, highlighting its effectiveness in training robust driving policies\.

## Related Work

##### Reinforcement learning for autonomous driving\.

Reinforcement learning \(RL\) has been widely studied for closed\-loop autonomous driving, where policies are optimized through continuous interaction with dynamic environments\. To support RL\-based training, various simulation platforms have been developed to improve scene diversity, scalability, and realism\. CARLA\(Dosovitskiyet al\.[2017](https://arxiv.org/html/2608.10403#bib.bib3)\)and SMARTS\(Zhouet al\.[2021](https://arxiv.org/html/2608.10403#bib.bib4)\)provide high\-fidelity simulation and multi\-agent interaction, while MetaDrive\(Liet al\.[2023](https://arxiv.org/html/2608.10403#bib.bib33)\)and GPUDrive\(Kazemkhaniet al\.[2025](https://arxiv.org/html/2608.10403#bib.bib2)\)further enhance scene generation and large\-scale parallel experience collection\. Recent works also explore more realistic and effective RL training paradigms, such as 3DGS\-based simulation in RAD\(Gaoet al\.[2025](https://arxiv.org/html/2608.10403#bib.bib41)\)and aligned world models in Raw2Drive\(Yanget al\.[2025](https://arxiv.org/html/2608.10403#bib.bib42)\)for driving policy optimization\. However, RL policies remain highly dependent on the quality and diversity of collected experiences, making rare and safety\-critical driving scenes difficult to discover through conventional sampling\.

##### Safety\-critical scene generation and perturbation\.

Existing works improve the exposure of driving policies to challenging situations through safety\-critical scene generation\. AdvSim\(Wanget al\.[2021](https://arxiv.org/html/2608.10403#bib.bib5)\)and KING\(Hanselmannet al\.[2022](https://arxiv.org/html/2608.10403#bib.bib21)\)optimize surrounding\-agent behaviors to construct safety\-critical interactions, while SafeBench\(Xuet al\.[2022](https://arxiv.org/html/2608.10403#bib.bib6)\)and CAT\(Zhanget al\.[2023](https://arxiv.org/html/2608.10403#bib.bib22)\)provide benchmarking and critical scene generation frameworks for robustness evaluation\. Recent approaches further explore learning\-based scene construction, including feasibility\-aware adversarial optimization\(Chenet al\.[2025](https://arxiv.org/html/2608.10403#bib.bib23)\), diffusion\-based safety\-critical synthesis\(Xuet al\.[2025](https://arxiv.org/html/2608.10403#bib.bib35)\), and collaborative adversarial scene evolution\(Liuet al\.[2026](https://arxiv.org/html/2608.10403#bib.bib36)\)\. However, these methods mainly optimize scene difficulty, failure likelihood, or adversarial object behaviors from environment\-level objectives, without explicitly considering the evolving policy and its specific learning needs\. TPSP addresses this limitation by introducing policy\-aware representations for targeted scene perturbation and optimizing perturbations with policy\-relevant threat signals\.

##### Policy\-aware reinforcement learning\.

Policy\-aware learning improves training efficiency by adapting experiences or interactions according to the current policy\. Self\-play methods such as AlphaZero\(Silveret al\.[2018](https://arxiv.org/html/2608.10403#bib.bib24)\)demonstrate the effectiveness of informative interaction generation, while several autonomous driving studies explore adaptive agent interactions and robust policy optimization\(Daiet al\.[2023](https://arxiv.org/html/2608.10403#bib.bib26); Caoet al\.[2023](https://arxiv.org/html/2608.10403#bib.bib25); Cusumano\-Towneret al\.[2025](https://arxiv.org/html/2608.10403#bib.bib20)\)\. Recent RL driving frameworks such as CaRL\(Jaegeret al\.[2025](https://arxiv.org/html/2608.10403#bib.bib39)\)and adaptive curriculum learning\(Abouelazm and others[2025](https://arxiv.org/html/2608.10403#bib.bib40)\)further adjust training experiences based on policy capability\. However, these methods focus mainly on policy optimization or experience scheduling, rather than constructing safety\-critical scenes tailored to the weaknesses of current policy\. In contrast, TPSP establishes a closed\-loop policy\-aware perturbation process, where the policy guides scene perturbation to provide more valuable experiences for online RL training\.

## Problem Formulation

We formulate closed\-loop autonomous driving as a Markov decision process \(MDP\),ℳ=\(𝒮,𝒜,𝒫,ℛ,γ\)\\mathcal\{M\}=\(\\mathcal\{S\},\\mathcal\{A\},\\mathcal\{P\},\\mathcal\{R\},\\gamma\), where𝒮\\mathcal\{S\},𝒜\\mathcal\{A\},𝒫\\mathcal\{P\},ℛ\\mathcal\{R\}, andγ\\gammadenote the state space, action space, transition function, reward function, and discount factor, respectively\. At timesteptt, the driving policyπθ​\(at\|st\)\\pi\_\{\\theta\}\(a\_\{t\}\|s\_\{t\}\)maps the current statest∈𝒮s\_\{t\}\\in\\mathcal\{S\}to an actionat∈𝒜a\_\{t\}\\in\\mathcal\{A\}\. The state contains observable information of the ego vehicle, surrounding objects, and driving environment\.

A driving policy rollout is represented as

τ=\(s0,a0,r0,…,sT\),\\displaystyle\\tau=\(s\_\{0\},a\_\{0\},r\_\{0\},\\ldots,s\_\{T\}\),\(1\)wherert∈ℛr\_\{t\}\\in\\mathcal\{R\}is the reward received at timestepttandTTrepresents the rollout horizon\.

To generate policy\-relevant safety\-critical scenes, we introduce a scene perturbation networkGϕG\_\{\\phi\}parameterized byϕ\\phi\. Given a policy\-aware scene representation,GϕG\_\{\\phi\}predicts perturbations including object speed adjustment, object spawn time shift, and ego initial state modification\.

The objective of TPSP is to optimizeGϕG\_\{\\phi\}to discover safety\-critical scenes with higher training value for policy optimization\. Specifically, TPSP updatesGϕG\_\{\\phi\}using the threat\-level difference between policy rollouts on perturbed and original scenes as the optimization signal\.

## Method

The proposed TPSP framework consists of a driving policyπθ\\pi\_\{\\theta\}and a scene perturbation networkGϕG\_\{\\phi\}\. Given policy\-aware scene representations,GϕG\_\{\\phi\}generates targeted scene perturbations, which are used to collect rollouts for policy optimization\. The perturbations are optimized through policy\-relevant threat signals derived from the comparison between policy rollouts on perturbed and original scenes, forming a closed\-loop process that enables efficient learning of robust safety behaviors\.

### Policy\-aware Scene Encoder

The effectiveness of scene perturbation depends not only on the traffic situations, but also on how the current policy interacts with the scene\. Therefore, TPSP learns a policy\-aware scene representation that integrates traffic configuration information and policy characteristics, enabling the scene perturbation network to generate perturbations aligned with the current policy\. Given a scene statests\_\{t\}, TPSP extracts heterogeneous features from the scene, including ego feature, surrounding object features, road feature, risk\-related feature and policy feature conditioned onsts\_\{t\}\.

The ego feature𝒆ego\\boldsymbol\{e\}^\{\\mathrm\{ego\}\}encodes the current kinematic state of the ego vehicle, including velocity, heading, and destination information\. For theii\-th surrounding object, we define its feature representation as𝒆iobj,i=1,…,N\\boldsymbol\{e\}^\{\\mathrm\{obj\}\}\_\{i\},i=1,\\dots,N, whereNNdenotes the number of observable objects\. Each object feature contains intrinsic attributes, such as object type and geometric properties, as well as interaction\-related information with the ego vehicle, including relative position, relative velocity, and Time\-to\-Collision \(TTC\)\. These interaction features characterize the dynamic relationship between the ego vehicle and surrounding objects\.

Moreover, the road feature𝒆road\\boldsymbol\{e\}^\{\\mathrm\{road\}\}encodes the information of the surrounding road structure, providing static environmental context for scene representation\. In addition, TPSP incorporates a scene\-level risk\-related feature𝒆risk\\boldsymbol\{e\}^\{\\mathrm\{risk\}\}to provide complementary safety information\. This feature aggregates several safety\-related indicators, including the inverse minimum distance to surrounding objects, the inverse minimum TTC, and the number of surrounding objects in the scene\.

To capture the characteristics of the current driving policy, TPSP extracts a policy feature from the hidden representation of the frozen policy network\. Specifically, letπθ¯\\pi\_\{\\bar\{\\theta\}\}denote a frozen copy of the current driving policy during rollout collection, whereθ¯\\bar\{\\theta\}represents the fixed policy parameters\. Given the original scene observation before perturbation, the hidden representation extracted fromπθ¯\\pi\_\{\\bar\{\\theta\}\}is denoted as𝐳πθ¯\\mathbf\{z\}\_\{\\pi\_\{\\bar\{\\theta\}\}\}\. The policy feature is obtained through a projection network:

𝒆policy=MLPpolicy​\(𝐳πθ¯\),\\boldsymbol\{e\}^\{\\mathrm\{policy\}\}=\\mathrm\{MLP\}^\{\\mathrm\{policy\}\}\\left\(\\mathbf\{z\}\_\{\\pi\_\{\\bar\{\\theta\}\}\}\\right\),\(2\)whereMLPpolicy​\(⋅\)\\mathrm\{MLP\}^\{\\mathrm\{policy\}\}\(\\cdot\)denotes the policy projection network that maps the hidden policy representation into the extracted policy feature𝒆policy\\boldsymbol\{e\}^\{\\mathrm\{policy\}\}\. The fixed policy parametersθ¯\\bar\{\\theta\}prevent gradients from propagating into the driving policy during scene perturbation network optimization\. This design allows the scene perturbation network to leverage the policy representation as auxiliary information while keeping policy optimization decoupled\.

These extracted features are then fed into the policy\-aware scene encoder:\(𝒆ego,𝒆iobj,𝒆road,𝒆risk,𝒆policy\)\(\\boldsymbol\{e\}^\{\\mathrm\{ego\}\},\\boldsymbol\{e\}^\{\\mathrm\{obj\}\}\_\{i\},\\boldsymbol\{e\}^\{\\mathrm\{road\}\},\\boldsymbol\{e\}^\{\\mathrm\{risk\}\},\\boldsymbol\{e\}^\{\\mathrm\{policy\}\}\)\. To model the interaction between the ego vehicle and surrounding objects, TPSP employs an attention module to aggregate object information conditioned on the ego vehicle:

𝒄obj=Attention​\(𝒆ego,\{𝒆iobj\}i=1N\),\\boldsymbol\{c\}^\{\\mathrm\{obj\}\}=\\mathrm\{Attention\}\\left\(\\boldsymbol\{e\}^\{\\mathrm\{ego\}\},\\\{\\boldsymbol\{e\}\_\{i\}^\{\\mathrm\{obj\}\}\\\}\_\{i=1\}^\{N\}\\right\),\(3\)with𝐜obj\\mathbf\{c\}^\{\\mathrm\{obj\}\}denoting the aggregated object interaction features and invalid objects masked before attention normalization\. The ego feature serves as the query, while surrounding object features provide the interaction information to be aggregated\.

Finally, the complete policy\-aware scene representation is obtained by fusing all feature components:

𝒉=MLPfusion​\(\[𝒆ego,𝒄obj,𝒆road,𝒆risk,𝒆policy\]\),\\boldsymbol\{h\}=\\mathrm\{MLP\}^\{\\mathrm\{fusion\}\}\\left\(\[\\boldsymbol\{e\}^\{\\mathrm\{ego\}\},\\boldsymbol\{c\}^\{\\mathrm\{obj\}\},\\boldsymbol\{e\}^\{\\mathrm\{road\}\},\\boldsymbol\{e\}^\{\\mathrm\{risk\}\},\\boldsymbol\{e\}^\{\\mathrm\{policy\}\}\]\\right\),\(4\)whereMLPfusion​\(⋅\)\\mathrm\{MLP\}^\{\\mathrm\{fusion\}\}\(\\cdot\)represents the feature fusion network that combines heterogeneous scene and policy feature into the final policy\-aware scene representation𝒉\\boldsymbol\{h\}\. Thus,𝒉\\boldsymbol\{h\}captures both the physical interaction context of the scene and the characteristics of the current driving policy\. This representation is then used by the subsequent scene perturbation network to generate policy\-aware scene perturbations\.

### Targeted Scene Perturbation

Given the policy\-aware scene representation𝒉\\boldsymbol\{h\}, TPSP generates targeted scene perturbations by first selecting critical objects and then modifying the corresponding scene elements\. Instead of applying uniform perturbations to all objects, TPSP focuses on objects that are more relevant to the current policy behavior\.

To identify important perturbation targets, TPSP assigns each observable object an importance score\. This is achieved by applying a Softmax function over all objects in the scene, which normalizes the raw scoring logits into a probability distribution over the object set\. Formally, the importance weight for each object is computed as:

𝒘obj=Softmax​\(\{fscore​\(\[𝒆iobj,𝒉\]\)\}i=1N\),\\boldsymbol\{w\}^\{\\mathrm\{obj\}\}=\\mathrm\{Softmax\}\\left\(\\left\\\{f^\{\\mathrm\{score\}\}\(\[\\boldsymbol\{e\}\_\{i\}^\{\\mathrm\{obj\}\},\\boldsymbol\{h\}\]\)\\right\\\}\_\{i=1\}^\{N\}\\right\),\(5\)where𝒘obj\\boldsymbol\{w\}^\{\\mathrm\{obj\}\}denotes the normalized importance scores of observable objects,NNis the number of objects in the scene, andfscore​\(⋅\)f^\{\\mathrm\{score\}\}\(\\cdot\)is an object scoring network that maps each object\-scene representation into an importance logit\. Based on these scores, TPSP selects the top\-KKobjects as perturbation targets, whose representations are denoted as:

𝑬Kobj=\{𝒆1obj,𝒆2obj,…,𝒆Kobj\}\.\\boldsymbol\{E\}^\{\\mathrm\{obj\}\}\_\{K\}=\\\{\\boldsymbol\{e\}^\{\\mathrm\{obj\}\}\_\{1\},\\boldsymbol\{e\}^\{\\mathrm\{obj\}\}\_\{2\},\\ldots,\\boldsymbol\{e\}^\{\\mathrm\{obj\}\}\_\{K\}\\\}\.\(6\)
The selected top\-KKobject representations are then combined with the policy\-aware scene representation𝒉\\boldsymbol\{h\}and provided to the scene perturbation networkGϕG\_\{\\phi\}\. Instead of directly generating deterministic modifications,GϕG\_\{\\phi\}models a Gaussian perturbation distribution and samples raw perturbation variables from this distribution\.

Specifically, the perturbation network predicts the mean and standard deviation of the Gaussian distribution:

𝝁=gμ​\(\[𝑬Kobj,𝒉\]\),\\displaystyle\\boldsymbol\{\\mu\}=g\_\{\\mu\}\(\[\\boldsymbol\{E\}^\{\\mathrm\{obj\}\}\_\{K\},\\boldsymbol\{h\}\]\),\(7\)𝝈=exp⁡\(gσ​\(\[𝑬Kobj,𝒉\]\)\),\\displaystyle\\boldsymbol\{\\sigma\}=\\exp\\left\(g\_\{\\sigma\}\(\[\\boldsymbol\{E\}^\{\\mathrm\{obj\}\}\_\{K\},\\boldsymbol\{h\}\]\)\\right\),\(8\)𝒛raw∼𝒩​\(𝝁,diag​\(𝝈2\)\),\\displaystyle\\boldsymbol\{z\}^\{\\mathrm\{raw\}\}\\sim\\mathcal\{N\}\\left\(\\boldsymbol\{\\mu\},\\mathrm\{diag\}\(\\boldsymbol\{\\sigma\}^\{2\}\)\\right\),\(9\)wheregμ​\(⋅\)g\_\{\\mu\}\(\\cdot\)andgσ​\(⋅\)g\_\{\\sigma\}\(\\cdot\)denote the mean and standard deviation prediction heads ofGϕG\_\{\\phi\}, respectively, with exponential function ensuring all standard deviation values positive\.𝝁\\boldsymbol\{\\mu\}and𝝈\\boldsymbol\{\\sigma\}represent the predicted mean and standard deviation of the perturbation distribution, and𝒛raw\\boldsymbol\{z\}^\{\\mathrm\{raw\}\}denotes the sampled unconstrained perturbation variables\. The diagonal covariance matrixdiag​\(𝝈2\)\\mathrm\{diag\}\(\\boldsymbol\{\\sigma\}^\{2\}\)assumes independent sampling across perturbation dimensions\.

TPSP then converts the raw perturbation variables into boundary indicators of the final state perturbation,

𝒛=λ​\(𝒛max⊙tanh⁡\(𝒛raw\)\),\\boldsymbol\{z\}=\\lambda\\left\(\\boldsymbol\{z\}^\{\\mathrm\{max\}\}\\odot\\tanh\(\\boldsymbol\{z\}^\{\\mathrm\{raw\}\}\)\\right\),\(10\)where𝒛max\\boldsymbol\{z\}^\{\\mathrm\{max\}\}specifies the maximum magnitude of each perturbation variable, and⊙\\odotdenotes element\-wise multiplication\.λ\\lambdais a dynamically adjusted scaling factor controlling the overall perturbation strength during training\. This transformation converts unconstrained Gaussian samples into physically feasible perturbations while preserving differentiability\.

The final perturbation variables𝒛\\boldsymbol\{z\}correspond to three types of scene modifications in TPSP: object speed adjustment, object spawn time shift, and ego initial state modification\. Object\-related perturbations are generated mainly based on the selected object representations, while ego initial state modification is conditioned on the policy\-aware scene representation𝒉\\boldsymbol\{h\}, which captures the current policy’s response to the scene, including predicted trajectory distribution or action\-value estimates\. By integrating object\-level and ego\-level adjustments in this manner, TPSP generates targeted perturbations rather than fixed or uniform modifications\.

### Threat\-guided Scene Perturbation Optimization

At the beginning of each training iteration, TPSP enriches the training scene set with informative safety\-critical experiences to improve policy robustness\. Since the original scene dataset𝒟\\mathcal\{D\}may contain insufficient high\-risk interactions, TPSP maintains a scene bufferℬ\\mathcal\{B\}to store previously discovered high\-threat perturbed scenes\. The original and perturbed scenes are combined to construct the training dataset𝒟train\\mathcal\{D\}^\{\\mathrm\{train\}\}, where their sampling proportions are dynamically adjusted throughout training\. Specifically, the proportion of high\-threat perturbed scenes is gradually increased during training, allowing the policy to progressively adapt to more challenging safety\-critical situations\.

Given any scene sampled from𝒟\\mathcal\{D\}orℬ\\mathcal\{B\}, the policy model performs rollouts based on the current state\. Each rollout is subsequently evaluated from multiple perspectives regarding environmental traffic threats\. To evaluate the threat score of a rolloutτ\\tau, TPSP first computes a step\-level threat score:

ct=\\displaystyle c\_\{t\}=\{\}wttc​ctttc\+wdist​ctdist\+wedge​ctedge\\displaystyle w^\{\\mathrm\{ttc\}\}c\_\{t\}^\{\\mathrm\{ttc\}\}\+w^\{\\mathrm\{dist\}\}c\_\{t\}^\{\\mathrm\{dist\}\}\+w^\{\\mathrm\{edge\}\}c\_\{t\}^\{\\mathrm\{edge\}\}\(11\)\+wcol​ctcol\+woff​ctoff,\\displaystyle\+w^\{\\mathrm\{col\}\}c\_\{t\}^\{\\mathrm\{col\}\}\+w^\{\\mathrm\{off\}\}c\_\{t\}^\{\\mathrm\{off\}\},wherect∈\[0,1\]c\_\{t\}\\in\[0,1\]denotes the threat score at timesteptt\. The five components represent different safety\-related factors, including time\-to\-collision, inter\-object distance, road\-edge proximity, collision occurrence, and off\-road status\. The coefficientswttcw^\{\\mathrm\{ttc\}\},wdistw^\{\\mathrm\{dist\}\},wedgew^\{\\mathrm\{edge\}\},wcolw^\{\\mathrm\{col\}\}, andwoffw^\{\\mathrm\{off\}\}are weighting factors that balance different threat components\.

Algorithm 1TPSP1:Initialize driving policy

πθ\\pi\_\{\\theta\}, scene perturbation network

GϕG\_\{\\phi\}, original dataset

𝒟\\mathcal\{D\}, scene buffer

ℬ\\mathcal\{B\}and perturbation scale

λ\\lambda
2:Set rollout policy

πθ¯←πθ\\pi\_\{\\bar\{\\theta\}\}\\leftarrow\\pi\_\{\\theta\}
3:foreach on\-policy training iteration

nndo

4:Sample scenes from the combination of the original scene dataset

𝒟\\mathcal\{D\}and the perturbed scene buffer

ℬ\\mathcal\{B\}\.

5:Extract detached policy feature from the sampled scenes using the rollout policy

πθ¯\\pi\_\{\\bar\{\\theta\}\}
6:Construct policy\-aware scene representations and generate targeted scene perturbations with

GϕG\_\{\\phi\}
7:Simulate perturbed scenes with

πθ¯\\pi\_\{\\bar\{\\theta\}\}and collect on\-policy rollouts

8:Update

πθ\\pi\_\{\\theta\}using the collected perturbed\-scene rollouts with the selected on\-policy RL algorithm

9:ifscene perturbation network updatethen

10:Simulate the same sampled original scenes without perturbations using identical simulator seeds

11:Compute threat differences using Eq\. \([15](https://arxiv.org/html/2608.10403#Sx4.E15)\)

12:Update

GϕG\_\{\\phi\}by minimizing

ℒper\\mathcal\{L\}^\{\\mathrm\{per\}\}in Eq\. \([18](https://arxiv.org/html/2608.10403#Sx4.E18)\)

13:endif

14:Update

ℬ\\mathcal\{B\}with high\-threat perturbed scenes

15:Adjust

λ\\lambdaaccording to the perturbation schedule

16:Synchronize the rollout policy:

πθ¯←πθ\\pi\_\{\\bar\{\\theta\}\}\\leftarrow\\pi\_\{\\theta\}
17:endfor

Since safety risks are usually concentrated in a small number of critical moments during a rollout, TPSP summarizes the overall rollout criticality by aggregating the most threatening timesteps\. Given a rollout trajectoryτ=\(s0,a0,r0,…,sT\)\\tau=\(s\_\{0\},a\_\{0\},r\_\{0\},\\ldots,s\_\{T\}\)with rollout horizonTT, we select the top\-kktimesteps with the highest step\-level threat scores and compute the rollout\-level criticality as:

𝒞​\(τ\)=1k′​∑t∈Top​\(τ,k\)ct,k′=min⁡\(k,T\+1\),\\mathcal\{C\}\(\\tau\)=\\frac\{1\}\{k^\{\\prime\}\}\\sum\_\{t\\in\\mathrm\{Top\}\(\\tau,k\)\}c\_\{t\},\\quad k^\{\\prime\}=\\min\(k,T\+1\),\(12\)whereTop​\(τ,k\)\\mathrm\{Top\}\(\\tau,k\)retrieves the temporal indices of the top\-kkselected critical timesteps with the highestctc\_\{t\}values\. Since a rollout of horizonTTcontainsT\+1T\+1states,k′k^\{\\prime\}adjusts the number of selected timesteps when the rollout length is shorter thankk\. By focusing on the most threatening moments rather than averaging all timesteps,𝒞​\(τ\)\\mathcal\{C\}\(\\tau\)better captures the safety\-critical characteristics of a rollout\.

In addition to the overall threat magnitude, TPSP considers the temporal instability of threat evolution during the rollout\. While𝒞​\(τ\)\\mathcal\{C\}\(\\tau\)captures critical risk levels, it does not reflect how rapidly the threat changes over time\. Therefore, TPSP introduces the threat instability metric:

𝒱​\(τ\)=1T​∑t=0T−1\|ct\+1−ct\|,\\mathcal\{V\}\(\\tau\)=\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\|c\_\{t\+1\}\-c\_\{t\}\|,\(13\)where𝒱​\(τ\)\\mathcal\{V\}\(\\tau\)measures the temporal instability of the threat evolution over the rollout\. A larger value indicates that the safety condition changes more rapidly, suggesting a more challenging interaction for the driving policy\.

Based on both threat severity and threat instability, the final threat score of a rollout is defined as:

ξ​\(τ\)=𝒞​\(τ\)​1\+β​𝒱​\(τ\)1\+β,\\mathcal\{\\xi\}\(\\tau\)=\\mathcal\{C\}\(\\tau\)\\frac\{1\+\\beta\\mathcal\{V\}\(\\tau\)\}\{1\+\\beta\},\(14\)whereβ\\betacontrols the contribution of threat variation\. The criticality term𝒞​\(τ\)\\mathcal\{C\}\(\\tau\)dominates the threat estimation to prioritize genuinely safety\-critical scenes, while𝒱​\(τ\)\\mathcal\{V\}\(\\tau\)acts as a modulation factor to emphasize temporally unstable interactions\. This formulation enables TPSP to identify informative safety\-critical scenes for policy improvement\.

![Refer to caption](https://arxiv.org/html/2608.10403v1/x2.png)Figure 2:Closed\-loop optimization between the driving policy and scene perturbation network in TPSP\.To optimize the scene perturbation network, TPSP compares the threat scores between policy rollouts on perturbed and original scenes from the same initial current state\. Specifically, both rollouts are generated using the same fixed driving policyπθ¯\\pi\_\{\\bar\{\\theta\}\}and identical simulator seeds:

Δ​ξ=ξ​\(τper\)−ξ​\(τorg\),\\Delta\\mathcal\{\\xi\}=\\mathcal\{\\xi\}\(\\tau^\{\\mathrm\{per\}\}\)\-\\mathcal\{\\xi\}\(\\tau^\{\\mathrm\{org\}\}\),\(15\)whereτper\\tau^\{\\mathrm\{per\}\}andτorg\\tau^\{\\mathrm\{org\}\}denote the perturbed and original rollouts of the same initial state, respectively\. The threat differenceΔ​ξ\\Delta\\mathcal\{\\xi\}measures the additional safety difficulty introduced by the generated perturbation\.

The perturbation networkGϕG\_\{\\phi\}is optimized using the threat difference as the learning signal\. Based on one certain initial scene or states∈𝒮s\\in\\mathcal\{S\}, for a sampled perturbation vector𝒛sraw\\boldsymbol\{z\}\_\{s\}^\{\\mathrm\{raw\}\}, its log probability is defined as:

log⁡pϕ​\(𝒛sraw\|𝑬Kobj,𝒉\)\.\\log p\_\{\\phi\}\(\\boldsymbol\{z\}\_\{s\}^\{\\mathrm\{raw\}\}\|\\boldsymbol\{E\}^\{\\mathrm\{obj\}\}\_\{K\},\\boldsymbol\{h\}\)\.\(16\)
![Refer to caption](https://arxiv.org/html/2608.10403v1/x3.png)
![Refer to caption](https://arxiv.org/html/2608.10403v1/x4.png)

Figure 3:Safety learning reward curves during online RL training\. Here, collision reward penalizes vehicle collision events, while off\-road reward penalizes curb collisions\. TPSP achieves faster improvements in collision avoidance and off\-road safety with fewer simulated driving mileage\.Scene AScene B![Refer to caption](https://arxiv.org/html/2608.10403v1/x5.png)![Refer to caption](https://arxiv.org/html/2608.10403v1/x6.png)![Refer to caption](https://arxiv.org/html/2608.10403v1/x7.png)![Refer to caption](https://arxiv.org/html/2608.10403v1/x8.png)Vanilla PPOTPSPVanilla PPOTPSPFigure 4:Qualitative comparison on two representative safety\-critical NAVSIM v2 scenes\. TPSP enables earlier hazard response and safer trajectory generation compared with Vanilla PPO\.The threat differences are normalized within each perturbation update batch to reduce the scale variation across training iterations\. The normalized threat advantage for scenessis computed as:

A^s=Δ​ξs−mean⁡\(Δ​ξ\)std⁡\(Δ​ξ\)\+ϵ,\\hat\{A\}\_\{s\}=\\frac\{\\Delta\\xi\_\{s\}\-\\operatorname\{mean\}\(\\Delta\\xi\)\}\{\\operatorname\{std\}\(\\Delta\\xi\)\+\\epsilon\},\(17\)whereA^s\\hat\{A\}\_\{s\}denotes the normalized threat advantage of sceness\. The operatorsmean⁡\(⋅\)\\operatorname\{mean\}\(\\cdot\)andstd⁡\(⋅\)\\operatorname\{std\}\(\\cdot\)compute the mean and standard deviation of the threat differences over the current perturbation update batch, respectively, andϵ\\epsilonis a small constant for numerical stability\. Thus, the scene perturbation network is optimized with:

ℒper=−𝔼s​\[A^s​log⁡pϕ​\(𝒛sraw\|𝑬Kobj,𝒉\)\]−cent​H𝒛,\\mathcal\{L\}^\{\\mathrm\{per\}\}=\-\\mathbb\{E\}\_\{s\}\[\\hat\{A\}\_\{s\}\\log p\_\{\\phi\}\(\\boldsymbol\{z\}^\{\\mathrm\{raw\}\}\_\{s\}\|\\boldsymbol\{E\}^\{\\mathrm\{obj\}\}\_\{K\},\\boldsymbol\{h\}\)\]\-c^\{\\mathrm\{ent\}\}H\_\{\\boldsymbol\{z\}\},\(18\)wherecentc^\{\\mathrm\{ent\}\}is the entropy regularization coefficient andH𝒛H\_\{\\boldsymbol\{z\}\}denotes the entropy of the perturbation distribution\. The entropy regularization encourages exploration of the perturbation space and prevents premature convergence\.

Algorithm[1](https://arxiv.org/html/2608.10403#alg1)and Figure[2](https://arxiv.org/html/2608.10403#Sx4.F2)illustrate the overall closed\-loop optimization procedure of TPSP\. During training, the generated perturbed scenes are used to collect rollouts for updating the driving policyπθ\\pi\_\{\\theta\}\. Meanwhile, the threat difference between perturbed and original rollouts provides the optimization signal for updating the scene perturbation networkGϕG\_\{\\phi\}\. By iteratively improving the driving policy based on informative perturbed scenes with PPO\(Schulmanet al\.[2017](https://arxiv.org/html/2608.10403#bib.bib1)\)algorithm and refiningGϕG\_\{\\phi\}with threat\-guided signals, TPSP establishes a closed\-loop optimization process for discovering safety\-critical training scenes\.

## Experiments

Our experiments assess TPSP’s safety and efficiency from three perspectives: learning speed, perturbation efficacy, and final performance\. We first determine if TPSP improves safety behaviors faster under fixed interaction budgets\. Next, ablation studies isolate the impact of targeted perturbations and policy\-aware representations\. We then compare TPSP’s final policy against baselines on the NAVSIM v2 benchmark\. Qualitative results further illustrate that TPSP effectively optimizes scene perturbations using threat signals from policy rollouts to generate informative safety\-critical interactions\.

### Experimental Setup

We train TPSP using the GPUDrive simulator with PPO and the NAVSIM v2navtraindataset\. The resulting driving policies are evaluated on the challenging NAVSIM v2navhard\_two\_stagebenchmark\. The evaluation considers three safety\-related metrics: no at\-fault collisions \(NC\), drivable area compliance \(DAC\), and time to collision \(TTC\)\. During training, all methods share the same policy architecture, reward function, interaction budgets, and PPO optimization hyperparameters to ensure a fair comparison\. Further implementation details are provided in Appendix A\.

### Safety Learning Efficiency Analysis

Figure[3](https://arxiv.org/html/2608.10403#Sx4.F3)illustrates the evolution of safety\-related rewards during online RL training under the same interaction budgets\. TPSP achieves faster improvements in both collision avoidance and off\-road safety rewards compared with Vanilla PPO, indicating that the policy can acquire safety\-related behaviors more efficiently with fewer simulated driving miles\. For example, at 2M miles, TPSP already reaches a collision\-reward level that Vanilla PPO does not attain until 4M miles\. This improvement comes from the targeted exposure to informative safety\-critical interactions, which provides more effective training signals than uniformly sampled experiences\.

To further analyze the learned safety behaviors, Figure[4](https://arxiv.org/html/2608.10403#Sx4.F4)presents two safety\-critical scenarios from NAVSIM v2 to analyze learned safety behaviors\. Under identical initial conditions, TPSP produces safer trajectories with earlier hazard responses\. In Scene A, TPSP proactively avoids a collision by adjusting its trajectory, while Vanilla PPO fails to react and crashes\. In Scene B, TPSP handles abrupt braking by decelerating and keeping a safe distance, whereas Vanilla PPO responds too late\. These cases highlight TPSP’s ability to improve safety learning efficiency by focusing on challenging interactions relevant to the policy’s weaknesses\.

Table 1:Ablation on NAVSIM v2navhard\_two\_stage\.
### Ablation Study

We conduct ablation studies to analyze the contribution of different components in TPSP\. We compare Vanilla PPO, random perturbation, TPSP without policy awareness \(TPSP w/o PA\), and the full TPSP framework with policy\-aware targeted perturbation\. Table[1](https://arxiv.org/html/2608.10403#Sx5.T1)reports the safety performance on the NAVSIM v2navhard\_two\_stagebenchmark\. Here, S1 and S2 denote Stage 1 and Stage 2 evaluations, respectively, where Stage 1 evaluates policies on original scenes and Stage 2 focuses on more challenging synthesized scenarios\. The upward arrows indicate that higher values correspond to better performance for the corresponding safety metrics\.

TPSP achieves the best performance across both stages, particularly in the more challenging S2 setting, where it obtains 96\.7% NC and 94\.6% TTC\. Random perturbation provides only limited improvements over Vanilla PPO, indicating that increasing scene diversity alone is insufficient for effective safety learning\. Moreover, the performance gap between TPSP and TPSP w/o PA demonstrates that incorporating policy\-specific information is essential for generating more informative safety\-critical scenes\.

### Safety Evaluation on NAVSIM v2

We show the safety of TPSP by comparing it with state\-of\-the\-art methods on the NAVSIM v2 leaderboard \(Table[2](https://arxiv.org/html/2608.10403#Sx5.T2)\)\. TPSP achieves superior performance in NC and TTC metrics across both evaluation stages, highlighting its effectiveness in collision avoidance and interaction safety\. Further details on the baseline methods are available in Appendix B\.

Table 2:Safety on NAVSIM v2navhard\_two\_stage\. External results are reported from the NAVSIM v2 benchmark leaderboard for reference\.\(Caoet al\.[2025](https://arxiv.org/html/2608.10403#bib.bib17)\)![Refer to caption](https://arxiv.org/html/2608.10403v1/x9.png)

![Refer to caption](https://arxiv.org/html/2608.10403v1/x10.png)

Figure 5:Visualization of threat\-guided policy\-aware perturbations learned by TPSP in GPUDrive simulation\. Left: a cut\-in situation; Right: a car\-following situation\. TPSP evaluates how different perturbations affect policy\-rollout threat signals and optimizes scene perturbations toward safety\-critical interactions\.
### Qualitative Analysis

Figure[5](https://arxiv.org/html/2608.10403#Sx5.F5)visualizes the learned perturbations generated by TPSP in GPUDrive simulation\. In the cut\-in situation \(left\), the generated perturbation increases the threat of the interaction withΔ​ξ=\+0\.1115\\Delta\\xi=\+0\.1115, corresponding to a more dangerous merging conflict\. In contrast, the car\-following situation \(right\) exhibits a less hazardous interaction pattern, and the estimated threat variationΔ​ξ=−0\.0540\\Delta\\xi=\-0\.0540is consistent with this intuitive observation\. These examples demonstrate that TPSP can effectively evaluate the risk variation induced by different perturbations and optimize scene modifications toward more informative safety\-critical interactions, rather than blindly increasing scene complexity\.

## Conclusion and Future Work

We proposed TPSP, a Threat\-guided Policy\-aware Scene Perturbation framework for improving safety learning efficiency in online RL for autonomous driving\. By generating targeted perturbations guided by policy weaknesses, TPSP enables more informative exploration under limited training budgets\. Experiments on NAVSIM v2 demonstrate that TPSP achieves strong safety performance with approximately 4 million kilometers of simulated driving mileage\. Ablation studies further validate the effectiveness of policy\-aware targeted perturbations\. Future work will explore more comprehensive perturbation spaces beyond the current scene\-level modifications, including richer semantic, map\-level, and multi\-agent interaction perturbations\. We also plan to investigate TPSP in larger\-scale simulation environments and real\-world driving settings, as well as its integration with more diverse RL algorithms and autonomous driving policy architectures\.

## References

- Automatic curriculum learning for driving scenarios: towards robust and efficient reinforcement learning\.InProceedings of the 36th IEEE Intelligent Vehicles Symposium,pp\. 2333–2340\.Cited by:[Policy\-aware reinforcement learning\.](https://arxiv.org/html/2608.10403#Sx2.SS0.SSS0.Px3.p1.1)\.
- M\. Bansal, A\. Krizhevsky, and A\. S\. Ogale \(2019\)ChauffeurNet: learning to drive by imitating the best and synthesizing the worst\.InProceedings of the 15th Robotics: Science and Systems,Cited by:[Introduction](https://arxiv.org/html/2608.10403#Sx1.p1.1)\.
- Y\. Bengio, J\. Louradour, R\. Collobert, and J\. Weston \(2009\)Curriculum learning\.InProceedings of the 26th Annual International Conference on Machine Learning,pp\. 41–48\.Cited by:[Introduction](https://arxiv.org/html/2608.10403#Sx1.p2.1)\.
- W\. Cao, M\. Hallgarten, T\. Li, D\. Dauner, X\. Gu, C\. Wang, Y\. Miron, M\. Aiello, H\. Li, I\. Gilitschenski, B\. Ivanovic, M\. Pavone, A\. Geiger, and K\. Chitta \(2025\)Pseudo\-simulation for autonomous driving\.InProceedings of the 9th Conference on Robot Learning,pp\. 4709–4722\.Cited by:[Appendix B](https://arxiv.org/html/2608.10403#A2.SSx1.p2.1),[Introduction](https://arxiv.org/html/2608.10403#Sx1.p5.1),[Table 2](https://arxiv.org/html/2608.10403#Sx5.T2)\.
- Z\. Cao, K\. Jiang, W\. Zhou, S\. Xu, H\. Peng, and D\. Yang \(2023\)Continuous improvement of self\-driving cars using dynamic confidence\-aware reinforcement learning\.Nature Machine Intelligence5\(2\),pp\. 145–158\.Cited by:[Policy\-aware reinforcement learning\.](https://arxiv.org/html/2608.10403#Sx2.SS0.SSS0.Px3.p1.1)\.
- K\. Chen, Y\. Lei, H\. Cheng, H\. Wu, W\. Sun, and S\. Zheng \(2025\)FREA: feasibility\-guided generation of safety\-critical scenarios with reasonable adversariality\.InProceedings of the 8th Conference on Robot Learning,pp\. 566–586\.Cited by:[Introduction](https://arxiv.org/html/2608.10403#Sx1.p2.1),[Safety\-critical scene generation and perturbation\.](https://arxiv.org/html/2608.10403#Sx2.SS0.SSS0.Px2.p1.1)\.
- F\. Codevilla, J\. Miura, A\. M\. Lopez, V\. Koltun, A\. Dosovitskiy, and R\. Urtasun \(2018\)End\-to\-end driving via conditional imitation learning\.InProceedings of the 35th IEEE International Conference on Robotics and Automation,pp\. 1–9\.Cited by:[Introduction](https://arxiv.org/html/2608.10403#Sx1.p1.1)\.
- M\. F\. Cusumano\-Towner, D\. Hafner, A\. Hertzberg, B\. Huval, A\. Petrenko, E\. Vinitsky, E\. Wijmans, T\. W\. Killian, S\. Bowers, O\. Sener, P\. Kraehenbuehl, and V\. Koltun \(2025\)Robust autonomy emerges from self\-play\.InProceedings of the 42nd International Conference on Machine Learning,pp\. 11710–11737\.Cited by:[Introduction](https://arxiv.org/html/2608.10403#Sx1.p1.1),[Policy\-aware reinforcement learning\.](https://arxiv.org/html/2608.10403#Sx2.SS0.SSS0.Px3.p1.1)\.
- Z\. Dai, T\. Zhou, K\. Shao, D\. H\. Mguni, B\. Wang, and J\. Hao \(2023\)Socially\-attentive policy optimization in multi\-agent self\-driving system\.InProceedings of the 6th Conference on Robot Learning,pp\. 946–955\.Cited by:[Policy\-aware reinforcement learning\.](https://arxiv.org/html/2608.10403#Sx2.SS0.SSS0.Px3.p1.1)\.
- W\. Ding, B\. Chen, M\. Xu, and D\. Zhao \(2020\)Learning to collide: an adaptive safety\-critical scenarios generating method\.InProceedings of the 33rd IEEE/RSJ International Conference on Intelligent Robots and Systems,pp\. 2243–2250\.Cited by:[Introduction](https://arxiv.org/html/2608.10403#Sx1.p2.1)\.
- A\. Dosovitskiy, G\. Ros, F\. Codevilla, A\. Lopez, and V\. Koltun \(2017\)CARLA: an open urban driving simulator\.InProceedings of the 1st Annual Conference on Robot Learning,pp\. 1–16\.Cited by:[Reinforcement learning for autonomous driving\.](https://arxiv.org/html/2608.10403#Sx2.SS0.SSS0.Px1.p1.1)\.
- L\. Feng, Y\. Gao, É\. Zablocki, Q\. Li, W\. Li, S\. Liu, M\. Cord, and A\. Alahi \(2026\)RAP: 3d rasterization augmented end\-to\-end planning\.InProceedings of the 14th International Conference on Learning Representations,Cited by:[Appendix B](https://arxiv.org/html/2608.10403#A2.SSx1.p2.1)\.
- H\. Gao, S\. Chen, B\. Jiang, B\. Liao, Y\. Shi, X\. Guo, Y\. Pu, H\. Yin, X\. Li, X\. Zhang,et al\.\(2025\)Rad: training an end\-to\-end driving policy via large\-scale 3dgs\-based reinforcement learning\.arXiv preprint arXiv:2502\.13144\.Cited by:[Reinforcement learning for autonomous driving\.](https://arxiv.org/html/2608.10403#Sx2.SS0.SSS0.Px1.p1.1)\.
- N\. Hanselmann, K\. Renz, K\. Chitta, A\. Bhattacharyya, and A\. Geiger \(2022\)KING: generating safety\-critical driving scenarios for robust imitation via kinematics gradients\.InProceedings of the 17th European Conference on Computer Vision,pp\. 335–352\.Cited by:[Safety\-critical scene generation and perturbation\.](https://arxiv.org/html/2608.10403#Sx2.SS0.SSS0.Px2.p1.1)\.
- B\. Jaeger, D\. Dauner, J\. Beißwenger, S\. Gerstenecker, K\. Chitta, and A\. Geiger \(2025\)CaRL: learning scalable planning policies with simple rewards\.arXiv preprint arXiv:2504\.17838\.Cited by:[Policy\-aware reinforcement learning\.](https://arxiv.org/html/2608.10403#Sx2.SS0.SSS0.Px3.p1.1)\.
- S\. Kazemkhani, A\. Pandya, D\. Cornelisse, B\. Shacklett, and E\. Vinitsky \(2025\)GPUDrive: data\-driven, multi\-agent driving simulation at 1 million FPS\.InProceedings of the 13th International Conference on Learning Representations,Cited by:[Introduction](https://arxiv.org/html/2608.10403#Sx1.p1.1),[Reinforcement learning for autonomous driving\.](https://arxiv.org/html/2608.10403#Sx2.SS0.SSS0.Px1.p1.1)\.
- B\. R\. Kiran, I\. Sobh, V\. Talpaert, P\. Mannion, A\. A\. Al Sallab, S\. Yogamani, and P\. Perez \(2022\)Deep reinforcement learning for autonomous driving: a survey\.IEEE Transactions on Intelligent Transportation Systems23\(6\),pp\. 4909–4926\.Cited by:[Introduction](https://arxiv.org/html/2608.10403#Sx1.p1.1)\.
- E\. Kirby, A\. Boulch, Y\. Xu, Y\. Yin, G\. Puy, É\. Zablocki, A\. Bursuc, S\. Gidaris, R\. Marlet, F\. Bartoccioni, A\. Cao, N\. Samet, T\. Vu, and M\. Cord \(2026\)Driving on registers\.arXiv preprint arXiv:2601\.05083\.Cited by:[Appendix B](https://arxiv.org/html/2608.10403#A2.SSx1.p2.1)\.
- Q\. Li, Z\. Peng, L\. Feng, Q\. Zhang, Z\. Xue, and B\. Zhou \(2023\)MetaDrive: composing diverse driving scenarios for generalizable reinforcement learning\.IEEE Transactions on Pattern Analysis and Machine Intelligence45\(3\),pp\. 3461–3475\.Cited by:[Reinforcement learning for autonomous driving\.](https://arxiv.org/html/2608.10403#Sx2.SS0.SSS0.Px1.p1.1)\.
- Z\. Li, W\. Yao, Z\. Wang, X\. Sun, J\. Chen, N\. Chang, M\. Shen, J\. Song, Z\. Wu, S\. Lan, and J\. M\. Alvarez \(2025\)ZTRS: zero\-imitation end\-to\-end autonomous driving with trajectory scoring\.arXiv preprint arXiv:2510\.24108\.Cited by:[Appendix B](https://arxiv.org/html/2608.10403#A2.SSx1.p2.1)\.
- J\. Liu, Y\. Guo, F\. Zhong, T\. Zhang, Z\. Jing, S\. Liang, J\. Wang, M\. Zhang, A\. Liu, and X\. Liu \(2026\)Adversarial generation and collaborative evolution of safety\-critical scenarios for autonomous vehicles\.InProceedings of the 40th AAAI Conference on Artificial Intelligence,pp\. 38926–38934\.Cited by:[Safety\-critical scene generation and perturbation\.](https://arxiv.org/html/2608.10403#Sx2.SS0.SSS0.Px2.p1.1)\.
- B\. Paden, M\. Cap, S\. Z\. Yong, D\. Yershov, and E\. Frazzoli \(2016\)A survey of motion planning and control techniques for self\-driving urban vehicles\.IEEE Transactions on Intelligent Vehicles1\(1\),pp\. 33–55\.Cited by:[Introduction](https://arxiv.org/html/2608.10403#Sx1.p1.1)\.
- T\. Schaul, J\. Quan, I\. Antonoglou, and D\. Silver \(2016\)Prioritized experience replay\.InProceedings of the 4th International Conference on Learning Representation,Cited by:[Introduction](https://arxiv.org/html/2608.10403#Sx1.p2.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[Threat\-guided Scene Perturbation Optimization](https://arxiv.org/html/2608.10403#Sx4.SSx3.p9.3)\.
- D\. Silver, T\. Hubert, J\. Schrittwieser, I\. Antonoglou, M\. Lai, A\. Guez, M\. Lanctot, L\. Sifre, D\. Kumaran, T\. Graepel,et al\.\(2018\)Mastering chess and shogi by self\-play with a general reinforcement learning algorithm\.arXiv preprint arXiv:1712\.01815\.Cited by:[Policy\-aware reinforcement learning\.](https://arxiv.org/html/2608.10403#Sx2.SS0.SSS0.Px3.p1.1)\.
- H\. Tian, T\. Li, H\. Liu, J\. Yang, Y\. Qiu, G\. Li, J\. Wang, Y\. Gao, Z\. Zhang, L\. Wang, H\. Ye, L\. Chen, and H\. Li \(2026\)SimScale: learning to drive via real\-world simulation at scale\.InProceedings of the 43rd IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 36365–36374\.Cited by:[Appendix B](https://arxiv.org/html/2608.10403#A2.SSx1.p2.1)\.
- J\. Wang, A\. Pun, J\. Tu, S\. Manivasagam, A\. Sadat, S\. Casas, M\. Ren, and R\. Urtasun \(2021\)AdvSim: generating safety\-critical scenarios for self\-driving vehicles\.InProceedings of the 38th IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 9909–9918\.Cited by:[Introduction](https://arxiv.org/html/2608.10403#Sx1.p2.1),[Safety\-critical scene generation and perturbation\.](https://arxiv.org/html/2608.10403#Sx2.SS0.SSS0.Px2.p1.1)\.
- C\. Xu, W\. Ding, W\. Lyu, Z\. Liu, S\. Wang, Y\. He, H\. Hu, D\. Zhao, and B\. Li \(2022\)SafeBench: a benchmarking platform for safety evaluation of autonomous vehicles\.InAdvances in Neural Information Processing Systems 35,pp\. 25667–25682\.Cited by:[Introduction](https://arxiv.org/html/2608.10403#Sx1.p2.1),[Safety\-critical scene generation and perturbation\.](https://arxiv.org/html/2608.10403#Sx2.SS0.SSS0.Px2.p1.1)\.
- C\. Xu, A\. Petiushko, D\. Zhao, and B\. Li \(2025\)DiffScene: diffusion\-based safety\-critical scenario generation for autonomous vehicles\.InProceedings of the 35th AAAI Conference on Artificial Intelligence,pp\. 8797–8805\.Cited by:[Safety\-critical scene generation and perturbation\.](https://arxiv.org/html/2608.10403#Sx2.SS0.SSS0.Px2.p1.1)\.
- Z\. Yang, X\. Jia, Q\. Li, X\. Yang, M\. Yao, and J\. Yan \(2025\)Raw2Drive: reinforcement learning with aligned world models for end\-to\-end autonomous driving \(in CARLA v2\)\.InAdvances in Neural Information Processing Systems 38,Cited by:[Reinforcement learning for autonomous driving\.](https://arxiv.org/html/2608.10403#Sx2.SS0.SSS0.Px1.p1.1)\.
- L\. Zhang, Z\. Peng, Q\. Li, and B\. Zhou \(2023\)CAT: closed\-loop adversarial training for safe end\-to\-end driving\.InProceedings of the 7th Conference on Robot Learning,pp\. 2357–2372\.Cited by:[Safety\-critical scene generation and perturbation\.](https://arxiv.org/html/2608.10403#Sx2.SS0.SSS0.Px2.p1.1)\.
- M\. Zhou, J\. Luo, J\. Villella, Y\. Yang, D\. Rusu, J\. Miao, W\. Zhang, M\. Alban, I\. Fadakar, Z\. Chen, C\. Huang, Y\. Wen, K\. Hassanzadeh, D\. Graves, Z\. Zhu, Y\. Ni, N\. Nguyen, M\. Elsayed, H\. Ammar, A\. Cowen\-Rivers, S\. Ahilan, Z\. Tian, D\. Palenicek, K\. Rezaee, P\. Yadmellat, K\. Shao, D\. Chen, B\. Zhang, H\. Zhang, J\. Hao, W\. Liu, and J\. Wang \(2021\)SMARTS: an open\-source scalable multi\-agent RL training school for autonomous driving\.InProceedings of the 4th Conference on Robot Learning,pp\. 264–285\.Cited by:[Reinforcement learning for autonomous driving\.](https://arxiv.org/html/2608.10403#Sx2.SS0.SSS0.Px1.p1.1)\.

## Appendix AAppendix A\. Implementation Details

This section provides additional implementation details of TPSP, including training hyperparameters, training infrastructure and safety\-related reward implementation\. These details are provided to facilitate the reproducibility of our experiments\.

### Training Hyperparameters

TPSP consists of a driving policyπθ\\pi\_\{\\theta\}optimized with PPO and a scene perturbation networkGϕG\_\{\\phi\}trained through threat\-guided optimization\. The main configurations of these components are summarized below\.

#### Driving Policy Configuration

The driving policyπθ\\pi\_\{\\theta\}is optimized using PPO in the GPUDrive environment\. The main architecture and training configurations are summarized in Table[3](https://arxiv.org/html/2608.10403#A1.T3)\.

Table 3:Architecture configuration of the driving policy\.
#### Scene Perturbation Network Configuration

The scene perturbation networkGϕG\_\{\\phi\}generates policy\-aware scene modifications based on encoded simulator states and detached policy features\. The main configurations are summarized in Table[4](https://arxiv.org/html/2608.10403#A1.T4)\.

Table 4:Configuration of the scene perturbation network\.HyperparameterValueEncoder hidden dimension128Fusion dimension256Policy feature dimension64Number of selected objectsKK8Initial log standard deviation\-1\.0OptimizerAdamLearning rate1×10−41\\times 10^\{\-4\}Entropy coefficient0\.01Gradient clipping0\.5Update interval4 rolloutsInitial perturbation scaleλ\\lambda0\.2
#### PPO Optimization Configuration

The detailed PPO optimization parameters used for updating the driving policy are summarized in Table[5](https://arxiv.org/html/2608.10403#A1.T5)\.

Table 5:PPO optimization configuration for training the driving policy\.HyperparameterValueOptimizerAdamWLearning rate3×10−43\\times 10^\{\-4\}PPO clip ratio0\.2Number of PPO epochs2Discount factorγ\\gamma0\.99GAE coefficientλGAE\\lambda\_\{\\mathrm\{GAE\}\}0\.95Experience batch size204800Mini\-batch size6400Value loss coefficient0\.5Entropy coefficient0\.01Gradient clipping0\.5

### Training Infrastructure

All experiments are conducted on the GPUDrive simulation platform\. The hardware and software configurations used for training are summarized in Table[6](https://arxiv.org/html/2608.10403#A1.T6)\.

Table 6:Training infrastructure configuration\.
### Safety\-related Reward Implementation

The driving policy is optimized using the reward provided by the GPUDrive simulator\. This section describes the implementation of two safety\-related reward components: collision and off\-road penalties\. At each simulation step, GPUDrive provides safety event indicators for each agent\. The collision signal combines collisions with vehicles and other objects:

dtcol=dtveh\+dtobj,d\_\{t\}^\{\\mathrm\{col\}\}=d\_\{t\}^\{\\mathrm\{veh\}\}\+d\_\{t\}^\{\\mathrm\{obj\}\},\(19\)wheredtvehd\_\{t\}^\{\\mathrm\{veh\}\}anddtobjd\_\{t\}^\{\\mathrm\{obj\}\}denote collision events with vehicles and other objects at timesteptt, respectively\. The off\-road signaldtoffd\_\{t\}^\{\\mathrm\{off\}\}indicates whether the ego vehicle leaves the drivable area\. The corresponding safety\-related reward terms are defined as:

rtsafe=−3\.5​dtcol−0\.75​dtoff\+0\.5​dtgoal,r\_\{t\}^\{\\mathrm\{safe\}\}=\-3\.5d\_\{t\}^\{\\mathrm\{col\}\}\-0\.75d\_\{t\}^\{\\mathrm\{off\}\}\+0\.5d\_\{t\}^\{\\mathrm\{goal\}\},\(20\)wheredtgoald\_\{t\}^\{\\mathrm\{goal\}\}denotes goal achievement\. The larger collision penalty encourages the policy to prioritize collision avoidance, while the off\-road penalty provides continuous guidance for maintaining valid driving areas\.

The collision penalty is applied only at the collision timestep since collided agents are removed by the simulator\. In contrast, off\-road penalties can accumulate when the vehicle remains outside the drivable area\. Other shaping reward components provided by GPUDrive remain unchanged during training\.

## Appendix BAppendix B\. Additional Discussion on NAVSIM Comparison and Training Pipeline

This section further discusses the comparison between TPSP and existing NAVSIM v2 leaderboard methods, and describes the training pipeline used to enable online reinforcement learning with NAVSIM scenes\. We clarify the differences in learning paradigms and input representations between TPSP and existing benchmark approaches\.

### Comparison with NAVSIM v2 Leaderboard Methods

Table[2](https://arxiv.org/html/2608.10403#Sx5.T2)compares TPSP with representative methods reported on the official NAVSIM v2 leaderboard\. These external results are included for reference under the same benchmark evaluation protocol\.

Most existing NAVSIM leaderboard methods follow an offline data\-driven end\-to\-end autonomous driving paradigm\(Caoet al\.[2025](https://arxiv.org/html/2608.10403#bib.bib17); Fenget al\.[2026](https://arxiv.org/html/2608.10403#bib.bib44); Liet al\.[2025](https://arxiv.org/html/2608.10403#bib.bib45); Tianet al\.[2026](https://arxiv.org/html/2608.10403#bib.bib46); Kirbyet al\.[2026](https://arxiv.org/html/2608.10403#bib.bib47)\)\. They typically learn driving policies or trajectory planners from large\-scale recorded driving data, where raw sensor observations provided by NAVSIM, such as multi\-view camera inputs, are directly used as model inputs\. Therefore, these approaches generally adopt a one\-stage learning pipeline that maps perception\-level observations to future trajectories or driving actions\.

In contrast, TPSP is designed as an online reinforcement learning framework that improves policy learning through interactive simulation\. Instead of directly consuming raw sensor observations, TPSP utilizes structured white\-box information available from the simulator, including ego states, surrounding object states, road information, and risk\-related features\. Based on these structured representations, TPSP first constructs a policy\-aware scene representation and then performs online RL optimization with targeted scene perturbations\.

Therefore, TPSP differs from existing NAVSIM methods in both learning paradigm and system pipeline\. Existing approaches mainly improve driving performance by developing stronger perception and planning models from offline data, whereas TPSP focuses on improving the quality of online policy training experiences through simulator\-based interaction\. Despite these differences, TPSP achieves competitive performance on NAVSIM v2 safety evaluation, demonstrating the effectiveness of online RL with policy\-aware scene optimization\.

### NAVSIM\-to\-GPUDrive Training Pipeline

TPSP requires interactive simulation for online reinforcement learning, whereas NAVSIM v2 provides recorded driving scenes for benchmark evaluation\. Therefore, the original NAVSIM v2 scenes cannot be directly used for GPUDrive simulation and training\.

To enable online training, we transform NAVSIM v2 scenes into GPUDrive\-compatible simulation environments\. The conversion process preserves essential scene information, including map structures, dynamic agent states, and temporal interactions, while enabling large\-scale parallel simulation in GPUDrive\.

After conversion, TPSP performs online reinforcement learning on the generated GPUDrive scenes\. The learned policy is finally evaluated on the official NAVSIM v2navhard\_two\_stagebenchmark following the standard evaluation protocol, allowing comparison with the publicly reported leaderboard results\.