ScenePilot: Controllable Boundary-Driven Critical Scenario Generation for Autonomous Driving

# ScenePilot: Controllable Boundary-Driven Critical Scenario Generation for Autonomous Driving
Source: [https://arxiv.org/html/2605.21168](https://arxiv.org/html/2605.21168)
###### Abstract

Safety\-critical scenarios are central to evaluating autonomous driving systems, yet their rarity in naturalistic logs makes simulation\-based stress testing indispensable\. Most scenario generation methods treat surrounding agents as adversaries, but they either \(i\) induce failures without explicitly modeling vehicle\-road physical limits, yielding visually extreme yet physically unsolvable crashes, or \(ii\) enforce physical feasibility or policy feasibility in isolation, which can over\-focus on aggressive maneuvers or remain tied to a controller\-dependent capability boundary\. We propose ScenePilot, a feasibility\-guided, boundary\-driven framework that targets the boundary band: scenarios that are physically solvable in principle yet still cause the deployed autonomy stack to fail\. We formulate generation as constrained multi\-objective reinforcement learning, combining an RSS\-derived physical\-feasibility scoreσ\\sigmawith an online\-learned AV\-risk predictorΦ\\Phi, and introduce step\-level feasibility\-aware shielding to keep exploration near the feasibility boundary while avoiding infeasible artifacts\. Experiments on SafeBench with multiple planners show that ScenePilot yields substantially higher collision rates \(\+6\.2 percentage points\) while preserving physical validity, and that adversarial fine\-tuning on these boundary\-band scenarios consistently reduces downstream crash rates\. The code is available at[https://github\.com/QiyuRuan/ScenePilot](https://github.com/QiyuRuan/ScenePilot)\.

Scenario Generation, Autonomouos Driving, Reinforcement Learning

## 1Introduction

Safety\-critical scenarios are rare in real traffic but decisive for autonomous vehicles \(AVs\)\. Large\-scale naturalistic driving logs cover everyday interactions, yet truly high\-consequence events occupy only a tiny fraction of the data \(Fig\.[1](https://arxiv.org/html/2605.21168#S1.F1)\(a\)\)\(Liu and Feng,[2024](https://arxiv.org/html/2605.21168#bib.bib18)\)\. As a result, AV stacks that appear reliable under passive replay may still fail on rare but consequential edge cases\. Although traffic\-rule compliance and online verification can check whether AV behavior satisfies formalized rules or remains within reachable safe sets during execution\(Althoffet al\.,[2025](https://arxiv.org/html/2605.21168#bib.bib59); Althoff and Dolan,[2014](https://arxiv.org/html/2605.21168#bib.bib60)\), they mainly serve as runtime assurance mechanisms rather than active tools for discovering diverse failure\-inducing interactions\. This motivates simulation\-based scenario generation as a complementary evaluation tool for systematically exposing rare and informative safety\-critical scenarios\(Sunet al\.,[2021](https://arxiv.org/html/2605.21168#bib.bib55)\)\.

![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/Fig1.png)Figure 1:Illustration of four interaction regimes relative to AV controller and physical feasibility\.In simulation, critical scenario generation is commonly posed as an adversarial problem: surrounding vehicles are treated as agents that perturb the ego vehicle toward failure and are trained with reinforcement learning or generative models to induce collisions beyond naturalistic replay frequency\(Wachi,[2019](https://arxiv.org/html/2605.21168#bib.bib27)\)\. While effective at increasing failure counts, many approaches implicitly optimize a single goal—*make the ego crash*—without controlling whether the resulting failures are informative or even physically meaningful\(Dinget al\.,[2020](https://arxiv.org/html/2605.21168#bib.bib35); Luet al\.,[2022](https://arxiv.org/html/2605.21168#bib.bib53); Niuet al\.,[2023](https://arxiv.org/html/2605.21168#bib.bib54)\)\. In particular, when vehicle–road physical limits are not modeled explicitly, adversaries may exploit unrealistic accelerations, timing, or geometry to create visually aggressive but physically unsalvageable crashes, where no admissible control could have avoided the collision\(Ghodsiet al\.,[2021](https://arxiv.org/html/2605.21168#bib.bib45); Rempeet al\.,[2022](https://arxiv.org/html/2605.21168#bib.bib2)\)\. These cases stress numerical robustness, but they conflate*difficulty*with*physical invalidity*, making it unclear whether failures reveal genuine deficiencies of the autonomy stack\.

A natural response is to incorporate*physical feasibility*into scenario generation\. Physics\-constrained methods restrict adversarial perturbations to physically realizable behaviors\. AdvSim\(Wanget al\.,[2021](https://arxiv.org/html/2605.21168#bib.bib36)\)perturbs agents’ trajectories while maintaining physically plausible motion and consistent sensor observations, and ACERO\(Songet al\.,[2023](https://arxiv.org/html/2605.21168#bib.bib48)\)searches for realistic, executable maneuvers that can reliably trigger failures in closed\-loop simulation\. These approaches rule out obviously unrealistic interactions and produce plausible stress tests \(Fig\.[1](https://arxiv.org/html/2605.21168#S1.F1)\(b\)\)\. However, using physical feasibility as the primary guiding constraint can cause generation to collapse onto extreme boundary\-pushing maneuvers near the physical limits, yielding a skewed set of highly adversarial samples that under\-represent more typical near\-boundary interactions\(Stoleret al\.,[2025](https://arxiv.org/html/2605.21168#bib.bib49)\)\. More broadly, heavy reliance on such highly adversarial samples can harm nominal performance and does not necessarily translate into actionable insights for improving the deployed autonomy stack\(Raghunathan\*et al\.,[2019](https://arxiv.org/html/2605.21168#bib.bib3)\)\.

Complementary to physics\-constrained generation, another line of work constrains adversariality by the*AV policy capability*\(Fig\.[1](https://arxiv.org/html/2605.21168#S1.F1)\(c\)\)\. FREA\(Chenet al\.,[2025](https://arxiv.org/html/2605.21168#bib.bib25)\), for example, leverages a policy\-feasibility signal to steer generation toward failures near the stack’s capability boundary\. This controller\-aware view can be actionable for diagnosing a specific stack, but it anchors generation to a controller\-dependent boundary and does not reveal how close the AV operates to the physical limits of the environment\. In particular, states deemed unrecoverable by the deployed stack may still be physically solvable under admissible dynamics, so policy\-constrained generation can miss the most informative regime where failures are*avoidable in principle*yet persist due to stack limitations\.

This paper adopts a simple organizing view: scenario difficulty is governed by two factors—physical feasibilityof vehicle–road interaction andAV policy capabilityof the deployed autonomy stack\. Their interplay defines four regimes \(Fig\.[1](https://arxiv.org/html/2605.21168#S1.F1)\): \(a\) nominal interactions far from both limits; \(b\) physically feasible but not necessarily informative adversarial cases produced by physics\-constrained generation; \(c\) controller\-bounded cases produced by policy\-constrained generation; and \(d\) a particularly informative boundary band where interactions remain physically solvable, yet still break the deployed stack\. We argue that this boundary band best isolates competence gaps: failures that are not inevitable from physics, but arise from limitations of the current autonomy stack\.

Building on this view, we proposeScenePilot, a feasibility\-guided adversarial scenario generation framework that targets the boundary band \(Fig\.[1](https://arxiv.org/html/2605.21168#S1.F1)\(d\)\)\. ScenePilot explicitly separates the two factors by quantifying physical feasibility using an RSS\-derived scoreσ\\sigmaand estimating stack vulnerability with an online\-learned risk predictorΦ\\Phi\. We formulate adversarial scenario generation as a constrained multi\-objective Markov decision process and introduce*step\-level feasibility\-aware shielding*that prioritizes feasibility recovery when the adversary approaches infeasible interactions, while maintaining pressure toward high\-risk failures\. To systematically explore the near\-boundary regime rather than collapsing onto a narrow set of extreme maneuvers, we further employ*feasibility\-threshold sweeping*withε\\varepsilon, which controls how close generation operates to the physical feasibility boundary and concentrates samples in the physically feasible yet policy\-infeasible band\.

In summary, our contributions are as follows:

- •We propose a feasibility\-guided formulation that disentangles physical feasibility from AV policy capability and targets their boundary band, where scenarios are physically solvable but still induce failures of the deployed autonomy stack\.
- •We develop a constrained multi\-objective adversarial generator that couples physical and policy signals\(σ,Φ\)\(\\sigma,\\Phi\)with step\-level feasibility\-aware shielding and feasibility\-threshold sweeping to concentrate on physically feasible yet policy\-infeasible near\-boundary scenarios\.
- •On SafeBench with multiple planners and controllers, ScenePilot generates more safety\-critical scenarios while keeping infeasible interactions rare, and adversarial fine\-tuning on boundary\-band scenarios consistently reduces downstream crash rates\.

## 2Related Work

### 2\.1Scenario\-based Safety Testing and Generation

Scenario\-based testing is a key approach for assessing AV safety\(Neurohret al\.,[2020](https://arxiv.org/html/2605.21168#bib.bib56)\)\. Existing methods for constructing safety\-critical scenarios are often grouped into three families: data\-driven, adversarial, and knowledge\-based generation\(Dinget al\.,[2023](https://arxiv.org/html/2605.21168#bib.bib8)\)\. Data\-driven methods replay or perturb real trajectories in large naturalistic datasets or simulators\(Liet al\.,[2019](https://arxiv.org/html/2605.21168#bib.bib9); Yanget al\.,[2023](https://arxiv.org/html/2605.21168#bib.bib10); Fenget al\.,[2021](https://arxiv.org/html/2605.21168#bib.bib7)\), and thus offer strong realism but are constrained by the long\-tail rarity and collection cost of critical events\(Luet al\.,[2026](https://arxiv.org/html/2605.21168#bib.bib17)\)\. Adversarial methods directly manipulate surrounding vehicles to induce near\-collisions or collisions\(Dinget al\.,[2021](https://arxiv.org/html/2605.21168#bib.bib13); Jiaet al\.,[2024](https://arxiv.org/html/2605.21168#bib.bib57)\), improving efficiency over passive logs but often optimizing failure frequency without strictly enforcing vehicle\-road physics\(Chenet al\.,[2021](https://arxiv.org/html/2605.21168#bib.bib22)\)\. Knowledge\-based methods encode traffic rules and expert priors into scenario templates or constraint systems\(Dinget al\.,[2025](https://arxiv.org/html/2605.21168#bib.bib15)\), improving controllability but relying on hand\-crafted coverage\. Overall, prior work emphasizes realism, adversariality, or rule coverage, with some methods incorporating physical feasibility constraints\(Donget al\.,[2025](https://arxiv.org/html/2605.21168#bib.bib46)\)\. However, existing methods often skew toward either unconstrained failure maximization or heavily physics\-focused boundary cases, underexploring the near\-boundary band of physically solvable yet AV policy\-breaking interactions\. We complement these lines by explicitly adopting a boundary\-focused view and targeting this near\-boundary feasible band\.

### 2\.2RL for Scenario Generation

Reinforcement learning \(RL\) is a natural tool for scenario generation, as AVs and surrounding agents interact in a sequential decision process\. RL\-based methods train adversarial agents to control nearby vehicles or modify traffic configurations and thereby synthesize safety\-critical encounters\(Liuet al\.,[2024](https://arxiv.org/html/2605.21168#bib.bib24); Weiet al\.,[2024](https://arxiv.org/html/2605.21168#bib.bib58)\)\. Many objectives directly encourage collisions or near\-misses, which is effective for exposing failures quickly but may also drive adversaries toward physically implausible or trivially infeasible behaviors, reducing the diagnostic value of the resulting scenarios\(Kuuttiet al\.,[2020](https://arxiv.org/html/2605.21168#bib.bib21)\)\. To improve physical validity, some works\(Haoet al\.,[2023](https://arxiv.org/html/2605.21168#bib.bib50); Caiet al\.,[2024](https://arxiv.org/html/2605.21168#bib.bib52)\)incorporate physics\-aware constraints modeling into RL\. However, such designs can bias generation toward extreme, boundary\-hugging feasible interactions, rather than the broader near\-boundary band that remains physically solvable yet exposes algorithmic weaknesses\. Chen et al\.\(Chenet al\.,[2025](https://arxiv.org/html/2605.21168#bib.bib25)\)further propose a boundary\-aware formulation by defining the Largest Feasible Region \(LFR\) for a given AV controller and shaping adversaries toward its boundary\. Yet LFR is controller\-dependent, mixing controller recoverability with physical feasibility\. We instead target a controller\-independent physical feasibility boundary and focus on physically solvable but AV policy\-breaking scenarios in the near\-boundary band\.

## 3Methods

![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/framework.png)Figure 2:Overview of our ScenePilot framework\. We characterize each rollout with an AV risk signal and a physics feasibility signal, and train a scenario policy to produce scenarios concentrated on the physically feasible yet AV policy\-infeasible boundary band\.ScenePilot is an adversarial scenario generation framework \(Fig\.[2](https://arxiv.org/html/2605.21168#S3.F2)\), where a learnable scenario policy controls non\-ego agents to evaluate ego autonomous driving policy under challenging interactions\. We characterize each rollout with two per\-step signals: an AV\-risk and a physics\-based feasibility\. The scenario policy is trained to concentrate rollouts in a boundary band that is physically feasible yet beyond the ego controller’s capability \(AV policy\-infeasible\), rather than producing trivial cases or physically impossible collisions\.

### 3\.1Problem Formulation

Adversarial MDP\.We formulate critical scenario generation as an adversarial discounted multi\-objective MDP⟨𝒮,𝒜,P,𝐫,γ⟩\\langle\\mathcal\{S\},\\mathcal\{A\},P,\\mathbf\{r\},\\gamma\\rangle, where a learnable scenario policyπ\(⋅∣st\)\\pi\(\\cdot\\mid s\_\{t\}\)controls all non\-ego agents\. At timett, the driving environment is in statest∈𝒮s\_\{t\}\\in\\mathcal\{S\}and the scenario policy samplesat∼π\(⋅∣st\)a\_\{t\}\\sim\\pi\(\\cdot\\mid s\_\{t\}\), whereata\_\{t\}denotes the joint action of the non\-ego agent set\. The ego vehicle follows a fixed driving policyπego\\pi^\{\\text\{ego\}\}and is absorbed into the environment dynamics, so that transitions satisfyst\+1∼P\(⋅∣st,at\)s\_\{t\+1\}\\sim P\(\\cdot\\mid s\_\{t\},a\_\{t\}\)\. Each transition is annotated with a two\-dimensional reward vector signal𝐫t=\(ϕt,σt\)\\mathbf\{r\}\_\{t\}=\(\\phi\_\{t\},\\sigma\_\{t\}\)\.

Two\-signal objective and boundary band\.The two components of𝐫t\\mathbf\{r\}\_\{t\}play complementary roles\. The AV\-risk signalϕt\\phi\_\{t\}measures how close the ego is to failure \(larger is riskier\), whileσt\\sigma\_\{t\}is a physics\-based feasibility derived from kinematic limits\. Specifically,σt≥0\\sigma\_\{t\}\\geq 0indicates physically feasible frames, andσt<0\\sigma\_\{t\}<0indicates physically infeasible \(i\.e\., unavoidable\-collision\) frames\. We define discounted objectivesJϕ​\(π\)=𝔼π​\[∑t=0T−1γt​ϕt\]J\_\{\\phi\}\(\\pi\)=\\mathbb\{E\}\_\{\\pi\}\\Big\[\\sum\_\{t=0\}^\{T\-1\}\\gamma^\{t\}\\phi\_\{t\}\\Big\]andJσ​\(π\)=𝔼π​\[∑t=0T−1γt​σt\]J\_\{\\sigma\}\(\\pi\)=\\mathbb\{E\}\_\{\\pi\}\\Big\[\\sum\_\{t=0\}^\{T\-1\}\\gamma^\{t\}\\sigma\_\{t\}\\Big\]\. Our goal is to generate scenarios that are physically feasible yet AV policy\-infeasible\. We therefore focus on the boundary band induced in the\(Jϕ,Jσ\)\(J\_\{\\phi\},J\_\{\\sigma\}\)plane,ℬ≜\{π\|Jσ​\(π\)≥0,Jϕ​\(π\)≥δ\}\\mathcal\{B\}\\triangleq\\left\\\{\\pi\\ \\middle\|\\ J\_\{\\sigma\}\(\\pi\)\\geq 0,\\ \\ J\_\{\\phi\}\(\\pi\)\\geq\\delta\\right\\\}, whereJσ​\(π\)≥0J\_\{\\sigma\}\(\\pi\)\\geq 0enforces physical feasibility andJϕ​\(π\)≥δJ\_\{\\phi\}\(\\pi\)\\geq\\deltafilters trivial low\-risk cases\. To traverse this band and obtain a spectrum of physically feasible yet AV policy\-infeasible scenarios, we optimize a family ofε\\varepsilon\-constrained objectives by Eq\. \([1](https://arxiv.org/html/2605.21168#S3.E1)\), whereε\\varepsilonis a control variable to select different feasibility levels during training\.

maxπ⁡Jϕ​\(π\)s\.t\.Jσ​\(π\)≥ε\.\\max\_\{\\pi\}\\ J\_\{\\phi\}\(\\pi\)\\qquad\\text\{s\.t\.\}\\qquad J\_\{\\sigma\}\(\\pi\)\\ \\geq\\ \\varepsilon\.\(1\)

### 3\.2Safety Signals

Physical Safety\.We develop our notion of physical safety based on the Mobileye Responsibility\-Sensitive Safety \(RSS\) model\(Shalev\-Shwartzet al\.,[2017](https://arxiv.org/html/2605.21168#bib.bib4)\), a widely adopted and principled framework for formalizing safety in autonomous driving\. RSS encodes “duty\-of\-care” driving rules and provides analytically checkable safe\-distance conditions in both longitudinal and lateral dimensions\. If all vehicles comply with these rules, traffic should be accident\-free\. As such, RSS provides a principled baseline for distinguishing physically feasible yet unsafe interactions from those consistent with safe driving\. However, the RSS safe\-distance are intentionally conservative\(Hassaninet al\.,[2022](https://arxiv.org/html/2605.21168#bib.bib5)\)\. Such conservatism is desirable for deployment\-level safety guarantees, but it is counterproductive for critical scenario exploration\. Many near\-crash yet still physically avoidable frames would be flagged as unsafe and thus prematurely excluded\. To better characterize the theoretically solvable region, we derive a physics\-limit variant of the RSS longitudinal and lateral distances in Eq\. \([2](https://arxiv.org/html/2605.21168#S3.E2)\)\.

dopplon=\(vrlon\)22​ar,maxlon,brake\+\(vflon\)22​af,maxlon,brake,\\displaystyle d\_\{\\mathrm\{opp\}\}^\{\\mathrm\{lon\}\}=\\dfrac\{\(v\_\{r\}^\{\\mathrm\{lon\}\}\)^\{2\}\}\{2a\_\{r,\\max\}^\{\\mathrm\{lon,brake\}\}\}\+\\dfrac\{\(v\_\{f\}^\{\\mathrm\{lon\}\}\)^\{2\}\}\{2a\_\{f,\\max\}^\{\\mathrm\{lon,brake\}\}\},\(2a\)dsamelon=max\[\(vrlon\)22​ar,maxlon,brake−\(vflon\)22​af,maxlon,brake⏟stop,\(vrlon−vflon\)22​\(ar,maxlon,brake−af,maxlon,brake\)⏟equal\],\\displaystyle\\begin\{aligned\} d\_\{\\mathrm\{same\}\}^\{\\mathrm\{lon\}\}=\\max&\\ \\Bigg\[\\underbrace\{\\dfrac\{\(v\_\{r\}^\{\\mathrm\{lon\}\}\)^\{2\}\}\{2a\_\{r,\\max\}^\{\\mathrm\{lon,brake\}\}\}\-\\dfrac\{\(v\_\{f\}^\{\\mathrm\{lon\}\}\)^\{2\}\}\{2a\_\{f,\\max\}^\{\\mathrm\{lon,brake\}\}\}\}\_\{\\text\{stop\}\},\\\\ &\\underbrace\{\\dfrac\{\(v\_\{r\}^\{\\mathrm\{lon\}\}\-v\_\{f\}^\{\\mathrm\{lon\}\}\)^\{2\}\}\{2\\big\(a\_\{r,\\max\}^\{\\mathrm\{lon,brake\}\}\-a\_\{f,\\max\}^\{\\mathrm\{lon,brake\}\}\\big\)\}\}\_\{\\text\{equal\}\}\\ \\Bigg\]\\end\{aligned\},\(2b\)dsamelat=\(vrlat\)22​ar,maxlat,brake−\(vflat\)22​af,maxlat,brake,\\displaystyle d\_\{\\mathrm\{same\}\}^\{\\mathrm\{lat\}\}=\\dfrac\{\(v\_\{r\}^\{\\mathrm\{lat\}\}\)^\{2\}\}\{2a\_\{r,\\max\}^\{\\mathrm\{lat,brake\}\}\}\-\\dfrac\{\(v\_\{f\}^\{\\mathrm\{lat\}\}\)^\{2\}\}\{2a\_\{f,\\max\}^\{\\mathrm\{lat,brake\}\}\},\(2c\)dopplat=\(vrlat\)22​ar,maxlat,brake\+\(vflat\)22​af,maxlat,brake\.\\displaystyle d\_\{\\mathrm\{opp\}\}^\{\\mathrm\{lat\}\}=\\dfrac\{\(v\_\{r\}^\{\\mathrm\{lat\}\}\)^\{2\}\}\{2a\_\{r,\\max\}^\{\\mathrm\{lat,brake\}\}\}\+\\dfrac\{\(v\_\{f\}^\{\\mathrm\{lat\}\}\)^\{2\}\}\{2a\_\{f,\\max\}^\{\\mathrm\{lat,brake\}\}\}\.\(2d\)
We index the two interacting vehicles byx∈\{r,f\}x\\in\\\{r,f\\\}, whererris the rear \(left in lateral\) vehicle andffis the front \(right in lateral\) one\. The scalarsvxlonv\_\{x\}^\{\\mathrm\{lon\}\}andvxlatv\_\{x\}^\{\\mathrm\{lat\}\}denote the longitudinal and lateral speed components of vehiclexx, andax,maxlon,brakea\_\{x,\\max\}^\{\\mathrm\{lon,brake\}\}andax,maxlat,brakea\_\{x,\\max\}^\{\\mathrm\{lat,brake\}\}denote its maximal feasible decelerations along the corresponding axis\. This variant corresponds to an idealized limit in which vehicles react instantaneously and decelerate at their maximal feasible rates under kinematic constraints, thereby providing an axis\-wise braking\-limit distancedlimitd\_\{\\mathrm\{limit\}\}\. Consequently, if the actual distance on an axis satisfiesdactual<dlimitd\_\{\\mathrm\{actual\}\}<d\_\{\\mathrm\{limit\}\}\(Eq\. \([2](https://arxiv.org/html/2605.21168#S3.E2)\)\), then a collision is unavoidable under braking\-only responses along that axis\.

𝒅actual=\[dactual,xdactual,y\]=\[cos⁡ψegosin⁡ψego−sin⁡ψegocos⁡ψego\]​\(\[xadvyadv\]−\[xegoyego\]\),𝒔​\(Δ​ψ\)=\[sxsy\]=\[Lego\+\|cos⁡Δ​ψ\|​Ladv\+\|sin⁡Δ​ψ\|​WadvWego\+\|cos⁡Δ​ψ\|​Wadv\+\|sin⁡Δ​ψ\|​Ladv\],σ=1−‖\(𝒅limit−\(\|𝒅actual\|−𝒔​\(Δ​ψ\)\)−𝒍\)\+⊘𝒅limit‖p\.\\begin\{aligned\} &\\boldsymbol\{d\}\_\{\\mathrm\{actual\}\}=\\begin\{bmatrix\}d\_\{\{\\mathrm\{actual\}\},x\}\\\\\[2\.0pt\] d\_\{\{\\mathrm\{actual\}\},y\}\\end\{bmatrix\}=\\begin\{bmatrix\}\\cos\\psi\_\{\\mathrm\{ego\}\}&\\sin\\psi\_\{\\mathrm\{ego\}\}\\\\ \-\\sin\\psi\_\{\\mathrm\{ego\}\}&\\cos\\psi\_\{\\mathrm\{ego\}\}\\end\{bmatrix\}\\left\(\\begin\{bmatrix\}x\_\{\\mathrm\{adv\}\}\\\\\[2\.0pt\] y\_\{\\mathrm\{adv\}\}\\end\{bmatrix\}\-\\begin\{bmatrix\}x\_\{\\mathrm\{ego\}\}\\\\\[2\.0pt\] y\_\{\\mathrm\{ego\}\}\\end\{bmatrix\}\\right\),\\\\\[4\.0pt\] &\\boldsymbol\{s\}\(\\Delta\\psi\)=\\begin\{bmatrix\}s\_\{x\}\\\\\[2\.0pt\] s\_\{y\}\\end\{bmatrix\}=\\begin\{bmatrix\}L\_\{\\mathrm\{ego\}\}\+\|\\cos\\Delta\\psi\|\\,L\_\{\\mathrm\{adv\}\}\+\|\\sin\\Delta\\psi\|\\,W\_\{\\mathrm\{adv\}\}\\\\\[4\.0pt\] W\_\{\\mathrm\{ego\}\}\+\|\\cos\\Delta\\psi\|\\,W\_\{\\mathrm\{adv\}\}\+\|\\sin\\Delta\\psi\|\\,L\_\{\\mathrm\{adv\}\}\\end\{bmatrix\},\\\\\[4\.0pt\] &\\sigma=1\-\\left\\\|\\left\(\\boldsymbol\{d\}\_\{\\mathrm\{limit\}\}\-\\bigl\(\|\\boldsymbol\{d\}\_\{\\mathrm\{actual\}\}\|\-\\boldsymbol\{s\}\(\\Delta\\psi\)\\bigr\)\-\\boldsymbol\{l\}\\right\)\_\{\+\}\\oslash\\boldsymbol\{d\}\_\{\\mathrm\{limit\}\}\\right\\\|\_\{p\}\.\\end\{aligned\}

\(3\)
Importantly, braking\-only infeasibility along the imminent\-collision axis does not preclude collision avoidance under coupled braking\-steering maneuvers, as the orthogonal axis may provide sufficient reachable displacement within the available time\-to\-collision\. To quantify instantaneous physical feasibility beyond braking, we decompose each frame in the ego\-fixed body frame into longitudinal \(xx\) and lateral \(yy\) axes and evaluate the residual margins on both\. We then incorporate time\-to\-collision and the reachable displacement under bounded accelerations to aggregate these margins into a unified physical safety scoreσ\\sigmain Eq\. \([3](https://arxiv.org/html/2605.21168#S3.E3)\)\.

𝒅actual\\boldsymbol\{d\}\_\{\\mathrm\{actual\}\}is the ego\-adversarial vehicle center distance expressed in the ego body frame,𝒔​\(Δ​ψ\)\\boldsymbol\{s\}\(\\Delta\\psi\)is the geometric clearance of the two oriented bounding boxes with relative yaw of the ego and adversarial vehicle,Δ​ψ=ψadv−ψego\\Delta\\psi=\\psi\_\{\\mathrm\{adv\}\}\-\\psi\_\{\\mathrm\{ego\}\}, and𝒅limit\\boldsymbol\{d\}\_\{\\mathrm\{limit\}\}is the physics\-limit safe distance from Eq\. \([2](https://arxiv.org/html/2605.21168#S3.E2)\) projected onto the longitudinal and lateral axes\. The operator\(⋅\)\+\(\\cdot\)\_\{\+\}acts elementwise as\[z\]\+=max⁡\(0,z\)\[z\]\_\{\+\}=\\max\(0,z\),⊘\\oslashdenotes elementwise division, and∥⋅∥p\\\|\\cdot\\\|\_\{p\}is theℓp\\ell\_\{p\}norm, which aggregates the normalized residual gaps on the longitudinal and lateral axes\. The vector𝒍=\[lx,ly\]⊤\\boldsymbol\{l\}=\[l\_\{x\},l\_\{y\}\]^\{\\top\}encodes the reachable displacements that can be compensated along the axis orthogonal to the imminent collision, withttdetermined by Eq\. \([4](https://arxiv.org/html/2605.21168#S3.E4)\)\.

t=min⁡\(\[dxactual−Wcar\]\+\[Δ​vx\]\+⏟TTC \(lon\),\[dyactual−Lcar\]\+\[Δ​vy\]\+⏟TTC \(lat\)\)t=\\min\\\!\\left\(\\underbrace\{\\frac\{\[\\,d\_\{x\}^\{\\mathrm\{actual\}\}\-W\_\{\\mathrm\{car\}\}\\,\]\_\{\+\}\}\{\[\\,\\Delta v\_\{x\}\\,\]\_\{\+\}\}\}\_\{\\text\{TTC \(lon\)\}\},\\;\\underbrace\{\\frac\{\[\\,d\_\{y\}^\{\\mathrm\{actual\}\}\-L\_\{\\mathrm\{car\}\}\\,\]\_\{\+\}\}\{\[\\,\\Delta v\_\{y\}\\,\]\_\{\+\}\}\}\_\{\\text\{TTC \(lat\)\}\}\\right\)\(4\)
Specifically, we compute the axial time to collision\(tx,ty\)\(t\_\{x\},t\_\{y\}\)and identify the first\-colliding axisc=arg⁡min⁡\{tx,ty\}c=\\arg\\min\\\{t\_\{x\},t\_\{y\}\\\}and its orthogonal axisn=\{x,y\}∖\{c\}n=\\\{x,y\\\}\\setminus\\\{c\\\}\. The displacement along the colliding axis is set to zero \(lc=0l\_\{c\}=0\), while that on the orthogonal axis is compensated asln=12​anrel​t2​1​\{Δ​vn\>0\}l\_\{n\}=\\tfrac\{1\}\{2\}a^\{\\mathrm\{rel\}\}\_\{n\}t^\{2\}\\,\\mathbf\{1\}\\\{\\Delta v\_\{n\}\>0\\\}, where𝟏​\{⋅\}\\mathbf\{1\}\\\{\\cdot\\\}is the indicator function,anrela^\{\\mathrm\{rel\}\}\_\{n\}is the maximal feasible relative acceleration andΔ​vn\>0\\Delta v\_\{n\}\>0denotes a closing motion\. By construction,σ≥0\\sigma\\geq 0indicates physically feasible frames, andσ<0\\sigma<0indicates physically infeasible frames where no admissible combination of braking and steering can prevent a collision\. Details are provided in Appendix[A\.1](https://arxiv.org/html/2605.21168#A1.SS1)\.

AV Risk\.FREA’s feasibility\-aware risk \(LFR\)\(Chenet al\.,[2025](https://arxiv.org/html/2605.21168#bib.bib25)\)offers a certificate indicating whether a state lies within the AV’s feasible set, making it well\-suited for characterizing the trajectory\-level feasibility boundary\. However, this certificate and its explicit constraint modeling are unnecessary in our work\. We therefore introduce a lightweight, online\-learnable AV\-risk network that estimates the ego vehicle’s instantaneous safety risk at each frame\. Specifically, we learn this predictor as a policy\-free critic \(value learning without an actor\)\. Given a scenario frame featurests\_\{t\}, we regress the network outputΦ^​\(st\)∈\[0,1\]\\hat\{\\Phi\}\(s\_\{t\}\)\\\!\\in\[0,1\]to a potential\-shaped temporal\-difference target:

yt\\displaystyle y\_\{t\}=𝟙​\{𝒞t\}⏟collision\+γ​Φ^​\(st\+1\)\+γ​F​\(st\+1\)−F​\(st\)⏟potential shaping\\displaystyle=\\underbrace\{\\mathbb\{1\}\\\{\\mathcal\{C\}\_\{t\}\\\}\}\_\{\\text\{collision\}\}\+\\gamma\\,\\hat\{\\Phi\}\(s\_\{t\+1\}\)\+\\underbrace\{\\gamma\\,F\(s\_\{t\+1\}\)\-F\(s\_\{t\}\)\}\_\{\\text\{potential shaping\}\}\(5\)where𝒞t\\mathcal\{C\}\_\{t\}denotes the collision event at timett\. The potential\-based shaping term densifies the otherwise sparse collision feedback and accelerates learning, while preserving optimal policies\(Nget al\.,[1999](https://arxiv.org/html/2605.21168#bib.bib26)\)\. In our implementation, we chooseF​\(s\)=κ/d​\(s\)F\(s\)=\\kappa/d\(s\), whered​\(s\)d\(s\)denotes the ego–nearest\-adversary distance andκ\>0\\kappa\>0is a scaling factor\.

The network is a two\-layer MLP with sigmoid output and is trained online by a weighted binary cross\-entropy to handle class imbalance in Eq\. \([6](https://arxiv.org/html/2605.21168#S3.E6)\), where a larger weightw\+≫1w\_\{\+\}\\\!\\gg\\\!1emphasizes rare collision targets\.

ℒt=wt⋅BCE​\(Φ^​\(st\),clip​\(yt,0,1\)\),\\displaystyle\\mathcal\{L\}\_\{t\}\\;=\\;w\_\{t\}\\cdot\\mathrm\{BCE\}\\big\(\\hat\{\\Phi\}\(s\_\{t\}\),\\mathrm\{clip\}\(y\_\{t\},0,1\)\\big\),\(6\)wt=\{w\+,𝒞t​is true,1,otherwise\.\\displaystyle w\_\{t\}=Under potential\-based shaping, the learned value is a shifted estimate,Φ^​\(s\)=Φ​\(s\)−F​\(s\)\\hat\{\\Phi\}\(s\)=\\Phi\(s\)\-F\(s\)\. At inference, we recover the unshifted AV risk by adding back the potential,Φ​\(s\)=clip​\(Φ^​\(s\)\+F​\(s\),0,1\)\\Phi\(s\)=\\mathrm\{clip\}\\big\(\\hat\{\\Phi\}\(s\)\+F\(s\),\\,0,1\\big\)\. This yields a continuous, differentiable score that provides an interpretable measure of the ego vehicle’s risk across states\. Details are provided in Appendix[A\.2](https://arxiv.org/html/2605.21168#A1.SS2)\.

### 3\.3ScenePilot

Given per\-frame AV\-risk and physical\-safety reward signals, our goal is to solve theε\\varepsilon\-constrained objective in Eq\. \([1](https://arxiv.org/html/2605.21168#S3.E1)\) and generate scenarios that are risky for AV yet physically solvable\. We use the two components of the per\-step reward vector𝐫t=\(ϕt,σt\)\\mathbf\{r\}\_\{t\}=\(\\phi\_\{t\},\\sigma\_\{t\}\)and sweep the feasibility thresholdε\\varepsilonduring training to traverse a near\-boundary feasible band\. Inspired by CMORL\-IPO\(Liuet al\.,[2025](https://arxiv.org/html/2605.21168#bib.bib28)\), a natural way to enforceJσ≥εJ\_\{\\sigma\}\\\!\\geq\\\!\\varepsilonis an interior\-point scalarization, i\.e\., optimizingJϕ\+λ​JσJ\_\{\\phi\}\+\\lambda J\_\{\\sigma\}withλ∝\(𝔼​\[Jσ\]−ε\)−1\\lambda\\propto\(\\mathbb\{E\}\[J\_\{\\sigma\}\]\-\\varepsilon\)^\{\-1\}\. While effective for exploring global trade\-offs, this batch\-mean construction can smooth out rare near\-boundary frames and weaken responsiveness to imminent violations\. We therefore adopt a step\-level shielding mechanism and learn dual value critics, following safe\-RL formulations\(Wageneret al\.,[2021](https://arxiv.org/html/2605.21168#bib.bib29); Yuet al\.,[2022](https://arxiv.org/html/2605.21168#bib.bib43)\)\. We use a deterministic state featurization that concatenates privileged simulator signals, including ego state, surrounding agents, route cues, and map features, into a vector statests\_\{t\}, which conditions the actor and a two\-headed vector\-valued critic\.

𝐕​\(st\)=𝔼​\[∑k=0T−1γk​𝐫t\+k\|st\],𝐫t=\(ϕt,σt\)\.\\mathbf\{V\}\(s\_\{t\}\)=\\mathbb\{E\}\\\!\\Big\[\\sum\_\{k=0\}^\{T\-1\}\\gamma^\{k\}\\,\\mathbf\{r\}\_\{t\+k\}\\,\\Big\|\\,s\_\{t\}\\Big\],\\quad\\mathbf\{r\}\_\{t\}=\(\\phi\_\{t\},\\sigma\_\{t\}\)\.\(7\)
From rollouts, we form the vector return𝐆t=∑k=0T−1γk​𝐫t\+k\\mathbf\{G\}\_\{t\}=\\sum\_\{k=0\}^\{T\-1\}\\gamma^\{k\}\\mathbf\{r\}\_\{t\+k\}and train the two\-head critic by a vector regression loss in Eq\. \([8](https://arxiv.org/html/2605.21168#S3.E8)\), where𝐕​\(st\)=\(Vϕ​\(st\),Vσ​\(st\)\)\\mathbf\{V\}\(s\_\{t\}\)=\(V\_\{\\phi\}\(s\_\{t\}\),V\_\{\\sigma\}\(s\_\{t\}\)\)\.

ℒV=𝔼​\[‖𝐕​\(st\)−𝐆t‖22\]\.\\mathcal\{L\}\_\{V\}=\\mathbb\{E\}\\\!\\Big\[\\big\\\|\\mathbf\{V\}\(s\_\{t\}\)\-\\mathbf\{G\}\_\{t\}\\big\\\|\_\{2\}^\{2\}\\Big\]\.\(8\)
We then compute generalized advantage estimates \(GAE\) for both components using the corresponding TD residuals\. Specifically, let𝜹t=𝐫t\+γ​𝐕​\(st\+1\)−𝐕​\(st\)\\boldsymbol\{\\delta\}\_\{t\}=\\mathbf\{r\}\_\{t\}\+\\gamma\\,\\mathbf\{V\}\(s\_\{t\+1\}\)\-\\mathbf\{V\}\(s\_\{t\}\)\. We apply GAE\(γ,λgae\)\(\\gamma,\\lambda\_\{\\text\{gae\}\}\)to each component of𝜹t\\boldsymbol\{\\delta\}\_\{t\}to obtain𝐀t=\(Atϕ,Atσ\)\\mathbf\{A\}\_\{t\}=\(A\_\{t\}^\{\\phi\},A\_\{t\}^\{\\sigma\}\)\. To retain sensitivity to imminent feasibility violations, we introduce a step\-level shielding maskmt≜𝕀​\[ht\>0∨ht\+1\>0\]m\_\{t\}\\triangleq\\mathbb\{I\}\[\\,h\_\{t\}\>0\\lor h\_\{t\+1\}\>0\\,\], with violation residualsht=\[εt−σt\]\+h\_\{t\}=\[\\,\\varepsilon\_\{t\}\-\\sigma\_\{t\}\\,\]\_\{\+\}andht\+1=\[εt−σt\+1\]\+h\_\{t\+1\}=\[\\,\\varepsilon\_\{t\}\-\\sigma\_\{t\+1\}\\,\]\_\{\+\}\. The thresholdεt\\varepsilon\_\{t\}follows the Gaussian\-shaped schedule in Eq\. \([33](https://arxiv.org/html/2605.21168#A1.E33)\) of Appendix[A\.3](https://arxiv.org/html/2605.21168#A1.SS3)\. The shielded advantage is

A~t=mt​Atσ\+\(1−mt\)​Atϕ\.\\tilde\{A\}\_\{t\}=m\_\{t\}\\,A\_\{t\}^\{\\sigma\}\+\(1\-m\_\{t\}\)\\,A\_\{t\}^\{\\phi\}\.\(9\)
This implements a step\-level shielding principle\. Whenever a feasibility violation is present or imminent, policy updates prioritize feasibility recovery viaAtσA\_\{t\}^\{\\sigma\}; otherwise they optimize risk viaAtϕA\_\{t\}^\{\\phi\}\. In contrast, an episode\- or batch\-mean feasibility penalty can dilute these short but decisive near\-boundary violations, allowing a few infeasible frames to be traded off against many feasible ones\. With a diagonal\-Gaussian actorπθ​\(at∣st\)\\pi\_\{\\theta\}\(a\_\{t\}\\\!\\mid\\\!s\_\{t\}\), we plug the shielded advantageA~t\\tilde\{A\}\_\{t\}into a PPO\-style clipped surrogate:

ℒpolicy​\(θ\)=\\displaystyle\\mathcal\{L\}\_\{\\text\{policy\}\}\(\\theta\)=−𝔼t​\[min⁡\(ρt​\(θ\)​A~t,ρ¯t​\(θ\)​A~t\)\]\\displaystyle\-\\mathbb\{E\}\_\{t\}\\\!\\left\[\\min\\\!\\Big\(\\rho\_\{t\}\(\\theta\)\\,\\tilde\{A\}\_\{t\},\\ \\bar\{\\rho\}\_\{t\}\(\\theta\)\\,\\tilde\{A\}\_\{t\}\\Big\)\\right\]\(10\)−cent𝔼t\[ℋ\(πθ\(⋅∣st\)\)\],\\displaystyle\-c\_\{\\text\{ent\}\}\\,\\mathbb\{E\}\_\{t\}\\\!\\Big\[\\mathcal\{H\}\\\!\\big\(\\pi\_\{\\theta\}\(\\cdot\\mid s\_\{t\}\)\\big\)\\Big\],whereρt​\(θ\)=exp⁡\(log⁡πθ​\(at∣st\)−log⁡πθold​\(at∣st\)\)\\rho\_\{t\}\(\\theta\)=\\exp\\\!\\big\(\\log\\pi\_\{\\theta\}\(a\_\{t\}\\\!\\mid\\\!s\_\{t\}\)\-\\log\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(a\_\{t\}\\\!\\mid\\\!s\_\{t\}\)\\big\)andρ¯t​\(θ\)=min⁡\{1\+η,max⁡\{1−η,ρt​\(θ\)\}\}\\bar\{\\rho\}\_\{t\}\(\\theta\)=\\min\\\{1\+\\eta,\\max\\\{1\-\\eta,\\rho\_\{t\}\(\\theta\)\\\}\\\}\. We use standard entropy regularization and gradient clipping, and train the critic with Eq\. \([8](https://arxiv.org/html/2605.21168#S3.E8)\)\. The moving thresholdε\\varepsilonsweeps the feasible band over training, while step\-level shielding in Eq\. \([9](https://arxiv.org/html/2605.21168#S3.E9)\) preserves sensitivity to rare near\-boundary events that episode/batch averages may wash out\. Together, these designs concentrate the scenario policy on physically feasible yet AV policy\-infeasible cases along the boundary band\.

## 4Experiments

Table 1:Statistics of scenario generation on SafeBench base scenarios\.We report the collision rate \(CR\) and overall score \(OS\) to quantify the safety\-criticality of the generated scenarios\. For each algorithm, metrics are averaged over three ego controllers \(SAC, PPO, TD3\) and all scenes under the same base scenario; the rightmost column summarizes the mean performance across all base scenarios, with bold entries indicating the best result among the compared methods\. LC \(Learning\-to\-Collide\), AS \(AdvSim\), CS \(CARLA Scenario Generator\), and AT \(Adversarial Trajectory Optimization\) follow the standard SafeBench setup, while ChatScene\(Zhanget al\.,[2024](https://arxiv.org/html/2605.21168#bib.bib39)\), SCSG\(Karaciket al\.,[2025](https://arxiv.org/html/2605.21168#bib.bib61)\), and SCENGE\(Liuet al\.,[2026](https://arxiv.org/html/2605.21168#bib.bib62)\)denote recent safety\-critical scenario generators\.↑⁣/⁣↓\\uparrow/\\downarrowdenote that higher/lower values are preferable for the corresponding metric\.In this section, we evaluate ScenePilot on SafeBench\(Xuet al\.,[2022](https://arxiv.org/html/2605.21168#bib.bib34)\)with two complementary goals\. First, we assess how effectively the generated scenarios expose safety\-critical ego failures while remaining physically plausible under vehicle–road limits\. Second, we test downstream utility by adversarially fine\-tuning ego policies on the generated scenarios and evaluating robustness on held\-out routes\.

All experiments are conducted in CARLA\(Dosovitskiyet al\.,[2017](https://arxiv.org/html/2605.21168#bib.bib33)\)under the SafeBench pipeline\(Xuet al\.,[2022](https://arxiv.org/html/2605.21168#bib.bib34)\), where our method controls adversarial agents and the ego is driven by a learning\-based policy\. This enables like\-for\-like comparison under the official protocol, and we additionally report physical plausibility statistics based on our RSS\-derived feasibility score\.

Our main findings are: \(1\) Compared with competitive baselines, ScenePilot generates more challenging yet physically plausible scenarios, avoiding failure cases dominated by physically infeasible artifacts\. \(2\) Although learned against a fixed surrogate ego policy, the generated failures transfer across heterogeneous controllers\. \(3\) Fine\-tuning on ScenePilot scenarios yields the strongest downstream robustness under an otherwise identical training pipeline\. \(4\) ScenePilot covers a broader portion of the physically feasible yet policy\-infeasible boundary band, rather than collapsing to a narrow set of extreme cases\.

### 4\.1Setup

Scenarios\.We follow SafeBench and generate/evaluate adversarial interactions on eight canonical traffic scenarios, each instantiated on ten pre\-defined routes that cover diverse map topologies and interaction patterns \(Straight/Turning Obstacle, Lane Changing, Vehicle Passing, Red\-light Running, Unprotected Left\-turn, Right Turn, and Crossing Negotiation\)\. Unless otherwise stated, all scenario\-generation statistics are aggregated over all routes within each base scenario\. For downstream ego\-policy fine\-tuning, we adopt a route\-disjoint split within each base scenario: routes 1–8 are used to collect training scenes, while routes 9–10 are held out for evaluation\. Implementation details of generated scenario sampling are provided in Appendix[B\.4](https://arxiv.org/html/2605.21168#A2.SS4)\.

AD algorithms\.To assess transfer and controller\-agnostic difficulty, we evaluate each selected scene with three representative deep RL controllers provided by SafeBench: Proximal Policy Optimization \(PPO\)\(Schulmanet al\.,[2017](https://arxiv.org/html/2605.21168#bib.bib40)\), an on\-policy stochastic method; Soft Actor\-Critic \(SAC\)\(Haarnojaet al\.,[2018](https://arxiv.org/html/2605.21168#bib.bib41)\), an off\-policy stochastic method; and Twin Delayed DDPG \(TD3\)\(Fujimotoet al\.,[2018](https://arxiv.org/html/2605.21168#bib.bib42)\), an off\-policy deterministic method\. Following SafeBench, the ego observation is a four\-dimensional vector capturing \(i\) distance to the next waypoint, \(ii\) longitudinal speed, \(iii\) angular speed, and \(iv\) a binary indicator of a front\-facing vehicle\. Using this interface preserves strict comparability with prior reports\. Beyond this standard RL\-controller evaluation, we further evaluate generated scenarios on a broader set of heterogeneous AV stacks, including CARLA Autopilot\(Dosovitskiyet al\.,[2017](https://arxiv.org/html/2605.21168#bib.bib33)\)\(Traffic\-Manager\-controlled policy\), AIM\-BEV\(Hanselmannet al\.,[2022](https://arxiv.org/html/2605.21168#bib.bib64)\)\(privileged\-input end\-to\-end planner\), TransFuser\(Prakashet al\.,[2021](https://arxiv.org/html/2605.21168#bib.bib63)\)\(raw\-sensor end\-to\-end planner\), BehaviorAgent\(Dosovitskiyet al\.,[2017](https://arxiv.org/html/2605.21168#bib.bib33)\)\(rule\-based CARLA agent\)\.

Baselines\.We compare against five SOTA scenario\-generation baselines in SafeBench, spanning two common design philosophies: \(i\) adversary\-driven approaches that learn perturbations of initial states or trajectories, including Learning\-to\-Collide \(LC\)\(Dinget al\.,[2020](https://arxiv.org/html/2605.21168#bib.bib35)\)and AdvSim \(AS\)\(Wanget al\.,[2021](https://arxiv.org/html/2605.21168#bib.bib36)\); \(ii\) rule/optimization\-driven approaches that encode traffic priors or trajectory costs, including the CARLA Scenario Generator \(CS\)\(CARLA Scenario Runner Contributors,[2019](https://arxiv.org/html/2605.21168#bib.bib37)\)and Adversarial Trajectory Optimization \(AT\)\(Zhanget al\.,[2022](https://arxiv.org/html/2605.21168#bib.bib38)\)\. We also include ChatScene\(Zhanget al\.,[2024](https://arxiv.org/html/2605.21168#bib.bib39)\), SCSG\(Karaciket al\.,[2025](https://arxiv.org/html/2605.21168#bib.bib61)\), and SCENGE\(Liuet al\.,[2026](https://arxiv.org/html/2605.21168#bib.bib62)\), some strong recent baselines that generates scenarios via scene code\. All methods are run in the same simulator with matched sampling budgets, identical initialization and termination conditions, and the same interaction interface\.

Metrics\.We report two primary metrics used in SafeBench: \(i\)Collision Rate \(CR\), the proportion of runs that end in a collision with the ego \(higher implies stronger safety\-criticality\)\. \(ii\)Overall Score \(OS\), the official composite metric aggregating Safety, Functionality, and Etiquette indicators \(lower implies harder scenes\)\. The definitions and computation follow the established Safebench and ChatScene to ensure strict comparability across methods\.

### 4\.2Evaluation of the Generated Safety\-Critical Scenarios

Table 2:Statistics of generated SafeBench Scenario 6 \(Unprotected Left\-turn\) cases under heterogeneous AV stacks\.We report the collision rate \(CR\) and overall score \(OS\) to quantify the safety\-criticality of generated scenarios under different ego stacks\. The evaluated stacks include CARLA Autopilot\(Dosovitskiyet al\.,[2017](https://arxiv.org/html/2605.21168#bib.bib33)\)\(Traffic\-Manager\-controlled\), AIM\-BEV\(Hanselmannet al\.,[2022](https://arxiv.org/html/2605.21168#bib.bib64)\)\(privileged\-input end\-to\-end\), TransFuser\(Prakashet al\.,[2021](https://arxiv.org/html/2605.21168#bib.bib63)\)\(raw\-sensor end\-to\-end\), BehaviorAgent\(Dosovitskiyet al\.,[2017](https://arxiv.org/html/2605.21168#bib.bib33)\)\(rule\-based\), and TD3/SAC/PPO \(deep\-RL\)\.↑⁣/⁣↓\\uparrow/\\downarrowdenote that higher/lower values are preferable for the corresponding metric\.We first evaluate the safety\-criticality of the generated scenes under the standard SafeBench protocol, where each method’s generated scenarios are replayed with three RL\-based ego controllers \(SAC, PPO, TD3\)\. Metrics are averaged over the three controllers and all scenes within the same base scenario\. As shown in Table[1](https://arxiv.org/html/2605.21168#S4.T1), ScenePilot achieves the strongest overall safety\-criticality across the eight SafeBench base scenarios, with the highest mean CR \(0\.893\) and the lowest mean OS \(0\.476\)\. Relative to prior SafeBench baselines \(LC/AS/CS/AT\) and the recent strong generator ChatScene, SCSG and SCENGE, ScenePilot improves CR on most scenarios while consistently lowering OS, indicating that the generated failures are not tailored to a single ego policy but transfer across controllers\. Qualitative examples are shown in Fig\.[3](https://arxiv.org/html/2605.21168#S4.F3), where the red vehicle denotes the ego AV, the green vehicle denotes the adversarial vehicle, and the right panel shows the ego vehicle’s front\-view observation\. The generated scenario stays close to the feasibility boundary while inducing failures across ego controllers\. We provide a component\-wise breakdown of OS \(Safety/Functionality/Etiquette\) in Appendix[C](https://arxiv.org/html/2605.21168#A3)for further diagnosis\.

![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/carla/7-1.png)
![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/carla/7-2.png)

Figure 3:Visualization of a near\-boundary scenario generated by ScenePilot\.To further examine whether ScenePilot remains effective across heterogeneous AV stacks, we conduct an additional study beyond the standard SafeBench RL\-controller evaluation\. We generate 100 SafeBench Scenario 6 cases using CARLA Autopilot as the ego stack, and replay the generated cases on Autopilot, AIM\-BEV, TransFuser, BehaviorAgent, and TD3/SAC/PPO\. We use KING\(Hanselmannet al\.,[2022](https://arxiv.org/html/2605.21168#bib.bib64)\)as the baseline under the same Autopilot\-based generation and SafeBench evaluation protocol\. As shown in Table[2](https://arxiv.org/html/2605.21168#S4.T2), ScenePilot remains more challenging than KING on average, increasing CR from 0\.380 to 0\.490 and reducing OS from 0\.766 to 0\.708\. These results indicate that ScenePilot is not limited to the standard RL\-controller setting, but remains effective for more capable AV stacks\.

### 4\.3Evaluation under Denser Traffic Conditions

Table 3:Statistics of generated scenarios under denser background traffic\.We report the collision rate \(CR\) and overall score \(OS\) after inserting different numbers of Traffic\-Manager\-controlled background vehicles \(BV\)\. The primary adversarial vehicle remains optimized by ScenePilot\.↑⁣/⁣↓\\uparrow/\\downarrowdenote that higher/lower values are preferable for the corresponding metric\.We also evaluate ScenePilot under denser background traffic\. Specifically, we insert 30 and 50 additional Carla Traffic\-Manager\-controlled background vehicles into the SafeBench Scenario 6 \(Unprotected Left\-turn\), while keeping the primary adversarial vehicle optimized by ScenePilot unchanged from standard SafeBench\-generated ScenePilot scenarios\. As shown in Table[3](https://arxiv.org/html/2605.21168#S4.T3), ScenePilot remains effective under denser traffic conditions\. Compared with the original setting without additional background vehicles, the average CR slightly increases from 0\.89 to 0\.91, while the average OS decreases from 0\.507 to 0\.492\. These results suggest that denser surrounding traffic affects the realized adversarial interaction through occupancy, gap availability, and timing, making the scenarios slightly more difficult\. However, since the added background vehicles are benign rather than adversarially optimized, the main stress\-testing pressure still comes from ScenePilot’s primary adversarial vehicle\. Thus, ScenePilot preserves its safety\-critical effect when the primary adversarial interaction is embedded in denser traffic\.

### 4\.4Effectiveness of Downstream Utility on Fine\-tuning

Table 4:Evaluating Ego\-Vehicle Performance Post\-Finetuning\.This table evaluates how effectively each method’s generated scenarios serve as adversarial fine\-tuning data to improve ego robustness, measured by collision rate \(CR\) and overall score \(OS\) on held\-out test scenes; entries report the mean performance over these test scenes\. The row labeled “PP” corresponds to the surrogate ego before fine\-tuning\. The rightmost column shows the average across all base scenarios\. For all metrics,↑⁣/⁣↓\\uparrow/\\downarrowindicate that larger/smaller values are preferred, respectively\.We further evaluate whether the generated scenes provide actionable learning signals for improving an ego policy via adversarial fine\-tuning\. Following ChatScene\(Zhanget al\.,[2024](https://arxiv.org/html/2605.21168#bib.bib39)\), we fine\-tune the same surrogate SAC ego policy used for scenario generation, ensuring an apples\-to\-apples comparison\. For each base scenario, we fine\-tune the ego separately on the safety\-critical scenes generated from the first eight routes, and evaluate on a held\-out set from the remaining two routes with test scenes pooled across generators \(about 40 cases per scenario; details in Appendix[B\.5](https://arxiv.org/html/2605.21168#A2.SS5)\)\. Table[4](https://arxiv.org/html/2605.21168#S4.T4)shows that fine\-tuning on generated critical scenarios substantially improves robustness over the pre\-trained surrogate \(PP\)\. On average, CR decreases from 0\.854 to 0\.134/0\.072 after fine\-tuning on ChatScene/ScenePilot scenes, while OS increases from 0\.525 to 0\.869/0\.898\. ScenePilot achieves the best post\-finetuning robustness with the lowest average CR \(0\.072\) and the highest OS \(0\.898\)\. Overall, these results suggest that ScenePilot produces training scenes with stronger downstream utility, leading to better generalization under the same fine\-tuning pipeline\.

### 4\.5AV–Physics Analysis of Generated Critical Scenarios

![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/results/phys_inv.png)\(a\)Phys\-invalid frame rate\.
![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/results/coverage_ratio.png)\(b\)Risk\-physics coverage ratio\.

Figure 4:Quantitative characterization of the AV–physics gap between ScenePilot and ChatScene\.\(a\) Physically invalid frame rate under different AV\-risk thresholds\. \(b\) Coverage ratio of ScenePilot to ChatScene in the AV\-risk–physical\-feasibility space\.To better understand the generated scenarios, we analyze their AV–physics characteristics\. Fig\.[4](https://arxiv.org/html/2605.21168#S4.F4)\(a\) compares the phys\-invalid frame rate between ScenePilot and ChatScene in pre\-collision frames \(excluding the final 0\.8 s unavoidable collision phase\)\. ChatScene produces a higher fraction of physically invalid frames \(i\.e\.,σphys<0\\sigma\_\{\\text\{phys\}\}<0\), suggesting that its failures more often involve infeasible interactions before collision\. In contrast, ScenePilot keeps the phys\-invalid rate much lower while still achieving a higher collision rate, indicating that its failures are less dependent on physically invalid behaviors\. We then examine how these frames are distributed in the AV\-risk–physical\-feasibility space\. For each threshold pair\(ϕ,σ\)\(\\phi,\\sigma\), we compute the fraction of frames satisfying both high AV risk and sufficient physical feasibility, and visualize the ScenePilot\-to\-ChatScene coverage ratio in Fig\.[4](https://arxiv.org/html/2605.21168#S4.F4)\(b\)\. While ChatScene is mainly concentrated in the lower\-left region, ScenePilot covers more high\-risk regions on the right side across different feasibility levels\. This suggests that ChatScene tends to induce failures under looser feasibility requirements, whereas ScenePilot more broadly explores hazardous yet physically feasible interactions across different safety margins\.

### 4\.6Ablation Study

Table 5:Ablation study of ScenePilot\.We report collision rate \(CR,↑\\uparrow\), overall score \(OS,↓\\downarrow\), andGap Coverage Score\(GCS,↑\\uparrow\) on two representative base traffic scenarios\. GCS measures the 2D occupancy coverage of the*AV\-infeasible yet physically\-feasible*gap band in the\(ϕ,σ\)\(\\phi,\\sigma\)space \(Appendix[C](https://arxiv.org/html/2605.21168#A3)\)\.We ablate ScenePilot by enabling each boundary component alone while keeping the same TTC\-driven maximization objective\. We consider: \(i\)Φrisk\\Phi\_\{\\text\{risk\}\}only \(constraintΦrisk≤1\\Phi\_\{\\text\{risk\}\}\\\!\\leq\\\!1\) and \(ii\)σphys\\sigma\_\{\\text\{phys\}\}only \(constraintσphys\>0\\sigma\_\{\\text\{phys\}\}\\\!\>\\\!0\)\. Table[5](https://arxiv.org/html/2605.21168#S4.T5)reports CR and OS averaged over SAC/PPO/TD3, together with the Gap Coverage Score \(GCS\), which measures occupancy coverage of the*AV\-infeasible yet physically\-feasible*band in the\(ϕ,σ\)\(\\phi,\\sigma\)space \(Appendix[C](https://arxiv.org/html/2605.21168#A3)\)\. UsingΦrisk\\Phi\_\{\\text\{risk\}\}alone yields the weakest safety\-criticality and smaller GCS, as risk guidance without feasibility control often remains within AV\-feasible interactions and covers a limited subset of failure modes\. Usingσphys\\sigma\_\{\\text\{phys\}\}alone pushes rollouts toward extreme near\-boundary behaviors, which appear adversarial but are less consistent across ego controllers and tend to collapse into a narrower region, reducing GCS\. Combining both boundaries, ScenePilot attains higher CR, lower OS, and the largest GCS, indicating broader coverage of physically feasible yet AV\-challenging interactions without drifting toward overly mild or physically implausible regimes\.

## 5Conclusion

We presented ScenePilot, a controllable boundary\-driven framework for generating safety\-critical autonomous\-driving scenarios\. By coupling a physics\-based feasibility scoreσ\\sigmawith an online\-learned AV\-risk predictorΦ\\Phiunder anϵ\\epsilon\-constrained multi\-objective policy, ScenePilot focuses scenario search on a near\-boundary band that is physically solvable yet highly challenging\. This design yields more critical and transferable adversarial cases across heterogeneous ego controllers, and can further improve ego robustness through adversarial fine\-tuning\.

Limitations\.First, our experiments follow fixed SafeBench base routes, so most diversity concentrates on a short interaction\-critical window rather than global trajectory differences\. Second, surrogate\-driven top\-kkselection may favor failure modes salient to the surrogate controller; although we replay retained episodes across multiple trained ego controllers, future work should consider more controller\-agnostic selection or full\-rollout distribution metrics\.

## Acknowledgements

This work was supported by the Science and Technology Development Fund of Macau \[0123/2022/AFJ, 0081/2022/A2, 0007/2025/RIC, 0122/2024/RIB2, 0215/2024/AGJ, 0074/2025/AMJ, 001/2024/SKL, 0002/2025/EQP\], the Research Services and Knowledge Transfer Office, University of Macau \[SRG2023\-00037\-IOTSC, MYRG\-GRG2024\-00284\-IOTSC\], the Shenzhen\-Hong Kong\-Macau Science and Technology Program Category C \[SGDX20230821095159012\], the Science and Technology Planning Project of Guangdong \[2025A0505010016\], National Natural Science Foundation of China \[52572354\], the State Key Lab of Intelligent Transportation System \[2024\-B001\], and the Jiangsu Provincial Science and Technology Program \[BZ2024055\]\.

## Impact Statement

This paper presents work aimed at enhancing the safety and reliability of autonomous\-driving systems by systematically generating boundary\-guided adversarial interactions that rigorously evaluate driving policies\. By exposing hard\-to\-handle corner cases and providing broader, more targeted coverage of challenging behaviors, our approach can support more rigorous safety evaluation and more robust policy improvement\. Ultimately, this may contribute to reducing failures in complex traffic situations and improving road safety for autonomous vehicles\.

## References

- M\. Althoff and J\. M\. Dolan \(2014\)Online verification of automated road vehicles using reachability analysis\.IEEE Transactions on Robotics30\(4\),pp\. 903–918\.External Links:[Document](https://dx.doi.org/10.1109/TRO.2014.2312453)Cited by:[§1](https://arxiv.org/html/2605.21168#S1.p1.1)\.
- M\. Althoff, S\. Maierhofer, G\. Würsching, Y\. Lin, F\. Lercher, and R\. Stolz \(2025\)No more traffic tickets: a tutorial to ensure traffic\-rule compliance of automated vehicles\.Proceedings of the IEEE\(\),pp\. 1–30\.External Links:[Document](https://dx.doi.org/10.1109/JPROC.2025.3570483)Cited by:[§1](https://arxiv.org/html/2605.21168#S1.p1.1)\.
- X\. Cai, X\. Bai, Z\. Cui, P\. Hang, H\. Yu, and Y\. Ren \(2024\)Adversarial stress test for autonomous vehicle via series reinforcement learning tasks with reward shaping\.IEEE Transactions on Intelligent Vehicles\.Cited by:[§2\.2](https://arxiv.org/html/2605.21168#S2.SS2.p1.1)\.
- CARLA Scenario Runner Contributors \(2019\)CARLA Scenario Runner\.Note:[https://github\.com/carla\-simulator/scenario\_runner](https://github.com/carla-simulator/scenario_runner)GitHub repository, accessed 2026\-01\-24Cited by:[§4\.1](https://arxiv.org/html/2605.21168#S4.SS1.p3.1)\.
- B\. Chen, X\. Chen, Q\. Wu, and L\. Li \(2021\)Adversarial evaluation of autonomous vehicles in lane\-change scenarios\.IEEE transactions on intelligent transportation systems23\(8\),pp\. 10333–10342\.Cited by:[§2\.1](https://arxiv.org/html/2605.21168#S2.SS1.p1.1)\.
- K\. Chen, Y\. Lei, H\. Cheng, H\. Wu, W\. Sun, and S\. Zheng \(2025\)FREA: feasibility\-guided generation of safety\-critical scenarios with reasonable adversariality\.InConference on Robot Learning,pp\. 566–586\.Cited by:[§1](https://arxiv.org/html/2605.21168#S1.p4.1),[§2\.2](https://arxiv.org/html/2605.21168#S2.SS2.p1.1),[§3\.2](https://arxiv.org/html/2605.21168#S3.SS2.p6.2)\.
- W\. Ding, B\. Chen, B\. Li, K\. J\. Eun, and D\. Zhao \(2021\)Multimodal safety\-critical scenarios generation for decision\-making algorithms evaluation\.IEEE Robotics and Automation Letters6\(2\),pp\. 1551–1558\.Cited by:[§2\.1](https://arxiv.org/html/2605.21168#S2.SS1.p1.1)\.
- W\. Ding, B\. Chen, M\. Xu, and D\. Zhao \(2020\)Learning to collide: an adaptive safety\-critical scenarios generating method\.In2020 IEEE/RSJ International Conference on Intelligent Robots and Systems \(IROS\),Vol\.,pp\. 2243–2250\.External Links:[Document](https://dx.doi.org/10.1109/IROS45743.2020.9340696)Cited by:[§1](https://arxiv.org/html/2605.21168#S1.p2.1),[§4\.1](https://arxiv.org/html/2605.21168#S4.SS1.p3.1)\.
- W\. Ding, H\. Lin, B\. Li, and D\. Zhao \(2025\)Semantically adversarial scene generation with explicit knowledge guidance\.IEEE Transactions on Intelligent Transportation Systems26\(2\),pp\. 1510–1521\.External Links:[Document](https://dx.doi.org/10.1109/TITS.2024.3510515)Cited by:[§2\.1](https://arxiv.org/html/2605.21168#S2.SS1.p1.1)\.
- W\. Ding, C\. Xu, M\. Arief, H\. Lin, B\. Li, and D\. Zhao \(2023\)A survey on safety\-critical driving scenario generation—a methodological perspective\.IEEE Transactions on Intelligent Transportation Systems24\(7\),pp\. 6971–6988\.Cited by:[§2\.1](https://arxiv.org/html/2605.21168#S2.SS1.p1.1)\.
- Y\. Dong, L\. Wang, Z\. Li, H\. Li, P\. Tang, C\. Hu, and S\. Guo \(2025\)Safe driving adversarial trajectory can mislead: toward more stealthy adversarial attack against autonomous driving prediction module\.ACM Transactions on Privacy and Security28\(2\),pp\. 1–28\.Cited by:[§2\.1](https://arxiv.org/html/2605.21168#S2.SS1.p1.1)\.
- A\. Dosovitskiy, G\. Ros, F\. Codevilla, A\. Lopez, and V\. Koltun \(2017\)CARLA: an open urban driving simulator\.InConference on robot learning,pp\. 1–16\.Cited by:[§4\.1](https://arxiv.org/html/2605.21168#S4.SS1.p2.1),[Table 2](https://arxiv.org/html/2605.21168#S4.T2),[Table 2](https://arxiv.org/html/2605.21168#S4.T2.2.1.1),[§4](https://arxiv.org/html/2605.21168#S4.p2.1)\.
- S\. Feng, X\. Yan, H\. Sun, Y\. Feng, and H\. X\. Liu \(2021\)Intelligent driving intelligence test for autonomous vehicles with naturalistic and adversarial environment\.Nature communications12\(1\),pp\. 748\.Cited by:[§2\.1](https://arxiv.org/html/2605.21168#S2.SS1.p1.1)\.
- S\. Fujimoto, H\. Hoof, and D\. Meger \(2018\)Addressing function approximation error in actor\-critic methods\.InInternational conference on machine learning,pp\. 1587–1596\.Cited by:[§4\.1](https://arxiv.org/html/2605.21168#S4.SS1.p2.1)\.
- Z\. Ghodsi, S\. K\. S\. Hari, I\. Frosio, T\. Tsai, A\. Troccoli, S\. W\. Keckler, S\. Garg, and A\. Anandkumar \(2021\)Generating and characterizing scenarios for safety testing of autonomous vehicles\.In2021 IEEE Intelligent Vehicles Symposium \(IV\),pp\. 157–164\.Cited by:[§1](https://arxiv.org/html/2605.21168#S1.p2.1)\.
- T\. Haarnoja, A\. Zhou, P\. Abbeel, and S\. Levine \(2018\)Soft actor\-critic: off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.InInternational conference on machine learning,pp\. 1861–1870\.Cited by:[§4\.1](https://arxiv.org/html/2605.21168#S4.SS1.p2.1)\.
- N\. Hanselmann, K\. Renz, K\. Chitta, A\. Bhattacharyya, and A\. Geiger \(2022\)King: generating safety\-critical driving scenarios for robust imitation via kinematics gradients\.InEuropean Conference on Computer Vision,pp\. 335–352\.Cited by:[§4\.1](https://arxiv.org/html/2605.21168#S4.SS1.p2.1),[§4\.2](https://arxiv.org/html/2605.21168#S4.SS2.p2.1),[Table 2](https://arxiv.org/html/2605.21168#S4.T2),[Table 2](https://arxiv.org/html/2605.21168#S4.T2.2.1.1)\.
- K\. Hao, W\. Cui, Y\. Luo, L\. Xie, Y\. Bai, J\. Yang, S\. Yan, Y\. Pan, and Z\. Yang \(2023\)Adversarial safety\-critical scenario generation using naturalistic human driving priors\.IEEE Transactions on Intelligent Vehicles\.Cited by:[§2\.2](https://arxiv.org/html/2605.21168#S2.SS2.p1.1)\.
- O\. Hassanin, X\. Wang, X\. Wu, and X\. Xu \(2022\)Efficiency performance and safety evaluation of the responsibility\-sensitive safety in freeway car\-following scenarios using automated longitudinal controls\.Accident Analysis & Prevention177,pp\. 106799\.Cited by:[§3\.2](https://arxiv.org/html/2605.21168#S3.SS2.p1.1)\.
- L\. Jia, D\. Yang, Y\. Ren, C\. Qian, Q\. Feng, B\. Sun, and Z\. Wang \(2024\)A dynamic test scenario generation method for autonomous vehicles based on conditional generative adversarial imitation learning\.Accident Analysis & Prevention194,pp\. 107279\.Cited by:[§2\.1](https://arxiv.org/html/2605.21168#S2.SS1.p1.1)\.
- N\. D\. Karacik, Y\. Xu, X\. Li, Y\. Hu, and Y\. Liu \(2025\)SCSG: real\-world report–augmented safety\-critical scenario generation for autonomous vehicles\.InEmbodied and Safe\-Assured Robotic Systems,External Links:[Link](https://openreview.net/forum?id=hk3JfyDR1l)Cited by:[§4\.1](https://arxiv.org/html/2605.21168#S4.SS1.p3.1),[Table 1](https://arxiv.org/html/2605.21168#S4.T1),[Table 1](https://arxiv.org/html/2605.21168#S4.T1.2.1.1)\.
- S\. Kuutti, S\. Fallah, and R\. Bowden \(2020\)Training adversarial agents to exploit weaknesses in deep control policies\.In2020 IEEE International Conference on Robotics and Automation \(ICRA\),pp\. 108–114\.Cited by:[§2\.2](https://arxiv.org/html/2605.21168#S2.SS2.p1.1)\.
- W\. Li, C\. Pan, R\. Zhang, J\. Ren, Y\. Ma, J\. Fang, F\. Yan, Q\. Geng, X\. Huang, H\. Gong,et al\.\(2019\)AADS: augmented autonomous driving simulation using data\-driven algorithms\.Science robotics4\(28\),pp\. eaaw0863\.Cited by:[§2\.1](https://arxiv.org/html/2605.21168#S2.SS1.p1.1)\.
- H\. Liu, L\. Zhang, S\. K\. S\. Hari, and J\. Zhao \(2024\)Safety\-critical scenario generation via reinforcement learning based editing\.In2024 IEEE International Conference on Robotics and Automation \(ICRA\),pp\. 14405–14412\.Cited by:[§2\.2](https://arxiv.org/html/2605.21168#S2.SS2.p1.1)\.
- H\. X\. Liu and S\. Feng \(2024\)Curse of rarity for autonomous vehicles\.nature communications15\(1\),pp\. 4808\.Cited by:[§1](https://arxiv.org/html/2605.21168#S1.p1.1)\.
- J\. Liu, Y\. Guo, F\. Zhong, T\. Zhang, Z\. Jing, S\. Liang, J\. Wang, M\. Zhang, A\. Liu, and X\. Liu \(2026\)Adversarial generation and collaborative evolution of safety\-critical scenarios for autonomous vehicles\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 38926–38934\.Cited by:[§4\.1](https://arxiv.org/html/2605.21168#S4.SS1.p3.1),[Table 1](https://arxiv.org/html/2605.21168#S4.T1),[Table 1](https://arxiv.org/html/2605.21168#S4.T1.2.1.1)\.
- R\. Liu, Y\. Pan, L\. Xu, L\. Song, P\. You, Y\. Chen, and J\. Bian \(2025\)Efficient discovery of pareto front for multi\-objective reinforcement learning\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=fDGPIuCdGi)Cited by:[§3\.3](https://arxiv.org/html/2605.21168#S3.SS3.p1.7)\.
- C\. Lu, Y\. Shi, H\. Zhang, M\. Zhang, T\. Wang, T\. Yue, and S\. Ali \(2022\)Learning configurations of operating environment of autonomous vehicles to maximize their collisions\.IEEE Transactions on Software Engineering49\(1\),pp\. 384–402\.Cited by:[§1](https://arxiv.org/html/2605.21168#S1.p2.1)\.
- Q\. Lu, M\. Ma, Z\. Wang, W\. Lu, M\. Ma, G\. Zhao, and S\. Feng \(2026\)Realistic corner case generation for autonomous vehicles with multimodal large language model\.Tsinghua Science and Technology31\(3\),pp\. 1440–1459\.Cited by:[§2\.1](https://arxiv.org/html/2605.21168#S2.SS1.p1.1)\.
- C\. Neurohr, L\. Westhofen, T\. Henning, T\. De Graaff, E\. Möhlmann, and E\. Böde \(2020\)Fundamental considerations around scenario\-based testing for automated driving\.In2020 IEEE intelligent vehicles symposium \(IV\),pp\. 121–127\.Cited by:[§2\.1](https://arxiv.org/html/2605.21168#S2.SS1.p1.1)\.
- A\. Y\. Ng, D\. Harada, and S\. Russell \(1999\)Policy invariance under reward transformations: theory and application to reward shaping\.InIcml,Vol\.99,pp\. 278–287\.Cited by:[§3\.2](https://arxiv.org/html/2605.21168#S3.SS2.p6.7)\.
- H\. Niu, K\. Ren, Y\. Xu, Z\. Yang, Y\. Lin, Y\. Zhang, and J\. Hu \(2023\)\(Re\) 2 h2o: autonomous driving scenario generation via reversely regularized hybrid offline\-and\-online reinforcement learning\.In2023 IEEE Intelligent Vehicles Symposium \(IV\),pp\. 1–8\.Cited by:[§1](https://arxiv.org/html/2605.21168#S1.p2.1)\.
- A\. Prakash, K\. Chitta, and A\. Geiger \(2021\)Multi\-modal fusion transformer for end\-to\-end autonomous driving\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 7077–7087\.Cited by:[§4\.1](https://arxiv.org/html/2605.21168#S4.SS1.p2.1),[Table 2](https://arxiv.org/html/2605.21168#S4.T2),[Table 2](https://arxiv.org/html/2605.21168#S4.T2.2.1.1)\.
- A\. Raghunathan\*, S\. M\. Xie\*, F\. Yang, J\. Duchi, and P\. Liang \(2019\)Adversarial training can hurt generalization\.InICML 2019 Workshop on Identifying and Understanding Deep Learning Phenomena,External Links:[Link](https://openreview.net/forum?id=SyxM3J256E)Cited by:[§1](https://arxiv.org/html/2605.21168#S1.p3.1)\.
- D\. Rempe, J\. Philion, L\. J\. Guibas, S\. Fidler, and O\. Litany \(2022\)Generating useful accident\-prone driving scenarios via a learned traffic prior\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 17305–17315\.Cited by:[§1](https://arxiv.org/html/2605.21168#S1.p2.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§4\.1](https://arxiv.org/html/2605.21168#S4.SS1.p2.1)\.
- S\. Shalev\-Shwartz, S\. Shammah, and A\. Shashua \(2017\)On a formal model of safe and scalable self\-driving cars\.arXiv preprint arXiv:1708\.06374\.Cited by:[§3\.2](https://arxiv.org/html/2605.21168#S3.SS2.p1.1)\.
- R\. Song, M\. O\. Ozmen, H\. Kim, R\. Muller, Z\. B\. Celik, and A\. Bianchi \(2023\)Discovering adversarial driving maneuvers against autonomous vehicles\.In32nd USENIX Security Symposium \(USENIX Security 23\),pp\. 2957–2974\.Cited by:[§1](https://arxiv.org/html/2605.21168#S1.p3.1)\.
- B\. Stoler, I\. Navarro, J\. Francis, and J\. Oh \(2025\)Seal: towards safe autonomous driving via skill\-enabled adversary learning for closed\-loop scenario generation\.IEEE Robotics and Automation Letters10\(9\),pp\. 9320–9327\.Cited by:[§1](https://arxiv.org/html/2605.21168#S1.p3.1)\.
- J\. Sun, H\. Zhang, H\. Zhou, R\. Yu, and Y\. Tian \(2021\)Scenario\-based test automation for highly automated vehicles: a review and paving the way for systematic safety assurance\.IEEE transactions on intelligent transportation systems23\(9\),pp\. 14088–14103\.Cited by:[§1](https://arxiv.org/html/2605.21168#S1.p1.1)\.
- A\. Wachi \(2019\)Failure\-scenario maker for rule\-based agent using multi\-agent adversarial reinforcement learning and its application to autonomous driving\.InInternational Joint Conference on Artificial Intelligence,Cited by:[§1](https://arxiv.org/html/2605.21168#S1.p2.1)\.
- N\. C\. Wagener, B\. Boots, and C\. Cheng \(2021\)Safe reinforcement learning using advantage\-based intervention\.InInternational Conference on Machine Learning,pp\. 10630–10640\.Cited by:[§3\.3](https://arxiv.org/html/2605.21168#S3.SS3.p1.7)\.
- J\. Wang, A\. Pun, J\. Tu, S\. Manivasagam, A\. Sadat, S\. Casas, M\. Ren, and R\. Urtasun \(2021\)Advsim: generating safety\-critical scenarios for self\-driving vehicles\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 9909–9918\.Cited by:[§1](https://arxiv.org/html/2605.21168#S1.p3.1),[§4\.1](https://arxiv.org/html/2605.21168#S4.SS1.p3.1)\.
- Z\. Wei, H\. Huang, G\. Zhang, R\. Zhou, X\. Luo, S\. Li, and H\. Zhou \(2024\)Interactive critical scenario generation for autonomous vehicles testing based on in\-depth crash data using reinforcement learning\.IEEE Transactions on Intelligent Vehicles\.Cited by:[§2\.2](https://arxiv.org/html/2605.21168#S2.SS2.p1.1)\.
- C\. Xu, W\. Ding, W\. Lyu, Z\. Liu, S\. Wang, Y\. He, H\. Hu, D\. Zhao, and B\. Li \(2022\)Safebench: a benchmarking platform for safety evaluation of autonomous vehicles\.Advances in Neural Information Processing Systems35,pp\. 25667–25682\.Cited by:[§4](https://arxiv.org/html/2605.21168#S4.p1.1),[§4](https://arxiv.org/html/2605.21168#S4.p2.1)\.
- Z\. Yang, Y\. Chen, J\. Wang, S\. Manivasagam, W\. Ma, A\. J\. Yang, and R\. Urtasun \(2023\)Unisim: a neural closed\-loop sensor simulator\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 1389–1399\.Cited by:[§2\.1](https://arxiv.org/html/2605.21168#S2.SS1.p1.1)\.
- H\. Yu, W\. Xu, and H\. Zhang \(2022\)Towards safe reinforcement learning with a safety editor policy\.Advances in Neural Information Processing Systems35,pp\. 2608–2621\.Cited by:[§3\.3](https://arxiv.org/html/2605.21168#S3.SS3.p1.7)\.
- J\. Zhang, C\. Xu, and B\. Li \(2024\)Chatscene: knowledge\-enabled safety\-critical scenario generation for autonomous vehicles\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 15459–15469\.Cited by:[§B\.4](https://arxiv.org/html/2605.21168#A2.SS4.p1.1),[Appendix B](https://arxiv.org/html/2605.21168#A2.p1.1),[Appendix C](https://arxiv.org/html/2605.21168#A3.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2605.21168#S4.SS1.p3.1),[§4\.4](https://arxiv.org/html/2605.21168#S4.SS4.p1.1),[Table 1](https://arxiv.org/html/2605.21168#S4.T1),[Table 1](https://arxiv.org/html/2605.21168#S4.T1.2.1.1)\.
- Q\. Zhang, S\. Hu, J\. Sun, Q\. A\. Chen, and Z\. M\. Mao \(2022\)On adversarial robustness of trajectory prediction for autonomous vehicles\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 15159–15168\.Cited by:[§4\.1](https://arxiv.org/html/2605.21168#S4.SS1.p3.1)\.

## Appendix AAdditional Details of ScenePilot

### A\.1Physical Safety Signalσ\\sigma

#### Physics\-limit braking distances\.

RSS prescribes appropriate responses for AVs via five common\-sense rules:

- •Rule 1:Maintain a safe longitudinal distance \(same\-direction and oncoming\)\.
- •Rule 2:Maintain a safe lateral distance \(e\.g\., merging/cut\-in\)\.
- •Rule 3:Do not fight for the right\-of\-way\.
- •Rule 4:Be cautious under limited visibility\.
- •Rule 5:If a collision can be avoided without causing another, it must be avoided\.

Under Rules 1\-2, RSS specifies explicit safe\-distance formulas in both longitudinal and lateral directions by Eq\. \([11](https://arxiv.org/html/2605.21168#A1.E11)\)\. Rule 3 only for posterior responsibility in our fully observable setting\. Letρ\\rhodenote the reaction time\. The two interacting vehicles are indexed byx∈\{r,f\}x\\in\\\{r,f\\\}, whererris the rear \(left in lateral\) vehicle andffis the front \(right in lateral\) one\. The scalarsvxlonv\_\{x\}^\{\\mathrm\{lon\}\}andvxlatv\_\{x\}^\{\\mathrm\{lat\}\}denote the longitudinal and lateral speed components of vehiclexx\. The maximal accelerations during the reaction window byax,maxlon,accela\_\{x,\\max\}^\{\\mathrm\{lon,accel\}\}andax,maxlat,accela\_\{x,\\max\}^\{\\mathrm\{lat,accel\}\}, and the conservative RSS braking bounds byax,minlon,brakea\_\{x,\\min\}^\{\\mathrm\{lon,brake\}\}andax,minlat,brakea\_\{x,\\min\}^\{\\mathrm\{lat,brake\}\}\. Define the reaction\-updated speeds asvx,ρlon=vxlon\+ρ​ax,maxlon,accelv\_\{x,\\rho\}^\{\\mathrm\{lon\}\}=v\_\{x\}^\{\\mathrm\{lon\}\}\+\\rho a\_\{x,\\max\}^\{\\mathrm\{lon,accel\}\}andvx,ρlat=vxlat\+ρ​ax,maxlat,accelv\_\{x,\\rho\}^\{\\mathrm\{lat\}\}=v\_\{x\}^\{\\mathrm\{lat\}\}\+\\rho a\_\{x,\\max\}^\{\\mathrm\{lat,accel\}\}\.

dsafelon,opp=vrlon\+vr,ρlon2​ρ\+\(vr,ρlon\)22​ar,minlon,brake\+\|vflon\|\+\|vf,ρlon\|2​ρ\+\(vf,ρlon\)22​af,minlon,brake,\\displaystyle d\_\{\\mathrm\{safe\}\}^\{\\mathrm\{lon,opp\}\}=\\dfrac\{v\_\{r\}^\{\\mathrm\{lon\}\}\+v\_\{r,\\rho\}^\{\\mathrm\{lon\}\}\}\{2\}\\rho\+\\dfrac\{\(v\_\{r,\\rho\}^\{\\mathrm\{lon\}\}\)^\{2\}\}\{2a\_\{r,\\min\}^\{\\mathrm\{lon,brake\}\}\}\+\\dfrac\{\|v\_\{f\}^\{\\mathrm\{lon\}\}\|\+\|v\_\{f,\\rho\}^\{\\mathrm\{lon\}\}\|\}\{2\}\\rho\+\\dfrac\{\(v\_\{f,\\rho\}^\{\\mathrm\{lon\}\}\)^\{2\}\}\{2a\_\{f,\\min\}^\{\\mathrm\{lon,brake\}\}\},\(11\)dsafelon,same=\[vrlon​ρ\+12​ar,maxlon,accel​ρ2\+\(vr,ρlon\)22​ar,minlon,brake−\(vflon\)22​af,maxlon,brake\]\+,\\displaystyle d\_\{\\mathrm\{safe\}\}^\{\\mathrm\{lon,same\}\}=\\Big\[v\_\{r\}^\{\\mathrm\{lon\}\}\\rho\+\\dfrac\{1\}\{2\}a\_\{r,\\max\}^\{\\mathrm\{lon,accel\}\}\\rho^\{2\}\+\\dfrac\{\(v\_\{r,\\rho\}^\{\\mathrm\{lon\}\}\)^\{2\}\}\{2a\_\{r,\\min\}^\{\\mathrm\{lon,brake\}\}\}\-\\dfrac\{\(v\_\{f\}^\{\\mathrm\{lon\}\}\)^\{2\}\}\{2a\_\{f,\\max\}^\{\\mathrm\{lon,brake\}\}\}\\Big\]\_\{\+\},dsafelat=μ\+\[vrlat\+vr,ρlat2​ρ\+\(vr,ρlat\)22​ar,minlat,brake−\(\|vflat\|\+\|vf,ρlat\|2​ρ−\(vf,ρlat\)22​af,minlat,brake\)\]\+\.\\displaystyle d\_\{\\mathrm\{safe\}\}^\{\\mathrm\{lat\}\}=\\mu\+\\Big\[\\dfrac\{v\_\{r\}^\{\\mathrm\{lat\}\}\+v\_\{r,\\rho\}^\{\\mathrm\{lat\}\}\}\{2\}\\rho\+\\dfrac\{\(v\_\{r,\\rho\}^\{\\mathrm\{lat\}\}\)^\{2\}\}\{2a\_\{r,\\min\}^\{\\mathrm\{lat,brake\}\}\}\-\\Big\(\\dfrac\{\|v\_\{f\}^\{\\mathrm\{lat\}\}\|\+\|v\_\{f,\\rho\}^\{\\mathrm\{lat\}\}\|\}\{2\}\\rho\-\\dfrac\{\(v\_\{f,\\rho\}^\{\\mathrm\{lat\}\}\)^\{2\}\}\{2a\_\{f,\\min\}^\{\\mathrm\{lat,brake\}\}\}\\Big\)\\Big\]\_\{\+\}\.
In RSS’s formulation \(Eq\. \([11](https://arxiv.org/html/2605.21168#A1.E11)\)\), the safe\-distance terms are conservative mainly due to: \(i\) a finite reaction windowρ\\rhoduring which the rear/merging vehicle may continue moving \(and potentially accelerating\), and \(ii\) conservative braking assumptions captured by the minimum deceleration boundsax,minlon,brakea\_\{x,\\min\}^\{\\mathrm\{lon,brake\}\}andax,minlat,brakea\_\{x,\\min\}^\{\\mathrm\{lat,brake\}\}\(together with the lateral bufferμ\\mufor cut\-in/merge\)\. Intuitively, this formulation enforces a sufficient separation as long as the follower has the intention to brake within the assumed response model, which can overestimate the distance required to avoid collision in near\-miss interactions\. For analysis, we therefore also consider an idealized limiting case that removes the reaction delay and characterizes the kinematic boundary under maximal feasible braking, yielding the physics\-limit longitudinal/lateral distances in Eq\. \([2](https://arxiv.org/html/2605.21168#S3.E2)\)\.

The opposite\-direction cases follow directly from summing the two axis\-wise stopping distances under maximal braking\. For the same\-direction longitudinal case, however, the tight braking\-only boundary depends on the interaction parameters and is therefore written as a maximum of two necessary lower bounds\. Consider 1D motion along the longitudinal axis for a rear vehiclerrand a front vehicleff, with initial speedsvrlonv\_\{r\}^\{\\mathrm\{lon\}\}andvflonv\_\{f\}^\{\\mathrm\{lon\}\}, and maximal feasible braking decelerationsar:=ar,maxlon,brakea\_\{r\}:=a\_\{r,\\max\}^\{\\mathrm\{lon,brake\}\}andaf:=af,maxlon,brakea\_\{f\}:=a\_\{f,\\max\}^\{\\mathrm\{lon,brake\}\}\. Letd0d\_\{0\}denote the initial longitudinal gap\. One necessary condition is obtained by comparing the two vehicles’ stopping distances, which yields

d0≥dstoplon=\(vrlon\)22​ar−\(vflon\)22​af\.d\_\{0\}\\ \\geq\\ d\_\{\\mathrm\{stop\}\}^\{\\mathrm\{lon\}\}=\\frac\{\(v\_\{r\}^\{\\mathrm\{lon\}\}\)^\{2\}\}\{2a\_\{r\}\}\-\\frac\{\(v\_\{f\}^\{\\mathrm\{lon\}\}\)^\{2\}\}\{2a\_\{f\}\}\.\(12\)A second necessary condition comes from requiring that the inter\-vehicle gap never becomes negative during simultaneous braking\. With constant decelerations, the relative distance evolves as

d​\(t\)=d0\+\(vflon−vrlon\)​t\+12​\(ar−af\)​t2\.d\(t\)=d\_\{0\}\+\(v\_\{f\}^\{\\mathrm\{lon\}\}\-v\_\{r\}^\{\\mathrm\{lon\}\}\)t\+\\tfrac\{1\}\{2\}\(a\_\{r\}\-a\_\{f\}\)t^\{2\}\.\(13\)When a closing interaction occurs \(vrlon\>vflonv\_\{r\}^\{\\mathrm\{lon\}\}\>v\_\{f\}^\{\\mathrm\{lon\}\}\) and the rear can decelerate more strongly \(ar\>afa\_\{r\}\>a\_\{f\}\), the quadratic in \([13](https://arxiv.org/html/2605.21168#A1.E13)\) attains its minimum at

t⋆=vrlon−vflonar−af,t^\{\\star\}=\\frac\{v\_\{r\}^\{\\mathrm\{lon\}\}\-v\_\{f\}^\{\\mathrm\{lon\}\}\}\{a\_\{r\}\-a\_\{f\}\},\(14\)which is exactly the time when the two longitudinal velocities become equal\. Requiring the minimum gap to remain nonnegative,d​\(t⋆\)≥0d\(t^\{\\star\}\)\\geq 0, gives

d0≥dequallon=\(vrlon−vflon\)22​\(ar−af\)\.d\_\{0\}\\ \\geq\\ d\_\{\\mathrm\{equal\}\}^\{\\mathrm\{lon\}\}=\\frac\{\(v\_\{r\}^\{\\mathrm\{lon\}\}\-v\_\{f\}^\{\\mathrm\{lon\}\}\)^\{2\}\}\{2\(a\_\{r\}\-a\_\{f\}\)\}\.\(15\)
To make the parameter dependence explicit, introduce the dimensionless ratiosκ:=araf\\kappa:=\\frac\{a\_\{r\}\}\{a\_\{f\}\},η:=vrlonvflon\\eta:=\\frac\{v\_\{r\}^\{\\mathrm\{lon\}\}\}\{v\_\{f\}^\{\\mathrm\{lon\}\}\}\. In the common closing regimevrlon\>vflonv\_\{r\}^\{\\mathrm\{lon\}\}\>v\_\{f\}^\{\\mathrm\{lon\}\}andar\>afa\_\{r\}\>a\_\{f\}\(i\.e\.,η\>1\\eta\>1andκ\>1\\kappa\>1\), the two bounds in \([12](https://arxiv.org/html/2605.21168#A1.E12)\)–\([15](https://arxiv.org/html/2605.21168#A1.E15)\) can be compared as

dstoplon−dequallon\\displaystyle d\_\{\\mathrm\{stop\}\}^\{\\mathrm\{lon\}\}\-d\_\{\\mathrm\{equal\}\}^\{\\mathrm\{lon\}\}=\(vflon\)22​af​\(η2κ−1−\(η−1\)2κ−1\)\\displaystyle=\\frac\{\(v\_\{f\}^\{\\mathrm\{lon\}\}\)^\{2\}\}\{2a\_\{f\}\}\\left\(\\frac\{\\eta^\{2\}\}\{\\kappa\}\-1\-\\frac\{\(\\eta\-1\)^\{2\}\}\{\\kappa\-1\}\\right\)=\(vflon\)22​af⋅\(η2−κ\)​\(κ−1\)−κ​\(η−1\)2κ​\(κ−1\)\.\\displaystyle=\\frac\{\(v\_\{f\}^\{\\mathrm\{lon\}\}\)^\{2\}\}\{2a\_\{f\}\}\\cdot\\frac\{\(\\eta^\{2\}\-\\kappa\)\(\\kappa\-1\)\-\\kappa\(\\eta\-1\)^\{2\}\}\{\\kappa\(\\kappa\-1\)\}\.\(16\)Thus, depending on\(vrlon,vflon,ar,af\)\(v\_\{r\}^\{\\mathrm\{lon\}\},v\_\{f\}^\{\\mathrm\{lon\}\},a\_\{r\},a\_\{f\}\), eitherdstoplond\_\{\\mathrm\{stop\}\}^\{\\mathrm\{lon\}\}ordequallond\_\{\\mathrm\{equal\}\}^\{\\mathrm\{lon\}\}can be the tighter constraint\. Since collision avoidance under braking\-only responses must satisfy both necessary conditions, and the active one varies with the interaction parameters, we adopt the unified expression

dsamelon=max⁡\(dstoplon,dequallon\),d\_\{\\mathrm\{same\}\}^\{\\mathrm\{lon\}\}=\\max\\\!\\Big\(d\_\{\\mathrm\{stop\}\}^\{\\mathrm\{lon\}\},\\,d\_\{\\mathrm\{equal\}\}^\{\\mathrm\{lon\}\}\\Big\),\(17\)which matches the same\-direction longitudinal term in Eq\. \([2](https://arxiv.org/html/2605.21168#S3.E2)\)\. Finally, this axis\-wise braking\-limit distance provides a tighter notion of what is physically avoidable: ifdactual<dlimitd\_\{\\mathrm\{actual\}\}<d\_\{\\mathrm\{limit\}\}on an axis, then a collision is unavoidable under braking\-only responses along that axis\.

#### Physical safety signalσ\\sigma\.

As discussed in Sec\. 3\.2, the physics\-limit distancedlimitd\_\{\\mathrm\{limit\}\}characterizes the boundary of braking\-only recoverability along a given axis\. However, near\-boundary interactions can still be solvable via orthogonal maneuvers \(e\.g\., evasive steering / lane change\) when sufficient time and dynamic margin exist\. We therefore aggregate the axis\-wise residual margins into a single physical safety signalσ\\sigmain Eq\. \([3](https://arxiv.org/html/2605.21168#S3.E3)\)\. Below we detail how each term in Eq\. \([3](https://arxiv.org/html/2605.21168#S3.E3)\) is computed\.

We first compute the adversary’s relative position in the ego\-fixed body frame\. Letpego=\[xego,yego\]⊤p\_\{\\mathrm\{ego\}\}=\[x\_\{\\mathrm\{ego\}\},y\_\{\\mathrm\{ego\}\}\]^\{\\top\}andpadv=\[xadv,yadv\]⊤p\_\{\\mathrm\{adv\}\}=\[x\_\{\\mathrm\{adv\}\},y\_\{\\mathrm\{adv\}\}\]^\{\\top\}be the world\-frame centers, and letR​\(−ψego\)R\(\-\\psi\_\{\\mathrm\{ego\}\}\)rotate world coordinates into the ego frame\. Then

\[dxactualdyactual\]=R​\(−ψego\)​\(padv−pego\),R​\(−ψego\)=\[cos⁡ψegosin⁡ψego−sin⁡ψegocos⁡ψego\]\.\\begin\{bmatrix\}d\_\{x\}^\{\\mathrm\{actual\}\}\\\\ d\_\{y\}^\{\\mathrm\{actual\}\}\\end\{bmatrix\}=R\(\-\\psi\_\{\\mathrm\{ego\}\}\)\\,\(p\_\{\\mathrm\{adv\}\}\-p\_\{\\mathrm\{ego\}\}\),\\qquad R\(\-\\psi\_\{\\mathrm\{ego\}\}\)=\\begin\{bmatrix\}\\cos\\psi\_\{\\mathrm\{ego\}\}&\\sin\\psi\_\{\\mathrm\{ego\}\}\\\\ \-\\sin\\psi\_\{\\mathrm\{ego\}\}&\\cos\\psi\_\{\\mathrm\{ego\}\}\\end\{bmatrix\}\.\(18\)To account for finite vehicle sizes and relative yaw, we use the envelope terms in Eq\. \([3](https://arxiv.org/html/2605.21168#S3.E3)\)\. GivenΔ​ψ=ψadv−ψego\\Delta\\psi=\\psi\_\{\\mathrm\{adv\}\}\-\\psi\_\{\\mathrm\{ego\}\}, define

sx​\(Δ​ψ\)\\displaystyle s\_\{x\}\(\\Delta\\psi\)=Lego\+\|cos⁡Δ​ψ\|​Ladv\+\|sin⁡Δ​ψ\|​Wadv,\\displaystyle=L\_\{\\mathrm\{ego\}\}\+\|\\cos\\Delta\\psi\|\\,L\_\{\\mathrm\{adv\}\}\+\|\\sin\\Delta\\psi\|\\,W\_\{\\mathrm\{adv\}\},\(19\)sy​\(Δ​ψ\)\\displaystyle s\_\{y\}\(\\Delta\\psi\)=Wego\+\|cos⁡Δ​ψ\|​Wadv\+\|sin⁡Δ​ψ\|​Ladv,\\displaystyle=W\_\{\\mathrm\{ego\}\}\+\|\\cos\\Delta\\psi\|\\,W\_\{\\mathrm\{adv\}\}\+\|\\sin\\Delta\\psi\|\\,L\_\{\\mathrm\{adv\}\},and the signed edge\-to\-edge clearances

δx=\|dxactual\|−sx​\(Δ​ψ\),δy=\|dyactual\|−sy​\(Δ​ψ\)\.\\delta\_\{x\}=\|d\_\{x\}^\{\\mathrm\{actual\}\}\|\-s\_\{x\}\(\\Delta\\psi\),\\qquad\\delta\_\{y\}=\|d\_\{y\}^\{\\mathrm\{actual\}\}\|\-s\_\{y\}\(\\Delta\\psi\)\.\(20\)
We estimate the time before the first axis\-wise collision event using the instantaneous closing speeds:

tx=\[dxactual−Wcar\]\+\[Δ​vx\]\+,ty=\[dyactual−Lcar\]\+\[Δ​vy\]\+,t=min⁡\(tx,ty\),t\_\{x\}=\\frac\{\[\\,d\_\{x\}^\{\\mathrm\{actual\}\}\-W\_\{\\mathrm\{car\}\}\\,\]\_\{\+\}\}\{\[\\,\\Delta v\_\{x\}\\,\]\_\{\+\}\},\\qquad t\_\{y\}=\\frac\{\[\\,d\_\{y\}^\{\\mathrm\{actual\}\}\-L\_\{\\mathrm\{car\}\}\\,\]\_\{\+\}\}\{\[\\,\\Delta v\_\{y\}\\,\]\_\{\+\}\},\\qquad t=\\min\(t\_\{x\},t\_\{y\}\),\(21\)where\[⋅\]\+\[\\cdot\]\_\{\+\}denotesmax⁡\(0,⋅\)\\max\(0,\\cdot\)\. This TTC \(time to collision\) is a local kinematic estimate computed from the current relative state\. Formally, lettingdc​\(τ\)d\_\{c\}\(\\tau\)be the edge\-to\-edge clearance on axisc∈\{x,y\}c\\in\\\{x,y\\\}andΔ​vc​\(τ\)\\Delta v\_\{c\}\(\\tau\)be the corresponding closing speed, the collision time under the instantaneous state satisfies

0≈dc​\(t\)=dc​\(0\)−∫0t\[Δ​vc​\(τ\)\]\+​𝑑τ\.0\\approx d\_\{c\}\(t\)\\;=\\;d\_\{c\}\(0\)\-\\int\_\{0\}^\{t\}\[\\,\\Delta v\_\{c\}\(\\tau\)\\,\]\_\{\+\}\\,d\\tau\.\(22\)Approximating the integrand by its current value \(a per\-frame constant\-velocity approximation\),\[Δ​vc​\(τ\)\]\+≈\[Δ​vc​\(0\)\]\+\[\\,\\Delta v\_\{c\}\(\\tau\)\\,\]\_\{\+\}\\approx\[\\,\\Delta v\_\{c\}\(0\)\\,\]\_\{\+\}, gives

t≈\[dc​\(0\)\]\+\[Δ​vc​\(0\)\]\+,t\\approx\\frac\{\[\\,d\_\{c\}\(0\)\\,\]\_\{\+\}\}\{\[\\,\\Delta v\_\{c\}\(0\)\\,\]\_\{\+\}\},\(23\)which yields Eq\. \([21](https://arxiv.org/html/2605.21168#A1.E21)\) after substituting the axis\-specific clearancesdx​\(0\)=dxactual−Wcard\_\{x\}\(0\)=d\_\{x\}^\{\\mathrm\{actual\}\}\-W\_\{\\mathrm\{car\}\}anddy​\(0\)=dyactual−Lcard\_\{y\}\(0\)=d\_\{y\}^\{\\mathrm\{actual\}\}\-L\_\{\\mathrm\{car\}\}and taking the earliest axis\-wise event viat=min⁡\(tx,ty\)t=\\min\(t\_\{x\},t\_\{y\}\)\. Because we evaluateσ\\sigmaat every simulator frame, this local approximation is repeatedly refreshed along the rollout and acts as a discrete\-time accumulation of the evolving relative motion\. In contrast, computing TTC by assuming future maximal braking would correspond to replacing\[Δ​vc​\(τ\)\]\+\[\\,\\Delta v\_\{c\}\(\\tau\)\\,\]\_\{\+\}in \([22](https://arxiv.org/html/2605.21168#A1.E22)\) with a braking\-shaped profile, which can systematically enlargettand thus over\-credit recoverability near the boundary\.

Letc=arg⁡min⁡\{tx,ty\}c=\\arg\\min\\\{t\_\{x\},t\_\{y\}\\\}be the first\-colliding axis andn=\{x,y\}∖\{c\}n=\\\{x,y\\\}\\setminus\\\{c\\\}its orthogonal axis\. We setlc=0l\_\{c\}=0and compensate only along the orthogonal axis by the reachable displacement within timett:

ln=12​anrel​t2​1​\{Δ​vn\>0\},l\_\{n\}=\\tfrac\{1\}\{2\}\\,a^\{\\mathrm\{rel\}\}\_\{n\}\\,t^\{2\}\\,\\mathbf\{1\}\\\{\\Delta v\_\{n\}\>0\\\},\(24\)so that\(lx,ly\)=\(0,ln\)\(l\_\{x\},l\_\{y\}\)=\(0,l\_\{n\}\)ifc=xc=xand\(lx,ly\)=\(ln,0\)\(l\_\{x\},l\_\{y\}\)=\(l\_\{n\},0\)ifc=yc=y\. This12​a​t2\\tfrac\{1\}\{2\}at^\{2\}form follows from the constant\-acceleration kinematic relation for displacement, and we use it as an upper bound under a physics\-limit assumption: once the imminent\-collision axis is identified, the most favorable solvable behavior is to generate separation along the orthogonal axis with maximal feasible relative acceleration\. Accordingly,lnl\_\{n\}can be interpreted as the additional clearance that could be created before the earliest collision time, while𝟏​\{Δ​vn\>0\}\\mathbf\{1\}\\\{\\Delta v\_\{n\}\>0\\\}avoids crediting compensation when the orthogonal gap is not closing\.

Finally, lettingdxlimitd\_\{x\}^\{\\mathrm\{limit\}\}anddylimitd\_\{y\}^\{\\mathrm\{limit\}\}be the axis\-wise physics\-limit distances from Eq\. \([2](https://arxiv.org/html/2605.21168#S3.E2)\) \(projected onto the egox/yx/yaxes\), Eq\. \([3](https://arxiv.org/html/2605.21168#S3.E3)\) can be evaluated equivalently in the explicit scalar form

σ=1−\(\\displaystyle\\sigma=1\-\\Bigg\(\[\[dxlimit−δx−lx\]\+dxlimit\]p\+\[\[dylimit−δy−ly\]\+dylimit\]p\)1/p\.\\displaystyle\\Bigg\[\\frac\{\\Bigl\[d\_\{x\}^\{\\mathrm\{limit\}\}\-\\delta\_\{x\}\-l\_\{x\}\\Bigr\]\_\{\+\}\}\{d\_\{x\}^\{\\mathrm\{limit\}\}\}\\Bigg\]^\{p\}\+\\Bigg\[\\frac\{\\Bigl\[d\_\{y\}^\{\\mathrm\{limit\}\}\-\\delta\_\{y\}\-l\_\{y\}\\Bigr\]\_\{\+\}\}\{d\_\{y\}^\{\\mathrm\{limit\}\}\}\\Bigg\]^\{p\}\\Bigg\)^\{\\\!1/p\}\.\(25\)In our implementation, we setp=2p=2, which corresponds to anℓ2\\ell\_\{2\}aggregation of the normalized axis\-wise gaps and yields an elliptical level\-set geometry in the\(x,y\)\(x,y\)plane \(a convenient surrogate for the vehicle footprint when combining longitudinal and lateral margins\)\. By construction,σ\>0\\sigma\>0indicates physically feasible frames,σ=0\\sigma=0lies on the kinematic boundary, andσ<0\\sigma<0indicates physically infeasible frames where no admissible combination of braking and steering can prevent a collision\.

### A\.2AV Risk Signalϕ\\phi

For per\-frame AV risk estimation, a lightweight predictor is needed that can be queried at every frame with negligible overhead\. Instead of introducing an additional planner or a heavy safety module, aa Bellman\-style formulation is trained to approximate the discounted collision\-to\-go return via Eq\. \([26](https://arxiv.org/html/2605.21168#A1.E26)\), which is theoretically sufficient\.

Φ~\(st\)=𝔼\[𝟙\{𝒞t\}\+γΦ~\(st\+1\)\|st\]\.\\tilde\{\\Phi\}\(s\_\{t\}\)\\;=\\;\\mathbb\{E\}\\\!\\left\[\\mathbb\{1\}\\\{\\mathcal\{C\}\_\{t\}\\\}\+\\gamma\\,\\tilde\{\\Phi\}\(s\_\{t\+1\}\)\\,\\middle\|\\,s\_\{t\}\\right\]\.\(26\)
However, collisions are rare and most frames satisfy𝟙​\{𝒞t\}=0\\mathbb\{1\}\\\{\\mathcal\{C\}\_\{t\}\\\}=0\. In long non\-collision segments, the recursion in Eq\. \([26](https://arxiv.org/html/2605.21168#A1.E26)\) is dominated by bootstrap termγ​Φ~​\(st\+1\)\\gamma\\,\\tilde\{\\Phi\}\(s\_\{t\+1\}\), yielding weak learning signal for earlier frames and causing the estimated risk to drift\. We therefore use the shaped target in Eq\. \([5](https://arxiv.org/html/2605.21168#S3.E5)\) by inserting a potential\-based shaping termγ​F​\(st\+1\)−F​\(st\)\\gamma F\(s\_\{t\+1\}\)\-F\(s\_\{t\}\)into the per\-step part of the TD target\. Concretely, along a rollouts0→s1→s2→⋯s\_\{0\}\\to s\_\{1\}\\to s\_\{2\}\\to\\cdots, the shaping contributions at successive steps take the form

γ​F​\(s1\)\\displaystyle\\gamma F\(s\_\{1\}\)−F​\(s0\),\\displaystyle\-F\(s\_\{0\}\),γ​F​\(s2\)\\displaystyle\\gamma F\(s\_\{2\}\)−F​\(s1\),\\displaystyle\-F\(s\_\{1\}\),γ​F​\(s3\)\\displaystyle\\gamma F\(s\_\{3\}\)−F​\(s2\),…\\displaystyle\-F\(s\_\{2\}\),\\ \\ldots
so each step is given an additional dense signal that depends on how the state potential changes\. That is, the per\-step signal is replaced from𝟙​\{𝒞t\}\\mathbb\{1\}\\\{\\mathcal\{C\}\_\{t\}\\\}to the shaped form𝟙​\{𝒞t\}\+γ​F​\(st\+1\)−F​\(st\)\\mathbb\{1\}\\\{\\mathcal\{C\}\_\{t\}\\\}\+\\gamma F\(s\_\{t\+1\}\)\-F\(s\_\{t\}\)\. Accordingly, the shaped per\-step reward as

rt^=1​\{𝒞t\}\+γ​F​\(st\+1\)−F​\(st\)\.\\hat\{r\_\{t\}\}\\;=\\;\\mathbb\{1\}\\\{\\mathcal\{C\}\_\{t\}\\\}\+\\gamma F\(s\_\{t\+1\}\)\-F\(s\_\{t\}\)\.\(27\)
so the TD target in Eq\. \([5](https://arxiv.org/html/2605.21168#S3.E5)\) can be written asyt=rt^\+γ​Φ^​\(st\+1\)y\_\{t\}=\\hat\{r\_\{t\}\}\+\\gamma\\hat\{\\Phi\}\(s\_\{t\+1\}\)\. The corresponding discounted shaped return is

G^\\displaystyle\\hat\{G\}=∑t=0Tγt​rt^=∑t=0Tγt​𝟙​\{𝒞t\}\+∑t=0Tγt​\(γ​F​\(st\+1\)−F​\(st\)\)\.\\displaystyle=\\sum\_\{t=0\}^\{T\}\\gamma^\{t\}\\hat\{r\_\{t\}\}=\\sum\_\{t=0\}^\{T\}\\gamma^\{t\}\\mathbb\{1\}\\\{\\mathcal\{C\}\_\{t\}\\\}\\;\+\\;\\sum\_\{t=0\}^\{T\}\\gamma^\{t\}\\big\(\\gamma F\(s\_\{t\+1\}\)\-F\(s\_\{t\}\)\\big\)\.\(28\)The shaping sum telescopes

∑t=0Tγt​\(γ​F​\(st\+1\)−F​\(st\)\)\\displaystyle\\sum\_\{t=0\}^\{T\}\\gamma^\{t\}\\big\(\\gamma F\(s\_\{t\+1\}\)\-F\(s\_\{t\}\)\\big\)=∑t=0Tγt\+1​F​\(st\+1\)−∑t=0Tγt​F​\(st\)\\displaystyle=\\sum\_\{t=0\}^\{T\}\\gamma^\{t\+1\}F\(s\_\{t\+1\}\)\-\\sum\_\{t=0\}^\{T\}\\gamma^\{t\}F\(s\_\{t\}\)\(29\)=γT\+1​F​\(sT\+1\)−F​\(s0\)\.\\displaystyle=\\gamma^\{T\+1\}F\(s\_\{T\+1\}\)\-F\(s\_\{0\}\)\.IfFFis bounded andγ∈\(0,1\)\\gamma\\in\(0,1\), thenlimT→∞γT\+1​F​\(sT\+1\)=0\\lim\_\{T\\to\\infty\}\\gamma^\{T\+1\}F\(s\_\{T\+1\}\)=0, and thus

G^=∑t=0∞γt​𝟙​\{𝒞t\}⏟G−F​\(s0\)\.\\hat\{G\}=\\underbrace\{\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\mathbb\{1\}\\\{\\mathcal\{C\}\_\{t\}\\\}\}\_\{G\}\\;\-\\;F\(s\_\{0\}\)\.\(30\)
Therefore, potential shaping preserves the underlying collision\-to\-go objective and only introduces a state\-dependent constant shift in the return at the starting state\. Taking conditional expectation yields the shifted value\-function relation

Φ^​\(s\)=𝔼​\[G^∣s0=s\]=𝔼​\[G∣s0=s\]⏟Φ​\(s\)−F​\(s\),\\hat\{\\Phi\}\(s\)=\\mathbb\{E\}\[\\hat\{G\}\\mid s\_\{0\}=s\]\\;=\\;\\underbrace\{\\mathbb\{E\}\[G\\mid s\_\{0\}=s\]\}\_\{\\Phi\(s\)\}\\;\-\\;F\(s\),\(31\)which impliesΦ​\(s\)=Φ^​\(s\)\+F​\(s\)\\Phi\(s\)=\\hat\{\\Phi\}\(s\)\+F\(s\)\. Consequently, when the critic is trained with Eq\. \([5](https://arxiv.org/html/2605.21168#S3.E5)\), we recover the AV\-risk signal on the original scale by adding back the potential

ϕt:=Φ​\(st\)=Φ^​\(st\)\+F​\(st\)\.\\phi\_\{t\}\\;:=\\;\\Phi\(s\_\{t\}\)\\;=\\;\\hat\{\\Phi\}\(s\_\{t\}\)\+F\(s\_\{t\}\)\.\(32\)

### A\.3Feasibility Thresholdε\\varepsilon

ScenePilot uses the feasibility thresholdε\\varepsilonto control the safety margin from the physical feasibility boundary\. Sinceσ=0\\sigma=0corresponds to the physical feasibility boundary, small positiveε\\varepsilonvalues probe near\-boundary interactions where failures can still be induced, while largerε\\varepsilonvalues impose stricter feasibility margins and move the search farther inside the feasible region\. As this margin increases, the generated interactions move farther away from the physical limit and are more likely to fall within the AV policy capability region, making collision\-inducing cases progressively sparser\. Thus,ε\\varepsilonis not merely a training coefficient, but the control variable that determines how ScenePilot sweeps the feasible\-yet\-adversarial boundary band\.

We empirically observe that the density of collision\-inducing cases is highly non\-uniform across different feasibility\-threshold ranges\. Table[6](https://arxiv.org/html/2605.21168#A1.T6)reports the collision rate observed under differentε\\varepsilonintervals during training for some generated scenarios\. The trend shows that, asε\\varepsilonincreases, collision\-inducing cases become progressively rarer\. This observation motivates both the use of non\-uniform threshold spacing and the choice ofεmax=0\.35\\varepsilon\_\{\\max\}=0\.35, since larger feasibility margins yield almost no informative failure cases\.

Table 6:Collision rate under different feasibility\-threshold intervals during training\. Largerε\\varepsilonvalues impose stricter feasibility margins, making collision\-inducing cases progressively sparser\.Based on this non\-uniform boundary structure, we adopt a Gaussian\-shaped schedule rather than uniformly spaced thresholds\. The schedule allocates denser constraint levels near the physical feasibility boundary and coarser levels farther inside the feasible region\. This design enables more fine\-grained probing of the near\-boundary region, where small changes inε\\varepsiloncan substantially affect whether a rollout remains physically feasible and still collision\-inducing\.

Concretely, we predefineNNconstraint levels by sampling evenly spaced points on a truncated standard\-normal axis and mapping them to\[0,εmax\]\[0,\\varepsilon\_\{\\max\}\]through the standard\-normal cumulative distribution function:

εi=εmax⋅F𝒩​\(ui\)−F𝒩​\(umin\)F𝒩​\(umax\)−F𝒩​\(umin\),ui=umin\+i−1N−1​\(umax−umin\),i=1,…,N,\\varepsilon\_\{i\}=\\varepsilon\_\{\\max\}\\cdot\\frac\{F\_\{\\mathcal\{N\}\}\(u\_\{i\}\)\-F\_\{\\mathcal\{N\}\}\(u\_\{\\min\}\)\}\{F\_\{\\mathcal\{N\}\}\(u\_\{\\max\}\)\-F\_\{\\mathcal\{N\}\}\(u\_\{\\min\}\)\},\\quad u\_\{i\}=u\_\{\\min\}\+\\frac\{i\-1\}\{N\-1\}\(u\_\{\\max\}\-u\_\{\\min\}\),\\quad i=1,\\ldots,N,\(33\)whereF𝒩​\(⋅\)F\_\{\\mathcal\{N\}\}\(\\cdot\)denotes the cumulative distribution function of the standard normal distribution\. This produces a sequence in which smallε\\varepsilonvalues are explored with finer granularity, while largerε\\varepsilonvalues are explored more sparsely\.

During training, we switch the active threshold every 100 episodes, cycling through\{εi\}i=1N\\\{\\varepsilon\_\{i\}\\\}\_\{i=1\}^\{N\}until reachingεmax=0\.35\\varepsilon\_\{\\max\}=0\.35\. Each time a threshold level is revisited, optimization resumes from the checkpoint previously saved for that same thresholdεi\\varepsilon\_\{i\}\. After reachingεmax\\varepsilon\_\{\\max\}, we restart the sweep from smallε\\varepsilonand continue this threshold\-specific optimization process\. This cyclic sweep stabilizes training while repeatedly revisiting the near\-boundary feasible region under different feasibility\-margin requirements\.

## Appendix BExperiment Details

We adopt the publicly released AV controllers provided by ChatScene\(Zhanget al\.,[2024](https://arxiv.org/html/2605.21168#bib.bib39)\)as our surrogate AVs\. Concretely, we use SAC as the default AV during training, and evaluate on a diverse set of AV planners including SAC and PPO \(with two SAC checkpoints of different training configurations\) as well as TD\-based baselines\. All AV checkpoints are directly taken from the official ChatScene releasing and are used as\-is without any additional fine\-tuning or modification\.

### B\.1Physical Feasibility Parameters

We compute the physical\-feasibility signalσ\\sigmaonline at every timestep in CARLA\. Vehicle geometry is obtained from the CARLA actor bounding box at runtime, and all checks follow the simulator step sizeΔ​t\\Delta t\. Table[7](https://arxiv.org/html/2605.21168#A2.T7)summarizes the constants used in Eq\.[3](https://arxiv.org/html/2605.21168#S3.E3)\. We aggregate axis\-wise residual margins intoσ\\sigmausing app\-norm withp=2p=2\.

Table 7:Physical\-feasibility parameters used for computingσ\\sigma\.
### B\.2AV Risk Network Training

#### Data Source\.

To learn a stable per\-frame AV risk predictor, it is important to expose the critic to sufficient safety\-critical interactions, since collisions are sparse under normal traffic\. Within the SafeBench benchmark, we train a PPO\-based scenario policy to actively generate hazardous encounters, and simultaneously update the AV risk network online using the same interaction stream\. Specifically, the PPO scenario policy is optimized with a composite reward consisting of \(i\) our physical\-safety signal \(Sec\.[A\.1](https://arxiv.org/html/2605.21168#A1.SS1)\) and \(ii\) a TTC\-based adversarial reward, which together encourage near\-miss and boundary interactions while maintaining meaningful traffic dynamics\. In practice, each scenario is run for 50k environment steps; we train across 8 scenarios, resulting in 400k interaction steps that jointly drive scenario learning and AV\-risk learning\.

#### Training details\.

We train a lightweight risk critic that maps a compact 6\-D interaction feature to a scalar risk value in\[0,1\]\[0,1\]\. To stabilize learning under sparse collisions, we introduce a distance\-based potentialF​\(s\)=k⋅inv​\_​d​\(s\)F\(s\)=k\\cdot\\mathrm\{inv\\\_d\}\(s\)\. The critic is optimized with TD\(0\) bootstrapping using a slowly\-updated target network:yt=clip​\(rt\+\(1−𝖽𝗈𝗇𝖾t\)​γ​Φ^tgt​\(st\+1\),0,1\)y\_\{t\}=\\mathrm\{clip\}\\\!\\left\(r\_\{t\}\+\(1\-\\mathsf\{done\}\_\{t\}\)\\gamma\\hat\{\\Phi\}\_\{\\mathrm\{tgt\}\}\(s\_\{t\+1\}\),\\,0,\\,1\\right\), wherertr\_\{t\}is a dense signal composed of a collision indicator and a potential\-based shaping term\. We minimize a weighted binary cross\-entropy with clipped soft targets in\[0,1\]\[0,1\]to address the severe imbalance between risky and non\-risky frames, up\-weighting high\-risk targets \(yt\>0\.85y\_\{t\}\>0\.85\) byw\+w\_\{\+\}\. All hyperparameters and architectural choices are reported in Table[8](https://arxiv.org/html/2605.21168#A2.T8)\.

Table 8:Hyperparameters for AV risk network training\.

### B\.3ScenePilot Policy Training

#### Training details\.

We train the ScenePilot scenario policy on the SafeBench benchmark \(CARLA\) using an on\-policy policy\-gradient optimizer, while keeping the surrogate AV controller fixed to the publicly released checkpoints from ChatScene \(without any additional fine\-tuning\)\. The trainable scenario policy is a small MLP\-based actor\-critic with only about 55K trainable parameters, so ScenePilot does not rely on large\-scale neural optimization\. All experiments were run on a server with 2× RTX 4090 GPUs; under this setup, we typically train the 10 routes of one base scenario in parallel, and the wall\-clock training time for a single base scenario is about 15 hours\. Unless otherwise stated, we set the discount factorγ=0\.99\\gamma=0\.99, GAEλ=0\.95\\lambda=0\.95, PPO clipping parameterϵclip=0\.25\\epsilon\_\{\\text\{clip\}\}=0\.25, Adam learning rate1×10−41\\times 10^\{\-4\}, batch size20482048, one update epoch per iteration, and entropy coefficient0\.030\.03\. The feasibility threshold follows the Gaussian\-shaped schedule in Appendix[A\.3](https://arxiv.org/html/2605.21168#A1.SS3), withN=8N=8predefined constraint levels\. Figures[5](https://arxiv.org/html/2605.21168#A2.F5)and[6](https://arxiv.org/html/2605.21168#A2.F6)report the aggregated training curves over all route\-wise runs, illustrating the typical periodic dynamics induced by the cyclic constraint sweep\. All hyperparameters are summarized in Table[9](https://arxiv.org/html/2605.21168#A2.T9)\.

![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/appendix/scenepilot_value_loss.png)Figure 5:Aggregated value loss during ScenePilot training\.![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/appendix/scenepilot_policy_loss.png)Figure 6:Aggregated policy loss during ScenePilot training\.Table 9:Hyperparameters for ScenePilot scenario\-policy training\.

### B\.4Scenario Sampling

Each base scenario contains 10 predefined routes in Safebench\. Using the fixed SAC ego policy released by ChatScene\(Zhanget al\.,[2024](https://arxiv.org/html/2605.21168#bib.bib39)\)as the surrogate controller, we train ScenePilot independently on each route for 3000 episodes under the above cyclicε\\varepsilonsweep to obtain route\-specialized adversarial behaviors\. For each \(base scenario, route\) pair, we keep the top\-10 most safety\-critical rollouts, resulting in 100 episodes per base scenario \(10 routes\)\. To avoid dependence on the surrogate, we report final results by replaying these episodes under three different ego controllers \(PPO, SAC, and TD3\) released by ChatScene and averaging over controllers and routes\.

### B\.5AV Policy Fine\-tuning

We follow the adversarial fine\-tuning procedure of ChatScene to adapt the surrogate AV policy under generated adversarial scenarios\. Concretely, we start from the publicly released SAC\-trained surrogate AV checkpoint from Chatscene and fine\-tune it in the same simulator setting without modifying the architecture\. We run fine\-tuning for 500 epochs, where one epoch corresponds to one full simulation rollout in one scene\. Both the policy learning rate and the Q\-value learning rate are set to1×10−41\\times 10^\{\-4\}\. To mitigate the extreme imbalance between collision and non\-collision experiences during adversarial interaction, we maintain two replay buffers: a non\-collision buffer of size20,00020\{,\}000and a collision buffer of size200200\. We use a batch size of 512; for each gradient step, 80% of samples are drawn from the non\-collision buffer and 20% from the collision buffer\. If the collision buffer contains fewer than0\.2×5120\.2\\times 512samples, we include all available collision samples and fill the remainder with non\-collision samples\. The entropy regularization coefficient is fixed to0\.010\.01, the discount factor is set toγ=0\.99\\gamma=0\.99, and the Q\-ensemble critic uses two Q\-networks, consistent with ChatScene\.

To address the failure mode that overly long fine\-tuning may induce a degenerate stopping strategy, we adopt the same checkpoint\-selection protocol as ChatScene\. Specifically, to obtain a faithful estimate of adversarial robustness, we evaluate all fine\-tuned checkpoints after 100 epochs with a stride of 50 epochs, and report the best checkpoint\. The selected checkpoint is defined as the one achieving the lowest collision rate while maintaining a reasonable route completion rate \(\>0\.3\>0\.3\), since adversarial events in our benchmark typically occur after the first 30% of route progress\.

## Appendix CResult Details

#### Detailed metrics\.

We adopt the evaluation metrics from Chatscene\(Zhanget al\.,[2024](https://arxiv.org/html/2605.21168#bib.bib39)\)and briefly summarize the intuition of each metric: CR \(collision rate\) evaluates the rate of collisions, reflecting accident\-avoidance capability; RR \(red\-light running frequency\) measures how often the ego vehicle runs red lights; SS \(stop\-sign violation frequency\) measures failures to stop at stop signs; OR \(out\-of\-road distance\) quantifies the average distance driven out of road, indicating lane/road discipline; RF \(route\-following stability\) examines how stably the ego vehicle follows its planned route; Comp \(route completion\) reports the percentage of the route completed; TS \(time spent\) evaluates the time efficiency to finish a route; ACC \(average acceleration\) measures driving smoothness; YV \(average yaw velocity\) reflects turning/handling intensity; and LI \(lane invasion frequency\) measures lane\-keeping accuracy\. The overall score \(OS\) is an aggregated metric combining the above factors, and we use the same weighting scheme as ChatScene\.

In addition, we reportGap Coverage Score \(GCS\), which quantifies how broadly the generated interactions populate the*AV\-infeasible yet physically\-feasible*gap band in the\(r,σ\)\(r,\\sigma\)space\. Given frame\-level pairs𝒮=\{\(rt,σt\)\}t=1T\\mathcal\{S\}=\\\{\(r\_\{t\},\\sigma\_\{t\}\)\\\}\_\{t=1\}^\{T\}collected along rollouts of the generated critical window, we define the*gap frames*as those satisfying physical feasibility while coming from failure episodes:

𝒮gap=\{\(rt,σt\)∈𝒮∣σt\>0∧collision\_rate​\(episode​\(t\)\)\>0\}\.\\mathcal\{S\}\_\{\\text\{gap\}\}=\\\{\(r\_\{t\},\\sigma\_\{t\}\)\\in\\mathcal\{S\}\\mid\\sigma\_\{t\}\>0\\ \\wedge\\ \\texttt\{collision\\\_rate\}\(\\text\{episode\}\(t\)\)\>0\\\}\.\(34\)We discretize\[0,1\]×\[0,1\]\[0,1\]\\times\[0,1\]in the\(r,σ\)\(r,\\sigma\)plane into a fixedK×KK\\times Kgrid \(withK=40K=40by default\), and denote the set of grid cells as𝒞K\\mathcal\{C\}\_\{K\}\. A cell is marked as occupied if it contains at least one point from𝒮gap\\mathcal\{S\}\_\{\\text\{gap\}\}, i\.e\.,

𝒪​\(𝒮gap\)=\{c∈𝒞K\|∃\(r,σ\)∈𝒮gap​s\.t\.​\(r,σ\)∈c\}\.\\mathcal\{O\}\(\\mathcal\{S\}\_\{\\text\{gap\}\}\)=\\left\\\{c\\in\\mathcal\{C\}\_\{K\}\\ \\middle\|\\ \\exists\(r,\\sigma\)\\in\\mathcal\{S\}\_\{\\text\{gap\}\}\\ \\text\{s\.t\.\}\\ \(r,\\sigma\)\\in c\\right\\\}\.\(35\)Then the Gap Coverage Score is defined as

GCS=\|𝒪​\(𝒮gap\)\|\|𝒞K\|=\|𝒪​\(𝒮gap\)\|K2\.\\mathrm\{GCS\}=\\frac\{\\left\|\\mathcal\{O\}\(\\mathcal\{S\}\_\{\\text\{gap\}\}\)\\right\|\}\{\\left\|\\mathcal\{C\}\_\{K\}\\right\|\}=\\frac\{\\left\|\\mathcal\{O\}\(\\mathcal\{S\}\_\{\\text\{gap\}\}\)\\right\|\}\{K^\{2\}\}\.\(36\)Larger GCS indicates that the generator covers a wider portion of the feasible gap, i\.e\., it discovers more diverse physically feasible yet AV\-challenging interaction regimes, rather than collapsing to overly mild or overly extreme behaviors\.

#### Results\.

We provide a detailed assessment of adversarial performance under multiple surrogate AV controllers\. Following the main protocol, we evaluate each scene set using three ego\-vehicle policies trained with different RL algorithms, namely SAC, PPO, and TD3, to test whether the generated adversarial scenes generalize across diverse AV training recipes\. Concretely, we report thecollision rate\(CR\) and theoverall score\(OS\) for each test ego vehicle under the same base traffic scenarios\. Detailed statistics of CR are summarized in Table[10](https://arxiv.org/html/2605.21168#A3.T10), while the OS results are reported in Table[11](https://arxiv.org/html/2605.21168#A3.T11)\. The evaluation protocol details \(including metrics, scoring rules\) follow the official evaluation settings of ChatScene\.

Table[12](https://arxiv.org/html/2605.21168#A3.T12)provides a multi\-level diagnostic by decomposing the SafeBench overall score \(OS\) into safety\-, functionality\-, and etiquette\-related indicators\. ScenePilot achieves the highest collision rate \(CR\) and the best OS, while keeping several non\-collision proxies comparatively low\. This is expected because our scenario generation operates on fixed SafeBench base routes and does not explicitly optimize traffic\-rule indicators such as red\-light or stop\-sign violations \(RR/SS\); instead, it targets safety\-criticality through adversarial interactions between the ego and non\-ego agents\. Even under this constraint, ScenePilot still establishes a new overall best result under the standard SafeBench evaluation, indicating that physics\-guided boundary shaping combined with AV\-risk\-aware optimization yields more transferable and genuinely safety\-critical scenarios\. Consequently, these results suggest that ScenePilot’s increased safety\-criticality primarily arises from interaction\-driven, near\-boundary failure cases—i\.e\., scenarios that expose control\-limit and collision\-prone interactions—rather than from trivial rule\-breaking or unstable driving behaviors that can artificially inflate risk\.

Overall, the results indicate that our method yields more transferable adversarial scenes and achieves consistently stronger average performance across different ego\-vehicle training algorithms\. We further provide visualizations of the generated adversarial interactions\. These examples illustrate representative near\-boundary behaviors and failure cases across different base scenarios\.

Table 10:Collision Rate \(CR\) Performance Across Different Models\.We report the mean CR for each model across base traffic scenarios\. Higher values of CR \(↑\\uparrow\) indicate stronger adversarial effectiveness\. Bold denotes the best performance for each scenario\.Table 11:Overall Score \(OS\) Performance Across Different Models\.We report the mean OS for each model across base traffic scenarios\. Lower values of OS \(↓\\downarrow\) are preferable\. Bold denotes the best performance for each scenario\.Table 12:Multi\-level analysis of gernerated scenario based on ego\-vehicle behavior\.This table summarizes the average performance of different scenario generation methods, evaluated with three ego controllers over eight base scenarios\. For each method, we report indicators at three levels, providing a comprehensive diagnostic of ego behavior\. Safety metrics include CR \(collision rate\), RR \(rate of running red lights\), SS \(rate of running stop signs\), and OR \(average distance traveled off road\)\. Functionality metrics cover RF \(route\-following stability\), Comp \(fraction of route completed\), and TS \(time to complete the route\)\. Etiquette metrics comprise ACC \(average acceleration\), YV \(average yaw velocity\), and LI \(lane\-invasion frequency\)\. OS denotes the overall score aggregated by SafeBench\.![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/carla/1-1.png)
![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/carla/1-2.png)
![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/carla/1-3.png)

Straight Obstacle![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/carla/2-1.png)
![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/carla/2-2.png)
![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/carla/2-3.png)

Turning Obstacle![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/carla/3-1.png)
![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/carla/3-2.png)
![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/carla/3-3.png)

Lane Changing![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/carla/4-1.png)
![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/carla/4-2.png)
![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/carla/4-3.png)

Vehicle Passing![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/carla/5-1.png)
![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/carla/5-2.png)
![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/carla/5-3.png)

Red\-light Running![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/carla/6-1.png)
![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/carla/6-2.png)
![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/carla/6-3.png)

Unprotected Left\-turn![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/carla/7-1.png)
![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/carla/7-2.png)
![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/carla/7-3.png)

Right\-turn![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/carla/8-1.png)
![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/carla/8-2.png)
![Refer to caption](https://arxiv.org/html/2605.21168v1/figure/carla/8-3.png)

Crossing Negotiation