SHRED: Self-Play for Heterogeneous Road-Users without Expert Demonstrations

# Self-Play for HeterogeneousRoad-Users without Expert Demonstrations
Source: [https://arxiv.org/html/2610.20866](https://arxiv.org/html/2610.20866)
## SHRED: Self\-Play for Heterogeneous Road\-Users without Expert Demonstrations

,Zhouchonghao WuAffiliation:Applied Intuition,Mountain View,CA,USA,Akshay RangeshAffiliation:Applied Intuition,Mountain View,CA,USA,Weixin LiAffiliation:Applied Intuition,Mountain View,CA,USA,Masayoshi TomizukaAffiliation:University of California, Berkeley,Berkeley,CA,USAandWei ZhanAffiliation:Applied Intuition,Mountain View,CA,USA

2026

###### Abstract\.

Realistic traffic simulation and autonomous planning both depend on modeling interactions among heterogeneous road users whose dynamics, constraints, and behaviors differ fundamentally\. Existing approaches predominantly learn realistic interaction behavior from large\-scale human driving demonstrations\. In this work, we ask whether such behavior can instead emerge purely through self\-play\. We present SHRED, a framework that jointly trains vehicles, pedestrians, and cyclists through large\-scale heterogeneous self\-play from random initialization, without demonstrations or logged\-trajectory replay\. SHRED combines type\-specific dynamics and objectives with procedural scenario generation and per\-agent randomization, allowing diverse behaviors to emerge through closed\-loop interaction\. On the Waymo Open Sim Agents Challenge \(WOSAC\), SHRED jointly controls all three road\-user types and outperforms both the closest demonstration\-free self\-play baseline and prior self\-play methods that use demonstration\-trained reference policies\. Policies trained only on nuPlan map geometry retain comparable realism under zero\-shot evaluation on Waymo\. For planning, SHRED outperforms all compared planners on the interaction\-heavy InterPlan benchmark\. Controlled ablations show that heterogeneity in self\-play improves both traffic realism and planning performance, with particularly large planning gains in interaction\-heavy scenarios\. Together, these results establish heterogeneous procedural self\-play as a scalable, demonstration\-free paradigm for learning realistic and interactive traffic behavior\.

###### Contents

1. [1Introduction](https://arxiv.org/html/2610.20866#S1)
2. [2Related Work](https://arxiv.org/html/2610.20866#S2)1. [2\.1Data\-Driven and Heterogeneous Traffic Simulation](https://arxiv.org/html/2610.20866#S2.SS1) 2. [2\.2Self\-Play Reinforcement Learning for Driving](https://arxiv.org/html/2610.20866#S2.SS2) 3. [2\.3Heterogeneity in Multi\-Agent Reinforcement Learning](https://arxiv.org/html/2610.20866#S2.SS3)
3. [3Method](https://arxiv.org/html/2610.20866#S3)1. [3\.1Heterogeneous Agent Modeling](https://arxiv.org/html/2610.20866#S3.SS1) 2. [3\.2Procedural Heterogeneous Self\-Play](https://arxiv.org/html/2610.20866#S3.SS2) 3. [3\.3Architecture and Algorithm Design Choices](https://arxiv.org/html/2610.20866#S3.SS3)
4. [4Experiments](https://arxiv.org/html/2610.20866#S4)1. [4\.1Heterogeneous Traffic Simulation](https://arxiv.org/html/2610.20866#S4.SS1) 2. [4\.2Interactive Ego Planning](https://arxiv.org/html/2610.20866#S4.SS2) 3. [4\.3Behavioral Analysis](https://arxiv.org/html/2610.20866#S4.SS3)
5. [5Conclusion](https://arxiv.org/html/2610.20866#S5)
6. [References](https://arxiv.org/html/2610.20866#bib)
7. [APer\-Type Dynamics](https://arxiv.org/html/2610.20866#A1)1. [A\.1Vehicle: Jerk\-Integrated Ackermann Arc](https://arxiv.org/html/2610.20866#A1.SS1) 2. [A\.2Pedestrian: Unicycle](https://arxiv.org/html/2610.20866#A1.SS2) 3. [A\.3Cyclist: Kinematic Bicycle](https://arxiv.org/html/2610.20866#A1.SS3) 4. [A\.4Domain Randomization Parameters](https://arxiv.org/html/2610.20866#A1.SS4)
8. [BRandom Initialization Details](https://arxiv.org/html/2610.20866#A2)
9. [CPer\-Type Reward Details](https://arxiv.org/html/2610.20866#A3)
10. [DPopArt Weight Correction](https://arxiv.org/html/2610.20866#A4)
11. [EWOSAC Evaluation Protocol and Additional Results](https://arxiv.org/html/2610.20866#A5)1. [E\.1WOSAC 2024 Validation](https://arxiv.org/html/2610.20866#A5.SS1) 2. [E\.2WOSAC 2025 Test Evaluation](https://arxiv.org/html/2610.20866#A5.SS2) 3. [E\.3Full InterPlan Comparison](https://arxiv.org/html/2610.20866#A5.SS3)

## 1\.Introduction

Developing autonomous vehicles that operate safely alongside human road users requires scalable and realistic closed\-loop traffic simulation\. Real\-world traffic, however, is inherently heterogeneous: beyond vehicles, it includes pedestrians, cyclists, and other agents with distinct physical dynamics, motion constraints, and behavioral patterns\. These differences shape how agents respond to one another and how their interactions unfold over time\. A pedestrian or scooter yielding at a crosswalk and a vehicle merging into traffic follow fundamentally different rules of engagement\. Capturing such heterogeneous interactions is therefore central to realistic closed\-loop simulation\. It is equally central to autonomous planning, where the ego vehicle must anticipate how diverse agents will react to its own actions\.

Recent advances in traffic simulation and autonomous planning have widely adopted data\-driven learning, leveraging large\-scale human driving datasets to model traffic behavior and train driving policies\. Tokenized and generative simulation models reproduce realistic traffic behaviors and achieve strong performance on large\-scale simulation benchmarks\([Wu et al\., 2024](https://arxiv.org/html/2610.20866#bib.bib4);[Philion et al\., 2024](https://arxiv.org/html/2610.20866#bib.bib21);[Zhou et al\., 2024](https://arxiv.org/html/2610.20866#bib.bib22)\)\. However, these approaches often rely on computationally expensive centralized models, limiting throughput in large\-scale closed\-loop simulation\([Chang et al\., 2026](https://arxiv.org/html/2610.20866#bib.bib3)\)\. In parallel, learning\-based planning methods have increasingly leveraged large\-scale driving datasets to learn scene representations, model interactions, and optimize driving decisions, achieving strong performance across challenging driving scenarios\([Huang et al\., 2023](https://arxiv.org/html/2610.20866#bib.bib64);[Huang et al\., 2024](https://arxiv.org/html/2610.20866#bib.bib59);[Zheng et al\., 2025](https://arxiv.org/html/2610.20866#bib.bib65);[Tan et al\., 2025](https://arxiv.org/html/2610.20866#bib.bib66)\)\. Despite their different objectives, both lines of work remain widely dependent on logged human driving data for learning realistic and effective interaction behaviors\. Their behavior remains tied to the interaction distributions in logged datasets, which can make coverage of diverse heterogeneous interactions and cross\-dataset transfer more challenging\. Large\-scale demonstration datasets are also expensive to collect and may not fully capture the diversity of reactive multi\-agent behaviors observed in real\-world traffic\.

Self\-play reinforcement learning offers a complementary direction: it can learn through online interaction without trajectory supervision, scales naturally to large numbers of agents, and enables diverse multi\-agent interactions to emerge through closed\-loop rollouts\. Recent work has shown that self\-play can produce robust driving policies at unprecedented scale\([Cusumano\-Towner et al\., 2025](https://arxiv.org/html/2610.20866#bib.bib2)\)and achieve competitive realism when anchored to pretrained reference models\([Chang et al\., 2026](https://arxiv.org/html/2610.20866#bib.bib3)\)\. However, prior self\-play systems focus on vehicle policies or homogeneous dynamics and rewards across road\-user types despite fundamentally different motion characteristics\. Scalable, fully reactive heterogeneous self\-play across vehicles, pedestrians, and cyclists without relying on logged trajectory data or pretrained imitation\-policy anchors therefore remains largely unexplored\.

To address these challenges, we propose SHRED \(Self\-Play forHeterogeneousRoad\-Users withoutExpertDemonstrations\), a scalable framework for learning heterogeneous traffic behavior through large\-scale self\-play\. SHRED jointly trains vehicles, pedestrians, and cyclists from random initialization, with type\-specific dynamics, objectives, and action spaces that reflect their distinct modes of interaction\. Procedural scenario generation and per\-agent randomization of dynamics and reward parameters expose agents to diverse traffic configurations and behaviors throughout training\. This enables a compact decentralized policy to learn realistic heterogeneous multi\-agent behaviors entirely through self\-play, without demonstration supervision\.

We show that demonstration\-free self\-play can achieve both realistic heterogeneous traffic simulation and strong interactive planning\. For simulation, we evaluate on the Waymo Open Sim Agents Challenge \(WOSAC\)\([Montali et al\., 2023](https://arxiv.org/html/2610.20866#bib.bib1)\)while jointly controlling vehicles, pedestrians, and cyclists\. SHRED outperforms the closest demonstration\-free self\-play baseline, while controlled ablations show that heterogeneous self\-play substantially improves traffic realism, including the behavior of vehicles themselves\. For planning, we evaluate SHRED on InterPlan\([Hallgarten et al\., 2024](https://arxiv.org/html/2610.20866#bib.bib19)\), where it outperforms all compared planners on interaction\-heavy closed\-loop scenarios\. Complementary planning ablations show that heterogeneous self\-play substantially outperforms vehicle\-only self\-play, with the largest gains on scenarios involving concentrated interactions with other road users\. Beyond these benchmarks, policies trained on nuPlan maps transfer zero\-shot to Waymo, while behavioral analyses show that self\-play produces diverse, plausible alternatives to logged trajectories \(Figure[1](https://arxiv.org/html/2610.20866#S1.F1)\)\.

Our main contributions are as follows:

- •We introduce SHRED, a demonstration\-free heterogeneous self\-play framework that jointly learns vehicle, pedestrian, and cyclist behaviors with distinct dynamics and objectives, entirely without demonstrations or logged\-trajectory replay\.
- •We develop the key components for scalable heterogeneous self\-play, including procedural scenario and goal generation, per\-agent randomization of dynamics and objectives, and type\-specific policy components that support joint learning across diverse road\-user types\.
- •We show that demonstration\-free self\-play is effective across both simulation and planning: SHRED achieves strong sim\-agent realism on WOSAC, the highest score among compared planners on the official 80\-scenario InterPlan split, and zero\-shot map transfer from nuPlan to Waymo\.

![Refer to caption](https://arxiv.org/html/2610.20866v1/fig/qualitative_viz.png)Figure 1\.Qualitative Rollout Distributions\.Qualitative trajectory distributions from SHRED across a diverse range of scenarios, from urban intersections to parking lots\. Each panel shows 32 closed\-loop policy rollouts \(blue: vehicle, red: pedestrian, green: cyclist\); orange traces show the ground\-truth log trajectory\. The panels show alternative maneuvers alongside the logged trajectory\.
## 2\.Related Work

### 2\.1\.Data\-Driven and Heterogeneous Traffic Simulation

Imitation from human driving logs has become the dominant paradigm for realistic traffic simulation\. Tokenized autoregressive models\([Wu et al\., 2024](https://arxiv.org/html/2610.20866#bib.bib4);[Philion et al\., 2024](https://arxiv.org/html/2610.20866#bib.bib21);[Zhou et al\., 2024](https://arxiv.org/html/2610.20866#bib.bib22);[Zhang et al\., 2025](https://arxiv.org/html/2610.20866#bib.bib5)\)and diffusion\-based approaches\([Jiang et al\., 2024](https://arxiv.org/html/2610.20866#bib.bib9)\)have driven strong performance on WOSAC\([Montali et al\., 2023](https://arxiv.org/html/2610.20866#bib.bib1)\), while closed\-loop fine\-tuning, promptable rollout, and RL/RLHF post\-training further improve rollout realism and controllability\([Zhang et al\., 2025](https://arxiv.org/html/2610.20866#bib.bib5);[Tan et al\., 2024](https://arxiv.org/html/2610.20866#bib.bib23);[Peng et al\., 2024](https://arxiv.org/html/2610.20866#bib.bib11);[Chen et al\., 2025](https://arxiv.org/html/2610.20866#bib.bib12);[Cao et al\., 2024](https://arxiv.org/html/2610.20866#bib.bib13);[Pei et al\., 2026](https://arxiv.org/html/2610.20866#bib.bib24)\)\. RLFTSim\([Ahmadi et al\., 2026](https://arxiv.org/html/2610.20866#bib.bib54)\)fine\-tunes a pretrained traffic simulator with a distribution\-alignment reward\. DecompGAIL\([Guo et al\., 2026](https://arxiv.org/html/2610.20866#bib.bib55)\)combines a SMART\-based model with decomposed adversarial imitation learning\. Both methods use a trajectory\-model prior; SHRED instead studies heterogeneous recurrent self\-play without one\. Pedestrian and cyclist behavior has additionally been studied through rule\-based crowd dynamics\([Helbing and Molnár, 1995](https://arxiv.org/html/2610.20866#bib.bib36);[Helbing et al\., 2005](https://arxiv.org/html/2610.20866#bib.bib38);[van den Berg et al\., 2011](https://arxiv.org/html/2610.20866#bib.bib39)\), multimodal trajectory forecasting models\([Alahi et al\., 2016](https://arxiv.org/html/2610.20866#bib.bib40);[Gupta et al\., 2018](https://arxiv.org/html/2610.20866#bib.bib41);[Salzmann et al\., 2020](https://arxiv.org/html/2610.20866#bib.bib42);[Xu et al\., 2022](https://arxiv.org/html/2610.20866#bib.bib43);[Rasouli and Kotseruba, 2023](https://arxiv.org/html/2610.20866#bib.bib44)\), and multi\-agent driving simulators\([Suo et al\., 2021](https://arxiv.org/html/2610.20866#bib.bib45);[Vinitsky et al\., 2022](https://arxiv.org/html/2610.20866#bib.bib46);[Gulino et al\., 2023](https://arxiv.org/html/2610.20866#bib.bib47);[Li et al\., 2022](https://arxiv.org/html/2610.20866#bib.bib48);[Kazemkhani et al\., 2025](https://arxiv.org/html/2610.20866#bib.bib8);[Rowe et al\., 2024](https://arxiv.org/html/2610.20866#bib.bib49)\)\. However, realistic heterogeneous interaction behavior is typically learned implicitly from demonstrations, while learned closed\-loop control remains largely vehicle\-centric or relies on shared dynamics across all road users\. Our work instead studies whether realistic heterogeneous traffic can emerge from demonstration\-free self\-play with type\-specific dynamics, rewards, goals, and policy components for vehicles, pedestrians, and cyclists jointly\.

### 2\.2\.Self\-Play Reinforcement Learning for Driving

Self\-play has driven landmark results in game playing\([Silver et al\., 2017](https://arxiv.org/html/2610.20866#bib.bib14);[Vinyals et al\., 2019](https://arxiv.org/html/2610.20866#bib.bib15);[Berner et al\., 2019](https://arxiv.org/html/2610.20866#bib.bib16);[Jaderberg et al\., 2019](https://arxiv.org/html/2610.20866#bib.bib17)\)and has recently been adopted for traffic simulation\. Gigaflow\([Cusumano\-Towner et al\., 2025](https://arxiv.org/html/2610.20866#bib.bib2)\)is the closest demonstration\-free precedent, training a unified policy for vehicles, pedestrians, and cyclists through massively parallel self\-play\. However, all agents share the same jerk\-actuated bicycle dynamics and goal\-generation mechanism, with diversity induced primarily through conditioning variables inside a shared backbone\. Other self\-play systems explore complementary directions:[Cornelisse et al\. \(2025\)](https://arxiv.org/html/2610.20866#bib.bib6)scale vehicle\-only PPO from logged WOMD initial states, while HR\-PPO\([Cornelisse and Vinitsky, 2024](https://arxiv.org/html/2610.20866#bib.bib7)\)and SPACeR\([Chang et al\., 2026](https://arxiv.org/html/2610.20866#bib.bib3)\)regularize self\-play with behavior\-cloned or tokenized policy anchors\. NOMAD\([Wang et al\., 2026](https://arxiv.org/html/2610.20866#bib.bib50)\)studies map\-transfer through self\-play, and GPUDrive\([Kazemkhani et al\., 2025](https://arxiv.org/html/2610.20866#bib.bib8)\)provides a GPU\-resident substrate for large\-scale simulation\. TerraTransfer\([Xiong et al\., 2026](https://arxiv.org/html/2610.20866#bib.bib69)\)aligns a self\-play policy with a pretrained vision backbone to obtain end\-to\-end driving policies without expert demonstrations\. In contrast, we jointly train vehicles, pedestrians, and cyclists from fully procedural initialization, without trajectory supervision, pretrained imitation\-policy anchors, or logged\-trajectory replay, using type\-specific dynamics, rewards, and goal generation\.

### 2\.3\.Heterogeneity in Multi\-Agent Reinforcement Learning

Most heterogeneous multi\-agent RL methods are developed for cooperative settings with shared rewards, including QMIX\([Rashid et al\., 2018](https://arxiv.org/html/2610.20866#bib.bib26)\), MAPPO\([Yu et al\., 2022](https://arxiv.org/html/2610.20866#bib.bib25)\), and HAPPO/HATRPO\([Kuba et al\., 2022](https://arxiv.org/html/2610.20866#bib.bib27);[Zhong et al\., 2024](https://arxiv.org/html/2610.20866#bib.bib28)\)\. Several of them rely on sequential per\-agent updates \(HAPPO/HATRPO\) or separate per\-agent policies, which become costly in traffic scenes containing dozens of agents with different dynamics and action spaces\. Recent modular and role\-conditioned approaches such as SePS\([Christianos et al\., 2021](https://arxiv.org/html/2610.20866#bib.bib29)\), ROMA\([Wang et al\., 2020](https://arxiv.org/html/2610.20866#bib.bib18)\), RODE\([Wang et al\., 2021](https://arxiv.org/html/2610.20866#bib.bib30)\), and MoE\-based RL architectures\([Yang et al\., 2020](https://arxiv.org/html/2610.20866#bib.bib31);[Obando\-Ceron et al\., 2024](https://arxiv.org/html/2610.20866#bib.bib32)\)reduce interference through selective parameter sharing or latent routing, but remain primarily focused on cooperative domains\.

Our setting is instead mixed\-motive: each road user optimizes its own goal under type\-specific rewards and dynamics without a shared team objective\. We therefore organize policies by physical agent type rather than agent identity, sharing perception while maintaining separate recurrent, actor, and value heads for vehicles, pedestrians, and cyclists\.

## 3\.Method

### 3\.1\.Heterogeneous Agent Modeling

We consider heterogeneous traffic environments containing agent types𝒦=\{VEH,PED,CYC\}\\mathcal\{K\}=\\\{\\mathrm\{VEH\},\\mathrm\{PED\},\\mathrm\{CYC\}\\\}, corresponding to vehicles, pedestrians, and cyclists\. Each agent type follows its own dynamics model, reward structure, and discrete action space while interacting through a shared closed\-loop environment\. At each timestep, agentiiselects an action according to a decentralized policy

ati∼πθ​\(oti,gti,ki,Cdyni,αi\),a\_\{t\}^\{i\}\\sim\\pi\_\{\\theta\}\\\!\\left\(o\_\{t\}^\{i\},\\,g\_\{t\}^\{i\},\\,k^\{i\},\\,C\_\{\\mathrm\{dyn\}\}^\{i\},\\,\\alpha^\{i\}\\right\),whereotio\_\{t\}^\{i\}denotes the local observation,gtig\_\{t\}^\{i\}denotes goal and waypoint conditioning,kik^\{i\}is the agent type,CdyniC\_\{\\mathrm\{dyn\}\}^\{i\}denotes kinematic parameters, andαi\\alpha^\{i\}parameterizes the reward function\.

#### Per\-Type Dynamics and Actions\.

Each agent type follows its own continuous\-control dynamics model\. Vehicles use jerk\-integrated Ackermann dynamics, pedestrians follow unicycle dynamics, and cyclists use kinematic bicycle dynamics\. Actions are discretized separately for each type to reflect their distinct motion characteristics and control regimes\. Vehicles operate over longitudinal and lateral jerk primitives, pedestrians use velocity and turning controls with an explicit stop action, and cyclists follow acceleration and steering controls\. Appendix[A](https://arxiv.org/html/2610.20866#A1)gives the full dynamics and action grids\.

#### Rewards\.

*Vehicle*rewards inherit the Gigaflow\([Cusumano\-Towner et al\., 2025](https://arxiv.org/html/2610.20866#bib.bib2)\)formulation, while pedestrian and cyclist rewards introduce type\-specific behavioral priors for heterogeneous traffic interaction\.*Pedestrian*rewards remove lane\-following objectives and instead encourage sidewalk, crosswalk, and waiting behavior through soft road\-intrusion penalties and walking\-speed regularization, allowing curb waiting, yielding, and occasional jaywalking while discouraging unrealistic motion\.*Cyclist*rewards retain vehicle\-like navigation objectives while introducing soft lane\-edge preferences and relaxed lane\-alignment constraints, encouraging curb\-side riding without enforcing strict vehicle lane\-centering behavior\.

We also introduce asymmetric cross\-type collision shaping: a vehicle’s collision penalty doubles when the other party is a pedestrian\. Detailed reward formulations, thresholds, and parameter ranges are provided in Appendix[C](https://arxiv.org/html/2610.20866#A3)\.

![Refer to caption](https://arxiv.org/html/2610.20866v1/training_architecture.png)Figure 2\.SHRED Training Architecture\.Simulation produces mixed transitions across all agent types\. The model consists of shared encoders followed by per\-type LSTM cells, action heads, and value heads\. Per\-type rewards and value normalization accommodate distinct reward scales across road\-user types\. Each type uses its own dynamics model: jerk\-integrated Ackermann dynamics for vehicles, unicycle dynamics for pedestrians, and kinematic bicycle dynamics for cyclists\.

### 3\.2\.Procedural Heterogeneous Self\-Play

Recent self\-play works often initialize agents directly from logged datasets or replay recorded trajectories during training and evaluation\([Cornelisse et al\., 2025](https://arxiv.org/html/2610.20866#bib.bib6);[Chang et al\., 2026](https://arxiv.org/html/2610.20866#bib.bib3)\)\. While effective for reproducing realistic traffic scenes, such approaches remain constrained by the finite scene layouts and trajectory horizons present in recorded data\. To improve coverage and interaction diversity during heterogeneous self\-play, we procedurally generate closed\-loop training episodes through randomized initialization, goal generation, and agent parameterization\.

Formally, each training episode is sampled from a factorized procedural distribution

𝒟ep=𝒟scene​∏k∈𝒦∏i∈𝒩k𝒟init\(k\)​𝒟goal\(k\)​𝒟b\(k\)​𝒟Cdyn\(k\)​𝒟α\(k\)​𝒟mask,\\mathcal\{D\}\_\{\\mathrm\{ep\}\}=\\mathcal\{D\}\_\{\\mathrm\{scene\}\}\\prod\_\{k\\in\\mathcal\{K\}\}\\prod\_\{i\\in\\mathcal\{N\}\_\{k\}\}\\mathcal\{D\}^\{\(k\)\}\_\{\\mathrm\{init\}\}\\mathcal\{D\}^\{\(k\)\}\_\{\\mathrm\{goal\}\}\\mathcal\{D\}^\{\(k\)\}\_\{b\}\\mathcal\{D\}^\{\(k\)\}\_\{C\_\{\\mathrm\{dyn\}\}\}\\mathcal\{D\}^\{\(k\)\}\_\{\\alpha\}\\mathcal\{D\}\_\{\\mathrm\{mask\}\},where each factor is parameterized independently per agent type\. These factors jointly control the distribution of scene layouts, navigation objectives, agent behaviors, and goal observability encountered during self\-play training\.

#### Per\-Type Initialization \(𝒟init\(k\)\\mathcal\{D\}\_\{\\mathrm\{init\}\}^\{\(k\)\}\)\.

Controlled agents are initialized procedurally from type\-specific spawn surfaces\. Vehicles and cyclists are sampled from lane centerlines, while pedestrians are initialized from crosswalks and road\-edge regions\. Agents are added sequentially through rejection sampling to ensure collision\-free initial states, with randomized heading perturbations applied at spawn time to increase local interaction diversity under high environment densities\. Because initialization is performed independently for each agent type, the resulting scenes span diverse heterogeneous compositions\. These include dense crosswalk traffic, mixed cyclist–vehicle flows, and multi\-type urban intersections, which are rare in recorded datasets\.

#### Procedural Goal Generation\.

Rather than replaying logged destinations, goals are procedurally sampled from map topology conditioned on agent type\. Vehicles and cyclists follow lane\-graph connectivity, while pedestrian goals are generated along crosswalk and sidewalk topology\. Unlike logged\-goal initialization, where agents often terminate after reaching a single recorded destination, goals are continuously regenerated after completion during training\. This decouples interaction horizon from the finite duration of recorded scenarios, allowing agents to experience substantially longer and more diverse closed\-loop interactions within a single environment rollout\.

#### Per\-Agent Diversity\.

Behavioral diversity is induced through independent randomization of agent geometry, kinematic coefficients, and reward parameterizations\. Each agent observes its own\(bi,Cdyni,αi\)\(b^\{i\},C\_\{\\mathrm\{dyn\}\}^\{i\},\\alpha^\{i\}\)but not those of surrounding agents, forcing the policy to model partner behavior as arising from a heterogeneous population rather than a fixed shared dynamics model\.

#### Goal Dropout\.

To reduce over\-reliance on explicit navigation conditioning, a subset of agents have their goal observations masked during training\. These agents must instead act using local scene context and interaction history alone, while continuing to participate in fully reactive closed\-loop self\-play\. This encourages the policy to produce robust and realistic behaviors even when long\-horizon navigation intent is partially specified or absent\.

### 3\.3\.Architecture and Algorithm Design Choices

#### Policy Architecture\.

The architecture of SHRED is illustrated in Figure[2](https://arxiv.org/html/2610.20866#S3.F2)\. All agent types share encoders for ego state, road geometry, nearby agents, traffic signals, goals, and conditioning variables\. The shared encoders produce a scene representation for each agent\. Vehicles, pedestrians, and cyclists then use separate LSTM cells, actor heads, and value heads\.

#### Value Normalization and Advantage Computation\.

Jointly optimizing heterogeneous agent populations introduces substantial optimization imbalance: different agent types exhibit distinct reward scales, motion statistics, and advantage distributions during self\-play\. As a result, naively optimizing all agents under a shared PPO objective can cause dominant agent classes or low\-signal transitions to overwhelm the optimization process\.

To stabilize heterogeneous self\-play, we maintain independent PopArt normalization statistics\([van Hasselt et al\., 2016](https://arxiv.org/html/2610.20866#bib.bib33)\)together with per\-type advantage standardization before PPO updates\. Advantages are computed jointly over mixed heterogeneous rollouts using GAE with V\-trace corrections\([Espeholt et al\., 2018](https://arxiv.org/html/2610.20866#bib.bib51)\)\. We further perform advantage\-weighted prioritized replay, replacing hard advantage filtering used in prior large\-scale self\-play systems\([Cusumano\-Towner et al\., 2025](https://arxiv.org/html/2610.20866#bib.bib2)\), where rollout segments are sampled proportionally to the magnitude of their estimated advantages\. This increases the contribution of high\-signal transitions during optimization while preserving balanced learning across heterogeneous agent populations\.

## 4\.Experiments

We evaluate demonstration\-free heterogeneous self\-play across both simulation and planning, focusing on its role in learning realistic traffic behavior and effective interactive planning\.

### 4\.1\.Heterogeneous Traffic Simulation

#### Training and Evaluation Protocol\.

We train the recurrent heterogeneous policy in the TerraZero simulator\([Wu et al\., 2026a](https://arxiv.org/html/2610.20866#bib.bib68)\)with distributed mixed\-precision self\-play on 32 NVIDIA A100 GPUs\. The policy shares scene encoders across road users and uses separate LSTM, actor, and value components for vehicles, pedestrians, and cyclists\. Training uses map geometry, procedural initialization, and map\-conditioned goals without trajectory supervision or a pretrained imitation\-policy anchor\. We evaluate WOSAC 2023 and WOSAC 2024 validation with 32 stochastic closed\-loop rollouts per scenario\. The WOSAC 2023 evaluation follows the leaderboard setting: agents receive the logged history and the map but no goal\. Appendix[E](https://arxiv.org/html/2610.20866#A5)describes the evaluator verification and reports the WOSAC 2025 test evaluation separately\.

#### Demonstration\-Free Self\-Play Comparison\.

Table[1](https://arxiv.org/html/2610.20866#S4.T1)compares SHRED with Gigaflow, the closest demonstration\-free self\-play baseline, on WOSAC 2023\. Gigaflow applies its learned policy to vehicles and cyclists during evaluation while controlling pedestrians with a scripted IDM\-like policy\([Cusumano\-Towner et al\., 2025](https://arxiv.org/html/2610.20866#bib.bib2)\)\. In contrast, SHRED learns policies for all three road\-user types through self\-play and achieves higher realism than Gigaflow \(0\.631 vs\. 0\.619\) over 10,000 validation scenarios, with a particularly strong improvement on an interaction metric, time\-to\-collision likelihood \(0\.894 vs\. 0\.808\)\. Gigaflow achieves a higher distance\-to\-object likelihood\.

To examine whether the gain arises from heterogeneous self\-play rather than self\-play alone, we additionally evaluate a homogeneous SHRED variant with shared dynamics, rewards, and policy components across road\-user types, which is the joint homogeneous design of Table[2](https://arxiv.org/html/2610.20866#S4.T2)\. The homogeneous variant scores 0\.604, compared with 0\.631 for the full heterogeneous policy\. The largest degradations appear in motion realism, with drops in linear\-speed and angular\-acceleration likelihood\. Vehicles exhibit the largest per\-type realism drop \(0\.641 to 0\.614\), suggesting that the benefits of heterogeneous modeling extend across road\-user types\.

Table 1\.Comparison with demonstration\-free self\-play on WOSAC 2023\.SHRED outperforms Gigaflow in overall realism, with the largest gains in time\-to\-collision and linear speed\. Without type\-specific rewards and dynamics, SHRED falls below Gigaflow, showing that its advantage comes from heterogeneous modeling\. Neither method observes logged destinations and goals are sampled from the map, following the official WOSAC setting\. Gigaflow controls pedestrians with a scripted IDM\-like policy, whereas SHRED learns all three road\-user types\.MethodRealism↑\\uparrowLin\.Speed↑\\uparrowLin\.Accel\.↑\\uparrowAng\.Speed↑\\uparrowAng\.Accel\.↑\\uparrowDist\.Obj\.↑\\uparrowTTC↑\\uparrowDist\.Rd\. Edge↑\\uparrowOff\-road↑\\uparrowColl\.↑\\uparrowGigaflow\([Cusumano\-Towner et al\., 2025](https://arxiv.org/html/2610.20866#bib.bib2)\)0\.6190\.2610\.2530\.5060\.4760\.3160\.8080\.5450\.9100\.947SHRED0\.6310\.3530\.2470\.5010\.4570\.2500\.8940\.6070\.9040\.947SHRED \(homogeneousrewards and dynamics\)0\.6040\.2470\.2390\.4950\.3580\.2480\.8790\.5630\.8960\.947

#### Role of Heterogeneity\.

Table 2\.Ablation of heterogeneous dynamics and rewards on WOSAC 2024\.Overall and per\-type realism for vehicles, pedestrians, and cyclists\. Removing either component modestly reduces realism, while removing both causes the largest drop, most strongly for vehicles\. Removing random initialization and goals also degrades realism\.Het\. dyn\.Het\. rew\.Overall↑\\uparrowVeh\.↑\\uparrowPed\.↑\\uparrowCyc\.↑\\uparrow✓✓0\.7210\.7420\.7240\.672✗✓0\.7180\.7390\.7230\.665✓✗0\.7140\.7330\.7110\.663✗✗0\.6860\.6990\.7080\.668w/o rand\. init\. & goals0\.7110\.7250\.7180\.676Table[2](https://arxiv.org/html/2610.20866#S4.T2)examines how the components of SHRED’s heterogeneous modeling contribute to simulation realism\. Homogeneous dynamics give every road\-user type the pedestrian unicycle model, and homogeneous rewards give every type the vehicle reward set\. Homogenizing either dynamics or rewards causes modest degradation, while the fully homogeneous design produces the largest drop in realism\. Vehicles benefit the most from heterogeneous modeling\. Removing procedural initialization and goals lowers overall realism by 0\.010, with the largest drop for vehicles\.

Table 3\.Realism and diversity on WOSAC 2024\.Unlike HR\-PPO and SPACeR, which anchor self\-play to demonstration\-trained policies, SHRED uses no demonstrations, yet achieves the highest interactive score and higher realism than both, with greater rollout diversity\. The nuPlan\-trained policy retains comparable realism zero\-shot on Waymo\.WOSAC 2024DiversityMethodDemoFreeRealism↑\\uparrowKinematic↑\\uparrowInteractive↑\\uparrowMap↑\\uparrowMean dist\. \(m\)↑\\uparrowSMART \(CAT\-K\)\([Wu et al\., 2024](https://arxiv.org/html/2610.20866#bib.bib4);[Zhang et al\., 2025](https://arxiv.org/html/2610.20866#bib.bib5)\)✗0\.7520\.4530\.7150\.8156\.97HR\-PPO\([Cornelisse and Vinitsky, 2024](https://arxiv.org/html/2610.20866#bib.bib7)\)✗0\.6900\.3080\.7310\.8554\.19SPACeR\([Chang et al\., 2026](https://arxiv.org/html/2610.20866#bib.bib3)\)✗0\.7170\.3780\.7540\.8631\.41SHRED✓0\.7210\.4020\.7780\.8325\.38SHRED \(zero\-shot, nuPlan\)✓0\.7150\.3950\.7760\.8214\.93
#### Self\-Play Realism, Diversity, and Map Transfer\.

Table[3](https://arxiv.org/html/2610.20866#S4.T3)compares SHRED with prior self\-play approaches on WOSAC 2024 across realism and trajectory diversity\. We measure diversity as the mean pairwise Wasserstein distance between each agent’s occupancy distributions across 32 rollouts, following the trajectory\-space analysis of BITS\([Xu et al\., 2023](https://arxiv.org/html/2610.20866#bib.bib10)\)\. Notably, unlike HR\-PPO and SPACeR, SHRED achieves competitive realism and greater trajectory diversity without demonstration data or a pretrained imitation\-policy anchor\. Moreover, the nuPlan\-trained policy retains comparable realism under zero\-shot evaluation on Waymo, suggesting that the learned traffic behaviors transfer across maps and datasets\.

### 4\.2\.Interactive Ego Planning

In this section, we evaluate whether policies learned through demonstration\-free heterogeneous self\-play can support interactive ego planning\. We use InterPlan\([Hallgarten et al\., 2024](https://arxiv.org/html/2610.20866#bib.bib19)\)because it emphasizes challenging interactions with other road users, particularly vulnerable road users \(VRUs\), through long\-tail scenarios including jaywalking pedestrians, overtaking, dense lane changes, construction zones, and accident sites\. The ego planner uses the SHRED policy trained on nuPlan maps\. To isolate the effect of heterogeneous self\-play, we compare against a separately trained vehicle\-only self\-play policy in the planning ablation\.

Table 4\.Interactive planning on InterPlan\.SHRED achieves the highest score among the compared planners on both the official 80\-scenario split and the full 335\-scenario evaluation\. Comparison with a vehicle\-only variant, trained with scripted pedestrians and no cyclists, showcases the benefit of heterogeneous self\-play for interactive planning\. Table[10](https://arxiv.org/html/2610.20866#A5.T10)in Appendix[E\.3](https://arxiv.org/html/2610.20866#A5.SS3)gives the full comparison\.PlannerTypeInterPlan \(80\)↑\\uparrowFull Scale \(335\)↑\\uparrowIDM\([Treiber et al\., 2000](https://arxiv.org/html/2610.20866#bib.bib56)\)Rule31—SPDM \(Np=60N\_\{p\}\{=\}60\)\([Distelzweig et al\., 2026b](https://arxiv.org/html/2610.20866#bib.bib20)\)Rule63\.66—HybridLLMPlanner \(Llama\-7B\)\([Hallgarten et al\., 2024](https://arxiv.org/html/2610.20866#bib.bib19)\)Hybr\.53—PDM\+PPO\([Distelzweig et al\., 2026a](https://arxiv.org/html/2610.20866#bib.bib60)\)Hybr\.43\.2—Diffusion Planner\([Zheng et al\., 2025](https://arxiv.org/html/2610.20866#bib.bib65)\)IL25\.8—FlowDrive\([Wang et al\., 2025](https://arxiv.org/html/2610.20866#bib.bib61)\)IL36\.96—Flow Planner\([Tan et al\., 2025](https://arxiv.org/html/2610.20866#bib.bib66)\)IL—61\.82PPO\([Distelzweig et al\., 2026a](https://arxiv.org/html/2610.20866#bib.bib60)\)RL42\.1—SHREDRL67\.5070\.14SHRED \(vehicle\-only\)RL54\.1966\.65

#### Planning Performance\.

Table[4](https://arxiv.org/html/2610.20866#S4.T4)presents representative baselines; Appendix[E\.3](https://arxiv.org/html/2610.20866#A5.SS3)reports the full comparison\. Beyond learning realistic heterogeneous traffic behavior, SHRED also learns a robust ego planner, achieving the highest score among the compared methods on both InterPlan evaluation sets: 67\.50 on the official 80\-scenario split and 70\.14 on the full 335\-scenario evaluation\. Notably, SHRED internalizes driving behavior through large\-scale heterogeneous self\-play in a compact 6\.7M\-parameter policy, orders of magnitude smaller than learned baselines such as the 7B\-parameter LLaMA backbone used by HybridLLMPlanner, while achieving substantially higher performance \(67\.50 vs\. 53\.00\)\. Even this 6\.7M parameter count reflects a deliberate latency–memory tradeoff: fused heterogeneous LSTM weights increase parameter count and memory usage to enable lower\-latency inference\.

#### Planning Ablations\.

Table[4](https://arxiv.org/html/2610.20866#S4.T4)asks whether heterogeneous self\-play benefits the ego planner itself\. We compare SHRED with a vehicle\-only self\-play policy trained on the same nuPlan maps with the same vehicle dynamics and rewards\. The vehicle\-only policy learns vehicle control through self\-play while pedestrians follow scripted crossing and jaywalking behaviors and cyclists are absent, mirroring the general setup of prior vehicle\-centric self\-play approaches such as Gigaflow\([Cusumano\-Towner et al\., 2025](https://arxiv.org/html/2610.20866#bib.bib2)\)\. SHRED improves the InterPlan score from 54\.19 to 67\.50 on the official 80\-scenario split and from 66\.65 to 70\.14 on the full 335\-scenario evaluation\. The substantially larger gain on the official split, which concentrates challenging interactions with other road users, suggests that heterogeneous self\-play is particularly beneficial for interactive planning\.

### 4\.3\.Behavioral Analysis

#### Qualitative Evaluation of Data Diversity\.

A key question for any simulation policy is whether it reproduces only the specific behaviors present in the training data, or whether it generalizes to a broader distribution of plausible maneuvers\. To assess this, we generate 32 closed\-loop rollouts per scenario and overlay the resulting trajectory distributions against the ground\-truth log \(Figure[1](https://arxiv.org/html/2610.20866#S1.F1)\)\. The policy distribution covers the log trajectory as a high\-probability mode while placing meaningful probability mass on alternative maneuvers: unexecuted turns at intersections, alternative branches at forks, and a range of merge timings\. Supplementary videos show planning around vulnerable road users and in dense interactive scenarios\.

Figure 3\.Group BehaviorMetrics\.Proximity clustering and directional alignment versus pedestrian populationNpedN\_\{\\rm ped\}\. Shaded bands:±1\\pm 1standard deviation across scenarios\. Left axis:in\-group fractionfgrpf\_\{\\rm grp\};local alignmentμalign\\mu\_\{\\rm align\}; right axis:mean group sizeg¯\\bar\{g\}\. All three GBMs increase monotonically with crowd density for the policy while remaining near zero / unity for the stationary baseline\.
#### Emergent Collective Behavior\.

We observe that structured collective pedestrian behavior emerges from purely decentralized, self\-interested agents without any group reward or shared objective\. To test this, we sweep the configured pedestrian populationNped∈\{64,128,256,512,1024\}N\_\{\\rm ped\}\\in\\\{64,\\,128,\\,256,\\,512,\\,1024\\\}while holding vehicles \(V=16V=16\) and cyclists \(C=8C=8\) fixed and evaluating over 32 validation scenarios of 60 s\. We make no changes to the individual reward or goal structure; pedestrians independently optimize their own progress and safety objectives under random goal assignments\.

We measure three*Group Behavior Metrics*\(GBMs\): i\)*in\-group fraction*fgrpf\_\{\\rm grp\}\([Küpper and Seyfried, 2023](https://arxiv.org/html/2610.20866#bib.bib34)\), the fraction of pedestrians that spend at least 50% of a rollout within 5 m of at least one co\-moving companion \(proximity\-linked group\); ii\)*local alignment*μalign\\mu\_\{\\rm align\}\([Luque Sánchez et al\., 2020](https://arxiv.org/html/2610.20866#bib.bib35)\), per\-frame mean cosine similarity between each pedestrian’s velocity and those of itskk\-nearest neighbors, averaged over moving frames; and iii\)*mean group size*g¯\\bar\{g\}\([Küpper and Seyfried, 2023](https://arxiv.org/html/2610.20866#bib.bib34)\), average headcount of identified proximity groups\. A*stationary baseline*, pedestrians with random positions and orientations \(used as proxy velocities\) frozen for the entire rollout, provides a nonmoving reference\.

Figure[3](https://arxiv.org/html/2610.20866#S4.F3.fig1)shows increasing proximity clustering and directional alignment as pedestrian density rises, despite the absence of an explicit group objective\. The stationary reference stays near its floor at every density, so static crowding alone does not produce the measured structure\. The result illustrates aggregate structure under individual objectives, consistent with the emergence of crowd patterns in local\-interaction models\([Helbing and Molnár, 1995](https://arxiv.org/html/2610.20866#bib.bib36);[Moussaïd et al\., 2011](https://arxiv.org/html/2610.20866#bib.bib37)\)\.

## 5\.Conclusion

We have presented SHRED, a framework for jointly training vehicles, pedestrians, and cyclists through self\-play reinforcement learning with type\-specific dynamics, rewards, and value normalization, from random initialization without trajectory supervision or logged\-trajectory replay\. On the Waymo Open Sim Agents Challenge, SHRED achieves strong realism without trajectory\-supervised policy training, while policies trained on nuPlan maps retain comparable realism under zero\-shot evaluation on Waymo\. Beyond simulation, the learned policy also serves as an effective ego planner, outperforming all compared planners on InterPlan\. Controlled ablations further show that heterogeneous self\-play improves both traffic realism and planning performance, with particularly large planning gains in interaction\-heavy scenarios\.

Our results position heterogeneous self\-play as a complementary paradigm to data\-driven approaches across both traffic simulation and autonomous planning\. Imitation\-based methods retain an advantage in overall distributional realism on WOSAC, while self\-play removes dependence on demonstration data and enables behaviors to emerge through closed\-loop interaction\. Together, our results suggest that heterogeneous self\-play provides a scalable, demonstration\-free paradigm for learning interactive traffic behavior across both simulation and planning\.

#### Limitations and Future Work\.

SHRED inherits the fidelity ceiling of its simulator and reward design: behaviors not captured by the dynamics or objectives, such as richer right\-of\-way conventions and traffic rules, may not emerge through self\-play\. Future work could address these limitations through richer traffic\-rule objectives, automatically learned or calibrated rewards, and additional road\-user types\. A realism gap to imitation\-based models also remains, particularly in map\-based realism \(0\.832 vs\. 0\.863 for SPACeR\), suggesting that lightweight data\-driven priors could complement self\-play where demonstrations are available\. Finally, WOSAC measures agreement with logged behavior rather than behavioral validity, and we do not evaluate transfer to real\-world driving\. Evaluating learned behaviors against real\-world interactions and using SHRED as interactive background traffic for downstream planner training and evaluation are important directions for future work\.

## References

- Ahmadiet al\.\(2026\)E\. Ahmadi, H\. Schofield, B\. Khamidehi, F\. Arasteh, J\. Shan, L\. Mou, D\. Bai, and K\. RezaeeRLFTSim: realistic and controllable multi\-agent traffic simulation via reinforcement learning fine\-tuning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 39734–39743\.Cited by:[§2\.1](https://arxiv.org/html/2610.20866#S2.SS1.p1.1)\.
- Alahiet al\.\(2016\)A\. Alahi, K\. Goel, V\. Ramanathan, A\. Robicquet, L\. Fei\-Fei, and S\. SavareseSocial LSTM: human trajectory prediction in crowded spaces\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 961–971\.Cited by:[§2\.1](https://arxiv.org/html/2610.20866#S2.SS1.p1.1)\.
- Berneret al\.\(2019\)C\. Berner, G\. Brockman, B\. Chan, V\. Cheung, P\. Dębiak, C\. Dennison, D\. Farhi, Q\. Fischer, S\. Hashme, C\. Hesse,et al\.Dota 2 with large scale deep reinforcement learning\.arXiv preprint arXiv:1912\.06680\.Cited by:[§2\.2](https://arxiv.org/html/2610.20866#S2.SS2.p1.1)\.
- Caoet al\.\(2024\)Y\. Cao, B\. Ivanovic, C\. Xiao, and M\. PavoneReinforcement learning with human feedback for realistic traffic simulation\.InIEEE International Conference on Robotics and Automation \(ICRA\),pp\. 14428–14434\.Cited by:[§2\.1](https://arxiv.org/html/2610.20866#S2.SS1.p1.1)\.
- Changet al\.\(2026\)W\. Chang, A\. Rangesh, K\. Joseph, M\. Strong, M\. Tomizuka, Y\. Hu, and W\. ZhanSPACeR: self\-play anchoring with centralized reference models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2610.20866#S1.p2.1),[§1](https://arxiv.org/html/2610.20866#S1.p3.1),[§2\.2](https://arxiv.org/html/2610.20866#S2.SS2.p1.1),[§3\.2](https://arxiv.org/html/2610.20866#S3.SS2.p1.1),[Table 3](https://arxiv.org/html/2610.20866#S4.T3.6.5.1.2.1.1.1)\.
- Chenet al\.\(2025\)K\. Chen, W\. Sun, H\. Cheng, and S\. ZhengRIFT: group\-relative RL fine\-tuning for realistic and controllable traffic simulation\.arXiv preprint arXiv:2505\.03344\.Cited by:[§2\.1](https://arxiv.org/html/2610.20866#S2.SS1.p1.1)\.
- Christianoset al\.\(2021\)F\. Christianos, G\. Papoudakis, A\. Rahman, and S\. V\. AlbrechtScaling multi\-agent reinforcement learning with selective parameter sharing\.InAdvances in Neural Information Processing Systems,Cited by:[§2\.3](https://arxiv.org/html/2610.20866#S2.SS3.p1.1)\.
- Cornelisseet al\.\(2025\)D\. Cornelisse, A\. Pandya, K\. Joseph, J\. Suárez, and E\. VinitskyBuilding reliable sim driving agents by scaling self\-play\.arXiv preprint arXiv:2502\.14706\.Cited by:[§2\.2](https://arxiv.org/html/2610.20866#S2.SS2.p1.1),[§3\.2](https://arxiv.org/html/2610.20866#S3.SS2.p1.1)\.
- Cornelisse and Vinitsky \(2024\)D\. Cornelisse and E\. VinitskyHuman\-compatible driving partners through data\-regularized self\-play reinforcement learning\.Reinforcement Learning Journal5,pp\. 2320–2344\.Cited by:[§2\.2](https://arxiv.org/html/2610.20866#S2.SS2.p1.1),[Table 3](https://arxiv.org/html/2610.20866#S4.T3.6.4.1.2.1.1.1)\.
- Cusumano\-Towneret al\.\(2025\)M\. Cusumano\-Towner, D\. Hafner, A\. Hertzberg, B\. Huval, A\. Petrenko, E\. Vinitsky, E\. Wijmans, T\. W\. Killian, S\. Bowers, O\. Sener, P\. Kraehenbuehl, and V\. KoltunRobust autonomy emerges from self\-play\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 11710–11737\.Cited by:[§1](https://arxiv.org/html/2610.20866#S1.p3.1),[§2\.2](https://arxiv.org/html/2610.20866#S2.SS2.p1.1),[§3\.1](https://arxiv.org/html/2610.20866#S3.SS1.SSS0.Px2.p1.1),[§3\.3](https://arxiv.org/html/2610.20866#S3.SS3.SSS0.Px2.p2.1),[§4\.1](https://arxiv.org/html/2610.20866#S4.SS1.SSS0.Px2.p1.1),[§4\.2](https://arxiv.org/html/2610.20866#S4.SS2.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2610.20866#S4.T1.6.1.2.1)\.
- Dauneret al\.\(2023\)D\. Dauner, M\. Hallgarten, A\. Geiger, and K\. ChittaParting with misconceptions about learning\-based vehicle motion planning\.InConference on Robot Learning \(CoRL\),Cited by:[Table 10](https://arxiv.org/html/2610.20866#A5.T10.5.1.16.1),[Table 10](https://arxiv.org/html/2610.20866#A5.T10.5.1.4.1)\.
- Distelzweiget al\.\(2026a\)A\. Distelzweig, F\. Janjoš, A\. Look, A\. Rothenhäusler, D\. Jost, O\. Scheel, R\. Rajan, D\. Cornelisse, E\. Vinitsky, and J\. BoedeckerBeyond self\-play and scale: a behavior benchmark for generalization in autonomous driving\.arXiv preprint arXiv:2605\.10034\.Cited by:[Table 10](https://arxiv.org/html/2610.20866#A5.T10.5.1.11.1),[Table 10](https://arxiv.org/html/2610.20866#A5.T10.5.1.21.1),[Table 4](https://arxiv.org/html/2610.20866#S4.T4.6.1.5.1),[Table 4](https://arxiv.org/html/2610.20866#S4.T4.6.1.9.1)\.
- Distelzweiget al\.\(2026b\)A\. Distelzweig, F\. Janjoš, O\. Scheel, S\. R\. Varra, R\. Rajan, and J\. BoedeckerPerfect prediction or plenty of proposals? what matters most in planning for autonomous driving\.InIEEE International Conference on Robotics and Automation \(ICRA\),Note:arXiv:2510\.15505Cited by:[Table 10](https://arxiv.org/html/2610.20866#A5.T10.5.1.5.1),[Table 10](https://arxiv.org/html/2610.20866#A5.T10.5.1.6.1),[Table 4](https://arxiv.org/html/2610.20866#S4.T4.6.1.3.1)\.
- Espeholtet al\.\(2018\)L\. Espeholt, H\. Soyer, R\. Munos, K\. Simonyan, V\. Mnih, T\. Ward, Y\. Doron, V\. Firoiu, T\. Harley, I\. Dunning,et al\.IMPALA: scalable distributed deep\-RL with importance weighted actor\-learner architectures\.InInternational Conference on Machine Learning \(ICML\),pp\. 1407–1416\.Cited by:[§3\.3](https://arxiv.org/html/2610.20866#S3.SS3.SSS0.Px2.p2.1)\.
- Gulinoet al\.\(2023\)C\. Gulino, J\. Fu, W\. Luo, G\. Tucker, E\. Bronstein, Y\. Lu, J\. Harb, X\. Pan, Y\. Wang, X\. Chen,et al\.Waymax: an accelerated, data\-driven simulator for large\-scale autonomous driving research\.InAdvances in Neural Information Processing Systems, Datasets and Benchmarks Track,Cited by:[§2\.1](https://arxiv.org/html/2610.20866#S2.SS1.p1.1)\.
- Guoet al\.\(2026\)K\. Guo, H\. Liu, X\. Wu, and C\. LvDecompGAIL: learning realistic traffic behaviors with decomposed multi\-agent generative adversarial imitation learning\.arXiv preprint arXiv:2510\.06913\.Note:Accepted to ICLRCited by:[§2\.1](https://arxiv.org/html/2610.20866#S2.SS1.p1.1)\.
- Guptaet al\.\(2018\)A\. Gupta, J\. Johnson, L\. Fei\-Fei, S\. Savarese, and A\. AlahiSocial GAN: socially acceptable trajectories with generative adversarial networks\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 2255–2264\.Cited by:[§2\.1](https://arxiv.org/html/2610.20866#S2.SS1.p1.1)\.
- Hallgartenet al\.\(2023\)M\. Hallgarten, M\. Stoll, and A\. ZellFrom prediction to planning with goal conditioned lane graph traversals\.InIEEE International Conference on Intelligent Transportation Systems \(ITSC\),Cited by:[Table 10](https://arxiv.org/html/2610.20866#A5.T10.5.1.14.1)\.
- Hallgartenet al\.\(2024\)M\. Hallgarten, J\. Zapata, M\. Stoll, K\. Renz, and A\. ZellCan vehicle motion planning generalize to realistic long\-tail scenarios?\.InIEEE/RSJ International Conference on Intelligent Robots and Systems \(IROS\),Cited by:[Table 10](https://arxiv.org/html/2610.20866#A5.T10.5.1.10.1),[Table 10](https://arxiv.org/html/2610.20866#A5.T10.5.1.8.1),[Table 10](https://arxiv.org/html/2610.20866#A5.T10.5.1.9.1),[§1](https://arxiv.org/html/2610.20866#S1.p5.1),[§4\.2](https://arxiv.org/html/2610.20866#S4.SS2.p1.1),[Table 4](https://arxiv.org/html/2610.20866#S4.T4.6.1.4.1)\.
- Helbinget al\.\(2005\)D\. Helbing, L\. Buzna, A\. Johansson, and T\. WernerSelf\-organized pedestrian crowd dynamics: experiments, simulations, and design solutions\.Transportation Science39\(1\),pp\. 1–24\.Cited by:[§2\.1](https://arxiv.org/html/2610.20866#S2.SS1.p1.1)\.
- Helbing and Molnár \(1995\)D\. Helbing and P\. MolnárSocial force model for pedestrian dynamics\.Physical Review E51\(5\),pp\. 4282–4286\.Cited by:[§2\.1](https://arxiv.org/html/2610.20866#S2.SS1.p1.1),[§4\.3](https://arxiv.org/html/2610.20866#S4.SS3.SSS0.Px2.p3.1)\.
- Huanget al\.\(2024\)Z\. Huang, P\. Karkus, B\. Ivanovic, Y\. Chen, M\. Pavone, and C\. LvDTPP: differentiable joint conditional prediction and cost evaluation for tree policy planning in autonomous driving\.InIEEE International Conference on Robotics and Automation \(ICRA\),Cited by:[Table 10](https://arxiv.org/html/2610.20866#A5.T10.5.1.7.1),[§1](https://arxiv.org/html/2610.20866#S1.p2.1)\.
- Huanget al\.\(2023\)Z\. Huang, H\. Liu, and C\. LvGameFormer: game\-theoretic modeling and learning of transformer\-based interactive prediction and planning for autonomous driving\.InIEEE/CVF International Conference on Computer Vision \(ICCV\),Cited by:[Table 10](https://arxiv.org/html/2610.20866#A5.T10.5.1.15.1),[§1](https://arxiv.org/html/2610.20866#S1.p2.1)\.
- Jaderberget al\.\(2019\)M\. Jaderberg, W\. M\. Czarnecki, I\. Dunning, L\. Marris, G\. Lever, A\. G\. Castañeda, C\. Beattie, N\. C\. Rabinowitz, A\. S\. Morcos, A\. Ruderman,et al\.Human\-level performance in 3D multiplayer games with population\-based reinforcement learning\.Science364\(6443\),pp\. 859–865\.Cited by:[§2\.2](https://arxiv.org/html/2610.20866#S2.SS2.p1.1)\.
- Jaegeret al\.\(2025\)B\. Jaeger, D\. Dauner, J\. Beißwenger, S\. Gerstenecker, K\. Chitta, and A\. GeigerCaRL: learning scalable planning policies with simple rewards\.arXiv preprint arXiv:2504\.17838\.Cited by:[Appendix C](https://arxiv.org/html/2610.20866#A3.SS0.SSS0.Px3.p1.1)\.
- Jianget al\.\(2024\)M\. Jiang, Y\. Bai, A\. Cornman, C\. Davis, X\. Huang, H\. Jeon, S\. Kulshrestha, J\. Lambert, S\. Li, X\. Zhou,et al\.SceneDiffuser: efficient and controllable driving simulation initialization and rollout\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 55729–55760\.Cited by:[§2\.1](https://arxiv.org/html/2610.20866#S2.SS1.p1.1)\.
- Kazemkhaniet al\.\(2025\)S\. Kazemkhani, A\. Pandya, D\. Cornelisse, B\. Shacklett, and E\. VinitskyGPUDrive: data\-driven, multi\-agent driving simulation at 1 million fps\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2\.1](https://arxiv.org/html/2610.20866#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2610.20866#S2.SS2.p1.1)\.
- Kestinget al\.\(2007\)A\. Kesting, M\. Treiber, and D\. HelbingGeneral lane\-changing model MOBIL for car\-following models\.Transportation Research Record1999\(1\),pp\. 86–94\.Cited by:[Table 10](https://arxiv.org/html/2610.20866#A5.T10.5.1.3.1)\.
- Kubaet al\.\(2022\)J\. G\. Kuba, R\. Chen, M\. Wen, Y\. Wen, F\. Sun, J\. Wang, and Y\. YangTrust region policy optimisation in multi\-agent reinforcement learning\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2\.3](https://arxiv.org/html/2610.20866#S2.SS3.p1.1)\.
- Küpper and Seyfried \(2023\)M\. Küpper and A\. SeyfriedIdentification of social groups and waiting pedestrians at railway platforms using trajectory data\.PLoS ONE18\(3\),pp\. e0282526\.External Links:[Document](https://dx.doi.org/10.1371/journal.pone.0282526)Cited by:[§4\.3](https://arxiv.org/html/2610.20866#S4.SS3.SSS0.Px2.p2.1)\.
- Liet al\.\(2022\)Q\. Li, Z\. Peng, L\. Feng, Q\. Zhang, Z\. Xue, and B\. ZhouMetaDrive: composing diverse driving scenarios for generalizable reinforcement learning\.IEEE Transactions on Pattern Analysis and Machine Intelligence\.Cited by:[§2\.1](https://arxiv.org/html/2610.20866#S2.SS1.p1.1)\.
- Luque Sánchezet al\.\(2020\)F\. Luque Sánchez, I\. Hupont, S\. Tabik, and F\. HerreraRevisiting crowd behaviour analysis through deep learning: taxonomy, anomaly detection, crowd emotions, datasets, opportunities and prospects\.Information Fusion64,pp\. 318–335\.External Links:[Document](https://dx.doi.org/10.1016/j.inffus.2020.07.008)Cited by:[§4\.3](https://arxiv.org/html/2610.20866#S4.SS3.SSS0.Px2.p2.1)\.
- Montaliet al\.\(2023\)N\. Montali, J\. Lambert, P\. Mougin, A\. Kuefler, N\. Rhinehart, M\. Li, C\. Gulino, T\. Emrich, Z\. Yang, S\. Whiteson, B\. White, and D\. AnguelovThe Waymo open sim agents challenge\.InAdvances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks Track,Vol\.36,pp\. 59151–59171\.Cited by:[§1](https://arxiv.org/html/2610.20866#S1.p5.1),[§2\.1](https://arxiv.org/html/2610.20866#S2.SS1.p1.1)\.
- Moussaïdet al\.\(2011\)M\. Moussaïd, D\. Helbing, and G\. TheraulazHow simple rules determine pedestrian behavior and crowd disasters\.Proceedings of the National Academy of Sciences \(PNAS\)108\(17\),pp\. 6884–6888\.Cited by:[§4\.3](https://arxiv.org/html/2610.20866#S4.SS3.SSS0.Px2.p3.1)\.
- Obando\-Ceronet al\.\(2024\)J\. Obando\-Ceron, G\. Sokar, T\. Willi, C\. Lyle, J\. Farebrother, J\. Foerster, G\. K\. Dziugaite, D\. Precup, and P\. S\. CastroMixtures of experts unlock parameter scaling for deep RL\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§2\.3](https://arxiv.org/html/2610.20866#S2.SS3.p1.1)\.
- Peiet al\.\(2026\)M\. Pei, S\. Shi, and S\. ShenAdvancing multi\-agent traffic simulation via R1\-style reinforcement fine\-tuning\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2\.1](https://arxiv.org/html/2610.20866#S2.SS1.p1.1)\.
- Penget al\.\(2024\)Z\. Peng, W\. Luo, Y\. Lu, T\. Shen, C\. Gulino, A\. Seff, and J\. FuImproving agent behaviors with RL fine\-tuning for autonomous driving\.InEuropean Conference on Computer Vision,pp\. 165–181\.Cited by:[§2\.1](https://arxiv.org/html/2610.20866#S2.SS1.p1.1)\.
- Philionet al\.\(2024\)J\. Philion, X\. B\. Peng, and S\. FidlerTrajeglish: traffic modeling as next\-token prediction\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2610.20866#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.20866#S2.SS1.p1.1)\.
- Rashidet al\.\(2018\)T\. Rashid, M\. Samvelyan, C\. Schroeder de Witt, G\. Farquhar, J\. Foerster, and S\. WhitesonQMIX: monotonic value function factorisation for deep multi\-agent reinforcement learning\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§2\.3](https://arxiv.org/html/2610.20866#S2.SS3.p1.1)\.
- Rasouli and Kotseruba \(2023\)A\. Rasouli and I\. KotserubaPedFormer: pedestrian behavior prediction via cross\-modal attention modulation and gated multitask learning\.InIEEE International Conference on Robotics and Automation \(ICRA\),Cited by:[§2\.1](https://arxiv.org/html/2610.20866#S2.SS1.p1.1)\.
- Roweet al\.\(2024\)L\. Rowe, R\. Girgis, A\. Gosselin, B\. Carrez, F\. Golemo, F\. Heide, L\. Paull, and C\. PalCtRL\-Sim: reactive and controllable driving agents with offline reinforcement learning\.InConference on Robot Learning \(CoRL\),Cited by:[§2\.1](https://arxiv.org/html/2610.20866#S2.SS1.p1.1)\.
- Salzmannet al\.\(2020\)T\. Salzmann, B\. Ivanovic, P\. Chakravarty, and M\. PavoneTrajectron\+\+: dynamically\-feasible trajectory forecasting with heterogeneous data\.InEuropean Conference on Computer Vision \(ECCV\),pp\. 683–700\.Cited by:[§2\.1](https://arxiv.org/html/2610.20866#S2.SS1.p1.1)\.
- Scheelet al\.\(2022\)O\. Scheel, L\. Bergamini, M\. Wolczyk, B\. Osiński, and P\. OndruskaUrban driver: learning to drive from real\-world demonstrations using policy gradients\.InConference on Robot Learning \(CoRL\),pp\. 718–728\.Cited by:[Table 10](https://arxiv.org/html/2610.20866#A5.T10.5.1.13.1)\.
- Silveret al\.\(2017\)D\. Silver, J\. Schrittwieser, K\. Simonyan, I\. Antonoglou, A\. Huang, A\. Guez, T\. Hubert, L\. Baker, M\. Lai, A\. Bolton,et al\.Mastering the game of Go without human knowledge\.Nature550\(7676\),pp\. 354–359\.Cited by:[§2\.2](https://arxiv.org/html/2610.20866#S2.SS2.p1.1)\.
- Suoet al\.\(2021\)S\. Suo, S\. Regalado, S\. Casas, and R\. UrtasunTrafficSim: learning to simulate realistic multi\-agent behaviors\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 10400–10409\.Cited by:[§2\.1](https://arxiv.org/html/2610.20866#S2.SS1.p1.1)\.
- Tanet al\.\(2024\)S\. Tan, B\. Ivanovic, Y\. Chen, B\. Li, X\. Weng, Y\. Cao, P\. Krähenbühl, and M\. PavonePromptable closed\-loop traffic simulation\.InConference on Robot Learning \(CoRL\),Cited by:[§2\.1](https://arxiv.org/html/2610.20866#S2.SS1.p1.1)\.
- Tanet al\.\(2025\)T\. Tan, Y\. Zheng, R\. Liang, Z\. Wang, K\. Zheng, J\. Zheng, J\. Li, X\. Zhan, and J\. LiuFlow matching\-based autonomous driving planning with advanced interactive behavior modeling\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2510\.11083Cited by:[Table 10](https://arxiv.org/html/2610.20866#A5.T10.5.1.19.1),[§1](https://arxiv.org/html/2610.20866#S1.p2.1),[Table 4](https://arxiv.org/html/2610.20866#S4.T4.6.1.8.1)\.
- Treiberet al\.\(2000\)M\. Treiber, A\. Hennecke, and D\. HelbingCongested traffic states in empirical observations and microscopic simulations\.Physical Review E62,pp\. 1805–1824\.Cited by:[Table 10](https://arxiv.org/html/2610.20866#A5.T10.5.1.2.1),[Table 10](https://arxiv.org/html/2610.20866#A5.T10.5.1.3.1),[Table 4](https://arxiv.org/html/2610.20866#S4.T4.6.1.2.1)\.
- van den Berget al\.\(2011\)J\. van den Berg, S\. J\. Guy, M\. Lin, and D\. ManochaReciprocalnn\-body collision avoidance\.InRobotics Research \(Springer Tracts in Advanced Robotics\),pp\. 3–19\.Cited by:[§2\.1](https://arxiv.org/html/2610.20866#S2.SS1.p1.1)\.
- van Hasseltet al\.\(2016\)H\. P\. van Hasselt, A\. Guez, M\. Hessel, V\. Mnih, and D\. SilverLearning values across many orders of magnitude\.InAdvances in Neural Information Processing Systems,Cited by:[§3\.3](https://arxiv.org/html/2610.20866#S3.SS3.SSS0.Px2.p2.1)\.
- Vinitskyet al\.\(2022\)E\. Vinitsky, N\. Lichtlé, X\. Yang, B\. Amos, and J\. FoersterNocturne: a scalable driving benchmark for bringing multi\-agent learning one step closer to the real world\.InAdvances in Neural Information Processing Systems, Datasets and Benchmarks Track,Cited by:[§2\.1](https://arxiv.org/html/2610.20866#S2.SS1.p1.1)\.
- Vinyalset al\.\(2019\)O\. Vinyals, I\. Babuschkin, W\. M\. Czarnecki, M\. Mathieu, A\. Dudzik, J\. Chung, D\. H\. Choi, R\. Powell, T\. Ewalds, P\. Georgiev,et al\.Grandmaster level in StarCraft II using multi\-agent reinforcement learning\.Nature575\(7782\),pp\. 350–354\.Cited by:[§2\.2](https://arxiv.org/html/2610.20866#S2.SS2.p1.1)\.
- Wanget al\.\(2025\)L\. Wang, Ö\. Ş\. Taş, M\. Steiner, and C\. StillerFlowDrive: moderated flow matching with data balancing for trajectory planning\.arXiv preprint arXiv:2509\.21961\.Cited by:[Table 10](https://arxiv.org/html/2610.20866#A5.T10.5.1.12.1),[Table 10](https://arxiv.org/html/2610.20866#A5.T10.5.1.18.1),[Table 4](https://arxiv.org/html/2610.20866#S4.T4.6.1.7.1)\.
- Wanget al\.\(2020\)T\. Wang, H\. Dong, V\. Lesser, and C\. ZhangROMA: multi\-agent reinforcement learning with emergent roles\.InProceedings of the 37th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.119,pp\. 9876–9886\.Cited by:[§2\.3](https://arxiv.org/html/2610.20866#S2.SS3.p1.1)\.
- Wanget al\.\(2021\)T\. Wang, T\. Gupta, A\. Mahajan, B\. Peng, S\. Whiteson, and C\. ZhangRODE: learning roles to decompose multi\-agent tasks\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2\.3](https://arxiv.org/html/2610.20866#S2.SS3.p1.1)\.
- Wanget al\.\(2026\)Z\. Wang, S\. Rahmani, D\. Cornelisse, B\. Sarkar, A\. D\. Goldie, J\. N\. Foerster, and S\. WhitesonLearning to drive in new cities without human demonstrations\.arXiv preprint arXiv:2602\.15891\.Cited by:[§2\.2](https://arxiv.org/html/2610.20866#S2.SS2.p1.1)\.
- Wuet al\.\(2024\)W\. Wu, X\. Feng, Z\. Gao, and Y\. KanSMART: scalable multi\-agent real\-time motion generation via next\-token prediction\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 114048–114071\.Cited by:[§1](https://arxiv.org/html/2610.20866#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.20866#S2.SS1.p1.1),[Table 3](https://arxiv.org/html/2610.20866#S4.T3.6.3.1.2.1.2.1)\.
- Wuet al\.\(2026a\)Z\. Wu, A\. Rangesh, W\. Li, W\. Chang, Z\. Lee, S\. Bonab, T\. Wang, and W\. ZhanTerraZero: procedural driving simulation for zero\-demonstration self\-play at scale\.External Links:2607\.13028,[Link](https://arxiv.org/abs/2607.13028)Cited by:[§4\.1](https://arxiv.org/html/2610.20866#S4.SS1.SSS0.Px1.p1.1)\.
- Wuet al\.\(2026b\)Z\. Wu, R\. Song, V\. Mundheda, L\. E\. Navarro\-Serment, C\. Schoenborn, and J\. SchneiderTADPO: reinforcement learning goes off\-road\.arXiv preprint arXiv:2603\.05995\.Cited by:[Appendix C](https://arxiv.org/html/2610.20866#A3.SS0.SSS0.Px3.p1.1)\.
- Xionget al\.\(2026\)Z\. Xiong, W\. Li, Z\. Wu, A\. Rangesh, S\. Bonde, G\. Hall, C\. Tang, Y\. Hu, and W\. ZhanTerraTransfer: learning end\-to\-end driving policies without expert demonstrations\.External Links:2606\.17386,[Link](https://arxiv.org/abs/2606.17386)Cited by:[§2\.2](https://arxiv.org/html/2610.20866#S2.SS2.p1.1)\.
- Xuet al\.\(2023\)D\. Xu, Y\. Chen, B\. Ivanovic, and M\. PavoneBITS: bi\-level imitation for traffic simulation\.InIEEE International Conference on Robotics and Automation \(ICRA\),Cited by:[§4\.1](https://arxiv.org/html/2610.20866#S4.SS1.SSS0.Px4.p1.1)\.
- Xuet al\.\(2022\)P\. Xu, J\. Hayet, and I\. KaramouzasSocialVAE: human trajectory prediction using timewise latents\.InEuropean Conference on Computer Vision \(ECCV\),pp\. 511–528\.Cited by:[§2\.1](https://arxiv.org/html/2610.20866#S2.SS1.p1.1)\.
- Yanget al\.\(2020\)R\. Yang, H\. Xu, Y\. Wu, and X\. WangMulti\-task reinforcement learning with soft modularization\.InAdvances in Neural Information Processing Systems,Cited by:[§2\.3](https://arxiv.org/html/2610.20866#S2.SS3.p1.1)\.
- Yuet al\.\(2022\)C\. Yu, A\. Velu, E\. Vinitsky, J\. Gao, Y\. Wang, A\. Bayen, and Y\. WuThe surprising effectiveness of PPO in cooperative, multi\-agent games\.Advances in Neural Information Processing Systems, Datasets and Benchmarks Track35\.Cited by:[§2\.3](https://arxiv.org/html/2610.20866#S2.SS3.p1.1)\.
- Yuet al\.\(2026\)H\. Yu, Y\. Jin, A\. Canevaro, J\. Schmidt, J\. Jordan, P\. Li, M\. Kaufeld, S\. Lindner, J\. Betz, and W\. StorkG2DP: diffusion planning with spatio\-temporal grid guidance\.arXiv preprint arXiv:2606\.26017\.Cited by:[Table 10](https://arxiv.org/html/2610.20866#A5.T10.5.1.20.1)\.
- Zhanget al\.\(2025\)Z\. Zhang, P\. Karkus, M\. Igl, W\. Ding, Y\. Chen, B\. Ivanovic, and M\. PavoneClosed\-loop supervised fine\-tuning of tokenized traffic models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 5422–5432\.Cited by:[§2\.1](https://arxiv.org/html/2610.20866#S2.SS1.p1.1),[Table 3](https://arxiv.org/html/2610.20866#S4.T3.6.3.1.2.1.2.1)\.
- Zhenget al\.\(2025\)Y\. Zheng, R\. Liang, K\. Zheng, J\. Zheng, L\. Mao, J\. Li, W\. Gu, R\. Ai, S\. E\. Li, X\. Zhan, and J\. LiuDiffusion\-based planning for autonomous driving with flexible guidance\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2501\.15564Cited by:[Table 10](https://arxiv.org/html/2610.20866#A5.T10.5.1.17.1),[§1](https://arxiv.org/html/2610.20866#S1.p2.1),[Table 4](https://arxiv.org/html/2610.20866#S4.T4.6.1.6.1)\.
- Zhonget al\.\(2024\)Y\. Zhong, J\. G\. Kuba, X\. Feng, S\. Hu, J\. Ji, and Y\. YangHeterogeneous\-agent reinforcement learning\.Journal of Machine Learning Research25\.Cited by:[§2\.3](https://arxiv.org/html/2610.20866#S2.SS3.p1.1)\.
- Zhouet al\.\(2024\)Z\. Zhou, H\. Hu, X\. Chen, J\. Wang, N\. Guan, K\. Wu, Y\. Li, Y\. Huang, and C\. J\. HoBehaviorGPT: smart agent simulation for autonomous driving with next\-patch prediction\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2610.20866#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.20866#S2.SS1.p1.1)\.

## Appendix APer\-Type Dynamics

This section gives the full continuous\-time integrators for each of the three dynamics models summarized in §[3\.1](https://arxiv.org/html/2610.20866#S3.SS1)\. All models share a common kinematic statest=\(x,y,θ,vx,vy,along,alat,δ\)s\_\{t\}=\(x,y,\\theta,v\_\{x\},v\_\{y\},a\_\{\\text\{long\}\},a\_\{\\text\{lat\}\},\\delta\)and operate atΔ​t=0\.1\\Delta t=0\.1\\,s\. Per\-type kinematic parameters and domain\-randomization ranges appear in Table[5](https://arxiv.org/html/2610.20866#A1.T5)\.

### A\.1\.Vehicle: Jerk\-Integrated Ackermann Arc

The vehicle model operates at the*jerk*level: the policy selects longitudinal and lateral jerk values from a discrete grida˙long∈\{−15,−4,0,4\}\\dot\{a\}\_\{\\text\{long\}\}\\in\\\{\-15,\\,\-4,\\,0,\\,4\\\}m/s3anda˙lat∈\{−4,0,4\}\\dot\{a\}\_\{\\text\{lat\}\}\\in\\\{\-4,\\,0,\\,4\\\}m/s3, giving\|𝒜\|=12\|\\mathcal\{A\}\|=12actions\.

#### Acceleration Integration\.

Per\-episode coefficientsCthrottle,CsteerC\_\{\\text\{throttle\}\},C\_\{\\text\{steer\}\}scale the jerk inputs:

\(1\)along\(t\)\\displaystyle a\_\{\\text\{long\}\}^\{\(t\)\}=along\(t−1\)\+Cthrottle​a˙long​Δ​t,\\displaystyle=a\_\{\\text\{long\}\}^\{\(t\-1\)\}\+C\_\{\\text\{throttle\}\}\\,\\dot\{a\}\_\{\\text\{long\}\}\\,\\Delta t,\(2\)alat\(t\)\\displaystyle a\_\{\\text\{lat\}\}^\{\(t\)\}=alat\(t−1\)\+Csteer​a˙lat​Δ​t\.\\displaystyle=a\_\{\\text\{lat\}\}^\{\(t\-1\)\}\+C\_\{\\text\{steer\}\}\\,\\dot\{a\}\_\{\\text\{lat\}\}\\,\\Delta t\.When longitudinal acceleration changes sign \(along\(t−1\)⋅along\(t\)<0a\_\{\\text\{long\}\}^\{\(t\-1\)\}\\cdot a\_\{\\text\{long\}\}^\{\(t\)\}<0\), it is snapped to zero, enabling smooth waiting and constant\-velocity trajectories\. A per\-episode coefficientCaccC\_\{\\text\{acc\}\}scales the upper acceleration bound before clamping:

\(3\)along\(t\)←clip⁡\(along\(t\),−5\.0,2\.5​Cacc\),alat\(t\)←clip⁡\(alat\(t\),−4\.0,4\.0\)\.a\_\{\\text\{long\}\}^\{\(t\)\}\\leftarrow\\mathrm\{clip\}\\\!\\left\(a\_\{\\text\{long\}\}^\{\(t\)\},\\;\-5\.0,\\;2\.5\\,C\_\{\\text\{acc\}\}\\right\),\\qquad a\_\{\\text\{lat\}\}^\{\(t\)\}\\leftarrow\\mathrm\{clip\}\\\!\\left\(a\_\{\\text\{lat\}\}^\{\(t\)\},\\;\-4\.0,\\;4\.0\\right\)\.

#### Velocity Update\.

Speed integrates by the trapezoidal rule and is clipped with aCvelC\_\{\\text\{vel\}\}\-scaled upper bound:

\(4\)v\(t\)=v\(t−1\)\+12​\(along\(t\)\+along\(t−1\)\)​Δ​t,v\(t\)←clip⁡\(v\(t\),−2\.0,20\.0​Cvel\)\.v^\{\(t\)\}=v^\{\(t\-1\)\}\+\\tfrac\{1\}\{2\}\\\!\\left\(a\_\{\\text\{long\}\}^\{\(t\)\}\+a\_\{\\text\{long\}\}^\{\(t\-1\)\}\\right\)\\Delta t,\\qquad v^\{\(t\)\}\\leftarrow\\mathrm\{clip\}\\\!\\left\(v^\{\(t\)\},\\;\-2\.0,\\;20\.0\\,C\_\{\\text\{vel\}\}\\right\)\.Velocity is snapped to zero on sign change, as with acceleration\.

#### Steering Angle\.

The target steering angle is derived from the desired curvatureκ=alat\(t\)/max⁡\(\(v\(t\)\)2,ϵ\)\\kappa=a\_\{\\text\{lat\}\}^\{\(t\)\}\\,/\\max\\\!\\left\(\(v^\{\(t\)\}\)^\{2\},\\,\\epsilon\\right\)withϵ=10−5\\epsilon=10^\{\-5\}:

\(5\)ϕtarget\\displaystyle\\phi\_\{\\text\{target\}\}=arctan⁡\(κ​lwb\),\\displaystyle=\\arctan\\\!\\left\(\\kappa\\,l\_\{\\text\{wb\}\}\\right\),\(6\)Δ​ϕ\\displaystyle\\Delta\\phi=clip⁡\(ϕtarget−ϕ\(t−1\),−δ˙max​Δ​t,δ˙max​Δ​t\),\\displaystyle=\\mathrm\{clip\}\\\!\\left\(\\phi\_\{\\text\{target\}\}\-\\phi^\{\(t\-1\)\},\\;\-\\dot\{\\delta\}\_\{\\max\}\\,\\Delta t,\\;\\dot\{\\delta\}\_\{\\max\}\\,\\Delta t\\right\),\(7\)ϕ\(t\)\\displaystyle\\phi^\{\(t\)\}=clip⁡\(ϕ\(t−1\)\+Δ​ϕ,−ϕmax,ϕmax\),\\displaystyle=\\mathrm\{clip\}\\\!\\left\(\\phi^\{\(t\-1\)\}\+\\Delta\\phi,\\;\-\\phi\_\{\\max\},\\;\\phi\_\{\\max\}\\right\),wherelwb=0\.6×lengthl\_\{\\text\{wb\}\}=0\.6\\times\\textit\{length\}is the wheelbase,δ˙max=0\.6\\dot\{\\delta\}\_\{\\max\}=0\.6rad/s is the slew rate limit, andϕmax=0\.55\\phi\_\{\\max\}=0\.55rad is the hard steering cap\. After clamping, curvature and lateral acceleration are recomputed from the realized steering angle to maintain consistency:κ←tan⁡ϕ\(t\)/lwb\\kappa\\leftarrow\\tan\\phi^\{\(t\)\}\\\!/\\,l\_\{\\text\{wb\}\},alat\(t\)←\(v\(t\)\)2​κa\_\{\\text\{lat\}\}^\{\(t\)\}\\leftarrow\(v^\{\(t\)\}\)^\{2\}\\,\\kappa\.

#### Position Update\.

The pose advances along a circular arc:

\(8\)d=12​\(v\(t\)\+v\(t−1\)\)​Δ​t,ϑ=d​κ,Δ​xlocal=sin⁡ϑκ,Δ​ylocal=1−cos⁡ϑκ,d=\\tfrac\{1\}\{2\}\(v^\{\(t\)\}\+v^\{\(t\-1\)\}\)\\,\\Delta t,\\qquad\\vartheta=d\\,\\kappa,\\qquad\\Delta x\_\{\\text\{local\}\}=\\frac\{\\sin\\vartheta\}\{\\kappa\},\\qquad\\Delta y\_\{\\text\{local\}\}=\\frac\{1\-\\cos\\vartheta\}\{\\kappa\},rotated into the global frame by the current heading; heading updates asθ\(t\)=θ\(t−1\)\+ϑ\\theta^\{\(t\)\}=\\theta^\{\(t\-1\)\}\+\\vartheta\. When\|κ\|<10−5\|\\kappa\|<10^\{\-5\}\(nearly straight\), the arc degenerates to a line segment\(Δ​xlocal,Δ​ylocal\)=\(d,0\)\(\\Delta x\_\{\\text\{local\}\},\\,\\Delta y\_\{\\text\{local\}\}\)=\(d,\\,0\)\.

### A\.2\.Pedestrian: Unicycle

Pedestrian motion is modeled as a unicycle where heading is an*integrated state*and velocity is constrained to lie along the heading direction \(zero slip\)\. The action is a pair\(av,ω\)\(a\_\{v\},\\,\\omega\), longitudinal acceleration and heading rate, selected from a nonuniform7×77\\times 7grid plus a dedicatedstopaction:

av∈\{−2\.0,−0\.8,−0\.2,0,0\.2,0\.8,2\.0\}​m/s2,\\displaystyle a\_\{v\}\\in\\\{\-2\.0,\\,\-0\.8,\\,\-0\.2,\\,0,\\,0\.2,\\,0\.8,\\,2\.0\\\}\\;\\text\{m/s\}^\{2\},ω∈\{−2\.0,−0\.5,−0\.1,0,0\.1,0\.5,2\.0\}​rad/s,\\displaystyle\\omega\\in\\\{\-2\.0,\\,\-0\.5,\\,\-0\.1,\\,0,\\,0\.1,\\,0\.5,\\,2\.0\\\}\\;\\text\{rad/s\},giving\|𝒜\|=50\|\\mathcal\{A\}\|=50\(49 grid \+ 1 stop\)\. The grid concentrates resolution near zero where typical pedestrian accelerations and heading changes lie \(Waymo WOMDp90p\_\{90\}:\|av\|≈1\.5\|a\_\{v\}\|\\approx 1\.5m/s2,\|ω\|≈1\.3\|\\omega\|\\approx 1\.3rad/s\)\. Thestopaction sets bothava\_\{v\}andω\\omegato zero and forces speed to zero, providing a clean discrete “stand still” primitive\.

#### Integration\.

After clipping\(av←clip⁡\(av,−2\.0,2\.0\)CLOSE\(a\_\{v\}\\leftarrow\\mathrm\{clip\}\(a\_\{v\},\\,\-2\.0,\\,2\.0\),OPENω←clip⁡\(ω,−2\.0,2\.0\)\)\\omega\\leftarrow\\mathrm\{clip\}\(\\omega,\\,\-2\.0,\\,2\.0\)\):

\(9\)v\(t\)\\displaystyle v^\{\(t\)\}=clip⁡\(v\(t−1\)\+av​Δ​t,0,3\.0\),\\displaystyle=\\mathrm\{clip\}\\\!\\left\(v^\{\(t\-1\)\}\+a\_\{v\}\\,\\Delta t,\\;0,\\;3\.0\\right\),\(10\)θ\(t\)\\displaystyle\\theta^\{\(t\)\}=θ\(t−1\)\+ω​Δ​t,\\displaystyle=\\theta^\{\(t\-1\)\}\+\\omega\\,\\Delta t,\(11\)𝐯\(t\)\\displaystyle\\mathbf\{v\}^\{\(t\)\}=v\(t\)​\(cos⁡θ\(t\),sin⁡θ\(t\)\),\\displaystyle=v^\{\(t\)\}\\\!\\left\(\\cos\\theta^\{\(t\)\},\\,\\sin\\theta^\{\(t\)\}\\right\),\(12\)𝐩\(t\)\\displaystyle\\mathbf\{p\}^\{\(t\)\}=𝐩\(t−1\)\+𝐯\(t\)​Δ​t\.\\displaystyle=\\mathbf\{p\}^\{\(t\-1\)\}\+\\mathbf\{v\}^\{\(t\)\}\\,\\Delta t\.Speed is clipped to\[0,3\.0\]\[0,\\,3\.0\]m/s \(walking/jogging cap\)\. Body\-frame accelerations for observation and reward parity with other models are derived numerically:along=\(v\(t\)−v\(t−1\)\)/Δ​ta\_\{\\text\{long\}\}=\(v^\{\(t\)\}\-v^\{\(t\-1\)\}\)/\\Delta t,alat=v\(t\)⋅\(θ\(t\)−θ\(t−1\)\)/Δ​ta\_\{\\text\{lat\}\}=v^\{\(t\)\}\\cdot\(\\theta^\{\(t\)\}\-\\theta^\{\(t\-1\)\}\)/\\Delta t\.

### A\.3\.Cyclist: Kinematic Bicycle

Cyclists use a kinematic bicycle model where the action directly controls accelerationaaand front\-wheel steering angleδ\\delta\. The discrete grid is7×77\\times 7withlogarithmic spacing:

a∈\{−5\.0,−2\.0,−0\.5,0,0\.5,2\.0,5\.0\}​m/s2,\\displaystyle a\\in\\\{\-5\.0,\\,\-2\.0,\\,\-0\.5,\\,0,\\,0\.5,\\,2\.0,\\,5\.0\\\}\\;\\text\{m/s\}^\{2\},δ∈\{−0\.20,−0\.06,−0\.02,0,0\.02,0\.06,0\.20\}​rad,\\displaystyle\\delta\\in\\\{\-0\.20,\\,\-0\.06,\\,\-0\.02,\\,0,\\,0\.02,\\,0\.06,\\,0\.20\\\}\\;\\text\{rad\},giving\|𝒜\|=49\|\\mathcal\{A\}\|=49\. The geometric ratio \(≈3\{\\approx\}\\,3\) concentrates roughly half of all actions within the low\-magnitude regime where fine steering matters most for typical urban cycling, providing about3×3\\timesfiner resolution near zero for both acceleration and steering than a uniform 7\-point grid with the same bounds\.

#### Scaling and Clipping\.

Per\-episode coefficientsCsteer,Cacc,CvelC\_\{\\text\{steer\}\},C\_\{\\text\{acc\}\},C\_\{\\text\{vel\}\}scale the action before integration:

\(13\)δ←Csteer​δ,a←clip⁡\(a~,−5\.0,3\.0\),a~=\{Cacc​aa\>0,aa≤0,\\delta\\leftarrow C\_\{\\text\{steer\}\}\\,\\delta,\\qquad a\\leftarrow\\mathrm\{clip\}\\\!\\left\(\\tilde\{a\},\\;\-5\.0,\\;3\.0\\right\),\\qquad\\tilde\{a\}=\\begin\{cases\}C\_\{\\text\{acc\}\}\\,a&a\>0,\\\\ a&a\\leq 0,\\end\{cases\}whereCaccC\_\{\\text\{acc\}\}scales only forward acceleration \(braking is not power\-limited\)\.

#### Velocity Update\.

\(14\)v\(t\)=v\(t−1\)\+a​Δ​t,v\(t\)←clip⁡\(v\(t\),−1\.0,10\.0​Cvel\)\.v^\{\(t\)\}=v^\{\(t\-1\)\}\+a\\,\\Delta t,\\qquad v^\{\(t\)\}\\leftarrow\\mathrm\{clip\}\\\!\\left\(v^\{\(t\)\},\\;\-1\.0,\\;10\.0\\,C\_\{\\text\{vel\}\}\\right\)\.Velocity is snapped to zero on sign change\.

#### Slip Angle and Yaw Rate\.

The slip angleβ\\betais computed via atanh\\tanh\-smoothed approximation that prevents numerical instability at large steering angles:

\(15\)β=tanh⁡\(12​tan⁡δ\),θ˙=v\(t\)​cos⁡β​tan⁡δL,\\beta=\\tanh\\\!\\left\(\\tfrac\{1\}\{2\}\\tan\\delta\\right\),\\qquad\\dot\{\\theta\}=\\frac\{v^\{\(t\)\}\\cos\\beta\\,\\tan\\delta\}\{L\},whereL=max⁡\(length,0\.1\)L=\\max\(\\textit\{length\},\\,0\.1\)m is the effective wheelbase, guarded against degenerate zero\-length entities\.

#### Position and Heading Update\.

\(16\)θ\(t\)\\displaystyle\\theta^\{\(t\)\}=θ\(t−1\)\+θ˙​Δ​t,\\displaystyle=\\theta^\{\(t\-1\)\}\+\\dot\{\\theta\}\\,\\Delta t,\(17\)𝐯\(t\)\\displaystyle\\mathbf\{v\}^\{\(t\)\}=v\(t\)​\(cos⁡\(θ\(t−1\)\+β\),sin⁡\(θ\(t−1\)\+β\)\),\\displaystyle=v^\{\(t\)\}\\\!\\left\(\\cos\(\\theta^\{\(t\-1\)\}\+\\beta\),\\;\\sin\(\\theta^\{\(t\-1\)\}\+\\beta\)\\right\),\(18\)𝐩\(t\)\\displaystyle\\mathbf\{p\}^\{\(t\)\}=𝐩\(t−1\)\+𝐯\(t\)​Δ​t\.\\displaystyle=\\mathbf\{p\}^\{\(t\-1\)\}\+\\mathbf\{v\}^\{\(t\)\}\\,\\Delta t\.Note that the velocity directionθ\(t−1\)\+β\\theta^\{\(t\-1\)\}\+\\betadiffers from the headingθ\(t\)\\theta^\{\(t\)\}by the slip angle during turns\. Lateral acceleration is derived asalat=v\(t\)⋅θ˙a\_\{\\text\{lat\}\}=v^\{\(t\)\}\\\!\\cdot\\\!\\dot\{\\theta\}\.

### A\.4\.Domain Randomization Parameters

Each dynamics model is parameterized by per\-episode coefficientsCthrottle,Csteer,Cacc,CvelC\_\{\\text\{throttle\}\},C\_\{\\text\{steer\}\},C\_\{\\text\{acc\}\},C\_\{\\text\{vel\}\}sampled uniformly from type\-specific ranges at each episode reset \(Table[5](https://arxiv.org/html/2610.20866#A1.T5)\)\. Vehicle coefficients span a wide\[0\.5,1\.5\]\[0\.5,\\,1\.5\]band to encourage robustness across car\-to\-truck\-scale dynamics; cyclist coefficients use a narrower\[0\.7,1\.3\]\[0\.7,\\,1\.3\]band reflecting the tighter physical envelope of bicycle kinematics; pedestrian coefficients are fixed at1\.01\.0since the unicycle model already covers the behaviorally relevant range\. All four coefficients are included as observation features so that the policy can condition on its current kinematic regime\.

Table 5\.Per\-type kinematic parameters and domain\-randomization ranges\. Coefficient ranges \(C∗C\_\{\*\}\) are sampled uniformly per agent per episode; scalar values are fixed\. Clip bounds are hard limits enforced by the dynamics integrator\. Bounding\-box ranges are sampled uniformly per agent at spawn, calibrated to the Waymo Open Motion Dataset distribution\. Vehicles additionally draw a Bernoulli truck/car bodywork class that selects from an extended length range\[6\.1,12\.2\]\[6\.1,\\,12\.2\]m for trucks\.ParameterVehiclePedestrianCyclistDynamics modeljerk\-Ackermannunicyclekin\. bicycleCthrottleC\_\{\\text\{throttle\}\}\[0\.5,1\.5\]\[0\.5,\\,1\.5\]1\.01\.01\.01\.0CsteerC\_\{\\text\{steer\}\}\[0\.5,1\.5\]\[0\.5,\\,1\.5\]1\.01\.0\[0\.7,1\.3\]\[0\.7,\\,1\.3\]CaccC\_\{\\text\{acc\}\}\[0\.5,1\.5\]\[0\.5,\\,1\.5\]1\.01\.0\[0\.7,1\.3\]\[0\.7,\\,1\.3\]CvelC\_\{\\text\{vel\}\}\[0\.5,1\.5\]\[0\.5,\\,1\.5\]1\.01\.0\[0\.7,1\.3\]\[0\.7,\\,1\.3\]vvclip \(m/s\)\[−2\.0,20\.0\]\[\-2\.0,\\,20\.0\]\[0\.0,3\.0\]\[0\.0,\\,3\.0\]\[−1\.0,10\.0\]\[\-1\.0,\\,10\.0\]alonga\_\{\\text\{long\}\}clip \(m/s2\)\[−5\.0,2\.5\]\[\-5\.0,\\,2\.5\]\[−2\.0,2\.0\]\[\-2\.0,\\,2\.0\]\[−5\.0,3\.0\]\[\-5\.0,\\,3\.0\]alata\_\{\\text\{lat\}\}clip \(m/s2\)\[−4\.0,4\.0\]\[\-4\.0,\\,4\.0\]\[−4\.0,4\.0\]\[\-4\.0,\\,4\.0\]\[−4\.0,4\.0\]\[\-4\.0,\\,4\.0\]\|δ\|\|\\delta\|cap \(rad\)0\.550\.55——\|δ˙\|\|\\dot\{\\delta\}\|slew \(rad/s\)0\.60\.6——\|ω\|\|\\omega\|cap \(rad/s\)—2\.02\.0—Length \(m\)\[3\.5,5\.5\]\[3\.5,\\,5\.5\]\[0\.65,1\.0\]\[0\.65,\\,1\.0\]\[1\.4,2\.0\]\[1\.4,\\,2\.0\]Width \(m\)\[1\.5,2\.5\]\[1\.5,\\,2\.5\]\[0\.65,1\.0\]\[0\.65,\\,1\.0\]\[0\.71,1\.0\]\[0\.71,\\,1\.0\]Height \(m\)\[1\.2,2\.0\]\[1\.2,\\,2\.0\]\[1\.5,1\.9\]\[1\.5,\\,1\.9\]\[1\.5,1\.9\]\[1\.5,\\,1\.9\]

## Appendix BRandom Initialization Details

Map memory is partitioned into atwo\-band entity layout: index range\[0,Nlog\)\[0,N\_\{\\text\{log\}\}\)for log\-replay entities and\[Nlog,Nlog\+Nrand\)\[N\_\{\\text\{log\}\},N\_\{\\text\{log\}\}\+N\_\{\\text\{rand\}\}\)for randomly placed controlled agents\. This layout is a simulator capability\. The reported SHRED training run uses only the randomly placed controlled band and does not replay logged future trajectories\.

For each typek∈𝒦k\\in\\mathcal\{K\}we draw up toNkN\_\{k\}agents byrejection sampling: vehicles snap to lane segments with heading noiseΔθ∼𝒰\(−π/7\.2,π/7\.2\)\\Delta\\theta\\sim\\mathcal\{U\}\(\-\\pi/7\.2,\\pi/7\.2\), initial speedv0∼𝒰⁡\(0,2\)v\_\{0\}\\sim\\mathcal\{U\}\(0,2\)m/s, and a Bernoulli truck/car bodywork draw; pedestrians sample a crosswalk with probabilitypcwp\_\{\\text\{cw\}\}and otherwise a road edge; cyclists use the same lane\-sampling pipeline as vehicles\. Each candidate is rejected on collision or off\-road, with at most100100attempts per slot\.

#### Per\-Type Goal Generation\.

When an agent is randomly spawned \(or reaches its current goal\), a new goal is synthesized from map topology rather than copied from the log\. The generation strategy is type\-specific:

- •Vehiclesdraw a target distanced∼𝒰⁡\(dmin,dmax\)d\\sim\\mathcal\{U\}\(d\_\{\\min\},d\_\{\\max\}\)and walk forward along the lane topology graph, randomly selecting successors at lane ends \(up to 10 transitions\), producing goals that follow the road structure\. Candidate goals are filtered for heading alignment \(≥cos⁡45∘\{\\geq\}\\cos 45^\{\\circ\}\)\.
- •Pedestriansspawn on either a crosswalk \(goal at the opposite edge\) or a road edge \(goal along the edge polyline with a sidewalk\-side lateral offset\), with distance boundsd∈\[10,30\]d\\in\[10,30\]m\. Subsequent goals chain along the nearest road edge using the pedestrian’s current heading for directionality\.
- •Cyclistsfollow the same lane\-topology walk as vehicles, with distance boundsd∈\[20,60\]d\\in\[20,60\]m\.

To prevent the policy from learning to exploit goal\-failure removal as a shortcut, agents that fail to obtain a valid goal are*not*removed from the scene\. Instead, they receive an additional binary observation feature indicating the absence of an active goal, and the policy must learn to act sensibly under partial supervision\.

## Appendix CPer\-Type Reward Details

Each agent’s reward is a weighted sum of terms spanning safety, progress, comfort, and type\-specific constraints\. Letrtk=∑iαik​ri,tr^\{k\}\_\{t\}=\\sum\_\{i\}\\alpha^\{k\}\_\{i\}\\,r\_\{i,t\}denote the reward for an agent of typekkat steptt, where each weightαik\\alpha^\{k\}\_\{i\}is sampled independently from a type\-specific uniform range at episode reset \(reward domain randomization\)\. Table[6](https://arxiv.org/html/2610.20866#A3.T6)lists the full per\-type reward structure\.

#### Comfort\.

The comfort term counts the number of kinematic channels that exceed a type\-specific threshold:

Ncomfort=∑i∈\{al,at,jl,jt\}\[\|xi\|\>τi\(k\)\],N\_\{\\text\{comfort\}\}=\\sum\_\{i\\in\\\{a\_\{l\},\\,a\_\{t\},\\,j\_\{l\},\\,j\_\{t\}\\\}\}\\mathbb\{1\}\\\!\\left\[\|x\_\{i\}\|\>\\tau^\{\(k\)\}\_\{i\}\\right\],withrcomfort=−αcomfort​Ncomfortr\_\{\\text\{comfort\}\}=\-\\alpha\_\{\\text\{comfort\}\}\\,N\_\{\\text\{comfort\}\}\. Table[7](https://arxiv.org/html/2610.20866#A3.T7)lists the thresholds, calibrated per type from an 80k\-scenario analysis of the Waymo Open Motion Dataset at thep95p\_\{95\}–p97\.5p\_\{97\.5\}percentile of real\-data magnitudes\.

#### Pedestrian Road Intrusion\.

Pedestrians receive a per\-step jaywalking penalty when their center is on the road surface and*not*within a crosswalk polygon:

rintrusion=−αintrusion⋅Δt⋅𝟙\[on\_road\]⋅𝟙\[¬on\_crosswalk\]\.r\_\{\\text\{intrusion\}\}=\-\\alpha\_\{\\text\{intrusion\}\}\\cdot\\Delta t\\cdot\\mathbb\{1\}\[\\text\{on\\\_road\}\]\\cdot\\mathbb\{1\}\[\\neg\\,\\text\{on\\\_crosswalk\}\]\.This term is zero for vehicles and cyclists, who are expected to occupy road lanes\.

#### Type\-Specific Lane and Speed Terms\.

Vehicles receive lane alignment \(αl​\-align\\alpha\_\{l\\text\{\-align\}\}\), lane centering \(αl​\-center\\alpha\_\{l\\text\{\-center\}\}\), velocity alignment \(αv​\-align\\alpha\_\{v\\text\{\-align\}\}\), and reverse penalties \(αreverse\\alpha\_\{\\text\{reverse\}\}\) that are absent for pedestrians and cyclists\. Cyclists receive a softer version of lane alignment plus an edge\-preference bonus \(αedge\\alpha\_\{\\text\{edge\}\}\) that encourages riding near the road margin\. Both pedestrians and cyclists receive a speed\-limit penaltyrspeed=−αspeed\-lim⋅Δt⋅\(v−vlim\)/vmaxr\_\{\\text\{speed\}\}=\-\\alpha\_\{\\text\{speed\-lim\}\}\\cdot\\Delta t\\cdot\(v\-v\_\{\\text\{lim\}\}\)/v\_\{\\max\}whenv\>vlimv\>v\_\{\\text\{lim\}\}, withvlimv\_\{\\text\{lim\}\}sampled per agent from\[1\.5,2\.5\]\[1\.5,\\,2\.5\]m/s for pedestrians and\[5\.0,8\.0\]\[5\.0,\\,8\.0\]m/s for cyclists \(Table[6](https://arxiv.org/html/2610.20866#A3.T6)\)\. Similar reward design strategies have been explored in off\-road\([Wu et al\., 2026b](https://arxiv.org/html/2610.20866#bib.bib52)\)and urban autonomous driving settings\([Jaeger et al\., 2025](https://arxiv.org/html/2610.20866#bib.bib53)\)\.

Table 6\.Per\-type reward terms and domain\-randomization ranges\. Dashes mark terms that are disabled \(α=0\\alpha=0\) for that type\.*Const\.*values are not randomized; ranges denote uniform sampling per agent per episode\. All weights are multiplied byΔ​t=0\.1\\Delta t=0\.1s per step except collision and goal, which fire as one\-shot events\.Reward termVehiclePedestrianCyclistαcollision\\alpha\_\{\\text\{collision\}\}\[2\.0,4\.0\]\[2\.0,\\,4\.0\]\[0\.5,2\.0\]\[0\.5,\\,2\.0\]\[1\.0,3\.0\]\[1\.0,\\,3\.0\]αcoll\-speed\\alpha\_\{\\text\{coll\-speed\}\}0\.10\.10\.00\.00\.00\.0mk′m\_\{k^\{\\prime\}\}\(partner mult\.\)\[1,2,1\]\[1,2,1\]——αboundary\\alpha\_\{\\text\{boundary\}\}\[2\.0,4\.0\]\[2\.0,\\,4\.0\]—\[0\.5,2\.0\]\[0\.5,\\,2\.0\]αcomfort\\alpha\_\{\\text\{comfort\}\}\[0\.0,0\.1\]\[0\.0,\\,0\.1\]——αl​\-align\\alpha\_\{l\\text\{\-align\}\}\[2\.5​e\-​4,2\.5​e\-​2\]\[2\.5\\text\{e\-\}4,\\,2\.5\\text\{e\-\}2\]—\[5​e\-​3,2​e\-​2\]\[5\\text\{e\-\}3,\\,2\\text\{e\-\}2\]αv​\-align\\alpha\_\{v\\text\{\-align\}\}\[0\.0,1\.0\]\[0\.0,\\,1\.0\]—\[0\.25,0\.75\]\[0\.25,\\,0\.75\]αl​\-center\\alpha\_\{l\\text\{\-center\}\}\[2\.5​e\-​4,7\.5​e\-​3\]\[2\.5\\text\{e\-\}4,\\,7\.5\\text\{e\-\}3\]——αreverse\\alpha\_\{\\text\{reverse\}\}\[2\.5​e\-​4,7\.5​e\-​3\]\[2\.5\\text\{e\-\}4,\\,7\.5\\text\{e\-\}3\]——αintrusion\\alpha\_\{\\text\{intrusion\}\}—\[0\.5,2\.0\]\[0\.5,\\,2\.0\]—αspeed\-lim\\alpha\_\{\\text\{speed\-lim\}\}—\[0\.5,2\.0\]\[0\.5,\\,2\.0\]\[0\.25,1\.0\]\[0\.25,\\,1\.0\]vlimv\_\{\\text\{lim\}\}\(m/s\)—\[1\.5,2\.5\]\[1\.5,\\,2\.5\]\[5\.0,8\.0\]\[5\.0,\\,8\.0\]αedge\\alpha\_\{\\text\{edge\}\}——\[0\.01,0\.05\]\[0\.01,\\,0\.05\]δgoal\\delta\_\{\\text\{goal\}\}\(m\)\[2\.0,12\.0\]\[2\.0,\\,12\.0\]\[2\.0,12\.0\]\[2\.0,\\,12\.0\]\[2\.0,12\.0\]\[2\.0,\\,12\.0\]αgoal\\alpha\_\{\\text\{goal\}\}1\.01\.01\.01\.01\.01\.0Table 7\.Per\-type comfort thresholdsτ\(k\)\\tau^\{\(k\)\}, calibrated at thep95p\_\{95\}–p97\.5p\_\{97\.5\}percentile of the Waymo Open Motion Dataset\. Pedestrian and cyclist thresholds are calibrated but unused in the reported run, whereαcomfort=0\\alpha\_\{\\text\{comfort\}\}=0for those types\.Typeτal\\tau\_\{a\_\{l\}\}\(m/s2\)τat\\tau\_\{a\_\{t\}\}\(m/s2\)τjl\\tau\_\{j\_\{l\}\}\(m/s3\)τjt\\tau\_\{j\_\{t\}\}\(m/s3\)Vehicle3\.03\.05\.05\.0Pedestrian1\.51\.012\.05\.0Cyclist3\.01\.540\.018\.0

## Appendix DPopArt Weight Correction

Each per\-type PopArt layer maintains running statistics\(μk,σk,νk\)\(\\mu\_\{k\},\\sigma\_\{k\},\\nu\_\{k\}\)updated by EMA over each epoch’s batch returns:

μk←\(1−βk\)​μk\+βk​G¯k,νk←\(1−βk\)​νk\+βk​G2¯k,σk←νk−μk2,\\mu\_\{k\}\\leftarrow\(1\-\\beta\_\{k\}\)\\mu\_\{k\}\+\\beta\_\{k\}\\,\\bar\{G\}\_\{k\},\\qquad\\nu\_\{k\}\\leftarrow\(1\-\\beta\_\{k\}\)\\nu\_\{k\}\+\\beta\_\{k\}\\,\\overline\{G^\{2\}\}\_\{k\},\\qquad\\sigma\_\{k\}\\leftarrow\\sqrt\{\\nu\_\{k\}\-\\mu\_\{k\}^\{\\,2\}\},whereβk=1−ρTbptt\\beta\_\{k\}=1\-\\rho^\{T\_\{\\text\{bptt\}\}\}with configurable decayρ=0\.9997\\rho=0\.9997\. To keep the real\-scale value function unchanged when\(μk,σk\)\(\\mu\_\{k\},\\sigma\_\{k\}\)shifts, the linear weights ofValuek\\mathrm\{Value\}^\{k\}are corrected:

Wk←σkoldσknew​Wk,bk←σkold​bk\+μkold−μknewσknew,W^\{k\}\\leftarrow\\frac\{\\sigma\_\{k\}^\{\\text\{old\}\}\}\{\\sigma\_\{k\}^\{\\text\{new\}\}\}\\,W^\{k\},\\qquad b^\{k\}\\leftarrow\\frac\{\\sigma\_\{k\}^\{\\text\{old\}\}b^\{k\}\+\\mu\_\{k\}^\{\\text\{old\}\}\-\\mu\_\{k\}^\{\\text\{new\}\}\}\{\\sigma\_\{k\}^\{\\text\{new\}\}\},ensuring no policy\-gradient discontinuity at the commit step\.

## Appendix EWOSAC Evaluation Protocol and Additional Results

### E\.1\.WOSAC 2024 Validation

Training remains procedural and does not use logged\-trajectory replay\.

We compute the realism meta\-score with the official weighted definition and prepend logged history when computing kinematic features\. On 50 WOSAC scenarios, the GPU evaluator and the official evaluator agree within3\.5×10−53\.5\\times 10^\{\-5\}on the realism meta\-score\.

Table 8\.Agreement between the GPU evaluator and the official WOSAC evaluator on 50 scenarios\.MetricGPU evaluatorOfficial evaluatorMaximum differenceDistance to nearest object0\.2582440\.2582441\.2×10−71\.2\\times 10^\{\-7\}Time to collision0\.8826760\.8826761\.3×10−71\.3\\times 10^\{\-7\}Collision indication0\.8509350\.8509358\.0×10−88\.0\\times 10^\{\-8\}Distance to road edge0\.6463140\.6463141\.4×10−71\.4\\times 10^\{\-7\}Off\-road indication0\.8677360\.8677368\.7×10−88\.7\\times 10^\{\-8\}Linear speed0\.2368080\.2362713\.7×10−33\.7\\times 10^\{\-3\}Linear acceleration0\.3654950\.3671871\.8×10−21\.8\\times 10^\{\-2\}Angular speed0\.3665930\.3658084\.0×10−34\.0\\times 10^\{\-3\}Angular acceleration0\.5861350\.5850678\.1×10−38\.1\\times 10^\{\-3\}Realism meta\-score0\.6861430\.6861083\.5×10−53\.5\\times 10^\{\-5\}
### E\.2\.WOSAC 2025 Test Evaluation

Table[9](https://arxiv.org/html/2610.20866#A5.T9)reports the WOSAC 2025 test evaluation\. The 2025 metric differs from the 2023 and 2024 metrics, so we report it separately\.

Table 9\.WOSAC 2025 test evaluation for SHRED\.MetricValueRealism meta\-score0\.722Linear speed0\.162Linear acceleration0\.337Angular speed0\.343Angular acceleration0\.574Distance to nearest object0\.316Collision0\.951Time to collision0\.806Distance to road edge0\.529Off\-road0\.907minADE \(m\)5\.920
### E\.3\.Full InterPlan Comparison

Table[10](https://arxiv.org/html/2610.20866#A5.T10)expands the representative comparison in Table[4](https://arxiv.org/html/2610.20866#S4.T4)to include all baselines and their listed variants\.

Table 10\.Full comparison of interactive planners on InterPlan\. InterPlan \(80\) is the official 80\-scenario split, and Full Scale \(335\) evaluates all 335 scenarios\. SHRED achieves the highest score among the compared planners on both evaluation sets\. Vehicle\-only self\-play learns vehicle control with scripted pedestrians and no cyclists\.PlannerTypeInterPlan \(80\)↑\\uparrowFull Scale \(335\)↑\\uparrowIDM\([Treiber et al\., 2000](https://arxiv.org/html/2610.20866#bib.bib56)\)Rule31—IDM\+MOBIL\([Treiber et al\., 2000](https://arxiv.org/html/2610.20866#bib.bib56);[Kesting et al\., 2007](https://arxiv.org/html/2610.20866#bib.bib57)\)Rule31—PDM\-Closed\([Dauner et al\., 2023](https://arxiv.org/html/2610.20866#bib.bib58)\)Rule42—SPDM \(Np=15N\_\{p\}\{=\}15\)\([Distelzweig et al\., 2026b](https://arxiv.org/html/2610.20866#bib.bib20)\)Rule42\.00—SPDM \(Np=60N\_\{p\}\{=\}60\)\([Distelzweig et al\., 2026b](https://arxiv.org/html/2610.20866#bib.bib20)\)Rule63\.66—DTPP\([Huang et al\., 2024](https://arxiv.org/html/2610.20866#bib.bib59)\)Hybr\.25—HybridLLMPlanner \(GPT\-3\.5\)\([Hallgarten et al\., 2024](https://arxiv.org/html/2610.20866#bib.bib19)\)Hybr\.40—HybridLLMPlanner \(Llama\-13B\)\([Hallgarten et al\., 2024](https://arxiv.org/html/2610.20866#bib.bib19)\)Hybr\.48—HybridLLMPlanner \(Llama\-7B\)\([Hallgarten et al\., 2024](https://arxiv.org/html/2610.20866#bib.bib19)\)Hybr\.53—PDM\+PPO\([Distelzweig et al\., 2026a](https://arxiv.org/html/2610.20866#bib.bib60)\)Hybr\.43\.2—FlowDrive w/ guidance \+ PDM scoring\([Wang et al\., 2025](https://arxiv.org/html/2610.20866#bib.bib61)\)Hybr\.44\.05—Urban Driver\([Scheel et al\., 2022](https://arxiv.org/html/2610.20866#bib.bib62)\)IL4—GC\-PGP\([Hallgarten et al\., 2023](https://arxiv.org/html/2610.20866#bib.bib63)\)IL10—GameFormer\([Huang et al\., 2023](https://arxiv.org/html/2610.20866#bib.bib64)\)IL11—PDM\-Open\([Dauner et al\., 2023](https://arxiv.org/html/2610.20866#bib.bib58)\)IL25—Diffusion Planner\([Zheng et al\., 2025](https://arxiv.org/html/2610.20866#bib.bib65)\)IL25\.8—FlowDrive\([Wang et al\., 2025](https://arxiv.org/html/2610.20866#bib.bib61)\)IL36\.96—Flow Planner\([Tan et al\., 2025](https://arxiv.org/html/2610.20866#bib.bib66)\)IL—61\.82G2DP\([Yu et al\., 2026](https://arxiv.org/html/2610.20866#bib.bib67)\)IL—61\.74PPO\([Distelzweig et al\., 2026a](https://arxiv.org/html/2610.20866#bib.bib60)\)RL42\.1—SHREDRL67\.5070\.14SHRED \(vehicle\-only\)RL54\.1966\.65