CMU-Drive and V2V-VLA: Cooperative Multi-agent Unified Driving with Reasoning Benchmark and Vehicle-to-Vehicle Vision-Language-Action Models

# Cooperative Multi-agent Unified Driving with Reasoning Benchmark and Vehicle-to-Vehicle Vision-Language-Action Models
Source: [https://arxiv.org/html/2608.07621](https://arxiv.org/html/2608.07621)
11institutetext:Carnegie Mellon University, Robotics Institute###### Abstract

Vision\-Language\-Action \(VLA\) models have recently achieved impressive performance for end\-to\-end autonomous driving, yet existing approaches are primarily designed for an individual single autonomous driving agent with limited support for cooperative perception, reasoning, and planning\. We presentCooperative Multi\-agent Unified Driving with Reasoning \(CMU\-Drive\), a closed\-loop end\-to\-end benchmark for evaluating cooperative autonomous driving with multiple connected autonomous vehicles \(CAVs\) operating in safety\-critical driving scenarios with background traffic participants\. We further proposeVehicle\-to\-Vehicle Vision\-Language\-Action \(V2V\-VLA\), a cooperative VLA model that integrates cooperative driving into a single forward pass by jointly generating driving actions, future waypoints, language reasoning, and communication policies\. Experiments on CMU\-Drive establish the first benchmark and baseline for cooperative VLA driving and provide a foundation for future research on multi\-agent, closed\-loop, end\-to\-end cooperative autonomous driving\. Our code, benchmark, and model checkpoint will be publicly released to facilitate open\-source research\.

## 1Introduction

Vision\-Language\-Action \(VLA\) models have recently demonstrated strong capabilities for end\-to\-end autonomous driving by jointly performing visual perception, language reasoning, and vehicle control\[Renz2025simlingo,jia2024bench,nvidia2025alpamayo,li2025spacedrive,zhou2025autovla,hwang2024emma,tian2024DriveVLM,hu2025vla4ad,xing2024openemma\]\. While existing VLA systems are primarily evaluated in single\-agent settings, cooperative autonomous driving\[xu2023v2v4real,xiang2024v2xreal,luo2025mixed,zimmer2024tumtraf,coscoy2026mdrive,liu2025colmdriver,zhou2024v2xpnp,zhou2025turbotrain,zhao2025coopre,zhao2025quantv2x,cho2025cocoon,chiu2026v2vllm,chiu2026v2vgot\]requires multiple autonomous vehicles to perceive, reason, and coordinate under partial observability\. This raises a new challenge: enabling VLA models to actively acquire complementary observations from neighboring vehicles while maintaining efficient closed\-loop driving\.

Existing benchmarks and methods address different aspects of this problem\. Bench2Drive\[jia2024bench\]establishes comprehensive closed\-loop evaluation for single\-agent end\-to\-end driving across diverse urban scenarios with interactions among vehicles, pedestrians, and cyclists, and SimLingo\[Renz2025simlingo\]achieves state\-of\-the\-art performance on this benchmark by unifying driving, vision\-language understanding, and language\-action alignment within a single model\. CoLMDriver\[liu2025colmdriver\]extends end\-to\-end evaluation to multi\-agent driving through curating the InterDrive\[liu2025colmdriver\]benchmark, and the proposed model resolves conflicting driving intentions using multi\-round LLM\-based negotiation\. However, this work does not consider much on safety\-critical driving scenarios with background traffic participants and vulnerable road users\. MDrive\[coscoy2026mdrive\]creates agentic\-generated interaction scenarios and further combines InterDrive\[liu2025colmdriver\]and V2X scenarios\[zhou2024v2xpnp\]for cooperative driving evaluation, but primarily benchmarks existing methods from prior works\[liu2025colmdriver,liu2025toward\]without introducing a new model or algorithm for cooperative driving\. Moreover, the aforementioned existing cooperative benchmarks\[liu2025colmdriver,coscoy2026mdrive\]involve relatively few cooperative autonomous driving agents as shown in[Tab\.˜1](https://arxiv.org/html/2608.07621#S1.T1)\. The number of cooperative autonomous driving agents is less than the typical number of vehicles in busy urban intersections\. Such a benchmark design limits the complexity and realism of the evaluation scenarios\.

To bridge these gaps, we introduceCooperative Multi\-agent Unified Driving with Reasoning \(CMU\-Drive\), a closed\-loop, end\-to\-end cooperative driving benchmark built upon Bench2Drive\[jia2024bench\]\. CMU\-Drive preserves rich interactions with background traffic participants while extending each scenario to include 2 to 16 cooperative autonomous vehicles, enabling simultaneous interactions among cooperative vehicles, non\-cooperative vehicles, pedestrians, and cyclists\. In addition, we implement our benchmark with a different coding architecture design from the approaches in the prior works\[liu2025colmdriver,coscoy2026mdrive\], so that CMU\-Drive only requires a single GPU to run closed\-loop evaluation with at most 16 cooperative driving agents\.[Tab\.˜1](https://arxiv.org/html/2608.07621#S1.T1)shows the statistics of CMU\-Drive, compared to other cooperative multi\-agent, closed\-loop, end\-to\-end driving benchmarks\.

Table 1:Statistics of the evaluation routes in CMU\-Drive, in comparison to other cooperative multi\-agent, end\-to\-end, closed\-loop driving benchmarks\. \# Routes: the number of evaluation routes\. \# CAVs: the number of connected autonomous vehicles per route\. \# GPUs: the number of GPUs required to run the evaluation per route\.We further proposeVehicle\-to\-Vehicle Vision\-Language\-Action \(V2V\-VLA\), a cooperative foundation model that incorporates cooperative perception, reasoning, and planning\. Unlike negotiation\-based approaches\[liu2025colmdriver\]that require multiple rounds of language interaction and LLM inference, V2V\-VLA jointly generates driving actions, future waypoints, language reasoning, and communication policies, allowing each vehicle to reason about when communication is beneficial and which neighboring vehicle can provide complementary observations to make driving decisions\. Together, CMU\-Drive and V2V\-VLA establish a unified benchmark and baseline for studying reasoning\-guided cooperative multi\-agent, closed\-loop, end\-to\-end autonomous driving\.

## 2CMU\-Drive: Cooperative Multi\-agent Unified Driving with Reasoning Benchmark

### 2\.1Benchmark Creation

CMU\-Drive extends the widely adopted Bench2Drive\[jia2024bench\]benchmark by transforming its single\-agent simulation and evaluation protocol into a cooperative multi\-agent setting while preserving its diverse urban traffic scenarios\. Specifically, CMU\-Drive inherits the 44 types of scenarios from Bench2Drive\[jia2024bench\], such as pedestrian crossing, emergency vehicles running a red light, and other safety\-critical urban driving scenarios\. Each scenario is instantiated under five different weather and lighting conditions, resulting in a total of 220 evaluation routes\.

Unlike Bench2Drive\[jia2024bench\], where only one autonomous vehicle is simulated and evaluated in each route, CMU\-Drive simultaneously simulates and evaluates 2 to 16 cooperative autonomous vehicles within the same closed\-loop CARLA\[dosovitskiy2017carla\]environment\. Each cooperative vehicle is assigned an individual starting location and destination while sharing the same dynamic traffic environment containing background vehicles, pedestrians, and cyclists\. The first cooperative vehicle follows the original Bench2Drive\[jia2024bench\]route configuration to preserve benchmark consistency and triggers the actions of the safety\-critical background traffic participants following the same conditions configured in Bench2Drive\[jia2024bench\]\. The starting locations and destinations of the additional cooperative vehicles are automatically generated according to the road topology of each scenario\. For example, in junction scenarios, cooperative vehicles are initialized from different entry lanes with corresponding exit destinations, enabling diverse interactions among multiple autonomous agents\. To ensure effective evaluation, we avoid setting the starting location for the additional cooperative driving agents in the lanes that may interfere with the safety\-critical event triggers defined in the original Bench2Drive\[jia2024bench\]benchmark, thereby preserving the intended interactive traffic behaviors and multi\-ability benchmarking\.[Fig\.˜1](https://arxiv.org/html/2608.07621#S2.F1)illustrates examples of CMU\-Drive evaluation routes\.

![Refer to caption](https://arxiv.org/html/2608.07621v1/x1.png)\(a\)Pedestrian crossing from behind a parked vehicle\. Total CAVs: 3\.
![Refer to caption](https://arxiv.org/html/2608.07621v1/x2.png)\(b\)Emergency vehicles running a red light\. Total CAVs: 8\.
![Refer to caption](https://arxiv.org/html/2608.07621v1/x3.png)\(c\)Bicycles crossing the road\. Total CAVs: 12\.
![Refer to caption](https://arxiv.org/html/2608.07621v1/x4.png)\(d\)Vehicles moving straight or making turns in a busy intersection\. Total CAVs: 16\.

Figure 1:Evaluation route samples in CMU\-Drive across different types of safety\-critical scenarios and varying numbers of connected autonomous vehicles \(CAVs\)\. Each solid arrow represents a feasible path from the configured starting location to the destination of a CAV\. Each dashed arrow represents the traffic flow of the configured background traffic participants within each safety\-critical driving scenario\. Each scenario also includes additional non\-cooperative background vehicles driving nearby\.
### 2\.2Multi\-agent Evaluation Metrics

We extend the single\-agent evaluation protocol adopted by CarlaLeaderboard2\.0\[dosovitskiy2017carla\], Bench2Drive\[jia2024bench\], and Simlingo\[Renz2025simlingo\]to support multi\-agent cooperative driving evaluation in CMU\-Drive\. Specifically, for each cooperative autonomous driving agentiiin an evaluation routejjof CMU\-Drive, we compute the agent\-level route completion scoreR​CijRC\_\{i\}^\{j\}and the infraction scoreI​SijIS\_\{i\}^\{j\}\. The route completionR​CijRC\_\{i\}^\{j\}is defined as the percentage of the assigned route completed by agentiiin routejj\. The infraction scoreI​SijIS\_\{i\}^\{j\}is initialized with a base score1\.01\.0and is multiplicatively penalized by a factor whenever the agent commits a driving infraction, including collisions with pedestrians, cyclists, vehicles, or static objects, running red lights or stop signs, driving off\-road, and failing to yield to emergency vehicles\.

Next, for each routejj, the route\-level route completion scoreR​CjRC^\{j\}is defined as the average agent\-level route completion scores of all autonomous driving agents in routejj, and the route\-level infraction scoreI​SjIS^\{j\}is defined as the product of all agent\-level infraction scores\. The route\-level driving scoreD​SjDS^\{j\}is defined as the product of route completion scoreR​CjRC^\{j\}and infraction scoreI​SjIS^\{j\}\. The route\-level success scoreS​SjSS^\{j\}is set to11ifD​Sj=100DS^\{j\}=100, and 0 otherwise\.

R​Cj\\displaystyle RC^\{j\}=1Nj​∑i=1i=NjR​Cij\\displaystyle=\\frac\{1\}\{N^\{j\}\}\\sum\_\{i=1\}^\{i=N^\{j\}\}RC\_\{i\}^\{j\}\(1\)I​Sj\\displaystyle IS^\{j\}=∏i=1i=NjI​Sij\\displaystyle=\\prod\_\{i=1\}^\{i=N^\{j\}\}IS\_\{i\}^\{j\}\(2\)D​Sj\\displaystyle DS^\{j\}=R​Cj⋅I​Sj\\displaystyle=RC^\{j\}\\cdot IS^\{j\}\(3\)S​Sj\\displaystyle SS^\{j\}=\{1,if​D​Sj=1000,otherwise\\displaystyle=\\begin\{cases\}1,\\text\{if \}DS^\{j\}=100\\\\ 0,\\text\{otherwise\}\\end\{cases\}\(4\), whereNjN^\{j\}represents the number of connected autonomous vehicles in routejj\.

Finally, the overall driving scoreD​SDSis defined as the average route\-level driving scoreD​SjDS^\{j\}\. The overall success rateS​RSRis the average route\-level success scoreS​SjSS^\{j\}\.

D​S\\displaystyle DS=1R​∑j=1j=RD​Sj\\displaystyle=\\frac\{1\}\{R\}\\sum\_\{j=1\}^\{j=R\}DS^\{j\}\(5\)S​R\\displaystyle SR=1R​∑j=1j=RS​Sj\\displaystyle=\\frac\{1\}\{R\}\\sum\_\{j=1\}^\{j=R\}SS^\{j\}\(6\), whereR=220R=220is the total number of the evaluation routes in the CMU\-Drive benchmark\.

## 3V2V\-VLA: Vehicle\-to\-Vehicle Vision\-Language\-Action Model for Cooperative Autonomous Driving

We propose V2V\-VLA, a new vehicle\-to\-vehicle vision\-language\-action model for cooperative autonomous driving, as the first baseline model with multi\-agent cooperation in CMU\-Drive, as shown in[Fig\.˜2](https://arxiv.org/html/2608.07621#S3.F2)\. In each frame of the closed\-loop simulation, each cooperative driving agent uses its own V2V\-VLA model to jointly generate driving actions, language reasoning, future waypoints, and an optional communication policy indicating whether another cooperative vehicle should be queried in the next timestep\.

![Refer to caption](https://arxiv.org/html/2608.07621v1/x5.png)Figure 2:V2V\-VLA model architecture\. The input consists of a language prompt, target waypoints, and vision tokens derived from the ego vehicle’s front\-camera image and a merged occupancy map from the ego and the selected communicating vehicle\. The output comprises language outputs for the action, reasoning, and communication, alongside the future waypoints\.### 3\.1Vehicle\-to\-Vehicle Communication Algorithm

For an ego vehicleC​A​Ve​g​oCAV\_\{ego\}, by default it selects its closest front cooperative autonomous vehicle as its communicating vehicleC​A​VcCAV\_\{c\}\. In addition, we also train the V2V\-VLA model to generate the suggested communication target vehicle in the language output whenever there exists a good candidate\. More specifically, if the V2V\-VLA model indicates that another cooperative vehicleC​A​VkCAV\_\{k\}is capable of detecting a critical object while the critical object is invisible to the ego vehicleC​A​Ve​g​oCAV\_\{ego\}, the ego vehicleC​A​Ve​g​oCAV\_\{ego\}will useC​A​VkCAV\_\{k\}as the communicating vehicleC​A​VcCAV\_\{c\}in the next time step, instead of the default choice\.

### 3\.2Model Architecture

[Fig\.˜2](https://arxiv.org/html/2608.07621#S3.F2)illustrates the overall architecture of V2V\-VLA model\. Each cooperative autonomous driving agent is equipped with RGB cameras, IMU, and GPS\. At every timestep, each driving agent feeds the language input, action input, and vision input to its own V2V\-VLA model, and generates the language output on action, reasoning, communication, and future waypoints\.

Language input: Language inputLi​n​p​u​tL\_\{input\}encodes the prompt template requesting the model to describe the driving action that the ego vehicleC​A​Ve​g​oCAV\_\{ego\}should take\. The language input additionally includes the relative locations of other cooperative driving vehicles in the coordinate system ofC​A​Ve​g​oCAV\_\{ego\}\.

Action input: Action inputAi​n​p​u​tA\_\{input\}encodes the target waypoint ofC​A​Ve​g​oCAV\_\{ego\}with a multi\-layer\-perceptron \(MLP\)\.

Vision input: Vision inputVi​n​p​u​tV\_\{input\}includes two parts\. The first one is fromC​A​Ve​g​oCAV\_\{ego\}’s front RGB camera imageIe​g​oI\_\{ego\}\. The second part is a merged camera\-based bird\-eye\-view \(BEV\) occupancy mapOm​e​r​g​e​dO\_\{merged\}\. Specifically, to createOm​e​r​g​e​dO\_\{merged\}, bothC​A​Ve​g​oCAV\_\{ego\}and its selected communicating vehicleC​A​VcCAV\_\{c\}generate their individual camera\-based BEV occupancy mapOe​g​oO\_\{ego\}andOcO\_\{c\}\. Subsequently,OcO\_\{c\}is warpped to generateOc−\>e​g​oO\_\{c\-\>ego\}via coordinate transformation based on the ego vehicle posePe​g​oP\_\{ego\}and communicating vehicle posePcP\_\{c\}\. Finally,Om​e​r​g​e​dO\_\{merged\}is generated by mergingOe​g​oO\_\{ego\}andOc−\>e​g​oO\_\{c\-\>ego\}with the element\-wise\-or operator\.Om​e​r​g​e​dO\_\{merged\}is then renderedas an RGB image, and is fed into a vision encoder, together withIe​g​oI\_\{ego\}to form the vision inputVi​n​p​u​tV\_\{input\}\. UniAD\[hu2023uniad\]is used to generate the camera\-based BEV occupancy maps, and InternViT\[chen2024internvl\]is used as the vision encoder\.

Oc−\>e​g​o\\displaystyle O\_\{c\-\>ego\}=C​o​o​r​d​i​n​a​t​e​T​r​a​n​s​f​o​r​m​\(Oc,Pe​g​o,Pc\)\\displaystyle=CoordinateTransform\(O\_\{c\},P\_\{ego\},P\_\{c\}\)\(7\)Om​e​r​g​e​d\\displaystyle O\_\{merged\}=E​l​e​m​e​n​t​w​i​s​e​O​r​\(Oe​g​o,Oc−\>e​g​o\)\\displaystyle=ElementwiseOr\(O\_\{ego\},O\_\{c\-\>ego\}\)\(8\)Vi​n​p​u​t\\displaystyle V\_\{input\}=V​i​s​i​o​n​E​n​c​o​d​e​r​\(\[Ie​g​o,Om​e​r​g​e​d\]\)\\displaystyle=VisionEncoder\(\[I\_\{ego\},O\_\{merged\}\]\)\(9\)
Vision\-Language\-Action model: V2V\-VLA uses an LLM to take the language input, action input, and vision input to generate the language outputLo​u​t​p​u​tL\_\{output\}and action outputAo​u​t​p​u​tA\_\{output\}\. We use Qwen2\[yang2024qwen2technicalreport\]as the LLM\.

Lo​u​t​p​u​t,Ao​u​t​p​u​t\\displaystyle L\_\{output\},A\_\{output\}=L​L​M​\(Li​n​p​u​t,Ai​n​p​u​t,Vi​n​p​u​t\)\\displaystyle=LLM\(L\_\{input\},A\_\{input\},V\_\{input\}\)\(10\)
Language output: Language outputLo​u​t​p​u​tL\_\{output\}explains the driving decision and identifies safety\-critical objects whenever necessary\. V2V\-VLA also generates reasoning comments related to critical objects invisible to the ego vehicle and the suggested communication target vehicleC​A​VcCAV\_\{c\}whenever there exist\.

Action output: We follow the same approach as in Simlingo\[Renz2025simlingo\]to format the action outputAo​u​t​p​u​tA\_\{output\}, which includes the suggested futureisochronousandequidistantwaypoints\. The waypoints are then used by PID\-Controllers\[emirler2014pid\]to generate the control, including steer, throttle, and brake, forC​A​Ve​g​oCAV\_\{ego\}\.

### 3\.3Training

### 3\.4Training Data Collection

First, we extend a subset of the single\-agent training routes in Simlingo\[Renz2025simlingo\]to build the multi\-agent training routes for our CMU\-Drive with the same approach as we extend the Bench2Drive\[jia2024bench\]single\-agent evaluation routes to our CMU\-Drive multi\-agent evaluation routes\. Then we collect the CMU\-Drive training data by using PDM\-lite\[Beibwenger2024PdmLite\]as the expert driving model in each autonomous driving agent\. PDM\-lite\[Beibwenger2024PdmLite\]is a driving model that uses the ground\-truth location and velocity information of all objects in the CARLA simulation environment to make driving decisions, regardless of whether objects are visible or invisible\.

At each time step, we save the future waypoints of the PDM\-lite agent as the ground\-truth annotation of the action outputAo​u​t​p​u​tA\_\{output\}\. We use a rule\-based approach, similar to DriveLM\[sima2023drivelm\]and Simlingo\[Renz2025simlingo\], to generate the suggested action and the reasoning parts ofLo​u​t​p​u​tL\_\{output\}\. Unlike previous works, we additionally annotate communication supervision\. At each timestep, we identify critical objects that influence the action of the expert driving agent\. If such a critical object is invisible to the ego vehicle but observable by another cooperative vehicle, the annotation recommends that vehicle as the communication partner\. Overall, we generate180​K180Ktraining samples for supervised fine\-tuning\.

### 3\.5Training Details

The training loss includes a smooth\-L1 loss on the future waypoints in the action outputAo​u​t​p​u​tA\_\{output\}and a cross\-entropy loss on the language outputLo​u​t​p​u​tL\_\{output\}\. We start our training by initializing the model weights from Simlingo\[Renz2025simlingo\]and UniAD\[hu2023uniad\]\. We fine\-tune the LoRA\[hu2022lora\]part of the LLM and freeze the UniAD\[hu2023uniad\]model\. And we fine\-tune all other trainable layers in V2V\-VLA for1212epochs, with a batch size88\. For other hyperparameters, we use the same values from Simlingo\[Renz2025simlingo\]\. We use 8 NVIDIA H100\-80GB GPUs for training, and it takes 48 hours\.

## 4Experimental Results

### 4\.1Quantitative Result

[Tab\.˜2](https://arxiv.org/html/2608.07621#S4.T2)summarizes the quantitative experimental result in the220220evaluation routes of CMU\-Drive\. Simlingo\[Renz2025simlingo\]is used as a baseline method that does not involve multi\-agent cooperation\. As shown in[Tab\.˜2](https://arxiv.org/html/2608.07621#S4.T2), our proposed V2V\-VLA achieves a better driving scoreD​SDSand a success rateS​RSRby a large margin\.

Table 2:Quantitative experimental result in the evaluation routes of CMU\-Drive\. DS: Driving Score, SR: Success Rate\.[Fig\.˜4](https://arxiv.org/html/2608.07621#S4.F4)shows the average driving score of V2V\-VLA in each of the4444different scenario types in CMU\-Drive, indicating that several safety\-critical scenarios are still challenging, such asNonSignalizedJunctionLeftTurnEnterFlow, which requires more future research\.

![Refer to caption](https://arxiv.org/html/2608.07621v1/x6.png)Figure 3:Average driving scores of V2V\-VLA in each scenario type of CMU\-Drive evaluation routes\.
![Refer to caption](https://arxiv.org/html/2608.07621v1/x7.png)Figure 4:Qualitative experimental results\.C​A​V0CAV\_\{0\}is unable to observe the leading bicycle clearly due to the long distance and rain\-induced image blur\. Nevertheless,C​A​V0CAV\_\{0\}still decides to reduce its speed by utilizing the perception information shared by the leadingC​A​V4CAV\_\{4\}, which is closer to and capable of detecting the bicycle\.

### 4\.2Qualitative Result

[Fig\.˜4](https://arxiv.org/html/2608.07621#S4.F4)illustrates a sample frame from an evaluation route in CMU\-Drive\. The detection of the leading bicycle by the ego vehicle \(C​A​V0CAV\_\{0\}\) is hindered by the long distance and rain\-induced image blur\. However, the bicycle is successfully identified by a leading connected autonomous vehicle \(C​A​V4CAV\_\{4\}\), which is closer to the hazard\. By utilizing vehicle\-to\-vehicle \(V2V\) communication, the ego vehicle \(C​A​V0CAV\_\{0\}\) safely reduces its speed to prevent a collision\.

## 5Conclusion

In this work, we present CMU\-Drive, a benchmark for cooperative multi\-agent, closed\-loop, end\-to\-end driving with reasoning\. Compared to prior related works, the proposed benchmark supports closed\-loop simulation with a larger number of connected autonomous vehicles and background traffic participants across more complex, safety\-critical driving scenarios, all while requiring fewer GPUs for evaluation\. Furthermore, we propose V2V\-VLA, a novel vehicle\-to\-vehicle vision\-language\-action model for cooperative autonomous driving\. The proposed model jointly generates driving actions, future waypoints, reasoning, and communication policies within a single forward inference\. Our experimental results demonstrate that V2V\-VLA achieves a superior driving score and a higher success rate compared to the baseline method\. To facilitate open\-source research, we will publicly release our code, benchmark, and model checkpoints\.

## References