Rethinking Training & Inference for Forecasting: Linking Winner-Take-All back to GMMs

# Rethinking Training & Inference for Forecasting: Linking Winner-Take-All back to GMMs
Source: [https://arxiv.org/html/2606.26424](https://arxiv.org/html/2606.26424)
11institutetext:1Cornell University,2Stanford University,3Boston University###### Abstract

Trajectory forecasting for autonomous driving has advanced rapidly, yet representative models often produce uninformative posteriors over forecast modes, causing problems for mode pruning\. We trace this to a modeling\-training mismatch: forecasters are typically modeled as conditional Gaussian mixture models \(GMMs\) but trained with a winner\-take\-all \(WTA\) loss that assigns each sample to its nearest mode\. We argue that this K\-means\-like hard assignment \(one\-hot\), while preventing mode collapse, is the source of uninformative mode probabilities: it over\-segments the trajectory space, ignores relatedness among nearby modes, and yields assignment instability under small perturbations\. Guided by this lens, we introduce two post\-hoc treatments: \(1\) test\-time posterior\-weighted merging that aggregates nearby candidate trajectories; and \(2\) a one\-step expectation\-maximization \(EM\) update that replaces hard labels with soft responsibilities, sharing probability mass across neighboring modes\. Across several WTA\-trained architectures, these lightweight steps produce more informative, faithfully ranked mode posteriors and strengthen final forecasts on popular displacement metrics—without retraining\. Our analysis unifies recent design choices through a GMM\-vs\-K\-means perspective and offers principled, practical corrections that better align training objectives with inference\.

## 1Introduction

Trajectory forecasting in autonomous driving, the task of predicting where other agents \(*i\.e\.*, cars, pedestrians, and cyclists\) will go, is critical for planning safe maneuvers for self\-driving vehicles in human\-populated traffic\. There have been numerous industry efforts to establish practical challenges\[[12](https://arxiv.org/html/2606.26424#bib.bib57),[34](https://arxiv.org/html/2606.26424#bib.bib64),[4](https://arxiv.org/html/2606.26424#bib.bib65)\], which in turn inspire learning\-based technical advances\. Fundamental to this is uncertainty estimation, since future motion is inherently multi\-modal and ambiguous\. For instance, a vehicle pausing at a stop sign can either go forward or be ready to turn, and a downstream self\-driving planning stack needs to account for both to ensure a safe action plan\. Beyond generating multiple forecasts, it is also crucial to rank them by likelihood: without a meaningful ordering, an abundance of low\-probability hypotheses can trigger overly conservative behaviors \(*e\.g\.*, abrupt braking\) in otherwise normal situations—both uncomfortable and potentially unsafe\.

![Refer to caption](https://arxiv.org/html/2606.26424v1/x1.png)Figure 1:64 modes predicted by Wayformer\.We visualize a Gaussian mixture of 5 components overlaid from directly predicting the smaller set\. Observe that it suffers from over\-segmentation, with many predictions per mode, despite many of them being unlikely\. The predicted trajectories are colored in black, the ground truth trajectory coloredblue, and the map polylines ingray\.This work focuses on one mainstream model family\[[9](https://arxiv.org/html/2606.26424#bib.bib24),[30](https://arxiv.org/html/2606.26424#bib.bib21),[31](https://arxiv.org/html/2606.26424#bib.bib23),[32](https://arxiv.org/html/2606.26424#bib.bib22),[23](https://arxiv.org/html/2606.26424#bib.bib28),[35](https://arxiv.org/html/2606.26424#bib.bib56)\], which generate multiple candidate trajectory predictions, each with an associated probability\. This design sensibly incorporates uncertainty directly into the predicted futures, and offering multiple candidates gives more flexibility for the subsequent downstream motion planning\. Prior works show that this class of motion forecasting models is indeed capable of producing future trajectory candidates that cover most of the true trajectories—often capturing even outliers\[[3](https://arxiv.org/html/2606.26424#bib.bib66)\]\. In practice, the model often predicts an initial set of over\-complete trajectories \(usually 64\) and adopts a post\-hoc mode sampling or aggregation process to obtain the final compact set of trajectories for downstream motion planning of the ego\-vehicle\. Today, this setting has become the de\-facto benchmark evaluation, as it balances the flexibility of obtaining a compact and tractable subset for real\-time deployment while providing a better coverage of safety\-critical possible future motions\. The bottleneck, however, is in the assigned probability to each candidate: we find that, while the prediction does model an associated probabilistic mixture weight for each predicted trajectory, the assigned probability is often uninformative,*e\.g\.*\., assuming low likelihood for candidates that are common and correct future behaviors, or fluctuating probabilities assigned to similar candidates\. Thus, a naive post\-hoc selection methods such as selecting the highest “likelihood” modes may yield trajectories that retain only a small fraction of the total predictive likelihood\.

This work takes a classical machine learning approach to answer the question:*What is a principled way of obtaining a compact set of trajectories despite uninformative predicted likelihoods*? To approach this, we begin by exploring why the likelihoods predicted under a broad swath of motion forecasting objectives\[[9](https://arxiv.org/html/2606.26424#bib.bib24),[33](https://arxiv.org/html/2606.26424#bib.bib25),[30](https://arxiv.org/html/2606.26424#bib.bib21),[32](https://arxiv.org/html/2606.26424#bib.bib22),[31](https://arxiv.org/html/2606.26424#bib.bib23),[23](https://arxiv.org/html/2606.26424#bib.bib28)\]are often uninformative\. Our investigations show a fundamental mismatch between their training objective and the assumed inference\-time probabilistic model\. To capture the uncertainty and diversity of future trajectories, these representative algorithms adopt a Gaussian mixture model \(GMM\) conditioned on the past trajectory\. The standard objective to learn GMM is the evidence lower bound \(ELBO\), which can be approximated by an Expectation\-Maximization \(EM\) style training—interleaving between assigning each sample*softly*to modes, and updating the candidate predictions and probability assignments\. However, such an objective is intractable over a large dataset, and directly optimizing with gradient descent often yields collapsed modes, which is especially undesirable\. In practice, these methods adopt a winner\-take\-all \(WTA\) assignment, where the soft assignment to modes is replaced with a hard assignment, thus preventing the collapse of all modes towards the single sample update\. While this simple change effectively enables the diverse predictions, our key insight is that such a hard assignment changes the objective to a K\-Means styled objective\[[19](https://arxiv.org/html/2606.26424#bib.bib60)\], thus losing its probabilistic interpretation\. This insight sheds light on why the predicted mixture likelihoods are often uninformative, and why models trained under such objective are able to effectively predict trajectories that specialize without mode collapse\.

Indeed, we argue that the winner\-take\-all objective has many merits, as it inherits the strengths of K\-Means predictions, and is capable of capturing the diverse behaviors necessary in motion forecasting methods\. This understanding also sheds light on why these models tend to over\-segment —*i\.e\.*, generate many predictions over— a single intention mode, such as “move\-forward”\. In section[4](https://arxiv.org/html/2606.26424#S4), we analyze the objective to demonstrate the perspective that the WTA objective maps closely to a K\-Means clustering objective\. With this insight, we propose to use an aggregation and single\-step expectation maximization in section[5](https://arxiv.org/html/2606.26424#S5)to mitigate the over\-segmentation characteristic and to replace hard labels with soft responsibilities for more informative, faithfully ranked mode posteriors\. Our key contribution are providing theoretical groundings and interpretations for this class of methods: why we need them, and how they work\. Finally, in section[6](https://arxiv.org/html/2606.26424#S6)we demonstrate that our simple treatments improve the distance error prediction performance over naive greedy selection over the predicted trajectory likelihoods without the need for retraining\. Our contributions are as follows:

- •We offer a unified GMM\-vs\-K\-means perspective on the winner\-take\-all trajectory forecasting objective\.
- •We offer a principled, practical correction to align the training objective with inference\-time probabilistic modeling assumptions\.
- •We empirically verify our findings with three representative winner\-take\-all models on post\-hoc aggregation strategies\.

## 2Related Works

#### Motion Forecasting for Self\-Driving\.

Motion forecasting has been a longstanding problem in self\-driving research\. The learning\-based methods primarily formulate it as a multi\-agent sequence prediction problem, where given some history of the agent’s \(*i\.e\.*another vehicle or a pedestrian\) location, the goal is to predict the future locations\[[2](https://arxiv.org/html/2606.26424#bib.bib2),[20](https://arxiv.org/html/2606.26424#bib.bib3)\]\. Early approaches predict one or more waypoints trajectories of the actors of interest\[[1](https://arxiv.org/html/2606.26424#bib.bib4),[24](https://arxiv.org/html/2606.26424#bib.bib7),[15](https://arxiv.org/html/2606.26424#bib.bib16)\]\. Other works explored how to model uncertainty either as independent Gaussian distributions over future timesteps\[[26](https://arxiv.org/html/2606.26424#bib.bib10),[7](https://arxiv.org/html/2606.26424#bib.bib6)\], or as non\-parametric occupancy predictions\[[16](https://arxiv.org/html/2606.26424#bib.bib14),[17](https://arxiv.org/html/2606.26424#bib.bib13),[21](https://arxiv.org/html/2606.26424#bib.bib20)\]\. Another branch of works aimed at identifying intentions\[[36](https://arxiv.org/html/2606.26424#bib.bib9),[18](https://arxiv.org/html/2606.26424#bib.bib19),[27](https://arxiv.org/html/2606.26424#bib.bib11),[14](https://arxiv.org/html/2606.26424#bib.bib8)\], either as the end goal or as a co\-objective towards multimodal motion forecasting\. These works often obtain very diverse, context\-dependent behaviors, but are slow to run in real\-time\. To handle the necessity of fast runtime forward passes, methods opted for implicit likelihoods to model intention as sampling\[[15](https://arxiv.org/html/2606.26424#bib.bib16),[29](https://arxiv.org/html/2606.26424#bib.bib18),[8](https://arxiv.org/html/2606.26424#bib.bib12),[10](https://arxiv.org/html/2606.26424#bib.bib17)\]\. However, the lack of an explicit likelihood makes these models hard to estimate uncertainty that is needed for downstream ego\-vehicle planning\. A few works instead leverage an explicit multimodal Gaussian mixture model \(GMM\) representation to predict implicit intentions as mixtures, and directly regress timestep\-independent Gaussian future predictions\[[11](https://arxiv.org/html/2606.26424#bib.bib15),[9](https://arxiv.org/html/2606.26424#bib.bib24),[8](https://arxiv.org/html/2606.26424#bib.bib12),[33](https://arxiv.org/html/2606.26424#bib.bib25),[30](https://arxiv.org/html/2606.26424#bib.bib21),[32](https://arxiv.org/html/2606.26424#bib.bib22)\]\. These yield interpretable uncertainties, but are often uncalibrated in practice\[[6](https://arxiv.org/html/2606.26424#bib.bib29)\]\. In the multimodal, GMM\-based approaches, popularized by\[[9](https://arxiv.org/html/2606.26424#bib.bib24)\], many works optimize their Gaussian Mixture models using a diversity\-inducing loss, where the likelihoods are trained with a winner\-take\-all objective\[[33](https://arxiv.org/html/2606.26424#bib.bib25),[30](https://arxiv.org/html/2606.26424#bib.bib21),[32](https://arxiv.org/html/2606.26424#bib.bib22),[23](https://arxiv.org/html/2606.26424#bib.bib28)\]\. This method of optimization yields diverse, multimodal predictions and stable convergences\.

#### Post\-hoc Trajectory Selection

Many motion predictors, such as MTR\[[30](https://arxiv.org/html/2606.26424#bib.bib21)\], Multipath\+\+\[[33](https://arxiv.org/html/2606.26424#bib.bib25)\], and Wayformer\[[23](https://arxiv.org/html/2606.26424#bib.bib28)\], standardly employ 64 initial modes \(K=64K=64\) for a better coverage of possible trajectories, and adopt a mode sampling or aggregation process to obtain the finalMMmodes for benchmark evaluation \(M≪KM\\ll K, usually 5 or 6\)\. This configuration is argued to be computationally tractable for the predictor while providing a sufficiently over\-complete set to enable the stable learning of diverse motion patterns\. Table[1](https://arxiv.org/html/2606.26424#S2.T1)summarizes the training objective and post\-hoc mode selection method of some models\. These post\-hoc methods allow practitioners to modify or down\-select trajectory modes without retraining, providing greater test\-time flexibility in the output representation\. However, despite their practical importance, there is currently no unified framework or systematic analysis that connects these post\-hoc trajectory selection/aggregation procedures with the underlying training objectives\.

Table 1:Training objective and post\-hoc method of motion prediction models\.

## 3Preliminaries

### 3\.1Motion Forecasting

Motion forecasting for autonomous driving requires predicting multiple plausible future trajectories of surrounding agents\. Key desiderata include: \(1\) multimodality to capture diverse behavioral intentions, \(2\) accurate likelihood estimates for uncertainty\-aware planning, and \(3\) computational efficiency for real\-time deployment\.

#### Problem Formulation\.

Given history states of all agents and scene map context, predict the target agent’s future trajectory in the form ofMMcandidates𝐲^m\(t\),t∈\{1,…,T\},m∈\{1,…,M\}\\hat\{\\mathbf\{y\}\}\_\{m\}^\{\(t\)\},t\\in\\\{1,\.\.\.,T\\\},m\\in\\\{1,\.\.\.,M\\\}over a prediction horizonTT\. Each timestep is predicted independently, conditioned on the inputs and the selected mode, so the model factorizes the joint prediction across time\. In this paper, we assume all predictions are conditioned on inputs \(which are dropped in the equations for simplicity\), and drop the datapoint index\.

#### GMM Representation\.

Gaussian Mixture Models \(GMMs\) naturally address many of these requirements and have become a popular choice for modeling future trajectories in motion forecasting\[[30](https://arxiv.org/html/2606.26424#bib.bib21),[32](https://arxiv.org/html/2606.26424#bib.bib22),[23](https://arxiv.org/html/2606.26424#bib.bib28)\]\. GMMs provide multimodality through mixture components representing distinct behavioral modes, principled likelihood estimates for uncertainty\-aware planning, and a compact parametric form that is efficient for training and inference\. These advantages have led many state\-of\-the\-art motion forecasting methods to adopt GMM\-based output representations, making them a de facto standard in the field\. Formally, givenKKmixture components, the GMM negative log\-likelihood objective is:

ℒGMM=−log​∑k=1Kp^k⋅∏t=1T𝒩​\(𝐲∗\(t\)\|𝐲^k\(t\),Σk\(t\)\)\\mathcal\{L\}\_\{\\text\{GMM\}\}=\-\\log\\sum\_\{k=1\}^\{K\}\\hat\{p\}\_\{k\}\\cdot\\prod\_\{t=1\}^\{T\}\\mathcal\{N\}\(\\mathbf\{y\}^\{\*\(t\)\}\|\\hat\{\\mathbf\{y\}\}\_\{k\}^\{\(t\)\},\\Sigma\_\{k\}^\{\(t\)\}\)\(1\)whereTTis the prediction horizon,𝐲∗\(t\)\\mathbf\{y\}^\{\*\(t\)\}is the ground truth at timett,𝐲^k\(t\)\\hat\{\\mathbf\{y\}\}\_\{k\}^\{\(t\)\}denotes thekk\-th predicted trajectory at timett,Σk\\Sigma\_\{k\}the covariance \(often simplified to a fixed isotropic covariance\), andp^k\\hat\{p\}\_\{k\}its mixture weight\.

#### Post\-hoc Trajectory Selection

In real\-world applications, there exists a tradeoff between more modes and the computation constraints: adding modes yields lower minimum displacement errors \(recall\) but at the cost of computation latency, a sensitivity of real\-time applications\. Previous works from Luo et al\., and Roh et al\.\[[22](https://arxiv.org/html/2606.26424#bib.bib67),[28](https://arxiv.org/html/2606.26424#bib.bib68)\]emphasize that incorporating multi\-modal predictions from multiple agents substantially increases planning complexity, often requiring mode pruning or reduction to meet real\-time constraints\. Additionally, training the predictor itself requires generating a large number of initial trajectory candidates\. Both\[[30](https://arxiv.org/html/2606.26424#bib.bib21)\]and\[[23](https://arxiv.org/html/2606.26424#bib.bib28)\]employ 64 initial trajectories, arguing that this number enables stable learning of diverse motion patterns while remaining computationally tractable, with downstream tasks subsequently selecting or pruning the topMMtrajectories based on their specific constraints\. Therefore, we consider a setting in which the predictor produces an over\-complete set ofKKcandidate trajectories, while only a limited subset ofMMcan be feasibly utilized for final evaluation\.

#### Evaluation Metrics\.

The motion forecasting community has developed several metrics to evaluate these desiderata\[[15](https://arxiv.org/html/2606.26424#bib.bib16),[7](https://arxiv.org/html/2606.26424#bib.bib6),[5](https://arxiv.org/html/2606.26424#bib.bib58),[9](https://arxiv.org/html/2606.26424#bib.bib24),[12](https://arxiv.org/html/2606.26424#bib.bib57)\]\. The most common displacement\-based metrics are minimum Average Displacement Error \(minADE\) and minimum Final Displacement Error \(minFDE\), which compute the L2 distance between the ground truth and the closest prediction from a set ofMMtrajectories:

minADE=minm∈\{1,…,M\}⁡1T​∑t=1T‖𝐲^m\(t\)−𝐲∗\(t\)‖2\\displaystyle=\\min\_\{m\\in\\\{1,\.\.\.,M\\\}\}\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\\|\\hat\{\\mathbf\{y\}\}\_\{m\}^\{\(t\)\}\-\\mathbf\{y\}^\{\*\(t\)\}\\\|\_\{2\}\(2\)minFDE=minm∈\{1,…,M\}⁡‖𝐲^m\(T\)−𝐲∗\(T\)‖2\\displaystyle=\\min\_\{m\\in\\\{1,\.\.\.,M\\\}\}\\\|\\hat\{\\mathbf\{y\}\}\_\{m\}^\{\(T\)\}\-\\mathbf\{y\}^\{\*\(T\)\}\\\|\_\{2\}\(3\)where𝐲^m\(t\)\\hat\{\\mathbf\{y\}\}\_\{m\}^\{\(t\)\}denotes themm\-th predicted trajectory at timett,𝐲∗\(t\)\\mathbf\{y\}^\{\*\(t\)\}is the ground truth at timett, andTTis the prediction horizon\. The Waymo Motion Prediction Challenge\[[12](https://arxiv.org/html/2606.26424#bib.bib57)\]introduced Miss Rate, which computes the ratio of scenes where all predictions exceed a scenario\-based final displacement error\.

### 3\.2Winner\-Take\-All Objective

Motion forecasting models ideally aim to learn a distribution over future trajectories, commonly approximated as a Gaussian Mixture Model \(GMM\)\. However, directly optimizing the GMM objective often leads to mode collapse, where the model learns to predict nearly identical trajectories across all mixture components to minimize the loss for any given ground truth\. This degeneracy defeats the purpose of multimodal prediction, as the model fails to capture the diversity of plausible future behaviors\. Moreover, GMM objective requires iterative EM training, which is computationally heavy\.

To address these issues, practitioners have adopted the winner\-take\-all \(WTA\) training paradigm\. Introduced in\[[9](https://arxiv.org/html/2606.26424#bib.bib24)\], instead of updating all predictions based on their likelihood under the ground truth, WTA selects only the best prediction and updates its parameters:

ℒWTA=−∑i=1n∑k=1K𝟙\{k=arg​minj⁡‖𝐲^j−𝐲∗‖F\}​\[log⁡p^k\+∑t=1Tlog⁡𝒩​\(𝐲∗\(t\)\|𝐲^k\(t\),Σk\(t\)\)\]\\mathcal\{L\}\_\{\\text\{WTA\}\}=\-\\sum\_\{i=1\}^\{n\}\\sum\_\{k=1\}^\{K\}\\mathbbm\{1\}\_\{\\\{k=\\operatorname\*\{arg\\,min\}\_\{j\}\\\|\\hat\{\\mathbf\{y\}\}\_\{j\}\-\\mathbf\{y\}^\{\*\}\\\|\_\{F\}\\\}\}\\Big\[\\log\\hat\{p\}\_\{k\}\+\\sum\_\{t=1\}^\{T\}\\log\\mathcal\{N\}\(\\mathbf\{y\}^\{\*\(t\)\}\|\\hat\{\\mathbf\{y\}\}\_\{k\}^\{\(t\)\},\\Sigma\_\{k\}^\{\(t\)\}\)\\Big\]\(4\)This objective encourages diverse predictions by ensuring that only one component needs to explain each training sample, preventing the collapse to a single mode, and is thus popular in various methods in motion forecasting that output GMM representations\[[30](https://arxiv.org/html/2606.26424#bib.bib21),[23](https://arxiv.org/html/2606.26424#bib.bib28),[32](https://arxiv.org/html/2606.26424#bib.bib22),[35](https://arxiv.org/html/2606.26424#bib.bib56)\]\. While this approach successfully maintains prediction diversity, we observe that it fundamentally changes the nature of what the model learns — a mismatch we analyze in detail in section[4](https://arxiv.org/html/2606.26424#S4)\.

## 4What is the WTA Objective Doing?

![Refer to caption](https://arxiv.org/html/2606.26424v1/figures/gmm-toy-example_2.png)Figure 2:Minimal 2D Example\.To demonstrate how the winner\-takes\-all objective influences the final predicted modes and probability, we construct a 3\-mode Gaussian mixture \(a\), and fit a GMM with gradient descent on the parameters \(b\) and with the WTA objective \(d\)\. Observe that the parameter estimates exhibit mode collapse; while the WTA objective covers diverse modes but over\-segments the space, which is identical to the K\-means result \(c\)\. We propose to update the mixture weights with a single EM step \(e\) or use weighted cluster merging \(f\), which yields more valid assignments\. Notice in \(d\)p^1\\hat\{p\}\_\{1\},p^4\\hat\{p\}\_\{4\}, andp^5\\hat\{p\}\_\{5\}are the top 3, causing a miss on the top true Gaussian; In \(e\), the top 3 clusters are nowp^1\\hat\{p\}\_\{1\},p^3\\hat\{p\}\_\{3\}, andp^4\\hat\{p\}\_\{4\}, covering all 3 true Gaussians\.In this work, we argue that WTA is actually a hybrid of GMM and K\-Means clustering\. We aim to study the winner\-take\-all objective that is prevalent in training many GMM outputs of prediction models\[[9](https://arxiv.org/html/2606.26424#bib.bib24),[33](https://arxiv.org/html/2606.26424#bib.bib25),[30](https://arxiv.org/html/2606.26424#bib.bib21),[32](https://arxiv.org/html/2606.26424#bib.bib22),[23](https://arxiv.org/html/2606.26424#bib.bib28),[31](https://arxiv.org/html/2606.26424#bib.bib23)\]\.

### 4\.1Revisiting K\-Means Clustering

Recall that the K\-means clustering objective is to learn a partition over the observations into clusters in which each observation belongs to the cluster with the nearest mean, or cluster exemplar\[[19](https://arxiv.org/html/2606.26424#bib.bib60)\]\. The objective is to minimize the total variance within the cluster\. Formally, given a set of observations\(𝐱1,𝐱2,⋯,𝐱n\)\(\\mathbf\{x\}\_\{1\},\\mathbf\{x\}\_\{2\},\\cdots,\\mathbf\{x\}\_\{n\}\), the objective is to findKKsets𝐒=\{S1,⋯,SK\}\\mathbf\{S\}=\\\{S\_\{1\},\\cdots,S\_\{K\}\\\}:

arg​min𝐒​∑k=1K∑𝐱∈𝐒𝐤‖𝐱−𝝁k‖2,\\operatorname\*\{arg\\,min\}\_\{\\mathbf\{S\}\}\\sum\_\{k=1\}^\{K\}\\sum\_\{\\mathbf\{x\\in S\_\{k\}\}\}\\\|\\mathbf\{x\}\-\\bm\{\\mu\}\_\{k\}\\\|^\{2\},\(5\)where each centroid𝝁k\\bm\{\\mu\}\_\{k\}is the mean of the clusterkk,*i\.e\.*𝝁k=1\|Sk\|​∑𝐱∈𝐒𝐤𝐱\\bm\{\\mu\}\_\{k\}=\\frac\{1\}\{\|S\_\{k\}\|\}\\sum\_\{\\mathbf\{x\\in S\_\{k\}\}\}\\mathbf\{x\}\.

Observe that we can rewrite this objective as an iteration over the data points instead,

arg​min𝐒​∑i=1n∑k=1K𝟙\{k=arg​minj⁡‖𝐱−𝝁k‖\}​‖𝐱−𝝁k‖2,\\operatorname\*\{arg\\,min\}\_\{\\mathbf\{S\}\}\\sum\_\{i=1\}^\{n\}\\sum\_\{k=1\}^\{K\}\\mathbbm\{1\}\_\{\\\{k=\\operatorname\*\{arg\\,min\}\_\{j\}\\\|\\mathbf\{x\}\-\\bm\{\\mu\}\_\{k\}\\\|\\\}\}\\\|\\mathbf\{x\}\-\\bm\{\\mu\}\_\{k\}\\\|^\{2\},\(6\)where the indicator𝟙\{k=arg​minj⁡‖𝐱−𝝁k‖\}\\mathbbm\{1\}\_\{\\\{k=\\operatorname\*\{arg\\,min\}\_\{j\}\\\|\\mathbf\{x\}\-\\bm\{\\mu\}\_\{k\}\\\|\\\}\}assigns the points to a cluster if it is closest to its centroid\. This can be viewed as optimizing the minimum distances across all datapoints within a cluster\. Typically, this objective is maximized using an iterative refinement technique\[[19](https://arxiv.org/html/2606.26424#bib.bib60)\], separating the cluster assignment step from the centroid update\.

### 4\.2Mapping WTA to K\-Means Objective

We hypothesize that the winner\-take\-all \(WTA\) objective’s hard minimum assignment exhibits similar behaviors to K\-means objective\. Observe that we can decompose the WTA objective in[Equation 4](https://arxiv.org/html/2606.26424#S3.E4)into the mixture weight likelihood term and the Gaussian component regression term:

ℒW​T​A=−∑i=1n\(∑k=1K𝟙\{⋅\}​log⁡p^k⏟Mode cross\-entropyℒc​l​a​s​s\+∑k=1K𝟙\{⋅\}​∑t=1Tlog⁡𝒩​\(𝐲∗\(t\)\|𝐲^k\(t\),Σk\(t\)\)⏟Mode Gaussian parameter regressionℒr​e​g\)\\mathcal\{L\}\_\{WTA\}=\-\\sum\_\{i=1\}^\{n\}\\Big\(\\underbrace\{\\sum\_\{k=1\}^\{K\}\\mathbbm\{1\}\_\{\\\{\\cdot\\\}\}\\log\\hat\{p\}\_\{k\}\}\_\{\\text\{Mode cross\-entropy $\\mathcal\{L\}\_\{class\}$\}\}\+\\underbrace\{\\sum\_\{k=1\}^\{K\}\\mathbbm\{1\}\_\{\\\{\\cdot\\\}\}\\sum\_\{t=1\}^\{T\}\\log\\mathcal\{N\}\(\\mathbf\{y\}^\{\*\(t\)\}\|\\hat\{\\mathbf\{y\}\}\_\{k\}^\{\(t\)\},\\Sigma\_\{k\}^\{\(t\)\}\)\}\_\{\\text\{Mode Gaussian parameter regression $\\mathcal\{L\}\_\{reg\}$\}\}\\Big\)\(7\)
Looking only at the regression term for a particular data pointii, observe that

𝐲^∗\\displaystyle\\hat\{\\mathbf\{y\}\}^\{\*\}=arg​min𝐲^​∑k=1K𝟙\{⋅\}​∑t=1T−log⁡𝒩​\(𝐲∗\(t\)\|𝐲^k\(t\),Σk\(t\)\)\\displaystyle=\\operatorname\*\{arg\\,min\}\_\{\\hat\{\\mathbf\{y\}\}\}\\sum\_\{k=1\}^\{K\}\\mathbbm\{1\}\_\{\\\{\\cdot\\\}\}\\sum\_\{t=1\}^\{T\}\-\\log\\mathcal\{N\}\(\\mathbf\{y\}^\{\*\(t\)\}\|\\hat\{\\mathbf\{y\}\}\_\{k\}^\{\(t\)\},\\Sigma\_\{k\}^\{\(t\)\}\)=arg​min𝐲^​∑k=1K𝟙\{⋅\}​∑t=1T‖\(Σk\(t\)\)−1/2​\(𝐲∗\(t\)−𝐲^k\(t\)\)‖2\\displaystyle=\\operatorname\*\{arg\\,min\}\_\{\\hat\{\\mathbf\{y\}\}\}\\sum\_\{k=1\}^\{K\}\\mathbbm\{1\}\_\{\\\{\\cdot\\\}\}\\sum\_\{t=1\}^\{T\}\\\|\(\\Sigma\_\{k\}^\{\(t\)\}\)^\{\-1/2\}\(\\mathbf\{y\}^\{\*\(t\)\}\-\\hat\{\\mathbf\{y\}\}\_\{k\}^\{\(t\)\}\)\\\|^\{2\}=arg​min𝐲^​∑k=1K𝟙\{⋅\}​‖\(Σk\)−1/2​\(𝐲∗−𝐲^k\)‖F2\\displaystyle=\\operatorname\*\{arg\\,min\}\_\{\\hat\{\\mathbf\{y\}\}\}\\sum\_\{k=1\}^\{K\}\\mathbbm\{1\}\_\{\\\{\\cdot\\\}\}\\\|\(\\Sigma\_\{k\}\)^\{\-1/2\}\(\\mathbf\{y\}^\{\*\}\-\\hat\{\\mathbf\{y\}\}\_\{k\}\)\\\|\_\{F\}^\{2\}=arg​min𝐲^​∑k=1K𝟙\{⋅\}​DM​\(𝐲∗,𝐲^k;Σk\)2,\\displaystyle=\\operatorname\*\{arg\\,min\}\_\{\\hat\{\\mathbf\{y\}\}\}\\sum\_\{k=1\}^\{K\}\\mathbbm\{1\}\_\{\\\{\\cdot\\\}\}D\_\{M\}\(\\mathbf\{y\}^\{\*\},\\hat\{\\mathbf\{y\}\}\_\{k\};\\Sigma\_\{k\}\)^\{2\},where𝐲∗\\mathbf\{y\}^\{\*\}and𝐲^k\\hat\{\\mathbf\{y\}\}\_\{k\}—with a slight abuse of notation—are the timestep\-stacked ground\-truth and prediction trajectory, andΣk\\Sigma\_\{k\}is the block diagonal stacked covariances at sampleii\. The condition of the indicator is from[Equation 4](https://arxiv.org/html/2606.26424#S3.E4)and dropped for brevity; it is the indicator for the closest prediction to the data point according to the Frobenius norm\.DMD\_\{M\}is the Mahalanobis distance under the Gaussian distribution of each modekk\.

Setting the gradient of WTA objective function with respect to𝐲^k\\hat\{\\mathbf\{y\}\}\_\{k\}to zero yields

∂ℒW​T​A∂𝐲^k=−2​Σk−1​∑i∈𝒞k\(𝐲i∗−𝐲^k\)=𝟎\\frac\{\\partial\\mathcal\{L\}\_\{WTA\}\}\{\\partial\\hat\{\\mathbf\{y\}\}\_\{k\}\}=\-2\\,\\Sigma\_\{k\}^\{\-1\}\\sum\_\{i\\in\\mathcal\{C\}\_\{k\}\}\\bigl\(\\mathbf\{y\}\_\{i\}^\{\*\}\-\\hat\{\\mathbf\{y\}\}\_\{k\}\\bigr\)=\\mathbf\{0\}\(8\)The optimal𝐲^k\\hat\{\\mathbf\{y\}\}\_\{k\}is exactly the mean of the cluster:𝐲^k∗=1Nk​∑i∈𝒞k𝐲i∗\\hat\{\\mathbf\{y\}\}\_\{k\}^\{\*\}=\\frac\{1\}\{N\_\{k\}\}\\sum\_\{i\\in\\mathcal\{C\}\_\{k\}\}\\mathbf\{y\}\_\{i\}^\{\*\}\. Here𝒞k=\{i:arg​minj⁡‖𝐲i∗−𝐲^j‖F=k\}\\mathcal\{C\}\_\{k\}=\\\{i:\\operatorname\*\{arg\\,min\}\_\{j\}\\\|\\mathbf\{y\}\_\{i\}^\{\*\}\-\\hat\{\\mathbf\{y\}\}\_\{j\}\\\|\_\{F\}=k\\\}withNk=\|𝒞k\|N\_\{k\}=\|\\mathcal\{C\}\_\{k\}\|,Σk≻0\\Sigma\_\{k\}\\succ 0\.The identical optimal prediction trajectory suggests an equivalence between WTA and K\-means\.Such a re\-mapping to the K\-means objective yields interesting insights into why the WTA objective exhibits behaviors that are mode covering, but often over\-segments the space\.

To demonstrate intuitively what the WTA objective optimizes as compared to the GMM model, we turn to a minimal example in Fig\.[2](https://arxiv.org/html/2606.26424#S4.F2), where we look at a simple 2\-dimensional case\. Observe that while the WTA objective \(Fig\.[2](https://arxiv.org/html/2606.26424#S4.F2)d\) does not exhibit mode collapse, like in the GMM representation \(Fig\.[2](https://arxiv.org/html/2606.26424#S4.F2)b\), it often breaks up modes and oversegments the space, which is characteristic of K\-Means clustering \(Fig\.[2](https://arxiv.org/html/2606.26424#S4.F2)c\)\. We analyze the consequences of this behavior in the following section\. In section[5](https://arxiv.org/html/2606.26424#S5), we introduce post\-hoc treatments to mitigate both failure modes — replacing hard assignments with soft responsibilities \(Fig\.[2](https://arxiv.org/html/2606.26424#S4.F2)e\) and merging oversegmented predictions into coherent modes \(Fig\.[2](https://arxiv.org/html/2606.26424#S4.F2)f\)\.

### 4\.3Consequences of Hard Assignment

The equivalence between WTA and K\-means clustering has two concrete negative consequences for trajectory forecasting:

#### Oversegmentation\.

When the number of learned modesKKexceeds the number of true behavioral modes, hard assignment encourages the model to partition a single true mode into multiple learned modes, each claiming exclusive ownership of a subset of training samples\. This is the characteristic oversegmentation behavior of K\-means: rather than concentrating multiple mode anchors on a single high\-density region, it spreads them across an artificially fragmented partition of that region \(Fig\.[2](https://arxiv.org/html/2606.26424#S4.F2)c\)\. The result is an over\-complete set of predicted trajectories that fragments the plausible trajectory space beyond what is semantically meaningful\. This characteristic oversegmentation is visualized in Fig\.[1](https://arxiv.org/html/2606.26424#S1.F1), where any single intention\-mode \(colored by Gaussian mixtures\) is fragmented across many predicted modes\.

#### Uninformative Mode Probabilities\.

Oversegmentation directly undermines the quality of predicted mode probabilities\. A single high\-probability true mode — say, “turn right” — may be represented by many learned modes, each capturing only a fraction of the associated training samples \(Fig\.[1](https://arxiv.org/html/2606.26424#S1.F1)\)\. Consequently, each individual learned mode accumulates only a small share of the probability mass, even though their union corresponds to a highly likely future behavior\. When the top\-KKpredictions are selected at inference time, these fragmented modes are each individually ranked as low\-probability, and may be suppressed or ignored — despite collectively representing a dominant intention\. This mismatch between individual mode probability and true behavioral likelihood is the source of the uninformative posteriors we observe in practice\.

## 5Proposals for Prediction Selection

### 5\.1Test\-Time Merging

When the model’s number of modes exceeds the number of components of the true distribution, the WTA objective tends to break up modes and over\-segment the space \(Fig\.[2](https://arxiv.org/html/2606.26424#S4.F2)d\)\. Merging the broken\-up clusters can recover the true distribution components, if clusters are grouped correctly \(Fig\.[2](https://arxiv.org/html/2606.26424#S4.F2)f\)\.

Therefore, we suggest a test\-time merging method based on weighted K\-means clustering to be used for models trained with WTA objective, which is simpler compared to the existing similar implementations in Wayformer\[[23](https://arxiv.org/html/2606.26424#bib.bib28)\]and MultiPath\+\+\[[33](https://arxiv.org/html/2606.26424#bib.bib25)\]\. In detail, it adopts an iterative process like K\-means clustering\. Given initially predicted trajectories𝐲^k,k∈\{1,…,K\}\\hat\{\\mathbf\{y\}\}\_\{k\},k\\in\\\{1,\.\.\.,K\\\}and we wantMMcluster centers𝐲¯m,m∈\{1,…,M\},M<K\\bar\{\\mathbf\{y\}\}\_\{m\},m\\in\\\{1,\.\.\.,M\\\},M<Kas our final output\. In each iteration, 1\) Assign initially predicted trajectories𝐲^k\\hat\{\\mathbf\{y\}\}\_\{k\}to the closest cluster \(Euclidean distance of the final waypoints‖𝐲^k\(T\)−𝐲¯m\(T\)‖\\\|\\hat\{\\mathbf\{y\}\}\_\{k\}^\{\(T\)\}\-\\bar\{\\mathbf\{y\}\}\_\{m\}^\{\(T\)\}\\\|\); 2\) Recompute cluster centers using the weighted mean formula:𝐲¯m=∑k∈Cmpk^​𝐲^k∑k∈Cmpk^\\bar\{\\mathbf\{y\}\}\_\{m\}=\\frac\{\\sum\_\{k\\in C\_\{m\}\}\\hat\{p\_\{k\}\}\\hat\{\\mathbf\{y\}\}\_\{k\}\}\{\\sum\_\{k\\in C\_\{m\}\}\\hat\{p\_\{k\}\}\}; 3\) Repeat until cluster centers stabilize\. The stabilized𝐲¯m\\bar\{\\mathbf\{y\}\}\_\{m\}is the final output\.

In the previous section, we have shown that this merging method theoretically aligns with the problem and targets exactly at the issues of WTA\. We also demonstrate the effectiveness of this merging method through experiments\.

### 5\.2One\-Step Expectation\-Maximization

Since the trajectories predicted under the WTA objective exhibit diverse, but yield uninformative mode prediction likelihoods, we can instead apply a post\-processing step to convert the final modeled outputs into a true mixture model with a Bayesian update on the predicted likelihoods\. We note that we can now turn to one step of expectation\-maximization \(EM\) to run another iteration of training, replacing the hard label assignments during training with soft responsibilities\. Specifically, the EM steps follow from the standard GMM updates, with the E\-step:

wk:=p​\(k\|𝐲∗\)=p​\(𝐲∗\|𝐲^k,Σk\)⋅p^k∑z=1Kp​\(𝐲∗\|𝐲^z,Σz\)⋅p^z,w\_\{k\}:=p\(k\|\\mathbf\{y\}^\{\*\}\)=\\frac\{p\(\\mathbf\{y\}^\{\*\}\|\\hat\{\\mathbf\{y\}\}\_\{k\},\\Sigma\_\{k\}\)\\cdot\\hat\{p\}\_\{k\}\}\{\\sum\_\{z=1\}^\{K\}p\(\\mathbf\{y\}^\{\*\}\|\\hat\{\\mathbf\{y\}\}\_\{z\},\\Sigma\_\{z\}\)\\cdot\\hat\{p\}\_\{z\}\},\(9\)where the likelihoods are obtained from our trained motion forecasting model, parameterized as a Gaussian\.

We select to compute the M\-step update over the training dataset maximizing the data distribution \(minimizing the negative log likelihoodℒ\\mathcal\{L\}\):

ℒ=−∑i=1nlog⁡p​\(𝐲∗\)=∑i=1n∑k=1K−wk​log⁡p^k,\\mathcal\{L\}=\-\\sum\_\{i=1\}^\{n\}\\log p\(\\mathbf\{y\}^\{\*\}\)=\\sum\_\{i=1\}^\{n\}\\sum\_\{k=1\}^\{K\}\-w\_\{k\}\\log\\hat\{p\}\_\{k\},\(10\)which can be interpreted as the cross\-entropy over thesoftassignments\.

![Refer to caption](https://arxiv.org/html/2606.26424v1/figures/qualitative.png)Figure 3:Qualitative results of Wayformer\. The predicted trajectory modes are colored inred\(high probability\) to black\(low probability\), logged trajectories are colored fromgreentoblue, progressing across time, the map polylines are ingray, and the ground truth future trajectory of the target vehicle is inorange\.

## 6Experiments

We compare one\-step expectation maximization with the original WTA method over multiple variants of mode selection methods, showing the improvement of predicted mode probabilities\.

#### Datasets and evaluation metrics

We evaluate accuracy metrics, informative metrics, and qualitative results on NuScenes Prediction\[[5](https://arxiv.org/html/2606.26424#bib.bib58)\]and Waymo Open Motion \(WOMD\)\[[12](https://arxiv.org/html/2606.26424#bib.bib57)\]datasets\. The standard prediction metrics \(minADE, minFDE, miss rate, and brier\-minFDE\) are used in this work\. We evaluate with the number of the final output modesM=5M=5\.

#### Mode selection variants

We have 3 major types of post\-hoc mode selection:

- •Greedy selection: choose the top M modes with the highest predicted scores\.
- •Non\-maximum suppression \(NMS\): choose the modes with the highest predicted scores while suppressing nearby modes whose predicted trajectories are too similar \(i\.e\., within a certain distance threshold\) to avoid redundant predictions and form the final M modes, which is adopted by MTR\[[32](https://arxiv.org/html/2606.26424#bib.bib22)\]\.
- •Clustering\-based merging: merge the initial predictions into M modes \(Section[5\.1](https://arxiv.org/html/2606.26424#S5.SS1)\)\. The initial predicted modes are K\-means clustered through an iterative process, and the M centroids are output as the final M modes after converging\. A similar method is used in Wayformer\[[23](https://arxiv.org/html/2606.26424#bib.bib28)\]\.

Additionally, we include a direct training approach that fixes the decoder to only predict M modes\. This method skips the post\-hoc mode selection, but suffers from the inflexibility when a different number of modes is needed at test time\.

#### Implementation details

The analysis is implemented with three existing models: Wayformer\[[23](https://arxiv.org/html/2606.26424#bib.bib28)\], MTR\-e2e\[[32](https://arxiv.org/html/2606.26424#bib.bib22)\], and EMP\[[25](https://arxiv.org/html/2606.26424#bib.bib69)\]\. We apply variants of post\-hoc mode selection to all models using a unified framework provided by\[[13](https://arxiv.org/html/2606.26424#bib.bib63)\]\. The model decoder predictsK=64K=64initial modes, and compresses toM=5M=5modes\. A two\-layer perceptron head is added to the mode probability head for one\-step EM, which is finetuned over training data\. Unless otherwise specified, the distance threshold of NMS is 1\.2m displacement of the final waypoint\.

Table 2:Accuracy of different post\-hoc selection on NuScenes\.Clustering is better than greedy selection, which shows the mode probability is uninformative and the modes are over\-segmented\.Table 3:Accuracy of different post\-hoc selection on WOMD\.
### 6\.1WTA\-Learned Mode Probability

We begin by evaluating different post\-hoc mode selection methods on the NuScenes dataset, and compare the results of directly using the predicted likelihoods for sampling\-based post\-hoc methods \(greedy and NMS selection\) with merging to obtain our final compact set\. Table[2](https://arxiv.org/html/2606.26424#S6.T2)and Table[3](https://arxiv.org/html/2606.26424#S6.T3)present the prediction accuracy with the final compact set, and their visualization is in Fig\.[3](https://arxiv.org/html/2606.26424#S5.F3)\. Observe that the clustering\-based merging is then better than NMS or greedy selection, which naively relies on the predicted likelihoods\. Note that the error lowerbound of directly training is only slightly better than merging in the case of Wayformer and EMP, and merging for MTR\-e2e even slightly outperforms directly training\. The results reveal two important observations: First, methods selecting a final model only relying on one single initial mode \(greedy selection and NMS\) do not work well, revealing that the predicted mode probabilities are uninformative about the overall pattern of all 64 modes or unaware of relatedness among nearby modes\. Second, the clustering method assigning cluster centroids to be the final modes performs relatively well, reflecting multiple elements \(elements in the same cluster\) segment the same single component of the true mixture distribution\. In other words, if each element learns a single mixture component, the cluster centroids would be placed between components\.

Table 4:Accuracy with one\-step EM finetuning on NuScenes\.Finetuning improves the accuracy of greedy selection significantly, reflecting a more informative mode probability\. EMP is excluded from this analysis since it does not output waypoint Gaussians\.Table 5:Accuracy with one\-step EM finetuning on WOMD\.Table 6:Average NLL before and after one\-step EM finetuning on NuScenes\.Top\-5 NLL is the neg\-log\-likelihood of the GMM composed by top\-5 modes\. Top\-1 NLL is that of the top\-1 Gaussian\.
### 6\.2Evaluation of One\-Step Expectation\-Maximization

We also validate our post\-hoc likelihood correction via the one\-step EM method, and compare it to the original predicted likelihoods\. Accuracy of one\-step EM is compared against the original model with sampling\-based post\-hoc selection \(greedy selection and NMS\) in Table[4](https://arxiv.org/html/2606.26424#S6.T4)and Table[5](https://arxiv.org/html/2606.26424#S6.T5)\. Beyond displacement metrics, negative\-log\-likelihood of the ground truth is reported in Table[6](https://arxiv.org/html/2606.26424#S6.T6)\. Note that themode trajectories predicted are kept consistent, and only the mode likelihoods are changed, leading to improved selection of the final compact trajectory set\.

![Refer to caption](https://arxiv.org/html/2606.26424v1/figures/FDE_distribution.png)\(a\)
![Refer to caption](https://arxiv.org/html/2606.26424v1/figures/ranked-prob-pred-nobar_mtr.png)\(b\)

Figure 4:\(a\) minFDE distribution before and after 1\-step EM finetuning\.Although the locations of the 64 trajectories have not been changed, 1\-step EM finetunes the score for trajectory selection, leading to a better final set that improves minFDE\.\(b\) Probability concentration of top\-10 predictions\.The likelihoods are more concentrated in candidates with correct future behavior after finetuning\. Both results are obtained on NuScenes\.We observe that correction via the finetuning step enables the greedy and NMS selection to improve significantly, in terms of both improved displacement accuracy and distribution quality\. To better understand this effect, we visualize the minFDE distribution before and after one\-step EM finetuning in Fig\.[4\(a\)](https://arxiv.org/html/2606.26424#S6.F4.sf1)\. As compared to before finetuning, the likelihoods shift towards more accurate predictions, enabling better final trajectory selection\. Further, Fig\.[4\(b\)](https://arxiv.org/html/2606.26424#S6.F4.sf2)exhibits that the likelihoods are more concentrated in candidates with correct future behavior after finetuning\. These results indicate that the 1\-step EM brings spatial awareness to the likelihoods, highlighting spatially representative trajectories to mitigate the uninformativeness of likelihoods caused by over\-segmentation\.

## 7Discussion

#### Why do we use the WTA objective?

The WTA objective avoids severe mode collapse caused by full likelihood training, allowing different modes to specialize in different future behaviors, which aligns naturally with benchmark metrics like minADE/minFDE\. Besides, the WTA objective also avoids computationally heavy EM processes\. However, training with WTA objective sacrifices at predicting uninformative mode probabilities\.

#### Implications for Practitioners\.

If the desired number of modes is fixed and training is feasible, training a model to directly produce that number of modes is the most effective strategy, as it provides the cleanest supervision signal, avoids test\-time selection, and leads to higher accuracy\. If flexibility in the number of modes is needed at test time and accuracy is prioritized over inference speed, we suggest using the clustering\-based merging method described in Section[5\.1](https://arxiv.org/html/2606.26424#S5.SS1), though note that clustering and merging will slow down inference\. Finally, if superior inference speed is desired, applying one\-step EM finetuning described in Section[5\.2](https://arxiv.org/html/2606.26424#S5.SS2)on the training data, followed by non\-maximum suppression or greedy selection, would help maintain accuracy\.

## 8Conclusion

In this work, we analyze the widely adopted winner\-take\-all \(WTA\) loss and provide a perspective linking Gaussian mixture models \(GMM\) to K\-means clustering objective for trajectory forecasting of autonomous driving\. We observe that WTA loss leads to uninformative mode probabilities and the modes over\-segment the true multi\-modal distribution instead of collapsing\. We further offer principled and practical post\-hoc treatments to align the training objective with inference\-time probabilistic modeling assumptions and mitigate probability uninformativeness: test\-time merging and one\-step EM finetuning\. Our findings and proposed treatments are empirically verified with three representative winner\-take\-all models on NuScenes and WOMD\.

## Acknowledgments

This research is supported in part by grants from the National Science Foundation \(IIS\-2107077, IIS\-2107161, CNS\-2211599, IIS\-2305532\)\. We thank the self\-driving collaboration group for insightful discussions culminating in this work\.

## References

- \[1\]A\. Alahi, K\. Goel, V\. Ramanathan, A\. Robicquet, L\. Fei\-Fei, and S\. Savarese\(2016\)Social lstm: human trajectory prediction in crowded spaces\.InProceedings of the IEEE conference on computer vision and pattern recognition,pp\. 961–971\.Cited by:[§2](https://arxiv.org/html/2606.26424#S2.SS0.SSS0.Px1.p1.1)\.
- \[2\]A\. Alahi, V\. Ramanathan, and L\. Fei\-Fei\(2014\)Socially\-aware large\-scale crowd forecasting\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition,pp\. 2203–2210\.Cited by:[§2](https://arxiv.org/html/2606.26424#S2.SS0.SSS0.Px1.p1.1)\.
- \[3\]M\. Baniodeh, K\. Goel, S\. Ettinger, C\. Fuertes, A\. Seff, T\. Shen, C\. Gulino, C\. Yang, G\. Jerfel, D\. Choe,et al\.\(2025\)Scaling laws of motion forecasting and planning–a technical report\.arXiv preprint arXiv:2506\.08228\.Cited by:[§1](https://arxiv.org/html/2606.26424#S1.p2.1)\.
- \[4\]H\. Caesar, V\. Bankiti, A\. H\. Lang, S\. Vora, V\. E\. Liong, Q\. Xu, A\. Krishnan, Y\. Pan, G\. Baldan, and O\. Beijbom\(2020\)Nuscenes: a multimodal dataset for autonomous driving\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 11621–11631\.Cited by:[§1](https://arxiv.org/html/2606.26424#S1.p1.1)\.
- \[5\]H\. Caesar, V\. Bankiti, A\. H\. Lang, S\. Vora, V\. E\. Liong, Q\. Xu, A\. Krishnan, Y\. Pan, G\. Baldan, and O\. Beijbom\(2019\)NuScenes: a multimodal dataset for autonomous driving\.arXiv preprint arXiv:1903\.11027\.Cited by:[§3\.1](https://arxiv.org/html/2606.26424#S3.SS1.SSS0.Px4.p1.1),[§6](https://arxiv.org/html/2606.26424#S6.SS0.SSS0.Px1.p1.1)\.
- \[6\]C\. Cao, X\. Chen, J\. Wang, Q\. Song, R\. Tan, and Y\. Li\(2024\)CCTR: calibrating trajectory prediction for uncertainty\-aware motion planning in autonomous driving\.InProceedings of the AAAI Conference on Artificial Intelligence,pp\. 20949–20957\.Cited by:[§2](https://arxiv.org/html/2606.26424#S2.SS0.SSS0.Px1.p1.1)\.
- \[7\]S\. Casas, C\. Gulino, R\. Liao, and R\. Urtasun\(2020\)Spagnn: spatially\-aware graph neural networks for relational behavior forecasting from sensor data\.In2020 IEEE International Conference on Robotics and Automation \(ICRA\),pp\. 9491–9497\.Cited by:[§2](https://arxiv.org/html/2606.26424#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2606.26424#S3.SS1.SSS0.Px4.p1.1)\.
- \[8\]S\. Casas, C\. Gulino, S\. Suo, K\. Luo, R\. Liao, and R\. Urtasun\(2020\)Implicit latent variable model for scene\-consistent motion forecasting\.InEuropean Conference on Computer Vision,pp\. 624–641\.Cited by:[§2](https://arxiv.org/html/2606.26424#S2.SS0.SSS0.Px1.p1.1)\.
- \[9\]Y\. Chai, B\. Sapp, M\. Bansal, and D\. Anguelov\(2019\)Multipath: multiple probabilistic anchor trajectory hypotheses for behavior prediction\.arXiv preprint arXiv:1910\.05449\.Cited by:[§1](https://arxiv.org/html/2606.26424#S1.p2.1),[§1](https://arxiv.org/html/2606.26424#S1.p3.1),[§2](https://arxiv.org/html/2606.26424#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2606.26424#S3.SS1.SSS0.Px4.p1.1),[§3\.2](https://arxiv.org/html/2606.26424#S3.SS2.p2.1),[§4](https://arxiv.org/html/2606.26424#S4.p1.1)\.
- \[10\]A\. Cui, S\. Casas, A\. Sadat, R\. Liao, and R\. Urtasun\(2021\)Lookout: diverse multi\-future prediction and planning for self\-driving\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 16107–16116\.Cited by:[§2](https://arxiv.org/html/2606.26424#S2.SS0.SSS0.Px1.p1.1)\.
- \[11\]N\. Deo and M\. M\. Trivedi\(2018\)Convolutional social pooling for vehicle trajectory prediction\.InProceedings of the IEEE conference on computer vision and pattern recognition workshops,pp\. 1468–1476\.Cited by:[§2](https://arxiv.org/html/2606.26424#S2.SS0.SSS0.Px1.p1.1)\.
- \[12\]S\. Ettinger, S\. Cheng, B\. Caine, C\. Liu, H\. Zhao, S\. Pradhan, Y\. Chai,et al\.\(2021\)Large scale interactive motion forecasting for autonomous driving: the waymo open motion dataset\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 9710–9719\.Cited by:[§1](https://arxiv.org/html/2606.26424#S1.p1.1),[§3\.1](https://arxiv.org/html/2606.26424#S3.SS1.SSS0.Px4.p1.1),[§3\.1](https://arxiv.org/html/2606.26424#S3.SS1.SSS0.Px4.p1.7),[§6](https://arxiv.org/html/2606.26424#S6.SS0.SSS0.Px1.p1.1)\.
- \[13\]L\. Feng, M\. Bahari, K\. M\. B\. Amor, É\. Zablocki, M\. Cord, and A\. Alahi\(2024\)Unitraj: a unified framework for scalable vehicle trajectory prediction\.InEuropean Conference on Computer Vision,pp\. 106–123\.Cited by:[§6](https://arxiv.org/html/2606.26424#S6.SS0.SSS0.Px3.p1.2)\.
- \[14\]K\. Gao, X\. Li, B\. Chen, L\. Hu, J\. Liu, R\. Du, and Y\. Li\(2023\)Dual transformer based prediction for lane change intentions and trajectories in mixed traffic environment\.IEEE Transactions on Intelligent Transportation Systems24\(6\),pp\. 6203–6216\.Cited by:[§2](https://arxiv.org/html/2606.26424#S2.SS0.SSS0.Px1.p1.1)\.
- \[15\]A\. Gupta, J\. Johnson, L\. Fei\-Fei, S\. Savarese, and A\. Alahi\(2018\)Social gan: socially acceptable trajectories with generative adversarial networks\.InProceedings of the IEEE conference on computer vision and pattern recognition,pp\. 2255–2264\.Cited by:[§2](https://arxiv.org/html/2606.26424#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2606.26424#S3.SS1.SSS0.Px4.p1.1)\.
- \[16\]Y\. Hu, W\. Zhan, and M\. Tomizuka\(2018\)Probabilistic prediction of vehicle semantic intention and motion\.In2018 IEEE Intelligent Vehicles Symposium \(IV\),pp\. 307–313\.Cited by:[§2](https://arxiv.org/html/2606.26424#S2.SS0.SSS0.Px1.p1.1)\.
- \[17\]A\. Jain, S\. Casas, R\. Liao, Y\. Xiong, S\. Feng, S\. Segal, and R\. Urtasun\(2020\)Discrete residual flow for probabilistic pedestrian behavior prediction\.InConference on Robot Learning,pp\. 407–419\.Cited by:[§2](https://arxiv.org/html/2606.26424#S2.SS0.SSS0.Px1.p1.1)\.
- \[18\]R\. Li, M\. Tapaswi, R\. Liao, J\. Jia, R\. Urtasun, and S\. Fidler\(2017\)Situation recognition with graph neural networks\.InProceedings of the IEEE international conference on computer vision,pp\. 4173–4182\.Cited by:[§2](https://arxiv.org/html/2606.26424#S2.SS0.SSS0.Px1.p1.1)\.
- \[19\]S\. Lloyd\(1982\)Least squares quantization in pcm\.IEEE Transactions on Information Theory28\(2\),pp\. 129–137\.External Links:[Document](https://dx.doi.org/10.1109/TIT.1982.1056489)Cited by:[§1](https://arxiv.org/html/2606.26424#S1.p3.1),[§4\.1](https://arxiv.org/html/2606.26424#S4.SS1.p1.3),[§4\.1](https://arxiv.org/html/2606.26424#S4.SS1.p2.1)\.
- \[20\]M\. Luber, J\. A\. Stork, G\. D\. Tipaldi, and K\. O\. Arras\(2010\)People tracking with human motion predictions from social forces\.2010 IEEE International Conference on Robotics and Automation,pp\. 464–469\.External Links:[Link](https://api.semanticscholar.org/CorpusID:1046089)Cited by:[§2](https://arxiv.org/html/2606.26424#S2.SS0.SSS0.Px1.p1.1)\.
- \[21\]K\. Luo, S\. Casas, R\. Liao, X\. Yan, Y\. Xiong, W\. Zeng, and R\. Urtasun\(2021\)Safety\-oriented pedestrian occupancy forecasting\.In2021 IEEE/RSJ International Conference on Intelligent Robots and Systems \(IROS\),pp\. 1015–1022\.Cited by:[§2](https://arxiv.org/html/2606.26424#S2.SS0.SSS0.Px1.p1.1)\.
- \[22\]W\. Luo, C\. Park, A\. Cornman, B\. Sapp, and D\. Anguelov\(2023\)Jfp: joint future prediction with interactive multi\-agent modeling for autonomous driving\.InConference on Robot Learning,pp\. 1457–1467\.Cited by:[§3\.1](https://arxiv.org/html/2606.26424#S3.SS1.SSS0.Px3.p1.3)\.
- \[23\]N\. Nayakanti, R\. Al\-Rfou, A\. Zhou, K\. Goel, K\. S\. Refaat, and B\. Sapp\(2022\)Wayformer: motion forecasting via simple & efficient attention networks\.arXiv preprint arXiv:2207\.05844\.Cited by:[§1](https://arxiv.org/html/2606.26424#S1.p2.1),[§1](https://arxiv.org/html/2606.26424#S1.p3.1),[§2](https://arxiv.org/html/2606.26424#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2606.26424#S2.SS0.SSS0.Px2.p1.3),[Table 1](https://arxiv.org/html/2606.26424#S2.T1.2.2.1.1),[§3\.1](https://arxiv.org/html/2606.26424#S3.SS1.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2606.26424#S3.SS1.SSS0.Px3.p1.3),[§3\.2](https://arxiv.org/html/2606.26424#S3.SS2.p2.2),[§4](https://arxiv.org/html/2606.26424#S4.p1.1),[§5\.1](https://arxiv.org/html/2606.26424#S5.SS1.p2.7),[3rd item](https://arxiv.org/html/2606.26424#S6.I1.i3.p1.1),[§6](https://arxiv.org/html/2606.26424#S6.SS0.SSS0.Px3.p1.2)\.
- \[24\]S\. H\. Park, B\. Kim, C\. M\. Kang, C\. C\. Chung, and J\. W\. Choi\(2018\)Sequence\-to\-sequence prediction of vehicle trajectory via lstm encoder\-decoder architecture\.In2018 IEEE intelligent vehicles symposium \(IV\),pp\. 1672–1678\.Cited by:[§2](https://arxiv.org/html/2606.26424#S2.SS0.SSS0.Px1.p1.1)\.
- \[25\]A\. Prutsch, H\. Bischof, and H\. Possegger\(2024\)Efficient motion prediction: a lightweight & accurate trajectory prediction model with fast training and inference speed\.In2024 IEEE/RSJ International Conference on Intelligent Robots and Systems \(IROS\),pp\. 9411–9417\.Cited by:[Table 1](https://arxiv.org/html/2606.26424#S2.T1.2.5.4.1),[§6](https://arxiv.org/html/2606.26424#S6.SS0.SSS0.Px3.p1.2)\.
- \[26\]N\. Rhinehart, K\. M\. Kitani, and P\. Vernaza\(2018\)R2p2: a reparameterized pushforward policy for diverse, precise generative path forecasting\.InProceedings of the European Conference on Computer Vision \(ECCV\),pp\. 772–788\.Cited by:[§2](https://arxiv.org/html/2606.26424#S2.SS0.SSS0.Px1.p1.1)\.
- \[27\]N\. Rhinehart, R\. McAllister, K\. Kitani, and S\. Levine\(2019\)Precog: prediction conditioned on goals in visual multi\-agent settings\.InProceedings of the IEEE/CVF international conference on computer vision,pp\. 2821–2830\.Cited by:[§2](https://arxiv.org/html/2606.26424#S2.SS0.SSS0.Px1.p1.1)\.
- \[28\]J\. Roh, C\. Mavrogiannis, R\. Madan, D\. Fox, and S\. Srinivasa\(2021\)Multimodal trajectory prediction via topological invariance for navigation at uncontrolled intersections\.InConference on Robot Learning,pp\. 2216–2227\.Cited by:[§3\.1](https://arxiv.org/html/2606.26424#S3.SS1.SSS0.Px3.p1.3)\.
- \[29\]A\. Sadeghian, V\. Kosaraju, A\. Sadeghian, N\. Hirose, H\. Rezatofighi, and S\. Savarese\(2019\)Sophie: an attentive gan for predicting paths compliant to social and physical constraints\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 1349–1358\.Cited by:[§2](https://arxiv.org/html/2606.26424#S2.SS0.SSS0.Px1.p1.1)\.
- \[30\]S\. Shi, L\. Jiang, D\. Dai, and B\. Schiele\(2022\)Motion transformer with global intention localization and local movement refinement\.Advances in Neural Information Processing Systems\.Cited by:[§1](https://arxiv.org/html/2606.26424#S1.p2.1),[§1](https://arxiv.org/html/2606.26424#S1.p3.1),[§2](https://arxiv.org/html/2606.26424#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2606.26424#S2.SS0.SSS0.Px2.p1.3),[Table 1](https://arxiv.org/html/2606.26424#S2.T1.2.4.3.1),[§3\.1](https://arxiv.org/html/2606.26424#S3.SS1.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2606.26424#S3.SS1.SSS0.Px3.p1.3),[§3\.2](https://arxiv.org/html/2606.26424#S3.SS2.p2.2),[§4](https://arxiv.org/html/2606.26424#S4.p1.1)\.
- \[31\]S\. Shi, L\. Jiang, D\. Dai, and B\. Schiele\(2022\)MTR\-a: 1st place solution for 2022 waymo open dataset challenge–motion prediction\.arXiv preprint arXiv:2209\.10033\.Cited by:[§1](https://arxiv.org/html/2606.26424#S1.p2.1),[§1](https://arxiv.org/html/2606.26424#S1.p3.1),[§4](https://arxiv.org/html/2606.26424#S4.p1.1)\.
- \[32\]S\. Shi, L\. Jiang, D\. Dai, and B\. Schiele\(2023\)MTR\+\+: multi\-agent motion prediction with symmetric scene modeling and guided intention querying\.arXiv preprint arXiv:2306\.17770\.Cited by:[§1](https://arxiv.org/html/2606.26424#S1.p2.1),[§1](https://arxiv.org/html/2606.26424#S1.p3.1),[§2](https://arxiv.org/html/2606.26424#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2606.26424#S3.SS1.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2606.26424#S3.SS2.p2.2),[§4](https://arxiv.org/html/2606.26424#S4.p1.1),[2nd item](https://arxiv.org/html/2606.26424#S6.I1.i2.p1.1),[§6](https://arxiv.org/html/2606.26424#S6.SS0.SSS0.Px3.p1.2)\.
- \[33\]B\. Varadarajan, A\. Hefny, A\. Srivastava, K\. S\. Refaat, N\. Nayakanti, A\. Cornman, K\. Chen, B\. Douillard, C\. P\. Lam, D\. Anguelov,et al\.\(2022\)Multipath\+\+: efficient information fusion and trajectory aggregation for behavior prediction\.In2022 International Conference on Robotics and Automation \(ICRA\),pp\. 7814–7821\.Cited by:[§1](https://arxiv.org/html/2606.26424#S1.p3.1),[§2](https://arxiv.org/html/2606.26424#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2606.26424#S2.SS0.SSS0.Px2.p1.3),[Table 1](https://arxiv.org/html/2606.26424#S2.T1.2.3.2.1),[§4](https://arxiv.org/html/2606.26424#S4.p1.1),[§5\.1](https://arxiv.org/html/2606.26424#S5.SS1.p2.7)\.
- \[34\]B\. Wilson, W\. Qi, T\. Agarwal, J\. Lambert, J\. Singh, S\. Khandelwal, B\. Pan, R\. Kumar, A\. Hartnett, J\. K\. Pontes,et al\.\(2023\)Argoverse 2: next generation datasets for self\-driving perception and forecasting\.arXiv preprint arXiv:2301\.00493\.Cited by:[§1](https://arxiv.org/html/2606.26424#S1.p1.1)\.
- \[35\]Z\. Zhou, J\. Wang, Y\. Li, and Y\. Huang\(2023\)Query\-centric trajectory prediction\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§1](https://arxiv.org/html/2606.26424#S1.p2.1),[§3\.2](https://arxiv.org/html/2606.26424#S3.SS2.p2.2)\.
- \[36\]A\. Zyner, S\. Worrall, J\. Ward, and E\. Nebot\(2017\)Long short term memory for driver intent prediction\.In2017 IEEE Intelligent Vehicles Symposium \(IV\),pp\. 1484–1489\.Cited by:[§2](https://arxiv.org/html/2606.26424#S2.SS0.SSS0.Px1.p1.1)\.