Predicting Consequences and Reinforcing Navigation Policies with Latent World Models

# Predicting Consequences and Reinforcing Navigation Policies with Latent World Models
Source: [https://arxiv.org/html/2608.26190](https://arxiv.org/html/2608.26190)
[Zengmao Wang](https://orcid.org/0009-0000-7029-6546)Affiliation:School of Advanced Interdisciplinary Sciences, University of Chinese Academy of SciencesAffiliation:Institute of Automation, Chinese Academy of Sciences[Wei Gao](https://orcid.org/0000-0003-2257-5684)Thanks:Corresponding author\.Affiliation:Institute of Automation, Chinese Academy of SciencesAffiliation:School of Artificial Intelligence, University of Chinese Academy of Sciences[Shuhan Shen](https://orcid.org/0000-0002-8704-7914)Affiliation:Institute of Automation, Chinese Academy of SciencesAffiliation:School of Artificial Intelligence, University of Chinese Academy of Sciences

###### Abstract

World models enable agents to reason about future outcomes and learn policies from their knowledge of state transition, but existing approaches primarily focus on reconstructing future observations or features, which introduces unnecessary complexity and limits their effectiveness for decision making\. In this work, we propose a compatibility prediction Latent World Model \(LWM\) for robot navigation that predicts action\-conditioned latent feature compatibility rather than reconstructing observations\. Our key insight is that spatial proximity correlates with latent feature similarity, enabling action consequences to be evaluated directly in latent space\. To support counterfactual training, our model leverages action sequences sampled across trajectories and learns to predict which sequences lead closer to the goal\. Furthermore, we demonstrate how the learned world model can supervise policy learning from unlabeled video data and further improve policies through reinforcement learning entirely within the world model\. This imagination\-driven framework eliminates the need for action annotations and additional environment interaction\. Extensive experiments on multiple real\-world robot navigation datasets show that our approach significantly outperforms prior world model and imitation learning methods in prediction accuracy, policy learning, and real\-world navigation performance\. The code, pretrained models, and additional materials are available at[https://wzm206\.github\.io/latent\-world\-model\-nav/](https://wzm206.github.io/latent-world-model-nav/)\.

*Keywords*World model⋅\\cdotRobot learning⋅\\cdotVisual navigation

## 1Introduction

Autonomous navigation requires robots to reason about the consequences of their actions before executing them in the real world\. World models have emerged as a promising paradigm for enabling such predictive reasoning by learning environment dynamics from offline data[13](https://arxiv.org/html/2608.26190#bib.bib14)\. By simulating future outcomes internally, world models allow agents to plan[2](https://arxiv.org/html/2608.26190#bib.bib11), evaluate[25](https://arxiv.org/html/2608.26190#bib.bib37), and improve policies[42](https://arxiv.org/html/2608.26190#bib.bib12)without costly real\-world interaction\. This paradigm is particularly attractive for robotics, where data collection is expensive and safety constraints limit online exploration\.

However, existing world models suffer from a fundamental limitation: they are typically trained to reconstruct future observations conditioned strictly on the factual actions executed in the dataset[2](https://arxiv.org/html/2608.26190#bib.bib11);[15](https://arxiv.org/html/2608.26190#bib.bib17)\. When we use the world model for planning or decision\-making, the model needs to infer situations that have not actually occurred, also known as counterfactual reasoning\. This creates a critical misalignment: while planning requires counterfactual reasoning to evaluate alternative actions, training relies solely on ground\-truth trajectories\. Furthermore, optimizing for pixel or latent reconstruction introduces unnecessary complexity, compounding errors, and representations that are misaligned with decision\-making\. We argue that world models should focus on predicting action consequences rather than generating exact observations\.

In this work, we propose a fundamentally different perspective: instead of predicting future observations, we train a world model to predict the compatibility between action consequences and future latent state which leads counterfactual reasoning into the training phase\. Our approach is motivated by a simple yet powerful observation: in navigation tasks, spatial proximity strongly correlates with visual feature similarity\. Therefore, rather than reconstructing exact future observations, it is sufficient to predict the compatibility\.

To achieve this counterfactual training, we introduce an imagination\-based world model that leverages action sequences from different trajectories to simulate counterfactual futures in latent space\. Given a current observation, our model predicts latent state features resulting from multiple candidate action sequences and learns to distinguish which sequences lead to states closer to the ground\-truth future\. Crucially, simulating these counterfactual scenarios during training significantly maximizes data utilization efficiency because collecting real\-world interaction data is notoriously expensive and challenging in robotics field\. This formulation enables the model to learn robust, decision\-centric representations without requiring pixel\-level prediction\. Therefore, we achieve training world models in counterfactual imagination\.

Beyond world model learning, we demonstrate how the learned model can be used to train and improve navigation policies from unlabeled video data\. We first use the world model to label candidate action sequences according to their predicted similarity to goal states, enabling supervised policy learning without ground\-truth actions\. We then further improve the policy using reinforcement learning entirely within the learned world model\. A well\-trained world model can naturally serve as a reward model\. This imagination\-driven reinforcement learning enables policy improvement without additional environment interaction\.

We evaluate our approach in real environments and three robot navigation datasets\. Our method demonstrates significant improvements over prior world model and navigation methods\. Our results show that compatibility prediction provides a powerful and efficient alternative to generative world modeling, enabling effective planning and policy learning from offline data\. We summarize our contributions as follows\.

- •A new world model formulation based on latent compatibility prediction\. We propose to predict action\-conditioned latent feature compatibility rather than reconstructing future observations, enabling efficient and decision\-oriented world modeling\.
- •A counterfactual imagination framework using cross\-trajectory action sequences\. Our method leverages action sequences from different trajectories to simulate alternative futures, enabling counterfactual reasoning from purely offline data\.
- •A unified framework for policy learning and reinforcement entirely within the world model\. We show how unlabeled video data can be used to supervise policy learning and further improve policies via imagination\-based reinforcement learning\.
- •State\-of\-the\-art performance in real\-world environment and robot navigation benchmarks\.

## 2Related Work

Driven by advancements in large language models[36](https://arxiv.org/html/2608.26190#bib.bib40), video generation[18](https://arxiv.org/html/2608.26190#bib.bib42), and embodied intelligence[7](https://arxiv.org/html/2608.26190#bib.bib41), world models have garnered significant attention[5](https://arxiv.org/html/2608.26190#bib.bib1)\. While the concept continues to evolve, it primarily serves two functions: learning internal representations to understand the environment, and utilizing predictive modeling to guide decision\-making for external actions\.

### 2\.1Building a World Model

Numerous efforts focus on constructing world models\. Video generation\-based approaches[37](https://arxiv.org/html/2608.26190#bib.bib2);[1](https://arxiv.org/html/2608.26190#bib.bib3);[23](https://arxiv.org/html/2608.26190#bib.bib4);[9](https://arxiv.org/html/2608.26190#bib.bib5)are intuitive and demonstrate remarkable physical consistency—such as understanding gravity and collisions—highlighting their potential as implicit world models[19](https://arxiv.org/html/2608.26190#bib.bib6);[8](https://arxiv.org/html/2608.26190#bib.bib7);[28](https://arxiv.org/html/2608.26190#bib.bib8);[20](https://arxiv.org/html/2608.26190#bib.bib9)\. These methods typically condition future frame prediction on current observations and actions\. For instance, LingBot\-World[35](https://arxiv.org/html/2608.26190#bib.bib10)employs a virtual rendering pipeline for interactive data generation, NWM[2](https://arxiv.org/html/2608.26190#bib.bib11)leverages large\-scale real\-world datasets to generate diverse action\-conditioned videos, and 3D\-VLA[44](https://arxiv.org/html/2608.26190#bib.bib35)extends this paradigm to 3D point clouds\.

However, pixel\-level video generation suffers from slow inference speeds and inconsistent physics\. Dino\-WM[45](https://arxiv.org/html/2608.26190#bib.bib20)addresses this by shifting to feature generation using a Dino\-v2 encoder[27](https://arxiv.org/html/2608.26190#bib.bib32), though its primary objective remains reconstruction\. In contrast, our method avoids pixel\-space generation entirely, preventing the model from fixating on irrelevant details\. By optimizing to predict the consequences of actions in the latent space rather than reconstructing features, our approach eliminates the need for iterative diffusion, enabling rapid prediction and decision\-making\.

### 2\.2Learning in the World Model

Ultimately, world models aim to facilitate decision\-making and learning for agents\. Learning paradigms within these models generally fall into two categories\.

#### 2\.2\.1Joint Training of World and Policy Models

Akin to model\-based reinforcement learning[14](https://arxiv.org/html/2608.26190#bib.bib13);[13](https://arxiv.org/html/2608.26190#bib.bib14);[16](https://arxiv.org/html/2608.26190#bib.bib15);[33](https://arxiv.org/html/2608.26190#bib.bib16), these methods jointly optimize both models\. PlaNet[16](https://arxiv.org/html/2608.26190#bib.bib15)introduces the RSSM architecture to enable dynamic learning in a compact latent space\. Dreamer[15](https://arxiv.org/html/2608.26190#bib.bib17)and DreamerV2[17](https://arxiv.org/html/2608.26190#bib.bib18)learn latent dynamics from images and optimize policies via imagined trajectories, excelling in long\-horizon tasks\. PWM[10](https://arxiv.org/html/2608.26190#bib.bib33)facilitates first\-order policy optimization within regularized world models\. However, these methods often require continuous agent\-environment interaction, confining them to simulated environments and limiting real\-world practicality\. While our method also leverages latent state representations, it can be deployed directly in the real world without requiring online interaction\.

#### 2\.2\.2Utilizing Frozen World Models as Simulators

Alternatively, frozen world models can serve as realistic simulators\. DreamZero[42](https://arxiv.org/html/2608.26190#bib.bib12)learns physical dynamics via joint video\-action prediction for real\-time control\. World\-env[40](https://arxiv.org/html/2608.26190#bib.bib19)replaces physical interactions with a virtual simulator for RL training\. RWM[24](https://arxiv.org/html/2608.26190#bib.bib36)uses a dual\-autoregressive model for robust sim\-to\-real transfer, and WorldVLA[3](https://arxiv.org/html/2608.26190#bib.bib34)unifies VLA and world models\. These approaches typically train policies on synthesized pixel\-level images, which is significantly more complex than predicting feature\-space similarities\. Furthermore, the generalization of video generation remains problematic[20](https://arxiv.org/html/2608.26190#bib.bib9)\. For instance, autonomous driving models struggle to imagine out\-of\-distribution scenarios like driving on sidewalks due to a lack of training data\. Conversely, our cross\-trajectory imagination approach can effectively evaluate such deviations, keeping the agent safely aligned with the goal\.

![Refer to caption](https://arxiv.org/html/2608.26190v1/mainloop.png)Figure 1:The overview of LWM\. \(a\) When training the world model, we need to predict the consequences of different action sequences and obtain compatibility with the goal\. \(b\) LWM can provide pseudo labels for the policy model\. \(c\) The policy model can be reinforced in LWM\.

## 3Method

We propose a compatibility prediction Latent World Model \(LWM\) that learns action\-conditioned latent dynamics by predicting feature compatibility rather than reconstructing future observations\. As shown in Fig\.[1](https://arxiv.org/html/2608.26190#S2.F1), our framework enables counterfactual reasoning from offline trajectories to alleviate the challenges of data annotation and collection\. Our framework supports policy learning and reinforcement entirely within LMW\. We will elaborate on our method in detail below\.

### 3\.1General Problem Formulation

We consider a dataset of offline trajectories:

𝒟=\{τi\}i=1N,τi=\{\(o1,a1\),\(o2,a2\),…,\(oT,aT\)\}\\mathcal\{D\}=\\\{\\tau\_\{i\}\\\}\_\{i=1\}^\{N\},\\quad\\tau\_\{i\}=\\\{\(o\_\{1\},a\_\{1\}\),\(o\_\{2\},a\_\{2\}\),\\dots,\(o\_\{T\},a\_\{T\}\)\\\}\(1\)whereot∈ℝH×W×3o\_\{t\}\\in\\mathbb\{R\}^\{H\\times W\\times 3\}is the visual observation andat∈ℝdaa\_\{t\}\\in\\mathbb\{R\}^\{d\_\{a\}\}is the robot action at timesteptt\. In visual navigation tasks,da=3\{d\_\{a\}\}=3represents the coordinates of motion\(x,y,θ\)\(x,y,\\theta\)\.

Our goal is to learn a world modelWMthat can evaluate the consequences of arbitrary action sequences starting from a given observation\. Within the world model, it can be divided into two modules: the prediction module and the discrimination module\. In the prediction module, the world model predicts the state features after executing the action\. In the discrimination module, the world model determines the compatibility between the predicted features and the goal\.

𝐳^t\+1:t\+K=WMpredict\(ot,At:t\+K−1\),Ct\+1:t\+K=WMdiscriminative\(𝐳^t\+1:t\+K,og\)\\hat\{\\mathbf\{z\}\}\_\{t\+1:t\+K\}=\\text\{WM\}\_\{\\text\{predict\}\}\(o\_\{t\},A\_\{t:t\+K\-1\}\),\\quad\{C\}\_\{t\+1:t\+K\}=\\text\{WM\}\_\{\\text\{discriminative\}\}\(\\hat\{\\mathbf\{z\}\}\_\{t\+1:t\+K\},o\_\{g\}\)\(2\)whereoto\_\{t\}is the observation at timett,At:t\+K−1=\{at,at\+1,…,at\+K−1\}A\_\{t:t\+K\-1\}=\\\{a\_\{t\},a\_\{t\+1\},\\dots,a\_\{t\+K\-1\}\\\}is a future action sequence,𝐳^t\+1:t\+K\\hat\{\\mathbf\{z\}\}\_\{t\+1:t\+K\}is the predicted features andCCis the compatibility\.

In video generation world models,𝐳^t\+1:t\+K\\hat\{\\mathbf\{z\}\}\_\{t\+1:t\+K\}in Eq\.\([2](https://arxiv.org/html/2608.26190#S3.E2)\) could be considered as images generated based on observation and action sequences\.CCin Eq\.\([2](https://arxiv.org/html/2608.26190#S3.E2)\) is obtained through pre\-trained feature loss, such as comparing the LPIPS[43](https://arxiv.org/html/2608.26190#bib.bib28)loss between the generated image and the goal image\. Unlike these works, we do not attempt to reconstruct future observations\. Instead, we bring the entire process into the latent space for better decision\-making and planning\.

### 3\.2Latent World Model

#### 3\.2\.1Latent Observation and Action Encoding

We first map observations into a latent feature space using a visual encoder:

zto=Eo​\(ot\),zt∈ℝNp×Dz\_\{t\}^\{o\}=E\_\{o\}\(o\_\{t\}\),\\quad z\_\{t\}\\in\\mathbb\{R\}^\{N\_\{p\}\\times D\}\(3\)whereEoE\_\{o\}is a Vision Transformer[6](https://arxiv.org/html/2608.26190#bib.bib38)encoder\.NpN\_\{p\}represents the number of patches in the images\. Similarly, we encode each action into a latent token:

𝐳t:t\+K−1a=Ea\(At:t\+K−1\),𝐳t:t\+K−1a∈ℝ\(K−1\)×D\\mathbf\{z\}\_\{t:t\+K\-1\}^\{a\}=E\_\{a\}\(A\_\{t:t\+K\-1\}\),\\quad\\mathbf\{z\}\_\{t:t\+K\-1\}^\{a\}\\in\\mathbb\{R\}^\{\(K\-1\)\\times D\}\(4\)
Our world model predicts latent state features conditioned on the current state and an action sequence\. We implement the world model prediction using a standard causal Transformer decoder:

𝐳^t\+1:t\+Ks=Decoder\(zto,𝐳t:t\+K−1a\)\\hat\{\\mathbf\{z\}\}^\{s\}\_\{t\+1:t\+K\}=\\text\{Decoder\}\(z\_\{t\}^\{o\},\\mathbf\{z\}\_\{t:t\+K\-1\}^\{a\}\)\(5\)Each prediction follows autoregressive causal masking\. For each action token, it can only see its previous actions\. At this point, we can predict\(K−1\)\(K\-1\)future features\. For previous methods, they often predicted pixels or features and optimized neural networks using reconstruction loss:L​o​s​s=M​S​E​\(𝐳^,𝐳\)Loss=MSE\(\\hat\{\\mathbf\{z\}\},\\mathbf\{z\}\)\. However, they did not take into account the assistance of other action sequences in the dataset\. Although we don’t know what the observations of performing other actions look like, we can estimate their compatibility with the goal images\.

#### 3\.2\.2Compatibility Learning Objective

A key limitation of offline data is that each observation is paired with only one action sequence\. To enable counterfactual reasoning, we construct alternative action sequences by sampling from other trajectories\. Given a stateztoz\_\{t\}^\{o\}, we sampleMcounterM\_\{\\text\{counter\}\}candidate action sequences:

𝒜=\{A\(1\),A\(2\),…,A\(Mcounter\)\}\\mathcal\{A\}=\\\{A^\{\(1\)\},A^\{\(2\)\},\\dots,A^\{\(M\_\{\\text\{counter\}\}\)\}\\\}\(6\)The world model predicts corresponding future latent state features:

𝒵^=\{𝐳^s⁡\(1\),𝐳^s⁡\(2\),…,𝐳^s⁡\(Mcounter\)\},𝐳^t\+1:t\+Ks⁡\(i\)=Decoder\(zo,Ea\(A\(i\)\)\)\\hat\{\\mathcal\{Z\}\}=\\\{\\hat\{\\mathbf\{z\}\}^\{s\(1\)\},\\hat\{\\mathbf\{z\}\}^\{s\(2\)\},\\dots,\\hat\{\\mathbf\{z\}\}^\{s\(M\_\{\\text\{counter\}\}\)\}\\\},\\quad\\hat\{\\mathbf\{z\}\}\_\{t\+1:t\+K\}^\{s\(i\)\}=\\text\{Decoder\}\(z^\{o\},E\_\{a\}\(A^\{\(i\)\}\)\)\(7\)
This allows the model to learn action\-conditioned compatibility across diverse trajectories, enabling counterfactual reasoning without environment interaction\. Our method relies on a simple and reasonable assumption: features𝐳\\mathbf\{z\}with closer spatial distances often more similar in the latent space\. For simplicity, we consider training with only one random goal imagezg=Eo​\(og\)z^\{g\}=E\_\{o\}\(o\_\{g\}\)in ground truth action sequence\{at,at\+1,…,at\+Kg​t\}\\\{a\_\{t\},a\_\{t\+1\},\\dots,a\_\{t\+K\_\{gt\}\}\\\}\. It is worth noting that the selection of goal image during training is random andKg​tK\_\{gt\}is smaller thanKK\. Therefore, for eachz^js⁡\(i\)\\hat\{z\}^\{s\(i\)\}\_\{j\}, we can calculate a compatibility with the ground truth goal image\. ForMcounterM\_\{\\text\{counter\}\}action sequences, we can obtainMcounter×\(K−1\)M\_\{\\text\{counter\}\}\\times\(K\-1\)compatibility score:

C=\{S⁡\(z^js⁡\(i\),zg\)\}j=1,2,…,K−1i=1,2,…,Mcounter,C∈ℝMcounter×\(K−1\)C=\\left\\\{S\(\\hat\{z\}^\{s\(i\)\}\_\{j\},z^\{g\}\)\\right\\\}^\{i=1,2,\.\.\.,M\_\{\\text\{counter\}\}\}\_\{j=1,2,\.\.\.,K\-1\},\\quad C\\in\\mathbb\{R\}^\{M\_\{\\text\{counter\}\}\\times\(K\-1\)\}\(8\)where S is a neural network used for compatibility prediction\. Because the state features are in the latent space, we still choose standard Transformer Decoder with a linear output layer instead of LPIPS[43](https://arxiv.org/html/2608.26190#bib.bib28)\. AmongCi​jC\_\{ij\}, the score corresponding to ground truth action sequence\{at,at\+1,…,at\+Kg​t\}\\\{a\_\{t\},a\_\{t\+1\},\\dots,a\_\{t\+K\_\{gt\}\}\\\}should be the highest\. This naturally reminds us of classification objectives similar to contrastive learning\. Specifically, we can consider one resultCg​tC\_\{gt\}obtained from ground truth action sequences as positive samples andMcounter×\(K−1\)−1M\_\{\\text\{counter\}\}\\times\(K\-1\)\-1results obtained from counterfactual action sequences as negative samples\. In this way, we can use Info\-NCE loss[12](https://arxiv.org/html/2608.26190#bib.bib44)to optimize the world model\.

ℒ=−log⁡exp⁡Cg​t∑i=1Mcounter∑j=1K−1exp⁡Ci​j\\mathcal\{L\}=\-\\log\\frac\{\\exp C\_\{gt\}\}\{\{\\textstyle\\sum\_\{i=1\}^\{M\_\{\\text\{counter\}\}\}\\sum\_\{j=1\}^\{K\-1\}\\exp C\_\{ij\}\}\}\(9\)However, we found that although this approach can obtain a world model, it ignores the spatial relationships between action sequences\. To better capture the underlying environment dynamics, we define the compatibility label in Eq\. \([10](https://arxiv.org/html/2608.26190#S3.E10)\) using a logarithmic mapping of spatial distances\. Physically, this transformation converts the geometric Euclidean metric into a high\-fidelity compatibility density\. The−log\-\\logoperator ensures that the supervision signal is highly sensitive to small spatial deviations, thereby forcing the world model to distinguish subtle differences between the ground\-truth trajectory and its spatially\-close neighbors \(i\.e\., hard negatives\)\.

Clabel=\{−log⁡\[d⁡\(aji,ag​t\)\+ϵ\]\}j=1,2,…,K−1i=1,2,…,McounterC\_\{\\text\{label\}\}=\\\{\-\\log\[\{d\(a^\{i\}\_\{j\},a\_\{gt\}\)\+\\epsilon\}\]\\\}^\{i=1,2,\.\.\.,M\_\{\\text\{counter\}\}\}\_\{j=1,2,\.\.\.,K\-1\}\(10\)whereddrepresents the metric distance of space, and we choose the simplest Euclidean distance\.ϵ\\epsilonis a small constant\. In the spatial distance, the smallest element value is 0, which corresponds to the ground truth action\. It is worth noting that this method can be directly used to train multiple goal images together with almost no increase in computational\. We only need to replacezgz\_\{g\}andag​ta\_\{gt\}with multiple ground truth on the same action sequence during training\. Afterwards, we use simple MSE loss function:ℒ=M​S​E​\(C,Clabel\)\\mathcal\{L\}=MSE\(C,C\_\{\\text\{label\}\}\)\.

#### 3\.2\.3Best practices

When we need to use world models for planning or decision\-making, we first give the current observationoto\_\{t\}, different action sequences𝒜\\mathcal\{A\}, and predict the executed features𝒵^\\hat\{\\mathcal\{Z\}\}\. Then we can get the compatibility scoreCCwith the goal image we want to reach, and select the action sequence with the highest compatibility score to execute\. In fact, this is still the paradigm of predicting first and then discriminating\. This reminds us of a question: Can we use the information of the goal when encoding the image from the beginning? If the goal features are given first during prediction in Eq\. \([3](https://arxiv.org/html/2608.26190#S3.E3)\), we can get better results\. That is to say, if the world model is for planning, we can provide goal information in advance when encoding observations and then determine the compatibility score\. Then Eq\. \([3](https://arxiv.org/html/2608.26190#S3.E3)\) becomeszto=Eo​\(ot,og\)z\_\{t\}^\{o\}=E\_\{o\}\(o\_\{t\},o\_\{g\}\)\. This approach is equivalent to giving a goal in advance, and then the world model imagines the compatibility score to the goal after performing different actions\. This goal can not only be an image target, but can also be extended to semantic targets[26](https://arxiv.org/html/2608.26190#bib.bib21)and language instructions[41](https://arxiv.org/html/2608.26190#bib.bib22)\. However, previous world model mainly focused on dynamic models of the environment without providing goal information in advance\. Therefore, for the best performance and fair comparison, we test two options separately: providing goal prompts in advance and not providing them\. The specific details can be further discussed in the experimental section\.

### 3\.3Policy Learning from World Model

Once the world model is trained, it can be leveraged to utilize large\-scale unlabeled video data for policy learning\. We define a goal\-conditioned policyπ⁡\(at∣ot,og\)\\pi\(a\_\{t\}\\mid o\_\{t\},o\_\{g\}\)that takes the current observationoto\_\{t\}and the goal observationogo\_\{g\}as inputs, and outputs an action sequence that drives the robot toward the goal state\.

Following prior autoregressive policy formulations such as OpenVLA[22](https://arxiv.org/html/2608.26190#bib.bib23)and ArMoNa[38](https://arxiv.org/html/2608.26190#bib.bib24), our policy generates actions as a sequence of discrete tokens\. Specifically, we first compute action incrementsΔ​a\\Delta aand apply k\-means clustering to obtainKcenter=64K\_\{\\text\{center\}\}=64cluster centers, which serve as discrete action tokens\. Any continuous action sequence can thus be represented as a sequence of these tokens\. Our policy model consists of a visual encoder and a causal transformer decoder\. The visual encoder extracts features from the current and the goal observation, while the causal decoder autoregressively predicts the action tokens\. Similar to next\-token prediction in natural language processing, the model generates tokens sequentially, starting from a special start token and continuing until an end token is produced\.

In the absence of ground\-truth action labels, we exploit the predictive capability of the world model to generate pseudo action labels from unlabeled video sequences\{ot,ot\+1,…\}\\\{o\_\{t\},o\_\{t\+1\},\\dots\\\}\. We first construct a candidate action set by clustering existing action sequences intoMc=64M\_\{c\}=64cluster centers, denoted as𝒜c=\{Ac\(1\),Ac\(2\),…,Ac\(Mc\)\}\\mathcal\{A\}\_\{c\}=\\\{A\_\{c\}^\{\(1\)\},A\_\{c\}^\{\(2\)\},\\dots,A\_\{c\}^\{\(M\_\{c\}\)\}\\\}\. We then randomly sample observation\-goal pairs from video trajectories\. For each pair\(ot,og\)\(o\_\{t\},o\_\{g\}\), we evaluate all candidate action sequences using the compatibility score from world model\. The action sequence with the highest compatibility score is selected as the pseudo label\. The discretized action sequence can be represented as\{as​t​a​r​t,at,at\+1,…,at\+Kpre,ae​n​d\}\.\\\{a\_\{start\},a\_\{t\},a\_\{t\+1\},\\dots,a\_\{t\+K\_\{\\text\{pre\}\}\},a\_\{end\}\\\}\.

Finally, the policy model is trained using a standard autoregressive cross\-entropy loss:

ℒ=−∑i=tt\+Kprelogπθ\(ai∣ot,og,a<i\),\\mathcal\{L\}=\-\\sum\_\{i=t\}^\{t\+K\_\{\\text\{pre\}\}\}\\log\\pi\_\{\\theta\}\\left\(a\_\{i\}\\mid o\_\{t\},o\_\{g\},a\_\{<i\}\\right\),\(11\)which encourages the model to generate action sequences that lead to the desired goal observation\.

### 3\.4Policy Reinforcement in Latent World Model

After the initial imitation stage, the policy model acquires the basic ability to generate action sequences conditioned on the current and goal observations\. However, this stage is limited by the fixed candidate action set constructed from clustering, which restricts policy expressiveness\. To further improve the policy, we perform reinforcement learning entirely within the latent world model\. This process is analogous to the cold\-start reinforcement paradigm used in DeepSeek\-R1[11](https://arxiv.org/html/2608.26190#bib.bib25), where the model is progressively improved beyond the initial imitation distribution\.

Unlike the imitation stage, where the policy is trained on a fixed set of clustered action sequences, the autoregressive policy can now freely generate novel action sequences by composing discrete tokens\. Given a current observationoto\_\{t\}and a goal observationogo\_\{g\}, we sample multiple candidate action sequences from the pretrained policy:

𝒜=\{A\(1\),A\(2\),…,A\(Mgen\)\},A\(i\)∼πθ\(⋅∣ot,og\)\.\\mathcal\{A\}=\\\{A^\{\(1\)\},A^\{\(2\)\},\\dots,A^\{\(M\_\{\\text\{gen\}\}\)\}\\\},\\quad A^\{\(i\)\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid o\_\{t\},o\_\{g\}\)\.\(12\)
After predicting the consequences of different actions, the discrimination module of LWM can naturally be seen as a reward model\. Specifically, for each sampled action sequenceA\(i\)A^\{\(i\)\}, the world model evaluates its compatibility with the goal observation, producing a scalar reward:

z^i=WMpredict​\(ot,A\(i\)\),ri=WMdiscriminative​\(z^i,og\)\\hat\{z\}^\{i\}=\\text\{WM\}\_\{\\text\{predict\}\}\(o\_\{t\},A^\{\(i\)\}\),\\quad r\_\{i\}=\\text\{WM\}\_\{\\text\{discriminative\}\}\(\\hat\{z\}^\{i\},o\_\{g\}\)\(13\)where higher reward indicates that the predicted outcome is closer to the goal state\. To stabilize training, we normalize the rewards across sampled sequences to obtain advantages:

Ai=ri−mean​\(𝐫\)std​\(𝐫\)\.A\_\{i\}=\\frac\{r\_\{i\}\-\\text\{mean\}\(\\mathbf\{r\}\)\}\{\\text\{std\}\(\\mathbf\{r\}\)\}\.\(14\)We optimize the policy using a GRPO\-style[11](https://arxiv.org/html/2608.26190#bib.bib25)objective, which encourages high\-reward actions while constraining the policy to remain close to a reference policy:

ℒGRPO\\displaystyle\\mathcal\{L\}\_\{\\text\{GRPO\}\}=−1Mgen∑i=1Mgen\(rratioAi−β𝔻K​L\(πθ∥πref\)\),\\displaystyle=\-\\frac\{1\}\{M\_\{\\text\{gen\}\}\}\\sum\_\{i=1\}^\{M\_\{\\text\{gen\}\}\}\\left\(r\_\{\\text\{ratio\}\}A\_\{i\}\-\\beta\\mathbb\{D\}\_\{KL\}\\left\(\\pi\_\{\\theta\}\\;\\\|\\;\\pi\_\{\\text\{ref\}\}\\right\)\\right\),\(15\)where the probability ratio is defined as:

rratio=min⁡\(πθ​\(𝐚i\)πθref​\(𝐚i\),clip​\(πθ​\(𝐚i\)πθref​\(𝐚i\),1−ϵ,1\+ϵ\)\)\.r\_\{\\text\{ratio\}\}=\\min\\left\(\\frac\{\\pi\_\{\\theta\}\(\\mathbf\{a\}\_\{i\}\)\}\{\\pi\_\{\\theta\_\{\\text\{ref\}\}\}\(\\mathbf\{a\}\_\{i\}\)\},\\;\\text\{clip\}\\left\(\\frac\{\\pi\_\{\\theta\}\(\\mathbf\{a\}\_\{i\}\)\}\{\\pi\_\{\\theta\_\{\\text\{ref\}\}\}\(\\mathbf\{a\}\_\{i\}\)\},\\;1\-\\epsilon,\\;1\+\\epsilon\\right\)\\right\)\.\(16\)
This objective encourages the policy to increase the likelihood of action sequences that lead to goal\-consistent future states\. Importantly, this reinforcement process is performed entirely within the learned latent world model, without requiring additional real\-world interaction or action annotations\. By iteratively sampling, evaluating, and optimizing in the latent space, the policy can continuously improve and discover more effective action sequence beyond the initial imitation distribution\.

## 4Experiment

### 4\.1Experimental Setting

#### 4\.1\.1Datasets\.

We conduct experiments using a total of three datasets\. For the public dataset, we use the datasets RECON[30](https://arxiv.org/html/2608.26190#bib.bib27)and SCAND[21](https://arxiv.org/html/2608.26190#bib.bib26)used by NWM[2](https://arxiv.org/html/2608.26190#bib.bib11)respectively\. The RECON dataset is a fisheye camera dataset collected in outdoor environments, which includes a large number of lawn houses, etc\. The SCAND dataset is collected in the campus environment, containing a large number of dynamic objects and human interaction scenarios\. We selected data from all wheeled robots for approximately three hours\. In addition, to validate the practicality of our framework, we collect our own dataset LWM \(navigation dataset in Latent World Model\) using the robot shown in Fig\.[3](https://arxiv.org/html/2608.26190#S4.F3)\. The dataset includes campus, residential area, and park scenes, covering approximately 6 hours and 600,000m2m^\{2\}\. We use wheel encoders and IMUs to obtain the position of the robot, sample one keyframe over 0\.2m, and set the maximum predicted frameK=64K=64\. We use 5% of the dataset as the test set and the remaining 95% as the training set\. Within the training set, half the data is used to train the world model and imitation learning, while the other half is used to reinforce the world model\.

#### 4\.1\.2Baselines\.

- •NWM[2](https://arxiv.org/html/2608.26190#bib.bib11)is a world model based on image generation for robot navigation tasks\. When planning and making decisions, the world model generates different image sequences based on different actions\. We follow the approach in the paper to select the action sequence with the minimum LPIPS[43](https://arxiv.org/html/2608.26190#bib.bib28)loss between the goal image and the generated image\. The model has already been trained on RECON and SCAND\. To align with our evaluation method, we fine tuned NWM\-CDiT/XL\-1B model for 10 epochs on these two datasets and 30 epochs on LWM\.
- •Dino\-WM[45](https://arxiv.org/html/2608.26190#bib.bib20)is a world model based on feature space\. All observed images are encoded into features by a Dino\-V2 encoder[45](https://arxiv.org/html/2608.26190#bib.bib20)\. The features of observation image and actions are fed into the world model to predict the feature of the goal image\. During deployment, Dino\-WM also used different action sequences for feature prediction, and ultimately selects the action sequence with the smallest error compared to the feature of goal image for execution\.
- •NoMaD[34](https://arxiv.org/html/2608.26190#bib.bib31)is a robot navigation method based on diffusion policy[4](https://arxiv.org/html/2608.26190#bib.bib43)\. The current image and the goal image are encoded as feature conditions, and then the noise is guided to transform into an action sequence\.
- •BC is a baseline robot navigation method we designed based on ViNT[32](https://arxiv.org/html/2608.26190#bib.bib29)and GNM[31](https://arxiv.org/html/2608.26190#bib.bib30)\. It is based on imitation learning\. The current and the goal image are input to an image encoder and then decoded into an action sequence\.

For fair comparison with baseline methods, we deploy three LWM: LWM\-S uses Dino\-v2\-small[27](https://arxiv.org/html/2608.26190#bib.bib32)as encoder which has the same 49M parameters as Dino\-WM[45](https://arxiv.org/html/2608.26190#bib.bib20)\. LWM\-B and LWM\-B\-P uses Croco\-v2\-base[39](https://arxiv.org/html/2608.26190#bib.bib39)as encoder which has 217M parameters\. LWM\-B\-P is given the goal prompt in advance as described in[3\.2\.3](https://arxiv.org/html/2608.26190#S3.SS2.SSS3)\. We compare policy model with the same number of parameters\. Specific implementation details can be found in the supplementary materials\.

#### 4\.1\.3Evaluation Metrics\.

For the evaluation of the world model, we measure prediction accuracy\. We use the 64 action sequences in Section[3\.3](https://arxiv.org/html/2608.26190#S3.SS3)obtained from clustering as candidate trajectories\. Following NWM[2](https://arxiv.org/html/2608.26190#bib.bib11), for each sample in the test dataset, we roll out all candidate action sequences in the world model and select the trajectory whose final state best matches the goal\.

- •PE and OE\. To quantify prediction error, we compute the Euclidean distance between the predicted goal coordinates and the corresponding ground\-truth goal coordinates\. Specifically, we report two metrics: Position Error \(translation error\) and Orientation Error \(rotation error\), corresponding to the discrepancies in spatial position and attitude, respectively\.
- •ACC\. In addition, we evaluate the consistency of the world model by measuring its ability to distinguish the correct action sequence from counterfactual alternatives\. Specifically, for each sample, we construct a candidate set consisting of the ground\-truth action sequence and several counterfactual action sequences from the dataset\. Given the current and goal observation, the world model can select an action sequence from candidate set\. We report the classification accuracy under different candidate set sizes, including 3 and 5 candidates, denoted by ACC3 and ACC5\.
- •SR and LPIPS\. To evaluate whether the policy model can enable robots to reach their goals, we deploy them in real\-world environments\. If the robot reaches within 0\.5m of the goal, it is considered successful\. To evaluate the accuracy of reaching the goal, when the robot succeeds in reaching the goal, we compare the minimum LPIPS between the robot observation and the goal image\.

Table 1:World Model Prediction and Planning Evaluation\. Comparison of prediction errors \(PE, OE\) and action selection accuracy \(ACC3, ACC5\)\. Our method achieve the lowest errors and highest accuracy compared to baseline methods\.

### 4\.2Latent World Model

Compared to other world models, can our framework better predict and plan? We compare our method with several baseline approaches in Table[1](https://arxiv.org/html/2608.26190#S4.T1)\. Our method achieves the lowest prediction error and the highest action selection accuracy, demonstrating the effectiveness of compatibility modeling\. Although Dino\-WM also performs prediction in the latent space, it does not use counterfactual action sequences for enhancement\. Moreover, the Dino\-V2 encoder remains frozen to avoid mode collapse during training, which constrains the adaptability of the representation to the navigation task\. As a result, its performance is bounded by the pre\-trained visual features and cannot fully exploit task\-specific dynamics\. Although NWM achieves strong performance in pixel\-space prediction, it performs worst when used for planning with predicted images, yielding the highest error and lowest accuracy\. This suggests that visually plausible pixel\-level generations do not necessarily preserve accurate spatial or geometric consistency, which is critical for robot navigation\.

![Refer to caption](https://arxiv.org/html/2608.26190v1/figure/grpo.png)Figure 2:Policy Reinforcement within the World Model\. Policies starting from different initializations achieve performance improvements when further optimized via GRPO within LWM\.Table 2:Policy Learning Performance\. Evaluation of policies trained under the supervision of different world models demonstrate that LWM provides superior pseudo\-labels for policy imitation\.
### 4\.3Learning and reinforcing policy in LWM

After training the world model, our ultimate objective is to leverage it for policy learning\. We evaluate the effectiveness of different world models for policy training in Table[2](https://arxiv.org/html/2608.26190#S4.T2)\. For NWM, we use the generated image as the pseudo goal image and the ground truth action sequence as the label\. Because the policy model is an autoregressive action generation model, we still report PE and OE as metrics\. Our method achieves the lowest error on both metrics, demonstrating that the learned world model provides reliable supervisory signals for training policies directly from unlabeled video data\. This capability is particularly valuable in robotics, where action annotations are expensive and scarce\.

During the initial policy training stage, we rely on a fixed set of 64 clustered action sequences, which limits the coverage of the action space\. To overcome this limitation, we further improve the policy through reinforcement learning within the world model\. In practice, policy initialization may come from different sources, so we use the LWM\-B\-P world model as a unified reward function to evaluate and optimize different initial policies\. The reinforcement results are shown in Fig\.[2](https://arxiv.org/html/2608.26190#S4.F2)\. We observe performance improvements across all policy initializations, demonstrating that our world model can effectively refine policies regardless of their origin\. This highlights the generality and practical applicability of our method for scalable policy learning from video data\.

![Refer to caption](https://arxiv.org/html/2608.26190v1/figure/real1.jpg)Figure 3:Navigation in real\-world environment\. LWM can learn policy models within the world model\.Table 3:Real\-World Navigation Performance\.
![Refer to caption](https://arxiv.org/html/2608.26190v1/figure/real_indoor.jpg)Figure 4:LWM demonstrates zero\-shot generalization to indoor environments\.

### 4\.4Navigation in the real world

To validate the effectiveness of our policy trained and reinforced within the world model, we deploy it in real\-world robotic environments\. We evaluate our method and baseline approaches across six distinct navigation scenarios\. The average success rate and the LPIPS are reported in Table[3](https://arxiv.org/html/2608.26190#S4.T3)\. Our method achieves the highest success rate and the lowest LPIPS, demonstrating that our world model has rich imagination and spatial knowledge, even in previously unseen environments\. Importantly, our approach does not rely on any action annotations during training\. The policy is learned entirely from video data using the world model\. As illustrated in Fig\.[3](https://arxiv.org/html/2608.26190#S4.F3), our method remains robust even when the goal is located far from the initial position\. The policy consistently maintains alignment with the goal state throughout execution\. In contrast, imitation learning and diffusion\-based policies can perform well in simple scenarios but often fail in more challenging settings, where they tend to lose track of the goal or produce inconsistent trajectories\.

Furthermore, we evaluate our method in indoor environments that are completely absent from the training data\. As shown in Fig\.[3](https://arxiv.org/html/2608.26190#S4.T3), our policy successfully reaches the same goal location from different starting positions\. This behavior indicates that the policy has learned meaningful spatial and geometric relationships, rather than merely memorizing action sequences\. More real\-world robot navigation deployment videos are provided in the supplementary materials\.

Table 4:Ablation on Counterfactual Action Sequences\. Different action sequences can introduce richer comparisons for supervision and improve performance\.
Table 5:Ablation on Loss Functions\. Our proposed method \(MSE with log\) effectively captures spatial relationships and subtle action consequences, achieving the best performance\.

### 4\.5Ablation

We first ablate the effect of modeling interactions among multiple candidate action sequences during training on SCAND dataset using LWM\-B\-P\. Specifically, we vary the number of imagined action sequences evaluated by the world model\. As shown in Table[5](https://arxiv.org/html/2608.26190#S4.T5),M\>1M\>1indicates that the model uses counterfactual reasoning during training which has better performance\. Different action sequences can introduce richer comparisons for supervision and allows the world model to better capture their consequences\. For completeness, more extensive results are presented in the supplementary material\.

We also study different loss functions on LWM dataset using LWM\-B\-P\. As shown in Table[5](https://arxiv.org/html/2608.26190#S4.T5), using a contrastive loss leads to higher prediction error compared to MSE\. This is because contrastive loss only enforces discrete separability between representations, without capturing the magnitude of differences between predicted and target features\. We further introduce a ranking loss to encourage better relative ordering among action sequences\. While this reduces the prediction error compared to pure contrastive loss, it still underperforms MSE loss\. In contrast, our proposed loss achieves the best performance\. This indicates that our loss formulation provides more informative supervision, enabling the model to better distinguish subtle differences between action outcomes and focus on harder action sequences\.

## 5Conclusion

We present the compatibility prediction Latent World Model \(LWM\) for visual navigation\. Instead of reconstructing observations, LWM predicts action\-conditioned latent feature compatibility via cross\-trajectory counterfactual reasoning\. This efficient latent space supports policy supervision from unlabeled videos and imagination\-driven reinforcement learning, significantly outperforming existing world models in real\-world robotic navigation\.

## Acknowledgements

This work was supported in part by the National Natural Science Foundation of China \(No\. U22B2055, 62273345\) and in part by the Beijing Natural Science Foundation \(No\. L223003\)\.

## Supplementary Material

## Appendix ADetails of the Model

The proposed LWM framework comprises two primary components: the world model and the policy model\. In this section, we provide detailed architectural specifications and training hyperparameters for both\.

### A\.1World Model

We evaluate three variants of our world model: LWM\-S utilizes DINOv2\-small[27](https://arxiv.org/html/2608.26190#bib.bib32)as its visual encoder, containing 49M parameters, which matches the capacity of Dino\-WM[45](https://arxiv.org/html/2608.26190#bib.bib20)\. LWM\-B and LWM\-B\-P employ the CroCo\-v2\-base[39](https://arxiv.org/html/2608.26190#bib.bib39)encoder, comprising 217M parameters\.To process the actions, we use a standard Multi\-Layer Perceptron \(MLP\) to embed the 3\-dimensional action inputs\. This MLP features a hidden dimension of 512, an output dimension of 384, and applies a ReLU activation function\. The resulting action embeddings are then treated as tokens and augmented with sinusoidal positional encodings\. For the generative and discriminative components, we employ standard Transformer decoder architectures\. The prediction module consists of 6 decoder layers, 6 attention heads, a feature dimension of 367, and a feed\-forward network \(FFN\) dimension of 2048\. The discrimination module follows a similar configuration but is shallower, comprising 4 decoder layers\. All networks are trained end\-to\-end using the Adam optimizer for 50 epochs\. We set the batch size to 8 and the peak learning rate to5×10−55\\times 10^\{\-5\}, accompanied by a linear warmup period of 5 epochs\.

### A\.2Policy Model

We evaluate three distinct policy architectures\. To ensure a fair comparison across all experiments, we consistently employ the CroCo\-v2\-base[39](https://arxiv.org/html/2608.26190#bib.bib39)visual encoder for all three methods\.

- •NoMaD[34](https://arxiv.org/html/2608.26190#bib.bib31):A robot navigation method based on Diffusion Policy[4](https://arxiv.org/html/2608.26190#bib.bib43)\. The current and goal images are encoded and processed by a Transformer encoder featuring 8 layers, 6 attention heads, and a feature dimension of 384\. The resulting image features are fed into a 1D U\-Net[29](https://arxiv.org/html/2608.26190#bib.bib45)alongside the diffusion timestepttand noisy actions\. The U\-Net, which outputs the estimated noise, utilizes three downsampling and three upsampling blocks\.
- •Behavioral Cloning \(BC\):A baseline navigation method inspired by ViNT[32](https://arxiv.org/html/2608.26190#bib.bib29)and GNM[31](https://arxiv.org/html/2608.26190#bib.bib30)\. Similar to NoMaD, current and goal observations are passed through a Transformer encoder \(8 layers, 6 attention heads, feature dimension of 384\)\. The output features are subsequently processed through a pooling layer and projected into action sequences via an MLP\.
- •LWM Policy \(Ours\):Our approach processes the encoded image features as context for an action decoder\. This decoder is a standard Transformer decoder consisting of 6 layers, 6 attention heads, and a feature dimension of 384\. Analogous to next\-token prediction in natural language processing, the model autoregressively generates action tokens, beginning with a special\[START\]token and concluding upon generating an\[END\]token\.

All policy models are trained using the Adam optimizer for 40 epochs with a batch size of 24\. We use a learning rate of1×10−41\\times 10^\{\-4\}with a linear warmup over the first 4 epochs\.

![Refer to caption](https://arxiv.org/html/2608.26190v1/figure/kmeans_zed2_all.png)Figure 5:The action sequence centers obtained through k\-means clustering\.

## Appendix BAblation of counterfactual action training

In the experimental section, we presented the ablation results of LWM\-B\-P on the SCAND[21](https://arxiv.org/html/2608.26190#bib.bib26)dataset\. Due to space limitations, we present more ablation experimental results on counterfactual training in LWM dataset here\.

Table 6:Ablation on Counterfactual Action Sequences using LWM\-STable 7:Ablation on Counterfactual Action Sequences using LWM\-B\-PFigure[5](https://arxiv.org/html/2608.26190#A1.F5)visualizes the action sequence centers obtained through k\-means clustering\. The quantitative results, presented in Tables[6](https://arxiv.org/html/2608.26190#A2.T6)and[7](https://arxiv.org/html/2608.26190#A2.T7), demonstrate that the absence of counterfactual training severely degrades model performance\. Conversely, the introduction of counterfactual training leads to progressive performance gains that eventually saturate\. This phenomenon indicates that our explicitly defined compatibility labels act as a strong initialization rather than an absolute upper bound\. The model leverages this supervisory signal to inherently refine the distance metric of the feature space\. Consequently, this confirms that a well\-designed inductive bias is instrumental in boosting the model’s performance\.

![Refer to caption](https://arxiv.org/html/2608.26190v1/figure/Compatibility_vis.jpg)Figure 6:The compatibility surface predicted by LWM\.
## Appendix CNavigation in real world

To investigate whether our world model can effectively predict compatibility in real\-world environments, we visualize its predictions on the LWM test dataset, as illustrated in Fig\.[6](https://arxiv.org/html/2608.26190#A2.F6)\. The visualization plots the distance function, where lower values along the z\-axis denote higher compatibility\. Notably, the world model yields a smooth surface, with the maximum compatibility correctly peaking near the ground\-truth value\. This topological smoothness is highly advantageous for policy learning and reinforcement within the world model\. Specifically, as the policy model explores the action space around the ground truth, this smooth landscape provides stable and informative gradients, effectively guiding the optimization process toward the optimal behavior\. We provide the video of robots deployed in the real world in another file\.

## Appendix DLimitations and Future Work

While our imagination\-based world model achieves state\-of\-the\-art performance in navigation tasks, we acknowledge certain boundary conditions\. A core inductive bias of this approach is the assumption that spatial proximity strongly correlates with visual similarity\. While this provides a highly efficient supervisory signal in standard scenarios, marginal spatial displacements in densely cluttered environments with severe occlusions can cause abrupt visual alterations\. Under such partial observability, visual similarity may temporarily decouple from spatial proximity, making compatibility predictions less discriminative\. To address these extreme edge cases, future work could explore integrating 3D\-aware latent representations or long\-term temporal memory, ensuring robust compatibility scores even during sudden, occlusion\-induced visual shifts\.

## References

- N\. Agarwal, A\. Ali, M\. Bala, Y\. Balaji, E\. Barker, T\. Cai, P\. Chattopadhyay, Y\. Chen, Y\. Cui, Y\. Ding,et al\.Cosmos world foundation model platform for physical ai\.arXiv preprint arXiv:2501\.03575\.Cited by:[§2\.1](https://arxiv.org/html/2608.26190#S2.SS1.p1.1)\.
- Baret al\.\(2025\)A\. Bar, G\. Zhou, D\. Tran, T\. Darrell, and Y\. LeCunNavigation world models\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 15791–15801\.Cited by:[§1](https://arxiv.org/html/2608.26190#S1.p1.1),[§1](https://arxiv.org/html/2608.26190#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.26190#S2.SS1.p1.1),[1st item](https://arxiv.org/html/2608.26190#S4.I1.i1.p1.1),[§4\.1\.1](https://arxiv.org/html/2608.26190#S4.SS1.SSS1.p1.1),[§4\.1\.3](https://arxiv.org/html/2608.26190#S4.SS1.SSS3.p1.1)\.
- Cenet al\.\(2025\)J\. Cen, C\. Yu, H\. Yuan, Y\. Jiang, S\. Huang, J\. Guo, X\. Li, Y\. Song, H\. Luo, F\. Wang,et al\.Worldvla: towards autoregressive action world model\.arXiv preprint arXiv:2506\.21539\.Cited by:[§2\.2\.2](https://arxiv.org/html/2608.26190#S2.SS2.SSS2.p1.1)\.
- Chiet al\.\(2025\)C\. Chi, Z\. Xu, S\. Feng, E\. Cousineau, Y\. Du, B\. Burchfiel, R\. Tedrake, and S\. SongDiffusion policy: visuomotor policy learning via action diffusion\.The International Journal of Robotics Research44\(10\-11\),pp\. 1684–1704\.Cited by:[1st item](https://arxiv.org/html/2608.26190#A1.I1.i1.p1.1),[3rd item](https://arxiv.org/html/2608.26190#S4.I1.i3.p1.1)\.
- Dinget al\.\(2025\)J\. Ding, Y\. Zhang, Y\. Shang, Y\. Zhang, Z\. Zong, J\. Feng, Y\. Yuan, H\. Su, N\. Li, N\. Sukiennik,et al\.Understanding world or predicting future? a comprehensive survey of world models\.ACM Computing Surveys58\(3\),pp\. 1–38\.Cited by:[§2](https://arxiv.org/html/2608.26190#S2.p1.1)\.
- Dosovitskiyet al\.\(2020\)A\. Dosovitskiy, L\. Beyer, A\. Kolesnikov, D\. Weissenborn, X\. Zhai, T\. Unterthiner, M\. Dehghani, M\. Minderer, G\. Heigold, S\. Gelly,et al\.An image is worth 16x16 words: transformers for image recognition at scale\.arXiv preprint arXiv:2010\.11929\.Cited by:[§3\.2\.1](https://arxiv.org/html/2608.26190#S3.SS2.SSS1.p2.1)\.
- Driesset al\.\(2023\)D\. Driess, F\. Xia, M\. S\. Sajjadi, C\. Lynch, A\. Chowdhery, B\. Ichter, A\. Wahid, J\. Tompson, Q\. Vuong, T\. Yu,et al\.Palm\-e: an embodied multimodal language model\.arXiv preprint arXiv:2303\.03378\.Cited by:[§2](https://arxiv.org/html/2608.26190#S2.p1.1)\.
- Duet al\.\(2023\)Y\. Du, S\. Yang, B\. Dai, H\. Dai, O\. Nachum, J\. Tenenbaum, D\. Schuurmans, and P\. AbbeelLearning universal policies via text\-guided video generation\.Advances in neural information processing systems36,pp\. 9156–9172\.Cited by:[§2\.1](https://arxiv.org/html/2608.26190#S2.SS1.p1.1)\.
- Gaoet al\.\(2025\)Y\. Gao, H\. Guo, T\. Hoang, W\. Huang, L\. Jiang, F\. Kong, H\. Li, J\. Li, L\. Li, X\. Li,et al\.Seedance 1\.0: exploring the boundaries of video generation models\.arXiv preprint arXiv:2506\.09113\.Cited by:[§2\.1](https://arxiv.org/html/2608.26190#S2.SS1.p1.1)\.
- Georgievet al\.\(2024\)I\. Georgiev, V\. Giridhar, N\. Hansen, and A\. GargPwm: policy learning with multi\-task world models\.arXiv preprint arXiv:2407\.02466\.Cited by:[§2\.2\.1](https://arxiv.org/html/2608.26190#S2.SS2.SSS1.p1.1)\.
- Guoet al\.\(2025\)D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§3\.4](https://arxiv.org/html/2608.26190#S3.SS4.p1.1),[§3\.4](https://arxiv.org/html/2608.26190#S3.SS4.p3.3)\.
- Gutmann and Hyvärinen \(2010\)M\. Gutmann and A\. HyvärinenNoise\-contrastive estimation: a new estimation principle for unnormalized statistical models\.InProceedings of the thirteenth international conference on artificial intelligence and statistics,pp\. 297–304\.Cited by:[§3\.2\.2](https://arxiv.org/html/2608.26190#S3.SS2.SSS2.p5.1)\.
- Ha and Schmidhuber \(2018a\)D\. Ha and J\. SchmidhuberRecurrent world models facilitate policy evolution\.Advances in neural information processing systems31\.Cited by:[§1](https://arxiv.org/html/2608.26190#S1.p1.1),[§2\.2\.1](https://arxiv.org/html/2608.26190#S2.SS2.SSS1.p1.1)\.
- Ha and Schmidhuber \(2018b\)D\. Ha and J\. SchmidhuberWorld models\.arXiv preprint arXiv:1803\.101222\(3\),pp\. 440\.Cited by:[§2\.2\.1](https://arxiv.org/html/2608.26190#S2.SS2.SSS1.p1.1)\.
- Hafneret al\.\(2019a\)D\. Hafner, T\. Lillicrap, J\. Ba, and M\. NorouziDream to control: learning behaviors by latent imagination\.arXiv preprint arXiv:1912\.01603\.Cited by:[§1](https://arxiv.org/html/2608.26190#S1.p2.1),[§2\.2\.1](https://arxiv.org/html/2608.26190#S2.SS2.SSS1.p1.1)\.
- Hafneret al\.\(2019b\)D\. Hafner, T\. Lillicrap, I\. Fischer, R\. Villegas, D\. Ha, H\. Lee, and J\. DavidsonLearning latent dynamics for planning from pixels\.InInternational conference on machine learning,pp\. 2555–2565\.Cited by:[§2\.2\.1](https://arxiv.org/html/2608.26190#S2.SS2.SSS1.p1.1)\.
- Hafneret al\.\(2020\)D\. Hafner, T\. Lillicrap, M\. Norouzi, and J\. BaMastering atari with discrete world models\.arXiv preprint arXiv:2010\.02193\.Cited by:[§2\.2\.1](https://arxiv.org/html/2608.26190#S2.SS2.SSS1.p1.1)\.
- Hoet al\.\(2022\)J\. Ho, T\. Salimans, A\. Gritsenko, W\. Chan, M\. Norouzi, and D\. J\. FleetVideo diffusion models\.Advances in neural information processing systems35,pp\. 8633–8646\.Cited by:[§2](https://arxiv.org/html/2608.26190#S2.p1.1)\.
- Huet al\.\(2023\)A\. Hu, L\. Russell, H\. Yeo, Z\. Murez, G\. Fedoseev, A\. Kendall, J\. Shotton, and G\. CorradoGaia\-1: a generative world model for autonomous driving\.arXiv preprint arXiv:2309\.17080\.Cited by:[§2\.1](https://arxiv.org/html/2608.26190#S2.SS1.p1.1)\.
- Kanget al\.\(2024\)B\. Kang, Y\. Yue, R\. Lu, Z\. Lin, Y\. Zhao, K\. Wang, G\. Huang, and J\. FengHow far is video generation from world model: a physical law perspective\.arXiv preprint arXiv:2411\.02385\.Cited by:[§2\.1](https://arxiv.org/html/2608.26190#S2.SS1.p1.1),[§2\.2\.2](https://arxiv.org/html/2608.26190#S2.SS2.SSS2.p1.1)\.
- Karnanet al\.\(2022\)H\. Karnan, A\. Nair, X\. Xiao, G\. Warnell, S\. Pirk, A\. Toshev, J\. Hart, J\. Biswas, and P\. StoneSocially compliant navigation dataset \(scand\): a large\-scale dataset of demonstrations for social navigation\.IEEE Robotics and Automation Letters7\(4\),pp\. 11807–11814\.Cited by:[Appendix B](https://arxiv.org/html/2608.26190#A2.p1.1),[§4\.1\.1](https://arxiv.org/html/2608.26190#S4.SS1.SSS1.p1.1)\.
- Kimet al\.\(2024\)M\. J\. Kim, K\. Pertsch, S\. Karamcheti, T\. Xiao, A\. Balakrishna, S\. Nair, R\. Rafailov, E\. Foster, G\. Lam, P\. Sanketi,et al\.Openvla: an open\-source vision\-language\-action model\.arXiv preprint arXiv:2406\.09246\.Cited by:[§3\.3](https://arxiv.org/html/2608.26190#S3.SS3.p2.1)\.
- Konget al\.\(2024\)W\. Kong, Q\. Tian, Z\. Zhang, R\. Min, Z\. Dai, J\. Zhou, J\. Xiong, X\. Li, B\. Wu, J\. Zhang,et al\.Hunyuanvideo: a systematic framework for large video generative models\.arXiv preprint arXiv:2412\.03603\.Cited by:[§2\.1](https://arxiv.org/html/2608.26190#S2.SS1.p1.1)\.
- Liet al\.\(2025a\)C\. Li, A\. Krause, and M\. HutterRobotic world model: a neural network simulator for robust policy optimization in robotics\.arXiv preprint arXiv:2501\.10100\.Cited by:[§2\.2\.2](https://arxiv.org/html/2608.26190#S2.SS2.SSS2.p1.1)\.
- Liet al\.\(2025b\)Y\. Li, Y\. Zhu, J\. Wen, C\. Shen, and Y\. XuWorldeval: world model as real\-world robot policies evaluator\.arXiv preprint arXiv:2505\.19017\.Cited by:[§1](https://arxiv.org/html/2608.26190#S1.p1.1)\.
- Nieet al\.\(2025\)D\. Nie, X\. Guo, Y\. Duan, R\. Zhang, and L\. ChenWmnav: integrating vision\-language models into world models for object goal navigation\.In2025 IEEE/RSJ International Conference on Intelligent Robots and Systems \(IROS\),pp\. 2392–2399\.Cited by:[§3\.2\.3](https://arxiv.org/html/2608.26190#S3.SS2.SSS3.p1.1)\.
- Oquabet al\.\(2023\)M\. Oquab, T\. Darcet, T\. Moutakanni, H\. Vo, M\. Szafraniec, V\. Khalidov, P\. Fernandez, D\. Haziza, F\. Massa, A\. El\-Nouby,et al\.Dinov2: learning robust visual features without supervision\.arXiv preprint arXiv:2304\.07193\.Cited by:[§A\.1](https://arxiv.org/html/2608.26190#A1.SS1.p1.1),[§2\.1](https://arxiv.org/html/2608.26190#S2.SS1.p2.1),[§4\.1\.2](https://arxiv.org/html/2608.26190#S4.SS1.SSS2.p1.1)\.
- Renet al\.\(2025\)Z\. Ren, Y\. Wei, X\. Guo, Y\. Zhao, B\. Kang, J\. Feng, and X\. JinVideoworld: exploring knowledge learning from unlabeled videos\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 29029–29039\.Cited by:[§2\.1](https://arxiv.org/html/2608.26190#S2.SS1.p1.1)\.
- Ronnebergeret al\.\(2015\)O\. Ronneberger, P\. Fischer, and T\. BroxU\-net: convolutional networks for biomedical image segmentation\.InInternational Conference on Medical image computing and computer\-assisted intervention,pp\. 234–241\.Cited by:[1st item](https://arxiv.org/html/2608.26190#A1.I1.i1.p1.1)\.
- Shahet al\.\(2021\)D\. Shah, B\. Eysenbach, G\. Kahn, N\. Rhinehart, and S\. LevineRapid exploration for open\-world navigation with latent goal models\.arXiv preprint arXiv:2104\.05859\.Cited by:[§4\.1\.1](https://arxiv.org/html/2608.26190#S4.SS1.SSS1.p1.1)\.
- Shahet al\.\(2023a\)D\. Shah, A\. Sridhar, A\. Bhorkar, N\. Hirose, and S\. LevineGnm: a general navigation model to drive any robot\.In2023 IEEE International Conference on Robotics and Automation \(ICRA\),pp\. 7226–7233\.Cited by:[2nd item](https://arxiv.org/html/2608.26190#A1.I1.i2.p1.1),[4th item](https://arxiv.org/html/2608.26190#S4.I1.i4.p1.1)\.
- Shahet al\.\(2023b\)D\. Shah, A\. Sridhar, N\. Dashora, K\. Stachowicz, K\. Black, N\. Hirose, and S\. LevineViNT: a foundation model for visual navigation\.arXiv preprint arXiv:2306\.14846\.Cited by:[2nd item](https://arxiv.org/html/2608.26190#A1.I1.i2.p1.1),[4th item](https://arxiv.org/html/2608.26190#S4.I1.i4.p1.1)\.
- Silveret al\.\(2017\)D\. Silver, H\. Hasselt, M\. Hessel, T\. Schaul, A\. Guez, T\. Harley, G\. Dulac\-Arnold, D\. Reichert, N\. Rabinowitz, A\. Barreto,et al\.The predictron: end\-to\-end learning and planning\.InInternational Conference on Machine Learning,pp\. 3191–3199\.Cited by:[§2\.2\.1](https://arxiv.org/html/2608.26190#S2.SS2.SSS1.p1.1)\.
- Sridharet al\.\(2024\)A\. Sridhar, D\. Shah, C\. Glossop, and S\. LevineNomad: goal masked diffusion policies for navigation and exploration\.In2024 IEEE International Conference on Robotics and Automation \(ICRA\),pp\. 63–70\.Cited by:[1st item](https://arxiv.org/html/2608.26190#A1.I1.i1.p1.1.1),[3rd item](https://arxiv.org/html/2608.26190#S4.I1.i3.p1.1)\.
- Teamet al\.\(2026\)R\. Team, Z\. Gao, Q\. Wang, Y\. Zeng, J\. Zhu, K\. L\. Cheng, Y\. Li, H\. Wang, Y\. Xu, S\. Ma,et al\.Advancing open\-source world models\.arXiv preprint arXiv:2601\.20540\.Cited by:[§2\.1](https://arxiv.org/html/2608.26190#S2.SS1.p1.1)\.
- Vaswaniet al\.\(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. PolosukhinAttention is all you need\.Advances in neural information processing systems30\.Cited by:[§2](https://arxiv.org/html/2608.26190#S2.p1.1)\.
- Wanet al\.\(2025\)T\. Wan, A\. Wang, B\. Ai, B\. Wen, C\. Mao, C\. Xie, D\. Chen, F\. Yu, H\. Zhao, J\. Yang,et al\.Wan: open and advanced large\-scale video generative models\.arXiv preprint arXiv:2503\.20314\.Cited by:[§2\.1](https://arxiv.org/html/2608.26190#S2.SS1.p1.1)\.
- Wanget al\.\(2025\)Z\. Wang, H\. Hu, W\. Gao, and S\. ShenCausal enhanced autoregressive model for monocular image\-goal navigation in unknown map environment\.IEEE Robotics and Automation Letters\.Cited by:[§3\.3](https://arxiv.org/html/2608.26190#S3.SS3.p2.1)\.
- Weinzaepfelet al\.\(2023\)P\. Weinzaepfel, T\. Lucas, V\. Leroy, Y\. Cabon, V\. Arora, R\. Brégier, G\. Csurka, L\. Antsfeld, B\. Chidlovskii, and J\. RevaudCroCo v2: Improved Cross\-view Completion Pre\-training for Stereo Matching and Optical Flow\.InICCV,Cited by:[§A\.1](https://arxiv.org/html/2608.26190#A1.SS1.p1.1),[§A\.2](https://arxiv.org/html/2608.26190#A1.SS2.p1.1),[§4\.1\.2](https://arxiv.org/html/2608.26190#S4.SS1.SSS2.p1.1)\.
- Xiaoet al\.\(2025\)J\. Xiao, Y\. Yang, X\. Chang, R\. Chen, F\. Xiong, M\. Xu, W\. Zheng, and Q\. ZhangWorld\-env: leveraging world model as a virtual environment for vla post\-training\.arXiv preprint arXiv:2509\.24948\.Cited by:[§2\.2\.2](https://arxiv.org/html/2608.26190#S2.SS2.SSS2.p1.1)\.
- Yaoet al\.\(2025\)X\. Yao, J\. Gao, and C\. XuNavmorph: a self\-evolving world model for vision\-and\-language navigation in continuous environments\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 5536–5546\.Cited by:[§3\.2\.3](https://arxiv.org/html/2608.26190#S3.SS2.SSS3.p1.1)\.
- Yeet al\.\(2026\)S\. Ye, Y\. Ge, K\. Zheng, S\. Gao, S\. Yu, G\. Kurian, S\. Indupuru, Y\. L\. Tan, C\. Zhu, J\. Xiang,et al\.World action models are zero\-shot policies\.arXiv preprint arXiv:2602\.15922\.Cited by:[§1](https://arxiv.org/html/2608.26190#S1.p1.1),[§2\.2\.2](https://arxiv.org/html/2608.26190#S2.SS2.SSS2.p1.1)\.
- Zhanget al\.\(2018\)R\. Zhang, P\. Isola, A\. A\. Efros, E\. Shechtman, and O\. WangThe unreasonable effectiveness of deep features as a perceptual metric\.InProceedings of the IEEE conference on computer vision and pattern recognition,pp\. 586–595\.Cited by:[§3\.1](https://arxiv.org/html/2608.26190#S3.SS1.p4.1),[§3\.2\.2](https://arxiv.org/html/2608.26190#S3.SS2.SSS2.p5.1),[1st item](https://arxiv.org/html/2608.26190#S4.I1.i1.p1.1)\.
- Zhenet al\.\(2024\)H\. Zhen, X\. Qiu, P\. Chen, J\. Yang, X\. Yan, Y\. Du, Y\. Hong, and C\. Gan3d\-vla: a 3d vision\-language\-action generative world model\.arXiv preprint arXiv:2403\.09631\.Cited by:[§2\.1](https://arxiv.org/html/2608.26190#S2.SS1.p1.1)\.
- Zhouet al\.\(2024\)G\. Zhou, H\. Pan, Y\. LeCun, and L\. PintoDino\-wm: world models on pre\-trained visual features enable zero\-shot planning\.arXiv preprint arXiv:2411\.04983\.Cited by:[§A\.1](https://arxiv.org/html/2608.26190#A1.SS1.p1.1),[§2\.1](https://arxiv.org/html/2608.26190#S2.SS1.p2.1),[2nd item](https://arxiv.org/html/2608.26190#S4.I1.i2.p1.1),[§4\.1\.2](https://arxiv.org/html/2608.26190#S4.SS1.SSS2.p1.1)\.