Does Latent Planning Survive Point Clouds? Action-Conditioned JEPA World Models for Geometric Observations

# Does Latent Planning Survive Point Clouds? Action-Conditioned JEPA World Models for Geometric Observations
Source: [https://arxiv.org/html/2608.29434](https://arxiv.org/html/2608.29434)
Fabio F\. Oberweger††thanks:Joint first authors: F\.F\.O\. led algorithm development, M\.S\. led data generation and environment design\.Affiliation:AIT Austrian Institute of TechnologyAffiliation:Assistive & Autonomous SystemsEmail:[fabio\.oberweger@ait\.ac\.at](mailto:)Michael Schwingshackl\*Affiliation:AIT Austrian Institute of TechnologyAffiliation:Assistive & Autonomous SystemsEmail:[michael\.schwingshackl@ait\.ac\.at](mailto:)

###### Abstract

JEPA world models make latent\-space planning a practical route to control, but they are built almost exclusively on images\. Whether latent prediction survives geometric observations is unclear: point clouds are sparse, unordered, and self\-occluded, and with 0\.3–15% of scene points moving, the slow\-feature optimum of latent prediction compounds with the geometric shortcut of 3D self\-supervision\. We lift three canonical JEPA designs to point clouds, frozen\-encoder, distribution\-prior, and action\-sensitive, and re\-sense thestable\-worldmodelbenchmark so that only the observation differs from the image baselines\. All three plan without collapse: the distribution\-prior model is statistically equivalent to its re\-evaluated image counterpart on every benchmark, and the action\-sensitive model attains the strongest result in our controlled comparison where the most geometry moves\. Probing explains why: object positions are almost perfectly linearly decodable and attention falls on the few moving points\. Planning withstands heavy dropout never seen in training, though range noise defeats the thinnest scene\. Geometry finally makes a commanded 3D target a natural goal interface: we construct the goal latent from the target and the current latent, at no cost in success rate, without a goal observation\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2608.29434v1/scene_pairs.png)Figure 1:From image environments to point cloud environments\.Top: the 3D scene\. Bottom: the generated point cloud of that same scene, which replaces the image as the observation\.A robot that can predict the consequences of its own actions need not be programmed for each task: control reduces to search\. World models promise this for physical AI\[[11](https://arxiv.org/html/2608.29434#bib.bib8),[12](https://arxiv.org/html/2608.29434#bib.bib9),[18](https://arxiv.org/html/2608.29434#bib.bib13)\]: learn dynamics from raw experience, offline and reward\-free, and plan new behaviors at test time\. Joint\-embedding predictive architectures \(JEPAs\) made this practical by predicting future*latent*states conditioned on actions rather than generating observations, whether over frozen features \(DINO\-WM\[[48](https://arxiv.org/html/2608.29434#bib.bib36)\]\), distribution\-prior \(LeWorldModel\[[21](https://arxiv.org/html/2608.29434#bib.bib15)\]\), or action\-sensitive \(Delta\-JEPA\[[46](https://arxiv.org/html/2608.29434#bib.bib33)\], SMWM\[[16](https://arxiv.org/html/2608.29434#bib.bib12)\]\)\. Yet almost all of it sees the world through a camera\.

Images are a rational default, with cheap sensors, dense semantics, and transferable foundation models\[[33](https://arxiv.org/html/2608.29434#bib.bib22)\]\. But many robots perceive the world geometrically, and range perception has matured to match, recovering 3D curves\[[26](https://arxiv.org/html/2608.29434#bib.bib20)\]and 6\-DoF object poses\[[32](https://arxiv.org/html/2608.29434#bib.bib23)\]from raw scans\. What is missing is*prediction*, a world model over geometric observations that a planner can interrogate, and it is not obvious that latent prediction survives the move\. Latent prediction admits a trivial optimum at temporally constant features\[[34](https://arxiv.org/html/2608.29434#bib.bib24)\], unusually accessible when only 1–15% of scene points move\[[15](https://arxiv.org/html/2608.29434#bib.bib11)\]\. Independently, 3D self\-supervision admits a geometric shortcut onto point height and surface normals, since coordinates enter the operators directly and cannot be masked\[[39](https://arxiv.org/html/2608.29434#bib.bib29)\]\. A point\-cloud JEPA thus faces two routes to collapse where an image JEPA faces one, and whether the known anti\-collapse mechanisms hold is an empirical question\.

Existing 3D world models do not answer it: driving models generate future scans or occupancy judged by geometric fidelity\[[44](https://arxiv.org/html/2608.29434#bib.bib32),[47](https://arxiv.org/html/2608.29434#bib.bib35),[41](https://arxiv.org/html/2608.29434#bib.bib30),[19](https://arxiv.org/html/2608.29434#bib.bib14)\], manipulation models decode explicit future geometry\[[15](https://arxiv.org/html/2608.29434#bib.bib11),[29](https://arxiv.org/html/2608.29434#bib.bib21)\], and AD\-L\-JEPA\[[51](https://arxiv.org/html/2608.29434#bib.bib37)\]predicts latents but is action\-free\. To our knowledge, no prior work learns an action\-conditioned latent\-prediction world model on point clouds and plans with it\.

We build that missing piece\. We lift the three canonical JEPA designs to point clouds, isolating each known anti\-collapse mechanism, and evaluate them against their image counterparts by re\-sensing thestable\-worldmodelsuite\[[20](https://arxiv.org/html/2608.29434#bib.bib16)\]with a simulated range sensor\.

Finally, latent planners require an encoded*goal observation*at inference, for a range sensor a scan of a configuration never visited\. Geometry affords a natural interface: we learn to construct the goal latent from the current latent and a commanded 3D target, so the planner is told where an object should end up rather than shown a picture of the world once it is there\. Our contributions:

- •Point\-cloud JEPA world models: frozen\-encoder \(Utonia\-WM, a novel variant over the frozen Utonia encoder\[[45](https://arxiv.org/html/2608.29434#bib.bib34)\]\), distribution\-prior \(Point\-LeWM\), and action\-sensitive \(Point\-Delta\-JEPA\), plus Vox\-WM, a parameter\-free geometric ablation that Utonia\-WM outperforms\. On the sparse, near\-static scans none collapses: positions stay linearly readable from the latents, attention settles on the moving entities, and planning largely withstands sensor degradation never seen in training\.
- •A controlled modality comparison: a re\-sensedstable\-worldmodelbenchmark where only the observation differs from the image datasets, verified by replay\. On it, Point\-LeWM is statistically equivalent to the image baseline on every benchmark, and Point\-Delta\-JEPA exceeds it where the most geometry moves \(above even its image counterpart’s reported, uncontrolled numbers\)\.
- •Planning toward 3D targets: a module predicting the goal latent from the current latent and a target object pose, replacing the inference\-time goal observation without losing success\.

## 2Related Work

![Refer to caption](https://arxiv.org/html/2608.29434v1/pointencoder.png)Figure 2:Encoder of Point\-LeWMandPoint\-Delta\-JEPA: FPS \+ ball query \(only a few groups are shown\), PointNet\[[30](https://arxiv.org/html/2608.29434#bib.bib45)\]tokenizer, and ViT\-Tiny\[[8](https://arxiv.org/html/2608.29434#bib.bib39)\]whose CLS token yieldsztz\_\{t\}\. The next latentzt\+1z\_\{t\+1\}, from the same encoder, is matched toz^t\+1\\hat\{z\}\_\{t\+1\}via MSE\. SIGReg\[[21](https://arxiv.org/html/2608.29434#bib.bib15)\]\(Point\-LeWM\) and the Action\-Reconstruction\-Loss\[[16](https://arxiv.org/html/2608.29434#bib.bib12),[46](https://arxiv.org/html/2608.29434#bib.bib33)\]\(Point\-Delta\-JEPA\) prevent collapse\.##### JEPA world models\.

Where classical world models train by reconstruction\[[11](https://arxiv.org/html/2608.29434#bib.bib8),[12](https://arxiv.org/html/2608.29434#bib.bib9)\]or reward prediction\[[13](https://arxiv.org/html/2608.29434#bib.bib10)\], JEPAs\[[18](https://arxiv.org/html/2608.29434#bib.bib13)\]train an action\-conditioned predictor in representation space: DINO\-WM\[[48](https://arxiv.org/html/2608.29434#bib.bib36)\]established the frozen\-encoder variant, PLDM\[[35](https://arxiv.org/html/2608.29434#bib.bib25)\]demonstrated end\-to\-end latent planning, and LeWorldModel\[[21](https://arxiv.org/html/2608.29434#bib.bib15)\]reduced end\-to\-end training to latent prediction plus one distributional regularizer \(SIGReg\[[5](https://arxiv.org/html/2608.29434#bib.bib5)\]\)\. Later work adds temporal\-difference targets\[[4](https://arxiv.org/html/2608.29434#bib.bib4)\], test\-time adaptation\[[37](https://arxiv.org/html/2608.29434#bib.bib26)\], action\-sensitivity\[[10](https://arxiv.org/html/2608.29434#bib.bib7)\], and video\-model post\-training\[[3](https://arxiv.org/html/2608.29434#bib.bib3)\]\. Thestable\-worldmodelplatform\[[20](https://arxiv.org/html/2608.29434#bib.bib16)\]consolidates these models and planners into the interface we re\-sense\. Two failure modes organize this space\. Against the slow\-feature optimum\[[34](https://arxiv.org/html/2608.29434#bib.bib24)\], remedies range from distributional regularization\[[5](https://arxiv.org/html/2608.29434#bib.bib5),[43](https://arxiv.org/html/2608.29434#bib.bib31)\]and value heads\[[13](https://arxiv.org/html/2608.29434#bib.bib10)\]to an inverse\-dynamics loss, that SMWM\[[16](https://arxiv.org/html/2608.29434#bib.bib12)\]and Delta\-JEPA\[[46](https://arxiv.org/html/2608.29434#bib.bib33)\]parameterize on the latent displacement\. Against one\-step training degrading under multi\-step rollouts, remedies include latent overshooting, multi\-step targets, and exposure to the model’s own predictions\[[12](https://arxiv.org/html/2608.29434#bib.bib9),[3](https://arxiv.org/html/2608.29434#bib.bib3),[4](https://arxiv.org/html/2608.29434#bib.bib4),[24](https://arxiv.org/html/2608.29434#bib.bib18)\]\. All are measured on images or proprioceptive states\.

##### World models on point clouds\.

The existing lanes never plan in latent space\.*Manipulation*models regress future geometry: PointWorld\[[15](https://arxiv.org/html/2608.29434#bib.bib11)\]predicts per\-point displacements and drives MPPI on a real arm, 3D Point World Models\[[29](https://arxiv.org/html/2608.29434#bib.bib21)\]place point completion before the dynamics, and both pay their planning cost on a decoded cloud\.*Driving*models are generative: Copilot4D\[[44](https://arxiv.org/html/2608.29434#bib.bib32)\]and OccWorld\[[47](https://arxiv.org/html/2608.29434#bib.bib35)\]synthesize future scans or occupancy, ViDAR\[[41](https://arxiv.org/html/2608.29434#bib.bib30)\]forecasts point clouds as pretraining, and LiSTAR\[[19](https://arxiv.org/html/2608.29434#bib.bib14)\]shows how strongly sensor geometry shapes architecture, but none searches over candidate actions\. Closest to us, AD\-L\-JEPA\[[51](https://arxiv.org/html/2608.29434#bib.bib37)\]and its temporal extension\[[50](https://arxiv.org/html/2608.29434#bib.bib38)\]bring masked*latent*prediction to automotive scans, the latter validating SIGReg on 3D latents, yet both are evaluated by fine\-tuning and neither conditions on actions\. FR3D\[[24](https://arxiv.org/html/2608.29434#bib.bib18)\]predicts in a frozen 3D reconstruction latent space but is monocular and action\-free\. The frozen\-encoder recipe also presupposes a general\-purpose 3D encoder: Sonata\[[39](https://arxiv.org/html/2608.29434#bib.bib29)\]identifies the geometric shortcut of Section[1](https://arxiv.org/html/2608.29434#S1)in 3D self\-supervision, and Utonia\[[45](https://arxiv.org/html/2608.29434#bib.bib34)\]trains a 3D foundation model across point\-cloud domains\.

##### Goal specification and planning\.

Latent world models plan by sampling\-based MPC\[[12](https://arxiv.org/html/2608.29434#bib.bib9),[13](https://arxiv.org/html/2608.29434#bib.bib10),[48](https://arxiv.org/html/2608.29434#bib.bib36)\], and in the reward\-free setting the goal enters as an*encoded goal observation*\[[20](https://arxiv.org/html/2608.29434#bib.bib16)\]\. This interface degrades sharply with goal horizon\[[22](https://arxiv.org/html/2608.29434#bib.bib17),[6](https://arxiv.org/html/2608.29434#bib.bib6)\], and it presumes an observation of the goal state exists at all\. FF\-JEPA\[[22](https://arxiv.org/html/2608.29434#bib.bib17)\]removes goal images by forecasting latent subgoals, but its objective is implicit in the demonstrations, so the planner cannot be commanded to an arbitrary target\. PRISM\[[38](https://arxiv.org/html/2608.29434#bib.bib27)\]and SAGE\[[6](https://arxiv.org/html/2608.29434#bib.bib6)\]keep the encoded goal and improve the search, and[Nguyen et al\. \[25\]](https://arxiv.org/html/2608.29434#bib.bib19)amortize that search into a goal\- and horizon\-conditioned inverse\-dynamics model\. In 3D, PointWorld\[[15](https://arxiv.org/html/2608.29434#bib.bib11)\]plans toward target positions for a selected subset of points, in effect a sparse goal cloud with known correspondences, a cost only a model predicting explicit geometry can evaluate\. Outside world models, coordinate goals are standard, from goal\-conditioned RL\[[2](https://arxiv.org/html/2608.29434#bib.bib2)\]to PointGoal navigation\[[1](https://arxiv.org/html/2608.29434#bib.bib1)\]\. We investigate another way: retain an explicit goal specification, represent it as the 3D pose of a task\-relevant object, and learn to construct the goal latent the planners above lack without a goal observation\.

![Refer to caption](https://arxiv.org/html/2608.29434v1/utoniawm.png)Figure 3:Utonia\-WM: frozen Utonia\[[40](https://arxiv.org/html/2608.29434#bib.bib28),[45](https://arxiv.org/html/2608.29434#bib.bib34)\]encodes the point cloud into super\-point features, which a fixed canonical\-frame voxel grid bins into≈256\{\\approx\}256tokens \(only a few cells shown\)\. Per cell, features are pooled and compressed by a fixed random orthogonal projection to the predictor width\. Only the predictor trains, with a token\-wise MSE against the frozen targetszt\+1z\_\{t\+1\}\.

## 3Methods

Section[3\.1](https://arxiv.org/html/2608.29434#S3.SS1)equips thestable\-worldmodelenvironments with simulated LiDAR\. Section[3\.2](https://arxiv.org/html/2608.29434#S3.SS2)instantiates the three canonical JEPA world\-model designs on the resulting point clouds and presents a complementary module that removes the need for a goal observation at inference time\.

### 3\.1Point\-cloud sensing and dataset replication

To compare observation modalities rather than datasets we re\-sense the released image benchmarks instead of collecting new data: a raycast range sensor is attached to the simulator behind each original dataset, every recorded state is replayed, and the resulting cloud is stored row\-aligned with the original observation\. Episodes, lengths, actions and goals are identical to the image datasets\. Table[1](https://arxiv.org/html/2608.29434#S3.T1)states per environment what else changes\.

Table 1:The four re\-sensed datasets\.*Ground share*and*moving returns*are the fractions of returns on the ground plane and on non\-static entities\. Both are constant within an environment\.##### Sensor\.

The sensor casts a fixed100×100100\\times 100grid of rays matched to its mount camera’s field of view, recording each ray’s first intersection within a per\-environment maximum range\. Returns are expressed in the sensor frame \(xxforward,yyleft,zzup\), and rays intersecting nothing take a sentinel value, so each frame is a fixed\(10,000,3\)\(10\{,\}000,3\)array\. The sensor is idealized, exact intersections, no noise, dropout or sweep distortion, so the comparison isolates observation geometry\. Only the OGBench cube\[[27](https://arxiv.org/html/2608.29434#bib.bib48)\]supplies a mountable camera\. The others offer a top\-down view alone, from which a scan is degenerate, so we place the sensor obliquely at45∘45^\{\\circ\}azimuth and elevation, derived from the scene’s bounding sphere rather than tuned, a harder view than the top\-down render the image baselines see\.

##### Dataset replication\.

Conversion writes each recorded state onto the environment and casts a scan without stepping physics\. The output table carries every source column alongside the cloud\. Two\-Room and Push\-T\[[7](https://arxiv.org/html/2608.29434#bib.bib49)\]are not 3D simulators, so we leave each engine unmodified and mirror its state into a lightweight MuJoCo\[[36](https://arxiv.org/html/2608.29434#bib.bib50)\]scene serving only as geometry for the sensor \(Appendix[C](https://arxiv.org/html/2608.29434#A3)\)\. Dynamics, rewards and termination remain the original environment’s\. Ground returns dominate every cloud and only 0\.3–15% fall on moving entities \(Table[1](https://arxiv.org/html/2608.29434#S3.T1)\), the regime Section[1](https://arxiv.org/html/2608.29434#S1)identifies as doubly hazardous for latent prediction\. At evaluation the same sensor is cast from each state the planner visits\.

### 3\.2Point cloud world models and planning toward 3D targets

We adopt the offline, reward\-free setting of prior JEPA world models\[[16](https://arxiv.org/html/2608.29434#bib.bib12),[21](https://arxiv.org/html/2608.29434#bib.bib15),[48](https://arxiv.org/html/2608.29434#bib.bib36)\]: the training data is a set ofnnaction\-interleaved trajectories𝒟=\{\(o1,a1,…,aTi−1,oTi\)\}i=1n\\mathcal\{D\}=\\\{\(o\_\{1\},a\_\{1\},\\dots,a\_\{T\_\{i\}\-1\},o\_\{T\_\{i\}\}\)\\\}\_\{i=1\}^\{n\}, without rewards or task labels\. Each observationoto\_\{t\}is a point cloud, an unordered point set inℝ3\\mathbb\{R\}^\{3\}of frame\-varying size, and each actionata\_\{t\}a continuous control input\. At test time the model must plan actions that reach a specified goal, conventionally given as a*goal observation*ogo\_\{g\}\[[20](https://arxiv.org/html/2608.29434#bib.bib16),[48](https://arxiv.org/html/2608.29434#bib.bib36)\]\. Our target\-to\-latent module will instead assume training\-time annotationsctc\_\{t\}of low\-dimensional 3D targets, the only supervision used in this paper\.

#### 3\.2\.1Model architecture

All models share the JEPA template\[[18](https://arxiv.org/html/2608.29434#bib.bib13)\]: an encoderfθf\_\{\\theta\}maps each observation to a latentzt=fθ​\(ot\)∈𝒵z\_\{t\}=f\_\{\\theta\}\(o\_\{t\}\)\\in\\mathcal\{Z\}, and a predictorpϕp\_\{\\phi\}predicts the next latent from a context ofℓ\\elllatents and actions,z^t\+1=pϕ\(zt−ℓ\+1:t,at−ℓ\+1:t\)\\hat\{z\}\_\{t\+1\}=p\_\{\\phi\}\(z\_\{t\-\\ell\+1:t\},a\_\{t\-\\ell\+1:t\}\)\. Observations are never reconstructed\. Point\-LeWM and Point\-Delta\-JEPA are trained end\-to\-end with an*identical*encoder and predictor and differ only in the objective, which isolates the anti\-collapse mechanism\. Utonia\-WM trains no encoder at all\.

![Refer to caption](https://arxiv.org/html/2608.29434v1/target2latent.png)Figure 4:Learning to plan toward 3D targets\.The target\-to\-latent module learns to reproduce the frozen encoder’s latent of a stored goal observation \(left\) from a task\-relevant 3D target, in the cube environment shown here simply thex​y​zxyz\-position of the cube\. At test time the 3D target is supplied directly, replacing the need for a goal observation\.##### Point\-LeWM & Point\-Delta\-JEPA\.

For the end\-to\-end pair,fθf\_\{\\theta\}is the point\-cloud analogue of LeWorldModel’s pixel encoder \(Figure[2](https://arxiv.org/html/2608.29434#S2.F2)\): a Point\-BERT\-style set tokenizer\[[42](https://arxiv.org/html/2608.29434#bib.bib42),[49](https://arxiv.org/html/2608.29434#bib.bib43)\]feeds the image model’s ViT\-Tiny trunk\[[8](https://arxiv.org/html/2608.29434#bib.bib39)\], whose CLS readout gives𝒵=ℝ192\\mathcal\{Z\}=\\mathbb\{R\}^\{192\}\. The predictor is LeWorldModel’s causal action\-conditioned transformer\[[21](https://arxiv.org/html/2608.29434#bib.bib15)\]\. Three choices depart from recognition practice\. Coordinates are normalized by a*fixed*workspace affine rather than per cloud, since per\-cloud statistics would erase the absolute object positions that constitute the task signal\. Points are grouped by metric radius\[[31](https://arxiv.org/html/2608.29434#bib.bib41)\]rather than bykk\-NN, so every token covers a fixed physical scale\. Finally, the encoder uses LayerNorm exclusively, R3D’s fix\[[14](https://arxiv.org/html/2608.29434#bib.bib44)\]for the BatchNorm\-induced 3D “scaling paradox” in control\.

##### Utonia\-WM & Vox\-WM\.

Utonia\-WM \(Figure[3](https://arxiv.org/html/2608.29434#S2.F3)\) follows DINO\-WM\[[48](https://arxiv.org/html/2608.29434#bib.bib36)\]: a frozen pretrained encoder supplies the state, and only the predictor trains\. Our frozen encoder is Utonia\[[45](https://arxiv.org/html/2608.29434#bib.bib34)\], a Point Transformer V3\[[40](https://arxiv.org/html/2608.29434#bib.bib28)\]consuming point coordinates alone\. Unlike an image backbone’s patch tokens, Utonia’s super\-points carry no persistent identity across frames, so DINO\-WM’s token\-wise prediction is undefined, and we impose the missing structure deterministically: each point cloud is expressed in a fixed gravity\-aligned frame, encoded by the frozen backbone, and pooled into a fixed grid of canonical\-frame cells,≈256\{\\approx\}256tokens \(Appendix[A](https://arxiv.org/html/2608.29434#A1)\)\. Each pooled token is then compressed from its 1152 pooled dimensions to the 192\-dimensional predictor width by a fixed random orthogonal projection \(Appendix[A](https://arxiv.org/html/2608.29434#A1)\), training\-free and approximately distance\-preserving in the sense of the Johnson–Lindenstrauss lemma\[[17](https://arxiv.org/html/2608.29434#bib.bib46)\]\. These tokens are predicted by DINO\-WM’s frame\-block\-causal transformer\[[48](https://arxiv.org/html/2608.29434#bib.bib36)\]\. Nothing downstream of the backbone is trainable, so the frozen latent space cannot collapse under the prediction loss\. Vox\-WM ablates the frozen encoder at fixed tokenization: it keeps the canonical frame, the token grid, and the predictor, but each cell’s token holds 25 plain statistics of the cell’s points \(occupancy, count, and per\-axis moments, Appendix[A](https://arxiv.org/html/2608.29434#A1)\) instead of pooled frozen features, so whatever Utonia\-WM gains over Vox\-WM is attributable to the learned features rather than to the voxel representation itself\.

##### Joint training objectives\.

All models minimize the teacher\-forced latent prediction lossℒpred=𝔼𝒟\[∥pϕ\(zt−ℓ\+1:t,at−ℓ\+1:t\)−zt\+1∥22\]\\mathcal\{L\}\_\{\\mathrm\{pred\}\}=\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\bigl\[\\\|p\_\{\\phi\}\(z\_\{t\-\\ell\+1:t\},a\_\{t\-\\ell\+1:t\}\)\-z\_\{t\+1\}\\\|\_\{2\}^\{2\}\\bigr\]\[[48](https://arxiv.org/html/2608.29434#bib.bib36),[21](https://arxiv.org/html/2608.29434#bib.bib15)\], i\.e\. the predicted next latent must match the encoded next observation\. Trained end\-to\-end, this loss alone is degenerate, since a constant encoder attains zero loss\[[34](https://arxiv.org/html/2608.29434#bib.bib24)\]\. Our three models occupy the three known escape routes\. Point\-LeWM optimizesℒpred\+λ​ΩSIGReg​\(θ\)\\mathcal\{L\}\_\{\\mathrm\{pred\}\}\+\\lambda\\,\\Omega\_\{\\mathrm\{SIGReg\}\}\(\\theta\), where the regularizer SIGReg\[[5](https://arxiv.org/html/2608.29434#bib.bib5),[21](https://arxiv.org/html/2608.29434#bib.bib15)\], weighted byλ\\lambda, drives the latent distribution toward an isotropic Gaussian, which a collapsed encoder cannot produce\. Point\-Delta\-JEPA instead optimizesℒpred\+λ𝔼𝒟\[1N∥hψ\(zt\+N−zt\)−at:t\+N−1∥22\]\\mathcal\{L\}\_\{\\mathrm\{pred\}\}\+\\lambda\\,\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\bigl\[\\tfrac\{1\}\{N\}\\\|h\_\{\\psi\}\(z\_\{t\+N\}\-z\_\{t\}\)\-a\_\{t:t\+N\-1\}\\\|\_\{2\}^\{2\}\\bigr\], in which a training\-only decoderhψh\_\{\\psi\}, discarded after training, must recover theN=5N\{=\}5executed actions from the net latent displacement, forcing latents that discriminate between actions\[[46](https://arxiv.org/html/2608.29434#bib.bib33),[16](https://arxiv.org/html/2608.29434#bib.bib12)\]\. Utonia\-WM needs neither term, since only the predictor is trained, against the fixed targets of the frozen encoder\[[48](https://arxiv.org/html/2608.29434#bib.bib36)\], at the price that the representation cannot adapt to the control task\.

##### Implementation\.

All models are implemented in PyTorch\[[28](https://arxiv.org/html/2608.29434#bib.bib51)\]on a deliberately modest budget: every world model trains on two A100 GPUs\. The shared end\-to\-end encoder is a ViT\-Tiny trunk of≈5\.5\{\\approx\}5\.5M parameters with a 192\-dimensional latent, trained for 10 epochs, whereas Utonia\-WM pairs its frozen≈137\{\\approx\}137M\-parameter backbone with a predictor trained for 100 epochs on precomputed features\. All models step at frameskip 5 and plan from a context ofℓ=3\\ell=3past latents\. Architecture and optimization constants otherwise follow the image pipelines\[[21](https://arxiv.org/html/2608.29434#bib.bib15),[48](https://arxiv.org/html/2608.29434#bib.bib36)\]\(Appendix[A](https://arxiv.org/html/2608.29434#A1)\)\.

#### 3\.2\.2Planning toward 3D targets

Planning follows the pixel models\[[21](https://arxiv.org/html/2608.29434#bib.bib15),[48](https://arxiv.org/html/2608.29434#bib.bib36)\]: receding\-horizon MPC in latent space, minimizing the distance∥z^t\+H−zg∥22\\lVert\\hat\{z\}\_\{t\+H\}\-z\_\{g\}\\rVert\_\{2\}^\{2\}to a goal latentzgz\_\{g\}over action sequences of horizonHHwith the cross\-entropy method, wherez^t\+H\\hat\{z\}\_\{t\+H\}is the autoregressive rollout ofpϕp\_\{\\phi\}\. Planner and CEM hyperparameters are inherited unchanged from the image pipelines \(Appendix[A](https://arxiv.org/html/2608.29434#A1)\), so the comparison never rests on planner tuning\. Conventionally, the goal latent is an encoded goal observation,zg=fθ​\(og\)z\_\{g\}=f\_\{\\theta\}\(o\_\{g\}\), for LiDAR a complete scan of a configuration the robot has never visited\. What is actually available in practice\[[32](https://arxiv.org/html/2608.29434#bib.bib23)\]is a low\-dimensional 3D targetcg∈𝒞c\_\{g\}\\in\\mathcal\{C\}, instantiated per environment as the cube position, the agent position \(Two\-Room\), or agent and block positions plus the block’s planar angle \(Push\-T\)\. A target alone need not index a latent:fθf\_\{\\theta\}encodes the entire scene whilecgc\_\{g\}constrains only the task\-relevant coordinates, so the map from target to latent is one\-to\-many\. Conditioning on the*current*latentztz\_\{t\}, which supplies exactly the scene content the target leaves unconstrained, resolves this ambiguity\. Without it, the regression target of Eq\. \([1](https://arxiv.org/html/2608.29434#S3.E1)\) averages over every scene consistent with the target and need not be the latent of*any*reachable observation \(Appendix[B](https://arxiv.org/html/2608.29434#A2)\)\. We evaluate both variants in Table[3](https://arxiv.org/html/2608.29434#S4.T3)\. Withfθf\_\{\\theta\}frozen, we trainmξ:𝒞×𝒵→𝒵m\_\{\\xi\}:\\mathcal\{C\}\\times\\mathcal\{Z\}\\to\\mathcal\{Z\}on pairs of frames from the same trajectory, separated by a random offsetk≤kmaxk\\leq k\_\{\\max\}:

ℒgoal\(ξ\)=𝔼\(ot,ct\+k,ot\+k\)∼𝒟,k∼𝒰⁡\{1,…,kmax\}\[∥mξ\(ct\+k,zt\)−zt\+k∥22\]\.\\mathcal\{L\}\_\{\\mathrm\{goal\}\}\(\\xi\)=\\mathbb\{E\}\_\{\(o\_\{t\},\\,c\_\{t\+k\},\\,o\_\{t\+k\}\)\\sim\\mathcal\{D\},\\;k\\sim\\mathcal\{U\}\\\{1,\\dots,k\_\{\\max\}\\\}\}\\Bigl\[\\bigl\\lVert m\_\{\\xi\}\(c\_\{t\+k\},z\_\{t\}\)\-z\_\{t\+k\}\\bigr\\rVert\_\{2\}^\{2\}\\Bigr\]\.\(1\)We instantiatemξm\_\{\\xi\}either as a residual MLP over Fourier\-featurized target coordinates, which fits the conditional mean of Eq\. \([1](https://arxiv.org/html/2608.29434#S3.E1)\), or as a shortcut model\[[9](https://arxiv.org/html/2608.29434#bib.bib47)\], a generative alternative that sampleszgz\_\{g\}from the full distribution of latents consistent with a target, in a single step at the MLP’s inference cost \(architectures and training constants in Appendix[A](https://arxiv.org/html/2608.29434#A1)\)\. The unconditioned variant dropsztz\_\{t\}from both the module and the loss\. At test timezg=mξ​\(cg,zt\)z\_\{g\}=m\_\{\\xi\}\(c\_\{g\},z\_\{t\}\)orzg=mξ​\(cg\)z\_\{g\}=m\_\{\\xi\}\(c\_\{g\}\)replace the encoded goal observation in the planning objective\. World model and planner are untouched, so the module is model\- and modality\-agnostic\. Appendix[B](https://arxiv.org/html/2608.29434#A2)discusses the alternative of casting a synthetic goal scan\.

## 4Experiments

All models are trained on the re\-sensed datasets of Section[3\.1](https://arxiv.org/html/2608.29434#S3.SS1)and evaluated by goal\-conditioned planning behind thestable\-worldmodelsolver layer\[[20](https://arxiv.org/html/2608.29434#bib.bib16)\]: receding\-horizon CEM over 10 seeds of 50 episodes each, on a fixed list of start–goal pairs shared across all models and image baselines\.

### 4\.1Baselines

Table 2:Planning success rate \(%\), mean±sd\{\}\_\{\\pm\\text\{sd\}\}over 10 seeds×\\times50 episodes \(sd across seeds, here and throughout\), with CEM behind thestable\-worldmodelsolver\[[20](https://arxiv.org/html/2608.29434#bib.bib16)\]on an identical fixed list of start–goal pairs\. Rows marked†\\daggerare copied from prior work under a different protocol \(DINO\-WM with 1 seed and no sd\[[21](https://arxiv.org/html/2608.29434#bib.bib15)\], Delta\-JEPA with 3 seeds\[[46](https://arxiv.org/html/2608.29434#bib.bib33)\]\) and serve as context only\.FamilyObs\.MethodTwo\-RoomReacherPush\-TOGB\-CubeRandom25\.2±4\.110\.8±4\.02\.4±1\.346\.0±8\.5SIGRegImageLeWM\[[21](https://arxiv.org/html/2608.29434#bib.bib15)\]85\.6±5\.782\.0±5\.686\.4±5\.269\.8±9\.3PointsPoint\-LeWM87\.0±4\.480\.4±4\.683\.6±3\.466\.0±5\.9Act\.\-recon\.ImageDelta\-JEPA†\[[46](https://arxiv.org/html/2608.29434#bib.bib33)\]100\.0±0\.081\.3±0\.589\.1±1\.979\.3±1\.8PointsPoint\-Delta\-JEPA100\.0±0\.077\.6±4\.470\.8±7\.783\.4±3\.8Frozen enc\.ImageDINO\-WM†\[[48](https://arxiv.org/html/2608.29434#bib.bib36)\]100\.079\.074\.086\.0PointsUtonia\-WM94\.0±1\.971\.2±3\.746\.6±6\.671\.2±7\.7PointsVox\-WM81\.4±4\.854\.4±6\.235\.2±6\.363\.4±7\.8Table[2](https://arxiv.org/html/2608.29434#S4.T2)answers the title question in the affirmative: every point\-cloud world model plans far above Random in every environment, despite moving fractions of only 0\.3–15% \(Table[1](https://arxiv.org/html/2608.29434#S3.T1)\), so neither route to degeneracy of Section[1](https://arxiv.org/html/2608.29434#S1)is realized\. Against image\-space LeWM, re\-evaluated from its released checkpoints\[[21](https://arxiv.org/html/2608.29434#bib.bib15)\]on identical start–goal pairs, Point\-LeWM is statistically equivalent on every benchmark, with only a small but significant 2\.8\-point deficit on Push\-T \(tests in Appendix[E](https://arxiv.org/html/2608.29434#A5)\)\. On Two\-Room it even attains the higher mean\. Sparsity, missing ordering and self\-occlusion thus cost at most a few points of planning performance, and little compute: 0\.53 s per CEM solve versus 0\.48 s \(Appendix[A](https://arxiv.org/html/2608.29434#A1)\)\.

Objective and modality interact\. Point\-Delta\-JEPA saturates Two\-Room and attains the best OGB\-Cube result in our controlled evaluation \(83\.4±3\.883\.4\\pm 3\.8, above even its image\-space counterpart\[[46](https://arxiv.org/html/2608.29434#bib.bib33)\]\), significantly outperforming Point\-LeWM on both \(\+13\.0\+13\.0and\+17\.4\+17\.4points,p<0\.001p<0\.001\)\. Push\-T reverses the ranking \(70\.8 vs\. 83\.6,p<0\.001p<0\.001\)\. The probing below offers a mechanism: the task scores the block’s orientation, which Point\-Delta\-JEPA encodes less linearly \(Table[4](https://arxiv.org/html/2608.29434#A4.T4)\)\. On Reacher the two objectives are statistically indistinguishable \(77\.6 vs\. 80\.4,p=0\.30p=0\.30\)\. No single objective dominates, so the anti\-collapse objective is a first\-order, task\-dependent design choice\.

Utonia\-WM shows the frozen\-encoder recipe transfers to point clouds: it plans far above Random everywhere, is strongest on navigation \(94\.0 on Two\-Room\), and on OGB\-Cube attains a higher mean than both SIGReg models \(71\.2 vs\. 66\.0 and 69\.8, within noise\)\. Manipulation on Push\-T is its weak point \(46\.6\), the price of a representation that cannot adapt to the control task\. Its features are nonetheless useful rather than an artifact of the voxel representation: it beats its training\-free counterpart Vox\-WM on every benchmark \(\+7\.8\+7\.8to\+16\.8\+16\.8points\), Push\-T included\.

### 4\.2Ablation

The remaining analyses dissect Point\-LeWM and Point\-Delta\-JEPA: what the latents encode, how planning survives a degraded sensor, and where the encoders look\.

##### Physical latent probing\.

We regress privileged simulator state from the exact 192\-dimensional latent the CEM cost uses, with a linear \(ridge\) and an MLP probe on episode\-disjoint splits \(Appendix[D\.1](https://arxiv.org/html/2608.29434#A4.SS1)\)\. Across all three environments,*positions*are decodable almost perfectly by both models, linearly atR2≥0\.95R^\{2\}\\\!\\geq\\\!0\.95and near\-exactly with the MLP \(Tables[4](https://arxiv.org/html/2608.29434#A4.T4)–[6](https://arxiv.org/html/2608.29434#A4.T6)\)\.*Angles*are where the objectives separate: on Push\-T \(Table[4](https://arxiv.org/html/2608.29434#A4.T4)\) the block angle is linearly decodable from Point\-LeWM atR2=0\.89R^\{2\}=0\.89but from Point\-Delta\-JEPA only at0\.660\.66, while the MLP recovers it from both, so the information is present but not linearly organized\. Near\-perfect position probing does not imply planning success: on Push\-T both models decode positions equally well, yet Point\-LeWM plans 13 points better\. The planning gap follows the angle probe, since the task scores the block’s orientation, which Point\-LeWM encodes far more linearly\. Conversely, on OGB\-Cube only Point\-Delta\-JEPA decodes joint*velocity*at all \(MLPR2R^\{2\}0\.63 vs\. 0\.10, Table[6](https://arxiv.org/html/2608.29434#A4.T6)\), consistent with its action\-reconstruction objective and with its stronger planning there\.

##### Point cloud perturbations\.

We degrade the scan at evaluation time only, with along\-ray range noise ofσ∈\{0\.25,0\.5\}%\\sigma\\in\\\{0\.25,0\.5\\\}\\%of the per\-frame scan extent and dropout turning2525or50%50\\%of returns into misses, applied to the live observation alone or to observation and goal cloud alike\. Table[7](https://arxiv.org/html/2608.29434#A4.T7)\(Appendix[D\.2](https://arxiv.org/html/2608.29434#A4.SS2)\) reports success relative to each model’s clean performance\. Degrading the goal cloud as well costs a further, roughly uniform∼6\{\\sim\}6relative points, so the numbers below keep the goal clean\. Point\-LeWM degrades more gracefully than Point\-Delta\-JEPA in nearly every condition: under0\.25%0\.25\\%range noise it retains 80% of its clean Two\-Room performance where Point\-Delta\-JEPA keeps 43%, and it retains 99% and 95% under 50% dropout on Two\-Room and Reacher where Point\-Delta\-JEPA drops to 64% and 87%, so the distribution\-matching objective appears to yield the smoother, more redundant latent\. Push\-T is the shared failure case: noise collapses both models to below 5% relative\. Its scan is 98% ground plane and its task entities are thin, so a noise scale set by the whole scene is comparable to their height and effectively erases them\. OGB\-Cube remains at clean level for both models \(98–103% under noise, 99–102% under dropout\)\. It is also where the most points move \(∼\\sim15%\), suggesting latents anchored to the actuated geometry rather than to incidental static structure\. Thecls\-attention rollouts \(Appendix[D\.3](https://arxiv.org/html/2608.29434#A4.SS3)\) agree: they concentrate on the moving entities although these carry as little as0\.3%0\.3\\%of the returns, so neither objective settled into the slow\-feature optimum\.

### 4\.3Towards planning with 3D\-targets

Table 3:Planning success rate \(%\) toward 3D targets, mean±\\pmstd over 10 seeds \(50 episodes each\)\. Target encoders \(P\-LeWM/PD\-JEPA abbreviate Point\-LeWM/Point\-Delta\-JEPA\) map the 3D target to a goal latent\.ztz\_\{t\}marks whether the current latent is an additional input\.*Goal cloud*encodes the stored goal cloud \(reference\)\. Best per column in bold, separately with and withoutztz\_\{t\}\.Table[3](https://arxiv.org/html/2608.29434#S4.T3)evaluates the target\-to\-latent module of Section[3\.2](https://arxiv.org/html/2608.29434#S3.SS2)against planning toward the encoded goal cloud, and shows the goal observation is dispensable: with the current latent as input, the learned encoders match the goal\-cloud reference within seed noise in every column, so a planner can be commanded by a low\-dimensional target pose at no cost\. Conditioning onztz\_\{t\}matters where Section[3\.2](https://arxiv.org/html/2608.29434#S3.SS2)predicts: on OGB\-Cube, whose target leaves the arm configuration unconstrained, droppingztz\_\{t\}costs Point\-Delta\-JEPA up to 12 points, while on Push\-T and Two\-Room, whose targets constrain all moving entities, the unconditioned variants suffice\. The generative shortcut is best or tied in every conditioned column, but its margin over the MLP lies within seed noise\.

## 5Conclusion

##### Limitations\.

Utonia\-WM plans slowly: its predictor rolls out≈256\{\\approx\}256latent tokens where the end\-to\-end models roll out one, a≈50×\{\\approx\}50\\timesgap that mirrors the DINO\-WM\[[21](https://arxiv.org/html/2608.29434#bib.bib15)\]and excludes it from the perturbation ablation\. Its fixed grid pooling and random projection also impose token structure the end\-to\-end models learn, a plausible source of its manipulation gap, and the Vox\-WM comparison carries a predictor\-width confound \(Appendix[A](https://arxiv.org/html/2608.29434#A1)\)\. The target\-to\-latent module assumes training\-time target annotations and is modality\-agnostic \(Appendix[B](https://arxiv.org/html/2608.29434#A2)\)\. The study spans four simulated tabletop scenes with an idealized sensor, degraded only synthetically at test time \(Appendix[D\.2](https://arxiv.org/html/2608.29434#A4.SS2)\)\. The†\\daggerbaselines remain uncontrolled: no image Delta\-JEPA checkpoint is available, and the compute already spent precluded retraining it or re\-evaluating DINO\-WM\.

##### Conclusion\.

Latent planning survives point clouds\. We lifted the three canonical JEPA designs to geometric observations and showed that, despite sparse, self\-occluded scans where as little as0\.3%0\.3\\%of returns move, none collapses and all plan, to our knowledge the first point\-cloud world models interrogated with candidate actions rather than judged by reconstruction\. Re\-sensing thestable\-worldmodelbenchmark holds episodes, actions, goals, and planners fixed\. Under this controlled comparison the distribution\-prior model is statistically equivalent to its image counterpart and the action\-sensitive model exceeds it where the most geometry moves\. Our analysis explains why: positions are almost perfectly linearly decodable, attention lands on the task\-relevant entities, and planning withstands heavy unseen dropout, while range noise on thin structures remains the failure mode\. Geometry then pays a practical dividend, a planner commanded by a 3D target pose instead of a goal observation at no cost\. This work establishes range sensing as a first\-class modality for world\-model planning, and points toward the robot that must predict and act where cameras fail\.

## Acknowledgments

The computational results presented have been achieved using the Vienna Scientific Cluster \(VSC\)\.

## References

- \[1\]P\. Anderson, A\. Chang, D\. S\. Chaplot, A\. Dosovitskiy, S\. Gupta, V\. Koltun, J\. Kosecka, J\. Malik, R\. Mottaghi, M\. Savva, and A\. R\. Zamir\(2018\)On evaluation of embodied navigation agents\.External Links:1807\.06757Cited by:[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px3.p1.1)\.
- \[2\]M\. Andrychowicz, F\. Wolski, A\. Ray, J\. Schneider, R\. Fong, P\. Welinder, B\. McGrew, J\. Tobin, P\. Abbeel, and W\. Zaremba\(2017\)Hindsight experience replay\.InNeurIPS,Cited by:[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px3.p1.1)\.
- \[3\]M\. Assran, A\. Bardes, D\. Fan, Q\. Garrido, R\. Howes, M\. Komeili, M\. Muckley, A\. Rizvi, C\. Roberts, K\. Sinha, A\. Zholus, S\. Arnaud, A\. Gejji, A\. Martin, F\. R\. Hogan, D\. Dugas, P\. Bojanowski, V\. Khalidov, P\. Labatut, F\. Massa, M\. Szafraniec, K\. Krishnakumar, Y\. Li, X\. Ma, S\. Chandar, F\. Meier, Y\. LeCun, M\. Rabbat, and N\. Ballas\(2025\)V\-JEPA 2: self\-supervised video models enable understanding, prediction and planning\.External Links:2506\.09985Cited by:[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px1.p1.1)\.
- \[4\]\(2025\)TD\-JEPA: latent\-predictive representations for zero\-shot reinforcement learning\.External Links:2510\.00739Cited by:[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px1.p1.1)\.
- \[5\]R\. Balestriero and Y\. LeCun\(2025\)LeJEPA: provable and scalable self\-supervised learning without the heuristics\.External Links:2511\.08544Cited by:[Appendix A](https://arxiv.org/html/2608.29434#A1.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px1.p1.1),[§3\.2\.1](https://arxiv.org/html/2608.29434#S3.SS2.SSS1.Px3.p1.1)\.
- \[6\]L\. Cheng, Q\. Zhang, and Y\. Wang\(2026\)SAGE: subgoal\-conditioned action generation for latent world model planning\.External Links:2607\.17973Cited by:[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px3.p1.1)\.
- \[7\]C\. Chi, S\. Feng, Y\. Du, Z\. Xu, E\. Cousineau, B\. Burchfiel, and S\. Song\(2023\)Diffusion policy: visuomotor policy learning via action diffusion\.InRobotics: Science and Systems \(RSS\),Cited by:[Appendix A](https://arxiv.org/html/2608.29434#A1.SS0.SSS0.Px8.p1.1),[§3\.1](https://arxiv.org/html/2608.29434#S3.SS1.SSS0.Px2.p1.1)\.
- \[8\]A\. Dosovitskiy, L\. Beyer, A\. Kolesnikov, D\. Weissenborn, X\. Zhai, T\. Unterthiner, M\. Dehghani, M\. Minderer, G\. Heigold, S\. Gelly, J\. Uszkoreit, and N\. Houlsby\(2021\)An image is worth 16x16 words: transformers for image recognition at scale\.InICLR,Cited by:[Appendix A](https://arxiv.org/html/2608.29434#A1.SS0.SSS0.Px1.p1.1),[Figure 2](https://arxiv.org/html/2608.29434#S2.F2),[§3\.2\.1](https://arxiv.org/html/2608.29434#S3.SS2.SSS1.Px1.p1.1)\.
- \[9\]K\. Frans, D\. Hafner, S\. Levine, and P\. Abbeel\(2025\)One step diffusion via shortcut models\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 34668–34684\.Cited by:[Appendix A](https://arxiv.org/html/2608.29434#A1.SS0.SSS0.Px6.p1.1),[§3\.2\.2](https://arxiv.org/html/2608.29434#S3.SS2.SSS2.p1.2)\.
- \[10\]Z\. Gan, Z\. Zeng, J\. Cheng, Y\. Song, Y\. Tang, and X\. Wang\(2026\)ActSWM: action\-sensitive world models for long\-horizon planning in open\-world games\.External Links:2607\.26712Cited by:[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px1.p1.1)\.
- \[11\]D\. Ha and J\. Schmidhuber\(2018\)World models\.External Links:1803\.10122,[Document](https://dx.doi.org/10.5281/zenodo.1207631)Cited by:[§1](https://arxiv.org/html/2608.29434#S1.p1.1),[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px1.p1.1)\.
- \[12\]D\. Hafner, T\. Lillicrap, I\. Fischer, R\. Villegas, D\. Ha, H\. Lee, and J\. Davidson\(2019\)Learning latent dynamics for planning from pixels\.InICML,Cited by:[§1](https://arxiv.org/html/2608.29434#S1.p1.1),[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px3.p1.1)\.
- \[13\]N\. Hansen, H\. Su, and X\. Wang\(2023\)TD\-MPC2: scalable, robust world models for continuous control\.InICLR,Cited by:[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px3.p1.1)\.
- \[14\]Z\. Hong, S\. Wu, H\. Cui, B\. Zhao, R\. Ji, Y\. He, H\. Zhang, Z\. Ke, J\. Wang, G\. Zhang, and J\. Gu\(2026\)R3D: revisiting 3D policy learning\.External Links:2604\.15281Cited by:[Appendix A](https://arxiv.org/html/2608.29434#A1.SS0.SSS0.Px1.p1.1),[§3\.2\.1](https://arxiv.org/html/2608.29434#S3.SS2.SSS1.Px1.p1.1)\.
- \[15\]W\. Huang, Y\. Chao, A\. Mousavian, M\. Liu, D\. Fox, K\. Mo, and L\. Fei\-Fei\(2026\)PointWorld: scaling 3D world models for in\-the\-wild robotic manipulation\.External Links:2601\.03782Cited by:[§1](https://arxiv.org/html/2608.29434#S1.p2.1),[§1](https://arxiv.org/html/2608.29434#S1.p3.1),[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px3.p1.1)\.
- \[16\]P\. Ivashkov, R\. Balestriero, and B\. Schölkopf\(2026\)Sensorimotor world models: perception for action via inverse dynamics\.External Links:2606\.20104Cited by:[Appendix A](https://arxiv.org/html/2608.29434#A1.SS0.SSS0.Px5.p1.1),[§1](https://arxiv.org/html/2608.29434#S1.p1.1),[Figure 2](https://arxiv.org/html/2608.29434#S2.F2),[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px1.p1.1),[§3\.2\.1](https://arxiv.org/html/2608.29434#S3.SS2.SSS1.Px3.p1.1),[§3\.2](https://arxiv.org/html/2608.29434#S3.SS2.p1.1)\.
- \[17\]W\. B\. Johnson and J\. Lindenstrauss\(1984\)Extensions of Lipschitz mappings into a Hilbert space\.Contemporary Mathematics26,pp\. 189–206\.Cited by:[§3\.2\.1](https://arxiv.org/html/2608.29434#S3.SS2.SSS1.Px2.p1.1)\.
- \[18\]Y\. LeCun\(2022\)A path towards autonomous machine intelligence\.External Links:[Link](https://openreview.net/forum?id=BZ5a1r-kVsf)Cited by:[§1](https://arxiv.org/html/2608.29434#S1.p1.1),[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px1.p1.1),[§3\.2\.1](https://arxiv.org/html/2608.29434#S3.SS2.SSS1.p1.1)\.
- \[19\]P\. Liu, S\. Wang, L\. Zhang, X\. Peng, Y\. Lyu, J\. Deng, S\. Lu, W\. Ma, X\. Zhang, Y\. Zhan, X\. Lang, and J\. Ma\(2025\)LiSTAR: ray\-centric world models for 4D LiDAR sequences in autonomous driving\.External Links:2511\.16049Cited by:[§1](https://arxiv.org/html/2608.29434#S1.p3.1),[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px2.p1.1)\.
- \[20\]L\. Maes, Q\. L\. Lidec, L\. Facury, N\. Massaudi, A\. Chaurasia, F\. Capuano, R\. Gao, T\. Gillin, D\. Haramati, D\. Scieur, Y\. LeCun, and R\. Balestriero\(2026\)Stable\-worldmodel: a platform for reproducible world modeling research and evaluation\.External Links:2605\.21800Cited by:[Appendix A](https://arxiv.org/html/2608.29434#A1.SS0.SSS0.Px5.p1.1),[Appendix A](https://arxiv.org/html/2608.29434#A1.SS0.SSS0.Px8.p1.1),[§1](https://arxiv.org/html/2608.29434#S1.p4.1),[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px3.p1.1),[§3\.2](https://arxiv.org/html/2608.29434#S3.SS2.p1.1),[Table 2](https://arxiv.org/html/2608.29434#S4.T2),[§4](https://arxiv.org/html/2608.29434#S4.p1.1)\.
- \[21\]L\. Maes, Q\. L\. Lidec, D\. Scieur, Y\. LeCun, and R\. Balestriero\(2026\)LeWorldModel: stable end\-to\-end joint\-embedding predictive architecture from pixels\.External Links:2603\.19312Cited by:[Appendix A](https://arxiv.org/html/2608.29434#A1.SS0.SSS0.Px2.p1.1),[Appendix A](https://arxiv.org/html/2608.29434#A1.SS0.SSS0.Px5.p1.1),[§1](https://arxiv.org/html/2608.29434#S1.p1.1),[Figure 2](https://arxiv.org/html/2608.29434#S2.F2),[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px1.p1.1),[§3\.2\.1](https://arxiv.org/html/2608.29434#S3.SS2.SSS1.Px1.p1.1),[§3\.2\.1](https://arxiv.org/html/2608.29434#S3.SS2.SSS1.Px3.p1.1),[§3\.2\.1](https://arxiv.org/html/2608.29434#S3.SS2.SSS1.Px4.p1.1),[§3\.2\.2](https://arxiv.org/html/2608.29434#S3.SS2.SSS2.p1.1),[§3\.2](https://arxiv.org/html/2608.29434#S3.SS2.p1.1),[§4\.1](https://arxiv.org/html/2608.29434#S4.SS1.p1.1),[Table 2](https://arxiv.org/html/2608.29434#S4.T2),[Table 2](https://arxiv.org/html/2608.29434#S4.T2.4.3.3),[§5](https://arxiv.org/html/2608.29434#S5.SS0.SSS0.Px1.p1.1)\.
- \[22\]S\. Masip, J\. Swinnen, Y\. Hu, R\. Detry, and T\. Tuytelaars\(2026\)FF\-JEPA: long\-horizon planning in world models with latent planners\.External Links:2606\.09311Cited by:[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px3.p1.1)\.
- \[23\]I\. Misra, R\. Girdhar, and A\. Joulin\(2021\)An end\-to\-end transformer model for 3D object detection\.InICCV,Cited by:[Appendix A](https://arxiv.org/html/2608.29434#A1.SS0.SSS0.Px1.p1.1),[Appendix A](https://arxiv.org/html/2608.29434#A1.SS0.SSS0.Px6.p1.1)\.
- \[24\]N\. Morbitzer, J\. Evers, A\. Savkin, T\. Stauner, N\. Navab, F\. Tombari, and S\. Gasperini\(2026\)Future dynamic 3D reconstruction: a 3D world model with disentangled ego\-motion\.InICML,Cited by:[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px2.p1.1)\.
- \[25\]H\. Nguyen, X\. Xu, and X\. Huang\(2026\)Latent geometry beyond search: amortizing planning in world models\.External Links:2605\.08732Cited by:[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px3.p1.1)\.
- \[26\]F\. F\. Oberweger, M\. Schwingshackl, and V\. Staderini\(2026\)PI3DETR: parametric instance detection of 3D point cloud edges with a geometry\-aware 3DETR\.In3DV,External Links:[Document](https://dx.doi.org/10.1109/3DV69130.2026.00052)Cited by:[§1](https://arxiv.org/html/2608.29434#S1.p2.1)\.
- \[27\]S\. Park, K\. Frans, B\. Eysenbach, and S\. Levine\(2025\)OGBench: benchmarking offline goal\-conditioned RL\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[Appendix A](https://arxiv.org/html/2608.29434#A1.SS0.SSS0.Px8.p1.1),[§3\.1](https://arxiv.org/html/2608.29434#S3.SS1.SSS0.Px1.p1.1)\.
- \[28\]A\. Paszke, S\. Gross, F\. Massa, A\. Lerer, J\. Bradbury, G\. Chanan, T\. Killeen, Z\. Lin, N\. Gimelshein, L\. Antiga, A\. Desmaison, A\. Köpf, E\. Yang, Z\. DeVito, M\. Raison, A\. Tejani, S\. Chilamkurthy, B\. Steiner, L\. Fang, J\. Bai, and S\. Chintala\(2019\)PyTorch: an imperative style, high\-performance deep learning library\.InNeurIPS,Cited by:[Appendix A](https://arxiv.org/html/2608.29434#A1.SS0.SSS0.Px8.p1.1),[§3\.2\.1](https://arxiv.org/html/2608.29434#S3.SS2.SSS1.Px4.p1.1)\.
- \[29\]S\. Peri, H\. Nguyen, C\. Kim, L\. Fuxin, and S\. Lee\(2026\)3D point world models: point completion enables more accurate dynamics learning\.External Links:2607\.00148Cited by:[§1](https://arxiv.org/html/2608.29434#S1.p3.1),[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px2.p1.1)\.
- \[30\]C\. R\. Qi, H\. Su, K\. Mo, and L\. J\. Guibas\(2017\)PointNet: deep learning on point sets for 3D classification and segmentation\.InCVPR,Cited by:[Appendix A](https://arxiv.org/html/2608.29434#A1.SS0.SSS0.Px1.p1.1),[Figure 2](https://arxiv.org/html/2608.29434#S2.F2)\.
- \[31\]C\. R\. Qi, L\. Yi, H\. Su, and L\. J\. Guibas\(2017\)PointNet\+\+: deep hierarchical feature learning on point sets in a metric space\.InNeurIPS,Cited by:[Appendix A](https://arxiv.org/html/2608.29434#A1.SS0.SSS0.Px1.p1.1),[§3\.2\.1](https://arxiv.org/html/2608.29434#S3.SS2.SSS1.Px1.p1.1)\.
- \[32\]M\. Schwingshackl, F\. F\. Oberweger, M\. Niedermeyer, J\. Huemer, and M\. Murschitz\(2026\)PIRATR: parametric object inference for robotic applications with transformers in 3D point clouds\.InICRA,Cited by:[§1](https://arxiv.org/html/2608.29434#S1.p2.1),[§3\.2\.2](https://arxiv.org/html/2608.29434#S3.SS2.SSS2.p1.1)\.
- \[33\]M\. Schwingshackl, F\. F\. Oberweger, and M\. Murschitz\(2025\)Few\-shot structure\-informed machinery part segmentation with foundation models and graph neural networks\.InWACV,External Links:[Document](https://dx.doi.org/10.1109/WACV61041.2025.00200)Cited by:[§1](https://arxiv.org/html/2608.29434#S1.p2.1)\.
- \[34\]V\. Sobal, J\. S, S\. Jalagam, N\. Carion, K\. Cho, and Y\. LeCun\(2022\)Joint embedding predictive architectures focus on slow features\.InNeurIPS,Cited by:[§1](https://arxiv.org/html/2608.29434#S1.p2.1),[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px1.p1.1),[§3\.2\.1](https://arxiv.org/html/2608.29434#S3.SS2.SSS1.Px3.p1.1)\.
- \[35\]V\. Sobal, W\. Zhang, K\. Cho, R\. Balestriero, T\. G\. J\. Rudner, and Y\. LeCun\(2025\)Learning from reward\-free offline data: a case for planning with latent dynamics models\.InNeurIPS,Cited by:[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px1.p1.1)\.
- \[36\]E\. Todorov, T\. Erez, and Y\. Tassa\(2012\)MuJoCo: a physics engine for model\-based control\.InIEEE/RSJ International Conference on Intelligent Robots and Systems \(IROS\),pp\. 5026–5033\.Cited by:[Appendix A](https://arxiv.org/html/2608.29434#A1.SS0.SSS0.Px8.p1.1),[§3\.1](https://arxiv.org/html/2608.29434#S3.SS1.SSS0.Px2.p1.1)\.
- \[37\]Y\. Wang, O\. Bounou, Y\. LeCun, and M\. Ren\(2026\)AdaJEPA: an adaptive latent world model\.External Links:2606\.32026Cited by:[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px1.p1.1)\.
- \[38\]Y\. Wang, J\. Xia, R\. Zhou, X\. Hu, Y\. Shi, J\. Du, and Y\. Ye\(2026\)PRISM: PRior\-guided imagination sampling in world models\.External Links:2606\.07974Cited by:[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px3.p1.1)\.
- \[39\]X\. Wu, D\. DeTone, D\. Frost, T\. Shen, C\. Xie, N\. Yang, J\. Engel, R\. Newcombe, H\. Zhao, and J\. Straub\(2025\)Sonata: self\-supervised learning of reliable point representations\.InCVPR,Cited by:[§1](https://arxiv.org/html/2608.29434#S1.p2.1),[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px2.p1.1)\.
- \[40\]X\. Wu, L\. Jiang, P\. Wang, Z\. Liu, X\. Liu, Y\. Qiao, W\. Ouyang, T\. He, and H\. Zhao\(2023\)Point transformer V3: simpler, faster, stronger\.InCVPR,Cited by:[Appendix A](https://arxiv.org/html/2608.29434#A1.SS0.SSS0.Px3.p1.1),[Figure 3](https://arxiv.org/html/2608.29434#S2.F3),[§3\.2\.1](https://arxiv.org/html/2608.29434#S3.SS2.SSS1.Px2.p1.1)\.
- \[41\]Z\. Yang, L\. Chen, Y\. Sun, and H\. Li\(2024\)Visual point cloud forecasting enables scalable autonomous driving\.InCVPR,External Links:[Document](https://dx.doi.org/10.1109/CVPR52733.2024.01390)Cited by:[§1](https://arxiv.org/html/2608.29434#S1.p3.1),[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px2.p1.1)\.
- \[42\]X\. Yu, L\. Tang, Y\. Rao, T\. Huang, J\. Zhou, and J\. Lu\(2022\)Point\-BERT: pre\-training 3D point cloud transformers with masked point modeling\.InCVPR,Cited by:[Appendix A](https://arxiv.org/html/2608.29434#A1.SS0.SSS0.Px1.p1.1),[§3\.2\.1](https://arxiv.org/html/2608.29434#S3.SS2.SSS1.Px1.p1.1)\.
- \[43\]Z\. Yu, X\. Hu, and X\. Xu\(2026\)QQWorld: quantile\-quantile matching for world model regularization\.External Links:2607\.28415Cited by:[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px1.p1.1)\.
- \[44\]L\. Zhang, Y\. Xiong, Z\. Yang, S\. Casas, R\. Hu, and R\. Urtasun\(2023\)Copilot4D: learning unsupervised world models for autonomous driving via discrete diffusion\.InICLR,Cited by:[§1](https://arxiv.org/html/2608.29434#S1.p3.1),[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px2.p1.1)\.
- \[45\]Y\. Zhang, X\. Wu, Y\. Yang, X\. Fan, H\. Li, Y\. Zhang, Z\. Huang, N\. Wang, and H\. Zhao\(2026\)Utonia: toward one encoder for all point clouds\.External Links:2603\.03283Cited by:[Appendix A](https://arxiv.org/html/2608.29434#A1.SS0.SSS0.Px3.p1.1),[Appendix A](https://arxiv.org/html/2608.29434#A1.SS0.SSS0.Px8.p1.1),[1st item](https://arxiv.org/html/2608.29434#S1.I1.i1.p1.1),[Figure 3](https://arxiv.org/html/2608.29434#S2.F3),[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px2.p1.1),[§3\.2\.1](https://arxiv.org/html/2608.29434#S3.SS2.SSS1.Px2.p1.1)\.
- \[46\]Z\. Zhang, Y\. Wang, Z\. Guan, Y\. Yang, B\. Shi, T\. Zong, H\. Yi, G\. Chao, X\. Chen, T\. Yang, C\. Bao, T\. Yu, J\. Zhou, and J\. Xu\(2026\)Delta\-JEPA: learning action\-sensitive world models via latent difference decoding\.External Links:2606\.31232Cited by:[Appendix A](https://arxiv.org/html/2608.29434#A1.SS0.SSS0.Px2.p1.1),[Appendix A](https://arxiv.org/html/2608.29434#A1.SS0.SSS0.Px5.p1.1),[§1](https://arxiv.org/html/2608.29434#S1.p1.1),[Figure 2](https://arxiv.org/html/2608.29434#S2.F2),[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px1.p1.1),[§3\.2\.1](https://arxiv.org/html/2608.29434#S3.SS2.SSS1.Px3.p1.1),[§4\.1](https://arxiv.org/html/2608.29434#S4.SS1.p2.1),[Table 2](https://arxiv.org/html/2608.29434#S4.T2),[Table 2](https://arxiv.org/html/2608.29434#S4.T2.4.5.3)\.
- \[47\]W\. Zheng, W\. Chen, Y\. Huang, B\. Zhang, Y\. Duan, and J\. Lu\(2024\)OccWorld: learning a 3D occupancy world model for autonomous driving\.InECCV,External Links:[Document](https://dx.doi.org/10.1007/978-3-031-72624-8%5F4)Cited by:[§1](https://arxiv.org/html/2608.29434#S1.p3.1),[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px2.p1.1)\.
- \[48\]G\. Zhou, H\. Pan, Y\. LeCun, and L\. Pinto\(2025\)DINO\-WM: world models on pre\-trained visual features enable zero\-shot planning\.InICML,Cited by:[Appendix A](https://arxiv.org/html/2608.29434#A1.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2608.29434#S1.p1.1),[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px3.p1.1),[§3\.2\.1](https://arxiv.org/html/2608.29434#S3.SS2.SSS1.Px2.p1.1),[§3\.2\.1](https://arxiv.org/html/2608.29434#S3.SS2.SSS1.Px3.p1.1),[§3\.2\.1](https://arxiv.org/html/2608.29434#S3.SS2.SSS1.Px4.p1.1),[§3\.2\.2](https://arxiv.org/html/2608.29434#S3.SS2.SSS2.p1.1),[§3\.2](https://arxiv.org/html/2608.29434#S3.SS2.p1.1),[Table 2](https://arxiv.org/html/2608.29434#S4.T2.4.7.3)\.
- \[49\]J\. Zhou, J\. Wang, B\. Ma, Y\. Liu, T\. Huang, and X\. Wang\(2024\)Uni3D: exploring unified 3D representation at scale\.InICLR,Cited by:[§3\.2\.1](https://arxiv.org/html/2608.29434#S3.SS2.SSS1.Px1.p1.1)\.
- \[50\]H\. Zhu and A\. Choromanska\(2026\)Self\-supervised JEPA\-based world models for LiDAR occupancy completion and forecasting\.External Links:2602\.12540Cited by:[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px2.p1.1)\.
- \[51\]H\. Zhu, Z\. Dong, K\. Topollai, B\. Sha, and A\. Choromanska\(2026\)Self\-supervised representation learning with joint embedding predictive architecture for automotive LiDAR object detection\.InAAAI,External Links:[Document](https://dx.doi.org/10.1609/AAAI.V40I16.38402)Cited by:[§1](https://arxiv.org/html/2608.29434#S1.p3.1),[§2](https://arxiv.org/html/2608.29434#S2.SS0.SSS0.Px2.p1.1)\.

## Appendix AImplementation details

This section records the architecture and optimization constants behind Section[3\.2](https://arxiv.org/html/2608.29434#S3.SS2)\. The released configurations remain authoritative\. All world models are trained on two A100 GPUs\.

##### Point\-cloud encoder \(Point\-LeWM, Point\-Delta\-JEPA\)\.

The shared encoder \(Figure[2](https://arxiv.org/html/2608.29434#S2.F2)\) consumes geometry only: the per\-point input features are the sensor\-frame coordinates, with no intensity or color channel\. Each cloud is reduced to 256 farthest\-point\-sampled centres, and a ball query of per\-environment metric radius \(0\.15 m Two\-Room, 0\.18 m Push\-T, 0\.04 m Reacher\) draws up to 32 points per centre uniformly at random\[[31](https://arxiv.org/html/2608.29434#bib.bib41)\]\. Coordinates are first normalized by the fixed workspace map of Section[3\.2](https://arxiv.org/html/2608.29434#S3.SS2), the affinep↦\(p−c\)/sp\\mapsto\(p\-c\)/swith a per\-environment centrec∈ℝ3c\\in\\mathbb\{R\}^\{3\}and isotropic scalessmeasured once on the dataset, which brings the workspace to approximately\[−1,1\]3\[\-1,1\]^\{3\}while preserving absolute object positions\. The centre\-relative offsets of each group then pass through a two\-stage mini\-PointNet\[[30](https://arxiv.org/html/2608.29434#bib.bib45),[42](https://arxiv.org/html/2608.29434#bib.bib42)\]with hidden widths\(128,256\)\(128,256\), yielding one 192\-dimensional token to which a log\-spaced sinusoidal embedding of the group centre\[[23](https://arxiv.org/html/2608.29434#bib.bib40)\]is added\. The tokens, prepended by a learnedclstoken, feed a LayerNorm\-only\[[14](https://arxiv.org/html/2608.29434#bib.bib44)\]ViT\-Tiny trunk \(12 layers, 3 heads, width 192,≈5\.5\{\\approx\}5\.5M parameters\)\[[8](https://arxiv.org/html/2608.29434#bib.bib39)\], whoseclsoutput is the latentztz\_\{t\}\. OGB\-Cube uses 512 tokens with radius 0\.03 m instead\. The denser sampling helped only there, since the other scenes are dominated by the ground plane\.

##### Predictor, actions, and optimization\.

The predictor is LeWorldModel’s causal action\-conditioned transformer\[[21](https://arxiv.org/html/2608.29434#bib.bib15)\]: 6 blocks, 16 heads of dimension 64, MLP width 2048 at the 192\-dimensional latent width, with a context ofℓ=3\\ell=3past latents\. Latents pass a BatchNorm MLP projector \(hidden 2048\), as in the image model\. The world models step at frameskip 5, the five intervening environment actions concatenated and embedded by a shared linear action encoder\. Both end\-to\-end models train for 10 epochs with AdamW \(learning rate5×10−55\\times 10^\{\-5\}, weight decay10−310^\{\-3\}\) at effective batch size 128, bf16 precision, and gradient clipping 1\.0\. The SIGReg weight is 0\.09 \(17 knots, 1024 projections\)\[[5](https://arxiv.org/html/2608.29434#bib.bib5),[21](https://arxiv.org/html/2608.29434#bib.bib15)\]\. The action\-reconstruction weight is 10 at horizonN=5N\{=\}5\[[46](https://arxiv.org/html/2608.29434#bib.bib33)\], for which Point\-Delta\-JEPA trains on 6\-frame windows while planning with the same 3\-latent context\. On OGB\-Cube both models follow a 50\-epoch cosine schedule, with the reported checkpoints still taken after 10 epochs\.

##### Utonia\-WM\.

Completing Section[3\.2](https://arxiv.org/html/2608.29434#S3.SS2): point clouds are expressed in a canonical frame derived from the fitted ground plane, rescaled by Utonia’s own tabletop factor of 4\.0, and voxel\-downsampled at a grid size of 0\.01, keeping one point per occupied voxel, before entering the frozen≈137\{\\approx\}137M\-parameter PTv3 backbone\[[40](https://arxiv.org/html/2608.29434#bib.bib28),[45](https://arxiv.org/html/2608.29434#bib.bib34)\]\. Per grid cell, the backbone’s coarsest\-stage 576\-dimensional super\-point features are mean\- and max\-pooled \(1152 dimensions\) and compressed to the 192\-dimensional token width by the fixed random orthogonal projection of Section[3\.2](https://arxiv.org/html/2608.29434#S3.SS2), a Gaussian matrix drawn once from a fixed seed and orthonormalized via QR\. The grids are8×8×4=2568\\times 8\\times 4=256tokens \(OGB\-Cube, Reacher\),12×11×2=26412\\times 11\\times 2=264\(Push\-T\), and10×9×3=27010\\times 9\\times 3=270\(Two\-Room\), spanning a per\-environment workspace box\. The predictor is DINO\-WM’s frame\-block\-causal token transformer\[[48](https://arxiv.org/html/2608.29434#bib.bib36)\]at the same depth and widths as above, with the action embedding tiled over a frame’s tokens and concatenated onto each\. Only the predictor and action encoder train, against precomputed frozen features, for 100 epochs with AdamW \(learning rate10−310^\{\-3\}, weight decay10−310^\{\-3\}\) at effective batch size 512\.

##### Vox\-WM\.

Vox\-WM is a parameter\-free geometric control for Utonia\-WM that ablates the frozen foundation encoder at fixed tokenization: same canonical frame, same token grid, same predictor and prediction\-only loss, but each cell’s token holds plain statistics of the cell’s points instead of pooled frozen features\. After the voxel downsampling above, a cell’s points are expressed as offsets from the cell centre, and the 25\-dimensional token stacks an occupancy bit \(1\), the point count aslog⁡\(1\+n\)/log⁡\(1\+256\)\\log\(1\{\+\}n\)/\\log\(1\{\+\}256\)\(1\), the per\-axis mean \(3\), standard deviation \(3\), minimum \(3\), and maximum \(3\) of the offsets, their three off\-diagonal covariances \(3\), and the occupancy bits of a2×2×22\\times 2\\times 2subdivision of the cell \(8\)\. Empty cells are zero tokens, which the occupancy bit disambiguates\. The encoding is deterministic and costs microseconds per cloud, so it runs live without a feature cache, and the predictor is identical but runs at the native 25\-dimensional token width\. Training follows the Utonia\-WM protocol\. The native width leaves a capacity confound: Utonia\-WM’s predictor operates at width 192, so part of its advantage over Vox\-WM in Table[2](https://arxiv.org/html/2608.29434#S4.T2)may reflect the wider predictor rather than the frozen features\. A matched control projecting the 25 statistics to width 192, mirroring the random projection above, was not run\.

##### Planning\.

Planner and CEM hyperparameters are inherited unchanged from the image pipelines\[[21](https://arxiv.org/html/2608.29434#bib.bib15),[16](https://arxiv.org/html/2608.29434#bib.bib12),[46](https://arxiv.org/html/2608.29434#bib.bib33)\], which share one configuration behind thestable\-worldmodelsolver layer\[[20](https://arxiv.org/html/2608.29434#bib.bib16)\]\.

##### Target\-to\-latent module\.

Both instantiations ofmξm\_\{\\xi\}\(Section[3\.2](https://arxiv.org/html/2608.29434#S3.SS2)\) share one conditioning interface\. The target coordinates, namely the cube position \(3\-d\), the Two\-Room agent position \(3\-d\), or the Push\-T agent and block positions together with the block’s heading vector \(9\-d\), are expressed in the sensor frame, normalized by the same workspace mapp↦\(p−c\)/sp\\mapsto\(p\-c\)/sas the encoder inputs, and lifted by log\-spaced sinusoidal Fourier features \(32 bands\)\[[23](https://arxiv.org/html/2608.29434#bib.bib40)\], which make sub\-centimetre offsets linearly separable\. The cube target carries no orientation channel, for the symmetry reason of Appendix[D\.1](https://arxiv.org/html/2608.29434#A4.SS1)\. The*MLP*head is a residual MLP \(width 1024, hidden 2048, 4 blocks\) regressing the normalized goal latent\. The heading channels and, when used,ztz\_\{t\}enter through a linear embedding added to the Fourier code\. The*shortcut*head\[[9](https://arxiv.org/html/2608.29434#bib.bib47)\]is an adaptive\-LayerNorm\-modulated residual MLP of the same widths over the normalized latent space, trained with flow matching plus the self\-consistency loss against an EMA copy \(decay 0\.999\) and sampled in a single step at evaluation\. Here the heading channels andztz\_\{t\}join the conditioning vector that modulates every block\. Training pairs are frames of one trajectory separated by±5\\pm 5to±25\\pm 25environment steps, withztz\_\{t\}the latent of the earlier frame and the target read from the later\. At evaluationztz\_\{t\}is re\-encoded at every replan\. All heads train for 60 epochs with AdamW \(learning rate3×10−43\\times 10^\{\-4\}, weight decay10−410^\{\-4\}\) at batch size 4096 on precomputed frozen latents, with 5 mm Gaussian jitter on the position channels, without which the Fourier code is sharp enough to memorize individual frames\.

##### Compute\.

Every world model trains on two A100 GPUs of an internal cluster\. One end\-to\-end training run \(Point\-LeWM or Point\-Delta\-JEPA\) takes roughly two to three days, and the Utonia\-WM predictor trains in comparable time on precomputed frozen features\. One evaluation sweep of 10 seeds×\\times50 episodes costs a few GPU\-hours per model and environment, considerably more for Utonia\-WM \(Section[5](https://arxiv.org/html/2608.29434#S5)\)\. Completing the planning\-cost numbers of Section[4](https://arxiv.org/html/2608.29434#S4): one CEM solve \(300 candidates, 30 iterations, horizon 25\) takes 0\.53 s for the point\-cloud model and 0\.48 s for the image model on a single A100, and with replanning every 25 steps a 50\-step episode requires about 1\.1 s and 1\.0 s of total planning time, an overhead shared by Point\-Delta\-JEPA whose encoder is identical\. Including preliminary and failed runs, the full project consumed on the order of several thousand A100\-hours\.

##### Licenses\.

All existing assets are used under their published licenses: thestable\-worldmodelsuite \(0\.1\.1\) and its released LeWM checkpoints\[[20](https://arxiv.org/html/2608.29434#bib.bib16)\]\(MIT\), OGBench 1\.2\.1\[[27](https://arxiv.org/html/2608.29434#bib.bib48)\]\(MIT\), the Push\-T environment\[[7](https://arxiv.org/html/2608.29434#bib.bib49)\]\(MIT\), MuJoCo 3\.10\.0\[[36](https://arxiv.org/html/2608.29434#bib.bib50)\]\(Apache 2\.0\), the pretrained Utonia backbone\[[45](https://arxiv.org/html/2608.29434#bib.bib34)\]\(Apache 2\.0\), and PyTorch 2\.7\.0\[[28](https://arxiv.org/html/2608.29434#bib.bib51)\]\(BSD\-3\-Clause\)\.

## Appendix BThe unconditioned target\-to\-latent objective

This section makes precise the claim of Section[3\.2](https://arxiv.org/html/2608.29434#S3.SS2)that, without the current latent, the regression target of Eq\. \([1](https://arxiv.org/html/2608.29434#S3.E1)\) need not be the latent of any reachable observation\. Write𝒵f=\{fθ​\(o\):o∈supp⁡𝒟\}\\mathcal\{Z\}\_\{f\}=\\\{f\_\{\\theta\}\(o\):o\\in\\operatorname\{supp\}\\mathcal\{D\}\\\}for the set of latents the frozen encoder assigns to observations in the support of the data distribution𝒟\\mathcal\{D\}, that is, to observations that actually occur in the data, and let\(zt,ct\+k,zt\+k\)\(z\_\{t\},c\_\{t\+k\},z\_\{t\+k\}\)be distributed as in Eq\. \([1](https://arxiv.org/html/2608.29434#S3.E1)\)\. The unconditioned variant minimizes𝔼⁡\[∥mξ​\(ct\+k\)−zt\+k∥22\]\\mathbb\{E\}\\bigl\[\\lVert m\_\{\\xi\}\(c\_\{t\+k\}\)\-z\_\{t\+k\}\\rVert\_\{2\}^\{2\}\\bigr\], and over unconstrained functionsm:𝒞→𝒵m:\\mathcal\{C\}\\to\\mathcal\{Z\}this mean\-squared error is minimized pointwise by the conditional mean

m∗​\(c\)=𝔼⁡\[zt\+k\|ct\+k=c\],m^\{\\ast\}\(c\)=\\mathbb\{E\}\\bigl\[z\_\{t\+k\}\\,\\big\|\\,c\_\{t\+k\}=c\\bigr\],\(2\)the barycenter of the latents of*all*data scenes whose annotation equalscc, covering every configuration of the scene content the target leaves free, on OGB\-Cube every arm pose\.𝒵f\\mathcal\{Z\}\_\{f\}is the image of the low\-dimensional scene configuration manifold under the continuous encoderfθf\_\{\\theta\}, hence itself a low\-dimensional subset ofℝ192\\mathbb\{R\}^\{192\}, and no term of the training objective encourages it to be convex\. SIGReg in particular shapes one\-dimensional projections of the latent distribution rather than the geometry of its support\. A barycenter of distinct points of a non\-convex set need not lie in the set, andm∗​\(c\)m^\{\\ast\}\(c\)can therefore be a latent that no observation in the data distribution produces\. A planner minimizing∥z^t\+H−m∗​\(c\)∥22\\lVert\\hat\{z\}\_\{t\+H\}\-m^\{\\ast\}\(c\)\\rVert\_\{2\}^\{2\}is then steered toward a goal latent it may be unable to attain\.

Conditioning changes the average, not the argument: withztz\_\{t\}as input the minimizer becomes𝔼\[zt\+k∣ct\+k=c,zt\]\\mathbb\{E\}\[z\_\{t\+k\}\\mid c\_\{t\+k\}=c,\\,z\_\{t\}\], an average over futures of the current scene only\. Scene content thatztz\_\{t\}determines and that the dynamics leave unchanged over the offset window contributes no variance to this average, so the residual spread stems only from the moving entities the target leaves unconstrained\. Whether that residue matters is an empirical question, and Table[3](https://arxiv.org/html/2608.29434#S4.T3)answers it per environment: on Push\-T and Two\-Room the target constrains all moving entities and the unconditioned variant already matches the goal\-cloud reference, while on OGB\-Cube the arm remains free at the goal even givenztz\_\{t\}, and droppingztz\_\{t\}costs up to 12 points\. The two heads of Section[3\.2](https://arxiv.org/html/2608.29434#S3.SS2)also treat the average differently: the MLP fits the conditional mean of Eq\. \([2](https://arxiv.org/html/2608.29434#A2.E2)\) directly, while the shortcut model samples from the conditional distribution and so returns one consistent latent rather than a barycenter, although Table[3](https://arxiv.org/html/2608.29434#S4.T3)separates the two heads only within seed noise\.

##### Why not synthesize the goal scan?

Since we control the mirror geometry, one could place the task\-relevant entity at the commanded target, cast a synthetic scan of that state, and encode it, with no learned module\. The construction founders on the same ambiguity as the unconditioned regression: a scene must be fully configured before a scan can be cast, the target constrains only the task\-relevant coordinates, so the synthesizer has to invent the remaining content, on OGB\-Cube an arm pose, and an invented configuration need not be reachable or consistent with the present scene\. The conditioned module resolves this fromztz\_\{t\}without reconstructing geometry, and in deployment the mismatch grows, since no perfect mirror of the scene exists to render from\. The module itself is also modality\-agnostic: given the same target annotations it could be trained on image latents\. What geometry specifically contributes is that the target already lives in the metric space of the observation, so no cross\-modal correspondence must be learned\.

## Appendix CPoint\-cloud sensing and dataset replication: details

Exact sensor parameters, mirror geometry and scene constants are given by the released configurations\. This section records only the choices behind them\.

##### Ray budget\.

The tokenizer reduces every cloud to 256 farthest\-point\-sampling centres, so the ray grid fixes the sampling density of the observation rather than the sequence length seen by the model\. We use100×100100\\times 100as the coarsest grid at which the smallest task\-relevant entity, the Push\-T pusher or the Two\-Room agent, still returns on the order of a hundred points\.

##### Viewpoint\.

Oblique viewpoints are derived rather than hand\-placed: given bounding\-sphere radiusRRthe camera sits atd=2\.5​Rd=2\.5Ralong the45∘45^\{\\circ\}azimuth and elevation bearing, with vertical field of view2​arcsin⁡\(R/d\)2\\arcsin\(R/d\), so no pose is tuned per environment\. The OGBench cube is the exception: its scene already contains the oblique camera its released images were rendered from, and the sensor is mounted on it, so that its scan and the image baseline’s observation share a viewpoint\.

##### Mirror scenes\.

Entity heights in the two mirrored environments are ours to choose rather than inherited from the benchmark, and were fixed once per environment: large enough that an entity is separable from the floor in an oblique scan, small enough that self\-occlusion does not dominate\. Reacher3D is native, but the geometry of the released environment did not match the dataset it shipped with\. We recovered the two link lengths by least\-squares fitting to the recorded joint angles and fingertip positions, after which forward kinematics reproduces the recorded fingertip\.

##### Goal handling\.

The goal marker is excluded from every scan, by an alpha threshold in the cube and by omission from the mirror geometry elsewhere, since a goal specifies a task rather than a surface a range sensor would return, and retaining it would permit a policy to read the goal off the observation rather than infer it\. The goal reaches the planner as the stored cloud of the goal frame, indexed from the same converted table by the same goal offset the image pipeline uses, so no separate goal scan is cast\.

## Appendix DAblations

### D\.1Physical latent probing

##### Protocol\.

For each environment the frozen checkpoints of the two end\-to\-end world models encode every tenth frame of its training table\. The probed representation isz=fθ​\(o\)z=f\_\{\\theta\}\(o\)after the projector, i\.e\. exactly the 192\-dimensional latent the CEM cost operates on\. Two probes per target regress privileged simulator state fromzz: a*linear*probe, closed\-form ridge regression on standardized features with the regularization weight selected on a validation split over a logarithmic grid, and an*MLP*probe \(→→→d192\\\!\\to\\\!512\\\!\\to\\\!512\\\!\\to\\\!d, GELU, AdamW with cosine schedule, early\-stopped on the same validation split\)\. Splits are drawn at*episode*granularity \(72/8/20 train/val/test\), so every test frame comes from a trajectory no probe ever saw\. We report test\-set MSE in native units and pooledR2R^\{2\}\. All targets live in the sensor frame of the stored clouds, and planar angles enter as unit heading vectors,\(cos⁡a,sin⁡a,0\)\(\\cos a,\\sin a,0\)rotated into the sensor frame, so that no probe has to fit a wrap\-around discontinuity\.

##### Results\.

Table[4](https://arxiv.org/html/2608.29434#A4.T4)reports Push\-T, analyzed in the main text\. Tables[5](https://arxiv.org/html/2608.29434#A4.T5)and[6](https://arxiv.org/html/2608.29434#A4.T6)complement it on Reacher and OGB\-Cube\. Reacher is saturated: the arm is fully visible from the oblique viewpoint, and every property, positions and link headings alike, is linearly decodable atR2≈1R^\{2\}\\\!\\approx\\\!1from both models, which is why it offers no signal for separating the objectives\. On OGB\-Cube the two yaw rows require care: a cube is 24\-fold rotationally symmetric, so its orientation is not identifiable from geometry at all, and the raw yaw label is not folded into a fundamental domain\. The near\-zeroR2R^\{2\}therefore reflects an unidentifiable*label*, not a representation failure\. The end\-effector yaw is similarly flat for both models\. The wrist subtends only a handful of returns at the100×100100\\times 100ray budget, and neither objective rewards retaining it\. Joint velocity, finally, is not observable from a single scan and can be decoded only through its correlation with configuration along the data distribution\. That Point\-Delta\-JEPA alone recovers a substantial share of it \(R2R^\{2\}0\.63 vs\. 0\.10 with the MLP probe\) is consistent with its objective, which forces the latent*displacement*to be decodable into the executed actions\.

Table 4:Physical latent probing results on Push\-T\. All targets are 3\-dimensional: locations are coordinates in the sensor frame, and the block angle is encoded as a heading vector in the sensor frame\. Lower MSE and higherR2R^\{2\}indicate better representation quality\.Table 5:Physical latent probing results on Reacher\. Positions are 3D coordinates in the sensor frame, and joint angles are encoded as per\-link unit heading vectors in the sensor frame\. Lower MSE and higherR2R^\{2\}indicate better representation quality\.Table 6:Physical latent probing results on OGB\-Cube\. Positions are 3D coordinates in the sensor frame, stacked across the five arm joints for*Joint Positions*\. Yaw is encoded as a heading vector in the sensor frame, and joint velocity is the raw six\-dimensional vector in rad/s\. Lower MSE and higherR2R^\{2\}indicate better representation quality\. The near\-zero yaw rows reflect label unidentifiability \(a cube is rotationally symmetric, and the wrist returns few points\), not representation failure, as discussed in the text\.

### D\.2Point cloud perturbations

##### Protocol\.

Perturbations are applied at evaluation time only, downstream of a faithful scan\. No model ever saw a degraded cloud in training\. Test\-time\-only degradation is the probe matched to our question: it asks whether latents trained on clean geometry anchor to structure a worse sensor still measures, the situation of a model deployed on hardware noisier than its training data\. Degrading the training data instead would measure whether the objectives can absorb noise, a property of training rather than of the learned representation\.*Noise*is along\-ray rather than isotropic, since a range sensor errs in the range it reports and each return therefore moves along its own line of sight\. Its standard deviationσ\\sigmais set as a fraction of the per\-frame bounding\-box diagonal of the valid returns\. The relative scale is what makes the ablation comparable across environments: an absoluteσ\\sigmaof 2 cm is a light haze on the 4\.9 m Push\-T arena and a catastrophe on the 1\.4 m Reacher workspace\.*Dropout*marks a random fraction of returns as misses, which is what a lost return physically is, so the dropped points leave through the same invalid\-return filter the pipeline applies to genuine misses\. Because the goal cloud is read from the dataset rather than scanned live, it stays clean under the first condition \(a degraded sensor pursuing a stored target, the deployment case\) and is degraded identically under the second, which tests the representation itself but no longer distinguishes “cannot see” from “aiming at the wrong thing”\. Success rates in Table[7](https://arxiv.org/html/2608.29434#A4.T7)are reported relative to each model’s own clean performance, so values may exceed 100 within seed noise\.

Table 7:Point cloud degradation\.Planning success rate over 10 seeds with 50 episodes each, reported*relative to each model’s own clean performance*\(100 = no degradation\)\. The top row gives the absolute clean success rates used as denominators\. The two point\-cloud world models are shown side by side under each condition, better of the pair in bold\. Due to Utonia\-WM’s long computation times, it is not part of this ablation table\. Noiseσ\\sigmais a fraction of the frame’s scan extent, and dropout is the fraction of returns lost\.Beyond the trends discussed in the main text, the table shows that degrading the goal in addition to the observation costs a further, roughly uniform margin \(e\.g\. Two\-Room noise0\.25%0\.25\\%: 80\.2 vs\. 61\.6 relative for Point\-LeWM\), and that dropout is benign wherever the surviving returns still sample the entities\. Only Push\-T, whose pusher is near the ray\-budget floor of Appendix[C](https://arxiv.org/html/2608.29434#A3), breaks under50%50\\%dropout\.

### D\.3Attention

Figure[5](https://arxiv.org/html/2608.29434#A4.F5)visualizes where the two end\-to\-end encoders look: thecls\-token attention is propagated through the ViT trunk by attention rollout, assigned from each group token to the points it covers, and normalised per cloud\. Both models place their attention mass on the moving entities\. In all four environments Point\-LeWM is sharply peaked on the entities, while Point\-Delta\-JEPA allots visibly more attention to ground and table\.

![Refer to caption](https://arxiv.org/html/2608.29434v1/attention_grid_one.png)Figure 5:Encoder attention\.cls\-token attention rollout of the PointViT encoder mapped onto the input points\. Warmer colours indicate higher attention, and points no group token covers are omitted\. Columns are environments, rows the two models\. Attention is normalised per point cloud\. Utonia\-WM is absent because its frozen PTv3 encoder has noclstoken and attends only locally within serialised patches, so it admits no comparable attention map\.

## Appendix EStatistical tests

This section quantifies two claims of Section[4](https://arxiv.org/html/2608.29434#S4): that Point\-LeWM plans on par with its image counterpart, and that the objective\-family gaps are real\. All tests operate on the ten per\-seed success rates behind Table[2](https://arxiv.org/html/2608.29434#S4.T2)\. The evaluation seed fixes the same 50 start–goal windows for every model, so any two models are compared on identical episodes and the tests are paired by seed, which removes the seed\-to\-seed variation in episode difficulty from the comparison\.

Two complementary tests answer two different questions\. A paired two\-sidedtt\-test asks whether the data show any difference at all\. A significant result means the two models genuinely differ on the shared episodes, but a non\-significant result is not evidence of equality\. Positive evidence of parity requires an equivalence test, and we use paired two one\-sided tests \(TOST\): equivalence at a margin of±10\\pm 10points is established \(p<0\.05p<0\.05\) exactly when the data rule out a true difference larger than ten points in either direction\. The margin is chosen below the smallest effect the paper interprets, the 13\.0\-point objective\-family gap on Two\-Room, so established equivalence is tighter than any gap we reason about\. Passing TOST atp<0\.05p<0\.05is equivalent to the 90% confidence interval of the paired difference lying inside the margin\. Table[8](https://arxiv.org/html/2608.29434#A5.T8)reports the more familiar 95% intervals, whose containment in the margin establishes equivalence even at the stricter levelα=0\.025\\alpha=0\.025\.

Table 8:Modality comparison of Table[2](https://arxiv.org/html/2608.29434#S4.T2), Point\-LeWM against the re\-evaluated image LeWM: difference of means \(points\) with its 95% confidence interval, pairedtt\-testpp, and paired TOST equivalenceppat the±10\\pm 10\-point margin\. Equivalence holds on every benchmark, and the pairing is sensitive enough to resolve the small Push\-T deficit\.Table[8](https://arxiv.org/html/2608.29434#A5.T8)shows the modality comparison\. Equivalence holds on every benchmark \(p≤0\.005p\\leq 0\.005\), and Two\-Room and Push\-T pass even at the stricter±5\\pm 5\-point margin, so a modality gap of practical size is ruled out everywhere\. Every 95% interval lies inside the±10\\pm 10margin, with the widest edge at−8\.1\-8\.1points on OGB\-Cube\. In the difference direction, only Push\-T is significant, a deficit of 2\.8 points \(p=0\.025p=0\.025\) that is real but small next to the image baseline’s 86\.4% success\. No other benchmark separates the modalities \(p≥0\.08p\\geq 0\.08\), and on Two\-Room the point\-cloud model attains the higher mean\. The objective\-family gaps of Section[4](https://arxiv.org/html/2608.29434#S4)are all confirmed by the same paired test: Point\-Delta\-JEPA exceeds Point\-LeWM by\+13\.0\+13\.0points on Two\-Room \(p<10−5p<10^\{\-5\}\) and\+17\.4\+17\.4on OGB\-Cube \(p<10−6p<10^\{\-6\}\), and Point\-LeWM exceeds Point\-Delta\-JEPA by\+12\.8\+12\.8on Push\-T \(p<10−3p<10^\{\-3\}\)\. On Reacher the two objectives do not differ \(−2\.8\-2\.8,p=0\.30p=0\.30\), the basis for the main text’s statement that Reacher separates neither objective\.

## Appendix FBroader impacts

This work is foundational research on world\-model learning in simulated tabletop environments and is not tied to a particular application or deployment\. Its societal effects are those generic to improved robot autonomy\. Range\-based world models could benefit assistive and industrial robots that must operate where cameras fail, and lower the cost of adapting robots to new tasks\. Conversely, more capable autonomous systems can displace labor and could be misused, for example in surveillance platforms that favor range sensors precisely because they work in darkness\. The models released here are trained on four simulated benchmarks and pose no direct risk, and we see no mitigations required beyond the norms of open release\.