XP-JEPA: Cross-Predictive Physics Grounding for Forecastable Latent Dynamics

# XP-JEPA: Cross-Predictive Physics Grounding for Forecastable Latent Dynamics
Source: [https://arxiv.org/html/2608.24044](https://arxiv.org/html/2608.24044)
Ziming LiAffiliation:Department of Electrical and Computer Engineering, NUSSiyuan LuoAffiliation:Department of Electrical and Computer Engineering, NUSFan ShiAffiliation:Department of Electrical and Computer Engineering, NUS

###### Abstract

Latent world models plan by predicting how candidate actions transform learned representations\. In self\-predictive models, however, the encoder and predictor are optimized jointly and can co\-adapt to latent transitions that are easy to predict but only weakly constrained by the physical evolution of the scene\. We introduce the cross\-predictive JEPA \(XP\-JEPA\), which grounds visual latent dynamics in privileged physical trajectories\. XP\-JEPA separately encodes visual observations and physical states, advances both through a shared action\-conditioned predictor, and matches each prediction to both future representations\. This objective encourages unified latent dynamics across the two modalities, grounded in the underlying physical transitions\. The physical branch is discarded after training, leaving a visual\-only model at deployment\. On a multi\-task suite spanning six evaluation subfamilies, XP\-JEPA reduces rollout drift of a newly fitted predictor from0\.3610\.361to0\.1040\.104and increases mean control success from53\.6%53\.6\\%to78\.2%78\.2\\%\. Direct physical\-state regression raises position decodability but leaves forecastability and control near the visual\-only baseline\. Cross\-predictive physical grounding can therefore produce more forecastable latent dynamics for rollout\-based control without privileged inputs at test time\.

## 1Introduction

Planning with a latent world model casts control as search in representation space\([Ha and Schmidhuber, 2018](https://arxiv.org/html/2608.24044#bib.bib1);[Hafner et al\., 2019](https://arxiv.org/html/2608.24044#bib.bib28);[Hafner et al\., 2025](https://arxiv.org/html/2608.24044#bib.bib3);[Hansen et al\., 2024](https://arxiv.org/html/2608.24044#bib.bib4)\): the agent encodes the current scene, rolls candidate actions forward through learned dynamics, and selects actions whose predicted outcomes approach the goal\. This requires representations with reliable action\-conditioned dynamics\. In self\-predictive world models\([LeCun, 2022](https://arxiv.org/html/2608.24044#bib.bib11);[Assran et al\., 2023](https://arxiv.org/html/2608.24044#bib.bib12);[Zhou et al\., 2024](https://arxiv.org/html/2608.24044#bib.bib5);[Sobal et al\., 2025](https://arxiv.org/html/2608.24044#bib.bib7)\), however, the representation and predictor are optimized jointly and may co\-adapt to latent transitions that are internally predictable but only weakly constrained by the underlying physical evolution\. Figure[1](https://arxiv.org/html/2608.24044#S1.F1)shows the failure mode this permits: two states that project to nearly the same image can evolve in opposite ways under the same action, and a self\-predictive objective does not require their latent transitions to separate\.

![Refer to caption](https://arxiv.org/html/2608.24044v1/motivation.png)Figure 1:Physical grounding distinguishes interactions that look alike but evolve differently\.*Left:*in A the block is held; in B it rests farther along the same camera ray, so the same lift moves it only in A \(insets magnify the same object window\)\.*Top right \(schematic\):*the two observations encode almost identically, and a self\-predictive objective is satisfied whether or not their predicted futures separate; matching each predicted visual transition to the corresponding physical future pulls them apart, with holding inferred from geometry rather than given as a contact label\. The physical branch is used only in training\.*Bottom right:*three\-seed mean control success, the single\-task value averaging the four environments of Table[2](https://arxiv.org/html/2608.24044#S4.T2), where gains vary by task\.RegressmatchesVisualon the suite \(53\.753\.7against53\.653\.6\) despite the highest object\-position decodability \(R2=0\.991R^\{2\}=0\.991against0\.9780\.978\); a position probe does not capture the gripper–object relation\.Privileged physical state provides an external reference for reducing this ambiguity\. Physical trajectories describe how the scene evolves under action in a common coordinate system, allowing supervision to constrain both the information a representation preserves and how that information evolves\. We use this training\-time signal to encourage visual latent dynamics that are compatible with the corresponding physical trajectory\.

Existing approaches commonly use privileged information to supervise representation content or downstream behavior through distillation, state regression, cross\-modal alignment, or training\-time critics\([Chen et al\., 2020](https://arxiv.org/html/2608.24044#bib.bib8);[Gupta et al\., 2016](https://arxiv.org/html/2608.24044#bib.bib19);[Tian et al\., 2020](https://arxiv.org/html/2608.24044#bib.bib20);[Kumar et al\., 2021](https://arxiv.org/html/2608.24044#bib.bib9);[Pinto et al\., 2018](https://arxiv.org/html/2608.24044#bib.bib10)\)\. For latent planning, however, making physical state decodable from individual latent snapshots does not necessarily constrain how those latents evolve under action\. We instead ask how privileged physical trajectories can supervise the predictive transition itself\.

We distinguish three properties that are relevant to latent planning\. Decodability asks whether physical variables can be recovered from an individual latent state\. Forecastability asks whether the representation admits reliable action\-conditioned prediction over time\. Planning utility further depends on whether predicted latent outcomes preserve the distinctions needed to rank candidate actions\. These properties need not coincide: making physical state accessible from a snapshot does not necessarily simplify its predictive evolution, and predictable latent dynamics need not by themselves provide a useful geometry for control\. Our goal is therefore not simply to encode more physical information, but to constrain predictive evolution using the corresponding physical trajectory\.

We introduce the*cross\-predictive JEPA*\(XP\-JEPA\), which treats visual observations and privileged physical states as two views of the same trajectory\. A shared action\-conditioned predictor advances histories from either view and matches each prediction to the corresponding visual and physical futures\. Within\-modality terms retain self\-prediction, while cross\-modal terms ground each predicted transition in the other modality’s future\. The physical branch is discarded after training, leaving the same visual encoder–predictor architecture and planner as the visual\-only baseline at deployment\.

Our experiments separate the decodability of physical variables from the forecastability of latent dynamics, and both from planning utility\. A single model trained across multiple object–interaction pairs has lower rollout drift and higher control success across all six evaluation subfamilies\. Regressing a comprehensive pose\-based state target improves position decodability but leaves forecastability and control near the visual\-only baseline\. This contrast separates snapshot content from transition structure: making physical variables accessible from individual latents does not by itself produce the forecastable dynamics observed with XP\-JEPA\. More forecastable latent dynamics can benefit rollout\-based control, but the ablations show that forecastability alone does not guarantee planning utility\.

Our contributions are:

- •Cross\-predictive JEPA: physical grounding of latent transitions\.XP\-JEPA grounds latent transitions in privileged physical trajectories by advancing visual and physical histories through a shared action\-conditioned predictor and matching each prediction to both modalities’ futures, while retaining a visual\-only architecture at deployment\.
- •A unified privileged\-state interface\.We design a unified privileged\-state interface that allows a single physical encoder to operate across diverse rigid\-body manipulation scenes without task\-specific labels\.
- •Forecastable dynamics and improved control\.XP\-JEPA produces more forecastable visual dynamics across the multi\-task suite and all four matched single\-task environments, while increasing mean multi\-task control success from53\.6%53\.6\\%to78\.2%78\.2\\%\.

## 2Related Work

### Predictive latent world models\.

World models differ in the structure that their learned representations preserve\. Reconstruction\-based approaches learn dynamics through pixels or generative latent variables\([Ha and Schmidhuber, 2018](https://arxiv.org/html/2608.24044#bib.bib1);[Hafner et al\., 2019](https://arxiv.org/html/2608.24044#bib.bib28);[Hafner et al\., 2020](https://arxiv.org/html/2608.24044#bib.bib2);[Hafner et al\., 2025](https://arxiv.org/html/2608.24044#bib.bib3)\), whereas value\-centric methods shape representations around reward and control objectives\([Schrittwieser et al\., 2020](https://arxiv.org/html/2608.24044#bib.bib29);[Hansen et al\., 2024](https://arxiv.org/html/2608.24044#bib.bib4)\)\. Representation\-predictive methods instead model future features directly: DINO\-WM plans in pretrained visual features\([Zhou et al\., 2024](https://arxiv.org/html/2608.24044#bib.bib5)\), PLDM learns latent dynamics from reward\-free offline data\([Sobal et al\., 2025](https://arxiv.org/html/2608.24044#bib.bib7)\), and LeWM jointly learns visual representations and predictive dynamics\([Maes et al\., 2026](https://arxiv.org/html/2608.24044#bib.bib6)\)\. More broadly, joint\-embedding objectives learn representations by predicting feature\-space targets rather than reconstructing observations\([LeCun, 2022](https://arxiv.org/html/2608.24044#bib.bib11);[Assran et al\., 2023](https://arxiv.org/html/2608.24044#bib.bib12);[Bardes et al\., 2023](https://arxiv.org/html/2608.24044#bib.bib13);[Bardes et al\., 2024](https://arxiv.org/html/2608.24044#bib.bib14)\)\. Our work builds on this predictive paradigm and asks how corresponding physical trajectories can constrain the transition structure learned by a jointly optimized visual encoder and predictor\.

### Privileged supervision across heterogeneous tasks\.

Learning with privileged information uses signals available during training but unavailable at deployment\([Vapnik and Izmailov, 2015](https://arxiv.org/html/2608.24044#bib.bib26)\)\. Prior work transfers such information through distillation, cross\-modal alignment, state regression, privileged sensing, or asymmetric actor–critic training\([Chen et al\., 2020](https://arxiv.org/html/2608.24044#bib.bib8);[Lee et al\., 2020](https://arxiv.org/html/2608.24044#bib.bib27);[Gupta et al\., 2016](https://arxiv.org/html/2608.24044#bib.bib19);[Tian et al\., 2020](https://arxiv.org/html/2608.24044#bib.bib20);[Kumar et al\., 2021](https://arxiv.org/html/2608.24044#bib.bib9);[Pinto et al\., 2018](https://arxiv.org/html/2608.24044#bib.bib10)\)\. These approaches establish several ways to exploit additional physical information during training, but our multi\-task setting introduces a further requirement: the privileged representation must retain a consistent meaning across heterogeneous object–interaction configurations\. We therefore represent scenes through a shared rigid\-body schema based on object geometry, extent, pose, and effector state, allowing a single physical encoder to provide the privileged stream across all configurations without task\-specific labels\.

### Physical grounding of predictive dynamics\.

Several recent world\-model approaches bring privileged physical information closer to the learned dynamics\. TWIST transfers state\-based dynamics to an image\-based student\([Yamada et al\., 2024](https://arxiv.org/html/2608.24044#bib.bib30)\), Scaffolder uses privileged sensors during policy learning\([Hu et al\., 2024](https://arxiv.org/html/2608.24044#bib.bib31)\), PIGDreamer aligns world\-model representations with privileged information\([Huang et al\., 2025](https://arxiv.org/html/2608.24044#bib.bib32)\), and Pri4R introduces privileged 4D prediction during vision–language–action training\([Kim et al\., 2026](https://arxiv.org/html/2608.24044#bib.bib33)\)\. Closest to our setting, the concurrent Phys\-JEPA imposes physical consistency on latent states and transitions for multivariate time\-series forecasting\([Nie et al\., 2026](https://arxiv.org/html/2608.24044#bib.bib34)\), while PhyLatent adds training\-only physical grounding and future\-alignment objectives to the LeWM backbone\([Zeng et al\., 2026](https://arxiv.org/html/2608.24044#bib.bib35)\)\. XP\-JEPA instead treats privileged state as a second online predictive view: histories from either modality predict the corresponding future representations in both modalities, directly coupling visual–physical correspondence with action\-conditioned evolution\. Unlike one\-way distillation toward a fixed physical target, both representations remain online, and the privileged branch is removed entirely at deployment\. We further study this coupling in the multi\-task setting, where one model and one unified privileged\-state interface span many object–interaction configurations\.

## 3Cross\-Predictive JEPA

XP\-JEPA augments a JEPA\-style visual world model with privileged physical state during training\. It treats physical state as a second predictive view of the same trajectory, using physical transitions to constrain visual latent dynamics rather than merely making state decodable\. An overview of XP\-JEPA is shown in Figure[2](https://arxiv.org/html/2608.24044#S3.F2)\.

### Core objective

We build on LeWM\([Maes et al\., 2026](https://arxiv.org/html/2608.24044#bib.bib6)\), which jointly learns a visual encoderfθf\_\{\\theta\}and an action\-conditioned predictorgψg\_\{\\psi\}\. An observationoto\_\{t\}is encoded aszto=fθ​\(ot\)z\_\{t\}^\{o\}=f\_\{\\theta\}\(o\_\{t\}\)\. Given a history of visual latentsZtoZ\_\{t\}^\{o\}and an action chunkata\_\{t\}, the predictor estimates the representation of the future observation\. LeWM trainsfθf\_\{\\theta\}andgψg\_\{\\psi\}jointly with the self\-prediction term∥gψ​\(Zto,at\)−zt\+1o∥22\\lVert g\_\{\\psi\}\(Z^\{o\}\_\{t\},a\_\{t\}\)\-z^\{o\}\_\{t\+1\}\\rVert\_\{2\}^\{2\}and the isotropy regularizerβ​Ω​\(zo\)\\beta\\Omega\(z^\{o\}\)defined below\. XP\-JEPA retains this term and augments it with a physical branch—a second encoderhϕh\_\{\\phi\}over privileged state—together with physical self\-prediction and two cross\-modal prediction terms\. At deployment, candidate action sequences are rolled forward through these latent dynamics, and their predicted outcomes are compared with an encoded goal\.

We introduce privileged statests\_\{t\}as a second training\-time description of the same scene\. A physical encoderhϕh\_\{\\phi\}maps it tozts=hϕ​\(st\)z\_\{t\}^\{s\}=h\_\{\\phi\}\(s\_\{t\}\)using the unified privileged\-state interface described below\. The privileged input contains only the instantaneous scene configuration; it excludes velocities, contact forces, wrenches, and goal\-relative quantities\. Both modalities must infer motion from temporal context and action\.

LetZtmZ\_\{t\}^\{m\}denote a latent history from modalitym∈\{o,s\}m\\in\\\{o,s\\\}\. Histories from both modalities are advanced by the same action\-conditioned predictor\.

We optimize

ℒXP\-JEPA=∑m,n∈\{o,s\}‖gψ​\(Ztm,at\)−zt\+1n‖22\+β⁡\[Ω⁡\(zo\)\+Ω⁡\(zs\)\],\\mathcal\{L\}\_\{\\text\{XP\-JEPA\}\}=\\sum\_\{m,n\\in\\\{o,s\\\}\}\\left\\\|g\_\{\\psi\}\(Z\_\{t\}^\{m\},a\_\{t\}\)\-z\_\{t\+1\}^\{n\}\\right\\\|\_\{2\}^\{2\}\+\\beta\\left\[\\Omega\(z^\{o\}\)\+\\Omega\(z^\{s\}\)\\right\],\(1\)whereΩ\\Omegais a per\-branch isotropy regularizer\([Balestriero and LeCun, 2025](https://arxiv.org/html/2608.24044#bib.bib15), SIGReg;\), and the action modulates the predictor through AdaLN\([Peebles and Xie, 2023](https://arxiv.org/html/2608.24044#bib.bib17)\)\.

Equation[1](https://arxiv.org/html/2608.24044#S3.E1)contains four source–target prediction terms\. The two within\-modality terms, obtained whenm=nm=n, retain ordinary visual and physical self\-prediction\. The two cross\-modal terms, obtained whenm≠nm\\neq n, train a history from either modality to predict the corresponding future representation in the other modality\. Each source history is matched to both future representations of the same trajectory; we call this*corresponding cross\-prediction*\. In particular, the predicted visual transition is constrained by the corresponding physical future, not only by the visual future\.

Equation[1](https://arxiv.org/html/2608.24044#S3.E1)also implies predictor sharing\. The predictorgψg\_\{\\psi\}receives no explicit modality indicator, so a single action\-conditioned transition model must operate on histories produced by either encoder\. Predictor sharing does not impose correspondence between the two latent spaces, so we ablate it separately from corresponding cross\-prediction in Section[4](https://arxiv.org/html/2608.24044#S4)\.

The visual and physical encoders are optimized jointly throughout training, with collapse controlled independently in each branch by the isotropy regularizer\. The physical latent therefore adapts to the predictive objective rather than serving as a fixed target for the visual branch\.

Figure 2:Physical supervision can constrain representation content or predictive dynamics\.Visual self\-prediction jointly trains a visual representation and predictor, while state regression makes physical information decodable from individual visual latents\. Neither objective directly constrains a predicted visual transition using the corresponding physical future\. XP\-JEPA advances visual and privileged histories through a shared predictor and matches each prediction to the corresponding future representations in both modalities\. The physical branch is removed after training, leaving a visual\-only model at deployment\.
### A unified privileged\-state interface across tasks

Applying cross\-modal grounding across heterogeneous tasks requires a consistent way to represent privileged physical state\. We therefore describe every scene using a unified rigid\-body schema, as shown in Figure[3](https://arxiv.org/html/2608.24044#S3.F3)\.

Each object contributes a token

τi=\[Gi∥ei∥Ri∥ti\],\\tau\_\{i\}=\[\\,G\_\{i\}\\,\\\|\\,e\_\{i\}\\,\\\|\\,R\_\{i\}\\,\\\|\\,t\_\{i\}\\,\],whereGiG\_\{i\}is a fixed\-width descriptor of its canonical geometry \(a pooled signed\-distance field,[Park et al\., 2019](https://arxiv.org/html/2608.24044#bib.bib25)\),eie\_\{i\}contains its half\-extents, and\(Ri,ti\)\(R\_\{i\},t\_\{i\}\)specifies its pose\. An effector token contains the end\-effector rotation, position, and finger opening, while a learned table token completes the set\. Scenes are padded to six object slots, with unused slots masked from both self\-attention and pooling\.

Because objects are represented by their geometry rather than their identity, the schema requires neither category labels nor task\-specific state fields\. A single physical encoderhϕh\_\{\\phi\}can therefore process every configuration in the suite\. Appendix[D](https://arxiv.org/html/2608.24044#A4)provides the complete dimensions and normalization procedure\.

The unified rigid\-body schema provides a consistent physical description across configurations\. This consistency enables cross\-predictive coupling between visual and physical trajectories across tasks, allowing the shared predictor to learn common rigid\-body and interaction dynamics without adapting to task\-specific state formats\.

![Refer to caption](https://arxiv.org/html/2608.24044v1/unified_representation.png)Figure 3:Each scene is represented by object, end\-effector, and table tokens under a shared schema\.The physical encoder \(hϕh\_\{\\phi\}\) first applies masked self\-attention over the valid scene tokens, then uses a learned query to attend to and pool the token set into a single physical latent \(ztsz\_\{t\}^\{s\}\)\. Unused object slots are masked in both stages, allowing the same encoder to process scenes with different numbers of objects\.
### What physical grounding provides

The privileged state captures object geometry and pose together with the end\-effector pose and gripper aperture, exposing spatial relations that camera projection can obscure\. As illustrated in Figure[1](https://arxiv.org/html/2608.24044#S1.F1), the privileged state distinguishes a block held between the gripper’s fingers from one resting farther along the same camera ray\. Although no contact label is provided, the physical trajectory reveals both the current configuration and its consequence under action: the held block follows the lift, whereas the separated block remains on the table\.

Corresponding cross\-prediction transfers this relational information to the visual model by predicting each modality’s future from the other modality’s history\. Unlike state regression or pointwise alignment, it connects interaction\-relevant configuration with action\-conditioned evolution\. The physical stream is removed after training, leaving this structure in the deployed visual representation and predictor\.

### Objective decomposition

The interaction between representation correspondence and predictive evolution becomes explicit under uniform weighting of the four prediction terms\. Define

pm=gψ​\(Ztm,at\),μt\+1=12​\(zt\+1o\+zt\+1s\)\.p^\{m\}=g\_\{\\psi\}\(Z\_\{t\}^\{m\},a\_\{t\}\),\\qquad\\mu\_\{t\+1\}=\\tfrac\{1\}\{2\}\\left\(z\_\{t\+1\}^\{o\}\+z\_\{t\+1\}^\{s\}\\right\)\.By the parallelogram identity,

∑m,n∈\{o,s\}‖pm−zt\+1n‖22=2​‖po−μt\+1‖22\+2​‖ps−μt\+1‖22\+‖zt\+1o−zt\+1s‖22\.\\sum\_\{m,n\\in\\\{o,s\\\}\}\\left\\\|p^\{m\}\-z\_\{t\+1\}^\{n\}\\right\\\|\_\{2\}^\{2\}=2\\left\\\|p^\{o\}\-\\mu\_\{t\+1\}\\right\\\|\_\{2\}^\{2\}\+2\\left\\\|p^\{s\}\-\\mu\_\{t\+1\}\\right\\\|\_\{2\}^\{2\}\+\\left\\\|z\_\{t\+1\}^\{o\}\-z\_\{t\+1\}^\{s\}\\right\\\|\_\{2\}^\{2\}\.\(2\)
Equation[2](https://arxiv.org/html/2608.24044#S3.E2)separates representation correspondence from predictive evolution\. The final term aligns the two representations of the future scene; the first two train each modality’s history to predict that shared future under action\. Because both targets are online encoder outputs, the coupling is symmetric rather than a one\-way distillation toward a fixed physical target\. XP\-JEPA thus couples pointwise correspondence to action\-conditioned prediction instead of relying on alignment alone\.

### Deployment

Privileged state is used only during training\. At deployment, the physical encoderhϕh\_\{\\phi\}is discarded, leaving the visual encoder and predictor\(fθ,gψ\)\(f\_\{\\theta\},g\_\{\\psi\}\)\. The deployed model therefore receives the same observations and has the same inference architecture and computational cost as the visual\-only baseline\.

Candidate action sequences are rolled forward in the learned latent dynamics and scored by the distance between their predicted terminal latent and the encoded goal:

J\(a1:T\)=‖z^T\(a1:T\)−fθ\(og\)‖22,z^i\+1=gψ\(Z^i,ai\)\.J\(a\_\{1:T\}\)=\\left\\\|\\hat\{z\}\_\{T\}\(a\_\{1:T\}\)\-f\_\{\\theta\}\(o\_\{g\}\)\\right\\\|\_\{2\}^\{2\},\\qquad\\hat\{z\}\_\{i\+1\}=g\_\{\\psi\}\(\\hat\{Z\}\_\{i\},a\_\{i\}\)\.\(3\)The selected sequence is executed and planning repeats from the resulting state\. At test time, the effect of privileged state is carried entirely by the visual representation and predictive dynamics learned during training; it is neither observed nor reconstructed during planning\. More forecastable dynamics reduce one source of error in this rollout\-based score, although successful control also requires latent distance to remain aligned with physical outcomes\.

On the multi\-task suite, candidates are drawn withzz\-CEM: a small unconditional variational autoencoder is trained on action chunks from the same corpus, CEM searches its latent space, and each candidate decodes to a temporally coherent action chunk\. Searching this space avoids direct search over raw action sequences, which yields incoherent candidates far from the action distribution seen in training\. The autoencoder observes neither the task nor the scene, so the same action prior serves every method\. The single\-task experiments retain LeWM’s released action\-space CEM planner\. The autoencoder specification and full planner hyperparameters are given in Appendix[E](https://arxiv.org/html/2608.24044#A5)\.

## 4Experiments

### Setup

We evaluate on four single\-task LeWM environments—*Push\-T*\([Chi et al\., 2025](https://arxiv.org/html/2608.24044#bib.bib23)\),*OGBench\-Block*\([Park et al\., 2025](https://arxiv.org/html/2608.24044#bib.bib21)\),*Two\-Room*, and*Reacher*\([Tassa et al\., 2018](https://arxiv.org/html/2608.24044#bib.bib22)\)—using the released datasets, planners, and success predicates\([Maes et al\., 2026](https://arxiv.org/html/2608.24044#bib.bib6)\)\. We additionally evaluate on a multi\-task tabletop suite built on Meta\-World\([Yu et al\., 2020](https://arxiv.org/html/2608.24044#bib.bib18);[Todorov et al\., 2012](https://arxiv.org/html/2608.24044#bib.bib24)\), where a single model is trained jointly across 22 object–task configurations spanning 13 distinct assets\. Control is evaluated on six evaluation subfamilies; some configurations contribute training transitions but no evaluated scenario\. Forecastability and decodability are measured on held\-out episodes spanning every configuration\. Appendix[A](https://arxiv.org/html/2608.24044#A1)gives the exact configuration counts and the mapping between configurations and evaluated families\. Core comparisons share the training data, visual encoder, optimization schedule, and deployment planner; each ablation changes only the stated objective or architectural component\. Unless stated otherwise, all results are reported over three training seeds\.

We evaluate three properties\.*Control*is closed\-loop task success\.*Forecastability*measures how readily the learned representation supports action\-conditioned prediction independently of its co\-trained predictor\. For each method, we freeze the visual encoder, discard the co\-trained predictor, and fit the same lightweight action\-conditioned predictor from scratch\. Refitting the predictor separates representation forecastability from encoder–predictor co\-adaptation during training\. Because the same predictor family and fitting protocol are used for every frozen representation, differences in rollout error more directly reflect how readily each latent space supports action\-conditioned prediction\. We report rollout drift relative to a temporal\-persistence baseline, with lower values indicating more forecastable dynamics\.

Figure 4:XP\-JEPA produces more forecastable latent dynamics throughout the rollout\.Multi\-task suite, three training seeds\.*Decodability*is measured by theR2R^\{2\}of physical variables predicted from the frozen visual latent; we report the position of the manipulated object, and Appendix[C](https://arxiv.org/html/2608.24044#A3)specifies the full decoding target and the remaining groups\. Appendices[A](https://arxiv.org/html/2608.24044#A1)and[C](https://arxiv.org/html/2608.24044#A3)provide the complete evaluation protocols\.

### XP\-JEPA improves forecastability

A fresh predictor models XP\-JEPA’s action\-conditioned dynamics more accurately than the baselines throughout the rollout\.

On the multi\-task suite, relative rollout drift falls from0\.3610\.361forVisualto0\.1040\.104for XP\-JEPA\. Figure[4](https://arxiv.org/html/2608.24044#S4.F4)shows that the separation persists at every horizon, including beyondh=3h=3, when the rollout becomes fully autoregressive\. Similar temporal\-persistence errors and rank\-matched PCA show that neither reduced temporal variation nor effective rank alone explains the gap \(Appendix[C](https://arxiv.org/html/2608.24044#A3)\)\.

Figure 5:Grounding improves multi\-task control\.XP\-JEPA raises overall success from53\.6%53\.6\\%to78\.2%78\.2\\%, with improvements across all six interaction families\. Regressing a comprehensive pose\-based physical\-state target from the visual latent does not recover the gain\. Bars show means±\\pmsample SD over three training seeds; dots show individual seeds\.The improvement is consistent across all four matched single\-task environments \(Table[1](https://arxiv.org/html/2608.24044#S4.T1)\)\. Relative drift decreases from0\.5030\.503to0\.2210\.221on Two\-Room, from0\.5070\.507to0\.2690\.269on OGBench\-Block, from0\.3990\.399to0\.2580\.258on Push\-T, and from0\.3130\.313to0\.2330\.233on Reacher\.

Table 1:XP\-JEPA produces the most forecastable visual dynamics across all four matched single\-task environments\.Forecastability is measured using a predictor fitted from scratch to each frozen representation\. Object\-positionR2R^\{2\}measures how well the environment’s object position is decoded from the frozen visual latent\. Values are means±\\pmsample SD over three training seeds\. Bold marks the best mean in each comparison; exact ties are all bold\.
### Grounding improves multi\-task control

On the multi\-task suite, XP\-JEPA raises mean task success from53\.6%53\.6\\%to78\.2%78\.2\\%, a gain of24\.624\.6percentage points under the same planner, action prior, and candidate budget \(Figure[5](https://arxiv.org/html/2608.24044#S4.F5)\)\. The improvement holds across all six interaction families\.

Figure[6](https://arxiv.org/html/2608.24044#S4.F6)compares the two models’ predictions under identical action sequences\. Because both models predict in latent space, we fit a separate state decoder, decode the manipulated\-object and end\-effector positions, and render the resulting rollouts alongside the ground truth\. XP\-JEPA imagines the eraser arriving on top of the block, whereasVisualplaces it on the table beside the block\. Across the three planning cycles of this episode, the decoded terminal position error averages1\.41\.4cm for XP\-JEPA and3\.13\.1cm forVisual\(Appendix[C](https://arxiv.org/html/2608.24044#A3.SS0.SSS0.Px4)\)\. The decoded rollouts make the drift difference concrete: the same actions lead the two models to different imagined terminal configurations\.

![Refer to caption](https://arxiv.org/html/2608.24044v1/imagined_rollout.png)Figure 6:Rolled over identical expert actions, XP\-JEPA imagines the eraser reaching the block;Visualimagines it beside the block\.Middle row: ground truth\. Top and bottom: each model’s own rollout over the same action chunks from the same observation, with the decoded object and end\-effector positions written back into the simulator and rendered\.State regression does not recover these forecastability or control gains\.Regressdirectly predicts a comprehensive pose\-based target from the visual latent and raises manipulated\-object position decodability toR2=0\.991R^\{2\}=0\.991, from0\.9780\.978for both XP\-JEPA andVisual\. Its control success remains at53\.7%53\.7\\%and its relative rollout drift at0\.3730\.373\. Position can therefore be readily available to a probe even when the latent transition remains difficult to forecast\. On the matched single\-task environments, XP\-JEPA significantly improves control on Two\-Room and OGBench\-Block, while its mean success is within one percentage point ofVisualon Push\-T and Reacher \(Table[2](https://arxiv.org/html/2608.24044#S4.T2)\)\. Forecastability improves across all four environments, whereas the control gains are task dependent\. This separation helps clarify the role of forecastability in planning\. XP\-JEPA makes the latent dynamics easier to predict in all four environments, but the downstream control benefit depends on whether rollout error is consequential for action selection in the task\. The single\-task results therefore support a more limited claim: forecastable dynamics improve one component required by rollout\-based planning, rather than guaranteeing higher control success on every task\.

Table 2:Matched single\-task control success\.Values are mean success rates \(%\)±\\pmsample SD over three training seeds\. Bold marks the best mean in each row; exact ties are all bold\.
### Correspondence and cross\-prediction play distinct roles

We next separate the effects of cross\-modal correspondence, predictor sharing, and exposure to physical trajectories\.Cross\-onlyretains corresponding cross\-prediction but uses separate predictors for the visual and physical streams\.Share\-onlyretains the shared predictor but removes the cross\-modal prediction terms\.Align\-onlyreplaces cross\-modal prediction with symmetric frame\-level alignment while retaining within\-modality prediction\.Distillreplaces predictive coupling with pointwise regression onto a stop\-gradient physical latent\. Finally,Shuffleretains the complete XP\-JEPA architecture but uses a fixed corpus\-wide derangement to pair each visual trajectory with an intact, incorrect privileged trajectory\.

Table 3:Correspondence supports control, while direct cross\-modal prediction improves forecastability beyond alignment\.The design columns indicate cross\-modal prediction, predictor sharing, and correct visual–physical correspondence \(✓ present,×\\timespresent but mispaired, – absent\)\. Drift and control are means±\\pmsample SD over three training seeds; bold marks the best mean in each column\.Table[3](https://arxiv.org/html/2608.24044#S4.T3)shows that corresponding cross\-prediction contributes more to control than predictor sharing alone\.Cross\-onlyreaches73\.0%73\.0\\%control, recovering roughly four fifths of the gain fromVisualto XP\-JEPA, with comparable rollout drift to XP\-JEPA\.Share\-onlyreaches56\.7%56\.7\\%control, close to theVisualbaseline\. Most of the control gain therefore survives without a common predictor, whereas sharing a transition model without cross\-modal coupling adds little\.

Align\-onlyrecovers most of the control gain \(72\.6%72\.6\\%against78\.2%78\.2\\%\) but has higher drift than XP\-JEPA \(0\.1580\.158against0\.1040\.104\), indicating that direct cross\-modal prediction improves forecastability beyond pointwise alignment\. Its aligned states are also trained with within\-modality prediction, so it still carries cross\-modal information forward indirectly\. Together, these comparisons separate two effects: correspondence accounts for much of the control improvement, while direct prediction across modalities further reduces rollout drift\.

Pointwise distillation does not reproduce XP\-JEPA’s gains:Distillreaches57\.7%57\.7\\%control and increases relative rollout drift to0\.5160\.516, compared with0\.3610\.361forVisual\. Together withRegress, this shows that making physical state decodable or matching it pointwise is not a substitute for grounding the transition itself\.

Shuffleisolates pairing quality: despite retaining the predictive architecture and intact physical trajectories, the fixed incorrect pairing reduces control to16\.6%16\.6\\%while drift remains below the visual baseline at0\.1670\.167\. This combination of low drift and poor control shows that predictable dynamics can still organize outcomes in a way the planner cannot use\.

These ablations suggest two partly separable roles for physical grounding\. Correct visual–physical correspondence makes the learned transition geometry relevant to physical outcomes and therefore useful for control, whereas direct cross\-modal prediction more strongly constrains how that geometry evolves under action\. Neither property is sufficient by itself: alignment recovers much of the control gain without matching XP\-JEPA’s forecastability, while SHUFFLE retains relatively low rollout drift under incorrect correspondence but fails at control\.

## 5Discussion and Conclusion

Across the multi\-task suite, XP\-JEPA reduces relative rollout drift from0\.3610\.361to0\.1040\.104and raises mean control success from53\.6%53\.6\\%to78\.2%78\.2\\%\. Direct state regression improves position decodability without comparable gains in forecastability or control, suggesting that snapshot content alone does not determine whether a representation supports reliable rollout prediction\. XP\-JEPA instead grounds visual latent evolution in corresponding physical trajectories\. Because both encoders are learned jointly, this does not impose a canonical physical representation; privileged state serves as a second structured view that constrains predictive dynamics\.

The ablations separate two roles of this grounding\. Correct visual–physical correspondence supports control, while direct cross\-modal prediction most clearly improves forecastability beyond alignment\. Predictor sharing alone contributes little, andShuffleretains relatively low drift but collapses to16\.6%16\.6\\%control\. Forecastability is therefore useful but not sufficient: predicted trajectories must also preserve distinctions relevant to action selection\.

This matters for rollout\-based planning, where candidate actions are ranked by distances between predicted terminal latents and the goal\. XP\-JEPA retains its lower drift after replacing the co\-trained predictor, indicating that the gain is reflected in the learned representation\. At the same time, the task\-dependent control gains show that improved forecastability does not guarantee higher success in every environment\.

Our results are limited to simulation, paired privileged trajectories during training, and a fixed rigid\-body state schema\. Extending this approach to real\-world observations and broader physical interactions remains future work\.

## References

- Assranet al\.\(2023\)M\. Assran, Q\. Duval, I\. Misra, P\. Bojanowski, P\. Vincent, M\. Rabbat, Y\. LeCun, and N\. BallasSelf\-supervised learning from images with a joint\-embedding predictive architecture\.In2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 15619–15629\.Cited by:[§1](https://arxiv.org/html/2608.24044#S1.p1.1),[§2](https://arxiv.org/html/2608.24044#S2.SS0.SSS0.Px1.p1.1)\.
- Balestriero and LeCun \(2025\)R\. Balestriero and Y\. LeCunLeJEPA: provable and scalable self\-supervised learning without the heuristics\.arXiv preprint arXiv:2511\.08544\.Cited by:[§3](https://arxiv.org/html/2608.24044#S3.SS0.SSS0.Px1.p4.2)\.
- Bardeset al\.\(2024\)A\. Bardes, Q\. Garrido, J\. Ponce, X\. Chen, M\. Rabbat, Y\. LeCun, M\. Assran, and N\. BallasRevisiting feature prediction for learning visual representations from video\.arXiv preprint arXiv:2404\.08471\.Cited by:[§2](https://arxiv.org/html/2608.24044#S2.SS0.SSS0.Px1.p1.1)\.
- Bardeset al\.\(2023\)A\. Bardes, J\. Ponce, and Y\. LeCunMC\-JEPA: a joint\-embedding predictive architecture for self\-supervised learning of motion and content features\.arXiv preprint arXiv:2307\.12698\.Cited by:[§2](https://arxiv.org/html/2608.24044#S2.SS0.SSS0.Px1.p1.1)\.
- Chenet al\.\(2020\)D\. Chen, B\. Zhou, V\. Koltun, and P\. KrähenbühlLearning by cheating\.InProceedings of the Conference on Robot Learning,Vol\.100,pp\. 66–75\.Cited by:[§1](https://arxiv.org/html/2608.24044#S1.p3.1),[§2](https://arxiv.org/html/2608.24044#S2.SS0.SSS0.Px2.p1.1)\.
- Chiet al\.\(2025\)C\. Chi, Z\. Xu, S\. Feng, E\. Cousineau, Y\. Du, B\. Burchfiel, R\. Tedrake, and S\. SongDiffusion policy: visuomotor policy learning via action diffusion\.The International Journal of Robotics Research44\(10\-11\),pp\. 1684–1704\.Cited by:[§4](https://arxiv.org/html/2608.24044#S4.SS0.SSS0.Px1.p1.1)\.
- Dosovitskiyet al\.\(2021\)A\. Dosovitskiy, L\. Beyer, A\. Kolesnikov, D\. Weissenborn, X\. Zhai, T\. Unterthiner, M\. Dehghani, M\. Minderer, G\. Heigold, S\. Gelly,et al\.An image is worth 16x16 words: transformers for image recognition at scale\.InInternational Conference on Learning Representations,Cited by:[Appendix E](https://arxiv.org/html/2608.24044#A5.SS0.SSS0.Px1.p1.1)\.
- Guptaet al\.\(2016\)S\. Gupta, J\. Hoffman, and J\. MalikCross modal distillation for supervision transfer\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition,pp\. 2827–2836\.Cited by:[§1](https://arxiv.org/html/2608.24044#S1.p3.1),[§2](https://arxiv.org/html/2608.24044#S2.SS0.SSS0.Px2.p1.1)\.
- Ha and Schmidhuber \(2018\)D\. Ha and J\. SchmidhuberRecurrent world models facilitate policy evolution\.InAdvances in Neural Information Processing Systems,Vol\.31\.Cited by:[§1](https://arxiv.org/html/2608.24044#S1.p1.1),[§2](https://arxiv.org/html/2608.24044#S2.SS0.SSS0.Px1.p1.1)\.
- Hafneret al\.\(2020\)D\. Hafner, T\. Lillicrap, J\. Ba, and M\. NorouziDream to control: learning behaviors by latent imagination\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2608.24044#S2.SS0.SSS0.Px1.p1.1)\.
- Hafneret al\.\(2019\)D\. Hafner, T\. Lillicrap, I\. Fischer, R\. Villegas, D\. Ha, H\. Lee, and J\. DavidsonLearning latent dynamics for planning from pixels\.InInternational Conference on Machine Learning,pp\. 2555–2565\.Cited by:[§1](https://arxiv.org/html/2608.24044#S1.p1.1),[§2](https://arxiv.org/html/2608.24044#S2.SS0.SSS0.Px1.p1.1)\.
- Hafneret al\.\(2025\)D\. Hafner, J\. Pasukonis, J\. Ba, and T\. LillicrapMastering diverse control tasks through world models\.Nature640,pp\. 647–653\.External Links:[Document](https://dx.doi.org/10.1038/s41586-025-08744-2)Cited by:[§1](https://arxiv.org/html/2608.24044#S1.p1.1),[§2](https://arxiv.org/html/2608.24044#S2.SS0.SSS0.Px1.p1.1)\.
- Hansenet al\.\(2024\)N\. Hansen, H\. Su, and X\. WangTD\-MPC2: scalable, robust world models for continuous control\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.24044#S1.p1.1),[§2](https://arxiv.org/html/2608.24044#S2.SS0.SSS0.Px1.p1.1)\.
- Huet al\.\(2024\)E\. Hu, J\. Springer, O\. Rybkin, and D\. JayaramanPrivileged sensing scaffolds reinforcement learning\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2608.24044#S2.SS0.SSS0.Px3.p1.1)\.
- Huanget al\.\(2025\)D\. Huang, J\. Wang, Y\. Li, C\. Xia, T\. Zhang, and K\. ZhangPIGDreamer: privileged information guided world models for safe partially observable reinforcement learning\.arXiv preprint arXiv:2508\.02159\.Cited by:[§2](https://arxiv.org/html/2608.24044#S2.SS0.SSS0.Px3.p1.1)\.
- Kimet al\.\(2026\)J\. Kim, J\. Cho, S\. Chu, A\. Bal, J\. Kim, G\. Lee, S\. Lee, S\. H\. Kim, B\. Han, H\. Lee,et al\.Pri4R: learning world dynamics for vision\-language\-action models with privileged 4D representation\.arXiv preprint arXiv:2603\.01549\.Cited by:[§2](https://arxiv.org/html/2608.24044#S2.SS0.SSS0.Px3.p1.1)\.
- Kumaret al\.\(2021\)A\. Kumar, Z\. Fu, D\. Pathak, and J\. MalikRMA: rapid motor adaptation for legged robots\.InProceedings of Robotics: Science and Systems,External Links:[Document](https://dx.doi.org/10.15607/RSS.2021.XVII.011)Cited by:[§1](https://arxiv.org/html/2608.24044#S1.p3.1),[§2](https://arxiv.org/html/2608.24044#S2.SS0.SSS0.Px2.p1.1)\.
- LeCun \(2022\)Y\. LeCunA path towards autonomous machine intelligence\.Note:OpenReviewVersion 0\.9\.2Cited by:[§1](https://arxiv.org/html/2608.24044#S1.p1.1),[§2](https://arxiv.org/html/2608.24044#S2.SS0.SSS0.Px1.p1.1)\.
- Leeet al\.\(2020\)J\. Lee, J\. Hwangbo, L\. Wellhausen, V\. Koltun, and M\. HutterLearning quadrupedal locomotion over challenging terrain\.Science Robotics5\(47\),pp\. eabc5986\.Cited by:[§2](https://arxiv.org/html/2608.24044#S2.SS0.SSS0.Px2.p1.1)\.
- Maeset al\.\(2026\)L\. Maes, Q\. L\. Lidec, D\. Scieur, Y\. LeCun, and R\. BalestrieroLeWorldModel: stable end\-to\-end joint\-embedding predictive architecture from pixels\.arXiv preprint arXiv:2603\.19312\.Cited by:[§2](https://arxiv.org/html/2608.24044#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2608.24044#S3.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2608.24044#S4.SS0.SSS0.Px1.p1.1)\.
- Nieet al\.\(2026\)W\. Nie, W\. Liu, H\. Guo, and Y\. SuPhys\-JEPA: physics\-informed latent world models for multivariate time\-series forecasting\.arXiv preprint arXiv:2606\.16076\.Cited by:[§2](https://arxiv.org/html/2608.24044#S2.SS0.SSS0.Px3.p1.1)\.
- Parket al\.\(2019\)J\. J\. Park, P\. Florence, J\. Straub, R\. Newcombe, and S\. LovegroveDeepSDF: learning continuous signed distance functions for shape representation\.In2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 165–174\.Cited by:[§3](https://arxiv.org/html/2608.24044#S3.SS0.SSS0.Px2.p2.2)\.
- Parket al\.\(2025\)S\. Park, K\. Frans, B\. Eysenbach, and S\. LevineOGBench: benchmarking offline goal\-conditioned RL\.InInternational Conference on Learning Representations,Cited by:[§4](https://arxiv.org/html/2608.24044#S4.SS0.SSS0.Px1.p1.1)\.
- Peebles and Xie \(2023\)W\. Peebles and S\. XieScalable diffusion models with transformers\.In2023 IEEE/CVF International Conference on Computer Vision \(ICCV\),pp\. 4172–4182\.Cited by:[§3](https://arxiv.org/html/2608.24044#S3.SS0.SSS0.Px1.p4.2)\.
- Pintoet al\.\(2018\)L\. Pinto, M\. Andrychowicz, P\. Welinder, W\. Zaremba, and P\. AbbeelAsymmetric actor critic for image\-based robot learning\.InProceedings of Robotics: Science and Systems,External Links:[Document](https://dx.doi.org/10.15607/RSS.2018.XIV.008)Cited by:[§1](https://arxiv.org/html/2608.24044#S1.p3.1),[§2](https://arxiv.org/html/2608.24044#S2.SS0.SSS0.Px2.p1.1)\.
- Schrittwieseret al\.\(2020\)J\. Schrittwieser, I\. Antonoglou, T\. Hubert, K\. Simonyan, L\. Sifre, S\. Schmitt, A\. Guez, E\. Lockhart, D\. Hassabis, T\. Graepel,et al\.Mastering Atari, Go, chess and shogi by planning with a learned model\.Nature588\(7839\),pp\. 604–609\.Cited by:[§2](https://arxiv.org/html/2608.24044#S2.SS0.SSS0.Px1.p1.1)\.
- Sobalet al\.\(2025\)U\. Sobal, W\. Zhang, K\. Cho, R\. Balestriero, T\. G\. J\. Rudner, and Y\. LeCunLearning from reward\-free offline data: a case for planning with latent dynamics models\.Advances in Neural Information Processing Systems38,pp\. 43905–43941\.Cited by:[§1](https://arxiv.org/html/2608.24044#S1.p1.1),[§2](https://arxiv.org/html/2608.24044#S2.SS0.SSS0.Px1.p1.1)\.
- Tassaet al\.\(2018\)Y\. Tassa, Y\. Doron, A\. Muldal, T\. Erez, Y\. Li, D\. d\. L\. Casas, D\. Budden, A\. Abdolmaleki, J\. Merel, A\. Lefrancq,et al\.DeepMind control suite\.arXiv preprint arXiv:1801\.00690\.Cited by:[§4](https://arxiv.org/html/2608.24044#S4.SS0.SSS0.Px1.p1.1)\.
- Tianet al\.\(2020\)Y\. Tian, D\. Krishnan, and P\. IsolaContrastive multiview coding\.InEuropean Conference on Computer Vision,pp\. 776–794\.Cited by:[§1](https://arxiv.org/html/2608.24044#S1.p3.1),[§2](https://arxiv.org/html/2608.24044#S2.SS0.SSS0.Px2.p1.1)\.
- Todorovet al\.\(2012\)E\. Todorov, T\. Erez, and Y\. TassaMuJoCo: a physics engine for model\-based control\.In2012 IEEE/RSJ International Conference on Intelligent Robots and Systems,pp\. 5026–5033\.Cited by:[§4](https://arxiv.org/html/2608.24044#S4.SS0.SSS0.Px1.p1.1)\.
- Vapnik and Izmailov \(2015\)V\. Vapnik and R\. IzmailovLearning using privileged information: similarity control and knowledge transfer\.The Journal of Machine Learning Research16\(1\),pp\. 2023–2049\.Cited by:[§2](https://arxiv.org/html/2608.24044#S2.SS0.SSS0.Px2.p1.1)\.
- Yamadaet al\.\(2024\)J\. Yamada, M\. Rigter, J\. Collins, and I\. PosnerTWIST: teacher\-student world model distillation for efficient sim\-to\-real transfer\.In2024 IEEE International Conference on Robotics and Automation \(ICRA\),pp\. 9190–9196\.Cited by:[§2](https://arxiv.org/html/2608.24044#S2.SS0.SSS0.Px3.p1.1)\.
- Yuet al\.\(2020\)T\. Yu, D\. Quillen, Z\. He, R\. Julian, K\. Hausman, C\. Finn, and S\. LevineMeta\-World: a benchmark and evaluation for multi\-task and meta reinforcement learning\.InProceedings of the Conference on Robot Learning,Vol\.100,pp\. 1094–1100\.Cited by:[§4](https://arxiv.org/html/2608.24044#S4.SS0.SSS0.Px1.p1.1)\.
- Zenget al\.\(2026\)X\. Zeng, H\. Ren, and Z\. SongPhyLatent: learning dynamics\-relevant representations for JEPA world models\.arXiv preprint arXiv:2608\.05720\.Cited by:[§2](https://arxiv.org/html/2608.24044#S2.SS0.SSS0.Px3.p1.1)\.
- Zhouet al\.\(2024\)G\. Zhou, H\. Pan, Y\. LeCun, and L\. PintoDINO\-WM: world models on pre\-trained visual features enable zero\-shot planning\.arXiv preprint arXiv:2411\.04983\.Cited by:[§1](https://arxiv.org/html/2608.24044#S1.p1.1),[§2](https://arxiv.org/html/2608.24044#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix ASuite and Evaluation Details

### Suite composition\.

Our Meta\-World\-based corpus contains2222object–task configurations over1313distinct assets, with450450clean episodes per configuration before noise augmentation\. A single model is trained jointly across all configurations\.

Table 4:The2222configurations in the multi\-task suite\.
### Success predicates\.

Success is determined from the executed physical trajectory rather than latent distance to the goal\. Positional tolerances are asset\-specific, and orientation is evaluated modulo each asset’s declared symmetry\.

Table 5:Success predicates for the six evaluated interaction families\.
### Scenario sampling\.

Each evaluated interaction family contains5050held\-out scenarios per training seed, giving300300scenarios per model\. Scenarios are stratified across configuration–subtask pairs and sampled with a fixed seed, so all methods are evaluated from the same initial states\. The300300scenarios are drawn from266266source episodes; statistical tests therefore cluster segments originating from the same episode\.

The interaction groups in Table[4](https://arxiv.org/html/2608.24044#A1.T4)label corpus configurations, whereas the six evaluated families are subtask labels annotated within each episode\. Scenarios are keyed by configuration–subtask pair, and each family’s quota is filled round\-robin across every configuration containing that subtask\. A grasp–move–place configuration therefore contributes segments to the grasping and placement families, the peg configurations to insertion, and the pushing, stacking and tossing configurations to their like\-named families\.

For the four LeWM environments, we use the benchmark’s released success predicates and evaluation loop unchanged\. Each controller is evaluated on the sameN=150N=150paired scenarios\.

## Appendix BFull Control Results

### Per\-family results\.

Table[6](https://arxiv.org/html/2608.24044#A2.T6)reports the per\-family results underlying Figure[5](https://arxiv.org/html/2608.24044#S4.F5)\.

Table 6:Per\-family control success \(%\), mean±\\pmsample SD over three training seeds\. Families are ordered by the XP\-JEPA–Visualgap, matching Figure[5](https://arxiv.org/html/2608.24044#S4.F5)\.
### Statistical tests\.

For each training seed, suite comparisons use an episode\-clustered paired sign\-flip test with10510^\{5\}Monte Carlo assignments\. Scenarios originating from the same source episode share a sign\. The per\-seed XP\-JEPA–Visualdifferences are\+28\.3\+28\.3,\+22\.0\+22\.0, and\+23\.7\+23\.7percentage points, while the corresponding XP\-JEPA–Distilldifferences are\+17\.3\+17\.3,\+18\.3\+18\.3, and\+26\.0\+26\.0percentage points\. All six seed\-level comparisons yieldpMC≤10−5p\_\{\\mathrm\{MC\}\}\\leq 10^\{\-5\}, the smallest value10510^\{5\}assignments can resolve\. These tests quantify paired scenario\-level differences for each trained model; consistency across training seeds is reported separately\.

Single\-task comparisons use exact McNemar tests on the sameN=150N=150paired scenarios\. XP\-JEPA exceedsVisualin every seed on both Two\-Room and OGBench\-Block, with the largestpp\-value equal to0\.0230\.023\.

For the comparison between XP\-JEPA andCross\-only, the training run is the unit of analysis\. Mean suite control is72\.3/74\.0/72\.772\.3/74\.0/72\.7forCross\-onlyand79\.0/76\.7/79\.079\.0/76\.7/79\.0for XP\-JEPA\.

## Appendix CForecastability Protocol and Additional Results

### Fresh\-predictor probe\.

To measure representation forecastability independently of the transition model used during representation learning, we freeze each encoder, cache its latents, discard the original predictor, and train a new predictor from scratch\. We use the same lightweight probe family across methods, equalizing predictor capacity so that differences primarily reflect predictive structure in the learned representation rather than the co\-trained transition model\.

For the single\-task experiments, the probe is action\-conditioned\. Its input concatenates theH=3H=3most recent latent frames with the correspondingH=3H=3action chunks\. A three\-layer MLP of width512512with GELU activations predicts the next latent as a residual update to the most recent frame\. Latents are standardized per dimension using statistics from the fitting episodes; actions are used as stored\. We train with AdamW at learning rate10−310^\{\-3\}and weight decay10−510^\{\-5\}for30,00030\{,\}000steps, with batch size256256on10001000episodes and a randomized fit/evaluation episode split\.

At evaluation, the probe is rolled out autoregressively for2020steps, feeding back its own predictions while consuming the recorded action sequence\. The resulting error therefore measures prediction under actions rather than latent smoothness alone\. Normalization by a constant\-latent baseline further removes credit for temporal persistence\.

The multi\-task results in Table[7](https://arxiv.org/html/2608.24044#A3.T7)and Figure[4](https://arxiv.org/html/2608.24044#S4.F4)use the same probe family with a direct rather than residual output head,20,00020\{,\}000training steps, batch size512512, cosine learning\-rate decay, and an episode\-level fit/evaluation split\.

Probe fitting and evaluation always use disjoint episodes\. Relative drift is computed independently at each rollout horizon and normalized by a constant\-latent predictor:

drift⁡\(h\)=𝔼​∥z^t\+h−zt\+h∥2𝔼​∥zt−zt\+h∥2\.\\mathrm\{drift\}\(h\)=\\frac\{\\mathbb\{E\}\\lVert\\hat\{z\}\_\{t\+h\}\-z\_\{t\+h\}\\rVert\_\{2\}\}\{\\mathbb\{E\}\\lVert z\_\{t\}\-z\_\{t\+h\}\\rVert\_\{2\}\}\.\(4\)The reported aggregate is the mean of these per\-horizon ratios overh=1,…,20h=1,\\ldots,20\.

### Manipulated\-object decoding probe\.

Decodability is measured using the same frozen encoder checkpoints and episode\-level split as the dynamics probe, so both columns of Table[7](https://arxiv.org/html/2608.24044#A3.T7)evaluate the same representations under matched data partitions\. A single three\-layer MLP of width512512with GELU activations maps one frozen latent frame to physical variables; it is trained with AdamW at learning rate10−310^\{\-3\}and weight decay10−510^\{\-5\}for60006000steps at batch size10241024, and evaluated on the held\-out episodes\. The decoding target is a1515\-dimensional vector comprising the manipulated object’s normalized position, its orientation as a flattened3×33\\times 3rotation matrix, and the normalized end\-effector position\. We computeR2=1−SSE/SSTR^\{2\}=1\-\\mathrm\{SSE\}/\\mathrm\{SST\}separately for each target group over all held\-out frames, with the total sum of squares taken about the held\-out mean\. The tables reportR2R^\{2\}for the object\-position group\. Canonical geometry, bounding\-box extents, and masked object slots are not included in the decoding target\.

### Multi\-task forecastability\.

Table[7](https://arxiv.org/html/2608.24044#A3.T7)reports the quantities underlying the multi\-task comparison\. Under the controlled fresh\-predictor protocol, XP\-JEPA has lower raw rollout error and normalized drift than the visual\-only and state\-regression baselines\.RegressandDistillmake the manipulated object’s position the most decodable of any method, while XP\-JEPA andVisualhave the same mean at the reported precision\. Relative toVisual, neitherRegressnorDistillsubstantially reduces rollout drift, whereas XP\-JEPA does\.Shufflealso yields low drift despite poor control, showing that forecastability alone is insufficient\. Decodability of the physical variable and forecastability of the latent dynamics are therefore separate properties, and the ordering of one does not predict the ordering of the other\.

Table 7:Forecastability and manipulated\-object position decodability on the multi\-task suite\. Both metrics use the same frozen encoders and held\-out split, with a separately fitted probe for each metric\. Drift andR2R^\{2\}are means±\\pmsample SD over three training seeds; bold marks the best reported mean\. Raw and Copy are averaged over horizons and seeds; Drift is the mean of the per\-horizon ratios and therefore differs from their ratio\.The larger Copy denominator forShuffle\(15\.1115\.11, compared with approximately1212for the other methods\) contributes to its lower normalized drift, but its raw rollout error is also lower thanVisual’s \(2\.092\.09versus4\.434\.43\)\. The mispaired objective therefore remains relatively predictable while control fails, confirming that forecastability alone is insufficient under incorrect visual–physical supervision\.

### Imagined trajectories under a fixed action sequence\.

Relative drift summarizes forecastability as a scalar; Figure[7](https://arxiv.org/html/2608.24044#A3.F7)shows what it looks like on one trajectory in physical units\. We take a stacking episode and, at the start of each planning cycle, roll each model’s predictor from the same observation over the same recorded expert action chunks\. The manipulated object’s position is decoded from the predicted latents by a probe fitted on training episodes only \(R2=0\.984R^\{2\}=0\.984and0\.9880\.988for the two models\); first\-step decoding error remains below22cm in every cycle\. With identical actions and starting observations, the plots compare decoded rollout behavior; residual state\-decoder error remains\. Mean terminal error across the three cycles is1\.41\.4cm for XP\-JEPA and3\.13\.1cm forVisual\.

Figure 7:Under the same actions, XP\-JEPA imagines the object coming to rest on its support\.Decoded height of the manipulated object during a stacking episode: ground truth against each model’s own rollout over the identical expert action chunks, one panel per planning cycle\.Visual’s imagined object passes through the support and stays there; XP\-JEPA has a mean terminal error of1\.41\.4cm across the three cycles\.
### Rank\-controlled forecastability\.

Relative drift compares a fitted predictor against a temporal\-persistence baseline, so a representation of lower intrinsic dimension could in principle be easier for a fixed\-capacity probe to fit\. XP\-JEPA does produce a lower\-dimensional latent: measured as the participation ratio of the eigenspectrum, its effective rank is56\.5±0\.256\.5\_\{\\pm 0\.2\}against95\.7±2\.495\.7\_\{\\pm 2\.4\}forVisual, out of192192dimensions, and this holds for every training seed\. To separate dimensionality from content we project each representation onto its leadingkkprincipal components and refit the identical probe\. The basis is fitted on the training episodes only and then applied to all episodes, so the held\-out split does not enter the projection\. Everything else is held fixed: the same episodes, split, actions, probe architecture, and optimization budget, with three probe seeds per condition\.

Table 8:Matching effective rank does not close the forecastability gap\.Relative drift after projection onto the leadingkkprincipal components, reported as mean±\\pmsample SD over three training seeds\.Atk=57k=57, chosen to approximate XP\-JEPA’s native effective rank,Visualand XP\-JEPA reach measured ranks of52\.852\.8and51\.751\.7, with drift of0\.4530\.453and0\.1210\.121, respectively\. Under more aggressive projection both arms degrade and the ratio narrows, from3\.5×3\.5\\timesat native width to1\.9×1\.9\\timesat rank7\.97\.9, but the gap never closes\. The retained variance shows why the two arms respond differently: atk=57k=57, XP\-JEPA retains95\.1%95\.1\\%of its variance whileVisualretains67\.9%67\.9\\%, so the same nominal width discards far more of the visual\-only representation\. Compression never improves the visual baseline, and XP\-JEPA atk=8k=8remains more forecastable thanVisualat any tested width\. The forecastability gap is therefore not explained by effective rank alone\.

## Appendix DPrivileged\-State Interface

This section gives the exact dimensions and normalization of the privileged\-state interface introduced in Section[3](https://arxiv.org/html/2608.24044#S3)\. Each object token concatenates a fixed\-width canonical\-geometry descriptor, bounding\-box half\-extents, a flattened3×33\\times 3rotation matrix, and position\. Our experiments use an838^\{3\}average\-pooled signed\-distance field as the geometry descriptor\. The effector token carries end\-effector rotation, position, and finger opening, while a learned table token completes the set\. Scenes are padded to six object slots with unused slots masked from both self\-attention and pooling, giving eight tokens in total\. All physical quantities are normalized using training\-set statistics\.

The privileged input describes only the instantaneous physical configuration and excludes velocities, wrenches, contact forces, and goal\-relative quantities\. Both branches receive length\-HHhistories, so they infer motion from temporal context and action\.

## Appendix EImplementation Details

### Encoders\.

The visual encoder is a ViT\-Tiny\([Dosovitskiy et al\., 2021](https://arxiv.org/html/2608.24044#bib.bib16)\)with192192\-dimensional features, patch size1414, and224×224224\\times 224input resolution \(5\.50M parameters\)\. The physical encoder is a three\-layer, four\-head Transformer of width192192with hidden width256256\(1\.55M parameters\), consuming the token set defined in Appendix[D](https://arxiv.org/html/2608.24044#A4)\.

### Predictor and objective\.

The action\-conditioned predictor is a six\-layer,1616\-head Transformer operating onH=3H=3latent frames, with head dimension6464, feed\-forward width20482048, and dropout0\.10\.1\. Actions enter through zero\-initialized AdaLN\. Each action chunk spans five environment steps with a one\-step prediction offset\. The four source–target terms in Equation[1](https://arxiv.org/html/2608.24044#S3.E1)receive equal weight\. SIGReg is applied separately to the two branches with weightβ=0\.09\\beta=0\.09,1717knots, and10241024projections\.

Both future latents are online encoder outputs; the XP\-JEPA prediction path contains no stop\-gradient, exponential\-moving\-average encoder, or separate target network\.

### Optimization\.

Core matched methods use AdamW with learning rate7×10−57\\times 10^\{\-5\}and weight decay10−310^\{\-3\}, batch size256256, bfloat16 precision, gradient clipping at1\.01\.0, and a linear\-warmup cosine\-annealing learning\-rate schedule\. All multi\-task models are trained for4040epochs over the augmented corpus\. No image augmentation is applied at training time; input diversity comes from the corpus\-level action\-noise augmentation described under*Data*below\. Training\-time auxiliary components differ only where required by the corresponding objective or ablation\.

### Planning\.

The multi\-task experiments use a horizon of five macro\-steps, each containing five environment actions, with300300candidates,3030CEM refinement iterations, and3030elites \(10%10\\%\)\. The search distribution is initialized at zero mean and unit variance in the latent action space and re\-fitted to the elites at each iteration, with the current mean always evaluated as one candidate\. The planner executes the entire five\-macro\-step sequence, that is2525environment actions, before replanning from the resulting state; the executed segment is therefore open\-loop and planning is repeated once per segment rather than at every macro\-step\. The unconditional action autoencoder used byzz\-CEM is a variational autoencoder with a3232\-dimensional latent whose encoder and decoder are two\-hidden\-layer MLPs of width512512, mapping a flattened125125\-dimensional action chunk to and from the latent\. It is trained on the same corpus for20,00020\{,\}000steps with AdamW at learning rate10−310^\{\-3\}, weight decay10−510^\{\-5\}, and cosine decay, under a reconstruction plus KL objective with KL weightβ=0\.1\\beta=0\.1\. It receives neither observations nor goals, and the same action prior is used for every method\.

Directly sampling raw action sequences can produce temporally incoherent candidates far from the action distribution seen during training\. Searching in the autoencoder latent space instead provides temporal structure: each latent candidate decodes to a coherent action chunk, and CEM refines a distribution over latent codes without conditioning candidates on the current task or scene\.

Single\-task experiments use the released LeWM action\-space CEM planner and its original evaluation budget\.

### Data\.

The multi\-task corpus is the union of one clean shard and four action\-noise shards, giving29,26029\{,\}260episodes and approximately1\.441\.44M transitions across the2222configurations\. The clean shard contributes9,9009\{,\}900episodes \(450450per configuration\) collected with noise disabled\. The four noise shards contribute5,1045\{,\}104,4,7524\{,\}752,4,7524\{,\}752and4,7524\{,\}752episodes and differ only in noise scale\.

Noise is injected into the expert controller during collection rather than added to a recorded trajectory, so every noisy episode is a physically consistent rollout of a perturbed policy\. It acts at two levels\. At each replanning tick the target waypoint is displaced by𝒩⁡\(0,σp2​I3\)\\mathcal\{N\}\(0,\\sigma\_\{p\}^\{2\}I\_\{3\}\)and the target yaw by𝒩⁡\(0,σyaw2\)\\mathcal\{N\}\(0,\\sigma\_\{\\mathrm\{yaw\}\}^\{2\}\); the displacement is held fixed for the whole replan segment, so replanning cannot average it away, and the perturbed target is then clipped into the reachable workspace\. At every control step an Ornstein–Uhlenbeck process perturbs the action itself,

x←\(1−θ\)​x\+ε,ε∼𝒩⁡\(0,σstep2​s2\),σstep=σa​2​θ−θ2,x\\leftarrow\(1\-\\theta\)\\,x\+\\varepsilon,\\qquad\\varepsilon\\sim\\mathcal\{N\}\\\!\\bigl\(0,\\sigma\_\{\\mathrm\{step\}\}^\{2\}\\,s^\{2\}\\bigr\),\\qquad\\sigma\_\{\\mathrm\{step\}\}=\\sigma\_\{a\}\\sqrt\{2\\theta\-\\theta^\{2\}\},\(5\)withθ=0\.15\\theta=0\.15and per\-dimension scales=\[1,1,1,1,0\.25\]s=\[1,1,1,1,0\.25\], so the gripper channel receives a quarter of the amplitude\. The step variance is chosen so the stationary per\-dimension standard deviation equalsσa\\sigma\_\{a\}\. The sample is added to the controller’s output and the sum is clipped to\[−1,1\]\[\-1,1\]\.

Theσ\\sigmavalues carry a per\-family calibration factorkk, fitted by bisection so the expert’s failure rate on that family lands in a target band\. The four shards are the resulting tiers: the band shard targets a2020–30%30\\%failure rate, and the s50, s20 and s0 shards target4545–55%55\\%,7575–85%85\\%and at least98%98\\%, givingk=0\.0625k=0\.0625,0\.08590\.0859,0\.10940\.1094and0\.3250\.325with\(σp,σa\)\(\\sigma\_\{p\},\\sigma\_\{a\}\)of\(2\.4×10−4,6\.3×10−3\)\(2\.4\\times 10^\{\-4\},6\.3\\times 10^\{\-3\}\),\(3\.3×10−4,8\.6×10−3\)\(3\.3\\times 10^\{\-4\},8\.6\\times 10^\{\-3\}\),\(4\.2×10−4,1\.1×10−2\)\(4\.2\\times 10^\{\-4\},1\.1\\times 10^\{\-2\}\)and\(1\.2×10−3,3\.3×10−2\)\(1\.2\\times 10^\{\-3\},3\.3\\times 10^\{\-2\}\)\. Yaw waypoint noise is disabled throughout \(σyaw=0\\sigma\_\{\\mathrm\{yaw\}\}=0\)\. Noise is therefore graded by how much it degrades the expert, not by a fixed magnitude, and the corpus spans from a lightly perturbed expert to one that essentially never succeeds\.

The LeWM datasets are used as released\. Code, the corpus\-generation pipeline, and evaluation harnesses will be released\.

## Appendix FAblation Details

All ablations use the same training data, visual backbone, latent dimensionality, optimization hyperparameters, and training schedule as XP\-JEPA\. Predictor parameterization and cross\-modal prediction terms are changed only where required by the intervention being tested\.

### Cross\-only\.

Cross\-onlyretains all four source–target prediction terms from XP\-JEPA and therefore preserves corresponding cross\-prediction, but replaces the shared predictor with separate visual and physical predictors\. This isolates the effect of corresponding cross\-prediction without requiring a common transition model across modalities\.

### Share\-only\.

Share\-onlyretains the shared action\-conditioned predictor but removes the two cross\-modal prediction terms\. Each modality is therefore trained only to predict its own future representation through the common predictor\. This isolates predictor sharing without directly grounding either prediction in the corresponding future of the other modality\.

### Regress\.

Regresskeeps the visual branch and its self\-prediction term unchanged and adds a per\-frame state\-regression head\. A two\-layer MLP of width512512maps each visual latent to an8585\-dimensional target formed by six object slots, each contributing a flattened3×33\\times 3rotation matrix and a position, followed by the effector’s rotation, position, and finger opening:

ℒreg=∑dmd​\(\[rω​\(zto\)\]d−st,d\)2∑dmd,\\mathcal\{L\}\_\{\\mathrm\{reg\}\}=\\frac\{\\sum\_\{d\}m\_\{d\}\\left\(\\left\[r\_\{\\omega\}\(z\_\{t\}^\{o\}\)\\right\]\_\{d\}\-s\_\{t,d\}\\right\)^\{2\}\}\{\\sum\_\{d\}m\_\{d\}\},\(6\)whererωr\_\{\\omega\}is the regression head andmdm\_\{d\}masks the dimensions of unoccupied object slots; the effector dimensions are always included\. Targets use the same training\-set normalization as the privileged branch\. The term is added to the visual self\-prediction and isotropy losses with weight1\.01\.0\. No physical encoder or cross\-modal prediction is present, and the regression head is unused at deployment\.

### Distill\.

Distillreplaces predictive cross\-modal coupling with pointwise regression\. Each branch retains its own within\-modality prediction term and is advanced by its own predictor, each followed by its own output projection that maps the predictor’s hidden state back to the embedding dimension\. These projections belong to the prediction path only: they are applied to predictor outputs and play no part in the cross\-branch coupling\. Both within\-modality prediction terms \(m=nm=nin Equation[1](https://arxiv.org/html/2608.24044#S3.E1)\) therefore remain active, and the physical encoder is still trained, by its own prediction term together with the isotropy regularizer\. The two modalities are coupled only by aligning the raw encoder outputs, with no projection applied on either side, the visual latent being regressed onto a stop\-gradient physical latent:

‖zto−sg⁡\(zts\)‖22\.\\left\\lVert z\_\{t\}^\{o\}\-\\operatorname\{sg\}\(z\_\{t\}^\{s\}\)\\right\\rVert\_\{2\}^\{2\}\.\(7\)The physical target remains paired with its corresponding observation, so this baseline preserves sample\-level correspondence while supervising representation content rather than corresponding predictive transitions\.

### Align\-only\.

Align\-onlykeeps the shared predictor and both branches’ within\-modality prediction terms but excludes the two cross\-modal terms from the loss, and adds an explicit symmetric alignment

λalign∥zo−zs∥22,\\lambda\_\{\\mathrm\{align\}\}\\,\\bigl\\lVert z^\{o\}\-z^\{s\}\\bigr\\rVert\_\{2\}^\{2\},\(8\)withλalign=1\\lambda\_\{\\mathrm\{align\}\}=1\. The term is applied to the full encoded window rather than only to the prediction targets, and neither branch is detached\.

### Shuffle\.

Shuffleretains all four XP\-JEPA source–target terms and the shared predictor, but reads the privileged stream from a fixed partner episode:

Zts←Zts,π⁡\(e\),zt\+1s←zt\+1s,π⁡\(e\),Z\_\{t\}^\{s\}\\leftarrow Z\_\{t\}^\{s,\\pi\(e\)\},\\qquad z\_\{t\+1\}^\{s\}\\leftarrow z\_\{t\+1\}^\{s,\\pi\(e\)\},\(9\)whereeeis the current episode andπ\\piis a derangement of the training episodes, drawn once before training and held fixed\. No episode is paired with itself, and each visual trajectory sees the same incorrect partner throughout training, so the mis\-pairing is a consistent alternative correspondence rather than a fresh random pairing at every step\. Partners are drawn across the whole corpus, so a visual trajectory may be paired with a privileged trajectory from a different configuration; the privileged geometry, slot occupancy, and pair\-validity masks follow the partner, and the partner window is taken at the same relative position within the partner episode\. Because context and future are read from the same partner, each privileged sequence remains an intact trajectory that obeys the environment dynamics, and the isotropy regularizer operates on these intact trajectories as the privileged branch’s own task\. Pixels and actions stay on the true episode\. The intervention therefore preserves the physical trajectories themselves while replacing their correspondence with the visual stream by a fixed, incorrect one\.

## Appendix GQualitative Rollouts

Figures[8](https://arxiv.org/html/2608.24044#A7.F8)and[9](https://arxiv.org/html/2608.24044#A7.F9)show paired qualitative comparisons between XP\-JEPA andVisualacross all six evaluated interaction families\. For each family, both methods are executed from the same initial state using the same planner\. We show a representative scenario on which the methods disagree, with XP\-JEPA succeeding andVisualfailing\. Frames are cropped to the working volume because the manipulated object occupies only a small fraction of the uncropped camera view, making criteria such as a55cm lift difficult to inspect at print scale\. Quantitative comparisons are reported in Figure[5](https://arxiv.org/html/2608.24044#S4.F5)and Table[6](https://arxiv.org/html/2608.24044#A2.T6)\.

![Refer to caption](https://arxiv.org/html/2608.24044v1/figures/rollouts_metaworld.png)Figure 8:XP\-JEPA againstVisualon the same scenario: grasping, insertion, and placement\.Each family contributes a pair of rows executed from the same initial state using the same planner, XP\-JEPA above andVisualbelow, cropped to the working volume\. The goal is shared by both arms and is therefore shown once per comparison, spanning the family’s two rows\. Scenarios are the representative episodes in which the two arms disagree, so each pair is a case XP\-JEPA completes andVisualdoes not; per\-family success rates are in Table[6](https://arxiv.org/html/2608.24044#A2.T6)\.![Refer to caption](https://arxiv.org/html/2608.24044v1/figures/rollouts_metaworld2.png)Figure 9:XP\-JEPA againstVisualon the same scenario: pushing, stacking, and tossing\.Same protocol as Figure[8](https://arxiv.org/html/2608.24044#A7.F8): paired rows from one initial state, XP\-JEPA above andVisualbelow, with the shared goal shown once per comparison\. In stacking,Visualleaves the eraser beside the block; in tossing, it leaves the object outside the carton\.