Hamiltonian JEPA: Action-Conditioned World Models with an Inherited Control State

https://t.co/izoGpqysAw

Hamiltonian JEPA: Action-Conditioned World Models with an Inherited Control State

Source: https://arxiv.org/html/2609.33497 Ameen AliLior WolfThe Blavatnik School of Computer ScienceTel Aviv UniversityAffiliation:{tamimzoabi, ameenali023}@mail.tau.ac.il, [email protected]

Abstract

Planning from pixels needs more than a latent space that is stable and predictable. The state the planner scores must also be organized by how actions move the system. Joint-embedding predictive architectures (JEPAs) avoid pixel reconstruction by predicting future representations, but existing action-conditioned JEPAs ask one embedding to serve both perception and control. We introduce H-JEPA, which separates the two. A wide perceptual code is regularized toward a well-scaled isotropic geometry with a Bures-Wasserstein prior, and a fixed orthonormal slice of that code is the control state, which inherits the code’s covariance without any objective of its own. The state evolves under phase-conditioned dissipative port-Hamiltonian dynamics whose input port has orthonormal columns. Port-inverse consistency (PIC) reads the executed action back through the transpose of that port. We show that this readout is exactly the rollout error projected onto the port directions, so PIC is a parameter-free reweighting of prediction error and not an auxiliary action decoder. Untying the readout from the port breaks this identity and loses half of the gain. H-JEPA matches or exceeds reconstruction-free baselines, including the action-decoding Delta-JEPA, on four pixel-based control benchmarks after at most1010training epochs, and its largest gain is on OGB-Cube (91.991.9against79.379.3percent). Ablations on PushT and OGB-Cube separate the contributions of the structured predictor, PIC, the prediction horizon, the state rank, and the anti-collapse prior.

1Introduction

An agent that can predict the consequences of its actions can plan: it scores candidate action sequences in imagination and executes the best one(Ha and Schmidhuber, 2018;Hafner et al., 2019). Trained on an offline corpus of interaction data, model-predictive control on a learned world model reaches goal images never demonstrated during training, with no reward function and no test-time gradient updates(Zhou et al., 2025;Sobal et al., 2025). The plan is only as good as the model’s response to actions. A model that captures appearance but blurs the difference between action sequences ranks all plans alike, so action sensitivity is the central property of the learned dynamics.

Learning such models from pixels raises the question of what to predict. Reconstruction-based models regress future frames and can spend capacity on visual detail that is irrelevant to control(Watter et al., 2015;Hafner et al., 2019;Hafner et al., 2020). Joint-embedding predictive architectures (JEPAs) instead predict future representations(LeCun, 2022;Assran et al., 2023;Bardes et al., 2024), which removes the reconstruction target but exposes training to representation collapse. LeWM(Maes et al., 2026)addressed collapse for action-conditioned world models by attaching SIGReg(Balestriero and LeCun, 2025), a sketched Gaussian regularizer, to the embedding the planner scores.

This paper starts from the observation that stability is not the same as control. Matching a marginal embedding distribution certifies that no direction has died. It does not say which directions actions move, which coordinates should enter the planning metric, or how much capacity control requires. Three consequences follow. First, LeWM plans in the full embedding, so the metric that ranks candidates mixes control-relevant variation with dimensions that satisfy the prior without informing action selection. Second, the regularizer and the prediction loss act on the same coordinates, so their balance must be re-tuned whenever any other pressure changes. Third, latent-MSE objectives admit scale-degenerate solutions, since shrinking the targets makes prediction easy. Sub-JEPA(Zhao et al., 2026)found the full-space prior too restrictive for low-dimensional dynamics and relaxed it to random subspaces with clear gains, but its subspaces are views for the regularizer and not an extracted state.

H-JEPA (Hamiltonian JEPA) separates the perceptual code from the control state and lets the state inherit its geometry. A wide codeh∈ℝDh\in\mathbb{R}^{D}is placed on a sphere by per-sample affine-free normalization and regularized toward isotropy with a Bures-Wasserstein moment penalty(Gelbrich, 1990). A fixed orthonormal projectionUUdefines the states=U⊤​hs=U^{\top}h, whose covariance is a fixed function of the code’s covariance, so no separate state-space objective is needed. On this state we place a dissipative port-Hamiltonian step whose input port has orthonormal columns.Port-inverse consistency(PIC) reads the executed action back from the realized displacement through the transpose of the port, which is its exact pseudoinverse. The readout has no parameters and no free scale, and its residual equals the rollout error projected onto the port directions, so PIC reweights prediction error toward the directions in which actions act.

H-JEPA matches or exceeds reconstruction, full-space-prior, and action-decoding baselines on Two-Room, Reacher, PushT, and OGB-Cube under the same planning protocol, with the largest gain on OGB-Cube (91.991.9percent against79.379.3for the strongest baseline). Our contributions are as follows. (i) A two-space world model in which one distributional objective shapes perception and a fixed orthonormal map carries covariance to the planner-facing state, with an exact description of the inherited covariance (§3.1to §3.3). (ii) An analysis of anti-collapse penalties near vanishing variance, showing that the Bures-Wasserstein penalty keeps a constant restoring force on each standard deviation while SIGReg and the Euclidean covariance penalty lose theirs (§3.3). (iii) A phase-conditioned dissipative port-Hamiltonian predictor with an orthonormal input port, and PIC, which we show to be an exact error reweighting, with a control showing that the tie between readout and port accounts for half of its gain (§3.4, §3.5, §4.1). (iv) Planning, probing, and ablation results at a matched representation on two tasks, with diagnostics showing that the planner ranks candidates along directions beyond the probed physical variables (§4).

2Related work

JEPA world models.LeCun (2022)proposed prediction in representation space as the basis of world models, validated by I-JEPA and V-JEPA(Assran et al., 2023;Bardes et al., 2024). LeJEPA(Balestriero and LeCun, 2025)motivates isotropic Gaussian embeddings and enforces them with SIGReg. LeWM(Maes et al., 2026)brought this program to action-conditioned world models from pixels, and Sub-JEPA(Zhao et al., 2026)improved planning by applying the Gaussian constraint inKKrandom subspaces, withKKselected per benchmark. Delta-JEPA(Zhang et al., 2026)adds action sensitivity through a learned linear decoder on latent differences, and sensorimotor world models(Ivashkov et al., 2026)make inverse dynamics the perception objective. H-JEPA differs in three respects. The subspace is a single fixed slice thatisthe control state, the prior stays full-space at LeWM’s placement and reaches the state by inheritance, and the inverse readout is the forward model’s own input port.

Latent dynamics models for control.E2C, DVBF, World Models, PlaNet, and Dreamer learn latent dynamics through pixel reconstruction and often reward prediction(Watter et al., 2015;Karl et al., 2017;Ha and Schmidhuber, 2018;Hafner et al., 2019;Hafner et al., 2020). DINO-WM plans with a frozen pretrained encoder(Zhou et al., 2025), and PLDM trains an end-to-end pixel JEPA with a VICReg-style objective(Sobal et al., 2025). H-JEPA uses neither pixels nor rewards as targets and trains its encoder end-to-end without teachers or stop-gradients(Grill et al., 2020). Its collapse defense is a moment-matching penalty in the Barlow Twins and VICReg family(Zbontar et al., 2021;Bardes et al., 2022), with the Euclidean covariance metric replaced by Bures-Wasserstein on Gaussians(Bhatia et al., 2019).

Structured dynamics.Hamiltonian neural networks learn a scalar energy whose gradient generates conservative flow(Greydanus et al., 2019), with controlled and generalized variants(Zhong et al., 2020;Course et al., 2020;Chen et al., 2021)and image-based generative forms(Toth et al., 2020). That literature targets a specified physical state or reconstruction. H-JEPA instead adopts the port-Hamiltonian form(van der Schaft and Jeltsema, 2014)as an inductive bias for a reconstruction-free representation trained for planning, and uses port orthonormality to remove a scale freedom that would otherwise interact with the representation objective.

Inverse dynamics and state discovery.Recovering actions from transitions is a classical route to control-relevant features(Agrawal et al., 2016;Pathak et al., 2017), as are DeepMDP and bisimulation objectives(Gelada et al., 2019;Zhang et al., 2021). In exogenous-block MDPs, forward models with multi-step inverse prediction identify a control-endogenous state under stated assumptions(Lamb et al., 2023), with known limits on when multi-step inverse prediction is sufficient(Levine et al., 2024), and displacement-level action decoding appears in recent JEPA world models(Zhang et al., 2026;Ivashkov et al., 2026). PIC ties the readout to the transpose of the forward input map. Tying a readout to the transpose of an input map is standard practice for language-model embeddings(Press and Wolf, 2017), where the transpose is not the inverse of that map in general(Gu et al., 2026). In H-JEPA the port is constrained to be orthonormal, which makes the transpose its exact pseudoinverse. The bottleneck view relates to the information bottleneck(Tishby et al., 2000;Alemi et al., 2017), predictive state representations(Littman and Sutton, 2001), and balanced truncation(Moore, 1981).

3Method

Table 1:Design choices we relaxed during development, the failure each one produced in this architecture, and the constraint adopted instead. These observations motivate the constraints of §3and complement the controlled ablations of §4.1.Setting.We are given offline trajectories of RGB framesxtx_{t}and continuous actions, with no rewards and no simulator state. The goal is a latent model in which a planner can roll out candidate action sequences and rank them by distance to an encoded goal image. An encoder produces aperceptual codehth_{t}(§3.1). A fixed linear slice of the code is thecontrol statests_{t}, the only quantity the planner scores (§3.2). One distributional prior acts on the code and keeps it from collapsing (§3.3). A port-Hamiltonian predictor advances the state under actions (§3.4). Training adds port-inverse consistency (PIC), which asks the predictor’s own action-input map to recover the executed action from the observed displacement (§3.5). One model step is

ht=E⁡(xt),st=U⊤​ht,vt=st−st−1,ct=[st,vt],s^t+1=ℱθ​(st,a¯t∣ct).h_{t}=E(x_{t}),\qquad s_{t}=U^{\top}h_{t},\qquad v_{t}=s_{t}-s_{t-1},\qquad c_{t}=[s_{t},v_{t}],\qquad\hat{s}_{t+1}=\mathcal{F}_{\theta}(s_{t},\bar{a}_{t}\mid c_{t}).(1)Reading equation1from left to right,EEis the image encoder andht∈ℝDh_{t}\in\mathbb{R}^{D}is the code of the single framextx_{t}. The matrixU∈ℝD×rU\in\mathbb{R}^{D\times r}has orthonormal columns and is never trained, andst∈ℝrs_{t}\in\mathbb{R}^{r}withr≤Dr\leq Dis the control state. Sincests_{t}is computed from one frame it carries no motion information, so we form the finite-difference velocityvt∈ℝrv_{t}\in\mathbb{R}^{r}and concatenate it with the state into thephasect∈ℝ2​rc_{t}\in\mathbb{R}^{2r}(square brackets denote concatenation). The actiona¯t∈ℝda\bar{a}_{t}\in\mathbb{R}^{d_{a}}is the normalized control applied between framesttandt+1t+1. The predictorℱθ\mathcal{F}_{\theta}with parametersθ\thetaadvancessts_{t}by one step undera¯t\bar{a}_{t}. The notationℱθ(⋅∣ct)\mathcal{F}_{\theta}(\cdot\mid c_{t})means that the vector fields inside the predictor are functions of the phase, while only the state is advanced. The velocity is recomputed from consecutive states, withv0=0v_{0}=0at the start of a window, and in open-loop rollouts it is formed from predictions,v^t+i=s^t+i−s^t+i−1\hat{v}_{t+i}=\hat{s}_{t+i}-\hat{s}_{t+i-1}. An overview figure is given in AppendixA.

One rule organizes the design. The distributional objective acts only on perception, every map between the code and the planner is fixed or explicitly constrained, and no batch statistic or learned gain sits on that path. Each constraint answers a failure we observed when it was relaxed (Table1).

3.1Perception on a sphere

Following the LeWM perception stack, each frame is encoded independently as

ht=LN0​(MLP⁡(ViT⁡(xt)))∈ℝD,LN0​(z)=D​z−mean⁡(z)​𝟏‖z−mean⁡(z)​𝟏‖2,h_{t}=\mathrm{LN}_{0}\!\bigl(\mathrm{MLP}(\mathrm{ViT}(x_{t}))\bigr)\in\mathbb{R}^{D},\qquad\mathrm{LN}_{0}(z)=\sqrt{D}\,\frac{z-\operatorname{mean}(z)\mathbf{1}}{\|z-\operatorname{mean}(z)\mathbf{1}\|_{2}},with a ViT-T/14 trained from scratch(Dosovitskiy et al., 2021)and an MLP projector atD=192D=192.LN0\mathrm{LN}_{0}is LayerNorm without learned scale or bias. It centers each sample across its coordinates and fixes‖h‖2=D\|h\|_{2}=\sqrt{D}, so every code satisfiese⊤​h=0e^{\top}h=0withe=𝟏/De=\mathbf{1}/\sqrt{D}and lies on a common sphere. Fixing the radius removes a scale freedom that would let the encoder inflate or shrink transition magnitudes instead of improving geometry, so the prior of §3.3redistributes variance and cannot grow it.

Each model step spansmmenvironment steps (the frameskip), and the action is the mean commanded control over that window,a¯t=1m​∑j=1mat,j\bar{a}_{t}=\tfrac{1}{m}\sum_{j=1}^{m}a_{t,j}, in normalized coordinates. We use no learned action embedding, which co-adapted with the dynamics in our experiments (Table1).

3.2A control state with inherited covariance

The control state isst=U⊤​ht∈ℝrs_{t}=U^{\top}h_{t}\in\mathbb{R}^{r}, whereUUis drawn once from a seeded QR factorization of a Gaussian matrix. BecauseUUis fixed and linear,

Cov⁡(s)=U⊤​Cov​(h)​U,\mathrm{Cov}(s)=U^{\top}\mathrm{Cov}(h)\,U,(2)so the covariance geometry imposed on the perceptual code is inherited by the control state without introducing a separate state-space objective or normalization. We refer to this property ascovariance inheritance. At the prior’s targetCov⁡(h)=I−e​e⊤\mathrm{Cov}(h)=I-ee^{\top}(§3.3), Eq. equation2gives

Cov⁡(s)=Ir−(U⊤​e)​(U⊤​e)⊤.\mathrm{Cov}(s)=I_{r}-(U^{\top}e)(U^{\top}e)^{\top}. Thus the target covariance is isotropic up to a single exceptional direction, whose eigenvalue is1−|U⊤​e|221-|U^{\top}e|_{2}^{2}. For a random orthonormalUU,|U⊤​e|22|U^{\top}e|_{2}^{2}concentrates aroundr/Dr/D, so the deviation from isotropy is controlled directly by the relative state dimension. Whenr<Dr<D, the state is therefore nearly isotropic with one direction receiving the corresponding shrinkage. Whenr=Dr=D, the centering directioneeis contained in the span ofUU, giving one exact null direction. Trained codes need not reach the prior exactly, and AppendixGreports the resulting covariance spectra and the predicted null direction. KeepingUUfixed also prevents the projection from co-adapting with the dynamics, while experiments with a learnableUUinstead produced degenerate low-rank states (Table1).

3.3A Bures-Wasserstein isotropy prior

Letℋ={hj}j=1n\mathcal{H}=\{h_{j}\}_{j=1}^{n}be the codes of all frames in a batch, pooled over batch and time, with empirical meanμ^=1n​∑jhj\hat{\mu}=\tfrac{1}{n}\sum_{j}h_{j}and covarianceΣ^=1n​∑j(hj−μ^)​(hj−μ^)⊤\widehat{\Sigma}=\tfrac{1}{n}\sum_{j}(h_{j}-\hat{\mu})(h_{j}-\hat{\mu})^{\top}. Since every code satisfiese⊤​h=0e^{\top}h=0, the covariance always has a zero eigenvalue alongee, and the most isotropic covariance available is the projectorP=I−e​e⊤P=I-ee^{\top}onto the hyperplane orthogonal toee. We penalize the squared22-Wasserstein distanceW22W_{2}^{2}between the Gaussian with the empirical moments and the Gaussian with the target moments, which has the closed Bures-Wasserstein form(Gelbrich, 1990;Bhatia et al., 2019):

ℒiso​(ℋ)=1D​W22​(𝒩⁡(μ^,Σ^),𝒩⁡(0,P))=1D​(‖μ^‖22+∑i=1D−1(λ^i−1)2)\boxed{\ \mathcal{L}_{\mathrm{iso}}(\mathcal{H})=\frac{1}{D}\,W_{2}^{2}\bigl(\mathcal{N}(\hat{\mu},\widehat{\Sigma}),\,\mathcal{N}(0,P)\bigr)=\frac{1}{D}\Bigl(\|\hat{\mu}\|_{2}^{2}+\textstyle\sum_{i=1}^{D-1}\bigl(\sqrt{\hat{\lambda}_{i}}-1\bigr)^{2}\Bigr)\ }(3)whereλ^1,…,λ^D−1≥0\hat{\lambda}_{1},\dots,\hat{\lambda}_{D-1}\geq 0are the eigenvalues ofΣ^\widehat{\Sigma}on that hyperplane. The second equality holds becauseΣ^\widehat{\Sigma}andPPcommute. Equation3constrains only the first two moments and does not requirehhto be Gaussian. It is the model’s only distributional objective.

Why Bures-Wasserstein.Writeσi=λi\sigma_{i}=\sqrt{\lambda_{i}}for the standard deviation of the code along one covariance direction. In H-JEPA theD−rD-rcode directions outside the slice receive no gradient from prediction or PIC, so the prior alone must hold them open, and what matters is the force a penalty exerts onσi\sigma_{i}asσi→0\sigma_{i}\to 0. The three candidates behave differently:

∂∂σ​(σ−1)2=2​(σ−1)→−2⏟Bures-Wasserstein,∂∂σ​(σ2−1)2=4​σ​(σ2−1)→0⏟Euclidean​‖Σ^−I‖F2,∂∂σ​T​(σ)=O⁡(σ)→0⏟SIGReg.\underbrace{\tfrac{\partial}{\partial\sigma}(\sigma-1)^{2}=2(\sigma-1)\to-2}_{\text{Bures-Wasserstein}},\quad\underbrace{\tfrac{\partial}{\partial\sigma}(\sigma^{2}-1)^{2}=4\sigma(\sigma^{2}-1)\to 0}_{\text{Euclidean }\|\widehat{\Sigma}-I\|_{F}^{2}},\quad\underbrace{\tfrac{\partial}{\partial\sigma}T(\sigma)=O(\sigma)\to 0}_{\text{SIGReg}}.(4)Bures-Wasserstein is a quadratic penalty on standard deviations, close in spirit to the variance term of VICReg(Bardes et al., 2022), and keeps a constant pull on a dying direction. The Euclidean penalty is quadratic in the variance, so its pull fades linearly withσ\sigma. SIGReg, the sketched Epps-Pulley normality statisticTT(EPPS and PULLEY, 1983)used by LeJEPA and LeWM, is a smooth bounded function of the samples and is therefore also quadratic inσ\sigmanear a point mass (AppendixH). This matches what we saw in training. In collapse runs within H-JEPA the SIGReg score saturated as the code concentrated and its gradient became negligible, exactly where latent prediction admits a constant solution, and Table6shows the same contrast in planning success.

3.4Phase-conditioned port-Hamiltonian dynamics

Instead of an unconstrained transition map, the predictor has the form of a port-Hamiltonian system(van der Schaft and Jeltsema, 2014): autonomous motion is generated by a scalar energy and damped by a dissipation term, and the action enters through a separate input map called theport. We use this form as an inductive bias and do not assume that the latent coordinates are physical positions and momenta. For evenrr, the predictor consists of

J=[0−Ir/2Ir/20],Hθ:ℝ2​r→ℝ,Rθ​(c)=Lθ​(c)​Lθ​(c)⊤⪰0,Gθ​(c)∈ℝr×da.J=\begin{bmatrix}0&-I_{r/2}\\ I_{r/2}&0\end{bmatrix},\qquad H_{\theta}:\mathbb{R}^{2r}\rightarrow\mathbb{R},\qquad R_{\theta}(c)=L_{\theta}(c)L_{\theta}(c)^{\top}\succeq 0,\qquad G_{\theta}(c)\in\mathbb{R}^{r\times d_{a}}.(5)HereJ∈ℝr×rJ\in\mathbb{R}^{r\times r}is the constant canonical symplectic matrix, withIr/2I_{r/2}the identity of sizer/2r/2. It is skew-symmetric, so the flowJ​∇s​HJ\nabla_{s}HconservesHHin continuous time.HθH_{\theta}is a scalar energy computed by an MLP from the phasec=[s,v]c=[s,v], and∇sHθ\nabla_{s}H_{\theta}denotes its gradient with respect to the state argument only, withvvtreated as a constant input.Lθ​(c)∈ℝr×rL_{\theta}(c)\in\mathbb{R}^{r\times r}is a lower-triangular matrix output by a second MLP, so the dissipation matrixRθ​(c)R_{\theta}(c)is positive semidefinite by construction and can only remove energy.Gθ​(c)G_{\theta}(c)is the input port, which maps an action inℝda\mathbb{R}^{d_{a}}to a state displacement. It is the output of a third MLP followed by a differentiable thin QR factorization, so its columns are orthonormal,Gθ​(c)⊤​Gθ​(c)=IdaG_{\theta}(c)^{\top}G_{\theta}(c)=I_{d_{a}}(it lies on the Stiefel manifold). All three networks take the phase as input, which is how velocity information reaches the dynamics. The continuous-time fields˙=(J−Rθ​(c))​∇sHθ​(s,v)+Gθ​(c)​a¯\dot{s}=(J-R_{\theta}(c))\nabla_{s}H_{\theta}(s,v)+G_{\theta}(c)\bar{a}is discretized with one explicit Euler step of sizeΔ​t=1\Delta t=1:

s^t+1=ℱθ​(st,a¯t∣ct)=st+Δ​t​[(J−Rθ​(ct))​∇sHθ​(st,vt)⏟passive flow+Gθ​(ct)​a¯t⏟control flow].\hat{s}_{t+1}=\mathcal{F}_{\theta}\bigl(s_{t},\bar{a}_{t}\mid c_{t}\bigr)=s_{t}+\Delta t\Big[\underbrace{\bigl(J-R_{\theta}(c_{t})\bigr)\nabla_{s}H_{\theta}(s_{t},v_{t})}_{\text{passive flow}}+\underbrace{G_{\theta}(c_{t})\,\bar{a}_{t}}_{\text{control flow}}\Big].(6)Continuous-time energy properties need not hold exactly after Euler discretization (§5).

The orthonormal port fixes action geometry and scale.An unconstrained input mapB⁡(c)B(c)can be written asG⁡(c)​S​(c)G(c)S(c)with orthonormalGGand gainSS. The gain can trade off against latent scale and PIC readout, causing scale drift without improving content (Table1). Constraining the port to orthonormal columns removes this freedom. The action Jacobian∂ℱ/∂a¯=Δ​t​Gθ​(c)\partial\mathcal{F}/\partial\bar{a}=\Delta t\,G_{\theta}(c)then has all singular values equal toΔ​t\Delta t, so one unit of normalized action produces one unit of direct latent displacement. State-dependent control effectiveness is therefore expressed through the port orientation and state geometry(Murray et al., 1994).

3.5Training objective

Training uses three terms,

ℒ=ℒpred+ℒPIC+ℒiso​(ℋ),\boxed{\mathcal{L}=\mathcal{L}_{\mathrm{pred}}+\mathcal{L}_{\mathrm{PIC}}+\mathcal{L}_{\mathrm{iso}}(\mathcal{H})},(7)with no pixel reconstruction, learned inverse model, contrastive loss, teacher network, or normalization ofss.

Open-loop prediction.We supervise the same multi-step composition the planner uses. Let𝒯\mathcal{T}be the set of rollout start times in a batch. From an encoded statests_{t}, the model rolls forward forKpK_{p}steps under the executed actions, feeding back its own predictions, and matches the encoded future at every step:

ℒpred=1|𝒯|​Kp​∑t∈𝒯∑i=1Kp1r​‖δt+i‖22,δt+i=st+i−s^t+i,s^t+i=ℱθ​(s^t+i−1,a¯t+i−1∣c^t+i−1),\mathcal{L}_{\mathrm{pred}}=\frac{1}{|\mathcal{T}|K_{p}}\sum_{t\in\mathcal{T}}\sum_{i=1}^{K_{p}}\tfrac{1}{r}\left\|\delta_{t+i}\right\|_{2}^{2},\quad\delta_{t+i}=s_{t+i}-\hat{s}_{t+i},\quad\hat{s}_{t+i}=\mathcal{F}_{\theta}\bigl(\hat{s}_{t+i-1},\bar{a}_{t+i-1}\mid\hat{c}_{t+i-1}\bigr),(8)withs^t=st\hat{s}_{t}=s_{t}andc^t+i=[s^t+i,v^t+i]\hat{c}_{t+i}=[\hat{s}_{t+i},\hat{v}_{t+i}]. Squared errors are averaged over coordinates, so all three terms are of order one. ForKp=1K_{p}=1this is the next-latent regression of LeWM. We useKp=5K_{p}=5, equal to the planning horizon, so training penalizes the composed rollouts that the planner ranks.

Port-inverse consistency (PIC).The port maps an action into a state displacement. PIC asks the same port, transposed, to map the observed displacement back to the action. At rollout stepii, we subtract the passive flow from the observed displacement and read the remainder through the transpose of the port:

u^t+i=Gθ​(c^t+i)⊤​[st+i+1−s^t+iΔ​t−(J−Rθ​(c^t+i))​∇sHθ​(s^t+i,v^t+i)],\hat{u}_{t+i}=G_{\theta}(\hat{c}_{t+i})^{\top}\left[\frac{s_{t+i+1}-\hat{s}_{t+i}}{\Delta t}-\bigl(J-R_{\theta}(\hat{c}_{t+i})\bigr)\nabla_{s}H_{\theta}\bigl(\hat{s}_{t+i},\hat{v}_{t+i}\bigr)\right],(9)and penalize its distance to the executed action overKPICK_{\mathrm{PIC}}rollout steps:

ℒPIC=1|𝒯|​KPIC​∑t∈𝒯∑i=0KPIC−11da​‖u^t+i−a¯t+i‖22.\boxed{\mathcal{L}_{\mathrm{PIC}}=\frac{1}{|\mathcal{T}|K_{\mathrm{PIC}}}\sum_{t\in\mathcal{T}}\sum_{i=0}^{K_{\mathrm{PIC}}-1}\tfrac{1}{d_{a}}\left\|\hat{u}_{t+i}-\bar{a}_{t+i}\right\|_{2}^{2}}.(10)Because the port has orthonormal columns,Gθ⊤G_{\theta}^{\top}is its exact pseudoinverse, as in tied input-output maps(Press and Wolf, 2017), so the readout needs no learned inverse model and has no scale of its own. Substituting the Euler step equation6into equation9(AppendixB) gives

u^t+i−a¯t+i=1Δ​t​Gθ​(c^t+i)⊤​δt+i+1,\hat{u}_{t+i}-\bar{a}_{t+i}=\tfrac{1}{\Delta t}\,G_{\theta}(\hat{c}_{t+i})^{\top}\delta_{t+i+1},(11)whereδt+i+1\delta_{t+i+1}is the rollout error of equation8. The prediction and PIC terms therefore combine into a single quadratic form in that error,

ℒpred+ℒPIC=1|𝒯|​Kp​r​∑t∈𝒯∑i=1Kpδt+i⊤​(I+rda​Δ​t2​Gt+i−1​Gt+i−1⊤)​δt+i,\mathcal{L}_{\mathrm{pred}}+\mathcal{L}_{\mathrm{PIC}}=\frac{1}{|\mathcal{T}|K_{p}\,r}\sum_{t\in\mathcal{T}}\sum_{i=1}^{K_{p}}\delta_{t+i}^{\top}\Bigl(I+\tfrac{r}{d_{a}\,\Delta t^{2}}\,G_{t+i-1}G_{t+i-1}^{\top}\Bigr)\delta_{t+i},(12)whenKPIC=KpK_{\mathrm{PIC}}=K_{p}, withGt+i−1=Gθ​(c^t+i−1)G_{t+i-1}=G_{\theta}(\hat{c}_{t+i-1}).

Equation12tells us what PIC actually measures. PIC is not a separate task of predicting actions. It is the same rollout error asℒpred\mathcal{L}_{\mathrm{pred}}, restricted to thedad_{a}port directions through which the action enters the dynamics. Adding the two losses therefore gives a single rollout error, measured with a state-dependent metric that counts errors along the port directions1+r/(da​Δ​t2)1+r/(d_{a}\Delta t^{2})times more than errors along any other direction.

This view also explains why the model cannot satisfy PIC through a shortcut. There are two obvious shortcuts, and neither works. The first is to rescale the readout until actions become easy to decode. This fails because actions are in fixed normalized units, the port has unit singular values, and the scale ofssis inherited rather than set by the readout. The second is to rotate the port away from the directions where the rollout is wrong. This fails because the same port also injects the action in Equation equation6, so a port that no longer points at the error makes the prediction worse and raisesℒpred\mathcal{L}_{\mathrm{pred}}.

Both arguments rest on one design choice: the action is read out through the same portGθG_{\theta}that injected it. §4.1tests this directly by replacingGθ⊤G_{\theta}^{\top}in the readout with a fixed random orthonormal matrix, which breaks the link between injection and readout. We useKPIC=Kp=5K_{\mathrm{PIC}}=K_{p}=5throughout.

Planning.At test time we plan with the cross-entropy method (CEM) in a model-predictive control loop(Rubinstein, 1999). The goal image is encoded tosgoal=U⊤​E​(xgoal)s_{\mathrm{goal}}=U^{\top}E(x_{\mathrm{goal}}), candidate action sequences are rolled out with equation6forKpK_{p}steps, and each is scored by the terminal distance‖s^t+Kp−sgoal‖22\|\hat{s}_{t+K_{p}}-s_{\mathrm{goal}}\|_{2}^{2}. The phase conditions the dynamics but is not part of the cost. Main results follow the Delta-JEPA budget of500500episodes. Controlled ablations use the lower-cost LeWM budget with otherwise identical planning settings.

4Experiments

Setup.We evaluate on the four pixel-based planning benchmarks of the LeWM suite: Two-Room navigation(Maes et al., 2026), DeepMind Control Reacher(Tassa et al., 2018), PushT(Chi et al., 2025), and OGBench Cube manipulation(Park et al., 2025), hereafter OGB-Cube. All models train on the same offline image-action trajectories. We compare against PLDM(Sobal et al., 2025), LeWM(Maes et al., 2026), Sub-JEPA(Zhao et al., 2026), and Delta-JEPA(Zhang et al., 2026)under the Delta-JEPA evaluation protocol of500500test episodes per task(Zhang et al., 2026), in which Delta-JEPA is trained from scratch for5050epochs. We train H-JEPA for1010epochs on every task. One held-out pilot seed, excluded from the reported averages, is used only to choose which epoch’s checkpoint to evaluate on each task, and this choice is then fixed for the remaining seeds: epoch88on Two-Room,66on Reacher,1010on PushT, and33on OGB-Cube. H-JEPA uses AdamW(Loshchilov and Hutter, 2019), windows ofT=2+Kp=7T=2+K_{p}=7frames, andKp=KPIC=5K_{p}=K_{\mathrm{PIC}}=5. The only task-specific architectural choice is the state rank:3232on Two-Room and OGB-Cube,6464on Reacher, and192192on PushT, chosen once from the lower-budget sweep on a held-out seed in §4.1and then fixed for the500500-episode evaluation. Per-benchmark capacity selection is common in this suite: Sub-JEPA selects its number of subspaces per task (K=32K=32, exceptK=16K=16on PushT, whereK=32K=32fails)(Zhao et al., 2026). All other hyperparameters are shared (AppendixC).

Table 2:Planning success rate (%) on the four LeWM benchmarks over500500test episodes per task, mean±\pmstd over three training seeds. Baselines are evaluated under the Delta-JEPA protocol, in which Delta-JEPA trains for5050epochs. H-JEPA trains for1010epochs and is evaluated at the per-task checkpoint epoch selected on a held-out seed (§4).Planning results.Table2reports planning success. H-JEPA is best or tied on every task. It saturates Two-Room together with Delta-JEPA, is within one standard deviation of Delta-JEPA on PushT, and leads by4.84.8points on Reacher and12.712.7points on OGB-Cube. H-JEPA trains for1010epochs, against5050for Delta-JEPA. The Sub-JEPA paper reports higher numbers for itself (95.095.0,84.084.0,89.089.0, and76.376.3on the four tasks) under a different protocol with5050evaluation episodes, against500500here(Zhao et al., 2026). H-JEPA is above those numbers as well. As a check on identical episodes, a paired comparison against the official LeWM checkpoint onN=200N{=}200matched episodes gives88.588.5against82.082.0on PushT (McNemarp≈0.053p\approx 0.053). Absolute rates in this paired test are not comparable to Table2. Because Table2changes representation and predictor together, §4.1separates them.

What the state encodes.Physical probing (AppendixD) shows a consistent division of labor. H-JEPA is strongest on the directly controlled variable (PushT agent location, linear MSE0.0280.028against0.0620.062and0.0650.065), while LeWM and Sub-JEPA decode the PushT block pose better. On OGB-Cube, H-JEPA is far stronger on joint velocity and end-effector position, and the methods are at parity on Reacher. Planning quality is thus not monotone in object-pose decodability.

What the predictor learns.AppendixEaudits the learned fields on held-out trajectories. The balance between passive and control flow in equation6is task dependent. On PushT the control flow is only0.130.13times the passive flow, the passive flow is dominated by dissipation, and linear probes from the passive flow, control flow, and energy gradient each explain8888to9090percent of the physical state increment. On OGB-Cube the control flow is1.531.53times the passive flow. On Reacher and OGB-Cube the component-wise physical alignment is weaker, even though task variables remain strongly encoded inss.

4.1Ablations and analysis

Predictor class at a matched representation.The JEPA baselines use a causal AdaLN transformer over three latents. We keep the H-JEPA perception stack, the fixedUU, the statess, andKp=5K_{p}{=}5, and swap only the dynamics for that transformer (Table3). WithKp=5K_{p}{=}5, LeWM’s transformer on its native SIGReg embedding reaches2020percent. The same transformer on the H-JEPA representation reaches8484percent. Replacing it by the port-Hamiltonian predictor with PIC raises success to9292percent atr=64r{=}64while shrinking the dynamics module from3.33.3M to0.860.86M parameters. The selectedr=192r{=}192model reaches9494percent with11.611.6M parameters in total, below the18.018.0M of LeWM. The last swap changes temporal-state construction and action injection together with the predictor, so it compares complete predictor formulations at a matched representation and horizon.

Table 3:Predictor class on PushT at a matched representation and horizon (Kp=5K_{p}{=}5in all rows). Parameters are total model followed by dynamics module. H-JEPA rows share the ViT-T/14 encoder and0.790.79M projector.Table 4:Mechanism ablations, success (%) atN=300N{=}300. H is the port-Hamiltonian predictor.†Value from Table2. Table 5:Horizons on PushT, success (%).Table 6:State rank sweep, success (%).

Structured dynamics and PIC.Table6isolates the remaining choices at matched rank on PushT (r=64r{=}64, epoch1010) and OGB-Cube (r=32r{=}32, seed1111), both atN=300N{=}300. The structured predictor helps by a similar amount on both: with PIC in place, replacing the port-Hamiltonian step by an unstructured MLP costs8.38.3points on PushT (93.393.3to85.085.0) and7.67.6on OGB-Cube (91.991.9to84.384.3). The contribution of PIC is task dependent. Removing it costs18.318.3points on PushT and2.92.9on OGB-Cube, where the full-model reference is the main-benchmark value and the gap is within evaluation noise atN=300N{=}300. This agrees with the reading of PIC as a reweighting equation12and with the predictor audit. On PushT the control flow is a small part of each displacement (0.130.13of the passive flow), so an unweighted rollout loss is dominated by passive error and says little about the action-coupled part, and extra weight on the port directions matters. On OGB-Cube the control flow is the larger part (1.531.53), so the rollout loss already emphasizes it, and the extra weight from PIC is also smaller there (7.47.4against3333, sincer=32r=32andda=5d_{a}=5). Without the Hamiltonian structure, PIC matters most: the MLP falls from85.085.0to32.032.0on PushT without it.

Tying the readout to the port.To test the tie between readout and port, we keep the full recipe, including the action target and the learned portGθG_{\theta}in the forward step, and replace onlyGθ⊤G_{\theta}^{\top}in the readout equation9byQ⊤Q^{\top}for a fixed random orthonormalQ∈ℝr×daQ\in\mathbb{R}^{r\times d_{a}}. The identity equation11then fails. The residual gains a term that vanishes only when the port coincides withQQ(AppendixB), which competes with the state-dependent orientation that prediction asks of the port. This untied readout reaches83.783.7percent, against75.075.0without PIC and93.393.3with the tied readout. An action readout through a fixed orthonormal map therefore already helps, and tying it to the port, which turns it into a pure reweighting of rollout error, accounts for the other half of the gain.

Prior type and placement.Moving the Bures-Wasserstein prior from the codehhonto the scored statesslowers PushT success from93.393.3to86.386.3, which supports regularizing perception and letting the state inherit. Replacing Bures-Wasserstein by SIGReg onhhreaches36.736.7percent at the same epoch1010, consistent with equation4. This is a statement about SIGReg inside H-JEPA, where the prior alone must hold open the code directions outside the slice. It is not a statement about LeWM, where SIGReg acts on the same coordinates as the prediction loss.

Training at the planning horizon.Table6(PushT,r=192r{=}192,N=50N{=}50, epoch1010) shows that one-step prediction does not compose into a plannable rollout, with or without one-step PIC (1212and3434percent). Matching both horizons at three or five steps reaches9494percent. Cutting PIC to one step while keepingKp=5K_{p}{=}5gives9090percent. We therefore setKp=KPICK_{p}=K_{\mathrm{PIC}}to the planning horizon.

Rank is the capacity knob.Table6shows that the useful control dimension is task dependent (bold: rank used in Table2, parentheses: larger evaluation). Two-Room saturates at every tested rank, Reacher and OGB-Cube degrade at full rank, and PushT improves up tor=Dr{=}D. Atr=Dr=Dthe slice compresses nothing but still separates the scored coordinates from the regularized ones. The selected ranks are capacity choices for the planning metric and not estimates of physical state dimension. An audit of the CEM cost on ther=64r{=}64PushT checkpoint makes this concrete. On the candidate sets scored during evaluation, we fit a linear map from the probed physical variables of AppendixDto the terminal state, split every terminal state into that reconstruction and the orthogonal remainder, and rank candidates by the terminal cost computed from each part alone. The remainder alone has Spearman correlation0.9970.997with the full candidate ranking and recovers8989percent of the CEM elites, against0.660.66and3838percent for the physical part. Planning therefore uses predictive directions beyond the variables exposed by standard physical probes.

AppendixFreports the history ablation that led to the parameter-free phase, and AppendixIthe training curves.

5Limitations

H-JEPA uses mechanics as a structural bias and makes no claim of latent system identification. The learned coordinates are free to rotate around the planning objective, and the audit in AppendixEconfirms that component-level physical alignment is task dependent. The explicit Euler step does not preserve the energy properties of the continuous-time field, which our short-horizon planner did not require. The orthonormal port fixes the singular values of the instantaneous action map. This removes a scale ambiguity and makes PIC well conditioned, but it gives up state-dependent action gain. The state rank is selected per task and not learned, and atr=Dr=Dthe state has one null direction (§3.2). The inherited covariance is only as isotropic as the trained code, and the measured spectra in AppendixGremain spread around one. Finally, our evidence is limited to offline pixel control in simulation with short-horizon CEM planning. Longer horizons, real observations, and online data collection remain to be tested.

6Conclusion

We introduced H-JEPA, a world model that learns to plan from pixels without reconstruction by separating perceptual representation learning from the state used for control. A Bures-Wasserstein prior regularizes the perceptual code toward isotropy, while a fixed orthonormal projection transfers its covariance geometry to a control state with task-dependent dimension. A phase-conditioned port-Hamiltonian predictor structures the latent dynamics, and tied port-inverse consistency yields an exact reweighting of rollout error along action-input directions without an additional decoder. Ablations support the benefits of structured dynamics and, on PushT, regularizing perception and tying the inverse readout to the action port. H-JEPA matches or exceeds all evaluated baselines across four pixel-based control benchmarks. Together, these results establish covariance inheritance and structured, action-sensitive prediction as a promising combination for learning representations that support effective planning.

Ethics statement

This work studies representation learning and planning in standard simulated control benchmarks. It involves no human subjects, no personal data, and no dataset releases beyond configurations for public benchmarks. Although the present experiments are confined to simulation, deploying learned world models on physical systems would require application-specific validation of safety, robustness, and failure behavior.

Reproducibility statement

Section3specifies the model, training objective, and planning procedure, with equation3, equation6, equation8, and equation10defining the distributional objective, dynamics update, multi-step prediction loss, and PIC loss. AppendixBderives the PIC identity. Default hyperparameters and the per-task checkpoint epochs are listed in AppendixC. The probing protocol followsZhang et al. (2026)andMaes et al. (2026)and is restated in AppendixD. Predictor diagnostics are in AppendixE, the geometry and collapse analyses in AppendicesGandH, and optimization traces in AppendixI. An anonymized code release with training configurations for all reported runs is included as supplementary material.

Acknowlegments

This work was supported by a Tel Aviv University Center for AI and Data Science (TAD) grant and by Len Blavatnik and the Blavatnik Family foundation. This research was also supported by the Ministry of Innovation, Science & Technology, Israel (1001576154) and the Michael J. Fox Foundation (MJFF-022407). The contribution of Tamim Zoabi is part of a PhD thesis research conducted at Tel Aviv University.

References

• Agrawalet al.(2016)P. Agrawal, A. V. Nair, P. Abbeel, J. Malik, and S. LevineLearning to poke by poking: experiential learning of intuitive physics.InAdvances in Neural Information Processing Systems,D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (Eds.),Vol.29,pp..External Links:LinkCited by:§2.
• Alemiet al.(2017)A. A. Alemi, I. Fischer, J. V. Dillon, and K. MurphyDeep variational information bottleneck.InInternational Conference on Learning Representations,External Links:LinkCited by:§2.
• Assranet al.(2023)M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, and N. BallasSelf-Supervised Learning from Images with a Joint-Embedding Predictive Architecture.In2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),Vol.,Los Alamitos, CA, USA,pp. 15619–15629.External Links:ISSN,Document,LinkCited by:§1,§2.
• Balestriero and LeCun (2025)R. Balestriero and Y. LeCunLeJEPA: provable and scalable self-supervised learning without the heuristics.External Links:2511.08544,LinkCited by:§1,§2.
• Bardeset al.(2024)A. Bardes, Q. Garrido, J. Ponce, X. Chen, M. Rabbat, Y. LeCun, M. Assran, and N. BallasRevisiting feature prediction for learning visual representations from video.Transactions on Machine Learning Research.Note:Featured CertificationExternal Links:ISSN 2835-8856,LinkCited by:§1,§2.
• Bardeset al.(2022)A. Bardes, J. Ponce, and Y. LeCunVICReg: variance-invariance-covariance regularization for self-supervised learning.InInternational Conference on Learning Representations,External Links:LinkCited by:§2,§3.3.
• Bhatiaet al.(2019)R. Bhatia, T. Jain, and Y. LimOn the bures–wasserstein distance between positive definite matrices.Expositiones Mathematicae37(2),pp. 165–191.External Links:ISSN 0723-0869,Document,LinkCited by:§2,§3.3.
• Chenet al.(2021)Y. Chen, T. Matsubara, and T. YaguchiNeural symplectic form: learning hamiltonian equations on general coordinate systems.InAdvances in Neural Information Processing Systems,M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. W. Vaughan (Eds.),Vol.34,pp. 16659–16670.External Links:LinkCited by:§2.
• Chiet al.(2025)C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. SongDiffusion policy: visuomotor policy learning via action diffusion.The International Journal of Robotics Research44(10-11),pp. 1684–1704.External Links:Document,Link,https://doi.org/10.1177/02783649241273668Cited by:§4.
• Courseet al.(2020)K. Course, T. Evans, and P. NairWeak form generalized hamiltonian learning.InAdvances in Neural Information Processing Systems,H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.),Vol.33,pp. 18716–18726.External Links:LinkCited by:§2.
• Dosovitskiyet al.(2021)A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. HoulsbyAn image is worth 16x16 words: transformers for image recognition at scale.InInternational Conference on Learning Representations,External Links:LinkCited by:§3.1.
• EPPS and PULLEY (1983)T. W. EPPS and L. B. PULLEYA test for normality based on the empirical characteristic function.Biometrika70(3),pp. 723–726.External Links:ISSN 0006-3444,Document,Link,https://academic.oup.com/biomet/article-pdf/70/3/723/687464/70-3-723.pdfCited by:Appendix H,§3.3.
• Geladaet al.(2019)C. Gelada, S. Kumar, J. Buckman, O. Nachum, and M. G. BellemareDeepMDP: learning continuous latent space models for representation learning.InProceedings of the 36th International Conference on Machine Learning,K. Chaudhuri and R. Salakhutdinov (Eds.),Proceedings of Machine Learning Research, Vol.97,pp. 2170–2179.External Links:LinkCited by:§2.
• Gelbrich (1990)M. GelbrichOn a formula for the l2 wasserstein metric between measures on euclidean and hilbert spaces.Mathematische Nachrichten147(1),pp. 185–203.External Links:Document,Link,https://onlinelibrary.wiley.com/doi/pdf/10.1002/mana.19901470121Cited by:§1,§3.3.
• Greydanuset al.(2019)S. Greydanus, M. Dzamba, and J. YosinskiHamiltonian neural networks.InAdvances in Neural Information Processing Systems,H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, and R. Garnett (Eds.),Vol.32,pp..External Links:LinkCited by:§2.
• Grillet al.(2020)J. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, B. Piot, k. kavukcuoglu, R. Munos, and M. ValkoBootstrap your own latent - a new approach to self-supervised learning.InAdvances in Neural Information Processing Systems,H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.),Vol.33,pp. 21271–21284.External Links:LinkCited by:§2.
• Guet al.(2026)J. Gu, A. Aleti, C. Chen, and H. ZhangRethinking weight tying: pseudo-inverse tying for lm stable training and updates.External Links:2602.04556,LinkCited by:§2.
• Ha and Schmidhuber (2018)D. Ha and J. SchmidhuberRecurrent world models facilitate policy evolution.InAdvances in Neural Information Processing Systems,S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.),Vol.31,pp..External Links:LinkCited by:§1,§2.
• Hafneret al.(2020)D. Hafner, T. Lillicrap, J. Ba, and M. NorouziDream to control: learning behaviors by latent imagination.External Links:1912.01603,LinkCited by:§1,§2.
• Hafneret al.(2019)D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. DavidsonLearning latent dynamics for planning from pixels.InProceedings of the 36th International Conference on Machine Learning,K. Chaudhuri and R. Salakhutdinov (Eds.),Proceedings of Machine Learning Research, Vol.97,pp. 2555–2565.External Links:LinkCited by:§1,§1,§2.
• Ivashkovet al.(2026)P. Ivashkov, R. Balestriero, and B. SchölkopfSensorimotor world models: perception for action via inverse dynamics.External Links:2606.20104,LinkCited by:§2,§2.
• Karlet al.(2017)M. Karl, M. Soelch, J. Bayer, and P. van der SmagtDeep variational bayes filters: unsupervised learning of state space models from raw data.InInternational Conference on Learning Representations,External Links:LinkCited by:§2.
• Lambet al.(2023)A. Lamb, R. Islam, Y. Efroni, A. R. Didolkar, D. Misra, D. J. Foster, L. P. Molu, R. Chari, A. Krishnamurthy, and J. LangfordGuaranteed discovery of control-endogenous latent states with multi-step inverse models.Transactions on Machine Learning Research.Note:External Links:ISSN 2835-8856,LinkCited by:§2.
• LeCun (2022)Y. LeCunA path towards autonomous machine intelligence.OpenReview preprint.Cited by:§1,§2.
• Levineet al.(2024)A. Levine, P. Stone, and A. ZhangMultistep inverse is not all you need.InReinforcement Learning Conference,External Links:LinkCited by:§2.
• Littman and Sutton (2001)M. Littman and R. S. SuttonPredictive representations of state.InAdvances in Neural Information Processing Systems,T. Dietterich, S. Becker, and Z. Ghahramani (Eds.),Vol.14,pp..External Links:LinkCited by:§2.
• Loshchilov and Hutter (2019)I. Loshchilov and F. HutterDecoupled weight decay regularization.InInternational Conference on Learning Representations,External Links:LinkCited by:§4.
• Maeset al.(2026)L. Maes, Q. L. Lidec, D. Scieur, Y. LeCun, and R. BalestrieroLeWorldModel: stable end-to-end joint-embedding predictive architecture from pixels.External Links:2603.19312,LinkCited by:Appendix D,§1,§2,§4,§6.
• Marčenko and Pastur (1967)V. A. Marčenko and L. A. PasturDISTRIBUTION of eigenvalues for some sets of random matrices.Mathematics of the USSR-Sbornik1(4),pp. 457.External Links:Document,LinkCited by:Table 13.
• Mishkin and Matas (2016)D. Mishkin and J. MatasAll you need is a good init.External Links:1511.06422,LinkCited by:Table 1.
• Moore (1981)B. MoorePrincipal component analysis in linear systems: controllability, observability, and model reduction.IEEE Transactions on Automatic Control26(1),pp. 17–32.External Links:DocumentCited by:Appendix B,§2.
• Murrayet al.(1994)R. M. Murray, Z. Li, and S. S. SastryA mathematical introduction to robotic manipulation.CRC Press.Cited by:§3.4.
• Parket al.(2025)S. Park, K. Frans, B. Eysenbach, and S. LevineOGBench: benchmarking offline goal-conditioned rl.InInternational Conference on Learning Representations,Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (Eds.),Vol.2025,pp. 94937–94982.External Links:LinkCited by:§4.
• Pathaket al.(2017)D. Pathak, P. Agrawal, A. A. Efros, and T. DarrellCuriosity-driven exploration by self-supervised prediction.InProceedings of the 34th International Conference on Machine Learning,D. Precup and Y. W. Teh (Eds.),Proceedings of Machine Learning Research, Vol.70,pp. 2778–2787.External Links:LinkCited by:§2.
• Press and Wolf (2017)O. Press and L. WolfUsing the output embedding to improve language models.InProceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers,M. Lapata, P. Blunsom, and A. Koller (Eds.),Valencia, Spain,pp. 157–163.External Links:LinkCited by:§2,§3.5.
• Rubinstein (1999)R. RubinsteinThe cross-entropy method for combinatorial and continuous optimization.Methodology and Computing in Applied Probability1(2),pp. 127–190.External Links:Document,LinkCited by:§3.5.
• Sobalet al.(2025)U. Sobal, W. Zhang, K. Cho, R. Balestriero, T. G. J. Rudner, and Y. LeCunLearning from reward-free offline data: a case for planning with latent dynamics models.InAdvances in Neural Information Processing Systems,D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (Eds.),Vol.38, Main Conference,pp. 43905–43941.External Links:Document,LinkCited by:§1,§2,§4.
• Tassaet al.(2018)Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. de Las Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, T. Lillicrap, and M. RiedmillerDeepMind control suite.External Links:1801.00690,LinkCited by:§4.
• Tishbyet al.(2000)N. Tishby, F. C. Pereira, and W. BialekThe information bottleneck method.External Links:physics/0004057,LinkCited by:§2.
• Tothet al.(2020)P. Toth, D. J. Rezende, A. Jaegle, S. Racanière, A. Botev, and I. HigginsHamiltonian generative networks.InInternational Conference on Learning Representations,External Links:LinkCited by:§2.
• van der Schaft and Jeltsema (2014)A. van der Schaft and D. JeltsemaPort-hamiltonian systems theory: an introductory overview.Foundations and Trends in Systems and Control1(2-3),pp. 173–378.External Links:ISSN 2325-6818,Document,Link,https://www.emerald.com/ftsys/article-pdf/1/2-3/173/11146089/2600000002en.pdfCited by:§2,§3.4.
• Watteret al.(2015)M. Watter, J. Springenberg, J. Boedecker, and M. RiedmillerEmbed to control: a locally linear latent dynamics model for control from raw images.InAdvances in Neural Information Processing Systems,C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.),Vol.28,pp..External Links:LinkCited by:§1,§2.
• Zbontaret al.(2021)J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. DenyBarlow twins: self-supervised learning via redundancy reduction.InProceedings of the 38th International Conference on Machine Learning,M. Meila and T. Zhang (Eds.),Proceedings of Machine Learning Research, Vol.139,pp. 12310–12320.External Links:LinkCited by:§2.
• Zhanget al.(2021)A. Zhang, R. T. McAllister, R. Calandra, Y. Gal, and S. LevineLearning invariant representations for reinforcement learning without reconstruction.InInternational Conference on Learning Representations,External Links:LinkCited by:§2.
• Zhanget al.(2026)Z. Zhang, Y. Wang, Z. Guan, Y. Yang, B. Shi, T. Zong, H. Yi, G. Chao, X. Chen, T. Yang, C. Bao, T. Yu, J. Zhou, and J. XuDelta-jepa: learning action-sensitive world models via latent difference decoding.External Links:2606.31232,LinkCited by:Table 7,Appendix C,Appendix D,Appendix I,§2,§2,§4,§6.
• Zhaoet al.(2026)K. Zhao, D. Nie, Y. Lin, Z. Luo, Y. Gu, D. Fan, and D. ZengSub-jepa: subspace gaussian regularization for stable end-to-end world models.External Links:2605.09241,LinkCited by:§1,§2,§4,§4.
• Zhonget al.(2020)Y. D. Zhong, B. Dey, and A. ChakrabortySymplectic ode-net: learning hamiltonian dynamics with control.InInternational Conference on Learning Representations,External Links:LinkCited by:§2.
• Zhouet al.(2025)G. Zhou, H. Pan, Y. LeCun, and L. PintoDINO-WM: world models on pre-trained visual features enable zero-shot planning.InForty-second International Conference on Machine Learning,External Links:LinkCited by:§1,§2.

Appendix AArchitecture overview

Figure1summarizes the training-time model: perception on a sphere, the fixed orthonormal slice, the port-Hamiltonian predictor, and the three losses.

Figure 1:H-JEPA.The Bures-Wasserstein priorℒiso\mathcal{L}_{\mathrm{iso}}acts only on the spherical codehth_{t}. A fixed orthonormal sliceU⊤U^{\top}yields the control statests_{t}. The phase-conditioned port-Hamiltonian predictorℱθ\mathcal{F}_{\theta}is supervised open-loop (ℒpred\mathcal{L}_{\mathrm{pred}}), PIC reads actions back through the port transpose (ℒPIC\mathcal{L}_{\mathrm{PIC}}), and planning scoresssalone.

Appendix BDerivation of the PIC identity

Fix a rollout stepiiand abbreviatec=c^t+ic=\hat{c}_{t+i},s=s^t+is=\hat{s}_{t+i},v=v^t+iv=\hat{v}_{t+i}, anda¯=a¯t+i\bar{a}=\bar{a}_{t+i}. The model’s one-step prediction from the rollout state is

s^t+i+1=s+Δ​t​[(J−Rθ​(c))​∇sHθ​(s,v)+Gθ​(c)​a¯].\hat{s}_{t+i+1}=s+\Delta t\Bigl[\bigl(J-R_{\theta}(c)\bigr)\nabla_{s}H_{\theta}(s,v)+G_{\theta}(c)\,\bar{a}\Bigr].(13)Substituting the readout of equation9and usingGθ​(c)⊤​Gθ​(c)=IG_{\theta}(c)^{\top}G_{\theta}(c)=I,

u^t+i−a¯\displaystyle\hat{u}_{t+i}-\bar{a}=Gθ​(c)⊤​[st+i+1−sΔ​t−(J−Rθ​(c))​∇sHθ​(s,v)]−Gθ​(c)⊤​Gθ​(c)​a¯\displaystyle=G_{\theta}(c)^{\top}\Bigl[\tfrac{s_{t+i+1}-s}{\Delta t}-\bigl(J-R_{\theta}(c)\bigr)\nabla_{s}H_{\theta}(s,v)\Bigr]-G_{\theta}(c)^{\top}G_{\theta}(c)\,\bar{a}(14)=1Δ​t​Gθ​(c)⊤​[st+i+1−s−Δ​t​(J−Rθ​(c))​∇sHθ​(s,v)−Δ​t​Gθ​(c)​a¯]\displaystyle=\tfrac{1}{\Delta t}\,G_{\theta}(c)^{\top}\Bigl[s_{t+i+1}-s-\Delta t\bigl(J-R_{\theta}(c)\bigr)\nabla_{s}H_{\theta}(s,v)-\Delta t\,G_{\theta}(c)\,\bar{a}\Bigr](15)=1Δ​t​Gθ​(c)⊤​(st+i+1−s^t+i+1)=1Δ​t​Gθ​(c)⊤​δt+i+1,\displaystyle=\tfrac{1}{\Delta t}\,G_{\theta}(c)^{\top}\bigl(s_{t+i+1}-\hat{s}_{t+i+1}\bigr)=\tfrac{1}{\Delta t}\,G_{\theta}(c)^{\top}\delta_{t+i+1},(16)which is equation11. The PIC residual is the projection of the open-loop state error onto the port subspace, scaled by1/Δ​t1/\Delta t. Summing1r​‖δ‖22\tfrac{1}{r}\|\delta\|_{2}^{2}and1da​Δ​t2​‖G⊤​δ‖22\tfrac{1}{d_{a}\Delta t^{2}}\|G^{\top}\delta\|_{2}^{2}over the same steps gives the quadratic form in equation12, which weights error along thedad_{a}port directions by a factor1+r/(da​Δ​t2)1+r/(d_{a}\Delta t^{2})relative to the remainingr−dar-d_{a}directions. WithΔ​t=1\Delta t=1this factor is1717on Two-Room,3333on Reacher,9797on PushT atr=192r=192, and7.47.4on OGB-Cube.

The identity does not remove the action readout. On a transition that the model predicts exactly, the transpose map returnsu^=a¯\hat{u}=\bar{a}. Off such transitions the readout is an orthogonal projection of the residual displacement, so it never amplifies noise, its magnitude is bounded by the transition energy, andℒPIC\mathcal{L}_{\mathrm{PIC}}is of order one at initialization in normalized units. The identity also explains the control in §4.1, where the readout uses a fixed random orthonormalQ⊤Q^{\top}in place ofGθ​(c)⊤G_{\theta}(c)^{\top}while the forward step keepsGθG_{\theta}. The same substitution now gives

u^t+i−a¯=1Δ​t​Q⊤​δt+i+1+(Q⊤​Gθ​(c)−I)​a¯.\hat{u}_{t+i}-\bar{a}=\tfrac{1}{\Delta t}\,Q^{\top}\delta_{t+i+1}+\bigl(Q^{\top}G_{\theta}(c)-I\bigr)\bar{a}.(17)The first term is again a projected rollout error. The second is new and vanishes only ifGθ​(c)=QG_{\theta}(c)=Qfor every visited phase, so the untied loss pulls the port toward a fixed frame whileℒpred\mathcal{L}_{\mathrm{pred}}asks it to follow the state. With the tied readout this term is identically zero and the port is left free. The control recovers about half of the gain (75.075.0to83.783.7, against93.393.3with the tied readout), and the tie supplies the rest. Weighting error along the directions in which actions enter is qualitatively related to the controllability side of balanced realizations(Moore, 1981), with predictive accuracy playing the analogous role for observability.

Appendix CDefault configuration and training details

Table 7:Default H-JEPA configuration. Task-specific entries are shown explicitly for rank, action dimension, and checkpoint epoch.Table7lists the defaults. The three losses are summed without weights, which places PIC and prediction on comparable scales in normalized action and state units. For context,Zhang et al. (2026)sweep the weight of their learned action decoder in the same action units and report collapse below0.10.1with a broad workable range from11to5050. The two losses are not equivalent, so we treat this as context and not as hyperparameter transfer. The readout itself is two matrix products plus one thin QR factorization per step, with no linear solve and no regularizer, run under bf16 autocast with the factorization in a small fp32 island, as is standard for spectral operations.

Appendix DFull physical probing tables

The protocol followsZhang et al. (2026)andMaes et al. (2026):20,00020{,}000frames, trajectory-level train and test split, targetszz-scored on the train split, an OLS linear probe and a2×2562{\times}256ReLU MLP probe trained for100100epochs, and three probe seeds governing MLP initialization and minibatch order (the OLS fit is deterministic given the split), reported as mean±\pmstd. H-JEPA probes the control states=U⊤​hs=U^{\top}hat the task-specific converged checkpoints defined in §4(PushT seed1111atr=192r{=}192, Reacher seed30723072atr=64r{=}64, OGB-Cube seed30723072atr=32r{=}32), and LeWM and Sub-JEPA use their official projected192192-d embeddings. Angular and quaternion targets are evaluated in raw coordinates, so wrap-around and sign ambiguity depress correlations for all methods alike, and those rows should be read comparatively. Tables8,9, and10report PushT, Reacher, and OGB-Cube in turn.

Table 8:Physical latent probing on PushT (ρ\rhois the Pearson correlation).Table 9:Physical latent probing on Reacher (ρ\rhois the Pearson correlation).Table 10:Physical latent probing on OGB-Cube (ρ\rhois the Pearson correlation).

Appendix EInterpreting the port-Hamiltonian predictor

The port-Hamiltonian parameterization is introduced as an inductive bias rather than an assumption that the learned coordinates recover physical positions, momenta, or energies. We therefore ask a more limited question:*after training for planning, does the predictor use its structured components in a meaningful and task-dependent way?*We audit the best planning checkpoint for each benchmark on24,57624{,}576held-out expert transitions, evaluating the learned fields only on encoded states encountered in the data.

The imposed structure is realized numerically.For each transition we decompose the predicted velocity into

s˙=(J−R⁡(c))​∇sH​(s,v)⏟passive flow+G⁡(c)​a¯⏟control flow.\dot{s}=\underbrace{\bigl(J-R(c)\bigr)\nabla_{s}H(s,v)}_{\text{passive flow}}+\underbrace{G(c)\bar{a}}_{\text{control flow}}.(18)The Stiefel constraint is satisfied to numerical precision on all three tasks,‖G⊤​G−I‖∞≈4.8×10−7\|G^{\top}G-I\|_{\infty}\approx 4.8\times 10^{-7}, whileR⁡(c)=L⁡(c)​L​(c)⊤R(c)=L(c)L(c)^{\top}remains positive semidefinite by construction. Thus, the analysis below probes the learned fields within the intended decomposition and not artifacts of constraint violation.

The balance of passive and controlled dynamics is task dependent.Table11shows that the predictor does not use the same decomposition across environments. PushT is strongly dominated by passive, dissipative flow:∥R∇H∥/∥J∇H∥=4.35\|R\nabla H\|/\|J\nabla H\|=4.35, while the direct control flow is only0.130.13times the passive flow on expert trajectories. Reacher is more balanced, and OGB-Cube becomes control dominated, with‖G​a¯‖/‖passive‖=1.53\|G\bar{a}\|/\|\mathrm{passive}\|=1.53. The overlaptr⁡(G⊤​R​G)/tr⁡(R)\operatorname{tr}(G^{\top}RG)/\operatorname{tr}(R)is nearly zero on PushT but substantially larger on OGB-Cube, indicating that the learned dissipative and actuated subspaces themselves adapt to the task.

Table 11:Learned port-Hamiltonian structure on held-out transitions.The relative contribution of dissipation, passive dynamics, and direct control changes markedly across tasks.PushT provides the clearest physical interpretation. Its learned dissipation increases with latent speed (corr⁡(tr⁡R,‖v‖)=0.48\operatorname{corr}(\operatorname{tr}R,\|v\|)=0.48), and the dissipative component dominates the conservative component, consistent with a representation organized around the strongly dissipative contact dynamics of pushing. OGB-Cube exhibits the opposite regime: direct actuation is larger than the passive field and actuation overlaps more strongly with the learned dissipative subspace. We emphasize that these quantities are measured in learned latent coordinates and should be interpreted as structural signatures, not estimates of physical coefficients in SI units.

When the task permits it, the learned fields align with physical change.To test whether the decomposition contains interpretable information beyond its relative magnitudes, we fit linear ridge probes from each field to the corresponding physical state incrementΔ​y\Delta y. Results are shown in Table12. On PushT, the passive field, control field, and Hamiltonian gradient each explain approximately8888to9090percent of the variance in the physical increment. The control state itself predicts the physical state withR2=0.95R^{2}=0.95. Thus, on this task the latent decomposition is not only structurally valid but closely aligned with observable physical evolution.

Table 12:Linear physical alignment of the learned predictor.R2R^{2}is measured on held-out transitions using ridge regression.The same factor-wise interpretation is weaker on Reacher and OGB-Cube. This does not imply that the learned state is non-physical: complementary probes recover finger position in Reacher with correlation0.9990.999, and end-effector and block position in OGB-Cube with correlations0.9970.997and0.9890.989, respectively. Rather, these tasks appear to represent the variables useful for planning more holistically inss, without requiring each of theHH,RR, andGGcomponents to be linearly aligned with a privileged physical coordinate.

Interpretation.These results suggest two levels at which the structured predictor is useful. First, the port-Hamiltonian parameterization provides a calibrated decomposition into passive, dissipative, and directly controlled change whose relative use adapts across tasks. Second, when the underlying task admits a simple physical factorization, as in PushT, the learned fields themselves become strongly interpretable in terms of physical evolution. Successful planning does not require this stronger property: on Reacher and OGB-Cube, control-relevant physical information remains clearly present in the planning state even when it is distributed across the nonlinear predictor. We therefore view the port-Hamiltonian structure primarily as an inductive bias that organizes latent dynamics, with physical interpretability emerging where supported by the task rather than being imposed by construction.

Appendix FHistory mechanism ablation

An earlier variant used a three-frame history fused by a learned mixer with LayerNorm,c=LN⁡(Whist​[s,Δ1,Δ2])c=\mathrm{LN}(W_{\mathrm{hist}}[s,\Delta_{1},\Delta_{2}]). After training, about9090percent of the mixer’s Frobenius energy concentrated on the latest state, and zeroing the difference channels barely changed one-step predictions (cosine similarity about0.990.99), so the mixer was learning to ignore the history it was built to fuse. The current parameter-free phasec=[s,v]c=[s,v]with a two-frame window removes the mixer and its normalization entirely, keepsssa single-frame representation, and leaves the planner’s metric untouched, and it is the variant used in all selected recipes.

Appendix GGeometry of the inherited control state

This appendix makes the inheritance statement of §3.2exact and compares it with trained models.

Target covariance of the state.Lete=𝟏/De=\mathbf{1}/\sqrt{D}.LN0\mathrm{LN}_{0}givese⊤​h=0e^{\top}h=0for every sample, soCov⁡(h)​e=0\mathrm{Cov}(h)\,e=0for any encoder, and the target of equation3isP=I−e​e⊤P=I-ee^{\top}. The exact covariance of a uniform distribution on the radius-D\sqrt{D}sphere inside the hyperplane isDD−1​P\tfrac{D}{D-1}P, which differs fromPPby0.50.5percent atD=192D=192, and we use the unit target as a fixed convention. IfCov⁡(h)=P\mathrm{Cov}(h)=P, then by equation2

Cov⁡(s)=U⊤​(I−e​e⊤)​U=Ir−w​w⊤,w=U⊤​e.\mathrm{Cov}(s)=U^{\top}(I-ee^{\top})U=I_{r}-ww^{\top},\qquad w=U^{\top}e.(19)This matrix has eigenvalue11with multiplicityr−1r-1and one exceptional eigenvalue1−‖w‖221-\|w\|_{2}^{2}alongww. ForUUdrawn uniformly among orthonormal frames,‖w‖22\|w\|_{2}^{2}follows aBeta⁡(r2,D−r2)\mathrm{Beta}\bigl(\tfrac{r}{2},\tfrac{D-r}{2}\bigr)distribution with meanr/Dr/D. Whenr=Dr=D,UUis an orthogonal matrix,‖w‖2=1\|w\|_{2}=1, and the exceptional eigenvalue is exactly zero: the state then has one null direction, the image of the centering direction.

Table 13:Measured geometry of the control state onn=512n=512encoded frames from the selected checkpoints. The Marchenko-Pastur (MP) interval is the range of sample eigenvalues that an exactly isotropic state would show at thisnnandrr(Marčenko and Pastur, 1967).(a)PushT,r=64r=64(b)PushT,r=192r=192(c)OGB-Cube,r=32r=32Figure 2:Eigenvalues ofCov⁡(s)\mathrm{Cov}(s)for the selected checkpoints.The dashed line marks the exceptional eigenvalue1−‖U⊤​e‖221-\|U^{\top}e\|_{2}^{2}of the target covariance and the dotted line marks one. Atr=D=192r=D=192the predicted null direction appears as the last eigenvalue.What trained models show.Table13and Figure2support three statements. First, the measured‖U⊤​e‖22\|U^{\top}e\|_{2}^{2}is close tor/Dr/Dfor each seeded projection, as expected. Second, the structural prediction atr=Dr=Dis confirmed: the smallest eigenvalue ofCov⁡(s)\mathrm{Cov}(s)is1.3×10−61.3\times 10^{-6}, so the full-rank PushT state lives in a hyperplane of dimensionD−1D-1. Third, forr<Dr<Devery state direction retains nonzero variance, but the spectrum is spread around one and is not flat, and the smallest eigenvalue on OGB-Cube is well below the Marchenko-Pastur range. Part of this spread is a finite-sample effect, as the MP intervals show, and part is residual anisotropy of the code, whose own covariance eigenvalues reach about2.752.75in these checkpoints. The smallest measured eigenvalue lies below the exceptional value of the target in bothr<Dr<Dmodels, so the tail of the spectrum is set by this residual anisotropy and not by the centering direction. The tail is lowest on OGB-Cube, whose selected checkpoint is from epoch33. We therefore describe the state as inheriting a full-rank, well-scaled covariance from the code, which is exactly isotropic only at the prior’s optimum.

Appendix HAnti-collapse penalties near vanishing variance

Consider one covariance direction with varianceλ\lambdaand standard deviationσ=λ\sigma=\sqrt{\lambda}, and write the samples along it aszj=σ​xjz_{j}=\sigma x_{j}with standardizedxjx_{j}. For any penaltyℓ\ell, the chain rule gives

∂ℓ∂σ=2​σ​∂ℓ∂λ,\frac{\partial\ell}{\partial\sigma}=2\sigma\,\frac{\partial\ell}{\partial\lambda},(20)so a penalty whose eigenvalue derivative stays bounded asλ→0\lambda\to 0exerts a force onσ\sigmathat vanishes linearly, and only a derivative that diverges likeλ−1/2\lambda^{-1/2}keeps a constant force. The three penalties of equation4fall on the two sides of this line, as Figure3shows.

Bures-Wasserstein.ℓ=(λ−1)2\ell=(\sqrt{\lambda}-1)^{2}has∂λℓ=1−λ−1/2\partial_{\lambda}\ell=1-\lambda^{-1/2}, which diverges at exactly the compensating rate, and∂σℓ=2​(σ−1)→−2\partial_{\sigma}\ell=2(\sigma-1)\to-2.

Euclidean covariance penalty.ℓ=(λ−1)2\ell=(\lambda-1)^{2}has∂λℓ=2​(λ−1)→−2\partial_{\lambda}\ell=2(\lambda-1)\to-2, bounded, and∂σℓ=4​σ​(σ2−1)→0\partial_{\sigma}\ell=4\sigma(\sigma^{2}-1)\to 0.

SIGReg.The Epps-Pulley statistic(EPPS and PULLEY, 1983)compares the empirical characteristic functionφn​(τ)=1n​∑jexp⁡(i​τ​σ​xj)\varphi_{n}(\tau)=\tfrac{1}{n}\sum_{j}\exp(\mathrm{i}\tau\sigma x_{j})with that of a standard Gaussian under a weightω\omega,T(σ)=n∫|φn(τ)−e−τ2/2|2ω(τ)dτT(\sigma)=n\int|\varphi_{n}(\tau)-e^{-\tau^{2}/2}|^{2}\,\omega(\tau)\,d\tau. Expanding at smallσ\sigmawith standardizedxjx_{j}givesφn​(τ)=1−12​τ2​σ2+O⁡(σ3)\varphi_{n}(\tau)=1-\tfrac{1}{2}\tau^{2}\sigma^{2}+O(\sigma^{3}), where the cubic term is purely imaginary, hence

T(σ)=T(0)−κσ2+O(σ4),κ=n∫τ2(1−e−τ2/2)ω(τ)dτ>0.T(\sigma)=T(0)-\kappa\,\sigma^{2}+O(\sigma^{4}),\qquad\kappa=n\int\tau^{2}\bigl(1-e^{-\tau^{2}/2}\bigr)\omega(\tau)\,d\tau>0.(21)The statistic is bounded, since|φn(τ)−e−τ2/2|≤2|\varphi_{n}(\tau)-e^{-\tau^{2}/2}|\leq 2, its eigenvalue derivative tends to the constant−κ-\kappa, and its force onσ\sigmais−2​κ​σ→0-2\kappa\sigma\to 0. A collapsing direction therefore sees a saturated SIGReg score and a vanishing restoring force, as we observed in training.

Figure 3:Eigenvalue derivatives of the three penalties on synthetic data.The plot shows|∂ℓ/∂λ||\partial\ell/\partial\lambda|againstλ\lambda. Bures-Wasserstein rises asλ−1/2\lambda^{-1/2}whenλ→0\lambda\to 0, while the Euclidean penalty and SIGReg flatten to constants. By equation20, a flat curve means a force on the standard deviation that vanishes likeλ\sqrt{\lambda}, and theλ−1/2\lambda^{-1/2}rise means a constant force. Vertical offsets reflect the native normalization of each statistic and are not comparable across curves. The comparison is between slopes.

Appendix ITraining behavior

All H-JEPA models were trained end-to-end for1010epochs on a single NVIDIA RTX A5000 GPU with the optimization settings of AppendixC. A held-out pilot seed, excluded from the three-seed benchmark average, was used once to choose which epoch’s checkpoint to evaluate on each task. We then fixed this choice at epoch88for Two-Room,66for Reacher,1010for PushT, and33for OGB-Cube for every reported seed. For comparison, Delta-JEPA is trained for5050epochs under its evaluation protocol(Zhang et al., 2026). Figures4and5show the available extended traces for PushT and OGB-Cube, including prediction, PIC, and Bures-Wasserstein isotropy on both training and validation data.

The curves remain closely matched between training and validation. PushT improves steadily over all1010epochs. OGB-Cube reaches its useful regime within the first three epochs, after which the representation-level losses continue to decrease without a corresponding need for a later checkpoint. The traces also show that all three objectives can be optimized jointly without loss-specific warm-up schedules or alternating optimization.

(a)Prediction(b)Port-inverse consistency(c)IsotropyFigure 4:H-JEPA training on PushT.Training and validation losses across the1010-epoch run for(a)open-loop latent prediction,(b)port-inverse consistency (PIC), and(c)the Bures-Wasserstein isotropy objective. All objectives decrease smoothly, with closely tracking training and validation curves.(a)Prediction(b)Port-inverse consistency(c)IsotropyFigure 5:H-JEPA training on OGB-Cube.Training and validation losses for(a)open-loop latent prediction,(b)PIC, and(c)isotropy. All three objectives fall rapidly during the first epochs and subsequently improve smoothly, with little separation between training and validation curves. The benchmark model uses the epoch-33OGB-Cube checkpoint selected in §4. The extended trace shows that lower representation-level losses after convergence are not required for the reported planning result.