VPFlow: Value-Predictive Planning with Imperfect World Models for Offline Goal Reaching

# VPFlow: Value-Predictive Planning with Imperfect World Models for Offline Goal Reaching
Source: [https://arxiv.org/html/2610.27280](https://arxiv.org/html/2610.27280)
###### Abstract

Offline goal\-conditioned reinforcement learning enables skill acquisition by reaching any goal from any state without dense rewards\. Model\-based methods allow for planning by scaling test\-time compute to solve complex tasks\. Combining both holds appealing promise for pre\-training on suboptimal demonstrations and zero\-shot adaptation via goal specification and planning\. Yet existing methods assume effective planning requires accurate next\-state prediction in pixel or latent space\. In this work, we show that value prediction is more useful for planning, whereas accurate state prediction is often unnecessary\. Our key insight is that offline goal reaching naturally admits a closed\-form Monte Carlo return, enabling value regression to replace temporal\-difference bootstrapping\. We use this return as the sole supervision for latent world models: imagined latents are trained to predict goal\-reaching value rather than match future states, implicitly preventing representation collapse\. These planning\-oriented representations let imperfect world models imagine on the level sets of value field beyond dataset transitions\. We then presentVPFlow, a value\-predictive planner that pairs the world models with a one\-step flow policy and plans in its noise space via evolutionary search\. It enables policy improvement with imagined generalized advantage and reduces out\-of\-distribution queries common in action\-space planning\. On 30 state and pixel OGBench datasets,VPFlowsurpasses model\-free baselines by at least 30\.3 and 10\.1 percentage points in success rate, respectively, and outperforms latent planners LeWorldModel, PLDM, and TD\-MPC2\. It achieves this despite a latent prediction error orders of magnitude larger than with state prediction, while minimizing this error lowers success rates\. This demonstrates that value\-predictive representations, not exact state prediction, are what offline planning requires\.

††correspondence:[yzhu@cs\.hku\.hk](mailto:[email protected])## 1Introduction

Offline reinforcement learning \(RL\) seeks to learn reusable behaviors purely from previously collected trajectories with no additional environment interaction\([Levine et al\., 2020](https://arxiv.org/html/2610.27280#bib.bib44)\)\. In its goal\-conditioned setting, an agent is tasked with reaching a desired goal at test time without optimizing a hand\-designed reward\([Kaelbling, 1993](https://arxiv.org/html/2610.27280#bib.bib37);[Schaul et al\., 2015](https://arxiv.org/html/2610.27280#bib.bib69);[Park et al\., 2025a](https://arxiv.org/html/2610.27280#bib.bib60)\)\. This makes offline goal reaching especially attractive for reward\-free skill acquisition: relabeling future observations as goals turns one unlabeled dataset into supervision for many tasks\([Andrychowicz et al\., 2017](https://arxiv.org/html/2610.27280#bib.bib1)\)\. Pairing it with model\-based planning is a natural next step: pre\-train on offline data once, then adapt to new goals in zero shot by searching over imagined futures at test time\. This combination is promising, yet comparatively unexplored\.

Figure 1:State\-predictive and value\-predictive action\-conditioned latent world models\.*Left*: Joint\-embedding predictive architecture \(JEPA\) aligns predicted and future latents with a joint encoder and explicit regularization,*e\.g\.*, variance\-invariance\-covariance regularization \(VICReg\) or sketched isotropic Gaussian regularization \(SIGReg\)\.*Middle*: TD\-MPC learns values from one\-step temporal\-difference \(TD\) targets, with source and target encoders, stop\-gradient, and exponential moving average \(EMA\) heuristics\.*Right*:VPFlowtrains the latent dynamics and encoders jointly via value prediction: predicted latent states seek accurate goal\-reaching value estimates, not future states\. Closed\-form hindsight Monte Carlo \(MC\) returns provide the temporal signal and act as an implicit regularizer\.Strong offline goal\-conditioned methods, spanning implicit Q\-learning\([Kostrikov et al\., 2022](https://arxiv.org/html/2610.27280#bib.bib41)\), contrastive learning\([Eysenbach et al\., 2022](https://arxiv.org/html/2610.27280#bib.bib20)\), and quasi\-metric representations\([Wang et al\., 2023](https://arxiv.org/html/2610.27280#bib.bib83);[Myers et al\., 2025](https://arxiv.org/html/2610.27280#bib.bib53)\), learn a reactive policy or value function from relabeled trajectories and act greedily at deployment\. While robust and efficient, they are static at test time: once training ends, additional inference compute cannot improve decisions on challenging tasks\. Model\-based planning offers a complementary advantage\. From Dyna to TD\-MPC\([Sutton, 1991](https://arxiv.org/html/2610.27280#bib.bib75);[Hafner et al\., 2019](https://arxiv.org/html/2610.27280#bib.bib29);[Hafner et al\., 2020](https://arxiv.org/html/2610.27280#bib.bib30);[Hansen et al\., 2022](https://arxiv.org/html/2610.27280#bib.bib33)\), learned world models convert extra test\-time compute into better decisions by searching over imagined futures\. Yet these methods share a foundational assumption that has migrated from pixels to latents largely unquestioned: the world model must accurately predict the next state, or the next latent embedding, for planning to succeed\([Hafner et al\., 2019](https://arxiv.org/html/2610.27280#bib.bib29);[Schwarzer et al\., 2021](https://arxiv.org/html/2610.27280#bib.bib72);[Sobal et al\., 2025](https://arxiv.org/html/2610.27280#bib.bib73);[Maes et al\., 2026b](https://arxiv.org/html/2610.27280#bib.bib52)\)\.

We argue that this state\-predictive assumption is both overly restrictive and misdirected\. What a planner needs is not a model that reproduces the next observation, but one that preserves the relative ordering of candidate futures according to their goal\-reaching value\. This view builds on a rich lineage: value prediction networks learn dynamics only as much as they reproduce Bellman backups\([Oh et al\., 2017](https://arxiv.org/html/2610.27280#bib.bib56)\); universal value functions and successor representations factor value into future occupancy and a goal\-dependent readout\([Dayan, 1993](https://arxiv.org/html/2610.27280#bib.bib16);[Schaul et al\., 2015](https://arxiv.org/html/2610.27280#bib.bib69);[Barreto et al\., 2017](https://arxiv.org/html/2610.27280#bib.bib8)\); and the value\-equivalence principle formalizes when a model that is inaccurate as a simulator remains exact for control\([Grimm et al\., 2020](https://arxiv.org/html/2610.27280#bib.bib26)\)\. We take this principle a step further: an imagined latent only needs to land on the right level set of the value field, so the representation worth learning for planning is*value\-predictive*rather than*state\-predictive*\. Offline goal reaching makes this directly actionable\. Because the relabeled reward is sparse, the Monte Carlo \(MC\) return from any state to any future goal on its trajectory is a closed\-form function of their temporal distance\. Value learning thus simplifies from temporal\-difference bootstrapping to direct regression, and that regression target can supervise the latent dynamics end to end, acting as an implicit regularizer against representation collapse\. A model trained this way may be an imperfect simulator yet remain an effective planner\.

Motivated by this view, we introduceValue\-Predictive Flow \(VPFlow\), which plans for offline goal\-reaching tasks with a deliberately imperfect world model\.VPFlowjointly learns an observation encoder, a goal\-agnostic latent dynamics model, and goal\-conditioned value heads alongside a distilled flow\-matching policy with action chunks\. Imagined latents are trained purely to be*value\-predictive*: rolled forward with dataset actions, the value predicted at each imagined latent is regressed towards the closed\-form return of the corresponding future state\. No observation is reconstructed, no target latent is matched, and no bootstrapped target network is required\. Figure[1](https://arxiv.org/html/2610.27280#S1.F1)contrasts alternative latent\-predictive architectures with ours\.

To translate a value\-predictive model into reliable control, the planner must propose feasible, in\-support actions\. Standard zeroth\-order planners such as the cross\-entropy method \(CEM\) and model predictive path integral \(MPPI\)\([Rubinstein, 1999](https://arxiv.org/html/2610.27280#bib.bib67);[Williams et al\., 2017](https://arxiv.org/html/2610.27280#bib.bib85)\)perturb raw action sequences, which in offline settings quickly leave the data support and expose the value function to out\-of\-distribution \(OOD\) queries\.VPFlowinstead searches in the noise space of its one\-step flow policy\([Park et al\., 2025c](https://arxiv.org/html/2610.27280#bib.bib62)\), where each candidate noise vector decodes into action chunks at the current latent state\. Because of behavior\-cloning regularization, the noise search warm\-starts at multi\-modal, in\-support proposals and stays anchored to the data manifold\. We guide this noise steering using the generalized advantage estimator \(GAE\)\([Schulman et al\., 2016](https://arxiv.org/html/2610.27280#bib.bib71)\)over imagined rollouts, instantiated as an evolution strategy via annealed mutation and elite selection\.

Across 30 locomotion and manipulation datasets from OGBench\([Park et al\., 2025a](https://arxiv.org/html/2610.27280#bib.bib60)\),VPFlowraises the mean success rate from32\.9%32\.9\\%to63\.2%63\.2\\%over 17 state datasets and from27\.5%27\.5\\%to37\.6%37\.6\\%over 13 pixel datasets relative to the strongest non\-hierarchical baselines, while substantially outperforming the latent state\-predictive planners LeWorldModel\([Maes et al\., 2026b](https://arxiv.org/html/2610.27280#bib.bib52)\), PLDM\([Sobal et al\., 2025](https://arxiv.org/html/2610.27280#bib.bib73)\), and TD\-MPC2\([Hansen et al\., 2024](https://arxiv.org/html/2610.27280#bib.bib34)\)\. Crucially, these gains hold while the model’s next\-latent prediction error sits orders of magnitude above that of its state\-predictive counterpart; conversely, enforcing state prediction closes that gap at the cost of lowering the success rate\. This demonstrates that offline planning does not require a faithful state simulator, but rather a value\-predictive one\. Our contributions are as follows:

1. 1\.We argue and demonstrate that value\-predictive latent world models, trained solely via closed\-form Monte Carlo return regression, avoid representation collapse and suffice for offline planning even as state\-prediction error grows, challenging the pervasive assumption of learning state\-predictive world models for planning\.
2. 2\.We propose a test\-time planning algorithm tailored to generative flow policies in their noise space, steering initial noise towards higher imagined generalized advantage while avoiding the out\-of\-distribution action queries of existing action\-space planning methods\.
3. 3\.We show across 30 state and pixel datasets on OGBench thatVPFlowsurpasses model\-free baselines and latent state\-predictive planners, and scales favorably with test\-time compute on long\-horizon tasks\.

## 2

0pt2\.4ex plus 0\.5ex minus 0\.2ex1\.4ex plus 0\.2ex

## 3Related Work

We positionVPFlowwithin three lines of work with broader literature in Appendix[A](https://arxiv.org/html/2610.27280#A1)\.

Offline goal\-conditioned RL\.Offline RL learns a policy from pre\-collected datasets without interacting with the environment, and needs techniques to keep actions on the data manifold during value learning\([Levine et al\., 2020](https://arxiv.org/html/2610.27280#bib.bib44);[Kostrikov et al\., 2022](https://arxiv.org/html/2610.27280#bib.bib41);[Kumar et al\., 2020](https://arxiv.org/html/2610.27280#bib.bib42);[Fujimoto and Gu, 2021](https://arxiv.org/html/2610.27280#bib.bib22);[Park et al\., 2025c](https://arxiv.org/html/2610.27280#bib.bib62)\)\. In the goal\-conditioned setting, an agent aims to reach any goal from any state\([Kaelbling, 1993](https://arxiv.org/html/2610.27280#bib.bib37)\), and hindsight relabeling\([Andrychowicz et al\., 2017](https://arxiv.org/html/2610.27280#bib.bib1)\)can convert any trajectories, even failed ones, into successful ones with the goal actually reached\. With high\-coverage datasets, this may enable learning a universal value function that empowers the policy to generalize to unseen goals in zero shot\([Schaul et al\., 2015](https://arxiv.org/html/2610.27280#bib.bib69);[Park et al\., 2024](https://arxiv.org/html/2610.27280#bib.bib59)\)\.

Representations for planning with world models\.World models seek a compact representation and transition dynamics from observations, where value can be learned in imagination\([Sutton, 1991](https://arxiv.org/html/2610.27280#bib.bib75);[Ha and Schmidhuber, 2018](https://arxiv.org/html/2610.27280#bib.bib28);[Hafner et al\., 2020](https://arxiv.org/html/2610.27280#bib.bib30);[Hansen et al\., 2022](https://arxiv.org/html/2610.27280#bib.bib33)\)\. Yet what representations are more effective for planning is still an open question\. Reconstruction\-based methods harness the dense supervision from pixels and have achieved remarkable success\([Hafner et al\., 2019](https://arxiv.org/html/2610.27280#bib.bib29);[Hafner et al\., 2025a](https://arxiv.org/html/2610.27280#bib.bib31)\)\. Self\-predictive representation holds promise for a more powerful model\([Schwarzer et al\., 2021](https://arxiv.org/html/2610.27280#bib.bib72);[LeCun, 2022](https://arxiv.org/html/2610.27280#bib.bib43);[Bagatella et al\., 2026](https://arxiv.org/html/2610.27280#bib.bib5)\), but needs stop\-gradient, EMA, or regularization to prevent collapse\([Sobal et al\., 2025](https://arxiv.org/html/2610.27280#bib.bib73);[Maes et al\., 2026b](https://arxiv.org/html/2610.27280#bib.bib52)\)\. More critically, it fails to exploit the rich temporal\-distance supervision naturally available from goal reaching\. This makes its usefulness for planning not directly obvious\. Our work aims to reveal what is essential and necessary for planning\-oriented representations\.

Test\-time search\.With a learned world model, previous studies primarily plan actions by sampling\-based optimization\([Williams et al\., 2017](https://arxiv.org/html/2610.27280#bib.bib85);[Janner et al\., 2019](https://arxiv.org/html/2610.27280#bib.bib36);[Hafner et al\., 2019](https://arxiv.org/html/2610.27280#bib.bib29);[Hansen et al\., 2022](https://arxiv.org/html/2610.27280#bib.bib33);[Park et al\., 2026](https://arxiv.org/html/2610.27280#bib.bib57)\)or search\([Schrittwieser et al\., 2020](https://arxiv.org/html/2610.27280#bib.bib70)\)\. Yet action\-space planning may suffer heavily in the offline setting, especially with low\-coverage data, in that off\-manifold actions can severely exploit the planning objective\. Advances in generative policies have turned to steering the noise at test time to keep actions in\-distribution, but they need reward or other external signals and have yet to show scaling effectiveness\([Yang et al\., 2024](https://arxiv.org/html/2610.27280#bib.bib87);[Wagenmaker et al\., 2025](https://arxiv.org/html/2610.27280#bib.bib81);[Patil et al\., 2026](https://arxiv.org/html/2610.27280#bib.bib63)\)\. In contrast,VPFlowcan improve itself at test time by noise steering111We use “steering”, “planning”, and “searching” interchangeably in this paper\.with imagined rollouts, even with imperfect world models\.

## 4Problem Statement and Preliminaries

Offline goal reaching\.We consider the offline goal\-conditioned reinforcement learning problem as a reward\-free controlled Markov decision process \(MDP\)ℳ=\(𝒮,𝒜,γ,P\)\\mathcal\{M\}=\(\\mathcal\{S\},\\mathcal\{A\},\\gamma,P\), where𝒮\\mathcal\{S\}denotes the state space,𝒜\\mathcal\{A\}denotes the action space,γ∈\[0,1\)\\gamma\\in\[0,1\)denotes the discount factor, andP:𝒮×𝒜→Δ⁡\(𝒮\)P:\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\Delta\(\\mathcal\{S\}\)denotes the transition dynamics mapping state and action to another state distributionΔ⁡\(⋅\)\\Delta\(\\cdot\)\. Given a dataset𝒟=\{τ\(i\)\}i=1N𝒟\\mathcal\{D\}=\\\{\\tau^\{\(i\)\}\\\}\_\{i=1\}^\{N\_\{\\mathcal\{D\}\}\}ofN𝒟N\_\{\\mathcal\{D\}\}trajectoriesτ\(i\)=\(s0\(i\),a0\(i\),s1\(i\),…,sTi\(i\)\)\\tau^\{\(i\)\}=\(s\_\{0\}^\{\(i\)\},a\_\{0\}^\{\(i\)\},s\_\{1\}^\{\(i\)\},\\dots,s\_\{T\_\{i\}\}^\{\(i\)\}\)of lengthTiT\_\{i\}, collected by a behavior policy222No assumption on its optimality\., the objective is to learn a goal\-conditioned policyπ⁡\(a∣s,g\)\\pi\(a\\mid s,g\)that maximizes the expected discounted utility𝔼τ​\[∑t=0∞γt​δg​\(st\)\]\\mathbb\{E\}\_\{\\tau\}\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\delta\_\{g\}\(s\_\{t\}\)\]for any goalg∈𝒮g\\in\\mathcal\{S\}\. Here,δg​\(st\)\\delta\_\{g\}\(s\_\{t\}\)has been referred to as the “Dirac delta function”\([Park et al\., 2025a](https://arxiv.org/html/2610.27280#bib.bib60)\)\. In practice, we consider it as the indicator function𝟙\{st=g\}\\mathds\{1\}\_\{\\\{s\_\{t\}=g\\\}\}that represents the relabeled sparse reward:

r⁡\(st,g\)≔\{1,if​st=g,0,otherwise\.r\(s\_\{t\},g\)\\coloneqq\\begin\{cases\}1,&\\text\{if \}s\_\{t\}=g,\\\\ 0,&\\text\{otherwise\}\.\\end\{cases\}\(1\)Model predictive control\.Given a transition dynamics modelf:𝒮×𝒜→𝒮f:\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\mathcal\{S\}, model predictive control \(MPC\) addresses an online constrained optimal control problem with a lookahead horizonHHand an action sequenceat:t\+H−1=\(at,…,at\+H−1\)a\_\{t:t\+H\-1\}=\(a\_\{t\},\\dots,a\_\{t\+H\-1\}\)at the current statests\_\{t\}, then executes the first actionata\_\{t\}, and re\-plans at the next state\. More formally, MPC solves the following optimization problem at time steptt:

minat:t\+H−1∑h=0H−1ℓt\(st\+h,at\+h\)\+ℓf\(st\+H\),s\.t\.st\+1=f\(st,at\),\\min\_\{a\_\{t:t\+H\-1\}\}\\sum\_\{h=0\}^\{H\-1\}\\ell\_\{t\}\(s\_\{t\+h\},a\_\{t\+h\}\)\+\\ell\_\{f\}\(s\_\{t\+H\}\),\\quad\\text\{s\.t\.\}\\quad s\_\{t\+1\}=f\(s\_\{t\},a\_\{t\}\),\(2\)whereℓt​\(s,a\)\\ell\_\{t\}\(s,a\)andℓf​\(s\)\\ell\_\{f\}\(s\)are the running and terminal cost functions of interest\. After finding the minimizerat:t\+H−1⋆a\_\{t:t\+H\-1\}^\{\\star\}of Equation[2](https://arxiv.org/html/2610.27280#S4.E2), only the first actionat⋆a\_\{t\}^\{\\star\}is applied due to the compounding error in the dynamics\. In the model\-based RL context, the cost has been instantiated using negative rewardrt​\(st,at\)r\_\{t\}\(s\_\{t\},a\_\{t\}\)and the value functionQ⁡\(st,at\)Q\(s\_\{t\},a\_\{t\}\)orV⁡\(st\)V\(s\_\{t\}\)\. Both are learned as part of the world modelf⁡\(st,at\)f\(s\_\{t\},a\_\{t\}\), which can be used to plan without interacting with the real environment\([Hafner et al\., 2019](https://arxiv.org/html/2610.27280#bib.bib29);[Hafner et al\., 2020](https://arxiv.org/html/2610.27280#bib.bib30);[Hansen et al\., 2022](https://arxiv.org/html/2610.27280#bib.bib33)\)\. Planning is widely done by sampling\-based trajectory optimizers\([Rubinstein, 1999](https://arxiv.org/html/2610.27280#bib.bib67);[Williams et al\., 2017](https://arxiv.org/html/2610.27280#bib.bib85)\)\. Another emerging way is to learn planning\-enabled representations without the reward and value function, and use a simpleℓ2\\ell\_\{2\}distance to the goal as the cost\([Assran et al\., 2025](https://arxiv.org/html/2610.27280#bib.bib3);[Sobal et al\., 2025](https://arxiv.org/html/2610.27280#bib.bib73);[Maes et al\., 2026b](https://arxiv.org/html/2610.27280#bib.bib52)\)\. Both lines of work assume the optimal objective is to predict the next state when learning world models\. In this work, we provide evidence for another value\-predictive representation view and show that it is more useful for planning\.

## 5VPFlow: Planning with Imperfect World Models

### 5\.1Value Equivalence Principle

![Refer to caption](https://arxiv.org/html/2610.27280v1/value-equivalence-crop.png)Figure 2:The value equivalence principle view of world modeling\.The value\-predictive representations of the same timestep constitute a level set of the value field\.We motivateVPFlowthrough the value equivalence principle\([Grimm et al\., 2020](https://arxiv.org/html/2610.27280#bib.bib26)\), which states that two*dynamics and reward models*are value\-equivalent with respect to a set of value functions if they induce the same Bellman operator\. Consequently, one can learn another set of dynamics and reward models beyond the training examples but still keep the same effect for policy learning\.

We extend this view to world modeling for offline goal reaching: two*predictive representations*s^,s^′∈𝒮\\hat\{s\},\\hat\{s\}^\{\\prime\}\\in\\mathcal\{S\}are value\-equivalent with respect to a goal if they are evaluated identically by the goal\-conditioned value function,*i\.e\.*,Vϕ​\(s^,g\)=Vϕ​\(s^′,g\)V\_\{\\phi\}\(\\hat\{s\},g\)=V\_\{\\phi\}\(\\hat\{s\}^\{\\prime\},g\)\(Figure[2](https://arxiv.org/html/2610.27280#S5.F2)\)\. In this setting, the Bellman operator can be written in closed form with sparse reward given a goal state, reducing recursive temporal\-difference learning\([Oh et al\., 2017](https://arxiv.org/html/2610.27280#bib.bib56);[Schrittwieser et al\., 2020](https://arxiv.org/html/2610.27280#bib.bib70)\)to direct regression on fixed targets\. Intuitively, rather than constraining the dynamics models to match the future states, this principle only requires the predicted representations to lie on the corresponding level sets of value functions, and this relaxation can potentially expand the reachable space of dynamics models\.

### 5\.2Hindsight Monte Carlo Return Supervision

![Refer to caption](https://arxiv.org/html/2610.27280v1/vpflow-learning-crop.png)Figure 3:VPFlowobjectives\.VPFlowalternately trains \(a\) the goal\-conditioned value functionVϕV\_\{\\phi\}, latent dynamicsfϕf\_\{\\phi\}, and encoderEϕE\_\{\\phi\}jointly, and \(b\) a distilled one\-step flow policyπθ\\pi\_\{\\theta\}\. The value function regresses the closed\-form Monte Carlo returnsRhR\_\{h\}on imagined rollouts\. The policy is improved by maximizing the imagined value from the learned dynamics model, regularized by conditional flow matching to stay close to the data manifold\.Q\-learning typically uses bootstrapping that predicts Bellman backup online with a target network and EMA to learn the action\-value functionQ⁡\(s,a\)Q\(s,a\)\. Though this heuristic stabilizes training\([Tang et al\., 2023](https://arxiv.org/html/2610.27280#bib.bib76)\), it makes TD learning less principled for scaling on larger datasets\([Park et al\., 2025b](https://arxiv.org/html/2610.27280#bib.bib61)\)\.

With hindsight relabeling, every trajectory in the dataset can be relabeled with a goal as if the agent in that trajectory has reached that goal\. This naturally admits a cleaner and simpler objective for value learning\. Concretely, the target value for a statests\_\{t\}is the discounted Monte Carlo return fromsts\_\{t\}to the relabeled goalg=st\+dg=s\_\{t\+d\}on the same trajectory:

R⁡\(st,g\)=R⁡\(d\)=∑k=0dγk​r​\(st\+k,g\)=\{γd,r\(s,g\)=𝟙\{s=g\}1−γdγ−1,r\(s,g\)=𝟙\{s=g\}−1,R\(s\_\{t\},g\)=R\(d\)=\\sum\_\{k=0\}^\{d\}\\gamma^\{k\}r\(s\_\{t\+k\},g\)=\\begin\{cases\}\\gamma^\{d\},&r\(s,g\)=\\mathds\{1\}\_\{\\\{s=g\\\}\}\\\\ \\frac\{1\-\\gamma^\{d\}\}\{\\gamma\-1\},&r\(s,g\)=\\mathds\{1\}\_\{\\\{s=g\\\}\}\-1\\end\{cases\},\(3\)whered≥0d\\geq 0is the temporal distance,*i\.e\.*, the number of steps fromsts\_\{t\}to the goalggon the same trajectory\. In practice, we use the sparse rewardr\(s,g\)=𝟙\{s=g\}−1r\(s,g\)=\\mathds\{1\}\_\{\\\{s=g\\\}\}\-1, following prior work\([Physical Intelligence et al\., 2025](https://arxiv.org/html/2610.27280#bib.bib65);[Park et al\., 2025a](https://arxiv.org/html/2610.27280#bib.bib60)\)\. Intuitively, this quantity measures how far the agent is from the goal as a progress signal\.

The training ofVPFlowalternates between the value prediction and policy learning objectives, as shown in Figure[3](https://arxiv.org/html/2610.27280#S5.F3)and elaborated below\.

### 5\.3Value\-Predictive Latent Dynamics

Given this closed\-form Monte Carlo return in Equation[3](https://arxiv.org/html/2610.27280#S5.E3),VPFlowlearns the observation encoderEϕ:𝒮→𝒵E\_\{\\phi\}:\\mathcal\{S\}\\to\\mathcal\{Z\}for a latent space𝒵\\mathcal\{Z\}, latent dynamicsfϕf\_\{\\phi\}, and a goal\-conditioned value functionVϕ:𝒵×𝒵→ℝV\_\{\\phi\}:\\mathcal\{Z\}\\times\\mathcal\{Z\}\\to\\mathbb\{R\}\. All three components are optimized jointly with the value\-prediction \(VP\) objective\. In particular, we use the action chunk representation of horizonHHas𝒂t=at:t\+H−1\{\\bm\{a\}\}\_\{t\}=a\_\{t:t\+H\-1\}, which allows for a certain level of temporal abstraction\([Li et al\., 2025](https://arxiv.org/html/2610.27280#bib.bib46)\)\. After encoding the state and goal withEϕE\_\{\\phi\}:

zt=Eϕ​\(st\),zg=Eϕ​\(g\),z\_\{t\}=E\_\{\\phi\}\(s\_\{t\}\),\\quad z\_\{g\}=E\_\{\\phi\}\(g\),\(4\)the latent dynamics predicts each future latent with an action prefix:

z^t\+h=fϕ\(zt,at:t\+h−1\),h=1,…,H,z^t≔zt\.\\hat\{z\}\_\{t\+h\}=f\_\{\\phi\}\(z\_\{t\},a\_\{t:t\+h\-1\}\),\\quad h=1,\\ldots,H,\\quad\\hat\{z\}\_\{t\}\\coloneqq z\_\{t\}\.\(5\)Unrollingfϕf\_\{\\phi\}with the action chunk𝒂t\{\\bm\{a\}\}\_\{t\}yields intermediate latentsz^t\+1,…,z^t\+H\\hat\{z\}\_\{t\+1\},\\dots,\\hat\{z\}\_\{t\+H\}, wherez^t\+H=fϕ​\(zt,𝒂t\)\\hat\{z\}\_\{t\+H\}=f\_\{\\phi\}\(z\_\{t\},\{\\bm\{a\}\}\_\{t\}\)denotes the final predicted latent\. The VP loss is then defined as:

ℒVP​\(ϕ\)≔𝔼\(st,𝒂t\)∼𝒟,g∼𝒮​\[1H\+1​∑h=0HMt\+h⊙‖Vϕ​\(z^t\+h,zg\)−Rt\+h‖22\],z^t≔zt,\\mathcal\{L\}\_\{\\mathrm\{VP\}\}\(\\phi\)\\coloneqq\\mathbb\{E\}\_\{\(s\_\{t\},\{\\bm\{a\}\}\_\{t\}\)\\sim\\mathcal\{D\},g\\sim\\mathcal\{S\}\}\\left\[\\frac\{1\}\{H\+1\}\\sum\_\{h=0\}^\{H\}M\_\{t\+h\}\\odot\\\|V\_\{\\phi\}\(\\hat\{z\}\_\{t\+h\},z\_\{g\}\)\-R\_\{t\+h\}\\\|\_\{2\}^\{2\}\\right\],\\quad\\hat\{z\}\_\{t\}\\coloneqq z\_\{t\},\(6\)whereRt\+h=R⁡\(st\+h,g\)R\_\{t\+h\}=R\(s\_\{t\+h\},g\)is the target of Equation[3](https://arxiv.org/html/2610.27280#S5.E3)for the dataset statest\+hs\_\{t\+h\}and does not depend onϕ\\phi,⊙\\odotis the Hadamard product, and the maskMt\+h∈\{0,1\}M\_\{t\+h\}\\in\\\{0,1\\\}filters out goals behind timet\+ht\+hand across trajectories such that the value function only exploits the clean temporal distance signal on the same trajectory\.

Equation[6](https://arxiv.org/html/2610.27280#S5.E6)trains the encoder, latent dynamics, and value function jointly\. The value supervision nudges the predictive representation of each rollout horizon towards the level set of the value function defined by the state\-goal temporal distance, instead of aiming for perfect state prediction\. It structures the representation space with explicit meaning as progress towards reaching the goal, unlike the regularizers that are only statistically meaningful\([Bardes et al\., 2022](https://arxiv.org/html/2610.27280#bib.bib7);[Balestriero and LeCun, 2025](https://arxiv.org/html/2610.27280#bib.bib6)\)\. It also expands the hypothesis space for the policy without being too conservative inside the data manifold, as we discussed in Section[5\.1](https://arxiv.org/html/2610.27280#S5.SS1)\.

### 5\.4Flow Policy as a Behavior Prior

For a more expressive policy to handle the multimodality of offline datasets, we opt for a flow\-based policy similar to Flow Q\-learning\([Park et al\., 2025c](https://arxiv.org/html/2610.27280#bib.bib62)\)\. Beyond that, the generative policy as a behavior prior is also a key design choice to enable planning with the data manifold constraint at test time\.

The policy improvement objective is to maximize the Q\-function of an action chunk\([Li et al\., 2025](https://arxiv.org/html/2610.27280#bib.bib46)\)given encoded state and goal,zt=Eϕ​\(st\),zg=Eϕ​\(g\)z\_\{t\}=E\_\{\\phi\}\(s\_\{t\}\),z\_\{g\}=E\_\{\\phi\}\(g\), using the learned world models:

Q⁡\(zt,𝒂t,zg\)\\displaystyle Q\(z\_\{t\},\{\\bm\{a\}\}\_\{t\},z\_\{g\}\)≔𝔼z^t\+H∼fϕ​\(zt,𝒂t\)​\[∑h=0H−1γh​r​\(zt\+h,zg\)\+γH​Vϕ​\(z^t\+H,zg\)\]\\displaystyle\\coloneqq\\mathbb\{E\}\_\{\\hat\{z\}\_\{t\+H\}\\sim f\_\{\\phi\}\(z\_\{t\},\{\\bm\{a\}\}\_\{t\}\)\}\\left\[\\sum\_\{h=0\}^\{H\-1\}\\gamma^\{h\}r\(z\_\{t\+h\},z\_\{g\}\)\+\\gamma^\{H\}V\_\{\\phi\}\(\\hat\{z\}\_\{t\+H\},z\_\{g\}\)\\right\]\(7\)≈𝔼z^t\+H∼fϕ​\(zt,𝒂t\)\[γHVϕ\(z^t\+H,zg\)\]\(noθin reward\),\\displaystyle\\approx\\mathbb\{E\}\_\{\\hat\{z\}\_\{t\+H\}\\sim f\_\{\\phi\}\(z\_\{t\},\{\\bm\{a\}\}\_\{t\}\)\}\\left\[\\gamma^\{H\}V\_\{\\phi\}\(\\hat\{z\}\_\{t\+H\},z\_\{g\}\)\\right\]\\qquad\\text\{\(no $\\theta$ in reward\)\},where the action chunk𝒂t=at:t\+H−1\{\\bm\{a\}\}\_\{t\}=a\_\{t:t\+H\-1\}is decoded from Gaussian noise𝜺∼𝒩⁡\(𝟎,I\)\\bm\{\\varepsilon\}\\sim\\mathcal\{N\}\(\{\\bm\{0\}\},I\)by the one\-step flow policyπθ​\(zt,zg,𝜺\)\\pi\_\{\\theta\}\(z\_\{t\},z\_\{g\},\\bm\{\\varepsilon\}\)with parametersθ\\theta\. Together with a velocity fieldvθv\_\{\\theta\}, the policy learning loss is defined as:

ℒπ​\(θ\)≔−Q⁡\(zt,𝒂t,zg\)\+αCFM​ℒCFM\+αDistill​ℒDistill\.\\mathcal\{L\}\_\{\\pi\}\(\\theta\)\\coloneqq\-Q\(z\_\{t\},\{\\bm\{a\}\}\_\{t\},z\_\{g\}\)\+\\alpha\_\{\\mathrm\{CFM\}\}\\mathcal\{L\}\_\{\\mathrm\{CFM\}\}\+\\alpha\_\{\\mathrm\{Distill\}\}\\mathcal\{L\}\_\{\\mathrm\{Distill\}\}\.\(8\)Here,ℒCFM\\mathcal\{L\}\_\{\\mathrm\{CFM\}\}is the conditional flow matching \(CFM\) loss that fitsvθv\_\{\\theta\}to the dataset actions,ℒDistill\\mathcal\{L\}\_\{\\mathrm\{Distill\}\}distills the multi\-step decoding process usingvθv\_\{\\theta\}intoπθ\\pi\_\{\\theta\}, andαCFM,αDistill\\alpha\_\{\\mathrm\{CFM\}\},\\alpha\_\{\\mathrm\{Distill\}\}are their weights\. The one\-step policy makes noise steering more efficient\. Detailed objective functions are explained in Appendix[B\.3](https://arxiv.org/html/2610.27280#A2.SS3)\.

### 5\.5Steering Noise as Planning

Planning objective\.VPFlowinstantiates the cost objective in Equation[2](https://arxiv.org/html/2610.27280#S4.E2)with the reward functionrtr\_\{t\}defined in Equation[1](https://arxiv.org/html/2610.27280#S4.E1)and optimizes the action chunk𝒂t=at:t\+H−1\{\\bm\{a\}\}\_\{t\}=a\_\{t:t\+H\-1\}at stateztz\_\{t\}\. Yet, credit assignment is challenging for sparse reward\. As such, we densify it with the potential\-based reward shaping\([Ng et al\., 1999](https://arxiv.org/html/2610.27280#bib.bib54)\)using imagined rollouts, which yields the reshaped reward as:

δt=rt\+γ​Vϕ​\(z^t\+1,zg\)−Vϕ​\(z^t,zg\)\.\\delta\_\{t\}=r\_\{t\}\+\\gamma V\_\{\\phi\}\(\\hat\{z\}\_\{t\+1\},z\_\{g\}\)\-V\_\{\\phi\}\(\\hat\{z\}\_\{t\},z\_\{g\}\)\.\(9\)We design the planning objective to maximize the\(γ​λ\)\(\\gamma\\lambda\)\-discounted sum of the reshaped reward over the planning horizonHH, whereλ∈\[0,1\]\\lambda\\in\[0,1\]is the trace\-decay coefficient:

A\(zt,𝒂t,zg\)=At≔∑h=0H−1\(γλ\)hδt\+h,s\.t\.z^t\+h=fϕ\(z^t,at:t\+h−1\),z^t≔zt\.A\(z\_\{t\},\{\\bm\{a\}\}\_\{t\},z\_\{g\}\)=A\_\{t\}\\coloneqq\\sum\_\{h=0\}^\{H\-1\}\(\\gamma\\lambda\)^\{h\}\\delta\_\{t\+h\},\\quad\\text\{s\.t\.\}\\quad\\hat\{z\}\_\{t\+h\}=f\_\{\\phi\}\(\\hat\{z\}\_\{t\},a\_\{t:t\+h\-1\}\),\\quad\\hat\{z\}\_\{t\}\\coloneqq z\_\{t\}\.\(10\)
Algorithm 1ε\\varepsilon\-ES: evolutionary search in noise space with the generalized advantage estimator\.Input:state

sts\_\{t\}, goal

gg; chunk length

HH; number of chunks

ncn\_\{c\}; elites

1≤Ne≤N1\\leq N\_\{\\mathrm\{e\}\}\\leq N; population

NN; iterations

K≥0K\\geq 0; mutation scale bounds

σmin,σmax\\sigma\_\{\\min\},\\sigma\_\{\\max\}; decay

ρ\\rho; inverse temperature

β\\beta; execution length

HexecH\_\{\\mathrm\{exec\}\}
Output:first

HexecH\_\{\\mathrm\{exec\}\}actions \(

\[:Hexec\]\[\{:\}H\_\{\\mathrm\{exec\}\}\]denotes slicing\)

1

zt←Eϕ​\(st\)z\_\{t\}\\leftarrow E\_\{\\phi\}\(s\_\{t\}\),

zg←Eϕ​\(g\)z\_\{g\}\\leftarrow E\_\{\\phi\}\(g\)
2function*Dec⁡\(𝛆=\(𝛆0,…,𝛆nc−1\)\)\\operatorname\{Dec\}\(\\bm\{\\varepsilon\}=\(\\bm\{\\varepsilon\}\_\{0\},\\ldots,\\bm\{\\varepsilon\}\_\{n\_\{c\}\-1\}\)\)*

3

z^t←zt\\hat\{z\}\_\{t\}\\leftarrow z\_\{t\}
4for*c=0,…,nc−1c=0,\\ldots,n\_\{c\}\-1*do

5

𝒂c←clip⁡\(πθ​\(z^t\+c​H,zg,𝜺c\),−1,1\)\{\\bm\{a\}\}\_\{c\}\\leftarrow\\operatorname\{clip\}\(\\pi\_\{\\theta\}\(\\hat\{z\}\_\{t\+cH\},z\_\{g\},\\bm\{\\varepsilon\}\_\{c\}\),\-1,1\)
6

z^t\+\(c\+1\)​H←fϕ​\(z^t\+c​H,𝒂c\)\\hat\{z\}\_\{t\+\(c\+1\)H\}\\leftarrow f\_\{\\phi\}\(\\hat\{z\}\_\{t\+cH\},\{\\bm\{a\}\}\_\{c\}\)⊳\\trianglerightEquation[5](https://arxiv.org/html/2610.27280#S5.E5), latent rollouts

7returnconcatenation of

\(𝒂0,…,𝒂nc−1\)\(\{\\bm\{a\}\}\_\{0\},\\ldots,\{\\bm\{a\}\}\_\{n\_\{c\}\-1\}\)
8if*K=0K=0*then

9return

Dec\(𝟎\)\[:Hexec\]\\operatorname\{Dec\}\(\{\\bm\{0\}\}\)\[\{:\}H\_\{\\mathrm\{exec\}\}\]⊳\\trianglerightNo search

10

A0←A⁡\(zt,Dec⁡\(𝟎\),zg\)A\_\{0\}\\leftarrow A\(z\_\{t\},\\operatorname\{Dec\}\(\{\\bm\{0\}\}\),z\_\{g\}\)⊳\\trianglerightEquation[10](https://arxiv.org/html/2610.27280#S5.E10), GAE

11

\(𝜺\(e\),Ae\)←\(𝟎,A0\)\(\\bm\{\\varepsilon\}^\{\(e\)\},A\_\{e\}\)\\leftarrow\(\{\\bm\{0\}\},A\_\{0\}\),

e=1,…,Nee=1,\\ldots,N\_\{\\mathrm\{e\}\}⊳\\trianglerightPolicy initialization at zero

12for*k=0,…,K−1k=0,\\ldots,K\-1*do

13

σk←max⁡\(σmin,σmax​ρk\)\\sigma\_\{k\}\\leftarrow\\max\(\\sigma\_\{\\min\},\\sigma\_\{\\max\}\\rho^\{k\}\)⊳\\trianglerightAnnealed mutation scale

14for*e=1,…,Nee=1,\\ldots,N\_\{\\mathrm\{e\}\}*do

15for*j=1,…,N/Nej=1,\\ldots,N/N\_\{\\mathrm\{e\}\}*do

16

𝜺~\(j\)∼𝒩⁡\(𝜺\(e\),σk2​I\)\\tilde\{\\bm\{\\varepsilon\}\}^\{\(j\)\}\\sim\\mathcal\{N\}\(\\bm\{\\varepsilon\}^\{\(e\)\},\\sigma\_\{k\}^\{2\}I\)⊳\\trianglerightMutate elites

17

A~j←A⁡\(zt,Dec⁡\(𝜺~\(j\)\),zg\)\\tilde\{A\}\_\{j\}\\leftarrow A\(z\_\{t\},\\operatorname\{Dec\}\(\\tilde\{\\bm\{\\varepsilon\}\}^\{\(j\)\}\),z\_\{g\}\)⊳\\trianglerightEvaluate rollouts with GAE

18

j⋆←arg​maxj∈\{1,…,N/Ne\}⁡A~jj^\{\\star\}\\leftarrow\\argmax\_\{j\\in\\\{1,\\dots,N/N\_\{\\mathrm\{e\}\}\\\}\}\\tilde\{A\}\_\{j\}
19if*A~j⋆\>Ae\\tilde\{A\}\_\{j^\{\\star\}\}\>A\_\{e\}*then

20

\(𝜺\(e\),Ae\)←\(𝜺~\(j⋆\),A~j⋆\)\(\\bm\{\\varepsilon\}^\{\(e\)\},A\_\{e\}\)\\leftarrow\(\\tilde\{\\bm\{\\varepsilon\}\}^\{\(j^\{\\star\}\)\},\\tilde\{A\}\_\{j^\{\\star\}\}\)⊳\\trianglerightGreedy improvement

21

we←exp⁡\(β​Ae\)/∑j=1Neexp⁡\(β​Aj\)w\_\{e\}\\leftarrow\\exp\(\\beta A\_\{e\}\)/\\sum\_\{j=1\}^\{N\_\{\\mathrm\{e\}\}\}\\exp\(\\beta A\_\{j\}\),

e=1,…,Nee=1,\\ldots,N\_\{\\mathrm\{e\}\}
22

e⋆∼Categorical⁡\(\(we\)e=1Ne\)e^\{\\star\}\\sim\\operatorname\{Categorical\}\(\(w\_\{e\}\)\_\{e=1\}^\{N\_\{\\mathrm\{e\}\}\}\)⊳\\trianglerightSample elite

23return

Dec\(𝜺\(e⋆\)\)\[:Hexec\]\\operatorname\{Dec\}\(\\bm\{\\varepsilon\}^\{\(e^\{\\star\}\)\}\)\[\{:\}H\_\{\\mathrm\{exec\}\}\]⊳\\trianglerightReceding\-horizon execution

It corresponds to the generalized advantage estimator \(GAE\)\([Schulman et al\., 2016](https://arxiv.org/html/2610.27280#bib.bib71)\)of horizonHH\. The scoreAAmeasures predicted progress towards the goal relative to the current state\. We discuss this planning objective in Appendix\.

Noise space steering\.Planning has been widely conducted by directly optimizing*open\-loop*actions on an unconstrained support\. For long\-horizon rollouts, actions tend to drift away from the data distribution, making the planning objective unreliable\. One of the key benefits of a generative policy is that after training, its noise space can be optimized to search for better actions while still being close to the training distribution\([Yang et al\., 2024](https://arxiv.org/html/2610.27280#bib.bib87);[Wagenmaker et al\., 2025](https://arxiv.org/html/2610.27280#bib.bib81)\)\. We adopt this strategy to steer the noise to optimize Equation[10](https://arxiv.org/html/2610.27280#S5.E10)that uses the imagined states unrolled from the learned dynamics\.

ε\\varepsilon\-ES: evolution strategy in noise space\.CEM planners used in model\-based methods maintain an elite set of actions, from which an online mean and standard deviation are updated\([Janner et al\., 2019](https://arxiv.org/html/2610.27280#bib.bib36);[Hansen et al\., 2022](https://arxiv.org/html/2610.27280#bib.bib33);[Maes et al\., 2026b](https://arxiv.org/html/2610.27280#bib.bib52)\)\. However, this design choice does not directly translate to the noise space, as their distribution differs\. Inspired by a previous gradient\-free optimizer for diffusion noise\([Yang et al\., 2024](https://arxiv.org/html/2610.27280#bib.bib87)\), we instead develop an evolution strategy that mutates samples of each elite in noise space and greedily keeps the ones that score higher by the objective in Equation[10](https://arxiv.org/html/2610.27280#S5.E10)\. This allows the policy prior learned during training to improve itself by adding more compute at test time without weight updates\. We present the search process, which runs forKKiterations, in Algorithm[1](https://arxiv.org/html/2610.27280#algorithm1)with more details in Appendix[B\.6](https://arxiv.org/html/2610.27280#A2.SS6)\.

### 5\.6

0pt1\.8ex plus 0\.4ex minus 0\.2ex0\.8ex plus 0\.2ex

## 6Experiments

We aim to address four questions in this section:Q1: How doesVPFlowcompare with model\-free baselines and model\-based methods that similarly plan with latent world models for offline goal\-reaching tasks?Q2: Are state\-predictive representations useful for planning, or do the value\-predictive ones play a more essential role?Q3: Does test\-time planning help, and does the search space matter more than the search algorithm?Q4: How does the learned latent actually behave during rollouts?

Setup\.We use the official OGBench datasets and tasks\([Park et al\., 2025a](https://arxiv.org/html/2610.27280#bib.bib60)\): 17 state\-based datasets and 13 pixel\-based datasets\. The metric is the goal\-reaching success rate \(%\\%,↑\\uparrow\), averaged over the five tasks within each dataset\. Model\-free baselines are the non\-hierarchical methods reported by OGBench: goal\-conditioned behavioral cloning \(GCBC\)\([Ghosh et al\., 2021](https://arxiv.org/html/2610.27280#bib.bib25)\), goal\-conditioned implicit value learning \(GCIVL\) and goal\-conditioned implicit Q\-learning \(GCIQL\)\([Kostrikov et al\., 2022](https://arxiv.org/html/2610.27280#bib.bib41)\), quasi\-metric RL \(QRL\)\([Wang et al\., 2023](https://arxiv.org/html/2610.27280#bib.bib83)\), and contrastive RL \(CRL\)\([Eysenbach et al\., 2022](https://arxiv.org/html/2610.27280#bib.bib20)\)\. Model\-based baselines are the latent plannersLeWorldModel \(LeWM\)\([Maes et al\., 2026b](https://arxiv.org/html/2610.27280#bib.bib52)\),PLDM\([Sobal et al\., 2025](https://arxiv.org/html/2610.27280#bib.bib73)\), andTD\-MPC2\([Hansen et al\., 2024](https://arxiv.org/html/2610.27280#bib.bib34)\)\. Implementation details are in Appendix[B](https://arxiv.org/html/2610.27280#A2), and hyperparameters are in Appendix\.

Table 1:Success rate \(%\\%,↑\\uparrow\) on OGBench compared against model\-free baselines\.Entries are mean success rates over the datasets in each environment, with the number of datasets in parentheses\. The best and second best results in each row are in bold and underlined, respectively\.InputTask TypeEnvironmentGCBCGCIVLGCIQLQRLCRLVPFlowStateLocomotionPointMaze\(4 datasets\)16\.038\.227\.860\.029\.887\.8AntMaze\(4 datasets\)19\.831\.835\.053\.061\.868\.0ManipulationCube\(4 datasets\)2\.022\.527\.81\.58\.242\.2Scene\(1 dataset\)5\.042\.051\.05\.019\.078\.0Puzzle\(4 datasets\)0\.59\.036\.80\.22\.051\.0Overall\(17 datasets\)9\.326\.432\.927\.325\.163\.2PixelLocomotionAntMaze\(4 datasets\)5\.09\.05\.21\.568\.261\.0ManipulationCube\(4 datasets\)7\.221\.013\.216\.813\.534\.8Scene\(1 dataset\)12\.025\.012\.010\.011\.033\.0Puzzle\(4 datasets\)4\.228\.26\.50\.24\.818\.3Overall\(13 datasets\)6\.019\.88\.66\.527\.537\.6### 6\.1Q1: How DoesVPFlowPerform for Offline Goal Reaching?

Comparison against model\-free baselines\.Table[1](https://arxiv.org/html/2610.27280#S6.T1)reports the results grouped by environment \(full numerics in Appendix\)\. Averaged over the1717state datasets,VPFlownearly doubles the strongest baseline \(63\.263\.2vs\.32\.932\.9for GCIQL\) and leads every environment group\. The gains concentrate on manipulation and shrink on navigation \(AntMaze68\.068\.0vs\.61\.861\.8\), where a reactive policy may already solve much of the task\. This suggests that model\-based planning can be more suitable for contact\-rich tasks, where test\-time compute translates more directly to success\. On pixel\-based datasets, the gains become relatively modest \(37\.637\.6vs\.27\.527\.5for CRL\) asVPFlowtrails behind in a long\-horizon Visual AntMaze dataset\.

Figure 4:Success rate of planners with latent world modelson*the only*pixel dataset where the baselines do not completely fail\.Comparison against latent planning methods\.To our surprise, on 12 of the 13 pixel datasets,*all three model\-based baselines score zero*, so Figure[4](https://arxiv.org/html/2610.27280#S6.F4)reports the only dataset on which any one of them achieves a non\-zero success rate\. All three train their latent to predict a future state and then plan over action sequences\. On diverse play data, neither practice stands out as the optimal design choice\. We next reveal more about the possible failure modes on these two patterns\.

### 6\.2Q2: Value\-Predictive or State\-Predictive Latents?

This experiment aims to verify which representations are more favorable for planning\. The default model trains the rollout with the value\-prediction lossℒVP\\mathcal\{L\}\_\{\\mathrm\{VP\}\}\(Equation[6](https://arxiv.org/html/2610.27280#S5.E6)\); the state\-predictive model trains it with the latent\-prediction lossℒLP\\mathcal\{L\}\_\{\\mathrm\{LP\}\}\(Equation[13](https://arxiv.org/html/2610.27280#A2.E13)\), which matches the predicted latents to the encoded states\. Other configurations stay the same\. Figure[5](https://arxiv.org/html/2610.27280#S6.F5)tracks the success rate and the next\-latent prediction error during training on four datasets\.

\(\(a\)\)scene\-play\-v0\(\(b\)\)visual\-scene\-play\-v0\(\(c\)\)visual\-cube\-single\-
play\-v0\(\(d\)\)visual\-antmaze\-large\-
navigate\-v0
Figure 5:Comparison of value\-predictive and state\-predictive representations\.Success rate \(left axis\) and next\-latent prediction error \(right axis, log scale\) during training, for value\-predictive \(solid\) and state\-predictive \(dashed\) models\. The error is the mean Euclidean distance‖z^t\+h−zt\+h‖2\\\|\\hat\{z\}\_\{t\+h\}\-z\_\{t\+h\}\\\|\_\{2\}between the predicted and encoded next latents\. On state and pixel inputs, the value\-predictive representation achieves higher success even when its state prediction error is larger by orders of magnitude\.Lower state prediction error, worse planning\.The state\-predictive controller reduces the next\-latent error by one to eight orders of magnitude, yet its success does not improve on any of the four datasets and falls most sharply onscene\-play\(78\.4%78\.4\\%to28\.0%28\.0\\%, Figure[5\(a\)](https://arxiv.org/html/2610.27280#S6.F5.sf1)\) andvisual\-cube\-single\(79\.2%79\.2\\%to0\.0%0\.0\\%, Figure[5\(c\)](https://arxiv.org/html/2610.27280#S6.F5.sf3)\)\. A model that predicts its own next latent almost perfectly is therefore not necessarily a model that plans well\. Predicting the value is more essential for planning\.

The failure mode is collapse, and value prediction prevents it\.Onvisual\-cube\-single\(Figure[5\(c\)](https://arxiv.org/html/2610.27280#S6.F5.sf3)\), the latent prediction term collapses to near zero: the encoder finds a trivially predictable latent and the agent never reaches the goal, the degeneracy that self\-predictive objectives prevent through explicit anti\-collapse techniques, none of which our architecture has\. In contrast, value prediction in Equation[6](https://arxiv.org/html/2610.27280#S5.E6)promotes temporally ordered latents that implicitly prevent collapse\. Appendixprovides ablations on explicit regularization over latent prediction\.

### 6\.3Q3: Does Steering Help, and Where Should It Search?

PlannerSpaceLoco\.Manip\.Allw/o steering–57\.58\.328\.0w/𝒜\\mathcal\{A\}\-CEMaction22\.61\.610\.0w/ε\\varepsilon\-CEMnoise53\.231\.540\.2VPFlownoise72\.238\.752\.1Table 2:Ablations on steering algorithm and search space\.Success rate \(%\\%,↑\\uparrow\) averaged over the 12 locomotion \(Loco\.\), 18 manipulation \(Manip\.\), and all 30 datasets\. Action\-space planning worsens the success rate relative to direct action output\.
Figure 6:Success rate \(%\\%\) against the number of steering iterationsKKoncube\-double\-play\-v0,scene\-play\-v0, andvisual\-scene\-play\-v0\. More iterations generally steer the policy towards a higher success rate\.K=0K=0means the reactive policy without steering\.
Test\-time steering enables policy improvement with imagination\.Without steering \(K=0K\{=\}0\),VPFlowexecutes the policy’s actions directly and achieves inferior performance \(Table[2](https://arxiv.org/html/2610.27280#S6.T2)\)\. Extra test\-time compute, by updating the noise greedily towards maximal advantage in the imagined future, raises success asKKgrows \(Figure[6](https://arxiv.org/html/2610.27280#S6.F6)\)\. This supports Equation[10](https://arxiv.org/html/2610.27280#S5.E10)as an effective objective enabling policy improvement via search without external signals, and planning with more compute can be made possible with imperfect world models\.

Noise space proves more effective than action space for planning\.Replacingε\\varepsilon\-ESwithε\\varepsilon\-CEMthat updates noise with standard CEM lowers the success rate\. This is expected, as noise is updated by advantage on the decoded actions rather than the noise itself\. More notably, the success rate of CEM in action space,𝒜\\mathcal\{A\}\-CEM, is even worse than the direct output \(10\.010\.0vs\.28\.028\.0\) and not even close to its noise\-space counterpart \(10\.010\.0vs\.40\.240\.2\)\. This can be attributed to the advantage exploitation of off\-manifold actions, and it explains why the action\-space planning design choices in PLDM and LeWM\([Sobal et al\., 2025](https://arxiv.org/html/2610.27280#bib.bib73);[Maes et al\., 2026b](https://arxiv.org/html/2610.27280#bib.bib52)\)could be suboptimal with offline data\. The*closed\-loop*planning in the noise space of a generative policy prevents this in that its decoded actions are trained to be in the support of the data distribution\.

### 6\.4Q4: How Do the Latents Behave in Rollouts?

![Refer to caption](https://arxiv.org/html/2610.27280v1/tsne-crop.png)\(\(a\)\)visual\-antmaze\-large\-navigate\-v0Task 1
![Refer to caption](https://arxiv.org/html/2610.27280v1/tsne2-crop.png)\(\(b\)\)visual\-antmaze\-large\-navigate\-v0Task 3

Figure 7:t\-SNE visualization of latent trajectories from successful Visual AntMaze episodes\.The solid circle is the encoded current state, and the dashed circles are the imagined latents via dynamics for that step\. Under value\-predictive training, the imagined latents leave the trajectory manifold at test time, but the agent can still reach the goal\.Figure[7](https://arxiv.org/html/2610.27280#S6.F7)depicts the geometry of imagined rollouts on Visual AntMaze to demonstrate the behavior of imperfect yet planning\-effective learned world models\. Under value\-predictive training, the dynamics model evolves latents to where the correct goal\-reaching value is forecasted by the value function, without being constrained to match the corresponding embeddings from the encoder\. This helps explain why the value\-predictive controller plans effectively despite the large next\-latent prediction error demonstrated in Section[6\.2](https://arxiv.org/html/2610.27280#S6.SS2)\. It also suggests how value prediction harnesses exploratory play data more effectively: closed\-form value regression structures the latent space by temporal proximity to the goal, allowing it to leave out irrelevant routes and discover a latent geometry suited to reaching the goal\. In contrast, a state\-predictive world model is trained to faithfully fit latent transitions in the dataset, which may limit its generalization to unseen goals\.

To examine this value\-equivalent behavior more quantitatively, Figure[8](https://arxiv.org/html/2610.27280#S6.F8)compares agents trained with value predictionℒVP\\mathcal\{L\}\_\{\\mathrm\{VP\}\}and latent predictionℒLP\\mathcal\{L\}\_\{\\mathrm\{LP\}\}on successful episodes fromvisual\-scene\-play\-v0\. Starting from the same encoded state, each agent unrolls its own executed actions in both the real environment and the learned world model\. Although VP exhibits larger relative latent error across the 16\-step rollout, its value gap is typically smaller than LP’s \(Figure[8\(a\)](https://arxiv.org/html/2610.27280#S6.F8.sf1)\)\. The value field provides a microscopic illustration: while VP’s observed and imagined latents are further apart in Euclidean space, they lie closer in the value space \(Figure[8\(b\)](https://arxiv.org/html/2610.27280#S6.F8.sf2)\)\. The value\-predictive latent also moves towards higher value, reflecting closer proximity to the goal\.

\(\(a\)\)Relative latent error‖z^h−zh‖2/‖zh‖2\\\|\\hat\{z\}\_\{h\}\-z\_\{h\}\\\|\_\{2\}/\\\|z\_\{h\}\\\|\_\{2\}and value error\|V⁡\(z^h,zg\)−V⁡\(zh,zg\)\|\|V\(\\hat\{z\}\_\{h\},z\_\{g\}\)\-V\(z\_\{h\},z\_\{g\}\)\|along rollout horizons with mean and standard deviation over 4 evaluation seeds\.\(\(b\)\)Value fields and level sets of each agent’s learned critic measured on the current, observed, and predicted next latents\.
Figure 8:Rollouts in the environment and learned world models, with their value fields\.With value prediction, the agent maintains a lower value error and reaches the goal successfully despite substantially larger latent prediction error\.Taken together, these results demonstrate that accurate latent\-state prediction is not strictly necessary for successful planning\. Imperfect world models can remain effective and even yield superior performance \(Figure\), provided that they preserve value equivalence\. The evidence that the agent can successfully reach the goal despite substantial latent prediction error suggests that latent error is an unreliable indicator of planning success; instead, value prediction should serve as a more informative metric\.

## 7Conclusions

We have challenged the pervasive assumption that planning requires a state\-predictive world model\.VPFlowtrains its observation encoder and latent dynamics solely via closed\-form Monte Carlo return regression—a supervision signal naturally available in sparse\-reward offline goal reaching—and plans at test time by steering a distilled one\-step flow policy within its noise space to maximize imagined generalized advantage\. Across 30 state and pixel datasets on OGBench,VPFlowsubstantially improves upon model\-free baselines and latent\-predictive planners despite much higher next\-latent prediction error\. In contrast, enforcing state prediction closes that error gap at the cost of degraded planning performance\. Moreover, test\-time noise steering reliably elevates the reactive policy prior, demonstrating that searching within an in\-support noise space matters more than the specific optimization algorithm\. More broadly, these findings indicate that effective latent planning favors value\-predictive representations for world models and an in\-distribution search space for planning, rather than a faithful state simulator\. We discuss limitations and future directions in Appendix\.

## References

- Andrychowicz et al\. \(2017\)Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba\.Hindsight experience replay\.In*Advances in Neural Information Processing Systems*, volume 30, 2017\.
- Argenson and Dulac\-Arnold \(2021\)Arthur Argenson and Gabriel Dulac\-Arnold\.Model\-based offline planning\.In*International Conference on Learning Representations*, 2021\.
- Assran et al\. \(2025\)Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et al\.V\-JEPA 2: Self\-supervised video models enable understanding, prediction and planning\.*arXiv preprint arXiv:2506\.09985*, 2025\.
- Ba et al\. \(2016\)Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton\.Layer normalization\.*arXiv preprint arXiv:1607\.06450*, 2016\.
- Bagatella et al\. \(2026\)Marco Bagatella, Matteo Pirotta, Ahmed Touati, Alessandro Lazaric, and Andrea Tirinzoni\.TD\-JEPA: Latent\-predictive representations for zero\-shot reinforcement learning\.In*International Conference on Learning Representations*, 2026\.
- Balestriero and LeCun \(2025\)Randall Balestriero and Yann LeCun\.LeJEPA: Provable and scalable self\-supervised learning without the heuristics\.*arXiv preprint arXiv:2511\.08544*, 2025\.
- Bardes et al\. \(2022\)Adrien Bardes, Jean Ponce, and Yann LeCun\.VICReg: Variance\-invariance\-covariance regularization for self\-supervised learning\.In*International Conference on Learning Representations*, 2022\.
- Barreto et al\. \(2017\)André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado Van Hasselt, and David Silver\.Successor features for transfer in reinforcement learning\.*Advances in Neural Information Processing Systems*, 30, 2017\.
- Barreto et al\. \(2018\)Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos\.Transfer in deep reinforcement learning using successor features and generalised policy improvement\.In*International Conference on Machine Learning*, pages 501–510, 2018\.
- Black et al\. \(2025\)Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al\.π0\\pi\_\{0\}: A vision\-language\-action flow model for general robot control\.In*Robotics: Science and Systems*, 2025\.
- Blier et al\. \(2021\)Léonard Blier, Corentin Tallec, and Yann Ollivier\.Learning successor states and goal\-dependent values: A mathematical viewpoint\.*arXiv preprint arXiv:2101\.07123*, 2021\.
- Borsa et al\. \(2019\)Diana Borsa, Andre Barreto, John Quan, Daniel J\. Mankowitz, Hado van Hasselt, Remi Munos, David Silver, and Tom Schaul\.Universal successor features approximators\.In*International Conference on Learning Representations*, 2019\.
- Bradbury et al\. \(2018\)James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Yash Katariya, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman\-Milne, and Qiao Zhang\.JAX: composable transformations of Python\+NumPy programs, 2018\.[http://github\.com/jax\-ml/jax](http://github.com/jax-ml/jax)\.
- Carvalho et al\. \(2024\)Wilka Carvalho, Momchil S\. Tomov, William de Cothi, Caswell Barry, and Samuel J\. Gershman\.Predictive representations: Building blocks of intelligence\.*Neural Computation*, 36\(11\):2225–2298, 2024\.[10\.1162/neco\_a\_01705](https://doi.org/10.1162/neco_a_01705)\.
- Dawid and LeCun \(2024\)Anna Dawid and Yann LeCun\.Introduction to latent variable energy\-based models: a path toward autonomous machine intelligence\.*Journal of Statistical Mechanics: Theory and Experiment*, 2024\(10\):104011, 2024\.[10\.1088/1742\-5468/ad292b](https://doi.org/10.1088/1742-5468/ad292b)\.
- Dayan \(1993\)Peter Dayan\.Improving generalization for temporal difference learning: The successor representation\.*Neural Computation*, 5\(4\):613–624, 1993\.[10\.1162/neco\.1993\.5\.4\.613](https://doi.org/10.1162/neco.1993.5.4.613)\.
- Deisenroth and Rasmussen \(2011\)Marc Deisenroth and Carl E Rasmussen\.PILCO: A model\-based and data\-efficient approach to policy search\.In*International Conference on Machine Learning*, pages 465–472, 2011\.
- Du et al\. \(2020\)Yilun Du, Toru Lin, and Igor Mordatch\.Model\-based planning with energy\-based models\.In*Conference on Robot Learning*, volume 100, pages 374–383\. PMLR, 2020\.
- Espeholt et al\. \(2018\)Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al\.IMPALA: Scalable distributed deep\-RL with importance weighted actor\-learner architectures\.In*International Conference on Machine Learning*, pages 1407–1416, 2018\.
- Eysenbach et al\. \(2022\)Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R Salakhutdinov\.Contrastive learning as goal\-conditioned reinforcement learning\.*Advances in Neural Information Processing Systems*, 35:35603–35620, 2022\.
- Finn and Levine \(2017\)Chelsea Finn and Sergey Levine\.Deep visual foresight for planning robot motion\.In*IEEE International Conference on Robotics and Automation*, pages 2786–2793\. IEEE, 2017\.
- Fujimoto and Gu \(2021\)Scott Fujimoto and Shixiang Shane Gu\.A minimalist approach to offline reinforcement learning\.*Advances in Neural Information Processing Systems*, 34:20132–20145, 2021\.
- Fujimoto et al\. \(2019\)Scott Fujimoto, David Meger, and Doina Precup\.Off\-policy deep reinforcement learning without exploration\.In*International Conference on Machine Learning*, pages 2052–2062, 2019\.
- Fujimoto et al\. \(2025\)Scott Fujimoto, Pierluca D'Oro, Amy Zhang, Yuandong Tian, and Michael Rabbat\.Towards general\-purpose model\-free reinforcement learning\.In*International Conference on Learning Representations*, 2025\.
- Ghosh et al\. \(2021\)Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Manon Devin, Benjamin Eysenbach, and Sergey Levine\.Learning to reach goals via iterated supervised learning\.In*International Conference on Learning Representations*, 2021\.
- Grimm et al\. \(2020\)Christopher Grimm, André Barreto, Satinder Singh, and David Silver\.The value equivalence principle for model\-based reinforcement learning\.*Advances in Neural Information Processing Systems*, 33:5541–5552, 2020\.
- Guo et al\. \(2022\)Zhaohan Guo, Shantanu Thakoor, Miruna Pîslar, Bernardo Avila Pires, Florent Altché, Corentin Tallec, Alaa Saade, Daniele Calandriello, Jean\-Bastien Grill, Yunhao Tang, et al\.BYOL\-Explore: Exploration by bootstrapped prediction\.*Advances in Neural Information Processing Systems*, 35:31855–31870, 2022\.
- Ha and Schmidhuber \(2018\)David Ha and Jürgen Schmidhuber\.Recurrent world models facilitate policy evolution\.*Advances in neural information processing systems*, 31, 2018\.
- Hafner et al\. \(2019\)Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson\.Learning latent dynamics for planning from pixels\.In*International Conference on Machine Learning*, pages 2555–2565, 2019\.
- Hafner et al\. \(2020\)Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi\.Dream to control: Learning behaviors by latent imagination\.In*International Conference on Learning Representations*, 2020\.
- Hafner et al\. \(2025a\)Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap\.Mastering diverse control tasks through world models\.*Nature*, 640\(8059\):647–653, 2025a\.[10\.1038/s41586\-025\-08744\-2](https://doi.org/10.1038/s41586-025-08744-2)\.
- Hafner et al\. \(2025b\)Danijar Hafner, Wilson Yan, and Timothy Lillicrap\.Training agents inside of scalable world models\.*arXiv preprint arXiv:2509\.24527*, 2025b\.
- Hansen et al\. \(2022\)Nicklas Hansen, Hao Su, and Xiaolong Wang\.Temporal difference learning for model predictive control\.In*International Conference on Machine Learning*, volume 162, pages 8387–8406\. PMLR, 2022\.
- Hansen et al\. \(2024\)Nicklas Hansen, Hao Su, and Xiaolong Wang\.TD\-MPC2: Scalable, robust world models for continuous control\.In*International Conference on Learning Representations*, 2024\.
- Hendrycks and Gimpel \(2016\)Dan Hendrycks and Kevin Gimpel\.Gaussian error linear units \(GELUs\)\.*arXiv preprint arXiv:1606\.08415*, 2016\.
- Janner et al\. \(2019\)Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine\.When to trust your model: Model\-based policy optimization\.*Advances in Neural Information Processing Systems*, 32, 2019\.
- Kaelbling \(1993\)Leslie Pack Kaelbling\.Learning to achieve goals\.In*International Joint Conference on Artificial Intelligence*, pages 1094–8, 1993\.
- Kidambi et al\. \(2020\)Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims\.MOReL: Model\-based offline reinforcement learning\.*Advances in Neural Information Processing Systems*, 33:21810–21823, 2020\.
- Kidger and Garcia \(2021\)Patrick Kidger and Cristian Garcia\.Equinox: neural networks in JAX via callable PyTrees and filtered transformations\.*arXiv preprint arXiv:2111\.00254*, 2021\.
- Kingma and Ba \(2015\)Diederik P\. Kingma and Jimmy Ba\.Adam: A method for stochastic optimization\.In*International Conference on Learning Representations*, 2015\.
- Kostrikov et al\. \(2022\)Ilya Kostrikov, Ashvin Nair, and Sergey Levine\.Offline reinforcement learning with implicit Q\-learning\.In*International Conference on Learning Representations*, 2022\.
- Kumar et al\. \(2020\)Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine\.Conservative Q\-learning for offline reinforcement learning\.*Advances in Neural Information Processing Systems*, 33:1179–1191, 2020\.
- LeCun \(2022\)Yann LeCun\.A path towards autonomous machine intelligence\.OpenReview preprint, 2022\.Version 0\.9\.2\.
- Levine et al\. \(2020\)Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu\.Offline reinforcement learning: Tutorial, review, and perspectives on open problems\.*arXiv preprint arXiv:2005\.01643*, 2020\.
- Li et al\. \(2022\)Jinning Li, Chen Tang, Masayoshi Tomizuka, and Wei Zhan\.Hierarchical planning through goal\-conditioned offline reinforcement learning\.*IEEE Robotics and Automation Letters*, 7\(4\):10216–10223, 2022\.[10\.1109/LRA\.2022\.3190100](https://doi.org/10.1109/LRA.2022.3190100)\.
- Li et al\. \(2025\)Qiyang Li, Zhiyuan \(Paul\) Zhou, and Sergey Levine\.Reinforcement learning with action chunking\.In*Advances in Neural Information Processing Systems*, volume 38, pages 55518–55553, 2025\.[10\.52202/085713\-1856](https://doi.org/10.52202/085713-1856)\.
- Lipman et al\. \(2023\)Yaron Lipman, Ricky T\. Q\. Chen, Heli Ben\-Hamu, Maximilian Nickel, and Matthew Le\.Flow matching for generative modeling\.In*International Conference on Learning Representations*, 2023\.
- Littman and Sutton \(2001\)Michael Littman and Richard S\. Sutton\.Predictive representations of state\.*Advances in Neural Information Processing Systems*, 14, 2001\.
- Lynch et al\. \(2020\)Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet\.Learning latent plans from play\.In*Conference on Robot Learning*, pages 1113–1132, 2020\.
- M\. Moerland et al\. \(2023\)Thomas M\. Moerland, Joost Broekens, Aske Plaat, and Catholijn M\. Jonker\.Model\-based reinforcement learning: A survey\.*Foundations and Trends in Machine Learning*, 16\(1\):1–118, 2023\.
- Maes et al\. \(2026a\)Lucas Maes, Quentin Le Lidec, Luiz Facury, Nassim Massaudi, Ayush Chaurasia, Francesco Capuano, Richard Gao, Taj Gillin, Dan Haramati, Damien Scieur, Yann LeCun, and Randall Balestriero\.stable\-worldmodel: A platform for reproducible world modeling research and evaluation, 2026a\.
- Maes et al\. \(2026b\)Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann LeCun, and Randall Balestriero\.LeWorldModel: Stable end\-to\-end joint\-embedding predictive architecture from pixels\.*arXiv preprint arXiv:2603\.19312*, 2026b\.
- Myers et al\. \(2025\)Vivek Myers, Bill Zheng, Benjamin Eysenbach, and Sergey Levine\.Offline goal\-conditioned reinforcement learning with quasimetric representations\.In*Advances in Neural Information Processing Systems*, 2025\.[10\.52202/085713\-0661](https://doi.org/10.52202/085713-0661)\.
- Ng et al\. \(1999\)Andrew Y Ng, Daishi Harada, and Stuart Russell\.Policy invariance under reward transformations: Theory and application to reward shaping\.In*International Conference on Machine Learning*, pages 278–287, 1999\.
- Ni et al\. \(2024\)Tianwei Ni, Benjamin Eysenbach, Erfan Seyedsalehi, Michel Ma, Clement Gehring, Aditya Mahajan, and Pierre\-Luc Bacon\.Bridging state and history representations: Understanding self\-predictive RL\.In*International Conference on Learning Representations*, 2024\.
- Oh et al\. \(2017\)Junhyuk Oh, Satinder Singh, and Honglak Lee\.Value prediction network\.*Advances in Neural Information Processing Systems*, 30, 2017\.
- Park et al\. \(2026\)Kwanyoung Park, Seohong Park, Youngwoon Lee, and Sergey Levine\.Scalable offline model\-based RL with action chunks\.In*International Conference on Learning Representations*, 2026\.
- Park et al\. \(2023\)Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine\.HIQL: Offline goal\-conditioned RL with latent states as actions\.*Advances in Neural Information Processing Systems*, 36:34866–34891, 2023\.
- Park et al\. \(2024\)Seohong Park, Tobias Kreiman, and Sergey Levine\.Foundation policies with Hilbert representations\.In*International Conference on Machine Learning*, 2024\.
- Park et al\. \(2025a\)Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine\.OGBench: Benchmarking offline goal\-conditioned RL\.In*International Conference on Learning Representations*, 2025a\.
- Park et al\. \(2025b\)Seohong Park, Kevin Frans, Deepinder Mann, Benjamin Eysenbach, Aviral Kumar, and Sergey Levine\.Horizon reduction makes RL scalable\.*Advances in Neural Information Processing Systems*, 38:8350–8389, 2025b\.
- Park et al\. \(2025c\)Seohong Park, Qiyang Li, and Sergey Levine\.Flow Q\-learning\.In*International Conference on Machine Learning*, 2025c\.
- Patil et al\. \(2026\)Omkar Patil, Ondrej Biza, Thomas Weng, Karl Schmeckpeper, Wil Thomason, Xiaohan Zhang, Kausik Sivakumar, Robin Walters, Nakul Gopalan, Sebastian Castro, et al\.You’ve got a golden ticket: Improving generative robot policies with a single noise vector\.*arXiv preprint arXiv:2603\.15757*, 2026\.
- Peng et al\. \(2019\)Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine\.Advantage\-weighted regression: Simple and scalable off\-policy reinforcement learning\.*arXiv preprint arXiv:1910\.00177*, 2019\.
- Physical Intelligence et al\. \(2025\)Physical Intelligence, Ali Amin, Raichelle Aniceto, Ashwin Balakrishna, Kevin Black, Ken Conley, Grace Connors, James Darpinian, Karan Dhabalia, Jared DiCarlo, et al\.π0\.6∗\\pi^\{\*\}\_\{0\.6\}: A VLA that learns from experience\.*arXiv preprint arXiv:2511\.14759*, 2025\.
- Pirotta et al\. \(2024\)Matteo Pirotta, Andrea Tirinzoni, Ahmed Touati, Alessandro Lazaric, and Yann Ollivier\.Fast imitation via behavior foundation models\.In*International Conference on Learning Representations*, 2024\.
- Rubinstein \(1999\)Reuven Rubinstein\.The cross\-entropy method for combinatorial and continuous optimization\.*Methodology and Computing in Applied Probability*, 1\(2\):127–190, 1999\.[10\.1023/A:1010091220143](https://doi.org/10.1023/A:1010091220143)\.
- Salimans et al\. \(2017\)Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever\.Evolution strategies as a scalable alternative to reinforcement learning\.*arXiv preprint arXiv:1703\.03864*, 2017\.
- Schaul et al\. \(2015\)Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver\.Universal value function approximators\.In*International Conference on Machine Learning*, pages 1312–1320, 2015\.
- Schrittwieser et al\. \(2020\)Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al\.Mastering Atari, Go, chess and shogi by planning with a learned model\.*Nature*, 588\(7839\):604–609, 2020\.[10\.1038/s41586\-020\-03051\-4](https://doi.org/10.1038/s41586-020-03051-4)\.
- Schulman et al\. \(2016\)John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel\.High\-dimensional continuous control using generalized advantage estimation\.In*International Conference on Learning Representations*, 2016\.
- Schwarzer et al\. \(2021\)Max Schwarzer, Ankesh Anand, Rishab Goel, R\. Devon Hjelm, Aaron Courville, and Philip Bachman\.Data\-efficient reinforcement learning with self\-predictive representations\.In*International Conference on Learning Representations*, 2021\.
- Sobal et al\. \(2025\)Uladzislau Sobal, Wancong Zhang, Kyunghyun Cho, Randall Balestriero, Tim G\. J\. Rudner, and Yann LeCun\.Learning from reward\-free offline data: A case for planning with latent dynamics models\.In*Advances in Neural Information Processing Systems*, 2025\.
- Su et al\. \(2024\)Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu\.RoFormer: Enhanced transformer with rotary position embedding\.*Neurocomputing*, 568:127063, 2024\.
- Sutton \(1991\)Richard S\. Sutton\.Dyna, an integrated architecture for learning, planning, and reacting\.*ACM SIGART Bulletin*, 2\(4\):160–163, 1991\.[10\.1145/122344\.122377](https://doi.org/10.1145/122344.122377)\.
- Tang et al\. \(2023\)Yunhao Tang, Zhaohan Daniel Guo, Pierre Harvey Richemond, Bernardo Avila Pires, Yash Chandak, Rémi Munos, Mark Rowland, Mohammad Gheshlaghi Azar, Charline Le Lan, Clare Lyle, et al\.Understanding self\-predictive learning for reinforcement learning\.In*International Conference on Machine Learning*, pages 33632–33656, 2023\.
- Tian et al\. \(2021\)Stephen Tian, Suraj Nair, Frederik Ebert, Sudeep Dasari, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine\.Model\-based visual planning with self\-supervised functional distances\.In*International Conference on Learning Representations*, 2021\.
- Touati and Ollivier \(2021\)Ahmed Touati and Yann Ollivier\.Learning one representation to optimize all rewards\.*Advances in Neural Information Processing Systems*, 34:13–23, 2021\.
- Van Assel et al\. \(2025\)Hugues Van Assel, Mark Ibrahim, Tommaso Biancalani, Aviv Regev, and Randall Balestriero\.Joint\-embedding vs reconstruction: Provable benefits of latent space prediction for self\-supervised learning\.In*Advances in Neural Information Processing Systems*, 2025\.
- Vaswani et al\. \(2017\)Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin\.Attention is all you need\.In*Advances in Neural Information Processing Systems*, volume 30, 2017\.
- Wagenmaker et al\. \(2025\)Andrew Wagenmaker, Yunchu Zhang, Mitsuhiko Nakamoto, Seohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, and Sergey Levine\.Steering your diffusion policy with latent space reinforcement learning\.In*Conference on Robot Learning*, 2025\.
- Wang et al\. \(2024\)Mianchu Wang, Rui Yang, Xi Chen, Hao Sun, Meng Fang, and Giovanni Montana\.GOPlan: Goal\-conditioned offline reinforcement learning by planning with learned models\.*Transactions on Machine Learning Research*, 2024\.ISSN 2835\-8856\.
- Wang et al\. \(2023\)Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang\.Optimal goal\-reaching reinforcement learning via quasimetric learning\.In*International Conference on Machine Learning*, pages 36411–36430, 2023\.
- Watter et al\. \(2015\)Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller\.Embed to control: A locally linear latent dynamics model for control from raw images\.*Advances in Neural Information Processing Systems*, 28, 2015\.
- Williams et al\. \(2017\)Grady Williams, Andrew Aldrich, and Evangelos A\. Theodorou\.Model predictive path integral control: From theory to parallel computation\.*Journal of Guidance, Control, and Dynamics*, 40\(2\):344–357, 2017\.[10\.2514/1\.G001921](https://doi.org/10.2514/1.G001921)\.
- Wu et al\. \(2019\)Yifan Wu, George Tucker, and Ofir Nachum\.Behavior regularized offline reinforcement learning\.*arXiv preprint arXiv:1911\.11361*, 2019\.
- Yang et al\. \(2024\)Brian Yang, Huangyuan Su, Nikolaos Gkanatsios, Tsung\-Wei Ke, Ayush Jain, Jeff Schneider, and Katerina Fragkiadaki\.Diffusion\-ES: Gradient\-free planning with diffusion for autonomous and instruction\-guided driving\.In*IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 15342–15353\. IEEE, 2024\.
- Yu et al\. \(2020\)Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma\.MOPO: Model\-based offline policy optimization\.*Advances in Neural Information Processing Systems*, 33:14129–14142, 2020\.
- Yu et al\. \(2021\)Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn\.COMBO: Conservative offline model\-based policy optimization\.*Advances in Neural Information Processing Systems*, 34:28954–28967, 2021\.
- Zhang et al\. \(2026\)Wancong Zhang, Basile Terver, Artem Zholus, Soham Chitnis, Harsh Sutaria, Mido Assran, Randall Balestriero, Amir Bar, Adrien Bardes, Yann LeCun, et al\.Hierarchical planning with latent world models\.*arXiv preprint arXiv:2604\.03208*, 2026\.

## Appendix AExtended Related Work

This appendix expands upon the three lines of work discussed in Section[3](https://arxiv.org/html/2610.27280#S3), positioningVPFlowwithin the broader literature\.

### A\.1Offline and Goal\-Conditioned Reinforcement Learning

Offline RL learns policies from static datasets without environment interaction, mitigating distribution shift either by biasing the policy towards the behavior distribution\([Fujimoto et al\., 2019](https://arxiv.org/html/2610.27280#bib.bib23);[Wu et al\., 2019](https://arxiv.org/html/2610.27280#bib.bib86);[Peng et al\., 2019](https://arxiv.org/html/2610.27280#bib.bib64);[Fujimoto and Gu, 2021](https://arxiv.org/html/2610.27280#bib.bib22)\)or by regularizing critic updates\([Kumar et al\., 2020](https://arxiv.org/html/2610.27280#bib.bib42);[Kostrikov et al\., 2022](https://arxiv.org/html/2610.27280#bib.bib41)\)\. To capture multi\-modal play data and improve long\-horizon credit assignment, Flow Q\-learning \(FQL\) distills a conditional flow\-matching policy into a one\-step network\([Lipman et al\., 2023](https://arxiv.org/html/2610.27280#bib.bib47);[Park et al\., 2025c](https://arxiv.org/html/2610.27280#bib.bib62)\), and action chunking introduces temporal abstraction\([Li et al\., 2025](https://arxiv.org/html/2610.27280#bib.bib46);[Park et al\., 2025b](https://arxiv.org/html/2610.27280#bib.bib61)\)\. In goal\-reaching settings\([Kaelbling, 1993](https://arxiv.org/html/2610.27280#bib.bib37)\), universal value function approximators\([Schaul et al\., 2015](https://arxiv.org/html/2610.27280#bib.bib69)\)and hindsight relabeling\([Andrychowicz et al\., 2017](https://arxiv.org/html/2610.27280#bib.bib1)\)extract supervision from reward\-free trajectories, which offline methods combine with iterated supervised learning\([Ghosh et al\., 2021](https://arxiv.org/html/2610.27280#bib.bib25)\), latent plans from play\([Lynch et al\., 2020](https://arxiv.org/html/2610.27280#bib.bib49)\), contrastive learning\([Eysenbach et al\., 2022](https://arxiv.org/html/2610.27280#bib.bib20)\), quasi\-metric representations\([Wang et al\., 2023](https://arxiv.org/html/2610.27280#bib.bib83);[Myers et al\., 2025](https://arxiv.org/html/2610.27280#bib.bib53)\), and subgoal hierarchies\([Park et al\., 2023](https://arxiv.org/html/2610.27280#bib.bib58)\), standardized by OGBench\([Park et al\., 2025a](https://arxiv.org/html/2610.27280#bib.bib60)\)\.VPFlowadopts a distilled action\-chunk flow policy as its behavior prior\([Park et al\., 2025c](https://arxiv.org/html/2610.27280#bib.bib62);[Li et al\., 2025](https://arxiv.org/html/2610.27280#bib.bib46)\)but departs from these static, model\-free methods in two ways: it replaces TD, contrastive, or quasi\-metric representation learning with closed\-form Monte Carlo return regression over multi\-step imagined rollouts, and it scales test\-time compute via receding\-horizon noise steering rather than acting greedily at deployment\.

### A\.2Model\-Based RL and World Models

Model\-based RL dates back to Dyna\([Sutton, 1991](https://arxiv.org/html/2610.27280#bib.bib75)\)and Gaussian\-process policy search\([Deisenroth and Rasmussen, 2011](https://arxiv.org/html/2610.27280#bib.bib17)\), with deep variants learning dynamics via pixel reconstruction\([Watter et al\., 2015](https://arxiv.org/html/2610.27280#bib.bib84);[Finn and Levine, 2017](https://arxiv.org/html/2610.27280#bib.bib21);[Hafner et al\., 2019](https://arxiv.org/html/2610.27280#bib.bib29)\)or latent imagination\([Ha and Schmidhuber, 2018](https://arxiv.org/html/2610.27280#bib.bib28);[Hafner et al\., 2020](https://arxiv.org/html/2610.27280#bib.bib30);[Hafner et al\., 2025a](https://arxiv.org/html/2610.27280#bib.bib31);[Hafner et al\., 2025b](https://arxiv.org/html/2610.27280#bib.bib32)\)and limiting compounding error through short rollouts\([Janner et al\., 2019](https://arxiv.org/html/2610.27280#bib.bib36);[Park et al\., 2026](https://arxiv.org/html/2610.27280#bib.bib57)\)\. Offline model\-based methods such as MOPO\([Yu et al\., 2020](https://arxiv.org/html/2610.27280#bib.bib88)\), MOReL\([Kidambi et al\., 2020](https://arxiv.org/html/2610.27280#bib.bib38)\), COMBO\([Yu et al\., 2021](https://arxiv.org/html/2610.27280#bib.bib89)\), and MBOP\([Argenson and Dulac\-Arnold, 2021](https://arxiv.org/html/2610.27280#bib.bib2)\)penalize or truncate rollouts where the model is uncertain, or plan with an ensemble model and a behavior prior like MBOP\([Argenson and Dulac\-Arnold, 2021](https://arxiv.org/html/2610.27280#bib.bib2)\)\. Yet they still assume that accurate state prediction is a prerequisite for planning\. Value\-equivalent models take an alternative position\([Grimm et al\., 2020](https://arxiv.org/html/2610.27280#bib.bib26);[M\. Moerland et al\., 2023](https://arxiv.org/html/2610.27280#bib.bib50)\): value prediction networks \(VPN\)\([Oh et al\., 2017](https://arxiv.org/html/2610.27280#bib.bib56)\)and MuZero\([Schrittwieser et al\., 2020](https://arxiv.org/html/2610.27280#bib.bib70)\)train dynamics purely through predicted rewards, values, and policies\.VPFlowbelongs to the value\-equivalent family, but unlike online methods VPN and MuZero, it exploits closed\-form hindsight Monte Carlo returns in offline goal reaching as the sole supervision for both the encoder and dynamics, eliminating target networks, EMA, state\-matching loss, and TD learning from training\.

### A\.3Predictive Representations

Predictive representations characterize a state by its future outcomes rather than its sensory appearance\([Littman and Sutton, 2001](https://arxiv.org/html/2610.27280#bib.bib48);[Carvalho et al\., 2024](https://arxiv.org/html/2610.27280#bib.bib14)\)\. One line of work builds on successor representations and features\([Dayan, 1993](https://arxiv.org/html/2610.27280#bib.bib16);[Barreto et al\., 2017](https://arxiv.org/html/2610.27280#bib.bib8);[Barreto et al\., 2018](https://arxiv.org/html/2610.27280#bib.bib9);[Borsa et al\., 2019](https://arxiv.org/html/2610.27280#bib.bib12)\), forward\-backward representations\([Touati and Ollivier, 2021](https://arxiv.org/html/2610.27280#bib.bib78);[Blier et al\., 2021](https://arxiv.org/html/2610.27280#bib.bib11)\), and behavior foundation models\([Pirotta et al\., 2024](https://arxiv.org/html/2610.27280#bib.bib66);[Park et al\., 2024](https://arxiv.org/html/2610.27280#bib.bib59)\)to predict discounted future occupancy for zero\-shot reactive readout\. A parallel line learns with a self\-predictive objective that forecasts future latent embeddings, relying on target encoders, stop\-gradients\([Schwarzer et al\., 2021](https://arxiv.org/html/2610.27280#bib.bib72);[Guo et al\., 2022](https://arxiv.org/html/2610.27280#bib.bib27);[Tang et al\., 2023](https://arxiv.org/html/2610.27280#bib.bib76);[Ni et al\., 2024](https://arxiv.org/html/2610.27280#bib.bib55);[Fujimoto et al\., 2025](https://arxiv.org/html/2610.27280#bib.bib24);[Bagatella et al\., 2026](https://arxiv.org/html/2610.27280#bib.bib5)\), or explicit regularizers\([LeCun, 2022](https://arxiv.org/html/2610.27280#bib.bib43);[Dawid and LeCun, 2024](https://arxiv.org/html/2610.27280#bib.bib15);[Assran et al\., 2025](https://arxiv.org/html/2610.27280#bib.bib3);[Van Assel et al\., 2025](https://arxiv.org/html/2610.27280#bib.bib79)\)to prevent collapse\. Whereas occupancy\-based representations omit multi\-step lookahead search at test time and self\-predictive world models require explicit anti\-collapse heuristics without exploiting temporal\-distance signals,VPFlowbridges both perspectives: it supervises multi\-step latent rollouts with goal\-conditioned values, which implicitly prevents collapse and organizes the latent space around temporal proximity to the goal\.

### A\.4Planning in Latent Space

Sampling\-based optimizers such as CEM and MPPI\([Rubinstein, 1999](https://arxiv.org/html/2610.27280#bib.bib67);[Williams et al\., 2017](https://arxiv.org/html/2610.27280#bib.bib85)\)are standard planners in world models\([Hafner et al\., 2019](https://arxiv.org/html/2610.27280#bib.bib29);[Janner et al\., 2019](https://arxiv.org/html/2610.27280#bib.bib36);[Hansen et al\., 2022](https://arxiv.org/html/2610.27280#bib.bib33)\), complemented in visual and offline domains by energy\-based or functional\-distance costs\([Du et al\., 2020](https://arxiv.org/html/2610.27280#bib.bib18);[Tian et al\., 2021](https://arxiv.org/html/2610.27280#bib.bib77)\), hierarchical subgoal search\([Li et al\., 2022](https://arxiv.org/html/2610.27280#bib.bib45);[Zhang et al\., 2026](https://arxiv.org/html/2610.27280#bib.bib90)\), and generative adversarial behavior priors in GOPlan\([Wang et al\., 2024](https://arxiv.org/html/2610.27280#bib.bib82)\)\. Closest to our setting, PLDM\([Sobal et al\., 2025](https://arxiv.org/html/2610.27280#bib.bib73)\)and LeWorldModel\([Maes et al\., 2026b](https://arxiv.org/html/2610.27280#bib.bib52)\)train latent\-predictive dynamics with VICReg\([Bardes et al\., 2022](https://arxiv.org/html/2610.27280#bib.bib7)\)or SIGReg\([Balestriero and LeCun, 2025](https://arxiv.org/html/2610.27280#bib.bib6)\)regularization and plan in action space via CEM to minimize Euclidean distance to the goal embedding, whereas recent generative policy methods steer diffusion or flow noise at test time using external rewards or online interaction\([Salimans et al\., 2017](https://arxiv.org/html/2610.27280#bib.bib68);[Yang et al\., 2024](https://arxiv.org/html/2610.27280#bib.bib87);[Wagenmaker et al\., 2025](https://arxiv.org/html/2610.27280#bib.bib81);[Patil et al\., 2026](https://arxiv.org/html/2610.27280#bib.bib63)\)\. In contrast,VPFlowavoids both the out\-of\-distribution action queries of open\-loop action\-space CEM and the external reward requirements of prior noise\-steering methods: it searches directly in the noise space of its distilled flow policy to maximize imagined generalized advantage under its value\-predictive world model, keeping planned actions on the data manifold\.

## Appendix BImplementation Details

This section specifies the datasets \(Section[B\.1](https://arxiv.org/html/2610.27280#A2.SS1)\), network details \(Section[B\.2](https://arxiv.org/html/2610.27280#A2.SS2)\), training objectives \(Section[B\.3](https://arxiv.org/html/2610.27280#A2.SS3)\), optimization \(Sections[B\.4](https://arxiv.org/html/2610.27280#A2.SS4)and[B\.5](https://arxiv.org/html/2610.27280#A2.SS5)\), evaluation protocol \(Section[B\.6](https://arxiv.org/html/2610.27280#A2.SS6)\), planner variants ofVPFlow\(Section[B\.7](https://arxiv.org/html/2610.27280#A2.SS7)\), other model\-based planning baselines in the latent space \(Section[B\.8](https://arxiv.org/html/2610.27280#A2.SS8)\), and software and hardware \(Section\)\. Configurations and hyperparameters are detailed in Appendix\.

### B\.1Datasets and Preprocessing

Datasets\.We use the official OGBench datasets\([Park et al\., 2025a](https://arxiv.org/html/2610.27280#bib.bib60)\)covering both locomotion and robotic manipulation tasks without modification\. The state suite comprises 17 datasets:pointmaze\-\{medium, large, giant, teleport\}\-navigate,antmaze\-\{medium, large, giant, teleport\}\-navigate,cube\-\{single, double, triple, quadruple\}\-play,scene\-play, andpuzzle\-\{3x3, 4x4, 4x5, 4x6\}\-play\. The pixel suite comprises the 13visual\-counterparts of the AntMaze, Cube, Scene, and Puzzle datasets, with64×64×364\\times 64\\times 3observations\. Each dataset has five evaluation tasks specified by start\-goal pairs\.

Training samples and goal relabeling\.A training sample is a length\-HHsegment\(st,at,…,at\+H−1\)\(s\_\{t\},a\_\{t\},\\ldots,a\_\{t\+H\-1\}\)drawn uniformly from the dataset trajectories, with the chunk lengthHHin Table\. The value goalgVg^\{V\}\(ggin Section[5\.3](https://arxiv.org/html/2610.27280#S5.SS3)\) is sampled from a mixture of three sources, with the probabilities in Table: the current observationsts\_\{t\}; a future observationst\+ds\_\{t\+d\}on the same trajectory, with the offsetdddrawn from the geometric distributionGeom⁡\(1−γ\)\\mathrm\{Geom\}\(1\-\\gamma\); and a uniformly random dataset observation\. For each offseth∈\{0,…,H\}h\\in\\\{0,\\ldots,H\\\}, the MC target isRt\+h=\(1−γdh\)/\(γ−1\)R\_\{t\+h\}=\(1\-\\gamma^\{d\_\{h\}\}\)/\(\\gamma\-1\)withdhd\_\{h\}as the number of temporal steps fromst\+hs\_\{t\+h\}togVg^\{V\}, and the maskMt\+hM\_\{t\+h\}is one if and only ifgVg^\{V\}is on the same trajectory at or after indext\+ht\+h\. Random goals therefore contribute no value target\. The policy goalgπg^\{\\pi\}\(ggin Section[5\.4](https://arxiv.org/html/2610.27280#S5.SS4)\) is randomly sampled from a future observation on the same trajectory\.

Data augmentation\.For pixel\-based tasks, we apply random\-crop augmentation \(edge padding, then cropping back to64×6464\\times 64; Table\) using one shared crop per sample for the sampled observation and both goals, except for the Visual AntMaze environment, following the practice in OGBench\.

### B\.2Networks

Observation encoder\.State\-based tasks use the identity encoderEϕ​\(s\)=sE\_\{\\phi\}\(s\)=s\. Pixel\-based tasks use one IMPALA\-Small convolutional encoder\([Espeholt et al\., 2018](https://arxiv.org/html/2610.27280#bib.bib19)\)shared by observations and goals, followed by a multilayer perceptron \(MLP\) head that produces the latent \(Table\)\.

Dynamics model\.The predictive world modelfϕf\_\{\\phi\}maps the current latentztz\_\{t\}and the action sequenceat:t\+H−1a\_\{t:t\+H\-1\}to subsequent latentsz^t\+1:t\+H\\hat\{z\}\_\{t\+1:t\+H\}\. To enable parallel rollouts, we instantiatefϕf\_\{\\phi\}as a Transformer\([Vaswani et al\., 2017](https://arxiv.org/html/2610.27280#bib.bib80)\)with cross\-attention and causal mask in which the current latentztz\_\{t\}is the query and action chunksat:t\+H−1a\_\{t:t\+H\-1\}are keys and values, as depicted in Figure[9](https://arxiv.org/html/2610.27280#A2.F9)\. It uses rotary position embeddings\([Su et al\., 2024](https://arxiv.org/html/2610.27280#bib.bib74)\), and its configuration is listed in Table\. The prediction for prefixhhattends only toat:t\+h−1a\_\{t:t\+h\-1\}, so allHHprefix latents of a chunk are produced in one parallel call\.

Figure 9:World model architecture\.A Transformer with cross\-attention and causal mask enables parallel rollouts of future latents induced by action chunks\.BBis the batch size,HHthe chunk length,dad\_\{a\}the action dimension,dzd\_\{z\}the latent dimension,ddthe Transformer width\.Value heads\.Following previous practices to handle value overestimation and uncertainty\([Fujimoto and Gu, 2021](https://arxiv.org/html/2610.27280#bib.bib22);[Kostrikov et al\., 2022](https://arxiv.org/html/2610.27280#bib.bib41)\), we adopt a twin\-value\-head structure, where each head is an MLP with layer normalization\([Ba et al\., 2016](https://arxiv.org/html/2610.27280#bib.bib4)\)and GELU activations\([Hendrycks and Gimpel, 2016](https://arxiv.org/html/2610.27280#bib.bib35)\)\(Table\), acting on the concatenation of the encoded statezzand goalzgz\_\{g\}\. Policy improvement and planning use the minimum of the two heads,*i\.e\.*,min⁡\{Vϕ,1,Vϕ,2\}\\min\\\{V\_\{\\phi,1\},V\_\{\\phi,2\}\\\}\. The default MC objective uses no target value network\.

Flow policy\.The velocity fieldvθv\_\{\\theta\}and the distilled one\-step decoderπθ\\pi\_\{\\theta\}are Transformers with sinusoidal position embeddings whose size is tuned per dataset \(Table\)\. Each network outputs action chunks of lengthHH, the same as the horizon of training samples\.

State and goal conditioning adopts a similar design to that ofπ0\\pi\_\{0\}\([Black et al\., 2025](https://arxiv.org/html/2610.27280#bib.bib10)\): the projected latentzzand goalzgz\_\{g\}are prefix tokens that theHHaction tokens attend to, and the flow timeuuenters through a sinusoidal embedding fused into each action token by a two\-layer MLP\. The prefix tokens attend only to each other while the action tokens attend to both the prefix and themselves bidirectionally\. The velocity fieldvθv\_\{\\theta\}and the one\-step policyπθ\\pi\_\{\\theta\}share this design\.

### B\.3Objectives

Value\-predictive model\.VPFlowtrains its dynamics model and observation encoders by minimizingℒVP\\mathcal\{L\}\_\{\\mathrm\{VP\}\}in Equation[6](https://arxiv.org/html/2610.27280#S5.E6)\. The squared error is summed over the two value heads, and the per\-sample loss is clipped at a dataset\-specific threshold\.

Conditional flow matching\.Behavior cloning regularization is a critical technique to reduce OOD actions in offline RL\([Fujimoto and Gu, 2021](https://arxiv.org/html/2610.27280#bib.bib22)\)\. Nonetheless, trajectories in the offline play dataset𝒟\\mathcal\{D\}are mostly multi\-modal, so a uni\-modal Gaussian policy is not expressive enough as the behavior prior\. We follow FQL\([Park et al\., 2025c](https://arxiv.org/html/2610.27280#bib.bib62)\)in using conditional flow matching as the behavior\-cloning loss and a distilled one\-step flow policy to predict multimodal actions\.

Conditional flow matching\([Lipman et al\., 2023](https://arxiv.org/html/2610.27280#bib.bib47)\)learns a velocity fieldvθv\_\{\\theta\}that transports samples from a source distribution𝜺∼𝒩⁡\(𝟎,I\)\\bm\{\\varepsilon\}\\sim\\mathcal\{N\}\(\{\\bm\{0\}\},I\)to a provided action distribution𝒂\{\\bm\{a\}\}along the linear path𝒙u=\(1−u\)​𝜺\+u​𝒂\{\\bm\{x\}\}\_\{u\}=\(1\-u\)\\bm\{\\varepsilon\}\+u\{\\bm\{a\}\}with flow timeu∈\[0,1\]u\\in\[0,1\], whose target velocity is𝒂−𝜺\{\\bm\{a\}\}\-\\bm\{\\varepsilon\}:

ℒCFM​\(θ\)≔𝔼\(s,𝒂,g\)∼𝒟,𝜺,u​\[‖vθ​\(𝒙u,u,Eϕ​\(s\),Eϕ​\(g\)\)−\(𝒂−𝜺\)‖22\]\.\\mathcal\{L\}\_\{\\mathrm\{CFM\}\}\(\\theta\)\\coloneqq\\mathbb\{E\}\_\{\(s,\{\\bm\{a\}\},g\)\\sim\\mathcal\{D\},\\,\\bm\{\\varepsilon\},\\,u\}\\\!\\left\[\\left\\\|v\_\{\\theta\}\(\{\\bm\{x\}\}\_\{u\},u,E\_\{\\phi\}\(s\),E\_\{\\phi\}\(g\)\)\-\(\{\\bm\{a\}\}\-\\bm\{\\varepsilon\}\)\\right\\\|\_\{2\}^\{2\}\\right\]\.\(11\)
Distillation\.The distillation trains the one\-step flow policyπθ​\(z,zg,𝜺\)\\pi\_\{\\theta\}\(z,z\_\{g\},\\bm\{\\varepsilon\}\), conditioned on the encoded statezzand goalzgz\_\{g\}, to match the action chunks decoded from the multi\-step integration of velocityvθv\_\{\\theta\}using the same noise𝜺∼𝒩⁡\(𝟎,I\)\\bm\{\\varepsilon\}\\sim\\mathcal\{N\}\(\{\\bm\{0\}\},I\)by minimizing:

ℒDistill​\(θ\)≔𝔼\(s,g\)∼𝒟,𝜺​\[‖πθ​\(Eϕ​\(s\),Eϕ​\(g\),𝜺\)−sg⁡\(EulerIntegration⁡\(vθ,𝜺\)\)‖22\],\\mathcal\{L\}\_\{\\mathrm\{Distill\}\}\(\\theta\)\\coloneqq\\mathbb\{E\}\_\{\(s,g\)\\sim\\mathcal\{D\},\\,\\bm\{\\varepsilon\}\}\\Big\[\\big\\\|\\pi\_\{\\theta\}\(E\_\{\\phi\}\(s\),E\_\{\\phi\}\(g\),\\bm\{\\varepsilon\}\)\-\\operatorname\{sg\}\\\!\\big\(\\operatorname\{EulerIntegration\}\(v\_\{\\theta\},\\bm\{\\varepsilon\}\)\\big\)\\big\\\|\_\{2\}^\{2\}\\Big\],\(12\)wheresg\\operatorname\{sg\}is the stop\-gradient operation andEulerIntegration⁡\(vθ,𝜺\)\\operatorname\{EulerIntegration\}\(v\_\{\\theta\},\\bm\{\\varepsilon\}\)is the action chunk obtained by integratingvθv\_\{\\theta\}with the Euler method fromu=0u=0\(the noise𝜺\\bm\{\\varepsilon\}\) tou=1u=1with the number of steps in Table\.

Policy improvement\.The policy minimizes Equation[8](https://arxiv.org/html/2610.27280#S5.E8)with the predictive model frozen\. Policy improvement imagines the future states with one chunk \(HHactions\) with the value term in Equation[7](https://arxiv.org/html/2610.27280#S5.E7)divided by its stop\-gradient batch mean\|Qθ\|\|Q\_\{\\theta\}\|, stabilized by a small constantϵQ\\epsilon\_\{Q\}\.

State\-predictive model\.The \(latent\) state\-predictive model described in Section[6\.2](https://arxiv.org/html/2610.27280#S6.SS2)trains the value heads on encoded stateszt\+h=Eϕ​\(st\+h\),h∈\[0,H\]z\_\{t\+h\}=E\_\{\\phi\}\(s\_\{t\+h\}\),h\\in\[0,H\]and includes an additional term with the weightλstate\\lambda\_\{\\mathrm\{state\}\}\(Table\) to train the rolloutz^t\+h\\hat\{z\}\_\{t\+h\}to predict the next latentzt\+hz\_\{t\+h\}, using the latent\-prediction \(LP\) loss:

ℒLP​\(ϕ,λstate\)≔𝔼\(𝒔t,𝒂t,𝒔t\+1\)∼𝒟,g∼𝒮​\[1H\+1​∑h=0HMt\+h⊙‖Vϕ​\(zt\+h,zg\)−Rt\+h‖22\+λstate​1H​∑h=1H‖z^t\+h−zt\+h‖22\]\.\\mathcal\{L\}\_\{\\mathrm\{LP\}\}\(\\phi;\\lambda\_\{\\mathrm\{state\}\}\)\\coloneqq\\mathbb\{E\}\_\{\(\{\\bm\{s\}\}\_\{t\},\{\\bm\{a\}\}\_\{t\},\{\\bm\{s\}\}\_\{t\+1\}\)\\sim\\mathcal\{D\},\\,g\\sim\\mathcal\{S\}\}\\left\[\\frac\{1\}\{H\+1\}\\sum\_\{h=0\}^\{H\}M\_\{t\+h\}\\odot\\\|V\_\{\\phi\}\(z\_\{t\+h\},z\_\{g\}\)\-R\_\{t\+h\}\\\|\_\{2\}^\{2\}\+\\lambda\_\{\\mathrm\{state\}\}\\frac\{1\}\{H\}\\sum\_\{h=1\}^\{H\}\\\|\\hat\{z\}\_\{t\+h\}\-z\_\{t\+h\}\\\|\_\{2\}^\{2\}\\right\]\.\(13\)The target latentzt\+hz\_\{t\+h\}has no stop\-gradient operation, so the pixel encoder can be updated by gradients from self\-prediction\.

### B\.4Training Loop

Algorithm 2VPFlowtraining\.Input:offline dataset

𝒟\\mathcal\{D\}; chunk length

HH; policy update interval

UU; training steps

TT; learning rates

ηϕ,ηθ\\eta\_\{\\phi\},\\eta\_\{\\theta\}
Output:trained parameters

ϕ\\phiand

θ\\theta
1Initialize

ϕ\\phifor

\(Eϕ,fϕ,Vϕ,1,Vϕ,2\)\(E\_\{\\phi\},f\_\{\\phi\},V\_\{\\phi,1\},V\_\{\\phi,2\}\)and

θ\\thetafor

\(vθ,πθ\)\(v\_\{\\theta\},\\pi\_\{\\theta\}\)
2for*k=1,…,Tk=1,\\ldots,T*do

3

\(st:t\+H,𝒂t=at:t\+H−1\)∼𝒟\(s\_\{t:t\+H\},\{\\bm\{a\}\}\_\{t\}=a\_\{t:t\+H\-1\}\)\\sim\\mathcal\{D\}; sample goals

gV,gπg^\{V\},g^\{\\pi\}
4

zt←Eϕ​\(st\)z\_\{t\}\\leftarrow E\_\{\\phi\}\(s\_\{t\}\),

zgV←Eϕ​\(gV\)z\_\{g^\{V\}\}\\leftarrow E\_\{\\phi\}\(g^\{V\}\)
5

z^t←zt\\hat\{z\}\_\{t\}\\leftarrow z\_\{t\}
6

z^t\+h←fϕ\(zt,at:t\+h−1\)\\hat\{z\}\_\{t\+h\}\\leftarrow f\_\{\\phi\}\(z\_\{t\},a\_\{t:t\+h\-1\}\),

h=1,…,Hh=1,\\ldots,H⊳\\trianglerightEquation[5](https://arxiv.org/html/2610.27280#S5.E5)

7

dh←steps⁡\(st\+h→gV\)d\_\{h\}\\leftarrow\\operatorname\{steps\}\(s\_\{t\+h\}\\to g^\{V\}\),

h=0,…,Hh=0,\\ldots,H⊳\\trianglerightTemporal distance to goal

8

Rt\+h←\(1−γdh\)/\(γ−1\)R\_\{t\+h\}\\leftarrow\(1\-\\gamma^\{d\_\{h\}\}\)/\(\\gamma\-1\)⊳\\trianglerightEquation[3](https://arxiv.org/html/2610.27280#S5.E3)

9

Mt\+h←𝟙\{dh≥0\}M\_\{t\+h\}\\leftarrow\\mathds\{1\}\_\{\\\{d\_\{h\}\\geq 0\\\}\}
10

ϕ←AdamStep⁡\(ϕ,∇ϕℒVP,ηϕ\)\\phi\\leftarrow\\operatorname\{AdamStep\}\(\\phi,\\nabla\_\{\\phi\}\\mathcal\{L\}\_\{\\mathrm\{VP\}\};\\eta\_\{\\phi\}\)⊳\\trianglerightEquation[6](https://arxiv.org/html/2610.27280#S5.E6),ℒVP\\mathcal\{L\}\_\{\\mathrm\{VP\}\}

11if*kmodU=0k\\bmod U=0*then

12

zt←sg⁡\(Eϕ​\(st\)\)z\_\{t\}\\leftarrow\\operatorname\{sg\}\(E\_\{\\phi\}\(s\_\{t\}\)\),

zgπ←sg⁡\(Eϕ​\(gπ\)\)z\_\{g^\{\\pi\}\}\\leftarrow\\operatorname\{sg\}\(E\_\{\\phi\}\(g^\{\\pi\}\)\)
13

𝜺∼𝒩⁡\(𝟎,I\)\\bm\{\\varepsilon\}\\sim\\mathcal\{N\}\(\{\\bm\{0\}\},I\)
14

𝒂←clip⁡\(πθ​\(zt,zgπ,𝜺\),−1,1\)\{\\bm\{a\}\}\\leftarrow\\operatorname\{clip\}\(\\pi\_\{\\theta\}\(z\_\{t\},z\_\{g^\{\\pi\}\},\\bm\{\\varepsilon\}\),\-1,1\)
15

z^t\+H←fϕ​\(zt,𝒂\)\\hat\{z\}\_\{t\+H\}\\leftarrow f\_\{\\phi\}\(z\_\{t\},\{\\bm\{a\}\}\)
16

Qθ←γH​minj∈\{1,2\}​Vϕ,j​\(z^t\+H,zgπ\)Q\_\{\\theta\}\\leftarrow\\gamma^\{H\}\\min\_\{j\\in\\\{1,2\\\}\}V\_\{\\phi,j\}\(\\hat\{z\}\_\{t\+H\},z\_\{g^\{\\pi\}\}\)
17

θ←AdamStep⁡\(θ,∇θℒπ,ηθ\)\\theta\\leftarrow\\operatorname\{AdamStep\}\(\\theta,\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\pi\};\\eta\_\{\\theta\}\)⊳\\trianglerightEquation[8](https://arxiv.org/html/2610.27280#S5.E8),ℒπ\\mathcal\{L\}\_\{\\pi\}

18return

\(ϕ,θ\)\(\\phi,\\theta\)

Algorithm[2](https://arxiv.org/html/2610.27280#algorithm2)summarizes the training process in Figure[3](https://arxiv.org/html/2610.27280#S5.F3)\. The operatorAdamStep\\operatorname\{AdamStep\}denotes an Adam update step\. The goal\-sampling rules and target masks follow Section[B\.1](https://arxiv.org/html/2610.27280#A2.SS1);steps\\operatorname\{steps\}returns−1\-1for goals on another trajectory or before the predicted state\. Hyperparameters are specified in Table\.

### B\.5Optimization

All networks are trained with Adam\([Kingma and Ba, 2015](https://arxiv.org/html/2610.27280#bib.bib40)\), with the optimizer settings in Tablesand\. The number of training steps follows the benchmarking standards in OGBench\([Park et al\., 2025a](https://arxiv.org/html/2610.27280#bib.bib60)\)\. The predictive model with value heads and the policy networks are updated alternately at every step,*i\.e\.*, the policy update intervalU=1U=1\.

### B\.6Planning and Evaluation Protocol

Planning setup\.At each planning step, we encode the current observation and goal and maximize the imagined advantageAAin Equation[10](https://arxiv.org/html/2610.27280#S5.E10), keeping the model and policy weights fixed\. The default horizon is one training\-sized action chunk \(nc=1n\_\{c\}=1,H=16H=16\)\. We execute the firstHexecH\_\{\\mathrm\{exec\}\}actions and replan from the resulting observation\. Shared planner hyperparameters like the population sizeNNand elite countNeN\_\{\\mathrm\{e\}\}are listed in Table, and dataset\-specific hyperparameters like execution horizonsHexecH\_\{\\mathrm\{exec\}\}are listed in Table\.

Noise\-space decoding\.The distilled policy maps source noise to an action chunk,𝒂=πθ​\(z,zg,𝜺\)\{\\bm\{a\}\}=\\pi\_\{\\theta\}\(z,z\_\{g\},\\bm\{\\varepsilon\}\), with𝜺∼𝒩⁡\(𝟎,I\)\\bm\{\\varepsilon\}\\sim\\mathcal\{N\}\(\{\\bm\{0\}\},I\)inℝH​da\\mathbb\{R\}^\{Hd\_\{a\}\}during training, wheredad\_\{a\}is the action dimension\. Planning optimizes this noise through the decoderDec\\operatorname\{Dec\}in Algorithm[1](https://arxiv.org/html/2610.27280#algorithm1);𝜺=𝟎\\bm\{\\varepsilon\}=\{\\bm\{0\}\}gives the deterministic policy output, which we call the policy mode\. For multiple chunks \(nc≥1n\_\{c\}\\geq 1\), the decoder conditions each chunkccon the latent reached by the preceding chunk using its noise vector𝜺c\\bm\{\\varepsilon\}\_\{c\}, where𝜺=\(𝜺0,…,𝜺nc−1\)\\bm\{\\varepsilon\}=\(\\bm\{\\varepsilon\}\_\{0\},\\ldots,\\bm\{\\varepsilon\}\_\{n\_\{c\}\-1\}\), keeping dynamics queries within the training context length\. The score in Equation[10](https://arxiv.org/html/2610.27280#S5.E10)then extends overnc​Hn\_\{c\}Hsteps\. Shorter planning horizons use a prefix of the decoded chunk\. SettingK=0K=0bypasses search and executes the policy mode\.

\(\(a\)\)Evolution of the noise population\.![Refer to caption](https://arxiv.org/html/2610.27280v1/es_gae-crop.png)\(\(b\)\)GAE is optimized via noise\-space search\.

Figure 10:Noise\-space evolutionary search withε\\varepsilon\-ESonpointmaze\-giant\-navigate\-v0\.Gray points denote candidate noises, orange points denote elites, and the blue star marks the selected noise at each generation\. The right panel shows the corresponding advantage scores\.Zeroth\-order steering\.ε\\varepsilon\-ES\(Algorithm[1](https://arxiv.org/html/2610.27280#algorithm1)\) uses forward evaluations ofAA, allowing search without informative gradients\. It initializesNeN\_\{\\mathrm\{e\}\}elites at the policy mode and generatesNNGaussian mutations per iteration, distributed evenly among the elites\. The mutation scale is annealed asσk=max⁡\(σmin,σmax​ρk\)\\sigma\_\{k\}=\\max\(\\sigma\_\{\\min\},\\sigma\_\{\\max\}\\rho^\{k\}\)fork=0,…,K−1k=0,\\ldots,K\-1\. Each parent is replaced only by a child with a higher score, so every elite’s score remains at least as high as the initial policy score and the best score is non\-decreasing\. The final plan is sampled from the elites with probabilities proportional toexp⁡\(β​Ae\)\\exp\(\\beta A\_\{e\}\), retaining several high\-scoring alternatives rather than selecting only the maximum\. The selected plan therefore also matches or exceeds the policy mode’s score under the learned objectiveAA\. Figure[10](https://arxiv.org/html/2610.27280#A2.F10)illustrates the population and score evolution on PointMaze\.

Algorithm 3ε\\varepsilon\-SGDandε\\varepsilon\-Adam: first\-order noise steering\.Input:state

sts\_\{t\}, goal

gg; optimizer

o∈\{SGD,Adam\}o\\in\\\{\\mathrm\{SGD\},\\mathrm\{Adam\}\\\}; steps

K≥0K\\geq 0; step size

η\\eta; decay

ρ\\rho; execution length

HexecH\_\{\\mathrm\{exec\}\}; decoder

Dec\\operatorname\{Dec\}of Algorithm[1](https://arxiv.org/html/2610.27280#algorithm1)

Output:first

HexecH\_\{\\mathrm\{exec\}\}actions of the final iterate

1

zt←Eϕ​\(st\)z\_\{t\}\\leftarrow E\_\{\\phi\}\(s\_\{t\}\),

zg←Eϕ​\(g\)z\_\{g\}\\leftarrow E\_\{\\phi\}\(g\)
2if*K=0K=0*then

3return

Dec\(𝟎\)\[:Hexec\]\\operatorname\{Dec\}\(\{\\bm\{0\}\}\)\[\{:\}H\_\{\\mathrm\{exec\}\}\]
4

𝜺←𝟎\\bm\{\\varepsilon\}\\leftarrow\{\\bm\{0\}\}⊳\\trianglerightPolicy initialization at zero

5

𝒎←𝟎\{\\bm\{m\}\}\\leftarrow\{\\bm\{0\}\},

𝒗←𝟎\{\\bm\{v\}\}\\leftarrow\{\\bm\{0\}\}
6for*k=1,…,Kk=1,\\ldots,K*do

7

ηk←η​ρk−1\\eta\_\{k\}\\leftarrow\\eta\\rho^\{k\-1\}⊳\\trianglerightAnnealed step size

8

𝒖←∇𝜺A​\(zt,Dec⁡\(𝜺\),zg\)\{\\bm\{u\}\}\\leftarrow\\nabla\_\{\\bm\{\\varepsilon\}\}A\(z\_\{t\},\\operatorname\{Dec\}\(\\bm\{\\varepsilon\}\),z\_\{g\}\)⊳\\trianglerightEquation[14](https://arxiv.org/html/2610.27280#A2.E14)

9if*o=SGDo=\\mathrm\{SGD\}*then

10

𝜺←𝜺\+ηk​𝒖\\bm\{\\varepsilon\}\\leftarrow\\bm\{\\varepsilon\}\+\\eta\_\{k\}\{\\bm\{u\}\}
11else

12

𝒎←0\.9​𝒎\+0\.1​𝒖\{\\bm\{m\}\}\\leftarrow 0\.9\\,\{\\bm\{m\}\}\+0\.1\\,\{\\bm\{u\}\}
13

𝒗←0\.999​𝒗\+0\.001​𝒖2\{\\bm\{v\}\}\\leftarrow 0\.999\\,\{\\bm\{v\}\}\+0\.001\\,\{\\bm\{u\}\}^\{2\}
14

𝒎^←𝒎/\(1−0\.9k\)\\hat\{\\bm\{m\}\}\\leftarrow\{\\bm\{m\}\}/\(1\-0\.9^\{k\}\),

𝒗^←𝒗/\(1−0\.999k\)\\hat\{\\bm\{v\}\}\\leftarrow\{\\bm\{v\}\}/\(1\-0\.999^\{k\}\)
15

𝜺←𝜺\+ηk​𝒎^/\(𝒗^\+10−8\)\\bm\{\\varepsilon\}\\leftarrow\\bm\{\\varepsilon\}\+\\eta\_\{k\}\\hat\{\\bm\{m\}\}/\(\\sqrt\{\\hat\{\\bm\{v\}\}\}\+10^\{\-8\}\)
16return

Dec\(𝜺\)\[:Hexec\]\\operatorname\{Dec\}\(\\bm\{\\varepsilon\}\)\[\{:\}H\_\{\\mathrm\{exec\}\}\]

First\-order steering\.In practice, we find that first\-order planners generally yield higher success rates thanε\\varepsilon\-ESon locomotion tasks \(Table\)\. As a result, we steer the noise by gradient ascent with stochastic gradient descent \(SGD\) or Adam, and denote the resulting plannersε\\varepsilon\-SGDandε\\varepsilon\-Adam, respectively\. The main results in Table[1](https://arxiv.org/html/2610.27280#S6.T1), with complete evaluations in Table, use these first\-order planners for locomotion and state\-basedcube\-single\-play\-v0datasets, andε\\varepsilon\-ESfor the remaining manipulation datasets\.

Both first\-order planners \(Algorithm[3](https://arxiv.org/html/2610.27280#algorithm3)\) optimize a single noise candidate initialized at𝜺\(0\)=𝟎\\bm\{\\varepsilon\}^\{\(0\)\}=\{\\bm\{0\}\}\. Each of theKKascent steps differentiates through the policy decoder, latent dynamics, and value function:

𝒖\(k\)=∇𝜺A​\(zt,Dec⁡\(𝜺\),zg\)\|𝜺=𝜺\(k−1\)\.\{\\bm\{u\}\}^\{\(k\)\}=\\nabla\_\{\\bm\{\\varepsilon\}\}A\\big\(z\_\{t\},\\operatorname\{Dec\}\(\\bm\{\\varepsilon\}\),z\_\{g\}\\big\)\\Big\|\_\{\\bm\{\\varepsilon\}=\\bm\{\\varepsilon\}^\{\(k\-1\)\}\}\.\(14\)ε\\varepsilon\-SGDupdates𝜺\(k\)=𝜺\(k−1\)\+ηk​𝒖\(k\)\\bm\{\\varepsilon\}^\{\(k\)\}=\\bm\{\\varepsilon\}^\{\(k\-1\)\}\+\\eta\_\{k\}\{\\bm\{u\}\}^\{\(k\)\}, withηk=η​ρk−1\\eta\_\{k\}=\\eta\\rho^\{k\-1\}\.ε\\varepsilon\-Adamuses bias\-corrected Adam ascent\([Kingma and Ba, 2015](https://arxiv.org/html/2610.27280#bib.bib40)\)with the same step\-size schedule and the moment constants in Table\. Adam rescales coordinates using running gradient moments, so its step size is tuned separately from that ofε\\varepsilon\-SGD\. Powers, square roots, and vector division are element\-wise\. Noise is unconstrained, while decoded actions are clipped to\[−1,1\]\[\-1,1\]\. We execute the final iterate without the elitist score guarantee ofε\\varepsilon\-ES\.

Planning complexity\.Ignoring initialization and final decoding,ε\\varepsilon\-ESevaluatesK​NKNcandidate rollouts, whereas first\-order steering usesKKforward\-backward evaluations of one candidate\. For a fixed model and rollout horizon, letCfwdC\_\{\\mathrm\{fwd\}\}andCbwdC\_\{\\mathrm\{bwd\}\}denote the per\-candidate complexity of decoding and scoring, and of backpropagation, respectively\. Under sequential computation, the total time complexities are𝒪⁡\(K​N​Cfwd\)\\mathcal\{O\}\(KNC\_\{\\mathrm\{fwd\}\}\)forε\\varepsilon\-ESand𝒪⁡\(K⁡\(Cfwd\+Cbwd\)\)\\mathcal\{O\}\(K\(C\_\{\\mathrm\{fwd\}\}\+C\_\{\\mathrm\{bwd\}\}\)\)for first\-order steering\. The planner comparisons use separately tuned settings rather than matched compute budgets\.

Evaluation\.We evaluate the final training checkpoint on each dataset’s five official OGBench tasks, using2525episodes per task in vectorized environments\([Park et al\., 2025a](https://arxiv.org/html/2610.27280#bib.bib60)\)\. Each episode retains its task goal, and success is determined by the environment’s native indicator within its time limit\. For each training seed, the dataset success rate is averaged over the five tasks\. We report the mean and standard deviation of success rate across 8 training seeds for state inputs and 4 for pixel inputs, the same as in OGBench\([Park et al\., 2025a](https://arxiv.org/html/2610.27280#bib.bib60)\)\.

### B\.7Planner Variants

The CEM variants in Table[2](https://arxiv.org/html/2610.27280#S6.T2)optimize the same advantageAAasε\\varepsilon\-ES, but refit a diagonal Gaussian at each iteration rather than retain and mutate individual elites\. Each iteration scoresNNcandidates and refits the mean and standard deviation using the topNeN\_\{\\mathrm\{e\}\}candidates with normalized weightswe∝exp⁡\(β​Ae\)w\_\{e\}\\propto\\exp\(\\beta A\_\{e\}\)\. This distribution refit does not preserveε\\varepsilon\-ES’s monotonic elite\-score guarantee\.

Noise\-space CEM\.ε\\varepsilon\-CEM\(Algorithm[4](https://arxiv.org/html/2610.27280#algorithm4)\) applies the CEM refitting scheme in the policy’s noise space, initializing the sampling distribution as𝒩⁡\(𝟎,I\)\\mathcal\{N\}\(\{\\bm\{0\}\},I\)\. Candidate noises are decoded throughDec\\operatorname\{Dec\}of Algorithm[1](https://arxiv.org/html/2610.27280#algorithm1)and ranked byAA\. After each refit, the coordinate\-wise standard deviations𝝈\\bm\{\\sigma\}are clipped betweenσmin\\sigma\_\{\\min\}and the annealed upper boundσk=max⁡\(σmin,σmax​ρk\)\\sigma\_\{k\}=\\max\(\\sigma\_\{\\min\},\\sigma\_\{\\max\}\\rho^\{k\}\), withkkindexed from zero\. The final plan is decoded from an elite sampled with the same softmax weights\.

Action\-space CEM\.𝒜\\mathcal\{A\}\-CEM\(Algorithm[5](https://arxiv.org/html/2610.27280#algorithm5)\) searches directly over action sequences, warm\-starting the initial mean at the policy’s direct output𝝁a=Dec⁡\(𝟎\)\{\\bm\{\\mu\}\}\_\{a\}=\\operatorname\{Dec\}\(\{\\bm\{0\}\}\)and setting the initial standard deviation to𝝈a=σmax​𝟏\\bm\{\\sigma\}\_\{a\}=\\sigma\_\{\\max\}\{\\bm\{1\}\}\. This initialization is recomputed at each planning step from the current observation, without carrying over the previous plan\. Candidates are clipped to\[−1,1\]\[\-1,1\]before scoring, and the sampling standard deviations use the same annealed bounds\. The executed plan is the final fitted action mean\.

Both variants execute the firstHexecH\_\{\\mathrm\{exec\}\}actions, return the direct policy output whenK=0K=0, and add no execution noise\. They reuseε\\varepsilon\-ES’s planning hyperparameters on each dataset \(Tablesand\)\. In the pseudocode below,TopK\\operatorname\{TopK\}returns elite indices and𝟏\{\\bm\{1\}\}is the all\-ones vector; vector squaring, square roots, and clipping are element\-wise\.

Algorithm 4ε\\varepsilon\-CEM: CEM in noise space\.Input:state

sts\_\{t\}, goal

gg; population

NN; elites

1≤Ne≤N1\\leq N\_\{\\mathrm\{e\}\}\\leq N; iterations

K≥0K\\geq 0; scale bounds

σmin,σmax\\sigma\_\{\\min\},\\sigma\_\{\\max\}; decay

ρ\\rho; inverse temperature

β\\beta; execution length

HexecH\_\{\\mathrm\{exec\}\}; decoder

Dec\\operatorname\{Dec\}of Algorithm[1](https://arxiv.org/html/2610.27280#algorithm1)

Output:first

HexecH\_\{\\mathrm\{exec\}\}actions of the selected plan

1

zt←Eϕ​\(st\)z\_\{t\}\\leftarrow E\_\{\\phi\}\(s\_\{t\}\),

zg←Eϕ​\(g\)z\_\{g\}\\leftarrow E\_\{\\phi\}\(g\)
2if*K=0K=0*then

3return

Dec\(𝟎\)\[:Hexec\]\\operatorname\{Dec\}\(\{\\bm\{0\}\}\)\[\{:\}H\_\{\\mathrm\{exec\}\}\]
4

𝝁←𝟎\{\\bm\{\\mu\}\}\\leftarrow\{\\bm\{0\}\},

𝝈←𝟏\\bm\{\\sigma\}\\leftarrow\{\\bm\{1\}\}⊳\\trianglerightGaussian noise initialization

5for*k=0,…,K−1k=0,\\ldots,K\-1*do

6

σk←max⁡\(σmin,σmax​ρk\)\\sigma\_\{k\}\\leftarrow\\max\(\\sigma\_\{\\min\},\\sigma\_\{\\max\}\\rho^\{k\}\)
7for*i=1,…,Ni=1,\\ldots,N*do

8

𝜺\(i\)∼𝒩⁡\(𝝁,diag⁡\(𝝈2\)\)\\bm\{\\varepsilon\}^\{\(i\)\}\\sim\\mathcal\{N\}\(\{\\bm\{\\mu\}\},\\operatorname\{diag\}\(\\bm\{\\sigma\}^\{2\}\)\)⊳\\trianglerightNoise\-space sampling

9

Ai←A⁡\(zt,Dec⁡\(𝜺\(i\)\),zg\)A\_\{i\}\\leftarrow A\(z\_\{t\},\\operatorname\{Dec\}\(\\bm\{\\varepsilon\}^\{\(i\)\}\),z\_\{g\}\)⊳\\trianglerightEquation[10](https://arxiv.org/html/2610.27280#S5.E10)

10

ℰ←TopK⁡\(\(Ai\)i=1N,Ne\)\\mathcal\{E\}\\leftarrow\\operatorname\{TopK\}\(\(A\_\{i\}\)\_\{i=1\}^\{N\},N\_\{\\mathrm\{e\}\}\)
11

wi←exp⁡\(β​Ai\)/∑j∈ℰexp⁡\(β​Aj\)w\_\{i\}\\leftarrow\\exp\(\\beta A\_\{i\}\)/\\sum\_\{j\\in\\mathcal\{E\}\}\\exp\(\\beta A\_\{j\}\),

i∈ℰi\\in\\mathcal\{E\}
12

𝝁←∑i∈ℰwi​𝜺\(i\)\{\\bm\{\\mu\}\}\\leftarrow\\sum\_\{i\\in\\mathcal\{E\}\}w\_\{i\}\\bm\{\\varepsilon\}^\{\(i\)\}
13

𝝈←∑i∈ℰwi​\(𝜺\(i\)−𝝁\)2\\bm\{\\sigma\}\\leftarrow\\sqrt\{\\sum\_\{i\\in\\mathcal\{E\}\}w\_\{i\}\(\\bm\{\\varepsilon\}^\{\(i\)\}\-\{\\bm\{\\mu\}\}\)^\{2\}\}
14

𝝈←clip⁡\(𝝈,σmin,σk\)\\bm\{\\sigma\}\\leftarrow\\operatorname\{clip\}\(\\bm\{\\sigma\},\\sigma\_\{\\min\},\\sigma\_\{k\}\)
15

i⋆∼Categorical⁡\(\(wi\)i∈ℰ\)i^\{\\star\}\\sim\\operatorname\{Categorical\}\(\(w\_\{i\}\)\_\{i\\in\\mathcal\{E\}\}\)
16return

Dec\(𝜺\(i⋆\)\)\[:Hexec\]\\operatorname\{Dec\}\(\\bm\{\\varepsilon\}^\{\(i^\{\\star\}\)\}\)\[\{:\}H\_\{\\mathrm\{exec\}\}\]

Algorithm 5𝒜\\mathcal\{A\}\-CEM: CEM in action space\.Input:state

sts\_\{t\}, goal

gg; population

NN; elites

1≤Ne≤N1\\leq N\_\{\\mathrm\{e\}\}\\leq N; iterations

K≥0K\\geq 0; scale bounds

σmin,σmax\\sigma\_\{\\min\},\\sigma\_\{\\max\}; decay

ρ\\rho; inverse temperature

β\\beta; execution length

HexecH\_\{\\mathrm\{exec\}\}; decoder

Dec\\operatorname\{Dec\}of Algorithm[1](https://arxiv.org/html/2610.27280#algorithm1)

Output:first

HexecH\_\{\\mathrm\{exec\}\}actions of the fitted mean

1

zt←Eϕ​\(st\)z\_\{t\}\\leftarrow E\_\{\\phi\}\(s\_\{t\}\),

zg←Eϕ​\(g\)z\_\{g\}\\leftarrow E\_\{\\phi\}\(g\)
2if*K=0K=0*then

3return

Dec\(𝟎\)\[:Hexec\]\\operatorname\{Dec\}\(\{\\bm\{0\}\}\)\[\{:\}H\_\{\\mathrm\{exec\}\}\]
4

𝝁a←Dec⁡\(𝟎\)\{\\bm\{\\mu\}\}\_\{a\}\\leftarrow\\operatorname\{Dec\}\(\{\\bm\{0\}\}\)⊳\\trianglerightAction warm start

5

𝝈a←σmax​𝟏\\bm\{\\sigma\}\_\{a\}\\leftarrow\\sigma\_\{\\max\}\{\\bm\{1\}\}
6for*k=0,…,K−1k=0,\\ldots,K\-1*do

7

σk←max⁡\(σmin,σmax​ρk\)\\sigma\_\{k\}\\leftarrow\\max\(\\sigma\_\{\\min\},\\sigma\_\{\\max\}\\rho^\{k\}\)
8

𝝈a←clip⁡\(𝝈a,σmin,σk\)\\bm\{\\sigma\}\_\{a\}\\leftarrow\\operatorname\{clip\}\(\\bm\{\\sigma\}\_\{a\},\\sigma\_\{\\min\},\\sigma\_\{k\}\)
9for*i=1,…,Ni=1,\\ldots,N*do

10

𝒂\(i\)∼𝒩⁡\(𝝁a,diag⁡\(𝝈a2\)\)\{\\bm\{a\}\}^\{\(i\)\}\\sim\\mathcal\{N\}\(\{\\bm\{\\mu\}\}\_\{a\},\\operatorname\{diag\}\(\\bm\{\\sigma\}\_\{a\}^\{2\}\)\)⊳\\trianglerightAction\-space sampling

11

𝒂\(i\)←clip⁡\(𝒂\(i\),−1,1\)\{\\bm\{a\}\}^\{\(i\)\}\\leftarrow\\operatorname\{clip\}\(\{\\bm\{a\}\}^\{\(i\)\},\-1,1\)
12

Ai←A⁡\(zt,𝒂\(i\),zg\)A\_\{i\}\\leftarrow A\(z\_\{t\},\{\\bm\{a\}\}^\{\(i\)\},z\_\{g\}\)⊳\\trianglerightEquation[10](https://arxiv.org/html/2610.27280#S5.E10)

13

ℰ←TopK⁡\(\(Ai\)i=1N,Ne\)\\mathcal\{E\}\\leftarrow\\operatorname\{TopK\}\(\(A\_\{i\}\)\_\{i=1\}^\{N\},N\_\{\\mathrm\{e\}\}\)
14

wi←exp⁡\(β​Ai\)/∑j∈ℰexp⁡\(β​Aj\)w\_\{i\}\\leftarrow\\exp\(\\beta A\_\{i\}\)/\\sum\_\{j\\in\\mathcal\{E\}\}\\exp\(\\beta A\_\{j\}\),

i∈ℰi\\in\\mathcal\{E\}
15

𝝁a←∑i∈ℰwi​𝒂\(i\)\{\\bm\{\\mu\}\}\_\{a\}\\leftarrow\\sum\_\{i\\in\\mathcal\{E\}\}w\_\{i\}\{\\bm\{a\}\}^\{\(i\)\}
16

𝝈a←∑i∈ℰwi​\(𝒂\(i\)−𝝁a\)2\\bm\{\\sigma\}\_\{a\}\\leftarrow\\sqrt\{\\sum\_\{i\\in\\mathcal\{E\}\}w\_\{i\}\(\{\\bm\{a\}\}^\{\(i\)\}\-\{\\bm\{\\mu\}\}\_\{a\}\)^\{2\}\}
17

𝝈a←clip⁡\(𝝈a,σmin,σk\)\\bm\{\\sigma\}\_\{a\}\\leftarrow\\operatorname\{clip\}\(\\bm\{\\sigma\}\_\{a\},\\sigma\_\{\\min\},\\sigma\_\{k\}\)
18return

𝝁a\[:Hexec\]\{\\bm\{\\mu\}\}\_\{a\}\[\{:\}H\_\{\\mathrm\{exec\}\}\]

### B\.8Model\-Based Latent Planners

We benchmark LeWM\([Maes et al\., 2026b](https://arxiv.org/html/2610.27280#bib.bib52)\), PLDM\([Sobal et al\., 2025](https://arxiv.org/html/2610.27280#bib.bib73)\), and TD\-MPC2\([Hansen et al\., 2024](https://arxiv.org/html/2610.27280#bib.bib34)\)on the 13 pixel\-based datasets in Appendix[B\.1](https://arxiv.org/html/2610.27280#A2.SS1)\. Our implementations are adapted from the official LeWM333[https://github\.com/lucas\-maes/le\-wm](https://github.com/lucas-maes/le-wm)and stable\-worldmodel\([Maes et al\., 2026a](https://arxiv.org/html/2610.27280#bib.bib51)\)444[https://github\.com/galilai\-group/stable\-worldmodel](https://github.com/galilai-group/stable-worldmodel)repositories\. Each baseline learns a latent world model from the offline data and uses CEM\([Rubinstein, 1999](https://arxiv.org/html/2610.27280#bib.bib67)\)to plan towards a goal image\. Table[B\.8](https://arxiv.org/html/2610.27280#A2.SS8)summarizes the configurations; the objectives and OGBench adaptations are described below\.

Table 3:Configurations of latent planning baselines on pixel\-based OGBench\.Merged cells indicate shared settings; – denotes an absent component or an inapplicable setting\.SettingLeWMPLDMTD\-MPC2b