Reinforced Planning with Latent World Models

# Reinforced Planning with Latent WorldModels
Source: [https://arxiv.org/html/2608.18669](https://arxiv.org/html/2608.18669)
## Reinforced Planning with Latent World Models

Armin SommerJannik SchillingAffiliation:Pantheon Industries

###### Abstract

Humans solve complex problems by constructing plans and mentally simulating their outcomes with an internal model of the world\. Machine learning has produced world models that similarly predict the outcomes of action sequences, but the improvement of candidate plans still isn’t fully learned\. Current planners are either hand\-designed, distilled from a hand\-designed optimizer, or learned only to inform an amortized policy rather than to revise the plan itself\. We introduce Reinforced Planning, a method based on the idea that search can be learned by reinforcing good search rules into a neural planner\. Our implementation RP1 learns both how to evaluate imagined outcomes through a critic, as well as how to improve multi\-step plans through an optimizer trained fully offline from imagined world\-model roll\-outs\. To our knowledge, RP1 is the first method to fully learn how to improve multi\-step plans\. Furthermore, it can be trained independently of and attached to any pretrained latent world model\. Across visual navigation, arm reaching, and robotic manipulation on two world\-model backbones, RP1 significantly outperforms hand\-designed search algorithms, reaching near\-perfect success in several settings while using1,000×1,000\\timesfewer world\-model rollouts and being up to67×67\\timesfaster than the strongest alternative under concurrent inference\.

## 1Introduction

Humans are commonly understood to solve complex problems by imagining possible futures and evaluating their consequences\. Hippocampal activity can represent prospective trajectories before an action is taken\[[20](https://arxiv.org/html/2608.18669#bib.bib16);[37](https://arxiv.org/html/2608.18669#bib.bib17)\], supporting the view that the brain uses a learned cognitive map for internal simulation\[[49](https://arxiv.org/html/2608.18669#bib.bib10)\]\. Computationally, this separates planning into two components: a*world model*predicts the consequences of hypothetical actions, while a*planner*determines how candidate action sequences are generated, evaluated, and improved\.

Machine learning has made substantial progress on the first component\. Latent world models now support high\-dimensional visual prediction\[[7](https://arxiv.org/html/2608.18669#bib.bib9);[9](https://arxiv.org/html/2608.18669#bib.bib23);[10](https://arxiv.org/html/2608.18669#bib.bib3)\], self\-supervised predictive representations\[[23](https://arxiv.org/html/2608.18669#bib.bib24);[3](https://arxiv.org/html/2608.18669#bib.bib25)\], and planning with pretrained, reward\-free models\[[54](https://arxiv.org/html/2608.18669#bib.bib4);[44](https://arxiv.org/html/2608.18669#bib.bib5);[28](https://arxiv.org/html/2608.18669#bib.bib6)\]\. Yet the planner operating on top of these models is still usually hand\-designed\. Given a candidate action sequence, the world model can predict its outcome, but it does not specify how that sequence should be changed to produce a better one\. Existing systems therefore rely on fixed search rules, often requiring thousands of world\-model evaluations per decision and configurations that must be chosen separately for different tasks and models\[[9](https://arxiv.org/html/2608.18669#bib.bib23);[12](https://arxiv.org/html/2608.18669#bib.bib2);[54](https://arxiv.org/html/2608.18669#bib.bib4);[44](https://arxiv.org/html/2608.18669#bib.bib5);[38](https://arxiv.org/html/2608.18669#bib.bib7)\]\.

While planning has been explored in several forms, learning the update rule in model\-based multi\-step planning has not been achieved so far: the planning rule is either fixed or inherited from a conventional optimizer, learned through online interaction, or applied only to the next action rather than to an entire plan \(Sec\.[2](https://arxiv.org/html/2608.18669#S2)\)\.

We introduce theReinforced Planningmethod, and its first implementation RP1, which learns both how imagined outcomes should be evaluated and how multi\-step plans should be improved\. RP1 learns a goal\-conditioned quasimetric critic\[[26](https://arxiv.org/html/2608.18669#bib.bib27);[52](https://arxiv.org/html/2608.18669#bib.bib28)\]from offline trajectories using temporal\-difference learning, then trains a neural planner to repeatedly improve action\-plans at inference time, by reinforcing good planning rules into the network weights\. At each refinement step, RP1 receives the current action\-plan, and evaluates its outcome via world\-model rollouts\. No conventional optimizer is executed inside this update or used as a training target\. To our knowledge, RP1 is the first model\-based planner to fully learn an update rule over a multi\-step action plan \(Sec\.[2](https://arxiv.org/html/2608.18669#S2)\)\.

We evaluate RP1 with two pretrained world\-model backbones, LeWorldModel and PLDM, across visual navigation \(TwoRoom\), continuous\-control reaching \(Reacher\), and contact\-rich manipulation \(OGBench Cube\)\. Across these three domains, RP1 exceeds the strongest existing planners while using only99world\-model rollouts per decision, compared with9,0009\{,\}000for the strongest competitor method, and reduces planning latency by up to67×67\\timeswhen multiple control loops share one GPU\.

## 2Background

#### Fixed or inherited planning rules\.

Most model\-based planners use a hand\-designed update rule: CEM in PlaNet and DINO\-WM, MPPI in the TD\-MPC family, or gradient descent through differentiable world\-model rollouts\[[9](https://arxiv.org/html/2608.18669#bib.bib23);[12](https://arxiv.org/html/2608.18669#bib.bib2);[54](https://arxiv.org/html/2608.18669#bib.bib4);[44](https://arxiv.org/html/2608.18669#bib.bib5);[38](https://arxiv.org/html/2608.18669#bib.bib7)\]\. Universal Planning Networks optimize multi\-step action sequences, but fix gradient descent as the plan optimizer\[[45](https://arxiv.org/html/2608.18669#bib.bib48)\]\. DMPO retains an MPPI update and shift operation and learns modifications to them from online task return\[[40](https://arxiv.org/html/2608.18669#bib.bib50)\]\. L2O\-MPC learns the runtime update, but only by imitating a higher\-budget MPPI expert that must be run during training\[[39](https://arxiv.org/html/2608.18669#bib.bib49)\]\. In all these, the planning rules are either \(partly\) hand\-designed or learned from a hand\-designed optimizer\.

#### Amortized model\-based control\.

The Dreamer methods use their world model to train an amortized policy, but do not plan through the world model at inference time\[[8](https://arxiv.org/html/2608.18669#bib.bib12)\]\. Diffuser learns a generative model over state–action trajectories that jointly captures dynamics and planning, refining trajectories directly through denoising rather than explicitly rolling out and evaluating successive candidate action plans through a separate world model\[[19](https://arxiv.org/html/2608.18669#bib.bib51)\]\.

#### Planning to inform amortized policies\.

The Imagination Based Planner \(IBP\) and Thinker methods learn which imagined trajectories to construct or inspect, but do not iteratively improve a candidate plan\. Instead, the information gathered through imagination conditions the agent’s amortized action policy\[[36](https://arxiv.org/html/2608.18669#bib.bib46);[5](https://arxiv.org/html/2608.18669#bib.bib47)\]\. Their learned planning behaviour therefore serves to improve fixed action distributions, rather than improving the candidate plan itself, and necessitates online learning\.

#### Iterative next\-action optimization\.

Iterative Amortized Policy Optimization \(IAPO\) instead learns an iterative optimizer for the current\-state action distributionπ⁡\(at∣st\)\\pi\(a\_\{t\}\\mid s\_\{t\}\)\[[30](https://arxiv.org/html/2608.18669#bib.bib45)\]\. Even in its model\-based variant, future terms in a world\-model rollout remain amortized policy outputs rather than jointly optimized decision variables\. The learned optimization therefore remains one\-step improvement, rather than learning an update rule over a multi\-step plan\.

#### Objectives for imagined plans\.

JEPA\-based world model literature tends to score imagined outcomes by their Euclidean distance to the goal in latent space\[[28](https://arxiv.org/html/2608.18669#bib.bib6);[44](https://arxiv.org/html/2608.18669#bib.bib5);[54](https://arxiv.org/html/2608.18669#bib.bib4)\]\. This choice has a biological analogy in grid\-cell representations, which have been argued to provide a spatial metric for vector\-based navigation\[[11](https://arxiv.org/html/2608.18669#bib.bib21);[4](https://arxiv.org/html/2608.18669#bib.bib22)\]\. However, latent proximity need not reflect temporal reachability \(Theorem[1](https://arxiv.org/html/2608.18669#Thmtheorem1)\), with recent work showing that learned reachability objectives can outperform latent distance\[[24](https://arxiv.org/html/2608.18669#bib.bib52)\]\. Evaluating outcomes with learned value functions is also consistent with evidence implicating the orbitofrontal and ventromedial prefrontal cortex in prospective value evaluation\[[33](https://arxiv.org/html/2608.18669#bib.bib18);[53](https://arxiv.org/html/2608.18669#bib.bib19);[42](https://arxiv.org/html/2608.18669#bib.bib20)\], and is well established in model\-based control\[[13](https://arxiv.org/html/2608.18669#bib.bib1);[12](https://arxiv.org/html/2608.18669#bib.bib2)\]\. We therefore estimate temporal cost\-to\-go from experience using a goal\-conditioned quasimetric\-style critic\[[21](https://arxiv.org/html/2608.18669#bib.bib37);[41](https://arxiv.org/html/2608.18669#bib.bib36);[1](https://arxiv.org/html/2608.18669#bib.bib26);[14](https://arxiv.org/html/2608.18669#bib.bib38);[26](https://arxiv.org/html/2608.18669#bib.bib27);[52](https://arxiv.org/html/2608.18669#bib.bib28);[51](https://arxiv.org/html/2608.18669#bib.bib34)\]\.

## 3Preliminaries

We consider a goal\-conditioned MDP\(𝒮,𝒜,𝒯,g,ρ0\)\(\\mathcal\{S\},\\mathcal\{A\},\\mathcal\{T\},g,\\rho\_\{0\}\)with state space𝒮\\mathcal\{S\}, action space𝒜⊂ℝ\|a\|\\mathcal\{A\}\\subset\\mathbb\{R\}^\{\|a\|\}, goal stateg⊆𝒮g\\subseteq\\mathcal\{S\}, transition function𝒯:𝒮×𝒜→𝒮\\mathcal\{T\}:\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\mathcal\{S\}and initial\-state distributionρ0\\rho\_\{0\}\. The agent does not observessdirectly, but instead receives an observationooin the form of a visual image\.

#### World model\.

A world model predicts the next state, given the current state and some candidate action\. Specifically, the neural network maps an observationoto\_\{t\}to its latent representationztz\_\{t\}through an encoderzt=Eϕ​\(ot\)∈𝒵z\_\{t\}=E\_\{\\phi\}\(o\_\{t\}\)\\in\\mathcal\{Z\}\. The world model then predicts the next latent state via a prediction maphϕ​\(z,a\)=z^h\_\{\\phi\}\(z,a\)=\\hat\{z\}, wherez^\\hat\{z\}is the predicted \(orimagined\) latent of the next state\. We define theNN\-step rollout operator of the world model via

Hϕ​\(𝐚,z^0\)\\displaystyle H\_\{\\phi\}\(\\mathbf\{a\},\\hat\{z\}\_\{0\}\):=hϕ\(hϕ\(⋯hϕ\(z^0,a0\)⋯,aN−2\),aN−1\)=z^N,\\displaystyle:=h\_\{\\phi\}\\bigl\(h\_\{\\phi\}\(\\cdots h\_\{\\phi\}\(\\hat\{z\}\_\{0\},a\_\{0\}\)\\cdots,a\_\{N\-2\}\),\\,a\_\{N\-1\}\\bigr\)\\;=\\;\\hat\{z\}\_\{N\},\(1\)a composition ofNNforward rolls of the world model\.

#### Model\-based Planning\.

Given a start latentztz\_\{t\}and encoded goalzgz\_\{g\}, a plan is scored by a terminal costCCapplied to the final latent state, as predicted by the world modelHϕH\_\{\\phi\}\. This plan aims to optimize the objective

J⁡\(𝐚,zt,zg\):=C⁡\(Hϕ​\(𝐚,zt\),zg\)\.J\(\\mathbf\{a\};z\_\{t\},z\_\{g\}\)\\;:=\\;C\\bigl\(H\_\{\\phi\}\(\\mathbf\{a\},z\_\{t\}\),\\,z\_\{g\}\\bigr\)\.\(2\)A model\-based planner is a search procedure over action sequences: it holds candidate plans and queries the world modelHϕH\_\{\\phi\}to evaluate them underJJ, and applies an update rule

F:𝐚k↦𝐚k\+1\\displaystyle F:\\mathbf\{a\}\_\{k\}\\mapsto\\mathbf\{a\}\_\{k\+1\}\(3\)forKKrounds\. Model\-based planners differ only in their instantiation ofFF\. A policyπ⁡\(a∣zt,zg\)\\pi\(a\\mid z\_\{t\},z\_\{g\}\), by contrast, instead amortizes the objective[2](https://arxiv.org/html/2608.18669#S3.E2)into a direct state\-to\-action mapping and performs no planning\.

Our method follows an actor\-critic architecture: a*critic*scores latent states with respect to a goal, and an actor, here a*learned planner*, optimizes the action plan against the critic’s final state estimate\.

#### Critic\.

The critic is a goal\-conditioned value functionVψ​\(zt,zg\)V\_\{\\psi\}\(\{z\}\_\{t\},\{z\}\_\{g\}\)that estimates the cost\-to\-go from latent statezt\{z\_\{t\}\}to an encoded goal statezg\{z\}\_\{g\}\. Here, lower values correspond to fewer steps to goal and thus signify occupancy of better states\. We learn this critic via offline temporal\-difference \(TD\) learning, although it could in theory be any cost\-function\. During planning, the critic only ever evaluates terminal statesz^N\\hat\{z\}\_\{N\}produced by the rollout operator\.

#### Planner\.

The planner is a learned operator

ℱθ:\(𝐚k⏟current plan,vk⏟terminal value,𝐠k⏟value gradient\)⟼𝐚k\+1⏟improved plan\.\\mathcal\{F\}\_\{\\theta\}:\\left\(\\underbrace\{\\mathbf\{a\}\_\{k\}\}\_\{\\text\{current plan\}\},\\underbrace\{v\_\{k\}\}\_\{\\text\{terminal value\}\},\\underbrace\{\\mathbf\{g\}\_\{k\}\}\_\{\\text\{value gradient\}\}\\right\)\\longmapsto\\underbrace\{\\mathbf\{a\}\_\{k\+1\}\}\_\{\\text\{improved plan\}\}\.\(4\)parametrized byθ\\theta, that outputs an improved plan from the current plan, the critic’s value at the plan’s terminal state, and the plan’s value gradient\. Starting from an initial plan𝐚0\\mathbf\{a\}\_\{0\}, planning does three things per step:

*roll out:*z^N\(k\)=Hϕ​\(𝐚k,zt\),\\displaystyle\\hat\{z\}\_\{N\}^\{\(k\)\}=H\_\{\\phi\}\\bigl\(\\mathbf\{a\}\_\{k\},\{z\}\_\{t\}\\bigr\),\(5\)*evaluate:*vk=Vψ​\(z^N\(k\),zg\),𝐠k=∇𝐚kVψ​\(z^N\(k\),zg\),\\displaystyle v\_\{k\}=V\_\{\\psi\}\\bigl\(\\hat\{z\}\_\{N\}^\{\(k\)\},\{z\}\_\{g\}\\bigr\),\\qquad\\mathbf\{g\}\_\{k\}=\\nabla\_\{\\mathbf\{a\}\_\{k\}\}V\_\{\\psi\}\\bigl\(\\hat\{z\}\_\{N\}^\{\(k\)\},\{z\}\_\{g\}\\bigr\),\(6\)*improve:*𝐚k\+1=ℱθ​\(𝐚k,vk,𝐠k\)\.\\displaystyle\\mathbf\{a\}\_\{k\+1\}=\\mathcal\{F\}\_\{\\theta\}\\bigl\(\\mathbf\{a\}\_\{k\},\\,v\_\{k\},\\,\\mathbf\{g\}\_\{k\}\\bigr\)\.\(7\)The optimized plan is the final iteration,𝐚⋆=𝐚K\\mathbf\{a\}^\{\\star\}=\\mathbf\{a\}\_\{K\}\. While the planner has access to the value and gradient, it is not constrained to follow the plan’s gradient−𝐠k\-\\mathbf\{g\}\_\{k\}and can learn when to trust and distrust it\. Task\-specific information reaches the planner only throughvkv\_\{k\}and𝐠k\\mathbf\{g\}\_\{k\}, forcing it to learn a plan\-update rule rather than a direct state\-and\-goal\-to\-action mapping\.

#### Reinforcing good planning rules\.

Applyingℱθ\\mathcal\{F\}\_\{\\theta\}produces an imagined optimization trajectory

𝐚0→ℱθ𝐚1→ℱθ⋯→ℱθ𝐚K,\\displaystyle\\mathbf\{a\}\_\{0\}\\xrightarrow\{\\mathcal\{F\}\_\{\\theta\}\}\\mathbf\{a\}\_\{1\}\\xrightarrow\{\\mathcal\{F\}\_\{\\theta\}\}\\cdots\\xrightarrow\{\\mathcal\{F\}\_\{\\theta\}\}\\mathbf\{a\}\_\{K\},\(8\)where each new plan is obtained by applying the same learned update rule to the preceding plan\. The planner is trained to minimize the terminal cost\-to\-go predicted by the frozen world modelHϕH\_\{\\phi\}and value functionVψV\_\{\\psi\}:

θ⋆=arg⁡minθ​𝔼\(z0,zg\)∼𝒟​\[Vψ​\(Hϕ​\(𝐚K,zt\),zg\)\]\+𝒞,\\displaystyle\\theta^\{\\star\}=\\arg\\min\_\{\\theta\}\\mathbb\{E\}\_\{\(z\_\{0\},z\_\{g\}\)\\sim\\mathcal\{D\}\}\\left\[V\_\{\\psi\}\\\!\\left\(H\_\{\\phi\}\(\\mathbf\{a\}\_\{K\},z\_\{t\}\),z\_\{g\}\\right\)\\right\]\+\\mathcal\{C\},\(9\)where𝒞\\mathcal\{C\}is some regularizer on intermediate plans’ value\. Updates that produce lower\-cost imagined plans reduce the optimization objective and are reinforced in the shared parameters ofℱθ\\mathcal\{F\}\_\{\\theta\}, while updates that produce higher\-cost plans are suppressed\. The planner therefore learns rules to improve action sequences, not the action sequences themselves\.

## 5Implementation

As a first realization of a Reinforced Planner, we implement a residual version we call*RP1*\. Starting from𝐚0=𝟎\\mathbf\{a\}\_\{0\}=\\mathbf\{0\}, the planner optimizes the action trajectory via

ℱθ\\displaystyle\\mathcal\{F\}\_\{\\theta\}\(𝐚k,vk,𝐠k\)\\displaystyle\\bigl\(\\mathbf\{a\}\_\{k\},\\,v\_\{k\},\\,\\mathbf\{g\}\_\{k\}\\bigr\)\(10\)=clip\[−amax,amax\]⁡\(𝐚k⏟prev\. plan\+fθ​\(𝐚k,vk,𝐠k\)⏟residual​Δ​𝐚k\),\\displaystyle=\\operatorname\{clip\}\_\{\[\-a\_\{\\max\},a\_\{\\max\}\]\}\\\!\\Bigl\(\\underbrace\{\\mathbf\{a\}\_\{k\}\}\_\{\\text\{prev\.\\ plan\}\}\+\\underbrace\{f\_\{\\theta\}\\bigl\(\\mathbf\{a\}\_\{k\},\\,v\_\{k\},\\,\\mathbf\{g\}\_\{k\}\\bigr\)\}\_\{\\text\{residual \}\\Delta\\mathbf\{a\}\_\{k\}\}\\Bigr\),wherefθ:𝒜N×ℝ×ℝN×\|a\|→ℝN×\|a\|f\_\{\\theta\}\\colon\\mathcal\{A\}^\{N\}\\times\\mathbb\{R\}\\times\\mathbb\{R\}^\{N\\times\|a\|\}\\to\\mathbb\{R\}^\{N\\times\|a\|\}is a neural network producing the plan changeΔ​𝐚k=fθ​\(𝐚k,vk,𝐠k\)\\Delta\\mathbf\{a\}\_\{k\}=f\_\{\\theta\}\(\\mathbf\{a\}\_\{k\},v\_\{k\},\\mathbf\{g\}\_\{k\}\)\.

This residual updateΔ​𝐚k\\Delta\\mathbf\{a\}\_\{k\}onto the previous plan𝐚k\\mathbf\{a\}\_\{k\}keeps the gradient flow stable to avoid the vanishing gradient problem\[[16](https://arxiv.org/html/2608.18669#bib.bib53);[15](https://arxiv.org/html/2608.18669#bib.bib54)\]\. The clip is a projection of each plan iterate onto the box\[−amax,amax\]\[\-a\_\{\\max\},a\_\{\\max\}\]\(anℓ∞\\ell\_\{\\infty\}constraint on the action trajectory, not on the update\), bounding actions toamaxa\_\{\\max\}standard deviations of the demonstrated distribution so that rollouts stay on the world model’s support\. We use open\-loop planning for our experiments\.

We realize the critic as a metric residual network\[[27](https://arxiv.org/html/2608.18669#bib.bib32)\]trained with Implicit Q\-Learning through Hindsight Experience Replay\[[22](https://arxiv.org/html/2608.18669#bib.bib30);[2](https://arxiv.org/html/2608.18669#bib.bib35)\]\. For specifics see Appendix[B](https://arxiv.org/html/2608.18669#A2)\.

roll out & evaluatefθf\_\{\\theta\}\+\+roll out & evaluatefθf\_\{\\theta\}\+\+𝐚0=𝟎\\mathbf\{a\}\_\{0\}=\\mathbf\{0\}𝐚1\\mathbf\{a\}\_\{1\}⋮\\vdots𝐚K\\mathbf\{a\}\_\{K\}

Figure 1:RP1 visual\.

## 6Theoretical Results

We formalize two motivations for Reinforced Planning\. First, predictive world\-model learning does not determine a Euclidean latent geometry suitable for planning without additional training incentives\. Second, under any fixed information interface, a learned neural planner can adapt its update rule across tasks, whereas a conventional optimizer uses one fixed configuration throughout the task distribution\.

### 6\.1Latent Norms and Cost\-to\-go

Recall the encoderEϕE\_\{\\phi\}and latent transition modelhϕh\_\{\\phi\}from Section[3](https://arxiv.org/html/2608.18669#S3)\. For notational simplicity, we writeEϕ​\(s\)E\_\{\\phi\}\(s\)for the encoding of the observation generated by statess\. Let𝒯:𝒮×𝒜→𝒮\\mathcal\{T\}:\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\mathcal\{S\}denote deterministic environment dynamics\. We call the latent world model exact when

hϕ​\(Eϕ​\(s\),a\)=Eϕ​\(𝒯⁡\(s,a\)\)for all​\(s,a\)∈𝒮×𝒜\.h\_\{\\phi\}\(E\_\{\\phi\}\(s\),a\)=E\_\{\\phi\}\(\\mathcal\{T\}\(s,a\)\)\\qquad\\text\{for all \}\(s,a\)\\in\\mathcal\{S\}\\times\\mathcal\{A\}\.\(11\)
###### Theorem 1\(Prediction does not identify Euclidean latent geometry\)\.

Suppose\(Eϕ,hϕ\)\(E\_\{\\phi\},h\_\{\\phi\}\)is exact\. If the latent displacements from some statessto two goalsg1g\_\{1\}andg2g\_\{2\}are linearly independent, then there exist two equally exact latent reparameterizations that reverse which goal is closer tossunder Euclidean distance\. The ratio between the two distances can be made arbitrarily large\.

###### Proof sketch\.

Any invertible linear change of latent coordinates can be absorbed into both the encoder and transition model without changing predictive exactness\. By mapping the two goal displacements to separate coordinate axes and stretching either axis, either goal can be made arbitrarily farther than the other\. The full proof is given in Appendix[A\.1](https://arxiv.org/html/2608.18669#A1.SS1)\. ∎

Theorem[1](https://arxiv.org/html/2608.18669#Thmtheorem1)does not imply that latent distance is necessarily a poor planning objective\. Rather, it shows that predictive accuracy alone cannot determine whether it is a good one: two equally predictive world models can rank the same candidate goals in opposite orders\. Agreement between latent distance and temporal cost\-to\-go is therefore an additional property that must be learned or imposed separately\. We learn this property through a goal\-conditioned critic trained directly from temporal transitions\.

This non\-identifiability holds even when the environment is reversible and temporal reachability is symmetric\. Appendix[A\.2](https://arxiv.org/html/2608.18669#A1.SS2)gives the complementary result that temporal reachability can additionally be asymmetric, in which case no symmetric latent norm can represent it exactly\.

### 6\.2Advantage of Learned Planning under Task Heterogeneity

For a fixed world model, action space, planning horizon, and objective, letx=\(zt,zg\)∼μx=\(z\_\{t\},z\_\{g\}\)\\sim\\mudenote a planning task\. A planner stateωk∈Ωpl\\omega\_\{k\}\\in\\Omega\_\{\\mathrm\{pl\}\}contains all information carried from one refinement round to the next, including the current candidate plans and any optimizer memory\. At each round, task\-dependent information is exposed through a fixed interfaceℐ\\mathcal\{I\}\. Starting from a shared initializationω0\\omega\_\{0\}, an update ruleFFis applied forKKrounds:

ωF,0​\(x\)=ω0,ωF,k\+1​\(x\)=F⁡\(ωF,k​\(x\),ℐ⁡\(x,ωF,k​\(x\)\)\)\.\\omega\_\{F,0\}\(x\)=\\omega\_\{0\},\\qquad\\omega\_\{F,k\+1\}\(x\)=F\\\!\\left\(\\omega\_\{F,k\}\(x\),\\mathcal\{I\}\\bigl\(x,\\omega\_\{F,k\}\(x\)\\bigr\)\\right\)\.\(12\)Its expected loss is

ℒ⁡\(F\)=𝔼x∼μ​\[ℓ⁡\(x,ωF,K​\(x\)\)\],\\mathcal\{L\}\(F\)=\\mathbb\{E\}\_\{x\\sim\\mu\}\\left\[\\ell\\bigl\(x,\\omega\_\{F,K\}\(x\)\\bigr\)\\right\],\(13\)whereℓ\\ellis the cost of the plan returned from the final planner state\.

Let𝒲\\mathcal\{W\}denote the compact set of feasible planner inputs, and let𝔉ℐ\\mathfrak\{F\}\_\{\\mathcal\{I\}\}denote the continuous feasible update rulesF:𝒲→ΩplF:\\mathcal\{W\}\\to\\Omega\_\{\\mathrm\{pl\}\}\. The precise ambient spaces and regularity conditions are given in Appendix[A\.3](https://arxiv.org/html/2608.18669#A1.SS3)\.

###### Assumption 1\(Universal search\-rule approximation\)\.

Assume the neural\-planner class\{ℱθ:θ∈Θ\}⊆𝔉ℐ\\\{\\mathcal\{F\}\_\{\\theta\}:\\theta\\in\\Theta\\\}\\subseteq\\mathfrak\{F\}\_\{\\mathcal\{I\}\}can uniformly approximate every rule in𝔉ℐ\\mathfrak\{F\}\_\{\\mathcal\{I\}\}: for everyF∈𝔉ℐF\\in\\mathfrak\{F\}\_\{\\mathcal\{I\}\}and everyε\>0\\varepsilon\>0, there existsθ∈Θ\\theta\\in\\Thetasuch that

supw∈𝒲‖ℱθ​\(w\)−F⁡\(w\)‖<ε\.\\sup\_\{w\\in\\mathcal\{W\}\}\\left\\\|\\mathcal\{F\}\_\{\\theta\}\(w\)\-F\(w\)\\right\\\|<\\varepsilon\.\(14\)

Now let𝒳1,…,𝒳r\\mathcal\{X\}\_\{1\},\\ldots,\\mathcal\{X\}\_\{r\}be a measurable partition of the task distribution, withpi=Pr⁡\(x∈𝒳i\)\>0\.p\_\{i\}=\\Pr\(x\\in\\mathcal\{X\}\_\{i\}\)\>0\.For any update ruleFF, define its regional loss by

ℒi​\(F\)=𝔼⁡\[ℓ⁡\(x,ωF,K​\(x\)\)\|x∈𝒳i\]\.\\mathcal\{L\}\_\{i\}\(F\)=\\mathbb\{E\}\\left\[\\ell\\bigl\(x,\\omega\_\{F,K\}\(x\)\\bigr\)\\,\\middle\|\\,x\\in\\mathcal\{X\}\_\{i\}\\right\]\.\(15\)
Letℬ\\mathcal\{B\}be a family of fixed search configurations\. EachB∈ℬB\\in\\mathcal\{B\}induces an update ruleFBF\_\{B\}, and the same configuration is used on every task\. We write

ℒi​\(B\)=ℒi​\(FB\),ℒ⁡\(B\)=∑i=1rpi​ℒi​\(B\)\.\\mathcal\{L\}\_\{i\}\(B\)=\\mathcal\{L\}\_\{i\}\(F\_\{B\}\),\\qquad\\mathcal\{L\}\(B\)=\\sum\_\{i=1\}^\{r\}p\_\{i\}\\mathcal\{L\}\_\{i\}\(B\)\.\(16\)
###### Theorem 2\(Strict advantage under task heterogeneity\)\.

Under Assumption[1](https://arxiv.org/html/2608.18669#Thmassumption1)and the regularity, regional incompatibility, and interface\-composability conditions stated in Appendix[A\.3](https://arxiv.org/html/2608.18669#A1.SS3),

infθ∈Θℒ⁡\(ℱθ\)≤∑i=1rpi​minB∈ℬ​ℒi​\(B\)<minB∈ℬ⁡ℒ⁡\(B\)\.\\inf\_\{\\theta\\in\\Theta\}\\mathcal\{L\}\(\\mathcal\{F\}\_\{\\theta\}\)\\leq\\sum\_\{i=1\}^\{r\}p\_\{i\}\\min\_\{B\\in\\mathcal\{B\}\}\\mathcal\{L\}\_\{i\}\(B\)<\\min\_\{B\\in\\mathcal\{B\}\}\\mathcal\{L\}\(B\)\.\(17\)Thus, a sufficiently expressive learned planner can strictly outperform every single fixed search configuration by adapting its update behavior across task regions through the shared interface\.

Note that Theorem[2](https://arxiv.org/html/2608.18669#Thmtheorem2)is an idealized matched\-interface expressivity result\. It identifies an advantage available to a sufficiently expressive learned update rule under the stated assumptions; it does not establish that the finite RP1 architecture contains the resulting rule, that training finds it, or that the empirical planners satisfy the theorem’s deterministic, continuous, and matched\-interface setup\.

## 7Experiments

![Refer to caption](https://arxiv.org/html/2608.18669v1/final_three_envs.png)Figure 2:Experiment environments\.In Reacher \(left\) the agent moves a two\-link arm to a goal configuration, here shaded\. In OGBench Cube \(middle\) a robot arm picks up a cube and moves it to a goal position, also shaded\. In TwoRoom \(right\) an agent navigates to a goal position, marked by a star\.#### Evaluation design\.

We evaluate RP1 in three visual\-control domains, TwoRoom, Reacher, and OGBench Cube, on two world\-model bases: LeWorldModel \(LeWM\) and PLDM\. All world\-model encoders and dynamics predictors remain frozen during critic and planner training, so differences in performance arise from how imagined trajectories are scored and improved rather than from changes to the world models\. We use benchmarks from the StableWorldModel environment\[[29](https://arxiv.org/html/2608.18669#bib.bib11)\]\.

#### Baselines and controlled comparisons\.

We compare RP1 against three popular hand\-designed planning algorithms and two partly\-learned hybrid planners\. The hand\-designed planners constitute the state of the art for planning with pretrained world models: essentially all recent latent\-planning systems use one of them or a close variant\[[9](https://arxiv.org/html/2608.18669#bib.bib23);[12](https://arxiv.org/html/2608.18669#bib.bib2);[54](https://arxiv.org/html/2608.18669#bib.bib4);[44](https://arxiv.org/html/2608.18669#bib.bib5);[38](https://arxiv.org/html/2608.18669#bib.bib7)\]\. We evaluate each with both latent distance∥z^N−zg∥22\\lVert\\hat\{z\}\_\{N\}\-z\_\{g\}\\rVert\_\{2\}^\{2\}and the learned objectiveV⁡\(z^N,zg\)V\(\\hat\{z\}\_\{N\},z\_\{g\}\)\. The hybrid planners DMPO and L2O\-MPC use a learned critic, following their original design\[[40](https://arxiv.org/html/2608.18669#bib.bib50);[39](https://arxiv.org/html/2608.18669#bib.bib49)\]\.

#### Evaluation metrics\.

We report task success and planning compute cost, measured as the number of world\-model rollouts per decision\. All planners use identical action chunking: each planned action comprises five primitive actions, and each planner optimizes a sequence of five such chunks, corresponding to a horizon of 25 primitive actions\. All methods therefore plan over the same horizon in the same normalized action space\. Full evaluation details are provided in Appendix[C](https://arxiv.org/html/2608.18669#A3)\.

### 7\.1TwoRoom

TwoRoom tests whether model\-based agents are capable of appropriate planning when geometric proximity differs from temporal reachability\. The agent must pass through a doorway to reach the opposite room, so states that are geometrically close across the wall might still require a long detour\.

\(a\)![Refer to caption](https://arxiv.org/html/2608.18669v1/value_landscape.png)

\(b\)![Refer to caption](https://arxiv.org/html/2608.18669v1/rlp_plan_timeseries_2.png)

Figure 3:RP1 in TwoRoom\.\(a\) The learned critic better captures temporal cost\-to\-go than latentL2L\_\{2\}distance\. \(b\) RP1 iteratively refines its plan, with later updates focusing on fine corrections to the final actions\. Additional visualizations are provided in Appendix[C\.2](https://arxiv.org/html/2608.18669#A3.SS2)\.We find that the latent\-distance objective does not capture distance\-to\-goal in the queried world\-models, whereas a learned value function saturates the benchmark across implemented planners\. Once planners are given the learned value critic, their performance largely converges: most methods reach near\-saturated success, despite using very different search rules and compute budgets\. This suggests that in TwoRoom the dominant difficulty is not how candidate plans are improved, but whether they are evaluated with an objective that reflects temporal reachability rather than latent proximity\.

TwoRoomTable 1:Success rate \(%\)\.Bold marks the best two entries per column\. For CEM, MPPI, and Adam, the main number uses latentL2L\_\{2\}while the gray parenthesized number uses the learned value critic\.
### 7\.2Reacher

Reacher is a two\-link arm under torque control, observed only as visual frames\. The task is to bring both joints into a target configuration\. Success follows the benchmark’s first\-hit convention at a loose and a tight tolerance \(τ=0\.1\\tau\{=\}0\.1andτ=0\.05\\tau\{=\}0\.05rad\)\. As in all domains, planners optimize five blocks of five primitive actions, so the planning horizon exactly covers the nominal2525\-step distance to the goal\.

Reacher complements TwoRoom by removing the objective as a confound: the arm moves in free space, meets no obstacles, and every configuration is reachable from every other, so geometric proximity and temporal reachability essentially coincide\. Empirically, latentL2L\_\{2\}distance is already an adequate surrogate for cost\-to\-go, and substituting the learned critic barely moves any baseline \(Table[3](https://arxiv.org/html/2608.18669#S7.T3)\)\. Whatever separates the planners in this domain must therefore come from how plans are improved, not from how they are scored\.

Table 2:ReacherTable 3:First\-hit success \(%\)\.Goal toleranceτ\\tau\(rad\), bold marks best number\. For CEM, MPPI, and Adam, the main number uses latentL2L\_\{2\}while the gray parenthesized number uses the learned value critic\.Even this near\-saturated task discriminates between planners once the tolerance is tightened\. Atτ=0\.1\\tau\{=\}0\.1, every competent planner brings the arm into the neighborhood of the goal: margins are within a point or two, and the relevant difference is cost, with RP1 matching the best baseline on three orders of magnitude fewer world\-model rollouts\. Halving the tolerance separates reaching a region from stopping inside it\. All methods degrade, but RP1 degrades the least and retains the best score in every column, and its margin over the strongest baseline widens from at most one point atτ=0\.1\\tau\{=\}0\.1to6\.76\.7points on LeWM and2\.02\.0on PLDM atτ=0\.05\\tau\{=\}0\.05\. We attribute this to terminal precision rather than coverage: plans that fail atτ=0\.05\\tau\{=\}0\.05typically find the right approach and miss only in the final action blocks, which seem to be refined more accurately in RP1 than other methods\.

### 7\.3OGBench Cube

OGBench Cube\[[34](https://arxiv.org/html/2608.18669#bib.bib8)\]is a vision\-based manipulation benchmark: a robot arm must pick up a cube and place it at a goal position, observed only from pixels, with goals placed2525or100100primitive steps away \(h​25h25,h​100h100\)\. The difficulty of the task comes from contact\. A small change early in a plan decides whether the gripper closes on the cube or misses it entirely, so the objective over plans is discontinuous and multimodal, a poor fit for both smooth gradient descent and a unimodal sampling distribution\. Contact also makes reachability directed: a dropped or knocked\-away cube cannot be undone\.

A complication of the benchmark is that its success criterion is partially satisfied at reset: executing no actions at all already scores56\.0%56\.0\\%ath​25h25and45\.3%45\.3\\%ath​100h100under the identical evaluation protocol \(Appendix[C\.4](https://arxiv.org/html/2608.18669#A3.SS4)\)\. Raw success rates, which we report as*easy*, therefore compress exactly the episodes that require manipulation, and differences between planners are partly masked by a floor every method inherits for free\. Alongside the easy score we report a*hard*score, the same runs normalized against the measured no\-op floorffas\(s−f\)/\(100−f\)⋅100\(s\-f\)/\(100\-f\)\\cdot 100, which measures the fraction of headroom above doing nothing that a planner actually converts\. The hard score is our primary number; easy is kept for comparability with the benchmark’s convention\.

OGBench Cube\.Table 4:Success rate \(%\\%\)\.We report easy numbers and hard numbers\. For normalized \(hard\) scores, we set 0 if the method performed worse than floor, e\.g\. L2O\-MPC on PLDM\.The table separates the two contributions\. The learned critic matters mainly at the long horizon: under latentL2L\_\{2\}, CEM’s hard score on LeWM falls from40\.940\.9ath​25h25to23\.223\.2ath​100h100, while the same planner scoring with the learned value holds57\.457\.4: once the goal is far away, latent distance stops ordering plans by how long they take to realize\. The learned search accounts for the rest: RP1 posts the best score in every column using99rollouts per decision against3,0003\{,\}000–9,0009\{,\}000for the hand\-designed planners\. The normalization itself is informative about the baselines: several hand\-designed search algorithms end up within a few points of the no\-op policy, so most of their raw success was inherited from not\-moving\. RP1 does significantly better, getting up to twice the success rate on PLDM on the hard evals of its closest competitor CEM\.

### 7\.4World\-Model Hallucination and Dyna Finetuning

Training the planner through a frozen world model lets it exploit model error\. Inspecting RP1’s failure episodes in OGBench Cube, we found the world model*hallucinating*contact outcomes: for LeWM, grasps that miss the cube are nevertheless predicted "magically" to attach it to the arm\. No improvement in search can fix such hallucinations\. We therefore correct the model rather than the planner: one Dyna iteration\[[47](https://arxiv.org/html/2608.18669#bib.bib29)\]deploys the trained planner, collects its \(failure\) rollouts, finetunes the world model on them, and retrains the planner \(Appendix[B\.3](https://arxiv.org/html/2608.18669#A2.SS3)\)\.

Table 5:Effect of one Dyna iteration \(hard success, %\)\.Gray parentheses give the unnormalized easy score\. Rollouts are collected onh​25h25tasks only; the finetuned model is reused unchanged ath​100h100\.One iteration recovers a large part of the exploitation gap, and we found empirically that the "grasp\-and\-miss" hallucinations were significantly reduced in LeWM\. However, despite mitigating exploitation, characterizing when it recurs remains open\.

### 7\.5Planning Speed

We measure end\-to\-end planning latency on OGBench Cube 25\-step goal offset with LeWM, including the complete computation from the input latents to the returned action plan\. All methods run in fp32 on a single NVIDIA H200 and are benchmarked using both CUDA\-graph\-captured and eager execution, with the faster mean reported\. We consider one planner running alone \(B=1B=1\) and5050independent planners running concurrently on the same GPU \(B=50B=50\), representing multiple control loops sharing one accelerator\.

101001k10k13×13\\timesfaster3045833914209591001k10k100k67×67\\timesfaster791882365\.3k10\.0k17\.1k\(a\) latency in ms,B=1B=1\(b\) latency in ms,B=50B=50RP1L2O\-MPCDMPOCEMMPPIAdamFigure 4:End\-to\-end planning latency\.On OGBench Cube with LeWM \(one NVIDIA H200, fp32\), with RP113×13\\timesfaster than CEM for one planner and67×67\\timesfaster for5050concurrent planners\.The1,000×1\{,\}000\\timesreduction in world\-model rollouts does not translate one\-for\-one into single\-planner latency because the GPU can evaluate many of a sampling planner’s candidate trajectories in parallel\. Nevertheless, RP1 completes a planning request in3030ms, compared with391391ms for CEM, the strongest conventional baseline, yielding a13×13\\timesspeedup\. The advantage grows substantially under concurrent inference: RP1 processes5050planners in7979ms, whereas CEM requires5\.315\.31s, yielding a67×67\\timesspeedup and reducing the amortized GPU time per planner from106\.2106\.2to1\.581\.58ms\. RP1 also remains3\.0×3\.0\\timesfaster than DMPO and2\.4×2\.4\\timesfaster than L2O\-MPC in this setting\. Thus, the rollout reduction becomes most consequential when one accelerator serves several control loops, such as multiple robot arms planning in tandem\.

## 8Discussion

Our results support the two hypotheses that motivated RP1\. First, on tasks where geometric proximity differs from reachability, replacing the latent\-distance objective with a learned quasimetric\-style critic resolves failures that no amount of additional search can fix \(Sec\.[7\.1](https://arxiv.org/html/2608.18669#S7.SS1)\)\. Second, learning the search procedure itself yields large gains where the plan landscape is discontinuous or multimodal: RP1 matches or exceeds the strongest hand\-designed planners while issuing two to three orders of magnitude fewer world\-model queries\. Together, these findings suggest that for current latent world models, planning quality is often the binding constraint on downstream performance, not prediction fidelity\.

Several limitations remain\. First and foremost, our evaluations are for different hyperparameters between environments\. We believe that this can be resolved at least for the critic, and intend on updating the paper once we have found a configuration that works across environments\. Our results use open\-loop execution; closed\-loop replanning may change the relative standing of the methods, so we intend to report this in future work\. Because the planner is trained through the frozen world model, it can exploit model errors in regions of poor data coverage; the Dyna\-style finetuning loop of Sec\.[B\.3](https://arxiv.org/html/2608.18669#A2.SS3)mitigates but does not eliminate this failure mode, and when planner exploitation occurs is still open for characterization\. Finally, our evaluation covers two world\-model bases and three domains: broader coverage across model families and longer\-horizon, multi\-object tasks is needed before claiming generality, and the learned planner currently assumes a fixed horizon and interface, whereas hand\-designed planners transfer across these choices without retraining\.

## 9Acknowledgements

The authors want to thank Xiao\-ke Lu, Sambhav Gupta and Kunvar Thaman for their insightful suggestions on initial drafts\.

## References

- Andrychowiczet al\.\(2017a\)M\. Andrychowicz, F\. Wolski, A\. Ray, J\. Schneider, R\. Fong, P\. Welinder, B\. McGrew, J\. Tobin, P\. Abbeel, and W\. ZarembaHindsight experience replay\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px5.p1.1)\.
- Andrychowiczet al\.\(2017b\)M\. Andrychowicz, F\. Wolski, A\. Ray, J\. Schneider, R\. Fong, P\. Welinder, B\. McGrew, J\. Tobin, P\. Abbeel, and W\. ZarembaHindsight experience replay\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§B\.1](https://arxiv.org/html/2608.18669#A2.SS1.p1.3),[§5](https://arxiv.org/html/2608.18669#S5.p2.1)\.
- Assranet al\.\(2023\)M\. Assran, Q\. Duval, I\. Misra, P\. Bojanowski, P\. Vincent, M\. Rabbat, Y\. LeCun, and N\. BallasSelf\-supervised learning from images with a joint\-embedding predictive architecture\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 15619–15629\.Cited by:[§1](https://arxiv.org/html/2608.18669#S1.p2.1)\.
- Baninoet al\.\(2018\)A\. Banino, C\. Barry, B\. Uria, C\. Blundell, T\. Lillicrap, P\. Mirowski, A\. Pritzel, M\. J\. Chadwick, T\. Degris, J\. Modayil, G\. Wayne, H\. Soyer, F\. Viola, B\. Zhang, R\. Goroshin, N\. Rabinowitz, R\. Pascanu, C\. Beattie, S\. Petersen, A\. Sadik, S\. Gaffney, H\. King, K\. Kavukcuoglu, D\. Hassabis, R\. Hadsell, and D\. KumaranVector\-based navigation using grid\-like representations in artificial agents\.Nature557\(7705\),pp\. 429–433\.External Links:[Document](https://dx.doi.org/10.1038/s41586-018-0102-6)Cited by:[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px5.p1.1)\.
- Chunget al\.\(2023\)S\. Chung, I\. Anokhin, and D\. KruegerThinker: learning to plan and act\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2307\.14993Cited by:[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px3.p1.1)\.
- Feinberget al\.\(2018\)V\. Feinberg, A\. Wan, I\. Stoica, M\. I\. Jordan, J\. E\. Gonzalez, and S\. LevineModel\-based value estimation for efficient model\-free reinforcement learning\.External Links:1803\.00101,[Link](https://arxiv.org/abs/1803.00101)Cited by:[§C\.4](https://arxiv.org/html/2608.18669#A3.SS4.SSS0.Px2.p1.1)\.
- Ha and Schmidhuber \(2018\)D\. Ha and J\. SchmidhuberRecurrent world models facilitate policy evolution\.Advances in Neural Information Processing Systems31\.Note:Extended version “World Models”, arXiv:1803\.10122Cited by:[§1](https://arxiv.org/html/2608.18669#S1.p2.1)\.
- Hafneret al\.\(2020\)D\. Hafner, T\. Lillicrap, J\. Ba, and M\. NorouziDream to control: learning behaviors by latent imagination\.External Links:1912\.01603,[Link](https://arxiv.org/abs/1912.01603)Cited by:[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px2.p1.1)\.
- Hafneret al\.\(2019\)D\. Hafner, T\. Lillicrap, I\. Fischer, R\. Villegas, D\. Ha, H\. Lee, and J\. DavidsonLearning latent dynamics for planning from pixels\.InInternational Conference on Machine Learning \(ICML\),pp\. 2555–2565\.Cited by:[§1](https://arxiv.org/html/2608.18669#S1.p2.1),[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2608.18669#S7.SS0.SSS0.Px2.p1.1)\.
- Hafneret al\.\(2025\)D\. Hafner, J\. Pasukonis, J\. Ba, and T\. LillicrapMastering diverse control tasks through world models\.Nature640,pp\. 647–653\.Cited by:[§1](https://arxiv.org/html/2608.18669#S1.p2.1)\.
- Haftinget al\.\(2005\)T\. Hafting, M\. Fyhn, S\. Molden, M\. Moser, and E\. I\. MoserMicrostructure of a spatial map in the entorhinal cortex\.Nature436\(7052\),pp\. 801–806\.External Links:[Document](https://dx.doi.org/10.1038/nature03721)Cited by:[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px5.p1.1)\.
- Hansenet al\.\(2024\)N\. Hansen, H\. Su, and X\. WangTD\-MPC2: scalable, robust world models for continuous control\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2608.18669#S1.p2.1),[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px5.p1.1),[§7](https://arxiv.org/html/2608.18669#S7.SS0.SSS0.Px2.p1.1)\.
- Hansenet al\.\(2022\)N\. Hansen, X\. Wang, and H\. SuTemporal difference learning for model predictive control\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px5.p1.1)\.
- Hartikainenet al\.\(2020\)K\. Hartikainen, X\. Geng, T\. Haarnoja, and S\. LevineDynamical distance learning for semi\-supervised and unsupervised skill discovery\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§B\.1](https://arxiv.org/html/2608.18669#A2.SS1.p1.2),[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px5.p1.1)\.
- Heet al\.\(2015\)K\. He, X\. Zhang, S\. Ren, and J\. SunDeep residual learning for image recognition\.CoRRabs/1512\.03385\.External Links:[Link](http://arxiv.org/abs/1512.03385),1512\.03385Cited by:[§5](https://arxiv.org/html/2608.18669#S5.p1.1)\.
- Hochreiter \(1991\)S\. HochreiterUntersuchungen zu dynamischen neuronalen netzen\.Master’s Thesis,Technische Universität München\.Cited by:[§5](https://arxiv.org/html/2608.18669#S5.p1.1)\.
- Huber \(1964\)P\. J\. HuberRobust estimation of a location parameter\.The Annals of Mathematical Statistics35\(1\),pp\. 73–101\.Cited by:[§B\.1](https://arxiv.org/html/2608.18669#A2.SS1.p1.6)\.
- Jafferjeeet al\.\(2020\)T\. Jafferjee, E\. Imani, E\. Talvitie, M\. White, and M\. BowlingHallucinating value: a pitfall of dyna\-style planning with imperfect environment models\.arXiv preprint arXiv:2006\.04363\.Cited by:[§B\.3](https://arxiv.org/html/2608.18669#A2.SS3.p1.1)\.
- Janneret al\.\(2022\)M\. Janner, Y\. Du, J\. B\. Tenenbaum, and S\. LevinePlanning with diffusion for flexible behavior synthesis\.External Links:2205\.09991,[Link](https://arxiv.org/abs/2205.09991)Cited by:[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px2.p1.1)\.
- Johnson and Redish \(2007\)A\. Johnson and A\. D\. RedishNeural ensembles in CA3 transiently encode paths forward of the animal at a decision point\.Journal of Neuroscience27\(45\),pp\. 12176–12189\.External Links:[Document](https://dx.doi.org/10.1523/JNEUROSCI.3761-07.2007)Cited by:[§1](https://arxiv.org/html/2608.18669#S1.p1.1)\.
- Kaelbling \(1993\)L\. P\. KaelblingLearning to achieve goals\.InInternational Joint Conference on Artificial Intelligence \(IJCAI\),Cited by:[§B\.1](https://arxiv.org/html/2608.18669#A2.SS1.p1.1),[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px5.p1.1)\.
- Kostrikovet al\.\(2022\)I\. Kostrikov, A\. Nair, and S\. LevineOffline reinforcement learning with implicit q\-learning\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§B\.1](https://arxiv.org/html/2608.18669#A2.SS1.p1.6),[§5](https://arxiv.org/html/2608.18669#S5.p2.1)\.
- LeCun \(2022\)Y\. LeCunA path towards autonomous machine intelligence\.Note:Version 0\.9\.2[https://openreview\.net/forum?id=BZ5a1r\-kVsf](https://openreview.net/forum?id=BZ5a1r-kVsf)Cited by:[§1](https://arxiv.org/html/2608.18669#S1.p2.1)\.
- Liet al\.\(2026\)L\. Li, S\. Wang, and Q\. LiuBeyond euclidean proximity: repairing latent world models with horizon\-matched trajectory reachability metrics\.External Links:2605\.22164,[Link](https://arxiv.org/abs/2605.22164)Cited by:[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px5.p1.1)\.
- Lillicrapet al\.\(2016\)T\. P\. Lillicrap, J\. J\. Hunt, A\. Pritzel, N\. Heess, T\. Erez, Y\. Tassa, D\. Silver, and D\. WierstraContinuous control with deep reinforcement learning\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§B\.1](https://arxiv.org/html/2608.18669#A2.SS1.p1.5)\.
- Liuet al\.\(2023a\)B\. Liu, Y\. Feng, Q\. Liu, and P\. StoneMetric residual network for sample efficient goal\-conditioned reinforcement learning\.InProceedings of the Thirty\-Seventh AAAI Conference on Artificial Intelligence \(AAAI\),pp\. 8799–8806\.Note:arXiv:2208\.08133Cited by:[§1](https://arxiv.org/html/2608.18669#S1.p4.1),[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px5.p1.1)\.
- Liuet al\.\(2023b\)B\. Liu, Y\. Feng, Q\. Liu, and P\. StoneMetric residual networks for sample efficient goal\-conditioned reinforcement learning\.InProceedings of the AAAI Conference on Artificial Intelligence,Cited by:[§B\.1](https://arxiv.org/html/2608.18669#A2.SS1.p1.2),[§5](https://arxiv.org/html/2608.18669#S5.p2.1)\.
- Maeset al\.\(2026a\)L\. Maes, Q\. Le Lidec, D\. Scieur, Y\. LeCun, and R\. BalestrieroLeWorldModel: stable end\-to\-end joint\-embedding predictive architecture from pixels\.arXiv preprint arXiv:2603\.19312\.External Links:2603\.19312Cited by:[§1](https://arxiv.org/html/2608.18669#S1.p2.1),[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px5.p1.1)\.
- Maeset al\.\(2026b\)L\. Maes, Q\. L\. Lidec, L\. Facury, N\. Massaudi, A\. Chaurasia, F\. Capuano, R\. Gao, T\. Gillin, D\. Haramati, D\. Scieur, Y\. LeCun, and R\. BalestrieroStable\-worldmodel: a platform for reproducible world modeling research and evaluation\.External Links:2605\.21800,[Link](https://arxiv.org/abs/2605.21800)Cited by:[§7](https://arxiv.org/html/2608.18669#S7.SS0.SSS0.Px1.p1.1)\.
- Marinoet al\.\(2021\)J\. Marino, A\. Piché, A\. D\. Ialongo, and Y\. YueIterative amortized policy optimization\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px4.p1.1)\.
- Mnihet al\.\(2015\)V\. Mnih, K\. Kavukcuoglu, D\. Silver,et al\.Human\-level control through deep reinforcement learning\.Nature518\(7540\),pp\. 529–533\.Cited by:[§B\.1](https://arxiv.org/html/2608.18669#A2.SS1.p1.5)\.
- Newey and Powell \(1987\)W\. K\. Newey and J\. L\. PowellAsymmetric least squares estimation and testing\.Econometrica55\(4\),pp\. 819–847\.Cited by:[§B\.1](https://arxiv.org/html/2608.18669#A2.SS1.p1.6)\.
- Padoa\-Schioppa and Assad \(2006\)C\. Padoa\-Schioppa and J\. A\. AssadNeurons in the orbitofrontal cortex encode economic value\.Nature441\(7090\),pp\. 223–226\.External Links:[Document](https://dx.doi.org/10.1038/nature04676)Cited by:[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px5.p1.1)\.
- Parket al\.\(2025\)S\. Park, K\. Frans, B\. Eysenbach, and S\. LevineOGBench: benchmarking offline goal\-conditioned RL\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§7\.3](https://arxiv.org/html/2608.18669#S7.SS3.p1.1)\.
- Parket al\.\(2023\)S\. Park, D\. Ghosh, B\. Eysenbach, and S\. LevineHIQL: offline goal\-conditioned rl with latent states as actions\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§B\.1](https://arxiv.org/html/2608.18669#A2.SS1.p1.6)\.
- Pascanuet al\.\(2017\)R\. Pascanu, Y\. Li, O\. Vinyals, N\. Heess, L\. Buesing, S\. Racanière, D\. Reichert, T\. Weber, D\. Wierstra, and P\. BattagliaLearning model\-based planning from scratch\.arXiv preprint arXiv:1707\.06170\.Cited by:[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px3.p1.1)\.
- Pfeiffer and Foster \(2013\)B\. E\. Pfeiffer and D\. J\. FosterHippocampal place\-cell sequences depict future paths to remembered goals\.Nature497\(7447\),pp\. 74–79\.External Links:[Document](https://dx.doi.org/10.1038/nature12112)Cited by:[§1](https://arxiv.org/html/2608.18669#S1.p1.1)\.
- S Vet al\.\(2023\)J\. S V, S\. Jalagam, Y\. LeCun, and V\. SobalGradient\-based planning with world models\.arXiv preprint arXiv:2312\.17227\.Cited by:[§1](https://arxiv.org/html/2608.18669#S1.p2.1),[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2608.18669#S7.SS0.SSS0.Px2.p1.1)\.
- Sacks and Boots \(2022\)J\. Sacks and B\. BootsLearning to optimize in model predictive control\.InIEEE International Conference on Robotics and Automation \(ICRA\),pp\. 10549–10556\.Cited by:[§C\.5](https://arxiv.org/html/2608.18669#A3.SS5.SSS0.Px5.p1.1),[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2608.18669#S7.SS0.SSS0.Px2.p1.1)\.
- Sackset al\.\(2024\)J\. Sacks, R\. Rana, K\. Huang, A\. Spitzer, G\. Shi, and B\. BootsDeep model predictive optimization\.InIEEE International Conference on Robotics and Automation \(ICRA\),Cited by:[§C\.5](https://arxiv.org/html/2608.18669#A3.SS5.SSS0.Px4.p1.1),[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2608.18669#S7.SS0.SSS0.Px2.p1.1)\.
- Schaulet al\.\(2015\)T\. Schaul, D\. Horgan, K\. Gregor, and D\. SilverUniversal value function approximators\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§B\.1](https://arxiv.org/html/2608.18669#A2.SS1.p1.1),[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px5.p1.1)\.
- Schucket al\.\(2016\)N\. W\. Schuck, M\. B\. Cai, R\. C\. Wilson, and Y\. NivHuman orbitofrontal cortex represents a cognitive map of state space\.Neuron91\(6\),pp\. 1402–1412\.External Links:[Document](https://dx.doi.org/10.1016/j.neuron.2016.08.019)Cited by:[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px5.p1.1)\.
- Smith and Winkler \(2006\)J\. E\. Smith and R\. L\. WinklerThe optimizer’s curse: skepticism and postdecision surprise in decision analysis\.Management Science52\(3\),pp\. 311–322\.External Links:[Document](https://dx.doi.org/10.1287/mnsc.1050.0451)Cited by:[§B\.3](https://arxiv.org/html/2608.18669#A2.SS3.p1.1)\.
- Sobalet al\.\(2025\)V\. Sobal, W\. Zhang, K\. Cho, R\. Balestriero, T\. G\. J\. Rudner, and Y\. LeCunLearning from reward\-free offline data: a case for planning with latent dynamics models\.arXiv preprint arXiv:2502\.14819\.Cited by:[§1](https://arxiv.org/html/2608.18669#S1.p2.1),[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px5.p1.1),[§7](https://arxiv.org/html/2608.18669#S7.SS0.SSS0.Px2.p1.1)\.
- Srinivaset al\.\(2018\)A\. Srinivas, A\. Jabri, P\. Abbeel, S\. Levine, and C\. FinnUniversal planning networks: learning generalizable representations for visuomotor control\.InInternational Conference on Machine Learning \(ICML\),pp\. 4732–4741\.Cited by:[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px1.p1.1)\.
- Sutton and Barto \(2018\)R\. S\. Sutton and A\. G\. BartoReinforcement learning: an introduction\.2nd edition,MIT Press\.Cited by:[§B\.1](https://arxiv.org/html/2608.18669#A2.SS1.p1.3)\.
- Sutton \(1991\)R\. S\. SuttonDyna, an integrated architecture for learning, planning, and reacting\.ACM SIGART Bulletin2\(4\),pp\. 160–163\.External Links:[Document](https://dx.doi.org/10.1145/122344.122377)Cited by:[§B\.3](https://arxiv.org/html/2608.18669#A2.SS3.p2.1),[§7\.4](https://arxiv.org/html/2608.18669#S7.SS4.p1.1)\.
- Talvitie \(2017\)E\. TalvitieSelf\-correcting models for model\-based reinforcement learning\.InProceedings of the Thirty\-First AAAI Conference on Artificial Intelligence \(AAAI\),pp\. 2597–2603\.Cited by:[§B\.3](https://arxiv.org/html/2608.18669#A2.SS3.p1.1)\.
- Tolman \(1948\)E\. C\. TolmanCognitive maps in rats and men\.Psychological Review55\(4\),pp\. 189–208\.Cited by:[§1](https://arxiv.org/html/2608.18669#S1.p1.1)\.
- Wang and Isola \(2022\)T\. Wang and P\. IsolaImproved representation of asymmetrical distances with interval quasimetric embeddings\.InNeurIPS Workshop on Symmetry and Geometry in Neural Representations,Cited by:[§B\.1](https://arxiv.org/html/2608.18669#A2.SS1.p1.2)\.
- Wanget al\.\(2023a\)T\. Wang, A\. Torralba, P\. Isola, and A\. ZhangOptimal goal\-reaching reinforcement learning via quasimetric learning\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§B\.1](https://arxiv.org/html/2608.18669#A2.SS1.p1.3),[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px5.p1.1)\.
- Wanget al\.\(2023b\)T\. Wang, A\. Torralba, P\. Isola, and A\. ZhangOptimal goal\-reaching reinforcement learning via quasimetric learning\.InInternational Conference on Machine Learning \(ICML\),pp\. 36411–36430\.Cited by:[§1](https://arxiv.org/html/2608.18669#S1.p4.1),[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px5.p1.1)\.
- Wilsonet al\.\(2014\)R\. C\. Wilson, Y\. K\. Takahashi, G\. Schoenbaum, and Y\. NivOrbitofrontal cortex as a cognitive map of task space\.Neuron81\(2\),pp\. 267–279\.External Links:[Document](https://dx.doi.org/10.1016/j.neuron.2013.11.005)Cited by:[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px5.p1.1)\.
- Zhouet al\.\(2025\)G\. Zhou, H\. Pan, Y\. LeCun, and L\. PintoDINO\-WM: world models on pre\-trained visual features enable zero\-shot planning\.InInternational Conference on Machine Learning \(ICML\),Note:arXiv:2411\.04983Cited by:[§1](https://arxiv.org/html/2608.18669#S1.p2.1),[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.18669#S2.SS0.SSS0.Px5.p1.1),[§7](https://arxiv.org/html/2608.18669#S7.SS0.SSS0.Px2.p1.1)\.

## Appendix AProofs of Theoretical Results

We formalize two motivations for Reinforced Planning\. First, predictive world\-model learning does not determine a Euclidean latent geometry suitable for planning\. Second, a learned neural planner can adapt its optimization rule to the task, whereas conventional optimizers use one configuration across the task distribution\.

### A\.1Proof of Theorem[1](https://arxiv.org/html/2608.18669#Thmtheorem1)

###### Proof\.

LetA∈ℝd×dA\\in\\mathbb\{R\}^\{d\\times d\}be invertible and define

EA​\(s\)=A​Eϕ​\(s\),hA​\(z,a\)=A​hϕ​\(A−1​z,a\)\.E\_\{A\}\(s\)=AE\_\{\\phi\}\(s\),\\qquad h\_\{A\}\(z,a\)=Ah\_\{\\phi\}\(A^\{\-1\}z,a\)\.\(18\)Then

hA​\(EA​\(s\),a\)\\displaystyle h\_\{A\}\(E\_\{A\}\(s\),a\)=A​hϕ​\(A−1​A​Eϕ​\(s\),a\)\\displaystyle=Ah\_\{\\phi\}\(A^\{\-1\}AE\_\{\\phi\}\(s\),a\)\(19\)=A​hϕ​\(Eϕ​\(s\),a\)\\displaystyle=Ah\_\{\\phi\}\(E\_\{\\phi\}\(s\),a\)\(20\)=A​Eϕ​\(𝒯⁡\(s,a\)\)\\displaystyle=AE\_\{\\phi\}\(\\mathcal\{T\}\(s,a\)\)\(21\)=EA​\(𝒯​\(s,a\)\)\.\\displaystyle=E\_\{A\}\(\\mathcal\{T\}\(s,a\)\)\.\(22\)Thus,\(EA,hA\)\(E\_\{A\},h\_\{A\}\)is exact whenever\(Eϕ,hϕ\)\(E\_\{\\phi\},h\_\{\\phi\}\)is exact\. The same argument applied recursively shows that all multi\-step trajectories remain exactly predicted under the transformed coordinates\.

Now define

u=Eϕ​\(g1\)−Eϕ​\(s\),v=Eϕ​\(g2\)−Eϕ​\(s\)\.u=E\_\{\\phi\}\(g\_\{1\}\)\-E\_\{\\phi\}\(s\),\\qquad v=E\_\{\\phi\}\(g\_\{2\}\)\-E\_\{\\phi\}\(s\)\.\(23\)Becauseuuandvvare linearly independent, there exists an invertible matrixBBsuch that

B​u=e1,B​v=e2,Bu=e\_\{1\},\\qquad Bv=e\_\{2\},\(24\)wheree1e\_\{1\}ande2e\_\{2\}are the first two standard basis vectors\.

For anyR\>1R\>1, define

A1\\displaystyle A\_\{1\}=diag⁡\(R,1,…,1\)​B,\\displaystyle=\\operatorname\{diag\}\(R,1,\\ldots,1\)B,\(25\)A2\\displaystyle A\_\{2\}=diag⁡\(1,R,1,…,1\)​B\.\\displaystyle=\\operatorname\{diag\}\(1,R,1,\\ldots,1\)B\.\(26\)Both matrices are invertible and therefore induce exact latent world models\. Under the first transformation,

∥EA1​\(g1\)−EA1​\(s\)∥2∥EA1​\(g2\)−EA1​\(s\)∥2=R,\\frac\{\\lVert E\_\{A\_\{1\}\}\(g\_\{1\}\)\-E\_\{A\_\{1\}\}\(s\)\\rVert\_\{2\}\}\{\\lVert E\_\{A\_\{1\}\}\(g\_\{2\}\)\-E\_\{A\_\{1\}\}\(s\)\\rVert\_\{2\}\}=R,\(27\)whereas under the second,

∥EA2​\(g1\)−EA2​\(s\)∥2∥EA2​\(g2\)−EA2​\(s\)∥2=1R\.\\frac\{\\lVert E\_\{A\_\{2\}\}\(g\_\{1\}\)\-E\_\{A\_\{2\}\}\(s\)\\rVert\_\{2\}\}\{\\lVert E\_\{A\_\{2\}\}\(g\_\{2\}\)\-E\_\{A\_\{2\}\}\(s\)\\rVert\_\{2\}\}=\\frac\{1\}\{R\}\.\(28\)The two exact world models therefore induce opposite Euclidean distance orderings\. SinceRRis arbitrary, the separation between the distances can be made arbitrarily large\. ∎

### A\.2Temporal reachability as a directed distance

Consider a deterministic controlled system with state space𝒮\\mathcal\{S\}\. Define

d⋆​\(s,g\):=inf\{T∈ℕ0:some length\-Taction sequence takesstog\},d^\{\\star\}\(s,g\):=\\inf\\left\\\{T\\in\\mathbb\{N\}\_\{0\}:\\text\{some length\-$T$ action sequence takes $s$ to $g$\}\\right\\\},\(29\)withd⋆​\(s,g\)=\+∞d^\{\\star\}\(s,g\)=\+\\inftywhenggis unreachable fromss\.

###### Proposition 1\(Temporal reachability is an extended directed quasimetric\)\.

For all statess,y,gs,y,g,

d⋆​\(s,g\)≥0,d⋆​\(s,g\)=0⇔s=g,d⋆​\(s,g\)≤d⋆​\(s,y\)\+d⋆​\(y,g\)\.d^\{\\star\}\(s,g\)\\geq 0,\\qquad d^\{\\star\}\(s,g\)=0\\iff s=g,\\qquad d^\{\\star\}\(s,g\)\\leq d^\{\\star\}\(s,y\)\+d^\{\\star\}\(y,g\)\.\(30\)However,d⋆​\(s,g\)d^\{\\star\}\(s,g\)need not equald⋆​\(g,s\)d^\{\\star\}\(g,s\)\. Consequently, when temporal reachability is asymmetric, no symmetric distance such as a latent norm∥E⁡\(s\)−E⁡\(g\)∥2\\lVert E\(s\)\-E\(g\)\\rVert\_\{2\}can represent it exactly on all ordered state pairs\.

###### Proof\.

The empty action sequence takes each state to itself, sod⋆​\(s,s\)=0d^\{\\star\}\(s,s\)=0\. Conversely, a length\-zero sequence cannot change the state, sod⋆​\(s,g\)=0d^\{\\star\}\(s,g\)=0impliess=gs=g\. Nonnegativity follows because action sequence lengths belong toℕ0\\mathbb\{N\}\_\{0\}\.

The triangle inequality is immediate if eitherd⋆​\(s,y\)d^\{\\star\}\(s,y\)ord⋆​\(y,g\)d^\{\\star\}\(y,g\)is infinite\. Otherwise, concatenate a shortest sequence fromsstoyywith a shortest sequence fromyytogg\. The resulting sequence takessstoggand has lengthd⋆​\(s,y\)\+d⋆​\(y,g\)d^\{\\star\}\(s,y\)\+d^\{\\star\}\(y,g\)\.

Finally, consider two states for which an action takessstogg, but no action sequence returns fromggtoss\. Thend⋆​\(s,g\)=1d^\{\\star\}\(s,g\)=1whiled⋆​\(g,s\)=\+∞d^\{\\star\}\(g,s\)=\+\\infty\. Because every symmetric distance assigns the same value to\(s,g\)\(s,g\)and\(g,s\)\(g,s\), it cannot representd⋆d^\{\\star\}exactly in this case\. ∎

### A\.3Formal Statement and Proof of Theorem[2](https://arxiv.org/html/2608.18669#Thmtheorem2)

We first make the ambient spaces and regularity conditions explicit\. Let

𝒵⊆ℝdz,𝒳⊆𝒵×𝒵⊆ℝ2​dz,Ωpl⊆ℝdω,𝒴⊆ℝdy\.\\mathcal\{Z\}\\subseteq\\mathbb\{R\}^\{d\_\{z\}\},\\qquad\\mathcal\{X\}\\subseteq\\mathcal\{Z\}\\times\\mathcal\{Z\}\\subseteq\\mathbb\{R\}^\{2d\_\{z\}\},\\qquad\\Omega\_\{\\mathrm\{pl\}\}\\subseteq\\mathbb\{R\}^\{d\_\{\\omega\}\},\\qquad\\mathcal\{Y\}\\subseteq\\mathbb\{R\}^\{d\_\{y\}\}\.\(31\)Here,𝒳\\mathcal\{X\}is the task space, withx=\(zt,zg\)∈𝒳x=\(z\_\{t\},z\_\{g\}\)\\in\\mathcal\{X\},Ωpl\\Omega\_\{\\mathrm\{pl\}\}is the planner\-state space, and𝒴\\mathcal\{Y\}is the output space of the planner interface\. We assume that𝒳\\mathcal\{X\}andΩpl\\Omega\_\{\\mathrm\{pl\}\}are nonempty and compact, and equip all finite\-dimensional spaces and product spaces with their Euclidean norms\.

Letμ\\mube a probability distribution supported on𝒳\\mathcal\{X\}, letω0∈Ωpl\\omega\_\{0\}\\in\\Omega\_\{\\mathrm\{pl\}\}be the common planner initialization, and letK<∞K<\\inftybe the number of refinement rounds\. Assume that

ℐ:𝒳×Ωpl⟶𝒴\\mathcal\{I\}:\\mathcal\{X\}\\times\\Omega\_\{\\mathrm\{pl\}\}\\longrightarrow\\mathcal\{Y\}\(32\)and

ℓ:𝒳×Ωpl⟶ℝ\\ell:\\mathcal\{X\}\\times\\Omega\_\{\\mathrm\{pl\}\}\\longrightarrow\\mathbb\{R\}\(33\)are continuous\.

Define the set of feasible planner inputs by

𝒲=\{\(ω,ℐ\(x,ω\)\):x∈𝒳,ω∈Ωpl\}⊆ℝdω\+dy\.\\mathcal\{W\}=\\left\\\{\\left\(\\omega,\\mathcal\{I\}\(x,\\omega\)\\right\):x\\in\\mathcal\{X\},\\;\\omega\\in\\Omega\_\{\\mathrm\{pl\}\}\\right\\\}\\subseteq\\mathbb\{R\}^\{d\_\{\\omega\}\+d\_\{y\}\}\.\(34\)Because𝒳×Ωpl\\mathcal\{X\}\\times\\Omega\_\{\\mathrm\{pl\}\}is compact and\(x,ω\)↦\(ω,ℐ⁡\(x,ω\)\)\(x,\\omega\)\\mapsto\(\\omega,\\mathcal\{I\}\(x,\\omega\)\)is continuous,𝒲\\mathcal\{W\}is compact\.

Let

𝔉ℐ=\{F:𝒲→Ωpl:Fis continuous\}\\mathfrak\{F\}\_\{\\mathcal\{I\}\}=\\left\\\{F:\\mathcal\{W\}\\to\\Omega\_\{\\mathrm\{pl\}\}\\;:\\;F\\text\{ is continuous\}\\right\\\}\(35\)be the class of continuous feasible search rules\. For anyF∈𝔉ℐF\\in\\mathfrak\{F\}\_\{\\mathcal\{I\}\}, setωF,0​\(x\)=ω0,\\omega\_\{F,0\}\(x\)=\\omega\_\{0\},and, fork=0,…,K−1k=0,\\ldots,K\-1,

ωF,k\+1​\(x\)=F⁡\(ωF,k​\(x\),ℐ⁡\(x,ωF,k​\(x\)\)\)\.\\omega\_\{F,k\+1\}\(x\)=F\\\!\\left\(\\omega\_\{F,k\}\(x\),\\mathcal\{I\}\\bigl\(x,\\omega\_\{F,k\}\(x\)\\bigr\)\\right\)\.\(36\)Since every rule maps𝒲\\mathcal\{W\}intoΩpl\\Omega\_\{\\mathrm\{pl\}\}, all planner states remain feasible\.

The expected loss ofFFis

ℒ⁡\(F\)=𝔼x∼μ​\[ℓ⁡\(x,ωF,K​\(x\)\)\]\.\\mathcal\{L\}\(F\)=\\mathbb\{E\}\_\{x\\sim\\mu\}\\left\[\\ell\\bigl\(x,\\omega\_\{F,K\}\(x\)\\bigr\)\\right\]\.\(37\)Assumption[1](https://arxiv.org/html/2608.18669#Thmassumption1)states that\{ℱθ:θ∈Θ\}⊆𝔉ℐ\\\{\\mathcal\{F\}\_\{\\theta\}:\\theta\\in\\Theta\\\}\\subseteq\\mathfrak\{F\}\_\{\\mathcal\{I\}\}and that, for everyF∈𝔉ℐF\\in\\mathfrak\{F\}\_\{\\mathcal\{I\}\}and everyε\>0\\varepsilon\>0, there existsθ∈Θ\\theta\\in\\Thetasatisfying

supw∈𝒲‖ℱθ​\(w\)−F⁡\(w\)‖<ε\.\\sup\_\{w\\in\\mathcal\{W\}\}\\left\\\|\\mathcal\{F\}\_\{\\theta\}\(w\)\-F\(w\)\\right\\\|<\\varepsilon\.\(38\)
Let𝒳1,…,𝒳r\\mathcal\{X\}\_\{1\},\\ldots,\\mathcal\{X\}\_\{r\}be a measurable partition of𝒳\\mathcal\{X\}, up to sets ofμ\\mu\-measure zero, with

pi=μ⁡\(𝒳i\)\>0\.p\_\{i\}=\\mu\(\\mathcal\{X\}\_\{i\}\)\>0\.\(39\)For everyF∈𝔉ℐF\\in\\mathfrak\{F\}\_\{\\mathcal\{I\}\}, define

ℒi​\(F\)=𝔼⁡\[ℓ⁡\(x,ωF,K​\(x\)\)\|x∈𝒳i\]\.\\mathcal\{L\}\_\{i\}\(F\)=\\mathbb\{E\}\\left\[\\ell\\bigl\(x,\\omega\_\{F,K\}\(x\)\\bigr\)\\,\\middle\|\\,x\\in\\mathcal\{X\}\_\{i\}\\right\]\.\(40\)
Let

ℬ⊆ℝdB\\mathcal\{B\}\\subseteq\\mathbb\{R\}^\{d\_\{B\}\}\(41\)be a nonempty compact family of fixed search configurations\. EachB∈ℬB\\in\\mathcal\{B\}induces a ruleFB∈𝔉ℐF\_\{B\}\\in\\mathfrak\{F\}\_\{\\mathcal\{I\}\}, with the same configurationBBused on every task\. Define

ℒi​\(B\)=ℒi​\(FB\),ℒ⁡\(B\)=∑i=1rpi​ℒi​\(B\),\\mathcal\{L\}\_\{i\}\(B\)=\\mathcal\{L\}\_\{i\}\(F\_\{B\}\),\\qquad\\mathcal\{L\}\(B\)=\\sum\_\{i=1\}^\{r\}p\_\{i\}\\mathcal\{L\}\_\{i\}\(B\),\(42\)and assume thatB↦ℒi​\(B\)B\\mapsto\\mathcal\{L\}\_\{i\}\(B\)is continuous for every regionii\. Consequently, the regional minimum

bi:=minB∈ℬ⁡ℒi​\(B\)b\_\{i\}:=\\min\_\{B\\in\\mathcal\{B\}\}\\mathcal\{L\}\_\{i\}\(B\)\(43\)exists for everyii\.

###### Assumption 2\(Regional incompatibility\)\.

No fixed configuration minimizes every regional loss:

⋂i=1rargminB∈ℬℒi​\(B\)=∅\.\\bigcap\_\{i=1\}^\{r\}\\operatorname\*\{argmin\}\_\{B\\in\\mathcal\{B\}\}\\mathcal\{L\}\_\{i\}\(B\)=\\varnothing\.\(44\)

###### Assumption 3\(Interface composability\)\.

There exists a continuous feasible ruleF⋆∈𝔉ℐF^\{\\star\}\\in\\mathfrak\{F\}\_\{\\mathcal\{I\}\}that attains the best fixed\-configuration loss in every region:

ℒi\(F⋆\)=bi=minB∈ℬℒi\(B\),i=1,…,r\.\\mathcal\{L\}\_\{i\}\(F^\{\\star\}\)=b\_\{i\}=\\min\_\{B\\in\\mathcal\{B\}\}\\mathcal\{L\}\_\{i\}\(B\),\\qquad i=1,\\ldots,r\.\(45\)

###### Theorem 3\(Strict advantage under task heterogeneity; restatement of Theorem[2](https://arxiv.org/html/2608.18669#Thmtheorem2)\)\.

Under Assumption[1](https://arxiv.org/html/2608.18669#Thmassumption1), Assumption[2](https://arxiv.org/html/2608.18669#Thmassumption2), and Assumption[3](https://arxiv.org/html/2608.18669#Thmassumption3),

infθ∈Θℒ⁡\(ℱθ\)≤∑i=1rpi​minB∈ℬ​ℒi​\(B\)<minB∈ℬ⁡ℒ⁡\(B\)\.\\inf\_\{\\theta\\in\\Theta\}\\mathcal\{L\}\(\\mathcal\{F\}\_\{\\theta\}\)\\leq\\sum\_\{i=1\}^\{r\}p\_\{i\}\\min\_\{B\\in\\mathcal\{B\}\}\\mathcal\{L\}\_\{i\}\(B\)<\\min\_\{B\\in\\mathcal\{B\}\}\\mathcal\{L\}\(B\)\.\(46\)

###### Proof\.

By Assumption[3](https://arxiv.org/html/2608.18669#Thmassumption3), there existsF⋆∈𝔉ℐF^\{\\star\}\\in\\mathfrak\{F\}\_\{\\mathcal\{I\}\}such thatℒi​\(F⋆\)=bi\\mathcal\{L\}\_\{i\}\(F^\{\\star\}\)=b\_\{i\}for everyii\. Since the regions partition the task distribution,

ℒ⁡\(F⋆\)=∑i=1rpi​ℒi​\(F⋆\)=∑i=1rpi​bi\.\\mathcal\{L\}\(F^\{\\star\}\)=\\sum\_\{i=1\}^\{r\}p\_\{i\}\\mathcal\{L\}\_\{i\}\(F^\{\\star\}\)=\\sum\_\{i=1\}^\{r\}p\_\{i\}b\_\{i\}\.\(47\)
We next show that the neural\-planner class can approach this loss\. For each integerm≥1m\\geq 1, apply Assumption[1](https://arxiv.org/html/2608.18669#Thmassumption1)withε=1/m\\varepsilon=1/m\. This gives a parameterθm∈Θ\\theta\_\{m\}\\in\\Thetasatisfying

supw∈𝒲‖ℱθm​\(w\)−F⋆​\(w\)‖<1m\.\\sup\_\{w\\in\\mathcal\{W\}\}\\left\\\|\\mathcal\{F\}\_\{\\theta\_\{m\}\}\(w\)\-F^\{\\star\}\(w\)\\right\\\|<\\frac\{1\}\{m\}\.\(48\)For brevity, write

ωm,k​\(x\)=ωℱθm,k​\(x\),ωk⋆​\(x\)=ωF⋆,k​\(x\)\.\\omega\_\{m,k\}\(x\)=\\omega\_\{\\mathcal\{F\}\_\{\\theta\_\{m\}\},k\}\(x\),\\qquad\\omega^\{\\star\}\_\{k\}\(x\)=\\omega\_\{F^\{\\star\},k\}\(x\)\.\(49\)We prove by induction that, for every fixedk≤Kk\\leq K,

supx∈𝒳‖ωm,k​\(x\)−ωk⋆​\(x\)‖⟶0as​m→∞\.\\sup\_\{x\\in\\mathcal\{X\}\}\\left\\\|\\omega\_\{m,k\}\(x\)\-\\omega^\{\\star\}\_\{k\}\(x\)\\right\\\|\\longrightarrow 0\\qquad\\text\{as \}m\\to\\infty\.\(50\)
The claim holds fork=0k=0, because all planners share the initializationω0\\omega\_\{0\}\. Suppose that it holds at roundkk\. Define

wm,k​\(x\)\\displaystyle w\_\{m,k\}\(x\)=\(ωm,k​\(x\),ℐ⁡\(x,ωm,k​\(x\)\)\),\\displaystyle=\\left\(\\omega\_\{m,k\}\(x\),\\mathcal\{I\}\\bigl\(x,\\omega\_\{m,k\}\(x\)\\bigr\)\\right\),\(51\)wk⋆​\(x\)\\displaystyle w^\{\\star\}\_\{k\}\(x\)=\(ωk⋆​\(x\),ℐ⁡\(x,ωk⋆​\(x\)\)\)\.\\displaystyle=\\left\(\\omega^\{\\star\}\_\{k\}\(x\),\\mathcal\{I\}\\bigl\(x,\\omega^\{\\star\}\_\{k\}\(x\)\\bigr\)\\right\)\.\(52\)Continuity ofℐ\\mathcal\{I\}on the compact set𝒳×Ωpl\\mathcal\{X\}\\times\\Omega\_\{\\mathrm\{pl\}\}implies uniform continuity\. Therefore, the induction hypothesis gives

supx∈𝒳‖wm,k​\(x\)−wk⋆​\(x\)‖⟶0\.\\sup\_\{x\\in\\mathcal\{X\}\}\\left\\\|w\_\{m,k\}\(x\)\-w^\{\\star\}\_\{k\}\(x\)\\right\\\|\\longrightarrow 0\.\(53\)
Using the planner recursion and adding and subtractingF⋆​\(wm,k​\(x\)\)F^\{\\star\}\(w\_\{m,k\}\(x\)\), we obtain

supx∈𝒳‖ωm,k\+1​\(x\)−ωk\+1⋆​\(x\)‖\\displaystyle\\sup\_\{x\\in\\mathcal\{X\}\}\\left\\\|\\omega\_\{m,k\+1\}\(x\)\-\\omega^\{\\star\}\_\{k\+1\}\(x\)\\right\\\|\(54\)≤supx∈𝒳‖ℱθm​\(wm,k​\(x\)\)−F⋆​\(wm,k​\(x\)\)‖\\displaystyle\\quad\\leq\\sup\_\{x\\in\\mathcal\{X\}\}\\left\\\|\\mathcal\{F\}\_\{\\theta\_\{m\}\}\\bigl\(w\_\{m,k\}\(x\)\\bigr\)\-F^\{\\star\}\\bigl\(w\_\{m,k\}\(x\)\\bigr\)\\right\\\|\+supx∈𝒳‖F⋆\(wm,k\(x\)\)−F⋆\(wk⋆\(x\)\)‖\.\\displaystyle\\qquad\\quad\+\\sup\_\{x\\in\\mathcal\{X\}\}\\left\\\|F^\{\\star\}\\bigl\(w\_\{m,k\}\(x\)\\bigr\)\-F^\{\\star\}\\bigl\(w^\{\\star\}\_\{k\}\(x\)\\bigr\)\\right\\\|\.\(55\)The first term is at most1/m1/mby Eq\.[48](https://arxiv.org/html/2608.18669#A1.E48)\. The second converges to zero becauseF⋆F^\{\\star\}is uniformly continuous on the compact set𝒲\\mathcal\{W\}and Eq\.[53](https://arxiv.org/html/2608.18669#A1.E53)holds\. This proves Eq\.[50](https://arxiv.org/html/2608.18669#A1.E50)for every finitek≤Kk\\leq K\.

Becauseℓ\\ellis continuous on the compact set𝒳×Ωpl\\mathcal\{X\}\\times\\Omega\_\{\\mathrm\{pl\}\}, it is uniformly continuous\. Applying Eq\.[50](https://arxiv.org/html/2608.18669#A1.E50)atk=Kk=Ktherefore yields

supx∈𝒳\|ℓ⁡\(x,ωm,K​\(x\)\)−ℓ⁡\(x,ωK⋆​\(x\)\)\|⟶0\.\\sup\_\{x\\in\\mathcal\{X\}\}\\left\|\\ell\\bigl\(x,\\omega\_\{m,K\}\(x\)\\bigr\)\-\\ell\\bigl\(x,\\omega^\{\\star\}\_\{K\}\(x\)\\bigr\)\\right\|\\longrightarrow 0\.\(56\)Consequently,

ℒ⁡\(ℱθm\)⟶ℒ⁡\(F⋆\)\.\\mathcal\{L\}\(\\mathcal\{F\}\_\{\\theta\_\{m\}\}\)\\longrightarrow\\mathcal\{L\}\(F^\{\\star\}\)\.\(57\)Together with Eq\.[47](https://arxiv.org/html/2608.18669#A1.E47), this gives

infθ∈Θℒ⁡\(ℱθ\)≤∑i=1rpi​bi\.\\inf\_\{\\theta\\in\\Theta\}\\mathcal\{L\}\(\\mathcal\{F\}\_\{\\theta\}\)\\leq\\sum\_\{i=1\}^\{r\}p\_\{i\}b\_\{i\}\.\(58\)
It remains to show that every single fixed configuration has strictly larger expected loss\. Define its excess over the regional minima by

Δ⁡\(B\)=∑i=1rpi​\(ℒi​\(B\)−bi\)\.\\Delta\(B\)=\\sum\_\{i=1\}^\{r\}p\_\{i\}\\bigl\(\\mathcal\{L\}\_\{i\}\(B\)\-b\_\{i\}\\bigr\)\.\(59\)Every term in this sum is nonnegative\. By Assumption[2](https://arxiv.org/html/2608.18669#Thmassumption2), eachB∈ℬB\\in\\mathcal\{B\}is strictly suboptimal in at least one region\. Since everypi\>0p\_\{i\}\>0,

Δ⁡\(B\)\>0for every​B∈ℬ\.\\Delta\(B\)\>0\\qquad\\text\{for every \}B\\in\\mathcal\{B\}\.\(60\)The functionΔ\\Deltais continuous because it is a finite weighted sum of the continuous functionsℒi\\mathcal\{L\}\_\{i\}\. Sinceℬ\\mathcal\{B\}is compact,Δ\\Deltaattains its minimum\. Its pointwise strict positivity implies

η:=minB∈ℬ⁡Δ⁡\(B\)\>0\.\\eta:=\\min\_\{B\\in\\mathcal\{B\}\}\\Delta\(B\)\>0\.\(61\)Hence

minB∈ℬ⁡ℒ⁡\(B\)\\displaystyle\\min\_\{B\\in\\mathcal\{B\}\}\\mathcal\{L\}\(B\)=minB∈ℬ⁡\[∑i=1rpi​bi\+Δ⁡\(B\)\]\\displaystyle=\\min\_\{B\\in\\mathcal\{B\}\}\\left\[\\sum\_\{i=1\}^\{r\}p\_\{i\}b\_\{i\}\+\\Delta\(B\)\\right\]\(62\)=∑i=1rpi​bi\+η\\displaystyle=\\sum\_\{i=1\}^\{r\}p\_\{i\}b\_\{i\}\+\\eta\(63\)\>∑i=1rpi​bi\.\\displaystyle\>\\sum\_\{i=1\}^\{r\}p\_\{i\}b\_\{i\}\.\(64\)Combining Eq\.[58](https://arxiv.org/html/2608.18669#A1.E58)with Eq\.[64](https://arxiv.org/html/2608.18669#A1.E64)proves

infθ∈Θℒ⁡\(ℱθ\)≤∑i=1rpi​minB∈ℬ​ℒi​\(B\)<minB∈ℬ⁡ℒ⁡\(B\)\.\\inf\_\{\\theta\\in\\Theta\}\\mathcal\{L\}\(\\mathcal\{F\}\_\{\\theta\}\)\\leq\\sum\_\{i=1\}^\{r\}p\_\{i\}\\min\_\{B\\in\\mathcal\{B\}\}\\mathcal\{L\}\_\{i\}\(B\)<\\min\_\{B\\in\\mathcal\{B\}\}\\mathcal\{L\}\(B\)\.\(65\)∎

## Appendix BMethod Details

### B\.1Value Learning

For each environment and world model, we train a separate goal\-conditioned cost\-to\-go function\[[21](https://arxiv.org/html/2608.18669#bib.bib37),[41](https://arxiv.org/html/2608.18669#bib.bib36)\]

Vψ​\(z,zg\):𝒵×𝒵→ℝ≥0\.V\_\{\\psi\}\(z,z\_\{g\}\)\\colon\\mathcal\{Z\}\\times\\mathcal\{Z\}\\rightarrow\\mathbb\{R\}\_\{\\geq 0\}\.\(66\)Lower values represent shorter predicted temporal distance\[[14](https://arxiv.org/html/2608.18669#bib.bib38)\]to the goal, as we assume a cost of11per step\. The world\-model encoder is frozen, and the value function is trained entirely from cached offline latents\. The value is represented by a metric residual network\[[27](https://arxiv.org/html/2608.18669#bib.bib32),[50](https://arxiv.org/html/2608.18669#bib.bib33)\],

Vψ​\(z,zg\)=∥uψ​\(z\)−uψ​\(zg\)∥2\+maxj⁡ReLU⁡\(vψ,j​\(zg\)−vψ,j​\(z\)\)\.V\_\{\\psi\}\(z,z\_\{g\}\)=\\lVert u\_\{\\psi\}\(z\)\-u\_\{\\psi\}\(z\_\{g\}\)\\rVert\_\{2\}\+\\max\_\{j\}\\operatorname\{ReLU\}\\left\(v\_\{\\psi,j\}\(z\_\{g\}\)\-v\_\{\\psi,j\}\(z\)\\right\)\.\(67\)Hereuψu\_\{\\psi\}is the first half of the latent vector the critic head computes andvψv\_\{\\psi\}is the second half\. The first term is symmetric, while the second permits directed temporal distance\[[51](https://arxiv.org/html/2608.18669#bib.bib34)\]\. For each update, we sample an anchorztz\_\{t\}, annn\-step successorzt\+neffz\_\{t\+n\_\{\\mathrm\{eff\}\}\}, and a hindsight goal\[[2](https://arxiv.org/html/2608.18669#bib.bib35)\]zgz\_\{g\}, whereneff=min⁡\{n,Tepisode−t\}\.n\_\{\\mathrm\{eff\}\}=\\min\\\{n,T\_\{\\mathrm\{episode\}\}\-t\\\}\.In\-episode goals are sampled from future states with temporal offsets balanced across the available episode horizon\. Cross\-episode goals are additionally sampled to train long\-range state pairs\. If an in\-episode goal lies within the backup window, its exact temporal distanceδ\\deltais used\. Otherwise, the target is bootstrapped with annn\-step backup\[[46](https://arxiv.org/html/2608.18669#bib.bib42)\]:

yt=\{δ,δ≤neff,cγ​\(neff\)\+γneff​V¯ψ¯​\(zt\+neff,zg\),otherwise,y\_\{t\}=\\begin\{cases\}\\delta,&\\delta\\leq n\_\{\\mathrm\{eff\}\},\\\\\[3\.0pt\] c\_\{\\gamma\}\(n\_\{\\mathrm\{eff\}\}\)\+\\gamma^\{n\_\{\\mathrm\{eff\}\}\}\\bar\{V\}\_\{\\bar\{\\psi\}\}\(z\_\{t\+n\_\{\\mathrm\{eff\}\}\},z\_\{g\}\),&\\text\{otherwise\},\\end\{cases\}\(68\)with

cγ​\(n\)=∑i=0n−1γi=\{n,γ=1,1−γn1−γ,γ<1\.c\_\{\\gamma\}\(n\)=\\sum\_\{i=0\}^\{n\-1\}\\gamma^\{i\}=\\begin\{cases\}n,&\\gamma=1,\\\\\[2\.0pt\] \\dfrac\{1\-\\gamma^\{n\}\}\{1\-\\gamma\},&\\gamma<1\.\\end\{cases\}\(69\)The target parameters\[[31](https://arxiv.org/html/2608.18669#bib.bib40)\]are updated by Polyak averaging\[[25](https://arxiv.org/html/2608.18669#bib.bib39)\],

ψ¯←\(1−η\)​ψ¯\+η​ψ\.\\bar\{\\psi\}\\leftarrow\(1\-\\eta\)\\bar\{\\psi\}\+\\eta\\psi\.\(70\)Following implicit Q\-learning\[[22](https://arxiv.org/html/2608.18669#bib.bib30),[35](https://arxiv.org/html/2608.18669#bib.bib43)\], the value function is trained by asymmetric expectile regression\[[32](https://arxiv.org/html/2608.18669#bib.bib31)\], replacing the squared penalty with a Huber penalty\[[17](https://arxiv.org/html/2608.18669#bib.bib41)\]for robustness:

ℒV\(ψ\)=𝔼\(zt,zg\)∼𝒟\[\|τ−1\[Vψ\(zt,zg\)−yt\>0\]\|ℓHuber\(Vψ\(zt,zg\)−yt\)\]\.\\mathcal\{L\}\_\{V\}\(\\psi\)=\\mathbb\{E\}\_\{\(z\_\{t\},z\_\{g\}\)\\sim\\mathcal\{D\}\}\\Bigl\[\\bigl\|\\tau\-\\textbf\{1\}\\\!\\left\[V\_\{\\psi\}\(z\_\{t\},z\_\{g\}\)\-y\_\{t\}\>0\\right\]\\bigr\|\\,\\ell\_\{\\mathrm\{Huber\}\}\\\!\\left\(V\_\{\\psi\}\(z\_\{t\},z\_\{g\}\)\-y\_\{t\}\\right\)\\Bigr\]\.\(71\)SinceVψV\_\{\\psi\}is a cost\-to\-go rather than a return, we useτ<0\.5\\tau<0\.5: the weight on overestimation exceeds the weight on underestimation, soVψV\_\{\\psi\}regresses toward a lower expectile of the target distribution, approximating the shortest temporal distance realizable in the data rather than the behavior\-policy average\.

### B\.2RP1 Training

RP1 is trained entirely offline while the world\-model encoder and dynamics predictor remain frozen\. For each world model, the planner is trained from a stride\-five latent cache aligned with five\-step action blocks\. RP1 consists of three fully\-connected layers with ReLU activations and hidden width512512, mappingℝ2​N​\|a\|\+1→ℝ512→ℝ512→ℝN​\|a\|\\mathbb\{R\}^\{2N\|a\|\+1\}\\rightarrow\\mathbb\{R\}^\{512\}\\rightarrow\\mathbb\{R\}^\{512\}\\rightarrow\\mathbb\{R\}^\{N\|a\|\}, where the input concatenates the flattened plan𝐚k∈ℝN​\|a\|\\mathbf\{a\}\_\{k\}\\in\\mathbb\{R\}^\{N\|a\|\}, its value gradient𝐠k∈ℝN​\|a\|\\mathbf\{g\}\_\{k\}\\in\\mathbb\{R\}^\{N\|a\|\}, and the scalar terminal valuevkv\_\{k\}, and the output is the residual plan update\. For example in OGBench Cube, with a planning horizon ofN=5N=5action blocks and\|a\|=25\|a\|=25\(five primitive steps of the five\-dimensional arm actions\), the refiner is251→512→512→125251\\rightarrow 512\\rightarrow 512\\rightarrow 125, i\.e\.0\.460\.46M parameters, applied with tied weights at allK=8K=8refinement steps\.

The RP1 actor is a weight\-tied residual plan refiner\. At refinement stepkk, it receives the current plan, its terminal value, and the value gradient with respect to the plan:

z^N\(k\)\\displaystyle\\hat\{z\}\_\{N\}^\{\(k\)\}=Hϕ​\(𝐚k,z0\),\\displaystyle=H\_\{\\phi\}\(\\mathbf\{a\}\_\{k\},z\_\{0\}\),\(72\)vk\\displaystyle v\_\{k\}=Vψ¯​\(z^N\(k\),zg\),\\displaystyle=V\_\{\\bar\{\\psi\}\}\(\\hat\{z\}\_\{N\}^\{\(k\)\},z\_\{g\}\),\(73\)𝐠k\\displaystyle\\mathbf\{g\}\_\{k\}=∇𝐚kVψ¯​\(z^N\(k\),zg\)\.\\displaystyle=\\nabla\_\{\\mathbf\{a\}\_\{k\}\}V\_\{\\bar\{\\psi\}\}\(\\hat\{z\}\_\{N\}^\{\(k\)\},z\_\{g\}\)\.\(74\)The plan is updated by

𝐚k\+1=clip\[−amax,amax\]⁡\[𝐚k\+fθ​\(𝐚k,vk,𝐠k\)\]\.\\mathbf\{a\}\_\{k\+1\}=\\operatorname\{clip\}\_\{\[\-a\_\{\\max\},a\_\{\\max\}\]\}\\left\[\\mathbf\{a\}\_\{k\}\+f\_\{\\theta\}\(\\mathbf\{a\}\_\{k\},v\_\{k\},\\mathbf\{g\}\_\{k\}\)\\right\]\.\(75\)The actor receives no raw current\-state or goal latent\. Goal information reaches it only throughvkv\_\{k\}and𝐠k\\mathbf\{g\}\_\{k\}\.

The actor is trained by differentiating the terminal value through the frozen world\-model rollout\. The value and gradient supplied as refiner inputs are detached, while the training loss remains differentiable through the refined action sequence and its resulting rollout\. No environment interaction is used during this stage\.

Letvkv\_\{k\}be the terminal value after refinement stepkk\. The planner objective is

JRP1​\(θ\)=𝔼z^\(K\)∼ℱθ​\[vK\+λmean​1K​∑k=1Kvk\]\.J\_\{\\mathrm\{RP1\}\}\(\\theta\)=\\mathbb\{E\}\_\{\\hat\{z\}^\{\(K\)\}\\sim\\mathcal\{F\}\_\{\\theta\}\}\\Bigl\[v\_\{K\}\+\\lambda\_\{\\mathrm\{mean\}\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}v\_\{k\}\\Bigr\]\.\(76\)The initial value from Section[B\.1](https://arxiv.org/html/2608.18669#A2.SS1)initializes the RP1 critic\. When critic co\-training is enabled, it continues to receive the same cached\-data TD updates while a Polyak\-averaged copy suppliesvkv\_\{k\}and𝐠k\\mathbf\{g\}\_\{k\}\.

We are doing open\-loop planning\. For closed\-loop control, letτ^\\hat\{\\tau\}be the first imagined goal\-arrival step, measured byvk≤ϵv\_\{k\}\\leq\\epsilon, or else justNNif the goal is not reached\. Choosing the telescoped per\-step costs∑i=0τ^−1γi​\(1\+γ​V​\(z^t\+i\+1\(K\),zg\)−V⁡\(z^t\+i\(K\),zg\)\)\\sum\_\{i=0\}^\{\\hat\{\\tau\}\-1\}\\gamma^\{i\}\(1\+\\gamma V\(\\hat\{z\}\_\{t\+i\+1\}^\{\(K\)\},z\_\{g\}\)\-V\(\\hat\{z\}\_\{t\+i\}^\{\(K\)\},z\_\{g\}\)\)as the planner’s optimization objective yields an arrival\-aware loss that favors reaching the goal earlier\.

### B\.3Dyna Loop

The valuesvkv\_\{k\}in the planner\-loss are read off latentsHϕ​\(𝐚k,z^0\)H\_\{\\phi\}\(\\mathbf\{a\}\_\{k\},\\hat\{z\}\_\{0\}\)that the world modelshϕh\_\{\\phi\}produced\. Shouldhϕh\_\{\\phi\}be wrong, or not have coverage for the dataset𝒟\\mathcal\{D\}, the planner can exploit inaccuracies, as is well reported in literature[43](https://arxiv.org/html/2608.18669#bib.bib13),[48](https://arxiv.org/html/2608.18669#bib.bib14),[18](https://arxiv.org/html/2608.18669#bib.bib15)\.

Much of this can be fixed by finetuning the world model on actual roll\-out data, as originally proposed in the Dyna loop[47](https://arxiv.org/html/2608.18669#bib.bib29)\. For this we deployθr\\theta\_\{r\}in the real environment, collect the \(failure\) trajectories it produces, mix them into the training data, and finetune the world model on the mixture\. Then we retrain the planner and repeat\.

## Appendix CEmpirical Results

### C\.1General Setup

#### Data and evaluation protocol\.

All world\-model encoders and dynamics predictors are frozen throughout; critics and planners are trained purely from cached latents\. Each domain provides10,00010\{,\}000episodes: value functions and planners train on episodes00–7,9997\{,\}999, and all evaluations draw start/goal states from the held\-out episodes8,0008\{,\}000–9,9999\{,\}999\. Hyperparameters are selected on the disjoint evaluation draws\{50,51\}\\\{50,51\\\}and never reported\. Unless stated otherwise, reported numbers average over the three predeclared evaluation seeds\{42,43,44\}\\\{42,43,44\\\}and, for RP1, over three planner training seeds\{0,1,2\}\\\{0,1,2\\\}; Reacher uses a wider protocol \(Sec\.[C\.3](https://arxiv.org/html/2608.18669#A3.SS3)\)\.

#### Planning protocol\.

All planners use55\-step action chunks and optimizeH=5H\{=\}5chunks \(2525primitive steps\) open loop, replanning every55chunks \(receding horizon55\)\. The goal is the statehhprimitive steps ahead and the episode budget is2​h2hsteps; TwoRoom and Cube evaluateh∈\{25,100\}h\\in\\\{25,100\\\}\(h​25h25,h​100h100\), Reacherh=25h\{=\}25\. Simulator evaluations run under EGL with a pinned render device, serialized per node\.

#### Objectives\.

Every planner scores the predicted terminal state with one of the two objectives of Sec\.[C\.5](https://arxiv.org/html/2608.18669#A3.SS5): the latent\-distance objectiveClatent​\(z^N,zg\)=∥z^N−zg∥22C\_\{\\mathrm\{latent\}\}\(\\hat\{z\}\_\{N\},z\_\{g\}\)=\\lVert\\hat\{z\}\_\{N\}\-z\_\{g\}\\rVert\_\{2\}^\{2\}, or the value objectiveCvalue​\(z^N,zg\)=Vψ​\(z^N,zg\)C\_\{\\mathrm\{value\}\}\(\\hat\{z\}\_\{N\},z\_\{g\}\)=V\_\{\\psi\}\(\\hat\{z\}\_\{N\},z\_\{g\}\), the goal\-conditioned temporal\-distance critic of Sec\.[B\.1](https://arxiv.org/html/2608.18669#A2.SS1)\(MRN quasimetric\-style head\) trained on the frozen cached latents of each base with the per\-domain settings of Table[6](https://arxiv.org/html/2608.18669#A3.T6)\(offline\-value block\)\.

Table 6:Selected RP1 configurations across domains\.Per\-base/per\-cell entries are listed*LeWM / PLDM*for Cube and Reacher, and*LeWM⋅h​25\\cdot h25/ LeWM⋅h​100\\cdot h100/ PLDM⋅h​25\\cdot h25/ PLDM⋅h​100\\cdot h100*for TwoRoom; all other values are shared across bases within a domain\. The actor LR is cosine\-annealed to1/101/10of the listed value for Cube and Reacher and held constant for TwoRoom\. The offline value of Sec\.[B\.1](https://arxiv.org/html/2608.18669#A2.SS1)initializes the co\-trained critic\.
#### RP1 training\.

All domains share the actor–critic recipe of Sec\.[B\.2](https://arxiv.org/html/2608.18669#A2.SS2):K=8K\{=\}8refinement iterations over theH=5H\{=\}5\-chunk plan; the co\-trained critic is initialized from the offline value, continues TD updates on cached data for the listed number of live steps \(one critic step per actor step, its EMA withτ=0\.005\\tau\{=\}0\.005serving as the actor’s teacher\), and is then frozen; the TD batch size is1,0241\{,\}024and the cross\-episode goal probability is0\.30\.3\. Table[6](https://arxiv.org/html/2608.18669#A3.T6)lists every selected per\-domain setting; anything not shown there is shared across domains and bases\.

### C\.2TwoRoom

#### Specific setup\.

All cells use three fresh actor and critic seeds\{0,1,2\}\\\{0,1,2\\\}, averaged over task\-seeds\{42,43,44\}\\\{42,43,44\\\}\. Success is judged by whether the final distance to the goal is within1616pixels\. Figure[5](https://arxiv.org/html/2608.18669#A3.F5)probes the learned critic, comparing latent distance to the critic’s value landscape on sampled tasks; Fig\.[6](https://arxiv.org/html/2608.18669#A3.F6)traces plan refinement against the hand\-designed planners\.

![Refer to caption](https://arxiv.org/html/2608.18669v1/final_3environments.png)Figure 5:Latent\-Distance vs\. learned Cost\-to\-go\.3 randomly sampled tasks from seeds \(42,43,44\) and their corresponding cost landscapes\.![Refer to caption](https://arxiv.org/html/2608.18669v1/image_comparison_planners_1.png)

![Refer to caption](https://arxiv.org/html/2608.18669v1/image_comparison_planners_2.png)

![Refer to caption](https://arxiv.org/html/2608.18669v1/image_comparison_planners3.png)

Figure 6:Plan refinement in TwoRoom\.Each panel shows the planner’s best\-scoring candidate plan at refinement iterationkkon the same task, with all planners scoring plans using the learned value critic; “final” isk=30k\{=\}30for Adam and CEM andk=8k\{=\}8for RP1\. Iterations differ greatly in cost: one CEM iteration evaluates300300sampled plans \(9,0009\{,\}000rollouts in total\), one Adam iteration takes a gradient step on100100plans in parallel \(3,0003\{,\}000rollouts\), whereas one RP1 iteration is a single forward pass of the learned refiner costing one rollout \(99in total, including the initial evaluation\)\.

### C\.3Reacher

#### Specific setup\.

Reacher widens the seed protocol: we report on evaluation draws\{42,…,47\}\\\{42,\\ldots,47\\\}averaged over six training seeds\{0,…,5\}\\\{0,\\ldots,5\\\}\(3636evaluations per base and tolerance\), and each world\-model base uses a single configuration fixed a priori\. Success is first\-hit: all joints withinτ\\tauradians of the goal configuration, scored in a separate simulator pass perτ\\tauwith termination on success\.

#### Cost windows\.

On Reacher, both objectives read the predicted terminal state through a latent window ofwwterminal frames\. We takew=1w\{=\}1as the primary setting and reportw=3w\{=\}3as an ablation \(Tab\.[7](https://arxiv.org/html/2608.18669#A3.T7)\)\. The Reacher value critic additionally uses window lag55and standardized latents, and the RP1 co\-trained critic is initialized from the offline value trained at the matching cost window \(single\-frame for the primaryw=1w\{=\}1result\)\. Wideningwwfrom11to33lets the cost read first\-order \(velocity\) information, which we expect to sharpen the estimate, most visibly at the tightτ=0\.05\\tau\{=\}0\.05tolerance\.

\(a\) single\-frame costsLeWMPLDMplannerroll\.τ=\.1\\tau\{=\}\.1τ=\.05\\tau\{=\}\.05τ=\.1\\tau\{=\}\.1τ=\.05\\tau\{=\}\.05*latent objective*CEM9k98\.780\.396\.780\.0MPPI9k63\.739\.364\.735\.7Adam3k94\.066\.094\.366\.0*value objective*CEM9k97\.382\.096\.076\.0MPPI9k74\.042\.060\.038\.7Adam3k88\.064\.792\.766\.7RP1†998\.788\.797\.882\.0
\(b\) 3\-frame costsLeWMPLDMplannerroll\.τ=\.1\\tau\{=\}\.1τ=\.05\\tau\{=\}\.05τ=\.1\\tau\{=\}\.1τ=\.05\\tau\{=\}\.05*latent objective*CEM9k99\.094\.398\.389\.3MPPI9k87\.768\.085\.764\.3Adam3k97\.380\.096\.777\.3*value objective*CEM9k99\.389\.398\.384\.7MPPI9k86\.066\.083\.761\.7Adam3k98\.381\.097\.376\.7RP1 \(ours\)999\.997\.199\.491\.2

Table 7:Reacher, first\-hit success \(%\) by cost window\(completes Tab\.[3](https://arxiv.org/html/2608.18669#S7.T3)\)\. The single\-frame cost \(a, our primary setting\) feeds only the terminal latent; the three\-frame cost \(b\) additionally feeds the two preceding latents, capturing first\-order information and, as expected, tightening success atτ=0\.05\\tau\{=\}0\.05\. RP1 leads every column in both windows\.

### C\.4OGBench Cube

#### Specific setup\.

Every cell is5050episodes per evaluation seed\. Success follows the benchmark’s cube\-placement criterion; the no\-op floors \(56\.056\.0ath​25h25,45\.345\.3ath​100h100\) are measured by executing zero actions under the identical protocol\.

#### Value expansion\.

On Cube, value expansion is part of the selected configuration[6](https://arxiv.org/html/2608.18669#bib.bib44): the critic bootstraps on imagined terminal states whose arrival velocity a single\-frame latent cannot represent, letting actor and critic jointly exploit the world model\.

#### Dyna iteration\.

On\-policy episodes are collected with the trained \(PRE\) planner onh​25h25tasks from the training split \(episodes00–79997999, no termination at goal\), mixed50:5050\{:\}50with the original data and outcome\-labeled; the world model is finetuned for22epochs at LR10−510^\{\-5\}\(epoch11kept\); latent caches and the TD critic are rebuilt under the finetuned model; POST actors retrain with the unchanged recipe\. The finetuned model is reused as\-is forh​100h100evaluation \(Sec\.[B\.3](https://arxiv.org/html/2608.18669#A2.SS3)\)\.

### C\.5Planning Baselines

Each conventional planner is evaluated with two terminal objectives\. The latent\-distance objective scores the predicted terminal latent by

Clatent​\(z^N,zg\)=∥z^N−zg∥22\.C\_\{\\mathrm\{latent\}\}\(\\hat\{z\}\_\{N\},z\_\{g\}\)=\\lVert\\hat\{z\}\_\{N\}\-z\_\{g\}\\rVert\_\{2\}^\{2\}\.\(77\)The value objective uses the goal\-conditioned value trained for the corresponding environment and world model:

Cvalue​\(z^N,zg\)=Vψ​\(z^N,zg\)\.C\_\{\\mathrm\{value\}\}\(\\hat\{z\}\_\{N\},z\_\{g\}\)=V\_\{\\psi\}\(\\hat\{z\}\_\{N\},z\_\{g\}\)\.\(78\)All baselines plan in the same normalized55\-chunk action space as RP1 and follow the identical receding\-horizon protocol; they differ only in how the action sequence is optimized\. Per decision, CEM and MPPI evaluate9,0009\{,\}000forward rollouts; Adam evaluates3,0003\{,\}000forward rollouts and the corresponding3,0003\{,\}000backward passes\.

#### Cross Entropy Method \(CEM\)\.

CEM samples complete action sequences from a factorized Gaussian, retains the lowest\-cost elite set, and refits the sampling distribution after every iteration\. We use300300samples per iteration for3030iterations with an elite set of3030\(top10%10\\%\); the initial distribution is zero\-mean with unit variance in the normalized action space\.

#### Model\-Predictive Path\-Integral \(MPPI\)\.

MPPI samples Gaussian perturbations around the current action sequence and updates the sequence using exponentially weighted trajectory costs\. We use300300samples per iteration for3030iterations with temperatureλ=0\.5\\lambda=0\.5\.

#### Adam\.

Adam directly differentiates the terminal objective through the frozen world\-model rollout and optimizes a batch of action sequences\. We optimize300300sequences in parallel for1010steps with AdamW at learning rate0\.10\.1and execute the lowest\-cost sequence\. For TwoRoom we optimized100100sequences in parallel at3030steps\.

#### Deep Model\-Predictive Optimization \(DMPO\)\.

DMPO keeps the MPPI update and learns a residual on it: a network reads the sampling distribution and theNNrollout costs \(no state, no gradient\) and emits a gated mean correction, a covariance update, and a learned warm\-start shift\[[40](https://arxiv.org/html/2608.18669#bib.bib50)\]\. The paper trains this online with PPO; we instead train the same networks offline by pathwise gradients through the frozen world model against the criticVψV\_\{\\psi\}\. Budget:256256rollouts per decision \(256×1256\{\\times\}1\)\.

#### Learning\-to\-Optimize MPC \(L2O\-MPC\)\.

L2O\-MPC, DMPO’s predecessor, learns the*whole*sampling update rather than a residual: a network reads the mean, covariance, andNNcosts and emits a gated replacement mean\[[39](https://arxiv.org/html/2608.18669#bib.bib49)\]\. As it is not a working optimizer untrained, it is trained by DAgger imitation of a larger\-budget MPPI expert \(computed here through the frozen world model and criticVψV\_\{\\psi\}\)\. Budget:256256rollouts per decision \(64×464\{\\times\}4\)\.