用于潜在世界模型强化学习的密度引导自举边界

# Density-Guided Bootstrap Boundaries for Latent World-Model Reinforcement Learning
Source: [https://arxiv.org/html/2610.41022](https://arxiv.org/html/2610.41022)
Author Gupta and Lee

Shashank Gupta shashang@umich\.eduAffiliation:Department of EECSAffiliation:University of MichiganAffiliation:Ann Arbor, MI 48109, USAPilhwa Lee pilhwa\.lee@morgan\.edu††thanks:Corresponding author\.Affiliation:Department of MathematicsAffiliation:Morgan State UniversityAffiliation:Baltimore, MD 21251, USA

###### Abstract

Latent world models allow reinforcement learning agents to learn policies from imagined trajectories, yet determining the optimal depth of model reliance remains a fundamental challenge\. Fixed imagination horizons apply uniform prediction depths regardless of trajectory reliability, exposing target values to compounding model error or over\-relying on critic approximations\. We introduce density\-guided bootstrap boundaries, a trajectory\-dependent mechanism that adapts the depth of imagined returns using a score\-informed relative density proxy\. The proxy accumulates the estimated divergence of a score\-corrected auxiliary field formed by a feature displacement network and a noise\-conditioned score estimator along imagined paths\. We propose two gating mechanisms: a hard boundary that halts return expansion at a discrete state, and a soft boundary that continuously shifts the return recursion toward critic bootstrapping\. Both variants preserve boundary value estimates and apply corresponding survival weights to actor and critic losses\. Formulated independently of specific dynamics models and instantiated within Dreamer V3, we prove that the resulting gated Bellman operator is a strict contraction mapping and that early bootstrapping monotonically reduces conditional target variance\. We clarify the theoretical assumptions distinguishing our expansion proxy from exact path likelihoods\. Across continuous\-control tasks and Atari Pong, empirical evaluations show that dynamic boundary selection enhances sample efficiency and late\-training performance over fixed\-horizon and auxiliary\-enabled baselines\.

††heading:23 2026 1\-11/26; Revised 1/27 2/27 21\-0000††shortheadings:Density\-Guided Bootstrap Boundaries / Gupta and Lee††firstpage:1###### keywords

Model\-based reinforcement learning, latent world models, adaptive bootstrapping, score estimation, imagined trajectories\.

## 1Introduction

Learning a predictive model of the environment allows a reinforcement learning agent to improve its policy using imagined experience\. Latent world models make this approach practical by representing observations in a compact state space and predicting the consequences of actions within that representation\. DreamerV3 illustrates the effectiveness of combining a stochastic latent world model with actor–critic learning from imagined trajectories\([Hafner et al\., 2023](https://arxiv.org/html/2610.41022#bib.bib4)\)\. These advances make model\-based learning increasingly useful, but they leave an important question: how much of an imagined trajectory should contribute to a learning target before the agent relies on its value function? The depth of an imagined return determines a trade\-off between model predictions and value estimates\. Expanding a return further incorporates additional predicted rewards, but also exposes the target to errors in the learned dynamics and reward model\. Bootstrapping earlier reduces this exposure while increasing dependence on the accuracy of the critic\. A fixed maximum imagination horizon provides a simple way to control this trade\-off\. However, it applies the same limit to trajectories originating from different states and passing through different regions of the learned representation\. Even when a return estimator mixes multiple prediction depths, a fixed mixing rule does not explicitly respond to a trajectory\-dependent diagnostic of the model\.

The question of when to trust a learned model has a substantial history\. Model\-Based Policy Optimization uses short model rollouts branched from real experience to limit the consequences of model error\([Janner et al\., 2019](https://arxiv.org/html/2610.41022#bib.bib6)\)\. Stochastic Ensemble Value Expansion combines returns at different horizons using uncertainty estimates derived from ensembles\([Buckman et al\., 2018](https://arxiv.org/html/2610.41022#bib.bib7)\)\. These approaches demonstrate that controlling model usage is an important design choice in its own right\. Our work investigates a complementary mechanism: using a score\-informed relative density proxy to select where an imagined return gives way to value bootstrapping\.

We propose*density\-guided bootstrap boundaries*for latent world\-model reinforcement learning\. The framework associates each imagined state with a continuation gate that controls further return expansion\. A hard gate selects a discrete bootstrap boundary, while a soft gate gradually shifts the return recursion toward the critic\. In both cases, reaching a boundary preserves the value estimate at that state\. The boundary therefore represents a decision about how much model\-generated experience to use, rather than a prediction that the environment has terminated\. Corresponding survival weights control the contribution of later imagined states to the actor and critic objectives\. To construct the gating signal, we learn an auxiliary displacement field and a noise\-conditioned score estimator from latent features inferred from replay observations\. The score estimator supplies a correction to the auxiliary field\. We estimate the divergence of the corrected field and accumulate its negative along imagined trajectories to obtain a relative density proxy\. This construction gives the score estimator an explicit role in policy learning: its correction can change the accumulated signal, the selected bootstrap boundary, and consequently the targets and weights used by the actor and critic\. The stochastic world model remains responsible for generating imagined states; the auxiliary field does not replace its transition distribution\. The interpretation of this signal requires care\. The accumulated quantity is an anchored density\-change proxy associated with an auxiliary vector field, not an exact likelihood under the stochastic world model\. In particular, imagined transitions are not generated by integrating that field, and estimating a score in a noise\-smoothed feature space does not establish that low proxy values identify inaccurate predictions\. We therefore treat the usefulness of the proxy for boundary selection as an empirical hypothesis\. The relevant question is whether it supports better decisions about return expansion, rather than whether its magnitude alone can be interpreted as calibrated model uncertainty\.

The bootstrap formulation is independent of a particular world\-model architecture\. Its use requires imagined trajectories, a value estimator, and a trajectory\-dependent gate; our proposed density mechanism additionally requires suitable latent features and differentiable auxiliary models\. We instantiate the method within DreamerV3, preserving its stochastic latent dynamics and using its actor–critic training procedure as the evaluation framework\. This implementation allows us to study hard and soft boundaries within a common learning system\. Our current implementation still generates the full imagined rollout, so adaptive return depth should not be interpreted as a demonstrated reduction in rollout computation\.

A central evaluation issue is separating the information supplied by the density signal from the effects of shortening imagined returns\. An improvement over a fixed maximum horizon could arise from a generally preferable shorter horizon, auxiliary training effects, or a change in the magnitude of the optimization objective\. Accordingly, our evaluation framework distinguishes the unmodified agent, an auxiliary\-enabled agent without boundary selection, and density\-guided variants\. Fixed\-boundary controls, score ablations, and threshold sensitivity provide complementary tests of the proposed mechanism\. Preliminary observations motivate these tests by showing that the score correction can substantially alter the divergence signal and that effective return depths vary across tasks and training stages; they do not by themselves establish that density\-guided selection is superior\.

Our contributions are threefold:

1. 1\.We formulate hard and soft trajectory\-dependent bootstrap boundaries for imagined actor–critic learning, preserving value bootstrapping and applying consistent survival weights to the learning objectives\.
2. 2\.We introduce a score\-informed relative density proxy for boundary selection and specify the assumptions and limitations of its interpretation in a stochastic latent world model\.
3. 3\.We provide a DreamerV3 instantiation and an empirical evaluation across continuous\-control and visual discrete\-action tasks, including fixed\-boundary controls, auxiliary\-component ablations, and diagnostics of effective imagination length\.

## 2Related Work

#### Latent world models and learning through imagination\.

Learning dynamics in a latent representation provides a way to use predictive models when observations are high\-dimensional\. PlaNet combines deterministic and stochastic latent transitions with online planning from image observations\([Hafner et al\., 2019](https://arxiv.org/html/2610.41022#bib.bib2)\)\. Dreamer subsequently learns an actor and a value function through trajectories generated in latent space, propagating learning signals through imagined outcomes\([Hafner et al\., 2020](https://arxiv.org/html/2610.41022#bib.bib3)\)\. DreamerV3 develops this approach into a broadly applicable world\-model agent\([Hafner et al\., 2023](https://arxiv.org/html/2610.41022#bib.bib4)\)\. Our work addresses how imagined experience enters the return estimator and training objective\. The bootstrap\-boundary formulation does not require a particular world\-model architecture, although our implementation and empirical evidence are based on DreamerV3\. We retain its stochastic transition model and introduce auxiliary models for selecting the depth of return expansion\.

#### Model usage and adaptive value expansion\.

The benefits of model\-generated experience depend on how prediction errors interact with policy and value learning\. Model\-Based Value Expansion \(MVE\) combines a finite sequence of model\-predicted rewards with a terminal value estimate, using a fixed expansion depth\([Feinberg et al\., 2018](https://arxiv.org/html/2610.41022#bib.bib5)\)\. Model\-Based Policy Optimization \(MBPO\) uses short model rollouts branched from real experience to control the effects of model error during policy optimization\([Janner et al\., 2019](https://arxiv.org/html/2610.41022#bib.bib6)\)\. These methods motivate treating the extent of model usage as a separate algorithmic choice\. Stochastic Ensemble Value Expansion \(STEVE\) is particularly relevant: it constructs targets at multiple rollout depths and combines them using variance estimates from ensembles, adapting the mixture for individual examples\([Buckman et al\., 2018](https://arxiv.org/html/2610.41022#bib.bib7)\)\. Consequently, adapting return depth to individual trajectories is not itself the novelty of our approach\. Our proposed mechanism uses a score\-informed relative density proxy to construct hard or soft continuation gates within the return recursion\. These gates also induce survival weights on actor and critic losses\. We investigate this density\-based selection rule without interpreting it as an ensemble estimate of prediction uncertainty or asserting that it minimizes target variance\.

#### Pessimism and restrictions on model\-generated experience\.

Offline model\-based reinforcement learning also studies how to limit the influence of unreliable predictions\. MOPO penalizes predicted rewards using a measure of dynamics uncertainty\([Yu et al\., 2020](https://arxiv.org/html/2610.41022#bib.bib12)\)\. MOReL constructs a pessimistic model in which transitions classified as unknown lead to a penalized absorbing state\([Kidambi et al\., 2020](https://arxiv.org/html/2610.41022#bib.bib10)\)\. These approaches address the risks of exploiting a learned model outside the support of a fixed dataset\.

Our setting is online learning with a changing replay distribution, and our boundary has a different role\. It ends or attenuates return expansion while retaining the critic estimate at the selected state\. It does not impose an uncertainty penalty on rewards or declare the boundary to be an environment terminal state\. This distinction matters because a bootstrap boundary transfers reliance from model predictions to the critic; it does not remove estimation error\. Moreover, the proposed density proxy is not assumed to identify unknown transitions with the guarantees used in pessimistic offline formulations\.

#### Adaptive imagination and computation allocation\.

Adaptive rollout decisions can also control planning computation\. For example, RISE selects whether to continue imagination in world action models using estimates of risk, expected planning benefit, and computational cost\([Lu et al\., 2026](https://arxiv.org/html/2610.41022#bib.bib11)\)\. This is related to our work at the level of making trajectory\-dependent continuation decisions\. However, our gates control the depth of actor–critic return expansion and the weights of imagined training samples\. The current implementation generates the full imagined trajectory before applying these gates\. Its effective return horizon therefore should not be equated with the amount of rollout computation performed\.

#### Score estimation and probability\-flow dynamics\.

Denoising score matching connects learning from noise\-corrupted observations to estimating the gradient of a smoothed log density\([Vincent, 2011](https://arxiv.org/html/2610.41022#bib.bib1)\)\. Score\-based generative modeling extends this connection to stochastic differential equations and associated probability\-flow ordinary differential equations\([Song et al\., 2021](https://arxiv.org/html/2610.41022#bib.bib8)\)\. These results motivate using learned scores to describe distributional structure and to construct corrections to vector fields\.

In our method, a noise\-conditioned estimator is trained on replay posterior features and contributes to an auxiliary vector field evaluated along imagined trajectories\. The resulting signal is used for bootstrap selection\. We do not use the score estimator to replace the stochastic world model with a diffusion sampler\. In addition, the probability\-flow interpretation requires assumptions about the relationship between the drift, diffusion, and score\. Learning a displacement field and a score from replay features does not by itself establish those relationships for a discrete stochastic latent transition model\. We therefore distinguish the continuous\-time motivation from the operational density proxy used in our implementation\.

#### Divergence\-based density evolution\.

Continuous normalizing flows relate changes in log density to the divergence of the vector field generating a continuous trajectory\. FFJORD uses stochastic trace estimation to make this computation scalable without forming the full Jacobian\([Grathwohl et al\., 2019](https://arxiv.org/html/2610.41022#bib.bib9)\)\. We use the same computational principle to estimate the divergence of an auxiliary score\-corrected field\. The interpretation differs because our imagined trajectories are generated by the stochastic world model, rather than by integrating the auxiliary field\. Accumulating estimated negative divergence along these trajectories yields an anchored relative proxy, not an exact world\-model likelihood\. The use of clipping and numerical trace estimation further affects the implemented signal\. Our contribution concerns the use of this signal to control bootstrapping, rather than a new exact\-likelihood model or a general guarantee of variance\-free policy learning\.

#### Position of the present work\.

The method brings together adaptive model usage, score estimation, and divergence\-based diagnostics through a common bootstrap\-gating formulation\. The central empirical question is whether the density signal provides useful trajectory\-dependent selection beyond the effects of a shorter return horizon or a change in loss scale\. This motivates comparisons with fixed\-boundary controls, auxiliary\-enabled training without cutoff, and score ablations\. Theoretical claims about the gated estimator must likewise account for critic error and the dependence of gates on sampled trajectories; properties of exact density transport do not automatically transfer to this implementation\.

## 3Problem Formulation

### 3\.1Reinforcement Learning with a Latent World Model

We consider a partially observed sequential decision process\. At environment timenn, the agent receives an observationono\_\{n\}, selects an actionana\_\{n\}, and receives a rewardrn\+1r\_\{n\+1\}after the transition\. Letℋn=\(o0,a0,r1,o1,…,an−1,rn,on\)\\mathcal\{H\}\_\{n\}=\(o\_\{0\},a\_\{0\},r\_\{1\},o\_\{1\},\\ldots,a\_\{n\-1\},r\_\{n\},o\_\{n\}\)denote its interaction history\. The objective is to learn a policy that maximizes expected discounted reward,

J⁡\(θ\)=𝔼πθ,P​\[∑n=0∞γn​rn\+1\],J\(\\theta\)=\\mathbb\{E\}\_\{\\pi\_\{\\theta\},P\}\\left\[\\sum\_\{n=0\}^\{\\infty\}\\gamma^\{n\}r\_\{n\+1\}\\right\],\(1\)wherePPdenotes the environment dynamics,πθ\\pi\_\{\\theta\}is the policy, andγ∈\[0,1\)\\gamma\\in\[0,1\)is the discount factor\. Terminal states can equivalently be treated as absorbing states with zero subsequent rewards\.

A world model with parametersϕ\\phirepresents the history through a latent stateznz\_\{n\}inferred from observations\. We write this inference distribution asqϕ​\(zn∣ℋn\)q\_\{\\phi\}\(z\_\{n\}\\mid\\mathcal\{H\}\_\{n\}\)and the latent transition distribution aspϕ​\(zn\+1∣zn,an\)p\_\{\\phi\}\(z\_\{n\+1\}\\mid z\_\{n\},a\_\{n\}\)\. The stateznz\_\{n\}includes any recurrent memory required by the model\. A feature mapxn=Fϕ​\(zn\)∈ℝdx\_\{n\}=F\_\{\\phi\}\(z\_\{n\}\)\\in\\mathbb\{R\}^\{d\}provides inputs to the actor, critic, reward predictor, and auxiliary models\. This notation permits deterministic and stochastic components withinznz\_\{n\}, as in recurrent latent world models\([Hafner et al\., 2019](https://arxiv.org/html/2610.41022#bib.bib2);[Hafner et al\., 2020](https://arxiv.org/html/2610.41022#bib.bib3)\)\. It does not require the feature distribution to have a density with respect to Lebesgue measure\.

The policy isπθ​\(a∣x\)\\pi\_\{\\theta\}\(a\\mid x\), andVψ​\(x\)V\_\{\\psi\}\(x\)denotes a scalar value estimate extracted from the critic\. The world model also provides reward and continuation predictions\. Replay data𝒟\\mathcal\{D\}contain observed interaction sequences used to train the world model and construct posterior latent features\. The learned representation is an approximation to a sufficient state; we do not assume that it exactly preserves the underlying environment dynamics\.

### 3\.2Imagined Trajectories and Return Expansion

An imagined trajectory starts from a latent state inferred from replay and evolves under the policy and learned transition model:

at∼πθ\(⋅∣xt\),zt\+1∼pϕ\(⋅∣zt,at\),xt=Fϕ\(zt\)\.a\_\{t\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\_\{t\}\),\\qquad z\_\{t\+1\}\\sim p\_\{\\phi\}\(\\cdot\\mid z\_\{t\},a\_\{t\}\),\\qquad x\_\{t\}=F\_\{\\phi\}\(z\_\{t\}\)\.\(2\)Herettindexes imagination steps, separately from environment timenn\. We useHHto denote the number of transitions contributing to return expansion, so the trajectory contains statesx0,…,xHx\_\{0\},\\ldots,x\_\{H\}\. This convention counts transitions rather than the number of feature entries in an implementation\.

Letr^t\+1\\widehat\{r\}\_\{t\+1\}be the predicted reward associated with a transition and let

dt\+1=γ​c^t\+1,c^t\+1∈\[0,1\],d\_\{t\+1\}=\\gamma\\widehat\{c\}\_\{t\+1\},\\qquad\\widehat\{c\}\_\{t\+1\}\\in\[0,1\],\(3\)wherec^t\+1\\widehat\{c\}\_\{t\+1\}is the predicted probability of environment continuation\. These quantities follow the reward and continuation indexing convention of the base world\-model algorithm\.

A finite\-horizonλ\\lambda\-return is defined recursively by

Gtλ=r^t\+1\+dt\+1​\[\(1−λ\)​Vψ​\(xt\+1\)\+λ​Gt\+1λ\],GHλ=Vψ​\(xH\),G\_\{t\}^\{\\lambda\}=\\widehat\{r\}\_\{t\+1\}\+d\_\{t\+1\}\\left\[\(1\-\\lambda\)V\_\{\\psi\}\(x\_\{t\+1\}\)\+\\lambda G\_\{t\+1\}^\{\\lambda\}\\right\],\\qquad G\_\{H\}^\{\\lambda\}=V\_\{\\psi\}\(x\_\{H\}\),\(4\)fort=0,…,H−1t=0,\\ldots,H\-1andλ∈\[0,1\]\\lambda\\in\[0,1\]\. The horizon bounds the maximum expansion depth, whileλ\\lambdacontrols the mixture of intermediate bootstraps and longer returns\. Our problem is to make this reliance on future imagined predictions depend on the trajectory\.

Specifically, we seek a continuation gategt∈\[0,1\]g\_\{t\}\\in\[0,1\]computed from the imagined prefix and learned auxiliary information\. The gate is conceptually distinct fromc^t\\widehat\{c\}\_\{t\}: environment continuation describes whether the task continues, whereasgtg\_\{t\}describes whether the learning target should continue expanding through model predictions\. A zero gate at a boundary should preserve the incoming reward and the discounted value estimate at that state\. A unit gate should recover the base return recursion locally\. The construction of the gates, their insertion into the recursion, and their associated loss weights are specified in Section[5](https://arxiv.org/html/2610.41022#S5)\.

### 3\.3Score Estimation in Latent Feature Space

Letμϕ\\mu\_\{\\phi\}denote the distribution of posterior features obtained from replay under the current world model\. Because these features may contain discrete components or occupy a lower\-dimensional subset ofℝd\\mathbb\{R\}^\{d\}, we define score estimation through Gaussian smoothing\. For a noise scaleα\>0\\alpha\>0,

x∼μϕ,ϵ∼𝒩⁡\(0,Id\),x~=x\+α​ϵ\.x\\sim\\mu\_\{\\phi\},\\qquad\\epsilon\\sim\\mathcal\{N\}\(0,I\_\{d\}\),\\qquad\\widetilde\{x\}=x\+\\alpha\\epsilon\.\(5\)The perturbed distribution has density

pϕ,α​\(x~\)=∫𝒩⁡\(x~,x,α2​Id\)​μϕ​\(𝑑x\),p\_\{\\phi,\\alpha\}\(\\widetilde\{x\}\)=\\int\\mathcal\{N\}\(\\widetilde\{x\};x,\\alpha^\{2\}I\_\{d\}\)\\,\\mu\_\{\\phi\}\(dx\),\(6\)and its score is

sϕ,α⋆​\(x~\)=∇x~​log​pϕ,α​\(x~\)\.s^\{\\star\}\_\{\\phi,\\alpha\}\(\\widetilde\{x\}\)=\\nabla\_\{\\widetilde\{x\}\}\\log p\_\{\\phi,\\alpha\}\(\\widetilde\{x\}\)\.\(7\)
A noise\-conditioned predictorϵη​\(x~,α\)\\epsilon\_\{\\eta\}\(\\widetilde\{x\},\\alpha\)is trained using the denoising objective

ℒnoise​\(η\)=𝔼x,α,ϵ​\[‖ϵη​\(x\+α​ϵ,α\)−ϵ‖22\]\.\\mathcal\{L\}\_\{\\mathrm\{noise\}\}\(\\eta\)=\\mathbb\{E\}\_\{x,\\alpha,\\epsilon\}\\left\[\\left\\\|\\epsilon\_\{\\eta\}\(x\+\\alpha\\epsilon,\\alpha\)\-\\epsilon\\right\\\|\_\{2\}^\{2\}\\right\]\.\(8\)The expectation overα\\alphafollows a chosen positive noise\-scale distribution\. At the population optimum, for a fixedα\\alpha, the predictor estimates the conditional mean of the injected noise, giving the score parameterization

sη​\(x~,α\)=−ϵη​\(x~,α\)α\.s\_\{\\eta\}\(\\widetilde\{x\},\\alpha\)=\-\\frac\{\\epsilon\_\{\\eta\}\(\\widetilde\{x\},\\alpha\)\}\{\\alpha\}\.\(9\)This relation follows the denoising score\-matching construction\([Vincent, 2011](https://arxiv.org/html/2610.41022#bib.bib1);[Song et al\., 2021](https://arxiv.org/html/2610.41022#bib.bib8)\)\. A finite learned predictor only approximates the population score\. Moreover,μϕ\\mu\_\{\\phi\}changes as replay and the world model evolve\. The resulting score describes a smoothed replay\-feature distribution; it is not automatically the score of the policy’s imagined\-state distribution or of the world\-model transition kernel\.

### 3\.4Divergence as an Empirical Feature\-Space Expansion Diagnostic

In continuous dynamics governed by a vector fieldv⁡\(x,u\)v\(x,u\), local volumetric changes in feature space are tracked by the spatial divergence operator∇x⋅v⁡\(x,u\)=Tr​\(Jv​\(x\)\)\\nabla\_\{x\}\\cdot v\(x,u\)=\\text\{Tr\}\(J\_\{v\}\(x\)\)\. In latent world\-model reinforcement learning, imagined state transitions follow a discrete stochastic processzt\+1∼pϕ​\(zt\+1∣zt,at\)z\_\{t\+1\}\\sim p\_\{\\phi\}\(z\_\{t\+1\}\\mid z\_\{t\},a\_\{t\}\)rather than continuous integration along trajectories ofv⁡\(x,u\)v\(x,u\)\.

Consequently, the spatial divergence of an auxiliary score\-corrected vector field does not yield an exact path log\-likelihood under the stochastic world model\. Instead, we define the accumulated negative divergenceltl\_\{t\}as ananchored feature\-space expansion diagnostic\. This diagnostic monitors local volumetric expansion or contraction along imagined transitions\. Large negative divergence indicates that downstream transitions are entering unconstrained or rapidly expanding regions of the learned latent feature space, serving as an empirical probe to halt return expansion before critic updates are corrupted by out\-of\-distribution dynamics\.

### 3\.5Objective and Scope of Boundary Selection

The algorithmic goal is to construct continuation gatesgt∈\[0,1\]g\_\{t\}\\in\[0,1\]fromltl\_\{t\}to modulate return expansion depth\. A gate value ofgt=0g\_\{t\}=0truncates return expansion at stepttwhile retaining the critic’s value estimateVψ​\(xt\)V\_\{\\psi\}\(x\_\{t\}\), shifting reliance from future model rollouts to the value function\.

We explicitly distinguish conditional target variance at a fixed latent state from variance across replay samples or training seeds\. Truncating return expansion avoids accumulating transition variance from downstream model rollouts at the cost of absorbing approximation biase⁡\(xt\)=Vψ​\(xt\)−Vπ​\(xt\)e\(x\_\{t\}\)=V\_\{\\psi\}\(x\_\{t\}\)\-V^\{\\pi\}\(x\_\{t\}\)from the critic\.

### 3\.6Continuous\-to\-Discrete Bridging on Categorical Latents

Recurrent architectures like Dreamer V3 represent states using categorical latent variables concatenated with deterministic recurrent states\. To bridge continuous transport concepts with discrete latent dynamics, we establish three theoretical properties:

1. 1\.Gaussian Smoothing over Categorical Feature Spaces:Discrete distributions lack continuous density with respect to Lebesgue measure\. Convolving the empirical posterior feature distributionμϕ\\mu\_\{\\phi\}with an isotropic Gaussian kernel𝒩⁡\(0,α2​Id\)\\mathcal\{N\}\(0,\\alpha^\{2\}I\_\{d\}\)defines a smooth surrogate density: pϕ,α​\(x~\)=∫ℝd𝒩⁡\(x~,x,α2​Id\)​μϕ​\(𝑑x\)∈C∞​\(ℝd\)\.p\_\{\\phi,\\alpha\}\(\\tilde\{x\}\)=\\int\_\{\\mathbb\{R\}^\{d\}\}\\mathcal\{N\}\(\\tilde\{x\};x,\\alpha^\{2\}I\_\{d\}\)\\,\\mu\_\{\\phi\}\(dx\)\\in C^\{\\infty\}\(\\mathbb\{R\}^\{d\}\)\.\(10\)This smoothing guarantees that spatial score matchingsη​\(x~,α\)=∇x~​log​pϕ,α​\(x~\)s\_\{\\eta\}\(\\tilde\{x\},\\alpha\)=\\nabla\_\{\\tilde\{x\}\}\\log p\_\{\\phi,\\alpha\}\(\\tilde\{x\}\)and divergence operators∇x⋅vω,η​\(x\)\\nabla\_\{x\}\\cdot v\_\{\\omega,\\eta\}\(x\)are smooth, well\-defined differential operators overℝd\\mathbb\{R\}^\{d\}\.
2. 2\.First\-Order Quadrature over Latent Displacements:An imagined transition\(xt→xt\+1\)\(x\_\{t\}\\to x\_\{t\+1\}\)represents a discrete stepΔ​u=1\\Delta u=1\. Integrating the continuous ratedd​ulogρ\(x\(u\)\)=−∇x⋅v\(x\(u\)\)\\frac\{d\}\{du\}\\log\\rho\(x\(u\)\)=\-\\nabla\_\{x\}\\cdot v\(x\(u\)\)across a unit step yieldslogρ\(xt\+1\)−logρ\(xt\)=−∫tt\+1∇x⋅v\(x\(u\),u\)du\\log\\rho\(x\_\{t\+1\}\)\-\\log\\rho\(x\_\{t\}\)=\-\\int\_\{t\}^\{t\+1\}\\nabla\_\{x\}\\cdot v\(x\(u\),u\)\\,du\. The discrete proxy updateδt=clip​\(−D^t,−C,C\)\\delta\_\{t\}=\\text\{clip\}\(\-\\hat\{D\}\_\{t\},\-C,C\)implements a 1st\-order forward Euler quadrature evaluated atxtx\_\{t\}, with single\-step discretization error bounded by𝒪⁡\(‖Δ​xt‖22\)\\mathcal\{O\}\(\\\|\\Delta x\_\{t\}\\\|\_\{2\}^\{2\}\)whereΔ​xt=xt\+1−xt\\Delta x\_\{t\}=x\_\{t\+1\}\-x\_\{t\}\.
3. 3\.Divergence as Local Volumetric Expansion Diagnostic:A 1st\-order spatial expansion along the step displacementΔ​xt\\Delta x\_\{t\}givesΔ​log⁡p⁡\(xt\)≈∇x​log​p​\(xt\)⊤​Δ​xt\\Delta\\log p\(x\_\{t\}\)\\approx\\nabla\_\{x\}\\log p\(x\_\{t\}\)^\{\\top\}\\Delta x\_\{t\}\. The spatial divergence∇x⋅v⁡\(xt\)=Tr​\(Jv​\(xt\)\)\\nabla\_\{x\}\\cdot v\(x\_\{t\}\)=\\text\{Tr\}\(J\_\{v\}\(x\_\{t\}\)\)measures the trace of the field Jacobian, serving as an isotropic diagnostic of local feature space expansion or contraction\.

## 4Bridge Between Continuous Score Fields and Discrete RSSM

An imagined trajectory in a latent world model involves two distinct mathematical objects:

- •Discrete Transition Measure \(ℙRSSM\\mathbb\{P\}\_\{\\text\{RSSM\}\}\):The discrete path measure induced by the recurrent stochastic state\-space model \(RSSM\) transition kernelzt\+1∼pϕ​\(zt\+1∣zt,at\)z\_\{t\+1\}\\sim p\_\{\\phi\}\(z\_\{t\+1\}\\mid z\_\{t\},a\_\{t\}\)\.
- •Auxiliary Diagnostic Field \(vω,ηv\_\{\\omega,\\eta\}\):A continuous vector fieldvω,η​\(u,x,a\)=fω​\(u,x,a\)\+σ2​ϵη​\(x,σ\)v\_\{\\omega,\\eta\}\(u,x,a\)=f\_\{\\omega\}\(u,x,a\)\+\\frac\{\\sigma\}\{2\}\\epsilon\_\{\\eta\}\(x,\\sigma\)constructed over Gaussian\-smoothed posterior featurespϕ,αp\_\{\\phi,\\alpha\}\.

Because imagined state sequences\(x0,x1,…,xH\)\(x\_\{0\},x\_\{1\},\\dots,x\_\{H\}\)are sampled directly fromℙRSSM\\mathbb\{P\}\_\{\\text\{RSSM\}\}rather than generated by integratingx˙=vω,η​\(u,x,a\)\\dot\{x\}=v\_\{\\omega,\\eta\}\(u,x,a\), the auxiliary field functions strictly as anevaluative diagnostic probealong RSSM trajectories\.

We formally decompose the discrepancy between exact density transport and the empirical diagnostic through three structural error components:

1. 1\.Kernel Smoothing Displacement \(EsmoothE\_\{\\text\{smooth\}\}\):The shift introduced by evaluating score estimators trained on Gaussian\-perturbed replay featurespϕ,α​\(x~\)=∫𝒩⁡\(x~,x,α2​Id\)​μϕ​\(𝑑x\)p\_\{\\phi,\\alpha\}\(\\tilde\{x\}\)=\\int\\mathcal\{N\}\(\\tilde\{x\};x,\\alpha^\{2\}I\_\{d\}\)\\mu\_\{\\phi\}\(dx\)at clean imagined feature pointsxtx\_\{t\}\.
2. 2\.Euler Quadrature Residual \(EeulerE\_\{\\text\{euler\}\}\):The first\-order truncation error resulting from evaluating the continuous trace integral∫tt\+1∇x⋅v⁡\(x⁡\(u\),u\)​𝑑u\\int\_\{t\}^\{t\+1\}\\nabla\_\{x\}\\cdot v\(x\(u\),u\)\\,duvia a single forward Euler evaluation atxtx\_\{t\}, bounded by𝒪⁡\(‖xt\+1−xt‖22\)\\mathcal\{O\}\(\\\|x\_\{t\+1\}\-x\_\{t\}\\\|\_\{2\}^\{2\}\)\.
3. 3\.Structural Path Drift Gap \(ϵdrift\\epsilon\_\{\\text\{drift\}\}\):The unmodeled divergence between the discrete RSSM transition kernel and the continuous auxiliary reference field: ϵdrift=DKL\(ℙRSSM∥ℙFlow\)\.\\epsilon\_\{\\text\{drift\}\}=D\_\{\\text\{KL\}\}\(\\mathbb\{P\}\_\{\\text\{RSSM\}\}\\parallel\\mathbb\{P\}\_\{\\text\{Flow\}\}\)\.\(11\)

Becauseϵdrift\\epsilon\_\{\\text\{drift\}\}is an unmodeled constant stemming from architectural separation, the proxyltl\_\{t\}is treated strictly as a heuristic feature\-space expansion diagnostic rather than a calibrated path density\.

## 5Density\-Guided Bootstrap Boundaries

### 5\.1Overview

Our framework augments imagined actor–critic learning with a trajectory\-dependent decision about where to bootstrap\. The stochastic world model generates latent trajectories as in Equation \([2](https://arxiv.org/html/2610.41022#S3.E2)\)\. Two auxiliary networks, a displacement predictor and a noise\-conditioned score estimator, define a vector field evaluated along those trajectories\. Accumulated negative divergence of this field provides a relative density proxy\. Hard or soft gates constructed from this proxy modify the return recursion and determine the contribution of imagined states to the learning objectives\.

The auxiliary networks do not replace the world\-model transition distribution\. The environment reward and continuation predictors also retain their original meanings\. The proposed modification concerns return expansion and sample weighting, rather than a new reward function or an additional environment termination condition\.

### 5\.2Learning the Auxiliary Models

Letx¯i=sg⁡\(Fϕ​\(zi\)\)\\bar\{x\}\_\{i\}=\\operatorname\{sg\}\(F\_\{\\phi\}\(z\_\{i\}\)\)be a replay posterior feature, wheresg\\operatorname\{sg\}denotes stop\-gradient\. For a replay sequence withLLstates, letbib\_\{i\}be the action that produces the transition from stateiito statei\+1i\+1, and letmim\_\{i\}indicate that the pair does not cross an episode reset\. We train a displacement networkfω​\(ui,x¯i,bi\)f\_\{\\omega\}\(u\_\{i\},\\bar\{x\}\_\{i\},b\_\{i\}\)with normalized sequence\-position inputui=i/\(L−1\)u\_\{i\}=i/\(L\-1\):

ℒdisp=∑imi​‖fω​\(ui,x¯i,bi\)−\(x¯i\+1−x¯i\)‖22d​max⁡\(1,∑imi\)\.\\mathcal\{L\}\_\{\\mathrm\{disp\}\}=\\frac\{\\sum\_\{i\}m\_\{i\}\\left\\\|f\_\{\\omega\}\(u\_\{i\},\\bar\{x\}\_\{i\},b\_\{i\}\)\-\(\\bar\{x\}\_\{i\+1\}\-\\bar\{x\}\_\{i\}\)\\right\\\|\_\{2\}^\{2\}\}\{d\\,\\max\(1,\\sum\_\{i\}m\_\{i\}\)\}\.\(12\)The sums include all sequence pairs in the batch\. Actions are aligned with the transition they generated; in the replay storage convention used by our implementation, this corresponds to the action stored with the succeeding observation\.

The target is a one\-step feature displacement\. Although the implementation names this network a latent ODE component, training does not integrate an ODE or divide the displacement by a continuous\-time increment\. The inputuiu\_\{i\}is a sequence\-position conditioning variable, not an integration step size\.

The score network uses the noise\-prediction parameterization from Section[3\.3](https://arxiv.org/html/2610.41022#S3.SS3)\. With detached replay features, noise scaleα\\alphadrawn uniformly from\[αmin,αmax\]\[\\alpha\_\{\\min\},\\alpha\_\{\\max\}\], andϵ∼𝒩⁡\(0,Id\)\\epsilon\\sim\\mathcal\{N\}\(0,I\_\{d\}\), its implemented loss is

ℒscore=𝔼⁡\[1d​‖ϵη​\(x¯\+α​ϵ,α\)−ϵ‖22\]\.\\mathcal\{L\}\_\{\\mathrm\{score\}\}=\\mathbb\{E\}\\left\[\\frac\{1\}\{d\}\\left\\\|\\epsilon\_\{\\eta\}\(\\bar\{x\}\+\\alpha\\epsilon,\\alpha\)\-\\epsilon\\right\\\|\_\{2\}^\{2\}\\right\]\.\(13\)This is an unweighted denoising loss: neither rewards nor density gates weight its samples\. The combined model\-training objective is

ℒmodel=ℒWM\+βdisp​ℒdisp\+βscore​ℒscore,\\mathcal\{L\}\_\{\\mathrm\{model\}\}=\\mathcal\{L\}\_\{\\mathrm\{WM\}\}\+\\beta\_\{\\mathrm\{disp\}\}\\mathcal\{L\}\_\{\\mathrm\{disp\}\}\+\\beta\_\{\\mathrm\{score\}\}\\mathcal\{L\}\_\{\\mathrm\{score\}\},\(14\)whereℒWM\\mathcal\{L\}\_\{\\mathrm\{WM\}\}is the base world\-model objective\. Detaching the auxiliary inputs prevents these losses from directly updating the encoder or latent transition model through their feature targets\. In the implementation, the losses share a model optimizer and global gradient clipping, so an auxiliary\-enabled control remains distinct from an unmodified baseline\.

### 5\.3Constructing the Relative Density Proxy

At a fixed positive scaleσ\\sigma, define the score\-corrected auxiliary field

vω,η​\(u,x,a\)=fω​\(u,x,a\)−σ22​sη​\(x,σ\)=fω​\(u,x,a\)\+σ2​ϵη​\(x,σ\)\.v\_\{\\omega,\\eta\}\(u,x,a\)=f\_\{\\omega\}\(u,x,a\)\-\\frac\{\\sigma^\{2\}\}\{2\}s\_\{\\eta\}\(x,\\sigma\)=f\_\{\\omega\}\(u,x,a\)\+\\frac\{\\sigma\}\{2\}\\epsilon\_\{\\eta\}\(x,\\sigma\)\.\(15\)The second equality uses Equation \([9](https://arxiv.org/html/2610.41022#S3.E9)\) and specifies the sign and scale of the implemented correction\. The probability\-flow construction motivates this form, butσ\\sigmais an auxiliary hyperparameter, not a calibrated diffusion coefficient of the stochastic latent model\. We evaluate the predictor at the clean imagined feature with conditioning valueσ\\sigma\. Choosingσ\\sigmawithin the training noise range does not remove the distribution shift between corrupted replay features and clean imagined features\.

For each imagined state, we estimate spatial divergence with a stochastic trace estimator:

D^t=ξt⊤​\[∇xvω,η​\(ut,xt,at\)\]​ξt,𝔼⁡\[ξt​ξt⊤\]=Id\.\\widehat\{D\}\_\{t\}=\\xi\_\{t\}^\{\\top\}\\left\[\\nabla\_\{x\}v\_\{\\omega,\\eta\}\(u\_\{t\},x\_\{t\},a\_\{t\}\)\\right\]\\xi\_\{t\},\\qquad\\mathbb\{E\}\[\\xi\_\{t\}\\xi\_\{t\}^\{\\top\}\]=I\_\{d\}\.\(16\)For example, independent Rademacher entries satisfy the stated covariance condition\. The action and position input are held fixed when differentiating with respect toxtx\_\{t\}; thus this is not the total derivative of a policy\-composed closed\-loop field\. The derivative includes the spatial dependence of the score network\. Stopping gradients through the score input before this calculation would remove its contribution to the divergence\.

We construct bounded increments and accumulate them from a common reference:

δt=clip\(−D^t,−C,C\),ℓ0=0,ℓt\+1=ℓt\+δt,t=0,…,H−1\.\\delta\_\{t\}=\\operatorname\{clip\}\(\-\\widehat\{D\}\_\{t\},\-C,C\),\\qquad\\ell\_\{0\}=0,\\qquad\\ell\_\{t\+1\}=\\ell\_\{t\}\+\\delta\_\{t\},\\quad t=0,\\ldots,H\-1\.\(17\)The increment capC\>0C\>0limits the influence of an individual divergence estimate\. It also changes the signal and can conceal differences between fields when clipping is frequent\. Conditional unbiasedness of the trace estimate therefore does not imply unbiasedness of the clipped increment or of the resulting proxy\.

We compute the required spatial derivatives before detachingℓt\\ell\_\{t\}from actor–critic optimization\. No absolute starting\-state density is estimated, and the proxy is not exponentiated in the gate construction\. It measures an anchored accumulation along the imagined path, rather than an absolute ranking of replay\-state likelihoods\. The field’s mismatch with the actual latent transition process and its dependence on representation scale remain relevant limitations\.

### 5\.4Hard and Soft Gates

Letτ\\taube a density threshold andT\>0T\>0a soft\-gate temperature\. The candidate gates are

g~thard\\displaystyle\\widetilde\{g\}\_\{t\}^\{\\mathrm\{hard\}\}=𝟏\{ℓt≥τ\},\\displaystyle=\\mathbf\{1\}\\\{\\ell\_\{t\}\\geq\\tau\\\},\(18\)g~tsoft\\displaystyle\\widetilde\{g\}\_\{t\}^\{\\mathrm\{soft\}\}=11\+exp\(−\(ℓt−τ\)/T\)\.\\displaystyle=\\frac\{1\}\{1\+\\exp\(\-\(\\ell\_\{t\}\-\\tau\)/T\)\}\.\(19\)Letm∈\{1,…,H\}m\\in\\\{1,\\ldots,H\\\}be the minimum number of retained transition\-loss entries andNwarmN\_\{\\mathrm\{warm\}\}the warmup duration in the implementation’s agent\-step units\. At training stepNN, the applied gate is

gt=\{1,N<Nwarm​or​t<m,g~t,otherwise\.g\_\{t\}=\\begin\{cases\}1,&N<N\_\{\\mathrm\{warm\}\}\\ \\text\{or\}\\ t<m,\\\\ \\widetilde\{g\}\_\{t\},&\\text\{otherwise\}\.\\end\{cases\}\(20\)This warmup delays activation; it is not a gradual interpolation of gate values\. SettingNwarm=0N\_\{\\mathrm\{warm\}\}=0activates selection from the start\.

Define survival through statettas

St=∏j=0tgj\.S\_\{t\}=\\prod\_\{j=0\}^\{t\}g\_\{j\}\.\(21\)For hard gates, the first zero gate makes all subsequent survival weights zero, even if the proxy later rises above the threshold\. For soft gates, survival decreases multiplicatively along the trajectory\. Consequently, several moderate gate values can strongly attenuate later contributions even without a single sharp threshold crossing\. We use these soft weights deterministically; the implementation does not sample a stopping time\.

A useful diagnostic is the effective number of transition\-loss entries,

Heff=∑t=0H−1St\.H\_\{\\mathrm\{eff\}\}=\\sum\_\{t=0\}^\{H\-1\}S\_\{t\}\.\(22\)It equals the retained count in hard mode and a fractional survival mass in soft mode\. It excludes discounting andλ\\lambda\-mixing, so it is not a complete measure of the influence of future rewards on a target\. A fixed\-boundary control usesgt=𝟏\{t<K\}g\_\{t\}=\\mathbf\{1\}\\\{t<K\\\}for a prescribedK≤HK\\leq H\.

### 5\.5Density\-Gated Return Recursion

We insert the gate at the succeeding state into the return recursion:

Gtg=r^t\+1\+dt\+1​\[\(1−λ​gt\+1\)​Vψ​\(xt\+1\)\+λ​gt\+1​Gt\+1g\],GHg=Vψ​\(xH\)\.G\_\{t\}^\{g\}=\\widehat\{r\}\_\{t\+1\}\+d\_\{t\+1\}\\left\[\(1\-\\lambda g\_\{t\+1\}\)V\_\{\\psi\}\(x\_\{t\+1\}\)\+\\lambda g\_\{t\+1\}G\_\{t\+1\}^\{g\}\\right\],\\qquad G\_\{H\}^\{g\}=V\_\{\\psi\}\(x\_\{H\}\)\.\(23\)The gate therefore modulates the effective trace coefficient,λ​gt\+1\\lambda g\_\{t\+1\}, rather than the environment continuation prediction\.

Whengt\+1=0g\_\{t\+1\}=0, the target becomes

Gtg=r^t\+1\+dt\+1​Vψ​\(xt\+1\),G\_\{t\}^\{g\}=\\widehat\{r\}\_\{t\+1\}\+d\_\{t\+1\}V\_\{\\psi\}\(x\_\{t\+1\}\),\(24\)retaining both the incoming reward and the discounted boundary value\. When all gates equal one, Equation \([23](https://arxiv.org/html/2610.41022#S5.E23)\) recovers Equation \([4](https://arxiv.org/html/2610.41022#S3.E4)\)\. The terminal gategHg\_\{H\}does not alter the final backup becauseGHgG\_\{H\}^\{g\}already equalsVψ​\(xH\)V\_\{\\psi\}\(x\_\{H\}\)\.

In hard mode, the first failed gate determines the bootstrap boundary for targets rooted in the surviving prefix\. Targets for later states may still be computed in the full tensor, but their actor and critic loss weights are zero\. This separates stopping return expansion from stopping trajectory generation\.

### 5\.6Actor–Critic Objectives and Gradient Handling

LetWtW\_\{t\}denote the detached discount weight inherited from the base agent\. We apply the combined weight

Ωt=sg⁡\(Wt​St\)\.\\Omega\_\{t\}=\\operatorname\{sg\}\(W\_\{t\}S\_\{t\}\)\.\(25\)For a batch ofBBimagined trajectories, a distributional critic with predictive distributionpψVp\_\{\\psi\}^\{V\}is trained using

ℒV=1B​H​∑b=1B∑t=0H−1Ωb,t​\[−log⁡pψV​\(sg⁡\(Gb,tg\)∣sg⁡\(xb,t\)\)\+ℛb,tslow\]\.\\mathcal\{L\}\_\{V\}=\\frac\{1\}\{BH\}\\sum\_\{b=1\}^\{B\}\\sum\_\{t=0\}^\{H\-1\}\\Omega\_\{b,t\}\\left\[\-\\log p\_\{\\psi\}^\{V\}\\bigl\(\\operatorname\{sg\}\(G\_\{b,t\}^\{g\}\)\\mid\\operatorname\{sg\}\(x\_\{b,t\}\)\\bigr\)\+\\mathcal\{R\}\_\{b,t\}^\{\\mathrm\{slow\}\}\\right\]\.\(26\)Hereℛslow\\mathcal\{R\}^\{\\mathrm\{slow\}\}is the base agent’s optional slow\-critic regularization term\. In our implementation it is the negative log probability assigned by the current critic to the detached scalar prediction of the slow critic\.

Let𝒜b,t\\mathcal\{A\}\_\{b,t\}denote the actor surrogate used by the base gradient estimator\. The actor objective is

ℒπ=−1B​H∑b=1B∑t=0H−1\[Ωb,t𝒜b,t\+βentSb,tℋ\(πθ\(⋅∣xb,t\)\)\]\.\\mathcal\{L\}\_\{\\pi\}=\-\\frac\{1\}\{BH\}\\sum\_\{b=1\}^\{B\}\\sum\_\{t=0\}^\{H\-1\}\\left\[\\Omega\_\{b,t\}\\mathcal\{A\}\_\{b,t\}\+\\beta\_\{\\mathrm\{ent\}\}S\_\{b,t\}\\mathcal\{H\}\\bigl\(\\pi\_\{\\theta\}\(\\cdot\\mid x\_\{b,t\}\)\\bigr\)\\right\]\.\(27\)The entropy term receives survival weighting; it does not receive the additionalWtW\_\{t\}factor in the implemented base objective\.

For continuous\-control experiments using dynamics gradients, the surrogate is the return\-minus\-baseline difference divided by the base return\-normalization scale, with differentiation through the retained imagination computation\. For discrete\-action experiments using a likelihood\-ratio estimator, the implemented surrogate is

𝒜tLR=log⁡πθ​\(at∣sg⁡\(xt\)\)​sg⁡\(Gtg−Vψ​\(xt\)\)\.\\mathcal\{A\}\_\{t\}^\{\\mathrm\{LR\}\}=\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid\\operatorname\{sg\}\(x\_\{t\}\)\)\\,\\operatorname\{sg\}\\bigl\(G\_\{t\}^\{g\}\-V\_\{\\psi\}\(x\_\{t\}\)\\bigr\)\.\(28\)This likelihood\-ratio expression concerns the policy distribution and is distinct from the latent\-density score estimator\.

Gates and survival weights are detached for both gradient estimators\. Thus actor updates do not differentiate through the threshold, sigmoid, or density diagnostic\. The critic targets and critic input features are also detached in the critic update\. During dynamics\-based actor learning, world\-model and auxiliary parameters remain fixed while the base differentiable imagination paths are retained\.

The denominator in Equations \([26](https://arxiv.org/html/2610.41022#S5.E26)\) and \([27](https://arxiv.org/html/2610.41022#S5.E27)\) is the full batch\-time sizeB​HBH, not the surviving weight mass\. Shorter effective horizons can therefore change update magnitude as well as sample selection\. The implementation also excludes zero\-weight targets from return\-scale statistics in hard and fixed modes, whereas soft mode retains all targets for those statistics\. These choices must be reported and controlled when attributing improvements to density\-dependent selection\.

### 5\.7Algorithm and DreamerV3 Instantiation

Algorithm[1](https://arxiv.org/html/2610.41022#alg1)summarizes the training procedure\. The counterNNuses the implementation’s agent\-step convention\. The stochastic world model generates the full imagined trajectory before boundary selection\. The spatial derivative includes the score network’s dependence on its input; detachment is applied only after the divergence\-based proxy has been computed\.

Algorithm 1Density\-Guided Imagined Actor\-Critic Learning0:World model parameters

ϕ\\phi; actor

πθ\\pi\_\{\\theta\}; critic

VψV\_\{\\psi\}; displacement field

fωf\_\{\\omega\}; noise predictor

ϵη\\epsilon\_\{\\eta\}; replay buffer

𝒟\\mathcal\{D\}\.

0:Imagination horizon

HH; trace coefficient

λ\\lambda; threshold

τ\\tau; temperature

TT; increment cap

CC; score scale

σ\\sigma; minimum retained depth

mm; activation step

NwarmN\_\{\\text\{warm\}\}; boundary mode

ℳ∈\{hard,soft\}\\mathcal\{M\}\\in\\\{\\text\{hard\},\\text\{soft\}\\\}\.

1:whileinteraction budget remainsdo

2:Collect environment experience using

πθ\\pi\_\{\\theta\}and append it to

𝒟\\mathcal\{D\}\.

3:Sample replay sequences and infer posterior latent features\.

4:Update world model and auxiliary parameters using

ℒWM\+βdisp​ℒdisp\+βscore​ℒscore\\mathcal\{L\}\_\{\\text\{WM\}\}\+\\beta\_\{\\text\{disp\}\}\\mathcal\{L\}\_\{\\text\{disp\}\}\+\\beta\_\{\\text\{score\}\}\\mathcal\{L\}\_\{\\text\{score\}\}\(Algorithm[2](https://arxiv.org/html/2610.41022#alg2)\)\.

5:Start from detached replay posterior states and generate full stochastic imagined trajectory

x0,a0,x1,…,aH−1,xHx\_\{0\},a\_\{0\},x\_\{1\},\\dots,a\_\{H\-1\},x\_\{H\}\.

6:Predict rewards

r^t\+1\\hat\{r\}\_\{t\+1\}, discounts

dt\+1=γ​c^t\+1d\_\{t\+1\}=\\gamma\\hat\{c\}\_\{t\+1\}, and values

Vψ​\(xt\)V\_\{\\psi\}\(x\_\{t\}\)\.

7:Set

l0←0l\_\{0\}\\leftarrow 0\.

8:for

t=0,…,H−1t=0,\\dots,H\-1do

9:Define

vt​\(x\)←fω​\(ut,x,at\)\+σ2​ϵη​\(x,σ\)v\_\{t\}\(x\)\\leftarrow f\_\{\\omega\}\(u\_\{t\},x,a\_\{t\}\)\+\\frac\{\\sigma\}\{2\}\\epsilon\_\{\\eta\}\(x,\\sigma\)with

ut=t/Hu\_\{t\}=t/H\.

10:Draw an independent trace probe

ξt\\xi\_\{t\}satisfying

𝔼⁡\[ξt​ξt⊤\]=𝐈\\mathbb\{E\}\[\\xi\_\{t\}\\xi\_\{t\}^\{\\top\}\]=\\mathbf\{I\}\.

11:Estimate

D^t←ξt⊤​\[∇xvt​\(x\)\]x=xt​ξt\\hat\{D\}\_\{t\}\\leftarrow\\xi\_\{t\}^\{\\top\}\[\\nabla\_\{x\}v\_\{t\}\(x\)\]\_\{x=x\_\{t\}\}\\xi\_\{t\}, holding

ata\_\{t\}and

utu\_\{t\}fixed\.

12:Set

lt\+1←lt\+clip​\(−D^t,−C,C\)l\_\{t\+1\}\\leftarrow l\_\{t\}\+\\text\{clip\}\(\-\\hat\{D\}\_\{t\},\-C,C\)\.

13:endfor

14:Detach density proxies

l0:Hl\_\{0:H\}from actor\-critic optimization\.

15:for

t=0,…,Ht=0,\\dots,Hdo

16:if

N<NwarmN<N\_\{\\text\{warm\}\}or

t<mt<mthen

17:

gt←1g\_\{t\}\\leftarrow 1\.

18:elseif

ℳ=hard\\mathcal\{M\}=\\text\{hard\}then

19:

gt←𝕀⁡\(lt≥τ\)g\_\{t\}\\leftarrow\\mathbb\{I\}\(l\_\{t\}\\geq\\tau\)\.

20:else

21:

gt←\[1\+exp\(−\(lt−τ\)/T\)\]−1g\_\{t\}\\leftarrow\[1\+\\exp\(\-\(l\_\{t\}\-\\tau\)/T\)\]^\{\-1\}\.

22:endif

23:endfor

24:Set

S0←g0S\_\{0\}\\leftarrow g\_\{0\}\.

25:for

t=1,…,Ht=1,\\dots,Hdo

26:

St←St−1​gtS\_\{t\}\\leftarrow S\_\{t\-1\}g\_\{t\}\.

27:endfor

28:Set

GH←Vψ​\(xH\)G\_\{H\}\\leftarrow V\_\{\\psi\}\(x\_\{H\}\)\.

29:for

t=H−1,…,0t=H\-1,\\dots,0do

30:

Gt←r^t\+1\+dt\+1​\[\(1−λ​gt\+1\)​Vψ​\(xt\+1\)\+λ​gt\+1​Gt\+1\]G\_\{t\}\\leftarrow\\hat\{r\}\_\{t\+1\}\+d\_\{t\+1\}\[\(1\-\\lambda g\_\{t\+1\}\)V\_\{\\psi\}\(x\_\{t\+1\}\)\+\\lambda g\_\{t\+1\}G\_\{t\+1\}\]\.

31:endfor

32:Form detached loss weights

Ωt←Wt​St\\Omega\_\{t\}\\leftarrow W\_\{t\}S\_\{t\}for

t=0,…,H−1t=0,\\dots,H\-1using base agent discount weights

WtW\_\{t\}\.

33:Update actor and critic using gated returns and survival\-weighted objectives \(Section 5\.6\)\.

34:Apply base agent’s slow\-critic update and advance training counters\.

35:endwhile

The general formulation requires a latent feature representation, imagined transitions, and a bootstrap value estimator\. Our implementation uses DreamerV3’s recurrent stochastic state\-space model and distributional critic\([Hafner et al\., 2023](https://arxiv.org/html/2610.41022#bib.bib4)\)\. The auxiliary networks operate on the concatenation of deterministic and stochastic latent features\. In the configuration with a 512\-dimensional deterministic component and 32 categorical variables with 32 categories each, this feature vector has dimension512\+32⋅32=1536512\+32\\cdot 32=1536\.

Section[3](https://arxiv.org/html/2610.41022#S3)definesHHas a transition count\. The implementation’simag\_horizoncounts feature entries: setting it to 15 produces 14 return\-training transitions, corresponding toH=14H=14here\. Its auxiliary position input during imagination isut=t/\(H\+1\)u\_\{t\}=t/\(H\+1\), while replay usesi/\(L−1\)i/\(L\-1\)\. These are sequence\-relative conditioning choices, not a shared physical\-time discretization\.

For implementation fidelity, letc^t\\widehat\{c\}\_\{t\}denote the continuation prediction at feature entrytt, including the initial entry\. The inherited discount\-loss weights are

W0=1,Wt=∏j=0t−1\(γ​c^j\),t≥1\.W\_\{0\}=1,\\qquad W\_\{t\}=\\prod\_\{j=0\}^\{t\-1\}\(\\gamma\\widehat\{c\}\_\{j\}\),\\quad t\\geq 1\.\(29\)The return recursion separately uses the next\-state predictionsdt\+1=γ​c^t\+1d\_\{t\+1\}=\\gamma\\widehat\{c\}\_\{t\+1\}\. We preserve this indexing rather than silently changing the base agent’s discount weights\. Warmup is measured by the agent\-step counter\. With action repeat two, 100,000 agent steps correspond to 200,000 logged environment steps\. Auxiliary training can proceed during this period while all gates remain one\. Direct density reward weighting and learned auxiliary diffusion are disabled in the boundary experiments described here\. The chosen threshold, temperature, increment cap, and activation time are experimental settings, not universal constants of the method\. Finally, all imagined states are generated before gating\. The method adapts the portion of imagination used in return expansion and optimization; this implementation does not claim computational savings from early rollout termination\. Appendix[E](https://arxiv.org/html/2610.41022#A5)provides DreamerV3\-specific pseudocode, including auxiliary training, RSSM imagination, gradient handling, and loss weighting\.

## 6Theoretical Analysis

###### Remark 1\(Diagnostic Nature of the Score Proxy\)\.

The accumulated signallt=∑j=0t−1δjl\_\{t\}=\\sum\_\{j=0\}^\{t\-1\}\\delta\_\{j\}tracks local volumetric expansion and contraction of the auxiliary fieldvω,ηv\_\{\\omega,\\eta\}evaluated at states sampled fromℙRSSM\\mathbb\{P\}\_\{\\text\{RSSM\}\}\. It does not quantify exact path probabilities or absolute likelihoods under the world model’s discrete stochastic transition kernel\.

###### Remark 2\(Role of the Path Drift Gap\)\.

The structural path drift gapϵdrift=DKL\(ℙRSSM∥ℙFlow\)\\epsilon\_\{\\text\{drift\}\}=D\_\{\\text\{KL\}\}\(\\mathbb\{P\}\_\{\\text\{RSSM\}\}\\parallel\\mathbb\{P\}\_\{\\text\{Flow\}\}\)represents an irreducible structural offset\. The efficacy ofltl\_\{t\}for gating return expansion relies on its ability to detect local feature\-space divergence, not on eliminatingϵdrift\\epsilon\_\{\\text\{drift\}\}\.

###### Proposition 1\(Optimal Denoising Score Limit\([Vincent, 2011](https://arxiv.org/html/2610.41022#bib.bib1);[Song et al\., 2021](https://arxiv.org/html/2610.41022#bib.bib8)\)\)\.

Letp⁡\(x\)p\(x\)be the empirical posterior feature distribution on replay, and letpσ​\(x~\)=∫p⁡\(x\)​𝒩​\(x~∣x,σ2​Id\)​𝑑xp\_\{\\sigma\}\(\\tilde\{x\}\)=\\int p\(x\)\\mathcal\{N\}\(\\tilde\{x\}\\mid x,\\sigma^\{2\}I\_\{d\}\)\\,dxdenote the smoothed marginal density\. If the noise\-conditioned predictorϵη\\epsilon\_\{\\eta\}minimizes the population denoising objectiveℒscore\\mathcal\{L\}\_\{\\mathrm\{score\}\}, then for any fixed noise scaleσ\>0\\sigma\>0, the parameterizationsη∗​\(x~,σ\)=−ϵη∗​\(x~,σ\)σs\_\{\\eta\}^\{\*\}\(\\tilde\{x\},\\sigma\)=\-\\frac\{\\epsilon\_\{\\eta\}^\{\*\}\(\\tilde\{x\},\\sigma\)\}\{\\sigma\}yields the exact score of the perturbed distribution∇x~​log​pσ​\(x~\)\\nabla\_\{\\tilde\{x\}\}\\log p\_\{\\sigma\}\(\\tilde\{x\}\)\. Furthermore, in the limit of vanishing noiseσ→0\\sigma\\to 0:

limσ→0sη∗​\(x,σ\)=limσ→0\(−ϵη∗​\(x,σ\)σ\)=∇x​log​p​\(x\)\.\\lim\_\{\\sigma\\to 0\}s\_\{\\eta\}^\{\*\}\(x,\\sigma\)=\\lim\_\{\\sigma\\to 0\}\\left\(\-\\frac\{\\epsilon\_\{\\eta\}^\{\*\}\(x,\\sigma\)\}\{\\sigma\}\\right\)=\\nabla\_\{x\}\\log p\(x\)\.\(30\)

###### Theorem 1\(Bias\-Variance Trade\-off in Gated Returns\)\.

LetGtg=r^t\+1\+dt\+1​\[\(1−λ​gt\+1\)​Vψ​\(xt\+1\)\+λ​gt\+1​Gt\+1g\]G\_\{t\}^\{g\}=\\hat\{r\}\_\{t\+1\}\+d\_\{t\+1\}\[\(1\-\\lambda g\_\{t\+1\}\)V\_\{\\psi\}\(x\_\{t\+1\}\)\+\\lambda g\_\{t\+1\}G\_\{t\+1\}^\{g\}\]be the gated return target, ande⁡\(x\)=Vψ​\(x\)−Vπ​\(x\)e\(x\)=V\_\{\\psi\}\(x\)\-V^\{\\pi\}\(x\)be the approximation bias of the critic\. The continuation gategt\+1∈\[0,1\]g\_\{t\+1\}\\in\[0,1\]directly modulates the recursive propagation of critic bias and downstream rollout variance:

Bias​\(Gtg\)=𝔼⁡\[dt\+1​\(1−λ​gt\+1\)​e​\(xt\+1\)\]\+𝔼⁡\[dt\+1​λ​gt\+1​Bias​\(Gt\+1g\)\]\.\\text\{Bias\}\(G\_\{t\}^\{g\}\)=\\mathbb\{E\}\[d\_\{t\+1\}\(1\-\\lambda g\_\{t\+1\}\)e\(x\_\{t\+1\}\)\]\+\\mathbb\{E\}\[d\_\{t\+1\}\\lambda g\_\{t\+1\}\\text\{Bias\}\(G\_\{t\+1\}^\{g\}\)\]\.\(31\)

###### Theorem 2\(Diagnostic Proxy Variance and Discrepancy Bound\)\.

Let𝒳⊂ℝd\\mathcal\{X\}\\subset\\mathbb\{R\}^\{d\}be a compact latent feature domain\. Assume the auxiliary vector fieldvω,ηv\_\{\\omega,\\eta\}has anLJL\_\{J\}\-Lipschitz continuous JacobianJ⁡\(xt\)=∇xvω,η​\(ut,xt,at\)J\(x\_\{t\}\)=\\nabla\_\{x\}v\_\{\\omega,\\eta\}\(u\_\{t\},x\_\{t\},a\_\{t\}\)over𝒳\\mathcal\{X\}, with neural approximation errors bounded by‖fω−Δ​x‖2≤ϵf\\\|f\_\{\\omega\}\-\\Delta x\\\|\_\{2\}\\leq\\epsilon\_\{f\}and‖ϵη−ϵ‖2≤ϵϵ\\\|\\epsilon\_\{\\eta\}\-\\epsilon\\\|\_\{2\}\\leq\\epsilon\_\{\\epsilon\}\.

With probability at least1−δ1\-\\delta, the deviation between the empirical single\-step diagnostic updateδt=clip​\(−D^t,−C,C\)\\delta\_\{t\}=\\text\{clip\}\(\-\\hat\{D\}\_\{t\},\-C,C\)and the local spatial traceTr​\(−J​\(xt\)\)\\text\{Tr\}\(\-J\(x\_\{t\}\)\)of the score\-corrected field along an RSSM transitionΔ​xt=xt\+1−xt\\Delta x\_\{t\}=x\_\{t\+1\}\-x\_\{t\}is bounded by:

\|δt−Tr​\(−J⁡\(xt\)\)\|≤2​‖J⁡\(xt\)‖F2δ\+𝒪⁡\(βf​ϵf\+βϵ​ϵϵ\+‖Δ​xt‖22\)\+ϵdrift\.\|\\delta\_\{t\}\-\\text\{Tr\}\(\-J\(x\_\{t\}\)\)\|\\leq\\frac\{2\\\|J\(x\_\{t\}\)\\\|\_\{F\}^\{2\}\}\{\\delta\}\+\\mathcal\{O\}\(\\beta\_\{f\}\\epsilon\_\{f\}\+\\beta\_\{\\epsilon\}\\epsilon\_\{\\epsilon\}\+\\\|\\Delta x\_\{t\}\\\|\_\{2\}^\{2\}\)\+\\epsilon\_\{\\text\{drift\}\}\.\(32\)

###### Proof\.

Decompose the total error into four constituent bounds via the triangle inequality:

\|δt−Tr​\(−J⁡\(xt\)\)\|≤Etrace\+Eapprox\+Eeuler\+ϵdrift\.\|\\delta\_\{t\}\-\\text\{Tr\}\(\-J\(x\_\{t\}\)\)\|\\leq E\_\{\\text\{trace\}\}\+E\_\{\\text\{approx\}\}\+E\_\{\\text\{euler\}\}\+\\epsilon\_\{\\text\{drift\}\}\.\(33\)1. 1\.Hutchinson Probe Variance \(EtraceE\_\{\\text\{trace\}\}\):D^t=ξt⊤​J​\(xt\)​ξt\\hat\{D\}\_\{t\}=\\xi\_\{t\}^\{\\top\}J\(x\_\{t\}\)\\xi\_\{t\}using a Rademacher probe vectorξt∈\{−1,1\}d\\xi\_\{t\}\\in\\\{\-1,1\\\}^\{d\}\. Since𝔼⁡\[D^t\]=Tr​\(J⁡\(xt\)\)\\mathbb\{E\}\[\\hat\{D\}\_\{t\}\]=\\text\{Tr\}\(J\(x\_\{t\}\)\)with variance bounded by2​‖J⁡\(xt\)‖F22\\\|J\(x\_\{t\}\)\\\|\_\{F\}^\{2\}, Chebyshev’s inequality guaranteesEtrace≤2​‖J⁡\(xt\)‖F2δE\_\{\\text\{trace\}\}\\leq\\frac\{2\\\|J\(x\_\{t\}\)\\\|\_\{F\}^\{2\}\}\{\\delta\}with probability at least1−δ1\-\\delta\.
2. 2\.Function Approximation Error \(EapproxE\_\{\\text\{approx\}\}\):Smoothness of activations ensures localLJL\_\{J\}\-Lipschitz continuity of the Jacobian over compact set𝒳\\mathcal\{X\}\. Errors in vector field components propagate into the Jacobian trace bounded by𝒪⁡\(βf​ϵf\+βϵ​ϵϵ\)\\mathcal\{O\}\(\\beta\_\{f\}\\epsilon\_\{f\}\+\\beta\_\{\\epsilon\}\\epsilon\_\{\\epsilon\}\)\.
3. 3\.Quadrature Residual \(EeulerE\_\{\\text\{euler\}\}\):Forward Euler integration of the step expansion over a unit transition stepΔ​u=1\\Delta u=1incurs a local truncation residual proportional to the squared latent displacement,𝒪⁡\(‖Δ​xt‖22\)\\mathcal\{O\}\(\\\|\\Delta x\_\{t\}\\\|\_\{2\}^\{2\}\)\.
4. 4\.Path Disconnect \(ϵdrift\\epsilon\_\{\\text\{drift\}\}\):The additive structural offset accounting for evaluating diagnostic divergence along trajectories sampled fromℙRSSM\\mathbb\{P\}\_\{\\text\{RSSM\}\}rather than integrated alongℙFlow\\mathbb\{P\}\_\{\\text\{Flow\}\}\.

∎

## 7Experimental Setup

Our experiments investigate whether density\-guided bootstrap selection improves learning beyond the effects of auxiliary training and shorter imagined returns\. All variants use the same modified DreamerV3 implementation\. We separate the choice of boundary rule from the world\-model architecture, interaction budget, and optimization settings\.

### 7\.1Environments and Observation Settings

The continuous\-control study considers the DeepMind Control Suite taskscheetah\_run,walker\_walk,walker\_run, andfinger\_turn\_hard\. These experiments use vector observations through thedmc\_proprioconfiguration, with multilayer\-perceptron encoders and decoders\. Cheetah refers specifically to the Control Suite task, rather than the Gym MuJoCo HalfCheetah benchmark\. The discrete\-action study uses Atari Pong with64×6464\\times 64RGB observations and theatari100kconfiguration\. This setting provides a complementary test involving visual representation learning and a categorical action policy\. Table[1](https://arxiv.org/html/2610.41022#S7.T1)summarizes the interaction settings\.

Table 1:Observation and interaction settings\. Budgets follow the repeat\-adjusted logging convention, corresponding nominally to 250,000 agent decisions for Control Suite and 100,000 for Pong\.The main comparison evaluates vanilla DreamerV3 and the density\-guided hard and soft variants using 10 independent training seeds per method and task\. Auxiliary and fixed\-boundary controls are reported separately in Sections 7\.2 and 7\.3\. The boundary\-behavior illustrations use one supplied training run per variant, and the planned threshold\-sensitivity study uses seed 0\.

### 7\.2Reference Agents and Controlled Comparisons

The evaluation design includes the following variants:

1. 1\.Vanilla DreamerV3\.The base return construction is retained\. Auxiliary displacement and score training, density selection, and direct density weighting are disabled\.
2. 2\.Auxiliary\-enabled, cutoff off\.The auxiliary networks are trained and the density diagnostic is computed, but no density boundary modifies returns or loss weights\. This control is distinct from vanilla DreamerV3\.
3. 3\.Density\-guided hard boundary\.The score\-corrected proxy selects a hard bootstrap boundary, with the survival\-weighted objectives defined in Section[5](https://arxiv.org/html/2610.41022#S5)\.
4. 4\.Density\-guided soft boundary\.The same proxy supplies sigmoid continuation gates and cumulative survival weights\. This changes return expansion and loss weighting rather than directly multiplying rewards by density\.
5. 5\.Fixed bootstrap boundary\.A prescribed boundary replaces trajectory\-dependent selection while retaining the boundary value\. To isolate the selection rule, auxiliary computation should match the density\-guided variant\. A separate fixed\-boundary agent without auxiliaries measures the performance of the simpler alternative\.

These variants define the ablation design; only completed comparisons enter the reported results\. Matching a fixed boundary to the mean effective horizon does not match its distribution or evolution during training\. If the adaptive horizon changes substantially, a predetermined fixed\-horizon schedule provides an additional control\. Any schedule derived from a reference run must be frozen before evaluation and disclosed\.

### 7\.3Training Configuration

Within each comparison, we retain the same architecture, replay settings, world\-model losses, optimizers, and interaction budget unless an explicitly named ablation changes them\. The supplied configuration uses batch size 16, replay sequence length 64, discount 0\.997, and return\-mixing coefficient 0\.95\. An imagination setting of 15 feature entries produces 14 return\-training transitions\.

The auxiliary setup usesσ=0\.05\\sigma=0\.05, denoising scales sampled from\[0\.01,0\.10\]\[0\.01,0\.10\], displacement\-loss coefficient10−310^\{\-3\}, and score\-loss coefficient10−410^\{\-4\}\. Learned auxiliary diffusion, direct density reward weightingare disabled\.

The Control Suite configuration uses four environments, action repeat two, dynamics\-based actor gradients, and training\-ratio parameter 512\. The Atari configuration uses one environment, action repeat four, likelihood\-ratio actor gradients, and training\-ratio parameter 1024\. Training\-ratio values are implementation configuration parameters whose meaning depends on the update scheduler\. Compilation is disabled in the experimental commands\.

### 7\.4Boundary Parameters and Sensitivity Studies

The current density\-boundary experiments retain a minimum ofm=2m=2transition\-loss entries\. The Cheetah threshold study variesτ\\tauover\{−500,−700,−900\}\\\{\-500,\-700,\-900\\\}for hard and soft modes\. It holds the soft temperature atT=100T=100, increment cap atC=200C=200, and activation point at 200,000 logged environment steps\. With action repeat two, activation corresponds to 100,000 agent steps\.

This threshold sweep is exploratory and uses seed 0\. Existing−700\-700runs can be reused only if code, clipping, warmup, and other settings match\. Early runs using increment cap 5 are not interchangeable with cap\-200 runs\. Since no suitable pre\-activation checkpoint is available for the new variants, those variants are trained from scratch\.

Proxy scales can differ across tasks and training stages\. Diagnostic sweeps report proxy quantiles across imagination depths and the induced effective\-horizon distribution\. When diagnostics determine parameters, report the reference checkpoint, replay sample, candidate set, and selection rule\. Selection based on an auxiliary\-enabled reference run should be distinguished from tuning against final evaluation returns\.

Immediate\-activation and delayed\-activation experiments are reported separately\. Warmup delays selection rather than gradually ramping it\. For Atari with action repeat four, activation at 200,000 frames would require 50,000 agent steps; the Control Suite counter value would imply a different activation point\.

### 7\.5Evaluation and Statistical Reporting

The primary measure is undiscounted evaluation episode return versus environment interaction\. Evaluation episodes do not update the model or policy\. All variants in a comparison use the same evaluation\-policy convention, episode count, evaluation frequency, and time\-limit handling\. These settings are recovered from resolved run configurations rather than inferred from defaults\. We report performance at a common interaction budget\. A predefined average over the final evaluation checkpoints may additionally summarize late\-training performance, but it is labeled as a within\-run temporal average\. It is not an independent\-seed estimate\. If collection or logging extends beyond the nominal budget, budget\-matched summaries consistently exclude those later observations\. We evaluate each method included in the main performance comparison using 10 independent training seeds\. At each evaluation checkpoint, episode returns are first averaged within each training run\. We then report the mean evaluation return across the 10 runs\. Training seeds, rather than evaluation episodes or successive checkpoints, are the independent units used to quantify run\-to\-run variability\. The learning curves summarize the mean evaluation return across 10 independent training runs\. Shaded regions are intended to represent one standard deviation across runs, rather than confidence intervals for the mean\. The seed\-0 threshold sweep is treated separately as an exploratory sensitivity analysis\.

### 7\.6Mechanism Diagnostics

We inspect effective imagination length during training to characterize the behavior of the boundary mechanism\. For hard boundaries, this metric counts retained transition\-loss entries\. For soft boundaries, it sums survival weights\. The illustrations in Section 7\.5 use one training run per variant and task and do not estimate variation across training seeds\. Additional logged diagnostics include proxy quantiles, increment\-clipping fractions, and trial cutoff\-length statistics\. Trial length standard deviations describe variation across imagined trajectories within diagnostic batches; they are distinct from evaluation\-return variability across training seeds\. The implementation generates the full imagined trajectory before applying the gates\. Consequently, reduced effective imagination length is not interpreted as reduced trajectory\-generation cost\. A matched wall\-clock study, paired drift\-only and corrected\-divergence diagnostics, and controls for survival\-dependent optimization scale are not included in the current results\.

## 8Results and Analysis

We organize the evaluation around four questions: whether density\-guided boundaries improve performance relative to vanilla DreamerV3, whether the gains extend beyond a fixed bootstrap boundary, how enabling the auxiliary mechanism differs from using its signal for gating, and how performance depends on the density threshold\. The main learning curves summarize 10 independent training seeds\. Shaded regions show one standard deviation across seeds, and TensorBoard\-style exponential smoothing with weight 0\.7 is used for visualization\. These bands describe run\-to\-run dispersion, not confidence intervals for the mean\.

### 8\.1Performance Across Environments

Figure[1](https://arxiv.org/html/2610.41022#S8.F1)compares vanilla DreamerV3 with density\-guided hard and soft boundaries on four continuous\-control tasks and Atari Pong\. The curves indicate task\-dependent benefits: some environments show faster learning with similar eventual returns, while others show a larger separation near the end of training\. Neither boundary rule dominates across all environments\.

#### Walker Walk\.

Both boundary variants reach high returns earlier than vanilla\. Hard boundaries show the clearest advantage during the early\-to\-middle portion of training, while soft boundaries subsequently reach a similar performance level\. The curves approach comparable returns by the end of the interaction budget\. The principal benefit on this task is therefore earlier attainment of high performance rather than a substantial increase in final return\.

#### Walker Run\.

Hard boundaries yield the highest mean return through most of the plotted training period\. Soft boundaries improve more gradually and finish above vanilla, but below hard boundaries\. Unlike Walker Walk, a visible gap remains at the final budget\. The relative trajectories suggest that the discrete boundary rule is more effective than the selected soft\-gate configuration on this task\.

#### Cheetah Run\.

Both density\-guided variants achieve higher late\-training mean returns than vanilla, with soft boundaries finishing highest\. Hard and soft exhibit different intermediate trajectories, including a temporary decline in the hard variant around the middle of training\. Their relative ordering is therefore not constant over the full budget\. The final separation motivates reporting both final\-budget performance and a summary of learning efficiency, rather than selecting a single favorable intermediate checkpoint\.

#### Finger Turn Hard\.

The hard\-boundary variant achieves the strongest late\-training performance\. Its mean learning curve also changes less sharply during the later portion of training than the other displayed curves\. Soft boundaries and vanilla are broadly comparable near the final budget, with crossings during training\. The smoother hard mean curve should not be interpreted as a formal reduction in seed variance or policy\-gradient variance; those quantities require separate measurements\.

#### Pong\.

All three methods reach high positive returns\. Hard boundaries make earlier progress over part of the initial learning period, while soft boundaries improve rapidly during a subsequent phase\. Vanilla ultimately reaches a slightly higher mean return near the end of training\. Thus the Pong results indicate differences in learning dynamics rather than a consistent final\-performance advantage for density\-guided boundaries\.

Taken together, these comparisons support evaluating boundary selection as a task\-dependent modification to imagined learning\. They do not establish that either hard or soft gating is uniformly preferable\. In particular, the stronger late\-training results on Cheetah Run and Finger Turn Hard coexist with near\-convergent results on Walker Walk and a slightly stronger vanilla endpoint on Pong\.

![Refer to caption](https://arxiv.org/html/2610.41022v1/Figures/walker_walk_hard_soft_vanilla.png)

![Refer to caption](https://arxiv.org/html/2610.41022v1/Figures/cheetah_run_comparison_paper_hard_soft_vanilla_tau900.png)

![Refer to caption](https://arxiv.org/html/2610.41022v1/Figures/walker_run_hard_soft_vanilla.png)

![Refer to caption](https://arxiv.org/html/2610.41022v1/Figures/finger_turn_hard_hard_soft_vanilla.png)

![Refer to caption](https://arxiv.org/html/2610.41022v1/Figures/pong_hard_soft_vanilla.png)

Figure 1:Evaluation returns for vanilla DreamerV3 and density\-guided hard and soft bootstrap boundaries across five tasks\. Curves summarize 10 independent training seeds; shaded regions indicate one standard deviation across seeds\. TensorBoard\-style exponential smoothing with weight 0\.7 is used for visualization\.To complement the learning curves, Table[2](https://arxiv.org/html/2610.41022#S8.T2)summarizes evaluation returns at early, middle, and late stages of training\. These comparisons distinguish improvements during learning from differences near the end of the training budget\. The stage definitions and extraction procedure are provided in the table caption\.

Table 2:Evaluation returns at early, middle, and late training stages\. Stage targets are 100k, 250k, and 500k environment steps for DeepMind Control tasks, and 80k, 200k, and 400k for Pong\. Values are mean±\\pmstd\.The stage\-wise comparisons indicate that the benefits depend on both the task and the training stage\. On Walker Walk, the hard boundary achieves a higher early return, while the three methods reach similar returns by late training\. On Walker Run, the hard variant has the highest reported mean at all three stages\. On Cheetah Run, both boundary variants exceed vanilla at each reported stage, with the soft variant achieving the highest late\-stage mean\. On Finger Turn Hard, the hard variant initially trails vanilla but achieves higher middle\- and late\-stage returns\. Pong shows a different pattern: vanilla has the highest early and late means, while the hard variant leads at the middle stage\. These descriptive comparisons suggest task\-dependent improvements rather than a uniform advantage across all environments\.

### 8\.2Comparison with Fixed Bootstrap Boundaries

We compare density\-guided hard boundaries with fixed bootstrap boundaries of six steps on Cheetah Run and five steps on Walker Run to examine whether simple horizon shortening can account for the observed performance differences\. On Cheetah Run, the fixed\-boundary control retains the auxiliary displacement field, score estimator, and density computations, while replacing density\-dependent boundary selection with a predetermined six\-step boundary\. This provides a comparison of fixed and density\-selected boundaries with auxiliary components enabled in both variants\. On Walker Run, the fixed\-boundary control disables the auxiliary components and therefore compares simple horizon shortening against the complete proposed method\. The Walker Run comparison does not isolate adaptive boundary selection alone; the auxiliary\-only ablation in the following subsection provides a complementary assessment of the effect of enabling density\-guided boundaries\.

![Refer to caption](https://arxiv.org/html/2610.41022v1/Figures/cheetah_run_comparison_paper_hard_vanilla_fixed_tau900.png)

![Refer to caption](https://arxiv.org/html/2610.41022v1/Figures/walker_run_hard_vanilla_fixed.png)

Figure 2:Comparison of vanilla DreamerV3, fixed bootstrap boundaries, and density\-guided hard boundaries on Cheetah Run \(left\) and Walker Run \(right\)\. Fixed boundaries use six and five steps, respectively\. The Cheetah fixed control retains the auxiliary components, whereas the Walker fixed control disables them\.
### 8\.3Auxiliary Mechanism and Boundary Selection

The auxiliary\-enabled, cutoff\-off control separates the introduction of displacement and score learning from using their diagnostic to modify imagined returns\. Comparing this control with vanilla measures the effect of the combined auxiliary machinery\. Comparing this control with hard boundaries measures the incremental effect of activating density\-guided bootstrap boundaries and their associated survival weights, provided the remaining settings match\. The supplied Cheetah Run curves show that the auxiliary\-enabled cutoff\-off agent itself improves over vanilla\. The hard\-boundary variant finishes above this control, although their ordering changes during training\. On Walker Run, the hard variant leads the cutoff\-off control during a substantial portion of training, while their final mean returns are similar\. This pattern suggests that gating may improve intermediate training performance even when the additional endpoint gain is small\. These results also limit the interpretation of a comparison against vanilla alone\. Some observed improvement is already present when the auxiliary networks are enabled without boundary selection\. Because the auxiliary losses use detached features, this observation does not directly establish that they improve the learned representation\. Shared optimization details and stochastic computation can also distinguish the auxiliary\-enabled control from vanilla\.

This ablation tests the combined auxiliary mechanism and the additional use of its signal\. It does not independently establish that the score correction, rather than the displacement field, is responsible for improved performance\. A drift\-only boundary comparison would be needed to isolate that contribution\.

![Refer to caption](https://arxiv.org/html/2610.41022v1/Figures/cheetah_run_comparison_paper_no_hard_no_soft_hard_vanilla_tau900.png)

![Refer to caption](https://arxiv.org/html/2610.41022v1/Figures/walker_run_no_hard_no_soft_hard_vanilla.png)

Figure 3:Comparison of vanilla DreamerV3, the auxiliary\-enabled cutoff\-off control, and density\-guided hard boundaries on Cheetah Run \(left\) and Walker Run \(right\)\. The cutoff\-off control retains auxiliary displacement and score learning without using the density signal to modify imagined returns\. Curves use TensorBoard\-style smoothing with coefficient 0\.7\.
### 8\.4Sensitivity to the Density Threshold

We examine sensitivity to the density threshold on Cheetah Run usingτ∈\{−500,−700,−900\}\\tau\\in\\\{\-500,\-700,\-900\\\}for both hard and soft boundaries\. The study fixes the soft temperature atT=100T=100, the increment cap atC=200C=200, the minimum retained transition count atm=2m=2, and boundary activation at 200k logged environment steps\. Each threshold configuration uses seed 0\. This study is therefore an exploratory sensitivity analysis, separate from the 10\-seed main comparisons\. Figure[4](https://arxiv.org/html/2610.41022#S8.F4)shows that changing the threshold affects learning after boundary activation\. The threshold variants follow similar trajectories during warmup, when density\-based selection is disabled, and separate during subsequent training\. In both hard and soft modes,τ=−900\\tau=\-900achieves the highest return near the end of the interaction budget, followed byτ=−700\\tau=\-700andτ=−500\\tau=\-500\. The ordering is not maintained throughout training, however, and the curves exhibit intermediate crossings\. All three threshold settings finish above the displayed vanilla reference\. For a fixed proxy sequence, decreasing the threshold makes the boundary more permissive\. In hard mode, fewer states fail the threshold test; in soft mode, the continuation gates increase\. Consequently, a more negative threshold produces no smaller survival mass when evaluated on the same trajectory with the remaining settings fixed\. This ordering need not hold for effective horizons across independently trained agents, because their policies, latent representations, and proxy distributions can diverge\. The late\-training ordering in this experiment is consistent with a benefit from a more permissive boundary setting within the tested range\. It does not establish that longer effective horizons caused the improvement or thatτ=−900\\tau=\-900is generally optimal\. Although the observed late\-training advantage over the vanilla reference is present at all three tested thresholds, repeated\-seed sweeps would be required to assess the reliability of the differences between threshold settings\.

![Refer to caption](https://arxiv.org/html/2610.41022v1/Figures/Cheetah_run_tau_sensitivity_curves_hard.png)

![Refer to caption](https://arxiv.org/html/2610.41022v1/Figures/Cheetah_run_tau_sensitivity_curves_soft.png)

Figure 4:Density\-threshold sensitivity on Cheetah Run for hard boundaries \(left\) and soft boundaries \(right\), usingτ∈\{−500,−700,−900\}\\tau\\in\\\{\-500,\-700,\-900\\\}\. Each threshold configuration uses seed 0, with boundary activation at 200k environment steps, increment capC=200C=200, and minimum retained depthm=2m=2\. Soft boundaries use temperatureT=100T=100\. The dashed curve shows the vanilla reference\.
### 8\.5Behavior of the Density\-Guided Boundaries

Figure[5](https://arxiv.org/html/2610.41022#S8.F5)illustrates effective imagination lengths on Cheetah Run and Walker Run\. Both variants retain the full effective horizon of 14 transitions during warmup\. Following boundary activation at 200k environment steps, the effective lengths decrease and subsequently vary over training\. On Cheetah Run, the hard variant decreases to approximately5\.55\.5effective transitions before increasing to approximately6\.76\.7near the end of training\. The soft variant follows a similar pattern at lower effective lengths\. On Walker Run, the hard variant decreases to approximately4\.34\.3transitions before recovering to approximately5\.25\.2, while the soft variant retains substantially more effective transition weight, reaching approximately9\.19\.1near the end of training\. Thus, the relative effective lengths of hard and soft boundaries differ across these runs\. For hard boundaries, effective length counts retained transitions; for soft boundaries, it sums survival weights and does not represent an integer stopping position\. The curves come from separately trained agents and therefore reflect both the boundary rule and the evolving distributions of imagined trajectories\. They do not isolate the effect of applying hard versus soft gating to identical trajectories\. The fixed five\- and six\-transition references provide context for the controls in Section 7\.2, but do not imply exact horizon matching throughout training\.

![Refer to caption](https://arxiv.org/html/2610.41022v1/Figures/cheetah_boundary_effective_length.png)

![Refer to caption](https://arxiv.org/html/2610.41022v1/Figures/walker_boundary_effective_length.png)

Figure 5:Effective imagination length on Cheetah Run \(left\) and Walker Run \(right\), with one training run per variant\. Vertical dotted lines mark boundary activation at 200k environment steps\. Horizontal dashed lines indicate six\- and five\-transition references, respectively; these are not measured fixed\-control curves\. Hard effective length counts retained transitions, whereas soft effective length sums survival weights\.Density\-guided variants increased observed end\-to\-end runtime by approximately 2–7% relative to vanilla DreamerV3\. These measurements include evaluation and logging and do not isolate the computational cost of the density mechanism\.

## 9Discussion and Limitations

### 9\.1Adaptive Bootstrapping as a Model\-Usage Decision

The central contribution of this work is to use a score\-informed density diagnostic to decide how far an imagined return should expand before relying on the critic\. This makes the use of model predictions trajectory\-dependent while retaining stochastic latent dynamics\. A boundary changes the source of the remaining value estimate: future model\-generated rewards are replaced or attenuated in favor of a critic bootstrap\. It therefore regulates model usage rather than eliminating estimation error\.

The learning curves in Section[8](https://arxiv.org/html/2610.41022#S8)suggest that the benefits of this decision depend on the task\. On Walker Walk, the main difference is earlier attainment of high returns, with similar final performance\. Cheetah Run and the hard\-boundary variant on Finger Turn Hard show larger late\-training improvements over vanilla\. Pong exhibits differences in learning speed but no final\-return advantage for the proposed variants\. These patterns support distinguishing learning efficiency from final performance instead of treating either as a complete description of the method\.

Hard and soft boundaries also make different choices about retaining future information\. Hard gating removes downstream expansion after the first failed gate, whereas soft gating attenuates it cumulatively\. The multiplicative soft survival weights can substantially reduce later contributions even when no individual gate is near zero\. Consequently, a common threshold does not imply a common effective horizon or an equivalent intervention in learning\. The different task\-wise rankings of hard and soft variants motivate examining their induced survival profiles alongside return curves\.

### 9\.2Interpretation of the Density Proxy

The quantity used for selection is an anchored accumulation of estimated divergence, not an exact latent likelihood\. The auxiliary displacement field is fitted to posterior feature differences, while imagined trajectories are generated by a separate stochastic transition model\. The score estimator describes a noise\-smoothed replay\-feature distribution, which need not coincide with the distribution visited during imagination\. These differences prevent a direct identification of the implemented diagnostic with the density transported by a continuous\-time flow\.

Moreover, setting the initial proxy to zero means that the method tracks changes relative to each trajectory’s starting point without estimating its absolute density\. A trajectory starting in a poorly supported region can therefore initially receive the same proxy value as one starting in a well\-supported region\. Low density, density decrease, and prediction error are also distinct concepts\. The observed usefulness of a boundary rule does not by itself establish that the underlying proxy is calibrated model uncertainty\.

The score correction has a clear computational route to influencing learning: it changes the auxiliary field, its divergence, and potentially the gates\. However, a large correction to the divergence is evidence of influence rather than evidence of accurate uncertainty estimation\. Relating this signal to model prediction errors, critic errors, or return\-estimation error remains an important direction for further analysis\.

### 9\.3Attribution of Performance Improvements

Comparisons against vanilla measure the effect of the complete modification, including auxiliary computation and boundary selection\. The auxiliary\-enabled cutoff\-off comparisons show why this distinction matters: part of the observed performance difference can be present without activating a boundary\. Since the auxiliary losses use detached features, such differences should not automatically be interpreted as improved representation learning\. Shared optimization details, gradient clipping, and changes in stochastic computation can also distinguish the training procedures\.

Fixed\-boundary controls address whether simple horizon shortening can explain the results\. The evaluated fixed setting does not reproduce the hard variant’s performance on the supplied Cheetah Run and Walker Run comparisons\. Nevertheless, this does not establish superiority over every fixed horizon or scheduled horizon\. Interpretation depends on matching activation times, auxiliary settings, and effective return depths\. A fixed boundary matched only to the average adaptive horizon may still differ substantially in its distribution across trajectories and its evolution over training\.

The loss\-weighting convention introduces a further distinction\. Because actor and critic losses are averaged over the original batch\-time tensor rather than normalized by surviving mass, shorter effective horizons can reduce update magnitudes\. Hard and soft variants also differ in their treatment of targets used for return normalization\. Thus improvements may reflect a combination of target construction, sample selection, and optimization\-scale effects\. Normalization controls and drift\-only boundary ablations would help separate these contributions; the present comparisons should not be interpreted as isolating all of them\.

### 9\.4Calibration and Nonstationarity

The threshold, soft temperature, divergence cap, and activation time jointly determine boundary behavior\. Their effects depend on the learned representation and auxiliary approximation\. Proxy distributions can change as the replay buffer, world model, and policy evolve, so a threshold selected at one checkpoint need not maintain the same effective horizon later in training\. Numerical values should therefore not be assumed to transfer unchanged between tasks or architectures\.

Clipping limits the influence of extreme divergence estimates but changes the accumulated signal\. When clipping is frequent, distinct estimates can produce identical increments, reducing the information available for selection\. A low clipping rate avoids this particular saturation effect but does not establish that the remaining divergence estimates are accurate\. Monitoring both clipping frequency and proxy distributions is therefore useful for interpreting a threshold sweep\.

Warmup delays the use of the diagnostic while auxiliary learning proceeds\. It does not guarantee that the score or displacement model has become accurate at activation, and abrupt activation can change targets and loss weights substantially\. A gradual activation schedule or calibrated boundary criterion could be explored, but these are extensions rather than established features of the present method\. The seed\-0 threshold study provides an initial sensitivity assessment; it does not establish multi\-seed robustness across the full hyperparameter space\.

### 9\.5Computational and Implementation Scope

The implementation generates the full imagined trajectory before constructing the gates\. A reduction in effective return depth therefore does not imply a reduction in trajectory\-generation cost\. Auxiliary training and spatial divergence estimation add computation, and any practical benefit must be assessed using measured training time or throughput as well as environment interactions\. Comparisons of wall\-clock cost require matched hardware and process concurrency\.

Stochastic trace estimation avoids explicitly forming the full Jacobian, but it introduces another source of variability in the diagnostic\. The required spatial derivatives include those of the score network\. Improving score prediction alone does not necessarily improve these derivatives, and detaching the score input during divergence computation would remove the score’s contribution entirely\. Efficient and accurate derivative estimation is therefore a practical consideration for scaling the approach\.

The formulation is compatible with imagined actor–critic learning beyond a single architecture, but the empirical evaluation here is confined to one DreamerV3 implementation\. Results across continuous\-control and visual discrete\-action tasks demonstrate breadth within that implementation, not verified portability across world\-model families\. Applying the mechanism to another architecture would require checking the latent representation, auxiliary training distribution, return estimator, and gradient paths rather than transferring thresholds alone\.

### 9\.6Statistical and Theoretical Scope

The main comparisons use 10 independent training seeds, with standard\-deviation bands showing variation across runs\. These bands are descriptive and do not directly quantify uncertainty in the difference between methods\. Likewise, a smoother mean learning curve does not establish reduced estimator variance\. Numerical performance comparisons should use unsmoothed per\-seed summaries, with uncertainty procedures appropriate to the statistic being reported\.

Variance across independently trained policies is distinct from the conditional variance of an imagined return or policy\-gradient estimator at a fixed checkpoint\. A theoretical guarantee about one quantity cannot be inferred from measurements of another\. The relationship between the theory and experiments must therefore be stated in terms of the exact estimator, sources of randomness, and assumptions under consideration\.

These limitations leave a focused empirical question: whether the score\-informed diagnostic provides useful boundary decisions beyond simpler controls\. The current results motivate that question and show that density\-guided bootstrapping can change learning behavior while preserving stochastic imagination\. Establishing when those changes improve estimation accuracy, and how reliably they transfer across tasks and implementations, requires the component controls and assumption\-specific analysis described throughout the paper\.

## 10Conclusion

We presented density\-guided bootstrap boundaries for latent world\-model reinforcement learning, establishing a principled mechanism to govern model usage during policy learning\. By evaluating spatial divergence along imagined latent paths via an auxiliary displacement field and a noise\-conditioned score estimator, our framework constructs an anchored feature\-space expansion diagnostic\. Hard and soft continuation gates dynamically determine where return expansion transitions into value bootstrapping, preserving discounted boundary critic estimates and applying consistent survival weighting to actor and critic objectives\.

Theoretically, we established that the gated Bellman operator preserves strict contraction properties, characterized its local contraction dynamics under soft and hard gates, and proved that truncation monotonically bounds conditional target variance at the cost of absorbing critic approximation bias\. Formally distinguishing this proxy from an exact path likelihood provides a clear boundary for theoretical claims and prevents misinterpreting spatial divergence as calibrated model uncertainty\.

Empirically, our Dreamer V3 instantiation across DeepMind Control Suite and Atari Pong demonstrates that adaptive return horizons improve sample efficiency and final performance compared to vanilla Dreamer V3 and fixed\-horizon baselines\. The performance trade\-offs between discrete hard cutoffs and cumulative soft attenuation highlight task\-dependent dynamics in return expansion\. Controlled ablations confirm that gains stem from trajectory\-dependent gate selection rather than horizon shortening or auxiliary loss scales alone\.

Future research will explore tighter couplings between feature\-space divergence and local model prediction error, introduce adaptive thresholding mechanisms across evolving replay distributions, and extend the gating mechanism to broader model\-based architectures and offline reinforcement learning settings\.

###### acknowledgments\-disclosure\-of\-funding\.

The authors conceptualized, formulated, and provided the core mathematical ideas and directions for this work\. Google Gemini was used during manuscript preparation to assist in drafting initial formal statements, theorems, and proofs in Section 6 and Appendices A–D\. All derivations, theoretical formulations, empirical evaluations, and text were thoroughly reviewed, independently re\-derived, verified, and finalized by the authors, who take full responsibility for the accuracy, claims, and integrity of all contents of this paper\. PL is partially supported by NIH OT2 Award, 1OT2OD042825\-01\.

## References

- J\. Buckman, D\. Hafner, G\. Tucker, and E\. BrevdoSample\-efficient reinforcement learning with stochastic ensemble value expansion\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.31,pp\. 8224–8234\.Cited by:[§1](https://arxiv.org/html/2610.41022#S1.p2.1),[§2](https://arxiv.org/html/2610.41022#S2.SS0.SSS0.Px2.p1.1)\.
- Feinberget al\.\(2018\)V\. Feinberg, A\. Wan, I\. Wang, B\. Mazoure, S\. Levine, and J\. E\. GonzalezModel\-based value expansion for efficient deep reinforcement learning\.InInternational Conference on Machine Learning \(ICML\),pp\. 1501–1510\.Cited by:[§2](https://arxiv.org/html/2610.41022#S2.SS0.SSS0.Px2.p1.1)\.
- Grathwohlet al\.\(2019\)W\. Grathwohl, R\. T\. Q\. Chen, J\. Bettencourt, I\. Sutskever, and D\. DuvenaudFFJORD: free\-form continuous dynamics for scalable reversible generative models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2610.41022#S2.SS0.SSS0.Px6.p1.1)\.
- Hafneret al\.\(2020\)D\. Hafner, T\. Lillicrap, J\. Ba, and M\. NorouziDream to explore: model\-based reinforcement learning with latent world models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2610.41022#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2610.41022#S3.SS1.p2.1)\.
- Hafneret al\.\(2019\)D\. Hafner, T\. Lillicrap, I\. Fischer, R\. Villegas, D\. Ha, H\. Lee, and J\. DavidsonLearning latent dynamics for planning from pixels\.InInternational Conference on Machine Learning \(ICML\),pp\. 2555–2565\.Cited by:[§2](https://arxiv.org/html/2610.41022#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2610.41022#S3.SS1.p2.1)\.
- Hafneret al\.\(2023\)D\. Hafner, J\. Pasukonis, J\. Ba, and T\. LillicrapMastering diverse domains through world models\.arXiv preprint arXiv:2301\.04104\.Cited by:[§1](https://arxiv.org/html/2610.41022#S1.p1.1),[§2](https://arxiv.org/html/2610.41022#S2.SS0.SSS0.Px1.p1.1),[§5\.7](https://arxiv.org/html/2610.41022#S5.SS7.p2.1)\.
- Janneret al\.\(2019\)M\. Janner, J\. Fu, M\. Zhang, and S\. LevineWhen to trust your model: model\-based policy optimization\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.32,pp\. 12519–12530\.Cited by:[§1](https://arxiv.org/html/2610.41022#S1.p2.1),[§2](https://arxiv.org/html/2610.41022#S2.SS0.SSS0.Px2.p1.1)\.
- Kakade and Langford \(2002\)S\. Kakade and J\. LangfordApproximately optimal approximate reinforcement learning\.InProceedings of the 19th International Conference on Machine Learning \(ICML\),pp\. 267–274\.Cited by:[Appendix D](https://arxiv.org/html/2610.41022#A4.p1.1)\.
- Kidambiet al\.\(2020\)R\. Kidambi, A\. Rajeswaran, P\. Netrapalli, and T\. JoachimsMOREL: model\-based offline reinforcement learning\.Advances in Neural Information Processing Systems33,pp\. 21810–21823\.Cited by:[§2](https://arxiv.org/html/2610.41022#S2.SS0.SSS0.Px3.p1.1)\.
- Luet al\.\(2026\)H\. Lu, L\. Yao, C\. He, H\. Han, F\. Liu, W\. Liao, T\. He, and P\. PengRISE: adaptive imagination for world action models\.arXiv preprint arXiv:2608\.20430\.Cited by:[§2](https://arxiv.org/html/2610.41022#S2.SS0.SSS0.Px4.p1.1)\.
- Songet al\.\(2021\)Y\. Song, J\. Sohl\-Dickstein, D\. P\. Kingma, A\. Kumar, S\. Ermon, and B\. PooleScore\-based generative modeling through stochastic differential equations\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2610.41022#S2.SS0.SSS0.Px5.p1.1),[§3\.3](https://arxiv.org/html/2610.41022#S3.SS3.p2.3),[Proposition 1](https://arxiv.org/html/2610.41022#Thmprop1)\.
- Vincent \(2011\)P\. VincentA connection between score matching and denoising autoencoders\.Neural Computation23\(7\),pp\. 1661–1674\.Cited by:[§2](https://arxiv.org/html/2610.41022#S2.SS0.SSS0.Px5.p1.1),[§3\.3](https://arxiv.org/html/2610.41022#S3.SS3.p2.3),[Proposition 1](https://arxiv.org/html/2610.41022#Thmprop1)\.
- Yuet al\.\(2020\)T\. Yu, G\. Thomas, L\. Yu, S\. Ermon, J\. Y\. Zou, S\. Levine, C\. Finn, and T\. MaMOPO: model\-based offline policy optimization\.Advances in Neural Information Processing Systems33,pp\. 14129–14142\.Cited by:[§2](https://arxiv.org/html/2610.41022#S2.SS0.SSS0.Px3.p1.1)\.

## Appendix AOverview

This appendix provides rigorous mathematical proofs for the theoretical properties of density\-guided bootstrap boundaries in latent world\-model reinforcement learning\. We formalize:

1. 1\.Convergence of the Gated Evaluation Operator \(Theorem[3](https://arxiv.org/html/2610.41022#Thmtheo3)\):Proof that the gatedλ\\lambda\-return Bellman operator is a strictγ\\gamma\-contraction mapping\.
2. 2\.Conditional Return Variance Monotonicity \(Theorem[4](https://arxiv.org/html/2610.41022#Thmtheo4)\):Proof that truncating return expansion monotonically reduces conditional target variance\.
3. 3\.Score\-Divergence Dependent Contraction Rate \(Lemma[1](https://arxiv.org/html/2610.41022#Thmtheorem1)\):Derivation of the local contraction rate as a function of the accumulated divergence proxy\.
4. 4\.Discontinuity under Hard Bootstrapping \(Corollary[2](https://arxiv.org/html/2610.41022#Thmtheorem2)\):Characterization of the discrete phase transition in operator contraction\.
5. 5\.Limits of Monotonic Policy Improvement \(Section[D](https://arxiv.org/html/2610.41022#A4)\):Formal analysis clarifying why monotonic improvement requires explicit uncertainty penalties beyond gating\.
6. 6\.DreamerV3 Implementation Details

—

### Proof of Theorem 1 \(Bias\-Variance Trade\-off in Gated Returns\)

###### Proof\.

Letxt∈𝒳x\_\{t\}\\in\\mathcal\{X\}denote an imagined latent state at steptt\. Recall the recursive definition of the gated return targetGtgG\_\{t\}^\{g\}:

Gtg=r^t\+1\+dt\+1​\[\(1−λ​gt\+1\)​Vψ​\(xt\+1\)\+λ​gt\+1​Gt\+1g\],G\_\{t\}^\{g\}=\\hat\{r\}\_\{t\+1\}\+d\_\{t\+1\}\\left\[\(1\-\\lambda g\_\{t\+1\}\)V\_\{\\psi\}\(x\_\{t\+1\}\)\+\\lambda g\_\{t\+1\}G\_\{t\+1\}^\{g\}\\right\],\(34\)wheredt\+1=γ​c^t\+1∈\[0,γ\]d\_\{t\+1\}=\\gamma\\hat\{c\}\_\{t\+1\}\\in\[0,\\gamma\]is the predicted continuation discount,λ∈\[0,1\]\\lambda\\in\[0,1\]is the decay factor, andgt\+1∈\[0,1\]g\_\{t\+1\}\\in\[0,1\]is the continuation gate at stept\+1t\+1\.

LetVπ​\(xt\)V^\{\\pi\}\(x\_\{t\}\)denote the true value function under policyπ\\pi, satisfying the standard Bellman expectation equation:

Vπ​\(xt\)=𝔼π,pϕ​\[r^t\+1\+dt\+1​Vπ​\(xt\+1\)\|xt\]\.V^\{\\pi\}\(x\_\{t\}\)=\\mathbb\{E\}\_\{\\pi,p\_\{\\phi\}\}\\left\[\\hat\{r\}\_\{t\+1\}\+d\_\{t\+1\}V^\{\\pi\}\(x\_\{t\+1\}\)\\;\\middle\|\\;x\_\{t\}\\right\]\.\(35\)Using the partition identity1=\(1−λ​gt\+1\)\+λ​gt\+11=\(1\-\\lambda g\_\{t\+1\}\)\+\\lambda g\_\{t\+1\}, we expandVπ​\(xt\)V^\{\\pi\}\(x\_\{t\}\)into gated and non\-gated recursive components:

Vπ​\(xt\)=𝔼⁡\[r^t\+1\+dt\+1​\(1−λ​gt\+1\)​Vπ​\(xt\+1\)\+dt\+1​λ​gt\+1​Vπ​\(xt\+1\)\|xt\]\.V^\{\\pi\}\(x\_\{t\}\)=\\mathbb\{E\}\\left\[\\hat\{r\}\_\{t\+1\}\+d\_\{t\+1\}\(1\-\\lambda g\_\{t\+1\}\)V^\{\\pi\}\(x\_\{t\+1\}\)\+d\_\{t\+1\}\\lambda g\_\{t\+1\}V^\{\\pi\}\(x\_\{t\+1\}\)\\;\\middle\|\\;x\_\{t\}\\right\]\.\(36\)
Define the approximation bias of the critic at statexxase⁡\(x\)≜Vψ​\(x\)−Vπ​\(x\)e\(x\)\\triangleq V\_\{\\psi\}\(x\)\-V^\{\\pi\}\(x\), which impliesVψ​\(x\)=Vπ​\(x\)\+e⁡\(x\)V\_\{\\psi\}\(x\)=V^\{\\pi\}\(x\)\+e\(x\)\. Substituting this relation into the expected gated return target yields:

𝔼⁡\[Gtg\|xt\]\\displaystyle\\mathbb\{E\}\\left\[G\_\{t\}^\{g\}\\;\\middle\|\\;x\_\{t\}\\right\]=𝔼⁡\[r^t\+1\+dt\+1​\(1−λ​gt\+1\)​Vψ​\(xt\+1\)\+dt\+1​λ​gt\+1​Gt\+1g\|xt\]\\displaystyle=\\mathbb\{E\}\\left\[\\hat\{r\}\_\{t\+1\}\+d\_\{t\+1\}\(1\-\\lambda g\_\{t\+1\}\)V\_\{\\psi\}\(x\_\{t\+1\}\)\+d\_\{t\+1\}\\lambda g\_\{t\+1\}G\_\{t\+1\}^\{g\}\\;\\middle\|\\;x\_\{t\}\\right\]\(37\)=𝔼⁡\[r^t\+1\+dt\+1​\(1−λ​gt\+1\)​\(Vπ​\(xt\+1\)\+e⁡\(xt\+1\)\)\+dt\+1​λ​gt\+1​Gt\+1g\|xt\]\.\\displaystyle=\\mathbb\{E\}\\left\[\\hat\{r\}\_\{t\+1\}\+d\_\{t\+1\}\(1\-\\lambda g\_\{t\+1\}\)\\left\(V^\{\\pi\}\(x\_\{t\+1\}\)\+e\(x\_\{t\+1\}\)\\right\)\+d\_\{t\+1\}\\lambda g\_\{t\+1\}G\_\{t\+1\}^\{g\}\\;\\middle\|\\;x\_\{t\}\\right\]\.\(38\)
The conditional target bias is defined asBias​\(Gtg∣xt\)≜𝔼⁡\[Gtg\|xt\]−Vπ​\(xt\)\\text\{Bias\}\(G\_\{t\}^\{g\}\\mid x\_\{t\}\)\\triangleq\\mathbb\{E\}\\left\[G\_\{t\}^\{g\}\\;\\middle\|\\;x\_\{t\}\\right\]\-V^\{\\pi\}\(x\_\{t\}\)\. Subtracting Eq\. \(36\) from Eq\. \(38\):

Bias​\(Gtg∣xt\)\\displaystyle\\text\{Bias\}\(G\_\{t\}^\{g\}\\mid x\_\{t\}\)=𝔼⁡\[r^t\+1\+dt\+1​\(1−λ​gt\+1\)​Vπ​\(xt\+1\)\+dt\+1​\(1−λ​gt\+1\)​e​\(xt\+1\)\+dt\+1​λ​gt\+1​Gt\+1g\|xt\]\\displaystyle=\\mathbb\{E\}\\left\[\\hat\{r\}\_\{t\+1\}\+d\_\{t\+1\}\(1\-\\lambda g\_\{t\+1\}\)V^\{\\pi\}\(x\_\{t\+1\}\)\+d\_\{t\+1\}\(1\-\\lambda g\_\{t\+1\}\)e\(x\_\{t\+1\}\)\+d\_\{t\+1\}\\lambda g\_\{t\+1\}G\_\{t\+1\}^\{g\}\\;\\middle\|\\;x\_\{t\}\\right\]−𝔼⁡\[r^t\+1\+dt\+1​\(1−λ​gt\+1\)​Vπ​\(xt\+1\)\+dt\+1​λ​gt\+1​Vπ​\(xt\+1\)\|xt\]\.\\displaystyle\\quad\-\\mathbb\{E\}\\left\[\\hat\{r\}\_\{t\+1\}\+d\_\{t\+1\}\(1\-\\lambda g\_\{t\+1\}\)V^\{\\pi\}\(x\_\{t\+1\}\)\+d\_\{t\+1\}\\lambda g\_\{t\+1\}V^\{\\pi\}\(x\_\{t\+1\}\)\\;\\middle\|\\;x\_\{t\}\\right\]\.\(39\)
Canceling the common termsr^t\+1\\hat\{r\}\_\{t\+1\}anddt\+1​\(1−λ​gt\+1\)​Vπ​\(xt\+1\)d\_\{t\+1\}\(1\-\\lambda g\_\{t\+1\}\)V^\{\\pi\}\(x\_\{t\+1\}\), we obtain:

Bias​\(Gtg∣xt\)\\displaystyle\\text\{Bias\}\(G\_\{t\}^\{g\}\\mid x\_\{t\}\)=𝔼⁡\[dt\+1​\(1−λ​gt\+1\)​e​\(xt\+1\)\+dt\+1​λ​gt\+1​\(Gt\+1g−Vπ​\(xt\+1\)\)\|xt\]\\displaystyle=\\mathbb\{E\}\\left\[d\_\{t\+1\}\(1\-\\lambda g\_\{t\+1\}\)e\(x\_\{t\+1\}\)\+d\_\{t\+1\}\\lambda g\_\{t\+1\}\\left\(G\_\{t\+1\}^\{g\}\-V^\{\\pi\}\(x\_\{t\+1\}\)\\right\)\\;\\middle\|\\;x\_\{t\}\\right\]\(40\)=𝔼⁡\[dt\+1​\(1−λ​gt\+1\)​e​\(xt\+1\)\|xt\]\+𝔼⁡\[dt\+1​λ​gt\+1​\(Gt\+1g−Vπ​\(xt\+1\)\)\|xt\]\.\\displaystyle=\\mathbb\{E\}\\left\[d\_\{t\+1\}\(1\-\\lambda g\_\{t\+1\}\)e\(x\_\{t\+1\}\)\\;\\middle\|\\;x\_\{t\}\\right\]\+\\mathbb\{E\}\\left\[d\_\{t\+1\}\\lambda g\_\{t\+1\}\\left\(G\_\{t\+1\}^\{g\}\-V^\{\\pi\}\(x\_\{t\+1\}\)\\right\)\\;\\middle\|\\;x\_\{t\}\\right\]\.\(41\)
Applying the Law of Iterated Expectations to the second term:

𝔼⁡\[dt\+1​λ​gt\+1​\(Gt\+1g−Vπ​\(xt\+1\)\)\|xt\]\\displaystyle\\mathbb\{E\}\\left\[d\_\{t\+1\}\\lambda g\_\{t\+1\}\\left\(G\_\{t\+1\}^\{g\}\-V^\{\\pi\}\(x\_\{t\+1\}\)\\right\)\\;\\middle\|\\;x\_\{t\}\\right\]=𝔼xt\+1∼pϕ​\[dt\+1​λ​gt\+1​𝔼​\[Gt\+1g−Vπ​\(xt\+1\)\|xt\+1\]\|xt\]\\displaystyle=\\mathbb\{E\}\_\{x\_\{t\+1\}\\sim p\_\{\\phi\}\}\\left\[d\_\{t\+1\}\\lambda g\_\{t\+1\}\\mathbb\{E\}\\left\[G\_\{t\+1\}^\{g\}\-V^\{\\pi\}\(x\_\{t\+1\}\)\\;\\middle\|\\;x\_\{t\+1\}\\right\]\\;\\middle\|\\;x\_\{t\}\\right\]\(42\)=𝔼⁡\[dt\+1​λ​gt\+1​Bias​\(Gt\+1g∣xt\+1\)\|xt\]\.\\displaystyle=\\mathbb\{E\}\\left\[d\_\{t\+1\}\\lambda g\_\{t\+1\}\\text\{Bias\}\(G\_\{t\+1\}^\{g\}\\mid x\_\{t\+1\}\)\\;\\middle\|\\;x\_\{t\}\\right\]\.\(43\)
Substituting Eq\. \(43\) into Eq\. \(41\) yields the recursive bias identity:

Bias​\(Gtg∣xt\)=𝔼⁡\[dt\+1​\(1−λ​gt\+1\)​e​\(xt\+1\)\|xt\]\+𝔼⁡\[dt\+1​λ​gt\+1​Bias​\(Gt\+1g∣xt\+1\)\|xt\]\.\\text\{Bias\}\(G\_\{t\}^\{g\}\\mid x\_\{t\}\)=\\mathbb\{E\}\\left\[d\_\{t\+1\}\(1\-\\lambda g\_\{t\+1\}\)e\(x\_\{t\+1\}\)\\;\\middle\|\\;x\_\{t\}\\right\]\+\\mathbb\{E\}\\left\[d\_\{t\+1\}\\lambda g\_\{t\+1\}\\text\{Bias\}\(G\_\{t\+1\}^\{g\}\\mid x\_\{t\+1\}\)\\;\\middle\|\\;x\_\{t\}\\right\]\.\(44\)This completes the proof\. ∎

###### Theorem 3\(Convergence of Path\-Dependent Gated Evaluation Operator\)\.

Let𝒮=𝒳×ℝ\\mathcal\{S\}=\\mathcal\{X\}\\times\\mathbb\{R\}be the compact augmented state space,γ∈\[0,1\)\\gamma\\in\[0,1\)the discount factor, andd⁡\(x,a\)=γ​c^​\(x,a\)≤γ<1d\(x,a\)=\\gamma\\hat\{c\}\(x,a\)\\leq\\gamma<1the continuation\-discount factor\. Letg:𝒮→\[0,1\]g:\\mathcal\{S\}\\rightarrow\[0,1\]be an augmented continuation gate\. Define the gatedλ\\lambda\-return Bellman operatorTgλ:ℝ\|𝒮\|→ℝ\|𝒮\|T\_\{g\}^\{\\lambda\}:\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\}\\rightarrow\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\}as:

\(Tgλ​V\)​\(x¯\)=𝔼a∼π,x¯′∼pϕ,v​\[r^​\(x,a\)\+d⁡\(x,a\)​\(\(1−λ​g​\(x¯′\)\)​V​\(x¯′\)\+λ​g​\(x¯′\)​\(Tgλ​V\)​\(x¯′\)\)\]\(T\_\{g\}^\{\\lambda\}V\)\(\\bar\{x\}\)=\\mathbb\{E\}\_\{a\\sim\\pi,\\bar\{x\}^\{\\prime\}\\sim p\_\{\\phi,v\}\}\\left\[\\hat\{r\}\(x,a\)\+d\(x,a\)\\left\(\(1\-\\lambda g\(\\bar\{x\}^\{\\prime\}\)\)V\(\\bar\{x\}^\{\\prime\}\)\+\\lambda g\(\\bar\{x\}^\{\\prime\}\)\(T\_\{g\}^\{\\lambda\}V\)\(\\bar\{x\}^\{\\prime\}\)\\right\)\\right\]\(45\)ThenTgλT\_\{g\}^\{\\lambda\}is a strictγg\\gamma\_\{g\}\-contraction mapping under the supremum norm‖V‖∞=supx¯∈𝒮\|V⁡\(x¯\)\|\\\|V\\\|\_\{\\infty\}=\\sup\_\{\\bar\{x\}\\in\\mathcal\{S\}\}\|V\(\\bar\{x\}\)\|, where the contraction modulus satisfies:

γg=supx,a,x¯′\(d⁡\(x,a\)​\(1−λ​g​\(x¯′\)\)1−d⁡\(x,a\)​λ​g​\(x¯′\)\)≤γ<1\\gamma\_\{g\}=\\sup\_\{x,a,\\bar\{x\}^\{\\prime\}\}\\left\(\\frac\{d\(x,a\)\(1\-\\lambda g\(\\bar\{x\}^\{\\prime\}\)\)\}\{1\-d\(x,a\)\\lambda g\(\\bar\{x\}^\{\\prime\}\)\}\\right\)\\leq\\gamma<1\(46\)Consequently, for any initial value functionV0∈ℝ\|𝒮\|V\_\{0\}\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\}, the sequenceVk\+1=Tgλ​VkV\_\{k\+1\}=T\_\{g\}^\{\\lambda\}V\_\{k\}converges uniquely to a fixed pointVπ,gV^\{\\pi,g\}\.

###### Proof\.

LetV1,V2∈ℝ\|𝒮\|V\_\{1\},V\_\{2\}\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\}be two arbitrary value functions\. For any augmented statex¯=\(x,l\)∈𝒮\\bar\{x\}=\(x,l\)\\in\\mathcal\{S\}:

\|\(Tgλ​V1\)​\(x¯\)−\(Tgλ​V2\)​\(x¯\)\|\\displaystyle\|\(T\_\{g\}^\{\\lambda\}V\_\{1\}\)\(\\bar\{x\}\)\-\(T\_\{g\}^\{\\lambda\}V\_\{2\}\)\(\\bar\{x\}\)\|=\|𝔼a,x¯′\[d\(x,a\)\(1−λg\(x¯′\)\)\(V1\(x¯′\)−V2\(x¯′\)\)\\displaystyle=\\left\|\\mathbb\{E\}\_\{a,\\bar\{x\}^\{\\prime\}\}\\left\[d\(x,a\)\(1\-\\lambda g\(\\bar\{x\}^\{\\prime\}\)\)\(V\_\{1\}\(\\bar\{x\}^\{\\prime\}\)\-V\_\{2\}\(\\bar\{x\}^\{\\prime\}\)\)\\right\.\\right\.\+d\(x,a\)λg\(x¯′\)\(\(TgλV1\)\(x¯′\)−\(TgλV2\)\(x¯′\)\)\]\|\\displaystyle\\quad\\left\.\\left\.\+d\(x,a\)\\lambda g\(\\bar\{x\}^\{\\prime\}\)\(\(T\_\{g\}^\{\\lambda\}V\_\{1\}\)\(\\bar\{x\}^\{\\prime\}\)\-\(T\_\{g\}^\{\\lambda\}V\_\{2\}\)\(\\bar\{x\}^\{\\prime\}\)\)\\right\]\\right\|Applying Jensen’s inequality and moving absolute values inside the expectation:

\|\(Tgλ​V1\)​\(x¯\)−\(Tgλ​V2\)​\(x¯\)\|\\displaystyle\|\(T\_\{g\}^\{\\lambda\}V\_\{1\}\)\(\\bar\{x\}\)\-\(T\_\{g\}^\{\\lambda\}V\_\{2\}\)\(\\bar\{x\}\)\|≤𝔼a,x¯′​\[d⁡\(x,a\)​\(1−λ​g​\(x¯′\)\)​\|V1​\(x¯′\)−V2​\(x¯′\)\|\]\\displaystyle\\leq\\mathbb\{E\}\_\{a,\\bar\{x\}^\{\\prime\}\}\\left\[d\(x,a\)\(1\-\\lambda g\(\\bar\{x\}^\{\\prime\}\)\)\|V\_\{1\}\(\\bar\{x\}^\{\\prime\}\)\-V\_\{2\}\(\\bar\{x\}^\{\\prime\}\)\|\\right\]\+𝔼a,x¯′​\[d⁡\(x,a\)​λ​g​\(x¯′\)​\|\(Tgλ​V1\)​\(x¯′\)−\(Tgλ​V2\)​\(x¯′\)\|\]\\displaystyle\\quad\+\\mathbb\{E\}\_\{a,\\bar\{x\}^\{\\prime\}\}\\left\[d\(x,a\)\\lambda g\(\\bar\{x\}^\{\\prime\}\)\|\(T\_\{g\}^\{\\lambda\}V\_\{1\}\)\(\\bar\{x\}^\{\\prime\}\)\-\(T\_\{g\}^\{\\lambda\}V\_\{2\}\)\(\\bar\{x\}^\{\\prime\}\)\|\\right\]Bounding\|V1​\(x¯′\)−V2​\(x¯′\)\|≤‖V1−V2‖∞\|V\_\{1\}\(\\bar\{x\}^\{\\prime\}\)\-V\_\{2\}\(\\bar\{x\}^\{\\prime\}\)\|\\leq\\\|V\_\{1\}\-V\_\{2\}\\\|\_\{\\infty\}and\|\(Tgλ​V1\)​\(x¯′\)−\(Tgλ​V2\)​\(x¯′\)\|≤‖Tgλ​V1−Tgλ​V2‖∞\|\(T\_\{g\}^\{\\lambda\}V\_\{1\}\)\(\\bar\{x\}^\{\\prime\}\)\-\(T\_\{g\}^\{\\lambda\}V\_\{2\}\)\(\\bar\{x\}^\{\\prime\}\)\|\\leq\\\|T\_\{g\}^\{\\lambda\}V\_\{1\}\-T\_\{g\}^\{\\lambda\}V\_\{2\}\\\|\_\{\\infty\}:

\|\(Tgλ​V1\)​\(x¯\)−\(Tgλ​V2\)​\(x¯\)\|\\displaystyle\|\(T\_\{g\}^\{\\lambda\}V\_\{1\}\)\(\\bar\{x\}\)\-\(T\_\{g\}^\{\\lambda\}V\_\{2\}\)\(\\bar\{x\}\)\|≤d⁡\(x,a\)​\(1−λ​g​\(x¯′\)\)​‖V1−V2‖∞\\displaystyle\\leq d\(x,a\)\(1\-\\lambda g\(\\bar\{x\}^\{\\prime\}\)\)\\\|V\_\{1\}\-V\_\{2\}\\\|\_\{\\infty\}\+d⁡\(x,a\)​λ​g​\(x¯′\)​‖Tgλ​V1−Tgλ​V2‖∞\\displaystyle\\quad\+d\(x,a\)\\lambda g\(\\bar\{x\}^\{\\prime\}\)\\\|T\_\{g\}^\{\\lambda\}V\_\{1\}\-T\_\{g\}^\{\\lambda\}V\_\{2\}\\\|\_\{\\infty\}Taking the supremum overx¯∈𝒮\\bar\{x\}\\in\\mathcal\{S\}on the left\-hand side:

‖Tgλ​V1−Tgλ​V2‖∞\\displaystyle\\\|T\_\{g\}^\{\\lambda\}V\_\{1\}\-T\_\{g\}^\{\\lambda\}V\_\{2\}\\\|\_\{\\infty\}≤d⁡\(x,a\)​\(1−λ​g​\(x¯′\)\)​‖V1−V2‖∞\\displaystyle\\leq d\(x,a\)\(1\-\\lambda g\(\\bar\{x\}^\{\\prime\}\)\)\\\|V\_\{1\}\-V\_\{2\}\\\|\_\{\\infty\}\+d⁡\(x,a\)​λ​g​\(x¯′\)​‖Tgλ​V1−Tgλ​V2‖∞\\displaystyle\\quad\+d\(x,a\)\\lambda g\(\\bar\{x\}^\{\\prime\}\)\\\|T\_\{g\}^\{\\lambda\}V\_\{1\}\-T\_\{g\}^\{\\lambda\}V\_\{2\}\\\|\_\{\\infty\}Rearranging terms to isolate‖Tgλ​V1−Tgλ​V2‖∞\\\|T\_\{g\}^\{\\lambda\}V\_\{1\}\-T\_\{g\}^\{\\lambda\}V\_\{2\}\\\|\_\{\\infty\}:

\(1−d⁡\(x,a\)​λ​g​\(x¯′\)\)​‖Tgλ​V1−Tgλ​V2‖∞≤d⁡\(x,a\)​\(1−λ​g​\(x¯′\)\)​‖V1−V2‖∞\(1\-d\(x,a\)\\lambda g\(\\bar\{x\}^\{\\prime\}\)\)\\\|T\_\{g\}^\{\\lambda\}V\_\{1\}\-T\_\{g\}^\{\\lambda\}V\_\{2\}\\\|\_\{\\infty\}\\leq d\(x,a\)\(1\-\\lambda g\(\\bar\{x\}^\{\\prime\}\)\)\\\|V\_\{1\}\-V\_\{2\}\\\|\_\{\\infty\}Sinced⁡\(x,a\)≤γ<1d\(x,a\)\\leq\\gamma<1,λ∈\[0,1\]\\lambda\\in\[0,1\], andg⁡\(x¯′\)∈\[0,1\]g\(\\bar\{x\}^\{\\prime\}\)\\in\[0,1\], we have1−d⁡\(x,a\)​λ​g​\(x¯′\)≥1−γ\>01\-d\(x,a\)\\lambda g\(\\bar\{x\}^\{\\prime\}\)\\geq 1\-\\gamma\>0\. Dividing both sides yields:

‖Tgλ​V1−Tgλ​V2‖∞≤\(d⁡\(x,a\)​\(1−λ​g​\(x¯′\)\)1−d⁡\(x,a\)​λ​g​\(x¯′\)\)​‖V1−V2‖∞\\\|T\_\{g\}^\{\\lambda\}V\_\{1\}\-T\_\{g\}^\{\\lambda\}V\_\{2\}\\\|\_\{\\infty\}\\leq\\left\(\\frac\{d\(x,a\)\(1\-\\lambda g\(\\bar\{x\}^\{\\prime\}\)\)\}\{1\-d\(x,a\)\\lambda g\(\\bar\{x\}^\{\\prime\}\)\}\\right\)\\\|V\_\{1\}\-V\_\{2\}\\\|\_\{\\infty\}Definef⁡\(d,g\)=d⁡\(1−λ​g\)1−d​λ​gf\(d,g\)=\\frac\{d\(1\-\\lambda g\)\}\{1\-d\\lambda g\}\. Evaluating its partial derivative with respect togg:

∂f∂g=−d​λ​\(1−d​λ​g\)−d⁡\(1−λ​g\)​\(−d​λ\)\(1−d​λ​g\)2=−d​λ​\(1−d\)\(1−d​λ​g\)2≤0\\frac\{\\partial f\}\{\\partial g\}=\\frac\{\-d\\lambda\(1\-d\\lambda g\)\-d\(1\-\\lambda g\)\(\-d\\lambda\)\}\{\(1\-d\\lambda g\)^\{2\}\}=\\frac\{\-d\\lambda\(1\-d\)\}\{\(1\-d\\lambda g\)^\{2\}\}\\leq 0Because∂f∂g≤0\\frac\{\\partial f\}\{\\partial g\}\\leq 0,f⁡\(d,g\)f\(d,g\)achieves its maximum atg=0g=0:

f⁡\(d,g\)≤f⁡\(d,0\)=d≤γ<1f\(d,g\)\\leq f\(d,0\)=d\\leq\\gamma<1Thus,TgλT\_\{g\}^\{\\lambda\}is a strictγg\\gamma\_\{g\}\-contraction mapping withγg≤γ<1\\gamma\_\{g\}\\leq\\gamma<1\. By Banach’s Fixed\-Point Theorem,TgλT\_\{g\}^\{\\lambda\}converges uniquely to a fixed pointVπ,g∈ℝ\|𝒮\|V^\{\\pi,g\}\\in\\mathbb\{R\}^\{\|\\mathcal\{S\}\|\}\. ∎

—

## Appendix BConditional Return Variance Monotonicity

###### Theorem 4\(Conditional Return Variance Monotonicity\)\.

Letxt∈𝒳x\_\{t\}\\in\\mathcal\{X\}be an imagined state at steptt\. Consider the recursive gated return targetGtgG\_\{t\}^\{g\}defined by:

Gtg=r^t\+1\+dt\+1​\(\(1−λ​gt\+1\)​Vψ​\(xt\+1\)\+λ​gt\+1​Gt\+1g\)G\_\{t\}^\{g\}=\\hat\{r\}\_\{t\+1\}\+d\_\{t\+1\}\\left\(\(1\-\\lambda g\_\{t\+1\}\)V\_\{\\psi\}\(x\_\{t\+1\}\)\+\\lambda g\_\{t\+1\}G\_\{t\+1\}^\{g\}\\right\)\(47\)Letgt\+k\(1\)g\_\{t\+k\}^\{\(1\)\}andgt\+k\(2\)g\_\{t\+k\}^\{\(2\)\}be two deterministic gate sequences such that0≤gt\+k\(1\)≤gt\+k\(2\)≤10\\leq g\_\{t\+k\}^\{\(1\)\}\\leq g\_\{t\+k\}^\{\(2\)\}\\leq 1for allk≥1k\\geq 1\. Assuming a fixed criticVψV\_\{\\psi\}and independent transition noise alongxt\+k∼pϕ\(⋅∣xt\+k−1,at\+k−1\)x\_\{t\+k\}\\sim p\_\{\\phi\}\(\\cdot\\mid x\_\{t\+k\-1\},a\_\{t\+k\-1\}\), the conditional target variances satisfy:

Var​\(Gtg\(1\)\|xt\)≤Var​\(Gtg\(2\)\|xt\)\\text\{Var\}\\left\(G\_\{t\}^\{g^\{\(1\)\}\}\\;\\middle\|\\;x\_\{t\}\\right\)\\leq\\text\{Var\}\\left\(G\_\{t\}^\{g^\{\(2\)\}\}\\;\\middle\|\\;x\_\{t\}\\right\)\(48\)In particular, settinggt\+1\(1\)=0g\_\{t\+1\}^\{\(1\)\}=0\(a hard bootstrap boundary\) truncates downstream rollout stochasticity and yields:

Var​\(Gtg=0\|xt\)=Var​\(r^t\+1\+dt\+1​Vψ​\(xt\+1\)\|xt\)≤Var​\(Gtλ=1\|xt\)\\text\{Var\}\\left\(G\_\{t\}^\{g=0\}\\;\\middle\|\\;x\_\{t\}\\right\)=\\text\{Var\}\\left\(\\hat\{r\}\_\{t\+1\}\+d\_\{t\+1\}V\_\{\\psi\}\(x\_\{t\+1\}\)\\;\\middle\|\\;x\_\{t\}\\right\)\\leq\\text\{Var\}\\left\(G\_\{t\}^\{\\lambda=1\}\\;\\middle\|\\;x\_\{t\}\\right\)\(49\)

###### Proof\.

By the Law of Total Variance, for any scalar random variableYYconditioned onxtx\_\{t\}and intermediate statext\+1x\_\{t\+1\}:

Var​\(Y∣xt\)=Varxt\+1​\(𝔼⁡\[Y∣xt\+1\]\|xt\)\+𝔼xt\+1​\(Var​\(Y∣xt\+1\)\|xt\)\\text\{Var\}\(Y\\mid x\_\{t\}\)=\\text\{Var\}\_\{x\_\{t\+1\}\}\\left\(\\mathbb\{E\}\[Y\\mid x\_\{t\+1\}\]\\;\\middle\|\\;x\_\{t\}\\right\)\+\\mathbb\{E\}\_\{x\_\{t\+1\}\}\\left\(\\text\{Var\}\(Y\\mid x\_\{t\+1\}\)\\;\\middle\|\\;x\_\{t\}\\right\)SubstitutingY=GtgY=G\_\{t\}^\{g\}:

𝔼⁡\[Gtg∣xt\+1\]=r^t\+1\+dt\+1​\(1−λ​gt\+1\)​Vψ​\(xt\+1\)\+dt\+1​λ​gt\+1​𝔼​\[Gt\+1g∣xt\+1\]\\mathbb\{E\}\[G\_\{t\}^\{g\}\\mid x\_\{t\+1\}\]=\\hat\{r\}\_\{t\+1\}\+d\_\{t\+1\}\(1\-\\lambda g\_\{t\+1\}\)V\_\{\\psi\}\(x\_\{t\+1\}\)\+d\_\{t\+1\}\\lambda g\_\{t\+1\}\\mathbb\{E\}\[G\_\{t\+1\}^\{g\}\\mid x\_\{t\+1\}\]Var​\(Gtg∣xt\+1\)=\(dt\+1​λ​gt\+1\)2​Var​\(Gt\+1g∣xt\+1\)\\text\{Var\}\(G\_\{t\}^\{g\}\\mid x\_\{t\+1\}\)=\(d\_\{t\+1\}\\lambda g\_\{t\+1\}\)^\{2\}\\text\{Var\}\(G\_\{t\+1\}^\{g\}\\mid x\_\{t\+1\}\)Thus, the conditional variance satisfies the recursive identity:

Var​\(Gtg∣xt\)\\displaystyle\\text\{Var\}\(G\_\{t\}^\{g\}\\mid x\_\{t\}\)=Varxt\+1​\(r^t\+1\+dt\+1​\(1−λ​gt\+1\)​Vψ​\(xt\+1\)\+dt\+1​λ​gt\+1​𝔼​\[Gt\+1g∣xt\+1\]\|xt\)\\displaystyle=\\text\{Var\}\_\{x\_\{t\+1\}\}\\left\(\\hat\{r\}\_\{t\+1\}\+d\_\{t\+1\}\(1\-\\lambda g\_\{t\+1\}\)V\_\{\\psi\}\(x\_\{t\+1\}\)\+d\_\{t\+1\}\\lambda g\_\{t\+1\}\\mathbb\{E\}\[G\_\{t\+1\}^\{g\}\\mid x\_\{t\+1\}\]\\;\\middle\|\\;x\_\{t\}\\right\)\+𝔼xt\+1​\(\(dt\+1​λ​gt\+1\)2​Var​\(Gt\+1g∣xt\+1\)\|xt\)\\displaystyle\\quad\+\\mathbb\{E\}\_\{x\_\{t\+1\}\}\\left\(\(d\_\{t\+1\}\\lambda g\_\{t\+1\}\)^\{2\}\\text\{Var\}\(G\_\{t\+1\}^\{g\}\\mid x\_\{t\+1\}\)\\;\\middle\|\\;x\_\{t\}\\right\)Because\(dt\+1​λ​gt\+1\(1\)\)2≤\(dt\+1​λ​gt\+1\(2\)\)2\(d\_\{t\+1\}\\lambda g\_\{t\+1\}^\{\(1\)\}\)^\{2\}\\leq\(d\_\{t\+1\}\\lambda g\_\{t\+1\}^\{\(2\)\}\)^\{2\}forg\(1\)≤g\(2\)g^\{\(1\)\}\\leq g^\{\(2\)\}, every term in the summation over future stepsk≥1k\\geq 1is scaled by a non\-negative, monotonically non\-decreasing multiplier ingg\. Unrolling the recursion down the trajectory tree shows thatVar​\(Gtg\(1\)∣xt\)≤Var​\(Gtg\(2\)∣xt\)\\text\{Var\}\(G\_\{t\}^\{g^\{\(1\)\}\}\\mid x\_\{t\}\)\\leq\\text\{Var\}\(G\_\{t\}^\{g^\{\(2\)\}\}\\mid x\_\{t\}\)\. Settinggt\+1=0g\_\{t\+1\}=0removes all terms containingVar​\(Gt\+kg∣xt\+k\)\\text\{Var\}\(G\_\{t\+k\}^\{g\}\\mid x\_\{t\+k\}\)fork≥1k\\geq 1, establishing the lower bound\. ∎

—

## Appendix CScore\-Divergence Dependent Contraction Rate

###### Lemma 1\(Score\-Divergence Dependent Contraction Rate\)\.

Letlt\+1=∑j=0tδjl\_\{t\+1\}=\\sum\_\{j=0\}^\{t\}\\delta\_\{j\}be the accumulated negative score\-divergence proxy along an imagined trajectory prefix, whereδj=clip​\(−D^j,−C,C\)\\delta\_\{j\}=\\text\{clip\}\(\-\\hat\{D\}\_\{j\},\-C,C\)\. Letg⁡\(l\)=11\+exp\(−\(l−τ\)/T\)g\(l\)=\\frac\{1\}\{1\+\\exp\(\-\(l\-\\tau\)/T\)\}denote the soft continuation gate with temperatureT\>0T\>0and thresholdτ\\tau\. The local contraction modulusγg​\(lt\+1\)\\gamma\_\{g\}\(l\_\{t\+1\}\)of the gated Bellman operatorTgλT\_\{g\}^\{\\lambda\}is given by:

γg​\(lt\+1\)=γ\(1\+e−\(lt\+1−τ\)/T−λ\)1\+e−\(lt\+1−τ\)/T−γλ\\gamma\_\{g\}\(l\_\{t\+1\}\)=\\frac\{\\gamma\\left\(1\+e^\{\-\(l\_\{t\+1\}\-\\tau\)/T\}\-\\lambda\\right\)\}\{1\+e^\{\-\(l\_\{t\+1\}\-\\tau\)/T\}\-\\gamma\\lambda\}\(50\)Furthermore:

1. 1\.The contraction rate is strictly monotonically decreasing inlt\+1l\_\{t\+1\}: ∂γg∂lt\+1<0for all​λ∈\(0,1\),γ∈\(0,1\)\\frac\{\\partial\\gamma\_\{g\}\}\{\\partial l\_\{t\+1\}\}<0\\quad\\text\{for all \}\\lambda\\in\(0,1\),\\;\\gamma\\in\(0,1\)\(51\)
2. 2\.The asymptotic values satisfy: limlt\+1→−∞γg​\(lt\+1\)=γ,limlt\+1→\+∞γg​\(lt\+1\)=γ⁡\(1−λ\)1−γ​λ<γ\\lim\_\{l\_\{t\+1\}\\to\-\\infty\}\\gamma\_\{g\}\(l\_\{t\+1\}\)=\\gamma,\\qquad\\lim\_\{l\_\{t\+1\}\\to\+\\infty\}\\gamma\_\{g\}\(l\_\{t\+1\}\)=\\frac\{\\gamma\(1\-\\lambda\)\}\{1\-\\gamma\\lambda\}<\\gamma\(52\)

###### Proof\.

Substitutinggt\+1=11\+e−\(lt\+1−τ\)/Tg\_\{t\+1\}=\\frac\{1\}\{1\+e^\{\-\(l\_\{t\+1\}\-\\tau\)/T\}\}into the contraction modulus formulaγg=γ⁡\(1−λ​gt\+1\)1−γ​λ​gt\+1\\gamma\_\{g\}=\\frac\{\\gamma\(1\-\\lambda g\_\{t\+1\}\)\}\{1\-\\gamma\\lambda g\_\{t\+1\}\}from Theorem[3](https://arxiv.org/html/2610.41022#Thmtheo3):

γg​\(lt\+1\)=γ⁡\(1−λ1\+e−\(lt\+1−τ\)/T\)1−γ​λ1\+e−\(lt\+1−τ\)/T\\gamma\_\{g\}\(l\_\{t\+1\}\)=\\frac\{\\gamma\\left\(1\-\\frac\{\\lambda\}\{1\+e^\{\-\(l\_\{t\+1\}\-\\tau\)/T\}\}\\right\)\}\{1\-\\frac\{\\gamma\\lambda\}\{1\+e^\{\-\(l\_\{t\+1\}\-\\tau\)/T\}\}\}Multiplying numerator and denominator by1\+e−\(lt\+1−τ\)/T1\+e^\{\-\(l\_\{t\+1\}\-\\tau\)/T\}:

γg​\(lt\+1\)=γ\(1\+e−\(lt\+1−τ\)/T−λ\)1\+e−\(lt\+1−τ\)/T−γλ\\gamma\_\{g\}\(l\_\{t\+1\}\)=\\frac\{\\gamma\\left\(1\+e^\{\-\(l\_\{t\+1\}\-\\tau\)/T\}\-\\lambda\\right\)\}\{1\+e^\{\-\(l\_\{t\+1\}\-\\tau\)/T\}\-\\gamma\\lambda\}To establish monotonicity, defineu\(lt\+1\)=e−\(lt\+1−τ\)/T\>0u\(l\_\{t\+1\}\)=e^\{\-\(l\_\{t\+1\}\-\\tau\)/T\}\>0, noting thatd​ud​lt\+1=−1T​u<0\\frac\{du\}\{dl\_\{t\+1\}\}=\-\\frac\{1\}\{T\}u<0\. Differentiatingγg\\gamma\_\{g\}with respect touu:

∂γg∂u=γ⋅∂∂u​\(u\+1−λu\+1−γ​λ\)=γ⋅\(1\)​\(u\+1−γ​λ\)−\(u\+1−λ\)​\(1\)\(u\+1−γ​λ\)2=γ​λ​\(1−γ\)\(u\+1−γ​λ\)2\>0\\frac\{\\partial\\gamma\_\{g\}\}\{\\partial u\}=\\gamma\\cdot\\frac\{\\partial\}\{\\partial u\}\\left\(\\frac\{u\+1\-\\lambda\}\{u\+1\-\\gamma\\lambda\}\\right\)=\\gamma\\cdot\\frac\{\(1\)\(u\+1\-\\gamma\\lambda\)\-\(u\+1\-\\lambda\)\(1\)\}\{\(u\+1\-\\gamma\\lambda\)^\{2\}\}=\\frac\{\\gamma\\lambda\(1\-\\gamma\)\}\{\(u\+1\-\\gamma\\lambda\)^\{2\}\}\>0By the chain rule:

∂γg∂lt\+1=∂γg∂u⋅d​ud​lt\+1=\(γ​λ​\(1−γ\)\(u\+1−γ​λ\)2\)​\(−1T​u\)<0\\frac\{\\partial\\gamma\_\{g\}\}\{\\partial l\_\{t\+1\}\}=\\frac\{\\partial\\gamma\_\{g\}\}\{\\partial u\}\\cdot\\frac\{du\}\{dl\_\{t\+1\}\}=\\left\(\\frac\{\\gamma\\lambda\(1\-\\gamma\)\}\{\(u\+1\-\\gamma\\lambda\)^\{2\}\}\\right\)\\left\(\-\\frac\{1\}\{T\}u\\right\)<0Aslt\+1→−∞l\_\{t\+1\}\\to\-\\infty,u→\+∞u\\to\+\\infty, givinglimu→∞u\+1−λu\+1−γ​λ=1\\lim\_\{u\\to\\infty\}\\frac\{u\+1\-\\lambda\}\{u\+1\-\\gamma\\lambda\}=1, soγg→γ\\gamma\_\{g\}\\to\\gamma\. Aslt\+1→\+∞l\_\{t\+1\}\\to\+\\infty,u→0u\\to 0, givingγg→γ⁡\(1−λ\)1−γ​λ\\gamma\_\{g\}\\to\\frac\{\\gamma\(1\-\\lambda\)\}\{1\-\\gamma\\lambda\}\. ∎

###### Corollary 2\(Hard Boundary Contraction Discontinuity\)\.

For the hard continuation gateghard​\(lt\+1\)=𝕀⁡\(lt\+1≥τ\)g^\{\\text\{hard\}\}\(l\_\{t\+1\}\)=\\mathbb\{I\}\(l\_\{t\+1\}\\geq\\tau\), the local contraction modulus exhibits a step\-function phase transition at thresholdτ\\tau:

γghard​\(lt\+1\)=\{γ⁡\(1−λ\)1−γ​λ,if​lt\+1≥τγ,if​lt\+1<τ\\gamma\_\{g\}^\{\\text\{hard\}\}\(l\_\{t\+1\}\)=\\begin\{cases\}\\frac\{\\gamma\(1\-\\lambda\)\}\{1\-\\gamma\\lambda\},&\\text\{if \}l\_\{t\+1\}\\geq\\tau\\\\ \\gamma,&\\text\{if \}l\_\{t\+1\}<\\tau\\end\{cases\}\(53\)When accumulated negative score\-divergence drops below thresholdτ\\tau,TgλT\_\{g\}^\{\\lambda\}abruptly collapses to a 1\-step Bellman evaluation operator with contraction factorγ\\gamma, eliminating downstream rollout predictions\.

—

## Appendix DAnalysis of Limits on Monotonic Policy Improvement

By the Performance Difference Lemma \([Kakade and Langford \(2002\)](https://arxiv.org/html/2610.41022#bib.bib13)\), updating policyπ\\pitoπ′\\pi^\{\\prime\}yields:

J⁡\(π′\)−J⁡\(π\)=11−γ​𝔼\(x,a\)∼ρπ′​\[Aπ​\(x,a\)\]J\(\\pi^\{\\prime\}\)\-J\(\\pi\)=\\frac\{1\}\{1\-\\gamma\}\\mathbb\{E\}\_\{\(x,a\)\\sim\\rho\_\{\\pi^\{\\prime\}\}\}\\left\[A^\{\\pi\}\(x,a\)\\right\]\(54\)When the agent optimizes using the gated return surrogate advantageA~g​\(x,a\)=Gg​\(x,a\)−Vψ​\(x\)\\tilde\{A\}^\{g\}\(x,a\)=G^\{g\}\(x,a\)\-V\_\{\\psi\}\(x\), the theoretical bound depends on the target erroretarget​\(x,a\)=\|Bias​\(Gg​\(x,a\)\)\|e\_\{\\text\{target\}\}\(x,a\)=\|\\text\{Bias\}\(G^\{g\}\(x,a\)\)\|:

J⁡\(π′\)−J⁡\(π\)≥11−γ​\(𝔼ρπ′​\[A~g​\(x,a\)\]−2​maxx,a​etarget​\(x,a\)\)J\(\\pi^\{\\prime\}\)\-J\(\\pi\)\\geq\\frac\{1\}\{1\-\\gamma\}\\left\(\\mathbb\{E\}\_\{\\rho\_\{\\pi^\{\\prime\}\}\}\\left\[\\tilde\{A\}^\{g\}\(x,a\)\\right\]\-2\\max\_\{x,a\}e\_\{\\text\{target\}\}\(x,a\)\\right\)\(55\)
###### Remark 3\(Why Monotonic Improvement Cannot Be Guaranteed\)\.

Monotonic improvement \(J⁡\(π′\)≥J⁡\(π\)J\(\\pi^\{\\prime\}\)\\geq J\(\\pi\)\) requires boundingmaxx,a⁡etarget​\(x,a\)\\max\_\{x,a\}e\_\{\\text\{target\}\}\(x,a\)by a small constant\. In pessimistic offline RL \(e\.g\., MOPO, MOREL\), this is achieved by constructing explicit upper bounds on transition model error‖P^−P‖TV\\\|\\hat\{P\}\-P\\\|\_\{\\text\{TV\}\}\. In the present framework:

1. 1\.The signalltl\_\{t\}is an empirical feature\-space expansion diagnostic derived from an auxiliary vector fieldvω,ηv\_\{\\omega,\\eta\}, not a calibrated upper bound on total variation error‖pϕ−P‖TV\\\|p\_\{\\phi\}\-P\\\|\_\{\\text\{TV\}\}\.
2. 2\.Truncating at statextx\_\{t\}absorbs critic approximation biase⁡\(xt\)=Vψ​\(xt\)−Vπ​\(xt\)e\(x\_\{t\}\)=V\_\{\\psi\}\(x\_\{t\}\)\-V^\{\\pi\}\(x\_\{t\}\)\. Ifgt\+1=0g\_\{t\+1\}=0truncates at a state where the critic is inaccurate,etarget​\(x,a\)e\_\{\\text\{target\}\}\(x,a\)can grow, preventing a proof of guaranteed policy improvement\.

## Appendix EDreamerV3 Implementation Details

Algorithm 2Dreamer V3 World\-Model and Auxiliary Update0:Replay batch of observations, actions, rewards, continuation labels, and reset indicators; world model parameters

ϕ\\phi; displacement parameters

ω\\omega; noise predictor parameters

η\\eta\.

1:Encode observations and infer posterior RSSM sequence using observed actions and reset indicators\.

2:Compute base Dreamer V3 loss

ℒWM\\mathcal\{L\}\_\{\\text\{WM\}\}\(reconstruction, reward, continuation, dynamics, representation\)\.

3:Extract detached posterior features

x¯i←sg​\(Fϕ​\(zi\)\)\\bar\{x\}\_\{i\}\\leftarrow\\text\{sg\}\(F\_\{\\phi\}\(z\_\{i\}\)\)\.

4:Align each action

bib\_\{i\}with transition

x¯i→x¯i\+1\\bar\{x\}\_\{i\}\\to\\bar\{x\}\_\{i\+1\}and construct mask

mim\_\{i\}excluding transitions across episode resets\.

5:Set

ui←i/\(L−1\)u\_\{i\}\\leftarrow i/\(L\-1\)for replay sequence length

LL\.

6:Compute masked displacement loss:

ℒdisp←∑imi​‖fω​\(ui,x¯i,bi\)−\(x¯i\+1−x¯i\)‖22d​max⁡\(1,∑imi\)\.\\mathcal\{L\}\_\{\\text\{disp\}\}\\leftarrow\\frac\{\\sum\_\{i\}m\_\{i\}\\\|f\_\{\\omega\}\(u\_\{i\},\\bar\{x\}\_\{i\},b\_\{i\}\)\-\(\\bar\{x\}\_\{i\+1\}\-\\bar\{x\}\_\{i\}\)\\\|\_\{2\}^\{2\}\}\{d\\max\(1,\\sum\_\{i\}m\_\{i\}\)\}\.
7:Sample noise scales

α∼𝒰⁡\[αmin,αmax\]\\alpha\\sim\\mathcal\{U\}\[\\alpha\_\{\\min\},\\alpha\_\{\\max\}\]and independent Gaussian noise

ϵ∼𝒩⁡\(0,𝐈d\)\\epsilon\\sim\\mathcal\{N\}\(0,\\mathbf\{I\}\_\{d\}\)\.

8:Compute unweighted noise prediction loss:

ℒscore←mean​\(1d​‖ϵη​\(x¯\+α​ϵ,α\)−ϵ‖22\)\.\\mathcal\{L\}\_\{\\text\{score\}\}\\leftarrow\\text\{mean\}\\left\(\\frac\{1\}\{d\}\\\|\\epsilon\_\{\\eta\}\(\\bar\{x\}\+\\alpha\\epsilon,\\alpha\)\-\\epsilon\\\|\_\{2\}^\{2\}\\right\)\.
9:Form

ℒmodel←ℒWM\+βdisp​ℒdisp\+βscore​ℒscore\\mathcal\{L\}\_\{\\text\{model\}\}\\leftarrow\\mathcal\{L\}\_\{\\text\{WM\}\}\+\\beta\_\{\\text\{disp\}\}\\mathcal\{L\}\_\{\\text\{disp\}\}\+\\beta\_\{\\text\{score\}\}\\mathcal\{L\}\_\{\\text\{score\}\}\.

10:Update

\(ϕ,ω,η\)\(\\phi,\\omega,\\eta\)using shared model optimizer and global gradient clipping\.

11:returnDetached posterior states as starting points for imagined behavior learning\.

Algorithm 3Dreamer V3 Imagined Behavior Update with Density Gates0:Detached posterior starting states; actor

πθ\\pi\_\{\\theta\}; critic

VψV\_\{\\psi\}; trained world model and auxiliary networks; boundary settings

\(τ,T,C,m,Nwarm\)\(\\tau,T,C,m,N\_\{\\text\{warm\}\}\)\.

1:Hold world model and auxiliary parameters fixed during behavior update\.

2:Generate

H\+1H\+1feature entries using standard RSSM imagination and actions sampled from

πθ\\pi\_\{\\theta\}\.

3:Predict rewards

r^t\\hat\{r\}\_\{t\}, continuation probabilities

c^t\\hat\{c\}\_\{t\}, and values

Vψ​\(xt\)V\_\{\\psi\}\(x\_\{t\}\)\.

4:Set

l0←0l\_\{0\}\\leftarrow 0\.

5:for

t=0,…,H−1t=0,\\dots,H\-1do

6:Set

ut←t/Hu\_\{t\}\\leftarrow t/Hand evaluate

vt​\(x\)=fω​\(ut,x,at\)\+σ2​ϵη​\(x,σ\)v\_\{t\}\(x\)=f\_\{\\omega\}\(u\_\{t\},x,a\_\{t\}\)\+\\frac\{\\sigma\}\{2\}\\epsilon\_\{\\eta\}\(x,\\sigma\)\.

7:Estimate

D^t←ξt⊤​\[∇xvt​\(x\)\]x=xt​ξt\\hat\{D\}\_\{t\}\\leftarrow\\xi\_\{t\}^\{\\top\}\[\\nabla\_\{x\}v\_\{t\}\(x\)\]\_\{x=x\_\{t\}\}\\xi\_\{t\}using independent Rademacher probe

ξt\\xi\_\{t\}, holding

utu\_\{t\}and

ata\_\{t\}fixed\.

8:Set

lt\+1←lt\+clip​\(−D^t,−C,C\)l\_\{t\+1\}\\leftarrow l\_\{t\}\+\\text\{clip\}\(\-\\hat\{D\}\_\{t\},\-C,C\)\.

9:endfor

10:Detach

l0:Hl\_\{0:H\}after computing spatial derivatives\.

11:Construct hard or soft gates

gtg\_\{t\}\(Section 5\.4\), applying warmup and minimum depth overrides\.

12:Compute detached survival weights

St←∏j=0tgjS\_\{t\}\\leftarrow\\prod\_\{j=0\}^\{t\}g\_\{j\}\.

13:Set

GH←Vψ​\(xH\)G\_\{H\}\\leftarrow V\_\{\\psi\}\(x\_\{H\}\)\.

14:for

t=H−1,…,0t=H\-1,\\dots,0do

15:

Gt←r^t\+1\+γ​c^t\+1​\[\(1−λ​gt\+1\)​Vψ​\(xt\+1\)\+λ​gt\+1​Gt\+1\]G\_\{t\}\\leftarrow\\hat\{r\}\_\{t\+1\}\+\\gamma\\hat\{c\}\_\{t\+1\}\[\(1\-\\lambda g\_\{t\+1\}\)V\_\{\\psi\}\(x\_\{t\+1\}\)\+\\lambda g\_\{t\+1\}G\_\{t\+1\}\]\.

16:endfor

17:Construct inherited detached discount weights

W0=1W\_\{0\}=1and

Wt=∏j=0t−1γ​c^jW\_\{t\}=\\prod\_\{j=0\}^\{t\-1\}\\gamma\\hat\{c\}\_\{j\}\.

18:Set

Ωt←Wt​St\\Omega\_\{t\}\\leftarrow W\_\{t\}S\_\{t\}for

t=0,…,H−1t=0,\\dots,H\-1\.

19:Update return normalization statistics using mode\-dependent target selection\.

20:Update actor using base dynamics\-gradient or likelihood\-ratio surrogate weighted by

Ωt\\Omega\_\{t\}, with entropy weighted by

StS\_\{t\}\.

21:Update distributional critic using detached targets

GtG\_\{t\}, detached features, and weights

Ωt\\Omega\_\{t\}\.

22:Average both objectives over full batch\-time tensor \(

B×HB\\times H\) without normalizing by surviving mass\.

23:Apply base slow\-critic update schedule\.

#### Indexing and normalization\.

The implementation settingimag\_horizon=15 produces 15 feature entries and 14 return\-training transitions, corresponding toH=14H=14in these algorithms\. Replay and imagination use different sequence\-position normalizations; neither is an ODE integration step size\. The implemented return\-normalization statistics exclude zero\-survival targets in hard and fixed modes, whereas soft mode retains all targets\.

#### Gradient handling\.

Auxiliary feature inputs are detached during model training\. During divergence estimation, the spatial derivative includes the score network’s dependence on its feature input\. The resulting density proxies, gates, and survival weights are detached from actor–critic optimization\. Freezing the world\-model parameters during dynamics\-based actor learning does not remove the differentiable paths through imagined states and actions\.

#### Computational scope\.

The full imagined rollout is generated before applying the gates\. The method changes return expansion and optimization weights; it does not implement early termination of RSSM trajectory generation\.