三思而行：通过世界模型中的比较想象进行探索与规划

# Second Thoughts: Exploring and Planning through Comparative Imagination in World Models
Source: [https://arxiv.org/html/2610.28779](https://arxiv.org/html/2610.28779)
Hong YuAffiliation:University of Massachusetts Amherst University of Massachusetts Lowell

###### Abstract

World models let agents compare imagined outcomes before acting, but an agent must still decide which alternatives to explore and when a proposed plan should replace its learned policy\. We introduceSDI\(SiblingDisagreement inImagination\), which connects these decisions through a shared actor\-based reference\.SDIcompares action sequences imagined from the same model state, using repeated rollouts of the actor’s own sequence to measure variation under fixed behavior\. This reference scales both the spread across candidate returns, which provides an exploration signal, and a proposed plan’s predicted advantage, which governs action acceptance\. The agent executes a planned action when its advantage exceeds the reference variability by a fixed factor; otherwise, it follows the actor\.SDIreuses the base agent’s world model, actor, and critic without requiring an additional disagreement ensemble\. Exploration and planning guide training data collection, while evaluation uses the learned actor alone\. Across Meta\-World, Adroit, and DM\-Control,SDIachieves a higher observed task\-average score than every compared baseline in each suite and matches or exceeds the strongest baseline’s performance on1515of1818tasks\. Component ablations and robustness analyses further examine the roles of exploration, planning exposure, and action acceptance\. Paired evaluations from identical simulator states show that the gate improves both recovery from actor failures and preservation of successful behavior relative to ungated planning, while substantially reducing return degradation\. Together, these results support comparing imagined alternatives against a shared actor reference as a unified approach to exploring and planning with world models\.111Code and configurations are available in the anonymous repository:[https://anonymous\.4open\.science/r/sdi\-EEDA](https://anonymous.4open.science/r/sdi-EEDA)\.

Figure 1:Overview ofSDI\.Imagine:candidate action sequences \(siblings, blue\) and repeated rollouts of the actor’s fixed sequence \(dashed gray\) start at the same latent stateztz\_\{t\}\.Compare:fresh plan and actor rollouts provide the mean return gapg^\\hat\{g\}and actor\-return standard deviationss; dots represent candidate mean returns, whose spread is measured bysib\\mathrm\{sib\}\.Act/explore:the upper branch selects the planned actionμ0\\mu\_\{0\}or actor fallbackaπa^\{\\pi\}; dark cells indicate the executed action\. The lower branch normalizes, clips, and weights the exploration bonusbbto augment the first imagined rewardr~\\tilde\{r\}\. The bonus anneals after the first training success\. Exploration and gating use separate rollout settings; evaluation uses the actor alone\.## 1Introduction

Learned world models support two complementary ways to improve control\. Agents such as Dreamer train a fast actor on imagined experience\([Hafner et al\., 2020](https://arxiv.org/html/2610.28779#bib.bib33);[Hafner et al\., 2025](https://arxiv.org/html/2610.28779#bib.bib19)\), whereas PlaNet and TD\-MPC2 search through predicted futures when an action is needed\([Hafner et al\., 2019](https://arxiv.org/html/2610.28779#bib.bib3);[Hansen et al\., 2024](https://arxiv.org/html/2610.28779#bib.bib9)\)\. Hybrid approaches use a learned actor or value function to guide online search\([Sikchi et al\., 2022](https://arxiv.org/html/2610.28779#bib.bib39)\)\. These approaches make imagination useful both for learning a policy and for revising its decisions\. An actor acts quickly, whereas a planner searches at each step\. During online learning, planned actions also change the actor’s training data\. When learning stalls, when should a searched action replace the actor’s choice?

A planner optimizes predictions from an imperfect model\. Its highest\-scoring sequence can therefore exploit errors in those predictions rather than improve the real outcome\([Talvitie, 2017](https://arxiv.org/html/2610.28779#bib.bib45);[Lambert et al\., 2020](https://arxiv.org/html/2610.28779#bib.bib44);[Sims et al\., 2024](https://arxiv.org/html/2610.28779#bib.bib1)\)\. Searching many alternatives can favor prediction errors, and even an accurately predicted high return offers little reason to intervene if the actor would do as well\. Probabilistic ensembles and short model rollouts limit some forms of this risk\([Chua et al\., 2018](https://arxiv.org/html/2610.28779#bib.bib2);[Janner et al\., 2019](https://arxiv.org/html/2610.28779#bib.bib4)\), but a decision still requires comparing a proposed plan with the action the agent would otherwise take\. During training, that choice also changes the experience from which the actor learns\. It is thus important to distinguish the effect of planning on the eventual policy from the effect of a particular intervention\([Hamrick et al\., 2021](https://arxiv.org/html/2610.28779#bib.bib46)\)\.

Exploration poses a related problem\. Prediction error, novelty, and model disagreement can direct an agent toward unfamiliar experience\([Pathak et al\., 2017](https://arxiv.org/html/2610.28779#bib.bib40);[Burda et al\., 2019](https://arxiv.org/html/2610.28779#bib.bib41);[Pathak et al\., 2019](https://arxiv.org/html/2610.28779#bib.bib42);[Sekar et al\., 2020](https://arxiv.org/html/2610.28779#bib.bib22)\)\. These signals are valuable for learning the environment, but familiarity with the dynamics and progress on a particular task are different properties\. Even a predictable state may offer actions with very different task outcomes\. When a task reward is specified, the agent may therefore seek actions with meaningfully different predicted returns\. A large spread across imagined actions is hard to interpret on its own, however: stochastic rollouts can also disagree about the return of one fixed action sequence\. A useful comparison needs to account for that variation while remaining relevant to the actor’s current behavior\.

We introduceSDI\(SiblingDisagreement inImagination\), which compares imagined action sequences with repeated rollouts of the actor’s fixed sequence from the same model state\. This actor\-based reference guides exploration and plan acceptance during data collection; evaluation uses the actor alone\. SDI reuses the existing world model and search optimizer\. Across Meta\-World\([Yu et al\., 2020a](https://arxiv.org/html/2610.28779#bib.bib17)\), Adroit\([Rajeswaran et al\., 2018](https://arxiv.org/html/2610.28779#bib.bib32)\), and DM\-Control\([Tassa et al\., 2018](https://arxiv.org/html/2610.28779#bib.bib18)\),SDImatches or exceeds the strongest compared baseline on1515of1818tasks\. In a replicated, matched\-occupancy ablation, it raises mean precision\-manipulation success from32%32\\%with ungated planning to53%53\\%\.

Our contributions are:

- •A task\-directed measure of imagined disagreement\.We define sibling disagreement as variation in predicted task returns across action sequences imagined from the same latent state, measured relative to repeated rollouts of the actor’s fixed sequence\. This comparison separates action\-dependent return variation from stochastic variation under fixed behavior\.
- •A unified rule for exploration and selective planning\.We introduceSDI, which uses the same actor\-repeat reference to reward informative imagined alternatives and to accept a planned action only when its predicted advantage exceeds local actor variability\. SDI requires no additional disagreement ensemble, guides data collection during training, and retains actor\-only evaluation\.
- •Broad and intervention\-level empirical evidence\.We demonstrate that SDI achieves the highest observed task\-average score among compared methods in each of three benchmark suites and matches or exceeds the strongest baseline mean on1515of1818tasks\. Matched\-occupancy ablations and paired\-state executions further show improved actor learning, recovery, success preservation, and return retention relative to ungated planning\.

## 2Related work

World\-model learning and planning\.Dreamer trains an actor in imagined rollouts and executes it without further search\([Hafner et al\., 2020](https://arxiv.org/html/2610.28779#bib.bib33);[Hafner et al\., 2021](https://arxiv.org/html/2610.28779#bib.bib34);[Hafner et al\., 2025](https://arxiv.org/html/2610.28779#bib.bib19)\); R2\-Dreamer provides the primary base agent in our experiments\([Morihira et al\., 2026](https://arxiv.org/html/2610.28779#bib.bib20)\)\. Decision\-time planners instead search through model\-based imagination\([Hafner et al\., 2019](https://arxiv.org/html/2610.28779#bib.bib3);[Schrittwieser et al\., 2020](https://arxiv.org/html/2610.28779#bib.bib35);[Ye et al\., 2021](https://arxiv.org/html/2610.28779#bib.bib36);[Hansen et al\., 2022](https://arxiv.org/html/2610.28779#bib.bib37);[Hansen et al\., 2024](https://arxiv.org/html/2610.28779#bib.bib9);[Nagabandi et al\., 2020](https://arxiv.org/html/2610.28779#bib.bib38);[Zhou et al\., 2025](https://arxiv.org/html/2610.28779#bib.bib5)\), while hybrid methods combine search with learned policies or values\([Sikchi et al\., 2022](https://arxiv.org/html/2610.28779#bib.bib39);[Anthony et al\., 2017](https://arxiv.org/html/2610.28779#bib.bib8)\)\. LOOP combines finite\-horizon model lookahead with an off\-policy terminal value and constrains search relative to an actor\([Sikchi et al\., 2022](https://arxiv.org/html/2610.28779#bib.bib39)\)\. SDI adds an explicit post\-search comparison against repeated actor rollouts and uses that same reference construction for exploration\. This targets the acceptance of a proposal as well as the experience collected through planning\([Hamrick et al\., 2021](https://arxiv.org/html/2610.28779#bib.bib46)\)\.

Model error and uncertainty\.Model error can compound over a rollout\([Talvitie, 2017](https://arxiv.org/html/2610.28779#bib.bib45)\), prediction objectives need not align with control\([Lambert et al\., 2020](https://arxiv.org/html/2610.28779#bib.bib44)\), and planning can exploit poorly covered regions\([Sims et al\., 2024](https://arxiv.org/html/2610.28779#bib.bib1)\)\. Short rollouts, ensembles, and uncertainty\-aware updates address these problems in different ways\([Janner et al\., 2019](https://arxiv.org/html/2610.28779#bib.bib4);[Chua et al\., 2018](https://arxiv.org/html/2610.28779#bib.bib2);[Buckman et al\., 2018](https://arxiv.org/html/2610.28779#bib.bib16);[Yu et al\., 2020b](https://arxiv.org/html/2610.28779#bib.bib43);[Pan et al\., 2020](https://arxiv.org/html/2610.28779#bib.bib15);[Frauenknecht et al\., 2024](https://arxiv.org/html/2610.28779#bib.bib14)\)\. PETS propagates uncertainty through probabilistic ensembles and trajectory sampling\([Chua et al\., 2018](https://arxiv.org/html/2610.28779#bib.bib2)\)\. SDI obtains its reference from stochastic repeats within the base world model\. This estimates rollout variability for a fixed actor sequence; model bias remains possible\([Berger et al\., 2026](https://arxiv.org/html/2610.28779#bib.bib23)\)\. Our paired\-state experiments test whether a margin scaled by that reference identifies executed improvements\.

Exploration and action comparisons\.Intrinsic rewards use prediction error, novelty, information gain, or ensemble disagreement to encourage exploration\([Pathak et al\., 2017](https://arxiv.org/html/2610.28779#bib.bib40);[Pathak et al\., 2019](https://arxiv.org/html/2610.28779#bib.bib42);[Burda et al\., 2019](https://arxiv.org/html/2610.28779#bib.bib41);[Houthooft et al\., 2016](https://arxiv.org/html/2610.28779#bib.bib26);[Sekar et al\., 2020](https://arxiv.org/html/2610.28779#bib.bib22);[Kauvar et al\., 2023](https://arxiv.org/html/2610.28779#bib.bib29)\)\. Other signals measure action influence or reward disagreement\([Seitzer et al\., 2021](https://arxiv.org/html/2610.28779#bib.bib24);[Bierling et al\., 2025](https://arxiv.org/html/2610.28779#bib.bib31)\)\. Adaptive mechanisms switch exploration modes or balance intrinsic and extrinsic rewards\([Pîslar et al\., 2022](https://arxiv.org/html/2610.28779#bib.bib27);[Chen et al\., 2022](https://arxiv.org/html/2610.28779#bib.bib25)\), while dormant\-representation diagnostics provide a different basis for adapting training interventions\([Sokar et al\., 2023](https://arxiv.org/html/2610.28779#bib.bib30);[Xu et al\., 2024](https://arxiv.org/html/2610.28779#bib.bib21)\)\. Plan2Explore rewards disagreement among predictions of the next observation embedding to seek expected information gain\([Sekar et al\., 2020](https://arxiv.org/html/2610.28779#bib.bib22)\)\. SDI instead compares task\-return predictions across action sequences, relative to actor\-repeat variability\. This makes the signal task\-directed and ties it to the reference used for plan acceptance\. The construction relates to interventional advantage, counterfactual credit assignment, and twin models\([Pan et al\., 2022](https://arxiv.org/html/2610.28779#bib.bib10);[Pan and Schölkopf, 2024](https://arxiv.org/html/2610.28779#bib.bib11);[Mesnard et al\., 2021](https://arxiv.org/html/2610.28779#bib.bib13);[Balke and Pearl, 1994](https://arxiv.org/html/2610.28779#bib.bib6);[Vlontzos et al\., 2023](https://arxiv.org/html/2610.28779#bib.bib12)\); interventional causal representation learning likewise studies information obtained under interventions\([Ahuja et al\., 2023](https://arxiv.org/html/2610.28779#bib.bib7)\)\. We therefore evaluate SDI’s predictions using executed alternatives\.

## 3Method

SDIaugments a world\-model actor–critic during training while leaving the base learning objectives unchanged\. At each latent state, it constructs a local actor reference from repeated imagined rollouts of a fixed actor sequence\. The same reference scales a task\-directed exploration bonus and the advantage required to execute a planned action\. Planning changes the experience collected for learning; evaluation executes the learned actor’s mode without planning\. Algorithm[1](https://arxiv.org/html/2610.28779#alg1)summarizes the complete training\-time interaction\.

### 3\.1Actor\-referenced imagined returns

The base agent provides a recurrent latent dynamics modelzτ\+1=fimg​\(zτ,aτ\)z\_\{\\tau\+1\}=f\_\{\\mathrm\{img\}\}\(z\_\{\\tau\},a\_\{\\tau\}\), reward headrr, continuation headcont\\mathrm\{cont\}, actorπ\\pi, and criticVV\. For anHH\-step action sequence, its continuation\-weighted imagined return is

G\(a0:H−1\)=∑τ<Hdτr\(zτ\+1\)\+dHV\(zH\),d0=1,dτ\+1=dτγcont\(zτ\+1\)\.G\(a\_\{0:H\-1\}\)=\\sum\_\{\\tau<H\}d\_\{\\tau\}r\(z\_\{\\tau\+1\}\)\+d\_\{H\}V\(z\_\{H\}\),\\qquad d\_\{0\}=1,\\quad d\_\{\\tau\+1\}=d\_\{\\tau\}\\gamma\\,\\mathrm\{cont\}\(z\_\{\\tau\+1\}\)\.Model predictive path integral \(MPPI\) search in latent space\([Williams et al\., 2016](https://arxiv.org/html/2610.28779#bib.bib28)\)evaluates candidate action sequences by averaging stochastic model rollouts\. After search,SDIre\-evaluates the selected plan and the actor’s fixed mode sequence withRgR\_\{g\}fresh rollouts each\. LetG¯plan\\overline\{G\}\_\{\\mathrm\{plan\}\}andG¯π\\overline\{G\}\_\{\\pi\}denote their mean returns, and letsπs\_\{\\pi\}denote the standard deviation across actor\-sequence rollouts\. The imagined advantage and its normalized score are

g^​\(z\)=G¯plan−G¯π,IU⁡\(z\)=g^​\(z\)sπ​\(z\)\.\\hat\{g\}\(z\)=\\overline\{G\}\_\{\\mathrm\{plan\}\}\-\\overline\{G\}\_\{\\pi\},\\qquad\\mathrm\{IU\}\(z\)=\\frac\{\\hat\{g\}\(z\)\}\{s\_\{\\pi\}\(z\)\}\.\(1\)Both sequences start from the same latent state, so their mean\-return gap compares the proposed change in behavior against what the actor would do there\. The denominator answers a different question: how much can the predicted return vary when the action sequence is held fixed? A plan whose advantage is small relative to that local variation provides less reason to replace the actor than the same raw gap in a stable state\. This scale does not correct model errors shared by the two sequences or turnIU\\mathrm\{IU\}into a calibrated probability of improvement\. The actor sequence is generated once and held fixed across its repeated rollouts, sosπs\_\{\\pi\}measures return variation under fixed behavior\. Exploration and action gating share the actor\-referenced comparison, not the sampled trajectories: exploration uses short sibling rollouts and its own actor repeats, whereas the gate freshly re\-scores a selected plan and actor sequence over the planning horizon\. All imagined rollouts use independent model noise; Appendix[A\.1](https://arxiv.org/html/2610.28779#A1.SS1.SSS0.Px1)distinguishes this sampling from the paired simulator randomness used only for analysis\.

Algorithm 1SDItraining\-time interaction\.Exploration modifies imagined rewards, while selective planning modifies environment actions\.1:latent

ztz\_\{t\}, actor

π\\pi, critic

VV, model

fimgf\_\{\\mathrm\{img\}\}
2:Draw fallback action

at∼π\(⋅∣zt\)a\_\{t\}\\sim\\pi\(\\cdot\\mid z\_\{t\}\)
3:Update bonus weight from the first\-training\-success schedule

4:ifthe exploration bonus is activethen

5:Compute

bbfrom sibling and actor\-reference rollouts \(Eq\.[2](https://arxiv.org/html/2610.28779#S3.E2)\)

6:Weight

bband augment the first imagined actor–critic reward

7:Update planner availability from episode\-return history

8:ifthe planner is availablethen

9:

μ←MPPI⁡\(zt,N,H,I,R\)\\mu\\leftarrow\\mathrm\{MPPI\}\(z\_\{t\};N,H,I,R\)
10:Fix the actor’s mode sequence and re\-score it and

μ\\muwith

RgR\_\{g\}fresh rollouts

11:Compute

g^\\hat\{g\}and

sπs\_\{\\pi\}\(Eq\.[1](https://arxiv.org/html/2610.28779#S3.E1)\)

12:if

g^\>κ​sπ\\hat\{g\}\>\\kappa s\_\{\\pi\}then

at←μ0a\_\{t\}\\leftarrow\\mu\_\{0\}
13:Execute

ata\_\{t\}, append the transition to replay, and update the base agent

Table 1:Cross\-domain benchmark performance under actor\-only evaluation\.Meta\-World\([Yu et al\., 2020a](https://arxiv.org/html/2610.28779#bib.bib17)\)and Adroit\([Rajeswaran et al\., 2018](https://arxiv.org/html/2610.28779#bib.bib32)\)report success; DM\-Control\([Tassa et al\., 2018](https://arxiv.org/html/2610.28779#bib.bib18)\)reports episode return\. Compared baselines are Plan2Explore\([Sekar et al\., 2020](https://arxiv.org/html/2610.28779#bib.bib22)\), Curious Replay\([Kauvar et al\., 2023](https://arxiv.org/html/2610.28779#bib.bib29)\), TD\-MPC2\([Hansen et al\., 2024](https://arxiv.org/html/2610.28779#bib.bib9)\), DreamerV3\([Hafner et al\., 2025](https://arxiv.org/html/2610.28779#bib.bib19)\), and R2\-Dreamer\([Morihira et al\., 2026](https://arxiv.org/html/2610.28779#bib.bib20)\)\. Columns show half and full training budgets, and suite means weight tasks equally\.Sibling disagreement for exploration\.Before the first successful environment transition,SDIevaluatesKKaction sequences imagined from the same latent state\. We call these sequences*siblings*\. WithG^k\\hat\{G\}\_\{k\}the estimated mean return of siblingkk, their normalized return spread is

sib⁡\(z\)=stdk​\(G^k\),b⁡\(z\)=sib⁡\(z\)sπ​\(z\)\.\\mathrm\{sib\}\(z\)=\\mathrm\{std\}\_\{k\}\(\\hat\{G\}\_\{k\}\),\\qquad b\(z\)=\\frac\{\\mathrm\{sib\}\(z\)\}\{s\_\{\\pi\}\(z\)\}\.\(2\)The sibling means summarize differences among action choices, whereassπs\_\{\\pi\}summarizes variation from repeating one choice\. Their ratio favors states where imagined actions differ in task return relative to that fixed\-behavior variation\. The bonus affects the actor–critic update in imagination; it is not added to the environment reward or used as the gate’s plan advantage\. We divide this score by its running mean, clip it to\[0,3\]\[0,3\], and add it to the first reward of the imagined actor–critic rollout with weightβ=0\.5\\beta=0\.5\. Exploration usesK=5K=5siblings,Kr=3K\_\{r\}=3actor\-reference repeats, and horizon33\. The first training transition carrying the environment’s success flag starts an anneal over2020k optimizer updates\. The environment reward itself is unchanged\. The anneal gradually removes the auxiliary signal after discovery, leaving the original task reward to guide later learning\.

### 3\.2Search, fresh rescoring, and plan acceptance

SDI separates proposing a plan from deciding whether to execute it\. MPPI first searches for a high\-return sequence by iteratively sampling candidates and refitting its action distribution to the elites\. Because the selected sequence is optimized on these search rollouts, their returns are not reused as evidence for acceptance\. Instead, SDI re\-evaluates the selected plan and the actor’s fixed mode sequence from the same latent state using independent model noise andRg=4R\_\{g\}=4fresh rollouts per sequence\. This post\-search comparison suppliesg^t\\hat\{g\}\_\{t\}andsπ​\(zt\)s\_\{\\pi\}\(z\_\{t\}\)in Eq\.[1](https://arxiv.org/html/2610.28779#S3.E1); the complete MPPI update and initialization appear in Appendix[A](https://arxiv.org/html/2610.28779#A1)\. Using the actor’s repeats to set the threshold also keeps acceptance tied to the current policy, rather than to the spread of candidates that search has already optimized\.

Scoring holds the actor’s mode sequence fixed, while a rejected plan executes a fresh actor sample\. This separates the gate’s repeatable reference from stochastic training behavior\.

At a planning step, the agent executes the first planned actionμ0\\mu\_\{0\}only when the freshly estimated advantage exceeds the actor\-reference variability by a factorκ\\kappa:

at=\{μ0,g^t\>κ​sπ​\(zt\),atactor∼π\(⋅∣zt\),otherwise,κ=2\.a\_\{t\}=\\begin\{cases\}\\mu\_\{0\},&\\hat\{g\}\_\{t\}\>\\kappa s\_\{\\pi\}\(z\_\{t\}\),\\\\ a\_\{t\}^\{\\mathrm\{actor\}\}\\sim\\pi\(\\cdot\\mid z\_\{t\}\),&\\text\{otherwise\},\\end\{cases\}\\qquad\\kappa=2\.\(3\)When a plan is rejected, the agent samples a fresh actor action\. Becausesπs\_\{\\pi\}is a rollout standard deviation rather than a standard error,κ\\kappais an empirical operating threshold rather than a statistical confidence level\. Only the first planned action is executed; the agent re\-plans from its next latent state\. These two uses of the actor reference address different decisions\. Sibling disagreement identifies states where alternative actions have meaningfully different imagined returns; the gate tests whether a selected plan clears the actor’s local return variability\. The scores use separate rollout batches and horizons\. A state can therefore be useful to explore even when no proposed plan warrants replacing the actor at that step\. Exploration changes the actor’s imagined learning signal; an accepted plan changes the next environment transition and the experience stored in replay\.

Figure 2:Cross\-domain benchmark learning curves\.Lines show average rolling success on Meta\-World\([Yu et al\., 2020a](https://arxiv.org/html/2610.28779#bib.bib17)\)and Adroit\([Rajeswaran et al\., 2018](https://arxiv.org/html/2610.28779#bib.bib32)\)and average episode return on DM\-Control\([Tassa et al\., 2018](https://arxiv.org/html/2610.28779#bib.bib18)\)\. Black triangles mark the Table[1](https://arxiv.org/html/2610.28779#S3.T1)scoring depths; curves for the latter suite may extend toward the nominal500500k\-step budget\.
### 3\.3Training schedule and planner configuration

Planning is disabled for the firsttmin=20t\_\{\\min\}=20k environment steps to allow initial model learning\. Thereafter, a return\-based predicate compares an exponential average of episode return with its valueW=50W=50k steps earlier and activates planning when relative improvement falls below\.05\.05\. Once activated, planning remains available\. We call the fraction of acting steps with planner availability*occupancy*; this is distinct from the fraction on which a planned action passes the acceptance rule\.

The planner usesN=256N=256candidates, horizonH=8H=8,I=4I=4refinement iterations,E=32E=32elites, andR=4R=4stochastic rollouts per candidate\. The selected plan and actor reference are then evaluated withRg=4R\_\{g\}=4fresh rollouts each\. These settings andκ\\kappaare fixed across domains\. Search and re\-scoring occur before acceptance, so a rejected proposal still requires the full planning computation\. Appendix[A](https://arxiv.org/html/2610.28779#A1)provides optimizer updates, initialization, action bounds, and variant configurations\.

## 4Experimental setup

Benchmarks and budgets\.Table[1](https://arxiv.org/html/2610.28779#S3.T1)covers three pixel\-based suites\. Meta\-World v3\([Yu et al\., 2020a](https://arxiv.org/html/2610.28779#bib.bib17)\)uses64×6464\\times 64observations and sparse success rewards\. The pre\-specified discovery set comprises box\-close, sweep\-into, and hammer, where vanilla DreamerV3\([Hafner et al\., 2025](https://arxiv.org/html/2610.28779#bib.bib19)\)has no success by150150k steps\. The precision\-plateau set comprises hand\-insert, assembly, and disassemble\. The Meta\-World budget is700700k, except for pick\-out\-of\-hole, which uses500500k\. Adroit\([Rajeswaran et al\., 2018](https://arxiv.org/html/2610.28779#bib.bib32)\)includes hammer, door, and pen with dense training rewards and success\-based evaluation at11M steps\. DM\-Control\([Tassa et al\., 2018](https://arxiv.org/html/2610.28779#bib.bib18)\)includes five tasks at a nominal500500k budget; comparisons use each task’s common observed depth\.

Baselines and evaluation\.We compare with DreamerV3\([Hafner et al\., 2025](https://arxiv.org/html/2610.28779#bib.bib19)\), R2\-Dreamer\([Morihira et al\., 2026](https://arxiv.org/html/2610.28779#bib.bib20)\), Plan2Explore\([Sekar et al\., 2020](https://arxiv.org/html/2610.28779#bib.bib22)\), Curious Replay\([Kauvar et al\., 2023](https://arxiv.org/html/2610.28779#bib.bib29)\), and TD\-MPC2\([Hansen et al\., 2024](https://arxiv.org/html/2610.28779#bib.bib9)\)\. All reported baseline results come from our runs\. For Meta\-World\([Yu et al\., 2020a](https://arxiv.org/html/2610.28779#bib.bib17)\)and Adroit\([Rajeswaran et al\., 2018](https://arxiv.org/html/2610.28779#bib.bib32)\), the endpoint is mean success over the final100100evaluation episodes \(the last1010evaluation calls\); an episode succeeds if its success flag is set at any step\. DM\-Control\([Tassa et al\., 2018](https://arxiv.org/html/2610.28779#bib.bib18)\)averages episode return over five evaluation calls at a common scoring depth for each task\.*Crossing*means reaching a last\-1010evaluation success rate of at least0\.50\.5at the task budget\.

Table 2:Ablation study on precision\-manipulation tasks\.Entries report mean evaluation success at350350k and700700k environment steps; the mean column gives the equally weighted average of the three700700k\-step task means\. Occupancy is the fraction of acting steps with planner availability, not planned\-action execution\. Base\-agent controls are R2\-Dreamer\([Morihira et al\., 2026](https://arxiv.org/html/2610.28779#bib.bib20)\)and DreamerV3\([Hafner et al\., 2025](https://arxiv.org/html/2610.28779#bib.bib19)\)\.AblationComponenthand\-insertassemblydisassemble*mean**occupancy*350k700k350k700k350k700k700kmeanSDI\.11\.49\.07\.75\.00\.35\.53\.97Base agentR2\-Dreamer\.04\.13\.00\.00\.01\.12\.08\.00DreamerV3\.05\.16\.00\.00\.00\.00\.05\.37PlanningUngated planner\.20\.53\.00\.29\.03\.14\.32\.97Half occupancy\.14\.47\.00\.00\.00\.01\.16\.47ExplorationAlways\-on bonus\.02\.17\.00\.00\.01\.00\.06\.00Reset when stuck\.03\.16\.00\.08\.00\.00\.08\.00ReplaySpread\-prioritized replay\.27\.40\.00\.00\.00\.00\.13\.47
## 5Results

### 5\.1Cross\-domain benchmark performance

Meta\-World\.Table[1](https://arxiv.org/html/2610.28779#S3.T1)shows that across the ten Meta\-World tasks\([Yu et al\., 2020a](https://arxiv.org/html/2610.28779#bib.bib17)\),SDIachieves the highest observed suite mean at both the half and full training budgets, improving from\.43\.43to\.84\.84; the strongest baseline means are\.42\.42and\.60\.60, respectively, from R2\-Dreamer\([Morihira et al\., 2026](https://arxiv.org/html/2610.28779#bib.bib20)\)\. At the full budget,SDIis highest or tied highest on every task\. Its largest gains over the strongest alternative occur on assembly \(\.75\.75versus\.09\.09\), pick\-out\-of\-hole \(\.96\.96versus\.41\.41\), and hand\-insert \(\.49\.49versus\.34\.34\)\.

Adroit\.On Adroit\([Rajeswaran et al\., 2018](https://arxiv.org/html/2610.28779#bib.bib32)\),SDIalso has the highest observed suite mean at both budgets, reaching\.60\.60halfway through training and\.76\.76at the full budget, compared with\.33\.33and\.40\.40for the strongest baseline, R2\-Dreamer\. At the full budget, it ties the best reported result on hammer \(\.99\.99\) and door \(\.85\.85\), and leads on pen with\.44\.44, versus\.23\.23for the strongest alternative\. Thus,SDImatches or exceeds the best baseline endpoint on all three tasks\.

DM\-Control\.DM\-Control\([Tassa et al\., 2018](https://arxiv.org/html/2610.28779#bib.bib18)\)shows a more mixed task\-level ranking despite the strongest suite averages\.SDIobtains means of359\.3359\.3and546\.2546\.2at the half and full budgets, compared with241\.8241\.8and521\.6521\.6for the strongest baseline, R2\-Dreamer\. At the full budget, it leads on cartpole\-swingup \(279279\) and reacher\-hard \(896896\), and remains close to the best result on ball\-in\-cup\-catch \(961961versus973973\)\. It trails DreamerV3\([Hafner et al\., 2025](https://arxiv.org/html/2610.28779#bib.bib19)\)on acrobot\-swingup \(1616versus6969\) and finger\-turn\-hard \(579579versus616616\)\. Across all three domains, these results correspond to matching or exceeding the strongest reported baseline mean on1515of1818tasks\.

### 5\.2Training dynamics across domains

Figure[2](https://arxiv.org/html/2610.28779#S3.F2)shows how the endpoint comparisons in Table[1](https://arxiv.org/html/2610.28779#S3.T1)emerge over training; black triangles mark the half\- and full\-budget scoring depths\. On several Meta\-World\([Yu et al\., 2020a](https://arxiv.org/html/2610.28779#bib.bib17)\)tasks, endpoint convergence masks different routes to high success\. On box\-close and sweep\-into,SDIenters the near\-saturated regime after an initial learning phase and then remains stable\. On hammer, TD\-MPC2\([Hansen et al\., 2024](https://arxiv.org/html/2610.28779#bib.bib9)\)and R2\-Dreamer\([Morihira et al\., 2026](https://arxiv.org/html/2610.28779#bib.bib20)\)solve the task earlier, whereasSDIfirst plateaus near intermediate success and makes a second, late transition to saturation\. The ordering is reversed on lever\-pull:SDIrises rapidly to near\-perfect success and maintains it while the alternatives improve more slowly\. On bin\-picking and peg\-insert\-side, R2\-Dreamer\([Morihira et al\., 2026](https://arxiv.org/html/2610.28779#bib.bib20)\)makes earlier progress, butSDIcontinues improving and reaches the strongest full\-budget performance\.

The precision\-manipulation tasks show the clearest separation\.SDIremains near zero early on assembly and pick\-out\-of\-hole, then undergoes a sustained late rise; hand\-insert improves more gradually across the second half of training\. Disassemble is different again: DreamerV3\([Hafner et al\., 2025](https://arxiv.org/html/2610.28779#bib.bib19)\)improves first, butSDI’s final increase produces the best endpoint\. The Adroit\([Rajeswaran et al\., 2018](https://arxiv.org/html/2610.28779#bib.bib32)\)curves also distinguish early learning from eventual performance\. R2\-Dreamer\([Morihira et al\., 2026](https://arxiv.org/html/2610.28779#bib.bib20)\)reaches high hammer success fastest andSDIcatches up later, whileSDIcontinues to gain on door and pen despite visible fluctuations in the rolling success curves\.

The DM\-Control\([Tassa et al\., 2018](https://arxiv.org/html/2610.28779#bib.bib18)\)trajectories reflect the mixed endpoint pattern\. Ball\-in\-cup\-catch is largely an early\-saturation task for the strongest methods, whereas cartpole\-swingup and reacher\-hard continue to separate methods later in training; on reacher\-hard,SDI’s advantage appears early and is largely maintained\. By contrast, DreamerV3\([Hafner et al\., 2025](https://arxiv.org/html/2610.28779#bib.bib19)\)exhibits the strongest late\-stage rise on acrobot\-swingup and narrowly exceedsSDIat the finger\-turn\-hard scoring depth\. Thus, the suite\-level averages summarize task\-dependent learning dynamics, including early breakthroughs,delayed transitions, and differences in plateau stability, rather than uniform dominance throughout training\.

### 5\.3Ablation study

Table[2](https://arxiv.org/html/2610.28779#S4.T2)isolates the contributions of the base agent, planning exposure, action acceptance, exploration schedule, and replay prioritization on the three precision\-manipulation tasks\. The complete method obtains a final equal\-task mean of\.53\.53at\.97\.97planner occupancy, compared with\.08\.08for R2\-Dreamer\([Morihira et al\., 2026](https://arxiv.org/html/2610.28779#bib.bib20)\)and\.05\.05for the DreamerV3\([Hafner et al\., 2025](https://arxiv.org/html/2610.28779#bib.bib19)\)transfer variant\.

At matched\.97\.97occupancy, removing the acceptance gate reduces the three\-task mean from\.53\.53to\.32\.32\. The gate improves assembly from\.29\.29to\.75\.75and disassemble from\.14\.14to\.35\.35, while hand\-insert changes from\.53\.53to\.49\.49\. Reducing planner occupancy to\.47\.47lowers the mean to\.16\.16and leaves assembly and disassemble near zero, indicating that sustained planning exposure is important in this setting\. The always\-on bonus and reset controls reach means of\.06\.06and\.08\.08, respectively\. Spread\-prioritized replay improves hand\-insert to\.40\.40but does not transfer to assembly or disassemble, yielding a mean of\.13\.13\. Overall, no individual control reproduces the full method’s cross\-task result\.

Figure 3:Paired\-state planning experiments\.\(a\) Realized one\-decision improvement over the actor by normalized\-margin decile; the dashed line is the all\-plan rate\. \(b\) Proposal acceptance under four rules\. \(c\) Recovery and success preservation during5050\-step continuations from restored states\. \(d\) Checkpoint mean return difference between gated and ungated continuations\. Shading in \(a\) shows the standard error; intervals in \(c,d\) are95%95\\%\.
### 5\.4Paired\-state analysis

#### Gate reliability and selectivity\.

For Figure[3](https://arxiv.org/html/2610.28779#S5.F3)a, we sample500500decisions from each of1818checkpoints across five Meta\-World\([Yu et al\., 2020a](https://arxiv.org/html/2610.28779#bib.bib17)\)tasks and execute the actor and planned branches from the same state\. A decision is counted as beneficial only when the planned branch produces a strictly positive realized return difference\. This probability rises monotonically from about\.24\.24in the lowest normalized\-margin decile to\.42\.42in the highest, crossing the all\-plan rate of\.33\.33in the upper half of the ranking\. The imagined margin therefore contains useful ordinal information about realized improvement\. Figure[3](https://arxiv.org/html/2610.28779#S5.F3)b shows the corresponding operating points\. Always executing uses every proposal, a positive\-margin rule uses91%91\\%, theSDIgate uses47%47\\%, and the constant\-spread rule uses14%14\\%\.SDIthus occupies an intermediate regime, rejecting a slight majority of proposals without becoming as restrictive as the constant\-spread rule\. These execution shares measure selectivity rather than quality; comparisons among rules must also account for which decisions each rule retains\.

#### Recovery and success preservation\.

The sustained\-planning study in Figure[3](https://arxiv.org/html/2610.28779#S5.F3)c uses1,8001\{,\}800restored states with planning available throughout each5050\-step continuation\. Among the1,0541\{,\}054states where the actor fails, gated planning recovers success in4\.2%4\.2\\%of cases, compared with2\.4%2\.4\\%for ungated planning:4444versus2525recoveries\. Among the746746states where the actor succeeds, the gate preserves98\.3%98\.3\\%of successes, compared with95\.8%95\.8\\%under ungated planning, corresponding to1313versus3131lost successes\. Selective execution therefore improves both sides of the intervention tradeoff: it recovers more actor failures while damaging fewer actor successes\. The pooled net success change is\+\.0172\+\.0172for gated planning, compared with−\.0033\-\.0033for ungated planning\. Figure[3](https://arxiv.org/html/2610.28779#S5.F3)d directly compares the two planning rules at each checkpoint\.

## 6Conclusion

We introduced Sibling Disagreement in Imagination \(SDI\), which uses repeated actor rollouts as a shared reference for task\-directed exploration and selective planning\. The same actor\-referenced comparison measures variation across imagined alternatives and determines when a plan should replace the actor, without requiring an additional disagreement ensemble\. Planning guides data collection during training, while evaluation retains the learned actor alone\. Thus, the benchmark endpoints assess policy improvement, while the paired\-state experiments separately test the quality of the planner’s training\-time interventions\. Across1818tasks in three domains,SDImatches or exceeds the strongest reported baseline on1515tasks\. Overall, the findings support actor\-referenced comparison as a practical way to connect exploration and planning in world\-model learning\.

## AI Use Statement

Generative AI tools assisted with methodological and experimental feedback; code scaffolding and experiment orchestration; analysis scripts and interpretation checks; literature search and bibliography verification; scientific\-figure drafting; and manuscript organization, title development, and language editing\. The tools did not generate synthetic datasets or autonomously collect experimental data, and this work makes no theoretical\-proof claims\. The authors ran and inspected all AI\-assisted code, checked analyses and numerical claims against the retained experiment records, verified citations against primary bibliographic sources, and manually reviewed all generated text and figures\. The authors take responsibility for the final content of this work, including text, claims, code, and artifacts produced with AI assistance\.

## Ethics Statement

This work studies model\-based reinforcement learning in simulated control benchmarks and does not involve human participants, animal subjects, personal or sensitive data, or deployment on physical systems\. Although the method is intended for general\-purpose control research, improved planning and autonomous decision making could be misapplied in safety\-critical or harmful settings\. The reported benchmark results do not establish safety, robustness, or suitability for real\-world deployment; in particular, world\-model bias and planning error remain important limitations\. Any transfer to physical or high\-stakes domains should therefore include domain\-specific risk assessment, human oversight, fail\-safe constraints, and testing under distribution shift\. We provide code, configurations, and methodological details to support scrutiny and responsible reuse\.

## Reproducibility Statement

Section[3](https://arxiv.org/html/2610.28779#S3)specifies the return construction, exploration bonus, acceptance rule, planning schedule, and search settings\. Section[4](https://arxiv.org/html/2610.28779#S4)defines the task budgets, endpoints, and replication used for the main claims\. Evaluation always uses the learned actor without planning\. The supplementary material contains only results with complete reported cohorts\. Code, configurations, and instructions are provided in the anonymous repository[https://anonymous\.4open\.science/r/sdi\-EEDA](https://anonymous.4open.science/r/sdi-EEDA)\.

## References

- Ahujaet al\.\(2023\)K\. Ahuja, D\. Mahajan, Y\. Wang, and Y\. BengioInterventional causal representation learning\.InInternational Conference on Machine Learning \(ICML\),External Links:2209\.11924,[Link](https://arxiv.org/abs/2209.11924)Cited by:[§2](https://arxiv.org/html/2610.28779#S2.p3.1)\.
- Anthonyet al\.\(2017\)T\. Anthony, Z\. Tian, and D\. BarberThinking fast and slow with deep learning and tree search\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2610.28779#S2.p1.1)\.
- Balke and Pearl \(1994\)A\. Balke and J\. PearlProbabilistic evaluation of counterfactual queries\.InAAAI Conference on Artificial Intelligence,pp\. 230–237\.External Links:[Link](https://aaai.org/Papers/AAAI/1994/AAAI94-035.pdf)Cited by:[§2](https://arxiv.org/html/2610.28779#S2.p3.1)\.
- Bergeret al\.\(2026\)J\. Berger, B\. Frauenknecht, S\. Trimpe, and B\. LeibeBiased dreams: limitations to epistemic uncertainty quantification in latent dynamics models\.Reinforcement Learning Journal7\.External Links:2604\.25416,[Link](https://arxiv.org/abs/2604.25416)Cited by:[§2](https://arxiv.org/html/2610.28779#S2.p2.1)\.
- Bierlinget al\.\(2025\)L\. Bierling, D\. Pasero, J\. H\. Bertrand, and K\. Van GerwenDreamerV3\-XP: optimizing exploration through uncertainty estimation\.arXiv preprint arXiv:2510\.21418\.External Links:2510\.21418,[Link](https://arxiv.org/abs/2510.21418)Cited by:[§2](https://arxiv.org/html/2610.28779#S2.p3.1)\.
- Buckmanet al\.\(2018\)J\. Buckman, D\. Hafner, G\. Tucker, E\. Brevdo, and H\. LeeSample\-efficient reinforcement learning with stochastic ensemble value expansion\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:1807\.01675,[Link](https://arxiv.org/abs/1807.01675)Cited by:[§2](https://arxiv.org/html/2610.28779#S2.p2.1)\.
- Burdaet al\.\(2019\)Y\. Burda, H\. Edwards, A\. Storkey, and O\. KlimovExploration by random network distillation\.InInternational Conference on Learning Representations \(ICLR\),External Links:1810\.12894Cited by:[§1](https://arxiv.org/html/2610.28779#S1.p3.1),[§2](https://arxiv.org/html/2610.28779#S2.p3.1)\.
- Chenet al\.\(2022\)E\. Chen, Z\. Hong, J\. Pajarinen, and P\. AgrawalRedeeming intrinsic rewards via constrained optimization\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2610.28779#S2.p3.1)\.
- Chuaet al\.\(2018\)K\. Chua, R\. Calandra, R\. McAllister, and S\. LevineDeep reinforcement learning in a handful of trials using probabilistic dynamics models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2610.28779#S1.p2.1),[§2](https://arxiv.org/html/2610.28779#S2.p2.1)\.
- Frauenknechtet al\.\(2024\)B\. Frauenknecht, A\. Eisele, D\. Subhasish, F\. Solowjow, and S\. TrimpeTrust the model where it trusts itself—model\-based actor\-critic with uncertainty\-aware rollout adaption\.InInternational Conference on Machine Learning \(ICML\),pp\. 13973–14005\.External Links:2405\.19014,[Link](https://arxiv.org/abs/2405.19014)Cited by:[§2](https://arxiv.org/html/2610.28779#S2.p2.1)\.
- Hafneret al\.\(2020\)D\. Hafner, T\. Lillicrap, J\. Ba, and M\. NorouziDream to control: learning behaviors by latent imagination\.InInternational Conference on Learning Representations \(ICLR\),External Links:1912\.01603Cited by:[§1](https://arxiv.org/html/2610.28779#S1.p1.1),[§2](https://arxiv.org/html/2610.28779#S2.p1.1)\.
- Hafneret al\.\(2019\)D\. Hafner, T\. Lillicrap, I\. Fischer, R\. Villegas, D\. Ha, H\. Lee, and J\. DavidsonLearning latent dynamics for planning from pixels\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2610.28779#S1.p1.1),[§2](https://arxiv.org/html/2610.28779#S2.p1.1)\.
- Hafneret al\.\(2021\)D\. Hafner, T\. Lillicrap, M\. Norouzi, and J\. BaMastering atari with discrete world models\.InInternational Conference on Learning Representations \(ICLR\),External Links:2010\.02193Cited by:[§2](https://arxiv.org/html/2610.28779#S2.p1.1)\.
- Hafneret al\.\(2025\)D\. Hafner, J\. Pasukonis, J\. Ba, and T\. LillicrapMastering diverse control tasks through world models\.Nature640,pp\. 647–653\.External Links:[Document](https://dx.doi.org/10.1038/s41586-025-08744-2),[Link](https://doi.org/10.1038/s41586-025-08744-2)Cited by:[§1](https://arxiv.org/html/2610.28779#S1.p1.1),[§2](https://arxiv.org/html/2610.28779#S2.p1.1),[Table 1](https://arxiv.org/html/2610.28779#S3.T1),[Table 2](https://arxiv.org/html/2610.28779#S4.T2),[§4](https://arxiv.org/html/2610.28779#S4.p1.1),[§4](https://arxiv.org/html/2610.28779#S4.p2.1),[§5\.1](https://arxiv.org/html/2610.28779#S5.SS1.p3.1),[§5\.2](https://arxiv.org/html/2610.28779#S5.SS2.p2.1),[§5\.2](https://arxiv.org/html/2610.28779#S5.SS2.p3.1),[§5\.3](https://arxiv.org/html/2610.28779#S5.SS3.p1.1)\.
- Hamricket al\.\(2021\)J\. B\. Hamrick, A\. L\. Friesen, F\. Behbahani, A\. Guez, F\. Viola, S\. Witherspoon, T\. Anthony, L\. Buesing, P\. Veličković, and T\. WeberOn the role of planning in model\-based deep reinforcement learning\.InInternational Conference on Learning Representations,External Links:2011\.04021,[Link](https://openreview.net/forum?id=IrM64DGB21)Cited by:[§1](https://arxiv.org/html/2610.28779#S1.p2.1),[§2](https://arxiv.org/html/2610.28779#S2.p1.1)\.
- Hansenet al\.\(2024\)N\. Hansen, H\. Su, and X\. WangTD\-MPC2: scalable, robust world models for continuous control\.InInternational Conference on Learning Representations \(ICLR\),External Links:2310\.16828Cited by:[§1](https://arxiv.org/html/2610.28779#S1.p1.1),[§2](https://arxiv.org/html/2610.28779#S2.p1.1),[Table 1](https://arxiv.org/html/2610.28779#S3.T1),[§4](https://arxiv.org/html/2610.28779#S4.p2.1),[§5\.2](https://arxiv.org/html/2610.28779#S5.SS2.p1.1)\.
- Hansenet al\.\(2022\)N\. Hansen, X\. Wang, and H\. SuTemporal difference learning for model predictive control\.InInternational Conference on Machine Learning \(ICML\),External Links:2203\.04955Cited by:[§2](https://arxiv.org/html/2610.28779#S2.p1.1)\.
- Houthooftet al\.\(2016\)R\. Houthooft, X\. Chen, Y\. Duan, J\. Schulman, F\. De Turck, and P\. AbbeelVIME: variational information maximizing exploration\.InAdvances in Neural Information Processing Systems \(NIPS\),Cited by:[§2](https://arxiv.org/html/2610.28779#S2.p3.1)\.
- Janneret al\.\(2019\)M\. Janner, J\. Fu, M\. Zhang, and S\. LevineWhen to trust your model: model\-based policy optimization\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2610.28779#S1.p2.1),[§2](https://arxiv.org/html/2610.28779#S2.p2.1)\.
- Kauvaret al\.\(2023\)I\. Kauvar, C\. Doyle, L\. Zhou, and N\. HaberCurious replay for model\-based adaptation\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§2](https://arxiv.org/html/2610.28779#S2.p3.1),[Table 1](https://arxiv.org/html/2610.28779#S3.T1),[§4](https://arxiv.org/html/2610.28779#S4.p2.1)\.
- Lambertet al\.\(2020\)N\. Lambert, B\. Amos, O\. Yadan, and R\. CalandraObjective mismatch in model\-based reinforcement learning\.InConference on Learning for Dynamics and Control \(L4DC\),pp\. 761–770\.External Links:2002\.04523Cited by:[§1](https://arxiv.org/html/2610.28779#S1.p2.1),[§2](https://arxiv.org/html/2610.28779#S2.p2.1)\.
- Mesnardet al\.\(2021\)T\. Mesnard, T\. Weber, F\. Viola, S\. Thakoor, A\. Saade, A\. Harutyunyan, W\. Dabney, T\. S\. Stepleton, N\. Heess, A\. Guez, E\. Moulines, M\. Hutter, L\. Buesing, and R\. MunosCounterfactual credit assignment in model\-free reinforcement learning\.InInternational Conference on Machine Learning \(ICML\),pp\. 7654–7664\.External Links:2011\.09464,[Link](https://arxiv.org/abs/2011.09464)Cited by:[§2](https://arxiv.org/html/2610.28779#S2.p3.1)\.
- Morihiraet al\.\(2026\)N\. Morihira, A\. Nahar, K\. Bharadwaj, Y\. Kato, A\. Hayashi, and T\. HaradaR2\-Dreamer: redundancy\-reduced world models without decoders or augmentation\.InInternational Conference on Learning Representations \(ICLR\),External Links:2603\.18202,[Link](https://openreview.net/forum?id=Je2QqXrcQq)Cited by:[§2](https://arxiv.org/html/2610.28779#S2.p1.1),[Table 1](https://arxiv.org/html/2610.28779#S3.T1),[Table 2](https://arxiv.org/html/2610.28779#S4.T2),[§4](https://arxiv.org/html/2610.28779#S4.p2.1),[§5\.1](https://arxiv.org/html/2610.28779#S5.SS1.p1.1),[§5\.2](https://arxiv.org/html/2610.28779#S5.SS2.p1.1),[§5\.2](https://arxiv.org/html/2610.28779#S5.SS2.p2.1),[§5\.3](https://arxiv.org/html/2610.28779#S5.SS3.p1.1)\.
- Nagabandiet al\.\(2020\)A\. Nagabandi, K\. Konolige, S\. Levine, and V\. KumarDeep dynamics models for learning dexterous manipulation\.InConference on Robot Learning \(CoRL\),pp\. 1101–1112\.Cited by:[§2](https://arxiv.org/html/2610.28779#S2.p1.1)\.
- Panet al\.\(2020\)F\. Pan, J\. He, D\. Tu, and Q\. HeTrust the model when it is confident: masked model\-based actor\-critic\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:2010\.04893,[Link](https://arxiv.org/abs/2010.04893)Cited by:[§2](https://arxiv.org/html/2610.28779#S2.p2.1)\.
- Panet al\.\(2022\)H\. Pan, N\. Gürtler, A\. Neitz, and B\. SchölkopfDirect advantage estimation\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:2109\.06093,[Link](https://arxiv.org/abs/2109.06093)Cited by:[§2](https://arxiv.org/html/2610.28779#S2.p3.1)\.
- Pan and Schölkopf \(2024\)H\. Pan and B\. SchölkopfSkill or luck? return decomposition via advantage functions\.InInternational Conference on Learning Representations \(ICLR\),External Links:2402\.12874,[Link](https://arxiv.org/abs/2402.12874)Cited by:[§2](https://arxiv.org/html/2610.28779#S2.p3.1)\.
- Pathaket al\.\(2017\)D\. Pathak, P\. Agrawal, A\. A\. Efros, and T\. DarrellCuriosity\-driven exploration by self\-supervised prediction\.InInternational Conference on Machine Learning \(ICML\),pp\. 2778–2787\.Cited by:[§1](https://arxiv.org/html/2610.28779#S1.p3.1),[§2](https://arxiv.org/html/2610.28779#S2.p3.1)\.
- Pathaket al\.\(2019\)D\. Pathak, D\. Gandhi, and A\. GuptaSelf\-supervised exploration via disagreement\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2610.28779#S1.p3.1),[§2](https://arxiv.org/html/2610.28779#S2.p3.1)\.
- Pîslaret al\.\(2022\)M\. Pîslar, D\. Szepesvari, G\. Ostrovski, D\. Borsa, and T\. SchaulWhen should agents explore?\.InInternational Conference on Learning Representations \(ICLR\),External Links:2108\.11811,[Link](https://openreview.net/forum?id=Ewfxf14bca)Cited by:[§2](https://arxiv.org/html/2610.28779#S2.p3.1)\.
- Rajeswaranet al\.\(2018\)A\. Rajeswaran, V\. Kumar, A\. Gupta, G\. Vezzani, J\. Schulman, E\. Todorov, and S\. LevineLearning complex dexterous manipulation with deep reinforcement learning and demonstrations\.InRobotics: Science and Systems \(RSS\),Cited by:[§1](https://arxiv.org/html/2610.28779#S1.p4.1),[Figure 2](https://arxiv.org/html/2610.28779#S3.F2),[Table 1](https://arxiv.org/html/2610.28779#S3.T1),[§4](https://arxiv.org/html/2610.28779#S4.p1.1),[§4](https://arxiv.org/html/2610.28779#S4.p2.1),[§5\.1](https://arxiv.org/html/2610.28779#S5.SS1.p2.1),[§5\.2](https://arxiv.org/html/2610.28779#S5.SS2.p2.1)\.
- Schrittwieseret al\.\(2020\)J\. Schrittwieser, I\. Antonoglou, T\. Hubert, K\. Simonyan, L\. Sifre, S\. Schmitt, A\. Guez, E\. Lockhart, D\. Hassabis, T\. Graepel, T\. Lillicrap, and D\. SilverMastering atari, go, chess and shogi by planning with a learned model\.Nature588\(7839\),pp\. 604–609\.Cited by:[§2](https://arxiv.org/html/2610.28779#S2.p1.1)\.
- Seitzeret al\.\(2021\)M\. Seitzer, B\. Schölkopf, and G\. MartiusCausal influence detection for improving efficiency in reinforcement learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2610.28779#S2.p3.1)\.
- Sekaret al\.\(2020\)R\. Sekar, O\. Rybkin, K\. Daniilidis, P\. Abbeel, D\. Hafner, and D\. PathakPlanning to explore via self\-supervised world models\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2610.28779#S1.p3.1),[§2](https://arxiv.org/html/2610.28779#S2.p3.1),[Table 1](https://arxiv.org/html/2610.28779#S3.T1),[§4](https://arxiv.org/html/2610.28779#S4.p2.1)\.
- Sikchiet al\.\(2022\)H\. Sikchi, W\. Zhou, and D\. HeldLearning off\-policy with online planning\.InConference on Robot Learning \(CoRL\),pp\. 1622–1633\.External Links:2008\.10066Cited by:[§1](https://arxiv.org/html/2610.28779#S1.p1.1),[§2](https://arxiv.org/html/2610.28779#S2.p1.1)\.
- Simset al\.\(2024\)A\. Sims, C\. Lu, J\. Foerster, and Y\. W\. TehThe edge\-of\-reach problem in offline model\-based reinforcement learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:2402\.12527,[Link](https://arxiv.org/abs/2402.12527)Cited by:[§1](https://arxiv.org/html/2610.28779#S1.p2.1),[§2](https://arxiv.org/html/2610.28779#S2.p2.1)\.
- Sokaret al\.\(2023\)G\. Sokar, R\. Agarwal, P\. S\. Castro, and U\. EvciThe dormant neuron phenomenon in deep reinforcement learning\.InInternational Conference on Machine Learning \(ICML\),Note:arXiv:2302\.12902Cited by:[§2](https://arxiv.org/html/2610.28779#S2.p3.1)\.
- Talvitie \(2017\)E\. TalvitieSelf\-correcting models for model\-based reinforcement learning\.InAAAI Conference on Artificial Intelligence \(AAAI\),External Links:1612\.06018Cited by:[§1](https://arxiv.org/html/2610.28779#S1.p2.1),[§2](https://arxiv.org/html/2610.28779#S2.p2.1)\.
- Tassaet al\.\(2018\)Y\. Tassa, Y\. Doron, A\. Muldal, T\. Erez, Y\. Li, D\. de Las Casas, D\. Budden, A\. Abdolmaleki, J\. Merel, A\. Lefrancq, T\. Lillicrap, and M\. RiedmillerDeepMind control suite\.Note:arXiv:1801\.00690Cited by:[§1](https://arxiv.org/html/2610.28779#S1.p4.1),[Figure 2](https://arxiv.org/html/2610.28779#S3.F2),[Table 1](https://arxiv.org/html/2610.28779#S3.T1),[§4](https://arxiv.org/html/2610.28779#S4.p1.1),[§4](https://arxiv.org/html/2610.28779#S4.p2.1),[§5\.1](https://arxiv.org/html/2610.28779#S5.SS1.p3.1),[§5\.2](https://arxiv.org/html/2610.28779#S5.SS2.p3.1)\.
- Vlontzoset al\.\(2023\)A\. Vlontzos, B\. Kainz, and C\. M\. Gilligan\-LeeEstimating categorical counterfactuals via deep twin networks\.Nature Machine Intelligence5,pp\. 159–168\.External Links:2109\.01904,[Link](https://arxiv.org/abs/2109.01904)Cited by:[§2](https://arxiv.org/html/2610.28779#S2.p3.1)\.
- Williamset al\.\(2016\)G\. Williams, P\. Drews, B\. Goldfain, J\. M\. Rehg, and E\. A\. TheodorouAggressive driving with model predictive path integral control\.InIEEE International Conference on Robotics and Automation \(ICRA\),pp\. 1433–1440\.External Links:[Document](https://dx.doi.org/10.1109/ICRA.2016.7487277),[Link](https://doi.org/10.1109/ICRA.2016.7487277)Cited by:[§3\.1](https://arxiv.org/html/2610.28779#S3.SS1.p1.2)\.
- Xuet al\.\(2024\)G\. Xu, R\. Zheng, Y\. Liang, X\. Wang, Z\. Yuan, T\. Ji, Y\. Luo, X\. Liu, J\. Yuan, P\. Hua, S\. Li, Y\. Ze, H\. Daumé III, F\. Huang, and H\. XuDrM: mastering visual reinforcement learning through dormant ratio minimization\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2610.28779#S2.p3.1)\.
- Yeet al\.\(2021\)W\. Ye, S\. Liu, T\. Kurutach, P\. Abbeel, and Y\. GaoMastering atari games with limited data\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:2111\.00210Cited by:[§2](https://arxiv.org/html/2610.28779#S2.p1.1)\.
- Yuet al\.\(2020a\)T\. Yu, D\. Quillen, Z\. He, R\. Julian, K\. Hausman, C\. Finn, and S\. LevineMeta\-world: a benchmark and evaluation for multi\-task and meta reinforcement learning\.InConference on Robot Learning \(CoRL\),External Links:1910\.10897,[Link](https://arxiv.org/abs/1910.10897)Cited by:[§1](https://arxiv.org/html/2610.28779#S1.p4.1),[Figure 2](https://arxiv.org/html/2610.28779#S3.F2),[Table 1](https://arxiv.org/html/2610.28779#S3.T1),[§4](https://arxiv.org/html/2610.28779#S4.p1.1),[§4](https://arxiv.org/html/2610.28779#S4.p2.1),[§5\.1](https://arxiv.org/html/2610.28779#S5.SS1.p1.1),[§5\.2](https://arxiv.org/html/2610.28779#S5.SS2.p1.1),[§5\.4](https://arxiv.org/html/2610.28779#S5.SS4.SSS0.Px1.p1.1)\.
- Yuet al\.\(2020b\)T\. Yu, G\. Thomas, L\. Yu, S\. Ermon, J\. Y\. Zou, S\. Levine, C\. Finn, and T\. MaMOPO: model\-based offline policy optimization\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:2005\.13239Cited by:[§2](https://arxiv.org/html/2610.28779#S2.p2.1)\.
- Zhouet al\.\(2025\)G\. Zhou, H\. Pan, Y\. LeCun, and L\. PintoDINO\-WM: world models on pre\-trained visual features enable zero\-shot planning\.InInternational Conference on Machine Learning \(ICML\),External Links:2411\.04983,[Link](https://arxiv.org/abs/2411.04983)Cited by:[§2](https://arxiv.org/html/2610.28779#S2.p1.1)\.

## Limitations

#### What the actor reference measures\.

Repeated actor rollouts quantify stochastic return variation for one fixed imagined sequence; they do not capture model bias shared by all rollouts\. The acceptance thresholdκ\\kappais an empirical operating point rather than a statistical confidence level\. Restored\-state experiments show that the normalized margin ranks the frequency of realized improvement, but even its highest decile improves on the actor in fewer than half of paired decisions\. The score should therefore not be interpreted as a calibrated probability, despite the better aggregate outcomes of gated than ungated continuations\.

#### Planning cost and simulator access\.

SDI reuses the base world model, actor, and critic, but search and fresh re\-scoring add acting\-time computation even when a proposal is rejected\. The paired\-state analysis also requires simulator\-state restoration and common random numbers; these are analysis tools and are not used during training\. The experiments measure finite continuations from frozen checkpoints and do not isolate the long\-term learning contribution of each accepted or rejected intervention\.

## Appendix AMethod details

### A\.1Search and rescoring

The planner samplesN=256N=256candidate sequences over horizonH=8H=8, withI=4I=4refinement iterations,E=32E=32elites, andR=4R=4independent stochastic rollouts per candidate\. Search temperature is1\.01\.0; initial and minimum action standard deviations are\.3\.3and\.05\.05\. Actions are clipped to the environment’s bounds\. At each iteration,

a\(n\)\\displaystyle a^\{\(n\)\}=clip⁡\(μ\+σ⊙ϵ\(n\),amin,amax\),ϵ\(n\)∼𝒩⁡\(0,I\),\\displaystyle=\\mathrm\{clip\}\(\\mu\+\\sigma\\odot\\epsilon^\{\(n\)\},a\_\{\\min\},a\_\{\\max\}\),\\qquad\\epsilon^\{\(n\)\}\\sim\\mathcal\{N\}\(0,I\),\(4\)G^\(n\)\\displaystyle\\hat\{G\}^\{\(n\)\}=1R​∑j=1RGj​\(a\(n\)\),ζ\(n\)=G^\(n\)−meann​G^\(n\)sdn​G^\(n\),\\displaystyle=\\frac\{1\}\{R\}\\sum\_\{j=1\}^\{R\}G\_\{j\}\(a^\{\(n\)\}\),\\qquad\\zeta^\{\(n\)\}=\\frac\{\\hat\{G\}^\{\(n\)\}\-\\mathrm\{mean\}\_\{n\}\\hat\{G\}^\{\(n\)\}\}\{\\mathrm\{sd\}\_\{n\}\\hat\{G\}^\{\(n\)\}\},\(5\)w\(e\)\\displaystyle w^\{\(e\)\}=softmaxe∈elites​\(ζ\(e\)/λ\),μ←∑ew\(e\)​a\(e\),\\displaystyle=\\mathrm\{softmax\}\_\{e\\in\\mathrm\{elites\}\}\(\\zeta^\{\(e\)\}/\\lambda\),\\qquad\\mu\\leftarrow\\sum\_\{e\}w^\{\(e\)\}a^\{\(e\)\},\(6\)σ2\\displaystyle\\sigma^\{2\}←clip⁡\(∑ew\(e\)​\(a\(e\)−μ\)2,σmin2,σmax2\)\.\\displaystyle\\leftarrow\\mathrm\{clip\}\\\!\\left\(\\sum\_\{e\}w^\{\(e\)\}\(a^\{\(e\)\}\-\\mu\)^\{2\},\\sigma\_\{\\min\}^\{2\},\\sigma\_\{\\max\}^\{2\}\\right\)\.\(7\)Scores are standardized separately for each environment\. The initial mean follows the actor’s imagined mode sequence\. On subsequent acting steps the previous sequence is shifted forward; episode boundaries reset this initialization\.

After search, the plan and the actor’s mode sequence receiveRg=max⁡\(R,4\)=4R\_\{g\}=\\max\(R,4\)=4fresh rollouts each\. The gate compares their mean returns using the standard deviation of those actor rollouts\. It executes only the first planned action; rejection falls back to an action sampled from the actor\. The reference used for scoring is therefore the actor’s fixed mode sequence, whereas the training\-time fallback is stochastic\. Evaluation uses the actor’s mode alone\.

#### Independent model noise\.

Candidate rollouts, repeated actor rollouts, and fresh re\-scoring all use independent model noise\. Exploration uses a separate batch with its own rollout settings\. Sharing the starting latent state does not imply sharing the model’s random draws\. In contrast, the executed\-alternative studies restore simulator state and use common random numbers where supported\. These are evaluation procedures, not additional simulator access required by SDI during learning\.