Dueling World Models: Advantage-Style Action Channels for Common-Mode Distractor Rejection

# Advantage-Style Action Channels for Common-Mode Distractor Rejection
Source: [https://arxiv.org/html/2608.06706](https://arxiv.org/html/2608.06706)
###### Abstract

Latent world models plan by predicting future states from an action, but when a scene contains motion the agent does not control, they quietly go action\-blind: predictions for different actions become indistinguishable even as the training loss keeps improving\. Existing remedies suppress this distraction with reconstruction, task reward, or auxiliary objectives, each adding machinery or assumptions\. We show that a minimal alternative suffices, borrowed from the dueling decomposition of value into a state baseline and an action advantage: in latent dynamics, subtracting a prediction’s mean effect over actions cancels whatever the actions share—the action\-independent variation where distractors live—leaving a clean, controllable channel, with no reward, no reconstruction, and no distractor\-specific auxiliary loss\. Because this is only a subtraction at readout time, it applies unchanged to any action\-conditioned world model, including frozen pretrained ones\. Across a gridworld, synthetic generators with known factors, distracting continuous control, and natural\-pixel Atari, the isolated channel recovers the agent’s own effect where entangled predictors fail, with nuisance leak indistinguishable from zero; applied post hoc it surfaces an action channel in off\-the\-shelf models that their raw readouts miss, and it converts into goal\-reaching control in the gridworld\. We prove the cancellation is exact in finite samples for both discrete and sampled action sets, and we state its measured boundary—distractors whose motion tracks the action—together with the remaining limitations in the appendix\.

## 1Introduction

Latent world models\(Ha and Schmidhuber[2018](https://arxiv.org/html/2608.06706#bib.bib53); Hafneret al\.[2023](https://arxiv.org/html/2608.06706#bib.bib54)\)increasingly follow the joint\-embedding predictive architecture \(JEPA\) recipe\(LeCun[2022](https://arxiv.org/html/2608.06706#bib.bib35)\): encode observations, predict future*embeddings*rather than pixels, and plan in the learned latent space\(Assranet al\.[2025](https://arxiv.org/html/2608.06706#bib.bib7); Zhouet al\.[2025](https://arxiv.org/html/2608.06706#bib.bib8); Maeset al\.[2026](https://arxiv.org/html/2608.06706#bib.bib5)\)\. Under action\-conditioned instances, one monolithic predictorP​\(z,a\)P\(z,a\)absorbs everything predictable about the next embedding: the consequences of the agent’s action, but also the autonomous evolution of whatever else fills the frame\. That distraction degrades learned action and dynamics channels is by now established—latent actions absorb distractor dynamics unless grounded by true action labels\(Nikulinet al\.[2025](https://arxiv.org/html/2608.06706#bib.bib9)\)or purged with external segmentation masks\(Fechneret al\.[2026](https://arxiv.org/html/2608.06706#bib.bib10)\)\. Existing remedies buy the separation with extra machinery: reconstruction and reward\-based factorisation\(Fuet al\.[2021](https://arxiv.org/html/2608.06706#bib.bib23); Wanget al\.[2022](https://arxiv.org/html/2608.06706#bib.bib24)\), separate controllable and uncontrollable branches\(Panet al\.[2022](https://arxiv.org/html/2608.06706#bib.bib16)\), bisimulation\-style objectives\(Zhuet al\.[2023](https://arxiv.org/html/2608.06706#bib.bib26)\), or an auxiliary inverse\-dynamics incentive on a monolithic predictor\(Ivashkovet al\.[2026](https://arxiv.org/html/2608.06706#bib.bib4)\)\. Each adds a loss, a decoder, or an assumption about the distractor process, and none isolates the action channel as a property of the predictor itself\. Under action\-labeled circumstances, we ask the mechanism question: what structure inside the predictor makes the given actions count?

We first quantify the failure\. In a13×1313\{\\times\}13FourRooms gridworld we addnnrolling binary distractor cells that are action\-independent and track*action separation*\(AS\): the mean pairwise distance between the predictor’s outputs for different actions at the same state\. Asnngrows to3030, AS collapses from≈1\.28\{\\approx\}1\.28to≈0\.002\{\\approx\}0\.002—for*every*architecture we train, including ours \(Fig\.[2](https://arxiv.org/html/2608.06706#S4.F2)\): the one\-step prediction is dominated by the distractor field, and the action’s contribution shrinks below numerical relevance\. Probing shows the information is not lost, as the action’s effect remains recoverable from the representation\. Yet the predictor simply stops routing it anywhere readable\. The failure is silent \(embedding\-prediction validation loss keeps improving\) but not benign: goal\-reaching control with the standard predictor degrades sharply, and standard model selection actively prefers the pathology—on Atari Freeway, validation\-loss checkpoint selection returned action\-collapsed checkpoints in 17 of 36 runs\.

Therefore, we propose a fix that restructures the predictor into two streams: a passive stream that predicts how the scene evolves with the action marginalised out, and an action stream that is*centered*, in the sense that the mean prediction over actions is subtracted from it before it is added back\. That subtraction is an algebraic identity rather than a learned tendency: whatever enters the action stream*identically across actions*—common\-mode variation, including whatever action\-independent distractors contribute—is removed exactly, leaving only what distinguishes one action from another\. It survives estimation, staying exact when the mean is taken over sampled actions rather than enumerated \(Sec\.[3](https://arxiv.org/html/2608.06706#S3)\)\. This is a claim about*per\-transition common\-mode structure*, not about the distractor process: it needs no process\-level exogeneity assumption\(Efroniet al\.[2022](https://arxiv.org/html/2608.06706#bib.bib18)\), no reward, and no reconstruction\(Fuet al\.[2021](https://arxiv.org/html/2608.06706#bib.bib23); Wanget al\.[2022](https://arxiv.org/html/2608.06706#bib.bib24); Panet al\.[2022](https://arxiv.org/html/2608.06706#bib.bib16)\)\. Whether real distractors actually enter as common mode is an empirical question; our results across gridworld and Atari indicate that they overwhelmingly do\. The subtraction is Dueling DQN’s identityQ=V\+\(A−A¯\)Q=V\+\(A\-\\bar\{A\}\)\(Wanget al\.[2016](https://arxiv.org/html/2608.06706#bib.bib1)\)transplanted from scalar values to vector latent dynamics, where centering—an identifiability device in the original—acquires a second role: common\-mode rejection\. The action\-marginal baseline itself appears inSeitzeret al\.\([2021](https://arxiv.org/html/2608.06706#bib.bib2)\)as a post\-hoc scalar influence score for exploration, and concurrent work\(Ivashkovet al\.[2026](https://arxiv.org/html/2608.06706#bib.bib4)\)pursues the same distractor\-discarding outcome through an inverse\-dynamics auxiliary loss on a monolithic predictor\. Our claim is the readout, not the architecture: a vector\-valued, common\-mode–invariant action channel that centering exposes in any action\-conditioned predictor—post hoc on frozen models, or embedded in the parameterization, where its invariance needs no auxiliary objective\. We call the resulting model AD\-JEPA \(Action\-Decomposed JEPA\)\.111Not to be confused with AD\-L\-JEPA\(Zhuet al\.[2026](https://arxiv.org/html/2608.06706#bib.bib34)\), a self\-supervised LiDAR pre\-training method for autonomous driving; the acronym collision is coincidental—there “AD” abbreviates*autonomous driving*, here*action\-decomposed*\.

The same scoping that gives the guarantee defines its limit: distractors whose dynamics correlate with the agent’s actions are not common mode and are not cancelled\. We construct such cases and measure the failure—offset probes collapse to chance and nuisance rejection breaks—consistent with the published stress cases, reafferent distractors\(Hutsonet al\.[2024](https://arxiv.org/html/2608.06706#bib.bib28)\)and agent\-like distractors\(Wanget al\.[2024](https://arxiv.org/html/2608.06706#bib.bib22)\); incentive\-based separation concedes the same failure mode\(Ivashkovet al\.[2026](https://arxiv.org/html/2608.06706#bib.bib4)\)\. We report this boundary alongside every positive result\. We also report a negative result on our own ladder: a learned gate stacked on centering is inert \(it degenerates to uniform down\-scaling rather than sparse selection\), so centering alone is the mechanism\.

#### Contributions\.

- •We propose*action\-mean centering*, a one\-line restructuring of an action\-conditioned latent predictor that makes its action channel common\-mode–invariant, and we analyse it: the cancellation is exact in finite samples for discrete and sampled action sets alike \(Propositions[1](https://arxiv.org/html/2608.06706#Thmproposition1)–[2](https://arxiv.org/html/2608.06706#Thmproposition2)\), with a precisely scoped boundary—action\-correlated variation passes through untouched\.
- •We give an implementation and validate it across four settings—a gridworld, synthetic generators with known factors, distracting continuous control, and natural\-pixel Atari—where the centered channel preserves the agent’s own effect with nuisance leak indistinguishable from zero, and converts into goal\-reaching control in the gridworld\.
- •Because the mechanism lives in the readout rather than the architecture, it applies to models we did not train: the same subtraction surfaces an action channel in frozen RePo and TIA world models whose raw readouts show none, which suggests such a channel is latent in action\-conditioned predictors generally and merely unrouted \(Sec\.[4\.4](https://arxiv.org/html/2608.06706#S4.SS4)\)\.

All results use three seeds unless a caption states otherwise \(frozen\-host cells: one training run each, readout±\\pms\.d\. over five probe seeds\)\.

## 2Related Work

Our contribution intersects three lines of work: advantage\-style decompositions that center predictions on an action marginal, latent\-action and JEPA world models, and factored world models that separate controllable from exogenous dynamics\. That distractors contaminate the action channels of learned dynamics models is by now an established problem\(Nikulinet al\.[2025](https://arxiv.org/html/2608.06706#bib.bib9); Fechneret al\.[2026](https://arxiv.org/html/2608.06706#bib.bib10)\); we claim a mechanism, not the problem statement\.

Dueling DQN\(Wanget al\.[2016](https://arxiv.org/html/2608.06706#bib.bib1)\)writesQ=V\+\(A−A¯\)Q=V\+\(A\-\\bar\{A\}\), where subtracting the action mean is purely an identifiability device for scalar values\. Transplanted to vector\-valued latent dynamics, the same subtraction acquires a second semantics: any variation entering the per\-action offsets*identically across actions*\(common\-mode\) cancels exactly, in the discrete and the sampled continuous case alike \(§[3](https://arxiv.org/html/2608.06706#S3)\)\. The arithmetic—comparing action\-conditioned predictions with their mean over sampled actions—is CAI’s\(Seitzeret al\.[2021](https://arxiv.org/html/2608.06706#bib.bib2)\), applied there post hoc as a scalar causal\-influence score for exploration; the Feedback World Model\(Anet al\.[2026](https://arxiv.org/html/2608.06706#bib.bib3)\)distills counterfactual variation over sampled actions into per\-dimension controllability weights, estimated offline and applied only during diffusion\-policy guidance\. Our claim is the promotion of this baseline from scalar diagnostic to channel: a vector\-valued, common\-mode–invariant action readout that applies post hoc to frozen predictors and, optionally, embeds in the parameterization—where it exists throughout learning and adds no auxiliary objective\.

Action\-conditioned JEPA world models—LeWM\(Maeset al\.[2026](https://arxiv.org/html/2608.06706#bib.bib5)\)with LeJEPA\-style regularization\(Balestriero and LeCun[2025](https://arxiv.org/html/2608.06706#bib.bib6)\), V\-JEPA 2\-AC\(Assranet al\.[2025](https://arxiv.org/html/2608.06706#bib.bib7)\), and DINO\-WM\(Zhouet al\.[2025](https://arxiv.org/html/2608.06706#bib.bib8)\)—train or post\-train a single monolithic predictorP​\(z,a\)P\(z,a\)in which controllable and exogenous dynamics remain entangled\. SMWM\(Ivashkovet al\.[2026](https://arxiv.org/html/2608.06706#bib.bib4)\)shares our outcome claim—a reward\- and reconstruction\-free JEPA world model that discards uncontrollable distractors—but pursues it by incentive, an inverse\-dynamics auxiliary loss on a monolithic transformer; our separation is structural, an identity of the parameterization rather than a learned tendency\. In the latent\-action literature, LAPO\(Schmidt and Jiang[2024](https://arxiv.org/html/2608.06706#bib.bib13)\), Genie\(Bruceet al\.[2024](https://arxiv.org/html/2608.06706#bib.bib14)\), and DynaMo\(Cuiet al\.[2024](https://arxiv.org/html/2608.06706#bib.bib15)\)recover actions through inverse/forward\-dynamics bottlenecks but leave the forward predictor undecomposed; LAOM\(Nikulinet al\.[2025](https://arxiv.org/html/2608.06706#bib.bib9)\)shows observation\-only latent actions require action supervision under distractors—precisely our action\-labeled regime, where centering isolates the controllable channel with no further signals\. MaskLAM\(Fechneret al\.[2026](https://arxiv.org/html/2608.06706#bib.bib10)\)purges distractors with external segmentation masks plus reconstruction, presupposing spatial separability\. AC\-LAM\(Weiet al\.[2026](https://arxiv.org/html/2608.06706#bib.bib11)\)is the nearest structural relative: an additivity prior forbids scene\-constant offsets in inferred pseudo\-actions, but as a soft constraint with no action\-marginal baseline; centering cancels*any*common\-mode component on true actions, as a parameterization\. PLSM\(Saanumet al\.[2024](https://arxiv.org/html/2608.06706#bib.bib12)\)regularizes how the transition depends on the state; we decompose how it depends on the action—orthogonal axes\.

TIA\(Fuet al\.[2021](https://arxiv.org/html/2608.06706#bib.bib23)\)and Denoised MDPs\(Wanget al\.[2022](https://arxiv.org/html/2608.06706#bib.bib24)\)factor the latent state using reward, reconstruction, and independence structure—the three dependencies centering removes\. Iso\-Dream and Iso\-Dream\+\+\(Panet al\.[2022](https://arxiv.org/html/2608.06706#bib.bib16),[2023](https://arxiv.org/html/2608.06706#bib.bib17)\)train separate controllable/noncontrollable RSSM branches with reconstruction and inverse\-dynamics losses: a learned, incentive\-based split where ours is identity\-level cancellation\. The Ex\-BMDP line\(Efroniet al\.[2022](https://arxiv.org/html/2608.06706#bib.bib18); Lambet al\.[2022](https://arxiv.org/html/2608.06706#bib.bib19); Islamet al\.[2023](https://arxiv.org/html/2608.06706#bib.bib20); Levineet al\.[2024](https://arxiv.org/html/2608.06706#bib.bib21)\)proves exogenous filtering under process\-level exogeneity and finite\-state conditions; we require only per\-transition common\-mode structure—an architectural identity per transition, not an asymptotic recovery result\. DreamerPro\(Denget al\.[2022](https://arxiv.org/html/2608.06706#bib.bib25)\), RePo\(Zhuet al\.[2023](https://arxiv.org/html/2608.06706#bib.bib26)\), and HRSSM\(Sunet al\.[2024](https://arxiv.org/html/2608.06706#bib.bib27)\)obtain reconstruction\-free or bisimulation\-style robustness without an explicit controllability channel \(the latter two reward\-dependent\), and InfoPower\(Bharadhwajet al\.[2022](https://arxiv.org/html/2608.06706#bib.bib29)\)contrasts action\-conditioned against action\-marginal information inside reward\-driven training—a variational cousin of centering\. Earlier controllability partitions separate factors via per\-factor policies or adversarial losses\(Thomaset al\.[2017](https://arxiv.org/html/2608.06706#bib.bib30),[2018](https://arxiv.org/html/2608.06706#bib.bib31); Sawada[2018](https://arxiv.org/html/2608.06706#bib.bib32); Kooiet al\.[2022](https://arxiv.org/html/2608.06706#bib.bib33)\); ours is algebraic\. Finally, two works delimit the boundary of our guarantee: AD3\(Wanget al\.[2024](https://arxiv.org/html/2608.06706#bib.bib22)\)generatively infers implicit actions for agent\-like distractors, and Policy\-Shaped Prediction\(Hutsonet al\.[2024](https://arxiv.org/html/2608.06706#bib.bib28)\)introduces the Reafferent DMC benchmark of action\-correlated distractors\. Both exemplify the action\-modulated regime that common\-mode cancellation explicitly excludes—a failure mode SMWM concedes as well—and we return to it in §[5\.1](https://arxiv.org/html/2608.06706#S5.SS1)\.

## 3Method

![Refer to caption](https://arxiv.org/html/2608.06706v1/x1.png)Figure 1:Centering as a universal plug\-in\.\(a\) For any action\-conditioned world model—our JEPA variants, an RSSM prior, or a frozen third\-party host—subtracting the mean prediction over candidate actions yields the effective action channelu​\(z,a\)u\(z,a\): a readout, applied post hoc with no retraining\. \(b\) What “distraction” means \(FourRooms\): distractor cells move identically under every action \(common\-mode\), so they enterΔ¯\\bar\{\\Delta\}and cancel; the agent’s next cell depends on the action and survives inuu\. Action\-correlated distractors break the premise—the stated boundary\.AD\-JEPA learns a latent world model from reward\-free, action\-labeled transitions\(ot,at,ot\+1\)\(o\_\{t\},a\_\{t\},o\_\{t\+1\}\), with no decoder and no reconstruction\. The design goal is not to remove distractor information from the representation, but to control*where*it lives: the predictor is restructured so that its action\-dependent pathway cannot carry variation that is common across actions\.

### 3\.1JEPA Backbone

Following the joint\-embedding predictive architecture\(LeCun[2022](https://arxiv.org/html/2608.06706#bib.bib35)\), an online encoder maps observations to latentszt∈ℝdz\_\{t\}\\in\\mathbb\{R\}^\{d\}and a predictor maps\(zt,at\)\(z\_\{t\},a\_\{t\}\)toz^t\+1\\hat\{z\}\_\{t\+1\}; targets come from an EMA target encoder held fixed by a stop\-gradient, as in BYOL\(Grillet al\.[2020](https://arxiv.org/html/2608.06706#bib.bib39)\)\. Action\-conditioned JEPA world models\(Zhouet al\.[2025](https://arxiv.org/html/2608.06706#bib.bib8); Assranet al\.[2025](https://arxiv.org/html/2608.06706#bib.bib7)\)implement the predictor as one monolithic networkP​\(zt,at\)P\(z\_\{t\},a\_\{t\}\); nothing in that parameterization separates what the action changes from what would have happened anyway, and under distractors the two entangle freely\. AD\-JEPA is a drop\-in restructuring of this predictor\.

### 3\.2Action\-Decomposed Predictor

We decompose the predictor into a passive head and a centered action\-offset head:

z^t\+1=B​\(zt\)\+\[Δ​\(zt,at\)−Δ¯​\(zt\)\]⏟u​\(zt,at\),\\hat\{z\}\_\{t\+1\}\\;=\\;B\(z\_\{t\}\)\\;\+\\;\\underbrace\{\\bigl\[\\Delta\(z\_\{t\},a\_\{t\}\)\-\\bar\{\\Delta\}\(z\_\{t\}\)\\bigr\]\}\_\{u\(z\_\{t\},\\,a\_\{t\}\)\},\(1\)whereΔ¯​\(z\)=𝔼a′∼ν​\[Δ​\(z,a′\)\]\\bar\{\\Delta\}\(z\)=\\mathbb\{E\}\_\{a^\{\\prime\}\\sim\\nu\}\\\!\\left\[\\Delta\(z,a^\{\\prime\}\)\\right\]is the mean offset under a proposalν\\nuover actions\. The passive headBBpredicts how the scene evolves with the action marginalized out; the*effective action channel*u​\(z,a\)=Δ​\(z,a\)−Δ¯​\(z\)u\(z,a\)=\\Delta\(z,a\)\-\\bar\{\\Delta\}\(z\)carries only the action\-contrastive part of the dynamics\. \(The*gated*ablation of Sec\.[4\.1](https://arxiv.org/html/2608.06706#S4.SS1)additionally scales the channel elementwise by a learnedC​\(z\)∈\(0,1\)dC\(z\)\\in\(0,1\)^\{d\}; Appendix[F](https://arxiv.org/html/2608.06706#A6)shows this gate degenerates to uniform down\-scaling, and we report it as a negative result\.\)

The construction transplants the dueling decompositionQ=V\+\(A−meana⁡A\)Q=V\+\(A\-\\operatorname\{mean\}\_\{a\}A\)ofWanget al\.\([2016](https://arxiv.org/html/2608.06706#bib.bib1)\)from scalar values to vector latent dynamics, where the subtraction acquires a second semantics\. As in dueling Q\-learning it is an identifiability device—for anyc​\(z\)c\(z\), replacingΔ↦Δ\+c\\Delta\\mapsto\\Delta\+cleavesuuunchanged, so the channel is well defined thoughΔ\\Deltaalone is not—*and*it performs exact common\-mode rejection \(Proposition[1](https://arxiv.org/html/2608.06706#Thmproposition1)\)\. Without it, the same degeneracy lets gradient descent park arbitrary state\- and distractor\-dependent variation in the offsets, and empirically it does \(Secs\.[4\.3](https://arxiv.org/html/2608.06706#S4.SS3),[4\.2](https://arxiv.org/html/2608.06706#S4.SS2)\)\.

#### Estimating the action mean\.

For discrete action sets𝒜\\mathcal\{A\}we takeν\\nuuniform and enumerate:Δ¯​\(z\)=1\|𝒜\|​∑a′∈𝒜Δ​\(z,a′\)\\bar\{\\Delta\}\(z\)=\\frac\{1\}\{\|\\mathcal\{A\}\|\}\\sum\_\{a^\{\\prime\}\\in\\mathcal\{A\}\}\\Delta\(z,a^\{\\prime\}\), computed exactly at every step\. For continuous actions, enumeration is unavailable andΔ¯\\bar\{\\Delta\}is replaced by a Monte\-Carlo mean overK=16K\{=\}16candidate actions: the executed action, with the remainingK−1K\{\-\}1split evenly between actions resampled from the replay buffer and Gaussian perturbations of the executed action \(standard deviation0\.30\.3of the action half\-range, clipped to the action bounds\)\. Proposition[2](https://arxiv.org/html/2608.06706#Thmproposition2)\(Sec\.[3\.4](https://arxiv.org/html/2608.06706#S3.SS4)\) shows this estimate leaves the cancellation untouched—exact at any finiteKK—and confines theO​\(1/K\)O\(1/\\sqrt\{K\}\), proposal\-dependent sampling error to the*centering point*, an action\-independent shift; only under discrete enumeration is the centering point itself exact\. The conditioning ofΔ¯\\bar\{\\Delta\}onzzis load\-bearing: on frozen hosts, subtracting the*global*mean offset instead leaves the channel unrecovered \(Appendix[G\.1](https://arxiv.org/html/2608.06706#A7.SS1)\)—the common mode being cancelled is state\-conditional\. The arithmetic of comparing action\-conditioned predictions against their sampled mean is CAI’s\(Seitzeret al\.[2021](https://arxiv.org/html/2608.06706#bib.bib2)\), applied there as a post\-hoc scalar influence score; AD\-JEPA promotes it to a trained, vector\-valued channel centered at every forward pass\.

### 3\.3Training Objective

The model is trained with

ℒ=ℒpred\+λact​ℒact\+λoff​ℒoff\+λreg​ℒreg​\(\+λgate​ℒgate\),\\begin\{split\}\\mathcal\{L\}=\{\}&\\mathcal\{L\}\_\{\\mathrm\{pred\}\}\+\\lambda\_\{\\mathrm\{act\}\}\\,\\mathcal\{L\}\_\{\\mathrm\{act\}\}\+\\lambda\_\{\\mathrm\{off\}\}\\,\\mathcal\{L\}\_\{\\mathrm\{off\}\}\\\\ &\{\}\+\\lambda\_\{\\mathrm\{reg\}\}\\,\\mathcal\{L\}\_\{\\mathrm\{reg\}\}\\;\(\+\\,\\lambda\_\{\\mathrm\{gate\}\}\\,\\mathcal\{L\}\_\{\\mathrm\{gate\}\}\),\\end\{split\}\(2\)with the gate term active only in the gated variant\.ℒpred=2−2​cos⁡\(z^t\+1,sg⁡\[z~t\+1\]\)\\mathcal\{L\}\_\{\\mathrm\{pred\}\}=2\-2\\cos\\\!\\bigl\(\\hat\{z\}\_\{t\+1\},\\operatorname\{sg\}\[\\tilde\{z\}\_\{t\+1\}\]\\bigr\)is the scale\-free JEPA prediction loss against the stop\-gradient EMA target\.ℒact\\mathcal\{L\}\_\{\\mathrm\{act\}\}is an InfoNCE term\(van den Oordet al\.[2018](https://arxiv.org/html/2608.06706#bib.bib40)\)over theKKcounterfactual\-action predictions of Eq\. \([1](https://arxiv.org/html/2608.06706#S3.E1)\): a cross\-entropy over cosine similarities to the target requires the executed action’s prediction to be closest—penalizing action\-insensitive predictors\.ℒoff\\mathcal\{L\}\_\{\\mathrm\{off\}\}is a compactness penalty, the squared norm of the executed action’s effective offset, discouraging the channel from absorbing state persistence \(λoff=10−4\\lambda\_\{\\mathrm\{off\}\}=10^\{\-4\}\)\.ℒreg\\mathcal\{L\}\_\{\\mathrm\{reg\}\}is a VICReg\-style anti\-collapse regularizer on the online embeddings—batch\-mean centering, a variance hinge, and a covariance penalty\(Bardeset al\.[2022](https://arxiv.org/html/2608.06706#bib.bib38)\)—withλreg=0\.1\\lambda\_\{\\mathrm\{reg\}\}=0\.1\. For the gated variant,ℒgate\\mathcal\{L\}\_\{\\mathrm\{gate\}\}is an L1 penalty on the gate values \(λgate=10−5\\lambda\_\{\\mathrm\{gate\}\}=10^\{\-5\}\), held at zero for the first 20% of training and then ramped in\. All variants train on fixed offline datasets under identical schedules; data collection, architectures, remaining hyperparameters, and the action\-separation–gated checkpoint\-selection protocol are in Appendix[A](https://arxiv.org/html/2608.06706#A1)\.

### 3\.4Common\-Mode Cancellation

The property we claim for Eq\. \([1](https://arxiv.org/html/2608.06706#S3.E1)\) is an identity of the parameterization, not an incentive supplied by a loss\.

###### Proposition 1\(Common\-mode cancellation\)\.

Fixzzand suppose the offset head decomposes asΔ​\(z,a\)=g​\(z,a\)\+h​\(z\)\\Delta\(z,a\)=g\(z,a\)\+h\(z\)for someggand some action\-independenthh\. Then, with the exact action meanΔ¯​\(z\)=1\|𝒜\|​∑a′∈𝒜Δ​\(z,a′\)\\bar\{\\Delta\}\(z\)=\\frac\{1\}\{\|\\mathcal\{A\}\|\}\\sum\_\{a^\{\\prime\}\\in\\mathcal\{A\}\}\\Delta\(z,a^\{\\prime\}\), for every actionaa

Δ​\(z,a\)−Δ¯​\(z\)=g​\(z,a\)−1\|𝒜\|​∑a′g​\(z,a′\):\\Delta\(z,a\)\-\\bar\{\\Delta\}\(z\)\\;=\\;g\(z,a\)\-\\tfrac\{1\}\{\|\\mathcal\{A\}\|\}\\textstyle\\sum\_\{a^\{\\prime\}\}g\(z,a^\{\\prime\}\):the common\-mode componenthhcancels exactly, for every value of the parameters and regardless of whathhencodes\.

###### Proof\.

h​\(z\)h\(z\)appears once inΔ​\(z,a\)\\Delta\(z,a\)and once in every term of the mean, so it is subtracted exactly\. ∎

The same holds for the sampled estimate of the continuous regime:

###### Proposition 2\(Sampled centering\)\.

LetΔ¯K​\(z\)=1K​∑k=1KΔ​\(z,ak\)\\bar\{\\Delta\}\_\{K\}\(z\)=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\Delta\(z,a\_\{k\}\)for candidate actions\{ak\}k=1K\\\{a\_\{k\}\\\}\_\{k=1\}^\{K\}drawn from a proposalν\\nu, and supposeΔ​\(z,a\)=g​\(z,a\)\+h​\(z\)\\Delta\(z,a\)=g\(z,a\)\+h\(z\)as above\. Then, for everyKKand every draw: \(i\)Δ​\(z,a\)−Δ¯K​\(z\)=g​\(z,a\)−1K​∑kg​\(z,ak\)\\Delta\(z,a\)\-\\bar\{\\Delta\}\_\{K\}\(z\)=g\(z,a\)\-\\frac\{1\}\{K\}\\sum\_\{k\}g\(z,a\_\{k\}\)—the common\-mode componenthhcancels exactly; and \(ii\) the sampling errorΔ¯K​\(z\)−𝔼a′∼ν​\[Δ​\(z,a′\)\]\\bar\{\\Delta\}\_\{K\}\(z\)\-\\mathbb\{E\}\_\{a^\{\\prime\}\\sim\\nu\}\[\\Delta\(z,a^\{\\prime\}\)\]does not depend on the query action, isO​\(1/K\)O\(1/\\sqrt\{K\}\)under bounded second moments ofg​\(z,⋅\)g\(z,\\cdot\), and cancels in any action\-differential comparisonu​\(z,a\)−u​\(z,a′\)u\(z,a\)\-u\(z,a^\{\\prime\}\)\. \(Proof and remarks on self\-inclusion and proposal bias: Appendix[B](https://arxiv.org/html/2608.06706#A2)\.\)

Three remarks scope this statement\.*First*, the identity is per\-transition and architectural—it holds at initialization, at every step, and at convergence, with no process\-level exogeneity assumption\(Efroniet al\.[2022](https://arxiv.org/html/2608.06706#bib.bib18)\)\. Becausez^t\+1=B\+u\\hat\{z\}\_\{t\+1\}=B\+umust still match the target, common\-mode variation is routed intoBB: the model is not blind to distractors—zzmay represent them andBBmust track them—butuuhas nowhere to hold them\. Rejection is routing, not blindness \(Sec\.[4\.2](https://arxiv.org/html/2608.06706#S4.SS2)\)\.*Second*, the guarantee is exactly as strong as the common\-mode premise: action\-correlated \(reafferent\) variation entersggand passes through untouched \(Sec\.[5\.1](https://arxiv.org/html/2608.06706#S5.SS1)\); whether real distractors enter as common\-mode is empirical—Sec\.[4\.3](https://arxiv.org/html/2608.06706#S4.SS3)tests it with known factors, and Secs\.[4\.2](https://arxiv.org/html/2608.06706#S4.SS2)and[4\.2](https://arxiv.org/html/2608.06706#S4.SS2)show gridworld and Atari distractors do behave as common\-mode in practice\.*Third*, the separation is a property of the function class, not the training signal: no auxiliary objective rewards discarding uncontrollable variation, in contrast to inverse\-dynamics incentives on a monolithic predictor\(Ivashkovet al\.[2026](https://arxiv.org/html/2608.06706#bib.bib4)\)\.

## 4Experiments

We ask three questions in turn\. Does the centered channel survive distraction where entangled predictors lose it, across environments of different character? What exactly does it contain, when the ground\-truth factors are known? And is the effect a property of the readout, so that it transfers to models we did not build? Every configuration uses three seeds; tables report means, with s\.e\.m\. where error bars are shown\.

### 4\.1Setup and Evaluation Protocol

#### Variant ladder\.

All variants share the encoder, EMA targets, and losses of Sec\.[3](https://arxiv.org/html/2608.06706#S3)and differ only in the predictor:*standard*z^′=P​\(z,a\)\\hat\{z\}^\{\\prime\}=P\(z,a\);*residual*z\+Δ​\(z,a\)z\+\\Delta\(z,a\);*noncentered*B​\(z\)\+Δ​\(z,a\)B\(z\)\+\\Delta\(z,a\);*centered*\(ours\)B​\(z\)\+\[Δ​\(z,a\)−Δ¯​\(z\)\]B\(z\)\+\[\\Delta\(z,a\)\-\\bar\{\\Delta\}\(z\)\]; and*gated*, which adds a learned channel gateC​\(z\)C\(z\)and proves inert \(Appendix[F](https://arxiv.org/html/2608.06706#A6)\), so centering is the operative mechanism throughout\. The noncentered→\\,\\to\\,centered rung isolates the contribution under test\. In the discrete\-action environmentsΔ¯\\bar\{\\Delta\}is the exact uniform mean over all actions\.

#### Probing protocol\.

We measure*where*information lives, not just whether it is present: for the encoder representationzz, the raw predicted offset, and the*effective offset*\[Δ−Δ¯\]\[\\Delta\-\\bar\{\\Delta\}\], we fit ridge probes to ground\-truth factors \(5\-fold CV, held\-outR2R^\{2\}\)\. Comparing features distinguishes*routing*from*blindness*: a factor can be decodable fromzzyet absent from the effective offset\. Action separation \(AS\) is the mean pairwise distance among per\-action predictions from a common state; AS→0\{\\to\}\\,0means the predictor’s*output*ignores the action\. Because validation loss alone is an unsafe selection signal here—on Freeway, cosine\-loss selection picked action\-collapsed checkpoints in 17 of 36 runs—we gate checkpoint selection on AS with a regime\-aware threshold\. Environments, budgets, distractor knobs and all per\-experiment hyperparameters are in Appendix[A](https://arxiv.org/html/2608.06706#A1)\.

### 4\.2The Centered Channel Survives Distraction

Three environments of different character make the same point: as distraction grows, the centered offset keeps decoding the agent’s own effect while the uncentered offset is destroyed\.

![Refer to caption](https://arxiv.org/html/2608.06706v1/x2.png)Figure 2:Gridworld sweep\(nnrolling distractor cells, 3 seeds\)\.*Left:*the centered offset keeps decoding the agent’s effect while the noncentered offset collapses\.*Right:*output\-level action separation collapses for*every*variant—the channel is preserved but unused\. Legends use the code name*dueling*for centered\.#### Constructed distractors \(gridworld\)\.

In a13×1313\{\\times\}13FourRooms gridworld we addnnrolling binary distractor cells whose dynamics do not depend on the action, so per transition their contribution to the offsets is common\-mode\. Acrossn=0/10/30/60n=0/10/30/60the centered offset probes the agent factor at0\.31/0\.55/0\.73/0\.690\.31/0\.55/\\mathbf\{0\.73\}/\\mathbf\{0\.69\}while the noncentered offset falls to0\.18/0\.21/0\.00/−0\.000\.18/0\.21/0\.00/\{\-0\.00\}\(Table[7](https://arxiv.org/html/2608.06706#A10.T7), Appendix[J](https://arxiv.org/html/2608.06706#A10)\); in the clean environment the gap is modest, so the benefit is specifically common\-mode rejection rather than a generic representation improvement\. The channel is preserved but*unused*: AS collapses from≈1\.28\{\\approx\}1\.28to≈0\.002\{\\approx\}0\.002for*every*variant byn≥30n\{\\geq\}30\(Fig\.[2](https://arxiv.org/html/2608.06706#S4.F2)\), because once distractor variance dominates the target the one\-step objective is nearly indifferent to the action\.

#### Pixel distraction \(continuous control\)\.

On three DMC tasks\(Tassaet al\.[2018](https://arxiv.org/html/2608.06706#bib.bib48)\)we composite1616action\-independent moving occluders onto the observation—a pixel\-level analogue of Distracting\-Control distraction\(Stoneet al\.[2021](https://arxiv.org/html/2608.06706#bib.bib45)\)that leaves the physics\-state probe target clean \(Fig\.[5](https://arxiv.org/html/2608.06706#A10.F5)\)—withΔ¯\\bar\{\\Delta\}aK=16K\{=\}16Monte\-Carlo mean\. Per\-seed sign consistency is full \(9/9\): under independent distraction the centered channel probes at0\.250\.25–0\.530\.53against0\.000\.00–0\.130\.13noncentered \(Table[5](https://arxiv.org/html/2608.06706#A10.T5), Appendix[J](https://arxiv.org/html/2608.06706#A10)\)\. As retention, centering keeps 95–106% of its own distraction\-free accuracy where the noncentered ablation keeps 32% and 2\.6%\. Pre\-registered criteria and their verbatim outcomes are in Appendix[D](https://arxiv.org/html/2608.06706#A4)\.

#### Natural distractors \(Atari\)\.

On*Freeway*\(Bellemareet al\.[2013](https://arxiv.org/html/2608.06706#bib.bib49)\)the distraction is generated by the environment itself—ten lanes of streaming traffic—with RAM probe ground truth\(Anandet al\.[2019](https://arxiv.org/html/2608.06706#bib.bib46)\)and frameskip as the distractor\-strength knob\. The centered offset decodes traffic change and traffic state atR2∈\[−0\.001,0\.000\]R^\{2\}\\in\[\-0\.001,0\.000\]at every frameskip, an exactly clean channel on natural pixels, while keeping the agent’s displacement decodable*above*the action\-identity ceiling \(0\.877/0\.838/0\.7950\.877/0\.838/0\.795vs\.0\.810/0\.757/0\.6830\.810/0\.757/0\.683; Table[4](https://arxiv.org/html/2608.06706#A10.T4), Appendix[J](https://arxiv.org/html/2608.06706#A10)\)\. Rejection is routing, not blindness: traffic remains decodable from the full latent atR2≈0\.87R^\{2\}\\approx 0\.87for every variant\. The noncentered ablation is not merely leaky—the leak crowds out the signal, dropping the agent’s own displacement to0\.1490\.149–0\.4850\.485, below even the ceiling\.

#### The channel converts to behaviour where the model is accurate\.

Goal\-reaching model\-predictive control in latent space, on the same gridworld sweep, holds at0\.920\.92–0\.930\.93for the centered variants acrossn=10n\{=\}10–6060where the standard predictor falls to0\.57±0\.030\.57\{\\pm\}0\.03\(Table[10](https://arxiv.org/html/2608.06706#A10.T10), Appendix[J](https://arxiv.org/html/2608.06706#A10)\)\. Control therefore survives the AS collapse above: the planner adds no robustness mechanism, it converts the channel that centering preserves\. A pre\-registered attempt to repeat this at DMC scale returns a null, reported in full in Sec\.[5\.1](https://arxiv.org/html/2608.06706#S5.SS1)\.

#### The boundary is shared, and predicted\.

Making the distractors move*with*the action breaks the common\-mode premise, and every readout fails together: the gridworld offset probe drops to−0\.00\-0\.00and control collapses to0\.550\.55–0\.570\.57for all variants at exactly the configuration where the probe reads chance, and the continuous\-action gate breaks rejection likewise\. The representation boundary predicts the control boundary, and both fall where the analysis says they must\.

### 4\.3What the Channel Contains: Recovery without Leakage

The environments above show the channel survives; they cannot show*what*it keeps, because the true factors are not separately observable\. We therefore build generators whose factors are known by construction: a controllable displacementΔc\\Delta\_\{\\mathrm\{c\}\}that depends on the action, a nuisanceΔw\\Delta\_\{\\mathrm\{w\}\}that does not, and an encoder that sees only a mixed observation\. The nuisance is action\-independent*per transition*\(common\-mode\), not exogenous at the process level\(Efroniet al\.[2022](https://arxiv.org/html/2608.06706#bib.bib18)\)\. Two probes score the channel: held\-outR2R^\{2\}againstΔc\\Delta\_\{\\mathrm\{c\}\}\(*recover*,↑\\uparrow\) and againstΔw\\Delta\_\{\\mathrm\{w\}\}\(*reject*,↓\\downarrow\), over 120 configurations spanning both generators and factor dimensionalities\.

Only the centered variant does both, atRc2=0\.87R^\{2\}\_\{\\mathrm\{c\}\}=0\.87–0\.930\.93withRw2≈−0\.00R^\{2\}\_\{\\mathrm\{w\}\}\\approx\-0\.00across the whole grid \(Table[6](https://arxiv.org/html/2608.06706#A10.T6), Fig\.[6](https://arxiv.org/html/2608.06706#A10.F6), Appendix[J](https://arxiv.org/html/2608.06706#A10)\)\. The noncentered ablation is the diagnostic case: it recovers nearly as well but leaks heavily \(0\.740\.74–0\.970\.97\), so predictive quality and channel purity are different properties\. The standard predictor loses the controllable factor outright once the nuisance dominates\. Residual structure buys recovery; subtractingΔ¯\\bar\{\\Delta\}buys rejection\.

#### Purity is not an artifact of unpredictability\.

A skeptic can attribute a low nuisance\-R2R^\{2\}to unlearnability—no model predicts an unpredictable nuisance—so we sweep the nuisance from fully predictable to nearly stochastic and measure leak two ways: against the nuisance displacement and against the nuisance*state*\. The confound is real \(Table[8](https://arxiv.org/html/2608.06706#A10.T8)\): the noncenteredΔ\\Delta\-leak decays0\.947→0\.0220\.947\\to 0\.022as the nuisance becomes stochastic—read naively, stochasticity “purifies” the model—while its state\-leak persists at0\.800\.80, because the channel still carries where the nuisance*is*\. Centered is flat at every predictability level, as expected of a per\-transition cancellation rather than an incentive that depends on what is learnable\. Hence the rule we apply throughout: under stochastic nuisance, measure leak against the nuisance state\.

#### The sampled estimator preserves the picture\.

Repeating the same generator with continuous actions and theK=16K\{=\}16Monte\-Carlo mean of Sec\.[3](https://arxiv.org/html/2608.06706#S3), centered recovers at0\.870\.87and rejects at−0\.00\-0\.00; noncentered recovers while leaking \(0\.960\.96\); standard does neither \(Table[9](https://arxiv.org/html/2608.06706#A10.T9), Appendix[J](https://arxiv.org/html/2608.06706#A10)\)\. The failure that does register is the intended one—an action\-correlated nuisance breaks rejection to\+0\.19\+0\.19—so the binding assumption is common\-mode structure, not the quality of the estimate\.

### 4\.4Centering as a Plug\-in: Frozen RePo and TIA Hosts

![Refer to caption](https://arxiv.org/html/2608.06706v1/x3.png)Figure 3:Centering as an inference\-only plug\-in\(frozen RePo/TIA, 500k steps, DAVIS video\)\. Raw action\-delta probes≈0\{\\approx\}0; the same model’s centered readout recoversR2R^\{2\}0\.190\.19–0\.520\.52; trained\-in \(hatched\) matches or exceeds post hoc on five of six cells\. Open markers: full\-latent reference\. Per\-cell numbers: Table[2](https://arxiv.org/html/2608.06706#A5.T2)\.The mechanism is a property of the readout, not of our architecture: any world model exposing an action\-conditioned latent transition admits the same subtraction\. We test this on RePo\(Zhuet al\.[2023](https://arxiv.org/html/2608.06706#bib.bib26)\)and TIA\(Fuet al\.[2021](https://arxiv.org/html/2608.06706#bib.bib23)\), each trained 500k steps on the three DMC tasks under DAVIS video\(Pont\-Tusetet al\.[2017](https://arxiv.org/html/2608.06706#bib.bib57)\)—their own regime, not ours—comparing three readouts of the*same*frozen model under one held\-out ridge protocol: the full latent, the raw action\-deltaΔ​\(z,a\)\\Delta\(z,a\), and the post\-hoc centered channelΔ​\(z,a\)−Δ¯​\(z\)\\Delta\(z,a\)\-\\bar\{\\Delta\}\(z\)\(K=16K\{=\}16uniform\-action mean\)—inference\-only, no retraining, no training data\.

The raw action\-delta reads≈0\{\\approx\}0on every cell \(−0\.002\-0\.002to\+0\.052\+0\.052\) while the centered readout of the same frozen predictor reaches0\.190\.19–0\.520\.52\(Fig\.[3](https://arxiv.org/html/2608.06706#S4.F3); per\-cell numbers in Table[2](https://arxiv.org/html/2608.06706#A5.T2)\)—including on TIA\-cartpole, whose full latent has collapsed \(0\.030\.03\) yet still yields a0\.190\.19centered channel\. Training the centering in \(both hosts, same budget and regime\) passes the same bar on all six cells; against the post\-hoc readout it matches or exceeds on RePo \(0\.2680\.268/0\.3490\.349/0\.5770\.577vs\.0\.2640\.264/0\.3230\.323/0\.4570\.457\) and is comparable on TIA for cartpole and reacher \(0\.1980\.198/0\.5060\.506vs\.0\.1900\.190/0\.5190\.519\) but weaker on TIA\-cheetah \(0\.1680\.168vs\.0\.3450\.345\): the mechanism’s benefit is fully available at readout time on a frozen model, and training it in is neither required nor uniformly better\. In the trained\-in models, linearly decodable action\-effect information concentrates in the offset channel—the full\-latent probe falls to between−0\.04\-0\.04and0\.130\.13across all six cells \(frozen baselines:0\.030\.03–0\.790\.79\); we return to what this concentration implies in Sec\.[5\.1](https://arxiv.org/html/2608.06706#S5.SS1)\.

#### The matched baseline: post\-hoc centering of our own standard predictor\.

If the mechanism is a readout it must work without our architecture: for any monolithic predictor,uP​\(z,a\)=P​\(z,a\)−meana′⁡P​\(z,a′\)u\_\{P\}\(z,a\)=P\(z,a\)\-\\operatorname\{mean\}\_\{a^\{\\prime\}\}P\(z,a^\{\\prime\}\)can be formed at inference time\. On FourRooms the centered readout of the plain*standard*predictor probes at0\.23/0\.66/0\.70/0\.590\.23/0\.66/0\.70/0\.59acrossn=0/10/30/60n\{=\}0/10/30/60vs\.0\.31/0\.54/0\.71/0\.660\.31/0\.54/0\.71/0\.66trained\-in \(differences≤0\.12\\leq 0\.12either way\), and resurrects the trained noncentered variant’s dead channel \(0\.00→0\.690\.00\\to 0\.69atn=30n\{=\}30\)\. On DMC the parity is exact to two decimals \(0\.533/0\.328/0\.1860\.533/0\.328/0\.186vs\.0\.528/0\.323/0\.1920\.528/0\.323/0\.192, reacher/cheetah/cartpole\), and within one model the contrast is stark: the standard predictor’s raw one\-step\-change probe reads−9\.2\-9\.2\(cheetah, video distraction\) while the*same*frozen model’s centered readout reads\+0\.33\+0\.33\. The action\-correlated boundary is readout\-independent \(every post\-hoc channel collapses on d30ac\)\. The channel is a property of action\-conditioned prediction itself; the trained decomposition is one convenient instantiation—the one that carries the control result—not a requirement\. For existing models we recommend the inference\-time form: retrofitting the decomposition into a host’s recurrence degrades its control \(returns fall on five of six RePo/TIA cells\), while the post\-hoc readout leaves behavior untouched by construction\.

#### Perturbation test with controls\.

Probes are correlational, so we also test the channel’s*direction*: push the host’s state along the post\-hoc centered offset and decode the induced physics displacement \(first\-order pushes; the host is not rolled forward—direction, not control; Appendix[G\.1](https://arxiv.org/html/2608.06706#A7.SS1)\)\. On the five hosts whose instrument validates, the decoded displacement aligns with the action’s counterfactual\-centered effect in four \(cosine\+0\.09\+0\.09to\+0\.41\+0\.41\); an equal\-norm random push scores≈0\{\\approx\}0, permuting the action pairing collapses the alignment, and subtracting a*global*mean instead of the state\-conditional marginal recovers nothing\. The fifth cell is an instructive null—that host keeps physics in its deterministic belief—and across hosts the alignment tracks the pushed block’s physics content \(ρ=0\.90\\rho\{=\}0\.90,n=5n\{=\}5\)\.

## 5Discussion

### 5\.1Limitations

Centering cancels exactly what enters the action offsets identically across actions; it needs no process\-level exogeneity assumption\(cf\. Efroniet al\.[2022](https://arxiv.org/html/2608.06706#bib.bib18)\), but by the same token it offers no guarantee once a distractor’s dynamics are modulated by the agent’s action\. Our stress tests locate this boundary: in the action\-correlated FourRooms configuration the centered\-offset probe collapses to chance and control degrades for every variant alike, and rejection breaks in the continuous gate \(Secs\.[4\.2](https://arxiv.org/html/2608.06706#S4.SS2),[4\.2](https://arxiv.org/html/2608.06706#S4.SS2),[4\.3](https://arxiv.org/html/2608.06706#S4.SS3)\)—a failure mode shared by the published reafferent and agent\-like stress cases\(Hutsonet al\.[2024](https://arxiv.org/html/2608.06706#bib.bib28); Wanget al\.[2024](https://arxiv.org/html/2608.06706#bib.bib22)\)and conceded by incentive\-based separation\(Ivashkovet al\.[2026](https://arxiv.org/html/2608.06706#bib.bib4)\); what distinguishes centering is that the boundary is legible in the algebra\. Nor does the identity itself guarantee distractor removal: it removes what the*learned offsets*represent identically across actions \(a zero\-mean interactionsdist​\(z\)​ϕ​\(a\)s\_\{\\mathrm\{dist\}\}\(z\)\\phi\(a\)would survive\)—so the channel is common\-mode–rejecting by construction, distractor\-rejecting only as measured \(Secs\.[4\.3](https://arxiv.org/html/2608.06706#S4.SS3),[4\.2](https://arxiv.org/html/2608.06706#S4.SS2)\)\. Finally, the preserved channel does not by itself restore control at scale: a multi\-step planner converts it into behavior in FourRooms, but a pre\-registered DMC\-scale attempt missed its gate across a4×4\\timestraining and32×32\\timesplanning ladder—a null whose signature \(transient\-correct steering, late rollout drift\) points to model accuracy rather than a channel defect, though it does not exclude a composition failure at the control horizon \(Appendix[H](https://arxiv.org/html/2608.06706#A8)\); a frozen\-model perturbation corroborates the channel side \(Appendix[G](https://arxiv.org/html/2608.06706#A7)\)\. The control claim is therefore sufficiency, never superiority at scale; the payoff we defend is representation\-level—the post\-hoc channel on frozen predictors \(Sec\.[4\.4](https://arxiv.org/html/2608.06706#S4.SS4)\)\. Centering marginalizes only the ego action, so entities that react to the agent inherit the reafferent boundary\.

### 5\.2Outlook

The decomposition is not tied to one\-step prediction: a successor\-feature head\(Dayan[1993](https://arxiv.org/html/2608.06706#bib.bib55); Barretoet al\.[2017](https://arxiv.org/html/2608.06706#bib.bib56)\)written in centered form,Ψ​\(z,a\)=ΨB​\(z\)\+\[ΨΔ​\(z,a\)−𝔼a′​ΨΔ​\(z,a′\)\]\\Psi\(z,a\)=\\Psi\_\{B\}\(z\)\+\[\\Psi\_\{\\Delta\}\(z,a\)\-\\mathbb\{E\}\_\{a^\{\\prime\}\}\\Psi\_\{\\Delta\}\(z,a^\{\\prime\}\)\], extends common\-mode rejection to accumulated features and, with rewards linear in features, reproduces the dueling value decomposition—the analogy closes\. A common\-mode–invariantΨΔ\\Psi\_\{\\Delta\}is a natural substrate for generalized policy improvement; validating this, and scaling the representation\-to\-control conversion, is future work\.

## 6Conclusion

Action\-mean centering restructures an action\-conditioned latent predictor asz^′=B​\(z\)\+\[Δ​\(z,a\)−Δ¯​\(z\)\]\\hat\{z\}^\{\\prime\}=B\(z\)\+\[\\Delta\(z,a\)\-\\bar\{\\Delta\}\(z\)\]: any variation entering the action offsets identically across actions cancels—exactly, for discrete and sampled action sets alike—with no reward, no reconstruction, and no distractor\-specific supervision\. Across synthetic, gridworld, distracting\-DMC, and Atari testbeds, the centered channel keeps the agent’s own effect decodable where entangled predictors go action\-blind, with nuisance leak indistinguishable from zero\. The mechanism is a property of the readout, not the architecture: applied post hoc, the same subtraction recovers an action\-effect channel from a plain entangled predictor and from frozen RePo and TIA models—no retraining\. Control gains come from the passive/offset decomposition rather than centering and do not yet extend beyond the gridworld; the channel’s value is representational, and its boundary—action\-correlated distractors—is explicit in its algebra\. A learned gate proved inert and is reported as a negative result\. We hope centering becomes a default readout for action\-conditioned world models\.

## References

- T\. An, J\. Jia, G\. Li, J\. Li, C\. Zhou, P\. Liu, B\. Lyu, J\. Bai, X\. Guo, G\. Li, and J\. Yang \(2026\)Feedback world model enables precise guidance of diffusion policy\.arXiv preprint arXiv:2605\.15705\.Cited by:[§2](https://arxiv.org/html/2608.06706#S2.p2.1)\.
- A\. Anand, E\. Racah, S\. Ozair, Y\. Bengio, M\. Côté, and R\. D\. Hjelm \(2019\)Unsupervised state representation learning in Atari\.InAdvances in Neural Information Processing Systems 32 \(NeurIPS\),Note:arXiv:1906\.08226Cited by:[§4\.2](https://arxiv.org/html/2608.06706#S4.SS2.SSS0.Px3.p1.6)\.
- M\. Assran, A\. Bardes, D\. Fan, Q\. Garrido, R\. Howes, M\. Komeili, M\. Muckley, A\. Rizvi, C\. Roberts, K\. Sinha,et al\.\(2025\)V\-JEPA 2: self\-supervised video models enable understanding, prediction and planning\.arXiv preprint arXiv:2506\.09985\.Cited by:[§1](https://arxiv.org/html/2608.06706#S1.p1.1),[§2](https://arxiv.org/html/2608.06706#S2.p3.1),[§3\.1](https://arxiv.org/html/2608.06706#S3.SS1.p1.4)\.
- J\. L\. Ba, J\. R\. Kiros, and G\. E\. Hinton \(2016\)Layer normalization\.arXiv preprint arXiv:1607\.06450\.Cited by:[Appendix A](https://arxiv.org/html/2608.06706#A1.SS0.SSS0.Px2.p1.21)\.
- R\. Balestriero and Y\. LeCun \(2025\)LeJEPA: provable and scalable self\-supervised learning without the heuristics\.arXiv preprint arXiv:2511\.08544\.Cited by:[§2](https://arxiv.org/html/2608.06706#S2.p3.1)\.
- A\. Bardes, J\. Ponce, and Y\. LeCun \(2022\)VICReg: variance\-invariance\-covariance regularization for self\-supervised learning\.InProceedings of the 10th International Conference on Learning Representations \(ICLR\),Note:arXiv:2105\.04906Cited by:[§3\.3](https://arxiv.org/html/2608.06706#S3.SS3.p1.9)\.
- A\. Barreto, W\. Dabney, R\. Munos, J\. J\. Hunt, T\. Schaul, H\. van Hasselt, and D\. Silver \(2017\)Successor features for transfer in reinforcement learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§5\.2](https://arxiv.org/html/2608.06706#S5.SS2.p1.2)\.
- M\. G\. Bellemare, Y\. Naddaf, J\. Veness, and M\. Bowling \(2013\)The arcade learning environment: an evaluation platform for general agents\.Journal of Artificial Intelligence Research47,pp\. 253–279\.Cited by:[Table 1](https://arxiv.org/html/2608.06706#A1.T1),[§4\.2](https://arxiv.org/html/2608.06706#S4.SS2.SSS0.Px3.p1.6)\.
- H\. Bharadhwaj, M\. Babaeizadeh, D\. Erhan, and S\. Levine \(2022\)Information prioritization through empowerment in visual model\-based RL\.InProceedings of the 10th International Conference on Learning Representations \(ICLR\),Note:arXiv:2204\.08585Cited by:[§2](https://arxiv.org/html/2608.06706#S2.p4.1)\.
- J\. Bruce, M\. D\. Dennis, A\. Edwards, J\. Parker\-Holder, Y\. Shi, E\. Hughes, M\. Lai, A\. Mavalankar, R\. Steigerwald, C\. Apps,et al\.\(2024\)Genie: generative interactive environments\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),Note:arXiv:2402\.15391Cited by:[§2](https://arxiv.org/html/2608.06706#S2.p3.1)\.
- Z\. J\. Cui, H\. Pan, A\. Iyer, S\. Haldar, and L\. Pinto \(2024\)DynaMo: in\-domain dynamics pretraining for visuo\-motor control\.InAdvances in Neural Information Processing Systems 37 \(NeurIPS\),Note:arXiv:2409\.12192Cited by:[§2](https://arxiv.org/html/2608.06706#S2.p3.1)\.
- P\. Dayan \(1993\)Improving generalization for temporal difference learning: the successor representation\.Neural Computation5\(4\),pp\. 613–624\.Cited by:[§5\.2](https://arxiv.org/html/2608.06706#S5.SS2.p1.2)\.
- P\. de Boer, D\. P\. Kroese, S\. Mannor, and R\. Y\. Rubinstein \(2005\)A tutorial on the cross\-entropy method\.Annals of Operations Research134\(1\),pp\. 19–67\.Cited by:[Appendix H](https://arxiv.org/html/2608.06706#A8.SS0.SSS0.Px2.p1.12)\.
- F\. Deng, I\. Jang, and S\. Ahn \(2022\)DreamerPro: reconstruction\-free model\-based reinforcement learning with prototypical representations\.InProceedings of the 39th International Conference on Machine Learning \(ICML\),Note:arXiv:2110\.14565Cited by:[§2](https://arxiv.org/html/2608.06706#S2.p4.1)\.
- Y\. Efroni, D\. Misra, A\. Krishnamurthy, A\. Agarwal, and J\. Langford \(2022\)Provably filtering exogenous distractors using multistep inverse dynamics\.InProceedings of the 10th International Conference on Learning Representations \(ICLR\),Note:arXiv:2110\.08847Cited by:[§1](https://arxiv.org/html/2608.06706#S1.p3.1),[§2](https://arxiv.org/html/2608.06706#S2.p4.1),[§3\.4](https://arxiv.org/html/2608.06706#S3.SS4.p3.6),[§4\.3](https://arxiv.org/html/2608.06706#S4.SS3.p1.7),[§5\.1](https://arxiv.org/html/2608.06706#S5.SS1.p1.3)\.
- M\. Fechner, H\. Adnan, C\. C\. Lüth, M\. T\. Jackson, A\. Zakharov, and J\. M\. Zöllner \(2026\)Segment to focus: guiding latent action models in the presence of distractors\.arXiv preprint arXiv:2602\.02259\.Cited by:[§1](https://arxiv.org/html/2608.06706#S1.p1.1),[§2](https://arxiv.org/html/2608.06706#S2.p1.1),[§2](https://arxiv.org/html/2608.06706#S2.p3.1)\.
- X\. Fu, G\. Yang, P\. Agrawal, and T\. Jaakkola \(2021\)Learning task informed abstractions\.InProceedings of the 38th International Conference on Machine Learning \(ICML\),Note:arXiv:2106\.15612Cited by:[Table 2](https://arxiv.org/html/2608.06706#A5.T2),[§1](https://arxiv.org/html/2608.06706#S1.p1.1),[§1](https://arxiv.org/html/2608.06706#S1.p3.1),[§2](https://arxiv.org/html/2608.06706#S2.p4.1),[§4\.4](https://arxiv.org/html/2608.06706#S4.SS4.p1.3)\.
- J\. Grill, F\. Strub, F\. Altché, C\. Tallec, P\. H\. Richemond, E\. Buchatskaya, C\. Doersch, B\. Avila Pires, Z\. D\. Guo, M\. Gheshlaghi Azar,et al\.\(2020\)Bootstrap your own latent: a new approach to self\-supervised learning\.InAdvances in Neural Information Processing Systems 33 \(NeurIPS\),Note:arXiv:2006\.07733Cited by:[§3\.1](https://arxiv.org/html/2608.06706#S3.SS1.p1.4)\.
- D\. Ha and J\. Schmidhuber \(2018\)World models\.arXiv preprint arXiv:1803\.10122\.Cited by:[§1](https://arxiv.org/html/2608.06706#S1.p1.1)\.
- D\. Hafner, T\. Lillicrap, I\. Fischer, R\. Villegas, D\. Ha, H\. Lee, and J\. Davidson \(2019\)Learning latent dynamics for planning from pixels\.InProceedings of the 36th International Conference on Machine Learning \(ICML\),Note:arXiv:1811\.04551Cited by:[Table 2](https://arxiv.org/html/2608.06706#A5.T2)\.
- D\. Hafner, J\. Pasukonis, J\. Ba, and T\. Lillicrap \(2023\)Mastering diverse domains through world models\.arXiv preprint arXiv:2301\.04104\.Cited by:[§1](https://arxiv.org/html/2608.06706#S1.p1.1)\.
- M\. Hutson, I\. Kauvar, and N\. Haber \(2024\)Policy\-shaped prediction: avoiding distractions in model\-based reinforcement learning\.InAdvances in Neural Information Processing Systems 37 \(NeurIPS\),Note:arXiv:2412\.05766Cited by:[§1](https://arxiv.org/html/2608.06706#S1.p4.1),[§2](https://arxiv.org/html/2608.06706#S2.p4.1),[§5\.1](https://arxiv.org/html/2608.06706#S5.SS1.p1.3)\.
- R\. Islam, M\. Tomar, A\. Lamb, Y\. Efroni, H\. Zang, A\. Didolkar, D\. Misra, X\. Li, H\. van Seijen, R\. Tachet des Combes, and J\. Langford \(2023\)Agent\-controller representations: principled offline RL with rich exogenous information\.InProceedings of the 40th International Conference on Machine Learning \(ICML\),Note:arXiv:2211\.00164Cited by:[§2](https://arxiv.org/html/2608.06706#S2.p4.1)\.
- P\. Ivashkov, R\. Balestriero, and B\. Schölkopf \(2026\)Sensorimotor world models: perception for action via inverse dynamics\.arXiv preprint arXiv:2606\.20104\.Note:Concurrent workCited by:[§1](https://arxiv.org/html/2608.06706#S1.p1.1),[§1](https://arxiv.org/html/2608.06706#S1.p3.1),[§1](https://arxiv.org/html/2608.06706#S1.p4.1),[§2](https://arxiv.org/html/2608.06706#S2.p3.1),[§3\.4](https://arxiv.org/html/2608.06706#S3.SS4.p3.6),[§5\.1](https://arxiv.org/html/2608.06706#S5.SS1.p1.3)\.
- J\. E\. Kooi, M\. Hoogendoorn, and V\. François\-Lavet \(2022\)Disentangled \(un\)controllable features\.arXiv preprint arXiv:2211\.00086\.Cited by:[§2](https://arxiv.org/html/2608.06706#S2.p4.1)\.
- A\. Lamb, R\. Islam, Y\. Efroni, A\. Didolkar, D\. Misra, D\. Foster, L\. Molu, R\. Chari, A\. Krishnamurthy, and J\. Langford \(2022\)Guaranteed discovery of control\-endogenous latent states with multi\-step inverse models\.arXiv preprint arXiv:2207\.08229\.Cited by:[§2](https://arxiv.org/html/2608.06706#S2.p4.1)\.
- Y\. LeCun \(2022\)A path towards autonomous machine intelligence\.Note:OpenReview preprint, version 0\.9\.2Cited by:[§1](https://arxiv.org/html/2608.06706#S1.p1.1),[§3\.1](https://arxiv.org/html/2608.06706#S3.SS1.p1.4)\.
- A\. Levine, P\. Stone, and A\. Zhang \(2024\)Multistep inverse is not all you need\.Reinforcement Learning Journal2,pp\. 884–925\.Note:Presented at the Reinforcement Learning Conference \(RLC\); arXiv:2403\.11940Cited by:[§2](https://arxiv.org/html/2608.06706#S2.p4.1)\.
- C\. Louizos, M\. Welling, and D\. P\. Kingma \(2018\)Learning sparse neural networks throughL0L\_\{0\}regularization\.InProceedings of the 6th International Conference on Learning Representations \(ICLR\),Note:arXiv:1712\.01312Cited by:[Appendix A](https://arxiv.org/html/2608.06706#A1.SS0.SSS0.Px2.p1.21),[Appendix F](https://arxiv.org/html/2608.06706#A6.p1.5)\.
- L\. Maes, Q\. Le Lidec, D\. Scieur, Y\. LeCun, and R\. Balestriero \(2026\)LeWorldModel: stable end\-to\-end joint\-embedding predictive architecture from pixels\.arXiv preprint arXiv:2603\.19312\.Cited by:[§1](https://arxiv.org/html/2608.06706#S1.p1.1),[§2](https://arxiv.org/html/2608.06706#S2.p3.1)\.
- A\. Nikulin, I\. Zisman, D\. Tarasov, N\. Lyubaykin, A\. Polubarov, I\. Kiselev, and V\. Kurenkov \(2025\)Latent action learning requires supervision in the presence of distractors\.InProceedings of the 42nd International Conference on Machine Learning \(ICML\),Note:arXiv:2502\.00379Cited by:[§1](https://arxiv.org/html/2608.06706#S1.p1.1),[§2](https://arxiv.org/html/2608.06706#S2.p1.1),[§2](https://arxiv.org/html/2608.06706#S2.p3.1)\.
- M\. Pan, X\. Zhu, Y\. Wang, and X\. Yang \(2022\)Iso\-Dream: isolating and leveraging noncontrollable visual dynamics in world models\.InAdvances in Neural Information Processing Systems 35 \(NeurIPS\),Note:arXiv:2205\.13817Cited by:[§1](https://arxiv.org/html/2608.06706#S1.p1.1),[§1](https://arxiv.org/html/2608.06706#S1.p3.1),[§2](https://arxiv.org/html/2608.06706#S2.p4.1)\.
- M\. Pan, X\. Zhu, Y\. Zheng, Y\. Wang, and X\. Yang \(2023\)Model\-based reinforcement learning with isolated imaginations\.arXiv preprint arXiv:2303\.14889\.Cited by:[§2](https://arxiv.org/html/2608.06706#S2.p4.1)\.
- J\. Pont\-Tuset, F\. Perazzi, S\. Caelles, P\. Arbeláez, A\. Sorkine\-Hornung, and L\. Van Gool \(2017\)The 2017 DAVIS challenge on video object segmentation\.arXiv preprint arXiv:1704\.00675\.Cited by:[Table 2](https://arxiv.org/html/2608.06706#A5.T2),[§4\.4](https://arxiv.org/html/2608.06706#S4.SS4.p1.3)\.
- T\. Saanum, P\. Dayan, and E\. Schulz \(2024\)Simplifying latent dynamics with softly state\-invariant world models\.InAdvances in Neural Information Processing Systems 37 \(NeurIPS\),Note:arXiv:2401\.17835Cited by:[§2](https://arxiv.org/html/2608.06706#S2.p3.1)\.
- Y\. Sawada \(2018\)Disentangling controllable and uncontrollable factors of variation by interacting with the world\.arXiv preprint arXiv:1804\.06955\.Cited by:[§2](https://arxiv.org/html/2608.06706#S2.p4.1)\.
- D\. Schmidt and M\. Jiang \(2024\)Learning to act without actions\.InProceedings of the 12th International Conference on Learning Representations \(ICLR\),Note:arXiv:2312\.10812Cited by:[§2](https://arxiv.org/html/2608.06706#S2.p3.1)\.
- M\. Seitzer, B\. Schölkopf, and G\. Martius \(2021\)Causal influence detection for improving efficiency in reinforcement learning\.InAdvances in Neural Information Processing Systems 34 \(NeurIPS\),Note:arXiv:2106\.03443Cited by:[§1](https://arxiv.org/html/2608.06706#S1.p3.1),[§2](https://arxiv.org/html/2608.06706#S2.p2.1),[§3\.2](https://arxiv.org/html/2608.06706#S3.SS2.SSS0.Px1.p1.11)\.
- A\. Stone, O\. Ramirez, K\. Konolige, and R\. Jonschkowski \(2021\)The distracting control suite – a challenging benchmark for reinforcement learning from pixels\.arXiv preprint arXiv:2101\.02722\.Cited by:[§4\.2](https://arxiv.org/html/2608.06706#S4.SS2.SSS0.Px2.p1.7)\.
- R\. Sun, H\. Zang, X\. Li, and R\. Islam \(2024\)Learning latent dynamic robust representations for world models\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),Note:arXiv:2405\.06263Cited by:[§2](https://arxiv.org/html/2608.06706#S2.p4.1)\.
- Y\. Tassa, Y\. Doron, A\. Muldal, T\. Erez, Y\. Li, D\. de Las Casas, D\. Budden, A\. Abdolmaleki, J\. Merel, A\. Lefrancq,et al\.\(2018\)DeepMind control suite\.arXiv preprint arXiv:1801\.00690\.Cited by:[Table 1](https://arxiv.org/html/2608.06706#A1.T1),[§4\.2](https://arxiv.org/html/2608.06706#S4.SS2.SSS0.Px2.p1.7)\.
- V\. Thomas, E\. Bengio, W\. Fedus, J\. Pondard, P\. Beaudoin, H\. Larochelle, J\. Pineau, D\. Precup, and Y\. Bengio \(2018\)Disentangling the independently controllable factors of variation by interacting with the world\.arXiv preprint arXiv:1802\.09484\.Cited by:[§2](https://arxiv.org/html/2608.06706#S2.p4.1)\.
- V\. Thomas, J\. Pondard, E\. Bengio, M\. Sarfati, P\. Beaudoin, M\. Meurs, J\. Pineau, D\. Precup, and Y\. Bengio \(2017\)Independently controllable factors\.arXiv preprint arXiv:1708\.01289\.Cited by:[§2](https://arxiv.org/html/2608.06706#S2.p4.1)\.
- A\. van den Oord, Y\. Li, and O\. Vinyals \(2018\)Representation learning with contrastive predictive coding\.arXiv preprint arXiv:1807\.03748\.Cited by:[§3\.3](https://arxiv.org/html/2608.06706#S3.SS3.p1.9)\.
- T\. Wang, S\. S\. Du, A\. Torralba, P\. Isola, A\. Zhang, and Y\. Tian \(2022\)Denoised MDPs: learning world models better than the world itself\.InProceedings of the 39th International Conference on Machine Learning \(ICML\),Note:arXiv:2206\.15477Cited by:[§1](https://arxiv.org/html/2608.06706#S1.p1.1),[§1](https://arxiv.org/html/2608.06706#S1.p3.1),[§2](https://arxiv.org/html/2608.06706#S2.p4.1)\.
- Y\. Wang, S\. Wan, L\. Gan, S\. Feng, and D\. Zhan \(2024\)AD3: implicit action is the key for world models to distinguish the diverse visual distractors\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),Note:arXiv:2403\.09976Cited by:[§1](https://arxiv.org/html/2608.06706#S1.p4.1),[§2](https://arxiv.org/html/2608.06706#S2.p4.1),[§5\.1](https://arxiv.org/html/2608.06706#S5.SS1.p1.3)\.
- Z\. Wang, T\. Schaul, M\. Hessel, H\. van Hasselt, M\. Lanctot, and N\. de Freitas \(2016\)Dueling network architectures for deep reinforcement learning\.InProceedings of the 33rd International Conference on Machine Learning \(ICML\),pp\. 1995–2003\.Cited by:[§1](https://arxiv.org/html/2608.06706#S1.p3.1),[§2](https://arxiv.org/html/2608.06706#S2.p2.1),[§3\.2](https://arxiv.org/html/2608.06706#S3.SS2.p2.5)\.
- H\. Wei, X\. Chen, C\. Zhang, T\. Pearce, J\. Chen, A\. Lamb, L\. Zhao, and J\. Bian \(2026\)Learning additively compositional latent actions for embodied AI\.arXiv preprint arXiv:2604\.03340\.Cited by:[§2](https://arxiv.org/html/2608.06706#S2.p3.1)\.
- G\. Zhou, H\. Pan, Y\. LeCun, and L\. Pinto \(2025\)DINO\-WM: world models on pre\-trained visual features enable zero\-shot planning\.InProceedings of the 42nd International Conference on Machine Learning \(ICML\),Note:arXiv:2411\.04983Cited by:[§1](https://arxiv.org/html/2608.06706#S1.p1.1),[§2](https://arxiv.org/html/2608.06706#S2.p3.1),[§3\.1](https://arxiv.org/html/2608.06706#S3.SS1.p1.4)\.
- C\. Zhu, M\. Simchowitz, S\. Gadipudi, and A\. Gupta \(2023\)RePo: resilient model\-based reinforcement learning by regularizing posterior predictability\.InAdvances in Neural Information Processing Systems 36 \(NeurIPS\),Note:arXiv:2309\.00082Cited by:[Table 2](https://arxiv.org/html/2608.06706#A5.T2),[§1](https://arxiv.org/html/2608.06706#S1.p1.1),[§2](https://arxiv.org/html/2608.06706#S2.p4.1),[§4\.4](https://arxiv.org/html/2608.06706#S4.SS4.p1.3)\.
- H\. Zhu, Z\. Dong, K\. Topollai, B\. Sha, and A\. Choromanska \(2026\)Self\-supervised representation learning with joint embedding predictive architecture for automotive LiDAR object detection\.InProceedings of the 40th AAAI Conference on Artificial Intelligence \(AAAI\),Note:AD\-L\-JEPA; arXiv:2501\.04969Cited by:[footnote 1](https://arxiv.org/html/2608.06706#footnote1)\.

## Appendix AImplementation and Training Details

#### Shared setup\.

All variants share the encoder, an EMA target encoder \(τ=0\.996\\tau\{=\}0\.996, buffers copied\), the optimizer \(AdamW, base learning rate3×10−43\\times 10^\{\-4\}, weight decay10−610^\{\-6\}, cosine\-annealed over training withTmaxT\_\{\\max\}equal to the step budget, gradient\-norm clip1010\), and the loss weights \(λact=0\.1\\lambda\_\{\\mathrm\{act\}\}\{=\}0\.1with softmax temperature0\.10\.1,λoff=10−4\\lambda\_\{\\mathrm\{off\}\}\{=\}10^\{\-4\},λgate=10−5\\lambda\_\{\\mathrm\{gate\}\}\{=\}10^\{\-5\}with a warm\-up over the first20%20\\%of steps,λreg=0\.1\\lambda\_\{\\mathrm\{reg\}\}\{=\}0\.1\)\. Only the predictor head differs across the ladder \(Sec\.[3\.2](https://arxiv.org/html/2608.06706#S3.SS2)\)\.

#### Architecture\.

The MLP encoder flattens the observation and appliesLinear→SiLU→Linear→SiLU→Linear\\mathrm\{Linear\}\{\\to\}\\mathrm\{SiLU\}\{\\to\}\\mathrm\{Linear\}\{\\to\}\\mathrm\{SiLU\}\{\\to\}\\mathrm\{Linear\}\(two hidden layers of width256256\) to dimensiondd, followed byLayerNorm\(Baet al\.[2016](https://arxiv.org/html/2608.06706#bib.bib58)\)\. The CNN encoder applies three3×33\{\\times\}3convolutions \(c→32→64→64c\{\\to\}32\{\\to\}64\{\\to\}64channels, strides1,2,21,2,2, padding11, SiLU\), flattens, and maps toddthrough a256256\-wide hidden layer andLayerNorm\(a64×6464\{\\times\}64input reduces to16×16×6416\{\\times\}16\{\\times\}64before the head\)\. Every prediction head is a two\-hidden\-layer, width\-256256SiLU MLP: the passive headB:d→dB\{:\}~d\{\\to\}d; the discrete offset headd→\|𝒜\|⋅dd\{\\to\}\|\\mathcal\{A\}\|\{\\cdot\}d\(all offsets in one pass, soΔ¯\\bar\{\\Delta\}is exact and free\); the continuous offset head\(d\+m\)→d\(d\{\+\}m\)\{\\to\}d; and the entangled baselinePP, which embeds the action to3232dimensions \(learned embedding for discrete, linear map for continuous\) before an MLP\(d\+32\)→d\(d\{\+\}32\)\{\\to\}d\. The gateCCis a one\-hidden\-layer MLP to a sigmoid \(or a hard\-concreteL0L\_\{0\}gate;Louizoset al\.[2018](https://arxiv.org/html/2608.06706#bib.bib41)\), initialized open\.

Table 1:Per\-environment settings, for the FourRooms gridworld, DMC\(Tassaet al\.[2018](https://arxiv.org/html/2608.06706#bib.bib48)\), Atari\(Bellemareet al\.[2013](https://arxiv.org/html/2608.06706#bib.bib49)\)and the synthetic testbed\. Continuous DMC action dimension is task\-dependent \(11cartpole\-swingup,22reacher\-easy,66cheetah\-run\);ncfn\_\{\\mathrm\{cf\}\}counterfactual states=1000=1000\(discrete\) /500500\(continuous\)\. Synthetic transitions are3030k train /88k test, self\-generated per cell over33seeds\.∗FourRooms:6060k steps for the probe\-sweep runs,100100k for the goal\-reaching control runs\.
#### EstimatingΔ¯\\bar\{\\Delta\}for continuous actions\.

K=16K\{=\}16candidate actions per state: the executed action \(index0\),77actions resampled from the replay buffer, and88Gaussian perturbations of the executed action with standard deviation0\.30\.3of the action half\-range, clipped to the action bounds—the most even split available of the remainingK−1=15K\{\-\}1\{=\}15candidates \(the main text’s “split evenly”\)\. For discrete actionsΔ¯\\bar\{\\Delta\}is the exact mean over all\|𝒜\|\|\\mathcal\{A\}\|offsets\.

#### FourRooms distractors\.

The13×1313\{\\times\}13layout adds a fourth binary observation channel ofnncells that deterministically roll one column per step \(np\.roll\), independent of the action; we sweepn∈\{0,10,30,60\}n\\in\\\{0,10,30,60\\\}\. In the action\-correlated stress configuration \(d30ac\), the same field instead rolls by the agent’s own\(Δ​row,Δ​col\)\(\\Delta\\mathrm\{row\},\\Delta\\mathrm\{col\}\)displacement each step, so it is no longer common\-mode and centering does not cancel it—the measured boundary\. The five actions are up/down/left/right/no\-op\.

#### Synthetic identifiability generator\.

A controllable factorc∈ℝ2c\\in\\mathbb\{R\}^\{2\}is moved by fixed per\-action displacements \(no\-op and±0\.3\\pm 0\.3along each axis;55actions\), clipped to\[−1,1\]\[\-1,1\]\. A distractorw∈ℝdww\\in\\mathbb\{R\}^\{d\_\{w\}\}evolves action\-independently asw←R​ww\\leftarrow RwwithRRa fixed block\-rotation \(angles∼𝒰​\(0\.2,0\.5\)\\sim\\mathcal\{U\}\(0\.2,0\.5\)\); in the correlated variantw←R​w\+Vaw\\leftarrow Rw\+V\_\{a\}withVa∼𝒩​\(0,0\.152\)V\_\{a\}\\sim\\mathcal\{N\}\(0,0\.15^\{2\}\)per action\. The observation is a fixed random mixingGGof\[c;w\]\[c;w\]intoℝ64\\mathbb\{R\}^\{64\}, either linear \(orthonormal columns\) or nonlinear \(twotanhlayers, width6464\)\. Episodes have length5050; we probe the held\-out change feature \(the centered offset for centered/noncentered/gated, the one\-step change for standard\) against the trueΔ​c\\Delta c\(Rc2R^\{2\}\_\{c\}, recover\) andΔ​w\\Delta w\(Rw2R^\{2\}\_\{w\}, reject\)\.

#### Checkpoint selection\.

The reported “best” checkpoint is the lowest validation cosine loss among checkpoints whose action separation exceeds a regime\-aware threshold\. An explicit override always wins; otherwise FourRooms with rolling action\-independent distractors uses threshold0—there even healthy runs sit at AS≈0\.002\{\\approx\}0\.002\(with offset\-probeR2≈0\.74R^\{2\}\\approx 0\.74\) while dead ones reach≈0\.008\{\\approx\}0\.008, so AS carries no collapse signal and the ridge probe is the health diagnostic—and every other setting uses0\.010\.01\. If no checkpoint passes, the highest\-AS checkpoint is used and the run is flagged collapsed\.

#### Code and configurations\.

Every model\-training run has a named configuration file carrying the per\-experiment overrides above; the synthetic testbeds are self\-contained scripts with their settings inlined\. Those configuration files, the run scripts, and our full training and evaluation code are provided in the supplementary code archive\.

## Appendix BProof of Proposition[2](https://arxiv.org/html/2608.06706#Thmproposition2)\(Sampled Centering\)

We restate the proposition, which is proved in full here; it appears as Proposition[2](https://arxiv.org/html/2608.06706#Thmproposition2)in the main text\.

###### Proposition 2\(Sampled centering\)\.

Fixzzand letΔ¯K​\(z\)=1K​∑k=1KΔ​\(z,ak\)\\bar\{\\Delta\}\_\{K\}\(z\)=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\Delta\(z,a\_\{k\}\)for candidate actions\{ak\}k=1K\\\{a\_\{k\}\\\}\_\{k=1\}^\{K\}drawn from a proposalν\\nu, and suppose the offset head decomposes asΔ​\(z,a\)=g​\(z,a\)\+h​\(z\)\\Delta\(z,a\)=g\(z,a\)\+h\(z\)for someggand some action\-independenthh\. Then, for everyKKand every draw:

1. \(i\)Δ​\(z,a\)−Δ¯K​\(z\)=g​\(z,a\)−1K​∑k=1Kg​\(z,ak\)\\displaystyle\\Delta\(z,a\)\-\\bar\{\\Delta\}\_\{K\}\(z\)=g\(z,a\)\-\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}g\(z,a\_\{k\}\), so the common\-mode componenthhcancels exactly; and
2. \(ii\)the sampling errorΔ¯K​\(z\)−𝔼a′∼ν​\[Δ​\(z,a′\)\]\\bar\{\\Delta\}\_\{K\}\(z\)\-\\mathbb\{E\}\_\{a^\{\\prime\}\\sim\\nu\}\[\\Delta\(z,a^\{\\prime\}\)\]does not depend on the query actionaa, isO​\(1/K\)O\(1/\\sqrt\{K\}\)under bounded second moments ofg​\(z,⋅\)g\(z,\\cdot\), and cancels in any action\-differential comparisonu​\(z,a\)−u​\(z,a′\)u\(z,a\)\-u\(z,a^\{\\prime\}\)\.

###### Proof\.

*Part \(i\)\.*SubstitutingΔ​\(z,ak\)=g​\(z,ak\)\+h​\(z\)\\Delta\(z,a\_\{k\}\)=g\(z,a\_\{k\}\)\+h\(z\)into the definition ofΔ¯K\\bar\{\\Delta\}\_\{K\}and using1K​∑k=1K1=1\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}1=1,

Δ¯K​\(z\)=1K​∑k=1K\[g​\(z,ak\)\+h​\(z\)\]=1K​∑k=1Kg​\(z,ak\)\+h​\(z\),\\begin\{split\}\\bar\{\\Delta\}\_\{K\}\(z\)&\\;=\\;\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\bigl\[g\(z,a\_\{k\}\)\+h\(z\)\\bigr\]\\\\ &\\;=\\;\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}g\(z,a\_\{k\}\)\\;\+\\;h\(z\),\\end\{split\}sinceh​\(z\)h\(z\)is constant across the averaged terms\. Subtracting this fromΔ​\(z,a\)=g​\(z,a\)\+h​\(z\)\\Delta\(z,a\)=g\(z,a\)\+h\(z\)gives

Δ​\(z,a\)−Δ¯K​\(z\)=g​\(z,a\)−1K​∑k=1Kg​\(z,ak\),\\Delta\(z,a\)\-\\bar\{\\Delta\}\_\{K\}\(z\)\\;=\\;g\(z,a\)\-\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}g\(z,a\_\{k\}\),which is claim \(i\)\. The cancellation is term by term and therefore holds for everyK≥1K\\geq 1and every realised candidate set; no independence, coverage, or unbiasedness property ofν\\nuis used\.

*Part \(ii\)\.*By the same substitution, and because thehhcomponents coincide in both terms and drop,

Δ¯K​\(z\)−𝔼a′∼ν​\[Δ​\(z,a′\)\]=1K​∑k=1Kg​\(z,ak\)−𝔼a′∼ν​\[g​\(z,a′\)\]\.\\begin\{split\}\\bar\{\\Delta\}\_\{K\}\(z\)&\-\\mathbb\{E\}\_\{a^\{\\prime\}\\sim\\nu\}\\\!\\left\[\\Delta\(z,a^\{\\prime\}\)\\right\]\\\\ &\\;=\\;\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}g\(z,a\_\{k\}\)\-\\mathbb\{E\}\_\{a^\{\\prime\}\\sim\\nu\}\\\!\\left\[g\(z,a^\{\\prime\}\)\\right\]\.\\end\{split\}The right\-hand side involves onlyggand is a single quantity that does not depend on the query actionaa; it is therefore an action\-independent shift of the channel\. If theaka\_\{k\}are drawn i\.i\.d\. fromν\\nuandg​\(z,⋅\)g\(z,\\cdot\)has bounded second moments, it is a sample mean minus its expectation, so its standard deviation isO​\(1/K\)O\(1/\\sqrt\{K\}\)\. Finally, since the shift is common to every action, it cancels in any action\-differential comparison:

u​\(z,a\)−u​\(z,a′\)=\[Δ​\(z,a\)−Δ¯K​\(z\)\]−\[Δ​\(z,a′\)−Δ¯K​\(z\)\]=Δ​\(z,a\)−Δ​\(z,a′\)\.∎\\begin\{split\}u\(z,a\)\-u\(z,a^\{\\prime\}\)&\\;=\\;\\bigl\[\\Delta\(z,a\)\-\\bar\{\\Delta\}\_\{K\}\(z\)\\bigr\]\\\\ &\\qquad\-\\bigl\[\\Delta\(z,a^\{\\prime\}\)\-\\bar\{\\Delta\}\_\{K\}\(z\)\\bigr\]\\\\ &\\;=\\;\\Delta\(z,a\)\-\\Delta\(z,a^\{\\prime\}\)\.\\qed\\end\{split\}

Two remarks\.*Self\-inclusion\.*In practice the executed action occupies one candidate slot \(a1=aa\_\{1\}=a\); for that query the channel is attenuated by exactly\(1−1/K\)\(1\-1/K\):u​\(z,a\)=K−1K​\[g​\(z,a\)−1K−1​∑k≥2g​\(z,ak\)\]u\(z,a\)=\\frac\{K\-1\}\{K\}\\bigl\[g\(z,a\)\-\\frac\{1\}\{K\-1\}\\sum\_\{k\\geq 2\}g\(z,a\_\{k\}\)\\bigr\]—anO​\(1/K\)O\(1/K\)effect onggonly, with thehh\-cancellation unaffected\.*Proposal bias\.*A biased or narrowν\\nu\(e\.g\. a state\-dependent behavior policy\) shifts𝔼ν​\[g\]\\mathbb\{E\}\_\{\\nu\}\[g\]and hence where the channel is centered; by \(i\) it cannot re\-admit the common mode\.*Empirics\.*AKK\-sweep over all108108DMC checkpoints \(K∈\{2,4,8,16,32\}K\\in\\\{2,4,8,16,32\\\}\) matches the proposition: channel probeR2R^\{2\}rises from≈0\{\\approx\}0atK=2K\{=\}2—where the\(1−1/K\)\(1\-1/K\)self\-inclusion attenuation is12\\tfrac\{1\}\{2\}and the centering point rests on a single fresh sample—to its plateau byK≈16K\\approx 16\(e\.g\.0\.520\.52of theK=32K\{=\}32value0\.560\.56on reacher\-independent\), while the noncentered channel, which uses no sampled mean, is exactly flat inKK—a built\-in negative control\. The resampling dispersion of the post\-hoc centering point decays monotonically inKK\(fitted log\-log slope−0\.26±0\.03\-0\.26\\pm 0\.03across noncentered and standard hosts; shallower than the i\.i\.d\.−12\-\\tfrac\{1\}\{2\}because the proposal is structured—the executed action is pinned and the replay/perturbation composition shifts withKK\)\. For the*trained\-in*decomposition the candidate\-mean prediction isB​\(z\)B\(z\)*exactly*, for every draw—1K​∑k\[Δ​\(z,ak\)−Δ¯K​\(z\)\]=0\\frac\{1\}\{K\}\\sum\_\{k\}\[\\Delta\(z,a\_\{k\}\)\-\\bar\{\\Delta\}\_\{K\}\(z\)\]=0—so its output’s centering point is draw\-invariant by construction \(measured dispersion at float precision,∼10−7\{\\sim\}10^\{\-7\}\): a small but provable advantage of embedding the readout in the parameterization\.

## Appendix CThe Out\-of\-Distribution Goal Pathology

Episodes that terminate on goal contact leave the on\-goal state absent from the training marginal: in 200k FourRooms transitions it never appears as a start state\. The converged encoder then decodes the on\-goal state10\.510\.5cells off \(all other cells:∼0\.005\{\\sim\}0\.005\), so no imagined plan can ever terminate “at the goal” and MPC success is0\.000\.00despite perfect in\-distribution prediction, rollout, and position\-decoding metrics — metrics that are all blind to the missing state\. Allowing the collection policy \(only\) to walk through the goal restores the state to the training distribution and MPC success to1\.001\.00across both variants and all seeds, with in\-distribution metrics unchanged\. We report this as a cautionary protocol note: rollout fidelity does not imply planning success when the plan’s target is itself out of distribution\.

## Appendix DDistracting DMC: Pre\-Registered Gate — Criteria and Outcomes

The DMC decision criteria were fixed before the matrix ran: \(i\) a centered\-minus\-noncentered probe gap≥0\.2\\geq 0\.2under independent distraction with 3/3 seed sign\-consistency per task; \(ii\) an absolute bar ofR2≥0\.5R^\{2\}\\geq 0\.5under distraction; \(iii\) a correlated\-regime gap within±0\.05\\pm 0\.05\. Outcomes, verbatim: \(i\)passon 3/3 tasks, 9/9 seeds \(gaps\+0\.212\+0\.212/\+0\.244\+0\.244/\+0\.493\+0\.493on cheetah/cartpole/reacher\); \(ii\)fail, diagnosed as ceiling mis\-calibration—the distraction\-free probe ceilings are0\.260\.26\(cartpole\) and≈0\.32\{\\approx\}0\.32–0\.400\.40\(cheetah\), so a fixed absolute bar cannot be met even without distractors on two of three tasks \(only reacher’s ceiling exceeds0\.50\.5\); \(iii\) within tolerance on 2/3 tasks \(\+0\.033\+0\.033cheetah,−0\.040\-0\.040cartpole\), with the reacher inversion \(−0\.153\-0\.153\) analyzed as contamination\-boost: the noncentered offset’s correlated\-regime probe \(0\.5350\.535\) exceeds its own distraction\-free level \(0\.1690\.169\), so the surplus is distractor signal masquerading as action effect\. Per\-seed values for the four variants run at DMC scale—standard \(which exposes no isolated channel\), noncentered, centered, and gated \(whose gate is inert\)—are given in Table[11](https://arxiv.org/html/2608.06706#A10.T11); the*residual*rung of the main text’s ladder was not part of the DMC matrix\.

## Appendix EPlug\-in: Per\-Cell Numbers

Table[2](https://arxiv.org/html/2608.06706#A5.T2)gives the exact per\-cell probe values behind Fig\.[3](https://arxiv.org/html/2608.06706#S4.F3)\. Readout variance over five probe seeds \(same checkpoints, resampled evaluation data and probe splits\): the centered column varies by s\.d\.0\.0090\.009–0\.0300\.030per cell \(RePo0\.260±\.0240\.260\{\\pm\}\.024/0\.300±\.0160\.300\{\\pm\}\.016/0\.454±\.0140\.454\{\\pm\}\.014, TIA0\.184±\.0300\.184\{\\pm\}\.030/0\.336±\.0090\.336\{\\pm\}\.009/0\.504±\.0110\.504\{\\pm\}\.011for cartpole/cheetah/reacher\), with rawΔ\\Deltaat≤0\.061\\leq 0\.061throughout; the one\-training\-run caveat is a host property, not readout noise\.

Table 2:Centering as a plug\-in\(probeR2R^\{2\}, higher is better except rawΔ\\Deltawhich diagnoses the entangled readout\)\. RePo\(Zhuet al\.[2023](https://arxiv.org/html/2608.06706#bib.bib26)\)and TIA\(Fuet al\.[2021](https://arxiv.org/html/2608.06706#bib.bib23)\)trained 500k steps under DAVIS video backgrounds\(Pont\-Tusetet al\.[2017](https://arxiv.org/html/2608.06706#bib.bib57)\), one training seed per cell; probes on frozen final checkpoints over5,0005\{,\}000held\-out random\-policy transitions, target = change in physics state\.*RawΔ\\Delta*vs\.*centered*: identical model, identical data—the only change is subtracting theK=16K\{=\}16action\-mean at readout\.*Trained\-in*: centering inside the RSSM prior\(Hafneret al\.[2019](https://arxiv.org/html/2608.06706#bib.bib42)\)for the full 500k run \(both hosts\)\. Budgets and protocols differ from ours; no cross\-method return comparison is implied\.
## Appendix FThe Gate Ablation Is Inert

The gated variant was designed as a learned per\-dimension controllability mask: a sigmoid gateC​\(z\)∈\(0,1\)dC\(z\)\\in\(0,1\)^\{d\}under an L1 sparsity penalty, intended to select which latent dimensions the action channel may write to\. It does not learn this\. Across all testbeds the gate converges to near\-uniform down\-scaling rather than sparse selection—an inert reparameterization, since a uniform gate can be absorbed into the scale ofΔ\\Delta, and L1\-on\-sigmoid admits exactly this shrink\-everything solution\. Replacing L1 with a hard\-concreteL0L\_\{0\}relaxation\(Louizoset al\.[2018](https://arxiv.org/html/2608.06706#bib.bib41)\)does not rescue it: theL0L\_\{0\}gate collapses at every sparsity coefficient we tested\. The gate does not improve channel purity over plain centering and adds a genuine failure mode at the strongest distractor setting \(Sec\.[4\.2](https://arxiv.org/html/2608.06706#S4.SS2)\)\. We therefore report it as a negative result: AD\-JEPA setsC≡1C\\equiv 1, centering is the sole mechanism, and the gated rung survives only as an ablation\.

## Appendix GSteering Along the Channel Under Distraction

We test the action channel by*perturbation*rather than by correlation\. For a frozen DMC checkpoint we push the latent along the model’s own channelu​\(z,a\)u\(z,a\)\(centered offset for centered/gated, raw offset for noncentered,z^′−z\\hat\{z\}^\{\\prime\}\-zfor standard\), project back to the encoder’s layer\-norm manifold, and decode the induced state displacement with a nonlinear MLP trained only as a measurement instrument\. The steering direction comes from the channel alone—no privileged signal enters, unlike the compass\-scored planner of Appendix[H](https://arxiv.org/html/2608.06706#A8)\. We report the cosine between the decoded displacement and the action’s counterfactual state effectΔ​s​\(a\)−1K​∑kΔ​s​\(ak\)\\Delta s\(a\)\-\\frac\{1\}\{K\}\\sum\_\{k\}\\Delta s\(a\_\{k\}\)\(cf\-cos\) and, as the load\-bearing control, the same quantity for an*equal\-norm random push*through the identical decoder—absolute decoder quality cancels, so the comparison is instrument\-relative\.33tasks×\\times33distractor modes×\\times44variants×\\times33seeds; push scalesγ∈\{0\.5,1,2\}\\gamma\\in\\\{0\.5,1,2\\\}; we quoteγ=1\\gamma\{=\}1throughout \(cf\-cos is positive and monotone inγ\\gammawherever it is nonzero, so no push scale is selected post hoc\)\. Pushes are first\-order and the predictor is not rolled forward from the pushed latent: the measurement is directional—channel alignment, not closed\-loop control\.

#### Result \(cheetah\-run\)\.

Without distractors every channel steers \(cf\-cos0\.42/0\.42/0\.29/0\.430\.42/0\.42/0\.29/0\.43for centered/gated/noncentered/ standard\)\. Under*independent*distraction only the centered variants survive \(0\.27/0\.260\.27/0\.26vs\.0\.00/0\.020\.00/0\.02; random control≈0\.01\{\\approx\}0\.01throughout;3/33/3seeds\): centering retains∼65%\{\\sim\}65\\%of its no\-distractor steering where the raw\-offset and standard channels retain essentially none\. Two properties sharpen this\.*\(i\) Norm does not buy efficacy*: the channel that steers has∥u∥≈0\.13\\lVert u\\rVert\\approx 0\.13; the standard channel that does not has∥u∥≈16\\lVert u\\rVert\\approx 16\.*\(ii\) Selectivity*: a centered push moves the decoded*distractor*state about half as much as an equal\-norm random push, and about5×5\\timesless than the noncentered channel’s push\. Under*action\-correlated*distraction centered steering collapses to≈0\.07\{\\approx\}0\.07even at the most favorable push scale—the pre\-registered boundary\.

#### Scope and instrument\.

The instrument is a nonlinear MLP; because the matched random push absorbs decoder quality, the claim is directional and instrument\-relative—what a low absolute decodeR2R^\{2\}under distraction permits\. The contrast is not an instrument artifact: across five decoder configurations \(widths128128–512512, depths11–33, with and without input noise; testR2R^\{2\}spanning−1\.8\-1\.8to−3\.6\-3\.6\), the centered channel’s alignment is flat \(0\.250\.25–0\.310\.31\) while the noncentered and standard channels stay at≈0\{\\approx\}0in every configuration\. reacher\-easy is uninformative: no channel steers even without distractors, consistent with its documented undertraining at this budget \(Appendix[D](https://arxiv.org/html/2608.06706#A4)\) rather than a task property—fully trained baselines reach stateR2R^\{2\}0\.800\.80–0\.920\.92on the same task, and post\-hoc centering on those frozen models yields a0\.460\.46–0\.580\.58channel \(Sec\.[4\.4](https://arxiv.org/html/2608.06706#S4.SS4)\)\. cartpole\-swingup is weakly positive for the centered variant only \(0\.060\.06vs\. random0\.020\.02\)\. We therefore report cheetah\-run as the clean case and the other two as gated out by a no\-distractor model\-quality check, per our report\-either\-way pre\-commitment\.

### G\.1Frozen Third\-Party Hosts

The same intervention ports to the frozen RePo/TIA hosts of Sec\.[4\.4](https://arxiv.org/html/2608.06706#S4.SS4)—RSSM architectures and objectives we did not design, trained with the authors’ released code and frozen before the readout is applied\. These hosts run under DAVIS video \(action\-independent distraction only, no distractor ground truth\), so the test measures*action\-specificity*of the channel direction, not common\-mode rejection\. We push the host’s stochastic state along the post\-hoc channelu​\(z,a\)=m​\(z,a\)−116​∑km​\(z,ak\)u\(z,a\)=m\(z,a\)\-\\frac\{1\}\{16\}\\sum\_\{k\}m\(z,a\_\{k\}\)of the RSSM prior meanmm, decode with an MLP fitted per host \(shuffled\-split calibration, matching the reference ridge protocol\), and score the decoded displacement against the action’s counterfactual\-centered physics effect at fixedγ=1\\gamma\{=\}1; three evaluation seeds per host, one training run per cell\.

Table 3:Steering frozen third\-party hosts\(cosine to the counterfactual\-centered physics effect,γ=1\\gamma\{=\}1, mean±\\pms\.d\. over 3 evaluation seeds\)\. Controls:*glob\.*subtracts the global mean offset instead of the state\-conditional marginal;*rand*is an equal\-norm random push through the same decoder;*perm*scores against permuted \(deranged\) targets\.R2R^\{2\}/Rs2R^\{2\}\_\{s\}: decoder fit from the full latent / from the pushed state block alone\. TIA\-cartpole is excluded: its instrument fails \(R2=−0\.07R^\{2\}\{=\}\-0\.07\) on a host whose full latent is independently known to be collapsed \(0\.030\.03, Table[2](https://arxiv.org/html/2608.06706#A5.T2)\)\.Four observations\.*\(i\)*The three controls isolate the channel: the equal\-norm random push and the permutation null are≈0\{\\approx\}0everywhere \(the alignment carries per\-sample action information\), and the global\-mean control tracks raw rather than centered—only the state\-conditional marginal recovers the channel, the observation behind the design note in Sec\.[3](https://arxiv.org/html/2608.06706#S3)\.*\(ii\)*repo\-reacher is an honest null on a validated instrument \(R2=0\.88R^\{2\}\{=\}0\.88\): that host keeps physics almost entirely in its deterministic belief \(Rbelief2=0\.89R^\{2\}\_\{\\text\{belief\}\}=0\.89vs\.Rs2=0\.04R^\{2\}\_\{s\}=0\.04\), so the pushed block has nothing to steer; across the five validated cells the alignment tracksRs2R^\{2\}\_\{s\}\(Spearmanρ=0\.90\\rho=0\.90,n=5n\{=\}5—suggestive, not confirmatory\)\.*\(iii\)*Stability tracks instrument validity: across three independent evaluation runs, validated\-instrument cells replicate closely \(repo\-cartpole moves by0\.0030\.003\) while broken\-instrument numbers moved arbitrarily in both directions—we therefore report only validated cells\.*\(iv\)*The result is directional \(first\-order pushes; the host is never rolled forward\) and says nothing about distractor invariance, which DAVIS hosts cannot measure\.

## Appendix HExtended Discussion

This appendix gives the full\-length analysis of the control null summarized in Sec\.[5\.1](https://arxiv.org/html/2608.06706#S5.SS1)\.

#### The one\-step predictor does not use the channel unaided\.

Action separation \(AS; the mean pairwise distance between per\-action predictions\) collapses from∼1\.28\{\\sim\}1\.28to∼0\.002\{\\sim\}0\.002for*every*variant—including centered—once FourRooms distraction reachesn≥30n\\geq 30cells, even though the centered offset probe still reads the agent factor atR2=0\.73R^\{2\}=0\.73from the same checkpoints \(Sec\.[4\.2](https://arxiv.org/html/2608.06706#S4.SS2)\)\. The channel is preserved but unused: when distractor variance dominates the prediction target, the one\-step objective is nearly indifferent to the action, and the predictor shrinks its action sensitivity toward zero\. A consumer with a longer horizon can restore it at small scale: in the FourRooms grid, the gridworld planner, trained with a multi\-step rollout loss, reaches0\.92±0\.020\.92\\pm 0\.02goal\-reaching success atn=30n\{=\}30where the standard predictor manages0\.57±0\.030\.57\\pm 0\.03\(Sec\.[4\.2](https://arxiv.org/html/2608.06706#S4.SS2)\); this conversion does not replicate at DMC scale \(below\)\. The practical corollary is the AS\-gated checkpoint\-selection rule of Sec\.[4\.1](https://arxiv.org/html/2608.06706#S4.SS1); we suspect the same selection pathology affects other latent world models trained under heavy distraction\.

#### The longer\-horizon consumer does not restore control at DMC scale\.

The multi\-step planner converts the preserved channel into behavior in FourRooms, but the conversion does not replicate at DMC scale within our compute budget\. We fixed a pre\-registered gate—beat a random\-action reference at goal tolerances0\.05/0\.10\.05/0\.1—before running, ported the gridworld recipe to reacher\-easy \(three seeds per variant trained to 200k steps with the same multi\-step rollout loss, planned with CEM MPC\(de Boeret al\.[2005](https://arxiv.org/html/2608.06706#bib.bib44)\)at 30 episodes per run, five of the six runs evaluated, and scored by the same position\-decoding compass head the gridworld planner uses—privileged, and used only to score candidate plans\), and missed the gate across the full effort ladder\. The outcome is not merely at\-chance: at the tolerances where a random policy has traction, the centered planner is*sub\-random*\(goal\-reaching success0\.03/0\.100\.03/0\.10vs\.0\.23/0\.570\.23/0\.57at tolerance0\.2/0\.30\.2/0\.3, two seeds\)—planning through the model is worse there than not planning—while at the pre\-registered tight tolerances both planner and random are at zero, and the aggregate final distance matches random \(2\.352\.35vs\.2\.442\.44\)\. The standard ablation is strictly worse: zero success at*every*tolerance across all three of its seeds \(0/900/90episodes\) at mean final distance4\.644\.64\. The null holds across4×4\\timestraining \(50​k→200​k50\\mathrm\{k\}\\\!\\to\\\!200\\mathrm\{k\}steps\),32×32\\timesplanning budget \(shipped→\\tohuge CEM\), the multi\-step rollout loss, and oracle scoring; on the earlier 50k checkpoints performance additionally*degraded*as the planning horizon grew—the signature of search through a drifting model\. Because the compass scorer is privileged—an upper bound on the scoring information any reward\-free readout could supply—the null is generous\-case: the planner lost even with the best available plan scorer, which forecloses a “needs a better reward” rescue\.

#### The null is consistent with a model\-accuracy bottleneck, not a channel defect\.

Within an episode the centered planner transiently steers the right way—best in\-episode distance falls to∼0\.15\{\\sim\}0\.15—before drifting back out to final distances of∼1\{\\sim\}1–55: the expected signature of CEM planning through an inaccurate model, not of a channel that misdirects\. Consistent with a representation that is not itself the bottleneck, the centered readout is the more accurate open\-loop model: five\-step decoded\-state rollout error \(standardized\) is0\.620\.62for centered vs\.0\.740\.74for standard, non\-overlapping across three seeds; but both sit far above the sub\-0\.40\.4target we pre\-set when porting the recipe, and both drift upward late in training\. We therefore read the null as a*candidate*model\-accuracy limit rather than a proven one\. A direct intervention on the same frozen checkpoints corroborates this localization: pushing the latent along the model’s own centered channel reproduces the executed action’s counterfactual state effect on cheetah\-run under independent distraction \(cf\-cos≈0\.27\{\\approx\}0\.27atγ=1\\gamma\{=\}1vs\.≈0\.01\{\\approx\}0\.01for an equal\-norm random push through the same decoder\), where the noncentered and standard channels—which steer comparably*without*distractors—collapse to≈0\{\\approx\}0; the effect is monotone in the push scale and respects the action\-correlated boundary \(Appendix[G](https://arxiv.org/html/2608.06706#A7)\)\. The failure is also distinct from the out\-of\-distribution\-goal artifact of Appendix[C](https://arxiv.org/html/2608.06706#A3): there the goal state was absent from the training marginal; here the goal is in\-distribution and the rollout itself drifts\.

## Appendix IScope of the Claim

#### What the identity does—and does not—guarantee\.

Proposition[1](https://arxiv.org/html/2608.06706#Thmproposition1)removes whatever the*learned offsets*represent identically across actions; it does not by itself guarantee that all action\-independent distractors are removed\. If the offset head learns a zero\-mean action interaction,Δ​\(z,a\)=h​\(z\)\+sdist​\(z\)​ϕ​\(a\)\\Delta\(z,a\)=h\(z\)\+s\_\{\\mathrm\{dist\}\}\(z\)\\,\\phi\(a\)with𝔼a′∼ν​ϕ​\(a′\)=0\\mathbb\{E\}\_\{a^\{\\prime\}\\sim\\nu\}\\,\\phi\(a^\{\\prime\}\)=0, centering leavessdist​\(z\)​ϕ​\(a\)s\_\{\\mathrm\{dist\}\}\(z\)\\,\\phi\(a\)intact even though the distractor’s environmental dynamics never depend on the action\. Invariance to a distractor is therefore a joint property of the identity and of what the trained offsets happen to represent—which is precisely what the channel\-purity probes measure rather than assume\. Empirically, such interactions do not arise where the nuisance is action\-independent: centered\-channel leak is indistinguishable from zero \(Secs\.[4\.3](https://arxiv.org/html/2608.06706#S4.SS3),[4\.2](https://arxiv.org/html/2608.06706#S4.SS2)\)\.

#### Sufficiency, not superiority\.

The representation\-to\-control conversion is demonstrated in the13×1313\\times 13FourRooms gridworld \(centered0\.920\.92vs\. standard0\.570\.57atn=30n\{=\}30; Sec\.[4\.2](https://arxiv.org/html/2608.06706#S4.SS2)\) and does not replicate at DMC scale at this compute budget; we treat it as a bounded existence proof that the preserved channel*can*convert to control where the world model is accurate enough to plan through—which our gridworld models are and our continuous\-control models, at this budget, are not—and never as a control\-superiority claim at scale\.

#### Single\-agent scope\.

Centering marginalizes over the ego agent’s action alone\. Entities driven by their own policies cancel only while their transitions are common\-mode with respect to that action; the moment they react to the agent—pursuit, evasion, coordination—they become action\-correlated and inherit the reafferent boundary\. A multi\-agent extension would center over joint actions or per\-agent marginals, which presupposes observing or inferring the other agents’ actions—exactly the supervision our action\-labeled setting assumes only for the ego agent\.

## Appendix JSupplementary Figures and Tables

This section collects the floats relocated from the main text for the page limit, in ascending float number\. Each is reproduced exactly as generated; the surrounding text says only what the float contains and how to read it\.

We begin with the two natural\-distraction testbeds\. Table[4](https://arxiv.org/html/2608.06706#A10.T4)gives the Freeway channel\-purity numbers behind the discussion in the main text: the agent’s own displacement read out of the effective offset, scored against the one\-hot\(action\) ceiling, alongside the traffic leak measured against both the traffic change and the traffic state\.

agent own\-Δ\\DeltaR2↑R^\{2\}\\uparrowtrafficR2↓R^\{2\}\\downarrowseparation↑\\uparrowfeaturesfs 2fs 4fs 8\(Δ\\Delta& state\)\(mean±\\pms\.d\.\)one\-hot\(a\)\(a\)ceiling0\.8100\.7570\.683——centered\(ours\)0\.8770\.8380\.795\[−0\.001,0\.000\]\[\-0\.001,0\.000\]\+0\.837±0\.036\\mathbf\{\+0\.837\{\\pm\}0\.036\}excess†\+0\.19\+0\.19–0\.230\.23\(9/9\)gated0\.8250\.8510\.788‡\[−0\.001,0\.000\]\[\-0\.001,0\.000\]\+0\.736±0\.272\+0\.736\\pm 0\.272noncentered0\.1490\.149–0\.4850\.4850\.5450\.545–0\.6120\.612−0\.256±0\.302\-0\.256\\pm 0\.302full latentzz\(any var\.\)—≈0\.87\\approx 0\.87—Table 4:Freeway channel purity\(end\-of\-training checkpoints, 3 seeds per frameskip; the 3/36 collapsed runs are kept in pooled separation statistics, excluded from per\-frameskip aggregates\)\. RidgeR2R^\{2\}from the effective offset for the agent’s displacement \(read against the one\-hot ceiling\) and the traffic factors; per the predictability control the leak column covers traffic*state*as well asΔ\\Delta\. Separation==agentR2R^\{2\}−\-traffic\-stateR2R^\{2\}per run\.†Excess==agentR2R^\{2\}after regressing out one\-hot\(a\)\(a\)\.‡Gated fs 8: mean of the two healthy seeds\. Bottom row: traffic state decoded from the*full latent*zzrather than the offset—rejection is routing, not blindness\.Table[5](https://arxiv.org/html/2608.06706#A10.T5)is the continuous\-control counterpart, covering three DMC tasks under the three distractor regimes\. The independent column is the one the claim rests on; the correlated column is the stated boundary, and reacher\-easy’s noncentered cell there is the contamination\-as\-signal case discussed above\.

Table 5:Distracting DMC\(3 seeds; per\-seed values and seed s\.d\. in Table[11](https://arxiv.org/html/2608.06706#A10.T11)—s\.d\.≤0\.014\\leq 0\.014outside reacher\-easy, whose correlated cells reach0\.0940\.094\)\. ProbeR2R^\{2\}of the controllable state change from the effective offset\. Under independent distraction \(bold\) the centered channel survives on all tasks \(9/9 seeds\); the correlated column is the stated boundary, and reacher’s noncentered correlated cell is contamination\-as\-signal\.![Refer to caption](https://arxiv.org/html/2608.06706v1/figs/frame_pair.png)Figure 4:What “distraction” is, concretely\(FourRooms observations, rendered from the environment\)\.*Left:*n=0n\{=\}0\.*Right:*n=30n\{=\}30rolling distractor cells \(blue\)\. The room, the agent \(orange\) and the goal \(green\) are identical; only the distractor field differs, and it rolls one column per step*regardless of the action*\. That per\-transition invariance across actions is what the action mean absorbs\. In the action\-correlated configuration the same field instead rolls by the agent’s own displacement, which is the boundary of the guarantee\.Figure[5](https://arxiv.org/html/2608.06706#A10.F5)shows the same contrast for continuous control, at the exact settings every DMC run used\.

![Refer to caption](https://arxiv.org/html/2608.06706v1/figs/dmc_pair.png)Figure 5:The same, in continuous control\(cheetah\-run observations, rendered from the environment at the settings every DMC run used:64×6464\{\\times\}64,1616occluders\)\.*Left:*clean\.*Right:*the same transition under distraction\. Both panels are stepped with the same action sequence and the occluders are composited onto the pixels only, so the physics—and with it the probe target—is bit\-identical between them while13\.6%13\.6\\%of pixels change\. In the independent configuration the occluders translate a fixed22px per frame whatever the action; in the action\-correlated configuration they translate by the action instead\. A single frame cannot tell the two apart: they differ only in the motion rule, which is precisely the property the action mean is sensitive to\.The next group covers the synthetic generator, where the controllable and nuisance factors are known by construction and both halves of the claim can be scored directly\. Table[6](https://arxiv.org/html/2608.06706#A10.T6)summarises recovery and rejection across the full grid\.

Table 6:Synthetic identifiability: ranges of held\-out probeR2R^\{2\}across the 120\-cell grid \(3 seeds per cell\)\.†At nuisance dimensiondw≥16d\_\{\\mathrm\{w\}\}\\geq 16; recovery fails outright, so rejection is moot \(cell not reported\)\.−0\.00\-0\.00denotes a value that is negative and rounds to zero; held\-outR2R^\{2\}can be slightly negative\.Table[7](https://arxiv.org/html/2608.06706#A10.T7)gives the gridworld sweep the main text quotes, including the action\-correlated configuration at which both variants fail\.

Table 7:Gridworld distractor sweep\(held\-out probeR2R^\{2\}, agent factor from the effective offset; 3 seeds\)\. Both variants die under action\-correlated d30ac—the stated boundary\.Figure[6](https://arxiv.org/html/2608.06706#A10.F6)plots the same synthetic study against the nuisance dimension, so the two requirements can be read against each other as the distractor grows\. The curves sweepdwd\_\{w\}on the*linear*generator; the×\\timesmarkers, offset slightly to the right ofdw=16d\_\{w\}\{=\}16, carry the action\-correlated configuration at that same dimension, so the boundary can be compared against the action\-independent case without a second panel\. Reading the two panels together separates the two ways a channel can fail\. Panel \(a\) is the recovery requirement: both centered variants hold at0\.910\.91–0\.930\.93across the sweep, and the noncentered ablation recovers just as well, so recovery alone does not discriminate between them\. Panel \(b\) is the rejection requirement, and there the two separate completely—the centered variants sit at−0\.00\-0\.00at every dimension while the noncentered offset carries the distractor at0\.880\.88–0\.950\.95\. The standard predictor fails the first requirement outright rather than the second: its recovery decays from0\.880\.88to0\.020\.02asdwd\_\{w\}grows, which is why its rejection column is not informative and is left unreported in Table[6](https://arxiv.org/html/2608.06706#A10.T6)\. At the boundary markers, rejection breaks for the centered variants as well \(−0\.00→0\.17\-0\.00\\to 0\.17\), which is the same failure the gridworld sweep shows under its action\-correlated configuration\.

![Refer to caption](https://arxiv.org/html/2608.06706v1/x4.png)Figure 6:Synthetic identifiability on the*linear*generator: recover \(Rc2R^\{2\}\_\{\\mathrm\{c\}\}, top\) versus reject \(Rw2R^\{2\}\_\{\\mathrm\{w\}\}, bottom\) against the nuisance dimensiondwd\_\{w\}\. The×\\timesmarkers are the action\-correlated boundary atdw=16d\_\{w\}\{=\}16\. Error bars: s\.e\.m\. over 3 seeds\. Values are discussed in the text; the nonlinear generator and the full120120\-cell grid are in Table[6](https://arxiv.org/html/2608.06706#A10.T6)\.Two controls follow\. Table[8](https://arxiv.org/html/2608.06706#A10.T8)varies how predictable the nuisance dynamics are, separating a leak of the nuisance*change*from a leak of the nuisance*state*—the distinction that makes the leak metric informative when the nuisance is stochastic\.

Table 8:Predictability control: leak of the action channel under increasingly stochastic nuisance dynamics\.Δ\\Delta\-leak vanishes for the noncentered ablation as the nuisance becomes unpredictable, but its state\-leak persists: the channel still encodes where the nuisance*is*\. The centered variant is clean at every predictability level\. Entries are means over 3 seeds\.Table[9](https://arxiv.org/html/2608.06706#A10.T9)repeats the recover\-and\-reject test in the continuous\-action regime, where the action mean is a Monte\-Carlo estimate rather than an exact enumeration\.

Table 9:Stage 0 \(continuous actions\): sampled centering \(K=16K\{=\}16; entries are means over 3 seeds\)\. The discrete\-case pattern survives Monte\-Carlo estimation ofΔ¯\\bar\{\\Delta\}; an action\-correlated nuisance breaks rejection, marking the common\-mode boundary rather than an estimator failure\.The remaining floats concern behaviour rather than representation\. Table[10](https://arxiv.org/html/2608.06706#A10.T10)reports goal\-reaching success under latent MPC across the gridworld sweep\.

Table 10:Gridworld control: the planner converts the preserved channel into control\.Goal\-reaching success under latent MPC \(FourRooms; variants run here: standard, centered, gated; 3 seeds, mean±\\pms\.e\.m\.\)\. Both centered variants stay near ceiling across the sweep while the standard predictor degrades\. Under the action\-correlated d30ac configuration, all three variants fall to0\.550\.55–0\.570\.57— the same boundary as the probe sweep\.Figure[7](https://arxiv.org/html/2608.06706#A10.F7)places the probe and control curves side by side, so the representation boundary and the control boundary can be compared directly\.

![Refer to caption](https://arxiv.org/html/2608.06706v1/x5.png)Figure 7:Gridworld control across the distractor sweep\.Decomposed variants hold near ceiling where the standard predictor degrades; representation and control break at the same action\-correlated point\. Variants shown: standard, centered, gated; error bars are s\.e\.m\. over 3 seeds\. Legends use*dueling*for centered\.Figure[8](https://arxiv.org/html/2608.06706#A10.F8)shows the Freeway readout across frameskips in the same format\.

![Refer to caption](https://arxiv.org/html/2608.06706v1/x6.png)Figure 8:Freeway\.Agent vs\. traffic probeR2R^\{2\}from the effective offset across frameskips \(dotted==one\-hot ceiling\): centered decodes the agent above the ceiling with traffic leak pinned at zero; noncentered inverts the pattern\. Error bars: s\.d\. over 3 seeds\.Finally, Table[11](https://arxiv.org/html/2608.06706#A10.T11)lists every DMC run individually, so the per\-seed spread behind the aggregated cells above can be inspected directly\.

Table 11:Per\-seed effective\-offset probeR2R^\{2\}for the four variants run at DMC scale \(standard, noncentered, centered, gated\)×\\times3 tasks×\\times3 distractor regimes \(108108runs\); the*residual*rung of the main\-text ladder was not run in this matrix\. The*standard*predictor is monolithic and exposes no offset channel, so it has no entry here\. Seed s\.d\. is at most0\.0140\.014outside reacher\-easy; within reacher\-easy the correlated cells reach0\.1840\.184\(gated\),0\.0940\.094\(noncentered\) and0\.0670\.067\(centered\), and the distraction\-free cells reach0\.0520\.052\(gated\) and0\.0300\.030\(centered\)\. The independent\-distraction column—the one the claim rests on—is tight everywhere \(s\.d\.≤0\.013\\leq 0\.013\)\.