Predictive Closure: A Set-Level Characterization of JEPA's Feature Selection

# Predictive Closure: A Set-Level Characterization of JEPA’s Feature Selection
Source: [https://arxiv.org/html/2610.27364](https://arxiv.org/html/2610.27364)
Mosh LevyAffiliation:Bar\-Ilan UniversityAffiliation:University of California, BerkeleyEmail:[moshe0110@gmail\.com](mailto:)Yoav GoldbergAffiliation:Bar\-Ilan UniversityAffiliation:Allen Institute for AI

###### Abstract

In joint\-embedding predictive architectures \(JEPA\), an encoder and a predictor are trained together to produce a representation that minimizes the error in predicting the representation of the next observation\. Because the representations of both the current and the next observation are learned, it is unclear which features they will include\. Earlier characterizations describe the features JEPA selects in terms of their individual predictability, raising the concern that unpredictable features, like a traffic light, will not be encoded into the representation\. However, the prediction error is also affected by the relationships between features, e\.g\., the light’s color helps predict whether the traffic will stop\. We propose thepredictive closurecharacterization, which accounts for these relationships\. Based on a reformulation of the JEPA training objective, we show that selection depends on properties of feature*sets*\. This view agrees with earlier works when features evolve independently, but makes new predictions when they do not, for instance that a feature like the traffic light can still be retained\. We test this characterization in controlled experiments, which show that unpredictable features can become better represented when they help predict other features, and that changing one feature’s predictive relationships can affect how well others are represented\. We also demonstrate implications for steering feature selection: targeting one feature with an auxiliary objective, or supplying it to the predictor as side information, can strengthen the representation of another feature whose future values it helps predict, so steering can be indirect\.

## 1Introduction

We argue for a set\-level characterization of how JEPA selects which features to encode\. A joint\-embedding predictive architecture \(JEPA\) aims to learn representations useful for planning and other downstream tasks, without explicitly specifying which features those representations should include\([LeCun, 2022](https://arxiv.org/html/2610.27364#bib.bib7)\)\. JEPA\-based systems have shown strong performance across a range of applications\([Assran et al\., 2025](https://arxiv.org/html/2610.27364#bib.bib26);[Maes et al\., 2026](https://arxiv.org/html/2610.27364#bib.bib17);[Rao et al\., 2026](https://arxiv.org/html/2610.27364#bib.bib9);[Larey et al\., 2026](https://arxiv.org/html/2610.27364#bib.bib27);[Zhu and Choromanska, 2026](https://arxiv.org/html/2610.27364#bib.bib18);[Kuhn et al\., 2026](https://arxiv.org/html/2610.27364#bib.bib20)\), yet which features they encode is still far from obvious\. This matters because their representation acts as the world model JEPA builds for each observation\([García et al\., 1989](https://arxiv.org/html/2610.27364#bib.bib34);[Schmidhuber, 1990](https://arxiv.org/html/2610.27364#bib.bib5);[Sutton, 1991](https://arxiv.org/html/2610.27364#bib.bib33);[Ha and Schmidhuber, 2018](https://arxiv.org/html/2610.27364#bib.bib6)\), so features absent from it cannot inform planning or other downstream tasks, however accurate JEPA’s predictions are\.

Since JEPA’s encoder determines both sides of the prediction problem, JEPA learns what to predict together with how to predict it\. The encoder maps each observation \(for example, three consecutive frames from a video\) to a latent representation and a predictor that uses the current observation’s representation to forecast the representation of the next observation\. The training objective combines a prediction loss with regularization designed to prevent the latent representation from collapsing to the same point for every input\. With a limited latent size, the encoder must leave some features out when the observation has more than fit\. Will JEPA then represent a feature whose next value is hard or impossible to predict, such as a traffic light?

![Refer to caption](https://arxiv.org/html/2610.27364v1/jepa_features.png)Figure 1:Will the traffic light be included in the latent representation?The top row shows successive observations, and the rows below show their latent representations with the light kept or dropped; crosses and dashed frames mark the resulting prediction errors\. In the first step, the light changes color at random, so keeping it costs an error on its own next color\. In the second step, the light’s current color helps predict whether the cyclist will ride or stop next, so dropping the light costs an error on the cyclist\. Earlier characterizations judge the light only by whether its own next color can be predicted, and so would drop it; they disregard what its current color contributes to predicting the cyclist’s next move, which is what can make the light worth keeping\. Cyclist artwork adapted from Scott McCloud’s*Understanding Comics*\(1993\)\.In recent work,[Pendharkar \(2026\)](https://arxiv.org/html/2610.27364#bib.bib8)argues that unpredictable features such as the traffic light will not be represented\. This is consistent with the characterizations of feature selection in other works that portray it as dependent on the feature’s own rate of change\([Sobal et al\., 2022](https://arxiv.org/html/2610.27364#bib.bib13);[Karmann et al\., 2026](https://arxiv.org/html/2610.27364#bib.bib25)\)or predictability\([Littwin et al\., 2024](https://arxiv.org/html/2610.27364#bib.bib12);[Enigma Team, 2026](https://arxiv.org/html/2610.27364#bib.bib4)\)\. All of these works judge each feature by its own behavior\. In this paper, we argue that such features can still be represented because, while hard to predict, they can help predict other features that JEPA encodes: the traffic light’s current color helps predict whether a cyclist waiting at it will ride or stop next \(Figure[1](https://arxiv.org/html/2610.27364#S1.F1)\)\. Reasoning about one feature at a time can then feel like chasing one’s own tail: what is useful to represent depends on what must be predicted, while what must be predicted depends on what is represented\.

To account for such relationships, we propose a novel characterization namedpredictive closure, which judges sets of features as wholes rather than each feature on its own\. It is based on a reformulation of the JEPA training objective in which selection depends on*joint*properties of the selected*set*of features \(Section[3](https://arxiv.org/html/2610.27364#S3)\)\. The reformulation shows that among the sets the latent can represent, the objective favors the set with the greatest mutual information between the current and the next values of its features\. This means that what a feature adds to JEPA’s objective depends on the other features in the set\. Adding it raises the mutual information if its current value helps predict their next values, or if its next value can be predicted from their current values\. The traffic light can therefore belong to the set the objective favors, even though its own next state is impossible to predict\.

Predictive closure explains what earlier works found about the features JEPA encodes, and agrees with them when features evolve independently\. When features help predict other features, it makes new predictions, including about steering methods \(Section[4](https://arxiv.org/html/2610.27364#S4)\)\. We construct image sequences with controlled predictive relationships, train JEPA systems on them with a latent too small to hold every feature, and use probes to measure which features the latent keeps \(Section[5](https://arxiv.org/html/2610.27364#S5)\)\. First, we find that an unpredictable traffic\-light\-like feature becomes more decodable as its usefulness for predicting other features increases\. Second, strengthening one predictive connection within a set of features that help predict one another can increase its members’ representation while reducing that of a competing set whose predictive relationships are unchanged\. Third, even a highly predictive set of features can remain poorly represented if it requires more capacity than the latent provides\.

Predictive closure further predicts that targeting one feature during training affects the representation of its predictive partners and competitors\. We test these steering predictions in Section[6](https://arxiv.org/html/2610.27364#S6)\. We find that adding an auxiliary loss to reconstruct a single image can strengthen the representation of its untargeted predictive partner while reducing that of competing feature sets\. We also find that supplying a feature directly to the predictor strengthens the representation of some features while weakening others\. This suggests using side information from observations to steer feature selection, even when it is not an action \(Section[6\.3](https://arxiv.org/html/2610.27364#S6.SS3)\)\. Both interventions can strengthen a predictive partner of the targeted feature, so steering can be indirect\. If we view actions as side information, predictive closure also suggests that action\-conditioned JEPA favors the features an agent has an impact on\.

## 2Background

A joint\-embedding predictive architecture \(JEPA\) learns a representation by predicting the latent representation of the next observation\. Unlike reconstruction\-based approaches, which train the latent to compress the information needed to recover the input\([Kingma and Welling, 2014](https://arxiv.org/html/2610.27364#bib.bib2)\), JEPA does not impose this requirement, so its latent can omit some features of the observation\([Van Assel et al\., 2025](https://arxiv.org/html/2610.27364#bib.bib39)\)\. This raises the question of which features JEPA will encode, which is the main focus of this paper\.

### 2\.1JEPA Doesn’t Explicitly Specify Features

JEPA includes an encoderffthat maps an observationxtx\_\{t\}at timettto a latent representationf⁡\(xt\)f\(x\_\{t\}\), and a predictorggthat takesf⁡\(xt\)f\(x\_\{t\}\)as input to predictf⁡\(xt\+1\)f\(x\_\{t\+1\}\)\(Figure[1](https://arxiv.org/html/2610.27364#S1.F1)\)\. Training combines a prediction loss on observation pairs with a regularization loss on the latent distribution\([LeCun, 2022](https://arxiv.org/html/2610.27364#bib.bib7)\):

ℒJEPA≜𝔼​‖g⁡\(f⁡\(Xt\)\)−f⁡\(Xt\+1\)‖22\+λ​ℛ​\(f⁡\(Xt\)\)\.\\mathcal\{L\}\_\{\\mathrm\{JEPA\}\}\\triangleq\\mathbb\{E\}\\left\\\|g\(f\(X\_\{t\}\)\)\-f\(X\_\{t\+1\}\)\\right\\\|\_\{2\}^\{2\}\+\\lambda\\mathcal\{R\}\(f\(X\_\{t\}\)\)\.\(1\)Here,XtX\_\{t\}denotes the random observation at timett, and the expectation is over consecutive observation pairs\. The parameterλ\>0\\lambda\>0controls the weight of the regularizerℛ\\mathcal\{R\}, which acts on the distribution off⁡\(Xt\)f\(X\_\{t\}\)and encourages it to match a chosen distribution to prevent collapse \(such as the distributions proposed in[Balestriero and LeCun, 2025](https://arxiv.org/html/2610.27364#bib.bib21);[Wu et al\., 2026](https://arxiv.org/html/2610.27364#bib.bib22);[Kuang et al\., 2026](https://arxiv.org/html/2610.27364#bib.bib24);[Nicollier et al\., 2026](https://arxiv.org/html/2610.27364#bib.bib23)\)\.

Within the bounds of the enforced distribution, the objective does not specify what the latent should contain, and because the encoder produces both the predictor’s input and its target, the predictor adjusts to whatever the encoder produces\.

To describe what the latent represents, we use the concept of features\.111An alternative way to describe the latent is to ask which differences in the observation result in the same representation, so that the latent is defined by what it omits\([Li et al\., 2006](https://arxiv.org/html/2610.27364#bib.bib29)\)\.We define features as variables of the observation, some of which correspond to named properties such as the traffic light’s color or the cyclist’s speed, and we say that the latent represents a feature when the feature’s value can be recovered from the latent\. See Appendix[A\.1](https://arxiv.org/html/2610.27364#A1.SS1)for the formal definition that we use in our proofs\. Features allow us to treat the latent as representing separate pieces of information and to leave aside how it encodes them together\. Such a description makes it tempting to think about the latent one piece at a time and to overlook the relationships between the pieces\. It is nonetheless a common way to describe information\-processing systems, both biological\([Hubel and Wiesel, 1962](https://arxiv.org/html/2610.27364#bib.bib40);[Marr, 1982](https://arxiv.org/html/2610.27364#bib.bib38);[Hung et al\., 2005](https://arxiv.org/html/2610.27364#bib.bib41)\)and artificial\([Bengio et al\., 2013](https://arxiv.org/html/2610.27364#bib.bib16);[Alain and Bengio, 2016](https://arxiv.org/html/2610.27364#bib.bib42)\), and it is also how earlier work has characterized the information JEPA stores, as we review next\.

### 2\.2Earlier Characterizations of Feature Selection

Some earlier works characterize JEPA as focusing on slow features\([Sobal et al\., 2022](https://arxiv.org/html/2610.27364#bib.bib13)\), as encoding the most predictable features\([Enigma Team, 2026](https://arxiv.org/html/2610.27364#bib.bib4)\), or as discarding unpredictable ones\([Pendharkar, 2026](https://arxiv.org/html/2610.27364#bib.bib8)\)\. Other works describe the information selected into the latent as a subspace of the observation space\([Tang et al\., 2023](https://arxiv.org/html/2610.27364#bib.bib10);[Littwin et al\., 2024](https://arxiv.org/html/2610.27364#bib.bib12);[Khetarpal et al\., 2024](https://arxiv.org/html/2610.27364#bib.bib15);[Van Assel, 2026](https://arxiv.org/html/2610.27364#bib.bib1)\), under two assumptions: that the encoder and predictor are linear, and that features evolve independently, so the current value of each feature carries no information about the next values of the others\.

These per\-feature descriptions disregard the predictive relationships*between features*and assume that each feature carries information about its own future only, not about the future of other features\. In the characterization we propose in this work, we take this relation into account by assessing features jointly\.

## 3Predictive Closure

We reformulate JEPA’s prediction objective as selecting*a feature set*whose present values provide the most information about*its own future*\(known as*predictive information*\) while remaining feasible under the regularization applied to the latent representation\. This yields the characterization we callpredictive closure\(Section[3\.1](https://arxiv.org/html/2610.27364#S3.SS1)\), which we then decompose into two terms that give a more interpretable view of the selection \(Section[3\.2](https://arxiv.org/html/2610.27364#S3.SS2)\)\.

### 3\.1JEPA Selects Sets, Not Individual Features

We begin by identifying which feature sets JEPA can select\. We writeSSfor*a set of features*,Yt​\(S\)Y\_\{t\}\(S\)for their*joint values*in an observation at timett, which together form one random variable, andS⁡\(f\)S\(f\)for the*complete set*of features recoverable from the latent of an encoderff\. Our first observation is that JEPA’s regularization loss constrainsYt​\(S​\(f\)\)Y\_\{t\}\(S\(f\)\)to a specific entropy \(Section[2\.1](https://arxiv.org/html/2610.27364#S2.SS1)\), and we call a set of featuresSSfeasibleif its values have exactly this entropy \(Appendix[A\.1](https://arxiv.org/html/2610.27364#A1.SS1)gives the full condition\)\. The question is then what determines which feasible set JEPA’s training objective selects\.

We prove that the setS⁡\(f⋆\)S\(f^\{\\star\}\)represented by an ideally trained encoderf⋆f^\{\\star\}, one that minimizes the training objective, is a feasible set with the greatest predictive informationIpredI\_\{\\mathrm\{pred\}\}\([Bialek and Tishby, 1999](https://arxiv.org/html/2610.27364#bib.bib11)\):

S⋆≔S⁡\(f⋆\)∈arg​maxS​feasible⁡I⁡\(Yt​\(S\),Yt\+1​\(S\)\)⏟Ipred​\(S\)\.\\boxed\{S^\{\\star\}\\coloneqq S\(f^\{\\star\}\)\\in\\operatorname\*\{arg\\,max\}\_\{S\\ \\mathrm\{feasible\}\}\\underbrace\{I\\bigl\(Y\_\{t\}\(S\);Y\_\{t\+1\}\(S\)\\bigr\)\}\_\{I\_\{\\mathrm\{pred\}\}\(S\)\}\.\}\(2\)
The optimization can be thought of as seeking a set that is most “closed under prediction”, meaning that the current values of its features determine their next values\. We call the feasible set that optimizes this thepredictive closure\.

#### Proof sketch\.

With the entropy of the latent’s distribution fixed, the prediction loss can be lowered only by making the next latent less uncertain given the current one, which increases the mutual information between them\. Because knowing the values ofS⁡\(f\)S\(f\)is the same as knowing the latent, the mutual information between consecutive latents equals the predictive information ofS⁡\(f\)S\(f\)\. Maximizing it over encoders is therefore the same as maximizing predictive information over feasible sets\. See Appendix[A](https://arxiv.org/html/2610.27364#A1)for the full proof\.

Intuitively, adding a feature to a set of features raises the set’s predictive information if the feature’s value at timetthelps predict the values of the other features in the set at timet\+1t\+1, or if its value at timet\+1t\+1can be predicted from their values at timett\. Here, one value helps predict another when the two have high mutual information\.

The earlier characterizations in Section[2\.2](https://arxiv.org/html/2610.27364#S2.SS2)and predictive closure both imply that JEPA keeps features that are predictable and help predict others, and omits features that are unpredictable and help predict nothing, but they differ fundamentally in how the choice is made\. Individual selection amounts to ranking the list of features and keeping the topnnthat fit in the latent, whereas predictive closure requires comparing alternative sets and not alternative features, because what a feature adds to the predictive information of a set depends on which other features the set contains\. The two views coincide when features evolve independently of one another: each feature then adds the same predictive information and takes up the same capacity whatever else is represented, so features compete only for capacity and ranking them is enough \(see Appendix[A\.5](https://arxiv.org/html/2610.27364#A1.SS5)for details\)\. Predictive closure makes new predictions when features are predictively related, in the sense that the current values of some features carry information about the value of another at the next time step\.

Because features are assessed jointly, neither how easily a feature can be predicted nor how much information it provides determines by itself whether it belongs to the predictive closure\.A featureϕ\\phiwill be represented if it is a member of some feasible feature setSϕS\_\{\\phi\}with greater predictive information than any other feasible set\.These competing sets can differ in many features besidesϕ\\phi, and some may share only constant features\. Whether one part of an observation is represented can therefore depend on whether another part is represented, even when neither helps predict the other\.

### 3\.2Aleatoric Uncertainty and Predictive Leakage

We decompose the predictive information to gain further insight into the feature selection objective:

S⋆∈arg​minS​feasible⁡\[H⁡\(Yt\+1​\(S\)∣Xt\)⏟aleatoric uncertainty\+I⁡\(Xt;Yt\+1​\(S\)∣Yt​\(S\)\)⏟predictive leakage\]\.S^\{\\star\}\\in\\operatorname\*\{arg\\,min\}\_\{S\\ \\mathrm\{feasible\}\}\\Bigl\[\\underbrace\{H\\bigl\(Y\_\{t\+1\}\(S\)\\mid X\_\{t\}\\bigr\)\}\_\{\\text\{aleatoric uncertainty\}\}\+\\underbrace\{I\\bigl\(X\_\{t\};Y\_\{t\+1\}\(S\)\\mid Y\_\{t\}\(S\)\\bigr\)\}\_\{\\text\{predictive leakage\}\}\\Bigr\]\.\(3\)The aleatoric uncertainty is how uncertain the set’s next valuesYt\+1​\(S\)Y\_\{t\+1\}\(S\)remain even when the full current observationXtX\_\{t\}is known\. The predictive leakage is how well the current observation predicts the set’s next values beyond what can be predicted from the set’s current values, and it is zero when the set’s features together predict all of their next values\. See Appendix[A\.4](https://arxiv.org/html/2610.27364#A1.SS4)for the derivation\.

This decomposition gives us more language to describe the choice of the best set: a feature whose current value helps predict the set’s next values lowers the set’s predictive leakage, and one whose next value is determined by the set’s current values does not add to either term\. Conversely, a noisy feature only adds to the aleatoric uncertainty unless it helps predict other features in the set, so a noisy TV that helps predict nothing\([Burda et al\., 2019](https://arxiv.org/html/2610.27364#bib.bib3)\)will not be represented\.

## 4Implications of Predictive Closure

We now show how predictive closure explains earlier empirical findings on which features JEPA\-based systems encode and makes predictions about what features will be selected in scenarios where features are predictively related\. Throughout this section we reason with Equation \([3](https://arxiv.org/html/2610.27364#S3.E3)\): the encoder favors features that add little to its sum\.

### 4\.1Explaining Earlier Findings

[Sobal et al\. \(2022\)](https://arxiv.org/html/2610.27364#bib.bib13)found that a moving object was better represented when its background was independently resampled between observations than when it remained fixed within each sequence\. A fixed background adds nothing to the sum, since its next values equal its current ones, while a resampled one adds its entire entropy as aleatoric uncertainty and is left out in favor of the moving object\.[Enigma Team \(2026\)](https://arxiv.org/html/2610.27364#bib.bib4)found that even a small pattern added to the observations took over most of the latent when held fixed within each sequence, and was ignored when resampled at every observation\. Here too the resampled pattern adds its entire entropy, while every feature of the fixed pattern, however fine, adds nothing, so a set made only of such features has the smallest sum and leaves no room for the rest of the scene\.

[Karmann et al\. \(2026\)](https://arxiv.org/html/2610.27364#bib.bib25)found that a JEPA world model encoded the slowly changing score of a Pong game while suppressing the fast\-moving ball, and, like[Sobal et al\. \(2022\)](https://arxiv.org/html/2610.27364#bib.bib13), describe JEPA as biased toward slow features\. This bias follows from the time step betweenttandt\+1t\+1in Equation \([3](https://arxiv.org/html/2610.27364#S3.E3)\): a feature that changes little within one step is almost determined by its current value and adds almost nothing to the sum, while a fast\-changing one can add far more\. What counts as slow therefore depends on the time step, so a feature can be kept at one temporal resolution and omitted at another\.

[Pendharkar \(2026\)](https://arxiv.org/html/2610.27364#bib.bib8)showed that an independently resampled visual cue, appearing over a predictably changing background, was barely decodable from JEPA’s latent despite determining which action earned a reward \(reward was not included in the training objective\)\. The cue only adds to the sum: its next value is pure aleatoric uncertainty, and its current value removes no predictive leakage\.

### 4\.2Predicting Feature Selection

#### Cross\-subsidization\.

An unpredictable featureϕ\\phiadds the aleatoric uncertainty of its own next value to the sum, but ifϕ\\phiis highly predictive of other represented features, including it also removes predictive leakage, which subsidizes this cost\. Hence, predictive closure predicts thatan unpredictable feature will be represented if it is sufficiently predictive of other represented features\. In scenarios like the one depicted in Figure[1](https://arxiv.org/html/2610.27364#S1.F1), where the cyclist’s features are determined by the traffic light and the light itself is unpredictable, the traffic light will be represented whenever the cyclist’s features are\.

#### Set switching\.

Consider a scenario where the observations contain two feature sets with predictive relationships within each set, but not between them, and the latent has the capacity to fully represent either set but not both\. Increasing the strength of the relationships within one set lowers the aleatoric uncertainty of its features’ next values, and once its sum falls below that of the second set, it will be represented instead, even if the second set remains unchanged\. Hence, predictive closure predicts thatstrengthening the predictive relationships within one set of features will eventually displace a disjoint set that competes with it for capacity\. For example, adding sensors for an independent, highly predictable process can displace features of the original observations from the latent\.222When we write of changes to the observations, we consider the counterfactual case where a JEPA system would be trained on the sequence after the change\.

#### Feasibility dependence\.

Equation \([3](https://arxiv.org/html/2610.27364#S3.E3)\) compares only feasible sets, and a set whose entropy exceeds the latent capacity is not feasible; each feasible part of such a set has predictive leakage through the features it leaves out\. Hence, predictive closure predicts thatenlarging a set of mutually predictive features beyond the latent capacity will reduce its representation, even if its features remain as predictable as before, which can lead to an entirely different set being represented\.

#### Steering with an auxiliary loss\.

When adding an auxiliary loss that requires some features, we force those features into the predictive closure, and the remaining features are selected such that together with the fixed features they form the set with the smallest sum\. The included features will be not only those that the auxiliary loss requires, but also their predictive partners: features whose current values help predict the required features’ next values and so reduce predictive leakage, and features whose next values the required features help predict\. Hence, predictive closure predicts thatan auxiliary loss on some features will also strengthen the representation of their predictive partners, at the expense of competing features\. For example, an auxiliary loss for reconstructing the wind measurements can favor representation of a flag’s state\.

#### Steering with side information\.

When supplying the predictor with side informationUtU\_\{t\}that it is not trained to predict \(as is usually done for actions\), both terms of Equation \([3](https://arxiv.org/html/2610.27364#S3.E3)\) are computed givenUtU\_\{t\}as well\. This extends the prediction\-error assumption to condition on both predictor inputs \(see Appendix[A\.6](https://arxiv.org/html/2610.27364#A1.SS6)for the derivation\)\. WhenUtU\_\{t\}is derived from the observation, only the predictive leakage changes: it becomesI\(Xt;Yt\+1\(S\)∣Yt\(S\),Ut\)I\\bigl\(X\_\{t\};Y\_\{t\+1\}\(S\)\\mid Y\_\{t\}\(S\),U\_\{t\}\\bigr\), so whatUtU\_\{t\}tells about the set’s next values no longer leaks\. When it is not, as with actions,UtU\_\{t\}can also lower the aleatoric uncertainty\. Hence, predictive closure predicts that addingside information can strengthen the representation of features whose next values it helps predict, without requiring the side information itself to be represented in the latent\. For example, supplying wind measurements to the predictor can favor representation of a flag’s state\.

## 5Testing the Selection Predictions

We test whether trained JEPA systems follow the three selection predictions from Section[4\.2](https://arxiv.org/html/2610.27364#S4.SS2), in image sequences that are simple by design\. Testing a prediction requires a sequence whose predictive relationships are known and can be changed one at a time, which natural data does not allow\. Each experiment therefore creates a synthetic sequence and changes a single property of it, with the architecture and the training budget held fixed, so that the change in what is represented can be attributed to that property\.

Experimental setup\.333Code:[https://github\.com/Mosh0110/predictive\-closure](https://github.com/Mosh0110/predictive-closure)All experiments train JEPA on constructed observation sequences of CIFAR\-10 image collages\([Krizhevsky, 2009](https://arxiv.org/html/2610.27364#bib.bib36)\)and approximate whether specific*target features*are represented by how accurately probes recover their values from the frozen latent on held\-out observations\. These targets are the traffic\-light state or binary indicators of target\-image presence\. We approximate how well features of the image at each position are represented by averaging accuracy across the binary target\-image detection tasks associated with that position\. We use representation of an image as shorthand for representation of its features\. We use a three\-dimensional latent representation and a training objective combining the prediction loss and SIGReg regularization with coefficients of0\.90\.9and0\.10\.1, respectively\. We choose this small latent so that even these simple sequences exceed its capacity and features must compete, as they do in larger latents on richer observations\. Unless otherwise specified, we train each configuration with three optimization seeds and report mean decoding accuracy, with error bars indicating standard errors across seeds\. Except where noted, our findings are not specific to a single choice ofλ\\lambda\(see sensitivity analysis in Appendix[D](https://arxiv.org/html/2610.27364#A4)\); additional training and probing details are in Appendix[B](https://arxiv.org/html/2610.27364#A2)\. We validate the predictive performance of the traffic\-light and two\-pair trained systems in Appendix[C](https://arxiv.org/html/2610.27364#A3)\.

### 5\.1Cross\-subsidization: representing unpredictable features

#### Scenario\.

We place a two\-valued traffic light alongside two images in a collage\. We writeAAandBBfor the images andLLfor the light’s value, green or red\. At every transition, exactly one image is replaced by a newly sampled image while the other persists\. With probabilityqq, the current lightLtL\_\{t\}determines which position is resampled; otherwise, the position is chosen uniformly \(Figure[2](https://arxiv.org/html/2610.27364#S5.F2), left\)\. Green selectsAAfor replacement and red selectsBB\. The next lightLt\+1L\_\{t\+1\}is an independent uniform draw, so the light is unpredictable and representing it adds one bit of aleatoric uncertainty at every value ofqq, while a largerqqletsLtL\_\{t\}remove more predictive leakage from the images’ next values\.

Figure 2:Cross\-subsidization: the unpredictable light becomes more decodable when it helps predict the images\.Increasingqqmakes the current light more useful for predicting which images persist, while its future remains unpredictable\. Dashed line:50%50\\%chance accuracy\.
#### Results\.

Asqqincreases, the light featureLLbecomes better represented even though its future is impossible to predict \(Figure[2](https://arxiv.org/html/2610.27364#S5.F2)\)\. As predictive closure predicts, features are not selected solely according to how easy they are to predict\. Note that the transition in this scenario involves a nonlinear interaction between the light and the current images, which cannot be captured by frameworks that assume the predictor is a linear function such as[Littwin et al\. \(2024\)](https://arxiv.org/html/2610.27364#bib.bib12)\.

### 5\.2Set Switching

#### Scenario\.

The two\-pair setup places imagesA/BA/BandC/DC/Din a2×22\\times 2collage\. Within each pair, an image can be copied to the other position in the next frame; otherwise, a new image is sampled independently\. We vary only the probabilitypDp\_\{D\}thatCtC\_\{t\}is copied toDt\+1D\_\{t\+1\}while keeping all other copying probabilities fixed \(Figure[3](https://arxiv.org/html/2610.27364#S5.F3), left; Appendix[B\.3](https://arxiv.org/html/2610.27364#A2.SS3)\)\.

Figure 3:Set switching: changing predictive relationships reverses which pair is more strongly represented\.IncreasingpDp\_\{D\}strengthens decoding ofC/DC/Dand weakensA/BA/B, whose predictive relationships remain fixed; neither pair helps predict the other\. Dashed line:50%50\\%chance\.
#### Results\.

IncreasingpDp\_\{D\}shifts representation from features ofA/BA/Btoward features ofC/DC/D, although the predictive relationships withinA/BA/Bremain unchanged and neither pair helps predict the other \(Figure[3](https://arxiv.org/html/2610.27364#S5.F3)\)\. We also find similar decoding accuracy within each pair, consistent with the set\-level characterization\.

### 5\.3Feasibility Dependence

#### Scenario\.

To test feasibility dependence, we keepA/BA/Band the latent size fixed and increase the competing cycle toN∈\{2,4,8,16\}N\\in\\\{2,4,8,16\\\}images \(Figure[4](https://arxiv.org/html/2610.27364#S5.F4), left\)\. Average within\-cycle copying probabilities remain unchanged, so retaining the larger cycle requires representing features of more images; tile scale and layout also change withNN\(Appendix[B\.3](https://arxiv.org/html/2610.27364#A2.SS3)\)\.

Figure 4:Feasibility dependence: adding images to a cycle can reverse which images’ features are more strongly represented\.Mean decoding accuracy across the competing cycle’s positions decreases asNNgrows, despite unchanged average copying probabilities; atN=16N=16, the fixedA/BA/Bpair is better represented\. Dashed line:50%50\\%chance\.
#### Results\.

Mean decoding accuracy for features of the competing images decreases at every increase inNNin all three optimization seeds \(Figure[4](https://arxiv.org/html/2610.27364#S5.F4)\)\. AtN=16N=16, features ofA/BA/Bare better represented in all three seeds; this endpoint reversal also holds for the worst and best positions in each cycle \(Appendix[E](https://arxiv.org/html/2610.27364#A5)\)\. In one out of the three sensitivity\-analysis configurations, the increase in representation ofA/BA/B’s features is within the error bars \(see Appendix[D](https://arxiv.org/html/2610.27364#A4)\)\.

## 6Testing the Steering Predictions

We now test the two steering predictions from Section[4\.2](https://arxiv.org/html/2610.27364#S4.SS2)\. Since selection operates jointly, we examine how steering that targets features of a single image affects both their predictive partners and competing features\. Both experiments use the two\-pair setup from Section[5\.2](https://arxiv.org/html/2610.27364#S5.SS2), keeping the sequence’s predictive relationships, image pool, three\-dimensional main latent representation, and training budget unchanged\. We measure image\-detection accuracy from the frozen main encoder alone, training probes on collages of images held out from encoder training and evaluating them on separate collages\. Chance decoding accuracy is50%50\\%for each feature individually\.

### 6\.1An Auxiliary Reconstruction Objective

Auxiliary objectives have been proposed to steer JEPA representations toward task\-relevant information\([LeCun, 2022](https://arxiv.org/html/2610.27364#bib.bib7);[Destrade et al\., 2025](https://arxiv.org/html/2610.27364#bib.bib37);[Zhu et al\., 2026](https://arxiv.org/html/2610.27364#bib.bib19);[Vujinovic and Kovacevic, 2026](https://arxiv.org/html/2610.27364#bib.bib35)\)\. An auxiliary reconstruction objective encourages the encoder to retain features of the image being reconstructed\. Section[4\.2](https://arxiv.org/html/2610.27364#S4.SS2)predicts that this incentive will also favor their predictive partners, at the expense of competing features\.

#### Steering\.

To test this prediction, we add an auxiliary objective that reconstructs a coarse version of the currentCCimage from the main latent representation; onlyCCis a reconstruction target, and the decoder is discarded before probing\. We vary the reconstruction weightβ\\betaacross all five original values ofpDp\_\{D\}, using one optimization seed\. The nonzero weights come from a preliminary search with bothCCandDDas reconstruction targets \(Appendix[B\.4](https://arxiv.org/html/2610.27364#A2.SS4)\)\.

#### Results\.

Rewarding reconstruction ofCCalone can makeDD’s features more strongly represented andA/BA/B’s less so \(Figure[5](https://arxiv.org/html/2610.27364#S6.F5)\)\. At the largest weight,DD, which is never reconstructed, is decoded almost as accurately asCCat everypDp\_\{D\}\. This supports the prediction that steering can affect representation beyond the directly targeted features\.

Figure 5:Reconstructing one image can strengthen representation of its predictive partner\.The changes are not uniformly monotonic, and the baseline already favorsC/DC/Dat highpDp\_\{D\}\. Each cell uses one optimization seed; all panels use the same5050–90%90\\%color scale\.

### 6\.2Predictor Side Information

Supplying information directly to the predictor changes how well it can predict different features\. Section[4\.2](https://arxiv.org/html/2610.27364#S4.SS2)predicts that this can strengthen the representation of features whose next values the additional information helps predict\.

#### Steering\.

To test this prediction, we supply the currentCCimage through an additional predictor input, without an auxiliary loss or masking it from the main input\. A separate encoder with the same architecture as the main encoder mapsCCto a three\-dimensional vector, which we concatenate with the main latent representation before prediction; this adds encoder capacity while leaving the main representation’s dimension unchanged\. The side encoder is trained jointly with the main encoder and predictor through the prediction loss; SIGReg is applied only to the main latent representation\. We evaluate all five values ofpDp\_\{D\}, paired with their no\-side\-input controls\. We validate these systems’ predictive performance in Appendix[C](https://arxiv.org/html/2610.27364#A3)\.

#### Results\.

SupplyingCCchanges how well features of the other images are represented, but produces a different redistribution from reconstruction \(Figure[6](https://arxiv.org/html/2610.27364#S6.F6)\)\. For most values ofpDp\_\{D\}, we see an increase in the representation ofDD’s features and a decrease in that ofBB’s, while representation ofAA’s features increases throughout the sweep\. These changes support the steering prediction forDD; predictive closure does not account forAAandBBseparating, since it values them only as a pair\. Low decoding accuracy from the main latent alone does not establish that a feature is unavailable to the predictor: information aboutCCor other features may be recoverable jointly from the main and side representations\.

Figure 6:SupplyingCCto the predictor can strengthen representation of its predictive partnerDD\.Data and probe tasks are matched; probes use only the main latent\.DDbecomes more represented forpD≥0\.75p\_\{D\}\\geq 0\.75and less atpD=0\.60p\_\{D\}=0\.60, whereCCis least often copied toDD;CCbecomes less represented at the highestpDp\_\{D\}\. Dotted line:50%50\\%chance\.

### 6\.3Implications for representation learning

#### Action conditioning\.

In action\-conditioned JEPA, the predictor receives the agent’s action alongside the current latent\. If we view the side information as an action, features that the agent*has an impact on*are more likely to be represented\. The action helps predict their future values, so the sets that include them gain the predictive information induced by their relationships with the action \(Appendix[A\.6](https://arxiv.org/html/2610.27364#A1.SS6)\)\. Features that the agent does not affect gain no such information and are less likely to be represented, unless they are predictive partners of features the agent affects\. In our side\-information experiment,CCplays the role of the action andDDof a feature it affects: forpD=0\.75,0\.85,0\.95p\_\{D\}=0\.75,0\.85,0\.95, supplyingCCstrengthens representation ofDD\. The consequences of genuine action conditioning remain to be tested\.

#### Steering through extracted information\.

A practitioner can instead choose side information specifically to steer predictive closure\. If a desired set of predictive features is poorly represented and one of its features can be extracted from the observation, supplying that feature to the predictor may strengthen representation of the features whose next values it helps predict\. Our side\-information experiment illustrates this possibility: supplyingCCcan strengthen representation ofDDwithout directly supervisingDD\. The mixed changes across the remaining features show why steering should be evaluated against all features needed downstream\.

#### Steering can be indirect\.

Both forms of steering target onlyCC, yet both can strengthen representation ofDD\. This shows that steering can be indirect: it need not target the desired feature itself\. Indirect steering is particularly useful when a related feature can be targeted but no loss can be written for the desired feature\. An auxiliary loss can target a feature that helps predict its next values, or one whose next values it helps predict; side information can supply only a feature of the first kind\. For example, an object’s velocity determines where the object will be next, so an auxiliary loss on its velocity, or supplying the velocity to the predictor, can strengthen the representation of its position\.

## 7Conclusion

We characterize JEPA representation learning through the joint predictive value of feature sets\. We reformulate the training objective as selecting apredictive closure: a feasible feature set that maximizes the mutual information between its features’ current values and their next values\. This view explains earlier findings on other JEPA systems, and our experiments match its predictions: an unpredictable feature becomes more decodable when it helps predict others, and changing a competing set can weaken a feature’s representation even when its own predictive relationships are unchanged\. Experiments with auxiliary objectives and predictor side information further show that targeting one feature can strengthen the representation of its predictive partners, so steering can be indirect\. If we view actions as side information, predictive closure also suggests that action\-conditioned JEPA favors the features an agent has an impact on\. These experiments illustrate how this view can inform the design of JEPA systems to better represent task\-relevant features\. Our experiments test the direction of each predicted effect; measuring its size in large JEPA systems trained on natural data remains for future work\. This set\-level view may also be relevant to models of biological representation learning that link representation to prediction\([Palmer et al\., 2015](https://arxiv.org/html/2610.27364#bib.bib31);[Chalk et al\., 2018](https://arxiv.org/html/2610.27364#bib.bib30);[Singer et al\., 2023](https://arxiv.org/html/2610.27364#bib.bib32)\)\. We hope predictive closure provides a clearer way to reason about which features JEPA encodes and informs the design and use of these systems in downstream tasks\.

## References

- Alain and Bengio \(2016\)G\. Alain and Y\. BengioUnderstanding intermediate layers using linear classifier probes\.arXiv preprint arXiv:1610\.01644\.External Links:1610\.01644,[Link](https://arxiv.org/abs/1610.01644)Cited by:[§2\.1](https://arxiv.org/html/2610.27364#S2.SS1.p3.1)\.
- Assranet al\.\(2025\)M\. Assran, A\. Bardes, D\. Fan, Q\. Garrido, R\. Howes, M\. Komeili, M\. Muckley, A\. Rizvi, C\. Roberts, K\. Sinha, A\. Zholus, S\. Arnaud, A\. Gejji, A\. Martin, F\. R\. Hogan, D\. Dugas, P\. Bojanowski, V\. Khalidov, P\. Labatut, F\. Massa, M\. Szafraniec, K\. Krishnakumar, Y\. Li, X\. Ma, S\. Chandar, F\. Meier, Y\. LeCun, M\. Rabbat, and N\. BallasV\-JEPA 2: self\-supervised video models enable understanding, prediction and planning\.arXiv preprint arXiv:2506\.09985\.External Links:2506\.09985,[Link](https://arxiv.org/abs/2506.09985)Cited by:[§1](https://arxiv.org/html/2610.27364#S1.p1.1)\.
- Balestriero and LeCun \(2025\)R\. Balestriero and Y\. LeCunLeJEPA: provable and scalable self\-supervised learning without the heuristics\.arXiv preprint arXiv:2511\.08544\.External Links:2511\.08544,[Link](https://arxiv.org/abs/2511.08544)Cited by:[§B\.1](https://arxiv.org/html/2610.27364#A2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2610.27364#S2.SS1.p1.2),[footnote 4](https://arxiv.org/html/2610.27364#footnote4)\.
- Bengioet al\.\(2013\)Y\. Bengio, A\. Courville, and P\. VincentRepresentation learning: a review and new perspectives\.IEEE Transactions on Pattern Analysis and Machine Intelligence35\(8\),pp\. 1798–1828\.Cited by:[§2\.1](https://arxiv.org/html/2610.27364#S2.SS1.p3.1)\.
- Bialek and Tishby \(1999\)W\. Bialek and N\. TishbyPredictive information\.arXiv preprint cond\-mat/9902341\.External Links:cond\-mat/9902341,[Link](https://arxiv.org/abs/cond-mat/9902341)Cited by:[§3\.1](https://arxiv.org/html/2610.27364#S3.SS1.p2.1)\.
- Burdaet al\.\(2019\)Y\. Burda, H\. Edwards, D\. Pathak, A\. Storkey, T\. Darrell, and A\. A\. EfrosLarge\-scale study of curiosity\-driven learning\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/1808.04355)Cited by:[§3\.2](https://arxiv.org/html/2610.27364#S3.SS2.p2.1)\.
- Chalket al\.\(2018\)M\. Chalk, O\. Marre, and G\. TkačikToward a unified theory of efficient, predictive, and sparse coding\.Proceedings of the National Academy of Sciences115\(1\),pp\. 186–191\.External Links:[Document](https://dx.doi.org/10.1073/pnas.1711114115)Cited by:[§7](https://arxiv.org/html/2610.27364#S7.p1.1)\.
- Destradeet al\.\(2025\)M\. Destrade, O\. Bounou, Q\. Le Lidec, J\. Ponce, and Y\. LeCunValue\-guided action planning with JEPA world models\.arXiv preprint arXiv:2601\.00844\.External Links:2601\.00844,[Link](https://arxiv.org/abs/2601.00844)Cited by:[§6\.1](https://arxiv.org/html/2610.27364#S6.SS1.p1.1)\.
- Enigma Team \(2026\)Enigma TeamThe obsessed encoder\.External Links:[Link](https://enigma.inc/posts/obsessed-encoder)Cited by:[§1](https://arxiv.org/html/2610.27364#S1.p3.1),[§2\.2](https://arxiv.org/html/2610.27364#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2610.27364#S4.SS1.p1.1)\.
- Garcíaet al\.\(1989\)C\. E\. García, D\. M\. Prett, and M\. MorariModel predictive control: theory and practice – a survey\.Automatica25\(3\),pp\. 335–348\.External Links:[Document](https://dx.doi.org/10.1016/0005-1098%2889%2990002-2),[Link](https://doi.org/10.1016/0005-1098(89)90002-2)Cited by:[§1](https://arxiv.org/html/2610.27364#S1.p1.1)\.
- Givanet al\.\(2003\)R\. Givan, T\. Dean, and M\. GreigEquivalence notions and model minimization in Markov decision processes\.Artificial Intelligence147\(1–2\),pp\. 163–223\.External Links:[Document](https://dx.doi.org/10.1016/S0004-3702%2802%2900376-4)Cited by:[§A\.1](https://arxiv.org/html/2610.27364#A1.SS1.p1.1)\.
- Ha and Schmidhuber \(2018\)D\. Ha and J\. SchmidhuberWorld models\.arXiv preprint arXiv:1803\.10122\.External Links:1803\.10122,[Link](https://arxiv.org/abs/1803.10122)Cited by:[§1](https://arxiv.org/html/2610.27364#S1.p1.1)\.
- Hubel and Wiesel \(1962\)D\. H\. Hubel and T\. N\. WieselReceptive fields, binocular interaction and functional architecture in the cat’s visual cortex\.The Journal of Physiology160\(1\),pp\. 106–154\.External Links:[Document](https://dx.doi.org/10.1113/jphysiol.1962.sp006837)Cited by:[§2\.1](https://arxiv.org/html/2610.27364#S2.SS1.p3.1)\.
- Hunget al\.\(2005\)C\. P\. Hung, G\. Kreiman, T\. Poggio, and J\. J\. DiCarloFast readout of object identity from macaque inferior temporal cortex\.Science310\(5749\),pp\. 863–866\.External Links:[Document](https://dx.doi.org/10.1126/science.1117593)Cited by:[§2\.1](https://arxiv.org/html/2610.27364#S2.SS1.p3.1)\.
- Karmannet al\.\(2026\)M\. Karmann, S\. Li, C\. Internò, B\. Andreis, D\. Klindt, R\. Balestriero, J\. Gu, P\. Torr, Q\. Zhang, P\. Jiang, H\. Zhang, B\. Li, and O\. UrfaliogluMotionJEPA: preventing temporal feature collapse by capturing visual changes in latent space\.External Links:2609\.23881,[Link](https://arxiv.org/abs/2609.23881)Cited by:[§1](https://arxiv.org/html/2610.27364#S1.p3.1),[§4\.1](https://arxiv.org/html/2610.27364#S4.SS1.p2.1)\.
- Khetarpalet al\.\(2024\)K\. Khetarpal, Z\. D\. Guo, B\. Avila Pires, Y\. Tang, C\. Lyle, M\. Rowland, N\. Heess, D\. Borsa, A\. Guez, and W\. DabneyA unifying framework for action\-conditional self\-predictive reinforcement learning\.arXiv preprint arXiv:2406\.02035\.External Links:2406\.02035,[Link](https://arxiv.org/abs/2406.02035)Cited by:[§2\.2](https://arxiv.org/html/2610.27364#S2.SS2.p1.1)\.
- Kingma and Welling \(2014\)D\. P\. Kingma and M\. WellingAuto\-encoding variational Bayes\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/1312.6114)Cited by:[§2](https://arxiv.org/html/2610.27364#S2.p1.1)\.
- Klindtet al\.\(2026\)D\. Klindt, C\. O’Neill, P\. Reizinger, H\. Maurer, and N\. MiolaneA unifying framework from neural superposition to sparse interpretable codes\.Nature Machine Intelligence8\(7\),pp\. 1025–1037\.External Links:[Document](https://dx.doi.org/10.1038/s42256-026-01259-z)Cited by:[§A\.1](https://arxiv.org/html/2610.27364#A1.SS1.SSS0.Px3.p1.1)\.
- Krizhevsky \(2009\)A\. KrizhevskyLearning multiple layers of features from tiny images\.Technical reportUniversity of Toronto\.External Links:[Link](https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf)Cited by:[§5](https://arxiv.org/html/2610.27364#S5.p2.1)\.
- Kuanget al\.\(2026\)Y\. Kuang, Y\. Dagade, D\. Chakraborty, E\. Learned\-Miller, R\. Balestriero, T\. G\. J\. Rudner, and Y\. LeCunRadial\-VCReg: more informative representation learning through radial gaussianization\.arXiv preprint arXiv:2602\.14272\.External Links:2602\.14272,[Link](https://arxiv.org/abs/2602.14272)Cited by:[§2\.1](https://arxiv.org/html/2610.27364#S2.SS1.p1.2)\.
- Kuhnet al\.\(2026\)L\. Kuhn, G\. Serra, and F\. BuettnerNon\-contrastive vision\-language learning with predictive embedding alignment\.arXiv preprint arXiv:2602\.00653\.External Links:2602\.00653,[Link](https://arxiv.org/abs/2602.00653)Cited by:[§1](https://arxiv.org/html/2610.27364#S1.p1.1)\.
- Lareyet al\.\(2026\)A\. Larey, E\. Dahan, A\. Bleiweiss, R\. Kellerman, G\. Leib, O\. Nayshool, D\. Ofer, T\. Zinger, D\. Dominissini, G\. Rechavi, N\. Bussola, S\. Lee, S\. O’Connell, D\. Hoang, M\. Wirth, A\. W\. Charney, N\. Daniel, and Y\. ShavitJEPA\-DNA: grounding genomic foundation models through joint\-embedding predictive architectures\.arXiv preprint arXiv:2602\.17162\.External Links:2602\.17162,[Link](https://arxiv.org/abs/2602.17162)Cited by:[§1](https://arxiv.org/html/2610.27364#S1.p1.1)\.
- LeCun \(2022\)Y\. LeCunA path towards autonomous machine intelligence\.OpenReview preprint\.External Links:[Link](https://openreview.net/forum?id=BZ5a1r-kVsf)Cited by:[§1](https://arxiv.org/html/2610.27364#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.27364#S2.SS1.p1.1),[§6\.1](https://arxiv.org/html/2610.27364#S6.SS1.p1.1)\.
- Liet al\.\(2006\)L\. Li, T\. J\. Walsh, and M\. L\. LittmanTowards a unified theory of state abstraction for MDPs\.InProceedings of the Ninth International Symposium on Artificial Intelligence and Mathematics,External Links:[Link](https://lihongli.github.io/pubs.html)Cited by:[§A\.1](https://arxiv.org/html/2610.27364#A1.SS1.p1.1),[footnote 1](https://arxiv.org/html/2610.27364#footnote1)\.
- Littwinet al\.\(2024\)E\. Littwin, O\. Saremi, M\. Advani, V\. Thilak, P\. Nakkiran, C\. Huang, and J\. SusskindHow JEPA avoids noisy features: the implicit bias of deep linear self distillation networks\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 91300–91336\.External Links:[Document](https://dx.doi.org/10.52202/079017-2897),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/a600f0f740605205133553cb74a1c107-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2610.27364#S1.p3.1),[§2\.2](https://arxiv.org/html/2610.27364#S2.SS2.p1.1),[§5\.1](https://arxiv.org/html/2610.27364#S5.SS1.SSS0.Px2.p1.1)\.
- Maeset al\.\(2026\)L\. Maes, Q\. Le Lidec, D\. Scieur, Y\. LeCun, and R\. BalestrieroLeWorldModel: stable end\-to\-end joint\-embedding predictive architecture from pixels\.arXiv preprint arXiv:2603\.19312\.External Links:2603\.19312,[Link](https://arxiv.org/abs/2603.19312)Cited by:[§1](https://arxiv.org/html/2610.27364#S1.p1.1)\.
- Marr \(1982\)D\. MarrVision: a computational investigation into the human representation and processing of visual information\.W\. H\. Freeman and Company,San Francisco\.Cited by:[§2\.1](https://arxiv.org/html/2610.27364#S2.SS1.p3.1)\.
- Nicollieret al\.\(2026\)L\. Nicollier, M\. Dunitz, M\. Pic, P\. Musé, E\. Meinhardt\-Llopis, and G\. FaccioloSPHERE\-JEPA: spherical prediction with homogeneous embeddings\.arXiv preprint arXiv:2605\.26900\.External Links:2605\.26900,[Link](https://arxiv.org/abs/2605.26900)Cited by:[§2\.1](https://arxiv.org/html/2610.27364#S2.SS1.p1.2)\.
- Palmeret al\.\(2015\)S\. E\. Palmer, O\. Marre, M\. J\. Berry, and W\. BialekPredictive information in a sensory population\.Proceedings of the National Academy of Sciences112\(22\),pp\. 6908–6913\.External Links:[Document](https://dx.doi.org/10.1073/pnas.1506855112)Cited by:[§7](https://arxiv.org/html/2610.27364#S7.p1.1)\.
- Pendharkar \(2026\)A\. PendharkarPredictive objectives discard exogenous control\-relevant features: a controlled mechanistic study\.arXiv preprint arXiv:2606\.30068\.External Links:2606\.30068,[Link](https://arxiv.org/abs/2606.30068)Cited by:[§1](https://arxiv.org/html/2610.27364#S1.p3.1),[§2\.2](https://arxiv.org/html/2610.27364#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2610.27364#S4.SS1.p3.1)\.
- Raoet al\.\(2026\)P\. Rao, W\. Zhang, R\. Balestriero, Y\. LeCun, and G\. LoiannoSkyJEPA: learning long\-horizon world models for zero\-shot sim\-to\-real control of quadrotors\.arXiv preprint arXiv:2606\.23444\.External Links:2606\.23444,[Link](https://arxiv.org/abs/2606.23444)Cited by:[§1](https://arxiv.org/html/2610.27364#S1.p1.1)\.
- Schmidhuber \(1990\)J\. SchmidhuberMaking the world differentiable: on using self\-supervised fully recurrent neural networks for dynamic reinforcement learning and planning in non\-stationary environments\.Technical reportTechnical ReportFKI\-126\-90,Institut für Informatik, Technische Universität München\.External Links:[Link](https://people.idsia.ch/~juergen/FKI-126-90ocr.pdf)Cited by:[§1](https://arxiv.org/html/2610.27364#S1.p1.1)\.
- Singeret al\.\(2023\)Y\. Singer, L\. Taylor, B\. D\. B\. Willmore, A\. J\. King, and N\. S\. HarperHierarchical temporal prediction captures motion processing along the visual pathway\.eLife12,pp\. e52599\.External Links:[Document](https://dx.doi.org/10.7554/eLife.52599)Cited by:[§7](https://arxiv.org/html/2610.27364#S7.p1.1)\.
- Sobalet al\.\(2022\)V\. Sobal, Jyothir S V, S\. Jalagam, N\. Carion, K\. Cho, and Y\. LeCunJoint embedding predictive architectures focus on slow features\.arXiv preprint arXiv:2211\.10831\.Note:Presented at the NeurIPS 2022 Workshop on Self\-Supervised Learning: Theory and PracticeExternal Links:2211\.10831,[Link](https://arxiv.org/abs/2211.10831)Cited by:[§1](https://arxiv.org/html/2610.27364#S1.p3.1),[§2\.2](https://arxiv.org/html/2610.27364#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2610.27364#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.27364#S4.SS1.p2.1)\.
- Sutton \(1991\)R\. S\. SuttonDyna, an integrated architecture for learning, planning, and reacting\.ACM SIGART Bulletin2\(4\),pp\. 160–163\.External Links:[Document](https://dx.doi.org/10.1145/122344.122377),[Link](https://doi.org/10.1145/122344.122377)Cited by:[§1](https://arxiv.org/html/2610.27364#S1.p1.1)\.
- Tanget al\.\(2023\)Y\. Tang, Z\. D\. Guo, P\. H\. Richemond, B\. Avila Pires, Y\. Chandak, R\. Munos, M\. Rowland, M\. Gheshlaghi Azar, C\. Le Lan, C\. Lyle, A\. Győrgy, S\. Thakoor, W\. Dabney, B\. Piot, D\. Calandriello, and M\. ValkoUnderstanding self\-predictive learning for reinforcement learning\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 33632–33656\.External Links:[Link](https://proceedings.mlr.press/v202/tang23d.html)Cited by:[§2\.2](https://arxiv.org/html/2610.27364#S2.SS2.p1.1)\.
- Van Asselet al\.\(2025\)H\. Van Assel, M\. Ibrahim, T\. Biancalani, A\. Regev, and R\. BalestrieroJoint\-embedding vs reconstruction: provable benefits of latent space prediction for self\-supervised learning\.InAdvances in Neural Information Processing Systems,External Links:2505\.12477,[Link](https://arxiv.org/abs/2505.12477)Cited by:[§2](https://arxiv.org/html/2610.27364#S2.p1.1)\.
- Van Assel \(2026\)H\. Van AsselFeature selection in world models: a spectral view\.Note:Blog postExternal Links:[Link](https://huguesva.github.io/blog/2026/world-models/)Cited by:[§2\.2](https://arxiv.org/html/2610.27364#S2.SS2.p1.1)\.
- Vujinovic and Kovacevic \(2026\)A\. Vujinovic and A\. KovacevicACT\-JEPA: novel joint\-embedding predictive architecture for efficient policy representation learning\.IEEE Access14,pp\. 78895–78906\.External Links:[Document](https://dx.doi.org/10.1109/ACCESS.2026.3696039),[Link](https://doi.org/10.1109/ACCESS.2026.3696039)Cited by:[§6\.1](https://arxiv.org/html/2610.27364#S6.SS1.p1.1)\.
- Wuet al\.\(2026\)H\. Wu, R\. Balestriero, and M\. LevineVISReg: variance\-invariance\-sketching regularization for JEPA training\.arXiv preprint arXiv:2606\.02572\.External Links:2606\.02572,[Link](https://arxiv.org/abs/2606.02572)Cited by:[§2\.1](https://arxiv.org/html/2610.27364#S2.SS1.p1.2)\.
- Zhu and Choromanska \(2026\)H\. Zhu and A\. ChoromanskaSelf\-supervised JEPA\-based world models for LiDAR occupancy completion and forecasting\.arXiv preprint arXiv:2602\.12540\.External Links:2602\.12540,[Link](https://arxiv.org/abs/2602.12540)Cited by:[§1](https://arxiv.org/html/2610.27364#S1.p1.1)\.
- Zhuet al\.\(2026\)P\. Zhu, S\. Penachio, K\. Mukherjee, and A\. JonelagaddaSpectral\-target physical latent structuring for JEPA\-style world models\.arXiv preprint arXiv:2609\.04264\.External Links:2609\.04264,[Link](https://arxiv.org/abs/2609.04264)Cited by:[§6\.1](https://arxiv.org/html/2610.27364#S6.SS1.p1.1)\.

## Appendix AFormalization of Predictive Closure

We provide a supporting derivation for the predictive\-closure characterization in Equation \([2](https://arxiv.org/html/2610.27364#S3.E2)\) of Section[3](https://arxiv.org/html/2610.27364#S3)\. We first give the definitions and assumptions, then connect prediction loss to mutual information and express the resulting objective in terms of feature sets\.

### A\.1Definitions and Assumptions

We use features to describe observations through identifiable properties, such as the traffic light’s color or whether the cyclist is riding or stopping\. This lets us characterize what JEPA encodes in terms of those properties and their predictive relationships\. The latent could also be described by which observations it maps to the same representation, so that the latent is defined by what it omits: whatever differs among observations that share a representation\([Givan et al\., 2003](https://arxiv.org/html/2610.27364#bib.bib28);[Li et al\., 2006](https://arxiv.org/html/2610.27364#bib.bib29)\)\.

#### Features\.

A featureϕi\\phi\_\{i\}describes a property of an observation, such as the traffic light’s color or the cyclist’s speed\. Formally,ϕi\\phi\_\{i\}is a mapping from observations to scalar values, and we writeyti=ϕi​\(xt\)y\_\{t\}^\{i\}=\\phi\_\{i\}\(x\_\{t\}\)for its value at timett\. This excludes properties that cannot be inferred from the observation, such as which watch the cyclist wears when it is completely hidden and no other clues reveal it\. Different mappings define different features\. For example, a mapping that matches whether the cyclist is riding or stopping on some observations but not others defines a different feature\. Afeature setS=\{ϕ1,…,ϕm\}S=\\\{\\phi\_\{1\},\\ldots,\\phi\_\{m\}\\\}is a collection of these mappings\. We writeYt​\(S\)=\(ϕi​\(Xt\)\)ϕi∈SY\_\{t\}\(S\)=\(\\phi\_\{i\}\(X\_\{t\}\)\)\_\{\\phi\_\{i\}\\in S\}for their values on an observationXtX\_\{t\}at timett\. Likewise, areaderddis a mapping from latents to scalar values, like those the predictor or a downstream model uses to read values off the latent\.

#### Represented feature sets\.

We say that a featureϕi\\phi\_\{i\}isrepresentedin the latentf⁡\(xt\)f\(x\_\{t\}\)when a readerdid\_\{i\}whose output changes only slightly under small changes to the latent \(formalized in Appendix[A\.3](https://arxiv.org/html/2610.27364#A1.SS3)\) recovers its value across observations:di​\(f⁡\(xt\)\)=ytid\_\{i\}\(f\(x\_\{t\}\)\)=y\_\{t\}^\{i\}for alltt\. We writeS⁡\(f\)S\(f\)for the complete set of features recoverable in this way from the latent produced by encoderff\.

#### Fixed latent entropy\.

Representing a feature requires capacity\([Klindt et al\., 2026](https://arxiv.org/html/2610.27364#bib.bib14)\)and we assume that all encoders match the same Gaussian target prescribed by regularization at each time, withPPdenoting a finite\-precision latent distribution\.444All encoders use the same numerical precision, and readers access the stored values\. This idealizes the distribution matching encouraged by regularizers such as SIGReg, for which[Balestriero and LeCun \(2025, Section 4\.3\)](https://arxiv.org/html/2610.27364#bib.bib21)provide empirical support\.The shared distribution fixes the Shannon entropy of the latent distributionH⁡\(P\)H\(P\), which we refer to as the latent’sentropy constraint\. We assume that the entropy constraintH⁡\(P\)H\(P\)is less than the observations’ entropy so not all features can be represented\. We call a feature setSSfeasiblewhen some encoder represents all of its features andH⁡\(Yt​\(S\)\)=H⁡\(P\)H\(Y\_\{t\}\(S\)\)=H\(P\)\. The complete setS⁡\(f\)S\(f\)of any encoderffis feasible, since its values and the latent determine one another, so thatH⁡\(Yt​\(S⁡\(f\)\)\)=H⁡\(f⁡\(Xt\)\)=H⁡\(P\)H\(Y\_\{t\}\(S\(f\)\)\)=H\(f\(X\_\{t\}\)\)=H\(P\)\(Appendix[A\.3](https://arxiv.org/html/2610.27364#A1.SS3)\)\. If we could directly choose which features are represented, adding a feature that provides new information would require removing others to keep the resulting set feasible\.

Throughout, the prescribed distributionPPand latent dimensionkkare fixed across encoders\.

### A\.2Prediction Loss and Information

Because the regularization term in Equation \([1](https://arxiv.org/html/2610.27364#S2.E1)\) depends only on the distribution off⁡\(Xt\)f\(X\_\{t\}\), which isPPfor every encoder, an encoder minimizes the training objective exactly when it minimizes the prediction loss\. Under a Gaussian prediction\-error assumption, we show that minimizing expected squared prediction error is equivalent to maximizing the mutual information between consecutive representations\. We writef⋆f^\{\\star\}for any such encoder:

f⋆∈arg​minf⁡ming​𝔼​‖g⁡\(f⁡\(Xt\)\)−f⁡\(Xt\+1\)‖22=arg​maxf⁡I​\(f⁡\(Xt\),f⁡\(Xt\+1\)\)\.f^\{\\star\}\\in\\operatorname\*\{arg\\,min\}\_\{f\}\\;\\min\_\{g\}\\,\\mathbb\{E\}\\left\\\|g\(f\(X\_\{t\}\)\)\-f\(X\_\{t\+1\}\)\\right\\\|\_\{2\}^\{2\}=\\operatorname\*\{arg\\,max\}\_\{f\}\\;I\\bigl\(f\(X\_\{t\}\);f\(X\_\{t\+1\}\)\\bigr\)\.\(4\)The expectation is over consecutive observation pairs\.

#### Prediction\-error assumption\.

WriteZt=f⁡\(Xt\)Z\_\{t\}=f\(X\_\{t\}\)\. We assume that all encoders producekk\-dimensional latents with the same marginal distributionPP, whose Shannon entropyH⁡\(P\)H\(P\)is finite\. For each encoderff, assume that among the functions the predictor network can represent there is a predictorgfg\_\{f\}such that, at the latent’s numerical resolution,

Zt\+1\|Zt=z∼𝒩⁡\(gf​\(z\),σf​Ik\),Z\_\{t\+1\}\\mid Z\_\{t\}=z\\sim\\mathcal\{N\}\\\!\\left\(g\_\{f\}\(z\),\\sigma\_\{f\}I\_\{k\}\\right\),\(5\)whereσf\>0\\sigma\_\{f\}\>0is a scalar variance that may differ between encoders\.

#### Proof\.

For each current latent, squared prediction error is minimized by the mean of this Gaussian, whichgfg\_\{f\}predicts\. Each error coordinate has zero mean and varianceσf\\sigma\_\{f\}, so its expected squared error isσf\\sigma\_\{f\}\. Summing over thekkcoordinates gives

ming⁡𝔼​‖Zt\+1−g⁡\(Zt\)‖22=k​σf\.\\min\_\{g\}\\mathbb\{E\}\\left\\\|Z\_\{t\+1\}\-g\(Z\_\{t\}\)\\right\\\|\_\{2\}^\{2\}=k\\sigma\_\{f\}\.\(6\)The conditional Gaussian distribution has entropy

h⁡\(Zt\+1∣Zt\)=k2​log⁡\(2​π​e​σf\)\.h\(Z\_\{t\+1\}\\mid Z\_\{t\}\)=\\frac\{k\}\{2\}\\log\(2\\pi e\\sigma\_\{f\}\)\.\(7\)At the latent’s fixed numerical precision, we assume the discrete conditional entropy equals this Gaussian value plus an additive correctionccshared across encoders\. \(cccomes from rounding the continuous latent to that precision, so it depends only on the precision\.\) Since the next latent has the fixed marginal distributionPP,

I⁡\(Zt,Zt\+1\)\\displaystyle I\(Z\_\{t\};Z\_\{t\+1\}\)=H⁡\(P\)−H⁡\(Zt\+1∣Zt\)\\displaystyle=H\(P\)\-H\(Z\_\{t\+1\}\\mid Z\_\{t\}\)\(8\)=H⁡\(P\)−k2​log⁡\(2​π​e​σf\)−c\.\\displaystyle=H\(P\)\-\\frac\{k\}\{2\}\\log\(2\\pi e\\sigma\_\{f\}\)\-c\.BecauseH⁡\(P\)H\(P\),kk, andccare fixed, predictive information is strictly decreasing inσf\\sigma\_\{f\}, and so in the optimized prediction errork​σfk\\sigma\_\{f\}\. Therefore, minimizing expected squared prediction error and maximizing predictive information select exactly the same encoders\. Without the Gaussian assumption, and with the same precision correctioncc, Equation \([8](https://arxiv.org/html/2610.27364#A1.E8)\) holds with≥\\geqin place of its second equality for every encoder that matchesPP, withσf\\sigma\_\{f\}taken as its optimized prediction error divided bykk, because among distributions with a given summed variance the isotropic Gaussian has the largest entropy \(averaging overZtZ\_\{t\}uses the concavity of the logarithm\)\. Minimizing the prediction loss then maximizes a lower bound on the mutual information between consecutive latents\.

### A\.3From Representations to Feature Sets

We now write the objective of Equation \([4](https://arxiv.org/html/2610.27364#A1.E4)\) in terms of the complete setS⁡\(f\)S\(f\)\.

#### Reader condition\.

ForZ=f⁡\(Xt\)∼PZ=f\(X\_\{t\}\)\\sim P, we control reader sensitivity using common tolerancesε,δ\>0\\varepsilon,\\delta\>0and0<α<10<\\alpha<1, keeping these tolerances unchanged across encoders:

Pr\[sup‖z′−Z‖2≤ε\|d\(z′\)−d\(Z\)\|\>δ\]≤α\.\\Pr\\\!\\left\[\\sup\_\{\\\|z^\{\\prime\}\-Z\\\|\_\{2\}\\leq\\varepsilon\}\|d\(z^\{\\prime\}\)\-d\(Z\)\|\>\\delta\\right\]\\leq\\alpha\.Thus, latent perturbations of size at mostε\\varepsilonchange the reader’s output by at mostδ\\delta, except on a set of probability at mostα\\alpha\. We require exact recovery from the unperturbed latent\.

#### Information preservation\.

The latent determines every feature value inYt​\(S​\(f\)\)Y\_\{t\}\(S\(f\)\)\. Conversely, the readersdj​\(z\)=\(δ/ε\)​zjd\_\{j\}\(z\)=\(\\delta/\\varepsilon\)z\_\{j\}, forj=1,…,kj=1,\\ldots,k, satisfy the sensitivity condition everywhere, so the corresponding featuresx↦dj​\(f​\(x\)\)x\\mapsto d\_\{j\}\(f\(x\)\)belong toS⁡\(f\)S\(f\)\. Their values recover every latent coordinate\. The latent and the valuesYt​\(S​\(f\)\)Y\_\{t\}\(S\(f\)\)therefore determine one another, so they carry the same information, and their mutual information across consecutive observations is equal:

I⁡\(f⁡\(Xt\),f⁡\(Xt\+1\)\)=I⁡\(Yt​\(S⁡\(f\)\),Yt\+1​\(S⁡\(f\)\)\)\.I\\bigl\(f\(X\_\{t\}\);f\(X\_\{t\+1\}\)\\bigr\)=I\\bigl\(Y\_\{t\}\(S\(f\)\);Y\_\{t\+1\}\(S\(f\)\)\\bigr\)\.\(9\)The same equality holds for every feasible setSS, withffan encoder that represents it: the latent determines the valuesYt​\(S\)Y\_\{t\}\(S\), and a function of the latent that keeps its entropyH⁡\(P\)H\(P\)is one\-to\-one, so these values also determine the latent\. This equality and Appendix[A\.2](https://arxiv.org/html/2610.27364#A1.SS2)together show thatS⁡\(f⋆\)S\(f^\{\\star\}\)maximizes predictive information over feasible feature sets, proving Equation \([2](https://arxiv.org/html/2610.27364#S3.E2)\)\.

### A\.4Aleatoric Uncertainty and Predictive Leakage

For a feature setSS, we writeYt=Yt​\(S\)Y\_\{t\}=Y\_\{t\}\(S\)andYt\+1=Yt\+1​\(S\)Y\_\{t\+1\}=Y\_\{t\+1\}\(S\)\. By definition,

I⁡\(Yt,Yt\+1\)=H⁡\(Yt\+1\)−H⁡\(Yt\+1∣Yt\),I\(Y\_\{t\};Y\_\{t\+1\}\)=H\(Y\_\{t\+1\}\)\-H\(Y\_\{t\+1\}\\mid Y\_\{t\}\),and for a feasible set the first term isH⁡\(P\)H\(P\)\. Conditional mutual information is defined in the same way, givenYtY\_\{t\}:

I⁡\(Xt;Yt\+1∣Yt\)=H⁡\(Yt\+1∣Yt\)−H⁡\(Yt\+1∣Xt,Yt\)\.I\(X\_\{t\};Y\_\{t\+1\}\\mid Y\_\{t\}\)=H\(Y\_\{t\+1\}\\mid Y\_\{t\}\)\-H\(Y\_\{t\+1\}\\mid X\_\{t\},Y\_\{t\}\)\.BecauseYtY\_\{t\}is computed fromXtX\_\{t\}, knowingXtX\_\{t\}already means knowingYtY\_\{t\}, soH⁡\(Yt\+1∣Xt,Yt\)=H⁡\(Yt\+1∣Xt\)H\(Y\_\{t\+1\}\\mid X\_\{t\},Y\_\{t\}\)=H\(Y\_\{t\+1\}\\mid X\_\{t\}\)and

H⁡\(Yt\+1∣Yt\)=H⁡\(Yt\+1∣Xt\)\+I⁡\(Xt;Yt\+1∣Yt\)\.H\(Y\_\{t\+1\}\\mid Y\_\{t\}\)=H\(Y\_\{t\+1\}\\mid X\_\{t\}\)\+I\(X\_\{t\};Y\_\{t\+1\}\\mid Y\_\{t\}\)\.Substituting this into the first equation gives

I⁡\(Yt,Yt\+1\)=H⁡\(P\)−H⁡\(Yt\+1∣Xt\)−I⁡\(Xt;Yt\+1∣Yt\),I\(Y\_\{t\};Y\_\{t\+1\}\)=H\(P\)\-H\(Y\_\{t\+1\}\\mid X\_\{t\}\)\-I\(X\_\{t\};Y\_\{t\+1\}\\mid Y\_\{t\}\),\(10\)so among feasible sets, maximizing predictive information is the same as minimizing the sum of the last two terms, as in Equation \([3](https://arxiv.org/html/2610.27364#S3.E3)\)\.

### A\.5Relation to Per\-Feature Selection

For a feature setSSand a featureϕ∉S\\phi\\notin S, we writeYt=Yt​\(S\)Y\_\{t\}=Y\_\{t\}\(S\)andϕt=ϕ⁡\(Xt\)\\phi\_\{t\}=\\phi\(X\_\{t\}\)\. By the chain rule of mutual information, addingϕ\\phitoSSchanges the set’s predictive information by

Δ⁡\(ϕ∣S\)≔Ipred​\(S∪\{ϕ\}\)−Ipred​\(S\)=I⁡\(ϕt;Yt\+1∣Yt\)\+I⁡\(\(Yt,ϕt\);ϕt\+1∣Yt\+1\),\\Delta\(\\phi\\mid S\)\\coloneqq I\_\{\\mathrm\{pred\}\}\(S\\cup\\\{\\phi\\\}\)\-I\_\{\\mathrm\{pred\}\}\(S\)=I\\bigl\(\\phi\_\{t\};Y\_\{t\+1\}\\mid Y\_\{t\}\\bigr\)\+I\\bigl\(\(Y\_\{t\},\\phi\_\{t\}\);\\phi\_\{t\+1\}\\mid Y\_\{t\+1\}\\bigr\),\(11\)and changes its entropy byH⁡\(ϕt∣Yt\)H\(\\phi\_\{t\}\\mid Y\_\{t\}\), theentropy costofϕ\\phi\. The first term is the information thatϕ\\phi’s current value adds about the set’s future\. The second is the information that the set’s present, together withϕ\\phi’s, provides aboutϕ\\phi’s future beyond the set’s own future\. Both terms and the entropy cost generally depend onSS\. In optimization terminology, predictive information is generally a non\-additive, or non\-modular, set objective: its value depends on the combination of selected features rather than a sum of fixed individual contributions\.

#### Independent features\.

Suppose featuresϕ1,…,ϕn\\phi\_\{1\},\\ldots,\\phi\_\{n\}evolve independently, so that the pairs\(ϕi​\(Xt\),ϕi​\(Xt\+1\)\)\(\\phi\_\{i\}\(X\_\{t\}\),\\phi\_\{i\}\(X\_\{t\+1\}\)\)are mutually independent acrossii\. For a setSSof these features andϕj∉S\\phi\_\{j\}\\notin S, the first term of Equation \([11](https://arxiv.org/html/2610.27364#A1.E11)\) vanishes, the second equalsI⁡\(ϕj​\(Xt\),ϕj​\(Xt\+1\)\)I\(\\phi\_\{j\}\(X\_\{t\}\);\\phi\_\{j\}\(X\_\{t\+1\}\)\), and the entropy cost equalsH⁡\(ϕj​\(Xt\)\)H\(\\phi\_\{j\}\(X\_\{t\}\)\), none of which depends onSS\. Hence

Ipred​\(S\)=∑ϕi∈SI⁡\(ϕi​\(Xt\),ϕi​\(Xt\+1\)\),H⁡\(Yt​\(S\)\)=∑ϕi∈SH⁡\(ϕi​\(Xt\)\)\.I\_\{\\mathrm\{pred\}\}\(S\)=\\sum\_\{\\phi\_\{i\}\\in S\}I\\bigl\(\\phi\_\{i\}\(X\_\{t\}\);\\phi\_\{i\}\(X\_\{t\+1\}\)\\bigr\),\\qquad H\\bigl\(Y\_\{t\}\(S\)\\bigr\)=\\sum\_\{\\phi\_\{i\}\\in S\}H\\bigl\(\\phi\_\{i\}\(X\_\{t\}\)\\bigr\)\.If these features all have the same entropy andH⁡\(P\)H\(P\)iskktimes it, then every feasible set formed from whole featuresϕi\\phi\_\{i\}containskkof them, and the one with the greatest predictive information consists of thekkfeatures with the greatest individual predictive informationI⁡\(ϕi​\(Xt\),ϕi​\(Xt\+1\)\)I\(\\phi\_\{i\}\(X\_\{t\}\);\\phi\_\{i\}\(X\_\{t\+1\}\)\)\. Selection among such sets then reduces to ranking individual features\. With unequal entropies, it becomes a knapsack problem in which each feature’s value and cost are still independent of the other features selected\.

#### Dependent features\.

For the traffic lightLLof Section[5\.1](https://arxiv.org/html/2610.27364#S5.SS1)and a setSSof image features,Lt\+1L\_\{t\+1\}is independent of\(Yt,Lt,Yt\+1\)\(Y\_\{t\},L\_\{t\},Y\_\{t\+1\}\), soΔ⁡\(L∣S\)=I⁡\(Lt;Yt\+1∣Yt\)\\Delta\(L\\mid S\)=I\(L\_\{t\};Y\_\{t\+1\}\\mid Y\_\{t\}\)\. This contribution is zero unlessSScontains the images whose persistence the light controls, while its entropy costH⁡\(Lt∣Yt\)H\(L\_\{t\}\\mid Y\_\{t\}\)is one bit at everyqq\. In the cycle of Section[5\.3](https://arxiv.org/html/2610.27364#S5.SS3), the information an image carries through its copy link counts only when the image it is copied to is also in the set\. A set holding part of a long cycle therefore keeps only the links whose two ends it contains\.

### A\.6Side Information

Section[4\.2](https://arxiv.org/html/2610.27364#S4.SS2)considers a predictor that also receives side informationUtU\_\{t\}, so that it predictsf⁡\(Xt\+1\)f\(X\_\{t\+1\}\)byg⁡\(f⁡\(Xt\),Ut\)g\(f\(X\_\{t\}\),U\_\{t\}\)\. We derive how this changes the two terms of Equation \([3](https://arxiv.org/html/2610.27364#S3.E3)\)\. For a feature setSS, we again writeYt=Yt​\(S\)Y\_\{t\}=Y\_\{t\}\(S\)andYt\+1=Yt\+1​\(S\)Y\_\{t\+1\}=Y\_\{t\+1\}\(S\)\.

#### Objective\.

We extend the prediction\-error assumption of Appendix[A\.2](https://arxiv.org/html/2610.27364#A1.SS2)to condition on both predictor inputs: writingZt=f⁡\(Xt\)Z\_\{t\}=f\(X\_\{t\}\), for each encoderffthere is a predictorgfg\_\{f\}such that

Zt\+1∣Zt=z,Ut=u∼𝒩\(gf\(z,u\),σfIk\)\.Z\_\{t\+1\}\\mid Z\_\{t\}=z,\\,U\_\{t\}=u\\sim\\mathcal\{N\}\\\!\\left\(g\_\{f\}\(z,u\),\\sigma\_\{f\}I\_\{k\}\\right\)\.The proof in Appendix[A\.2](https://arxiv.org/html/2610.27364#A1.SS2)then holds withUtU\_\{t\}added to every condition, so the optimized prediction error is strictly increasing inH⁡\(Zt\+1∣Zt,Ut\)H\(Z\_\{t\+1\}\\mid Z\_\{t\},U\_\{t\}\)\. Because the latent and the values of the complete represented set determine one another \(Appendix[A\.3](https://arxiv.org/html/2610.27364#A1.SS3)\), this entropy equalsH⁡\(Yt\+1∣Yt,Ut\)H\(Y\_\{t\+1\}\\mid Y\_\{t\},U\_\{t\}\)forS=S⁡\(f\)S=S\(f\)\. An ideally trained encoder therefore represents a feasible set that minimizesH⁡\(Yt\+1∣Yt,Ut\)H\(Y\_\{t\+1\}\\mid Y\_\{t\},U\_\{t\}\), which is the same as maximizingI⁡\(\(Yt,Ut\),Yt\+1\)=H⁡\(P\)−H⁡\(Yt\+1∣Yt,Ut\)I\\bigl\(\(Y\_\{t\},U\_\{t\}\);Y\_\{t\+1\}\\bigr\)=H\(P\)\-H\(Y\_\{t\+1\}\\mid Y\_\{t\},U\_\{t\}\)\.

#### The two terms\.

The steps of Appendix[A\.4](https://arxiv.org/html/2610.27364#A1.SS4)hold withUtU\_\{t\}added to every condition\. By the definition of conditional mutual information,

I\(Xt;Yt\+1∣Yt,Ut\)=H\(Yt\+1∣Yt,Ut\)−H\(Yt\+1∣Xt,Yt,Ut\),I\(X\_\{t\};Y\_\{t\+1\}\\mid Y\_\{t\},U\_\{t\}\)=H\(Y\_\{t\+1\}\\mid Y\_\{t\},U\_\{t\}\)\-H\(Y\_\{t\+1\}\\mid X\_\{t\},Y\_\{t\},U\_\{t\}\),and becauseYtY\_\{t\}is computed fromXtX\_\{t\}, the last term equalsH⁡\(Yt\+1∣Xt,Ut\)H\(Y\_\{t\+1\}\\mid X\_\{t\},U\_\{t\}\), so

H\(Yt\+1∣Yt,Ut\)=H\(Yt\+1∣Xt,Ut\)\+I\(Xt;Yt\+1∣Yt,Ut\)\.H\(Y\_\{t\+1\}\\mid Y\_\{t\},U\_\{t\}\)=H\(Y\_\{t\+1\}\\mid X\_\{t\},U\_\{t\}\)\+I\(X\_\{t\};Y\_\{t\+1\}\\mid Y\_\{t\},U\_\{t\}\)\.\(12\)The minimized quantity is thus the sum of the two terms of Equation \([3](https://arxiv.org/html/2610.27364#S3.E3)\), each computed givenUtU\_\{t\}as well\.

#### Change in each term\.

By the definition of conditional mutual information, supplyingUtU\_\{t\}lowers the sum by

H⁡\(Yt\+1∣Yt\)−H⁡\(Yt\+1∣Yt,Ut\)=I⁡\(Ut;Yt\+1∣Yt\),H\(Y\_\{t\+1\}\\mid Y\_\{t\}\)\-H\(Y\_\{t\+1\}\\mid Y\_\{t\},U\_\{t\}\)=I\(U\_\{t\};Y\_\{t\+1\}\\mid Y\_\{t\}\),which is whatUtU\_\{t\}tells about the set’s next values beyond the set’s current values, so sets for which this quantity is larger are favored\. SupplyingUtU\_\{t\}lowers the aleatoric uncertainty by

H⁡\(Yt\+1∣Xt\)−H⁡\(Yt\+1∣Xt,Ut\)=I⁡\(Ut;Yt\+1∣Xt\),H\(Y\_\{t\+1\}\\mid X\_\{t\}\)\-H\(Y\_\{t\+1\}\\mid X\_\{t\},U\_\{t\}\)=I\(U\_\{t\};Y\_\{t\+1\}\\mid X\_\{t\}\),which is whatUtU\_\{t\}tells about the set’s next values beyond the whole current observation\. WhenUtU\_\{t\}is derived from the observation, knowingXtX\_\{t\}already means knowingUtU\_\{t\}, so this second quantity is zero\. The aleatoric uncertainty is then unchanged, and the whole reduction comes from the predictive leakage:

I\(Xt;Yt\+1∣Yt,Ut\)=I\(Xt;Yt\+1∣Yt\)−I\(Ut;Yt\+1∣Yt\)\.I\(X\_\{t\};Y\_\{t\+1\}\\mid Y\_\{t\},U\_\{t\}\)=I\(X\_\{t\};Y\_\{t\+1\}\\mid Y\_\{t\}\)\-I\(U\_\{t\};Y\_\{t\+1\}\\mid Y\_\{t\}\)\.WhenUtU\_\{t\}is not determined by the observation, as with an action, the second quantity can be positive\. For example, an agent’s next position can be uncertain given the whole current observation and certain once its action is known\.

## Appendix BExperimental Setup and Evaluation Details

### B\.1Shared training and probing protocol

In each experiment in Section[5](https://arxiv.org/html/2610.27364#S5), we vary a single design parameter while keeping the remaining data\-generation rules fixed\. At each value of the varied parameter, we train the same end\-to\-end JEPA from scratch on sequences of collages composed of CIFAR\-10 images, using a three\-dimensional latent and the objective0\.9​ℒpred\+0\.1​ℒSIGReg0\.9\\mathcal\{L\}\_\{\\mathrm\{pred\}\}\+0\.1\\mathcal\{L\}\_\{\\mathrm\{SIGReg\}\}\([Balestriero and LeCun, 2025](https://arxiv.org/html/2610.27364#bib.bib21)\)\. Training draws from a class\-balanced set of 5,000 images from the official training split; after training, we freeze the encoder and train probes to decode designated features from newly sampled collages drawn from a disjoint class\-balanced set of 5,000 test images\.

#### Architecture and gradients\.

The encoder has four3×33\\times 3convolution/ReLU/2×22\\times 2max\-pooling blocks with 32, 64, 128, and 256 channels, followed by a4096→1024→34096\\to 1024\\to 3projection with a GELU hidden activation\. The group\-size encoder additionally uses adaptive max\-pooling to produce a4×44\\times 4spatial output\. Predictors use3→16→16→33\\to 16\\to 16\\to 3GELU MLPs, with six inputs for side information; the reconstruction decoder is an affine3→483\\to 48map\. We share the main encoder across frames and backpropagate through both prediction\-loss branches, without stop\-gradient\. All JEPA systems train with AdamW at learning rate10−310^\{\-3\}with cosine decay to10−510^\{\-5\}, and SIGReg uses 1,024 random projections\.

### B\.2Two\-image traffic\-light setup and readouts

To vary the light’s usefulness for predicting the next transition while keeping each image’s marginal persistence fixed, we let the light control which image is replaced\. The two32×3232\\times 32\-pixel images occupy the upper row of a64×6464\\times 64\-pixel canvas, and a4×44\\times 4\-pixel light patch appears in the lower\-right quadrant; the remaining pixels are black\. Each transition replaces exactly one image with a different identity; the current light selects the position with probabilityqq, and a uniform draw selects it otherwise\. The light is sampled independently and uniformly at every frame\. Consequently, each image persists with marginal probability0\.50\.5at everyqq, while the light identifies the replaced position with probability\(1\+q\)/2\(1\+q\)/2\. We train three runs at eachq∈\{0,0\.25,0\.5,0\.75,1\}q\\in\\\{0,0\.25,0\.5,0\.75,1\\\}, each for 25 epochs of 100,000 eight\-frame sequences with batch size 512\.

A two\-hidden\-layer probe of width 16 decodes the current light from the frozen current\-frame latent, using separately sampled training, validation, and test datasets containing 20,000, 5,000, and 5,000 examples from the held\-out image pool, respectively\. The probe trains for 1,200 steps with batch size 512; chance accuracy is50%50\\%\.

### B\.3Setups and readouts for the two\-pair and cycle experiments

#### Changing predictability\.

The two\-pair setup groups image identities into\{A,B\}\\\{A,B\\\}and\{C,D\}\\\{C,D\\\}at the four locations of a2×22\\times 2collage\. Within each pair, an image identity can be copied to the other position in the next frame:BtB\_\{t\}determinesAt\+1A\_\{t\+1\}with probability11,AtA\_\{t\}determinesBt\+1B\_\{t\+1\}with probability0\.60\.6, andDtD\_\{t\}determinesCt\+1C\_\{t\+1\}with probability0\.90\.9\. The probabilitypDp\_\{D\}thatCtC\_\{t\}determinesDt\+1D\_\{t\+1\}is the only parameter we vary; when a copy does not occur, the next image is sampled independently\. We train three runs at eachpD∈\{0\.60,0\.75,0\.85,0\.95,1\}p\_\{D\}\\in\\\{0\.60,0\.75,0\.85,0\.95,1\\\}, each for 25 epochs of 100,000 eight\-frame sequences with batch size 512\.

#### Changing group size\.

TheA/BA/Bcycle is fixed at two identities with copy probability0\.850\.85, and the competingC/DC/Dcycle hasN∈\{2,4,8,16\}N\\in\\\{2,4,8,16\\\}identities, orderedC,D,E,F,…C,D,E,F,\\ldotsthrough the firstNNpositions, with copy probabilities alternating between0\.900\.90and0\.960\.96\. Thus the average per\-feature copy probabilities remain0\.850\.85forA/BA/Band0\.930\.93forC/DC/Dat every size\. We train three runs per configuration, each for 100 epochs of 100,000 eight\-frame sequences, with batch size 512 and a three\-dimensional latent\. Within each design,\{A,B\}\\\{A,B\\\}remains unchanged, allowing us to assess whether changes in the competing group alter its representation\. We used the first run during configuration development and kept the configuration unchanged for the two replications; all three runs share the image pools and data\-generation randomness\.

#### Rendering control\.

Keeping individual image size fixed would make the largerC/DC/Dgroup occupy more of the input, confounding feature count with the group’s total spatial extent\. We therefore keep the128×256128\\times 256\-pixel canvas and the two native32×3232\\times 32\-pixelA/BA/Bimages fixed, while distributing approximately the same total displayed area among theC/DC/Dimages\. To avoid removing image detail through downsampling, we use the native CIFAR\-10 resolution as the minimum tile size and enlarge images when fewer positions share this area\. ForN=2,4,8,16N=2,4,8,16, individualC/DC/Dimages have side lengths of 90, 64, 45, and 32 pixels, respectively, giving occupied areas of 16,200, 16,384, 16,200, and 16,384 pixels, which differ by less than1\.2%1\.2\\%\. TheC/DC/Dimages occupy centered row\-major grids of1×21\\times 2,2×22\\times 2,2×42\\times 4, and4×44\\times 4within the same right\-hand region; unused pixels remain black\. Larger tiles use bilinear enlargement of the native CIFAR\-10 images, without cropping or reducing any image below its original resolution\. Enlargement is greatest atN=2N=2and goes back to the original size atN=16N=16\. Tile scale and layout still change withNN, so this control does not isolate representation cost from the difficulty of extracting the images’ information\.

#### Image\-detection probes\.

To assess image\-identity information in the latent, we train target\-image presence detectors separately for each fixed position\. The group\-size experiment uses the same primary image\-detection probe architecture, per\-detector training budget, and held\-out image pool as the four\-image predictability and steering experiments\. We train a separate binary classifier for each of 50 target images at every fixed position, with five targets per CIFAR\-10 class\. Each classifier receives only the frozen latent representation: positive examples contain its target at the designated position, while negative examples exclude that target everywhere and use a different image of the same class at the designated position\. Other positions are sampled independently, excluding the target\. Because negative examples exclude the target everywhere, a representation of the unordered images within a pair could pass both position\-associated probes without distinguishing their positions\. These readouts measure target\-image detection under the stated sampling protocol and do not establish position binding\.

Each detector has two hidden layers of width 16 and trains for 1,000 steps with batch size 128 on 512 training examples, with 512 validation and 1,024 test examples, balanced between positive and negative cases\. We use AdamW with learning rate10−310^\{\-3\}decaying to10−510^\{\-5\}, 50 warmup steps, weight decay10−410^\{\-4\}, gradient clipping at 10, and the same probe optimization seed across comparisons\. Latent coordinates are standardized using 100,000 separately sampled collages from the held\-out image pool\. All probe images are unseen during encoder training; target identities recur across probe training, validation, and testing, while the collages are sampled separately from the same held\-out pool\. We report ordinary binary classification accuracy at the zero\-logit threshold, with chance accuracy50%50\\%\. Increasing the number of positions increases the number of independent detectors, rather than reducing the training or evaluation budget per position\. We first average accuracy over target images within each position, then average positions within each group, and finally report the mean and standard error across optimization seeds\.

### B\.4Steering interventions

Both interventions use the two\-pair setup, keeping the sequence’s predictive relationships, image pool, three\-dimensional main latent representation, and training budget unchanged\. Held\-out image\-detection probes receive only the frozen main encoder’s representation\.

#### Auxiliary reconstruction\.

To directly reward retaining information aboutCC, we add an auxiliary reconstruction objective\. The decoder reconstructs a standardized4×44\\times 4average\-pooled version of the currentCCimage from the current latent representation, giving the objective0\.9​ℒpred\+0\.1​ℒSIGReg\+β​ℒreconC0\.9\\mathcal\{L\}\_\{\\mathrm\{pred\}\}\+0\.1\\mathcal\{L\}\_\{\\mathrm\{SIGReg\}\}\+\\beta\\mathcal\{L\}\_\{\\mathrm\{recon\}\}^\{C\}\. OnlyCCis a reconstruction target; the decoder is discarded before probing\. The four nonzero weights were identified in a preliminary search atpD=0\.75p\_\{D\}=0\.75with bothCCandDDas reconstruction targets\. We then evaluate reconstruction ofCCalone atβ∈\{0,0\.0425,0\.105,0\.135,0\.36\}\\beta\\in\\\{0,\\allowbreak 0\.0425,\\allowbreak 0\.105,\\allowbreak 0\.135,\\allowbreak 0\.36\\\}and all five original values ofpDp\_\{D\}, using one optimization seed\.

#### Predictor side information\.

To examine how direct predictor access toCCchanges the main representation, we supply the currentCCimage through a separate encoder\. The side encoder has the same architecture as the main encoder and maps the current32×3232\\times 32\-pixelCCimage, bilinearly enlarged to64×6464\\times 64pixels, to a three\-dimensional vector\. We concatenate this vector with the current main latent to form the predictor’s six\-dimensional input; its target remains the next\-frame main latent\. The main encoder continues to receive the full collage, includingCC, and retains its three\-dimensional output\. Training uses0\.9​ℒpred\+0\.1​ℒSIGReg0\.9\\mathcal\{L\}\_\{\\mathrm\{pred\}\}\+0\.1\\mathcal\{L\}\_\{\\mathrm\{SIGReg\}\}, with the side encoder trained through the prediction loss and SIGReg applied only to the main latent\. This intervention adds encoder capacity while keeping the main latent dimension fixed\. We evaluate all five values ofpDp\_\{D\}with three training runs per condition, paired with their corresponding no\-side\-input controls\.

## Appendix CNext\-Observation Prediction

To validate that the JEPA systems learned to predict, we test whether their predictions distinguish the actual next observation from an alternative\. We evaluate the traffic\-light systems trained with light\-control probabilityq=1q=1, in which the current light always determines which image is replaced \(Section[5\.1](https://arxiv.org/html/2610.27364#S5.SS1)\)\. We also evaluate the two\-pair systems at all five values ofpDp\_\{D\}, the probability thatCtC\_\{t\}is copied toDt\+1D\_\{t\+1\}\(Section[5\.2](https://arxiv.org/html/2610.27364#S5.SS2)\)\. For these, we test both the ordinary systems and those whose predictor also receives the currentCCimage as side information \(Section[6\.2](https://arxiv.org/html/2610.27364#S6.SS2)\)\. Training settings follow Appendix[B](https://arxiv.org/html/2610.27364#A2)\.

We evaluate each trained system on5,0005\{,\}000transitions generated from images held out from encoder training, using the same transitions across runs\. For each transition, the system predicts the latent of the next observation from the current observationxtx\_\{t\}\. We count the prediction as correct if it is closer, in squared Euclidean distance, to the latentf⁡\(xt\+1\)f\(x\_\{t\+1\}\)of the actual next observation than to the latentf⁡\(x~\)f\(\\tilde\{x\}\)of an alternative observationx~\\tilde\{x\}\. We drawx~\\tilde\{x\}from the same distribution as the observations and redraw it whenever the transition rules allow it to followxtx\_\{t\}\. It is therefore as likely asxt\+1x\_\{t\+1\}overall and differs from it only in that it cannot followxtx\_\{t\}\. As a control, we shuffle predictions across transitions while keeping the actual and alternative observations fixed, so that each prediction comes from an unrelated current observation\.

In every tested setting, the systems identify the actual next observation well above chance, whereas shuffled predictions remain near chance \(Table[1](https://arxiv.org/html/2610.27364#A3.T1)\)\.

Table 1:Predictors distinguish actual next observations from incompatible alternatives\.Values are percentage correct, reported as mean±\\pmstandard error across three training runs\. Chance is50%50\\%for choosing between the actual next observation and the alternative\. Each system is evaluated in its own learned latent space\. The side\-information condition supplies the currentCCimage while keeping it visible in the main input\.
## Appendix DRegularization Sensitivity

We test how the findings change with the regularization weight in all five experiments, using

ℒr=\(1−r\)​ℒpred\+r​ℒSIGReg,r∈\{0\.05,0\.10,0\.20\}\.\\mathcal\{L\}\_\{r\}=\(1\-r\)\\mathcal\{L\}\_\{\\mathrm\{pred\}\}\+r\\mathcal\{L\}\_\{\\mathrm\{SIGReg\}\},\\qquad r\\in\\\{0\.05,0\.10,0\.20\\\}\.\(13\)The main setting usesr=0\.10r=0\.10; the corresponding weights relative to a unit prediction coefficient areλ=r/\(1−r\)\\lambda=r/\(1\-r\), about0\.050\.05,0\.110\.11, and0\.250\.25\. For reconstruction, we addβ​ℒrec,C\\beta\\mathcal\{L\}\_\{\\mathrm\{rec\},C\}and holdβ\\betafixed acrossrr\. All other training and probing settings follow Appendix[B](https://arxiv.org/html/2610.27364#A2)\. Figures[7](https://arxiv.org/html/2610.27364#A4.F7)–[11](https://arxiv.org/html/2610.27364#A4.F11)show every tested condition\. We report mean decoding accuracy with standard errors across three runs, except for reconstruction, which uses one run\.

Across the testedλ\\lambdavalues, we recover the qualitative findings reported in the main text for cross\-subsidization, predictability\-driven set switching, and steering through reconstruction or side information\. The group\-size result holds at the largerλ\\lambda; at the smallerλ\\lambda, the enlarged group still becomes less represented, but the switch in favor of the competing group occurs in only one of three runs\.

Figure 7:Cross\-subsidization\.The current light becomes more strongly represented asqqincreases, although stronger regularization weakens this effect\. Dashed lines mark50%50\\%chance light\-decoding accuracy\.Figure 8:Changing predictability\.IncreasingpDp\_\{D\}from0\.600\.60to11changes the better\-represented pair fromA/BA/BtoC/DC/Din all three runs at every regularization weight\. Stronger regularization reduces the separation between the pairs\. The change need not be monotonic: atr=0\.20r=0\.20, mean decoding accuracy forC/DC/Ddecreases frompD=0\.95p\_\{D\}=0\.95topD=1p\_\{D\}=1\. Dashed lines mark50%50\\%chance accuracy\.Figure 9:Changing group size\.Enlarging theC/DC/Dgroup reduces its decoding accuracy in every run at every regularization weight\. AtN=16N=16,A/BA/Bbecomes better represented in all three runs atr=0\.10,0\.20r=0\.10,0\.20, but only one of three atr=0\.05r=0\.05\. We average decoding accuracy across positions within each group; dashed lines mark50%50\\%chance accuracy\.Figure 10:Auxiliary reconstruction\.ReconstructingCCcan also strengthen its predictive partnerDD, which receives no reconstruction supervision\. Atβ=0\.36\\beta=0\.36,C/DC/Dis better represented thanA/BA/Bat everypDp\_\{D\}and regularization weight\. Increasingβ\\betadoes not always improve decoding: atr=0\.20,pD=0\.75r=0\.20,p\_\{D\}=0\.75, bothCCandDDdecline fromβ=0\.135\\beta=0\.135toβ=0\.36\\beta=0\.36\. All four images remain visible to the encoder\. Cells show decoding accuracy from one training run, using a shared5050–90%90\\%color scale; numerical values below50%50\\%remain visible\.Figure 11:Predictor side information\.SupplyingCCstrengthens representation ofAAandDDwhile weakeningBBatpD=0\.75,0\.85p\_\{D\}=0\.75,0\.85in all three runs at every regularization weight\. The effect varies across the sweep and does not uniformly replace theA/BA/Bpair withC/DC/D\. Colored curves supplyCCto the predictor; gray curves omit side input\. All probes read the main latent without access to the side input; supplyingCCalso adds encoder capacity\. Panels show decoding accuracy with standard errors across three runs; dotted lines mark50%50\\%chance accuracy\.
## Appendix EWorst\- and Best\-Position Readouts for Group Size

We check whether the reversal in Figure[4](https://arxiv.org/html/2610.27364#S5.F4)also appears at the worst and best positions, using the same frozen encoders and position\-specific image\-detection probes described in Appendix[B\.3](https://arxiv.org/html/2610.27364#A2.SS3)\. For each optimization seed, we first average accuracy across target identities at each position, then take the minimum or maximum across positions within each group\. Both measures show the same endpoint reversal in all three seeds: the competing cycle is more strongly represented atN=2N=2, whereas the smallerA/BA/Bgroup is more strongly represented atN=16N=16\. The average probability of copying an image from the preceding position in each cycle remains unchanged across these sizes\.

Figure 12:Worst\- and best\-position readouts show the same reversal\.Both measures favor the competing cycle at the smallest size andA/BA/Bat the largest, showing that the reversal also holds for the least and most decodable positions within each group\. Curves show means across three training runs; error bars show standard errors\. Probes are trained on collages from images held out from encoder training and evaluated on separate collages\. The dashed line marks the chance image\-detection accuracy of50%50\\%\.C/DC/Daccuracy decreases across all four sizes in every seed, butA/BA/Baccuracy is not monotonic, and the intermediate\-size preferences vary across seeds\. AtN=16N=16,A/BA/Bdecoding also varies more across optimization seeds\. We hold the training and evaluation budget per position fixed across group sizes\. Appendix[B\.3](https://arxiv.org/html/2610.27364#A2.SS3)describes the constant\-area rendering procedure\.