Observation-Grounded Self-Predictive Reinforcement Learning for Visual Continuous Control

# Observation-Grounded Self-Predictive Reinforcement Learning for Visual Continuous Control
Source: [https://arxiv.org/html/2608.05989](https://arxiv.org/html/2608.05989)
###### Abstract

Sample\-efficient policy learning from pixels is a long\-standing challenge in reinforcement learning \(RL\)\. Recent dynamics\-based representation learning methods have significantly improved the sample efficiency of model\-free visual RL by learning dynamics\-aware representations through auxiliary prediction performed either in latent space \(self\-prediction\) or observation space \(observation prediction\)\. However, state\-of\-the\-art methods from both categories still struggle on challenging visual control tasks when training data is limited\. We posit that relying on either predictive objective alone may be insufficient\. Self\-prediction encourages latent representations to be temporally predictive over multiple future steps, but does not explicitly require latent transitions to align with observation\-level dynamics\. In contrast, observation prediction grounds learned representations in observation\-level dynamics, but does not directly regularize the temporal predictability of latent representations over extended horizons\. In this paper, we propose Observation\-Grounded Self\-Predictive Representations \(OG\-SPR\), a model\-free visual RL algorithm for continuous control that learns representations that are both temporally predictive in latent space and grounded in observation\-level dynamics\. Building on an actor\-critic framework, OG\-SPR incorporates two core auxiliary objectives: multi\-step latent self\-prediction and next\-observation prediction\. In addition, OG\-SPR uses short\-term value prediction as an auxiliary task to further stabilize value learning\. We empirically show that directly imposing latent self\-prediction on the shared representation may over\-constrain it and does not necessarily improve performance\. To address this issue, OG\-SPR introduces two lightweight adapters for latent self\-prediction, allowing the shared representation to benefit from temporally predictive signals without being forced to directly satisfy the self\-prediction objective\. Experiments on 28 visual control tasks from the DeepMind Control Suite show that OG\-SPR improves aggregate performance over state\-of\-the\-art self\-predictive and observation\-predictive RL methods, with particularly pronounced gains in challenging domains such as dog and humanoid\.

## Introduction

![Refer to caption](https://arxiv.org/html/2608.05989v1/Figures/improvement.png)Figure 1:Performance improvements of OG\-SPR over state\-of\-the\-art baselines on the DeepMind Control Suite at 500k environment steps\. For each task, the baseline score is defined as the stronger result between the recent self\-predictive method MR\.Q\(Fujimotoet al\.[2025](https://arxiv.org/html/2608.05989#bib.bib3)\)and observation\-predictive method AnonMethod\(Anonymous[2026](https://arxiv.org/html/2608.05989#bib.bib46)\)\.Learning policies from pixel inputs is an important problem in reinforcement learning \(RL\)\. While prior work has shown the potential of mastering visual continuous control by model\-free methods, such as DrQ\(Yaratset al\.[2021](https://arxiv.org/html/2608.05989#bib.bib2)\)and DrQ\-v2\(Yaratset al\.[2022](https://arxiv.org/html/2608.05989#bib.bib1)\), sample efficiency remains a major challenge\. Recently, dynamics\-based representation learning has made significant progress in data\-efficient visual RL\(Schwarzeret al\.[2021](https://arxiv.org/html/2608.05989#bib.bib5),[2023](https://arxiv.org/html/2608.05989#bib.bib6); Fujimotoet al\.[2025](https://arxiv.org/html/2608.05989#bib.bib3); Kimet al\.[2025](https://arxiv.org/html/2608.05989#bib.bib7); Anonymous[2026](https://arxiv.org/html/2608.05989#bib.bib46)\)\. These methods learn dynamics\-aware representations for model\-free RL algorithms, without using a learned dynamics model for planning or value estimation, as done in model\-based RL methods\(Hansenet al\.[2022](https://arxiv.org/html/2608.05989#bib.bib36),[2024](https://arxiv.org/html/2608.05989#bib.bib11); Hafneret al\.[2025](https://arxiv.org/html/2608.05989#bib.bib10)\)\. Two representative lines of work are self\-predictive methods and observation\-predictive methods\. Self\-predictive approaches\(Zhenget al\.[2023](https://arxiv.org/html/2608.05989#bib.bib32); Scannellet al\.[2024](https://arxiv.org/html/2608.05989#bib.bib31); Fujimotoet al\.[2025](https://arxiv.org/html/2608.05989#bib.bib3)\), inspired by self\-supervised learning\(Grillet al\.[2020](https://arxiv.org/html/2608.05989#bib.bib8)\), learn representations by predicting latent embeddings of future observations over multiple time steps\. In contrast, observation\-predictive methods\(Geladaet al\.[2019](https://arxiv.org/html/2608.05989#bib.bib9); Anonymous[2026](https://arxiv.org/html/2608.05989#bib.bib46)\)train representations by predicting next observations\.

Both lines of work have recently achieved performance matching or surpassing state\-of\-the\-art model\-based methods, such as DreamerV3\(Hafneret al\.[2025](https://arxiv.org/html/2608.05989#bib.bib10)\)and TD\-MPC2\(Hansenet al\.[2024](https://arxiv.org/html/2608.05989#bib.bib11)\), on diverse tasks\(Fujimotoet al\.[2025](https://arxiv.org/html/2608.05989#bib.bib3)\)\. This suggests that learning meaningful representations is a key enabler for data\-efficient visual RL\. Despite this progress, the success of these methods remains uneven across tasks\. Strong recent methods from both lines still struggle in challenging visual control domains under limited data budgets, such as 500k environment steps\(Fujimotoet al\.[2025](https://arxiv.org/html/2608.05989#bib.bib3); Anonymous[2026](https://arxiv.org/html/2608.05989#bib.bib46)\)\. We posit that relying on either latent self\-prediction or next\-observation prediction alone may be insufficient for data\-efficient visual continuous control\. Self\-prediction encourages latent representations to be temporally predictive over multiple future steps, but does not explicitly require latent transitions to align with observation\-level dynamics\. In contrast, observation prediction grounds learned representations in observation\-level dynamics through future observation reconstruction, but does not directly enforce long\-horizon temporal predictability in latent space\. These two objectives therefore impose complementary inductive biases, motivating the following question:

*Can a model\-free visual RL agent become more data\-efficient in continuous control by learning representations that are both temporally predictive in latent space and grounded in observation\-level dynamics?*

In this paper, we provide a positive answer to this question by exploring how to leverage the complementary strengths of latent self\-prediction and observation prediction\. Specifically, we present Observation\-Grounded Self\-Predictive Representations \(OG\-SPR\), a model\-free RL algorithm that learns representations satisfying both objectives\. Building on an off\-policy actor\-critic framework\(Suttonet al\.[1998](https://arxiv.org/html/2608.05989#bib.bib12)\), OG\-SPR learns state\-action representations through two dynamics\-oriented auxiliary tasks: multi\-step latent self\-prediction and next\-observation prediction\. In addition to these dynamics\-oriented objectives, OG\-SPR also includes short\-term value prediction as an auxiliary objective to stabilize value learning\(Anonymous[2026](https://arxiv.org/html/2608.05989#bib.bib46)\)\. However, as shown in our ablation study, naively combining these auxiliary objectives does not effectively improve performance\. We hypothesize that directly imposing the latent self\-prediction objective on the shared representation may over\-constrain it\. To mitigate this issue, OG\-SPR introduces two lightweight adapters for latent self\-prediction, with one placed after the observation encoder and the other after the state\-action encoder\. These adapters form an adapter\-mediated self\-prediction branch, so that the shared representation can receive temporally predictive learning signals without being forced to directly satisfy the self\-prediction objective\.

We evaluate OG\-SPR on 28 visual control tasks from the DeepMind Control Suite \(DMControl\)\(Tassaet al\.[2018](https://arxiv.org/html/2608.05989#bib.bib13)\)under limited data budgets\. Experimental results show that OG\-SPR achieves better average performance than state\-of\-the\-art self\-predictive and observation\-predictive RL methods\(Fujimotoet al\.[2025](https://arxiv.org/html/2608.05989#bib.bib3); Anonymous[2026](https://arxiv.org/html/2608.05989#bib.bib46)\), with particularly strong gains in challenging domains such asdogandhumanoid, as shown in Figure[1](https://arxiv.org/html/2608.05989#Sx1.F1)\. OG\-SPR represents an initial step toward data\-efficient RL for complex visual continuous control tasks by integrating self\-predictive and observation\-predictive methods\. We hope this work can inspire further research on their intersection to improve the data efficiency of model\-free visual RL\.

## Related Work

### Representation Learning for Data\-Efficient RL

Representation learning is widely used to improve the sample efficiency of RL\. Prior work has explored a wide range of representation learning approaches for RL\(Echchahed and Castro[2025](https://arxiv.org/html/2608.05989#bib.bib38)\)\. Metric\-based methods\(Liaoet al\.[2023](https://arxiv.org/html/2608.05989#bib.bib20); Zhanget al\.[2025](https://arxiv.org/html/2608.05989#bib.bib21)\)shape the embedding space by enforcing task\-relevant similarities among state representations\. Data\-augmentation methods\(Yaratset al\.[2022](https://arxiv.org/html/2608.05989#bib.bib1); Maet al\.[2024](https://arxiv.org/html/2608.05989#bib.bib16)\)encourage the learned representations to be invariant to irrelevant visual changes\. Contrastive learning methods\(Eysenbachet al\.[2022](https://arxiv.org/html/2608.05989#bib.bib18); Liuet al\.[2025](https://arxiv.org/html/2608.05989#bib.bib19)\)learn representations by contrasting positive pairs against negative pairs\. Our work focuses on dynamics\-based representation learning, which learns dynamics\-aware representations for model\-free RL through dynamics prediction\(Geladaet al\.[2019](https://arxiv.org/html/2608.05989#bib.bib9); Fujimotoet al\.[2025](https://arxiv.org/html/2608.05989#bib.bib3)\)\.

### Dynamics\-Based Representation Learning

Recent dynamics\-based methods have achieved performance competitive with or better than model\-based methods, such as DreamerV3\(Hafneret al\.[2025](https://arxiv.org/html/2608.05989#bib.bib10)\)and TD\-MPC2\(Hansenet al\.[2024](https://arxiv.org/html/2608.05989#bib.bib11)\), on diverse tasks with less algorithmic and computational complexity\(Fujimotoet al\.[2025](https://arxiv.org/html/2608.05989#bib.bib3); Anonymous[2026](https://arxiv.org/html/2608.05989#bib.bib46)\)\. This branch of research can be broadly divided into self\-predictive and observation\-predictive methods\. Self\-predictive methods learn representations by predicting latent embeddings of future observations\. TD7\(Fujimotoet al\.[2023](https://arxiv.org/html/2608.05989#bib.bib22)\)learns state\-action representations in the latent dynamics space for low\-dimensional continuous control tasks\. Building on TD7, MR\.Q\(Fujimotoet al\.[2025](https://arxiv.org/html/2608.05989#bib.bib3)\)is a state\-of\-the\-art self\-predictive method that augments latent self\-prediction with additional predictive objectives, such as reward and termination prediction\. Observation\-predictive methods also train representations through dynamics modeling, but ground the prediction objective in observation space\. OFENet\(Otaet al\.[2020](https://arxiv.org/html/2608.05989#bib.bib23)\)is a representative observation\-predictive approach for low\-dimensional settings\. AnonMethod\(Anonymous[2026](https://arxiv.org/html/2608.05989#bib.bib46)\)is a recent observation\-predictive method\. It addresses a bottleneck of observation\-predictive representation learning in low\-dimensional settings, where reconstruction losses tend to be dominated by observation dimensions with large value ranges\. To mitigate this issue, AnonMethod performs both representation learning and RL in a normalized observation space, making observation\-predictive learning applicable across diverse domains\. However, existing dynamics\-based methods typically focus on prediction either in latent space or in observation space, and these two directions have largely evolved as separate lines of work\. The potential of training representations simultaneously in latent and observation spaces for model\-free RL remains underexplored\. OG\-SPR takes an initial step toward addressing this gap\.

### Model\-Based Reinforcement Learning

Model\-based methods, such as the Dreamer series\(Hafneret al\.[2020](https://arxiv.org/html/2608.05989#bib.bib24),[2021](https://arxiv.org/html/2608.05989#bib.bib25),[2025](https://arxiv.org/html/2608.05989#bib.bib10)\), also learn latent dynamics with observation prediction\. Our work differs from these methods in both learning objectives and how the learned dynamics are used\. In terms of learning objectives, OG\-SPR overlaps with these methods only in observation prediction, while the remaining objectives are distinct\. In terms of usage, OG\-SPR does not maintain a learned dynamics model for RL training\. Instead, it learns representations through dynamics modeling in both latent and observation spaces for model\-free RL\.

## Preliminaries

### Reinforcement Learning

Reinforcement learning \(RL\) addresses the problem of sequential decision making, usually formulated as a Markov Decision Process \(MDP\)\. An MDP can be represented by a tuple\(𝒮,𝒜,P,r,γ\)\(\\mathcal\{S\},\\mathcal\{A\},P,r,\\gamma\), where𝒮\\mathcal\{S\}and𝒜\\mathcal\{A\}denote the state and action spaces, respectively;P​\(st\+1\|st,at\)P\(s\_\{t\+1\}\|s\_\{t\},a\_\{t\}\)denotes the transition probability of the next statest\+1s\_\{t\+1\}given the current statests\_\{t\}and actionata\_\{t\};r:𝒮×𝒜→ℝr:\\mathcal\{S\}\\times\\mathcal\{A\}\\rightarrow\\mathbb\{R\}is the reward function;γ∈\[0,1\)\\gamma\\in\[0,1\)is the discount factor\. The objective of RL is to learn a policyπ:𝒮→𝒜\\pi:\\mathcal\{S\}\\rightarrow\\mathcal\{A\}that maximizes the discounted cumulative reward∑t=0∞γt​rt\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}r\_\{t\}\. Actor\-critic methods\(Suttonet al\.[1998](https://arxiv.org/html/2608.05989#bib.bib12)\)typically learn an action\-value functionQπ​\(s,a\)=𝔼π​\[∑t=0∞γt​rt\|s0=s,a0=a\]Q^\{\\pi\}\(s,a\)=\\mathbb\{E\}\_\{\\pi\}\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}r\_\{t\}\|s\_\{0\}=s,a\_\{0\}=a\]which models the expected return, from an initial statessand actionaa\.

In low\-dimensional settings, the observation an agent receives at time steptt, denoted asoto\_\{t\}, is usually treated as the statests\_\{t\}\(st:=ots\_\{t\}:=o\_\{t\}\)\. In complex settings \(e\.g\., visual RL\),oto\_\{t\}does not necessarily satisfy the Markov property and therefore may not be directly treated assts\_\{t\}\. This induces a partially observable MDP \(POMDP\)\. Following common practice\(Mnihet al\.[2013](https://arxiv.org/html/2608.05989#bib.bib37); Yaratset al\.[2022](https://arxiv.org/html/2608.05989#bib.bib1)\), we approximate the Markov state by stacking three consecutive prior observations into a statest:=\[ot−2,ot−1,ot\]s\_\{t\}:=\[o\_\{t\-2\},o\_\{t\-1\},o\_\{t\}\]\.

## Methodology

As shown in Figure[2](https://arxiv.org/html/2608.05989#Sx4.F2), OG\-SPR is built on an actor\-critic framework with value and policy networks\.

![Refer to caption](https://arxiv.org/html/2608.05989v1/Figures/framework1.png)

\(a\) Value network with two long\-term value prediction heads and three auxiliary prediction heads\. ![Refer to caption](https://arxiv.org/html/2608.05989v1/Figures/framework2.png) \(b\) Policy network with detached inputs\.

Figure 2:An illustration of the full OG\-SPR method\.An image\-based statessis first passed through an online encoderfffollowed by Adapter 1u1u\_\{1\}, producing a low\-dimensional latent state representationhh\. The latent state is then combined with the actionaaand fed into a state\-action encoderggto obtain a state\-action representationzz:

h=u1​\(f​\(s\)\),z=g​\(h,a\)\.\\displaystyle h=u\_\{1\}\\left\(f\(s\)\\right\),\\quad\\quad z=g\(h,a\)\.\(1\)Adapter 1 maps encoder features into the latent state space on which policy learning, value learning, and latent self\-prediction are built\. Similar to AnonMethod, the policy network takeshhwith gradients stopped, while the state\-action representationzzis used for long\-term value learning and three auxiliary tasks\.

### Value Learning

The value target is constructed based on TD3\(Fujimotoet al\.[2018](https://arxiv.org/html/2608.05989#bib.bib26)\)\. Specifically, we train two separate long\-term value predictorsQ1Q\_\{1\}andQ2Q\_\{2\}, with identical architectures\. Both predictors take the same state\-action representations as inputs\. To stabilize bootstrapping, we maintain target copies of the networks used for target value estimation\. Target networks and their outputs are denoted by adding a prime to their online counterparts \(e\.g\.,f′f^\{\\prime\}andh′h^\{\\prime\}\)\. Target parameters are not updated via gradient descent, but periodically synchronized with their online counterparts\(Fujimotoet al\.[2025](https://arxiv.org/html/2608.05989#bib.bib3)\)\. The smoothed target actionaπ′a^\{\\pi^\{\\prime\}\}is produced by the target policy networkπ′\\pi^\{\\prime\}with clipped Gaussian noise:

aπ′=clip⁡\(a′,−1,1\),a′=π′​\(h′\)\+clip⁡\(ϵ,−c,c\),ϵ∼𝒩​\(0,σ2\),\\begin\{gathered\}a^\{\\pi^\{\\prime\}\}=\\operatorname\{clip\}\(a^\{\\prime\},\-1,1\),\\\\ a^\{\\prime\}=\\pi^\{\\prime\}\(h^\{\\prime\}\)\+\\operatorname\{clip\}\(\\epsilon,\-c,c\),\\quad\\epsilon\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}\),\\end\{gathered\}\(2\)whereh′=u1′​\(f′​\(s\)\)h^\{\\prime\}=u\_\{1\}^\{\\prime\}\\left\(f^\{\\prime\}\(s\)\\right\)\. We usenn\-step returns for value learning\(Yaratset al\.[2022](https://arxiv.org/html/2608.05989#bib.bib1)\)\. Given a sampled transition sequenceτ=\(st,at,rt:t\+n−1,st\+n\)\\tau=\(s\_\{t\},a\_\{t\},r\_\{t:t\+n\-1\},s\_\{t\+n\}\), the value targetyyis:

y=∑i=0n−1γi​rt\+i\+γn​minj∈\{1,2\}⁡Qj′​\(zt\+n′\),zt\+n′=g′​\(ht\+n′,at\+nπ′\),\\begin\{gathered\}y=\\sum\_\{i=0\}^\{n\-1\}\\gamma^\{i\}r\_\{t\+i\}\+\\gamma^\{n\}\\min\_\{j\\in\\\{1,2\\\}\}\{Q^\{\\prime\}\_\{j\}\(z^\{\\prime\}\_\{t\+n\}\)\},\\\\ z^\{\\prime\}\_\{t\+n\}=g^\{\\prime\}\\left\(h^\{\\prime\}\_\{t\+n\},a\_\{t\+n\}^\{\\pi^\{\\prime\}\}\\right\),\\end\{gathered\}\(3\)whereht\+n′=u1′​\(f′​\(st\+n\)\)h^\{\\prime\}\_\{t\+n\}=u\_\{1\}^\{\\prime\}\(f^\{\\prime\}\(s\_\{t\+n\}\)\), andat\+nπ′a\_\{t\+n\}^\{\\pi^\{\\prime\}\}is computed based onst\+ns\_\{t\+n\}via Equation[2](https://arxiv.org/html/2608.05989#Sx4.E2)\. We use the Huber loss instead of mean squared error \(MSE\) for value learning to eliminate bias from prioritized sampling\(Fujimotoet al\.[2020](https://arxiv.org/html/2608.05989#bib.bib39),[2025](https://arxiv.org/html/2608.05989#bib.bib3)\):

ℒValue=Huber⁡\(Q1​\(zt\),y\)\+Huber⁡\(Q2​\(zt\),y\)\.\\displaystyle\\mathcal\{L\}\_\{\\text\{Value\}\}=\\operatorname\{Huber\}\\left\(Q\_\{1\}\(z\_\{t\}\),y\\right\)\+\\operatorname\{Huber\}\\left\(Q\_\{2\}\(z\_\{t\}\),y\\right\)\.\(4\)

### Policy Learning

The policy networkπ\\piis optimized using the deterministic policy gradient\(Silveret al\.[2014](https://arxiv.org/html/2608.05989#bib.bib33)\)\. An L2 regularization term is added to the pre\-activation policy outputsa~π\\tilde\{a\}^\{\\pi\}, which helps avoid local minima when rewards and value estimates are sparse\(Schrittwieseret al\.[2020](https://arxiv.org/html/2608.05989#bib.bib34); Fujimotoet al\.[2025](https://arxiv.org/html/2608.05989#bib.bib3)\):

aπ=tanh⁡\(a~π\),ℒPolicy=−12​∑i∈\{1,2\}Qi​\(z\)\+λpre\-activ​‖a~π‖22,\\begin\{gathered\}a^\{\{\\pi\}\}=\\tanh\(\\tilde\{a\}^\{\\pi\}\),\\\\ \\mathcal\{L\}\_\{\\text\{Policy\}\}=\-\\frac\{1\}\{2\}\\sum\_\{i\\in\\\{1,2\\\}\}\{Q\_\{i\}\(z\)\}\+\\lambda\_\{\\text\{pre\-activ\}\}\\\|\\tilde\{a\}^\{\\pi\}\\\|\_\{2\}^\{2\},\\end\{gathered\}\(5\)wherez=g​\(u1​\(f​\(s\)\),aπ\)z=g\\left\(u\_\{1\}\(f\(s\)\),a^\{\\pi\}\\right\)\.aπa^\{\\pi\}is computed fromssusing the online encoder, Adapter 1, and policy network without policy smoothing noise\.

### Auxiliary Tasks

#### Latent Self\-Prediction\.

Starting from the current latent statehth\_\{t\}, future latent states are predicted recursively by applying the state\-action encoderggand Adapter 2u2u\_\{2\}with the corresponding actions from the sampled transition sequence:

h^t\+1\\displaystyle\\hat\{h\}\_\{t\+1\}=u2​\(g​\(ht,at\)\),\\displaystyle=u\_\{2\}\(g\(\{h\}\_\{t\},a\_\{t\}\)\),\(6\)h^t\+i\+1\\displaystyle\\hat\{h\}\_\{t\+i\+1\}=u2​\(g​\(h^t\+i,at\+i\)\),i=1,…,K−1\.\\displaystyle=u\_\{2\}\(g\(\\hat\{h\}\_\{t\+i\},a\_\{t\+i\}\)\),\\quad i=1,\\ldots,K\-1\.Adapter 2 maps state\-action representations back to the latent space defined by Adapter 1\. Both adapters use the same output activation function to maintain consistent output spaces\. By inserting this adapter between the state\-action representation and the latent prediction target, this design prevents the self\-prediction objective from being directly imposed on the shared state\-action representation, thereby alleviating the constraint on it\. The targets for latent self\-prediction are computed using target networks\(Schwarzeret al\.[2021](https://arxiv.org/html/2608.05989#bib.bib5); Fujimotoet al\.[2025](https://arxiv.org/html/2608.05989#bib.bib3)\)\. Given the target latent embeddings\(h′t\+1,…,h′t\+K\)\(\{h^\{\\prime\}\}\_\{t\+1\},\\ldots,\{h^\{\\prime\}\}\_\{t\+K\}\)produced by the target encoderf′f^\{\\prime\}and target Adapter 1u1′u\_\{1\}^\{\\prime\}, the self\-prediction loss is computed using the mean squared error \(MSE\):

ℒSelf\-predictive=∑k=1KMSE⁡\(h^t\+k,h′t\+k\)\.\\mathcal\{L\}\_\{\\text\{Self\-predictive\}\}=\\sum\_\{k=1\}^\{K\}\\operatorname\{MSE\}\\left\(\\hat\{h\}\_\{t\+k\},\{h^\{\\prime\}\}\_\{t\+k\}\\right\)\.\(7\)

#### Next\-Observation Prediction\.

The second auxiliary task predicts the next observation from the state\-action representation\. Specifically, a decoderDec\\operatorname\{Dec\}takesztz\_\{t\}as input and reconstructs the next observationot\+1o\_\{t\+1\}using the MSE loss:

ℒRec=‖Dec⁡\(zt\)−ot\+1‖22\.\\displaystyle\\mathcal\{L\}\_\{\\text\{Rec\}\}=\\left\\\|\\operatorname\{Dec\}\(z\_\{t\}\)\-o\_\{t\+1\}\\right\\\|\_\{2\}^\{2\}\.\(8\)

#### Short\-Term Value Prediction\.

We adopt short\-term value prediction as an auxiliary stabilizing objective, which provides lower\-variance training signals than bootstrapped value targets and stabilizes representation learning when value estimates are noisy\(Anonymous[2026](https://arxiv.org/html/2608.05989#bib.bib46)\)\. A predictorqqtakes state\-action representationszzas input and predicts the cumulative discounted reward over the nextnnsteps\. The auxiliary loss is computed as the cross\-entropy \(CE\) loss between the predicted logits and a two\-hot encoding of the target:

Rt=∑j=0n−1γj​rt\+j,ℒn\-step=CE⁡\(q​\(zt\),TwoHot⁡\(Rt\)\),\\begin\{gathered\}R\_\{t\}=\\sum\_\{j=0\}^\{n\-1\}\\gamma^\{j\}r\_\{t\+j\},\\\\ \\mathcal\{L\}\_\{\\text\{n\-step\}\}=\\operatorname\{CE\}\\left\(q\(z\_\{t\}\),\\operatorname\{TwoHot\}\(R\_\{t\}\)\\right\),\\end\{gathered\}\(9\)whereRtR\_\{t\}is the non\-bootstrapped term in the value target in Equation[3](https://arxiv.org/html/2608.05989#Sx4.E3)\. We use a non\-uniform return support for the two\-hot encoding, with bin locations generated by thesymexp\\operatorname\{symexp\}transform,symexp⁡\(x\)=sign⁡\(x\)​\(exp⁡\(\|x\|\)−1\)\\operatorname\{symexp\}\(x\)=\\operatorname\{sign\}\(x\)\(\\exp\(\|x\|\)\-1\)\(Hafneret al\.[2025](https://arxiv.org/html/2608.05989#bib.bib10)\)\.

During training, gradients from the policy loss are not propagated to the online encoder and Adapter 1, whereas all components of the value network are jointly optimized using the combined lossℒ\\mathcal\{L\}:

ℒ=ℒValue\+λRec​ℒRec\+λn\-step​ℒn\-step\\displaystyle\\mathcal\{L\}=\\mathcal\{L\}\_\{\\text\{Value\}\}\+\\lambda\_\{\\text\{Rec\}\}\\mathcal\{L\}\_\{\\text\{Rec\}\}\+\\lambda\_\{\\text\{n\-step\}\}\\mathcal\{L\}\_\{\\text\{n\-step\}\}\(10\)\+λSelf\-predictive​ℒSelf\-predictive,\\displaystyle\+\\lambda\_\{\\text\{Self\-predictive\}\}\\mathcal\{L\}\_\{\\text\{Self\-predictive\}\},whereλRec\\lambda\_\{\\text\{Rec\}\},λn\-step\\lambda\_\{\\text\{n\-step\}\}andλSelf\-predictive\\lambda\_\{\\text\{Self\-predictive\}\}denote the auxiliary loss weights for next\-observation prediction, short\-term value prediction, and latent self\-prediction, respectively\. The pseudocode of OG\-SPR is provided in the appendix\.

## Experiments

In this section, we first compare OG\-SPR with baselines in terms of performance and computational efficiency on continuous control tasks\. We then conduct ablation studies to analyze the key components of OG\-SPR, followed by a representation analysis to examine how self\-prediction and observation prediction affect the learned state and state\-action representations\. Finally, we provide an additional evaluation on discrete\-action domains to examine the behavior of OG\-SPR beyond continuous\-control settings\.

### Experimental Setup

#### Environments\.

We primarily evaluate OG\-SPR on the DMControl benchmark\(Tassaet al\.[2018](https://arxiv.org/html/2608.05989#bib.bib13)\), a collection of continuous control tasks built on the MuJoCo simulator\(Todorovet al\.[2012](https://arxiv.org/html/2608.05989#bib.bib27)\)\. The maximum episode score for each task is 1000\. We consider 28 tasks used in prior work\(Fujimotoet al\.[2025](https://arxiv.org/html/2608.05989#bib.bib3); Anonymous[2026](https://arxiv.org/html/2608.05989#bib.bib46)\)\. Agents are trained for 500k environment steps, equivalent to 1M frames in the original environment under an action repeat of 2\. The input state is constructed by stacking three previous frames, which are resized to 84×\\times84 pixels in RGB format\.

#### Baselines\.

We compare OG\-SPR with representative data\-efficient RL methods, including both model\-based and model\-free approaches: \(1\) TD\-MPC2\(Hansenet al\.[2024](https://arxiv.org/html/2608.05989#bib.bib11)\), a strong model\-based method for continuous control tasks, \(2\) DreamerV3\(Hafneret al\.[2025](https://arxiv.org/html/2608.05989#bib.bib10)\), a general\-purpose model\-based RL algorithm that performs well across diverse domains, \(3\) MR\.Q\(Fujimotoet al\.[2025](https://arxiv.org/html/2608.05989#bib.bib3)\), a state\-of\-the\-art self\-predictive method that achieves performance competitive with model\-based RL approaches, DreamerV3 and TD\-MPC2, \(4\) AnonMethod\(Anonymous[2026](https://arxiv.org/html/2608.05989#bib.bib46)\), a recent observation\-predictive method achieving performance competitive with or better than MR\.Q, \(5\) DrQ\-v2\(Yaratset al\.[2022](https://arxiv.org/html/2608.05989#bib.bib1)\), a simple yet strong model\-free RL baseline for visual continuous control tasks based on data augmentation\.

#### Implementation Details\.

We follow MR\.Q in constructing the actor\-critic backbone of OG\-SPR\. Each adapter is implemented as a linear layer\. We apply random shift augmentation to pixel observations\(Yaratset al\.[2022](https://arxiv.org/html/2608.05989#bib.bib1)\), but use different augmentation strategies for the observation\-predictive and self\-predictive objectives\. For next\-observation prediction, we apply the same sampled spatial shift to both the input observation and the prediction target to preserve spatial alignment\. For latent self\-prediction, we apply independently sampled shifts to future target observations\. The prediction horizon for latent self\-prediction is set to 5, following prior self\-predictive methods\(Schwarzeret al\.[2021](https://arxiv.org/html/2608.05989#bib.bib5); Fujimotoet al\.[2025](https://arxiv.org/html/2608.05989#bib.bib3)\)\. The auxiliary loss weights are kept fixed across tasks\. We adopt the same default settings as AnonMethod for the observation\-predictive and short\-term value objectives, settingλRec=0\.1\\lambda\_\{\\text\{Rec\}\}=0\.1andλn\-step=1\.0\\lambda\_\{\\text\{n\-step\}\}=1\.0\. To determineλSelf\-predictive\\lambda\_\{\\text\{Self\-predictive\}\}, we start from the default weight used in SPR\(Schwarzeret al\.[2021](https://arxiv.org/html/2608.05989#bib.bib5)\), 2\.0, although SPR uses a cosine\-similarity loss while OG\-SPR uses MSE\. We then selectλSelf\-predictive\\lambda\_\{\\text\{Self\-predictive\}\}using a small subset containing onlyquadruped\-runanddog\-run, searching over candidate values spaced by 1\.0\. The resulting value, 5\.0, is used for all 28 tasks without per\-task tuning\. Additional details are provided in the appendix\.

#### Evaluation Protocol\.

All experiments are run for 5 seeds\. For each seed, we evaluate the agent every 5k environment steps over 10 episodes and report the average episode return as the evaluation score\. Results for all baselines are obtained by re\-running their official implementations, except for DrQ\-v2\. For DrQ\-v2, we use the results reported in MR\.Q\.

Most prior work reports results on DMControl using raw, unnormalized scores\(Yaratset al\.[2021](https://arxiv.org/html/2608.05989#bib.bib2),[2022](https://arxiv.org/html/2608.05989#bib.bib1); Zhenget al\.[2023](https://arxiv.org/html/2608.05989#bib.bib32); Hansenet al\.[2024](https://arxiv.org/html/2608.05989#bib.bib11); Hafneret al\.[2025](https://arxiv.org/html/2608.05989#bib.bib10); Fujimotoet al\.[2025](https://arxiv.org/html/2608.05989#bib.bib3)\)\. However, aggregate statistics over raw scores can be dominated by relatively easy tasks with high returns, such ascartpole\-balanceandreacher\-easy\. This may obscure improvements on more challenging tasks, such asdog\-runandhumanoid\-walk, where raw scores are typically much lower\. To provide a complementary view of aggregate performance, we additionally report DrQv2\-normalized scores, which measure relative improvement over DrQ\-v2, a strong model\-free baseline for visual continuous control\. The metric is computed in a way similar to the human\-normalized score\(Wanget al\.[2016](https://arxiv.org/html/2608.05989#bib.bib35)\)commonly used for Atari games\(Bellemareet al\.[2013](https://arxiv.org/html/2608.05989#bib.bib30)\):

DrQv2−Normalized⁡\(x\)=x−random scoreDrQ\-v2 score−random score\.\\displaystyle\\operatorname\{DrQv2\-Normalized\}\(x\)=\\frac\{x\-\\text\{random score\}\}\{\{\\text\{DrQ\-v2 score\}\-\\text\{random score\}\}\}\.This normalization reduces the influence of tasks where DrQ\-v2 already performs well and highlights improvements on tasks where DrQ\-v2 remains weak\. Benchmark\-level aggregate performance is reported using the mean and interquartile mean \(IQM\) over task\-level scores, where each task\-level score is averaged across seeds\.

### Main Results

As shown in Table[1](https://arxiv.org/html/2608.05989#Sx5.T1), OG\-SPR achieves the best aggregate performance on both raw scores and DrQv2\-normalized scores\.

TasksDrQ\-v2TD\-MPC2DreamerV3MR\.QAnonMethodOG\-SPR \(ours\)Raw Scores of Selected Challenging Tasksdog\-run10 \[9, 12\]8 \[5, 11\]29 \[23, 36\]51 \[45, 56\]55 \[46, 69\]90\[64, 115\]dog\-stand43 \[37, 49\]161 \[147, 175\]118 \[78, 157\]251 \[233, 269\]239 \[214, 263\]357\[267, 448\]dog\-trot14 \[11, 18\]16 \[12, 18\]44 \[34, 54\]70 \[61, 80\]61 \[59, 63\]105\[79, 128\]dog\-walk22 \[18, 29\]14 \[12, 17\]42 \[31, 52\]90 \[79, 101\]81 \[72, 91\]108\[102, 118\]humanoid\-run1 \[1, 1\]1 \[1, 1\]1 \[1, 2\]1 \[1, 2\]1 \[1, 1\]3\[2, 4\]humanoid\-run \(2M\)2 \[2, 3\]1 \[1, 1\]4 \[1, 9\]27 \[4, 50\]1 \[1, 1\]100\[94, 106\]humanoid\-stand6 \[6, 7\]6 \[6, 7\]7 \[3, 10\]8 \[7, 8\]6 \[5, 7\]12\[8, 20\]humanoid\-stand \(2M\)9 \[8, 11\]7 \[6, 7\]11 \[7, 17\]79 \[8, 200\]31 \[8, 55\]339\[284, 401\]humanoid\-walk2 \[2, 2\]2 \[1, 2\]2 \[1, 3\]3 \[2, 5\]2 \[2, 3\]15\[7, 22\]humanoid\-walk \(2M\)19 \[2, 52\]2 \[2, 2\]2 \[1, 3\]48 \[3, 93\]54 \[3, 153\]316\[299, 333\]Aggregate Raw Results of 28 TasksMean510 \[497, 523\]485 \[467, 505\]536 \[507, 563\]599 \[591, 606\]607 \[602, 612\]626\[620, 632\]IQM545 \[519, 564\]489 \[453, 524\]571 \[516, 622\]685 \[670, 699\]702 \[692, 711\]730\[718, 741\]Aggregate DrQv2\-Normalized Results of 28 TasksMean1\.001\.35 \[1\.27, 1\.42\]1\.94 \[1\.64, 2\.24\]2\.84 \[2\.72, 2\.96\]2\.65 \[2\.51, 2\.80\]5\.05\[4\.40, 5\.71\]IQM1\.000\.97 \[0\.92, 1\.01\]1\.21 \[1\.14, 1\.32\]1\.29 \[1\.24, 1\.36\]1\.25 \[1\.20, 1\.31\]1\.61\[1\.50, 1\.76\]Table 1:Results on selected challenging DMControl tasks and aggregate results over all 28 tasks at default 500k environment steps\.2Mdenotes results evaluated at 2M environment steps\. See the appendix for humanoid\-domain results evaluated at 1M and 1\.5M environment steps\. Brackets denote 95% bootstrap confidence intervals\. Bold numbers indicate the best performance\.Notably, OG\-SPR substantially outperforms both MR\.Q and AnonMethod in terms of DrQv2\-normalized scores\. Its mean DrQ\-v2\-normalized score is approximately1\.8×1\.8\\timesthat of MR\.Q and1\.9×1\.9\\timesthat of AnonMethod\. Figure[1](https://arxiv.org/html/2608.05989#Sx1.F1)shows the percentage improvement of OG\-SPR over the stronger of MR\.Q and AnonMethod on each task, computed using raw scores\. OG\-SPR brings particularly large relative gains in challenging visual control domains, achieving20\.9–63\.8%and61\.7–333\.7%improvements in thedogandhumanoiddomains, respectively\. Under the default 500k\-step budget, all methods obtain relatively low scores in thehumanoiddomain, making it difficult to fully reveal performance differences in this challenging domain\. We therefore additionally evaluate the humanoid tasks with a relaxed training budget of 2M environment steps in Table[1](https://arxiv.org/html/2608.05989#Sx5.T1)\. Under this setting, OG\-SPR still achieves substantial gains over all baselines\. We also provide aggregate learning curves computed using raw scores and DrQv2\-normalized scores in Figures[3](https://arxiv.org/html/2608.05989#Sx5.F3)\. Full per\-task results are provided in the appendix\.

![Refer to caption](https://arxiv.org/html/2608.05989v1/Figures/main_results_anon_raw_agg.png)

![Refer to caption](https://arxiv.org/html/2608.05989v1/Figures/main_results_anon_agg.png)

Figure 3:Aggregate learning curves on DMControl, computed as the mean over 28 tasks using raw scores and DrQ\-v2\-normalized scores\. Shaded area captures a 95% bootstrap confidence interval\.Since OG\-SPR incorporates both observation\-predictive and self\-predictive objectives, it incurs higher training cost than AnonMethod and MR\.Q\. OG\-SPR requires 18\.3%\\%more training time than MR\.Q, and trains substantially faster than DreamerV3 \(see the appendix\) when all methods are implemented in PyTorch\(Paszkeet al\.[2019](https://arxiv.org/html/2608.05989#bib.bib28)\)\.

### Ablation Study

We conduct ablation studies for two purposes:

1. 1\.To evaluate the effectiveness of learning representations that are both temporally predictive in latent space and grounded in observation\-level dynamics\.
2. 2\.To examine the importance of the adapter design\.

The aggregate ablation results are summarized in Table[2](https://arxiv.org/html/2608.05989#Sx5.T2)\. We denote byOG\-SPR \- OPthe variant that removes the observation\-predictive objectiveℒRec\\mathcal\{L\}\_\{\\text\{Rec\}\}\.OG\-SPR \- SPindicates the variant that removes the self\-predictive objectiveℒSelf\-predictive\\mathcal\{L\}\_\{\\text\{Self\-predictive\}\}\.OG\-SPR \- SVPdenotes removing the short\-term value prediction lossℒn\-step\\mathcal\{L\}\_\{\\text\{n\-step\}\}\. Overall, removing any auxiliary objective degrades the performance of OG\-SPR\. The performance drop is especially pronounced when removing the self\-predictive objective\.

To study the adapter design, we compare OG\-SPR with three variants\. InShared Adapter, the two adapters share parameters, reducing the flexibility of the mappings used for recursive self\-prediction\. InNo Adapter, both adapters are removed, and the self\-prediction objective is directly imposed on the shared representation\. These two variants progressively reduce the flexibility of the adapter\-mediated self\-prediction branch\. BothShared AdapterandNo Adapterunderperform complete OG\-SPR, indicating that reducing or removing adapter flexibility hurts performance\. We also considerAdapter 2 Only, where Adapter 1 is removed, which is the design used in MR\.Q\. Compared with the full OG\-SPR,Adapter 2 Onlyperforms worse because removing Adapter 1 makes the state encoder output space inconsistent with the output space of Adapter 2, as the former uses an ELU activation whereas the latter uses no activation function\. These results support the importance of using two separate adapters with matched output spaces for latent self\-prediction\.

Table 2:Aggregate ablation results on the DMControl benchmark\. The mean and IQM are reported for both raw scores and DrQv2\-Normalized scores\. Bold numbers indicate the best performance\.
### Analyzing State and State\-Action Representations

To understand how self\-prediction and observation prediction shape representations, we analyze the state and state\-action representations learned by OG\-SPR and its two ablated variants,OG\-SPR \- OPandOG\-SPR \- SP\. We freeze the learned representation networks and train lightweight linear predictors to recover the proprioceptive states corresponding to the visual observations\. We consider two prediction tasks\. The first predicts the current proprioceptive state from the latent state representation, with the prediction loss denoted byℒstate\\mathcal\{L\}\_\{\\text\{state\}\}\. The second predicts the next proprioceptive state from the state\-action representation, with the loss denoted byℒnext\-state\\mathcal\{L\}\_\{\\text\{next\-state\}\}\. Details and results are provided in the appendix\.

The two ablated variants exhibit distinct strengths\. The observation\-predictive variantOG\-SPR \- SPgenerally achieves lowerℒnext\-state\\mathcal\{L\}\_\{\\text\{next\-state\}\}than the self\-predictive variantOG\-SPR \- OP, suggesting that observation prediction helps state\-action representations capture more accurate dynamics information\. In contrast, the self\-predictive variant tends to achieve lowerℒstate\\mathcal\{L\}\_\{\\text\{state\}\}, suggesting that self\-prediction learns state representations that are more informative for recovering the underlying proprioceptive state\. These observations are consistent with the different inductive biases of the two predictive objectives\. Multi\-step latent self\-prediction primarily regularizes the state representation to be temporally predictive over horizons, without explicitly constraining latent transitions to align with observation\-level dynamics\. In contrast, next\-observation prediction learns the state\-action representation through next\-observation reconstruction, encouraging it to capture observation\-grounded dynamics information, while not directly imposing a multi\-step predictability constraint on the latent state representation\.

OG\-SPR inherits the characteristics of both objectives\. Itsℒstate\\mathcal\{L\}\_\{\\text\{state\}\}is close to that ofOG\-SPR \- OP, while itsℒnext\-state\\mathcal\{L\}\_\{\\text\{next\-state\}\}often lies between those ofOG\-SPR \- OPandOG\-SPR \- SP\. Overall, OG\-SPR strikes a balance between extracting useful information from the current pixel input and capturing accurate dynamics information\.

### Evaluation on Discrete\-Action Tasks

Although our main focus is visual continuous control, we also evaluate OG\-SPR on Atari100k\(Kaiseret al\.[2020](https://arxiv.org/html/2608.05989#bib.bib29)\), a benchmark for data\-efficient discrete\-action visual RL, to examine how the proposed design behaves when adapted to a discrete\-action domain\. We adopt a lightweight discrete\-action adaptation following MR\.Q and AnonMethod, by modifying the output activation of the policy network\. The implementation details and hyperparameters are provided in the appendix\. We report aggregated human\-normalized scores, the standard evaluation metric for the Atari benchmark\. Similar to the DrQv2\-normalized score, we also consider SPR\-normalized scores, where each game score is normalized by the corresponding score of SPR\(Schwarzeret al\.[2021](https://arxiv.org/html/2608.05989#bib.bib5)\), a strong self\-predictive baseline for this benchmark\. As shown in Table[3](https://arxiv.org/html/2608.05989#Sx5.T3), OG\-SPR remains competitive on Atari100k, ranking first or second on three of the four aggregate metrics\. Together with the main DMControl results, these findings suggest that OG\-SPR is particularly effective in continuous\-control domains while remaining competitive in discrete\-action settings\.

One possible explanation for the less pronounced improvements on Atari100k is that adapting continuous\-control algorithms to discrete\-action domains may require nontrivial algorithmic modifications for cross\-domain adaptation\. The Dreamer family provides a representative example\. DreamerV2\(Hafneret al\.[2021](https://arxiv.org/html/2608.05989#bib.bib25)\)introduced discrete latent representations when extending Dreamer\(Hafneret al\.[2020](https://arxiv.org/html/2608.05989#bib.bib24)\)to Atari, while DreamerV3\(Hafneret al\.[2025](https://arxiv.org/html/2608.05989#bib.bib10)\)further introduced a set of cross\-domain stabilization techniques for stable learning across diverse domains\. Our Atari experiments instead use a minimal discrete\-action adaptation, allowing us to examine whether OG\-SPR transfers to discrete\-action domains without substantial cross\-domain modifications\. We leave the development of robust cross\-domain mechanisms for model\-free visual RL to future work\.

Table 3:Aggregate results on Atari100k\. We report the mean and IQM under both human\-normalized and SPR\-normalized scores\. Bold and underline indicate the best and second\-best results, respectively\.

## Conclusion and Limitations

In this paper, we presented Observation\-Grounded Self\-Predictive Representations \(OG\-SPR\), a model\-free visual RL algorithm that improves data efficiency by integrating multi\-step latent self\-prediction, next\-observation prediction, and short\-term value prediction within an actor\-critic framework\. To alleviate the direct constraint imposed by latent self\-prediction on the shared representation, OG\-SPR introduces two lightweight adapters that form an adapter\-mediated self\-prediction branch\. Experiments on DMControl show that OG\-SPR improves sample efficiency and achieves new state\-of\-the\-art aggregate performance on the benchmark, with particularly pronounced gains in challenging domains\. Despite the considerable room for improvement in these challenging domains under limited data budgets, OG\-SPR provides an initial step toward data\-efficient RL for complex visual continuous control tasks\.

Several limitations remain\. Although OG\-SPR can be adapted to discrete\-action domains, its gains are less consistent than those observed in visual continuous control\. Further work is needed to make the method robust across different action spaces\. Besides, OG\-SPR uses fixed auxiliary loss weights across tasks, which may not be optimal for all tasks\. Future work could explore adaptive weighting strategies to better coordinate multiple auxiliary objectives\.

## References

- Anonymous \(2026\)Title withheld for anonymous review\.Note:Citation suppressed to preserve double\-blind reviewCited by:[Figure 1](https://arxiv.org/html/2608.05989#Sx1.F1),[Introduction](https://arxiv.org/html/2608.05989#Sx1.p1.1),[Introduction](https://arxiv.org/html/2608.05989#Sx1.p2.1),[Introduction](https://arxiv.org/html/2608.05989#Sx1.p4.1),[Introduction](https://arxiv.org/html/2608.05989#Sx1.p5.1),[Dynamics\-Based Representation Learning](https://arxiv.org/html/2608.05989#Sx2.SSx2.p1.1),[Short\-Term Value Prediction\.](https://arxiv.org/html/2608.05989#Sx4.SSx3.SSS0.Px3.p1.3),[Environments\.](https://arxiv.org/html/2608.05989#Sx5.SSx1.SSS0.Px1.p1.1),[Baselines\.](https://arxiv.org/html/2608.05989#Sx5.SSx1.SSS0.Px2.p1.1)\.
- M\. G\. Bellemare, Y\. Naddaf, J\. Veness, and M\. Bowling \(2013\)The arcade learning environment: an evaluation platform for general agents\.J\. Artif\. Intell\. Res\.47,pp\. 253–279\.External Links:[Link](https://doi.org/10.1613/jair.3912),[Document](https://dx.doi.org/10.1613/JAIR.3912)Cited by:[Evaluation Protocol\.](https://arxiv.org/html/2608.05989#Sx5.SSx1.SSS0.Px4.p2.1)\.
- A\. Echchahed and P\. S\. Castro \(2025\)A survey of state representation learning for deep reinforcement learning\.Trans\. Mach\. Learn\. Res\.2025\.External Links:[Link](https://openreview.net/forum?id=gOk34vUHtz)Cited by:[Representation Learning for Data\-Efficient RL](https://arxiv.org/html/2608.05989#Sx2.SSx1.p1.1)\.
- B\. Eysenbach, T\. Zhang, S\. Levine, and R\. Salakhutdinov \(2022\)Contrastive learning as goal\-conditioned reinforcement learning\.InAdvances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 \- December 9, 2022,S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2022/hash/e7663e974c4ee7a2b475a4775201ce1f-Abstract-Conference.html)Cited by:[Representation Learning for Data\-Efficient RL](https://arxiv.org/html/2608.05989#Sx2.SSx1.p1.1)\.
- S\. Fujimoto, W\. Chang, E\. J\. Smith, S\. Gu, D\. Precup, and D\. Meger \(2023\)For SALE: state\-action representation learning for deep reinforcement learning\.InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 \- 16, 2023,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2023/hash/c20ac0df6c213db6d3a930fe9c7296c8-Abstract-Conference.html)Cited by:[Dynamics\-Based Representation Learning](https://arxiv.org/html/2608.05989#Sx2.SSx2.p1.1)\.
- S\. Fujimoto, P\. D’Oro, A\. Zhang, Y\. Tian, and M\. Rabbat \(2025\)Towards general\-purpose model\-free reinforcement learning\.InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025,External Links:[Link](https://openreview.net/forum?id=R1hIXdST22)Cited by:[Figure 1](https://arxiv.org/html/2608.05989#Sx1.F1),[Introduction](https://arxiv.org/html/2608.05989#Sx1.p1.1),[Introduction](https://arxiv.org/html/2608.05989#Sx1.p2.1),[Introduction](https://arxiv.org/html/2608.05989#Sx1.p5.1),[Representation Learning for Data\-Efficient RL](https://arxiv.org/html/2608.05989#Sx2.SSx1.p1.1),[Dynamics\-Based Representation Learning](https://arxiv.org/html/2608.05989#Sx2.SSx2.p1.1),[Value Learning](https://arxiv.org/html/2608.05989#Sx4.SSx1.p1.13),[Value Learning](https://arxiv.org/html/2608.05989#Sx4.SSx1.p1.6),[Policy Learning](https://arxiv.org/html/2608.05989#Sx4.SSx2.p1.2),[Latent Self\-Prediction\.](https://arxiv.org/html/2608.05989#Sx4.SSx3.SSS0.Px1.p1.6),[Environments\.](https://arxiv.org/html/2608.05989#Sx5.SSx1.SSS0.Px1.p1.1),[Baselines\.](https://arxiv.org/html/2608.05989#Sx5.SSx1.SSS0.Px2.p1.1),[Implementation Details\.](https://arxiv.org/html/2608.05989#Sx5.SSx1.SSS0.Px3.p1.4),[Evaluation Protocol\.](https://arxiv.org/html/2608.05989#Sx5.SSx1.SSS0.Px4.p2.1)\.
- S\. Fujimoto, D\. Meger, and D\. Precup \(2020\)An equivalence between loss functions and non\-uniform sampling in experience replay\.InAdvances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6\-12, 2020, virtual,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\. Balcan, and H\. Lin \(Eds\.\),External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/a3bf6e4db673b6449c2f7d13ee6ec9c0-Abstract.html)Cited by:[Value Learning](https://arxiv.org/html/2608.05989#Sx4.SSx1.p1.13)\.
- S\. Fujimoto, H\. van Hoof, and D\. Meger \(2018\)Addressing function approximation error in actor\-critic methods\.InProceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10\-15, 2018,J\. G\. Dy and A\. Krause \(Eds\.\),Proceedings of Machine Learning Research, Vol\.80,pp\. 1582–1591\.External Links:[Link](http://proceedings.mlr.press/v80/fujimoto18a.html)Cited by:[Value Learning](https://arxiv.org/html/2608.05989#Sx4.SSx1.p1.6)\.
- C\. Gelada, S\. Kumar, J\. Buckman, O\. Nachum, and M\. G\. Bellemare \(2019\)DeepMDP: learning continuous latent space models for representation learning\.InProceedings of the 36th International Conference on Machine Learning, ICML 2019, 9\-15 June 2019, Long Beach, California, USA,K\. Chaudhuri and R\. Salakhutdinov \(Eds\.\),Proceedings of Machine Learning Research, Vol\.97,pp\. 2170–2179\.External Links:[Link](http://proceedings.mlr.press/v97/gelada19a.html)Cited by:[Introduction](https://arxiv.org/html/2608.05989#Sx1.p1.1),[Representation Learning for Data\-Efficient RL](https://arxiv.org/html/2608.05989#Sx2.SSx1.p1.1)\.
- J\. Grill, F\. Strub, F\. Altché, C\. Tallec, P\. H\. Richemond, E\. Buchatskaya, C\. Doersch, B\. Á\. Pires, Z\. Guo, M\. G\. Azar, B\. Piot, K\. Kavukcuoglu, R\. Munos, and M\. Valko \(2020\)Bootstrap your own latent \- A new approach to self\-supervised learning\.InAdvances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6\-12, 2020, virtual,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\. Balcan, and H\. Lin \(Eds\.\),External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/f3ada80d5c4ee70142b17b8192b2958e-Abstract.html)Cited by:[Introduction](https://arxiv.org/html/2608.05989#Sx1.p1.1)\.
- D\. Hafner, T\. P\. Lillicrap, J\. Ba, and M\. Norouzi \(2020\)Dream to control: learning behaviors by latent imagination\.In8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26\-30, 2020,External Links:[Link](https://openreview.net/forum?id=S1lOTC4tDS)Cited by:[Model\-Based Reinforcement Learning](https://arxiv.org/html/2608.05989#Sx2.SSx3.p1.1),[Evaluation on Discrete\-Action Tasks](https://arxiv.org/html/2608.05989#Sx5.SSx5.p2.1)\.
- D\. Hafner, T\. P\. Lillicrap, M\. Norouzi, and J\. Ba \(2021\)Mastering atari with discrete world models\.In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3\-7, 2021,External Links:[Link](https://openreview.net/forum?id=0oabwyZbOu)Cited by:[Model\-Based Reinforcement Learning](https://arxiv.org/html/2608.05989#Sx2.SSx3.p1.1),[Evaluation on Discrete\-Action Tasks](https://arxiv.org/html/2608.05989#Sx5.SSx5.p2.1)\.
- D\. Hafner, J\. Pasukonis, J\. Ba, and T\. Lillicrap \(2025\)Mastering diverse control tasks through world models\.Nature,pp\. 1–7\.Cited by:[Introduction](https://arxiv.org/html/2608.05989#Sx1.p1.1),[Introduction](https://arxiv.org/html/2608.05989#Sx1.p2.1),[Dynamics\-Based Representation Learning](https://arxiv.org/html/2608.05989#Sx2.SSx2.p1.1),[Model\-Based Reinforcement Learning](https://arxiv.org/html/2608.05989#Sx2.SSx3.p1.1),[Short\-Term Value Prediction\.](https://arxiv.org/html/2608.05989#Sx4.SSx3.SSS0.Px3.p1.6),[Baselines\.](https://arxiv.org/html/2608.05989#Sx5.SSx1.SSS0.Px2.p1.1),[Evaluation Protocol\.](https://arxiv.org/html/2608.05989#Sx5.SSx1.SSS0.Px4.p2.1),[Evaluation on Discrete\-Action Tasks](https://arxiv.org/html/2608.05989#Sx5.SSx5.p2.1)\.
- N\. Hansen, H\. Su, and X\. Wang \(2022\)Temporal difference learning for model predictive control\.InInternational Conference on Machine Learning, ICML 2022, 17\-23 July 2022, Baltimore, Maryland, USA,K\. Chaudhuri, S\. Jegelka, L\. Song, C\. Szepesvári, G\. Niu, and S\. Sabato \(Eds\.\),Proceedings of Machine Learning Research, Vol\.162,pp\. 8387–8406\.External Links:[Link](https://proceedings.mlr.press/v162/hansen22a.html)Cited by:[Introduction](https://arxiv.org/html/2608.05989#Sx1.p1.1)\.
- N\. Hansen, H\. Su, and X\. Wang \(2024\)TD\-MPC2: scalable, robust world models for continuous control\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=Oxh5CstDJU)Cited by:[Introduction](https://arxiv.org/html/2608.05989#Sx1.p1.1),[Introduction](https://arxiv.org/html/2608.05989#Sx1.p2.1),[Dynamics\-Based Representation Learning](https://arxiv.org/html/2608.05989#Sx2.SSx2.p1.1),[Baselines\.](https://arxiv.org/html/2608.05989#Sx5.SSx1.SSS0.Px2.p1.1),[Evaluation Protocol\.](https://arxiv.org/html/2608.05989#Sx5.SSx1.SSS0.Px4.p2.1)\.
- L\. Kaiser, M\. Babaeizadeh, P\. Milos, B\. Osinski, R\. H\. Campbell, K\. Czechowski, D\. Erhan, C\. Finn, P\. Kozakowski, S\. Levine, A\. Mohiuddin, R\. Sepassi, G\. Tucker, and H\. Michalewski \(2020\)Model based reinforcement learning for atari\.In8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26\-30, 2020,External Links:[Link](https://openreview.net/forum?id=S1xCPJHtDB)Cited by:[Evaluation on Discrete\-Action Tasks](https://arxiv.org/html/2608.05989#Sx5.SSx5.p1.1)\.
- K\. Kim, J\. Ha, and Y\. Kim \(2025\)Self\-predictive dynamics for generalization of vision\-based reinforcement learning\.CoRRabs/2506\.05418\.External Links:[Link](https://doi.org/10.48550/arXiv.2506.05418),[Document](https://dx.doi.org/10.48550/ARXIV.2506.05418),2506\.05418Cited by:[Introduction](https://arxiv.org/html/2608.05989#Sx1.p1.1)\.
- W\. Liao, Z\. Zhang, and Y\. Yu \(2023\)Policy\-independent behavioral metric\-based representation for deep reinforcement learning\.InThirty\-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty\-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7\-14, 2023,B\. Williams, Y\. Chen, and J\. Neville \(Eds\.\),pp\. 8746–8754\.External Links:[Link](https://doi.org/10.1609/aaai.v37i7.26052),[Document](https://dx.doi.org/10.1609/AAAI.V37I7.26052)Cited by:[Representation Learning for Data\-Efficient RL](https://arxiv.org/html/2608.05989#Sx2.SSx1.p1.1)\.
- G\. Liu, M\. Tang, and B\. Eysenbach \(2025\)A single goal is all you need: skills and exploration emerge from contrastive RL without rewards, demonstrations, or subgoals\.InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025,External Links:[Link](https://openreview.net/forum?id=xCkgX4Xfu0)Cited by:[Representation Learning for Data\-Efficient RL](https://arxiv.org/html/2608.05989#Sx2.SSx1.p1.1)\.
- G\. Ma, L\. Li, S\. Zhang, Z\. Liu, Z\. Wang, Y\. Chen, L\. Shen, X\. Wang, and D\. Tao \(2024\)Revisiting plasticity in visual reinforcement learning: data, modules and training stages\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=0aR1s9YxoL)Cited by:[Representation Learning for Data\-Efficient RL](https://arxiv.org/html/2608.05989#Sx2.SSx1.p1.1)\.
- V\. Mnih, K\. Kavukcuoglu, D\. Silver, A\. Graves, I\. Antonoglou, D\. Wierstra, and M\. A\. Riedmiller \(2013\)Playing atari with deep reinforcement learning\.CoRRabs/1312\.5602\.External Links:[Link](http://arxiv.org/abs/1312.5602),1312\.5602Cited by:[Reinforcement Learning](https://arxiv.org/html/2608.05989#Sx3.SSx1.p2.7)\.
- K\. Ota, T\. Oiki, D\. K\. Jha, T\. Mariyama, and D\. Nikovski \(2020\)Can increasing input dimensionality improve deep reinforcement learning?\.InProceedings of the 37th International Conference on Machine Learning, ICML 2020, 13\-18 July 2020, Virtual Event,Proceedings of Machine Learning Research, Vol\.119,pp\. 7424–7433\.External Links:[Link](http://proceedings.mlr.press/v119/ota20a.html)Cited by:[Dynamics\-Based Representation Learning](https://arxiv.org/html/2608.05989#Sx2.SSx2.p1.1)\.
- A\. Paszke, S\. Gross, F\. Massa, A\. Lerer, J\. Bradbury, G\. Chanan, T\. Killeen, Z\. Lin, N\. Gimelshein, L\. Antiga, A\. Desmaison, A\. Köpf, E\. Z\. Yang, Z\. DeVito, M\. Raison, A\. Tejani, S\. Chilamkurthy, B\. Steiner, L\. Fang, J\. Bai, and S\. Chintala \(2019\)PyTorch: an imperative style, high\-performance deep learning library\.InAdvances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8\-14, 2019, Vancouver, BC, Canada,H\. M\. Wallach, H\. Larochelle, A\. Beygelzimer, F\. d’Alché\-Buc, E\. B\. Fox, and R\. Garnett \(Eds\.\),pp\. 8024–8035\.External Links:[Link](https://proceedings.neurips.cc/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html)Cited by:[Main Results](https://arxiv.org/html/2608.05989#Sx5.SSx2.p3.1)\.
- A\. Scannell, K\. Kujanpää, Y\. Zhao, M\. Nakhaei, A\. Solin, and J\. Pajarinen \(2024\)IQRL \- implicitly quantized representations for sample\-efficient reinforcement learning\.CoRRabs/2406\.02696\.External Links:[Link](https://doi.org/10.48550/arXiv.2406.02696),[Document](https://dx.doi.org/10.48550/ARXIV.2406.02696),2406\.02696Cited by:[Introduction](https://arxiv.org/html/2608.05989#Sx1.p1.1)\.
- J\. Schrittwieser, I\. Antonoglou, T\. Hubert, K\. Simonyan, L\. Sifre, S\. Schmitt, A\. Guez, E\. Lockhart, D\. Hassabis, T\. Graepel, T\. P\. Lillicrap, and D\. Silver \(2020\)Mastering atari, go, chess and shogi by planning with a learned model\.Nat\.588\(7839\),pp\. 604–609\.External Links:[Link](https://doi.org/10.1038/s41586-020-03051-4),[Document](https://dx.doi.org/10.1038/S41586-020-03051-4)Cited by:[Policy Learning](https://arxiv.org/html/2608.05989#Sx4.SSx2.p1.2)\.
- M\. Schwarzer, A\. Anand, R\. Goel, R\. D\. Hjelm, A\. C\. Courville, and P\. Bachman \(2021\)Data\-efficient reinforcement learning with self\-predictive representations\.In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3\-7, 2021,External Links:[Link](https://openreview.net/forum?id=uCQfPZwRaUu)Cited by:[Introduction](https://arxiv.org/html/2608.05989#Sx1.p1.1),[Latent Self\-Prediction\.](https://arxiv.org/html/2608.05989#Sx4.SSx3.SSS0.Px1.p1.6),[Implementation Details\.](https://arxiv.org/html/2608.05989#Sx5.SSx1.SSS0.Px3.p1.4),[Evaluation on Discrete\-Action Tasks](https://arxiv.org/html/2608.05989#Sx5.SSx5.p1.1)\.
- M\. Schwarzer, J\. S\. Obando\-Ceron, A\. C\. Courville, M\. G\. Bellemare, R\. Agarwal, and P\. S\. Castro \(2023\)Bigger, better, faster: human\-level atari with human\-level efficiency\.InInternational Conference on Machine Learning, ICML 2023, 23\-29 July 2023, Honolulu, Hawaii, USA,A\. Krause, E\. Brunskill, K\. Cho, B\. Engelhardt, S\. Sabato, and J\. Scarlett \(Eds\.\),Proceedings of Machine Learning Research, Vol\.202,pp\. 30365–30380\.External Links:[Link](https://proceedings.mlr.press/v202/schwarzer23a.html)Cited by:[Introduction](https://arxiv.org/html/2608.05989#Sx1.p1.1)\.
- D\. Silver, G\. Lever, N\. Heess, T\. Degris, D\. Wierstra, and M\. A\. Riedmiller \(2014\)Deterministic policy gradient algorithms\.InProceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21\-26 June 2014,JMLR Workshop and Conference Proceedings, Vol\.32,pp\. 387–395\.External Links:[Link](http://proceedings.mlr.press/v32/silver14.html)Cited by:[Policy Learning](https://arxiv.org/html/2608.05989#Sx4.SSx2.p1.2)\.
- R\. S\. Sutton, A\. G\. Barto,et al\.\(1998\)Reinforcement learning: an introduction\.Vol\.1,MIT press Cambridge\.Cited by:[Introduction](https://arxiv.org/html/2608.05989#Sx1.p4.1),[Reinforcement Learning](https://arxiv.org/html/2608.05989#Sx3.SSx1.p1.14)\.
- Y\. Tassa, Y\. Doron, A\. Muldal, T\. Erez, Y\. Li, D\. de Las Casas, D\. Budden, A\. Abdolmaleki, J\. Merel, A\. Lefrancq, T\. P\. Lillicrap, and M\. A\. Riedmiller \(2018\)DeepMind control suite\.CoRRabs/1801\.00690\.External Links:[Link](http://arxiv.org/abs/1801.00690),1801\.00690Cited by:[Introduction](https://arxiv.org/html/2608.05989#Sx1.p5.1),[Environments\.](https://arxiv.org/html/2608.05989#Sx5.SSx1.SSS0.Px1.p1.1)\.
- E\. Todorov, T\. Erez, and Y\. Tassa \(2012\)MuJoCo: A physics engine for model\-based control\.In2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2012, Vilamoura, Algarve, Portugal, October 7\-12, 2012,pp\. 5026–5033\.External Links:[Link](https://doi.org/10.1109/IROS.2012.6386109),[Document](https://dx.doi.org/10.1109/IROS.2012.6386109)Cited by:[Environments\.](https://arxiv.org/html/2608.05989#Sx5.SSx1.SSS0.Px1.p1.1)\.
- Z\. Wang, T\. Schaul, M\. Hessel, H\. van Hasselt, M\. Lanctot, and N\. de Freitas \(2016\)Dueling network architectures for deep reinforcement learning\.InProceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19\-24, 2016,M\. Balcan and K\. Q\. Weinberger \(Eds\.\),JMLR Workshop and Conference Proceedings, Vol\.48,pp\. 1995–2003\.External Links:[Link](http://proceedings.mlr.press/v48/wangf16.html)Cited by:[Evaluation Protocol\.](https://arxiv.org/html/2608.05989#Sx5.SSx1.SSS0.Px4.p2.1)\.
- D\. Yarats, R\. Fergus, A\. Lazaric, and L\. Pinto \(2022\)Mastering visual continuous control: improved data\-augmented reinforcement learning\.InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25\-29, 2022,External Links:[Link](https://openreview.net/forum?id=%5C_SJ-%5C_yyes8)Cited by:[Introduction](https://arxiv.org/html/2608.05989#Sx1.p1.1),[Representation Learning for Data\-Efficient RL](https://arxiv.org/html/2608.05989#Sx2.SSx1.p1.1),[Reinforcement Learning](https://arxiv.org/html/2608.05989#Sx3.SSx1.p2.7),[Value Learning](https://arxiv.org/html/2608.05989#Sx4.SSx1.p1.10),[Baselines\.](https://arxiv.org/html/2608.05989#Sx5.SSx1.SSS0.Px2.p1.1),[Implementation Details\.](https://arxiv.org/html/2608.05989#Sx5.SSx1.SSS0.Px3.p1.4),[Evaluation Protocol\.](https://arxiv.org/html/2608.05989#Sx5.SSx1.SSS0.Px4.p2.1)\.
- D\. Yarats, I\. Kostrikov, and R\. Fergus \(2021\)Image augmentation is all you need: regularizing deep reinforcement learning from pixels\.In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3\-7, 2021,External Links:[Link](https://openreview.net/forum?id=GY6-6sTvGaf)Cited by:[Introduction](https://arxiv.org/html/2608.05989#Sx1.p1.1),[Evaluation Protocol\.](https://arxiv.org/html/2608.05989#Sx5.SSx1.SSS0.Px4.p2.1)\.
- L\. Zhang, Z\. Wang, X\. Li, and Y\. Li \(2025\)Revisiting bisimulation metric for robust representations in reinforcement learning\.CoRRabs/2507\.18519\.External Links:[Link](https://doi.org/10.48550/arXiv.2507.18519),[Document](https://dx.doi.org/10.48550/ARXIV.2507.18519),2507\.18519Cited by:[Representation Learning for Data\-Efficient RL](https://arxiv.org/html/2608.05989#Sx2.SSx1.p1.1)\.
- R\. Zheng, X\. Wang, Y\. Sun, S\. Ma, J\. Zhao, H\. Xu, H\. D\. III, and F\. Huang \(2023\)TACO: temporal latent action\-driven contrastive loss for visual reinforcement learning\.InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 \- 16, 2023,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2023/hash/96d00450ed65531ffe2996daed487536-Abstract-Conference.html)Cited by:[Introduction](https://arxiv.org/html/2608.05989#Sx1.p1.1),[Evaluation Protocol\.](https://arxiv.org/html/2608.05989#Sx5.SSx1.SSS0.Px4.p2.1)\.