Integrating Novelty and Surprise for Experience Prioritization and Exploration in Image-Based Reinforcement Learning

# Integrating Novelty and Surprise for Experience Prioritization and Exploration in Image-Based Reinforcement Learning
Source: [https://arxiv.org/html/2608.17373](https://arxiv.org/html/2608.17373)
Henry WilliamsBruce A\. MacDonaldThanks:Robot Learning Team and CARES Robotics Lab, Department of Electrical, Computer, and Software Engineering, University of Auckland, Auckland, New Zealand\. Corresponding author: hyam650@aucklanduni\.ac\.nz\.

###### Abstract

Sample efficiency is a central challenge in reinforcement learning \(RL\), particularly in image\-based domains where agents must learn from high\-dimensional visual inputs\. Traditional sampling often relies on random or suboptimal experience selection, leading to redundant updates and slow learning\. Improving efficiency requires mechanisms that prioritize informative experiences while also encouraging effective exploration\. Prioritized Experience Replay \(PER\) addresses part of this challenge by reusing high\-value transitions, while intrinsic rewards promote the exploration of novel or uncertain states\. However, their integration has not been extensively studied\. This paper introduces Novelty and Surprise Prioritized Experience Replay \(NSPER\), which uses novelty to capture underrepresented states and surprise to expose gaps in the agent’s understanding of the environment\. We further extend this with NSPER\+R, integrating these signals as intrinsic rewards to jointly improve replay quality and exploration\. Experiments on DeepMind Control Suite tasks show that NSPER and NSPER\+R improve training efficiency and convergence speed compared to existing methods in image\-based RL\.

\{IEEEkeywords\}

Reinforcement learning, prioritized experience replay, sample efficiency, novelty, surprise, intrinsic rewards, continuous control, image\-based control\.

![Refer to caption](https://arxiv.org/html/2608.17373v1/new-pics/main/nsper-.png)Figure 1:Schematic overview of Novelty and Surprise Prioritized Experience Replay \(NSPER\)\.## 1INTRODUCTION

Reinforcement learning \(RL\) has shown great promise in image\-based applications, allowing agents to learn directly from visual input\[[44](https://arxiv.org/html/2608.17373#bib.bib6),[17](https://arxiv.org/html/2608.17373#bib.bib7)\]\. This paradigm has been widely applied in domains such as gaming\[[26](https://arxiv.org/html/2608.17373#bib.bib17)\], robotics\[[13](https://arxiv.org/html/2608.17373#bib.bib9)\], and autonomous driving\[[33](https://arxiv.org/html/2608.17373#bib.bib10)\], where vision serves as the primary source of environmental feedback\. However, high\-dimensional image data introduces redundancy, high memory demands, and substantial computational overhead, making policy learning more complex and resource\-intensive\[[9](https://arxiv.org/html/2608.17373#bib.bib8)\]\. Since RL inherently relies on large volumes of interaction data, improving sample efficiency is critical\[[11](https://arxiv.org/html/2608.17373#bib.bib3)\]\. A promising direction is to prioritize informative experiences during training and to guide exploration toward those experiences that provide the most learning value\. This prioritization can be guided by intrinsic learning signals that highlight which transitions are the most useful for learning\[[5](https://arxiv.org/html/2608.17373#bib.bib31)\]\.

Inspired by human cognition, where intrinsic signals foster curiosity\[[4](https://arxiv.org/html/2608.17373#bib.bib25),[10](https://arxiv.org/html/2608.17373#bib.bib24)\]and guide memory systems to prioritize novel or salient experiences\[[35](https://arxiv.org/html/2608.17373#bib.bib26),[7](https://arxiv.org/html/2608.17373#bib.bib27)\], RL agents can adopt similar mechanisms to enhance exploration and accelerate learning\. Such signals provide a foundation for both directing exploratory behavior and selecting the most informative experiences during training\[[30](https://arxiv.org/html/2608.17373#bib.bib32),[16](https://arxiv.org/html/2608.17373#bib.bib33)\], thereby improving sample efficiency in two complementary ways\. First, they can be formulated as intrinsic rewards, encouraging agents to acquire diverse and informative experiences that accelerate policy refinement\[[28](https://arxiv.org/html/2608.17373#bib.bib29)\]\. Second, they can be used as prioritization scores within experience replay, enabling agents to focus on transitions with the highest learning potential\.

Among intrinsic signals, novelty and surprise have proven especially effective as intrinsic rewards\[[6](https://arxiv.org/html/2608.17373#bib.bib30)\]\. Novelty reflects the unfamiliarity of a state, driving exploration toward underrepresented or unexplored regions\[[18](https://arxiv.org/html/2608.17373#bib.bib35),[20](https://arxiv.org/html/2608.17373#bib.bib36)\]\. Surprise, on the other hand, arises when outcomes deviate substantially from predictions, exposing gaps in the agent’s environmental model and its limited ability to generalize from past experiences\[[1](https://arxiv.org/html/2608.17373#bib.bib34)\]\. Combining these two signals enables agents to identify regions that require further learning and adjust their exploration strategies, resulting in more efficient and targeted exploration\[[22](https://arxiv.org/html/2608.17373#bib.bib28),[39](https://arxiv.org/html/2608.17373#bib.bib1)\]\.

Prioritized Experience Replay \(PER\) is a well\-established approach in RL that exploits such intrinsic signals within the replay buffer to improve sample efficiency\[[32](https://arxiv.org/html/2608.17373#bib.bib19)\]\. In its original form, which we refer to as TD\-PER, transitions are prioritized based on the temporal difference \(TD\) error, which measures the discrepancy between predicted and actual outcomes following an agent’s action\. While effective in discrete action spaces, TD\-PER struggles in continuous control environments\[[29](https://arxiv.org/html/2608.17373#bib.bib12),[27](https://arxiv.org/html/2608.17373#bib.bib21)\]\. To overcome these limitations, several extensions have been proposed, including augmenting TD error with auxiliary signals or using biologically inspired mechanisms for prioritization\[[31](https://arxiv.org/html/2608.17373#bib.bib22),[27](https://arxiv.org/html/2608.17373#bib.bib21),[40](https://arxiv.org/html/2608.17373#bib.bib16),[37](https://arxiv.org/html/2608.17373#bib.bib23)\]\. Despite these advances, integrating intrinsic rewards with PER in image\-based RL remains limited\. For instance, the Contrastive Curiosity Learning Framework \(CCLF\)\[[37](https://arxiv.org/html/2608.17373#bib.bib23)\]uses curiosity to prioritize surprising augmented inputs\. However, the broader use of intrinsic rewards as scoring signals for experience prioritization remains largely underexplored\.

To address this gap, we propose NSPER, a novel prioritization method that uses two distinct intrinsic signals, novelty and surprise, to guide experience selection within PER\. By combining these signals to guide sampling, NSPER identifies diverse and informative transitions with high learning potential, improving the quality of replayed experiences\. We further introduce NSPER\+R, an extension of NSPER that integrates novelty and surprise not only for prioritization but also as intrinsic rewards\. This dual mechanism improves experience selection and exploratory behavior, leading to faster convergence and more generalizable policies\. To evaluate our approach, we integrate it into PixelTD3, an image\-based variant of the TD3 algorithm\[[14](https://arxiv.org/html/2608.17373#bib.bib20)\], and assess its performance on challenging tasks from the DeepMind Control Suite\[[38](https://arxiv.org/html/2608.17373#bib.bib4)\], demonstrating promising improvements in complex, high\-dimensional visual environments\. The model architecture is illustrated in Fig\.[1](https://arxiv.org/html/2608.17373#S0.F1)\. The implementation is publicly available at[https://github\.com/UoA\-CARES/NSPER](https://github.com/UoA-CARES/NSPER)\.

The main contributions of this paper are summarized as follows:

1. 1\.We introduceNSPER, a novel approach that uses intrinsic rewards as prioritization signals within PER to improve sample efficiency in image\-based RL\.
2. 2\.We extend this idea inNSPER\+R, which uses novelty and surprise both for prioritization and as intrinsic rewards, further enhancing exploration and policy learning\.
3. 3\.We provide a comprehensive evaluation of different prioritization strategies and their integration with intrinsic rewards, analyzing their impact on training efficiency and learning performance\.
4. 4\.We conduct an ablation study to isolate the individual contributions of novelty and surprise, as well as their combined effect when used with intrinsic rewards\.

## 2RELATED WORKS

Experience prioritization and intrinsic motivation are key strategies for improving sample efficiency and exploration in RL\. Prior work has explored various mechanisms for identifying informative experiences and guiding agents toward novel or surprising transitions, providing a foundation for methods that combine these ideas to enhance learning in complex environments\.

### 2\.1Experience Prioritization Methods

PER methods enhance sample efficiency by assigning higher sampling probability to transitions with greater learning potential\[[42](https://arxiv.org/html/2608.17373#bib.bib18)\]\. TD\-PER\[[32](https://arxiv.org/html/2608.17373#bib.bib19)\], a common variant, ranks transitions based on TD error, under the assumption that transitions with larger TD errors are less familiar and therefore more informative\. However, in continuous high\-dimensional spaces, small perturbations in actions or observations can lead to large fluctuations in value estimates, even for well\-learned transitions, making TD error a noisy and unstable prioritization signal\[[29](https://arxiv.org/html/2608.17373#bib.bib12)\]\. This sensitivity makes TD error a noisy and unreliable prioritization signal, motivating alternative strategies that offer more stable and informative learning signals in continuous control tasks\[[8](https://arxiv.org/html/2608.17373#bib.bib13),[25](https://arxiv.org/html/2608.17373#bib.bib14)\]\.

To address these limitations, LA3P\[[31](https://arxiv.org/html/2608.17373#bib.bib22)\]decouples learning signals by assigning low TD\-error transitions to the actor to stabilize policy updates and high TD\-error transitions to the critic to improve value estimation\. Another method\[[42](https://arxiv.org/html/2608.17373#bib.bib18)\]prioritizes recent transitions with low TD error, thereby focusing on up\-to\-date experiences, but this risks catastrophic forgetting by discarding older yet valuable data\. Model\-Augmented PER \(MaPER\)\[[27](https://arxiv.org/html/2608.17373#bib.bib21)\]incorporates auxiliary signals derived from learned environmental dynamics to complement the TD error, thereby improving the identification of important experiences\. Despite these enhancements, all of these methods primarily rely on the TD error as the primary prioritization criterion\.

In contrast, Reward Prediction Error \(RPE\) prioritization\[[40](https://arxiv.org/html/2608.17373#bib.bib16)\]introduces an alternative signal inspired by biological learning\. By measuring the discrepancy between expected and actual rewards, RPE provides informative feedback on its prediction accuracy, allowing it to adjust its policies and enhance decision\-making over time\[[36](https://arxiv.org/html/2608.17373#bib.bib40)\]\. Similarly, our method proposes a novel prioritization strategy that uses alternative intrinsic signals, fostering exploration and emphasizing unfamiliar transitions\.

### 2\.2Intrinsic Motivation Strategies

Intrinsic motivation has been widely explored in RL to encourage agents to explore and learn beyond extrinsic rewards\[[28](https://arxiv.org/html/2608.17373#bib.bib29)\]\. This approach encourages agents to seek out novel states and behaviors within the environment\[[3](https://arxiv.org/html/2608.17373#bib.bib37),[37](https://arxiv.org/html/2608.17373#bib.bib23)\]\. Several intrinsic motivation mechanisms have been proposed, including state novelty\[[6](https://arxiv.org/html/2608.17373#bib.bib30)\], state prediction errors\[[6](https://arxiv.org/html/2608.17373#bib.bib30)\], uncertainty about outcomes\[[24](https://arxiv.org/html/2608.17373#bib.bib38)\], and environment dynamics\[[34](https://arxiv.org/html/2608.17373#bib.bib39)\]\. Among these, novelty and surprise enable more focused and efficient learning by directly targeting meaningful opportunities, guiding exploration toward unfamiliar states and significant prediction errors, and outperforming methods that rely solely on uncertainty or environmental dynamics\[[39](https://arxiv.org/html/2608.17373#bib.bib1)\]\.

Novelty encourages visiting rarely encountered states, promoting a broader understanding of the environment\[[18](https://arxiv.org/html/2608.17373#bib.bib35)\]\. Surprise is measured through prediction errors in environment dynamics, such as next\-state errors and state transition divergences\[[30](https://arxiv.org/html/2608.17373#bib.bib32),[1](https://arxiv.org/html/2608.17373#bib.bib34)\]\. Combining novelty and surprise captures complementary aspects of agent\-environment interaction, improving exploration and representation learning\[[18](https://arxiv.org/html/2608.17373#bib.bib35),[20](https://arxiv.org/html/2608.17373#bib.bib36)\]and allowing better balance between exploration and exploitation in complex tasks\[[22](https://arxiv.org/html/2608.17373#bib.bib28)\]\.

Despite these advances, integrating intrinsic motivation signals into memory prioritization mechanisms remains an ongoing challenge\. Several recent studies have focused on combining curiosity\-based metrics with experience replay to further refine prioritization strategies\.

### 2\.3Integrating Intrinsic Signals in Experience Replay

Recent studies have explored the incorporation of intrinsic motivation into memory prioritization\. Curiosity\-driven experience prioritization \(CDP\)\[[43](https://arxiv.org/html/2608.17373#bib.bib42)\]prioritizes trajectories that lead to rare goal states, ensuring the agent focuses on underexplored areas\. However, CDP is based on the replay of hindsight experiences \(HER\)\[[2](https://arxiv.org/html/2608.17373#bib.bib11)\], which limits its applicability to goal\-oriented tasks\. Augmented Curiosity\-Driven Experience Replay \(ACDER\)\[[23](https://arxiv.org/html/2608.17373#bib.bib41)\]also integrates curiosity signals into HER but remains restricted to goal\-conditioned exploration\.

A more relevant approach, CCLF\[[37](https://arxiv.org/html/2608.17373#bib.bib23)\], introduces a self\-supervised prioritization mechanism in image\-based RL\. CCLF selects surprising augmented inputs to enhance sample efficiency, but it focuses primarily on representation learning rather than on optimizing experience selection for policy improvement\.

Our method extends these insights by integrating novelty and surprise into experience prioritization, ensuring that selected transitions support both exploration and policy optimization\. By combining intrinsic rewards with a structured prioritization mechanism, it overcomes the limitations of PER and leverages intrinsic motivation to enable more effective learning in challenging environments

## 3METHODOLOGY

This section outlines the components underlying our approach, beginning with a review of PER, PixelTD3, and the computation of novelty and surprise signals\. We then introduce our framework, NSPER, and its extension, NSPER\+R\.

### 3\.1PER

PER improves sample efficiency by replaying transitions with higher learning potential\. Transitions in the bufferBBare stored as:

Bi=\(si,ai,ri,si\+1\),B\_\{i\}=\(s\_\{i\},a\_\{i\},r\_\{i\},s\_\{i\+1\}\),\(1\)whereiiindexes the time step\. The priority of each transition is defined as:

σi=\|δi\|\+ϵ,\\sigma\_\{i\}=\|\\delta\_\{i\}\|\+\\epsilon,\(2\)In the standard variant, TD\-PER,δi\\delta\_\{i\}is the TD error, andϵ\>0\\epsilon\>0ensures non\-zero priority\.

Transitions are sampled according to:

pi=σiα∑kσkαp\_\{i\}=\\frac\{\\sigma\_\{i\}^\{\\alpha\}\}\{\\sum\_\{k\}\\sigma\_\{k\}^\{\\alpha\}\}\(3\)
wi=\(1\|B\|​pi\)βw\_\{i\}=\\Big\(\\tfrac\{1\}\{\|B\|\\,p\_\{i\}\}\\Big\)^\{\\beta\}\(4\)
wherepip\_\{i\}favors transitions with higher priorityσi\\sigma\_\{i\}, andwiw\_\{i\}corrects for the bias introduced by non\-uniform sampling\. Here,α\\alphacontrols prioritization sharpness, andβ\\betacompensates for sampling bias\. While TD\-PER definesσi\\sigma\_\{i\}using TD error, PER is flexible and allows alternative prioritization signals by redefiningσi\\sigma\_\{i\}\.

### 3\.2PixelTD3

PixelTD3 is an image\-based extension of TD3\[[14](https://arxiv.org/html/2608.17373#bib.bib20)\], inspired by SACAE\[[41](https://arxiv.org/html/2608.17373#bib.bib5)\]and extended in NaSATD3\[[39](https://arxiv.org/html/2608.17373#bib.bib1)\]\. PixelTD3 employs a convolutional autoencoder that maps raw observationssts\_\{t\}into a compact latent representationzz\. This latent vector is shared between the actor and critic networks, ensuring that policy learning is grounded in a consistent feature space\. The encoder–decoder is jointly trained with a reconstruction loss that preserves the structural content of the input images while maintaining task\-relevant features\.

Unlike SACAE, which trains separate encoders for actor and critic, PixelTD3 uses a single shared autoencoder that is updated end\-to\-end at every step, including both convolutional and linear layers\. This provides a stable and compact representation for policy learning\. We adopt this setup as the backbone of NSPER \(see Fig\.[1](https://arxiv.org/html/2608.17373#S0.F1)\)\. Complete architectural specifications of the encoder, decoder, actor, critics, and predictive ensemble are reported in Table[3](https://arxiv.org/html/2608.17373#S3.T3)\.

### 3\.3Novelty And Surprise Computation

Noveltymeasures how different an observation is from what the agent has already encoded\. It is estimated by reconstructing the inputsts\_\{t\}through the autoencoder and comparing it to the original observation\. Lets^t=D​e​c​\(E​n​c​\(st\)\)\\hat\{s\}\_\{t\}=Dec\(Enc\(s\_\{t\}\)\)denote the reconstruction ofsts\_\{t\}\. The novelty score is then defined as the complement of the Structural Similarity Index \(SSIM\) betweensts\_\{t\}ands^t\\hat\{s\}\_\{t\}:

Novelty​\(st\)=1−SSIM​\(st,s^t\)\.\\text\{Novelty\}\(s\_\{t\}\)=1\-\\text\{SSIM\}\(s\_\{t\},\\hat\{s\}\_\{t\}\)\.\(5\)A higher value indicates that the observation is less well\-reconstructed, and therefore more novel to the agent\. Fig\.[1](https://arxiv.org/html/2608.17373#S0.F1)illustrates how novelty is obtained from the autoencoder reconstruction error of the input image\.

Surprisemeasures the deviation between the agent’s predicted and observed dynamics\. It is computed in latent space using an ensemble of dynamics modelsPρP\_\{\\rho\}, which predict the next latent statez^t\+1\\hat\{z\}\_\{t\+1\}given the current latent representationztz\_\{t\}and actionata\_\{t\}:

z^t\+1=Pρ​\(zt,at\)\.\\hat\{z\}\_\{t\+1\}=P\_\{\\rho\}\(z\_\{t\},a\_\{t\}\)\.\(6\)Surprise is defined as the mean squared error between the predicted and observed latent states:

Surprise​\(st,at\)=MSE​\(zt\+1,z^t\+1\)\.\\text\{Surprise\}\(s\_\{t\},a\_\{t\}\)=\\text\{MSE\}\(z\_\{t\+1\},\\hat\{z\}\_\{t\+1\}\)\.\(7\)As illustrated in Fig\.[1](https://arxiv.org/html/2608.17373#S0.F1), this signal corresponds to the mismatch between the latent representation of the observed next state and the ensemble prediction\. The ensemble consists ofMMmodels trained with mean squared error loss, and averaging their predictions reduces variance and improves stability under non\-stationary encoder updates\.

Novelty and Surprise Combination: Combining novelty and surprise reward signals encourages more targeted exploration of informative regions within the environment\. This integration forms a new intrinsic signal, referred to as theNovelty\-Surprise Signal \(NSS\)\. TheNSS​\(s,a\)\\text\{NSS\}\(s,a\), derived from \([5](https://arxiv.org/html/2608.17373#S3.E5)\) and \([7](https://arxiv.org/html/2608.17373#S3.E7)\), captures the joint contribution of novelty and surprise and is formulated as:

NSS​\(st,at\)=Novelty​\(st\)\+Surprise​\(st,at\)\\text\{NSS\}\(s\_\{t\},a\_\{t\}\)=\\text\{Novelty\}\(s\_\{t\}\)\+\\text\{Surprise\}\(s\_\{t\},a\_\{t\}\)\(8\)

### 3\.4NSPER: Novelty and Surprise Prioritization

We propose a prioritization strategy that leverages novelty and surprise as intrinsic signals to guide experience replay\. Rather than relying on the TD error, as in standard PER, NSPER replacesδi\\delta\_\{i\}in \([2](https://arxiv.org/html/2608.17373#S3.E2)\) with the Novelty–Surprise SignalNSS​\(s,a\)\\text\{NSS\}\(s,a\)from \([8](https://arxiv.org/html/2608.17373#S3.E8)\)\. The priority assigned to each transition is therefore defined as:

σt=\|NSS​\(st,at\)\|\+ϵ,\\sigma\_\{t\}=\\left\|\\text\{NSS\}\(s\_\{t\},a\_\{t\}\)\\right\|\+\\epsilon,\(9\)whereϵ\>0\\epsilon\>0ensures all transitions remain eligible for replay\.

These priorities are then used in the sampling equations \([3](https://arxiv.org/html/2608.17373#S3.E3)\) and \([4](https://arxiv.org/html/2608.17373#S3.E4)\) to compute transition probabilities and importance weights\. By emphasizing transitions with higher novelty and surprise, NSPER favors diverse and informative experiences that better support critic training\. The parameterα\\alpharegulates the strength of this prioritization, ensuring that informative transitions are emphasized without over\-concentrating sampling on a few extreme cases\. This balance allows NSPER to prioritize informative experiences while maintaining diversity in the sampled transitions\.

### 3\.5Enhancing Exploration with Intrinsic Rewards

Intrinsic rewards are agent\-generated signals that promote exploration, especially when extrinsic rewards are sparse or delayed\[[5](https://arxiv.org/html/2608.17373#bib.bib31)\]\. In our approach, we define the intrinsic reward using novelty and surprise \([8](https://arxiv.org/html/2608.17373#S3.E8)\) as:

ri​n​t​\(st,at\)=N​S​S​\(st,at\)r\_\{int\}\(s\_\{t\},a\_\{t\}\)=NSS\(s\_\{t\},a\_\{t\}\)\(10\)As depicted in Fig\.[1](https://arxiv.org/html/2608.17373#S0.F1), the NSS is used both to prioritize experiences during replay and to define the intrinsic reward signal\. The total reward at each step combines extrinsic and intrinsic components:

rt=re​x​t\+ri​n​tr\_\{t\}=r\_\{ext\}\+r\_\{int\}\(11\)
We refer to the full version of our framework as NSPER\+R, which integrates intrinsic rewards with prioritized experience replay\. This variant improves learning by prioritizing transitions that contribute to both task success and deeper environmental understanding\. Algorithm[1](https://arxiv.org/html/2608.17373#alg1)summarises the full procedure for NSPER and its intrinsic\-reward variant NSPER\+R\.

Algorithm 1NSPER and NSPER\+R with NSS \(Novelty–Surprise Signal\)1:Inputs:PER exponents

α,β\\alpha,\\beta, batch size

mm, small

ε\>0\\varepsilon\>0
2:Initialize:actor

Θ\\Theta, critic

θ\\theta, encoder–decoder

\(Enc,Dec\)\(\\mathrm\{Enc\},\\mathrm\{Dec\}\), replay buffer

BB, priorities

\{σi\}\\\{\\sigma\_\{i\}\\\}
3:foreach environment step

ttdo

4:Observe state

sts\_\{t\}; select action

ata\_\{t\}\(with exploration noise\); receive

rextr\_\{\\text\{ext\}\}and

st\+1s\_\{t\+1\}
5:Novelty:

Novelty⁡\(st\)←1−SSIM⁡\(st,Dec⁡\(Enc⁡\(st\)\)\)\\displaystyle\\mathrm\{Novelty\}\(s\_\{t\}\)\\leftarrow 1\-\\mathrm\{SSIM\}\\\!\\big\(s\_\{t\},\\,\\mathrm\{Dec\}\(\\mathrm\{Enc\}\(s\_\{t\}\)\)\\big\)
6:Surprise:obtain latents

\(zt\+1,zt\+1′\)\(z\_\{t\+1\},z^\{\\prime\}\_\{t\+1\}\)and set

Surprise⁡\(st,at\)←MSE⁡\(zt\+1′,zt\+1\)\\displaystyle\\mathrm\{Surprise\}\(s\_\{t\},a\_\{t\}\)\\leftarrow\\mathrm\{MSE\}\(z^\{\\prime\}\_\{t\+1\},z\_\{t\+1\}\)
7:NSS:

NSS⁡\(st,at\)←Novelty⁡\(st\)\+Surprise⁡\(st,at\)\\displaystyle\\mathrm\{NSS\}\(s\_\{t\},a\_\{t\}\)\\leftarrow\\mathrm\{Novelty\}\(s\_\{t\}\)\+\\mathrm\{Surprise\}\(s\_\{t\},a\_\{t\}\)
8:Intrinsic reward:

rint←NSS⁡\(st,at\)r\_\{\\text\{int\}\}\\leftarrow\\mathrm\{NSS\}\(s\_\{t\},a\_\{t\}\)
9:Total reward:

rt←\{rext\+rint,NSPER\+Rrext,NSPERr\_\{t\}\\leftarrow\\begin\{cases\}r\_\{\\text\{ext\}\}\+r\_\{\\text\{int\}\},&\\text\{NSPER\+R\}\\\\ r\_\{\\text\{ext\}\},&\\text\{NSPER\}\\end\{cases\}
10:Push

\(st,at,rt,st\+1,NSS⁡\(st,at\)\)\(s\_\{t\},a\_\{t\},r\_\{t\},s\_\{t\+1\},\\mathrm\{NSS\}\(s\_\{t\},a\_\{t\}\)\)into

BB
11:Set priority for new item:

σnew←\|NSS⁡\(st,at\)\|\+ε\\sigma\_\{\\text\{new\}\}\\leftarrow\|\\mathrm\{NSS\}\(s\_\{t\},a\_\{t\}\)\|\+\\varepsilon
12:foreach update stepdo

13:Sampleindices

IIof size

mmwith PER:

p⁡\(i\)=σiα∑kσkαp\(i\)=\\frac\{\\sigma\_\{i\}^\{\\alpha\}\}\{\\sum\_\{k\}\\sigma\_\{k\}^\{\\alpha\}\}
14:IS weights:

w⁡\(i\)←\(1\|B\|⋅p⁡\(i\)\)βw\(i\)\\leftarrow\\Big\(\\frac\{1\}\{\|B\|\\cdot p\(i\)\}\\Big\)^\{\\beta\}\(optionally normalize by

maxj∈I⁡w⁡\(j\)\\max\_\{j\\in I\}w\(j\)\)

15:Update critic/actor:compute losses on batch, weight by

w⁡\(i\)w\(i\), update

θ,Θ\\theta,\\Theta
16:Refresh priorities:for each

i∈Ii\\in I, set

σi←\|NSS⁡\(si,ai\)\|\+ε\\sigma\_\{i\}\\leftarrow\|\\mathrm\{NSS\}\(s\_\{i\},a\_\{i\}\)\|\+\\varepsilon
17:endfor

18:endfor

![Refer to caption](https://arxiv.org/html/2608.17373v1/new-pics/dmcs-images/cartpole-balance.png)\(a\)Cartpole\-Balance
![Refer to caption](https://arxiv.org/html/2608.17373v1/new-pics/dmcs-images/finger.png)\(b\)Finger\-Spin
![Refer to caption](https://arxiv.org/html/2608.17373v1/new-pics/dmcs-images/ball-in-cup.png)\(c\)Ball\-in\-Cup
![Refer to caption](https://arxiv.org/html/2608.17373v1/new-pics/dmcs-images/walker.png)\(d\)Walker\-Walk
![Refer to caption](https://arxiv.org/html/2608.17373v1/new-pics/dmcs-images/cheetah.png)\(e\)Cheetah\-Run

Figure 2:Five continuous control tasks from the DeepMind Control Suite used in our evaluation\.Table 1:Summary of DeepMind Control Suite tasks used in our experiments\.Table 2:Experimental setup for NSPER across DeepMind Control Suite environments\.Table 3:Network architectures used in NSPER and NSPER\+R\.

## 4EXPERIMENTS

![Refer to caption](https://arxiv.org/html/2608.17373v1/new-pics/main/Cartpole-Balance.png)\(a\)
![Refer to caption](https://arxiv.org/html/2608.17373v1/new-pics/main/Finger-Spin.png)\(b\)
![Refer to caption](https://arxiv.org/html/2608.17373v1/new-pics/main/Ball_In_Cup.png)\(c\)
![Refer to caption](https://arxiv.org/html/2608.17373v1/new-pics/main/Walker-Walk.png)\(d\)
![Refer to caption](https://arxiv.org/html/2608.17373v1/new-pics/main/Cheetah-Run.png)\(e\)
NSPER\+RNSPERTD\-PER\+RTD\-PERRPE\-PER\+RRPE\-PERUniform\+RUniformCCLF

Figure 3:Learning curves for continuous control tasks using PixelTD3\. Solid lines represent the algorithm’s performance with intrinsic rewards \(\+R\), while dashed lines indicate performance without intrinsic rewards\. Shaded regions denote the 95% confidence interval across 5 runs\.This section describes the experimental setup, simulation tasks, algorithmic settings, and baseline methods used in our evaluation, and then presents an analysis of the learning performance of NSPER, NSPER\+R, and competing approaches\.

### 4\.1Experimental Setup

We evaluate our algorithms and baselines on five image\-based continuous control tasks from the DeepMind Control Suite \(DMC\)\[[38](https://arxiv.org/html/2608.17373#bib.bib4)\]:Cartpole\-Balance,Finger\-Spin,Ball\-in\-Cup,Walker\-Walk, andCheetah\-Run\. These environments constitute the standard benchmark suite for visual reinforcement learning and have been widely used in prior pixel\-based studies\[[19](https://arxiv.org/html/2608.17373#bib.bib43),[21](https://arxiv.org/html/2608.17373#bib.bib44),[39](https://arxiv.org/html/2608.17373#bib.bib1)\]\. This set provides representative coverage of balance, manipulation, and locomotion tasks and offers sufficient diversity for assessing prioritization strategies in image\-based continuous control\. Visual examples of the environments are given in Fig\.[2](https://arxiv.org/html/2608.17373#S3.F2), and their key properties are summarised in Table[1](https://arxiv.org/html/2608.17373#S3.T1)\.

All algorithms are trained under a unified experimental configuration following established practice in continuous\-control evaluation\. Training settings, optimization parameters, evaluation schedules, and replay specifications are listed in Table[2](https://arxiv.org/html/2608.17373#S3.T2)\. Using identical configurations across all tasks ensures that performance differences arise from the prioritization mechanism rather than from hyperparameter choices or implementation bias\. NSPER and NSPER\+R employ a shared latent\-space architecture for representation learning, prediction, and action\-value estimation\. The components of this architecture are outlined in Table[3](https://arxiv.org/html/2608.17373#S3.T3)\.

### 4\.2Experimental Approach

To evaluate the effectiveness of our approach, we consider two proposed variants, NSPER and NSPER\+R:

NSPER:Uses novelty and surprise solely for replay prioritization, isolating their effect on experience selection\.

NSPER\+R:Extends NSPER by also incorporating novelty and surprise as intrinsic rewards, combined with extrinsic rewards \([11](https://arxiv.org/html/2608.17373#S3.E11)\), allowing us to assess their joint impact on prioritization and exploration\.

Baseline Algorithms:For comparison, we evaluate a range of established methods within the PixelTD3 framework, which differ in their replay strategies and use of intrinsic rewards\. All implementations share the same codebase for fairness\. Although CCLF was originally proposed with a pixel\-based SAC backbone, we adapt it to PixelTD3 for consistency\. The baselines include:

1. 1\.Uniform: Uniformly samples transitions from the buffer without PER or intrinsic rewards\[[14](https://arxiv.org/html/2608.17373#bib.bib20),[12](https://arxiv.org/html/2608.17373#bib.bib2)\]\.
2. 2\.Uniform\+R \(NaSATD3\): Uniform sampling with intrinsic rewards, equivalent to PixelTD3\+R\[[39](https://arxiv.org/html/2608.17373#bib.bib1)\]\.
3. 3\.TD\-PER: PER based on TD error\[[32](https://arxiv.org/html/2608.17373#bib.bib19)\], without intrinsic rewards\.
4. 4\.TD\-PER\+R: TD\-PER augmented with intrinsic rewards\.
5. 5\.RPE\-PER: PER based on reward prediction error \(RPE\)\[[40](https://arxiv.org/html/2608.17373#bib.bib16)\], without intrinsic rewards\.
6. 6\.RPE\-PER\+R: RPE\-PER extended with intrinsic rewards\.
7. 7\.CCLF: A contrastive learning\-based prioritization method with intrinsic rewards to improve exploration and representation learning\[[37](https://arxiv.org/html/2608.17373#bib.bib23)\]\.

This setup enables a comprehensive evaluation of NSPER and NSPER\+R against diverse baselines that vary in their use of prioritization strategies and intrinsic rewards, providing a robust comparison in image\-based RL\.

### 4\.3Learning Efficiency Analysis

The learning curves in Fig\.[3](https://arxiv.org/html/2608.17373#S4.F3)illustrate the learning efficiency of NSPER and baseline methods across five challenging DeepMind continuous control environments\. These curves depict the average evaluation reward over training steps, providing insights into convergence speed and performance differences among NSPER, NSPER\+R, and the baseline algorithms\.

Dashed\-line plots represent PixelTD3 with different replay buffer sampling methods but without intrinsic rewards, while solid\-line plots of the same color correspond to the same sampling methods with intrinsic rewards\. These separate plots directly compare how sampling strategies influence performance with and without intrinsic rewards\.

It is crucial to note that Baseline methods did not always replicate the performance reported in prior work\. Such deviations are not uncommon in deep reinforcement learning and can arise from factors such as environmental stochasticity, initialization randomness, and subtle implementation or runtime differences\. As emphasized by Henderson et al\.\[[15](https://arxiv.org/html/2608.17373#bib.bib15)\], ensuring fair comparison is essential; in our case, the relative ranking of methods remained consistent across multiple runs, supporting the robustness of our evaluations\.

## 5Results and discussion

Performance results across various tasks in Fig\.[3](https://arxiv.org/html/2608.17373#S4.F3)show that NSPER \(red plot\) results are better than baseline algorithms\. It enhances PixelTD3’s performance on four of five tasks while achieving comparable results on the fifth task to the best\-performing methods among image\-based RL approaches\. Our evaluation highlights the crucial role of novelty and surprise signals in improving learning efficiency\. In contrast, Uniform sampling \(orange plot\) exhibits suboptimal performance across all tasks, demonstrating that relying solely on a uniform experience buffer is insufficient\. These findings suggest that PixelTD3 could significantly benefit from integrating prioritized sampling or intrinsic reward mechanisms, even in relatively simple environments\.

Incorporating prioritization strategies alongside intrinsic rewards improves performance across nearly all algorithms, highlighting the advantages of these techniques in fostering more robust and adaptive learning\. In particular, NSPER\+R achieves the best results among the methods\. While CCLF \(light green plot\) consistently outperforms Uniform across all tasks, its overall performance remains unremarkable\. This suggests that CCLF’s prioritization strategy differs fundamentally from that of a prioritization buffer, and, as the results indicate, it does not integrate well with PixelTD3\.

In addition, the comparable performance of the RPE\-PER algorithm \(blue plot\) to NSPER across all tasks demonstrates its potential as an effective prioritization method\. However, NSPER consistently achieves better results\. These findings indicate that using a complex prioritization scale in image\-based RL can be beneficial, while also reaffirming that novelty and surprise signals are stronger heuristics for selecting informative experiences, ultimately contributing to a more sample\-efficient learning process\.

![Refer to caption](https://arxiv.org/html/2608.17373v1/new-pics/Ablition/Cartpole-Balance.png)\(a\)
![Refer to caption](https://arxiv.org/html/2608.17373v1/new-pics/Ablition/Finger-Spin.png)\(b\)
![Refer to caption](https://arxiv.org/html/2608.17373v1/new-pics/Ablition/Ball_In_Cup.png)\(c\)
![Refer to caption](https://arxiv.org/html/2608.17373v1/new-pics/Ablition/Walker-Walk.png)\(d\)
![Refer to caption](https://arxiv.org/html/2608.17373v1/new-pics/Ablition/Cheetah-Run.png)\(e\)
NSPER\+RNSPERNoveltyPER\+RNoveltyPERSurprisePER\+RSurprisePER

Figure 4:Ablation results comparing experience prioritization based on novelty, surprise, and their combination, with and without intrinsic rewards\. Shaded regions represent the 95% confidence interval over five runs\.### 5\.1Task\-Specific Analysis

This section examines the performance of different methods across individual tasks, highlighting their strengths and limitations in each environment\.

In Cartpole\-Balance \(Fig\.[3](https://arxiv.org/html/2608.17373#S4.F3), plot a\), a classic control task requiring a cart to balance a pole, all methods using either a prioritized buffer or intrinsic rewards perform well\. This suggests that PixelTD3 benefits from PER or intrinsic rewards, even in simpler environments\.

The Finger\-Spin task, which involves manipulating an object with a simulated finger, presents challenges due to complex contact dynamics and sparse rewards\. As shown in Fig\.[3](https://arxiv.org/html/2608.17373#S4.F3), plot b, NSPER variants outperform other methods, emphasizing the importance of novelty and surprise signals for learning efficiency\. NSPER\+R, which integrates intrinsic rewards with PER, further enhances performance\. Comparing TD\-PER \(prioritization based on TD error\) with Uniform\+R \(intrinsic rewards without prioritization\) reveals that intrinsic rewards yield better results in this environment than PER alone\. However, TD\-PER\+R underperforms relative to TD\-PER, suggesting that conflicting signals from the TD error and novelty\-surprise signals can hinder efficiency\. In contrast, NSPER\+R aligns both prioritization and intrinsic rewards using the same features, thereby improving sample efficiency\.

The Ball\-in\-Cup task, characterized by sparse rewards granted only upon successfully catching the ball, provides another challenging test\. In \(Fig\.[3](https://arxiv.org/html/2608.17373#S4.F3), plot c\), NSPER\+R outperforms other approaches, demonstrating the benefits of combining novelty and surprise for prioritization with intrinsic rewards\. While Uniform\+R performs well, NSPER\+R and RPE\-PER\+R achieve consistently better results, reinforcing that properly aligning prioritization with intrinsic rewards enhances performance\.

The Walker\-Walk environment, which requires controlling a bipedal walker to maintain dynamic balance, presents a complex, multi\-stage control challenge despite offering dense rewards\. \(Fig\.[3](https://arxiv.org/html/2608.17373#S4.F3), plot d\) shows that NSPER\+R and RPE\-PER\+R improve performance, highlighting the importance of context\-aware experience selection\. In contrast, TD error\-based PER leads to suboptimal results even when combined with intrinsic rewards\. This suggests that prioritizing solely based on high TD error can misguide learning, as such experiences may not always be the most informative\.

In Cheetah\-Run \(Fig\.[3](https://arxiv.org/html/2608.17373#S4.F3), plot e\), TD\-PER performs best, but NSPER closely follows, demonstrating its adaptability to high\-speed locomotion tasks\. This result indicates that prioritization strategies should be tailored to specific environments, as more straightforward prioritization methods can sometimes be more effective\. However, TD\-PER exhibits inconsistency, performing well in Cheetah\-Run but poorly in Walker\-Walk, as noted in prior studies\[[29](https://arxiv.org/html/2608.17373#bib.bib12)\]\. In contrast, NSPER maintains stable performance across diverse tasks, suggesting it provides a more reliable prioritization strategy\.

### 5\.2Ablation Study: The Role of Novelty and Surprise in Prioritization

This study evaluates various prioritization strategies to isolate the individual and combined effects of novelty and surprise\. NoveltyPER prioritizes experiences based solely on novelty, SurprisePER based solely on surprise, and NSPER jointly leverages both signals\. Solid curves denote variants that additionally use intrinsic rewards, while dashed curves show performance without them\. As shown in Fig\.[4](https://arxiv.org/html/2608.17373#S5.F4), prioritization based on both novelty and surprise outperforms using either signal alone\. This combination improves experience selection and, when also incorporated as intrinsic rewards, encourages more effective exploration\. The results highlight the complementary nature of novelty and surprise in RL\. Furthermore, integrating these signals as intrinsic rewards yields additional performance gains by promoting a more balanced selection of experiences, ultimately enhancing policy learning\.

In the Cheetah\-Run task, prioritization alone is sufficient, and incorporating surprise as an intrinsic reward negatively impacts performance\. Another key observation is that NoveltyPER outperforms SurprisePER in most tasks, suggesting that novelty\-based prioritization, which encourages exploration of diverse states, is more beneficial than surprise\-based prioritization, which focuses on unexpected transitions\. Additionally, the performance of SurprisePER and SurprisePER\+R remains similar in most cases, indicating that adding surprise as an intrinsic reward does not significantly contribute to performance improvements\. However, when both novelty and surprise are used in prioritization and intrinsic rewards, the agent achieves the best overall performance, demonstrating their complementary roles in improving learning efficiency\.

## 6Conclusion

In this work, we addressed the challenge of sample inefficiency in image\-based RL by introducing NSPER and its extension NSPER\+R, which integrate novelty and surprise as signals for replay prioritization and, in the case of NSPER\+R, also as intrinsic rewards\. This dual mechanism improves both replay quality and exploration, leading to faster convergence and more generalizable policies\. Through an extensive evaluation of the DeepMind Control Suite using the PixelTD3 backbone, we showed that NSPER and NSPER\+R outperform existing baselines, improving training efficiency in complex image\-based tasks\. Ablation studies further confirmed the complementary contributions of novelty and surprise to prioritization and exploration\. Our research highlights the importance of using internal stimuli and enhanced sampling buffers to improve RL systems\.

Future work should explore additional intrinsic motivators and adaptively weight novelty and surprise\. Our ablation study indicates that these signals impact learning efficiency differently, suggesting that dynamically adjusting their balance could refine the agent’s exploration strategy\.

## References

- \[1\]J\. Achiam and S\. Sastry\(2017\)Surprise\-based intrinsic motivation for deep reinforcement learning\.arXiv preprint arXiv:1703\.01732\.Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p3.1),[§2\.2](https://arxiv.org/html/2608.17373#S2.SS2.p2.1)\.
- \[2\]M\. Andrychowicz, F\. Wolski, A\. Ray, J\. Schneider, R\. Fong, P\. Welinder, B\. McGrew, J\. Tobin, O\. Pieter Abbeel, and W\. Zaremba\(2017\)Hindsight experience replay\.Advances in neural information processing systems30\.Cited by:[§2\.3](https://arxiv.org/html/2608.17373#S2.SS3.p1.1)\.
- \[3\]A\. Aubret, L\. Matignon, and S\. Hassas\(2019\)A survey on intrinsic motivation in reinforcement learning\.arXiv preprint arXiv:1908\.06976\.Cited by:[§2\.2](https://arxiv.org/html/2608.17373#S2.SS2.p1.1)\.
- \[4\]G\. Baldassarre\(2011\)What are intrinsic motivations? a biological perspective\.In2011 IEEE international conference on development and learning \(ICDL\),Vol\.2,pp\. 1–8\.Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p2.1)\.
- \[5\]A\. G\. Barto\(2013\)Intrinsic motivation and reinforcement learning\.Intrinsically motivated learning in natural and artificial systems,pp\. 17–47\.Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p1.1),[§3\.5](https://arxiv.org/html/2608.17373#S3.SS5.p1.1)\.
- \[6\]M\. Bellemare, S\. Srinivasan, G\. Ostrovski, T\. Schaul, D\. Saxton, and R\. Munos\(2016\)Unifying count\-based exploration and intrinsic motivation\.Advances in neural information processing systems29\.Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p3.1),[§2\.2](https://arxiv.org/html/2608.17373#S2.SS2.p1.1)\.
- \[7\]E\. K\. Braun, G\. E\. Wimmer, and D\. Shohamy\(2018\)Retroactive and graded prioritization of memory by reward\.Nature communications9\(1\),pp\. 4886\.Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p2.1)\.
- \[8\]R\. Carrasco\-Davis, S\. Lee, C\. Clopath, and W\. Dabney\(2025\)Uncertainty prioritized experience replay\.arXiv preprint arXiv:2506\.09270\.Cited by:[§2\.1](https://arxiv.org/html/2608.17373#S2.SS1.p1.1)\.
- \[9\]L\. Chen, K\. Lee, A\. Srinivas, and P\. Abbeel\(2021\)Improving computational efficiency in visual reinforcement learning via stored embeddings\.Advances in Neural Information Processing Systems34,pp\. 26779–26791\.Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p1.1)\.
- \[10\]E\. L\. Deci and R\. M\. Ryan\(2013\)Intrinsic motivation and self\-determination in human behavior\.Springer Science & Business Media\.Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p2.1)\.
- \[11\]F\. E\. Dorner\(2021\)Measuring progress in deep reinforcement learning sample efficiency\.arXiv preprint arXiv:2102\.04881\.Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p1.1)\.
- \[12\]W\. Fedus, P\. Ramachandran, R\. Agarwal, Y\. Bengio, H\. Larochelle, M\. Rowland, and W\. Dabney\(2020\)Revisiting fundamentals of experience replay\.InInternational conference on machine learning,pp\. 3061–3071\.Cited by:[item 1](https://arxiv.org/html/2608.17373#S4.I1.i1.p1.1)\.
- \[13\]S\. Ferraro, T\. Van de Maele, P\. Mazzaglia, T\. Verbelen, and B\. Dhoedt\(2022\)Computational optimization of image\-based reinforcement learning for robotics\.Sensors22\(19\),pp\. 7382\.Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p1.1)\.
- \[14\]S\. Fujimoto, H\. Hoof, and D\. Meger\(2018\)Addressing function approximation error in actor\-critic methods\.InInternational conference on machine learning,pp\. 1587–1596\.Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p5.1),[§3\.2](https://arxiv.org/html/2608.17373#S3.SS2.p1.1),[item 1](https://arxiv.org/html/2608.17373#S4.I1.i1.p1.1)\.
- \[15\]P\. Henderson, R\. Islam, P\. Bachman, J\. Pineau, D\. Precup, and D\. Meger\(2018\)Deep reinforcement learning that matters\.InProceedings of the AAAI conference on artificial intelligence,Vol\.32\.Cited by:[§4\.3](https://arxiv.org/html/2608.17373#S4.SS3.p3.1)\.
- \[16\]R\. Houthooft, X\. Chen, Y\. Duan, J\. Schulman, F\. De Turck, and P\. Abbeel\(2016\)Vime: variational information maximizing exploration\.Advances in neural information processing systems29\.Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p2.1)\.
- \[17\]Y\. Huang, P\. Peng, Y\. Zhao, Y\. Zhai, H\. Xu, and Y\. Tian\(2023\)Simoun: synergizing interactive motion\-appearance understanding for vision\-based reinforcement learning\.In2023 IEEE/CVF International Conference on Computer Vision \(ICCV\),Vol\.,pp\. 176–185\.External Links:[Document](https://dx.doi.org/10.1109/ICCV51070.2023.00023)Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p1.1)\.
- \[18\]A\. Jaegle, V\. Mehrpour, and N\. Rust\(2019\)Visual novelty, curiosity, and intrinsic reward in machine learning and the brain\.Current opinion in neurobiology58,pp\. 167–174\.Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p3.1),[§2\.2](https://arxiv.org/html/2608.17373#S2.SS2.p2.1)\.
- \[19\]I\. Kostrikov, D\. Yarats, and R\. Fergus\(2020\)Image augmentation is all you need: regularizing deep reinforcement learning from pixels\.arXiv preprint arXiv:2004\.13649\.Cited by:[§4\.1](https://arxiv.org/html/2608.17373#S4.SS1.p1.1)\.
- \[20\]M\. Kubovčík, I\. Dirgová Luptáková, and J\. Pospíchal\(2023\)Signal novelty detection as an intrinsic reward for robotics\.Sensors23\(8\),pp\. 3985\.Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p3.1),[§2\.2](https://arxiv.org/html/2608.17373#S2.SS2.p2.1)\.
- \[21\]M\. Laskin, A\. Srinivas, and P\. Abbeel\(2020\)Curl: contrastive unsupervised representations for reinforcement learning\.InInternational conference on machine learning,pp\. 5639–5650\.Cited by:[§4\.1](https://arxiv.org/html/2608.17373#S4.SS1.p1.1)\.
- \[22\]H\. Le, K\. Do, D\. Nguyen, and S\. Venkatesh\(2024\)Beyond surprise: improving exploration through surprise novelty\.\.InAAMAS,pp\. 1084–1092\.Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p3.1),[§2\.2](https://arxiv.org/html/2608.17373#S2.SS2.p2.1)\.
- \[23\]B\. Li, T\. Lu, J\. Li, N\. Lu, Y\. Cai, and S\. Wang\(2020\)ACDER: augmented curiosity\-driven experience replay\.In2020 IEEE International Conference on Robotics and Automation \(ICRA\),pp\. 4218–4224\.Cited by:[§2\.3](https://arxiv.org/html/2608.17373#S2.SS3.p1.1)\.
- \[24\]K\. Li, A\. Gupta, A\. Reddy, V\. H\. Pong, A\. Zhou, J\. Yu, and S\. Levine\(2021\)Mural: meta\-learning uncertainty\-aware rewards for outcome\-driven reinforcement learning\.InInternational conference on machine learning,pp\. 6346–6356\.Cited by:[§2\.2](https://arxiv.org/html/2608.17373#S2.SS2.p1.1)\.
- \[25\]X\. Liu, Z\. Xue, J\. Pang, S\. Jiang, F\. Xu, and Y\. Yu\(2021\)Regret minimization experience replay in off\-policy reinforcement learning\.Advances in neural information processing systems34,pp\. 17604–17615\.Cited by:[§2\.1](https://arxiv.org/html/2608.17373#S2.SS1.p1.1)\.
- \[26\]V\. Mnih, A\. P\. Badia, M\. Mirza, A\. Graves, T\. Lillicrap, T\. Harley, D\. Silver, and K\. Kavukcuoglu\(2016\)Asynchronous methods for deep reinforcement learning\.InInternational conference on machine learning,pp\. 1928–1937\.Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p1.1)\.
- \[27\]Y\. Oh, J\. Shin, E\. Yang, and S\. J\. Hwang\(2021\)Model\-augmented prioritized experience replay\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p4.1),[§2\.1](https://arxiv.org/html/2608.17373#S2.SS1.p2.1)\.
- \[28\]P\. Oudeyer and F\. Kaplan\(2007\)What is intrinsic motivation? a typology of computational approaches\.Frontiers in neurorobotics1,pp\. 6\.Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.17373#S2.SS2.p1.1)\.
- \[29\]Y\. Pan, J\. Mei, A\. Farahmand, M\. White, H\. Yao, M\. Rohani, and J\. Luo\(2022\)Understanding and mitigating the limitations of prioritized experience replay\.InUncertainty in Artificial Intelligence,pp\. 1561–1571\.Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p4.1),[§2\.1](https://arxiv.org/html/2608.17373#S2.SS1.p1.1),[§5\.1](https://arxiv.org/html/2608.17373#S5.SS1.p6.1)\.
- \[30\]D\. Pathak, P\. Agrawal, A\. A\. Efros, and T\. Darrell\(2017\)Curiosity\-driven exploration by self\-supervised prediction\.InInternational conference on machine learning,pp\. 2778–2787\.Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.17373#S2.SS2.p2.1)\.
- \[31\]B\. Saglam, F\. B\. Mutlu, D\. C\. Cicek, and S\. S\. Kozat\(2022\)Actor prioritized experience replay\.arXiv preprint arXiv:2209\.00532\.Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p4.1),[§2\.1](https://arxiv.org/html/2608.17373#S2.SS1.p2.1)\.
- \[32\]T\. Schaul, J\. Quan, I\. Antonoglou, and D\. Silver\(2015\)Prioritized experience replay\.arXiv preprint arXiv:1511\.05952\.Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p4.1),[§2\.1](https://arxiv.org/html/2608.17373#S2.SS1.p1.1),[item 3](https://arxiv.org/html/2608.17373#S4.I1.i3.p1.1)\.
- \[33\]A\. Scorsoglio, R\. Furfaro, R\. Linares, and B\. Gaudet\(2020\)Image\-based deep reinforcement learning for autonomous lunar landing\.InAIAA Scitech 2020 Forum,pp\. 1910\.Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p1.1)\.
- \[34\]Y\. Seo, L\. Chen, J\. Shin, H\. Lee, P\. Abbeel, and K\. Lee\(2021\)State entropy maximization with random encoders for efficient exploration\.InInternational Conference on Machine Learning,pp\. 9443–9454\.Cited by:[§2\.2](https://arxiv.org/html/2608.17373#S2.SS2.p1.1)\.
- \[35\]D\. Shohamy and R\. A\. Adcock\(2010\)Dopamine and adaptive memory\.Trends in cognitive sciences14\(10\),pp\. 464–472\.Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p2.1)\.
- \[36\]R\. Simmons\-Edler, B\. Eisner, D\. Yang, A\. Bisulco, E\. Mitchell, S\. Seung, and D\. Lee\(2019\)Reward prediction error as an exploration objective in deep rl\.arXiv preprint arXiv:1906\.08189\.Cited by:[§2\.1](https://arxiv.org/html/2608.17373#S2.SS1.p3.1)\.
- \[37\]C\. Sun, H\. Qian, and C\. Miao\(2022\)Cclf: a contrastive\-curiosity\-driven learning framework for sample\-efficient reinforcement learning\.arXiv preprint arXiv:2205\.00943\.Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p4.1),[§2\.2](https://arxiv.org/html/2608.17373#S2.SS2.p1.1),[§2\.3](https://arxiv.org/html/2608.17373#S2.SS3.p2.1),[item 7](https://arxiv.org/html/2608.17373#S4.I1.i7.p1.1)\.
- \[38\]Y\. Tassa, Y\. Doron, A\. Muldal, T\. Erez, Y\. Li, D\. d\. L\. Casas, D\. Budden, A\. Abdolmaleki, J\. Merel, A\. Lefrancq,et al\.\(2018\)Deepmind control suite\.arXiv preprint arXiv:1801\.00690\.Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p5.1),[§4\.1](https://arxiv.org/html/2608.17373#S4.SS1.p1.1)\.
- \[39\]D\. Valencia, H\. Williams, Y\. Xing, T\. Gee, M\. Liarokapis, and B\. A\. MacDonald\(2024\)Image\-based deep reinforcement learning with intrinsically motivated stimuli: on the execution of complex robotic tasks\.In2024 IEEE/RSJ International Conference on Intelligent Robots and Systems \(IROS\),pp\. 587–594\.Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p3.1),[§2\.2](https://arxiv.org/html/2608.17373#S2.SS2.p1.1),[§3\.2](https://arxiv.org/html/2608.17373#S3.SS2.p1.1),[item 2](https://arxiv.org/html/2608.17373#S4.I1.i2.p1.1),[§4\.1](https://arxiv.org/html/2608.17373#S4.SS1.p1.1)\.
- \[40\]H\. Yamani, Y\. Xing, L\. V\. C\. Ong, B\. A\. MacDonald, and H\. Williams\(2025\)Reward prediction error prioritisation in experience replay: the rpe\-per method\.arXiv preprint arXiv:2501\.18093\.Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p4.1),[§2\.1](https://arxiv.org/html/2608.17373#S2.SS1.p3.1),[item 5](https://arxiv.org/html/2608.17373#S4.I1.i5.p1.1)\.
- \[41\]D\. Yarats, A\. Zhang, I\. Kostrikov, B\. Amos, J\. Pineau, and R\. Fergus\(2021\)Improving sample efficiency in model\-free reinforcement learning from images\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.35,pp\. 10674–10681\.Cited by:[§3\.2](https://arxiv.org/html/2608.17373#S3.SS2.p1.1)\.
- \[42\]D\. Zha, K\. Lai, K\. Zhou, and X\. Hu\(2019\)Experience replay optimization\.arXiv preprint arXiv:1906\.08387\.Cited by:[§2\.1](https://arxiv.org/html/2608.17373#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2608.17373#S2.SS1.p2.1)\.
- \[43\]R\. Zhao and V\. Tresp\(2019\)Curiosity\-driven experience prioritization via density estimation\.arXiv preprint arXiv:1902\.08039\.Cited by:[§2\.3](https://arxiv.org/html/2608.17373#S2.SS3.p1.1)\.
- \[44\]J\. Zheng and Y\. Song\(2024\)Effective representation learning is more effective in reinforcement learning than you think\.In2024 IEEE International Conference on Robotics and Automation \(ICRA\),Vol\.,pp\. 9176–9182\.External Links:[Document](https://dx.doi.org/10.1109/ICRA57147.2024.10611330)Cited by:[§1](https://arxiv.org/html/2608.17373#S1.p1.1)\.