Repetition as Reinforcement: Enhancing Sample Efficiency via Instant Episode Repetition in Reinforcement Learning
Summary
The paper introduces Instant Episode Repetition (IER), a biologically inspired mechanism that enhances sample efficiency in reinforcement learning by actively repeating action sequences from successful episodes during data collection, demonstrating improved performance on MuJoCo, DeepMind Control Suite, and robotic tasks.
arXiv:2608.17347v1 Announce Type: new
Abstract: Repetition is a fundamental mechanism in human learning, where revisiting successful experiences strengthens memory, consolidates skills, and improves future performance. Motivated by this biological principle, we introduce Instant Episode Repetition (IER), a simple and novel mechanism that improves sample efficiency by immediately repeating action sequences from successful episodes during environment interaction. Unlike conventional approaches such as Experience Replay and Self-Imitation Learning (SIL), which passively reuse past experience during training updates, IER directly influences the data collection process. Upon identifying a high-reward episode, the agent repeats its action sequence for a fixed number of subsequent episodes, reinforcing valuable behaviors through renewed interaction with the environment. We integrate IER into state-of-the-art SAC and TD3 algorithms and evaluate its effectiveness on continuous-control benchmarks, including MuJoCo, the DeepMind Control Suite, and a real-world dynamic object translation task with a robotic manipulator. Experimental results demonstrate that this simple mechanism improves learning performance over standard and self-imitation-based baselines.
View Cached Full Text
Cached at: 08/19/26, 10:27 AM
# Enhancing Sample Efficiency via Instant Episode Repetition in Reinforcement Learning
Source: [https://arxiv.org/html/2608.17347](https://arxiv.org/html/2608.17347)
Repetition as Reinforcement: Enhancing Sample Efficiency via Instant Episode Repetition in Reinforcement Learning
Hoda Yamani, Yuning Xing, Koen van Rijnsoever, Bruce A\. MacDonald, Henry Williams
Keywords:Reinforcement Learning, Experience Replay, Episode Repetition, Sample Efficiency, Continuous Control, Self\-Imitation Learning, TD3, SAC
SummaryRepetition is a central mechanism in biological learning, where revisiting successful experiences strengthens memory and stabilizes skill acquisition\. Inspired by this principle, we proposeInstant Episode Repetition \(IER\), a simple mechanism that improves sample efficiency in reinforcement learning by actively re\-executing action sequences from high\-reward episodes during environment interaction\. Unlike replay\-buffer methods, which passively reuse stored transitions during policy updates, IER directly modifies the data collection process: when an episode achieves a higher cumulative reward than previously observed, the agent immediately re\-executes the same action sequence under different initial conditions for a fixed number of repetitions\. IER integrates seamlessly with off\-policy continuous\-control algorithms, including SAC and TD3, without architectural modifications\. Experiments on MuJoCo, the DeepMind Control Suite, and a real\-world robotic manipulation task show that IER improves learning efficiency and accelerates policy refinement compared with standard off\-policy and SIL\-enhanced baselines, suggesting that structured action\-sequence repetition provides a simple yet effective mechanism for enhancing reinforcement learning\.
Contribution\(s\)1\.We introduceIER, a biologically inspired mechanism that modifies the RL interaction loop by immediately re\-executing the action sequence from a prior high\-reward episode\. Context:Unlike replay\-buffer methods that reuse experience only after storage, IER intervenes during data collection by replacing policy\-selected actions with actions from a successful prior episode\.2\.IER introduces episode\-level behavioral consolidation by repeating complete action sequences from high\-reward episodes during environment interaction, rather than treating experience as independent transition samples\. Context:This shifts experience reuse from passive replay\-buffer sampling toward active episode\-level action replay, enabling successful behaviors to be revisited while new transitions are still being collected\.3\.We provide a simple and general integration of IER into off\-policy continuous\-control algorithms, demonstrating compatibility with SAC and TD3 without architectural modifications\. Context:The method operates at the interaction level and does not require changes to network structures or loss functions\.4\.Through evaluation on MuJoCo, the DeepMind Control Suite, and a real\-world robotic manipulation task, we demonstrate that immediate episode repetition improves sample efficiency and accelerates convergence under consistent training settings\. Context:The empirical analysis compares IER against standard off\-policy baselines and self\-imitation methods under consistent training settings\.
###### Abstract
Repetition is a fundamental mechanism in human learning, where revisiting successful experiences strengthens memory, consolidates skills, and improves future performance\. Motivated by this biological principle, we introduce Instant Episode Repetition \(IER\), a simple and novel mechanism that improves sample efficiency by immediately repeating action sequences from successful episodes during environment interaction\. Unlike conventional approaches such as Experience Replay and Self\-Imitation Learning \(SIL\), which passively reuse past experience during training updates, IER directly influences the data collection process\. Upon identifying a high\-reward episode, the agent repeats its action sequence for a fixed number of subsequent episodes, reinforcing valuable behaviors through renewed interaction with the environment\. We integrate IER into state\-of\-the\-art SAC and TD3 algorithms and evaluate its effectiveness on continuous\-control benchmarks, including MuJoCo, the DeepMind Control Suite, and a real\-world dynamic object translation task with a robotic manipulator\. Experimental results demonstrate that this simple mechanism improves learning performance over standard and self\-imitation\-based baselines\.
## 1Introduction
Reinforcement Learning \(RL\) offers a powerful framework for training agents to make sequential decisions by interacting with their environment and learning from feedback[35](https://arxiv.org/html/2608.17347#bib.bib36)\. RL has achieved notable success in various domains, from robotic manipulation and autonomous driving to game playing[16](https://arxiv.org/html/2608.17347#bib.bib4);[34](https://arxiv.org/html/2608.17347#bib.bib5);[23](https://arxiv.org/html/2608.17347#bib.bib6)\. However, one of the key challenges that continues to limit its broader applicability is sample efficiency, the ability to learn effective policies from a limited number of interactions[7](https://arxiv.org/html/2608.17347#bib.bib7)\. In contrast to biological agents, which can often learn complex behaviors from relatively few experiences, RL agents typically require millions of interactions to achieve comparable performance[25](https://arxiv.org/html/2608.17347#bib.bib8);[2](https://arxiv.org/html/2608.17347#bib.bib9)\. This inefficiency presents a significant barrier, especially in real\-world environments where interactions are expensive and time\-consuming[9](https://arxiv.org/html/2608.17347#bib.bib10)\.
Neuro\-scientific research highlights reward\-driven repetition as a fundamental mechanism underlying efficient learning[33](https://arxiv.org/html/2608.17347#bib.bib19);[8](https://arxiv.org/html/2608.17347#bib.bib20)\. When humans or animals experience a rewarding outcome, they are more likely to reengage in the same sequence of actions that led to that reward[22](https://arxiv.org/html/2608.17347#bib.bib21)\. This action repetition reinforces the link between actions and outcomes and supports the development of procedural memory and motor skills[17](https://arxiv.org/html/2608.17347#bib.bib28);[12](https://arxiv.org/html/2608.17347#bib.bib25)\. For example, when learning to ride a bike, a person may initially struggle, trying different ways to balance, steer, and pedal\. However, once they successfully ride a short distance, they instinctively repeat the same coordination of movements[10](https://arxiv.org/html/2608.17347#bib.bib18);[22](https://arxiv.org/html/2608.17347#bib.bib21)\. Each repetition improves their competence, and the successful action pattern becomes more stable and efficient\. At the neural level, this process strengthens synaptic connections through mechanisms such as synaptic plasticity and dopaminergic modulation, which support the consolidation and refinement of effective strategies over time[18](https://arxiv.org/html/2608.17347#bib.bib22);[32](https://arxiv.org/html/2608.17347#bib.bib24)\.
Figure 1:Illustration of the Instant Episode Repetition Algorithm: At the end of each episode \(when done is true\), if the agent identifies anepisode with a high reward, it immediately initiates repetition by executing the actions from that episode sequentially, from start to end, forRNRNrepetitions within the environment\.Inspired by this, we investigate how repetition can be explicitly incorporated into RL frameworks to reinforce high\-value experiences and improve learning\. While existing techniques such as experience replay[28](https://arxiv.org/html/2608.17347#bib.bib30)and PER[31](https://arxiv.org/html/2608.17347#bib.bib35)enhance training efficiency by reusing stored transitions, they do so passively during policy updates and do not directly influence how the agent behaves during interaction\. Consequently, even successful behaviors are rarely repeated in the environment, delaying their reinforcement\.
Recent methods, such as Self\-Imitation Learning \(SIL\)[29](https://arxiv.org/html/2608.17347#bib.bib11), build on this idea by encouraging agents to imitate past experiences with high cumulative rewards obtained within an episode\. SIL samples episodes from the replay buffer and prioritizes high returns to guide learning\. However, these approaches remain passive and do not directly alter the agent’s interaction with the environment[6](https://arxiv.org/html/2608.17347#bib.bib14)\.
We presentInstant Episode Repetition \(IER\), a novel and conceptually simple mechanism designed to improve sample efficiency in RL by reinforcing successful behaviors immediately upon discovery\. In contrast to conventional off\-policy approaches, which execute policy\-generated actions during interaction and rely primarily on passive reuse of stored transitions in replay buffers, IER modifies the data collection process through repetition of action sequences from newly discovered high\-reward episodes\. When the agent identifies an episode achieving a new highest reward, it stores the corresponding action sequence and repeats it immediately after the current episode for a fixed number of subsequent episodes, instead of sampling actions from the current policy\. By generating additional samples around behaviors that have already demonstrated high value, IER increases the density of informative experience in promising regions of the state–action space\. Importantly, this intervention affects only the interaction strategy and does not alter the underlying network architectures, objective functions, or optimization procedures\.
We integrate IER into standard off\-policy frameworks and evaluate its effectiveness across continuous\-control benchmarks in both simulated and real\-world robotic environments\. The experimental results demonstrate that IER outperforms the baseline algorithms, highlighting its potential to improve learning efficiency and performance\.Code availability:The implementation of IER is publicly available at[https://github\.com/UoA\-CARES/instant\-episode\-repetition](https://github.com/UoA-CARES/instant-episode-repetition)\.
## 2Background and Related Work
### 2\.1Behavioral and Neural Foundations of Repetition
Repetition and reinforcement are fundamental mechanisms of adaptive learning in biological systems[15](https://arxiv.org/html/2608.17347#bib.bib27)\. Reinforcement increases the likelihood of repeating actions associated with rewarding outcomes, a principle formalized in operant conditioning and supported by dopaminergic reward prediction signaling[37](https://arxiv.org/html/2608.17347#bib.bib29);[32](https://arxiv.org/html/2608.17347#bib.bib24)\. Repetition, in turn, stabilizes and refines reinforced behaviors through repeated activation of neural circuits and structured practice, strengthening synaptic efficacy and supporting long\-term memory consolidation[18](https://arxiv.org/html/2608.17347#bib.bib22);[4](https://arxiv.org/html/2608.17347#bib.bib23)\.
At the neural level, repeated activation of task\-relevant circuits strengthens synaptic connections, a process captured by Hebbian learning[18](https://arxiv.org/html/2608.17347#bib.bib22)\. Long\-term potentiation provides physiological evidence that sustained co\-activation enhances synaptic efficacy and supports memory consolidation[4](https://arxiv.org/html/2608.17347#bib.bib23)\. Through these mechanisms, repetition transforms transient successes into stable behavioral patterns and progressively increases execution efficiency[12](https://arxiv.org/html/2608.17347#bib.bib25)\.
Importantly, repetition is not mechanical duplication\. Behavioral and motor learning research emphasizes “repetition without repetition,” where each execution occurs under slightly varying conditions, promoting robustness and generalization[3](https://arxiv.org/html/2608.17347#bib.bib2);[26](https://arxiv.org/html/2608.17347#bib.bib3)\. Neuroimaging studies further demonstrate repetition suppression, reflecting increased processing efficiency as familiar patterns are re\-engaged[20](https://arxiv.org/html/2608.17347#bib.bib26)\. Together, these findings indicate that effective learning arises from repeatedly re\-engaging successful behaviors under natural variability\.
These principles suggest that repetition consolidates success by stabilizing effective behaviors while preserving adaptability\. Translating this insight to RL motivates the deliberate re\-execution of high\-reward episodes as a means of reinforcing valuable strategies during interaction\. Motivated by this biological perspective, we introduce IER as an interaction\-level repetition mechanism for policy learning\.
### 2\.2Self\-Imitation Learning \(SIL\)
SIL is a value\-based RL technique that improves policy learning by leveraging past experiences with high discounted return[29](https://arxiv.org/html/2608.17347#bib.bib11)\. It performs off\-policy updates by increasing the likelihood of actions whose discounted returns exceed the current value estimate\. However, SIL remains a passive method: although it updates the policy based on successful past episodes, the agent still samples actions from its current policy, and high\-reward trajectories do not directly shape new experience\. This separation between learning and behavior limits SIL’s ability to reinforce successful strategies during interaction\.
Recent extensions of SIL improve learning efficiency by integrating additional learning signals or auxiliary mechanisms\. For example,[27](https://arxiv.org/html/2608.17347#bib.bib15)present A\-SILfD, which incorporates expert demonstrations and constrains updates using ensemble Q\-functions to mitigate overestimation\. However, their approach relies on expert data emphasizing key behaviors, limiting its applicability where such data is unavailable or difficult to obtain\.
[1](https://arxiv.org/html/2608.17347#bib.bib16)combine SIL with intrinsic motivation to guide exploration toward previously successful behaviors, with intrinsic signals further emphasizing the prioritization of high\-reward episodes\.[6](https://arxiv.org/html/2608.17347#bib.bib14)introduce an episodic\-level replay strategy combined with hindsight to improve policy learning\. While useful in goal\-conditioned settings, relying on Hindsight Experience Replay \(HER\) limits its use in unstructured or general\-purpose environments\.[24](https://arxiv.org/html/2608.17347#bib.bib17)incorporate adversarial training to improve robustness within the SIL framework\. However, their method operates primarily through offline replay and does not influence action selection during interaction, reducing its effectiveness for real\-time adaptation and behavior shaping\.
In contrast, the proposed IER approach moves beyond purely policy\-driven sampling by allowing the agent to re\-execute entire successful trajectories, thereby embedding the recall of past successes directly into the interaction process\. By reinforcing temporally coherent action sequences rather than isolated transitions, IER strengthens valuable behavioral patterns while enabling on\-policy evaluation under dynamically evolving conditions\. To our knowledge, this is a novel direction in self\-imitation that can be integrated into a wide range of algorithms, including both value\-based and actor\-critic methods, without requiring structural modifications or external demonstrations\.
## 3Methodology
In this section, we introduce IER and describe how it integrates into standard off\-policy actor–critic RL\. IER operates at episode boundaries and alters the interaction dynamics while preserving the underlying learning objective and optimization procedure\.
### 3\.1Preliminaries
RL is formulated as a Markov Decision Process \(MDP\)
ℳ=\(𝒮,𝒜,P,R,γ\),\\mathcal\{M\}=\(\\mathcal\{S\},\\mathcal\{A\},P,R,\\gamma\),\(1\)where𝒮\\mathcal\{S\}and𝒜\\mathcal\{A\}denote the state and action spaces,P\(s′∣s,a\)P\(s^\{\\prime\}\\mid s,a\)is the transition kernel,R\(s,a\)R\(s,a\)the reward function, andγ∈\[0,1\)\\gamma\\in\[0,1\)the discount factor\.
An episode \(trajectory\) is defined as
τ=\{\(s0,a0,r0\),…,\(sT,aT,rT\)\},\\tau=\\\{\(s\_\{0\},a\_\{0\},r\_\{0\}\),\\ldots,\(s\_\{T\},a\_\{T\},r\_\{T\}\)\\\},\(2\)representing a complete interaction sequence from an initial state to termination\.
In off\-policy actor–critic methods such as TD3 and SAC, transitions\(st,at,rt,st\+1\)\(s\_\{t\},a\_\{t\},r\_\{t\},s\_\{t\+1\}\)are stored in a replay bufferℳ\\mathcal\{M\}and sampled to update the actorπθ\\pi\_\{\\theta\}and criticQQnetworks through temporal\-difference learning\.
For episode\-level selection in IER, we define the episode reward as the sum of rewards accumulated over an episode,
Rep\(τ\)=∑t=0Trt\.R\_\{\\text\{ep\}\}\(\\tau\)=\\sum\_\{t=0\}^\{T\}r\_\{t\}\.\(3\)
IER uses thisRep\(τ\)R\_\{\\text\{ep\}\}\(\\tau\)as the criterion for determining whether an episode should be repeated\.
### 3\.2Instant Episode Repetition
IER modifies the data collection phase of off\-policy RL by temporarily reinforcing high\-reward episodes\. It does not alter the reward function, loss formulation, network architecture, or optimization procedure\. Instead, it modifies how episodes are generated\.
At the end of each episode \(when thedoneflag is true\), the agent computes the episode rewardRepR\_\{\\text\{ep\}\}and updates the best observed value:
rmax←max\(rmax,Rep\)\.r\_\{\\max\}\\leftarrow\\max\(r\_\{\\max\},R\_\{\\text\{ep\}\}\)\.\(4\)
For the subsequent episode, the agent operates in one of two modes, as illustrated in Figure[1](https://arxiv.org/html/2608.17347#S1.F1):
- •Repetition:If the current episode achieves a new maximum episode reward, the full action sequence 𝐚⋆=\(a0⋆,…,aT⋆\)\\mathbf\{a\}^\{\\star\}=\(a\_\{0\}^\{\\star\},\\dots,a\_\{T\}^\{\\star\}\)\(5\)is stored\. The agent then enters a repetition phase lastingRNRNconsecutive episodes, whereRNRNis a hyperparameter controlling repetition strength\. During this phase, the stored action sequence𝐚⋆\\mathbf\{a\}^\{\\star\}is executed sequentially instead of sampling from the policy\.
- •Exploitation:If no new maximum\-reward episode is detected, or once repetition ends, the agent resumes standard behavior by sampling actions from its current policy: at∼πθ\(⋅∣st\)\.a\_\{t\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid s\_\{t\}\)\.\(6\)
All transitions collected during both repetition and exploitation are inserted into the replay bufferℳ\\mathcal\{M\}and used for standard off\-policy updates\. Consequently, IER modifies only the visitation distribution of newly collected episodes while preserving the underlying learning dynamics\. The complete training procedure is summarized in pseudo\-code in the supplementary material\.
### 3\.3Re\-execution Under New Initial Conditions
A key property of IER is that repetition does not duplicate identical episodes\. Although several benchmark environments are deterministic in their transition dynamics, each episode starts from an initial state sampled from a reset distributionp0p\_\{0\}\. As a result, re\-executing the same stored action sequence𝐚⋆\\mathbf\{a\}^\{\\star\}does not reproduce the original episode exactly\. Instead, it generates a family of locally related episodes in the neighborhood of the previously observed high\-reward behavior\.
Rather than cloning past experience, IER replays successful action sequences under perturbed initial conditions\. While the action sequence remains fixed, the resulting trajectories differ at the state level due to variations in the starting state and subsequent state evolution\. Consequently, repetition increases sampling density in regions of the state space that have empirically yielded high rewards, without duplicating identical transitions\. This concentrates visitation around informative regions while avoiding collapse to a single deterministic trajectory\.
In stochastic environments or real\-world robotic systems, additional variability from sensor noise, contact dynamics, and actuation uncertainty further broadens this local sampling distribution\. Thus, IER preserves replay diversity while reinforcing behaviorally salient regions, supporting stable and generalizable learning\.
## 4Experiments
We evaluate the proposed IER method on a diverse set of high\-dimensional continuous control tasks in both simulated and real\-world environments\. The goal is to examine how episodic repetition affects learning performance when combined with state\-of\-the\-art RL algorithms\. The following subsections introduce the baseline algorithms used for comparison and describe the simulated and real\-world tasks employed in the experiments\.
### 4\.1Baseline Algorithms and Variants
We integrate IER into two widely used off\-policy RL algorithms: TD3[11](https://arxiv.org/html/2608.17347#bib.bib37)and SAC[14](https://arxiv.org/html/2608.17347#bib.bib33), yielding the variantsIER\-TD3andIER\-SAC\. To isolate the effect of IER, we compare these agents with:
- •TD3 and SAC:Standard implementations with no episode repetition\.
- •SIL\-TD3andSIL\-SAC: We integrate SIL[29](https://arxiv.org/html/2608.17347#bib.bib11)with TD3 following the implementation in[21](https://arxiv.org/html/2608.17347#bib.bib13), and extend the same formulation to SAC for a fair comparison\. We deliberately use the standard SIL formulation rather than newer variants to avoid introducing additional mechanisms \(e\.g\., demonstrations, intrinsic rewards\) discussed in Background and Related Works section, which could confound the comparison\. This provides a clean baseline to highlight the behavioral difference between passive experience reuse in SIL and active episodic repetition in IER\.
### 4\.2Simulation Environments
We evaluated IER and baseline algorithms on eight continuous control tasks spanning both theDeepMind Control Suite[36](https://arxiv.org/html/2608.17347#bib.bib31)andMuJoCo[38](https://arxiv.org/html/2608.17347#bib.bib32)\. These simulation platforms are widely recognized benchmarks for RL, providing diverse dynamics, varying levels of task complexity, and high\-dimensional state\-action spaces that closely resemble real\-world continuous control challenges\. By selecting tasks from both environments, we ensure a comprehensive evaluation of IER’s robustness and generalization across different domains and difficulty levels\. Detailed descriptions and configurations of the specific tasks used in our experiments are provided in the supplementary material\.
### 4\.3Real\-World Task: Dynamic Object Translation via In\-Hand Manipulation
Figure 2:Real\-world setup for dynamic object translation\. The cube must be moved laterally while suspended in the air by a custom\-built gripper\.We evaluated our method on a stochastic dexterous in\-hand manipulation task that requires moving a cube sideways while maintaining a stable grasp\. As shown in Figure[2](https://arxiv.org/html/2608.17347#S4.F2), each episode starts with a reset mechanism that autonomously repositions the cube to ensure consistent trial conditions and minimize human intervention\. The gripper must perform precise and coordinated movements to keep the cube from slipping or falling, relying entirely on its dexterity without external support\.
To provide a reliable perception, the workspace is uniformly illuminated by a ring of LEDs, while a camera tracks the pose of the cube in real time using ArUco markers\. The agent receives rewards for maintaining a stable grasp and increasing lateral displacement between time steps, encouraging fine\-grained control while preserving grasp integrity\. This setup captures key real\-world challenges, including contact dynamics, gravitational effects, sensor noise, and constraints on data collection, while supporting scalable, physically grounded experimentation\. Additional details on the hardware, reward function, and implementation are provided in the supplementary material\.
SAC
TD3
BaselineSILIER
Figure 3:Performance comparison of Baseline, SIL, and the proposed IER method across eight continuous\-control tasks under SAC \(top\) and TD3 \(bottom\)\. Curves show the mean episodic return over five independent seeds, and shaded regions represent the standard deviation across five random seeds\.
## 5Experimental Results
We evaluate the performance of the proposed repetition\-based methods,IER\-SACandIER\-TD3, across eight simulated continuous\-control tasks and a real\-world robotic task\. These variants are compared against the standard SAC and TD3 baselines, along with their respective self\-imitation learning variants \(SIL\-SAC and SIL\-TD3\)\. The learning curves for simulated tasks are shown in Figure[3](https://arxiv.org/html/2608.17347#S4.F3), which reports SAC\-based methods \(top\) and TD3\-based methods \(bottom\)\. Results for the real\-world setup are presented in Figure[4](https://arxiv.org/html/2608.17347#S5.F4)\.
Across most tasks,IER\-SACandIER\-TD3consistently outperform their respective baselines, demonstrating that repetition of high\-reward episodes improves learning efficiency and convergence speed with minimal additional complexity\.
In some environments, baseline performance is slightly lower than values reported in the original publications, likely due to simulator stochasticity and random seed variation\. However, relative performance differences between competing methods remain consistent across seeds\. Our evaluation follows established best practices for fair RL benchmarking, as outlined by[19](https://arxiv.org/html/2608.17347#bib.bib34)\.
We begin by presenting results from simulated environments for both SAC\- and TD3\-based variants, followed by evaluation in a real\-world setting\. Finally, we conduct a sensitivity analysis to examine the impact of RN\.
### 5\.1Performance in Simulated Environments
As shown in Figure[3](https://arxiv.org/html/2608.17347#S4.F3),IER\-SACimproves over the SAC baseline in six out of eight tasks and performs comparably in the remaining two\. The strongest gains occur in dynamic locomotion tasks such asAnt\-v4,HalfCheetah\-v4,Walker\-Walk, andCheetah\-Run, where agents must learn coordinated and temporally extended movement patterns\. In these domains, repetition reinforces stable behavioral sequences, leading to faster convergence and improved long\-term rewards\.
SIL\-SACperforms competitively in selected environments, includingHalfCheetah\-v4andWalker\-Walk, but shows inconsistent performance in more unstable tasks such asHumanoid\-v4andCartpole\-Swingup\. In some cases, performance falls below the SAC baseline, reflecting a known limitation of self\-imitation learning: the risk of reinforcing suboptimal trajectories when exploration remains incomplete[29](https://arxiv.org/html/2608.17347#bib.bib11);[13](https://arxiv.org/html/2608.17347#bib.bib12)\.
Under TD3 \(bottom row of Figure[3](https://arxiv.org/html/2608.17347#S4.F3)\),IER\-TD3outperforms standard TD3 in six out of eight tasks and exceeds SIL\-TD3 in five tasks, with notable improvements inHalfCheetah\-v4,Humanoid\-v4, andWalker\-Walk\. These results further confirm that repetition strengthens beneficial trajectory segments that contribute to improved cumulative reward\.
While both IER variants outperform their respective baselines, the relative improvements observed withIER\-TD3are generally smaller than those seen withIER\-SAC\. This difference may stem from the underlying exploration mechanisms\. SAC incorporates entropy regularization, promoting broader exploration and maintaining policy diversity\. The repetition mechanism complements this objective by reinforcing effective behaviors without substantially limiting exploration\.
In contrast, TD3 employs deterministic policy updates, which may be more sensitive to repeated behavior sequences and thus more prone to premature convergence if repetition is excessive\. This architectural difference likely explains the comparatively smaller but still consistent gains observed forIER\-TD3\.
### 5\.2Real\-World Evaluation
\(a\)SAC variants
\(b\)TD3 variants
BaseSILIER
Figure 4:Performance comparison of standard, SIL\-enhanced, and IER\-enhanced SAC and TD3 variants in the real\-world gripper manipulation task\. Curves show the mean episodic return over five independent seeds, and shaded regions represent the standard deviation across five random seeds\.To evaluate the robustness of the proposed IER method under real\-world stochasticity, we conduct experiments on a physical robotic manipulation task \(see Experiments Section\)\. Unlike simulated benchmarks, real\-world environments introduce unavoidable sources of variability, including sensor noise, friction and contact inconsistencies, and actuation delays\. These factors make trajectory execution inherently stochastic, even when identical action commands are applied\.
As shown in Figure[4](https://arxiv.org/html/2608.17347#S5.F4),IER\-SACachieves faster convergence and higher final performance compared to both SAC and SIL\-SAC\. A similar pattern is observed in the TD3\-based setting, whereIER\-TD3outperforms TD3 and SIL\-TD3, although with narrower margins\. While SIL\-TD3 exhibits a temporary early improvement, its performance quickly saturates and fails to sustain gains, allowing both TD3 and IER\-TD3 to surpass it after relatively few environment interactions\.
These results indicate that the repetition strategy remains effective under real\-world stochasticity\. In particular, IER improves data efficiency and promotes more stable learning dynamics in this setting\.
### 5\.3Sensitivity Analysis of the Repetition Number \(RN\)
This section examines how the RN affects IER performance\. RN specifies how many consecutive episodes re\-execute a high\-reward episode once it is identified\. We evaluate RN values from\{0,1,…,7\}\\\{0,1,\\ldots,7\\\}for both IER\-SAC and IER\-TD3 across all benchmark tasks\. Performance is measured using the normalized Area Under the Learning Curve \(AUC\), capturing both learning speed and stability\. We report the relative improvement over RN=0 \(no repetition\):
ΔAUC%=AUCRN−AUCRN0AUCRN0×100\.\\Delta\\mathrm\{AUC\}\\%=\\frac\{\\mathrm\{AUC\}\_\{\\mathrm\{RN\}\}\-\\mathrm\{AUC\}\_\{\\mathrm\{RN0\}\}\}\{\\mathrm\{AUC\}\_\{\\mathrm\{RN0\}\}\}\\times 100\.\(7\)
#### 5\.3\.1Aggregate Trends Across Tasks
Figure 5:MeanΔ\\DeltaAUC% across all eight tasks as a function of the RN, jointly averaged over SAC and TD3\.Figure[5](https://arxiv.org/html/2608.17347#S5.F5)shows the meanΔ\\DeltaAUC% across all eight tasks as a function of RN, jointly averaged over SAC and TD3\. Performance peaks at RN3 \(10\.66%\), followed by RN2 and RN4, and declines at larger repetition levels\. The resulting unimodal pattern indicates that moderate repetition yields the strongest overall acceleration of learning\. Small RN values may insufficiently reinforce high\-reward episodes, whereas excessively large RN values can reduce state diversity, leading to diminishing returns\.
We report the joint average across SAC and TD3 to provide a unified view of repetition effects across baseline algorithms\. Although task\-wise analyses are presented separately, the aggregate curve highlights repetition dynamics that remain consistent across settings and provides a compact estimate of the most effective repetition range\.
#### 5\.3\.2Task\-Specific Trends and Algorithm Sensitivity
Figure[6](https://arxiv.org/html/2608.17347#S5.F6)presents the per\-taskΔ\\DeltaAUC% as a function of the RN for both TD3 and SAC\. Results suggest that environments with higher instability or reward variance tend to benefit more strongly from repetition\. For example,Finger\-Turn\-HardandWalker\-Walkshow substantial gains at moderate or higher RN values\. InWalker\-WalkandCheetah\-Run, performance often improves with increasing RN and typically peaks at intermediate levels \(RN3–RN5\)\. In contrast, tasks such asHopper\-v4andCartpole\-Swingupexhibit smaller and more variable changes, consistent with their comparatively stable reward dynamics\.
The impact of repetition varies across baseline algorithms\. For TD3, performance commonly peaks at moderate RN values \(RN2–RN3\), yielding consistent improvements across multiple tasks\. In contrast, SAC tends to benefit from smaller RN values, with performance exhibiting greater sensitivity at larger repetition levels\. This difference reflects their respective update mechanisms: TD3 gains from stronger reinforcement of high\-reward episodes, while SAC’s entropy regularization inherently promotes exploration, increasing its sensitivity to excessive repetition\.
Although the optimal RN varies across environments and occasional performance declines are observed at certain repetition levels, moderate repetition \(RN2–RN4\) often achieves competitive and near\-peak performance\. These results indicate that RN should be treated as a tunable hyperparameter rather than a fixed setting\.
Figure 6:Per\-taskΔ\\DeltaAUC% as a function of the Repetition Number \(RN\), from RN0 \(no repetition\) to RN7, for SAC and TD3\.Additional analyses, including complete RN0–RN7 performance curves, domain\-level comparisons, and agreement analysis between TD3 and SAC, are provided in the supplementary material\. These extended results further demonstrate the robustness of moderate repetition and confirm its effectiveness across both MuJoCo and DeepMind Control environments\.
## 6Conclusion
This work introduced IER, a repetition\-based mechanism that enhances RL by directly reinforcing action sequences from high\-reward episodes through immediate re\-execution\. Unlike traditional approaches that rely solely on replayed transitions, IER intervenes during data collection, enabling agents to revisit successful behaviours during data collection\.
IER provides a simple, general, and biologically inspired extension to off\-policy RL that requires no architectural modifications\. Empirical results across simulated benchmarks and real\-world robotic tasks demonstrate consistent improvements in data efficiency and training stability\.
## References
- A\. Andres, E\. Villar\-Rodriguez, and J\. Del SerTowards improving exploration in self\-imitation learning using intrinsic motivation\.In2022 IEEE Symposium Series on Computational Intelligence \(SSCI\),pp\. 890–899\.Cited by:[§2\.2](https://arxiv.org/html/2608.17347#S2.SS2.p3.1)\.
- Bellemareet al\.\(2016\)M\. Bellemare, S\. Srinivasan, G\. Ostrovski, T\. Schaul, D\. Saxton, and R\. MunosUnifying count\-based exploration and intrinsic motivation\.Advances in neural information processing systems29\.Cited by:[§1](https://arxiv.org/html/2608.17347#S1.p1.1)\.
- Bernstein \(1967\)N\. BernsteinThe coordination and regulation of movements\.\(No Title\)\.Cited by:[§2\.1](https://arxiv.org/html/2608.17347#S2.SS1.p3.1)\.
- Bliss and Lømo \(1973\)T\. V\. Bliss and T\. LømoLong\-lasting potentiation of synaptic transmission in the dentate area of the anaesthetized rabbit following stimulation of the perforant path\.The Journal of physiology232\(2\),pp\. 331–356\.Cited by:[§2\.1](https://arxiv.org/html/2608.17347#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2608.17347#S2.SS1.p2.1)\.
- Dafleet al\.\(2014\)N\. C\. Dafle, A\. Rodriguez, R\. Paolini, B\. Tang, S\. S\. Srinivasa, M\. Erdmann, M\. T\. Mason, I\. Lundberg, H\. Staab, and T\. FuhlbriggeExtrinsic dexterity: in\-hand manipulation with external forces\.In2014 IEEE International Conference on Robotics and Automation \(ICRA\),pp\. 1578–1585\.Cited by:[Real\-World Robotic Task](https://arxiv.org/html/2608.17347#Sx3.SSx1.p2.1)\.
- Daiet al\.\(2020\)T\. Dai, H\. Liu, and A\. Anthony BharathEpisodic self\-imitation learning with hindsight\.Electronics9\(10\),pp\. 1742\.Cited by:[§1](https://arxiv.org/html/2608.17347#S1.p4.1),[§2\.2](https://arxiv.org/html/2608.17347#S2.SS2.p3.1)\.
- Ding and Dong \(2020\)Z\. Ding and H\. DongChallenges of reinforcement learning\.Deep Reinforcement Learning: Fundamentals, Research and Applications,pp\. 249–272\.Cited by:[§1](https://arxiv.org/html/2608.17347#S1.p1.1)\.
- Doya \(2008\)K\. DoyaModulators of decision making\.Nature neuroscience11\(4\),pp\. 410–416\.Cited by:[§1](https://arxiv.org/html/2608.17347#S1.p2.1)\.
- Dulac\-Arnoldet al\.\(2021\)G\. Dulac\-Arnold, N\. Levine, D\. J\. Mankowitz, J\. Li, C\. Paduraru, S\. Gowal, and T\. HesterChallenges of real\-world reinforcement learning: definitions, benchmarks and analysis\.Machine Learning110\(9\),pp\. 2419–2468\.Cited by:[§1](https://arxiv.org/html/2608.17347#S1.p1.1)\.
- Ericssonet al\.\(1993\)K\. A\. Ericsson, R\. T\. Krampe, and C\. Tesch\-RömerThe role of deliberate practice in the acquisition of expert performance\.\.Psychological review100\(3\),pp\. 363\.Cited by:[§1](https://arxiv.org/html/2608.17347#S1.p2.1)\.
- Fujimotoet al\.\(2018\)S\. Fujimoto, H\. Hoof, and D\. MegerAddressing function approximation error in actor\-critic methods\.InInternational conference on machine learning,pp\. 1587–1596\.Cited by:[§4\.1](https://arxiv.org/html/2608.17347#S4.SS1.p1.1),[Appendix D: Configuration and Hyperparameter Settings](https://arxiv.org/html/2608.17347#Sx4.p1.1)\.
- Graybiel \(2008\)A\. M\. GraybielHabits, rituals, and the evaluative brain\.Annu\. Rev\. Neurosci\.31\(1\),pp\. 359–387\.Cited by:[§1](https://arxiv.org/html/2608.17347#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.17347#S2.SS1.p2.1)\.
- Guoet al\.\(2019\)Y\. Guo, J\. Choi, M\. Moczulski, S\. Bengio, M\. Norouzi, and H\. LeeSelf\-imitation learning via trajectory\-conditioned policy for hard\-exploration tasks\.Cited by:[§5\.1](https://arxiv.org/html/2608.17347#S5.SS1.p2.1)\.
- Haarnojaet al\.\(2018\)T\. Haarnoja, A\. Zhou, P\. Abbeel, and S\. LevineSoft actor\-critic: off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.InInternational conference on machine learning,pp\. 1861–1870\.Cited by:[§4\.1](https://arxiv.org/html/2608.17347#S4.SS1.p1.1)\.
- Haleemet al\.\(2015\)A\. Haleem, M\. K\. Khan, S\. Sufia, S\. Chaudhry, M\. I\. Siddiqui, and A\. A\. KhanThe role of repetition and reinforcement in school\-based oral health education\-a cluster randomized controlled trial\.BMC Public Health16,pp\. 1–11\.Cited by:[§2\.1](https://arxiv.org/html/2608.17347#S2.SS1.p1.1)\.
- Hanet al\.\(2023\)D\. Han, B\. Mulyana, V\. Stankovic, and S\. ChengA survey on deep reinforcement learning algorithms for robotic manipulation\.Sensors23\(7\),pp\. 3762\.Cited by:[§1](https://arxiv.org/html/2608.17347#S1.p1.1)\.
- Hartley \(2008\)J\. HartleyLearning and studying: a research perspective\.Cited by:[§1](https://arxiv.org/html/2608.17347#S1.p2.1)\.
- Hebb \(2005\)D\. O\. HebbThe organization of behavior: a neuropsychological theory\.Psychology press\.Cited by:[§1](https://arxiv.org/html/2608.17347#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.17347#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2608.17347#S2.SS1.p2.1)\.
- Hendersonet al\.\(2018\)P\. Henderson, R\. Islam, P\. Bachman, J\. Pineau, D\. Precup, and D\. MegerDeep reinforcement learning that matters\.InProceedings of the AAAI conference on artificial intelligence,Vol\.32\.Cited by:[§5](https://arxiv.org/html/2608.17347#S5.p3.1)\.
- Henson \(2003\)R\. N\. HensonNeuroimaging studies of priming\.Progress in neurobiology70\(1\),pp\. 53–81\.Cited by:[§2\.1](https://arxiv.org/html/2608.17347#S2.SS1.p3.1)\.
- Huet al\.\(2021\)H\. Hu, J\. Ye, G\. Zhu, Z\. Ren, and C\. ZhangGeneralizable episodic memory for deep reinforcement learning\.arXiv preprint arXiv:2103\.06469\.Cited by:[2nd item](https://arxiv.org/html/2608.17347#S4.I1.i2.p1.1)\.
- Kahanaet al\.\(2024\)M\. J\. Kahana, N\. B\. Diamond, and A\. AkaLaws of human memory\.The Oxford Handbook of Human Memory, Two Volume Pack: Foundations and Applications,pp\. 29–63\.Cited by:[§1](https://arxiv.org/html/2608.17347#S1.p2.1)\.
- Kiranet al\.\(2021\)B\. R\. Kiran, I\. Sobh, V\. Talpaert, P\. Mannion, A\. A\. Al Sallab, S\. Yogamani, and P\. PérezDeep reinforcement learning for autonomous driving: a survey\.IEEE transactions on intelligent transportation systems23\(6\),pp\. 4909–4926\.Cited by:[§1](https://arxiv.org/html/2608.17347#S1.p1.1)\.
- Kuanget al\.\(2025\)Y\. Kuang, L\. J\. Manso, and G\. VogiatzisGoal\-based self\-adaptive generative adversarial imitation learning \(goal\-sagail\) for multi\-goal robotic manipulation tasks\.arXiv preprint arXiv:2506\.12676\.Cited by:[§2\.2](https://arxiv.org/html/2608.17347#S2.SS2.p3.1)\.
- Lakeet al\.\(2017\)B\. M\. Lake, T\. D\. Ullman, J\. B\. Tenenbaum, and S\. J\. GershmanBuilding machines that learn and think like people\.Behavioral and brain sciences40,pp\. e253\.Cited by:[§1](https://arxiv.org/html/2608.17347#S1.p1.1)\.
- Latash \(2012\)M\. L\. LatashThe bliss \(not the problem\) of motor abundance \(not redundancy\)\.Experimental brain research217\(1\),pp\. 1–5\.Cited by:[§2\.1](https://arxiv.org/html/2608.17347#S2.SS1.p3.1)\.
- Liet al\.\(2023\)C\. Li, F\. Wu, and J\. ZhaoAccelerating self\-imitation learning from demonstrations via policy constraints and q\-ensemble\.In2023 International Joint Conference on Neural Networks \(IJCNN\),pp\. 1–8\.Cited by:[§2\.2](https://arxiv.org/html/2608.17347#S2.SS2.p2.1)\.
- Lin \(1992\)L\. LinSelf\-improving reactive agents based on reinforcement learning, planning and teaching\.Machine learning8,pp\. 293–321\.Cited by:[§1](https://arxiv.org/html/2608.17347#S1.p3.1)\.
- Ohet al\.\(2018\)J\. Oh, Y\. Guo, S\. Singh, and H\. LeeSelf\-imitation learning\.InInternational conference on machine learning,pp\. 3878–3887\.Cited by:[§1](https://arxiv.org/html/2608.17347#S1.p4.1),[§2\.2](https://arxiv.org/html/2608.17347#S2.SS2.p1.1),[2nd item](https://arxiv.org/html/2608.17347#S4.I1.i2.p1.1),[§5\.1](https://arxiv.org/html/2608.17347#S5.SS1.p2.1)\.
- Ohet al\.\(2021\)Y\. Oh, J\. Shin, E\. Yang, and S\. J\. HwangModel\-augmented prioritized experience replay\.InInternational Conference on Learning Representations,Cited by:[Appendix D: Configuration and Hyperparameter Settings](https://arxiv.org/html/2608.17347#Sx4.p1.1)\.
- Schaulet al\.\(2015\)T\. Schaul, J\. Quan, I\. Antonoglou, and D\. SilverPrioritized experience replay\.arXiv preprint arXiv:1511\.05952\.Cited by:[§1](https://arxiv.org/html/2608.17347#S1.p3.1)\.
- Schultzet al\.\(1997\)W\. Schultz, P\. Dayan, and P\. R\. MontagueA neural substrate of prediction and reward\.Science275\(5306\),pp\. 1593–1599\.Cited by:[§1](https://arxiv.org/html/2608.17347#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.17347#S2.SS1.p1.1)\.
- Schultz \(2006\)W\. SchultzBehavioral theories and the neurophysiology of reward\.Annu\. Rev\. Psychol\.57\(1\),pp\. 87–115\.Cited by:[§1](https://arxiv.org/html/2608.17347#S1.p2.1)\.
- Silveret al\.\(2016\)D\. Silver, A\. Huang, C\. J\. Maddison, A\. Guez, L\. Sifre, G\. Van Den Driessche, J\. Schrittwieser, I\. Antonoglou, V\. Panneershelvam, M\. Lanctot,et al\.Mastering the game of go with deep neural networks and tree search\.nature529\(7587\),pp\. 484–489\.Cited by:[§1](https://arxiv.org/html/2608.17347#S1.p1.1)\.
- Suttonet al\.\(1998\)R\. S\. Sutton A\. G\. Bartoet al\.Reinforcement learning: an introduction\.Vol\.1,MIT press Cambridge\.Cited by:[§1](https://arxiv.org/html/2608.17347#S1.p1.1)\.
- Tassaet al\.\(2018\)Y\. Tassa, Y\. Doron, A\. Muldal, T\. Erez, Y\. Li, D\. d\. L\. Casas, D\. Budden, A\. Abdolmaleki, J\. Merel, A\. Lefrancq,et al\.Deepmind control suite\.arXiv preprint arXiv:1801\.00690\.Cited by:[§4\.2](https://arxiv.org/html/2608.17347#S4.SS2.p1.1),[Appendix C: Environment Descriptions](https://arxiv.org/html/2608.17347#Sx3.p1.1)\.
- Thorndike \(2017\)E\. ThorndikeAnimal intelligence: experimental studies\.Routledge\.Cited by:[§2\.1](https://arxiv.org/html/2608.17347#S2.SS1.p1.1)\.
- Todorovet al\.\(2012\)E\. Todorov, T\. Erez, and Y\. TassaMujoco: a physics engine for model\-based control\.In2012 IEEE/RSJ international conference on intelligent robots and systems,pp\. 5026–5033\.Cited by:[§4\.2](https://arxiv.org/html/2608.17347#S4.SS2.p1.1),[Appendix C: Environment Descriptions](https://arxiv.org/html/2608.17347#Sx3.p1.1)\.
- Valenciaet al\.\(2024\)D\. Valencia, H\. Williams, Y\. Xing, T\. Gee, M\. Liarokapis, and B\. A\. MacDonaldImage\-based deep reinforcement learning with intrinsically motivated stimuli: on the execution of complex robotic tasks\.In2024 IEEE/RSJ International Conference on Intelligent Robots and Systems,External Links:[Link](https://arxiv.org/abs/2407.21338)Cited by:[Real\-World Robotic Task](https://arxiv.org/html/2608.17347#Sx3.SSx1.p3.1)\.
Supplementary Materials
*The following content was not necessarily subject to peer review\.*
This supplementary material provides additional methodological details and experimental results that complement the main paper\. The content is organized as follows:
- •Appendix A– Pseudo\-code of the proposed IER algorithm\.
- •Appendix B– Extended Analysis of the Repetition Number \(RN\)
- •Appendix C– Description of the experimental environments, including MuJoCo tasks, the DeepMind Control Suite, and the real\-worldDynamic Object Translation with Gripper Manipulationsetup\.
- •Appendix D– Detailed hyperparameter configurations for IER and baseline algorithms\.
## Appendix A: IER Algorithm
In this section, we present the pseudocode of the Instant Episode Repetition \(IER\) algorithm, as shown in Algorithm[1](https://arxiv.org/html/2608.17347#alg1)\.
Algorithm 1Policy Training with Instant Episode Repetition \(IER\)1:Initialize:Policy
πθ\\pi\_\{\\theta\}, environment
ℰ\\mathcal\{E\}, buffer
ℳ\\mathcal\{M\}, exploration steps
TexploreT\_\{\\text\{explore\}\}, training steps
TtrainT\_\{\\text\{train\}\}, repetition length
RNR\_\{N\}
2:
Rmax←0R\_\{\\text\{max\}\}\\leftarrow 0,
r←0r\\leftarrow 0,
𝐚∗←\[\]\\mathbf\{a\}^\{\*\}\\leftarrow\[\],
s0←ℰ\.reset\(\)s\_\{0\}\\leftarrow\\mathcal\{E\}\.\\text\{reset\}\(\),
Re←0R\_\{e\}\\leftarrow 0,
step←0\\text\{step\}\\leftarrow 0
3:for
t=1t=1to
TtrainT\_\{\\text\{train\}\}do
4:if
t≤Texploret\\leq T\_\{\\text\{explore\}\}then
5:
at∼𝒰\(𝒜\)a\_\{t\}\\sim\\mathcal\{U\}\(\\mathcal\{A\}\)
6:elseif
r\>0r\>0then
7:
at←𝐚∗\[step\]a\_\{t\}\\leftarrow\\mathbf\{a\}^\{\*\}\[\\text\{step\}\]
8:else
9:
at←πθ\(st\)\+ϵta\_\{t\}\\leftarrow\\pi\_\{\\theta\}\(s\_\{t\}\)\+\\epsilon\_\{t\}
10:endif
11:Execute
ata\_\{t\}, observe
st\+1,rt,dones\_\{t\+1\},r\_\{t\},\\text\{done\}
12:Store
\(st,at,rt,st\+1,done\)\(s\_\{t\},a\_\{t\},r\_\{t\},s\_\{t\+1\},\\text\{done\}\)in
ℳ\\mathcal\{M\}
13:
Re←Re\+rtR\_\{e\}\\leftarrow R\_\{e\}\+r\_\{t\},
step←step\+1\\text\{step\}\\leftarrow\\text\{step\}\+1
14:Update policy every
KKsteps
15:ifdone or truncatedthen
16:if
Re\>RmaxR\_\{e\}\>R\_\{\\text\{max\}\}then
17:
Rmax←ReR\_\{\\text\{max\}\}\\leftarrow R\_\{e\},
𝐚∗←\\mathbf\{a\}^\{\*\}\\leftarrowactions from episode
18:
r←RNr\\leftarrow R\_\{N\}
19:elseif
r\>0r\>0then
20:
r←r−1r\\leftarrow r\-1
21:endif
22:Reset:
s0←ℰ\.reset\(\)s\_\{0\}\\leftarrow\\mathcal\{E\}\.\\text\{reset\}\(\),
Re←0R\_\{e\}\\leftarrow 0,
step←0\\text\{step\}\\leftarrow 0
23:endif
24:endfor
The algorithm begins by initializing the policy, environment, replay buffer, and relevant parameters, such as the number of exploration stepsTexploreT\_\{\\text\{explore\}\}, total training stepsTtrainT\_\{\\text\{train\}\}, and repetition lengthRNR\_\{N\}\. During the exploration phase, actions are sampled uniformly from the action space to promote state diversity\. After the exploration phase, the policy’s actions are selected with added exploration noise\.
When a new episode achieves a total rewardReR\_\{e\}greater than the maximum episode rewardRmaxR\_\{\\text\{max\}\}, the sequence of actions from that episode is stored as𝐚∗\\mathbf\{a\}^\{\*\}\. This sequence is then repeated for the nextRNR\_\{N\}episodes\. Repetition ensures that the policy can consistently revisit and learn from the most successful trajectories discovered so far\. The variablerrtracks the number of remaining repeated episodes\.
At each step, the agent interacts with the environment, stores transitions in the replay buffer, and updates the policy at a fixed interval\. When an episode terminates, the algorithm checks whether the episode reward exceedsRmaxR\_\{\\text\{max\}\}and either updates the best sequence or decrements the repetition counter\. This process continues untilTtrainT\_\{\\text\{train\}\}steps are completed\.
RN0RN1RN2RN3RN4RN5RN6RN7
Figure 7:Impact of repetition number RN onIER\-SACperformance across all tasks, showing how repeated execution of the highest\-reward episode affects learning\. Curves show the mean episodic return over ten independent seeds and are smoothed using a sliding window of size 5 for visual clarity\.
RN0RN1RN2RN3RN4RN5RN6RN7
Figure 8:Impact of repetition number RN onIER\-TD3performance across all tasks, showing how repeated execution of the highest\-reward episode affects learning\. Curves show the mean episodic return over ten independent seeds and are smoothed using a sliding window of size 5 for visual clarity\.\(a\)Ant\-v4
\(b\)HalfCheetah\-v4
\(c\)Humanoid\-v4
\(d\)Hopper\-v4
\(e\)Walker\-Walk
\(f\)Cheetah\-Run
\(g\)Cartpole\-Swingup
\(h\)Finger\-Turn\-Hard
Figure 9:tasks used in our experiments\.
## Appendix B: Extended Analysis of the Repetition Number \(RN\)
This appendix provides additional experimental results analyzing the impact of the RN used in the IER algorithm on agent performance\.
Figures[7](https://arxiv.org/html/2608.17347#Sx1.F7)and[8](https://arxiv.org/html/2608.17347#Sx1.F8)present complete performance curves across all RN values and tasks, allowing for the comparison of performance trends under varying repetition frequencies\.
Across nearly all tasks, RN=0 consistently results in one of the weakest performances, confirming that learning without repetition underutilizes successful trajectories\. Even a small amount of repetition \(e\.g\., RN=1\) consistently improves performance, indicating that revisiting high\-reward episodes enhances learning efficiency\. Higher RNs, such as RN=6, continue to provide strong performance in many tasks; however, performance occasionally declines at RN=7, likely due to overexploitation or reduced diversity in the sampled experiences\.
##### Domain\-Level Analysis\.
Figures[10\(a\)](https://arxiv.org/html/2608.17347#Sx2.F10.sf1)and[10\(b\)](https://arxiv.org/html/2608.17347#Sx2.F10.sf2)separate MuJoCo and DeepMind Control tasks\.
In MuJoCo locomotion environments, improvements are greatest at moderate repetition levels, particularly under TD3\. In DeepMind Control tasks, repetition effects are smoother and less sensitive to RN magnitude\. This suggests that repetition interacts with environment dynamics, with higher\-dimensional locomotion tasks benefiting more strongly from moderate replay\.
##### Agreement Between Baseline Algorithms\.
To evaluate robustness beyond single\-task optima, we first identify, for each task and each baseline \(IER\-SAC and IER\-TD3\), the set of RN values whose normalized AUC is within 95% of that task’s best AUC\. We refer to these as the near\-optimal RN sets\.
An agreement is defined as selecting an RN who performs well for both baselines on the same task\. When the near\-optimal RN sets of SAC and TD3 overlap, we choose the RN from their intersection, i\.e\., an RN that is simultaneously near\-optimal for both algorithms\.
If no intersection exists, we apply a rank\-based criterion\. Specifically, RN values are ranked according to their normalized AUC under each baseline, and we select the RN with the best combined rank across SAC and TD3, ensuring a balanced compromise rather than favoring a single algorithm \(Table[1](https://arxiv.org/html/2608.17347#Sx2.T1)\)\.
Several tasks exhibit direct intersection\-based agreement, while others require rank\-based compromise selection\. Figure[11](https://arxiv.org/html/2608.17347#Sx2.F11)summarizes the frequency of selected agreement RN values across tasks\. RN2, RN3, and RN5 appear most frequently, indicating that moderate repetition levels tend to generalize across algorithms\.
The results suggest that moderate repetition provides the most reliable performance across tasks and baseline algorithms\. Very small RN values underutilize successful trajectories, whereas excessively large RN values yield diminishing returns\. Across MuJoCo and DeepMind Control environments, RN2–RN4 offers a stable and effective operating range for IER\.
\(a\)MeanΔ\\DeltaAUC% across MuJoCo tasks\. Moderate repetition \(RN3–RN4\) yields the strongest gains, particularly under TD3\.
\(b\)MeanΔ\\DeltaAUC% across DeepMind Control tasks\. Improvements are more uniform and less sensitive to RN magnitude\.
Figure 10:MeanΔ\\DeltaAUC% across environment groups\. Left: MuJoCo tasks\. Right: DeepMind Control tasks\.Table 1:Per\-task RN agreement based on 95% near\-best criterion\.Figure 11:Frequency of RN values selected as robust joint candidates across tasks\. RN2, RN3, and RN5 appear most frequently\.
## Appendix C: Environment Descriptions
Table 2:Summary of environments used in evaluation\.Table 3:Observation and action space dimensions and episode horizon for each environment\.ℝd\\mathbb\{R\}^\{d\}denotes add\-dimensional real vector;\[−1,1\]d\[\-1,1\]^\{d\}denotes a bounded action space\.We evaluate our approach across a diverse set of continuous control environments from MuJoCo[38](https://arxiv.org/html/2608.17347#bib.bib32)and the DeepMind Control Suite \(DMC\)[36](https://arxiv.org/html/2608.17347#bib.bib31), two widely adopted platforms for studying motor control in physically realistic simulations\. All tasks are configured with low\-dimensional, vector\-based observations consisting of joint positions, velocities, orientations, and other proprioceptive signals—excluding high\-dimensional visual input\.
##### MuJoCo Environments\.
MuJoCo tasks, accessed via the Gym interface, focus on high\-speed and dynamic locomotion with varying complexity:
- •Ant\-v4:A quadrupedal agent learns to walk using a high\-dimensional action space\. Coordination and contact dynamics make this task nontrivial\.
- •HalfCheetah\-v4:A planar biped with a flexible spine learns to run\. It offers dense, unbounded rewards, with cumulative rewards often exceeding 12,000, making it suitable for evaluating long\-horizon learning and experience prioritization\.
- •Humanoid\-v4:A high\-DOF biped must learn to walk upright\. Its instability and dimensionality pose significant challenges for balance, control, and exploration\.
- •Hopper\-v4:A single\-legged agent must learn to hop forward while maintaining balance\. The task requires precise control and stability, as failure to maintain upright posture results in early episode termination\. Its sensitivity to instability makes it useful for assessing learning robustness\.
Table 4:Real\-World Robotic Gripper Task Characteristics
##### DeepMind Control Suite Environments\.
DMC tasks are designed for general motor control using structured state vectors\. Unlike MuJoCo, rewards are typically bounded in\[0,1000\]\[0,1000\], focusing more on precision than accumulation:
- •Walker\-Walk:A biped learns forward locomotion while maintaining balance\. It involves moderately complex dynamics and fewer degrees of freedom than Humanoid\.
- •Cheetah\-Run:A planar cheetah learns forward locomotion under aggressive dynamics and bounded rewards\. Although structurally similar to HalfCheetah\-v4, it exhibits greater control sensitivity and performance saturation\. We deliberately include this task to evaluate robustness and generalization in environments with similar morphology but different reward scales and control challenges\.
- •Cartpole\-Swingup:An underactuated classic control problem requiring swing\-up and stabilization of a pole mounted on a cart\. Sparse rewards and nonlinear dynamics make learning nontrivial\.
- •Finger\-TurnHard:A single robotic finger must rotate an object to a target orientation\. Unlike the easy variant, it requires more precise torque control and sustained actuation under stricter success conditions\.
Figure 12:Workspace boundaries and cube tracking for reward evaluation\. Green and red markers indicate current and previous cube positions used to compute displacement\-based rewards\.Table 5:Hyperparameters used in training\. Shared parameters apply to both SAC and TD3; SAC, TD3, and SIL sections list algorithm\-specific settings\.
### Real\-World Robotic Task
This section describes the Dynamic Object Translation task involving gripper\-based manipulation, including the hardware setup, sensing modalities, and key challenges\.
The task, illustrated in Fig\.[12](https://arxiv.org/html/2608.17347#Sx3.F12), is a dexterous in\-hand manipulation challenge requiring the gripper to securely grasp a cube and manipulate it to a desired position within image space\. Unlike tasks leveraging extrinsic dexterity[5](https://arxiv.org/html/2608.17347#bib.bib39)or caging strategies, this setup demands sustained force closure to maintain grasp stability throughout the manipulation\.
This task builds upon a prior two\-finger push manipulation setup from[39](https://arxiv.org/html/2608.17347#bib.bib1), with a 90\-degree rotation along the x\-axis that orients the gripper downward\. This modification eliminates the supporting base beneath the cube and intensifies gravitational effects, forcing the gripper to develop stable grasp and finger manipulation strategies to avoid dropping the object\.
Success in this task is defined by continuously manipulating the cube, maximizing its displacement while maintaining a stable grasp\. The task pushes beyond simulation by involving intrinsic gripper dexterity in a physical environment, introducing real\-world challenges such as limited data collection and hardware constraints\.
Fig\.[12](https://arxiv.org/html/2608.17347#Sx3.F12)shows the workspace from a camera viewpoint\. The blue rectangle denotes the gripper’s operational area\. The cube’s current and previous positions are tracked using ArUco markers, visualized as green and red circles, respectively, providing intuitive feedback for monitoring and debugging\. The hardware combines aluminum extrusions and 3D\-printed joints, with a Logitech webcam capturing the scene\. Experiments run on a single PC, ensuring real\-time control\. The gripper hardware is built with consumer\-level components and extensive 3D printing, yielding a cost\-effective setup\. An automated reset mechanism is integrated to facilitate efficient data collection and consistent training\. A rack\-and\-pinion elevator lifts the cube to a fixed start position between the gripper’s fingertips at the end of each episode, enabling rapid and repeatable task restarts\. A detailed summary of the task’s characteristics is provided in Table[4](https://arxiv.org/html/2608.17347#Sx3.T4)\.
##### Reward Structure:
Rewards are computed based on two main criteria: \(1\) successful force\-closure grasping of the object, and \(2\) the magnitude of lateral displacement of the cube between consecutive time steps\. This reward structure encourages the agent to maintain grip stability while steadily moving the object over longer distances\.
##### State and Action Spaces:
Thestate spaceis a vector consisting of the positions of the servos, Aruco markers in the image \(xy\), and cube positions \(xy\)\. Similarly, the action space is represented as a vector containing the desired positions of each servo joint\. Each position value in this vector corresponds to a specific angular position for the Dynamixel servos\. The vector’s elements directly control the orientation and configuration of the gripper, allowing for precise manipulation of objects\.
##### Training Setup:
Agents are trained for 20,000 interaction steps\. Each episode comprises 20 real\-time steps, reflecting physical environment constraints\.
## Appendix D: Configuration and Hyperparameter Settings
This appendix provides the complete configuration and hyperparameter settings used in all experiments\. All agents were implemented in PyTorch, following architectures and optimization settings consistent with prior work[11](https://arxiv.org/html/2608.17347#bib.bib37);[30](https://arxiv.org/html/2608.17347#bib.bib38)\. To ensure fair comparisons, all tasks were trained under the same configuration unless specified otherwise\. Experiments in the main study were averaged over five random seeds, while the ablation study on RN used ten seeds to improve statistical reliability\.
Table[5](https://arxiv.org/html/2608.17347#Sx3.T5)summarizes the hyperparameters used in training\. Shared parameters apply to both SAC and TD3, while algorithm\-specific settings for SAC, TD3, and the SIL extension are listed separately\. These configurations were kept consistent across tasks to isolate the effects of algorithmic modifications, such as varying the RN or incorporating intrinsic motivation mechanisms\.Similar Articles
Hindsight Experience Replay
OpenAI presents Hindsight Experience Replay (HER), a technique enabling sample-efficient reinforcement learning from sparse binary rewards without complex reward engineering. It is demonstrated on robotic arm manipulation tasks including pushing, sliding, and pick-and-place, and validated on physical robots.
Replay What Matters: Off-Policy Replay for Efficient LLM Reinforcement Unlearning
This paper introduces ReRULE, an off-policy replay method for reinforcement unlearning in LLMs, improving forgetting and retention efficiency on benchmarks like RWKU and MUSE.
Integrating Novelty and Surprise for Experience Prioritization and Exploration in Image-Based Reinforcement Learning
This paper introduces Novelty and Surprise Prioritized Experience Replay (NSPER) to enhance sample efficiency in image-based reinforcement learning by combining novelty and surprise for experience prioritization and exploration.
Freshness-Aware Prioritized Experience Replay for LLM/VLM Reinforcement Learning
FreshPER introduces a freshness-aware prioritized experience replay method for LLM/VLM reinforcement learning that addresses the 'priority staleness' problem by applying exponential age decay to stored priorities, enabling off-policy reuse of trajectories. Evaluated on eight agentic, reasoning, and math tasks, FreshPER significantly outperforms on-policy baselines with gains up to +367% on Sokoban.
Neurosymbolic Reasoning with Incremental Knowledge for Sample Efficient Hierarchical Reinforcement Learning
This paper proposes InK, a neurosymbolic hierarchical reinforcement learning approach that uses incremental knowledge for symbolic planning and reward-shaped low-level neural modules, achieving improved sample efficiency in navigation tasks.