V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Control

# Unleashing the Architectural Potential of RL in Visual Continuous Control
Source: [https://arxiv.org/html/2608.07870](https://arxiv.org/html/2608.07870)
V\-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Control

Donghu Kim, Youngdo Lee, Hojoon Lee, Johan Obando\-Ceron,

Byungkun Lee, Aaron Courville, Pablo Samuel Castro,

Jaegul Choo, Clare Lyle

Keywords:Deep RL, Visual RL, Neural Network Design, Plasticity, Normalization

SummaryImproving sample efficiency remains a core challenge in reinforcement learning \(RL\), especially in real\-world settings like robotics, where data collection is costly\. This challenge is pronounced in visual RL, where high\-dimensional inputs often obscure learning signals\. While prior work in visual RL has focused on algorithmic solutions, such as better dynamics models or exploration strategies, recent advances in state\-based RL show that architectural design alone can lead to significant gains in sample efficiency\. This raises an important question:Can these architectural principles transfer to visual RL?In response, we introduceV\-Simba, a simple yet effective visual RL architecture inspired by the Simba architecture from state\-based RL\. Built on top of Soft Actor\-Critic \(SAC\) with data augmentation, V\-Simba modifies the architecture by adding normalization layers to stabilize training and using pointwise convolutions to reduce computation\. Despite its simplicity, V\-Simba matches or outperforms the state\-of\-the\-art methods across the DMC, Adroit, and Meta\-World benchmarks, while being more computationally efficient than DrQ\-v2\.

Contribution\(s\)1\.We identify and analyze severe training instabilities such as sharp loss landscapes, dormant units, and feature collapse that are inherent in standard convolutional architectures widely adopted in visual reinforcement learning\. Context:While previous visual RL research has primarily focused on algorithmic innovations, the underlying neural architectures have largely remained simple, often following the standard DrQ\-v2 baseline\. Our analysis reveals the vulnerabilities and capacity loss associated with these heavily utilized architectures\.2\.We propose V\-Simba, a novel visual RL architecture integrating normalization layers, weight regularization, and a distributional critic to stabilize training, alongside large\-stride and pointwise convolutions to maintain computational efficiency\. Context:Recent state\-based models have improved learning by constraining parameters and feature growth, but translating this to pixel\-based environments is hindered by the severe computational bottlenecks of high\-dimensional observations\. V\-Simba addresses this gap by combining targeted normalization to stabilize optimization with spatial downsampling to remain lightweight and computationally feasible for visual tasks\.3\.We experimentally demonstrate that V\-Simba matches or outperforms state\-of\-the\-art visual RL methods across 29 tasks in the DeepMind Control Suite, Adroit, and Meta\-World benchmarks\. Context:We evaluate V\-Simba against strong algorithmic baselines including DrQ\-v2, MR\.Q, TD\-MPC2, DrM, and TACO, showing that robust architectural design alone can achieve superior sample and compute efficiency without introducing complex algorithmic add\-ons\.

###### Abstract

Improving sample efficiency remains a core challenge in reinforcement learning \(RL\), especially in real\-world settings like robotics, where data collection is costly\. This challenge is pronounced in visual RL, where high\-dimensional inputs often obscure learning signals\. While prior work in visual RL has focused on algorithmic solutions, such as better dynamics models or exploration strategies, recent advances in state\-based RL show that architectural design alone can lead to significant gains in sample efficiency\. This raises an important question:Can these architectural principles transfer to visual RL?In response, we introduceV\-Simba, a simple yet effective visual RL architecture inspired by the Simba architecture from state\-based RL\. Built on top of Soft Actor\-Critic \(SAC\) with data augmentation, V\-Simba modifies the architecture by adding normalization layers to stabilize training and using pointwise convolutions to reduce computation\. Despite its simplicity, V\-Simba matches or outperforms the state\-of\-the\-art methods across the DMC, Adroit, and Meta\-World benchmarks, while being more computationally efficient than DrQ\-v2\.We make our code publicly available at[https://github\.com/DAVIAN\-Robotics/V\-Simba](https://github.com/DAVIAN-Robotics/V-Simba)\.

![Refer to caption](https://arxiv.org/html/2608.07870v1/x1.png)Figure 1:Benchmark Summary\.We evaluate the effectiveness of V\-Simba across 29 visual continuous control tasks spanning multiple domains, with asingleset of hyperparameters\. By incorporating V\-Simba into Soft Actor\-Critic with data augmentation, it matches or outperforms state\-of\-the\-art visual RL methods, demonstrating better sample and compute efficiency\.## 1Introduction

Deep reinforcement learning \(RL\) has long been a prominent approach for solving continuous control tasks\. However, RL typically relies on an extensive amount of trial\-and\-error within the environment, which is often expensive in terms of time, compute, and real\-world constraints\. This issue is exacerbated in visual RL, where agents must learn from high\-dimensional, noisy, and often partially observable image inputs\. Consequently, improving sample efficiency \(i\.e\., learning effectively from limited interaction data\) has become a key research topic in visual RL\.

To improve sample efficiency, recent work has largely concentrated on algorithmic innovations, including enhanced representation learning\(yarats2021sacae;echchahed2025survey;obando2026simplicial\), latent dynamics modeling\(fujimoto2025mrq;zheng2023taco\), world models\(hansen2023tdmpcv2;hafner2023dreamerv3\), and improved exploration strategies\(burda2018rnd;xu2023drm\)\. Yet, despite these advances, the underlying neural architectures have remained relatively simple\. A prominent example is DrQ\-v2\(yarats2021drqv2\), which combines the DDPG algorithm\(lillicrap2015ddpg\)with data augmentation\(laskin2020rad\)\. Its architecture consists of a shallow convolutional encoder, followed by a large fully connected layer and a single layer normalization layer\(lei2016layer\_norm\)in\-between\. Due to its simplicity and strong empirical performance, DrQ\-v2 has become the de facto standard in visual RL, and many state\-of\-the\-art methods\(xu2023drm;zheng2023taco;cetin2022alix;sukhija2024maxinforl\)adopt DrQ\-v2’s architecture with minimal modifications\.

![Refer to caption](https://arxiv.org/html/2608.07870v1/x2.png)Figure 2:DrQ\-v2 vs\. V\-Simba\.Comparison of the DrQ\-v2 architecture and our V\-Simba in theDog Standenvironment\. Both are evaluated on Soft Actor\-Critic \(SAC\) with results averaged over 5 seeds\.\(a\)V\-Simba is substantially more efficient than DrQ\-v2 in terms of both sample and compute\.\(b\)Unlike DrQ\-v2, V\-Simba has stable learning dynamics, indicated by a smooth loss landscape, a low dormant ratio, high feature diversity, and well\-controlled feature, parameter, and gradient norms\. Detailed explanations of each metric are provided in Appendix[7\.2](https://arxiv.org/html/2608.07870#S7.SS2)\.However, our analysis reveals that this commonly adopted architecture suffers from severe training instabilities\. As shown in the top row of Figure[2](https://arxiv.org/html/2608.07870#S1.F2)\(b\), DrQ\-v2 exhibits sharp loss landscapes that correlate with poor generalization\(foret2020sharpness;lee2024plastic\), a high fraction of dormant units indicating plasticity loss\(sokar2023redo;liu2026measure\), and low feature diversity indicating feature collapse111We use these metrics as diagnostic probes that surface the failure modes our architecture is designed to address, rather than as direct predictors of final return; we discuss their scope and limitations in Appendix[7](https://arxiv.org/html/2608.07870#S7)\.\(woo2023convnextv2\)\.

In contrast, recent advances in state\-based RL\(lee2024simba;bhatt2024crossq;lee2025simbav2;palenicek2025scaling;castanyer2025stable\)demonstrate that carefully designed architectures can effectively mitigate these instabilities\. Notably, the Simba series of architectures\(lee2024simba;lee2025simbav2\)introduce principled architectural guidelines that stabilize training by constraining the growth of features, weights, and gradients through targeted normalization and regularization\.

While effective in state\-based tasks, the Simba architecture lacks a suitable inductive bias for visual data\. Convolutional architectures in RL face distinct challenges, such as severe plasticity loss and capacity degradation driven by unconstrained parameter growth across spatial hierarchies\(lyle2024disentangling\)\. Furthermore, directly applying extensive normalization to high\-resolution convolutional feature maps introduces prohibitive computational bottlenecks\.

In response, we proposeV\-Simba, an architecture specifically tailored to overcome these visual RL challenges\. Built on top of Soft Actor\-Critic \(SAC\)\(haarnoja2018sac\), V\-Simba incorporates three core architectural components: \(1\) layer normalization \(LN\) to control feature norms, \(2\)ℓ2\\ell\_\{2\}weight regularization to limit parameter growth, and \(3\) a distributional critic with reward normalization to stabilize gradients\. Crucially, to make these stabilizing mechanisms computationally viable for high\-dimensional image inputs, V\-Simba employs aggressive early spatial reduction, alongside extensive use of lightweight pointwise convolutions\(hua2018pointwise\)and max\-pooling operations\(krizhevsky2012imagenet\)\.

We evaluate V\-Simba on three standard benchmarks: DMControl\(tassa2018dmc\), Adroit\(rajeswaran2017adroit\), and Meta\-World\(yu2020metaworld\)using a single set of hyperparameters across all tasks\. Despite its simplicity, V\-Simba significantly outperforms DrQ\-v2, while reducing both model size \(7\.2M → 5\.0M\) and training time \(5\.4 → 4\.8 hours for 1M DMControl steps\)\. Most notably, V\-Simba outperforms highly complex algorithmic methods like DrM\(xu2023drm\)across all benchmarks, as well as MR\.Q\(fujimoto2025mrq\), TD\-MPC2\(hansen2023tdmpcv2\)and TACO\(zheng2023taco\)\.

V\-Simba is intended to offer a strong, stable, and efficient architectural foundation for advancing visual continuous control\. We hope our work highlights the untapped potential of principled architecture design within the visual RL community\.

## 2Related Work

Learning solely from high\-dimensional visual observations poses significant challenges in RL\. Due to the partially observable nature of the observation space \(Section[3\.1](https://arxiv.org/html/2608.07870#S3.SS1)\), visual RL agents suffer from poor sample efficiency and large generalization gaps compared to their state\-based counterparts\(ma2022visualrlaug\)\.

Algorithmic approaches for visual RLhave primarily focused on: \(1\) representation learning via auxiliary tasks predicting future latent states, either as auxiliary losses in model\-free methods\(stooke2021atc;zheng2023taco;schwarzer2020spr;schwarzer2021sgi;kim2022self;lee2020stochastic;van2016stable;yu2022mask;gelada2019deepmdp;seo2022reinforcement;ni2024bridging;yu2021playvirtual;fujimoto2021deep;mcinroe2021learning;fujimoto2025mrq;echchahed2025survey;obando2026simplicial;pasand2026stable\)or separate dynamics models in model\-based methods\(hansen2023tdmpcv2;hafner2023dreamerv3;lin2025tdmpcsquared;wu2023daydreamer;ha2018world;finn2016deep;watter2015embed\); \(2\) data augmentation, especially random shifts\(laskin2020rad;kostrikov2020drq;yarats2021drqv2\), for better efficiency and generalization; and \(3\) exploration methods, including planning\(sekar2020planning;wang2023coplanner\), curiosity\-driven\(pathak2017curiosity;burda2018exploration;guo2022byolexplore\), and information maximization\(sukhija2024maxinforl\)\. These algorithmic innovations have driven rapid progress in visual RL, leading to continuous improvements in sample efficiency\.

Architectural design for visual RLhas received comparatively little attention compared to algorithmic innovations\. The field has largely maintained shallow convolutional neural network \(CNN\) architectures similar to the one established by DQN\(mnih2015dqn\)over a decade ago\. While some works have incorporated architectural elements from computer vision—such as ResNet\-like architectures in Impala\(espeholt2018impala\), BBF\(schwarzer2023bbf\), and EfficientZero\(ye2021mastering\), or transformers in DTQN\(esslinger2022deep\)—these modifications were often introduced alongside complex algorithmic methods\. This entanglement has obscured the true contribution of architectural design to performance improvements\. Recent studies have identified the benefits of normalization techniques\(lyle2023understanding\_plasticity;ball2023efficient;lyle2024normalization\), but were generally applied to conventional CNN encoders with minimal architectural modifications\.

Studies that do revisit visual encoders, such as global average pooling\(trumpp2025impoola;sokar2025mindthegap\), Hadamard max\-pooling encoders\(kooi2025hadamax\), or Mixture\-of\-Experts\(obando2024mixtures;willi2024mixture;sokar2024don\), have largely targeted discrete\-action, Atari\-style domains\. This mirrors a broader resurgence of architectural and representational design across RL, including high\-capacity categorical value functions\(nauman2025brc\), dynamic network expansion for scaling continuous control\(kang2025fog;liu2025neuroplastic\), constraining initial representations to stabilize temporal\-difference learning\(lyu2026constrained\), and reinitialization schemes that balance the stability\-plasticity tradeoff\(han2026fire\)\. In visual continuous control, however, architectural innovation remains scarce\(espeholt2018impala;huang2024mentor\)\.

## 3Preliminary

### 3\.1Visual Reinforcement Learning

Reinforcement learning \(RL\) is typically formulated as a Markov Decision Process \(MDP\)\(bellman1957markovian\), defined by the tuple\(𝒮,𝒜,P,R,γ\)\(\\mathcal\{S\},\\mathcal\{A\},P,R,\\gamma\)of state space𝒮\\mathcal\{S\}, action space𝒜\\mathcal\{A\}, transition functionP:𝒮×𝒜→𝒫​\(𝒮\)P:\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\mathcal\{P\}\(\\mathcal\{S\}\), reward functionR:𝒮×𝒜→ℝR:\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\mathbb\{R\}, and discount factorγ∈\[0,1\)\\gamma\\in\[0,1\)\. From an initial states0∈𝒮s\_\{0\}\\in\\mathcal\{S\}, the objective is to find an optimal policyπ∗:𝒮→𝒫​\(𝒜\)\\pi^\{\*\}:\\mathcal\{S\}\\to\\mathcal\{P\}\(\\mathcal\{A\}\)that maximizes the expected discounted return𝔼π​\[∑t=0∞γt​R​\(st,at\)\]\\mathbb\{E\}\_\{\\pi\}\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}R\(s\_\{t\},a\_\{t\}\)\]\. Visual RL is a subclass of this problem where the agent does not have access to the true states∈𝒮s\\in\\mathcal\{S\}, but instead receives high\-dimensional pixel observationso∈𝒪o\\in\\mathcal\{O\}of the system\. Since these observations may not fully capture the true state, the problem is modeled as a Partially Observable Markov Decision Process \(POMDP\)\(bellman1957markovian\)represented by the tuple\(𝒮,𝒪,𝒜,P,R,γ\)\(\\mathcal\{S\},\\mathcal\{O\},\\mathcal\{A\},P,R,\\gamma\), where𝒪\\mathcal\{O\}denotes the observation space\.

### 3\.2Data\-regularized Q\-learning

Data\-regularized Q\-learning \(DrQ\-v2\)\(yarats2021drqv2\)is a model\-free RL algorithm that has emerged as a strong baseline in visual RL due to its simplicity, efficiency, and competitive performance\. It builds upon the Deep Deterministic Policy Gradient \(DDPG\) algorithm\(lillicrap2015ddpg\)by incorporating two key modifications: \(1\) extensive use of data augmentation via random shift transformations, and \(2\) target Q\-function stabilization through exponential moving average \(EMA\) updates\.

At its core, DrQ\-v2 improves sample efficiency in off\-policy learning by generating augmented views of each observation, thereby increasing data diversity\. This augmentation acts as a regularizer, mitigating overfitting to specific visual patterns\. Despite its empirical effectiveness, DrQ\-v2 employs a notably lightweight architecture: a shallow convolutional encoder followed by an MLP\-based prediction head, with a single normalization layer\(lei2016layer\_norm\)in between\.

While DrQ\-v2 has become the de facto architecture for many recent visual RL algorithms\(xu2023drm;zheng2023taco\), its architectural simplicity leaves room for improvement in stability and representational capacity\.

### 3\.3Soft Actor\-Critic \(SAC\)

Soft Actor\-Critic \(SAC\) is a prominent off\-policy algorithm for continuous control\. It aims to maximize both expected cumulative reward and policy entropy, whereτ=\(o,a,r,o′\)\\tau=\(o,a,r,o^\{\\prime\}\)represents a transition tuple\. SAC comprises a stochastic policyπθ​\(a\|o\)\\pi\_\{\\theta\}\(a\|o\), a Q\-functionQϕ​\(o,a\)Q\_\{\\phi\}\(o,a\), and an entropy coefficientα\\alphathat balances reward maximization and entropy regularization\. The policy network is optimized to maximize the expected return while encouraging exploration through entropy\. This objective is formalized as:

ℒπ=𝔼a¯∼πθ​\[α​log⁡πθ​\(a¯\|o\)−Qϕ​\(o,a¯\)\]\.\\mathcal\{L\}\_\{\\pi\}=\\mathbb\{E\}\_\{\\bar\{a\}\\sim\\pi\_\{\\theta\}\}\\left\[\\alpha\\log\\pi\_\{\\theta\}\(\\bar\{a\}\|o\)\-Q\_\{\\phi\}\(o,\\bar\{a\}\)\\right\]\.\(1\)
The Q\-functionQϕ​\(o,a\)Q\_\{\\phi\}\(o,a\)is trained to minimize the Bellman residual:

ℒQ=\(Qϕ​\(o,a\)−\(r\+γ​Qϕ¯​\(o′,a′\)−α​log⁡πθ​\(a′\|o′\)\)\)2,\\mathcal\{L\}\_\{Q\}=\(Q\_\{\\phi\}\(o,a\)\-\\left\(r\+\\gamma Q\_\{\\bar\{\\phi\}\}\(o^\{\\prime\},a^\{\\prime\}\)\-\\alpha\\log\\pi\_\{\\theta\}\(a^\{\\prime\}\|o^\{\\prime\}\)\\right\)\)^\{2\},\(2\)wherea′∼πθ\(⋅\|o′\)a^\{\\prime\}\\sim\\pi\_\{\\theta\}\(\\cdot\|o^\{\\prime\}\), andQϕ¯Q\_\{\\bar\{\\phi\}\}represents the target Q\-network updated via an exponential moving average ofϕ\\phi\.

## 4Method

V\-Simba leverages architectural design from state\-based RL to stabilize optimization dynamics and improve computational efficiency in visual RL\. Our design follows two core principles: \(1\) stabilizing optimization \(Section[4\.1](https://arxiv.org/html/2608.07870#S4.SS1)\), and \(2\) maintaining computational efficiency \(Section[4\.2](https://arxiv.org/html/2608.07870#S4.SS2)\)\. The final architecture builds on these principles \(Section[4\.3](https://arxiv.org/html/2608.07870#S4.SS3)\)\.

### 4\.1Design Philosophy I: Stabilizing Optimization

As shown in Figure[2](https://arxiv.org/html/2608.07870#S1.F2), DrQ\-v2 suffers from unstable optimization during training\. While layer normalization \(LN\)\(lei2016layer\_norm\)and residual connections\(he2020why\_resnet\)effectively stabilize supervised learning, visual RL methods often underuse them—DrQ\-v2, for instance, employs only a single normalization layer without residuals\. We incorporate both components to improve stability\.

However, when adding LayerNorm, one must consider its relationship with the gradient\. Concretely, LayerNorm introduces scale invariance: for any scalarc\>0c\>0and weight matrixWW,

Norm​\(c​W​x\)=Norm​\(W​x\),\\text\{Norm\}\(cWx\)=\\text\{Norm\}\(Wx\),\(3\)which causes gradients to scale inversely with parameter magnitude:

∇WNorm​\(c​W​x\)=1c​∇WNorm​\(W​x\)\.\\nabla\_\{W\}\\text\{Norm\}\(cWx\)=\\frac\{1\}\{c\}\\nabla\_\{W\}\\text\{Norm\}\(Wx\)\.\(4\)As parameter norms grow during training, gradients diminish, reducing learning ability\(lyle2024normalization;ceron2024in;palenicek2025scaling;mayor2025the;castanyer2025stable\)\. Moreover, uneven growth across layers causes inconsistent gradient scales, destabilizing optimization\(lee2025simbav2\)\. This highlights the importance of controlling weight and gradient norms, in addition to the feature norm\. Thus, we employ the following design choices to achieve stable norms\.

We first opt LayerNorm as the forefront layer of both encoder and critic module, in order to control the norm of not only their intermediate features but also their inputs\. While unusual for convolutional networks, this resembles the Dual PatchNorm design\(kumar2023dual\)which has been empirically shown to stabilize the gradients of the embedding layer222In practice, we adopt the shift\-and\-norm strategy introduced in SimbaV2\(lee2025simbav2\)to preserve magnitude information\. We useℓ2\\ell\_\{2\}\-norm for action inputs however, as when\|𝒜\|=1\|\\mathcal\{A\}\|=1shift\-and\-LN always outputs\[−1,1\]\[\-1,1\]\.\. For preventing parameter growth, we surprisingly found a simpleℓ2\\ell\_\{2\}weight regularization to be sufficient, as shown in Figure[2](https://arxiv.org/html/2608.07870#S1.F2)\.

We further stabilize gradients by employing a distributional critic with KL divergence loss\(bellemare2017distributional\)and reward normalization\(lee2025simbav2\)\. The KL divergence loss is more robust to noisy targets than mean squared error due to its smoother loss landscape\(farebrother2024stop\), while reward normalization ensures consistent learning signals despite varying reward scales\.

Specifically, reward normalization maintains unit variance in expected returns\. Given rewardrtr\_\{t\}at timett, we track the discounted return:

Gt←γ​Gt−1\+rtG\_\{t\}\\leftarrow\\gamma G\_\{t\-1\}\+r\_\{t\}\(5\)withGtG\_\{t\}re\-initialized to0at the start of each episode\. Letσt,G2\\sigma^\{2\}\_\{t,G\}denote the running variance ofGtG\_\{t\}\. Each reward is then scaled as:

r¯t←rtσt,G2\+ϵ,\\bar\{r\}\_\{t\}\\leftarrow\\frac\{r\_\{t\}\}\{\\sqrt\{\\sigma\_\{t,G\}^\{2\}\+\\epsilon\}\},\(6\)

### 4\.2Design Philosophy II: Maintaining Computational Efficiency

Adding normalization layers and regularization increases training cost, so reducing computation is crucial\. We find that most of DrQ\-v2’s computational cost comes from early convolutional layers processing high\-resolution inputs\. We apply early downsampling via large\-stride convolutions, a common practice in ResNet\(he2016deep\), ConvNeXt\(liu2022convnext\), and Vision Transformer\(dosovitskiy2020image\)\. This results in an early reduction in spatial resolution and in turn, the computational cost of subsequent convolution layers\.

We further cut computation by replacing convolutional layers into more cost\-effective alternatives\. For spatial convolutions, we instead utilize pointwise \(1×11\\times 1kernel\) convolutions\(hua2018pointwise\), which operate channel\-wise without mixing spatial information, preserving spatial details at a lower cost\. For downsampling, we adopt parameter\-free2×22\\times 2max\-pooling\(krizhevsky2012imagenet\)in place of strided convolutions, which retain the strongest local activations while inducing significantly less overhead in both training and inference\. Empirically, we found that these substitutions do not meaningfully alter the learning dynamics or the learning curves, indicating that the computational savings come at no cost to stability\.

![Refer to caption](https://arxiv.org/html/2608.07870v1/x3.png)Figure 3:V\-Simba architecture\.We aim to develop an architecture that constrains its feature norm, weight norm, and gradient norm for better stability and generalization\. Precisely, we make extensive use of layer normalization and residual connections for stable features and gradients, and incorporate point\-wise convolutions in tandem with spatial convolutions for computational efficiency\.
### 4\.3V\-Simba Architecture

Building on our design principles, we now detail the V\-Simba architecture \(Figure[3](https://arxiv.org/html/2608.07870#S4.F3)\)\.

Image Preprocessing\.The inputo∈ℝ84×84×9o\\in\\mathbb\{R\}^\{84\\times 84\\times 9\}is a stack of the last three RGB frames\. We first apply an alternating sequence of normalization and reduction steps\. Specifically, the input passes through an initial LayerNorm, followed by a3×33\\times 3convolution \(stride 2\), a second LayerNorm, and finally a2×22\\times 2max\-pooling layer \(stride 2\)\. This results in downsampled and normalized featuresf0∈ℝ21×21×32f\_\{0\}\\in\\mathbb\{R\}^\{21\\times 21\\times 32\}\.

Encoder\.The encoder consists of two sequential blocks transforming and downsampling features:f0→Block1f1→Block2f2f\_\{0\}\\xrightarrow\{\\text\{Block\}\_\{1\}\}f\_\{1\}\\xrightarrow\{\\text\{Block\}\_\{2\}\}f\_\{2\}, wheref1∈ℝ10×10×32,f2∈ℝ5×5×32\.f\_\{1\}\\in\\mathbb\{R\}^\{10\\times 10\\times 32\},f\_\{2\}\\in\\mathbb\{R\}^\{5\\times 5\\times 32\}\.

Each encoder block processes inputfif\_\{i\}as follows:

1. 1\.A3×33\\times 3convolution to aggregate spatial features without changing resolution\.
2. 2\.Feature normalization \(LayerNorm\) to provide stable features to the subsequent layers\.
3. 3\.Two pointwise \(1×11\\times 1\) convolutions with nonlinearities to refine and filter features, followed by a residual connection for stable gradient flow\. Here, we employ an inverted bottleneck with 4×\\timesexpansion, following ConvNext\(liu2022convnext\)and Simba\(lee2024simba\),
4. 4\.Downsampling with a2×22\\times 2maxpool layer with stride 2 to reduce spatial resolution\.
5. 5\.Applying LayerNorm to normalize the features before passing to the next block\.

After the second block,f2f\_\{2\}is flattened into the latent state vectorz∈ℝ800z\\in\\mathbb\{R\}^\{800\}\.

Predictor\.The latent vectorzzfeeds into separate actor and critic heads, each followed by a linear layer and LayerNorm\. For the critic, actions are separately embedded and concatenated with image embedding\. The actor and critic embeddings have dimensionszπ∈ℝ128z\_\{\\pi\}\\in\\mathbb\{R\}^\{128\},zQ∈ℝ512z\_\{Q\}\\in\\mathbb\{R\}^\{512\}followinglee2024simba\.

Each embedding passes through residual nonlinear blocks: one block for the actor and two for the critic\. Finally, the actor output passes through LayerNorm, a linear layer, and atanh\\tanhactivation, while the critic output passes through LayerNorm and a linear layer modeling the Q\-value distribution\.

![Refer to caption](https://arxiv.org/html/2608.07870v1/figures/environments/dog.png)
![Refer to caption](https://arxiv.org/html/2608.07870v1/figures/environments/humanoid.png)

\(a\)DeepMind Control Suite
![Refer to caption](https://arxiv.org/html/2608.07870v1/figures/environments/hammer.png)
![Refer to caption](https://arxiv.org/html/2608.07870v1/figures/environments/pen.png)

\(b\)Adroit
![Refer to caption](https://arxiv.org/html/2608.07870v1/figures/environments/assembly.png)
![Refer to caption](https://arxiv.org/html/2608.07870v1/figures/environments/shelf-place.png)

\(c\)Meta\-World

Figure 4:Environment Visualization\.We evaluate our V\-Simba on 3 visual continuous control benchmarks: DeepMind Control Suite\(tassa2018dmc\), Adroit\(rajeswaran2017adroit\), and Meta\-World\(yu2020metaworld\)\.

## 5Experiments

We now provide an empirical evaluation of V\-Simba:

1. 1\.Performance Evaluation\(Sections[5\.2](https://arxiv.org/html/2608.07870#S5.SS2)\), comparing V\-Simba against leading visual RL methods to demonstrate its effectiveness across diverse benchmarks\.
2. 2\.Ablation Study\(Section[5\.3](https://arxiv.org/html/2608.07870#S5.SS3)\), conducting experiments demonstrating the contribution of each architectural component in V\-Simba\.

### 5\.1Experimental Setup

Environment\.We consider a total of 29 continuous control tasks spanning 3 benchmarks: DeepMind Control \(DMC\) Suite\(tassa2018dmc\), Adroit\(rajeswaran2017adroit\), and Meta\-World\(yu2020metaworld\)\. Figure[4](https://arxiv.org/html/2608.07870#S4.F4)shows the visualization of each task\. These environments pose diverse challenges, including high\-dimensional action spaces, sparse rewards, and complex dexterous manipulation, often under rich visual observations with shading and textures\. Consequently, to solve the tasks, prior visual RL methods typically require either large volumes of frames or privileged information such as low\-level robot states\. Comprehensive descriptions of the benchmarks are provided in Appendix[11](https://arxiv.org/html/2608.07870#S11)\.

Baselines\.In experiments, we compare V\-Simba against a diverse set of state\-of\-the\-art visual RL methods exemplifying three key algorithmic strategies: data and model regularization \(DrQ\-v2\(yarats2021drqv2\), A\-LIX\(cetin2022alix\)\), advanced exploration \(DrM\(xu2023drm\)\), and model\-based representation learning \(TACO\(zheng2023taco\), TD\-MPC2\(hansen2023tdmpcv2\), MR\.Q\(fujimoto2025mrq\)\)\. Notably, A\-LIX, TACO, and DrM build upon DrQ\-v2 \(see Section[3\.2](https://arxiv.org/html/2608.07870#S3.SS2)\): A\-LIX stabilizes training by adaptively regularizing the encoder’s gradients; TACO leverages a latent dynamics loss for richer representations; and DrM integrates dormant ratio\(sokar2023redo\)\-guided mechanisms that balance exploration\-exploitation dynamically\. While these variants benefit from task\-specific hyperparameter tuning, our method uses thesamehyperparameters across all tasks \(see Appendix[8](https://arxiv.org/html/2608.07870#S8)for the complete hyperparameter list\)\. Whenever possible, we report original paper results; otherwise, we run the authors’ official implementations\.

### 5\.2Performance Evaluation

DMC Medium\.We begin by evaluating V\-Simba on DMC Medium, consisting of 11 mid\-difficulty tasks from DMC\. As shown in Figure[6](https://arxiv.org/html/2608.07870#S5.F6), our base algorithm, DrQ\-v2, falls behind model\-based methods such as TD\-MPC2 and MR\.Q\. However, simply replacing DrQ\-v2’s neural network with our proposed architecture, V\-Simba, yields substantial performance gains\. As a result, V\-Simba surpasses TD\-MPC2 and achieves results competitive with leading algorithm, MR\.Q, highlighting the impact of architectural improvements\.

DMC Hard\.We further assess V\-Simba on DMC Hard, a set of 7 high\-difficulty tasks in DMC, characterized by complex kinematics and high\-dimensional control\. Figure[5](https://arxiv.org/html/2608.07870#S5.F5)shows that V\-Simba performs competitively with MR\.Q, though full task success remains elusive\. We believe that this observation suggests that concurrent advances in both algorithm design and architectural representation are needed in visual RL to close the gap with state\-based performance\.

![Refer to caption](https://arxiv.org/html/2608.07870v1/x4.png)Figure 5:DeepMind Control Suite \- Hard\.Average episode returns on 7 hard\-level tasks from DeepMind Control Suite\(tassa2018dmc\)\. Each curve represents the mean performance across 5 random seeds per algorithm; shaded areas indicate 95% bootstrap confidence intervals\.![Refer to caption](https://arxiv.org/html/2608.07870v1/x5.png)Figure 6:DeepMind Control Suite \- Medium\.Average episode returns on 11 medium\-level tasks from DeepMind Control Suite\(tassa2018dmc\)\. Each curve represents the mean performance across 5 random seeds per algorithm; shaded areas indicate 95% bootstrap confidence intervals\.![Refer to caption](https://arxiv.org/html/2608.07870v1/x6.png)Figure 7:Adroit \- Sparse\.Average success rates on 3 sparse\-reward tasks from Adroit\(rajeswaran2017adroit\)\. Each curve represents the mean performance across 5 random seeds per algorithm; shaded areas indicate 95% bootstrap confidence intervals\.![Refer to caption](https://arxiv.org/html/2608.07870v1/x7.png)Figure 8:Meta\-World\.Average success rates on 8 tasks from the Meta\-World\(yu2020metaworld\)\. Each curve represents the mean performance across 5 random seeds per algorithm; shaded areas indicate 95% bootstrap confidence intervals\.Adroit \- Sparse\.Moving to more intricate scenarios, we evaluate V\-Simba on Adroit under the challengingsparse\-rewardsetting\. In this domain, the agent must control a dexterous hand\-arm system to perform complex manipulation such as opening a door or using tools like a hammer\. These tasks pose significant challenges for visual RL, often requiring over 5 million environment frames and access to privileged robot state inputs for successful learning\. The comparison results are shown in Figure[7](https://arxiv.org/html/2608.07870#S5.F7)\. V\-Simba reliably solves or nearly solves all tasks using only 1 million frames\. In contrast, DrM–the previous state\-of\-the\-art method–fails to learn meaningful behavior, despite leveraging privileged state vectors\. Notably, V\-Simba is the only method to solveHammerwith 1 million environment steps\. These results underscore V\-Simba’s strong sample\-efficiency and effectiveness in high\-dimensional visual control settings\.

Meta\-World\.We also benchmark V\-Simba on Meta\-World, which demands precise object manipulation\. We consider 4 medium\-difficulty tasks:Coffee Push,Soccer,Sweep Into, andHammer, and 4 high\-difficulty tasks:Assembly,Stick Pull,Pick Place Wall, andDisassemble\. For the medium tasks, we adopt a sparse\-reward setting by replacing the ground\-truth reward functions with binary success signals, followingyu2020metaworld, to increase task difficulty\. As shown in Figure[8](https://arxiv.org/html/2608.07870#S5.F8), while DrQ\-v2 struggles to learn in most tasks, V\-Simba significantly improves over DrQ\-v2 and matches or surpasses leading baselines, demonstrating superior sample efficiency\. A notable performance improvement can be seen in theDisassembleandHammertasks, where V\-Simba was able to consistently achieve almost perfect success rate, whereas prior works have failed in few trials\.

Table 1:Ablation Study\.We exclude or modify each component in V\-Simba and report their final performance on each benchmark, averaged over 3 random seeds\. Each cell is highlighted base on their relative percentile difference to V\-Simba, namely: positive\(\>0\.01\)\(\>0\.01\), mildly negative\[−0\.05,−0\.01\)\[\-0\.05,\-0\.01\), damaging\[−0\.1,−0\.05\)\[\-0\.1,\-0\.05\), and catastrophic\[−1\.0,−0\.1\)\[\-1\.0,\-0\.1\)\.
### 5\.3Ablation Study

To assess the impact of each component on V‑Simba’s overall performance, we evaluate variants that remove or modify one component at a time\. The results are reported in Table[1](https://arxiv.org/html/2608.07870#S5.T1)\.

We first investigate the effect of normalization layers \(Table[1](https://arxiv.org/html/2608.07870#S5.T1)\.\(a\)\-\(c\)\)\.No Normalization Layersremoves LayerNorm entirely from the network, whereasNo Input Normalizationonly removes two LayerNorms: for image and action inputs in encoder and critic respectively\.LN w/oγ,β\\gamma,\\betaremoves the bias and scale parameters of LayerNorm\. In summary, by removing certain normalization layers or components, the network loses control over the feature norms, leading to degradation\.

Next, we quantify the importance of residual connections and weight decay \(Table[1](https://arxiv.org/html/2608.07870#S5.T1)\.\(d\)\-\(e\)\)\. Both residual connections and weight decay, along with their well\-known benefits, are also known to bias the network towards simple solutions for improved robustness\(teney2024neuralredshift;lee2024simba\)\. Removing such components led to a visible drop in performance, similar to removing normalization layers\.

Finally, the categorical critic and reward scaling are critical components, as they reformulate the regression problem into a categorical prediction, giving a much more stable gradient and learning dynamics\. Reverting back to regression loss led to diminished performance \(Table[1](https://arxiv.org/html/2608.07870#S5.T1)\.\(f\)\)\. Even with categorical loss, leaving no bounds to the reward scales led to similar consequences \(Table[1](https://arxiv.org/html/2608.07870#S5.T1)\.\(g\)\), highlighting the importance of assuring the Q\-values to stay in a certain range\.

## 6Lessons and Opportunities

In this work, we introduce V\-Simba, a simple yet effective neural network architecture for visual continuous control, inspired by the Simba architecture from state\-based RL\(lee2024simba\)\. By combining feature normalization, weight regularization, and a distributional critic, V\-Simba achieves superior performance over prior visual RL methods across multiple benchmarks with minimal algorithmic changes\. Additionally, V\-Simba reduces computational cost by integrating early downsampling through large\-stride convolutions and pointwise convolution layers, enabling faster training than DrQ\-v2\(yarats2021drqv2\)\. We believe our work does not oppose the current trend of adopting model\-based learning or exploration strategies; rather, it offers a complementary approach that can be integrated with subsequent studies\.

Regardless, several opportunities remain\. Because V\-Simba constrains the growth of features, weights, and gradients, it offers a natural substrate for scaling to larger widths and depths, a promising avenue we leave to future work\. Seed variance can also be large on a few tasks \(e\.g\.,Adroit\-Hammer\), though comparable spread arises across visual\-RL methods rather than being specific to V\-Simba, and reducing it remains important\. Finally, our evaluation protocol focuses on evaluating wide range of environments with a single hyperparameter set with 5 seeds; more seeds or longer horizons would further sharpen these estimates\.

In recent years, reinforcement learning for robotic control has gained increased attention\. However, limited sample efficiency remains a significant barrier to real\-world adoption\. While simulators provide valuable virtual environments\(makoviychuk2021isaac;zakka2025mujoco\), rendering high\-resolution images with complex object interactions is still computationally expensive and difficult to parallelize\. This underscores the importance of improving sample efficiency\. V\-Simba offers a lightweight architectural solution using well\-established components that are easy to integrate into existing algorithms\. Its simplicity allows practitioners to adopt and extend it with minimal overhead\. We hope V\-Simba serves as an architectural foundation to accelerate progress in the robotics community\.

#### Acknowledgments

This work was supported by Institute for Information & Communications Technology Planning & Evaluation\(IITP\) grant funded by the Korea government\(MSIT\) \(RS\-2019\-II190075, Artificial Intelligence Graduate School Program\(KAIST\)\)\. This work was supported by the National Research Foundation of Korea\(NRF\) grant funded by the Korea government\(MSIT\) \(No\. RS\-2025\-00555621\)\. This research was supported by the “Advanced GPU Utilization Support Program” funded by the Government of the Republic of Korea \(Ministry of Science and ICT\)\.

Pablo Samuel Castro acknowledges funding from NSERC Discovery Grant\. Aaron Courville acknowledge funding from National Sciences and Engineering Council of Canada \(NSERC\) and the Canadian Institute for Advanced Research \(CIFAR\)\. Johan Obando\-Ceron acknowledge funding support from Google and CIFAR AI\. We would also like to thank the Python community\(van1995python;4160250\)for developing tools that enabled this work, including NumPy\(harris2020array\), Matplotlib\(hunter2007matplotlib\), Jupyter\(2016ppap\), and Pandas\(McKinney2013Python\)\.

## References

Supplementary Materials

*The following content was not necessarily subject to peer review\.*

## 7Extended Metric Analysis

This section provides the details and extended results of the metric analysis presented in Figure[2](https://arxiv.org/html/2608.07870#S1.F2)\.

### 7\.1Setup

Our goal is to analyze and compare neural network architectures in visual RL in terms of learning dynamics and stability\. We use two baseline algorithms—DDPGlillicrap2015ddpgand SAChaarnoja2018sac—both with data augmentationyarats2021drqv2\. We then evaluate four neural architectures proposed by DrQ\-v2yarats2021drqv2, IMPALAespeholt2018impala, MR\.Qfujimoto2025mrqand our proposed V\-Simba on each algorithm\. For IMPALA, we only employ the encoder with residual blocks, combined with DrQ\-v2 predictors\. For MR\.Q, we exclude the dynamics learning components and learn the encoder and critic end\-to\-end \(dubbed ‘no MR’ in their ablation experiments\)\.

We follow the original paper for any architecture\-specific hyperparameters such as the number of layers, hidden dimension, and the use of clipped double Q\-learning \(CDQ\)fujimoto2018td3\. Otherwise, we use the same set of hyperparameters for all experiments\. We measure the metrics \(Section[7\.2](https://arxiv.org/html/2608.07870#S7.SS2)\) every 10,000 update steps \(20,000 environment steps\), using a mini\-batch of size 256\.

### 7\.2Metrics

We employ the following metrics for analysis:

Sharpness of the loss landscape\.Sharpness is often considered indicative of a neural network’s ability to generalize\. In reinforcement learning, the underlying data distribution is inherently non\-stationary, making consistent generalization crucial\. We quantify sharpness by the largest eigenvalue of the Hessian matrix \(λmax​\(∇2ℒ\)\\lambda\_\{\\max\}\(\\nabla^\{2\}\\mathcal\{L\}\)\)lee2024plastic;foret2020sharpness, which can be approximated using the Lanczos algorithm\(golub1996lanczos\)\.

Dormant ratio\.A neuron is said to be inactive ordormantwhen its absolute activation value tends to be small compared to the layer’s average\. Formally, theii\-th neuron of layerℓ\\ellisτ\\tau\-dormant ifsiℓ=𝔼x∈D​\|hiℓ​\(x\)\|1Hℓ​∑k∈h𝔼x∈D​\|hiℓ​\(x\)\|≤τs^\{\\ell\}\_\{i\}=\\frac\{\\mathbb\{E\}\_\{x\\in D\}\|h^\{\\ell\}\_\{i\}\(x\)\|\}\{\\frac\{1\}\{H^\{\\ell\}\}\\sum\_\{k\\in h\}\\mathbb\{E\}\_\{x\\in D\}\|h^\{\\ell\}\_\{i\}\(x\)\|\}\\leq\\tau, wherehiℓh^\{\\ell\}\_\{i\}are the activation values of layerℓ\\ellsokar2023redo\. A high proportion of dormant neurons implies that the network’s decisions rely heavily on only a few neurons, indicating capacity loss\. We useτ=0\.1\\tau=0\.1in our analysis\.

Feature diversity\.While maximizing feature diversity itself might not be crucial for RL, preventing feature collapse is critical, as it reduces the network’s capacity and hampers learning capability\. Inspired by ConvNext\-v2woo2023convnextv2, we measure the average cosine distance between the samples within a batch:1B2​∑iB∑jB1−c​o​s​\(Xi,Xj\)2\\frac\{1\}\{B^\{2\}\}\\sum^\{B\}\_\{i\}\\sum^\{B\}\_\{j\}\\frac\{1\-cos\(X\_\{i\},X\_\{j\}\)\}\{2\}, whereBBis the batch size, andX∈ℝB×DX\\in\\mathbb\{R\}^\{B\\times D\}is the feature matrix\.

Norms\.Prior works in state\-based RL have shown that controlling the growth of features, weights and gradient norms can stabilize the learning process and thus performancelee2024simba;lee2025simbav2;palenicek2025scaling\. We investigate whether the same argument could be made for visual RL as well\. Followinglee2025simbav2, we define theeffectivenorm of a set of vectors and matrices using dimension\-based weightswi​\(z\)=dim​\(zi\)∑j=1Ndim​\(zj\)w\_\{i\}\(z\)=\\frac\{\\mathrm\{dim\}\(z\_\{i\}\)\}\{\\sum\_\{j=1\}^\{N\}\\mathrm\{dim\}\(z\_\{j\}\)\}, which captures dimensional contributions across vectors and matrices\. For example, for a neural network’s parameter setθ=\{θi\}i=1N\\theta=\\\{\\theta\_\{i\}\\\}\_\{i=1\}^\{N\}, the effective parameter norm is defined as‖θ‖eff2≜∑i=1Nwi​\(θ\)​‖θi‖2\\\|\\theta\\\|\_\{\\mathrm\{eff\}\}^\{2\}\\triangleq\\sum\_\{i=1\}^\{N\}w\_\{i\}\(\\theta\)\\\|\\theta\_\{i\}\\\|\_\{2\}where∥⋅∥22\\\|\\cdot\\\|\_\{2\}^\{2\}denotes the standardℓ2\\ell\_\{2\}\-norm \(or Frobenius norm∥⋅∥F\\\|\\cdot\\\|\_\{F\}for matrices\)\.

### 7\.3Results

We visualize the results for DDPG in Figure[9](https://arxiv.org/html/2608.07870#S7.F9), and SAC in Figure[10](https://arxiv.org/html/2608.07870#S7.F10)\. DrQ\-v2 and IMPALA architectures exhibit significant instability across both algorithms, showing high sharpness and dormant ratio, low feature diversity, and exploding feature, parameter and gradient norms\. Collectively, these issues hinder the learning process and their capacity to learn meaningful behaviors\.

Meanwhile, MR\.Q maintains better stability by incorporating numerous normalization layers into its design\. Notably, MR\.Q achieves lower dormant ratios and higher feature diversity compared to V\-Simba, highlighting the importance of normalization layers in stabilizing learning dynamics\. Despite the strengths, MR\.Q still experiences relatively high sharpness and norm magnitudes, although their growth is better controlled\.

Finally, our proposed V\-Simba further enhances stability by rigorously controlling the norm scales\. This leads to smoother loss landscape, low dormant ratio, high feature diversity, and overall superior performance compared to all other evaluated methods\.

We stress that these metrics are diagnostic rather than predictive: no single metric cleanly orders methods by return\. MR\.Q, for instance, matches or exceeds V\-Simba on dormant ratio and feature diversity, yet attains a lower average return, meaning low dormancy or high diversity alone is not sufficient for strong performance\. Among the metrics we track, sharpness aligns most consistently with the observed performance ordering, and MR\.Q’s comparatively high sharpness may partially account for its gap to V\-Simba, in line with the established connection between sharpness and generalization\(foret2020sharpness;lee2024plastic\)\. Accordingly, we treat these metrics as complementary lenses on optimization stability that motivate our architectural choices, rather than as standalone predictors of performance\.

![Refer to caption](https://arxiv.org/html/2608.07870v1/x8.png)Figure 9:Comparison of Neural Architectures under DDPG\.We evaluate and compare the neural network architectures proposed by DrQ\-v2, IMPALA, MR\.Q and our V\-Simba, using DDPG with data augmentation in theDog Standenvironment\. V\-Simba maintains greater stability throughout training and outperforms other baselines\.![Refer to caption](https://arxiv.org/html/2608.07870v1/x9.png)Figure 10:Comparison of Neural Architectures under SAC\.We evaluate and compare the neural network architectures proposed by DrQ\-v2, IMPALA, MR\.Q and our V\-Simba, using SAC with data augmentation in theDog Standenvironment\. V\-Simba maintains greater stability throughout training and outperforms other baselines\.

## 8Hyperparameters

Table[2](https://arxiv.org/html/2608.07870#S8.T2)lists the hyperparameters used across all experiments\. Unless otherwise specified, we follow the default hyperparameters from prior work for consistency and computational efficiency, despite the well\-known sensitivity of deep RL agents to these choices\(ceron2024on\)\.

Table 2:Hyperparameters Table\.We use the consistent hyperparameters across all benchmarks, which are listed below\. The discount factorγ\\gammais automatically determined by heuristics from\(hansen2023tdmpcv2\)\.HyperparameterNotationValueCommonDiscount factorγ\\gammaHeuristic\(hansen2023tdmpcv2\)Replay buffer capacity\-11MBuffer sampling\-UniformBatch size\-256256Observation Shape\|𝒪\|\|\\mathcal\{O\}\|3×84×843\\times 84\\times 84Update\-to\-data \(UTD\) ratio\-11TD stepsnn33EncoderNumber of blocksLL22Hidden dimension \(channels\)dhd\_\{h\}3232Predictor \- ActorNumber of blocksLL1Hidden dimensiondhd\_\{h\}128Initial temperatureα0\\alpha\_\{0\}1​e−21\\mathrm\{e\}\{\-2\}Target entropyℍ∗\\mathbb\{H\}^\{\*\}\|𝒜\|/2\|\\mathcal\{A\}\|/2Predictor \- CriticNumber of blocksLL2Hidden dimensiondhd\_\{h\}512Number of atomsnatomsn\_\{\\text\{atoms\}\}101Target critic momentumτ\\tau5​e−35\\mathrm\{e\}\{\-3\}Clipped double Q\-NoAction embedding dimensiondad\_\{a\}128OutputNumber of return binsnatomsn\_\{\\text\{atoms\}\}101101Support of return\[Gmin,Gmax\]\[G\_\{\\min\},G\_\{\\max\}\]\[−5,5\]\[\-5,5\]Reward scaler epsilonϵ\\epsilon1​e−81\\mathrm\{e\}\{\-8\}OptimizerOptimizer\-AdamOptimizer momentum\(β1,β2\)\(\\beta\_\{1\},\\beta\_\{2\}\)\(0\.9, 0\.999\)Weight Decay\-1e−2\\mathrm\{e\}\{\-2\}Learning rateη\\eta1​e−41\\mathrm\{e\}\{\-4\}

## 9Compute Resources

We mainly use RTX3090 GPUs in our experiments, which takes approximately4\.84\.8hours to finish a single seed experiment with V\-Simba\. We have optimized the replay buffer to be more memory\-friendly, requiring around 11GB of RAM memory for each experiment\.

## 10Baselines

DrQ\-v2\(yarats2021drqv2\)\. Data\-regularized Q\-learning \(DrQ\-v2\) incorporates data augmentation via random shift transformations into DDPG to avoid overfitting of visual encoder to specific visual patterns\. We provide a detailed explanation of the algorithm in Section[3\.2](https://arxiv.org/html/2608.07870#S3.SS2)\. Results for DMC Medium and Meta\-World were obtained from\(yarats2021drqv2\)and\(xu2023drm\), respectively\. We run the official repository \([https://github\.com/facebookresearch/drqv2](https://github.com/facebookresearch/drqv2)\) over 3 random seeds for Adroit, and 5 random seeds for DMC Hard results\.

A\-LIX\(cetin2022alix\)\. Adaptive Local Signal Mixing \(A\-LIX\) modifies the convolutional layer of DrQ\-v2 by performing bilinear interpolation with weights derived from random shifts, regularizing gradients and reducing overfitting\. Meta\-World results are from\(xu2023drm\), which are averaged over 4 random seeds\. We obtained Adroit results by running the official repository \([https://github\.com/Aladoro/Stabilizing\-Off\-Policy\-RL](https://github.com/Aladoro/Stabilizing-Off-Policy-RL)\) over 3 seeds\.

TACO\(zheng2023taco\)\. Temporal Action\-driven Contrastive Learning \(TACO\) jointly learns state and action representations introducing contrastive learning to DrQ\-v2, which promotes to generalize its knowledge more effectively across diverse state\-action pairs, enhancing the sample efficiency of RL algorithms\. Meta\-World results are from\(xu2023drm\), which are averaged over 4 random seeds\. We obtained Adroit results by running the official repository \([https://github\.com/FrankZheng2022/TACO](https://github.com/FrankZheng2022/TACO)\) over 3 seeds\.

DrM\(xu2023drm\)\. Dormant Ratio Minimization \(DrM\) extends DrQ\-v2 with three mechanisms that reduce the agent’s dormant ratio and leverage it to balance exploration and exploitation\. Meta\-World results are from\(xu2023drm\), which are averaged over 4 random seeds\. We obtained Adroit results by running the official repository \([https://github\.com/XuGW\-Kevin/DrM](https://github.com/XuGW-Kevin/DrM)\) over 3 seeds\.

DreamerV3\(hafner2023dreamerv3\)\. DreamerV3 builds a latent world model by encoding the observation into a compact latent space for long\-horizon behavior and value learning\. DMC results are from\(fujimoto2025mrq\), reproduced with the official codebase \([https://github\.com/danijar/dreamerv3](https://github.com/danijar/dreamerv3)\) over 10 seeds\.

TD\-MPC2\(hansen2023tdmpcv2\)\. TD\-MPC2 learns a decoder\-free world model via multi\-task dynamics prediction and performs latent\-space planning\. DMC results are from\(fujimoto2025mrq\), reproduced with the official codebase \([https://github\.com/nicklashansen/tdmpc2](https://github.com/nicklashansen/tdmpc2)\) over 10 seeds\.

MR\.Q\(fujimoto2025mrq\)\. Model\-based Representations for Q\-learning \(MR\.Q\) is a model\-free algorithm that leverages model\-based auxiliary tasks to learn rich actor\-critic representations\. DMC results are from\(fujimoto2025mrq\), averaged over 10 seeds\.

## 11Environment Details

This section describes the benchmark environments used in our evaluation\. A complete list of tasks, including state and action dimensions, is provided at the end of the section\. Although state vectors are not used during training, we report them to reflect task difficulty\. Table[3](https://arxiv.org/html/2608.07870#S11.T3)details episode length, frame stack, action repeat, total environment steps, and performance metrics\. Figure[4](https://arxiv.org/html/2608.07870#S4.F4)shows visualizations of each environment\.

### 11\.1DeepMind Control Suite

The DeepMind Control Suite\(tassa2018dmc, DMC\)is a standard benchmark for continuous control benchmarks with varying levels of complexity\. Tasks in this benchmark range from simple low\-dimensional \(𝒮∈ℝ3\\mathcal\{S\}\\in\\mathbb\{R\}^\{3\},𝒜∈ℝ1\\mathcal\{A\}\\in\\mathbb\{R\}^\{1\}\) to highly complex continuous control \(𝒮∈ℝ223\\mathcal\{S\}\\in\\mathbb\{R\}^\{223\},𝒜∈ℝ38\\mathcal\{A\}\\in\\mathbb\{R\}^\{38\}\)\. We evaluate 18 tasks, grouped into DMC Medium and DMC Hard\. DMC Easy tasks are excluded due to their low difficulty\. Full task lists appear in Tables[4](https://arxiv.org/html/2608.07870#S11.T4.24)and[5](https://arxiv.org/html/2608.07870#S11.T5.16)\.

### 11\.2Adroit

Adroit\(rajeswaran2017adroit\)comprises dexterous manipulation tasks involving in\-hand manipulation, tool use, and articulated object control, performed using a 24 degree\-of\-freedom \(DoF\) anthropomorphic Shadow Hand\. Due to the tasks’ complexity, the state\-of\-the\-art method DrM\(xu2023drm\)uses a privileged robot sensor vector alongside image observations \(see official code:[https://github\.com/XuGW\-Kevin/DrM](https://github.com/XuGW-Kevin/DrM)\)\. In contrast, we do not use privileged information\. To increase difficulty, we also benchmark under sparse reward settings\. The full task list is provided in Table[6](https://arxiv.org/html/2608.07870#S11.T6.8)\.

### 11\.3Meta\-World

Meta\-World\(yu2020metaworld\)consists of 50 diverse robotic manipulation tasks using a simulated 7\-DoF Sawyer arm in a tabletop setting\. Following\(xu2023drm\), we select 8 tasks spanning object interaction, tool use, and precise motion control to cover a range of manipulation challenges\. For the easier half, we replace the ground\-truth dense reward function with a binary task completion signal \(i\.e\. asparse task completion reward\) and mark them assparse\. For details on the success metric, we refer the reader to\(yu2020metaworld\)\. Full task list is provided in Table[7](https://arxiv.org/html/2608.07870#S11.T7.18)\.

Table 3:Environment details\.We list the episode length, frame stack, action repeat for each domain, total environment steps, and performance metrics used for benchmarking\.Table 4:DMC Medium Task List\.We evaluate 11 tasks from the DMC Medium benchmark, listed below\. Performance for each task is reported at 1M environment steps\.Table 5:DMC Hard Task List\.We evaluate 7 tasks from DMC Hard benchmark, listed below\. Performance for each task is reported at 1M environment steps\.Table 6:Adroit Task List\.We evaluate 3 tasks from Adroit benchmark, listed below\. Performance for each task is reported at 1M environment steps\.Table 7:Meta\-World Task List\.We evaluate 8 tasks from Meta\-World benchmark, listed below\. Performance for each task is reported at 1M environment steps\.