Quasi-Monte Carlo Initialization for Meta-Reinforcement Learning

# Quasi-Monte Carlo Initialization for Meta-Reinforcement Learning
Source: [https://arxiv.org/html/2607.21637](https://arxiv.org/html/2607.21637)
###### Abstract\.

This paper explores the efficacy of quasi\-Monte Carlo \(QMC\) weight initialization for meta\-reinforcement learning within modern benchmark environments\. Various sampling methods are used to bound a population\-based search and aggregate an optimal prior from a baseline set of tasks\. The QMC meta\-priorsθi∗\\theta\_\{i\}^\{\*\}show improvements in training convergence compared to modern orthogonal \(SB3\) defaults when extrapolated to similar unseen continuous control environments\. In dissimilar tasks, the orthogonal orientation was globally superior for an unbiased search\.

## 1\.Introduction

Reinforcement learning \(RL\) is highly influenced by the model’s initial conditions\(Kwoket al\.,[2025](https://arxiv.org/html/2607.21637#bib.bib1)\)\. This study uses various quasi\-Monte Carlo \(QMC\) sampling methodsii\(Fig\.[1](https://arxiv.org/html/2607.21637#S1.F1)\): Sobol, Latin Hypercube \(LHS\), and Hyperellipsoid Density Sampling \(HDS\)\(Soltes,[2026](https://arxiv.org/html/2607.21637#bib.bib3)\), to create both Euclidean and non\-Euclidean search spacesΩi\\Omega\_\{i\}from which to sample initial model weightsθi∈Ωi\\theta\_\{i\}\\in\\Omega\_\{i\}\. An efficient one\-step training sweep through this sample population identifies an optimal meta\-priorθi∗\\theta^\{\*\}\_\{i\}to accelerate learning when applied to similar unseen environments\.

Quasi\-Monte Carlo sequences are commonly explored for one\-shot optimization problems\(Bosseket al\.,[2020](https://arxiv.org/html/2607.21637#bib.bib2)\)\. Extrapolating to meta\-reinforcement learning, the results of this study show that the exploitative HDS, low\-discrepancy Sobol, and high\-stratification LHS meta\-priors \(θH∗\\theta\_\{H\}^\{\*\},θS∗\\theta^\{\*\}\_\{S\},θL∗\\theta^\{\*\}\_\{L\}\) improve training performance when initializing unseen environments with similar kinematic profiles\.

![Refer to caption](https://arxiv.org/html/2607.21637v1/samples.png)Figure 1\.Representative 2\-D sample populations\.
## 2\.Methodology

The experimental setup is configured using modern RL architectures \(section[2\.1](https://arxiv.org/html/2607.21637#S2.SS1)\)\. The optimal meta\-prior \(θi∗\\theta\_\{i\}^\{\*\}\) is identified and evaluated via one\-step adaptation \([2\.2](https://arxiv.org/html/2607.21637#S2.SS2)\) and zero\-shot transfer \([2\.3](https://arxiv.org/html/2607.21637#S2.SS3)\)\.

### 2\.1\.Experimental Setup

The experimental trials are conducted using proximal policy optimization \(PPO\) viaStableBaseline3\(SB3\)\(Raffinet al\.,[2021](https://arxiv.org/html/2607.21637#bib.bib4)\)on variousGymnasium\(Towerset al\.,[2024](https://arxiv.org/html/2607.21637#bib.bib6)\)environments\. Quasi\-Monte Carlo samples were generated usinghdim\_optv1\.4\.71\. All trials are bootstrapped over 16 randomly\-seeded iterations to achieve statistically significant results \(p<0\.05p<0\.05\) via the rank\-sum test\. Each configuration’s random seed is defined relative to the experimental trial iteration to ensure reproducibility\.

#### 2\.1\.1\.RL Architecture

The actor and critic networks are configured as multi\-layer perceptrons consisting of two 32\-neuron layers to prevent over\- or under\-fitting to the initial search environments; for generalizability across tasks, the initial sample weights are injected exclusively into these networks, bypassing the environment\-specific output layers\. The injected sample weights are internally and dynamically scaled via LeCun / Xavier initialization \(∝1/D\\propto 1/\\sqrt\{D\}\)\(Glorot and Bengio,[2010](https://arxiv.org/html/2607.21637#bib.bib5)\)\.

The observation spaces are padded to 27 dimensions \(the total position and velocity states within Gymnasium’s Ant\-v5\) to account for varying observation shapes between environments\. This relatively low dimensionality provides the additional benefit of reducing network overfitting to the one\-step search environment\(s\)\.

### 2\.2\.One\-Step Adaptation

A weight populationN=210N=2^\{10\}was generated using HDS, Sobol, LHS, and random baselines \(Fig\.[1](https://arxiv.org/html/2607.21637#S1.F1)\)\. Each sample was injected into the policy and value network torsos and evaluated within three baseline search environments: HalfCheetah\-v5, BipedalWalker\-v3, and Hopper\-v5\.

The PPO is trained for a single timestep within each search environment, using Huber loss \(smoothL1L\_\{1\}\) and gradient norm clipping \(\|\|g\|\|≤21\.0\|\{\}\|\{\}g\|\{\}\|\{\}\_\{2\}\\leq 1\.0\) to stabilize the stochastic gradient descent update with an explorative learning rate of 0\.01\. The resulting adaptation score𝒜\\mathcal\{A\}is quantified as the Euclidean shift in the action distribution mean, Z\-score normalized and averaged across the three search environments\. The sample weightθi\\theta\_\{i\}with the highest score𝒜\+\\mathcal\{A\}^\{\+\}is extracted as the optimal meta\-priorθi∗\\theta\_\{i\}^\{\*\}for zero\-shot transfer, tested at various sample sizesN∈\{22,…,210\}N\\in\\\{2^\{2\},\\dots,2^\{10\}\\\}\(Fig\.[2](https://arxiv.org/html/2607.21637#S2.F2)\) to visualize𝒜\+\\mathcal\{A\}^\{\+\}as a function ofNN\.

![Refer to caption](https://arxiv.org/html/2607.21637v1/initial_adaptation.png)Figure 2\.Best adaptation scores𝒜\+\\mathcal\{A\}^\{\+\}using various population sizes\.
### 2\.3\.Zero\-Shot Transfer

The PPO was injected with the optimal meta\-priorθi∗\\theta\_\{i\}^\{\*\}and trained for10610^\{6\}timesteps in three similar \(and two dissimilar\) unseen continuous control environments: Ant\-v5, Walker2d\-v5, BipedalWalkerHardcore\-v3, Swimmer\-v5, and LunarLander\-v3\. The latter two are included to evaluate theθi∗\\theta\_\{i\}^\{\*\}on tasks with dissimilar profiles to the search environments\. The results are compared to training initialized with the SB3 PPO’s defaults \(’Orthogonal’ and ’Random’\)\.

## 3\.Results

The results confirm the hypothesis that the QMC meta\-priors improve performance on environments with similar kinematic profiles, while showing negative effects on dissimilar tasks\.

The global converged training performance is aggregated across all evaluation environments and shown in Table[1](https://arxiv.org/html/2607.21637#S3.T1), where thepp\-value represents the difference in performance compared to the default SB3 Orthogonal\.

Table 1\.Zero\-shot transfer performance, Z\-normalized across all evaluation environments\.pp\-values are calculated relative to the SB3 Orthogonal baseline\.### 3\.1\.Similar Tasks

The similar tasks saw improvements in performance forθi∗\\theta\_\{i\}^\{\*\}compared to SB3 Random and Orthogonal initializations, with varying levels of statistical significance \(Table[1](https://arxiv.org/html/2607.21637#S3.T1)\)\. SB3 Orthogonal performed lowest compared to all methods on Ant\-v5; its best task was Walker2d, where it was beat only, and insignificantly, by SobolθS∗\\theta\_\{S\}^\{\*\}\(p≈0\.87p\\approx 0\.87\)\. While all QMC meta\-priors globally out\-performed SB3 Orthogonal, only SobolθS∗\\theta\_\{S\}^\{\*\}showed statistically significant \(p<0\.05p<0\.05\) improvements\.

![Refer to caption](https://arxiv.org/html/2607.21637v1/global_boxplot_similar.png)Figure 3\.Aggregated performance of the meta\-priors in similar environments\.
### 3\.2\.Dissimilar Tasks

The dissimilar tasks measured global decreases in performance forθi∗\\theta\_\{i\}^\{\*\}compared to SB3 Orthogonal, where the mathematically unbiased initial orientation proved superior within foreign environments \(p<0\.001p<0\.001\)\. The uniform QMC meta\-priors \(SobolθS∗\\theta\_\{S\}^\{\*\}and LHSθL∗\\theta\_\{L\}^\{\*\}\) retain higher performance in these environments compared to the non\-Euclidean HDSθH∗\\theta\_\{H\}^\{\*\}due to less overfitting\.

![Refer to caption](https://arxiv.org/html/2607.21637v1/global_boxplot_dissimilar.png)Figure 4\.Aggregated converged performance of the meta\-priors in dissimilar environments\.

## 4\.Conclusion

The results validate quasi\-Monte Carlo sampling methods as effective search geometries for zero\-shot weight initialization in meta\-reinforcement learning problems\. The optimal meta\-priorsθi∗\\theta\_\{i\}^\{\*\}identified from the efficient one\-step search are found to accelerate training when initializing similar unseen environments\. For dissimilar tasks, the unbiased geometry of SB3 Orthogonal initialization showed globally superior performance\.

###### Acknowledgements\.

The author would like to thank Dr\. Mike Busch for the continued instruction on reinforcement learning architectures\. Monet is a source of inspiration, as always\.

## Appendix

![Refer to caption](https://arxiv.org/html/2607.21637v1/combined_results_similar.png)Figure 5\.Per\-environment performance of the meta\-priors in similar environments\.![Refer to caption](https://arxiv.org/html/2607.21637v1/combined_results_dissimilar.png)Figure 6\.Per\-environment performance of the meta\-priors in dissimilar environments\.
## References

- J\. Bossek, C\. Doerr, P\. Kerschke, A\. Neumann, and F\. Neumann \(2020\)Evolving sampling strategies for one\-shot optimization tasks\.InParallel Problem Solving from Nature – PPSN XVI,T\. Bäck, M\. Preuss, A\. Deutz, H\. Wang, C\. Doerr, M\. Emmerich, and H\. Trautmann \(Eds\.\),Cham,pp\. 111–124\.External Links:ISBN 978\-3\-030\-58112\-1Cited by:[§1](https://arxiv.org/html/2607.21637#S1.p2.3)\.
- X\. Glorot and Y\. Bengio \(2010\)Understanding the difficulty of training deep feedforward neural networks\.InProceedings of the thirteenth international conference on artificial intelligence and statistics,pp\. 249–256\.Cited by:[§2\.1\.1](https://arxiv.org/html/2607.21637#S2.SS1.SSS1.p1.1)\.
- D\. Kwok, G\. S\. Altıntaş, C\. Raffel, and D\. Rolnick \(2025\)The butterfly effect: neural network training trajectories are highly sensitive to initial conditions\.InInternational Conference on Machine Learning,Note:arXiv:2506\.13234External Links:[Link](https://doi.org/10.48550/arXiv.2506.13234)Cited by:[§1](https://arxiv.org/html/2607.21637#S1.p1.4)\.
- A\. Raffin, A\. Hill, A\. Gleave, A\. Kanervisto, M\. Ernestus, and N\. Dormann \(2021\)Stable\-baselines3: reliable reinforcement learning implementations\.Journal of Machine Learning Research22\(268\),pp\. 1–8\.External Links:[Link](http://jmlr.org/papers/v22/20-1364.html)Cited by:[§2\.1](https://arxiv.org/html/2607.21637#S2.SS1.p1.1)\.
- J\. G\. Soltes \(2026\)Hyperellipsoid density sampling: exploitative sequences to accelerate high\-dimensional numerical optimization\.External Links:2511\.07836,[Link](https://arxiv.org/abs/2511.07836)Cited by:[§1](https://arxiv.org/html/2607.21637#S1.p1.4)\.
- M\. Towers, J\. Terry, J\. Kwiatkowski, J\. U\. Balis, M\. de Cola, T\. Deleu, M\. Gouläo, A\. Kallinteris, A\. Krimmel, A\. KG, R\. Perez\-Hidalgo, and A\. Rassledov \(2024\)Gymnasium: a standard interface for reinforcement learning environments\.arXiv preprint arXiv:2407\.17032\.Cited by:[§2\.1](https://arxiv.org/html/2607.21637#S2.SS1.p1.1)\.