JumpStart Your Policy Learning with Lessons from 160,000 Training Runs

# JumpStart Your Policy Learning with Lessons from 160,000 Training Runs
Source: [https://arxiv.org/html/2609.13730](https://arxiv.org/html/2609.13730)
Eric BaeAffiliation:University of WashingtonChung Yik Edward YeungAffiliation:University of WashingtonSiddhartha SenAffiliation:MicrosoftAli FarhadiAffiliation:University of WashingtonAffiliation:Microsoft

###### Abstract

Reliable progress in offline policy learning depends on careful reporting, well\-tuned baselines, and evaluation across diverse conditions\. Prior work has shown that results can be sensitive to reporting choices, hyperparameter tuning, and dataset properties, but these sources of variability have not been systematically investigated together at the scale needed to understand how they shape conclusions\. To address this gap, we present a large\-scale empirical study of offline reinforcement and imitation learning, training over 160,000 policies across 114 datasets\. At this scale, no algorithm dominates: aggregate performance among the strongest methods is often close, but the leaders differ substantially across environments\. We find that proper hyperparameter tuning frequently reshuffles perceived algorithm rankings and that benchmark composition can produce conflicting conclusions\. We also study hyperparameter sensitivity and transfer across environments, identifying a simple strategy for deriving strong default configurations\. We use our findings to develop a dataset\-conditioned recommender that provides task\-specific algorithm recommendations for practitioners\. Finally, we release JumpStart: a resource suite containing every trained policy, per\-model scores and hyperparameters, strong baselines across all environments, training and evaluation code, and an extensible website for retrieving, analyzing, and contributing results\. Together, these resources aim to make offline policy\-learning research more reliable and enable future work beyond the scope of this study\.

Figure 1:Our tuned models substantially outperform published baselines on Minari and change the relative ordering of algorithms\. Original paper implementations use the authors’ recommended or closest available hyperparameters\. Further details are provided in Appendix[B](https://arxiv.org/html/2609.13730#A2)\.## 1Introduction

Learning policies from fixed datasets offers an attractive alternative when online interaction is expensive, slow, or unsafe\. Offline Reinforcement Learning \(ORL\) uses reward\-labeled data to optimize a policy without further interaction, while Imitation Learning \(IL\) learns to reproduce behavior from demonstrations\. We use the term*offline policy learning*to encompass both settings, since they share the central empirical challenge studied here: reliably training and comparing policies using pre\-collected data\.

This comparison is harder than standard benchmark results often suggest\. Performance can change substantially with the choice of dataset, environment, hyperparameters, evaluation protocol, and aggregation rule\. When these factors vary across studies, it becomes difficult to determine whether an apparent improvement comes from the algorithm itself or from the comparison surrounding it\. Small evaluation suites and limited tuning budgets make these sensitivities particularly difficult to see\.

Prior work has exposed individual parts of this problem\. Offline RL performance can depend strongly on reward normalization, initialization, policy parameterization, network architecture, learning\-rate schedules, and other implementation choices\[kang2023improving\]\. Hidden layer size\[cetin2024simple\], dataset coverage and trajectory quality\[schweighofer2021dataset\], reward sparsity, horizon, and stochasticity\[bhargava2023when\], and the use of human rather than synthetic demonstrations\[mandlekar2021matterslearningofflinehuman\]can all change which methods appear strongest\. Aggregation and uncertainty reporting introduce further ambiguity\[agarwal2021deep\]\. Cross\-paper comparisons compound these problems: studies use different environment subsets, evaluation budgets, and benchmark versions, and numerical results are sometimes transferred between settings where they are not directly comparable, such as different environment versions\[kostrikov2021offline\]\.

We present a large\-scale empirical study designed to make these sensitivities visible\. Using nearly one million GPU\-hours, we train more than160,000160\{,\}000policies across 114 datasets spanning continuous control, robotics, navigation, and Atari\. Figure[1](https://arxiv.org/html/2609.13730#S0.F1)shows that our tuned models not only substantially outperform reference runs constructed from the original implementations and published hyperparameters, but also change the relative ordering of the algorithms\. Across the 49 shared datasets, IQL overtakes BCQ under both IQM and optimality gap after tuning\. Hyperparameter selection can therefore change the conclusion of a comparison, not simply the absolute scores it reports\. The reference implementations are modified only as needed to load the standardized datasets we use, we otherwise follow the authors’ tuning guidance or use the configuration from the most similar reported dataset\. Full matching and tuning details are provided in Appendix[B](https://arxiv.org/html/2609.13730#A2)\. We use the resulting corpus to study how tuning, benchmark composition, aggregation, and algorithm design affect empirical conclusions\.

Our primary contributions are:

- •We conduct an extensively tuned comparison of offline policy\-learning algorithms across 114 datasets, producing more than160,000160\{,\}000trained policies and a controlled view of performance across continuous\- and discrete\-control domains\.
- •We characterize how hyperparameter choice, tuning budget, and random seed affect absolute performance and algorithm rankings, identify the most influential hyperparameters and practical tuning budgets, and derive strong default configurations that transfer across datasets\.
- •We show that benchmark composition can produce conflicting algorithm rankings even when the selected subsets are comparable in size to established evaluation suites, and we propose reporting practices that make this sensitivity visible\.
- •We systematically analyze how dataset characteristics relate to algorithm performance and use these relationships to recommend promising algorithms for new tasks\.
- •We release the JumpStart corpus, including trained policies, hyperparameters, training code, and a website for retrieving, comparing, analyzing, and extending results\.

## 2Background and Related Work

In this section, we briefly review the background needed to place our study in context\. We first outline the main algorithm families we study, then summarize the datasets and benchmarks used in offline policy learning, and finally discuss prior empirical studies on what makes algorithms work or fail, along with related issues in evaluation, reporting, and tuning\.

Algorithms\.We study a mix of popular imitation learning and offline RL methods\. Behavior Cloning \(BC\) fits actions directly through supervised learning\. Action Chunking with Transformers \(ACT;zhao2023learning\) predicts short action sequences to reduce compounding error in manipulation settings\. VQ\-BeT\[lee2024vqbet\]uses vector\-quantized latent actions to capture multimodal behavior\. Diffusion Policy\[chi2023diffusion\]models action sequences through conditional denoising, allowing it to represent multimodal action distributions and perform receding\-horizon control\. Decision Transformer \(DT;chen2021decision\) models trajectories autoregressively and conditions its predictions on a desired return\-to\-go\. Implicit Q\-Learning \(IQL;kostrikov2021offline\) improves a policy through implicit value estimation without querying out\-of\-distribution actions\. Conservative Q\-Learning \(CQL;kumar2020conservative\) regularizes value estimates conservatively to stabilize offline Q\-learning\. ReBRAC\[tarasov2023revisiting\]builds on TD3\+BC with a set of specific design choices, including layer normalization, a deeper critic, a higher discount factor, and decoupled actor and critic penalties\. Batch\-Constrained deep Q\-learning \(BCQ;fujimoto2019offpolicydeepreinforcementlearning\) constrains candidate actions to remain close to the support of the dataset\.

Datasets\.The data used for offline policy learning can vary widely in quality, coverage, and domain, and these properties can strongly influence which algorithms perform well\. The most widely used benchmark is D4RL\[fu2021d4rldatasetsdeepdatadriven\], which introduced a standardized suite for evaluating offline, or data\-driven, reinforcement learning\. The original suite includes Maze2D and AntMaze\[fu2021d4rldatasetsdeepdatadriven\], Gym\-MuJoCo\[brockman2016openaigym,todorov2012mujoco\], Adroit\[rajeswaran2018learning\], FrankaKitchen\[gupta2019relay\], Flow\[wu2017flow\], and CARLA\[dosovitskiy2017carla\]\. In practice, most subsequent evaluations focus on the first five groups and omit Flow and CARLA\.

Prior to D4RL, studies generally generated their own non\-standardized evaluation datasets\[fujimoto2019offpolicydeepreinforcementlearning,kumar2019stabilizingoffpolicyqlearningbootstrapping,wu2019behaviorregularizedofflinereinforcement\]\. These datasets were commonly produced by training an expert through online reinforcement learning and then retaining either its training replay buffer or trajectories generated by selected checkpoints\. Later,mandlekar2021matterslearningofflinehumanintroduced RoboMimic, a collection of human teleoperation trajectories for the RoboSuite framework\[zhu2020robosuite\]\. They found that conclusions obtained from machine\-generated datasets did not necessarily transfer to human demonstrations, a distinction that we examine more broadly in Section[4\.2](https://arxiv.org/html/2609.13730#S4.SS2)\.

More recent benchmarks have aimed to address limitations of D4RL\.lu2023challengesintroduce V\-D4RL, which provides pixel\-based analogues of D4RL tasks for evaluating offline RL from visual observations\.liu2023liberorelease LIBERO, a procedurally generated suite of 130 language\-conditioned manipulation tasks built around RoboSuite and designed to study lifelong knowledge transfer in imitation learning\.rafailov2024d5rlpropose D5RL as a successor to D4RL, motivated by saturation on the original tasks, with more realistic manipulation and locomotion environments and heterogeneous data sources spanning scripted, teleoperated, and play\-style data\. Finally,park2025ogbenchintroduce OGBench, an offline goal\-conditioned RL benchmark spanning eight environment types and 85 datasets designed to probe capabilities such as stitching, long\-horizon reasoning, and robustness to stochasticity\.

In parallel, the Farama Foundation released Minari\[minari\], a standardized dataset format and Python API that hosts reimplementations of D4RL and other reference offline RL datasets\. Minari has partially supplanted the original D4RL codebase, which is no longer actively maintained\. For reproducibility and ease of maintenance, as discussed further in Section[3\.1](https://arxiv.org/html/2609.13730#S3.SS1), we use a 114\-dataset subset of Minari\. These datasets were collected from several sources, including humans, RL agents, and scripted planners\. This diversity allows us to study not only transfer between human\- and machine\-generated data, but transfer across expert sources more generally in Section[4\.2](https://arxiv.org/html/2609.13730#S4.SS2)\.

Reporting practices and reproducibility\.Inconsistent reporting has long been an issue in deep RL\.agarwal2021deeprecommend robust aggregate metrics accompanied by uncertainty estimates\. We adopt two such metrics throughout: the interquartile mean \(IQM\), which discards the top and bottom25%25\\%of normalized scores before averaging, and the optimality gap, which measures the average shortfall from a target performance level; we use expert performance as the target\.

Canonical offline policy\-learning papers differ in ways that make direct comparison difficult\. For example, BCQ\[fujimoto2019offpolicydeepreinforcementlearning\], BEAR\[kumar2019stabilizingoffpolicyqlearningbootstrapping\], and BRAC\[wu2019behaviorregularizedofflinereinforcement\]primarily present reward\-over\-training curves rather than consolidated benchmark tables, while Decision Transformer\[chen2021decision\]reports results on a comparatively restricted subset of D4RL\. In Section[4](https://arxiv.org/html/2609.13730#S4), we show that benchmark subsets comparable in size to established evaluation suites can nevertheless produce sharply different algorithm rankings\. Results are also sometimes copied across papers despite differences in the underlying environment versions, such as v0 versus v2, as noted bykostrikov2021offline\.

tarasov2023revisitingaddress part of this problem by providing single\-file implementations of several algorithms and evaluating them on a unified set of D4RL dataset versions\. Their primary goal, however, is implementation standardization rather than extensive per\-dataset tuning\. Our results show why this distinction matters: tuning can substantially change both absolute performance and the relative ordering of algorithms\. For example, BC frequently appears near the bottom of their leaderboards, whereas after tuning we find it substantially more competitive\.

Empirical studies\.andrychowicz2020mattersonpolicyreinforcementlearningtrain tens of thousands of on\-policy RL agents while varying more than 50 implementation choices, finding that low\-level details such as policy initialization, observation normalization, and action\-distribution parameterization can matter as much as the high\-level algorithm\.orsini2021mattersadversarialimitationlearningreach a similar conclusion for adversarial imitation learning and additionally show that algorithm rankings shift substantially between synthetic and human demonstrations\.

paine2020hyperparameterselectionofflinereinforcementconduct an early study of hyperparameter selection in offline RL\. Across three algorithms and ten continuous\-control tasks, they train 2,560 policies and find that performance varies substantially with hyperparameter choice\. They also show that Fitted Q Evaluation can often rank the resulting policies without additional environment interaction\. This addresses a different problem from ours: their policies have already been trained, and the goal is to select a strong candidate when the budget for evaluating those policies is limited\.

This setup treats training many candidate policies as feasible while evaluation is the primary bottleneck\. Although this remains relevant when deployment is expensive or risky, the balance is less clear for modern policy learning, where models may contain billions of parameters and be trained on large datasets\[black2026pi0visionlanguageactionflowmodel\]\. Our goal is instead to compare algorithms after adequate tuning while reducing the cost of reaching that point\. Across 162,864 policies from ten algorithms and 114 datasets, we identify which hyperparameters warrant attention and how their importance changes across datasets \(Section[5\.1](https://arxiv.org/html/2609.13730#S5.SS1)\)\. We then use the same sweeps to derive stronger default parameters, improving the performance of an initial run and reducing the amount of target\-specific tuning required \(Section[5\.2](https://arxiv.org/html/2609.13730#S5.SS2)\)\.

kang2023improvingablate 20 low\-level implementation choices across CQL, CRR, and IQL on D4RL and RL Unplugged\. They show that many reported gains arise from these choices rather than the central algorithm and that their effects depend strongly on the data distribution\. Some of their recommendations do not transfer uniformly across our broader suite; for example, we find that the importance of learning rate varies substantially across datasets\.cetin2024simplesimilarly argue that scale can matter more than novelty in offline RL, showing that appropriately scaled simple methods can match or outperform more elaborate algorithms on D4RL\. To control for model scale, we either include larger hidden sizes in our search spaces or use networks comparable in scale to the larger configurations in their study while varying critic count and layer width\.

Dataset properties provide another explanation for inconsistent results\.schweighofer2021datasetfind that exploration coverage and trajectory quality strongly predict which methods perform well, whilebhargava2023whenshow that the relative ordering of offline RL and imitation\-style methods depends on properties such as reward sparsity, horizon, and demonstration source\. We extend this line of work by automatically parsing environment and dataset descriptions into structured characteristics and using them in a practitioner\-facing recommender that identifies promising algorithms for a given task\. Prior studies therefore establish that implementations, hyperparameters, model scale, and dataset properties can all affect performance; our study brings these factors together and makes the resulting relationships directly actionable\.

Our study brings these lines of analysis together across ten algorithms, 114 datasets, continuous\- and discrete\-control domains, and more than160,000160\{,\}000trained policies\. Because we extensively tune each algorithm on each dataset, we can analyze dataset effects without relying on potentially weak default configurations\. In Section[6](https://arxiv.org/html/2609.13730#S6), we study these effects systematically and use environment and dataset characteristics to recommend promising algorithms for a given task\. We additionally release the complete corpus of trained policies, hyperparameters, and training code so that future studies can reproduce, inspect, and extend our analysis\. Table[1](https://arxiv.org/html/2609.13730#S2.T1)summarizes the components combined in our study relative to the most closely related empirical work\.

Table 1:Comparison with selected large\-scale empirical studies in offline policy learning\. Prior\-work counts describe the principal offline benchmarks; JumpStart counts include the full hyperparameter\-tuning corpus\.Notes\.NR denotes an unreported total;△\\triangledenotes a partial release\. Method counts include separately evaluated variants\.†Counts are reconstructed from reported experimental settings and exclude auxiliary ablations and offline\-to\-online fine\-tuning\. Policy metadata comprises policy\-level scores paired with hyperparameters or experiment configurations\. An extensible leaderboard accepts externally contributed results\.

## 3Training Methods

### 3\.1Datasets

![Refer to caption](https://arxiv.org/html/2609.13730v1/environment_collage.png)Figure 2:An 18\-environment subset of the diverse tasks used in the study\. A full list of environments and datasets can be found in Appendix[A](https://arxiv.org/html/2609.13730#A1)\.Minari follows the same recipe as D4RL, generating offline datasets from trained policies\[minari\], and additionally includes 7 human\-collected datasets\. We use Minari rather than the original D4RL release because it is the actively maintained successor, with newer environment versions, clearer documentation, and openly available dataset\-generation code\. Because the underlying datasets are not bit\-identical to D4RL, some absolute numbers \(e\.g\., CQL on AntMaze\) may differ from prior reports\. A subset of the tasks used in our study is visualized in Figure[2](https://arxiv.org/html/2609.13730#S3.F2)\. For Figure[1](https://arxiv.org/html/2609.13730#S0.F1), we adapted the source code of prior work\[chen2021decision,tarasov2023revisiting,fujimoto2019offpolicydeepreinforcementlearning,kostrikov2021offline,kumar2020conservative\]to support Minari and adapted hyper\-parameters using the procedure fromtarasov2023revisiting\(additionally following tuning suggestions from their papers, if available\), and re\-evaluated their models against ours\.

We made two additions\. The Atari datasets in Minari were too small for the discrete\-control comparisons we wanted, so we collected new ones with100100k states per game using their dataset generation code, following the Atari100100k setting\[kaiser2019modelbased\]\. We also omit a number of environments whose default datasets are trivially solvable across all methods we tested\. The remaining114114datasets \(5757continuous control,5757discrete\) cover a wide range of distributional shapes, including some near\-zero\-reward degenerate datasets that we filter from our analyses, though they are still included in artifact releases for future studies\. After filters, we land at5252discrete and5757continuous datasets\. We provide the full set of datasets and their descriptions in Appendix[A](https://arxiv.org/html/2609.13730#A1)\.

D4RL provides fixed expert and random reference returns for each environment, whereas Minari currently provides these values only for datasets imported from D4RL\. In our work, we define "expert" as the policy that generated the dataset used for offline policy learning\. To apply a consistent normalization procedure across all environments, we estimate the expert reference directly from each dataset and estimate the random reference by running a random policy for 100 trajectories in each unique environment\.

For a trajectoryτ\\tau, let

R⁡\(τ\)=∑t=0TτrtR\(\\tau\)=\\sum\_\{t=0\}^\{T\_\{\\tau\}\}r\_\{t\}denote its undiscounted return\. For dataset𝒟\\mathcal\{D\}from environmentee, we define

Rexpert​\(𝒟\)=1\|𝒟\|​∑τ∈𝒟R⁡\(τ\),Rrandom​\(e\)=1100​∑i=1100R⁡\(τirand\)\.R\_\{\\mathrm\{expert\}\}\(\\mathcal\{D\}\)=\\frac\{1\}\{\|\\mathcal\{D\}\|\}\\sum\_\{\\tau\\in\\mathcal\{D\}\}R\(\\tau\),\\qquad R\_\{\\mathrm\{random\}\}\(e\)=\\frac\{1\}\{100\}\\sum\_\{i=1\}^\{100\}R\(\\tau\_\{i\}^\{\\mathrm\{rand\}\}\)\.Given an evaluation returnRevalR\_\{\\mathrm\{eval\}\}, we report

Rnorm=Reval−Rrandom​\(e\)Rexpert​\(𝒟\)−Rrandom​\(e\)\.R\_\{\\mathrm\{norm\}\}=\\frac\{R\_\{\\mathrm\{eval\}\}\-R\_\{\\mathrm\{random\}\}\(e\)\}\{R\_\{\\mathrm\{expert\}\}\(\\mathcal\{D\}\)\-R\_\{\\mathrm\{random\}\}\(e\)\}\.

### 3\.2Model training

Our pipeline is built on d3rlpy\[d3rlpy\]for verified algorithm implementations and Ray Tune\[liaw2018tuneresearchplatformdistributed\]for distributed hyperparameter tuning\. We completed full hyperparameter tuning runs with BC, BCP \(BC with percentile\-based data filtering, also called BC%, a common baseline\[tarasov2023revisiting,chen2021decision\]\), BCQ, CQL, and DT across all114114continuous and discrete datasets; VQ\-BeT, ACT, IQL, and ReBRAC run on5757continuous control datasets\. We start with the LeRobot111[https://github\.com/huggingface/lerobot/tree/main/src/lerobot/policies/vqbet](https://github.com/huggingface/lerobot/tree/main/src/lerobot/policies/vqbet),[https://github\.com/huggingface/lerobot/tree/main/src/lerobot/policies/act](https://github.com/huggingface/lerobot/tree/main/src/lerobot/policies/act)implementations for ACT and VQ\-BeT and modify them to work with our codebase\. For Diffusion Policy, we used a modified implementation from the original paper’s code release222[https://github\.com/real\-stanford/diffusion\_policy](https://github.com/real-stanford/diffusion_policy)\. We validated the modified implementations on a subset of MuJoCo tasks in Minari by converting datasets to a format supported by the original code\.

Training budgets\.Value\-based methods \(CQL, BCQ, IQL, ReBRAC\) use500,000500\{,\}000gradient steps on continuous environments and250,000250\{,\}000on discrete\. BC, BCP, and DT use5050epochs\. Diffusion and ACT uses100100epochs, and VQ\-BeT uses 300 total split among the stages\. These budgets were chosen via an exponentially increasing sweep on a small set of testing datasets \(33MuJoCo HalfCheetah datasets and D4RL AntMaze\) under default and four random hyperparameter configurations, picking the smallest budget at which rewards plateau\.

Table 2:Number of trained policies by algorithm and action space\.Table 3:Search\-space boundary occupancy\. We report the percentage of trials containing at least one hyperparameter within 5% of either search\-range boundary\. Enrichment is the difference between the top 10% of trials and all trials\.Hyperparameter tuning\.We use random search\[bergstra2012random\]and the Tree\-structured Parzen Estimator \(TPE\), which models better\- and worse\-performing configurations to bias subsequent trials toward promising regions\[bergstra2011algorithms\]\. We run TPE through Ray Tune’s OptunaSearch wrapper\[akiba2019optuna\]\. The search spaces are reported in Tables[4](https://arxiv.org/html/2609.13730#S8.T4)and[5](https://arxiv.org/html/2609.13730#S8.T5), and model counts are reported in Table[2](https://arxiv.org/html/2609.13730#S3.T2)\. To test whether these ranges constrain performance, we compare how often the top 10% of trials and the full sweep contain at least one hyperparameter within 5% of a search\-range boundary\. If there is a significant difference in prevalence within the boundaries, the sweep may have been limited in width\. As shown in Table[3](https://arxiv.org/html/2609.13730#S3.T3), the median enrichment among top trials is only\+0\.1\+0\.1percentage points, with no consistent increase across algorithms\. Strong configurations therefore do not systematically accumulate near the edges of our search spaces, suggesting that the selected ranges are sufficiently broad\.

Evaluation\.For all evaluations, we use the final checkpoint evaluated across 100 episodes\. Minari provides a method to recover the environment used to collect each dataset, but sometimes omits wrappers \(e\.g\., resizing or frame stacking\), as in Atari\. In these cases, we recover the missing wrappers from Minari’s data\-generation scripts\.333[https://github\.com/Farama\-Foundation/minari\-dataset\-generation\-scripts/tree/main](https://github.com/Farama-Foundation/minari-dataset-generation-scripts/tree/main)

## 4Evaluation Sensitivity

Aggregates can hide more than they show\. The apparent best algorithm depends materially on the choice ofmetric,data, andtuning budget\. We treat each as a distinct failure mode and recommend evaluation practices that make the resulting ambiguity legible\.

Figure 3:A visual example of optimality gap and the IQM overlaid on a survival chart for behavior cloning on our suite\.The choice of evaluation metric has been studied extensively in reinforcement learning\. Followingagarwal2021deep, we avoid reporting only means or medians over many runs and datasets, and instead use the interquartile mean \(IQM\) and optimality gap as robust aggregate metrics\. For normalized scoresx1,…,xnx\_\{1\},\\ldots,x\_\{n\}, letq0\.25q\_\{0\.25\}andq0\.75q\_\{0\.75\}denote the empirical first and third quartiles\. The IQM is

IQM⁡\(x\)=1\|ℐ\|​∑i∈ℐxi,ℐ=\{i:q0\.25≤xi≤q0\.75\}\.\\operatorname\{IQM\}\(x\)=\\frac\{1\}\{\|\\mathcal\{I\}\|\}\\sum\_\{i\\in\\mathcal\{I\}\}x\_\{i\},\\qquad\\mathcal\{I\}=\\\{i:q\_\{0\.25\}\\leq x\_\{i\}\\leq q\_\{0\.75\}\\\}\.For an optimality thresholdγ\\gamma, the optimality gap is

OGγ⁡\(x\)=1n​∑i=1nmax⁡\(γ−xi,0\)\.\\operatorname\{OG\}\_\{\\gamma\}\(x\)=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\max\(\\gamma\-x\_\{i\},0\)\.In our setting, normalized expert performance corresponds toγ=1\\gamma=1\. Since metric aggregation is already well explored, we adopt these methodologies and focus our analysis on the remaining sources of uncertainty\. We provide a visual explanation of IQM and optimality gap in Figure[3](https://arxiv.org/html/2609.13730#S4.F3)\.

Figure 4:15\-dataset subset leaderboard constructions with different Kendall’sτ\\tauto the full\-environment leaderboard\. Even with 15\-20 different datasets, it is possible to have rankings in different orders, with the top 3 \(gold, silver, bronze outlined\) varying substantially\. Letters A\-J indicate different algorithms\.We compare algorithm orderings, or leaderboards, using Kendall’sτ\\tau\. Letri\(a\)r\_\{i\}^\{\(a\)\}andri\(b\)r\_\{i\}^\{\(b\)\}denote the rank of algorithmiiunder two leaderboardsaaandbb, respectively, with rank11corresponding to the best\-performing method\. For each pair of algorithms\(i,j\)\(i,j\), we compute

ci​j=sign⁡\[\(ri\(a\)−rj\(a\)\)​\(ri\(b\)−rj\(b\)\)\],c\_\{ij\}=\\operatorname\{sign\}\\left\[\\left\(r\_\{i\}^\{\(a\)\}\-r\_\{j\}^\{\(a\)\}\\right\)\\left\(r\_\{i\}^\{\(b\)\}\-r\_\{j\}^\{\(b\)\}\\right\)\\right\],so thatci​j=1c\_\{ij\}=1if the two leaderboards order the pair consistently,ci​j=−1c\_\{ij\}=\-1if they reverse the pair, andci​j=0c\_\{ij\}=0if the pair is tied in either ordering\. To visualize the disagreement values, we show a reference leaderboard and a few alternatives with varying Kendall’sτ\\tauvalues in Figure[4](https://arxiv.org/html/2609.13730#S4.F4)\. Letters A\-J indicate different algorithms, and as ordering deviates from the reference leaderboard, the Kendall’sτ\\taudrops\.

We begin by analyzing how the tuning and budget can materially change the conclusions of a study in Section[4\.1](https://arxiv.org/html/2609.13730#S4.SS1)\. We additionally compare the variance across different hyperparameter sets to the seed variance, and suggest an allocation of resources that removes more uncertainty than simply running more seeds\.

In Section[4\.2](https://arxiv.org/html/2609.13730#S4.SS2), we analyze how the separation, choice and quantity of datasets impacts reliability, and generalize on prior work finding human\-generated datasets behaving differently from machine\-generated ones, finding that this can be applied toanydifferent expert source: planner\-based solvers, reinforcement learning, cloned datasets, and human\.

Finally, in Section[4\.3](https://arxiv.org/html/2609.13730#S4.SS3), we construct leaderboards and summary tables showing final algorithm performance after hyperparameter tuning\.

Takeaway\.We use boxes like this to close each analysis with its main takeaway\.

### 4\.1Tuning budget and seed variance

Figure 5:Tuning progress in continuous\-control\. Most of the improvement in best observed normalized score occurs within 25–50 trials, whereas agreement with the full\-budget algorithm ranking improves more gradually\. Confidence intervals are across datasets, where peak performance can vary substantially\.Figure 6:Tuning progress in discrete\-control\. Agreement with the full\-budget ranking increases rapidly during the first 10–25 trials, while the best observed normalized score continues to improve with additional tuning\. Confidence intervals are across datasets, where peak performance can vary substantially\.Running 200 hyperparameter trials for every algorithm–dataset pair is beyond the budget of many evaluations\. We therefore use the complete sweep as a reference and ask how quickly smaller searches recover both its best observed performance and its algorithm ordering\.

In continuous\-control datasets, these two quantities converge at different rates\. Figure[5](https://arxiv.org/html/2609.13730#S4.F5)shows that most of the improvement in normalized score occurs within the first 25–50 trials, with smaller gains thereafter\. Agreement with the full\-budget ranking improves more slowly\. A search may therefore find strong configurations before it provides a reliable comparison between algorithms: continuous\-control leaderboards remain sensitive to tuning budget even after performance begins to plateau\.

Ranking agreement increases more quickly in discrete control\. Figure[6](https://arxiv.org/html/2609.13730#S4.F6)shows a sharp improvement during the first 10–25 trials, after which Kendall’sτ\\tauremains generally high despite some fluctuations\. The best observed score continues to increase throughout the search\.

Figure 7:Variability across hyperparameter configurations and random seeds\. The left panel reports the median per\-cell coefficient of variation for each algorithm, and the right panel summarizes 731 algorithm–dataset cells\. Hyperparameter variability exceeds seed\-wise variability in 92% of cells, or 83% after removing outlier configurations\. Within the top 25% of configurations, however, variability is comparable to that observed across seeds\.Figure 8:Comparison between continuous\-control leaderboards constructed from the top\-quartile mean of the hyperparameter sweep and from independently seeded runs\. IQM and optimality\-gap estimates are nearly identical, and the resulting algorithm rankings have Kendall correlations ofτ=0\.94\\tau=0\.94for IQM andτ=0\.83\\tau=0\.83for optimality gap\.Tuning and seed replication answer different questions\. Additional configurations explore sensitivity to hyperparameter choice and increase the chance of finding a strong policy, whereas repeated seeds estimate the stochastic variability of a fixed configuration\. A finite evaluation budget must therefore balance search\-space exploration against replication of the selected configurations\.

Across the complete sweep, hyperparameter choice is the larger source of variation\. As shown in Figure[7](https://arxiv.org/html/2609.13730#S4.F7), variation across configurations exceeds seed\-wise variation in92%92\\%of algorithm–dataset cells and in83%83\\%after removing outlier configurations\. Results based on only a few configurations can therefore depend more strongly on which hyperparameters were tried than on which random seeds were used\. Seed replication is necessary for estimating uncertainty between runs, but additional seeds cannot compensate for an inadequately explored search space and leaves a much larger source of uncertainty unexplored\. For our experiments we pick the 95th percentile hyperparameter for each dataset–algorithm pair tested, and train for 5 seeds\.

The upper\-performing region is considerably more concentrated\. Among the top25%25\\%of configurations, variation across hyperparameters is comparable to variation across seeds\. We therefore average the normalized scores of the top25%25\\%of configurations within each algorithm–dataset pair, producing one robust near\-optimal score per pair, and compute IQM and optimality gap across these dataset\-level scores\.

Figure[8](https://arxiv.org/html/2609.13730#S4.F8)validates this approximation against independently seeded runs\. The two procedures produce nearly identical IQM and optimality\-gap estimates, while their continuous\-control rankings agree strongly: Kendall’sτ\\tauis0\.940\.94for IQM and0\.830\.83for optimality gap\.

Takeaway\.Hyperparameter searches may find many strong configurations long before providing a stable comparison between algorithms\.

Takeaway\.Hyperparameter choice dominates variation across the full search, while the top of the search is much more concentrated\. A budget of 25–50 trials provides a useful practical compromise\.

### 4\.2Datasets and Environments

The datasets and environments included in an evaluation shape both the stability and interpretation of its leaderboard\. We examine how rankings change with benchmark size and composition, and whether they transfer across different expert sources\.

#### 4\.2\.1Benchmark Size and Composition

Figure 9:Selected benchmark subsets that favor different algorithms\. We sample 2,000 subsets at each tested size \(n∈\{10,12,15,20\}n\\in\\\{10,12,15,20\\\}\)\. Among these candidates, subsets containing 12, 10, and 12 environment–dataset pairs place ACT, DT, and BCQ first by a margin, respectively, under both IQM and optimality gap\.Benchmark composition can materially change the conclusion of an evaluation\. Small benchmarks are naturally vulnerable to unrepresentative sampling, but size alone may not solve the problem: we ask whether subsets comparable in size to established evaluation suites can still produce substantially different algorithm rankings\.

We draw 2,000 random subsets independently at each of four sizes,n∈\{10,12,15,20\}n\\in\\\{10,12,15,20\\\}, producing 8,000 candidate benchmarks\. We then search these candidates for subsets that favor different algorithms\. Figure[9](https://arxiv.org/html/2609.13730#S4.F9)shows that ACT, DT, and BCQ can each be placed first under both IQM and optimality gap\. The selected subsets contain 12, 10, and 12 environment–dataset pairs, respectively, and are therefore not unusually small\. An algorithm other than the winner on the full 57\-dataset continuous\-control benchmark ranks first under both IQM and optimality gap in 24\.70%, 23\.80%, 20\.80%, and 15\.45% of subsets of sizesn=10,12,15,20n=10,12,15,20, respectively\. For comparison, the continuous\-control evaluation in Decision Transformer contains four environments under three dataset regimes each, yielding 12 environment–dataset pairs\[chen2021decision\]\.

These examples show that favorable leaderboards can be found through a moderate random search over benchmark composition\. A selectively composed benchmark can therefore match the size of a complete suite used in prior work while supporting a different conclusion about which algorithm performs best\.

Figure 10:Agreement between subset and full\-suite leaderboards as a function of subset size\. Lines show the mean Kendall’sτ\\tauover randomly sampled subsets, and shading shows one standard deviation\. Agreement increases and variability across subsets decreases as more environment–dataset pairs are included\.Figure 11:Probability of displaying a winner conflicting with the true top algorithm as the benchmark size increases\.Random subsets nevertheless become more reliable as they grow\. Figure[10](https://arxiv.org/html/2609.13730#S4.F10)shows that their mean agreement with the full\-suite leaderboard increases with subset size, while variability across subsets decreases\. We show in Figure[11](https://arxiv.org/html/2609.13730#S4.F11)that the probability of producing a conflicting winner to the full\-suite leaderboard decreases as more benchmarks are added\.

Within our suite, evaluations therefore become less sensitive to the particular pairs sampled as more pairs are included\. Strongly discordant leaderboards become less likely to arise through random selection\. Carefully selected subsets remain a concern, however: Figure[9](https://arxiv.org/html/2609.13730#S4.F9)shows that benchmarks comparable in size to prior evaluation suites can still be composed to favor different algorithms\.

Takeaway\.Larger benchmark suites produce more stable algorithm rankings\. In our continuous\-control experiments, subsets of 10–15 environment–dataset pairs, comparable to prior evaluations, yield a different winner from the full benchmark under both metrics roughly 20–25% of the time\. Increasing the subset size to 25 pairs reduces this rate to about 10%\.

#### 4\.2\.2Transfer Across Expert Sources

Prior work has found that results on agent\-generated datasets do not necessarily transfer to human demonstrations\[mandlekar2021matterslearningofflinehuman\]\. We examine whether this is part of a broader pattern by comparing algorithm rankings across several sources of expert data\.

We divide the 57 continuous\-control datasets according to how their trajectories were generated:

- •Cloned \(4 datasets\)\.Datasets generated wholly or partly by policies trained through behavior cloning\.
- •Human \(7 datasets\)\.Datasets collected through direct human control or teleoperation\.
- •RL \+ Planner \(6 datasets\)\.Datasets for which both a planner and a policy trained through reinforcement learning contribute to trajectory generation\.
- •RL \(32 datasets\)\.Datasets generated by expert policies trained through reinforcement learning\.
- •Scripted \(8 datasets\)\.Datasets generated by a predefined planner or scripted controller rather than a learned policy\.

![Refer to caption](https://arxiv.org/html/2609.13730v1/transfer_continuous.png)Figure 12:Agreement between algorithm rankings across expert sources\. The left and center panels report pairwise Kendall’sτ\\tauunder optimality gap and IQM\. The right panel reports mean off\-diagonal agreement before and after controlling for the log number of datasets in each source\. The diamond with whiskers represents the count\-matched random baseline, with whiskers representing 1 standard deviation\.Figure[12](https://arxiv.org/html/2609.13730#S4.F12)shows that algorithm rankings transfer unevenly across expert sources\. Pairwise Kendall’sτ\\tauranges from−0\.16\-0\.16to0\.600\.60under optimality gap and from−0\.07\-0\.07to0\.470\.47under IQM\. Most pairs exhibit little or negative agreement\. In particular, poor transfer is not confined to comparisons between human and machine\-generated data; it also occurs between different methods of generating trajectories computationally\.

Averaged over the ten source pairs, Kendall’sτ\\tauis0\.1640\.164under optimality gap and0\.1730\.173under IQM\. Randomly constructed groups with the same sizes reach approximately0\.260\.26under both metrics\. Rankings transfer unusually poorly, even compared to sampling random datasets to predict the ranking\.

Group size explains part of the variation between sources\. The raw averages make RL appear more transferable, but this category contains 32 datasets, compared with only 4–8 for each remaining source\. After controlling for the log number of datasets, the apparent advantage of RL largely disappears, and every source has lower mean agreement than the random baseline\. No single expert source consequently produces a ranking that is broadly representative of the others\.

These results extend the distinction previously observed between human and agent\-generated demonstrations\. Changes in algorithm ordering seem to occur not only between human and machine\-generated data, but also between cloned, scripted, planned, and RL\-generated datasets\.

Takeaway\.Algorithm rankings transfer poorly across expert sources and agree less than rankings obtained from size\-matched random groups\. This failure extends beyond human\-versus\-machine comparisons to different sources of machine\-generated data\.

### 4\.3Leaderboard and evaluations

Figure 13:Overall algorithm performance on continuous\- and discrete\-control datasets\. We report interquartile mean \(IQM; higher is better\) and optimality gap with thresholdγ=1\\gamma=1\(lower is better\)\. Markers denote point estimates, while boxes and whiskers summarize bootstrap uncertainty\. ACT leads the continuous\-control aggregate, whereas BCP leads the discrete\-control aggregate under both metrics\.Figure 14:Continuous\-control performance separated by environment collection\. The leading algorithm changes across collections: ACT leads on AntMaze and Kitchen, BCQ on PointMaze, DT on Adroit, and IQL or ACT on MuJoCo depending on the aggregation metric\.Figure 15:Continuous\-control performance separated by the source of the trajectories in each dataset\. The leading algorithm changes from ACT on human and RL \+ planner data, to DT on cloned data, IQL on RL data, and BCQ on scripted data\.We next ask which algorithms perform well across the suite and where their relative strengths change\. Figure[13](https://arxiv.org/html/2609.13730#S4.F13)presents the overall continuous\- and discrete\-control leaderboards, while Figures[14](https://arxiv.org/html/2609.13730#S4.F14)and[15](https://arxiv.org/html/2609.13730#S4.F15)separate continuous\-control results by environment collection and expert source\. Full per\-dataset scores are reported in Tables[6](https://arxiv.org/html/2609.13730#S8.T6)and[7](https://arxiv.org/html/2609.13730#S8.T7)\.

ACT is the strongest algorithm in the overall continuous\-control aggregate, ranking first under both IQM and optimality gap\. Its advantage is broad rather than universal: ACT also leads on AntMaze, Kitchen, human demonstrations, and RL \+ planner data, but different methods take the lead elsewhere\. BCQ performs best on PointMaze and scripted datasets, DT leads on Adroit and cloned data, and IQL obtains the highest IQM on MuJoCo and RL\-generated data\. The aggregate leaderboard therefore provides a useful summary, but it obscures substantial changes in algorithm ordering across evaluation regimes\.

The human\-data results illustrate this distinction\. ACT remains first and VQ\-BeT rises to second under both metrics, consistent with VQ\-BeT’s design for multimodal behavior generation from demonstration data\[lee2024vqbet\]\. DT follows, while BCP places fourth under IQM\.

The simple cloning baselines are especially competitive\. BCP uses a small MLP and differs from ordinary behavior cloning primarily through data filtering, yet it remains close to the leading group in several continuous\-control settings\. Its discrete\-control performance is even stronger: BCP ranks first on Atari under both IQM and optimality gap, with unfiltered BC second\. Prior studies such as CORL\[tarasov2023revisiting\]often have behavior cloning at the bottom of their leaderboards, but in our continuous control suite they appear to be closer to the middle post\-tuning\. These are by far the cheapest and simplest methods we study, the performance here is thus quite impressive\.

Diffusion and CQL produce the weakest continuous\-control aggregates, although for different reasons\. We evaluate the U\-Net variant of Diffusion Policy as a standalone policy trained separately on each dataset\[chi2023diffusion\]\. It achieves competitive returns on many AntMaze, Adroit, and MuJoCo datasets but fails severely on others\. These results should not be interpreted as evidence that diffusion\-based action generation is flawed\. In foundation VLAs such asπ0\\pi\_\{0\}, the related flow\-matching objective is used to model multimodal action chunks within a much larger pretrained system\[black2026pi0visionlanguageactionflowmodel\]\. Its action expert is also substantially smaller than the VLM backbone, and cached representations avoid repeatedly evaluating the full model during generation\. This differs considerably from training Diffusion Policy from scratch as the complete policy\. Action chunking itself is unlikely to explain our results because other chunked policies perform strongly under the same evaluation pipeline\. Finally, extending training from 100 to 300 epochs produced little additional reward at substantially greater cost\.

CQL exhibits a sharp domain reversal\. It ranks last in the overall continuous\-control comparison but third on Atari, ahead of DT and far ahead of BCQ\. Because the original CQL results were obtained on a different set of datasets\[kumar2020conservative\], we checked whether this discrepancy could be explained by our implementation\. We adapted the original repository to load our Minari datasets and evaluated it on our suite of environments and datasets, represented in Figure[1](https://arxiv.org/html/2609.13730#S0.F1), excluding PointMaze because the original work reports neither results nor tuning guidance for that collection\. Following the procedure used by CORL, we applied the authors’ tuning recommendations when available and otherwise used the hyperparameters reported for the most similar dataset\. When these configurations failed, we additionally evaluated a small set of existing published configurations\. The original implementation produced performance comparable to our tuned models across the overlapping settings, making a simple implementation failure an unlikely explanation for CQL’s continuous\-control performance\. Prior reproduction and benchmarking efforts similarly find that CQL is highly sensitive to hyperparameters, low\-level implementation choices, and the distribution of the offline data\[tarasov2023revisiting,kang2023improving\]\. Further implementation and tuning details are provided in Appendix[B](https://arxiv.org/html/2609.13730#A2)\.

Finally, the value\-function\-based methods do not behave as a single coherent group\. IQL is strongest on RL\-generated and MuJoCo data, ReBRAC is competitive on RL \+ planner data, and BCQ leads on scripted and PointMaze datasets\. Their relative performance changes sharply elsewhere: BCQ performs poorly on Atari despite leading PointMaze, while IQL and ReBRAC are much less competitive on human demonstrations\.

Takeaway\.ACT is the strongest overall continuous\-control method, while the simple BCP baseline leads Atari\. No algorithm dominates every data regime: environment collection and expert source substantially change which method performs best, particularly beyond the top position\.

## 5Hyperparameter Sensitivity and Transfer

The previous section showed that tuning budget can materially change algorithm rankings\. We now ask which hyperparameters drive those changes, whether the same hyperparameters matter across environments, and how a broad sweep can be distilled into a strong default configuration\. We examine sensitivity first and then turn to transfer\.

### 5\.1Hyperparameter sensitivities

Figure 16:Hyperparameter sensitivity across the 57 continuous\-control environments\. For each algorithm–environment pair, we compute the absolute Spearman correlation\|ρh\|\|\\rho\_\{h\}\|between each hyperparameter and trial score and normalize these magnitudes assh=\|ρh\|/∑j\|ρj\|s\_\{h\}=\|\\rho\_\{h\}\|/\\sum\_\{j\}\|\\rho\_\{j\}\|\. Each vertical bar represents one environment and shows the resulting sensitivity shares\. The bottom panel reports the median cumulative share of the one through five most influential hyperparameters within each environment; the displayed value is the median top\-one share\. Environment ordering is chosen separately within each algorithm for readability\.Figure 17:Hyperparameter sensitivity across the 54 Atari environments, computed as in Figure[16](https://arxiv.org/html/2609.13730#S5.F16)\. The sensitivity profiles are more consistent than in continuous control: learning rate is the most influential hyperparameter for every algorithm, with particularly strong concentration for CQL and BCQ\.Figures[16](https://arxiv.org/html/2609.13730#S5.F16)and[17](https://arxiv.org/html/2609.13730#S5.F17)visualize how sensitivity is distributed across algorithms and environments\. For each algorithm–environment pair, we measure the marginal association between hyperparameterhhand trial performance using Spearman’s rank correlationρh\\rho\_\{h\}\. Because the direction of the relationship is secondary to whether a hyperparameter affects performance, we take the absolute correlation and normalize it within the environment:

sh=\|ρh\|∑j\|ρj\|\.s\_\{h\}=\\frac\{\|\\rho\_\{h\}\|\}\{\\sum\_\{j\}\|\\rho\_\{j\}\|\}\.
The resulting value measures each hyperparameter’s share of the total marginal rank association within an environment\. We report both the median\|ρh\|\|\\rho\_\{h\}\|and the median normalized share in Tables[8](https://arxiv.org/html/2609.13730#S8.T8)and[9](https://arxiv.org/html/2609.13730#S8.T9)\.

We find that most algorithms are primarily sensitive to one to three hyperparameters, but their identities often change across datasets\. Although learning rate is the most important hyperparameter for ACT and VQ\-BeT in aggregate, action horizon dominates ACT on kitchen/complete, humanoid/simple, humanoid/medium, pointmaze/medium\-dense, and door/cloned\. For VQ\-BeT, it dominates on both inverteddoublependulum datasets, invertedpendulum/expert, and antmaze/medium\-diverse\. Even rarely dominant parameters can matter locally: VQ\-BeT’s commitment weight leads on hammer/human, while its secondary\-code weight leads on pen/human\. Full per\-environment heatmaps are provided in Figure[25](https://arxiv.org/html/2609.13730#A3.F25)of Appendix[C](https://arxiv.org/html/2609.13730#A3)\.

Sensitivity remains concentrated even after accounting for differences in search\-space size\. For VQ\-BeT, the three most important hyperparameters within an environment account for a median of74\.5%74\.5\\%of total sensitivity, compared with25%25\\%under a uniform allocation across its 12 tuned parameters\. The corresponding values are70\.9%70\.9\\%versus27\.3%27\.3\\%for BCQ and85\.8%85\.8\\%versus33\.3%33\.3\\%for ReBRAC\. By comparison, ACT’s top\-three share is higher in absolute terms at88\.7%88\.7\\%, but its six\-parameter search space gives a uniform reference of50%50\\%\.

The dominant hyperparameter is not always the one conventionally emphasized\. Learning rate ranks first for every discrete algorithm, but the continuous\-control results are more varied:γ\\gammaleads for BCQ, hidden\-unit count for DT, actor regularizationβ\\betafor ReBRAC, the conservative\-loss threshold for CQL, and the number of diffusion steps for Diffusion Policy\. Consequently, tuning only learning rate and a few standard optimization parameters can miss the principal source of variation for several algorithms\.

We additionally investigate whether the sensitivities are driven by suboptimal \(too wide\) ranges, resulting in divergent runs\. We trimmed 10% and 25% of the runs at the boundaries, and find that the original top hyperparameter remains completely unchanged\. The same leading hyperparameter remained ranked first for all algorithms under both trims\.

Takeaway\.Sensitivity is usually concentrated, but its identity is not always transferable\. In continuous control, the top three hyperparameters capture54%54\\%–78%78\\%of total absolute rank association, yet the dominant parameters often change across environments\.

### 5\.2Better Default Hyperparameters

Given the cost of running 25–50 tuning trials, we attempt to reduce the burden on practitioners by deriving strong default configurations for existing algorithms\. Algorithm libraries commonly provide a single default configuration, but these defaults can leave substantial performance on the table\. We therefore ask whether sweeps already completed by an algorithm’s developers can be distilled into data\-driven defaults that transfer to previously unseen environments without target\-specific tuning\.

Deriving improved defaults\.Our heuristic is motivated by the hypothesis that, although the best hyperparameter value may vary across environments, good values occupy broad, overlapping regions of the search space\. Values near the center of high\-performing configurations may therefore transfer more reliably than either a library default or the best configuration from a single environment\.

For each algorithm and training environment, we retain the top10%10\\%of trials by score\. We pool these trials across environments and select each numerical hyperparameter by its median, computing the median in log\-space for log\-scaled parameters\. For categorical parameters, we use the mode\. Restricting the calculation to high\-performing trials avoids clearly poor regions, while the median reduces the influence of noisy trials and environment\-specific extremes\. This procedure produces one fixed configuration per algorithm that can directly replace its existing library default\.

Figure 18:Performance of existing d3rlpy defaults, median\-transfer defaults, and fully tuned JumpStart top\-quartile results\. The evaluation contains five folds of five held\-out environments, yielding 25 environments and 140 matched algorithm–environment cells\. Each fold’s test environments are excluded when constructing its transferred defaults\. Error bars are95%95\\%bootstrap confidence intervals\.Do the improved defaults work?We evaluate BC, BCQ, CQL, DT, IQL, and ReBRAC against their existing d3rlpy defaults\[d3rlpy\]\. The evaluation uses five folds containing five held\-out environments each\. For every fold, its test environments are excluded from the construction of the transferred configuration\.

As shown in Figure[18](https://arxiv.org/html/2609.13730#S5.F18), our defaults increase IQM from0\.470\.47to0\.700\.70and reduce optimality gap from0\.550\.55to0\.410\.41\. Fully tuned JumpStart results reach an IQM of0\.930\.93and an optimality gap of0\.270\.27\. Median transfer therefore recovers roughly half of the improvement obtained through full target\-specific tuning without evaluating any configuration on the target environment\. Recovering the remaining improvement without target\-specific search is difficult because both the identities of the dominant hyperparameters and their best values vary across environments, preventing a single coordinate\-wise default from reproducing each target’s specialized joint configuration\.

These aggregate results establish that the complete configurations transfer, but do not explain why\. We next test whether their individual hyperparameters fall within regions associated with strong performance on unseen environments\.

Figure 19:Marginal support for candidate defaults across five folds\. Within each fold, its five final test environments are excluded, and each remaining environment is held out in turn\. A value is supported when it lies inside the80%80\\%highest\-density region \(HDR\) of hyperparameter values among the target environment’s top25%25\\%of trials\. We compare our median\-transfer defaults with the existing d3rlpy defaults and the exact coverage of a value drawn from the configured search distribution\. The dotted line denotes the80%80\\%probability mass contained by the HDR by construction, not an expected transfer rate\. Error bars are95%95\\%environment\-bootstrap confidence intervals\.Why do the defaults transfer?Within each fold, we exclude its five final test environments and hold out each remaining training environment in turn\. For every algorithm, target environment, and hyperparameter, we recompute the transferred default using only the other training environments\. We then estimate the distributionp^e,h\\hat\{p\}\_\{e,h\}of good values on the target from its top25%25\\%of trials, using a boundary\-corrected kernel density estimate for numerical parameters and a lightly smoothed empirical probability mass function for categorical parameters\.

For numerical parameters, we evaluatep^e,h\\hat\{p\}\_\{e,h\}on a dense grid spanning the configured search bounds and normalize it using numerical integration\. We then lower a density threshold until the grid intervals above that threshold contain80%80\\%of the estimated probability mass\. Boundary correction is applied when estimating the density, before this numerical integration\. For categorical parameters, we analogously select the highest\-probability categories containing at least80%80\\%of the smoothed probability mass\. We record whether each candidate default lies inside the resulting region\. Random search\-space coverage is computed directly from the probability assigned to the same region by the original tuning distribution\.

Figure[19](https://arxiv.org/html/2609.13730#S5.F19)shows that median\-transfer coverage ranges from approximately61%61\\%to83%83\\%across algorithms, compared with33%33\\%–63%63\\%for the existing defaults and52%52\\%–63%63\\%for a random search\-space value\. The median\-transfer default has the highest coverage for every evaluated algorithm\. This supports the intuition behind the heuristic: high\-performing configurations commonly occupy regions shared across environments rather than isolated, environment\-specific optima\.

This density analysis is marginal and does not capture interactions between hyperparameters\. Nevertheless, its agreement with the held\-out evaluation suggests that the full configurations transfer because their individual values repeatedly fall within broad regions associated with strong performance\. The transferred configurations should therefore be interpreted as strong starting points rather than replacements for full tuning\.

Takeaway\.Taking the coordinate\-wise median of high\-performing configurations from prior environments produces strong transferable defaults\. Across 25 held\-out environments, these defaults raise IQM from0\.470\.47to0\.700\.70and reduce optimality gap from0\.550\.55to0\.410\.41, recovering roughly half of the benefit of full tuning without target\-specific search\.

## 6Recommendation System for Practitioners

The preceding analyses show that the best algorithm depends on the dataset\. We therefore train a dataset\-conditioned recommender that uses observable properties of a continuous\-action dataset and its environment to rank ten offline policy\-learning algorithms and return the three most promising\.

Prediction task\.For each algorithm–dataset pair, the target is the normalized mean performance of the trials in the top25%25\\%,

ya,d=R¯a,dtop​\-​25%−RdrandomRddataset−Rdrandom,y\_\{a,d\}=\\frac\{\\overline\{R\}\_\{a,d\}^\{\\,\\mathrm\{top\}\\text\{\-\}25\\%\}\-R\_\{d\}^\{\\mathrm\{random\}\}\}\{R\_\{d\}^\{\\mathrm\{dataset\}\}\-R\_\{d\}^\{\\mathrm\{random\}\}\},whereRddatasetR\_\{d\}^\{\\mathrm\{dataset\}\}is the mean trajectory return in the dataset\. Each dataset is one training example, with target vector𝐲d=\(y1,d,…,y10,d\)\\mathbf\{y\}\_\{d\}=\(y\_\{1,d\},\\ldots,y\_\{10,d\}\)\. The recommender predicts this vector and returns the three algorithms with the highest predicted scores\.

Features include dataset size, episode statistics, observation and action dimensionality, task domain, expert source, reward type, and trajectory\-return statistics\. We exclude collection names, dataset identities, exact data\-generation algorithms, and benchmark results\. Missing categorical values are represented explicitly as unknown, while numerical values are imputed within each training fold\.

Candidate regressors\.We evaluate Ridge, Elastic Net, partial least squares,kk\-nearest neighbors, RBF\-SVR, random forest, histogram gradient boosting, and ExtraTrees\. These models span linear, low\-rank, local, kernel, and tree\-based approaches suited to our small tabular dataset; we omit neural predictors because only 57 dataset\-level examples are available\.

Algorithm 1Evaluation of each candidate recommender1:for allrecommender models

mmdo

2:for allenvironment families

ggdo

3:Hold out every dataset from family

gg
4:Tune

mmon the remaining datasets using three\-fold grouped cross\-validation

5:Refit

mmon all remaining datasets

6:Predict scores for the ten algorithms on each held\-out dataset

7:Recommend the three algorithms with the highest predicted scores

8:endfor

9:Evaluate all out\-of\-family recommendations

10:endfor

Evaluation protocol\.We evaluate on 57 datasets from 30 environment families using leave\-one\-environment\-family\-out validation, as summarized in Algorithm[1](https://arxiv.org/html/2609.13730#alg1)\. All variants of the target environment are removed together\. Model selection, preprocessing, and hyperparameter tuning occur entirely within the outer training split, so the held\-out family is used only for final evaluation\.

We compare against a fold\-local global\-mean baseline, which recommends the three algorithms with the highest mean performance on the outer training datasets regardless of target characteristics\. This represents the common strategy of selecting whichever algorithms are strongest overall\.

We report top\-three overlap and best\-of\-three regret\. Top\-three overlap is the fraction of the true top three recovered by the recommendation\. Best\-of\-three regret measures the score lost after trying all three recommendations,

regretd=maxa⁡ya,d−maxa∈𝒜^d\(3\)⁡ya,d,\\operatorname\{regret\}\_\{d\}=\\max\_\{a\}y\_\{a,d\}\-\\max\_\{a\\in\\widehat\{\\mathcal\{A\}\}\_\{d\}^\{\(3\)\}\}y\_\{a,d\},where𝒜^d\(3\)\\widehat\{\\mathcal\{A\}\}\_\{d\}^\{\(3\)\}is the recommended set\. Regret is low whenever at least one recommended algorithm performs close to the true winner, even if the complete ordering is imperfect\.

Figure 20:Algorithm\-recommendation performance under leave\-one\-environment\-family\-out validation\. The global\-mean baseline recommends the fold\-local leaderboard to every target, while the learned models condition on observable dataset features and are tuned entirely within each outer training split\. Higher top\-three overlap and lower best\-of\-three regret are better\. Error bars show95%95\\%environment\-clustered bootstrap confidence intervals\.Main recommendation results\.Figure[20](https://arxiv.org/html/2609.13730#S6.F20)shows that both random forest and ExtraTrees improve substantially over the global leaderboard\. Random forest increases top\-three overlap from0\.4090\.409to0\.4850\.485, includes the true winner among its recommendations on63\.2%63\.2\\%rather than47\.4%47\.4\\%of datasets, and reduces best\-of\-three regret from0\.0910\.091to0\.0300\.030\. It also improves NDCG@3 from0\.7690\.769to0\.8470\.847and Kendall’sτ\\taufrom0\.3240\.324to0\.4010\.401\.

ExtraTrees obtains slightly stronger point estimates on the main evaluation, with0\.5030\.503overlap and0\.0230\.023regret, but the two tree models are statistically indistinguishable\. Its overlap advantage over random forest is0\.0180\.018, with a paired95%95\\%confidence interval of\[−0\.035,0\.077\]\[\-0\.035,0\.077\], and its regret advantage is similarly uncertain\. Relative to the global baseline \(ranked by mean performance across all training datasets\), random forest improves overlap by0\.0760\.076and reduces regret by0\.0610\.061; the paired confidence interval for the regret reduction is\[0\.011,0\.130\]\[0\.011,0\.130\]\.

Figure 21:Impurity\-based feature importance for the random\-forest recommender\. Expert source and mean episode length provide the strongest predictive signal, followed by observation dimensionality, task domain, and properties of the return distribution\.What information supports the recommendations?Figure[21](https://arxiv.org/html/2609.13730#S6.F21)shows that expert source and mean episode length are the most informative features, followed by observation dimensionality, return skew, and task domain\. Several additional return\-distribution statistics provide secondary signal, whereas dataset size contributes relatively little\. Because expert source is the strongest feature, the source\-held\-out evaluation below is an important test of whether the recommender can generalize when this information has not been observed during training\.

Figure 22:Transfer to minority expert\-source categories\. The top row removes each target source from training; the bottom row additionally removes all training datasets from environment families represented in the corresponding test set\. “Index” aggregates the 25 datasets across Human, BC\-clone, RL\-plus\-planner, and Scripted sources\. Error bars show95%95\\%environment\-clustered bootstrap confidence intervals\.Transfer to unseen expert sources\.We stress\-test the recommender on four minority source categories: Human demonstrations, BC clones, RL\-plus\-planner data, and Scripted data\. Each test dataset is first predicted using a model trained without any datasets from its source category\. As shown in Figure[22](https://arxiv.org/html/2609.13730#S6.F22), random forest obtains0\.4270\.427aggregate top\-three overlap, includes the winner on64\.0%64\.0\\%of datasets, and achieves regret of0\.0730\.073\. ExtraTrees obtains0\.2670\.267overlap,28\.0%28\.0\\%winner coverage, and0\.2020\.202regret, while the global baseline obtains0\.3870\.387,32\.0%32\.0\\%, and0\.1940\.194, respectively\.

In the stricter test, we also remove every training dataset from environment families represented in the test set\. Random forest retains0\.4130\.413overlap and64\.0%64\.0\\%winner coverage, compared with0\.3600\.360and56\.0%56\.0\\%for ExtraTrees\.

We do not perform a corresponding source\-held\-out test for RL\-generated data\. These datasets account for 32 of the 57 datasets, contain all locomotion coverage, and supply much of the observed variation in trajectory and action dimensionality\. Removing them would therefore confound expert\-source shift with task\-domain and feature\-support shifts\. Generalization among RL\-generated datasets remains covered by the main environment\-family holdout\.

Figure 23:Recommendation quality as test features are randomly marked unknown\. The tree models remain effective with half of the features missing\. When fewer than35%35\\%are available, the recommender explicitly falls back to the fold\-local global leaderboard, causing all methods to converge to the baseline at high missingness\.Incomplete dataset information\.Figure[23](https://arxiv.org/html/2609.13730#S6.F23)shows that the tree recommenders remain stable under moderate feature masking\. With50%50\\%of features unknown, random forest obtains0\.4970\.497top\-three overlap and0\.0370\.037regret, compared with0\.5150\.515and0\.0440\.044for ExtraTrees\. At higher missingness, both models converge to the explicit global\-leaderboard fallback rather than making poorly supported predictions\.

Figure 24:Reliability of ensemble agreement as a recommendation\-confidence measure\. Left: empirical top\-three frequency against the fraction of trees placing an algorithm in the top three\. Right: Brier score and calibration error\. Random forest is better calibrated\.Interpreting confidence\.Tree agreement provides a measure of how stable a recommendation is across the ensemble\. Random forest produces better\-calibrated agreement scores than ExtraTrees, with a Brier score of0\.1880\.188rather than0\.2130\.213and expected calibration error of0\.0720\.072rather than0\.1330\.133\.

We select random forest as the final recommender because it appears to be the most robust, while providing strong predictive performance\.

## 7Takeaways and Discussion

Reliable evaluation and comparison have been longstanding challenges across machine learning\. A benchmark should do more than produce a single ordering of methods: it should reveal where each method succeeds or fails, and how strongly its conclusions depend on tuning, randomness, and benchmark composition\. We distill our findings into practical guidance for developing, evaluating, and reporting offline policy\-learning methods\.

Pilot experiments\.Early in a project, researchers commonly compare a proposed method with a small set of baselines to decide whether further investment is justified\. Such pilots are only informative when the baselines are competitive; otherwise, an apparent improvement may reflect weak configurations rather than a better method\. Figure[1](https://arxiv.org/html/2609.13730#S0.F1)shows that tuning can substantially improve published baselines and change their relative ordering\. JumpStart provides a consistent set of strong, downloadable baselines and their configurations, allowing new methods to be compared against competitive reference points without first reproducing a large tuning sweep\. We hope this makes pilot experiments more informative and allows unpromising directions to be identified earlier\.

Hyperparameters\.Hyperparameters should be treated as part of a method’s evaluation rather than as incidental implementation details\. Studies should report their search spaces, search procedure, tuning budget, and model\-selection rule, along with the hyperparameters to which performance is most sensitive\. Repeating a fixed configuration across seeds answers a different question from exploring additional configurations and cannot compensate for an inadequately searched space\. In our experiments, 25–50 trials provides a useful practical compromise, although continuous\-control rankings remain sensitive to further tuning\. When target\-specific tuning is infeasible, the transferred defaults developed in Section[5\.2](https://arxiv.org/html/2609.13730#S5.SS2)provide a stronger starting point than existing library defaults, but they should not be interpreted as a replacement for tuning when accurate comparisons are required\.

Evaluation suites and methodology\.Broader evaluation generally produces more stable conclusions, although size alone does not guarantee a representative benchmark\. For context, the main Decision Transformer comparisons contain nine D4RL locomotion datasets and four Atari games\[chen2021decision\]; d3rlpy evaluates 17 datasets\[d3rlpy\]; and the unified evaluations ofkang2023improvingandtarasov2023revisitingcover 26 and 30 tasks or datasets, respectively\. Our results show that subsets of only 10–12 environment–dataset pairs can produce conflicting winners, and that even similarly sized suites can support different conclusions depending on their composition\. Researchers should therefore report the complete benchmark composition, use as broad a suite as practical, and disaggregate results across important dataset properties\. In particular, rankings transfer poorly across expert sources, so evaluations containing human, scripted, planned, or RL\-generated data should report these regimes separately rather than relying only on a pooled aggregate\.

Evaluation protocols must also be consistent across methods\. We recommend robust aggregate metrics such as IQM and optimality gap, accompanied by uncertainty estimates and per\-dataset results\[agarwal2021deep\]\. Baseline scores should be obtained using the same environment versions, wrappers, normalization procedure, checkpoint\-selection rule, and evaluation budget as the proposed method\. Numbers should not be copied from prior papers unless these details are known to match, particularly when environment versions differ\. By releasing trained policies, their scores and hyperparameters, and the corresponding training and evaluation code, JumpStart allows baselines to be reevaluated as protocols change rather than leaving their published scores permanently tied to one implementation\.

Choosing an algorithm\.Our results do not support a single universally best algorithm\. Performance depends on observable properties of the dataset and environment, and rankings can change across task domains and expert sources\. The practitioner recommender in Section[6](https://arxiv.org/html/2609.13730#S6)uses these properties to produce a shortlist of three promising algorithms rather than asserting a universal winner\. This shortlist is intended to guide an initial pilot: when its recommendations are confident, it can reduce the number of methods that must be tested; when confidence is low or task information is limited, the global leaderboard remains a safer fallback\.

Living benchmarks and leaderboards\.Static leaderboards become outdated as algorithms, datasets, and evaluation practices change\. We therefore release an open\-source website and extensible result database that support external submissions and self\-hosting\. New algorithms and datasets can be added while retaining the policies, scores, hyperparameters, and evaluation provenance underlying each entry\. Because we release every trained policy, existing results can also be reevaluated as environments, metrics, and evaluation practices evolve\. This allows the benchmark to develop with the field while keeping its comparisons reproducible and auditable\.

## 8Conclusion

We presented JumpStart, an extensively tuned comparison of ten offline policy\-learning algorithms across 114 datasets and more than 160,000 trained policies\. Our results show that algorithm rankings depend substantially on hyperparameter tuning, benchmark composition, and dataset characteristics, limiting the conclusions that can be drawn from small or weakly tuned evaluations\. We translate these findings into practical tuning guidance, stronger default configurations, and dataset\-conditioned algorithm recommendations\. By releasing every trained policy with its metadata and code through an extensible leaderboard, JumpStart can incorporate new results and reevaluate existing models as evaluation practices change, providing an evolving foundation for future offline policy\-learning research\.

Table 4:Hyperparameter search spaces for continuous\-control algorithms\. Fixed parameters are omitted\.*Notation:*U\\mathrm\{U\}is uniform,LogU\\mathrm\{LogU\}is log\-uniform,Int\\mathrm\{Int\}is inclusive integer, and braces denote uniformly sampled choices\.

BC

BCP

BCQ

CQL

DT

IQL

ReBRAC

ACT

VQ\-BeT

Diffusion Policy

Table 5:Hyperparameter search spaces for discrete\-control algorithms\. Fixed parameters are omitted\.*Notation:*U\\mathrm\{U\}is uniform,LogU\\mathrm\{LogU\}is log\-uniform,Int\\mathrm\{Int\}is inclusive integer, and braces denote uniformly sampled choices\.

BC

BCP

BCQ

CQL

DT

Table 6:Normalized scores on continuous\-control datasets, grouped by benchmark collection\. Entries report the point estimate±\\pmhalf\-width of its 95% confidence interval\. Dataset rows summarize the top 25% of evaluated hyperparameter configurations; shaded rows report collection\-level interquartile mean \(IQM;↑\\uparrow\) and optimality gap \(↓\\downarrow\)\. The best point estimate in each row is bold\.Table 7:Normalized scores on Atari environments\. Entries report the point estimate±\\pmhalf\-width of its 95% confidence interval\. Environment rows summarize the top 25% of evaluated hyperparameter configurations; shaded rows report the collection\-level interquartile mean \(IQM;↑\\uparrow\) and optimality gap \(↓\\downarrow\)\. The best point estimate in each row is bold\.Table 8:Hyperparameter sensitivity in continuous control\. Within each environment,ρ\\rhois the Spearman correlation between the hyperparameter and trial score, and Share is its L1\-normalized magnitude,\|ρh\|/∑j\|ρj\|\|\\rho\_\{h\}\|/\\sum\_\{j\}\|\\rho\_\{j\}\|\. Entries report the median magnitude and share across 57 environments; rows are ordered by median\|ρ\|\|\\rho\|\. Because each statistic is aggregated independently, the displayed shares need not sum to 100%\.BC

BCP

BCQ

CQL

DT

IQL

ReBRAC

ACT

VQ\-BeT

Diffusion

Table 9:Hyperparameter sensitivity in discrete control \(Atari\)\. Within each environment,ρ\\rhois the Spearman correlation between the hyperparameter and trial score, and Share is its L1\-normalized magnitude,\|ρh\|/∑j\|ρj\|\|\\rho\_\{h\}\|/\\sum\_\{j\}\|\\rho\_\{j\}\|\. Entries report the median magnitude and share across 54 environments; rows are ordered by median\|ρ\|\|\\rho\|\. Because each statistic is aggregated independently, the displayed shares need not sum to 100%\.BC

BCP

BCQ

CQL

DT

## References

## Appendix ADatasets

We describe the collection process for each dataset used in our analysis\. We did not create or contribute these datasets, but include these details as they may be relevant to readers\. Dataset identifiers below follow the Minari registry\. Our benchmark contains 114 environment–dataset pairs: 29 from the Minari release of D4RL \(commit 6c51941431a72f66721e5b4ed92c7ce6f116c816\), 28 from the Minari MuJoCo collection \(commit 8e62dc7f7fcb4a19f8f869c65402d4bb60049117\), and 57 from our larger reproduction of Minari’s Atari collection, the 100,000\-transition omi\-n/atari\-medium collection\.

AntMaze\.The six AntMaze datasets require a MuJoCo Ant to navigate a maze under sparse reward\. They were collected with a hierarchical controller: a Q\-iteration planner selected waypoints and a goal\-conditioned SAC policy supplied the low\-level actions\[fu2020d4rl,haarnoja2018softactorcriticoffpolicymaximum\]\. umaze\-v1 uses a fixed start and goal, whereas umaze\-diverse\-v1 varies both over a set of valid maze cells\. The medium and large layouts each have a play dataset, built from hand\-selected start–goal pairs, and a diverse dataset, which samples both starts and goals more broadly\. These give the four datasets medium\-play\-v1, medium\-diverse\-v1, large\-play\-v1, and large\-diverse\-v1\.

Adroit Door\.Door uses a 24\-degree\-of\-freedom Adroit hand to unlatch and open a door\. human\-v2 contains 25 teleoperated human demonstrations\. expert\-v2 contains rollouts of a policy trained by demo\-augmented policy gradient \(DAPG\)\[rajeswaran2018learning\]\. cloned\-v2 is deliberately imperfect: a behavioral\-cloning policy was trained on the union of the human and expert data, and its rollouts were mixed approximately equally with the source demonstrations\[fu2020d4rl\]\.

Adroit Hammer\.Hammer requires the Adroit hand to pick up a hammer and drive a nail\. Its human\-v2, expert\-v2, and cloned\-v2 datasets were collected in the same three regimes as Door: 25 human teleoperation demonstrations, DAPG\-policy rollouts, and an approximately equal mixture of source demonstrations and rollouts from a behavioral clone, respectively\[rajeswaran2018learning,fu2020d4rl\]\.

Adroit Pen\.Pen requires reorienting a pen to a target pose\. human\-v2 contains 25 human teleoperation demonstrations, expert\-v2 contains DAPG\-policy rollouts, and cloned\-v2 combines the source demonstrations with rollouts of a behavioral clone trained on the human and expert data\[rajeswaran2018learning,fu2020d4rl\]\.

Adroit Relocate\.Relocate requires the Adroit hand to move an object to a target position\. As in the other Adroit environments, human\-v2 contains 25 teleoperated demonstrations, expert\-v2 contains DAPG\-policy trajectories, and cloned\-v2 mixes the source demonstrations approximately equally with trajectories from a behavioral\-cloning policy\[rajeswaran2018learning,fu2020d4rl\]\.

Kitchen\.The three Franka Kitchen datasets contain human demonstrations collected through virtual\-reality teleoperation\[gupta2019relay,fu2020d4rl\]\. Each trajectory may include interactions with several appliances or fixtures\. In complete\-v2, all four target subtasks are completed in order\. partial\-v2 contains mixed\-task demonstrations with sub\-trajectories that complete the target subtasks in sequence, while mixed\-v2 contains broader subtask combinations in which the four target subtasks are never all completed in sequence\. Unlike the locomotion datasets below, these variants therefore differ in task coverage rather than in the training checkpoint of a single behavior policy\.

PointMaze—Open\.PointMaze controls a point mass rather than a legged robot\. A hand\-coded proportional–derivative controller followed waypoints produced by a Q\-iteration planner toward sampled goals\[fu2020d4rl,minari\]\. open\-v2 records the sparse\-reward version of the open layout, and open\-dense\-v2 uses dense distance\-based reward; the behavior policy is otherwise the same\.

PointMaze—U\-Maze\.umaze\-v2 and umaze\-dense\-v2 were collected in the U\-shaped maze with the same scripted controller and Q\-iteration planner\. The former uses sparse goal reward and the latter dense shaped reward\.

PointMaze—Medium\.medium\-v2 and medium\-dense\-v2 increase the size and path complexity of the maze\. Both were generated by the scripted planner–controller pair; they differ only in whether the recorded reward is sparse or dense\.

PointMaze—Large\.large\-v2 and large\-dense\-v2 use the largest PointMaze layout in the benchmark\. As above, a Q\-iteration planner supplied waypoints to a proportional–derivative controller, and the two datasets use sparse and dense reward, respectively\.

MuJoCo collection\.The MuJoCo datasets were generated by rolling out checkpoints of policies trained in the corresponding Gymnasium environments\[todorov2012mujoco,minari\]\. The behavior policies were implemented with Stable\-Baselines3 and use SAC, PPO, or TQC\[stable\-baselines3,haarnoja2018softactorcriticoffpolicymaximum,schulman2017proximalpolicyoptimizationalgorithms,pmlr\-v119\-kuznetsov20a\]\. The labels simple, medium, and expert denote increasing training budgets, not mixtures of trajectories from policies of different quality\. Except where noted below, the variants for an environment are checkpoints from one training run\.

Ant\.Ant is a quadrupedal locomotion task\. simple\-v0 and medium\-v0 were collected from SAC checkpoints after 1 and 10 million training steps\. expert\-v0 was generated separately from an externally trained SAC checkpoint selected as the best pretrained model\.

HalfCheetah\.HalfCheetah asks a planar articulated agent to run forward\. Its three datasets use truncated quantile critics \(TQC\), a distributional variant of SAC: simple\-v0, medium\-v0, and expert\-v0 correspond to checkpoints after 1, 6, and 25 million training steps\.

Hopper\.Hopper is one\-legged forward locomotion\. The simple\-v0, medium\-v0, and expert\-v0 datasets were generated from SAC checkpoints after 1, 4, and 25 million training steps, respectively\.

Walker2d\.Walker2d is planar bipedal locomotion\. Its simple\-v0, medium\-v0, and expert\-v0 datasets use SAC checkpoints after 1\.5, 6, and 25 million training steps\.

Swimmer\.Swimmer propels a three\-link body through fluid by actuating its joints\. It has no simple variant\. medium\-v0 and expert\-v0 were collected from PPO checkpoints after 300 thousand and 100 million training steps\. PPO was trained with an undiscounted objective because SAC\-style discounting was ineffective for this environment\.

Reacher\.Reacher asks a two\-link arm to move its end effector to a target\. It has two SAC\-generated datasets: medium\-v0 uses a checkpoint after 200 thousand training steps and expert\-v0 a checkpoint after 10 million steps\.

Pusher\.Pusher requires a robotic arm to push an object toward a target\. medium\-v0 and expert\-v0 were generated from SAC checkpoints after 1 and 10 million training steps; no simple variant is provided\.

Humanoid\.Humanoid is high\-dimensional bipedal locomotion\. Its trajectories were collected from TQC policies: simple\-v0, medium\-v0, and expert\-v0 use checkpoints after 2, 5, and 20 million training steps\.

HumanoidStandup\.HumanoidStandup requires a fallen humanoid to rise to an upright posture\. The behavior policy was SAC trained with a reduced observation representation\. simple\-v0, medium\-v0, and expert\-v0 were collected after 2, 10, and 50 million training steps\.

InvertedPendulum\.InvertedPendulum requires balancing a pole on a moving cart\. medium\-v0 and expert\-v0 contain rollouts of SAC checkpoints after 10 thousand and 1 million training steps, respectively\.

InvertedDoublePendulum\.InvertedDoublePendulum extends the balancing task to two hinged links\. Its medium\-v0 and expert\-v0 datasets were collected from SAC checkpoints after 50 thousand and 10 million training steps\.

Atari\.The discrete\-control portion of the benchmark contains one expert\-v0 dataset for each of 57 Atari games in the Arcade Learning Environment\[Bellemare\_2013\]: Alien, Amidar, Assault, Asterix, Asteroids, Atlantis, Bank Heist, Battle Zone, Beam Rider, Berzerk, Bowling, Boxing, Breakout, Centipede, Chopper Command, Crazy Climber, Defender, Demon Attack, Double Dunk, Enduro, Fishing Derby, Freeway, Frostbite, Gopher, Gravitar, H\.E\.R\.O\., Ice Hockey, James Bond, Kangaroo, Krull, Kung\-Fu Master, Montezuma’s Revenge, Ms\. Pac\-Man, Name This Game, Phoenix, Pitfall\!, Pong, Private Eye, Q\*bert, River Raid, Road Runner, Robot Tank, Seaquest, Skiing, Solaris, Space Invaders, Star Gunner, Surround, Tennis, Time Pilot, Tutankham, Up ’n Down, Venture, Video Pinball, Wizard of Wor, Yars’ Revenge, and Zaxxon\. For each game, the same pretrained CleanBA PPO policy with an IMPALA\-style convolutional encoder that Minari used was rolled out in the corresponding ALE environment using the original Minari data generation scripts until a target of 100,000 transitions was reached\[huang2021cleanrlhighqualitysinglefileimplementations,schulman2017proximalpolicyoptimizationalgorithms,espeholt2018impalascalabledistributeddeeprl\]\. Actions were sampled stochastically from the policy logits rather than chosen greedily, and sticky actions were disabled\. The policy received resized grayscale stacks of four frames, but collection occurred before those wrappers, so the datasets retain raw image observations\. The atari\-medium label describes dataset size; all 57 datasets use the same expert\-policy collection procedure and do not represent medium\-quality behavior\.

## Appendix BOriginal Paper Baselines

In Figure[1](https://arxiv.org/html/2609.13730#S0.F1), we compare our trained models with the original authors’ implementations, each trained across three random seeds\. We use the official repositories for BCQ, CQL, DT, IQL, and ReBRAC\.444Repositories:[BCQ](https://github.com/sfujim/BCQ),[CQL](https://github.com/aviralkumar2907/CQL),[DT](https://github.com/kzl/decision-transformer),[IQL](https://github.com/ikostrikov/implicit_q_learning), and[ReBRAC](https://github.com/DT6A/ReBRAC)\.

To preserve the original training implementations, we restrict our modifications to data\-ingestion adapters and environment construction\. For CQL, we load the Minari HDF5 data and insert its transitions into rlkit’s EnvReplayBuffer\. For DT, we convert each Minari dataset to the expected pickle format\. For BCQ, we reconstruct episode buffers from Minari’s stored episodes and load the observations, actions, rewards, terminal indicators, and next observations into BCQ’s ReplayBuffer\. We apply the same principle to the remaining methods, converting each dataset into the format expected by the original framework while leaving the algorithm and training procedure unchanged\. This avoids introducing unintended differences in preprocessing, numerical precision, or trajectory construction\.

To ensure that all baselines are evaluated consistently, we adapt each repository’s evaluation procedure to match our framework: we evaluate only the final checkpoint over 100 episodes\. We reconstruct the exact environment and version associated with each Minari dataset and add any omitted constructor arguments or wrappers according to Minari’s data\-generation scripts\. Because most original implementations use the Gym API\[brockman2016openaigym\], these changes are limited primarily to environment construction and evaluation, leaving the underlying policies and training procedures unchanged\.

## Appendix CHyperparameter Sensitivity Breakdown

We provide detailed heatmaps of the hyperparameter sensitivities from Section[5](https://arxiv.org/html/2609.13730#S5)in Figure[25](https://arxiv.org/html/2609.13730#A3.F25)\.

![Refer to caption](https://arxiv.org/html/2609.13730v1/act.png)

![Refer to caption](https://arxiv.org/html/2609.13730v1/bc.png)

Figure 25:Per\-environment hyperparameter\-sensitivity breakdowns for the continuous\-control algorithms\. Each heatmap reports relative absolute Spearman associations between hyperparameters and trial performance\. Zoom in for environment and hyperparameter labels\.![Refer to caption](https://arxiv.org/html/2609.13730v1/bcp.png)

![Refer to caption](https://arxiv.org/html/2609.13730v1/bcq.png)

Figure 26:Hyperparameter\-sensitivity breakdowns \(continued\)\.![Refer to caption](https://arxiv.org/html/2609.13730v1/cql.png)

![Refer to caption](https://arxiv.org/html/2609.13730v1/diffusion.png)

Figure 27:Hyperparameter\-sensitivity breakdowns \(continued\)\.![Refer to caption](https://arxiv.org/html/2609.13730v1/dt.png)

![Refer to caption](https://arxiv.org/html/2609.13730v1/iql.png)

Figure 28:Hyperparameter\-sensitivity breakdowns \(continued\)\.![Refer to caption](https://arxiv.org/html/2609.13730v1/rebrac.png)

![Refer to caption](https://arxiv.org/html/2609.13730v1/vqbet.png)

Figure 29:Hyperparameter\-sensitivity breakdowns \(continued\)\.