Principled Analysis of Deep Reinforcement Learning Evaluation and Design Paradigms

# Principled Analysis of Deep Reinforcement Learning Evaluation and Design Paradigms
Source: [https://arxiv.org/html/2607.07769](https://arxiv.org/html/2607.07769)
###### Abstract

Starting from the utilization of deep neural networks to approximate the state\-action value function that led to winning one of the most challenging games, to algorithmic advancements that allowed solving problems without even explicitly stating the rules of the challenge at hand, reinforcement learning research has been the center of remarkable scientific progress for the past decade\. In this paper, we focus on the key ingredients of this research progress and we analyze the canonical evaluation and design paradigms in reinforcement learning\. We introduce the theoretical foundations of scaling laws in reinforcement learning and show that the asymptotic performance of reinforcement learning algorithms does not have a monotone relationship between performance rankings and data\-regimes\. We conduct large\-scale experiments and our results demonstrate that a line of reinforcement learning research under the canonical design and evaluation paradigms resulted in incorrect conclusions\. Our analysis and results provide a core analysis on scaling, capacity and complexity of deep reinforcement learning\.

## 1Introduction

Founded on rigorous theoretical guarantees, reinforcement learning research achieved high acceleration upon the proposal of the initial study on approximating the state\-action value function via deep neural networks\(Mnihet al\.[2015](https://arxiv.org/html/2607.07769#bib.bib126); Stiennonet al\.[2020](https://arxiv.org/html/2607.07769#bib.bib84); Schrittwieseret al\.[2020](https://arxiv.org/html/2607.07769#bib.bib19); Leeet al\.[2024](https://arxiv.org/html/2607.07769#bib.bib85); Korkmaz[2025](https://arxiv.org/html/2607.07769#bib.bib51)\)\. A line of highly successful deep reinforcement learning algorithms have been proposed\(Hasseltet al\.[2016](https://arxiv.org/html/2607.07769#bib.bib72); Wanget al\.[2016](https://arxiv.org/html/2607.07769#bib.bib163); Hesselet al\.[2018](https://arxiv.org/html/2607.07769#bib.bib61),[2021](https://arxiv.org/html/2607.07769#bib.bib48); Kapturowskiet al\.[2023](https://arxiv.org/html/2607.07769#bib.bib21); Korkmaz[2024](https://arxiv.org/html/2607.07769#bib.bib42)\)from focusing on different architectural ideas to foundations targeting overestimation, all of which were designed and tested in the high\-data regime, i\.e\. two hundred million frame training\. An alternative recent line of research with an extensive amount of publications focused on pushing the performance bounds of deep reinforcement learning policies in the low\-data regime, i\.e\. with one hundred thousand environment interaction training\. Many different concepts in current reinforcement learning research, from architectural proposals to learning underlying dynamics of the environment, experienced accelerated progress and significant attention, growing into several major research fields, solely based on policy performance comparisons demonstrated in the low\-data regime benchmark\.

In this paper, we focus on evaluation paradigms, implicit assumptions and canonical methodological choices made in deep reinforcement learning research and demonstrate that there is a significant overlooked underlying premise driving this line of research without being explicitly discussed: that the performance profiles of deep reinforcement learning algorithms have a monotonic relationship with different sample\-complexity regimes\. We show that this implicit assumption, that is commonly shared amongst a large collection of low\-data regime studies, shapes how the canonical design and evaluation choices are made in deep reinforcement learning research and represents a prominent misdirection in scientific progress\. The suboptimal conclusions obtained from these canonical choices shape future research directions with incorrect reasoning\. We show that these methodological decisions fuel incorrect justifications and conclusions, thereby misdirecting research efforts toward certain concepts for several years\. Thus, in our paper we target these underlying premises and aim to answer the following questions:

*What are the implicit assumptions and canonical choices in deep reinforcement learning research that fundamentally affect the conclusions made?*

*What is the foundational relationship between sample complexity and the algorithmic performance from the data\-scarce regime to the asymptotic regime?*

Hence, to be able to answer the questions raised above, in our paper we focus on underlying design and evaluation paradigms in deep reinforcement learning and make the following contributions:

- •We analyze the evaluation paradigms and canonical methodological choices in deep reinforcement learning research, and introduce the theoretical foundations on how these methodological choices affect algorithm design, performance comparisons and algorithmic conclusions\. Our analysis lays the foundations on scaling, capacity and complexity of deep reinforcement learning\.
- •Our theoretical analysis proves that the performance profile has a non\-monotonic relationship with the asymptotic sample complexity and the low\-data sample complexity regime\. Regarding the central focus of the large scale implicit assumption instances, our results reveal that the canonical methodological choices made in a line of deep reinforcement learning research have led to incorrect justifications and conclusions\.
- •We conduct large scale extensive experiments for a comprehensive and a diverse portfolio of deep reinforcement learning baseline algorithms in both the low\-data regime and the high\-data regime Arcade Learning Environment benchmark\. Our results demonstrate that recent algorithms proposed and evaluated in the Arcade Learning Environment 100K benchmark are significantly affected by the implicit assumption on the relationship between performance profiles and sample complexity resulting in systematic bias in algorithmic evaluation\.

## 2Background and Preliminaries

The reinforcement learning problem is formalized as a Markov Decision Process \(MDP\) represented as a tuple⟨S,A,𝒫,ℛ,γ,ρ0⟩\\langle S,A,\\mathcal\{P\},\\mathcal\{R\},\\gamma,\\rho\_\{0\}\\ranglewhereSSrepresents the state space,AArepresents the set of actions,𝒫\\mathcal\{P\}represents the transition probability distribution onS×A×SS\\times A\\times S,ℛ:S×A→ℝ\\mathcal\{R\}:S\\times A\\to\\mathbb\{R\}represents the reward function, andγ∈\(0,1\]\\gamma\\in\(0,1\]represents the discount factor\. The aim in reinforcement learning is to learn an optimal policyπ​\(s,a\)\\pi\(s,a\)that maps state observations to actionsπ:S→Δ​\(A\)\\pi:S\\to\\Delta\(A\), which maximizes the expected cumulative discounted rewardsR=𝔼at∼π​\(st,⋅\)​∑tγt​ℛ​\(st,at,st\+1\)R=\\mathbb\{E\}\_\{a\_\{t\}\\sim\\pi\(s\_\{t\},\\cdot\)\}\\sum\_\{t\}\\gamma^\{t\}\\mathcal\{R\}\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\)\. This objective is achieved by constructing a state\-action value function that learns for each state\-action pair the expected cumulative discounted rewards that will be obtained if actiona∈Aa\\in Ais executed in states∈Ss\\in S\.

𝒬​\(s,a\)=∑s′𝒫​\(s′\|s,a\)​\[ℛ​\(s,a,s′\)\+γ​𝒱​\(s′\)\]\\mathcal\{Q\}\(s,a\)=\\sum\_\{s^\{\\prime\}\}\\mathcal\{P\}\(s^\{\\prime\}\|s,a\)\[\\mathcal\{R\}\(s,a,s^\{\\prime\}\)\+\\gamma\\mathcal\{V\}\(s^\{\\prime\}\)\]In settings where the state space and/or action space is large enough that the state\-action value function𝒬​\(s,a\)\\mathcal\{Q\}\(s,a\)cannot be held in a tabular form, a function approximator is used\. Thus, for deep reinforcement learning the𝒬\\mathcal\{Q\}\-function is approximated via deep neural networks

θt\+1=θt\+α\(\\displaystyle\\theta\_\{t\+1\}=\\theta\_\{t\}\+\\alpha\(ℛ​\(st,at,st\+1\)\\displaystyle\\mathcal\{R\}\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\)\+γ​𝒬​\(st\+1,arg​maxa⁡𝒬​\(st\+1,a;θt\);θt\)\\displaystyle\+\\gamma\\mathcal\{Q\}\(s\_\{t\+1\},\\operatorname\*\{arg\\,max\}\_\{a\}\\mathcal\{Q\}\(s\_\{t\+1\},a;\\theta\_\{t\}\);\\theta\_\{t\}\)−𝒬\(st,at;θt\)\)∇θt𝒬\(st,at;θt\)\.\\displaystyle\\qquad\\qquad\\qquad\\qquad\\qquad\-\\mathcal\{Q\}\(s\_\{t\},a\_\{t\};\\theta\_\{t\}\)\)\\nabla\_\{\\theta\_\{t\}\}\\mathcal\{Q\}\(s\_\{t\},a\_\{t\};\\theta\_\{t\}\)\.Dueling Architecture:The dueling architecture\(Wanget al\.[2016](https://arxiv.org/html/2607.07769#bib.bib163)\)outputs two streams of fully connected layers for both estimating the advantage𝒜​\(s,a\)\\mathcal\{A\}\(s,a\)for each action in a given statess,𝒜​\(s,a\)=𝒬​\(s,a\)−maxa⁡𝒬​\(s,a\)\\mathcal\{A\}\(s,a\)=\\mathcal\{Q\}\(s,a\)\-\\max\_\{a\}\\mathcal\{Q\}\(s,a\)and the state values𝒱​\(s\)\\mathcal\{V\}\(s\)\. In particular, the last layer of the dueling architecture contains the forward mapping𝒬​\(s,a;θ,α,β\)=𝒱​\(s;θ,β\)\+\(𝒜​\(s,a;θ,α\)−maxa′∈A⁡𝒜​\(s,a′;θ,α\)\)\\mathcal\{Q\}\(s,a;\\theta,\\alpha,\\beta\)=\\mathcal\{V\}\(s;\\theta,\\beta\)\+\\big\(\\mathcal\{A\}\(s,a;\\theta,\\alpha\)\-\\max\_\{a^\{\\prime\}\\in A\}\\mathcal\{A\}\(s,a^\{\\prime\};\\theta,\\alpha\)\\big\)whereθ\\thetarepresents the parameters of the convolutional layers andα\\alphaandβ\\betarepresent the parameters of the fully connected layers outputting the advantage and state value estimates respectively\.

Inherent High\-Capacity Models:The initial algorithm that has been proposed to have inherent high\-capacity is C51\. In particular, the projected Bellman update for theithi^\{\\textrm\{th\}\}atom is computed as

\(Φ​𝒯​𝒵θ​\(st,at\)\)i\\displaystyle\(\\Phi\\mathcal\{T\}\\mathcal\{Z\}\_\{\\theta\}\(s\_\{t\},a\_\{t\}\)\)\_\{i\}=∑j𝒩−1\[1−\|\[𝒯​zj\]vminvmax−zi\|Δ​z\]01\\displaystyle=\\sum\_\{j\}^\{\\mathcal\{N\}\-1\}\\big\[1\-\\dfrac\{\|\[\\mathcal\{T\}z\_\{j\}\]^\{v\_\{\\textrm\{max\}\}\}\_\{v\_\{\\textrm\{min\}\}\}\-z\_\{i\}\|\}\{\\Delta z\}\\big\]^\{1\}\_\{0\}τj​\(st\+1,maxa∈A⁡𝔼​𝒵θ​\(st\+1,a\)\)\\displaystyle\\qquad\\qquad\\quad\\tau\_\{j\}\(s\_\{t\+1\},\\max\_\{a\\in A\}\\mathbb\{E\}\\mathcal\{Z\}\_\{\\theta\}\(s\_\{t\+1\},a\)\)wherezi=vmin\+i​Δ​z:0≤i<𝒩z\_\{i\}=v\_\{\\textrm\{min\}\}\+i\\Delta z:0\\leq i<\\mathcal\{N\}represents the set of atoms in categorical learning, and the atom probabilities are learnt as a parametric model\(Bellemareet al\.[2017](https://arxiv.org/html/2607.07769#bib.bib165)\)

τi​\(st,maxa∈A⁡𝔼​𝒵θ​\(st,a\)\)=eθi​\(st,at\)∑jeθj​\(st,at\)​,​Δ​z:=vmax−vmin𝒩−1\\tau\_\{i\}\(s\_\{t\},\\max\_\{a\\in A\}\\mathbb\{E\}\\mathcal\{Z\}\_\{\\theta\}\(s\_\{t\},a\)\)=\\dfrac\{e^\{\\theta\_\{i\}\(s\_\{t\},a\_\{t\}\)\}\}\{\\sum\_\{j\}e^\{\\theta\_\{j\}\(s\_\{t\},a\_\{t\}\)\}\}\\\>\\\>\\textrm\{,\}\\\>\\\>\\Delta z:=\\dfrac\{v\_\{\\textrm\{max\}\}\-v\_\{\\textrm\{min\}\}\}\{\\mathcal\{N\}\-1\}Following this baseline the𝒬\\mathcal\{Q\}RD𝒬\\mathcal\{Q\}N algorithm\(Dabneyet al\.[2018b](https://arxiv.org/html/2607.07769#bib.bib53)\)is proposed to learn the quantile projection

𝒯𝒵\(st,at\)=ℛ\(\\displaystyle\\mathcal\{T\}\\mathcal\{Z\}\(s\_\{t\},a\_\{t\}\)=\\mathcal\{R\}\(st,at,st\+1\)\\displaystyle s\_\{t\},a\_\{t\},s\_\{t\+1\}\)\+γ​𝒵​\(st\+1,arg​maxa∈A⁡𝔼z∼𝒵​\(st\+1,at\+1\)​\[z\]\)\\displaystyle\+\\gamma\\mathcal\{Z\}\(s\_\{t\+1\},\\operatorname\*\{arg\\,max\}\_\{a\\in A\}\\mathbb\{E\}\_\{z\\sim\\mathcal\{Z\}\(s\_\{t\+1\},a\_\{t\+1\}\)\}\[z\]\)withst\+1∼𝒫\(⋅\|st,at\)s\_\{t\+1\}\\sim\\mathcal\{P\}\(\\cdot\|s\_\{t\},a\_\{t\}\)where𝒵∈Z\\mathcal\{Z\}\\in Zrepresents the quantile distribution of an arbitrary value function\. Following this study the I𝒬\\mathcal\{Q\}N algorithm\(Dabneyet al\.[2018a](https://arxiv.org/html/2607.07769#bib.bib47)\)is proposed to learn the full quantile function instead of learning a discrete set of quantiles as in the𝒬\\mathcal\{Q\}RD𝒬\\mathcal\{Q\}N algorithm\. The I𝒬\\mathcal\{Q\}N algorithm objective is to minimize the loss function

ℒ=1𝒦\\displaystyle\\mathcal\{L\}=\\dfrac\{1\}\{\\mathcal\{K\}\}∑i=1𝒦∑j=1𝒦′ρδ\(ℛ\(st,at,st\+1\)\\displaystyle\\sum\_\{i=1\}^\{\\mathcal\{K\}\}\\sum\_\{j=1\}^\{\\mathcal\{K^\{\\prime\}\}\}\\rho\_\{\\delta\}\(\\mathcal\{R\}\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\)\(1\)\+γ𝒵δj′\(st\+1,arg​maxa∈A𝒬β\(st,at\)\)−𝒵δi\(st,at\)\)\\displaystyle\+\\gamma\\mathcal\{Z\}\_\{\{\\delta^\{\\prime\}\_\{j\}\}\}\(s\_\{t\+1\},\\operatorname\*\{arg\\,max\}\_\{a\\in A\}\\mathcal\{Q\}\_\{\\beta\}\(s\_\{t\},a\_\{t\}\)\)\-\\mathcal\{Z\}\_\{\\delta\_\{i\}\}\(s\_\{t\},a\_\{t\}\)\)whereρδ\\rho\_\{\\delta\}represents the Huber quantile regression loss, and𝒬β=∫01ℱ𝒵−1​\(δ\)​𝑑β​\(δ\)\\mathcal\{Q\}\_\{\\beta\}=\\int^\{1\}\_\{0\}\\mathcal\{F\}^\{\-1\}\_\{\\mathcal\{Z\}\}\(\\delta\)d\\beta\(\\delta\)\. Note that𝒵δ=ℱ𝒵−1​\(δ\)\\mathcal\{Z\}\_\{\\delta\}=\\mathcal\{F\}^\{\-1\}\_\{\\mathcal\{Z\}\}\(\\delta\)is the quantile function of the random variable𝒵\\mathcal\{Z\}atδ∈\[0,1\]\\delta\\in\[0,1\]\.

## 3Low\-data Regime versus Asymptotic Performance

Our paper discovers both with extensive empirical analysis and theoretical investigation that asymptotic performance of reinforcement learning algorithms does not necessarily provide any information nor indication on their relative performance ranking in the low\-data regime\. The results provided in Section[6](https://arxiv.org/html/2607.07769#S6)extensively demonstrate that a large body of work in reinforcement learning research carried this assumption and resulted in incorrect conclusions\. In this section, we introduce the foundational basis for our discovery revealed by our extensive empirical analysis in Section[6](https://arxiv.org/html/2607.07769#S6)in optimization of non\-stationary policies, i\.e\. rewards and transitions that can vary with each step in an episode, in undiscounted, finite\-horizon MDPs with linear function approximation\. In particular, a finite horizon MDP is represented as a tuple⟨S,A,𝒫,ℛ,ℋ⟩\\langle S,A,\\mathcal\{P\},\\mathcal\{R\},\\mathcal\{H\}\\ranglewhereSSis the set of states, andAArepresents the set of actions\. For each time stept∈\[ℋ\]=\{1,…,ℋ\}t\\in\[\\mathcal\{H\}\]=\\\{1,\\dots,\\mathcal\{H\}\\\}, statess, and actionaathe transition probability kernel𝒫t​\(s′\|s,a\)\\mathcal\{P\}\_\{t\}\(s^\{\\prime\}\|s,a\)gives the probability distribution over the next state, and the rewardℛt​\(s,a,s′\)\\mathcal\{R\}\_\{t\}\(s,a,s^\{\\prime\}\)gives the immediate rewards\. A non\-stationary policyπ=\(π1,…,πℋ\)\\pi=\(\\pi\_\{1\},\\dots,\\pi\_\{\\mathcal\{H\}\}\)induces a state\-action value function given by

𝒬tπ​\(s,a\)=𝔼​\[∑h=tℋℛh​\(sh,πh​\(sh\),sh\+1\)\|sh=s,ah=a\]\\displaystyle\\mathcal\{Q\}\_\{t\}^\{\\pi\}\(s,a\)=\\mathbb\{E\}\\left\[\\sum\_\{h=t\}^\{\\mathcal\{H\}\}\\mathcal\{R\}\_\{h\}\(s\_\{h\},\\pi\_\{h\}\(s\_\{h\}\),s\_\{h\+1\}\)\\bigg\|s\_\{h\}=s,a\_\{h\}=a\\right\]where we letah∼πh​\(sh\)a\_\{h\}\\sim\\pi\_\{h\}\(s\_\{h\}\), and the corresponding value function𝒱tπ​\(s\)=𝒬t​\(s,πt​\(s\)\)\\mathcal\{V\}\_\{t\}^\{\\pi\}\(s\)=\\mathcal\{Q\}\_\{t\}\(s,\\pi\_\{t\}\(s\)\)\. The optimal non\-stationary policyπ∗\\pi^\{\*\}has value function𝒱t∗​\(s\)=𝒱tπ∗​\(s\)\\mathcal\{V\}\_\{t\}^\{\*\}\(s\)=\\mathcal\{V\}\_\{t\}^\{\\pi^\{\*\}\}\(s\)satisfying𝒱t∗​\(s\)=supπ𝒱tπ​\(s\)\.\\mathcal\{V\}\_\{t\}^\{\*\}\(s\)=\\sup\_\{\\pi\}\\mathcal\{V\}\_\{t\}^\{\\pi\}\(s\)\.The objective is to learn a sequence of non\-stationary policiesπk\\pi^\{k\}fork∈\{1,…,𝒦\}k\\in\\\{1,\\dots,\\mathcal\{K\}\\\}while interacting with an unknown MDP in order to minimize the regret, which is measured asymptotically over𝒦\\mathcal\{K\}episodes of lengthℋ\\mathcal\{H\},Regret​\(𝒦\)=∑k=1𝒦\(𝒱1∗​\(s1k\)−𝒱1πk​\(s1k\)\)\\textsc\{Regret\}\(\\mathcal\{K\}\)=\\sum\_\{k=1\}^\{\\mathcal\{K\}\}\\left\(\\mathcal\{V\}\_\{1\}^\{\*\}\(s^\{k\}\_\{1\}\)\-\\mathcal\{V\}\_\{1\}^\{\\pi^\{k\}\}\(s^\{k\}\_\{1\}\)\\right\)wheres1k∈Ss^\{k\}\_\{1\}\\in Sis the starting state of thekk\-th episode\. Regret sums up the gap between the expected rewards obtained by the sequence of learned policiesπk\\pi^\{k\}and those obtained byπ∗\\pi^\{\*\}when learning for𝒦\\mathcal\{K\}episodes\. In the linear function approximation setting there is a feature mapϕt:S×A→ℝdt\\phi\_\{t\}:S\\times A\\to\\mathbb\{R\}^\{d\_\{t\}\}for eacht∈\[ℋ\]t\\in\[\\mathcal\{H\}\]that sends a state\-action pair\(s,a\)\(s,a\)to thedtd\_\{t\}\-dimensional vectorϕt​\(s,a\)\\phi\_\{t\}\(s,a\)\. Then, the state\-action value function𝒬t​\(s,a\)\\mathcal\{Q\}\_\{t\}\(s,a\)is parameterized by a vectorθt∈ℝdt\\theta\_\{t\}\\in\\mathbb\{R\}^\{d\_\{t\}\}so that𝒬t​\(θt\)​\(s,a\)=ϕt​\(s,a\)⊤​θt\\mathcal\{Q\}\_\{t\}\(\\theta\_\{t\}\)\(s,a\)=\\phi\_\{t\}\(s,a\)^\{\\top\}\\theta\_\{t\}\. Recent theoretical work in this setting gives an algorithm along with a lower bound that matches the regret achieved by the algorithm up to logarithmic factors\.

###### Theorem 3\.1\(\(Zanetteet al\.[2020](https://arxiv.org/html/2607.07769#bib.bib55)\)\)\.

Under appropriate normalization assumptions there is an algorithm that learns a sequence of policiesπk\\pi^\{k\}achieving regretRegret​\(𝒦\)=O~​\(∑t=1ℋdt​𝒦\+∑t=1ℋdt​ℐ​𝒦\)\\textsc\{Regret\}\(\\mathcal\{K\}\)=\\tilde\{O\}\\left\(\\sum\_\{t=1\}^\{\\mathcal\{H\}\}d\_\{t\}\\sqrt\{\\mathcal\{K\}\}\+\\sum\_\{t=1\}^\{\\mathcal\{H\}\}\\sqrt\{d\_\{t\}\}\\mathcal\{I\}\\mathcal\{K\}\\right\), whereℐ\\mathcal\{I\}is the inherent Bellman error\. Furthermore, this regret bound is optimal for this setting up to logarithmic factors indt,𝒦d\_\{t\},\\mathcal\{K\}andℋ\\mathcal\{H\}whenever𝒦=Ω​\(\(∑t=1ℋdt\)2\)\\mathcal\{K\}=\\Omega\(\(\\sum\_\{t=1\}^\{\\mathcal\{H\}\}d\_\{t\}\)^\{2\}\), in the sense that for any level of inherent Bellman errorℐ\\mathcal\{I\}and sequence of feature dimensions\{dt\}t=1H\\\{d\_\{t\}\\\}\_\{t=1\}^\{H\}, there exists a class of MDPs𝒞​\(ℐ,\{dt\}t=1H\)\\mathcal\{C\}\(\\mathcal\{I\},\\\{d\_\{t\}\\\}\_\{t=1\}^\{H\}\)where any algorithm achieves at least as much regret on at least one MDP in the class\.

The class of MDPs𝒞​\(ℐ,\{dt\}t=1H\)\\mathcal\{C\}\(\\mathcal\{I\},\\\{d\_\{t\}\\\}\_\{t=1\}^\{H\}\)constructed in Theorem[3\.1](https://arxiv.org/html/2607.07769#S3.Thmtheorem1)additionally satisfies the following properties\. First, every MDP in∪ℐ,\{dt\}t=1H𝒞​\(ℐ,\{dt\}t=1H\)\\cup\_\{\\mathcal\{I\},\\\{d\_\{t\}\\\}\_\{t=1\}^\{H\}\}\\mathcal\{C\}\(\\mathcal\{I\},\\\{d\_\{t\}\\\}\_\{t=1\}^\{H\}\)has the same transitions \(up to renaming of states and actions\)\. Second, for each fixed value of the inherent Bellman errorℐ\\mathcal\{I\}and the dimensions\{dt\}t=1H\\\{d\_\{t\}\\\}\_\{t=1\}^\{H\}, every MDP in𝒞​\(ℐ,\{dt\}t=1H\)\\mathcal\{C\}\(\\mathcal\{I\},\\\{d\_\{t\}\\\}\_\{t=1\}^\{H\}\)utilizes the same feature mapϕt​\(st,at\)\\phi\_\{t\}\(s\_\{t\},a\_\{t\}\)\. Thus one can view the class𝒞​\(ℐ,\{dt\}t=1H\)\\mathcal\{C\}\(\\mathcal\{I\},\\\{d\_\{t\}\\\}\_\{t=1\}^\{H\}\)as encoding one ”underlying” true environment defined by the transitions, with varying values ofℐ\\mathcal\{I\}and\{dt\}t=1H\\\{d\_\{t\}\\\}\_\{t=1\}^\{H\}corresponding to varying levels of function approximation accuracy, and model capacity for the underlying environment\. For simplicity of notation we will focus on the setting wheredt=dd\_\{t\}=dfor allt∈\{1,…​H\}t\\in\\\{1,\\dots H\\\}and write𝒞​\(ℐ,d\)\\mathcal\{C\}\(\\mathcal\{I\},d\)for the class of MDPs constructed in Theorem[3\.1](https://arxiv.org/html/2607.07769#S3.Thmtheorem1)for this setting\. Utilizing this point of view, we can then prove the following theorem on the relationship between the performance in the asymptotic and low\-data regimes\.

###### Theorem 3\.2\(*Non\-monotonicity Across Regimes*\)\.

For anyϵ\>0\\epsilon\>0, letdαd\_\{\\alpha\}be any feature dimension, and letdβ=dα1−ϵ/2d\_\{\\beta\}=d\_\{\\alpha\}^\{1\-\\epsilon/2\}\. Then there exist thresholds𝒦low<𝒦high\\mathcal\{K\}\_\{\\text\{low\}\}<\\mathcal\{K\}\_\{\\text\{high\}\}and inherent Bellman error levelsℐβ\>ℐα\\mathcal\{I\}\_\{\\beta\}\>\\mathcal\{I\}\_\{\\alpha\}such that

1. 1\.There is an algorithm achieving regretRegretlow​\(𝒦\)\\textsc\{Regret\}\_\{\\text\{low\}\}\(\\mathcal\{K\}\)when𝒦<𝒦low\\mathcal\{K\}<\\mathcal\{K\}\_\{\\text\{low\}\}for all MDPs in𝒞​\(ℐβ,dβ\)\\mathcal\{C\}\(\\mathcal\{I\}\_\{\\beta\},d\_\{\\beta\}\)\. However, every algorithm has regret at leastΩ~​\(dβϵ/2​Regretlow​\(𝒦\)\)\\widetilde\{\\Omega\}\\left\(d\_\{\\beta\}^\{\\epsilon/2\}\\textsc\{Regret\}\_\{\\text\{low\}\}\(\\mathcal\{K\}\)\\right\)when𝒦<𝒦low\\mathcal\{K\}<\\mathcal\{K\}\_\{\\text\{low\}\}on some MDPM∈𝒞​\(ℐα,dα\)M\\in\\mathcal\{C\}\(\\mathcal\{I\}\_\{\\alpha\},d\_\{\\alpha\}\)\.
2. 2\.There is an algorithm achieving regretRegrethigh​\(𝒦\)\\textsc\{Regret\}\_\{\\text\{high\}\}\(\\mathcal\{K\}\)when𝒦\>𝒦high\\mathcal\{K\}\>\\mathcal\{K\}\_\{\\text\{high\}\}for all MDPs in𝒞​\(ℐα,dα\)\\mathcal\{C\}\(\\mathcal\{I\}\_\{\\alpha\},d\_\{\\alpha\}\)\. However, every algorithm has regret at leastΩ~​\(dαϵ​Regrethigh​\(𝒦\)\)\\widetilde\{\\Omega\}\\left\(d\_\{\\alpha\}^\{\\epsilon\}\\textsc\{Regret\}\_\{\\text\{high\}\}\(\\mathcal\{K\}\)\\right\)on some MDPM∈𝒞​\(ℐβ,dβ\)M\\in\\mathcal\{C\}\(\\mathcal\{I\}\_\{\\beta\},d\_\{\\beta\}\)when𝒦\>𝒦high\\mathcal\{K\}\>\\mathcal\{K\}\_\{\\text\{high\}\}\.

###### Proof\.

Letϵ\>0\\epsilon\>0and considerdβ=dα1−ϵ2,ℐβ=1dαϵ​dβ,ℐα=1dα12\+2​ϵ,𝒦low=dα2\+ϵ,𝒦high=dα2\+4​ϵd\_\{\\beta\}=d\_\{\\alpha\}^\{1\-\\frac\{\\epsilon\}\{2\}\},\\mathcal\{I\}\_\{\\beta\}=\\frac\{1\}\{d\_\{\\alpha\}^\{\\epsilon\}\\sqrt\{d\_\{\\beta\}\}\},\\mathcal\{I\}\_\{\\alpha\}=\\frac\{1\}\{d\_\{\\alpha\}^\{\\frac\{1\}\{2\}\+2\\epsilon\}\},\\mathcal\{K\}\_\{\\text\{low\}\}=d\_\{\\alpha\}^\{2\+\\epsilon\},\\mathcal\{K\}\_\{\\text\{high\}\}=d\_\{\\alpha\}^\{2\+4\\epsilon\}We begin with the proof of part 1\. Therefore, for𝒦<𝒦low\\mathcal\{K\}<\\mathcal\{K\}\_\{\\text\{low\}\},dβ​ℐβ​𝒦=dα−ϵ​𝒦<dα1−ϵ2​𝒦=dβ​𝒦\\sqrt\{d\_\{\\beta\}\}\\mathcal\{I\}\_\{\\beta\}\\mathcal\{K\}=d\_\{\\alpha\}^\{\-\\epsilon\}\\mathcal\{K\}<d\_\{\\alpha\}^\{1\-\\frac\{\\epsilon\}\{2\}\}\\sqrt\{\\mathcal\{K\}\}=d\_\{\\beta\}\\sqrt\{\\mathcal\{K\}\}\. Therefore, by Theorem[3\.1](https://arxiv.org/html/2607.07769#S3.Thmtheorem1)there exists an algorithm achieving regret

Regretlow​\(𝒦\)\\displaystyle\\textsc\{Regret\}\_\{\\text\{low\}\}\(\\mathcal\{K\}\)=O~​\(ℋ​dβ​𝒦\+ℋ​dβ​ℐβ​𝒦\)\\displaystyle=\\tilde\{O\}\\left\(\\mathcal\{H\}d\_\{\\beta\}\\sqrt\{\\mathcal\{K\}\}\+\\mathcal\{H\}\\sqrt\{d\_\{\\beta\}\}\\mathcal\{I\}\_\{\\beta\}\\mathcal\{K\}\\right\)=O~​\(dβ​𝒦\)\\displaystyle=\\widetilde\{O\}\\left\(d\_\{\\beta\}\\sqrt\{\\mathcal\{K\}\}\\right\)in every MDPM∈𝒞​\(ℐβ,dβ\)M\\in\\mathcal\{C\}\(\\mathcal\{I\}\_\{\\beta\},d\_\{\\beta\}\)\. Further, since𝒦low=dα2\+ϵ\>Ω~​\(dα2\)\\mathcal\{K\}\_\{\\text\{low\}\}=d\_\{\\alpha\}^\{2\+\\epsilon\}\>\\widetilde\{\\Omega\}\\left\(d\_\{\\alpha\}^\{2\}\\right\), the lower bound from Theorem[3\.1](https://arxiv.org/html/2607.07769#S3.Thmtheorem1)applies to the class of MDPs𝒞​\(ℐα,dα\)\\mathcal\{C\}\(\\mathcal\{I\}\_\{\\alpha\},d\_\{\\alpha\}\)for all𝒦∈\[Ω~​\(dα2\),𝒦low\]\\mathcal\{K\}\\in\\left\[\\widetilde\{\\Omega\}\\left\(d\_\{\\alpha\}^\{2\}\\right\),\\mathcal\{K\}\_\{\\text\{low\}\}\\right\]\. In particular, every algorithm receives regret at least

Regret​\(𝒦\)\\displaystyle\\textsc\{Regret\}\(\\mathcal\{K\}\)=Ω~​\(ℋ​dα​𝒦\+ℋ​dα​ℐα​𝒦\)\\displaystyle=\\widetilde\{\\Omega\}\\left\(\\mathcal\{H\}d\_\{\\alpha\}\\sqrt\{\\mathcal\{K\}\}\+\\mathcal\{H\}\\sqrt\{d\_\{\\alpha\}\}\\mathcal\{I\}\_\{\\alpha\}\\mathcal\{K\}\\right\)\>Ω~​\(ℋ​dβ11−ϵ/2​𝒦\)\\displaystyle\>\\widetilde\{\\Omega\}\\left\(\\mathcal\{H\}d\_\{\\beta\}^\{\\frac\{1\}\{1\-\\epsilon/2\}\}\\sqrt\{\\mathcal\{K\}\}\\right\)\>Ω~​\(ℋ​dβϵ/21−ϵ/2​dβ​𝒦\)\\displaystyle\>\\widetilde\{\\Omega\}\\left\(\\mathcal\{H\}d\_\{\\beta\}^\{\\frac\{\\epsilon/2\}\{1\-\\epsilon/2\}\}d\_\{\\beta\}\\sqrt\{\\mathcal\{K\}\}\\right\)Thus,Regret​\(𝒦\)\>Ω~​\(dβϵ/2​Regretlow​\(𝒦\)\)\\textsc\{Regret\}\(\\mathcal\{K\}\)\>\\widetilde\{\\Omega\}\\left\(d\_\{\\beta\}^\{\\epsilon/2\}\\textsc\{Regret\}\_\{\\text\{low\}\}\(\\mathcal\{K\}\)\\right\)\. For part 2, note that for𝒦\>𝒦high\\mathcal\{K\}\>\\mathcal\{K\}\_\{\\text\{high\}\}we have bothdα​ℐα​𝒦=dα−2​ϵ​𝒦\>dα−2​ϵ​𝒦⋅𝒦high\>dα​𝒦\\sqrt\{d\_\{\\alpha\}\}\\mathcal\{I\}\_\{\\alpha\}\\mathcal\{K\}=d\_\{\\alpha\}^\{\-2\\epsilon\}\\mathcal\{K\}\>d\_\{\\alpha\}^\{\-2\\epsilon\}\\sqrt\{\\mathcal\{K\}\\cdot\\mathcal\{K\}\_\{\\text\{high\}\}\}\>d\_\{\\alpha\}\\sqrt\{\\mathcal\{K\}\}anddβ​ℐβ​𝒦\>dα−ϵ​𝒦⋅𝒦low=dα1\+ϵ​𝒦\>dβ​𝒦\\sqrt\{d\_\{\\beta\}\}\\mathcal\{I\}\_\{\\beta\}\\mathcal\{K\}\>d\_\{\\alpha\}^\{\-\\epsilon\}\\sqrt\{\\mathcal\{K\}\\cdot\\mathcal\{K\}\_\{\\text\{low\}\}\}=d\_\{\\alpha\}^\{1\+\\epsilon\}\\sqrt\{\\mathcal\{K\}\}\>d\_\{\\beta\}\\sqrt\{\\mathcal\{K\}\}\.

![Refer to caption](https://arxiv.org/html/2607.07769v1/lowandhigh.png)Asymptotic vs Low\-Data

![Refer to caption](https://arxiv.org/html/2607.07769v1/lowdata2.png)Regret in the Low\-data Regime

![Refer to caption](https://arxiv.org/html/2607.07769v1/highdata2.png)Regret in the High\-data Regime

Figure 1:Left: Scaling laws of reinforcement learning: Baseline comparison of algorithms that were proposed and developed in the high\-data regime in the Arcade Learning Environment in both high\-data regime and low\-data regime\. Middle: Regret in the low\-data regime\. Right: Regret in the high\-data regime\.Therefore by Theorem[3\.1](https://arxiv.org/html/2607.07769#S3.Thmtheorem1)that for𝒦\>𝒦high\\mathcal\{K\}\>\\mathcal\{K\}\_\{\\text\{high\}\}there exists an algorithm achieving regret

Regrethigh​\(𝒦\)\\displaystyle\\textsc\{Regret\}\_\{\\text\{high\}\}\(\\mathcal\{K\}\)=O~​\(ℋ​dα​𝒦\+ℋ​dα​ℐα​𝒦\)\\displaystyle=\\tilde\{O\}\\left\(\\mathcal\{H\}d\_\{\\alpha\}\\sqrt\{\\mathcal\{K\}\}\+\\mathcal\{H\}\\sqrt\{d\_\{\\alpha\}\}\\mathcal\{I\}\_\{\\alpha\}\\mathcal\{K\}\\right\)=O~​\(ℋ​dα​ℐα​𝒦\)\.\\displaystyle=\\tilde\{O\}\\left\(\\mathcal\{H\}\\sqrt\{d\_\{\\alpha\}\}\\mathcal\{I\}\_\{\\alpha\}\\mathcal\{K\}\\right\)\.for every MDPM∈𝒞​\(ℐα,dα\)M\\in\\mathcal\{C\}\(\\mathcal\{I\}\_\{\\alpha\},d\_\{\\alpha\}\)\. However, by the lower bound in Theorem[3\.1](https://arxiv.org/html/2607.07769#S3.Thmtheorem1), for𝒦\>𝒦high\\mathcal\{K\}\>\\mathcal\{K\}\_\{\\text\{high\}\}every algorithm receives regret at least

Regret​\(𝒦\)\\displaystyle\\textsc\{Regret\}\(\\mathcal\{K\}\)=Ω~​\(ℋ​dβ​𝒦\+ℋ​dβ​ℐβ​𝒦\)\\displaystyle=\\widetilde\{\\Omega\}\\left\(\\mathcal\{H\}d\_\{\\beta\}\\sqrt\{\\mathcal\{K\}\}\+\\mathcal\{H\}\\sqrt\{d\_\{\\beta\}\}\\mathcal\{I\}\_\{\\beta\}\\mathcal\{K\}\\right\)\>Ω~​\(ℋ​dβ​ℐβ​𝒦\)=Ω~​\(ℋ​dα−ϵ​𝒦\)\\displaystyle\>\\widetilde\{\\Omega\}\\left\(\\mathcal\{H\}\\sqrt\{d\_\{\\beta\}\}\\mathcal\{I\}\_\{\\beta\}\\mathcal\{K\}\\right\)=\\widetilde\{\\Omega\}\\left\(\\mathcal\{H\}d\_\{\\alpha\}^\{\-\\epsilon\}\\mathcal\{K\}\\right\)=Ω~​\(dαϵ​ℋ​dα−2​ϵ​𝒦\)=Ω~​\(dαϵ​ℋ​dα​ℐα​𝒦\)\\displaystyle=\\widetilde\{\\Omega\}\\left\(d\_\{\\alpha\}^\{\\epsilon\}\\mathcal\{H\}d\_\{\\alpha\}^\{\-2\\epsilon\}\\mathcal\{K\}\\right\)=\\widetilde\{\\Omega\}\\left\(d\_\{\\alpha\}^\{\\epsilon\}\\mathcal\{H\}\\sqrt\{d\_\{\\alpha\}\}\\mathcal\{I\}\_\{\\alpha\}\\mathcal\{K\}\\right\)\>Ω~​\(dαϵ​Regrethigh​\(𝒦\)\)\\displaystyle\>\\widetilde\{\\Omega\}\\left\(d\_\{\\alpha\}^\{\\epsilon\}\\textsc\{Regret\}\_\{\\text\{high\}\}\(\\mathcal\{K\}\)\\right\)∎

Theorem[3\.2](https://arxiv.org/html/2607.07769#S3.Thmtheorem2)introduces the provable trade\-off between performance in the low\-data regime, i\.e\.𝒦<𝒦low\\mathcal\{K\}<\\mathcal\{K\}\_\{\\text\{low\}\}, and the high\-data regime, i\.e\.𝒦\>𝒦high\\mathcal\{K\}\>\\mathcal\{K\}\_\{\\text\{high\}\}\. In particular, in the low\-data regime lower capacity function approximation, i\.e\. lower feature dimensiondβd\_\{\\beta\}, with larger approximation error, i\.e\. larger inherent Bellman errorℐβ\\mathcal\{I\}\_\{\\beta\}, can provably outperform larger capacity models, i\.e\. feature dimensiondαd\_\{\\alpha\}, with smaller approximation error, i\.e\. inherent Bellman errorℐα\\mathcal\{I\}\_\{\\alpha\}\. Furthermore, the relative performance is reversed in the high\-data regime𝒦\>𝒦high\\mathcal\{K\}\>\\mathcal\{K\}\_\{\\text\{high\}\}\. Thus, asymptotic performance of an algorithm is neither indicative nor carries any relevant information on the expected performance of the algorithm when training data is scarce \(i\.e\. limited\)\.

## 4The Assumption of Monotonicity and Performance Rankings

The instances of the implicit assumption that the performance profile of an algorithm in the high\-data regime will translate to the low\-data regime monotonically appear in almost all of the studies conducted in the low\-data regime\. In particular, we see that when this line of work was being conducted the best performing algorithm in the high\-data regime was an inherently high capacity model, i\.e\. based on learning the state action value distribution\. Hence, there are many cases in the literature \(e\.g\. DRQ, OTR, DER, CURL, SimPLE, Efficient\-Zero\) where all the newly proposed algorithms in the low\-data regime are being compared to an algorithm that inherently produces a higher capacity modelunder the implicit assumptionthat an algorithm that is state\-of\-the\-art in the high\-data regime must be the state\-of\-the\-art in the low\-data regime\. The large scale experiments provided in Section[6](https://arxiv.org/html/2607.07769#S6)demonstrate the impact of this implicit assumption and provide a guideline for a principled analysis and evaluation\. In particular, the results reported in Section[6](https://arxiv.org/html/2607.07769#S6)prove that the performance profile of an algorithm in the high\-data regime does not monotonically transfer to the low\-data regime\. Due to this extensive focus throughout the literature on low\-data regime comparisons to algorithms that inherently learn higher capacity models, we provide additional theoretical analysis for the empirically observed sample complexity results in the low to high\-data regime in deep reinforcement learning\. The following proposition demonstrates a precise justification of these issues: whenever there are two different actions where the true mean state\-action values are withinϵ\\epsilon, an approximation error ofϵ\\epsilonin total variation distancedT​Vd\_\{TV\}for𝒟​\(s,a\)\\mathcal\{D\}\(s,a\)of one of the actions can be sufficient to reverse the order of the means\.

###### Proposition 4\.1\(*Sufficiency of error ofϵ\\epsilonin total variation distance*\)\.

Fix a statessand consider two actionsa,a^a,\\hat\{a\}\. Let𝒟​\(s,a\)\\mathcal\{D\}\(s,a\)be the true state\-action value distribution of\(s,a\)\(s,a\), and let𝒵​\(s,a\)∼𝒟​\(s,a\)\\mathcal\{Z\}\(s,a\)\\sim\\mathcal\{D\}\(s,a\)\. Suppose that𝔼​\[𝒵​\(s,a\)\]=𝔼​\[𝒵​\(s,a^\)\]\+ϵ\\mathbb\{E\}\[\\mathcal\{Z\}\(s,a\)\]=\\mathbb\{E\}\[\\mathcal\{Z\}\(s,\\hat\{a\}\)\]\+\\epsilon\. Then there is a random variable𝒴\\mathcal\{Y\}such thatdT​V​\(𝒴,𝒵​\(s,a\)\)≤ϵ​and​𝔼​\[𝒵​\(s,a^\)\]≥𝔼​\[𝒴\]\.d\_\{TV\}\(\\mathcal\{Y\},\\mathcal\{Z\}\(s,a\)\)\\leq\\epsilon\\\>\\textrm\{ and \}\\\>\\mathbb\{E\}\[\\mathcal\{Z\}\(s,\\hat\{a\}\)\]\\geq\\mathbb\{E\}\[\\mathcal\{Y\}\]\.

The proof is provided in the supplementary material\. Proposition[4\.1](https://arxiv.org/html/2607.07769#S4.Thmtheorem1)shows that to have the correct ranking of the actions the state\-action value distribution must be learnt with error at mostϵ\\epsilon\. Standard results on sample complexity for discrete distributions then imply that algorithms that learn the state\-action value distribution with fixed support sizekk, i\.e C51, requirek/ϵ2k/\\epsilon^\{2\}samples to achieve total variation distance at mostϵ\\epsilon\. More advanced algorithms such as𝒬\\mathcal\{Q\}RD𝒬\\mathcal\{Q\}N and I𝒬\\mathcal\{Q\}N do away with the assumption that the support is known\. This allows a more flexible representation in order to more accurately represent state\-action values, but, as we will show, leads to a further increase in the sample complexity\. The𝒬\\mathcal\{Q\}RD𝒬\\mathcal\{Q\}N algorithm models it as a uniform mixture of𝒩\\mathcal\{N\}Dirac deltas on the reals i\.e\.𝒵​\(s,a\)=1𝒩​∑i=1𝒩δθi​\(s,a\)\\mathcal\{Z\}\(s,a\)=\\frac\{1\}\{\\mathcal\{N\}\}\\sum\_\{i=1\}^\{\\mathcal\{N\}\}\\delta\_\{\\theta\_\{i\}\(s,a\)\}, whereθi​\(s,a\)∈ℝ\\theta\_\{i\}\(s,a\)\\in\\mathbb\{R\}is a parametric model\.

###### Proposition 4\.2\(*Sample Complexity with Unknown Support*\)\.

Let𝒩\>ℳ≥2\\mathcal\{N\}\>\\mathcal\{M\}\\geq 2,ϵ\>ℳ4​𝒩\\epsilon\>\\frac\{\\mathcal\{M\}\}\{4\\mathcal\{N\}\}, andθi∈ℝ\\theta\_\{i\}\\in\\mathbb\{R\}fori∈\[𝒩\]i\\in\[\\mathcal\{N\}\]\. The number of samples required to learn a model of the form𝒵=1𝒩​∑i=1𝒩δθi\\mathcal\{Z\}=\\frac\{1\}\{\\mathcal\{N\}\}\\sum\_\{i=1\}^\{\\mathcal\{N\}\}\\delta\_\{\\theta\_\{i\}\}to within total variation distanceϵ\\epsilonisΩ​\(ℳϵ2\)\\Omega\\left\(\\frac\{\\mathcal\{M\}\}\{\\epsilon^\{2\}\}\\right\)\.

The proof is provided in the supplementary material\. Note that the lower bound in Proposition[4\.2](https://arxiv.org/html/2607.07769#S4.Thmtheorem2)can be significantly larger thank/ϵ2k/\\epsilon^\{2\}samples\.

![Refer to caption](https://arxiv.org/html/2607.07769v1/x1.png)Median

![Refer to caption](https://arxiv.org/html/2607.07769v1/x2.png)Mean

![Refer to caption](https://arxiv.org/html/2607.07769v1/x3.png)20th\{\}^\{\\textrm\{th\}\}Percentile

![Refer to caption](https://arxiv.org/html/2607.07769v1/x4.png)Median

![Refer to caption](https://arxiv.org/html/2607.07769v1/x5.png)Mean

![Refer to caption](https://arxiv.org/html/2607.07769v1/x6.png)20th\{\}^\{\\textrm\{th\}\}Percentile

Figure 2:Up: Human normalized median, mean and 20th\{\}^\{\\textrm\{th\}\}percentile results for the dueling algorithm, C51, I𝒬\\mathcal\{Q\}N and𝒬\\mathcal\{Q\}RD𝒬\\mathcal\{Q\}N in the Arcade Learning Environment 100K benchmark\. Down: Human normalized median, mean, and 20th\{\}^\{\\textrm\{th\}\}percentile results for the dueling algorithm, C51, I𝒬\\mathcal\{Q\}N and𝒬\\mathcal\{Q\}RD𝒬\\mathcal\{Q\}N in the high\-data regime towards 200 million frame\.Table 1:Large scale comparison of𝒬\\mathcal\{Q\}\-based deep reinforcement learning algorithms with human normalized mean, median and 20th\{\}^\{\\textrm\{th\}\}percentile results in the Arcade Learning Environment 100K benchmark for D𝒬\\mathcal\{Q\}N\(Mnihet al\.[2015](https://arxiv.org/html/2607.07769#bib.bib126)\), deep Double\-𝒬\\mathcal\{Q\}\(Hasseltet al\.[2016](https://arxiv.org/html/2607.07769#bib.bib72)\), dueling\(Wanget al\.[2016](https://arxiv.org/html/2607.07769#bib.bib163)\), Prior\(Schaulet al\.[2016](https://arxiv.org/html/2607.07769#bib.bib64)\), C51,𝒬\\mathcal\{Q\}RD𝒬\\mathcal\{Q\}N and I𝒬\\mathcal\{Q\}N\(Dabneyet al\.[2018a](https://arxiv.org/html/2607.07769#bib.bib47)\)\.
## 5Principled Evaluation Framework

In Section[6](https://arxiv.org/html/2607.07769#S6), we systematically explain and discuss the underlying design paradigms, the implicit assumptions and the methodological choices made in deep reinforcement learning research that led to incorrect conclusions\. In this section we introduce the principled evaluation framework to ensure the research progress we obtain in deep reinforcement learning is reliable and scientifically robust\.

I\. Assumptions matter:Performance rankings across regimes are non\-monotone\. II\. Biases in Evaluation:Including algorithms in the comparison benchmark based on the monotonicity assumption will create biased evaluation\. III\. Core Algorithms:Core algorithms must be included in the comparison benchmarks\. IV\. Inherent Capacity:Inherent capacity and dimensionality will provide insights on performance rankings across regimes\. V\. Biases in Datasets:Creating datasets based on the monotonicity assumption will create biased benchmarks\.

## 6Large Scale Empirical Analysis

The empirical analysis is conducted in the Arcade Learning Environment \(ALE\)\(Mnihet al\.[2015](https://arxiv.org/html/2607.07769#bib.bib126)\)\. The Double𝒬\\mathcal\{Q\}\-learning algorithm is trained viaHasseltet al\.\([2016](https://arxiv.org/html/2607.07769#bib.bib72)\)initially proposed byvan Hasselt \([2010](https://arxiv.org/html/2607.07769#bib.bib86)\)\. The dueling algorithm is trained viaWanget al\.\([2016](https://arxiv.org/html/2607.07769#bib.bib163)\)\. The prior algorithm refers to the prioritized experience replay algorithm proposed bySchaulet al\.\([2016](https://arxiv.org/html/2607.07769#bib.bib64)\)\. The experiments are run with Haiku as the neural network library, Optax\(Hesselet al\.[2020](https://arxiv.org/html/2607.07769#bib.bib83)\)as the optimization library, and RLax for the reinforcement learning library\(Babuschkinet al\.[2020](https://arxiv.org/html/2607.07769#bib.bib82)\)\. All of the results are reported with the standard error of the mean\. For the full list of algorithms, details on the hyperparameters, direct references and the detailed explanations of the baselines please see the supplementary material\. To provide a complete picture of the sample complexity we conducted our experiments in both low\-data, i\.e\. the Arcade Learning Environment 100K benchmark, and high data regime, i\.e\. baseline 200 million frame training\. Note that human normalized score is computed as follows:ScoreHN=\(Scoreagent−Scorerandom\)/\(Scorehuman−Scorerandom\)\\textrm\{Score\}\_\{\\textrm\{HN\}\}=\(\\textrm\{Score\}\_\{\\textit\{agent\}\}\-\\textrm\{Score\}\_\{\\textit\{random\}\}\)/\(\\textrm\{Score\}\_\{\\textit\{human\}\}\-\\textrm\{Score\}\_\{\\textit\{random\}\}\)\.

![Refer to caption](https://arxiv.org/html/2607.07769v1/alien2.png)Alien

![Refer to caption](https://arxiv.org/html/2607.07769v1/amidar2.png)Amidar

![Refer to caption](https://arxiv.org/html/2607.07769v1/assault2.png)Assault

![Refer to caption](https://arxiv.org/html/2607.07769v1/asterix2.png)Asterix

![Refer to caption](https://arxiv.org/html/2607.07769v1/bank2.png)BankHeist

![Refer to caption](https://arxiv.org/html/2607.07769v1/chopper2.png)ChopperCommand

![Refer to caption](https://arxiv.org/html/2607.07769v1/hero2.png)Hero

![Refer to caption](https://arxiv.org/html/2607.07769v1/james2.png)JamesBond

![Refer to caption](https://arxiv.org/html/2607.07769v1/kangaroo2.png)Kangaroo

![Refer to caption](https://arxiv.org/html/2607.07769v1/crazy2.png)CrazyClimber

![Refer to caption](https://arxiv.org/html/2607.07769v1/pacman2.png)MsPacman

![Refer to caption](https://arxiv.org/html/2607.07769v1/frost2.png)FrostBite

![Refer to caption](https://arxiv.org/html/2607.07769v1/road2.png)RoadRunner

![Refer to caption](https://arxiv.org/html/2607.07769v1/sea2.png)Seaquest

![Refer to caption](https://arxiv.org/html/2607.07769v1/up2.png)UpNDown

![Refer to caption](https://arxiv.org/html/2607.07769v1/qbert2.png)Qbert

Figure 3:The learning curves of Alien, Amidar, Asterix, BankHeist, ChopperCommand, Hero, CrazyClimber, JamesBond, Kangaroo, MsPacman, FrostBite, Qbert, RoadRunner, Seaquest and UpNDown with dueling architecture, C51, I𝒬\\mathcal\{Q\}N and𝒬\\mathcal\{Q\}RD𝒬\\mathcal\{Q\}N algorithms in the Arcade Learning Environment with 100K environment interaction training\.Implicit Assumptions on Monotonicity Cause Suboptimal and Incorrect Conclusions\.Our extensive large\-scale empirical analysis demonstrates that a major line of research conducted in the past five years resulted in incorrect conclusions\. We show that a simple baseline algorithmfrom 2016\(Wanget al\.[2016](https://arxiv.org/html/2607.07769#bib.bib163)\), by a systematic methodological choice was never included in the comparison benchmark, following the implicit assumption that appears in all of the recent line of research that we have discussed in detail in Section[4](https://arxiv.org/html/2607.07769#S4)\. We demonstrate that this baseline algorithm in fact performs much better than many recent algorithms that claimed to be better than the baselines, even including algorithms that are specifically built on top of the baseline algorithm\. Figure[3](https://arxiv.org/html/2607.07769#S6.F3)reports learning curves for the I𝒬\\mathcal\{Q\}N,𝒬\\mathcal\{Q\}RD𝒬\\mathcal\{Q\}N, dueling and C51 in the Arcade Learning Environment low\-data regime benchmark\. These results demonstrate that the simple base algorithm dueling performs significantly better than a series of algorithms that were included in the comparison benchmark which inherently produced higher capacity models when the training samples are limited\. Note that DR𝒬\\mathcal\{Q\}uses the dueling architecture without any high capacity inducing components\. One intriguing takeaway from the results provided in Table[1](https://arxiv.org/html/2607.07769#S4.T1)and Figure[4](https://arxiv.org/html/2607.07769#S6.F4)111DER2021\{\}^\{\\textrm\{2021\}\}refers to the re\-implementation with random seed variations of the original paper data\-efficient Rainbow \(i\.e\. DER2019\{\}^\{\\textrm\{2019\}\}\) byvan Hasseltet al\.\([2019](https://arxiv.org/html/2607.07769#bib.bib60)\)\. OTR refers to further implementation of the Rainbow algorithm byKielak \([2019](https://arxiv.org/html/2607.07769#bib.bib49)\)\. DR𝒬NeurIPS\\mathcal\{Q\}^\{\\textrm\{NeurIPS\}\}refers to the re\-implementation of the original DR𝒬\\mathcal\{Q\}algorithm with the goal of achieving reproducibility with variation on the number of random seeds\(Agarwalet al\.[2021](https://arxiv.org/html/2607.07769#bib.bib52)\)\.is the fact that the simple baseline dueling algorithm performs 15% better than the DR𝒬NeurIPS\\mathcal\{Q\}^\{\\textrm\{NeurIPS\}\}implementation, and 11% less than the DR𝒬ICLR\\mathcal\{Q\}^\{\\textrm\{ICLR\}\}implementation instead of 82% gain reported in the original paper\.

Providing Direct Comparison to Core Algorithms\.Algorithms that are built on top of a core reinforcement learning algorithm must provide a direct comparison to the algorithm they are built on top of\. The case of DR𝒬\\mathcal\{Q\}demonstrates the significance of the direct comparison to the core algorithm\. As our paper discovers and describes extensively, the monotonicity assumption on the performance ranking across regimes led a line of work to benchmark against certain algorithms in the low\-data regime, assuming that if an algorithm has the highest performance in the high\-data regime it must have the top\-ranked performance in the low\-data regime\. However, as we pointed out in our theoretical analysis this is a dangerous and incorrect assumption\. The results reported in Figure[1](https://arxiv.org/html/2607.07769#S3.F1)and Figure[4](https://arxiv.org/html/2607.07769#S6.F4)demonstrate that these implicit assumptions in fact lead to incorrect and suboptimal conclusions\.

Non\-Monotonicity of Performance Ranking Across Regimes\.Table[1](https://arxiv.org/html/2607.07769#S4.T1)reports the human normalized median, mean and 20th\{\}^\{\\textrm\{th\}\}percentile results over all of the MDPs from the 100K ALE benchmark for D𝒬\\mathcal\{Q\}N, Double\-𝒬\\mathcal\{Q\}, dueling, C51,𝒬\\mathcal\{Q\}RD𝒬\\mathcal\{Q\}N, I𝒬\\mathcal\{Q\}N and Prior\. One important takeaway from the results reported in the Table[1](https://arxiv.org/html/2607.07769#S4.T1)is the fact that one particular algorithm performance profile in 200 million frame training will not directly transfer to the low\-data region as predicted by our theoretical analysis in Section[3](https://arxiv.org/html/2607.07769#S3)\. Figure[2](https://arxiv.org/html/2607.07769#S4.F2)reports the learning curves of human normalized median, mean and 20th\{\}^\{\\textrm\{th\}\}percentile for the dueling algorithm, C51,𝒬\\mathcal\{Q\}RD𝒬\\mathcal\{Q\}N, and I𝒬\\mathcal\{Q\}N\. These results once more demonstrate that the performance profile of the simple base algorithm dueling is significantly better than any core algorithm which inherently produced higher capacity models that was included in the comparison benchmark of the extensive low\-data regime literature when the number of environment interactions are limited\.

![Refer to caption](https://arxiv.org/html/2607.07769v1/samples.png)Sample Complexity of C51

![Refer to caption](https://arxiv.org/html/2607.07769v1/iqnsamples.png)Sample Complexity of I𝒬\\mathcal\{Q\}N

![Refer to caption](https://arxiv.org/html/2607.07769v1/comparison3.png)Overall Comparison

Figure 4:Left: Number of samples, i\.e\. environment interactions, required by the base algorithms that inherently produce higher capacity models to achieve the performance level achieved by the dueling algorithm\. Center: Number of samples required by I𝒬\\mathcal\{Q\}N to achieve the performance level achieved by dueling\. Right: Overall comparison of algorithms recently developed in the low\-data regime ALE 100K benchmark to the dueling algorithm that was designed in the high\-data region\.Biases in the Evaluation Criteria\.The original paper of the DR𝒬ICLR\\mathcal\{Q\}^\{\\textrm\{ICLR\}\}algorithm\(Yaratset al\.[2021](https://arxiv.org/html/2607.07769#bib.bib56)\)benchmarks against data\-efficient Rainbow \(DER\)\(van Hasseltet al\.[2019](https://arxiv.org/html/2607.07769#bib.bib60)\)which inherently learns a higher capacity model\. Our results show that the fact that the original paper that proposed data augmentation for reinforcement learning, i\.e\. DR𝒬ICLR\\mathcal\{Q\}^\{\\textrm\{ICLR\}\}, on top of the dueling algorithm did not provide comparisons against the core algorithm that they are built on, i\.e\. dueling\(Wanget al\.[2016](https://arxiv.org/html/2607.07769#bib.bib163)\), resulted in inflated performance profiles for the DR𝒬ICLR\\mathcal\{Q\}^\{\\textrm\{ICLR\}\}algorithm\. For a fair, direct and transparent comparison we kept the hyperparameters for the baseline algorithms in the low\-data regime exactly the same with the DR𝒬ICLR\\mathcal\{Q\}^\{\\textrm\{ICLR\}\}paper \(see supplementary material for the full list and high\-data regime hyperparameter settings\)\. More intriguingly, the comparisons provided in the DR𝒬ICLR\\mathcal\{Q\}^\{\\textrm\{ICLR\}\}paper to the DER and OTR algorithms report the performance gained by DR𝒬ICLR\\mathcal\{Q\}^\{\\textrm\{ICLR\}\}over DER is 82% and over OTR is 35%\. However, if a direct comparison is made to the simple dueling algorithm as Table[1](https://arxiv.org/html/2607.07769#S4.T1)demonstrates the performance gain is utterly restricted to11%\. Moreover, when it is compared to the reproduced results of DR𝒬NeurIPS\\mathcal\{Q\}^\{\\textrm\{NeurIPS\}\}our results reveal that in fact there is a performance decrease due to utilizing DR𝒬\\mathcal\{Q\}over dueling\. Thus, while our paper introduces the foundations on the non\-monotonicity of the performance profiles from large\-data regime to low\-data regime, it further provides the basis on how we can compare algorithms with a principled approach and scientific rigor allowing more concrete and accurate evaluation across data\-regimes\.

Theoretical Analysis and the Inherent Bellman Error vs Dimensionality\.The right and center plots of Figure[1](https://arxiv.org/html/2607.07769#S3.F1)report regret curves corresponding to the theoretical analysis in Theorem[3\.2](https://arxiv.org/html/2607.07769#S3.Thmtheorem2)for various choices of the feature dimensionalityddand the inherent Bellman errorℐ\\mathcal\{I\}\. In particular, the center plot shows the low\-data regime where the number of episodes𝒦<1000\\mathcal\{K\}<1000, while the right plot shows the high\-data regime where𝒦\\mathcal\{K\}is as large as500000500000\. Notably, the relative ordering of the regret across the different choices ofddandℐ\\mathcal\{I\}is completely reversed in the high\-data regime when compared to the low\-data regime\. Recall from Theorem[3\.1](https://arxiv.org/html/2607.07769#S3.Thmtheorem1)that the inherent Bellman error is a measure of the accuracy of function approximation under the Bellman operator corresponding to an MDP\. Thus, the varying values ofℐ\\mathcal\{I\}andddin Figure[1](https://arxiv.org/html/2607.07769#S3.F1)correspond to a natural setting where increasing the number of model parameters \(i\.e\. increasingdd\) corresponds to an increase in the accuracy of function approximation \(i\.e\. a decrease inℐ\\mathcal\{I\}\)\. Thus the results reported in Figure[1](https://arxiv.org/html/2607.07769#S3.F1)demonstrate that, even in the natural setting where increased model capacity leads to increased accuracy, there can be a complete reversal in the ordering of algorithm performance between the low and high\-data regimes\. Figure[4](https://arxiv.org/html/2607.07769#S6.F4)reports results on the number of samples required for training with the baseline algorithm that inherently produces higher capacity models to reach the same performance levels achieved by the dueling algorithm for every MDP from ALE low\-data regime benchmark\. These results once more demonstrate that to reach the same performance levels with the dueling algorithm, baseline algorithms that inherently learn higher capacity models require orders of magnitude more samples to train on\. As discussed in Section[4](https://arxiv.org/html/2607.07769#S4), more complex representations for broader classes of distributions come at the cost of a higher sample complexity required for learning\. One intriguing fact is that the original SimPLE paper in the low\-data regime benchmarked against the Rainbow algorithm which is essentially a higher capacity model designed in the high\-data regime by having the implicit assumption that the state\-of\-the art performance profile must transfer monotonically to the low\-data regime\. These instances of implicit assumptions also occur in DR𝒬ICLR\\mathcal\{Q\}^\{\\textrm\{ICLR\}\}, CURL, SPR and Efficient\-Zero\(Yeet al\.[2021](https://arxiv.org/html/2607.07769#bib.bib110)\)even when comparisons are made for more advanced algorithms such as MuZero\.

Datasets are Created and Founded on Implicit Assumptions\.Thus far we have discussed the pivotal role of implicit assumptions on the algorithmic comparisons and developing baselines in deep reinforcement learning\. However, this issue further extends back to even how the entire low\-data regime benchmark was established, i\.e\. ALE 100K\. The ALE 100K was initially created to allow researchers to work on a subset of games instead of full set of games used in the high\-data regime\(Kaiseret al\.[2020](https://arxiv.org/html/2607.07769#bib.bib43)\), and this benchmark is currently used by any algorithm developed for the low data regime\. However, the entire ALE 100K benchmark was in fact built on the selection bias of choosing games that performed better either with the proposed algorithm of the paper that proposed the entire benchmark\(Kaiseret al\.[2020](https://arxiv.org/html/2607.07769#bib.bib43)\), or with Rainbow, which we extensively demonstrated throughout the paper is an algorithm that is subjected to the implicit assumption bias on monotonicity across regimes\. Thus the issues we explicitly discover and analyze in our paper are not limited to baselines but further extend to canonical benchmarks that we evaluate reinforcement learning algorithms on\. Our paper discovers that the canonical methodological choices made in a major line of deep reinforcement learning research that is based on these implicit assumptions, give incorrect signals on why and what makes these algorithms work, and hence affect future research directions while misdirecting the possible current research efforts from ideas that could have worked during the algorithm design process\.

## 7Conclusion

In this paper we aimed to answer the following questions:*\(i\) How are the scaling laws of reinforcement learning formally characterized with respect to capacity and complexity?**\(ii\) What are the canonical methodological choices that fundamentally affect the progress in deep reinforcement learning research?*and*\(iii\) What is the underlying theoretical relationship between monotonicity, the performance profiles and sample complexity regimes?*To be able to answer these questions we provide theoretical analysis on the sample complexity of the baseline deep reinforcement learning algorithms\. We conduct extensive experiments both in the low\-data regime 100K Arcade Learning Environment and high\-data regime baseline 200 million frame training\. Both theoretical and empirical analysis provided in our paper demonstrate that the performance profiles of deep reinforcement learning algorithms do not have a monotonic relationship across sample complexity regimes\. Our analysis reveals that the underlying assumption of the monotonic relationship of the performance characteristics and the sample complexity regimes is currently present in a major line of research including many recent state\-of\-the\-art studies and this implicit assumption led these studies to result in incorrect conclusions\. Our paper demonstrates that several baseline𝒬\\mathcal\{Q\}algorithms perform better than a line of recent algorithms claimed to be the state\-of\-the\-art\. Our paper establishes a principled analysis of deep reinforcement learning that characterizes the fundamental relationship between scaling, capacity and complexity\.

## References

- R\. Agarwal, M\. Schwarzer, P\. S\. Castro, A\. C\. Courville, and M\. G\. Bellemare \(2021\)Deep reinforcement learning at the edge of the statistical precipice\.InAdvances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021,M\. Ranzato, A\. Beygelzimer, Y\. N\. Dauphin, P\. Liang, and J\. W\. Vaughan \(Eds\.\),pp\. 29304–29320\.Cited by:[footnote 1](https://arxiv.org/html/2607.07769#footnote1)\.
- I\. Babuschkin, K\. Baumli, A\. Bell, S\. Bhupatiraju, J\. Bruce, P\. Buchlovsky, D\. Budden, T\. Cai, A\. Clark, I\. Danihelka, C\. Fantacci, J\. Godwin, C\. Jones, T\. Hennigan, M\. Hessel, S\. Kapturowski, T\. Keck, I\. Kemaev, M\. King, L\. Martens, H\. Merzic, V\. Mikulik, T\. Norman, J\. Quan, G\. Papamakarios, R\. Ring, F\. Ruiz, A\. Sanchez, R\. Schneider, E\. Sezener, S\. Spencer, S\. Srinivasan, W\. Stokowiec, and F\. Viola \(2020\)The DeepMind EcosystemExternal Links:[Link](http://github.com/deepmind)Cited by:[§6](https://arxiv.org/html/2607.07769#S6.p1.2)\.
- M\. G\. Bellemare, W\. Dabney, and R\. Munos \(2017\)A distributional perspective on reinforcement learning\.InProceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6\-11 August 2017,Proceedings of Machine Learning Research, Vol\.70,pp\. 449–458\.Cited by:[§2](https://arxiv.org/html/2607.07769#S2.p2.2)\.
- W\. Dabney, G\. Ostrovski, D\. Silver, and R\. Munos \(2018a\)Implicit quantile networks for distributional reinforcement learning\.InProceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10\-15, 2018,J\. G\. Dy and A\. Krause \(Eds\.\),Proceedings of Machine Learning Research, Vol\.80,pp\. 1104–1113\.Cited by:[§2](https://arxiv.org/html/2607.07769#S2.p2.10),[Table 1](https://arxiv.org/html/2607.07769#S4.T1)\.
- W\. Dabney, M\. Rowland, M\. G\. Bellemare, and R\. Munos \(2018b\)Distributional reinforcement learning with quantile regression\.InProceedings of the Thirty\-Second AAAI Conference on Artificial Intelligence, \(AAAI\-18\), the 30th innovative Applications of Artificial Intelligence \(IAAI\-18\), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence \(EAAI\-18\), New Orleans, Louisiana, USA, February 2\-7, 2018,S\. A\. McIlraith and K\. Q\. Weinberger \(Eds\.\),pp\. 2892–2901\.Cited by:[§2](https://arxiv.org/html/2607.07769#S2.p2.4)\.
- H\. v\. Hasselt, A\. Guez, and D\. Silver \(2016\)Deep reinforcement learning with double q\-learning\.Association for the Advancement of Artificial Intelligence \(AAAI\)\.Cited by:[§1](https://arxiv.org/html/2607.07769#S1.p1.1),[Table 1](https://arxiv.org/html/2607.07769#S4.T1),[§6](https://arxiv.org/html/2607.07769#S6.p1.2)\.
- M\. Hessel, D\. Budden, F\. Viola, M\. Rosca, E\. Sezener, and T\. Hennigan \(2020\)Optax: composable gradient transformation and optimisationExternal Links:[Link](http://github.com/deepmind/optax)Cited by:[§6](https://arxiv.org/html/2607.07769#S6.p1.2)\.
- M\. Hessel, I\. Danihelka, F\. Viola, A\. Guez, S\. Schmitt, L\. Sifre, T\. Weber, D\. Silver, and H\. van Hasselt \(2021\)Muesli: combining improvements in policy optimization\.InProceedings of the 38th International Conference on Machine Learning, ICML 2021, 18\-24 July 2021, Virtual Event,M\. Meila and T\. Zhang \(Eds\.\),Proceedings of Machine Learning Research, Vol\.139,pp\. 4214–4226\.Cited by:[§1](https://arxiv.org/html/2607.07769#S1.p1.1)\.
- M\. Hessel, J\. Modayil, H\. van Hasselt, T\. Schaul, G\. Ostrovski, W\. Dabney, D\. Horgan, B\. Piot, M\. G\. Azar, and D\. Silver \(2018\)Rainbow: combining improvements in deep reinforcement learning\.InProceedings of the Thirty\-Second AAAI Conference on Artificial Intelligence, \(AAAI\-18\), the 30th innovative Applications of Artificial Intelligence \(IAAI\-18\), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence \(EAAI\-18\), New Orleans, Louisiana, USA, February 2\-7, 2018,S\. A\. McIlraith and K\. Q\. Weinberger \(Eds\.\),pp\. 3215–3222\.Cited by:[§1](https://arxiv.org/html/2607.07769#S1.p1.1)\.
- L\. Kaiser, M\. Babaeizadeh, P\. Milos, B\. Osinski, R\. H\. Campbell, K\. Czechowski, D\. Erhan, C\. Finn, P\. Kozakowski, S\. Levine, A\. Mohiuddin, R\. Sepassi, G\. Tucker, and H\. Michalewski \(2020\)Model based reinforcement learning for atari\.In8th International Conference on Learning Representations, ICLR 2020 \[Spotlight Presentation\],Cited by:[§6](https://arxiv.org/html/2607.07769#S6.p7.1)\.
- S\. Kapturowski, V\. Campos, R\. Jiang, N\. Rakicevic, H\. van Hasselt, C\. Blundell, and A\. P\. Badia \(2023\)Human\-level atari 200x faster\.InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1\-5, 2023,Cited by:[§1](https://arxiv.org/html/2607.07769#S1.p1.1)\.
- K\. P\. Kielak \(2019\)Do recent advancements in model\-based deep reinforcement learning really improve data efficiency?\.CoRR\.Cited by:[footnote 1](https://arxiv.org/html/2607.07769#footnote1)\.
- E\. Korkmaz \(2024\)Understanding and Diagnosing Deep Reinforcement Learning\.InInternational Conference on Machine Learning, ICML 2024,Cited by:[§1](https://arxiv.org/html/2607.07769#S1.p1.1)\.
- E\. Korkmaz \(2025\)Counteractive rl: rethinking core principles for efficient and scalable deep reinforcement learning\.Advances in Neural Information Processing Systems 39: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025 \[Spotlight Presentation\]\.Cited by:[§1](https://arxiv.org/html/2607.07769#S1.p1.1)\.
- H\. Lee, S\. Phatale, H\. Mansoor, T\. Mesnard, J\. Ferret, K\. Lu, C\. Bishop, E\. Hall, V\. Carbune, A\. Rastogi, and S\. Prakash \(2024\)RLAIF vs\. RLHF: scaling reinforcement learning from human feedback with AI feedback\.InForty\-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21\-27, 2024,Cited by:[§1](https://arxiv.org/html/2607.07769#S1.p1.1)\.
- V\. Mnih, K\. Kavukcuoglu, D\. Silver, A\. A\. Rusu, J\. Veness, a\. G\. Bellemare, A\. Graves, M\. Riedmiller, A\. Fidjeland, G\. Ostrovski, S\. Petersen, C\. Beattie, A\. Sadik, Antonoglou, H\. King, D\. Kumaran, D\. Wierstra, S\. Legg, and D\. Hassabis \(2015\)Human\-level control through deep reinforcement learning\.Nature518,pp\. 529–533\.Cited by:[§1](https://arxiv.org/html/2607.07769#S1.p1.1),[Table 1](https://arxiv.org/html/2607.07769#S4.T1),[§6](https://arxiv.org/html/2607.07769#S6.p1.2)\.
- T\. Schaul, J\. Quan, I\. Antonogloua, and D\. Silver \(2016\)Prioritized experience replay\.International Conference on Learning Representations \(ICLR\)\.Cited by:[Table 1](https://arxiv.org/html/2607.07769#S4.T1),[§6](https://arxiv.org/html/2607.07769#S6.p1.2)\.
- J\. Schrittwieser, I\. Antonoglou, T\. Hubert, K\. Simonyan, L\. Sifre, S\. Schmitt, A\. Guez, E\. Lockhart, D\. Hassabis, T\. Graepel, T\. P\. Lillicrap, and D\. Silver \(2020\)Mastering atari, go, chess and shogi by planning with a learned model\.Nat\.588\(7839\),pp\. 604–609\.External Links:[Link](https://doi.org/10.1038/s41586-020-03051-4),[Document](https://dx.doi.org/10.1038/S41586-020-03051-4)Cited by:[§1](https://arxiv.org/html/2607.07769#S1.p1.1)\.
- N\. Stiennon, L\. Ouyang, J\. Wu, D\. M\. Ziegler, R\. Lowe, C\. Voss, A\. Radford, D\. Amodei, and P\. F\. Christiano \(2020\)Learning to summarize with human feedback\.InAdvances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6\-12, 2020, virtual,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\. Balcan, and H\. Lin \(Eds\.\),Cited by:[§1](https://arxiv.org/html/2607.07769#S1.p1.1)\.
- H\. van Hasselt, M\. Hessel, and J\. Aslanides \(2019\)When to use parametric models in reinforcement learning?\.InAdvances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8\-14, 2019, Vancouver, BC, Canada,H\. M\. Wallach, H\. Larochelle, A\. Beygelzimer, F\. d’Alché\-Buc, E\. B\. Fox, and R\. Garnett \(Eds\.\),pp\. 14322–14333\.Cited by:[§6](https://arxiv.org/html/2607.07769#S6.p5.8),[footnote 1](https://arxiv.org/html/2607.07769#footnote1)\.
- H\. van Hasselt \(2010\)Double q\-learning\.InAdvances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010\. Proceedings of a meeting held 6\-9 December 2010, Vancouver, British Columbia, Canada,J\. D\. Lafferty, C\. K\. I\. Williams, J\. Shawe\-Taylor, R\. S\. Zemel, and A\. Culotta \(Eds\.\),pp\. 2613–2621\.Cited by:[§6](https://arxiv.org/html/2607.07769#S6.p1.2)\.
- Z\. Wang, T\. Schaul, M\. Hessel, H\. Van Hasselt, M\. Lanctot, and Nando\. De Freitas \(2016\)Dueling network architectures for deep reinforcement learning\.\.Internation Conference on Machine Learning ICML\.,pp\. 1995–2003\.Cited by:[§1](https://arxiv.org/html/2607.07769#S1.p1.1),[§2](https://arxiv.org/html/2607.07769#S2.p1.22),[Table 1](https://arxiv.org/html/2607.07769#S4.T1),[§6](https://arxiv.org/html/2607.07769#S6.p1.2),[§6](https://arxiv.org/html/2607.07769#S6.p2.6),[§6](https://arxiv.org/html/2607.07769#S6.p5.8)\.
- D\. Yarats, I\. Kostrikov, and R\. Fergus \(2021\)Image augmentation is all you need: regularizing deep reinforcement learning from pixels\.In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3\-7, 2021 \[Spotlight Presentation\],Cited by:[§6](https://arxiv.org/html/2607.07769#S6.p5.8)\.
- W\. Ye, S\. Liu, T\. Kurutach, P\. Abbeel, and Y\. Gao \(2021\)Mastering atari games with limited data\.InAdvances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6\-14, 2021,,Cited by:[§6](https://arxiv.org/html/2607.07769#S6.p6.12)\.
- A\. Zanette, A\. Lazaric, M\. J\. Kochenderfer, and E\. Brunskill \(2020\)Learning near optimal policies with low inherent bellman error\.InProceedings of the 37th International Conference on Machine Learning, ICML 2020, 13\-18 July 2020, Virtual Event,Proceedings of Machine Learning Research, Vol\.119,pp\. 10978–10989\.External Links:[Link](http://proceedings.mlr.press/v119/zanette20a.html)Cited by:[Theorem 3\.1](https://arxiv.org/html/2607.07769#S3.Thmtheorem1)\.