Gated Q-learning: Add Off-Policy Bias to Taste

# Add Off-Policy Bias to Taste
Source: [https://arxiv.org/html/2607.28916](https://arxiv.org/html/2607.28916)
Gated Q\-learning: Add Off\-Policy Bias to Taste

Brett Daley

Keywords:Q\-learning, Off\-Policy Learning, Bias\-Variance Trade\-Off, Multistep Returns, Eligibility Traces\.

SummaryMultistep credit assignment is critical for sample\-efficient reinforcement learning, yet managing off\-policy bias in Q\-learning remains a fundamental challenge\. For 30 years, practitioners have been limited to a binary choice: eliminate the bias at the cost of severely truncated eligibility traces \(Watkins’ Q\(λ\\lambda\)\), or ignore the bias to learn faster while injecting detrimental errors into the value estimates \(Peng’s Q\(λ\\lambda\)\)\. Modern off\-policy estimators fail to resolve this tension, as importance\-sampling ratios collapse under Q\-learning’s greedy target policy\. We introduce Gated Q\-learning, a novel algorithmic framework that ends this dilemma by smoothly interpolating between the two historical extremes\. Rather than relying on importance sampling, our approach employs a continuous, state\-action\-dependent gating mechanism to selectively attenuate eligibility traces in an exploration\-aware manner\. We provide a rigorous theoretical foundation for this mechanism, proving that the expected operator remains a contraction mapping and deriving its exact fixed point\. Empirical evaluations verify that intermediate gating safely enables longer credit\-assignment horizons, yielding faster initial learning than either extreme\. Gated Q\-learning offers a simple alternative to importance sampling while enabling customization of the effective multistep horizon and the amount of off\-policy bias in Q\-learning agents\.

Contribution\(s\)1\.We identify a new, general class of Q\(λ\\lambda\) algorithms that utilize state\-action\-dependent trace\-decay values\. This offers a novel perspective on partial off\-policy bias correction in Q\-learning methods, where importance sampling cannot be applied\. Context:State\-action\-dependent traces for Expected Sarsa have been previously studied to control off\-policy bias in conjunction with importance sampling\(e\.g\., Munos et al\.,[2016](https://arxiv.org/html/2607.28916#bib.bib27); Sutton & Barto,[2018](https://arxiv.org/html/2607.28916#bib.bib33), Ch\. 12\.8\)\. To the best of our knowledge, this idea has never been explored in Q\-learning beyond Watkins’ Q\(λ\\lambda\)\(Watkins,[1989](https://arxiv.org/html/2607.28916#bib.bib34)\), which applies exploration\-conditional trace cuts to eliminate off\-policy bias\.2\.We propose Gated Q\(λ\\lambda\), which implements soft, exploration\-conditional trace cuts to mitigate off\-policy bias while preserving multistep credit assignment\. We very briefly discuss annn\-step version as well\. Context:Gated Q\(λ\\lambda\) interpolates smoothly between the classic methods of Watkins’ Q\(λ\\lambda\)\(Watkins,[1989](https://arxiv.org/html/2607.28916#bib.bib34)\)and Peng’s Q\(λ\\lambda\)\(Peng & Williams,[1996](https://arxiv.org/html/2607.28916#bib.bib28)\)\.3\.We conduct a large\-scale hyperparameter sweep in a random\-walk environment adapted for off\-policy control, generating detailed heatmaps to visualize the influence of step size, trace decay, and gating on Gated Q\(λ\\lambda\)\. Our heatmaps clearly illustrate a performance trade\-off between Watkins’ Q\(λ\\lambda\) and Peng’s Q\(λ\\lambda\)\. Context:Sutton & Barto \([2018](https://arxiv.org/html/2607.28916#bib.bib33), Ex\. 7\.1\)describes the 19\-state random walk that we adapt for our experiment\.4\.We derive the value\-function operator underlying this general class of Q\(λ\\lambda\) algorithms with state\-action\-dependent traces, formally proving how the chosen traces impact its contraction rate and fixed point\. Context:Kozuno et al\. \([2021](https://arxiv.org/html/2607.28916#bib.bib21)\)derived similar results for Peng’s Q\(λ\\lambda\)\. Our theorems significantly generalize these to the newly identified class of Q\(λ\\lambda\) algorithms\.

###### Abstract

Multistep credit assignment is critical for sample\-efficient reinforcement learning, yet managing off\-policy bias in Q\-learning remains a fundamental challenge\. For 30 years, practitioners have been limited to a binary choice: eliminate the bias at the cost of severely truncated eligibility traces \(Watkins’ Q\(λ\\lambda\)\), or ignore the bias to learn faster while injecting detrimental errors into the value estimates \(Peng’s Q\(λ\\lambda\)\)\. Modern off\-policy estimators fail to resolve this tension, as importance\-sampling ratios collapse under Q\-learning’s greedy target policy\. We introduce Gated Q\-learning, a novel algorithmic framework that ends this dilemma by smoothly interpolating between the two historical extremes\. Rather than relying on importance sampling, our approach employs a continuous, state\-action\-dependent gating mechanism to selectively attenuate eligibility traces in an exploration\-aware manner\. We provide a rigorous theoretical foundation for this mechanism, proving that the expected operator remains a contraction mapping and deriving its exact fixed point\. Empirical evaluations verify that intermediate gating safely enables longer credit\-assignment horizons, yielding faster initial learning than either extreme\. Gated Q\-learning offers a simple alternative to importance sampling while enabling customization of the effective multistep horizon and the amount of off\-policy bias in Q\-learning agents\.

## 1Introduction

Despite its simplicity, Q\-learning\(Watkins,[1989](https://arxiv.org/html/2607.28916#bib.bib34)\)remains a staple of modern reinforcement learning \(RL\)\. The appeal of Q\-learning lies in its theoretical elegance and its decoupling of the behavior policy from the target policy, allowing agents to continuously refine estimates of optimal values while exploring the environment or learning from historical replay buffers\. In deep RL specifically, where neural networks serve as function approximators, Q\-learning underpins the success of some of the most sample\-efficient methods to date, including Deep Q\-Networks\(DQN; Mnih et al\.,[2015](https://arxiv.org/html/2607.28916#bib.bib24)\), Rainbow\(Hessel et al\.,[2018](https://arxiv.org/html/2607.28916#bib.bib17)\), and Parallel Q\-Networks\(PQN; Gallici et al\.,[2025](https://arxiv.org/html/2607.28916#bib.bib13)\)\. It also plays a critical role in offline RL\(e\.g\., Fujimoto et al\.,[2019](https://arxiv.org/html/2607.28916#bib.bib12); Kumar et al\.,[2019](https://arxiv.org/html/2607.28916#bib.bib22);[2020](https://arxiv.org/html/2607.28916#bib.bib23); Kostrikov et al\.,[2022](https://arxiv.org/html/2607.28916#bib.bib20)\), where its off\-policy nature is ideal for learning from static datasets\. Consequently, advancing the algorithmic foundations of Q\-learning directly translates to broader improvements across a vast array of deep RL architectures\.

Standard Q\-learning is rooted in11\-step temporal\-difference \(TD\) learning\(Sutton,[1988](https://arxiv.org/html/2607.28916#bib.bib31)\), which struggles to assign credit quickly over long time horizons\. Multistep learning is crucial for accelerating this process, but the naive application of forward\-view return estimators such asnn\-step returns orλ\\lambda\-returns is strongly biased in off\-policy settings\. This bias stems from the distributional mismatch between the agent’s exploratory behavior and the targeted greedy behavior\. The theoretically correct approach is to eliminate this bias by truncating the multistep return estimates whenever an exploratory action is taken, as in Watkins’ Q\(λ\\lambda\)\(Watkins,[1989](https://arxiv.org/html/2607.28916#bib.bib34)\)\. However, empirical evidence indicates that simply*ignoring*these corrections often yields superior performance\(Daley & Amato,[2019](https://arxiv.org/html/2607.28916#bib.bib7); Hernandez\-Garcia & Sutton,[2018](https://arxiv.org/html/2607.28916#bib.bib16)\), which is the exact motivation behind Peng’s Q\(λ\\lambda\)\(Peng & Williams,[1996](https://arxiv.org/html/2607.28916#bib.bib28)\)\. Practitioners are thus faced with a limited choice: either strictly eliminate the bias at the cost of severe truncation and slower learning, or accept the bias and ultimately limit the length of multistep returns that can be safely deployed\.

Although multistep off\-policy estimators that enable finer\-grained control over off\-policy bias do exist, they rely on importance sampling\(Kahn & Marshall,[1953](https://arxiv.org/html/2607.28916#bib.bib19)\)and are therefore not compatible with Q\-learning\. Key examples include Tree Backup\(Precup et al\.,[2000](https://arxiv.org/html/2607.28916#bib.bib29)\), Retrace\(Munos et al\.,[2016](https://arxiv.org/html/2607.28916#bib.bib27)\), and Recency\-Bounded Importance Sampling\(Daley et al\.,[2023](https://arxiv.org/html/2607.28916#bib.bib8)\)—all of which adapt the degree of reinforcement based on the ratio between the action probabilities assigned by the target and behavior policies\. However, in Q\-learning, the target policy is strictly greedy, causing the importance\-sampling ratio to become binary\-valued\. Consequently, this whole class of methods degenerates into the same aggressive Watkins\-style correction, which, as previously mentioned, fails to preserve the long credit\-assignment horizons needed for fast learning\.

There is a clear need for a new mechanism to regulate off\-policy bias in multistep Q\-learning, without the use of importance sampling\. We propose a novel approach that utilizes adaptiveλ\\lambda\-values to partially “gate” the propagation of the eligibility trace specifically when exploratory \(non\-greedy\) actions are taken\. This algorithm, which we call*Gated Q\(λ\\lambda\)*, mitigates some but not all of the off\-policy bias while preserving the trace along greedy trajectories\. This strategy interpolates smoothly between the principled \(but slow\) Watkins’ update and the biased \(but fast\) Peng’s update\. We hypothesize that balancing this trade\-off leads to superior learning compared to either extreme\. Gated Q\-learning is conceptually analogous to the gating mechanisms found in Long Short\-Term Memory \(LSTM\) networks\(Hochreiter & Schmidhuber,[1997](https://arxiv.org/html/2607.28916#bib.bib18)\), Gated Recurrent Units\(GRUs; Cho et al\.,[2014](https://arxiv.org/html/2607.28916#bib.bib4); Chung et al\.,[2014](https://arxiv.org/html/2607.28916#bib.bib5)\), and gated attention\(Xu et al\.,[2015](https://arxiv.org/html/2607.28916#bib.bib36); Dhingra et al\.,[2017](https://arxiv.org/html/2607.28916#bib.bib10)\), which inspire its name, although its role is distinct—it modulates credit assignment in RL\.

Our paper is dedicated to deeply understanding the properties and implications of this gating mechanism in off\-policy credit assignment\. We primarily focus on eligibility traces andλ\\lambda\-returns, although we briefly discuss annn\-step variant as well \(see[Section˜4\.2](https://arxiv.org/html/2607.28916#S4.SS2)\)\. We first derive Gated Q\(λ\\lambda\) as a special case of a new, more general Q\(λ\\lambda\) class which permits state\-action\-dependentλ\\lambda\-values while targeting a greedy policy—the latter being the key differentiator from ’s\([2016](https://arxiv.org/html/2607.28916#bib.bib27)\)per\-decision operator\. This greatly broadens the scope of our theoretical analysis while helping to contextualize and justify the specific choice of our adaptive gating strategy\. We then conduct a focused hyperparameter study in a random walk to illustrate how the gating mechanism impacts credit assignment and learning speed\. Building upon these empirical insights, we formally analyze Gated Q\(λ\\lambda\) to establish its contraction rate and fixed point, providing clear theoretical proof of the trade\-off between trace preservation and off\-policy bias\. Our results demonstrate that there are still fundamental Q\-learning improvements to be discovered, and that faster learning and lower asymptotic error can be simultaneously achieved by adjusting off\-policy bias to a desired, intermediate amount\.

## 2Background

The RL problem considers an agent whose objective is to learn to act in its environment in a way that maximizes its expected cumulative discounted reward\. Since the foundational work ofWatkins \([1989](https://arxiv.org/html/2607.28916#bib.bib34)\), which introduced Q\-learning, the RL problem is most commonly framed as solving a Markov Decision Process \(MDP\) from sample\-based interaction\. The MDP is typically described by a tuple\(𝒮,𝒜,p,r\)\(\\mathcal\{S\},\\mathcal\{A\},p,r\)\. At each discrete time stept≥0t\\geq 0, the agent observes a stateSt∈𝒮S\_\{t\}\\in\\mathcal\{S\}and selects an actionAt∈𝒜A\_\{t\}\\in\\mathcal\{A\}according to a behavior policyb​\(a\|s\)b\(a\|s\), which maps states to probability distributions over actions\. The environment then transitions to a new stateSt\+1S\_\{t\+1\}according to the transition dynamicsp​\(s′∣s,a\)p\(s^\{\\prime\}\\mid s,a\), and the agent receives a scalar rewardRt\+1R\_\{t\+1\}governed by the reward functionr​\(s,a\)r\(s,a\)\. The fundamental objective of the agent is to maximize the expected return, defined as the cumulative sum of discounted future rewards,Gt≔∑i=0∞γi​Rt\+i\+1G\_\{t\}\\coloneqq\\sum\_\{i=0\}^\{\\infty\}\\gamma^\{i\}R\_\{t\+i\+1\}, whereγ∈\[0,1\)\\gamma\\in\[0,1\)is the discount factor\. In off\-policy learning, we explicitly distinguish between this behavior policy \(which explores the environment and generates trajectory data\) and the target policyπ​\(a\|s\)\\pi\(a\|s\), the distinct policy that the algorithm is attempting to evaluate or optimize\.

To measure the quality of a policyπ\\pi, we define the action\-value functionqπ​\(s,a\)≔𝔼π​\[Gt∣St=s,At=a\]q\_\{\\pi\}\(s,a\)\\coloneqq\{\\mathbb\{E\}\_\{\\pi\}\[G\_\{t\}\\mid S\_\{t\}=s,A\_\{t\}=a\]\}, which represents the expected return for taking actionaain statessand subsequently followingπ\\pi\. Q\-learning aims to learn the optimal action\-value functionq∗q\_\{\*\}by representing these estimates as a tabular matrix or parameterized functionQ∈ℝ\|𝒮×𝒜\|Q\\in\\mathbb\{R\}^\{\\absolutevalue\{\\mathcal\{S\}\\times\\mathcal\{A\}\}\}\. The distinguishing feature of Q\-learning is that its target policy is always*greedy*with respect to its current value estimates\. The estimated value of a state is therefore given by

V​\(s\)≔maxa∈𝒜⁡Q​\(s,a\),V\(s\)\\coloneqq\\max\_\{a\\in\\mathcal\{A\}\}Q\(s,a\)\\,,where the value of a terminal state is always defined to be0\. The classic 1\-step Q\-learning update rule is then defined as

Q​\(St,At\)←Q​\(St,At\)\+α​\(Rt\+1\+γ​V​\(St\+1\)−Q​\(St,At\)⏟δt′\),Q\(S\_\{t\},A\_\{t\}\)\\leftarrow Q\(S\_\{t\},A\_\{t\}\)\+\\alpha\\Bigl\(\\underbrace\{R\_\{t\+1\}\+\\gamma V\(S\_\{t\+1\}\)\-Q\(S\_\{t\},A\_\{t\}\)\}\_\{\\delta^\{\\prime\}\_\{t\}\}\\Bigr\)\\,,whereα∈\(0,1\]\\alpha\\in\(0,1\]is the step size\. We refer to the quantityδt′\\delta^\{\\prime\}\_\{t\}as the*Q\-learning \(QL\) error*to differentiate it from the classic state\-value TD error\. Rooted in the Bellman optimality equation\(Bellman,[1957](https://arxiv.org/html/2607.28916#bib.bib2)\), this 1\-step update is highly robust and guaranteed to converge toq∗q\_\{\*\}in tabular settings\(Watkins & Dayan,[1992](https://arxiv.org/html/2607.28916#bib.bib35)\)\. However, because information about future rewards is solely conveyed through immediate bootstrapping, its 1\-step nature results in painfully slow credit assignment, as rewards must propagate backward through the state\-action space one step at a time over repeated episodic interactions\.

To overcome the slow credit assignment, we can consider*multistep*versions of Q\-learning by substituting the 1\-step target with a generalized multistep return estimatorG^t\\hat\{G\}\_\{t\}:

Q​\(St,At\)←Q​\(St,At\)\+α​\(G^t−Q​\(St,At\)\)\.Q\(S\_\{t\},A\_\{t\}\)\\leftarrow Q\(S\_\{t\},A\_\{t\}\)\+\\alpha\\Bigl\(\\hat\{G\}\_\{t\}\-Q\(S\_\{t\},A\_\{t\}\)\\Bigr\)\\,\.DefiningG^t\\hat\{G\}\_\{t\}to safely accelerate learning in off\-policy settings has proven to be highly nontrivial\. If one were to simply use eligibility traces to accumulate a fading record of recent11\-step errors, analogous to on\-policy TD\(λ\\lambda\)\(Sutton,[1988](https://arxiv.org/html/2607.28916#bib.bib31)\), it would produce the following forward\-view target:

Gtλ​\(naive\)=Q​\(St,At\)\+∑i=0∞\(γ​λ\)i​δt\+i′\.G^\{\\lambda\\,\\text\{\(naive\)\}\}\_\{t\}=Q\(S\_\{t\},A\_\{t\}\)\+\\sum\_\{i=0\}^\{\\infty\}\(\\gamma\\lambda\)^\{i\}\\delta^\{\\prime\}\_\{t\+i\}\\,\.This update has become known as*naive*Q\(λ\\lambda\)\(Sutton & Barto,[1998](https://arxiv.org/html/2607.28916#bib.bib32), Sec\. 7\.6\)because it completely fails to address the off\-policy distributional mismatch between the exploratory behavior policy and the greedy target policy\. As a result, it suffers from severe bias and does not converge in off\-policy settings unlessλ\\lambdais kept impractically small\(Harutyunyan et al\.,[2016](https://arxiv.org/html/2607.28916#bib.bib15)\), which heavily counteracts the original multistep benefits\.

Watkins \([1989](https://arxiv.org/html/2607.28916#bib.bib34)\)recognized that off\-policy bias can be completely avoided by cutting the eligibility trace whenever a non\-greedy action is taken\. Define the following greedy indicator function:

greedy\(s,a\)≔a∈arg​maxa′∈𝒜⁡Q​\(s,a′\),\\operatorname\*\{greedy\}\(s,a\)\\coloneqq a\\in\\operatorname\*\{arg\\,max\}\_\{a^\{\\prime\}\\in\\mathcal\{A\}\}Q\(s,a^\{\\prime\}\)\\,,where ties are broken arbitrarily but consistently\. Written as a forward\-view return, Watkins’ Q\(λ\\lambda\), which cuts traces, generates the following target:

Gtλ​\(Watkins\)\\displaystyle G^\{\\lambda\\,\\text\{\(Watkins\)\}\}\_\{t\}=Q​\(St,At\)\+∑i=0∞γi​\(∏j=1iλt\+j\)​δt\+i′,\\displaystyle=Q\(S\_\{t\},A\_\{t\}\)\+\\sum\_\{i=0\}^\{\\infty\}\\gamma^\{i\}\\left\(\\prod\_\{j=1\}^\{i\}\\lambda\_\{t\+j\}\\right\)\\delta^\{\\prime\}\_\{t\+i\}\\,,\(1\)whereλt\\displaystyle\\text\{where\}\\quad\\lambda\_\{t\}=\{λif​greedy\(St,At\),0otherwise,\\displaystyle=\\begin\{cases\}\\lambda&\\text\{if \}\\operatorname\*\{greedy\}\(S\_\{t\},A\_\{t\}\),\\\\ 0&\\text\{otherwise,\}\\end\{cases\}and∏j=10λt\+j≔1\\prod\_\{j=1\}^\{0\}\\lambda\_\{t\+j\}\\coloneqq 1to correctly initialize the first weight\. Watkins’ Q\(λ\\lambda\) fully eliminates off\-policy bias, and in this sense is the theoretically “correct” implementation of Q\-learning with eligibility traces\. Unfortunately, because exploratory actions are frequent during training, traces are cut early and often\. This severe truncation mostly counteracts the speed benefits of multistep learning, making it seem as though Q\-learning is simply at odds with the goals of extended credit assignment\.

To circumvent the severe trace cutting of Watkins’ method,Peng & Williams \([1996](https://arxiv.org/html/2607.28916#bib.bib28)\)introduced a modified version of Q\(λ\\lambda\) that prioritizes rapid credit assignment over strict off\-policy correctness\. To formalize this, we first define the state\-value TD error as

δt≔Rt\+1\+γ​V​\(St\+1\)−V​\(St\)\.\\delta\_\{t\}\\coloneqq R\_\{t\+1\}\+\\gamma V\(S\_\{t\+1\}\)\-V\(S\_\{t\}\)\\,\.Then, Peng’s Q\(λ\\lambda\) target becomes a hybrid of Q\-learning and TD\(λ\\lambda\):

Gtλ​\(Peng\)≔Q​\(St,At\)\+δt′\+∑i=1∞\(γ​λ\)i​δt\+i\.G^\{\\lambda\\,\\text\{\(Peng\)\}\}\_\{t\}\\coloneqq Q\(S\_\{t\},A\_\{t\}\)\+\\delta^\{\\prime\}\_\{t\}\+\\sum\_\{i=1\}^\{\\infty\}\(\\gamma\\lambda\)^\{i\}\\delta\_\{t\+i\}\\,\.Unlike Watkins’ Q\(λ\\lambda\), this estimator never cuts the eligibility trace\. While this makes the update strongly biased in off\-policy settings, it is empirically much faster\. Furthermore, this estimator satisfies an elegant recursive equation:

Gtλ​\(Peng\)=Rt\+1\+γ​\(\(1−λ\)​V​\(St\+1\)\+λ​Gt\+1λ​\(Peng\)\),G^\{\\lambda\\,\\text\{\(Peng\)\}\}\_\{t\}=R\_\{t\+1\}\+\\gamma\\Bigl\(\(1\-\\lambda\)V\(S\_\{t\+1\}\)\+\\lambda G^\{\\lambda\\,\\text\{\(Peng\)\}\}\_\{t\+1\}\\Bigr\)\\,,where the recursion is initialized byV​\(ST\)V\(S\_\{T\}\)at the end of a truncated trajectory orRTR\_\{T\}at the end of an episode\. This makes it very efficient to compute over offline trajectories of experience\. As a result, Peng’s Q\(λ\\lambda\) has become a popular choice in trajectory\-based deep RL\(e\.g\., Harb & Precup,[2016](https://arxiv.org/html/2607.28916#bib.bib14); Mousavi et al\.,[2017](https://arxiv.org/html/2607.28916#bib.bib26); Daley & Amato,[2019](https://arxiv.org/html/2607.28916#bib.bib7); Kozuno et al\.,[2021](https://arxiv.org/html/2607.28916#bib.bib21); Gallici et al\.,[2025](https://arxiv.org/html/2607.28916#bib.bib13); Elelimy et al\.,[2025](https://arxiv.org/html/2607.28916#bib.bib11)\)and is the most common multistep alternative to the widely usednn\-step return\(e\.g\., Hessel et al\.,[2018](https://arxiv.org/html/2607.28916#bib.bib17)\)\. However, in both cases, relying on uncorrected bias is fundamentally flawed, as it ultimately limits the safe effective horizon of the multistep return\.

In a separate line of research, modern methods for off\-policy learning have successfully managed this bias\-variance trade\-off using variations of importance sampling\(Kahn & Marshall,[1953](https://arxiv.org/html/2607.28916#bib.bib19)\)\. These include algorithms such as Tree Backup\(Precup et al\.,[2000](https://arxiv.org/html/2607.28916#bib.bib29)\), Retrace\(Munos et al\.,[2016](https://arxiv.org/html/2607.28916#bib.bib27)\), and Recency\-Bounded Importance Sampling\(Daley et al\.,[2023](https://arxiv.org/html/2607.28916#bib.bib8)\)\. While highly effective for off\-policy learning, these approaches inherently fall under the Sarsa class of algorithms, meaning they evaluate or optimize stochastic \(non\-greedy\) target policies\. They are fundamentally inapplicable to Q\-learning due to its greedy target policy, which causes importance\-sampling ratios to collapse to either zero or nonzero\. This structural limitation precludes the use of modern importance\-sampling methods, including resampling\(e\.g\., Schlegel et al\.,[2019](https://arxiv.org/html/2607.28916#bib.bib30)\), severely limiting the degree to which off\-policy corrections can be controlled in Q\-learning\. As a consequence, there has been surprisingly little progress on multistep*Q\-learning*estimators, leaving practitioners caught between the two extremes of Watkins’ and Peng’s Q\(λ\\lambda\)\.

## 3Variable Q\(λ\\lambda\)

Before introducing our Gated Q\(λ\\lambda\), we begin by formalizing a more general class of*variable*Q\(λ\\lambda\) methods, of which our algorithm is a special case\. When applying multistep Q\-learning, practitioners have historically been faced with a limited choice: either fully correct the off\-policy bias as in Watkins’ Q\(λ\\lambda\), or ignore the bias as in Peng’s Q\(λ\\lambda\)\. As previously established, relying on uncorrected bias is bad because it limits the length of the credit\-assignment window that can be safely deployed before value estimates diverge\. Furthermore, an ideal multistep Q\-learning method must have several properties that make it practically useful: most notably the ability to mediate this bias without sacrificing computational efficiency\. To unify these different approaches, we adopt a variableλ\\lambdaframework, which serves as a prime but underexplored candidate for resolving this dilemma\.

The concept of state\-action\-dependent trace parameters has been discussed bySutton & Barto \([2018](https://arxiv.org/html/2607.28916#bib.bib33), Ch\. 12\.8\), but only for*Sarsa*methods and not Q\-learning\. In our notation, the Q\-learning return is

G~tλ≔Q​\(St,At\)\+δt′\+∑i=1∞γi​\(∏j=1iλt\+j\)​δt\+i,\\tilde\{G\}^\{\\lambda\}\_\{t\}\\coloneqq Q\(S\_\{t\},A\_\{t\}\)\+\\delta^\{\\prime\}\_\{t\}\+\\sum\_\{i=1\}^\{\\infty\}\\gamma^\{i\}\\left\(\\prod\_\{j=1\}^\{i\}\\lambda\_\{t\+j\}\\right\)\\delta\_\{t\+i\}\\,,\(2\)where the overloaded functionλ:𝒮×𝒜→\[0,1\]\\lambda\\colon\\mathcal\{S\}\\times\\mathcal\{A\}\\to\[0,1\]now determinesλt≔λ​\(St,At\)\\lambda\_\{t\}\\coloneqq\\lambda\(S\_\{t\},A\_\{t\}\)\. This return admits a recursive form:

G~tλ=Rt\+1\+γ​\(\(1−λt\+1\)​V​\(St\+1\)\+λt\+1​G~t\+1λ\)\.\\tilde\{G\}^\{\\lambda\}\_\{t\}=R\_\{t\+1\}\+\\gamma\\Bigl\(\(1\-\\lambda\_\{t\+1\}\)V\(S\_\{t\+1\}\)\+\\lambda\_\{t\+1\}\\tilde\{G\}^\{\\lambda\}\_\{t\+1\}\\Bigr\)\\,\.\(3\)The equivalence between[Eq\.˜2](https://arxiv.org/html/2607.28916#S3.E2)and[Eq\.˜3](https://arxiv.org/html/2607.28916#S3.E3)follows from a slight generalization of the fixed\-λ\\lambdaderivation given byDaley \([2025](https://arxiv.org/html/2607.28916#bib.bib6), Sec\. 5\.2\), and it is exactly this recursion that makes the return target efficient to compute, whether with eligibility traces or replayed trajectories\. The only instance of this equation that we found is Eq\. 12\.20 ofSutton & Barto \([2018](https://arxiv.org/html/2607.28916#bib.bib33)\), where it is given for Expected Sarsa but not Q\-learning\. The key distinction is how the target policies are defined \(non\-greedy for Expected Sarsa, greedy for Q\-learning\), which in turn changes the definition ofV​\(St\+1\)V\(S\_\{t\+1\}\)as well as the applicability of importance sampling\. To our knowledge, Watkins’ Q\(λ\\lambda\) is the only existing Q\-learning algorithm that leverages this particular variable\-trace formulation to manage off\-policy data, leaving its broader potential for fine\-grained bias control entirely unexamined\.111Thus, to use this formula with any other choice ofλ​\(s,a\)\\lambda\(s,a\), we must accept some bias and sacrifice convergence toq∗q\_\{\*\}\.

Table 1:Comparison of different Q\(λ\\lambda\) methods that can be expressed by[Algorithm˜1](https://arxiv.org/html/2607.28916#algorithm1)\.Crucially, this framework reveals that both Watkins’ and Peng’s Q\(λ\\lambda\) are special cases of variable Q\(λ\\lambda\) in[Eq\.˜2](https://arxiv.org/html/2607.28916#S3.E2)\. Peng’s Q\(λ\\lambda\) is achieved by simply substituting a constant value ofλ\\lambda\. Watkins’ Q\(λ\\lambda\) is less obvious; it follows becauseλ\\lambdais nonzero if and only if an action is greedy, and thereforeδt′=δt\\delta^\{\\prime\}\_\{t\}=\\delta\_\{t\}*only*in[Eq\.˜1](https://arxiv.org/html/2607.28916#S2.E1)\. We present pseudocode for the universal eligibility\-trace template in[Algorithm˜1](https://arxiv.org/html/2607.28916#algorithm1), offering the specificλ\\lambdadefinitions for the methods in[Table˜1](https://arxiv.org/html/2607.28916#S3.T1)\. In line 9, we indicate the specific choice ofλ\\lambdato produce our flagship algorithm Gated Q\(λ\\lambda\) that is introduced in the next section, but this line can be modified to implement Watkins’ Q\(λ\\lambda\), Peng’s Q\(λ\\lambda\), and any other variable Q\(λ\\lambda\) method described by[Eq\.˜2](https://arxiv.org/html/2607.28916#S3.E2)\.

1Initialize

Q​\(s,a\)Q\(s,a\)arbitrarily and

Z​\(s,a\)←0Z\(s,a\)\\leftarrow 0for all

\(s,a\)\(s,a\)
2

31exfor*t=0,1,…t=0,1,\\dots*do

4Compute greedy action

At∗≔arg​maxa∈𝒜⁡Q​\(St,a\)A^\{\*\}\_\{t\}\\coloneqq\\operatorname\*\{arg\\,max\}\_\{a\\in\\mathcal\{A\}\}Q\(S\_\{t\},a\)
Take action

AtA\_\{t\}according to policy in state

StS\_\{t\}
//ReuseAt∗A^\{\*\}\_\{t\}if needed

5Observe reward

Rt\+1R\_\{t\+1\}and next state

St\+1S\_\{t\+1\}
6

1ex

Gt\(1\)≔\{Rt\+1if​terminal\(St\+1\)Rt\+1\+γ​maxa′∈𝒜⁡Q​\(St\+1,a′\)otherwiseG^\{\(1\)\}\_\{t\}\\coloneqq\\begin\{cases\}R\_\{t\+1\}&\\text\{if \}\\operatorname\*\{terminal\}\(S\_\{t\+1\}\)\\\\ R\_\{t\+1\}\+\\gamma\\max\\limits\_\{a^\{\\prime\}\\in\\mathcal\{A\}\}Q\(S\_\{t\+1\},a^\{\\prime\}\)&\\text\{otherwise\}\\end\{cases\}
//

11\-step return

δt′≔Gt\(1\)−Q​\(St,At\)\\delta^\{\\prime\}\_\{t\}\\coloneqq G^\{\(1\)\}\_\{t\}\-Q\(S\_\{t\},A\_\{t\}\)
//QL error

δt≔Gt\(1\)−Q​\(St,At∗\)\\delta\_\{t\}\\coloneqq G^\{\(1\)\}\_\{t\}\-Q\(S\_\{t\},A^\{\*\}\_\{t\}\)
//TD error

7

1ex

λt≔\{λif​At=At∗λ​χotherwise\\lambda\_\{t\}\\coloneqq\\begin\{cases\}\\lambda&\\text\{if \}A\_\{t\}=A^\{\*\}\_\{t\}\\\\ \\lambda\\chi&\\text\{otherwise\}\\end\{cases\}
//Change to implement other algorithms

8

91exforeach*\(s,a\)\(s,a\)*do

Z​\(s,a\)←γ​λt​Z​\(s,a\)Z\(s,a\)\\leftarrow\\gamma\\lambda\_\{t\}Z\(s,a\)
//Decay trace

Q​\(s,a\)←Q​\(s,a\)\+α​δt​Z​\(s,a\)Q\(s,a\)\\leftarrow Q\(s,a\)\+\\alpha\\delta\_\{t\}Z\(s,a\)
//Apply TD error to past

10

11end foreach

12

1ex

Q​\(St,At\)←Q​\(St,At\)\+α​δt′Q\(S\_\{t\},A\_\{t\}\)\\leftarrow Q\(S\_\{t\},A\_\{t\}\)\+\\alpha\\delta^\{\\prime\}\_\{t\}
//Apply QL error to present

13

141exif*terminal\(St\+1\)\\operatorname\*\{terminal\}\(S\_\{t\+1\}\)*then

Z​\(s,a\)←0Z\(s,a\)\\leftarrow 0for all

\(s,a\)\(s,a\)
//Reset traces

Reset environment state

St\+1S\_\{t\+1\}
//Next episode

15

16else

Z​\(St,At\)←Z​\(St,At\)\+1Z\(S\_\{t\},A\_\{t\}\)\\leftarrow Z\(S\_\{t\},A\_\{t\}\)\+1
//Increment trace

17

18end if

19

20end for

Algorithm 1Gated Q\(λ\\lambda\)
## 4Gated Q\-learning

Variable Q\(λ\\lambda\) offers a large, untapped space of potential multistep Q\-learning methods\. To address the original problem of balancing trace preservation with off\-policy bias mitigation, we focus on a specific subset that we name*Gated Q\-learning*\.

Gated Q\-learning leverages the variable nature ofλt\\lambda\_\{t\}to target and attenuate credit along*exploratory*trajectories\. Unlike Watkins’ Q\(λ\\lambda\), it does this in a smooth and forgiving way; a trajectory earns only partial credit for each non\-greedy action\. It turns out that this strategy achieves a pure interpolation between Watkins’ and Peng’s Q\(λ\\lambda\), yet without using any importance sampling\. We call this method*Gated Q\(λ\\lambda\)*and present it as the main algorithmic contribution of our paper \(see[Section˜4\.1](https://arxiv.org/html/2607.28916#S4.SS1)\), but briefly discuss the special case of annn\-step variant as well \(see[Section˜4\.2](https://arxiv.org/html/2607.28916#S4.SS2)\)\.

### 4\.1Gated Q\(λ\\lambda\)

We can conceptualize credit assignment in Q\(λ\\lambda\) methods as productive errors flowing backward in time to update previous value estimates, like water through a series of pipe segments\. In any segment, the value ofλ\\lambdadetermines the instantaneous flow rate: what proportion of the water passes through and affects downstream values\. A maximum value ofλ=1\\lambda=1allows unconstrained flow, flooding the value estimates with noise \(high variance\)\. A minimum value ofλ=0\\lambda=0shuts off the flow entirely, starving the pipeline of precious water \(high bias\)\. Ideally, these two concerns would be balanced\.

Continuing with this metaphor, non\-greedy actions “pollute” the water from a Q\-learning perspective; exploratory behavior taints the value estimation with bias because such actions do not match the greedy target policy\. However, contaminated water is preferable to no water—the intuitive reason why Peng’s Q\(λ\\lambda\) outperforms Watkins’ Q\(λ\\lambda\), which discards the entire stream at the first sign of pollution\. It is better to selectively regulate the intake of contaminated water to maintain a reasonable flow rate while balancing cleanliness\. This is the core motivation behind Gated Q\(λ\\lambda\)\.

Formally, we introduce a hyperparameterχ∈\[0,1\]\\chi\\in\[0,1\], which we refer to as the “gate\.” When an exploratory action is taken, we attenuate the eligibility of the TD error by an additional factor ofχ\\chi:

λt=\{λif​greedy\(St,At\),λ​χotherwise\.\\lambda\_\{t\}=\\begin\{cases\}\\lambda&\\text\{if \}\\operatorname\*\{greedy\}\(S\_\{t\},A\_\{t\}\),\\\\ \\lambda\\chi&\\text\{otherwise\.\}\\end\{cases\}This definition makes it clear that Gated Q\(λ\\lambda\) achieves a pure mathematical interpolation between the strict, safe updates of Watkins’ Q\(λ\\lambda\) \(χ=0\\chi=0\) and the fast, uncorrected updates of Peng’s Q\(λ\\lambda\) \(χ=1\\chi=1\)\. This targeting mechanism is conceptually analogous to the gating architectures found in LSTM networks\(Hochreiter & Schmidhuber,[1997](https://arxiv.org/html/2607.28916#bib.bib18)\), GRUs\(Cho et al\.,[2014](https://arxiv.org/html/2607.28916#bib.bib4); Chung et al\.,[2014](https://arxiv.org/html/2607.28916#bib.bib5)\), and gated attention\(Xu et al\.,[2015](https://arxiv.org/html/2607.28916#bib.bib36); Dhingra et al\.,[2017](https://arxiv.org/html/2607.28916#bib.bib10)\)\. Just as these architectures smoothly regulate information flow to protect a network’s internal memory state, our mechanism smoothly regulates error flow to protect the action\-value estimates\. However,*unlike*these architectures, the gate here is not a learnable parameter and remains independent of the agent’s function approximator\.

To understand the theoretical implication of this mechanism, letkt:t\+i∈\{0,…,i\}k\_\{t:t\+i\}\\in\\\{0,\\dots,i\\\}denote the total number of non\-greedy actions taken from time stept\+1t\+1throught\+it\+i\. The forward\-view return of Gated Q\(λ\\lambda\) can be expressed as:

Gtλ​\(Gated\)=Q​\(St,At\)\+δt′\+∑i=1∞\(γ​λ\)i​χkt:t\+i​δt\+i\.G^\{\\lambda\\,\\text\{\(Gated\)\}\}\_\{t\}=Q\(S\_\{t\},A\_\{t\}\)\+\\delta^\{\\prime\}\_\{t\}\+\\sum\_\{i=1\}^\{\\infty\}\(\\gamma\\lambda\)^\{i\}\\chi^\{k\_\{t:t\+i\}\}\\delta\_\{t\+i\}\\,\.\(4\)This perspective clearly illustrates that the multistep error decays unconditionally by\(γ​λ\)i\(\\gamma\\lambda\)^\{i\}as in Peng’s method, but is strictly attenuated by an additional factor ofχ\\chifor every non\-greedy choice made along the trajectory\. While this forward view provides theoretical clarity regarding the geometric accumulation of the gate, we note that practical implementations will rely on either the equivalent recursive formula in[Eq\.˜3](https://arxiv.org/html/2607.28916#S3.E3)or standard backward\-view eligibility traces as detailed in[Algorithm˜1](https://arxiv.org/html/2607.28916#algorithm1)\.

### 4\.2nn\-step Gated Q\-learning

As an aside, we note that we can derive annn\-step return target that applies the same gating mechanism, and thus our idea is not specific toλ\\lambda\-returns\. We achieve this by truncating the Gated Q\(λ\\lambda\) target in[Eq\.˜4](https://arxiv.org/html/2607.28916#S4.E4)to justnnsteps and then settingλ=1\\lambda=1\. This yields the following return estimate:

Gt\(n\)≔Q​\(St,At\)\+δt′\+∑i=1n−1γi​χkt:t\+i​δt\+i\.G^\{\(n\)\}\_\{t\}\\coloneqq Q\(S\_\{t\},A\_\{t\}\)\+\\delta^\{\\prime\}\_\{t\}\+\\sum\_\{i=1\}^\{n\-1\}\\gamma^\{i\}\\chi^\{k\_\{t:t\+i\}\}\\delta\_\{t\+i\}\\,\.Although thisnn\-step estimator is very interesting, we do not pursue it further in this work\. We imagine, though, that it could be beneficial to large\-buffer deep RL algorithms in the DQN family\(Mnih et al\.,[2015](https://arxiv.org/html/2607.28916#bib.bib24); Hessel et al\.,[2018](https://arxiv.org/html/2607.28916#bib.bib17)\), which commonly use biased, uncorrectednn\-step returns to improve sample efficiency\. A slight drawback to ournn\-step estimate is that its computational cost scales linearly withnndue to the individual TD errors\. However, with massive hardware parallelization, the cost may not be noticeable for typical values ofnn\.

## 5Hyperparameter Study

Before we formally analyze and evaluate Gated Q\(λ\\lambda\) in[Section˜6](https://arxiv.org/html/2607.28916#S6), we conduct a focused hyperparameter sweep to gain an intuition for how the gating mechanism impacts credit assignment\. Code to reproduce this experiment is available online\.222[https://github\.com/brett\-daley/gated\-q\-learning](https://github.com/brett-daley/gated-q-learning)

For our test environment, we adapt the 19\-state random walk fromSutton & Barto \([2018](https://arxiv.org/html/2607.28916#bib.bib33), Sec\. 12\.1\)\. This environment has 19 linearly connected states and two actions to move left or right\. The agent starts each episode in the central state\. The agent’s behavior policy chooses either action with equal probability\. Reaching the extreme ends of the walk yields a−1\-1or\+1\+1reward \(left and right, respectively\) and terminates the episode\. The simple, linear topology of this environment is ideal for isolating and measuring credit assignment\.

Because the traditional random\-walk experiment is set up for on\-policy prediction, we must modify it for off\-policy control\. We first apply a slight discount factor ofγ=0\.99\\gamma=0\.99, to incentivize the agent to earn rewards expediently\. We then calculate the optimal action\-value function,q∗q\_\{\*\}\. The key difference in our setup is that we train the agents for a fixed number of steps \(500\) instead of episodes, to capture the initial learning speed of the agent\. Each agent’s value function is initialized with negligible Gaussian noise \(σ=10−9\\sigma=10^\{\-9\}\) to break ties at the start\. We record the root\-mean\-square \(RMS\) error betweenQQandq∗q\_\{\*\}on every time step\. Note that if we trained for*too*long, the results would be biased in favor of Watkins’ Q\(λ\\lambda\) because it has no asymptotic bias\. This would fail to capture the phenomenon we really want to study: the ability for trace preservation across off\-policy trajectories to accelerate learning in spite of the bias\. We thus want to examine the early phase of training where trace preservation can be a key performance differentiator\.

![Refer to caption](https://arxiv.org/html/2607.28916v1/images/curves.png)Figure 1:Learning curves for Gated, Peng’s, and Watkins’ Q\(λ\\lambda\) methods with their respective best hyperparameters\. Prediction accuracy is defined as the overall RMS error reduction, where 0% represents no progress and 100% represents perfect convergence toq∗q\_\{\*\}\. Results are averaged over 300 random seeds each\. The shading represents 95% confidence intervals\.We evaluate Gated Q\(λ\\lambda\) with eligibility traces, described in[Algorithm˜1](https://arxiv.org/html/2607.28916#algorithm1), by sweeping overα\\alpha,λ\\lambda, andχ\\chi\. We divide each axis into 21 uniform values from0to11\. This results in 9,261 hyperparameter combinations, each averaged across 300 random seeds, for a total of almost 3 million independent trials\. Note that the extremal values ofχ=0\\chi=0andχ=1\\chi=1correspond to Watkins’ and Peng’s, respectively, so the vast majority of evaluated configurations represent new and previously untested variations of Q\(λ\\lambda\)\.

Table 2:Hyperparameters used in[Figure˜1](https://arxiv.org/html/2607.28916#S5.F1)\.Let us consider how the results are theoretically affected by preserving traces\. Each time the agent moves right, it receives an optimal experience\. Clearly, this experience should be reinforced, and all of the agents weight its influence on past state\-action pairs by the same value:λ\\lambda\. The sole difference in the methods lies in the case where the agent moves*left*—a suboptimal exploratory move\. If the agent were to move right again afterwards, the methods that do not cut traces will ultimately reinforce the correct action in spite of this exploration\. This is the principal mechanism that makes Peng’s Q\(λ\\lambda\) learn faster in practice\. However, if the agent*does*end up receiving the negative reward on the left, it will reinforce a bad experience that would never be taken by the optimal policy, incurring bias\.

We generate learning curves for each hyperparameter configuration\. We invert and normalize the RMS errors to instead measure*prediction accuracy*, where 0% represents the initial error \(no learning\) and 100% represents perfect convergence toq∗q\_\{\*\}\(optimal performance\)\. We take the area under the curve \(AUC\) of each learning curve to produce a single summarizing scalar\. The curves for the best\-performing hyperparameter selection for each method \(withχ\\chifixed at0or11for Watkins’ and Peng’s, respectively; see[Table˜2](https://arxiv.org/html/2607.28916#S5.T2)\) are plotted in[Figure˜1](https://arxiv.org/html/2607.28916#S5.F1)\.

Collectively, the AUC scalars form a 3\-D cube of21×21×2121\\times 21\\times 21voxels\. We identify the globally optimal point atα=0\.95\\alpha=0\.95,λ=1\.00\\lambda=1\.00, andχ=0\.45\\chi=0\.45\. We then cross section the cube through this point along orthogonal planes to generate three 2\-D heatmap visualizations in[Figure˜2](https://arxiv.org/html/2607.28916#S5.F2)\.

![Refer to caption](https://arxiv.org/html/2607.28916v1/images/3D_heatmap.png)Figure 2:Cross\-sectional slices of the 3\-D heatmap generated for the random walk\. Each slice passes through the coordinate of the best hyperparameters found by the search, indicated by a red X \(see[Table˜2](https://arxiv.org/html/2607.28916#S5.T2)\)\. The colors indicate the area under the curve achieved by each hyperparameter configuration, with warmer colors indicating larger values\. Each voxel is averaged over 300 trials\.This experiment reveals three major insights: \(i\) Gated Q\(λ\\lambda\) learns significantly faster than both baselines; \(ii\) the gating mechanism enables the safe use of higherλ\\lambda\-values; and \(iii\) performance is remarkably robust across a wide range of intermediate gate values \(χ∈\[0\.2,0\.6\]\\chi\\in\[0\.2,0\.6\]\)\. Clearly, an intermediateχ\\chiimproves the best performance in this task\. This gives merit to the idea that balancing the bias\-variance trade\-off through a gating mechanism is advantageous\.

## 6Analysis

In this section, we identify and analyze a general value\-function operator underpinning the variable Q\(λ\\lambda\) method introduced in[Section˜3](https://arxiv.org/html/2607.28916#S3)\. Our analysis here is not intended to capture the stochastic behavior of the online eligibility\-trace method presented in[Algorithm˜1](https://arxiv.org/html/2607.28916#algorithm1)\. Instead, we primarily wish to understand the properties of Gated Q\(λ\\lambda\)’s forward\-view return target\.333Standard results in stochastic approximation\(e\.g\., Bertsekas & Tsitsiklis,[1996](https://arxiv.org/html/2607.28916#bib.bib3), Ch\. 4–5\)suggest that a stochastic algorithm with a corresponding contractive operator converges to the same fixed point, given appropriate step\-size schedules and sufficient exploration\. Formally establishing this here is left for future work\.This is especially relevant for deep RL, where value estimates are typically produced by a frozen target network and backward\-view eligibility traces are generally not used\.444We refer the reader toDaley \([2025](https://arxiv.org/html/2607.28916#bib.bib6), Ch\. 1, 7\)for a detailed account\. Forward\-view returns are more compatible with minibatch sampling, which draws transitions out of temporal order\. Although some recent algorithms have revisited backward\-view eligibility traces for deep RL, they currently tend to be less stable and sample\-efficient than replay methods\.Operator theory provides exactly the right tool to understand how the gating mechanism addresses the trade\-off between expected convergence speed and off\-policy bias\.

### 6\.1Variable Q\(λ\\lambda\) Operator

We start by derivingTΛ:ℝ\|𝒮×𝒜\|→ℝ\|𝒮×𝒜\|T^\{\\Lambda\}\\colon\\mathbb\{R\}^\{\\absolutevalue\{\\mathcal\{S\}\\times\\mathcal\{A\}\}\}\\to\\mathbb\{R\}^\{\\absolutevalue\{\\mathcal\{S\}\\times\\mathcal\{A\}\}\}, a general operator capable of expressing the variable Q\(λ\\lambda\) methods introduced in[Section˜3](https://arxiv.org/html/2607.28916#S3), including Watkins’ Q\(λ\\lambda\), Peng’s Q\(λ\\lambda\), and our Gated Q\(λ\\lambda\)\. Here, vectors represent value functions \(e\.g\.,𝒒∈ℝ\|𝒮×𝒜\|\{\\bm\{q\}\}\\in\\mathbb\{R\}^\{\\absolutevalue\{\\mathcal\{S\}\\times\\mathcal\{A\}\}\}representsQQ\), where each element corresponds to the value estimate for a particular state\-action pair\. The operator acts on the state\-action value estimates in the update𝒒←TΛ​𝒒\{\\bm\{q\}\}\\leftarrow T^\{\\Lambda\}\{\\bm\{q\}\}, thus producing the expected value of the targetsG~tλ\\tilde\{G\}^\{\\lambda\}\_\{t\}from[Eq\.˜2](https://arxiv.org/html/2607.28916#S3.E2)for each state\-action pair\. Similarly, if we were to setα\\alphato be very small and holdQQfixed while executing[Algorithm˜1](https://arxiv.org/html/2607.28916#algorithm1)and averaging updates on the side, then the net update would be approximately equal toTΛ​𝒒−𝒒T^\{\\Lambda\}\{\\bm\{q\}\}\-\{\\bm\{q\}\}\. This is the classic forward\-backward equivalence of eligibility traces\(Sutton & Barto,[1998](https://arxiv.org/html/2607.28916#bib.bib32), Sec\. 7\.4\), allowing us to analyze the expected behavior of the return target and the eligibility traces purely through the lens ofTΛT^\{\\Lambda\}\.

To constructTΛT^\{\\Lambda\}, we first define a few fundamental operators based on the bipartite MDP decomposition fromDaley \([2025](https://arxiv.org/html/2607.28916#bib.bib6), Ch\. 2\)\. LetP:ℝ\|𝒮\|→ℝ\|𝒮×𝒜\|P\\colon\\mathbb\{R\}^\{\\absolutevalue\{\\mathcal\{S\}\}\}\\to\\mathbb\{R\}^\{\\absolutevalue\{\\mathcal\{S\}\\times\\mathcal\{A\}\}\}be the transition dynamics operator\. We also define a policy operatorEπ:ℝ\|𝒮×𝒜\|→ℝ\|𝒮\|E\_\{\\pi\}\\colon\\mathbb\{R\}^\{\\absolutevalue\{\\mathcal\{S\}\\times\\mathcal\{A\}\}\}\\to\\mathbb\{R\}^\{\\absolutevalue\{\\mathcal\{S\}\}\}which computes the expected state values under any policyπ\\pi\. LetE∗E\_\{\*\}denote the special case of a*greedy*policyπgreedy\\pi\_\{\\text\{greedy\}\}with respect toQQ\. Finally, to simplify the analysis, we introduce an expansion operatorJ:ℝ\|𝒮\|→ℝ\|𝒮×𝒜\|J\\colon\\mathbb\{R\}^\{\\absolutevalue\{\\mathcal\{S\}\}\}\\to\\mathbb\{R\}^\{\\absolutevalue\{\\mathcal\{S\}\\times\\mathcal\{A\}\}\}, which broadcasts a state’s value to each of its action values\. Note that the reward function is itself treated as a value function,𝒓∈ℝ\|𝒮×𝒜\|\{\\bm\{r\}\}\\in\\mathbb\{R\}^\{\\absolutevalue\{\\mathcal\{S\}\\times\\mathcal\{A\}\}\}, to which dimensionally compatible operators can be applied\.

With these definitions, the Bellman operator for action values is expressed as

T​𝒒≔𝒓\+γ​𝑷​E∗​𝒒\.T\{\\bm\{q\}\}\\coloneqq\{\\bm\{r\}\}\+\\gamma\{\\bm\{P\}\}E\_\{\*\}\{\\bm\{q\}\}\\,\.\(5\)Note that we bold both value functions and*linear*operators whenever they appear as such algebraic quantities\. Next, we define the trace\-decay matrix𝚲\{\\bm\{\\Lambda\}\}\. Because the trace\-decay parameterλ​\(s,a\)\\lambda\(s,a\)can vary by state\-action pair,𝚲\{\\bm\{\\Lambda\}\}is a diagonal matrix of size\|𝒮×𝒜\|×\|𝒮×𝒜\|\\absolutevalue\{\\mathcal\{S\}\\times\\mathcal\{A\}\}\\times\\absolutevalue\{\\mathcal\{S\}\\times\\mathcal\{A\}\}\. The elements are defined by

Λi,j≔\{λ​\(si,ai\)ifi=j,0if​i≠j,∀i,j∈\{1,…,\|𝒮×𝒜\|\}\.\{\\Lambda\}\_\{i,j\}\\coloneqq\\begin\{cases\}\\lambda\(s\_\{i\},a\_\{i\}\)&\\text\{if $i=j$\}\\,,\\\\ 0&\\text\{if \}i\\neq j\\,,\\end\{cases\}\\quad\\forall\\,i,j\\in\\\{1,\\dots,\\absolutevalue\{\\mathcal\{S\}\\times\\mathcal\{A\}\}\\\}\\,\.It is crucial to note that because methods like Gated Q\(λ\\lambda\) defineλ​\(s,a\)\\lambda\(s,a\)as a function of the current value estimates,𝚲\{\\bm\{\\Lambda\}\}is an implicit function ofQQin general\. This theoretically makes it a nonlinear operator\. However, because we analyze only the pure policy\-evaluation setting in our work, we can safely assume thatQQis a fixed quantity and therefore treat𝚲\{\\bm\{\\Lambda\}\}as a linear operator\. This is a standard and necessary assumption for policy evaluation, which makes the analysis tractable while perfectly mirroring the stale\-target setting common in modern RL architectures\.

###### Proposition 1\.

The operatorTΛT^\{\\Lambda\}for variable Q\(λ\\lambda\) has the closed\-form definition

TΛ​𝒒≔\(𝑰−γ​𝑷​𝑬b​𝚲\)−1​\(T​𝒒−γ​𝑷​𝑬b​𝚲​𝑱​E∗​𝒒\)\.T^\{\\Lambda\}\{\\bm\{q\}\}\\coloneqq\(\{\\bm\{I\}\}\-\\gamma\{\\bm\{P\}\}\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\)^\{\-1\}\(T\{\\bm\{q\}\}\-\\gamma\{\\bm\{P\}\}\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\{\\bm\{J\}\}E\_\{\*\}\{\\bm\{q\}\}\)\\,\.\(6\)

###### Proof\.

See[Section˜A\.1](https://arxiv.org/html/2607.28916#A1.SS1)\. The result follows by converting the expectation of the recursive target in[Eq\.˜3](https://arxiv.org/html/2607.28916#S3.E3)to operator form, and then manipulating it algebraically to isolateTΛT^\{\\Lambda\}\. ∎

[Eq\.˜6](https://arxiv.org/html/2607.28916#S6.E6)reveals that theTΛT^\{\\Lambda\}operator takes the generic form𝒁−1​\(𝒚\+𝑿​𝒒\)\{\\bm\{Z\}\}^\{\-1\}\(\{\\bm\{y\}\}\+\{\\bm\{X\}\}\{\\bm\{q\}\}\)\. Here, the matrix𝒁−1=\(𝑰−γ​𝑷​𝑬b​𝚲\)−1\{\\bm\{Z\}\}^\{\-1\}=\(\{\\bm\{I\}\}\-\\gamma\{\\bm\{P\}\}\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\)^\{\-1\}represents the expected eligibility trace for every state\-action pair under the behavior policybband the chosen decay scheme𝚲\{\\bm\{\\Lambda\}\}\. The inner vector term\(𝒚\+𝑿​𝒒\)=𝒓\+γ​𝑷​E∗​𝒒−γ​𝑷​𝑬b​𝚲​𝑱​E∗​𝒒\(\{\\bm\{y\}\}\+\{\\bm\{X\}\}\{\\bm\{q\}\}\)=\{\\bm\{r\}\}\+\\gamma\{\\bm\{P\}\}E\_\{\*\}\{\\bm\{q\}\}\-\\gamma\{\\bm\{P\}\}\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\{\\bm\{J\}\}E\_\{\*\}\{\\bm\{q\}\}fills in the complementary gaps of the decayed trace with bootstrapped state\-value estimates derived frommaxa∈𝒜⁡Q​\(s,a\)\\max\_\{a\\in\\mathcal\{A\}\}Q\(s,a\)\.

### 6\.2Contraction Rate

To quantify the expected error reduction achieved by a single application ofTΛT^\{\\Lambda\}, our next step is to prove that it is a contraction mapping\. Let∥⋅∥∞\\norm\{\\cdot\}\_\{\\infty\}denote the maximum norm of a vector or matrix \(i\.e\., the maximum row sum of the absolute values\)\. We formally define a contraction mapping with respect to this norm below\.

###### Definition 1\.

A value\-function operatorH:ℝ\|𝒮×𝒜\|→ℝ\|𝒮×𝒜\|H\\colon\\mathbb\{R\}^\{\\absolutevalue\{\\mathcal\{S\}\\times\\mathcal\{A\}\}\}\\to\\mathbb\{R\}^\{\\absolutevalue\{\\mathcal\{S\}\\times\\mathcal\{A\}\}\}is a contraction mapping if and only if there exists a constantβ∈\[0,1\)\\beta\\in\[0,1\)such that,∀𝐪1,𝐪2∈ℝ\|𝒮×𝒜\|\\forall\\,\{\\bm\{q\}\}\_\{1\},\{\\bm\{q\}\}\_\{2\}\\in\\mathbb\{R\}^\{\\absolutevalue\{\\mathcal\{S\}\\times\\mathcal\{A\}\}\},

‖H​𝒒1−H​𝒒2‖∞≤β​‖𝒒1−𝒒2‖∞\.\\norm\{H\{\\bm\{q\}\}\_\{1\}\-H\{\\bm\{q\}\}\_\{2\}\}\_\{\\infty\}\\leq\\beta\\norm\{\{\\bm\{q\}\}\_\{1\}\-\{\\bm\{q\}\}\_\{2\}\}\_\{\\infty\}\\,\.

We seek to find a*contraction modulus*forTΛT^\{\\Lambda\}: a constantβ\\betathat satisfies[Definition˜1](https://arxiv.org/html/2607.28916#Thmdefinition1)\. To facilitate this, let𝟏𝒮∈ℝ\|𝒮\|\{\\bm\{1\}\}\_\{\\mathcal\{S\}\}\\in\\mathbb\{R\}^\{\\absolutevalue\{\\mathcal\{S\}\}\}and𝟏𝒮×𝒜∈ℝ\|𝒮×𝒜\|\{\\bm\{1\}\}\_\{\\mathcal\{S\}\\times\\mathcal\{A\}\}\\in\\mathbb\{R\}^\{\\absolutevalue\{\\mathcal\{S\}\\times\\mathcal\{A\}\}\}denote the all\-ones vectors\.

###### Theorem 1\.

Let𝐜≔𝐄b​𝚲​𝟏𝒮×𝒜\{\\bm\{c\}\}\\coloneqq\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\{\\bm\{1\}\}\_\{\\mathcal\{S\}\\times\\mathcal\{A\}\}andc−≔mins∈𝒮⁡c​\(s\)c^\{\-\}\\coloneqq\\min\_\{s\\in\\mathcal\{S\}\}c\(s\)\. The operatorTΛT^\{\\Lambda\}is a contraction mapping with modulus

β\\displaystyle\\beta=γ​\(1−c−\)1−γ​c−\.\\displaystyle=\\frac\{\\gamma\(1\-c^\{\-\}\)\}\{1\-\\gamma c^\{\-\}\}\\,\.

###### Proof\.

See[Section˜A\.2](https://arxiv.org/html/2607.28916#A1.SS2)\. ∎

The valuec−c^\{\-\}represents the smallest state\-conditional expected trace\-decay value across all states\. Becauseβ\\betais monotonically decreasing withc−c^\{\-\}, this formalizes the intuition that the trace preservation inherent to methods like Peng’s Q\(λ\\lambda\) is indeed beneficial to convergence speed in expectation\. When we apply a constant value ofλ\\lambdato all state\-action pairs, the formula gracefully collapses back toβ=γ​\(1−λ\)/\(1−γ​λ\)\\beta=\\gamma\(1\-\\lambda\)\\mathbin\{/\}\(1\-\\gamma\\lambda\), which exactly matches the known contraction modulus for standard TD\(λ\\lambda\) and Peng’s Q\(λ\\lambda\)\(see, e\.g\., Daley et al\.,[2024](https://arxiv.org/html/2607.28916#bib.bib9), Proof of Prop\. 3\.11\)\.

For Gated Q\(λ\\lambda\), the algorithm’s trace decay depends on whether a greedy action is taken:λ\\lambdaif so andλ​χ\\lambda\\chiif not\. If the behavior policy has probabilitypg​\(s\)p\_\{g\}\(s\)of acting greedily in statess, then the expected trace value isc​\(s\)=pg​\(s\)⋅λ\+\(1−pg​\(s\)\)⋅λ​χc\(s\)=p\_\{g\}\(s\)\\cdot\\lambda\+\(1\-p\_\{g\}\(s\)\)\\cdot\\lambda\\chi\. This meansc−c^\{\-\}is determined by the*least\-greedy*state: the state where the agent currently explores the most\.555If the agent is executing anϵ\\epsilon\-greedy policy, then exploration is state\-wise uniform and we simply substitute the probability of taking a greedy action:pg​\(s\)=1−ϵ\+ϵ/\|𝒜\|p\_\{g\}\(s\)=1\-\\epsilon\+\\epsilon\\mathbin\{/\}\\absolutevalue\{\\mathcal\{A\}\}\.This implies that a gate ofχ<1\\chi<1actually*reduces*expected convergence speed, making the method more like Watkins’ Q\(λ\\lambda\)\. However, this can be compensated for by increasingλ\\lambdabeyond what would be considered safe for Peng’s Q\(λ\\lambda\), as we show with the fixed\-point analysis in the next subsection\.

### 6\.3Fixed Point and Off\-Policy Bias

We have now established thatTΛT^\{\\Lambda\}is a contraction mapping\. By the Banach fixed\-point theorem\(Banach,[1922](https://arxiv.org/html/2607.28916#bib.bib1)\),TΛT^\{\\Lambda\}admits a*unique*fixed point, to which repeated application of the operator converges: i\.e\.,limi→∞\(TΛ\)i​𝒒\\lim\_\{i\\to\\infty\}\(T^\{\\Lambda\}\)^\{i\}\{\\bm\{q\}\}exists and is the same for every𝒒∈ℝ\|𝒮×𝒜\|\{\\bm\{q\}\}\\in\\mathbb\{R\}^\{\\absolutevalue\{\\mathcal\{S\}\\times\\mathcal\{A\}\}\}\. We must now identify the nature of this fixed point\.

We once again leverage the recursive formula to unpack the components of the operator\. By doing so, we find that the algorithm implicitly evaluates a composite policy that interpolates between the behavior and \(greedy\) target policies\.

###### Theorem 2\.

Letπmix\\pi\_\{\\text\{mix\}\}be the mixture policy defined by the following policy operator:

𝑬πmix≔𝑬b​𝚲\+\(𝑰𝒮−𝑬b​𝚲​𝑱\)​E∗\.\{\\bm\{E\}\}\_\{\\pi\_\{\\text\{mix\}\}\}\\coloneqq\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\+\(\{\\bm\{I\}\}\_\{\\mathcal\{S\}\}\-\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\{\\bm\{J\}\}\)E\_\{\*\}\\,\.The unique fixed point of operatorTΛT^\{\\Lambda\}is exactly the action\-value function ofπmix\\pi\_\{\\text\{mix\}\}:

𝒒πmix≔\(𝑰−γ​𝑷​𝑬πmix\)−1​𝒓\.\{\\bm\{q\}\}\_\{\\pi\_\{\\text\{mix\}\}\}\\coloneqq\(\{\\bm\{I\}\}\-\\gamma\{\\bm\{P\}\}\{\\bm\{E\}\}\_\{\\pi\_\{\\text\{mix\}\}\}\)^\{\-1\}\{\\bm\{r\}\}\\,\.

###### Proof\.

See[Section˜A\.3](https://arxiv.org/html/2607.28916#A1.SS3)\. ∎

For every choice of theλ​\(s,a\)\\lambda\(s,a\)function, variable Q\(λ\\lambda\) converges to the value function corresponding to a policy which mixes the behavior and greedy policies\. For clarity, we show the point\-wise definition ofπmix\\pi\_\{\\text\{mix\}\}which is derived directly from the operators’ definitions:

πmix​\(a\|s\)≔b​\(a\|s\)​λ​\(s,a\)\+\(1−c​\(s\)\)​πgreedy​\(a\|s\)\.\\pi\_\{\\text\{mix\}\}\(a\|s\)\\coloneqq b\(a\|s\)\\lambda\(s,a\)\+\(1\-c\(s\)\)\\pi\_\{\\text\{greedy\}\}\(a\|s\)\\,\.Kozuno et al\. \([2021](https://arxiv.org/html/2607.28916#bib.bib21), Th\. 2\)proved that Peng’s Q\(λ\\lambda\) converges to the value function of a mixture policy formed by the convex mixtureλ​b​\(a\|s\)\+\(1−λ\)​πgreedy​\(a\|s\)\\lambda b\(a\|s\)\+\(1\-\\lambda\)\\pi\_\{\\text\{greedy\}\}\(a\|s\), which matches our result above whenλ​\(s,a\)=λ\\lambda\(s,a\)=\\lambdafor all\(s,a\)\(s,a\)\. Thus, our[Theorem˜2](https://arxiv.org/html/2607.28916#Thmtheorem2)is a pure generalization\.

This result is fascinating in that it formalizes the precise way in which variable Q\(λ\\lambda\) methods trade off trace preservation with off\-policy bias\. In particular, on\-policy values seep into the return estimation whenever we simultaneously haveλ​\(s,a\)\>0\\lambda\(s,a\)\>0andb​\(a\|s\)\>0b\(a\|s\)\>0for a non\-greedy pair\(s,a\)\(s,a\), indicating that any amount of preserved trace during exploration contributes bias\. This is because such a pair hasπmix​\(a\|s\)=b​\(a\|s\)​λ​\(s,a\)\>0\\pi\_\{\\text\{mix\}\}\(a\|s\)=b\(a\|s\)\\lambda\(s,a\)\>0, whereasπgreedy​\(a\|s\)=0\\pi\_\{\\text\{greedy\}\}\(a\|s\)=0, altering the fixed point because the mixture policy is no longer greedy\. This affirms that Watkins’ Q\(λ\\lambda\) is the only way to truly eliminate off\-policy bias by settingλ​\(s,a\)=0\\lambda\(s,a\)=0whenever\(s,a\)\(s,a\)is exploratory\. Nevertheless, biased Q\(λ\\lambda\) methods can still converge to a*nearby*value function, and converge to it much faster than Watkins’ Q\(λ\\lambda\), making them practically useful\. Adaptingλ\\lambdabased on the state\-action pair provides the exact capability to adjust where and how much off\-policy bias is added to the return estimation\. Gated Q\(λ\\lambda\) offers this capability via the gating mechanismχ∈\[0,1\]\\chi\\in\[0,1\]which only targets exploratory actions, enabling users to fine\-tune the degree of off\-policy bias without negatively impacting unbiased greedy actions\.

## 7Conclusion

We introduced Gated Q\-learning, a novel algorithmic framework that resolves the 30\-year tension between the aggressive trace\-cutting of Watkins’ Q\(λ\\lambda\) and the uncorrected bias of Peng’s Q\(λ\\lambda\)\. By formalizing partial off\-policy corrections through a state\-action\-dependent gating mechanism, we provide a new continuum of algorithms that finely balances long\-term credit assignment with off\-policy bias\. Our theoretical analysis proves that this strategy always guarantees a contraction mapping and convergence, with smoothly bounded \(but generally nonzero\) fixed\-point bias\. Key limitations include the newly added gate hyperparameterχ\\chi, which in theory must be tuned in conjunction withλ\\lambda\. However, the wide plateau nearχ≈0\.5\\chi\\approx 0\.5in[Figure˜2](https://arxiv.org/html/2607.28916#S5.F2)suggests a favorable margin of error when setting this hyperparameter, at least in the tested environment\. Another limitation is that we did not make any direct empirical comparisons with importance\-sampled estimators like Retrace\(Munos et al\.,[2016](https://arxiv.org/html/2607.28916#bib.bib27)\), as we focused specifically on greedy Q\-learning algorithms\. Evaluating the relative effectiveness of these two distinct off\-policy corrections remains an important direction for future work\. Finally, our analysis did not consider the variance of the return estimates due to the complexity of the problem, though it seems likely that forgoing importance\-sampling ratios is a significant boon to variance reduction\.

Although not evaluated in the deep RL setting, Gated Q\-learning can seamlessly integrate into existing agents, including trajectory\-replay methods that useλ\\lambda\-returns such as A3C\(Mnih et al\.,[2016](https://arxiv.org/html/2607.28916#bib.bib25)\), DQN\(λ\\lambda\)\(Daley & Amato,[2019](https://arxiv.org/html/2607.28916#bib.bib7)\), and PQN\(Gallici et al\.,[2025](https://arxiv.org/html/2607.28916#bib.bib13)\), as well as minibatch\-replay methods in the DQN family\(Mnih et al\.,[2015](https://arxiv.org/html/2607.28916#bib.bib24); Hessel et al\.,[2018](https://arxiv.org/html/2607.28916#bib.bib17)\)that usenn\-step returns \(recall[Section˜4\.2](https://arxiv.org/html/2607.28916#S4.SS2)\)\. We see significant potential to improve performance in these algorithms, as off\-policy bias tends to be extreme under experience replay\. Here, Gated Q\-learning can serve as a simpler and more efficient alternative to importance sampling, though we make no claim of superior performance yet\.

## References

- Banach \(1922\)Stefan Banach\.Sur les opérations dans les ensembles abstraits et leur application aux équations intégrales\.*Fundamenta Mathematicae*, 3\(1\):133–181, 1922\.
- Bellman \(1957\)Richard Bellman\.*Dynamic Programming*\.Princeton University Press, 1957\.
- Bertsekas & Tsitsiklis \(1996\)Dimitri P\. Bertsekas and John N\. Tsitsiklis\.*Neuro\-Dynamic Programming*\.Athena Scientific, 1996\.
- Cho et al\. \(2014\)Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio\.Learning phrase representations using RNN encoder\-decoder for statistical machine translation\.In*Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2014\.
- Chung et al\. \(2014\)Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio\.Empirical evaluation of gated recurrent neural networks on sequence modeling\.In*NeurIPS Deep Learning and Representation Learning Workshop*, 2014\.
- Daley \(2025\)Brett Daley\.*Multistep Credit Assignment in Deep Reinforcement Learning*\.PhD thesis, University of Alberta, 2025\.
- Daley & Amato \(2019\)Brett Daley and Christopher Amato\.Reconcilingλ\\lambda\-returns with experience replay\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2019\.
- Daley et al\. \(2023\)Brett Daley, Martha White, Christopher Amato, and Marlos C\. Machado\.Trajectory\-aware eligibility traces for off\-policy reinforcement learning\.In*International Conference on Machine Learning \(ICML\)*, 2023\.
- Daley et al\. \(2024\)Brett Daley, Martha White, and Marlos C\. Machado\.Averagingnn\-step returns reduces variance in reinforcement learning\.In*International Conference on Machine Learning \(ICML\)*, 2024\.
- Dhingra et al\. \(2017\)Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov\.Gated\-attention readers for text comprehension\.In*Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2017\.
- Elelimy et al\. \(2025\)Esraa Elelimy, Brett Daley, Andrew Patterson, Marlos C\. Machado, Adam White, and Martha White\.Deep reinforcement learning with gradient eligibility traces\.*Reinforcement Learning Journal*, 2025\.
- Fujimoto et al\. \(2019\)Scott Fujimoto, David Meger, and Doina Precup\.Off\-policy deep reinforcement learning without exploration\.In*International Conference on Machine Learning \(ICML\)*, 2019\.
- Gallici et al\. \(2025\)Matteo Gallici, Mattie Fellows, Benjamin Ellis, Bartomeu Pou, Ivan Masmitja, Jakob Nicolaus Foerster, and Mario Martin\.Simplifying deep temporal difference learning\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.
- Harb & Precup \(2016\)Jean Harb and Doina Precup\.Investigating recurrence and eligibility traces in deep Q\-networks\.In*NeurIPS Deep Reinforcement Learning Workshop*, 2016\.
- Harutyunyan et al\. \(2016\)Anna Harutyunyan, Marc G\. Bellemare, Tom Stepleton, and Rémi Munos\.Q\(λ\\lambda\) with off\-policy corrections\.In*International Conference on Algorithmic Learning Theory \(ALT\)*, 2016\.
- Hernandez\-Garcia & Sutton \(2018\)J\. Fernando Hernandez\-Garcia and Richard S\. Sutton\.Understanding multi\-step deep reinforcement learning: A systematic study of the DQN target\.In*NeurIPS Deep Reinforcement Learning Workshop*, 2018\.
- Hessel et al\. \(2018\)Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver\.Rainbow: Combining improvements in deep reinforcement learning\.In*AAAI Conference on Artificial Intelligence \(AAAI\)*, 2018\.
- Hochreiter & Schmidhuber \(1997\)Sepp Hochreiter and Jürgen Schmidhuber\.Long short\-term memory\.*Neural Computation*, 9\(8\):1735–1780, 1997\.
- Kahn & Marshall \(1953\)Herman Kahn and Andy W\. Marshall\.Methods of reducing sample size in Monte Carlo computations\.*Journal of the Operations Research Society of America*, 1\(5\):263–278, 1953\.
- Kostrikov et al\. \(2022\)Ilya Kostrikov, Ashvin Nair, and Sergey Levine\.Offline reinforcement learning with implicit Q\-learning\.In*International Conference on Learning Representations \(ICLR\)*, 2022\.
- Kozuno et al\. \(2021\)Tadashi Kozuno, Yunhao Tang, Mark Rowland, Rémi Munos, Steven Kapturowski, Will Dabney, Michal Valko, and David Abel\.Revisiting Peng’s Q\(λ\\lambda\) for modern reinforcement learning\.In*International Conference on Machine Learning \(ICML\)*, 2021\.
- Kumar et al\. \(2019\)Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine\.Stabilizing off\-policy Q\-learning via bootstrapping error reduction\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2019\.
- Kumar et al\. \(2020\)Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine\.Conservative Q\-learning for offline reinforcement learning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2020\.
- Mnih et al\. \(2015\)Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A\. Rusu, Joel Veness, Marc G\. Bellemare, Alex Graves, Martin Riedmiller, Andreas K\. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis\.Human\-level control through deep reinforcement learning\.*Nature*, 518\(7540\):529–533, 2015\.
- Mnih et al\. \(2016\)Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu\.Asynchronous methods for deep reinforcement learning\.In*International Conference on Machine Learning \(ICML\)*, 2016\.
- Mousavi et al\. \(2017\)Seyed Sajad Mousavi, Michael Schukat, Enda Howley, and Patrick Mannion\.Applying Q\(λ\\lambda\)\-learning in deep reinforcement learning to play Atari games\.In*AAMAS Adaptive Learning Agents Workshop*, 2017\.
- Munos et al\. \(2016\)Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc G\. Bellemare\.Safe and efficient off\-policy reinforcement learning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2016\.
- Peng & Williams \(1996\)Jing Peng and Ronald J\. Williams\.Incremental multi\-step Q\-learning\.*Machine Learning*, 22:283–290, 1996\.
- Precup et al\. \(2000\)Doina Precup, Richard S\. Sutton, and Satinder Singh\.Eligibility traces for off\-policy policy evaluation\.In*International Conference on Machine Learning \(ICML\)*, 2000\.
- Schlegel et al\. \(2019\)Matthew Schlegel, Wesley Chung, Daniel Graves, Jian Qian, and Martha White\.Importance resampling for off\-policy prediction\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2019\.
- Sutton \(1988\)Richard S\. Sutton\.Learning to predict by the methods of temporal differences\.*Machine Learning*, 3\(1\):9–44, 1988\.
- Sutton & Barto \(1998\)Richard S\. Sutton and Andrew G\. Barto\.*Reinforcement Learning: An Introduction*\.The MIT Press, 1st edition, 1998\.
- Sutton & Barto \(2018\)Richard S\. Sutton and Andrew G\. Barto\.*Reinforcement Learning: An Introduction*\.The MIT Press, 2nd edition, 2018\.
- Watkins \(1989\)Christopher J\. C\. H\. Watkins\.*Learning from Delayed Rewards*\.PhD thesis, University of Cambridge, 1989\.
- Watkins & Dayan \(1992\)Christopher J\. C\. H\. Watkins and Peter Dayan\.Q\-learning\.*Machine Learning*, 8:279–292, 1992\.
- Xu et al\. \(2015\)Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio\.Show, attend and tell: Neural image caption generation with visual attention\.In*International Conference on Machine Learning \(ICML\)*, 2015\.

Supplementary Materials

*The following content was not necessarily subject to peer review\.*

## Appendix AProofs

This section contains the full mathematical proofs omitted from the main paper for exposition ease\.

### A\.1Proof of[˜1](https://arxiv.org/html/2607.28916#Thmprop1)

See[1](https://arxiv.org/html/2607.28916#Thmprop1)

###### Proof\.

We first rewrite the recursiveλ\\lambda\-return formula from[Eq\.˜3](https://arxiv.org/html/2607.28916#S3.E3)as

G~tλ=Rt\+1\+γ​V​\(St\+1\)\+γ​λt\+1​\(G~t\+1λ−V​\(St\+1\)\)\.\\tilde\{G\}^\{\\lambda\}\_\{t\}=R\_\{t\+1\}\+\\gamma V\(S\_\{t\+1\}\)\+\\gamma\\lambda\_\{t\+1\}\\Bigl\(\\tilde\{G\}^\{\\lambda\}\_\{t\+1\}\-V\(S\_\{t\+1\}\)\\Bigr\)\\,\.To convert this to an operator equation, we must convert the expected values of these quantities to their matrix or vector equivalents:

TΛ​𝒒=T​𝒒\+γ​𝑷​𝑬b​𝚲​\(TΛ​𝒒−𝑱​E∗​𝒒\)\.T^\{\\Lambda\}\{\\bm\{q\}\}=T\{\\bm\{q\}\}\+\\gamma\{\\bm\{P\}\}\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\(T^\{\\Lambda\}\{\\bm\{q\}\}\-\{\\bm\{J\}\}E\_\{\*\}\{\\bm\{q\}\}\)\\,\.\(7\)This is because the first term is the Bellman operator from[Eq\.˜5](https://arxiv.org/html/2607.28916#S6.E5), the time\-step shift and decay are handled by multiplyingγ​𝑷​𝑬b​𝚲\\gamma\{\\bm\{P\}\}\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}, and the inner term references the operator itself again, minus the state\-value estimate \(which must be expanded by𝑱\{\\bm\{J\}\}to match the𝒮×𝒜\\mathcal\{S\}\\times\\mathcal\{A\}dimension\)\.

We conclude by solving[Eq\.˜7](https://arxiv.org/html/2607.28916#A1.E7)algebraically forTΛT^\{\\Lambda\}\. \(Here and throughout, the unsubscripted identity matrix𝑰\{\\bm\{I\}\}has dimension\|𝒮×𝒜\|×\|𝒮×𝒜\|\\absolutevalue\{\\mathcal\{S\}\\times\\mathcal\{A\}\}\\times\\absolutevalue\{\\mathcal\{S\}\\times\\mathcal\{A\}\}\.\) Rearranging gives

\(𝑰−γ​𝑷​𝑬b​𝚲\)​TΛ​𝒒=T​𝒒−γ​𝑷​𝑬b​𝚲​𝑱​E∗​𝒒,\(\{\\bm\{I\}\}\-\\gamma\{\\bm\{P\}\}\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\)T^\{\\Lambda\}\{\\bm\{q\}\}=T\{\\bm\{q\}\}\-\\gamma\{\\bm\{P\}\}\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\{\\bm\{J\}\}E\_\{\*\}\{\\bm\{q\}\}\\,,\(8\)and left\-multiplying both sides by\(𝑰−γ​𝑷​𝑬b​𝚲\)−1\(\{\\bm\{I\}\}\-\\gamma\{\\bm\{P\}\}\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\)^\{\-1\}yields[Eq\.˜6](https://arxiv.org/html/2607.28916#S6.E6), completing the derivation\. ∎

### A\.2Proof of[Theorem˜1](https://arxiv.org/html/2607.28916#Thmtheorem1)

See[1](https://arxiv.org/html/2607.28916#Thmtheorem1)

###### Proof\.

Let𝒒1,𝒒2∈ℝ\|𝒮×𝒜\|\{\\bm\{q\}\}\_\{1\},\{\\bm\{q\}\}\_\{2\}\\in\\mathbb\{R\}^\{\\absolutevalue\{\\mathcal\{S\}\\times\\mathcal\{A\}\}\}\. To simplify notation, we define the differencesΔ​TΛ​𝒒≔TΛ​𝒒1−TΛ​𝒒2\\Delta T^\{\\Lambda\}\{\\bm\{q\}\}\\coloneqq\{T^\{\\Lambda\}\{\\bm\{q\}\}\_\{1\}\-T^\{\\Lambda\}\{\\bm\{q\}\}\_\{2\}\},Δ​𝒒≔𝒒1−𝒒2\\Delta\{\\bm\{q\}\}\\coloneqq\{\{\\bm\{q\}\}\_\{1\}\-\{\\bm\{q\}\}\_\{2\}\}, andΔ​𝒗≔E∗​𝒒1−E∗​𝒒2\\Delta\{\\bm\{v\}\}\\coloneqq\{E\_\{\*\}\{\\bm\{q\}\}\_\{1\}\-E\_\{\*\}\{\\bm\{q\}\}\_\{2\}\}\. Note thatΔ​T​𝒒=γ​𝑷​Δ​𝒗\\Delta T\{\\bm\{q\}\}=\\gamma\{\\bm\{P\}\}\\Delta\{\\bm\{v\}\}\. From substitution into the recursive definition ofTΛT^\{\\Lambda\}in[Eq\.˜7](https://arxiv.org/html/2607.28916#A1.E7)and by linearity, we have

Δ​TΛ​𝒒\\displaystyle\\Delta T^\{\\Lambda\}\{\\bm\{q\}\}=γ​𝑷​Δ​𝒗\+γ​𝑷​𝑬b​𝚲​\(Δ​TΛ​𝒒−𝑱​Δ​𝒗\)\\displaystyle=\\gamma\{\\bm\{P\}\}\\Delta\{\\bm\{v\}\}\+\\gamma\{\\bm\{P\}\}\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\(\\Delta T^\{\\Lambda\}\{\\bm\{q\}\}\-\{\\bm\{J\}\}\\Delta\{\\bm\{v\}\}\)=γ​𝑷​𝑬b​𝚲​Δ​TΛ​𝒒\+γ​𝑷​\(𝑰𝒮−𝑬b​𝚲​𝑱\)​Δ​𝒗\.\\displaystyle=\\gamma\{\\bm\{P\}\}\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\\Delta T^\{\\Lambda\}\{\\bm\{q\}\}\+\\gamma\{\\bm\{P\}\}\(\{\\bm\{I\}\}\_\{\\mathcal\{S\}\}\-\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\{\\bm\{J\}\}\)\\Delta\{\\bm\{v\}\}\\,\.To form an upper bound, we apply the*element\-wise*absolute value to each vector and invoke the triangle inequality\. We then use the fact that\|𝒙\|≤‖𝒙‖∞​𝟏\\absolutevalue\{\{\\bm\{x\}\}\}\\leq\\norm\{\{\\bm\{x\}\}\}\_\{\\infty\}\{\\bm\{1\}\}holds element\-wise\. Additionally, observe that\(𝑰𝒮−𝑬b​𝚲​𝑱\)​𝟏𝒮=𝟏𝒮−𝑬b​𝚲​𝟏𝒮×𝒜=𝟏𝒮−𝒄\(\{\\bm\{I\}\}\_\{\\mathcal\{S\}\}\-\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\{\\bm\{J\}\}\)\{\\bm\{1\}\}\_\{\\mathcal\{S\}\}=\{\\bm\{1\}\}\_\{\\mathcal\{S\}\}\-\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\{\\bm\{1\}\}\_\{\\mathcal\{S\}\\times\\mathcal\{A\}\}=\{\\bm\{1\}\}\_\{\\mathcal\{S\}\}\-\{\\bm\{c\}\}\. Most of the terms have nonnegative components and can be safely pulled out of the absolute value:

\|Δ​TΛ​𝒒\|\\displaystyle\\absolutevalue\{\\Delta T^\{\\Lambda\}\{\\bm\{q\}\}\}≤γ​𝑷​𝑬b​𝚲​\|Δ​TΛ​𝒒\|\+γ​𝑷​\(𝑰𝒮−𝑬b​𝚲​𝑱\)​\|Δ​𝒗\|\\displaystyle\\leq\\gamma\{\\bm\{P\}\}\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\\absolutevalue\{\\Delta T^\{\\Lambda\}\{\\bm\{q\}\}\}\+\\gamma\{\\bm\{P\}\}\(\{\\bm\{I\}\}\_\{\\mathcal\{S\}\}\-\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\{\\bm\{J\}\}\)\\absolutevalue\{\\Delta\{\\bm\{v\}\}\}≤γ​𝑷​\(‖Δ​TΛ​𝒒‖∞​𝑬b​𝚲​𝟏𝒮×𝒜\+‖Δ​𝒗‖∞​\(𝑰𝒮−𝑬b​𝚲​𝑱\)​𝟏𝒮\)\\displaystyle\\leq\\gamma\{\\bm\{P\}\}\\Bigl\(\\norm\{\\Delta T^\{\\Lambda\}\{\\bm\{q\}\}\}\_\{\\infty\}\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\{\\bm\{1\}\}\_\{\\mathcal\{S\}\\times\\mathcal\{A\}\}\+\\norm\{\\Delta\{\\bm\{v\}\}\}\_\{\\infty\}\(\{\\bm\{I\}\}\_\{\\mathcal\{S\}\}\-\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\{\\bm\{J\}\}\)\{\\bm\{1\}\}\_\{\\mathcal\{S\}\}\\Bigr\)=γ​𝑷​\(‖Δ​TΛ​𝒒‖∞​𝒄\+‖Δ​𝒗‖∞​\(𝟏𝒮−𝒄\)\)\\displaystyle=\\gamma\{\\bm\{P\}\}\\Bigl\(\\norm\{\\Delta T^\{\\Lambda\}\{\\bm\{q\}\}\}\_\{\\infty\}\{\\bm\{c\}\}\+\\norm\{\\Delta\{\\bm\{v\}\}\}\_\{\\infty\}\(\{\\bm\{1\}\}\_\{\\mathcal\{S\}\}\-\{\\bm\{c\}\}\)\\Bigr\)≤γ​𝑷​\(‖Δ​TΛ​𝒒‖∞​𝒄\+‖Δ​𝒒‖∞​\(𝟏𝒮−𝒄\)\),\\displaystyle\\leq\\gamma\{\\bm\{P\}\}\\Bigl\(\\norm\{\\Delta T^\{\\Lambda\}\{\\bm\{q\}\}\}\_\{\\infty\}\{\\bm\{c\}\}\+\\norm\{\\Delta\{\\bm\{q\}\}\}\_\{\\infty\}\(\{\\bm\{1\}\}\_\{\\mathcal\{S\}\}\-\{\\bm\{c\}\}\)\\Bigr\)\\,,where the last step follows because themax\\maxoperator is non\-expansive, hence‖Δ​𝒗‖∞≤‖Δ​𝒒‖∞\\norm\{\\Delta\{\\bm\{v\}\}\}\_\{\\infty\}\\leq\\norm\{\\Delta\{\\bm\{q\}\}\}\_\{\\infty\}\.

Note that‖𝑷‖∞=1\\norm\{\{\\bm\{P\}\}\}\_\{\\infty\}=1because𝑷\{\\bm\{P\}\}is a stochastic matrix\. Taking the norm of both sides yields

‖Δ​TΛ​𝒒‖∞\\displaystyle\\norm\{\\Delta T^\{\\Lambda\}\{\\bm\{q\}\}\}\_\{\\infty\}≤γ​‖‖Δ​TΛ​𝒒‖∞​𝒄\+‖Δ​𝒒‖∞​\(𝟏𝒮−𝒄\)‖∞\\displaystyle\\leq\\gamma\\Bigl\\\|\\norm\{\\Delta T^\{\\Lambda\}\{\\bm\{q\}\}\}\_\{\\infty\}\{\\bm\{c\}\}\+\\norm\{\\Delta\{\\bm\{q\}\}\}\_\{\\infty\}\(\{\\bm\{1\}\}\_\{\\mathcal\{S\}\}\-\{\\bm\{c\}\}\)\\Bigr\\\|\_\{\\infty\}=γ​maxs⁡\(c​\(s\)​‖Δ​TΛ​𝒒‖∞\+\(1−c​\(s\)\)​‖Δ​𝒒‖∞\)\.\\displaystyle=\\gamma\\max\_\{s\}\\Bigl\(c\(s\)\\norm\{\\Delta T^\{\\Lambda\}\{\\bm\{q\}\}\}\_\{\\infty\}\+\(1\-c\(s\)\)\\norm\{\\Delta\{\\bm\{q\}\}\}\_\{\\infty\}\\Bigr\)\\,\.\(9\)This inequality presents a convex combination of‖Δ​TΛ​𝒒‖∞\\norm\{\\Delta T^\{\\Lambda\}\{\\bm\{q\}\}\}\_\{\\infty\}and‖Δ​𝒒‖∞\\norm\{\\Delta\{\\bm\{q\}\}\}\_\{\\infty\}\. If we assume for a moment that‖Δ​TΛ​𝒒‖∞\>‖Δ​𝒒‖∞\\norm\{\\Delta T^\{\\Lambda\}\{\\bm\{q\}\}\}\_\{\\infty\}\>\\norm\{\\Delta\{\\bm\{q\}\}\}\_\{\\infty\}, the convex combination would be strictly less than‖Δ​TΛ​𝒒‖∞\\norm\{\\Delta T^\{\\Lambda\}\{\\bm\{q\}\}\}\_\{\\infty\}\. This would imply‖Δ​TΛ​𝒒‖∞<γ​‖Δ​TΛ​𝒒‖∞\\norm\{\\Delta T^\{\\Lambda\}\{\\bm\{q\}\}\}\_\{\\infty\}<\\gamma\\norm\{\\Delta T^\{\\Lambda\}\{\\bm\{q\}\}\}\_\{\\infty\}, which is impossible sinceγ≤1\\gamma\\leq 1\. Therefore, it must be universally true that‖Δ​TΛ​𝒒‖∞≤‖Δ​𝒒‖∞\\norm\{\\Delta T^\{\\Lambda\}\{\\bm\{q\}\}\}\_\{\\infty\}\\leq\\norm\{\\Delta\{\\bm\{q\}\}\}\_\{\\infty\}\.

Because‖Δ​𝒒‖∞\\norm\{\\Delta\{\\bm\{q\}\}\}\_\{\\infty\}is the larger of the two quantities, the convex combination is maximized by placing as much weight as possible on it\. This means maximizing1−c​\(s\)1\-c\(s\)in[Eq\.˜9](https://arxiv.org/html/2607.28916#A1.E9), which is achieved by settingc​\(s\)=c−c\(s\)=c^\{\-\}\. Substitutingc−c^\{\-\}into the bound yields

‖Δ​TΛ​𝒒‖∞≤γ​c−​‖Δ​TΛ​𝒒‖∞\+γ​\(1−c−\)​‖Δ​𝒒‖∞,\\norm\{\\Delta T^\{\\Lambda\}\{\\bm\{q\}\}\}\_\{\\infty\}\\leq\\gamma c^\{\-\}\\norm\{\\Delta T^\{\\Lambda\}\{\\bm\{q\}\}\}\_\{\\infty\}\+\\gamma\(1\-c^\{\-\}\)\\norm\{\\Delta\{\\bm\{q\}\}\}\_\{\\infty\}\\,,which rearranges to

‖Δ​TΛ​𝒒‖∞≤γ​\(1−c−\)1−γ​c−⏟β​‖Δ​𝒒‖∞\.\\norm\{\\Delta T^\{\\Lambda\}\{\\bm\{q\}\}\}\_\{\\infty\}\\leq\\underbrace\{\\frac\{\\gamma\(1\-c^\{\-\}\)\}\{1\-\\gamma c^\{\-\}\}\}\_\{\\beta\}\\norm\{\\Delta\{\\bm\{q\}\}\}\_\{\\infty\}\\,\.Because0≤c−≤10\\leq c^\{\-\}\\leq 1by definition ofλ​\(s,a\)\\lambda\(s,a\), the coefficientβ\\betais strictly less than11wheneverγ<1\\gamma<1, confirming thatTΛT^\{\\Lambda\}is a contraction mapping with the stated modulus\. ∎

### A\.3Proof of[Theorem˜2](https://arxiv.org/html/2607.28916#Thmtheorem2)

See[2](https://arxiv.org/html/2607.28916#Thmtheorem2)

###### Proof\.

At the fixed point, we must haveTΛ​𝒒=𝒒T^\{\\Lambda\}\{\\bm\{q\}\}=\{\\bm\{q\}\}\. Starting from[Eq\.˜8](https://arxiv.org/html/2607.28916#A1.E8), expandingT​𝒒T\{\\bm\{q\}\}using[Eq\.˜5](https://arxiv.org/html/2607.28916#S6.E5), and then substituting𝒒\{\\bm\{q\}\}forTΛ​𝒒T^\{\\Lambda\}\{\\bm\{q\}\}, we obtain

\(𝑰−γ​𝑷​𝑬b​𝚲\)​𝒒\\displaystyle\(\{\\bm\{I\}\}\-\\gamma\{\\bm\{P\}\}\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\)\{\\bm\{q\}\}=𝒓\+γ​𝑷​E∗​𝒒−γ​𝑷​𝑬b​𝚲​𝑱​E∗​𝒒\\displaystyle=\{\\bm\{r\}\}\+\\gamma\{\\bm\{P\}\}E\_\{\*\}\{\\bm\{q\}\}\-\\gamma\{\\bm\{P\}\}\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\{\\bm\{J\}\}E\_\{\*\}\{\\bm\{q\}\}𝒒\\displaystyle\{\\bm\{q\}\}=𝒓\+γ​𝑷​𝑬b​𝚲​𝒒\+γ​𝑷​E∗​𝒒−γ​𝑷​𝑬b​𝚲​𝑱​E∗​𝒒\\displaystyle=\{\\bm\{r\}\}\+\\gamma\{\\bm\{P\}\}\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\{\\bm\{q\}\}\+\\gamma\{\\bm\{P\}\}E\_\{\*\}\{\\bm\{q\}\}\-\\gamma\{\\bm\{P\}\}\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\{\\bm\{J\}\}E\_\{\*\}\{\\bm\{q\}\}=𝒓\+γ​𝑷​\(𝑬b​𝚲\+E∗−𝑬b​𝚲​𝑱​E∗\)​𝒒\\displaystyle=\{\\bm\{r\}\}\+\\gamma\{\\bm\{P\}\}\(\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\+E\_\{\*\}\-\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\{\\bm\{J\}\}E\_\{\*\}\)\{\\bm\{q\}\}\(10\)=𝒓\+γ​𝑷​𝑬πmix​𝒒,\\displaystyle=\{\\bm\{r\}\}\+\\gamma\{\\bm\{P\}\}\{\\bm\{E\}\}\_\{\\pi\_\{\\text\{mix\}\}\}\{\\bm\{q\}\}\\,,\(11\)where the last step substitutes the policy operator defined in the theorem\. Note that factoring out𝒒\{\\bm\{q\}\}in[Eq\.˜10](https://arxiv.org/html/2607.28916#A1.E10)is justified because, as established earlier,QQis assumed to be fixed for the operator evaluation\. This allows us to locally treat the greedy expectationE∗E\_\{\*\}as a linear operator\.

[Eq\.˜11](https://arxiv.org/html/2607.28916#A1.E11)reveals that the fixed point takes the standard Bellman form for the previously defined mixture policy\. To complete the proof, we must verify that𝑬πmix\{\\bm\{E\}\}\_\{\\pi\_\{\\text\{mix\}\}\}is a valid stochastic matrix, thereby representing a realizable policy\. This means𝑬πmix\{\\bm\{E\}\}\_\{\\pi\_\{\\text\{mix\}\}\}must comprise exclusively nonnegative elements and its rows must sum to one\.

We first establish nonnegativity\. By definition, the policy operators𝑬b\{\\bm\{E\}\}\_\{b\}andE∗E\_\{\*\}, as well as the trace matrix𝚲\{\\bm\{\\Lambda\}\}, contain exclusively nonnegative elements\. The only term that could theoretically introduce negative values is the subtraction in𝑰𝒮−𝑬b​𝚲​𝑱\{\\bm\{I\}\}\_\{\\mathcal\{S\}\}\-\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\{\\bm\{J\}\}\. However, the operator𝑬b​𝚲​𝑱\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\{\\bm\{J\}\}effectively maps a state to itself with the conditionally expected trace decayc​\(s\)=∑ab​\(a\|s\)​λ​\(s,a\)c\(s\)=\\sum\_\{a\}b\(a\|s\)\\lambda\(s,a\)\. Becauseλ​\(s,a\)∈\[0,1\]\\lambda\(s,a\)\\in\[0,1\]and the behavior policybbis a valid probability distribution, it is guaranteed thatc​\(s\)∈\[0,1\]c\(s\)\\in\[0,1\]for alls∈𝒮s\\in\\mathcal\{S\}\. Consequently,𝑰𝒮−𝑬b​𝚲​𝑱\{\\bm\{I\}\}\_\{\\mathcal\{S\}\}\-\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\{\\bm\{J\}\}is a diagonal matrix whose diagonal entries are1−c​\(s\)≥01\-c\(s\)\\geq 0\.

Finally, we verify the row sums by applying the operator to the all\-ones vector\. Recall from the proof of[Theorem˜1](https://arxiv.org/html/2607.28916#Thmtheorem1)that𝒄=𝑬b​𝚲​𝟏𝒮×𝒜\{\\bm\{c\}\}=\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\{\\bm\{1\}\}\_\{\\mathcal\{S\}\\times\\mathcal\{A\}\}and𝟏𝒮−𝒄=\(𝑰𝒮−𝑬b​𝚲​𝑱\)​𝟏𝒮\{\\bm\{1\}\}\_\{\\mathcal\{S\}\}\-\{\\bm\{c\}\}=\(\{\\bm\{I\}\}\_\{\\mathcal\{S\}\}\-\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\{\\bm\{J\}\}\)\{\\bm\{1\}\}\_\{\\mathcal\{S\}\}\. Therefore,

𝑬πmix​𝟏𝒮×𝒜\\displaystyle\{\\bm\{E\}\}\_\{\\pi\_\{\\text\{mix\}\}\}\{\\bm\{1\}\}\_\{\\mathcal\{S\}\\times\\mathcal\{A\}\}=𝑬b​𝚲​𝟏𝒮×𝒜\+\(𝑰𝒮−𝑬b​𝚲​𝑱\)​E∗​𝟏𝒮×𝒜\\displaystyle=\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\{\\bm\{1\}\}\_\{\\mathcal\{S\}\\times\\mathcal\{A\}\}\+\(\{\\bm\{I\}\}\_\{\\mathcal\{S\}\}\-\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\{\\bm\{J\}\}\)E\_\{\*\}\{\\bm\{1\}\}\_\{\\mathcal\{S\}\\times\\mathcal\{A\}\}=𝒄\+\(𝑰𝒮−𝑬b​𝚲​𝑱\)​E∗​𝟏𝒮×𝒜\\displaystyle=\{\\bm\{c\}\}\+\(\{\\bm\{I\}\}\_\{\\mathcal\{S\}\}\-\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\{\\bm\{J\}\}\)E\_\{\*\}\{\\bm\{1\}\}\_\{\\mathcal\{S\}\\times\\mathcal\{A\}\}=𝒄\+\(𝑰𝒮−𝑬b​𝚲​𝑱\)​𝟏𝒮\\displaystyle=\{\\bm\{c\}\}\+\(\{\\bm\{I\}\}\_\{\\mathcal\{S\}\}\-\{\\bm\{E\}\}\_\{b\}\{\\bm\{\\Lambda\}\}\{\\bm\{J\}\}\)\{\\bm\{1\}\}\_\{\\mathcal\{S\}\}=𝒄\+𝟏𝒮−𝒄\\displaystyle=\{\\bm\{c\}\}\+\{\\bm\{1\}\}\_\{\\mathcal\{S\}\}\-\{\\bm\{c\}\}=𝟏𝒮\.\\displaystyle=\{\\bm\{1\}\}\_\{\\mathcal\{S\}\}\\,\.Since applying𝑬πmix\{\\bm\{E\}\}\_\{\\pi\_\{\\text\{mix\}\}\}to the all\-ones vector perfectly recovers the all\-ones vector \(of the appropriate dimensions\), the row sums of the implied transition matrix all equal11\. Coupled with the fact that it has nonnegative components,𝑬πmix\{\\bm\{E\}\}\_\{\\pi\_\{\\text\{mix\}\}\}represents a valid policy and the proof is complete\. ∎