Collaborative Weighting with Pessimistic Critic for Mitigating Overestimation in Off-Policy Reinforcement Learning

# Collaborative Weighting with Pessimistic Critic for Mitigating Overestimation in Off-Policy Reinforcement Learning
Source: [https://arxiv.org/html/2607.26509](https://arxiv.org/html/2607.26509)
###### Abstract

Deep off\-policy reinforcement learning algorithms for continuous control typically rely on neural value function approximation to guide policy improvement\. However, temporal\-difference \(TD\) learning introduces noisy targets, resulting in non\-stationary optimization, while greedy policy updates amplify early\-stage estimation errors\. The recursive propagation of such errors leads to persistent overestimation bias and degraded training stability in actor\-critic methods\. Existing approaches attempt to alleviate this issue via prioritized sampling or modified value learning objectives, but often overemphasize high\-uncertainty transitions caused by limited data coverage or bootstrapping errors, thereby further amplifying bias\. In this paper, we propose Collaborative Weighting Actor\-Critic \(CWAC\), a unified framework that explicitly accounts for predictive uncertainty in value estimation\. CWAC employs distributional critic to model return uncertainty and introduces a collaborative weighting mechanism that jointly reweights TD\-errors and uncertainty, enabling robust learning from reliable samples while suppressing noisy updates\. In addition, we incorporate a stochastic pessimistic value estimation scheme via sampling from the return distribution, which effectively mitigates error propagation during policy improvement\. CWAC can be seamlessly integrated into existing off\-policy algorithm frameworks such as SAC, TD3, and DDPG with minimal overhead\. Empirical results demonstrate that our proposed method significantly enhances performance across a diverse range of simulated tasks\.

###### keywords:

off\-policy reinforcement learning , greedy policy updates , stochastic pessimistic value estimation , collaborative weighting mechanism

††journal:Knowledge\-Based Systems\\csdef

WGMwgm\\csdefQEqe\\csdefEPep\\csdefPMSpms\\csdefBECbec\\csdefDEde

\\affiliation

\[1\]organization=School of Computer Science and Technology, Tongji University, postcode=200092, city=Shanghai, country=China

\\affiliation

\[2\]organization=Shanghai University of Engineering Science, postcode=201620‌, city=Shanghai, country=China

## 1Introduction

Off\-policy reinforcement learning \(RL\) algorithms achieve high sample efficiency and have been widely adopted in robot control\[[36](https://arxiv.org/html/2607.26509#bib.bib10),[17](https://arxiv.org/html/2607.26509#bib.bib9)\], autonomous driving\[[29](https://arxiv.org/html/2607.26509#bib.bib8),[37](https://arxiv.org/html/2607.26509#bib.bib7)\], and industrial scheduling\[[19](https://arxiv.org/html/2607.26509#bib.bib6),[24](https://arxiv.org/html/2607.26509#bib.bib5),[14](https://arxiv.org/html/2607.26509#bib.bib4)\]\. These methods estimate long\-term discounted returns usingQQ\-functions trained from reused experience in a replay buffer\. Modern actor\-critic \(AC\) frameworks\[[12](https://arxiv.org/html/2607.26509#bib.bib88),[7](https://arxiv.org/html/2607.26509#bib.bib1)\]parameterize theseQQ\-functions with deep neural networks and optimize them via temporal\-difference \(TD\) learning\[[32](https://arxiv.org/html/2607.26509#bib.bib3)\]\.

A key challenge in off\-policy RL is the weak learning signal arising from approximate temporal\-difference targets, together with the propagation and amplification of estimation errors during bootstrapping\. As illustrated in Figure[1](https://arxiv.org/html/2607.26509#S1.F1), the greedy operatormaxa⁡Qk​\(s,a\)\\max\_\{a\}Q^\{k\}\(s,a\)inherently favors actions associated with positive estimation noise\. This selection bias amplifies early estimation errors, triggering a self\-reinforcing feedback loop that leads to systematic overestimation and undermines training stability\.

![Refer to caption](https://arxiv.org/html/2607.26509v1/x1.png)

Figure 1:Illustration of two fundamental challenges in off\-policy RL: \(a\) approximation errors in Q\-function estimation, and \(b\) error amplification induced by greedy policy improvement\. Here,QkQ^\{k\}andQk\+1Q^\{k\+1\}denote the value estimates at thekk\-th and\(k\+1\)\(k\+1\)\-th training iterations, respectively\.Gπ​\(s,a\)G^\{\\pi\}\(s,a\)represents the ground\-truth action\-value function under policyπ\\pi, defined by the standard infinite\-horizon discounted return asGπ\(s,a\)=𝔼a∼πϕ\[∑t=0∞γtℛ\(s,a\)\|s0=s,a0=a\]\.G^\{\\pi\}\(s,a\)=\\mathbb\{E\}\_\{a\\sim\\pi\_\{\\phi\}\}\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\mathcal\{R\}\(s,a\)\\;\\middle\|\\;s\_\{0\}=s,\\,a\_\{0\}=a\\right\]\.Existing methods typically tackle this issue from two perspectives: pessimistic value estimation and adaptive sample weighting\. Pessimistic approaches\[[4](https://arxiv.org/html/2607.26509#bib.bib184),[23](https://arxiv.org/html/2607.26509#bib.bib179),[34](https://arxiv.org/html/2607.26509#bib.bib120)\]mitigate overestimation by penalizing high\-uncertainty actions\. While a moderate degree of pessimism can be effective, excessive conservatism in online settings may hinder exploration, leading to suboptimal performance\[[16](https://arxiv.org/html/2607.26509#bib.bib41),[30](https://arxiv.org/html/2607.26509#bib.bib27)\]\. On the other hand, adaptive sample weighting methods emphasize transitions with large TD\-error\[[28](https://arxiv.org/html/2607.26509#bib.bib186),[8](https://arxiv.org/html/2607.26509#bib.bib11),[40](https://arxiv.org/html/2607.26509#bib.bib116)\]\. However, such signals are often entangled with uncertainty, which can amplify noisy interactions and introduce biased updates during training\.

To address these challenges, a lightweight uncertainty modeling approach based on a distributional critic is first introduced to estimate predictive uncertainty from return distributions\. Building upon this formulation, Collaborative Weighting Actor\-Critic \(CWAC\) is proposed, integrating stochastic pessimistic value estimation with an uncertainty\-aware collaborative weighting mechanism\. The proposed collaborative weighting jointly and adaptively calibrates the influence of TD\-errors and predictive uncertainty: predictive uncertainty reweights TD\-errors to suppress unreliable learning signals, while TD\-errors reciprocally refine uncertainty estimation\. In addition, adaptive pessimistic value maximization is achieved via stochastic sampling from the return distribution, reducing value overestimation and error propagation while maintaining sufficient exploration\. As a result, CWAC improves both sample efficiency and training stability in off\-policy reinforcement learning\. Our contributions can be summarized in threefold:

- 1\.We propose CWAC, which incorporates a collaborative weighting mechanism that adaptively favors optimization according to estimation error and predictive uncertainty, thereby suppressing unreliable updates and improving learning stability\.
- 2\.We develop a stochastic pessimistic value estimation scheme based on distributional critic, where policy improvement is performed via sampling from the return distribution rather than direct maximization of expected Q\-values, thereby mitigating error amplification induced by greedy updates\.
- 3\.Extensive experiments on continuous control benchmarks demonstrate that CWAC achieves competitive or superior performance compared to state\-of\-the\-art methods, while significantly improving sample efficiency and training stability\.

The remainder of this paper is organized as follows\. Section[2](https://arxiv.org/html/2607.26509#S2)introduces the background\. Related work is reviewed in Section[3](https://arxiv.org/html/2607.26509#S3)\. In Section[4](https://arxiv.org/html/2607.26509#S4), we present the proposed Collaborative Weighting Actor\-Criti \(CWAC\), including the overall framework and implementation details\. Section[5](https://arxiv.org/html/2607.26509#S5)reports experimental results on 8 challenging continuous control tasks, demonstrating the effectiveness and practicality of CWAC\. Finally, Section[6](https://arxiv.org/html/2607.26509#S6)concludes the paper and outlines potential future directions\.

## 2Preliminaries

Reinforcement Learning\.Within the standard framework of the Markov decision process \(MDP\), RL can be formulated asℳ=⟨𝒮,𝒜,𝒫,ℛ,γ⟩\\mathcal\{M\}=\\langle\\mathcal\{S\},\\mathcal\{A\},\\mathcal\{P\},\\mathcal\{R\},\\gamma\\rangle\. Here,𝒮\{\\cal S\}denotes the state space,𝒜\{\\cal A\}denotes the action space,𝒫\(⋅∣s,a\)\\mathcal\{P\}\(\\cdot\\mid s,a\)stands for transition dynamics,ℛ:𝒮×𝒜→ℝ\\mathcal\{R\}:\{\\cal S\\times A\}\\to\\mathbb\{R\}denotes the reward function, andγ∈\(0,1\]\\gamma\\in\(0,1\]is the discount factor\. Reinforcement learning aims at finding a policyπ\(⋅∣s\)\\pi\(\\cdot\\mid s\)such that the expected cumulative long\-term rewardsJπ​\(ϕ\)=𝔼s∼ℬ,a∼πϕ​\[∑t=0∞γt​ℛ​\(st,at\)\]J\_\{\\pi\}\(\\phi\)=\\mathbb\{E\}\_\{s\\sim\\mathcal\{B\},a\\sim\\pi\_\{\\phi\}\}\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\mathcal\{R\}\(s\_\{t\},a\_\{t\}\)\]are maximized\. The state and state\-action distributions induced byπ\\piare denoted byρπ​\(s\)\\rho\_\{\\pi\}\(s\)andρπ​\(s,a\)\\rho\_\{\\pi\}\(s,a\), respectively\.

The optimal policy can be obtained via a maximum entropy variant of policy iteration, which comprises two alternating steps: \(a\) policy evaluation and \(b\) policy improvement\. Given a policyπ\\pi, the corresponding Q\-function is learned via the soft Bellman operator𝒯π\\mathcal\{T\}^\{\\pi\}:

𝒯π​Q​\(s,a\)=ℛ​\(s,a\)\+γ​𝔼s′∼ρπ,a′∼πϕ​\[Qθ′​\(s′,a′\)−α​log⁡πϕ​\(a′\|s′\)\],\\mathcal\{T\}^\{\\pi\}Q\(s,a\)=\\mathcal\{R\}\(s,a\)\+\\gamma\\mathbb\{E\}\_\{s^\{\{\}^\{\\prime\}\}\\sim\\rho\_\{\\pi\},a^\{\{\}^\{\\prime\}\}\\sim\\pi\_\{\\phi\}\}\\left\[Q\_\{\\theta^\{\{\}^\{\\prime\}\}\}\(s^\{\{\}^\{\\prime\}\},a^\{\{\}^\{\\prime\}\}\)\-\\alpha\\log\\pi\_\{\\phi\}\(a^\{\{\}^\{\\prime\}\}\|s^\{\{\}^\{\\prime\}\}\)\\right\],\(1\)whereθ\\thetadenotes the parameters of the Q\-network, andϕ\\phidenotes the parameters of the Gaussian policy\. The entropy coefficientα\\alphamitigates overestimation bias by penalizing overly deterministic policies\. The parameters of the Q\-function are optimized by minimizing the soft Bellman residual, which can be formalized as

ℒQ​\(θ\)=𝔼\(s,a\)∼ℬ​\[\(Qθ​\(s,a\)−𝒯π​Q​\(s,a\)\)2\]\.\\mathcal\{L\}\_\{Q\}\(\\theta\)=\\mathbb\{E\}\_\{\(s,a\)\\sim\\mathcal\{B\}\}\\left\[\\left\(Q\_\{\\theta\}\(s,a\)\-\\mathcal\{T\}^\{\\pi\}Q\(s,a\)\\right\)^\{2\}\\right\]\.\(2\)
Weighted Q\-learning\.Weighted Q\-learning\[[39](https://arxiv.org/html/2607.26509#bib.bib47)\]introduces adaptive importance weights into temporal\-difference learning to mitigate value estimation bias\. Instead of treating all transitions equally, weighted Q\-learning assigns state\-action dependent coefficients to modulate the contribution of each sample during value updates\. This framework integrates a range of approaches, including prioritized experience replay\[[28](https://arxiv.org/html/2607.26509#bib.bib186)\]and uncertainty\-aware weighted learning methods\[[35](https://arxiv.org/html/2607.26509#bib.bib183)\]\.

Weighted Q\-learning generalizes this objective by introducing an adaptive weight functionω​\(s,a\)\\omega\(s,a\)

ℒQω​\(θ\)=𝔼\(s,a\)∼ℬ​\[ω​\(s,a\)⋅\(Qθ​\(s,a\)−𝒯π​Q​\(s,a\)\)2\],\\mathcal\{L\}\_\{Q\}^\{\\omega\}\(\\theta\)=\\mathbb\{E\}\_\{\(s,a\)\\sim\\mathcal\{B\}\}\\left\[\\omega\(s,a\)\\cdot\\left\(Q\_\{\\theta\}\(s,a\)\-\\mathcal\{T\}^\{\\pi\}Q\(s,a\)\\right\)^\{2\}\\right\],\(3\)whereω​\(s,a\)\\omega\(s,a\)modulates the contribution of each transition to the critic update\.

## 3Related Work

Overestimation Bias in Reinforcement Learning\.Overestimation bias originates from the interaction between stochastic value estimation errors and the maximization operator in TD learning\. Even in tabular settings, applying themax\\maxoperator to noisy estimates induces systematic overestimation\. While DQN\[[20](https://arxiv.org/html/2607.26509#bib.bib44)\]improves training stability via target networks, it does not eliminate this issue, which is further exacerbated under function approximation\. To address this, Double Q\-learning\[[33](https://arxiv.org/html/2607.26509#bib.bib36)\]decouples action selection from evaluation to reduce maximization bias\. Building on this idea, Maxmin DQN\[[18](https://arxiv.org/html/2607.26509#bib.bib38)\]explicitly mitigates overestimation by maintaining an ensemble of Q\-networks and using the minimum overestimates, providing a more conservative value approximation\.

In continuous control, actor–critic methods further mitigate overestimation via coordinated architectural and algorithmic refinements\. For instance, TD3\[[7](https://arxiv.org/html/2607.26509#bib.bib1)\]incorporates clipped double Q\-learning, delayed policy updates, and target policy smoothing to reduce estimation bias\. Similarly, maximum entropy methods such as SAC\[[12](https://arxiv.org/html/2607.26509#bib.bib88)\]introduce entropy regularization to soften policy evaluation, thereby improving robustness while partially alleviating overestimation\. Nevertheless, these approaches still suffer from overestimation bias\. For instance,\[[27](https://arxiv.org/html/2607.26509#bib.bib33)\]and\[[9](https://arxiv.org/html/2607.26509#bib.bib39)\]argue that off\-policy algorithms exhibit value overestimation, which hampers efficient policy exploitation, and accordingly propose to improve the decision\-making process to enhance sample efficiency\. In contrast,\[[25](https://arxiv.org/html/2607.26509#bib.bib65)\]attributes suboptimal performance of neural network–parameterized actor–critic methods to insufficiently strong value guidance, and advocates for more aggressive, greedier value evaluation to provide stronger policy improvement signals\. However, these methods fail to tackle the root cause of overestimation: the coupling between function approximation errors in parameterized Q\-networks and the error amplification introduced by greedy policy updates\.

Uncertainty\-Aware Value Estimation and Pessimism\.A prominent line of work addresses overestimation bias via uncertainty\-aware value estimation\. These approaches introduce pessimism by penalizing high\-uncertainty Q\-values or by optimizing lower confidence bounds\. For instance,\[[35](https://arxiv.org/html/2607.26509#bib.bib183),[2](https://arxiv.org/html/2607.26509#bib.bib115)\]adopt Bayesian formulations to quantify epistemic uncertainty and construct pessimistic value estimates, thereby mitigating overestimation bias\.\[[10](https://arxiv.org/html/2607.26509#bib.bib26)\]further leverages both uncertainty and familiarity to regulate value estimation, enabling adaptive adjustment across state–action pairs and effectively suppressing error amplification\.

Building on this direction,\[[11](https://arxiv.org/html/2607.26509#bib.bib42),[34](https://arxiv.org/html/2607.26509#bib.bib120)\]propose to dynamically adapt the degree of pessimism, treating it as either a learnable parameter or a constrained variable to better balance conservatism and exploration\. Moreover,\[[21](https://arxiv.org/html/2607.26509#bib.bib29)\]provide empirical evidence supporting this perspective, while\[[16](https://arxiv.org/html/2607.26509#bib.bib41),[30](https://arxiv.org/html/2607.26509#bib.bib27)\]establish theoretically that estimation bias is not universally detrimental; in certain environments, a controlled degree of overestimation can even be beneficial\. In contrast to prior approaches, the proposed CWAC algorithm introduces adaptive stochastic pessimism into value estimation, effectively mitigating overestimation while preserving exploratory capacity\.

Importance Weighting Mechanisms\.A line of work seeks to improve sample efficiency by reweighting transitions to emphasize informative updates\. Existing approaches often upweight samples using advantage\-guided schemes\[[26](https://arxiv.org/html/2607.26509#bib.bib69),[22](https://arxiv.org/html/2607.26509#bib.bib173),[5](https://arxiv.org/html/2607.26509#bib.bib123)\]or energy\-based criteria\[[38](https://arxiv.org/html/2607.26509#bib.bib66),[1](https://arxiv.org/html/2607.26509#bib.bib70)\]\. However, these approaches typically lack explicit quantification of the uncertainty associated with the guiding signals, which may result in misleading updates when the guidance is inaccurate\.

Another line of research focus on adaptive reweighting strategies that prioritize transitions deemed critical for learning\. Representative approaches include prioritized experience replay \(PER\)\[[28](https://arxiv.org/html/2607.26509#bib.bib186),[8](https://arxiv.org/html/2607.26509#bib.bib11),[13](https://arxiv.org/html/2607.26509#bib.bib35)\]\. PER\[[28](https://arxiv.org/html/2607.26509#bib.bib186)\]prioritizes samples according to the magnitude of the TD\-error, which, however, may overemphasize noisy transitions\. LAP\[[8](https://arxiv.org/html/2607.26509#bib.bib11)\]refines this strategy by adopting loss\-aware prioritization, selecting samples based on gradient magnitude\. BETDQNet\[[13](https://arxiv.org/html/2607.26509#bib.bib35)\]further constructs more reliable weighting schemes by jointly considering one\-step TD\-errors and state\-wise averaged errors\. While these methods can alleviate Q\-value overestimation to some extent, TD\-error is often correlated with estimation uncertainty, and thus, such priority\-based mechanisms may inadvertently amplify bias and induce training instability if not properly controlled\.

## 4Methods

### 4\.1Stochastic Pessimistic Value Estimation

###### Definition 1\(Stochastic Pessimistic Value Sampling\)\.

Given distributional critic to characterize the return distribution via its meanQθ​\(s,a\)Q\_\{\\theta\}\(s,a\)and standard deviationσθ​\(s,a\)\\sigma\_\{\\theta\}\(s,a\)\. Let the biased Q\-value estimator be defined as:

𝒵θ​\(s,a\)=Qθ​\(s,a\)−b​\(σθ​\(s,a\)\),\\mathcal\{Z\}\_\{\\theta\}\(s,a\)=Q\_\{\\theta\}\(s,a\)\-b\(\\sigma\_\{\\theta\}\(s,a\)\),\(4\)whereb​\(σ\)=\|ϵ\|⋅σb\(\\sigma\)=\|\\epsilon\|\\cdot\\sigmadenotes a stochastically sampled value,ϵ∼𝒩​\(0,μ⋅I\)\\epsilon\\sim\\mathcal\{N\}\(0,\\mu\\cdot\\mathrm\{I\}\)is a Gaussian noise term sampled from a zero\-mean Gaussian distribution with covarianceμ⋅I\\mu\\cdot\\mathrm\{I\}, whereμ\\mucontrolling the magnitude of stochastic perturbations\.

###### Intuition 1\.

Consider the decomposition of the action\-value estimateQθ​\(s,a\)=Gπ​\(s,a\)\+e​\(s,a\)Q\_\{\\theta\}\(s,a\)=G^\{\\pi\}\(s,a\)\+e\(s,a\), wheree​\(⋅\)e\(\\cdot\)denotes the function approximation error\. By substituting the raw Q\-valueQθQ\_\{\\theta\}with the pessimistic estimator𝒵θ=Qθ​\(s,a\)−b​\(σθ\)=Gπ​\(s,a\)\+e​\(s,a\)−b​\(σθ\)\\mathcal\{Z\}\_\{\\theta\}=Q\_\{\\theta\}\(s,a\)\-b\(\\sigma\_\{\\theta\}\)=G^\{\\pi\}\(s,a\)\+e\(s,a\)\-b\(\\sigma\_\{\\theta\}\), the residual between the predicted and ground\-truth action\-values under policyπ\\piis redefined ase~​\(s,a\)=e​\(s,a\)−b​\(σθ\)\\tilde\{e\}\(s,a\)=e\(s,a\)\-b\(\\sigma\_\{\\theta\}\)\. The resulting learning dynamics are thus governed by two competing mechanisms:

Mitigation of Error Amplification \(Appropriate Bias\)\.A properly moderate bias shifts the mean of the error distributione~\\tilde\{e\}such that𝔼​\[max⁡𝒵θ\]≈Gπ\\mathbb\{E\}\[\\max\\mathcal\{Z\}\_\{\\theta\}\]\\approx G^\{\\pi\}\. This neutralizing effect arrests the recursive amplification of approximation errors, effectively mitigating the overestimation bias\.

Suppression of Exploration \(Excessive Pessimism\)\.Let𝒟train\\mathcal\{D\}\_\{\\text\{train\}\}denote the replay buffer and𝒟test\\mathcal\{D\}\_\{\\text\{test\}\}denote the rollout distribution\. We assume the uncertainty of value estimation satisfiesσ​\(s,a\)\(s,a\)∼𝒟train≤σ​\(s,a\)\(s,a\)∼𝒟test,\\sigma\(s,a\)\_\{\(s,a\)\\sim\\mathcal\{D\}\_\{\\text\{train\}\}\}\\;\\leq\\;\\sigma\(s,a\)\_\{\(s,a\)\\sim\\mathcal\{D\}\_\{\\text\{test\}\}\},reflecting higher uncertainty in unvisited interaction\. Consider a statessand two actionsa1a\_\{1\}anda2a\_\{2\}\. The state\-action pair\(s,a1\)\(s,a\_\{1\}\)corresponds to a previously observed interaction, i\.e\.,\(s,a1\)∼𝒟train\(s,a\_\{1\}\)\\sim\\mathcal\{D\}\_\{\\text\{train\}\}, whereas\(s,a2\)\(s,a\_\{2\}\)corresponds to an unvisited interaction, i\.e\.,\(s,a2\)∼𝒟test\(s,a\_\{2\}\)\\sim\\mathcal\{D\}\_\{\\text\{test\}\}\. Assume both actions have comparable true value:Gπ​\(s,a1\)<Gπ​\(s,a2\),G^\{\\pi\}\(s,a\_\{1\}\)<G^\{\\pi\}\(s,a\_\{2\}\),so thata2a\_\{2\}is the truly better action\.

Under pessimistic estimation, we introduce a pessimistically adjusted value𝒵\\mathcal\{Z\}by penalizing the Q\-value with uncertainty\. The difference between the pessimistic values is given by𝒵​\(s,a2\)−𝒵​\(s,a1\)=e~​\(s,a2\)−e~​\(s,a1\)\.\\mathcal\{Z\}\(s,a\_\{2\}\)\-\\mathcal\{Z\}\(s,a\_\{1\}\)=\\tilde\{e\}\(s,a\_\{2\}\)\-\\tilde\{e\}\(s,a\_\{1\}\)\.Consider a simple deterministic MDP with initial statessand two actionsa1a\_\{1\}anda2a\_\{2\}\. The true returns satisfyGπ​\(s,a1\)=1G^\{\\pi\}\(s,a\_\{1\}\)=1andGπ​\(s,a2\)=2G^\{\\pi\}\(s,a\_\{2\}\)=2\. Suppose the approximation errors aree​\(s,a1\)=0\.01e\(s,a\_\{1\}\)=0\.01ande​\(s,a2\)=0\.1e\(s,a\_\{2\}\)=0\.1, with predictive uncertaintiesb​\(σ​\(s,a1\)\)=0\.01b\(\\sigma\(s,a\_\{1\}\)\)=0\.01andb​\(σ​\(s,a2\)\)=0\.5b\(\\sigma\(s,a\_\{2\}\)\)=0\.5\. Under pessimistic adjustment, the effective errors becomee~​\(s,a1\)=0\\tilde\{e\}\(s,a\_\{1\}\)=0ande~​\(s,a2\)=−0\.4\\tilde\{e\}\(s,a\_\{2\}\)=\-0\.4\. Consequently,𝒵​\(s,a1\)\>𝒵​\(s,a2\),\\mathcal\{Z\}\(s,a\_\{1\}\)\>\\mathcal\{Z\}\(s,a\_\{2\}\),indicating that excessive pessimism can reverse the true action preference and suppress the exploratory action\.

### 4\.2Collaborative Weighting Actor\-Critic

###### Theorem 1\(Weighted Q\-learning is not enough\)\.

Consider the weighted TD\-error objectiveℒω=𝔼​\[ω​\(s,a\)⋅δ​\(s,a\)2\]\\mathcal\{L\}^\{\\omega\}=\\mathbb\{E\}\[\\omega\(s,a\)\\cdot\\delta\(s,a\)^\{2\}\], where the weighting functionω​\(s,a\)∝\|δ​\(s,a\)\|p\\omega\(s,a\)\\propto\|\\delta\(s,a\)\|^\{p\}\(e\.g\., Prioritized Experience Replay\[[28](https://arxiv.org/html/2607.26509#bib.bib186)\]\)\. Suppose the soft Bellman target𝒯π​Q\\mathcal\{T\}^\{\\pi\}Qis corrupted by heteroscedastic noiseη​\(s,a\)\\eta\(s,a\)with𝔼​\[η\]\>0\\mathbb\{E\}\[\\eta\]\>0, the prioritization weightω​\(s,a\)\\omega\(s,a\)induces a biased gradient update that increases with the noise magnitude\. Consequently, high\-variance state\-action regions are systematically overemphasized, amplifying overestimation bias and potentially leading to unstable or divergent value estimates\.

Proof sketch:Consider the observed TD\-errorδ=δ⋆\+η\\delta=\\delta^\{\\star\}\+\\eta, whereδ⋆\\delta^\{\\star\}is the true residual andη\\etais the approximation noise\. The weighted gradient is given by∇θℒω≈𝔼​\[\|δ⋆\+η\|p⋅\(δ⋆\+η\)⋅∇θQθ\]\\nabla\_\{\\theta\}\\mathcal\{L\}^\{\\omega\}\\approx\\mathbb\{E\}\[\|\\delta^\{\\star\}\+\\eta\|^\{p\}\\cdot\(\\delta^\{\\star\}\+\\eta\)\\cdot\\nabla\_\{\\theta\}Q\_\{\\theta\}\]\. Due to the positive correlation between the weight\|δ⋆\+η\|p\|\\delta^\{\\star\}\+\\eta\|^\{p\}and the noiseη\\eta, the expected update is skewed toward samples whereη\\etais large and positive\. The exponentppcontrols the degree of this weighting effect\. In RL, these high\-residual samples are often located in unvisited or high\-uncertainty regions\. Consequently, the optimizer disproportionately updates the critic toward these noisy targets, effectively transforming stochastic noise into systemic overestimation\.

Why is this a problem?The fundamental flaw of heuristic weighting based purely on\|δ\|\|\\delta\|is its inability to distinguish between learning progress and stochastic noise\. In the nascent stages of training or in complex environments with high epistemic uncertainty, large TD\-errors are frequently artifacts of function approximation limits or aleatoric noise rather than meaningful Bellman residuals\. By reinforcing these “hard” samples, the agent triggers a positive feedback loop: overestimation leads to higher prioritization, which in turn leads to further overestimation\. This instability prevents the critic from grounding the value surface, ultimately degrading policy optimization\.

###### Definition 2\(Collaborative Weighting Value Learning\)\.

Consider the gradient of the collaborative loss functionℒ𝒵ω,ξ\\mathcal\{L\}\_\{\\mathcal\{Z\}\}^\{\\omega,\\xi\}with respect to the parametersθ\\theta:

ℒ𝒵ω,ξ​\(θ\)=𝔼\(s,a\)∼ℬ,s′∼ρπ,a′∼πϕ​\[ω​\(s,a\)⋅Huber​\(Qθ​\(s,a\)−y\)\+ξ​\(s,a\)⋅σ​\(s,a\)\],\\mathcal\{L\}\_\{\\mathcal\{Z\}\}^\{\\omega,\\xi\}\(\\theta\)=\\mathbb\{E\}\_\{\(s,a\)\\sim\\mathcal\{B\},s^\{\{\}^\{\\prime\}\}\\sim\\rho\_\{\\pi\},a^\{\{\}^\{\\prime\}\}\\sim\\pi\_\{\\phi\}\}\\left\[\\omega\(s,a\)\\cdot\\,\\mathrm\{Huber\}\\left\(Q\_\{\\theta\}\(s,a\)\-y\\ \\right\)\+\\xi\(s,a\)\\cdot\\sigma\(s,a\)\\right\],\(5\)whereyydenotes the bootstrapped learning target\. To improve robustness against noisy value estimates, the critic is optimized using the Huber loss\[[15](https://arxiv.org/html/2607.26509#bib.bib31)\]rather than the standard MSE objective\. The target value functionQθ′Q\_\{\\theta^\{\\prime\}\}is further replaced by its stochastic pessimistic counterpart𝒵θ′\\mathcal\{Z\}\_\{\\theta^\{\\prime\}\}:

y=r\+γ\[mini=1,2Zθi′\(s′,a′\)−αlogπϕ\(a′∣s′\)\],a′∼πϕ\(⋅∣s′\)\.y=r\+\\gamma\\left\[\\min\_\{i=1,2\}Z\_\{\\theta^\{\{\}^\{\\prime\}\}\_\{i\}\}\(s^\{\\prime\},a^\{\\prime\}\)\-\\alpha\\log\\pi\_\{\\phi\}\(a^\{\\prime\}\\mid s^\{\\prime\}\)\\right\],a^\{\\prime\}\\sim\\pi\_\{\\phi\}\(\\cdot\\mid s^\{\\prime\}\)\.\(6\)
We define the TD\-error asδ​\(s,a\)=Qθ​\(s,a\)−y\\delta\(s,a\)=Q\_\{\\theta\}\(s,a\)\-y, based on which we further construct uncertainty\-aware and TD\-error\-driven weighting functions

ω​\(s,a\)=\(𝔼​\[σθ​\(s,a\)\]σθ​\(s,a\)\+c\)βω,ξ​\(s,a\)=\(𝔼​\[∣δ​\(s,a\)∣\]∣δ​\(s,a\)∣\+c\)βξ,\\omega\(s,a\)=\\left\(\\frac\{\\mathbb\{E\}\[\\sigma\_\{\\theta\}\(s,a\)\]\}\{\\sigma\_\{\\theta\}\(s,a\)\+c\}\\right\)^\{\\beta\_\{\\omega\}\},\\quad\\xi\(s,a\)=\\left\(\\frac\{\\mathbb\{E\}\[\\mid\\delta\(s,a\)\\mid\]\}\{\\mid\\delta\(s,a\)\\mid\+c\}\\right\)^\{\\beta\_\{\\xi\}\},\(7\)whereβω\\beta\_\{\\omega\}andβξ\\beta\_\{\\xi\}control the sharpness of uncertainty and TD\-error reweighting, respectively, andccis a small constant for numerical stability\.

###### Intuition 2\.

The corresponding gradient is given by∇θℒ𝒵ω,ξ=𝔼s∼ℬ,a∼πϕ\[2ω⋅δ⋅∇θQθ\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\mathcal\{Z\}\}^\{\\omega,\\xi\}=\\mathbb\{E\}\_\{s\\sim\\mathcal\{B\},a\\sim\\pi\_\{\\phi\}\}\[2\\omega\\cdot\\delta\\cdot\\nabla\_\{\\theta\}Q\_\{\\theta\}\+ξ⋅∇θσθ\]\+\\xi\\cdot\\nabla\_\{\\theta\}\\sigma\_\{\\theta\}\]\. The resulting optimization dynamics are adaptively governed by reciprocal interactions between the weighting terms and the learning targets, establishing a critical equilibrium between overestimation suppression and efficient value learning\.

Uncertainty\-Aware Value Regularization\.In high\-noise or unvisited regions where epistemic uncertaintyσθ↑\\sigma\_\{\\theta\}\\uparrow, the weighting coefficientω↓\\omega\\downarrow\. This mechanism introduces a principled adaptive pessimism: it suppresses the recursive amplification of overestimation bias by attenuating the influence of unreliable, high\-variance samples on the value surfaceQθQ\_\{\\theta\}\.

Error\-Conditioned Uncertainty Optimization\.When the model encounters significant Bellman residuals\|δ\|↑\|\\delta\|\\uparrow, the weightξ↓\\xi\\downarrow\. This serves as a vital safeguard against excessive pessimism: by preventing the premature minimization ofσθ\\sigma\_\{\\theta\}\(i\.e\., avoiding over\-confident uncertainty collapse\), the agent maintains a sufficiently diverse uncertainty manifold\. This ensures that exploratory gradients are not prematurely extinguished, allowing the agent to continue gathering informative transitions until the value estimateQθQ\_\{\\theta\}is sufficiently grounded\.

Gradient Synthesis and Stability\.As learning converges \(\|δ\|↓,σθ↓\|\\delta\|\\downarrow,\\sigma\_\{\\theta\}\\downarrow\), the weights reciprocally increase \(ξ↑,ω↑\\xi\\uparrow,\\omega\\uparrow\), facilitating fine\-grained refinement of the value surface\. This dual\-track regulation ensures that the total gradient norm remains bounded and is preferentially oriented toward state\-action regions with high statistical confidence, achieving a robust trade\-off between conservative value estimation and active exploration\.

###### Definition 3\(Mitigating Maximization Bias in Actor Updates\)\.

Based on the stochastic pessimistic sampling in Eq\. \([4](https://arxiv.org/html/2607.26509#S4.E4)\), we further incorporate pessimism into policy improvement\. The actor’s objective is optimized by maximizing

Jπ​\(ϕ\)=𝔼s∼ℬ,a∼πϕ​\[𝒵θ​\(s,a\)−α​log⁡πϕ​\(a\|s\)\]\.\\displaystyle J\_\{\\pi\}\(\\phi\)=\\mathbb\{E\}\_\{s\\sim\\mathcal\{B\},a\\sim\\pi\_\{\\phi\}\}\\Big\[\\mathcal\{Z\}\_\{\\theta\}\(s,a\)\-\\alpha\\log\\pi\_\{\\phi\}\(a\|s\)\\Big\]\.\(8\)

Initialize critic networks

Qθ1Q\_\{\\theta\_\{1\}\},

Qθ2Q\_\{\\theta\_\{2\}\}and actor\-network

πϕ\\pi\_\{\\phi\}with random parameters

Initialize target networks

θ1′←θ1\\theta^\{\{\}^\{\\prime\}\}\_\{1\}\\leftarrow\\theta\_\{1\},

θ2′←θ2\\theta^\{\{\}^\{\\prime\}\}\_\{2\}\\leftarrow\\theta\_\{2\}and replay buffer

ℬ\\mathcal\{B\}
for

t=1t=1to

TTdo

Calculate action

a∼πϕ\(⋅\|s\)a\\sim\\pi\_\{\\phi\}\(\\cdot\|s\)
Get reward

rrand new state

s′s^\{\{\}^\{\\prime\}\}and store transition tuple

\(s,a,r,s′\)\(s,a,r,s^\{\{\}^\{\\prime\}\}\)in

ℬ\\mathcal\{B\}
Sample mini\-batch of transitions

\(s,a,r,s′\)∼ℬ\(s,a,r,s^\{\{\}^\{\\prime\}\}\)\\sim\\mathcal\{B\}
Get target value:

y=r\+γ\[mini=1,2Zθi′\(s′,a′\)−αlogπϕ\(a′∣s′\)\],a′∼πϕ\(⋅∣s′\)y=r\+\\gamma\\left\[\\min\_\{i=1,2\}Z\_\{\\theta^\{\{\}^\{\\prime\}\}\_\{i\}\}\(s^\{\\prime\},a^\{\\prime\}\)\-\\alpha\\log\\pi\_\{\\phi\}\(a^\{\\prime\}\\mid s^\{\\prime\}\)\\right\],a^\{\\prime\}\\sim\\pi\_\{\\phi\}\(\\cdot\\mid s^\{\\prime\}\)
Update critic with gradient descent by minimizing Eq\. \([5](https://arxiv.org/html/2607.26509#S4.E5)\)

Stochastic pessimistic value estimation via Eq\. \([4](https://arxiv.org/html/2607.26509#S4.E4)\)

Update actor with gradient ascent by maximizing Eq\. \([8](https://arxiv.org/html/2607.26509#S4.E8)\)

Update weights:

θi′←τ​θi\+\(1−τ\)​θi′,i∈1,2\\theta^\{\{\}^\{\\prime\}\}\_\{i\}\\leftarrow\\tau\\theta\_\{i\}\+\(1\-\\tau\)\\theta^\{\{\}^\{\\prime\}\}\_\{i\},i\\in 1,2
endfor

Algorithm 1Collaborative Weighting Actor\-Critic \(CWAC\)We present the detailed learning procedure of the proposed CWAC in Algorithm[1](https://arxiv.org/html/2607.26509#alg1)\. In contrast to\[[8](https://arxiv.org/html/2607.26509#bib.bib11)\], our method does not rely on prioritized sampling over historical interactions\. Instead of modifying the sampling distribution, it incorporates a pessimistic value sampling strategy together with a collaborative weighting mechanism that jointly and adaptively calibrates the influence of TD\-errors and predictive uncertainty during critic optimization\. Compared to the vanilla SAC algorithm, these modifications provide a systematic approach to mitigating overestimation bias in off\-policy RL\.

## 5Experiments

Our experiments are designed to address the following questions: \(1\) Does CWAC achieve consistent and stable performance improvements across diverse environments? \(2\) How does the collaborative weighting mechanism facilitate Bellman error minimization and stabilize value learning? \(3\) Can CWAC stabilize value estimation under non\-stationary reward perturbations?

To answer \(1\), we evaluate CWAC on 6 continuous control tasks from OpenAI Gym\[[3](https://arxiv.org/html/2607.26509#bib.bib23)\], 4 tasks from PyBullet\[[6](https://arxiv.org/html/2607.26509#bib.bib2)\]and 12 tasks from DeepMind Control \(DMC\)\[[31](https://arxiv.org/html/2607.26509#bib.bib22)\]using a fixed set of hyperparameters\. For \(3\), we evaluate performance and TD\-error dynamics in environments with stochastic reward noise\.

### 5\.1Experimental Setup

Hyperparameters\.In Section[4](https://arxiv.org/html/2607.26509#S4), we introduce three key hyperparameters: the pessimism coefficientμ=0\.8\\mu=0\.8, the uncertainty weighting exponentβω=2\\beta\_\{\\omega\}=2, and the TD\-error reweighting exponentβξ=1\\beta\_\{\\xi\}=1\. Unless otherwise specified, these hyperparameters are fixed across all experiments\. Our model architecture follows the standard design of SAC\. Further details can be found in Table[5](https://arxiv.org/html/2607.26509#S5.T5)\.

Baselines\.We compare CWAC against several representative actor\-critic baselines, including value\-improved actor\-critic method VIAC\[[25](https://arxiv.org/html/2607.26509#bib.bib65)\], the decision augmentation approach ALH\[[27](https://arxiv.org/html/2607.26509#bib.bib33)\], the prioritized sampling method LAP\[[8](https://arxiv.org/html/2607.26509#bib.bib11)\], the maximum\-entropy actor\-critic algorithm SAC\[[12](https://arxiv.org/html/2607.26509#bib.bib88)\], and the deterministic policy gradient method TD3\[[7](https://arxiv.org/html/2607.26509#bib.bib1)\]\.

### 5\.2Evaluation Environments

We evaluate the effectiveness of the proposed algorithm on 6 Gym environments, 4 PyBullet environments, and 12 DMC benchmarks\. Detailed descriptions of these experimental environments are provided below\.

Table 1:State and action space dimensions of the continuous control environments used in experiments: Gym tasks \(a\), PyBullet tasks \(b\), and DMC tasks \(c\)\.\(a\)
\(b\)
\(c\)

Gym\.Gym is a fast and powerful physics engine that has become a standard platform for simulating complex robotic dynamics and control tasks\. It is extensively utilized in the robotics and RL communities for training agents to perform tasks such as locomotion and manipulation, and it offers a diverse set of benchmarks for evaluating RL algorithms\. In our experiments, we adopt 6 challenging tasks from the Gym environments: HalfCheetah, Ant, Hopper, Walker2d, Humanoid, and BipedalWalker\.

PyBullet\.PyBullet is an open\-source physics engine that supports general\-purpose physical simulation as well as high\-fidelity modeling for robotics tasks\. It provides an efficient control interface for fast simulation of rigid\-body dynamics, collision detection, and various control problems\. Compared to MuJoCo, PyBullet typically exhibits higher levels of environment\-induced dynamical noise, making it a more challenging testbed for evaluating the robustness of reinforcement learning algorithms\. In this work, we evaluate the proposed algorithm on four PyBullet benchmark tasks: HalfCheetahBulletEnv, AntBulletEnv, HopperBulletEnv, and Walker2DBulletEnv\.

DMC\.The DeepMind Control \(DMC\) suite is a set of continuous control environments mainly designed for robotic manipulation tasks, but it also covers a wide range of industrial control challenges\. In our experiments, we evaluated the proposed algorithm on four continuous control tasks from the DMC suite: reacher\-hard, walker\-walk, hopper\-hop, hopper\-stand, fish\-swim, swimmer\-swimmer6, pendulum\-swingup, cheetah\-run, walker\-run, quadruped\-walk, quadruped\-run, and finger\-turnhard task\.

### 5\.3Performance Evaluation on Gym Environments

We evaluate CWAC against several representative baselines on 6 continuous control tasks from the Gym benchmark suite\. Learning curves are shown in Figure[2](https://arxiv.org/html/2607.26509#S5.F2), with final performance summarized in Table[2](https://arxiv.org/html/2607.26509#S5.T2)\.

Overall, CWAC demonstrates consistent and superior performance over VIAC, ALH, and LAP across all 6 Gym tasks\. On the Ant task, VIAC exhibits a clear performance degradation after 1\.5M training steps, which we attribute to its aggressive, greedy value improvement scheme that tends to amplify noise\-dominated updates and induce non\-stationary training dynamics\. The ALH method constructs actions by combining representations from the previous and current timesteps, which may lead to temporally correlated actions and thus limit performance gains\. LAP yields only marginal improvements over SAC and even shows performance degradation on Ant and BipedalWalker, suggesting that TD\-error\-based prioritized sampling may overemphasize noisy transitions, potentially resulting in suboptimal performance\. In contrast, CWAC systematically mitigates overestimation bias through uncertainty\-aware weighting and pessimistic value estimation, resulting in more stable and efficient performance improvements\.

As shown in Table[2](https://arxiv.org/html/2607.26509#S5.T2), CWAC achieves the best overall performance among all benchmark methods, including VIAC, ALH, LAP, SAC, and TD3, with relative improvements of 15\.3%, 31\.5%, 29\.5%, 40\.3%, and 32\.9%, respectively, in average evaluation metrics\. Moreover, CWAC demonstrates consistent performance gains across all 6 Gym tasks, indicating that the proposed collaborative weighting mechanism enables more stable Q\-value estimation and provides more reliable guidance for policy improvement\.

![Refer to caption](https://arxiv.org/html/2607.26509v1/x2.png)

Figure 2:Learning curves on 6 Gym tasks\. The shaded region represents half a standard deviation of the average evaluation over 10 trials\. Curves are smoothed uniformly for visual clarity\.Table 2:Average return of the last 10 evaluation scores over 10 random seeds\. The maximum values of each row are bolded\.±\\pmcorresponds to a standard deviation over trials\.
### 5\.4Performance on PyBullet Environments

We evaluate CWAC and all baseline algorithms on the PyBullet benchmark suite\. Each algorithm is independently trained with 10 random seeds for 2 million environment interaction steps\. The learning curves are shown in Figure[3](https://arxiv.org/html/2607.26509#S5.F3), and the final performance results are reported in Table[3](https://arxiv.org/html/2607.26509#S5.T3)\.

![Refer to caption](https://arxiv.org/html/2607.26509v1/x3.png)

Figure 3:Learning curves on four PyBullet tasks\. The shaded areas represent half of the standard deviation of the average evaluation returns over 10 independent trials\. For better visualization, all curves are uniformly smoothed\.As shown in Figure[3](https://arxiv.org/html/2607.26509#S5.F3), CWAC achieves faster policy convergence and superior performance compared with all baseline methods across the four tasks\. Specifically, LAP obtains only marginal improvements on HalfCheetahBulletEnv, AntBulletEnv, and Walker2DBulletEnv, indicating that prioritized experience replay improves sample utilization efficiency but provides limited benefits for policy improvement\. VIAC achieves higher returns on several tasks by adopting a more aggressive value estimation and value\-guided optimization mechanism\. However, such aggressive value updates may exacerbate value overestimation bias, thereby compromising training stability\. In contrast, CWAC consistently demonstrates stable and significant performance gains across all four tasks\. These results suggest that the uncertainty\-aware value weighting mechanism effectively mitigates the misleading effects of overestimated states during policy optimization\. By leveraging valuable value information while reducing the adverse impact of value error accumulation and propagation, CWAC enables a more robust policy learning process\. Furthermore, as an extension of TD3, ALH underperforms the standard TD3 across all four PyBullet tasks\. We hypothesize that this degradation may be attributed to introducing the implicit representation of the previous observation as an additional input to the actor network\. Such a design encourages the policy to generate more similar actions for adjacent states, thereby reducing behavioral diversity and exploration capability during policy optimization, ultimately limiting further performance improvement\.

Table 3:Average evaluation returns over the final 10 evaluation episodes on PyBullet environments across 10 random seeds\. The maximum values of each row are bolded\.±\\pmcorresponds to a standard deviation over trials\.From Table[3](https://arxiv.org/html/2607.26509#S5.T3), we observe that SAC\-based CWAC, VIAC, and LAP consistently outperform the original SAC on all four PyBullet continuous control tasks\. Specifically, CWAC, VIAC, and LAP improve the average return over SAC by 37\.6%, 26\.6%, and 27\.5%, respectively\. These results demonstrate that incorporating value information into policy optimization can effectively improve both learning efficiency and final performance\.

### 5\.5Performance Evaluation on DMC Environments

We conduct experiments on the DMC benchmark suite, where each algorithm is trained for 500K environment steps with 10 random seeds\. The aggregated learning curves are presented in Figure[4](https://arxiv.org/html/2607.26509#S5.F4), and the final performance comparisons are summarized in Table[4](https://arxiv.org/html/2607.26509#S5.T4)\.

Across the 12 evaluated tasks, CWAC achieves the SOTA performance on 8 tasks, highlighting its robustness and effectiveness in diverse continuous control scenarios\. In particular, CWAC built upon SAC consistently demonstrates superior and more stable performance compared with the vanilla SAC\. These improvements suggest that the proposed collaborative weighting and pessimistic value estimation mechanisms effectively alleviate value estimation errors, leading to improved sample efficiency and more reliable policy optimization\.

![Refer to caption](https://arxiv.org/html/2607.26509v1/x4.png)

Figure 4:Learning curves on 12 DMC tasks\. The shaded region represents half a standard deviation of the average evaluation over 10 trials\. Curves are smoothed uniformly for visual clarity\.Table 4:Average evaluation returns over the final 10 evaluation episodes on DMC environments across 10 random seeds\. The maximum values of each row are bolded\.±\\pmcorresponds to a standard deviation over trials\.
### 5\.6Integration with More Off\-Policy Algorithms

To further validate the generality of the proposed CWAC algorithm, we integrate CWAC with two representative off\-policy value\-based algorithms, namely TD3 and DDPG, and conduct corresponding experimental studies\. The experiments are carried out on six Gym continuous control environments, with each environment independently run for 10 trials\. All algorithms adopt a unified hyperparameter configuration and identical experimental conditions to ensure a fair comparison\. Figure[5](https://arxiv.org/html/2607.26509#S5.F5)presents the learning curve comparisons among CWAC\(TD3\), CWAC\(DDPG\), and the original TD3 and DDPG algorithms\.

![Refer to caption](https://arxiv.org/html/2607.26509v1/x5.png)

Figure 5:Learning curves of CWAC, TD3, and DDPG on six Gym continuous control tasks\. Shaded regions denote half a standard deviation of the average evaluation over 10 trials\. All curves are uniformly smoothed for visual clarity\.As shown in Figure[5](https://arxiv.org/html/2607.26509#S5.F5), integrating CWAC with TD3 consistently improves learning performance across most evaluated tasks\. Compared with the original TD3, CWAC\(TD3\) achieves significant performance gains on three out of six benchmark environments, while maintaining comparable performance on the remaining tasks without introducing performance degradation\. Specifically, on the HalfCheetah, Walker2d, and Humanoid environments, CWAC\(TD3\) demonstrates faster convergence and achieves higher final cumulative returns\. Moreover, CWAC\(DDPG\) also yields substantial improvements on these three tasks, further validating the effectiveness and generalizability of the proposed method across different value\-based off\-policy algorithms\. Interestingly, we observe that vanilla DDPG fails to achieve stable convergence on the BipedalWalker task, whereas CWAC\(DDPG\) exhibits consistent performance improvements throughout training\. This result suggests that the proposed method can effectively alleviate unstable optimization issues in value\-based policy learning\. Overall, these empirical results demonstrate that the collaborative weighting mechanism introduced in CWAC effectively improves the utilization efficiency of high\-value experience samples in off\-policy reinforcement learning\. Through the adaptive coordination of TD\-errors and predictive uncertainty during optimization, CWAC effectively balances the influence of different training samples, leading to more stable policy learning, higher sample efficiency, and improved final policy performance\. These findings indicate that CWAC is not limited to maximum\-entropy reinforcement learning frameworks, but can also serve as a general enhancement module for a broader class of policy gradient algorithms\.

### 5\.7Sensitivity Analysis of Core Hyperparameters

CWAC introduces three hyperparameters to control the strength of stochastic samplingμ\\mu, the uncertainty weighting coefficientβω\\beta\_\{\\omega\}, and the temporal\-difference error weighting coefficientβξ\\beta\_\{\\xi\}, respectively\. To evaluate the sensitivity of these hyperparameters, we conduct experiments on the Walker2d task, where each configuration is trained for 2 million time steps using 10 different random seeds, and analyze the resulting training dynamics under varying parameter settings\.

![Refer to caption](https://arxiv.org/html/2607.26509v1/x6.png)

Figure 6:Hyperparameter sensitivity analysis\. The shaded region represents half a standard deviation of the average evaluation over 10 trials\. Curves are smoothed uniformly for visual clarity\.Pessimistic value coefficientμ\\mu\.The parameterμ\\mucontrols the strength of stochastic pessimism, as described in Section[4](https://arxiv.org/html/2607.26509#S4)\. In this study, we setβω=1\\beta\_\{\\omega\}=1andβξ=2\\beta\_\{\\xi\}=2as the default setting\. Whenμ=0\\mu=0, the stochastic pessimistic sampling mechanism is disabled\. Whenμ\\mulies within the interval\[0\.6,1\.2\]\[0\.6,1\.2\], CWAC demonstrates consistently stable performance improvements across different choices ofμ\\mu, as shown in Figure[6](https://arxiv.org/html/2607.26509#S5.F6)\(left\)\. However, overly large values ofμ\\mumay amplify estimation bias and introduce training instability, as observed for CWAC withμ=1\.4\\mu=1\.4\.

Uncertainty coefficientβω\\beta\_\{\\omega\}\.The coefficientβω\\beta\_\{\\omega\}controls the weighting of TD\-error according to the uncertainty in the Q\-value estimates, directly modulating each sample’s contribution\. Whenβω=0\\beta\_\{\\omega\}=0, no uncertainty weighting is applied, resulting in a notable performance drop compared to settings withβω∈\[0\.5,2\]\\beta\_\{\\omega\}\\in\[0\.5,2\], as shown in Figure[6](https://arxiv.org/html/2607.26509#S5.F6)\(center\)\. Asβω\\beta\_\{\\omega\}increases, TD\-errors associated with high uncertainty are increasingly down\-weighted, while those with low uncertainty are correspondingly up\-weighted, as described in Section[4](https://arxiv.org/html/2607.26509#S4)\. However, further increasingβω\\beta\_\{\\omega\}\(e\.g\., to 4\) amplifies the overall fluctuations of TD\-errors, which in turn leads to unstable performance\.

TD\-error coefficientβξ\\beta\_\{\\xi\}\.Similarly, as shown in Figure[6](https://arxiv.org/html/2607.26509#S5.F6)\(right\), the weighting of samples with high TD\-error is suppressed, while that of samples with low TD\-error is increased\. Asβξ\\beta\_\{\\xi\}increases, the variability of sample\-wise error uncertainty is amplified, which can adversely affect training stability and lead to degraded performance\.

### 5\.8Value Estimation Error Visualization

To further investigate the training stability of the proposed method, we visualize the Q\-value estimation errorQ​\(s,a\)−Gπ​\(s,a\)Q\(s,a\)\-G^\{\\pi\}\(s,a\)over 2M training steps across 6 continuous control tasks\. The empirical results are illustrated in Figure[7](https://arxiv.org/html/2607.26509#S5.F7)\.

![Refer to caption](https://arxiv.org/html/2607.26509v1/x7.png)

Figure 7:Value estimation error on 6 Gym tasks\. The shaded region represents half a standard deviation of the average evaluation over 10 trials\. Curves are smoothed uniformly for visual clarity\. The black dashed line indicates the training target of the value function\.Overall, CWAC exhibits consistently smoother and more stable TD\-error convergence across all benchmarks, indicating improved stability in critic optimization\. In contrast, conventional off\-policy baselines such as SAC and TD3 show elevated TD\-error during the early stages of training, accompanied by pronounced stochastic oscillations, which suggest unstable value estimation\. LAP reduces TD\-error relative to SAC, particularly in high\-dimensional tasks such as HalfCheetah, Walker2d, and Humanoid\. However, its prioritized sampling strategy may overemphasize noisy transitions, potentially amplifying aleatoric noise and inducing distributional shift, which can introduce estimation bias and destabilize policy updates\. We observe that ALH exhibits pronounced value underestimation on the Hopper and BipedalWalker tasks compared to TD3\. This phenomenon can be attributed to its design, as ALH introduces a decision augmentation mechanism with hypothesis modeling, building upon TD3’s value smoothing to further increase conservatism in value estimation\. While this design improves the stability of policy updates, it reduces adaptability to new data, thereby inducing systematic underestimation in value predictions\. For VIAC, we observe pronounced TD\-error fluctuations on the Ant task, which correlate with its performance degradation\. This instability likely stems from its aggressive value improvement operator, potentially compromising the consistency of the learning dynamics\. In contrast, CWAC maintains low\-variance and well\-behaved TD\-error trajectories throughout training, effectively balancing conservative estimation with learning efficiency\.

### 5\.9Value Learning in Noisy Reward Environments

To investigate the robustness of CWAC under non\-stationary reward perturbations, we compare its value learning dynamics against the vanilla SAC algorithm on the HalfCheetah, Ant, and Hopper tasks\. Specifically, we introduce stochastic perturbations to the reward signal during training by injecting Gaussian noise with a positive mean \(ϵr∼𝒩​\(1,0\.2\)\\epsilon\_\{r\}\\sim\\mathcal\{N\}\(1,0\.2\)\) into the immediate rewardrr\. The noise is applied with a frequency of 2%, i\.e\., once every 50 environment interactions\. The resulting transitions are stored in the replay bufferℬ\\mathcal\{B\}\. Validation performance and estimation error are compared under noisy and noise\-free rewards, with the complete learning dynamics depicted in Figure[8](https://arxiv.org/html/2607.26509#S5.F8)\.

![Refer to caption](https://arxiv.org/html/2607.26509v1/x8.png)

Figure 8:Comparison of evaluation performance and estimation error in noisy\-reward and noise\-free environments\. The shaded region represents half a standard deviation of the average evaluation over 10 trials\. Curves are smoothed uniformly for visual clarity\. The black dashed line indicates the training target of the value function\.From the results, we observe that CWAC \(w/ noise\) exhibits a significant increase in TD\-error across all three tasks, while its performance remains comparable to the noise\-free CWAC variant\. In contrast, SAC \(w/ noise\) shows degraded performance on the HalfCheetah and Ant tasks compared to its noise\-free counterpart, suggesting that non\-uniform reward noise induces non\-stationarity in the value signal, which leads to suboptimal policy performance\. Interestingly, on the Hopper task, SAC \(w/ noise\) outperforms its noise\-free baseline\. This suggests that overestimation in value estimation may encourage exploration, leading to improved performance\. Furthermore, we observe that CWAC \(w/ noise\) exhibits smoother TD\-error trajectories than SAC \(w/ noise\), demonstrating robustness to noisy reward signals\. In contrast, SAC \(w/ noise\) shows pronounced estimation error oscillations during the early stages, particularly on the HalfCheetah and Ant tasks, indicating that the injected noise introduces non\-stationarity into value estimates, which slows policy convergence\.

Table 5:A complete comparison of hyper\-parameter choices between CWAC and 5 baselines, including VIAC, ALH, LAP, SAC, and TD3\.Hyper\-parameterCWACVIACALHLAPSACTD3OptimizerAdamAdamAdamAdamAdamAdamCritic learning rate3⋅10−43\\cdot 10^\{\-4\}3⋅10−43\\cdot 10^\{\-4\}3⋅10−43\\cdot 10^\{\-4\}3⋅10−43\\cdot 10^\{\-4\}3⋅10−43\\cdot 10^\{\-4\}3⋅10−43\\cdot 10^\{\-4\}Actor learning rate3⋅10−43\\cdot 10^\{\-4\}3⋅10−43\\cdot 10^\{\-4\}3⋅10−43\\cdot 10^\{\-4\}3⋅10−43\\cdot 10^\{\-4\}3⋅10−43\\cdot 10^\{\-4\}3⋅10−43\\cdot 10^\{\-4\}Target update rateτ\\tau5⋅10−35\\cdot 10^\{\-3\}5⋅10−35\\cdot 10^\{\-3\}5⋅10−35\\cdot 10^\{\-3\}5⋅10−35\\cdot 10^\{\-3\}5⋅10−35\\cdot 10^\{\-3\}5⋅10−35\\cdot 10^\{\-3\}Batch size256256256256256256256256256256256256Discount factor0\.990\.990\.990\.990\.990\.990\.990\.990\.990\.990\.990\.99Number of critics222222Hidden dimension256256256256256256256256256256256256Start timesteps1e41e425e325e325e325e3Policy update frequencydd112112Policy noise\-\-𝒩​\(0,0\.2\)\\mathcal\{N\}\(0,0\.2\)\-\-𝒩​\(0,0\.2\)\\mathcal\{N\}\(0,0\.2\)Noise clip range\-\-\[−0\.5,0\.5\]\[\-0\.5,0\.5\]\-\-\[−0\.5,0\.5\]\[\-0\.5,0\.5\]Priority scaling factorpp\-\-\-0\.4\-\-Value functionHuberexpectileMSEHuberMSEMSEPessimistic value coefficientμ\\mu0\.8\-\-\-\-\-Uncertainty coefficientβω\\beta\_\{\\omega\}1\-\-\-\-\-TD\-error coefficientβξ\\beta\_\{\\xi\}2\-\-\-\-\-
### 5\.10Runtime Analysis

To evaluate the computational efficiency of the proposed CWAC algorithm, we conducted comprehensive experiments on the HalfCheetah\-v3 task\. The experimental environment was equipped with a 56\-core CPU and an Nvidia RTX 4090 GPU running on Linux\. Each algorithm was trained for 2 million time steps to ensure a fair comparison of total wall\-clock time\.

Table 6:Comparison of total training duration \(Wall\-clock time\) across baseline algorithms on the HalfCheetah\-v3 task\.As illustrated in Table[6](https://arxiv.org/html/2607.26509#S5.T6), CWAC introduces only a marginal increase in training duration compared to the vanilla SAC \(7%\)\. This modest overhead is primarily attributed to the gradient updates for the distributional Q\-networks and the corresponding computation of uncertainty parameters used in collaborative weighting\. These results demonstrate that CWAC attains superior stability and performance without incurring prohibitive computational overhead, making it practical for high\-dimensional control tasks\.

## 6Conclusions

We propose CWAC, a dynamic weighting framework that integrates two complementary mechanisms: stochastic pessimistic value estimation and a collaborative weighting scheme\. By jointly weighting the TD\-error and predictive uncertainty under a stochastic pessimistic value estimation formulation, CWAC produces more reliable value estimates and facilitates stable policy optimization\. Extensive experiments demonstrate that, using a single set of hyperparameters, CWAC consistently suppresses value overestimation while maintaining stable value learning across a wide range of continuous control benchmarks\. Comprehensive ablation studies further verify the effectiveness of each individual component, confirming that their synergy is essential to the observed performance improvements\.

Despite these advantages, stochastic pessimistic value estimation inevitably introduces a degree of conservatism\. Although such conservatism effectively mitigates overestimation bias, excessive pessimism may lead to suboptimal policy performance by discouraging beneficial exploration\. An important direction for future work is therefore to develop unbiased or adaptively calibrated value learning mechanisms that better balance overestimation suppression with effective exploration, thereby achieving both robust value estimation and improved policy performance\.

## 7Declarations

### 7\.1Funding

This study was supported by National Major Science and Technology Special Project \(CN\) \(Grant Number 2025ZD1604900\)\.

### 7\.2Conflicts of interest/Competing interests

The authors declare that they have no competing interests\.

### 7\.3Availability of data and material \(data transparency\)

The data that has been used is confidential\.

### 7\.4Code availability \(software application or custom code\)

The code used in this study is available upon request to the corresponding author\.

### 7\.5Authors’ contributions

Gong Gao: Conceptualization, Methodology, Visualization, Writing – original draft, Writing – review & editing \. Xiao Lai\*: Project administration, Resources\. Ziqi Xie: Writing – review & editing\. Guojie Chen: Writing – original draft\. Xianhui Liu: Validation, Funding acquisition\. Weidong Zhao: Validation, Funding acquisition\.

## References

- \[1\]M\. Alles, N\. Chen, P\. van der Smagt, and B\. Cseke\(2025\)FlowQ: energy\-guided flow policies for offline reinforcement learning\.arXiv preprint arXiv:2505\.14139\.Cited by:[§3](https://arxiv.org/html/2607.26509#S3.p5.1)\.
- \[2\]C\. Bai, L\. Wang, Z\. Yang, Z\. Deng, A\. Garg, P\. Liu, and Z\. Wang\(2022\)Pessimistic bootstrapping for uncertainty\-driven offline reinforcement learning\.arXiv preprint arXiv:2202\.11566\.Cited by:[§3](https://arxiv.org/html/2607.26509#S3.p3.1)\.
- \[3\]G\. Brockman, V\. Cheung, L\. Pettersson, J\. Schneider, J\. Schulman, J\. Tang, and W\. Zaremba\(2016\)Openai gym\.arXiv preprint arXiv:1606\.01540\.Cited by:[§5](https://arxiv.org/html/2607.26509#S5.p2.1)\.
- \[4\]E\. Cetin and O\. Celiktutan\(2023\)Learning pessimism for reinforcement learning\.InProceedings of the AAAI conference on artificial intelligence,Vol\.37,pp\. 6971–6979\.Cited by:[§1](https://arxiv.org/html/2607.26509#S1.p3.1)\.
- \[5\]X\. Chen, A\. Ghadirzadeh, T\. Yu, J\. Wang, A\. Y\. Gao, W\. Li, L\. Bin, C\. Finn, and C\. Zhang\(2022\)Lapo: latent\-variable advantage\-weighted policy optimization for offline reinforcement learning\.Advances in Neural Information Processing Systems35,pp\. 36902–36913\.Cited by:[§3](https://arxiv.org/html/2607.26509#S3.p5.1)\.
- \[6\]E\. Coumans and Y\. Bai\(2016\)Pybullet, a python module for physics simulation for games, robotics and machine learning\.Cited by:[§5](https://arxiv.org/html/2607.26509#S5.p2.1)\.
- \[7\]S\. Fujimoto, H\. Hoof, and D\. Meger\(2018\)Addressing function approximation error in actor\-critic methods\.InInternational conference on machine learning,pp\. 1587–1596\.Cited by:[§1](https://arxiv.org/html/2607.26509#S1.p1.2),[§3](https://arxiv.org/html/2607.26509#S3.p2.1),[§5\.1](https://arxiv.org/html/2607.26509#S5.SS1.p2.1)\.
- \[8\]S\. Fujimoto, D\. Meger, and D\. Precup\(2020\)An equivalence between loss functions and non\-uniform sampling in experience replay\.Advances in neural information processing systems33,pp\. 14219–14230\.Cited by:[§1](https://arxiv.org/html/2607.26509#S1.p3.1),[§3](https://arxiv.org/html/2607.26509#S3.p6.1),[§4\.2](https://arxiv.org/html/2607.26509#S4.SS2.p3.1),[§5\.1](https://arxiv.org/html/2607.26509#S5.SS1.p2.1)\.
- \[9\]G\. Gao, W\. Zhao, X\. Liu, and N\. Jia\(2026\)Improving policy exploitation in online reinforcement learning with instant retrospect action\.Neural Networks,pp\. 108667\.Cited by:[§3](https://arxiv.org/html/2607.26509#S3.p2.1)\.
- \[10\]X\. Gong, S\. Lü, J\. Yu, S\. Zhu, and Z\. Li\(2023\)Adaptive estimation q\-learning with uncertainty and familiarity\.InProceedings of the Thirty\-Second International Joint Conference on Artificial Intelligence,pp\. 3750–3758\.Cited by:[§3](https://arxiv.org/html/2607.26509#S3.p3.1)\.
- \[11\]K\. Guo, S\. Yunfeng, and Y\. Geng\(2022\)Model\-based offline reinforcement learning with pessimism\-modulated dynamics belief\.Advances in Neural Information Processing Systems35,pp\. 449–461\.Cited by:[§3](https://arxiv.org/html/2607.26509#S3.p4.1)\.
- \[12\]T\. Haarnoja, A\. Zhou, K\. Hartikainen, G\. Tucker, S\. Ha, J\. Tan, V\. Kumar, H\. Zhu, A\. Gupta, P\. Abbeel,et al\.\(2018\)Soft actor\-critic algorithms and applications\.arXiv preprint arXiv:1812\.05905\.Cited by:[§1](https://arxiv.org/html/2607.26509#S1.p1.2),[§3](https://arxiv.org/html/2607.26509#S3.p2.1),[§5\.1](https://arxiv.org/html/2607.26509#S5.SS1.p2.1)\.
- \[13\]H\. Hassani, S\. Nikan, and A\. Shami\(2025\)Improved exploration–exploitation trade\-off through adaptive prioritized experience replay\.Neurocomputing614,pp\. 128836\.Cited by:[§3](https://arxiv.org/html/2607.26509#S3.p6.1)\.
- \[14\]L\. Hua, H\. Liu, and Y\. Pan\(2025\)Solving industrial chain job scheduling problems through a deep reinforcement learning method with decay strategy\.Information Sciences702,pp\. 121906\.Cited by:[§1](https://arxiv.org/html/2607.26509#S1.p1.2)\.
- \[15\]P\. J\. Huber\(1992\)Robust estimation of a location parameter\.InBreakthroughs in statistics: Methodology and distribution,pp\. 492–518\.Cited by:[Definition 2](https://arxiv.org/html/2607.26509#Thmdefinition2.p1.5.3)\.
- \[16\]T\. G\. Karimpanal, H\. Le, M\. Abdolshah, S\. Rana, S\. Gupta, T\. Tran, and S\. Venkatesh\(2023\)Balanced q\-learning: combining the influence of optimistic and pessimistic targets\.Artificial Intelligence325,pp\. 104021\.Cited by:[§1](https://arxiv.org/html/2607.26509#S1.p3.1),[§3](https://arxiv.org/html/2607.26509#S3.p4.1)\.
- \[17\]D\. Kim, S\. Park, H\. Jang, J\. Shin, J\. Kim, and Y\. Seo\(2026\)Robot\-r1: reinforcement learning for enhanced embodied reasoning in robotics\.Advances in Neural Information Processing Systems38,pp\. 161472–161507\.Cited by:[§1](https://arxiv.org/html/2607.26509#S1.p1.2)\.
- \[18\]Q\. Lan, Y\. Pan, A\. Fyshe, and M\. White\(2020\)Maxmin q\-learning: controlling the estimation bias of q\-learning\.arXiv preprint arXiv:2002\.06487\.Cited by:[§3](https://arxiv.org/html/2607.26509#S3.p1.1)\.
- \[19\]Y\. Liu, J\. Fan, C\. Zhang, and W\. Shen\(2025\)A knowledge\-driven deep reinforcement learning approach for dynamic scheduling of re\-entrant hybrid flow shop with in\-line product quality inspection\.Knowledge\-Based Systems,pp\. 114418\.Cited by:[§1](https://arxiv.org/html/2607.26509#S1.p1.2)\.
- \[20\]V\. Mnih, K\. Kavukcuoglu, D\. Silver, A\. A\. Rusu, J\. Veness, M\. G\. Bellemare, A\. Graves, M\. Riedmiller, A\. K\. Fidjeland, G\. Ostrovski,et al\.\(2015\)Human\-level control through deep reinforcement learning\.nature518\(7540\),pp\. 529–533\.Cited by:[§3](https://arxiv.org/html/2607.26509#S3.p1.1)\.
- \[21\]T\. Moskovitz, J\. Parker\-Holder, A\. Pacchiano, M\. Arbel, and M\. Jordan\(2021\)Tactical optimism and pessimism for deep reinforcement learning\.Advances in Neural Information Processing Systems34,pp\. 12849–12863\.Cited by:[§3](https://arxiv.org/html/2607.26509#S3.p4.1)\.
- \[22\]A\. Nair, A\. Gupta, M\. Dalal, and S\. Levine\(2020\)Awac: accelerating online reinforcement learning with offline datasets\.arXiv preprint arXiv:2006\.09359\.Cited by:[§3](https://arxiv.org/html/2607.26509#S3.p5.1)\.
- \[23\]M\. Nauman, M\. Bortkiewicz, P\. Miłoś, T\. Trzcinski, M\. Ostaszewski, and M\. Cygan\(2024\)Overestimation, overfitting, and plasticity in actor\-critic: the bitter lesson of reinforcement learning\.InInternational Conference on Machine Learning,pp\. 37342–37364\.Cited by:[§1](https://arxiv.org/html/2607.26509#S1.p3.1)\.
- \[24\]C\. Ngwu, Y\. Liu, and R\. Wu\(2026\)Reinforcement learning in dynamic job shop scheduling: a comprehensive review of ai\-driven approaches in modern manufacturing\.Journal of Intelligent Manufacturing37\(3\),pp\. 1093–1108\.Cited by:[§1](https://arxiv.org/html/2607.26509#S1.p1.2)\.
- \[25\]Y\. Oren, M\. A\. Zanger, P\. R\. V\. der Vaart, M\. M\. Çelikok, W\. Boehmer, and M\. T\. J\. Spaan\(2025\)Value improved actor critic algorithms\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,Cited by:[§3](https://arxiv.org/html/2607.26509#S3.p2.1),[§5\.1](https://arxiv.org/html/2607.26509#S5.SS1.p2.1)\.
- \[26\]X\. B\. Peng, A\. Kumar, G\. Zhang, and S\. Levine\(2019\)Advantage\-weighted regression: simple and scalable off\-policy reinforcement learning\.arXiv preprint arXiv:1910\.00177\.Cited by:[§3](https://arxiv.org/html/2607.26509#S3.p5.1)\.
- \[27\]N\. M\. Quang and H\. W\. Lauw\(2024\)Augmenting decision with hypothesis in reinforcement learning\.InForty\-first International Conference on Machine Learning,Cited by:[§3](https://arxiv.org/html/2607.26509#S3.p2.1),[§5\.1](https://arxiv.org/html/2607.26509#S5.SS1.p2.1)\.
- \[28\]T\. Schaul, J\. Quan, I\. Antonoglou, and D\. Silver\(2015\)Prioritized experience replay\.arXiv preprint arXiv:1511\.05952\.Cited by:[§1](https://arxiv.org/html/2607.26509#S1.p3.1),[§2](https://arxiv.org/html/2607.26509#S2.p3.1),[§3](https://arxiv.org/html/2607.26509#S3.p6.1),[Theorem 1](https://arxiv.org/html/2607.26509#Thmtheorem1.p1.6.6)\.
- \[29\]H\. Shi, Y\. Zhou, K\. Wu, S\. Chen, B\. Ran, and Q\. Nie\(2023\)Physics\-informed deep reinforcement learning\-based integrated two\-dimensional car\-following control strategy for connected automated vehicles\.Knowledge\-Based Systems269,pp\. 110485\.Cited by:[§1](https://arxiv.org/html/2607.26509#S1.p1.2)\.
- \[30\]T\. Tan, H\. Xie, and D\. Lian\(2024\)Adaptive order q\-learning\.\.InInternational Joint Conference on Artificial Intelligence,pp\. 4946–4954\.Cited by:[§1](https://arxiv.org/html/2607.26509#S1.p3.1),[§3](https://arxiv.org/html/2607.26509#S3.p4.1)\.
- \[31\]Y\. Tassa, Y\. Doron, A\. Muldal, T\. Erez, Y\. Li, D\. d\. L\. Casas, D\. Budden, A\. Abdolmaleki, J\. Merel, A\. Lefrancq,et al\.\(2018\)Deepmind control suite\.arXiv preprint arXiv:1801\.00690\.Cited by:[§5](https://arxiv.org/html/2607.26509#S5.p2.1)\.
- \[32\]G\. Tesauroet al\.\(1995\)Temporal difference learning and td\-gammon\.Communications of the ACM38\(3\),pp\. 58–68\.Cited by:[§1](https://arxiv.org/html/2607.26509#S1.p1.2)\.
- \[33\]H\. Van Hasselt, A\. Guez, and D\. Silver\(2016\)Deep reinforcement learning with double q\-learning\.InProceedings of the AAAI conference on artificial intelligence,Vol\.30\.Cited by:[§3](https://arxiv.org/html/2607.26509#S3.p1.1)\.
- \[34\]J\. Wang, L\. Li, W\. Wei, Y\. Zhang, and X\. Yang\(2025\)Dynamic uncertainty estimation for offline reinforcement learning\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 21126–21134\.Cited by:[§1](https://arxiv.org/html/2607.26509#S1.p3.1),[§3](https://arxiv.org/html/2607.26509#S3.p4.1)\.
- \[35\]Y\. Wu, S\. Zhai, N\. Srivastava, J\. Susskind, J\. Zhang, R\. Salakhutdinov, and H\. Goh\(2021\)Uncertainty weighted actor\-critic for offline reinforcement learning\.arXiv preprint arXiv:2105\.08140\.Cited by:[§2](https://arxiv.org/html/2607.26509#S2.p3.1),[§3](https://arxiv.org/html/2607.26509#S3.p3.1)\.
- \[36\]H\. Xiao, C\. Chen, G\. Zhang, and C\. P\. Chen\(2024\)Reinforcement learning\-driven dynamic obstacle avoidance for mobile robot trajectory tracking\.Knowledge\-Based Systems297,pp\. 111974\.Cited by:[§1](https://arxiv.org/html/2607.26509#S1.p1.2)\.
- \[37\]Z\. Yang, X\. Jia, Q\. Li, X\. Yang, M\. Yao, and J\. Yan\(2026\)Raw2drive: reinforcement learning with aligned world models for end\-to\-end autonomous driving \(in carla v2\)\.Advances in Neural Information Processing Systems38,pp\. 134122–134147\.Cited by:[§1](https://arxiv.org/html/2607.26509#S1.p1.2)\.
- \[38\]S\. Zhang, W\. Zhang, and Q\. Gu\(2025\)Energy\-weighted flow matching for offline reinforcement learning\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§3](https://arxiv.org/html/2607.26509#S3.p5.1)\.
- \[39\]Z\. Zhang, Z\. Pan, and M\. J\. Kochenderfer\(2017\)Weighted double q\-learning\.\.InProceedings of the 26th International Joint Conference on Artificial Intelligence,pp\. 3455–3461\.Cited by:[§2](https://arxiv.org/html/2607.26509#S2.p3.1)\.
- \[40\]H\. Zhu, P\. Rashidinejad, and J\. Jiao\(2023\)Importance weighted actor\-critic for optimal conservative offline reinforcement learning\.Advances in Neural Information Processing Systems36,pp\. 49579–49602\.Cited by:[§1](https://arxiv.org/html/2607.26509#S1.p3.1)\.