Revisiting Action Factorization for Complex Action Spaces

# Revisiting Action Factorization for Complex Action Spaces
Source: [https://arxiv.org/html/2606.26574](https://arxiv.org/html/2606.26574)
###### Abstract\.

Many real\-world control problems involve hybrid discrete\-continuous action spaces\. For example, steering and signaling in autonomous driving, and aiming and firing in robotics or video\-games\. Despite real\-world hybrid factorization and reinforcement learning framework support for complex action spaces \(e\.g\., Gymnasium, PettingZoo, TorchRL, SeedRL, Mujoco, etc\), the default environments within those frameworks often implement uniform action space configurations \(LunarLander, Walker2D, Cheetah, SMAC, SUMO, Ant, Atari\)\. Landmark hybrid\-action benchmarks \(RoboCup 2D HFO, SC2LE, Platform, CARLA, etc\) are mostly heavyweight or archival implementations originating from papers which test one or a small number of competing factorization methods on one kind of control\. This article provides a cross\-sectional study of factorization methods \[independent networks, shared encoder, VDN, QPLEX, Joint, Auto\-Regressive\] on each of three families of algorithms \[PPO, SAC, DQN\] across three action spaces \[discretized, hybrid, continuous\] over four lightweight environments \[Platform, hybrid\-LunarLander, Hybrid\-Shoot, CoopPush\]\. Accounting for some invalid pairings such as joint\-continuous, we are left with 220 configurations to analyze each method\. We provide two new C\+\+ parallel gymnasium and petting\-zoo compliant environments \[CoopPush, Hybrid\-Shoot\] to isolate particular challenges such as state\-dependent inter\-action dependence\. Finally, we introduce VDN\-PPO and PPO\-MIX which use a branching critic to assign credit to multi\-headed PPO\. These variants out\-perform all other tested PPO factorizations\. Our results suggest that branching dueling architectures balance compute and performance most effectively, with Auto\-Regressive actions reaching the highest performance overall and native continuous SAC outperforming discrete and hybrid algorithms, albeit both at increased computational cost\.

Action Factorization, Hybrid Action Space, DQN, SAC, PPO

††journal:TAAS††copyright:none††ccs:Computing methodologies Reinforcement learning## 1\.Introduction

While neural networks have been widely accepted in Deep Reinforcement Learning \(D\-RL\) for their capacity to generalize over dependent hybrid feature spaces, there remains an open debate on factorizing complex action spaces\. In the single\-agent domains, sub\-actions𝒜1,…,𝒜H\\mathcal\{A\}\_\{1\},\\ldots,\\mathcal\{A\}\_\{H\}may control individual motor torques or buttons that combine to form the agent’s full joint\-action space𝒜=∏h=1H𝒜h\\mathcal\{A\}=\\prod\_\{h=1\}^\{H\}\\mathcal\{A\}\_\{h\}\(exponential size in the number of sub\-actions\)\. In multi\-agent RL \(MARL\) actions𝒜1,…,𝒜K\\mathcal\{A\}\_\{1\},\\ldots,\\mathcal\{A\}\_\{K\}refers to individual agents’ action spaces which may themselves be decomposed into sub\-actions\. Not all actions are interdependent, so the full joint action space need not always be explored\. For example, the value of movement that avoids danger may not depend upon where an agent is looking or firing, but the decision to fire is highly dependent upon where the agent is aiming\. There are a range of approaches in the literature covering the spectrum from full independence to joint action learning \(JAL\)\. In this paper we cover the following general strategies:

1. \(1\)Independent:Each sub\-action is sampled from a fully independent policy networkπh​\(ah∣s\)\\pi\_\{h\}\(a\_\{h\}\\mid s\), with all networks updated using the same global training signal\.
2. \(2\)Shared Encoder:A shared state representationϕ​\(s\)\\phi\(s\)branches into multiple heads: 1. \(a\)No Mixing:Each action head is exposed to the global reward signal where the impact of the other heads is considered independent noise\. The gradients combine in the encoder so the result is not truly independent\. 2. \(b\)Value Decomposition \(VDN\-style\):A single state\-valueV​\(s\)V\(s\)and per\-head advantagesAh​\(s,ah\)A\_\{h\}\(s,a\_\{h\}\)are learned\. The joint action\-value is additively factored:Q​\(s,𝐚\)=V​\(s\)\+∑h=1HAh​\(s,ah\)Q\(s,\\mathbf\{a\}\)=V\(s\)\+\\sum\_\{h=1\}^\{H\}A\_\{h\}\(s,a\_\{h\}\)\. 3. \(c\)Monotonic \(QPLEX\-style\):A single state\-valueV​\(s\)V\(s\)is learned, and per\-head advantages are combined via a monotonic mixing network:Q​\(s,𝐚\)=V​\(s\)\+fmix​\(A1​\(s,a1\),…,AH​\(s,aH\)\)Q\(s,\\mathbf\{a\}\)=V\(s\)\+f\_\{\\text\{mix\}\}\\big\(A\_\{1\}\(s,a\_\{1\}\),\\dots,A\_\{H\}\(s,a\_\{H\}\)\\big\), enforcing∂fmix∂Ah≥0\\frac\{\\partial f\_\{\\text\{mix\}\}\}\{\\partial A\_\{h\}\}\\geq 0\. 4. \(d\)Concatenated SAC \(SAC\-Concat\):The actor jointly samples continuous dimensions via a squashed Gaussian and discrete dimensions via Gumbel\-Softmax\. The critic treats these as a single input vector𝐚=\[ac,ad\]\\mathbf\{a\}=\[a\_\{c\},a\_\{d\}\]to approximate a unified action\-valueQ​\(s,𝐚\)Q\(s,\\mathbf\{a\}\)\. 5. \(e\)Branching Dueling SAC \(SAC\-BDQ\):The critic utilizes a branching architecture conditioned on the continuous actions\. The network encodes the state and continuous actions to compute a base valueV​\(s,ac\)V\(s,a\_\{c\}\)and branches to output discrete advantagesAh​\(s,ad,h\)A\_\{h\}\(s,a\_\{d,h\}\)per categorical head, yieldingQ​\(s,ac,𝐚d\)=V​\(s,ac\)\+∑h=1HAh​\(s,ad,h\)Q\(s,a\_\{c\},\\mathbf\{a\}\_\{d\}\)=V\(s,a\_\{c\}\)\+\\sum\_\{h=1\}^\{H\}A\_\{h\}\(s,a\_\{d,h\}\)\.
3. \(3\)Auto\-Regressive \(AR\):The joint policy is factored sequentially using the chain rule\. Each action dimensionhhconditions on the statessand previously sampled actionsa<h=\(a1,…,ah−1\)a\_\{<h\}=\(a\_\{1\},\\dots,a\_\{h\-1\}\)\. Sub\-actions are drawn asah∼πh\(⋅∣s,a<h\)a\_\{h\}\\sim\\pi\_\{h\}\(\\cdot\\mid s,a\_\{<h\}\), yielding the joint policyπ​\(𝐚∣s\)=∏h=1Hπh​\(ah∣s,a<h\)\\pi\(\\mathbf\{a\}\\mid s\)=\\prod\_\{h=1\}^\{H\}\\pi\_\{h\}\(a\_\{h\}\\mid s,a\_\{<h\}\)\.
4. \(4\)Joint:The action space is the Cartesian product of all sub\-actions𝒜=∏h=1H𝒜h\\mathcal\{A\}=\\prod\_\{h=1\}^\{H\}\\mathcal\{A\}\_\{h\}\. Continuous actions can be discretized into bins for a single Multinomial policy distribution\.

While these factorization strategies exist for each algorithm individually, their performance is often evaluated in isolation within a single algorithmic family \(e\.g\., value\-based methods\) on a set of benchmarks that may or may not be available for comparison across families\. It remains an open question how the effectiveness of a given strategy, such as value decomposition, translates between value\-based \(DQN\)\(Mnihet al\.,[2013](https://arxiv.org/html/2606.26574#bib.bib1)\), policy gradient \(PG,PPO\)\(Schulmanet al\.,[2017](https://arxiv.org/html/2606.26574#bib.bib5); Suttonet al\.,[1999](https://arxiv.org/html/2606.26574#bib.bib2)\), and actor\-critic \(DDPG,SAC\)\(Haarnojaet al\.,[2018](https://arxiv.org/html/2606.26574#bib.bib3); Lillicrapet al\.,[2015](https://arxiv.org/html/2606.26574#bib.bib4)\)algorithms\. Furthermore, it is unclear how the choice between discrete, continuous, or hybrid action spaces interacts with these factorization methods\. For example, continuous actions can approach arbitrary precision, but mixing continuous and discrete gradients may destructively interfere in a shared encoder\. Auto\-regressive actions can tractably factorize a large joint space, but the “best” action requires a sequence of evaluations that adds learning variance, inference latency, and potential implementation complexity by normalizing flows\(Rezende and Mohamed,[2015](https://arxiv.org/html/2606.26574#bib.bib48)\)\. This paper provides a cross\-sectional comparison of factorization methods with regards to runtime, performance, and implementation effort, to guide future researchers on model selection\.

In order to measure performance, a tunable environment is needed\. Benchmarks like Gymnasium\(Towerset al\.,[2024](https://arxiv.org/html/2606.26574#bib.bib38)\)and Atari100k\(Yeet al\.,[2021](https://arxiv.org/html/2606.26574#bib.bib16)\)for single agent discrete RL, PettingZoo\(Terryet al\.,[2021](https://arxiv.org/html/2606.26574#bib.bib39)\)and SMACv2\(Elliset al\.,[2023](https://arxiv.org/html/2606.26574#bib.bib37)\)for cooperative MARL, and Deepmind\(Tassaet al\.,[2018](https://arxiv.org/html/2606.26574#bib.bib40)\)and Berkeley’s\(Duanet al\.,[2016](https://arxiv.org/html/2606.26574#bib.bib36)\)continuous control suites are imperative for developing and contextualizing new algorithms and their ability to scale to reasonably complex tasks\. We believe that a principled benchmark environment is missing for factorization with tunable inter\-action dependence, action\-rate, and action datatype\.

The remainder of this paper is structured as follows\. Section[4](https://arxiv.org/html/2606.26574#S4)introduces the four benchmark environments including our two novel contributions\. Section[5](https://arxiv.org/html/2606.26574#S5)describes each model family and introduces VDN\-PPO and PPO\-MIX\. Section[6](https://arxiv.org/html/2606.26574#S6)details the experimental setup and runtime normalization methodology\. Section[7](https://arxiv.org/html/2606.26574#S7)presents results and analysis\. Our key findings are: \(i\) shared encoder architectures provide the best compute to performance trade\-off for most settings\. \(ii\) VDN\-PPO and PPO\-MIX substantially outperform shared\-encoder PPO on discrete action spaces by redistributing credit to action heads with higher state agency\. \(iii\) monotonic mixing \(QPLEX\) does not improve over shared encoder in simple single\-agent settings\. \(iv\) Action type \(discrete, hybrid, continuous\) has less impact on performance than factorization strategy\.

## 2\.Background / Preliminaries

### 2\.1\.Reinforcement Learning

Standard single\-agent reinforcement learning problems are often modeled as a Markov Decision Process \(MDP\) consisting of the tuple\(𝒮,𝒜,P,R,γ\)\(\\mathcal\{S\},\\mathcal\{A\},P,R,\\gamma\)\.𝒮⊂ℝN\\mathcal\{S\}\\subset\\mathbb\{R\}^\{N\}is the set of all environment states,𝒜=∏h=1H𝒜h\\mathcal\{A\}=\\prod\_\{h=1\}^\{H\}\\mathcal\{A\}\_\{h\}is the joint action space factored overHHaction heads,P:𝒮×𝒜→Δ​\(𝒮\)P:\\mathcal\{S\}\\times\\mathcal\{A\}\\rightarrow\\Delta\(\\mathcal\{S\}\)the transition distribution, andR:𝒮×𝒜→ℝR:\\mathcal\{S\}\\times\\mathcal\{A\}\\rightarrow\\mathbb\{R\}is the reward function withγ\\gammaas the discount factor andrt∈Rr\_\{t\}\\in Rthe reward at time t\. We denote the state and joint action at timettasst,𝐚ts\_\{t\},\\mathbf\{a\}\_\{t\}sampled from factored policyπ​\(𝐚t\|st\)=∏h=1Hπh​\(ah,t∣st\)\\pi\(\\mathbf\{a\}\_\{t\}\|s\_\{t\}\)=\\prod\_\{h=1\}^\{H\}\\pi\_\{h\}\(a\_\{h,t\}\\mid s\_\{t\}\)\. Our goal is to find a policyπ\\pithat maximizes the expected sum of discounted rewards,𝔼​\[Gt\]\\mathbb\{E\}\[G\_\{t\}\]whereGt=∑τ=t∞γτ−t​R​\(sτ,𝐚τ\)G\_\{t\}=\\sum\_\{\\tau=t\}^\{\\infty\}\\gamma^\{\\tau\-t\}R\(s\_\{\\tau\},\\mathbf\{a\}\_\{\\tau\}\)and𝐚t∼π\(⋅\|st\),st\+1∼P\(⋅\|st,𝐚t\)\\mathbf\{a\}\_\{t\}\\sim\\pi\(\\cdot\|s\_\{t\}\),s\_\{t\+1\}\\sim P\(\\cdot\|s\_\{t\},\\mathbf\{a\}\_\{t\}\)\. We do not knowPPorRRin advance\. The expected return from timettonward givenπ\\pi, \(𝔼​\[Gt\|π,st\]\\mathbb\{E\}\[G\_\{t\}\|\\pi,s\_\{t\}\]\) is the state\-value function:

Vπ​\(s\):=𝔼π​\[∑τ=t∞γτ−t​r​\(sτ,𝐚τ\)∣st=s\]V^\{\\pi\}\(s\):=\\mathbb\{E\}\_\{\\pi\}\[\\sum\_\{\\tau=t\}^\{\\infty\}\\gamma^\{\\tau\-t\}r\(s\_\{\\tau\},\\mathbf\{a\}\_\{\\tau\}\)\\mid s\_\{t\}=s\]and the state\-action value \(𝔼​\[Gt\|π,st,at\]\\mathbb\{E\}\[G\_\{t\}\|\\pi,s\_\{t\},a\_\{t\}\]\) is:

Qπ​\(s,𝐚\):=𝔼π​\[∑τ=t∞γτ−t​r​\(sτ,𝐚τ\)∣st=s,𝐚t=𝐚\]Q^\{\\pi\}\(s,\\mathbf\{a\}\):=\\mathbb\{E\}\_\{\\pi\}\[\\sum\_\{\\tau=t\}^\{\\infty\}\\gamma^\{\\tau\-t\}r\(s\_\{\\tau\},\\mathbf\{a\}\_\{\\tau\}\)\\mid s\_\{t\}=s,\\mathbf\{a\}\_\{t\}=\\mathbf\{a\}\]

### 2\.2\.Multi\-Action as a Special Case of Multi\-Agent Control

A complex action space can be factored into a joint space𝒜=∏h=1H𝒜h\\mathcal\{A\}=\\prod\_\{h=1\}^\{H\}\\mathcal\{A\}\_\{h\}, encompassing discrete, continuous, or hybrid dimensions\. For a multi\-agent system withKKagents, each takingDDdiscrete andCCcontinuous actions, we can formulate this as a multi\-action problem withH:=K​\(D\+C\)H:=K\(D\+C\)distinct action heads\.

A cooperative Decentralized Partially Observable MDP \(Dec\-POMDP\) defines an observation function𝒪\\mathcal\{O\}mapping the global state to local observationsoio\_\{i\}for each agentii, with a shared global reward functionR​\(s,𝐚\)R\(s,\\mathbf\{a\}\)\. When such an environment grants full central observability to all agents \(oi≡so\_\{i\}\\equiv sfor allii\), the Dec\-POMDP collapses into a standard single\-agent MDP with a high\-dimensional, multi\-action space\. Consequently, contemporary Multi\-Agent RL \(MARL\) factorization techniques can be applied directly to our single\-agent setting\.

State\-of\-the\-art MARL relies on Centralized Training with Decentralized Execution \(CTDE\) to bridge the gap between global value estimation and local policy execution\. Because our single\-agent actors inherently possess full\-state observability during execution, we are freed from the strict decentralization constraints of CTDE\. This allows us to simplify MARL architectures, such as QPLEX, while taking advantage of their credit assignment capabilities to bypass the computational infeasibility of full JAL\.

## 3\.Related Work

Factored action spaces across single\-agent reinforcement learning \(RL\) and cooperative multi\-agent RL \(MARL\) provides a rich set of methods for breaking down complex actions into tractable forms\.

### 3\.1\.Independent vs Centralized Critics

At one extreme, independent learners \(e\.g\., IQL\(Tan,[1993](https://arxiv.org/html/2606.26574#bib.bib8)\), IPPO\(Yuet al\.,[2022](https://arxiv.org/html/2606.26574#bib.bib9)\)\) treat each action dimension or agent as a separate entity learning from a shared reward signal\. While linearly scalable, this approach can suffer from non\-stationary instability, local equilibria, and credit assignment difficulties\(Matignonet al\.,[2012](https://arxiv.org/html/2606.26574#bib.bib19); Hernandez\-Lealet al\.,[2019](https://arxiv.org/html/2606.26574#bib.bib20)\)\. Centralized actor\-critic methods like MADDPG\(Loweet al\.,[2017](https://arxiv.org/html/2606.26574#bib.bib10)\)conditioned their value function on all actions, providing a separate gradient signal to each decentralized actor for credit assignment\. Centralized critics can struggle with representational capacity over the full joint state\-action space, especially for many agents\(Foersteret al\.,[2018](https://arxiv.org/html/2606.26574#bib.bib25); Rashidet al\.,[2020](https://arxiv.org/html/2606.26574#bib.bib14)\)\.

### 3\.2\.Value Function Factorization

Another body of work focuses on factoring the joint\-action value functionQt​o​t​\(s,𝐚\)Q\_\{tot\}\(s,\\mathbf\{a\}\)into individual per\-action components\. For greedy action selection to be tractable in combinatorial spaces, it is imperative that the factorization satisfies the Individual\-Global\-Max \(IGM\) principle\. The IGM principle is equally applicable in single\-agent settings with factored action spaces; its primary function is computational tractability of joint argmax \(exponential inHHwithout IGM, polynomial with it\), rather than decentralized execution\. IGM requiresarg⁡max𝐚⁡Qt​o​t​\(s,𝐚\)=\(arg⁡maxa1⁡Q1,…,arg⁡maxah⁡Qh\)\\arg\\max\_\{\\mathbf\{a\}\}Q\_\{tot\}\(s,\\mathbf\{a\}\)=\(\\arg\\max\_\{a\_\{1\}\}Q\_\{1\},\\dots,\\arg\\max\_\{a\_\{h\}\}Q\_\{h\}\)\. Value Decomposition Networks \(VDN\)\(Sunehaget al\.,[2017](https://arxiv.org/html/2606.26574#bib.bib12)\)strictly enforce IGM by assuming additive contributions, definingQt​o​tQ\_\{tot\}as the sum of individualQhQ\_\{h\}values\. QMIX\(Rashidet al\.,[2020](https://arxiv.org/html/2606.26574#bib.bib14)\)broadens this representational capacity while maintaining IGM by passing the individual values through a monotonic mixing network, ensuring∂Qt​o​t∂Qh≥0\\frac\{\\partial Q\_\{tot\}\}\{\\partial Q\_\{h\}\}\\geq 0\. Q\-PLEX\(Wanget al\.,[2020](https://arxiv.org/html/2606.26574#bib.bib15)\)mixes advantages instead of full Q values to allow the value\-head to absorb some of the Q\-value approximation variance\.

Adopting a dueling architecture\(Tavakoliet al\.,[2018](https://arxiv.org/html/2606.26574#bib.bib13)\)shifts IGM from Q\-values to advantages:V​\(s\)V\(s\)is factored out andarg⁡max𝐚⁡At​o​t=\(arg⁡maxa1⁡A1,…,arg⁡maxah⁡Ah\)\\arg\\max\_\{\\mathbf\{a\}\}A\_\{tot\}=\(\\arg\\max\_\{a\_\{1\}\}A\_\{1\},\\dots,\\arg\\max\_\{a\_\{h\}\}A\_\{h\}\); sinceQt​o​t=V​\(s\)\+At​o​tQ\_\{tot\}=V\(s\)\+A\_\{tot\}, maximizing the joint advantage guarantees maximizingQt​o​tQ\_\{tot\}\. Methods like QPLEX\(Wanget al\.,[2020](https://arxiv.org/html/2606.26574#bib.bib15)\)leverage this by combining dueling architectures with monotonic mixing networks over the advantages, effectively capturing complex inter\-action dependencies while strictly guaranteeing IGM\. QTRAN\(Sonet al\.,[2019](https://arxiv.org/html/2606.26574#bib.bib24)\)approaches this by relaxing structural constraints and enforcing IGM through loss regularization\. For policy gradient methods, COMA\(Foersteret al\.,[2018](https://arxiv.org/html/2606.26574#bib.bib25)\)forms per\-agent advantages via a counterfactual baseline at an increased evaluation cost\. We instead use a first\-order Taylor approximation of the mixer gradient to save critic evaluations\. VDMPO\(Wanget al\.,[2025](https://arxiv.org/html/2606.26574#bib.bib54)\)scales global reward per agent using individual value function contributions, but it does not perform credit assignment to individual actions/agents\. We scale the global advantage per action head by its state\-level importance \(see Section[5\.2](https://arxiv.org/html/2606.26574#S5.SS2)\)\.

### 3\.3\.Structured Policy Decomposition

An alternative paradigm imposes an explicit structure on the policy itself\. Auto\-regressive methods factorize the joint policyπ​\(a\|s\)\\pi\(a\|s\)into a chain of conditional probabilities,Πi​π​\(ai\|s,a<i\)\\Pi\_\{i\}\\pi\(a\_\{i\}\|s,a\_\{<i\}\), where each sub\-action is selected sequentially, conditioned on the preceding ones\(Korenkevychet al\.,[2019](https://arxiv.org/html/2606.26574#bib.bib49); Liet al\.,[2023a](https://arxiv.org/html/2606.26574#bib.bib32); Rezende and Mohamed,[2015](https://arxiv.org/html/2606.26574#bib.bib48); Vinyalset al\.,[2019](https://arxiv.org/html/2606.26574#bib.bib33)\)\. This approach can capture arbitrary dependencies between actions at the cost of sequential computation time and often requires a pre\-defined action ordering\.

### 3\.4\.Hybrid Action Decompositions

Common solutions to mixed discrete\-continuous spaces can involve parameterizing discrete actions with continuous values\(Xionget al\.,[2018](https://arxiv.org/html/2606.26574#bib.bib30)\)or using networks with mixed output heads\(Fanet al\.,[2019](https://arxiv.org/html/2606.26574#bib.bib42); Chenet al\.,[2022](https://arxiv.org/html/2606.26574#bib.bib43)\)\. For actor\-critic methods, this often requires differentiable relaxations of discrete distributions, such as the Gumbel\-Softmax\(Janget al\.,[2016](https://arxiv.org/html/2606.26574#bib.bib26)\)estimator, to enable backpropagation through discrete action choices\(Chenet al\.,[2022](https://arxiv.org/html/2606.26574#bib.bib43); Fanet al\.,[2019](https://arxiv.org/html/2606.26574#bib.bib42)\)\. There exist hybrid implementations for PPO\(Fanet al\.,[2019](https://arxiv.org/html/2606.26574#bib.bib42)\)and SAC\(Chenet al\.,[2022](https://arxiv.org/html/2606.26574#bib.bib43); Liuet al\.,[2024](https://arxiv.org/html/2606.26574#bib.bib44); Delalleauet al\.,[2019](https://arxiv.org/html/2606.26574#bib.bib17); Camposet al\.,[2022](https://arxiv.org/html/2606.26574#bib.bib18)\), with discretization such as bang\-bang control for hybrid DQN\(Seydeet al\.,[2021](https://arxiv.org/html/2606.26574#bib.bib45)\)\. For SAC in particular \(which is typically fully continuous\), it is unclear whether it is better to train a critic which takes both relaxed discrete and continuous actions as input, or continuous actions as input with discreteQQvalue outputs to serve as a parameterized model\. We present the single\-value criticQ​\(𝐚𝐝,𝐚𝐜,s\)Q\(\\mathbf\{a\_\{d\}\},\\mathbf\{a\_\{c\}\},s\)as SAC\-Concat, and the parameterized criticQ=V​\(s,𝐚𝐜\)\+∑hdAh​\(s,𝐚𝐜\)Q=V\(s,\\mathbf\{a\_\{c\}\}\)\+\\sum\_\{h\}^\{d\}A\_\{h\}\(s,\\mathbf\{a\_\{c\}\}\)as SAC\-BDQ\.

### 3\.5\.Action Embeddings

Latent action embedding methods such as HyAR\(Liet al\.,[2021](https://arxiv.org/html/2606.26574#bib.bib29)\)and action representation learning\(Chandaket al\.,[2019](https://arxiv.org/html/2606.26574#bib.bib46)\)map hybrid spaces into compact continuous latent spaces\. Because no canonical method exists to implement such embeddings uniformly across DQN, PPO, and SAC, we exclude them from our comparisons\.

## 4\.Environments

![Refer to caption](https://arxiv.org/html/2606.26574v1/envs.png)Figure 1\.Lunar\-Landerv3: Shown top left,CoopPush:Particles,Boulders, andlandmarks\. Default \(bottom left\) and independent \(bottom center\)\.Hybrid\-Shoot: Targets:•Selected:oShoot Location:o\.Platform: Agent \(Purple\) Obstacles \(Grey\)For each environment below, we take the continuous\-action variant and discretize continuous intervals into discrete bins by the following transform: Let a continuous valuex∈\[xmin,xmax\]x\\in\[x\_\{\\min\},x\_\{\\max\}\]be mapped to a zero\-indexed discrete bin indexb∈\{0,1,…,N−1\}b\\in\\\{0,1,\\dots,N\-1\\\}\. The uniform bin width is defined inline asΔ​x=xmax−xminN\\Delta x=\\frac\{x\_\{\\max\}\-x\_\{\\min\}\}\{N\}, and the resulting discretized bin index is computed viab=min⁡\(⌊x−xminΔ​x⌋,N−1\)b=\\min\\left\(\\left\\lfloor\\frac\{x\-x\_\{\\min\}\}\{\\Delta x\}\\right\\rfloor,N\-1\\right\), where⌊⋅⌋\\lfloor\\cdot\\rfloordenotes the floor function\. CoopPush and Hybrid\-Shoot natively support parallel execution in C\+\+ and we use EnvPool\(Wenget al\.,[2022](https://arxiv.org/html/2606.26574#bib.bib21)\)for lunar lander\. For hybrid\-platform we fall back on gymnasium’s SyncVecEnv\(Towerset al\.,[2024](https://arxiv.org/html/2606.26574#bib.bib38)\)API for vectorized execution\.

### 4\.1\.Contextual\-Decoupler

We introduce Contextual\-Decoupler as a minimal benchmark environment designed to isolate and evaluate factored credit assignment in multi\-head action spaces\. At each timestep, the environment state is defined by a discrete tuples=\(c,τ0,τ1\)s=\(c,\\tau\_\{0\},\\tau\_\{1\}\)sampled uniformly from\{0,1\}×\{0,…,N−1\}2\\\{0,1\\\}\\times\\\{0,\\dots,N\-1\\\}^\{2\}, wherec∈\{0,1\}c\\in\\\{0,1\\\}denotes which action head is active andτ0,τ1\\tau\_\{0\},\\tau\_\{1\}represent the target actions for action headsa0a\_\{0\}anda1a\_\{1\}respectively\. The agent selects a joint actiona=\(a0,a1\)∈\{0,…,N−1\}2a=\(a\_\{0\},a\_\{1\}\)\\in\\\{0,\\dots,N\-1\\\}^\{2\}\. The scalar rewardR​\(s,a\)R\(s,a\)is governed by the active headcc, choosing the target actionτc\\tau\_\{c\}while slightly penalizing the inactive head for any action besides zero:R​\(s,a\)=𝕀​\(ac=τc\)−𝕀​\(ac≠τc\)−0\.1⋅𝕀​\(a1−c≠0\)R\(s,a\)=\\mathbb\{I\}\(a\_\{c\}=\\tau\_\{c\}\)\-\\mathbb\{I\}\(a\_\{c\}\\neq\\tau\_\{c\}\)\-0\.1\\cdot\\mathbb\{I\}\(a\_\{1\-c\}\\neq 0\), where𝕀​\(⋅\)\\mathbb\{I\}\(\\cdot\)is the indicator function\. The transition dynamics are completely i\.i\.d\. across timesteps, meaning the next states′s^\{\\prime\}is drawn uniformly at random such that𝒫​\(s′∣s,a\)=12​N2\\mathcal\{P\}\(s^\{\\prime\}\\mid s,a\)=\\frac\{1\}\{2N^\{2\}\}for alls,as,a\. The episode truncates deterministically after a fixed horizon ofTTsteps\. By eliminating temporal state dependencies, this formulation isolates structural action coordination from sequential credit assignment difficulties\.

### 4\.2\.Platform

We include the Platform domain\(Massonet al\.,[2016](https://arxiv.org/html/2606.26574#bib.bib11)\), a classic benchmark designed to study parameterized action spaces\. In Platform, an agent \(pictured in purple in Figure[1](https://arxiv.org/html/2606.26574#S4.F1)\) must traverse a sequence of platforms separated by gaps while avoiding falling or colliding with enemies \(grey obstacles\)\. At each step, the agent selects a discrete action typead∈\{0,1,2\}a\_\{d\}\\in\\\{0,1,2\\\}corresponding toRun\(move horizontally\),Hop\(small jump to clear gaps\), andLeap\(large jump\), respectively\. Each discrete action is parameterized by a continuous scalar controlling the magnitude of the movement: running speedp0∈\[0,30\]p\_\{0\}\\in\[0,30\], hopping powerp1∈\[0,720\]p\_\{1\}\\in\[0,720\], and leaping powerp2∈\[0,430\]p\_\{2\}\\in\[0,430\]\. When an action is taken, the environment executes the selected actionada\_\{d\}with its corresponding parameterpadp\_\{a\_\{d\}\}while ignoring the others\. The observation space is a tuple containing a scaled 9\-dimensional state vector \(representing the kinematics of the agent and enemies alongside the geometric properties of the platforms and gaps\) and a discrete time step constraint\. By requiring the simultaneous selection of an action type and its continuous magnitude, Platform isolates the difficulty of evaluating discrete choices whose values are inherently dependent on the precision of their continuous parameters\. As with the other environments, we discretize the continuous parameter intervals intoNNdiscrete bins using our uniform mapping to evaluate algorithms across different levels of control granularity\.

### 4\.3\.Lunar Lander

We discretize actions into 3 bins for actions markeddiscretein the wrapper and 5 bins forcontinuousso that DQN can be viewed as the baseline for discretization granularity\. LunarLanderv3 provides a grounded baseline to confirm each model family works as expected under known factorizations with limited inter\-action dependence\.

### 4\.4\.Hybrid\-Shoot

We introduce Hybrid\-Shoot as a configurable version of a classic parameterized action space problem\. In Hybrid\-Shoot,NNtargets positioned randomly on an 2D\-plane generate a negative reward each round\. One action selects a target, and the other action selects an\[ax,ay\]\[a\_\{x\},a\_\{y\}\]location\. In dependent mode, a target can only be shot when it is selected and‖⟨ax,ay⟩−⟨tx,ty⟩‖<c\|\|\\langle a\_\{x\},a\_\{y\}\\rangle\-\\langle t\_\{x\},t\_\{y\}\\rangle\|\|<cwhereccis the hit radius\. Positionsaxa\_\{x\}andaya\_\{y\}are chosen by separate sub\-actions\. A target that is currently selected produces no negative reward, and a shot target produces no negative reward starting on the next turn\. In independent mode, a target may be shot whether it is selected or not and a single parameterppdetermines⟨ax,ay⟩\\langle a\_\{x\},a\_\{y\}\\rangleby a 16x16 Hilbert curve so that 2D locality is maintained by a single scalar action\. This way, the best selection and shoot location do not depend on each\-other and there is no dependentx,yx,ycomponents\. By altering between dependent and independent modes, and by changingccand the action types of any action, we can isolate inter\-action dependence vs precision based learning difficulties\. Thex,yx,ycomponents are discretized into a 10x10 grid with hit radius 0\.1 so that all algorithms can can reach all targets\.

### 4\.5\.CoopPush

Finally we introduce a cooperative push environment where there are three classes of entities 1\. Particles, 2\. Boulders, and 3\. Landmarks\. Particles and boulders collide with one\-another and a reward is given when a boulder overlaps with a landmark that it has not visited\. The boulders do not move on their own, so the particles need to push them\. The environment supports a dense reward for the change in distance between each boulder and its nearest unvisited landmark\. The environment also has two termination modes,visit\_oneandvisit\_allwhere the game ends when each boulder has visited at least one, or all possible landmarks respectively\. The states∈ℝ2​\(Np\+Nb\+Nl\)s\\in\\mathbb\{R\}^\{2\(N\_\{p\}\+N\_\{b\}\+N\_\{l\}\)\}concatenates the 2D positions of allNpN\_\{p\}particles,NbN\_\{b\}boulders, andNlN\_\{l\}landmarks\. Each particle takes\(δx,δy\)∈\[−1,1\]2\(\\delta\_\{x\},\\delta\_\{y\}\)\\in\[\-1,1\]^\{2\}as a continuous action with movement capped at magnitude 1\.0, or discrete actions with cardinality 9 for no\-op and 8 directions\. Particles are assigned to starting positions in a random order upon reset so that models cannot memorize a state\-independent policy\.

We consider two layouts pictured in Figure[1](https://arxiv.org/html/2606.26574#S4.F1)\.DefaultvsIndependentmeasures initial action dependence\. In theIndependentsetting there is not enough time to push the other particle’s boulder, so the optimal action does not depend on the other particle\. InDefault, if the left particles are pushing the boulder to the right, it is better for the right particles to get out of the way than to stalemate the boulder\. Finally, the engine supportsδt\\delta\_\{t\}andnum\_physics\_stepsto control action granularity as in Mujoco\(Todorovet al\.,[2012](https://arxiv.org/html/2606.26574#bib.bib22)\)\. By increasing the steps taken andδt\\delta\_\{t\}\(lower control frequency\) continuous precision becomes more advantageous than duty\-cycle discrete control\.

## 5\.Model Families

To study action factorization across algorithm families, we implement variants of DQN, SAC, and PPO\. Rather than treating each as a bespoke architecture, we build them from two reusable building blocks: a*Stochastic Actor*that emits factored discrete and continuous sub\-actions, and a*Branching Dueling Critic*\(denotedQS\) that decomposes the joint action\-value into a shared state value and per\-head advantages\. The data flows for these blocks, together with the wiring that realizes monolithic, branching, and auto\-regressive factorizations, are shown in the topologies below\. A single mechanism ties the families together: each factorization corresponds to a choice of advantage\-weighting termWhW\_\{h\}applied to the heads of the QS critic before aggregation \(see the weighting table in the QS topology\)\. The subsections that follow specify the mathematical formulations, constraints, and credit\-assignment mechanics that these shared structures induce\.

### 1\. Stochastic Actor Topology

TheStochasticActorserves as the foundational policy network for both PPO and SAC\. It splits into continuous and discrete branches\. The discrete branch offers two pathways depending on the algorithm: standard Categorical sampling \(PPO\) or a differentiable Gumbel\-Softmax relaxation \(SAC\)\.

XXActorEncoderCont\.HeadDisc\.Heads𝒩​\(μ,σ\)\\mathcal\{N\}\(\\mu,\\sigma\)\+tanh\\tanhaca\_\{c\}Categorical\(PPO\)Gumbel\-Softmax\(SAC/Optional\)ada\_\{d\}ada\_\{d\}μ,log⁡σ\\mu,\\log\\sigmalogits

### 2\. Branching Dueling Critic Topology \(QS\)

TheQSnetwork is the unified critic\. It factors the action\-value into a state\-valueVVand dimensional advantagesAhA\_\{h\}\. Crucially, the combination is mediated by an algorithm\-dependent weightWhW\_\{h\}applied to each advantage head before summation\.

XXCriticEncoderValueHeadAdvantageHeadsV​\(X\)V\(X\)Ah​\(X\)A\_\{h\}\(X\)⊗\\otimesWhW\_\{h\}⊕\\oplusQ​\(X\)Q\(X\)Algorithm Weighting \(WhW\_\{h\}\): Shared / VDN:Wh=1W\_\{h\}=1 VDN\-PPO:Wh=whα​\(X\)W\_\{h\}=w^\{\\alpha\}\_\{h\}\(X\)\(Importance\) PPO\-MIX / QPLEX:Wh=∂fmix∂AhW\_\{h\}=\\frac\{\\partial f\_\{\\text\{mix\}\}\}\{\\partial A\_\{h\}\}WeightedAhA\_\{h\}

### 3\. Factorization Wiring Arrangements

Using the abstract topologies defined above, we wire the specific inputsXXto isolate different factorization strategies\.

A\. SAC\-Concat \(Monolithic\)ssaca\_\{c\}ada\_\{d\}∥\\parallelStandardCriticQ​\(s,𝐚\)Q\(s,\\mathbf\{a\}\)XXB\. SAC\-BDQssaca\_\{c\}ad,ha\_\{d,h\}∥\\parallelQS CriticBlock⊕\\oplusQ​\(s,ac,𝐚d\)Q\(s,a\_\{c\},\\mathbf\{a\}\_\{d\}\)XXV,AhV,A\_\{h\}selectsC\. Auto\-Regressive ActorssActorHead 1a1a\_\{1\}∥\\parallelActorHead 2a2a\_\{2\}…\\dotsX1X\_\{1\}X2X\_\{2\}

### 5\.1\.Branching Dueling Deep Q\-Learning \(BDQ\)

Our branching\(Tavakoliet al\.,[2018](https://arxiv.org/html/2606.26574#bib.bib13)\)dueling\(Wanget al\.,[2016](https://arxiv.org/html/2606.26574#bib.bib6)\)DQN instantiates the QS Critic topology directly: the critic encoder feeds a single global state\-value headV​\(s\)V\(s\)and an advantage head per action branchhh, producing a max\-zero advantage vectorA→h\\vec\{A\}^\{h\}withmax⁡\(A→h\)=0\\max\(\\vec\{A\}^\{h\}\)=0\. Max\-centering is critical in two cases because it makes the shared value function identifiable when advantages are monotonically mixed \(see Appendix[B](https://arxiv.org/html/2606.26574#A2)\)\. Otherwise, the mixing network can learn to represent an arbitrary portion of the value function\.

The joint action\-valueQ​\(s,𝐚\)Q\(s,\\mathbf\{a\}\)is assembled from these heads according to the weightingWhW\_\{h\}of the topology\. In Shared Encoder and VDN modes \(Wh=1W\_\{h\}=1\), the heads sum to the joint advantage,

\(1\)Q​\(s,𝐚\)=V​\(s\)\+∑h=1HAh​\(s,ah\),Q\(s,\\mathbf\{a\}\)=V\(s\)\+\\sum\_\{h=1\}^\{H\}A^\{h\}\(s,a\_\{h\}\),so that in the shared\-encoder limit each branch independently recoversQh​\(s,ah\)=V​\(s\)\+Ah​\(s,ah\)Q^\{h\}\(s,a\_\{h\}\)=V\(s\)\+A^\{h\}\(s,a\_\{h\}\)\. In QPLEX mode, the advantages are instead combined by a state\-conditioned monotonic mixing network,Q​\(s,𝐚\)=V​\(s\)\+fmix​\(A1,…,AH∣s\)Q\(s,\\mathbf\{a\}\)=V\(s\)\+f\_\{\\text\{mix\}\}\(A^\{1\},\\dots,A^\{H\}\\mid s\)with∂fmix/∂Ah≥0\\partial f\_\{\\text\{mix\}\}/\\partial A^\{h\}\\geq 0\. This is a single\-agent reduction of QPLEX\(Wanget al\.,[2020](https://arxiv.org/html/2606.26574#bib.bib15)\): because the dueling architecture supplies full state observability through one globalV​\(s\)V\(s\), the mixer operates over max\-zero*advantages*rather than rawQQ\-values\. Finally, in Auto\-Regressive mode the joint action is rolled out by taking the argmax of each advantage head in sequence, feeding earlier sub\-actions forward as in the Auto\-Regressive Actor wiring\. Because DQN selects actions by hard switching rather than sampling on the forward pass, this sequential evaluation can inflate critic variance relative to the single\-pass stochastic sampling of SAC or PPO\. Finally, we use 11 action bins for all ”continuous” action dimensions\. 11 dims gives finer control for continuous actions such as engine thrusts in lunar lander, but it also increases the observation size for centralized mixing networks and the overestimation bias caused by argmax targeting\. The purpose of including “continuous” DQN is to show the impact of bin granularity on the various factorizations\.

### 5\.2\.Branching Critic PPO: VDN\-PPO and PPO\-MIX

We adapt Proximal Policy Optimization\(Schulmanet al\.,[2017](https://arxiv.org/html/2606.26574#bib.bib5)\)to complex action spaces using the Stochastic Actor topology, with categorical heads for discrete sub\-actions and squashed Gaussian heads for continuous ones\. A standard PPO critic emits a scalar baselineV​\(s\)V\(s\)that feeds an identical Generalized Advantage Estimation \(GAE\) signal to every head\. We instead reuse the QS critic as a*branching*critic, decomposing the joint state\-action value as

\(2\)Qϕ​\(s,𝐚\)=Vϕ​\(s\)\+∑h=1HAϕ,h​\(s,ah\),Q\_\{\\phi\}\(s,\\mathbf\{a\}\)=V\_\{\\phi\}\(s\)\+\\sum\_\{h=1\}^\{H\}A\_\{\\phi,h\}\(s,a\_\{h\}\),where each advantage streamAϕ,hA\_\{\\phi,h\}is normalized to zero mean under the*current*policy:𝔼ah∼πh​\[Aϕ,h​\(s,ah\)\]=0\\mathbb\{E\}\_\{a\_\{h\}\\sim\\pi\_\{h\}\}\[A\_\{\\phi,h\}\(s,a\_\{h\}\)\]=0\. This on\-policy normalization is what distinguishes our critic from standard BDQ, which centers advantages under a uniform or greedy distribution: the policy gradient requiresVϕ​\(s\)V\_\{\\phi\}\(s\)to target the on\-policy state valueVπ​\(s\)V^\{\\pi\}\(s\), so the per\-head advantages remain valid baselines for the current policy\.

Discrete heads are centered by weighting each advantage with its categorical action probability\. For continuous heads, the critic produces several advantage ‘bins’ spanning the action range, and centering requires the probability massmh,cm\_\{h,c\}the policy places in each binccof headhh\. For actions drawn from a squashed Gaussianπh​\(s\)→\(μh,σh\)\\pi\_\{h\}\(s\)\\rightarrow\(\\mu\_\{h\},\\sigma\_\{h\}\), this mass follows from the bin’s interval endpointsxc,xc\+1x\_\{c\},x\_\{c\+1\}:

\(3\)ah∼tanh⁡\(𝒩​\(μh,σh\)\)⇒mh,c=CDF​\(tanh−1⁡\(xc\+1\);μh,σh\)−CDF​\(tanh−1⁡\(xc\);μh,σh\)\.a\_\{h\}\\sim\\tanh\\left\(\\mathcal\{N\}\(\\mu\_\{h\},\\sigma\_\{h\}\)\\right\)\\;\\Rightarrow\\;m\_\{h,c\}=\\mathrm\{CDF\}\\left\(\\tanh^\{\-1\}\(x\_\{c\+1\}\);\\mu\_\{h\},\\sigma\_\{h\}\\right\)\-\\mathrm\{CDF\}\\left\(\\tanh^\{\-1\}\(x\_\{c\}\);\\mu\_\{h\},\\sigma\_\{h\}\\right\)\.We computemh,cm\_\{h,c\}once for all states in the rollout buffer before any updates, so that the advantage means and the resulting GAE are both grounded in the behavior policy that produced the buffer\.

##### VV\-only target\.

Including the next\-step joint actionQ​\(st\+1,𝐚t\+1\)Q\(s\_\{t\+1\},\\mathbf\{a\}\_\{t\+1\}\)in the regression target would inject substantial variance and compute overhead from noisy advantage estimation across allHHheads\. We therefore train the branching critic with aVV\-only targetyt=rt\+γ​Vϕ​\(st\+1\)y\_\{t\}=r\_\{t\}\+\\gamma V\_\{\\phi\}\(s\_\{t\+1\}\)\. Expanding the squared error ofQϕ=Vϕ\+∑hAϕ,hQ\_\{\\phi\}=V\_\{\\phi\}\+\\sum\_\{h\}A\_\{\\phi,h\}against this target yields

\(4\)ℒ​\(ϕ\)=12​𝔼τ∼π​\[\(∑h=1HAϕ,h​\(st,ah,t\)−δtV\)2\],δtV=rt\+γ​Vϕ​\(st\+1\)−Vϕ​\(st\),\\mathcal\{L\}\(\\phi\)=\\tfrac\{1\}\{2\}\\,\\mathbb\{E\}\_\{\\tau\\sim\\pi\}\\left\[\\left\(\\sum\_\{h=1\}^\{H\}A\_\{\\phi,h\}\(s\_\{t\},a\_\{h,t\}\)\-\\delta\_\{t\}^\{V\}\\right\)^\{2\}\\right\],\\quad\\delta\_\{t\}^\{V\}=r\_\{t\}\+\\gamma V\_\{\\phi\}\(s\_\{t\+1\}\)\-V\_\{\\phi\}\(s\_\{t\}\),which shows that training fits the*sum*of per\-head advantages to theVV\-based TD residualδtV\\delta\_\{t\}^\{V\}, directly bridging the BDQ architecture with GAE \(the full derivation and its convergence properties are given in Appendix \([A](https://arxiv.org/html/2606.26574#A1)\)\. As shown in Figure[3](https://arxiv.org/html/2606.26574#S7.F3), an offline memory buffer for the critic stabilizes the estimated advantages at the cost of some bias; remaining unbiased under such a buffer would require importance\-sampling corrections as in V\-Trace\(Espeholtet al\.,[2018](https://arxiv.org/html/2606.26574#bib.bib53); Singh and Sutton,[1996](https://arxiv.org/html/2606.26574#bib.bib52)\)\.

#### 5\.2\.1\.VDN\-PPO: Additive Credit Assignment

Under the additive decomposition \([2](https://arxiv.org/html/2606.26574#S5.E2)\), VDN\-PPO realizes theWh=whα​\(s\)W\_\{h\}=w\_\{h\}^\{\\alpha\}\(s\)entry of the QS weighting table: it assigns credit to each head through an*importance\-weighted*GAE\. We measure a head’s importance by the range of its advantage function and form the corresponding annealed weight,

\(5\)Ih​\(s\)=maxa⁡Aϕ,h​\(s,a\)−mina⁡Aϕ,h​\(s,a\),whα​\(s\)=Ih​\(s\)α∑k=1HIk​\(s\)α,I\_\{h\}\(s\)=\\max\_\{a\}A\_\{\\phi,h\}\(s,a\)\-\\min\_\{a\}A\_\{\\phi,h\}\(s,a\),\\qquad w\_\{h\}^\{\\alpha\}\(s\)=\\frac\{I\_\{h\}\(s\)^\{\\alpha\}\}\{\\sum\_\{k=1\}^\{H\}I\_\{k\}\(s\)^\{\\alpha\}\},whereα∈\[0,1\]\\alpha\\in\[0,1\]is annealed during training and the convention00=10^\{0\}=1yields uniform weightswh0=1/Hw\_\{h\}^\{0\}=1/Hatα=0\\alpha=0\. The importance\-weighted GAE for headhhis

\(6\)A^t\(h\)=∑l=0T−t−1\(γ​λ\)l​whα​\(st\+l\)⋅δt\+lV,\\hat\{A\}\_\{t\}^\{\(h\)\}=\\sum\_\{l=0\}^\{T\-t\-1\}\(\\gamma\\lambda\)^\{l\}\\;w\_\{h\}^\{\\alpha\}\(s\_\{t\+l\}\)\\cdot\\delta\_\{t\+l\}^\{V\},and the weightswhαw\_\{h\}^\{\\alpha\}are treated as stop\-gradient constants, so no gradient flows through them into the policy parametersθ\\theta\. Becausewhα​\(s\)w\_\{h\}^\{\\alpha\}\(s\)depends on the state alone, this design admits three formal guarantees, proved in Appendix[A](https://arxiv.org/html/2606.26574#A1):

1. \(1\)Unbiasedness \(Appendix Theorem[A\.9](https://arxiv.org/html/2606.26574#A1.Thmtheorem9)\)\.Sincewhα​\(st\+l\)w\_\{h\}^\{\\alpha\}\(s\_\{t\+l\}\)isσ​\(st\+l\)\\sigma\(s\_\{t\+l\}\)\-measurable, it is conditionally independent of the sampled action and factors out of the inner expectation,𝔼​\[whα​\(st\+l\)​δt\+lV∣st\+l,𝐚t\+l\]=whα​\(st\+l\)​𝔼​\[δt\+lV∣st\+l,𝐚t\+l\]\\mathbb\{E\}\[w\_\{h\}^\{\\alpha\}\(s\_\{t\+l\}\)\\,\\delta\_\{t\+l\}^\{V\}\\mid s\_\{t\+l\},\\mathbf\{a\}\_\{t\+l\}\]=w\_\{h\}^\{\\alpha\}\(s\_\{t\+l\}\)\\,\\mathbb\{E\}\[\\delta\_\{t\+l\}^\{V\}\\mid s\_\{t\+l\},\\mathbf\{a\}\_\{t\+l\}\]\. Under a converged critic \(Vϕ=VπV\_\{\\phi\}=V^\{\\pi\}\) the inner expectation equals the true advantageAπ​\(st\+l,𝐚t\+l\)A^\{\\pi\}\(s\_\{t\+l\},\\mathbf\{a\}\_\{t\+l\}\), soA^t\(h\)\\hat\{A\}\_\{t\}^\{\(h\)\}is unbiased for the importance\-weighted advantage along the trajectory\.
2. \(2\)Global gradient recovery \(Corollary[A\.10](https://arxiv.org/html/2606.26574#A1.Thmtheorem10)\)\.Because∑h=1Hwhα​\(s\)=1\\sum\_\{h=1\}^\{H\}w\_\{h\}^\{\\alpha\}\(s\)=1identically, summing the per\-head estimates recovers standard GAE,∑hA^t\(h\)=A^tGAE\\sum\_\{h\}\\hat\{A\}\_\{t\}^\{\(h\)\}=\\hat\{A\}\_\{t\}^\{\\mathrm\{GAE\}\}\. The joint policy gradient therefore remains unbiased for every value ofα\\alpha\.
3. \(3\)Variance redistribution \(Proposition[A\.12](https://arxiv.org/html/2606.26574#A1.Thmtheorem12)\)\.WhenIh​\(s\)\>0I\_\{h\}\(s\)\>0for allhh\(generically true after exploration\), low\-importance heads satisfy\(whα\)2<1/H2\(w\_\{h\}^\{\\alpha\}\)^\{2\}<1/H^\{2\}, strictly reducing their variance contribution, while heads withIh=0I\_\{h\}=0receivewh=0w\_\{h\}=0and inject no noise at all\. The freed variance budget is reallocated to high\-importance heads where signal dominates noise\.

Operationally, the annealing schedule interpolates between these regimes: early in trainingα≈0\\alpha\\approx 0gives uniform credit \(standard GAE\), and as the critic convergesα→1\\alpha\\to 1concentrates credit on the heads whose sub\-actions most affect theQQ\-value, suppressing pure noise from heads with no agency at the current state\.

#### 5\.2\.2\.PPO\-MIX: Monotonic Non\-Linear Advantage Mixing

The additive structure \([2](https://arxiv.org/html/2606.26574#S5.E2)\) assumes each head contributes to the joint advantage independently\. When sub\-actions are strictly coupled \(e\.g\., aiming and firing\), such interactions may be unrepresentable\. FACMAC\(Penget al\.,[2021](https://arxiv.org/html/2606.26574#bib.bib55)\)shows that non\-monotonic factorization can represent tasks monotonic methods cannot; PPO\-MIX instead retains monotonic mixing to preserve the IGM property and the tractability of per\-head greedy action selection, trading representational capacity for that guarantee\. It replaces the linear sum with the QPLEX\-style mixer \(Wh=∂fmix/∂AhW\_\{h\}=\\partial f\_\{\\text\{mix\}\}/\\partial A\_\{h\}in the QS table\),

\(7\)Qϕ​\(s,𝐚\)=Vϕ​\(s\)\+fmix​\(Aϕ,1​\(s,a1\),…,Aϕ,H​\(s,aH\)∣s\),∂fmix∂Ah≥0,Q\_\{\\phi\}\(s,\\mathbf\{a\}\)=V\_\{\\phi\}\(s\)\+f\_\{\\text\{mix\}\}\\left\(A\_\{\\phi,1\}\(s,a\_\{1\}\),\\dots,A\_\{\\phi,H\}\(s,a\_\{H\}\)\\mid s\\right\),\\quad\\frac\{\\partial f\_\{\\text\{mix\}\}\}\{\\partial A\_\{h\}\}\\geq 0,wherefmixf\_\{\\text\{mix\}\}is realized by a state\-conditioned hypernetwork with strictly non\-negative weights, guaranteeing monotonicity and hence IGM\.

Credit assignment now requires the gradient of the mixed output at the*current joint action*\. A first\-order Taylor expansion offmixf\_\{\\text\{mix\}\}gives the per\-head credit

\(8\)w~h​\(s,𝐚\)=∂fmix∂Ah\|𝐚⋅Aϕ,h​\(s,ah\),whmix​\(s,𝐚\)=w~h∑kw~k\.\\tilde\{w\}\_\{h\}\(s,\\mathbf\{a\}\)=\\frac\{\\partial f\_\{\\text\{mix\}\}\}\{\\partial A\_\{h\}\}\\bigg\|\_\{\\mathbf\{a\}\}\\\!\\cdot A\_\{\\phi,h\}\(s,a\_\{h\}\),\\qquad w\_\{h\}^\{\\text\{mix\}\}\(s,\\mathbf\{a\}\)=\\frac\{\\tilde\{w\}\_\{h\}\}\{\\sum\_\{k\}\\tilde\{w\}\_\{k\}\}\.Evaluating∂fmix/∂Ah\\partial f\_\{\\text\{mix\}\}/\\partial A\_\{h\}at the sampled action is*necessary*: the mixer is non\-linear in the advantages, so the local gradient depends on where in advantage\-space the joint action falls\. A state\-only surrogate, evaluating the gradient at the advantage means, which are zero by construction—would collapse to a degenerate zero weight\. COMA\(Foersteret al\.,[2018](https://arxiv.org/html/2606.26574#bib.bib25)\)compares the zero value to the value at the action as a finite difference instead of the Taylor approximation that we employ\.

##### Variance cost of action\-dependent weights\.

This action\-dependence breaks the factorization that underwrites VDN\-PPO’s unbiasedness\. Marginalizing over future actions at stept\+lt\+l, the expectation of the weighted residual no longer decomposes \(conditioning onst\+ls\_\{t\+l\}left implicit\):

\(9\)𝔼𝐚t\+l​\[whmix​\(st\+l,𝐚t\+l\)​δt\+lV\]=𝔼​\[whmix\]​𝔼​\[δt\+lV\]\+Cov​\(whmix,δt\+lV\)⏟≠0\.\\mathbb\{E\}\_\{\\mathbf\{a\}\_\{t\+l\}\}\\left\[w\_\{h\}^\{\\text\{mix\}\}\(s\_\{t\+l\},\\mathbf\{a\}\_\{t\+l\}\)\\;\\delta\_\{t\+l\}^\{V\}\\right\]=\\mathbb\{E\}\[w\_\{h\}^\{\\text\{mix\}\}\]\\,\\mathbb\{E\}\[\\delta\_\{t\+l\}^\{V\}\]\+\\underbrace\{\\mathrm\{Cov\}\\left\(w\_\{h\}^\{\\text\{mix\}\},\\;\\delta\_\{t\+l\}^\{V\}\\right\)\}\_\{\\neq\\,0\}\.Becausewhmixw\_\{h\}^\{\\text\{mix\}\}andδt\+lV\\delta\_\{t\+l\}^\{V\}share the sampled action𝐚t\+l\\mathbf\{a\}\_\{t\+l\}—actions producing large advantages also inflate the mixing gradient—the covariance is generically non\-zero\. Two consequences follow: \(i\) the per\-head estimates no longer sum to standard GAE, since∑h∂fmix/∂Ah≠1\\sum\_\{h\}\\partial f\_\{\\text\{mix\}\}/\\partial A\_\{h\}\\neq 1for a non\-linear mixer, so the requirements for policy gradient validity in Theorem[A\.11](https://arxiv.org/html/2606.26574#A1.Thmtheorem11)are not met, and \(ii\) the residual covariance injects additional variance into every head’s estimate beyond what VDN weighting produces\. In short, PPO\-MIX trades the theoretical guarantees of VDN\-PPO for the representational capacity needed to model strongly coupled sub\-actions\. Section[7\.1](https://arxiv.org/html/2606.26574#S7.SS1)shows that PPO\-MIX can learn state\-dependent importance, and in practice it outperforms VDN slightly for continuous action spaces while matching it for discrete actions\.

##### Continuous Action Advantage Centering\.

The mechanism of computing bin masses via the CDF is theoretically necessary to preserve the on\-policy value baselineVϕ​\(s\)V\_\{\\phi\}\(s\)in continuous domains\. By definition, the value function and advantages must satisfyV​\(s\)=𝔼𝐚​\[Q​\(s,𝐚\)\]V\(s\)=\\mathbb\{E\}\_\{\\mathbf\{a\}\}\[Q\(s,\\mathbf\{a\}\)\], which requires the expected advantage to be strictly zero\. In discrete action spaces, this constraint is trivially satisfied by evaluating the sumV​\(s\)=V​\(s\)\+∑aπ​\(a\|s\)​A​\(s,a\)V\(s\)=V\(s\)\+\\sum\_\{a\}\\pi\(a\|s\)A\(s,a\), allowing us to center the advantages by scaling each discrete action’s advantage by its corresponding policy probabilityπa\\pi\_\{a\}\. However, for continuous actions, this discrete summation is mathematically invalid and the expectation instead demands taking the integral over the action dimension,∫π​\(a\|s\)​A​\(s,a\)​𝑑a=0\\int\\pi\(a\|s\)A\(s,a\)\\,da=0\. By discretizing the continuous advantage function into intervals and evaluating the Cumulative Distribution Function at the bin boundaries, we analytically integrate the policy’s probability density over each segment to find the exact probability massmh,cm\_\{h,c\}\. Using this mass as the proportion by which to scale the bin’s advantage acts as a rigorous, tractable approximation of the continuous integral\. This ensures the continuous advantage streams are correctly zero\-centered under the current policy without introducing the high variance that would result from relying on Monte Carlo action sampling to approximate the expectation\.

### 5\.3\.Hybrid SAC

We adapt SAC to complex action spaces with two variants, both drawn from the factorization wiring diagram\. For monolithic evaluation \(SAC\-Concat\)\(Delalleauet al\.,[2019](https://arxiv.org/html/2606.26574#bib.bib17)\), the state and both sub\-actions are concatenated into the standard critic, using a Gumbel\-Softmax\(Janget al\.,[2016](https://arxiv.org/html/2606.26574#bib.bib26)\)relaxation to keep the discrete branch differentiable\. Credit assignment through the deep gradient is natural here, but the entropy signals must be separated: a single target entropy drives one distribution to collapse while the other explodes, so we maintain distinct temperature coefficientsαd,αc\\alpha\_\{d\},\\alpha\_\{c\}for the discrete and continuous dimensions as in\(Delalleauet al\.,[2019](https://arxiv.org/html/2606.26574#bib.bib17)\)\. Alternatively,SAC\-BDQ\(hybrid\-sac\(Chenet al\.,[2022](https://arxiv.org/html/2606.26574#bib.bib43)\)\+ branching dueling\) routes discrete sub\-actions into the QS critic, drawing them from a softmax over advantages as in Munchausen Q\-learning\(Vieillardet al\.,[2020](https://arxiv.org/html/2606.26574#bib.bib7)\)—analogous to P\-DQN\(Xionget al\.,[2018](https://arxiv.org/html/2606.26574#bib.bib30)\)under entropy constraints—so that the branching critic itself performs the discrete credit assignment\. SAC\-BDQ reduces to Discrete\-SAC for the discrete action space\.

## 6\.Experimental Setup

On each of the environments shown in Figure[1](https://arxiv.org/html/2606.26574#S4.F1), we analyze Discrete, Hybrid and continuous action types\. For the hybrid action configurations, the firstDDactions are discrete and the followingCCactions are continuous\.

Table 1\.Action space factorizations per environment\.𝒟k\\mathcal\{D\}\_\{k\}denotes a discrete action head of cardinalitykk, andℝn\\mathbb\{R\}^\{n\}denotes a continuous action head ofnndimensions\.### 6\.1\.Hyperparameter Configuration

Across all evaluated environments, we established a standardized set of core hyperparameters for Proximal Policy Optimization \(PPO\), Soft Actor\-Critic \(SAC\), and Deep Q\-Networks \(DQN\)\. The neural network architectures utilized two hidden layers of 256 units each for thePush,Shoot, andPlatformtasks, while theLandertasks utilized 128 units per layer\. A uniform learning rate of5\.0×10−45\.0\\times 10^\{\-4\}was maintained across all algorithms and environments\. The discount factor \(γ\\gamma\) was set to0\.980\.98for allPushtasks and0\.990\.99for the remaining environments\. Due to differing sample efficiencies and task complexities, the total maximum environment steps \(max\_steps\) varied by environment based on the number of steps required for the canonical versions of each algorithm to converge \(Discrete PPO, BD\-DQN, SAC\-Concat\)\. Notably, SAC and DQN required identical step counts for convergence across all configurations, while PPO consistently required higher sample limits \(offset computationally by parallel environment execution where collection time becomes 1/20th runtime for all algorithms\), as summarized in Table[2](https://arxiv.org/html/2606.26574#S6.T2)\.

Table 2\.Total Environment Interactions \(max\_steps\) by Task and Algorithm
### 6\.2\.Contextual Decomposition Verification

The Contextual\-Decoupler of Section[4](https://arxiv.org/html/2606.26574#S4)is a contextual bandit \(γ\\gamma\-discounted but i\.i\.d\. in state\) whose ground\-truth per\-head credit assignment is known in closed form: the active headh=ch=cdrives the dominant±1\\pm 1reward while the inactive headh≠ch\\neq ccontributes only a−0\.1\-0\.1shaping penalty, so the true importance isIh⋆​\(s\)=𝕀​\[h=c\]I\_\{h\}^\{\\star\}\(s\)=\\mathbb\{I\}\[h=c\]and the active head’s±1\\pm 1variance acts as pure noise on the inactive head’s policy\-gradient signal\. This is precisely the regime in which Proposition[A\.12](https://arxiv.org/html/2606.26574#A1.Thmtheorem12)\(Appendix[A](https://arxiv.org/html/2606.26574#A1)\) predicts a benefit from importance reweighting, because the uniform estimator \(α=0\\alpha=0\) propagates the active head’sδV\\delta^\{V\}variance onto the inactive head undiminished\. Because the task isolates pure algorithmic capability rather than environment\-specific control, it is not included in the “practical” sweep of Table[2](https://arxiv.org/html/2606.26574#S6.T2); instead we run two targeted studies onN=5N=5actions per head \(𝒟2×𝒟5×𝒟5\\mathcal\{D\}\_\{2\}\\times\\mathcal\{D\}\_\{5\}\\times\\mathcal\{D\}\_\{5\}\) over10510^\{5\}environment steps\.

##### Cross\-family reward sweep\.

To analyze the impact of critic factorization performance across*all*three algorithm families, we run every pairing of \{PPO, SAC, DQN\} with critic structure \{shared\-nomix, VDN, Q\-PLEX\}\. SAC is evaluated in itsSAC\-BDQ\(QQ\-critic\) andSAC\-Concat\(VV\-critic\) forms\. All agents collect10510^\{5\}steps and we report the smoothed vectorized reward curve over training \(Figure[2](https://arxiv.org/html/2606.26574#S7.F2)\)\.

##### Factorial PPO study\.

To isolate the*magnitude*of each source of gain within PPO and to test Proposition[A\.12](https://arxiv.org/html/2606.26574#A1.Thmtheorem12)directly, we run a full factorial over critic structure and importance estimator\. The critic is one ofnomix\(a single scalar value baseline, no per\-head decomposition\),VDN\(additive mixer∑hAϕ,h\\sum\_\{h\}A\_\{\\phi,h\}, \([2](https://arxiv.org/html/2606.26574#S5.E2)\)\), orQ\-PLEX\(the state\-conditioned monotonic hypernetwork mixer of \([7](https://arxiv.org/html/2606.26574#S5.E7)\)\)\. The importance estimator is one of:

- •uniform\(α≡0\\alpha\\equiv 0\): weights pinned towh=1/Hw\_\{h\}=1/H, disabling reweighting; this isolates the critic\-factorization effect\.
- •grad: the first\-order mixer\-gradient surrogatewh∝\|Aϕ,h​\(s,ah\)⋅∂AhQtot\|αw\_\{h\}\\propto\|A\_\{\\phi,h\}\(s,a\_\{h\}\)\\cdot\\partial\_\{A\_\{h\}\}Q\_\{\\text\{tot\}\}\|^\{\\alpha\}of \([8](https://arxiv.org/html/2606.26574#S5.E8)\), evaluated at the observed action\.
- •range: the exact range\-based importance of \([5](https://arxiv.org/html/2606.26574#S5.E5)\),wh∝Ih​\(s\)αw\_\{h\}\\propto I\_\{h\}\(s\)^\{\\alpha\}withIh​\(s\)=maxa⁡Aϕ,h−mina⁡Aϕ,hI\_\{h\}\(s\)=\\max\_\{a\}A\_\{\\phi,h\}\-\\min\_\{a\}A\_\{\\phi,h\}\.

Forgradandrangethe annealing parameter is ramped linearlyα:0→1\\alpha\\\!:0\\to 1over the first4040updates, reaching its terminal value well within the≈48\\approx 48updates of a run\. All agents use PPO with44epochs, minibatch size128128, learning rate10−310^\{\-3\}, a\[64,64\]\[64,64\]trunk, and \(for Q\-PLEX\) a6464\-dimensional mixing embedding\. We collect20482048\-transition rollouts across1616synchronous environments and reportn=16n=16random seeds per cell\. We track three learning\-dynamics metrics and one direct variance metric:

- •Final reward: mean smoothed episode return over the last5050episodes\.
- •AUC: mean of the smoothed return curve over all of training \(a sample\-efficiency proxy\)\.
- •Steps→\\to50: environment steps to first reach a smoothed return of5050\(lower is better; censored at10510^\{5\}\)\.
- •inact/act ratio: the empirical variance of the*inactive*head’s importance\-weighted GAE advantageA^\(h\)\\hat\{A\}^\{\(h\)\}relative to that of the*active*head, measured over the final third of updates\. This is the direct empirical analogue of the\(whα\)2\(w\_\{h\}^\{\\alpha\}\)^\{2\}scaling in Proposition[A\.12](https://arxiv.org/html/2606.26574#A1.Thmtheorem12)\.

We additionally score the learned weightswhα​\(s\)w\_\{h\}^\{\\alpha\}\(s\)against the ground\-truth importanceIh⋆​\(s\)I\_\{h\}^\{\\star\}\(s\)to test whether they recover the active head \(Figure[3](https://arxiv.org/html/2606.26574#S7.F3)\)\.

## 7\.Results and Analysis

### 7\.1\.Branching Critics on the Contextual Decoupler

The Contextual\-Decoupler \(Section[4](https://arxiv.org/html/2606.26574#S4), design in Section[6\.2](https://arxiv.org/html/2606.26574#S6.SS2)\) lets us decompose the performance of branching critics into separable sources, because the ground\-truth credit assignmentIh⋆​\(s\)=𝕀​\[h=c\]I\_\{h\}^\{\\star\}\(s\)=\\mathbb\{I\}\[h=c\]is known in closed form\. We report three studies: a cross\-family reward sweep \(Figure[2](https://arxiv.org/html/2606.26574#S7.F2)\) showing that critic factorization helps every algorithm family, a factorial PPO study \(Tables[3](https://arxiv.org/html/2606.26574#S7.T3)\-[4](https://arxiv.org/html/2606.26574#S7.T4)\) isolating the magnitude of each source within PPO, and an importance\-recovery probe \(Figure[3](https://arxiv.org/html/2606.26574#S7.F3)\) confirming the range weights track the true active head\.

##### Critic factorization helps every algorithm family\.

Figure[2](https://arxiv.org/html/2606.26574#S7.F2)sweeps \{PPO, SAC, DQN\} against \{shared\-nomix, VDN, Q\-PLEX\} critics\. The factorization effect is visible across all three families\.SAC\-BDQroutes the discrete heads through the branching dueling critic and solves the task almost immediately, reaching a near\-perfect score within a few thousand steps\. All threeDQNvariants plateau around7575, capped not by the critic but by the residualϵ\\epsilon\-greedy exploration actions that the on\-policy reward curve still pays for\.VDN\-PPO and PPO\-MIX \(Q\-PLEX\)converge to roughly8080by the end of training, whereas the unfactoredPPO shared\-nomixbaseline crawls to only∼15\\sim 15: with a single scalar value, the inactive head’s gradient is dominated by the active head’s±1\\pm 1fluctuation and the weak−0\.1\-0\.1shaping signal is never recovered within the budget\.SAC\-Concatreaches only∼50\\sim 50; SAC requires a very low target entropy on this task and is sensitive to its setting, since a single temperature drives one head’s distribution to collapse while the other explodes \(§[5\.2](https://arxiv.org/html/2606.26574#S5.SS2)\), the engineering motivation for the separateαd,αc\\alpha\_\{d\},\\alpha\_\{c\}coefficients of SAC\-Concat \(as in the original paper\(Delalleauet al\.,[2019](https://arxiv.org/html/2606.26574#bib.bib17)\)\)\.

![Refer to caption](https://arxiv.org/html/2606.26574v1/XoRcheck.png)Figure 2\.Vectorized reward curves on the Contextual\-Decoupler \(10510^\{5\}steps\) for each algorithm family paired with a shared\-nomix, VDN, or Q\-PLEX critic\. SAC\-BDQ \(QQ\-critic\) solves the task almost immediately; the DQN variants plateau near7575\(held back byϵ\\epsilon\-greedy exploration actions\); VDN\-PPO and PPO\-MIX \(Q\-PLEX\) converge to∼80\\sim 80while the unfactored PPO shared\-nomix baseline reaches only∼15\\sim 15\. SAC\-Concat \(VV\-critic\) reaches∼50\\sim 50and is sensitive to the target entropy\. \(5 seeds; shaded region: SEM\.\)Table[3](https://arxiv.org/html/2606.26574#S7.T3)runs the factorial over critic structure and importance estimator \(Section[6\.2](https://arxiv.org/html/2606.26574#S6.SS2)\)\. Replacing the scalar baseline \(nomix\) with any factored critic raises the final reward from6\.76\.7to8282\-8989: the gap is highly significant on every metric \(nomix vs\. uniform mixers gives Welcht≈28t\\approx 28\-4040,p<10−14p<10^\{\-14\}, Cohen’sd≈11d\\approx 11–1414\), and nomix never reaches the threshold of5050within budget\. The factored critic absorbs the action\-dependent return into the per\-head advantage streams, soVϕV\_\{\\phi\}fits the clean state value and each head receives an advantage scoped to its own action, theVV\-only mechanism of \([4](https://arxiv.org/html/2606.26574#S5.E4)\) \(Appendix[A](https://arxiv.org/html/2606.26574#A1)\)\.

Table 3\.Aggregate results on the Contextual\-Decoupler \(n=16n=16seeds, mean±\\pmstd\)\. Final reward and AUC: higher is better; Steps→\\to50: lower is better\. The inact/act ratio is the inactive\-head advantage variance relative to the active head; values below11indicate the variance suppression predicted by Proposition[A\.12](https://arxiv.org/html/2606.26574#A1.Thmtheorem12)\(Appendix[A](https://arxiv.org/html/2606.26574#A1)\)\.
##### Importance weighting reduces inactive\-head variance exactly as predicted\.

Atα=0\\alpha=0the inactive and active\-head advantages are identical, so the ratio is exactly1\.0001\.000\. Withα→1\\alpha\\to 1the inactive\-head variance is suppressed to0\.830\.83–0\.86×0\.86\\timesthe active head’s \(Table[3](https://arxiv.org/html/2606.26574#S7.T3)\); a paired within\-run one\-sided test confirms the suppression for every weighted configuration \(t∈\[−25,−11\]t\\in\[\-25,\-11\],p<10−8p<10^\{\-8\}\)\. This is a direct empirical confirmation of the\(whα\)2\(w\_\{h\}^\{\\alpha\}\)^\{2\}variance scaling in Proposition[A\.12](https://arxiv.org/html/2606.26574#A1.Thmtheorem12)\(Appendix[A](https://arxiv.org/html/2606.26574#A1)\)\. Because the weights sum to one and are state\-measurable, reweighting is unbiased \(Corollary[A\.10](https://arxiv.org/html/2606.26574#A1.Thmtheorem10)\), so we expect it to improve*dynamics*rather than the asymptotic optimum\. Table[4](https://arxiv.org/html/2606.26574#S7.T4)confirms this: the uniform→\\toweighted effect is consistent in sign across all three metrics and both mixers: higher final reward, higher AUC, fewer steps to threshold, reaching significance in most cells and most strongly for Q\-PLEX \(final\+3\.9\+3\.9,p=10−5p=10^\{\-5\},d=1\.9d=1\.9\)\. Magnitudes are modest \(∼3\\sim 3–44reward,∼4\\sim 4–6,0006\{,\}000fewer steps, i\.e\.∼4%\\sim 4\\%reward and∼8\\sim 8–10%10\\%sample efficiency\), second\-order relative to the≈77\\approx 77\-point factorization effect, but robust\. Weighting also tightens seed\-to\-seed spread \(VDN final std4\.5→2\.54\.5\\to 2\.5; Q\-PLEX2\.4→1\.52\.4\\to 1\.5\), an outer\-level stability gain from the same mechanism\.

Table 4\.Significance of the variance\-reduction effect: uniform \(α=0\\alpha=0\) vs\. weighted estimators \(n=16n=16, Welch’s two\-sidedtt\)\.Δ\\Deltais \(weighted−\-uniform\); for Steps→\\to50 a negativeΔ\\Deltais an improvement in the number of steps taken to get to a score of 50 compared to uniform weighting\.ddis Cohen’sdd\. Significance:∗\\astp<0\.05p<0\.05,∗⁣∗\\ast\\astp<0\.01p<0\.01,∗⁣∗⁣∗\\ast\\ast\\astp<0\.001p<0\.001\.
##### The exact and first\-order estimators are equivalent for control, but only the exact one is interpretable\.

Thegrad\(first\-order, \([8](https://arxiv.org/html/2606.26574#S5.E8)\)\) andrange\(exact, \([5](https://arxiv.org/html/2606.26574#S5.E5)\)\) weights are statistically indistinguishable for learning: VDN slightly favoursrangeand Q\-PLEX slightly favoursgradon final reward \(\|t\|<2\.5\|t\|<2\.5\), with no consistent winner\. Both deliver the full variance\-reduction benefit\. We nonetheless recommendrangeas the default, since it doubles as a faithful interpretability probe: as shown in Figure[3](https://arxiv.org/html/2606.26574#S7.F3), its weights track the ground\-truth active head over training, withr≈0\.97r\\approx 0\.97correlation toIh⋆I\_\{h\}^\{\\star\}, whereas the first\-order surrogate is not a reliable estimate of which head is responsible despite equal learning performance\. Figure[3](https://arxiv.org/html/2606.26574#S7.F3)also shows that giving the critic an offline buffer \(breaking the on\-policy requirement of Theorem[A\.9](https://arxiv.org/html/2606.26574#A1.Thmtheorem9)and incurring the bias bounded by Proposition[A\.13](https://arxiv.org/html/2606.26574#A1.Thmtheorem13)in Appendix[A](https://arxiv.org/html/2606.26574#A1)\) can improve weight correlation, but in practice we found the added bias damages the policy gradient too greatly and leads to degraded performance\. For this reason and theoretical correctness we use the online buffer for all experiments going forward\.

![Refer to caption](https://arxiv.org/html/2606.26574v1/importance.png)

![Refer to caption](https://arxiv.org/html/2606.26574v1/offlineReward.png)

Figure 3\.Orange is Correlation between the true state’s active headIh⋆I\_\{h\}^\{\\star\}and the internal range\-based importance weighting over training\. Blue is accuracy \(percent of the time the more important head was correctly chosen\) Both VDN and QPLEX progress towards true importance\. Giving the critic an offline buffer adds theoretical bias \(Proposition[A\.13](https://arxiv.org/html/2606.26574#A1.Thmtheorem13), Appendix[A](https://arxiv.org/html/2606.26574#A1)\) leading to a worse env reward, but it but improves importance estimation\. \(5 seeds; shaded region: SEM\.\) We use the online\-only critic throughout this paper’s other experiments for it’s empirical performance and theoretical correctness, but future work may investigate dual critics with the offline critic handling importance while the online critic supplies the GAE baseline\.
##### Summary of sources and magnitudes\.

The environment cleanly separates two contributions, ordered by magnitude: \(i\)Policy Gradient*critic factorization*is the headline effect \(\+77\+77reward,d≈11d\\approx 11\), arising because the per\-head advantage streams absorb action\-dependent return and letVϕV\_\{\\phi\}trackVπV^\{\\pi\}; \(ii\)*importance\-weighted variance reduction*is a statistically robust second\-order refinement to suppress inactive\-head estimator variance∼15%\\sim 15\\%\(p<10−8p<10^\{\-8\}\), translating to∼4%\\sim 4\\%higher reward,∼8\\sim 8–10%10\\%better sample efficiency, and reduced seed variance, all without changing the asymptotic optimum as Corollary[A\.10](https://arxiv.org/html/2606.26574#A1.Thmtheorem10)requires\. PPO\-MIX retains state\-dependent importance but pays the action\-dependent covariance cost of \([9](https://arxiv.org/html/2606.26574#S5.E9)\), which is why it tracks rather than beats VDN\-PPO here\.

### 7\.2\.Summary Results

Grouping by \{env, algorithm, action dtype\} results in 45 plots with up to 5 factorizations each\. For completeness, we include each individual result in AppendixLABEL:app:appendixD, but we sum over environments to isolate the global impacts on algorithm \(PPO, DQN, SAC\) and Datatype \(Discrete, Hybrid, Continuous\) over our benchmark suite\. We perform min\-max normalization on a per\-environment basis so that the highest and lowest score achieved by each algorithm, any seed, represent the Y\-axis scale \[0\.0,1\.0\] for all pairings\. We used BD\-DQN, Discrete PPO, and SAC to determine the number of steps required to converge on each environment for each family of algorithm\. Steps were tuned for each algorithm because this paper is not intended to choose between DQN, PPO and SAC, but rather to compare the impact of factorization on each\. By treating the X axis as steps from \[0\.0,1\.0\] and interpolating, we report the mean normalized performance per normalized env step for pairing in Figure[4](https://arxiv.org/html/2606.26574#S7.F4)over 5 seeds per experiment for a total of 1,100 runs\.

![Refer to caption](https://arxiv.org/html/2606.26574v1/vectorized_graphs/model_action_grid.png)

Figure 4\.Mean results over all environments, grouped by action datatype \(columns\) and algorithm family \(rows\)\. Note that hybrid actions were given longer to train on some environments to account for gradient interference:

#### 7\.2\.1\.PPO

For continuous action spaces, the Monotonic critic seems to slightly benefit PPO, but ”branching” PPO otherwise reduces to the standard continuous implementation\. The improvement is not substantial given extra implementation effort\. For discrete actions, however, we show that two reductions in variance strongly improve performance over baseline implementations of PPO, including IPPO\. Finally, Joint action factorization lags behind the others, likely because our action spaces fall mostly between 300 and 6,000 choices\. All 3 families get relatively similar Joint performance, probably limited by encoder functional capacity at 64 or 256 neurons feeding into hundreds of actions\.

#### 7\.2\.2\.SAC

Pure continuous action SAC outperforms all other factorizations in this environment sweep, while discrete SAC at∼\\sim0\.5 underperforms\. In the hybrid regime, we find that parameterized / SAC\-BDQ outperforms the standard concatenation\.

#### 7\.2\.3\.DQN

Independent networks and VDN/BDQ\-DQN style factorizations consistently perform well here, but QPLEX does not scale well to continuous actions relative to discrete ones\. Independent networks scale in the number of total parameters with the action space size, so they perform consistently across environments\.

### 7\.3\.Notable Environment Results

The complete set of environment training curves are included as supplementary material, but we have chosen a few to highlight here as either instructive or surprising examples\.

![Refer to caption](https://arxiv.org/html/2606.26574v1/vectorized_graphs/hybrid_independent_shoot_sac.png)\(a\)Independent Shoot Results SAC continuous gets∼8\\sim 8
![Refer to caption](https://arxiv.org/html/2606.26574v1/vectorized_graphs/discrete_dependent_shoot_sac.png)\(b\)Dependent Shoot Results SAC \(Continuous gets∼35\\sim 35

Figure 5\.The discrete and hybrid shoot environments are SAC’s worst performance because 300 action choices combined with a discrete entropy target allows auto\-tunedα\\alphato overpower the learning signal, especially because hitting a target reduces the number of optimal actions \(and therefore entropy reward\)As shown in Figure[5](https://arxiv.org/html/2606.26574#S7.F5), discrete actions in the shoot environment perform poorly under SAC\. We identify that this performance is closely tied to the target entropy formulation,ℋt​a​r​g​e​t=ϵH​log⁡\(\|Ad\|\)\\mathcal\{H\}\_\{target\}=\\epsilon\_\{H\}\\log\(\|A\_\{d\}\|\), whereϵH\\epsilon\_\{H\}is the target entropy percentage hyperparameter\. WhileϵH=0\.5\\epsilon\_\{H\}=0\.5is effective across standard environments, the shoot environment’s large action space and low\-entropy optimal policy necessitate a lowerϵH\\epsilon\_\{H\}of 0\.1 or 0\.2; otherwise, the entropy signal dominates the extrinsic environment reward\.

We hypothesize that this performance degradation occurs because the soft exploration objective is fundamentally misaligned with the environment’s task dynamics\. At the beginning of an episode, three active targets exist, meaning the optimal policy distributes probability mass across a corresponding subset of the valid target locations and jammer choices\. As targets are successfully hit, the number of plausible optimal actions strictly decreases\. Consequently, the maximum possible entropy of the optimal policy must monotonically decrease as the episode progresses toward completion\. Likewise, the discrete tuner usesπ​\(s,a\)=eQ​\(s,a\)/α∑beQ​\(s,b\)/α\\pi\(s,a\)=\\frac\{e^\{Q\(s,a\)/\\alpha\}\}\{\\sum\_\{b\}e^\{Q\(s,b\)/\\alpha\}\}to generate actions\. This means that BDQ mode can directly force the entropy to meet the target while Concat mode uses reward to apply feedback to the discrete actor network\. Finally, continuous actions do not scale with action granularity so the target entropy is less important\.

Because SAC’s soft objective formulation rewards the agent for occupying states that support higher\-entropy policies, it inherently penalizes the transition into these progressed, lower\-entropy states\. Furthermore, because the observation remains static unless a target is successfully hit, taking sub\-optimal exploratory actions does not yield state transitions\. Advancing the task requires a strictly less stochastic policy, placing the environment’s underlying goal in direct mathematical conflict with SAC’s soft exploration mechanism\.

![Refer to caption](https://arxiv.org/html/2606.26574v1/vectorized_graphs/dtypes.png)Figure 6\.All datatype results aggregated by algorithm\. SAC performs best with native continuous actions while PPO and DQN perform best with discrete, but the differences are small relative to factorization \(Hybrid environments that got extra steps during training were truncated to make the comparison fair\)We see in Figures[6](https://arxiv.org/html/2606.26574#S7.F6)and[4](https://arxiv.org/html/2606.26574#S7.F4)that action type has relatively less impact than factorization with the exception of SAC as discussed above\. For off\-policy learners, auto\-regressive action offers the most universal solution to action factorization as it is able model dependence directly while scaling model parameters linearly with the number of actions\. For on\-policy, PPO with a factored critic provides superior performance at a significantly lessened computational cost relative to SAC/DQN and AR\-PPO\. Policy gradient methods are very sensitive to variance, so we hypothesize that the chain of normally distributed continuous action inputs offered more noise than signal\. Conversely, variance reduction is the exact method which improves performance\.

## 8\.Discussion

### 8\.1\.Computational Complexity

The computational footprint of a factorization method is a critical criterion for model selection, governed not only by theoretical operation counts but also by hardware\-level parallelizability\. LetEEdenote the operation cost of the state encoder,ddthe final hidden layer dimension, and\|𝒜h\|\|\\mathcal\{A\}\_\{h\}\|the cardinality of action headh∈\{1,…,H\}h\\in\\\{1,\\dots,H\\\}\.

The theoretical forward\-pass complexities scale as follows:

- •Joint:O​\(E\+d​∏h=1H\|𝒜h\|\)O\\big\(E\+d\\prod\_\{h=1\}^\{H\}\|\\mathcal\{A\}\_\{h\}\|\\big\)\. The final fully connected layer scales exponentially with the number of sub\-actions\.
- •Branching \(Shared Encoder\):O​\(E\+d​∑h=1H\|𝒜h\|\)O\\big\(E\+d\\sum\_\{h=1\}^\{H\}\|\\mathcal\{A\}\_\{h\}\|\\big\)\. The exponential output requirement is reduced to a linear sum by attaching distinct linear heads to a single shared encoder pass\.
- •Independent & Auto\-Regressive \(AR\):O​\(H⋅E\+d​∑h=1H\|𝒜h\|\)O\\big\(H\\cdot E\+d\\sum\_\{h=1\}^\{H\}\|\\mathcal\{A\}\_\{h\}\|\\big\)\. Because each action dimension utilizes a completely distinct network, the base state\-encoding costEEscales linearly withHH\.

However, theoretical complexity does not directly translate to wall\-clock runtime due to architectural sequentiality\. While Independent networks possess the highest total operation count, their decoupled forward passes can be batched and executed simultaneously on modern parallel hardware, effectively reducing inference latency to the speed of the slowest network:maxh⁡O​\(E\+d​\|𝒜h\|\)\\max\_\{h\}O\(E\+d\|\\mathcal\{A\}\_\{h\}\|\)\. Conversely, the chain\-rule dependencies inherent in AR policies strictly prohibit parallel execution; headhhcannot be evaluated until headh−1h\-1completes, forcing a sequential bottleneck bounded by the sum of all operations\.

Furthermore, the dense, monolithic matrix multiplications characteristic of Joint factorization are highly optimized on GPU architectures\. Provided the exponentially scaled joint action space does not exceed VRAM limits, a single massive parallel operation can approximate constant time in practice\. Consequently, despite requiring vastly more theoretical FLOPs, Joint architectures frequently outpace Branching and AR methods in wall\-clock inference speed by avoiding kernel\-launch overhead and sequential routing\.

## 9\.Conclusion

We presented a cross\-sectional evaluation of six action factorization strategies across DQN, PPO, and SAC on discrete, continuous, and hybrid action spaces in over 220 configurations, alongside two novel lightweight benchmarks \(CoopPush, Hybrid\-Shoot\) with tunable inter\-action dependence\. Auto\-regressive actions offer the best action dependence performance across the board, at the cost of O\(N\) latency and total parameters\. Shared encoder architectures offer the best compute to performance ratio in fully observable settings, where VDN in particular is appealing due to its trivial implementation effort\. VDN\-PPO and PPO\-MIX delivers a substantial improvement over shared\-encoder PPO in discrete spaces by redistributing variance to high\-agency heads\. In continuous action spaces the improvement is less significant\. A notable direction for future work is equilibrium selection: both environments contain multiple symmetric optima that differ in exploration difficulty, which may differentially affect methods that select actions and explore using joint spaces compared to factored spaces\. In the case of DQN, it can be seen that monotonic mixing networks can degrade performance as action spaces get larger \(11311^\{3\}for shoot and11811^\{8\}for push\)\. Investigating how these factorization representational capacity and stability interact with asynchronous parallelization frameworks \(IMPALA\(Espeholtet al\.,[2018](https://arxiv.org/html/2606.26574#bib.bib53)\), PQN\(Liet al\.,[2023b](https://arxiv.org/html/2606.26574#bib.bib23)\)\) is another open question\.

## 10\.Practitioner Recommendations

Based on the results across all 220\+ configurations, we offer the following model\-selection recommendations: \(i\)Default to Branching Dueling / VDN, especially for Policy Gradient methodsas it requires minimal implementation overhead and computational cost and it performs competitively across all three algorithm families\. \(ii\)Auto\-Regressiveactions are the most generally performant method if the computational overhead and latency is affordable\. It is an open question whether action ordering has a strong impact in general\. \(iii\)Avoid Q\-PLEX/PPO\-MIXwhen the action space or number of actions grow large\. Despite the highest performance on several configurations, DQN shows clear degradation when moving from smaller discrete action spaces size[4](https://arxiv.org/html/2606.26574#S7.F4)\. Representational capacity or overestimation bias are both potential candidate causes for the degraded ability to accurately assign credit\. It is very possible that scaling the monotonic mixer parameter count with the action space size would alleviate some of the degradation, but the concatenated action space is still exponential in the number of agents so future work is required to study mixing\-scaling at length\. Implementation overhead and added compute may not be worth it with multiple subtle implementation pitfalls \(such as max centering being required for identifiability instead of the more stable mean\-centered approach for dueling implementations, Elu vs Abs optimizer stability, and mixing network size as an additional hyper parameter\)\. \(iv\)Concat vs BDQ SACdepend heavily on the environment which will perform better\. In our test there is not a clear default parameterization for hybrid spaces and the separate continuous and discrete entropy must be handled with care\.

We hope that this article serves two primary purposes\. 1: The introduction and application of a novel variance reduction algorithm for policy gradient methods which trains a branching dueling critic on value\-only targets with sub\-action GAE importance weighting to reduce monte carlo return variance\. 2: A Cross\-sectional comparison over standard \(AR, Concat\-SAC, BD\-DQN\) and lesser known \(BDQ/P\-SAC, Single\-Agent QPLEX, and AR\-Hybrid\-PPO\) factorization strategies in a minimal but principled benchmark suite of dependent action environments\. Future work will include scaling properties to significantly larger action spaces and environments with difficult exploration dynamics where branching or deterministic action selection may struggle relative to joint or auto\-regressive learners\. We hope to expand the list of benchmarks and supported algorithms to provide a consolidated selection guide for practitioners facing real world problems that are naturally represented with both continuous and discrete components\.

## References

- G\. Campos, N\. H\. El\-Farra, and A\. Palazoglu \(2022\)Soft actor\-critic deep reinforcement learning with hybrid mixed\-integer actions for demand responsive scheduling of energy systems\.Industrial & Engineering Chemistry Research61\(24\),pp\. 8443–8461\.Cited by:[§3\.4](https://arxiv.org/html/2606.26574#S3.SS4.p1.3)\.
- Y\. Chandak, G\. Theocharous, J\. Kostas, S\. Jordan, and P\. Thomas \(2019\)Learning action representations for reinforcement learning\.InInternational conference on machine learning,pp\. 941–950\.Cited by:[§3\.5](https://arxiv.org/html/2606.26574#S3.SS5.p1.1)\.
- S\. Chen, X\. Qiu, X\. Tan, Z\. Fang, and Y\. Jin \(2022\)A model\-based hybrid soft actor\-critic deep reinforcement learning algorithm for optimal ventilator settings\.Information sciences611,pp\. 47–64\.Cited by:[§3\.4](https://arxiv.org/html/2606.26574#S3.SS4.p1.3),[§5\.3](https://arxiv.org/html/2606.26574#S5.SS3.p1.1)\.
- O\. Delalleau, M\. Peter, E\. Alonso, and A\. Logut \(2019\)Discrete and continuous action representation for practical rl in video games\.arXiv preprint arXiv:1912\.11077\.Cited by:[§3\.4](https://arxiv.org/html/2606.26574#S3.SS4.p1.3),[§5\.3](https://arxiv.org/html/2606.26574#S5.SS3.p1.1),[§7\.1](https://arxiv.org/html/2606.26574#S7.SS1.SSS0.Px1.p1.8)\.
- Y\. Duan, X\. Chen, R\. Houthooft, J\. Schulman, and P\. Abbeel \(2016\)Benchmarking deep reinforcement learning for continuous control\.InInternational conference on machine learning,pp\. 1329–1338\.Cited by:[§1](https://arxiv.org/html/2606.26574#S1.p4.1)\.
- B\. Ellis, J\. Cook, S\. Moalla, M\. Samvelyan, M\. Sun, A\. Mahajan, J\. Foerster, and S\. Whiteson \(2023\)Smacv2: an improved benchmark for cooperative multi\-agent reinforcement learning\.Advances in Neural Information Processing Systems36,pp\. 37567–37593\.Cited by:[§1](https://arxiv.org/html/2606.26574#S1.p4.1)\.
- L\. Espeholt, H\. Soyer, R\. Munos, K\. Simonyan, V\. Mnih, T\. Ward, Y\. Doron, V\. Firoiu, T\. Harley, I\. Dunning,et al\.\(2018\)Impala: scalable distributed deep\-rl with importance weighted actor\-learner architectures\.InInternational conference on machine learning,pp\. 1407–1416\.Cited by:[§5\.2](https://arxiv.org/html/2606.26574#S5.SS2.SSS0.Px1.p1.7),[§9](https://arxiv.org/html/2606.26574#S9.p1.2)\.
- Z\. Fan, R\. Su, W\. Zhang, and Y\. Yu \(2019\)Hybrid actor\-critic reinforcement learning in parameterized action space\.arXiv preprint arXiv:1903\.01344\.Cited by:[§3\.4](https://arxiv.org/html/2606.26574#S3.SS4.p1.3)\.
- J\. Foerster, G\. Farquhar, T\. Afouras, N\. Nardelli, and S\. Whiteson \(2018\)Counterfactual multi\-agent policy gradients\.InProceedings of the AAAI conference on artificial intelligence,Vol\.32\.Cited by:[§3\.1](https://arxiv.org/html/2606.26574#S3.SS1.p1.1),[§3\.2](https://arxiv.org/html/2606.26574#S3.SS2.p2.4),[§5\.2\.2](https://arxiv.org/html/2606.26574#S5.SS2.SSS2.p2.2)\.
- T\. Haarnoja, A\. Zhou, P\. Abbeel, and S\. Levine \(2018\)Soft actor\-critic: off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.InInternational conference on machine learning,pp\. 1861–1870\.Cited by:[§1](https://arxiv.org/html/2606.26574#S1.p3.1)\.
- P\. Hernandez\-Leal, B\. Kartal, and M\. E\. Taylor \(2019\)A survey and critique of multiagent deep reinforcement learning\.Autonomous Agents and Multi\-Agent Systems33\(6\),pp\. 750–797\.Cited by:[§3\.1](https://arxiv.org/html/2606.26574#S3.SS1.p1.1)\.
- E\. Jang, S\. Gu, and B\. Poole \(2016\)Categorical reparameterization with gumbel\-softmax\.arXiv preprint arXiv:1611\.01144\.Cited by:[§3\.4](https://arxiv.org/html/2606.26574#S3.SS4.p1.3),[§5\.3](https://arxiv.org/html/2606.26574#S5.SS3.p1.1)\.
- D\. Korenkevych, A\. R\. Mahmood, G\. Vasan, and J\. Bergstra \(2019\)Autoregressive policies for continuous control deep reinforcement learning\.arXiv preprint arXiv:1903\.11524\.Cited by:[§3\.3](https://arxiv.org/html/2606.26574#S3.SS3.p1.2)\.
- B\. Li, H\. Tang, Y\. Zheng, J\. Hao, P\. Li, Z\. Wang, Z\. Meng, and L\. Wang \(2021\)Hyar: addressing discrete\-continuous action reinforcement learning via hybrid action representation\.arXiv preprint arXiv:2109\.05490\.Cited by:[§3\.5](https://arxiv.org/html/2606.26574#S3.SS5.p1.1)\.
- C\. Li, J\. Liu, Y\. Zhang, Y\. Wei, Y\. Niu, Y\. Yang, Y\. Liu, and W\. Ouyang \(2023a\)Ace: cooperative multi\-agent q\-learning with bidirectional action\-dependency\.InProceedings of the AAAI conference on artificial intelligence,Vol\.37,pp\. 8536–8544\.Cited by:[§3\.3](https://arxiv.org/html/2606.26574#S3.SS3.p1.2)\.
- Z\. Li, T\. Chen, Z\. Hong, A\. Ajay, and P\. Agrawal \(2023b\)ParallelQQ\-learning: scaling off\-policy reinforcement learning under massively parallel simulation\.InInternational Conference on Machine Learning,pp\. 19440–19459\.Cited by:[§9](https://arxiv.org/html/2606.26574#S9.p1.2)\.
- T\. P\. Lillicrap, J\. J\. Hunt, A\. Pritzel, N\. Heess, T\. Erez, Y\. Tassa, D\. Silver, and D\. Wierstra \(2015\)Continuous control with deep reinforcement learning\.arXiv preprint arXiv:1509\.02971\.Cited by:[§1](https://arxiv.org/html/2606.26574#S1.p3.1)\.
- Z\. E\. Liu, Y\. Li, Q\. Zhou, Y\. Li, B\. Shuai, H\. Xu, M\. Hua, G\. Tan, and L\. Xu \(2024\)Deep reinforcement learning\-based energy management for heavy duty hev considering discrete\-continuous hybrid action space\.IEEE Transactions on Transportation Electrification10\(4\),pp\. 9864–9876\.External Links:[Document](https://dx.doi.org/10.1109/TTE.2024.3363650)Cited by:[§3\.4](https://arxiv.org/html/2606.26574#S3.SS4.p1.3)\.
- R\. Lowe, Y\. I\. Wu, A\. Tamar, J\. Harb, O\. Pieter Abbeel, and I\. Mordatch \(2017\)Multi\-agent actor\-critic for mixed cooperative\-competitive environments\.Advances in neural information processing systems30\.Cited by:[§3\.1](https://arxiv.org/html/2606.26574#S3.SS1.p1.1)\.
- W\. Masson, P\. Ranchod, and G\. Konidaris \(2016\)Reinforcement learning with parameterized actions\.InProceedings of the AAAI conference on artificial intelligence,Vol\.30\.Cited by:[§4\.2](https://arxiv.org/html/2606.26574#S4.SS2.p1.7)\.
- L\. Matignon, G\. J\. Laurent, and N\. Le Fort\-Piat \(2012\)Independent reinforcement learners in cooperative markov games: a survey regarding coordination problems\.The Knowledge Engineering Review27\(1\),pp\. 1–31\.Cited by:[§3\.1](https://arxiv.org/html/2606.26574#S3.SS1.p1.1)\.
- V\. Mnih, K\. Kavukcuoglu, D\. Silver, A\. Graves, I\. Antonoglou, D\. Wierstra, and M\. Riedmiller \(2013\)Playing atari with deep reinforcement learning\.arXiv preprint arXiv:1312\.5602\.Cited by:[§1](https://arxiv.org/html/2606.26574#S1.p3.1)\.
- B\. Peng, T\. Rashid, C\. Schroeder de Witt, P\. Kamienny, P\. Torr, W\. Böhmer, and S\. Whiteson \(2021\)Facmac: factored multi\-agent centralised policy gradients\.Advances in neural information processing systems34,pp\. 12208–12221\.Cited by:[§5\.2\.2](https://arxiv.org/html/2606.26574#S5.SS2.SSS2.p1.1)\.
- T\. Rashid, M\. Samvelyan, C\. S\. De Witt, G\. Farquhar, J\. Foerster, and S\. Whiteson \(2020\)Monotonic value function factorisation for deep multi\-agent reinforcement learning\.Journal of Machine Learning Research21\(178\),pp\. 1–51\.Cited by:[§3\.1](https://arxiv.org/html/2606.26574#S3.SS1.p1.1),[§3\.2](https://arxiv.org/html/2606.26574#S3.SS2.p1.6)\.
- D\. Rezende and S\. Mohamed \(2015\)Variational inference with normalizing flows\.InInternational conference on machine learning,pp\. 1530–1538\.Cited by:[§1](https://arxiv.org/html/2606.26574#S1.p3.1),[§3\.3](https://arxiv.org/html/2606.26574#S3.SS3.p1.2)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§1](https://arxiv.org/html/2606.26574#S1.p3.1),[§5\.2](https://arxiv.org/html/2606.26574#S5.SS2.p1.1)\.
- T\. Seyde, I\. Gilitschenski, W\. Schwarting, B\. Stellato, M\. Riedmiller, M\. Wulfmeier, and D\. Rus \(2021\)Is bang\-bang control all you need? solving continuous control with bernoulli policies\.Advances in Neural Information Processing Systems34,pp\. 27209–27221\.Cited by:[§3\.4](https://arxiv.org/html/2606.26574#S3.SS4.p1.3)\.
- S\. P\. Singh and R\. S\. Sutton \(1996\)Reinforcement learning with replacing eligibility traces\.Machine learning22\(1\),pp\. 123–158\.Cited by:[§5\.2](https://arxiv.org/html/2606.26574#S5.SS2.SSS0.Px1.p1.7)\.
- K\. Son, D\. Kim, W\. J\. Kang, D\. E\. Hostallero, and Y\. Yi \(2019\)Qtran: learning to factorize with transformation for cooperative multi\-agent reinforcement learning\.InInternational conference on machine learning,pp\. 5887–5896\.Cited by:[§3\.2](https://arxiv.org/html/2606.26574#S3.SS2.p2.4)\.
- P\. Sunehag, G\. Lever, A\. Gruslys, W\. M\. Czarnecki, V\. Zambaldi, M\. Jaderberg, M\. Lanctot, N\. Sonnerat, J\. Z\. Leibo, K\. Tuyls,et al\.\(2017\)Value\-decomposition networks for cooperative multi\-agent learning\.arXiv preprint arXiv:1706\.05296\.Cited by:[§3\.2](https://arxiv.org/html/2606.26574#S3.SS2.p1.6)\.
- R\. S\. Sutton, D\. McAllester, S\. Singh, and Y\. Mansour \(1999\)Policy gradient methods for reinforcement learning with function approximation\.Advances in neural information processing systems12\.Cited by:[§1](https://arxiv.org/html/2606.26574#S1.p3.1)\.
- M\. Tan \(1993\)Multi\-agent reinforcement learning: independent vs\. cooperative agents\.InProceedings of the tenth international conference on machine learning,pp\. 330–337\.Cited by:[§3\.1](https://arxiv.org/html/2606.26574#S3.SS1.p1.1)\.
- Y\. Tassa, Y\. Doron, A\. Muldal, T\. Erez, Y\. Li, D\. d\. L\. Casas, D\. Budden, A\. Abdolmaleki, J\. Merel, A\. Lefrancq,et al\.\(2018\)Deepmind control suite\.arXiv preprint arXiv:1801\.00690\.Cited by:[§1](https://arxiv.org/html/2606.26574#S1.p4.1)\.
- A\. Tavakoli, F\. Pardo, and P\. Kormushev \(2018\)Action branching architectures for deep reinforcement learning\.InProceedings of the aaai conference on artificial intelligence,Vol\.32\.Cited by:[§3\.2](https://arxiv.org/html/2606.26574#S3.SS2.p2.4),[§5\.1](https://arxiv.org/html/2606.26574#S5.SS1.p1.4)\.
- J\. Terry, B\. Black, N\. Grammel, M\. Jayakumar, A\. Hari, R\. Sullivan, L\. S\. Santos, C\. Dieffendahl, C\. Horsch, R\. Perez\-Vicente,et al\.\(2021\)Pettingzoo: gym for multi\-agent reinforcement learning\.Advances in Neural Information Processing Systems34,pp\. 15032–15043\.Cited by:[§1](https://arxiv.org/html/2606.26574#S1.p4.1)\.
- E\. Todorov, T\. Erez, and Y\. Tassa \(2012\)Mujoco: a physics engine for model\-based control\.In2012 IEEE/RSJ international conference on intelligent robots and systems,pp\. 5026–5033\.Cited by:[§4\.5](https://arxiv.org/html/2606.26574#S4.SS5.p2.2)\.
- M\. Towers, A\. Kwiatkowski, J\. Terry, J\. U\. Balis, G\. De Cola, T\. Deleu, M\. Goulão, A\. Kallinteris, M\. Krimmel, A\. KG,et al\.\(2024\)Gymnasium: a standard interface for reinforcement learning environments\.arXiv preprint arXiv:2407\.17032\.Cited by:[§1](https://arxiv.org/html/2606.26574#S1.p4.1),[§4](https://arxiv.org/html/2606.26574#S4.p1.5)\.
- N\. Vieillard, O\. Pietquin, and M\. Geist \(2020\)Munchausen reinforcement learning\.Advances in Neural Information Processing Systems33,pp\. 4235–4246\.Cited by:[§5\.3](https://arxiv.org/html/2606.26574#S5.SS3.p1.1)\.
- O\. Vinyals, I\. Babuschkin, W\. M\. Czarnecki, M\. Mathieu, A\. Dudzik, J\. Chung, D\. H\. Choi, R\. Powell, T\. Ewalds, P\. Georgiev,et al\.\(2019\)Grandmaster level in starcraft ii using multi\-agent reinforcement learning\.nature575\(7782\),pp\. 350–354\.Cited by:[§3\.3](https://arxiv.org/html/2606.26574#S3.SS3.p1.2)\.
- J\. Wang, Z\. Ren, T\. Liu, Y\. Yu, and C\. Zhang \(2020\)Qplex: duplex dueling multi\-agent q\-learning\.arXiv preprint arXiv:2008\.01062\.Cited by:[§3\.2](https://arxiv.org/html/2606.26574#S3.SS2.p1.6),[§3\.2](https://arxiv.org/html/2606.26574#S3.SS2.p2.4),[§5\.1](https://arxiv.org/html/2606.26574#S5.SS1.p2.8)\.
- Z\. Wang, N\. Li, and G\. Gong \(2025\)VDMPO: policy optimization for cooperative multi\-agent reinforcement learning based on joint value decomposition\.Neurocomputing,pp\. 131193\.Cited by:[§3\.2](https://arxiv.org/html/2606.26574#S3.SS2.p2.4)\.
- Z\. Wang, T\. Schaul, M\. Hessel, H\. Hasselt, M\. Lanctot, and N\. Freitas \(2016\)Dueling network architectures for deep reinforcement learning\.InInternational conference on machine learning,pp\. 1995–2003\.Cited by:[§5\.1](https://arxiv.org/html/2606.26574#S5.SS1.p1.4)\.
- J\. Weng, M\. Lin, S\. Huang, B\. Liu, D\. Makoviichuk, V\. Makoviychuk, Z\. Liu, Y\. Song, T\. Luo, Y\. Jiang,et al\.\(2022\)Envpool: a highly parallel reinforcement learning environment execution engine\.Advances in Neural Information Processing Systems35,pp\. 22409–22421\.Cited by:[§4](https://arxiv.org/html/2606.26574#S4.p1.5)\.
- J\. Xiong, Q\. Wang, Z\. Yang, P\. Sun, L\. Han, Y\. Zheng, H\. Fu, T\. Zhang, J\. Liu, and H\. Liu \(2018\)Parametrized deep q\-networks learning: reinforcement learning with discrete\-continuous hybrid action space\.arXiv preprint arXiv:1810\.06394\.Cited by:[§3\.4](https://arxiv.org/html/2606.26574#S3.SS4.p1.3),[§5\.3](https://arxiv.org/html/2606.26574#S5.SS3.p1.1)\.
- W\. Ye, S\. Liu, T\. Kurutach, P\. Abbeel, and Y\. Gao \(2021\)Mastering atari games with limited data\.Advances in neural information processing systems34,pp\. 25476–25488\.Cited by:[§1](https://arxiv.org/html/2606.26574#S1.p4.1)\.
- C\. Yu, A\. Velu, E\. Vinitsky, J\. Gao, Y\. Wang, A\. Bayen, and Y\. Wu \(2022\)The surprising effectiveness of ppo in cooperative multi\-agent games\.Advances in neural information processing systems35,pp\. 24611–24624\.Cited by:[§3\.1](https://arxiv.org/html/2606.26574#S3.SS1.p1.1)\.

## Appendix ATheoretical Analysis of Importance\-Weighted GAE for Multi\-Head Branching Dueling Q\-Network Critics

### A\.1\.Markov Decision Process

Let\(𝒮,𝒜,P,r,γ\)\(\\mathcal\{S\},\\mathcal\{A\},P,r,\\gamma\)denote a discounted MDP where𝒮\\mathcal\{S\}is the state space,𝒜=∏h=1H𝒜h\\mathcal\{A\}=\\prod\_\{h=1\}^\{H\}\\mathcal\{A\}\_\{h\}is the joint action space factored overHHaction heads,P:𝒮×𝒜→Δ​\(𝒮\)P:\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\Delta\(\\mathcal\{S\}\)is the transition kernel,r:𝒮×𝒜→ℝr:\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\mathbb\{R\}is the scalar reward function, andγ∈\[0,1\)\\gamma\\in\[0,1\)is the discount factor\.

The policyπ\\piis factored as:

\(10\)π​\(𝐚∣s\)=∏h=1Hπh​\(ah∣s\),\\pi\(\\mathbf\{a\}\\mid s\)=\\prod\_\{h=1\}^\{H\}\\pi\_\{h\}\(a\_\{h\}\\mid s\),where each headπh:𝒮→Δ​\(𝒜h\)\\pi\_\{h\}:\\mathcal\{S\}\\to\\Delta\(\\mathcal\{A\}\_\{h\}\)is parameterised byθh\\theta\_\{h\}and is conditionally independent of all other heads given the statess\. This conditional independence is a*design constraint*of the branching architecture \(BDQ\), not an assumption imposed on the environment\.

### A\.2\.Branching Dueling Q\-Network \(BDQ\) Critic

###### Definition A\.1 \(BDQ Value Decomposition\)\.

The BDQ critic represents the joint state\-action value as:

\(11\)Qϕ​\(s,𝐚\)=Vϕ​\(s\)\+∑h=1HAϕ,h​\(s,ah\),Q\_\{\\phi\}\(s,\\mathbf\{a\}\)=V\_\{\\phi\}\(s\)\+\\sum\_\{h=1\}^\{H\}A\_\{\\phi,h\}\(s,a\_\{h\}\),whereVϕ:𝒮→ℝV\_\{\\phi\}:\\mathcal\{S\}\\to\\mathbb\{R\}is the shared state\-value stream andAϕ,h:𝒮×𝒜h→ℝA\_\{\\phi,h\}:\\mathcal\{S\}\\times\\mathcal\{A\}\_\{h\}\\to\\mathbb\{R\}is the advantage stream for headhh, both parameterised byϕ\\phi\.

To ensure identifiability ofVϕV\_\{\\phi\}and eachAϕ,hA\_\{\\phi,h\}, a zero\-mean normalisation must be enforced with respect to the*current policy*:

\(12\)𝔼ah∼πh\(⋅∣s\)​\[Aϕ,h​\(s,ah\)\]=0∀s∈𝒮,h∈ℋ\.\\mathbb\{E\}\_\{a\_\{h\}\\sim\\pi\_\{h\}\(\\cdot\\mid s\)\}\\\!\\left\[A\_\{\\phi,h\}\(s,a\_\{h\}\)\\right\]=0\\quad\\forall\\,s\\in\\mathcal\{S\},\\;h\\in\\mathcal\{H\}\.Equation \([12](https://arxiv.org/html/2606.26574#A1.E12)\) ensures that the value functionVϕ​\(s\)V\_\{\\phi\}\(s\)strictly targets the on\-policy state valueVπ​\(s\)V^\{\\pi\}\(s\)without absorbing expected policy advantages\.

### A\.3\.Per\-Head Importance Measure and Weights

###### Definition A\.2 \(Per\-Head Importance and Annealed Weights\)\.

For statessand action headhh, define the importance as the range of the advantage function:

\(13\)Ih​\(s\)=maxah∈𝒜h⁡Aϕ,h​\(s,ah\)−minah∈𝒜h⁡Aϕ,h​\(s,ah\)≥0\.I\_\{h\}\(s\)=\\max\_\{a\_\{h\}\\in\\mathcal\{A\}\_\{h\}\}A\_\{\\phi,h\}\(s,a\_\{h\}\)\-\\min\_\{a\_\{h\}\\in\\mathcal\{A\}\_\{h\}\}A\_\{\\phi,h\}\(s,a\_\{h\}\)\\geq 0\.Given an annealing parameterα∈\[0,1\]\\alpha\\in\[0,1\], the importance weight is:

\(14\)whα​\(s\)=Ih​\(s\)α∑h′=1HIh′​\(s\)α,w\_\{h\}^\{\\alpha\}\(s\)=\\frac\{I\_\{h\}\(s\)^\{\\alpha\}\}\{\\sum\_\{h^\{\\prime\}=1\}^\{H\}I\_\{h^\{\\prime\}\}\(s\)^\{\\alpha\}\},where00=10^\{0\}=1enforces uniform weightswh0​\(s\)=1/Hw\_\{h\}^\{0\}\(s\)=1/Hatα=0\\alpha=0\.

### A\.4\.Critic Optimisation and TD Error Structure

#### A\.4\.1\.TheVV\-only Target and Critic Loss

In environments with highly factored action spaces, using the full state\-action valueQ​\(st\+1,𝐚t\+1\)Q\(s\_\{t\+1\},\\mathbf\{a\}\_\{t\+1\}\)in the regression target introduces severe variance due to noisy advantage estimation\. To stabilise learning and ensure convergence to the on\-policy state value, the BDQ critic is trained using aVV\-only target:

\(15\)yt=rt\+γ​Vϕ​\(st\+1\)\.y\_\{t\}=r\_\{t\}\+\\gamma V\_\{\\phi\}\(s\_\{t\+1\}\)\.Under this target formulation, the regression loss for the parameterised criticQϕ=Vϕ\+∑hAϕ,hQ\_\{\\phi\}=V\_\{\\phi\}\+\\sum\_\{h\}A\_\{\\phi,h\}is:

ℒ​\(ϕ\)\\displaystyle\\mathcal\{L\}\(\\phi\)=12​𝔼τ∼π​\[\(Qϕ​\(st,𝐚t\)−yt\)2\]\\displaystyle=\\frac\{1\}\{2\}\\mathbb\{E\}\_\{\\tau\\sim\\pi\}\\\!\\left\[\\left\(Q\_\{\\phi\}\(s\_\{t\},\\mathbf\{a\}\_\{t\}\)\-y\_\{t\}\\right\)^\{2\}\\right\]\(16\)=12​𝔼τ∼π​\[\(Vϕ​\(st\)\+∑h=1HAϕ,h​\(st,ah,t\)−\(rt\+γ​Vϕ​\(st\+1\)\)\)2\]\.\\displaystyle=\\frac\{1\}\{2\}\\mathbb\{E\}\_\{\\tau\\sim\\pi\}\\\!\\left\[\\left\(V\_\{\\phi\}\(s\_\{t\}\)\+\\sum\_\{h=1\}^\{H\}A\_\{\\phi,h\}\(s\_\{t\},a\_\{h,t\}\)\-\\left\(r\_\{t\}\+\\gamma V\_\{\\phi\}\(s\_\{t\+1\}\)\\right\)\\right\)^\{2\}\\right\]\.

#### A\.4\.2\.Temporal Difference Residuals

Given a trajectoryτ=\(s0,𝐚0,r0,…\)\\tau=\(s\_\{0\},\\mathbf\{a\}\_\{0\},r\_\{0\},\\dots\), define the one\-stepVV\-based TD residual:

\(17\)δtV=rt\+γ​Vϕ​\(st\+1\)−Vϕ​\(st\)\.\\delta\_\{t\}^\{V\}=r\_\{t\}\+\\gamma V\_\{\\phi\}\(s\_\{t\+1\}\)\-V\_\{\\phi\}\(s\_\{t\}\)\.
Rearranging Equation \([16](https://arxiv.org/html/2606.26574#A1.E16)\) reveals that optimising the BDQ architecture with aVV\-only target is mathematically equivalent to fitting the sum of the per\-head advantages to the TD residual:

\(18\)ℒ​\(ϕ\)=12​𝔼τ∼π​\[\(∑h=1HAϕ,h​\(st,ah,t\)−δtV\)2\]\.\\mathcal\{L\}\(\\phi\)=\\frac\{1\}\{2\}\\mathbb\{E\}\_\{\\tau\\sim\\pi\}\\\!\\left\[\\left\(\\sum\_\{h=1\}^\{H\}A\_\{\\phi,h\}\(s\_\{t\},a\_\{h,t\}\)\-\\delta\_\{t\}^\{V\}\\right\)^\{2\}\\right\]\.This explicitly bridges the architecture’s loss function with the theoretical construct of Generalized Advantage Estimation \(GAE\)\.

###### Lemma A\.3 \(Unbiasedness of the TD Residual\)\.

Letℱt=σ​\(s0,𝐚0,…,st,𝐚t\)\\mathcal\{F\}\_\{t\}=\\sigma\(s\_\{0\},\\mathbf\{a\}\_\{0\},\\dots,s\_\{t\},\\mathbf\{a\}\_\{t\}\)be the filtration up to action𝐚t\\mathbf\{a\}\_\{t\}\. LetAπ​\(s,𝐚\)=Qπ​\(s,𝐚\)−Vπ​\(s\)A^\{\\pi\}\(s,\\mathbf\{a\}\)=Q^\{\\pi\}\(s,\\mathbf\{a\}\)\-V^\{\\pi\}\(s\)\. Then:

\(19\)𝔼\[δtV\|ℱt\]=Aπ\(st,𝐚t\)\+γ\(Vπ\(st\+1\)−Vϕ\(st\+1\)\)−\(Vπ\(st\)−Vϕ\(st\)\)\.\\mathbb\{E\}\\\!\\left\[\\delta\_\{t\}^\{V\}\\;\\middle\|\\;\\mathcal\{F\}\_\{t\}\\right\]=A^\{\\pi\}\(s\_\{t\},\\mathbf\{a\}\_\{t\}\)\+\\gamma\\\!\\left\(V^\{\\pi\}\(s\_\{t\+1\}\)\-V\_\{\\phi\}\(s\_\{t\+1\}\)\\right\)\-\\left\(V^\{\\pi\}\(s\_\{t\}\)\-V\_\{\\phi\}\(s\_\{t\}\)\\right\)\.When the critic has converged \(Vϕ=VπV\_\{\\phi\}=V^\{\\pi\}\),𝔼​\[δtV∣ℱt\]=Aπ​\(st,𝐚t\)\\mathbb\{E\}\[\\delta\_\{t\}^\{V\}\\mid\\mathcal\{F\}\_\{t\}\]=A^\{\\pi\}\(s\_\{t\},\\mathbf\{a\}\_\{t\}\)\.

### A\.5\.The Modified Per\-Head Estimator

###### Definition A\.4 \(Importance\-Weighted Per\-Head GAE\)\.

For each action headh∈ℋh\\in\\mathcal\{H\}, the importance\-weighted GAE is:

\(20\)A^t\(h\)=∑l=0T−t−1\(γ​λ\)l​whα​\(st\+l\)⋅δt\+lV\.\\hat\{A\}\_\{t\}^\{\(h\)\}=\\sum\_\{l=0\}^\{T\-t\-1\}\(\\gamma\\lambda\)^\{l\}\\,w\_\{h\}^\{\\alpha\}\(s\_\{t\+l\}\)\\cdot\\delta\_\{t\+l\}^\{V\}\.The weightswhα​\(st\+l\)w\_\{h\}^\{\\alpha\}\(s\_\{t\+l\}\)are computed using thecurrentparameter valuesϕ\\phiand are treated as fixed scalars \(no gradient flows through them\)\. Crucially,whα​\(st\+l\)w\_\{h\}^\{\\alpha\}\(s\_\{t\+l\}\)depends only on the statest\+ls\_\{t\+l\}, not the action sampled att\+lt\+l\.

### A\.6\.Formal Assumptions

###### Assumption A\.5 \(Factored Policy Independence\)\.

The policy satisfies \([10](https://arxiv.org/html/2606.26574#A1.E10)\):π​\(𝐚∣s\)=∏h=1Hπh​\(ah∣s\)\\pi\(\\mathbf\{a\}\\mid s\)=\\prod\_\{h=1\}^\{H\}\\pi\_\{h\}\(a\_\{h\}\\mid s\)\.

###### Assumption A\.6 \(On\-Policy Advantage Normalisation\)\.

The critic ensures zero\-mean advantage under the*current*policy distributionπ\\pi:𝔼ah∼πh​\[Aϕ,h​\(s,ah\)\]=0\\mathbb\{E\}\_\{a\_\{h\}\\sim\\pi\_\{h\}\}\[A\_\{\\phi,h\}\(s,a\_\{h\}\)\]=0\.

###### Assumption A\.7 \(Critic Convergence\)\.

For the primary bias analysis,Vϕ​\(s\)=Vπ​\(s\)V\_\{\\phi\}\(s\)=V^\{\\pi\}\(s\)for alls∈𝒮s\\in\\mathcal\{S\}\.

###### Assumption A\.8 \(Non\-Degeneracy and Stop\-Gradient\)\.

At every statess,∑h=1HIh​\(s\)\>0\\sum\_\{h=1\}^\{H\}I\_\{h\}\(s\)\>0\. The weightswhαw\_\{h\}^\{\\alpha\}are treated as constants with respect to policy parametersθ\\theta\.

### A\.7\.Main Results

#### A\.7\.1\.Unbiasedness of the Estimator

###### Theorem A\.9 \(Unbiasedness of the Weighted Advantage\)\.

Under Assumptions[A\.5](https://arxiv.org/html/2606.26574#A1.Thmtheorem5)–[A\.7](https://arxiv.org/html/2606.26574#A1.Thmtheorem7), the modified estimatorA^t\(h\)\\hat\{A\}\_\{t\}^\{\(h\)\}satisfies:

\(21\)𝔼τ∼π\[A^t\(h\)\|st,𝐚t\]=𝔼τ∼π\[∑l=0T−t−1\(γλ\)lwhα\(st\+l\)⋅Aπ\(st\+l,𝐚t\+l\)\|st,𝐚t\]\.\\mathbb\{E\}\_\{\\tau\\sim\\pi\}\\\!\\left\[\\hat\{A\}\_\{t\}^\{\(h\)\}\\;\\middle\|\\;s\_\{t\},\\mathbf\{a\}\_\{t\}\\right\]=\\mathbb\{E\}\_\{\\tau\\sim\\pi\}\\\!\\left\[\\sum\_\{l=0\}^\{T\-t\-1\}\(\\gamma\\lambda\)^\{l\}\\,w\_\{h\}^\{\\alpha\}\(s\_\{t\+l\}\)\\cdot A^\{\\pi\}\(s\_\{t\+l\},\\mathbf\{a\}\_\{t\+l\}\)\\;\\middle\|\\;s\_\{t\},\\mathbf\{a\}\_\{t\}\\right\]\.That is, it is an unbiased estimator of the expected discounted sum of importance\-weighted advantages along the trajectory\.

###### Proof\.

Step 1: Expand the expectation and define filtrations\.

\(22\)𝔼\[A^t\(h\)\|st,𝐚t\]=∑l=0T−t−1\(γλ\)l𝔼\[whα\(st\+l\)⋅δt\+lV\|st,𝐚t\]\.\\mathbb\{E\}\\\!\\left\[\\hat\{A\}\_\{t\}^\{\(h\)\}\\;\\middle\|\\;s\_\{t\},\\mathbf\{a\}\_\{t\}\\right\]=\\sum\_\{l=0\}^\{T\-t\-1\}\(\\gamma\\lambda\)^\{l\}\\;\\mathbb\{E\}\\\!\\left\[w\_\{h\}^\{\\alpha\}\(s\_\{t\+l\}\)\\cdot\\delta\_\{t\+l\}^\{V\}\\;\\middle\|\\;s\_\{t\},\\mathbf\{a\}\_\{t\}\\right\]\.Define the extended filtration𝒢t\+l=σ​\(s0,𝐚0,…,st\+l,𝐚t\+l,st\+l\+1\)\\mathcal\{G\}\_\{t\+l\}=\\sigma\(s\_\{0\},\\mathbf\{a\}\_\{0\},\\dots,s\_\{t\+l\},\\mathbf\{a\}\_\{t\+l\},s\_\{t\+l\+1\}\)\. Note thatδt\+lV\\delta\_\{t\+l\}^\{V\}is𝒢t\+l\\mathcal\{G\}\_\{t\+l\}\-measurable, whilewhα​\(st\+l\)w\_\{h\}^\{\\alpha\}\(s\_\{t\+l\}\)is strictlyσ​\(st\+l\)\\sigma\(s\_\{t\+l\}\)\-measurable\.

Step 2: Factor using the tower property\.Apply the tower property conditioning on the statest\+ls\_\{t\+l\}and action𝐚t\+l\\mathbf\{a\}\_\{t\+l\}:

\(23\)𝔼\[whα\(st\+l\)δt\+lV\|st,𝐚t\]\\displaystyle\\mathbb\{E\}\\\!\\left\[w\_\{h\}^\{\\alpha\}\(s\_\{t\+l\}\)\\delta\_\{t\+l\}^\{V\}\\;\\middle\|\\;s\_\{t\},\\mathbf\{a\}\_\{t\}\\right\]=𝔼\[𝔼\[whα\(st\+l\)δt\+lV\|st\+l,𝐚t\+l\]\|st,𝐚t\]\.\\displaystyle=\\mathbb\{E\}\\\!\\left\[\\mathbb\{E\}\\\!\\left\[w\_\{h\}^\{\\alpha\}\(s\_\{t\+l\}\)\\delta\_\{t\+l\}^\{V\}\\;\\middle\|\\;s\_\{t\+l\},\\mathbf\{a\}\_\{t\+l\}\\right\]\\;\\middle\|\\;s\_\{t\},\\mathbf\{a\}\_\{t\}\\right\]\.
Step 3: Measurability of importance weights\.Becausewhα​\(st\+l\)w\_\{h\}^\{\\alpha\}\(s\_\{t\+l\}\)is a deterministic function of the statest\+ls\_\{t\+l\}alone, it is fully determined given the inner conditioning\. It can be factored out:

\(24\)𝔼\[whα\(st\+l\)δt\+lV\|st\+l,𝐚t\+l\]\\displaystyle\\mathbb\{E\}\\\!\\left\[w\_\{h\}^\{\\alpha\}\(s\_\{t\+l\}\)\\delta\_\{t\+l\}^\{V\}\\;\\middle\|\\;s\_\{t\+l\},\\mathbf\{a\}\_\{t\+l\}\\right\]=whα\(st\+l\)⋅𝔼\[δt\+lV\|st\+l,𝐚t\+l\]\.\\displaystyle=w\_\{h\}^\{\\alpha\}\(s\_\{t\+l\}\)\\cdot\\mathbb\{E\}\\\!\\left\[\\delta\_\{t\+l\}^\{V\}\\;\\middle\|\\;s\_\{t\+l\},\\mathbf\{a\}\_\{t\+l\}\\right\]\.
Step 4: Invoke Lemma[A\.3](https://arxiv.org/html/2606.26574#A1.Thmtheorem3)\.Under Assumption[A\.7](https://arxiv.org/html/2606.26574#A1.Thmtheorem7)\(Vϕ=VπV\_\{\\phi\}=V^\{\\pi\}\), Lemma[A\.3](https://arxiv.org/html/2606.26574#A1.Thmtheorem3)gives𝔼​\[δt\+lV∣st\+l,𝐚t\+l\]=Aπ​\(st\+l,𝐚t\+l\)\\mathbb\{E\}\[\\delta\_\{t\+l\}^\{V\}\\mid s\_\{t\+l\},\\mathbf\{a\}\_\{t\+l\}\]=A^\{\\pi\}\(s\_\{t\+l\},\\mathbf\{a\}\_\{t\+l\}\)\. Substituting this back completes the proof:

\(25\)𝔼\[A^t\(h\)\|st,𝐚t\]=𝔼\[∑l=0T−t−1\(γλ\)lwhα\(st\+l\)⋅Aπ\(st\+l,𝐚t\+l\)\|st,𝐚t\]\.\\mathbb\{E\}\\\!\\left\[\\hat\{A\}\_\{t\}^\{\(h\)\}\\;\\middle\|\\;s\_\{t\},\\mathbf\{a\}\_\{t\}\\right\]=\\mathbb\{E\}\\\!\\left\[\\sum\_\{l=0\}^\{T\-t\-1\}\(\\gamma\\lambda\)^\{l\}\\,w\_\{h\}^\{\\alpha\}\(s\_\{t\+l\}\)\\cdot A^\{\\pi\}\(s\_\{t\+l\},\\mathbf\{a\}\_\{t\+l\}\)\\;\\middle\|\\;s\_\{t\},\\mathbf\{a\}\_\{t\}\\right\]\.∎

###### Corollary A\.10 \(Sum Recovers Standard GAE\)\.

Because∑h=1Hwhα​\(s\)=1\\sum\_\{h=1\}^\{H\}w\_\{h\}^\{\\alpha\}\(s\)=1identically for all statesss:

\(26\)∑h=1H𝔼\[A^t\(h\)\|st,𝐚t\]=𝔼\[∑l=0T−t−1\(γλ\)lAπ\(st\+l,𝐚t\+l\)\|st,𝐚t\]=𝔼\[A^tGAE\|st,𝐚t\]\.\\sum\_\{h=1\}^\{H\}\\mathbb\{E\}\\\!\\left\[\\hat\{A\}\_\{t\}^\{\(h\)\}\\;\\middle\|\\;s\_\{t\},\\mathbf\{a\}\_\{t\}\\right\]=\\mathbb\{E\}\\\!\\left\[\\sum\_\{l=0\}^\{T\-t\-1\}\(\\gamma\\lambda\)^\{l\}A^\{\\pi\}\(s\_\{t\+l\},\\mathbf\{a\}\_\{t\+l\}\)\\;\\middle\|\\;s\_\{t\},\\mathbf\{a\}\_\{t\}\\right\]=\\mathbb\{E\}\\\!\\left\[\\hat\{A\}\_\{t\}^\{\\mathrm\{GAE\}\}\\;\\middle\|\\;s\_\{t\},\\mathbf\{a\}\_\{t\}\\right\]\.The global policy gradient remains unbiased\.

#### A\.7\.2\.Validity of the Policy Gradient

###### Theorem A\.11 \(Valid Policy Gradient Direction\)\.

Under Assumptions[A\.5](https://arxiv.org/html/2606.26574#A1.Thmtheorem5)–[A\.8](https://arxiv.org/html/2606.26574#A1.Thmtheorem8), the per\-head gradient estimatorg^h=A^t\(h\)​∇θhlog⁡πh​\(at,h∣st\)\\hat\{g\}\_\{h\}=\\hat\{A\}\_\{t\}^\{\(h\)\}\\nabla\_\{\\theta\_\{h\}\}\\log\\pi\_\{h\}\(a\_\{t,h\}\\mid s\_\{t\}\)satisfies:

\(27\)sign​\(𝔼​\[g^h\]⊤​∇θhJ\)\>0\.\\mathrm\{sign\}\\\!\\left\(\\mathbb\{E\}\[\\hat\{g\}\_\{h\}\]^\{\\top\}\\nabla\_\{\\theta\_\{h\}\}J\\right\)\>0\.

###### Proof\.

By the stop\-gradient constraint \(Assumption[A\.8](https://arxiv.org/html/2606.26574#A1.Thmtheorem8)\), the weights do not interfere with the score function\. The expected update is a positive, state\-dependent rescaling of the true gradient, guaranteeing ascent in expectation\. ∎

### A\.8\.Variance Reduction Analysis

###### Proposition A\.12 \(Monotonic Variance Reduction\)\.

Under Assumptions[A\.5](https://arxiv.org/html/2606.26574#A1.Thmtheorem5)–[A\.7](https://arxiv.org/html/2606.26574#A1.Thmtheorem7), the per\-step contribution to estimator variance from headhhis strictly reduced for low\-importance heads relative to uniform weighting \(α=0\\alpha=0\)\.

\(28\)\(whα​\(s\)\)2≤1H2for heads with​Ih​\(s\)≤1H​∑j=1HIj​\(s\)\.\(w\_\{h\}^\{\\alpha\}\(s\)\)^\{2\}\\leq\\frac\{1\}\{H^\{2\}\}\\quad\\text\{for heads with \}I\_\{h\}\(s\)\\leq\\frac\{1\}\{H\}\\sum\_\{j=1\}^\{H\}I\_\{j\}\(s\)\.

###### Proof\.

For anyα\>0\\alpha\>0, the functionx↦xα/∑xjαx\\mapsto x^\{\\alpha\}/\\sum x\_\{j\}^\{\\alpha\}is monotonic with respect toxx\. Therefore, if the advantage rangeIh​\(s\)I\_\{h\}\(s\)is below the arithmetic mean importanceI¯​\(s\)=1H​∑Ij​\(s\)\\bar\{I\}\(s\)=\\frac\{1\}\{H\}\\sum I\_\{j\}\(s\), its corresponding weightwhα​\(s\)w\_\{h\}^\{\\alpha\}\(s\)is strictly less than its uniform share1/H1/H\. Since the variance of the estimator sum scales with\(whα​\(s\)\)2​Var​\[δV∣s\]\(w\_\{h\}^\{\\alpha\}\(s\)\)^\{2\}\\mathrm\{Var\}\[\\delta^\{V\}\\mid s\], the variance injected by uninformative heads is systematically suppressed\. ∎

### A\.9\.Finite\-Sample and Non\-Converged Critic Analysis

###### Proposition A\.13 \(Bias Under Critic Error\)\.

Define the finite\-sample critic error asϵϕ​\(s\)=Vϕ​\(s\)−Vπ​\(s\)\\epsilon\_\{\\phi\}\(s\)=V\_\{\\phi\}\(s\)\-V^\{\\pi\}\(s\)\. Because the target isVV\-only, the bias of the modified estimator is precisely:

\(29\)Bias​\[A^t\(h\)\]=∑l=0T−t−1\(γ​λ\)l​𝔼​\[whα​\(st\+l\)​\(γ​ϵϕ​\(st\+l\+1\)−ϵϕ​\(st\+l\)\)\]\.\\displaystyle\\mathrm\{Bias\}\\\!\\left\[\\hat\{A\}\_\{t\}^\{\(h\)\}\\right\]=\\sum\_\{l=0\}^\{T\-t\-1\}\(\\gamma\\lambda\)^\{l\}\\;\\mathbb\{E\}\\\!\\left\[w\_\{h\}^\{\\alpha\}\(s\_\{t\+l\}\)\\left\(\\gamma\\epsilon\_\{\\phi\}\(s\_\{t\+l\+1\}\)\-\\epsilon\_\{\\phi\}\(s\_\{t\+l\}\)\\right\)\\right\]\.Sincewhα​\(s\)∈\(0,1\)w\_\{h\}^\{\\alpha\}\(s\)\\in\(0,1\), the magnitude of the bias contribution from noisy heads is attenuated relative to standard unweighted GAE\.

## Appendix BAdvantage Centering in Monotonic Mixing Architectures

In factored dueling architectures such as those used for DQN and SAC, the total action\-valueQt​o​tQ\_\{tot\}is decomposed into a global state\-valueV​\(s\)V\(s\)and a set of agent\-specific advantagesAh​\(s,ah\)A\_\{h\}\(s,a\_\{h\}\), recombined via a monotonic mixerfm​i​xf\_\{mix\}:

\(30\)Qt​o​t​\(s,𝐚\)=V​\(s\)\+fm​i​x​\(A1​\(s,a1\),…,An​\(s,an\);s\)Q\_\{tot\}\(s,\\mathbf\{a\}\)=V\(s\)\+f\_\{mix\}\(A\_\{1\}\(s,a\_\{1\}\),\\dots,A\_\{n\}\(s,a\_\{n\}\);s\)
While this decomposition provides a powerful inductive bias, it introduces a criticalidentifiability crisisregarding the magnitude of the advantage terms\. When using a monotonic mixer \(e\.g\., QPLEX\), the requirement formax\-centering\(maxa⁡A​\(s,a\)=0\\max\_\{a\}A\(s,a\)=0\) becomes a structural necessity rather than a convention\.

### B\.1\.The Threat of Gain Inflation

In Q\-PLEX mode, the mixerfm​i​xf\_\{mix\}is parameterized by a hypernetwork that generates non\-negative weightswh​\(s\)≥0w\_\{h\}\(s\)\\geq 0on the advantages\. This ensures the monotonicity constraint∂Qt​o​t∂Ah≥0\\frac\{\\partial Q\_\{tot\}\}\{\\partial A\_\{h\}\}\\geq 0\. However, because these weights are state\-dependent and effectively unbounded, the network can minimize TD error by arbitrarily “inflating” the gain of the mixer\.

If the advantages are not strictly grounded \(e\.g\., if we use mean\-centering or no centering\), a positive feedback loop can emerge during bootstrapping:

1. \(1\)The value headV​\(s\)V\(s\)drifts \(e\.g\., decreases\)\.
2. \(2\)To maintain the targetQQ, the hypernetwork increases the weightswhw\_\{h\}\.
3. \(3\)The largerwhw\_\{h\}“amplifies” the advantage signal, makingQt​o​tQ\_\{tot\}highly sensitive to small changes inAhA\_\{h\}\.
4. \(4\)This increased gain drives larger gradients back into the encoder, further ratcheting the weights and leading to abootstrapped divergence\(loss and weights exploding toward infinity\)\.

### B\.2\.Max\-Centering as a Structural Clamp

Max\-centering \(A=A−max⁡AA=A\-\\max A\) enforces the constraintA​\(s,a\)≤0A\(s,a\)\\leq 0for all actions\. This grounding serves as a “structural clamp” in two ways:

- •Fixed Upper Bound:By forcing the maximum advantage to be zero, the termfm​i​x​\(0,…,0;s\)f\_\{mix\}\(0,\\dots,0;s\)becomes the fixed upper bound of the mixer’s contribution\. If the mixer is implemented without state\-dependent biases \(as in theQSclass whendueling=True\), thenfm​i​x​\(𝟎;s\)=0f\_\{mix\}\(\\mathbf\{0\};s\)=0\.
- •Decoupling V from Gain:Withmax⁡A=0\\max A=0, the identityQt​o​t​\(s,𝐚∗\)=V​\(s\)Q\_\{tot\}\(s,\\mathbf\{a\}^\{\*\}\)=V\(s\)is strictly enforced for the optimal joint action𝐚∗\\mathbf\{a\}^\{\*\}\. This prevents the hypernetwork from using gain inflation to “swallow” the state\-valueV​\(s\)V\(s\)into the advantage terms\. The only way to increase the total Q\-value of the optimal action is to increaseV​\(s\)V\(s\)directly, which is subject to the standard TD backup and is not amplified by the mixer’s weights\.

### B\.3\.Mathematical Identifiability in SAC and DQN

For off\-policy algorithms like DQN and SAC, the objective is to estimate the optimal valueV∗​\(s\)=maxa⁡Q​\(s,a\)V^\{\*\}\(s\)=\\max\_\{a\}Q\(s,a\)\. Max\-centering is the only constraint that aligns the internal state\-value headV​\(s\)V\(s\)with this mathematical definition\.

Without max\-centering, the mixer’s asymmetry \(often caused by activations likeReLUorELU\) can lead to a “representational gap”: the mixer can represent large positive contributions easily but struggles to represent very negative values\. Max\-centering forces all sub\-optimal actions to be represented as negative offsets fromV​\(s\)V\(s\)\. This ensures that the “noise” or “drift” from sub\-optimal actions does not leak into the estimation of the state value, preserving the stability of the greedy policy\.

## Appendix CHyperparameter Configurations

The complete list of hyperparameter settings across all reinforcement learning evaluations is detailed below\.

Table 5\.Hyperparameter configurations for all evaluation environments\.DependentPush\(Hidden:
\[256, 256\]\)DiscretePPO2,000,000Buffer: 4096, Batch: 128, Epochs: 4,γ=0\.98\\gamma=0\.98, Clip: 0\.2,λGAE=0\.95\\lambda\_\{\\text\{GAE\}\}=0\.95SAC1,000,000Replay: 50k, Batch: 128,γ=0\.98\\gamma=0\.98,τSAC=0\.01\\tau\_\{\\text\{SAC\}\}=0\.01, Ent%: 0\.5DQN1,000,000Replay: 50k, Batch: 128,γ=0\.98\\gamma=0\.98,τ=0\.01\\tau=0\.01, Update: 4ContinuousPPO2,000,000Buffer: 4096, Batch: 128, Epochs: 4,γ=0\.98\\gamma=0\.98, Clip: 0\.2,λGAE=0\.95\\lambda\_\{\\text\{GAE\}\}=0\.95SAC1,000,000Replay: 50k, Batch: 256,γ=0\.98\\gamma=0\.98,τSAC=0\.01\\tau\_\{\\text\{SAC\}\}=0\.01, Ent%: 0\.5DQN1,000,000Replay: 50k, Batch: 256,γ=0\.98\\gamma=0\.98,τ=0\.01\\tau=0\.01, Update: 4HybridPPO3,000,000Buffer: 4096, Batch: 128, Epochs: 5,γ=0\.98\\gamma=0\.98, Clip: 0\.2,λGAE=0\.95\\lambda\_\{\\text\{GAE\}\}=0\.95SAC2,000,000Replay: 50k, Batch: 256,γ=0\.98\\gamma=0\.98,τSAC=0\.01\\tau\_\{\\text\{SAC\}\}=0\.01, Ent%: 0\.5DQN2,000,000Replay: 50k, Batch: 256,γ=0\.98\\gamma=0\.98,τ=0\.01\\tau=0\.01, Update: 4Lander\(Hidden Dims:
\[128, 128\]\)DiscretePPO600,000Buffer: 2048, Batch: 64, Epochs: 4,γ=0\.99\\gamma=0\.99, Clip: 0\.2,λGAE=0\.95\\lambda\_\{\\text\{GAE\}\}=0\.95SAC400,000Replay: 50k, Batch: 128,γ=0\.99\\gamma=0\.99,τSAC=0\.01\\tau\_\{\\text\{SAC\}\}=0\.01, Ent%: 0\.5DQN400,000Replay: 50k, Batch: 128,γ=0\.99\\gamma=0\.99,τ=0\.01\\tau=0\.01, Update: 4ContinuousPPO600,000Buffer: 2048, Batch: 64, Epochs: 4,γ=0\.99\\gamma=0\.99, Clip: 0\.2,λGAE=0\.95\\lambda\_\{\\text\{GAE\}\}=0\.95SAC400,000Replay: 50k, Batch: 128,γ=0\.99\\gamma=0\.99,τSAC=0\.01\\tau\_\{\\text\{SAC\}\}=0\.01, Ent%: 0\.5DQN400,000Replay: 50k, Batch: 128,γ=0\.99\\gamma=0\.99,τ=0\.01\\tau=0\.01, Update: 4HybridPPO1,000,000Buffer: 2048, Batch: 64, Epochs: 4,γ=0\.99\\gamma=0\.99, Clip: 0\.2,λGAE=0\.95\\lambda\_\{\\text\{GAE\}\}=0\.95SAC600,000Replay: 50k, Batch: 128,γ=0\.99\\gamma=0\.99,τSAC=0\.01\\tau\_\{\\text\{SAC\}\}=0\.01, Ent%: 0\.5DQN600,000Replay: 50k, Batch: 128,γ=0\.99\\gamma=0\.99,τ=0\.01\\tau=0\.01, Update: 4Shoot\(Hidden Dims:
\[256, 256\]\)Dep\. HybridPPO1,500,000Buffer: 4096, Batch: 128, Epochs: 5,γ=0\.99\\gamma=0\.99, Clip: 0\.2,λGAE=0\.95\\lambda\_\{\\text\{GAE\}\}=0\.95SAC800,000Replay: 100k, Batch: 256,γ=0\.99\\gamma=0\.99,τSAC=0\.01\\tau\_\{\\text\{SAC\}\}=0\.01, Ent%: 0\.5DQN800,000Replay: 100k, Batch: 256,γ=0\.99\\gamma=0\.99,τ=0\.01\\tau=0\.01, Update: 4Indep\. HybridPPO1,500,000Buffer: 4096, Batch: 128, Epochs: 5,γ=0\.99\\gamma=0\.99, Clip: 0\.2,λGAE=0\.95\\lambda\_\{\\text\{GAE\}\}=0\.95SAC800,000Replay: 100k, Batch: 256,γ=0\.99\\gamma=0\.99,τSAC=0\.01\\tau\_\{\\text\{SAC\}\}=0\.01, Ent%: 0\.5DQN800,000Replay: 100k, Batch: 256,γ=0\.99\\gamma=0\.99,τ=0\.01\\tau=0\.01, Update: 4Dep\. DiscretePPO1,500,000Buffer: 4096, Batch: 128, Epochs: 4,γ=0\.99\\gamma=0\.99, Clip: 0\.2,λGAE=0\.95\\lambda\_\{\\text\{GAE\}\}=0\.95SAC800,000Replay: 100k, Batch: 256,γ=0\.99\\gamma=0\.99,τSAC=0\.01\\tau\_\{\\text\{SAC\}\}=0\.01, Ent%: 0\.5DQN800,000Replay: 100k, Batch: 256,γ=0\.99\\gamma=0\.99,τ=0\.01\\tau=0\.01, Update: 4Indep\. DiscretePPO1,500,000Buffer: 4096, Batch: 128, Epochs: 4,γ=0\.99\\gamma=0\.99, Clip: 0\.2,λGAE=0\.95\\lambda\_\{\\text\{GAE\}\}=0\.95SAC800,000Replay: 100k, Batch: 256,γ=0\.99\\gamma=0\.99,τSAC=0\.01\\tau\_\{\\text\{SAC\}\}=0\.01, Ent%: 0\.5DQN800,000Replay: 100k, Batch: 256,γ=0\.99\\gamma=0\.99,τ=0\.01\\tau=0\.01, Update: 4Dep\. ContinuousPPO1,500,000Buffer: 4096, Batch: 128, Epochs: 4,γ=0\.99\\gamma=0\.99, Clip: 0\.2,λGAE=0\.95\\lambda\_\{\\text\{GAE\}\}=0\.95SAC800,000Replay: 100k, Batch: 256,γ=0\.99\\gamma=0\.99,τSAC=0\.01\\tau\_\{\\text\{SAC\}\}=0\.01, Ent%: 0\.5DQN800,000Replay: 100k, Batch: 256,γ=0\.99\\gamma=0\.99,τ=0\.01\\tau=0\.01, Update: 4Indep\. ContinuousPPO1,500,000Buffer: 4096, Batch: 128, Epochs: 4,γ=0\.99\\gamma=0\.99, Clip: 0\.2,λGAE=0\.95\\lambda\_\{\\text\{GAE\}\}=0\.95SAC800,000Replay: 100k, Batch: 256,γ=0\.99\\gamma=0\.99,τSAC=0\.01\\tau\_\{\\text\{SAC\}\}=0\.01, Ent%: 0\.5DQN800,000Replay: 100k, Batch: 256,γ=0\.99\\gamma=0\.99,τ=0\.01\\tau=0\.01, Update: 4Platform\(Hidden Dims:
\[256, 256\]\)DiscretePPO500,000Buffer: 4096, Batch: 128, Epochs: 4,γ=0\.99\\gamma=0\.99, Clip: 0\.2,λGAE=0\.95\\lambda\_\{\\text\{GAE\}\}=0\.95SAC200,000Replay: 100k, Batch: 256,γ=0\.99\\gamma=0\.99,τSAC=0\.01\\tau\_\{\\text\{SAC\}\}=0\.01, Ent%: 0\.5DQN200,000Replay: 100k, Batch: 256,γ=0\.99\\gamma=0\.99,τ=0\.01\\tau=0\.01, Update: 4ContinuousPPO500,000Buffer: 4096, Batch: 128, Epochs: 4,γ=0\.99\\gamma=0\.99, Clip: 0\.2,λGAE=0\.95\\lambda\_\{\\text\{GAE\}\}=0\.95SAC200,000Replay: 100k, Batch: 256,γ=0\.99\\gamma=0\.99,τSAC=0\.01\\tau\_\{\\text\{SAC\}\}=0\.01, Ent%: 0\.5DQN200,000Replay: 100k, Batch: 256,γ=0\.99\\gamma=0\.99,τ=0\.01\\tau=0\.01, Update: 4HybridPPO500,000Buffer: 4096, Batch: 128, Epochs: 5,γ=0\.99\\gamma=0\.99, Clip: 0\.2,λGAE=0\.95\\lambda\_\{\\text\{GAE\}\}=0\.95SAC200,000Replay: 100k, Batch: 256,γ=0\.99\\gamma=0\.99,τSAC=0\.01\\tau\_\{\\text\{SAC\}\}=0\.01, Ent%: 0\.5DQN200,000Replay: 100k, Batch: 256,γ=0\.99\\gamma=0\.99,τ=0\.01\\tau=0\.01, Update: 4
Note:All experimental runs globally shared a learning rate \(l​rlr\) value of5\.0×10−45\.0\\times 10^\{\-4\}\.

## Appendix DAdditional Figures and Results

### D\.1\.Aggregated and Consolidated Results

![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/agg_dqn.png)![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/agg_dqn_continuous.png)![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/agg_dqn_discrete.png)![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/agg_dqn_hybrid.png)![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/agg_ppo.png)![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/agg_ppo_continuous.png)![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/agg_ppo_discrete.png)![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/agg_ppo_hybrid.png)![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/agg_sac.png)![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/agg_sac_continuous.png)![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/agg_sac_discrete.png)![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/agg_sac_hybrid.png)![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/cons_dqn.png)![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/cons_ppo.png)![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/cons_sac.png)
### D\.2\.Detailed Results by Environment and Action Space

![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/continuous_dependent_push_dqn.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/continuous_dependent_push_ppo.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/continuous_dependent_push_sac.png)

![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/continuous_dependent_shoot_dqn.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/continuous_dependent_shoot_ppo.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/continuous_dependent_shoot_sac.png)

![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/continuous_independent_shoot_dqn.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/continuous_independent_shoot_ppo.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/continuous_independent_shoot_sac.png)

![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/continuous_lander_dqn.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/continuous_lander_ppo.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/continuous_lander_sac.png)

![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/continuous_platform_dqn.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/continuous_platform_ppo.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/continuous_platform_sac.png)

![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/discrete_dependent_push_dqn.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/discrete_dependent_push_ppo.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/discrete_dependent_push_sac.png)

![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/discrete_dependent_shoot_dqn.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/discrete_dependent_shoot_ppo.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/discrete_dependent_shoot_sac.png)

![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/discrete_independent_shoot_dqn.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/discrete_independent_shoot_ppo.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/discrete_independent_shoot_sac.png)

![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/discrete_lander_dqn.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/discrete_lander_ppo.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/discrete_lander_sac.png)

![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/discrete_platform_dqn.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/discrete_platform_ppo.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/discrete_platform_sac.png)

![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/hybrid_dependent_push_dqn.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/hybrid_dependent_push_ppo.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/hybrid_dependent_push_sac.png)

![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/hybrid_dependent_shoot_dqn.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/hybrid_dependent_shoot_ppo.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/hybrid_dependent_shoot_sac.png)

![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/hybrid_independent_shoot_dqn.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/hybrid_independent_shoot_ppo.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/hybrid_independent_shoot_sac.png)

![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/hybrid_lander_dqn.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/hybrid_lander_ppo.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/hybrid_lander_sac.png)

![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/hybrid_platform_dqn.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/hybrid_platform_ppo.png)
![[Uncaptioned image]](https://arxiv.org/html/2606.26574v1/vectorized_graphs/hybrid_platform_sac.png)