重新审视复杂动作空间中的动作分解

# 重访复合动作空间的动作分解 来源：https://arxiv.org/html/2606.26574 ###### 摘要. 许多现实世界中的控制问题涉及混合离散-连续动作空间。例如，自动驾驶中的转向与信号控制，以及机器人或电子游戏中的瞄准与射击。尽管现实世界支持混合分解，且强化学习框架（如Gymnasium、PettingZoo、TorchRL、SeedRL、Mujoco等）也支持复杂动作空间，但这些框架中的默认环境通常采用统一的动作空间配置（如LunarLander、Walker2D、Cheetah、SMAC、SUMO、Ant、Atari）。标志性的混合动作基准（如RoboCup 2D HFO、SC2LE、Platform、CARLA等）大多是来自论文的重型或存档实现，这些论文仅测试一种或少数几种竞争性分解方法在单一控制类型上的表现。本文对三类算法系列[PPO、SAC、DQN]在三种动作空间[离散化、混合、连续]上，针对四种轻量级环境[Platform、hybrid-LunarLander、Hybrid-Shoot、CoopPush]，对多种分解方法[独立网络、共享编码器、VDN、QPLEX、联合、自回归]进行了横截面研究。考虑到一些无效配对（如联合-连续），我们共有220种配置需要分析每种方法。我们提供了两个新的符合Gymnasium和PettingZoo标准的C++并行环境[CoopPush、Hybrid-Shoot]，以隔离特定挑战，如状态相关的动作间依赖性。最后，我们引入了VDN-PPO和PPO-MIX，它们使用分支评论家为多头PPO分配信用。这些变体在所有测试的PPO分解中表现最佳。我们的结果表明，分支决斗架构在计算和性能之间取得了最有效的平衡，自回归动作达到了最高性能，而原生连续SAC虽然计算成本更高，但表现优于离散和混合算法。 动作分解、混合动作空间、DQN、SAC、PPO ††期刊:TAAS ††版权:无 ††ccs:计算方法论 强化学习## 1. 引言 尽管神经网络因其对依赖混合特征空间的泛化能力而在深度强化学习（D-RL）中被广泛接受，但对于如何分解复杂动作空间仍存在开放争议。在单智能体领域，子动作A1,...,AH\mathcal{A}_{1},\ldots,\mathcal{A}_{H}可能控制单个电机扭矩或按钮，这些组合起来形成智能体的完整联合动作空间A=∏h=1HAh\mathcal{A}=\prod_{h=1}^{H}\mathcal{A}_{h}（其大小为子动作数量的指数级）。在多智能体RL（MARL）中，动作A1,...,AK\mathcal{A}_{1},\ldots,\mathcal{A}_{K}指的是各个智能体的动作空间，而这些空间本身可能进一步分解为子动作。并非所有动作都是相互依赖的，因此完整的联合动作空间并不总是需要被探索。例如，避免危险的运动的价值可能不依赖于智能体看向何处或射击，但射击的决定高度依赖于智能体瞄准的位置。文献中有一系列方法涵盖了从完全独立到联合动作学习（JAL）的整个谱系。在本文中，我们涵盖以下一般策略： 1. (1) 独立：每个子动作从一个完全独立的策略网络πh(ah∣s)\pi_{h}(a_{h}\mid s)中采样，所有网络使用相同的全局训练信号进行更新。 2. (2) 共享编码器：一个共享的状态表示φ(s)\phi(s)分支到多个头： 1. (a) 无混合：每个动作头暴露于全局奖励信号，其他头的影响被视为独立噪声。梯度在编码器中合并，因此结果并非真正独立。 2. (b) 价值分解（VDN风格）：学习单个状态价值V(s)V(s)和每个头的优势Ah(s,ah)A_{h}(s,a_{h})。联合动作价值通过加法分解：Q(s,a)=V(s)+∑h=1HAh(s,ah)Q(s,\mathbf{a})=V(s)+\sum_{h=1}^{H}A_{h}(s,a_{h})。 3. (c) 单调（QPLEX风格）：学习单个状态价值V(s)V(s)，并通过一个单调混合网络组合每个头的优势：Q(s,a)=V(s)+fmix(A1(s,a1),…,AH(s,aH))Q(s,\mathbf{a})=V(s)+f_{\text{mix}}\big(A_{1}(s,a_{1}),\dots,A_{H}(s,a_{H})\big)，其中要求∂fmix∂Ah≥0\frac{\partial f_{\text{mix}}}{\partial A_{h}}\geq 0。 4. (d) 拼接SAC（SAC-Concat）：演员通过压缩高斯分布联合采样连续维度，通过Gumbel-Softmax采样离散维度。评论家将这些视为单个输入向量a=[ac,ad]\mathbf{a}=[a_{c},a_{d}]来近似一个统一的动作价值Q(s,a)Q(s,\mathbf{a})。 5. (e) 分支决斗SAC（SAC-BDQ）：评论家使用一个以连续动作为条件的分支架构。网络编码状态和连续动作来计算一个基础价值V(s,ac)V(s,a_{c})，并分支输出每个分类头的离散优势Ah(s,ad,h)A_{h}(s,a_{d,h})，得到Q(s,ac,ad)=V(s,ac)+∑h=1HAh(s,ad,h)Q(s,a_{c},\mathbf{a}_{d})=V(s,a_{c})+\sum_{h=1}^{H}A_{h}(s,a_{d,h})。 3. (3) 自回归（AR）：联合策略通过链式法则顺序分解。每个动作维度hh以状态ss和之前采样的动作a0I_{h}(s)>0I_{h}(s)>0 for all hh (generically true after exploration), low-importance heads satisfy (whα)2<1/H2(w_{h}^{\alpha})^{2}<1/H^{2}, strictly reducing their variance contribution, while heads with Ih=0I_{h}=0 receive wh=0w_{h}=0 and inject no noise at all. The freed variance budget is reallocated to high-importance heads where signal dominates noise. Operationally, the annealing schedule interpolates between these regimes: early in training α≈0\alpha\approx 0 gives uniform credit (standard GAE), and as the critic converges α→1\alpha\to 1 concentrates credit on the heads whose sub-actions most affect the QQ-value, suppressing pure noise from heads with no agency at the current state. #### 5.2.2. PPO-MIX: Monotonic Non-Linear Advantage Mixing The additive structure (2 (https://arxiv.org/html/2606.26574#S5.E2)) assumes each head contributes to the joint advantage independently. When sub-actions are strictly coupled (e.g., aiming and firing), such interactions may be unrepresentable. FACMAC (Peng et al., 2021 (https://arxiv.org/html/2606.26574#bib.bib55)) shows that non-monotonic factorization can represent tasks monotonic methods cannot; PPO-MIX instead retains monotonic mixing to preserve the IGM property and the tractability of per-head greedy action selection, trading representational capacity for that guarantee. It replaces the linear sum with the QPLEX-style mixer (Wh=∂fmix/∂AhW_{h}=\partial f_{\text{mix}}/\partial A_{h} in the QS table), (7) Qφ(s,a)=Vφ(s)+fmix(Aφ,1(s,a1),…,Aφ,H(s,aH)∣s),∂fmix∂Ah≥0,Q_{\phi}(s,\mathbf{a})=V_{\phi}(s)+f_{\text{mix}}\left(A_{\phi,1}(s,a_{1}),\dots,A_{\phi,H}(s,a_{H})\mid s\right),\quad\frac{\partial f_{\text{mix}}}{\partial A_{h}}\geq 0, where fmixf_{\text{mix}} is realized by a state-conditioned hypernetwork with strictly non-negative weights, guaranteeing monotonicity and hence IGM. Credit assignment now requires the gradient of the mixed output at the *current joint action*. A first-order Taylor expansion of fmixf_{\text{mix}} gives the per-head credit (8) w~h(s,a)=∂fmix∂Ah|a⋅Aφ,h(s,ah),whmix(s,a)=w~h∑kw~k.\tilde{w}_{h}(s,\mathbf{a})=\frac{\partial f_{\text{mix}}}{\partial A_{h}}\bigg|_{\mathbf{a}}\!\cdot A_{\phi,h}(s,a_{h}),\qquad w_{h}^{\text{mix}}(s,\mathbf{a})=\frac{\tilde{w}_{h}}{\sum_{k}\tilde{w}_{k}}. Evaluating ∂fmix/∂Ah\partial f_{\text{mix}}/\partial A_{h} at the sampled action is *necessary*: the mixer is non-linear in the advantages, so the local gradient depends on where in advantage-space the joint action falls. A state-only surrogate, evaluating the gradient at the advantage means, which are zero by construction—would collapse to a degenerate zero weight. COMA (Foerster et al., 2018 (https://arxiv.org/html/2606.26574#bib.bib25)) compares the zero value to the value at the action as a finite difference instead of the Taylor approximation that we employ. ##### Variance cost of action-dependent weights. This action-dependence breaks the factorization that underwrites VDN-PPO’s unbiasedness. Marginalizing over future actions at step t+lt+l, the expectation of the weighted residual no longer decomposes (conditioning on st+ls_{t+l} left implicit): (9) Eat+l[whmix(st+l,at+l)δt+lV]=E[whmix]E[δt+lV]+Cov(whmix,δt+lV)⏟≠0.\mathbb{E}_{\mathbf{a}_{t+l}}\left[w_{h}^{\text{mix}}(s_{t+l},\mathbf{a}_{t+l})\;\delta_{t+l}^{V}\right]=\mathbb{E}[w_{h}^{\text{mix}}]\,\mathbb{E}[\delta_{t+l}^{V}]+\underbrace{\mathrm{Cov}\left(w_{h}^{\text{mix}},\;\delta_{t+l}^{V}\right)}_{\neq\,0}. Because whmixw_{h}^{\text{mix}} and δt+lV\delta_{t+l}^{V} share the sampled action at+l\mathbf{a}_{t+l}—actions producing large advantages also inflate the mixing gradient—the covariance is generically non-zero. Two consequences follow: (i) the per-head estimates no longer sum to standard GAE, since ∑h∂fmix/∂Ah≠1\sum_{h}\partial f_{\text{mix}}/\partial A_{h}\neq 1 for a non-linear mixer, so the requirements for policy gradient validity in Theorem A.11 (https://arxiv.org/html/2606.26574#A1.Thmtheorem11) are not met, and (ii) the residual covariance injects additional variance into every head’s estimate beyond what VDN weighting produces. In short, PPO-MIX trades the theoretical guarantees of VDN-PPO for the representational capacity needed to model strongly coupled sub-actions. Section 7.1 (https://arxiv.org/html/2606.26574#S7.SS1) shows that PPO-MIX can learn state-dependent importance, and in practice it outperforms VDN slightly for continuous action spaces while matching it for discrete actions. ##### Continuous Action Advantage Centering. The mechanism of computing bin masses via the CDF is theoretically necessary to preserve the on-policy value baseline Vφ(s)V_{\phi}(s) in continuous domains. By definition, the value function and advantages must satisfy V(s)=Ea[Q(s,a)]V(s)=\mathbb{E}_{\mathbf{a}}[Q(s,\mathbf{a})], which requires the expected advantage to be strictly zero. In discrete action spaces, this constraint is trivially satisfied by evaluating the sum V(s)=V(s)+∑aπ(a|s)A(s,a)V(s)=V(s)+\sum_{a}\pi(a|s)A(s,a), allowing us to center the advantages by scaling each discrete action’s advantage by its corresponding policy probability πa\pi_{a}. However, for continuous actions, this discrete summation is mathematically invalid and the expectation instead demands taking the integral over the action dimension, ∫π(a|s)A(s,a)da=0\int\pi(a|s)A(s,a)\,da=0. By discretizing the continuous advantage function into intervals and evaluating the Cumulative Distribution Function at the bin boundaries, we analytically integrate the policy’s probability density over each segment to find the exact probability mass mh,cm_{h,c}. Using this mass as the proportion by which to scale the bin’s advantage acts as a rigorous, tractable approximation of the continuous integral. This ensures the continuous advantage streams are correctly zero-centered under the current policy without introducing the high variance that would result from relying on Monte Carlo action sampling to approximate the expectation. ### 5.3. Hybrid SAC We adapt SAC to complex action spaces with two variants, both drawn from the factorization wiring diagram. For monolithic evaluation (SAC-Concat) (Delalleau et al., 2019 (https://arxiv.org/html/2606.26574#bib.bib17)), the state and both sub-actions are concatenated into the standard critic, using a Gumbel-Softmax (Jang et al., 2016 (https://arxiv.org/html/2606.26574#bib.bib26)) relaxation to keep the discrete branch differentiable. Credit assignment through the deep gradient is natural here, but the entropy signals must be separated: a single target entropy drives one distribution to collapse while the other explodes, so we maintain distinct temperature coefficients αd,αc\alpha_{d},\alpha_{c} for the discrete and continuous dimensions as in (Delalleau et al., 2019 (https://arxiv.org/html/2606.26574#bib.bib17)). Alternatively, SAC-BDQ (hybrid-sac (Chen et al., 2022 (https://arxiv.org/html/2606.26574#bib.bib43)) + branching dueling) routes discrete sub-actions into the QS critic, drawing them from a softmax over advantages as in Munchausen Q-learning (Vieillard et al., 2020 (https://arxiv.org/html/2606.26574#bib.bib7))—analogous to P-DQN (Xiong et al., 2018 (https://arxiv.org/html/2606.26574#bib.bib30)) under entropy constraints—so that the branching critic itself performs the discrete credit assignment. SAC-BDQ reduces to Discrete-SAC for the discrete action space. ## 6. Experimental Setup On each of the environments shown in Figure 1 (https://arxiv.org/html/2606.26574#S4.F1), we analyze Discrete, Hybrid and continuous action types. For the hybrid action configurations, the first DD actions are discrete and the following CC actions are continuous. Table 1. Action space factorizations per environment. Dk\mathcal{D}_{k} denotes a discrete action head of cardinality kk, and Rn\mathbb{R}^{n} denotes a continuous action head of nn dimensions.### 6.1. Hyperparameter Configuration Across all evaluated environments, we established a standardized set of core hyperparameters for Proximal Policy Optimization (PPO), Soft Actor-Critic (SAC), and Deep Q-Networks (DQN). The neural network architectures utilized two hidden layers of 256 units each for the Push, Shoot, and Platform tasks, while the Lander tasks utilized 128 units per layer. A uniform learning rate of 5.0×10−45.0\times 10^{-4} was maintained across all algorithms and environments. The discount factor (γ\gamma) was set to 0.980.98 for all Push tasks and 0.990.99 for the remaining environments. Due to differing sample efficiencies and task complexities, the total maximum environment steps (max\_steps) varied by environment based on the number of steps required for the canonical versions of each algorithm to converge (Discrete PPO, BD-DQN, SAC-Concat). Notably, SAC and DQN required identical step counts for convergence across all configurations, while PPO consistently required higher sample limits (offset computationally by parallel environment execution where collection time becomes 1/20th runtime for all algorithms), as summarized in Table 2 (https://arxiv.org/html/2606.26574#S6.T2). Table 2. Total Environment Interactions (max\_steps) by Task and Algorithm ### 6.2. Contextual Decomposition Verification The Contextual-Decoupler of Section 4 (https://arxiv.org/html/2606.26574#S4) is a contextual bandit (γ\gamma-discounted but i.i.d. in state) whose ground-truth per-head credit assignment is known in closed form: the active head h=ch=c drives the dominant ±1\pm 1 reward while the inactive head h≠ch\neq c contributes only a −0.1-0.1 shaping penalty, so the true importance is Ih⋆(s)=I[h=c]I_{h}^{\star}(s)=\mathbb{I}[h=c] and the active head’s ±1\pm 1 variance acts as pure noise on the inactive head’s policy-gradient signal. This is precisely