HiComm: Hierarchical Communication for Multi-agent Reinforcement Learning

# HiComm: Hierarchical Communication for Multi-agent Reinforcement Learning
Source: [https://arxiv.org/html/2606.29126](https://arxiv.org/html/2606.29126)
Runze Zhao1Dongruo Zhou1Sumit Kumar Jha2Nathaniel D\. Bastian3Ankit Shah41Luddy School of Informatics, Computing, and Engineering, Indiana University Bloomington, Bloomington, IN 474082Department of Computer & Information Science & Engineering, University of Florida, Gainesville, FL 326113Department of Electrical Engineering and Computer Science, United States Military Academy, West Point, NY 109964Department of Operations and Decision Technologies, Indiana University Bloomington, Bloomington, IN 47405

###### Abstract

Cooperative multi\-agent reinforcement learning \(MARL\) often relies on communication to mitigate partial observability, yet most existing protocols treat messages as flat dense vectors detached from the structure of the observations they summarize\. This design overlooks an important source of inductive bias in many cooperative environments, where observations naturally follow a hierarchy such as groups and entities\. We proposeHiComm, a plug\-in communication module that grounds messages in the sender’s hierarchical observation\.HiCommis receiver\-driven: the receiver issues a query, and the hierarchy is resolved through a three\-stage decoding process that first selects a group, then a sender, and then an entity within that group, returning the corresponding feature slice as the message\. This converts communication from unstructured vector transmission into structured information retrieval over the sender’s observation hierarchy\. We instantiate this mechanism with Straight\-Through Gumbel\-Softmax for differentiable discrete selection and a lightweight shared projection design that attaches to standard MARL pipelines\. Experiments across cooperative MARL tasks with different observation structures and coordination demands show thatHiCommmatches or outperforms representative learned communication baselines while reducing communication volume by up to23×23\\timesper receiver per episode\.

## 1Introduction

Cooperative multi\-agent reinforcement learning \(MARL\) studies how a team of decentralized agents can jointly maximize a shared return when each agent observes only part of the environment\. Compared with single\-agent RL, the central challenge in cooperative MARL arises from multi\-agency under partial observability: each agent must make decisions based on its own local observation, while the information needed for effective coordination may be distributed across teammates\. Communication provides a natural mechanism to mitigate this bottleneck\. By exchanging previously observed information, agents can avoid inferring hidden state solely from teammates’ actions, reduce redundant exploration of the joint state space, and learn coordinated policies with fewer environment interactions\[[31](https://arxiv.org/html/2606.29126#bib.bib1),[6](https://arxiv.org/html/2606.29126#bib.bib2),[11](https://arxiv.org/html/2606.29126#bib.bib16),[2](https://arxiv.org/html/2606.29126#bib.bib3),[8](https://arxiv.org/html/2606.29126#bib.bib5)\]\.

In this work, we focus on cooperative environments whose observations exhibitrich hierarchical structure\. A representative example is cyber defense, as instantiated by the CAGE Challenge 4 environment\[[13](https://arxiv.org/html/2606.29126#bib.bib18),[12](https://arxiv.org/html/2606.29126#bib.bib19)\]\. In such settings, hosts are organized into subnets, agents may observe only their assigned hosts or subnets, and effective coordination is required to track activity across the system\. At enterprise scale, with thousands of assets and rapidly evolving events, such coordination must be both efficient and scalable\. These constraints apply equally to coordinated attackers \(red teams\), which operate under similar partial observability and distributed control; consequently, effective red\-team evaluation depends on communication mechanisms that capture these limitations while enabling coherent multi\-agent behavior\. However, designing communication protocols that efficiently exploit these hierarchical observation structures remains challenging\.

Existing communication methods largely treat messages as flat objects\. A sender typically encodes its local observation into a fixed\-width vector, and the receiver consumes this vector without explicitly indexing the structure of the underlying observation\. A related line of hierarchical communication has explored hierarchy in other parts of the system: feudal architectures stratify the agent population into managers and workers\[[1](https://arxiv.org/html/2606.29126#bib.bib42),[21](https://arxiv.org/html/2606.29126#bib.bib43)\], while clustered or order\-based protocols impose hierarchy on the communication graph or the message\-passing schedule\[[26](https://arxiv.org/html/2606.29126#bib.bib9),[20](https://arxiv.org/html/2606.29126#bib.bib10)\]\. However, these approaches do not organize the message body according to the hierarchy of the sender’s observation\. As a result, a readily available source of structure in many cooperative environments remains unused\.

We address this gap withHiComm, a communication module for cooperative MARL that grounds each message in the sender’s hierarchical observation\. The module is receiver\-driven: a receiver first issues a query, and the hierarchy is resolved through a three\-stage decoding process\. Specifically, the receiver first selects a group, then a sender, and the addressed sender finally selects an entity within that group on its own observation, returning the corresponding feature slice as the message\. In this way,HiCommturns communication from transmitting an unstructured dense vector into retrieving structured information from the sender’s observation hierarchy\. Our contributions are listed as follows:

- •We proposeHiComm, a plug\-in communication module for cooperative MARL settings whose observations admit a natural group\-entity hierarchy\. Instead of transmitting an unstructured dense vector,HiCommlets the receiver query a group\-entity location in a selected sender’s observation hierarchy, and the sender returns the corresponding feature slice as the message\.
- •We instantiate this hierarchical query\-response mechanism as a differentiable and lightweight module\.HiCommuses Straight\-Through Gumbel\-Softmax for discrete group, sender, and entity selection, shares a single projection head across hierarchical encoders, and keeps the receiver query compact, allowing it to attach to standard MARL training pipelines without changing the underlying policy or critic architecture\.
- •We empirically validateHiCommacross cooperative MARL tasks with different observation structures and coordination demands\. As an add\-on communication module,HiCommmatches or outperforms representative learned communication baselines while reducing communication volume by up to23×23\\timesper receiver per episode\.

## 2Related Work

Cooperative MARL under partial observability has long relied on interagent communication\[[6](https://arxiv.org/html/2606.29126#bib.bib2),[31](https://arxiv.org/html/2606.29126#bib.bib1)\], with prior work refining the practice along three axes\. On*message content*, NDQ\[[33](https://arxiv.org/html/2606.29126#bib.bib20)\], VBC\[[39](https://arxiv.org/html/2606.29126#bib.bib48)\], and TMC\[[40](https://arxiv.org/html/2606.29126#bib.bib49)\]regularise messages to be succinct, while MAIC\[[37](https://arxiv.org/html/2606.29126#bib.bib35)\]and CACOM\[[17](https://arxiv.org/html/2606.29126#bib.bib12)\]condition each sender’s projection on a model of the intended receiver\. On*scheduling*, IC3Net\[[28](https://arxiv.org/html/2606.29126#bib.bib50)\]and ATOC\[[11](https://arxiv.org/html/2606.29126#bib.bib16)\]let agents gate themselves silent, SchedNet\[[14](https://arxiv.org/html/2606.29126#bib.bib51)\]and SeqComm\[[4](https://arxiv.org/html/2606.29126#bib.bib13)\]prioritise or sequence communicators across the team, and Who2com\[[19](https://arxiv.org/html/2606.29126#bib.bib57)\]together with When2com\[[18](https://arxiv.org/html/2606.29126#bib.bib58)\]use a multi\-stage handshake in which the receiver requests, candidate senders return a brief match score, and the receiver finally connects to the best\-matching peer\. On*message integration*, TarMAC\[[2](https://arxiv.org/html/2606.29126#bib.bib3)\]and SARNet\[[23](https://arxiv.org/html/2606.29126#bib.bib4)\]apply receiver\-side soft attention over incoming messages, MASIA\[[7](https://arxiv.org/html/2606.29126#bib.bib8)\]aggregates teammate embeddings with permutation\-invariant self\-supervised objectives, T2MAC\[[32](https://arxiv.org/html/2606.29126#bib.bib11)\]weighs evidence under uncertainty, and CoDe\[[29](https://arxiv.org/html/2606.29126#bib.bib15)\]aligns intent with timeliness for delay\-robust collaboration\. A parallel line learns the routing graph itself: DGN\[[10](https://arxiv.org/html/2606.29126#bib.bib59)\]convolves over a multi\-head attention graph, DICG\[[16](https://arxiv.org/html/2606.29126#bib.bib60)\]infers an implicit adjacency via self\-attention, CASEC\[[34](https://arxiv.org/html/2606.29126#bib.bib61)\]sparsifies the coordination graph using payoff\-variance scores, MAGIC\[[22](https://arxiv.org/html/2606.29126#bib.bib7)\], CommFormer\[[8](https://arxiv.org/html/2606.29126#bib.bib5)\], and TGCNet\[[41](https://arxiv.org/html/2606.29126#bib.bib14)\]learn graph attention, differentiable adjacency, or a dynamic directed graph, and G\-Designer\[[38](https://arxiv.org/html/2606.29126#bib.bib63)\]learns a task\-adaptive topology via a variational graph auto\-encoder\. Two cross\-cutting properties hold across this entire body of work\. First, the protocols are sender\-driven by default: the sender unilaterally decides what to encode, and even MAIC and CACOM adjust the sender’s projection toward an intended receiver rather than letting the receiver actively query\. Second, the channel payload is a learned, and therefore lossy, projection of the sender’s observation into a fixed\-width vector, decoupled from any structure intrinsic to that observation; even the Who2com and When2com handshakes select*whom*to ask without specifying*which part*of the sender’s observation to return\. A complementary efficiency line attacks bandwidth from outside the message form: NDQ, VBC, and TMC compress the payload, CASEC and message\-pruning approaches sparsify the graph, and graph\-learning methods like CommFormer and G\-Designer thin out edges, but none of them changes the form of what is sent\.

A subset of MARL work labels its mechanism as*hierarchical*, but the hierarchy invariably lives outside the message body\. Feudal Multiagent Hierarchies\[[1](https://arxiv.org/html/2606.29126#bib.bib42)\]stratify the agent pool into managers and workers via subgoal signals, and GMAH\[[35](https://arxiv.org/html/2606.29126#bib.bib44)\]keeps the same vertical decomposition inside each agent’s policy stack with a QMIX\-style goal\-mixing network in lieu of any explicit channel; HiMPo\[[21](https://arxiv.org/html/2606.29126#bib.bib43)\]marries this feudal layout with peer\-to\-peer GNN exchange of flat hidden states\. LSC\[[26](https://arxiv.org/html/2606.29126#bib.bib9)\]hierarchises the routing graph by electing group leaders and propagating learned embeddings within and between groups, and HAMA\[[24](https://arxiv.org/html/2606.29126#bib.bib62)\]hierarchises the attention itself by stacking inter\-agent and inter\-group attention layers over a fixed agent partition\. DHCG\[[20](https://arxiv.org/html/2606.29126#bib.bib10)\]hierarchises the message\-passing order via a learned DAG with edges carrying flat action and trajectory embeddings\. In every case the hierarchy sits on agents, attention, routing, order, or the policy stack, and the transmitted content is either an abstract subgoal token or a flat hidden\-state embedding, never indexed against the observation hierarchy itself\. That last dimension is whereHiCommintroduces structure: the receiver actively issues a query that selects a coordinate in the sender’s intrinsic group\-then\-entity partition, and the response is delivered*verbatim*as the raw feature vector at that coordinate, exchanging the encoder’s lossy projection for an in\-distribution observation entry \(the precise payload form is given in Sec\.[4](https://arxiv.org/html/2606.29126#S4)\)\.

## 3Preliminaries

##### Cooperative MARL with communication\.

We study cooperative multi\-agent tasks with partial observability and learnable communication, formalised as a tupleℳ=⟨S,U,P,Z,O,N,r,γ⟩\\mathcal\{M\}=\\langle S,U,P,Z,O,N,r,\\gamma\\rangle\. HereSSis the global state space,UUis the per\-agent action space, andN∈ℕN\\in\\mathbb\{N\}is the number of agents\. The transition kernelP​\(s′∣s,𝐮\)P\(s^\{\\prime\}\\mid s,\\mathbf\{u\}\)governs the dynamics under joint actions𝐮∈UN\\mathbf\{u\}\\in U^\{N\}, the functionr:S×UN→ℝr:S\\times U^\{N\}\\to\\mathbb\{R\}is the team\-shared reward, andγ∈\[0,1\)\\gamma\\in\[0,1\)is the discount factor\. Each agent perceives the environment through the per\-agent observation spaceZZvia the observation functionO:S×\{1,…,N\}→ZO:S\\times\\\{1,\\dots,N\\\}\\to Z\.

At every step, agentiireceives a local observationzi=O​\(s,i\)z^\{i\}=O\(s,i\), exchanges a fixed\-width messagemi∈ℝdmm^\{i\}\\in\\mathbb\{R\}^\{d\_\{m\}\}with its peers, and selects an action according toπi​\(ui∣zi,m1:N\)\\pi^\{i\}\(u^\{i\}\\mid z^\{i\},m^\{1:N\}\), wherem1:N=\(m1,…,mN\)m^\{1:N\}=\(m^\{1\},\\dots,m^\{N\}\)collects the per\-agent message slots\. When self\-loops are excluded, we setmi=𝟎m^\{i\}=\\mathbf\{0\}on the self\-slot by convention, anddm∈ℕd\_\{m\}\\in\\mathbb\{N\}denotes the message bandwidth\. We call an observation*flat*if it is represented as a single ungrouped list of entity slots, with all entities placed at the same address level\. In a flat observation, any notion of which entities form a meaningful unit must be inferred from features during training\.

##### Hierarchical observations\.

We considerhierarchical observationziz^\{i\}in this work\. We assume that each agent’s observation is a three\-axis hierarchical tensorzi∈𝒢×ℰ×ℱz^\{i\}\\in\{\\mathcal\{G\}\\times\\mathcal\{E\}\\times\\mathcal\{F\}\}, where\|𝒢\|,\|ℰ\|,\|ℱ\|∈ℕ\>0\|\\mathcal\{G\}\|,\|\\mathcal\{E\}\|,\|\\mathcal\{F\}\|\\in\\mathbb\{N\}\_\{\>0\}, and write\[K\]:=\{1,…,K\}\[K\]:=\\\{1,\\dots,K\\\}for an axis of sizeKK\. The hierarchy consists of the following components:

- •𝒢\\mathcal\{G\}is the coarse*group*axis, indexing𝒢\\mathcal\{G\}groups\.
- •ℰ\\mathcal\{E\}is the finer*entity*axis, indexingℰ\\mathcal\{E\}entities within each group\.
- •ℱ\\mathcal\{F\}is the per\-entity*feature*axis, storing a feature vector for each group–entity pair\.

Thus, an entryzi​\[g,ℓ,:\]z^\{i\}\[g,\\ell,:\]is addressed by a group indexg∈\[𝒢\]g\\in\[\\mathcal\{G\}\]and an entity indexℓ∈\[ℰ\]\\ell\\in\[\\mathcal\{E\}\], and returns the feature vectorF:=zi​\[g,ℓ,:\]∈ℝ\|ℱ\|F:=z^\{i\}\[g,\\ell,:\]\\in\\mathbb\{R\}^\{\|\\mathcal\{F\}\|\}\. Hereggandℓ\\ellare discrete coordinates, whileFFis the continuous feature vector stored at the coordinate\(g,ℓ\)\(g,\\ell\)\.

The partition\[𝒢\]×\[ℰ\]\[\\mathcal\{G\}\]\\times\[\\mathcal\{E\}\]is supplied by the environment rather than invented by the model\. It may arise in either of two ways:

- •as a*native hierarchy*, where the observation tensor already has separate group and entity axes;
- •as a*semantic hierarchy*, where an observable per\-entity attribute induces the group label\.

This hierarchy frames the communication problem studied in this paper: givenNNagents whose observations are jointly indexed by\[𝒢\]×\[ℰ\]\[\\mathcal\{G\}\]\\times\[\\mathcal\{E\}\], can each per\-round messagemi∈ℝdmm^\{i\}\\in\\mathbb\{R\}^\{d\_\{m\}\}be constructed from the environment\-provided partition, namely from a selected coordinate\(g,ℓ\)\(g,\\ell\)and the feature vectorzi​\[g,ℓ,:\]z^\{i\}\[g,\\ell,:\], rather than from a flattened sender\-side code that ignores this structure?

##### Communication on hierarchical observations\.

We call a communication protocol*observation\-space hierarchical*if every transmitted message is an address–value pair\(\(g,ℓ\),F\)\\big\(\(g,\\ell\),\\,F\\big\)whose valueF=z​\[g,ℓ,:\]∈ℝℱF=z\[g,\\ell,:\]\\in\\mathbb\{R\}^\{\\mathcal\{F\}\}is the verbatim feature vector at the coordinate\(g,ℓ\)∈\[𝒢\]×\[ℰ\]\(g,\\ell\)\\in\[\\mathcal\{G\}\]\\\!\\times\\\!\[\\mathcal\{E\}\], with the address drawn from a*factored*discrete distributionp​\(g\)​p​\(ℓ∣g\)p\(g\)\\,p\(\\ell\\mid g\)over the same partition that organises the observation, rather than the message being a flat sender\-side encoding of the whole tensor or the address being drawn from a single flat𝒢⋅ℰ\\mathcal\{G\}\\\!\\cdot\\\!\\mathcal\{E\}\-way categorical\. The protocol then addresses information at the same granularity at which the observation is already organised, and the open design problem reduces to specifyingp​\(g\)​p​\(ℓ∣g\)p\(g\)\\,p\(\\ell\\mid g\)jointly with the policy\. InNN\-agent settings the same hierarchy can be reused to choose*whom*to query: drawing a peerj∈\[N\]∖\{i\}j\\in\[N\]\\setminus\\\{i\\\}conditional on the topicggviap​\(j∣g\)p\(j\\mid g\)extends the descent top​\(g\)​p​\(j∣g\)​p​\(ℓ∣g,j\)p\(g\)\\,p\(j\\mid g\)\\,p\(\\ell\\mid g,j\), factoring sender selection through the same partition rather than along an orthogonal axis\.

## 4Methodology

Algorithm 1Cooperative MARL withHiCommCommunication Module0:Actor

πΘ\\pi\_\{\\Theta\}with

Θ=\(θpol,θcomm\)\\Theta=\(\\theta\_\{\\text\{pol\}\},\\theta\_\{\\text\{comm\}\}\); critic

VϕV\_\{\\phi\}; surrogate loss

ℒ​\(Θ,ϕ\)\\mathcal\{L\}\(\\Theta,\\phi\)supplied by the outer cooperative MARL algorithm; horizon

TT, iterations

II
1:foriteration

=1,…,I=1,\\dots,Ido

2:

𝒟←∅\\mathcal\{D\}\\leftarrow\\varnothing
3:for

t=0,…,T−1t=0,\\dots,T\-1do

4:Observe

\{zti,Mti\}i=1N\\\{z^\{i\}\_\{t\},M^\{i\}\_\{t\}\\\}\_\{i=1\}^\{N\}
5:

\{z~ti\}i=1N←HiComm​\(\{zti,Mti\}i=1N;θcomm\)\\\{\\widetilde\{z\}^\{i\}\_\{t\}\\\}\_\{i=1\}^\{N\}\\leftarrow\\textsc\{HiComm\}\\big\(\\\{z^\{i\}\_\{t\},M^\{i\}\_\{t\}\\\}\_\{i=1\}^\{N\};\\,\\theta\_\{\\text\{comm\}\}\\big\)\{Augment obs space via Alg\.[2](https://arxiv.org/html/2606.29126#alg2)\}

6:

uti∼πθpol\(⋅∣z~ti\)u^\{i\}\_\{t\}\\sim\\pi\_\{\\theta\_\{\\text\{pol\}\}\}\(\\cdot\\mid\\widetilde\{z\}^\{i\}\_\{t\}\)for each

i=1,…,Ni=1,\\dots,N\{decentralised per\-agent action sampling\}

7:Execute

𝐮t\\mathbf\{u\}\_\{t\}; observe

rtr\_\{t\}; append

\(\{zti,Mti,z~ti,uti\}i,rt\)\\big\(\\\{z^\{i\}\_\{t\},M^\{i\}\_\{t\},\\widetilde\{z\}^\{i\}\_\{t\},u^\{i\}\_\{t\}\\\}\_\{i\},\\,r\_\{t\}\\big\)to

𝒟\\mathcal\{D\}
8:endfor

9:Update

\(Θ,ϕ\)\(\\Theta,\\phi\)by minimizing

ℒ​\(Θ,ϕ\)\\mathcal\{L\}\(\\Theta,\\phi\)on

𝒟\\mathcal\{D\}
10:endfor

11:return

\(Θ,ϕ\)\(\\Theta,\\phi\)

We introduceHiComm, a communication protocol that turns the hierarchical structure of the observation space introduced in §[3](https://arxiv.org/html/2606.29126#S3)into the addressing space of every transmitted message\.HiCommis a learned, stateless transform parameterised byθcomm\\theta\_\{\\text\{comm\}\}that, at every environment step, ingests the joint observation\{zti,Mti\}i=1N\\\{z^\{i\}\_\{t\},M^\{i\}\_\{t\}\\\}\_\{i=1\}^\{N\}, whereMti∈\{0,1\}\|𝒢\|×\|ℰ\|M^\{i\}\_\{t\}\\in\\\{0,1\\\}^\{\|\\mathcal\{G\}\|\\times\|\\mathcal\{E\}\|\}is agentii’s observability mask over the\[𝒢\]×\[ℰ\]\[\\mathcal\{G\}\]\\\!\\times\\\!\[\\mathcal\{E\}\]partition, and returns the joint augmented observation\{z~ti\}i=1N\\\{\\widetilde\{z\}^\{i\}\_\{t\}\\\}\_\{i=1\}^\{N\}that the decentralised policy then consumes\. Because the transform leaves every other component ofℳ\\mathcal\{M\}unchanged,HiCommis independent of the cooperative MARL algorithm wrapped around it: its parameters ride on the surrounding actor\-critic gradient, complemented by a scalar information\-gain shaping prior added to the actor loss\. §[4\.1](https://arxiv.org/html/2606.29126#S4.SS1)fixes this outer loop and shows whereHiCommplugs in \(Algorithm[1](https://arxiv.org/html/2606.29126#alg1)\); §[4\.2](https://arxiv.org/html/2606.29126#S4.SS2)specifies the three\-phaseHiCommstep that builds messages from the\[𝒢\]×\[ℰ\]\[\\mathcal\{G\}\]\\\!\\times\\\!\[\\mathcal\{E\}\]partition \(Algorithm[2](https://arxiv.org/html/2606.29126#alg2)\)\. Throughout,score​\(q,k\)=⟨q,k⟩\\mathrm\{score\}\(q,k\)=\\langle q,k\\ranglerefers to the dot\-product scoring, and ST\-GS refers to the Straight\-Through Gumbel\-Softmax estimator\[[9](https://arxiv.org/html/2606.29126#bib.bib29)\], which returns a one\-hot sample on the forward pass and a temperature\-τ\\tausoft surrogate on the backward pass\.

### 4\.1MARL Framework

##### Cooperative MARL backbone\.

We build on the standard cooperative MARL rollout\-and\-update framework, which we briefly review for completeness\. At each training iteration, the agents jointly execute a horizon\-TTrollout in the environmentℳ\\mathcal\{M\}under the current policyπθpol\\pi\_\{\\theta\_\{\\mathrm\{pol\}\}\}, and store the resulting transitions in a replay buffer𝒟\\mathcal\{D\}\. A surrogate objectiveℒ\\mathcal\{L\}is then constructed from𝒟\\mathcal\{D\}and optimized using a standard cooperative MARL algorithm, such as IPPO\[[3](https://arxiv.org/html/2606.29126#bib.bib22)\]or MAPPO\[[36](https://arxiv.org/html/2606.29126#bib.bib23)\]\. This backbone determines how the policy and critic are updated, while leaving the communication mechanism modular\.

##### WhereHiCommplugs in\.

HiCommpreserves this rollout\-and\-update loop and inserts a differentiable communication transform between the joint observation and the per\-agent action sampling step:

\{z~ti\}i=1N←HiComm​\(\{zti,Mti\}i=1N;θcomm\)⏟Algorithm[2](https://arxiv.org/html/2606.29126#alg2)⟶uti∼πθpol\(⋅∣z~ti\)\(∀i\)⏟decentralized action sampling\.\\underbrace\{\\big\\\{\\widetilde\{z\}^\{i\}\_\{t\}\\big\\\}\_\{i=1\}^\{N\}\\leftarrow\\textsc\{HiComm\}\\Big\(\\big\\\{z^\{i\}\_\{t\},\\,M^\{i\}\_\{t\}\\big\\\}\_\{i=1\}^\{N\};\\,\\theta\_\{\\mathrm\{comm\}\}\\Big\)\}\_\{\\text\{Algorithm~\\ref\{alg:hicomm\}\}\}\\;\\longrightarrow\\;\\underbrace\{u^\{i\}\_\{t\}\\sim\\pi\_\{\\theta\_\{\\mathrm\{pol\}\}\}\\\!\\big\(\\cdot\\mid\\widetilde\{z\}^\{i\}\_\{t\}\\big\)\\quad\(\\forall i\)\}\_\{\\text\{decentralized action sampling\}\}\.Herez~ti\\widetilde\{z\}^\{i\}\_\{t\}denotes the augmented observation used by agentiifor action selection\. It concatenates the local observationztiz^\{i\}\_\{t\}with fixed\-size message slots of widthdmd\_\{m\}, one slot per peer as in §[3](https://arxiv.org/html/2606.29126#S3)\. Under our sender\-selection design, only the slot corresponding to the selected senderj∗j^\{\*\}is populated, while the remaining peer slots are set to zero; the construction ofj∗j^\{\*\}and the messagemj∗→im\_\{j^\{\*\}\\to i\}is given in §[4\.2](https://arxiv.org/html/2606.29126#S4.SS2)\. We bundle the communication parameters with the actor parameters asΘ=\(θpol,θcomm\)\\Theta=\(\\theta\_\{\\mathrm\{pol\}\},\\theta\_\{\\mathrm\{comm\}\}\), so that a single backward pass onℒ​\(Θ,ϕ\)\\mathcal\{L\}\(\\Theta,\\phi\)propagates task gradients through both the policy and theHiCommmodule, together with the critic parametersϕ\\phiwhen required by the chosen actor\-critic backbone\. Consequently, replacingℒ\\mathcal\{L\}swaps the underlying MARL algorithm, such as IPPO or MAPPO, without changing theHiCommmodule\. Algorithm[1](https://arxiv.org/html/2606.29126#alg1)summarizes the full procedure\.

### 4\.2Communication via Hierarchical Observation Structure

Algorithm 2Bandwidth\-Efficient MARL viaHierarchicalCommunication \(HiComm\)0:Joint observation

zi∈𝒢×ℰ×ℱz^\{i\}\\in\{\\mathcal\{G\}\\times\\mathcal\{E\}\\times\\mathcal\{F\}\}; HiComm modules

θcomm=\{fgap,fq,\{κg\},Φ,ψ\}\\theta\_\{\\text\{comm\}\}=\\\{f\_\{\\text\{gap\}\},\\,f\_\{\\text\{q\}\},\\,\\\{\\kappa\_\{g\}\\\},\\,\\Phi,\\,\\psi\\\}\(gap encoder, query head, group keys, context\-conditioned affinity, entity projector\)\.

0:Augmented observations

\{z~i\}i=1N\\\{\\widetilde\{z\}^\{i\}\\\}\_\{i=1\}^\{N\}\.

1:foreach agent

iiin paralleldo

2:// Phase 1: query from observation context \(receiver\-side\)

3:

hi←fgap​\(zi,Mi\)h\_\{i\}\\leftarrow f\_\{\\text\{gap\}\}\(z^\{i\},M^\{i\}\);

qi←tanh⁡\(fq​\(hi\)\)q\_\{i\}\\leftarrow\\tanh\\\!\\big\(f\_\{\\text\{q\}\}\(h\_\{i\}\)\\big\)
4:// Phase 2 \-\- Stage 1: pick topic groupg∗g^\{\*\}via group keysκg\\kappa\_\{g\}\(receiver\-side\)

5:

g∗←arg⁡maxg⁡ST\-GS​\(\{score​\(qi,κg\)\}g=1𝒢\)g^\{\*\}\\leftarrow\\arg\\max\_\{g\}\\,\\text\{ST\-GS\}\\\!\\Big\(\\big\\\{\\text\{score\}\(q\_\{i\},\\,\\kappa\_\{g\}\)\\big\\\}\_\{g=1\}^\{\\mathcal\{G\}\}\\Big\)
6:// Phase 2 \-\- Stage 2: pick senderj∗j^\{\*\}via context\-conditioned affinityΦ​\(j,g∗;hi\)\\Phi\(j,g^\{\*\};h\_\{i\}\)\(receiver\-side\)

7:

j∗←arg⁡maxj≠i⁡ST\-GS​\(\{score​\(qi,Φ​\(j,g∗;hi\)\)\}\)j^\{\*\}\\leftarrow\\arg\\max\_\{j\\neq i\}\\,\\text\{ST\-GS\}\\\!\\Big\(\\big\\\{\\text\{score\}\\big\(q\_\{i\},\\,\\Phi\(j,\\,g^\{\*\};\\,h\_\{i\}\)\\big\)\\big\\\}\\Big\)
8:// Phase 3: senderj∗j^\{\*\}receives\(qi,g∗\)\(q\_\{i\},g^\{\*\}\)and picks entityℓ∗\\ell^\{\*\}on its localzj∗,Mj∗z^\{j^\{\*\}\},M^\{j^\{\*\}\}

9:

ℓ∗←arg⁡maxℓ⁡ST​\-​GS​\{score​\(qi,ψ​\(zj∗​\[g∗,ℓ,:\]\)\)\+Mj∗\(ent\)​\(g∗\)​\[ℓ\]\}\\ell^\{\*\}\\leftarrow\\arg\\max\_\{\\ell\}\\,\\mathrm\{ST\\text\{\-\}GS\}\\\!\\big\\\{\\text\{score\}\\big\(q\_\{i\},\\,\\psi\(z^\{j^\{\*\}\}\\\!\\\!\[g^\{\*\},\\ell,:\]\)\\big\)\+M^\{\(\\text\{ent\}\)\}\_\{j^\{\*\}\}\\\!\(g^\{\*\}\)\[\\ell\]\\big\\\}\.

10:

mj∗→i←\(\(g∗,ℓ∗\),zj∗​\[g∗,ℓ∗,:\]\)m\_\{j^\{\*\}\\to i\}\\leftarrow\\big\(\(g^\{\*\},\\ell^\{\*\}\),\\,z^\{j^\{\*\}\}\[g^\{\*\},\\ell^\{\*\},:\]\\big\)\{sender returns address

\+\+verbatim feature vector to receiver

ii\}

11:

z~i←zi∥\[𝟎,…,mj∗→i⏟j∗​\-slot,…,𝟎\]\\widetilde\{z\}^\{i\}\\leftarrow z^\{i\}\\,\\\|\\,\\big\[\\mathbf\{0\},\\dots,\\underbrace\{m\_\{j^\{\*\}\\to i\}\}\_\{j^\{\*\}\\text\{\-slot\}\},\\dots,\\mathbf\{0\}\\big\]
12:endfor

13:return

\{z~i\}\\\{\\widetilde\{z\}^\{i\}\\\}

We now specify the three\-phaseHiCommstep \(Algorithm[2](https://arxiv.org/html/2606.29126#alg2)\)\. The protocol rests on a single structural principle:*the\[𝒢\]×\[ℰ\]\[\\mathcal\{G\}\]\\\!\\times\\\!\[\\mathcal\{E\}\]hierarchy is the addressing scheme not just for what the message carries, but also for whom to ask*\. The receiver does not pick a peer out of thin air and then walk the hierarchy; it descends the hierarchy and the peer identity*emerges*along the way\. Concretely, each round commits to three discrete picks made in order: an integer topic\-group indexg∗∈\[\|𝒢\|\]g^\{\*\}\\\!\\in\\\!\[\|\\mathcal\{G\}\|\]that names what the receiver wants to ask about; a senderj∗∈\[N\]∖\{i\}j^\{\*\}\\\!\\in\\\!\[N\]\\\!\\setminus\\\!\\\{i\\\}whose learnable affinity forg∗g^\{\*\}best matches the query; and an integer entity indexℓ∗∈\[\|ℰ\|\]\\ell^\{\*\}\\\!\\in\\\!\[\|\\mathcal\{E\}\|\]within groupg∗g^\{\*\}on the chosen sender’s observation\. In brief, Phase 1 generates the receiver’s queryqiq\_\{i\}; Phase 2 performs the two\-stage hierarchical sender selection \(groupg∗g^\{\*\}, then agentj∗∣g∗j^\{\*\}\\\!\\mid\\\!g^\{\*\}\) entirely on the receiver side; Phase 3 transmits\(qi,g∗\)\(q\_\{i\},g^\{\*\}\)toj∗j^\{\*\}, who locally selects the entityℓ∗\\ell^\{\*\}on its own observation and returns the addressed feature vectorF=zj∗​\[g∗,ℓ∗,:\]∈ℝ\|ℱ\|F=z^\{j^\{\*\}\}\[g^\{\*\},\\ell^\{\*\},:\]\\in\\mathbb\{R\}^\{\|\\mathcal\{F\}\|\}\. The receiver initiates and steers each round, while the sender executes only the final entity stage on its private observation; we therefore callHiComm*query\-driven*rather than fully receiver\-resolved\.

This descent has three structural consequences\. First, the hierarchy is not relegated to a retrieval\-only role: it drives sender selection itself, sincej∗j^\{\*\}is chosen*conditional on*g∗g^\{\*\}rather than orthogonally to it\. Second, the total selector\-head output dimension is\|𝒢\|\+\(N−1\)\+\|ℰ\|\|\\mathcal\{G\}\|\+\(N\{\-\}1\)\+\|\\mathcal\{E\}\|, summed across the three stage\-wise picks, rather than the\|𝒢\|⋅\|ℰ\|⋅\(N−1\)\|\\mathcal\{G\}\|\\\!\\cdot\\\!\|\\mathcal\{E\}\|\\\!\\cdot\\\!\(N\{\-\}1\)of a flat joint pick over \(group, entity, sender\) \(self\-loops masked in both cases\), reducing the discrete decision space the policy gradient must traverse and giving each stage a dedicated parameter set into which to localise credit\. Third, the transmitted feature vectorF=zj∗​\[g∗,ℓ∗,:\]F=z^\{j^\{\*\}\}\[g^\{\*\},\\ell^\{\*\},:\]is in\-distribution and verbatim: it is one of the sender’s own observation entries, never a learned mixture across cells\. Empirical bandwidth comparisons against alternative protocols are reported in §[5](https://arxiv.org/html/2606.29126#S5)\.

##### Phase 1: query generation\.

LetMi∈\{0,1\}\|𝒢\|×\|ℰ\|M^\{i\}\\in\\\{0,1\\\}^\{\|\\mathcal\{G\}\|\\times\|\\mathcal\{E\}\|\}be receiverii’s observability mask, withMi​\[g,ℓ\]=1M^\{i\}\[g,\\ell\]=1iff entity\(g,ℓ\)\(g,\\ell\)is observed inziz^\{i\}\. Note thatMiM^\{i\}is a deterministic function ofziz^\{i\}, the indicator of which\(g,ℓ\)\(g,\\ell\)slots carry observed values rather than an unobserved sentinel, and we expose it explicitly only for notational convenience\. Surfacing the mask alongsideziz^\{i\}letsqiq\_\{i\}treat the receiver’s epistemic footprint as a first\-class signal: when evidence is sparse, the query can deliberately steer toward blind spots; when local support is too uncertain to commit to a confident pick, it can soften its retrieval rather than commit to a request over an unreliable basis\. The receiver summarises its observation context together with its observability footprint into a query,

hi=fgap​\(zi,Mi\),qi=tanh⁡\(fq​\(hi\)\)∈\[−1,1\]dq,h\_\{i\}=f\_\{\\text\{gap\}\}\(z^\{i\},M^\{i\}\),\\qquad q\_\{i\}=\\tanh\\\!\\big\(f\_\{\\text\{q\}\}\(h\_\{i\}\)\\big\)\\in\[\-1,1\]^\{d\_\{q\}\},wherefgapf\_\{\\text\{gap\}\}andfqf\_\{\\text\{q\}\}are learnable MLPs andhih\_\{i\}is the GapNet\-encoded query feature \(kept distinct from the action symboluiu^\{i\}of §[3](https://arxiv.org/html/2606.29126#S3)\)\. The queryqiq\_\{i\}thus represents a prior over what to ask given the receiver’s current state, and is reused without recomputation across all subsequent stages so that whom\-to\-ask, what\-group, and what\-entity decisions are anchored in a single information\-need representation\.

##### Phase 2: hierarchical sender selection \(receiver\-side, two stages\)\.

The receiver descends the hierarchy in two ST\-GS draws, both executed on its local device\.

*Stage 1, group selection\.*The receiver picks the topic groupg∗g^\{\*\}by scoring its query against\|𝒢\|\|\\mathcal\{G\}\|learnable group\-key embeddingsκg∈ℝdq\\kappa\_\{g\}\\in\\mathbb\{R\}^\{d\_\{q\}\}\(one per group, parameter\-shared across receivers and senders\),

g∗=arg⁡maxg⁡ST​\-​GS​\(score​\(qi,κg\)\)\.g^\{\*\}=\\arg\\max\_\{g\}\\,\\mathrm\{ST\\text\{\-\}GS\}\\\!\\Big\(\\mathrm\{score\}\\big\(q\_\{i\},\\,\\kappa\_\{g\}\\big\)\\Big\)\.The keys\{κg\}g=1\|𝒢\|\\\{\\kappa\_\{g\}\\\}\_\{g=1\}^\{\|\\mathcal\{G\}\|\}belong toθcomm\\theta\_\{\\text\{comm\}\}and are updated by the same task gradient as the rest ofHiComm\. Stage 1 commits the receiver to a topic before naming any peer\.

*Stage 2, agent selection conditional ong∗g^\{\*\}\.*The receiver scores the sameqiq\_\{i\}against a*context\-conditioned agent\-group affinity*Φ​\(j,g∗;hi\)∈ℝdq\\Phi\(j,g^\{\*\};\\,h\_\{i\}\)\\in\\mathbb\{R\}^\{d\_\{q\}\}based on the receiver’s own GapNet outputhih\_\{i\}from Phase 1 \(re\-used without recomputation\):

si​j∣g∗=score​\(qi,Φ​\(j,g∗;hi\)\),j∗=arg⁡maxj≠i⁡ST​\-​GS​\(si,:∣g∗\)\.s\_\{ij\\mid g^\{\*\}\}=\\mathrm\{score\}\\\!\\big\(q\_\{i\},\\,\\Phi\(j,g^\{\*\};\\,h\_\{i\}\)\\big\),j^\{\*\}=\\arg\\max\_\{j\\neq i\}\\,\\mathrm\{ST\\text\{\-\}GS\}\\\!\\big\(s\_\{i,:\\mid g^\{\*\}\}\\big\)\.The hard top\-11is structural rather than approximate: the address\(g∗,ℓ∗\)\(g^\{\*\},\\ell^\{\*\}\)produced in Phase 3 indexes a specific*sender’s*tensor, so a soft mixture over senders would leave the address ill\-defined\.

##### Phase 3: entity retrieval \(executed at senderj∗j^\{\*\}\)\.

The receiver transmits the pair\(qi,g∗\)\(q\_\{i\},g^\{\*\}\)toj∗j^\{\*\}\. The sender then resolves the entityℓ∗\\ell^\{\*\}within groupg∗g^\{\*\}*on its own device*, masking unobserved entries with its own mask:

Mj∗\(ent\)​\(g∗\)​\[ℓ\]=\{0if​Mj∗​\[g∗,ℓ\]=1,−∞otherwise,M^\{\(\\text\{ent\}\)\}\_\{j^\{\*\}\}\(g^\{\*\}\)\[\\ell\]=\\begin\{cases\}0&\\text\{if \}M^\{j^\{\*\}\}\[g^\{\*\},\\ell\]=1,\\\\ \-\\infty&\\text\{otherwise\},\\end\{cases\}ℓ∗∣g∗,j∗=arg⁡maxℓ⁡ST​\-​GS​\(score​\(qi,ψ​\(zj∗​\[g∗,ℓ,:\]\)\)\+Mj∗\(ent\)​\(g∗\)​\[ℓ\]\),\\ell^\{\*\}\\mid g^\{\*\},j^\{\*\}=\\arg\\max\_\{\\ell\}\\,\\mathrm\{ST\\text\{\-\}GS\}\\\!\\Big\(\\mathrm\{score\}\\big\(q\_\{i\},\\,\\psi\(z^\{j^\{\*\}\}\[g^\{\*\},\\ell,:\]\)\\big\)\+M^\{\(\\text\{ent\}\)\}\_\{j^\{\*\}\}\(g^\{\*\}\)\[\\ell\]\\Big\),whereψ:ℝ\|ℱ\|→ℝdq\\psi:\\mathbb\{R\}^\{\|\\mathcal\{F\}\|\}\\to\\mathbb\{R\}^\{d\_\{q\}\}is a sender\-side MLP that projects each entity feature vectorF=zj∗​\[g∗,ℓ,:\]∈ℝ\|ℱ\|F=z^\{j^\{\*\}\}\[g^\{\*\},\\ell,:\]\\in\\mathbb\{R\}^\{\|\\mathcal\{F\}\|\}into the scoring space\. Because the entity keysψ​\(zj∗​\[g∗,ℓ,:\]\)\\psi\(z^\{j^\{\*\}\}\[g^\{\*\},\\ell,:\]\)and the maskMj∗M^\{j^\{\*\}\}are derived only fromj∗j^\{\*\}’s private observation, no part ofzj∗z^\{j^\{\*\}\}ever leaves the sender; only\(qi,g∗\)\(q\_\{i\},g^\{\*\}\)enter, and the address\-value pair

mj∗→i=\(\(g∗,ℓ∗\),F\),F:=zj∗​\[g∗,ℓ∗,:\]∈ℝ\|ℱ\|,m\_\{j^\{\*\}\\to i\}=\\big\(\(g^\{\*\},\\ell^\{\*\}\),\\ F\\big\),\\qquad F:=z^\{j^\{\*\}\}\[g^\{\*\},\\ell^\{\*\},:\]\\in\\mathbb\{R\}^\{\|\\mathcal\{F\}\|\},returns to the receiver\. The transmitted feature vectorFFis in\-distribution and verbatim: it equals one of the sender’s own observation entries, never a learned mixture across cells \(during training the soft surrogate of ST\-GS converges to one\-hot atτ→τmin\\tau\\to\\tau\_\{\\min\}\)\. All three key types, namely the group keysκg\\kappa\_\{g\}, the context\-conditioned affinityΦ​\(j,g;hi\)\\Phi\(j,g;\\,h\_\{i\}\), and the entity projectionsψ​\(⋅\)\\psi\(\\cdot\), live in the commonℝdq\\mathbb\{R\}^\{d\_\{q\}\}scoring geometry, soqiq\_\{i\}uses the same dot\-product across all three stages without re\-projection\.

##### Training Objective

We updateΘ=\(θpol,θcomm\)\\Theta=\(\\theta\_\{\\text\{pol\}\},\\theta\_\{\\text\{comm\}\}\)in a single actor backward pass on

ℒactor​\(Θ\)=ℒPPO​\(Θ\)−cH​ℋ​\[πθpol\]\+λig​\(t\)​ℒig​\(θcomm\),\\mathcal\{L\}\_\{\\text\{actor\}\}\(\\Theta\)\\,=\\,\\mathcal\{L\}^\{\\text\{PPO\}\}\(\\Theta\)\\,\-\\,c\_\{H\}\\,\\mathcal\{H\}\\\!\\big\[\\pi\_\{\\theta\_\{\\text\{pol\}\}\}\\big\]\\,\+\\,\\lambda\_\{\\text\{ig\}\}\(t\)\\,\\mathcal\{L\}\_\{\\text\{ig\}\}\(\\theta\_\{\\text\{comm\}\}\),whereℒPPO\\mathcal\{L\}^\{\\text\{PPO\}\}supplies task gradient through every ST\-GS selector andℒig\\mathcal\{L\}\_\{\\text\{ig\}\}cross\-entropies the Stage 1, Stage 2 and Phase 3 pre\-ST\-GS logits against soft labels obtained by softmax\-normalising the per\-coordinate information gainΔi​j​\[g,ℓ,:\]=\[zj​\[g,ℓ,:\]−zi​\[g,ℓ,:\]\]\+\\Delta\_\{ij\}\[g,\\ell,:\]=\\big\[z^\{j\}\[g,\\ell,:\]\-z^\{i\}\[g,\\ell,:\]\\big\]\_\{\+\}along the matching axis, plus a small batch\-marginal entropy term that discourages sender collapse\. As joint coverage saturates,Δi​j→0\\Delta\_\{ij\}\\\!\\to\\\!0and the soft labels degenerate to uniform, soℒig\\mathcal\{L\}\_\{\\text\{ig\}\}vanishes structurally; the scheduleλig​\(t\)\\lambda\_\{\\text\{ig\}\}\(t\)only accelerates this fade\. The critic update onϕ\\phiis unchanged, leavingHiComm’s training\-side footprint at one scalar added to the actor loss\.

## 5Experiments

The empirical study answers three questions\.\(RQ1\) Performance:doesHiCommclose the gap between the no\-communicationPartialObsfloor and theFullObsceiling, and does it match or beat the strongest learned\-communication baselines,CACOM\[[17](https://arxiv.org/html/2606.29126#bib.bib12)\]andT2MAC\[[32](https://arxiv.org/html/2606.29126#bib.bib11)\]?\(RQ2\) Bandwidth:at matched performance, how much information doesHiCommactually ship per receiver per episode, relative to those baselines and toFullObs?\(RQ3\) Generality:do RQ1 and RQ2 hold across IPPO\[[3](https://arxiv.org/html/2606.29126#bib.bib22)\]and MAPPO\[[36](https://arxiv.org/html/2606.29126#bib.bib23)\], and across varied benchmarks?

We evaluate on three partially observable benchmarks chosen to span qualitatively different regimes: CAGE Challenge 4\[[13](https://arxiv.org/html/2606.29126#bib.bib18),[12](https://arxiv.org/html/2606.29126#bib.bib19)\], an adversarial cyber\-defense task with*accumulative*reconnaissance; SMACv2\[[5](https://arxiv.org/html/2606.29126#bib.bib26)\], cooperative micromanagement under*instantaneous*sight\-range fog of war; and Google Research Football \(GRF\)\[[15](https://arxiv.org/html/2606.29126#bib.bib33)\], set\-piece control under a*bounded sensing radius*\. All three instantiate the group\-entity hierarchy of §[3](https://arxiv.org/html/2606.29126#S3), with opponents \(where present\) collapsed into a single group\.

### 5\.1Setup

##### Methods\.

We compare five settings under two cooperative\-MARL backbones: IPPO\[[3](https://arxiv.org/html/2606.29126#bib.bib22)\]\(decentralised critic\) and MAPPO\[[36](https://arxiv.org/html/2606.29126#bib.bib23)\]\(centralised critic\)\.PartialObskeeps each receiver on its own local viewziz^\{i\}with allN−1N\{\-\}1message slots zeroed \(the no\-communication*floor*\);FullObsconcatenates every teammate’s local view\{zj\}j≠i\\\{z^\{j\}\\\}\_\{j\\neq i\}verbatim intoz~i\\widetilde\{z\}^\{i\}\(the*ceiling*on team reward, at the cost of shipping every full local view at every step\)\. The three communication settings share a fixed message widthdmd\_\{m\}\.CACOM\[[17](https://arxiv.org/html/2606.29126#bib.bib12)\]first lets each sender broadcast a short summary so everyone knows roughly what is going on, then tailors a separate message to each receiver via attention\.T2MAC\[[32](https://arxiv.org/html/2606.29126#bib.bib11)\]only talks to selected teammates instead of broadcasting, and on the receiver side it weighs each incoming message by how informative it looks before mixing them\. Both, however, treat the observation as a flat vector and ignore its𝒢×ℰ\\mathcal\{G\}\\\!\\times\\\!\\mathcal\{E\}group\-entity structure\.HiComm\(ours\) is the three\-phase receiver\-driven protocol of §[4](https://arxiv.org/html/2606.29126#S4): the receiver picks a topic groupg∗g^\{\*\}and a senderj∗j^\{\*\}in two ST\-GS stages, the chosen sender picks an entityℓ∗∈g∗\\ell^\{\*\}\\\!\\in\\\!g^\{\*\}on its own observation, and returns the address\-value pair\(\(g∗,ℓ∗\),F\)\\big\(\(g^\{\*\},\\ell^\{\*\}\),\\,F\\big\), whereF=zj∗​\[g∗,ℓ∗,:\]∈ℝℱF=z^\{j^\{\*\}\}\[g^\{\*\},\\ell^\{\*\},:\]\\in\\mathbb\{R\}^\{\\mathcal\{F\}\}is the feature vector at the addressed coordinate\.

##### Metrics\.

We report team*reward*and per\-receiver*communication cost*in KB per episode \(per\-step payload×\\timesepisode length\)\. Across all benchmarks the receiver\-side group pick \(Phase 2 Stage 1\) carries no data\-dependent mask; the sender\-side entity stage masks slots ing∗g^\{\*\}that senderj∗j^\{\*\}cannot currently observe, derived fromj∗j^\{\*\}’s own maskMj∗M^\{j^\{\*\}\}\. Per\-benchmark simulator, wrapper, and baseline\-port details are deferred to App\.[A\.1](https://arxiv.org/html/2606.29126#A1.SS1)\-[A\.3](https://arxiv.org/html/2606.29126#A1.SS3)\.

##### CAGE Challenge 4 \(CC4\)\.

CC4 is a procedurally generated enterprise\-network cyber\-defense benchmark in which five blue defenders face six red attackers over500500\-step episodes across nine subnets organized into five defended zones\. The official release ships CC4 as a blue\-side training problem with red implemented as a scripted finite\-state machine; we instead lift red to a learned cooperative MARL task, freezing blue at the pretrained IPPO checkpoint ofSinghet al\.\[[27](https://arxiv.org/html/2606.29126#bib.bib41)\]\. Each red agent is assigned to a single subnet and is initially inactive, with the sole exception of the agent in the Contractor Network; the remaining agents activate only after another red agent establishes a session within their subnet\. Partial observability is*accumulative*: at each step a red agent learns a few new bits about hosts it has reached \(an IP, a service, a privilege\) and that knowledge persists\. CC4 exposes a*native*subnet⊃\\,\\supset\\,host containment that aligns directly withHiComm, with𝒢=9\\mathcal\{G\}\{=\}9subnets andℰ=16\\mathcal\{E\}\{=\}16hosts\.

##### StarCraft Multi\-Agent Challenge \(SMACv2\)\.

SMACv2 is a cooperative StarCraft II micromanagement benchmark in which each unit is operated by an independent agent under instantaneous sight\-range fog of war and randomised start configurations\. We adopt the standard55v55setup across the three races \(protoss,terran,zerg\); per\-unit observations restricted to entities within sight, the discrete action set, and the damage\-shaped reward with terminal win bonus all follow SMACv2 defaults\. Because no native observation hierarchy exists, we induce one by indexing the group axis with ally unit class \(a one\-hot tag already exposed by the per\-unit observation\), with𝒢\\mathcal\{G\}the active ally classes plus a single*opponent*group\.HiCommunicasts every step to track rapid shifts in visibility\.

##### Google Research Football \(GRF\)\.

GRF is a 3D physics\-based football simulator with a discrete high\-level action set, widely used as a partially observable cooperative\-control benchmark\. We adoptacademy\_counterattack\_hard\(44v33\): a fast break in whichNa=3N\_\{a\}\{=\}3controlled attackers face two defenders plus a goalkeeper across400400steps, with the controlled side’s keeper scripted followingSonget al\.\[[30](https://arxiv.org/html/2606.29126#bib.bib45)\]\. Each agent observes thesimple115v2feature vector restricted to entities within its sensing radius \(ℱ=7\\mathcal\{F\}\{=\}7\); the1919\-action discrete control set and the*Dense*\(scoring,checkpoints\) reward ofSonget al\.\[[30](https://arxiv.org/html/2606.29126#bib.bib45)\]are GRF defaults\. Partial observability is again instantaneous\. We index the group axis with each ally’s on\-pitch*field\-position role*\(\{GK,DEF,MID,FWD\}\\\{\\text\{GK\},\\text\{DEF\},\\text\{MID\},\\text\{FWD\}\\\}, drawn from the engine\) plus a single opponent group\. This is the third group indexing under test, complementing the spatial \(CC4\) and unit\-class \(SMACv2\) cases\.

### 5\.2Results

Table[1](https://arxiv.org/html/2606.29126#S5.T1)reports the IPPO and MAPPO grid on a representative scenario per benchmark, together with per\-receiver per\-episode communication cost \(in kb\)\.

Table 1:Cross\-benchmark results \(IPPO, MAPPO\): team reward and per\-episode communication cost \(KB\) per receiver\.Boldmarks the top\-2 reward \(higher is better\) and bottom\-2 cost \(lower is better\) within each algorithm block\.CC4SMACv255v55GRFcthard44v33protossterranzergAlgorithmMethodRewardCostRewardCostRewardCostRewardCostRewardCostIPPOFullObs𝟏,𝟕𝟔𝟑\\mathbf\{1\{,\}763\}8420\.68420\.617\.1\\mathbf\{17\.1\}247\.77247\.7717\.2\\mathbf\{17\.2\}195\.13195\.1314\.3\\mathbf\{14\.3\}138\.86138\.860\.79\\mathbf\{0\.79\}351351IPPOPartialObs1,4361\{,\}436𝟎\\mathbf\{0\}14\.514\.5𝟎\\mathbf\{0\}14\.414\.4𝟎\\mathbf\{0\}11\.411\.4𝟎\\mathbf\{0\}0\.700\.70𝟎\\mathbf\{0\}IPPOCACOM1,5171\{,\}51741\.341\.315\.615\.652\.6752\.6715\.615\.634\.8434\.8412\.312\.319\.7619\.760\.730\.7316\.516\.5IPPOT2MAC𝟏,𝟔𝟐𝟑\\mathbf\{1\{,\}623\}143\.8143\.815\.215\.2201\.26201\.2615\.315\.3129\.74129\.7412\.312\.376\.2876\.280\.730\.73192192IPPOHiComm\(ours\)1,5271\{,\}5278\.4\\mathbf\{8\.4\}16\.3\\mathbf\{16\.3\}9\.77\\mathbf\{9\.77\}16\.4\\mathbf\{16\.4\}7\.95\\mathbf\{7\.95\}13\.5\\mathbf\{13\.5\}4\.85\\mathbf\{4\.85\}0\.76\\mathbf\{0\.76\}13\.6\\mathbf\{13\.6\}MAPPOFullObs𝟐,𝟏𝟏𝟎\\mathbf\{2\{,\}110\}8420\.68420\.617\.4\\mathbf\{17\.4\}243\.92243\.9216\.4\\mathbf\{16\.4\}187\.55187\.5514\.9\\mathbf\{14\.9\}138\.97138\.970\.89\\mathbf\{0\.89\}382382MAPPOPartialObs990990𝟎\\mathbf\{0\}14\.814\.8𝟎\\mathbf\{0\}13\.613\.6𝟎\\mathbf\{0\}12\.312\.3𝟎\\mathbf\{0\}0\.790\.79𝟎\\mathbf\{0\}MAPPOCACOM1,6921\{,\}69236\.836\.816\.016\.046\.7446\.7414\.714\.736\.9736\.9713\.113\.124\.2024\.200\.830\.8321\.421\.4MAPPOT2MAC1,5541\{,\}554134\.2134\.215\.615\.6168\.93168\.9314\.414\.4155\.82155\.8212\.712\.794\.7194\.710\.850\.85137137MAPPOHiComm\(ours\)𝟏,𝟗𝟒𝟏\\mathbf\{1\{,\}941\}7\.7\\mathbf\{7\.7\}16\.7\\mathbf\{16\.7\}10\.98\\mathbf\{10\.98\}15\.6\\mathbf\{15\.6\}6\.83\\mathbf\{6\.83\}13\.9\\mathbf\{13\.9\}4\.25\\mathbf\{4\.25\}0\.86\\mathbf\{0\.86\}14\.0\\mathbf\{14\.0\}

##### Performance \(RQ1\)\.

HiCommcloses a substantial fraction of the partial\-to\-full gap and outperforms both flat\-message baselines in99of1010algorithm\-scenario cells\. CC4 makes the algorithmic contrast especially clean\. Under IPPO the no\-communicationPartialObsfloor already reaches1,4361\{,\}436, roughly81%81\\%of the1,7631\{,\}763FullObsceiling, so the headroom available to any communication protocol is narrow on this row;CACOM,T2MAC, andHiCommaccordingly land within a tight band \(1,5171\{,\}517,1,6231\{,\}623,1,5271\{,\}527\) and CC4\-IPPO does not strongly differentiate the three communication methods on reward alone\. Under MAPPO the picture changes sharply: thePartialObsfloor collapses to990990, the partial\-to\-full gap more than triples, andHiCommdelivers a markedly larger lift than the flat\-message baselines, rising to1,9411\{,\}941to recover≈85%\\approx\\\!85\\%of the gap to the2,1102\{,\}110ceiling and beatingCACOMby\+249\+249andT2MACby\+387\+387\. The MAPPO row is therefore the clearest demonstration ofHiComm’s advantage on this benchmark: it is precisely when the no\-communication baseline leaves real headroom that the protocol’s gain becomes visible\. On SMACv2,HiCommcloses6969\-72%72\\%of the gap \(vs\.1515\-46%46\\%for the baselines\); on GRF it closes67%67\\%under IPPO and70%70\\%under MAPPO\. Querying coordinates in the group\-entity grid lets the receiver retrieve precisely the parts of teammate state it cannot reconstruct from its own view, which a flat receiver\-conditioned projection \(CACOM\) or a flat sender broadcast with evidence weighting \(T2MAC\) cannot match at the same channel width\.

##### Bandwidth \(RQ2\)\.

HiCommattains these gains while shipping one to three orders of magnitude less data than every other communication setting:55\-17×17\\timesless thanCACOM/T2MACon CC4,44\-23×23\\timesless on SMACv2, and≈10×\\approx\\\!10\\timesless thanT2MACon GRF \(withCACOM’s already\-narrow GRF channel only1\.21\.2\-1\.5×1\.5\\timeswider thanHiComm’s\); relative toFullObs, the saving exceeds1000×1000\\timeson CC4\. Mechanically,HiCommwrites one\(g∗,ℓ∗\)\(g^\{\*\},\\ell^\{\*\}\)address plus the\|ℱ\|\|\\mathcal\{F\}\|\-dim feature vectorFFinto a single message slot and zeros the rest, whereas the flat baselines populate every slot at every step\. Because the address\-value pair carries the feature vector verbatim rather than a lossy projection, the bandwidth saving comes*without*a fidelity penalty\.

##### Generality \(RQ3\)\.

The performance\-bandwidth pattern survives both critic styles and all three benchmarks’ group\-axis instantiations:HiCommranks first or tied\-first on reward in99of1010cells and is strictly cheapest in bandwidth in all1010, even though the underlying group axis is read off from a router map \(CC4, native\), a unit\-type one\-hot \(SMACv2, semantic\), or a coarse positional class \(GRF, semantic\)\. Switching between IPPO and MAPPO shifts absolute reward levels but does not perturb the relative ordering of the four communication settings, indicating that the gain attributable toHiCommlives in the message protocol rather than in the critic\.

## 6Conclusion and Limitations

In all, we presentedHiComm, a receiver\-driven communication protocol that grounds every message in the sender’s observation hierarchy\. A query selects a topic group, a sender, and an entity via three Straight\-Through Gumbel\-Softmax draws, and the sender returns the raw feature vector at the addressed coordinate\. Across CC4, SMACv2, and GRF under both IPPO and MAPPO,HiCommmatches or beats learned\-communication baselines in99of1010algorithm\-scenario cells while shipping up to23×23\\timesless data per receiver per episode\. The principal limitation of this work is the absence of a formal theoretical analysis ofHiComm’s communication efficiency: our bandwidth advantage rests on empirical comparison rather than an information\-theoretic bound that explains why hierarchical addressing is provably more efficient than flat encoding at matched task performance\. Establishing such a theory is the primary direction for follow\-up work\.

## Acknowledgments and Disclosure of Funding

This work was supported in part by the Defense Advanced Research Projects Agency \(DARPA\) under Cooperative Agreement No\. HR0011\-24\-2\-0004\.

Distribution statement: Approved for public release; distribution is unlimited\.

## References

- \[1\]S\. Ahilan and P\. Dayan\(2019\)Feudal multi\-agent hierarchies for cooperative reinforcement learning\.InWorkshop on Structure & Priors in Reinforcement Learning \(SPiRL 2019\) at ICLR 2019,pp\. 1–11\.Cited by:[§1](https://arxiv.org/html/2606.29126#S1.p3.1),[§2](https://arxiv.org/html/2606.29126#S2.p2.1),[Remark 4\.2](https://arxiv.org/html/2606.29126#S4.Thmtheorem2.p1.2)\.
- \[2\]A\. Das, T\. Gervet, J\. Romoff, D\. Batra, D\. Parikh, M\. Rabbat, and J\. Pineau\(2019\)Tarmac: targeted multi\-agent communication\.InInternational Conference on machine learning,pp\. 1538–1546\.Cited by:[§1](https://arxiv.org/html/2606.29126#S1.p1.1),[§2](https://arxiv.org/html/2606.29126#S2.p1.1),[Remark 4\.2](https://arxiv.org/html/2606.29126#S4.Thmtheorem2.p1.2)\.
- \[3\]C\. S\. De Witt, T\. Gupta, D\. Makoviichuk, V\. Makoviychuk, P\. H\. Torr, M\. Sun, and S\. Whiteson\(2020\)Is independent learning all you need in the starcraft multi\-agent challenge?\.arXiv preprint arXiv:2011\.09533\.Cited by:[§4\.1](https://arxiv.org/html/2606.29126#S4.SS1.SSS0.Px1.p1.6),[§5\.1](https://arxiv.org/html/2606.29126#S5.SS1.SSS0.Px1.p1.11),[§5](https://arxiv.org/html/2606.29126#S5.p1.1)\.
- \[4\]Z\. Ding, Z\. Liu, Z\. Fang, K\. Su, L\. Zhu, and Z\. Lu\(2024\)Multi\-agent coordination via multi\-level communication\.Advances in Neural Information Processing Systems37,pp\. 118513–118539\.Cited by:[§2](https://arxiv.org/html/2606.29126#S2.p1.1)\.
- \[5\]B\. Ellis, J\. Cook, S\. Moalla, M\. Samvelyan, M\. Sun, A\. Mahajan, J\. Foerster, and S\. Whiteson\(2023\)Smacv2: an improved benchmark for cooperative multi\-agent reinforcement learning\.Advances in Neural Information Processing Systems36,pp\. 37567–37593\.Cited by:[§A\.2\.1](https://arxiv.org/html/2606.29126#A1.SS2.SSS1.p1.3),[§A\.2\.3](https://arxiv.org/html/2606.29126#A1.SS2.SSS3.Px1.p1.2),[Table 5](https://arxiv.org/html/2606.29126#A1.T5),[§5](https://arxiv.org/html/2606.29126#S5.p2.1)\.
- \[6\]J\. N\. Foerster, Y\. M\. Assael, N\. de Freitas, and S\. Whiteson\(2016\)Learning to communicate with deep multi\-agent reinforcement learning\.InProceedings of the 30th International Conference on Neural Information Processing Systems,pp\. 2145–2153\.Cited by:[§1](https://arxiv.org/html/2606.29126#S1.p1.1),[§2](https://arxiv.org/html/2606.29126#S2.p1.1)\.
- \[7\]C\. Guan, F\. Chen, L\. Yuan, Z\. Zhang, and Y\. Yu\(2024\)Efficient communication via self\-supervised information aggregation for online and offline multiagent reinforcement learning\.IEEE Transactions on Neural Networks and Learning Systems36\(5\),pp\. 9044–9056\.Cited by:[§2](https://arxiv.org/html/2606.29126#S2.p1.1),[Remark 4\.2](https://arxiv.org/html/2606.29126#S4.Thmtheorem2.p1.2)\.
- \[8\]S\. Hu, L\. Shen, Y\. Zhang, and D\. Tao\(2024\)Learning multi\-agent communication from graph modeling perspective\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Qox9rO0kN0)Cited by:[§1](https://arxiv.org/html/2606.29126#S1.p1.1),[§2](https://arxiv.org/html/2606.29126#S2.p1.1),[Remark 4\.2](https://arxiv.org/html/2606.29126#S4.Thmtheorem2.p1.2)\.
- \[9\]E\. Jang, S\. Gu, and B\. Poole\(2017\)Categorical reparameterization with gumbel\-softmax\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=rkE3y85ee)Cited by:[§4](https://arxiv.org/html/2606.29126#S4.p1.10)\.
- \[10\]J\. Jiang, C\. Dun, T\. Huang, and Z\. Lu\(2020\)Graph convolutional reinforcement learning\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=HkxdQkSYDB)Cited by:[§2](https://arxiv.org/html/2606.29126#S2.p1.1),[Remark 4\.2](https://arxiv.org/html/2606.29126#S4.Thmtheorem2.p1.2)\.
- \[11\]J\. Jiang and Z\. Lu\(2018\)Learning attentional communication for multi\-agent cooperation\.Advances in neural information processing systems31\.Cited by:[§1](https://arxiv.org/html/2606.29126#S1.p1.1),[§2](https://arxiv.org/html/2606.29126#S2.p1.1),[Remark 4\.2](https://arxiv.org/html/2606.29126#S4.Thmtheorem2.p1.2)\.
- \[12\]M\. Kiely, M\. Ahiskali, E\. Borde, B\. Bowman, D\. Bowman, D\. Van Bruggen, K\. Cowan, P\. Dasgupta, E\. Devendorf, B\. Edwards,et al\.\(2025\)CAGE challenge 4: a scalable multi\-agent reinforcement learning gym for autonomous cyber defence\.AI Magazine46\(3\),pp\. e70021\.Cited by:[§A\.1\.1](https://arxiv.org/html/2606.29126#A1.SS1.SSS1.Px3.p1.4),[§A\.1\.1](https://arxiv.org/html/2606.29126#A1.SS1.SSS1.p1.1),[Table 3](https://arxiv.org/html/2606.29126#A1.T3),[§1](https://arxiv.org/html/2606.29126#S1.p2.1),[§5](https://arxiv.org/html/2606.29126#S5.p2.1)\.
- \[13\]M\. Kiely, M\. Ahiskali, E\. Borde, B\. Bowman, D\. Bowman, D\. Van Bruggen, K\. Cowan, P\. Dasgupta, E\. Devendorf, B\. Edwards,et al\.\(2025\)Exploring the efficacy of multi\-agent reinforcement learning for autonomous cyber defence: a cage challenge 4 perspective\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 28907–28913\.Cited by:[§A\.1\.1](https://arxiv.org/html/2606.29126#A1.SS1.SSS1.Px3.p1.4),[§A\.1\.1](https://arxiv.org/html/2606.29126#A1.SS1.SSS1.p1.1),[Table 3](https://arxiv.org/html/2606.29126#A1.T3),[§1](https://arxiv.org/html/2606.29126#S1.p2.1),[§5](https://arxiv.org/html/2606.29126#S5.p2.1)\.
- \[14\]D\. Kim, S\. Moon, D\. Hostallero, W\. J\. Kang, T\. Lee, K\. Son, and Y\. Yi\(2019\)Learning to schedule communication in multi\-agent reinforcement learning\.arXiv preprint arXiv:1902\.01554\.Cited by:[§2](https://arxiv.org/html/2606.29126#S2.p1.1),[Remark 4\.2](https://arxiv.org/html/2606.29126#S4.Thmtheorem2.p1.2)\.
- \[15\]K\. Kurach, A\. Raichuk, P\. Stańczyk, M\. Zając, O\. Bachem, L\. Espeholt, C\. Riquelme, D\. Vincent, M\. Michalski, O\. Bousquet,et al\.\(2020\)Google research football: a novel reinforcement learning environment\.InProceedings of the AAAI conference on artificial intelligence,Vol\.34,pp\. 4501–4510\.Cited by:[§A\.3\.1](https://arxiv.org/html/2606.29126#A1.SS3.SSS1.Px2.p1.5),[§A\.3\.1](https://arxiv.org/html/2606.29126#A1.SS3.SSS1.p1.7),[§5](https://arxiv.org/html/2606.29126#S5.p2.1)\.
- \[16\]S\. Li, J\. K\. Gupta, P\. Morales, R\. Allen, and M\. J\. Kochenderfer\(2021\)Deep implicit coordination graphs for multi\-agent reinforcement learning\.InProceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems,pp\. 764–772\.Cited by:[§2](https://arxiv.org/html/2606.29126#S2.p1.1)\.
- \[17\]X\. Li and J\. Zhang\(2024\)Context\-aware communication for multi\-agent reinforcement learning\.InProceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems,pp\. 1156–1164\.Cited by:[Appendix A](https://arxiv.org/html/2606.29126#A1.SS0.SSS0.Px1.p1.10),[§2](https://arxiv.org/html/2606.29126#S2.p1.1),[Remark 4\.2](https://arxiv.org/html/2606.29126#S4.Thmtheorem2.p1.2),[§5\.1](https://arxiv.org/html/2606.29126#S5.SS1.SSS0.Px1.p1.11),[§5](https://arxiv.org/html/2606.29126#S5.p1.1)\.
- \[18\]Y\. Liu, J\. Tian, N\. Glaser, and Z\. Kira\(2020\)When2com: multi\-agent perception via communication graph grouping\.InProceedings of the IEEE/CVF Conference on computer vision and pattern recognition,pp\. 4106–4115\.Cited by:[§2](https://arxiv.org/html/2606.29126#S2.p1.1),[Remark 4\.2](https://arxiv.org/html/2606.29126#S4.Thmtheorem2.p1.2)\.
- \[19\]Y\. Liu, J\. Tian, C\. Ma, N\. Glaser, C\. Kuo, and Z\. Kira\(2020\)Who2com: collaborative perception via learnable handshake communication\.In2020 IEEE International Conference on Robotics and Automation \(ICRA\),pp\. 6876–6883\.Cited by:[§2](https://arxiv.org/html/2606.29126#S2.p1.1),[Remark 4\.2](https://arxiv.org/html/2606.29126#S4.Thmtheorem2.p1.2)\.
- \[20\]Z\. Liu, L\. Wan, X\. Sui, Z\. Chen, K\. Sun, and X\. Lan\(2023\)Deep hierarchical communication graph in multi\-agent reinforcement learning\.InProceedings of the Thirty\-Second International Joint Conference on Artificial Intelligence,pp\. 208–216\.Cited by:[§1](https://arxiv.org/html/2606.29126#S1.p3.1),[§2](https://arxiv.org/html/2606.29126#S2.p2.1),[Remark 4\.2](https://arxiv.org/html/2606.29126#S4.Thmtheorem2.p1.2)\.
- \[21\]T\. Marzi, C\. Alippi, and A\. Cini\(2025\)Hierarchical message\-passing policies for multi\-agent reinforcement learning\.arXiv preprint arXiv:2507\.23604\.Cited by:[§1](https://arxiv.org/html/2606.29126#S1.p3.1),[§2](https://arxiv.org/html/2606.29126#S2.p2.1),[Remark 4\.2](https://arxiv.org/html/2606.29126#S4.Thmtheorem2.p1.2)\.
- \[22\]Y\. Niu, R\. Paleja, and M\. Gombolay\(2021\)Multi\-agent graph\-attention communication and teaming\.InProceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems,pp\. 964–973\.Cited by:[§2](https://arxiv.org/html/2606.29126#S2.p1.1),[Remark 4\.2](https://arxiv.org/html/2606.29126#S4.Thmtheorem2.p1.2)\.
- \[23\]M\. Rangwala and R\. Williams\(2020\)Learning multi\-agent communication through structured attentive reasoning\.Advances in Neural Information Processing Systems33,pp\. 10088–10098\.Cited by:[§2](https://arxiv.org/html/2606.29126#S2.p1.1),[Remark 4\.2](https://arxiv.org/html/2606.29126#S4.Thmtheorem2.p1.2)\.
- \[24\]H\. Ryu, H\. Shin, and J\. Park\(2020\)Multi\-agent actor\-critic with hierarchical graph attention network\.InProceedings of the AAAI conference on artificial intelligence,Vol\.34,pp\. 7236–7243\.Cited by:[§2](https://arxiv.org/html/2606.29126#S2.p2.1),[Remark 4\.2](https://arxiv.org/html/2606.29126#S4.Thmtheorem2.p1.2)\.
- \[25\]M\. Samvelyan, T\. Rashid, C\. Schroeder de Witt, G\. Farquhar, N\. Nardelli, T\. G\. Rudner, C\. Hung, P\. H\. Torr, J\. Foerster, and S\. Whiteson\(2019\)The starcraft multi\-agent challenge\.InProceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems,pp\. 2186–2188\.Cited by:[§A\.2\.1](https://arxiv.org/html/2606.29126#A1.SS2.SSS1.Px2.p1.7),[§A\.2\.1](https://arxiv.org/html/2606.29126#A1.SS2.SSS1.p1.3)\.
- \[26\]J\. Sheng, X\. Wang, B\. Jin, J\. Yan, W\. Li, T\. Chang, J\. Wang, and H\. Zha\(2022\)Learning structured communication for multi\-agent reinforcement learning\.Autonomous Agents and Multi\-Agent Systems36\(2\),pp\. 50\.Cited by:[§1](https://arxiv.org/html/2606.29126#S1.p3.1),[§2](https://arxiv.org/html/2606.29126#S2.p2.1),[Remark 4\.2](https://arxiv.org/html/2606.29126#S4.Thmtheorem2.p1.2)\.
- \[27\]A\. V\. Singh, E\. Rathbun, E\. Graham, L\. Oakley, S\. Boboila, P\. Chin, and A\. Oprea\(2025\)Hierarchical multi\-agent reinforcement learning for cyber network defense\.InProceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems,pp\. 2747–2749\.Cited by:[§5\.1](https://arxiv.org/html/2606.29126#S5.SS1.SSS0.Px3.p1.4)\.
- \[28\]A\. Singh, T\. Jain, and S\. Sukhbaatar\(2018\)Learning when to communicate at scale in multiagent cooperative and competitive tasks\.arXiv preprint arXiv:1812\.09755\.Cited by:[§2](https://arxiv.org/html/2606.29126#S2.p1.1),[Remark 4\.2](https://arxiv.org/html/2606.29126#S4.Thmtheorem2.p1.2)\.
- \[29\]S\. Song, Y\. Lin, S\. Han, C\. Yao, H\. Wu, S\. Wang, and K\. Lv\(2025\)Code: communication delay\-tolerant multi\-agent collaboration via dual alignment of intent and timeliness\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 23304–23312\.Cited by:[§2](https://arxiv.org/html/2606.29126#S2.p1.1),[Remark 4\.2](https://arxiv.org/html/2606.29126#S4.Thmtheorem2.p1.2)\.
- \[30\]Y\. Song, H\. Jiang, H\. Zhang, Z\. Tian, W\. Zhang, and J\. Wang\(2024\)Boosting studies of multi\-agent reinforcement learning on google research football environment: the past, present, and future\.InProceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS,Vol\.2024,pp\. 1772–1781\.Cited by:[§A\.3\.1](https://arxiv.org/html/2606.29126#A1.SS3.SSS1.Px2.p1.4),[§A\.3\.1](https://arxiv.org/html/2606.29126#A1.SS3.SSS1.Px2.p1.5),[§A\.3\.1](https://arxiv.org/html/2606.29126#A1.SS3.SSS1.p1.7),[Table 8](https://arxiv.org/html/2606.29126#A1.T8),[§5\.1](https://arxiv.org/html/2606.29126#S5.SS1.SSS0.Px5.p1.7)\.
- \[31\]S\. Sukhbaatar, A\. Szlam, and R\. Fergus\(2016\)Learning multiagent communication with backpropagation\.InProceedings of the 30th International Conference on Neural Information Processing Systems,pp\. 2252–2260\.Cited by:[§1](https://arxiv.org/html/2606.29126#S1.p1.1),[§2](https://arxiv.org/html/2606.29126#S2.p1.1)\.
- \[32\]C\. Sun, Z\. Zang, J\. Li, J\. Li, X\. Xu, R\. Wang, and C\. Zheng\(2024\)T2mac: targeted and trusted multi\-agent communication through selective engagement and evidence\-driven integration\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 15154–15163\.Cited by:[Appendix A](https://arxiv.org/html/2606.29126#A1.SS0.SSS0.Px1.p1.10),[§A\.3\.1](https://arxiv.org/html/2606.29126#A1.SS3.SSS1.Px3.p1.11),[§2](https://arxiv.org/html/2606.29126#S2.p1.1),[Remark 4\.2](https://arxiv.org/html/2606.29126#S4.Thmtheorem2.p1.2),[§5\.1](https://arxiv.org/html/2606.29126#S5.SS1.SSS0.Px1.p1.11),[§5](https://arxiv.org/html/2606.29126#S5.p1.1)\.
- \[33\]T\. Wang, J\. Wang, C\. Zheng, and C\. Zhang\(2020\)Learning nearly decomposable value functions via communication minimization\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=HJx-3grYDB)Cited by:[§2](https://arxiv.org/html/2606.29126#S2.p1.1),[Remark 4\.2](https://arxiv.org/html/2606.29126#S4.Thmtheorem2.p1.2)\.
- \[34\]T\. Wang, L\. Zeng, W\. Dong, Q\. Yang, Y\. Yu, and C\. Zhang\(2022\)Context\-aware sparse deep coordination graphs\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=wQfgfb8VKTn)Cited by:[§2](https://arxiv.org/html/2606.29126#S2.p1.1)\.
- \[35\]C\. Xu, Y\. Shi, C\. Zhang, R\. Wang, S\. Duan, Y\. Wan, and X\. Zhang\(2026\)Subgoal\-based hierarchical reinforcement learning for multiagent collaboration\.IEEE Transactions on Systems, Man, and Cybernetics: Systems\.Cited by:[§2](https://arxiv.org/html/2606.29126#S2.p2.1),[Remark 4\.2](https://arxiv.org/html/2606.29126#S4.Thmtheorem2.p1.2)\.
- \[36\]C\. Yu, A\. Velu, E\. Vinitsky, J\. Gao, Y\. Wang, A\. Bayen, and Y\. Wu\(2022\)The surprising effectiveness of ppo in cooperative multi\-agent games\.Advances in neural information processing systems35,pp\. 24611–24624\.Cited by:[§4\.1](https://arxiv.org/html/2606.29126#S4.SS1.SSS0.Px1.p1.6),[§5\.1](https://arxiv.org/html/2606.29126#S5.SS1.SSS0.Px1.p1.11),[§5](https://arxiv.org/html/2606.29126#S5.p1.1)\.
- \[37\]L\. Yuan, J\. Wang, F\. Zhang, C\. Wang, Z\. Zhang, Y\. Yu, and C\. Zhang\(2022\)Multi\-agent incentive communication via decentralized teammate modeling\.InProceedings of the AAAI conference on artificial intelligence,Vol\.36,pp\. 9466–9474\.Cited by:[§2](https://arxiv.org/html/2606.29126#S2.p1.1),[Remark 4\.2](https://arxiv.org/html/2606.29126#S4.Thmtheorem2.p1.2)\.
- \[38\]G\. Zhang, Y\. Yue, X\. Sun, G\. Wan, M\. Yu, J\. Fang, K\. Wang, T\. Chen, and D\. Cheng\(2025\)G\-designer: architecting multi\-agent communication topologies via graph neural networks\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=LpE54NUnmO)Cited by:[§2](https://arxiv.org/html/2606.29126#S2.p1.1),[Remark 4\.2](https://arxiv.org/html/2606.29126#S4.Thmtheorem2.p1.2)\.
- \[39\]S\. Q\. Zhang, Q\. Zhang, and J\. Lin\(2019\)Efficient communication in multi\-agent reinforcement learning via variance based control\.InAdvances in Neural Information Processing Systems,pp\. 3235–3244\.Cited by:[§2](https://arxiv.org/html/2606.29126#S2.p1.1),[Remark 4\.2](https://arxiv.org/html/2606.29126#S4.Thmtheorem2.p1.2)\.
- \[40\]S\. Q\. Zhang, Q\. Zhang, and J\. Lin\(2020\)Succinct and robust multi\-agent communication with temporal message control\.Advances in Neural Information Processing Systems33,pp\. 17271–17282\.Cited by:[§2](https://arxiv.org/html/2606.29126#S2.p1.1),[Remark 4\.2](https://arxiv.org/html/2606.29126#S4.Thmtheorem2.p1.2)\.
- \[41\]Z\. Zhang, B\. He, B\. Cheng, and G\. Li\(2025\)Bridging training and execution via dynamic directed graph\-based communication in cooperative multi\-agent systems\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 23395–23403\.Cited by:[§2](https://arxiv.org/html/2606.29126#S2.p1.1),[Remark 4\.2](https://arxiv.org/html/2606.29126#S4.Thmtheorem2.p1.2)\.

## Appendix AAdditional Details for Experiments

All experiments were conducted on a single node equipped with 4 NVIDIA L40S GPUs\. Representative single\-run wall\-clock times are approximately 24 hours for CC4 \(5M env steps\), 48 hours for SMACv2 5v5 \(10M env steps, averaged across protoss/terran/zerg and across IPPO/MAPPO/QMIX\), and 48 hours for GRF academy\_counterattack\_hard \(5M env steps\)\.

Sections[A\.1](https://arxiv.org/html/2606.29126#A1.SS1)through[A\.3](https://arxiv.org/html/2606.29126#A1.SS3)cover, for each benchmark, \(i\) simulator setup \(wrappers, reward, baseline ports\), \(ii\) the instantiation of𝒢,ℰ,ℱ\\mathcal\{G\},\\mathcal\{E\},\\mathcal\{F\}thatHiCommretrieves over, \(iii\) full tables extending Table[1](https://arxiv.org/html/2606.29126#S5.T1)\. Sec\.[A\.4](https://arxiv.org/html/2606.29126#A1.SS4)ablatesHiCommalong three axes \(payload form, retrieval factorisation, gradient pathway\)\.

##### Common conventions across all three benchmarks\.

*\(i\) Grouping\.*CC4 uses the simulator’s*native*subnet⊃\\supsethost containment; SMACv2 and GRF have no native containment, so we use a*semantic*grouping with one group per ally class plus one collapsed opponent group \(communication is between teammates only and class\-level addressing of an enemy is never queried\)\. The class tag \(unit type on SMACv2, on\-pitch position on GRF\) is read directly from the per\-agent observation, so no additional supervision is required to instantiate𝒢,ℰ\\mathcal\{G\},\\mathcal\{E\}\.*\(ii\) Stationary opponent\.*Each benchmark’s opposing side is frozen or scripted \(CC4: pretrained IPPO/MAPPO blue checkpoint; SMACv2: SC2 built\-in heuristic AI at difficulty7; GRF: built\-in scripted AI\), so from the controlled team’s perspective the environment is stationary throughout training and evaluation, and the only non\-stationarity comes from per\-episode randomisation \(unit composition / start positions on SMACv2; player start positions on GRF\)\.*\(iii\) Observation segments\.*On SMACv2 and GRF the per\-agent observation is a fixed\-length flat vector formed by concatenating semantically distinct segments in a canonical order; we expose this structure toHiCommas a list of\(slots,features per slot\)\(\\text\{slots\},\\,\\text\{features per slot\}\)pairs calledobs\_segs, with per\-benchmark segment lists given in the respective subsections\.*\(iv\) Baseline ports\.*T2MAC\[[32](https://arxiv.org/html/2606.29126#bib.bib11)\]andCACOM\[[17](https://arxiv.org/html/2606.29126#bib.bib12)\]were originally evaluated on SMACv1 / Hallway / MPE\-style benchmarks under QMIX; our SMACv2 and GRF ports share three structural adjustments: \(a\) feed the per\-agent observation into each baseline’s original observation encoder unchanged but hand it the canonicalobs\_segslayout, so its per\-entity attention head sees ally and enemy rows as separate tokens \(the same layoutHiCommuses, ensuring all three protocols index the same per\-agent state\); \(b\) additionally train both baselines under IPPO and MAPPO with the same actor/critic architectures, PPO hyperparameters, entropy floor, and gradient clip values as our other policy\-gradient baselines\.*\(v\) Mask scope\.*Per Sec\.[4\.2](https://arxiv.org/html/2606.29126#S4.SS2), the only data\-dependent mask consumed byHiCommis the Phase 3 entity\-stage maskMj∗\(ent\)​\(g∗\)M^\{\(\\text\{ent\}\)\}\_\{j^\{\*\}\}\(g^\{\*\}\), derived deterministically from senderj∗j^\{\*\}’s own observability maskMj∗M^\{j^\{\*\}\}\. We therefore only specify whenMj​\[g,ℓ\]=1M^\{j\}\[g,\\ell\]=1holds for each benchmark and let the Phase 3 mask follow mechanically\. TheHiCommmodules of Sec\.[4\.2](https://arxiv.org/html/2606.29126#S4.SS2)are unchanged across the three benchmarks; only the partition𝒢,ℰ\\mathcal\{G\},\\mathcal\{E\}, the feature spaceℱ\\mathcal\{F\}, and the semantics ofMjM^\{j\}differ\.

### A\.1CAGE Challenge 4 \(CC4\)

This subsection covers \(i\) the wrapper stack that turns CC4 into a red\-side MARL task, \(ii\) the native subnet⊃\\supsethost hierarchy used byHiCommon this benchmark, and \(iii\) the full results table extending Table[1](https://arxiv.org/html/2606.29126#S5.T1)\.

#### A\.1\.1Red\-Side MARL Reformulation

The original CC4 release\[[13](https://arxiv.org/html/2606.29126#bib.bib18),[12](https://arxiv.org/html/2606.29126#bib.bib19)\]distributes the environment as a*blue\-side only*training problem\. Concretely, the official PettingZoo interface exposes the five blue defenders as policy\-gradient agents with per\-step observations, an action mask, and a shared team reward, while the red team is hard\-wired to a scripted finite\-state machine \(FiniteStateRedAgent\) that selects actions from internal heuristics; red has no observation tensor, no reward channel, and no policy\-callable action interface\. Any attempt to train red with a standard RL algorithm against this stock release would therefore require either rewriting the simulator or replaying the FSM as a fixed environmental dynamic\. Because this paper contributes a communication protocol for the*attacker*side, we reformulate red as a first\-class MARL problem, keeping the simulator itself untouched and instead adding a thin wrapper stack around it that turns red into a standard PettingZoo multi\-agent task\. The result is drop\-in compatible with off\-the\-shelf IPPO, MAPPO, and QMIX runners and is what every red\-side training and evaluation number in this paper is run against\.

##### Observation and action spaces\.

ARedFlatWrapperexposes, for each of the six red agents, a722722\-dimensional observation that concatenates the9×16×5=7209\{\\times\}16\{\\times\}5\{=\}720host matrix \(subnet→\\tohost, five per\-host features: access level, recon level,neighbours\_discovered,is\_reachable,deception\_known\) with two global scalars \(current mission phase, previous\-action success flag\)\. The action space is factorised as a pair\(atype,\(asubnet,ahost\)\)\(a\_\{\\text\{type\}\},\(a\_\{\\text\{subnet\}\},a\_\{\\text\{host\}\}\)\)over ten primitive actions \(DiscoverRemoteSystems,StealthServiceDiscovery,AggressiveServiceDiscovery,ExploitRemoteService,DegradeServices,DiscoverDeception,Withdraw,PrivilegeEscalate,Impact,Sleep\) and all9×169\{\\times\}16target coordinates, filtered by a per\-step validity mask so the policy never commits to a syntactically illegal action\.

##### Attack chain intoImpact\.

Impactis not directly invocable on an arbitrary host: red must first discover the subnet, scan the host’s services, exploit a service for a user\-level session, and locally escalate to root\. Table[2](https://arxiv.org/html/2606.29126#A1.T2)names the five canonical stages with their advancing actions and per\-host preconditions\. BecauseImpactis the*terminal*stage of every successful trajectory, the count of successfulImpacts is exactly the number of hosts on which red completed the entire five\-stage chain in an episode; we report it alongside reward in Table[4](https://arxiv.org/html/2606.29126#A1.T4)as a behavioural sanity check, since a method that inflates reward by spamming intermediate\-stage actions \(e\.g\.DegradeServices\) would leave Successful \#Impact unchanged\.

Table 2:Canonical red attack chain intoImpacton CC4: stage, advancing action\(s\), and the per\-host precondition each stage establishes on\(access\_level,recon\_level,…\)\(\\texttt\{access\\\_level\},\\texttt\{recon\\\_level\},\\ldots\)\.Impactsucceeds only on root\-owned hosts with an active OT service \(OZA, OZB only\)\.\#StageCC4 action\(s\)Effect / pre\-condition established1DiscoverDiscoverRemoteSystemsSubnet recon: hosts in the target subnet become known \(recon\_level==11\)\.2ScanStealthServiceDiscovery,AggressiveServiceDiscoveryPer\-host service enumeration:recon\_level11→\\to22, exposing exploitable services\.3ExploitExploitRemoteServiceGains a user\-level session on the host:access\_level0→\\to11\.4EscalatePrivilegeEscalateLocal privilege escalation:access\_level11→\\to22\(root\)\.5ImpactImpactDisrupts the host’s active OT service; succeeds only on root\-owned hosts in OZA/OZB; yields the largest positive entries of Table[3](https://arxiv.org/html/2606.29126#A1.T3)\(in the red\-reward convention used throughout this paper\)\.
##### Reward function\.

On top of the stationary\-opponent setup of Sec\.[A](https://arxiv.org/html/2606.29126#A1)\(here aHybridFixedBlueWrapperplugs in a fixed pretrained IPPO or MAPPO blue policy, the standard CC4 blue baseline\), CC4’s official release\[[13](https://arxiv.org/html/2606.29126#bib.bib18),[12](https://arxiv.org/html/2606.29126#bib.bib19)\]bills three blue\-side event types per step:Local Work Fails\(a green agent’s local work action fails, e\.g\. while a blue agent is restoring its host\),Access Service Fails\(a green agent cannot connect to a service it relies on\), andRed impact/access\(red has runImpacton, or otherwise compromised, a host that a green agent occupies\)\. Penalties are zone\-specific and phase\-dependent\. We report all CC4 rewards in the*red\-reward convention*rr=−rbr^\{r\}=\-r^\{b\}to keep “higher is better” uniform across benchmarks; the simulator is unchanged\. Table[3](https://arxiv.org/html/2606.29126#A1.T3)consolidates CC4 Tables 4A\-4C into a single zone×\\timesphase grid in this convention \(OZA spikes in Phase 2a, OZB in Phase 2b, CON only in Phase 1, INT never contributes\)\. On each step our wrapper restricts the blue rewardsr1:5br^\{b\}\_\{1:5\}to the subsetℬt\\mathcal\{B\}\_\{t\}of defenders co\-occurring with red under the current phase’s connectivity policy and assigns every red agent the shared scalar

rtr=−1\|ℬt\|​∑b∈ℬtrtb,r^\{r\}\_\{t\}\\;=\\;\-\\frac\{1\}\{\|\\mathcal\{B\}\_\{t\}\|\}\\sum\_\{b\\in\\mathcal\{B\}\_\{t\}\}r^\{b\}\_\{t\},\(A\.1\)which is zero\-sum on the co\-occurring slice and is the only modification to CC4’s reward semantics\.

Table 3:Red rewards across the three CC4 mission phases \(CC4 Tables 4A through 4C of\[[13](https://arxiv.org/html/2606.29126#bib.bib18),[12](https://arxiv.org/html/2606.29126#bib.bib19)\]consolidated\)\.PhaseGreen action / compromiseHQCONRZAOZARZBOZBINTPhase 1 O&MLocal Work Fails110111111110Access Service Fails1155331133110Red impact/access3355111111110Phase 2a \(Mission A\)Local Work Fails11022101011110Access Service Fails11011011110Red impact/access33033101011110Phase 2b \(Mission B\)Local Work Fails11011112210100Access Service Fails11033111100Red impact/access33033113310100

#### A\.1\.2Hierarchy Instantiation

CC4’s native subnet⊃\\supsethost containment is used directly:𝒢=\{RZA, OZA, RZB, OZB, CON, PAZ, ADM, OFF, INT\}\\mathcal\{G\}\{=\}\\\{\\text\{RZA, OZA, RZB, OZB, CON, PAZ, ADM, OFF, INT\}\\\}\(the nine enterprise subnets\),\|ℰ\|=16\|\\mathcal\{E\}\|\{=\}16hosts per subnet,\|ℱ\|=5\|\\mathcal\{F\}\|\{=\}5per\-host features \(access level, recon level, neighbours discovered, reachable, deception known\)\. Observability mask:Mj​\[g,ℓ\]=1M^\{j\}\[g,\\ell\]=1iff hostℓ\\ellin subnetgghasrecon\_level≥1\\geq 1in red agentjj’s belief state\.

#### A\.1\.3Detailed Results

Table[4](https://arxiv.org/html/2606.29126#A1.T4)extends the CC4 columns of Table[1](https://arxiv.org/html/2606.29126#S5.T1)with \(i\) theNaiveComreference \(the canonical broadcast\-autoencoder protocol, run to expose the encoder/reward decoupling pathology that motivatesHiComm’s observation\-entry payload design over a learned reconstruction objective; Sec\.[4\.2](https://arxiv.org/html/2606.29126#S4.SS2)\); \(ii\) the FSM scripted opponent as a non\-RL reference; and \(iii\) Successful/TotalImpactcounts as a behavioural sanity check\. TheNaiveCompathology is visible directly: totalImpactattempts inflate while successful ones collapse belowPartialObsunder both algorithms, because reconstruction preserves what minimises MSE rather than what helps teammates succeed\.HiComm’s observation\-entry response avoids this failure mode, keeping successfulImpactat or abovePartialObswhile substantially improving reward\.

Table 4:CC4 full grid over100100evaluation episodes \(red\-reward convention\)\. Successful \#Impact countsImpactsteps that succeeded; Total \#Impact counts attempts\. The two columns serve as a behavioural sanity check against strategy drift\.Bold==overall best;underline==best comm method within each algorithm block\.AlgorithmObservationRewardSuccess \#ImpactTotal \#ImpactFSMpartial1811813463468,8608\{,\}860IPPOFullObs𝟏,𝟕𝟔𝟑\\mathbf\{1\{,\}763\}𝟕𝟐𝟎\\mathbf\{720\}49,28149\{,\}281IPPOPartialObs1,4361\{,\}43624624644,28244\{,\}282IPPONaiveCom1,603¯\\underline\{1\{,\}603\}16916952,04152\{,\}041IPPOHiComm\(ours\)1,5271\{,\}527262¯\\underline\{262\}49,49949\{,\}499MAPPOFullObs𝟐,𝟏𝟏𝟎\\mathbf\{2\{,\}110\}𝟒𝟗𝟕\\mathbf\{497\}50,29650\{,\}296MAPPOPartialObs99099023223230,28730\{,\}287MAPPONaiveCom1,5431\{,\}54316516554,02654\{,\}026MAPPOHiComm\(ours\)1,941¯\\underline\{1\{,\}941\}246¯\\underline\{246\}78,21678\{,\}216

### A\.2SMACv2

This subsection covers \(i\) the SMACv2 simulator setup we use, including the per agent observation and action spaces, the reward function in detail, and the ports of theT2MACandCACOMbaselines from SMACv1 to SMACv2 needed to run them on this benchmark, \(ii\) the semantic \(unit type\) hierarchy instantiationHiCommruns on this benchmark, and \(iii\) the full3×53\{\\times\}5grid extending Table[1](https://arxiv.org/html/2606.29126#S5.T1)\.

#### A\.2\.1Setup and Baseline Ports

The original SMACv2 release\[[5](https://arxiv.org/html/2606.29126#bib.bib26)\]ships the StarCraft II micromanagement benchmark ofSamvelyanet al\.\[[25](https://arxiv.org/html/2606.29126#bib.bib27)\]with two changes that defeat the memorisation shortcuts of SMACv1: per episode unit type compositions are sampled stochastically from a race specific distribution \(weighted\_teamsonstalker/zealot/colossusforprotoss, onmarine/marauder/medivacforterran, and onzergling/hydralisk/banelingforzerg\), and start positions are sampled from asurrounded\_and\_reflectdistribution on a32×3232\{\\times\}32map rather than fixed across episodes\. The reward schedule, per agent observation construction, and primitive action set are inherited from SMACv1 unchanged\. We adopt the released configs verbatim for the55v55grid \(n\_units=5,n\_enemies=5,step\_mul=8, difficulty7\) and turn oncapability\_config\.team\_gen\.observe=Trueso the per agent observation includes the sampled unit type one hot, which is what makes the role / position indexed hierarchy of HiComm addressable from the observation alone\.

##### Observation and action spaces\.

The per\-agent observation segments \(canonical order, exposed toHiCommasobs\_segs; cf\. Sec\.[A](https://arxiv.org/html/2606.29126#A1)\) are:

1. \(1\)*Move availability*,\(1,4\)\(1,4\)— one bit per cardinal direction \(N/S/E/W\) indicating whether the unit can step that way under map collision\.
2. \(2\)*Enemy entities*,\(Ne,\|ℱ\|\)\(N\_\{e\},\|\\mathcal\{F\}\|\)— one row per enemy slot\.
3. \(3\)*Ally entities \(excluding self\)*,\(Na−1,\|ℱ\|\)\(N\_\{a\}\{\-\}1,\|\\mathcal\{F\}\|\)— one row per teammate slot, in the same per entity format as the enemy rows\.
4. \(4\)*Own state*,\(1,oa\)\(1,o\_\{a\}\)— current health, current shield \(protossonly\), and a unit class one hot of length33\.
5. \(5\)*Last action one hot*,\(1,11\)\(1,11\)— the agent’s previous primitive action; the55v55action space hasNe\+6=11N\_\{e\}\+6=11entries \(see action space description below\)\.
6. \(6\)*Agent identity one hot*,\(1,5\)\(1,5\)— the agent’s slot index inside the controlled team of sizeNa=5N\_\{a\}=5\.

The per entity feature vectorF∈ℝ\|ℱ\|F\\in\\mathbb\{R\}^\{\|\\mathcal\{F\}\|\}shared by enemy and ally rows carries the same fields in the same order: visibility bit, relative distance, relativexx, relativeyy, current health, current shield \(protossonly\), and a33\-way unit class one hot\. This gives\|ℱ\|=9\|\\mathcal\{F\}\|\{=\}9onprotossand\|ℱ\|=8\|\\mathcal\{F\}\|\{=\}8onterran/zerg; the own state widthoao\_\{a\}tracks the same shield distinction, withoa=5o\_\{a\}\{=\}5onprotossandoa=4o\_\{a\}\{=\}4onterran/zerg\. Absent slots — a sampled team smaller than the per race upper bound, dead units, or units outside sight range — have their entire row zeroed and their visibility bit set to0\. On the55v55scenarios this resolves\(Ne,Na−1\)\(N\_\{e\},N\_\{a\}\{\-\}1\)to\(5,4\)\(5,4\), so the concreteobs\_segswe hand toHiCommis

\[\(1,4\),\(5,\|ℱ\|\),\(4,\|ℱ\|\),\(1,oa\),\(1,11\),\(1,5\)\],\\big\[\(1,4\),\\ \(5,\|\\mathcal\{F\}\|\),\\ \(4,\|\\mathcal\{F\}\|\),\\ \(1,o\_\{a\}\),\\ \(1,11\),\\ \(1,5\)\\big\],with\|ℱ\|∈\{8,9\}\|\\mathcal\{F\}\|\\in\\\{8,9\\\}andoa∈\{4,5\}o\_\{a\}\\in\\\{4,5\\\}chosen by race as above\. The action space is a discrete head of sizeNe\+6N\_\{e\}\+6\(NeN\_\{e\}unit targeted attacks plusno\-op,stop, and the four cardinal moves\), with a per step validity mask that zeroes out attacks on out of sight enemies, moves blocked by terrain or map borders, andno\-opunless the unit is dead\. Sight range and shoot range are inherited from the SMACv1 unit table and are unit class specific \(e\.g\. Stalker sight1010, Marine sight99\); we do not modify these\.

##### Reward function\.

SMACv2 inherits the dense battle reward of SMACv1\[[25](https://arxiv.org/html/2606.29126#bib.bib27)\]unchanged; SMACv2’s only deviations from v1 are the stochastic team composition and start positions, not the reward\. Withreward\_only\_positive=True\(default\), per\-step team reward isrt=Δ​Htenemy\+ρdeath⋅Δ​Dtenemyr\_\{t\}=\\Delta H\_\{t\}^\{\\text\{enemy\}\}\+\\rho\_\{\\text\{death\}\}\\cdot\\Delta D\_\{t\}^\{\\text\{enemy\}\}, whereΔ​H\\Delta His health\+\+shield damage dealt andΔ​D\\Delta Dis enemy units killed; on the terminal step a win bonusRwinR\_\{\\text\{win\}\}is added if the enemy team is fully eliminated\. The unscaled cumulative return is divided by a normaliserZZso an idealised maximum episode return is approximately2020:

R=1Z​\(∑t=1Trt\+Rwin⋅𝟙​\[win\]\),Z=Henemy,total\+ρdeath⋅Nenemy\+RwinS,R\\;=\\;\\frac\{1\}\{Z\}\\\!\\Big\(\\\!\\sum\_\{t=1\}^\{T\}r\_\{t\}\+R\_\{\\text\{win\}\}\\\!\\cdot\\\!\\mathbb\{1\}\[\\text\{win\}\]\\Big\),\\qquad Z\\;=\\;\\frac\{H\_\{\\text\{enemy,total\}\}\+\\rho\_\{\\text\{death\}\}\\cdot N\_\{\\text\{enemy\}\}\+R\_\{\\text\{win\}\}\}\{S\},\(A\.2\)withS=reward\_scale\_rateS\{=\}\\texttt\{reward\\\_scale\\\_rate\},Henemy,totalH\_\{\\text\{enemy,total\}\}the sum of initial health\+\+shield over enemy units, andNenemyN\_\{\\text\{enemy\}\}the enemy team size\. Table[5](https://arxiv.org/html/2606.29126#A1.T5)lists the coefficients used \(matched to the official SMACv2 release\)\. The shaping is identical acrossFullObs,PartialObs,CACOM,T2MAC, andHiComm, so the only mechanism by which a communication method can improve return is by changing what each agent observes\.

Table 5:SMACv2 reward / scaling coefficients \(verbatim from\[[5](https://arxiv.org/html/2606.29126#bib.bib26)\], used unchanged across all comm settings\)\.SymbolConfig flagValueρdeath\\rho\_\{\\text\{death\}\}reward\_death\_value1010RwinR\_\{\\text\{win\}\}reward\_win200200RdefeatR\_\{\\text\{defeat\}\}reward\_defeat0αneg\\alpha\_\{\\text\{neg\}\}reward\_negative\_scale−0\.5\-0\.5reward\_only\_positiveTruereward\_scaleTrueSSreward\_scale\_rate2020reward\_sparseFalse
##### Baseline ports from SMACv1 to SMACv2\.

On top of the common port adjustments of Sec\.[A](https://arxiv.org/html/2606.29126#A1), SMACv2 introduces one additional concern: SMACv1 has a fixed unit composition per scenario, so per\-teammate output heads in the original baselines were sized to a fixedNaN\_\{a\}, whereas SMACv2 keeps the team size fixed but resamples*which*unit class fills each slot per episode\. We rely on the unit\-type one\-hot inside the per\-entity row to disambiguate slot identity rather than baking it into the architecture, so the per\-teammate head dimensions ofT2MACandCACOMare unchanged across episodes\.

*Per\-baseline specifics on SMACv2\.*ForT2MAC, the evidence encoder emits ann⋅Kn\\\!\\cdot\\\!K\-dimensional Dirichlet concentration per teammate withK=Ne\+6=11K\{=\}N\_\{e\}\+6\{=\}11on55v55, and the selector network, uncertainty\-delta pseudo\-labels, DST combination rule, and auxiliary BCE loss are taken from the published code without modification\. ForCACOM, latent dim=8=8, attention dim=32=32, and the discrete payload uses22bits per channel, matching the released SMACv1 configs; the two\-round broadcast / gated\-reply schedule, mutual\-information loss weight \(10−310^\{\-3\}\), entropy regulariser \(10−210^\{\-2\}\), gating threshold, and gate training schedule are all unchanged\. We disclose these ports so readers do not conflate our SMACv2 numbers with the SMACv1 numbers in the original papers\.

##### Training budget\.

Every \(algorithm, method, scenario\) is trained for1010M environment steps, identically across all five communication settings \(FullObs,PartialObs,CACOM,T2MAC,HiComm\) and all three backbones \(QMIX, IPPO, MAPPO\)\. The budget is held fixed across the3×53\{\\times\}5grid so that any cross method or cross algorithm differences in win rate and scaled return reflect the protocol or the optimiser, not the wall clock allowance\.

#### A\.2\.2Hierarchy Instantiation

SMACv2 has no native containment, so per the common convention \(Sec\.[A](https://arxiv.org/html/2606.29126#A1)\) we use*semantic*grouping with ally unit type as the group axis\. Concretely,\|𝒢\|=\|𝒞ally\|\+1\|\\mathcal\{G\}\|\{=\}\|\\mathcal\{C\}\_\{\\text\{ally\}\}\|\+1\(one group per active ally type, plus one collapsed opponent group\); per\-race taxonomies areprotoss=\{=\\\{Zealot, Stalker, opponent\}\\\},terran=\{=\\\{Marine, Marauder, Medivac, opponent\}\\\},zerg=\{=\\\{Zergling, Hydralisk, Baneling, opponent\}\\\}\. Each group’sℰ\\mathcal\{E\}holds the unit slots padded to per\-group capacity\. The per\-unit feature vector matches the native per\-unit feature count:\|ℱ\|=9\|\\mathcal\{F\}\|\{=\}9onprotoss,\|ℱ\|=8\|\\mathcal\{F\}\|\{=\}8onterran/zerg\. Observability mask:Mj​\[g,ℓ\]=1M^\{j\}\[g,\\ell\]=1iff senderjjcurrently sees the unit at slotℓ\\ellof groupgg\(alive, within sight range, non\-padding\)\.

#### A\.2\.3Detailed Results

Table[6](https://arxiv.org/html/2606.29126#A1.T6)extends Table[1](https://arxiv.org/html/2606.29126#S5.T1)with the QMIX backbone, reporting reward alongside per\-episode communication cost; Table[7](https://arxiv.org/html/2606.29126#A1.T7)consolidates the underlying win rate across all three backbones \(QMIX, IPPO, MAPPO\)\. Bandwidth for IPPO and MAPPO is reported in the main paper and not duplicated here\.

##### Win rate definition and computation\.

FollowingElliset al\.\[[5](https://arxiv.org/html/2606.29126#bib.bib26)\], an evaluation episode is a*win*iff every enemy unit dies strictly before the horizon*and*at least one ally is alive at that moment; horizon timeouts and ally wipes both count as losses \(no draws\)\. Per \(algorithm, method, scenario, seed\) tuple we report the mean win rate over the lastK=10K\{=\}10evaluation checkpoints \(each running3232episodes with greedy/argmax action selection\)\.

The QMIX bandwidth signature is consistent with the IPPO and MAPPO ones in the main paper: under QMIX,HiCommis the best communication method on all three55v55scenarios \(\+2\.9\+2\.9,\+1\.6\+1\.6,\+1\.4\+1\.4reward points overCACOMand\+3\.3\+3\.3,\+2\.0\+2\.0,\+1\.6\+1\.6overT2MAC, equivalent to\+25\+25,\+12\+12,\+10\+10and\+28\+28,\+15\+15,\+12\+12win rate points respectively\), closing most of the gap toFullObs, and does so at roughly4×4\{\\times\}lower bandwidth thanCACOMand roughly18×18\{\\times\}lower thanT2MAC\(10\.310\.3vs\.39\.139\.1vs\.185\.1185\.1KB per episode onprotoss\_5v5\)\.

Table 6:SMACv2 QMIX block, extending Table[1](https://arxiv.org/html/2606.29126#S5.T1)\.protossterranzergAlgorithmMethodRewardComm \(KB\)RewardComm \(KB\)RewardComm \(KB\)QMIXFullObs17\.8\\mathbf\{17\.8\}193\.8193\.817\.9\\mathbf\{17\.9\}152\.5152\.514\.9\\mathbf\{14\.9\}95\.295\.2QMIXPartialObs16\.316\.3016\.016\.0012\.312\.30QMIXCACOM14\.514\.539\.139\.116\.016\.028\.328\.312\.512\.515\.915\.9QMIXT2MAC14\.114\.1185\.1185\.115\.615\.6142\.8142\.812\.312\.385\.585\.5QMIXHiComm\(ours\)17\.4¯\\underline\{17\.4\}10\.3¯\\underline\{\\mathbf\{10\.3\}\}17\.6¯\\underline\{17\.6\}8\.4¯\\underline\{\\mathbf\{8\.4\}\}13\.9¯\\underline\{13\.9\}5\.4¯\\underline\{\\mathbf\{5\.4\}\}

Table 7:SMACv255v55win rate \(%\), full3×53\{\\times\}5grid after1010M steps,3232evaluation episodes\.AlgorithmMethodprotossterranzergQMIXFullObs𝟖𝟏\\mathbf\{81\}𝟖𝟒\\mathbf\{84\}𝟔𝟑\\mathbf\{63\}QMIXPartialObs696969694444QMIXCACOM535369694646QMIXT2MAC505066664444QMIXHiComm\(ours\)78¯\\underline\{78\}81¯\\underline\{81\}56¯\\underline\{56\}IPPOFullObs𝟕𝟓\\mathbf\{75\}𝟕𝟖\\mathbf\{78\}𝟓𝟗\\mathbf\{59\}IPPOPartialObs535356563838IPPOCACOM636366664444IPPOT2MAC595963634444IPPOHiComm\(ours\)69¯\\underline\{69\}72¯\\underline\{72\}53¯\\underline\{53\}MAPPOFullObs𝟕𝟖\\mathbf\{78\}𝟕𝟐\\mathbf\{72\}𝟔𝟑\\mathbf\{63\}MAPPOPartialObs565650504444MAPPOCACOM666659595050MAPPOT2MAC636356564747MAPPOHiComm\(ours\)72¯\\underline\{72\}66¯\\underline\{66\}56¯\\underline\{56\}

### A\.3Google Research Football \(GRF\)

This subsection covers \(i\) the GRF simulator setup we use, including the per agent observation and action spaces, the reward function in detail, and the ports of theT2MACandCACOMbaselines to GRF, \(ii\) the semantic \(player position\) hierarchy instantiationHiCommruns on this benchmark, and \(iii\) the full results table extending the GRF column of Table[1](https://arxiv.org/html/2606.29126#S5.T1)with the underlying mean win rate \(%\), the GRF community convention metric, onacademy\_counterattack\_hard, the sole GRF scenario in our setup \(Section[5\.1](https://arxiv.org/html/2606.29126#S5.SS1.SSS0.Px5)\)\.

#### A\.3\.1Setup and Baseline Ports

The original GRF release\[[15](https://arxiv.org/html/2606.29126#bib.bib33)\]ships a continuous control football simulator with a discrete primitive action set and a small set of*academy*scenarios designed to isolate specific tactical motifs\. We adoptacademy\_counterattack\_hard\(44v33with keeper\) as the sole GRF scenario in our setup: a short fast break in which the controlled side hasNleft=4N\_\{\\text\{left\}\}\{=\}4left team players \(three controllable attackers and one AI keeper\) racing againstNright=3N\_\{\\text\{right\}\}\{=\}3right team players \(22defenders plus a goalkeeper\) before the defenders can reorganise\. The goalkeeper on the controlled side is handed off to GRF’s built in scripted policy following the academy scenario protocol ofSonget al\.\[[30](https://arxiv.org/html/2606.29126#bib.bib45)\], leavingNa=3N\_\{a\}\{=\}3controllable attackers; the right team is fully scripted\. Episodes are capped at400400steps and we use therepresentation="raw"observation channel so the per agent feature encoder has access to ball state, sticky actions, role indices, and team positions, matching the encoder ofKurachet al\.\[[15](https://arxiv.org/html/2606.29126#bib.bib33)\], Songet al\.\[[30](https://arxiv.org/html/2606.29126#bib.bib45)\]that subsequent learned communication work also builds on\.

##### Observation and action spaces\.

The per\-agent observation segments \(canonical order, exposed toHiCommasobs\_segs; cf\. Sec\.[A](https://arxiv.org/html/2606.29126#A1)\) are:

1. \(1\)*Own / player*,\(1,29\)\(1,29\)— the agent’s position, direction, speed, a role one hot of length1010, and four state flags \(ball far, tiredness, dribbling, sprinting\)\.
2. \(2\)*Ball*,\(1,18\)\(1,18\)— ball position, a ball zone one hot of length66, ball relative position, ball direction, ball speed, ball distance to the agent, ball ownership flag, and an “owned by us” flag\.
3. \(3\)*Enemy entities*,\(Ne,7\)\(N\_\{e\},7\)— one row per enemy slot, with scaled position, direction, speed, distance to agent, and tiredness\.
4. \(4\)*Ally entities \(excluding self\)*,\(Nleft−1,7\)\(N\_\{\\text\{left\}\}\{\-\}1,7\)— one row per left\-team teammate slot \(i\.e\. the other policy\-controlled attackers*and*the scripted goalkeeper, withNleft=4N\_\{\\text\{left\}\}\{=\}4\), in the same per\-entity format as the enemy rows\.
5. \(5\)*Closest left teammate summary*,\(1,7\)\(1,7\)— a single per entity row for the closest left team player to the agent\.
6. \(6\)*Closest right opponent summary*,\(1,7\)\(1,7\)— a single per entity row for the closest right team player to the agent\.

Crucially, the per entity feature vectorF∈ℝ\|ℱ\|F\\in\\mathbb\{R\}^\{\|\\mathcal\{F\}\|\}is identical for the ally and enemy segments with\|ℱ\|=7\|\\mathcal\{F\}\|\{=\}7, which is the requirement under which a singleHiCommresponse head can return either a teammate or an opponent row\. Onacademy\_counterattack\_hardthe entity counts instantiate as\(Ne,Nleft−1\)=\(3,3\)\(N\_\{e\},N\_\{\\text\{left\}\}\{\-\}1\)=\(3,3\)\(note the ally segment countsNleft−1N\_\{\\text\{left\}\}\{\-\}1teammates including the scripted keeper, notNa−1N\_\{a\}\{\-\}1controllable attackers\), so the concreteobs\_segswe hand toHiCommis

\[\(1,29\),\(1,18\),\(3,7\),\(3,7\),\(1,7\),\(1,7\)\],\\big\[\(1,29\),\\ \(1,18\),\\ \(3,7\),\\ \(3,7\),\\ \(1,7\),\\ \(1,7\)\\big\],giving a per agent observation size of29\+18\+21\+21\+7\+7=10329\+18\+21\+21\+7\+7=103dims\. TheFullObsupper bound concatenates the per agent observation across allNa=3N\_\{a\}\{=\}3controllable attackers \(each re centred around its ownactiveindex by the feature encoder\), giving an actor input of3×103=3093\\times 103=309dims\. The action space is the GRF default1919dim discrete head:idle, eight directional moves \(left, top left, top, top right, right, bottom right, bottom, bottom left\),long\_pass,high\_pass,short\_pass,shot,sprint,release\_move,release\_sprint,slide,dribble, andrelease\_dribble; a per step validity mask zeroes out passes / shots when the agent does not own the ball or is too far from it, slides when the team owns the ball, set piece restrictions during goal kicks / corner kicks / penalty kicks, and thereleaseactions whenever the matching sticky action is not currently engaged\.

##### Reward function\.

GRF natively exposes two reward signals,SCORINGandCHECKPOINT, defined in the original release\[[15](https://arxiv.org/html/2606.29126#bib.bib33)\]and re\-formalised as the*Sparse*and*Dense*variants bySonget al\.\[[30](https://arxiv.org/html/2606.29126#bib.bib45)\]\.SCORINGreturns

rtscore=\+1⋅𝟙​\[ally goal at​t\]−1⋅𝟙​\[enemy goal at​t\],r\_\{t\}^\{\\text\{score\}\}\\;=\\;\+1\\cdot\\mathbb\{1\}\[\\text\{ally goal at \}t\]\\;\-\\;1\\cdot\\mathbb\{1\}\[\\text\{enemy goal at \}t\],\(A\.3\)i\.e\. a\{−1,0,\+1\}\\\{\-1,0,\+1\\\}sparse signal that fires only on goal events\.CHECKPOINTpartitions the half pitch in front of the opponent goal into ten concentric checkpoint regions\{C1,…,C10\}\\\{C\_\{1\},\\ldots,C\_\{10\}\\\}ordered by decreasing distance to goal, and grants

rtcp=∑k=1100\.1⋅𝟙​\[first crossing of​Ck​by ally ball\-carrier at​t\],r\_\{t\}^\{\\text\{cp\}\}\\;=\\;\\sum\_\{k=1\}^\{10\}0\.1\\cdot\\mathbb\{1\}\\\!\\big\[\\text\{first crossing of \}C\_\{k\}\\text\{ by ally ball\-carrier at \}t\\big\],\(A\.4\)which sums to at most1\.01\.0per episode \(each region is rewarded at most once\) and resets on possession exchange\. The per step team reward we optimise is the*Dense*sum

rt=rtscore\+rtcp,r\_\{t\}\\;=\\;r\_\{t\}^\{\\text\{score\}\}\\;\+\\;r\_\{t\}^\{\\text\{cp\}\},\(A\.5\)shared across allNaN\_\{a\}controlled players\. We instantiate eq\. \([A\.5](https://arxiv.org/html/2606.29126#A1.E5)\) via the GRF flagrewards="scoring,checkpoints", matching the*Dense*setting thatSonget al\.\[[30](https://arxiv.org/html/2606.29126#bib.bib45)\]report as the strictly better variant on academy scenarios; Table[8](https://arxiv.org/html/2606.29126#A1.T8)lists the per\-step contributions\. Eq\. \([A\.5](https://arxiv.org/html/2606.29126#A1.E5)\) is used identically acrossFullObs,PartialObs,CACOM,T2MAC, andHiComm\(cf\. SMACv2\)\. The reported*win rate*depends only on theSCORINGcomponent and is therefore invariant to theCHECKPOINTshaping coefficient\.

Table 8:GRF per\-step reward onacademy\_counterattack\_hard,*Dense*setting ofSonget al\.\[[30](https://arxiv.org/html/2606.29126#bib.bib45)\]\(rewards="scoring,checkpoints"\)\. Each event adds to the shared scalarrtr\_\{t\}broadcast to every attacker\. TotalCHECKPOINTcapped at\+1\.0\+1\.0per episode\.EventRewardAlly goal at steptt\+1\.0\+1\.0Enemy goal at steptt−1\.0\-1\.0First crossing of checkpoint regionCkC\_\{k\}by ally ball\-carrier \(k=1,…,10k=1,\\ldots,10\)\+0\.1\+0\.1Possession transfer to enemy teamresets unclaimedCkC\_\{k\}Episode horizon reached without a goal0
##### Baseline ports to GRF\.

On top of the common port adjustments of Sec\.[A](https://arxiv.org/html/2606.29126#A1), the only GRF\-specific change is that the action head is the1919\-way GRF discrete set rather thanK=11K\{=\}11\(SMACv255v55\); bothT2MAC’s Dirichlet/Subjective\-Logic/DST machinery andCACOM’s gated\-reply/mutual\-information stack are defined over a discrete action space, so the port requires only the action\-size change\.T2MAC’s belief mass and uncertainty follow Eq\. \(2\) ofSunet al\.\[[32](https://arxiv.org/html/2606.29126#bib.bib11)\]withK=19K\{=\}19;CACOMreuses the SMACv2 port’s latent dim=8=8, attention dim=32=32,22\-bit channel, MI weight10−310^\{\-3\}, and entropy regulariser10−210^\{\-2\}verbatim, with the only environment\-specific quantity being the observation encoder input width \(103103onacademy\_counterattack\_hard\)\.

##### Training budget\.

Every \(algorithm, method\) configuration in Tables[1](https://arxiv.org/html/2606.29126#S5.T1)and[9](https://arxiv.org/html/2606.29126#A1.T9)is trained for55M environment steps onacademy\_counterattack\_hard, allocated equally across the five communication settings \(FullObs,PartialObs,CACOM,T2MAC,HiComm\) under each of IPPO and MAPPO\. The same55M\-step budget is used for the baseline ports above so that any reward / win rate gap is attributable to the message protocol rather than to a difference in optimisation horizon\. The reported win rate is averaged over3232evaluation episodes against the same scripted opponent at the end of training\.

#### A\.3\.2Hierarchy Instantiation

Per the common convention \(Sec\.[A](https://arxiv.org/html/2606.29126#A1)\), GRF uses*semantic*grouping with each ally’s on\-pitch position class as the group axis \(drawn from\{GK,DEF,MID,FWD\}\\\{\\text\{GK\},\\text\{DEF\},\\text\{MID\},\\text\{FWD\}\\\}via the role one\-hot of segment \(1\)\) plus one collapsed opponent group\.academy\_counterattack\_hard\(44v33\) instantiates\{\\\{ally MID, ally FWD, opponent\}\\\}\(the opponent group collapses enemy DEF and GK\)\. Each group’sℰ\\mathcal\{E\}holds the player slots padded to per\-group capacity;\|ℱ\|=7\|\\mathcal\{F\}\|\{=\}7matches segments \(3\)\-\(6\) above \(scaled position, direction, speed, distance to agent, tiredness\)\. Observability mask:Mj​\[g,ℓ\]=1M^\{j\}\[g,\\ell\]=1iff the player at slotℓ\\ellof groupggis within senderjj’s sensing radius and the slot is non\-padding\.

#### A\.3\.3Detailed Results

Table[9](https://arxiv.org/html/2606.29126#A1.T9)extends the GRF column of Table[1](https://arxiv.org/html/2606.29126#S5.T1)by reporting the underlying mean win rate \(%\), the GRF community convention metric, across the same IPPO and MAPPO grid onacademy\_counterattack\_hard, after the55M\-step training budget specified in Sec\.[A\.3](https://arxiv.org/html/2606.29126#A1.SS3)above\.

Table 9:GRFacademy\_counterattack\_hard\(44v33\) after55M steps; mean win rate \(%\) over3232evaluation episodes\.AlgorithmObservationWin rateIPPOFullObs5959IPPOPartialObs5050IPPOCACOM5353IPPOT2MAC5353IPPOHiComm\(ours\)56¯\\underline\{56\}MAPPOFullObs𝟔𝟗\\mathbf\{69\}MAPPOPartialObs5959MAPPOCACOM6363MAPPOT2MAC66¯\\underline\{66\}MAPPOHiComm\(ours\)66¯\\underline\{66\}

### A\.4Ablation Studies

We run a single ablation grid on CC4 against the same fixed pre\-trained blue opponent and under the same training budget as the main results in Table[1](https://arxiv.org/html/2606.29126#S5.T1)\. Every variant differs fromHiCommalong exactly one axis, holding the rest of the protocol \(state encoder, hierarchy\|𝒢\|=9\|\\mathcal\{G\}\|\{=\}9subnets→\|ℰ\|=16\\to\|\\mathcal\{E\}\|\{=\}16hosts→\|ℱ\|=5\\to\|\\mathcal\{F\}\|\{=\}5features, per\-agent observability maskMiM^\{i\}as defined in Sec\.[A\.1](https://arxiv.org/html/2606.29126#A1.SS1)together with the Phase 3 entity\-stage maskMj∗\(ent\)​\(g∗\)M^\{\(\\text\{ent\}\)\}\_\{j^\{\*\}\}\(g^\{\*\}\)of Sec\.[4\.2](https://arxiv.org/html/2606.29126#S4.SS2), unicastk=1k\{=\}1, PPO hyper\-parameters, blue checkpoint\) exactly as in Section[5\.1](https://arxiv.org/html/2606.29126#S5.SS1.SSS0.Px3)\. The variants partition into three axis groups \(*payload form*,*retrieval factorisation*,*gradient pathway*\) so that the marginal contribution of each design choice can be read off the resulting table\. Each variant is trained on both IPPO and MAPPO and evaluated under the same protocol as Table[1](https://arxiv.org/html/2606.29126#S5.T1)\.

##### Module naming\.

Throughout this section we use the implementation namesQueryNet,CommScheduler,DecodeNet, andhost\_selectorfor the fourHiCommsub\-modules of Sec\.[4\.2](https://arxiv.org/html/2606.29126#S4.SS2)\. Concretely,QueryNetproduces the receiver queryqi=tanh⁡\(fq​\(hi\)\)q\_\{i\}=\\tanh\(f\_\{\\text\{q\}\}\(h\_\{i\}\)\)\(Phase 1\);CommSchedulerproduces the Stage 2 sender\-selection logitssi​j∣g∗=score​\(qi,Φ​\(j,g∗;hi\)\)s\_\{ij\\mid g^\{\*\}\}=\\mathrm\{score\}\(q\_\{i\},\\Phi\(j,g^\{\*\};h\_\{i\}\)\);DecodeNetjointly hosts the Stage 1 group keys\{κg\}g=1\|𝒢\|\\\{\\kappa\_\{g\}\\\}\_\{g=1\}^\{\|\\mathcal\{G\}\|\}\(used at the receiver to pickg∗g^\{\*\}\) and the Phase 3 entity projectionψ​\(⋅\)\\psi\(\\cdot\)\(used at senderj∗j^\{\*\}to pickℓ∗\\ell^\{\*\}\); andhost\_selectoris the parameter\-free gatherzj∗​\[g∗,ℓ∗,:\]z^\{j^\{\*\}\}\[g^\{\*\},\\ell^\{\*\},:\]that returns the addressed feature vector\. Becauseκg\\kappa\_\{g\}andψ\\psilive inDecodeNet, detaching “DecodeNet’s subnet/host Gumbel logits” below means detaching both the Stage 1 receiver\-side group logits and the Phase 3 sender\-side entity logits\.

##### Payload form \(observation entry vs\. encoded\)\.

Two variants probe the claim of Section[4\.2](https://arxiv.org/html/2606.29126#S4.SS2)that the message should be the entry of the sender’s observation tensor at the retrieved coordinate, by replacing it with reconstruction\-style encoders\.HiComm\-encodedreplaces the raw\|ℱ\|=5\|\\mathcal\{F\}\|\{=\}5\-dim feature\-vector lookup at the queried coordinate with a learned message encoder over the sender’s full host matrix \(autoencoder\-style, comparable in capacity toCACOM’s message head\), so the receiver consumes a compressed summary rather than the feature vector itself; everything else is identical toHiComm\.HiComm\-encoded\-cellkeeps the two\-stage hierarchical retrieval and unicast scheduler untouched but appends a smallLinear\(5→85\{\\to\}8\)→\\toReLU→\\toLinear\(8→58\{\\to\}5\)MLP encoder afterhost\_selector’s55\-dim raw cell output, so the receiver consumes an encoded version of the same selected cell rather than the cell itself\.

##### Retrieval factorisation \(hierarchical vs\. flat\)\.

A single variant probes the contribution of the two\-stage subnet→\\tohost decode that givesHiCommits name\.HiComm\-Flatcollapses the receiver\-side Stage 1 group pick \(against\{κg\}g=1\|𝒢\|\\\{\\kappa\_\{g\}\\\}\_\{g=1\}^\{\|\\mathcal\{G\}\|\}, Sec\.[4\.2](https://arxiv.org/html/2606.29126#S4.SS2)\) and the sender\-side Phase 3 entity pick \(againstψ​\(zj∗​\[g,ℓ,:\]\)\\psi\(z^\{j^\{\*\}\}\[g,\\ell,:\]\)\) into a single144144\-way Gumbel\-Softmax over the flattened9×16=1449\{\\times\}16=144\(subnet, host\) coordinate set, executed at senderj∗j^\{\*\}oncej∗j^\{\*\}has been selected by Stage 2; the receiver no longer transmits a topic groupg∗g^\{\*\}and the sender alone produces the joint coordinate\(g∗,ℓ∗\)\(g^\{\*\},\\ell^\{\*\}\)\. Phase 1 query, Phase 2 Stage 2 sender selection, the unicast bandwidth budget, and the raw\-cell payload are all unchanged, so the only quantity that changes is whether the \(group, entity\) retrieval is factored hierarchically across the receiver/sender boundary or modelled jointly at the sender\.

##### Gradient pathway \(which modules receive PPO task\-reward gradient\)\.

The remaining variants stress\-test the task\-gradient pathway emphasised in Section[4\.2](https://arxiv.org/html/2606.29126#S4.SS2)by selectively detaching eachHiCommmodule’s output before the actor head, during the differentiable forward\. Recall that the actor objective in Sec\.[4\.2](https://arxiv.org/html/2606.29126#S4.SS2)isℒactor=ℒPPO−cH​ℋ\+λig​\(t\)​ℒig\\mathcal\{L\}\_\{\\text\{actor\}\}=\\mathcal\{L\}^\{\\text\{PPO\}\}\-c\_\{H\}\\mathcal\{H\}\+\\lambda\_\{\\text\{ig\}\}\(t\)\\,\\mathcal\{L\}\_\{\\text\{ig\}\}, whereℒig\\mathcal\{L\}\_\{\\text\{ig\}\}is the information\-gain shaping prior cross\-entropying the Stage 1, Stage 2, and Phase 3 pre\-ST\-GS logits against soft labels \(plus a small batch\-marginal entropy term that discourages sender collapse\)\. For the ablations only, we expose its three constituent terms separately so that the contribution of each pathway is readable:ℒdecode\\mathcal\{L\}\_\{\\text\{decode\}\}is the cross\-entropy on the Stage 1 subnet and Phase 3 host logits;ℒtarget\\mathcal\{L\}\_\{\\text\{target\}\}is the cross\-entropy on the Stage 2 sender logits;ℒexplore\\mathcal\{L\}\_\{\\text\{explore\}\}is the batch\-marginal entropy on sender selection\. In the baseHiCommall three run together asℒig\\mathcal\{L\}\_\{\\text\{ig\}\}alongside the PPO surrogate; the experiment isolates, for each module, how much of its learning is carried by the task\-reward gradient once the corresponding sub\-prior becomes the sole signal\.

HiComm\-NaiveComm\.detach\(\)s the outputs ofCommScheduler,DecodeNet, andQueryNetsimultaneously insideHiCommR\_Actor\.forward, so all three communication modules are trained only by the \(decomposed\) info\-gain prior and the policy receives the resulting messages without back\-propagating into the protocol\.HiComm\-NoCS\.detach\(\)s onlyCommScheduler’s output, leavingℒtarget\+ℒexplore\\mathcal\{L\}\_\{\\text\{target\}\}\+\\mathcal\{L\}\_\{\\text\{explore\}\}as the sole training signals for target selection while the query and decode paths still receive PPO gradient\.HiComm\-NoDec\.detach\(\)sDecodeNet’s subnet/host Gumbel logits \(i\.e\. both Stage 1 and Phase 3\) and trains the two\-stage retrieval purely fromℒdecode\\mathcal\{L\}\_\{\\text\{decode\}\}\.HiComm\-NoQuery\.detach\(\)sQueryNet’s outputqq; since none of the three sub\-priors is attached toQueryNetdirectly, this leaves it without any training signal and the query becomes a fixed random projection of the receiver’s hidden state\.

#### A\.4\.1Results

Table[10](https://arxiv.org/html/2606.29126#A1.T10)reports the ablation grid on CC4 over100100evaluation episodes against the same fixed pre\-trained blue opponent \(IPPO blue policy\) used in Table[4](https://arxiv.org/html/2606.29126#A1.T4)\. Each variant is trained independently under both IPPO and MAPPO with the training budget and hyper\-parameters of Section[A\.1](https://arxiv.org/html/2606.29126#A1.SS1); only the axis listed in the variant description differs from theHiCommreference\. We report mean episode reward, together with the number ofImpactattempts and successes summed across the100100episodes\.HiComm\(top row of each block\) is the reference configuration; the ordering inside each block follows the*payload→\\toretrieval→\\togradient*axis grouping of the variant descriptions above\.

Table 10:CC4 ablation grid over100100evaluation episodes against the fixed IPPO blue opponent \(red\-reward convention; higher is better\)\. Each variant differs fromHiCommalong exactly one axis; all other components, training budget, and PPO hyper\-parameters are held fixed\. \#Impact attempts/successes are summed over the100100episodes\.Bold==best reward within each algorithm block;underline==best Success \#Impact within each algorithm block\.AlgorithmVariantRewardTotal \#ImpactSuccess \#ImpactIPPOHiComm\(ref\.\)𝟏,𝟓𝟐𝟕\\mathbf\{1\{,\}527\}49,49949\{,\}499262¯\\underline\{262\}IPPOHiComm\-encoded96196141,08841\{,\}088152152IPPOHiComm\-encoded\-cell84484441,60741\{,\}607209209IPPOHiComm\-Flat1,3811\{,\}38139,27239\{,\}272243243IPPOHiComm\-NaiveComm1,3311\{,\}33139,32839\{,\}328198198IPPOHiComm\-NoCS1,4811\{,\}48146,77746\{,\}777245245IPPOHiComm\-NoDec1,3071\{,\}30748,45348\{,\}453216216IPPOHiComm\-NoQuery1,4221\{,\}42248,03248\{,\}032210210MAPPOHiComm\(ref\.\)𝟏,𝟗𝟒𝟏\\mathbf\{1\{,\}941\}78,21678\{,\}216246¯\\underline\{246\}MAPPOHiComm\-encoded1,1091\{,\}10940,99340\{,\}993197197MAPPOHiComm\-encoded\-cell1,5441\{,\}54442,45042\{,\}450214214MAPPOHiComm\-Flat69069041,10941\{,\}109141141MAPPOHiComm\-NaiveComm86486438,51538\{,\}515198198MAPPOHiComm\-NoCS1,6451\{,\}64532,72432\{,\}724223223MAPPOHiComm\-NoDec1,0831\{,\}08347,95647\{,\}956170170MAPPOHiComm\-NoQuery61861840,26840\{,\}268138138Across both algorithm blocks,HiComm\(ref\.\) is the unique row that is strictly best on both reward and Success \#Impact, and every single\-axis variant loses on both metrics simultaneously\. The Total \#Impact column further shows that the reward gap is not driven by attempt count: variants that lower Total \#Impact still lose Success \#Impact, so the regression reflects worse strategic choices rather than fewer attempts\. We read the three axis groups in turn\.

##### Payload form\.

Both encoded variants degrade on both algorithms, and Success \#Impact tracks the reward drop\. The cell\-level encoder is the milder of the two but still costly, even though it is only a small MLP appended tohost\_selector’s raw cell output: a learned bottleneck whose reconstruction objective is decoupled from policy reward is enough to disrupt the raw lookup\. The full\-host autoencoder ofHiComm\-encodedis worse still, consistent with the encoder being dominated by the mostly\-zero entries of an unexplored host matrix in CC4\. This confirms the design choice of Section[4\.2](https://arxiv.org/html/2606.29126#S4.SS2): with the receiver issuing a coordinate\-level query, the natural payload is the raw observation entry at that coordinate, and inserting any encoder degrades the protocol\.

##### Retrieval factorisation\.

HiComm\-Flatreplaces the two\-stage subnet→\\tohost Gumbel\-Softmax decode with a single Gumbel\-Softmax over the flattened entity set\. Reward drops modestly under IPPO but*collapses*under MAPPO, the largest single\-axis regression in the table, with Success \#Impact also dropping sharply\. The hierarchical decomposition is therefore not a parametric shortcut: it injects CC4’s native subnet⊃\\supsethost containment as an inductive bias that turns retrieval into two short categoricals, and the centralised MAPPO critic does not recover the lost optimisation tractability on its own\.

##### Gradient pathway\.

Detaching all three communication modules simultaneously \(HiComm\-NaiveComm, only the decomposed info\-gain prior remains\) hurts both algorithms, and substantially more under MAPPO: the prior partially compensates under IPPO but cannot recover the task\-gradient signal that carries credit assignment under a centralised critic\. The per\-module breakdown isolates which of the three modules is most reliant on PPO gradient\.HiComm\-NoQuery\(no signal at all, since none ofℒdecode\\mathcal\{L\}\_\{\\text\{decode\}\},ℒtarget\\mathcal\{L\}\_\{\\text\{target\}\},ℒexplore\\mathcal\{L\}\_\{\\text\{explore\}\}attaches toQueryNet\) suffers the most catastrophic regression on MAPPO: without any signal, the query collapses to a fixed random projection of the receiver hidden state and neither the retrieved coordinate nor the consumed payload can be aligned with what the policy actually needs\.HiComm\-NoDec\(onlyℒdecode\\mathcal\{L\}\_\{\\text\{decode\}\}, info\-gain over\(g,ℓ\)\(g,\\ell\)\) is also clearly worse than the reference, showing that an info\-gain prior is a useful but insufficient substitute for task gradient through the two\-stage retrieval, since informative entries are not always reward\-relevant\.HiComm\-NoCS\(onlyℒtarget\+ℒexplore\\mathcal\{L\}\_\{\\text\{target\}\}\+\\mathcal\{L\}\_\{\\text\{explore\}\}\) is the mildest perturbation: the per\-target novelty proxy is enough to keep target selection close to the reference, indicating that*which*sender to query is the easiest of the three sub\-problems to learn from the prior alone\. Together, the gradient\-pathway block establishes that the task\-reward gradient is essential precisely where the prior is least informative \(QueryNet, thenDecodeNet\), and motivates running the fullHiCommstack inside the actor’s differentiable forward as in Section[4\.2](https://arxiv.org/html/2606.29126#S4.SS2)\.

##### Takeaway\.

Across the results, we found that removing the raw observation\-entry payload, the hierarchical retrieval factorisation, or the task\-reward gradient pathway each induces a distinct failure mode, and no alternative training signal we tested recovers the lost performance\. Across all six axis perturbations and both PPO surrogates,HiComm\(ref\.\) is strictly best on reward and Success \#Impact, supporting the claim that each of the three design choices in Section[4\.2](https://arxiv.org/html/2606.29126#S4.SS2)is necessary for the protocol as a whole\.