When to Communicate: Belief Distributions and KL Divergence for Principled Gating in Multi-Agent RL

# Belief Distributions and KL Divergence for Principled Gating in Multi-Agent RL
Source: [https://arxiv.org/html/2608.14559](https://arxiv.org/html/2608.14559)
###### Abstract

Effective communication in multi\-agent reinforcement learning requires agents to decide not onlywhatto communicate, but when? Existing approaches either communicate at every timestep or learn a binary gate through REINFORCE policy gradients\[[2](https://arxiv.org/html/2608.14559#bib.bib2)\], a high\-variance signal that produces unstable and uninterpretable gating behavior\. I propose a principled alternative: agents communicate only when the KL divergence between their learned belief distributions exceeds a fixed threshold\. Each agent maintains a belief distribution over a latent world state computed as a softmax over its LSTM hidden state, and communicates only when belief disagreement is large enough to justify information exchange\. I evaluate this approach on the Predator\-Prey benchmark from IC3Net\[[2](https://arxiv.org/html/2608.14559#bib.bib2)\]across two environment sizes with 5 seeds each, and on MPE simple\_spread\[[6](https://arxiv.org/html/2608.14559#bib.bib6)\], comparing against IC3Net, CommNet, and an independent controller\. On PP 10×\\times10, IC3Net outperforms KL\-belief at all thresholds\. On the harder PP 20×\\times20, a threshold ablation overε∈\{0\.1,0\.3,0\.5,1\.0\}\\varepsilon\\in\\\{0\.1,0\.3,0\.5,1\.0\\\}reveals an inverted U\-shape:ε=0\.5\\varepsilon=0\.5achieves 73\.84 average steps and 42% success rate versus IC3Net’s 75\.31 steps and 31%, a gap of 1\.47 steps and 11 percentage points with tighter seed variance\. On MPE, the belief head improves mean reward by 12 points and reduces variance by 26×\\timeseven when gating is inactive, suggesting two orthogonal contributions: principled gating when beliefs can converge, and improved latent representations that benefit coordination regardless\.

## 1Introduction

In many real\-world multi\-agent problems, agents cannot observe the full state of the environment\. Each agent sees only a limited, local portion of the world, and must coordinate with other agents to complete a shared task\. Communication is a natural solution to this problem by sharing information, agents can build a better picture of the environment together than they could alone\. But communication has a cost\. In large systems, broadcasting information at every timestep is expensive and often unnecessary\. More importantly, in mixed or competitive settings, sharing information at the wrong moment can actually hurt performance\. A prey agent, for example, should not broadcast its location to predators\. This raises a fundamental question:when should an agent communicate?

Early work on multi\-agent communication, such as CommNet\[[1](https://arxiv.org/html/2608.14559#bib.bib1)\], took a simple approach: agents always communicate by sharing their hidden states through a continuous channel\. This works well in fully cooperative settings, but it has two important limitations\. First, it assumes all agents share the same global reward, which makes credit assignment difficult as the number of agents grows\. Second, because every agent always communicates with every other agent, the model cannot be used in competitive or mixed scenarios where agents have conflicting interests\.

IC3Net\[[2](https://arxiv.org/html/2608.14559#bib.bib2)\]addressed these limitations by introducing a learned binary gate trained with REINFORCE, allowing it to work across cooperative, competitive, and mixed settings\. However, the gate is a black box trained purely through reward signals with no explicit notion ofwhycommunication should happen at a given moment\. As a result, communication frequency fluctuates wildly during training, a direct consequence of REINFORCE’s high variance that makes IC3Net’s gating behavior difficult to interpret or trust\.

I argue that the decision to communicate should be grounded in something more principled: the degree to which agents disagree about the state of the world\. If two agents have observed the same things and formed similar beliefs, there is little new information to exchange\. But if their beliefs diverge significantly for example, one agent has located the prey while others are still searching communication becomes genuinely valuable\. This is not just an intuition\. From the perspective of POMDP theory, the belief state is the sufficient statistic for decision\-making under uncertainty\[[3](https://arxiv.org/html/2608.14559#bib.bib3)\]\. Grounding the communication trigger in belief disagreement is therefore a principled choice with a clear theoretical motivation\.

In this paper, I propose a simple modification to IC3Net’s gating mechanism\. Each agent computes a belief distribution over a latent world state by passing its LSTM hidden state through a small linear layer followed by a softmax, producing aKK\-dimensional probability vector:

bti=Softmax​\(fϕ​\(hti\)\)∈ℝK\.b^\{i\}\_\{t\}=\\text\{Softmax\}\(f\_\{\\phi\}\(h^\{i\}\_\{t\}\)\)\\in\\mathbb\{R\}^\{K\}\.\(1\)Agentiithen communicates with agentjjonly when the KL divergence between their belief distributions at the previous timestep exceeds a thresholdϵ\\epsilon:

gti​j=𝟏​\[DKL​\(bt−1i∥bt−1j\)\>ϵ\]\.g^\{ij\}\_\{t\}=\\mathbf\{1\}\\\!\\left\[D\_\{\\mathrm\{KL\}\}\\\!\\left\(b^\{i\}\_\{t\-1\}\\,\\\|\\,b^\{j\}\_\{t\-1\}\\right\)\>\\epsilon\\right\]\.\(2\)I use the previous timestep’s beliefs to avoid a circularity problem: to decide whether to communicate with agentjj, I would otherwise need agentjj’s current belief, which has not yet been received\. Usingbt−1b\_\{t\-1\}breaks this dependency cleanly\. This simple change replaces IC3Net’s black\-box REINFORCE gate with a principled, interpretable trigger that is directly tied to uncertainty disagreement between agents\.

I evaluate this approach on the Predator\-Prey benchmark from the IC3Net paper\[[2](https://arxiv.org/html/2608.14559#bib.bib2)\]across two environment sizes with 5 seeds each, and on the MPE simple\_spread cooperative benchmark, comparing against IC3Net, CommNet, and an independent controller\. My experiments reveal two consistent findings\. First, on PP 10×\\times10, IC3Net outperforms KL\-belief at all thresholds, suggesting the principled trigger requires sufficient partial observability to be effective\. Second, on PP 20×\\times20, a threshold ablation overε∈\{0\.1,0\.3,0\.5,1\.0\}\\varepsilon\\in\\\{0\.1,0\.3,0\.5,1\.0\\\}reveals a clear inverted U\-shape: atε=0\.5\\varepsilon=0\.5, KL\-belief achieves 73\.84 average steps and 42% success rate versus IC3Net’s 75\.31 steps and 31%, a gap of 1\.47 steps and 11 percentage points, with tighter variance across seeds \(±\\pm1\.20 versus±\\pm1\.82\)\. On MPE, the belief head provides representational benefits even when the gate does not selectively activate, improving mean reward by 12 points over IC3Net at properly calibrated thresholds\.

## 2Background and Related Work

### 2\.1Partially Observable Multi\-Agent Settings

In many real\-world coordination tasks, agents cannot observe the full state of the environment\. This is formally modelled as a Decentralised Partially Observable Markov Decision Process \(Dec\-POMDP\), where a set ofnnagents share a hidden statest∈𝒮s\_\{t\}\\in\\mathcal\{S\}\. At each timestep, agentiireceives a local observationotio^\{i\}\_\{t\}that depends on the true state but does not fully reveal it\. Each agent selects an actionatia^\{i\}\_\{t\}according to a decentralised policyπi​\(ati∣hti\)\\pi^\{i\}\(a^\{i\}\_\{t\}\\mid h^\{i\}\_\{t\}\), wherehtih^\{i\}\_\{t\}summarises its local observation history\. The environment transitions to the next state according toT​\(st\+1∣st,𝐚t\)T\(s\_\{t\+1\}\\mid s\_\{t\},\\mathbf\{a\}\_\{t\}\)and agents receive a reward signalrt=R​\(st,𝐚t\)r\_\{t\}=R\(s\_\{t\},\\mathbf\{a\}\_\{t\}\)\.

A key insight from POMDP theory is that thebelief statethe probability distribution over the hidden state given an agent’s observation history is the sufficient statistic for optimal decision\-making under uncertainty\[[3](https://arxiv.org/html/2608.14559#bib.bib3)\]\. In single\-agent settings, maintaining and acting on this belief state is well understood\. In multi\-agent settings, however, agents havedifferentbelief states because they observe different parts of the environment\. This difference in beliefs is precisely what makes communication valuable: agents can share what they know to resolve each other’s uncertainty\.

### 2\.2Learning to Communicate

CommNet\[[1](https://arxiv.org/html/2608.14559#bib.bib1)\]was one of the first deep learning approaches to learn communication in multi\-agent systems\. In CommNet, each agentjjmaintains a hidden statehjth^\{t\}\_\{j\}updated by an LSTM\. At each timestep, agents broadcast their hidden states to all others, and each agent receives the average of its teammates’ hidden states as a communication vector:

cjt=1J−1​∑j′≠jhj′t\.c^\{t\}\_\{j\}=\\frac\{1\}\{J\-1\}\\sum\_\{j^\{\\prime\}\\neq j\}h^\{t\}\_\{j^\{\\prime\}\}\.\(3\)This communication is continuous and differentiable, so CommNet can be trained end\-to\-end with backpropagation\. It works well in fully cooperative tasks, but has two important limitations\. First, it uses a global average reward for all agents, which makes it hard for individual agents to understand their own contribution to the team’s performance\. Second, because agents always communicate with everyone, CommNet cannot be used in competitive or mixed settings where sharing information may hurt an agent’s own performance\.

IC3Net\[[2](https://arxiv.org/html/2608.14559#bib.bib2)\]addressed both of these problems\. It gives each agent its own individual reward, which helps with credit assignment and allows the model to work in cooperative, competitive, and mixed settings\. More importantly for this paper, IC3Net adds a learned binary communication gate\. At each timestep, agentjjdecides whether to communicate using a small gating networkfgf^\{g\}trained with REINFORCE\[[5](https://arxiv.org/html/2608.14559#bib.bib5)\]:

gjt=fg​\(hjt\)∈\{0,1\}\.g^\{t\}\_\{j\}=f^\{g\}\(h^\{t\}\_\{j\}\)\\in\\\{0,1\\\}\.\(4\)The communication vector becomes the gated average of active agents’ hidden states:

cjt=1J−1​∑j′≠jhj′t⋅gj′t\.c^\{t\}\_\{j\}=\\frac\{1\}\{J\-1\}\\sum\_\{j^\{\\prime\}\\neq j\}h^\{t\}\_\{j^\{\\prime\}\}\\cdot g^\{t\}\_\{j^\{\\prime\}\}\.\(5\)IC3Net achieves strong performance on standard benchmarks and empirically learns to communicate only when it is profitable for example, predators learn to stop communicating once they reach prey in a competitive setting\. However, the gating mechanism is a black box: it is trained purely through reward signals with no explicit notion of uncertainty or belief disagreement\. As I show in this paper, this leads to unstable and erratic communication patterns during training\.

### 2\.3Related Work on Targeted and Structured Communication

Several works have tried to make multi\-agent communication more structured and selective\. TarMAC\[[8](https://arxiv.org/html/2608.14559#bib.bib8)\]uses an attention mechanism to allow agents to send targeted messages to specific teammates rather than broadcasting to everyone\. This makes communication more efficient, but the decision of what to communicate is still based on learned attention weights rather than a principled measure of uncertainty\.

DIAL\[[4](https://arxiv.org/html/2608.14559#bib.bib4)\]takes a different approach, allowing agents to communicate discrete symbols learned through a differentiable relaxation\. This makes the communication protocol more interpretable in terms of what is being said, but does not address the question ofwhento communicate\.

MADDPG\[[6](https://arxiv.org/html/2608.14559#bib.bib6)\]and QMIX\[[7](https://arxiv.org/html/2608.14559#bib.bib7)\]focus on centralised training with decentralised execution, but neither explicitly models communication gating\. These methods assume agents can implicitly coordinate through their trained policies without explicit message passing\.

Most recently, Hill et al\.\[[11](https://arxiv.org/html/2608.14559#bib.bib11)\]propose communicating predicted future plans rather than raw observations, using a learned world model to generate messages\. Their finding that engineered inductive biases outperform purely learned end\-to\-end communication as environment complexity increases directly supports the motivation for our work\. Where they addresswhatto communicate via world model predictions, we addresswhento communicate via belief disagreement\.

To my knowledge, no prior work has replaced IC3Net’s REINFORCE\-trained gate with a trigger based on KL divergence between learned belief distributions\. The closest related idea is the use of belief representations in single\-agent POMDPs, where the belief state directly informs decision\-making\. My work bridges this idea to the multi\-agent communication setting by using belief disagreement as a principled signal forwhento communicate\.

## 3Methodology

### 3\.1Base Architecture

My method builds directly on top of IC3Net\[[2](https://arxiv.org/html/2608.14559#bib.bib2)\], keeping its core architecture unchanged and replacing only the communication gate\. I describe the full architecture here for completeness\.

Each agentjjmaintains a hidden statehjth^\{t\}\_\{j\}and cell statesjts^\{t\}\_\{j\}using an LSTM\[[9](https://arxiv.org/html/2608.14559#bib.bib9)\]\. At each timestep, the agent receives its local observationojto^\{t\}\_\{j\}, encodes it through a linear encodere​\(⋅\)e\(\\cdot\), and updates its hidden state:

hjt\+1,sjt\+1=LSTM​\(e​\(ojt\)\+cjt,hjt,sjt\),h^\{t\+1\}\_\{j\},s^\{t\+1\}\_\{j\}=\\text\{LSTM\}\\\!\\left\(e\(o^\{t\}\_\{j\}\)\+c^\{t\}\_\{j\},\\ h^\{t\}\_\{j\},\\ s^\{t\}\_\{j\}\\right\),\(6\)wherecjtc^\{t\}\_\{j\}is the communication vector received from other agents\. The action policy and value head are both computed directly from the hidden statehjth^\{t\}\_\{j\}\. All agents share the same network parameters, which makes the model invariant to the ordering of agents and allows it to scale to different numbers of agents without retraining\.

The communication vectorcjtc^\{t\}\_\{j\}is computed as the gated average of other agents’ hidden states:

cjt=1J−1⋅C​∑j′≠jhj′t⋅gj′t,c^\{t\}\_\{j\}=\\frac\{1\}\{J\-1\}\\cdot C\\sum\_\{j^\{\\prime\}\\neq j\}h^\{t\}\_\{j^\{\\prime\}\}\\cdot g^\{t\}\_\{j^\{\\prime\}\},\(7\)whereCCis a learned linear transformation,JJis the number of alive agents, andgj′t∈\{0,1\}g^\{t\}\_\{j^\{\\prime\}\}\\in\\\{0,1\\\}is the communication gate for agentj′j^\{\\prime\}\. In the original IC3Net, this gate is trained by REINFORCE\. In my method, I replace it with a principled KL\-divergence trigger, described next\.

Each agent is trained with its own individual reward rather than a shared global reward, an important design choice from IC3Net that enables the model to work in mixed and competitive settings where global rewards fail\.

### 3\.2Belief Head

The key addition in my method is a belief head a small network that sits on top of each agent’s LSTM hidden state and produces a probability distribution overKKlatent world state categories\. I call this the agent’sbeliefabout the current state of the environment\.

Formally, the belief of agentiiat timestepttis:

bti=Softmax​\(fϕ​\(hti\)\)∈ℝK,∑kbti​\(k\)=1,b^\{i\}\_\{t\}=\\text\{Softmax\}\(f\_\{\\phi\}\(h^\{i\}\_\{t\}\)\)\\in\\mathbb\{R\}^\{K\},\\quad\\sum\_\{k\}b^\{i\}\_\{t\}\(k\)=1,\(8\)wherefϕf\_\{\\phi\}is a single linear layer with parametersϕ\\phi, and the softmax ensures the output is a valid probability distribution\. I useK=8K=8categories in all experiments, which I found to be sufficient to capture meaningful belief differences between agents without adding too many parameters\.

The belief head is trained end\-to\-end with the rest of the network\. It receives gradients through the main policy loss there is no separate loss for the belief head\. This means the belief representations are shaped entirely by what is useful for the agent’s policy, not by any explicit supervision about the world state\. In practice, I found that this is enough for beliefs to become meaningfully different across agents in partially observable environments, which is exactly what the KL gate needs to work\.

### 3\.3KL\-Divergence Communication Gate

Instead of training a gate network with REINFORCE, I compute the communication decision directly from the pairwise KL divergence between agents’ belief distributions\. Agentiicommunicates with agentjjat timestepttif and only if the KL divergence between their beliefs exceeds a fixed thresholdϵ\\epsilon:

gti​j=𝟏​\[DKL​\(bt−1i∥bt−1j\)\>ϵ\],g^\{ij\}\_\{t\}=\\mathbf\{1\}\\\!\\left\[D\_\{\\mathrm\{KL\}\}\\\!\\left\(b^\{i\}\_\{t\-1\}\\,\\\|\\,b^\{j\}\_\{t\-1\}\\right\)\>\\epsilon\\right\],\(9\)where:

DKL​\(bi∥bj\)=∑k=1Kbi​\(k\)​log⁡bi​\(k\)bj​\(k\)\.D\_\{\\mathrm\{KL\}\}\(b^\{i\}\\\|b^\{j\}\)=\\sum\_\{k=1\}^\{K\}b^\{i\}\(k\)\\log\\frac\{b^\{i\}\(k\)\}\{b^\{j\}\(k\)\}\.\(10\)A small constantδ=10−8\\delta=10^\{\-8\}is added to both distributions before computing the logarithm to ensure numerical stability when belief probabilities approach zero, giving the implemented form:

DKL​\(bi∥bj\)=∑k=1Kbi​\(k\)​\(log⁡\(bi​\(k\)\+δ\)−log⁡\(bj​\(k\)\+δ\)\)\.D\_\{\\mathrm\{KL\}\}\(b^\{i\}\\\|b^\{j\}\)=\\sum\_\{k=1\}^\{K\}b^\{i\}\(k\)\\left\(\\log\(b^\{i\}\(k\)\+\\delta\)\-\\log\(b^\{j\}\(k\)\+\\delta\)\\right\)\.\(11\)I use beliefs from theprevioustimestept−1t\-1rather than the current timesteptt\. This is necessary to avoid a circularity problem: to decide whether to communicate with agentjjat timesteptt, agentiiwould require agentjj’s current belief, which has not yet been received\. Usingbt−1b\_\{t\-1\}resolves this dependency cleanly, as the gate decision relies solely on information available to all agents before any communication occurs\. At the start of each episode, the previous beliefs are initialised to a uniform distribution over allKKcategories:

b0i=\(1K,1K,…,1K\)\.b^\{i\}\_\{0\}=\\left\(\\frac\{1\}\{K\},\\frac\{1\}\{K\},\\ldots,\\frac\{1\}\{K\}\\right\)\.\(12\)This means all agents communicate with each other at the first timestep of every episode, since uniform distributions have zero KL divergence\. As the episode progresses and agents accumulate different observations, their beliefs diverge and the gate starts selectively blocking communication\.

### 3\.4Training

The full model is trained with REINFORCE\[[5](https://arxiv.org/html/2608.14559#bib.bib5)\]using individual rewards for each agent, exactly as in IC3Net\. The belief head parametersϕ\\phiare updated through the same policy gradient with no separate training objective\. The communication gate is not trained at all: it is a deterministic function of the beliefs and the thresholdϵ\\epsilon, producing no gradient\. Only the policy is trained with REINFORCE; the gate follows automatically from the beliefs, making the system easier to train and more stable than IC3Net\.

## 4Experiments

### 4\.1Environment

I evaluate my method on the Predator\-Prey \(PP\) environment from the IC3Net benchmark\[[2](https://arxiv.org/html/2608.14559#bib.bib2)\]\. In this task,nnpredator agents with limited vision are placed randomly on a square grid and must cooperate to find a stationary prey\. Once a predator reaches the prey, it stays there and continues receiving a positive reward until the episode ends\. Each predator can take one of five movement actions at each timestep: up, down, left, right, or stay\. Agents have a vision radius of 1, meaning they can only observe the cells immediately surrounding their current position\. The episode ends either when all predators have reached the prey, or when the maximum number of steps is reached\.

I test on two difficulty levels\. The 10×\\times10 grid uses 5 agents and a maximum of 40 steps per episode\. The 20×\\times20 grid uses 10 agents and a maximum of 80 steps per episode\. Both settings use the mixed cooperation scenario from IC3Net, in which each agent receives an individual reward that does not depend on how many other agents have already reached the prey\. This mixed setting is more challenging than the cooperative setting because agents have no explicit incentive to help each other, and it is the setting where selective communication is most important\.

I additionally evaluate on the MPEsimple\_spreadcooperative task\[[6](https://arxiv.org/html/2608.14559#bib.bib6)\], in which 3 agents must cover 3 landmarks while minimising inter\-agent collisions\. Agents receive a shared penalty proportional to the minimum distance from each landmark to the nearest agent\. This environment tests whether KL\-belief generalises beyond Predator\-Prey to a second cooperative benchmark with different reward structure and observation space\.

### 4\.2Baselines

I compare against IC3Net\[[2](https://arxiv.org/html/2608.14559#bib.bib2)\], the primary baseline whose REINFORCE gate my method directly replaces; CommNet\[[1](https://arxiv.org/html/2608.14559#bib.bib1)\], which always communicates and serves as the always\-communicate baseline; and an Independent Controller \(IC\) with no communication, establishing the lower bound\. T2MAC\[[10](https://arxiv.org/html/2608.14559#bib.bib10)\], the most closely related prior work using uncertainty\-driven communication, is evaluated only on SMAC in its published codebase\. A direct comparison on shared benchmarks is left for future work pending codebase unification\.

### 4\.3Training Details

All models use an LSTM with hidden size 128, trained with REINFORCE using RMSProp with learning rate 0\.001\. All models are trained for 1000 epochs\. These settings match the original IC3Net paper\[[2](https://arxiv.org/html/2608.14559#bib.bib2)\]exactly\. For my KL\-belief method, I useK=8K=8belief categories\. PP experiments use 16 parallel processes with batch\_size=500, and are run with 5 random seeds\. MPE experiments use nprocesses=1 with epoch\_size=10 and batch\_size=500, running for 1000 epochs with 3 seeds for the ablation\. All experiments were run on Northeastern University’s Explorer HPC cluster using 16 CPU cores and 32GB RAM per job\. A belief size ablation overK∈\{4,8,16\}K\\in\\\{4,8,16\\\}is also conducted on PP 20×\\times20 withε=0\.5\\varepsilon=0\.5and 5 seeds each, using nprocesses=1\.

### 4\.4Evaluation Metrics

I report average steps per episode and success rate, both averaged over the final 100 training epochs\. For PP I additionally report communication rate standard deviation across all 1000 epochs as a measure of gate stability\. Lower steps and higher success rate are better; lower communication rate standard deviation indicates more consistent gating\.

### 4\.5Threshold Ablation

A key hyperparameter of my method is the KL divergence thresholdϵ\\epsilon, which controls how much belief disagreement is required before communication is triggered\. A threshold that is too low causes the gate to fire almost constantly, making the method behave similarly to CommNet\. A threshold that is too high causes agents to communicate too rarely, losing the coordination benefits of communication\. I run a systematic ablation overϵ∈\{0\.1,0\.3,0\.5,1\.0\}\\epsilon\\in\\\{0\.1,0\.3,0\.5,1\.0\\\}on PP 20×\\times20 with 5 seeds each, and overε∈\{0\.5,1\.0,2\.0,5\.0,10\.0\}\\varepsilon\\in\\\{0\.5,1\.0,2\.0,5\.0,10\.0\\\}on MPE simple\_spread with 3 seeds each\.

### 4\.6Belief Size Ablation

A secondary hyperparameter isKK, the number of belief categories\. Too few categories and the belief vector cannot capture meaningful differences between agents\. Too many and the representation becomes harder to learn consistently\. I run an ablation overK∈\{4,8,16\}K\\in\\\{4,8,16\\\}on PP 20×\\times20 withε=0\.5\\varepsilon=0\.5and 5 seeds each\.

## 5Results

### 5\.1Predator\-Prey 10×\\times10

Table[1](https://arxiv.org/html/2608.14559#S5.T1)shows results on the 10×\\times10 grid with 5 agents and 5 seeds\. IC3Net achieves the best task performance at 27\.22 average steps and 88\.6% success rate\. No KL\-belief threshold matches IC3Net on either metric, the closest isε=0\.3\\varepsilon=0\.3at 27\.26 steps and 84\.7% success\. CommNet and IC both fail to coordinate effectively, confirming that selective communication is necessary in the mixed setting\.

Table 1:Results on Predator\-Prey 10×\\times10, mixed setting, 5 seeds\. Mean±\\pmstd reported\. CommNet and IC use single seed\.
On the 10×\\times10 grid, IC3Net outperforms KL\-belief at every threshold\. Unlike the 20×\\times20 results, there is no inverted U\-shape, performance degrades monotonically asε\\varepsilonincreases\. This contrast between environments suggests that KL\-belief’s advantage depends on the degree of partial observability: with only 5 agents on a smaller grid, belief distributions do not diverge enough to provide a reliable communication signal, and the learned REINFORCE gate has an advantage\. As environment complexity grows, the principled KL trigger becomes more valuable\.

### 5\.2Predator\-Prey 20×\\times20

Table[2](https://arxiv.org/html/2608.14559#S5.T2)shows the baseline results on the 20×\\times20 grid with 10 agents, usingϵ=0\.1\\epsilon=0\.1and 5 seeds\. At this threshold, KL\-belief performs essentially the same as IC3Net, 75\.49 steps and 28% success versus IC3Net’s 75\.31 steps and 31% success\. CommNet and IC both fail completely, reaching 0% success rate across all seeds\.

Table 2:Results on Predator\-Prey 20×\\times20, mixed setting, nprocesses=16, 5 seeds\. Mean±\\pmstandard deviation reported\.
The poor performance of KL\-belief atϵ=0\.1\\epsilon=0\.1on the 20×\\times20 grid is not a failure of the method but a calibration issue\. With 10 agents spread across a larger grid, belief distributions are naturally more diverse because agents observe different parts of a larger space\. As a result, the KL divergence between any two agents’ beliefs exceedsϵ=0\.1\\epsilon=0\.1almost constantly, causing the gate to fire erratically, similar to always communicating\. This motivates the threshold ablation in the next subsection\.

We note that we were unable to reproduce the original IC3Net result of 52\.4 steps on PP 20×\\times20 reported by Singh et al\.\[[2](https://arxiv.org/html/2608.14559#bib.bib2)\]\. Our IC3Net baseline consistently achieved approximately 75 steps across all 5 seeds, which we attribute to unreported hyperparameter tuning in the original paper\. All baselines in this work use identical hyperparameters for fair comparison\.

### 5\.3Threshold Ablation

Table[3](https://arxiv.org/html/2608.14559#S5.T3)shows the results of theϵ\\epsilonablation with 5 seeds each\. The results reveal a clear inverted U\-shape: performance improves asϵ\\epsilonincreases from 0\.1 to 0\.5, then degrades atϵ=1\.0\\epsilon=1\.0\.

Table 3:Threshold ablation on Predator\-Prey 20×\\times20, mixed setting, nprocesses=16, 5 seeds\. Mean±\\pmstandard deviation reported\.
Atϵ=0\.1\\epsilon=0\.1, the threshold is too low and the gate fires almost constantly, producing worse results than IC3Net\. Asϵ\\epsilonincreases to 0\.3 and then 0\.5, the gate becomes more selective, only triggering communication when belief disagreement is genuinely large, and performance improves consistently\. Atϵ=0\.5\\epsilon=0\.5, KL\-belief achieves 73\.84 average steps and 42% success rate, beating IC3Net by 1\.47 steps and 11 percentage points in success rate\. Atϵ=1\.0\\epsilon=1\.0, the threshold is too high and agents communicate too rarely, causing performance to degrade back toward IC3Net level\. The variance across seeds also tightens at the sweet spot: atϵ=0\.5\\epsilon=0\.5, step variance is±\\pm1\.20 and success variance is±\\pm0\.04, compared to IC3Net’s±\\pm1\.82 and±\\pm0\.09\. This suggests that a well\-calibrated KL gate produces more consistent coordination behavior across different random initializations\.

Figure[1](https://arxiv.org/html/2608.14559#S5.F1)illustrates this contrast directly\. IC3Net’s communication rate fluctuates across the full training range, while KL\-Beliefε=0\.5\\varepsilon\{=\}0\.5converges to a stable band around 0\.38 after an initial transient period\. Figure[2](https://arxiv.org/html/2608.14559#S5.F2)shows the inverted U\-shape across allε\\varepsilonvalues\. Additional learning curves over training are shown in Appendix Figure[4](https://arxiv.org/html/2608.14559#A1.F4)\. Communication rates for allε\\varepsilonvalues over training are shown in Appendix Figure[5](https://arxiv.org/html/2608.14559#A1.F5)\.

![[Uncaptioned image]](https://arxiv.org/html/2608.14559v1/x1.png)

Figure 1:Communication rate over training on PP20×2020\{\\times\}20\(5 seeds, mean±\\pmstd\)\. KL\-Beliefε=0\.5\\varepsilon\{=\}0\.5settles at a stable rate \(≈0\.38\\approx 0\.38\) while IC3Net’s REINFORCE gate fluctuates throughout training\.
![[Uncaptioned image]](https://arxiv.org/html/2608.14559v1/plots/plot3_ablation_bar.png)

Figure 2:ε\\varepsilonablation on PP20×2020\{\\times\}20\(5 seeds\)\. Dashed line marks IC3Net’s mean\. KL\-Beliefε=0\.5\\varepsilon\{=\}0\.5achieves the best success rate \(0\.42±0\.040\.42\\pm 0\.04\) and lowest average steps \(73\.84±1\.2073\.84\\pm 1\.20\)\.

### 5\.4MPE Simple Spread

Table[4](https://arxiv.org/html/2608.14559#S5.T4)shows results on MPE simple\_spread with 3 agents\. Atε=0\.5\\varepsilon=0\.5, KL\-belief performs worse than IC3Net \(−237\.53\-237\.53versus−230\.06\-230\.06\), mirroring the PP 20×\\times20 pattern atε=0\.1\\varepsilon=0\.1: the threshold is too low and the gate fires constantly because agents in a continuous space always maintain highly divergent beliefs\. Asε\\varepsilonincreases to 1\.0 and above, KL\-belief consistently outperforms IC3Net by approximately 12 reward points with dramatically tighter variance \(±\\pm0\.32 versus±\\pm8\.49\)\.

Table 4:MPE simple\_spread results, 3 agents\. IC3Net and KLε\\varepsilon=0\.5 use 5 seeds; ablation values use 3 seeds\. Mean±\\pmstd reported\.
Notably, the communication rate remains 1\.0 at all thresholds on MPE, the gate never selectively suppresses communication\. Yet KL\-belief still outperforms IC3Net atε≥1\.0\\varepsilon\\geq 1\.0\. This suggests the belief head contributes two orthogonal benefits: principled gating when beliefs can converge \(as in PP 20×\\times20 atε=0\.5\\varepsilon=0\.5\), and improved latent state representations that benefit the policy even when gating is inactive\.

### 5\.5Belief Size Ablation

Table[5](https://arxiv.org/html/2608.14559#S5.T5)shows results forK∈\{4,8,16\}K\\in\\\{4,8,16\\\}on PP 20×\\times20 withε=0\.5\\varepsilon=0\.5and 5 seeds\.K=8K=8achieves the best performance and tightest variance across seeds\.K=4K=4lacks representational capacity, with high success variance \(±\\pm0\.082\)\.K=16K=16introduces instability: communication rate standard deviation of 0\.340 across seeds indicates some seeds communicate almost constantly while others rarely communicate\.K=8K=8balances expressiveness and learnability, confirming it as a principled choice rather than an arbitrary one\.

Table 5:Belief size ablation on PP 20×\\times20,ε=0\.5\\varepsilon=0\.5, 5 seeds\. Mean±\\pmstd reported\.
![[Uncaptioned image]](https://arxiv.org/html/2608.14559v1/x2.png)

Figure 3:Belief size ablation on PP20×2020\{\\times\}20\(5 seeds\)\.K=8K=8achieves the best success rate and lowest steps with tightest variance\.K=16K=16shows high communication rate instability across seeds\.

### 5\.6Summary

Across all three environments, a consistent pattern emerges\. On PP 10×\\times10, IC3Net outperforms KL\-belief at every threshold\. On PP 20×\\times20,ε=0\.5\\varepsilon=0\.5is the sweet spot, outperforming IC3Net by 1\.47 steps and 11 percentage points with tighter seed variance\. On MPE, the gate never selectively activates but the belief head still improves performance atε≥1\.0\\varepsilon\\geq 1\.0\. The belief size ablation confirmsK=8K=8as a principled choice, balancing expressiveness and learnability\. The unifying finding is that the optimal threshold scales with environment complexity and belief divergence: a threshold calibrated for one environment will be miscalibrated for another, directly motivating adaptive thresholding as future work\.

## 6Conclusion

In this paper, I proposed a principled communication gate for multi\-agent reinforcement learning that replaces IC3Net’s REINFORCE\-trained black\-box gate with a KL\-divergence trigger over learned belief distributions\. Each agent computes a belief distribution by passing its LSTM hidden state through a small linear layer followed by a softmax, and communicates with another agent only when the KL divergence between their beliefs exceeds a fixed threshold\. This simple modification requires no additional training signal beyond the existing policy gradient the belief head is trained end\-to\-end as part of the policy\.

My experiments across Predator\-Prey and MPE simple\_spread reveal two consistent findings\. First, KL\-belief’s performance advantage over IC3Net grows with environment complexity, but only when the threshold is properly calibrated: PP 10×\\times10 shows no advantage, PP 20×\\times20 atε=0\.5\\varepsilon=0\.5shows a clear inverted U\-shape with 1\.47 steps and 11pp improvement\. Second, even when the gate does not selectively activate, as on MPE where beliefs are always highly divergent, the belief head still improves performance and variance over IC3Net, suggesting two orthogonal contributions of the method\. The belief size ablation further confirms thatK=8K=8is a principled choice, balancing representational expressiveness with learnability across random seeds\. Future work includes a direct comparison against T2MAC\[[10](https://arxiv.org/html/2608.14559#bib.bib10)\]on SMAC, investigation of adaptive thresholding via a running estimate of average KL divergence, and evaluation on SMAC as a large\-scale cooperative benchmark where richer partial observability may allow the principled trigger to show a clearer advantage\.

## References

- \[1\]Sukhbaatar, S\., Fergus, R\., et al\.Learning multiagent communication with backpropagation\.InAdvances in Neural Information Processing Systems \(NeurIPS\), 2016\.
- \[2\]Singh, A\., Jain, T\., and Sukhbaatar, S\.Learning when to communicate at scale in multiagent cooperative and competitive tasks\.InInternational Conference on Learning Representations \(ICLR\), 2019\.
- \[3\]Åström, K\. J\.Optimal control of Markov decision processes with incomplete state estimation\.Journal of Mathematical Analysis and Applications, 10:174–205, 1965\.
- \[4\]Foerster, J\., Assael, Y\., de Freitas, N\., and Whiteson, S\.Learning to communicate with deep multi\-agent reinforcement learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\), 2016\.
- \[5\]Williams, R\. J\.Simple statistical gradient\-following algorithms for connectionist reinforcement learning\.Machine Learning, 8\(3–4\):229–256, 1992\.
- \[6\]Lowe, R\., Wu, Y\., Tamar, A\., Harb, J\., Abbeel, P\., and Mordatch, I\.Multi\-agent actor\-critic for mixed cooperative\-competitive environments\.InAdvances in Neural Information Processing Systems \(NeurIPS\), 2017\.
- \[7\]Rashid, T\., Samvelyan, M\., de Witt, C\., Farquhar, G\., Foerster, J\., and Whiteson, S\.QMIX: Monotonic value function factorisation for deep multi\-agent reinforcement learning\.InInternational Conference on Machine Learning \(ICML\), 2018\.
- \[8\]Das, A\., Gerber, S\., Datta, S\., Lee, S\., Abbeel, P\., Parikh, D\., and Batra, D\.TarMAC: Targeted multi\-agent communication\.InInternational Conference on Machine Learning \(ICML\), 2019\.
- \[9\]Hochreiter, S\. and Schmidhuber, J\.Long short\-term memory\.Neural Computation, 9\(8\):1735–1780, 1997\.
- \[10\]Sun, C\., Zang, Z\., Li, J\., Li, J\., Xu, X\., Wang, R\., and Zheng, C\.T2MAC: Targeted and Trusted Multi\-Agent Communication through Selective Engagement and Evidence\-Driven Integration\.InProceedings of the AAAI Conference on Artificial Intelligence, 2024\.
- \[11\]Hill, B\. A\., Wei, M\. K\. E\., and Jishnuanandh, T\.Communicating plans, not percepts: Scalable multi\-agent coordination with embodied world models\.InProceedings of the 39th Conference on Neural Information Processing Systems \(NeurIPS\) Workshops, 2025\.

## Appendix AAdditional Results

![Refer to caption](https://arxiv.org/html/2608.14559v1/x3.png)Figure 4:Average steps per episode over training on PP20×2020\{\\times\}20\(5 seeds, mean±\\pmstd\)\. KL\-Beliefε=0\.5\\varepsilon\{=\}0\.5converges to a lower plateau \(73\.84 steps\) than IC3Net \(75\.31 steps\)\.![Refer to caption](https://arxiv.org/html/2608.14559v1/x4.png)Figure 5:Communication rate over training for allε\\varepsilonvalues on PP20×2020\{\\times\}20\(5 seeds, mean±\\pmstd\)\. Higherε\\varepsilongenerally suppresses communication rate\. IC3Net remains lowest throughout\.