The Role of Network Topology and Opponent Information in Shaping Cooperation in Multi-Agent Reinforcement Learning Systems

# The Role of Network Topology and Opponent Information in Shaping Cooperation in Multi-Agent Reinforcement Learning Systems
Source: [https://arxiv.org/html/2608.28977](https://arxiv.org/html/2608.28977)
Seongho Son††thanks:Corresponding author:seong\.son\.22@ucl\.ac\.ukAffiliation:Department of Computer Science, University College LondonAffiliation:UCL Centre for Artificial Intelligence, University College LondonMirco MusolesiAffiliation:Department of Computer Science, University College LondonAffiliation:UCL Centre for Artificial Intelligence, University College LondonAffiliation:Department of Computer Science and Engineering, University of Bologna

###### Abstract

Several works have investigated the influence of graph topology on cooperation among artificial agents, while the majority of the literature has focused on modelling agents’ adaptation through strategy imitation, which relies solely on the cumulative payoffs of others\. This paper investigates scenarios in which each agent learns to play the two\-player Iterated Prisoner’s Dilemma \(IPD\) using deep reinforcement learning\. Each agent is represented as a node in a graph, where its neighbours constitute the pool of opponents with whom it can interact\. During each IPD episode, agents are provided with different types of information about their opponent, consisting of action history and opponent identity\. Experimental results across different graph topologies show that the number of neighbours per node and the average path length are the main factors affecting the emergence of cooperation\. We also show that, while partner selection fosters mutual cooperation by limiting the diversity of the opponent pool, providing agents with the identity of their opponent hinders the proliferation of cooperative strategies\.

Keywords:Iterated Prisoner’s Dilemma; cooperation; graph topology; deep reinforcement learning; partner selection; state information\.

## 1Introduction

A large portion of interactions within human societies consists of altruistic behaviour, ranging from donating blood\[[1](https://arxiv.org/html/2608.28977#bib.bib15),[2](https://arxiv.org/html/2608.28977#bib.bib16)\]to investing in early\-stage ventures with little expectation of direct returns\[[3](https://arxiv.org/html/2608.28977#bib.bib17)\]\. Similar forms of altruistic behaviour can also emerge among artificial agents, where agents may incur individual costs to benefit others or contribute to collective outcomes\.

If self\-interested behaviour of each individual is assumed, what could be the factors that affect their tendency to be cooperative? The emergence of cooperation among multiple agents has attracted a substantial amount of attention in academic research\[[4](https://arxiv.org/html/2608.28977#bib.bib8),[5](https://arxiv.org/html/2608.28977#bib.bib12)\]\. Social dilemmas in particular have been extensively studied, as they present mixed\-motive situations where individually optimal choices are to act selfishly while socially optimal choices are to be cooperative\[[6](https://arxiv.org/html/2608.28977#bib.bib9),[7](https://arxiv.org/html/2608.28977#bib.bib11),[8](https://arxiv.org/html/2608.28977#bib.bib10)\]\.

Reinforcement learning \(RL\) has been widely used to model the learning dynamics of societies of artificial and human agents\[[9](https://arxiv.org/html/2608.28977#bib.bib20),[10](https://arxiv.org/html/2608.28977#bib.bib4),[11](https://arxiv.org/html/2608.28977#bib.bib22),[12](https://arxiv.org/html/2608.28977#bib.bib21),[13](https://arxiv.org/html/2608.28977#bib.bib44),[14](https://arxiv.org/html/2608.28977#bib.bib41)\]\. In RL, agents learn by changing behaviour to maximise their return, which is defined as the discounted sum of rewards\. If the agent receives a rewardrk∈ℝr\_\{k\}\\in\\mathbb\{R\}at time stepkkand the discount rate is denoted asγ∈\(0,1\]\\gamma\\in\(0,1\], the returnGGis defined byG=∑k=0∞γk​rkG=\\sum\_\{k=0\}^\{\\infty\}\\gamma^\{k\}r\_\{k\}\. With RL, we can investigate the behaviour of individual agents learning with various reward structures and different levels of access to information about the surrounding environment\. The stationarity of the environment surrounding the agent is essential for the theoretical guarantee of the agent’s performance improvement, which is violated with multiple agents learning at the same time\[[12](https://arxiv.org/html/2608.28977#bib.bib21)\]\. Along with partial observability of the state, these features of multi\-agent environments have been the major obstacles to stable and efficient training\. Although approaches such as\[[9](https://arxiv.org/html/2608.28977#bib.bib20),[15](https://arxiv.org/html/2608.28977#bib.bib31)\]tried to address this issue by modelling or actually gaining access to the information of the agents’ opponents, the methods can become intractable due to weak scalability\. On the other hand, partner selection algorithms have been proposed as an approach to training multiple RL agents in decentralised scenarios, enabling agents to learn norm\-inducing behaviours and promote cooperation\[[10](https://arxiv.org/html/2608.28977#bib.bib4),[16](https://arxiv.org/html/2608.28977#bib.bib38),[17](https://arxiv.org/html/2608.28977#bib.bib39),[13](https://arxiv.org/html/2608.28977#bib.bib44)\]\.

Graph topology representing agents as nodes provides a useful way to analyse relation between individuals and draw insights from the structure\. Numerous approaches have been presented on the issue of cooperation using graph representation of the society\[[18](https://arxiv.org/html/2608.28977#bib.bib29),[19](https://arxiv.org/html/2608.28977#bib.bib30),[20](https://arxiv.org/html/2608.28977#bib.bib19),[21](https://arxiv.org/html/2608.28977#bib.bib23),[22](https://arxiv.org/html/2608.28977#bib.bib42),[23](https://arxiv.org/html/2608.28977#bib.bib27),[24](https://arxiv.org/html/2608.28977#bib.bib24),[25](https://arxiv.org/html/2608.28977#bib.bib26),[26](https://arxiv.org/html/2608.28977#bib.bib25),[27](https://arxiv.org/html/2608.28977#bib.bib43),[28](https://arxiv.org/html/2608.28977#bib.bib45),[29](https://arxiv.org/html/2608.28977#bib.bib46)\]\. Many existing works simulated agents’ behaviour with learning algorithm based on imitation, in which each agent copies one of its neighbour’s strategies with probability proportional to the difference between their accumulated payoffs\. This approach enables agents to adopt the strategy that results in higher payoff, while it requires access to information on others’ payoffs\[[20](https://arxiv.org/html/2608.28977#bib.bib19),[21](https://arxiv.org/html/2608.28977#bib.bib23),[26](https://arxiv.org/html/2608.28977#bib.bib25),[18](https://arxiv.org/html/2608.28977#bib.bib29),[19](https://arxiv.org/html/2608.28977#bib.bib30)\]\. Reinforcement learning does not rely on this requirement by using only the reward gained by the agent itself to adjust its behaviour\. Although previous studies have highlighted the significant impact of graph topology on the emergence of cooperation, relatively little research has explored how it shapes the behaviour of reinforcement learning–based agents\.

During interactions, agents’ behaviours strongly depend on the information about their opponents\. Previous work has investigated the role of reputation in the emergence of cooperation, often in the format of action history\[[10](https://arxiv.org/html/2608.28977#bib.bib4),[30](https://arxiv.org/html/2608.28977#bib.bib7),[31](https://arxiv.org/html/2608.28977#bib.bib18),[32](https://arxiv.org/html/2608.28977#bib.bib32)\]\. Agents can prefer other agents with cooperative action history as their interaction partners, which can motivate agents to build a reputation of being cooperative\. On the other hand, exposing information on their behaviours can render agents vulnerable to be targeted by defective agents and be exploited\. Agents’ behaviours can also be affected by the presence of their interaction partners’ identity information in their observation\[[33](https://arxiv.org/html/2608.28977#bib.bib33),[34](https://arxiv.org/html/2608.28977#bib.bib28),[35](https://arxiv.org/html/2608.28977#bib.bib34)\]\. When agents can observe the identity of their interaction partners, they can use different strategy\. Behaviours like unconditionally cooperating with some of the agents while exploiting the others are possible in this setting\. In this paper, we investigate cases where identity information is provided such that each individual can be distinguished from others by decoding state information\.

Our work investigates how graph topology and opponent identity influence the emergence of cooperation\. Each agent is represented as a node in a graph and interacts with its neighbours by playing two\-player Iterated Prisoner’s Dilemma \(IPD\)\[[4](https://arxiv.org/html/2608.28977#bib.bib8)\]\. We provide experimental results with three different types of synthetic graphs and show that longer average path length and sparser graphs lead to more mutual cooperation of the agents\. We also conduct experiments in which agents perform partner selection before IPD\. We demonstrate that access to longer histories of an opponent’s actions allows agents to more accurately discern the opponent’s strategic behaviour\. We report the results of the experiment of including identity information from the opponent in the state, in which the emergence of cooperation is hindered by enabling agents with defective strategies to spot entirely cooperative agents while avoiding retaliating agents, while providing more information on the action history alleviates the phenomenon\.

## 2Iterated Prisoner’s Dilemma on Graphs

### 2\.1Game Structure

We assume that each agent in the population is represented as a nodev∈𝒱v\\in\\mathcal\{V\}in an undirected graph𝒢=\(𝒱,ℰ\)\\mathcal\{G\}=\(\\mathcal\{V\},\\mathcal\{E\}\), with the set of edges in the graph denoted asℰ\\mathcal\{E\}\. Each agenti∈\{1,2,…,N\}i\\in\\\{1,2,\.\.\.,N\\\}participates in an interaction with one of the other agents whose nodes are in its neighbourhood𝒩v=\{u\|\(u,v\)∈ℰ∨\(v,u\)∈ℰ\}\\mathcal\{N\}\_\{v\}=\\\{u\\,\|\\,\(u,v\)\\in\\mathcal\{E\}\\vee\(v,u\)\\in\\mathcal\{E\}\\\}\. For each interaction agents decide whether to cooperate \(C\) or to defect \(D\), and receive payoffsrrbased on the result of each interaction characterised by a payoff matrix of Prisoner’s Dilemma \(PD\)\[[30](https://arxiv.org/html/2608.28977#bib.bib7),[4](https://arxiv.org/html/2608.28977#bib.bib8),[6](https://arxiv.org/html/2608.28977#bib.bib9),[10](https://arxiv.org/html/2608.28977#bib.bib4)\], as in Table[1](https://arxiv.org/html/2608.28977#S2.T1)\. A game is considered PD whenT\>R\>P\>ST\>R\>P\>Sand2​R\>T\+S2R\>T\+Sare satisfied\. We will denote 4 possible interactions in an episode of PD from the row player’s perspective: \(C, C\) as mutual cooperation, \(C, D\) as sucker, \(D, C\) as temptation, \(D, D\) as mutual defection, where\(ai,aj\)\(a\_\{i\},a\_\{j\}\)represents action pair of row playeriiand column playerjj\.

We further specify the relationship between payoffs using benefit to cost ratiob:c=5:1b:c=5:1\[[8](https://arxiv.org/html/2608.28977#bib.bib10)\], whereR=b−c,T=b,S=−c,P=0R=b\-c,T=b,S=\-c,P=0\. Payoff values are shifted such thatR\+P=T\+S=0R\+P=T\+S=0, which makes initialised strategies of agents be equally distributed across cooperative and defective strategies\. Payoff values are rescaled to prevent numerical instability during the training of neural networks\.

Table 1:\(a\) General format of payoff matrix used for an episode of Prisoner’s Dilemma \(left\)\. \(b\) Payoff matrix used in each interaction of Iterated Prisoner’s Dilemma \(right\)\.Each agentiiis randomly assigned an opponento∈𝒩io\\in\\mathcal\{N\}\_\{i\}for an interaction\. Due to the randomness of the assignment of the opponent and the structure of the graph, each agent can interact with different numbers of opponents in an interaction roundkk\. We index the interactions occurring in each round ast∈\{1,…,N−1\}t\\in\\\{1,\.\.\.,N\-1\\\}\. Agents are provided with information of the previous actions of the assigned opponents\. The action history of the opponent is given in the format of a concatenated one hot vector,si∈𝒮d​i​l=ℝ2×ls^\{i\}\\in\\mathcal\{S\}\_\{dil\}=\\mathbb\{R\}^\{2\\times l\}, wherelldenotes the length of the action history\. This is used for the state vector of the agent’s dilemma\-playing Deep Q\-networkfd​i​li​\(si\)f\_\{dil\}^\{i\}\(s^\{i\}\)\[[36](https://arxiv.org/html/2608.28977#bib.bib5)\]to select an actionai∈𝒜d​i​l=\{C,D\}a^\{i\}\\in\\mathcal\{A\}\_\{dil\}=\\\{C,D\\\}\. We usel=1l=1, and report the result of the experiment after running each setting for 100,000 interaction rounds\.

### 2\.2Synthetic Graphs

For experiments with random assignment of opponents, we use three graph types: Erdős\-Rényi \(ER\)\[[37](https://arxiv.org/html/2608.28977#bib.bib1)\], Watts\-Strogatz \(WS\)\[[38](https://arxiv.org/html/2608.28977#bib.bib2)\]and Barabási\-Albert \(BA\)\[[39](https://arxiv.org/html/2608.28977#bib.bib3)\]\. For ER graphs, we generate graphs with wiring probabilitypE​Rp\_\{ER\}sampled fromU⁡\[0\.1,1\]U\[0\.1,1\]\. For WS graphs, we first construct a regular graph with 4 edges per node and rewire edges with probabilitypW​S∼U⁡\[0,1\]p\_\{WS\}\\sim U\[0,1\]\. For BA graphs, we denote the number of edges added per each preferential attachment asm∈\{1,2,3,4\}m\\in\\\{1,2,3,4\\\}\. In BA graphs, preferential attachment is performed after initialmmnodes\.

We generate 100 graphs for each type with parameters sampled with uniform probability using the method mentioned above\. For experiments with partner selection, we only use graphs with full connectivity\. For all experiments, we useN=32N=32agents for the population\.

### 2\.3IPD with Partner Selection

Inspired by\[[10](https://arxiv.org/html/2608.28977#bib.bib4)\], we conduct experiments on cases where partner selection is involved\. Each agenti∈\{1,…,N\}i\\in\\\{1,\.\.\.,N\\\}gets an opportunity to select a partner using its partner selection moduleηs​e​l,πi\\eta\_\{sel,\\pi\}^\{i\}\. The input for this module consists of the concatenated state vector of neighbouring\(N−1\)\(N\-1\)agents in the graph,Ss​e​li∈ℝ\(N−1\)×dd​i​lS\_\{sel\}^\{i\}\\in\\mathbb\{R\}^\{\(N\-1\)\\times d\_\{dil\}\}\.dd​i​ld\_\{dil\}denotes the dimension of each state vector used for each interaction of Prisoner’s Dilemma\. The output of the partner selection module consists of selection probability for each candidate,ηs​e​l,πi​\(Ss​e​li\)∈ℝ\(N−1\)\\eta\_\{sel,\\pi\}^\{i\}\(S\_\{sel\}^\{i\}\)\\in\\mathbb\{R\}^\{\(N\-1\)\}\.

For the state vectorsSs​e​liS\_\{sel\}^\{i\}used for playing IPD in experiments with partner selection, we add an option to provide the identity of the opponent\. The opponent’s indexo∈\{1,…,i−1,i\+1,…,N\}o\\in\\\{1,\.\.\.,i\-1,i\+1,\.\.\.,N\\\}is provided in the format of binary encoding\. For example, in the population with 32 agents, an opponent with index number 13 will be represented as01101201101\_\{2\}\. This makesdd​i​l=2×l\+⌈log2⁡N⌉d\_\{dil\}=2\\times l\+\\lceil\\log\_\{2\}N\\rceilwhen opponent identity is included\. We run experiments with 6 different settings, where only one of two settings with the same valuel∈\{1,5,10\}l\\in\\\{1,5,10\\\}has opponent identity in the dilemma\-playing statess\. Each setting is run with 20 different seeds, with each run consisting of 200,000 interaction rounds\.

## 3Reinforcement Learning Implementation

### 3\.1Playing the IPD

The learning algorithm used for this experiment is a variant of Deep Q\-learning\[[36](https://arxiv.org/html/2608.28977#bib.bib5)\]\. Deep Q\-learning uses neural networks to estimate the optimal action\-value functionQ∗Q^\{\*\}\. When the agent selects an actionata\_\{t\}using policyπ\\piand receives rewardrtr\_\{t\}for the action, and when the discount rate isγ\\gamma,Q∗​\(st,at\)=maxπ⁡𝔼⁡\[∑j=0∞γj​rt\+j\|π\]Q^\{\*\}\(s\_\{t\},a\_\{t\}\)=\\max\_\{\\pi\}\\mathbb\{E\}\[\\sum\_\{j=0\}^\{\\infty\}\\gamma^\{j\}r\_\{t\+j\}\|\\pi\]\. Each agent is equipped with a separate neural network for Prisoner’s Dilemmafd​i​lif^\{i\}\_\{dil\}with its parameter denoted asθd​i​li\\theta\_\{dil\}^\{i\}\. Agents are trained independently with each of them having a buffer to store transitions\{\(sk,t,ak,t,rk,t,sk,t\+1\):k=1,…,K,t=1,…,N−1\}\\\{\(s\_\{k,t\},a\_\{k,t\},r\_\{k,t\},s\_\{k,t\+1\}\):k=1,\.\.\.,K,t=1,\.\.\.,N\-1\\\}used for training\. The policy of each agent is updated after each interaction roundkkand transitions are discarded from training buffer after each update, in order to maintain relevance to the current transition dynamics of the environment\[[10](https://arxiv.org/html/2608.28977#bib.bib4)\]\. The dilemma\-playing policy of agentiican be described as

πi​\(si\)=\{argmaxai∈𝒜d​i​l​Qi​\(si,ai\)with probability​1−ϵ𝒰⁡\(𝒜d​i​l\),otherwise,\\displaystyle\\pi^\{i\}\(s^\{i\}\)=\\begin\{cases\}\\text\{argmax\}\_\{a^\{i\}\\in\\mathcal\{A\}\_\{dil\}\}Q^\{i\}\(s^\{i\},a^\{i\}\)&\\text\{with probability\}\\ 1\-\\epsilon\\\\ \\mathcal\{U\}\(\\mathcal\{A\}\_\{dil\}\),&\\text\{otherwise,\}\\end\{cases\}\(1\)
where𝒰⁡\(𝒜d​i​l\)\\mathcal\{U\}\(\\mathcal\{A\}\_\{dil\}\)denotes sampling from a uniform distribution\[[10](https://arxiv.org/html/2608.28977#bib.bib4)\]\. After each interaction roundkk, dilemma\-playing networks update their Q\-values by minimising the objective functionLk​\(θd​i​l,ki\)L\_\{k\}\(\\theta\_\{dil,k\}^\{i\}\)as described below\. When calculating the objective function, we estimate the Q\-value in the next time stept\+1t\+1with another network called the target network to stabilise learning\[[36](https://arxiv.org/html/2608.28977#bib.bib5)\]\. The target network is initialised with the same architecture and parameter values as the dilemma\-playing network, while its parametersθd​i​li,−\\theta^\{i,\-\}\_\{dil\}are copied from the original network everyZ=16Z=16interaction rounds\. We use exploration rateϵ=0\.05\\epsilon=0\.05, discount rateγ=0\.99\\gamma=0\.99\. We use deep neural networks with two hidden layers, 32 and 16 nodes respectively\. Rectified Linear Unit is used for the activation function\. Withs′s^\{\\prime\}denoting the state at the next time stepsk,t\+1s\_\{k,t\+1\}, the loss function is defined as follows:

Lk​\(θd​i​l,ki\)=𝔼s,a,r,s′​\[\(yk,ti−Qi​\(sk,ti,ak,ti,θd​i​l,ki\)\)2\],yk,ti=rk,ti\+γ​maxa′∈𝒜d​i​l​Qi​\(sk,t\+1i,a′,θd​i​l,ki,−\)\\displaystyle\\begin\{split\}L\_\{k\}\(\\theta\_\{dil,k\}^\{i\}\)=\\mathbb\{E\}\_\{s,a,r,s^\{\\prime\}\}\\biggl\[\\bigl\(y^\{i\}\_\{k,t\}\-Q^\{i\}\(s^\{i\}\_\{k,t\},a^\{i\}\_\{k,t\};\\theta\_\{dil,k\}^\{i\}\)\\bigr\)^\{2\}\\biggr\],\\\\ y^\{i\}\_\{k,t\}=r^\{i\}\_\{k,t\}\+\\gamma\\max\_\{a^\{\\prime\}\\in\\mathcal\{A\}\_\{dil\}\}Q^\{i\}\(s^\{i\}\_\{k,t\+1\},a^\{\\prime\};\\theta^\{i,\-\}\_\{dil,k\}\)\\end\{split\}\(2\)
Figure[1](https://arxiv.org/html/2608.28977#S3.F1)summarises the resulting interaction and training loop from the perspective of agentii\.

Figure 1:Visual summary of IPD on graphs, from the perspective ofiith agent\. For each interaction round, each agent selects an opponent among its neighbours in the graph with uniform probability\. Both experiences of playing with opponents that agentiiselected or the agent being selected by one of its neighbours are used for training the dilemma\-playing policyπi\\pi^\{i\}\.
### 3\.2Partner Selection

In experiments with partner selection, each agent uses separate neural networks for learning how to choose the opponent\. These networks used for selectionfs​e​l,Qi,fs​e​l,πif\_\{sel,Q\}^\{i\},f\_\{sel,\\pi\}^\{i\}have the same structure as the dilemma\-playing network, except that their outputs are single values representing the Q\-value or action probability inℝ1\\mathbb\{R\}^\{1\}for selecting the given agent as a partner\. With the information of agentii’s neighbouroi,j∈𝒩i,j∈\{1,…,\|𝒩i\|\}o^\{i,j\}\\in\\mathcal\{N\}\_\{i\},j\\in\\\{1,\.\.\.,\|\\mathcal\{N\}\_\{i\}\|\\\}denoted asss​e​li,js\_\{sel\}^\{i,j\},Ss​e​liS\_\{sel\}^\{i\}is\[ss​e​li,1,ss​e​li,2,…,ss​e​li,\|𝒩i\|\]\[s\_\{sel\}^\{i,1\},s\_\{sel\}^\{i,2\},\.\.\.,s\_\{sel\}^\{i,\|\\mathcal\{N\}\_\{i\}\|\}\], and selection network is applied to each ofss​e​li,js\_\{sel\}^\{i,j\}\. The outputs of the partner selection module then becomeηs​e​l,Qi​\(Ss​e​li\)=\[fs​e​l,Qi​\(ss​e​li,1\),fs​e​l,Qi​\(ss​e​li,2\),…,fs​e​l,Qi​\(ss​e​li,\|𝒩i\|\)\]\\eta\_\{sel,Q\}^\{i\}\(S\_\{sel\}^\{i\}\)=\[f\_\{sel,Q\}^\{i\}\(s\_\{sel\}^\{i,1\}\),f\_\{sel,Q\}^\{i\}\(s\_\{sel\}^\{i,2\}\),\.\.\.,f\_\{sel,Q\}^\{i\}\(s\_\{sel\}^\{i,\|\\mathcal\{N\}\_\{i\}\|\}\)\]for the Q\-function andηs​e​l,πi​\(Ss​e​li\)=\[fs​e​l,πi​\(ss​e​li,1\),fs​e​l,πi​\(ss​e​li,2\),…,fs​e​l,πi​\(ss​e​li,\|𝒩i\|\)\]\\eta\_\{sel,\\pi\}^\{i\}\(S\_\{sel\}^\{i\}\)=\[f\_\{sel,\\pi\}^\{i\}\(s\_\{sel\}^\{i,1\}\),f\_\{sel,\\pi\}^\{i\}\(s\_\{sel\}^\{i,2\}\),\.\.\.,f\_\{sel,\\pi\}^\{i\}\(s\_\{sel\}^\{i,\|\\mathcal\{N\}\_\{i\}\|\}\)\]for the policy\. We use the information from agentii’s neighbours att=1t=1forss​e​li,js\_\{sel\}^\{i,j\}in the interaction roundkk, because each agent performs partner selection before any agent in the population plays an episode of the Prisoner’s Dilemma\. Figure[2](https://arxiv.org/html/2608.28977#S3.F2)summarises the corresponding procedure when partner selection is involved\.

Figure 2:Visual summary of IPD with partner selection, from the perspective ofiith agent\. For each interaction round, each agent selects an opponent among the otherN−1N\-1agents using partner selection moduleηs​e​l,πi\\eta^\{i\}\_\{sel,\\pi\}\. The partner selection module is trained with only the experience of agentiiselecting an opponent, as the action of partner selectionas​e​lia^\{i\}\_\{sel\}is the index of the selected opponent\.We use Soft Actor\-Critic \(SAC\)\[[40](https://arxiv.org/html/2608.28977#bib.bib40)\]for training the partner selection networks\. SAC adds an entropy termℍ\(π\(⋅\|s\)\)\\mathbb\{H\}\(\\pi\(\\cdot\|s\)\)as a regularisation term to the policy gradient objective, making the optimal policyπ∗=argmaxπ∑k=0K𝔼st∼p,at∼π\[r\(st,at\)\+αℍ\(π\(⋅\|st\)\)\]\\pi^\{\*\}=\\text\{argmax\}\_\{\\pi\}\\sum\_\{k=0\}^\{K\}\\mathbb\{E\}\_\{s\_\{t\}\\sim p,a\_\{t\}\\sim\\pi\}\[r\(s\_\{t\},a\_\{t\}\)\+\\alpha\\mathbb\{H\}\(\\pi\(\\cdot\|s\_\{t\}\)\)\]andℍ\(π\(⋅\|s\)\)=−∫π\(a\|s\)logπ\(a\|s\)da=𝔼a∼π\(⋅\|s\)\[−logπ\(a\|s\)\]\\mathbb\{H\}\(\\pi\(\\cdot\|s\)\)=\-\\int\\pi\(a\|s\)\\log\\pi\(a\|s\)da=\\mathbb\{E\}\_\{a\\sim\\pi\(\\cdot\|s\)\}\[\-\\log\\pi\(a\|s\)\]\.α=0\.01\\alpha=0\.01is the parameter determining the importance of the entropy term against the reward\. The soft state\-value function and soft Q\-function are defined asV⁡\(s\)=𝔼a∼π​\[Q⁡\(s,a\)−α​log⁡\(π⁡\(a\|s\)\)\]V\(s\)=\\mathbb\{E\}\_\{a\\sim\\pi\}\[Q\(s,a\)\-\\alpha\\log\(\\pi\(a\|s\)\)\]\. We denote the parameters of agentii’s selection policy asϕs​e​li\\phi^\{i\}\_\{sel\}, and parameters of the soft Q\-function for selection asθs​e​li\\theta\_\{sel\}^\{i\}\. We also use target network parametersθ¯s​e​l\\bar\{\\theta\}\_\{sel\}for soft Q\-function like we did for training policies to play IPD\.rkir^\{i\}\_\{k\}denotes the payoff gained by playing with the opponent which agentiiselected in roundkk\.θs​e​l\\theta\_\{sel\}are trained to minimise the soft Bellman residual:

Ls​e​l,kQ​\(θs​e​l,ki\)=𝔼s,a,r,s′​\[\(Qs​e​li​\(s,a,θs​e​l,ki\)−ys​e​l,ki\)2\],ys​e​l,ki=rs​e​l,ki\+γ​∑a′π⁡\(a′\|s′;ϕs​e​l,ki\)​\(Qs​e​li​\(s′,a′,θ¯s​e​l,ki\)−α​log⁡π⁡\(a′\|s′;ϕs​e​l,ki\)\),s=sis​e​l,k,a=ais​e​l,k,r=ris​e​l,k,s′=sis​e​l,k\+1,a′=ais​e​l,k\+1\.\\displaystyle\\begin\{split\}L\_\{sel,k\}^\{Q\}\(\\theta\_\{sel,k\}^\{i\}\)=\\mathbb\{E\}\_\{s,a,r,s^\{\\prime\}\}\\biggl\[\\bigl\(Q\_\{sel\}^\{i\}\(s,a;\\theta\_\{sel,k\}^\{i\}\)\-y^\{i\}\_\{sel,k\}\\bigr\)^\{2\}\\biggr\],\\\\ y^\{i\}\_\{sel,k\}=r^\{i\}\_\{sel,k\}\+\\gamma\\sum\_\{a^\{\\prime\}\}\\pi\(a^\{\\prime\}\|s^\{\\prime\};\\phi\_\{sel,k\}^\{i\}\)\(Q\_\{sel\}^\{i\}\(s^\{\\prime\},a^\{\\prime\};\\bar\{\\theta\}\_\{sel,k\}^\{i\}\)\-\\alpha\\log\\pi\(a^\{\\prime\}\|s^\{\\prime\};\\phi\_\{sel,k\}^\{i\}\)\),\\\\ s=s^\{i\}\_\{sel,k\},\\ a=a^\{i\}\_\{sel,k\},\\ r=r^\{i\}\_\{sel,k\},\\ s^\{\\prime\}=s^\{i\}\_\{sel,k\+1\},\\ a^\{\\prime\}=a^\{i\}\_\{sel,k\+1\}\.\\end\{split\}\(3\)
In SAC, policies are updated towards the exponential of the soft Q\-function:

Ls​e​l,kπ​\(ϕs​e​l,ki\)=𝔼s​\[∑aπ⁡\(a\|s;ϕs​e​l,ki\)​\(α​log⁡π⁡\(a\|s;ϕs​e​l,ki\)−Qs​e​li​\(s,a,θs​e​l,ki\)\)\],s=sis​e​l,k,a=ais​e​l,k\.\\displaystyle\\begin\{split\}L\_\{sel,k\}^\{\\pi\}\(\\phi\_\{sel,k\}^\{i\}\)=\\mathbb\{E\}\_\{s\}\\biggl\[\\sum\_\{a\}\\pi\(a\|s;\\phi\_\{sel,k\}^\{i\}\)\(\\alpha\\log\\pi\(a\|s;\\phi\_\{sel,k\}^\{i\}\)\-Q\_\{sel\}^\{i\}\(s,a;\\theta\_\{sel,k\}^\{i\}\)\)\\biggr\],\\\\ s=s^\{i\}\_\{sel,k\},\\ a=a^\{i\}\_\{sel,k\}\.\\end\{split\}\(4\)
For target networks of soft Q\-functions, we useτ=0\.05\\tau=0\.05to gradually apply update:θ¯s​e​l,ki←τ​θs​e​l,ki\+\(1−τ\)​θ¯s​e​l,ki\\bar\{\\theta\}\_\{sel,k\}^\{i\}\\leftarrow\\tau\\theta\_\{sel,k\}^\{i\}\+\(1\-\\tau\)\\bar\{\\theta\}\_\{sel,k\}^\{i\}\. This implementation of partner selection is different from that of\[[10](https://arxiv.org/html/2608.28977#bib.bib4)\], which gave implicit information on opponent identity to only selection networks\. Our architecture enables us to have the option of explicitly including identity information in the state vectors and investigate the effect of it on the interaction dynamics of playing IPD\. The implementation of the environment and of both learning algorithms, together with the configuration files reproducing every experiment reported below, is publicly available\.

## 4Results

### 4\.1IPD with Random Matching

We first present in Figure[3](https://arxiv.org/html/2608.28977#S4.F3)the proportion of mutual defectionpm​dp\_\{md\}in each run’s last 100 interaction rounds with the parameters used to generate graphs\. Most of the runs on ER graphs withpE​R\>0\.2p\_\{ER\}\>0\.2converge to complete mutual defection\. In WS graphs the tendency is less apparent compared to that of ER graphs, but as the value ofpW​Sp\_\{WS\}increases, the higher the level of mutual defection\. In BA graphs, as the value ofmmincreases, runs tend to converge to a higher level of mutual defection\.

We further investigate this phenomenon using metrics calculated from graphs\. Figure[4](https://arxiv.org/html/2608.28977#S4.F4)shows the results of the experiment considering the relationship with the average path length, which is calculated by averaging the shortest path length between each pair of nodes in the graph:dp​a​t​h​\(𝒢\)=∑v,u∈𝒱,v≠ud⁡\(v,u\)\|𝒱\|​\(\|𝒱\|−1\)d\_\{path\}\(\\mathcal\{G\}\)=\\frac\{\\sum\_\{v,u\\in\\mathcal\{V\},v\\neq u\}d\(v,u\)\}\{\|\\mathcal\{V\}\|\(\|\\mathcal\{V\}\|\-1\)\}whered⁡\(v,u\)d\(v,u\)denotes the shortest path length between verticesvvanduu\. Figure[5](https://arxiv.org/html/2608.28977#S4.F5)shows the relation between the graph generation parameterspE​R,pW​S,mp\_\{ER\},p\_\{WS\},mand the average path length of each generated graph\. Across all three types of graphs, more runs converge to a higher value ofpm​dp\_\{md\}as the average path length decreases\. In ER graphs, except one outlier, the other runs withdp​a​t​h<2\.5d\_\{path\}<2\.5converge topm​d\>0\.8p\_\{md\}\>0\.8\. BA graphs are characterised by a longerdp​a​t​hd\_\{path\}than ER graphs, reachingdp​a​t​h\>4d\_\{path\}\>4whenm=1m=1\. The level of mutual defection in BA graphs withdp​a​t​h\>3d\_\{path\}\>3does not increase above 0\.6, while many graphs generated withdp​a​t​h<2\.5d\_\{path\}<2\.5result inpm​d\>0\.8p\_\{md\}\>0\.8\. This corresponds to cases wherem=3,4m=3,4as shown in Figure[3](https://arxiv.org/html/2608.28977#S4.F3)\. A similar pattern is shown in the results of the WS networks, with the value ofpm​dp\_\{md\}staying below 0\.6 whendp​a​t​h\>3\.5d\_\{path\}\>3\.5and reaching above 0\.9 whendp​a​t​h<3d\_\{path\}<3\.

Figure 3:Experimental results per graph type with parameters for graph generation \(ER, WS and BA from the left, respectively\)\.Figure 4:Experimental results per graph type with average path length per graph \(ER, WS and BA from the left, respectively\)\.Figure 5:Relation between graph generation parameters and average path length \(ER, WS and BA from the left, respectively\)\.
### 4\.2IPD with Partner Selection

Figure[6](https://arxiv.org/html/2608.28977#S4.F6)shows the result of the experiments performed with partner selection\. The average proportion of mutual cooperation in the last 100 interaction rounds, denoted aspm​cp\_\{mc\}, is calculated across 20 runs with 95% confidence interval\. When no information on the identity of the opponent is included in the agent state information, the level of mutual cooperation after 200,000 interaction rounds does not show a statistically significant difference when the length of the action historyllchanges\. Significant difference occurs when binary encoding of the opponent’s index is provided in the state information\.pm​cp\_\{mc\}drops below 0\.3 whenl=1l=1, largely deviating from cases without the identity of the opponent\.pm​cp\_\{mc\}increases above 0\.5 whenl=5,10l=5,10, which makes the confidence intervals ofpm​cp\_\{mc\}overlap with the other experiments without identity information\. This implies that the effect of identity information on the proportion of mutual cooperation becomes insignificant as action history length increases\.

Figure 6:Experimental results of partner selection experiment\. Dots represent the average proportion of mutual cooperation in the last 100 interaction rounds of 20 runs, while vertical intervals represent 95% confidence intervals of each experiment setting\. Experiments conducted without having identity information in the states of dilemma\-playing agents are coloured in green, while experiments which included binary encoding of the opponent’s identity index are coloured in orange\. Except for the experiment with identity information and action history lengthl=1l=1, which showed significantly lower level of mutual cooperation, all the other experiments’ confidence intervals overlap\.![Refer to caption](https://arxiv.org/html/2608.28977v1/sac_32a_ps_hislen_ldynamics.png)Figure 7:Learning dynamics of partner selection experiment without opponent identity \(l=1,5,10l=1,5,10from the left, respectively\)\.![Refer to caption](https://arxiv.org/html/2608.28977v1/sac_32a_ps_opb_hislen_ldynamics.png)Figure 8:Learning dynamics of partner selection experiment with opponent identity \(l=1,5,10l=1,5,10from the left, respectively\)\.We also investigate the learning dynamics by measuring the proportion of four types of interactions between agents in Iterated Prisoner’s Dilemma every 100 interaction rounds\. Shaded areas show the standard deviation of interactions calculated across 20 runs\. Figure[7](https://arxiv.org/html/2608.28977#S4.F7)shows the learning dynamics of the partner selection experiments without the opponent identity provided\. Atl=1l=1, mutual cooperation spreads rapidly during the initial 50,000 rounds of interaction and maintains its level afterwards\. Forl=5l=5andl=10l=10, we observe learning dynamics similar to those reported in\[[10](https://arxiv.org/html/2608.28977#bib.bib4)\]: \(1\) mutual cooperation initially increases in the population; \(2\) unilateral exploitation of cooperative agents \(i\.e\., sucker and temptation outcomes\) increases, while mutual cooperation decreases; \(3\) mutual defection increases, while unilateral exploitation decreases; and \(4\) mutual cooperation increases again, accompanied by decreases in both unilateral exploitation and mutual defection\. Mutual cooperation reaches its highest level more slowly than forl=1l=1, with the peak occurring at around 100,000 rounds across the experimental runs for bothl=5l=5andl=10l=10\.

In Figure[8](https://arxiv.org/html/2608.28977#S4.F8)we present the learning dynamics of experiments with opponent identity in the state information\. Atl=1l=1, the level of mutual cooperation increases in the initial stage to 0\.4, but then decreases to 0\.2 and gradually continues to decrease\. All the other interactions, such as mutual defection or temptation, gradually increase throughout the experiment\. However, withl=5l=5andl=10l=10, mutual cooperation shows a stable increase after the first quarter of the experiments\. The learning dynamics described in\[[10](https://arxiv.org/html/2608.28977#bib.bib4)\]is also apparent, except that inl=10l=10, the initial surge of mutual cooperation is less significant\.

## 5Discussion

### 5\.1Effect of Graph Topology on the Emergence of Cooperation

We discuss here how differences in the graph structure influenced the strategy of RL\-based agents to play IPD\. For effective discussion, we consider cases wherel=1l=1and opponent identity is excluded from state information\. Because\|𝒜d​i​l\|=2\|\\mathcal\{A\}\_\{dil\}\|=2and\|𝒮d​i​l\|=2\|\\mathcal\{S\}\_\{dil\}\|=2, there are 4 possible strategies determined by the difference of Q\-value of the dilemma\-playing network as in Table[2](https://arxiv.org/html/2608.28977#S5.T2)\.sC,sDs\_\{C\},s\_\{D\}denote the states where the assigned opponent’s previous action was cooperation and defection, respectively\.

Table 2:Possible strategies in experiments wherel=1l=1, opponent identity excluded from state information\.It is well established that, in a single\-shot Prisoner’s Dilemma, defection is the optimal strategy for rational agents\[[41](https://arxiv.org/html/2608.28977#bib.bib6)\]\. Previous studies show that in the iterated Prisoner’s Dilemma, the discount factorγ\\gammaplays a significant role in the emergence of cooperation\[[5](https://arxiv.org/html/2608.28977#bib.bib12),[42](https://arxiv.org/html/2608.28977#bib.bib13),[43](https://arxiv.org/html/2608.28977#bib.bib14)\]\. In two\-player IPD with indefinite length of episodes, when the opponent’s strategy is tit\-for\-tat, the optimal strategy differs based on the value ofγ\\gamma\[[5](https://arxiv.org/html/2608.28977#bib.bib12)\]:

Always cooperate:Gc=R1−γ,Alternate between cooperation and defection:Ga=T\+γ​S1−γ2,Always defect:Gd=T\+γ​P1−γ\.\\displaystyle\\begin\{split\}&\\text\{Always cooperate:\}\\qquad G\_\{c\}=\\frac\{R\}\{1\-\\gamma\},\\\\ &\\text\{Alternate between cooperation and defection:\}\\qquad G\_\{a\}=\\frac\{T\+\\gamma S\}\{1\-\\gamma^\{2\}\},\\\\ &\\text\{Always defect:\}\\qquad G\_\{d\}=T\+\\frac\{\\gamma P\}\{1\-\\gamma\}\.\\end\{split\}\(5\)
Using the payoff values we defined in Section[2\.1](https://arxiv.org/html/2608.28977#S2.SS1)\(T=0\.3,R=0\.2,P=−0\.2,S=−0\.3\)\(T=0\.3,R=0\.2,P=\-0\.2,S=\-0\.3\)and whenγ=0\.99\\gamma=0\.99, values of the returns in each case above becomeGc=20,Ga=0\.151,Gd=−19\.5G\_\{c\}=20,G\_\{a\}=0\.151,G\_\{d\}=\-19\.5\. This shows that it is optimal to always cooperate when facing an agent with tit\-for\-tat strategy\. If we ignore the actions randomly selected byϵ\\epsilon\-greedy exploration, the agents with retaliator strategy in our setting shows the behaviour identical to that of ones with tit\-for\-tat strategy\. Hence, we can say that agents continuously playing against opponents with retaliator strategy will eventually shift to cooperator strategy\. Meanwhile, regardless of the value of discount factor, it is always optimal to defect against opponents with fixed strategy of cooperator, defector, and reverser\.

Figure 9:Interaction patterns in the IPD across different graph structures \(ER, WS and BA from the top, respectively\)\. Each graph shows results of one of the runs in each experiment setting\. Each node represents an agent in the population\. Colours of the nodes represent types of interactions the agent has experienced the most in the last 100 interaction rounds\. Colours of the edges represent types of interactions that have occurred the most between the two nodes each edge is connecting\.In our actual experiments of random opponent assignment, agents are continuously learning from their recent experiences, which leads to change of their strategies\. In addition, because the opponent for each interaction is assigned randomly from each agent’s neighbours𝒩\\mathcal\{N\}, agents are more likely to encounter an opponent with different strategy from that of the previous interaction when there are more neighbouring agents\. This means that the expected number of interaction rounds of continuously playing with opponents with the same strategy gets smaller as\|𝒩\|\|\\mathcal\{N\}\|increases, which has similar effects to decreasing the discount factorγ\\gamma\. The gap of values between the returnsGc,Ga,GdG\_\{c\},G\_\{a\},G\_\{d\}defined above decreases asγ\\gammadecreases from 0\.99, and three values become identical whenγ=0\.2\\gamma=0\.2, which will make agents no longer continuously cooperate with tit\-for\-tat agents\. Thus, we can say that an agent with larger neighbourhood in the graph is more likely to prefer defection\. This prediction is verified from the results of ER and BA graphs, where increasing the value of graph generation parameterpE​Rp\_\{ER\}andmmresults in graphs with denser connection, also enlarging the number of edges in the graph and thus increasing the overall size of neighbourhood of all the agents in the graph\. Figure[9](https://arxiv.org/html/2608.28977#S5.F9)visualises the resulting interaction patterns for one run of each graph type\.

While the analysis based on neighbourhood size can explain the results of ER and BA graphs, it cannot be applied to that of WS graphs\. In our experiments, we use WS graphs with fixed number of edges, only varying the rewiring probabilitypW​Sp\_\{WS\}\. This means that the average size of the neighbourhood stays the same across all 100 runs\. The factor that differs between each generated network is average path length, which decreases aspW​Sp\_\{WS\}increases\. With agents only interacting with others in adjacent nodes and changing its strategy through RL algorithms, the change of dilemma\-playing strategy of an agent affects its neighbours first\. With the same size of population and same value of average node degree, shorter average path length means that it needs less number of interaction rounds for a strategy to be propagated to all the other nodes\. This way, defector strategy is able to propagate faster to the population in graphs with shorter average path lengths\. Average path length also decreases as graph generation parameterspE​R,mp\_\{ER\},mincrease in ER and BA graph, which would further accelerate the proliferation of defective strategy in the population\.

### 5\.2Effect of Opponent Information on the Emergence of Cooperation

We now discuss the effect of different formats of opponent information on the experiments with partner selection\. In the experiments conducted by\[[10](https://arxiv.org/html/2608.28977#bib.bib4)\], agents were given action histories of opponents withl=1l=1, while selection module was implicitly aware of opponent identity\. Partner selection played a crucial role in the emergence of cooperation, by enabling agents with cooperative strategies to choose other cooperators to benefit from mutual cooperation\. Others with defective strategies also selected agents with cooperative history but were often retaliated against, which resulted in their shift to cooperative strategies as analysed in Section[5\.1](https://arxiv.org/html/2608.28977#S5.SS1)\.

We first discuss the results of experiments without opponent identity provided\. In these cases, partner selection module calculates Q\-value estimates purely from neighbours’ action histories\. Whenl=1l=1, action history only shows the action of the previous interaction round\. Assuming that the action was not chosen by exploration, an action history with actionCCimplies that the assigned opponent’s strategy is not defector\. This means that agents with defective strategies cannot distinguish cooperators from retaliators, not guaranteeing their benefits from unilaterally exploiting the opponent\. It results in a decrease of defective actions, while the proportion of mutual cooperation surges\. Mutual cooperation becomes prevalent in cases withl=5,10l=5,10as well, but more slowly\. One of the main reasons of delayed convergence is that longer action history enabled partner selection modules to target cooperators while avoiding selecting retaliators\. Because retaliators change their actions based on the action history of their opponents, their own history is more likely to contain defection\. The action history of cooperators mostly consists of cooperation, unless their strategies had changed recently or defection was selected by exploration\. Because agents with defective strategies can select another cooperator as a victim when their previous target changes its behaviour, it takes more time for retaliating behaviour to suppress exploiting behaviour\. The corresponding interaction patterns are visualised in Figure[10](https://arxiv.org/html/2608.28977#S5.F10)\. This is supported by Figure[11](https://arxiv.org/html/2608.28977#S5.F11)and Figure[12](https://arxiv.org/html/2608.28977#S5.F12), which show the inequality \(Gini coefficient\) dynamics between agents\. The number of interactions and the proportion of different types of interactions in 100 interaction rounds are measured\. Among experiments where identity information is not included, the inequality of the number of interactions, mutual cooperation and unilateral exploitation reaches its lowest level more slowly forl=5,10l=5,10than forl=1l=1\. The steady increase of mutual cooperation is delayed until most of the agents with cooperative action history changes their behaviour to retaliator strategy, being ready to suppress the defective behaviour in the population\.

Figure 10:Interaction patterns in the IPD with partner selection without identity \(l=1,5,10l=1,5,10from the top, respectively\)\. Each graph shows results of one of the 20 runs in each experiment setting\. Edges have directions as each agent can select their interaction partners\. Agents that mainly cooperate playing IPD \(blue, purple nodes\) are being selected from other agents, while agents that mainly exploit others \(green nodes\) are not being selected\.Figure 11:Dynamics of inequality \(Gini coefficient\) of partner selection between agents per 100 interaction rounds\. Shaded areas show 95% confidence interval across 20 runs\.Figure 12:Dynamics of inequality \(Gini coefficient\) of proportion of different types of interactions\.pm​cp\_\{mc\}\(upper left\),pu​ep\_\{ue\}\(upper right\) andpm​dp\_\{md\}\(bottom left\) denote the proportion of mutual cooperation, unilateral exploitation \(suckerdefined in Section[2\.1](https://arxiv.org/html/2608.28977#S2.SS1)\) and mutual defection in 100 interaction rounds respectively\. Shaded areas show 95% confidence interval across 20 runs\.Figure 13:IPD interaction patterns with partner selection and binary\-encoded identity \(l=1,5,10l=1,5,10from the top, respectively\)\. Each graph shows results of one of the 20 runs in each experiment setting\. When the length of the action history is 1, partner selections are highly concentrated\. This concentration is alleviated as the history length gets longer, enabling more mutual cooperation to emerge in the population as experiment proceeds\.In the experiments where opponent identity is included in state information, the overall dynamics differ most significantly from experiments without identity information withl=1l=1\. Mutual cooperation is not prevalent even with the presence of partner selection mechanism\. As both dilemma\-playing network and partner selection module share the same format of state vector, dilemma\-playing network chooses its action while also considering opponent identity\. Each agent can learn to have different behaviours against opponents with different identity\. This structure favours defectors over cooperators\. Defective agents are not preferred during partner selection, making them play with only the agents they selected in an interaction round\. Defectors are only required to learn to defect against their selection target\. On the other hand, cooperative agents are more likely to be selected, encountering several others in an interaction round\. Because they are likely to be exploited by defectors, cooperative agents have to learn to retaliate against all the opponents that are defecting\. This process requires more learning steps than defective agents, because cooperative agents need to adapt their behaviour against multiple agents to avoid exploitation, while defective agents can benefit by learning to defect against a single agent\. This unbalanced burden of learning is caused by introducing identity information and it hinders the emergence of cooperation\. In cases with longer action historiesl=5,10l=5,10, additional information of opponents’ behaviour make agents depend less on opponent identity\. This lessens the concentration of partner selection while promoting the generalisation of agents’ behaviour based on the opponents’ action history, which enables the proliferation of mutual cooperation, as shown in Figure[13](https://arxiv.org/html/2608.28977#S5.F13)\. Figure[11](https://arxiv.org/html/2608.28977#S5.F11)and Figure[12](https://arxiv.org/html/2608.28977#S5.F12)show that introducing identity information increased the inequality of the number of interactions, mutual cooperation and unilateral exploitation, while increasing action history length decreased the inequality\. The variance of the proportion of interactions in Figure[7](https://arxiv.org/html/2608.28977#S4.F7)and Figure[8](https://arxiv.org/html/2608.28977#S4.F8)is generally smaller in experiments with opponent identity than those without, because while the action history of agents continuously change throughout the experiment, identity information stays static and causes less variation in the choice of partner selection modules\.

## 6Conclusion

In this work, we have presented an analysis of the emergence of cooperation in a group of multiple agents interacting in the form of two\-player IPD over a specified network structure\. We have experimented with three different types of graphs, and have showed that in denser graphs and in those characterised by a shorter path length, mutual defection is more likely to spread\. We have provided discount factor\-based analysis to explain the relation between the results and the characteristics of graph structures\. We also have analysed experiments with partner selection mechanism, altering the format of state vectors\. We have considered scenarios composed of different lengths of action history, as well as binary\-encoded identity information of the opponent\. We have shown that longer action history slowed the emergence of mutual cooperation by enabling agents with defective behaviour to target cooperative agents while circumventing retaliators\. We have experimentally demonstrated that providing agents with opponent identity results in lower level of mutual cooperation when insufficient information is provided in action history\. Our work investigated the limited cases of IPD, which is one of the simplest games which does not affect the agents’ location in the given graphs\. We also only experimented with identity information that enables individual identification of the opponents\. We leave the experiments with environments where graph topology can change temporally and, more in general, the study of sequential social dilemma\[[44](https://arxiv.org/html/2608.28977#bib.bib35),[45](https://arxiv.org/html/2608.28977#bib.bib36),[46](https://arxiv.org/html/2608.28977#bib.bib37)\], for future work\. Investigation of learning dynamics of RL\-based agents with different types of identity information, such as group identity, would be another promising topic for future research\.

## Data and code availability

## Acknowledgements

This work was supported by the UK Engineering and Physical Sciences Research Council \(EPSRC\) under grant EP/X028569/1 \(“Satisficing Trust in Human Robot Teams”\)\.

## References

- \[1\]E\. Ferguson\(2015\)Mechanism of altruism approach to blood donor recruitment and retention: a review and future directions\.Transfusion Medicine25\(4\),pp\. 211–226\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p1.1)\.
- \[2\]S\. Z\. Attari, D\. H\. Krantz, and E\. U\. Weber\(2014\)Reasons for cooperation and defection in real\-world social dilemmas\.Judgment and Decision Making9\(4\),pp\. 316–334\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p1.1)\.
- \[3\]K\. Klyver, N\. J\. Lindsay, S\. K\. Kassicieh, and G\. Hancock\(2017\)Altruistic investment decision behavior in early\-stage ventures\.Small Business Economics48\(1\),pp\. 135–152\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p1.1)\.
- \[4\]R\. Axelrod and W\. D\. Hamilton\(1981\)The evolution of cooperation\.Science211\(4489\),pp\. 1390–1396\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p2.1),[§1](https://arxiv.org/html/2608.28977#S1.p6.1),[§2\.1](https://arxiv.org/html/2608.28977#S2.SS1.p1.1)\.
- \[5\]R\. Axelrod\(1984\)The evolution of cooperation\.Basic Books,New York, NY\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p2.1),[§5\.1](https://arxiv.org/html/2608.28977#S5.SS1.p2.1)\.
- \[6\]M\. A\. Nowak and K\. Sigmund\(1992\)Tit for tat in heterogeneous populations\.Nature355\(6357\),pp\. 250–253\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.28977#S2.SS1.p1.1)\.
- \[7\]N\. Masuda and K\. Aihara\(2003\)Spatial prisoner’s dilemma optimally played in small\-world networks\.Physics Letters A313\(1\-2\),pp\. 55–61\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p2.1)\.
- \[8\]S\. B\. Ale, J\. S\. Brown, and A\. T\. Sullivan\(2013\)Evolution of cooperation: combining kin selection and reciprocal altruism into matrix games with social dilemmas\.PLoS ONE8\(5\),pp\. e63761\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.28977#S2.SS1.p2.1)\.
- \[9\]R\. Lowe, Y\. Wu, A\. Tamar, J\. Harb, P\. Abbeel, and I\. Mordatch\(2017\)Multi\-agent actor\-critic for mixed cooperative\-competitive environments\.InAdvances in Neural Information Processing Systems 30 \(NeurIPS 2017\),pp\. 6379–6390\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p3.1)\.
- \[10\]N\. Anastassacos, S\. Hailes, and M\. Musolesi\(2020\)Partner selection for the emergence of cooperation in multi\-agent systems using reinforcement learning\.InProceedings of the 34th AAAI Conference on Artificial Intelligence \(AAAI 2020\),Vol\.34,pp\. 7047–7054\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p3.1),[§1](https://arxiv.org/html/2608.28977#S1.p5.1),[§2\.1](https://arxiv.org/html/2608.28977#S2.SS1.p1.1),[§2\.3](https://arxiv.org/html/2608.28977#S2.SS3.p1.1),[§3\.1](https://arxiv.org/html/2608.28977#S3.SS1.p1.1),[§3\.1](https://arxiv.org/html/2608.28977#S3.SS1.p2.1),[§3\.2](https://arxiv.org/html/2608.28977#S3.SS2.p6.1),[§4\.2](https://arxiv.org/html/2608.28977#S4.SS2.p2.1),[§4\.2](https://arxiv.org/html/2608.28977#S4.SS2.p3.1),[§5\.2](https://arxiv.org/html/2608.28977#S5.SS2.p1.1)\.
- \[11\]R\. Merhej, F\. P\. Santos, F\. S\. Melo, M\. Chetouani, and F\. C\. Santos\(2022\)Cooperation and learning dynamics under risk diversity and financial incentives\.InProceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems \(AAMAS 2022\),pp\. 908–916\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p3.1)\.
- \[12\]A\. Wong, T\. Bäck, A\. V\. Kononova, and A\. Plaat\(2023\)Deep multiagent reinforcement learning: challenges and directions\.Artificial Intelligence Review56\(6\),pp\. 5023–5056\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p3.1)\.
- \[13\]N\. Dasgupta and M\. Musolesi\(2025\)Investigating the impact of direct punishment on the emergence of cooperation in multi\-agent reinforcement learning systems\.Autonomous Agents and Multi\-Agent Systems39\(1\),pp\. 1–37\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p3.1)\.
- \[14\]C\. Leung, P\. Turrini, F\. P\. Santos, and M\. Musolesi\(2026\)Learning to cooperate with minimal observability\.InProceedings of the 40th Annual AAAI Conference on Artificial Intelligence \(AAAI 2026\),Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p3.1)\.
- \[15\]J\. N\. Foerster, R\. Y\. Chen, M\. Al\-Shedivat, S\. Whiteson, P\. Abbeel, and I\. Mordatch\(2018\)Learning with opponent\-learning awareness\.InProceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems \(AAMAS 2018\),pp\. 122–130\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p3.1)\.
- \[16\]S\. Koohborfardhaghighi and E\. Pauwels\(2021\)Reinforcement learning for modeling and capturing the effect of partner selection strategies on the emergence of cooperation\.InEconomics of Grids, Clouds, Systems, and Services: 18th International Conference \(GECON 2021\),Cham,pp\. 52–65\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p3.1)\.
- \[17\]T\. Gu and B\. An\(2023\)A flexi partner selection model for the emergence of cooperation in N\-person social dilemmas\.InDistributed Artificial Intelligence: 4th International Conference \(DAI 2022\),Cham,pp\. 15–28\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p3.1)\.
- \[18\]F\. C\. Santos and J\. M\. Pacheco\(2005\)Scale\-free networks provide a unifying framework for the emergence of cooperation\.Physical Review Letters95\(9\),pp\. 098104\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p4.1)\.
- \[19\]F\. C\. Santos, J\. M\. Pacheco, and T\. Lenaerts\(2006\)Evolutionary dynamics of social dilemmas in structured heterogeneous populations\.Proceedings of the National Academy of Sciences103\(9\),pp\. 3490–3494\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p4.1)\.
- \[20\]F\. C\. Santos, M\. D\. Santos, and J\. M\. Pacheco\(2008\)Social diversity promotes the emergence of cooperation in public goods games\.Nature454\(7201\),pp\. 213–216\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p4.1)\.
- \[21\]G\. Rezaei and M\. Kirley\(2012\)Dynamic social networks facilitate cooperation in the N\-player prisoner’s dilemma\.Physica A: Statistical Mechanics and its Applications391\(23\),pp\. 6199–6211\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p4.1)\.
- \[22\]L\. Barlow and D\. Ashlock\(2013\)The impact of connection topology and agent size on cooperation in the iterated prisoner’s dilemma\.InProceedings of the 9th IEEE Conference on Computational Inteligence in Games \(CIG 2013\),Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p4.1)\.
- \[23\]F\. J\. H\. Hol, P\. Galajda, K\. Nagy, R\. G\. Woolthuis, C\. Dekker, and J\. E\. Keymer\(2013\)Spatial structure facilitates cooperation in a social dilemma: empirical evidence from a bacterial community\.PLoS ONE8\(10\),pp\. e77042\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p4.1)\.
- \[24\]H\. Ishibuchi, T\. Sudo, K\. Hoshino, and Y\. Nojima\(2013\)Evolution of cooperative strategies for iterated prisoner’s dilemma on networks\.InProceedings of the 5th International Conference on Computational Aspects of Social Networks \(CASoN 2013\),pp\. 32–37\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p4.1)\.
- \[25\]G\. Cimini and A\. Sánchez\(2014\)Learning dynamics explains human behaviour in prisoner’s dilemma on networks\.Journal of the Royal Society Interface11\(94\),pp\. 20131186\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p4.1)\.
- \[26\]P\. Liu and J\. Liu\(2017\)Cooperation in the prisoner’s dilemma game on tunable community networks\.Physica A: Statistical Mechanics and its Applications472,pp\. 156–163\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p4.1)\.
- \[27\]J\. Bara, P\. Turrini, and G\. Andrighetto\(2022\)Enabling imitation\-based cooperation in dynamic social networks\.Autonomous Agents and Multi\-Agent Systems36\(2\),pp\. 34\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p4.1)\.
- \[28\]Z\. Song, C\. Shen, and T\. A\. Han\(2026\)Network reciprocity turns cheap talk into a force for cooperation\.Journal of Theoretical Biology617,pp\. 112303\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p4.1)\.
- \[29\]Z\. Yuan, G\. Jiang, S\. Hu, M\. Perc, C\. Chu, and J\. Liu\(2026\)Dynamics of q\-learning in networked stochastic games\.IEEE Transactions on Neural Networks and Learning Systems\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p4.1)\.
- \[30\]R\. Axelrod\(1980\)Effective choice in the prisoner’s dilemma\.Journal of Conflict Resolution24\(1\),pp\. 3–25\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p5.1),[§2\.1](https://arxiv.org/html/2608.28977#S2.SS1.p1.1)\.
- \[31\]F\. Fu, C\. Hauert, M\. A\. Nowak, and L\. Wang\(2008\)Reputation\-based partner choice promotes cooperation in social networks\.Physical Review E78\(2\),pp\. 026117\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p5.1)\.
- \[32\]N\. Anastassacos, J\. García, S\. Hailes, and M\. Musolesi\(2021\)Cooperation and reputation dynamics with reinforcement learning\.InProceedings of the 20th International Conference on Autonomous Agents and Multiagent Systems \(AAMAS 2021\),pp\. 115–123\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p5.1)\.
- \[33\]M\. B\. Brewer and R\. M\. Kramer\(1986\)Choice behavior in social dilemmas: effects of social identity, group size, and decision framing\.Journal of Personality and Social Psychology50\(3\),pp\. 543–549\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p5.1)\.
- \[34\]B\. Simpson\(2006\)Social identity and cooperation in social dilemmas\.Rationality and Society18\(4\),pp\. 443–470\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p5.1)\.
- \[35\]O\. Aksoy\(2019\)Crosscutting circles in a social dilemma: effects of social identity and inequality on cooperation\.Social Science Research82,pp\. 148–163\.Cited by:[§1](https://arxiv.org/html/2608.28977#S1.p5.1)\.
- \[36\]V\. Mnih, K\. Kavukcuoglu, D\. Silver, A\. A\. Rusu, J\. Veness, M\. G\. Bellemare, A\. Graves, M\. Riedmiller, A\. K\. Fidjeland, G\. Ostrovski,et al\.\(2015\)Human\-level control through deep reinforcement learning\.Nature518\(7540\),pp\. 529–533\.Cited by:[§2\.1](https://arxiv.org/html/2608.28977#S2.SS1.p3.1),[§3\.1](https://arxiv.org/html/2608.28977#S3.SS1.p1.1),[§3\.1](https://arxiv.org/html/2608.28977#S3.SS1.p2.1)\.
- \[37\]P\. Erdős and A\. Rényi\(1960\)On the evolution of random graphs\.Publication of the Mathematical Institute of the Hungarian Academy of Sciences5,pp\. 17–61\.Cited by:[§2\.2](https://arxiv.org/html/2608.28977#S2.SS2.p1.1)\.
- \[38\]D\. J\. Watts and S\. H\. Strogatz\(1998\)Collective dynamics of ‘small\-world’ networks\.Nature393\(6684\),pp\. 440–442\.Cited by:[§2\.2](https://arxiv.org/html/2608.28977#S2.SS2.p1.1)\.
- \[39\]A\. Barabási and R\. Albert\(1999\)Emergence of scaling in random networks\.Science286\(5439\),pp\. 509–512\.Cited by:[§2\.2](https://arxiv.org/html/2608.28977#S2.SS2.p1.1)\.
- \[40\]T\. Haarnoja, A\. Zhou, K\. Hartikainen, G\. Tucker, S\. Ha, J\. Tan, V\. Kumar, H\. Zhu, A\. Gupta, P\. Abbeel, and S\. Levine\(2018\)Soft actor\-critic algorithms and applications\.arXiv preprint arXiv:1812\.05905\.Cited by:[§3\.2](https://arxiv.org/html/2608.28977#S3.SS2.p2.1)\.
- \[41\]T\. W\. Sandholm and R\. H\. Crites\(1996\)Multiagent reinforcement learning in the iterated prisoner’s dilemma\.BioSystems37\(1\-2\),pp\. 147–166\.Cited by:[§5\.1](https://arxiv.org/html/2608.28977#S5.SS1.p2.1)\.
- \[42\]D\. M\. Kreps\(1990\)A course in microeconomic theory\.Princeton University Press,Princeton, NJ\.Cited by:[§5\.1](https://arxiv.org/html/2608.28977#S5.SS1.p2.1)\.
- \[43\]D\. Fudenberg and J\. Tirole\(1991\)Game theory\.MIT Press,Cambridge, MA\.Cited by:[§5\.1](https://arxiv.org/html/2608.28977#S5.SS1.p2.1)\.
- \[44\]J\. Z\. Leibo, V\. Zambaldi, M\. Lanctot, J\. Marecki, and T\. Graepel\(2017\)Multi\-agent reinforcement learning in sequential social dilemmas\.InProceedings of the 16th International Conference on Autonomous Agents and Multiagent Systems \(AAMAS 2017\),pp\. 464–473\.Cited by:[§6](https://arxiv.org/html/2608.28977#S6.p1.1)\.
- \[45\]N\. Jaques, A\. Lazaridou, E\. Hughes, C\. Gulcehre, P\. A\. Ortega, D\. Strouse, J\. Z\. Leibo, and N\. de Freitas\(2019\)Social influence as intrinsic motivation for multi\-agent deep reinforcement learning\.InProceedings of the 36th International Conference on Machine Learning \(ICML 2019\),pp\. 3040–3049\.Cited by:[§6](https://arxiv.org/html/2608.28977#S6.p1.1)\.
- \[46\]E\. Hughes, J\. Z\. Leibo, M\. G\. Phillips, K\. Tuyls, E\. A\. Duéñez\-Guzmán, A\. García Castañeda, I\. Dunning, T\. Zhu, K\. R\. McKee, R\. Koster, H\. Roff, and T\. Graepel\(2018\)Inequity aversion improves cooperation in intertemporal social dilemmas\.InAdvances in Neural Information Processing Systems 31 \(NeurIPS 2018\),pp\. 3330–3340\.Cited by:[§6](https://arxiv.org/html/2608.28977#S6.p1.1)\.