Calibrating Artificial Guilt: Neurally Grounded Reward Shaping for Prosocial Multi-Agent Reinforcement Learning

# Calibrating Artificial Guilt: Neurally Grounded Reward Shaping for Prosocial Multi-Agent Reinforcement Learning
Source: [https://arxiv.org/html/2608.04663](https://arxiv.org/html/2608.04663)
11institutetext:Mahatma Gandhi International School, Ahmedabad, India
11email:aadityamehta\.work@gmail\.com22institutetext:Indian Institute of Technology, Gandhinagar, India
22email:arya\.shah@iitgn\.ac\.in###### Abstract

Cooperative multi\-agent reinforcement learning often adds social terms to individual rewards, yet the scale of those terms is usually chosen by hand\. We ask whether a guilt signal can instead be calibrated from human neural and behavioural data and transferred to artificial agents\. Using the public SoDec responsibility fMRI dataset \(40 participants\), we fit a subject\-fixed\-effects regression of momentary\-happiness changes on outcome\-type counts and recover a guilt weight as the Partner\-negative minus Social\-negative contrast \(w^=1\.118\\hat\{w\}=1\.118, Cohen’sd=0\.214d=0\.214\)\. We embed this weight in a two\-agent Social Lottery environment and train independent Proximal Policy Optimization actor\-critics under four shaping regimes: neurally calibrated, uniform constant, zero \(selfish\), and a unit\-coefficient oracle\. Across 1,000 evaluation episodes per condition, the calibrated agents track the human Social safe\-choice rate most closely \(0\.4590\.459vs\. human0\.4840\.484;KL=0\.0012\\mathrm\{KL\}=0\.0012\), while the other three conditions deviate by one to three orders of magnitude in KL\. Human neurobehavioural priors can therefore act as quantitative constraints on prosocial reward shaping\.

## 1Introduction

Mixed\-motive multi\-agent reinforcement learning \(MARL\) is a common setting for studying cooperation among learning agents\[[27](https://arxiv.org/html/2608.04663#bib.bib1),[11](https://arxiv.org/html/2608.04663#bib.bib2)\]\. Independent self\-interested learners often defect in social dilemmas, so researchers augment individual rewards with social signals such as inequity aversion\[[22](https://arxiv.org/html/2608.04663#bib.bib3),[15](https://arxiv.org/html/2608.04663#bib.bib4)\], social influence\[[23](https://arxiv.org/html/2608.04663#bib.bib5)\], or diverse social preferences\[[42](https://arxiv.org/html/2608.04663#bib.bib6)\]\. Potential\-based reward shaping\[[34](https://arxiv.org/html/2608.04663#bib.bib7)\]gives a formal route for such bias without changing the optimal policy class under stated conditions\.

Two design choices usually remain heuristic\. The shaping magnitude is tuned by hand or by sweep, and the cooperative reference is often a normative ideal such as full equality rather than measured human choice\. Agents can then satisfy a designer criterion while missing the distribution of real social decisions\. Human social cognition offers a sharper alternative: momentary happiness in interactive tasks is predictable from simple outcome equations\[[43](https://arxiv.org/html/2608.04663#bib.bib8)\], and interpersonal guilt reliably engages the anterior insula\[[7](https://arxiv.org/html/2608.04663#bib.bib9),[55](https://arxiv.org/html/2608.04663#bib.bib10)\]\.

We calibrate an artificial guilt penalty from the SoDec responsibility dataset \(OpenNeuro ds005588\)\[[45](https://arxiv.org/html/2608.04663#bib.bib14),[17](https://arxiv.org/html/2608.04663#bib.bib11)\]\. Participants choose between a safe payoff and a risky lottery, either for themselves alone \(Solo\) or for themselves and a partner \(Social\)\. A published imaging analysis localises the interpersonal\-guilt contrast to the left anterior insula\. Our pipeline turns the corresponding behavioural contrast into a scalar weightw^\\hat\{w\}and injects it into independent Proximal Policy Optimization \(PPO\) agents\[[44](https://arxiv.org/html/2608.04663#bib.bib12)\]in a matching Social Lottery environment \(Fig\.[1](https://arxiv.org/html/2608.04663#S1.F1)\)\.

![Refer to caption](https://arxiv.org/html/2608.04663v1/x1.png)Figure 1:Teaching artificial agents a human\-like sense of guilt\. People feel worse when their choice harms a partner; we turn that measured feeling into a learning rule for two cooperating agents\. Agents trained with the human\-scaled rule choose more carefully for their partners and match human Social choices more closely than agents trained with no guilt, a fixed penalty, or a full shortfall penalty\.The transfer needs no new pairwise preference labels and no inverse reinforcement learning\[[10](https://arxiv.org/html/2608.04663#bib.bib15)\]\. After training, the neurally calibrated agents reach a safe rate of0\.4590\.459against a human Social baseline of0\.4840\.484, withKL​\(Phuman∥Pagent\)=0\.0012\\mathrm\{KL\}\(P\_\{\\mathrm\{human\}\}\\\|P\_\{\\mathrm\{agent\}\}\)=0\.0012\. That divergence is an order of magnitude smaller than a unit\-coefficient oracle penalty and hundreds of times smaller than a selfish baseline\. The calibrated coefficient \(w^=1\.118\\hat\{w\}=1\.118\) is slightly larger than the oracle coefficient \(w=1w=1\), yet produces lower cumulative penalty mass because agents choose the safe option more often\. Magnitude, not only functional form, therefore carries alignment information\.

The main contributions of this work are: \(1\) a calibration pipeline that turns a published anterior\-insula\-indexed behavioural contrast into a single transferable RL hyperparameter without extra human supervision; \(2\) a PettingZoo\[[47](https://arxiv.org/html/2608.04663#bib.bib13)\]Social Lottery environment that reproduces the responsibility paradigm with lottery parameters sampled from ds005588; and \(3\) evidence that the scale of a human moral signal improves behavioural alignment in MARL relative to designer\-chosen constants\.

## 2Related Work

We situate the work across five strands: cooperative MARL, reward shaping, prosocial social signals, computational and neural accounts of guilt, and alignment from human priors\.

##### Cooperative and mixed\-motive MARL\.

Sequential social dilemmas cast cooperation as a Markov game in which independent learners often converge to defection\[[27](https://arxiv.org/html/2608.04663#bib.bib1),[38](https://arxiv.org/html/2608.04663#bib.bib16)\], a failure mode that cooperative AI agendas also emphasise\[[11](https://arxiv.org/html/2608.04663#bib.bib2)\]\. Deep MARL addresses coordination with centralised critics\[[30](https://arxiv.org/html/2608.04663#bib.bib47)\], opponent learning awareness\[[16](https://arxiv.org/html/2608.04663#bib.bib48)\], value decomposition\[[46](https://arxiv.org/html/2608.04663#bib.bib17),[41](https://arxiv.org/html/2608.04663#bib.bib18)\], population game\-theoretic solvers\[[26](https://arxiv.org/html/2608.04663#bib.bib19)\], and strong independent PPO baselines\[[54](https://arxiv.org/html/2608.04663#bib.bib20)\]\. Large\-scale multi\-agent systems and surveys further map these ideas under complex dynamics\[[49](https://arxiv.org/html/2608.04663#bib.bib21),[21](https://arxiv.org/html/2608.04663#bib.bib22)\]\. Across this literature, success is usually defined by team return or equilibrium quality, not by match to a measured human choice distribution\.

##### Reward shaping and intrinsic motivation\.

Potential\-based reward shaping biases learning without changing the optimal policy class under stated conditions\[[34](https://arxiv.org/html/2608.04663#bib.bib7),[53](https://arxiv.org/html/2608.04663#bib.bib23)\]\. Intrinsic motivation supplies related additive signals for exploration and skill discovery, including prediction\-error curiosity\[[37](https://arxiv.org/html/2608.04663#bib.bib24)\], random network distillation\[[6](https://arxiv.org/html/2608.04663#bib.bib25)\], and diversity\-driven objectives\[[14](https://arxiv.org/html/2608.04663#bib.bib26)\]\. We adopt the additive shaping template, but treat the coefficient as an empirically estimated social prior rather than an exploration bonus or hand\-chosen constant\.

##### Social and prosocial reward signals\.

A parallel line injects explicitly social terms into MARL rewards: inequity aversion\[[22](https://arxiv.org/html/2608.04663#bib.bib3)\]grounded in Fehr and Schmidt preferences\[[15](https://arxiv.org/html/2608.04663#bib.bib4)\], counterfactual social influence\[[23](https://arxiv.org/html/2608.04663#bib.bib5)\], heterogeneous social preferences\[[42](https://arxiv.org/html/2608.04663#bib.bib6)\], reciprocity\[[12](https://arxiv.org/html/2608.04663#bib.bib27)\], prosocial objectives\[[39](https://arxiv.org/html/2608.04663#bib.bib49),[29](https://arxiv.org/html/2608.04663#bib.bib28)\], evolved intrinsic rewards for altruism\[[51](https://arxiv.org/html/2608.04663#bib.bib29)\], and human\-centred mechanism design with deep RL\[[25](https://arxiv.org/html/2608.04663#bib.bib30)\]\. Related behavioural models outside MARL include Rabin’s fairness equilibrium\[[40](https://arxiv.org/html/2608.04663#bib.bib31)\], Bolton and Ockenfels ERC\[[4](https://arxiv.org/html/2608.04663#bib.bib32)\], and Charness and Rabin social preferences\[[9](https://arxiv.org/html/2608.04663#bib.bib33)\]\. In these systems the functional form of the social term is carefully motivated; the magnitude is still typically swept\. Our setting is orthogonal: fix a guilt\-like form and import its scale from human data\.

##### Guilt, happiness, and the anterior insula\.

Behavioural economics treats guilt as an explicit utility component in psychological games\[[3](https://arxiv.org/html/2608.04663#bib.bib50),[8](https://arxiv.org/html/2608.04663#bib.bib34)\]\. Momentary happiness in interactive tasks is well approximated by linear functions of own and partner outcomes\[[43](https://arxiv.org/html/2608.04663#bib.bib8)\]\. Neuroimaging localises interpersonal guilt and related social\-affective monitoring to the anterior insula and connected prefrontal and temporal circuits\[[7](https://arxiv.org/html/2608.04663#bib.bib9),[55](https://arxiv.org/html/2608.04663#bib.bib10),[50](https://arxiv.org/html/2608.04663#bib.bib35),[24](https://arxiv.org/html/2608.04663#bib.bib36),[56](https://arxiv.org/html/2608.04663#bib.bib37),[33](https://arxiv.org/html/2608.04663#bib.bib38)\]\. The SoDec responsibility dataset and its associated analysis\[[45](https://arxiv.org/html/2608.04663#bib.bib14),[17](https://arxiv.org/html/2608.04663#bib.bib11)\], shared via OpenNeuro under BIDS with fMRIPrep derivatives\[[32](https://arxiv.org/html/2608.04663#bib.bib39),[18](https://arxiv.org/html/2608.04663#bib.bib40),[13](https://arxiv.org/html/2608.04663#bib.bib56)\], provide a public behavioural and neural handle on that contrast\. We use the behavioural events to estimate a transferable scalar and do not re\-analyse BOLD maps here\.

##### Alignment from human priors\.

Value alignment typically relies on inverse RL\[[35](https://arxiv.org/html/2608.04663#bib.bib51),[1](https://arxiv.org/html/2608.04663#bib.bib41)\], cooperative IRL\[[19](https://arxiv.org/html/2608.04663#bib.bib42)\], preference comparisons\[[10](https://arxiv.org/html/2608.04663#bib.bib15)\], scalable reward modelling\[[28](https://arxiv.org/html/2608.04663#bib.bib43)\], RLHF\[[36](https://arxiv.org/html/2608.04663#bib.bib52)\], or constitutional AI feedback\[[2](https://arxiv.org/html/2608.04663#bib.bib44)\]\. A complementary programme argues for structural priors drawn from neuroscience\[[20](https://arxiv.org/html/2608.04663#bib.bib53),[5](https://arxiv.org/html/2608.04663#bib.bib54),[31](https://arxiv.org/html/2608.04663#bib.bib45),[52](https://arxiv.org/html/2608.04663#bib.bib46)\]\. Our differentiation is narrow: we calibrate one social shaping coefficient from a published neurobehavioural contrast, instantiate it in a PettingZoo Social Lottery environment\[[47](https://arxiv.org/html/2608.04663#bib.bib13)\]with independent PPO learners\[[44](https://arxiv.org/html/2608.04663#bib.bib12)\], and score agents against the same human task distribution without collecting new preference labels\.

## 3Methodology

The pipeline has three stages: extract a guilt weight from the SoDec responsibility dataset, embed that weight in a two\-agent Social Lottery environment, and train independent PPO actor\-critics under four shaping regimes\.

### 3\.1Dataset

We use the SoDec responsibility dataset \(OpenNeuro ds005588\)\[[45](https://arxiv.org/html/2608.04663#bib.bib14),[17](https://arxiv.org/html/2608.04663#bib.bib11)\], which contains forty participants \(ages 22 to 50\) scanned across two runs of a responsibility task \(TR=2\.5\\mathrm\{TR\}=2\.5s\)\. On each trial, participants chose between a deterministic safe payoff and a 50/50 risky lottery under Solo \(self only\), Social \(self and partner\), or Partner \(partner chooses for both\) conditions\. Risky outcomes were drawn independently for the two players, and momentary happiness was rated on a 0 to 100 scale every two to four trials\. The original imaging analysis localised the Social\-versus\-Partner happiness\-decrement contrast to the left anterior insula\[[17](https://arxiv.org/html/2608.04663#bib.bib11)\]\. Our pipeline uses only the behavioural event records\.

### 3\.2Extracting the Guilt Weight

For each participant and run we group events into*happiness epochs*: the sequence of outcomes between consecutive happiness ratings\. Within each epoch we count nine outcome types defined by the\{self,other\}×\{pos,neg\}\\\{\\text\{self\},\\text\{other\}\\\}\\times\\\{\\text\{pos\},\\text\{neg\}\\\}valence cross the\{Solo,Social,Partner\}\\\{\\text\{Solo\},\\text\{Social\},\\text\{Partner\}\\\}condition factorial, plus three deterministic safe\-outcome types\. Following Rutledge et al\.\[[43](https://arxiv.org/html/2608.04663#bib.bib8)\], we regress the happiness changeΔ​hi,k\\Delta h\_\{i,k\}for participantiiat epochkkon these counts with subject fixed effects:

Δ​hi,k=α\+∑j=19βj​nj,k\+γi\+εi,k,\\Delta h\_\{i,k\}=\\alpha\+\\sum\_\{j=1\}^\{9\}\\beta\_\{j\}\\,n\_\{j,k\}\+\\gamma\_\{i\}\+\\varepsilon\_\{i,k\},\(1\)wherenj,kn\_\{j,k\}is the count of outcome typejjandγi\\gamma\_\{i\}is a participant intercept \(drop\-one dummy coding\)\. We estimate Eq\. \([1](https://arxiv.org/html/2608.04663#S3.E1)\) by ordinary least squares via the pseudo\-inverse, which remains stable under sparse outcome counts\.

Interpersonal guilt in this paradigm is the extra happiness decrement after a partner’s negative outcome when the participant was responsible \(Social\), relative to when the partner was responsible \(Partner\):

w≔βother​\_​neg​\_​partner−βother​\_​neg​\_​social\.w\\;\\coloneqq\\;\\beta\_\{\\mathrm\{other\\\_neg\\\_partner\}\}\-\\beta\_\{\\mathrm\{other\\\_neg\\\_social\}\}\.\(2\)On ds005588 we obtainw^=1\.118\\hat\{w\}=1\.118with pooled standard errorσw^=7\.39\\sigma\_\{\\hat\{w\}\}=7\.39, Cohen’sd=0\.214d=0\.214, and one\-sidedp=0\.44p=0\.44\. The point estimate has the predicted sign with a small\-to\-moderate standardised effect, but the behavioural sample alone does not reach classical significance\. We therefore treatw^\\hat\{w\}as a prior on penalty magnitude, not as a stand\-alone hypothesis test\. Figure[2](https://arxiv.org/html/2608.04663#S3.F2)shows the fitted coefficients and the contrast distribution\.

![Refer to caption](https://arxiv.org/html/2608.04663v1/x2.png)\(a\)Coefficientsβ^j\\hat\{\\beta\}\_\{j\}\(±1\\pm 1SE\)\.
![Refer to caption](https://arxiv.org/html/2608.04663v1/x3.png)\(b\)Δ​h\\Delta hafter one partner\-negative outcome\.

Figure 2:Extracting the neural guilt weight from ds005588\.
### 3\.3Social Lottery Environment

We cast the paradigm as a two\-agent Markov gameℳ=⟨𝒩,𝒮,\{𝒜i\},P,\{Ri\},T⟩\\mathcal\{M\}=\\langle\\mathcal\{N\},\\mathcal\{S\},\\\{\\mathcal\{A\}\_\{i\}\\\},P,\\\{R\_\{i\}\\\},T\\rangleimplemented as a PettingZoo ParallelEnv\[[47](https://arxiv.org/html/2608.04663#bib.bib13)\]on the Gymnasium API\[[48](https://arxiv.org/html/2608.04663#bib.bib55)\]\. Agents𝒩=\{0,1\}\\mathcal\{N\}=\\\{0,1\\\}alternate as*decider*acrossT=20T=20rounds per episode\. Both observe the same four\-dimensional vector

ot=\(vsafe/c,vhigh/c,vlow/c,1​\{decider\}\)⊤,o\_\{t\}=\\bigl\(v\_\{\\mathrm\{safe\}\}/c,\\;v\_\{\\mathrm\{high\}\}/c,\\;v\_\{\\mathrm\{low\}\}/c,\\;\\mathbb\{1\}\\\{\\mathrm\{decider\}\\\}\\bigr\)^\{\\\!\\top\},\(3\)with normaliserc=30c=30\. Actions are binary,𝒜i=\{Safe,Risky\}\\mathcal\{A\}\_\{i\}=\\\{\\textsc\{Safe\},\\textsc\{Risky\}\\\}; only the decider’s action is consequential\. We samplevsafev\_\{\\mathrm\{safe\}\}and the risky expected value from the empirical marginals in ds005588 \(Fig\.[3](https://arxiv.org/html/2608.04663#S3.F3)\) and draw the risky half\-spread from\|𝒩​\(15,5\)\|\|\\mathcal\{N\}\(15,5\)\|\.

![Refer to caption](https://arxiv.org/html/2608.04663v1/x4.png)Figure 3:Empirical safe\-option and risky expected\-value distributions in ds005588; environment lotteries are sampled from these\.Letrd,rpr\_\{d\},r\_\{p\}be round payoffs to decider and partner, and letEV=\(vhigh\+vlow\)/2\\mathrm\{EV\}=\(v\_\{\\mathrm\{high\}\}\+v\_\{\\mathrm\{low\}\}\)/2\. The partner shortfall after a risky choice isΔp=max⁡\(0,EV−rp\)\\Delta\_\{p\}=\\max\(0,\\mathrm\{EV\}\-r\_\{p\}\)\. Shaped rewards are

Rd​\(s,a\)\\displaystyle R\_\{d\}\(s,a\)=1c​\(rd−w​Δp​1​\{a=Risky\}\),\\displaystyle=\\tfrac\{1\}\{c\}\\bigl\(r\_\{d\}\-w\\,\\Delta\_\{p\}\\,\\mathbb\{1\}\\\{a=\\textsc\{Risky\}\\\}\\bigr\),\(4\)Rp​\(s,a\)\\displaystyle R\_\{p\}\(s,a\)=1c​rp\.\\displaystyle=\\tfrac\{1\}\{c\}\\,r\_\{p\}\.\(5\)The penalty applies only to the decider, and only when a risky action yields a low partner outcome\. Table[1](https://arxiv.org/html/2608.04663#S3.T1)lists the four values ofww\.

Table 1:Reward\-shaping conditions\. NeuroGuilt uses the fitted contrast; Oracle applies a unit coefficient on the same shortfall\.
### 3\.4Multi\-Agent PPO Training

We train two independent PPO actor\-critics\[[44](https://arxiv.org/html/2608.04663#bib.bib12)\], one per agent\. Each network is a two\-layer MLP trunk of6464tanh\\tanhunits with linear policy and value heads\. With GAE advantagesA^t\\hat\{A\}\_\{t\}and returnsR^t\\hat\{R\}\_\{t\}, the clipped objective is

ℒPPO​\(θ\)\\displaystyle\\mathcal\{L\}^\{\\mathrm\{PPO\}\}\(\\theta\)=𝔼t​\[min⁡\(ρt​A^t,clip​\(ρt,1−ϵ,1\+ϵ\)​A^t\)\]\\displaystyle=\\mathbb\{E\}\_\{t\}\\Bigl\[\\min\\bigl\(\\rho\_\{t\}\\hat\{A\}\_\{t\},\\,\\mathrm\{clip\}\(\\rho\_\{t\},1\-\\epsilon,1\+\\epsilon\)\\hat\{A\}\_\{t\}\\bigr\)\\Bigr\]\(6\)−cv𝔼t\[\(Vθ\(st\)−R^t\)2\]\+ce𝔼t\[ℋ\(πθ\(⋅\|st\)\)\],\\displaystyle\\quad\-c\_\{v\}\\,\\mathbb\{E\}\_\{t\}\\bigl\[\(V\_\{\\theta\}\(s\_\{t\}\)\-\\hat\{R\}\_\{t\}\)^\{2\}\\bigr\]\+c\_\{e\}\\,\\mathbb\{E\}\_\{t\}\\bigl\[\\mathcal\{H\}\(\\pi\_\{\\theta\}\(\\cdot\|s\_\{t\}\)\)\\bigr\],whereρt=πθ​\(at\|st\)/πθold​\(at\|st\)\\rho\_\{t\}=\\pi\_\{\\theta\}\(a\_\{t\}\|s\_\{t\}\)/\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\_\{t\}\|s\_\{t\}\)\. Hyperparameters \(Table[2](https://arxiv.org/html/2608.04663#S3.T2)\) were chosen once onZeroand held fixed across conditions so that differences isolateww\. Each condition trains for20,00020\{,\}000episodes with seed4242and is evaluated greedily on1,0001\{,\}000episodes with seed42\+99942\{\+\}999, so training and evaluation draw on disjoint stochastic streams\. End\-to\-end training takes on the order of one hour per condition on a single CPU\. We intentionally avoid condition\-specific learning\-rate or entropy retuning: any change in safe rate or KL is then attributable towwrather than to optimiser asymmetry\.

Table 2:Shared PPO hyperparameters for all four conditions\.

## 4Experimental Design

After training, each condition is evaluated for1,0001\{,\}000greedy episodes under identical environment dynamics and the shared hyperparameters of Table[2](https://arxiv.org/html/2608.04663#S3.T2)\. The design targets three questions: \(RQ1\) Does a neurally calibrated guilt penalty change learned policies relative to a selfish baseline? \(RQ2\) How doesw^=1\.118\\hat\{w\}=1\.118compare with designer coefficientsw∈\{0\.5,1\}w\\in\\\{0\.5,1\\\}? \(RQ3\) Does the resulting choice distribution align more closely with human Social behaviour than the baselines?

We report social welfare \(SW; episode\-summed return\), safe rate \(SR; fraction of decider rounds choosing safe\), reward inequality \(IE; absolute difference in agent returns\), and guilt\-penalty mass \(GP\)\. For RQ3 we compute KL divergence on the binary safe/risky distribution against the human Social baselinePH,Social=0\.484P\_\{H,\\mathrm\{Social\}\}=0\.484from ds005588\. Fixed seeds keep the comparison focused on the shaped reward in Eq\. \([4](https://arxiv.org/html/2608.04663#S3.E4)\)\.

## 5Results

We organise findings around the three research questions in Sec\.[4](https://arxiv.org/html/2608.04663#S4)\.

##### Training dynamics and trade\-offs \(RQ1, RQ2\)\.

Figure[4](https://arxiv.org/html/2608.04663#S5.F4)a shows social welfare and prosocial rate over20,00020\{,\}000training episodes\. Welfare curves separate early:Zeroclimbs highest because it never penalises risky play\. Safe\-rate trajectories separate more slowly\.ZeroandUniformsettle at low safe rates, whileOracleandNeuroGuiltplateau near0\.400\.40and0\.460\.46\. Table[3](https://arxiv.org/html/2608.04663#S5.T3)and Fig\.[4](https://arxiv.org/html/2608.04663#S5.F4)b report steady\-state metrics over1,0001\{,\}000evaluation episodes\. Most metrics move with effective guilt exposure, with one notable pattern:NeuroGuiltrecords lower reward inequality \(1\.5541\.554\) thanOracle\(1\.7761\.776\) and lower cumulative guilt mass \(2\.8632\.863vs\.3\.1053\.105\), even though its coefficient is larger \(1\.1181\.118vs\.11\)\. Safer policies trigger the penalty less often, so cumulative mass is not a monotone readout ofww\.

![Refer to caption](https://arxiv.org/html/2608.04663v1/x5.png)\(a\)Training curves\.
![Refer to caption](https://arxiv.org/html/2608.04663v1/x6.png)\(b\)Steady\-state outcomes\.

Figure 4:Training dynamics and group\-level trade\-offs across conditions\.Table 3:Steady\-state metrics over1,0001\{,\}000greedy episodes \(mean±\\pmSD where applicable\)\. SW: social welfare; SR: safe rate; IE: inequality; GP: guilt\-penalty mass\.
##### Alignment with the human distribution \(RQ3\)\.

The KL column of Table[3](https://arxiv.org/html/2608.04663#S5.T3)is the headline comparison\.NeuroGuilt’s safe rate \(0\.4590\.459\) lies within0\.0250\.025of the human Social rate \(0\.4840\.484\)\. In KL terms, its divergence is12\.8×12\.8\\timessmaller thanOracle’s and393×393\\timessmaller thanZero’s \(Fig\.[6](https://arxiv.org/html/2608.04663#S5.F6)\)\. The calibrated magnitude is therefore closer to human choice than either a weaker constant or a unit\-coefficient shortfall penalty\. Per\-episode safe\-rate distributions \(Fig\.[6](https://arxiv.org/html/2608.04663#S5.F6)\) show the same pattern:OracleandNeuroGuiltretain wider support that resembles human Social variability, whereas the low\-guilt baselines collapse toward near\-deterministic risky play\.

![Refer to caption](https://arxiv.org/html/2608.04663v1/x7.png)Figure 5:Human Social safe rate \(dashed\) versus agent safe rates under the four shaping conditions\. Annotated values areKL​\(Phuman∥Pagent\)\\mathrm\{KL\}\(P\_\{\\mathrm\{human\}\}\\\|P\_\{\\mathrm\{agent\}\}\)\.
![Refer to caption](https://arxiv.org/html/2608.04663v1/x8.png)Figure 6:Per\-episode prosocial \(safe\) choice\-rate distributions\.

## 6Discussion

The central empirical claim is that a single scalar taken from a public neurobehavioural contrast can constrain prosocial MARL more tightly than common designer defaults\. Relative to the promises in Sec\.[1](https://arxiv.org/html/2608.04663#S1), the results deliver on behavioural alignment:NeuroGuiltrecovers a human\-like safe rate and a KL divergence in the10−310^\{\-3\}regime without new preference labels\. They also qualify a simple “stronger penalty is better” intuition\.Oracleusesw=1w=1on the same partner shortfall;NeuroGuiltusesw^=1\.118\\hat\{w\}=1\.118\. The calibrated coefficient is not smaller, yet it yields higher safe rate, lower inequality, lower cumulative penalty mass, and much lower KL to humans\. Alignment here tracks the*data\-derived scale*, not a monotone ranking of coefficient size\.

Two limitations matter for interpretation\. First,w^\\hat\{w\}is estimated from behavioural happiness regressions with a non\-significant classical test \(p=0\.44p=0\.44\)\. We treat it as a prior transferred into RL, not as a confirmed neural effect size; subject\-level anterior\-insula betas from the accompanying imaging derivatives remain unused\. Second, the Social Lottery is a short\-horizon abstraction of the scanner task\. It preserves responsibility structure and empirical lottery values, but not delayed credit assignment, communication, or richer mixed\-motive grids used elsewhere in MARL\. Claims should stay inside that scope: magnitude calibration helps when the environment matches the human paradigm closely enough for the safe/risky distribution to be a meaningful target\.

The broader implication is modest but concrete\. Preference\-based alignment and neurobehavioural calibration need not compete\. The former collects new human judgments; the latter reuses measured social priors already published with open datasets\. For moral signals with an established neural and behavioural signature, that reuse can replace at least one layer of hand tuning\.

A practical reading for MARL design follows\. When a social penalty has a clear human operationalisation, sweep less and measure more: fit the scale on the same paradigm you evaluate against, hold the optimiser fixed, and report divergence to the human distribution rather than only welfare or cooperation rate\. Welfare alone would have preferredZeroin our tables; the alignment metrics reverse that ranking\. That separation is the reason to keep human choice as an explicit target\.

## 7Conclusion

This work contributed a calibration path from a published interpersonal\-guilt contrast to a transferable reward weight for multi\-agent reinforcement learning, a Social Lottery environment matched to the SoDec responsibility paradigm, and evidence that the scale of that weight improves alignment with human Social choices\. Withw^=1\.118\\hat\{w\}=1\.118, independent PPO agents reached a safe rate of0\.4590\.459against a human baseline of0\.4840\.484and a KL divergence of0\.00120\.0012, outperforming selfish, constant, and unit\-coefficient baselines on the same task\. Replacing the behavioural proxy with subject\-specific neural betas, and testing the same magnitude\-transfer idea in longer\-horizon dilemmas, are the natural next steps enabled by the existing dataset\.\{credits\}

#### 7\.0\.1\\discintname

The authors have no competing interests to declare that are relevant to the content of this article\.

## References

- \[1\]P\. Abbeel and A\. Y\. Ng\(2004\)Apprenticeship learning via inverse reinforcement learning\.InProceedings of the Twenty\-First International Conference on Machine Learning,ICML ’04,New York, NY, USA,pp\. 1\.External Links:ISBN 1581138385,[Link](https://doi.org/10.1145/1015330.1015430),[Document](https://dx.doi.org/10.1145/1015330.1015430)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px5.p1.1)\.
- \[2\]Y\. Bai, S\. Kadavath, S\. Kundu, A\. Askell, J\. Kernion, A\. Jones, A\. Chen, A\. Goldie, A\. Mirhoseini, C\. McKinnon, C\. Chen, C\. Olsson, C\. Olah, D\. Hernandez, D\. Drain, D\. Ganguli, D\. Li, E\. Tran\-Johnson, E\. Perez, J\. Kerr, J\. Mueller, J\. Ladish, J\. Landau, K\. Ndousse, K\. Lukošiūtė, L\. Lovitt, M\. Sellitto, N\. Elhage, N\. Schiefer, N\. Mercado, N\. Dassarma, R\. Lasenby, R\. Larson, S\. Ringer, S\. Johnston, S\. Kravec, S\. E\. Showk, S\. Fort, T\. Lanham, T\. Telleen\-Lawton, T\. Conerly, T\. Henighan, T\. Hume, S\. Bowman, Z\. Hatfield\-Dodds, B\. Mann, D\. Amodei, N\. Joseph, S\. McCandlish, T\. B\. Brown, and J\. Kaplan\(2022\)Constitutional ai: harmlessness from ai feedback\.ArXivabs/2212\.08073\.External Links:[Link](https://api.semanticscholar.org/CorpusID:254823489)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px5.p1.1)\.
- \[3\]P\. Battigalli and M\. A\. Dufwenberg\(2007\)Guilt in games\.The American Economic Review97,pp\. 170–176\.External Links:[Link](https://api.semanticscholar.org/CorpusID:144125584)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px4.p1.1)\.
- \[4\]G\. E\. Bolton and A\. Ockenfels\(2000\)ERC: a theory of equity, reciprocity, and competition\.The American Economic Review90\(1\),pp\. 166–193\.External Links:ISSN 00028282,[Link](http://www.jstor.org/stable/117286)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px3.p1.1)\.
- \[5\]M\. Botvinick, J\. X\. Wang, W\. Dabney, K\. J\. Miller, and Z\. Kurth\-Nelson\(2020\)Deep reinforcement learning and its neuroscientific implications\.External Links:2007\.03750,[Link](https://arxiv.org/abs/2007.03750)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px5.p1.1)\.
- \[6\]Y\. Burda, H\. Edwards, A\. J\. Storkey, and O\. Klimov\(2018\)Exploration by random network distillation\.ArXivabs/1810\.12894\.External Links:[Link](https://api.semanticscholar.org/CorpusID:53115163)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px2.p1.1)\.
- \[7\]L\. J\. Chang, A\. Smith, M\. Dufwenberg, and A\. G\. Sanfey\(2011\-05\)Triangulating the neural, psychological, and economic bases of guilt aversion\.Neuron70\(3\),pp\. 560–572\.External Links:ISSN 0896\-6273,[Link](http://dx.doi.org/10.1016/j.neuron.2011.02.056),[Document](https://dx.doi.org/10.1016/j.neuron.2011.02.056)Cited by:[§1](https://arxiv.org/html/2608.04663#S1.p2.1),[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px4.p1.1)\.
- \[8\]G\. Charness and M\. Dufwenberg\(2006\)Promises and partnership\.Econometrica74\(6\),pp\. 1579–1601\.External Links:ISSN 00129682, 14680262,[Link](http://www.jstor.org/stable/4123084)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px4.p1.1)\.
- \[9\]G\. Charness and M\. Rabin\(2002\)Understanding social preferences with simple tests\.The Quarterly Journal of Economics117\(3\),pp\. 817–869\.External Links:ISSN 00335533, 15314650,[Link](http://www.jstor.org/stable/4132490)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px3.p1.1)\.
- \[10\]P\. F\. Christiano, J\. Leike, T\. B\. Brown, M\. Martic, S\. Legg, and D\. Amodei\(2017\)Deep reinforcement learning from human preferences\.InProceedings of the 31st International Conference on Neural Information Processing Systems,NIPS’17,Red Hook, NY, USA,pp\. 4302–4310\.External Links:ISBN 9781510860964Cited by:[§1](https://arxiv.org/html/2608.04663#S1.p4.5),[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px5.p1.1)\.
- \[11\]A\. Dafoe, E\. Hughes, Y\. Bachrach, T\. Collins, K\. R\. McKee, J\. Z\. Leibo, K\. Larson, and T\. Graepel\(2020\)Open problems in cooperative ai\.ArXivabs/2012\.08630\.External Links:[Link](https://api.semanticscholar.org/CorpusID:229220772)Cited by:[§1](https://arxiv.org/html/2608.04663#S1.p1.1),[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px1.p1.1)\.
- \[12\]T\. Eccles, E\. Hughes, J\. Kramár, S\. Wheelwright, and J\. Z\. Leibo\(2019\)Learning reciprocity in complex sequential social dilemmas\.ArXivabs/1903\.08082\.External Links:[Link](https://api.semanticscholar.org/CorpusID:83458767)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px3.p1.1)\.
- \[13\]O\. Esteban, R\. Ciric, K\. Finc, R\. W\. Blair, C\. J\. Markiewicz, C\. A\. S\. Moodie, J\. D\. Kent, M\. Goncalves, E\. Dupre, D\. E\. P\. Gomez, Z\. Ye, T\. Salo, R\. Valabrègue, I\. K\. Amlien, F\. Liem, N\. Jacoby, H\. Stojić, M\. Cieslak, S\. Urchs, Y\. O\. Halchenko, S\. S\. Ghosh, A\. de la Vega, T\. Yarkoni, J\. Wright, W\. H\. Thompson, R\. A\. Poldrack, and K\. J\. Gorgolewski\(2019\)Analysis of task\-based functional mri data preprocessed with fmriprep\.Nature protocols15,pp\. 2186 – 2202\.External Links:[Link](https://api.semanticscholar.org/CorpusID:198238682)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px4.p1.1)\.
- \[14\]B\. Eysenbach, A\. Gupta, J\. Ibarz, and S\. Levine\(2018\)Diversity is all you need: learning skills without a reward function\.ArXivabs/1802\.06070\.External Links:[Link](https://api.semanticscholar.org/CorpusID:3521071)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px2.p1.1)\.
- \[15\]E\. Fehr and K\. M\. Schmidt\(1998\)A theory of fairness, competition and cooperation\.Munich Reprints in Economics4\.External Links:[Link](https://api.semanticscholar.org/CorpusID:2640717)Cited by:[§1](https://arxiv.org/html/2608.04663#S1.p1.1),[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px3.p1.1)\.
- \[16\]J\. N\. Foerster, R\. Y\. Chen, M\. Al\-Shedivat, S\. Whiteson, P\. Abbeel, and I\. Mordatch\(2017\)Learning with opponent\-learning awareness\.InAdaptive Agents and Multi\-Agent Systems,External Links:[Link](https://api.semanticscholar.org/CorpusID:8708073)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px1.p1.1)\.
- \[17\]M\. Gädeke, T\. E\. Willems, O\. S\. Ahmed, B\. Weber, R\. Hurlemann, and J\. Schultz\(2026\-03\)Contributions of insula and superior temporal sulcus to interpersonal guilt and responsibility in social decisions\.eLife14\.External Links:ISSN 2050\-084X,[Link](http://dx.doi.org/10.7554/eLife.105391),[Document](https://dx.doi.org/10.7554/elife.105391)Cited by:[§1](https://arxiv.org/html/2608.04663#S1.p3.1),[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px4.p1.1),[§3\.1](https://arxiv.org/html/2608.04663#S3.SS1.p1.1)\.
- \[18\]K\. J\. Gorgolewski, T\. Auer, V\. D\. Calhoun, R\. C\. Craddock, S\. Das, E\. P\. Duff, G\. Flandin, S\. S\. Ghosh, T\. Glatard, Y\. O\. Halchenko, D\. A\. Handwerker, M\. Hanke, D\. Keator, X\. Li, Z\. Michael, C\. Maumet, B\. N\. Nichols, T\. E\. Nichols, J\. Pellman, J\. Poline, A\. Rokem, G\. Schaefer, V\. Sochat, W\. Triplett, J\. A\. Turner, G\. Varoquaux, and R\. A\. Poldrack\(2016\)The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments\.Scientific Data3\(1\)\.External Links:ISSN 2052\-4463,[Link](http://dx.doi.org/10.1038/sdata.2016.44),[Document](https://dx.doi.org/10.1038/sdata.2016.44)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px4.p1.1)\.
- \[19\]D\. Hadfield\-Menell, S\. J\. Russell, P\. Abbeel, and A\. D\. Dragan\(2016\)Cooperative inverse reinforcement learning\.InNeural Information Processing Systems,External Links:[Link](https://api.semanticscholar.org/CorpusID:6208061)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px5.p1.1)\.
- \[20\]D\. Hassabis, D\. Kumaran, C\. Summerfield, and M\. Botvinick\(2017\)Neuroscience\-inspired artificial intelligence\.Neuron95\(2\),pp\. 245–258\.External Links:ISSN 0896\-6273,[Link](http://dx.doi.org/10.1016/j.neuron.2017.06.011),[Document](https://dx.doi.org/10.1016/j.neuron.2017.06.011)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px5.p1.1)\.
- \[21\]P\. Hernandez\-Leal, B\. Kartal, and M\. E\. Taylor\(2019\-11\)A survey and critique of multiagent deep reinforcement learning\.Autonomous Agents and Multi\-Agent Systems33\(6\),pp\. 750–797\.External Links:ISSN 1387\-2532,[Link](https://doi.org/10.1007/s10458-019-09421-1),[Document](https://dx.doi.org/10.1007/s10458-019-09421-1)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px1.p1.1)\.
- \[22\]E\. Hughes, J\. Z\. Leibo, M\. Phillips, K\. Tuyls, E\. Dueñez\-Guzman, A\. G\. Castañeda, I\. Dunning, T\. Zhu, K\. McKee, R\. Koster, H\. Roff, and T\. Graepel\(2018\)Inequity aversion improves cooperation in intertemporal social dilemmas\.InProceedings of the 32nd International Conference on Neural Information Processing Systems,NIPS’18,Red Hook, NY, USA,pp\. 3330–3340\.Cited by:[§1](https://arxiv.org/html/2608.04663#S1.p1.1),[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px3.p1.1)\.
- \[23\]N\. Jaques, A\. Lazaridou, E\. Hughes, Ç\. Gülçehre, P\. A\. Ortega, D\. Strouse, J\. Z\. Leibo, and N\. de Freitas\(2018\)Social influence as intrinsic motivation for multi\-agent deep reinforcement learning\.InInternational Conference on Machine Learning,External Links:[Link](https://api.semanticscholar.org/CorpusID:60440666)Cited by:[§1](https://arxiv.org/html/2608.04663#S1.p1.1),[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px3.p1.1)\.
- \[24\]L\. Koban, C\. Corradi\-Dell’Acqua, and P\. Vuilleumier\(2013\)Integration of error agency and representation of others’ pain in the anterior insula\.Journal of Cognitive Neuroscience25,pp\. 258–272\.External Links:[Link](https://api.semanticscholar.org/CorpusID:10237549)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px4.p1.1)\.
- \[25\]R\. Koster, J\. Balaguer, A\. Tacchetti, A\. Weinstein, T\. Zhu, O\. Hauser, D\. Williams, L\. Campbell\-Gillingham, P\. Thacker, M\. Botvinick, and C\. Summerfield\(2022\)Human\-centered mechanism design with democratic ai\.External Links:2201\.11441,[Link](https://arxiv.org/abs/2201.11441)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px3.p1.1)\.
- \[26\]M\. Lanctot, V\. Zambaldi, A\. Gruslys, A\. Lazaridou, K\. Tuyls, J\. Pérolat, D\. Silver, and T\. Graepel\(2017\)A unified game\-theoretic approach to multiagent reinforcement learning\.InProceedings of the 31st International Conference on Neural Information Processing Systems,NIPS’17,Red Hook, NY, USA,pp\. 4193–4206\.External Links:ISBN 9781510860964Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px1.p1.1)\.
- \[27\]J\. Z\. Leibo, V\. Zambaldi, M\. Lanctot, J\. Marecki, and T\. Graepel\(2017\)Multi\-agent reinforcement learning in sequential social dilemmas\.InProceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems,AAMAS ’17,Richland, SC,pp\. 464–473\.Cited by:[§1](https://arxiv.org/html/2608.04663#S1.p1.1),[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px1.p1.1)\.
- \[28\]J\. Leike, D\. Krueger, T\. Everitt, M\. Martic, V\. Maini, and S\. Legg\(2018\)Scalable agent alignment via reward modeling: a research direction\.ArXivabs/1811\.07871\.External Links:[Link](https://api.semanticscholar.org/CorpusID:53745764)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px5.p1.1)\.
- \[29\]A\. Lerer and A\. Peysakhovich\(2017\)Maintaining cooperation in complex social dilemmas using deep reinforcement learning\.ArXivabs/1707\.01068\.External Links:[Link](https://api.semanticscholar.org/CorpusID:3231252)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px3.p1.1)\.
- \[30\]R\. Lowe, Y\. Wu, A\. Tamar, J\. Harb, P\. Abbeel, and I\. Mordatch\(2017\)Multi\-agent actor\-critic for mixed cooperative\-competitive environments\.ArXivabs/1706\.02275\.External Links:[Link](https://api.semanticscholar.org/CorpusID:26419660)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px1.p1.1)\.
- \[31\]A\. H\. Marblestone, G\. Wayne, and K\. P\. Kording\(2016\)Toward an integration of deep learning and neuroscience\.Frontiers in Computational Neuroscience10\.External Links:[Link](https://api.semanticscholar.org/CorpusID:1994856)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px5.p1.1)\.
- \[32\]C\. J\. Markiewicz, K\. J\. Gorgolewski, F\. Feingold, R\. Blair, Y\. O\. Halchenko, E\. Miller, N\. Hardcastle, J\. Wexler, O\. Esteban, M\. Goncavles, A\. Jwa, and R\. Poldrack\(2021\-10\)The openneuro resource for sharing of neuroscience data\.eLife10\.External Links:ISSN 2050\-084X,[Link](http://dx.doi.org/10.7554/eLife.71774),[Document](https://dx.doi.org/10.7554/elife.71774)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px4.p1.1)\.
- \[33\]J\. Moll, F\. Krueger, R\. Zahn, M\. Pardini, R\. de Oliveira\-Souza, and J\. H\. Grafman\(2006\)Human fronto–mesolimbic networks guide decisions about charitable donation\.Proceedings of the National Academy of Sciences103,pp\. 15623 – 15628\.External Links:[Link](https://api.semanticscholar.org/CorpusID:9734435)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px4.p1.1)\.
- \[34\]A\. Ng, D\. Harada, and S\. J\. Russell\(1999\)Policy invariance under reward transformations: theory and application to reward shaping\.InInternational Conference on Machine Learning,External Links:[Link](https://api.semanticscholar.org/CorpusID:5730166)Cited by:[§1](https://arxiv.org/html/2608.04663#S1.p1.1),[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px2.p1.1)\.
- \[35\]A\. Y\. Ng and S\. J\. Russell\(2000\)Algorithms for inverse reinforcement learning\.InInternational Conference on Machine Learning,External Links:[Link](https://api.semanticscholar.org/CorpusID:266239314)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px5.p1.1)\.
- \[36\]L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. Christiano, J\. Leike, and R\. Lowe\(2022\)Training language models to follow instructions with human feedback\.InProceedings of the 36th International Conference on Neural Information Processing Systems,NIPS ’22,Red Hook, NY, USA\.External Links:ISBN 9781713871088Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px5.p1.1)\.
- \[37\]D\. Pathak, P\. Agrawal, A\. A\. Efros, and T\. Darrell\(2017\)Curiosity\-driven exploration by self\-supervised prediction\.2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops \(CVPRW\),pp\. 488–489\.External Links:[Link](https://api.semanticscholar.org/CorpusID:20045336)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px2.p1.1)\.
- \[38\]J\. Perolat, J\. Z\. Leibo, V\. Zambaldi, C\. Beattie, K\. Tuyls, and T\. Graepel\(2017\)A multi\-agent reinforcement learning model of common\-pool resource appropriation\.InProceedings of the 31st International Conference on Neural Information Processing Systems,NIPS’17,Red Hook, NY, USA,pp\. 3646–3655\.External Links:ISBN 9781510860964Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px1.p1.1)\.
- \[39\]A\. Peysakhovich and A\. Lerer\(2018\)Prosocial learning agents solve generalized stag hunts better than selfish ones\.InProceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems,AAMAS ’18,Richland, SC,pp\. 2043–2044\.Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px3.p1.1)\.
- \[40\]M\. Rabin\(1993\)Incorporating fairness into game theory and economics\.The American Economic Review83\(5\),pp\. 1281–1302\.External Links:ISSN 00028282,[Link](http://www.jstor.org/stable/2117561)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px3.p1.1)\.
- \[41\]T\. Rashid, M\. Samvelyan, C\. S\. De Witt, G\. Farquhar, J\. Foerster, and S\. Whiteson\(2020\-01\)Monotonic value function factorisation for deep multi\-agent reinforcement learning\.J\. Mach\. Learn\. Res\.21\(1\)\.External Links:ISSN 1532\-4435Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px1.p1.1)\.
- \[42\]M\. Rodriguez\-Soto, M\. Lopez\-Sanchez, and J\. A\. Rodriguez\-Aguilar\(2023\-08\)Multi\-objective reinforcement learning for designing ethical multi\-agent environments\.Neural Comput\. Appl\.37\(31\),pp\. 25619–25644\.External Links:ISSN 0941\-0643,[Link](https://doi.org/10.1007/s00521-023-08898-y),[Document](https://dx.doi.org/10.1007/s00521-023-08898-y)Cited by:[§1](https://arxiv.org/html/2608.04663#S1.p1.1),[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px3.p1.1)\.
- \[43\]R\. B\. Rutledge, N\. Skandali, P\. Dayan, and R\. J\. Dolan\(2014\)A computational and neural model of momentary subjective well\-being\.Proceedings of the National Academy of Sciences111\(33\),pp\. 12252–12257\.External Links:[Document](https://dx.doi.org/10.1073/pnas.1407535111),[Link](https://www.pnas.org/doi/abs/10.1073/pnas.1407535111),https://www\.pnas\.org/doi/pdf/10\.1073/pnas\.1407535111Cited by:[§1](https://arxiv.org/html/2608.04663#S1.p2.1),[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px4.p1.1),[§3\.2](https://arxiv.org/html/2608.04663#S3.SS2.p1.5)\.
- \[44\]J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov\(2017\)Proximal policy optimization algorithms\.ArXivabs/1707\.06347\.External Links:[Link](https://api.semanticscholar.org/CorpusID:28695052)Cited by:[§1](https://arxiv.org/html/2608.04663#S1.p3.1),[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px5.p1.1),[§3\.4](https://arxiv.org/html/2608.04663#S3.SS4.p1.4)\.
- \[45\]Schultz, Johannes, Gädeke, Maria, Willems, Tom, Ahmed, Omar Salah, Hurlemann, René, and Weber, Bernd\(2025\)SoDec \- responsibility fmri experiment dataset\.OpenNeuro\.External Links:[Document](https://dx.doi.org/10.18112/OPENNEURO.DS005588.V1.0.1),[Link](https://openneuro.org/datasets/ds005588/versions/1.0.1)Cited by:[§1](https://arxiv.org/html/2608.04663#S1.p3.1),[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px4.p1.1),[§3\.1](https://arxiv.org/html/2608.04663#S3.SS1.p1.1)\.
- \[46\]P\. Sunehag, G\. Lever, A\. Gruslys, W\. M\. Czarnecki, V\. Zambaldi, M\. Jaderberg, M\. Lanctot, N\. Sonnerat, J\. Z\. Leibo, K\. Tuyls, and T\. Graepel\(2018\)Value\-decomposition networks for cooperative multi\-agent learning based on team reward\.InProceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems,AAMAS ’18,Richland, SC,pp\. 2085–2087\.Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px1.p1.1)\.
- \[47\]J\. K\. Terry, B\. Black, M\. Jayakumar, A\. Hari, L\. Santos, C\. Dieffendahl, N\. L\. Williams, Y\. Lokesh, R\. Sullivan, C\. Horsch, and P\. Ravi\(2021\)PettingZoo: gym for multi\-agent reinforcement learning\.External Links:[Link](https://openreview.net/forum?id=WoLQsYU8aZ)Cited by:[§1](https://arxiv.org/html/2608.04663#S1.p5.1),[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px5.p1.1),[§3\.3](https://arxiv.org/html/2608.04663#S3.SS3.p1.3)\.
- \[48\]M\. Towers, A\. Kwiatkowski, J\. Terry, J\. U\. Balis, G\. D\. Cola, T\. Deleu, M\. Goulão, A\. Kallinteris, M\. Krimmel, A\. KG, R\. Perez\-Vicente, A\. Pierré, S\. Schulhoff, J\. J\. Tai, H\. Tan, and O\. G\. Younis\(2025\)Gymnasium: a standard interface for reinforcement learning environments\.External Links:2407\.17032,[Link](https://arxiv.org/abs/2407.17032)Cited by:[§3\.3](https://arxiv.org/html/2608.04663#S3.SS3.p1.3)\.
- \[49\]O\. Vinyals, I\. Babuschkin, W\. M\. Czarnecki, M\. Mathieu, A\. J\. Dudzik, J\. Chung, D\. Choi, R\. Powell, T\. Ewalds, P\. Georgiev, J\. Oh, D\. Horgan, M\. Kroiss, I\. Danihelka, A\. Huang, L\. Sifre, T\. Cai, J\. P\. Agapiou, M\. Jaderberg, A\. S\. Vezhnevets, R\. Leblond, T\. Pohlen, V\. Dalibard, D\. Budden, Y\. Sulsky, J\. Molloy, T\. L\. Paine, C\. Gulcehre, Z\. Wang, T\. Pfaff, Y\. Wu, R\. Ring, D\. Yogatama, D\. Wünsch, K\. McKinney, O\. Smith, T\. Schaul, T\. P\. Lillicrap, K\. Kavukcuoglu, D\. Hassabis, C\. Apps, and D\. Silver\(2019\)Grandmaster level in starcraft ii using multi\-agent reinforcement learning\.Nature575,pp\. 350 – 354\.External Links:[Link](https://api.semanticscholar.org/CorpusID:204972004)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px1.p1.1)\.
- \[50\]U\. Wagner, K\. N’Diaye, T\. Ethofer, and P\. Vuilleumier\(2011\-03\)Guilt\-specific processing in the prefrontal cortex\.Cerebral Cortex21\(11\),pp\. 2461–2470\.External Links:ISSN 1047\-3211,[Link](http://dx.doi.org/10.1093/cercor/bhr016),[Document](https://dx.doi.org/10.1093/cercor/bhr016)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px4.p1.1)\.
- \[51\]J\. X\. Wang, E\. Hughes, C\. Fernando, W\. M\. Czarnecki, E\. A\. Duenez\-Guzman, and J\. Z\. Leibo\(2019\)Evolving intrinsic motivations for altruistic behavior\.External Links:1811\.05931,[Link](https://arxiv.org/abs/1811.05931)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px3.p1.1)\.
- \[52\]J\. X\. Wang, Z\. Kurth\-Nelson, D\. Kumaran, D\. Tirumala, H\. Soyer, J\. Z\. Leibo, D\. Hassabis, and M\. Botvinick\(2018\-05\)Prefrontal cortex as a meta\-reinforcement learning system\.Nature Neuroscience21\(6\),pp\. 860–868\.External Links:ISSN 1546\-1726,[Link](http://dx.doi.org/10.1038/s41593-018-0147-8),[Document](https://dx.doi.org/10.1038/s41593-018-0147-8)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px5.p1.1)\.
- \[53\]E\. Wiewiora\(2003\-09\)Potential\-based shaping and q\-value initialization are equivalent\.J\. Artif\. Int\. Res\.19\(1\),pp\. 205–208\.External Links:ISSN 1076\-9757Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px2.p1.1)\.
- \[54\]C\. Yu, A\. Velu, E\. Vinitsky, J\. Gao, Y\. Wang, A\. Bayen, and Y\. Wu\(2022\)The surprising effectiveness of ppo in cooperative multi\-agent games\.InProceedings of the 36th International Conference on Neural Information Processing Systems,NIPS ’22,Red Hook, NY, USA\.External Links:ISBN 9781713871088Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px1.p1.1)\.
- \[55\]H\. Yu, J\. Hu, L\. Hu, and X\. Zhou\(2013\-08\)The voice of conscience: neural bases of interpersonal guilt and compensation\.Social Cognitive and Affective Neuroscience9\(8\),pp\. 1150–1158\.External Links:ISSN 1749\-5016,[Link](http://dx.doi.org/10.1093/scan/nst090),[Document](https://dx.doi.org/10.1093/scan/nst090)Cited by:[§1](https://arxiv.org/html/2608.04663#S1.p2.1),[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px4.p1.1)\.
- \[56\]R\. Zahn, J\. Moll, M\. Paiva, G\. Garrido, F\. Krueger, E\. D\. Huey, and J\. Grafman\(2008\-05\)The neural basis of human social values: evidence from functional mri\.Cerebral Cortex19\(2\),pp\. 276–283\.External Links:ISSN 1047\-3211,[Link](http://dx.doi.org/10.1093/cercor/bhn080),[Document](https://dx.doi.org/10.1093/cercor/bhn080)Cited by:[§2](https://arxiv.org/html/2608.04663#S2.SS0.SSS0.Px4.p1.1)\.