When Cooperation Must Change: Semantic Strategy Unlearning in Multi-Agent Reinforcement Learning

arXiv cs.AI Papers

Summary

This paper formalizes semantic coordination-strategy unlearning in cooperative MARL and proposes CARVE, which uses causal coordination attribution, retain-aware subspace updates, counterfactual re-coordination, and adversarial auditing to remove obsolete conventions while preserving useful cooperation, achieving 91.4% reduction in convention elicitation with only 5.9% median utility loss across five benchmarks.

arXiv:2611.17917v1 Announce Type: new Abstract: Cooperative multi-agent reinforcement learning (MARL) supports coordinated decision-making in distributed Web services, multi-robot control, and collaborative games through learned coordination conventions. When deployment requirements change, a previously useful convention may become obsolete or unsafe. For example, revised cache-sharing permissions in Web content delivery may require caches to abandon a learned division of storage and serving roles while maintaining efficient delivery. Teams must therefore remove the specified convention while preserving useful cooperation. This is challenging because the same relational pattern can be reproduced by different agents across different histories. Runtime shielding leaves the underlying coordination preference intact and can impair task progress, while existing reinforcement unlearning and trajectory-deletion methods do not directly target role-invariant conventions. This paper formalizes this problem as semantic coordination-strategy unlearning and proposes CARVE.CARVE uses causal coordination attribution to identify influential interactions and retain-aware subspace updates to limit interference with unrelated behaviors. Counterfactual re-coordination helps agents learn productive alternatives, while adversarial auditing tests whether the target convention can still be recovered. Accepted updates are merged into decentralized actors for direct deployment. Evaluation across five cooperative MARL benchmarks shows that CARVE reduces mean nominal convention-completion and worst-case elicitation rates by 91.4\% and 83.3\%, respectively.The median utility loss is 5.9\%, and measured inference latency matches that of the original policies.
Original Article
View Cached Full Text

Cached at: 11/16/26, 08:32 AM

# When Cooperation Must Change: Semantic Strategy Unlearning in Multi-Agent Reinforcement Learning
Source: [https://arxiv.org/html/2611.17917](https://arxiv.org/html/2611.17917)
\\correspondingauthor

DOI:[XXXXXXX\.XXXXXXX](https://doi.org/XXXXXXX.XXXXXXX)Conference:ACM Web Conference 2027; May 10–14, 2027; Dublin, IrelandISBN:XXX\-X\-XXXX\-XXXX\-X/XXXX/XXWenhan WuAffiliation:School of Computer Science, Wuhan University,Wuhan,Hubei,Chinaemail:[wenhanwu@whu\.edu\.cn](mailto:[email protected])Linghan ChenAffiliation:School of Computer Science and Information Technology, University of Adelaide,Adelaide,South Australia,Australiaemail:[chenlinghan2004@163\.com](mailto:[email protected]),Yudong GaoAffiliation:Department of Computer Science and Engineering, Hong Kong University of Science and Technology,Hong Kong SARemail:[ygaodj@connect\.ust\.hk](mailto:[email protected]),Fan YangAffiliation:School of Electromechanical and Automation, Huaqiao University,Xiameng,Fujian,Chinaemail:[25013088022@stu\.hqu\.edu\.cn](mailto:[email protected]),Honglong ChenAffiliation:College of Control Science and Engineering, China University of Petroleum \(East China\),Qingdao,Shandong,Chinaemail:[chenhl@upc\.edu\.cn](mailto:[email protected]),Kai YinAffiliation:School of Computer Science, Wuhan University,Wuhan,Hubei,Chinaemail:[yinkai@whu\.edu\.cn](mailto:[email protected]),Jiawei JiangAffiliation:School of Computer Science, Central China Normal University,Wuhan,Hubei,Chinaemail:[jiawei\.jiang@ccnu\.edu\.cn](mailto:[email protected])andChuang HuAffiliation:the Institute for Math & AI, Wuhan University,Wuhan,Hubei,Chinaemail:[handc@whu\.edu\.cn](mailto:[email protected])

2027

###### Abstract\.

Cooperative multi\-agent reinforcement learning \(MARL\) supports coordinated decision\-making in distributed Web services, multi\-robot control, and collaborative games through learned coordination conventions\. When deployment requirements change, a previously useful convention may become obsolete or unsafe\. For example, revised cache\-sharing permissions in Web content delivery may require caches to abandon a learned division of storage and serving roles while maintaining efficient delivery\. Teams must therefore remove the specified convention while preserving useful cooperation\. This is challenging because the same relational pattern can be reproduced by different agents across different histories\. Runtime shielding leaves the underlying coordination preference intact and can impair task progress, while existing reinforcement\-unlearning and trajectory\-deletion methods do not directly target role\-invariant conventions\. This paper formalizes this problem as semantic coordination\-strategy unlearning and proposes CARVE\. CARVE uses causal coordination attribution to identify influential interactions and retain\-aware subspace updates to limit interference with unrelated behaviors\. Counterfactual re\-coordination helps agents learn productive alternatives, while adversarial auditing tests whether the target convention can still be recovered\. Accepted updates are merged into decentralized actors for direct deployment\. Evaluation across five cooperative MARL benchmarks shows that CARVE reduces mean nominal convention\-completion and worst\-case elicitation rates by 91\.4% and 83\.3%, respectively\. The median utility loss is 5\.9%, and measured inference latency matches that of the original policies\.

###### Keywords:

Multi\-agent Reinforcement Learning, Machine Unlearning, Semantic Learning, Data Attribution

## 1\.Introduction

Cooperative multi\-agent reinforcement learning \(MARL\) enables agents to pursue shared objectives in distributed Web services, multi\-robot systems, and collaborative games\. Its effectiveness often depends on learned coordination conventions: distributed edge caches coordinate content placement and retrieve objects from neighboring caches to reduce content\-delivery latency\([Chang et al\., 2022](https://arxiv.org/html/2611.17917#bib.bib14)\), allied units focus fire in StarCraft micromanagement\([Samvelyan et al\., 2019](https://arxiv.org/html/2611.17917#bib.bib8);[Ellis et al\., 2023](https://arxiv.org/html/2611.17917#bib.bib10)\), partners divide cooking and delivery roles in Overcooked\([Carroll et al\., 2019](https://arxiv.org/html/2611.17917#bib.bib11);[Ruhdorfer et al\., 2024](https://arxiv.org/html/2611.17917#bib.bib12)\), and robots negotiate passage through narrow corridors\([Bettini et al\., 2022](https://arxiv.org/html/2611.17917#bib.bib13);[Papoudakis et al\., 2021](https://arxiv.org/html/2611.17917#bib.bib23)\)\. These conventions simplify joint decision\-making and can contribute substantially to policy return\. Once formed, however, they also become behavioral patterns that deployed systems may repeatedly reproduce\. Removing an obsolete or unsafe coordination pattern while preserving service efficiency and effectiveness is therefore an important problem for reliable multi\-agent deployment\.

In real\-world settings, changing deployment requirements may render some coordination conventions no longer permissible\. For example, in a cooperative Web caching service, controllers may learn a division of roles in which one cache leaves an object uncached locally while another retains it and serves subsequent peer fetches\. This convention can improve storage utilization and reduce origin requests\. If updated service rules prohibit sharing a designated content class across an administrative boundary, the team must abandon conventions that depend on the revoked sharing relation while continuing to serve requests through local caches, permitted peers, or the origin\. Such a request targets a specific mode of cooperation, not the entire task and not one particular agent\. Removing the target convention therefore requires targeted intervention: the system should abandon the specified convention while retaining the coordination skills needed to make progress in other ways\. We call this problem*semantic coordination\-strategy unlearning*\.

A natural response is to prohibit the target convention at execution time to satisfy the semantic unlearning objective\. In the caching example, a runtime rule can reject disallowed peer fetches, but learned controllers may continue to omit local copies, increasing origin traffic or delivery latency when those fetches are rejected\. Our passage experiment in the Vectorized Multi\-Agent Simulator \(VMAS\), a two\-dimensional physics simulator for multi\-agent robotic learning\([Bettini et al\., 2022](https://arxiv.org/html/2611.17917#bib.bib13)\), illustrates this tradeoff between convention suppression and task performance in a controlled coordination setting\. The policy is trained with Multi\-Agent Proximal Policy Optimization \(MAPPO\), which uses centralized value estimation during training while allowing agents to choose actions in a decentralized manner at execution time\([Yu et al\., 2022](https://arxiv.org/html/2611.17917#bib.bib15)\)\. In this task, the policy learns a bottleneck convention: one agent yields or blocks while another agent passes\. Starting from a restored checkpoint, we apply an automaton\-based runtime shield that intercepts actions completing the forbidden convention\([Hammond et al\., 2021](https://arxiv.org/html/2611.17917#bib.bib33);[Icarte et al\., 2018](https://arxiv.org/html/2611.17917#bib.bib18)\)\. Although the shield changes the executed action, the original actor network continues to propose the same action pattern\. The agents are repeatedly intervened on near the passage and fail to make efficient progress\. As shown in Table[1](https://arxiv.org/html/2611.17917#S1.T1), the shield reduces WER from0\.7550\.755to0\.3410\.341, whereWER=maxν∈𝒱⁡Prπ,ν⁡\(τ⊧φf\)\\mathrm\{WER\}=\\max\_\{\\nu\\in\\mathcal\{V\}\}\\Pr\_\{\\pi,\\nu\}\(\\tau\\models\\varphi\_\{f\}\)is the worst\-case elicitation, computed over held\-out or adversarial contextsν\\nu\. Here,τ\\taudenotes a trajectory,φf\\varphi\_\{f\}is a relational temporal formula specifying the forbidden convention, andτ⊧φf\\tau\\models\\varphi\_\{f\}means that the trajectory realizes an accepting instance of that convention, such as one agent blocking or yielding followed by another agent passing within a bounded time window\. Lower WER means the forbidden convention is harder to recover under role permutations, starts, or context changes\. However, the task return drops from47\.547\.5to29\.629\.6, and normalized task utility falls to0\.6230\.623\. Return is the episodic task reward measuring task performance, while utility normalizes how much of the original performance is retained\. Thus, an apparent improvement in a convention\-suppression metric can hide a severe loss of performance\. The real challenge is not only to block the old convention, but to change the learned coordination behavior while preserving a viable way to complete the task\.

This difficulty is not limited to runtime constraints, because a coordination convention is not owned by one agent or one trajectory\. Existing reinforcement\-unlearning and trajectory\-deletion methods do not directly address semantic strategy removal\([Ye et al\., 2025](https://arxiv.org/html/2611.17917#bib.bib1);[Gong et al\., 2024](https://arxiv.org/html/2611.17917#bib.bib2)\)\. For example, in the caching scenario, the revoked convention concerns the relation between a cache that omits a local copy and a peer that subsequently supplies the object across the restricted boundary\. After deleting trajectories involving one cache pair, other nodes may still reproduce the same placement and fetching pattern\. Similarly, forgetting focus fire must cover the same tactic across different allied units and enemy targets\. We call this property*role invariance*: the target convention is defined by roles and interactions rather than fixed agent identities\. Moreover, the influence of the convention may be distributed across shared policies, value networks, communication modules, and agents’ responses to teammates\. These properties show that fixed agents, trajectories, or state\-action pairs are too narrow to serve as the basic unit of such unlearning requests\.

These observations motivate a behavioral notion of semantic coordination\-strategy unlearning: an edited policy should suppress the specified convention across nominal and eliciting contexts while preserving useful cooperation\. We think target conventions should be specified after deployment, because shared team rewards entangle their contributions with useful cooperation, making it difficult to withhold positive credit selectively during training\. We therefore study request\-driven post\-training policy editing, with the accepted update merged into decentralized actors for direct deployment\. It provides a concrete comparison target beyond the frequency of forbidden events and includes the preservation of original coordination competence as part of the unlearning requirement\. The final deployed policy should satisfy these behavioral requirements without relying on a detachable inference\-time prohibition layer\.

Table 1\.Comparison on VMAS Passage from the same restored MAPPO checkpoint\. Similar WER values can correspond to different task performance\.To address this MARL problem, we proposeCARVE\. The framework first uses counterfactual action interventions to attribute target\-convention risk to agent\-time interactions\. It then selects parameter subspaces according to the behavior that should be retained and restricts updates to these subspaces, reducing interference with other coordination skills\. Counterfactual re\-coordination helps agents learn alternative ways to complete the task, avoiding the progress failures caused by merely blocking the original action pattern\. Finally, adversarial auditing tests whether the target convention can still be recovered\.CARVEconnects the semantic scope of an unlearning request with the cooperative behaviors that must be preserved, enabling targeted model updates for semantic coordination\-strategy unlearning\. Our evaluation spans five representative MARL environments and a set of representative baselines to assess the effectiveness and efficiency of policy unlearning\. Compared with runtime shielding,CARVEachieves relative improvements of167\.8%167\.8\\%in average task utility\. Unlearning is101\.92101\.92–584\.83584\.83times faster than original backbone training\. Meanwhile, the two methods achieve similar average reductions in WER, indicating that reporting target\-convention recoverability alone is insufficient; evaluation must also consider task utility, return retention\. Our main contributions are summarized as follows:

- •We first identify limitations of MARL runtime shield and provide the formalization of role\-invariant semantic coordination\-strategy unlearning in finite\-horizon cooperative Markov games\.
- •We proposeCARVE, combining causal interaction attribution, counterfactual re\-coordination, and adversarial auditing for efficient subspace updates that remove target strategies while preserving performance\.
- •We evaluateCARVEacross five MARL benchmarks through basic and stronger elicitation tests, efficiency measurements, and ablations\. Compared with the strongest parameter\-editing baseline,CARVEimproves unlearning effectiveness by16\.9%16\.9\\%\.

## 2\.Related Work

##### MARL benchmarks and algorithms\.

Reinforcement learning\([Arulkumaran et al\., 2017](https://arxiv.org/html/2611.17917#bib.bib43);[Dai et al\., 2025](https://arxiv.org/html/2611.17917#bib.bib39)\)is gradually evolving towards multi\-agent systems\. A large body of cooperative MARL research has developed benchmarks and algorithms for learning coordination\. SMAC established StarCraft micromanagement as a cooperative MARL benchmark\([Samvelyan et al\., 2019](https://arxiv.org/html/2611.17917#bib.bib8)\); SMACv2 introduced procedural generation and stronger closed\-loop evaluation pressure\([Ellis et al\., 2023](https://arxiv.org/html/2611.17917#bib.bib10)\)\. Overcooked\-AI studies human\-AI coordination\([Carroll et al\., 2019](https://arxiv.org/html/2611.17917#bib.bib11)\), and the Overcooked Generalisation Challenge \(OGC\) expands evaluation to novel partners and layouts\([Ruhdorfer et al\., 2024](https://arxiv.org/html/2611.17917#bib.bib12)\)\. VMAS provides fast vectorized multi\-robot simulation\([Bettini et al\., 2022](https://arxiv.org/html/2611.17917#bib.bib13)\)\. RWARE and LBF are sparse\-reward coordination benchmarks popularized through systematic cooperative MARL evaluation\([Papoudakis et al\., 2021](https://arxiv.org/html/2611.17917#bib.bib23)\)\. Melting Pot complements these with test\-time social generalization\([Leibo et al\., 2021](https://arxiv.org/html/2611.17917#bib.bib25)\)\. Standard backbones include MAPPO\([Yu et al\., 2022](https://arxiv.org/html/2611.17917#bib.bib15)\), QMIX\([Rashid et al\., 2020](https://arxiv.org/html/2611.17917#bib.bib17)\), VDN\([Sunehag et al\., 2018](https://arxiv.org/html/2611.17917#bib.bib24)\), MADDPG\([Lowe et al\., 2017](https://arxiv.org/html/2611.17917#bib.bib27)\), and COMA\([Foerster et al\., 2018](https://arxiv.org/html/2611.17917#bib.bib28)\)\. Our work builds on these established environments and backbones, but changes the evaluation question from whether agents can acquire coordination to whether a trained system can selectively forget it while retaining task competence\.

##### Reinforcement and machine unlearning\.

Machine unlearning\([Oesterling et al\., 2024](https://arxiv.org/html/2611.17917#bib.bib3);[Bourtoule et al\., 2021](https://arxiv.org/html/2611.17917#bib.bib22)\)is commonly motivated by retraining equivalence: the post\-unlearning model should resemble a model trained without the removed data or influence\([Liang et al\., 2026b](https://arxiv.org/html/2611.17917#bib.bib42);[Wu et al\., 2025c](https://arxiv.org/html/2611.17917#bib.bib19);[Wu et al\., 2025d](https://arxiv.org/html/2611.17917#bib.bib20);[Wu et al\., 2026](https://arxiv.org/html/2611.17917#bib.bib21);[Liang et al\., 2026a](https://arxiv.org/html/2611.17917#bib.bib41)\)due to the requirements of GDPR\([Zaeem and Barber, 2020](https://arxiv.org/html/2611.17917#bib.bib29)\), CCPA\([Illman and Temple, 2019](https://arxiv.org/html/2611.17917#bib.bib30)\)or users’ privacy needs\([Wu et al\., 2025b](https://arxiv.org/html/2611.17917#bib.bib31);[Wu et al\., 2025a](https://arxiv.org/html/2611.17917#bib.bib32)\)\. Reinforcement unlearning extends this idea to RL environments\([Ye et al\., 2025](https://arxiv.org/html/2611.17917#bib.bib1)\), while TrajDeleter targets offline RL trajectory removal\([Gong et al\., 2024](https://arxiv.org/html/2611.17917#bib.bib2)\)\. These settings do not cover a distributed, role\-invariant coordination convention that can be re\-enacted by different agents\.

##### Policy repair, safe MARL, and temporal logic\.

Another line of work studies how to constrain or repair policies after undesirable behavior has been identified\. Temporal logic and reward machines provide automata\-based structure for non\-Markovian tasks\([Hammond et al\., 2021](https://arxiv.org/html/2611.17917#bib.bib33);[Icarte et al\., 2018](https://arxiv.org/html/2611.17917#bib.bib18);[Chen et al\., 2026a](https://arxiv.org/html/2611.17917#bib.bib40)\)\. Safe MARL methods such as MACPO and MAPPO\-Lagrangian optimize constraints during policy improvement\([Gu et al\., 2021](https://arxiv.org/html/2611.17917#bib.bib5)\)\. PolicyCleanse mitigates RL backdoors\([Guo et al\., 2023](https://arxiv.org/html/2611.17917#bib.bib34)\), and MUTE removes low\-value communication\([Zuo et al\., 2026](https://arxiv.org/html/2611.17917#bib.bib4)\); both are relevant conditional baselines for triggered or communication\-mediated target strategies\. These approaches provide closely related risk–return objectives and are essential baselines for behavioral strategy unlearning\. CARVE focuses on request\-driven editing of a trained team, using interaction attribution, retain\-aware sparse updates, and teacher\-guided replacement coordination to suppress role\-invariant conventions while preserving cooperative task performance\.

## 3\.Preliminaries

We study a finite\-horizon cooperative Markov game as follows:

\(1\)𝒢=\(𝒩,𝒮,\{𝒪i\},\{𝒜i\},P,r,γ,H\),\\mathcal\{G\}=\(\\mathcal\{N\},\\mathcal\{S\},\\\{\\mathcal\{O\}\_\{i\}\\\},\\\{\\mathcal\{A\}\_\{i\}\\\},P,r,\\gamma,H\),where𝒩=\{1,…,N\}\\mathcal\{N\}=\\\{1,\\ldots,N\\\}is the set of agents,𝒮\\mathcal\{S\}is the global state space,𝒪i\\mathcal\{O\}\_\{i\}and𝒜i\\mathcal\{A\}\_\{i\}are the observation and action spaces of agentii,PPis the transition kernel,rris the shared team reward,γ\\gammais the discount factor, andHHis the horizon\. At timett, agentiiobservesotio\_\{t\}^\{i\}, acts withatia\_\{t\}^\{i\}, and conditions on its local historyhtih\_\{t\}^\{i\}\. We use𝐡t=\(ht1,…,htN\)\\mathbf\{h\}\_\{t\}=\(h\_\{t\}^\{1\},\\ldots,h\_\{t\}^\{N\}\)and𝐚t=\(at1,…,atN\)\\mathbf\{a\}\_\{t\}=\(a\_\{t\}^\{1\},\\ldots,a\_\{t\}^\{N\}\)for the joint history and joint action\. A decentralized policy factorizes as

\(2\)𝝅θ​\(𝐚t∣𝐡t\)=∏i=1Nπθi​\(ati∣hti\),\\bm\{\\pi\}\_\{\\theta\}\(\\mathbf\{a\}\_\{t\}\\mid\\mathbf\{h\}\_\{t\}\)=\\prod\_\{i=1\}^\{N\}\\pi\_\{\\theta\_\{i\}\}\(a\_\{t\}^\{i\}\\mid h\_\{t\}^\{i\}\),whereθ\\thetadenotes the actor parameters\. Parameter sharing is allowed: multiple agents may use the same parameter block while receiving different histories\. The return of a policy is

\(3\)J⁡\(π\)=𝔼τ∼π​\[∑t=0H−1γt​r​\(st,𝐚t\)\],J\(\\pi\)=\\mathbb\{E\}\_\{\\tau\\sim\\pi\}\\left\[\\sum\_\{t=0\}^\{H\-1\}\\gamma^\{t\}r\(s\_\{t\},\\mathbf\{a\}\_\{t\}\)\\right\],whereτ\\taudenotes a complete trajectory\.

### 3\.1\.Semantic Requests

The object to be forgotten is not a data point, one fixed agent, or one exact trajectory\. It is a*role\-invariant coordination convention*, which can be specified as a relational temporal formulaφf\\varphi\_\{f\}, whereffdenotes “forget\.” Atomic predicates are event labels computed from the state and joint action, such asBlock⁡\(i,door\)\\operatorname\{Block\}\(i,\\mathrm\{door\}\),Pass⁡\(j,door\)\\operatorname\{Pass\}\(j,\\mathrm\{door\}\), orAttack⁡\(i,e\)\\operatorname\{Attack\}\(i,e\)\. The variablesi,ji,jrange over agents, andeeranges over task entities, such as a specific enemy in a game\. For example, the semantic request “forget the convention where one agent blocks a door so another passes shortly after” can be written as

\(4\)φf=◇\[∃i≠j:Block\(i,door\)∧◇≤kPass\(j,door\)\]\.\\varphi\_\{f\}=\\Diamond\\\!\\left\[\\exists i\\neq j:\\operatorname\{Block\}\(i,\\mathrm\{door\}\)\\land\\Diamond\_\{\\leq k\}\\operatorname\{Pass\}\(j,\\mathrm\{door\}\)\\right\]\.Here◇\\Diamondmeans “eventually” and◇≤k\\Diamond\_\{\\leq k\}means “withinkksteps\.”∃i≠j\\exists i\\neq jmeans that the convention should be removed no matter which concrete agents instantiate the blocker and passer roles\. The same notation also covers combat conventions such as focus\-fire:

\(5\)φf=◇\[∃i≠j,e:Attack\(i,e\)∧◇≤kAttack\(j,e\)\],\\varphi\_\{f\}=\\Diamond\\\!\\left\[\\exists i\\neq j,e:\\operatorname\{Attack\}\(i,e\)\\land\\Diamond\_\{\\leq k\}\\operatorname\{Attack\}\(j,e\)\\right\],which asks the policy to forget the role\-invariant tactic where two allies successively attack the same enemyee\. For monitoring, we compileφf\\varphi\_\{f\}into a finite automaton

\(6\)𝒜φ=\(Q,q0,δ,F\)\.\\mathcal\{A\}\_\{\\varphi\}=\(Q,q\_\{0\},\\delta,F\)\.QQis the set of monitor states,q0q\_\{0\}is the initial state,δ\\deltaupdates the monitor after seeing the current event labels, andFFis the set of accepting states indicating that the forbidden convention has occurred\. The product state

\(7\)s¯t=\(st,qt\)\\bar\{s\}\_\{t\}=\(s\_\{t\},q\_\{t\}\)combines the environment statests\_\{t\}with the current monitor stateqtq\_\{t\}\. Thus the monitor turns a history\-dependent semantic request into a Markovian signal that can be used by critics, attribution, and evaluation\.

### 3\.2\.Unlearning Goal

Letc∈𝒞c\\in\\mathcal\{C\}denote an evaluation context, such as a layout, initial\-state distribution, teammate population, or adversarial partner\. The semantic risk of a policy is

\(8\)ℛφ\(π;c\)=Prτ∼\(𝒢c,π\)\[τ⊧φf\],ℛφrob\(π\)=supc∈𝒞ℛφ\(π;c\)\.\\mathcal\{R\}\_\{\\varphi\}\(\\pi;c\)=\\Pr\_\{\\tau\\sim\(\\mathcal\{G\}\_\{c\},\\pi\)\}\[\\tau\\models\\varphi\_\{f\}\],\\qquad\\mathcal\{R\}\_\{\\varphi\}^\{\\mathrm\{rob\}\}\(\\pi\)=\\sup\_\{c\\in\\mathcal\{C\}\}\\mathcal\{R\}\_\{\\varphi\}\(\\pi;c\)\.The first term is the probability that the policy executes the forbidden convention in contextcc\. The second is the worst\-case risk over the evaluation contexts\. Given an original policyπ0\\pi\_\{0\}, an unlearning algorithm returns a new policyπ−=U⁡\(π0,φf\)\\pi^\{\-\}=U\(\\pi\_\{0\},\\varphi\_\{f\}\)\. We wantπ−\\pi^\{\-\}to satisfy two behavioral requirements:

\(9\)ℛφrob​\(π−\)≤ϵf,Jr​\(π−\)≥Jr​\(π0\)−δJ\.\\mathcal\{R\}\_\{\\varphi\}^\{\\mathrm\{rob\}\}\(\\pi^\{\-\}\)\\leq\\epsilon\_\{f\},\\quad J\_\{r\}\(\\pi^\{\-\}\)\\geq J\_\{r\}\(\\pi\_\{0\}\)\-\\delta\_\{J\}\.The first condition asks for low residual semantic risk under nominal and adversarial evaluation contexts\. The second preserves retained task performance;JrJ\_\{r\}is return measured on target\-unrelated or retained contexts, andδJ\\delta\_\{J\}is the allowed loss\. We do not require a separately trained reference policy\. Instead, the evaluation directly measures whether the forbidden convention occurs or can be elicited again, using semantic monitors under nominal rollouts, role permutations, held\-out contexts, and adversarial partners\. This distinction is important because a runtime shield may block the final forbidden action while leaving the underlying coordination preference unchanged\. The request is intentionally behavioral in approximate unlearning\([Xu et al\., 2024](https://arxiv.org/html/2611.17917#bib.bib35);[Wang et al\., 2024](https://arxiv.org/html/2611.17917#bib.bib36);[Li et al\., 2025](https://arxiv.org/html/2611.17917#bib.bib37)\), which does not claim that every parameter\-level trace of the original strategy disappears\. Instead, they ask whether the delivered policy acts like one that no longer relies on the forbidden convention, while keeping the rest of its cooperative competence\.

## 4\.System Design:CARVE

The strategy unlearning MARL frameworkCARVEtakes as input the original policyπ0\\pi\_\{0\}, its parametersθ0\\theta\_\{0\}, rollout logs or simulator access, the semantic requestφf\\varphi\_\{f\}, and optional retain specifications of behaviors or task performance to retain\. It outputs a merged policyπ−\\pi^\{\-\}whose actor parameters are changed\. The automaton is used during unlearning and auditing, but not as an inference\-time shield\. Figure[1](https://arxiv.org/html/2611.17917#acmlabel1)summarizes the four stages\.

![Diagram showing the CARVE framework workflow, from input policy and data to edited policy, with steps for attribution, selection, replacement, and auditing.](https://arxiv.org/html/2611.17917v1/overall.png)Figure 1\.Overview ofCARVE\. The framework takes an original policy, rollout data or simulator access, and a semantic requestφf\\varphi\_\{f\}, and returns an edited policyπ−\\pi^\{\-\}\.CARVEfirst attributes forbidden\-convention risk to agent\-time interactions, then selects retain\-aware sparse edit directions, constructs replacement coordination through a counterfactual teacher, and finally audits the edited policy before merging the accepted parameter update\.Diagram showing the CARVE framework workflow, from input policy and data to edited policy, with steps for attribution, selection, replacement, and auditing\.### 4\.1\.Causal Coordination Attribution

The first stage asks a simple question: which agents, at which times, make the forbidden convention more likely? We estimate two centralized critics on the product states¯t=\(st,qt\)\\bar\{s\}\_\{t\}=\(s\_\{t\},q\_\{t\}\):

\(10\)QJ\(s¯t,𝐚t\)=𝔼\[∑k=tH−1γk−trk∣s¯t,𝐚t\],Qφ\(s¯t,𝐚t\)=Pr\(qH∈F∣s¯t,𝐚t\)\.Q^\{J\}\(\\bar\{s\}\_\{t\},\\mathbf\{a\}\_\{t\}\)=\\mathbb\{E\}\\\!\\left\[\\sum\_\{k=t\}^\{H\-1\}\\gamma^\{k\-t\}r\_\{k\}\\mid\\bar\{s\}\_\{t\},\\mathbf\{a\}\_\{t\}\\right\],Q^\{\\varphi\}\(\\bar\{s\}\_\{t\},\\mathbf\{a\}\_\{t\}\)=\\Pr\(q\_\{H\}\\in F\\mid\\bar\{s\}\_\{t\},\\mathbf\{a\}\_\{t\}\)\.QJQ^\{J\}estimates future task return, andQφQ^\{\\varphi\}estimates the probability that the monitor eventually accepts\. Becauseqtq\_\{t\}records formula progress,QφQ^\{\\varphi\}can assign credit to early decisions that only become visibly forbidden later\.

To measure an agent’s contribution, we compare the observed joint action with counterfactual actions sampled from a task\-plausible baselinebb\. For a coalitionS⊆𝒩S\\subseteq\\mathcal\{N\},

\(11\)vt​\(S\)=𝔼𝐚S¯,t′∼b​\[Qφ​\(s¯t,𝐚S,t,𝐚S¯,t′\)\],v\_\{t\}\(S\)=\\mathbb\{E\}\_\{\\mathbf\{a\}^\{\\prime\}\_\{\\bar\{S\},t\}\\sim b\}\\left\[Q^\{\\varphi\}\(\\bar\{s\}\_\{t\},\\mathbf\{a\}\_\{S,t\},\\mathbf\{a\}^\{\\prime\}\_\{\\bar\{S\},t\}\)\\right\],where𝐚S,t\\mathbf\{a\}\_\{S,t\}keeps the observed actions of agents inSS, and𝐚S¯,t′\\mathbf\{a\}^\{\\prime\}\_\{\\bar\{S\},t\}replaces the other agents’ current actions\. Specifically,bbsamples alternative actions from the original policy’s high\-probability actions at the current local histories, excluding the observed actions and renormalizing their probabilities\. A Shapley\-style attribution is then

\(12\)χi,t=𝔼S∼ωi​\[vt​\(S∪\{i\}\)−vt​\(S\)\],\\chi\_\{i,t\}=\\mathbb\{E\}\_\{S\\sim\\omega\_\{i\}\}\\left\[v\_\{t\}\(S\\cup\\\{i\\\}\)\-v\_\{t\}\(S\)\\right\],whereωi\\omega\_\{i\}is the standard Shapley distribution\([Shapley, 1952](https://arxiv.org/html/2611.17917#bib.bib6);[You et al\., 2026](https://arxiv.org/html/2611.17917#bib.bib9)\)over coalitions not containingii\. Thusχi,t\>0\\chi\_\{i,t\}\>0means that restoring agentii’s original action increases semantic risk on average\. To keep pairwise coordination visible\([Grabisch and Roubens, 1999](https://arxiv.org/html/2611.17917#bib.bib7)\), we also estimate

\(13\)χi​j,t\(2\)=𝔼S∼ωi​j​\[vt​\(S∪\{i,j\}\)−vt​\(S∪\{i\}\)−vt​\(S∪\{j\}\)\+vt​\(S\)\]\.\\chi\_\{ij,t\}^\{\(2\)\}=\\mathbb\{E\}\_\{S\\sim\\omega\_\{ij\}\}\\left\[v\_\{t\}\(S\\cup\\\{i,j\\\}\)\-v\_\{t\}\(S\\cup\\\{i\\\}\)\-v\_\{t\}\(S\\cup\\\{j\\\}\)\+v\_\{t\}\(S\)\\right\]\.This second score is positive when two agents jointly create more risk than their separate effects explain\. In practice, we can approximate these expectations with sampled coalitions rather than enumerating all subsets\.

### 4\.2\.Retain\-Aware Strategy Subspace

The second stage turns attribution into parameter directions\. Let

\(14\)ψi,t=∇θ​log​πθi​\(ati∣hti\)\\psi\_\{i,t\}=\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\_\{i\}\}\(a\_\{t\}^\{i\}\\mid h\_\{t\}^\{i\}\)be the policy score of agentii’s decision\. We convert the attribution scores into one nonnegative weight

\(15\)wi,t=\[χi,t\]\+\+η​∑j≠i\[χi​j,t\(2\)\]\+,w\_\{i,t\}=\[\\chi\_\{i,t\}\]\_\{\+\}\+\\eta\\sum\_\{j\\neq i\}\[\\chi\_\{ij,t\}^\{\(2\)\}\]\_\{\+\},where\[x\]\+=max⁡\(x,0\)\[x\]\_\{\+\}=\\max\(x,0\)andη\\etacontrols how much pairwise coordination affects the weight\. The forbidden\-strategy direction for trajectoryτ\\tauis

\(16\)gf​\(τ\)=∑i,twi,t​ψi,t\.g\_\{f\}\(\\tau\)=\\sum\_\{i,t\}w\_\{i,t\}\\psi\_\{i,t\}\.This vector points toward parameters that support decisions causally involved inφf\\varphi\_\{f\}\.

Aggregating over high\-risk trajectories gives

\(17\)Gf=𝔼⁡\[gf​\(τ\)​gf​\(τ\)⊤\],G\_\{f\}=\\mathbb\{E\}\[g\_\{f\}\(\\tau\)g\_\{f\}\(\\tau\)^\{\\top\}\],the forbidden\-strategy sensitivity matrix\.

We also estimate a retained Fisher matrix\([Yang et al\., 2022](https://arxiv.org/html/2611.17917#bib.bib38)\)FrF\_\{r\}from trajectories and decision points that do not trigger the target convention and should remain stable\. Its empirical estimate is computed as

\(18\)F^r=1M​∑\(i,t\)∈ℐrψi,t​ψi,t⊤,\\widehat\{F\}\_\{r\}=\\frac\{1\}\{M\}\\sum\_\{\(i,t\)\\in\\mathcal\{I\}\_\{r\}\}\\psi\_\{i,t\}\\psi\_\{i,t\}^\{\\top\},whereℐr\\mathcal\{I\}\_\{r\}is the set of retained decision points,M=\|ℐr\|M=\|\\mathcal\{I\}\_\{r\}\|, and all policy score vectors are evaluated at the original parametersθ0\\theta\_\{0\}\. Intuitively,u⊤​Gf​uu^\{\\top\}G\_\{f\}uis large if directionuuchanges the forbidden strategy, whileu⊤​Fr​uu^\{\\top\}F\_\{r\}uis large ifuudisturbs retained behavior\.

CARVEchooses edit directions by solving

\(19\)Gf​uk=λk​\(Fr\+λ0​I\)​uk\.G\_\{f\}u\_\{k\}=\\lambda\_\{k\}\(F\_\{r\}\+\\lambda\_\{0\}I\)u\_\{k\}\.Hereuku\_\{k\}is an edit direction,λk\\lambda\_\{k\}is its strategy\-to\-retain ratio, andλ0\>0\\lambda\_\{0\}\>0stabilizes the retain metric\. LetB=\[u1,…,uK\]B=\[u\_\{1\},\\ldots,u\_\{K\}\]contain the top\-KKcandidate directions\. To obtain sparse edits, we select a fixed coordinate maskmf∈\{0,1\}Pm\_\{f\}\\in\\\{0,1\\\}^\{P\}containing the top⌈f​P⌉\\lceil fP\\rceilcoordinates under the retain\-aware semantic sensitivity score, and formBf=diag⁡\(mf\)​BB\_\{f\}=\\operatorname\{diag\}\(m\_\{f\}\)B, which is a sparsification heuristic\. Subsequent optimization is restricted to

\(20\)θα=θ0\+Bf​α,α∈ℝK\.\\theta\_\{\\alpha\}=\\theta\_\{0\}\+B\_\{f\}\\alpha,\\qquad\\alpha\\in\\mathbb\{R\}^\{K\}\.Onlyα\\alphais optimized; the masked basis remains fixed\.

### 4\.3\.Counterfactual Re\-Coordination

Removing a convention can leave a coordination gap: agents may stop using the forbidden strategy but fail to coordinate on an alternative\. This stage therefore builds a counterfactual teacher at product states with highQφQ^\{\\varphi\}\. Here, counterfactual refers to alternative joint actions evaluated at the same decision context to construct replacement coordination\. For each such state, the teacher favors joint actions that keep task value high and semantic risk low:

\(21\)μ~t​\(𝐚∣s¯t\)=1Zt​𝝅0​\(𝐚∣𝐡t\)​exp⁡\(β​QJ​\(s¯t,𝐚\)−λ​Qφ​\(s¯t,𝐚\)\)\.\\widetilde\{\\mu\}\_\{t\}\(\\mathbf\{a\}\\mid\\bar\{s\}\_\{t\}\)=\\frac\{1\}\{Z\_\{t\}\}\\bm\{\\pi\}\_\{0\}\(\\mathbf\{a\}\\mid\\mathbf\{h\}\_\{t\}\)\\exp\\\!\\left\(\\beta Q^\{J\}\(\\bar\{s\}\_\{t\},\\mathbf\{a\}\)\-\\lambda Q^\{\\varphi\}\(\\bar\{s\}\_\{t\},\\mathbf\{a\}\)\\right\)\.ZtZ\_\{t\}normalizes the distribution over joint actions\. The coefficientβ≥0\\beta\\geq 0rewards useful alternatives, andλ≥0\\lambda\\geq 0penalizes actions likely to complete the forgotten convention\. This is a compact way to express a KL\-near replacement teacher: stay close toπ0\\pi\_\{0\}, but tilt probability mass away from the target strategy and toward valuable alternatives\. Because the teacher is centralized, we distill its agent\-wise marginalsμ~i,t\\widetilde\{\\mu\}\_\{i,t\}into the decentralized actors:

\(22\)ℒcf\(α\)=𝔼τ~∑i,twi,tDKL\(μ~i,t\(⋅∣s¯t\)∥πθα,i\(⋅∣hti\)\)\.\\mathcal\{L\}\_\{\\mathrm\{cf\}\}\(\\alpha\)=\\mathbb\{E\}\_\{\\widetilde\{\\tau\}\}\\sum\_\{i,t\}w\_\{i,t\}D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\widetilde\{\\mu\}\_\{i,t\}\(\\cdot\\mid\\bar\{s\}\_\{t\}\)\\\|\\pi\_\{\\theta\_\{\\alpha\},i\}\(\\cdot\\mid h\_\{t\}^\{i\}\)\\right\)\.τ~\\widetilde\{\\tau\}denotes teacher\-generated or teacher\-relabeled trajectories, andμ~i,t\\widetilde\{\\mu\}\_\{i,t\}is the teacher’s marginal distribution for agentii\. The same attribution weightwi,tw\_\{i,t\}focuses distillation on decisions that matter for the target convention, rather than forcing the entire policy to imitate the teacher everywhere\.

### 4\.4\.Robust Auditing and Policy Merge

The final stage choosesα\\alphaand repeatedly audits the candidate policyπθα\\pi\_\{\\theta\_\{\\alpha\}\}\. The training objective contains only two terms:

\(23\)minαℒcf​\(α\)\+λr​ℒretain​\(α\),\\min\_\{\\alpha\}\\quad\\mathcal\{L\}\_\{\\mathrm\{cf\}\}\(\\alpha\)\+\\lambda\_\{\\mathrm\{r\}\}\\mathcal\{L\}\_\{\\mathrm\{retain\}\}\(\\alpha\),where

\(24\)ℒretain\(α\)=𝔼τ∼𝒟r∑i,tDKL\(π0,i\(⋅∣hti\)∥πθα,i\(⋅∣hti\)\)\\mathcal\{L\}\_\{\\mathrm\{retain\}\}\(\\alpha\)=\\mathbb\{E\}\_\{\\tau\\sim\\mathcal\{D\}\_\{r\}\}\\sum\_\{i,t\}D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\pi\_\{0,i\}\(\\cdot\\mid h\_\{t\}^\{i\}\)\\\|\\pi\_\{\\theta\_\{\\alpha\},i\}\(\\cdot\\mid h\_\{t\}^\{i\}\)\\right\)keeps target\-unrelated behavior close to the original policy\. The semantic monitor is used for auditing:

\(25\)ℛφ^adv​\(π\)=maxc∈𝒞adv⁡ℛφ^​\(π,c\)\\widehat\{\\mathcal\{R\}\_\{\\varphi\}\}\_\{\\mathrm\{adv\}\}\(\\pi\)=\\max\_\{c\\in\\mathcal\{C\}\_\{\\mathrm\{adv\}\}\}\\widehat\{\\mathcal\{R\}\_\{\\varphi\}\}\(\\pi;c\)is the empirical worst\-case semantic risk over adversarial contexts\. These contexts include held\-out partners, starts, layouts, and role permutations selected to elicitφf\\varphi\_\{f\}\. If the audit risk is too high, the eliciting trajectories are added to the training set and the update is repeated\. We accept the final policy only if it passes the semantic\-risk and return checks from the formulation section\. The delivered model is obtained by mergingΔ​θ=Bf​α\\Delta\\theta=B\_\{f\}\\alphainto the actor parameters; no monitor or shield is required at deployment\.

## 5\.Experiments

We conduct experiments to answer four questions aboutCARVE:i\)\. RQ1: whether it removes the requested coordination strategy \(§[5\.2\.1](https://arxiv.org/html/2611.17917#S5.SS2.SSS1), §[5\.2\.3](https://arxiv.org/html/2611.17917#S5.SS2.SSS3), §[5\.4](https://arxiv.org/html/2611.17917#S5.SS4)\)?ii\)\. RQ2: whether it preserves MARL task performance \(§[5\.2\.2](https://arxiv.org/html/2611.17917#S5.SS2.SSS2)\)?iii\)\. RQ3: whether it is efficient enough to be a post\-training edit rather than retraining in disguise \(§[5\.3](https://arxiv.org/html/2611.17917#S5.SS3)\)?iv\)\. RQ4: which components and hyperparameters control the erasure–retention frontier \(§[5\.5](https://arxiv.org/html/2611.17917#S5.SS5),§[C](https://arxiv.org/html/2611.17917#A3)\)?

### 5\.1\.Experimental Setup

#### 5\.1\.1\.Testbed

Experiments are implemented in Python 3\.11\.8 with PyTorch 2\.6\.0\+cu124 and TorchRL 0\.7\.2\. They run on an Ubuntu 18\.04\.6 LTS server with two 26\-core Intel Xeon Platinum 8269CY CPUs, 256 GB RAM, and four NVIDIA A100\-40GB GPUs\. Unless otherwise stated, backbone training uses five random seeds\.

#### 5\.1\.2\.Evaluation Benchmarks and Unlearning Requests

Our evaluation cross five MARL environments pairs established cooperative MARL testbeds with role\-invariant unlearning requests\. For SMACv2\([Ellis et al\., 2023](https://arxiv.org/html/2611.17917#bib.bib10)\), the target is a relational combat tactic; for Overcooked\-AI/OGC\([Carroll et al\., 2019](https://arxiv.org/html/2611.17917#bib.bib11);[Ruhdorfer et al\., 2024](https://arxiv.org/html/2611.17917#bib.bib12)\), it is a bottleneck or handoff convention; for VMAS\([Bettini et al\., 2022](https://arxiv.org/html/2611.17917#bib.bib13)\), it is a continuous\-control yielding or leader\-follower rule; for RWARE\([Papoudakis et al\., 2021](https://arxiv.org/html/2611.17917#bib.bib23)\), it is a warehouse aisle convention; and for LBF\([Papoudakis et al\., 2021](https://arxiv.org/html/2611.17917#bib.bib23)\), it is a sparse\-reward partner\-selection convention, which is used as an additional sparse\-reward stress test\. Details of the scenarios, role\-invariant unlearning requests, retain specifications, backbone algorithms, and hyperparameters, are provided in Appendix[B](https://arxiv.org/html/2611.17917#A2)\.

#### 5\.1\.3\.Baselines

The baselines are chosen to answer distinct alternative explanations: whether runtime shielding merely hides the behavior, whether constrained MARL is enough, whether generic parameter unlearning is enough, and whether trajectory removal is enough\. We compare against the methods as shown in Table[2](https://arxiv.org/html/2611.17917#S5.T2)\.

Table 2\.Baseline suite used in the main result tables\.Table 3\.Unlearning performance across five MARL benchmarks\. OC\-AI denotes Overcooked\-AI\.
#### 5\.1\.4\.Evaluation Metrics

We assess convention removal using the nominal semantic satisfaction rateSSR⁡\(π\)=Prτ∼π⁡\(τ⊧φf\)\\mathrm\{SSR\}\(\\pi\)=\\Pr\_\{\\tau\\sim\\pi\}\(\\tau\\models\\varphi\_\{f\}\)and the worst\-case elicitation rateWER⁡\(π\)=maxν∈𝒱⁡Prπ,ν⁡\(τ⊧φf\)\.\\mathrm\{WER\}\(\\pi\)=\\max\_\{\\nu\\in\\mathcal\{V\}\}\\Pr\_\{\\pi,\\nu\}\(\\tau\\models\\varphi\_\{f\}\)\.Here,𝒱\\mathcal\{V\}denotes the red\-team elicitation set, including held\-out teammates or partners when applicable, held\-out starts, layouts, procedural context seeds, and role permutations\. Both metrics measure the occurrence of the forbidden convention, with lower values indicating more effective suppression\. We assess task retention using episodic return \(Return\) and normalized task\-utility retention \(Util\.\(π\)=clip\[0,1\]⁡\(1−\[J0−J⁡\(π\)\]\+Db\)\\mathrm\{Util\.\}\(\\pi\)=\\operatorname\{clip\}\_\{\[0,1\]\}\\\!\\left\(1\-\\frac\{\[J\_\{0\}\-J\(\\pi\)\]\_\{\+\}\}\{D\_\{b\}\}\\right\)\), whereJ0J\_\{0\}is the original policy’s mean return recorded in the same run,\[x\]\+=max⁡\(x,0\)\[x\]\_\{\+\}=\\max\(x,0\)\. The evaluation scripts useDb=max⁡\(\|J0\|,1\)D\_\{b\}=\\max\(\|J\_\{0\}\|,1\)for SMACv2 and VMAS, which accommodates negative and near\-zero original returns, andDb=J0D\_\{b\}=J\_\{0\}for Overcooked\-AI, RWARE, and LBF, whose reported original returns are positive\. Util\. is normalized relative to the restored original policy, with11indicating full task\-performance retention\. We measure efficiency using unlearning speedupTbackbone/TunlearnT\_\{\\mathrm\{backbone\}\}/T\_\{\\mathrm\{unlearn\}\}, inference latency \(Infer\.\), and the fraction of actor parameters changed relative to the original checkpoint \(ParamFrac\)\.

Table 4\.MARL retain performance across five benchmarks\. OC\-AI denotes Overcooked\-AI\.

### 5\.2\.Overall Performance

#### 5\.2\.1\.Unlearning Performance

Table[3](https://arxiv.org/html/2611.17917#S5.T3)shows thatCARVEreduces mean SSR and WER by 91\.4% and 83\.3%, respectively, relative to the original policies\. No target\-strategy completions are observed under either protocol in Overcooked\-AI, RWARE, and LBF\. In the two benchmarks with residual risk,CARVEmatches the lowest reported WER in SMACv2 and achieves the lowest WER in VMAS\. Relative to MACPO, the strongest parameter\-editing baseline by mean WER, the largest within\-benchmark SSR and WER reductions reach 29\.1% and 39\.8%, respectively\. Compared with Shield,CARVEmatches WER in four benchmarks and reduces it by 53\.8% in VMAS, while requiring no inference\-time monitor\. For Full\-FT, RP, and Lag\., the largest within\-benchmark WER reductions range from 33\.3% to 56\.0%\. These gains are consistent with targeting causally attributed coordination decisions, constructing useful alternatives, and incorporating eliciting trajectories during auditing, rather than relying solely on broad updates or semantic penalties\. Fisher leaves greater residual semantic risk, while TrajDel still permits target\-strategy completions in every benchmark\. This shows that parameter dampening may miss the interactions underlying a convention, while deleting selected trajectories does not prevent the strategy’s reproduction through other histories\. SMACv2 remains challenging, with a residual WER of0\.5000\.500\. This matches the lowest observed WER among the evaluated methods\. At this risk level, however,CARVEretains a utility of0\.9410\.941, compared with0\.7490\.749for Shield and0\.5440\.544for MACPO\. Thus, although substantial residual risk remains,CARVEpreserves the most task utility among methods achieving the same worst\-case suppression\. These results highlight the importance of evaluating convention removal jointly with retained coordination competence\.

#### 5\.2\.2\.MARL Retain Performance

Table[4](https://arxiv.org/html/2611.17917#S5.T4)showsCARVEreduces a median utility of only 5\.9% relative to the original policies, with four of the five benchmarks losing at most 6\.2%\. The retention comparison is particularly informative in Overcooked\-AI, RWARE, and LBF: Shield, Full\-FT, RP, Lag\., and MACPO achieve zero observed SSR/WER alongside zero task return\. Their suppression is therefore accompanied by task failure, whereasCARVEmaintains positive returns in all three settings\. In SMACv2,CARVEmatches Shield and MACPO in WER while improving return by up to 80\.6%\. This pattern is consistent with the intended combination of retain\-aware subspace selection, KL retention, and teacher\-guided replacement coordination\. TrajDel preserves slightly higher mean utility, but the target convention remains elicitable on every benchmark\. Relative to TrajDel,CARVEgives up 2\.9% in mean utility while reducing mean WER by 80\.8%\. Fisher preserves utility well in SMACv2 and VMAS but does not consistently retain performance across the remaining tasks\. Overcooked\-AI exhibits the largest utility loss with a45\.8%45\.8\\%reduction\. Productive workflows can depend strongly on the convention being removed, making effective replacement coordination difficult\. Nevertheless, among methods attaining zero observed SSR and WER,CARVEachieves the highest retained utility; all corresponding baselines yield zero task return\. These comparisons show thatCARVEoffers the strongest observed retention at the lowest reported semantic risk in this setting\. The remaining utility gap may reflect both dependence on the removed convention and limitations of replacement coordination\.

Table 5\.Stronger unlearning audit under two different attacks on three VMAS forgetting tasks\. Bold numbers indicate the largest reported value among the three audit settings\.
#### 5\.2\.3\.Robustness with Stronger Elicitation under Different Unlearning Requests

We further test whether the unlearning performance persist under stronger elicitation across the GiveWay, Passage, and Flocking requests in VMAS dataset, using their respective MADDPG\([Lowe et al\., 2017](https://arxiv.org/html/2611.17917#bib.bib27)\), MAPPO\([Yu et al\., 2022](https://arxiv.org/html/2611.17917#bib.bib15)\), and IPPO\([De Witt et al\., 2020](https://arxiv.org/html/2611.17917#bib.bib16)\)checkpoints\. We use two red\-team attacks\. The role\-permutation attack swaps agent identities before policy evaluation and maps the resulting actions back before stepping\. The role\-invariant semantic request remains unchanged, so this attack tests whether reassignment of the same roles to different agent indices can recover the target convention\. The held\-out\-context attack evaluates the same edited checkpoint on unseen VMAS context seeds\{4001,4002,4003\}\\\{4001,4002,4003\\\}, which vary initial states and procedural task contexts\. No additional parameter updates are performed during these tests;CARVEexecutes its merged actors, and the semantic monitor scores the resulting trajectories\.

Table[5](https://arxiv.org/html/2611.17917#S5.T5)reports standard\-context WER \(Norm\. WER\), WER under role permutation \(Role WER\), and the maximum WER across held\-out\-context seeds \(Ctx\. WER\)\. Under role permutation,CARVEachieves the lowest reported Role WER for all three unlearning requests\. Relative to the original policies, the reduction across requests has a median of 53\.0% and reaches 63\.6%\. Across parameter\-editing baselines, the median Role WER reductions range from 9\.3% to 58\.9%, with the largest reduction against MACPO reaching 46\.9%\. Compared with Shield, the median reduction is 12\.0%, and the largest reaches 42\.1%\. These comparisons show thatCARVE’s advantage persists when the same roles are reassigned to different agent indices\. Under held\-out contexts,CARVEreduces Ctx\. WER relative to the original policies by a median of 54\.6%, with the largest reduction reaching 64\.9%\. Its median Ctx\. WER reductions against parameter\-editing baselines range from 5\.2% to 59\.0%, including 22\.2% against MACPO\. Relative to Shield, the median reduction is 10\.2%, and the largest reaches 48\.2%\.CARVEachieves the lowest Ctx\. WER in GiveWay and Passage, while RP has slightly lower Ctx\. WER in Flocking\. When risk is summarized by the maximum over the nominal, role\-permutation, and held\-out\-context protocols for each method and request,CARVEnevertheless achieves the lowest value for all three requests\. This supports its comparative robustness across the tested role assignments and unseen contexts, while residual strategy completions remain possible\.

Table 6\.Unlearning efficiency across five MARL benchmarks\. Speedup isTbackbone/TunlearnT\_\{\\mathrm\{backbone\}\}/T\_\{\\mathrm\{unlearn\}\}; Infer\. is inference latency\.

### 5\.3\.Unlearning Efficiency

Table[6](https://arxiv.org/html/2611.17917#S5.T6)shows thatCARVEachieves the highest reported unlearning time speedup across all five benchmarks, ranging from101\.92×101\.92\\timesto584\.83×584\.83\\timesrelative to original backbone training, including loading data, gradient\-mask construction, and update time\. It is1\.251\.25–2\.292\.29times faster than Full\-FT\. Its inference latency matches the original policy, while Shield increases latency by19\.2%19\.2\\%–31\.2%31\.2\\%\. This overhead arises from additional runtime automaton monitoring and action filtering\. Figure[2](https://arxiv.org/html/2611.17917#acmlabel2)reportsParamFrac, the fraction of actor parameters changed relative to the original checkpoint\.CARVEmodifies13\.1%13\.1\\%on average, ranging from10\.5%10\.5\\%to20\.0%20\.0\\%\), compared with90\.6%90\.6\\%–100\.0%100\.0\\%for RP, Lag\., MACPO, Full\-FT, and TrajDel\. Fisher also has a small edit footprint of12\.0%12\.0\\%in every benchmark, but records lower unlearning speedups thanCARVE\. This illustrates that edit locality alone does not determine runtime, which also depends on parameter selection and optimization costs\.CARVEoptimizes only the coefficientsα\\alphain the retain\-aware updateΔ​θ=Bf​α\\Delta\\theta=B\_\{f\}\\alpha\. The accepted update is merged into the actors, enabling deployment without an inference\-time monitor or shield\.

Figure 2\.Fraction of actor parameters modified by each unlearning method across five MARL benchmarksBar chart showing the fraction of actor parameters modified by each unlearning method across five MARL benchmarks\.![Refer to caption](https://arxiv.org/html/2611.17917v1/flocking_original_action_norm.png)![Refer to caption](https://arxiv.org/html/2611.17917v1/flocking_carve_action_norm.png)\(a\) Original\(b\)CARVEFigure 3\.Visualization for VMAS Flocking\. Rows from top to bottom show the rotated physical trajectory, the semantic score for leader\-follower dominance, per\-agent action norms, and occupancy of the monitor statesq0,q1,q2q\_\{0\},q\_\{1\},q\_\{2\}\.
### 5\.4\.Visualization

We use VMAS Flocking to inspect whether semantic unlearning changes the policy itself\. The target convention is leader\-follower dominance: agents move cohesively while one agent exerts disproportionate control\. Figure[3](https://arxiv.org/html/2611.17917#S5.F3)compares the original restored actor andCARVEon the same rollout diagnostics\. In the monitor row,q0q\_\{0\},q1q\_\{1\}, andq2q\_\{2\}denote low\-risk, recoverable, and accepted forbidden\-convention states\. The original actor shows the target signature across all views: clustered trajectories, a semantic\-score peak above the accepting threshold, uneven action norms, and visibleq1/q2q\_\{1\}/q\_\{2\}monitor occupancy\.CARVEweakens this signature\. Its trajectories are less organized around a single leading path, the semantic score remains below the accepting region in the shown rollout, action magnitudes are more balanced, and monitor mass shifts towardq0q\_\{0\}\. This supports the main claim:CARVEremoves the leader\-follower convention while preserving coordinated flocking, rather than merely blocking actions at runtime\.

### 5\.5\.Ablation Study

We ablate five components on VMAS GiveWay, Passage, and Flocking using the same checkpoints, semantic requests, retain sets, and audit contexts\. The full model uses the complete pipeline in Eqs\. \([12](https://arxiv.org/html/2611.17917#S4.E12)\)–\([25](https://arxiv.org/html/2611.17917#S4.E25)\)\. The ablated variants replace the attribution scores in Eqs\. \([12](https://arxiv.org/html/2611.17917#S4.E12)\)–\([13](https://arxiv.org/html/2611.17917#S4.E13)\) with random sparse weights, replace the retain\-aware eigenspace in Eq\. \([19](https://arxiv.org/html/2611.17917#S4.E19)\) with semantic\-gradient coordinates, dropℒretain\\mathcal\{L\}\_\{\\mathrm\{retain\}\}from Eq\. \([23](https://arxiv.org/html/2611.17917#S4.E23)\), drop the counterfactual teacher in Eqs\. \([21](https://arxiv.org/html/2611.17917#S4.E21)\)–\([22](https://arxiv.org/html/2611.17917#S4.E22)\), or optimize the full actorθ\\thetainstead of the subspace update in Eq\. \([20](https://arxiv.org/html/2611.17917#S4.E20)\)\. Table[7](https://arxiv.org/html/2611.17917#S5.T7)reports task averages\. Causal attribution increasesΔ\\DeltaSSR andΔ\\DeltaWER by147\.1%147\.1\\%and142\.0%142\.0\\%, respectively, compared with random weights, supporting more effective targeting of the convention\. Retain\-aware selection improvesΔ\\DeltaWER by27\.3%27\.3\\%and utility by10\.5%10\.5\\%, while the retention loss improves utility by54\.9%54\.9\\%with nearly unchanged semantic reductions\. These components therefore provide complementary protection for retained cooperation\. Counterfactual replacement improvesΔ\\DeltaWER by43\.1%43\.1\\%and mean return by61\.6%61\.6\\%, supporting productive alternative coordination\. Subspace restriction improves utility by35\.5%35\.5\\%while reducing the changed\-parameter fraction by87\.8%87\.8\\%, indicating that limiting edit scope helps preserve task competence\. Together, these ablations show that the components complement one another in balancing effective convention removal with efficient retained cooperation\.

Table 7\.Ablation study on VMAS GiveWay, Passage, and Flocking\.Δ\\DeltaSSR andΔ\\DeltaWER are reductions relative to the restored original checkpoint\.

## 6\.Conclusion

This paper studied semantic coordination\-strategy unlearning, where a trained cooperative policy must abandon a role\-invariant convention while retaining useful cooperation\.CARVEcombines causal coordination attribution, retain\-aware subspace updates, counterfactual re\-coordination, and adversarial auditing, then merges the accepted update into decentralized actors\. Experiments across five MARL benchmarks show thatCARVEbalances target\-convention removal with the preservation of cooperative task performance and no additional inference latency\.

## References

- Arulkumaranet al\.\(2017\)K\. Arulkumaran, M\. P\. Deisenroth, M\. Brundage, and A\. A\. BharathDeep reinforcement learning: a brief survey\.IEEE signal processing magazine34\(6\),pp\. 26–38\.Cited by:[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px1.p1.1)\.
- Bettiniet al\.\(2022\)M\. Bettini, R\. Kortvelesy, J\. Blumenkamp, and A\. ProrokVmas: a vectorized multi\-agent simulator for collective robot learning\.InInternational symposium on distributed autonomous robotic systems,Cham,pp\. 42–56\.Cited by:[Table A1](https://arxiv.org/html/2611.17917#A0.T1.4.1.6.1.1.1),[Table A1](https://arxiv.org/html/2611.17917#A0.T1.4.1.7.1.1.1),[Table A1](https://arxiv.org/html/2611.17917#A0.T1.4.1.8.1.1.1),[§1](https://arxiv.org/html/2611.17917#S1.p1.1),[§1](https://arxiv.org/html/2611.17917#S1.p3.1),[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px1.p1.1),[§5\.1\.2](https://arxiv.org/html/2611.17917#S5.SS1.SSS2.p1.1)\.
- Bourtouleet al\.\(2021\)L\. Bourtoule, V\. Chandrasekaran, C\. A\. Choquette\-Choo, H\. Jia, A\. Travers, B\. Zhang, D\. Lie, and N\. PapernotMachine unlearning\.In2021 IEEE symposium on security and privacy \(SP\),Virtual Site,pp\. 141–159\.Cited by:[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px2.p1.1)\.
- Carrollet al\.\(2019\)M\. Carroll, R\. Shah, M\. K\. Ho, T\. Griffiths, S\. Seshia, P\. Abbeel, and A\. DraganOn the utility of learning about humans for human\-ai coordination\.Advances in neural information processing systems32\.Cited by:[Table A1](https://arxiv.org/html/2611.17917#A0.T1.4.1.3.1.1.1),[§1](https://arxiv.org/html/2611.17917#S1.p1.1),[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px1.p1.1),[§5\.1\.2](https://arxiv.org/html/2611.17917#S5.SS1.SSS2.p1.1)\.
- Changet al\.\(2022\)Q\. Chang, Y\. Jiang, F\. Zheng, M\. Bennis, and X\. YouCooperative edge caching via multi agent reinforcement learning in fog radio access networks\.InICC 2022\-IEEE International Conference on Communications,Seoul, South Korea,pp\. 3641–3646\.Cited by:[§1](https://arxiv.org/html/2611.17917#S1.p1.1)\.
- Chenet al\.\(2026a\)L\. Chen, Y\. Gao, J\. Wang, K\. Ji, and H\. ChenDo system prompts leave behavioral fingerprints? a large\-scale empirical study of clone detection via output similarity\.External Links:2608\.24461Cited by:[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px3.p1.1)\.
- Chenet al\.\(2026b\)L\. Chen, K\. Ji, and M\. GuoAttacking the trusted imagination: oracle\-level integrity attacks on imagine\-then\-act world models\.arXiv preprint arXiv:2606\.22966\.Cited by:[Appendix F](https://arxiv.org/html/2611.17917#A6.p1.1)\.
- Chiet al\.\(2025\)X\. Chi, H\. Chen, Z\. Ni, H\. Sun, P\. Sun, and D\. YuH\-STEP: Heuristic Stable Edge Service Entity Placement for Mobile Virtual Reality Systems\.IEEE Transactions on Mobile Computing24\(8\),pp\. 7377–7388\.Cited by:[Appendix F](https://arxiv.org/html/2611.17917#A6.p1.1)\.
- Daiet al\.\(2025\)Q\. Dai, H\. Chen, Z\. Ni, Y\. Yang, L\. Chen, X\. Fan, and X\. LiReinforcement learning\-based resource reservation for mobile edge computing with probable failure\.IEEE Transactions on Vehicular Technology74\(5\),pp\. 7985–7996\.Cited by:[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px1.p1.1)\.
- De Wittet al\.\(2020\)C\. S\. De Witt, T\. Gupta, D\. Makoviichuk, V\. Makoviychuk, P\. H\. Torr, M\. Sun, and S\. WhitesonIs independent learning all you need in the starcraft multi\-agent challenge?\.External Links:2011\.09533Cited by:[Table A1](https://arxiv.org/html/2611.17917#A0.T1.4.1.3.5.1.1),[Table A1](https://arxiv.org/html/2611.17917#A0.T1.4.1.8.5.1.1),[§5\.2\.3](https://arxiv.org/html/2611.17917#S5.SS2.SSS3.p1.1)\.
- Elliset al\.\(2023\)B\. Ellis, J\. Cook, S\. Moalla, M\. Samvelyan, M\. Sun, A\. Mahajan, J\. Foerster, and S\. WhitesonSmacv2: an improved benchmark for cooperative multi\-agent reinforcement learning\.Advances in neural information processing systems36,pp\. 37567–37593\.Cited by:[Table A1](https://arxiv.org/html/2611.17917#A0.T1.4.1.2.1.1.1),[§1](https://arxiv.org/html/2611.17917#S1.p1.1),[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px1.p1.1),[§5\.1\.2](https://arxiv.org/html/2611.17917#S5.SS1.SSS2.p1.1)\.
- Foersteret al\.\(2018\)J\. Foerster, G\. Farquhar, T\. Afouras, N\. Nardelli, and S\. WhitesonCounterfactual multi\-agent policy gradients\.InProceedings of the AAAI conference on artificial intelligence,Vol\.32,New Orleans, Louisiana, USA\.Cited by:[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px1.p1.1)\.
- Fosteret al\.\(2024\)J\. Foster, S\. Schoepf, and A\. BrintrupFast machine unlearning without retraining through selective synaptic dampening\.InProceedings of the AAAI conference on artificial intelligence,Vol\.38,Vancouver, Canada,pp\. 12043–12051\.Cited by:[Table 2](https://arxiv.org/html/2611.17917#S5.T2.2.9.1.1.1)\.
- Gonget al\.\(2024\)C\. Gong, K\. Li, J\. Yao, and T\. WangTrajdeleter: enabling trajectory forgetting in offline reinforcement learning agents\.InProceedings of the 31st Annual Network and Distributed System Security Symposium \(NDSS\),San Diego, CA, USA,pp\.\.Cited by:[§1](https://arxiv.org/html/2611.17917#S1.p4.1),[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px2.p1.1),[Table 2](https://arxiv.org/html/2611.17917#S5.T2.2.10.1.1.1)\.
- Grabisch and Roubens \(1999\)M\. Grabisch and M\. RoubensAn axiomatic approach to the concept of interaction among players in cooperative games\.International Journal of game theory28\(4\),pp\. 547–565\.Cited by:[§4\.1](https://arxiv.org/html/2611.17917#S4.SS1.p2.3)\.
- Guet al\.\(2021\)S\. Gu, J\. G\. Kuba, M\. Wen, R\. Chen, Z\. Wang, Z\. Tian, J\. Wang, A\. Knoll, and Y\. YangMulti\-agent constrained policy optimisation\.External Links:2110\.02793Cited by:[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px3.p1.1),[Table 2](https://arxiv.org/html/2611.17917#S5.T2.2.7.1.1.1),[Table 2](https://arxiv.org/html/2611.17917#S5.T2.2.8.1.1.1)\.
- Guoet al\.\(2023\)J\. Guo, A\. Li, L\. Wang, and C\. LiuPolicycleanse: backdoor detection and mitigation for competitive reinforcement learning\.In2023 IEEE/CVF International Conference on Computer Vision \(ICCV\),Paris, France,pp\. 4676–4685\.Cited by:[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px3.p1.1)\.
- Hammondet al\.\(2021\)L\. Hammond, A\. Abate, J\. Gutierrez, and M\. WooldridgeMulti\-agent reinforcement learning with temporal logic specifications\.InProceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems,Richland, SC,pp\. 583–592\.Cited by:[§1](https://arxiv.org/html/2611.17917#S1.p3.1),[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px3.p1.1)\.
- Icarteet al\.\(2018\)R\. T\. Icarte, T\. Klassen, R\. Valenzano, and S\. McIlraithUsing reward machines for high\-level task specification and decomposition in reinforcement learning\.InInternational Conference on Machine Learning,Stockholm, Sweden,pp\. 2107–2116\.Cited by:[§1](https://arxiv.org/html/2611.17917#S1.p3.1),[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px3.p1.1)\.
- Illman and Temple \(2019\)E\. Illman and P\. TempleCalifornia consumer privacy act\.The Business Lawyer75\(1\),pp\. 1637–1646\.Cited by:[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px2.p1.1)\.
- Leiboet al\.\(2021\)J\. Z\. Leibo, E\. A\. Dueñez\-Guzman, A\. Vezhnevets, J\. P\. Agapiou, P\. Sunehag, R\. Koster, J\. Matyas, C\. Beattie, I\. Mordatch, and T\. GraepelScalable evaluation of multi\-agent reinforcement learning with melting pot\.InInternational Conference on Machine Learning,Virtual Site,pp\. 6187–6199\.Cited by:[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2025\)N\. Li, C\. Zhou, Y\. Gao, H\. Chen, Z\. Zhang, B\. Kuang, and A\. FuMachine unlearning: taxonomy, metrics, applications, challenges, and prospects\.IEEE Transactions on Neural Networks and Learning Systems36\(8\),pp\. 13709–13729\.Cited by:[§3\.2](https://arxiv.org/html/2611.17917#S3.SS2.p1.3)\.
- Li and Wang \(2026\)Q\. Li and X\. WangJeva: system one models from first principles\.arXiv preprint arXiv:2610\.18773\.Cited by:[Appendix F](https://arxiv.org/html/2611.17917#A6.p1.1)\.
- Lianget al\.\(2026a\)H\. Liang, K\. Guan, W\. Wu, R\. Ge, J\. Jiang, and D\. ChengDecoupled attribute subspace unlearning for private federated recommender systems\.IEEE Transactions on Dependable and Secure Computingearly access,pp\. 1–13\.Cited by:[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px2.p1.1)\.
- Lianget al\.\(2026b\)H\. Liang, W\. Wu, Z\. Gong, Z\. He, Y\. Gong, J\. Jiang, and D\. ChengFederated unlearning via representation misdirection with adaptive anchor generation\.IEEE Transactions on Dependable and Secure Computingearly access,pp\. 1–14\.Cited by:[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px2.p1.1)\.
- Loweet al\.\(2017\)R\. Lowe, Y\. I\. Wu, A\. Tamar, J\. Harb, O\. Pieter Abbeel, and I\. MordatchMulti\-agent actor\-critic for mixed cooperative\-competitive environments\.Advances in neural information processing systems30,pp\. 6382–6393\.Cited by:[Table A1](https://arxiv.org/html/2611.17917#A0.T1.4.1.6.5.1.1),[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px1.p1.1),[§5\.2\.3](https://arxiv.org/html/2611.17917#S5.SS2.SSS3.p1.1)\.
- Maoet al\.\(2026\)X\. Mao, R\. Qian, L\. Wang, X\. Yi, M\. Li, L\. Chen, Y\. Gao, X\. Zheng, and C\. Wang" You’re right, let me fix it": how llm agents damage correct work when falsely accused\.arXiv preprint arXiv:2609\.32616\.Cited by:[Appendix F](https://arxiv.org/html/2611.17917#A6.p1.1)\.
- Oesterlinget al\.\(2024\)A\. Oesterling, J\. Ma, F\. Calmon, and H\. LakkarajuFair machine unlearning: data removal while mitigating disparities\.InInternational Conference on Artificial Intelligence and Statistics,Valencia, Spain,pp\. 3736–3744\.Cited by:[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px2.p1.1)\.
- Papoudakiset al\.\(2021\)G\. Papoudakis, F\. Christianos, L\. Schäfer, and S\. V\. AlbrechtBenchmarking multi\-agent deep reinforcement learning algorithms in cooperative tasks\.InAdvances in Neural Information Processing Systems 2021 Datasets and Benchmarks Track,Virtual Site,pp\.\.Cited by:[Table A1](https://arxiv.org/html/2611.17917#A0.T1.4.1.4.1.1.1),[Table A1](https://arxiv.org/html/2611.17917#A0.T1.4.1.5.1.1.1),[§1](https://arxiv.org/html/2611.17917#S1.p1.1),[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px1.p1.1),[§5\.1\.2](https://arxiv.org/html/2611.17917#S5.SS1.SSS2.p1.1)\.
- Rashidet al\.\(2020\)T\. Rashid, M\. Samvelyan, C\. S\. De Witt, G\. Farquhar, J\. Foerster, and S\. WhitesonMonotonic value function factorisation for deep multi\-agent reinforcement learning\.Journal of Machine Learning Research21\(178\),pp\. 1–51\.Cited by:[Table A1](https://arxiv.org/html/2611.17917#A0.T1.4.1.4.5.1.1),[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px1.p1.1)\.
- Ruhdorferet al\.\(2024\)R\. Ruhdorfer, K\. Bullard, A\. Gleave, E\. Grefenstette, T\. Rocktäschel, and M\. SamvelyanThe overcooked generalisation challenge: evaluating cooperation with novel partners in unknown environments using unsupervised environment design\.External Links:2406\.17949Cited by:[Table A1](https://arxiv.org/html/2611.17917#A0.T1.4.1.3.1.1.1),[§1](https://arxiv.org/html/2611.17917#S1.p1.1),[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px1.p1.1),[§5\.1\.2](https://arxiv.org/html/2611.17917#S5.SS1.SSS2.p1.1)\.
- Samvelyanet al\.\(2019\)M\. Samvelyan, T\. Rashid, C\. S\. De Witt, G\. Farquhar, N\. Nardelli, T\. G\. Rudner, C\. Hung, P\. H\. Torr, J\. Foerster, and S\. WhitesonThe starcraft multi\-agent challenge\.InProceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems,Richland, SC,pp\. 2186–2188\.Cited by:[§1](https://arxiv.org/html/2611.17917#S1.p1.1),[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px1.p1.1)\.
- Shapley \(1952\)L\. S\. ShapleyA value for n\-person games\.PaperTechnical ReportP\-295,RAND Corporation,Santa Monica, CA\.Cited by:[§4\.1](https://arxiv.org/html/2611.17917#S4.SS1.p2.3)\.
- Sunehaget al\.\(2018\)P\. Sunehag, G\. Lever, A\. Gruslys, W\. M\. Czarnecki, V\. Zambaldi, M\. Jaderberg, M\. Lanctot, N\. Sonnerat, J\. Z\. Leibo, K\. Tuyls,et al\.Value\-decomposition networks for cooperative multi\-agent learning\.InProceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems,Stockholm, Sweden,pp\. 2085–2087\.Cited by:[Table A1](https://arxiv.org/html/2611.17917#A0.T1.4.1.5.5.1.1),[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2024\)W\. Wang, Z\. Tian, C\. Zhang, and S\. YuMachine unlearning: a comprehensive survey\.External Links:2405\.07406Cited by:[§3\.2](https://arxiv.org/html/2611.17917#S3.SS2.p1.3)\.
- Wuet al\.\(2025a\)W\. Wu, Y\. Gong, J\. Jiang, C\. Hu, X\. Zhou, and D\. ChengDefending against attribute inference attacks in post\-training of recommendation systems via unlearning\.In2025 IEEE 41st International Conference on Data Engineering \(ICDE\),Hong Kong SAR,pp\. 2656–2669\.Cited by:[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px2.p1.1)\.
- Wuet al\.\(2026\)W\. Wu, Z\. He, H\. Liang, Y\. Gong, J\. Jiang, C\. Hu, and D\. ChengREMISVFU: vertical federated unlearning via representation misdirection for intermediate output feature\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,Singapore,pp\. 26983–26991\.Cited by:[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px2.p1.1)\.
- Wuet al\.\(2025b\)W\. Wu, J\. Jiang, and C\. HuAegis: post\-training attribute unlearning in federated recommender systems against attribute inference attacks\.InProceedings of the ACM on Web Conference 2025,Sydney, Australia,pp\. 3783–3793\.Cited by:[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px2.p1.1)\.
- Wuet al\.\(2025c\)W\. Wu, H\. Liang, T\. Tu, J\. Jiang, C\. Hu, and D\. ChengMimir: data\-free federated unlearning through client\-specific prompt generation for personalized models\.IEEE Transactions on Mobile Computing24\(10\),pp\. 10537–10556\.Cited by:[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px2.p1.1)\.
- Wuet al\.\(2025d\)W\. Wu, H\. Liang, J\. Yuan, J\. Jiang, K\. Y\. Wang, C\. Hu, X\. Zhou, and D\. ChengZero\-shot federated unlearning via transforming from data\-dependent to personalized model\-centric\.\.InIJCAI,Montreal, Canada,pp\. 6588–6596\.Cited by:[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px2.p1.1)\.
- Xuet al\.\(2024\)J\. Xu, Z\. Wu, C\. Wang, and X\. JiaMachine unlearning: solutions and challenges\.IEEE Transactions on Emerging Topics in Computational Intelligence8\(3\),pp\. 2150–2168\.Cited by:[§3\.2](https://arxiv.org/html/2611.17917#S3.SS2.p1.3)\.
- Yanget al\.\(2022\)M\. Yang, D\. Xu, Q\. Cui, Z\. Wen, and P\. XuAn efficient fisher matrix approximation method for large\-scale neural network optimization\.IEEE Transactions on Pattern Analysis and Machine Intelligence45\(5\),pp\. 5391–5403\.Cited by:[§4\.2](https://arxiv.org/html/2611.17917#S4.SS2.p3.1)\.
- Yeet al\.\(2025\)D\. Ye, T\. Zhu, C\. Zhu, D\. Wang, K\. Gao, Z\. Shi, S\. Shen, W\. Zhou, and M\. XueReinforcement unlearning\.InProceedings of the 31st Annual Network and Distributed System Security Symposium \(NDSS\),San Diego, CA, USA,pp\.\.Cited by:[§1](https://arxiv.org/html/2611.17917#S1.p4.1),[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px2.p1.1)\.
- Youet al\.\(2026\)L\. You, Y\. Bian, and L\. CaoJoint distribution–informed shapley values for sparse counterfactual explanations\.InInternational Conference on Learning Representations,Vol\.2026,Seoul, South Korea,pp\. 100183–100205\.Cited by:[§4\.1](https://arxiv.org/html/2611.17917#S4.SS1.p2.3)\.
- Yuet al\.\(2022\)C\. Yu, A\. Velu, E\. Vinitsky, J\. Gao, Y\. Wang, A\. Bayen, and Y\. WuThe surprising effectiveness of ppo in cooperative multi\-agent games\.Advances in neural information processing systems35,pp\. 24611–24624\.Cited by:[Table A1](https://arxiv.org/html/2611.17917#A0.T1.4.1.2.5.1.1),[Table A1](https://arxiv.org/html/2611.17917#A0.T1.4.1.7.5.1.1),[§1](https://arxiv.org/html/2611.17917#S1.p3.1),[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px1.p1.1),[§5\.2\.3](https://arxiv.org/html/2611.17917#S5.SS2.SSS3.p1.1)\.
- Zaeem and Barber \(2020\)R\. N\. Zaeem and K\. S\. BarberThe effect of the gdpr on privacy policies: recent progress and future promise\.ACM Transactions on Management Information Systems \(TMIS\)12\(1\),pp\. 1–20\.Cited by:[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px2.p1.1)\.
- Zuoet al\.\(2026\)R\. Zuo, Q\. Huang, M\. Li, Z\. Zhang, S\. Khan, and Q\. QiuMUTE: return\-preserving communication unlearning for efficient multi\-agent coordination\.External Links:2607\.03473Cited by:[§2](https://arxiv.org/html/2611.17917#S2.SS0.SSS0.Px3.p1.1)\.

Table A1\.Concrete scenarios used in the main experiments\. Each row reports one implemented semantic request, the corresponding retained behavior, and the single backbone used for the reported checkpoint\.## Appendix AAlgorithm Details

### A1\.CARVEPseudocode

As shown in Algorithm[1](https://arxiv.org/html/2611.17917#alg1),CARVEis a post\-training unlearning framework\. It starts from a restored cooperative policy, uses the automaton monitor to locate decisions that causally support the forbidden convention, restricts the update to retain\-aware strategy\-sensitive directions, distills replacement coordination, and finally audits the edited actor under stronger elicitation contexts\. The automaton and red\-team contexts are used during editing and verification; the delivered policy is the edited actor itself and does not require an inference\-time shield\.

### A2\.Complexity Analysis

LetMMbe the number of collected rollouts,HHthe horizon,NNthe number of agents,PPthe number of actor parameters,CCthe number of sampled coalitions per target\-positive decision,KKthe retained subspace rank,ffthe editable actor\-parameter fraction,UUthe number of projected optimization updates,RRthe number of audit contexts, andEEthe number of audit episodes per context\. Monitor annotation is linear in trajectory length,O⁡\(M​H\)O\(MH\), excluding simulator cost\. Attribution requiresO⁡\(M​H​N​C\)O\(MHNC\)semantic\-critic evaluations in the sampled\-coalition implementation, rather than enumerating all2N2^\{N\}coalitions\. Building the retain\-aware subspace costsO⁡\(M​H​N​P\)O\(MHNP\)to accumulate score\-gradient statistics plusO⁡\(Ieig​K​P\)O\(I\_\{\\mathrm\{eig\}\}KP\)forIeigI\_\{\\mathrm\{eig\}\}iterative generalized\-eigen updates when diagonal or low\-rank sensitivities are used; a denseP×PP\\times Peigensolver would beO⁡\(P3\)O\(P^\{3\}\)and is not the intended implementation\. Selecting the edit mask costsO⁡\(P\)O\(P\)to score coordinates plus a top\-⌈f​P⌉\\lceil fP\\rceilselection\. Projected optimization costsO⁡\(U​Bmb​P\)O\(UB\_\{\\mathrm\{mb\}\}P\)for minibatch actor forward/backward passes, but onlyf​PfPactor coordinates receive nonzero updates\. Materializing the masked subspace costsO⁡\(K​f​P\)O\(KfP\)\. The final audit costsO⁡\(R​E​H\)O\(REH\)simulator steps\. In practice, the dominant costs are simulator rollouts and actor\-gradient passes; memory isO⁡\(P\+K​f​P\)O\(P\+KfP\)for the actor and masked subspace, plus the rollout buffer\.

Algorithm 1CARVEsemantic coordination\-strategy unlearning\.0:Restored policy

π0\\pi\_\{0\}with actor parameters

θ0\\theta\_\{0\}; simulator

𝒢\\mathcal\{G\}; forbidden formula

φf\\varphi\_\{f\}; retain data

𝒟r\\mathcal\{D\}\_\{r\}; monitor automaton

𝒜φf\\mathcal\{A\}\_\{\\varphi\_\{f\}\}; edit rank

KK; actor edit fraction

ff; update budget

UU; audit contexts

𝒞audit\\mathcal\{C\}\_\{\\mathrm\{audit\}\}; minimum semantic reduction

δ\>0\\delta\>0; utility threshold

ρ\\rho; required consecutive passes

MM; audit interval

AA\.

0:Accepted edited policy

π−\\pi^\{\-\}, orNotAccepted\.

1:Evaluate baseline scores

S0=SSR⁡\(π0\)S\_\{0\}=\\mathrm\{SSR\}\(\\pi\_\{0\}\)and

W0=WER⁡\(π0\)W\_\{0\}=\\mathrm\{WER\}\(\\pi\_\{0\}\)on

𝒞audit\\mathcal\{C\}\_\{\\mathrm\{audit\}\}\.

2:Roll out

π0\\pi\_\{0\}in training and retain contexts; update product states

s¯t=\(st,qt\)\\bar\{s\}\_\{t\}=\(s\_\{t\},q\_\{t\}\)with

𝒜φf\\mathcal\{A\}\_\{\\varphi\_\{f\}\}\.

3:Mark target\-positive and retain decision points from the monitored trajectories\.

4:Fit

Qφ​\(s¯t,𝐚t\)Q^\{\\varphi\}\(\\bar\{s\}\_\{t\},\\mathbf\{a\}\_\{t\}\)for target\-convention risk and

QJ​\(s¯t,𝐚t\)Q^\{J\}\(\\bar\{s\}\_\{t\},\\mathbf\{a\}\_\{t\}\)for task value\.

5:foreach target\-positive decision pointdo

6:Sample coalitions and estimate

χi,t\\chi\_\{i,t\},

χi​j,t\(2\)\\chi^\{\(2\)\}\_\{ij,t\}by Eqs\. \([12](https://arxiv.org/html/2611.17917#S4.E12)\) and \([13](https://arxiv.org/html/2611.17917#S4.E13)\)\.

7:Convert attribution scores into weights

wi,tw\_\{i,t\}by Eq\. \([15](https://arxiv.org/html/2611.17917#S4.E15)\)\.

8:endfor

9:Compute

gf​\(τ\)g\_\{f\}\(\\tau\)by Eq\. \([16](https://arxiv.org/html/2611.17917#S4.E16)\); estimate

GfG\_\{f\}and

FrF\_\{r\}\.

10:Solve Eq\. \([19](https://arxiv.org/html/2611.17917#S4.E19)\); retain the top\-

KKcandidate directions

B=\[u1,…,uK\]B=\[u\_\{1\},\\ldots,u\_\{K\}\]\.

11:Select the top

⌈f​P⌉\\lceil fP\\rceilcoordinates by retain\-aware semantic sensitivity; form

mfm\_\{f\}and

Bf=diag⁡\(mf\)​BB\_\{f\}=\\operatorname\{diag\}\(m\_\{f\}\)Bas a heuristic sparsification\.

12:Fix

BfB\_\{f\}and restrict subsequent optimization to

θα=θ0\+Bf​α\\theta\_\{\\alpha\}=\\theta\_\{0\}\+B\_\{f\}\\alpha\.

13:Build the centralized replacement teacher

μ~t\\widetilde\{\\mu\}\_\{t\}on high\-risk product states using Eq\. \([21](https://arxiv.org/html/2611.17917#S4.E21)\)\.

14:Define the decentralized distillation loss

ℒcf\\mathcal\{L\}\_\{\\mathrm\{cf\}\}by Eq\. \([22](https://arxiv.org/html/2611.17917#S4.E22)\)\.

15:Initialize

α←0\\alpha\\leftarrow 0\.

16:for

u=1,…,Uu=1,\\ldots,Udo

17:Update

α\\alphawith Eq\. \([23](https://arxiv.org/html/2611.17917#S4.E23)\), using

ℒretain\\mathcal\{L\}\_\{\\mathrm\{retain\}\}from Eq\. \([24](https://arxiv.org/html/2611.17917#S4.E24)\)\.

18:if

umodA=0u\\bmod A=0or

u=Uu=Uthen

19:Audit

πθα\\pi\_\{\\theta\_\{\\alpha\}\}on

𝒞audit\\mathcal\{C\}\_\{\\mathrm\{audit\}\}; obtain

Su=SSR⁡\(πθα\)S\_\{u\}=\\mathrm\{SSR\}\(\\pi\_\{\\theta\_\{\\alpha\}\}\),

Wu=WER⁡\(πθα\)W\_\{u\}=\\mathrm\{WER\}\(\\pi\_\{\\theta\_\{\\alpha\}\}\), and

Vu=Util⁡\(πθα\)V\_\{u\}=\\mathrm\{Util\}\(\\pi\_\{\\theta\_\{\\alpha\}\}\)\.

20:if

S0−Su≥δS\_\{0\}\-S\_\{u\}\\geq\\deltaand

W0−Wu≥δW\_\{0\}\-W\_\{u\}\\geq\\deltaand

Vu≥ρV\_\{u\}\\geq\\rhothen

21:Accept

α⋆←α\\alpha^\{\\star\}\\leftarrow\\alpha; merge

Δ​θ=Bf​α⋆\\Delta\\theta=B\_\{f\}\\alpha^\{\\star\}into the actors\.

22:Report SSR/WER, retention metrics, and ParamFrac\.

23:return

π−=πθα⋆\\pi^\{\-\}=\\pi\_\{\\theta\_\{\\alpha^\{\\star\}\}\}\.

24:endif

25:endif

26:endfor

27:returnNotAccepted\.

## Appendix BEvaluation and Hyperparameters Details

Table[A1](https://arxiv.org/html/2611.17917#A0.T1)summarizes the task scenarios, role\-invariant semantic unlearning requests, retained behaviors, and backbone algorithms across five cooperative MARL environments\. Unless otherwise stated, SMACv2 and VMAS backbone training uses five random seeds,3\.0723\.072M environment frames per run, 32 evaluation episodes every122,880122\{,\}880frames, and checkpointing every307,200307\{,\}200frames\. SMACv2 uses four environment workers and 2,000 collected frames per batch, while VMAS uses 32 workers and 6,144 collected frames per batch\. For post\-checkpointCARVEruns, the default setting is 800 update steps, actor edit fractionf=0\.12f=0\.12, retain\-action weightλr=0\.5\\lambda\_\{r\}=0\.5, learning rate2×10−42\\times 10^\{\-4\},β=1\\beta=1,K=8K=8,λ0=10−3\\lambda\_\{0\}=10^\{\-3\}andλ=8\\lambda=8\. For deterministic actors such as MADDPG, candidate actions are generated by adding independent Gaussian noiseε∼𝒩⁡\(0,0\.152​I\)\\varepsilon\\sim\\mathcal\{N\}\(0,0\.15^\{2\}I\)in theatanh\\operatorname\{atanh\}latent space of normalized original actions, followed by atanh\\tanhtransformation and rescaling to the native action range\.

## Appendix CParameter Sensitivity

We sweep the hyperparameters that control the erasure–retention frontier on the three VMAS post\-checkpoint tasks: GiveWay, Passage, and Flocking\. The validation grid is update steps in \{200, 400, 800, 1200\}, edit fraction in \{0\.03, 0\.06, 0\.12, 0\.24, 0\.48\}, retain weight in \{0\.1, 0\.5, 1\.0, 2\.0, 4\.0\}, and learning rate in \{10−410^\{\-4\},2×10−42\\times 10^\{\-4\},5×10−45\\times 10^\{\-4\}\}\. Table[A2](https://arxiv.org/html/2611.17917#A3.T2)reports the average semantic reduction, retained utility, task reward, and parameter locality together\. Increasing the edit fraction from0\.060\.06to0\.120\.12improvesΔ\\DeltaWER by16\.7%16\.7\\%, whereas expanding it to0\.480\.48yields only a further1\.5%1\.5\\%gain with four times as many changed parameters\. Increasing the retain weight from0\.50\.5to4\.04\.0reducesΔ\\DeltaWER by44\.6%44\.6\\%, with little additional utility retention\. Among the tested learning rates,2×10−42\\times 10^\{\-4\}achieves the strongest semantic reductions\. More updates do not consistently improve forgetting:800800steps increase mean reward by50\.1%50\.1\\%over200200steps with comparable semantic reductions, while12001200steps weaken forgetting\. These results support the default configuration as a practical balance between convention removal, task performance, and edit locality\.

Table A2\.Parameter sensitivity on VMAS\.
## Appendix DTeacher–Actor Gap Experiment

Marginal distillation may discard joint\-action correlations encoded by the centralized teacher\. Table[A3](https://arxiv.org/html/2611.17917#A4.T3)therefore compares the joint teacherTTwith the independently sampled edited actorAsA\_\{s\}on paired nominal episodes, using one trained checkpoint per VMAS scenario\. Both are nominal\-context score diagnostics\. The observed gap is scenario\-dependent\. GiveWay shows nearly unchanged semantic scores and return\. In Passage, the actor’s SSR and WER point estimates decrease by0\.0230\.023and0\.0220\.022, respectively, alongside a higher return point estimate\. In Flocking, they increase by0\.0190\.019and0\.0390\.039, respectively, alongside a lower return point estimate\. Thus, the actor is not uniformly worse than the teacher\. The teacher’s local, critic\-based KL\-regularized objective does not guarantee dominance on episode\-level metrics\. All paired bootstrap 95% confidence intervals forJ⁡\(As\)−J⁡\(T\)J\(A\_\{s\}\)\-J\(T\)contain zero, leaving return differences statistically unresolved\. This does not establish equivalence, particularly given the wide Passage interval\. The comparison quantifies the combined effects of correlation loss, local observability, and constrained actor fitting\.

Table A3\.Joint\-teacher versus actor performance on three VMAS scenarios\. WER is measured in the nominal context\. Confidence intervals bootstrap paired evaluation episodes\.
## Appendix ETheory

We analyze attribution, surrogate subspace selection, the replacement teacher, conditional return retention, and decentralized information limits\.

##### Attribution consistency\.

###### Proposition 0\(Shapley efficiency and estimation error\)\.

For a fixed intervention baseline, letχi,t\\chi\_\{i,t\}andχ^i,t\\widehat\{\\chi\}\_\{i,t\}be the exact Shapley values ofvtv\_\{t\}and its estimatev^t\\widehat\{v\}\_\{t\}\. IfmaxS⁡\|v^t​\(S\)−vt​\(S\)\|≤εv\\max\_\{S\}\|\\widehat\{v\}\_\{t\}\(S\)\-v\_\{t\}\(S\)\|\\leq\\varepsilon\_\{v\}, then

\(A1\)∑iχi,t=vt​\(𝒩\)−vt​\(∅\),\|χ^i,t−χi,t\|≤2​εv\.\\sum\_\{i\}\\chi\_\{i,t\}=v\_\{t\}\(\\mathcal\{N\}\)\-v\_\{t\}\(\\emptyset\),\\qquad\|\\widehat\{\\chi\}\_\{i,t\}\-\\chi\_\{i,t\}\|\\leq 2\\varepsilon\_\{v\}\.The pair term in Eq\. \([13](https://arxiv.org/html/2611.17917#S4.E13)\) vanishes for additivevtv\_\{t\}\. These statements concern signed attributions before clipping\.

Shapley values average permutation marginals, whose sum telescopes fromvt​\(∅\)v\_\{t\}\(\\emptyset\)tovt​\(𝒩\)v\_\{t\}\(\\mathcal\{N\}\)\. Each estimated marginal has error at most2​εv2\\varepsilon\_\{v\}\. Additive coalition functions have zero second differences\. ∎

##### Retain\-aware subspace\.

LetGf,Fr⪰0G\_\{f\},F\_\{r\}\\succeq 0andM=Fr\+λ0​IM=F\_\{r\}\+\\lambda\_\{0\}I, withλ0\>0\\lambda\_\{0\}\>0\. The semantic sensitivity surrogate is

\(A2\)Δ​θ⊤​Gf​Δ​θ=𝔼⁡\[\(gf​\(τ\)⊤​Δ​θ\)2\]\.\\Delta\\theta^\{\\top\}G\_\{f\}\\Delta\\theta=\\mathbb\{E\}\[\(g\_\{f\}\(\\tau\)^\{\\top\}\\Delta\\theta\)^\{2\}\]\.
###### Theorem 2\(Retain\-normalized sensitivity\)\.

For1≤K≤d1\\leq K\\leq d, the topKKgeneralized eigenvectors ofGf​u=λ​M​uG\_\{f\}u=\\lambda Mu, normalized byB⊤​M​B=IKB^\{\\top\}MB=I\_\{K\}, solve

\(A3\)maxB∈ℝd×K⁡tr⁡\(B⊤​Gf​B\)s\.t\.B⊤​M​B=IK\.\\max\_\{B\\in\\mathbb\{R\}^\{d\\times K\}\}\\operatorname\{tr\}\(B^\{\\top\}G\_\{f\}B\)\\quad\\mathrm\{s\.t\.\}\\quad B^\{\\top\}MB=I\_\{K\}\.The optimum is the sum of the topKKgeneralized eigenvalues\.

SetY=M1/2​BY=M^\{1/2\}BandA=M−1/2GfM−1/2A=M^\{\-1/2\}G\_\{f\}M^\{\-1/2\}\. The problem becomesmaxY⊤​Y=IK⁡tr⁡\(Y⊤​A​Y\)\\max\_\{Y^\{\\top\}Y=I\_\{K\}\}\\operatorname\{tr\}\(Y^\{\\top\}AY\); the Ky Fan maximum principle gives the result\. ∎

Because Eq\. \([A2](https://arxiv.org/html/2611.17917#A5.E2)\) is unchanged byΔ​θ↦−Δ​θ\\Delta\\theta\\mapsto\-\\Delta\\theta, this theorem does not guarantee semantic\-risk reduction\. For a fixed coordinate mask, it applies to an eigenproblem solved on the selected principal submatrices, withKKno larger than the selected dimension; it does not imply optimality of the mask or of a subsequently masked unrestricted basis\.

##### Replacement teacher\.

At a fixed context, writep0​\(𝐚\)=π0​\(𝐚∣𝐡t\)p\_\{0\}\(\\mathbf\{a\}\)=\\pi\_\{0\}\(\\mathbf\{a\}\\mid\\mathbf\{h\}\_\{t\}\)andwt​\(𝐚\)=β​QJ​\(s¯t,𝐚\)−λ​Qφ​\(s¯t,𝐚\)w\_\{t\}\(\\mathbf\{a\}\)=\\beta Q^\{J\}\(\\bar\{s\}\_\{t\},\\mathbf\{a\}\)\-\\lambda Q^\{\\varphi\}\(\\bar\{s\}\_\{t\},\\mathbf\{a\}\)\.

###### Proposition 0\(KL\-regularized teacher\)\.

For boundedwtw\_\{t\}, the teacher in Eq\. \([21](https://arxiv.org/html/2611.17917#S4.E21)\) uniquely minimizes

\(A4\)DKL\(μ∥p0\)−𝔼μ\[wt\]D\_\{\\mathrm\{KL\}\}\(\\mu\\\|p\_\{0\}\)\-\\mathbb\{E\}\_\{\\mu\}\[w\_\{t\}\]over distributionsμ≪p0\\mu\\ll p\_\{0\}\.

WithZt=𝔼p0​\[ewt\]∈\(0,∞\)Z\_\{t\}=\\mathbb\{E\}\_\{p\_\{0\}\}\[e^\{w\_\{t\}\}\]\\in\(0,\\infty\),

\(A5\)DKL\(μ∥p0\)−𝔼μ\[wt\]=DKL\(μ∥μ~t\)−logZt,D\_\{\\mathrm\{KL\}\}\(\\mu\\\|p\_\{0\}\)\-\\mathbb\{E\}\_\{\\mu\}\[w\_\{t\}\]=D\_\{\\mathrm\{KL\}\}\(\\mu\\\|\\widetilde\{\\mu\}\_\{t\}\)\-\\log Z\_\{t\},which is minimized uniquely atμ=μ~t\\mu=\\widetilde\{\\mu\}\_\{t\}\. ∎This optimum concerns the centralized teacher; marginal distillation need not preserve its joint\-action correlations\.

##### KL\-based retention\.

Forπ−=πθα\\pi^\{\-\}=\\pi\_\{\\theta\_\{\\alpha\}\}, define

\(A6\)κα\(𝐡t\)=∑iDKL\(π0,i\(⋅∣hti\)∥πi−\(⋅∣hti\)\),ℒ¯retain=∑t𝔼qr,t\[κα\]\.\\kappa\_\{\\alpha\}\(\\mathbf\{h\}\_\{t\}\)=\\sum\_\{i\}D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\pi\_\{0,i\}\(\\cdot\\mid h\_\{t\}^\{i\}\)\\\|\\pi^\{\-\}\_\{i\}\(\\cdot\\mid h\_\{t\}^\{i\}\)\\right\),\\overline\{\\mathcal\{L\}\}\_\{\\mathrm\{retain\}\}=\\sum\_\{t\}\\mathbb\{E\}\_\{q\_\{r,t\}\}\[\\kappa\_\{\\alpha\}\]\.Hereqr,tq\_\{r,t\}is the population retain\-history marginal, with the time weights of Eq\. \([24](https://arxiv.org/html/2611.17917#S4.E24)\);ℒretain\\mathcal\{L\}\_\{\\mathrm\{retain\}\}is its empirical counterpart\.

###### Proposition 0\(Conditional return retention\)\.

Fix anHH\-step contextccwith common initial, transition, observation, and reward kernels,\|rt\|≤Rmax\|r\_\{t\}\|\\leq R\_\{\\max\}, and0≤γ≤10\\leq\\gamma\\leq 1\. Letdπ0,tcd\_\{\\pi\_\{0\},t\}^\{c\}be the original\-policy history law\. Suppose, for alltt,

\(A7\)dπ0,tc≤Cc​qr,t,ℒ¯retain​\(α\)≤ℒretain​\(α\)\+εgen,d\_\{\\pi\_\{0\},t\}^\{c\}\\leq C\_\{c\}q\_\{r,t\},\\qquad\\overline\{\\mathcal\{L\}\}\_\{\\mathrm\{retain\}\}\(\\alpha\)\\leq\\mathcal\{L\}\_\{\\mathrm\{retain\}\}\(\\alpha\)\+\\varepsilon\_\{\\mathrm\{gen\}\},where the first inequality is between measures,Cc<∞C\_\{c\}<\\infty, andεgen≥0\\varepsilon\_\{\\mathrm\{gen\}\}\\geq 0is finite and valid for the fitted edit\. WithRH=Rmax​∑t=0H−1γtR\_\{H\}=R\_\{\\max\}\\sum\_\{t=0\}^\{H\-1\}\\gamma^\{t\},

\(A8\)\|Jc​\(π−\)−Jc​\(π0\)\|≤2​RH​Cc​\(ℒretain​\(α\)\+εgen\)2\.\|J\_\{c\}\(\\pi^\{\-\}\)\-J\_\{c\}\(\\pi\_\{0\}\)\|\\leq 2R\_\{H\}\\sqrt\{\\frac\{C\_\{c\}\(\\mathcal\{L\}\_\{\\mathrm\{retain\}\}\(\\alpha\)\+\\varepsilon\_\{\\mathrm\{gen\}\}\)\}\{2\}\}\.

Common environment kernels and factorized actors give

\(A9\)DKL\(Pπ0c∥Pπ−c\)=∑t𝔼dπ0,tc\[κα\]≤Ccℒ¯retain≤Cc\(ℒretain\+εgen\)\.D\_\{\\mathrm\{KL\}\}\(P\_\{\\pi\_\{0\}\}^\{c\}\\\|P\_\{\\pi^\{\-\}\}^\{c\}\)=\\sum\_\{t\}\\mathbb\{E\}\_\{d\_\{\\pi\_\{0\},t\}^\{c\}\}\[\\kappa\_\{\\alpha\}\]\\leq C\_\{c\}\\overline\{\\mathcal\{L\}\}\_\{\\mathrm\{retain\}\}\\leq C\_\{c\}\(\\mathcal\{L\}\_\{\\mathrm\{retain\}\}\+\\varepsilon\_\{\\mathrm\{gen\}\}\)\.Bounded returns and Pinsker’s inequality yield\|Jc​\(π−\)−Jc​\(π0\)\|≤2​RH​DKL\(Pπ0c∥Pπ−c\)/2,\|J\_\{c\}\(\\pi^\{\-\}\)\-J\_\{c\}\(\\pi\_\{0\}\)\|\\leq 2R\_\{H\}\\sqrt\{D\_\{\\mathrm\{KL\}\}\(P\_\{\\pi\_\{0\}\}^\{c\}\\\|P\_\{\\pi^\{\-\}\}^\{c\}\)/2\},proving the claim\. ∎The KL direction matches Eq\. \([24](https://arxiv.org/html/2611.17917#S4.E24)\)\. Coverage and generalization are additional conditions, not certified by a small replay loss or action\-matching MSE; the experiments do not verify them\.

##### Decentralized information limit\.

###### Theorem 5\(Indistinguishable decision situations\)\.

If two same\-time situationsx,x′x,x^\{\\prime\}give every agent identical local histories but require disjoint joint\-action sets𝒜good​\(x\)\\mathcal\{A\}\_\{\\mathrm\{good\}\}\(x\)and𝒜good​\(x′\)\\mathcal\{A\}\_\{\\mathrm\{good\}\}\(x^\{\\prime\}\), no history\-conditioned decentralized policy satisfies both requirements with probability one\. At least one situation has violation probability at least1/21/2\.

Identical histories induce the same joint\-action lawμ\\mu\. Disjointness impliesμ⁡\(𝒜good​\(x\)\)\+μ⁡\(𝒜good​\(x′\)\)≤1,\\mu\(\\mathcal\{A\}\_\{\\mathrm\{good\}\}\(x\)\)\+\\mu\(\\mathcal\{A\}\_\{\\mathrm\{good\}\}\(x^\{\\prime\}\)\)\\leq 1,so at least one required set has probability at most1/21/2\. ∎This theorem identifies an information limit that parameter editing alone cannot resolve\. Its error bound is conditional on the two indistinguishable decision situations and does not directly bound overall SSR/WER or expected return\.

## Appendix FLimitations

The formulation assumes cooperative finite\-horizon MARL, CTDE training, target strategies expressible through observable relational predicates, and limited simulator access for suffix regeneration and red\-team auditing\. If the original reward continues to make the forgotten strategy uniquely optimal, subsequent unconstrained training can relearn it; preventing that is continual semantic revocation rather than one\-shot unlearning\. The proposed guarantees are behavioral and semantic, not direct legal\-data\-deletion guarantees\. Future work will investigate replacement policies that better preserve joint\-action correlations during decentralized distillation and broader auditing over partners and deployment contexts\.[8](https://arxiv.org/html/2611.17917#bib.bib45)[23](https://arxiv.org/html/2611.17917#bib.bib44)[7](https://arxiv.org/html/2611.17917#bib.bib46)[27](https://arxiv.org/html/2611.17917#bib.bib47)

Similar Articles