When Actions Disappear: Adversarial Action Removal in Self-Play Reinforcement Learning

# Adversarial Action Removal in Self-Play Reinforcement Learning
Source: [https://arxiv.org/html/2605.16312](https://arxiv.org/html/2605.16312)
## When Actions Disappear: Adversarial Action Removal in Self\-Play Reinforcement Learning

###### Abstract

We study adversarial action masking in self\-play reinforcement learning: an attacker selectively removes legal actions from a victim’s action set\. Unlike observation or action perturbations, removal eliminates decision options before the agent acts\. Across poker games scaling from 6 to 5,531 information states and two non\-poker domains, learned masking causes substantially more damage than random masking and learned perturbation baselines\. The attack persists across Q\-learning, PPO, NFSP, neural NFSP, and DQN victims; transfers across agents; is amplified by self\-play; and shows no recovery under extended masked training\. Mechanistically, the adversary targets high\-value decision points, captured by reach\-weighted contingent action capacity \(CACw\) and a value\-weighted refinement CACv\. These results identify action availability as a distinct robustness surface in self\-play RL\.

## 1Introduction

Multi\-agent reinforcement learning \(MARL\) agents trained via self\-play achieve strong performance in competitive domains\(Silveret al\.,[2018](https://arxiv.org/html/2605.16312#bib.bib7); Brown and Sandholm,[2019](https://arxiv.org/html/2605.16312#bib.bib11)\), but their robustness to structural environment changes remains poorly understood\. Prior work on adversarial attacks focuses on observation perturbations\(Huanget al\.,[2017](https://arxiv.org/html/2605.16312#bib.bib2); Gleaveet al\.,[2020](https://arxiv.org/html/2605.16312#bib.bib1)\)or reward manipulation\(Zhanget al\.,[2020](https://arxiv.org/html/2605.16312#bib.bib8)\)\. We study a different, more severe attack surface: the*action space*itself\.

An adversary that selectively removes actions—disabling specific capabilities rather than adding noise—poses a qualitatively different threat from bounded perturbation\. Such attacks arise naturally: hardware failures disable actuators, regulatory changes restrict strategies, API deprecations remove endpoints, and sandboxing constrains agent capabilities\. We formalise this as a bi\-level optimisation where the inner loop trains an RL agent under masked actions and the outer loop trains an adversary to choose which actions to remove\.

Our key finding is that adversarial masking is*dramatically more efficient*than random removal: across Leduc variants, learned removal is up to4\.8×4\.8\\timesmore damaging than random masking at comparable support\. The mechanism operates through selective minimisation of reach\-weighted contingent action capacity \(CACw\)—empirically verified by strong correlation between CACwand victim reward across budget levels\.

#### Contributions\.

- •We formalise adversarial action masking and show it is4×4\\timesmore damaging than learned perturbation \(RARL\-style\) with equal training budget\.
- •We demonstrate scaling from 6 to5,531information states across five algorithms \(QL, PPO, NFSP, neural NFSP, DQN\), with the adversary’s advantage increasing from2\.2×2\.2\\timesto4\.8×4\.8\\timeswithin the Leduc family as game complexity grows\.
- •We validate cross\-domain in two non\-poker environments \(competitive gridworld, resource collection\), confirming the phenomenon is not poker\-specific\.
- •We connect the mechanism to CACw\(r=0\.80r=0\.80\) and a refined CACv\(r=0\.81r=0\.81\), and show victims do not recover under extended training\.

## 2Related Work

#### Adversarial attacks on RL\.

Adversarial attacks on RL most commonly perturb observations\(Huanget al\.,[2017](https://arxiv.org/html/2605.16312#bib.bib2); Gleaveet al\.,[2020](https://arxiv.org/html/2605.16312#bib.bib1); Zhanget al\.,[2021](https://arxiv.org/html/2605.16312#bib.bib17); Sunet al\.,[2022](https://arxiv.org/html/2605.16312#bib.bib15)\)or poison rewards\(Zhanget al\.,[2020](https://arxiv.org/html/2605.16312#bib.bib8)\)\. In multi\-agent settings, recent work studies adversarial policies\(Gleaveet al\.,[2020](https://arxiv.org/html/2605.16312#bib.bib1)\), sparse or model\-based attacks against cooperative MARL\(Linet al\.,[2020](https://arxiv.org/html/2605.16312#bib.bib23); Hu and Zhang,[2022](https://arxiv.org/html/2605.16312#bib.bib24)\), and collusive policy\-level attackers\(Niuet al\.,[2026](https://arxiv.org/html/2605.16312#bib.bib25)\)\. These attacks manipulate observations, policies, or rewards while leaving the legal action set intact\. We instead study a structural attack on action availability itself\.

#### Robust and action\-robust RL\.

Robust MDPs\(Iyengar,[2005](https://arxiv.org/html/2605.16312#bib.bib18); Nilim and El Ghaoui,[2005](https://arxiv.org/html/2605.16312#bib.bib19)\)and constrained MDPs\(Altman,[1999](https://arxiv.org/html/2605.16312#bib.bib10)\)model uncertainty or constraints in the environment\. Robust Adversarial RL \(RARL\)\(Pintoet al\.,[2017](https://arxiv.org/html/2605.16312#bib.bib13)\)and action\-robust RL\(Tessleret al\.,[2019](https://arxiv.org/html/2605.16312#bib.bib14)\)train against perturbations of the action actually executed\. Those channels modify selected actions; our adversary removes actions from the legal set before selection\. This difference matters: bounded perturbations preserve the agent’s ability to choose, whereas removal can collapse a decision point to a singleton action set\.

#### Robust MARL defenses\.

Recent robust MARL methods train agents against structured adversaries, including game\-theoretic robust training for temporally coupled perturbations \(GRAD\)\(Lianget al\.,[2024](https://arxiv.org/html/2605.16312#bib.bib27)\)and fault\-switching MARL defenses such as MARTA\(Mguniet al\.,[2025](https://arxiv.org/html/2605.16312#bib.bib28)\)\. Fault\-tolerant control has long studied actuator failures and reconfiguration\(Blankeet al\.,[2006](https://arxiv.org/html/2605.16312#bib.bib29)\); our setting can be viewed as a learning\-based analogue where failures are state\-dependent and adversarially selected\. These approaches are complementary: they improve robustness to perturbations, malfunctions, or adversarial agents within an intact joint action space\. Our attack changes the feasible action set itself\. A mask\-aware version of such defenses is a natural direction, but existing guarantees do not directly cover state\-dependent removal of legal actions\.

#### Action sets and masking\.

Invalid action masking prevents agents from selecting impossible moves\(Huang and Ontañón,[2022](https://arxiv.org/html/2605.16312#bib.bib3)\)\. Decision\-theoretic planning has long studied structural leverage and action contingencies\(Boutilieret al\.,[1999](https://arxiv.org/html/2605.16312#bib.bib22)\), and empowerment measures quantify an agent’s available control over future outcomes\(Klyubinet al\.,[2005](https://arxiv.org/html/2605.16312#bib.bib30)\)\. Our CAC view is related in spirit but is game\-local and adversarial: it measures remaining multi\-action decision capacity at reached information states\. Our work reverses the usual masking motivation: the mask is not a safety or efficiency aid, but an attacker\-chosen capability removal\.

#### Self\-play and imperfect\-information games\.

Self\-play can overfit, cycle, or exploit non\-transitive structure\(Balduzziet al\.,[2019](https://arxiv.org/html/2605.16312#bib.bib4); Lanctotet al\.,[2019](https://arxiv.org/html/2605.16312#bib.bib5)\)\. Regret\-minimization methods such as CFR\(Zinkevichet al\.,[2007](https://arxiv.org/html/2605.16312#bib.bib20)\)and Deep CFR\(Brownet al\.,[2019](https://arxiv.org/html/2605.16312#bib.bib21)\)provide strong baselines for imperfect\-information games, while NFSP\(Heinrich and Silver,[2016](https://arxiv.org/html/2605.16312#bib.bib12)\)combines best\-response learning with average\-strategy tracking\. Population methods such as PSRO\(Lanctotet al\.,[2017](https://arxiv.org/html/2605.16312#bib.bib6)\)maintain diversity\. We show that these algorithmic stabilizers do not address a different failure mode: when the action set itself is structurally reduced, averaging and population diversity cannot restore eliminated strategic dimensions\.

## 3Problem Formulation

#### Game model\.

Consider a two\-player zero\-sum extensive\-form gameΓ\\Gammawith information setsℐ=ℐ0∪ℐ1\\mathcal\{I\}=\\mathcal\{I\}\_\{0\}\\cup\\mathcal\{I\}\_\{1\}\. At information seth∈ℐph\\in\\mathcal\{I\}\_\{p\}, playerppchooses from legal actionsA​\(h\)A\(h\)\. Player 0 is the*victim*; Player 1 is the opponent\. Both learn via self\-play\.

#### Adversary definition\.

An*action\-removal adversary*is a mappingℳ:ℐ0→2A\\mathcal\{M\}:\\mathcal\{I\}\_\{0\}\\to 2^\{A\}that, at each victim information sethh, selects a subsetℳ​\(h\)⊆A​\(h\)\\mathcal\{M\}\(h\)\\subseteq A\(h\)of actions to*retain*\. The victim observes onlyℳ​\(h\)\\mathcal\{M\}\(h\)and is unaware of the adversary\. Formally:

- •Input: information sethh, legal actionsA​\(h\)A\(h\), current playerpp\.
- •Output:ℳ​\(h\)⊆A​\(h\)\\mathcal\{M\}\(h\)\\subseteq A\(h\)with\|ℳ​\(h\)\|≥1\|\\mathcal\{M\}\(h\)\|\\geq 1\(at least one action retained\)\.
- •Constraint:\|supp​\(ℳ\)\|=\|\{h:\|ℳ​\(h\)\|<\|A​\(h\)\|\}\|≤k\|\\text\{supp\}\(\\mathcal\{M\}\)\|=\|\\\{h:\|\\mathcal\{M\}\(h\)\|<\|A\(h\)\|\\\}\|\\leq k\(budget\)\.
- •Objective: minimise the victim’s expected valueV0​\(πℳ∗\)V\_\{0\}\(\\pi^\{\*\}\_\{\\mathcal\{M\}\}\)\.

#### Bi\-level optimisation\.

The adversary and victim interact through a bi\-level problem:

Inner:πℳ∗=arg⁡maxπ⁡𝔼​\[∑trt∣π,ℳ\]\\displaystyle\\pi^\{\*\}\_\{\\mathcal\{M\}\}=\\arg\\max\_\{\\pi\}\\mathbb\{E\}\\left\[\\sum\_\{t\}r\_\{t\}\\mid\\pi,\\mathcal\{M\}\\right\]\(1\)Outer:ℳ∗=arg⁡minℳ∈𝒞k⁡V0​\(πℳ∗\)\\displaystyle\\mathcal\{M\}^\{\*\}=\\arg\\min\_\{\\mathcal\{M\}\\in\\mathcal\{C\}\_\{k\}\}V\_\{0\}\(\\pi^\{\*\}\_\{\\mathcal\{M\}\}\)\(2\)where𝒞k=\{ℳ:\|supp​\(ℳ\)\|≤k\}\\mathcal\{C\}\_\{k\}=\\\{\\mathcal\{M\}:\|\\text\{supp\}\(\\mathcal\{M\}\)\|\\leq k\\\}\. The inner loop trains the victim under the adversary’s mask via RL; the outer loop updates the adversary via REINFORCE with reward signal−V0\-V\_\{0\}\. In practice, we alternate: 500 episodes of inner training, then one adversary gradient step, for 20–25 outer iterations\.

#### Adversary implementation\.

*Tabular*: preference tableθ​\(h,a\)\\theta\(h,a\)withpremove​\(a\|h\)=softmax​\(θ​\(h,⋅\)\)p\_\{\\text\{remove\}\}\(a\|h\)=\\text\{softmax\}\(\\theta\(h,\\cdot\)\); the action with highest removal probability is removed\. Updated via REINFORCE\.*Neural*: MLPfϕ:ℝd→Δ\|A\|\+1f\_\{\\phi\}:\\mathbb\{R\}^\{d\}\\to\\Delta^\{\|A\|\+1\}mapping state features to a distribution over\|A\|\|A\|removal choices plus “no removal\.” Trained via REINFORCE with mean\-reward baseline\.

#### Connection to CACw\.

Define*reach\-weighted contingent action capacity*:

CACw​\(ℳ\)=∑h∈ℐ0ρ​\(h\)⋅𝟏​\[\|ℳ​\(h\)\|\>1\]\\text\{CAC\}\_\{w\}\(\\mathcal\{M\}\)=\\sum\_\{h\\in\\mathcal\{I\}\_\{0\}\}\\rho\(h\)\\cdot\\mathbf\{1\}\[\|\\mathcal\{M\}\(h\)\|\>1\]whereρ​\(h\)\\rho\(h\)is the reach probability ofhhunder current play\. CACwmeasures*remaining*multi\-action capacity: an unmasked decision state with at least two retained actions contributes positive capacity, while a state collapsed to a singleton contributes zero\. Stronger attacks therefore drive CACwdownward by turning high\-reach decision points into forced moves\. The adversary’s value\-minimisation objective \([2](https://arxiv.org/html/2605.16312#S3.E2)\) can be decomposed: masking statehhreducesV0V\_\{0\}by approximatelyρ​\(h\)⋅δ​\(h\)\\rho\(h\)\\cdot\\delta\(h\), whereδ​\(h\)=\|Q​\(h,a∗\)−Q​\(h,aforced\)\|\\delta\(h\)=\|Q\(h,a^\{\*\}\)\-Q\(h,a\_\{\\text\{forced\}\}\)\|is the*value gap*athh\. This motivates a refined measure:

CACv​\(ℳ\)=∑h∈ℐ0ρ​\(h\)⋅δ​\(h\)⋅𝟏​\[\|ℳ​\(h\)\|\>1\]\\text\{CAC\}\_\{v\}\(\\mathcal\{M\}\)=\\sum\_\{h\\in\\mathcal\{I\}\_\{0\}\}\\rho\(h\)\\cdot\\delta\(h\)\\cdot\\mathbf\{1\}\[\|\\mathcal\{M\}\(h\)\|\>1\]Empirically, CACvcorrelates with victim reward atr=0\.81r=0\.81vs\. CACw’sr=0\.77r=0\.77\(Section[5\.4](https://arxiv.org/html/2605.16312#S5.SS4)\)\.

#### Deterministic Exploitation Attractor \(DEA\)\.

###### Proposition 1\(DEA Convergence\)\.

Under CAC=w0\{\}\_\{w\}=0in self\-play Q\-learning withε\\varepsilon\-greedy \(ε\>0\\varepsilon\>0,α∈\(0,1\)\\alpha\\in\(0,1\)\): \(i\) the victim’s policy converges to the unique forced action at every information set; \(ii\) the opponent’s Q\-values converge toQ∗=VBR​\(σ∗\)Q^\{\*\}=V^\{\\text\{BR\}\(\\sigma^\{\*\}\)\}; \(iii\)\(σ∗,BR​\(σ∗\)\)\(\\sigma^\{\*\},\\text\{BR\}\(\\sigma^\{\*\}\)\)is a stable fixed point\.

*Proof sketch\.*Forced actions make the victim’s policy Q\-value\-independent\. The opponent faces a stationary MDP; Q\-learning converges under standard conditions\. The pair is stable: neither player can unilaterally deviate\.

###### Proposition 2\(Damage bound\)\.

Letℳ\\mathcal\{M\}mask a single information seth∗h^\{\*\}by retaining a singleton actionaf​\(h∗\)a\_\{f\}\(h^\{\*\}\)\. The victim’s value loss relative to unmasked play is bounded by:

Δ​V0≥ρ​\(h∗\)⋅\[Q0​\(h∗,a∗​\(h∗\)\)−Q0​\(h∗,af​\(h∗\)\)\]\\Delta V\_\{0\}\\geq\\rho\(h^\{\*\}\)\\cdot\[Q\_\{0\}\(h^\{\*\},a^\{\*\}\(h^\{\*\}\)\)\-Q\_\{0\}\(h^\{\*\},a\_\{f\}\(h^\{\*\}\)\)\]wherea∗​\(h\)=arg⁡maxa∈A​\(h\)⁡Q0​\(h,a\)a^\{\*\}\(h\)=\\arg\\max\_\{a\\in A\(h\)\}Q\_\{0\}\(h,a\)is the victim’s best action,af​\(h\)a\_\{f\}\(h\)is the action forced by the mask athh, andρ​\(h\)\\rho\(h\)is the reach probability\. For a budget\-kkadversary with supportS=supp​\(ℳ\)S=\\text\{supp\}\(\\mathcal\{M\}\), the additive approximation is:

Δ​V0​\(ℳ\)≈∑h∈Sρ​\(h\)⋅\[Q0​\(h,a∗​\(h\)\)−Q0​\(h,af​\(h\)\)\]\\Delta V\_\{0\}\(\\mathcal\{M\}\)\\approx\\sum\_\{h\\in S\}\\rho\(h\)\\cdot\[Q\_\{0\}\(h,a^\{\*\}\(h\)\)\-Q\_\{0\}\(h,a\_\{f\}\(h\)\)\]and is upper\-bounded by:

Δ​V0​\(ℳ\)≤∑h∈Sρ​\(h\)⋅maxaf∈A​\(h\)∖\{a∗​\(h\)\}⁡\[Q0​\(h,a∗​\(h\)\)−Q0​\(h,af\)\]\.\\Delta V\_\{0\}\(\\mathcal\{M\}\)\\leq\\sum\_\{h\\in S\}\\rho\(h\)\\cdot\\max\_\{a\_\{f\}\\in A\(h\)\\setminus\\\{a^\{\*\}\(h\)\\\}\}\[Q\_\{0\}\(h,a^\{\*\}\(h\)\)\-Q\_\{0\}\(h,a\_\{f\}\)\]\.Thus a greedy adversary selects states with highρ​\(h\)​δ​\(h\)\\rho\(h\)\\delta\(h\), whereδ​\(h\)=Q0​\(h,a∗​\(h\)\)−Q0​\(h,af​\(h\)\)\\delta\(h\)=Q\_\{0\}\(h,a^\{\*\}\(h\)\)\-Q\_\{0\}\(h,a\_\{f\}\(h\)\)—exactly the states that maximise per\-unit CACvreduction\.

*Proof\.*The bound follows from the definition of Q\-values and linearity of expectation over the reach\-weighted trajectory\. At stateh∗h^\{\*\}, forcingaf​\(h∗\)a\_\{f\}\(h^\{\*\}\)instead ofa∗​\(h∗\)a^\{\*\}\(h^\{\*\}\)losesQ0​\(h∗,a∗​\(h∗\)\)−Q0​\(h∗,af​\(h∗\)\)Q\_\{0\}\(h^\{\*\},a^\{\*\}\(h^\{\*\}\)\)\-Q\_\{0\}\(h^\{\*\},a\_\{f\}\(h^\{\*\}\)\)in expectation, weighted byρ​\(h∗\)\\rho\(h^\{\*\}\)\. Under the simplifying assumption that masked states lie on independent branches, the resulting coverage objective suggests an approximate greedy structure\.111This independence assumption is only approximate in extensive\-form games: masking one state can alter reach probabilities and values downstream\.

#### Theoretical scope\.

Propositions 1–2 provide*sufficient conditions*and*local bounds*, not a complete characterization of the optimal adversary\. The damage bound is per\-state and additive, ignoring cross\-state interactions \(masking statehhmay shift reach probabilities and Q\-values at downstream states\)\. A global characterization would require solving a combinatorial optimisation over\(\|ℐ0\|k\)\\binom\{\|\\mathcal\{I\}\_\{0\}\|\}\{k\}mask configurations—NP\-hard in general via reduction to weighted max coverage\. Under approximate independence, the value\-gap decomposition motivates a greedy heuristic over highρ⋅δ\\rho\\cdot\\deltastates; we do not claim a general approximation guarantee for arbitrary extensive\-form games\. We view these bounds as explaining*why*the adversary works rather than fully characterizing*how well*it can work\.

## 4Methods

#### Self\-play protocol\.

Both players share a single learning agent; the mask is applied to Player 0’s actions at every decision point during*both training and evaluation*\. In the self\-play regime, both sides adapt continuously\. In the fixed\-opponent regime, a snapshot of the agent’s value function is frozen before masking begins; this static copy plays as Player 1 while only Player 0 continues learning under the mask\.

#### Victim agents\.

We evaluate four algorithms spanning tabular and neural methods:

- •Tabular Q\-Learning:ε\\varepsilon\-greedy,ε=0\.15\\varepsilon=0\.15,α=0\.1\\alpha=0\.1\.
- •Tabular PPO: softmax policy, clipped surrogate, entropy bonus \(Appendix[A](https://arxiv.org/html/2605.16312#A1)\)\.
- •Tabular NFSP\(Heinrich and Silver,[2016](https://arxiv.org/html/2605.16312#bib.bib12)\): best\-response QL \+ average strategy,η=0\.1\\eta=0\.1\.
- •DQN: 2\-layer MLP \(64 hidden\), experience replay \(20k buffer\), target network, Adam \(10−310^\{\-3\}\)\. Full hyperparameters in Appendix[C](https://arxiv.org/html/2605.16312#A3)\.

#### Masking strategies\.

Table 1:Masking strategies evaluated\.
#### Adversary architectures\.

Both tabular and neural adversaries are defined formally in Section 3\. The neural adversary scales its architecture with game size: 2\-layer \(32 hidden\) for Kuhn/Leduc, 3\-layer \(128\-64\) for Leduc\-10/20\.

#### Environments\.

Kuhn Poker: 3 cards, 2 actions, 6 P0 info states\.Leduc Poker: 6 cards \(3 ranks×\\times2 suits\), 3 actions, 2 rounds,∼\\sim50 P0 info states\.Leduc\-5: 10 cards \(5 ranks\), 389 P0 info states\.Leduc\-10: 20 cards \(10 ranks\), 1,496 P0 info states\.Leduc\-20: 40 cards \(20 ranks\),5,531 P0 info states\.Competitive Gridworld: 5×\\times5, 5 actions, 149 P0 states\.Resource Collection: 4×\\times4, 4 actions, 18,604 P0 states\. All implemented from scratch withMaskedEnvwrappers\.

## 5Experiments

We structure experiments around five questions: \(1\) Does adversarial masking scale with game complexity? \(2\) Is the vulnerability algorithm\-invariant? \(3\) Does it generalise beyond poker? \(4\) How efficient is adversarial targeting? \(5\) What mechanism drives the attack?

### 5\.1Scaling with Game Complexity

Our strongest result uses function approximation on both sides: a DQN victim and a neural MLP adversary in Leduc variants whose state spaces range from∼\\sim50 to 5,531 victim information states\.

![Refer to caption](https://arxiv.org/html/2605.16312v1/x1.png)Figure 1:Scaling trend\. The adversarial/random damage ratio is plotted against victim information\-set count on a log scale\. Error bars show 95% CIs from the DQN scale experiments\.We implement Leduc\-N, parameterised by rank count\.Leduc\-5\(10 cards, 389 states\),Leduc\-10\(20 cards, 1,496 states\), andLeduc\-20\(40 cards,5,531 states\) test whether the adversary’s advantage persists as game complexity grows by920×920\\timesfrom Kuhn\. Figure[1](https://arxiv.org/html/2605.16312#S5.F1)and Table[2](https://arxiv.org/html/2605.16312#S5.T2)show a consistent empirical scaling trend\.

Table 2:DQN victim across poker scales \(5 seeds, 95% CIs\)\.At 5,531 states the neural adversary drives DQN to−3\.00\-3\.00per hand—4\.8×4\.8\\timesworse than random—with the tightest CIs of any scale \(±0\.15\\pm 0\.15\)\. A strict matched\-L0L\_\{0\}control in Leduc samples exactly the same number of states as the learned mask \(k=64\.8±4\.3k=64\.8\\pm 4\.3\): adversarial masking reaches−2\.32±0\.36\-2\.32\\pm 0\.36while matched random reaches−1\.03±0\.24\-1\.03\\pm 0\.24\(2\.24×2\.24\\timesgap\)\. The advantage is therefore*which*states are targeted, not support size\.

#### Threat\-model ablation\.

The main setting assumes environment\-level control: the adversary can observe the victim information state and alter legal actions before selection\. This is intentionally strong, matching settings where a simulator, platform, or operating layer controls capabilities\. To test whether private\-card access is essential, we restrict the adversary to public information only \(betting history and public card, private rank hidden\)\. In Leduc, public\-info adversarial masking still reaches−1\.71±0\.58\-1\.71\\pm 0\.58, substantially worse than random masking \(−0\.98±0\.08\-0\.98\\pm 0\.08\), though weaker than private\-info masking \(−2\.30±0\.39\-2\.30\\pm 0\.39\)\. Thus strong private observability is not required: public structural information is sufficient for damaging targeted removal\.

#### Neural NFSP on Leduc\-5\.

To confirm the result holds with a stronger victim, we run*Neural NFSP*\(128\-64 MLP average policy, DQN best\-response, reservoir\-sampled action buffer\) on Leduc\-5\. Pre\-attack:−0\.13±0\.09\-0\.13\\pm 0\.09; post\-adversarial:−1\.89±0\.16\\mathbf\{\-1\.89\\pm 0\.16\}; post\-random:−0\.89±0\.06\-0\.89\\pm 0\.06\. The adversary is2\.3×2\.3\\timesmore damaging than random, confirming that even proper neural game\-solving methods collapse under adversarial action removal\.

![Refer to caption](https://arxiv.org/html/2605.16312v1/x2.png)Figure 2:Learning under a fixed mask\. NFSP in Leduc\-5 does not recover under continued masked training; adversarial removal remains substantially worse than random removal\.
#### DQN masking mechanics\.

Masked actions are filtered from the legal action list*before*ε\\varepsilon\-greedy selection: the DQN computes Q\-values for all actions, then restricts argmax and exploration to the masked subset\. The victim is unaware of the masking\. Total DQN training: 20k pre\-training episodes \(no mask\) \+ 20 outer×\\times500 inner = 10k episodes under the mask\. Learning curves \(Appendix[F](https://arxiv.org/html/2605.16312#A6)\) confirm convergence: reward stabilises by episode∼\\sim8k under the mask\.

#### OpenSpiel/CFR context\.

Kuhn and Leduc rules match the standard imperfect\-information games used in OpenSpiel\(Lanctotet al\.,[2019](https://arxiv.org/html/2605.16312#bib.bib5)\); we implement them from scratch only to expose the action\-mask hook at every legal\-action query and to scale Leduc\-N by rank count\. CFR and Deep CFR\(Zinkevichet al\.,[2007](https://arxiv.org/html/2605.16312#bib.bib20); Brownet al\.,[2019](https://arxiv.org/html/2605.16312#bib.bib21)\)assume a fixed game tree\. Action removal changes the game tree itself, so CFR remains a useful equilibrium reference but is not a direct defense unless retrained on the masked game\. Our results should therefore be read as structural robustness tests of learning dynamics under game\-tree modification\.

### 5\.2Tabular Validation Across Algorithms

Table 3:Tabular agents in Kuhn and Leduc under masking \(5 seeds\)\. Both raw reward \(rr\) and normalised performance shown\.All algorithms collapse under adversarial masking—in both Kuhn and Leduc\. The vulnerability is*algorithm\-invariant*: Q\-Learning, PPO, and NFSP all degrade to comparable levels\. NFSP on Leduc drops from−0\.01\-0\.01to−1\.58\-1\.58\(raw\), confirming that NFSP’s average\-strategy component cannot compensate when the action space itself is reduced\. Combined with the DQN scaling result \(Table[2](https://arxiv.org/html/2605.16312#S5.T2)\), we havefour algorithms across two games and two paradigms \(tabular \+ neural\) all exhibiting collapse\.

### 5\.3Cross\-Domain Validation: Competitive Gridworld

To verify the phenomenon is not a poker artifact, we test on a*completely different*domain: a 5×\\times5 competitive gridworld where Player 0 \(prey\) navigates to a goal while Player 1 \(predator\) attempts to intercept\. Both players have 5 actions \(UP/DOWN/LEFT/RIGHT/STAY\), the game is turn\-based with perfect information, and 149 unique P0 states are observed\.

Table 4:Cross\-domain: Competitive Gridworld \(5×\\times5, 5 actions, 149 P0 states, 5 seeds\)\.The adversary causes1\.9×1\.9\\timesmore damage than random masking in a domain with no cards, no imperfect information, and a spatial rather than strategic structure\. Fixed removal \(always remove UP\) has negligible effect because the prey reroutes; the adversary*state\-dependently*removes the action that matters most at each position\.

#### Resource Collection \(4×\\times4\)\.

A second non\-poker domain: two agents compete to collect 4 resources on a grid, with reward = resources\_collected\_difference\. 4 actions, 18,604 unique P0 states observed\. Adversarial masking degrades P0 byΔ=−0\.13\\Delta=\-0\.13vs\. randomΔ=−0\.10\\Delta=\-0\.10\(1\.4×1\.4\\times\)\. Fixed removal \(UP\)*helps*P0 \(Δ=\+0\.32\\Delta=\+0\.32\), showing that only state\-dependent targeting produces consistent harm\. The smaller gap \(1\.4×1\.4\\timesvs\.1\.9×1\.9\\times\) reflects the symmetric, coordination\-like structure where many states have similar strategic value—there is less exploitable structure to target\.

Two non\-poker domains with different structures \(goal\-seeking vs\. resource competition, perfect information, spatial\) both show adversarial\>\>random\. The phenomenon is not poker\-specific\.

### 5\.4Attack Efficiency and CACwCorrelation

#### Budget sweep\.

Appendix[E](https://arxiv.org/html/2605.16312#A5)shows victim reward as a function of adversary budgetkk\(number of info states masked\)\. Targeted removal degrades the victim as support grows; random masking requires broad coverage to approach the same damage\.

#### CACwcorrelation\.

To verify that the adversary operates through CACwminimisation, we compute reach\-weighted CACwat each budget level and correlate with victim reward\. Across all budget levels and both adversarial and random conditions \(10 seeds each\), the Pearson correlation between CACwand reward isr=0\.77r=0\.77\(p<0\.001p<0\.001\); for adversarial\-only conditions,r=0\.80r=0\.80\. The full budget table is in Appendix[E](https://arxiv.org/html/2605.16312#A5)\.

#### CACv\-greedy oracle\.

We also test a direct oracle heuristic that selects the top\-kkKuhn states by estimatedρ​\(h\)​δ​\(h\)\\rho\(h\)\\delta\(h\)and removes the victim’s estimated best action\. Atk=3k=3, CACv\-greedy reaches−0\.29±0\.09\-0\.29\\pm 0\.09, stronger than random \(−0\.19±0\.06\-0\.19\\pm 0\.06\) and the learned adversary under the same short training budget \(−0\.22±0\.02\-0\.22\\pm 0\.02\)\. This supports CACv as a mechanistic proxy: when given the value\-gap statistic directly, greedy targeting produces the expected harm\.

#### Estimating CACv\.

We estimateρ​\(h\)\\rho\(h\)from on\-policy visitation frequencies during evaluation rollouts andδ​\(h\)\\delta\(h\)from the victim’s learned Q\-table or network Q\-values at the same checkpoint\. Thus CACv is a diagnostic proxy, not an oracle ground truth: Q\-estimation error can perturb rankings\. Despite this estimation noise, CACv rankings are stable across seeds and align with oracle performance: the top CACv states in Kuhn repeatedly include the same high\-impact information sets \(0pb,1,2pb\), and the CACv\-greedy ablation remains stronger than random\.

### 5\.5Action Removal vs\. Perturbation \(RARL\)

Table 5:Action removal vs\. perturbation: both with*learned*adversary \(Kuhn, 10 seeds, 95% CIs\)\. Both adversaries use the same REINFORCE training \(20 outer×\\times500 inner\)\. Removal is4×4\\timesmore damaging even when the perturbation adversary is given equal training budget\.Both adversaries use identical training \(REINFORCE, same budget, same iterations\), ensuring a fair comparison\. The learned perturbation adversary \(−0\.24\-0\.24\) is stronger than fixed\-ppRARL \(−0\.09\-0\.09, not shown\), but learned removal \(−1\.01\-1\.01\) remains4×4\\timesmore damaging\. The mechanism is qualitatively different: perturbation adds noise to execution but preserves the agent’s*ability to choose*; removal eliminates decision points entirely, reducing CACw\(Pintoet al\.,[2017](https://arxiv.org/html/2605.16312#bib.bib13); Tessleret al\.,[2019](https://arxiv.org/html/2605.16312#bib.bib14)\)\.

#### Perturbation protocol\.

In the learned perturbation baseline, the adversary observes the same information state and replaces the victim’s chosen action with another legal action \(or no\-op\) before execution\. The victim is unaware of the replacement and receives the same training/evaluation schedule as in the removal experiment\. This matches training budget and observability while preserving the legal action set\.

### 5\.6Additional Ablations

The attack is amplified by self\-play co\-adaptation \(self\-play:−0\.975\-0\.975, fixed opponent:−0\.841\-0\.841\), transfers across random seeds \(transferred:−1\.032\-1\.032vs\. per\-agent retrained:−0\.872\-0\.872\), transfers from Q\-learning to NFSP \(Δ=−0\.49\\Delta=\-0\.49\), and is not an artifact of parameter sharing: separate tabular Q\-tables match shared\-agent results, while separate DQN networks in Leduc still collapse to−1\.53±0\.48\-1\.53\\pm 0\.48\. Evaluation\-only masking already degrades a normally trained Leduc victim \(−0\.58\-0\.58\), while continued masked training and retraining from scratch on the fixed final learned mask converge to larger losses \(−2\.65\-2\.65and−2\.71\-2\.71\), showing both immediate structural harm and long\-run adaptation failure\. Simple action\-dropout helps modestly \(−1\.82\-1\.82vs\.−2\.45\-2\.45\), but a persistent random mask\-ensemble defense does not help \(−2\.64\-2\.64\)\. This negative result is informative: robustness to arbitrary stochastic unavailability is not enough; effective defenses likely need targeted protection or redundancy at high\-CACv states\. Full ablation tables are in Appendix[E](https://arxiv.org/html/2605.16312#A5)\.

## 6Discussion

#### Scaling trend\.

The scaling trajectory \(Table[2](https://arxiv.org/html/2605.16312#S5.T2)\) shows a consistent empirical trend\. Excluding Kuhn \(2\-action ceiling effect\), log\-linear regression of adversary damage ratio onlog10⁡\(\|ℐ0\|\)\\log\_\{10\}\(\|\\mathcal\{I\}\_\{0\}\|\)yieldsR2=0\.62R^\{2\}=0\.62\(Pearsonr=0\.79r=0\.79,n=4n=4game sizes\), with slope1\.561\.56\. We report this as an*empirical trend*, not a scaling law—four data points establish direction, not a precise functional form\. The absolute damage gap \(adversarial minus random reward\) grows from−1\.40\-1\.40at 50 states to−2\.51\-2\.51at 1,496 states, and the adversary’s variance*decreases*with scale \(CI±0\.47→±0\.15\\pm 0\.47\\to\\pm 0\.15\), suggesting more robust convergence in larger games where there is more targetable strategic heterogeneity\. Neural NFSP \(2\.3×2\.3\\timeson Leduc\-5\) confirms this holds for properly neural game\-solving methods\.

#### CACw: necessary but not sufficient\.

The CACw–reward correlation \(r=0\.80r=0\.80\) is consistent with the hypothesis that reducing strategic flexibility drives performance degradation\. However, CACw is a*necessary*condition for collapse, not a complete characterization: at intermediate budgets, the adversary sometimes achieves comparable or greater damage with higher CACw than random masking \(Table[19](https://arxiv.org/html/2605.16312#A5.T19),k=3k\{=\}3: adversarial CACw=0\.68=0\.68vs\. random0\.370\.37, yet adversarial reward=−0\.25=\-0\.25vs\. random−0\.21\-0\.21\)\. This occurs because CACw aggregates over all reachable states; the adversary targets the*strategically pivotal*subset where action removal most disrupts equilibrium play, whereas random removal may eliminate more states but at less important ones\. We test a refined measure,*counterfactual\-value\-weighted capacity*CACv=∑hρ​\(h\)⋅\|Q​\(h,a0\)−Q​\(h,a1\)\|⋅𝟏​\[\|ℳ​\(h\)\|\>1\]\\text\{CAC\}\_\{v\}=\\sum\_\{h\}\\rho\(h\)\\cdot\|Q\(h,a\_\{0\}\)\-Q\(h,a\_\{1\}\)\|\\cdot\\mathbf\{1\}\[\|\\mathcal\{M\}\(h\)\|\>1\], which weights each state by its Q\-value gap \(strategic importance\)\. CACvachieves Pearsonr=0\.81r=0\.81vs\. CACw’sr=0\.77r=0\.77, confirming that incorporating value sensitivity improves the structural explanation\. The remaining unexplained variance likely reflects higher\-order interactions \(e\.g\., how masking one state changes the value landscape at downstream states\)\.

#### Implications for deployment\.

Real\-world multi\-agent systems where an adversary can disable specific capabilities \(API endpoints, actuators, communication channels\) are vulnerable to targeted collapse\. Defences should maintain strategic flexibility at high\-reach, high\-CACv decision points\. Uniform robustness methods such as random action dropout or random mask ensembles are insufficient in our experiments; the defensive object is not raw action count, but redundancy at the strategically pivotal states where removal produces large value gaps\. Designing effective defenses likely requires identifying and preserving high\-CACv states, rather than optimizing for uniform robustness to arbitrary action unavailability\.

## 7Limitations

Our largest game \(Leduc\-20\) has∼\\sim5,500 information states—nearly three orders of magnitude larger than Kuhn\. The increasing adversary advantage with scale \(Table[2](https://arxiv.org/html/2605.16312#S5.T2)\) suggests the mechanism strengthens as games grow, but verification at full poker scale \(10510^\{5\}\+ states\) remains future work\.

#### Discrete scope\.

Action removal is inherently a*discrete*attack: it eliminates specific actions from a finite set\. Our results—across poker, gridworld, and resource collection—all use small discrete action spaces \(\|A\|≤5\|A\|\\leq 5\)\. In continuous\-action domains \(e\.g\., MuJoCo\), the analogous attack would be*region exclusion*\(disabling subsets of the action manifold\), which requires different formalisation and may exhibit qualitatively different dynamics\. We do not claim generality beyond discrete action spaces\.

#### Theory\.

The damage bound \(Proposition 2\) is local and additive, assuming independent masking effects across states\. Cross\-state interactions \(masking one state shifts Q\-values at others\) are not captured\. The scaling regression \(R2=0\.62R^\{2\}=0\.62, 4 game sizes\) establishes a trend but not a precise functional relationship\.

## 8Conclusion

We show that self\-play RL agents—from tabular Q\-Learning to neural NFSP and DQN, from 6\-state Kuhn to 5,531\-state Leduc\-20, across poker and non\-poker domains—are brittle to targeted action\-space attacks\. Within the Leduc family, the adversary’s advantage over random masking increases with game complexity \(2\.2×2\.2\\timesat 50 states to4\.8×4\.8\\timesat 5,531\), and the vulnerability persists across five algorithms and seven environments\. The mechanism operates through selective reduction of strategically important decision capacity \(CACw/v\), victims do not recover under extended training, and the attack transfers across agents\. These findings argue for robustness designs that preserve strategic flexibility at high\-reach decision points rather than raw action count\.

## Reproducibility Statement

All environments, agents, adversaries, and experiment scripts are implemented in the accompanying repository\. Each reported result is generated by a standalone script underexperiments/; random seeds are fixed in the scripts, and hyperparameters are listed in Appendix[C](https://arxiv.org/html/2605.16312#A3)\. Figures are generated byexperiments/generate\_neurips\_figures\.pyfrom completed experiment outputs\. The largest run, Leduc\-20, uses five seeds, 30k victim pre\-training episodes, and 25 adversary outer iterations with 500 inner episodes each\.

## References

- E\. Altman \(1999\)Constrained Markov decision processes: stochastic modeling\.CRC Press\.Cited by:[§2](https://arxiv.org/html/2605.16312#S2.SS0.SSS0.Px2.p1.1)\.
- D\. Balduzzi, M\. Garnelo, Y\. Bachrach, W\. Czarnecki, J\. Pérolat, M\. Jaderberg, and T\. Graepel \(2019\)Open\-ended learning in symmetric zero\-sum games\.InInternational Conference on Machine Learning,pp\. 434–443\.Cited by:[§2](https://arxiv.org/html/2605.16312#S2.SS0.SSS0.Px5.p1.1)\.
- M\. Blanke, M\. Kinnaert, J\. Lunze, and M\. Staroswiecki \(2006\)Diagnosis and fault\-tolerant control\.Springer\.Cited by:[§2](https://arxiv.org/html/2605.16312#S2.SS0.SSS0.Px3.p1.1)\.
- C\. Boutilier, T\. Dean, and S\. Hanks \(1999\)Decision\-theoretic planning: structural assumptions and computational leverage\.Journal of Artificial Intelligence Research11,pp\. 1–94\.Cited by:[§2](https://arxiv.org/html/2605.16312#S2.SS0.SSS0.Px4.p1.1)\.
- N\. Brown, A\. Lerer, S\. Gross, and T\. Sandholm \(2019\)Deep counterfactual regret minimization\.InInternational Conference on Machine Learning,pp\. 793–802\.Cited by:[§2](https://arxiv.org/html/2605.16312#S2.SS0.SSS0.Px5.p1.1),[§5\.1](https://arxiv.org/html/2605.16312#S5.SS1.SSS0.Px4.p1.1)\.
- N\. Brown and T\. Sandholm \(2019\)Superhuman AI for multiplayer poker\.Science365\(6456\),pp\. 885–890\.Cited by:[§1](https://arxiv.org/html/2605.16312#S1.p1.1)\.
- A\. Gleave, M\. Dennis, C\. Wild, N\. Kant, S\. Levine, and S\. Russell \(2020\)Adversarial policies: attacking deep reinforcement learning\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2605.16312#S1.p1.1),[§2](https://arxiv.org/html/2605.16312#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Heinrich and D\. Silver \(2016\)Deep reinforcement learning from self\-play in imperfect\-information games\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2605.16312#S2.SS0.SSS0.Px5.p1.1),[3rd item](https://arxiv.org/html/2605.16312#S4.I1.i3.p1.1)\.
- Y\. Hu and Z\. Zhang \(2022\)Sparse adversarial attack in multi\-agent reinforcement learning\.arXiv preprint arXiv:2205\.09362\.Cited by:[§2](https://arxiv.org/html/2605.16312#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Huang, N\. Papernot, I\. Goodfellow, Y\. Duan, and P\. Abbeel \(2017\)Adversarial attacks on neural network policies\.arXiv preprint arXiv:1702\.02284\.Cited by:[§1](https://arxiv.org/html/2605.16312#S1.p1.1),[§2](https://arxiv.org/html/2605.16312#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Huang and S\. Ontañón \(2022\)A closer look at invalid action masking in policy gradient algorithms\.InInternational FLAIRS Conference,Cited by:[§2](https://arxiv.org/html/2605.16312#S2.SS0.SSS0.Px4.p1.1)\.
- G\. N\. Iyengar \(2005\)Robust dynamic programming\.Mathematics of Operations Research30\(2\),pp\. 257–280\.Cited by:[§2](https://arxiv.org/html/2605.16312#S2.SS0.SSS0.Px2.p1.1)\.
- A\. S\. Klyubin, D\. Polani, and C\. L\. Nehaniv \(2005\)Empowerment: a universal agent\-centric measure of control\.InIEEE Congress on Evolutionary Computation,pp\. 128–135\.Cited by:[§2](https://arxiv.org/html/2605.16312#S2.SS0.SSS0.Px4.p1.1)\.
- M\. Lanctot, E\. Lockhart, J\. Lespiau, V\. Zambaldi,et al\.\(2019\)OpenSpiel: a framework for reinforcement learning in games\.arXiv preprint arXiv:1908\.09453\.Cited by:[§2](https://arxiv.org/html/2605.16312#S2.SS0.SSS0.Px5.p1.1),[§5\.1](https://arxiv.org/html/2605.16312#S5.SS1.SSS0.Px4.p1.1)\.
- M\. Lanctot, V\. Zambaldi, A\. Gruslys, A\. Lazaridou, K\. Tuyls, J\. Pérolat, D\. Silver, and T\. Graepel \(2017\)A unified game\-theoretic approach to multiagent reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[§2](https://arxiv.org/html/2605.16312#S2.SS0.SSS0.Px5.p1.1)\.
- Y\. Liang, Y\. Sun, R\. Zheng, X\. Liu, B\. Eysenbach, T\. Sandholm, F\. Huang, and S\. McAleer \(2024\)Game\-theoretic robust reinforcement learning handles temporally\-coupled perturbations\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2605.16312#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Lin, K\. Dzeparoska, S\. Q\. Zhang, A\. Leon\-Garcia, and N\. Papernot \(2020\)On the robustness of cooperative multi\-agent reinforcement learning\.InIEEE Security and Privacy Workshops,pp\. 62–68\.Cited by:[§2](https://arxiv.org/html/2605.16312#S2.SS0.SSS0.Px1.p1.1)\.
- D\. Mguni, Y\. Sun, H\. Chen, W\. Yang, A\. Darabi, L\. O\. Orimoloye, and Y\. Yang \(2025\)Learning robust multi\-agent policies via selective adversarial fault induction\.arXiv preprint arXiv:2508\.08800\.Cited by:[§2](https://arxiv.org/html/2605.16312#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Nilim and L\. El Ghaoui \(2005\)Robust control of Markov decision processes with uncertain transition matrices\.Operations Research53\(5\),pp\. 780–798\.Cited by:[§2](https://arxiv.org/html/2605.16312#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Niu, X\. Fan, Q\. Jing, S\. Luo, and Y\. Lu \(2026\)CAMA: exploring collusive adversarial attacks in cooperative multi\-agent reinforcement learning\.arXiv preprint arXiv:2603\.20390\.Cited by:[§2](https://arxiv.org/html/2605.16312#S2.SS0.SSS0.Px1.p1.1)\.
- L\. Pinto, J\. Davidson, R\. Sukthankar, and A\. Gupta \(2017\)Robust adversarial reinforcement learning\.InInternational Conference on Machine Learning,pp\. 2817–2826\.Cited by:[§2](https://arxiv.org/html/2605.16312#S2.SS0.SSS0.Px2.p1.1),[§5\.5](https://arxiv.org/html/2605.16312#S5.SS5.p1.6)\.
- D\. Silver, T\. Hubert, J\. Schrittwieser,et al\.\(2018\)A general reinforcement learning algorithm that masters chess, shogi, and Go through self\-play\.Science362\(6419\),pp\. 1140–1144\.Cited by:[§1](https://arxiv.org/html/2605.16312#S1.p1.1)\.
- Y\. Sun, R\. Zheng, Y\. Liang, and F\. Huang \(2022\)Who is the strongest enemy? towards optimal and efficient evasion attacks in deep RL\.International Conference on Learning Representations\.Cited by:[§2](https://arxiv.org/html/2605.16312#S2.SS0.SSS0.Px1.p1.1)\.
- C\. Tessler, Y\. Efroni, and S\. Mannor \(2019\)Action robust reinforcement learning and applications in continuous control\.International Conference on Machine Learning,pp\. 6215–6224\.Cited by:[§2](https://arxiv.org/html/2605.16312#S2.SS0.SSS0.Px2.p1.1),[§5\.5](https://arxiv.org/html/2605.16312#S5.SS5.p1.6)\.
- H\. Zhang, H\. Chen, D\. Boning, and C\. Hsieh \(2021\)Robust reinforcement learning on state observations with learned optimal adversary\.International Conference on Learning Representations\.Cited by:[§2](https://arxiv.org/html/2605.16312#S2.SS0.SSS0.Px1.p1.1)\.
- X\. Zhang, Y\. Ma, A\. Singla, and X\. Zhu \(2020\)Adaptive reward\-poisoning attacks against reinforcement learning\.International Conference on Machine Learning\.Cited by:[§1](https://arxiv.org/html/2605.16312#S1.p1.1),[§2](https://arxiv.org/html/2605.16312#S2.SS0.SSS0.Px1.p1.1)\.
- M\. Zinkevich, M\. Johanson, M\. Bowling, and C\. Piccione \(2007\)Regret minimization in games with incomplete information\.InAdvances in Neural Information Processing Systems,Vol\.20\.Cited by:[§2](https://arxiv.org/html/2605.16312#S2.SS0.SSS0.Px5.p1.1),[§5\.1](https://arxiv.org/html/2605.16312#S5.SS1.SSS0.Px4.p1.1)\.

## Appendix ATabular PPO Details

Our tabular PPO maintains a softmax policy over a preference tableθ​\(s,a\)\\theta\(s,a\)\. Action probabilities:π​\(a\|s\)=softmax​\(θ​\(s,⋅\)\)\\pi\(a\|s\)=\\text\{softmax\}\(\\theta\(s,\\cdot\)\)\. Updates use the clipped surrogate objectiveL=min⁡\(rt​At,clip​\(rt,1−ϵ,1\+ϵ\)​At\)L=\\min\(r\_\{t\}A\_\{t\},\\text\{clip\}\(r\_\{t\},1\{\-\}\\epsilon,1\{\+\}\\epsilon\)A\_\{t\}\)withrt=πnew/πoldr\_\{t\}=\\pi\_\{\\text\{new\}\}/\\pi\_\{\\text\{old\}\}, advantageAt=R−bA\_\{t\}=R\-b\(running baseline\), clip parameterϵ=0\.2\\epsilon=0\.2, entropy bonus coefficient0\.010\.01, learning rate0\.010\.01\.

## Appendix BExperimental Protocols

#### Self\-play protocol\.

Both victim and opponent are the same agent; the mask is applied to Player 0’s actions at every decision during both training and evaluation\. In the fixed\-opponent regime, a snapshot of the Q\-table is frozen before masking; this static copy serves as Player 1 while only Player 0 continues learning\.

#### Transfer experiment\.

The adversary is trained against one victim \(seed 42\) for 20 outer×\\times500 inner episodes\. This mask is applied without modification to fresh victims \(seeds 123–2048\), each pre\-trained 10k episodes then trained 10k under the transferred mask\.

#### Budget enforcement\.

Budgetkkis enforced via top\-kkprojection: the adversary trains over all states but at execution only the top\-kkby confidence \(max softmax prob\. minus uniform baseline\) have masks applied\. This approximates a hardL0L\_\{0\}constraint on mask support\.

#### Action\-removal granularity\.

Unless otherwise stated, a masked state removes exactly one legal action; neural adversaries output one removal choice or a no\-op\. Masks never remove all actions: if removing the selected action would leave an empty set, the original legal set is retained\. In two\-action games, removing one action naturally forces a singleton; in three\- or five\-action games, most masks leave multiple alternatives\.

## Appendix CHyperparameters

Table 6:Q\-Learning hyperparameters\.Table 7:DQN hyperparameters for Leduc\-scale experiments\.Table 8:Neural NFSP hyperparameters \(Leduc\-5\)\.Table 9:PPO hyperparameters\.Table 10:Adversary hyperparameters \(tabular / neural\)\.
## Appendix DNormalisation Details

Normalised performance usesnorm​\(r\)=\(r−rmin\)/\(rmax−rmin\)\\text\{norm\}\(r\)=\(r\-r\_\{\\min\}\)/\(r\_\{\\max\}\-r\_\{\\min\}\):

Table 11:Normalisation bounds \(max single\-hand loss/gain\)\.Normalisation aids cross\-game comparison but obscures absolute exploitability: a normalised\.37\.37in Leduc \(Δ​r=−3\.5\\Delta r=\-3\.5\) represents a larger absolute loss than\.27\.27in Kuhn \(Δ​r=−0\.9\\Delta r=\-0\.9\)\.

## Appendix EAdditional Ablations

Table 12:Mask timing controls in Leduc Q\-learning\.Table 13:Threat\-model ablation in Leduc Q\-learning: private vs\. public information adversary\.Table 14:CACv\-greedy oracle at budgetk=3k=3in Kuhn\.Table 15:Strict matched\-L0L\_\{0\}control in Leduc Q\-learning\. Matched random samples exactly the same number of information states as the learned adversarial mask\.Table 16:Precise effectiveL0L\_\{0\}support diagnostics in Leduc Q\-learning\.Table 17:Neural and defense ablations\.Table 18:Representative compute and sample costs\. Wall\-clock times are CPU runs on a desktop workstation\.Table 19:Budget sweep: victim reward \(raw\) under adversarial and random masking \(Kuhn, 10 seeds, 95% CIs\)\.Table 20:Secondary ablations in Kuhn Q\-learning\.Table 21:Learned Kuhn mask at full budget\.
## Appendix FLearning Curves Under Attack

To confirm that reported results are*asymptotic*\(converged\) rather than transient, we report reward in 500\-episode windows during post\-attack training\.

#### Kuhn \+ QL \(seed 42\)\.

Pre\-attack final window:−0\.16\-0\.16\. Post\-attack trajectory: reward drops to∼−0\.3\\sim\\\!\-0\.3within 2k episodes and oscillates around−0\.2\-0\.2to−0\.4\-0\.4for the remaining 13k episodes\. No recovery trend\.

#### Leduc \+ QL \(seed 42\)\.

Pre\-attack:\+0\.13\+0\.13\. Post\-attack: gradual degradation from0to−1\.0\-1\.0over 10k episodes, with oscillations\. Final window:−0\.40\-0\.40\. The attack takes∼\\sim5k episodes to fully manifest as the adversary’s policy\-gradient updates progressively sharpen\.

#### Leduc \+ NFSP \(seed 42\)\.

Pre\-attack:−0\.30\-0\.30\. Post\-attack: initial spike to\+1\.0\+1\.0\(NFSP’s average strategy briefly exploits the adversary’s early exploration\), followed by monotonic decline to−0\.51\-0\.51by episode 15k\. The adversary overcomes NFSP’s initial resistance\.

#### Leduc\-5 \+ NFSP \(5 seeds\)\.

Pre\-attack:−0\.06\-0\.06\. Under adversarial masking with 25 outer iterations: progressive degradation from−0\.5\-0\.5\(outer 5\) to−1\.9\-1\.9\(outer 25\)\. Continued training for 15k additional episodes under the converged mask: reward oscillates between−1\.2\-1\.2and−2\.1\-2\.1with no recovery trend\. The adversary overcomes NFSP’s average\-strategy buffer at this scale\.

These curves confirm that \(a\) the attack converges and is not a transient phenomenon, \(b\) victims do not recover even with3×3\\timesextended training under the mask, and \(c\) the no\-recovery property holds across algorithms and game sizes\.