A Structural Threshold in Decision Capacity Governs Collapse in Self-Play Reinforcement Learning

# A Structural Threshold in Decision Capacity Governs Collapse in Self-Play Reinforcement Learning
Source: [https://arxiv.org/html/2605.16315](https://arxiv.org/html/2605.16315)
###### Abstract

We show that a threshold in decision capacity determines whether self\-play reinforcement learning agents collapse under asymmetric rule perturbations\. Across poker variants \(Kuhn, Leduc, Leduc\-4\), matrix games \(Matching Pennies\), a dice game \(Liar’s Dice; 24,576 info sets\), and six learning algorithms \(Q\-Learning, SARSA, REINFORCE, PPO, DQN, NFSP\), eliminating all positive\-reach contingent decisions causes rapid convergence to a deterministic exploitation attractor \(DEA\)—a fixed point at near\-maximal loss\. Preserving even a single positive\-reach contingent decision point prevents this collapse\. A frozen baseline and fixed\-opponent control confirm the mechanism is co\-adaptation under constraint, not the perturbation itself\. The phenomenon is timing\-invariant, fully reversible upon action restoration, and intensifies under function approximation\. These results establish a practically sharp threshold induced by the discontinuity atCACw=0\\text\{CAC\}\_\{w\}=0, with severity scaling continuously via reach\-weighted capacity in all tested domains\.

## 1Introduction

Multi\-agent reinforcement learning \(MARL\) agents trained through self\-play have achieved superhuman performance in complex games\(Silveret al\.,[2018](https://arxiv.org/html/2605.16315#bib.bib4); Brown and Sandholm,[2019](https://arxiv.org/html/2605.16315#bib.bib8); Vinyalset al\.,[2019](https://arxiv.org/html/2605.16315#bib.bib12)\), yet their robustness to structural changes in the environment remains poorly understood\. Prior work has focused primarily on adversarial perturbations to observations or rewards\(Gleaveet al\.,[2020](https://arxiv.org/html/2605.16315#bib.bib6)\), opponent modelling under distribution shift\(Foersteret al\.,[2018](https://arxiv.org/html/2605.16315#bib.bib14)\), or training stability in population\-based methods\(Balduzziet al\.,[2019](https://arxiv.org/html/2605.16315#bib.bib15)\)\. Structural changes to the*action space*—where an agent permanently loses access to certain actions—remain largely unexplored\.

Such perturbations arise naturally in practice\. In robotics, hardware failures may disable actuators, eliminating actions from an agent’s repertoire mid\-deployment\. In financial trading, regulatory changes can restrict previously available strategies\. In multi\-agent software systems, API deprecations may remove action endpoints\. Understanding how self\-play agents respond to these asymmetric capability losses is essential for deploying RL systems in environments where the action space is not guaranteed to remain static\.

We study this question in discrete, imperfect\-information games by deterministically removing one player’s ability to bet or raise at specified subsets of decision nodes\. We discover a structural failure mode with a precise threshold: the agent’s*contingent action capacity*\(CAC\)—the number of information sets at which it retains more than one legal action—governs whether self\-play dynamics collapse or stabilise\. We report unweighted CAC for interpretability; the reach\-weighted variantCACw\\text\{CAC\}\_\{w\}discounts rarely reached decision points and is the quantity that aligns with best\-response discontinuities\.

Our central finding is a pronounced threshold effect\. WhenCACw\\text\{CAC\}\_\{w\}drops to zero \(every positive\-reach decision is forced\), adaptive self\-play agents converge to a*deterministic exploitation attractor*\(DEA\)—the unique optimal policy of the single\-agent MDP induced by the forced player’s policyσ0f\\sigma\_\{0\}^\{f\}\. Its stability arises from the absence of counterfactual branching underCACw=0\\text\{CAC\}\_\{w\}=0: the opponent faces a stationary, deterministic environment with a unique best response, andε\\varepsilon\-greedy perturbations bound deviation from the theoretical minimum\. The static fact that zero contingency reduces the game to best response againstσ0f\\sigma\_\{0\}^\{f\}is expected; the contribution is the observed learning transition from collapse at zero capacity to near\-Nash stability when even one positive\-reach decision point is preserved\. A frozen baseline and fixed\-opponent control isolate continued co\-adaptation as the mechanism\. Our contributions:

- •We identify a pronounced reach\-weighted CAC threshold governing collapse in self\-play RL and provide formal propositions characterising the zero\-contingency fixed point\.
- •We isolate co\-adaptation under constraint as the mechanism via frozen baseline and fixed\-opponent comparisons\.
- •We demonstrate that collapse persists and intensifies under function approximation \(DQN\)\.
- •We replicate across eight game variants \(1–24,576 info sets\), six algorithms, multiple perturbation schedules, and show full reversibility\. Boundary conditions \(IPD, Liar’s Dice, cooperative games\) sharpen the threshold definition\.

## 2Related Work

#### Solving imperfect\-information games\.

Counterfactual regret minimisation \(CFR\)\(Zinkevichet al\.,[2007](https://arxiv.org/html/2605.16315#bib.bib1)\)and its variants have solved increasingly large poker games, from heads\-up limit hold’em\(Bowlinget al\.,[2015](https://arxiv.org/html/2605.16315#bib.bib2)\)to no\-limit variants via DeepStack\(Moravcíket al\.,[2017](https://arxiv.org/html/2605.16315#bib.bib7)\), Libratus\(Brown and Sandholm,[2018](https://arxiv.org/html/2605.16315#bib.bib9)\), and Pluribus\(Brown and Sandholm,[2019](https://arxiv.org/html/2605.16315#bib.bib8)\)\. These systems assume a fixed game structure\. We study what happens when the game structure changes asymmetrically after training\.

#### Self\-play reinforcement learning\.

Self\-play has driven advances from TD\-Gammon\(Tesauro,[1994](https://arxiv.org/html/2605.16315#bib.bib3)\)through AlphaZero\(Silveret al\.,[2018](https://arxiv.org/html/2605.16315#bib.bib4)\)and AlphaStar\(Vinyalset al\.,[2019](https://arxiv.org/html/2605.16315#bib.bib12)\)\. Neural Fictitious Self\-Play \(NFSP\)\(Heinrich and Silver,[2016](https://arxiv.org/html/2605.16315#bib.bib10)\)combines RL with average\-strategy tracking\. Policy\-Space Response Oracles \(PSRO\)\(Lanctotet al\.,[2017](https://arxiv.org/html/2605.16315#bib.bib11)\)maintain diverse strategy populations\. However, self\-play dynamics can be unstable—agents may cycle, overfit to their own weaknesses, or exhibit non\-transitive behaviour\(Balduzziet al\.,[2019](https://arxiv.org/html/2605.16315#bib.bib15); Lanctotet al\.,[2019](https://arxiv.org/html/2605.16315#bib.bib5)\)\. Our work identifies a distinct failure mode: co\-adaptation\-driven collapse under asymmetric action\-space constraints\.

#### Robustness in multi\-agent RL\.

The MARL robustness literature encompasses adversarial policies\(Gleaveet al\.,[2020](https://arxiv.org/html/2605.16315#bib.bib6)\), opponent\-learning awareness\(Foersteret al\.,[2018](https://arxiv.org/html/2605.16315#bib.bib14)\), and distributional robustness\(Zhanget al\.,[2021](https://arxiv.org/html/2605.16315#bib.bib13)\)\. These typically perturb observations, rewards, or opponent behaviour\. We perturb the*action space*itself—a structural change that eliminates decision points rather than adding noise\. This reveals a qualitative threshold effect that continuous perturbations cannot produce\.

#### Action masking and constrained RL\.

Invalid action masking is standard practice in game AI\(Huang and Ontañón,[2022](https://arxiv.org/html/2605.16315#bib.bib19)\), and constrained MDPs formalise action restrictions\(Altman,[1999](https://arxiv.org/html/2605.16315#bib.bib20)\)\. Robust MDPs address uncertain transitions\(Iyengar,[2005](https://arxiv.org/html/2605.16315#bib.bib21); Nilim and El Ghaoui,[2005](https://arxiv.org/html/2605.16315#bib.bib22)\), and stochastic action\-set MDPs model settings where available actions vary\(Boutilieret al\.,[2018](https://arxiv.org/html/2605.16315#bib.bib23)\)\. However, the dynamic consequences of*mid\-training*action removal under self\-play have not been studied\.

#### Self\-play stabilisation\.

Opponent\-shaping methods such as LOLA\(Foersteret al\.,[2018](https://arxiv.org/html/2605.16315#bib.bib14)\)and SOS\(Letcheret al\.,[2019](https://arxiv.org/html/2605.16315#bib.bib24)\)stabilise self\-play by accounting for the opponent’s learning dynamics\. Population\-based methods including PSRO\(Lanctotet al\.,[2017](https://arxiv.org/html/2605.16315#bib.bib11)\)andα\\alpha\-PSRO\(Mulleret al\.,[2020](https://arxiv.org/html/2605.16315#bib.bib25)\)maintain diverse strategy populations to avoid cycling\. We test PSRO empirically and find it*mitigates*but does not eliminate collapse \(§[6\.3](https://arxiv.org/html/2605.16315#S6.SS3)\)\.

#### Exploitability and game\-theoretic evaluation\.

Exploitability—the gap between an agent’s value and the Nash equilibrium value—is the standard evaluation metric in computational game theory\(Johanson and Bowling,[2007](https://arxiv.org/html/2605.16315#bib.bib16); Timberset al\.,[2022](https://arxiv.org/html/2605.16315#bib.bib17)\)\. We connect our threshold effect to the best\-response structure of reduced games via formal propositions \(Section 5\)\.

## 3Background

#### Kuhn Poker\.

Three cards \(J<Q<KJ<Q<K\), two actions \(pass, bet\), ante 1\. Nash value for P0:−1/18≈−0\.056\-1/18\\approx\-0\.056\. 12 information sets\.

#### Leduc Poker\.

Six cards \(J,Q,K×2J,Q,K\\times 2suits\), three actions \(fold, check/call, raise\), two rounds, fixed\-limit betting\. Nash P0 value:≈−0\.087\\approx\-0\.087\. 288 information sets\. Our CFR implementation converges to−0\.0866\-0\.0866\.

#### Leduc\-4 Poker\.

A Leduc variant with four ranks \(J,Q,K,AJ,Q,K,A\) and three suits \(12 cards\)\. Same rules as Leduc\. 504 information sets\. Nash P0 value≈−0\.096\\approx\-0\.096\.

#### Liar’s Dice \(1 die\)\.

Two players each roll one six\-sided die \(private\)\. Players alternate claiming a minimum count of a face value across both dice; claims must strictly increase\. A player may challenge instead of claiming\. On challenge, the claim is verified—if true the challenger loses, otherwise the claimer loses \(±1\\pm 1\)\. 13 actions \(12 claims \+ challenge\)\. 24,576 information sets\. Our CFR implementation yields Nash P0 value≈−0\.076\\approx\-0\.076\.

#### Coordination Game\.

Two agents simultaneously choose among 3 actions to match a randomly revealed target\. Payoff \+1 if both match, 0 otherwise\. 10 rounds/episode\.*Cooperative*, not competitive—the first non\-zero\-sum domain tested\.

#### Matching Pennies\.

Two players, two actions \(heads, tails\), simultaneous single\-shot\. Nash: 50/50 mixed, value 0\. 1 information set per player\.

#### Agents\.

CFR: Nash equilibrium via full\-tree CFR, frozen post\-training\.Q\-Learning: Tabular,ε\\varepsilon\-greedy \(ε=0\.15\\varepsilon=0\.15\), MC terminal updates\.QL\-Frozen: Q\-Learning frozen at perturbation \(Q\-table andε\\varepsilonfixed\)\.DQN: 2\-layer MLP \(64 hidden\), experience replay, target network\.SARSA/REINFORCE: On\-policy tabular variants\.

## 4Methodology

Each experiment runs 20 seeds of 20,000 episodes \(50,000 for DQN\)\. Perturbation applied to Player 0 at the midpoint\.

#### Perturbation protocol\.

Unless stated otherwise, perturbations are deterministic: we fix a named action label or action set and remove it from Player 0’s legal set after the midpoint\. In poker domains this removes bet/raise; in Matching Pennies it removes heads; in IPD it removes cooperate; in Liar’s Dice boundary experiments it removes high/all claims or forces the lowest legal action\. Scope is controlled by the experiment: root\-only removal affects only Player 0’s first decision, while full removal affects every Player 0 information set where the named action is legal\. Thus the Kuhn CAC sweep compares full bet removal \(zero contingency\), root\-only bet removal \(one residual call/fold decision\), and the unperturbed control\. Stochastic masking, when used, is a separate ablation over whether this fixed mask is active in an episode, not a random choice of which action to delete\.

#### Contingent action capacity \(CAC\)\.

The number of reachable information sets at which the perturbed agent retains\>1\>1legal action\. All CAC\-based experiments use the unweighted count as a proxy;CACw\\text\{CAC\}\_\{w\}\(reach\-weighted, Section 5\) is the governing theoretical quantity\. All observed CAC thresholds correspond toCACw\>0\\text\{CAC\}\_\{w\}\>0vs\.CACw=0\\text\{CAC\}\_\{w\}=0regimes; the unweighted count serves as an interpretable proxy that aligns with the reach\-weighted distinction in all tested games\.

#### Self\-play setup\.

A single agent plays both roles, with Q\-values indexed by player\-specific information states\. This is equivalent to independent self\-play in our zero\-sum setting because each player’s policy is implicitly determined by separate information\-state entries\. We verify this empirically: separate\-agent experiments produce identical results \(Appendix[B](https://arxiv.org/html/2605.16315#A2)\)\.

#### Statistical analysis\.

Pairedtt\-tests across seeds, bootstrap 95% CIs \(10,000 resamples\), Cohen’sdd\. Due to low across\-seed variance under paired evaluation, effect sizes are large and should be interpreted as indicating direction and reliability\.

#### Normalization\.

Cross\-game comparison uses\(r−rmin\)/\(rmax−rmin\)∈\[0,1\]\(r\-r\_\{\\min\}\)/\(r\_\{\\max\}\-r\_\{\\min\}\)\\in\[0,1\]\. This normalization aids comparison across reward scales but obscures absolute exploitability; we report exact exploitability where computable \(Kuhn, Leduc\) to complement normalized rewards\. Normalized comparisons align with exploitability trends in both games where both metrics are available\.

## 5Theory

We formalise the threshold effect in two\-player zero\-sum extensive\-form games\. The propositions below apply to the zero\-sum case; Section[6\.5](https://arxiv.org/html/2605.16315#S6.SS5)presents empirical evidence that cooperative and mixed\-motive settings produce a qualitatively different \(bounded degradation\) response\. Extending the formal treatment to general\-sum games is left to future work\. LetG=\(H,P,A,ℐ,u\)G=\(H,P,A,\\mathcal\{I\},u\)be a two\-player zero\-sum game with history setHH, player functionPP, action setsAA, information partitionℐ\\mathcal\{I\}, and utilityuu\.

#### Contingent action capacity\.

For a reduced gameG′G^\{\\prime\}, the unweighted contingent action capacity of Playeriiis

CAC​\(Pi\)=∑h∈ℐi𝟏​\[\|A​\(h\)\|≥2\]\.\\text\{CAC\}\(P\_\{i\}\)=\\sum\_\{h\\in\\mathcal\{I\}\_\{i\}\}\\mathbf\{1\}\[\|A\(h\)\|\\geq 2\]\.\(1\)This counts information sets at which the player still has a real choice after perturbation\.

#### Reach\-weighted contingent action capacity\.

For a strategy profileσ\\sigma, the*reach probability*of information sethhisρσ​\(h\)=∏h′⊏hσP​\(h′\)​\(h′\)\\rho^\{\\sigma\}\(h\)=\\prod\_\{h^\{\\prime\}\\sqsubset h\}\\sigma\_\{P\(h^\{\\prime\}\)\}\(h^\{\\prime\}\), the product of action probabilities on the path tohh\. We define the*reach\-weighted CAC*:

CACw​\(Pi\)=∑h∈ℐiρσ​\(h\)⋅𝟏​\[\|A​\(h\)\|≥2\]\\text\{CAC\}\_\{w\}\(P\_\{i\}\)=\\sum\_\{h\\in\\mathcal\{I\}\_\{i\}\}\\rho^\{\\sigma\}\(h\)\\cdot\\mathbf\{1\}\[\|A\(h\)\|\\geq 2\]\(2\)This refines the unweighted count by discounting information sets that are rarely reached\. The unweighted CAC used in experiments is an upper bound:CAC≥CACw\\text\{CAC\}\\geq\\text\{CAC\}\_\{w\}\.

###### Proposition 1\(Zero\-contingency exploitation\)\.

LetG′G^\{\\prime\}be the reduced game withCAC​\(P0\)=0\\text\{CAC\}\(P\_\{0\}\)=0\(equivalentlyCACw​\(P0\)=0\\text\{CAC\}\_\{w\}\(P\_\{0\}\)=0\)\. ThenP0P\_\{0\}’s value inG′G^\{\\prime\}is:

v0​\(G′\)=−maxπ1∈Π1​∑z∈Zu1​\(z\)⋅ρσ0f,π1​\(z\)v\_\{0\}\(G^\{\\prime\}\)=\-\\max\_\{\\pi\_\{1\}\\in\\Pi\_\{1\}\}\\sum\_\{z\\in Z\}u\_\{1\}\(z\)\\cdot\\rho^\{\\sigma\_\{0\}^\{f\},\\pi\_\{1\}\}\(z\)whereσ0f\\sigma\_\{0\}^\{f\}isP0P\_\{0\}’s forced \(deterministic\) strategy\. Moreover,P1P\_\{1\}’s best response is pure and computable inO​\(\|Z\|\)O\(\|Z\|\)time\.

###### Proof\.

Under zero contingency,σ0f\\sigma\_\{0\}^\{f\}is the unique strategy forP0P\_\{0\}\. The game reduces to a single\-player MDP forP1P\_\{1\}with a deterministic environment\. The optimal policy is a pure best response computable by backward induction overP1P\_\{1\}’s decision nodes, with complexity linear in the number of terminal histories reachable underσ0f\\sigma\_\{0\}^\{f\}\. ∎

###### Proposition 2\(Residual contingency bound\)\.

SupposeP0P\_\{0\}retains at least one information seth∗∈ℐ0h^\{\*\}\\in\\mathcal\{I\}\_\{0\}with\|A​\(h∗\)\|≥2\|A\(h^\{\*\}\)\|\\geq 2and reachρσ​\(h∗\)\>0\\rho^\{\\sigma\}\(h^\{\*\}\)\>0\. Letv0∗v\_\{0\}^\{\*\}beP0P\_\{0\}’s minimax value in the subgame rooted ath∗h^\{\*\}, and letv0forcedv\_\{0\}^\{\\text\{forced\}\}be the value under the forced policy at all other nodes\. Then:

v0​\(G′\)≥ρσ​\(h∗\)⋅v0∗\+\(1−ρσ​\(h∗\)\)⋅v0forcedv\_\{0\}\(G^\{\\prime\}\)\\geq\\rho^\{\\sigma\}\(h^\{\*\}\)\\cdot v\_\{0\}^\{\*\}\+\(1\-\\rho^\{\\sigma\}\(h^\{\*\}\)\)\\cdot v\_\{0\}^\{\\text\{forced\}\}In particular,v0​\(G′\)\>v0zero\-contv\_\{0\}\(G^\{\\prime\}\)\>v\_\{0\}^\{\\text\{zero\-cont\}\}wheneverv0∗\>v0forcedv\_\{0\}^\{\*\}\>v\_\{0\}^\{\\text\{forced\}\}andρσ​\(h∗\)\>0\\rho^\{\\sigma\}\(h^\{\*\}\)\>0\. The improvement is proportional to the reach of the retained node:

δ​\(h∗\)=ρσ​\(h∗\)⋅\(v0∗−v0forced\)\\delta\(h^\{\*\}\)=\\rho^\{\\sigma\}\(h^\{\*\}\)\\cdot\(v\_\{0\}^\{\*\}\-v\_\{0\}^\{\\text\{forced\}\}\)

###### Proof\.

By linearity of expectation over reach probabilities,P0P\_\{0\}’s value decomposes into the contribution from histories passing throughh∗h^\{\*\}\(whereP0P\_\{0\}plays optimally\) and those not passing throughh∗h^\{\*\}\(whereP0P\_\{0\}is forced\)\. The bound follows directly\. Whenh∗h^\{\*\}is reached with probability 1 \(as in Kuhn root\-only removal where all games pass through the “pb” node\),δ​\(h∗\)=v0∗−v0forced\\delta\(h^\{\*\}\)=v\_\{0\}^\{\*\}\-v\_\{0\}^\{\\text\{forced\}\}and the full minimax value is recovered\. ∎

###### Corollary 1\.

The transition fromCACw\>0\\text\{CAC\}\_\{w\}\>0toCACw=0\\text\{CAC\}\_\{w\}=0qualitatively changes the best\-response structure: from a game with strategic interaction \(requiring mixed or adaptive responses, with value bounded byδ​\(h∗\)\\delta\(h^\{\*\}\)\) to one with a trivially computable pure best response\. A single retained decision point with positive reach is sufficient to prevent collapse; a retained decision with zero reach provides no protection\.

###### Proposition 3\(DEA as a fixed point of self\-play dynamics\)\.

Consider tabular Q\-learning self\-play under zero contingency \(CACw=0\\text\{CAC\}\_\{w\}=0\)\. LetQ1​\(s,a\)Q\_\{1\}\(s,a\)denoteP1P\_\{1\}’s Q\-values\. Under standard assumptions \(ε\>0\\varepsilon\>0,αt→0\\alpha\_\{t\}\\to 0,∑αt=∞\\sum\\alpha\_\{t\}=\\infty\),P1P\_\{1\}’s Q\-values converge to the unique fixed pointQ1∗=QBR​\(σ0f\)Q\_\{1\}^\{\*\}=Q^\{\\text\{BR\}\(\\sigma\_\{0\}^\{f\}\)\}—the optimal Q\-function for a single\-agent MDP defined byP0P\_\{0\}’s forced policyσ0f\\sigma\_\{0\}^\{f\}\. The resulting joint policy profile\(σ0f,π1∗\)\(\\sigma\_\{0\}^\{f\},\\pi\_\{1\}^\{\*\}\)is a Nash equilibrium of the reduced gameG′G^\{\\prime\}and constitutes the DEA\.

###### Proof\.

UnderCACw=0\\text\{CAC\}\_\{w\}=0,P0P\_\{0\}’s policy is fixed and deterministic at every reachable information set\. FromP1P\_\{1\}’s perspective, the environment is a stationary MDP with deterministic transitions \(conditioned on the card deal\)\. Standard Q\-learning convergence guarantees\(Tesauro,[1994](https://arxiv.org/html/2605.16315#bib.bib3)\)apply: with decaying learning rate and persistent exploration,Q1→Q1∗Q\_\{1\}\\to Q\_\{1\}^\{\*\}almost surely\. SinceP0P\_\{0\}cannot deviate fromσ0f\\sigma\_\{0\}^\{f\}, the profile\(σ0f,π1∗\)\(\\sigma\_\{0\}^\{f\},\\pi\_\{1\}^\{\*\}\)satisfies the Nash equilibrium conditions ofG′G^\{\\prime\}\. The fixed point is unique because the reduced MDP has a unique optimal Q\-function\. Theε\\varepsilon\-greedy policy induced byQ1∗Q\_\{1\}^\{\*\}places probability≥1−ε\\geq 1\-\\varepsilonon the best response at each information set, yieldingv0≥Vmin\+εfloorv\_\{0\}\\geq V\_\{\\min\}\+\\varepsilon\_\{\\text\{floor\}\}whereεfloor=O​\(ε\)\\varepsilon\_\{\\text\{floor\}\}=O\(\\varepsilon\)\. ∎

## 6Experiments

Results are organised as: phenomenon \(§[6\.1](https://arxiv.org/html/2605.16315#S6.SS1)\), threshold \(§[6\.2](https://arxiv.org/html/2605.16315#S6.SS2)\), mechanism \(§[6\.3](https://arxiv.org/html/2605.16315#S6.SS3)\), generalisation \(§[6\.4](https://arxiv.org/html/2605.16315#S6.SS4)\), boundary conditions \(§[6\.5](https://arxiv.org/html/2605.16315#S6.SS5)\), and dynamics \(§[6\.6](https://arxiv.org/html/2605.16315#S6.SS6)\)\. Additional experiments appear in the Appendix\.

### 6\.1The Phenomenon

#### Zero contingency \(CAC=0=0\)\.

Bet removed from P0 at all nodes in Kuhn\. P0 is forced to check then fold\.

Table 1:Kuhn, zero contingency\. Player 0 reward \(20 seeds, 95% CIs\)\.Q\-Learning converges to the DEA at−0\.926\-0\.926\(normalized:0\.270\.27;d=−42\.0d=\-42\.0\) within a mean of four episodes\. CFR drops to−0\.22\-0\.22, bounded by its static opponent\.

![Refer to caption](https://arxiv.org/html/2605.16315v1/figures/full_removal.png)Figure 1:Zero contingency \(Kuhn\): Q\-Learning converges to the DEA within four episodes, illustrating the transition to deterministic exploitation\.
#### Residual contingency \(CAC=1=1\)\.

Bet removed at root only; P0 retains call/fold\. Q\-Learning drops modestly \(Δ=−0\.024\\Delta=\-0\.024,d=−0\.8d=\-0\.8\) then stabilises near Nash \(Table[2](https://arxiv.org/html/2605.16315#S6.T2)\)\.

Table 2:Kuhn, residual contingency \(CAC=1=1\)\. 20 seeds\.

### 6\.2The Threshold

Table 3:CAC sweep \(Kuhn\): the discontinuity at CAC=0→1=0\\to 1\.The jump from CAC=0=0to11isΔ=0\.86\\Delta=0\.86\(\+0\.21\+0\.21normalized\)\. The jump from 1 to 2 is marginal \(Δ=0\.03\\Delta=0\.03\)\. Figure[2](https://arxiv.org/html/2605.16315#S6.F2)shows the discontinuity on a normalized scale\.

![Refer to caption](https://arxiv.org/html/2605.16315v1/figures/capacity_threshold.png)Figure 2:The CAC threshold \(normalized\)\. The discontinuity at 0→\\to1 confirms a structural threshold\.
### 6\.3The Mechanism

#### Frozen baseline\.

QL\-Frozen avoids the DEA \(−0\.141\-0\.141vs−0\.927\-0\.927;Δ=−0\.787\\Delta=\-0\.787,p=0\.004p=0\.004,d=−2\.6d=\-2\.6; 5 seeds\)\. Co\-adaptation—not the constraint—drives collapse\.

#### Fixed opponent\.

Against a static Nash opponent \(5 seeds\), Q\-Learning drops only to−0\.228\-0\.228\(identical to CFR\)\. Under self\-play the same condition produces−0\.926\-0\.926\(20 seeds\)\. Co\-adaptation is necessary for catastrophic collapse\.

#### Population\-based training \(PSRO\)\.

Under PSRO with a 5\-policy population, P0’s post\-perturbation reward improves to−0\.418±0\.062\-0\.418\\pm 0\.062\(3 seeds\)—substantially better than self\-play \(−0\.927\-0\.927\) but still degraded from pre\-perturbation \(≈−0\.05\\approx\-0\.05\)\. The diverse population prevents the opponent from fully specialising against P0’s forced policy, but cannot recover the strategic loss from zero contingency\. PSRO*mitigates*but does not*eliminate*collapse: the structural vulnerability remains, but its exploitation is bounded by population diversity\. With larger populations, P0’s post\-perturbation reward improves monotonically \(−0\.275\-0\.275at pop\. 3;−0\.598\-0\.598at pop\. 15; all vs\.−0\.927\-0\.927under self\-play\)\. Population diversity bounds the degree of exploitation but cannot reintroduce eliminated strategic dimensions: the structural vulnerability at CAC=0=0persists regardless of opponent diversity, though its severity is modulated\.

![Refer to caption](https://arxiv.org/html/2605.16315v1/figures/frozen_full_removal.png)Figure 3:QL\-Frozen avoids the DEA, isolating co\-adaptation as the mechanism\.

### 6\.4Generalisation

#### Algorithm invariance\.

Table 4:Algorithm invariance under zero contingency \(Kuhn, 5 seeds\)\.All algorithms collapse\. DQN reaches−0\.994\-0\.994—*more*severe than tabular agents\. PPO and REINFORCE both reach−0\.500\-0\.500, partially protected by their softmax policy parameterisation\. Notably, NFSP—which maintains an explicit average strategy via supervised learning, designed specifically for imperfect\-information games—also collapses to−0\.505\-0\.505\. This demonstrates that structural constraints override regret\-minimisation guarantees: NFSP’s best\-response component still learns to exploit the forced opponent, and the average strategy cannot compensate when the action space itself has collapsed, as there is no historical action diversity to average over when P0’s every decision is forced\. To isolate theε\\varepsilon\-schedule effect, we ran DQN with fixedε=0\.15\\varepsilon=0\.15\(no decay\): post\-perturbation reward improved modestly to−0\.923\-0\.923, confirming that the deeper collapse under decayingε\\varepsilonis driven by reduced exploration \(ε→0\.01\\varepsilon\\to 0\.01\), but the structural vulnerability persists regardless\. Figure[5](https://arxiv.org/html/2605.16315#S6.F5)provides mechanistic insight: after perturbation, DQN’s policy entropy drops to near zero and Q\-value gaps spike, confirming rapid convergence to a deterministic policy\.

![Refer to caption](https://arxiv.org/html/2605.16315v1/figures/algorithm_overlay.png)Figure 4:All four algorithms converge toward the DEA\. DQN collapses deepest\.![Refer to caption](https://arxiv.org/html/2605.16315v1/figures/dqn_analysis.png)Figure 5:DQN neural analysis under zero contingency \(Kuhn\)\. Left: policy entropy drops to near zero post\-perturbation, confirming deterministic policy convergence\. Right: Q\-value gap \(\|Q​\(a0\)−Q​\(a1\)\|\|Q\(a\_\{0\}\)\-Q\(a\_\{1\}\)\|\) spikes, reflecting the network’s growing certainty in the forced\-fold action\.
#### Cross\-game replication\.

Table 5:Cross\-game normalized collapse severity\.†Challenge\-only retains strategic value \(§[6\.5](https://arxiv.org/html/2605.16315#S6.SS5)\)\.‡Cooperative game; forced single action reduces coordination but reward remains positive \(p=0\.001p\\\!=\\\!0\.001\)\.

Collapse holds across poker and matrix games \(1–504 info sets\)\. Liar’s Dice at both scales \(24,576 and200,000\+200\{,\}000\{\+\}info sets\) shows no collapse because challenge\-only play retains strategic flexibility—a boundary condition consistent with Proposition[2](https://arxiv.org/html/2605.16315#Thmproposition2)\. Severity scales inversely with residual action options: Matching Pennies \(most severe, 0\.07\) has zero residual options; Leduc variants \(least severe, 0\.49\) retain fold/check\-call\. Under residual contingency, no game collapses\.

### 6\.5Boundary Conditions

#### IPD: when perturbation aligns with equilibrium\.

Removing “cooperate” in the Iterated Prisoner’s Dilemma produces no collapse \(post=\+1\.12=\+1\.12\)\. IPD Nash is always\-defect; removing cooperate pushes P0*toward*equilibrium\. The threshold operates only when the perturbation forces the agent into a dominated regime where the opponent can extract surplus\.

#### Liar’s Dice: strategic flexibility vs\. action type\.

Removing all claims from P0 in Liar’s Dice forces challenge\-only play\. Neither tabular Q\-Learning \(1\-die; post=−0\.032=\-0\.032\) nor DQN \(2\-dice; post=\+0\.008=\+0\.008\) collapses, because challenging at different points is a strategically contingent decision\. However, when we force P0 to play*deterministically*\(always the lowest legal action, eliminating timing choices\), Q\-Learning collapses to−0\.524\-0\.524\(normalized:0\.240\.24;p<0\.0001p<0\.0001\)—demonstrating collapse at 24,576 information sets\. This confirms that the threshold depends on the reach\-weighted CAC \(Eq\.[2](https://arxiv.org/html/2605.16315#S5.E2)\), not the action\-type count: challenge\-only play retains highCACw\\text\{CAC\}\_\{w\}because challenge decisions are contingent on private information, whereas deterministic\-lowest play setsCACw=0\\text\{CAC\}\_\{w\}=0\.

#### Non\-zero\-sum domains: degradation without collapse\.

In the cooperative Coordination game \(match\-the\-target\), forcing P0 to a single action \(CAC=0=0\) degrades team performance \(\+1\.57→\+1\.44\+1\.57\\to\+1\.44,p=0\.001p=0\.001,d=−3\.8d=\-3\.8\) but does not produce convergence to the DEA\. In the Negotiation game \(ultimatum, 11 offer actions\), forcing P0 to a single offer \(CAC=0=0\) degrades outcomes; crucially, P1’s policy shifts toward rejection because it can condition on P0’s inability to adapt its offer, but this rejection is bounded—P1 does not converge to unconditional rejection as it would under zero\-sum exploitation pressure\. Retaining partial flexibility \(offers 0–2, CAC=3=3\) reverses the degradation entirely\. In contrast to competitive settings where zero contingency produces collapse to the DEA, cooperative and mixed\-motive environments exhibit bounded degradation, suggesting that the threshold interacts with the underlying interaction structure: zero\-sum dynamics amplify the constraint into catastrophic exploitation, while cooperative dynamics produce performance loss without attractor convergence\.

#### Timing invariance\.

Perturbation at episodes 3k, 10k, 17k yields identical collapse \(−0\.926\-0\.926,−0\.927\-0\.927,−0\.925\-0\.925\)\. The DEA is a structural attractor independent of training stage\.

### 6\.6Dynamics

#### Recovery\.

Restoring actions at episode 15k produces full recovery \(Δ=\+0\.90\\Delta=\+0\.90\) within four episodes—symmetric with collapse speed\. The DEA is a maintained attractor, not a corrupted representation\. \(See Figure[6](https://arxiv.org/html/2605.16315#S6.F6)and Table[6](https://arxiv.org/html/2605.16315#S6.T6)\.\)

Table 6:Recovery \(Kuhn, 5 seeds\): three\-phase Q\-Learning reward\.![Refer to caption](https://arxiv.org/html/2605.16315v1/figures/recovery_trajectory.png)Figure 6:Collapse and recovery are symmetric\. The DEA is maintained only by the constraint\.
#### Exploitability trajectory\.

Figure[7](https://arxiv.org/html/2605.16315#S6.F7)plots exact exploitability \(computed via best\-response tree walk\) over training in Kuhn\. Pre\-perturbation, exploitability converges toward zero as the agent approaches Nash\. Post\-perturbation, exploitability spikes\. For Leduc Poker, exact exploitability \(120 deals, full tree walk\) increases from1\.631\.63pre\-perturbation to1\.931\.93post\-perturbation \(Δ=\+0\.30\\Delta=\+0\.30\), corroborating the reward\-based findings with a game\-theoretic metric\. Reward degradation aligns with increased exploitability where computable \(Kuhn, Leduc\), supporting generalisation of the DEA characterisation to games where only reward is available\.

![Refer to caption](https://arxiv.org/html/2605.16315v1/figures/exploitability_curve.png)Figure 7:Exploitability over time \(Kuhn, full removal\)\. Post\-perturbation exploitability spikes as the agent’s policy becomes deterministically exploitable\.
#### Variance decomposition\.

Post\-collapse reward variance is5×10−65\\times 10^\{\-6\}, confirming the DEA is a deterministic fixed point\.

## 7Conclusion

We have identified a pronounced threshold in contingent action capacity that governs whether self\-play RL agents collapse under asymmetric action\-space perturbations\. The zero\-capacity reduced game is, by construction, a best\-response problem against a forced policy; the nontrivial result is that self\-play dynamics switch abruptly between a deterministic exploitation attractor atCACw=0\\text\{CAC\}\_\{w\}=0and near\-Nash stability once positive\-reach contingency remains\. The severity of the transition depends on the reach\-weighted CAC of the retained decision points \(Proposition[2](https://arxiv.org/html/2605.16315#Thmproposition2)\): higher reach yields stronger protection\. The finding is, within the tested class of discrete, imperfect\-information and matrix games, invariant across all tested algorithms \(Q\-Learning, SARSA, REINFORCE, PPO, NFSP, DQN\), co\-adaptation\-driven, timing\-invariant, and fully reversible\. A single retained decision point prevents catastrophic exploitation by maintaining strategic coupling between players\.

#### Limitations\.

Games studied range from 1 to200,000\+200\{,\}000\{\+\}information sets, with confirmed collapse at 24,576 info sets under true zero contingency \(Liar’s Dice\)\. Experiments report unweighted CAC as a readable proxy forCACw\\text\{CAC\}\_\{w\}; larger games may require direct reach\-weighted reporting\. Reversibility may not hold under deeper networks where gradient\-based collapse could corrupt representations\.

#### Future work\.

Scaling to larger games; cooperative and general\-sum settings; formal exploitability bounds beyond Kuhn \(approximate BR for Leduc/Liar’s Dice\); testing regret\-minimisation self\-play \(NFSP\) as a dynamic opponent; formalising degradation attractors in non\-zero\-sum settings\.

## References

- E\. Altman \(1999\)Constrained Markov decision processes: stochastic modeling\.CRC Press\.Cited by:[§2](https://arxiv.org/html/2605.16315#S2.SS0.SSS0.Px4.p1.1)\.
- D\. Balduzzi, M\. Garnelo, Y\. Bachrach, W\. Czarnecki, J\. Pérolat, M\. Jaderberg, and T\. Graepel \(2019\)Open\-ended learning in symmetric zero\-sum games\.InInternational Conference on Machine Learning,pp\. 434–443\.Cited by:[§1](https://arxiv.org/html/2605.16315#S1.p1.1),[§2](https://arxiv.org/html/2605.16315#S2.SS0.SSS0.Px2.p1.1)\.
- C\. Boutilier, C\. Hsu, B\. Kveton, O\. Meshi, and Z\. Qin \(2018\)Planning and acting in stochastic action sets\.InInternational Joint Conference on Artificial Intelligence,pp\. 4674–4682\.Cited by:[§2](https://arxiv.org/html/2605.16315#S2.SS0.SSS0.Px4.p1.1)\.
- M\. Bowling, N\. Burch, M\. Johanson, and O\. Tammelin \(2015\)Heads\-up limit hold’em poker is solved\.Science347\(6218\),pp\. 145–149\.Cited by:[§2](https://arxiv.org/html/2605.16315#S2.SS0.SSS0.Px1.p1.1)\.
- N\. Brown and T\. Sandholm \(2018\)Superhuman AI for heads\-up no\-limit poker: Libratus beats top professionals\.Science359\(6374\),pp\. 418–424\.Cited by:[§2](https://arxiv.org/html/2605.16315#S2.SS0.SSS0.Px1.p1.1)\.
- N\. Brown and T\. Sandholm \(2019\)Superhuman AI for multiplayer poker\.Science365\(6456\),pp\. 885–890\.Cited by:[§1](https://arxiv.org/html/2605.16315#S1.p1.1),[§2](https://arxiv.org/html/2605.16315#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Foerster, R\. Y\. Chen, M\. Al\-Shedivat, S\. Whiteson, P\. Abbeel, and I\. Mordatch \(2018\)Learning with opponent\-learning awareness\.InInternational Conference on Autonomous Agents and Multiagent Systems,pp\. 122–130\.Cited by:[§1](https://arxiv.org/html/2605.16315#S1.p1.1),[§2](https://arxiv.org/html/2605.16315#S2.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2605.16315#S2.SS0.SSS0.Px5.p1.1)\.
- A\. Gleave, M\. Dennis, C\. Wild, N\. Kant, S\. Levine, and S\. Russell \(2020\)Adversarial policies: attacking deep reinforcement learning\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2605.16315#S1.p1.1),[§2](https://arxiv.org/html/2605.16315#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Heinrich and D\. Silver \(2016\)Deep reinforcement learning from self\-play in imperfect\-information games\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2605.16315#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Huang and S\. Ontañón \(2022\)A closer look at invalid action masking in policy gradient algorithms\.InInternational FLAIRS Conference,Cited by:[§2](https://arxiv.org/html/2605.16315#S2.SS0.SSS0.Px4.p1.1)\.
- G\. N\. Iyengar \(2005\)Robust dynamic programming\.Mathematics of Operations Research30\(2\),pp\. 257–280\.Cited by:[§2](https://arxiv.org/html/2605.16315#S2.SS0.SSS0.Px4.p1.1)\.
- M\. Johanson and M\. Bowling \(2007\)Computing robust counter\-strategies\.InAdvances in Neural Information Processing Systems,Vol\.20\.Cited by:[§2](https://arxiv.org/html/2605.16315#S2.SS0.SSS0.Px6.p1.1)\.
- M\. Lanctot, E\. Lockhart, J\. Lespiau, V\. Zambaldi, S\. Upadhyay, J\. Pérolat, S\. Srinivasan, F\. Timbers, K\. Tuyls, S\. Omidshafiei,et al\.\(2019\)OpenSpiel: a framework for reinforcement learning in games\.arXiv preprint arXiv:1908\.09453\.Cited by:[§2](https://arxiv.org/html/2605.16315#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Lanctot, V\. Zambaldi, A\. Gruslys, A\. Lazaridou, K\. Tuyls, J\. Pérolat, D\. Silver, and T\. Graepel \(2017\)A unified game\-theoretic approach to multiagent reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[§2](https://arxiv.org/html/2605.16315#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2605.16315#S2.SS0.SSS0.Px5.p1.1)\.
- A\. Letcher, J\. Foerster, D\. Balduzzi, T\. Rocktäschel, and S\. Whiteson \(2019\)Stable opponent shaping in differentiable games\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2605.16315#S2.SS0.SSS0.Px5.p1.1)\.
- M\. Moravcík, M\. Schmid, N\. Burch, V\. Lisý, D\. Morrill, N\. Bard, T\. Davis, K\. Waugh, M\. Johanson, and M\. Bowling \(2017\)DeepStack: expert\-level artificial intelligence in heads\-up no\-limit poker\.Science356\(6337\),pp\. 508–513\.Cited by:[§2](https://arxiv.org/html/2605.16315#S2.SS0.SSS0.Px1.p1.1)\.
- P\. Muller, S\. Omidshafiei, M\. Rowland, K\. Tuyls, J\. Perolat, S\. Liu, D\. Hennes, L\. Marris, M\. Lanctot, E\. Hughes,et al\.\(2020\)A generalized training approach for multiagent learning\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2605.16315#S2.SS0.SSS0.Px5.p1.1)\.
- A\. Nilim and L\. El Ghaoui \(2005\)Robust control of Markov decision processes with uncertain transition matrices\.Operations Research53\(5\),pp\. 780–798\.Cited by:[§2](https://arxiv.org/html/2605.16315#S2.SS0.SSS0.Px4.p1.1)\.
- D\. Silver, T\. Hubert, J\. Schrittwieser, I\. Antonoglou, M\. Lai, A\. Guez, M\. Lanctot, L\. Sifre, D\. Kumaran, T\. Graepel,et al\.\(2018\)A general reinforcement learning algorithm that masters chess, shogi, and Go through self\-play\.Science362\(6419\),pp\. 1140–1144\.Cited by:[§1](https://arxiv.org/html/2605.16315#S1.p1.1),[§2](https://arxiv.org/html/2605.16315#S2.SS0.SSS0.Px2.p1.1)\.
- G\. Tesauro \(1994\)TD\-Gammon, a self\-teaching backgammon program, achieves master\-level play\.Neural Computation6\(2\),pp\. 215–219\.Cited by:[§2](https://arxiv.org/html/2605.16315#S2.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2605.16315#S5.SS0.SSS0.Px2.3.p1.13)\.
- F\. Timbers, N\. Burch, M\. Schmid, M\. Bowling, and M\. Lanctot \(2022\)Approximate exploitability: learning a best response\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2605.16315#S2.SS0.SSS0.Px6.p1.1)\.
- O\. Vinyals, I\. Babuschkin, W\. M\. Czarnecki, M\. Mathieu, A\. Dudzik, J\. Chung, D\. H\. Choi, R\. Powell, T\. Ewalds, P\. Georgiev,et al\.\(2019\)Grandmaster level in StarCraft II using multi\-agent reinforcement learning\.Nature575\(7782\),pp\. 350–354\.Cited by:[§1](https://arxiv.org/html/2605.16315#S1.p1.1),[§2](https://arxiv.org/html/2605.16315#S2.SS0.SSS0.Px2.p1.1)\.
- K\. Zhang, Z\. Yang, and T\. Başar \(2021\)Multi\-agent reinforcement learning: a selective overview of theories and algorithms\.Handbook of Reinforcement Learning and Control,pp\. 321–384\.Cited by:[§2](https://arxiv.org/html/2605.16315#S2.SS0.SSS0.Px3.p1.1)\.
- M\. Zinkevich, M\. Johanson, M\. Bowling, and C\. Piccione \(2007\)Regret minimization in games with incomplete information\.InAdvances in Neural Information Processing Systems,Vol\.20\.Cited by:[§2](https://arxiv.org/html/2605.16315#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix AHyperparameter Sensitivity

Collapse persists across all 9 tested hyperparameter combinations under zero contingency in Kuhn Poker\. Learning rate \(α\\alpha\) has negligible effect; exploration rate \(ε\\varepsilon\) shifts only the floor \(lowerε\\varepsilon= deeper collapse, as expected from reduced residual stochasticity\)\.

The DEA is robust to hyperparameter choice\. The only variation is theε\\varepsilon\-floor effect predicted by Proposition[1](https://arxiv.org/html/2605.16315#Thmproposition1)\.

## Appendix BSeparate Self\-Play Verification

Experiments with independent P0/P1 Q\-tables produce identical collapse dynamics \(separate:−0\.926±0\.003\-0\.926\\pm 0\.003; shared:−0\.927\-0\.927; difference:\+0\.0008\+0\.0008\), confirming that shared\-table self\-play is equivalent in our zero\-sum setting\.

## Appendix CTabular PPO Details

Our “tabular PPO” maintains a softmax policy over a preference tableθ​\(s,a\)\\theta\(s,a\)indexed by information state\. Action probabilities:π​\(a\|s\)=softmax​\(θ​\(s,⋅\)\)\\pi\(a\|s\)=\\text\{softmax\}\(\\theta\(s,\\cdot\)\)\. Updates use the clipped surrogate objectiveL=min⁡\(rt​At,clip​\(rt,1−ϵ,1\+ϵ\)​At\)L=\\min\(r\_\{t\}A\_\{t\},\\text\{clip\}\(r\_\{t\},1\-\\epsilon,1\+\\epsilon\)A\_\{t\}\)wherert=πnew/πoldr\_\{t\}=\\pi\_\{\\text\{new\}\}/\\pi\_\{\\text\{old\}\},At=R−bA\_\{t\}=R\-b\(advantage with running baseline\), andϵ=0\.2\\epsilon=0\.2\. An entropy bonus−0\.01⋅H​\(π\)\-0\.01\\cdot H\(\\pi\)is added\. Learning rate:0\.010\.01\. The softmax parameterisation explains the partial protection: unlikeε\\varepsilon\-greedy, softmax policies cannot fully collapse to deterministic play under finiteθ\\thetaupdates, bounding post\-perturbation loss at≈−0\.50\\approx\-0\.50rather than−0\.93\-0\.93\.

## Appendix DFixed\-Opponent Calibration

Under forced check\-fold \(Kuhn, CAC=0=0\),P1P\_\{1\}’s best response is to always bet:P0P\_\{0\}checks,P1P\_\{1\}bets,P0P\_\{0\}folds, yielding\+1\+1forP1P\_\{1\}\(−1\-1forP0P\_\{0\}\) on every hand\. Theε\\varepsilon\-floor prevents exact convergence, so the theoretical DEA bound isv0≈−1\+εfloorv\_\{0\}\\approx\-1\+\\varepsilon\_\{\\text\{floor\}\}—consistent with the observed self\-play collapse at−0\.926\-0\.926\.

Under a*fixed Nash opponent*\(which does*not*best\-respond toσ0f\\sigma\_\{0\}^\{f\}\),P0P\_\{0\}drops only to−0\.228\-0\.228\. This is far from the BR value of−1\-1because the Nash opponent was trained before perturbation and does not adapt\. The gap between−0\.228\-0\.228\(fixed Nash\) and−0\.926\-0\.926\(self\-play\) directly quantifies the contribution of co\-adaptation: without an adapting opponent,P0P\_\{0\}loses only what the Nash strategy extracts from forced check\-fold; with co\-adaptation,P1P\_\{1\}converges toward the BR and extracts nearly the maximum\.

## Appendix EAdditional Experiments

#### Severity sweep\.

Perturbation timing \(×\\timesseverity\) under Kuhn Poker\. Collapse severity is invariant to timing\.

#### Stochastic masking\.

Root\-only perturbation applied with 50% probability per episode \(Kuhn\): QL post=−0\.049=\-0\.049\(p=0\.06p=0\.06\)\. Intermittent perturbation does not trigger collapse\.

#### Variance decomposition\.

Post\-collapse Q\-Learning reward variance:Vtotal=5×10−6V\_\{\\text\{total\}\}=5\\times 10^\{\-6\},Venv=7×10−6V\_\{\\text\{env\}\}=7\\times 10^\{\-6\},Vpolicy=6×10−6V\_\{\\text\{policy\}\}=6\\times 10^\{\-6\}\. Near\-zero total variance confirms the DEA is a deterministic fixed point\.

## Appendix FReach\-Weighted CAC Values

The “pb” node has average reachρ=0\.473\\rho=0\.473under Nash play \(P0 passes at root, P1 bets\)\. This positive reach confirms Proposition[2](https://arxiv.org/html/2605.16315#Thmproposition2): the retained node has sufficient strategic weight to prevent collapse\. The improvementδ​\(h∗\)∝ρ​\(h∗\)\\delta\(h^\{\*\}\)\\propto\\rho\(h^\{\*\}\)is directly observable in the data\.

## Appendix GDQN Hyperparameters and Normalization

#### Normalization bounds per game\.

#### Exploitability computation\.

Exact exploitability is computed for Kuhn Poker via full best\-response tree walk over all 6 deals:Exploit​\(π\)=\(BRP0​\(π\)\+BRP1​\(π\)\)/2\\text\{Exploit\}\(\\pi\)=\(\\text\{BR\}\_\{P\_\{0\}\}\(\\pi\)\+\\text\{BR\}\_\{P\_\{1\}\}\(\\pi\)\)/2, whereBRPi\\text\{BR\}\_\{P\_\{i\}\}is the best\-response value for playeriiagainstπ\\piplaying as the opponent\. This is exact \(no approximation\) for Kuhn’s 12 information sets\.

## Appendix HEntropy Regularisation

We test whether entropy regularisation \(Q​\(s,a\)←Q​\(s,a\)\+α​\(r\+τ​H​\(πs\)−Q​\(s,a\)\)Q\(s,a\)\\leftarrow Q\(s,a\)\+\\alpha\(r\+\\tau H\(\\pi\_\{s\}\)\-Q\(s,a\)\)\) can prevent collapse\. Under zero contingency in Kuhn:

Entropy regularisation has no measurable effect on collapse severity\. The DEA is robust to diversity\-encouraging bonuses because the zero\-contingency constraint eliminates the agent’s ability to act on any maintained diversity\. The structural nature of the threshold—not insufficient exploration—is what drives collapse\.

## Appendix IReach Sensitivity

We varyε\\varepsilon\(exploration rate\) to change the empirical reach of the retained “pb” node under root\-only perturbation in Kuhn:

Post\-perturbation reward remains stable near−0\.06\-0\.06to−0\.09\-0\.09across all reach values, confirming that the retained node provides effective protection onceCACw\>0\\text\{CAC\}\_\{w\}\>0, with only modest sensitivity to the exact reach probability\.