Towards Learning Representations of Policies in Two-Player Zero-Sum Imperfect-Information Games

# Towards Learning Representations of Policies in Two-Player Zero-Sum Imperfect-Information Games
Source: [https://arxiv.org/html/2607.01498](https://arxiv.org/html/2607.01498)
\\optauthor\\Name

Kevin Wang∗\\Emailkevin\_a\_wang@brown\.edu \\NameKevin Yang∗\\Emailkevin\_c\_yang@brown\.edu \\NameArjun Prakash\\Emailarjun\_prakash@brown\.edu \\NameAmy Greenwald\\Emailamy\_greenwald@brown\.edu \\addrBrown University

\*Equal contribution\.

###### Abstract

We investigate the problem of learning useful policy representations \(embeddings\) in two\-player zero\-sum imperfect\-information games\. We make three contributions: First, we introduce methods of creating datasets of policies for a given game\. Second, we propose methods to learn policy representations\. Third, we introduce downstream tasks to evaluate the effectiveness of such representations\.

We evaluate each dataset method, embedding method, and downstream task on Kuhn and Leduc Poker\. Although our methods are very basic, we demonstrate that useful behavioral representations are present in the learned embeddings\. To our knowledge, this work is among the first to systematically compare self\-supervised learning techniques for learning policy representations in games\. Our code is available at\\urlhttps://github\.com/VitamintK/ssl\-project for others to extend\.

## 1Introduction

In two\-player zero\-sum imperfect\-information games, agents can benefit from reasoning about their opponent’s policies, and their own\. In general, policies in such games are stochastic\.111In contrast, in single\-agent or perfect\-information settings, reasoning about individual actions typically suffices – e\.g\. lookahead search in chess and go\.

For example, an agent seeking to play the Nash equilibrium at some decision point in a hand of poker may perform lookahead search by imagining that each player plays some policy for a few steps, and then evaluating the resulting public belief state\. This depth\-limited search is typically done tabularly222e\.g\. using counterfactual regret minimization or linear programming\[[Sokota et al\. \(2021\)](https://arxiv.org/html/2607.01498#bib.bib21),[Brown et al\. \(2020\)](https://arxiv.org/html/2607.01498#bib.bib2)\], which is feasible since such a depth\-limited policy has a tractable size\. However, in games with larger public belief states, such a policy becomes intractable to enumerate\. Thus, to perform a version of such a search, an agent will need to reason with compact representations of policies\.

So we want to learn good, compact representations of policies\. However, there is little existing research towards this end, especially in the context of two\-player zero\-sum imperfect\-information games\. There are few existing methods to learn policy embeddings, or comparisons between methods\. In order to compare methods, we also need an extensive set of evaluations, which does not exist\.

In this work, we begin an extensive, systematic approach to the problem of learning compact, useful representations of policies, particularly in two\-player zero\-sum imperfect\-information games\. We are particularly interested in methods that allow decoding embeddings into policies and in representations that can predict future payoffs\.

We do this in three parts:

1. 1\.We introduce three methods of creating datasets of policies for a given game\. \(Section[3\.1](https://arxiv.org/html/2607.01498#S3.SS1)\)
2. 2\.We propose several methods of varying complexity for learning representations of policies\. We also reimplement an existing method\. \(Section[3\.2](https://arxiv.org/html/2607.01498#S3.SS2)\)
3. 3\.We introduce several downstream tasks to evaluate the usefulness of the policy representations, and we use them to evaluate the methods\. \(Section[3\.3](https://arxiv.org/html/2607.01498#S3.SS3)\)

## 2Preliminaries

#### Imperfect Information Games

An imperfect\-information game \(IIG\) is one in which each player may not know the true state of the world\. A two\-player zero\-sum game is one in which there are two players, and the players’ payoffs sum to 0\. Formally, an IIG is given by the tuple\[[Rudolph et al\. \(2025\)](https://arxiv.org/html/2607.01498#bib.bib19)\]:⟨𝕊,𝔸,𝕆,𝕀,ℛ,𝒯,𝒪,𝒞,tmax⟩,\\langle\\mathbb\{S\},\\mathbb\{A\},\\mathbb\{O\},\\mathbb\{I\},\{\\mathcal\{R\}\},\{\\mathcal\{T\}\},\{\\mathcal\{O\}\},\{\\mathcal\{C\}\},t\_\{\\text\{max\}\}\\rangle,where𝕊\\mathbb\{S\}is the space of game states,𝔸\\mathbb\{A\}is the action space,𝕆\\mathbb\{O\}is the observation space and𝕀=∪t\(𝕆×𝔸\)t×𝕆\\mathbb\{I\}=\\cup\_\{t\}\(\\mathbb\{O\}\\times\\mathbb\{A\}\)^\{t\}\\times\\mathbb\{O\}is the space of information sets\. The reward function is given byℛ:𝕊×𝔸→ℝ\{\\mathcal\{R\}\}:\\mathbb\{S\}\\times\\mathbb\{A\}\\to\{\\mathbb\{R\}\}and𝒯:𝕊×𝔸→Δ​\(𝕊\)\{\\mathcal\{T\}\}:\\mathbb\{S\}\\times\\mathbb\{A\}\\to\\Delta\(\\mathbb\{S\}\)is the transition function\. The observation function, which determines which observation is given to the acting player is𝒪:𝕊→𝕆\{\\mathcal\{O\}\}:\\mathbb\{S\}\\to\\mathbb\{O\}and𝒞:𝕊→\{1,2\}\{\\mathcal\{C\}\}:\\mathbb\{S\}\\to\\\{1,2\\\}is the choice function which determines the acting player andtmaxt\_\{\\text\{max\}\}is the timestep at which the game ends\. Players interact with the game using policiesπi:𝕀→Δ​\(𝔸\)\\pi\_\{i\}:\\mathbb\{I\}\\to\\Delta\(\\mathbb\{A\}\)fori∈\{1,2\}i\\in\\\{1,2\\\}which maps from information set to a distribution over actions\. The objective of player 1 is to maximize expected return:𝒥=𝔼π​\[∑t=0tmaxℛ​\(𝐬t,𝐚t\)\],\{\\mathcal\{J\}\}=\{\\mathbb\{E\}\}\_\{\\pi\}\\big\[\\sum^\{t\_\{\\text\{max\}\}\}\_\{t=0\}\{\\mathcal\{R\}\}\(\\mathbf\{s\}^\{t\},\\mathbf\{a\}^\{t\}\)\\big\],where𝐬t∈𝕊\\mathbf\{s\}^\{t\}\\in\\mathbb\{S\}and𝐚t∈𝔸\\mathbf\{a\}^\{t\}\\in\\mathbb\{A\}are the states and actions at timett\.

#### Related Works

To save space, we describe related works in detail in Appendix[L](https://arxiv.org/html/2607.01498#A12)\.

## 3Methods

### 3\.1Methods for Making Datasets of Policies

In order to learn and evaluate representations of policies, we need a dataset of policies\. We propose three methods of generating such datasets:

In our first method, we simply randomly initialize policy neural networks, producing a diverse population of behaviorally distinct agents\. We use a custom initialization for the parameters of the neural networks, as the PyTorch default initialization produces behaviorally homogenous policies \(Appendix[A](https://arxiv.org/html/2607.01498#A1)\)\.

Our second method of producing a set of diverse policies for a game is to run thePolicy Space Response Oracle \(PSRO\)algorithm\[[Lanctot et al\. \(2017\)](https://arxiv.org/html/2607.01498#bib.bib9)\], a method that iteratively trains best\-responses to mixtures of other policies to produce an expanding pool of policies\.

Our third method of producing a set of diverse policies for a game is to run a variant ofneural population learning \(NeuPL\)\[[Liu et al\. \(2022\)](https://arxiv.org/html/2607.01498#bib.bib12),[Liu et al\. \(2024\)](https://arxiv.org/html/2607.01498#bib.bib13)\], which is the same as PSRO, but all policies share the same conditional neural network\. After training the network \(which is multiple agents\), we sample from its latent embedding space to generate new populations of policies\.

### 3\.2Methods for Learning Embeddings

Here, we present several methods for learning vector embeddings that represent policies\. Some methods we present are simple and naive, and others are comparatively more complex\. We assume that policies are policy neural networks\. In this work, we use policy neural networks which are MLPs with 3 hidden layers of size 256, and we train them \(when needed\) via PPO\[[Schulman et al\. \(2017\)](https://arxiv.org/html/2607.01498#bib.bib20)\]\.

#### Weight Autoencoder

The first naive method is to autoencode the weights of the policy neural network\.333Here, we colloquially refer to all the parameters of the neural network \(both weights and biases\) as “weights”\.The weight autoencoder \(Figure[2](https://arxiv.org/html/2607.01498#S3.F2)\) takes as input the flattened vector of policy weights and reconstructs the weights from a bottleneck layer\. Given an encoderfϕ:ℝd→ℝkf\_\{\\phi\}:\{\\mathbb\{R\}\}^\{d\}\\to\{\\mathbb\{R\}\}^\{k\}and decodergψ:ℝk→ℝdg\_\{\\psi\}:\{\\mathbb\{R\}\}^\{k\}\\to\{\\mathbb\{R\}\}^\{d\}, we obtain embeddings𝐳=fϕ​\(𝐖\)\\mathbf\{z\}=f\_\{\\phi\}\(\\mathbf\{W\}\)and minimize the reconstruction loss:

ℒWAE=‖𝐖−gψ​\(fϕ​\(𝐖\)\)‖22\.\{\\mathcal\{L\}\}\_\{\\text\{WAE\}\}=\\\|\\mathbf\{W\}\-g\_\{\\psi\}\(f\_\{\\phi\}\(\\mathbf\{W\}\)\)\\\|\_\{2\}^\{2\}\.
While a policy*is*\(in a sense\) its parameter vector, making weight autoencoders a natural baseline, we hypothesize this approach will perform poorly: behavior is difficult to predict from raw weights, and the large input/output dimension forces either an intractably large autoencoder or an overly compressed bottleneck\.

We note that the decoder is a hypernetwork\[[Ha et al\. \(2016\)](https://arxiv.org/html/2607.01498#bib.bib6)\]\. In this work, we simply use an MLP for the autoencoder\. However, it is plausible that state\-of\-the\-art hypernetwork architectures may make this type of method more feasible\[[Chauhan et al\. \(2024\)](https://arxiv.org/html/2607.01498#bib.bib3)\]\.

![Refer to caption](https://arxiv.org/html/2607.01498v1/figs/wae.png)Figure 1:Weight autoencoder
![Refer to caption](https://arxiv.org/html/2607.01498v1/figs/tae.png)Figure 2:Trajectory encoder

#### Functional Encoder

The functional encoder444diagram in Appendix[B](https://arxiv.org/html/2607.01498#A2)uses the same encoder\-decoder architecture as the weight autoencoder, but instead of reconstructing weights directly, it evaluates both the original and reconstructed networks on randomly sampled information statesI∼𝕀I\\sim\\mathbb\{I\}\. Let𝐖^=gψ​\(fϕ​\(𝐖\)\)\\hat\{\\mathbf\{W\}\}=g\_\{\\psi\}\(f\_\{\\phi\}\(\\mathbf\{W\}\)\)denote the reconstructed weights\. The loss minimizes the KL divergence between the action distributions:

ℒFAE=𝔼I∼𝕀\[DKL\(π𝐖\(⋅∣I\)∥π𝐖^\(⋅∣I\)\)\]\.\{\\mathcal\{L\}\}\_\{\\text\{FAE\}\}=\{\\mathbb\{E\}\}\_\{I\\sim\\mathbb\{I\}\}\\Big\[D\_\{\\text\{KL\}\}\\\!\\big\(\\pi\_\{\\mathbf\{W\}\}\(\\cdot\\mid I\)\\;\\\|\\;\\pi\_\{\\hat\{\\mathbf\{W\}\}\}\(\\cdot\\mid I\)\\big\)\\Big\]\.In this way, the autoencoder reconstructs the*behavior*of the original network rather than the weights themselves\.

#### Trajectory Encoder

To capture the behavioral semantics of policies rather than their internal network architecture, we use a trajectory encoder \([Figure2](https://arxiv.org/html/2607.01498#S3.F2)\) trained with contrastive learning inspired by SimCLR\[[Chen et al\. \(2020\)](https://arxiv.org/html/2607.01498#bib.bib4)\]\. A trajectoryτ=\(o0,a0,r0,…,oT,aT,rT\)\\tau=\(o\_\{0\},a\_\{0\},r\_\{0\},\\ldots,o\_\{T\},a\_\{T\},r\_\{T\}\)is collected by rolling out a policy against a randomly sampled opponent from the population\. An encoderhϕh\_\{\\phi\}maps variable\-length trajectories to a fixed\-dimensional latent space:𝐳=hϕ​\(τ\)\\mathbf\{z\}=h\_\{\\phi\}\(\\tau\)\.

For a mini\-batch ofNNpolicies, we sample two trajectories per policy \(against different opponents\), yielding2​N2Nembeddings\. Let𝐳i\\mathbf\{z\}\_\{i\}and𝐳j\\mathbf\{z\}\_\{j\}be a positive pair \(two trajectories from the same policy\)\. We optimize the normalized temperature\-scaled cross\-entropy \(NT\-Xent\) loss:

ℒNT\-Xent=−12​N​∑\(i,j\)log⁡exp⁡\(sim​\(𝐳i,𝐳j\)/t\)∑k=1k≠i2​Nexp⁡\(sim​\(𝐳i,𝐳k\)/t\),\{\\mathcal\{L\}\}\_\{\\text\{NT\-Xent\}\}=\-\\frac\{1\}\{2N\}\\sum\_\{\(i,j\)\}\\log\\frac\{\\exp\\\!\\big\(\\text\{sim\}\(\\mathbf\{z\}\_\{i\},\\mathbf\{z\}\_\{j\}\)/t\\big\)\}\{\\displaystyle\\sum\_\{\\begin\{subarray\}\{c\}k=1\\\\ k\\neq i\\end\{subarray\}\}^\{2N\}\\exp\\\!\\big\(\\text\{sim\}\(\\mathbf\{z\}\_\{i\},\\mathbf\{z\}\_\{k\}\)/t\\big\)\},wheresim​\(⋅,⋅\)\\text\{sim\}\(\\cdot,\\cdot\)denotes cosine similarity andttis a temperature parameter\. This objective ensures embeddings are invariant to stochastic environmental noise yet discriminative of strategic intent\.

Like the previous methods, this allows us to embed a given policy\. In addition, we can learn and create embeddings for arbitrary policies with only access to their trajectories, without knowledge of the neural network internals, and without the requirement that policies are neural networks\. Unlike the other methods, this doesn’t directly give us a method for turning an embedding into a policy\.

![Refer to caption](https://arxiv.org/html/2607.01498v1/figs/neupl.png)Figure 3:NeuPL\-style conditional policy network
#### NeuPL\-style Conditional Embeddings

Training NeuPL induces an embedding space in which any vector can be decoded into a policy via the conditional network \([Figure3](https://arxiv.org/html/2607.01498#S3.F3)\)\. Unlike the other methods, this generates the population and embeddings jointly rather than post\-hoc\.

#### Grover et al\. Encoder

FollowingGrover et al\. \([2018](https://arxiv.org/html/2607.01498#bib.bib5)\), we implement a hybrid generative\-discriminative baseline that learns policy embeddings from episode data\. An MLP encoderfθf\_\{\\theta\}processes individual \(observation, action\) pairs and averages the resulting vectors across the episode to form the embedding𝐳=1T​∑t=1Tfθ​\(ot,at\)\\mathbf\{z\}=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}f\_\{\\theta\}\(o\_\{t\},a\_\{t\}\)\. A conditional policy networkπϕ,θ​\(𝐚\|o,𝐳\)\\pi\_\{\\phi,\\theta\}\(\\mathbf\{a\}\|o,\\mathbf\{z\}\)predicts the agent’s actions given an observation and the embedding\. The model is trained with a hybrid loss combining an imitation \(behavioral cloning\) term and a triplet\-based agent identification term\. It differs from the trajectory encoder in both architecture \(Transformer vs\. MLP with mean pooling\) and training objective \(contrastive NT\-Xent vs\. hybrid imitation and triplet identification\)\.

#### Tabular Baseline

As a non\-learned baseline, we compute each agent’s complete action distribution at every information state in the game by querying the policy network\. Concatenating these distributions yields a fixed\-dimensional vector that fully describes the policy’s behavior\. This tabular representation requires no training\.

#### Identity Baseline

We can also test the usefulness of using the weights of the neural network directly, without embedding them to a lower\-dimensional representation\. For some tasks, this very high\-dimensional representation is too expensive to be tractable\.

![Refer to caption](https://arxiv.org/html/2607.01498v1/figs/tsne_kuhn_poker_trajectory_encoder_P30.png)

\(a\) Trajectory\-encoder embeddings

![Refer to caption](https://arxiv.org/html/2607.01498v1/figs/tsne_kuhn_poker_weight_autoencoder_P30.png)

\(b\) Weight autoencoder embeddings

Figure 4:t\-SNE visualization of embeddings in Kuhn Poker for 1000 random agents, then colored by aggression level \(bet/raise frequency\)\. With the trajectory encoder, aggressive strategies \(red, top\) clearly separated from passive strategies \(blue, bottom\)\. With the weight autoencoder, no such patterns emerge\.

### 3\.3Downstream Tasks

In this section, we introduce a suite of downstream tasks to evaluate the usefulness of policy representations\. Detailed descriptions in Appendix[M](https://arxiv.org/html/2607.01498#A13)\.

Task A \(Payoff Prediction vs\. Uniform Random\)\.How well can a linear predictor, given the embedding of a policy, predict the policy’s expected payoff against a uniform random opponent?

Task B \(Payoff Prediction: Policy vs\. Policy\)\.How well can a linear predictor, given the embeddings of a player 1 policy and a player 2 policy, predict the expected payoff of the two policies?

Task C \(Exploitability Prediction\)\.How well can a linear predictor, given the embedding of a policy, predict the payoff of an opponent’s best\-response to that policy? We compute ground\-truth best\-response values exactly, which is feasible since the games we use are small enough\.

Task D \(zero\-shot best\-response\)\. For a given Player 1 policyπ1\\pi\_\{1\}with embedding𝐳1\\mathbf\{z\}\_\{1\}, we train a zero\-shot best\-responder which, given𝐳1\\mathbf\{z\}\_\{1\}, approximates a best\-response againstπ1\\pi\_\{1\}\. The zero\-shot best\-responder is itself a NeuPL\-style conditional policy neural network \(Figure[3](https://arxiv.org/html/2607.01498#S3.F3)\)\.

Task E \(Agent Identification\)\.FollowingGrover et al\. \([2018](https://arxiv.org/html/2607.01498#bib.bib5)\), we evaluate whether the embeddings are discriminative enough to identify which policy generated a given trajectory\. For each ofNNpolicies, we collect multiple held\-out trajectories and embed them independently\. A classifier \(k\-nearest neighbors or linear probe\) is trained to predict the policy ID from the embedding\.

## 4Experimental Results

We test our methods on two zero\-sum imperfect\-information games: Kuhn Poker \(12 information states\) and Leduc Poker \(936\)\. In Appendix[F\.1](https://arxiv.org/html/2607.01498#A6.SS1)we report preliminary results on two larger games: Liar’s Dice \(1 die, 4 sides;∼\{\\sim\}1024\), and Phantom Tic\-Tac\-Toe \(∼\{\\sim\}500K\)\. Rules for all games can be found in Appendix[K](https://arxiv.org/html/2607.01498#A11)\.

Table 1:Performance on payoff prediction\. Each method was evaluated on randomly initialized policies, except NeuPL, which used NeuPL policies\. Identity Task B uses 10K sampled train pairs and 2\.5K sampled validation pairs due to the computational cost of evaluating all pairwise payoffs\.Imp\.\(improvement\) denotes reduction in MSE from baseline to the method\. Negative improvement occurs when the method doesworsethan the baseline \(e\.g\. due to overfitting\)\.±\\pmdenotes std over 3 random seeds\.Kuhn PokerLeduc PokerEncoderTaskPayoffBaselineImp\.Upp\.PayoffBaselineImp\.Upp\.IdentityTask D0\.253±\\pm0\.0410\.256±\\pm0\.092\-0\.0030\.497±\\pm0\.015––––WeightTask D0\.472±\\pm0\.0290\.412±\\pm0\.127\+0\.0600\.485±\\pm0\.0342\.124±\\pm0\.1532\.292±\\pm0\.099\-0\.1683\.172±\\pm0\.182NeuPLTask D0\.085±\\pm0\.0120\.038±\\pm0\.043\+0\.0470\.179±\\pm0\.0100\.260±\\pm0\.0220\.080±\\pm0\.113\+0\.1800\.924±\\pm0\.015

Table 2:Performance on Task D\.Payoffis the empirical payoff of the best\-responder to embedded policies\.Baselineis the empirical payoff of the best\-responder when it is trained with random embeddings\.Imp\.denotes the improvement from Baseline to Payoff \(positive improvement is good\)\.Upp\.denotes the upper bound: the average of the true best\-response value for each embedded policy\.±\\pmdenotes std over 3 random seeds\.We evaluate the quality of our learned representations by training a linear predictor on the frozen embeddings for the downstream payoff prediction tasks defined in Section 4\. As abaselinepredictor for each task, we use the mean target across the training set\. A qualitative visualization of the latent space is presented in[Figure4](https://arxiv.org/html/2607.01498#S3.F4)\.

#### Tasks A and B\.

We evaluate the identity baseline, the weight encoders, and NeuPL embeddings on the payoff prediction tasks \(A and B\)\. The results are in Table[1](https://arxiv.org/html/2607.01498#S4.T1)\. Neither the identity nor the weight encoders improve over simply predicting the mean\.

In Kuhn Poker, where the tabular vector is compact and complete, it dominates on Task B\. On Task A, all methods perform comparably \(∼\\sim71–72%\)\. This is expected: a lossless 24\-dimensional representation of the full policy is hard to beat in a game this small\. However, as game complexity grows, the picture reverses\. In Leduc, the tabular representation balloons to 2808 dimensions and*underperforms*the learned 128\-dimensional encoders on Task A \(22\.0% vs\. 27\.8–32\.3%\)\.

#### Tasks C and D\.

In Task C, we embed Player 1 policies, and learn to predict the payoffs of Player 2’s best\-response to each Player 1 policy\. Neither the identity baseline nor the weight reconstruction autoencoder perform better than predicting the mean on the dataset of random policies\. NeuPL does improve over the baseline on the dataset of NeuPL policies\.

In Task D[2](https://arxiv.org/html/2607.01498#S4.T2), we similarly embed Player 1 policies and train a Player 2 zero\-shot best\-responder with PPO to best\-respond to any given embedding\. NeuPL again outperforms the naive baseline\.555It is difficult to compare NeuPL to the other methods, since the population of policies used for training and evaluation is different than for the other methods\.

Table 3:Task E: Agent identification top\-1 accuracy for 500 eval agents using 5 train / 5 test trajectories per agent\.±\\pmdenotes std over 3 random seeds\. Random baseline is 0\.2%\.
#### Trajectory Encoder vs\. Grover\.

We compare against the Grover baselineGrover et al\. \([2018](https://arxiv.org/html/2607.01498#bib.bib5)\), the most closely related prior work, which learns embeddings from episode data using an MLP encoder with a hybrid generative\-discriminative objective \(imitation \+ triplet identification\)\. Compared to our trajectory encoder, the two approaches show complementary strengths\. On payoff prediction \([Table1](https://arxiv.org/html/2607.01498#S4.T1)\), Grover’s hybrid objective yields a consistent advantage on Task A\. However, on Task E agent identification, our contrastive Transformer encoder substantially outperforms Grover, achieving 47\.9% vs\. 25\.7% top\-1 accuracy on Kuhn and 57\.8% vs\. 38\.9% on Leduc with a linear classifier \(Table[3](https://arxiv.org/html/2607.01498#S4.T3)\)\.666Results on larger games in Appendix[F\.2](https://arxiv.org/html/2607.01498#A6.SS2)

## 5Conclusion

In this work, we presented a comparative study of techniques and downstream tasks for learning policy representations in imperfect\-information games\.

The NeuPL\-style embeddings show promise, and future work includes adding additional loss terms to further shape the embeddings\.

Natural extensions to the downstream task suite include tasks involving belief\-state prediction or integration with decision\-time planning\.

## 6Acknowledgments

Thanks to Randall Balestriero for inspiring this work and giving helpful suggestions and discussions on it\.

This material is based upon work supported by the National Science Foundation CISE Graduate Fellowships under Grant No\. 2313998\.

## References

- Albrecht and Stone \(2018\)Stefano V\. Albrecht and Peter Stone\.Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problems\.*Artificial Intelligence*, 258:66–95, May 2018\.ISSN 00043702\.[10\.1016/j\.artint\.2018\.01\.002](https://arxiv.org/doi.org/10.1016/j.artint.2018.01.002)\.URL\\urlhttp://arxiv\.org/abs/1709\.08071\.arXiv:1709\.08071 \[cs\]\.
- Brown et al\. \(2020\)Noam Brown, Anton Bakhtin, Adam Lerer, and Qucheng Gong\.Combining Deep Reinforcement Learning and Search for Imperfect\-Information Games, November 2020\.URL\\urlhttp://arxiv\.org/abs/2007\.13544\.arXiv:2007\.13544 \[cs\]\.
- Chauhan et al\. \(2024\)Vinod Kumar Chauhan, Jiandong Zhou, Ping Lu, Soheila Molaei, and David A\. Clifton\.A Brief Review of Hypernetworks in Deep Learning\.*Artificial Intelligence Review*, 57\(9\):250, August 2024\.ISSN 1573\-7462\.[10\.1007/s10462\-024\-10862\-8](https://arxiv.org/doi.org/10.1007/s10462-024-10862-8)\.URL\\urlhttp://arxiv\.org/abs/2306\.06955\.arXiv:2306\.06955 \[cs\]\.
- Chen et al\. \(2020\)Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton\.A simple framework for contrastive learning of visual representations\.In*International conference on machine learning*, pages 1597–1607\. PmLR, 2020\.
- Grover et al\. \(2018\)Aditya Grover, Maruan Al\-Shedivat, Jayesh K Gupta, Yuri Burda, and Harrison Edwards\.Learning policy representations in multiagent systems\.In*International Conference on Machine Learning*, pages 1802–1811\. PMLR, 2018\.
- Ha et al\. \(2016\)David Ha, Andrew Dai, and Quoc V Le\.Hypernetworks\.*arXiv preprint arXiv:1609\.09106*, 2016\.
- He et al\. \(2016\)He He, Jordan Boyd\-Graber, Kevin Kwok, and Hal Daumé III\.Opponent modeling in deep reinforcement learning\.In*International Conference on Machine Learning*, pages 1804–1813\. PMLR, 2016\.
- Kuhn \(2016\)Harold W Kuhn\.A simplified two\-person poker\.*Contributions to the Theory of Games*, 1:97–103, 2016\.
- Lanctot et al\. \(2017\)Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel\.A unified game\-theoretic approach to multiagent reinforcement learning\.*Advances in neural information processing systems*, 30, 2017\.
- Lanctot et al\. \(2019\)Marc Lanctot, Edward Lockhart, Jean\-Baptiste Lespiau, Vinicius Zambaldi, Satyaki Upadhyay, Julien Pérolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafiei, Daniel Hennes, Dustin Morrill, Paul Muller, Timo Ewalds, Ryan Faulkner, János Kramár, Bart De Vylder, Brennan Saeta, James Bradbury, David Ding, Sebastian Borgeaud, Matthew Lai, Julian Schrittwieser, Thomas Anthony, Edward Hughes, Ivo Danihelka, and Jonah Ryan\-Davis\.OpenSpiel: A Framework for Reinforcement Learning in Games\.*CoRR*, abs/1908\.09453, 2019\.URL\\urlhttp://arxiv\.org/abs/1908\.09453\.\_eprint: 1908\.09453\.
- Laskin et al\. \(2020\)Michael Laskin, Aravind Srinivas, and Pieter Abbeel\.CURL: Contrastive unsupervised representations for reinforcement learning\.In*International Conference on Machine Learning*, pages 5639–5650\. PMLR, 2020\.
- Liu et al\. \(2022\)Siqi Liu, Luke Marris, Daniel Hennes, Josh Merel, Nicolas Heess, and Thore Graepel\.Neupl: Neural population learning\.*arXiv preprint arXiv:2202\.07415*, 2022\.
- Liu et al\. \(2024\)Siqi Liu, Luke Marris, Marc Lanctot, Georgios Piliouras, Joel Z\. Leibo, and Nicolas Heess\.Neural Population Learning beyond Symmetric Zero\-sum Games, January 2024\.URL\\urlhttps://arxiv\.org/abs/2401\.05133v1\.
- Lupu et al\. \(2021\)Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster\.Trajectory diversity for zero\-shot coordination\.In*International Conference on Machine Learning*, pages 7204–7213\. PMLR, 2021\.
- Ma et al\. \(2024\)Wenhao Ma, Yu\-Cheng Chang, Jie Yang, Yu\-Kai Wang, and Chin\-Teng Lin\.Contrastive learning\-based agent modeling for deep reinforcement learning\.*arXiv preprint arXiv:2401\.00132*, 2024\.
- McIlroy\-Young et al\. \(2022\)Reid McIlroy\-Young, Russell Wang, Siddhartha Sen, Jon Kleinberg, and Ashton Anderson\.Learning Models of Individual Behavior in Chess\.In*Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining*, pages 1253–1263, August 2022\.[10\.1145/3534678\.3539367](https://arxiv.org/doi.org/10.1145/3534678.3539367)\.URL\\urlhttp://arxiv\.org/abs/2008\.10086\.arXiv:2008\.10086 \[cs\]\.
- Nashed and Zilberstein \(2022\)Samer Nashed and Shlomo Zilberstein\.A Survey of Opponent Modeling in Adversarial Domains\.*Journal of Artificial Intelligence Research*, 73:277–327, January 2022\.ISSN 1076\-9757\.[10\.1613/jair\.1\.12889](https://arxiv.org/doi.org/10.1613/jair.1.12889)\.URL\\urlhttps://www\.jair\.org/index\.php/jair/article/view/12889\.
- Papoudakis et al\. \(2021\)Georgios Papoudakis, Filippos Christianos, and Stefano Albrecht\.Agent modelling under partial observability for deep reinforcement learning\.In*Advances in Neural Information Processing Systems*, volume 34, pages 19210–19222, 2021\.
- Rudolph et al\. \(2025\)Max Rudolph, Nathan Lichtle, Sobhan Mohammadpour, Alexandre Bayen, J Zico Kolter, Amy Zhang, Gabriele Farina, Eugene Vinitsky, and Samuel Sokota\.Reevaluating policy gradient methods for imperfect\-information games\.*arXiv preprint arXiv:2502\.08938*, 2025\.
- Schulman et al\. \(2017\)John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov\.Proximal Policy Optimization Algorithms, August 2017\.URL\\urlhttp://arxiv\.org/abs/1707\.06347\.arXiv:1707\.06347 \[cs\]\.
- Sokota et al\. \(2021\)Samuel Sokota, Edward Lockhart, Finbarr Timbers, Elnaz Davoodi, Ryan D’Orazio, Neil Burch, Martin Schmid, Michael Bowling, and Marc Lanctot\.Solving Common\-Payoff Games with Approximate Policy Iteration, January 2021\.URL\\urlhttp://arxiv\.org/abs/2101\.04237\.arXiv:2101\.04237 \[cs\]\.
- Unterthiner et al\. \(2020\)Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Ilya Tolstikhin\.Predicting neural network accuracy from weights\.In*arXiv preprint arXiv:2002\.11448*, 2020\.
- Wallace et al\. \(2025\)Conor Wallace, Umer Siddique, and Yongcan Cao\.TransAM: Transformer\-based agent modeling for multi\-agent systems via local trajectory encoding\.In*Reinforcement Learning Conference*, 2025\.
- Xie et al\. \(2020\)Annie Xie, Dylan P Losey, Ryan Tolsma, Chelsea Finn, and Dorsa Sadigh\.Learning latent representations to influence multi\-agent interaction\.In*Conference on Robot Learning*, pages 575–588\. PMLR, 2020\.

## Appendix ADetails on Randomly Initializing Policy Neural Networks

Our layer initialization initializes weights orthogonally with a scaling factor of2\.22\.2\. It initializes the biases to0, except for the last layer \(whose outputs are the action logits\), for which each bias is randomly initialized between−1\-1and11\.

This is in contrast to PyTorch’s default layer initialization for linear layers, where weights are initialized according to a Kaiming uniform distribution, and biases are initialized randomly between−b,b\\sqrt\{\-b\},\\sqrt\{b\}, wherebbis the fan\-in of the neuron\.

For the results in Table[4](https://arxiv.org/html/2607.01498#A1.T4), we used 1000 policies randomly initialized via PyTorch default, 1000 policies randomly initialized with our method, 1000 policies generated by interpolating NeuPL embeddings, and 611 policies generated by running PSRO multiple times\.

We train NeuPL with a population size of 25 per player\. To generate each of the 1000 random policies, we randomly pick 2 of the agents, and interpolate uniformly at random to a convex combination of their embeddings\.

Table 4:Policies with parameters randomly initialized with PyTorch’s default, with our random initialization, generated via PSRO, and generated via NeuPL\. Values are: payoffs vs\. a uniform random strategy in Kuhn poker, and probability of taking the “check” action as the first player in Kuhn poker with a jack\. Values are mean±\\pmstandard deviation\.Policies initialized with the default method are less diverse \(they have much lower standard deviation in both values\)
## Appendix BBaseline Encoder Diagrams

Illustration of functional encoder is in Fig[5](https://arxiv.org/html/2607.01498#A2.F5)\.

![Refer to caption](https://arxiv.org/html/2607.01498v1/figs/fae.png)Figure 5:Functional encoder
## Appendix CFuture Work

While our experiments span games from∼\{\\sim\}12 to∼\{\\sim\}500K information states, scaling to even larger games such as 3x3 dark hex \(millions of information states\) and games with continuous action spaces remains an important direction\.

Our opponent\-adaptive strategy selection results highlight a key gap: representations trained for behavioral discrimination do not encode payoff\-structural similarity in a linearly recoverable form, as demonstrated by our Mahalanobis projection experiment\. A natural next step is to define training objectives that explicitly optimize for payoff\-relevant structure, for example contrastive losses where positive pairs are agents with correlated payoff profiles rather than same\-agent trajectories, or auxiliary losses that directly predict pairwise payoffs from embedding distances\.

Finally, we are interested in downstream tasks that directly enable the creation of good policies in games, either via payoff prediction or as part of a decision\-time planning process\.

## Appendix DLearned Similarity Metric

To test whether a better similarity metric could close the opponent adaptation gap, we trained a*learned Mahalanobis projection*on Liar’s Dice: a linear mapW∈ℝ64×128W\\in\\mathbb\{R\}^\{64\\times 128\}optimized on library agents so thatcos⁡\(W​zi,W​zj\)\\cos\(Wz\_\{i\},Wz\_\{j\}\)predicts payoff correlation between agentsiiandjj\. The projection achievesr=0\.70r\{=\}0\.70correlation with ground\-truth payoff similarity during training, confirming that it learns a meaningful transformation\. However, kernel regression with the learned similarity \(0\.347±\\pm0\.030\) does not outperform the simpler temperature\-scaled cosine \(0\.351±\\pm0\.024\) for either encoder, indicating that the bottleneck is not the similarity metric but the information content of the embeddings themselves\.

## Appendix EPSRO Population Results

We additionally evaluated encoders on populations generated by the Policy Space Response Oracle algorithm \(PSRO\)Lanctot et al\. \([2017](https://arxiv.org/html/2607.01498#bib.bib9)\)\. PSRO iteratively expands a population by training best\-response policies via deep RL against Nash equilibrium mixtures of the current population\. This produces strategically structured agents, unlike the random populations used in the main experiments\.

Results on Kuhn Poker are shown in[Table5](https://arxiv.org/html/2607.01498#A5.T5)\. The Trajectory Encoder maintains its advantage over weight\-space methods on PSRO populations, consistent with the random population findings\. PSRO training on Leduc Poker did not converge reliably within our computational budget due to the cost of computing best responses in the larger game tree; investigating scalable population generation for larger games is an important direction for future work\.

Table 5:Downstream task performance on PSRO\-generated populations \(Kuhn Poker only\)\.±\\pmdenotes std over 3 random seeds where available\.
## Appendix FAdditional Results

### F\.1Preliminary results on larger games

We performed preliminary experiments on the larger games of Liar’s Dice and Phantom Tic\-Tac\-Toe\. Results are reported in Table[6](https://arxiv.org/html/2607.01498#A6.T6)

Table 6:Payoff prediction performance for encoders evaluated on the same shared agent pools\. The Tabular baseline uses the complete action distribution at every information state as a fixed\-dimensional representation \(24\-dim for Kuhn, 2808\-dim for Leduc, 9216\-dim for Liar’s Dice\)\. In Kuhn, the lossless tabular representation dominates on Task B\. As game complexity grows, learned encoders overtake the tabular baseline despite compressing into 128 dimensions, and the tabular Task B exceeds 32GB memory \(OOM\)\.±\\pmdenotes std over 3 random seeds\.
### F\.2Task E: Agent Identification

FollowingGrover et al\. \([2018](https://arxiv.org/html/2607.01498#bib.bib5)\), we evaluate whether the trajectory encoder embeddings can identify which policy generated a held\-out trajectory\. For 500 random agents, we collect 5 train and 5 test trajectories each, embed them independently, and train classifiers on the embeddings\.

Table 7:Task E: Agent identification top\-1 accuracy for 500 eval agents using 5 train / 5 test trajectories per agent\.±\\pmdenotes std over 3 random seeds\. Random baseline is 0\.2%\.
### F\.3Our Trajectory Encoder vs\. Grover

On the larger games \(Liar’s Dice and Phantom TTT\), both encoders achieve near\-perfect identification and retrieval, with diminishing gaps\. On strategy classification, both encoders perform comparably\. These results indicate that the contrastive objective produces more fine\-grained, discriminative representations suited to identification tasks, while the hybrid imitation objective better captures the smooth payoff structure needed for regression\. We hypothesize that this gap arises because Grover’s imitation loss directly optimizes for encoding action distributions that determine payoffs whereas the contrastive NT\-Xent objective optimizes for inter\-policy discrimination on a normalized embedding sphere, discarding magnitude information that may be relevant for regression\. An ablation over training trajectories per policy \(Appendix[G](https://arxiv.org/html/2607.01498#A7)\) shows that the contrastive encoder is sample\-efficient: downstream performance is stable across a 5×\\timesrange \(10 to 50 trajectories\), indicating that only modest behavioral data per agent is needed during pre\-training\.

#### Additional Identification and Retrieval Tasks\.

To further characterize how the two training objectives shape the embedding space, we evaluate on three additional downstream tasks: few\-shot \(1\-shot\) agent identification \(Task E\), policy retrieval \(Task F\), and strategy classification \(Task G\)\. These tasks probe different aspects of representation quality, from fine\-grained individual discrimination to coarse behavioral categorization\. Results are shown in[Table8](https://arxiv.org/html/2607.01498#A6.T8)\.

The contrastive Trajectory Encoder consistently outperforms the Grover baseline on identification and retrieval tasks across all four games, though the gap narrows substantially on larger games\. On Task E \([Table7](https://arxiv.org/html/2607.01498#A6.T7)\), the contrastive encoder achieves 89\.5% top\-1 accuracy on Liar’s Dice \(Linear\), compared to 76\.6% for Grover\. On Phantom TTT, both encoders achieve near\-perfect identification \(\>\>95%\), with the contrastive encoder maintaining a slight edge \(97\.5% vs\. 95\.8% Linear\)\. On policy retrieval \([Table8](https://arxiv.org/html/2607.01498#A6.T8)\), both encoders achieve Recall@10\>\>99% on Liar’s Dice and Phantom TTT, indicating that the correct agent appears in the top 10 nearest neighbors virtually every time\. The gap is particularly striking in the 1\-shot setting on Kuhn and Leduc, where the contrastive encoder achieves 33\.1% and 35\.1% \(k\-NN\) compared to 18\.2% and 17\.2% for Grover, though on Phantom TTT both converge \(91\.5% vs\. 91\.1%\)\.

On strategy classification \(Task G\), both encoders perform comparably: 79\.7% vs\. 75\.3% on Kuhn and 56\.0% vs\. 56\.3% on Leduc\. This suggests that coarse\-grained strategic style \(aggression level\) is captured by both training objectives\. The substantially higher accuracy on Kuhn compared to Leduc for both encoders reflects the structural simplicity of Kuhn Poker, where aggression \(bet frequency\) is a near\-complete description of a policy’s behavior due to the minimal game tree \(3 cards, one betting round\)\. In Leduc, with 6 cards, a community card, and two betting rounds, policies vary along many more behavioral dimensions, making a single aggression scalar a coarser summary and the classification task correspondingly harder\.

Taken together, these results reveal that different SSL objectives produce representations with different epistemic content: the contrastive objective captures fine\-grained identity \(knowing*who*\), while the hybrid objective better captures payoff structure \(knowing*how much*\), and both capture broad behavioral categories\. This decomposition, which form of knowledge about an opponent each objective prioritizes, is consistent with their designs: NT\-Xent maximizes separation between distinct policies, while the Grover hybrid loss optimizes for action\-distribution fidelity\.

Table 8:Additional downstream tasks comparing Trajectory and Grover encoders\. Few\-shot ID uses 1 train trajectory per agent \(Top\-1 accuracy\)\. Retrieval reports Recall@kk\. Strategy classification reports accuracy over 5 aggression buckets \(random baseline: 20%\)\.±\\pmdenotes std over 3 seeds\. “–” indicates tasks not applicable \(strategy classification requires a game\-specific style label not defined for dice/board games\)\.
#### Opponent\-Adaptive Strategy Selection\.

To evaluate whether the learned embeddings have practical utility beyond probe\-based evaluation, we design a downstream application: embedding\-based opponent\-adaptive strategy selection\. Given a novel opponent, the task is to select the best counter\-strategy from a library of agents with known pairwise payoffs\.

We split the 500 evaluation agents into 50 held\-out test opponents and a 450\-agent response library\. We use*kernel regression*to predict each library agent’s payoff against the test opponent as a similarity\-weighted combination of its known payoffs against all library agents, then select the agent with the highest predicted payoff\. We compare against three baselines:*Oracle*\(best library agent per test opponent using ground\-truth payoffs\),*Random*\(uniformly random library agent\), and*Best\-Avg*\(library agent with highest mean payoff across the library, equivalent to kernel regression with uniform weights\)\. Results on Liar’s Dice and Leduc Poker are shown in[Table9](https://arxiv.org/html/2607.01498#A6.T9)\.

Table 9:Opponent\-adaptive strategy selection\. Mean payoff from a 450\-agent library against 50 held\-out opponents\.±\\pmstd over 10 seeds\.On Liar’s Dice, kernel regression with Trajectory embeddings and temperature scaling \(τ=5\\tau\{=\}5\) achieves 0\.351 vs\. 0\.343 for Best\-Avg, winning on 9 of 10 seeds \(sign testp≈0\.02p\{\\approx\}0\.02\)\. We observe the same pattern on Leduc Poker: Trajectory kernel \(τ=5\\tau\{=\}5\) achieves 0\.935 vs\. 0\.923 for Best\-Avg, winning on 8 of 10 seeds, but the improvement is not statistically significant \(pairedtt\-testp=0\.68p\{=\}0\.68\)\. We additionally trained a learned Mahalanobis similarity metric on Liar’s Dice, which did not outperform temperature\-scaled cosine despite achievingr=0\.70r\{=\}0\.70correlation with ground\-truth payoff similarity \([AppendixD](https://arxiv.org/html/2607.01498#A4)\), indicating that the bottleneck is the information content of the embeddings themselves\.

The Grover encoder’s kernel regression degenerates to Best\-Avg on both games, consistent with its near\-uniform cosine similarities \(μ=0\.90\\mu\{=\}0\.90,σ=0\.06\\sigma\{=\}0\.06on Liar’s Dice;[Figure6](https://arxiv.org/html/2607.01498#A6.F6)\)\. The large Oracle–Best\-Avg gaps on both games confirm that opponent\-specific information is exploitable in principle\. These results, replicated across two games, multiple similarity metrics, and a supervised learned metric, reveal an epistemic blind spot in current SSL representations: they encode enough to recognize an opponent but not enough to know how to exploit them, and post\-hoc correction via learned metrics cannot compensate for information that was never captured\. Whether payoff\-aware training objectives or uncertainty\-aware representations that explicitly model what is*not yet known*about an opponent could close this gap remains an open question\.

![Refer to caption](https://arxiv.org/html/2607.01498v1/x1.png)Figure 6:Cosine similarity matrices for 100 evaluation agents under Trajectory \(contrastive\) and Grover \(hybrid\) encoders on Liar’s Dice\. Agents are sorted by mean similarity\. The Trajectory encoder produces structured similarity patterns, while the Grover encoder’s similarities collapse to near\-uniform values, explaining why its kernel regression degenerates to the Best\-Avg baseline\.

## Appendix GTraining\-Time Trajectory Ablation

We ablate the number of trajectories per policy used during contrastive pre\-training on Liar’s Dice\.[Table10](https://arxiv.org/html/2607.01498#A7.T10)shows that downstream performance is remarkably stable across a 5×\\timesrange of training trajectories \(10 to 50 per policy\)\. Even with only 10 trajectories per policy, the encoder achieves 55\.5% improvement on Task A, comparable to the default setting of 50 trajectories \(55\.6%\)\. This indicates that the contrastive encoder extracts discriminative policy representations efficiently, requiring only a modest number of behavioral observations per agent during pre\-training\.

Table 10:Trajectory encoder ablation on Liar’s Dice \(seed 42\): downstream payoff prediction as a function of training trajectories per policy\.
## Appendix HExperimental Details and Hyperparameters

### H\.1Code Implementation

We use the OpenSpielLanctot et al\. \([2019](https://arxiv.org/html/2607.01498#bib.bib10)\)library\. We extensively use a modified version of IIG\-RL\-BenchmarkRudolph et al\. \([2025](https://arxiv.org/html/2607.01498#bib.bib19)\)for PPO training, PSRO, and NeuPL\.

### H\.2Standard Training Configuration

All experiments were conducted using three random seeds \(42, 43, 44\) and results were averaged, except for the opponent\-adaptive strategy selection experiment which uses 10 seeds for statistical testing\. For the Trajectory Encoder, we utilized 50 trajectories per policy during the training phase\. Qualitative visualizations using t\-SNE were generated with a perplexity value of 30\.

Table 11:Hyperparameters for Trajectory and Functional EncodersThe Grover encoder uses the same hyperparameters as described in the original workGrover et al\. \([2018](https://arxiv.org/html/2607.01498#bib.bib5)\): a 2\-layer MLP encoder with 128\-dimensional embeddings, trained with a hybrid imitation and triplet loss objective\. All Grover models across the four games use the same architecture and training configuration\.

### H\.3Hardware and Infrastructure

Experiments were executed on anNVIDIA Quadro RTX 6000\(24GB VRAM\) and anIntel\(R\) Xeon\(R\) Platinum 8268 CPU @ 2\.90GHz\. For Identity Task B, we evaluate sampled agent pairs because computing payoffs for the full pairwise evaluation set was too expensive for our available compute budget\.

## Appendix IReproducibility: CLI Commands

To reproduce the specific checkpoints mentioned in the results, the following commands can be executed:

```
# Trajectory Encoder (Kuhn Random)
python trajectory_encoder.py --game kuhn_poker --num-agents 500 --batch-size 16 \
--epochs 200 --lr 1e-4 --val-split 0.2 --lr-scheduler cosine --normalize

# Trajectory Encoder (Leduc Random)
python trajectory_encoder.py --game leduc_poker --num-agents 500 --batch-size 16 \
--epochs 200 --lr 1e-4 --val-split 0.2 --lr-scheduler cosine --normalize

# Functional Encoder (Kuhn Random)
python functional_autoencoder.py --game kuhn_poker --num-agents 500 --batch-size 16 \
--epochs 100

# Trajectory Encoder (Liar’s Dice Random)
python trajectory_encoder.py --game "liars_dice(numdice=1,dice_sides=4)" \
--agent-pool agent_pools/liars_dice_numdice1_dice_sides4_seed42_n500.pt \
--num-agents 500 --batch-size 16 --epochs 200 --lr 1e-4 --val-split 0.2 \
--lr-scheduler cosine --normalize

# Trajectory Encoder (Phantom TTT Random)
python trajectory_encoder.py \
--game "phantom_ttt(obstype=reveal-nothing)" \
--agent-pool agent_pools/phantom_ttt_seed42_n500.pt \
--num-agents 500 --batch-size 8 --epochs 50 --lr 1e-4 --val-split 0.2 \
--lr-scheduler cosine --normalize

# Functional Encoder (Leduc Random)
python functional_autoencoder.py --game leduc_poker --num-agents 500 --batch-size 16 \
--epochs 50 --dataset-fraction 0.02 --device cuda
```

## Appendix JDetailed Performance Metrics: Random Agents

The following table summarizes the Mean Squared Error \(MSE\) and corresponding baseline MSE for Tasks A and B using 1,000 random agents \(500 for training/upstream and 500 for evaluation/downstream\)\. Trajectory and Grover results use shared pre\-generated agent pools to ensure identical evaluation populations across encoders\. For models where three seeds were available, results represent individual run samples\.

Table 12:MSE Results per Seed for Random Agent Encoders### J\.1Performance Observations

- •Identity Baseline: The identity encoder \(raw parameters\) does not improve over the mean predictor on Task B, even when evaluated with sampled agent pairs due to the computational cost of the full pairwise evaluation\.
- •Tabular Baseline: The complete action distribution dominates on Kuhn Task B \(MSE∼\\sim0\.035 vs\.∼\\sim0\.077–0\.084 for learned encoders\), confirming that in small games, the lossless tabular policy is hard to beat\. In Leduc and Liar’s Dice, the high\-dimensional tabular vectors \(2808\-dim, 9216\-dim\) underperform learned 128\-dim encoders on Task A, and Task B exceeds 32GB memory \(OOM\) due to the high\-dimensional embedding pairs\. On Phantom TTT \(∼\{\\sim\}500K information states\), the tabular representation is entirely infeasible \(OOM on both tasks\)\.
- •Grover vs\. Trajectory: The Grover encoder outperforms the Trajectory encoder on Tasks A and B across all four games\. The gap is most pronounced on Phantom TTT \(48\.5% vs\. 9\.0% on Task A\) and Liar’s Dice \(67\.2% vs\. 56\.6% on Task A, 49\.6% vs\. 41\.9% on Task B\)\.
- •Weight and Functional Stability: Both the Weight Autoencoder and Functional Autoencoder largely mirrored baseline performance on random agents, suggesting that simple parameter or action distribution reconstruction is less effective than trajectory\-based modeling for these tasks\.

## Appendix KGame Rules

#### Kuhn Poker

is the simplest possible version of poker\. It was developed by game theorist Harold Kuhn in 1950 to study simplified zero\-sum gamesKuhn \([2016](https://arxiv.org/html/2607.01498#bib.bib8)\)\. It strips poker down to its absolute mathematical core: one card, one bet and two players \(P1 and P2\)\. The deck consists of 3 cards one KingKK, one QueenQQand a JackJJ\. Each player enters a nominal ante into the pot before being dealt one card\. There is one betting round\. The first player can Check or Bet\. If P1 checks, P2 can check \(followed by an immediate showdown\), or Bet\. If P1 Bets, player 2 can fold \(P1 wins\) or call \(showdown\)\. If P2 bets \(following P1 check\), P1 can fold or call\. If neither player folds, both players reveal their cards in the showdown whereK\>Q\>JK\>Q\>J\. The player with the highest card wins the pot\. The game is fully solved\. We know the exact Nash Equilibrium strategy \(e\.g\., if holding a Jack, Player 1 should never bet, but if holding a King, they should bet 3 times as often as they bet with a Queen to balance their bluffing range\)\.

#### Leduc Poker

adds a community card and multiple rounds of betting making it significantly more complicated and a better proxy for Texas Hold’em\. In this game there are two players and 6 cards: two Kings, Queens and Jacks\. To start, both players pay an ante and a then dealt one private card\.Round 1:Players bet based on their private card\. There is a fixed bet size\.The Flop:One community card is revealed from the remaining deck\.Round 2:Players bet again now knowing their card and the community card\.The Showdown:Both players reveal their cards; A pair is a winning hand, otherwise the player with the highest card wins\. If both players have the same card, the pot is split\.

#### Liar’s Dice

\(1 die, 4 sides\) is a bluffing game structurally distinct from poker\. Each of the two players privately rolls a single four\-sided die\. Players alternate making claims about the total count of a particular face value across*both*dice combined \(e\.g\., “there are at least two 3s”\)\. Each claim must be strictly higher than the previous one \(either a higher count or a higher face value\)\. A player may instead challenge the opponent’s last claim\. If challenged, both dice are revealed: if the claim is true, the challenger loses; otherwise, the claimant loses\. The game requires reasoning about hidden information \(the opponent’s die\) and incentivizes bluffing where players can make inflated claims to pressure opponents into folding or to set up a trap\. With 1 die and 4 sides, the game has approximately 1024 information states and 9 possible actions per state\.

#### Phantom Tic\-Tac\-Toe

is a partially observable variant of Tic\-Tac\-Toe where players cannot see their opponent’s moves\. Each player places marks on a 3×\\times3 board, but only observes their own marks and whether an attempted move was rejected \(because the cell was already occupied\)\. If a player attempts to place on an occupied cell, they are informed the move is invalid and must choose again, but they do not learn who occupies that cell\. The game ends when a player completes a row, column, or diagonal, or when the board is full \(draw\)\. With the “reveal\-nothing” observation type, the game has approximately 500,000 information states, making it the largest game in our benchmark and a test of scalability beyond card games\.

## Appendix LExtended Related Works

#### Agent Modeling and Policy Representations

Learning representations of other agents’ behavior is a central challenge in multi\-agent systems\.Albrecht and Stone \([2018](https://arxiv.org/html/2607.01498#bib.bib1)\)andNashed and Zilberstein \([2022](https://arxiv.org/html/2607.01498#bib.bib17)\)performed comprehensive surveys on opponent modeling\. Early work byHe et al\. \([2016](https://arxiv.org/html/2607.01498#bib.bib7)\)introduced DRON for opponent modeling via deep RL, andGrover et al\. \([2018](https://arxiv.org/html/2607.01498#bib.bib5)\)learn low\-dimensional policy embeddings using variational inference, evaluating on agent identification and reward prediction \(our Task E follows their identification protocol\)\. WhileGrover et al\. \([2018](https://arxiv.org/html/2607.01498#bib.bib5)\)desire “simulating the agent’s policy” and “distinguishing the agent’s policy” as the two desiderata of good representations, we focus more on predicting payoffs in two\-player zero\-sum games as a measure of good representations\. More recently,Papoudakis et al\. \([2021](https://arxiv.org/html/2607.01498#bib.bib18)\)learn agent representations under partial observability by training an encoder\-decoder to predict other agents’ actions from local observations\.Xie et al\. \([2020](https://arxiv.org/html/2607.01498#bib.bib24)\)learn latent representations of other agents’ strategies to influence multi\-agent interactions\. Most closely related to our trajectory encoder, CLAMMa et al\. \([2024](https://arxiv.org/html/2607.01498#bib.bib15)\)uses contrastive learning on local observations to produce real\-time policy representations of other agents, and TransAMWallace et al\. \([2025](https://arxiv.org/html/2607.01498#bib.bib23)\)uses a transformer to encode local trajectories into embeddings that capture other agents’ policies\. Our work differs from these approaches in that we \(i\) focus specifically on imperfect\-information games, \(ii\) systematically compare multiple types of policy embeddings and \(iii\) evaluate on game\-theoretic downstream tasks such as payoff and exploitability prediction rather than cooperative returns\.

#### Behavioral Diversity and Trajectory Representations

Several works have used trajectory\-level representations to measure or encourage policy diversity\. TrajeDiLupu et al\. \([2021](https://arxiv.org/html/2607.01498#bib.bib14)\)defines behavioral distances between policies in trajectory space for zero\-shot coordination\. CURLLaskin et al\. \([2020](https://arxiv.org/html/2607.01498#bib.bib11)\)applies contrastive learning to state observations for sample\-efficient RL, though it learns state representations for a single agent rather than*policy*representations across agents\. Our trajectory encoder adapts contrastive learning to the latter setting, learning embeddings that are invariant to stochastic noise but discriminative of strategic intent\.

### L\.1Grover

Details onGrover et al\. \([2018](https://arxiv.org/html/2607.01498#bib.bib5)\):

The hybrid loss is defined as:

ℒGrover=ℒimit\+λ​ℒid\\displaystyle\{\\mathcal\{L\}\}\_\{\\text\{Grover\}\}=\{\\mathcal\{L\}\}\_\{\\text\{imit\}\}\+\\lambda\\,\{\\mathcal\{L\}\}\_\{\\text\{id\}\}ℒid=\(1\+exp⁡\(‖re−ne‖2−‖re−pe‖2\)\)−2,\\displaystyle\\quad\{\\mathcal\{L\}\}\_\{\\text\{id\}\}=\\big\(1\+\\exp\(\\\|r\_\{e\}\-n\_\{e\}\\\|\_\{2\}\-\\\|r\_\{e\}\-p\_\{e\}\\\|\_\{2\}\)\\big\)^\{\-2\},wherepep\_\{e\},rer\_\{e\},nen\_\{e\}are embeddings of a positive, reference, and negative episode respectively\.

### L\.2Additional Related Works

#### Chess style learning

Other relevant research directions include learning individual players’ styles in chess\. This was done byMcIlroy\-Young et al\. \([2022](https://arxiv.org/html/2607.01498#bib.bib16)\)\. Their approach fine\-tunes policy networks to represent styles of different players\. The methods in this paper could be used to represent styles of different players instead as low\-dimensional embeddings\.

#### Weight\-Space Learning

Several works have explored learning from or about neural network weights directly\.Unterthiner et al\. \([2020](https://arxiv.org/html/2607.01498#bib.bib22)\)showed that network properties can be predicted from weight statistics alone, while hypernetworksHa et al\. \([2016](https://arxiv.org/html/2607.01498#bib.bib6)\)generate weights for target networks\. Our weight and functional autoencoders draw on this line of work, but our results suggest that weight\-space proximity is a poor proxy for behavioral similarity in games\.

## Appendix MDetails on Downstream Tasks

### M\.1Task A

A linear predictor receives a single agent embedding𝐳1\\mathbf\{z\}\_\{1\}and predicts its expected payoff against a static, uniform random opponentπrand\\pi\_\{\\text\{rand\}\}\. This tests whether the embeddings correlate with general strategic competence\.

We want the linear prediction \(left side of the equation\) to approximate the expected returns \(right side of the equation\):

y^A=𝐰A⊤​𝐳1\+bA≈𝔼π1,πrand​\[∑t=0tmaxℛ​\(𝐬t,𝐚t\)\]\.\\hat\{y\}\_\{A\}=\\mathbf\{w\}\_\{A\}^\{\\top\}\\mathbf\{z\}\_\{1\}\+b\_\{A\}\\approx\{\\mathbb\{E\}\}\_\{\\pi\_\{1\},\\pi\_\{\\text\{rand\}\}\}\\bigg\[\\sum\_\{t=0\}^\{t\_\{\\text\{max\}\}\}\{\\mathcal\{R\}\}\(\\mathbf\{s\}^\{t\},\\mathbf\{a\}^\{t\}\)\\bigg\]\.

### M\.2Task B

The linear probe receives the concatenated embeddings of two agents and predicts the payoff of a head\-to\-head match:

y^B=𝐰B⊤​\[𝐳1;𝐳2\]\+bB≈𝔼π1,π2​\[∑t=0tmaxℛ​\(𝐬t,𝐚t\)\]\.\\hat\{y\}\_\{B\}=\\mathbf\{w\}\_\{B\}^\{\\top\}\[\\mathbf\{z\}\_\{1\};\\mathbf\{z\}\_\{2\}\]\+b\_\{B\}\\approx\{\\mathbb\{E\}\}\_\{\\pi\_\{1\},\\pi\_\{2\}\}\\bigg\[\\sum\_\{t=0\}^\{t\_\{\\text\{max\}\}\}\{\\mathcal\{R\}\}\(\\mathbf\{s\}^\{t\},\\mathbf\{a\}^\{t\}\)\\bigg\]\.This tests whether the latent space captures relational information about how specific strategies interact, not just their individual quality\.

### M\.3Task C

For a given Player 1 policyπ1\\pi\_\{1\}, the best\-response value is defined asBRV​\(π1\)=maxπ2⁡𝔼π1,π2​\[−∑tℛ​\(𝐬t,𝐚t\)\]\\text\{BRV\}\(\\pi\_\{1\}\)=\\max\_\{\\pi\_\{2\}\}\{\\mathbb\{E\}\}\_\{\\pi\_\{1\},\\pi\_\{2\}\}\[\-\\sum\_\{t\}\{\\mathcal\{R\}\}\(\\mathbf\{s\}^\{t\},\\mathbf\{a\}^\{t\}\)\]\. A linear probe predicts this from the embedding:

y^C=𝐰C⊤​𝐳1\+bC≈BRV​\(π1\)\.\\hat\{y\}\_\{C\}=\\mathbf\{w\}\_\{C\}^\{\\top\}\\mathbf\{z\}\_\{1\}\+b\_\{C\}\\approx\\text\{BRV\}\(\\pi\_\{1\}\)\.