通过偏好优化教会AI智能体进行策略性扑克博弈

# Teaching AI Agent to Play Poker Strategically Through Preference Optimization
Source: [https://arxiv.org/html/2610.27028](https://arxiv.org/html/2610.27028)
## Teaching AI Agent to Play Poker Strategically Through Preference OptimizationThanks:Accepted at the iSAI\-NLP 2026 Student Research Workshop \(SRW\)\.

Phurinat PolasaThanapon ThanadunpremdetAffiliation:Department of Computer Engineering Mahidol University Thailand thanapon\.tha@student\.mahidol\.eduAffiliation:Phatthanasak KraiduangAffiliation:Department of Computer Engineering Mahidol University Thailand phatthanasak\.kra@student\.mahidol\.ac\.thSaksorn RuangtanusakAffiliation:R&D SCB DataX, SCBX Group Thailand saksorn\.ruangtanusak@data\-x\.ai

###### Abstract

A poker agent can make a legal move and still make a strategically poor one\. We ask whether compact AI agents can exhibit strategic behavior under hidden information and whether preference optimization can improve beyond static and zero\-shot policies\. We first evaluate a static baseline and three frozen language models: SmallThinker, Qwen3, and Gemma\-3 whose performance varies substantially between exact\-game and six\-player evaluation\. We therefore propose Strategic Preference Optimization with KL\-anchored Equilibrium \(SPOKE\), combining teacher\-relative value penalties, policy anchoring, risk constraints, and structured rationale supervision, and compare it with SFT, DPO, GRPO, and GSPO\. Across three runs of a SmallThinker\-4B\-A0\.6B policy, SPOKE achieves the lowest mean exploitability in Kuhn \(0\.213\) and Leduc \(1\.081\) and the highest six\-max top\-1 agreement \(0\.775\), while maintaining teacher KL comparable to SFT \(0\.522 vs\. 0\.523\)\. These results provide preliminary evidence that strategic preference optimization can improve strategic quality while preserving teacher\-consistent behavior\.

###### Index Terms:

AI agents, Large Language Models, preference optimization

## IIntroduction

Legal poker actions can still be strategically poor, making poker a useful testbed for strategic decision\-making under hidden information\. CFR provides approximate equilibrium strategies in finite two\-player zero\-sum games\[[1](https://arxiv.org/html/2610.27028#bib.bib1)\], while systems such as Pluribus\[[2](https://arxiv.org/html/2610.27028#bib.bib2)\]motivate evaluation in more complex multiplayer settings\.

We first compare a static policy with three frozen compact language models: SmallThinker, Qwen3, and Gemma\-3\. SmallThinker performs best among them in exact games, yet remains weak on six\-max agreement\.

We therefore compare Supervised Fine\-Tuning \(SFT\), Direct Preference Optimization \(DPO\)\[[4](https://arxiv.org/html/2610.27028#bib.bib4)\], Group Relative Policy Optimization \(GRPO\)\[[5](https://arxiv.org/html/2610.27028#bib.bib5)\], and Group Sequence Policy Optimization \(GSPO\)\[[6](https://arxiv.org/html/2610.27028#bib.bib6)\], and propose Strategic Preference Optimization with KL\-anchored Equilibrium \(SPOKE\)\. SPOKE combines teacher\-relative value penalties, KL anchoring, risk constraints, and structured rationale supervision to improve strategic quality without sacrificing teacher\-consistent behavior\.

![Refer to caption](https://arxiv.org/html/2610.27028v1/figures/overviews.png)Fig\. 1:SPOKE overview: teaching a poker agent to act legally and strategically\.
## IIProposed Method

Each record contains the player’s information statess, legal action–size menuA⁡\(s\)A\(s\), teacher distributionπT\\pi\_\{T\}, optional action valuesQTQ\_\{T\}, risk features, and rationale tagstt\. Prompts exclude opponents’ hidden cards\. Canonical action strings receive model likelihood scores, normalized overA⁡\(s\)A\(s\)to formπθ\\pi\_\{\\theta\}; a separate output predicts structured tags \(Fig\.[2](https://arxiv.org/html/2610.27028#S2.F2)\)\.

We operationalize the objective as follows\. Let\[x\]\+=max⁡\(x,0\)\[x\]\_\{\+\}=\\max\(x,0\)andVT=𝔼a∼πT​QT​\(s,a\)V\_\{T\}=\\mathbb\{E\}\_\{a\\sim\\pi\_\{T\}\}Q\_\{T\}\(s,a\)\. Define

Rs\+\\displaystyle R\_\{s\}^\{\+\}=𝔼a∼πθ​\[VT−QT​\(s,a\)\]\+,\\displaystyle=\\mathbb\{E\}\_\{a\\sim\\pi\_\{\\theta\}\}\[V\_\{T\}\-Q\_\{T\}\(s,a\)\]\_\{\+\},Δs\+\\displaystyle\\Delta\_\{s\}^\{\+\}=\[VT−𝔼a∼πθ​QT​\(s,a\)\]\+\.\\displaystyle=\[V\_\{T\}\-\\mathbb\{E\}\_\{a\\sim\\pi\_\{\\theta\}\}Q\_\{T\}\(s,a\)\]\_\{\+\}\.\(1\)The first penalty discourages individual below\-reference actions; the second penalizes an average value shortfall\. These are teacher\-relative proxies, not cumulative CFR regret\. Withmsm\_\{s\}indicating validated value targets, we optimize the policy using

ℒ=𝔼s\[\\displaystyle\\mathcal\{L\}=\\mathbb\{E\}\_\{s\}\\big\[ms\(λrRs\+\+λvΔs\+\)\+λkDKL\(πT∥πθ\)\\displaystyle m\_\{s\}\(\\lambda\_\{r\}R\_\{s\}^\{\+\}\+\\lambda\_\{v\}\\Delta\_\{s\}^\{\+\}\)\+\\lambda\_\{k\}D\_\{\\mathrm\{KL\}\}\(\\pi\_\{T\}\\\|\\pi\_\{\\theta\}\)\+λb\[𝔼a∼πθc\(s,a\)−Bs\]\+\+λtℓtag\]\.\\displaystyle\+\\lambda\_\{b\}\[\\mathbb\{E\}\_\{a\\sim\\pi\_\{\\theta\}\}c\(s,a\)\-B\_\{s\}\]\_\{\+\}\+\\lambda\_\{t\}\\ell\_\{\\mathrm\{tag\}\}\\big\]\.\(2\)
Hereccis a risk cost,BsB\_\{s\}its budget, andℓtag\\ell\_\{\\mathrm\{tag\}\}is binary cross\-entropy over rationale tags\. KL anchoring discourages collapse of the teacher’s mixed policy\. We use value\-based terms only where validated \(ms=1m\_\{s\}=1for Kuhn/Leduc andms=0m\_\{s\}=0for six\-max\), with hyperparameters selected on the development split\.

Kuhn / limit LeducCFR:πT,QT,VT\\pi\_\{T\},\\,Q\_\{T\},\\,V\_\{T\}Six\-max preflopSmoothed behavior labelsDecision records \+ grouped splitsLegal menu, risk features, rationale tagsSPOKE trainingValues \+ KL \+ risk \+ tagsLegal\-action filteringRangeDiald∈\[−1,1\]d\\in\[\-1,1\]Separate evaluation \+ audit trailExact\-game exploitability / six\-max behaviorpolicy

Fig\. 2:Proposed SPOKE pipeline\. Teacher values are used only where validated; six\-max action labels provide behavioral supervision\. Legal\-action filtering is shared by all methods\.After restricting the policy to legal actions, RangeDial applies a bounded preference shift

πθ,d​\(a∣s\)∝πθ​\(a∣s\)​exp⁡\(α​d​b​\(a\)\),d∈\[−1,1\],\\pi\_\{\\theta,d\}\(a\\mid s\)\\propto\\pi\_\{\\theta\}\(a\\mid s\)\\exp\(\\alpha d\\,b\(a\)\),\\quad d\\in\[\-1,1\],\(3\)wherebbis11for raise,00for check/call, and−1\-1for fold;α≥0\\alpha\\geq 0sets the strength\. Increasingddfavors raising without adding illegal actions\.

## IIIExperimental Setups

Data and evaluation\.The record set holds 5,183 decisions after merging 670 repeated six\-max spots \(4,883 spots from 5,553 observations\), with teacher exploitability 0\.001486 in Kuhn \(10,000 CFR iterations\) and 0\.010877 in limit Leduc \(2,000 iterations\)\. Splits precede prompt augmentation and group information sets and public histories to prevent leakage: train/dev/test are 6/3/3 \(Kuhn\), 189/45/54 \(Leduc\) and 3,449/705/729 \(six\-max\)\. Full chance enumeration and best responses support exact profile value, NashConv, and exploitability on the declared small games \(over all information sets\)\.

Six\-max evaluation uses 100\-big\-blind preflop decisions with label\-smoothed, Pluribus\-derived behavioral actions\. The primary behavioral endpoint is held\-out top\-1 agreement; teacher KL is secondary\. Legality is checked separately\.

Models and controls\.The static baseline samples uniformly from legal actions in Kuhn and Leduc and uses a fixed preflop action chart in six\-max\. Frozen SmallThinker\-4B\-A0\.6B\[[3](https://arxiv.org/html/2610.27028#bib.bib3)\], Qwen3\-0\.6B\[[7](https://arxiv.org/html/2610.27028#bib.bib7)\], and Gemma\-3\-1B\-IT\[[8](https://arxiv.org/html/2610.27028#bib.bib8)\]were each evaluated once on the same held\-out sets, task prompt, and legal menu\. Action probabilities use first\-token scores for*fold*/*call*/*raise*; these serve as zero\-shot controls\.

We then train SFT, DPO\[[4](https://arxiv.org/html/2610.27028#bib.bib4)\], GRPO\[[5](https://arxiv.org/html/2610.27028#bib.bib5)\], GSPO\[[6](https://arxiv.org/html/2610.27028#bib.bib6)\], and SPOKE as a SmallThinker\-4B\-A0\.6B policy over 4,096 hashed prompt features, where DPO, GRPO, and GSPO are single\-step versions of those objectives, and evaluate them under the same protocol with three runs \(seeds 13/42/73\)\.

Sensitivity analyses\.KL\-weight and RangeDial sweeps use the same three runs\. SFT receives identical tag supervision\. Component\-removal ablations have no measured results, so the contribution of each SPOKE term cannot be isolated\.

## IVPreliminary Diagnostics and Limitations

TABLE I:Zero\-shot controls vs\. optimization methods \(SmallThinker\-4B\-A0\.6B\)\. Three\-run means; frozen controls are single runs\. Bold: best per metric\.Table[I](https://arxiv.org/html/2610.27028#S4.T1)shows that zero\-shot capability does not transfer uniformly across evaluation settings\. SmallThinker improves over the static baseline in Kuhn \(0\.326 vs\. 0\.458\) and Leduc \(1\.767 vs\. 2\.374\), yet reaches only 0\.159 six\-max top\-1 agreement\. Qwen3 and Gemma\-3 exhibit different failure modes, indicating that no frozen model performs consistently well across both strategic and behavioral endpoints\.

The optimization objectives reveal a similar trade\-off\. SFT achieves strong six\-max agreement \(0\.754\) and low teacher KL \(0\.523\), while GRPO and GSPO improve Leduc exploitability to 1\.239 and 1\.331 but increase KL to 2\.357 and 2\.354\. DPO performs poorly under this protocol\.

SPOKE provides the strongest combined result, best on all four metrics\. Variation across three runs remains small, although the sample size is insufficient to establish statistical superiority\.

Sensitivity analysis shows that KL weighting controls the trade\-off between behavioral agreement and teacher divergence while Leduc exploitability remains relatively stable\. RangeDial further changes aggression without introducing illegal actions\.

## VConclusion

Frozen compact language models demonstrated inconsistent performance in six\-max evaluations and exact\-game evaluations\. We proposed SPOKE, integrating teacher\-relative value penalties, KL anchoring, risk constraints, and structured rationale supervision\. Across three runs of a SmallThinker\-4B\-A0\.6B policy, SPOKE achieved the lowest mean exploitability in Kuhn \(0\.213\) and Leduc \(1\.081\), and the highest six\-max top\-1 agreement \(0\.775\), while maintaining teacher KL comparable to SFT\. These findings provide preliminary evidence that strategic performance and teacher\-consistent behavior can be jointly improved\. However, the limited number of runs, the absence of component ablations, and the evaluation of behavioral\-only at six\-maximums warrant further validation\.

## VIAcknowledgments

This research has received funding support from the NSRF via the Research and Innovation Acceleration Agency for Competitiveness and Area Development \(RCAD\) \(Program Management Unit for Frontier Brainpower and Future Industries\) \[grant number B13T690002\-1\]\.

## References

- \[1\]M\. Zinkevich, M\. Johanson, M\. Bowling, and C\. Piccione, “Regret minimization in games with incomplete information,” in*NeurIPS*, 2007\.
- \[2\]N\. Brown and T\. Sandholm, “Superhuman AI for multiplayer poker,”*Science*, vol\. 365, no\. 6456, pp\. 885–890, 2019\.
- \[3\]Y\. Song*et al\.*, “SmallThinker: A family of efficient large language models natively trained for local deployment,”[arXiv:2507\.20984](https://arxiv.org/abs/2507.20984), 2025\.
- \[4\]R\. Rafailov*et al\.*, “Direct preference optimization: Your language model is secretly a reward model,” in*NeurIPS*, 2023\.
- \[5\]Z\. Shao*et al\.*, “DeepSeekMath: Pushing the limits of mathematical reasoning in open language models,”[arXiv:2402\.03300](https://arxiv.org/abs/2402.03300), 2024\.
- \[6\]C\. Zheng*et al\.*, “Group sequence policy optimization,”[arXiv:2507\.18071](https://arxiv.org/abs/2507.18071), 2025\.
- \[7\]Qwen Team, “[Qwen3\-0\.6B](https://huggingface.co/Qwen/Qwen3-0.6B)model card,” Hugging Face, 2026\.
- \[8\]Google, “[Gemma\-3\-1B\-IT](https://huggingface.co/google/gemma-3-1b-it)model card,” Hugging Face, 2026\.