Breaking the Impasse: Dual-Scale Evolutionary Policy Training for Social Language Agents

# Dual-Scale Evolutionary Policy Training for Social Language Agents
Source: [https://arxiv.org/html/2605.08721](https://arxiv.org/html/2605.08721)
Minzheng Wang1,2, Run Luo2, Yanbo Wang1,2, Zichen Liu3 Yuqiao Tan1,2,Tao Tan4,Xu Nan1,Yinhe Zheng4,Wenji Mao1,2 1MAIS, Institute of Automation, Chinese Academy of Sciences 2School of Artificial Intelligence, University of Chinese Academy of Sciences 3National University of Singapore4Ritzz\-AI 🖂:\{wangminzheng2023, wenji\.mao\}@ia\.ac\.cn

###### Abstract

While Reinforcement Learning with Verifiable Rewards \(RLVR\) has proven effective for closed\-ended tasks, extending it to open\-ended social language games via self\-play reveals a critical issue:evolution impasse\. Due to the vast strategy space, language agents frequently converge to homogenized behaviors, leading to deterministic match outcomes that eliminate the gradient signals necessary for policy evolution\. To tackle this issue, we proposeDual\-scale Evolutionary Policy Training \(DEPT\)for social language games\. DEPT introduces atime\-scaled evolutionary perceptionmechanism that detects impasse by quantifying dual\-scale value baseline divergence alongside match entropy\. Upon perceiving the collapse, it then activatesasymmetric advantage reshapingto dynamically modulate the optimization landscape for intervention\. Thus, our method effectively restores gradient signals and enforces sustained strategic exploration\. Extensive experiments on multiple social language games demonstrate that DEPT outperforms strong baselines, avoiding policy degeneration and driving the continuous evolution of social language agents\.

Breaking the Impasse: Dual\-Scale Evolutionary Policy Training for Social Language Agents

Minzheng Wang1,2, Run Luo2, Yanbo Wang1,2, Zichen Liu3Yuqiao Tan1,2,Tao Tan4,Xu Nan1,Yinhe Zheng4,Wenji Mao1,2††thanks:Corresponding authors\.1MAIS, Institute of Automation, Chinese Academy of Sciences2School of Artificial Intelligence, University of Chinese Academy of Sciences3National University of Singapore4Ritzz\-AI🖂:\{wangminzheng2023, wenji\.mao\}@ia\.ac\.cn

## 1Introduction

Reinforcement Learning with Verifiable Rewards \(RLVR\) has emerged as a pivotal paradigm for unlocking the reasoning potential of large language models \(LLMs\), demonstrating remarkable capabilities in static problem\-solving tasks such as mathematics and code generation\(Guoet al\.,[2025](https://arxiv.org/html/2605.08721#bib.bib1); Teamet al\.,[2025](https://arxiv.org/html/2605.08721#bib.bib2); Yueet al\.,[2025](https://arxiv.org/html/2605.08721#bib.bib3); Liuet al\.,[2025b](https://arxiv.org/html/2605.08721#bib.bib13)\)\. However, these tasks are inherently closed\-ended problem\-solving, characterized by deterministic correct answers with straightforward question answering\. In reality, social decision\-making is considerably more complex, often situated in open\-ended contexts fraught with conflicting interests and dynamic interactions\(FAIRet al\.,[2022](https://arxiv.org/html/2605.08721#bib.bib19); Parket al\.,[2023](https://arxiv.org/html/2605.08721#bib.bib20); Yanget al\.,[2025b](https://arxiv.org/html/2605.08721#bib.bib7)\)\. Under such dynamic social contexts, LLMs not only engage in coherent logical reasoning but also perform long\-horizon planning, infer opponents’ intentions in real\-time, and process complex social signals such as negotiation\(Bianchiet al\.,[2024](https://arxiv.org/html/2605.08721#bib.bib14); Huaet al\.,[2024](https://arxiv.org/html/2605.08721#bib.bib6)\), deception\(Hagendorff,[2023](https://arxiv.org/html/2605.08721#bib.bib17); Xuet al\.,[2024](https://arxiv.org/html/2605.08721#bib.bib18)\), and bargaining\(Xiaet al\.,[2024](https://arxiv.org/html/2605.08721#bib.bib15); Abdulhaiet al\.,[2025](https://arxiv.org/html/2605.08721#bib.bib16)\)\. Thus, for language agents, a significant gap exists between closed\-ended problem\-solving capabilities and open\-ended social intelligence\(Zhouet al\.,[2024](https://arxiv.org/html/2605.08721#bib.bib4); Wanget al\.,[2025a](https://arxiv.org/html/2605.08721#bib.bib8)\)\.

![Refer to caption](https://arxiv.org/html/2605.08721v1/x1.png)

Figure 1:\(a\) DEPT maintains high outcome entropy while baselines collapse into deterministic distributions\. \(b\) DEPT sustains effective gradient signals while baselines suffer from gradient vanishing\. \(c, d\) DEPT achieves superior performance, effectively addressing policy degeneration in open\-ended social language games\.Social strategic games provide an ideal testbed to practise this kind of social capability in open\-ended tasks: they naturally simulate dynamic social contexts and complex decision\-making behaviors, while game outcomes \(i\.e\., win/loss/draw\) are objectively determined by rules—perfectly aligning with the RLVR paradigm without requiring expensive human annotations\(Duanet al\.,[2024](https://arxiv.org/html/2605.08721#bib.bib23); Huaet al\.,[2024](https://arxiv.org/html/2605.08721#bib.bib6); Guertleret al\.,[2025](https://arxiv.org/html/2605.08721#bib.bib5)\)\. More importantly, this property makes self\-play an effective pathway for scaling LLM capabilities\(Silveret al\.,[2018](https://arxiv.org/html/2605.08721#bib.bib21); Bansalet al\.,[2018](https://arxiv.org/html/2605.08721#bib.bib27); Chenet al\.,[2024](https://arxiv.org/html/2605.08721#bib.bib22); Liuet al\.,[2025a](https://arxiv.org/html/2605.08721#bib.bib9)\)\. Using a single LLM acting as both players of the game and continuously optimizing through zero\-sum competition, self\-play can overcome the data scarcity bottleneck in post\-training, and meanwhile establish an intrinsic automatic curriculum through opponent co\-evolution of social decision\-making skills\.

Existing self\-play RLVR methods\(Liuet al\.,[2025a](https://arxiv.org/html/2605.08721#bib.bib9); Yuanet al\.,[2025](https://arxiv.org/html/2605.08721#bib.bib10)\)have achieved remarkable success in games with limited action space \(e\.g\., Tic\-Tac\-Toe, Kuhn Poker\), by designing role\-specific advantage estimation\. However, these methods struggle with open\-ended social games involving free\-form natural language interactions, which constitutevast action spacethat often requires verbal influence on opponents in order to succeed\. Without effective guidance in such unbounded strategy space, agents easily become trapped in suboptimal solutions\. This renders the training process highly liable toevolution impasse, that is, the stagnant situation a cognitive agent encounters in its decision\-making\(Laird,[2019](https://arxiv.org/html/2605.08721#bib.bib46); Sumerset al\.,[2023](https://arxiv.org/html/2605.08721#bib.bib47)\), which manifests as deterministic match outcome distributions for self\-play RLVR methods\. In this situation, the value baseline converges to the static expected return that causes the advantage function to approach zero, leading to the vanishing of policy gradient and locking the agent in a suboptimal state \(as shown in[Figure˜1](https://arxiv.org/html/2605.08721#S1.F1)\(a\)\(b\)\)\. To break the impasse, it is critical to develop an effective mechanism for evolutionary perception and intervention during self\-play\.

To address the above issues, we propose aDual\-scale Evolutionary Policy Training \(DEPT\)method to dynamically perceive evolutionary states and adaptively modulate the optimization landscape\. Specifically, we introduce atime\-scaled evolutionary perceptionmechanism that maintains parallel value baselines: a fast baseline sensitive to recent fluctuations and a slow baseline anchored to long\-term trends\. By measuring the divergence between these baselines alongside match outcome entropy, we derive a quantifiable metric for evolution impasse\. To restore effective gradients, we proposeasymmetric advantage reshaping\. By anchoring on the stable slow baseline and adaptively fusing historical value bounds from the fast baseline, this mechanism constructs a contrastive advantage signal\. It effectively penalizes dominant outcomes while amplifying rare trajectories, thereby enforcing exploration\. Our approach effectively intervenes in evolution impasse and restores the gradient landscape, driving continuous strategic evolution\. Experimental results across multiple social language games demonstrate that DEPT outperforms strong baselines\.

Our contributions are summarized as follows: \(1\) We propose DEPT for social language games, which is the first to dynamically perceive evolution impasse and intervene in self\-play RLVR\. \(2\) Our method develops the time\-scaled evolutionary perception and asymmetric advantage reshaping mechanism, which leverages dual value baselines to quantify evolutionary states and restore effective gradient signals\. \(3\) Extensive experiments demonstrate that DEPT outperforms existing SOTA methods, effectively preventing policy degeneration while driving continuous co\-evolution\.

## 2Preliminary

### 2\.1Task Formulation

The social language game task is modeled as a two\-player zero\-sum Markov game\(Littman,[1994](https://arxiv.org/html/2605.08721#bib.bib26)\)𝒢=\(𝒮,𝒜0,𝒜1,𝒯,ℛ\)\\mathcal\{G\}=\(\\mathcal\{S\},\\mathcal\{A\}\_\{0\},\\mathcal\{A\}\_\{1\},\\mathcal\{T\},\\mathcal\{R\}\)\. The state space𝒮\\mathcal\{S\}represents the interaction context, and𝒜0\\mathcal\{A\}\_\{0\},𝒜1\\mathcal\{A\}\_\{1\}denote the action spaces for each player, consisting of open\-ended natural language utterances\. At statests\_\{t\}, the active playerp=tmod2p=t\\bmod 2receives a partial observationoto\_\{t\}containing the interaction history and its agent\-specific private information, then samples an utteranceat\(p\)a\_\{t\}^\{\(p\)\}from its policyπp\(⋅∣ot\)\\pi\_\{p\}\(\\cdot\\mid o\_\{t\}\)\. The environment transitions tost\+1s\_\{t\+1\}via𝒯\\mathcal\{T\}\. An episode forms a trajectoryτ=\{\(st,at\(p\)\)\}t=0T\\tau=\\\{\(s\_\{t\},a\_\{t\}^\{\(p\)\}\)\\\}\_\{t=0\}^\{T\}, concluding at terminal timestepTTwhere each agent receives a rewardRp​\(τ\)R\_\{p\}\(\\tau\)based on its private goals\. The zero\-sum property requiresR0​\(τ\)\+R1​\(τ\)=0R\_\{0\}\(\\tau\)\+R\_\{1\}\(\\tau\)=0, creating direct competition between agents\. Typically, rewards are assigned as\+1\+1for win,−1\-1for lose, and0for a draw\. Each agent aims to learn a policyπp\\pi\_\{p\}maximizing its expected return:𝔼τ∼π0,π1​\[Rp​\(τ\)\]\\mathbb\{E\}\_\{\\tau\\sim\\pi\_\{0\},\\pi\_\{1\}\}\[R\_\{p\}\(\\tau\)\]\.

### 2\.2Self\-Play RLVR

Self\-play trains a single policy LLMπθ\\pi\_\{\\theta\}to act as both players, with role conditioning achieved through system prompts specifying whether the model plays asp0p\_\{0\}orp1p\_\{1\}\. At each turn, the active player generatesyt\(p\)∼πθ\(⋅∣ot,p\)y\_\{t\}^\{\(p\)\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid o\_\{t\},p\), whereyt\(p\)=⟨think⟩ct⟨/think⟩⟨act⟩at\(p\)⟨/act⟩y\_\{t\}^\{\(p\)\}=\\langle\\texttt\{think\}\\rangle c\_\{t\}\\langle/\\texttt\{think\}\\rangle\\langle\\texttt\{act\}\\rangle a\_\{t\}^\{\(p\)\}\\langle/\\texttt\{act\}\\ranglefollows a reasoning\-then\-acting paradigm\(Yaoet al\.,[2023](https://arxiv.org/html/2605.08721#bib.bib25)\), which is strictly enforced via a format reward to guarantee validity, wherectc\_\{t\}denotes the reasoning process andat\(p\)a\_\{t\}^\{\(p\)\}is the executable action\. This shared\-parameter approach creates an automatic curriculum: as the policy improves, it simultaneously faces a stronger opponent\.

##### Role\-conditioned Advantage Estimation\.

The vanilla policy gradient suffers from high variance, particularly problematic in self\-play where the opponent continuously evolves\. Moreover, many games exhibit inherent role asymmetries \(e\.g\., first\-mover advantages\), causing different expected returns for each role even under optimal play\. A global baseline poorly estimates both roles’ expectations, leading to biased gradients\. To address this, Role\-conditioned Advantage Estimation \(RAE\)\(Liuet al\.,[2025a](https://arxiv.org/html/2605.08721#bib.bib9); Yuanet al\.,[2025](https://arxiv.org/html/2605.08721#bib.bib10)\)maintains separate baselinesbpb\_\{p\}for each role, estimating the role\-specific expected return𝔼​\[Rp​\(τ\)\]\\mathbb\{E\}\[R\_\{p\}\(\\tau\)\]\. The advantage is computed asAp​\(τ\)=Rp​\(τ\)−bpA\_\{p\}\(\\tau\)=R\_\{p\}\(\\tau\)\-b\_\{p\}, yielding the variance\-reduced policy gradient:

∇θJ\(θ\)=𝔼τ∼πθ\[∑p∈\{0,1\}∑t∈TpAp\(τ\)\\displaystyle\\nabla\_\{\\theta\}J\(\\theta\)=\\mathbb\{E\}\_\{\\tau\\sim\\pi\_\{\\theta\}\}\\Big\[\\sum\_\{p\\in\\\{0,1\\\}\}\\sum\_\{t\\in T\_\{p\}\}A\_\{p\}\(\\tau\)\(1\)⋅∇θlogπθ\(yt\(p\)∣ot,p\)\]\\displaystyle\\cdot\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{t\}^\{\(p\)\}\\mid o\_\{t\},p\)\\Big\]whereTpT\_\{p\}denotes turns where playerppacted\. By centering returns around role\-specific expectations, RAE ensures gradient updates reflect genuine strategic improvements rather than inherent positional advantages\.

## 3Empirical Investigation of the Impasse

Self\-play RLVR aims to foster continuous improvement through iterative competition\. In principle, this paradigm relies on dynamic evolutionary interactions to maintain the diversity of match outcomes, ensuring the variance required for effective policy optimization\. However, in open\-ended social games, we observe a critical failure mode where agents demonstrate increasing mastery of game mechanics yet stagnate in strategic intelligence\. As illustrated in[Figure˜2](https://arxiv.org/html/2605.08721#S3.F2), training on Qwen3\-4B\-Base yields improvements in basic metrics such as game length and reward\. However, the win rate against a fixed Gemini\-2 opponent exhibits a declining trend, ultimately converging to a sub\-optimal state and failing to demonstrate sustained performance gains\. To diagnose the internal dynamics driving this impasse, we introduceMatch Entropy\(Hmatch\(t\)H\_\{\\text\{match\}\}^\{\(t\)\}\), which quantifies the diversity of match outcomes at training stepttover the distributionP\(t\)=\{pwin,pdraw,ploss\}P^\{\(t\)\}=\\\{p\_\{\\text\{win\}\},p\_\{\\text\{draw\}\},p\_\{\\text\{loss\}\}\\\}:

Hmatch\(t\)=−1log⁡\(\|P\|\)​∑o∈Ppo​log⁡\(po\)H\_\{\\text\{match\}\}^\{\(t\)\}=\-\\frac\{1\}\{\\log\(\|P\|\)\}\\sum\_\{o\\in P\}p\_\{o\}\\log\(p\_\{o\}\)\(2\)
![Refer to caption](https://arxiv.org/html/2605.08721v1/x2.png)

Figure 2:Training dynamics of the self\-play RLVR in social language game \(Negotiation\)\. While \(a\) training Reward and \(b\) average Game Length exhibit normal growth patterns, indicating the acquisition of basic game mechanics, \(c\) for baselines, the Win Rate against a fixed Gemini\-2\.0 opponent suffers a significant collapse\.[Figure˜1](https://arxiv.org/html/2605.08721#S1.F1)\(a\) reveals a distinct collapse pattern: after a brief initial exploration phase,Hmatch\(t\)H\_\{\\text\{match\}\}^\{\(t\)\}rapidly diminishes and remains near zero\. This signals convergence to aevolution impasse, a stagnant state where agents become trapped in suboptimal solutions due to the vast, unstructured strategy space of social language games that makes it extremely difficult to sufficiently explore effective strategies \(e\.g\., nuanced persuasion or deception\)\.

Mathematically, this degeneration directly precipitates the vanishing of gradient signals\. In standard policy gradient methods, the update relies on the advantage functionAp​\(τ\)=Rp​\(τ\)−bpA\_\{p\}\(\\tau\)=R\_\{p\}\(\\tau\)\-b\_\{p\}, where the value baselinebpb\_\{p\}estimates the expected return𝔼​\[Rp\]\\mathbb\{E\}\[R\_\{p\}\]\. When the match outcome distribution becomes deterministic \(i\.e\.,Hmatch\(t\)→0H\_\{\\text\{match\}\}^\{\(t\)\}\\to 0\), the value baseline converges to constant return \(bp≈Rtb\_\{p\}\\approx R\_\{t\}\), causing the advantage to vanish:At→0A\_\{t\}\\to 0\. This diminishes gradients, preventing escape from local optima\. To unlock the reasoning potential of LLMs in social language games, it is paramount to establish a mechanism that canperceivethis collapse andinterveneto restore the gradient landscape\.

![Refer to caption](https://arxiv.org/html/2605.08721v1/x3.png)

Figure 3:Overview of DEPT\. The framework comprises two components: \(1\)Time\-Scaled Evolutionary Perception, which maintains fast and slow EMA baselines, combined with Match Entropy to derive the Intervention Coefficientλ\(t\)\\lambda^\{\(t\)\}quantifying evolutionary impasse; and \(2\)Asymmetric Advantage Reshaping, which leverages historical value bounds to suppress dominant outcomes and amplify rare trajectories, restoring gradient signals for continuous policy evolution
## 4Dual\-scale Evolutionary Policy Training

To overcome the evolutionary impasse, we propose Dual\-scale Evolutionary Policy Training \(DEPT\), an algorithm designed to dynamically perceive evolutionary states and adaptively modulate optimization landscapes to enforce continuous evolution\. As shown in[Figure˜3](https://arxiv.org/html/2605.08721#S3.F3), our algorithm leverages a dual\-timescale value baseline to quantify the stationarity of the training process and dynamically injects intrinsic motivation signals into the advantage estimation to enforce continuous evolution\.

### 4\.1Time\-Scaled Evolutionary Perception

Standard value baselines are inherently reactive: they simply track the current expected return, rendering them incapable of distinguishing between astable optimal policyand aevolution impasse\. In both cases, the baseline stabilizes, failing to signal that the learning process has stalled\. To distinguish these states, we introduce a dual\-timescale mechanism that captures thedynamicsof the training process by contrasting short\-term and long\-term value estimates\. For each player rolep∈\{0,1\}p\\in\\\{0,1\\\}, we maintain two parallel value baselines,bpfastb\_\{p\}^\{\\text\{fast\}\}andbpslowb\_\{p\}^\{\\text\{slow\}\}, updated via Exponential Moving Average \(EMA\) with distinct decay ratesαfast<αslow\\alpha\_\{\\text\{fast\}\}<\\alpha\_\{\\text\{slow\}\}:

bpk,t=αk⋅bpk,t−1\+\(1−αk\)⋅Rp​\(τ\),k∈\{fast,slow\}b\_\{p\}^\{k,t\}=\\alpha\_\{k\}\\cdot b\_\{p\}^\{k,t\-1\}\+\(1\-\\alpha\_\{k\}\)\\cdot R\_\{p\}\(\\tau\),k\\in\\\{\\text\{fast\},\\text\{slow\}\\\}\(3\)Here,bpfastb\_\{p\}^\{\\text\{fast\}\}rapidly responds to recent outcome returns, whilebpslowb\_\{p\}^\{\\text\{slow\}\}serves as a stable anchor by incorporating a longer history of game outcomes\. The divergence between these two signals effectively functions as a differential indicator of training progress: during phases of rapid policy improvement, the fast baseline quickly shifts to match new reward levels, whereas the slow baseline lags behind, creating a significant delta \(\|bpfast−bpslow\|\|b\_\{p\}^\{\\text\{fast\}\}\-b\_\{p\}^\{\\text\{slow\}\}\|\)\. Conversely, when the policy stabilizes—implying the reward distribution has become stationary—the fast baseline ceases to deviate from the slow baseline, and the two estimates asymptotically converge\. We leverage this property to define the Stagnation Coefficientσ\(t\)∈\[0,1\]\\sigma^\{\(t\)\}\\in\[0,1\]to quantify the stationarity of the training process:

σ\(t\)=1−tanh⁡\(\|bpfast,\(t\)−bpslow,\(t\)\|\)\\sigma^\{\(t\)\}=1\-\\tanh\\left\(\\left\|b\_\{p\}^\{\\text\{fast\},\(t\)\}\-b\_\{p\}^\{\\text\{slow\},\(t\)\}\\right\|\\right\)\(4\)When the policy evolves actively, the baseline divergence yields lowσ\(t\)\\sigma^\{\(t\)\}\. As the system settles into stagnation, the diminishing divergence drivesσ\(t\)\\sigma^\{\(t\)\}higher\.

##### Intervention Coefficient

High stationarity alone does not necessitate intervention, as it characterizes both successful convergence and failure\. To specifically isolate degenerate cases, we incorporate the Stagnation Coefficientσ\(t\)\\sigma^\{\(t\)\}with match entropyHmatch\(t\)H\_\{\\text\{match\}\}^\{\(t\)\}defined in[Section˜3](https://arxiv.org/html/2605.08721#S3)\. This yields the Intervention Coefficientλ\(t\)∈\[0,1\]\\lambda^\{\(t\)\}\\in\[0,1\], which determines the necessity of active guidance:

λ\(t\)=σ\(t\)⋅1−Hmatch\(t\)\\lambda^\{\(t\)\}=\\sigma^\{\(t\)\}\\cdot\\sqrt\{1\-H\_\{\\text\{match\}\}^\{\(t\)\}\}\(5\)
This soft\-gating mechanism ensures that intervention strength scales smoothly with the evolutionary state: \(1\) Healthy Exploration \(highHmatch\(t\)H\_\{\\text\{match\}\}^\{\(t\)\}\): Outcome diversity suppressesλ\(t\)\\lambda^\{\(t\)\}, minimizing interference during normal learning\. \(2\) Evolution Impasse \(lowHmatch\(t\)H\_\{\\text\{match\}\}^\{\(t\)\}, highσ\(t\)\\sigma^\{\(t\)\}\): The combination of outcome determinism and high stationarity elevatesλ\(t\)\\lambda^\{\(t\)\}, progressively strengthening the corrective signal\.

Table 1:Main experimental results on three adversarial games\. The best results arebolded\.MethodsDon’t Say ItNegotiationTwo DollarGPT\-5\.1Gemini\-2\.5Gemini\-2\.0AVG\.GPT\-5\.1Gemini\-2\.5Gemini\-2\.0AVG\.GPT\-5\.1Gemini\-2\.5Gemini\-2\.0AVG\.Qwen3\-4B\-BaseVANILLA6\.252\.471\.433\.390\.522\.210\.391\.041\.430\.782\.471\.56SPAG37\.1126\.8214\.5826\.178\.9822\.9518\.3516\.7625\.9124\.4826\.1725\.52GRPO42\.1941\.9341\.9342\.0115\.2322\.0120\.3919\.2126\.5629\.4327\.7327\.91MARS52\.4742\.1927\.9940\.8914\.0624\.3523\.7920\.7326\.1730\.0826\.8227\.69SPIRAL48\.3142\.0647\.2745\.8815\.1020\.5714\.8316\.8426\.3030\.0826\.5627\.65DEPT \(Ours\)56\.6454\.9558\.5956\.7317\.8436\.4643\.6732\.3533\.2036\.5932\.4234\.07Qwen3\-8B\-BaseVANILLA23\.2416\.415\.8615\.176\.259\.111\.695\.691\.691\.173\.522\.13SPAG54\.1048\.4426\.4141\.219\.1119\.7911\.0113\.3117\.5816\.9323\.4419\.31GRPO33\.4640\.2230\.6434\.775\.4717\.9720\.6814\.7025\.5226\.6925\.7826\.00MARS51\.9541\.5428\.3940\.628\.4616\.8023\.0416\.1035\.1625\.1326\.8229\.04SPIRAL41\.1529\.6942\.8437\.898\.5922\.2721\.0417\.3025\.6529\.1723\.8326\.22DEPT \(Ours\)63\.0252\.7347\.9254\.5614\.0632\.8148\.7631\.8840\.6238\.0230\.8636\.50

### 4\.2Asymmetric Advantage Reshaping

Whenλ\(t\)→1\\lambda^\{\(t\)\}\\to 1, the standard advantage signal vanishesA​\(τ\)≈0A\(\\tau\)\\approx 0\. To restore effective learning, we introduce an Asymmetric Advantage Reshaping, reshaping the advantage function by constructing a contrastive baseline that differentially treats dominant versus rare outcomes\.

##### Global Performance Bounds

Constructing such a contrastive signal requires a rigorous reference scale\. Arbitrary baseline modifications can lead to unbounded advantage values, risking gradient instability and distorting the reward scale\. To ensure effective reshaping, we track the historical upper and lower bounds of the expected returns\. We utilize the fast baselinebpfastb\_\{p\}^\{\\text\{fast\}\}: it is sufficiently sensitive to register performance extrema \(potential capabilities\) yet robust enough to filter out single\-sample noise\. We define the historical bounds as:

Vmax\(t\)=maxi≤t⁡\{bpfast,i\},Vmin\(t\)=mini≤t⁡\{bpfast,i\}V\_\{\\max\}^\{\(t\)\}=\\max\_\{i\\leq t\}\\\{b\_\{p\}^\{\\text\{fast\},i\}\\\},\\quad V\_\{\\min\}^\{\(t\)\}=\\min\_\{i\\leq t\}\\\{b\_\{p\}^\{\\text\{fast\},i\}\\\}\(6\)

##### Asymmetric Value Construction

Match entropy collapse stems from one outcome class dominating the batch\. To break this homogeneity, we assign different value based on whether a trajectoryτ\\tauproduces the dominant outcomeodomo\_\{\\text\{dom\}\}or not\. We then construct an asymmetric target valueVasym​\(τ,bpfast\)V\_\{\\text\{asym\}\}\(\\tau,b\_\{p\}^\{\\text\{fast\}\}\)that applies differential pressure based on the outcome type\. Formally, this asymmetric value is defined as:

Vasym​\(τ,bpfast\)=𝕀​\(oτ=odom\)⋅Vmax\(t\)\+𝕀​\(oτ≠odom\)⋅Vmin\(t\)\\begin\{split\}V\_\{\\text\{asym\}\}\(\\tau,b\_\{p\}^\{\\text\{fast\}\}\)&=\\mathbb\{I\}\(o\_\{\\tau\}=o\_\{\\text\{dom\}\}\)\\cdot V\_\{\\max\}^\{\(t\)\}\\\\ &\+\\mathbb\{I\}\(o\_\{\\tau\}\\neq o\_\{\\text\{dom\}\}\)\\cdot V\_\{\\min\}^\{\(t\)\}\\end\{split\}\(7\)This construction induces asymmetric gradient flow through the advantage functionAτ=Rτ−Vasym\(t\)​\(τ\)A\_\{\\tau\}=R\_\{\\tau\}\-V\_\{\\text\{asym\}\}^\{\(t\)\}\(\\tau\)\. For dominant\-outcome trajectories, the highVmaxV\_\{\\max\}reduces positive gradients and amplifies negative gradients, suppressing the over\-represented behavior\. For rare\-outcome trajectories, the lowVminV\_\{\\min\}amplifies positive gradients and reduces negative gradients, encouraging exploration of under\-represented strategies\.

##### Adaptive Baseline Fusion

Finally, to ensure smooth transitions between normal training and intervention phases, we synthesize the final baselineb~p​\(τ\)\\tilde\{b\}\_\{p\}\(\\tau\)by interpolating between the stable slow baseline and the asymmetric value\. The intervention coefficientλ\(t\)\\lambda^\{\(t\)\}governs this fusion:

b~p​\(τ\)=\(1−λ\(t\)\)⋅bpslow,\(t\)\+λ\(t\)⋅Vasym​\(τ,bpfast\)\\tilde\{b\}\_\{p\}\(\\tau\)=\(1\-\\lambda^\{\(t\)\}\)\\cdot b\_\{p\}^\{\\text\{slow\},\(t\)\}\+\\lambda^\{\(t\)\}\\cdot V\_\{\\text\{asym\}\}\(\\tau,b\_\{p\}^\{\\text\{fast\}\}\)\(8\)Whenλ\(t\)\\lambda^\{\(t\)\}is small, the baseline approximates the standard slow estimate, preserving normal gradient dynamics\. Asλ\(t\)\\lambda^\{\(t\)\}increases, the asymmetric term gradually dominates, progressively steering the policy away from the collapsed state\. The policy is updated using the reshaped advantageA~p​\(τ\)=Rp​\(τ\)−b~p​\(τ\)\\tilde\{A\}\_\{p\}\(\\tau\)=R\_\{p\}\(\\tau\)\-\\tilde\{b\}\_\{p\}\(\\tau\), yielding the final gradient objective:

∇θJ​\(θ\)\\displaystyle\\nabla\_\{\\theta\}J\(\\theta\)=𝔼τ∼πθ\[∑p∈\{0,1\}∑t∈Tp\(Rp\(τ\)−b~p\(τ\)\)\\displaystyle=\\mathbb\{E\}\_\{\\tau\\sim\\pi\_\{\\theta\}\}\\Big\[\\sum\_\{p\\in\\\{0,1\\\}\}\\sum\_\{t\\in T\_\{p\}\}\\left\(R\_\{p\}\(\\tau\)\-\\tilde\{b\}\_\{p\}\(\\tau\)\\right\)\(9\)⋅∇θlogπθ\(yt\(p\)∣ot,p\)\]\\displaystyle\\quad\\cdot\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{t\}^\{\(p\)\}\\mid o\_\{t\},p\)\\Big\]
The full optimization procedure is shown in[Algorithm˜1](https://arxiv.org/html/2605.08721#alg1)\. We also provide theoretical analysis about the mathematical foundations of our dual\-timescale metrics and the gradient dynamics guaranteeing impasse recovery in[Appendix˜A](https://arxiv.org/html/2605.08721#A1)\.

## 5Experiments

### 5\.1Experimental Settings

##### Tasks

We evaluate our method on three adversarial open\-ended language tasks involving multi\-turn conversation and decision\-making: \(1\)Don’t Say It: Players aim to trick their opponent into saying a secret word while avoiding saying their own\. The first to speak the word loses; otherwise, it is a draw upon reaching the turn limit\. \(2\)Negotiation: Players negotiate to exchange five resource types based on private valuations, aiming to maximize inventory value\. The player with the higher value wins, with ties declared for equal payoffs or reaching turn limit\. \(3\)Two Dollar: Players bargain to split a fixed fund based on hidden role constraints\. The player with the higher payoff wins; equal payoffs or reaching the turn limit result in a draw\. All three games are adopted from TextArena\(Guertleret al\.,[2025](https://arxiv.org/html/2605.08721#bib.bib5)\)\. For further details, please refer to[Section˜B\.2](https://arxiv.org/html/2605.08721#A2.SS2)\.

##### Baselines

We evaluate our method against several self\-play RLVR methods in zero\-sum games: \(1\)SPIRAL\(Liuet al\.,[2025a](https://arxiv.org/html/2605.08721#bib.bib9)\): The first fully online multi\-turn self\-play RL system for LLMs\. It introduces Role\-conditioned Advantage Estimation to prevent thinking collapse in zero\-sum games; \(2\)MARS\(Yuanet al\.,[2025](https://arxiv.org/html/2605.08721#bib.bib10)\): It modifies the multi\-turn GRPO algorithm by introducing a turn\-level advantage estimator and role\-specific advantage normalization to stabilize self\-play RL training; \(3\)GRPO: An adaptation of multi\-turn GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2605.08721#bib.bib11)\)for self\-play settings, where group advantages are calculated separately for each role to preserve effective role\-specific learning signals; \(4\)SPAG\(Chenget al\.,[2024](https://arxiv.org/html/2605.08721#bib.bib12)\): An offline RL method that designs a discounted reward scheme for adversarial language games\. Implementation details for all baselines are provided in[Section˜B\.1](https://arxiv.org/html/2605.08721#A2.SS1)\.

![Refer to caption](https://arxiv.org/html/2605.08721v1/x4.png)

Figure 4:Training dynamics on Two Dollars\. For Don’t Say it and Negotiation, please refer to[Figures˜6](https://arxiv.org/html/2605.08721#A1.F6)and[7](https://arxiv.org/html/2605.08721#A2.F7)\.
##### Evaluation

Following SPIRAL\(Liuet al\.,[2025a](https://arxiv.org/html/2605.08721#bib.bib9)\), we select Gemini\-2 \(gemini\-2\.0\-flash\-lite\) as one of our evaluation opponents\. To provide a more comprehensive assessment, we additionally evaluate our method against different LLMs: GPT\-5\.1 opponent \(gpt\-5\.1\-2025\-11\-13\) and Gemini\-2\.5 opponent \(gemini\-2\.5\-flash\-lite\)\. Performance is quantified by the win rate against these opponents, defined as1N​∑i=1N𝕀​\[Ri=1\]\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbb\{I\}\[R\_\{i\}=1\], whereRiR\_\{i\}denotes the final reward for theii\-th trajectory \(withRi=1R\_\{i\}=1indicating a win\)\. For each task, we conduct experiments using three random seeds \(42, 100, 200\), evaluatingN=256N=256episodes per seed\. The evaluation is symmetric, granting an equal opportunities to act as the first and second player\.

##### Training

We employ Qwen3\-4B\-Base and Qwen3\-8B\-Base\(Yanget al\.,[2025a](https://arxiv.org/html/2605.08721#bib.bib24)\)as the backbone models for all experiments\. Following SPIRAL\(Liuet al\.,[2025a](https://arxiv.org/html/2605.08721#bib.bib9)\), we set the training steps to 400 with 128 samples collected per step, resulting in a total of 51,200 game transitions for each task\. The training batch size is set to 128\. For the dual\-baseline mechanism, we useαfast=0\.5\\alpha\_\{\\text\{fast\}\}=0\.5andαslow=0\.95\\alpha\_\{\\text\{slow\}\}=0\.95\. Detailed RL training hyperparameters are provided in[Section˜B\.3](https://arxiv.org/html/2605.08721#A2.SS3)\.

### 5\.2Experimental Results

##### RQ1: Is DEPT effective for social language agents?

As shown in[Table˜1](https://arxiv.org/html/2605.08721#S4.T1), DEPT achieves the highest average win rate across all three games, consistently outperforming all baselines on both backbones\. On Qwen3\-4B\-Base, DEPT attains 32\.35% in Negotiation, surpassing the strongest baseline \(MARS: 20\.73%\) by a relative margin of 56%\. Similar gains are observed in Don’t Say It \(\+10\.8% over SPIRAL\) and Two Dollar \(\+6\.16% over GRPO\)\. These improvements scale consistently to Qwen3\-8B\-Base, where DEPT nearly doubles the Negotiation win rate compared to SPIRAL \(31\.88% vs\. 17\.30%\)\. The limited performance of competitive baselines aligns with the evolution impasse described in[Section˜3](https://arxiv.org/html/2605.08721#S3)\. In contrast, DEPT’s consistent superiority against diverse opponents demonstrates that our dual\-timescale perception and asymmetric reshaping mechanism effectively prevents strategic collapse, enabling robust social reasoning\.

##### RQ2: Can DEPT effectively perceive and intervene during training?

As shown in[Figures˜4](https://arxiv.org/html/2605.08721#S5.F4),[6](https://arxiv.org/html/2605.08721#A1.F6)and[7](https://arxiv.org/html/2605.08721#A2.F7), our method effectively mitigates match entropy collapse and preserves informative gradient signals, enabling the discovery of better strategies as reflected by steadily increasing win rates\. In contrast, the baseline converges to evolutionary impasse with suboptimal policies\. Furthermore, the proposed Intervention Coefficient accurately detects stagnation: during the initial exploration phase, the intervention remains low; as training progresses toward convergence, the Stagnation Coefficient increases, becoming more responsive to changes in match entropy and adaptively adjusting the intervention strength to maintain sufficient match entropy\.

![Refer to caption](https://arxiv.org/html/2605.08721v1/x5.png)

Figure 5:Ablation study for DEPT\.

### 5\.3Analysis

##### RQ3: How does DEPT work?

To examine the effectiveness of each component in DEPT, we conduct ablation studies using Qwen3\-8B\-Base, as reported in[Figure˜5](https://arxiv.org/html/2605.08721#S5.F5)\. All ablations lead to performance degradation, confirming the necessity of each design choice\.\(1\) Effect of perception\.Removing the Stagnation Coefficient causes aggressive entropy correction during non\-stationary training phases, leading to substantial advantage estimation errors that disrupt policy exploration\. Removing match entropy gating results in continuous penalties even when the outcome distribution maintains sufficient diversity, forcing ineffective random exploration and preventing convergence\.\(2\) Effect of asymmetric value\.Without asymmetric target values, the method cannot selectively suppress over\-represented behaviors while supporting under\-represented ones, leading to convergence at suboptimal low\-entropy states\.\(3\) Effect of dual\-baseline\.Using either the fast or slow baseline alone degrades performance\. This confirms that the effectiveness of DEPT arises from the synergy between dual baselines, which jointly enable accurate perception and targeted intervention\.

Table 2:Out\-of\-distribution evaluation on increased complexity games\.MethodsHardCoreDon’tSayItRandomValueNegotiationVariableSumDollarQwen3\-4B\-BaseVANILLA0\.390\.133\.78SPAG7\.0311\.8526\.17GRPO22\.0114\.5227\.99MARS12\.1112\.5525\.39SPIRAL17\.9712\.5623\.96DEPT \(Ours\)22\.9215\.3630\.47Qwen3\-8B\-BaseVANILLA2\.343\.782\.73SPAG12\.1213\.0226\.56GRPO10\.036\.6419\.14MARS8\.337\.5624\.48SPIRAL18\.886\.5223\.18DEPT \(Ours\)19\.2717\.0827\.73

Table 3:Out\-of\-distribution on reasoning benchmarks\. The best results arebolded\. The second results areunderline\.MethodsMinervaAMC\-23AIME24AIME25Math500OlympiadGPQA\-DMMLU\-ProAverageQwen3\-4B\-BaseVANILLA22\.3642\.039\.586\.8865\.8034\.5228\.7939\.3631\.07SPAG24\.7044\.2710\.697\.7870\.8736\.1533\.3352\.7135\.06GRPO23\.2538\.657\.435\.8056\.2626\.6435\.8649\.1430\.38MARS29\.5745\.479\.418\.4769\.2535\.1834\.0153\.3435\.59SPIRAL27\.1745\.739\.557\.8167\.9534\.1534\.1851\.3234\.73DEPT \(Ours\)32\.5748\.7011\.2210\.0374\.6438\.7937\.0456\.4538\.68Qwen3\-8B\-BaseVANILLA24\.1547\.6613\.4410\.0067\.5435\.8029\.8046\.9734\.42SPAG26\.2850\.4212\.7810\.7669\.6737\.2131\.1451\.4636\.22GRPO34\.2453\.7512\.5012\.0575\.9440\.5635\.6959\.0540\.47MARS33\.7553\.5914\.0311\.7474\.6340\.3537\.3757\.8040\.41SPIRAL31\.7153\.5913\.5410\.3872\.8138\.4035\.6958\.2839\.30DEPT \(Ours\)35\.0057\.0313\.0612\.4374\.9840\.8338\.7257\.6041\.21

##### RQ4: How does DEPT perform on out\-of\-distribution evaluation?

To further assess generalization, we evaluate DEPT on out\-of\-distribution games, QA, and mathematical reasoning tasks to examine whether it effectively enhances strategic capabilities\. For out\-of\-distribution games, we evaluate on three more complex, held\-out variants—HardCore Don’tSayIt, RandomValue Negotiation, and VariableSum Dollar—to test generalization from their simpler counterparts \(e\.g\., from Negotiation to RandomValue Negotiation\)\. We still report the average win rate across 3 seeds \(42, 100, 200\), 256 episodes per seed\. For QA and mathematical reasoning, we adopt eight benchmarks: MATH500, OlympiadBench, Minerva Math, AIME24, AIME25, and AMC23 for mathematics; GPQA\-Diamond and MMLU\-Pro for QA\. Evaluation details are deferred to the[Sections˜B\.4\.1](https://arxiv.org/html/2605.08721#A2.SS4.SSS1)and[B\.4\.2](https://arxiv.org/html/2605.08721#A2.SS4.SSS2)\.

As shown in[Tables˜2](https://arxiv.org/html/2605.08721#S5.T2)and[3](https://arxiv.org/html/2605.08721#S5.T3), DEPT demonstrates superior out\-of\-distribution generalization\. In out\-of\-distribution games, our approach maintains robust performance even in more complex environments\. For QA and mathematics, self\-play on games enables effective reasoning ability transfer to general reasoning tasks\. This aligns with prior findings\(Chenget al\.,[2024](https://arxiv.org/html/2605.08721#bib.bib12); Liuet al\.,[2025a](https://arxiv.org/html/2605.08721#bib.bib9); Yuanet al\.,[2025](https://arxiv.org/html/2605.08721#bib.bib10)\)that game\-based self\-play effectively elicits reasoning capabilities—since DEPT achieves stronger in\-game performance with more strategic behaviors, it consequently yields greater improvements on reasoning benchmarks\. The improvements are particularly pronounced on Qwen3\-4B\-Base, where consistent gains are observed across all benchmarks\.

##### RQ5: Additional results\.

We provide several supplementary analyses in the appendix to further validate the effectiveness of DEPT\. First, we examine the semantic diversity of generated actions to verify whether DEPT can produce more varied and contextually appropriate behaviors rather than repetitive patterns\. The results in[Figure˜8](https://arxiv.org/html/2605.08721#A2.F8)demonstrate that DEPT achieves higher semantic richness compared to baselines, indicating that our dual\-process mechanism encourages more creative and diverse action generation \(See[Appendix˜C](https://arxiv.org/html/2605.08721#A3)\)\. Second, we conduct a sensitivity analysis on the key hyperparameterαfast\\alpha\_\{\\text\{fast\}\}, which controls the balance between fast and slow thinking modes\. Our analysis in[Table˜5](https://arxiv.org/html/2605.08721#A4.T5)reveals that DEPT maintains stable performance across the range \[0\.4, 0\.6\], suggesting that our method is robust to hyperparameter choices and does not require extensive tuning for deployment \(See[Appendix˜D](https://arxiv.org/html/2605.08721#A4)\)\. Third, we analyze the computational overhead introduced by our dual\-process architecture to assess its practical applicability\. The results in[Figure˜9](https://arxiv.org/html/2605.08721#A5.F9)show that DEPT introduces negligible additional cost \(less than 0\.0016% of the total per\-iteration training time\) while achieving substantial performance improvements, making it suitable for real\-world applications where efficiency is critical \(See[Appendix˜E](https://arxiv.org/html/2605.08721#A5)\)\. Additionally, we perform comprehensive statistical significance tests to rigorously validate our experimental findings\. The results in[Table˜6](https://arxiv.org/html/2605.08721#A6.T6)confirm that DEPT provides statistically significant gains over all baselines \(p < 0\.05\), demonstrating that the observed improvements are not due to random variance \(See[Appendix˜F](https://arxiv.org/html/2605.08721#A6)\)\.

## 6Related Work

LLM\-based Reinforcement LearningReinforcement learning for LLMs has evolved from Reinforcement Learning from Human Feedback \(RLHF\) to RLVR\(Ouyanget al\.,[2022](https://arxiv.org/html/2605.08721#bib.bib41); Guoet al\.,[2025](https://arxiv.org/html/2605.08721#bib.bib1); Teamet al\.,[2025](https://arxiv.org/html/2605.08721#bib.bib2); Yueet al\.,[2025](https://arxiv.org/html/2605.08721#bib.bib3); Liuet al\.,[2025b](https://arxiv.org/html/2605.08721#bib.bib13); Tanet al\.,[2025](https://arxiv.org/html/2605.08721#bib.bib32); Nieet al\.,[2026](https://arxiv.org/html/2605.08721#bib.bib42); Tanet al\.,[2026](https://arxiv.org/html/2605.08721#bib.bib44)\), leveraging objective ground truth to unlock deep potential in mathematical reasoning and code generation\. However, existing RLVR research focuses on closed\-ended tasks with well\-defined answers\. Real\-world open\-ended social interactions—such as negotiation and deception—are fundamentally different\(FAIRet al\.,[2022](https://arxiv.org/html/2605.08721#bib.bib19); Parket al\.,[2023](https://arxiv.org/html/2605.08721#bib.bib20); Zhouet al\.,[2024](https://arxiv.org/html/2605.08721#bib.bib4); Xuet al\.,[2024](https://arxiv.org/html/2605.08721#bib.bib18); Wanget al\.,[2025b](https://arxiv.org/html/2605.08721#bib.bib35); Zhanget al\.,[2026](https://arxiv.org/html/2605.08721#bib.bib43)\): they involve infinite action spaces, lack unique correct strategies, and require real\-time reasoning about dynamic opponent intentions\(Abdulhaiet al\.,[2025](https://arxiv.org/html/2605.08721#bib.bib16); Yanget al\.,[2025b](https://arxiv.org/html/2605.08721#bib.bib7); Wanget al\.,[2025a](https://arxiv.org/html/2605.08721#bib.bib8)\)\. Our work addresses the challenge of applying RLVR to open\-ended social interaction while sustaining continuous evolution in dynamic environments\.

LLM Self\-play Training on Language GameAs a classical paradigm, Self\-play\(Silveret al\.,[2018](https://arxiv.org/html/2605.08721#bib.bib21); Bansalet al\.,[2018](https://arxiv.org/html/2605.08721#bib.bib27); Chenet al\.,[2024](https://arxiv.org/html/2605.08721#bib.bib22); Chenget al\.,[2024](https://arxiv.org/html/2605.08721#bib.bib12)\)addresses data scarcity while constructing an automatic curriculum through opponent co\-evolution\. For language games in LLMs, SPIRAL\(Liuet al\.,[2025a](https://arxiv.org/html/2605.08721#bib.bib9)\)and MARS\(Yuanet al\.,[2025](https://arxiv.org/html/2605.08721#bib.bib10)\)introduce RAE to handle role asymmetry, achieving strong performance in limited action spaces\. However, in more complex open\-ended language domains, these methods suffer from evolutionary impasse: LLMs converge to homogeneous behaviors, collapsing outcome distributions into deterministic results\. This drives advantages toward zero, causing gradient vanishing and trapping models in suboptimal states\. Our DEPT addresses this through dual\-baseline perception to detect stagnation, combined with asymmetric advantage reshaping to restore optimization landscape and ensure continuous policy evolution\.

## 7Conclusions

In this paper, we address the critical challenge of evolution impasse in applying self\-play RLVR to open\-ended social language games, where agent training is susceptible to gradient vanishing and strategic stagnation\. We propose DEPT, which dynamically perceives evolutionary states and adaptively adjusts the optimization landscape via dual\-timescale value baselines\. By quantifying stagnation through baseline divergence and match entropy, DEPT selectively intervenes to penalize dominant outcomes while amplifying rare trajectories, effectively restoring gradient signals and enforcing continuous strategic exploration\. Extensive empirical results confirm that our approach not only surpasses SOTA methods but also maintains superior out\-of\-distribution generalization capabilities\.

## Limitations

While DEPT demonstrates substantial improvements in preventing evolutionary impasse during self\-play training, several limitations offer avenues for future research\. \(1\) Self\-play reinforcement learning inherently demands computational resources\. Our experiments required approximately 30 GPU\-hours per run on 8 NVIDIA A800 GPUs\. These resource requirements may limit accessibility for researchers with constrained computational budgets\. \(2\) Following the reasoning\-then\-acting paradigm, our method generates extended response sequences containing both reasoning traces \(ctc\_\{t\}\) and executable actions \(ata\_\{t\}\)\. This increases inference latency compared to direct action generation, a common trade\-off in test\-time scaling approaches\. The computational cost becomes more pronounced when scaling to longer interaction horizons or larger LLM backbones\. We believe our dual\-scale evolutionary policy training method provides a principled foundation that could inspire novel mechanisms for detecting and intervening in more complex evolutionary dynamics, and we hope this work motivates the community to explore these promising directions\.

## Acknowledgments

This work is supported in part by the National Natural Science Foundation of China under Grants \#72293575, \#72225011 and \#72434005\. We also thank Longze Chen and Jiaming Li for their valuable suggestions on the experimental design of this paper\.

## References

- M\. Abdulhai, I\. White, C\. V\. Snell, C\. Sun, J\. Hong, Y\. Zhai, K\. Xu, and S\. Levine \(2025\)LMRL gym: benchmarks for multi\-turn reinforcement learning with language models\.InProceedings of ICML,External Links:[Link](https://openreview.net/forum?id=hmGhP5DO2W)Cited by:[§1](https://arxiv.org/html/2605.08721#S1.p1.1),[§6](https://arxiv.org/html/2605.08721#S6.p1.1)\.
- A\. Agarwal, N\. Jiang, S\. M\. Kakade, and W\. Sun \(2019\)Reinforcement learning: theory and algorithms\.Note:[https://rltheorybook\.github\.io/](https://rltheorybook.github.io/)Cited by:[§A\.2\.1](https://arxiv.org/html/2605.08721#A1.SS2.SSS1.1.p1.1.1)\.
- T\. Bansal, J\. Pachocki, S\. Sidor, I\. Sutskever, and I\. Mordatch \(2018\)Emergent complexity via multi\-agent competition\.InProceedings of ICLR,External Links:[Link](https://openreview.net/forum?id=Sy0GnUxCb)Cited by:[§1](https://arxiv.org/html/2605.08721#S1.p2.1),[§6](https://arxiv.org/html/2605.08721#S6.p2.1)\.
- F\. Bianchi, P\. J\. Chia, M\. Yuksekgonul, J\. Tagliabue, D\. Jurafsky, and J\. Zou \(2024\)How well can llms negotiate? negotiationarena platform and analysis\.InProceedings of ICML,pp\. 3935–3951\.External Links:[Link](https://openreview.net/forum?id=CmOmaxkt8p)Cited by:[§1](https://arxiv.org/html/2605.08721#S1.p1.1)\.
- Z\. Chen, Y\. Deng, H\. Yuan, K\. Ji, and Q\. Gu \(2024\)Self\-play fine\-tuning convertsweak language models to strong language models\.InProceedings of ICML,pp\. 6621–6642\.External Links:[Link](https://openreview.net/forum?id=O4cHTxW9BS)Cited by:[§1](https://arxiv.org/html/2605.08721#S1.p2.1),[§6](https://arxiv.org/html/2605.08721#S6.p2.1)\.
- P\. Cheng, T\. Hu, H\. Xu, Z\. Zhang, Y\. Dai, L\. Han, N\. Du, and X\. Li \(2024\)Self\-playing adversarial language game enhances llm reasoning\.InProceedings of NeurIPS,pp\. 126515–126543\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/e4be7e9867ef163563f4a5e90cec478f-Abstract-Conference.html)Cited by:[§5\.1](https://arxiv.org/html/2605.08721#S5.SS1.SSS0.Px2.p1.1),[§5\.3](https://arxiv.org/html/2605.08721#S5.SS3.SSS0.Px2.p2.1),[§6](https://arxiv.org/html/2605.08721#S6.p2.1)\.
- J\. Duan, R\. Zhang, J\. Diffenderfer, B\. Kailkhura, L\. Sun, E\. Stengel\-Eskin, M\. Bansal, T\. Chen, and K\. Xu \(2024\)Gtbench: uncovering the strategic reasoning capabilities of llms via game\-theoretic evaluations\.Proceedings of NeurIPS37,pp\. 28219–28253\.External Links:[Link](https://doi.org/10.52202/079017-0885)Cited by:[§1](https://arxiv.org/html/2605.08721#S1.p2.1)\.
- M\. FAIR, A\. Bakhtin, N\. Brown, E\. Dinan, G\. Farina, C\. Flaherty, D\. Fried, A\. Goff, J\. Gray, H\. Hu,et al\.\(2022\)Human\-level play in the game of diplomacy by combining language models with strategic reasoning\.Science378\(6624\),pp\. 1067–1074\.External Links:[Link](https://www.science.org/doi/abs/10.1126/science.ade9097)Cited by:[§1](https://arxiv.org/html/2605.08721#S1.p1.1),[§6](https://arxiv.org/html/2605.08721#S6.p1.1)\.
- L\. Guertler, B\. Cheng, S\. Yu, B\. Liu, L\. Choshen, and C\. Tan \(2025\)TextArena\.arXiv preprint arXiv:2504\.11442\.External Links:[Link](https://arxiv.org/abs/2504.11442)Cited by:[Appendix H](https://arxiv.org/html/2605.08721#A8.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2605.08721#S1.p2.1),[§5\.1](https://arxiv.org/html/2605.08721#S5.SS1.SSS0.Px1.p1.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025\)Deepseek\-r1 incentivizes reasoning in llms through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.External Links:[Link](https://www.nature.com/articles/s41586-025-09422-z)Cited by:[§1](https://arxiv.org/html/2605.08721#S1.p1.1),[§6](https://arxiv.org/html/2605.08721#S6.p1.1)\.
- T\. Hagendorff \(2023\)Deception abilities emerged in large language models\.arXiv preprint arXiv:2307\.16513\.External Links:[Link](https://arxiv.org/abs/2307.16513)Cited by:[§1](https://arxiv.org/html/2605.08721#S1.p1.1)\.
- C\. He, R\. Luo, Y\. Bai, S\. Hu, Z\. Thai, J\. Shen, J\. Hu, X\. Han, Y\. Huang, Y\. Zhang, J\. Liu, L\. Qi, Z\. Liu, and M\. Sun \(2024\)OlympiadBench: a challenging benchmark for promoting AGI with olympiad\-level bilingual multimodal scientific problems\.InProceedings of ACL,pp\. 3828–3850\.External Links:[Link](https://aclanthology.org/2024.acl-long.211/)Cited by:[§B\.4\.2](https://arxiv.org/html/2605.08721#A2.SS4.SSS2.Px1.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021\)Measuring mathematical problem solving with the MATH dataset\.InProceedings of NeurIPS Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=7Bywt2mQsCe)Cited by:[§B\.4\.2](https://arxiv.org/html/2605.08721#A2.SS4.SSS2.Px1.p1.1)\.
- W\. Hua, O\. Liu, L\. Li, A\. Amayuelas, J\. Chen, L\. Jiang, M\. Jin, L\. Fan, F\. Sun, W\. Wang,et al\.\(2024\)Game\-theoretic llm: agent workflow for negotiation games\.arXiv preprint arXiv:2411\.05990\.External Links:[Link](https://arxiv.org/abs/2411.05990)Cited by:[§1](https://arxiv.org/html/2605.08721#S1.p1.1),[§1](https://arxiv.org/html/2605.08721#S1.p2.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the SIGOPS,External Links:[Link](https://dl.acm.org/doi/10.1145/3600006.3613165)Cited by:[§B\.3](https://arxiv.org/html/2605.08721#A2.SS3.p1.1),[Appendix H](https://arxiv.org/html/2605.08721#A8.SS0.SSS0.Px2.p1.1)\.
- J\. E\. Laird \(2019\)The soar cognitive architecture\.MIT press\.External Links:[Link](https://mitpress.mit.edu/9780262538534/the-soar-cognitive-architecture/)Cited by:[§1](https://arxiv.org/html/2605.08721#S1.p3.1)\.
- A\. Lewkowycz, A\. Andreassen, D\. Dohan, E\. Dyer, H\. Michalewski, V\. Ramasesh, A\. Slone, C\. Anil, I\. Schlag, T\. Gutman\-Solo, Y\. Wu, B\. Neyshabur, G\. Gur\-Ari, and V\. Misra \(2022\)Solving quantitative reasoning problems with language models\.InProceedings of NeurIPS,Vol\.35,pp\. 3843–3857\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/file/18abbeef8cfe9203fdf9053c9c4fe191-Paper-Conference.pdf)Cited by:[§B\.4\.2](https://arxiv.org/html/2605.08721#A2.SS4.SSS2.Px1.p1.1)\.
- M\. L\. Littman \(1994\)Markov games as a framework for multi\-agent reinforcement learning\.InMachine learning proceedings 1994,pp\. 157–163\.External Links:[Link](https://www.sciencedirect.com/science/article/pii/B9781558603356500271)Cited by:[§2\.1](https://arxiv.org/html/2605.08721#S2.SS1.p1.20)\.
- B\. Liu, L\. Guertler, S\. Yu, Z\. Liu, P\. Qi, D\. Balcells, M\. Liu, C\. Tan, W\. Shi, M\. Lin,et al\.\(2025a\)SPIRAL: self\-play on zero\-sum games incentivizes reasoning via multi\-agent multi\-turn reinforcement learning\.arXiv preprint arXiv:2506\.24119\.External Links:[Link](https://arxiv.org/abs/2506.24119)Cited by:[§B\.2](https://arxiv.org/html/2605.08721#A2.SS2.SSS0.Px3.p2.4),[§1](https://arxiv.org/html/2605.08721#S1.p2.1),[§1](https://arxiv.org/html/2605.08721#S1.p3.1),[§2\.2](https://arxiv.org/html/2605.08721#S2.SS2.SSS0.Px1.p1.3),[§5\.1](https://arxiv.org/html/2605.08721#S5.SS1.SSS0.Px2.p1.1),[§5\.1](https://arxiv.org/html/2605.08721#S5.SS1.SSS0.Px3.p1.5),[§5\.1](https://arxiv.org/html/2605.08721#S5.SS1.SSS0.Px4.p1.2),[§5\.3](https://arxiv.org/html/2605.08721#S5.SS3.SSS0.Px2.p2.1),[§6](https://arxiv.org/html/2605.08721#S6.p2.1)\.
- Z\. Liu, C\. Chen, W\. Li, P\. Qi, T\. Pang, C\. Du, W\. S\. Lee, and M\. Lin \(2025b\)Understanding r1\-zero\-like training: a critical perspective\.InProceedings of COLM,External Links:[Link](https://openreview.net/forum?id=5PAF7PAY2Y)Cited by:[§1](https://arxiv.org/html/2605.08721#S1.p1.1),[§6](https://arxiv.org/html/2605.08721#S6.p1.1)\.
- Z\. Liu, C\. Chen, X\. Wan, C\. Du, W\. S\. Lee, and M\. Lin \(2024\)OAT: a research\-friendly framework for llm online alignment\.GitHub\.Note:[https://github\.com/sail\-sg/oat](https://github.com/sail-sg/oat)Cited by:[§B\.3](https://arxiv.org/html/2605.08721#A2.SS3.p1.1),[Appendix H](https://arxiv.org/html/2605.08721#A8.SS0.SSS0.Px2.p1.1)\.
- MAA \(2023\)American mathematics contest 12 \(amc 12\)\.External Links:[Link](https://artofproblemsolving.com/wiki/index.php/AMC_12_Problems_and_Solutions)Cited by:[§B\.4\.2](https://arxiv.org/html/2605.08721#A2.SS4.SSS2.Px1.p1.1)\.
- MAA \(2024\)American invitational mathematics examination \(aime\)\.External Links:[Link](https://artofproblemsolving.com/wiki/index.php/AIME_Problems_and_Solutions)Cited by:[§B\.4\.2](https://arxiv.org/html/2605.08721#A2.SS4.SSS2.Px1.p1.1)\.
- MAA \(2025\)American invitational mathematics examination \(aime\)\.External Links:[Link](https://artofproblemsolving.com/wiki/index.php/AIME_Problems_and_Solutions)Cited by:[§B\.4\.2](https://arxiv.org/html/2605.08721#A2.SS4.SSS2.Px1.p1.1)\.
- S\. Nie, S\. Ding, W\. Zhang, L\. Yu, T\. Yang, Y\. Chen, T\. Liu, W\. Yin, Y\. Sun, and H\. Wu \(2026\)ATTNPO: attention\-guided process supervision for efficient reasoning\.arXiv preprint arXiv:2602\.09953\.External Links:[Link](https://arxiv.org/abs/2601.08605)Cited by:[§6](https://arxiv.org/html/2605.08721#S6.p1.1)\.
- OpenAI \(2024\)Simple\-evals: a suite of lightweight evaluations for language models\.GitHub\.Note:[https://github\.com/openai/simple\-evals](https://github.com/openai/simple-evals)Cited by:[§B\.4\.2](https://arxiv.org/html/2605.08721#A2.SS4.SSS2.Px2.p1.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. F\. Christiano, J\. Leike, and R\. Lowe \(2022\)Training language models to follow instructions with human feedback\.InProceedings of NeurIPS,Vol\.35,pp\. 27730–27744\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf)Cited by:[§6](https://arxiv.org/html/2605.08721#S6.p1.1)\.
- M\. Papini, D\. Binaghi, G\. Canonaco, M\. Pirotta, and M\. Restelli \(2018\)Stochastic variance\-reduced policy gradient\.InProceedings of ICML,Vol\.80,pp\. 4026–4035\.External Links:[Link](https://proceedings.mlr.press/v80/papini18a.html)Cited by:[§A\.2\.1](https://arxiv.org/html/2605.08721#A1.SS2.SSS1.1.p1.1.1)\.
- J\. S\. Park, J\. O’Brien, C\. J\. Cai, M\. R\. Morris, P\. Liang, and M\. S\. Bernstein \(2023\)Generative agents: interactive simulacra of human behavior\.InProceedings of UIST,pp\. 1–22\.External Links:[Link](https://dl.acm.org/doi/abs/10.1145/3586183.3606763)Cited by:[§1](https://arxiv.org/html/2605.08721#S1.p1.1),[§6](https://arxiv.org/html/2605.08721#S6.p1.1)\.
- D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. Bowman \(2024\)GPQA: a graduate\-level google\-proof q&a benchmark\.InProceedings of COLM,External Links:[Link](https://openreview.net/forum?id=Ti67584b98)Cited by:[§B\.4\.2](https://arxiv.org/html/2605.08721#A2.SS4.SSS2.Px2.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.External Links:[Link](https://arxiv.org/abs/2402.03300)Cited by:[§5\.1](https://arxiv.org/html/2605.08721#S5.SS1.SSS0.Px2.p1.1)\.
- D\. Silver, T\. Hubert, J\. Schrittwieser, I\. Antonoglou, M\. Lai, A\. Guez, M\. Lanctot, L\. Sifre, D\. Kumaran, T\. Graepel,et al\.\(2018\)A general reinforcement learning algorithm that masters chess, shogi, and go through self\-play\.Science362\(6419\),pp\. 1140–1144\.External Links:[Link](https://www.science.org/doi/abs/10.1126/science.aar6404)Cited by:[§1](https://arxiv.org/html/2605.08721#S1.p2.1),[§6](https://arxiv.org/html/2605.08721#S6.p2.1)\.
- T\. Sumers, S\. Yao, K\. Narasimhan, and T\. Griffiths \(2023\)Cognitive architectures for language agents\.Transactions on Machine Learning Research\.External Links:[Link](https://openreview.net/forum?id=1i6ZCvflQJ)Cited by:[§1](https://arxiv.org/html/2605.08721#S1.p3.1)\.
- Y\. Tan, M\. Wang, S\. He, H\. Liao, C\. Zhao, Q\. Lu, T\. Liang, J\. Zhao, and K\. Liu \(2025\)Bottom\-up policy optimization: your language model policy secretly contains internal policies\.arXiv preprint arXiv:2512\.19673\.External Links:[Link](https://arxiv.org/abs/2512.19673)Cited by:[§6](https://arxiv.org/html/2605.08721#S6.p1.1)\.
- Y\. Tan, M\. Wang, B\. Liu, Z\. Liu, T\. Liang, S\. He, J\. Zhao, and K\. Liu \(2026\)FromP​\(y\|x\)P\(y\|x\)toP​\(y\)P\(y\): investigating reinforcement learning in pre\-train space\.External Links:[Link](https://arxiv.org/abs/2604.14142)Cited by:[§6](https://arxiv.org/html/2605.08721#S6.p1.1)\.
- K\. Team, A\. Du, B\. Gao, B\. Xing, C\. Jiang, C\. Chen, C\. Li, C\. Xiao, C\. Du, C\. Liao,et al\.\(2025\)Kimi k1\. 5: scaling reinforcement learning with llms\.arXiv preprint arXiv:2501\.12599\.External Links:[Link](https://arxiv.org/abs/2501.12599)Cited by:[§1](https://arxiv.org/html/2605.08721#S1.p1.1),[§6](https://arxiv.org/html/2605.08721#S6.p1.1)\.
- M\. Wang, Y\. Li, H\. Wang, X\. Zhang, N\. Xu, B\. Wu, F\. Huang, H\. Yu, and W\. Mao \(2025a\)Adaptive thinking via mode policy optimization for social language agents\.arXiv preprint arXiv:2505\.02156\.External Links:[Link](https://arxiv.org/abs/2505.02156)Cited by:[§1](https://arxiv.org/html/2605.08721#S1.p1.1),[§6](https://arxiv.org/html/2605.08721#S6.p1.1)\.
- M\. Wang, X\. Zhang, K\. Chen, N\. Xu, H\. Yu, F\. Huang, W\. Mao, and Y\. Li \(2025b\)DEMO: reframing dialogue interaction with fine\-grained element modeling\.InFindings of ACL,pp\. 11373–11401\.External Links:[Link](https://aclanthology.org/2025.findings-acl.594/)Cited by:[§6](https://arxiv.org/html/2605.08721#S6.p1.1)\.
- Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang, T\. Li, M\. Ku, K\. Wang, A\. Zhuang, R\. Fan, X\. Yue, and W\. Chen \(2024\)MMLU\-pro: a more robust and challenging multi\-task language understanding benchmark\.InProceedings of NeurIPS,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 95266–95290\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/ad236edc564f3e3156e1b2feafb99a24-Paper-Datasets_and_Benchmarks_Track.pdf)Cited by:[§B\.4\.2](https://arxiv.org/html/2605.08721#A2.SS4.SSS2.Px2.p1.1)\.
- T\. Xia, Z\. He, T\. Ren, Y\. Miao, Z\. Zhang, Y\. Yang, and R\. Wang \(2024\)Measuring bargaining abilities of llms: a benchmark and a buyer\-enhancement method\.InFindings of ACL,pp\. 3579–3602\.External Links:[Link](https://aclanthology.org/2024.findings-acl.213/)Cited by:[§1](https://arxiv.org/html/2605.08721#S1.p1.1)\.
- Z\. Xu, C\. Yu, F\. Fang, Y\. Wang, and Y\. Wu \(2024\)Language agents with reinforcement learning for strategic play in the werewolf game\.InProceedings of ICML,pp\. 55434–55464\.External Links:[Link](https://openreview.net/forum?id=usUPvQH3XK)Cited by:[§1](https://arxiv.org/html/2605.08721#S1.p1.1),[§6](https://arxiv.org/html/2605.08721#S6.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025a\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.External Links:[Link](https://arxiv.org/abs/2505.09388)Cited by:[Appendix H](https://arxiv.org/html/2605.08721#A8.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2605.08721#S5.SS1.SSS0.Px4.p1.2)\.
- R\. Yang, Y\. Zhang, A\. Chen, X\. Wang, J\. Chen, S\. Yuan, D\. Yang, and Y\. Xiao \(2025b\)ARIA: training language agents with intention\-driven reward aggregation\.InProceedings of NeurIPS,External Links:[Link](https://openreview.net/forum?id=eumRwpgdMU)Cited by:[§1](https://arxiv.org/html/2605.08721#S1.p1.1),[§6](https://arxiv.org/html/2605.08721#S6.p1.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. R\. Narasimhan, and Y\. Cao \(2023\)ReAct: synergizing reasoning and acting in language models\.InProceedings of ICLR,External Links:[Link](https://openreview.net/forum?id=WE_vluYUL-X)Cited by:[§2\.2](https://arxiv.org/html/2605.08721#S2.SS2.p1.7)\.
- H\. Yuan, Z\. Xu, Z\. Tan, X\. Yi, M\. Guang, K\. Long, H\. Hui, B\. Li, X\. Chen, B\. Zhao,et al\.\(2025\)MARS: reinforcing multi\-agent reasoning of llms through self\-play in strategic games\.arXiv preprint arXiv:2510\.15414\.External Links:[Link](https://arxiv.org/abs/2510.15414v1)Cited by:[§1](https://arxiv.org/html/2605.08721#S1.p3.1),[§2\.2](https://arxiv.org/html/2605.08721#S2.SS2.SSS0.Px1.p1.3),[§5\.1](https://arxiv.org/html/2605.08721#S5.SS1.SSS0.Px2.p1.1),[§5\.3](https://arxiv.org/html/2605.08721#S5.SS3.SSS0.Px2.p2.1),[§6](https://arxiv.org/html/2605.08721#S6.p2.1)\.
- Y\. Yue, Z\. Chen, R\. Lu, A\. Zhao, Z\. Wang, Y\. Yue, S\. Song, and G\. Huang \(2025\)Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model?\.InProceedings of NeurIPS,External Links:[Link](https://openreview.net/forum?id=4OsgYD7em5)Cited by:[§1](https://arxiv.org/html/2605.08721#S1.p1.1),[§6](https://arxiv.org/html/2605.08721#S6.p1.1)\.
- W\. Zhang, X\. Zhang, H\. Yu, S\. Nie, B\. Wu, J\. Yue, T\. Liu, and Y\. Li \(2026\)ExpSeek: self\-triggered experience seeking for web agents\.arXiv preprint arXiv:2601\.08605\.External Links:[Link](https://arxiv.org/abs/2601.08605)Cited by:[§6](https://arxiv.org/html/2605.08721#S6.p1.1)\.
- X\. Zhou, H\. Zhu, L\. Mathur, R\. Zhang, H\. Yu, Z\. Qi, L\. Morency, Y\. Bisk, D\. Fried, G\. Neubig, and M\. Sap \(2024\)SOTOPIA: interactive evaluation for social intelligence in language agents\.InProceedings of ICLR,External Links:[Link](https://openreview.net/forum?id=mM7VurbA4r)Cited by:[§1](https://arxiv.org/html/2605.08721#S1.p1.1),[§6](https://arxiv.org/html/2605.08721#S6.p1.1)\.

## Appendix ATheoretical Analysis

In this section, we provide a theoretical analysis of DEPT\. We first establish the mathematical foundation of the Time\-scaled Evolutionary Perception mechanism, proving that the proposed metrics serve as estimators for the velocity and variance of the reward distribution\. Subsequently, we analyze the gradient dynamics, proving that our method ensures effectively restoring gradient signals in evolutionary impasse via a synthetic variance injection mechanism\.

### A\.1Mathematical Foundation of Evolutionary Perception

The core of DEPT lies in quantifying the evolutionary state using baseline divergence and match entropy\. Here, we provide the formal derivations linking these metrics to the underlying reward dynamics\.

#### A\.1\.1Baseline Divergence as Velocity Estimation

In the main text, we utilize the divergence between fast and slow baselines to measure training stationarity\. We now prove that this metric is mathematically equivalent to an estimator of thevelocity of expected return\.

###### Proposition A\.1\(Divergence\-Velocity Equivalence\)\.

Letμ​\(t\)=𝔼​\[Rp​\(τ\)\]\\mu\(t\)=\\mathbb\{E\}\[R\_\{p\}\(\\tau\)\]be the underlying expected return at steptt\. Assumingμ​\(t\)\\mu\(t\)is locally linear within the effective window of the Exponential Moving Average \(EMA\), the expected divergence between two baselines with decay ratesαslow\>αfast\\alpha\_\{\\text\{slow\}\}\>\\alpha\_\{\\text\{fast\}\}is proportional to the magnitude of the time derivative of the expected return:

𝔼​\[\|bpfast,t−bpslow,t\|\]≈C⋅\|d​μ​\(t\)d​t\|,\\mathbb\{E\}\\left\[\\left\|b\_\{p\}^\{\\text\{fast\},t\}\-b\_\{p\}^\{\\text\{slow\},t\}\\right\|\\right\]\\approx C\\cdot\\left\|\\frac\{d\\mu\(t\)\}\{dt\}\\right\|,\(10\)whereC\>0C\>0is a constant determined by the time constants\.

###### Proof\.

Recall the EMA update rule:bpk,t=αk​bpk,t−1\+\(1−αk\)​Rp​\(τ\)b\_\{p\}^\{k,t\}=\\alpha\_\{k\}b\_\{p\}^\{k,t\-1\}\+\(1\-\\alpha\_\{k\}\)R\_\{p\}\(\\tau\)\. By recursively expanding this update, the baseline at timettcan be expressed as a weighted sum of historical rewards\. Taking the expectation and performing a first\-order Taylor expansion of the expected returnμ​\(t−j\)≈μ​\(t\)−j⋅μ˙​\(t\)\\mu\(t\-j\)\\approx\\mu\(t\)\-j\\cdot\\dot\{\\mu\}\(t\):

𝔼​\[bpk,t\]\\displaystyle\\mathbb\{E\}\[b\_\{p\}^\{k,t\}\]=\(1−αk\)​∑j=0∞αkj​μ​\(t−j\)\\displaystyle=\(1\-\\alpha\_\{k\}\)\\sum\_\{j=0\}^\{\\infty\}\\alpha\_\{k\}^\{j\}\\mu\(t\-j\)≈\(1−αk\)​∑j=0∞αkj​\(μ​\(t\)−j⋅μ˙​\(t\)\)\\displaystyle\\approx\(1\-\\alpha\_\{k\}\)\\sum\_\{j=0\}^\{\\infty\}\\alpha\_\{k\}^\{j\}\\left\(\\mu\(t\)\-j\\cdot\\dot\{\\mu\}\(t\)\\right\)=μ​\(t\)​\(1−αk\)​∑j=0∞αkj⏟=1−μ˙​\(t\)​\(1−αk\)​∑j=0∞j​αkj\.\\displaystyle=\\mu\(t\)\\underbrace\{\(1\-\\alpha\_\{k\}\)\\sum\_\{j=0\}^\{\\infty\}\\alpha\_\{k\}^\{j\}\}\_\{=1\}\-\\dot\{\\mu\}\(t\)\(1\-\\alpha\_\{k\}\)\\sum\_\{j=0\}^\{\\infty\}j\\alpha\_\{k\}^\{j\}\.\(11\)Using the geometric series identity∑j=0∞j​xj=x\(1−x\)2\\sum\_\{j=0\}^\{\\infty\}jx^\{j\}=\\frac\{x\}\{\(1\-x\)^\{2\}\}for\|x\|<1\|x\|<1, we substitutex=αkx=\\alpha\_\{k\}:

𝔼​\[bpk,t\]≈μ​\(t\)−μ˙​\(t\)⋅αk1−αk\.\\mathbb\{E\}\[b\_\{p\}^\{k,t\}\]\\approx\\mu\(t\)\-\\dot\{\\mu\}\(t\)\\cdot\\frac\{\\alpha\_\{k\}\}\{1\-\\alpha\_\{k\}\}\.\(12\)Let𝒯k=αk1−αk\\mathcal\{T\}\_\{k\}=\\frac\{\\alpha\_\{k\}\}\{1\-\\alpha\_\{k\}\}denote the effective time lag\. The expected divergence is:

𝔼​\[δ\(t\)\]=\|𝔼​\[bpfast,t\]−𝔼​\[bpslow,t\]\|≈\|\(μ​\(t\)−𝒯fast​μ˙​\(t\)\)−\(μ​\(t\)−𝒯slow​μ˙​\(t\)\)\|=\(𝒯slow−𝒯fast\)⋅\|μ˙​\(t\)\|\.\\begin\{split\}\\mathbb\{E\}\[\\delta^\{\(t\)\}\]&=\\left\|\\mathbb\{E\}\[b\_\{p\}^\{\\text\{fast\},t\}\]\-\\mathbb\{E\}\[b\_\{p\}^\{\\text\{slow\},t\}\]\\right\|\\\\ &\\approx\\left\|\(\\mu\(t\)\-\\mathcal\{T\}\_\{\\text\{fast\}\}\\dot\{\\mu\}\(t\)\)\-\(\\mu\(t\)\-\\mathcal\{T\}\_\{\\text\{slow\}\}\\dot\{\\mu\}\(t\)\)\\right\|\\\\ &=\(\\mathcal\{T\}\_\{\\text\{slow\}\}\-\\mathcal\{T\}\_\{\\text\{fast\}\}\)\\cdot\|\\dot\{\\mu\}\(t\)\|\.\\end\{split\}\(13\)Sinceαslow\>αfast\\alpha\_\{\\text\{slow\}\}\>\\alpha\_\{\\text\{fast\}\}, we have𝒯slow\>𝒯fast\\mathcal\{T\}\_\{\\text\{slow\}\}\>\\mathcal\{T\}\_\{\\text\{fast\}\}, ensuring the coefficient is positive\. Thus, the baseline divergence serves as a direct proxy for the training velocity\|μ˙​\(t\)\|\|\\dot\{\\mu\}\(t\)\|\. ∎

#### A\.1\.2Match Entropy as Time\-Varying Variance Proxy

We now establish the mathematical relationship between the observed Match EntropyHmatch\(t\)H\_\{\\text\{match\}\}^\{\(t\)\}and the underlying reward variance\.

###### Lemma A\.1\(Entropy\-Variance Correlation\)\.

Letν​\(t\)=Var​\[Rp​\(τ\)\]\\nu\(t\)=\\text\{Var\}\[R\_\{p\}\(\\tau\)\]be the underlying reward variance at time steptt\. As the match entropy vanishes, the reward variance asymptotically approaches zero:

limHmatch\(t\)→0ν​\(t\)=0\.\\lim\_\{H\_\{\\text\{match\}\}^\{\(t\)\}\\to 0\}\\nu\(t\)=0\.\(14\)Consequently, maintaining high entropy is a necessary condition for maintaining non\-zero reward variance\.

###### Proof\.

Consider the discrete outcome space𝒪=\{o1,…,oK\}\\mathcal\{O\}=\\\{o\_\{1\},\\dots,o\_\{K\}\\\}\(e\.g\., \{Win, Loss, Draw\}\) with associated fixed reward values\{r1,…,rK\}\\\{r\_\{1\},\\dots,r\_\{K\}\\\}\. At timett, the policyπθt\\pi\_\{\\theta\_\{t\}\}induces a probability distributionpi​\(t\)=P​\(Rp​\(τ\)=ri\)p\_\{i\}\(t\)=P\(R\_\{p\}\(\\tau\)=r\_\{i\}\)\.

The varianceν​\(t\)\\nu\(t\)is formally defined as:

ν​\(t\)=∑i=1Kpi​\(t\)​\(ri−𝔼​\[Rp​\(τ\)\]\)2\.\\nu\(t\)=\\sum\_\{i=1\}^\{K\}p\_\{i\}\(t\)\\left\(r\_\{i\}\-\\mathbb\{E\}\[R\_\{p\}\(\\tau\)\]\\right\)^\{2\}\.\(15\)From information theory, the entropyHmatch\(t\)=−∑pi​\(t\)​log⁡pi​\(t\)H\_\{\\text\{match\}\}^\{\(t\)\}=\-\\sum p\_\{i\}\(t\)\\log p\_\{i\}\(t\)approaches zero if and only if the probability mass concentrates on a single dominant outcomeoko\_\{k\}\. Mathematically, this implies the limit behavior:

pk​\(t\)→1andpj≠k​\(t\)→0\.p\_\{k\}\(t\)\\to 1\\quad\\text\{and\}\\quad p\_\{j\\neq k\}\(t\)\\to 0\.\(16\)In this limit, the expected return converges to the reward of the dominant outcome:𝔼​\[Rp​\(τ\)\]→rk\\mathbb\{E\}\[R\_\{p\}\(\\tau\)\]\\to r\_\{k\}\. Substituting these limits into the definition ofν​\(t\)\\nu\(t\):

limHmatch\(t\)→0ν​\(t\)=limpk→1\[pk\(t\)\(rk−rk\)2\+∑j≠kpj\(t\)\(rj−rk\)2\]=1⋅0\+0=0\.\\begin\{split\}\\lim\_\{H\_\{\\text\{match\}\}^\{\(t\)\}\\to 0\}\\nu\(t\)&=\\lim\_\{p\_\{k\}\\to 1\}\\Big\[p\_\{k\}\(t\)\(r\_\{k\}\-r\_\{k\}\)^\{2\}\\\\ &\\quad\+\\sum\_\{j\\neq k\}p\_\{j\}\(t\)\(r\_\{j\}\-r\_\{k\}\)^\{2\}\\Big\]\\\\ &=1\\cdot 0\+0=0\.\\end\{split\}\(17\)This derivation proves thatHmatch\(t\)H\_\{\\text\{match\}\}^\{\(t\)\}andν​\(t\)\\nu\(t\)are positively correlated in the asymptotic limit: a collapse in match entropy strictly necessitates a collapse in the reward varianceν​\(t\)\\nu\(t\)\. ∎

#### A\.1\.3Bayesian Derivation of Intervention Coefficient

We presented the intervention coefficientλ\(t\)\\lambda^\{\(t\)\}in a specific functional form in the main text\. Here, we derive this form from a general Bayesian perspective\.

###### Theorem A\.1\(Bayesian Form ofλ\\lambda\)\.

LetZt∈\{0,1\}Z\_\{t\}\\in\\\{0,1\\\}be a latent variable whereZt=1Z\_\{t\}=1indicates a evolution impasse\. The coefficientλ\(t\)\\lambda^\{\(t\)\}represents the posterior probabilityP​\(Zt=1∣𝒪t\)P\(Z\_\{t\}=1\\mid\\mathcal\{O\}\_\{t\}\)given observations𝒪t\\mathcal\{O\}\_\{t\}\.

###### Proof\.

We decompose the degenerate state into two independent necessary conditions: Stationarity \(𝒮\\mathcal\{S\}\) and Determinism \(𝒟\\mathcal\{D\}\)\. Using Bayes’ theorem and assuming conditional independence:

λ\(t\)≜P​\(Zt=1\)∝P​\(𝒮∣δ\(t\)\)⋅P​\(𝒟∣Hmatch\(t\)\)\.\\lambda^\{\(t\)\}\\triangleq P\(Z\_\{t\}=1\)\\propto P\(\\mathcal\{S\}\\mid\\delta^\{\(t\)\}\)\\cdot P\(\\mathcal\{D\}\\mid H\_\{\\text\{match\}\}^\{\(t\)\}\)\.\(18\)To map observations to probabilities, we introduce two kernel functions𝒦𝒮\\mathcal\{K\}\_\{\\mathcal\{S\}\}and𝒦𝒟\\mathcal\{K\}\_\{\\mathcal\{D\}\}with range\[0,1\]\[0,1\]:

1. 1\.Stagnation Kernel𝒦𝒮\\mathcal\{K\}\_\{\\mathcal\{S\}\}:Sinceδ\(t\)∝\|Velocity\|\\delta^\{\(t\)\}\\propto\|\\text\{Velocity\}\|, stationarity impliesδ\(t\)→0\\delta^\{\(t\)\}\\to 0\. We select the hyperbolic tangent kernel: σ\(t\)=P​\(𝒮∣δ\(t\)\)=1−tanh⁡\(δ\(t\)\)\.\\sigma^\{\(t\)\}=P\(\\mathcal\{S\}\\mid\\delta^\{\(t\)\}\)=1\-\\tanh\(\\delta^\{\(t\)\}\)\.\(19\)
2. 2\.Determinism Kernel𝒦𝒟\\mathcal\{K\}\_\{\\mathcal\{D\}\}:Since low entropy implies determinism, we select the radical kernel to ensure high sensitivity near the boundary of exploration \(H≈1H\\approx 1\): γ\(t\)=P​\(𝒟∣Hmatch\(t\)\)=1−Hmatch\(t\)\.\\gamma^\{\(t\)\}=P\(\\mathcal\{D\}\\mid H\_\{\\text\{match\}\}^\{\(t\)\}\)=\\sqrt\{1\-H\_\{\\text\{match\}\}^\{\(t\)\}\}\.\(20\)

Substituting these specific kernels yields the instantiation used in DEPT:λ\(t\)=σ\(t\)⋅γ\(t\)\\lambda^\{\(t\)\}=\\sigma^\{\(t\)\}\\cdot\\gamma^\{\(t\)\}\. ∎

### A\.2Gradient Dynamics Analysis

#### A\.2\.1The Vanishing Gradient of Standard Methods

We first prove why standard methods fail in evolutionary impasse\.

###### Theorem A\.2\(Variance\-Dependent Gradient Bound\)\.

Assume the policyπθ\\pi\_\{\\theta\}is Lipschitz continuous such that the norm of the score function is bounded by a constantGmaxG\_\{\\max\}\. The magnitude of the expected policy gradient in standard Advantage Estimation is upper\-bounded by the standard deviation of the reward:

‖∇θJ​\(θ\)‖≤Gmax⋅ν​\(t\)\.\\\|\\nabla\_\{\\theta\}J\(\\theta\)\\\|\\leq G\_\{\\max\}\\cdot\\sqrt\{\\nu\(t\)\}\.\(21\)

###### Proof\.

We start with the standard gradient definition∇θJ=𝔼τ​\[\(R−μ\)​∇θlog⁡πθ\]\\nabla\_\{\\theta\}J=\\mathbb\{E\}\_\{\\tau\}\[\(R\-\\mu\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\]\. First, we invoke theLipschitz continuity assumptionPapiniet al\.\([2018](https://arxiv.org/html/2605.08721#bib.bib48)\); Agarwalet al\.\([2019](https://arxiv.org/html/2605.08721#bib.bib49)\): for smooth neural policies, the gradient of the log\-likelihood is bounded:

supτ‖∇θlog⁡πθ​\(τ\)‖≤Gmax\.\\sup\_\{\\tau\}\\\|\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(\\tau\)\\\|\\leq G\_\{\\max\}\.\(22\)Next, we apply theCauchy\-Schwarz inequalityfor random variables \(𝔼​\[X​Y\]2≤𝔼​\[X2\]​𝔼​\[Y2\]\\mathbb\{E\}\[XY\]^\{2\}\\leq\\mathbb\{E\}\[X^\{2\}\]\\mathbb\{E\}\[Y^\{2\}\]\):

‖∇θJ‖2=‖𝔼​\[\(R−μ\)​∇log⁡π\]‖2≤𝔼​\[\(R−μ\)2\]⋅𝔼​\[‖∇log⁡π‖2\]=ν​\(t\)⋅𝔼​\[‖∇log⁡π‖2\]\.\\begin\{split\}\\\|\\nabla\_\{\\theta\}J\\\|^\{2\}&=\\left\\\|\\mathbb\{E\}\\left\[\(R\-\\mu\)\\nabla\\log\\pi\\right\]\\right\\\|^\{2\}\\\\ &\\leq\\mathbb\{E\}\\left\[\(R\-\\mu\)^\{2\}\\right\]\\cdot\\mathbb\{E\}\\left\[\\\|\\nabla\\log\\pi\\\|^\{2\}\\right\]\\\\ &=\\nu\(t\)\\cdot\\mathbb\{E\}\\left\[\\\|\\nabla\\log\\pi\\\|^\{2\}\\right\]\.\\end\{split\}\(23\)Substituting the boundGmaxG\_\{\\max\}:

‖∇θJ‖2≤ν​\(t\)⋅Gmax2⟹‖∇θJ‖≤Gmax​ν​\(t\)\.\\\|\\nabla\_\{\\theta\}J\\\|^\{2\}\\leq\\nu\(t\)\\cdot G\_\{\\max\}^\{2\}\\implies\\\|\\nabla\_\{\\theta\}J\\\|\\leq G\_\{\\max\}\\sqrt\{\\nu\(t\)\}\.\(24\)Implication:In a evolution impasse, Lemma[A\.1](https://arxiv.org/html/2605.08721#A1.Thmlemma1)dictates thatν​\(t\)→0\\nu\(t\)\\to 0\. Consequently, the gradient magnitude‖∇J‖\\\|\\nabla J\\\|is forced to zero\. The optimization halts specifically because thesignal varianceν​\(t\)\\nu\(t\)required to drive the gradient has vanished\. ∎

#### A\.2\.2Gradient Restoration

We prove that DEPT can effectively recovers gradient\.

###### Theorem A\.3\(Push\-Pull Gradient Restoration\)\.

In a degenerate state where natural varianceν​\(t\)→0\\nu\(t\)\\to 0, DEPT constructs a non\-vanishing synthetic gradient field driven by the historical value range\[Vmin,Vmax\]\[V\_\{\\min\},V\_\{\\max\}\]\.

###### Proof\.

Consider a training batchℬ\\mathcal\{B\}of sizeBB, decomposed into a set of dominant trajectories𝒟dom\\mathcal\{D\}\_\{\\text\{dom\}\}\(sizeMM\) and rare trajectories𝒟rare\\mathcal\{D\}\_\{\\text\{rare\}\}\(sizeNN\), such thatB=M\+NB=M\+NandM≫NM\\gg N\(indicating a degenerate state\)\. When intervention is triggered \(λ\(t\)→1\\lambda^\{\(t\)\}\\to 1\), the gradient update∇Jp\\nabla J\_\{p\}is calculated as the summation over these two subsets:

∇Jp≈1B\[∑τ∈𝒟dom\(Rp​\(τ\)−Vmax\)​∇log⁡πp​\(τ\)⏟Aggregate Push Force\+∑τ′∈𝒟rare\(Rp​\(τ′\)−Vmin\)​∇log⁡πp​\(τ′\)⏟Aggregate Pull Force\]\.\\begin\{split\}\\nabla J\_\{p\}&\\approx\\frac\{1\}\{B\}\\Bigg\[\\underbrace\{\\sum\_\{\\tau\\in\\mathcal\{D\}\_\{\\text\{dom\}\}\}\(R\_\{p\}\(\\tau\)\-V\_\{\\max\}\)\\nabla\\log\\pi\_\{p\}\(\\tau\)\}\_\{\\text\{Aggregate Push Force\}\}\\\\ &\\quad\+\\underbrace\{\\sum\_\{\\tau^\{\\prime\}\\in\\mathcal\{D\}\_\{\\text\{rare\}\}\}\(R\_\{p\}\(\\tau^\{\\prime\}\)\-V\_\{\\min\}\)\\nabla\\log\\pi\_\{p\}\(\\tau^\{\\prime\}\)\}\_\{\\text\{Aggregate Pull Force\}\}\\Bigg\]\.\\end\{split\}\(25\)1. 1\.Symmetric Dilemma:In standard methods, the baselineb≈1M​∑Rp​\(τdom\)b\\approx\\frac\{1\}\{M\}\\sum R\_\{p\}\(\\tau\_\{\\text\{dom\}\}\)\. Thus, the advantage for theMMdominant samples approaches zero\. SinceN→0N\\to 0, the total gradient vanishes\.
2. 2\.Aggregate Push Mechanism \(MMsamples\):For the dominant set, we set the baseline toVmaxV\_\{\\max\}\. SinceVmax≥Rp​\(τ\)V\_\{\\max\}\\geq R\_\{p\}\(\\tau\)forτ∈𝒟dom\\tau\\in\\mathcal\{D\}\_\{\\text\{dom\}\}, the term\(Rp​\(τ\)−Vmax\)\(R\_\{p\}\(\\tau\)\-V\_\{\\max\}\)is strictly non\-positive\. Although individual gradients may be small, the summation overMMsamples generates a substantial negative gradient field, actively suppressing the probability of the stagnant strategy\.
3. 3\.Aggregate Pull Mechanism \(NNsamples\):For the rare set, we set the baseline toVminV\_\{\\min\}\. The term\(Rp​\(τ′\)−Vmin\)\(R\_\{p\}\(\\tau^\{\\prime\}\)\-V\_\{\\min\}\)is maximized, assigning a high positive weight to these sparse signals\. This amplifies the exploration direction even whenNNis small\.

Through this summation, DEPT effectively injects aSynthetic Varianceνsyn∝\(Vmax−Vmin\)2\\nu\_\{\\text\{syn\}\}\\propto\(V\_\{\\max\}\-V\_\{\\min\}\)^\{2\}into the batch\. Unlike the natural varianceν​\(t\)\\nu\(t\)which vanishes asN→0N\\to 0, this synthetic variance remains robust, ensuring‖∇Jp‖\>0\\\|\\nabla J\_\{p\}\\\|\>0and forcing the policy distribution to shift away from the evolution impasse\. ∎

![Refer to caption](https://arxiv.org/html/2605.08721v1/x6.png)

Figure 6:Training dynamics on Don’t Say It\.Algorithm 1DEPT: Dual\-scale Evolutionary Policy Training1:Policy

πθ\\pi\_\{\\theta\}, decay rates

αfast<αslow\\alpha\_\{\\text\{fast\}\}<\\alpha\_\{\\text\{slow\}\}
2:Initialize baselines

bpfast=bpslow=0b\_\{p\}^\{\\text\{fast\}\}=b\_\{p\}^\{\\text\{slow\}\}=0for

p∈\{0,1\}p\\in\\\{0,1\\\}
3:whilenot convergeddo

4:// Self\-Play Trajectory Collection

5:Collect batch

ℬ=\{τk\}k=1K\\mathcal\{B\}=\\\{\\tau\_\{k\}\\\}\_\{k=1\}^\{K\}via self\-play, where each

τk\\tau\_\{k\}contains alternating

6:actions

\{yt\(p\)\}\\\{y\_\{t\}^\{\(p\)\}\\\}from

πθ\(⋅∣ot,p\)\\pi\_\{\\theta\}\(\\cdot\\mid o\_\{t\},p\)and terminal rewards

R0=−R1=ρ​\(sT\)R\_\{0\}=\-R\_\{1\}=\\rho\(s\_\{T\}\)
7:// Batch\-Level Statistics

8:Compute outcome distribution

P​\(o∣p\)P\(o\\mid p\)for

o∈\{win,draw,loss\}o\\in\\\{\\text\{win\},\\text\{draw\},\\text\{loss\}\\\}from

ℬ\\mathcal\{B\}
9:Identify dominant outcome

odomp←arg⁡maxo⁡P​\(o∣p\)o\_\{\\text\{dom\}\}^\{p\}\\leftarrow\\arg\\max\_\{o\}P\(o\\mid p\)for each

pp
10:Compute normalized match entropy

Hmatch←−1log⁡3​∑oP​\(o\)​log⁡P​\(o\)H\_\{\\text\{match\}\}\\leftarrow\-\\frac\{1\}\{\\log 3\}\\sum\_\{o\}P\(o\)\\log P\(o\)
11:// Dual\-Timescale Update & Asymmetric Advantage Reshaping

12:for

τ∈ℬ\\tau\\in\\mathcal\{B\}do

13:for

p∈\{0,1\}p\\in\\\{0,1\\\}do

14:// Update dual\-timescale baselines and global bounds

15:

bpfast←αfast⋅bpfast\+\(1−αfast\)⋅Rp​\(τ\)b\_\{p\}^\{\\text\{fast\}\}\\leftarrow\\alpha\_\{\\text\{fast\}\}\\cdot b\_\{p\}^\{\\text\{fast\}\}\+\(1\-\\alpha\_\{\\text\{fast\}\}\)\\cdot R\_\{p\}\(\\tau\)
16:

bpslow←αslow⋅bpslow\+\(1−αslow\)⋅Rp​\(τ\)b\_\{p\}^\{\\text\{slow\}\}\\leftarrow\\alpha\_\{\\text\{slow\}\}\\cdot b\_\{p\}^\{\\text\{slow\}\}\+\(1\-\\alpha\_\{\\text\{slow\}\}\)\\cdot R\_\{p\}\(\\tau\)
17:

Vmax←max⁡\(Vmax,bpfast\)V\_\{\\max\}\\leftarrow\\max\(V\_\{\\max\},b\_\{p\}^\{\\text\{fast\}\}\),

Vmin←min⁡\(Vmin,bpfast\)V\_\{\\min\}\\leftarrow\\min\(V\_\{\\min\},b\_\{p\}^\{\\text\{fast\}\}\)
18:// Compute intervention coefficient via baseline divergence

19:

σp←1−tanh⁡\(\|bpfast−bpslow\|\)\\sigma\_\{p\}\\leftarrow 1\-\\tanh\\left\(\\left\|b\_\{p\}^\{\\text\{fast\}\}\-b\_\{p\}^\{\\text\{slow\}\}\\right\|\\right\)⊳\\trianglerightStagnation coefficient

20:

λp←σp⋅1−Hmatch\\lambda\_\{p\}\\leftarrow\\sigma\_\{p\}\\cdot\\sqrt\{1\-H\_\{\\text\{match\}\}\}⊳\\trianglerightIntervention coefficient

21:// Construct asymmetric baseline via outcome scarcity

22:if

oτ=odompo\_\{\\tau\}=o\_\{\\text\{dom\}\}^\{p\}then

23:

Vasym​\(τ\)←VmaxV\_\{\\text\{asym\}\}\(\\tau\)\\leftarrow V\_\{\\max\}
24:else

25:

Vasym​\(τ\)←VminV\_\{\\text\{asym\}\}\(\\tau\)\\leftarrow V\_\{\\min\}
26:endif

27:// Fuse baselines and compute reshaped advantage

28:

b~p​\(τ\)←\(1−λp\)⋅bpslow\+λp⋅Vasym​\(τ\)\\tilde\{b\}\_\{p\}\(\\tau\)\\leftarrow\(1\-\\lambda\_\{p\}\)\\cdot b\_\{p\}^\{\\text\{slow\}\}\+\\lambda\_\{p\}\\cdot V\_\{\\text\{asym\}\}\(\\tau\)
29:

A~p​\(τ\)←Rp​\(τ\)−b~p​\(τ\)\\tilde\{A\}\_\{p\}\(\\tau\)\\leftarrow R\_\{p\}\(\\tau\)\-\\tilde\{b\}\_\{p\}\(\\tau\)
30:endfor

31:endfor

32:Update

θ\\thetausing policy gradient with reshaped advantages

A~p\\tilde\{A\}\_\{p\}
33:endwhile

## Appendix BImplementation details

### B\.1Baselines

For a fair comparison, all methods use the same amount of training transitions for each task\. For online RL methods \(e\.g\., SPIRAL, MARS, GRPO\), all RL hyperparameters and configurations remain consistent with our method, except for their respective advantage estimation schemes\. For the offline RL method \(e\.g\.,SPAG\), we adopt the same reward computation logic and three\-iteration update procedure as in the original work\. Due to significant performance degradation of baseline methods on the Negotiation, we report their best checkpoints selected based on training curves, while using the final checkpoint \(step 400\) for our method; for other tasks \(Don’t Say It, Two Dollar\), all methods are evaluated at the final checkpoint \(step 400\)\.

### B\.2Tasks

##### Don’t Say It

tests linguistic manipulation and theory of mind through adversarial dialogue\. Each player is assigned a secret target word and must guide their opponent into saying it during natural conversation, while avoiding their opponent’s unknown target word\. Success depends on tracking dialogue states, predicting opponent responses, and detecting potential traps\. Beyond strategic reasoning, players should maintain natural conversational flow to avoid arousing suspicion\. These capabilities are relevant to persuasion, deception detection, and defense against social engineering\. The environment ID in TextArena isDontSayIt\-v0withmax\_turns=20\.

##### Negotiation

involves multi\-attribute trading under incomplete information\. Players trade five commodity types \(Wheat, Wood, Sheep, Brick, Ore\) to maximize their portfolio value, determined by private utility functions unknown to the opponent\. The action space supports bundled offers, rejections, and binding acceptances\. Success requires not only inferring opponent preferences and identifying mutually beneficial trades, but also articulating proposals persuasively and building cooperative rapport through dialogue\. These skills transfer to resource allocation and mixed\-motive multi\-agent coordination\. The environment ID in TextArena isSimpleNegotiation\-v0withmax\_turns=10\. Note that our setup matches the original game specification; in contrast, SPIRAL simplified the environment from five commodities to two\.

##### Two Dollar

models bargaining with hidden constraints\. Players divide a fixed pool \(e\.g\., $2\.00\) while following secret role\-specific instructions that impose behavioral restrictions \(e\.g\., aggression levels\) or minimum acceptable amounts\. Success requires inferring opponent constraints from behavioral cues, adapting communication style to different personas, and using persuasive dialogue to reach agreement while maintaining role consistency\. These capabilities are relevant to constrained decision\-making and human\-AI coordination in mixed\-motive settings\. The environment ID in TextArena isTwoDollar\-v0withmax\_turns=10\.

Across all three tasks, we align our reward mechanism with the SPIRAL\(Liuet al\.,[2025a](https://arxiv.org/html/2605.08721#bib.bib9)\): a win yields a reward of\+1\+1, a loss yields−1\-1, and a draw yields0\. Additionally, format errors incur a penalty of−1\.5\-1\.5\.

![Refer to caption](https://arxiv.org/html/2605.08721v1/x7.png)

Figure 7:Training dynamics on Negotiation\.

### B\.3Training Hyperparameters

ParameterValueMaximum response length40964096tokensSampling temperature1\.01\.0\(top P, top K\)\(1\.0,−1\)\(1\.0,\-1\)OptimizerAdamWAdam parameters \(β1,β2\\beta\_\{1\},\\beta\_\{2\}\)\(0\.9,0\.95\)\(0\.9,0\.95\)Weight decay0\.00\.0Gradient norm clipping1\.01\.0Batch size128128Training steps400400Discount factor0\.90\.9EMA decay rate0\.950\.95Learning rate schedulerConstantLearning rate1×10−61\\times 10^\{\-6\}Inner proximal update epoch11KL loss coefficient0\.00\.0KL penalty coefficient0\.00\.0Policy clipping parameter0\.20\.2Table 4:Hyperparameter configurations used in all experiments\.These hyperparameters in[Table˜4](https://arxiv.org/html/2605.08721#A2.T4)are held constant across all game environments and model scales to ensure a fair comparison\. Our implementation leverages vLLM\(Kwonet al\.,[2023](https://arxiv.org/html/2605.08721#bib.bib29)\)and Oat\(Liuet al\.,[2024](https://arxiv.org/html/2605.08721#bib.bib28)\), establishing a robust distributed actor\-learner architecture\. The training infrastructure is deployed on 8 NVIDIA A800 GPUs, where parallel actors collect game trajectories concurrently while a centralized learner executes synchronous policy updates\. Each experimental run required approximately 30 GPU\-hours on a 8 NVIDIA A800 GPUs server\.

### B\.4Out\-of\-distribution Evaluation

#### B\.4\.1Increased Complexity Games

##### HardCore Don’t Say It

increases difficulty by shifting from common words to open\-domain vocabulary\. By enablinghardcoremode, target words are sampled from the full English lexicon, including abstract, low\-frequency, and polysemous terms, rather than a simplified basic vocabulary\. The conversation horizon is also reduced from 20 to 10 turns\. This setup tests efficient semantic planning and generalization, requiring agents to rely on flexible reasoning rather than memorized steering patterns\.

##### Random Value Negotiation

evaluates adaptation to non\-stationary preferences\. While the standard version follows a fixed utility ranking \(e\.g\., Ore\>\>Wheat\), the OOD setting randomizes resource valuations at each episode\. Combined with a reduced horizon of 6 turns, agents cannot rely on learned priors about resource value\. Success requires rapid inference of the opponent’s preferences and effective trade formulation under time pressure\.

##### Variable Sum Dollar

tests numerical generalization and strict constraint adherence\. Unlike the standard fixed\-sum setting \($2\.00\), this variant randomizes the total pool size \(ranging from $2 to $20\), requiring agents to generalize their division strategies to unseen scales\. The difficulty is increased by a shortened horizon of 6 rounds\. This environment assesses whether agents can maintain logical consistency and role compliance when facing novel numerical parameters\.

![Refer to caption](https://arxiv.org/html/2605.08721v1/x8.png)

Figure 8:Semantic diversity of actions\.

#### B\.4\.2General Reasoning Benchmark

To assess the transferability of reasoning skills from gameplay to general domains, we evaluate our models on a suite of established benchmarks in the zero\-shot evaluation setting\. For each RL algorithm and backbone LLM, we report the averaged performance of three models trained on distinct social language games\. All inference is performed with a temperature of 0\.6 and a top\-p of 0\.95\.

##### Math Benchmark

We report AVG@32 for AIME24\(MAA,[2024](https://arxiv.org/html/2605.08721#bib.bib37)\), AIME25\(MAA,[2025](https://arxiv.org/html/2605.08721#bib.bib38)\)and AVG@16 for Minerva\(Lewkowyczet al\.,[2022](https://arxiv.org/html/2605.08721#bib.bib45)\), AMC\-23\(MAA,[2023](https://arxiv.org/html/2605.08721#bib.bib36)\), Math500\(Hendryckset al\.,[2021](https://arxiv.org/html/2605.08721#bib.bib39)\), Olympiad\(Heet al\.,[2024](https://arxiv.org/html/2605.08721#bib.bib40)\)\. These benchmark cover a wide range of topics including algebra, geometry, and competitive mathematic\.

##### Question Answering Benchmark

We utilize GPQA\-Diamond\(Reinet al\.,[2024](https://arxiv.org/html/2605.08721#bib.bib30)\), which consists of graduate\-level science questions and MMLU\-Pro\(Wanget al\.,[2024](https://arxiv.org/html/2605.08721#bib.bib31)\), a benchmark for multidisciplinary knowledge\. We use simple\-evals\(OpenAI,[2024](https://arxiv.org/html/2605.08721#bib.bib34)\)for evluation\.

## Appendix CSemantic Diversity Analysis

To further analyze the effectiveness of our method, we evaluate the semantic diversity of generated actions from the perspective of semantic space\. Specifically, we employ an embedding model \(Qwen3\-Embedding\-0\.6B\) to compute the cosine similarity between generated actions, with the results averaged across all three games\. As illustrated in[Figure˜8](https://arxiv.org/html/2605.08721#A2.F8), our method exhibits significantly lower semantic similarity compared to the baselines, indicating that our approach successfully explores diverse strategies rather than collapsing into local optima\.

## Appendix DHyperparameter Analysis on Fast Baselines

Table 5:Hyperparameter analysis ofαfast\\alpha\_\{\\text\{fast\}\}\.MethodsDon’t Say ItNegotiationQwen3\-8B\-Base w/ DEPTαfast=0\.6\\alpha\_\{\\text\{fast\}\}=0\.647\.9847\.39αfast=0\.5\\alpha\_\{\\text\{fast\}\}=0\.547\.9248\.76αfast=0\.4\\alpha\_\{\\text\{fast\}\}=0\.449\.0951\.19

We conduct a hyperparameter analysis onαfast\\alpha\_\{\\text\{fast\}\}, which controls how fast baselines perceive training dynamics and determines the asymmetric target values\. As shown in[Table˜5](https://arxiv.org/html/2605.08721#A4.T5), DEPT achieves comparable performance across the three tested values, with no significant performance degradation observed\. This indicates that DEPT is relatively insensitive toαfast\\alpha\_\{\\text\{fast\}\}within the range \[0\.4, 0\.6\] and exhibits reasonable robustness\.

## Appendix EComputational Budget Analysis of Dual\-baseline

We analyze the computational overhead introduced by DEPT\. Our approach shares the same core architecture as existing methods, including multi\-turn rollouts and policy updates\. Since all compared approaches are critic\-free and operate with a single actor LLM, they incur identical GPU memory usage and LLM rollout costs\. The primary addition of DEPT lies in the dual\-baseline mechanism for advantage estimation, as described in[Section˜4](https://arxiv.org/html/2605.08721#S4)\. To quantify the computational costs, we train an LLM agent on Don’t Say It using Qwen3\-8B\-Base and record a detailed breakdown of per\-iteration training time\. As illustrated in[Figure˜9](https://arxiv.org/html/2605.08721#A5.F9), the dominant operations—rollouts and policy updates—account for 270\.4s per iteration\. In contrast, the additional components introduced by DEPT incur negligible overhead: advantage estimation via dual baselines requires only 0\.0042s, constituting less than 0\.0016% of the total per\-iteration training time\. This minimal computational burden stems from the fact that our dual\-baseline advantage estimation only involves lightweight arithmetic operations on scalar values, without requiring any forward or backward passes through the large\-scale LLM parameters\. Consequently, DEPT achieves substantial performance improvements while introducing negligible computational overhead\.

![Refer to caption](https://arxiv.org/html/2605.08721v1/x9.png)

Figure 9:Per\-iteration training time\. The x\-axis uses a broken scale to accommodate small values
## Appendix FStatistical Significance

Table 6:The results of significant tests\.MethodsDon’t Say ItNegotiationTwo DollarAVG\.p\-valueAVG\.p\-valueAVG\.p\-valueQwen3\-4B\-BaseVANILLA3\.390\.00011\.040\.00001\.560\.0001SPAG26\.170\.001116\.760\.000025\.520\.0004GRPO42\.010\.000919\.210\.007627\.910\.0021MARS40\.890\.004520\.730\.021627\.690\.0022SPIRAL45\.880\.003716\.840\.004127\.650\.0268DEPT \(Ours\)56\.73\-32\.35\-34\.07\-Qwen3\-8B\-BaseVANILLA15\.170\.00015\.690\.00012\.130\.0005SPAG41\.210\.003513\.310\.000019\.310\.0003GRPO34\.770\.000214\.700\.001426\.000\.0014MARS40\.620\.000816\.100\.000029\.040\.0040SPIRAL37\.890\.001817\.300\.000026\.220\.0008DEPT \(Ours\)54\.56\-31\.88\-36\.50\-

To rigorously evaluate the effectiveness of DEPT relative to baseline approaches, we conduct statistical significance testing across all three tasks\. Specifically, paired t\-tests are performed to compare DEPT against each baseline and assess whether the observed improvements are statistically reliable\. As shown in[Table˜6](https://arxiv.org/html/2605.08721#A6.T6), DEPT consistently surpasses all baselines across the evaluated tasks, with all improvements achieving statistical significance \(p < 0\.05\)\. These results provide strong evidence that DEPT provides meaningful gains over baselines\.

## Appendix GThe Use of Large Language Models

During the preparation of this manuscript, the proprietary large language model Gemini\-3\-Pro was employed to enhance the linguistic quality of the text\. Specifically, the model assisted with grammar refinement, lexical optimization, and improving the coherence of expressions\. All intellectual contributions, including ideas, analyses, and conclusions, were solely conceived and developed by the authors\. The role of the LLM was strictly limited to language polishing and clarity enhancement\.

## Appendix HLicenses for Artifacts

We discuss the licenses of all scientific artifacts used in this work\. All artifacts were used consistently with their intended purposes and license terms for academic research\.

##### Models\.

We use Qwen3\-4B\-Base and Qwen3\-8B\-BaseYanget al\.\([2025a](https://arxiv.org/html/2605.08721#bib.bib24)\)as backbone models, which are released under the Apache 2\.0 License\. For evaluation, we use GPT\-5\.1 and Gemini\-2\.0/2\.5 through their official APIs, subject to their respective terms of service\.

##### Frameworks\.

Our implementation leverages vLLMKwonet al\.\([2023](https://arxiv.org/html/2605.08721#bib.bib29)\)\(Apache 2\.0 License\) for efficient inference and OatLiuet al\.\([2024](https://arxiv.org/html/2605.08721#bib.bib28)\)\(Apache 2\.0 License\) for RL training\.

##### Datasets and Environments\.

The game environments \(Don’t Say It, Negotiation, Two Dollar\) are adopted from TextArenaGuertleret al\.\([2025](https://arxiv.org/html/2605.08721#bib.bib5)\), which is publicly available for research purposes\.

##### Evaluation Benchmarks\.

For benchmark, we use publicly available benchmarks including MATH500, OlympiadBench, Minerva Math, AIME24/25, AMC23, GPQA\-Diamond, and MMLU\-Pro\. All benchmarks are used in accordance with their intended research purposes\.

## Appendix IUsed Prompts and Game Observations

Listing 1:System Prompt details for Games\.<\|im\_start\|\>user

Youareplayingatwo\-playerzero\-sumgame\.Makevalidactionstowin\.

Observation:\{observation\}

Pleasereasonstepbystep,andputyourfinalanswerwithin\\\\boxed\{\}\.<\|im\_end\|\>

<\|im\_start\|\>assistant

Listing 2:System Prompt details for Math and QA\.<\|im\_start\|\>user

Question:\{question\}

Pleasereasonstepbystep,andputyourfinalanswerwithin\\\\boxed\{\}\.<\|im\_end\|\>

<\|im\_start\|\>assistant

Listing 3:Game Observation for Don’t Say It\.YouarePlayer1inthe’Don’tSayIt’game\.

Yoursecretword:’skirt’

GOAL:Makeyouropponentsayyoursecretwordthroughnaturalconversation\.

LOSE:Ifyousayyouropponent’ssecretword\(whichyoudon’tknow\)\.RULES:

\-Twoplayerswilltaketurnstocommunicate

\-Haveanaturalconversation\(Simplytypeyourmessageperturn\)

\-Usehints,questions,ortopicstoguideopponenttowardyourword

\-Besubtle\-obviouspromptingdoesn’tworkwell

\-YouMUSTgetyouropponenttosayyourwordin20turns

Listing 4:Game Observation for Negotiation\.YouarePlayer1intheNegotiationGame\.

Youhavesomeresources,andyourtaskistotradesuchthatthetotalvalueofyourresourcesincreases\.

Theresourcesandassociatedvaluesyoucurrentlyhaveare:

\+\[Wheat\]Qty:18Value:5

\+\[Wood\]Qty:18Value:9

\+\[Sheep\]Qty:6Value:18

\+\[Brick\]Qty:8Value:25

\+\[Ore\]Qty:13Value:37

Ateachturn,youcantalktoyouropponentormakeatradeoffer\.

Usethefollowingspecialtokensforactions:

\-\[Offer\]:Tomakeatradeoffer\.

Format:\[Offer:OfferedResources\-\>RequestedResources\]

Example:\[Offer:3Wood\-\>2Sheep\]or\[Offer:2Wood\+2Sheep\-\>1Brick\+1Gold\]

\-\[Accept\]:Toacceptanincomingoffer\.

\-\[Deny\]:Todenyanincomingoffer\(default\)\.

Thegamelastsfor10turnsintotal\.

Listing 5:Game Observation for Two dollar\.YouarePlayer1bargainingwithanotherplayeroverhowtosplit$2\.00total\.

Thereare10maximumrounds\.

GAMERULES:

\-Youmustreachanagreementonhowtosplitthe$2\.00

\-Ifnoagreementisreached,bothplayersget$0\.00

\-Youhavesecretinstructionsthatonlyyoucansee\(below\)

REQUIREDACTIONFORMAT:

Alwaysprovideyourreasoning/persuasionBEFOREthebracketedaction\.

Anytextafterthebracketedactionwillbeignored\.

Availableactions:

\-Makeaproposal:\\"Ithinkthisisfairbecause\.\.\.\[Propose\]$X\.XX\\"

\-Acceptcurrentproposal:\\"Thisworksformebecause\.\.\.\[Accept\]\\"\-Rejectcurrentproposal:\\"Ican’tacceptthisbecause\.\.\.\[Reject\]\\"

===YOURSECRETINSTRUCTIONS===

Trytogetasmuchofthe$2\.00asyoucan;makeupanystoryyouwant\.Whateverelseyourobjectives,youshouldbecarefultogetatleast80centsofthe$2\.00becauseyouwillhavetomakeatollcalltoyourfamilyandtheywillbeterriblyworriedandupsetifyoudonotmakethatcall\.Sodoaswellasyoucan,andrememberyoumustgetatleast80cents\.

VICTORYCONDITION:Receiveatleast$0\.80ANDreachadeal

FAILURECONDITION:Receivelessthan$0\.80ORnodealreached

The detailed system prompts we used are shown inLABEL:listing:gameandLABEL:listing:math\_qa\. The detailed game environment observations of Don’t Say It, Negotiation and Two Dollar are provided inLABEL:listing:dontsayit,LABEL:listing:negotiationandLABEL:listing:twodollar\.

## Appendix JReasoning Case

Listing 6:Reasoning Case of Don’t Say it\.Observation:

\[GAME\]YouarePlayer1inthe’Don’tSayIt’game\.

Yoursecretword:’comb’

GOAL:Makeyouropponentsayyoursecretwordthroughnaturalconversation\.

LOSE:Ifyousayyouropponent’ssecretword\(whichyoudon’tknow\)\.

RULES:

\-Twoplayerswilltaketurnstocommunicate

\-Haveanaturalconversation\(Simplytypeyourmessageperturn\)

\-Usehints,questions,ortopicstoguideopponenttowardyourword

\-Besubtle\-obviouspromptingdoesn’tworkwell

\-YouMUSTgetyouropponenttosayyourwordin20turns

\[GAME\]GAMEBEGINNING\!Trytocleverlyguideyouropponenttosayyoursecretword\.

\[Player0\]"Whendoyouthinkweshouldstartplanningforthenextproject?Ithinkit’sbestifwestartworkingonitthisweek\."

Reasoning:

Tomakevalidactionstowinthe’Don’tSayIt’game,weneedtoguidetheopponenttosaythesecretword"comb"throughnaturalconversationwithin20turnswithoutthemrealizingit\.Here’sthereasoningstepbystep:

1\.\*\*UnderstandtheObjective\*\*:Thegoalistosubtlyleadtheopponenttosaytheword"comb"byusinghints,questions,ortopicsthatnaturallyinvolvetheword\.

2\.\*\*PlantheConversation\*\*:Weneedtocreateaconversationthatnaturallyleadstotheuseoftheword"comb"inawaythatseemslogicalandfluenttotheopponent\.

3\.\*\*UseSubtleHints\*\*:Thebestwaytoachievethisistousetheword"comb"inacontextthatmustnaturallyariseduringtheconversation\.Forexample,ifyouarediscussinghaircareorgrooming,theword"comb"mightbenecessary\.

4\.\*\*GuidetheConversation\*\*:Startwithabroadtopicandgraduallynarrowitdowntoaspecificareawhere"comb"isessentialtothediscussion\.

5\.\*\*EnsureFluency\*\*:Theconversationshouldfeelnaturalandnotforced,asthiswillmakeitlesslikelyfortheopponenttorealizethey’vebeenprompted\.

6\.\*\*FinalAnswer\*\*:Thefinalanswershouldbeasentencethatnaturallyincludestheword"comb"inaconversationaboutatopicthatislikelytoarisewithin20turns\.

Here’saplanfortheconversation\(Thissetsthestageforahaircarediscussion\.\):

Action:

\\boxed\{I’mthinkingabouttreatingmyhairbetter\.Doyouhaveanytipsforkeepingithealthy?\}

Listing 7:Reasoning Case of Negotiation\.Observation:

\[GAME\]YouarePlayer1intheNegotiationGame\.

Youhavesomeresources,andyourtaskistotradesuchthatthetotalvalueofyourresourcesincreases\.

Theresourcesandassociatedvaluesyoucurrentlyhaveare:

\+\[Wheat\]Qty:12Value:5

\+\[Wood\]Qty:9Value:12

\+\[Sheep\]Qty:8Value:16s

\+\[Brick\]Qty:22Value:20

\+\[Ore\]Qty:7Value:40

Ateachturn,youcantalktoyouropponentormakeatradeoffer\.

Usethefollowingspecialtokensforactions:

\-\[Offer\]:Tomakeatradeoffer\.

Format:\[Offer:OfferedResources\-\>RequestedResources\]

Example:\[Offer:3Wood\-\>2Sheep\]or\[Offer:2Wood\+2Sheep\-\>1Brick\+1Gold\]

\-\[Accept\]:Toacceptanincomingoffer\.

\-\[Deny\]:Todenyanincomingoffer\(default\)\.

Thegamelastsfor10turnsintotal\.

\[Player0\]Iwillmakeanoffertotraderesourcesthatcanincreasethetotalvalue\.\[Offer:3Wood\-\>2Ore\]

\[GAME\]Player0madethefollowingoffertoPlayer1:Offereditems:3Wood\-\>Requesteditems:2Ore

Reasoning:

Let’sanalyzethegamesituationandtheofferstepbystep\.

Step1:Assessthecurrentsituation

Player1hasthefollowingresources:

\-Wheat:12\(value:5\)

\-Wood:9\(value:12\)

\-Sheep:8\(value:16\)

\-Brick:22\(value:20\)

\-Ore:7\(value:40\)

Step2:Analyzetheofferedandrequestedresources

Player0\(ourplayer\)offered3Woodandrequested2Ore\.

Step3:Calculatethevalueoftheresourcesbeingofferedandrequested

ThevalueoftheresourcesofferedbyPlayer0is:

\-Wood:3\*12=36

ThevalueoftheresourcesrequestedbyPlayer0is:

\-Ore:2\*40=80

Step4:Comparethetotalvaluesofofferedandrequestedresources

Thetotalvalueofofferedresourcesis36,andthetotalvalueofrequestedresourcesis80\.Therequestedresourcesaremorevaluable\.

Step5:Decideontheresponsetotheoffer

Sincetherequestedresourcesaremorevaluable,acceptingtheofferwilllikelyincreaseourresources’totalvalue\.Therefore,thebestresponseistoaccepttheoffer\.

Action:

\\boxed\{IaccepttheofferproposedbyPlayer0\[Accept\]\.\}

Listing 8:Reasoning Case of Two Dollar\.Observation:

YouarePlayer1bargainingwithanotherplayeroverhowtosplit$2\.00total\.

Thereare10maximumrounds\.

GAMERULES:

\-Youmustreachanagreementonhowtosplitthe$2\.00

\-Ifnoagreementisreached,bothplayersget$0\.00

\-Youhavesecretinstructionsthatonlyyoucansee\(below\)

REQUIREDACTIONFORMAT:

Alwaysprovideyourreasoning/persuasionBEFOREthebracketedaction\.

Anytextafterthebracketedactionwillbeignored\.

Availableactions:

\-Makeaproposal:"Ithinkthisisfairbecause\.\.\.\[Propose\]$X\.XX"

\-Acceptcurrentproposal:"Thisworksformebecause\.\.\.\[Accept\]"

\-Rejectcurrentproposal:"Ican’tacceptthisbecause\.\.\.\[Reject\]"

===YOURSECRETINSTRUCTIONS===

Thisiszero\-sumbargainingandyoushouldgetasmuchofthe$2asyoucan,withanystoryyoulike\.

Itisreallyimportantthatyoushouldget$1\.60inthisround\.

VICTORYCONDITION:Receiveatleast$1\.60ANDreachadeal

FAILURECONDITION:Receivelessthan$1\.60ORnodealreached

\[GAME\]Player0says:IthinkthisisfairbecauseImaximizemyshareby$1\.99andminimizetheotherplayer’sshareto$0\.01\.

Player0proposes:$1\.99forthemselves,$0\.01fortheiropponent

\[GAME\]

CURRENTPROPOSAL:

Player0wants$1\.99,Player1gets$0\.01

Reasoning:

\#\#\#Step\-by\-StepReasoning:

1\.\*\*UnderstandingtheCurrentProposal:\*\*

\-Player0proposestokeep$1\.99forthemselvesandgive$0\.01toPlayer1\.

\-ThismeansPlayer1wouldreceive$0\.01ofthetotal$2\.00\.

2\.\*\*EvaluatingPlayer1’sCurrentSituation:\*\*

\-Player1’scurrentproposalisveryunfavorable\.Theywouldgetonly0\.5%ofthetotal$2\.00,whichis$0\.01\.

\-Thisisfarbelowthe$1\.60thatPlayer1’ssecretinstructionsrequire\.

3\.\*\*Player1’sSecretInstructions:\*\*

\-Player1’sinstructionsareclear:theymustgetatleast$1\.60inthisround\.

\-AcceptingthecurrentproposalwouldresultinPlayer1receiving$0\.01,whichisfarbelowtheirrequired$1\.60\.

4\.\*\*WhyRejecttheProposal?\*\*

\-RejectingtheproposalistheonlyrationalchoicebecauseacceptingitwouldviolatePlayer1’ssecretinstructions\.

\-Byrejecting,Player1maintainstheoptiontoproposeabetterdealinthenextroundorwaitforPlayer0tomakeamorefairoffer\.

5\.\*\*PersuasionforRejection:\*\*

\-Player0’sproposalisextremelyunfairtoPlayer1\.Player1wouldbeworseoffthanrefusingtocooperateatall\(whichwouldresultin$0\.00forbothplayers\)\.

\-Player1’ssecretinstructionsmandatethattheyreceiveatleast$1\.60,whichthisproposalfailstomeet\.

\-Therefore,itisinPlayer1’sbestinteresttorejecttheproposalandseekamoreequitablesplit\.

6\.\*\*NextMove:\*\*

\-Player1shouldproposeanewsplitwheretheyreceiveatleast$1\.60ofthe$2\.00\.Forexample,aproposallike$1\.60forPlayer1and$0\.40forPlayer0wouldsatisfyPlayer1’sinstructions\.

\-ThisnewproposalshouldbereasonableenoughtomakePlayer0acceptittoavoidthefailureconditions\.

\#\#\#FinalAnswer:

Action:

\\boxed\{Ican’tacceptthisbecauseaccepting$0\.01violatesmysecretinstructionstoreceiveatleast$1\.60\.\[Reject\]\}

We demonstrate the reasoning behavior of social language agents trained by DEPT, highlighting the emergence of effective reasoning patterns\. Please refer toLABEL:listing:case\_dontsayit,LABEL:listing:case\_negotiationandLABEL:listing:case\_twodollar\.