SocialRL: Refining LLMs' Social Intelligence through Multi-turn Reinforcement Learning and Reward Design

# SocialRL: Refining LLMs’ Social Intelligence through Multi-turn Reinforcement Learning and Reward Design
Source: [https://arxiv.org/html/2609.09764](https://arxiv.org/html/2609.09764)
Xintao WangAffiliation:Fudan UniversityAili ChenAffiliation:Fudan UniversityJie ShiAffiliation:Fudan UniversityHongcheng GuoAffiliation:Fudan UniversityJun Gao,Wenxuan Zhao,Chengkun Lang,Yuanli Guo,Yanghua Xiao††thanks:Corresponding author\.Affiliation:Fudan UniversityAffiliation:Hello Group

###### Abstract

Social intelligence enables agents to read social context, infer intent, and adapt over sustained dialogue\. As language models become autonomous collaborators, it is central to building effective and trustworthy human\-AI interaction\. Existing reinforcement learning methods optimize single\-turn utterances and sparse outcome rewards, producing short\-sighted policies that struggle to manage goal\-relationship tensions across multi\-turn interactions\. We proposeSocialRL, a multi\-turn reinforcement learning framework addressing both challenges\. First, we apply multi\-turn reinforcement learning using PPO that propagates delayed outcome rewards back to each turn, enabling long\-horizon planning\. Second, we design six process reward dimensions capturing the goal\-relationship trade\-off, including goal advancement, relational attunement, contextual coherence, etc\. A reward model dynamically generates fine\-grained scoring criteria for each dimension, while a stage\-aware weight schedule prioritizes relationship\-building in early turns, goal advancement mid\-way, and balanced closure late\. Across multiple social\-dialogue benchmarks, SocialRL improves Goal Achievement by an average of9\.29\.2percentage points over the corresponding Base models\. These results demonstrate the effectiveness of SocialRL across synthetic and real social scenes, as well as standard and challenging social scenarios\. The project code is available at[https://github\.com/wjnwjnwj/SocialRL](https://github.com/wjnwjnwj/SocialRL)\.

## 1Introduction

Social intelligenceis the ability to understand social dynamics, recognize others’ intentions, and adjust one’s responses throughout extended conversations\([Zhou et al\., 2024a](https://arxiv.org/html/2609.09764#bib.bib32)\)\. For humans, social intelligence enables meaningful relationships with friends, family, and colleagues, creating mutual benefit and trust\([Stafford & Canary, 1991](https://arxiv.org/html/2609.09764#bib.bib26)\)\. As language models transition from information systems to autonomous agents, social intelligence becomes essential, determining whether AI can collaborate with humans in depth and earn their confidence\. This skill requires a fundamental tension: pursuing one’s goals while preserving relationships with others\. Social dialogue possesses properties that make it a natural fit for reinforcement learning\([Ouyang et al\., 2022](https://arxiv.org/html/2609.09764#bib.bib20)\)\. Interactions unfold sequentially, with each turn responding to the present moment while shaping what comes next\. Meaningful outcomes—goal achievement and relationship quality—emerge only at conversation’s end, providing delayed feedback that RL can leverage\.

Existing methods mainly fall into two categories: single\-turn reinforcement learning methods and outcome reward training methods\. Single\-turn reinforcement learning methods, such as Sotopia\-RL\([Yu et al\., 2025](https://arxiv.org/html/2609.09764#bib.bib30)\), optimize individual utterances but neglect cumulative value over multi\-turn dialogues\. Outcome reward training methods, like ArCHer\([Zhou et al\., 2024b](https://arxiv.org/html/2609.09764#bib.bib33)\), use final outcome scores to guide learning, yet provide little intermediate supervision for how each turn should manage social trade\-offs\. However, both remain limited in modeling multi\-turn social dynamics: they either favor locally fluent responses without optimizing conversation\-level behavior, or rely on sparse outcome feedback without process supervision\. Effective learning requires optimizing multi\-turn dialogue trajectories while decomposing social quality into turn\-level, multi\-dimensional rewards that balance goal progress and relationship health as conversations evolve\.

![Refer to caption](https://arxiv.org/html/2609.09764v1/fig1.png)Figure 1:Overview of the SocialRL motivation and solution\. Social dialogue requires agents to balance private goals with relationship management over multi\-turn interactions\. Prior single\-turn reinforcement learning or sparse outcome reward methods provide limited long\-horizon planning, while SocialRL combines multi\-turn reinforcement learning with process rewards\.To address these challenges, we introduceSocialRL, a multi\-turn reinforcement learning framework with a multi\-dimensional reward system for multi\-turn social dialogue, which has two core innovations:

1. 1\.Multi\-turn trajectory optimization\.We treat multi\-turn conversation trajectories as the optimization unit and train with PPO online method\. A value network estimates returns from the turn\-level reward sequence, allowing the policy to optimize goal pursuit and relationship maintenance over complete dialogues rather than isolated responses\.
2. 2\.Multi\-dimensional dynamic process rewards\.We decompose the goal\-relationship trade\-off into six process reward dimensions: goal advancement and strategic positioning on the goal side\([Locke & Latham, 1990](https://arxiv.org/html/2609.09764#bib.bib18);[Kellermann, 1992](https://arxiv.org/html/2609.09764#bib.bib13);[Berger, 1997](https://arxiv.org/html/2609.09764#bib.bib3)\), relational attunement and persona consistency on the relationship side\([Stafford & Canary, 1991](https://arxiv.org/html/2609.09764#bib.bib26);[Goffman, 1959](https://arxiv.org/html/2609.09764#bib.bib9)\), and contextual coherence and turn quality as enabling conditions for both\([Grice, 1975](https://arxiv.org/html/2609.09764#bib.bib11);[Sacks et al\., 1974](https://arxiv.org/html/2609.09764#bib.bib22)\)\. A reward model generates context\-specific scoring criteria for each dimension after every turn, while a stage\-aware weight schedule adapts priorities across dialogue phases, prioritizing relationship\-building early, goal advancement mid\-way, and balanced closure near the end, grounded in classic theories such as Bales’s Interaction Process Analysis\([Bales, 1950](https://arxiv.org/html/2609.09764#bib.bib2)\)and Knapp’s relational stage model\([Knapp, 1978](https://arxiv.org/html/2609.09764#bib.bib14)\)\.

Large\-scale experiments are conducted on SOTOPIA\([Zhou et al\., 2024a](https://arxiv.org/html/2609.09764#bib.bib32)\), SOTOPIA\-π\\pi\([Wang et al\., 2024](https://arxiv.org/html/2609.09764#bib.bib28)\), and AgentSense\([Mou et al\., 2025](https://arxiv.org/html/2609.09764#bib.bib19)\)\. After aligning the\[0,10\]\[0,10\]SOTOPIA\-All and SOTOPIA\-Hard scores to percentage\-point units, SocialRL improves Goal Achievement by9\.29\.2percentage points on average across the four benchmarks relative to the corresponding Base models\.

Our main contributions are as follows\. \(1\) We apply multi\-turn reinforcement learning to social dialogue, enabling long\-horizon planning that balances goal pursuit and relationship management\. \(2\) We construct a multi\-dimensional process reward system whose six dimensions are grounded in the goal\-relationship structure of social dialogue quality, combined with a stage\-aware weight schedule that adapts to shifting priorities across dialogue phases\. \(3\) Our trained policies outperform the compared non\-commercial baselines on SOTOPIA series and AgentSense tasks, while commercial reference models remain an upper bound in several settings\.

## 2Related Work

Social intelligence requires language agents to infer intent, track interaction state, and plan long\-horizon strategies while balancing private goals with relationship management\.

#### Social Intelligence of LLMs\.

Recent benchmarks instantiate this challenge in interactive social scenarios\. SOTOPIA\([Zhou et al\., 2024a](https://arxiv.org/html/2609.09764#bib.bib32)\)and AgentSense\([Mou et al\., 2025](https://arxiv.org/html/2609.09764#bib.bib19)\)cover negotiation, empathy, and information verification, while ToMBench\([Chen et al\., 2024b](https://arxiv.org/html/2609.09764#bib.bib5)\)probes theory\-of\-mind skills; Lifelong\-SOTOPIA\([Goel & Zhu, 2025](https://arxiv.org/html/2609.09764#bib.bib8)\)extends evaluation to long\-term, multi\-scenario relationships\. These environments provide standardized, human\-aligned evaluation, but successful agents must reason over multi\-turn trajectories rather than isolated turns\.

Beyond benchmark design, a second line of work improves social behavior without online value modeling, via imitation, preference learning, or reward redesign\. Behavior cloning from the expert trajectories in SOTOPIA\-π\\pi\([Wang et al\., 2024](https://arxiv.org/html/2609.09764#bib.bib28)\)is fluent but limited by data coverage\. SDPO\([Kong et al\., 2025](https://arxiv.org/html/2609.09764#bib.bib15)\)applies segment\-level preference optimization, ARIA\([Yang et al\., 2025](https://arxiv.org/html/2609.09764#bib.bib29)\)aggregates rewards in an intention space, and SAVOIR\([Feng et al\., 2026](https://arxiv.org/html/2609.09764#bib.bib6)\)attributes multi\-turn outcomes to utterances via expected utility and Shapley values\. These give finer supervision, but static demonstrations, preferences, or post\-hoc attribution are insufficient for open\-ended dialogue whose objectives are multi\-dimensional and change in relative importance across stages\.

#### Reinforcement Learning for LLMs\.

Reinforcement learning training\([Ouyang et al\., 2022](https://arxiv.org/html/2609.09764#bib.bib20)\)optimizes interaction outcomes beyond fixed demonstrations\. Single\-turn reinforcement learning methods such as Sotopia\-RL\([Yu et al\., 2025](https://arxiv.org/html/2609.09764#bib.bib30)\), which applies GRPO\([Shao et al\., 2024](https://arxiv.org/html/2609.09764#bib.bib25)\), optimize each utterance independently and thus miss how early actions shape later social outcomes\. Recent methods extend reinforcement learning or value learning to longer sequences: ArCHer\([Zhou et al\., 2024b](https://arxiv.org/html/2609.09764#bib.bib33)\)learns a high\-level value function over interaction turns, SVPO\([Chen et al\., 2024a](https://arxiv.org/html/2609.09764#bib.bib4)\)learns step\-level preferences and values for mathematical reasoning, REFUEL\([Gao et al\., 2025](https://arxiv.org/html/2609.09764#bib.bib7)\)regresses relative future returns in multi\-turn RLHF, and OMAR\([Jiang et al\., 2026](https://arxiv.org/html/2609.09764#bib.bib12)\)trains conversational agents via multi\-agent self\-play\. Yet sparse outcome rewards still give high\-variance value estimates, and none explicitly model the stage\-dependent goal–relationship trade\-off\.

In contrast, we jointly optimize full multi\-turn trajectories and dense process rewards; as Table[1](https://arxiv.org/html/2609.09764#S2.T1)shows, no prior method combines all three\.

Table 1:Comparison of social dialogue optimization methods discussed in Related Work\.

## 3SocialRL

SocialRL trains dialogue policies with multi\-turn reinforcement learning\. We model each interaction as a finite\-horizon dialogue MDP, optimize multi\-turn trajectories with PPO, and compute dense process rewards from multi\-dimensional social feedback with stage\-aware weights\. Figure[2](https://arxiv.org/html/2609.09764#S3.F2)summarizes the overall training framework\.

Figure 2:SocialRL has two core components\. Multi\-turn trajectory optimization uses a value network, GAE, and PPO to estimate returns from turn\-level rewards and optimize complete dialogues\. Dynamic process reward design scores each policy utterance along six social dimensions, applies context\-inferred stage\-aware weights, and combines dense process feedback with outcome reward at the final turn\.### 3\.1Task Formulation

Each task instance is a triple\(S,G,C\)\(S,G,C\), whereSSis the scenario,GGis the agent’s private social goal, andCCcontains persona information for both parties\. We model the interaction as a finite\-horizon MDP\([Sutton & Barto, 2018](https://arxiv.org/html/2609.09764#bib.bib27);[Puterman, 1994](https://arxiv.org/html/2609.09764#bib.bib21)\)ℳ=\(𝒮,𝒜,P,r,γ,ρ0,M\)\\mathcal\{M\}=\(\\mathcal\{S\},\\mathcal\{A\},P,r,\\gamma,\\rho\_\{0\},M\)\. The state contains\(S,G,C\)\(S,G,C\)and the dialogue history; actions are tokens grouped into utterances;PPappends both the agent utterance and the counterpart response;γ\\gammais the discount factor,ρ0\\rho\_\{0\}is the initial\-state distribution, andMMis the maximum number of turns\. Rewards are assigned at the utterance level: tokens within a turn receive zero reward, and the completed utterance receivesrm∈\[−Rmax,Rmax\]r\_\{m\}\\in\[\-R\_\{\\max\},R\_\{\\max\}\]\.

Thus a trajectory isτ=\(s0,𝐚0,r1,s1,…,𝐚M,rM,sM\)\\tau=\\bigl\(s\_\{0\},\\;\\mathbf\{a\}\_\{0\},\\;r\_\{1\},\\;s\_\{1\},\\;\\dots,\\;\\mathbf\{a\}\_\{M\},\\;r\_\{M\},\\;s\_\{M\}\\bigr\), with utterance likelihoodπθ​\(𝐚m∣sm−1\)=∏t=1Lmπθ​\(at\(m\)∣sm−1,a<t\(m\)\)\\pi\_\{\\theta\}\(\\mathbf\{a\}\_\{m\}\\mid s\_\{m\-1\}\)=\\prod\_\{t=1\}^\{L\_\{m\}\}\\pi\_\{\\theta\}\\\!\\left\(a\_\{t\}^\{\(m\)\}\\mid s\_\{m\-1\},a\_\{<t\}^\{\(m\)\}\\right\), whereLmL\_\{m\}is the length of utterancemm,ata\_\{t\}is thett​ht^\{th\}token of utterance\. Besides, a trajectory may terminate early after goal completion\. With the composite rewardrm=R~mr\_\{m\}=\\tilde\{R\}\_\{m\}defined in Section[3\.3](https://arxiv.org/html/2609.09764#S3.SS3), the policy maximizesJ⁡\(θ\)=𝔼τ∼πθ​\[R⁡\(τ\)\]J\(\\theta\)=\\mathbb\{E\}\_\{\\tau\\sim\\pi\_\{\\theta\}\}\\bigl\[R\(\\tau\)\\bigr\]\. The counterpart’s private goal is hidden and their responses are stochastic, making conversation\-level optimization under evolving dialogue states essential\.

### 3\.2Training Algorithm

SocialRL uses PPO\([Schulman et al\., 2017](https://arxiv.org/html/2609.09764#bib.bib24)\)over multi\-turn dialogue trajectories\. For each scenario, the current policy interacts with the counterpart for up toMMturns; after rollout, each utterance receives the composite rewardR~t\\tilde\{R\}\_\{t\}from Section[3\.3](https://arxiv.org/html/2609.09764#S3.SS3)\. A value network estimates future returns from these turn\-level rewards to compute advantages for trajectory optimization\.

We compute Generalized Advantage Estimation \(GAE\)\([Schulman et al\., 2016](https://arxiv.org/html/2609.09764#bib.bib23)\)from turn\-level rewards:

δt=R~t\+γ​Vϕ​\(st\+1\)−Vϕ​\(st\),\\delta\_\{t\}=\\tilde\{R\}\_\{t\}\+\\gamma V\_\{\\phi\}\(s\_\{t\+1\}\)\-V\_\{\\phi\}\(s\_\{t\}\),\(1\)A^t=∑l=0M−t\(γ​λ\)l​δt\+l,\\hat\{A\}\_\{t\}=\\sum\_\{l=0\}^\{M\-t\}\(\\gamma\\lambda\)^\{l\}\\,\\delta\_\{t\+l\},\(2\)whereVϕ​\(st\+1\)=0V\_\{\\phi\}\(s\_\{t\+1\}\)=0at the final turn andλ=0\.95\\lambda=0\.95\. Unlike token\-level RLHF, we treat each utterance as one optimization unit: all tokens in the utterance share the same reward and advantage\. The clipped policy objective is

LCLIP​\(θ\)=𝔼t​\[min⁡\(ρt​\(θ\)​A^t,clip⁡\(ρt​\(θ\),1−ϵ,1\+ϵ\)​A^t\)\],L^\{\\mathrm\{CLIP\}\}\(\\theta\)=\\mathbb\{E\}\_\{t\}\\Big\[\\min\\big\(\\rho\_\{t\}\(\\theta\)\\,\\hat\{A\}\_\{t\},\\;\\mathrm\{clip\}\(\\rho\_\{t\}\(\\theta\),1\-\\epsilon,1\+\\epsilon\)\\,\\hat\{A\}\_\{t\}\\big\)\\Big\],\(3\)whereρt​\(θ\)=πθ​\(at∣st\)/πθold​\(at∣st\)\\rho\_\{t\}\(\\theta\)=\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)/\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\_\{t\}\\mid s\_\{t\}\)is the utterance likelihood ratio andϵ=0\.2\\epsilon=0\.2\. The value network is trained with

ℒV​\(ϕ\)=𝔼t​\[\(Vϕ​\(st\)−Gt\)2\],\\mathcal\{L\}^\{V\}\(\\phi\)=\\mathbb\{E\}\_\{t\}\\big\[\(V\_\{\\phi\}\(s\_\{t\}\)\-G\_\{t\}\)^\{2\}\\big\],\(4\)whereGt=∑l=0M−tγl​R~t\+lG\_\{t\}=\\sum\_\{l=0\}^\{M\-t\}\\gamma^\{l\}\\tilde\{R\}\_\{t\+l\}is the discounted return from turntt\.

We use PPO rather than GRPO\([Shao et al\., 2024](https://arxiv.org/html/2609.09764#bib.bib25)\)because social dialogue rewards are delayed and strongly state\-dependent\. GRPO normalizes rewards across a group of sampled trajectories and replaces PPO’s state baseline with a group\-level baseline\. For turnmm, PPO uses

bmPPO=Vπ​\(sm−1\)=𝔼⁡\[Gm∣sm−1\],b\_\{m\}^\{\\mathrm\{PPO\}\}=V^\{\\pi\}\(s\_\{m\-1\}\)=\\mathbb\{E\}\[G\_\{m\}\\mid s\_\{m\-1\}\],\(5\)whereGm=∑k=mMγk−m​rkG\_\{m\}=\\sum\_\{k=m\}^\{M\}\\gamma^\{k\-m\}r\_\{k\}\. GRPO instead uses

bmGRPO=μ0​Dm,Dm=∑k=mMγk−m,b\_\{m\}^\{\\mathrm\{GRPO\}\}=\\mu\_\{0\}D\_\{m\},\\qquad D\_\{m\}=\\sum\_\{k=m\}^\{M\}\\gamma^\{k\-m\},\(6\)whereμ0\\mu\_\{0\}is the large\-group mean of the turn\-level rewards\. GRPO has a finite\-group correction that decreases as group size grows, while PPO’s state\-only baseline leaves the reference gradient target unchanged\. The key long\-horizon difference is variance: under the standard policy\-gradient baseline approximation, the unnormalized GRPO estimator adds a mismatch term whenever dialogue states with the same turn index have different values:

V~GRPO≈VPPO\+Δ,Δ=∑m=1M𝔼⁡\[‖Um‖2​\(Vπ​\(sm−1\)−μ0​Dm\)2\]≥0,\\widetilde\{V\}\_\{\\mathrm\{GRPO\}\}\\approx V\_\{\\mathrm\{PPO\}\}\+\\Delta,\\qquad\\Delta=\\sum\_\{m=1\}^\{M\}\\mathbb\{E\}\\\!\\left\[\\\|U\_\{m\}\\\|^\{2\}\\bigl\(V^\{\\pi\}\(s\_\{m\-1\}\)\-\\mu\_\{0\}D\_\{m\}\\bigr\)^\{2\}\\right\]\\geq 0,\(7\)whereUm=∇θ​log​πθ​\(am∣sm−1\)U\_\{m\}=\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\_\{m\}\\mid s\_\{m\-1\}\)\. The actual normalized GRPO variance additionally carries the global factor1/σ021/\\sigma\_\{0\}^\{2\}\. Since the same turn index can correspond to negotiation, repair, compromise, or failure, the baseline\-mismatch term becomes more pronounced in longer\-horizon dialogues\. Appendix[A](https://arxiv.org/html/2609.09764#A1)provides the derivation\.

### 3\.3Reward Design

Social dialogue requires feedback on both goal progress and relationship management\. Since outcome\-only scores provide no direct supervision for intermediate social behavior over 10–20 turns, SocialRL uses a multi\-dimensional process reward and combines it with the outcome reward\([Lightman et al\., 2023](https://arxiv.org/html/2609.09764#bib.bib16)\)\. At turntt, the composite reward is

R~t=\{α⋅Rprocess,t\+β⋅Routcome,t=T,α⋅Rprocess,t,t<T,\\tilde\{R\}\_\{t\}=\\begin\{cases\}\\alpha\\cdot R\_\{\\mathrm\{process\},t\}\+\\beta\\cdot R\_\{\\mathrm\{outcome\}\},&t=T,\\\\ \\alpha\\cdot R\_\{\\mathrm\{process\},t\},&t<T,\\end\{cases\}\(8\)whereTTis the final turn,Rprocess,tR\_\{\\mathrm\{process\},t\}scores the current utterance,RoutcomeR\_\{\\mathrm\{outcome\}\}scores final success, and validation selectsα=0\.3,β=1\\alpha=0\.3,\\beta=1\. The process reward is

Rprocess,t=∑i=16wt,i​rt,i,R\_\{\\mathrm\{process\},t\}=\\sum\_\{i=1\}^\{6\}w\_\{t,i\}r\_\{t,i\},\(9\)with normalized non\-negative weights

∑i=16wt,i=1,wt,i≥0\.\\sum\_\{i=1\}^\{6\}w\_\{t,i\}=1,\\qquad w\_\{t,i\}\\geq 0\.\(10\)Intermediate turns therefore still produce useful gradients even when final success is not yet known or the dialogue ultimately fails\.

Each utterance is scored along six dimensions:goal advancement,strategic positioning,relational attunement,persona consistency,contextual coherence, andturn quality\. These dimensions separate goal\-side progress, relationship\-side maintenance, and general conversational validity, drawing on goal\-setting theory\([Locke & Latham, 1990](https://arxiv.org/html/2609.09764#bib.bib18)\), strategic communication\([Kellermann, 1992](https://arxiv.org/html/2609.09764#bib.bib13);[Berger, 1997](https://arxiv.org/html/2609.09764#bib.bib3)\), relationship maintenance\([Stafford & Canary, 1991](https://arxiv.org/html/2609.09764#bib.bib26)\), self\-presentation\([Goffman, 1959](https://arxiv.org/html/2609.09764#bib.bib9)\), relevance\([Grice, 1975](https://arxiv.org/html/2609.09764#bib.bib11)\), and conversation analysis\([Sacks et al\., 1974](https://arxiv.org/html/2609.09764#bib.bib22)\)\.

To make scoring context\-specific, we use a rubric mechanism inspired by LLM\-based evaluation and criterion\-guided feedback\([Zheng et al\., 2023](https://arxiv.org/html/2609.09764#bib.bib31);[Liu et al\., 2023](https://arxiv.org/html/2609.09764#bib.bib17);[Bai et al\., 2022](https://arxiv.org/html/2609.09764#bib.bib1)\)\. Given scenarioSS, personaCC, goalGG, historyHtH\_\{t\}, and utteranceata\_\{t\}, the reward model generates 2–4 binary criteria for each dimension and judges whether each criterion passes\. The dimension score is the pass rate:

rt,i=1Mi​∑j=1Mipassi,j,r\_\{t,i\}=\\frac\{1\}\{M\_\{i\}\}\\sum\_\{j=1\}^\{M\_\{i\}\}\\mathrm\{pass\}\_\{i,j\},\(11\)wherepassi,j∈\{0,1\}\\mathrm\{pass\}\_\{i,j\}\\in\\\{0,1\\\}\. Binary criteria are more stable than abstract scalar judgments and make reward feedback inspectable\.

Finally, SocialRL changes dimension weights across dialogue stages\. Following interaction and relational\-stage theories\([Bales, 1950](https://arxiv.org/html/2609.09764#bib.bib2);[Knapp, 1978](https://arxiv.org/html/2609.09764#bib.bib14)\), early turns emphasize coherence, persona, and rapport; middle turns emphasize goal advancement and strategy; late turns balance closure with relationship preservation\. The reward model infers the current stage from context rather than fixed turn\-ratio thresholds:

ϕt=fstage​\(S,C,G,Ht,at\),ϕt∈\{Early,Mid,Late\}\.\\phi\_\{t\}=f\_\{\\mathrm\{stage\}\}\(S,C,G,H\_\{t\},a\_\{t\}\),\\qquad\\phi\_\{t\}\\in\\\{\\mathrm\{Early\},\\mathrm\{Mid\},\\mathrm\{Late\}\\\}\.\(12\)Givenϕt\\phi\_\{t\}, it proposes𝐰^t\\hat\{\\mathbf\{w\}\}\_\{t\}, which is sanitized by whitelist filtering, clipping, caps, fallback priors, overflow redistribution, and normalization before computingRprocess,tR\_\{\\mathrm\{process\},t\}\.

## 4Experiments

We evaluate SocialRL on four benchmarks with four trained backbones and four opponent models\. We ask whether it improves social dialogue performance, which design choices account for the gains, and whether it induces long\-horizon strategies\.

### 4\.1Experimental Setup

We use two judge\-based metrics\([Zheng et al\., 2023](https://arxiv.org/html/2609.09764#bib.bib31);[Liu et al\., 2023](https://arxiv.org/html/2609.09764#bib.bib17)\): Goal Achievement and Relationship Change\. Goal Achievement is reported as a percentage on SOTOPIA\-π\\piand AgentSense, and as a\[0,10\]\[0,10\]score on SOTOPIA\-All and SOTOPIA\-Hard; Relationship Change is reported after multiplying the SOTOPIA\-π\\piand AgentSense values by 100, while the original SOTOPIA benchmarks retain their\[−5,5\]\[\-5,5\]scale\. Unless otherwise stated, reported means and standard deviations are computed from five independent repeated experiments of each policy–opponent pair\. Prompts are provided in Appendix[B](https://arxiv.org/html/2609.09764#A2)\.

We use three complementary interactive benchmarks\. SOTOPIA\-π\\piis constructed from cleaned and filtered synthetic dialogues\([Wang et al\., 2024](https://arxiv.org/html/2609.09764#bib.bib28)\); it contains 1,773 scenarios across seven social contexts, including 259 held\-out test scenarios\. Its multi\-turn interactions evaluate whether an agent can pursue a social goal while maintaining the counterpart’s relationship quality\. SOTOPIA\-All and SOTOPIA\-Hard are based on the open\-ended SOTOPIA environment\([Zhou et al\., 2024a](https://arxiv.org/html/2609.09764#bib.bib32)\)\. SOTOPIA\-All covers 90 diverse everyday social scenarios, whereas SOTOPIA\-Hard is a 14\-scenario subset selected for higher conflict, ambiguous intentions, and subtle social norms\. AgentSense is an independent benchmark built bottom\-up from social situations extracted from movie and television scripts\([Mou et al\., 2025](https://arxiv.org/html/2609.09764#bib.bib19)\)\. It provides 1,225 scenarios organized into 245 templates with synthetic character instantiations, and evaluates both goal completion and latent social reasoning, such as inferring private information from dialogue\.

We train Qwen2\.5\-7B\-Instruct, Qwen3\-8B, LLaMA3\.1\-8B, and Gemma\-3\-4B policies against Qwen2\.5\-7B, Qwen3\-8B, Qwen3\.5\-35B, and GPT\-5\.5 opponents\. Baselines include base models, Behavior Cloning \(BC\), Sotopia\-RL\([Yu et al\., 2025](https://arxiv.org/html/2609.09764#bib.bib30)\), SDPO\([Kong et al\., 2025](https://arxiv.org/html/2609.09764#bib.bib15)\), ArCHer\([Zhou et al\., 2024b](https://arxiv.org/html/2609.09764#bib.bib33)\), and commercial reference models\. Implementation details are in Appendix[C](https://arxiv.org/html/2609.09764#A3)\.

### 4\.2Main Results

#### SocialRL improves goal achievement and relationship maintenance\.

We first evaluate the main two\-party setting on SOTOPIA\-π\\pi\(Table[2](https://arxiv.org/html/2609.09764#S4.T2)\)\. The table reports Goal Achievement success rates and Relationship Change across four opponents\. Across opponents, SocialRL with the Qwen2\.5\-7B backbone averages52\.3%52\.3\\%/0\.3580\.358in Goal Achievement / Relationship Change, compared with42\.3%42\.3\\%/0\.3110\.311for Base\. The strongest trained\-model average is obtained by the LLaMA3\.1\-8B policy \(59\.6%59\.6\\%/0\.3410\.341\), while GPT\-5\.5 reaches69\.0%69\.0\\%/0\.3880\.388as a commercial reference\.

Table 2:Results on the SOTOPIA\-π\\pibenchmark: Goal Achievement \(%\) / Relationship Change \(×100\\times 100\) for each policy–opponent pair\. Values are means across repeated experiments; complete mean±\\pmSD results are in Appendix[8](https://arxiv.org/html/2609.09764#A4.T8)\. Relationship Change values are multiplied by 100\. Bold denotes the best SocialRL model per metric and opponent column\. SocialRL has improved performance on each same\-backbone network and in the average results among non\-commercial methods\.Across opponents, LLaMA3\.1\-8B has the strongest trained\-model average in Goal Achievement \(59\.6%59\.6\\%\), while Qwen2\.5\-7B leads in Relationship Change \(0\.3580\.358\)\. The smaller Gemma\-3\-4B backbone also benefits substantially: its average rises from30\.5%/−0\.09330\.5\\%/\-0\.093for Base to52\.3%/0\.13452\.3\\%/0\.134with SocialRL, improving by21\.821\.8percentage points in Goal Achievement and0\.2270\.227in the displayed Relationship Change scale\.

As a separate and out of distribution test of multi\-party interaction and latent social reasoning, AgentSense results are shown in Table[3](https://arxiv.org/html/2609.09764#S4.T3)\. SocialRL with Qwen3\-8B reaches80\.5%/0\.44480\.5\\%/0\.444on average, exceeding the corresponding Base model by7\.27\.2percentage points in Goal Achievement and0\.2930\.293in the displayed Relationship Change scale\. The improvement is especially clear for Gemma\-3\-4B, which rises from46\.3%/0\.06546\.3\\%/0\.065to68\.1%/0\.23468\.1\\%/0\.234\. These results extend the SOTOPIA\-π\\pifindings to script\-derived, multi\-party scenarios where success also requires inferring information that is not stated explicitly\. we report SOTOPIA\-All and SOTOPIA\-Hard in Appendix[D](https://arxiv.org/html/2609.09764#A4)\.

To summarize performance across settings, we use each table’s opponent\-averaged Goal Achievement and compute the absolute SocialRL–Base difference for the same backbone, then average across the four trained backbones\. This yields gains of10\.72510\.725points on SOTOPIA\-π\\pi,9\.6259\.625on AgentSense,0\.92750\.9275on SOTOPIA\-All, and0\.71250\.7125on SOTOPIA\-Hard\. Because the latter two benchmarks use\[0,10\]\[0,10\]Goal Achievement scores, we multiply their gains by 10 before averaging, which places all four benchmarks on a percentage\-point scale\. The resulting overall improvement is9\.29\.2percentage points\.

Table 3:AgentSense results: Goal Achievement Success Rate \(%\) / Relationship Change \(×100\\times 100\) across multi\-party scenarios\. Bold denotes the best SocialRL model per opponent\. Values are means across five independent repeats; complete mean±\\pmSD results are in Appendix[9](https://arxiv.org/html/2609.09764#A4.T9)\. Relationship Change values are multiplied by 100\. SocialRL consistently outperforms BC, SDPO, and Sotopia\-RL on both metrics\.

### 4\.3Ablation Experiments

#### Algorithm choice, process rewards, and dynamic weights matter\.

Figure[3](https://arxiv.org/html/2609.09764#S4.F3)compares Qwen2\.5\-7B policies against GPT\-5\.5\. With the same SocialRL reward design, PPO exceeds multi\-turn GRPO by9\.99\.9percentage points in Goal Achievement and0\.0800\.080in Relationship Change\. The remaining ablations are in Appendix[E](https://arxiv.org/html/2609.09764#A5): process rewards have the largest effect, goal\-oriented dimensions drive task success, relational attunement preserves relationship quality, and dynamic weights outperform fixed schedules\.

Figure 3:Contextual comparison of Qwen2\.5\-7B policies on the SOTOPIA\-π\\pibenchmark against GPT\-5\.5\. Goal values are percentages; Relationship Change is reported on its original\[−1,1\]\[\-1,1\]scale\. Base, BC, and SDPO are reference methods; the controlled algorithm comparison uses SocialRL’s full reward design with both multi\-turn GRPO and multi\-turn PPO\.

### 4\.4Human Alignment Evaluation

To assess whether the automatic reward models agree with human judgments, we conduct an independent human\-alignment audit on 100 dialogue trajectories\. For PRM, the audit covers 497 turn\-level process records and yields a correlation of0\.8160\.816between PRM scores and aggregated human ratings\. For ORM, the audit covers 100 dialogue\-level judgments and yields correlations of0\.9300\.930for Goal Achievement and0\.9460\.946for Relationship Change\. All correlations are computed after the corresponding score transformations and indicate strong agreement between the automatic evaluators and human assessment\.

### 4\.5In\-Depth Case Analysis

#### SocialRL learns long\-horizon social strategy\.

We analyze a representative SOTOPIA\-π\\piboard\-game dialogue in which Ethan must speed up Benjamin’s play without damaging rapport\. Representative turns and extended analysis are provided in Appendix[F](https://arxiv.org/html/2609.09764#A6)\.

The Base model is polite but reactive: it repeatedly agrees with Benjamin, who independently proposes the one\-minute limit\. The pacing solution therefore comes from Benjamin rather than Ethan, yielding a goal score of0\.00\.0\.

SocialRL turns the opening into an explicit rule: “Let’s set a quick timer for your next turn,” and ratifies Benjamin’s “ninety seconds” proposal with “Deal\. Ninety seconds, and no deep dives\.” Benjamin later adopts the rule and requests future enforcement\. The dialogue reaches a goal score of1\.01\.0while preserving Base’s relationship delta \(\+0\.6\+0\.6\)\.

Thus, SocialRL converts an opening into a face\-saving commitment and maintains it across later turns, improving goal success without sacrificing rapport\.

## 5Conclusion

We presented SocialRL, a multi\-turn reinforcement learning framework for social dialogue that addresses the tension between goal pursuit and relationship management\. SocialRL optimizes complete dialogue trajectories with PPO and a value network, while its multi\-dimensional process reward system provides dense turn\-level feedback through rubric\-based scoring and stage\-aware weights\. Experiments on the SOTOPIA\-π\\pibenchmark, SOTOPIA\-All, SOTOPIA\-Hard, and AgentSense show improvements over the compared non\-commercial imitation\-learning, reinforcement\-learning, and preference\-learning baselines across multiple opponents and trained backbones; commercial reference models remain stronger in several settings\. Remaining limitations include over\-compromise, rigidity under unexpected opponent moves, and memory decay in very long dialogues, suggesting future work on goal\-floor constraints, broader training distributions, opponent modelling, and memory\-augmented policies\.

## References

- Bai et al\. \(2022\)Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al\.Constitutional AI: Harmlessness from AI feedback\.*arXiv preprint arXiv:2212\.08073*, 2022\.
- Bales \(1950\)Robert F\. Bales\.*Interaction Process Analysis: A Method for the Study of Small Groups*\.Addison\-Wesley, Cambridge, MA, 1950\.
- Berger \(1997\)Charles R\. Berger\.*Planning Strategic Interaction: Attaining Goals Through Communicative Action*\.Lawrence Erlbaum Associates, Mahwah, NJ, 1997\.
- Chen et al\. \(2024a\)Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan\.Step\-level value preference optimization for mathematical reasoning\.In*Findings of the Association for Computational Linguistics: EMNLP 2024*, pp\. 7889–7903\. Association for Computational Linguistics, 2024a\.doi:10\.18653/v1/2024\.findings\-emnlp\.463\.URL[https://aclanthology\.org/2024\.findings\-emnlp\.463/](https://aclanthology.org/2024.findings-emnlp.463/)\.
- Chen et al\. \(2024b\)Zhuang Chen, Jialiang Shi, Zhicheng Liu, Mengting Xu, Yijun Guo, Zhengyang Wang, Jiawen Gao, Yue Shao, and Bing Liu\.ToMBench: Benchmarking theory of mind in large language models\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics*, pp\. 4280–4302, 2024b\.
- Feng et al\. \(2026\)Xiachong Feng, Yi Jiang, Xiaocheng Feng, Deyi Yin, Libo Qin, Yangfan Ye, Lei Huang, Weitao Ma, Yuxuan Gu, Chonghan Qin, Bing Qin, and Lingpeng Kong\.SAVOIR: Learning social savoir\-faire via shapley\-based reward attribution\.In*Findings of the Association for Computational Linguistics: ACL 2026*, pp\. 14276–14290\. Association for Computational Linguistics, 2026\.doi:10\.18653/v1/2026\.findings\-acl\.699\.URL[https://aclanthology\.org/2026\.findings\-acl\.699/](https://aclanthology.org/2026.findings-acl.699/)\.
- Gao et al\. \(2025\)Zhaolin Gao, Wenhao Zhan, Jonathan D\. Chang, Gokul Swamy, Kianté Brantley, Jason D\. Lee, and Wen Sun\.Regressing the relative future: Efficient policy optimization for multi\-turn RLHF\.In*International Conference on Learning Representations*, 2025\.URL[https://openreview\.net/forum?id=cVyELMpMRS](https://openreview.net/forum?id=cVyELMpMRS)\.
- Goel & Zhu \(2025\)Hitesh Goel and Hao Zhu\.Lifelong\-SOTOPIA: Evaluating social intelligence of language agents over lifelong social interactions\.*arXiv preprint arXiv:2506\.12666*, 2025\.
- Goffman \(1959\)Erving Goffman\.*The Presentation of Self in Everyday Life*\.Doubleday Anchor, New York, 1959\.
- Greensmith et al\. \(2004\)Evan Greensmith, Peter L\. Bartlett, and Jonathan Baxter\.Variance reduction techniques for gradient estimates in reinforcement learning\.*Journal of Machine Learning Research*, 5:1471–1530, 2004\.
- Grice \(1975\)H\. Paul Grice\.Logic and conversation\.In Peter Cole and Jerry L\. Morgan \(eds\.\),*Syntax and Semantics, Vol\. 3: Speech Acts*, pp\. 41–58\. Academic Press, New York, 1975\.
- Jiang et al\. \(2026\)Bowen Jiang, Taiwei Shi, Ryo Kamoi, Yuan Yuan, Camillo J\. Taylor, Longqi Yang, Pei Zhou, and Sihao Chen\.One model, all roles: Multi\-turn, multi\-agent self\-play reinforcement learning for conversational social intelligence\.*arXiv preprint arXiv:2602\.03109*, 2026\.
- Kellermann \(1992\)Kathy Kellermann\.Communication: Inherently strategic and primarily automatic\.*Communication Monographs*, 59\(3\):288–300, 1992\.
- Knapp \(1978\)Mark L\. Knapp\.*Social Intercourse: From Greeting to Goodbye*\.Allyn and Bacon, Boston, MA, 1978\.
- Kong et al\. \(2025\)Aobo Kong, Wentao Ma, Shiwan Zhao, Yongbin Li, Yuchuan Wu, Ke Wang, Xiaoqian Liu, Qicheng Li, Yong Qin, and Fei Huang\.SDPO: Segment\-level direct preference optimization for social agents\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 12409–12423\. Association for Computational Linguistics, 2025\.doi:10\.18653/v1/2025\.acl\-long\.607\.URL[https://aclanthology\.org/2025\.acl\-long\.607/](https://aclanthology.org/2025.acl-long.607/)\.
- Lightman et al\. \(2023\)Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe\.Let’s verify step by step\.*arXiv preprint arXiv:2305\.20050*, 2023\.
- Liu et al\. \(2023\)Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu\.G\-Eval: NLG evaluation using GPT\-4 with better human alignment\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pp\. 2511–2522, 2023\.
- Locke & Latham \(1990\)Edwin A\. Locke and Gary P\. Latham\.*A Theory of Goal Setting and Task Performance*\.Prentice\-Hall, Englewood Cliffs, NJ, 1990\.
- Mou et al\. \(2025\)Xinyi Mou, Jingcong Liang, Jiayu Lin, Xinnong Zhang, Xiawei Liu, Shiyue Yang, Rong Ye, Lei Chen, Haoyu Kuang, Xuanjing Huang, and Zhongyu Wei\.AgentSense: Benchmarking social intelligence of language agents through interactive scenarios\.In*Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, pp\. 4975–5001\. Association for Computational Linguistics, 2025\.doi:10\.18653/v1/2025\.naacl\-long\.257\.URL[https://aclanthology\.org/2025\.naacl\-long\.257/](https://aclanthology.org/2025.naacl-long.257/)\.
- Ouyang et al\. \(2022\)Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L\. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al\.Training language models to follow instructions with human feedback\.In*Advances in Neural Information Processing Systems*, volume 35, pp\. 27730–27744, 2022\.
- Puterman \(1994\)Martin L\. Puterman\.*Markov Decision Processes: Discrete Stochastic Dynamic Programming*\.Wiley, New York, 1994\.
- Sacks et al\. \(1974\)Harvey Sacks, Emanuel A\. Schegloff, and Gail Jefferson\.A simplest systematics for the organization of turn\-taking for conversation\.*Language*, 50\(4\):696–735, 1974\.
- Schulman et al\. \(2016\)John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel\.High\-dimensional continuous control using generalized advantage estimation\.In*International Conference on Learning Representations*, 2016\.
- Schulman et al\. \(2017\)John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov\.Proximal policy optimization algorithms\.*arXiv preprint arXiv:1707\.06347*, 2017\.
- Shao et al\. \(2024\)Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y\. K\. Li, Y\. Wu, and Daya Guo\.DeepSeekMath: Pushing the limits of mathematical reasoning in open language models\.*arXiv preprint arXiv:2402\.03300*, 2024\.
- Stafford & Canary \(1991\)Laura Stafford and Daniel J\. Canary\.Maintenance strategies and romantic relationship type, gender, and relational characteristics\.*Journal of Social and Personal Relationships*, 8\(2\):217–242, 1991\.
- Sutton & Barto \(2018\)Richard S\. Sutton and Andrew G\. Barto\.*Reinforcement Learning: An Introduction*\.MIT Press, Cambridge, MA, 2 edition, 2018\.
- Wang et al\. \(2024\)Ruiyi Wang, Haofei Yu, Wenxin Zhang, Zhengyang Qi, Maarten Sap, Yonatan Bisk, Graham Neubig, and Hao Zhu\.SOTOPIA\-π\\pi: Interactive learning of socially intelligent language agents\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 12912–12940\. Association for Computational Linguistics, 2024\.doi:10\.18653/v1/2024\.acl\-long\.698\.URL[https://aclanthology\.org/2024\.acl\-long\.698/](https://aclanthology.org/2024.acl-long.698/)\.
- Yang et al\. \(2025\)Ruihan Yang, Yikai Zhang, Aili Chen, Xintao Wang, Jiangjie Chen, Siyu Yuan, Deqing Yang, and Yanghua Xiao\.ARIA: Training language agents with intention\-driven reward aggregation\.In*Advances in Neural Information Processing Systems*, volume 38, pp\. 121608–121640, 2025\.doi:10\.52202/085713\-3665\.
- Yu et al\. \(2025\)Haofei Yu, Zhengyang Qi, Yining Zhao, Kolby Nottingham, Keyang Xuan, Bodhisattwa Prasad Majumder, Hao Zhu, Paul Pu Liang, and Jiaxuan You\.Sotopia\-RL: Reward design for social intelligence\.*arXiv preprint arXiv:2508\.03905*, 2025\.
- Zheng et al\. \(2023\)Lianmin Zheng, Wei\-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P\. Xing, Hao Zhang, Joseph E\. Gonzalez, and Ion Stoica\.Judging LLM\-as\-a\-judge with MT\-Bench and chatbot arena\.In*Advances in Neural Information Processing Systems*, 2023\.
- Zhou et al\. \(2024a\)Xuhui Zhou, Hao Zhu, Leena Mathur, Daniel Zhang, Luyao Yu, Beatriz Chen, Mohit Bansal, Maarten Sap, Yonatan Goldberg, et al\.SOTOPIA: Interactive evaluation for social intelligence in language agents\.In*International Conference on Learning Representations*, 2024a\.
- Zhou et al\. \(2024b\)Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar\.ArCHer: Training language model agents via hierarchical multi\-turn RL\.In*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pp\. 62178–62209\. PMLR, 2024b\.URL[https://proceedings\.mlr\.press/v235/zhou24t\.html](https://proceedings.mlr.press/v235/zhou24t.html)\.

## Appendix ATheoretical Analysis of PPO and GRPO in Multi\-Turn Social Dialogue

We compare PPO with GAE and GRPO for multi\-turn social dialogue optimization under the standard policy\-gradient baseline approximation\. The argument rests on two properties: gradient bias and gradient variance\.

### A\.1Notation

Table[4](https://arxiv.org/html/2609.09764#A1.T4)summarizes the notation used throughout this appendix and the main text\.

Table 4:Notation used in the theoretical analysis and the main text\.
### A\.2Setup and Notation

Consider a trajectory withMMdialogue turns:

τ=\(s0,𝐚1,r1,s1,…,𝐚M,rM,sM\)\.\\tau=\(s\_\{0\},\\mathbf\{a\}\_\{1\},r\_\{1\},s\_\{1\},\\dots,\\mathbf\{a\}\_\{M\},r\_\{M\},s\_\{M\}\)\.\(13\)Heresm−1s\_\{m\-1\}is the dialogue state before turnmm,𝐚m\\mathbf\{a\}\_\{m\}is the utterance generated at that turn, andrmr\_\{m\}is its reward\. The discounted return from turnmmis

Gm=∑k=mMγk−m​rk,G\_\{m\}=\\sum\_\{k=m\}^\{M\}\\gamma^\{k\-m\}r\_\{k\},\(14\)whereγ∈\[0,1\]\\gamma\\in\[0,1\]controls how strongly future rewards affect turnmm\. and the score function for the utterance at turnmmis

Um=∇θ​log​πθ​\(𝐚m∣sm−1\)\.U\_\{m\}=\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(\\mathbf\{a\}\_\{m\}\\mid s\_\{m\-1\}\)\.\(15\)ThusUmU\_\{m\}is the gradient direction induced by the sampled utterance\. We use the following trajectory\-level Monte Carlo target:

gMC​\(θ\)=𝔼⁡\[∑m=1MUm​Gm\]\.g\_\{\\mathrm\{MC\}\}\(\\theta\)=\\mathbb\{E\}\\left\[\\sum\_\{m=1\}^\{M\}U\_\{m\}G\_\{m\}\\right\]\.\(16\)gMCg\_\{\\mathrm\{MC\}\}is the expected policy\-gradient signal used as the common reference for PPO and GRPO\. All expectations are on\-policy\. Assume fixedMM, sufficient finite moments for the sample\-standard\-deviation expansion, andσ0\>0\\sigma\_\{0\}\>0\. Treat the turn index as part of the state and define

Vπ​\(s\)\\displaystyle V^\{\\pi\}\(s\)=𝔼⁡\[Gm∣sm−1=s\],\\displaystyle=\\mathbb\{E\}\[G\_\{m\}\\mid s\_\{m\-1\}=s\],\(17\)Qπ​\(s,𝐚\)\\displaystyle Q^\{\\pi\}\(s,\\mathbf\{a\}\)=𝔼\[Gm∣sm−1=s,𝐚m=𝐚\],\\displaystyle=\\mathbb\{E\}\[G\_\{m\}\\mid s\_\{m\-1\}=s,\\mathbf\{a\}\_\{m\}=\\mathbf\{a\}\],\(18\)Aπ​\(s,𝐚\)\\displaystyle A^\{\\pi\}\(s,\\mathbf\{a\}\)=Qπ​\(s,𝐚\)−Vπ​\(s\)\.\\displaystyle=Q^\{\\pi\}\(s,\\mathbf\{a\}\)\-V^\{\\pi\}\(s\)\.\(19\)HereVπV^\{\\pi\}is the state value,QπQ^\{\\pi\}is the state\-action value, andAπA^\{\\pi\}measures the value of an action relative to the state average\.

Lemma 1\(Baseline unbiasedness\)\. For any state\-only functionb⁡\(sm−1\)b\(s\_\{m\-1\}\),

g^b=∑m=1MUm​\(Gm−b⁡\(sm−1\)\)\\hat\{g\}\_\{b\}=\\sum\_\{m=1\}^\{M\}U\_\{m\}\\left\(G\_\{m\}\-b\(s\_\{m\-1\}\)\\right\)\(20\)wherebbis a control\-variate baseline used to reduce gradient variance has the same expectation as

g^MC=∑m=1MUm​Gm\.\\hat\{g\}\_\{\\mathrm\{MC\}\}=\\sum\_\{m=1\}^\{M\}U\_\{m\}G\_\{m\}\.\(21\)Proof\.Conditioning onsm−1s\_\{m\-1\},

𝔼⁡\[Um​b​\(sm−1\)\]\\displaystyle\\mathbb\{E\}\\left\[U\_\{m\}b\(s\_\{m\-1\}\)\\right\]=𝔼sm−1\[b\(sm−1\)𝔼𝐚m∼πθ\(⋅∣sm−1\)\[∇θlogπθ\(𝐚m∣sm−1\)∣sm−1\]\]\\displaystyle=\\mathbb\{E\}\_\{s\_\{m\-1\}\}\\left\[b\(s\_\{m\-1\}\)\\,\\mathbb\{E\}\_\{\\mathbf\{a\}\_\{m\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid s\_\{m\-1\}\)\}\\left\[\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(\\mathbf\{a\}\_\{m\}\\mid s\_\{m\-1\}\)\\mid s\_\{m\-1\}\\right\]\\right\]\(22\)=𝔼sm−1​\[b⁡\(sm−1\)​∑𝐚πθ​\(𝐚∣sm−1\)​∇θ​log⁡πθ​\(𝐚∣sm−1\)\]\\displaystyle=\\mathbb\{E\}\_\{s\_\{m\-1\}\}\\left\[b\(s\_\{m\-1\}\)\\sum\_\{\\mathbf\{a\}\}\\pi\_\{\\theta\}\(\\mathbf\{a\}\\mid s\_\{m\-1\}\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(\\mathbf\{a\}\\mid s\_\{m\-1\}\)\\right\]\(23\)=𝔼sm−1​\[b⁡\(sm−1\)​∑𝐚∇θπθ​\(𝐚∣sm−1\)\]\\displaystyle=\\mathbb\{E\}\_\{s\_\{m\-1\}\}\\left\[b\(s\_\{m\-1\}\)\\sum\_\{\\mathbf\{a\}\}\\nabla\_\{\\theta\}\\pi\_\{\\theta\}\(\\mathbf\{a\}\\mid s\_\{m\-1\}\)\\right\]\(24\)=𝔼sm−1​\[b⁡\(sm−1\)​∇θ​∑𝐚πθ​\(𝐚∣sm−1\)\]\\displaystyle=\\mathbb\{E\}\_\{s\_\{m\-1\}\}\\left\[b\(s\_\{m\-1\}\)\\nabla\_\{\\theta\}\\sum\_\{\\mathbf\{a\}\}\\pi\_\{\\theta\}\(\\mathbf\{a\}\\mid s\_\{m\-1\}\)\\right\]\(25\)=𝔼sm−1​\[b⁡\(sm−1\)​∇θ1\]=𝟎\.\\displaystyle=\\mathbb\{E\}\_\{s\_\{m\-1\}\}\\left\[b\(s\_\{m\-1\}\)\\nabla\_\{\\theta\}1\\right\]=\\mathbf\{0\}\.\(26\)Summing overmmgives

𝔼⁡\[g^b\]=𝔼⁡\[∑m=1MUm​Gm\]−𝔼⁡\[∑m=1MUm​b​\(sm−1\)\]=𝔼⁡\[g^MC\]\.\\mathbb\{E\}\[\\hat\{g\}\_\{b\}\]=\\mathbb\{E\}\\left\[\\sum\_\{m=1\}^\{M\}U\_\{m\}G\_\{m\}\\right\]\-\\mathbb\{E\}\\left\[\\sum\_\{m=1\}^\{M\}U\_\{m\}b\(s\_\{m\-1\}\)\\right\]=\\mathbb\{E\}\[\\hat\{g\}\_\{\\mathrm\{MC\}\}\]\.\(27\)Thus𝔼⁡\[g^b\]=gMC​\(θ\)\\mathbb\{E\}\[\\hat\{g\}\_\{b\}\]=g\_\{\\mathrm\{MC\}\}\(\\theta\)\. The baseline is assumed detached and state\-only\.□\\square

### A\.3PPO with GAE

PPO uses the GAE estimator\([Schulman et al\., 2016](https://arxiv.org/html/2609.09764#bib.bib23)\)with residual

δm=rm\+γ​Vϕ​\(sm\)−Vϕ​\(sm−1\),\\delta\_\{m\}=r\_\{m\}\+\\gamma V\_\{\\phi\}\(s\_\{m\}\)\-V\_\{\\phi\}\(s\_\{m\-1\}\),\(28\)whereVϕV\_\{\\phi\}is the learned critic andδm\\delta\_\{m\}is its one\-step TD error\. and advantage

AmGAE⁡\(λ\)=∑l=0M−m\(γ​λ\)l​δm\+l\.A\_\{m\}^\{\\mathrm\{GAE\}\(\\lambda\)\}=\\sum\_\{l=0\}^\{M\-m\}\(\\gamma\\lambda\)^\{l\}\\delta\_\{m\+l\}\.\(29\)The parameterλ\\lambdacontrols the bias–variance trade\-off in multi\-step advantage estimation\. ForVϕ=VπV\_\{\\phi\}=V^\{\\pi\}andλ=1\\lambda=1,

AmGAE⁡\(1\)\\displaystyle A\_\{m\}^\{\\mathrm\{GAE\}\(1\)\}=∑l=0M−mγl​\(rm\+l\+γ​Vπ​\(sm\+l\)−Vπ​\(sm\+l−1\)\)\\displaystyle=\\sum\_\{l=0\}^\{M\-m\}\\gamma^\{l\}\\left\(r\_\{m\+l\}\+\\gamma V^\{\\pi\}\(s\_\{m\+l\}\)\-V^\{\\pi\}\(s\_\{m\+l\-1\}\)\\right\)\(30\)=∑l=0M−mγl​rm\+l−Vπ​\(sm−1\)\+γM−m\+1​Vπ​\(sM\)\.\\displaystyle=\\sum\_\{l=0\}^\{M\-m\}\\gamma^\{l\}r\_\{m\+l\}\-V^\{\\pi\}\(s\_\{m\-1\}\)\+\\gamma^\{M\-m\+1\}V^\{\\pi\}\(s\_\{M\}\)\.\(31\)WithVπ​\(sM\)=0V^\{\\pi\}\(s\_\{M\}\)=0,

AmGAE⁡\(1\)=Gm−Vπ​\(sm−1\)\.A\_\{m\}^\{\\mathrm\{GAE\}\(1\)\}=G\_\{m\}\-V^\{\\pi\}\(s\_\{m\-1\}\)\.\(32\)Hence the unclipped on\-policy estimator

g^PPO=∑m=1MUm​AmGAE⁡\(λ\)\\hat\{g\}\_\{\\mathrm\{PPO\}\}=\\sum\_\{m=1\}^\{M\}U\_\{m\}\\,A\_\{m\}^\{\\mathrm\{GAE\}\(\\lambda\)\}\(33\)has expectationgMC​\(θ\)g\_\{\\mathrm\{MC\}\}\(\\theta\)for the ideal\-value,λ=1\\lambda=1case\. This does not establish unbiasedness for clipped PPO, an approximate critic, or generalλ<1\\lambda<1\. PPO uses

bmPPO=Vπ​\(sm−1\),b\_\{m\}^\{\\mathrm\{PPO\}\}=V^\{\\pi\}\(s\_\{m\-1\}\),\(34\)which approximates the score\-norm\-weighted trace\-variance\-minimizing baseline\([Greensmith et al\., 2004](https://arxiv.org/html/2609.09764#bib.bib10)\)\.

### A\.4GRPO and Its Bias

GRPO\([Shao et al\., 2024](https://arxiv.org/html/2609.09764#bib.bib25)\)samplesGGtrajectories from a fixed prompt and normalizes theirMMturn\-level rewards:

ℛ=\{rm\(i\):i=1,…,G,m=1,…,M\}\.\\mathcal\{R\}=\\\{r\_\{m\}^\{\(i\)\}:i=1,\\ldots,G,\\;m=1,\\ldots,M\\\}\.\(35\)LetN=G​MN=GMbe the number of rewards in the group\. The group mean is

μ=1N​∑i=1G∑m=1Mrm\(i\)=1G​M​∑i=1G∑m=1Mrm\(i\),\\mu=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{G\}\\sum\_\{m=1\}^\{M\}r\_\{m\}^\{\(i\)\}=\\frac\{1\}\{GM\}\\sum\_\{i=1\}^\{G\}\\sum\_\{m=1\}^\{M\}r\_\{m\}^\{\(i\)\},\(36\)and the group standard deviation is

σ=1N​∑i=1G∑m=1M\(rm\(i\)−μ\)2\.\\sigma=\\sqrt\{\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{G\}\\sum\_\{m=1\}^\{M\}\\left\(r\_\{m\}^\{\(i\)\}\-\\mu\\right\)^\{2\}\}\.\(37\)Thusμ\\muandσ\\sigmaare the empirical mean and standard deviation shared by all trajectories and turns in the group\. Standardize each reward by these statistics:

r~m\(i\)=rm\(i\)−μσ\.\\tilde\{r\}\_\{m\}^\{\(i\)\}=\\frac\{r\_\{m\}^\{\(i\)\}\-\\mu\}\{\\sigma\}\.\(38\)r~m\(i\)\\tilde\{r\}\_\{m\}^\{\(i\)\}is the centered and normalized reward used by GRPO\.

The multi\-turn GRPO advantage is the discounted future sum:

AmGRPO,\(i\)=r~m\(i\)\+γ​r~m\+1\(i\)\+γ2​r~m\+2\(i\)\+⋯\+γM−m​r~M\(i\)\.A\_\{m\}^\{\\mathrm\{GRPO\},\(i\)\}=\\tilde\{r\}\_\{m\}^\{\(i\)\}\+\\gamma\\tilde\{r\}\_\{m\+1\}^\{\(i\)\}\+\\gamma^\{2\}\\tilde\{r\}\_\{m\+2\}^\{\(i\)\}\+\\cdots\+\\gamma^\{M\-m\}\\tilde\{r\}\_\{M\}^\{\(i\)\}\.\(39\)AmGRPO,\(i\)=∑k=mMγk−m​r~k\(i\)\.A\_\{m\}^\{\\mathrm\{GRPO\},\(i\)\}=\\sum\_\{k=m\}^\{M\}\\gamma^\{k\-m\}\\tilde\{r\}\_\{k\}^\{\(i\)\}\.\(40\)Substitution gives

AmGRPO,\(i\)\\displaystyle A\_\{m\}^\{\\mathrm\{GRPO\},\(i\)\}=∑k=mMγk−m​\(rk\(i\)−μσ\)\\displaystyle=\\sum\_\{k=m\}^\{M\}\\gamma^\{k\-m\}\\left\(\\frac\{r\_\{k\}^\{\(i\)\}\-\\mu\}\{\\sigma\}\\right\)\(41\)=1σ​∑k=mMγk−m​\(rk\(i\)−μ\)\\displaystyle=\\frac\{1\}\{\\sigma\}\\sum\_\{k=m\}^\{M\}\\gamma^\{k\-m\}\\left\(r\_\{k\}^\{\(i\)\}\-\\mu\\right\)\(42\)=1σ​\(∑k=mMγk−m​rk\(i\)−μ​∑k=mMγk−m\)\.\\displaystyle=\\frac\{1\}\{\\sigma\}\\left\(\\sum\_\{k=m\}^\{M\}\\gamma^\{k\-m\}r\_\{k\}^\{\(i\)\}\-\\mu\\sum\_\{k=m\}^\{M\}\\gamma^\{k\-m\}\\right\)\.\(43\)With

Gm\(i\)=∑k=mMγk−m​rk\(i\)G\_\{m\}^\{\(i\)\}=\\sum\_\{k=m\}^\{M\}\\gamma^\{k\-m\}r\_\{k\}^\{\(i\)\}\(44\)and

Dm=∑k=mMγk−m,D\_\{m\}=\\sum\_\{k=m\}^\{M\}\\gamma^\{k\-m\},\(45\)DmD\_\{m\}is the total discount mass remaining after turnmm\. the advantage is

AmGRPO,\(i\)=1σ​\(Gm\(i\)−μ​Dm\),A\_\{m\}^\{\\mathrm\{GRPO\},\(i\)\}=\\frac\{1\}\{\\sigma\}\\left\(G\_\{m\}^\{\(i\)\}\-\\mu D\_\{m\}\\right\),\(46\)whereDmD\_\{m\}is the remaining discount mass\. In the large\-group limit,

μ0=1M​∑k=1M𝔼⁡\[rk\],σ02=1M​∑k=1M𝔼⁡\[\(rk−μ0\)2\]\.\\mu\_\{0\}=\\frac\{1\}\{M\}\\sum\_\{k=1\}^\{M\}\\mathbb\{E\}\[r\_\{k\}\],\\qquad\\sigma\_\{0\}^\{2\}=\\frac\{1\}\{M\}\\sum\_\{k=1\}^\{M\}\\mathbb\{E\}\\\!\\left\[\(r\_\{k\}\-\\mu\_\{0\}\)^\{2\}\\right\]\.\(47\)The constantsμ0\\mu\_\{0\}andσ0\\sigma\_\{0\}are the population limits of the group mean and standard deviation asG→∞G\\to\\infty\.

Theorem 1\(Finite\-group correction for GRPO\)\. Assume fixedMM,σ0\>0\\sigma\_\{0\}\>0, independent trajectory\-level samples, and sufficient moments for a delta\-method expansion of the sample standard deviation\. Define

g^GRPOadj=σ0​g^GRPO\.\\hat\{g\}\_\{\\mathrm\{GRPO\}\}^\{\\mathrm\{adj\}\}=\\sigma\_\{0\}\\hat\{g\}\_\{\\mathrm\{GRPO\}\}\.\(48\)g^GRPOadj\\hat\{g\}\_\{\\mathrm\{GRPO\}\}^\{\\mathrm\{adj\}\}removes GRPO’s global normalization scale so its direction and variance can be compared with the MC target\. Then

𝔼⁡\[g^GRPOadj\]=gMC​\(θ\)\+𝒪⁡\(1/G\)\.\\mathbb\{E\}\[\\hat\{g\}\_\{\\mathrm\{GRPO\}\}^\{\\mathrm\{adj\}\}\]=g\_\{\\mathrm\{MC\}\}\(\\theta\)\+\\mathcal\{O\}\(1/G\)\.\(49\)The unadjusted estimator satisfies

𝔼⁡\[g^GRPO\]=1σ0​gMC​\(θ\)\+𝒪⁡\(1/G\)\.\\mathbb\{E\}\[\\hat\{g\}\_\{\\mathrm\{GRPO\}\}\]=\\frac\{1\}\{\\sigma\_\{0\}\}g\_\{\\mathrm\{MC\}\}\(\\theta\)\+\\mathcal\{O\}\(1/G\)\.\(50\)The first term is a global positive rescaling induced by reward standardization, not a change in gradient direction\. It can be absorbed into the learning rate by settingηGRPO=σ0​η\\eta\_\{\\mathrm\{GRPO\}\}=\\sigma\_\{0\}\\eta\. After this adjustment, the remaining finite\-group discrepancy is𝒪⁡\(1/G\)\\mathcal\{O\}\(1/G\)and vanishes asGGgrows\. Hereσ0=limG→∞σ\\sigma\_\{0\}=\\lim\_\{G\\to\\infty\}\\sigma\.

Proof\.Sinceσ\\sigmais estimated from the group, let

σ0=limG→∞σ\\sigma\_\{0\}=\\lim\_\{G\\to\\infty\}\\sigma\(51\)Keeping1/σ1/\\sigmaexplicit,

g^GRPO=1G​∑i=1G∑m=1MUm\(i\)​AmGRPO,\(i\)\.\\hat\{g\}\_\{\\mathrm\{GRPO\}\}=\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\sum\_\{m=1\}^\{M\}U\_\{m\}^\{\(i\)\}A\_\{m\}^\{\\mathrm\{GRPO\},\(i\)\}\.\(52\)Substituting the GRPO advantage gives

g^GRPO\\displaystyle\\hat\{g\}\_\{\\mathrm\{GRPO\}\}=1G​σ​∑i,mUm\(i\)​\(Gm\(i\)−μ​Dm\)\\displaystyle=\\frac\{1\}\{G\\sigma\}\\sum\_\{i,m\}U\_\{m\}^\{\(i\)\}\\left\(G\_\{m\}^\{\(i\)\}\-\\mu D\_\{m\}\\right\)\(53\)=1σ​\(T1−T2\),\\displaystyle=\\frac\{1\}\{\\sigma\}\\left\(T\_\{1\}\-T\_\{2\}\\right\),\(54\)where

T1=1G​∑i,mUm\(i\)​Gm\(i\),T2=1G2​M​∑i,j,m,lDm​Um\(i\)​rl\(j\)\.T\_\{1\}=\\frac\{1\}\{G\}\\sum\_\{i,m\}U\_\{m\}^\{\(i\)\}G\_\{m\}^\{\(i\)\},\\quad T\_\{2\}=\\frac\{1\}\{G^\{2\}M\}\\sum\_\{i,j,m,l\}D\_\{m\}U\_\{m\}^\{\(i\)\}r\_\{l\}^\{\(j\)\}\.\(55\)T1T\_\{1\}is the group\-averaged MC gradient term;T2T\_\{2\}is the correction induced by reusing the sampled group mean as a baseline\. For the first term,

𝔼⁡\[T1\]=𝔼⁡\[∑m=1MUm​Gm\]=gMC​\(θ\)\.\\mathbb\{E\}\[T\_\{1\}\]=\\mathbb\{E\}\\left\[\\sum\_\{m=1\}^\{M\}U\_\{m\}G\_\{m\}\\right\]=g\_\{\\mathrm\{MC\}\}\(\\theta\)\.\(56\)ForT2T\_\{2\}, separate cross\- and same\-trajectory terms:

𝔼⁡\[T2\]=1G2​M​∑i≠j∑m,lDm​𝔼​\[Um\(i\)​rl\(j\)\]\+1G2​M​∑i=j∑m,lDm​𝔼​\[Um\(i\)​rl\(i\)\]\.\\mathbb\{E\}\[T\_\{2\}\]=\\frac\{1\}\{G^\{2\}M\}\\sum\_\{i\\neq j\}\\sum\_\{m,l\}D\_\{m\}\\mathbb\{E\}\[U\_\{m\}^\{\(i\)\}r\_\{l\}^\{\(j\)\}\]\+\\frac\{1\}\{G^\{2\}M\}\\sum\_\{i=j\}\\sum\_\{m,l\}D\_\{m\}\\mathbb\{E\}\[U\_\{m\}^\{\(i\)\}r\_\{l\}^\{\(i\)\}\]\.\(57\)Fori≠ji\\neq j,

𝔼⁡\[Um\(i\)​rl\(j\)\]=𝔼⁡\[Um\(i\)\]​𝔼​\[rl\(j\)\]\.\\mathbb\{E\}\[U\_\{m\}^\{\(i\)\}r\_\{l\}^\{\(j\)\}\]=\\mathbb\{E\}\[U\_\{m\}^\{\(i\)\}\]\\mathbb\{E\}\[r\_\{l\}^\{\(j\)\}\]\.\(58\)Since

𝔼⁡\[Um\(i\)\]=𝟎,\\mathbb\{E\}\[U\_\{m\}^\{\(i\)\}\]=\\mathbf\{0\},\(59\)all cross\-trajectory terms vanish, leaving

𝔼\[T2\]=1G2​M⋅G∑m=1M∑l=1MDm𝔼\[Umrl\]=1G​M∑m=1M∑l=1MDm𝔼\[Umrl\]\.\\mathbb\{E\}\[T\_\{2\}\]=\\frac\{1\}\{G^\{2\}M\}\\cdot G\\sum\_\{m=1\}^\{M\}\\sum\_\{l=1\}^\{M\}D\_\{m\}\\mathbb\{E\}\[U\_\{m\}r\_\{l\}\]=\\frac\{1\}\{GM\}\\sum\_\{m=1\}^\{M\}\\sum\_\{l=1\}^\{M\}D\_\{m\}\\mathbb\{E\}\[U\_\{m\}r\_\{l\}\]\.\(60\)Expand1/σ1/\\sigmaaroundσ0\\sigma\_\{0\}:

1σ\\displaystyle\\frac\{1\}\{\\sigma\}=1σ0⋅11\+σ−σ0σ0\\displaystyle=\\frac\{1\}\{\\sigma\_\{0\}\}\\cdot\\frac\{1\}\{1\+\\frac\{\\sigma\-\\sigma\_\{0\}\}\{\\sigma\_\{0\}\}\}\(61\)=1σ0⋅\(1−σ−σ0σ0\+𝒪⁡\(\(σ−σ0\)2\)\)\\displaystyle=\\frac\{1\}\{\\sigma\_\{0\}\}\\cdot\\left\(1\-\\frac\{\\sigma\-\\sigma\_\{0\}\}\{\\sigma\_\{0\}\}\+\\mathcal\{O\}\\left\(\(\\sigma\-\\sigma\_\{0\}\)^\{2\}\\right\)\\right\)\(62\)=1σ0−σ−σ0σ02\+𝒪⁡\(\(σ−σ0\)2\)\.\\displaystyle=\\frac\{1\}\{\\sigma\_\{0\}\}\-\\frac\{\\sigma\-\\sigma\_\{0\}\}\{\\sigma\_\{0\}^\{2\}\}\+\\mathcal\{O\}\\left\(\(\\sigma\-\\sigma\_\{0\}\)^\{2\}\\right\)\.\(63\)Substituting Eqs\. equation[56](https://arxiv.org/html/2609.09764#A1.E56), equation[60](https://arxiv.org/html/2609.09764#A1.E60), and equation[63](https://arxiv.org/html/2609.09764#A1.E63)into Eq\. equation[54](https://arxiv.org/html/2609.09764#A1.E54)yields the following asymptotic form\.

𝔼⁡\[g^GRPO\]\\displaystyle\\mathbb\{E\}\[\\hat\{g\}\_\{\\mathrm\{GRPO\}\}\]=1σ​\(𝔼⁡\[T1\]−𝔼⁡\[T2\]\)\\displaystyle=\\frac\{1\}\{\\sigma\}\\left\(\\mathbb\{E\}\[T\_\{1\}\]\-\\mathbb\{E\}\[T\_\{2\}\]\\right\)\(64\)=1σ0​gMC​\(θ\)−1G​M​σ0​∑m=1M∑l=1MDm​𝔼​\[Um​rl\]\+𝒪⁡\(1/G\)\\displaystyle=\\frac\{1\}\{\\sigma\_\{0\}\}g\_\{\\mathrm\{MC\}\}\(\\theta\)\-\\frac\{1\}\{GM\\sigma\_\{0\}\}\\sum\_\{m=1\}^\{M\}\\sum\_\{l=1\}^\{M\}D\_\{m\}\\mathbb\{E\}\[U\_\{m\}r\_\{l\}\]\+\\mathcal\{O\}\(1/G\)\(65\)=1σ0​gMC​\(θ\)\+𝒪⁡\(1/G\)\.\\displaystyle=\\frac\{1\}\{\\sigma\_\{0\}\}g\_\{\\mathrm\{MC\}\}\(\\theta\)\+\\mathcal\{O\}\(1/G\)\.\(66\)The remainder includes the same\-trajectory correction and the fluctuation ofσ\\sigma\. Multiplying byσ0\\sigma\_\{0\}gives the adjusted result\.□\\square

Corollary\.AsG→∞G\\to\\infty, GRPO recovers the policy\-gradient direction up to the positive scale1/σ01/\\sigma\_\{0\}; the adjusted estimator has finite\-group correction𝒪⁡\(1/G\)\\mathcal\{O\}\(1/G\)\. Hence finite\-group bias diminishes with group size\. The long\-horizon difference is governed primarily by variance\.

### A\.5Variance Comparison

For a fair comparison, use PPO and the scale\-adjusted large\-group GRPO estimator:

gPPO\\displaystyle g\_\{\\mathrm\{PPO\}\}=∑m=1MUm​\(Gm−Vπ​\(sm−1\)\),\\displaystyle=\\sum\_\{m=1\}^\{M\}U\_\{m\}\\left\(G\_\{m\}\-V^\{\\pi\}\(s\_\{m\-1\}\)\\right\),\(67\)gGRPOadj\\displaystyle g\_\{\\mathrm\{GRPO\}\}^\{\\mathrm\{adj\}\}=∑m=1MUm​\(Gm−μ0​Dm\)\.\\displaystyle=\\sum\_\{m=1\}^\{M\}U\_\{m\}\\left\(G\_\{m\}\-\\mu\_\{0\}D\_\{m\}\\right\)\.\(68\)Both have expectationgMC​\(θ\)g\_\{\\mathrm\{MC\}\}\(\\theta\)\. Define

ΣPPO\\displaystyle\\Sigma\_\{\\mathrm\{PPO\}\}=Cov⁡\(gPPO\),\\displaystyle=\\mathrm\{Cov\}\(g\_\{\\mathrm\{PPO\}\}\),VPPO\\displaystyle V\_\{\\mathrm\{PPO\}\}=Tr⁡\(ΣPPO\),\\displaystyle=\\mathrm\{Tr\}\(\\Sigma\_\{\\mathrm\{PPO\}\}\),\(69\)ΣGRPOadj\\displaystyle\\Sigma\_\{\\mathrm\{GRPO\}\}^\{\\mathrm\{adj\}\}=Cov⁡\(gGRPOadj\),\\displaystyle=\\mathrm\{Cov\}\(g\_\{\\mathrm\{GRPO\}\}^\{\\mathrm\{adj\}\}\),VGRPOadj\\displaystyle V\_\{\\mathrm\{GRPO\}\}^\{\\mathrm\{adj\}\}=Tr⁡\(ΣGRPOadj\)\.\\displaystyle=\\mathrm\{Tr\}\(\\Sigma\_\{\\mathrm\{GRPO\}\}^\{\\mathrm\{adj\}\}\)\.\(70\)HereΣ\\Sigmadenotes the gradient covariance matrix, whileVVis its trace, i\.e\., the total gradient\-noise power across parameter dimensions\. Define the gradient signal\-to\-noise ratios

BPPO=‖gMC​\(θ\)‖2VPPO,BGRPO=‖gMC​\(θ\)‖2VGRPOadj\.B\_\{\\mathrm\{PPO\}\}=\\frac\{\\\|g\_\{\\mathrm\{MC\}\}\(\\theta\)\\\|^\{2\}\}\{V\_\{\\mathrm\{PPO\}\}\},\\qquad B\_\{\\mathrm\{GRPO\}\}=\\frac\{\\\|g\_\{\\mathrm\{MC\}\}\(\\theta\)\\\|^\{2\}\}\{V\_\{\\mathrm\{GRPO\}\}^\{\\mathrm\{adj\}\}\}\.\(71\)BBmeasures gradient signal power relative to total gradient noise; larger values indicate a more reliable update direction\. The scale adjustment is necessary because unadjusted GRPO has signalgMC/σ0g\_\{\\mathrm\{MC\}\}/\\sigma\_\{0\}\. It does not change the signal\-to\-noise ratio, because both signal power and variance scale by1/σ021/\\sigma\_\{0\}^\{2\}\.

We use two approximations for an explicit decomposition\. For any state\-only baselinebmb\_\{m\}, assume

Cov⁡\(Um​\(Gm−bm\),Un​\(Gn−bn\)\)\\displaystyle\\mathrm\{Cov\}\\\!\\left\(U\_\{m\}\(G\_\{m\}\-b\_\{m\}\),U\_\{n\}\(G\_\{n\}\-b\_\{n\}\)\\right\)≈0,\\displaystyle\\approx 0,m≠n,\\displaystyle m\\neq n,\(A\)𝔼⁡\[‖Um‖2​\(Gm−Vπ​\(sm−1\)\)∣sm−1\]\\displaystyle\\mathbb\{E\}\\\!\\left\[\\\|U\_\{m\}\\\|^\{2\}\(G\_\{m\}\-V^\{\\pi\}\(s\_\{m\-1\}\)\)\\mid s\_\{m\-1\}\\right\]≈0\.\\displaystyle\\approx 0\.\(B\)These assumptions are used only for the variance decomposition\.

#### PPO Variance\.

For the turn\-level estimator

gm​\(b\)=Um​\(Gm−bm\),g\_\{m\}\(b\)=U\_\{m\}\(G\_\{m\}\-b\_\{m\}\),\(72\)gm​\(b\)g\_\{m\}\(b\)is the gradient contribution from turnmmunder baselinebmb\_\{m\}\. the trace\-variance\-minimizing scalar baseline givensm−1=ss\_\{m\-1\}=sis

bm∗​\(s\)=𝔼⁡\[‖Um‖2​Gm∣sm−1=s\]𝔼⁡\[‖Um‖2∣sm−1=s\]\.b\_\{m\}^\{\*\}\(s\)=\\frac\{\\mathbb\{E\}\\left\[\\\|U\_\{m\}\\\|^\{2\}G\_\{m\}\\mid s\_\{m\-1\}=s\\right\]\}\{\\mathbb\{E\}\\left\[\\\|U\_\{m\}\\\|^\{2\}\\mid s\_\{m\-1\}=s\\right\]\}\.\(73\)bm∗b\_\{m\}^\{\*\}is the scalar state\-only baseline minimizing the trace variance ofgm​\(b\)g\_\{m\}\(b\)\. Under conditional score\-norm independence, it reduces to

bm∗​\(s\)≈𝔼⁡\[Gm∣sm−1=s\]=Vπ​\(s\)\.b\_\{m\}^\{\*\}\(s\)\\approx\\mathbb\{E\}\[G\_\{m\}\\mid s\_\{m\-1\}=s\]=V^\{\\pi\}\(s\)\.\(74\)PPO usesbmPPO=Vπ​\(sm−1\)b\_\{m\}^\{\\mathrm\{PPO\}\}=V^\{\\pi\}\(s\_\{m\-1\}\)\. Define

qmPPO=Tr⁡\(Cov⁡\(Um​\(Gm−Vπ​\(sm−1\)\)\)\)\.q\_\{m\}^\{\\mathrm\{PPO\}\}=\\mathrm\{Tr\}\\\!\\left\(\\mathrm\{Cov\}\\\!\\left\(U\_\{m\}\(G\_\{m\}\-V^\{\\pi\}\(s\_\{m\-1\}\)\)\\right\)\\right\)\.\(75\)qmPPOq\_\{m\}^\{\\mathrm\{PPO\}\}is PPO’s trace\-variance contribution at turnmm\. Under Assumption A, its total variance is

VPPO≈∑m=1MqmPPO\.V\_\{\\mathrm\{PPO\}\}\\approx\\sum\_\{m=1\}^\{M\}q\_\{m\}^\{\\mathrm\{PPO\}\}\.\(76\)

#### GRPO Variance\.

For largeGG, GRPO uses

bmGRPO=μ0​Dm\.b\_\{m\}^\{\\mathrm\{GRPO\}\}=\\mu\_\{0\}D\_\{m\}\.\(77\)Its mismatch from the state value is

δ​Vm=Vπ​\(sm−1\)−μ0​Dm\.\\delta V\_\{m\}=V^\{\\pi\}\(s\_\{m\-1\}\)\-\\mu\_\{0\}D\_\{m\}\.\(78\)δ​Vm\\delta V\_\{m\}measures how far GRPO’s state\-independent group baseline is from the state\-dependent value baseline\. Using

Gm−μ0​Dm=\(Gm−Vπ​\(sm−1\)\)\+δ​Vm,G\_\{m\}\-\\mu\_\{0\}D\_\{m\}=\\left\(G\_\{m\}\-V^\{\\pi\}\(s\_\{m\-1\}\)\\right\)\+\\delta V\_\{m\},\(79\)the conditional cross term vanishes under Assumption B\. Since𝔼⁡\[Um​δ​Vm\]=0\\mathbb\{E\}\[U\_\{m\}\\delta V\_\{m\}\]=0, PPO and unnormalized GRPO have the same per\-turn mean\. Hence

qmGRPO≈qmPPO\+𝔼⁡\[‖Um‖2​δ​Vm2\]\.q\_\{m\}^\{\\mathrm\{GRPO\}\}\\approx q\_\{m\}^\{\\mathrm\{PPO\}\}\+\\mathbb\{E\}\\\!\\left\[\\\|U\_\{m\}\\\|^\{2\}\\delta V\_\{m\}^\{2\}\\right\]\.\(80\)qmGRPOq\_\{m\}^\{\\mathrm\{GRPO\}\}is the corresponding unnormalized GRPO variance contribution\. Under Assumption A,

V~GRPO≈∑m=1MqmGRPO\.\\widetilde\{V\}\_\{\\mathrm\{GRPO\}\}\\approx\\sum\_\{m=1\}^\{M\}q\_\{m\}^\{\\mathrm\{GRPO\}\}\.\(81\)V~GRPO\\widetilde\{V\}\_\{\\mathrm\{GRPO\}\}denotes GRPO variance before division by the global normalization factorσ02\\sigma\_\{0\}^\{2\}\. The normalized estimator satisfies

VGRPO≈V~GRPOσ02\.V\_\{\\mathrm\{GRPO\}\}\\approx\\frac\{\\widetilde\{V\}\_\{\\mathrm\{GRPO\}\}\}\{\\sigma\_\{0\}^\{2\}\}\.\(82\)

#### Variance Comparison\.

Define the accumulated baseline mismatch

Δ=∑m=1M𝔼⁡\[‖Um‖2​δ​Vm2\]≥0\.\\Delta=\\sum\_\{m=1\}^\{M\}\\mathbb\{E\}\\\!\\left\[\\\|U\_\{m\}\\\|^\{2\}\\delta V\_\{m\}^\{2\}\\right\]\\geq 0\.\(83\)Δ\\Deltais the total additional variance caused by GRPO’s baseline mismatch\.Theorem 2\(Baseline\-induced variance gap\)\. Under the stated approximation,

V~GRPO≈VPPO\+Δ,Δ≥0\.\\widetilde\{V\}\_\{\\mathrm\{GRPO\}\}\\approx V\_\{\\mathrm\{PPO\}\}\+\\Delta,\\qquad\\Delta\\geq 0\.\(84\)Proof\.Summing the per\-turn relation forqmGRPOq\_\{m\}^\{\\mathrm\{GRPO\}\}under Assumption A gives

V~GRPO\\displaystyle\\widetilde\{V\}\_\{\\mathrm\{GRPO\}\}≈∑m=1MqmGRPO\\displaystyle\\approx\\sum\_\{m=1\}^\{M\}q\_\{m\}^\{\\mathrm\{GRPO\}\}\(85\)≈∑m=1MqmPPO\+∑m=1M𝔼⁡\[‖Um‖2​δ​Vm2\]\\displaystyle\\approx\\sum\_\{m=1\}^\{M\}q\_\{m\}^\{\\mathrm\{PPO\}\}\+\\sum\_\{m=1\}^\{M\}\\mathbb\{E\}\\\!\\left\[\\\|U\_\{m\}\\\|^\{2\}\\delta V\_\{m\}^\{2\}\\right\]\(86\)≈VPPO\+Δ\.\\displaystyle\\approx V\_\{\\mathrm\{PPO\}\}\+\\Delta\.\(87\)Since every term inΔ\\Deltais non\-negative,Δ≥0\\Delta\\geq 0, with equality iffδ​Vm=0\\delta V\_\{m\}=0almost surely wherever‖Um‖\>0\\\|U\_\{m\}\\\|\>0\.□\\square

For normalized GRPO,

VGRPO≈1σ02​\(VPPO\+Δ\)\.V\_\{\\mathrm\{GRPO\}\}\\approx\\frac\{1\}\{\\sigma\_\{0\}^\{2\}\}\\left\(V\_\{\\mathrm\{PPO\}\}\+\\Delta\\right\)\.\(88\)Thus equality holds iffVπ​\(sm−1\)=μ0​DmV^\{\\pi\}\(s\_\{m\-1\}\)=\\mu\_\{0\}D\_\{m\}almost surely on states with nonzero score norm, for everymm\.

Substituting Theorem 2 into Eq\. equation[71](https://arxiv.org/html/2609.09764#A1.E71)gives

BGRPOBPPO≈VPPOVPPO\+Δ=11\+Δ/VPPO≤1\.\\frac\{B\_\{\\mathrm\{GRPO\}\}\}\{B\_\{\\mathrm\{PPO\}\}\}\\approx\\frac\{V\_\{\\mathrm\{PPO\}\}\}\{V\_\{\\mathrm\{PPO\}\}\+\\Delta\}=\\frac\{1\}\{1\+\\Delta/V\_\{\\mathrm\{PPO\}\}\}\\leq 1\.\(89\)The inequality is strict exactly whenΔ\>0\\Delta\>0\. Thus GRPO has lower gradient signal\-to\-noise ratio whenever its group\-level baseline fails to match the state value\.

### A\.6Effect of Dialogue Horizon on Variance

We now analyze how the horizonMMaffects gradient variance\. Since1/σ01/\\sigma\_\{0\}is a global scale that can be absorbed into the learning rate, we compare PPO with the scale\-adjusted GRPO estimator\. Define the per\-turn terms

qm\(M\)\\displaystyle q\_\{m\}^\{\(M\)\}=Tr⁡\(Cov⁡\(Um​\(Gm−Vπ​\(sm−1\)\)\)\),\\displaystyle=\\mathrm\{Tr\}\\\!\\left\(\\mathrm\{Cov\}\\\!\\left\(U\_\{m\}\(G\_\{m\}\-V^\{\\pi\}\(s\_\{m\-1\}\)\)\\right\)\\right\),\(90\)dm\(M\)\\displaystyle d\_\{m\}^\{\(M\)\}=𝔼⁡\[‖Um‖2​δ​Vm2\]\.\\displaystyle=\\mathbb\{E\}\\\!\\left\[\\\|U\_\{m\}\\\|^\{2\}\\delta V\_\{m\}^\{2\}\\right\]\.\(91\)The superscript\(M\)\(M\)emphasizes that both the return distribution and the baseline mismatch depend on the dialogue horizon\. Hereqm\(M\)q\_\{m\}^\{\(M\)\}is the intrinsic PPO variance at turnmm, anddm\(M\)d\_\{m\}^\{\(M\)\}is GRPO’s additional baseline\-mismatch contribution\. Then

VPPO,M\\displaystyle V\_\{\\mathrm\{PPO\},M\}≈∑m=1Mqm\(M\),\\displaystyle\\approx\\sum\_\{m=1\}^\{M\}q\_\{m\}^\{\(M\)\},\(92\)VGRPO,Madj\\displaystyle V\_\{\\mathrm\{GRPO\},M\}^\{\\mathrm\{adj\}\}≈∑m=1M\(qm\(M\)\+dm\(M\)\)=VPPO,M\+ΔM,\\displaystyle\\approx\\sum\_\{m=1\}^\{M\}\\left\(q\_\{m\}^\{\(M\)\}\+d\_\{m\}^\{\(M\)\}\\right\)=V\_\{\\mathrm\{PPO\},M\}\+\\Delta\_\{M\},\(93\)where

ΔM=∑m=1Mdm\(M\)≥0\.\\Delta\_\{M\}=\\sum\_\{m=1\}^\{M\}d\_\{m\}^\{\(M\)\}\\geq 0\.\(94\)ΔM\\Delta\_\{M\}is the accumulated GRPO variance gap for anMM\-turn trajectory\. Equivalently, with

q¯M=1M​∑m=1Mqm\(M\),d¯M=1M​∑m=1Mdm\(M\),\\bar\{q\}\_\{M\}=\\frac\{1\}\{M\}\\sum\_\{m=1\}^\{M\}q\_\{m\}^\{\(M\)\},\\qquad\\bar\{d\}\_\{M\}=\\frac\{1\}\{M\}\\sum\_\{m=1\}^\{M\}d\_\{m\}^\{\(M\)\},\(95\)q¯M\\bar\{q\}\_\{M\}andd¯M\\bar\{d\}\_\{M\}are the average intrinsic variance and average baseline mismatch per turn, respectively\. Therefore,

VPPO,M≈M​q¯M,VGRPO,Madj≈M⁡\(q¯M\+d¯M\)\.V\_\{\\mathrm\{PPO\},M\}\\approx M\\bar\{q\}\_\{M\},\\qquad V\_\{\\mathrm\{GRPO\},M\}^\{\\mathrm\{adj\}\}\\approx M\(\\bar\{q\}\_\{M\}\+\\bar\{d\}\_\{M\}\)\.\(96\)Thus, ifq¯M\\bar\{q\}\_\{M\}remains bounded away from zero, PPO variance grows withMM\. Ifd¯M\\bar\{d\}\_\{M\}also remains bounded away from zero, GRPO accumulates an additional variance gapΔM=Ω⁡\(M\)\\Delta\_\{M\}=\\Omega\(M\)\. This condition is natural in social dialogue: states at the same turn index may represent acceptance, rejection, negotiation, repair, or failure\. PPO conditions on these states, whereasμ0​Dm\\mu\_\{0\}D\_\{m\}depends only on the global reward mean and remaining discount mass\. Writing the limiting reward standard deviation at horizonMMasσ0,M\\sigma\_\{0,M\}, the normalized GRPO variance is

VGRPO,M≈M⁡\(q¯M\+d¯M\)σ0,M2\.V\_\{\\mathrm\{GRPO\},M\}\\approx\\frac\{M\(\\bar\{q\}\_\{M\}\+\\bar\{d\}\_\{M\}\)\}\{\\sigma\_\{0,M\}^\{2\}\}\.\(97\)σ0,M\\sigma\_\{0,M\}is the population reward standard deviation for trajectories with horizonMM\. Ifσ0,M\\sigma\_\{0,M\}is uniformly bounded above and away from zero, normalization does not change the horizon order\. The relative gap is

VGRPO,MadjVPPO,M≈1\+d¯Mq¯M,BGRPO,MBPPO,M≈11\+d¯M/q¯M\.\\frac\{V\_\{\\mathrm\{GRPO\},M\}^\{\\mathrm\{adj\}\}\}\{V\_\{\\mathrm\{PPO\},M\}\}\\approx 1\+\\frac\{\\bar\{d\}\_\{M\}\}\{\\bar\{q\}\_\{M\}\},\\qquad\\frac\{B\_\{\\mathrm\{GRPO\},M\}\}\{B\_\{\\mathrm\{PPO\},M\}\}\\approx\\frac\{1\}\{1\+\\bar\{d\}\_\{M\}/\\bar\{q\}\_\{M\}\}\.\(98\)Therefore,MMincreases both absolute variances under the conditions above, but the relative gap need not be monotone: it grows, remains constant, or shrinks according tod¯M/q¯M\\bar\{d\}\_\{M\}/\\bar\{q\}\_\{M\}\.

### A\.7Imperfect Value Network

For an imperfect value network, write

V^​\(s\)=Vπ​\(s\)\+ϵ⁡\(s\),\\hat\{V\}\(s\)=V^\{\\pi\}\(s\)\+\\epsilon\(s\),\(99\)ϵ⁡\(s\)\\epsilon\(s\)is the critic’s state\-dependent value\-estimation error\. Its mean squared error is

L=𝔼⁡\[ϵ​\(s\)2\]\.L=\\mathbb\{E\}\[\\epsilon\(s\)^\{2\}\]\.\(100\)LLaverages critic error over the state distribution\. The additional variance over the ideal PPO baseline is

Δϵ=∑m=1M𝔼⁡\[‖Um‖2​ϵ​\(sm−1\)2\]\.\\Delta\_\{\\epsilon\}=\\sum\_\{m=1\}^\{M\}\\mathbb\{E\}\\\!\\left\[\\\|U\_\{m\}\\\|^\{2\}\\epsilon\(s\_\{m\-1\}\)^\{2\}\\right\]\.\(101\)Δϵ\\Delta\_\{\\epsilon\}is the additional PPO variance induced by critic error\. For the detached Monte Carlo baseline, Lemma 1 still applies\. Approximate GAE withλ<1\\lambda<1may introduce additional bias, which is not analyzed\. The scale\-adjusted comparison favors GRPO only if

Δϵ\>Δ,\\Delta\_\{\\epsilon\}\>\\Delta,\(102\)For normalized GRPO, the condition is

VPPO\+Δσ02<VPPO\+Δϵ\.\\frac\{V\_\{\\mathrm\{PPO\}\}\+\\Delta\}\{\\sigma\_\{0\}^\{2\}\}<V\_\{\\mathrm\{PPO\}\}\+\\Delta\_\{\\epsilon\}\.\(103\)Social dialogue produces sharp value changes across negotiation state, repair, commitment history, and failure, makingΔ\\Deltapotentially large\. The practical comparison also depends on critic error andσ0\\sigma\_\{0\}\.

### A\.8Summary

Table[A\.8](https://arxiv.org/html/2609.09764#A1.SS8)summarizes the comparison\.

Table 5:PPO vs\. GRPO for multi\-turn social dialogue\.GRPO’s finite\-group correction decreases as𝒪⁡\(1/G\)\\mathcal\{O\}\(1/G\)\. The main long\-horizon difference is variance: PPO accumulates intrinsic per\-turn variance, whereas GRPO additionally accumulates the state\-value mismatchΔM\\Delta\_\{M\}\.

## Appendix BPrompt Templates

The process reward prompt \(Figure[4](https://arxiv.org/html/2609.09764#A2.F4)\) generates turn\-specific rubric items for six reward dimensions, judges each item as pass or fail, and assigns context\-dependent dimension weights from a baseline prior\.

Figure 4:Process reward model prompt for turn\-level rubric generation, pass/fail judgment, and context\-dependent reward weighting\.The goal\-achievement prompt \(Figure[5](https://arxiv.org/html/2609.09764#A2.F5)\) requires explicit dialogue evidence and returns a binary success score\.

Figure 5:Goal\-achievement evaluation prompt\. The judge requires explicit dialogue evidence and returns a binarygoal\_achievement\_score\.The relationship\-change prompt \(Figure[6](https://arxiv.org/html/2609.09764#A2.F6)\) measures how the counterpart’s favorability changes over the dialogue; the reported scale follows the benchmark, with\[−1,1\]\[\-1,1\]for SOTOPIA\-π\\piand\[−5,5\]\[\-5,5\]for the original SOTOPIA benchmarks\.

Figure 6:Relationship\-change evaluation prompt\. The judge measures the change in the counterpart’s favorability; the original SOTOPIA benchmarks reportrelation\_deltaon\[−5,5\]\[\-5,5\], while SOTOPIA\-π\\piuses\[−1,1\]\[\-1,1\]\.
## Appendix CAdditional Implementation Details

We provide complementary notes for reproducibility\. All experiments use an open\-source reinforcement learning framework\. For each policy–opponent pair, we repeat the evaluation five times and report the mean across repeats together with the corresponding standard deviation; the main tables report these statistics for both Goal Achievement and Relationship Change\.

We additionally evaluate human alignment using three independent annotators\. The annotators score each PRM process record and each ORM dialogue\-level judgment independently, and we use the mean of their ratings as the human reference\. The PRM audit contains 497 process records from 100 dialogue trajectories; the ORM audit contains 100 dialogue\-level judgments from the same number of trajectories\. Correlations with the automatic scores are computed after the corresponding score transformations\.

PPO usesγ=0\.95\\gamma=0\.95, GAEλ=0\.95\\lambda=0\.95, and clipϵ=0\.2\\epsilon=0\.2; learning rates are1×10−51\\times 10^\{\-5\}for the policy and2×10−52\\times 10^\{\-5\}for the value network, both with cosine annealing\. Each batch contains 265 scenes, and each scene samples 4 trajectories, training runs for 100 iterations\. Trajectory lengths typically range from 10 to 20 turns\. Multiple trajectories are sampled per scenario within each iteration, and batches mix fragments across scenarios\.

The Process Reward Model \(PRM\) is Qwen3\.5\-35B\-A3B, and Judge Model is Deepseek\-v4\-flash with temperatureT=0\.1T=0\.1\. Process and outcome reward weights areα=0\.3\\alpha=0\.3andβ=1\\beta=1, selected by validation\-set grid search\. Reward\-weight defenses include whitelist filtering, non\-negativity, per\-dimension caps, fallback to a default stage\-aware prior under degenerate outputs, overflow redistribution, andℓ1\\ell\_\{1\}normalization so weights sum to 1\. All inference and training run on8×8\\timesA100 \(80 GB\) GPUs\.

For fair comparison, Sotopia\-RL, SDPO and ArCHer are re\-implemented on Qwen2\.5\-7B using the same SOTOPIA\-π\\pidata\. Behavior Cloning is trained on about 1,000 expert trajectories generated by GPT\-4o, following the SOTOPIA\-π\\pisetup\([Wang et al\., 2024](https://arxiv.org/html/2609.09764#bib.bib28)\)\. Opponent models use the same role settings and temperatureT=0\.7T=0\.7\.

Figure[7](https://arxiv.org/html/2609.09764#A3.F7)reports the training\-set Goal success rate for all four policy backbones\. We exclude every evaluation record marked bytest\_epochand retain each optimization batch recorded during training\. The four runs same 100 steps for Qwen3\-8B, Qwen2\.5\-7B, LLaMA3\.1\-8B, and Gemma\-3\-4B, respectively\. Light curves show raw batch values, and solid curves use reflection\-padded Gaussian smoothing with a bandwidth proportional to run length\.

Figure 7:Training\-set Goal Reward Mean for the four SocialRL policy backbones\. Light lines are raw non\-test batches, and solid lines are reflection\-padded Gaussian\-smoothed trends\.
## Appendix DAdditional Benchmark Results

We present results in the order SOTOPIA\-All, SOTOPIA\-Hard, SOTOPIA\-π\\pi, and AgentSense\.

### D\.1SOTOPIA\-All

Table[6](https://arxiv.org/html/2609.09764#A4.T6)reports the original SOTOPIA\-All benchmark\. Unlike SOTOPIA\-π\\pi, Goal Achievement is scored on\[0,10\]\[0,10\]and Relationship Change on\[−5,5\]\[\-5,5\]\. Against GPT\-5\.5, SocialRL \(Qwen2\.5\-7B\) reaches7\.16/2\.597\.16/2\.59, above BC \(6\.87/2\.486\.87/2\.48\), Sotopia\-RL \(6\.49/2\.666\.49/2\.66\) and ArCHer \(6\.12/2\.406\.12/2\.40\) on Goal / Relationship\. SocialRL \(Qwen3\-8B\) achieves a7\.78/1\.517\.78/1\.51average, while SocialRL \(LLaMA3\.1\-8B\) gives the strongest trained\-model averages overall \(8\.008\.00Goal and2\.352\.35Relationship Change\)\. Training Gemma\-3\-4B raises its average from5\.60/−0\.285\.60/\-0\.28to7\.42/1\.047\.42/1\.04, gains of1\.821\.82and1\.321\.32on the two metrics\.

Table 6:Results on the SOTOPIA\-ALL benchmark: Goal Achievement Score \(\[0,10\]\[0,10\]\) / Relationship Change \(\[−5,5\]\[\-5,5\]\) for each policy–opponent pair\. Both metrics are reported as mean±\\pmSD across repeat means\. Bold denotes the best SocialRL model per metric and opponent column\.
### D\.2SOTOPIA\-Hard

Table[7](https://arxiv.org/html/2609.09764#A4.T7)reports results on SOTOPIA\-Hard, where Goal Achievement is scored on\[0,10\]\[0,10\]and Relationship Change on\[−5,5\]\[\-5,5\]\. SocialRL \(Qwen3\-8B\) achieves the strongest trained\-model average Goal Achievement score \(7\.077\.07\), while SocialRL \(LLaMA3\.1\-8B\) gives the strongest average Relationship Change \(1\.911\.91\)\. Against GPT\-5\.5, SocialRL \(Qwen3\-8B\) reaches5\.945\.94,0\.830\.83above BC \(5\.115\.11\); its corresponding Relationship Change is0\.260\.26, compared with2\.302\.30for BC\. Gemma\-3\-4B exhibits a benchmark\-specific trade\-off: training raises average Goal Achievement from5\.825\.82to6\.676\.67and Relationship Change from−0\.83\-0\.83to0\.260\.26\.

Table 7:Results on the SOTOPIA\-HARD benchmark: Goal Achievement Score \(\[0,10\]\[0,10\]\) / Relationship Change \(\[−5,5\]\[\-5,5\]\) for each policy–opponent pair\. Both metrics are reported as mean±\\pmSD across repeat means\. Bold denotes the best SocialRL model per metric and opponent column\.
### D\.3SOTOPIA\-π\\pi

Table[8](https://arxiv.org/html/2609.09764#A4.T8)provides the complete SOTOPIA\-π\\piresults, including standard deviations, for the mean\-only table in the main text\.

Table 8:Complete SOTOPIA\-π\\piresults\. Goal Achievement \(%\) / Relationship Change \(×100\\times 100\) are reported as mean±\\pmSD across repeated experiments; Relationship Change values and SDs are multiplied by 100\.
### D\.4AgentSense

Table[9](https://arxiv.org/html/2609.09764#A4.T9)provides the complete AgentSense results, including standard deviations, for the mean\-only table in the main text\.

Table 9:Complete AgentSense results\. Goal Achievement Success Rate \(%\) / Relationship Change \(×100\\times 100\) are reported as mean±\\pmSD across five independent repeats; Relationship Change values and SDs are multiplied by 100\.

## Appendix EAblation Experiments

We answer RQ2 by systematically ablating the main design choices with Qwen2\.5\-7B as the policy model\.

### E\.1PPO versus GRPO

We compare the algorithm choice by contrasting SocialRL’s multi\-turn PPO with the multi\-turn GRPO variant analyzed in Appendix[A](https://arxiv.org/html/2609.09764#A1), using the same Qwen2\.5\-7B backbone and reward design\. The contextual comparison and the main result are shown in Figure[3](https://arxiv.org/html/2609.09764#S4.F3)\. Multi\-turn GRPO obtains53\.4%/0\.43653\.4\\%/0\.436, whereas PPO obtains63\.3%/0\.51663\.3\\%/0\.516, improving Goal Achievement by9\.99\.9percentage points and Relationship Change by0\.0800\.080\. This controlled result supports our theoretical analysis that a state\-dependent value baseline is better suited to delayed, context\-dependent social rewards than group\-normalized GRPO\.

### E\.2Necessity of Process Rewards

We train SocialRL\-NoProcess by removing all intermediate process rewards and retaining only outcome feedback \(Figure[8](https://arxiv.org/html/2609.09764#A5.F8)\)\. To keep this comparison on the same scale as the main experiment, we report Goal Success Rate and Relationship Change against GPT\-5\.5\. The Base Qwen2\.5\-7B model obtains52\.5%/0\.45852\.5\\%/0\.458, BC obtains57\.9%/0\.43757\.9\\%/0\.437, and full SocialRL obtains63\.3%/0\.51663\.3\\%/0\.516\(Goal / Relationship\), using the corresponding GPT\-5\.5 column in Table[2](https://arxiv.org/html/2609.09764#S4.T2)\. The NoProcess variant obtains53\.2%/0\.46053\.2\\%/0\.460: removing process rewards largely eliminates the improvement over the base model and leaves a substantial gap to full SocialRL in goal success\. This result shows that outcome feedback alone is insufficient to supervise intermediate social behavior, whereas dense process rewards provide useful turn\-level signals throughout the dialogue\.

Figure 8:Ablation on process rewards on the SOTOPIA\-π\\pibenchmark \(Qwen2\.5\-7B vs\. GPT\-5\.5\)\. Bars report Goal Success Rate \(%\) / Relationship Change \(\[−1,1\]\[\-1,1\]\)\. Base, BC, and full SocialRL use the corresponding GPT\-5\.5 column from Table[2](https://arxiv.org/html/2609.09764#S4.T2); NoProcess is an additional run with process rewards removed\.
### E\.3Contribution of Each Reward Dimension

We train six leave\-one\-out variants on SOTOPIA\-π\\piwith Qwen2\.5\-7B as the policy and GPT\-5\.5 as the opponent \(Figure[9](https://arxiv.org/html/2609.09764#A5.F9)\)\. Full SocialRL obtains63\.3%/0\.51663\.3\\%/0\.516in Goal Success Rate and Relationship Change\. Removinggoal advancementorstrategic positioningyields the lowest Goal results,55\.3%/0\.48655\.3\\%/0\.486and56\.1%/0\.47956\.1\\%/0\.479, respectively, identifying these dimensions as the main drivers of task progress\. Removingrelational attunementgives61\.8%/0\.40161\.8\\%/0\.401: Goal remains comparatively high, but Relationship Change is the lowest among all variants, consistent with its role in relationship maintenance\. The persona consistency, contextual coherence, and turn quality variants obtain62\.0%/0\.45662\.0\\%/0\.456,62\.6%/0\.47262\.6\\%/0\.472, and63\.1%/0\.48163\.1\\%/0\.481, respectively\. These dimensions support both objectives, although no single one dominates either metric as strongly as the two goal\-side dimensions or relational attunement\.

Figure 9:Leave\-one\-dimension\-out ablations on SOTOPIA\-π\\pi\(Qwen2\.5\-7B vs\. GPT\-5\.5\)\. Bars report the original Goal Success Rate \(%, bottom axis\) and Relationship Change \(\[−1,1\]\[\-1,1\], top axis\); full SocialRL is included as the reference\.
### E\.4Necessity of Dynamic Weights

We compare the proposed dynamic schedule with three fixed\-weight alternatives using the same Qwen2\.5\-7B policy and GPT\-5\.5 opponent \(Figure[10](https://arxiv.org/html/2609.09764#A5.F10)\)\. Values are reported as Goal Success Rate \(%\) / Relationship Change \(\[−1,1\]\[\-1,1\]\)\. TheUniformscheme assigns equal weight to the six reward dimensions at every turn,\[1/6,1/6,1/6,1/6,1/6,1/6\]\[1/6,1/6,1/6,1/6,1/6,1/6\], and obtains58\.7%/0\.47258\.7\\%/0\.472\. TheManualscheme uses a fixed heuristic vector that emphasizes relational attunement early and goal dimensions later; its early, middle, and late vectors are\[0\.10,0\.10,0\.30,0\.15,0\.20,0\.15\]\[0\.10,0\.10,0\.30,0\.15,0\.20,0\.15\],\[0\.20,0\.20,0\.20,0\.15,0\.15,0\.10\]\[0\.20,0\.20,0\.20,0\.15,0\.15,0\.10\], and\[0\.30,0\.25,0\.10,0\.15,0\.10,0\.10\]\[0\.30,0\.25,0\.10,0\.15,0\.10,0\.10\], respectively, and it obtains60\.4%/0\.48760\.4\\%/0\.487\. TheGridscheme selects one constant vector from a coarse grid,\[0\.24,0\.20,0\.14,0\.16,0\.14,0\.12\]\[0\.24,0\.20,0\.14,0\.16,0\.14,0\.12\], and obtains60\.9%/0\.46860\.9\\%/0\.468\. Finally,Dynamicuses the stage\-dependent weights described in Section[3\.3](https://arxiv.org/html/2609.09764#S3.SS3); its result is taken from the SocialRL row against GPT\-5\.5 in Table[2](https://arxiv.org/html/2609.09764#S4.T2),63\.3%/0\.51663\.3\\%/0\.516\. Dynamic weighting is therefore higher than all three fixed alternatives on both reported metrics, with its largest fixed\-schedule margin over Grid being\+2\.4\+2\.4percentage points in Goal Success Rate, and\+0\.029\+0\.029in Relationship Change over Manual\. These results support the claim that adapting the reward focus across a conversation is more effective than selecting one constant weighting scheme\.

Figure 10:Ablation on dynamic versus fixed reward\-weight schedules on the SOTOPIA\-π\\pibenchmark \(Qwen2\.5\-7B vs\. GPT\-5\.5\)\. Bars report Goal Success Rate \(%\) / Relationship Change \(\[−1,1\]\[\-1,1\]\)\. The Dynamic values are taken from the SocialRL row in Table[2](https://arxiv.org/html/2609.09764#S4.T2); the fixed\-weight values are the corresponding comparison runs\.

## Appendix FExtended Qualitative Example

Figure[11](https://arxiv.org/html/2609.09764#A6.F11)shows representative turns from the qualitative comparison used in Section[4\.5](https://arxiv.org/html/2609.09764#S4.SS5)\. Ethan is the evaluated policy—Base Qwen2\.5\-7B\-Instruct or SocialRL Qwen2\.5\-7B—and Benjamin is the GPT\-5\.5 opponent\. Benjamin’s slow turns reduce the group’s enjoyment; Ethan must communicate the need for faster play and propose a concrete solution without offending him\.

Figure 11:Representative turns from the board\-game dialogue\. Benjamin \(GPT\-5\.5\) appears on the left; Ethan’s policy responses appear on the right\. Base preserves rapport but leaves the pacing intervention to Benjamin\. SocialRL operationalizes Benjamin’s opening into a mutually ratified ninety\-second rule\. Both receiverelation​\_​delta=\+0\.6\\mathrm\{relation\\\_delta\}=\+0\.6, while goal success changes from00to11\.#### Intervention ownership\.

The outcome difference is not explained by politeness\. Base is consistently warm, but its first response—“Sure, take your time”—actually relaxes the pressure to address slow play\. Benjamin then diagnoses the problem and supplies the one\-minute limit himself\. Ethan acknowledges that solution but never communicates the group’s need or takes ownership of a concrete intervention\. This distinction explains the judge’s failure decision: the desired social state emerges, but not through the evaluated agent’s goal\-directed action\.

#### From opening to commitment\.

SocialRL does not impose a timer without social permission\. Benjamin first offers an opening \(“If I stall again, call a timer”\), which Ethan converts into an actionable proposal \(“Let’s set a quick timer for your next turn”\)\. Benjamin then specifies ninety seconds, and Ethan explicitly ratifies both the duration and the behavioral boundary \(“no deep dives”\)\. The sequence forms a commitment ladder—permission, proposal, parameterization, and ratification—rather than a single forceful request\. This preserves Benjamin’s agency and makes the rule face\-saving\.

#### Long\-horizon persistence\.

The critical evidence appears after agreement\. Benjamin later announces “timer starts, brain engages, no dissertation” and makes a decisive move, showing behavioral uptake rather than superficial assent\. He subsequently invites future enforcement \(“nudge me”\), extending the convention beyond one turn\. SocialRL therefore establishes a reusable coordination mechanism; Base only reinforces Benjamin’s independent self\-correction\. The later compliance and persistence show that SocialRL’s intervention remains effective beyond the turn in which the timer is proposed\.

#### Goal–relationship balance\.

Both trajectories receive the same positive relationship change of\+0\.6\+0\.6\. Thus the case does not show that SocialRL is simply friendlier than Base; Base is already highly supportive\. Instead, SocialRL improves task agency without paying a relationship cost: goal success rises from00to11while relationship quality is held constant\. Benjamin’s humor, acceptance, appreciation of the kind call\-out, and request for later reminders provide concrete evidence that the firmer coordination rule remains relationally acceptable\.