From Passive Delegates to Strategic Negotiators: Reinforcing Social Reasoning in Small Language Models with SocialRL

# From Passive Delegates to Strategic Negotiators:Reinforcing Social Reasoning in Small Language Models with SocialRL
Source: [https://arxiv.org/html/2608.13787](https://arxiv.org/html/2608.13787)
Wenyue HuaThanks:Corresponding authors: wenyuehua@microsoft\.com, aslic@microsoft\.com, samershi@microsoft\.comAffiliation:Microsoft Research, AI FrontiersTyler PayneAffiliation:Microsoft Research, AI FrontiersSafoora YousefiAffiliation:Microsoft Research, AI FrontiersSaleema AmershiAffiliation:Microsoft Research, AI FrontiersAsli CelikyilmazAffiliation:Microsoft Research, AI Frontiers

###### Abstract

AI agents increasingly act on their users’ behalf as representatives of their interests, handling tasks such as scheduling meetings, comparing offers, and haggling over prices\. These principal\-driven tasks routinely place the agent across from a counterpart, such as another user’s agent, a seller, or a recruiter, whose goals may conflict with those of its principal\. Yet the dispositions that make an assistant pleasant can make it a poor delegate: a friendly and helpful frontier model may disclose its principal’s private information unprompted, and concede the principal’s position at the first sign of resistance\. We presentSocialRL, a general recipe that trains social reasoning directly, and apply it to a 4B model across six interaction domains: Deal\-or\-No\-Deal, CaSiNo, Craigslist, Job Interview, Calendar, and Marketplace\. Every domain is trained in\-domain under the same recipe, and every trained policy is evaluated on all six domains\. We find that \(1\) in\-domain training reaches the frontier: on held\-out scenarios the 4B matches or exceeds the GPT\-5 family per domain, closing 73–122% of the baseline\-to\-frontier gap on the negotiation games, with the change visible at the trace level where 78% of buyer openings anchor below target versus 3% untrained; \(2\) cross\-domain transfer follows game structure: structurally paired games lift each other, a broad multi\-issue donor lifts nearly all domains, and structurally isolated games transfer nothing; \(3\) guided by this transfer structure, we propose two strategies to consolidate the per\-domain specialists into a single unified 4B that achieves 0\.627 average utility across all six environments, matching or exceeding GPT\-4\.1 \(0\.625\), GPT\-5\.1 \(0\.619\), GPT\-5\.2 \(0\.613\): cascade RL and multi\-teacher on\-policy distillation \(OPD\); \(4\) an explicit theory\-of\-mind \(ToM\) scaffold helps only through training: distilling the ToM trace, rather than actions alone, lifts utility on every environment and generalizes better across them, and of the two ToM skills, only next\-action prediction predicts negotiation outcomes\.

## 1Introduction

AI agents are no longer only tools for completing tasks\. A growing share of what they are asked to do carries the user’s stake: the agent acts on the user’s behalf, and the outcome bears directly on the user’s interests\([37](https://arxiv.org/html/2608.13787#bib.bib13);[35](https://arxiv.org/html/2608.13787#bib.bib14)\)\. Deployed systems can manage customer\-service requests, coordinate meetings, purchase goods and book reservations, and help users compare homes or schedule property tours\([23](https://arxiv.org/html/2608.13787#bib.bib1);[14](https://arxiv.org/html/2608.13787#bib.bib2);[33](https://arxiv.org/html/2608.13787#bib.bib3);[58](https://arxiv.org/html/2608.13787#bib.bib4)\)\. These applications position an AI agent as a*delegate*: a system entrusted with a principal’s preferences and authorized to take consequential actions in open\-ended environments\.

Delegation becomes especially challenging when an agent interacts with another party whose objectives differ from those of its principal\. A customer\-service agent\([6](https://arxiv.org/html/2608.13787#bib.bib15)\)may face a user seeking compensation, a purchasing agent may negotiate with a seller seeking a higher price\([15](https://arxiv.org/html/2608.13787#bib.bib16)\), and a scheduling agent\([59](https://arxiv.org/html/2608.13787#bib.bib17)\)may need to reconcile participants with competing constraints\. In such settings, merely completing the interaction is insufficient\. A capable delegate must protect private information\([20](https://arxiv.org/html/2608.13787#bib.bib18)\), infer the counterpart’s incentives\([30](https://arxiv.org/html/2608.13787#bib.bib22)\), determine when to concede or push back, and pursue an agreement that advances its principal’s interests\([36](https://arxiv.org/html/2608.13787#bib.bib23)\)\. We refer to this collection of capabilities as*social reasoning*: reasoning about another actor’s latent preferences, incentives, and likely behavior in order to choose strategically effective actions\.

![Refer to caption](https://arxiv.org/html/2608.13787v1/figs/main.png)Figure 1:SocialRL overview: interaction environments, event\-based agent interface, and decoupled training infrastructure\.\(a\)Example social\-reasoning environments for delegated coordination and negotiation\.\(b\)An event\-based, agent\-independent interface in which a stateful environment communicates with agents through private channels of observations, notifications, and actions\.\(c\)A decoupled training stack where the rollout proxy connects the environment and agent harness to inference and training, enabling interchangeable components and supporting both reinforcement learning and distillation\.Negotiation has long served as a testbed for language\-based strategic interaction, including multi\-item allocation, resource division, and price bargaining\([24](https://arxiv.org/html/2608.13787#bib.bib8);[16](https://arxiv.org/html/2608.13787#bib.bib9);[7](https://arxiv.org/html/2608.13787#bib.bib19)\)\. Recent work has extended these settings to evaluate the agency and strategic behavior of general\-purpose language models\([5](https://arxiv.org/html/2608.13787#bib.bib10);[53](https://arxiv.org/html/2608.13787#bib.bib11);[17](https://arxiv.org/html/2608.13787#bib.bib12)\)\. Although frontier models can conduct coherent negotiations and frequently reach agreements, strong aggregate outcomes do not guarantee faithful representation of the principal\. We find that frontier models routinely abandon contested positions after limited resistance\. These behaviors reflect dispositions that are useful for general assistance, such as transparency, agreeableness, and an eagerness to reach consensus, but they leave an agent predictable and exploitable when acting as a delegate\.

These failures reflect a mismatch between the objectives used to train general\-purpose assistants and those required for strategic delegation\. Instruction tuning and reinforcement learning from human feedback are typically designed to produce broadly helpful\([8](https://arxiv.org/html/2608.13787#bib.bib45)\), honest, harmless, and instruction\-following behavior in cooperative dialogue\([21](https://arxiv.org/html/2608.13787#bib.bib42);[54](https://arxiv.org/html/2608.13787#bib.bib43);[9](https://arxiv.org/html/2608.13787#bib.bib44)\)\. In interactions with conflicting objectives and private information, the same behavioral priors can manifest as premature disclosure, excessive accommodation, and a preference for agreement even when the resulting outcome is unfavorable to the principal\. Delegated agency therefore requires principal\-conditioned strategic behavior: selective information disclosure, calibrated reservation boundaries, and the willingness to reject or prolong an interaction when agreement would sacrifice the principal’s utility\. This objective mismatch motivates post\-training that targets social reasoning directly\.

We presentSocialRL, a complete infrastructure \+ recipe for training and studying social reasoning in language\-model agents, as shown in Figure[1](https://arxiv.org/html/2608.13787#S1.F1)\. We focus on a 4B model to test whether strategic delegation can be induced through targeted post\-training rather than depending on frontier\-scale capacity\. UsingSocialRL, the resulting 4B policies approach and, in some settings, exceed the performance of much larger GPT models while learning strategies that generalize across interaction structures\. We train and evaluate across six interaction domains: Deal\-or\-No\-Deal\([24](https://arxiv.org/html/2608.13787#bib.bib8)\), CaSiNo\([7](https://arxiv.org/html/2608.13787#bib.bib19)\), Job Interview\([44](https://arxiv.org/html/2608.13787#bib.bib20)\), Craigslist\([16](https://arxiv.org/html/2608.13787#bib.bib9)\), and Calendar and Marketplace from SocialReasoning\-Bench\([31](https://arxiv.org/html/2608.13787#bib.bib21)\)\. Together, these domains span single\- and multi\-issue bargaining, price negotiation\([57](https://arxiv.org/html/2608.13787#bib.bib46)\), and slot coordination\. To support training in these multi\-turn environments, we build a decoupled system\([52](https://arxiv.org/html/2608.13787#bib.bib47);[40](https://arxiv.org/html/2608.13787#bib.bib48);[27](https://arxiv.org/html/2608.13787#bib.bib49)\)that separates the environment, agent harness, inference engine, and trainer, allowing a common interaction infrastructure to support reinforcement learning, distillation, and heterogeneous or black\-box counterparts\. We first train a specialist within each domain and evaluate the full cross\-environment transfer matrix, then consolidate the specialists into a single policy across all six environments using cascade RL\([10](https://arxiv.org/html/2608.13787#bib.bib53)\)as well as multi\-teacher mode\-seeking OPD\.

On held\-out scenarios, our domain\-trained 4B policies achieve performance comparable to much larger GPT models across the six environments\. Cross\-environment transfer is substantial and follows interaction structure, with the strongest transfer occurring between structurally similar domains\. We consolidate these specialized policies through cascade reinforcement learning and multi\-teacher OPD \(MOPD\), producing a unified 4B model that achieves 0\.627 average utility across all six environments, comparable to GPT\-4\.1 at 0\.625, GPT\-5\.1 at 0\.619, GPT\-5\.2 at 0\.613\. Finally, we introduce explicit theory\-of\-mind supervision throughInfer→\\toAct→\\toAnticipate\. Distilling these reasoning traces improves performance and cross\-environment generalization, with next\-action prediction emerging as the theory\-of\-mind skill most predictive of negotiation outcomes\.

##### Contributions\.

- •Strategic Delegation Formulation & Six\-Environment Suite\.We formulate strategic delegation as a social\-reasoning post\-training problem and introduce a heterogeneous suite spanning allocation, multi\-issue bargaining, price negotiation, and preference\-based coordination\. We train domain specialists in each environment and evaluate every policy across the full cross\-environment transfer matrix\.
- •Agent\-Agnostic Environments & Decoupled Training Infrastructure\.We developSocialRL, combining an event\-based, agent\-independent environment interface with an OpenAI\-compatible rollout proxy that disentangles environments, agent harnesses, inference engines, and trainers\. The same stack supports reinforcement learning and distillation with local, remote, heterogeneous, or black\-box counterparts\.
- •Frontier\-Range 4B Policies & Transfer\-Aware Unification\.We show that domain\-trained 4B policies achieve aggregate performance in the range of GPT\-4\.1, GPT\-5\.1, and GPT\-5\.2, and reveal that cross\-environment transfer is strongly directional and structure\-dependent\. We exploit this transfer structure to consolidate the specialists: transfer\-aware cascade RL reaches0\.6270\.627Avg\-6, while multi\-teacher on\-policy distillation recovers92\.6%92\.6\\%of the specialists’ average advantage in only6060additional optimization steps\.
- •Explicit Theory\-of\-Mind Supervision\.We introduceInfer→\\rightarrowAct→\\rightarrowAnticipatesupervision and show that distilling the complete reasoning trace outperforms action\-only supervision on every evaluated negotiation environment and improves cross\-environment generalization\. We further identify next\-action prediction, rather than preference inference alone, as the theory\-of\-mind component most predictive of negotiation outcomes\.

## 2Related Work

##### Language\-model agents\.

A large body of work on language\-model agents has focused on enabling models to reason, plan, and act through external tools and digital interfaces\. ReAct\([49](https://arxiv.org/html/2608.13787#bib.bib24)\)introduced the interleaving of language\-based reasoning with environment actions, while Toolformer\([34](https://arxiv.org/html/2608.13787#bib.bib25)\)demonstrated that language models can learn when and how to invoke external APIs\. Subsequent benchmarks have evaluated increasingly realistic forms of web navigation, desktop control, and tool\-mediated interaction, including WebArena, OSWorld, andτ\\tau\-bench\([55](https://arxiv.org/html/2608.13787#bib.bib5);[43](https://arxiv.org/html/2608.13787#bib.bib6);[48](https://arxiv.org/html/2608.13787#bib.bib7)\)\. This literature has substantially advanced planning\([41](https://arxiv.org/html/2608.13787#bib.bib27)\), tool selection\([46](https://arxiv.org/html/2608.13787#bib.bib28)\), interface grounding\([3](https://arxiv.org/html/2608.13787#bib.bib26)\), and policy compliance\([11](https://arxiv.org/html/2608.13787#bib.bib29)\), with performance generally measured by whether the agent reaches a target environment state\. Strategic interaction introduces an additional dimension: the environment contains another adaptive decision\-maker\([2](https://arxiv.org/html/2608.13787#bib.bib30)\), so the value of an action depends on its immediate effect, the information it reveals, and the future responses it induces\.

##### LLMs as delegated agents\.

LLMs are increasingly studied as delegates that act on behalf of users or organizations in economically and socially consequential settings\.τ\\tau\-bench\([48](https://arxiv.org/html/2608.13787#bib.bib7)\)models agents handling retail and airline customer\-service requests under domain\-specific policies, while ScheduleMe\([2](https://arxiv.org/html/2608.13787#bib.bib30)\)applies multi\-agent coordination to personal calendar management\. CalBench further studies calendar assistants that coordinate under private information, exposing trade\-offs between scheduling efficiency, fairness, communication, and privacy\([59](https://arxiv.org/html/2608.13787#bib.bib17)\)\. In commerce, ACES evaluates agents that inspect marketplaces and make product choices on behalf of consumers\([1](https://arxiv.org/html/2608.13787#bib.bib31)\); in negotiation, recent work has compared advisory, coaching, and autonomous delegation interfaces in multi\-party bargaining\. These settings require the agent to preserve and act on a principal’s preferences while interacting with users, platforms, or other agents whose objectives may differ\. Negotiation is a particularly direct instance of delegated agency because success depends jointly on reaching an agreement, managing private information, and securing value for the represented principal\.

##### Training agents for delegated interaction\.

A growing line of work trains language agents through strategic interaction\. Within negotiation, prior work has explored self\-play and language feedback, iterative self\-play with behavior cloning, reinforcement learning with verifiable economic rewards, and pipelines combining supervised training on synthetic negotiations with reinforcement learning\([13](https://arxiv.org/html/2608.13787#bib.bib32);[25](https://arxiv.org/html/2608.13787#bib.bib33);[26](https://arxiv.org/html/2608.13787#bib.bib34);[4](https://arxiv.org/html/2608.13787#bib.bib35)\)\. These methods can induce substantially stronger bargaining strategies, but typically specialize to a particular interaction structure, such as bilateral price bargaining or resource division\. Complementary work has broadened the scope of interactive post\-training:Sotopia\-π\\piandSotopia\-RLtrain general social behavior from open\-ended interactions\([39](https://arxiv.org/html/2608.13787#bib.bib36);[50](https://arxiv.org/html/2608.13787#bib.bib37)\), while recent self\-play methods train transferable multi\-agent reasoning across cooperative and competitive games\([51](https://arxiv.org/html/2608.13787#bib.bib39);[19](https://arxiv.org/html/2608.13787#bib.bib38);[28](https://arxiv.org/html/2608.13787#bib.bib40)\)\. Evaluation frameworks have likewise highlighted the diversity of negotiation itself, spanning resource allocation, exchange, price bargaining, and realistic multi\-issue scenarios\([5](https://arxiv.org/html/2608.13787#bib.bib10);[56](https://arxiv.org/html/2608.13787#bib.bib41)\)\.

SocialRLfocuses on the intersection of these directions: post\-training a single small model across heterogeneous delegated interactions that demand different strategic capabilities but share a common need to reason about counterpart incentives and act on behalf of a principal\. We study transfer across these interaction structures and show that the resulting 4B policies reach the performance range of much larger GPT models\.

## 3Environment and Infrastructure for SocialRL

Training agents through negotiation requires infrastructure that differs from conventional single\-turn language\-model post\-training\. An environment may be stateful, partially observed, and populated by multiple agents acting on different schedules\. Each episode contains several model calls, while the reward is often available only after the complete interaction\. In addition, experiments may combine local trainable policies, remote frontier\-model opponents, scripted agents, and human participants\. We therefore design the system around two forms of decoupling: an event\-based interface separates environments from agents, and an OpenAI\-compatible proxy separates agent execution from model training\. Figure[2](https://arxiv.org/html/2608.13787#S3.F2)shows the environment stack, while Figure[3](https://arxiv.org/html/2608.13787#S3.F3)shows the training data flow\.

### 3\.1Environment and Agent Interface

We design a generic interface for multi\-agent interaction that supports sequential and simultaneous decisions, asynchronous communication, and arbitrary numbers of participants\. The same abstraction can represent two\-party negotiation environments such as Deal\-or\-No\-Deal and Craigslist, as well as more complex multi\-party environments such as Werewolf and Avalon\. The design follows four principles: environments are independent of agent implementation; interaction is represented as an event stream with explicit action semantics; agents consume this stream asynchronously; and failures are handled without terminating the episode\.

##### Agent\-independent environments\.

The environment is a stateful process that owns the game rules, hidden state, legal actions, and reward computation, while making no assumptions about how participants are implemented\. Each participant interacts with the environment through a private asynchronous channel and is treated as a black box implementing a single operation,decide\(context, actions\)→\\rightarrowaction\. An agent may therefore be backed by a language model, an external coding agent, a scripted policy, or a human interface\. We further separate the model layer from the agent harness: model inference exposes only standard chat messages and tool definitions through an OpenAI\-compatible API\. This separation allows the environment, agent implementation, model backend, and execution transport to vary independently\.

##### Event\-based interaction and action semantics\.

Communication between the environment and an agent is represented as an ordered stream of events rather than repeated snapshots of environment state\. We distinguish two event types: an*observation*records something that has occurred, while a*notification*requests a decision from the receiving agent and specifies its currently available actions\. The environment drives interaction through three primitives:ask\(\)for sequential decisions,ask\_all\(\)for simultaneous decisions, andbroadcast\(\)for events requiring no response\. The control flow of the environment coroutine therefore expresses the game logic directly, without requiring a separate state\-machine or dispatcher abstraction\.

An event stream preserves information that is important in partially observable multi\-agent interaction: what happened, when it happened, and which participants observed it\. It also naturally separates an agent’s*action*from its*effect*\. Submitting an action records a decision and returns only an acknowledgment or validation error; its consequences are emitted subsequently as observations\. This distinction is important because actions need not map one\-to\-one to outcomes\. An outcome may depend on several participants’ actions, occur only after additional environment logic, or be visible to different participants in different ways\. For example, a vote is an individual player’s action, whereas an elimination is a collective outcome produced after all votes have been received\.

Partial observability is enforced at this same boundary\. Each emitted event is filtered for its intended audience, so an agent’s channel contains exactly the information available to that participant\. The resulting event history can be shared by agent execution, player\-facing interfaces, replay tools, and post\-episode analysis, reducing the need for separate state projections or logs that may diverge from what the agent actually observed\.

![Refer to caption](https://arxiv.org/html/2608.13787v1/figs/sage_architecture.png)Figure 2:Environment and agent architecture\.A stateful environment communicates with black\-box agents through per\-agent asynchronous channels carrying structured observations, notifications, and actions\. The same interface supports LLM agents, external coding agents, humans, and scripted policies\. Model inference is isolated behind an OpenAI\-compatible interface, while the event stream supports agent execution, human play, observability, replay, and trajectory collection\.
##### Asynchronous event consumption\.

For local execution, agents consume their event streams directly through asynchronous iterators\. This interface matches the execution pattern of multi\-agent interaction: events arrive incrementally, agents may spend different amounts of time computing their decisions, and progress may depend on responses from other participants\. Asynchronous consumption allows agents and the environment to execute concurrently while preserving the ordering of events, without requiring polling or coupling agent execution to the environment’s control flow\. Remote agents access the same logical stream through a cursor\-based HTTP interface\. Events are stored in an append\-only history and indexed by sequence number, allowing a disconnected client to resume from its last cursor without losing observations\. Local and remote execution therefore share the same interaction semantics and differ only in transport\.

##### Failure and concurrency handling\.

The interface is designed so that transient agent and transport failures do not invalidate an episode\. Every submitted action identifies the notification to which it responds\. If the environment has advanced while an agent was generating its response, the action is rejected as stale and the agent consumes the newer events before deciding again\. Together with the append\-only event history, this provides resumable event delivery while ensuring that at most one action is accepted for each decision point\.

Invalid actions, timeouts, and unavailable agents are handled explicitly by the environment\. Invalid actions generate informative events and may be retried, while a timeout applies an inert default action and records the failure in the event stream\. A malformed response, failed model request, or dropped connection therefore becomes part of the trajectory rather than terminating the episode\. This property is particularly important for large\-scale rollout collection, where a single interaction may involve several independently executing models and individual requests may fail or exceed their latency budgets\.

![Refer to caption](https://arxiv.org/html/2608.13787v1/figs/train_diagram.png)Figure 3:Decoupled training architecture\.The agent runtime executes independently of the trainer and sends standard model requests through a rollout proxy\. The proxy forwards requests to the inference engine while recording the authentic model inputs and outputs associated with each episode\. Completed trajectories are written to a shared Parquet buffer consumed by interchangeable training backends, and updated policy weights are synchronized back to the inference engine\.

### 3\.2Decoupled Training Infrastructure

Our second design goal is to disentangle rollout generation from training\. Agentic rollouts may involve arbitrary environment logic, agent harnesses, memory systems, tool use, and remote counterparts, while the trainer should only require the model inputs, model outputs, and eventual rewards needed to optimize the policy\. We therefore connect rollout and training through a single abstraction: an OpenAI\-compatible*rollout proxy*placed between the agent runtime and the inference endpoint\.

##### Rollout\-training separation\.

The environment and agent harness execute exactly as they would at inference time\. The environment determines observations, actions, and rewards; the agent decides how to construct prompts, manage memory, invoke tools, and interpret model outputs\. Neither contains training\-specific logic\. From the trainer’s perspective, these implementation details are invisible: it consumes trajectories captured at the model API boundary and therefore need not know how the environment or agent harness is implemented\.

This separation is particularly useful for agentic interaction, where a single episode may contain many model calls, the terminal reward may become available only after the interaction ends, and counterparts may be remote or black\-box systems outside the training process\. It also allows the rollout stack and training stack to evolve independently\.

##### The rollout proxy as the training boundary\.

The proxy exposes the same OpenAI\-compatible interface as the underlying inference endpoint, so an agent can use it by changing only the client’s base URL\. For every model call, the proxy records the exact request received from the agent and the exact response returned by the model, while forwarding the response to the agent unchanged\. It therefore captures the authentic model inputs and outputs produced during execution, without requiring the harness to construct a separate training representation\.

Each episode is associated with a unique session key carried with every model request\. The session groups otherwise stateless HTTP calls into a trajectory\. When an episode begins, the runtime opens a session; once the interaction terminates, it attaches the final reward and closes the session\. Many rollout workers can consequently execute concurrently against a shared continuously batched inference server without exposing their internal control flow to the trainer\.

##### Trajectory reconstruction and memory compaction\.

Importantly, the proxy does not assume that the agent maintains its context in any particular way\. Agent harnesses commonly truncate histories, summarize earlier interactions, or otherwise compact memory during long rollouts\. In such cases, a later model request may no longer extend the prefix of an earlier request\. We handle this directly at the model\-call boundary\. Consecutive calls whose contexts share the expected prefix are treated as part of the same training segment\. When this prefix relation breaks, the proxy starts a new segment\. Thus, context truncation, summarization, and memory compaction require no special integration with the trainer: they simply induce multiple training segments within the same episode\. Each segment retains the exact context under which its response was generated, avoiding any need to reconstruct an artificial conversation history that the policy never actually observed\.

##### A common contract for RL and distillation\.

The proxy supports both reinforcement learning and distillation while preserving the same rollout interface\. For reinforcement learning, it records tokenized prompts and responses together with token\-level log probabilities from the generating policy\. Capturing these probabilities at generation time preserves the behavior policy required by PPO even if the policy is updated before optimization\. For distillation, the proxy stores the raw messages, tool definitions, responses, and tool calls, allowing trajectories from closed\-source or heterogeneous teachers to be tokenized later using the student’s tokenizer\.

Completed sessions are exported to a shared Parquet buffer that forms the data contract between rollout and training\. RL records include response tokens, rollout log probabilities, rewards, task identifiers, and policy versions; distillation records preserve the raw interaction trace\. Training losses are applied only to model response tokens\. Because trainers consume this common representation, the rollout system can be paired with a lightweight local trainer, distributed PPO through veRL, alternative algorithms through phitrain, or external training backends through a small adapter\.

##### Asynchronous rollout and optimization\.

Rollout collection, optimization, and inference proceed asynchronously\. Episode workers continuously generate trajectories and append completed sessions to the buffer; the trainer consumes them in batches, updates the policy, and synchronizes new weights to the inference server\. The rollout and training systems can therefore scale independently\. Asynchrony can introduce policy staleness when rollout generation runs ahead of optimization\. We bound this effect with a capacity\-based gate that limits the number of trajectories dispatched ahead of the trainer\. Each trajectory is tagged with the policy version that generated it, while remaining off\-policy differences are handled by the importance\-sampling correction used during PPO\. Together with the environment interface in Section[3\.1](https://arxiv.org/html/2608.13787#S3.SS1), this design separates the major axes of an agent\-training system: environments define interactions, harnesses define agent behavior, the proxy records authentic model calls, and trainers optimize over the resulting trajectories\. Changing any one of these components does not require rewriting the others\.

## 4Social Reasoning Environments and Reward Design

SocialRLtrains agents across six heterogeneous negotiation and coordination environments: Deal\-or\-No\-Deal, CaSiNo, Craigslist Bargains, Job Interview, Calendar, and Marketplace\. Our training recipe proceeds in two stages\. We first train a domain\-specialized policy in each environment, then consolidate these specialists into a single policy that performs well across all six domains\. We study two consolidation strategies in Section[5](https://arxiv.org/html/2608.13787#S5): cascade reinforcement learning, whose training order is chosen according to measured cross\-domain transfer, and MOPD, which provides a substantially more efficient route to unification\.

The six environments are deliberately heterogeneous\. They span multi\-issue allocation, multi\-issue contract negotiation, single\-issue price bargaining, and preference\-based coordination, with different action spaces, utility structures, information asymmetries, and interaction dynamics\. Their common structure is social reasoning: in every domain, an agent must reason about a counterpart with private objectives, protect information about its own principal, and decide when to propose, concede, reject, or close\. This diversity lets us study both specialization within individual interaction structures and transfer of social\-reasoning strategies across them\. Table[1](https://arxiv.org/html/2608.13787#S4.T1)summarizes the six environments\.

Table 1:The six social\-reasoning environments\.The suite spans heterogeneous negotiation and coordination structures while sharing a common requirement to reason strategically about a counterpart with private objectives\.### 4\.1Interaction Environments

##### Deal\-or\-No\-Deal \(DnD\)\.

Two agents divide a shared pool of items \(books, hats, and balls\), each holding private per\-item values\([24](https://arxiv.org/html/2608.13787#bib.bib8)\)\. A scenario specifies an item multiset and each player’s valuation, with every player’s total valuation normalized to1010points\. Agents alternate free\-form messages and structured proposals until a split is accepted, one side walks away, or the round limit is reached\. Because valuations are private, effective play requires identifying which items the counterpart values and trading low\-value items for high\-value ones\.

##### CaSiNo\.

Two campers divide a fixed stock of food, water, and firewood packages, with three units of each resource\([7](https://arxiv.org/html/2608.13787#bib.bib19)\)\. Each player has a private priority ordering over the resources, grounded in a persona and backstory\. Priorities map to per\-unit utilities \(High=5=5, Medium=4=4, Low=3=3\), giving each player a maximum score of3636\. CaSiNo shares DnD’s multi\-issue allocation structure while requiring agents to negotiate through naturalistic needs and justifications\.

##### Craigslist Bargains\.

A buyer and seller negotiate the price of a single listed item\([16](https://arxiv.org/html/2608.13787#bib.bib9)\)\. Scenarios are derived from Craigslist postings across multiple product categories, with private price objectives defining the bargaining range\. The environment is one\-dimensional and role\-asymmetric: the buyer seeks a lower price while the seller seeks a higher one\. Effective negotiation therefore depends on anchoring, concession timing, and deciding when the available surplus justifies agreement\.

##### Job Interview\.

A worker and recruiter negotiate a five\-issue employment package covering salary, weekly holiday, position, workplace, and company\([44](https://arxiv.org/html/2608.13787#bib.bib20)\)\. Each side has private utilities and issue weights, and the parties generally value different dimensions of the contract\. Efficient agreements therefore require identifying the counterpart’s priorities and exchanging concessions on low\-value issues for gains on high\-value ones\. With roughly10410^\{4\}possible agreements per scenario, Job Interview provides the largest structured outcome space in the suite\.

##### Calendar\.

Calendar is adapted from SocialReasoning\-Bench\([31](https://arxiv.org/html/2608.13787#bib.bib21)\)\. An agent acts on behalf of a principal to negotiate a meeting time with a requestor whose preferences differ from the principal’s\. Each side has private preferences over available time slots, and the initial request conflicts with the principal’s interests\. The agent must gather enough information about the requestor’s flexibility while steering the interaction toward a slot favorable to its principal\.

##### Marketplace\.

Marketplace is also adapted from SocialReasoning\-Bench\([31](https://arxiv.org/html/2608.13787#bib.bib21)\)\. A buyer agent negotiates a purchase with a seller while holding a private reservation price\. The seller begins from a price unfavorable to the buyer, while a zone of possible agreement is guaranteed to exist\. As in Craigslist, success requires strategic price negotiation, but the environment differs in its interface, counterpart behavior, and utility construction, making the pair useful for studying transfer across structurally related domains\.

### 4\.2Outcome Evaluation and Reward Design

All six environments use terminal, outcome\-only rewards: each episode receives a single scalar after agreement, walk\-away, or timeout, with no intermediate reward shaping\. Our reward design follows two principles\. First, every environment maps outcomes to a common\[0,1\]\[0,1\]range, providing consistent endpoints for cross\-environment reporting, aggregation, and checkpoint selection during multi\-domain training\. Second, the reward should reflect the quality of an agreement relative to the opportunities available in the particular scenario and role\. The appropriate normalization therefore depends on the interaction structure: DnD, CaSiNo, and Job Interview use scenario\- and role\-specific reference agreements; Craigslist and Marketplace normalize by the available price\-negotiation corridor; and Calendar normalizes by the value range over mutually feasible time slots\.

For a fixed scenario and reference pointmm, the transformation is strictly increasing inzzand therefore preserves the ranking of outcomes within that scenario\. Its purpose is to calibrate how outcomes are valued across scenarios\. For example, a normalized utility of0\.60\.6may exceed the fair\-and\-efficient reference in a highly conflicting scenario while falling well below it in a scenario with largely compatible preferences\. Centering the reward curve atmmassigns the greatest resolution to the transition between weak and strong agreements under the scenario\-specific benchmark\.

##### Difficulty\-aware rewards for multi\-issue negotiation\.

DnD, CaSiNo, and Job Interview evaluate an outcome relative to a scenario\-specific reference agreement\. This adjustment accounts for variation in the utility that a player can reasonably attain under different combinations of private preferences\.

*Allocation games\.*In DnD and CaSiNo, a scenario specifies item counts\{ck\}\\\{c\_\{k\}\\\}and private per\-item valuesvav^\{a\}andvbv^\{b\}\. For an allocationxx, where playeraareceivesxkx\_\{k\}units of itemkk, the two players obtain

sa​\(x\)=∑kxk​vka,sb​\(x\)=∑k\(ck−xk\)​vkb\.s\_\{a\}\(x\)=\\sum\_\{k\}x\_\{k\}v^\{a\}\_\{k\},\\qquad s\_\{b\}\(x\)=\\sum\_\{k\}\(c\_\{k\}\-x\_\{k\}\)v^\{b\}\_\{k\}\.\(1\)The maximum raw score isS=10S=10in DnD andS=36S=36in CaSiNo\. Because the allocation spaces are small, we enumerate all feasible outcomes and compute their Pareto frontiers exactly\.

We use allocations that are both Pareto\-optimal and envy\-free to define the reference point\. An allocation is envy\-free when each player weakly prefers its own bundle to the other player’s bundle under its private valuation\. For playerii, we define the*envy\-free Pareto maximum*

mi=1S​max⁡\{si​\(x\):x​is Pareto\-optimal and envy\-free\}\.m\_\{i\}=\\frac\{1\}\{S\}\\max\\left\\\{s\_\{i\}\(x\):x\\text\{ is Pareto\-optimal and envy\-free\}\\right\\\}\.\(2\)This reference is the greatest normalized value available to playeriiamong outcomes that remain both efficient and fair\. In scenarios for which no allocation satisfies both conditions, we use the Pareto\-optimal allocation that minimizes the larger of the two players’ envy violations\.

*Job Interview\.*Each Job Interview outcome is a dealddassigning one option to each issue\. The worker and recruiter receive normalized utilitiesuw​\(d\),ur​\(d\)∈\[0,1\]u\_\{\\mathrm\{w\}\}\(d\),u\_\{\\mathrm\{r\}\}\(d\)\\in\[0,1\]derived from their private issue weights and option utilities\. Because the complete outcome space is enumerable, we define the reference agreement through the egalitarian objective

d∗=arg⁡maxd⁡min⁡\(uw​\(d\),ur​\(d\)\),d^\{\*\}=\\arg\\max\_\{d\}\\min\\left\(u\_\{\\mathrm\{w\}\}\(d\),u\_\{\\mathrm\{r\}\}\(d\)\\right\),\(3\)breaking ties in favor of greater total utility\. The role\-specific reference points are

mw=uw​\(d∗\),mr=ur​\(d∗\)\.m\_\{\\mathrm\{w\}\}=u\_\{\\mathrm\{w\}\}\(d^\{\*\}\),\\qquad m\_\{\\mathrm\{r\}\}=u\_\{\\mathrm\{r\}\}\(d^\{\*\}\)\.\(4\)
*Common difficulty\-aware transformation\.*Letz∈\[0,1\]z\\in\[0,1\]denote the agent’s normalized outcome utility andm∈\[0,1\]m\\in\[0,1\]its scenario\- and role\-specific reference point\. DnD, CaSiNo, and Job Interview use

r=σ⁡\(z−mT\)−σ⁡\(−mT\)σ⁡\(1−mT\)−σ⁡\(−mT\),σ⁡\(x\)=11\+e−x,T=0\.2\.r=\\frac\{\\sigma\\\!\\left\(\\frac\{z\-m\}\{T\}\\right\)\-\\sigma\\\!\\left\(\\frac\{\-m\}\{T\}\\right\)\}\{\\sigma\\\!\\left\(\\frac\{1\-m\}\{T\}\\right\)\-\\sigma\\\!\\left\(\\frac\{\-m\}\{T\}\\right\)\},\\qquad\\sigma\(x\)=\\frac\{1\}\{1\+e^\{\-x\}\},\\qquad T=0\.2\.\(5\)The transformation preservesr=0r=0atz=0z=0andr=1r=1atz=1z=1, while placing the steepest part of the curve atz=mz=m\. Outcomes near the reference agreement therefore receive the greatest reward discrimination\. The same raw utility can represent strong play in a highly conflicting scenario and a weak agreement in a scenario with largely compatible preferences\.

This normalization retains a common absolute reward range while accounting for what was reasonably attainable in each scenario and role\. It therefore gives PPO a more comparable signal of agreement quality across scenarios with different utility frontiers\.

##### Price\-corridor rewards for single\-issue negotiation\.

Craigslist and Marketplace both measure how much of a one\-dimensional bargaining surplus each party captures\. Letpminp\_\{\\min\}denote the buyer\-favorable endpoint of the bargaining corridor,pmaxp\_\{\\max\}the seller\-favorable endpoint, andpdealp\_\{\\mathrm\{deal\}\}the agreed price, withpmin<pmaxp\_\{\\min\}<p\_\{\\max\}\. We define

rbuyer=clip⁡\(pmax−pdealpmax−pmin,0,1\),rseller=clip⁡\(pdeal−pminpmax−pmin,0,1\)\.r\_\{\\mathrm\{buyer\}\}=\\operatorname\{clip\}\\left\(\\frac\{p\_\{\\max\}\-p\_\{\\mathrm\{deal\}\}\}\{p\_\{\\max\}\-p\_\{\\min\}\},0,1\\right\),\\qquad r\_\{\\mathrm\{seller\}\}=\\operatorname\{clip\}\\left\(\\frac\{p\_\{\\mathrm\{deal\}\}\-p\_\{\\min\}\}\{p\_\{\\max\}\-p\_\{\\min\}\},0,1\\right\)\.\(6\)For any agreement within the corridor, the two rewards sum to11\. A deal atpminp\_\{\\min\}assigns the full available surplus to the buyer, while a deal atpmaxp\_\{\\max\}assigns it to the seller\.

In Craigslist, the corridor is defined by the buyer’s target priceptp\_\{t\}and the listing pricepℓp\_\{\\ell\}:

pmin=pt,pmax=pℓ\.p\_\{\\min\}=p\_\{t\},\\qquad p\_\{\\max\}=p\_\{\\ell\}\.\(7\)The resulting reward measures how much of the listing\-to\-target bargaining range each role captures\.

In Marketplace, the corridor is the zone of possible agreement defined by the seller’s and buyer’s private reservation prices\([31](https://arxiv.org/html/2608.13787#bib.bib21)\)\. Letpsp\_\{s\}be the seller’s reservation price andpbp\_\{b\}the buyer’s reservation price, withps<pbp\_\{s\}<p\_\{b\}\. We set

pmin=ps,pmax=pb\.p\_\{\\min\}=p\_\{s\},\\qquad p\_\{\\max\}=p\_\{b\}\.\(8\)The trained Marketplace delegate represents the buyer, so its terminal reward isrbuyerr\_\{\\mathrm\{buyer\}\}\. A deal at the seller’s reservation price captures all available surplus for the principal and scores11, while a deal at the buyer’s reservation price captures none and scores00\. A deal above the buyer’s reservation price also receives zero buyer value\. Thus, Craigslist and Marketplace use the same normalized surplus\-sharing reward and differ only in how the endpoints of the price corridor are specified: Craigslist uses the task\-defined target and listing prices, while Marketplace uses the parties’ private reservation prices\.

##### Preference\-based reward for slot coordination\.

In Calendar, the agent receives its principal’s value functionv⁡\(t\)∈\[0,1\]v\(t\)\\in\[0,1\]over candidate time slots\([31](https://arxiv.org/html/2608.13787#bib.bib21)\)\. The requestor has a separate value function over the same slots, constructed in opposition to the principal’s preferences\. Let𝒵\\mathcal\{Z\}denote the zone of possible agreement, defined as the set of time slots that are mutually free on both calendars\.

We define the principal\-best and counterparty\-best feasible values as

vmax=maxt∈𝒵⁡v⁡\(t\),vmin=mint∈𝒵⁡v⁡\(t\)\.v\_\{\\max\}=\\max\_\{t\\in\\mathcal\{Z\}\}v\(t\),\\qquad v\_\{\\min\}=\\min\_\{t\\in\\mathcal\{Z\}\}v\(t\)\.\(9\)For a meeting scheduled attdeal∈𝒵t\_\{\\mathrm\{deal\}\}\\in\\mathcal\{Z\}, the Calendar reward is

rcalendar=v⁡\(tdeal\)−vminvmax−vmin\.r\_\{\\mathrm\{calendar\}\}=\\frac\{v\(t\_\{\\mathrm\{deal\}\}\)\-v\_\{\\min\}\}\{v\_\{\\max\}\-v\_\{\\min\}\}\.\(10\)By construction, the feasible slots have different preference scores, sovmax\>vminv\_\{\\max\}\>v\_\{\\min\}\. The principal’s most preferred mutually feasible slot scores11, while the slot most favorable to the requestor scores00\.

This reward distinguishes agreement quality from task completion\. Scheduling any mutually feasible meeting completes the coordination task, while the reward measures whether the selected slot actually advances the principal’s preferences\.

##### Non\-agreement outcomes and common scale\.

Timeouts and aborted interactions receive zero reward\. Where an environment provides an explicit walk\-away action, we retain its task\-specific outside\-option payoff\. Calendar and Marketplace assign zero outcome value when no agreement is reached\.

Across all six environments, the resulting training signal lies in\[0,1\]\[0,1\]\. The environment\-specific constructions preserve the relevant utility structure: fair and efficient allocation, division of price surplus, or preference\-sensitive coordination, while the shared range makes reward statistics comparable and provides a consistent interface for the multi\-domain consolidation methods in Section[5](https://arxiv.org/html/2608.13787#S5)\.

## 5SocialRL Training and Unification

Our study proceeds in two stages\. In Stage 1, we train a domain\-specialized policy for each of the six environments\. PPO directly from the base model is sufficient for Deal\-or\-No\-Deal, CaSiNo, Job Interview, and Calendar, while Craigslist and Marketplace use an SFT warm start before PPO\. We then evaluate every specialist on all six environments, revealing substantial and highly asymmetric cross\-environment transfer that depends on interaction structure \(§[5\.2](https://arxiv.org/html/2608.13787#S5.SS2)\)\. In Stage 2, we consolidate the specialists into a single multi\-domain policy through two complementary approaches: transfer\-aware cascade RL, which uses the observed transfer structure to prioritize final performance, and multi\-teacher on\-policy distillation \(MOPD\), which transfers most of the specialists’ advantage with substantially less additional training \(§[5\.3](https://arxiv.org/html/2608.13787#S5.SS3)\)\. As a complementary study, we introduce explicit theory\-of\-mind supervision throughInfer→\\rightarrowAct→\\rightarrowAnticipateand test whether direct supervision of opponent modeling improves social reasoning and cross\-environment generalization \(§[5\.4](https://arxiv.org/html/2608.13787#S5.SS4)\)\. All experiments use Qwen3\-4B\-Instruct\-2507 as the base policy, providing a controlled test of whether targeted post\-training can induce broad strategic capabilities in a compact model\. Across our six\-environment evaluation, the resulting specialists and unified policy achieve aggregate performance in the range of GPT\-4\.1, GPT\-5\.1, and GPT\-5\.2\.

### 5\.1Stage 1: In\-Domain Specialist Training

We first train a specialized 4B policy for each of the six environments\. For Deal\-or\-No\-Deal, CaSiNo, Job Interview, and Calendar, we optimize the base model directly with PPO using the terminal outcome rewards defined in §[4\.2](https://arxiv.org/html/2608.13787#S4.SS2)\. For the two price\-negotiation environments, Craigslist and Marketplace, we initialize the policy with supervised fine\-tuning \(SFT\) before applying PPO\. This warm start places prerequisite bargaining behaviors within the policy’s support, improving exploration; PPO then optimizes these behaviors for the environment’s terminal utility\. All rollouts use the decoupled infrastructure described in §[3\.2](https://arxiv.org/html/2608.13787#S3.SS2)\. Agents interact with their environments through the standard harness, while the rollout proxy records the model inputs, outputs, rollout log probabilities, and terminal rewards required for training\. This separation allows the same PPO implementation to be used across environments with different interaction structures and agent implementations\.

##### PPO configuration\.

Unless otherwise stated, all PPO runs use a separate Qwen3\-1\.7B critic\. We first warm up the critic for3030optimization steps while keeping the 4B actor frozen, and then jointly update the actor and critic\. We use learning rate of2×10−62\\times 10^\{\-6\}for the actor with batch size 144, mini batch size 36, max gradient norm 5, and learning rate of1×10−51\\times 10^\{\-5\}for the critic\. Max training steps is set to be 200\. Under this configuration, the critic typically reaches an explained variance of approximately0\.80\.8before actor being trained, indicating that the value model captures most of the variation in episodic returns and provides a reliable baseline for PPO updates\.

Table 2:In\-domain specialist training\.Base and domain\-trained 4B scores are reported as mean±\\pmstandard deviation across repeated evaluations; GPT baselines are reported as mean scores\. Each domain\-trained checkpoint is evaluated on its training environment\. The six 4B specialists average 0\.619 utility, placing their aggregate performance in the range of GPT\-4\.1, GPT\-5\.1, and GPT\-5\.2\.As shown in Table[2](https://arxiv.org/html/2608.13787#S5.T2), in\-domain post\-training produces strong specialists across all six interaction structures111We ran evaluation for 5 times to compute the mean and variance throughout the paper\.\. Averaged across domains, the specialized 4B policies achieve 0\.615 utility, comparable to GPT\-4\.1 \(0\.625\), GPT\-5\.1 \(0\.619\), and GPT\-5\.2 \(0\.613\)\. At the environment level, the 4B specialists fall within or above the GPT\-4\.1/5\.1/5\.2 range on DnD, CaSiNo, Craigslist, and Marketplace, while remaining close on Job Interview and Calendar\. We next ask whether these domain\-specific capabilities transfer across interaction structures\.

### 5\.2Cross\-Environment Transfer

We next evaluate every domain\-trained policy on all six environments, producing the transfer matrix in Table[3](https://arxiv.org/html/2608.13787#S5.T3)\. The diagonal measures in\-domain specialization, while each off\-diagonal entry measures how training on one*donor*environment changes performance on another\. We summarize each donor by its*out\-transfer*, the mean change relative to the base model over the other five environments\.

Table 3:Cross\-environment transfer\.Each row evaluates a policy trained on one donor environment across all six domains \(⋆\\stardenotes the training domain\)\. Out\-transfer is the mean change from the base 4B policy over the five off\-domain environments\.†Craigslist uses SFT\+PPO\. Reported uncertainties are standard deviations across three SGLang benchmark runs where available\.Transfer depends strongly on the donor\-recipient configuration\. The clearest pattern is transfer between environments with similar interaction structure\. Craigslist and Marketplace, both price\-negotiation environments, exhibit the two largest cross\-domain gains: training on Craigslist raises Marketplace from0\.1740\.174to0\.4910\.491\(\+0\.317\+0\.317\), while training on Marketplace raises Craigslist from0\.3180\.318to0\.5020\.502\(\+0\.184\+0\.184\)\. The allocation pair DnD and CaSiNo also transfers positively in both directions, although more modestly\. Some environments transfer more broadly\. Job Interview is the strongest general\-purpose donor after Craigslist, with an average out\-transfer of\+0\.026\+0\.026and improvements on DnD, CaSiNo, Craigslist, and Calendar\. In contrast, a policy can become very strong in\-domain without becoming a useful donor: Marketplace reaches0\.8380\.838on its own task while its average effect on the other five environments is approximately neutral\. Calendar is the clearest negative donor in some domains, with out\-transfer of−0\.060\-0\.060, including large drops on Craigslist \(→0\.1670\.318\\\!\\rightarrow\\\!0\.167\) and Job Interview \(→0\.3110\.479\\\!\\rightarrow\\\!0\.311\)\.

These results show that transfer is strongly asymmetric and structured: what matters is not simply how strong a specialist is, but which capability it learns and which environment receives it\. This observation is central to unification\. In the next stage, we exploit the measured transfer structure to choose the ordering of environments for cascade RL, and compare this transfer\-aware ordering against random and anti\-transfer orderings\.

### 5\.3Stage 2: Consolidating Specialists into a Unified Model

Stage 1 produces a strong specialist for each environment\. Our next goal is to consolidate these capabilities into a single policy that performs well across all six interaction structures\. This is non\-trivial because training effects are coupled across domains: as shown by the transfer matrix in Table[3](https://arxiv.org/html/2608.13787#S5.T3), optimizing one environment can improve, preserve, or interfere with performance on another\. Sequential training can therefore both exploit positive transfer and induce catastrophic forgetting\. We study two complementary approaches to unification, targeting different points in the performance\-efficiency tradeoff\.

##### Two routes to unification\.

Our first approach,*cascade RL*\([38](https://arxiv.org/html/2608.13787#bib.bib52);[10](https://arxiv.org/html/2608.13787#bib.bib53)\), continues reinforcement learning sequentially across environments\. Because each stage directly optimizes environment reward, cascade RL can exploit positive transfer during training and may even improve previously learned capabilities beyond the corresponding single\-domain specialists\. Its cost is substantial: each environment requires another full RL stage, effectively relearning the capabilities rather than directly reusing the trained specialists\. Moreover, the final policy depends strongly on training order, since beneficial and destructive transfer are highly asymmetric\. We therefore explore the construction of a transfer\-aware curriculum based on the transfer matrix from Section[5\.2](https://arxiv.org/html/2608.13787#S5.SS2)\.

Our second approach, MOPD\([29](https://arxiv.org/html/2608.13787#bib.bib50);[45](https://arxiv.org/html/2608.13787#bib.bib51);[47](https://arxiv.org/html/2608.13787#bib.bib54);[12](https://arxiv.org/html/2608.13787#bib.bib55)\), reuses the Stage 1 specialists directly as teachers and distills their capabilities into a shared student\. This is substantially more efficient: fewer than100100additional optimization steps recover most of the specialists’ advantage\. The main challenge shifts from environment ordering to allocation of the distillation budget: different teachers provide very different amounts of additional capability over the student, and some have already been nearly matched through cross\-domain transfer\. We address this with a gap\-closed curriculum that focuses sampling on domains whose teacher advantage remains unabsorbed\.

##### Shared initialization\.

Both consolidation methods start from the same Craigslist\-SFT checkpoint\. This initialization is necessary because the base 4B model rarely explores the anchoring behavior required for successful Craigslist negotiation: as observed in Stage 1, direct PPO fails to improve the base policy on Craigslist\. We find the same limitation for OPD from the base model also fails to acquire the Craigslist capability when trained on Craigslist alone\. Supervised distillation first places this prerequisite behavior within the student’s support, after which either RL or OPD can optimize and combine it with capabilities from the other environments\. Using the same initialization for both methods also makes their comparison controlled\.

Cascade RL and MOPD thus provide complementary approaches to the same consolidation problem\. Cascade RL prioritizes final performance by continuing reward optimization and exploiting cross\-environment transfer, while MOPD prioritizes efficiency by directly transferring the capabilities already learned by the domain specialists\.

#### 5\.3\.1Stage 2A: Transfer\-Aware Unification with Cascade RL

##### Training principle\.

Cascade RL consolidates capabilities by continuing reinforcement learning sequentially across environments\. A single policy is carried through the cascade: at each stage, PPO optimizes the policy on one environment using the same outcome reward as in Stage 1, and the selected checkpoint initializes the next stage\. We periodically evaluate checkpoints on all environments encountered so far and select the one with the highest average utility, balancing progress on the current domain against retention of previously acquired capabilities\.

The central design choice is therefore the*order*of environments\. The transfer matrix in Table[3](https://arxiv.org/html/2608.13787#S5.T3)shows that cross\-environment effects are strongly directional: training on one domain may improve another, leave it largely unchanged, or substantially degrade it\. Sequential RL is consequently path\-dependent\. We use the observed transfer structure to construct a curriculum that places beneficially interacting domains together and schedules destructive donors before the capabilities they would otherwise overwrite\.

##### Transfer\-aware ordering\.

Three patterns in Table[3](https://arxiv.org/html/2608.13787#S5.T3)guide the curriculum\. First, Craigslist and Marketplace form the strongest transfer pair in the suite: Craigslist training improves Marketplace by\+0\.317\+0\.317, while Marketplace improves Craigslist by\+0\.184\+0\.184\. We therefore place them consecutively, with Craigslist first to exploit the stronger transfer direction\. Second, Calendar is a destructive donor for Craigslist and Job Interview: reducing Craigslist by0\.1510\.151and Job Interview by0\.1680\.168, and is the only domain with negative average out\-transfer\. We therefore place Calendar at the beginning of the cascade, before these bargaining capabilities are acquired\. Third, Job Interview exhibits strongly asymmetric transfer\. It transfers positively to several other domains, yet its own performance is poorly preserved after training elsewhere\. We therefore place it last, allowing the final stage to recover Job Interview capability without exposing it to subsequent interference\.

These considerations yield thetransfer\-awarecurriculum

Calendar→CaSiNo→DnD→Craigslist→Marketplace→Job Interview\.\\text\{Calendar\}\\rightarrow\\text\{CaSiNo\}\\rightarrow\\text\{DnD\}\\rightarrow\\text\{Craigslist\}\\rightarrow\\text\{Marketplace\}\\rightarrow\\text\{Job Interview\}\.\(11\)

##### CascadeRL configuration\.

All cascade experiments start from the same Craigslist\-SFT initialization and use the same PPO configuration and per\-stage training budget\. Each stage is an independent PPO with max 200 steps, and we evaluate each checkpoint every 20 training steps on a small validation dataset\.

##### Ordering controls\.

To test whether the measured transfer structure provides a useful curriculum signal, we compare the transfer\-aware ordering with alternative sequences while holding initialization and training budget fixed\. Our random\-order control uses

Craigslist→Job Interview→DnD→CaSiNo→Calendar→Marketplace\.\\text\{Craigslist\}\\rightarrow\\text\{Job Interview\}\\rightarrow\\text\{DnD\}\\rightarrow\\text\{CaSiNo\}\\rightarrow\\text\{Calendar\}\\rightarrow\\text\{Marketplace\}\.\(12\)We additionally construct an*anti\-transfer*curriculum that deliberately reverses the main ordering principles:

Job Interview→Marketplace→Craigslist→DnD→CaSiNo→Calendar\.\\text\{Job Interview\}\\rightarrow\\text\{Marketplace\}\\rightarrow\\text\{Craigslist\}\\rightarrow\\text\{DnD\}\\rightarrow\\text\{CaSiNo\}\\rightarrow\\text\{Calendar\}\.\(13\)This ordering places the difficult\-to\-preserve Job Interview capability first and the destructive Calendar stage last\.

##### Unified\-model performance\.

Table[5](https://arxiv.org/html/2608.13787#S5.T5)reports the final six\-environment performance of the transfer\-aware, random\-order, and anti\-transfer cascades, together with the GPT baselines\. The transfer\-aware cascade reaches an Avg\-6 of0\.627±0\.0040\.627\\pm 0\.004, placing the unified 4B policy in the same aggregate performance range as GPT\-4\.1 \(0\.6250\.625\), GPT\-5\.1 \(0\.6190\.619\), and GPT\-5\.2 \(0\.6130\.613\)\. In contrast, the random\-order cascade reaches only0\.584±0\.0080\.584\\pm 0\.008and the anti\-transfer cascade reaches only0\.562±0\.0100\.562\\pm 0\.010under the same overall training setup\.

The aggregate result masks substantial cross\-domain interaction\. Calendar is the clearest example of positive accumulation: although it is trained at the very beginning of the cascade, the final policy reaches0\.7420\.742, well above its single\-domain specialist score of0\.5400\.540and above every evaluated GPT baseline\. Marketplace is also strongly retained at0\.8030\.803, essentially matching GPT\-4\.1 at0\.8040\.804, while Craigslist reaches0\.5800\.580, above GPT\-4\.1 and GPT\-5\.2\. Thus, sequential RL can do more than preserve earlier specialists: subsequent stages can reinforce capabilities acquired earlier through positive cross\-environment transfer\.

##### Stage\-by\-stage transfer dynamics\.

The final benchmark does not reveal how capabilities evolve as the policy moves through the cascade\. Table[4](https://arxiv.org/html/2608.13787#S5.T4)traces the checkpoint selected at the end of each stage, evaluated on the environments included in checkpoint selection at that point on a small validation dataset\. Because every cascade starts from the Craigslist\-SFT initialization, Craigslist is tracked from the first stage even though its PPO stage occurs fourth\. These values are the within\-cascade evaluations used for checkpoint selection; the final repeated benchmark is reported separately in Table[5](https://arxiv.org/html/2608.13787#S5.T5)\.

Table 4:Stage\-by\-stage evolution of the transfer\-aware cascade\.Each row reports the checkpoint selected after optimizing the environment in the first column; bold denotes the environment optimized at that stage\. Craigslist is evaluated from the beginning because the cascade is initialized from its SFT checkpoint\. Dashes denote environments not yet included in checkpoint selection\. Mean seen averages the environments tracked at that stage and is therefore not directly comparable across rows as the set of environments expands\. These are within\-cascade checkpoint\-selection evaluations; Table[5](https://arxiv.org/html/2608.13787#S5.T5)reports the final repeated six\-environment benchmark\.The stagewise trajectory shows that cross\-environment transfer occurs during the cascade itself\. Calendar reaches0\.6210\.621after its own stage, then improves to0\.7330\.733after CaSiNo and0\.7580\.758after DnD despite receiving no additional Calendar\-specific updates\. Later bargaining stages introduce some forgetting, but Calendar remains strong at0\.6920\.692after the final Job Interview stage\. Its high final performance therefore reflects capability accumulated across multiple stages, rather than retention of the original Calendar checkpoint alone\.

The Craigslist\-Marketplace pair exhibits a second form of transfer\. Craigslist performance initially falls during the allocation stages \(0\.614→0\.517→0\.4890\.614\\rightarrow 0\.517\\rightarrow 0\.489\), recovers to0\.5440\.544when Craigslist is optimized directly, and then rises further to0\.5640\.564after Marketplace training\. At the same checkpoint, Marketplace reaches0\.8420\.842while DnD and Calendar remain at0\.6280\.628and0\.7040\.704, respectively\. The final Job Interview stage preserves most of these gains: Marketplace remains at0\.8230\.823and Craigslist rises to0\.5700\.570, hile also raising CaSiNo from0\.4830\.483to0\.5130\.513\.

These dynamics are neither monotonic nor uniformly positive\. Earlier capabilities can temporarily degrade, as Craigslist does during the allocation stages, and Calendar declines from its intermediate peak as later domains are introduced\. The role of the transfer\-aware curriculum is therefore to order these interactions so that destructive stages occur early and later stages can repair or reinforce related capabilities\. Checkpoint selection on the environments seen so far further limits forgetting at each transition\.

Table 5:Cascade RL is sensitive to environment ordering\.The transfer\-aware curriculum follows Calendar→\\rightarrowCaSiNo→\\rightarrowDnD→\\rightarrowCraigslist→\\rightarrowMarketplace→\\rightarrowJob Interview; the random\-order control follows Eq\.[12](https://arxiv.org/html/2608.13787#S5.E12); and the anti\-transfer control follows Eq\.[13](https://arxiv.org/html/2608.13787#S5.E13)\. Each final evaluation uses1010scenarios×\\times33opponents×\\times55trials \(150150games per environment\)\. Transfer\-aware scores are mean±\\pmstandard deviation over two independent evaluations, and random\-order scores are mean±\\pmstandard deviation over repeated evaluations\. Anti\-transfer performs way worse than both Transfer\-aware as well as Random\-order\. The transfer\-aware unified 4B policy achieves an Avg\-6 of0\.6270\.627, in the performance range of GPT\-4\.1, GPT\-5\.1, and GPT\-5\.2\.
##### Comparison with other curriculums\.

The comparison with random ordering and anti\-optimal ordering shows that this accumulation depends strongly on curriculum\. The transfer\-aware cascade improves Avg\-6 from0\.5840\.584to0\.6270\.627, with particularly large gains on Calendar \(0\.541→0\.7420\.541\\rightarrow 0\.742\), Job Interview \(0\.478→0\.5380\.478\\rightarrow 0\.538\), and Marketplace \(0\.783→0\.8030\.783\\rightarrow 0\.803\)\. The random ordering performs better on Craigslist in isolation \(0\.6370\.637vs\.0\.5800\.580\), but fails to preserve comparable performance across the full suite\. This is precisely the objective of the transfer\-aware curriculum: optimize the final multi\-domain policy rather than any single stage\.

These results support the use of the transfer matrix as a reasonable curriculum\-design signal\. The same environments, trained with the same initialization and learning procedure, can lead to substantially different unified policies depending on their order\. Cascade RL can therefore achieve strong multi\-domain performance by exploiting the directionality of cross\-environment transfer, at the cost of running a full sequence of reinforcement\-learning stages\.

#### 5\.3\.2Stage 2B: Efficient Unification with MOPD

##### Training principle\.

Cascade RL continues optimizing environment rewards, but requires a full RL stage for each additional domain\. MOPD instead treats the Stage\-1 specialists as teachers and directly transfers their capabilities into a common student\. At each rollout, we sample an environmenteeand pair it with its corresponding specialist\. The student generates trajectories on\-policy, and its responses are trained toward the teacher using a reverse\-KL distillation objective\. This construction preserves the original interaction stack: the environment and agent harness execute unchanged, while the rollout proxy records the student trajectory and queries the corresponding teacher for distillation\. Changing from one teacher to six therefore requires no modification to the environments or agent implementations\.

##### MOPD configuration\.

The MOPD runs use a reverse\-KL objective: the student generates every rollout on\-policy, and the corresponding domain specialist supplies token\-level teacher logits for the generated responses\. We truncate the teacher distribution to its top3232logits, use a distillation temperature of1\.01\.0, and apply the loss uniformly over all response tokens without position weighting\. We optimize with a constant learning rate of10−510^\{\-5\}without warmup\.

##### Initialization and teacher selection\.

We initialize the unified student from the Craigslist\-SFT checkpoint used throughout Stage 2\. We also exclude teachers that offer essentially no additional headroom over this initialization\. CaSiNo is the clearest case: the Craigslist\-SFT student already scores0\.5020\.502on CaSiNo, nearly matching the domain specialist at0\.5030\.503\. Distilling from such a teacher provides little additional signal while potentially introducing optimization noise\. We therefore omit CaSiNo from the MOPD training mixture and retain it only for evaluation\. The transfer results in Table[3](https://arxiv.org/html/2608.13787#S5.T3)further suggest that CaSiNo performance can be maintained through capabilities learned from other domains\.

##### Gap\-closed curriculum\.

The remaining teachers still differ substantially in how much capability the student has left to absorb\. Uniform sampling gives equal training budget to domains that are nearly solved and domains with large remaining gaps\. We instead adapt the sampling distribution to the fraction of each teacher’s advantage that remains unclosed: Letℰ\\mathcal\{E\}denote the candidate training environments\. For eache∈ℰe\\in\\mathcal\{E\}, letBeB\_\{e\}be the student’s pre\-distillation score,TeT\_\{e\}its specialist teacher’s score, ands¯e\(t\)\\bar\{s\}\_\{e\}^\{\(t\)\}the student’s rolling mean terminal reward over the most recentW=500W=500training games\. Before any observations are available, we initializes¯e\(t\)=Be\\bar\{s\}\_\{e\}^\{\(t\)\}=B\_\{e\}\.

We define

ge=Te−Be,ρe\(t\)=s¯e\(t\)−Bege,g\_\{e\}=T\_\{e\}\-B\_\{e\},\\qquad\\rho\_\{e\}^\{\(t\)\}=\\frac\{\\bar\{s\}\_\{e\}^\{\(t\)\}\-B\_\{e\}\}\{g\_\{e\}\},\(14\)wheregeg\_\{e\}is the teacher\-student headroom andρe\(t\)\\rho\_\{e\}^\{\(t\)\}is the fraction already closed\. The unnormalized sampling weight is

we\(t\)=\{λ,ge<δ,max⁡\(0,1−ρe\(t\)\)\+λ,otherwise,w\_\{e\}^\{\(t\)\}=\\begin\{cases\}\\lambda,&g\_\{e\}<\\delta,\\\\\[4\.0pt\] \\max\\\!\\left\(0,1\-\\rho\_\{e\}^\{\(t\)\}\\right\)\+\\lambda,&\\text\{otherwise\},\\end\{cases\}\(15\)with floorλ=0\.05\\lambda=0\.05and minimum\-gap thresholdδ=0\.03\\delta=0\.03\. We normalize

pe\(t\)=we\(t\)∑e′we′\(t\),p\_\{e\}^\{\(t\)\}=\\frac\{w\_\{e\}^\{\(t\)\}\}\{\\sum\_\{e^\{\\prime\}\}w\_\{e^\{\\prime\}\}^\{\(t\)\}\},\(16\)and band\-clamp the resulting probabilities to

pe\(t\)∈\[pmin,pmax\],p\_\{e\}^\{\(t\)\}\\in\[p\_\{\\min\},p\_\{\\max\}\],\(17\)iteratively renormalizing the remaining mass\.pmin,pmaxp\_\{\\min\},p\_\{\\max\}are set to be\[0\.10,0\.40\]\[0\.10,0\.40\]in our experiments\. The normalization by each domain’s own gap makes progress comparable across environments with different reward ranges\. At initialization,ρe=0\\rho\_\{e\}=0for every learnable environment\. As the student approaches its teacher, the corresponding sampling mass decreases; once it matches or surpasses the teacher, the environment receives only the floor weight\. The minimum\-gap guard prevents near\-zero teacher\-student gaps from turning reward noise into extreme sampling weights, while the probability band prevents any remaining environment from dominating or disappearing from training\.

##### Evaluation metrics\.

We evaluate every unified checkpoint on all six environments and report selected checkpoints as mean±\\pmstandard deviation over five independent evaluation runs\. We use three complementary aggregate metrics\.

The first is the unweighted mean utility which directly measures overall model performance:

Avg6⁡\(s\)=16​∑ese,\\mathrm\{Avg6\}\(s\)=\\frac\{1\}\{6\}\\sum\_\{e\}s\_\{e\},\(18\)
To measure the objective specific to specialist consolidation, we additionally compute the fraction of each teacher’s advantage absorbed by the student:

Ce​\(s\)=se−BeTe−Be\.C\_\{e\}\(s\)=\\frac\{s\_\{e\}\-B\_\{e\}\}\{T\_\{e\}\-B\_\{e\}\}\.\(19\)A value of11means that the unified student matches the specialist teacher, and values above11indicate that it surpasses the specialist\. CaSiNo is excluded because its teacher\-initialization gap is only0\.0010\.001\.

Finally, we report clipped gap closure whereℰ′\\mathcal\{E\}^\{\\prime\}excludes CaSiNo\.

C¯clip=1\|ℰ′\|​∑e∈ℰ′clip⁡\(Ce,0,1\),\\bar\{C\}\_\{\\mathrm\{clip\}\}=\\frac\{1\}\{\|\\mathcal\{E\}^\{\\prime\}\|\}\\sum\_\{e\\in\\mathcal\{E\}^\{\\prime\}\}\\operatorname\{clip\}\\\!\\left\(C\_\{e\},0,1\\right\),\(20\)Clipping prevents over\-performance on one environment from compensating for incomplete transfer on another\. Avg\-6 thus measures absolute capability, while the two gap\-closure metrics measure how completely the specialist capabilities have been consolidated\.

##### Curriculum ablations\.

Table[6](https://arxiv.org/html/2608.13787#S5.T6)compares adaptive and uniform environment sampling, with and without the saturated CaSiNo teacher\. The gap\-closed curriculum without CaSiNo performs best under all three metrics\. After only6060optimization steps, it reaches an Avg\-6 of0\.597±0\.0150\.597\\pm 0\.015, with92\.6%92\.6\\%mean gap closure and84\.5%84\.5\\%clipped gap closure\. Removing CaSiNo helps under both uniform and adaptive sampling, confirming that allocating distillation budget to a teacher with essentially zero headroom is counterproductive\.

Table 6:MOPD\.All configurations use reverse\-KL distillation without position weighting and a learning rate of10−510^\{\-5\}\. Scores are mean±\\pmstandard deviation over five independent evaluation runs\. The gap\-closed curriculum without the saturated CaSiNo teacher performs best under all three aggregate metrics\.
##### Per\-environment consolidation\.

Table[7](https://arxiv.org/html/2608.13787#S5.T7)shows how the best unified checkpoint compares with its initialization and specialist teachers\. The unified model recovers85%85\\%of the Craigslist teacher’s advantage,69%69\\%on Job Interview, and83%83\\%on Marketplace\. It also surpasses the corresponding specialist on DnD \(0\.6640\.664vs\.0\.6560\.656\) and Calendar \(0\.5640\.564vs\.0\.5400\.540\), yielding gap closures above100%100\\%\. Thus, MOPD can combine specialist capabilities without constraining the unified model to interpolate below each teacher\.

Table 7:Per\-environment performance of the best MOPD model\.We report the gap\-closed curriculum without CaSiNo at step 60\. Unified\-model scores are mean±\\pmstandard deviation over five independent evaluation runs\. Gap closure measures the fraction of each specialist’s advantage over the initialization recovered by the unified model\.†CaSiNo is excluded from MOPD training and gap\-closure aggregation because its teacher–initialization gap is only0\.0010\.001\.∗Mean gap closure over the remaining five environments\.Overall, MOPD provides a substantially cheaper consolidation route than sequential RL: most of the specialist advantage is transferred in fewer than100100additional optimization steps\. Together, the two Stage\-2 methods expose a performance\-efficiency tradeoff: cascade RL uses continued reward optimization and transfer\-aware ordering to pursue a stronger final policy, while MOPD directly compresses the specialists into a unified model with a much smaller additional training budget\.

### 5\.4Explicit Theory\-of\-Mind Supervision

The training recipe so far improves negotiation through outcome\-based reinforcement learning: the model is rewarded for reaching favorable agreements, while any reasoning about the counterpart is learned only indirectly\. Yet effective negotiation depends naturally on understanding the other party\([18](https://arxiv.org/html/2608.13787#bib.bib56);[32](https://arxiv.org/html/2608.13787#bib.bib57);[22](https://arxiv.org/html/2608.13787#bib.bib58);[42](https://arxiv.org/html/2608.13787#bib.bib59)\)\. An agent must infer what the counterpart values, identify where their preferences overlap or conflict, and anticipate how the counterpart is likely to respond to a proposed action\. We therefore study a complementary approach that supervises these intermediate reasoning capabilities explicitly, asking whether stronger*theory of mind*\(ToM\) can translate into stronger negotiation\.

##### Theory\-of\-mind capabilities\.

We operationalize theory of mind through two capabilities that are directly relevant to strategic interaction\.*Preference inference*\(Infer\) measures whether the agent can recover the counterpart’s latent preferences or private state from the interaction history\.*Next\-action prediction*\(Anticipate\) measures whether the agent can predict how the counterpart will respond to the agent’s planned move\. Together with the agent’s own action \(Act\), these form the reasoning scaffold

Infer→Act→Anticipate\.\\textsc\{Infer\}\\;\\rightarrow\\;\\textsc\{Act\}\\;\\rightarrow\\;\\textsc\{Anticipate\}\.\(21\)For each decision, the scaffold records the agent’s current belief about the counterpart, its chosen action, and its prediction of the counterpart’s subsequent reaction\. We evaluateInferby exact\-match accuracy against the counterpart’s private preferences andAnticipateby the accuracy of the predicted next action\.

##### Does prompting alone elicit theory of mind?

We first test whether the base 4B model can benefit from the scaffold without additional training\. The answer is negative: adding the explicit ToM prompt reduces average negotiation utility from0\.4540\.454to0\.3530\.353\(Table[8](https://arxiv.org/html/2608.13787#S5.T8)\), with the largest drop on Job Interview \(0\.476→0\.2210\.476\\rightarrow 0\.221\)\. Requiring a small model to produce an explicit reasoning structure therefore does not by itself improve strategic behavior\.

Direct evaluation of the two ToM capabilities helps explain this result\. The base model already performs reasonably well at preference inference, reaching0\.6180\.618accuracy, but is substantially weaker at next\-action prediction, at approximately54%54\\%\. Thus, the base model can often recover what the counterpart wants while remaining much less capable of anticipating what the counterpart will actually do\. Prompting exposes this limitation rather than resolving it\.

##### Training explicit theory of mind\.

We next train these capabilities directly through supervised distillation\. GPT\-5\.2 generates demonstrations containing both the ordinary negotiation action and the explicitInferandAnticipatereasoning\. We refer to SFT on these complete traces as*ExpToM SFT*\. The experiment uses no reinforcement learning, allowing us to isolate the effect of explicit reasoning supervision from outcome optimization\. We compare ExpToM SFT with standard SFT on action trajectories, under both per\-environment training and joint training across the four negotiation environments\.

Explicit ToM supervision substantially improves the model’s measured reasoning capabilities\. Next\-action prediction improves by roughly3030percentage points, while preference\-inference accuracy rises to approximately0\.630\.63–0\.710\.71, depending on the training environment\. More importantly, these gains translate into stronger negotiation\. Per\-environment ExpToM SFT reaches an average utility of0\.5460\.546, compared with0\.5000\.500for ordinary SFT and0\.4540\.454for the base model\. The improvement is consistent across all four environments\. The effect also survives multi\-domain training: mixed ExpToM reaches0\.5250\.525, compared with0\.4950\.495for mixed ordinary SFT\.

Table 8:Explicit theory\-of\-mind supervision improves negotiation\.Normalized utility for the base model, prompt\-only ToM, ordinary SFT, and explicit ToM SFT\. ExpToM distillsInfer→\\rightarrowAct→\\rightarrowAnticipatetraces from GPT\-5\.2\. Prompting the scaffold alone hurts the base 4B, while training on the same reasoning structure improves over ordinary SFT under both per\-environment and mixed training\.
##### Which theory\-of\-mind capability matters for negotiation?

Finally, we ask which component of ToM is associated with successful negotiation\. Across trained checkpoints, next\-action prediction accuracy is positively correlated with negotiation utility, while preference\-inference accuracy shows little corresponding relationship\. The distinction is visible already in the base model: it has relatively strong preference inference despite weak negotiation performance, while its largest ToM deficit is next\-action prediction\.

This suggests that identifying the counterpart’s preferences is only one part of effective social reasoning\. Strategic action additionally requires anticipating how those preferences translate into behavior in response to a particular move\. Explicit ToM supervision improves both capabilities, but the improvement inAnticipateis the component most closely associated with better negotiation outcomes\.

Taken together, these experiments provide a second route for improving social reasoning alongside outcome\-based RL\. Reinforcement learning trains strategic behavior from the quality of the final agreement; explicit ToM distillation instead supervises intermediate reasoning about the counterpart\. The latter improves negotiation even under pure SFT, showing that part of the capability learned implicitly through interaction can also be transferred directly through structured reasoning supervision\.

## 6Qualitative Analysis

The aggregate results in §[5](https://arxiv.org/html/2608.13787#S5)show that post\-training substantially improves negotiation utility\. We next inspect the interaction trajectories to understand*how*the learned policies differ from the base model\. Our current analysis covers the six negotiation environments: for each environment, we compare the base model with the final domain\-trained policy; where available, we also examine an SFT\-only checkpoint as a diagnostic for separating behaviors introduced by imitation from those selected by reinforcement learning\.

We organize the analysis around two questions\. First, how does training change the agent’s*strategic decisions*: where it anchors, what it concedes, when it rejects, and what value it preserves for its principal? Second, how does training change the*interaction itself*: communication style and action choice?

### 6\.1Strategic Behavior: More Selective Concession

Across the four negotiation environments, the most consistent strategic change is greater selectivity in concession\. The base model often moves quickly toward agreement after encountering resistance, even when doing so gives away substantial utility\. Post\-training makes the policy more willing to maintain a favorable position, reject an unfavorable proposal, and concede on dimensions that are relatively inexpensive\. The concrete behavior differs with the structure of each environment\.

##### Craigslist: anchor low and concede gradually\.

Craigslist gives the clearest view of how training reshapes the bargaining trajectory\. We normalize buyer offers by the listing\-to\-target gap, with00denoting the buyer’s target price,11the listing price, and negative values offers below the target\. Table[9](https://arxiv.org/html/2608.13787#S6.T9)shows the resulting concession curves\.

The base model opens essentially at its own target \(\+0\.04\+0\.04\), with only3%3\\%of openings below target, and then moves rapidly toward the seller\. SFT introduces the missing anchoring behavior: the mean opening moves to−1\.21\-1\.21, and71%71\\%of games begin below the buyer’s target\. PPO strengthens this strategy further\. The SFT\+\+PPO policy opens at−1\.48\-1\.48, remains below target through its third proposal on average, and raises the below\-target opening rate to78%78\\%\.

Table 9:Craigslist buyer concession curves\.Offer price is normalized by the listing\-target gap:00denotes the buyer’s target,11the listing price, and negative values offers below target\. SFT introduces below\-target anchoring; PPO produces a lower initial anchor and a slower subsequent concession trajectory\.Matched trajectories illustrate the difference\. With a $275 listing and a $209 target, the base model opens at $209 and eventually accepts $275, obtaining zero utility\. The trained policy instead negotiates $150→\\rightarrow$165, rejects $250, and eventually accepts $190, obtaining maximal utility\. Training therefore changes both the initial reference point and the willingness to maintain it under pressure\.

Craigslist also reveals a division of labor between SFT and PPO in the language used to support these offers\. The base buyer frequently reveals its own target price:52\.7%52\.7\\%of its messages explicitly mention that private reference point, while only3\.7%3\.7\\%invoke external market comparisons\. After SFT, target leakage falls to1\.0%1\.0\\%and market/comparable references rise to29\.6%29\.6\\%of messages\. PPO subsequently shifts toward commitment language: explicit “final offer,” “firm limit,” and related expressions rise to13\.3%13\.3\\%of messages\. Thus, SFT introduces useful bargaining moves and rhetorical forms, while PPO selects when to commit to them\.

One undesirable behavior also appears here: the environment provides no external comparable\-price information, so some learned market references are fabricated\. We treat this learned bluffing behavior as a limitation rather than as evidence of improved factual grounding\.

##### Deal\-or\-No\-Deal: resist capitulation after rejection\.

In Deal\-or\-No\-Deal, the main weakness of the base model appears after its initial proposal\. Its openings are often reasonable, but rejection can trigger a sharp concession or repeated proposals that eventually lead to failure\. For example, in one trajectory where hats and balls are each worth55and books are worth00, the base model responds to resistance by accepting only a book, giving away all of its value\. Other trajectories repeat nearly identical allocations until the counterpart walks away\.

PPO makes rejection less likely to trigger this collapse\. The trained policy more consistently preserves high\-value items and either counters or accepts when the available allocation is favorable\. Relative to the SFT diagnostic checkpoint, PPO also reduces the zero\-reward and no\-deal tail: zero\-reward episodes fall from4\.7%4\.7\\%to2\.5%2\.5\\%, and no\-deals from4\.5%4\.5\\%to0\.5%0\.5\\%\. The qualitative shift is therefore concentrated in how the policy responds when its preferred allocation is challenged\.

##### CaSiNo: maintain a floor on high\-priority resources\.

CaSiNo exhibits a closely related pattern\. Here, successful negotiation requires protecting high\-priority resources while using low\-priority resources as concessions\. The base model often concedes too much after encountering resistance\. In a matched example, it initially asks for all three food packages and, after one rejection, immediately drops to a single food package\.

PPO learns a more stable reservation strategy\. The trained policy explicitly maintains floors such as “I require at least 2 food packages,” preserves its high\- and medium\-priority resources, and gives ground primarily on its low\-priority resource\. At the distribution level, the fraction of over\-conceding games falls from17\.3%17\.3\\%to13\.6%13\.6\\%, while the fraction of games in which the agent strongly preserves its own value rises from17\.3%17\.3\\%to24\.2%24\.2\\%\. The SFT diagnostic moves in the opposite direction, raising over\-concession to29\.4%29\.4\\%, suggesting that imitation alone can favor agreeable behavior without calibrating how much value to surrender\.

##### Job Interview: protect high\-weight contract terms\.

Job Interview makes selective concession multi\-dimensional\. The two parties negotiate salary and holiday together with categorical terms such as position, workplace, and company\. A strong policy therefore needs to identify which dimensions carry the largest private utility and spend concessions on less important issues\.

The trained trajectories show greater preservation of high\-value categorical terms\. The worker’s categorical utility in completed agreements rises from approximately0\.640\.64for the base model to0\.730\.73–0\.780\.78for the trained checkpoints, while the fraction of agreements securing the worker’s top\-choice workplace increases from39%39\\%to6868–72%72\\%\. In matched examples, the base model can abandon several high\-value categorical terms in order to close quickly, whereas the trained policy holds position, workplace, and company fixed while negotiating over salary and holiday\.

##### Marketplace: create bargaining room and protect the reservation price\.

Marketplace exhibits a particularly sharp strategic shift\. The base buyer is already capable of completing transactions, but negotiates from an extremely weak position: its mean opening offer is0\.9560\.956of its private reservation price, leaving almost no room to bargain\. After PPO, the mean opening falls to0\.3610\.361of the reservation price\. The trained policy therefore begins far from its maximum willingness to pay and preserves substantial concession room\.

Training also eliminates a severe information\-leakage failure mode\. In the base policy,62%62\\%of opening messages reveal the buyer’s reservation price or explicitly state its budget constraint\. After PPO, this behavior essentially disappears\. The learned shift closely parallels Craigslist: the agent stops revealing its private boundary, anchors aggressively, and moves toward agreement gradually rather than beginning near its own limit\.

##### Calendar: optimize agreement quality rather than agreement rate\.

Calendar exposes a different failure mode\. The central challenge is not anchoring or bargaining over a scalar surplus, but selecting a mutually feasible time slot that also serves the principal’s private preferences\. Before Calendar training, the policy is highly willing to schedule a meeting but frequently accepts the counterpart’s proposed slot without comparing it against better alternatives\. Using the Marketplace\-trained checkpoint as a pre\-Calendar proxy,92%92\\%of games end in a scheduled meeting, yet only16%16\\%obtain a maximum\-preference slot and63%63\\%receive zero utility\.

Calendar PPO changes this objective\. The trained policy schedules fewer meetings overall \(74%74\\%\), but the quality of the selected slots improves substantially: the fraction of maximum\-utility outcomes rises from16%16\\%to38%38\\%, while zero\-utility outcomes fall from63%63\\%to35%35\\%\. The policy has therefore learned that completing the coordination task is insufficient; it should reject or counter\-propose when the available agreement poorly serves its principal\.

##### Cross\-domain pattern\.

Across all six environments, training makes agreement more explicitly conditional on the principal’s utility\. The base model often treats reaching an agreement as valuable in itself and gives ground too readily when the counterpart resists or proposes a feasible alternative\. Post\-training instead learns environment\-specific reservation behavior: Craigslist and Marketplace create bargaining room through aggressive anchors and slower price concessions; DnD resists capitulation after rejection; CaSiNo protects high\-priority resources; Job Interview preserves high\-weight contract terms; and Calendar rejects feasible but low\-value slots in favor of higher\-preference alternatives\.

The common strategic change is therefore*utility\-sensitive concession*: the policy becomes better at distinguishing what can be traded away from what should be protected\. The particular implementation depends on the interaction structure: a reservation price in bilateral bargaining, a preferred bundle in allocation, high\-weight issues in multi\-issue negotiation, or a preference\-weighted time slot in coordination\.

### 6\.2Interaction Behavior: Less Private Deliberation, More Strategic Communication

The strategic changes above are accompanied by a broad shift in how the policy conducts the interaction\. Table[10](https://arxiv.org/html/2608.13787#S6.T10)summarizes several trajectory\-level measures\. Across the four analyzed environments, trained policies generate substantially fewer total tokens per turn while sending longer and more informative messages to the counterpart\. They also use rejection more actively and exhibit more explicit commitment language\.

Table 10:Interaction behavior across the training ladder\.Turns denotes turns per game; public tok is the token length of the opponent\-facing message; total tok includes the full generation associated with each turn;<think\>is the fraction of turns containing an explicit reasoning block; polite and firm denote the fractions of public messages containing courteous and commitment language, respectively\. Tokens are measured with the Qwen3\-4B tokenizer\.##### Rejection becomes a strategic action\.

The base policies rely heavily on proposing and accepting, with explicit rejection appearing in only00–3%3\\%of actions in the analyzed traces\. Training makes rejection a meaningful part of the policy\. Reject actions rise to approximately14%14\\%in DnD and20%20\\%in Job Interview, and they are typically accompanied by a counterproposal or explanation of which terms are unacceptable\.

This change complements the selective\-concession pattern above\. The trained model has acquired a practical way to maintain a reservation boundary: instead of responding to an unfavorable proposal by immediately moving toward it, the policy can reject, explain the conflict, and propose an alternative\.

##### Commitment becomes firmer while remaining socially cooperative\.

Training also changes the linguistic form of bargaining\. Commitment language increases substantially in several environments: from1%1\\%to11%11\\%of messages in CaSiNo,3%3\\%to23%23\\%in Craigslist, and2%2\\%to43%43\\%in the analyzed Job Interview checkpoint\. These messages contain explicit boundaries such as “firm,” “final offer,” or “non\-negotiable\.” Firmness generally coexists with courteous language\. For example, politeness rises from16%16\\%to59%59\\%in CaSiNo and remains around half of Craigslist messages in the final policy\. The learned behavior is therefore closer to*polite commitment*: clear reservation boundaries expressed through cooperative language\.

##### SFT broadens behavior; RL selects behavior by utility\.

The intermediate SFT checkpoints provide a useful diagnostic of how the two training signals differ\. In Craigslist, SFT introduces capabilities that are almost absent from the base model: below\-target anchoring, external justifications, and reduced leakage of the buyer’s private target\. RL then changes how these behaviors are deployed, producing stronger commitment and slower concession\.

In the other environments where we evaluate SFT as an ablation, imitation can also introduce undesirable behavior: more looping in DnD, more over\-concession in CaSiNo, and weaker categorical preservation in the currently analyzed Job Interview traces\. Outcome optimization subsequently selects among these behaviors according to realized utility\. This provides a trajectory\-level interpretation of the training recipe: supervised data can expand the policy’s behavioral repertoire, while reinforcement learning determines which parts of that repertoire are strategically useful\.

## 7Conclusion

We introducedSocialRL, a framework for training and studying language\-model agents in heterogeneous delegated interactions\. At the systems level, we develop a general event\-based multi\-agent environment interface that is independent of agent implementation, together with a decoupled training infrastructure that separates rollout generation from optimization through an OpenAI\-compatible rollout proxy\. These abstractions allow the same environments and agent harnesses to support local or remote agents, heterogeneous and black\-box counterparts, reinforcement learning, and distillation without coupling the interaction stack to a particular trainer\.

On top of this infrastructure, we study social\-reasoning post\-training across six heterogeneous negotiation and coordination environments\. Domain\-specific training produces strong 4B specialists whose aggregate performance lies in the range of much larger GPT models\. Evaluating these specialists across all environments reveals substantial and highly directional transfer: structurally related interactions reinforce one another, while other training configurations introduce interference\. We exploit this structure in two complementary approaches to unification\. Transfer\-aware cascade RL produces a single 4B policy with an Avg\-6 of0\.6270\.627, comparable to GPT\-4\.1, GPT\-5\.1, and GPT\-5\.2, and substantially outperforms random and anti\-transfer curricula\. MOPD provides a more efficient alternative, recovering most of the specialists’ advantage in fewer than100100additional optimization steps\.

We further investigate explicit theory\-of\-mind supervision as a complementary source of social\-reasoning capability\. DistillingInfer→\\rightarrowAct→\\rightarrowAnticipatetraces improves both negotiation performance and measured theory\-of\-mind abilities, with next\-action prediction emerging as the component most closely associated with negotiation outcomes\. Trajectory analysis provides a consistent behavioral interpretation of these gains: trained agents become more sensitive to their principal’s utility, protect private information more effectively, concede more selectively, and adapt their reasoning strategy to the structure of the interaction\.

Taken together, the environment abstraction, decoupled training infrastructure, and training results provide a general platform for studying social reasoning in interactive agents\. Our results suggest that strategic capabilities learned in one interaction structure can transfer to others, that this transfer can be exploited when consolidating specialists, and that strong multi\-domain social behavior can be learned in relatively small models\.

## References

- Allouahet al\.\(2026\)A\. Allouah, O\. Besbes, J\. D\. Figueroa, Y\. Kanoria, and A\. KumarWhat is your ai agent buying? evaluation, biases, model dependence, & emerging implications of agentic e\-commerce\.InProceedings of the ACM Web Conference 2026,pp\. 8697–8700\.Cited by:[§2](https://arxiv.org/html/2608.13787#S2.SS0.SSS0.Px2.p1.1)\.
- Anantaprayoonet al\.\(2026\)P\. Anantaprayoon, N\. Babina, N\. Asgharbeygi, and J\. TarifiLearning to negotiate: multi\-agent deliberation for collective value alignment in llms\.arXiv preprint arXiv:2603\.10476\.Cited by:[§2](https://arxiv.org/html/2608.13787#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.13787#S2.SS0.SSS0.Px2.p1.1)\.
- Awadallahet al\.\(2025\)A\. Awadallah, Y\. Lara, R\. Magazine, H\. Mozannar, A\. Nambi, Y\. Pandya, A\. Rajeswaran, C\. Rosset, A\. Taymanov, V\. Vineet,et al\.Fara\-7b: an efficient agentic model for computer use\.arXiv preprint arXiv:2511\.19663\.Cited by:[§2](https://arxiv.org/html/2608.13787#S2.SS0.SSS0.Px1.p1.1)\.
- Bergemannet al\.\(2026\)D\. Bergemann, S\. Ghili, X\. Hu, C\. Li, and Z\. YangTraining language models for bilateral trade with private information\.arXiv preprint arXiv:2604\.16472\.Cited by:[§2](https://arxiv.org/html/2608.13787#S2.SS0.SSS0.Px3.p1.1)\.
- Bianchiet al\.\(2024\)F\. Bianchi, P\. J\. Chia, M\. Yuksekgonul, J\. Tagliabue, D\. Jurafsky, and J\. ZouHow well can llms negotiate? negotiationarena platform and analysis\.arXiv preprint arXiv:2402\.05863\.Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p3.1),[§2](https://arxiv.org/html/2608.13787#S2.SS0.SSS0.Px3.p1.1)\.
- Chaturvedi and Verma \(2023\)R\. Chaturvedi and S\. VermaOpportunities and challenges of ai\-driven customer service\.Artificial Intelligence in customer service: The next frontier for personalized engagement,pp\. 33–71\.Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p2.1)\.
- Chawlaet al\.\(2021\)K\. Chawla, J\. Ramirez, R\. Clever, G\. Lucas, J\. May, and J\. GratchCasino: a corpus of campsite negotiation dialogues for automatic negotiation systems\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 3167–3185\.Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p3.1),[§1](https://arxiv.org/html/2608.13787#S1.p5.1),[§4\.1](https://arxiv.org/html/2608.13787#S4.SS1.SSS0.Px2.p1.1)\.
- Dahlgren Lindströmet al\.\(2025\)A\. Dahlgren Lindström, L\. Methnani, L\. Krause, P\. Ericson, Í\. M\. de Rituerto de Troya, D\. Coelho Mollo, and R\. DobbeHelpful, harmless, honest? sociotechnical limits of ai alignment and safety through reinforcement learning from human feedback: ad lindström et al\.\.Ethics and Information Technology27\(2\),pp\. 28\.Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p4.1)\.
- Donget al\.\(2024\)H\. Dong, W\. Xiong, B\. Pang, H\. Wang, H\. Zhao, Y\. Zhou, N\. Jiang, D\. Sahoo, C\. Xiong, and T\. ZhangRlhf workflow: from reward modeling to online rlhf\.arXiv preprint arXiv:2405\.07863\.Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p4.1)\.
- Duet al\.\(2024\)Y\. Du, R\. Srikant, and W\. ChenCascading reinforcement learning\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 30263–30304\.Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p5.1),[§5\.3](https://arxiv.org/html/2608.13787#S5.SS3.SSS0.Px1.p1.1)\.
- Elkoussy and Perez \(2026\)L\. Elkoussy and J\. PerezAgentLTL: a trace\-verification framework for measuring, enforcing, and training procedural compliance in tool\-using llm agents\.arXiv preprint arXiv:2607\.02599\.Cited by:[§2](https://arxiv.org/html/2608.13787#S2.SS0.SSS0.Px1.p1.1)\.
- Fuet al\.\(2026\)S\. Fu, H\. Xu, R\. He, Z\. Fu, H\. Wang, J\. Huang, X\. Ma, M\. Zhong, W\. Huang, X\. He,et al\.Poly\-opd: heterogeneous multi\-teacher on\-policy distillation for capability\-selectable flow models\.arXiv preprint arXiv:2608\.04349\.Cited by:[§5\.3](https://arxiv.org/html/2608.13787#S5.SS3.SSS0.Px1.p2.1)\.
- Fuet al\.\(2023\)Y\. Fu, H\. Peng, T\. Khot, and M\. LapataImproving language model negotiation with self\-play and in\-context learning from ai feedback\.arXiv preprint arXiv:2305\.10142\.Cited by:[§2](https://arxiv.org/html/2608.13787#S2.SS0.SSS0.Px3.p1.1)\.
- Google \(2026\)GoogleGoogle scheduler\.https://workspace\.google\.com/resources/appointment\-scheduling/\.Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p1.1)\.
- Haurumet al\.\(2024\)K\. R\. Haurum, R\. Ma, and W\. LongReal estate with ai: an agent based on langchain\.Procedia Computer Science242,pp\. 1082–1088\.Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p2.1)\.
- Heet al\.\(2018\)H\. He, D\. Chen, A\. Balakrishnan, and P\. LiangDecoupling strategy and generation in negotiation dialogues\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,pp\. 2333–2343\.Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p3.1),[§1](https://arxiv.org/html/2608.13787#S1.p5.1),[§4\.1](https://arxiv.org/html/2608.13787#S4.SS1.SSS0.Px3.p1.1)\.
- Huaet al\.\(2024\)W\. Hua, O\. Liu, L\. Li, A\. Amayuelas, J\. Chen, L\. Jiang, M\. Jin, L\. Fan, F\. Sun, W\. Wang,et al\.Game\-theoretic llm: agent workflow for negotiation games\.arXiv preprint arXiv:2411\.05990\.Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p3.1)\.
- Hwanget al\.\(2026\)E\. Hwang, Y\. Yin, G\. Carenini, P\. West, and V\. ShwartzInfusing theory of mind into socially intelligent llm agents\.InFindings of the Association for Computational Linguistics: ACL 2026,pp\. 11327–11360\.Cited by:[§5\.4](https://arxiv.org/html/2608.13787#S5.SS4.p1.1)\.
- Jianget al\.\(2026\)B\. Jiang, T\. Shi, R\. Kamoi, Y\. Yuan, C\. J\. Taylor, L\. Yang, P\. Zhou, and S\. ChenOne model, all roles: multi\-turn, multi\-agent self\-play reinforcement learning for conversational social intelligence\.arXiv preprint arXiv:2602\.03109\.Cited by:[§2](https://arxiv.org/html/2608.13787#S2.SS0.SSS0.Px3.p1.1)\.
- Junejaet al\.\(2025\)G\. Juneja, J\. Pasupulati, A\. Albalak, W\. Hua, and W\. Y\. WangMAGPIE: a benchmark for multi\-agent contextual privacy evaluation\.InSocially Responsible and Trustworthy Foundation Models at NeurIPS 2025,Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p2.1)\.
- Kirket al\.\(2024\)R\. Kirk, I\. Mediratta, C\. Nalmpantis, J\. Luketina, E\. Hambro, E\. Grefenstette, and R\. RaileanuUnderstanding the effects of rlhf on llm generalisation and diversity\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 20620–20653\.Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p4.1)\.
- Kostka and Chudziak \(2025\)A\. Kostka and J\. A\. ChudziakEvaluating theory of mind and internal beliefs in llm\-based multi\-agent systems\.InInternational Conference on Computational Collective Intelligence,pp\. 18–32\.Cited by:[§5\.4](https://arxiv.org/html/2608.13787#S5.SS4.p1.1)\.
- Kovala \(2026\)T\. KovalaA complete guide to agentforce\.Springer Books\.Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p1.1)\.
- Lewiset al\.\(2017\)M\. Lewis, D\. Yarats, Y\. Dauphin, D\. Parikh, and D\. BatraDeal or no deal? end\-to\-end learning of negotiation dialogues\.InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing,pp\. 2443–2453\.Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p3.1),[§1](https://arxiv.org/html/2608.13787#S1.p5.1),[§4\.1](https://arxiv.org/html/2608.13787#S4.SS1.SSS0.Px1.p1.1)\.
- Liaoet al\.\(2024\)A\. Liao, N\. Tomlin, and D\. KleinEfficacy of language model self\-play in non\-zero\-sum games\.arXiv preprint arXiv:2406\.18872\.Cited by:[§2](https://arxiv.org/html/2608.13787#S2.SS0.SSS0.Px3.p1.1)\.
- Liuet al\.\(2026\)S\. D\. Liu, C\. Chen, J\. S\. Xiao, L\. Lei, Y\. Zhang, Y\. Yue, and D\. Simchi\-LeviInstructing llms to negotiate using reinforcement learning with verifiable rewards\.arXiv preprint arXiv:2604\.09855\.Cited by:[§2](https://arxiv.org/html/2608.13787#S2.SS0.SSS0.Px3.p1.1)\.
- Luoet al\.\(2025\)X\. Luo, Y\. Zhang, Z\. He, Z\. Wang, S\. Zhao, D\. Li, L\. K\. Qiu, and Y\. YangAgent lightning: train any ai agents with reinforcement learning\.arXiv preprint arXiv:2508\.03680\.Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p5.1)\.
- Lyuet al\.\(2026\)N\. Lyu, B\. Xu, X\. Tian, W\. Meng, Y\. Yuan, Y\. Zhang, Z\. Huang, T\. Chua, and H\. ShenGIFT: games as informal training for generalizable llms\.arXiv preprint arXiv:2601\.05633\.Cited by:[§2](https://arxiv.org/html/2608.13787#S2.SS0.SSS0.Px3.p1.1)\.
- Maet al\.\(2026\)W\. Ma, J\. Wei, L\. Zhao, H\. Zhang, B\. Xiao, L\. Li, Q\. Yang, B\. Gao, Y\. Wang, R\. Li,et al\.Mopd: multi\-teacher on\-policy distillation for capability integration in llm post\-training\.arXiv preprint arXiv:2606\.30406\.Cited by:[§5\.3](https://arxiv.org/html/2608.13787#S5.SS3.SSS0.Px1.p2.1)\.
- Matta \(2026\)D\. MattaArtificial intelligence and theory of mind\.Journal of Psychology and AI2\(1\),pp\. 2628373\.Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p2.1)\.
- Microsoft Research, AI Frontiers \(2026\)Microsoft Research, AI FrontiersSocialReasoning\-bench: measuring whether ai agents act in users’ best interests\.Note:[https://www\.microsoft\.com/en\-us/research/blog/socialreasoning\-bench\-measuring\-whether\-ai\-agents\-act\-in\-users\-best\-interests/](https://www.microsoft.com/en-us/research/blog/socialreasoning-bench-measuring-whether-ai-agents-act-in-users-best-interests/)Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p5.1),[§4\.1](https://arxiv.org/html/2608.13787#S4.SS1.SSS0.Px5.p1.1),[§4\.1](https://arxiv.org/html/2608.13787#S4.SS1.SSS0.Px6.p1.1),[§4\.2](https://arxiv.org/html/2608.13787#S4.SS2.SSS0.Px2.p3.1),[§4\.2](https://arxiv.org/html/2608.13787#S4.SS2.SSS0.Px3.p1.1)\.
- Muet al\.\(2026\)C\. Mu, Y\. Zeng, Q\. Zhang, K\. Shao, C\. Chu, H\. Guo, D\. Jia, Z\. Wang, and S\. HuAdaptive theory of mind for llm\-based multi\-agent coordination\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 29608–29616\.Cited by:[§5\.4](https://arxiv.org/html/2608.13787#S5.SS4.p1.1)\.
- OpenAI \(2025\)OpenAIOperator system card\.\.https://openai\.com/index/operator\-system\-card\.Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p1.1)\.
- Schicket al\.\(2023\)T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, E\. Hambro, L\. Zettlemoyer, N\. Cancedda, and T\. ScialomToolformer: language models can teach themselves to use tools\.Advances in neural information processing systems36,pp\. 68539–68551\.Cited by:[§2](https://arxiv.org/html/2608.13787#S2.SS0.SSS0.Px1.p1.1)\.
- Southet al\.\(2025\)T\. South, S\. Marro, T\. Hardjono, R\. Mahari, C\. D\. Whitney, D\. Greenwood, A\. Chan, and A\. PentlandAuthenticated delegation and authorized ai agents\.arXiv preprint arXiv:2501\.09674\.Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p1.1)\.
- Sunet al\.\(2025a\)C\. Sun, S\. Huang, and D\. PompiliLlm\-based multi\-agent decision\-making: challenges and future directions\.IEEE Robotics and Automation Letters10\(6\),pp\. 5681–5688\.Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p2.1)\.
- Sunet al\.\(2025b\)H\. Sun, Y\. Wu, Y\. Cheng, and X\. ChuGame theory meets large language models: a systematic survey\.arXiv preprint arXiv:2502\.09053\.Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p1.1)\.
- Wanget al\.\(2025\)B\. Wang, C\. Lee, N\. Lee, S\. Lin, W\. Dai, Y\. Chen, Y\. Chen, Z\. Yang, Z\. Liu, M\. Shoeybi,et al\.Nemotron\-cascade: scaling cascaded reinforcement learning for general\-purpose reasoning models\.arXiv preprint arXiv:2512\.13607\.Cited by:[§5\.3](https://arxiv.org/html/2608.13787#S5.SS3.SSS0.Px1.p1.1)\.
- Wanget al\.\(2024\)R\. Wang, H\. Yu, W\. Zhang, Z\. Qi, M\. Sap, Y\. Bisk, G\. Neubig, and H\. ZhuSotopia\-π\\pi: interactive learning of socially intelligent language agents\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 12912–12940\.Cited by:[§2](https://arxiv.org/html/2608.13787#S2.SS0.SSS0.Px3.p1.1)\.
- Wanget al\.\(2026a\)Y\. Wang, X\. Chen, X\. Jin, M\. Wang, and L\. YangOpenclaw\-rl: train any agent simply by talking\.arXiv preprint arXiv:2603\.10165\.Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p5.1)\.
- Wanget al\.\(2026b\)Z\. Wang, C\. Xu, B\. Liu, Y\. Wang, S\. Han, Z\. Yao, H\. Yao, and Y\. HeAgent world model: infinity synthetic environments for agentic reinforcement learning\.arXiv preprint arXiv:2602\.10090\.Cited by:[§2](https://arxiv.org/html/2608.13787#S2.SS0.SSS0.Px1.p1.1)\.
- Xiaoet al\.\(2025\)Y\. Xiao, J\. Wang, Q\. Xu, C\. Song, C\. Xu, Y\. Cheng, W\. Li, and P\. LiuTowards dynamic theory of mind: evaluating llm adaptation to temporal evolution of human states\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 24036–24057\.Cited by:[§5\.4](https://arxiv.org/html/2608.13787#S5.SS4.p1.1)\.
- Xieet al\.\(2024\)T\. Xie, D\. Zhang, J\. Chen, X\. Li, S\. Zhao, R\. Cao, T\. J\. Hua, Z\. Cheng, D\. Shin, F\. Lei,et al\.Osworld: benchmarking multimodal agents for open\-ended tasks in real computer environments\.Advances in Neural Information Processing Systems37,pp\. 52040–52094\.Cited by:[§2](https://arxiv.org/html/2608.13787#S2.SS0.SSS0.Px1.p1.1)\.
- Yamaguchiet al\.\(2021\)A\. Yamaguchi, K\. Iwasa, and K\. FujitaDialogue act\-based breakdown detection in negotiation dialogues\.InProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume,pp\. 745–757\.Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p5.1),[§4\.1](https://arxiv.org/html/2608.13787#S4.SS1.SSS0.Px4.p1.1)\.
- Yang and Zhang \(2024\)J\. Yang and J\. ZhangA multi\-teacher policy distillation framework for enhancing zero\-shot generalization of autonomous driving policies\.IEEE Transactions on Vehicular Technology73\(7\),pp\. 9734–9746\.Cited by:[§5\.3](https://arxiv.org/html/2608.13787#S5.SS3.SSS0.Px1.p2.1)\.
- Yanget al\.\(2026a\)S\. Yang, C\. Han, X\. Ma, Y\. Li, M\. R\. G\. Madani, and E\. HovyEvotool: self\-evolving tool\-use policy optimization in llm agents via blame\-aware mutation and diversity\-aware selection\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 43553–43572\.Cited by:[§2](https://arxiv.org/html/2608.13787#S2.SS0.SSS0.Px1.p1.1)\.
- Yanget al\.\(2026b\)W\. Yang, W\. Liu, R\. Xie, K\. Yang, S\. Yang, and Y\. LinLearning beyond teacher: generalized on\-policy distillation with reward extrapolation\.arXiv preprint arXiv:2602\.12125\.Cited by:[§5\.3](https://arxiv.org/html/2608.13787#S5.SS3.SSS0.Px1.p2.1)\.
- Yaoet al\.\(2024\)S\. Yao, N\. Shinn, P\. Razavi, and K\. NarasimhanTau\-bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.arXiv preprint arXiv:2406\.12045\.Cited by:[§2](https://arxiv.org/html/2608.13787#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.13787#S2.SS0.SSS0.Px2.p1.1)\.
- Yaoet al\.\(2022\)S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. CaoReact: synergizing reasoning and acting in language models\.arXiv preprint arXiv:2210\.03629\.Cited by:[§2](https://arxiv.org/html/2608.13787#S2.SS0.SSS0.Px1.p1.1)\.
- Yuet al\.\(2025\)H\. Yu, Z\. Qi, Y\. Zhao, K\. Nottingham, K\. Xuan, B\. P\. Majumder, H\. Zhu, P\. P\. Liang, and J\. YouSotopia\-rl: reward design for social intelligence\.arXiv preprint arXiv:2508\.03905\.Cited by:[§2](https://arxiv.org/html/2608.13787#S2.SS0.SSS0.Px3.p1.1)\.
- Yuanet al\.\(2025\)H\. Yuan, Z\. Xu, Z\. Tan, X\. Yi, M\. Guang, K\. Long, H\. Hui, B\. Li, X\. Chen, B\. Zhao,et al\.MARS: reinforcing multi\-agent reasoning of llms through self\-play in strategic games\.arXiv e\-prints,pp\. arXiv–2510\.Cited by:[§2](https://arxiv.org/html/2608.13787#S2.SS0.SSS0.Px3.p1.1)\.
- Zenget al\.\(2025\)A\. Zeng, X\. Lv, Q\. Zheng, Z\. Hou, B\. Chen, C\. Xie, C\. Wang, D\. Yin, H\. Zeng, J\. Zhang,et al\.Glm\-4\.5: agentic, reasoning, and coding \(arc\) foundation models\.arXiv preprint arXiv:2508\.06471\.Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p5.1)\.
- Zhanget al\.\(2026\)E\. Zhang, F\. Zhang, A\. Pappu, B\. El, J\. Blanchet, S\. Athey, J\. Liu, and J\. ZouTERMS\-bench: diagnosing llm negotiation agents beyond deal rate\.arXiv preprint arXiv:2605\.13909\.Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p3.1)\.
- Zhouet al\.\(2023\)J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. HouInstruction\-following evaluation for large language models\.arXiv preprint arXiv:2311\.07911\.Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p4.1)\.
- Zhouet al\.\(2024\)S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried,et al\.Webarena: a realistic web environment for building autonomous agents\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 15585–15606\.Cited by:[§2](https://arxiv.org/html/2608.13787#S2.SS0.SSS0.Px1.p1.1)\.
- Zhuet al\.\(2026\)C\. Zhu, S\. Cui, W\. S\. Dufallo, R\. Jin, Z\. Xu, L\. Zhang, and D\. CainPieArena: ranking and profiling language agents in realistic negotiation scenarios\.arXiv preprint arXiv:2602\.05302\.Cited by:[§2](https://arxiv.org/html/2608.13787#S2.SS0.SSS0.Px3.p1.1)\.
- Zhuet al\.\(2025\)S\. Zhu, J\. Sun, Y\. Nian, T\. South, A\. Pentland, and J\. PeiThe automated but risky game: modeling and benchmarking agent\-to\-agent negotiations and transactions in consumer markets\.arXiv preprint arXiv:2506\.00073\.Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p5.1)\.
- Zillow \(2026\)ZillowZillow debuts ai mode\.https://www\.zillow\.com/news/zillow\-debuts\-ai\-mode\.Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p1.1)\.
- Zouet al\.\(2026\)C\. Zou, Y\. Yao, S\. She, N\. Goodman, and R\. D\. HawkinsCalBench: evaluating coordination\-privacy trade\-offs in multi\-agent llms\.arXiv preprint arXiv:2605\.09823\.Cited by:[§1](https://arxiv.org/html/2608.13787#S1.p2.1),[§2](https://arxiv.org/html/2608.13787#S2.SS0.SSS0.Px2.p1.1)\.