Preference Estimation via Opponent Modeling in Multi-Agent Negotiation

# Preference Estimation via Opponent Modeling in Multi-Agent Negotiation

Source: https://arxiv.org/html/2604.15687

Yuta Konishi¹, Kento Yamamoto², Eisuke Sonomoto², Rikuho Takeda², Ryo Furukawa², Yusuke Muraki², Takafumi Shimizu¹, Kazuma Fukumura¹, Yuya Kanemoto²,†, Takayuki Ito¹, Shiyao Ding¹,†

¹Graduate School of Informatics, Kyoto University, Kyoto, Japan
²Accenture Japan Ltd, Tokyo, Japan

†[email protected], [email protected]

## Abstract

Automated negotiation in complex, multi-party and multi-issue settings critically depends on accurate opponent modeling. However, conventional numerical-only approaches fail to capture the qualitative information embedded in natural language interactions, resulting in unstable and incomplete preference estimation. Although Large Language Models (LLMs) enable rich semantic understanding of utterances, it remains challenging to quantitatively incorporate such information into consistent opponent modeling. To tackle this issue, we propose a novel preference estimation method integrating natural language information into a structured Bayesian opponent modeling framework. Our approach leverages LLMs to extract qualitative cues from utterances and converts them into probabilistic formats for dynamic belief tracking. Experimental results on a multi-party benchmark demonstrate that our framework improves the full agreement rate and preference estimation accuracy by integrating probabilistic reasoning with natural language understanding.

## 1 Introduction

In modern society, automated negotiation is a pivotal technology for conflict resolution and efficient consensus-building among diverse stakeholders (Memo et al., 2025; Bagga et al., 2021). Historically, the field has matured through integrated development environments like the General Environment for Negotiation with Intelligent multi-purpose Usage Simulation (GENIUS) (Lin et al., 2014) and international competitions such as the Automated Negotiating Agents Competition (ANAC) (Baarslags et al., 2012). A significant milestone was the BOA architecture (Baarslags et al., 2014), which standardized negotiating agents into three decoupled components: the bidding strategy, opponent model, and acceptance strategy. Within multi-party, multi-issue settings, opponent modeling remains essential for strategic decision-making (Baarslags et al., 2016). Traditionally, these models have evolved through Bayesian learning (ZENG and SYCARA, 1998; Hindriks and Tykhonov, 2008) and reinforcement learning (He et al., 2016), primarily estimating utility functions from numerical proposal histories. However, numerical-only methods struggle to capture qualitative contexts, leading to unstable estimation under high information uncertainty (Baarslags et al., 2016).

To address these limitations, integrating Large Language Models (LLMs) into negotiation and decision-making frameworks has gained traction (Abdelnabi et al., 2024; Fu et al., 2023). LLMs possess sophisticated capabilities for context understanding and Theory of Mind (ToM) (Kosinski, 2024; Chan et al., 2024), enabling the extraction of qualitative preference signals typically lost in conventional models. Nevertheless, directly applying reasoning techniques like Chain-of-Thought (CoT) (Wei et al., 2022), Tree of Thoughts (ToT) (Yao et al., 2023), or Multi-Agent Debate (MAD) (Liang et al., 2024) to LLM-based agents reveals new challenges: a lack of strategic consistency during prolonged negotiations (Chan et al., 2024), fragile generalization across different problem settings (Zhao et al., 2025), and an exponential increase in inference complexity as the amount of available information grows (Abdelnabi et al., 2024). In addition, prior work on natural language negotiation using LLMs (Chen et al., 2024; Chan et al., 2024) has largely focused on intent inference in static or short-horizon evaluation settings, where strategic dynamics are limited. Such approaches often lack a formal mechanism for belief updating over time, thereby hampering stable preference tracking in dynamic negotiation scenarios.

**Figure 1:** Overview of the negotiation flow (left) and the proposed Bayesian opponent modeling process (right). The left panel shows the three phases: initial proposal by p₁, intermediate rounds with deal and utterance exchanges among agents p₁...p₆, and the final proposal by p₁. Each deal at round t is represented as dₜ = (oₜ¹,...,oₜᴹ), where oₜᵐ denotes the selected option for issue iₘ. In this example, issues are denoted by capital letters and options by indices. For instance, the initial proposal d₁ = (A1, B3, C3, D1, E4) corresponds to d₁ = (o₁¹ = A1, o₁² = B3, o₁³ = C3, o₁⁴ = D1, o₁⁵ = E4) meaning that option 1 is selected for issue A, option 3 for issue B, and so on. The right panel illustrates p₁'s internal modeling for a hypothesis hₖ, consisting of: (1) parsing the opponent's utterance uₜ to estimate preference signals zₜ; (2) calculating the likelihood for zₜ (via Luce's Choice Axiom) and the likelihood of the proposed deal dₜ (via normal distribution approximation); and (3) updating the posterior probability of hₖ through Bayesian fusion.

To address these challenges, we propose a novel preference estimation method that integrates natural language signals from dialogue into a structured Bayesian framework. Our approach utilizes LLMs to extract qualitative cues and subsequently converts these cues into a format compatible with probabilistic models for dynamic belief tracking.

Our main contributions are summarized as follows. First, we propose an integrated framework that complements qualitative intent extraction via LLMs with quantitative preference estimation through Bayesian inference. Second, we demonstrate that the proposed method achieves superior preference estimation performance in complex multi-party scenarios compared to baselines relying solely on numerical data or direct LLM inference. Finally, we show that our framework improves agreement rates even under high uncertainty, thereby facilitating more effective autonomous negotiation.

## 2 Problem Formulation

We adopt the scorable negotiation framework proposed in Abdelnabi et al. (2024). Let P = {p₁,...,pₙ} denote the set of parties with pₙ ∈ P, and I = {i₁,...,iₘ} denote the set of issues with iₘ ∈ I, where each issue iₘ has a finite option set Oₘ = {o₁ᵐ,...,oₖₘᵐ} with oₖᵐ ∈ Oₘ. A deal dₜ proposed at round t is defined as a tuple of options dₜ = (oₜ¹,...,oₜᴹ), where each oₜᵐ ∈ Oₘ is selected for the corresponding issue iₘ.

Each party pₙ holds a private score function sₙᵐ: Oₘ → ℝ for each issue iₘ ∈ I, and the utility of a deal dₜ for party pₙ is defined as the sum of these scores:

Uₙ(dₜ) = ∑ₘ₌₁ᴹ sₙᵐ(oₜᵐ).     (1)

Upon reaching an agreement, each party receives the utility Uₙ(dₜ). Otherwise, each party receives their Best Alternative to a Negotiated Agreement (BATNA), represented by a private reservation threshold τₚₙ ∈ ℝ.

The left panel of Figure 1 illustrates the flow of the negotiation. The negotiation lasts for up to T rounds. In each round t, a designated party proposes a deal dₜ and a natural language utterance uₜ, without revealing score functions; parties infer other parties' preferences from the history of dₜ and uₜ. The success of the negotiation is determined by the deal dₜ in the final round. An agreement is reached if and only if at least a minimum required number of parties, including all veto holders, satisfy Uₙ(dₜ) > τₚₙ.

## 3 Bayesian Preference Estimation Method

In this section, we describe our method for explicitly estimating opponents' preferences at each negotiation round. Our approach builds on the Bayesian opponent modeling framework established by Hindriks and Tykhonov (2008), extending it to integrate natural language information using an LLM. The right panel of Figure 1 illustrates the specific mechanism of our proposed opponent modeling framework.

### 3.1 Model Representation and Hypothesis Space

First, we define the representation of the opponent's strategy and the space of possible preferences. We represent an opponent's preference model using two components: an issue-weight vector **w** = [w₁,...,wₘ], which captures the relative importance of each issue, and a set of evaluation functions **v** = [v₁,...,vₘ], where each vₘ specifies the preference shape over the options of issue iₘ. Based on these components, the agent maintains a finite hypothesis space over possible opponent preferences.

#### Hypothesis Space

To estimate the opponent's score function, the agent maintains a finite set of candidate hypotheses H = {h₁,...,hₖ}. Each hypothesis hₖ ∈ H represents a specific combination of a weight vector **w**(k) = [w₁(k),...,wₘ(k)], denoting the relative importance of each issue, and a vector of evaluation functions **v**(k) = [v₁(k),...,vₘ(k)], representing the preference shapes for each issue.

#### Estimated Utility Function

Under a given hypothesis hₖ, the estimated utility Û(dₜ;hₖ) of a deal dₜ is modeled as an additive utility function. It is calculated as the weighted sum of the evaluation functions:

Û(dₜ;hₖ) = ∑ₘ₌₁ᴹ wₘ(k) · vₘ(k)(oₜᵐ).     (2)

#### Likelihood Based on Numerical Offers

Assuming the opponent follows a concession-based strategy, we define the likelihood P(dₜ|hₖ) of observing a deal dₜ under hypothesis hₖ. This is based on the proximity between the estimated utility Û(dₜ;hₖ) and u'(t):

P(dₜ|hₖ) ∝ exp(−(Û(dₜ;hₖ) − u'(t))²/(2σ²)).     (3)

Here, u'(t) denotes the opponent's assumed target utility at round t, reflecting a concession-based aspiration level over time. Although we adopt a concession-style strategy here as a standard baseline for the offer-based likelihood, this component is modular and can be replaced with other behavioral models without changing the linguistic likelihood or the Bayesian fusion rule.

In addition, despite the factorial growth of the hypothesis space as the number of issues increases, prior work (Hindriks and Tykhonov, 2008) has proposed scalable approximations for this class of Bayesian opponent modeling, which can be incorporated into our framework without changing the Bayesian update itself.

### 3.2 Linguistic Likelihood Estimation via LLM

We describe how linguistic utterances are converted into probabilities over opponent preferences. As a complement to the likelihood from observed deals in Eq. (3), we define a linguistic likelihood over opponent preferences based on utterances.

#### Signal Extraction via LLM

We use an LLM to parse an utterance uₜ into a structured signal zₜ. Each signal zₜ is represented by the following two attributes:

- **Target:** the issue or option referred to by the signal. A target can take one of four forms: (i) a single issue, (ii) a comparison between two issues, (iii) a single option, or (iv) a comparison between two options.
- **Stance:** the attitude toward the target, such as "prefer" or "oppose." Together, these attributes allow the agent to convert qualitative information—for example, "Issue i₁ is important" or "Option o₁¹ is preferable to o₁²"—into a form suitable for probabilistic computation.

In the current formulation, we assume that such linguistic signals are broadly truthful. This assumption could be relaxed in future work by introducing a reliability parameter that dynamically controls the contribution of linguistic evidence based on its consistency with observed offers and dialogue over the course of the negotiation.

#### Likelihood Calculation based on Luce's Axiom

To quantify the likelihood P(zₜ|hₖ), we apply Luce's Choice Axiom. For instance, the probability of observing a signal that indicates a preference for issue iₓ is defined as:

P(zₜ ∈ Z_{iₓ,pref}|hₖ) = wₓ(k) / ∑ₘ₌₁ᴹ wₘ(k),     (4)

where Z_{iₓ,pref} denotes the set of signals representing a "prefer" stance toward issue iₓ. Similarly, likelihoods for comparison or opposition are calculated based on the relative ratios of components within **w**(k) and **v**(k) for each hypothesis hₖ.

### 3.3 Preference Update via Multimodal Observations

We now integrate the numerical-offer likelihood in Eq. (3) and the linguistic likelihood in Eq. (4) into a unified Bayesian update rule. For simplicity, we adopt a Naive Bayes assumption under which the numerical offer dₜ and the linguistic signal zₜ are conditionally independent given a hypothesis hₖ. Under this assumption...