Preference Tree Optimization: Enhancing Goal-Oriented Dialogue with Look-Ahead Simulations

# Preference Tree Optimization: Enhancing Goal-Oriented Dialogue with Look-Ahead Simulations
Source: [https://arxiv.org/html/2608.12062](https://arxiv.org/html/2608.12062)
Lior BaruchAffiliation:School of Computer ScienceAffiliation:Reichman University, Herzliya, IsraelEmail:[lior95bar@gmail\.com](mailto:)Moshe ButmanAffiliation:School of Computer ScienceAffiliation:Reichman University, Herzliya, IsraelEmail:[moshe\.butman@gmail\.com](mailto:)Kfir BarAffiliation:School of Computer ScienceAffiliation:Reichman University, Herzliya, IsraelEmail:[kfir\.bar@runi\.ac\.il](mailto:)Doron FriedmanAffiliation:School of CommunicationsAffiliation:Reichman University, Herzliya, IsraelEmail:[doronf@runi\.ac\.il](mailto:)

August 12, 2026

###### Abstract

Developing dialogue systems capable of engaging in multi\-turn, goal\-oriented conversations remains a significant challenge, especially in specialized domains with limited data\. This research proposes a novel framework calledPreference Tree Optimization \(PTO\), designed to iteratively improve agent models in such dialogue systems, by generating preference data using a method calledPreference Tree with Look\-Ahead\. Focusing on Motivational Interviewing \(MI\)—a counseling technique aimed at facilitating behavioral change—we leverage virtual patients and an oracle evaluator to simulate conversations and generate rich preference datasets\. By combining this method with Direct Preference Optimization \(DPO\), we aim to enhance the agent’s decision\-making capabilities over iterative training cycles\. The proposed framework addresses data scarcity and advances the development of more nuanced and effective dialogue systems in goal\-oriented domains\.

Experimental evaluations demonstrate that the PTO framework enhances dialogue agents’ performance in goal\-oriented conversations within the domain of Motivational Interviewing \(MI\)\. Models trained with PTO consistently outperformed the baseline in key metrics such as session satisfaction and working alliance\. Additionally, incorporating look\-ahead simulations led to improved long\-term planning and more effective conversational strategies, with deeper look\-ahead configurations yielding the most stable and high\-scoring results\.

## 1Introduction

Goal\-oriented dialogue systems are designed to achieve specific objectives through interactive conversations\. Developing such systems in specialized domains is challenging due to the complexity of interactions and the scarcity of domain\-specific data\. Motivational Interviewing \(MI\) is such a domain – it is a counseling approach that facilitates behavioral change through collaborative, client\-centered dialogue, requiring nuanced understanding and adaptability from the conversational agent[Miller & Rollnick 1991](https://arxiv.org/html/2608.12062#bib.bib5)\.

This research introduces a framework for iteratively improving agent models in goal\-oriented dialogue systems, calledPreference Tree Optimization \(PTO\)\(see[Figure1](https://arxiv.org/html/2608.12062#S1.F1)\), by generating preference data using a novel method calledPreference Tree with Look\-Ahead\. This method systematically simulates various conversational paths and evaluates them using an oracle to generate preference data\. We use this preference data with Direct Preference Optimization \(DPO\)[Rafailov et al\. 2023](https://arxiv.org/html/2608.12062#bib.bib8)to iteratively refine the agent model, enhancing its decision\-making capabilities\.

Our approach leverages existing virtual patients and evaluators from previous research in MI[Yosef et al\. 2024](https://arxiv.org/html/2608.12062#bib.bib10), making it an ideal testbed for our framework\. By addressing the challenges of data scarcity and the need for nuanced interactions, we aim to contribute to the advancement of dialogue systems capable of effective, goal\-oriented conversations\.

Similar preference\-based strategies have improved models in well\-defined analytic tasks like games, coding, and math\. However, their application to human\-centric domains like Motivational Interviewing—where objectives are subjective and nuanced communication is key—remains largely unexplored\.

![Refer to caption](https://arxiv.org/html/2608.12062v1/fig_framework.png)Figure 1:Preference Tree Optimization \(PTO\) Framework\. The framework operates in two iterative steps: \(i\)Preference Data Generation: The User Model is prompted with a range of attributes to simulate diverse user personalities\. For each digital user personality, thePreference Tree with Look\-Ahead\(Section[3\.1](https://arxiv.org/html/2608.12062#S3.SS1)\) method is used in conjunction with the Oracle Evaluator and the current agent model \(A​g​e​n​t​\_​M​o​d​e​ltAgent\\\_Model\_\{t\}\) to generate a preference tree that explores various conversational pathways\. These trees are aggregated into a comprehensive preference dataset\. \(ii\)Model Training: The current agent model is trained on the newly generated preference dataset using Direct Preference Optimization \(DPO\), resulting in an improved model \(A​g​e​n​t​\_​M​o​d​e​lt\+1Agent\\\_Model\_\{t\+1\}\)\. The updated agent model is then used for the next iteration, repeating the process for continuous improvement\.The PTO framework is designed exclusively as an offline training paradigm\. Although the DPO process is computationally intensive—since it is applied at each simulated decision point during training—this cost is incurred only once during model development\. Once trained, the automated therapist is deployed for real\-time conversation, where the inference process is fast and efficient\.

This work makes several contributions to the advancement of goal\-oriented dialogue systems\. First, we introduce thePreference Tree with Look\-Ahead, a novel method that systematically simulates and evaluates potential conversational trajectories to generate high\-quality preference data, thereby facilitating more effective learning from interactions\. Second, we propose thePreference Tree Optimization \(PTO\)framework, which integrates this preference data withDirect Preference Optimization \(DPO\)to iteratively refine an agent’s decision\-making capabilities over successive training cycles\. Third, we validate our approach in the challenging domain ofMotivational Interviewing \(MI\)by leveraging virtual patients and oracle evaluators to simulate realistic, high\-stakes conversational scenarios\. Finally, our methodology offers broad insights and generalizable strategies for applying preference\-based optimization to other specialized dialogue domains\.

## 2Background and Related Work

Recent breakthroughs in Natural Language Processing \(NLP\) and the development of Large Language Models \(LLMs\) have dramatically advanced dialogue systems\. However, designing goal\-oriented systems for specialized domains—such as Motivational Interviewing \(MI\)—remains particularly challenging due to the limited availability of domain\-specific data and the complexity of managing nuanced, multi\-turn interactions\. Pure generative models, which primarily rely on likelihood estimation, may not naturally exhibit goal\-directed behavior\. While reinforcement learning \(RL\) offers a potential path to integrating goal orientation, identifying suitable reward functions in domains like psychology is far from straightforward—unlike more structured fields such as mathematics or gaming, where clear optimal strategies exist\. Additionally, approaches like Direct Preference Optimization \(DPO\) raise questions: Can they sufficiently promote goal\-oriented behavior, and if so, what implicit reward mechanisms do they employ?

### 2\.1Preference Optimization in Language Models

One of the key approaches to improving language models involves aligning them with human preferences\. This alignment helps models generate responses that are not only coherent, but also contextually appropriate and tailored to specific conversational objectives\. Traditional approaches, such as Reinforcement Learning from Human Feedback \(RLHF\)[Christiano et al\. 2017](https://arxiv.org/html/2608.12062#bib.bib2), involve training a separate reward model based on human evaluations of model outputs\. This reward model then guides the language model through reinforcement learning to produce preferred responses\. Although effective, RLHF can be complex and resource\-intensive due to the necessity of maintaining a distinct reward model and implementing reinforcement learning algorithms[Ouyang et al\. 2022](https://arxiv.org/html/2608.12062#bib.bib6)\.

Direct Preference Optimization \(DPO\)[Rafailov et al\. 2023](https://arxiv.org/html/2608.12062#bib.bib8)offers a more streamlined alternative by directly optimizing the language model using preference data, eliminating the need for a separate reward model and the complexities of reinforcement learning\. DPO establishes a direct mapping between LLM policies and reward functions, enabling the training of an LLM to satisfy preference data through a straightforward cross\-entropy loss\.

### 2\.2Synthetic Data Generation and Iterative Self\-Improvement

Addressing the challenge of data scarcity in specialized dialogue domains has motivated researchers to develop methods that combine synthetic data generation with iterative self\-improvement\. Broadly, these approaches can be grouped into three categories: score\-based synthetic data generation, self\-evaluation–driven improvement, and search\-based tree\-structured methods\.

Score\-Based Synthetic Data Generation:Pace et al\.[Pace et al\. 2024](https://arxiv.org/html/2608.12062#bib.bib7)introducedWest\-of\-N, a method that leverages language models to produce multiple candidate responses for a given prompt\. A reward model then scores these responses, and by selecting the best and worst outputs, the approach forms synthetic preference pairs used to refine the reward model’s alignment with human preferences\. In a similar vein,[Guo et al\. 2024](https://arxiv.org/html/2608.12062#bib.bib3)propose an online variant of direct alignment from preferences\. Their method employs an LLM as an annotator to provide on\-the\-fly feedback on pairs of responses sampled from the current model\. This Online AI Feedback \(OAIF\) approach addresses the distribution shift inherent in offline datasets by continuously updating preference data, thereby enhancing alignment performance, particularly in soft domains where nuanced judgment is critical\.

Self\-Evaluation–Driven Improvement:Another line of work harnesses the model’s internal evaluation mechanisms to self\-generate and refine synthetic data\. Yuan et al\.[Yuan et al\. 2024b](https://arxiv.org/html/2608.12062#bib.bib13)presentSelf\-Rewarding Language Models, wherein the model generates multiple responses and uses an LLM\-as\-a\-judge to rank them\. The resulting preference data is then used with Direct Preference Optimization \(DPO\) to iteratively enhance both response generation and internal reward estimation\. Similarly, Liang et al\.[Liang et al\. 2024](https://arxiv.org/html/2608.12062#bib.bib4)proposeI\-SHEEP\(Iterative Self\-EnHancEmEnt Paradigm\), a framework in which the model synthesizes data, self\-assesses its quality, and filters out low\-quality responses before applying supervised fine\-tuning\. While these self\-assessment–based methods efficiently leverage the model’s own capabilities, they risk perpetuating internal biases if the self\-evaluation is not sufficiently robust\.

Search\-Based and Tree\-Structured Approaches:A third category of methods employs search strategies to systematically explore potential outcomes\. Xie et al\.[Xie et al\. 2024](https://arxiv.org/html/2608.12062#bib.bib9)integrate Monte Carlo Tree Search \(MCTS\) with iterative preference learning to generate and evaluate fine\-grained, step\-level reasoning paths\. The collected preference data is then used to refine the model via DPO\. In addition, prior work onPreference Trees[Yuan et al\. 2024a](https://arxiv.org/html/2608.12062#bib.bib12)demonstrates how tree\-structured methods can effectively manage complex reasoning tasks in domains such as coding, math, and logic\. Yu et al\.[Yu et al\. 2023](https://arxiv.org/html/2608.12062#bib.bib11)introduce a prompt\-based search method where an LLM plays multiple roles in planning without additional training\. Recently, Chen et al\.[Chen et al\. 2025](https://arxiv.org/html/2608.12062#bib.bib1)developed a conversation planning approach that reduces reliance on direct LLM\-based simulation, by exploiting the dense semantic representation of conversations\. Building on these ideas, our framework employs aPreference Tree with Look\-Aheadto simulate full conversational trajectories using a dedicated user model, specifically targeting goal\-oriented dialogue systems like those used in Motivational Interviewing\.

In summary, these diverse methodologies illustrate the potential of combining synthetic data generation with iterative self\-improvement\. They differ in how preference data is generated, whether through score\-based selection, self\-assessment, or search\-based exploration, and in the application domains they target\. Our work bridges search\-based and score\-based paradigms: thePreference Tree with Look\-Aheadmethod employs tree\-structured exploration of conversational trajectories, while the oracle evaluator provides score\-driven comparisons, enabling iterative refinement viaDirect Preference Optimization \(DPO\)to enhance goal\-oriented dialogue agents in specialized domains such asMotivational Interviewing\. Unlike prior work, which has primarily focused on structured tasks such as coding, math, or games, our approach explores preference\-based optimization in a domain that requires deep human understanding, where objectives are inherently subjective and harder to quantify\.

### 2\.3Motivational Interviewing and AI Dialogue Systems

Motivational Interviewing \(MI\) is a client\-centered counseling approach aimed at eliciting behavioral change by helping clients explore and resolve ambivalence[Miller & Rollnick 1991](https://arxiv.org/html/2608.12062#bib.bib5)\. Implementing MI in AI dialogue systems presents unique challenges due to the need for empathy, adaptability, and the ability to interpret subtle conversational cues\.

Previous research has explored the potential of LLMs in simulating MI sessions\. Yosef et al\.[Yosef et al\. 2024](https://arxiv.org/html/2608.12062#bib.bib10)utilized AI\-generated patient simulations to assess MI sessions, highlighting the feasibility of virtual patients in training and evaluating therapeutic dialogues\. Their work demonstrated that AI agents could engage in MI conversations to a certain extent but also underscored the limitations in capturing the full depth of human therapist\-patient interactions\.

In addition, Yosef et al\. fine\-tuned therapist models using existing datasets specific to MI, demonstrating that such fine\-tuning can improve model performance in therapeutic settings[Yosef et al\. 2024](https://arxiv.org/html/2608.12062#bib.bib10)\. Unlike methods that rely on pre\-existing datasets, ourPreference Tree Optimization \(PTO\)framework iteratively generates training data from simulated conversations using thePreference Tree with Look\-Aheadmethod, refining the model at each iteration via Direct Preference Optimization\.

## 3Method

Our methodology involves two main components: thePreference Tree with Look\-Aheadmethod for preference data generation and an iterative training process to refine the agent model using DPO\.

### 3\.1Preference Tree with Look\-Ahead

ThePreference Tree with Look\-Aheadmethod systematically explores potential conversational paths by simulating multiple agent responses and their subsequent dialogue trajectories, as shown in[SectionA\.1](https://arxiv.org/html/2608.12062#A1.SS1)\. This is intended to allow the agent to anticipate the long\-term impact of its responses\. The process is as follows:

1. 1\.Agent Decision Point: At each turn, the agent model generatesNNpossible responses\.
2. 2\.Branch Initialization: For each response, a new branch is created, and the response is appended to the conversation history\.
3. 3\.Look\-Ahead Simulation: Each branch simulatesKKfuture steps, alternating between the agent and the virtual patient, to anticipate the long\-term implications of the agent’s response\.
4. 4\.Oracle Evaluation: An oracle evaluator assesses each branch based on predefined criteria \(e\.g\., adherence to MI principles, empathy, goal progression\) and assigns scores\.
5. 5\.Preference Recording: The response with the highest score is considered the preferred response, and the one with the lowest score is the least preferred\. The preference tuple is recorded in the dataset\.
6. 6\.Conversation Update: The conversation continues with the preferred response, and the process repeats until a termination condition is met \(e\.g\., reaching maximum conversation length or achieving the goal\)\.

By considering future conversation trajectories, the agent is expected to learn to make decisions that are not only immediately appropriate but also beneficial in the long term \(see[Figure2](https://arxiv.org/html/2608.12062#S3.F2)\)\.

![Refer to caption](https://arxiv.org/html/2608.12062v1/fig_preftree.png)Figure 2:Preference Tree Generation Process\.The figure shows how a preference tree is used to generate preference data\. At each conversation stepii, the agent generatesNNpossible responses, and each branch simulates the conversation through several look\-ahead steps\. These branches represent possible future dialogue paths\. An oracle evaluates each path, assigning scores to determine the best \(r​e​s​p​o​n​s​ei,w​i​nresponse\_\{i,win\}\) and worst \(r​e​s​p​o​n​s​ei,l​o​s​eresponse\_\{i,lose\}\) outcomes\. After selecting the winning response, the user model replies, advancing to the next conversation stepc​o​n​v​e​r​s​a​t​i​o​ni\+1conversation\_\{i\+1\}, and the process repeats\. This way, each preference tree produces multiple preference samples, with each sample consisting of a tuple\(c​o​n​v​e​r​s​a​t​i​o​ni,r​e​s​p​o​n​s​ei,l​o​s​e,r​e​s​p​o​n​s​ei,w​i​n\)\(conversation\_\{i\},response\_\{i,lose\},response\_\{i,win\}\)\.
### 3\.2Preference Tree Optimization \(PTO\) Framework

This process forms the Preference Tree Optimization \(PTO\) Framework\. The agent model is iteratively improved through cycles of preference data generation and training using DPO\.

1. 1\.Initial Training: The agent model is initially trained on available data or pre\-trained weights\.
2. 2\.Preference Data Generation: Using the current agent model, thePreference Tree with Look\-Aheadmethod generates new preference data, capturing the agent’s strengths and weaknesses\.
3. 3\.Preference Data Filtering: We retain a preference sample only if the winning score surpasses the losing score by a predefined threshold\. In our experiments, we used a threshold value of 0\.1, ensuring that only clearly distinguishable preference pairs contribute to training\.
4. 4\.Model Update: The agent model is fine\-tuned using DPO on the newly generated preference data, optimizing it directly based on preferences without the need for a reward model\.
5. 5\.Evaluation: The updated model is evaluated using predefined metrics to assess improvements\.
6. 6\.Iteration: Steps 2\-5 are repeated, allowing the agent to improve over time through continuous learning\.

This process balances exploration \(generating new conversational paths\) and exploitation \(refining the agent’s responses\), leading to incremental enhancements in performance\.

Algorithm 1Preference Tree Optimization \(PTO\) Framework1:Initial agent model

A\(0\)A^\{\(0\)\}, user model

UU, oracle evaluator

OO, maximum conversation length

LL, look\-ahead steps

KK, branching factor

NN, trees per iteration

TT, total iterations

II, filtering threshold

τ\\tau
2:Sequence of optimized agent models

\{A\(1\),A\(2\),…,A\(I\)\}\\\{A^\{\(1\)\},A^\{\(2\)\},\\dots,A^\{\(I\)\}\\\}
3:for

i←1i\\leftarrow 1to

IIdo

4:Initialize preference dataset:

D\(i\)←∅D^\{\(i\)\}\\leftarrow\\varnothing
5:for

t←1t\\leftarrow 1to

TTdo

6:Assign user role:

Ut←UU\_\{t\}\\leftarrow U
7:

P\(t\)←GeneratePreferenceTree​\(A\(i−1\),Ut,O,L,K,N\)P^\{\(t\)\}\\leftarrow\\textbf\{GeneratePreferenceTree\}\(A^\{\(i\-1\)\},U\_\{t\},O,L,K,N\)⊳\\trianglerightSee[Algorithm2](https://arxiv.org/html/2608.12062#alg2)

8:Aggregate preferences:

D\(i\)←D\(i\)∪P\(t\)D^\{\(i\)\}\\leftarrow D^\{\(i\)\}\\cup P^\{\(t\)\}
9:endfor

10:

D\(i\)←Filter​\(D\(i\),τ\)D^\{\(i\)\}\\leftarrow\\textbf\{Filter\}\(D^\{\(i\)\},\\tau\)⊳\\trianglerightRetain samples where the winning score exceeds the losing score by at leastτ\\tau

11:

A\(i\)←DPO​\(A\(i−1\),D\(i\)\)A^\{\(i\)\}\\leftarrow\\textbf\{DPO\}\(A^\{\(i\-1\)\},D^\{\(i\)\}\)
12:endfor

13:returnOptimized agent models

\{A\(1\),A\(2\),…,A\(I\)\}\\\{A^\{\(1\)\},A^\{\(2\)\},\\dots,A^\{\(I\)\}\\\}

## 4Experimental Setup

During our experiments, GPT\-3\.5 served as both the user simulator and the oracle evaluator\. Notably, GPT\-3\.5 was used in fixed, separate roles \(with distinct prompts for the user simulation and the oracle evaluation\), and it was not updated or fine\-tuned at any point during the training process\. This ensured that the model’s parameters remained unchanged throughout, providing a consistent but unlearned behavior in each role\.

To evaluate our proposed framework, we conducted a series of initial experiments in the Motivational Interviewing \(MI\) domain\. The experimental setup is detailed as follows:

### 4\.1Models and Tools

- •Agent Model: We utilizedLlama\-2\-7Bas the base model for the therapist agent\.
- •User Model: Virtual patients were simulated usingGPT\-3\.5, based on guidelines from previous MI research[Yosef et al\. 2024](https://arxiv.org/html/2608.12062#bib.bib10)\. Each patient is defined by parameters such as gender, age, problem \(smoking/obesity\), duration, prior attempts to resolve the issue, and cooperation level, creating 96 unique profiles to capture diverse challenges and attitudes toward counseling\.
- •Oracle Evaluator: GPT\-3\.5 model is used as the oracle evaluator, using specific questionnaires designed to assess MI adherence and conversational quality based on the guidelines from previous research[Yosef et al\. 2024](https://arxiv.org/html/2608.12062#bib.bib10)and detailed in[SectionA\.2](https://arxiv.org/html/2608.12062#A1.SS2)\. The final score is calculated as the average of the two questionnaire scores, where each questionnaire score is the average of its respective question scores\.

### 4\.2Experimental Variables

- •Look\-Ahead Depths: We tested two different look\-ahead depths: 0 \(no look\-ahead\) and 5\. This variable assesses the impact of anticipating future conversational turns on the agent’s performance\.
- •Iterations per Look\-Ahead: For each look\-ahead depth, we conducted 7 iterative training cycles\. Each iteration involved: 1. 1\.Preference Data Generation: Utilizing thePreference Tree with Look\-Aheadmethod to generate preference tuples based on simulated conversational paths\. 2. 2\.Model Fine\-Tuning: Applying Direct Preference Optimization \(DPO\) to fine\-tune the agent model using the newly generated preference data\.

### 4\.3Data Collection

After each iteration, we generated a set of conversations to evaluate the agent’s performance:

- •Number of Conversations: For each trained model, we conducted 96 separate conversations with virtual patients to ensure a comprehensive assessment\.
- •Evaluation Metrics: Each conversation was scored by the oracle evaluator based on two distinct questionnaires designed to measure MI adherence and overall conversational quality, detailed in Table[3](https://arxiv.org/html/2608.12062#A1.T3)\.

## 5Results

To assess the efficacy of the proposed Preference Tree Optimization \(PTO\) Framework, we conducted experiments concentrating on two distinct look\-ahead depths: 0 and 5\. Each configuration was subjected to seven iterative training cycles, and their performances were compared against the baseline model,Llama\-2\-7B\.

### 5\.1Performance Metrics

The agent’s effectiveness was evaluated using two primary metrics derived from the oracle evaluator’s questionnaires \(see[Table3](https://arxiv.org/html/2608.12062#A1.T3)\):

- •Session Satisfaction \(Q1\): This metric aggregates scores from Questionnaire 1 \(as detailed in[Yosef et al\. 2024](https://arxiv.org/html/2608.12062#bib.bib10)\), assessing overall satisfaction, content relevance, motivation facilitation, learning outcomes, and applicability to everyday life\.
- •Working Alliance \(Q2\): This metric aggregates scores from Questionnaire 2 \(see[Yosef et al\. 2024](https://arxiv.org/html/2608.12062#bib.bib10)\), evaluating the therapist’s interpersonal skills, empathy, communication effectiveness, and ability to establish a collaborative relationship\.
- •Final Score: Calculated as the average of Session Satisfaction and Working Alliance scores, this provides a comprehensive indicator of overall performance\.

### 5\.2Results Overview

Table 1:Average Performance Scores and Standard Deviations Across Models[Table1](https://arxiv.org/html/2608.12062#S5.T1)presents the mean scores and standard deviations forSession Satisfaction \(Q1\),Working Alliance \(Q2\), and theFinal Scoreacross all evaluated models, including the baseline \(Llama\-2\-7B\) and PTO\-enhanced models at look\-ahead depths of 0 and 5\. The lowest standard deviation values for each metric are underlined in the table\.

Across all evaluated metrics, every PTO\-trained model \(L0\_Mx and L5\_Mx\) outperforms the baseline \(see[Figure3](https://arxiv.org/html/2608.12062#S5.F3)\), demonstrating that preference\-based optimization improves goal\-oriented dialogue performance\. Additionally, models trained with deeper look\-ahead \(depth\-5\) achieve higher scores than those trained with no look\-ahead \(depth\-0\), suggesting that anticipating future conversational paths enhances both session satisfaction and the working alliance\.

![Refer to caption](https://arxiv.org/html/2608.12062v1/fig_barplot_q1q2.png)Figure 3:Comparative Performance Analysis
Bar charts illustrating the average scores for theFinal Score\(left\),Session Satisfaction \(Q1\)\(middle\), andWorking Alliance \(Q2\)\(right\) across theBaselinemodel \(Llama\-2\-7B\) and thePTO\-enhancedmodels with varying look\-ahead depths\. Error bars represent the 95% confidence intervals\. This comparison highlights the performance improvements achieved through the Preference Tree Optimization Framework\.A one\-way ANOVA confirms that model choice significantly influences Q1, Q2, Final Score, and conversation length as shown in[Table2](https://arxiv.org/html/2608.12062#S5.T2)\. Post\-hoc Tukey HSD tests were conducted to compare the baseline model \(Llama\-2\-7B\) against the best\-performing models from each look\-ahead depth:L0\_M4\(best\-performing depth\-0 model\) andL5\_M7\(best\-performing depth\-5 model\) \([SectionA\.3](https://arxiv.org/html/2608.12062#A1.SS3)\)\. Results indicate that bothL0\_M4andL5\_M7significantly outperform the baseline across all three metrics \(Q1, Q2, and Final Score\)\. While L5\_M7 achieves the highest Final Score, its improvement over L0\_M4 is only statistically significant for Q2, indicating that deeper look\-ahead particularly strengthens the working alliance\.

Examining the standard deviations in Table[1](https://arxiv.org/html/2608.12062#S5.T1)further supports the stability of PTO\-trained models\. Among all evaluated models,L5\_M7exhibits the lowest variance across Q1, Q2, and Final Score \(underlined in the table\), suggesting that deeper look\-ahead not only enhances performance but also ensures more consistent and reliable motivation interventions\.

Furthermore, Figure[4](https://arxiv.org/html/2608.12062#S5.F4)illustrates that PTO\-trained models tend to reduce conversation length compared to the baseline, reflecting more focused interactions\. Notably,L5\_M7achieves the most substantial reduction, decreasing the average number of dialogue turns from 43\.7 \(baseline\) to 34\.4\. This underscores the role of look\-ahead in streamlining interactions while maintaining high conversation quality\.

Table 2:One\-Way ANOVA Results for Model Performance![Refer to caption](https://arxiv.org/html/2608.12062v1/fig_barplot_len.png)Figure 4:Barplot of Conversation Length
This barplot displays the average conversation lengths for each model, comparing theBaselinemodel with thePTO\-enhancedmodels at different look\-ahead depths\. Error bars represent the 95% confidence intervals\. It highlights how the Preference Tree Optimization Framework influences the efficiency and duration of dialogues\.

## 6Discussion

Our experimental results demonstrate that the Preference Tree Optimization \(PTO\) framework consistently improves dialogue performance compared to the baseline\. Importantly, these improvements were achieved using a base pre\-trained model \(Llama\-2\-7B\) that was neither instruction\-tuned nor fine\-tuned via supervised learning; instead, all training was conducted solely with data generated by the Preference Tree with Look\-Ahead method\. Both look\-ahead configurations \(depth\-0 and depth\-5\) yield significant gains in Session Satisfaction \(Q1\), Working Alliance \(Q2\), and overall Final Score\. Notably, the best\-performing depth\-5 model \(L5\_M7\) not only achieved the highest scores but also exhibited the lowest variance, indicating more stable and reliable interactions\. This suggests that incorporating look\-ahead enables the agent to anticipate future conversational turns, leading to more effective, empathetic, and streamlined dialogues\.

Potential biases in automated evaluation remain a concern\. For instance, positional bias may occur if the evaluator assigns different weights to responses depending on their position in the conversation\. Our analysis shows that while there is minor variability in the evaluation of the initial utterances, the oracle’s scoring remains largely consistent throughout the dialogue\. Similarly, preference bias can emerge if the evaluator favors certain stylistic or content\-related features—such as preferring responses typical of language models over those created by humans—which could lead the agent to optimize for superficial attributes rather than genuine conversational quality\. This can be considered a type of “reward hacking”\.

In our case, both the oracle evaluator and the virtual patients are implemented as fixed, pre\-trained models\. Thus, the fact that we employ the same underlying model for both roles is not the primary source of risk for “reward hacking”\. Reward hacking is an inherent challenge in frameworks that rely on automated evaluation, regardless of whether identical or heterogeneous models are used\. Importantly, our oracle evaluator was validated by human assessments—although the correlation was moderate, this validation indicates that the evaluation criteria capture meaningful aspects of effective counseling\.

Future work will focus on elucidating whether and why deeper look\-ahead \(L5\) offers advantages over no look\-ahead \(L0\) in this soft domain\. We plan to investigate the underlying mechanisms that contribute to improved long\-term planning, such as better management of conversational dynamics and enhanced anticipatory decision\-making\. Additionally, we plan to benchmark our approach against leading state\-of\-the\-art methods—specifically, the online alignment framework from[Guo et al\. 2024](https://arxiv.org/html/2608.12062#bib.bib3)and the self\-rewarding language model approach from[Yuan et al\. 2024b](https://arxiv.org/html/2608.12062#bib.bib13)—to further elucidate the advantages and limitations of long\-term planning in goal\-oriented dialogue\.

## Acknowledgments

This work was partially supported by the European Commission Horizon 2020 project GuestXR \(\#101017884\)\.

## References

- Chen et al\. \(2025\)Zhiliang Chen, Xinyuan Niu, Chuan\-Sheng Foo, and Bryan Kian Hsiang Low\.Broaden your scope\! efficient multi\-turn conversation planning for llms with semantic space\.In*The Thirteenth International Conference on Learning Representations*, 2025\.
- Christiano et al\. \(2017\)Paul F\. Christiano, Jan Leike, Tom B\. Brown, Miljan Martic, Shane Legg, and Dario Amodei\.Deep reinforcement learning from human preferences\.*arXiv preprint arXiv:1706\.03741*, 2017\.URL[https://arxiv\.org/abs/1706\.03741](https://arxiv.org/abs/1706.03741)\.Presented at the 31st Conference on Neural Information Processing Systems \(NeurIPS 2017\)\.
- Guo et al\. \(2024\)Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Ramé, Thomas Mesnard, Yao Zhao, Bilal Piot, Johan Ferret, and Mathieu Blondel\.Direct language model alignment from online ai feedback\.*arXiv preprint arXiv:2402\.04792*, 2024\.URL[https://arxiv\.org/abs/2402\.04792](https://arxiv.org/abs/2402.04792)\.
- Liang et al\. \(2024\)Yiming Liang, Ge Zhang, Xingwei Qu, Tianyu Zheng, Jiawei Guo, Xinrun Du, ZhenZhu Yang, Jiaheng Liu, Chenghua Lin, Lei Ma, Wenhao Huang, and Jiajun Zhang\.I\-sheep: Self\-alignment of llm from scratch through an iterative self\-enhancement paradigm\.*Proceedings of the AAAI Conference on Artificial Intelligence*, 2024\.URL[https://www\.arxiv\.org/abs/2408\.08072](https://www.arxiv.org/abs/2408.08072)\.Copyright © 2024, Association for the Advancement of Artificial Intelligence \(www\.aaai\.org\)\. All rights reserved\.
- Miller & Rollnick \(1991\)W\.R\. Miller and S\. Rollnick\.*Motivational Interviewing: Preparing People to Change Addictive Behavior*\.Guilford Publications, 1991\.ISBN 9780898625660\.URL[https://books\.google\.co\.il/books?id=h16\_QgAACAAJ](https://books.google.co.il/books?id=h16_QgAACAAJ)\.
- Ouyang et al\. \(2022\)Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L\. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe\.Training language models to follow instructions with human feedback\.*arXiv preprint arXiv:2203\.02155*, 2022\.URL[https://arxiv\.org/abs/2203\.02155](https://arxiv.org/abs/2203.02155)\.Work by the OpenAI team\.
- Pace et al\. \(2024\)Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn\.West\-of\-n: Synthetic preference generation for improved reward modeling\.*arXiv preprint arXiv:2401\.12086*, 2024\.URL[https://arxiv\.org/abs/2401\.12086](https://arxiv.org/abs/2401.12086)\.
- Rafailov et al\. \(2023\)Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D\. Manning, and Chelsea Finn\.Direct preference optimization: Your language model is secretly a reward model\.*arXiv preprint arXiv:2305\.18290*, 2023\.URL[https://arxiv\.org/abs/2305\.18290](https://arxiv.org/abs/2305.18290)\.Accepted at the 37th Conference on Neural Information Processing Systems \(NeurIPS 2023\)\.
- Xie et al\. \(2024\)Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min\-Yen Kan, Timothy Lillicrap, Kenji Kawaguchi, and Michael Shieh\.Monte carlo tree search boosts reasoning via iterative preference learning\.*arXiv preprint arXiv:2405\.00451v2*, 2024\.URL[https://github\.com/YuxiXie/MCTS\-DPO](https://github.com/YuxiXie/MCTS-DPO)\.
- Yosef et al\. \(2024\)Stav Yosef, Moreah Zisquit, Ben Cohen, Anat Brunstein Klomek, Kfir Bar, and Doron Friedman\.The journey towards an automatic mental health therapist\.*Preprint*, 2024\.
- Yu et al\. \(2023\)Xiao Yu, Maximillian Chen, and Zhou Yu\.Prompt\-based monte\-carlo tree search for goal\-oriented dialogue policy planning\.*arXiv preprint arXiv:2305\.13660*, 2023\.
- Yuan et al\. \(2024a\)Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun\.Advancing llm reasoning generalists with preference trees\.*arXiv preprint arXiv:2404\.02078*, 2024a\.URL[https://arxiv\.org/abs/2404\.02078](https://arxiv.org/abs/2404.02078)\.Preprint\.
- Yuan et al\. \(2024b\)Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston\.Self\-rewarding language models\.*arXiv preprint arXiv:2401\.10020*, 2024b\.URL[https://arxiv\.org/abs/2401\.10020](https://arxiv.org/abs/2401.10020)\.

## Appendix AAppendix

### A\.1Preference Tree with Look\-Ahead Algorithm

Algorithm 2Preference Tree with Look\-Ahead1:

- •Agent modelAA
- •User modelUU
- •Oracle evaluatorOO
- •Maximum conversation lengthLL
- •Look\-ahead depthKK
- •Number of candidate responsesNN

2:Preference dataset

DD
3:

D←∅D\\leftarrow\\emptyset⊳\\trianglerightInitialize the preference dataset

4:

C←∅C\\leftarrow\\emptyset⊳\\trianglerightInitialize the conversation history

5:Initialize

CCwith the starting context

6:whilelength\(

CC\)

<<LLdo

7:Agent Decision Phase:

8:Generate

NNcandidate responses:

R←\{r1,r2,…,rN\}R\\leftarrow\\\{r\_\{1\},r\_\{2\},\\dots,r\_\{N\}\\\}from

AA
9:

S←∅S\\leftarrow\\emptyset⊳\\trianglerightInitialize the list to store branch scores

10:foreach response

ri∈Rr\_\{i\}\\in Rdo

11:Initialize Branch:

12:

Ci←CC\_\{i\}\\leftarrow C⊳\\trianglerightClone the current conversation history

13:Append

rir\_\{i\}to

CiC\_\{i\}
14:Simulate Look\-Ahead:

15:

s​t​e​p​s←0steps\\leftarrow 0
16:

c​u​r​r​e​n​t​\_​t​u​r​n←Usercurrent\\\_turn\\leftarrow\\text\{User\}
17:while

s​t​e​p​s<Ksteps<Kandtermination condition not metdo

18:if

c​u​r​r​e​n​t​\_​t​u​r​n=Usercurrent\\\_turn=\\text\{User\}then

19:

u←U⁡\(Ci\)u\\leftarrow U\(C\_\{i\}\)⊳\\trianglerightGenerate a user response

20:Append

uuto

CiC\_\{i\}
21:

c​u​r​r​e​n​t​\_​t​u​r​n←Agentcurrent\\\_turn\\leftarrow\\text\{Agent\}
22:else

23:

a←A⁡\(Ci\)a\\leftarrow A\(C\_\{i\}\)⊳\\trianglerightGenerate an agent response

24:Append

aato

CiC\_\{i\}
25:

c​u​r​r​e​n​t​\_​t​u​r​n←Usercurrent\\\_turn\\leftarrow\\text\{User\}
26:endif

27:

s​t​e​p​s←s​t​e​p​s\+1steps\\leftarrow steps\+1
28:endwhile

29:Evaluate Branch:

30:Compute branch score

si←O⁡\(Ci\)s\_\{i\}\\leftarrow O\(C\_\{i\}\)
31:Add

sis\_\{i\}to

SS
32:endfor

33:Determine Preferences:

34:

w←arg⁡max⁡\(S\)w\\leftarrow\\arg\\max\(S\)⊳\\trianglerightIndex of the preferred response

35:

l←arg⁡min⁡\(S\)l\\leftarrow\\arg\\min\(S\)⊳\\trianglerightIndex of the least preferred response

36:Let

rw←R⁡\[w\]r\_\{w\}\\leftarrow R\[w\]and

rl←R⁡\[l\]r\_\{l\}\\leftarrow R\[l\]
37:Record Preference Tuple:

38:

D←D∪\{\(C,rw,rl\)\}D\\leftarrow D\\cup\\\{\(C,r\_\{w\},r\_\{l\}\)\\\}
39:Update Conversation History:

40:Append

rwr\_\{w\}to

CC
41:

u←U⁡\(C\)u\\leftarrow U\(C\)⊳\\trianglerightGenerate the subsequent user reply

42:Append

uuto

CC
43:iftermination condition is metthen

44:Exit Loop:break

45:endif

46:endwhile

47:return

DD

### A\.2Evaluation Questionnaires for Therapist Performance

Table 3:The questions posed to the LLM for evaluating the performance of the therapist\.
### A\.3Tukey HSD Post\-Hoc Analysis

Table 4:Tukey HSD Post\-Hoc Test Results for Pairwise Model Comparisons