ToSCA: Leveraging Hierarchical Reinforcement Learning on Temporal and Strategic Abstractions of Conversational Agents

# ToSCA: Leveraging Hierarchical Reinforcement Learning on Temporal and Strategic Abstractions of Conversational Agents
Source: [https://arxiv.org/html/2608.21969](https://arxiv.org/html/2608.21969)
Qingqing GuYue ZhaoTeng ChenYuqi CaoAffiliation:Geely AI Lab, Beijing Institute of Technology, Peking University,Correspondence:[jiluoaaron@hotmail\.com](mailto:[email protected])Xiaokai ChenHongyan LiLuo Ji

###### Abstract

Humans have multiple levels of temporal abstractions on daily interaction and thinking, such as concept perception and strategic planning\. Inspired by this nature, we propose a two\-level hierarchical reinforcement learning \(RL\) framework for conversational agents, bridging the gap between previous token\-level or utterance\-level RL methods\. Developed on a two\-level MDP, the token\-level response decoding is conditioned on the utterance\-level action, the explicit textual strategies\. Based on theoretical derivation and efficiency consideration, we use DQN to solve the high\-level critic and PPO to solve the low\-level actor\-critic\. To further alleviate the reward sparsity and facilitate the convergence, we also design the dual\-granularity reward mechanism, in which the utterance\-level satisfaction score is integrated with token\-level intrinsic motivation and K\-L penalty\. Experiments on both daily and emotional support conversations show that our method outperforms versatile baselines in strategy determination and response quality\. Our implementation is available at[https://github\.com/AaronJi/ToSCA](https://github.com/AaronJi/ToSCA)\.

## 1Introduction

Large Language Models \(LLMs\) have made remarkable advances in conversational generation and reasoning[30](https://arxiv.org/html/2608.21969#bib.bib2);[1](https://arxiv.org/html/2608.21969#bib.bib1)\. However, when dealing with conversations, LLMs may encounter substantial obstacles: i\)the bottleneck of data supervision, ii\)alignment on the nuance and complicated human preference, as well as iii\)the wide distribution of topics, challenging the adaptation and generalization of traditional finetuning methods \(e\.g\., SFT or LoRA\)[7](https://arxiv.org/html/2608.21969#bib.bib49);[49](https://arxiv.org/html/2608.21969#bib.bib51);[12](https://arxiv.org/html/2608.21969#bib.bib50)\. These issues motivate the adoption of reinforcement learning from human feedback \(RLHF\) for conversation exploration and long\-term return exploitation, such as Proximal Policy Optimization \(PPO\)[30](https://arxiv.org/html/2608.21969#bib.bib2)and Group Relative Policy Optimization \(GRPO\)[36](https://arxiv.org/html/2608.21969#bib.bib52)\.

Despite the success, in practice, most RL approaches operate at the token\-level[15](https://arxiv.org/html/2608.21969#bib.bib45);[55](https://arxiv.org/html/2608.21969#bib.bib44);[4](https://arxiv.org/html/2608.21969#bib.bib43), suffering from reward sparsity and high computational overheads[55](https://arxiv.org/html/2608.21969#bib.bib44)\. These issues are more evident in multi\-turn conversations, in which the meaningful user feedback is not available until the entire assistant utterance is completed[16](https://arxiv.org/html/2608.21969#bib.bib31)\. Consequently, exploration over the vast token\-level action space becomes prohibitively expensive, preventing LLM\-based agents from optimal, anthropomorphic conversational behaviors\.

![Refer to caption](https://arxiv.org/html/2608.21969v1/wxy_p1_v8.png)Figure 1:The paradigm of ToSCA with two\-level MDP\. A high\-level criticQHQ^\{\\text\{H\}\}determines the optimal strategy \(aHa^\{\\text\{H\}\}\), which guides the low\-level actorπL\\pi^\{\\text\{L\}\}to produce the enhanced response \(aLa^\{\\text\{L\}\}\)\. The strategic determination is depicted by aswitch⊥\\botover thecontacts⊸\\multimap\.QHQ^\{\\text\{H\}\}also bootstraps the user’s satisfaction reward, while the low\-level criticQLQ^\{\\text\{L\}\}also considers the intrinsic motivation\. Case and strategies are sampled from DailyDialog\.One possible remedy is inspired by humans’ multi\-level abstractions capabilities[27](https://arxiv.org/html/2608.21969#bib.bib3);[50](https://arxiv.org/html/2608.21969#bib.bib4)\. For example, humans often first decide a strategy, such as commissive, directive, question, and inform, then formulate the detailed response\. Such strategies can be extracted and annotated for human conversation corpus, based on sociological or psychological theories, such as the Helping Skills Theory[8](https://arxiv.org/html/2608.21969#bib.bib12)in ESConv[23](https://arxiv.org/html/2608.21969#bib.bib14), and the annotation criteria ISO 24617\-2[32](https://arxiv.org/html/2608.21969#bib.bib13)in DailyDialog[18](https://arxiv.org/html/2608.21969#bib.bib48)\. This hierarchical nature of human thinking motivates prompting methods[53](https://arxiv.org/html/2608.21969#bib.bib10);[29](https://arxiv.org/html/2608.21969#bib.bib19), supervised learning[39](https://arxiv.org/html/2608.21969#bib.bib8);[22](https://arxiv.org/html/2608.21969#bib.bib7), retrieval\-based framework[47](https://arxiv.org/html/2608.21969#bib.bib9), and simulation\-based studies[54](https://arxiv.org/html/2608.21969#bib.bib6)\. Although recently there are also strategy\-based RL methods such as StraQ\*\([45](https://arxiv.org/html/2608.21969#bib.bib32)\), they lack token\-level action determinations, lacking the alignment of actions on strategy and token\-levels\.

Hierarchical Reinforcement Learning \(HRL\)[41](https://arxiv.org/html/2608.21969#bib.bib38);[25](https://arxiv.org/html/2608.21969#bib.bib34)has addressed analogous challenges in robotics and control through temporal abstraction and action space reduction[37](https://arxiv.org/html/2608.21969#bib.bib53);[40](https://arxiv.org/html/2608.21969#bib.bib54);[46](https://arxiv.org/html/2608.21969#bib.bib55)\. However, due to fundamental differences in state and action representations, it is difficult to transfer traditional HRL methods to human conversations directly\. Although recent studies also apply HRL on dialogue systems[55](https://arxiv.org/html/2608.21969#bib.bib44);[4](https://arxiv.org/html/2608.21969#bib.bib43), to explore the dual\-granularity of utterance\-level and token\-level generations, they may still have deficiencies\. For example, the high\-level policy of ArCher[55](https://arxiv.org/html/2608.21969#bib.bib44)determines the abstract intent in a continuous latent space, lacking explicit textual expressions, resulting in limited interpretability\. In contrast, utterance\-level RL approaches such as DAT[15](https://arxiv.org/html/2608.21969#bib.bib45)directly leverage explicit strategies to train a planner, while neglecting the rich demonstrations on the token level\.

To bridge the gap between ArCher and DAT, in this paper, we propose a novel dual\-level HRL framework calledTemporal\-Strategic Abstractions ofConversationalAgents \(ToSCA\), to guide a token\-level policy with an utterance\-level strategic planner\. ToSCA belongs to a famous HRL paradigm calledoptions[41](https://arxiv.org/html/2608.21969#bib.bib38);[38](https://arxiv.org/html/2608.21969#bib.bib39), with the low\-level action \(response tokens\) conditioned on the high\-level action \(strategies\), as visualized in Figure[1](https://arxiv.org/html/2608.21969#S1.F1)\. By defining a two\-level MDP, we implement the high\-level MDP with a value\-based critic, operating on a discrete action space, solved by DQN; and the low\-level MDP with the actor\-critic framework, solved by PPO\. To alleviate the reward sparsity and enhance the intrinsic consistency, we also design a dual\-granularity reward mechanism, in which the low\-level reward has two token\-level auxiliary components, the K\-L penalty and the intrinsic motivation, besides the utterance\-level user satisfaction score\. We train and evaluate ToSCA on two domains, daily\-life conversation \(DailyDialog\) and emotional support conversation \(ESConv\), as well as an out\-of\-domain test on EmpatheticDialogues\. ToSCA surpasses prompting, supervised, and RL baselines, showing good generalization and reasonable strategic planning\. Major contributions of this paper include:

- •We propose a novel HRL\-based dialogue system, bridging the gap between explicit utterance\-level strategic planning and token\-level response generation\.
- •We introduce intrinsic motivation into conversation generation, formulating a dual\-granularity reward mechanism, to alleviate reward sparsity and facilitate the convergence\.
- •We design substantial experiments on both daily\-life and emotional support conversations to verify the method’s effectiveness, and validate its generalization on out\-of\-domain tests\.

## 2Preliminaries

### 2\.1Markov Decision Process

The Markov decision process \(MDP\) is usually defined as a 5\-tuple\(𝒮,𝒜,ℛ,𝒯,γ\)\(\\mathcal\{S\},\\mathcal\{A\},\\mathcal\{R\},\\mathcal\{T\},\\gamma\), where𝒮\\mathcal\{S\}is the state set,𝒜\\mathcal\{A\}is the action set,ℛ\\mathcal\{R\}is the reward set,γ\\gammais the discounting factor of rewards, and𝒯:𝒮×𝒜→𝒮\\mathcal\{T\}:\\mathcal\{S\}\\times\\mathcal\{A\}\\rightarrow\\mathcal\{S\}is the state transition function\.

At a specific timett, Reinforcement Learning \(RL\) can be employed to provide a policyπ⁡\(a\|s\)\\pi\(a\|s\)by optimizing the following discounted cumulative expected return \(JJ\):

maxπ⁡𝔼s,a∼𝒯,π​∑t=0∞γt​rt:=J⁡\(ℳ\)\\displaystyle\\max\_\{\\pi\}\{\\mathbb\{E\}\_\{s,a\\sim\\mathcal\{T\},\\pi\}\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}r\_\{t\}\}:=J\(\\mathcal\{M\}\)\(1\)s∈𝒮,a∈𝒜,r∈ℛ\\displaystyle s\\in\\mathcal\{S\},a\\in\\mathcal\{A\},r\\in\\mathcal\{R\}

### 2\.2Q\-Learning

Q\-Learning belongs to value\-based RL, which aims to learn the optimal state\-action value functionQ∗​\(s,a\)Q^\{\*\}\(s,a\), such that the determined action approximately optimizesJJby maximizingQQ:

a⋆=\\displaystyle a^\{\\star\}=arg⁡maxa​Q∗​\(s,a\)←arg⁡max⁡J\\displaystyle\\arg\\max\_\{a\}Q^\{\*\}\(s,a\)\\leftarrow\\arg\\max JQ⁡\(s,a\)←\\displaystyle Q\(s,a\)\\leftarrowr⁡\(s,a\)\+γ​maxa′⁡Q⁡\(s′,a′\)\\displaystyle r\(s,a\)\+\\gamma\\max\_\{a^\{\\prime\}\}Q\(s^\{\\prime\},a^\{\\prime\}\)\(2\)in which the second equation is the famous Bellman Equation, the superscript∗\*means the optimal value, and′indicates the next step\.

Instead of explicitly implementing the above equation, Deep Q\-learning \(DQN\)[26](https://arxiv.org/html/2608.21969#bib.bib5)approximates the minimization of the error of Equation[2](https://arxiv.org/html/2608.21969#S2.E2)with deep value networks:

ℒQ​\(ϕ\)=\|r⁡\(s,a\)\+γ​maxa′​Qϕ¯​\(s′,a′\)−Qϕ​\(s,a\)\|2\\displaystyle\\mathcal\{L\}\_\{Q\}\(\\phi\)=\|r\(s,a\)\+\\gamma\\max\_\{a^\{\\prime\}\}Q\_\{\\bar\{\\phi\}\}\(s^\{\\prime\},a^\{\\prime\}\)\-Q\_\{\\phi\}\(s,a\)\|^\{2\}\(3\)whereϕ\\phiandϕ¯\\bar\{\\phi\}are parameters of the Q and target Q\-net, respectively\.ϕ¯\\bar\{\\phi\}is usually periodically synchronized fromϕ\\phi\.

### 2\.3Proximal Policy Optimization

Proximal policy optimization \(PPO\)\([35](https://arxiv.org/html/2608.21969#bib.bib33)\)follows the Actor\-Critic framework and is widely applied to LLMs\([30](https://arxiv.org/html/2608.21969#bib.bib2)\)\. Initialized from an LLM backbone, the*actor*behaves as a generative policy network, while a separate value network serves as the*critic*\. The actor is trained by maximizing the objective

ℒπ​\(θ\)=\\displaystyle\\mathcal\{L\}\_\{\\pi\}\(\\theta\)=𝔼\(s,a\)∼𝒯,πθold​\[πθ​\(a\|s\)πθold​\(a\|s\)​A​\(s,a\)\]\\displaystyle\\;\\mathbb\{E\}\_\{\(s,a\)\\sim\\mathcal\{T\},\\,\\pi\_\{\\theta\_\{\\text\{old\}\}\}\}\\Big\[\\frac\{\\pi\_\{\\theta\}\(a\|s\)\}\{\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(a\|s\)\}\\,A\(s,a\)\\Big\]−λ1𝔼s∼𝒯\[DKL\(πθ\(⋅\|s\)∥πSFT\(⋅\|s\)\)\]\\displaystyle\\;\-\\lambda\_\{1\}\\,\\mathbb\{E\}\_\{s\\sim\\mathcal\{T\}\}\\Big\[D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\pi\_\{\\theta\}\(\\cdot\|s\)\\,\\\|\\,\\pi^\{\\text\{SFT\}\}\(\\cdot\|s\)\\right\)\\Big\]\+λ2​𝔼\(s,a\)∼𝒟pretrain​\[log⁡πθ​\(a\|s\)\]\\displaystyle\\;\+\\lambda\_\{2\}\\,\\mathbb\{E\}\_\{\(s,a\)\\sim\\mathcal\{D\}\_\{\\text\{pretrain\}\}\}\\Big\[\\log\\pi\_\{\\theta\}\(a\|s\)\\Big\]\(4\)whereθ\\thetaandθold\\theta\_\{\\text\{old\}\}denote the parameters of the current policy and the behavior policy, respectively, andA⁡\(s,a\)A\(s,a\)is the advantage function estimated by the critic\.λ1\\lambda\_\{1\}controls the strength of the KL regularization that constrains the policy to remain close to the SFT model, whileλ2\\lambda\_\{2\}balances the auxiliary pretraining objective\. The critic is trained to regress the state\-value function using

ℒQ​\(ψ\)=𝔼⁡\[\(Qψ​\(s\)−A⁡\(s,a\)−Qψ¯​\(s\)\)2\]\\displaystyle\\mathcal\{L\}\_\{\\text\{Q\}\}\(\\psi\)=\\mathbb\{E\}\\Big\[\\big\(Q\_\{\\psi\}\(s\)\-A\(s,a\)\-Q\_\{\\bar\{\\psi\}\}\(s\)\\big\)^\{2\}\\Big\]\(5\)whereψ\\psiandψ¯\\bar\{\\psi\}are the parameters of the current and previous value networks, respectively\.

## 3Method

### 3\.1Task Formulation

#### Multi\-turn conversation\.

ATT\-turn user\-agent conversation can be represented by an interleaved sequence ofq​u​e​r​y​\(t\)query\(t\)andr​e​s​p​o​n​s​e​\(t\)response\(t\),1≤t≤T1\\leq t\\leq T\. This paper studies strategy\-based conversations[23](https://arxiv.org/html/2608.21969#bib.bib14);[34](https://arxiv.org/html/2608.21969#bib.bib15), with each turn annotated with the user’s emotion \(e​m​oemo\) and the agent’s strategy \(s​t​r​astra\)\. Such an augmented conversation is then described by\{d​e​s​c,h​i​s​t​\(T\),q​u​e​r​y​\(T\),e​m​o​\(T\),s​t​r​a​\(T\),r​e​s​p​o​n​s​e​\(T\)\}\\\{desc,\\allowbreak hist\(T\),\\allowbreak query\(T\),\\allowbreak emo\(T\),\\allowbreak stra\(T\),\\allowbreak response\(T\)\\\}, in whichd​e​s​cdescis the session\-level description, andhist\(T\):=\{query\(t\),response\(t\)\}0:T−1hist\(T\):=\\\{query\(t\),\\allowbreak response\(t\)\\\}\_\{0:T\-1\}is the history context attt\.

#### Hierarchical MDP\.

Similar to ArCHer[55](https://arxiv.org/html/2608.21969#bib.bib44), we define the task of multi\-turn conversations as a hierarchical MDP, conducting the temporal abstraction:

1. 1\.The high\-level MDPℳH\\mathcal\{M\}^\{\\text\{H\}\}: the agent makes conceptual decisions at the utterance level, guiding the low\-level action\.
2. 2\.The low\-level MDPℳL\\mathcal\{M\}^\{\\text\{L\}\}: Similar to conventional RLHF, this MDP operates at the token level, where the agent generates the response token by token, until completion\.

where we use the superscripts H and L to represent the high and low levels\. Different from[55](https://arxiv.org/html/2608.21969#bib.bib44), our high\-level action performs the strategic abstraction, which can be learned and verified by strategy\-annotated datasets\.

### 3\.2System Configuration

#### Variable definitions\.

According to the environment, we define its system variables as Table[1](https://arxiv.org/html/2608.21969#S3.T1), wherer​e​s​p​o​n​s​eiresponse\_\{i\}is the current generated part ofr​e​s​p​o​n​s​eresponse\. Based the above definitions,ℳH\\mathcal\{M\}^\{\\text\{H\}\}andℳL\\mathcal\{M\}^\{\\text\{L\}\}can be depicted with tuples of\(𝒮H,𝒜H,ℛH\)\(\\mathcal\{S\}^\{\\text\{H\}\},\\mathcal\{A\}^\{\\text\{H\}\},\\mathcal\{R\}^\{\\text\{H\}\}\)and\(𝒮L,𝒜L,ℛL\)\(\\mathcal\{S\}^\{\\text\{L\}\},\\mathcal\{A\}^\{\\text\{L\}\},\\mathcal\{R\}^\{\\text\{L\}\}\), respectively\.

LevelVariableDefinitionHighsH∈𝒮Hs^\{\\text\{H\}\}\\in\\mathcal\{S\}^\{\\text\{H\}\}State\(q​u​e​r​y,h​i​s​t​o​r​y,e​m​o\)\(query,\\;history,\\;emo\)aH∈𝒜Ha^\{\\text\{H\}\}\\in\\mathcal\{A\}^\{\\text\{H\}\}Strategy selection\(s​t​r​a\)\(stra\)rH∈ℛHr^\{\\text\{H\}\}\\in\\mathcal\{R\}^\{\\text\{H\}\}Satisfaction score\(rsat\)\(r^\{\\text\{sat\}\}\)LowsL∈𝒮Ls^\{\\text\{L\}\}\\in\\mathcal\{S\}^\{\\text\{L\}\}State\(sH,r​e​s​p​o​n​s​ei\)\(s^\{\\text\{H\}\},\\;response\_\{i\}\)aL∈𝒜La^\{\\text\{L\}\}\\in\\mathcal\{A\}^\{\\text\{L\}\}Current decoded tokenrL∈ℛLr^\{\\text\{L\}\}\\in\\mathcal\{R\}^\{\\text\{L\}\}Token\-level reward

Table 1:Variable definitions in the HRL framework\.
#### RL components and objectives\.

In this work, we employ HRL to jointly optimize the targets ofℳH\\mathcal\{M\}^\{\\text\{H\}\}andℳL\\mathcal\{M\}^\{\\text\{L\}\}\. Different from previous configurations such as ActorH\{\}^\{\\text\{H\}\}\-CriticL\{\}^\{\\text\{L\}\}[55](https://arxiv.org/html/2608.21969#bib.bib44), ActorH\{\}^\{\\text\{H\}\}\-CriticH\{\}^\{\\text\{H\}\}\-ActorL\{\}^\{\\text\{L\}\}[2](https://arxiv.org/html/2608.21969#bib.bib40)or double Actor\-Critic[51](https://arxiv.org/html/2608.21969#bib.bib42), the high\-level MDP employs a single critic \(QHQ^\{\\text\{H\}\}\) to provide a strategic bootstrapping on discrete options, while the low\-level MDP inherits the actor\-critic framework \(πL\\pi^\{\\text\{L\}\},QLQ^\{\\text\{L\}\}\), taking advantage of robustness of token\-level PPO\. Therefore, our architecture can be named as*CriticH\{\}^\{\\text\{H\}\}\-ActorL\{\}^\{\\text\{L\}\}\-CriticL\{\}^\{\\text\{L\}\}*\. All these components are initialized from the same pretrained LLM\. ToSCA can be considered as the following objective:

πH←max⁡J⁡\(ℳH\);πL,QL←max⁡J⁡\(ℳL\)\\displaystyle\\pi^\{\\text\{H\}\}\\leftarrow\\max J\(\\mathcal\{M\}^\{\\text\{H\}\}\);\\pi^\{\\text\{L\}\},Q^\{\\text\{L\}\}\\leftarrow\\max J\(\\mathcal\{M\}^\{\\text\{L\}\}\)\(6\)where definitions inherit from Section[2](https://arxiv.org/html/2608.21969#S2)\.

Figure 2:Illustration of the dual\-scale pipeline of ToSCA\. The high\-level MDP operates on the utterance scale \(tt\), receives the user’s satisfaction reward \(rtH=rts​a​tr^\{H\}\_\{t\}=r^\{sat\}\_\{t\}\), and determines thes​t​r​a​t​e​g​ystrategy\(atHa^\{H\}\_\{t\}\) which conditions the low\-level action \(akLa^\{L\}\_\{k\}\)\. The low\-level MDP operates on the token scale \(kk\), learned by the actor\-critic framework\. The low\-level reward \(rtLr^\{L\}\_\{t\}\) consists of the satisfaction reward \(rts​a​tr^\{sat\}\_\{t\}\), KL penalty \(rkK​Lr^\{KL\}\_\{k\}\) and intrinsic motivation \(rki​mr^\{im\}\_\{k\}\)\.QHQ^\{H\}andQLQ^\{L\}are optimized by the critic lossℒ𝒬\\mathcal\{L\_\{Q\}\}, whileπH\\pi^\{H\}is optimized byℒπ\\mathcal\{L\_\{\\pi\}\}\.

### 3\.3The High\-Level MDP

#### Training\.

Instead of augmenting the value head \(as is done in conventional RLHF\), we implement this strategic value function in a purely generative manner\. Assuming there is an instruction template with the placeholder ofss, denoted byℐ⁡\(s\)\\mathcal\{I\}\(s\)\. Then the state\-action value can be obtained from the averaged logits of the action tokens, from the conditional forward pass of LLM:

QϕH​\(s,a\)\\displaystyle Q\_\{\\phi\}^\{\\text\{H\}\}\(s,a\)←LLMϕ​\(ℐH​\(s\)⊕a\)\\displaystyle\\leftarrow\\text\{LLM\}\_\{\\phi\}\(\\mathcal\{I\}^\{\\text\{H\}\}\(s\)\\oplus a\)\(7\)where⊕\\oplusrepresents the textual concatenation\. We conduct DQN \(Equation[3](https://arxiv.org/html/2608.21969#S2.E3)\) to trainQϕHQ\_\{\\phi\}^\{\\text\{H\}\}\.

#### Prompt\.

We briefly exhibit our instructionℐ⁡\(s\)\\mathcal\{I\}\(s\)here:

The high\-level promptℐH\\mathcal\{I\}^\{\\text\{H\}\}Description:\{d​e​s​cdesc\}User’s emotion:\{e​m​oemo\} History:\{h​i​s​thist\}Query:\{q​u​e​r​yquery\} Please select the best strategy:\(1\)\{s​t​r​astra1\}\(2\)\{s​t​r​astra2\}⋯\\cdots\(K\)\{s​t​r​astraK\}

with a full version in Appendix[A](https://arxiv.org/html/2608.21969#A1)\. To further strengthen the understanding capability of LLM on the strategy selection, we formulateℐ\\mathcal\{I\}as a multiple\-choice question \(MCQ\), instead of a plain question, forcing the LLM to choose one of the options\. Accordingly, the action set becomes the set of possible strategy indicesa∈𝒜:=\{1,2,⋯,K\}a\\in\\mathcal\{A\}:=\\\{1,2,\\cdots,K\\\}whereKKis the total number of strategies\.

#### Inference\.

Instead of decoding the next token, the finetuned LLM produces logits of available strategies, and the optimal strategy can be determined from the maximum logit

aH←arg⁡max⁡LLMϕ​\(ℐH​\(s\)⊕a\),a∈𝒜a^\{\\text\{H\}\}\\leftarrow\\arg\\max\\text\{LLM\}\_\{\\phi\}\(\\mathcal\{I\}^\{\\text\{H\}\}\(s\)\\oplus a\),a\\in\\mathcal\{A\}\(8\)

### 3\.4The Low\-Level MDP

#### Training\.

Another LLM backbone is employed as the low\-level policyπL\\pi^\{\\text\{L\}\}, with the trainable parameterθ\\theta\. Similar to RLHF[30](https://arxiv.org/html/2608.21969#bib.bib2), a copy of LLM backbone is used as the low\-level criticQLQ^\{\\text\{L\}\}, with the value head augmented\. We use token\-level PPO to trainπL\\pi^\{\\text\{L\}\}andQLQ^\{\\text\{L\}\}jointly\.

#### Inference and prompt\.

The high\-level action guides the low\-level action by in\-context learning \(ICL\):

aL←LLMθ​\(ℐL​\(sL,aH\)\):=πaHL​\(sL\)\\displaystyle a^\{\\text\{L\}\}\\leftarrow\\text\{LLM\}\_\{\\theta\}\(\\mathcal\{I\}^\{\\text\{L\}\}\(s^\{\\text\{L\}\},a^\{\\text\{H\}\}\)\):=\\pi^\{\\text\{L\}\}\_\{a^\{\\text\{H\}\}\}\(s^\{\\text\{L\}\}\)\(9\)
The low\-level promptℐL\\mathcal\{I\}^\{\\text\{L\}\}Description:\{d​e​s​cdesc\}User’s emotion:\{e​m​oemo\} History:\{h​i​s​t​o​r​yhistory\}Query:\{q​u​e​r​yquery\} Your strategy:\{s​t​r​astra\}Please response now:

The prompt has the above template with detailed content in Appendix[A](https://arxiv.org/html/2608.21969#A1)\.

### 3\.5The Reward Mechanisms

To provide multi\-timescale feedback, we define our high\-level reward \(rHr^\{\\text\{H\}\}\) as the satisfaction score \(rsatr^\{\\text\{sat\}\}\) at the real timett, solely coming from the environment; while the low\-level reward \(rLr^\{\\text\{L\}\}\) has another two token\-level \(kk\) components

rtH=rtsat,rkL=rtsat\+β1​rkKL\+β2​rkim\\displaystyle r^\{\\text\{H\}\}\_\{t\}=r^\{\\text\{sat\}\}\_\{t\},\\quad r^\{\\text\{L\}\}\_\{k\}=r^\{\\text\{sat\}\}\_\{t\}\+\\beta\_\{1\}r^\{\\text\{KL\}\}\_\{k\}\+\\beta\_\{2\}r^\{\\text\{im\}\}\_\{k\}\(10\)with definitions of reward components below\.

#### The satisfaction score \(rsatr^\{\\text\{sat\}\}\)\.

Choice of rewards may be crucial, especially when an offline dataset constrains the sampling\. In this paper, we leverage the LLM\-as\-the\-Judge approach to automatically generate online scores\. A strong\-basis LLM, such as GPT\-4o, is employed as the oracle \(or the teacher\) to produce the satisfaction scorersatr^\{\\text\{sat\}\}from 0 to 5\. In Appendix[A](https://arxiv.org/html/2608.21969#A1), we provided the detailed evaluation prompt; in Appendix[F](https://arxiv.org/html/2608.21969#A6), we conduct statistical experiments to illustrate the consistency between oracle and human scores\.

#### The KL reward \(rKLr^\{\\text\{KL\}\}\)\.

The penalty of Kullback–Leibler \(K\-L\) divergence preventsπL\\pi^\{\\text\{L\}\}from deviating the ref policy too much, which has the form ofrKL=−DKL\(πL∥πref\)r\_\{\\text\{KL\}\}=\-D\_\{\\mathrm\{KL\}\}\(\\pi^\{\\text\{L\}\}\\\|\\pi^\{\\text\{ref\}\}\)\.

#### Intrinsic motivation \(rimr^\{\\text\{im\}\}\)\.

Here we define the reward of intrinsic motivation as the inner\-agent alignment between the low\-level context \(state plus strategy\) and response:

rim=log⁡LLMθ​\(sL⊕aH⊕aL\)\\displaystyle r^\{\\text\{im\}\}=\\log\{\\text\{LLM\}\_\{\\theta\}\(s^\{\\text\{L\}\}\\oplus a^\{\\text\{H\}\}\\oplus a^\{\\text\{L\}\}\)\}\(11\)where the right\-hand side can also be observed as the cross\-entropy loss ofLLMwith all textual inputs, similar to the pretrained loss in[30](https://arxiv.org/html/2608.21969#bib.bib2)\. By augmentingrimr^\{\\text\{im\}\}, we encourage the low\-level policy to be more proficient with the strategy\-steering response style, in contrast with the policy update driven solely by human feedback\.

### 3\.6Algorithm

Algorithm[1](https://arxiv.org/html/2608.21969#alg1)outlines our overall framework\. For each session, we sample an initial query from the dataset and perform the rollout\. At each turn, the high\-level critic first determines a strategy, then the low\-level policy generates the response, conditioned on both state and strategy\. During training, the high\-level critic is learned by DQN, while the low\-level actor and critic are trained by PPO\. Figure[2](https://arxiv.org/html/2608.21969#S3.F2)illustrates the entire pipeline\. Further implementation details are provided in Appendix[A](https://arxiv.org/html/2608.21969#A1)\.

Algorithm 1HRL\-ToSCAInput:QHQ^\{\\text\{H\}\},QLQ^\{\\text\{L\}\},πL\\pi^\{\\text\{L\}\}from pretrained checkpoints; batch sizesBHB^\{\\text\{H\}\}andBLB^\{\\text\{L\}\}

1:Initialize buffer

ℬH,ℬL=\{\}\\mathcal\{B\}^\{\\text\{H\}\},\\mathcal\{B\}^\{\\text\{L\}\}=\\\{\\\}\.

2:whilenot convergesdo

3:Initialize

sH,sLs^\{\\text\{H\}\},s^\{\\text\{L\}\}with queries from dataset

4:\\\\Roll\-out

5:fort =

1,…,T1,\\dots,Tdo

6:Inference

atHa^\{\\text\{H\}\}\_\{t\}by

arg⁡max\\arg\\max\(Eq[8](https://arxiv.org/html/2608.21969#S3.E8)\)

7:fork =

1,…,K1,\\dots,Kdo

8:Inference

atL∼πL​\(stL\)a^\{\\text\{L\}\}\_\{t\}\\sim\\pi^\{\\text\{L\}\}\(s^\{\\text\{L\}\}\_\{t\}\)\(Eq[9](https://arxiv.org/html/2608.21969#S3.E9)\)

9:Increment

stLs^\{\\text\{L\}\}\_\{t\}to

st\+1Ls^\{\\text\{L\}\}\_\{t\+1\}
10:break if

atL=<E​O​S\>a^\{\\text\{L\}\}\_\{t\}=<EOS\>
11:endfor

12:collect

rtsatr\_\{t\}^\{\\text\{sat\}\}from the oracle LLM

13:Append buffer

ℬL∪\{\(skL,akL,rtsat\)\}k=1K\\mathcal\{B\}^\{\\text\{L\}\}\\cup\\\{\(s^\{\\text\{L\}\}\_\{k\},a^\{\\text\{L\}\}\_\{k\},r\_\{t\}^\{\\text\{sat\}\}\)\\\}\_\{k=1\}^\{K\}
14:Append buffer

ℬH∪\(stH,atH,rtsat\)\\mathcal\{B\}^\{\\text\{H\}\}\\cup\(s^\{\\text\{H\}\}\_\{t\},a^\{\\text\{H\}\}\_\{t\},r\_\{t\}^\{\\text\{sat\}\}\)
15:endfor

16:\\\\High\-level Training

17:Sample

\(stH,atH,rtsat,st\+1H,at\+1H\)\(s^\{\\text\{H\}\}\_\{t\},a^\{\\text\{H\}\}\_\{t\},r\_\{t\}^\{\\text\{sat\}\},s^\{\\text\{H\}\}\_\{t\+1\},a^\{\\text\{H\}\}\_\{t\+1\}\)from

ℬH\\mathcal\{B\}^\{\\text\{H\}\}
18:update

QHQ^\{\\text\{H\}\}by DQN \(Eq[3](https://arxiv.org/html/2608.21969#S2.E3)\)

19:\\\\Low\-level Training

20:fork =

1,…,K−11,\\dots,K\-1do

21:Get

\(skL,akL,rtsat,sk\+1L,ak\+1L\)\(s^\{\\text\{L\}\}\_\{k\},a^\{\\text\{L\}\}\_\{k\},r\_\{t\}^\{\\text\{sat\}\},s^\{\\text\{L\}\}\_\{k\+1\},a^\{\\text\{L\}\}\_\{k\+1\}\)from

ℬL\\mathcal\{B\}^\{\\text\{L\}\}
22:Calculate

rkKLr\_\{k\}^\{\\text\{KL\}\},

rkimr\_\{k\}^\{\\text\{im\}\},

rkLr\_\{k\}^\{\\text\{L\}\}by Eq[10](https://arxiv.org/html/2608.21969#S3.E10)

23:Update

πL\\pi^\{\\text\{L\}\}and

QLQ^\{\\text\{L\}\}jointly by PPO \(Eq[4](https://arxiv.org/html/2608.21969#S2.E4)\)

24:endfor

25:endwhile

### 3\.7Theoretical Derivation

We start from two assumptions proposed by theoptionsframework[41](https://arxiv.org/html/2608.21969#bib.bib38), which are the foundation of the subsequent main theorem\.

###### Assumption 1\.

The strategies, as special types of options, are everywhere:

∀s∈𝒮,∀aH∈𝒜H:∃s∈ℐ⁡\(s\|aH\)\\forall s\\in\\mathcal\{S\},\\forall a^\{\\text\{H\}\}\\in\\mathcal\{A^\{\\text\{H\}\}\}:\\exists s\\in\\mathcal\{I\}\(s\|a^\{\\text\{H\}\}\)\(12\)whereℐ\\mathcal\{I\}is the initial state set\.

###### Assumption 2\.

The strategies are Markovian:

P\(stH,atH\|s0:t−1H,a0:t−1H\)=P\(stH,atH\|st−1H,at−1H\)\\text\{P\}\(s\_\{t\}^\{\\text\{H\}\},a\_\{t\}^\{\\text\{H\}\}\|s\_\{0:t\-1\}^\{\\text\{H\}\},a\_\{0:t\-1\}^\{\\text\{H\}\}\)=\\text\{P\}\(s\_\{t\}^\{\\text\{H\}\},a\_\{t\}^\{\\text\{H\}\}\|s\_\{t\-1\}^\{\\text\{H\}\},a\_\{t\-1\}^\{\\text\{H\}\}\)\(13\)

Considering the features of the dialogue system, it is obvious to see both assumptions are reasonable\. The agent can choose any strategy or any response, no matter what the current state is\. The prior history and query strategy are also prerequisites for determining the next strategy\. Within a hierarchical framework, we propose the main theorem:

###### Theorem 1\.

With Assumptions[1](https://arxiv.org/html/2608.21969#Thmassumption1)and[2](https://arxiv.org/html/2608.21969#Thmassumption2)hold, the value function \(Eq[7](https://arxiv.org/html/2608.21969#S3.E7)\) converges as long as the high\-level policy \(Eq[8](https://arxiv.org/html/2608.21969#S3.E8)\) is deterministic, and the high\- and low\-level critics are solved interleaved\.

Due to page limits, we leave detailed derivations in Appendix[E](https://arxiv.org/html/2608.21969#A5)\. Theorem[1](https://arxiv.org/html/2608.21969#Thmtheorem1)suggests that our determined strategy can converge to the ground truth\.

## 4Experiment

In this section, we mainly address two research questions \(RQ\):RQ1, whether our framework can adapt to different conversation domains;RQ2, whether our method can generalize well when strategy annotation is not available\. ForRQ1, we investigate two domains in Section[4\.5](https://arxiv.org/html/2608.21969#S4.SS5)and[4\.6](https://arxiv.org/html/2608.21969#S4.SS6), respectively; forRQ2, we differentiate the tests into in\-domain and out\-of\-domain tests, depending on the availability of strategy annotations\.

### 4\.1Implementation

We use LLaMA3\.2\-1B\-Instruct and LLaMA3\.1\-8B\-Instruct[1](https://arxiv.org/html/2608.21969#bib.bib1)as high\-level and low\-level backbones, respectively\. The framework is trained on OpenRLHF[9](https://arxiv.org/html/2608.21969#bib.bib30)using 16 A100 GPUs\. We employ GPT\-4o for both user simulation and satisfaction scoring\. We use learning rates of 5e\-6, 9e\-7, and 9e\-4 for the training ofQHQ^\{\\text\{H\}\},πL\\pi^\{\\text\{L\}\}, andQLQ^\{\\text\{L\}\}, respectively\. The batch size is 64, and the window length is set to 128\.

### 4\.2Dataset

#### Daily\-life conversations\.

DailyDialog[18](https://arxiv.org/html/2608.21969#bib.bib48)is a manually constructed multi\-turn dialogue dataset with approximately 13,000 conversations on everyday topics\. Each utterance is annotated with a dialogue strategy111Denoted as ‘Act’ in the original dataset\., includingQuestion, Inform, Directive,andCommissive\. During training, only the strategy set is provided, and the model infers strategies without access to gold labels\. The annotations are reserved exclusively for the evaluation of strategy classification and response quality\.

#### Emotional support conversations \(ESC\)\.

Classical datasets in this field include ESConv[23](https://arxiv.org/html/2608.21969#bib.bib14), annotating with88strategies; and EmpatheticDialogues[34](https://arxiv.org/html/2608.21969#bib.bib15), which hasnostrategy annotation\. As a result, we employ ESConv as the in\-domain \(ID\) test while EmpatheticDialogues as the out\-of\-domain \(OOD\) test\. Appendix[B](https://arxiv.org/html/2608.21969#A2)provides more detailed introductions and statistics of these datasets\.

### 4\.3Evaluation

#### Classification metrics\.

We employ classification metrics of accuracy \(Acc\) and Macro\-F1 \(MaF1\)\. We also refer to the evaluation methods proposed by[10](https://arxiv.org/html/2608.21969#bib.bib16), which propose theb​i​a​sbiasof strategies based on the Bradley\-Terry model\([3](https://arxiv.org/html/2608.21969#bib.bib24)\)\. A smallerb​i​a​sbiasindicates better\.

#### Automatic metrics on response\.

For generative tasks, we utilize similarity\-based metrics like Bleu\-2 \(B\-2\) and Rouge\-L \(R\-L\); as well as Distinct\-2 \(D\-2\), which indicates response diversity\.

#### Human scoring\.

Similar with[10](https://arxiv.org/html/2608.21969#bib.bib16), we annotate with dimensions ofAcceptance,Effectiveness,Sensitivity,Fluency, andEmotion, and the ultimate purpose, seeker’sSatisfaction\. Detailed scoring settings and principles are in Appendix[C](https://arxiv.org/html/2608.21969#A3)\.

### 4\.4Baselines

We consider several types of baselines, including \(1\) Direct inference; \(2\) Prompting methods such as Self\-Refine[24](https://arxiv.org/html/2608.21969#bib.bib18), ECoT[20](https://arxiv.org/html/2608.21969#bib.bib17), SoT[29](https://arxiv.org/html/2608.21969#bib.bib19), ToT[48](https://arxiv.org/html/2608.21969#bib.bib20), Plan\-and\-Solve \(PS\)[43](https://arxiv.org/html/2608.21969#bib.bib21), Finite State Machine \(FSM\)[44](https://arxiv.org/html/2608.21969#bib.bib11); \(3\) finetuning methods such as SFT, and the supervised versions of ECoT and FSM \(EmoFSM\)[53](https://arxiv.org/html/2608.21969#bib.bib10); and 4\) RL methods such as DQN and PPO, and two HRL baselines: DAT[16](https://arxiv.org/html/2608.21969#bib.bib31)and ArCher[55](https://arxiv.org/html/2608.21969#bib.bib44)\. Detailed introductions and implementation details of baselines can be found in Appendix[D](https://arxiv.org/html/2608.21969#A4)\.

### 4\.5Results on Daily\-Life Conversations

#### Reward and Value\.

Loss convergence and reward&value increase are observed in our training\. Table[2](https://arxiv.org/html/2608.21969#S4.T2)shows the averaged rewards and values of ToSCA on DailyDialog, compared to Direct and Raw dataset\. Loss curves are in Appendix[F](https://arxiv.org/html/2608.21969#A6)\.

Method<QH\><Q^\{\\text\{H\}\}\><rL\><r^\{\\text\{L\}\}\>Direct486\.23\.21Raw Dataset551\.43\.53ToSCA\(ours\)616\.54\.03Table 2:Averaged values and rewards on DailyDialog\.
#### Strategy determination\.

In Table[3](https://arxiv.org/html/2608.21969#S4.T3), although ToSCA does not receive reward signals from the ground\-truth strategy labels, it still achieves the Acc, benefiting from the alignment between human\-annotated data and the automatic evaluation of correct strategies\. While ToSCA does not have the bestb​i​a​sbiasscore, it performs competitively, which collectively leads to the best MaF1 of 58\.91\.

#### Response quality\.

Table[3](https://arxiv.org/html/2608.21969#S4.T3)also illustrates that ToSCA produces responses with the highest similarity to ground\-truth \(as measured by B\-2 and R\-L\), while simultaneously achieving the best diversity \(D\-2\), surpassing prompt, finetuning, or RL\-based baselines, as well as specific structured models on DailyDialog\. More detailed results \(including full lists of baselines\) are in Appendix[F](https://arxiv.org/html/2608.21969#A6)\.

MethodsAcc↑\\uparrowMaF1↑\\uparrowb​i​a​sbias↓\\downarrowB\-2↑\\uparrowR\-L↑\\uparrowD\-2↑\\uparrowDirect52\.6018\.031\.663\.3510\.3344\.74\+ Direct\-Refine48\.2728\.280\.702\.568\.7043\.67\+ Self\-Refine[24](https://arxiv.org/html/2608.21969#bib.bib18)49\.7622\.151\.182\.407\.7534\.01\+ ECoT[20](https://arxiv.org/html/2608.21969#bib.bib17)38\.9429\.990\.271\.786\.0055\.26\+ SoT[29](https://arxiv.org/html/2608.21969#bib.bib19)N/AN/AN/A2\.537\.9759\.98\+ ToT[48](https://arxiv.org/html/2608.21969#bib.bib20)N/AN/AN/A2\.528\.8443\.19\+ PS[43](https://arxiv.org/html/2608.21969#bib.bib21)N/AN/AN/A2\.607\.7639\.73\+ FSM[44](https://arxiv.org/html/2608.21969#bib.bib11)46\.8621\.221\.302\.709\.4438\.75SFT60\.1944\.820\.826\.8118\.5243\.36\+ECoT60\.1144\.90\.666\.6118\.0742\.87\+FSM[53](https://arxiv.org/html/2608.21969#bib.bib10)60\.0346\.020\.555\.8521\.7747\.43DQN54\.0150\.100\.624\.1813\.0959\.27DAT[16](https://arxiv.org/html/2608.21969#bib.bib31)N/AN/AN/A3\.4511\.800\.90ArCher[55](https://arxiv.org/html/2608.21969#bib.bib44)50\.4142\.670\.215\.1714\.3555\.16ToSCA\(ours\)63\.6458\.910\.6316\.3535\.2262\.67

Table 3:ID results of automatic metrics including Acc, MaF1,b​i​a​sbias, B\-2, R\-L and D\-2 on DailyDialog\. The best result of the methods isbolded\.

### 4\.6Results on ESC

#### In\-domain results\.

In Table[4](https://arxiv.org/html/2608.21969#S4.T4), as the scenario shifts from daily dialogues to emotional support conversations, ToSCA achieves the best or second\-best performance across all metrics, demonstrating its ability to employ appropriate empathetic strategies and achieve better emotional alignment\. We further conduct human evaluation in Appendix[F\.3](https://arxiv.org/html/2608.21969#A6.SS3); as well as comparisons to previous dataset\-specific studies[17](https://arxiv.org/html/2608.21969#bib.bib46);[6](https://arxiv.org/html/2608.21969#bib.bib23);[10](https://arxiv.org/html/2608.21969#bib.bib16)in Appendix[F\.4](https://arxiv.org/html/2608.21969#A6.SS4)\. Typical cases are in Appendix[F\.7](https://arxiv.org/html/2608.21969#A6.SS7)\.

#### Transfer to OOD\.

In this setting, we omit the finetuning baselines to mimic the knowledge transfer scenario\. Table[5](https://arxiv.org/html/2608.21969#S4.T5)demonstrates the generalization ability of ToSCA on EmpatheticDialogues, with higher performance than the baselines on this zero\-shot test\. This success benefits from the inclusion of empathy elements in the reward signal derived from daily dialogues\.

MethodsAcc↑\\uparrowMaF1↑\\uparrowb​i​a​sbias↓\\downarrowB\-2↑\\uparrowR\-L↑\\uparrowD\-2↑\\uparrowDirect11\.8010\.261\.613\.4710\.6433\.45\+ Direct\-Refine17\.0811\.071\.273\.106\.1314\.22\+ Self\-Refine[24](https://arxiv.org/html/2608.21969#bib.bib18)17\.5813\.611\.923\.349\.7114\.61\+ ECoT[20](https://arxiv.org/html/2608.21969#bib.bib17)15\.3210\.381\.693\.1610\.5030\.38\+ SoT[29](https://arxiv.org/html/2608.21969#bib.bib19)N/AN/AN/A3\.078\.7626\.15\+ ToT[48](https://arxiv.org/html/2608.21969#bib.bib20)N/AN/AN/A2\.659\.8114\.2\+ PS[43](https://arxiv.org/html/2608.21969#bib.bib21)N/AN/AN/A2\.818\.2719\.12\+ FSM[44](https://arxiv.org/html/2608.21969#bib.bib11)17\.3711\.150\.814\.1211\.8335\.43SFT32\.4321\.291\.286\.9716\.5950\.45\+ECoT30\.8017\.701\.356\.5115\.0034\.96\+FSM[53](https://arxiv.org/html/2608.21969#bib.bib10)28\.0023\.700\.415\.8815\.3051\.48DQN37\.6934\.570\.593\.5911\.7444\.14DAT[16](https://arxiv.org/html/2608.21969#bib.bib31)18\.508\.361\.743\.2611\.2435\.82ArCher[55](https://arxiv.org/html/2608.21969#bib.bib44)24\.5019\.600\.505\.3013\.1054\.80ToSCA\(ours\)39\.2636\.850\.486\.9316\.2852\.42

Table 4:ID results of automatic metrics including Acc, MaF1,b​i​a​sbias, B\-2, R\-L and D\-2 on ESConv\. The best result isboldedand the second best isunderlined\.MethodsB\-2↑\\uparrowR\-L↑\\uparrowD\-2↑\\uparrowDirect3\.099\.9125\.23\+ ECoT[20](https://arxiv.org/html/2608.21969#bib.bib17)2\.919\.7932\.65\+ SoT[29](https://arxiv.org/html/2608.21969#bib.bib19)1\.795\.6648\.59\+ ToT[48](https://arxiv.org/html/2608.21969#bib.bib20)2\.319\.0529\.09\+ PS[43](https://arxiv.org/html/2608.21969#bib.bib21)2\.696\.9324\.02\+ FSM[44](https://arxiv.org/html/2608.21969#bib.bib11)3\.3310\.8033\.37DQN3\.619\.1539\.60DAT[16](https://arxiv.org/html/2608.21969#bib.bib31)3\.3810\.9035\.05ArCHer[55](https://arxiv.org/html/2608.21969#bib.bib44)3\.8410\.7545\.19ToSCA\(ours\)4\.4912\.9346\.53

Table 5:OOD results of B\-2, R\-L and D\-2 on EmpatheticDialogues\. The best result of the methods isboldedand the second best isunderlined\.MethodAcc↑\\uparrowMaF1↑\\uparrowb​i​a​sbias↓\\downarrowB\-2↑\\uparrowR\-L↑\\uparrowD\-2↑\\uparroww/o high\-levelN/AN/AN/A7\.8525\.1650\.59w/o low\-level53\.3949\.020\.534\.1713\.1559\.44w/o im60\.0453\.010\.6816\.2135\.0660\.95ToSCA\(ours\)63\.6458\.910\.6316\.3535\.2262\.67

Table 6:Ablation study of ToSCA on DailyDialog\.
#### Ablation Study\.

Here we conduct three ablation studies: \(1\)w/o high\-level, which degenerates to token\-level PPO by removing the high\-level critic; \(2\)w/o low\-level, which employs a DQN\-based strategy planner \(i\.e\., StraQ\*[45](https://arxiv.org/html/2608.21969#bib.bib32)\) to guide the token generation of a frozen LLM; and \(3\)w/o im, which removes the intrinsic motivation from the low\-level reward\. Table[6](https://arxiv.org/html/2608.21969#S4.T6)shows that removing either the high\-level or low\-level component results in a significant performance drop, verifying the hierarchical design\.w/o imalso causes a moderate but evident decrease in automatic metrics, indicating the effectiveness of intrinsic motivation\. Appendix[B](https://arxiv.org/html/2608.21969#A2)further presents sensitivity analyses onγ\\gammaand reward weights \(β1\\beta\_\{1\}andβ2\\beta\_\{2\}\)\.

### 4\.7Discussion

#### Method variance across domains\.

From Table[3](https://arxiv.org/html/2608.21969#S4.T3)and Table[4](https://arxiv.org/html/2608.21969#S4.T4), one can observe difference performances for several baselines\. For example, SFT and ArCher perform similarly or even outperform ToSCA on ESConv, yet perform poorly on DailyDialog\. This difference may be because ESConv falls within the ESC domain, with limited topics and clear objectives, resulting in relatively well\-defined behavior patterns \(Exploration\-\>Comforting\-\>Action, as in[23](https://arxiv.org/html/2608.21969#bib.bib14)\) which are easier for supervised methods to imitate\. On the contrary, DailyDialog has open\-ended topics, more spontaneous emotional exchanges, and much broader behavior distributions, which may challenge pure imitations\. Furthermore, the high\-level policy of ArCher maps abstract intents into a continuous latent space, lacking explicit textual expressions \(e\.g\., “Questioning” or “Acknowledging” in ToSCA\), leading to a deficiency in semantic alignment\. Despite these obstacles, our ToSCA has robust performance across all three datasets, indicating its strong generalization and adaptation\.

#### Intrinsic motivation of strategic chatting\.

In ToSCA, high\-level strategies guide the low\-level policy via prompts, but fine\-tuning may cause the model to ignore these constraints for higher rewards\. As shown in Table[7](https://arxiv.org/html/2608.21969#S4.T7), without intrinsic motivation, the model fails to follow theQuestionstrategy\. Intrinsic motivation enforces strategy\-aligned behavior during multi\-turn training\. Additional examples are in Appendix[F](https://arxiv.org/html/2608.21969#A6)\.

UserIt’s a portable TV\. It’s a popular thing now\.AssistantOh, that’s new to me\. So what’s on everyday?UserNews about current affairs, documentaries, music, movies, noncommercial ads and so on\.w/o im\(Question\) I bet there’s something fun to watch on it\.w/ im\(Question\) Got anything exciting on there?

Table 7:Comparison of responses with and without intrinsic motivation \(im\)\. The response without intrinsic motivation \(w/ow/oim\) shows lower self\-consistency\.
#### Lookahead strategy determination\.

To better understand the strategy determination, we also investigate the strategy transitions, with each grid cell\(i,j\)\(i,j\)representing the frequency from the strategy onii\-th row to the strategy onjj\-th column\.

Figure[3](https://arxiv.org/html/2608.21969#S5.F3)\(Left\) exhibits the normalized occurrences of strategy distributions from user to ToSCA in the same turn\. Clearly,“Question → Inform”and“Directive → Commissive”are the two most frequent transitions, which are consistent with the conversation logic, indicating the agent aims to answer the user’s question or provide help\.

Figure[3](https://arxiv.org/html/2608.21969#S5.F3)\(Right\) then shows how ToSCA’s strategies transition between adjacent utterances of itself, revealing ToSCA’s lookahead capability on strategy abstraction\. The upper\-right triangle suggests a dialogue pattern statistically:Inform → Question → Directive → Commissive, which shows the agent’s strategy pipeline, consistent with the Helping Skills Theory discussed in[23](https://arxiv.org/html/2608.21969#bib.bib14)\.

## 5Related Work

### 5\.1Hierarchical Reinforcement Learning

Hierarchical Reinforcement Learning \(HRL\) structures hierarchical MDPs via multi\-level decision\-making, mainly includinggoal\-conditionedframeworks[11](https://arxiv.org/html/2608.21969#bib.bib35);[42](https://arxiv.org/html/2608.21969#bib.bib36);[13](https://arxiv.org/html/2608.21969#bib.bib37)andoptions\-based frameworks with discrete subskills[2](https://arxiv.org/html/2608.21969#bib.bib40);[5](https://arxiv.org/html/2608.21969#bib.bib41);[51](https://arxiv.org/html/2608.21969#bib.bib42)\. Our framework belongs to the latter with the semantic connections between two\-level actions\.

### 5\.2Utterance\-level RL for Dialogue Systems

Utterance\-level RL extends token\-level RLHF[30](https://arxiv.org/html/2608.21969#bib.bib2)to alleviate reward sparsity\. DAT models utterance actions in a continuous space[15](https://arxiv.org/html/2608.21969#bib.bib45), while ArCHer adopts a two\-level HRL framework with a learned high\-level critic[55](https://arxiv.org/html/2608.21969#bib.bib44)\. Our method follows a two\-level HRL structure but introduces discrete strategy abstraction at the high level\.

![Refer to caption](https://arxiv.org/html/2608.21969v1/trans_2_persons.png)

![Refer to caption](https://arxiv.org/html/2608.21969v1/trans_1_person.png)

Figure 3:Occurrences of strategy \(in abbrev\.\) transitions\.Left:Strategy transitions from user \(row\) to ToSCA \(column\)\.Right:Strategies transitions of ToSCA between adjacent turns \(row→\\rightarrowcolumn\)\.

## 6Conclusion

In this paper, we propose ToSCA, an HRL\-based dialogue agent with temporal and strategic abstractions\. It includes a high\-level critic that produces strategy\-level actions, guiding the low\-level policy to generate the detailed response\. To ensure sampling efficiency and convergence, we train the high\-level critic by DQN, and train the low\-level actor and critic by PPO\. To improve the semantic alignment between the dual\-level actions, we further incorporate a KL penalty and intrinsic motivation reward\. Experiments on daily\-life and emotional support conversations, as well as the OOD test, demonstrate the effectiveness of ToSCA\.

## Limitations

Compared to conventional supervised and RL methods, the hierarchical framework introduces additional training complexity, which may require careful tuning to ensure stable convergence\. Furthermore, this study mainly focuses on conversations, while it would also be promising to develop a generalized, self\-evolving hierarchical reasoning and thinking AI\.

## References

- AI@Meta \(2024\)AI@MetaLlama 3 model card\.External Links:[Link](https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md)Cited by:[§1](https://arxiv.org/html/2608.21969#S1.p1.1),[§4\.1](https://arxiv.org/html/2608.21969#S4.SS1.p1.1)\.
- Baconet al\.\(2017\)P\. Bacon, J\. Harb, and D\. PrecupThe option\-critic architecture\.InProceedings of the Thirty\-First AAAI Conference on Artificial Intelligence,AAAI’17,pp\. 1726–1734\.Cited by:[§3\.2](https://arxiv.org/html/2608.21969#S3.SS2.SSS0.Px2.p1.1),[§5\.1](https://arxiv.org/html/2608.21969#S5.SS1.p1.1)\.
- Bradley and Terry \(1952\)R\. A\. Bradley and M\. E\. TerryRank analysis of incomplete block designs: i\. the method of paired comparisons\.Biometrika39\(3/4\),pp\. 324–345\.Cited by:[§C\.1](https://arxiv.org/html/2608.21969#A3.SS1.SSS0.Px3.p1.1),[§4\.3](https://arxiv.org/html/2608.21969#S4.SS3.SSS0.Px1.p1.1)\.
- Choet al\.\(2024\)I\. Cho, R\. Takahashi, Y\. Yanase, and H\. SaitoDeep reinforcement learning with hierarchical action exploration for dialogue generation\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),N\. Calzolari, M\. Kan, V\. Hoste, A\. Lenci, S\. Sakti, and N\. Xue \(Eds\.\),Torino, Italia,pp\. 4566–4579\.External Links:[Link](https://aclanthology.org/2024.lrec-main.408/)Cited by:[§1](https://arxiv.org/html/2608.21969#S1.p2.1),[§1](https://arxiv.org/html/2608.21969#S1.p4.1)\.
- Chunduru and Precup \(2020\)R\. Chunduru and D\. PrecupAttention option\-critic\.In4th Lifelong Machine Learning Workshop at ICML 2020,External Links:[Link](https://openreview.net/forum?id=96wFs9pOFkl)Cited by:[§5\.1](https://arxiv.org/html/2608.21969#S5.SS1.p1.1)\.
- Farahani \(2023\)M\. FarahaniAn empirical study of multitask learning to improve open domain dialogue systems\.InProceedings of the 24th Nordic Conference on Computational Linguistics \(NoDaLiDa\),T\. Alumäe and M\. Fishel \(Eds\.\),Tórshavn, Faroe Islands,pp\. 347–357\.External Links:[Link](https://aclanthology.org/2023.nodalida-1.36/)Cited by:[2nd item](https://arxiv.org/html/2608.21969#A4.I4.i2.p1.1),[Table 14](https://arxiv.org/html/2608.21969#A6.T14.2.15.1),[§4\.6](https://arxiv.org/html/2608.21969#S4.SS6.SSS0.Px1.p1.1)\.
- Hedayatniaet al\.\(2022\)B\. Hedayatnia, D\. Jin, Y\. Liu, and D\. Hakkani\-TurA Systematic Evaluation of Response Selection for Open Domain Dialogue\.InProceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue,O\. Lemon, D\. Hakkani\-Tur, J\. J\. Li, A\. Ashrafzadeh, D\. H\. Garcia, M\. Alikhani, D\. Vandyke, and O\. Dušek \(Eds\.\),Edinburgh, UK,pp\. 298–311\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.sigdial-1.30)Cited by:[§1](https://arxiv.org/html/2608.21969#S1.p1.1)\.
- Hill \(2009\)C\. E\. HillHelping skills: facilitating, exploration, insight, and action\.American Psychological Association\.Cited by:[§1](https://arxiv.org/html/2608.21969#S1.p3.1)\.
- Huet al\.\(2024\)J\. Hu, X\. Wu, Z\. Zhu, Xianyu, W\. Wang, D\. Zhang, and Y\. CaoOpenRLHF: an easy\-to\-use, scalable and high\-performance rlhf framework\.arXiv preprint arXiv:2405\.11143\.Cited by:[§4\.1](https://arxiv.org/html/2608.21969#S4.SS1.p1.1)\.
- Kanget al\.\(2024\)D\. Kang, S\. Kim, T\. Kwon, S\. Moon, H\. Cho, Y\. Yu, D\. Lee, and J\. YeoCan large language models be good emotional supporter? mitigating preference bias on emotional support conversation\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 15232–15261\.External Links:[Link](https://aclanthology.org/2024.acl-long.813/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.813)Cited by:[§C\.1](https://arxiv.org/html/2608.21969#A3.SS1.SSS0.Px3.p2.2),[§C\.3](https://arxiv.org/html/2608.21969#A3.SS3.SSS0.Px2.p1.1),[3rd item](https://arxiv.org/html/2608.21969#A4.I4.i3.p1.1),[Table 15](https://arxiv.org/html/2608.21969#A6.T15.2.16.1),[Table 15](https://arxiv.org/html/2608.21969#A6.T15.2.17.1),[Table 15](https://arxiv.org/html/2608.21969#A6.T15.2.18.1),[Table 15](https://arxiv.org/html/2608.21969#A6.T15.2.19.1),[Table 15](https://arxiv.org/html/2608.21969#A6.T15.2.20.1),[Table 15](https://arxiv.org/html/2608.21969#A6.T15.2.21.1),[Table 15](https://arxiv.org/html/2608.21969#A6.T15.2.22.1),[Table 15](https://arxiv.org/html/2608.21969#A6.T15.2.23.1),[§4\.3](https://arxiv.org/html/2608.21969#S4.SS3.SSS0.Px1.p1.1),[§4\.3](https://arxiv.org/html/2608.21969#S4.SS3.SSS0.Px3.p1.1),[§4\.6](https://arxiv.org/html/2608.21969#S4.SS6.SSS0.Px1.p1.1)\.
- Kulkarniet al\.\(2016\)T\. D\. Kulkarni, K\. Narasimhan, A\. Saeedi, and J\. TenenbaumHierarchical deep reinforcement learning: integrating temporal abstraction and intrinsic motivation\.InProceedings of the 29th International Conference on Neural Information Processing Systems,NIPS ’16\.Cited by:[§5\.1](https://arxiv.org/html/2608.21969#S5.SS1.p1.1)\.
- Leeet al\.\(2026\)J\. Y\. Lee, K\. Lee, and W\. GanModeling the One\-to\-Many Property in Open\-Domain Dialogue with LLMs\.arXiv\.External Links:2506\.15131,[Document](https://dx.doi.org/10.48550/arXiv.2506.15131)Cited by:[§1](https://arxiv.org/html/2608.21969#S1.p1.1)\.
- Levyet al\.\(2019\)A\. Levy, G\. Konidaris, R\. Platt, and K\. SaenkoLearning multi\-level hierarchies with hindsight\.InProceedings of the 7th\. International Conference on Learning Representation,ICLR ’19\.Cited by:[§5\.1](https://arxiv.org/html/2608.21969#S5.SS1.p1.1)\.
- Liet al\.\(2015\)J\. Li, M\. Galley, C\. Brockett, J\. Gao, and B\. DolanA diversity\-promoting objective function for neural conversation models\.arXiv preprint arXiv:1510\.03055\.Cited by:[§C\.2](https://arxiv.org/html/2608.21969#A3.SS2.SSS0.Px3.p1.1)\.
- Liet al\.\(2024a\)K\. Li, Y\. Wang, F\. Viégas, and M\. WattenbergDialogue action tokens: steering language models in goal\-directed dialogue with a multi\-turn planner\.arXiv preprint arXiv:2406\.11978\.Cited by:[§1](https://arxiv.org/html/2608.21969#S1.p2.1),[§1](https://arxiv.org/html/2608.21969#S1.p4.1),[§5\.2](https://arxiv.org/html/2608.21969#S5.SS2.p1.1)\.
- Liet al\.\(2024b\)K\. Li, Y\. Wang, F\. Viégas, and M\. WattenbergDialogue Action Tokens: Steering Language Models in Goal\-Directed Dialogue with a Multi\-Turn Planner\.arXiv\.External Links:2406\.11978,[Document](https://dx.doi.org/10.48550/arXiv.2406.11978)Cited by:[4th item](https://arxiv.org/html/2608.21969#A4.I5.i4.p1.1),[Table 14](https://arxiv.org/html/2608.21969#A6.T14.2.17.1),[§1](https://arxiv.org/html/2608.21969#S1.p2.1),[§4\.4](https://arxiv.org/html/2608.21969#S4.SS4.p1.1),[Table 3](https://arxiv.org/html/2608.21969#S4.T3.2.1.14.1),[Table 4](https://arxiv.org/html/2608.21969#S4.T4.2.1.14.1),[Table 5](https://arxiv.org/html/2608.21969#S4.T5.2.1.9.1)\.
- Li \(2020\)P\. LiAn Empirical Investigation of Pre\-Trained Transformer Language Models for Open\-Domain Dialogue Generation\.arXiv\.External Links:2003\.04195,[Document](https://dx.doi.org/10.48550/arXiv.2003.04195)Cited by:[1st item](https://arxiv.org/html/2608.21969#A4.I4.i1.p1.1),[Table 14](https://arxiv.org/html/2608.21969#A6.T14.2.13.1),[Table 14](https://arxiv.org/html/2608.21969#A6.T14.2.14.1),[§4\.6](https://arxiv.org/html/2608.21969#S4.SS6.SSS0.Px1.p1.1)\.
- Liet al\.\(2017a\)Y\. Li, H\. Su, X\. Shen, W\. Li, Z\. Cao, and S\. NiuDailyDialog: a manually labelled multi\-turn dialogue dataset\.InProceedings of the Eighth International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),G\. Kondrak and T\. Watanabe \(Eds\.\),Taipei, Taiwan,pp\. 986–995\.External Links:[Link](https://aclanthology.org/I17-1099/)Cited by:[§1](https://arxiv.org/html/2608.21969#S1.p3.1),[§4\.2](https://arxiv.org/html/2608.21969#S4.SS2.SSS0.Px1.p1.1)\.
- Liet al\.\(2017b\)Y\. Li, H\. Su, X\. Shen, W\. Li, Z\. Cao, and S\. NiuDailyDialog: A Manually Labelled Multi\-turn Dialogue Dataset\.InProceedings of the Eighth International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),G\. Kondrak and T\. Watanabe \(Eds\.\),Taipei, Taiwan,pp\. 986–995\.Cited by:[1st item](https://arxiv.org/html/2608.21969#A2.I1.i1.p1.1)\.
- Liet al\.\(2024c\)Z\. Li, G\. Chen, R\. Shao, Y\. Xie, D\. Jiang, and L\. NieEnhancing emotional generation capability of large language models via emotional chain\-of\-thought\.External Links:2401\.06836,[Link](https://arxiv.org/abs/2401.06836)Cited by:[2nd item](https://arxiv.org/html/2608.21969#A4.I2.i2.p1.1),[Table 14](https://arxiv.org/html/2608.21969#A6.T14.2.5.1),[Table 15](https://arxiv.org/html/2608.21969#A6.T15.2.5.1),[§4\.4](https://arxiv.org/html/2608.21969#S4.SS4.p1.1),[Table 3](https://arxiv.org/html/2608.21969#S4.T3.2.1.5.1),[Table 4](https://arxiv.org/html/2608.21969#S4.T4.2.1.5.1),[Table 5](https://arxiv.org/html/2608.21969#S4.T5.2.1.3.1)\.
- Lin \(2004\)C\. LinRouge: a package for automatic evaluation of summaries\.InText summarization branches out,pp\. 74–81\.Cited by:[§C\.2](https://arxiv.org/html/2608.21969#A3.SS2.SSS0.Px2.p1.1)\.
- Liuet al\.\(2024\)Q\. Liu, J\. Qin, W\. Ye, H\. Mou, Y\. He, and K\. WangAdaptive prompt routing for arbitrary text style transfer with pre\-trained language models\.InAAAI Conference on Artificial Intelligence,Cited by:[§1](https://arxiv.org/html/2608.21969#S1.p3.1)\.
- Liuet al\.\(2021\)S\. Liu, C\. Zheng, O\. Demasi, S\. Sabour, Y\. Li, Z\. Yu, Y\. Jiang, and M\. HuangTowards emotional support dialog systems\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 3469–3483\.External Links:[Link](https://aclanthology.org/2021.acl-long.269),[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.269)Cited by:[2nd item](https://arxiv.org/html/2608.21969#A2.I1.i2.p1.1),[Table 15](https://arxiv.org/html/2608.21969#A6.T15.2.13.1),[Table 15](https://arxiv.org/html/2608.21969#A6.T15.2.14.1),[Table 15](https://arxiv.org/html/2608.21969#A6.T15.2.15.1),[§1](https://arxiv.org/html/2608.21969#S1.p3.1),[§3\.1](https://arxiv.org/html/2608.21969#S3.SS1.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2608.21969#S4.SS2.SSS0.Px2.p1.1),[§4\.7](https://arxiv.org/html/2608.21969#S4.SS7.SSS0.Px1.p1.1),[§4\.7](https://arxiv.org/html/2608.21969#S4.SS7.SSS0.Px3.p3.1)\.
- Madaan \(2023\)A\. MadaanSelf\-refine: iterative refinement with self\-feedback\.ArXivabs/2303\.17651\.External Links:[Link](https://api.semanticscholar.org/CorpusID:257900871)Cited by:[1st item](https://arxiv.org/html/2608.21969#A4.I2.i1.p1.1),[Table 14](https://arxiv.org/html/2608.21969#A6.T14.2.4.1),[Table 15](https://arxiv.org/html/2608.21969#A6.T15.2.4.1),[§4\.4](https://arxiv.org/html/2608.21969#S4.SS4.p1.1),[Table 3](https://arxiv.org/html/2608.21969#S4.T3.2.1.4.1),[Table 4](https://arxiv.org/html/2608.21969#S4.T4.2.1.4.1)\.
- McGovern and Barto \(2001\)A\. McGovern and A\. G\. BartoAutomatic discovery of subgoals in reinforcement learning using diverse density\.InProceedings of the Eighteenth International Conference on Machine Learning,pp\. 361–368\.Cited by:[§1](https://arxiv.org/html/2608.21969#S1.p4.1)\.
- Mnihet al\.\(2015\)V\. Mnih, K\. Kavukcuoglu, D\. Silver, A\. A\. Rusu, J\. Veness, M\. G\. Bellemare, A\. Graves, M\. Riedmiller, A\. K\. Fidjeland, G\. Ostrovski, S\. Petersen, C\. Beattie, A\. Sadik, I\. Antonoglou, H\. King, D\. Kumaran, D\. Wierstra, S\. Legg, and D\. HassabisHuman\-level control through deep reinforcement learning\.Nature518,pp\. 529–533\.External Links:[Document](https://dx.doi.org/10.1038/nature14236)Cited by:[§2\.2](https://arxiv.org/html/2608.21969#S2.SS2.p2.1)\.
- Murrayet al\.\(2014\)J\. D\. Murray, A\. Bernacchia, D\. J\. Freedman, R\. Romo, J\. D\. Wallis, X\. Cai, C\. Padoa\-Schioppa, T\. Pasternak, H\. Seo, D\. Lee, and X\. WangA hierarchy of intrinsic timescales across primate cortex\.Nature Neuroscience17\(12\),pp\. 1661–1663\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1038/nn.3862)Cited by:[§1](https://arxiv.org/html/2608.21969#S1.p3.1)\.
- Newman \(2023\)M\. E\. J\. NewmanEfficient computation of rankings from pairwise comparisons\.Journal of Machine Learning Research24\(238\),pp\. 1–25\.External Links:[Link](http://jmlr.org/papers/v24/22-1086.html)Cited by:[§C\.1](https://arxiv.org/html/2608.21969#A3.SS1.SSS0.Px3.p1.1)\.
- Ninget al\.\(2024\)X\. Ning, Z\. Lin, Z\. Zhou, Z\. Wang, H\. Yang, and Y\. WangSkeleton\-of\-thought: prompting LLMs for efficient parallel generation\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=mqVgBbNCm9)Cited by:[3rd item](https://arxiv.org/html/2608.21969#A4.I2.i3.p1.1),[Table 14](https://arxiv.org/html/2608.21969#A6.T14.2.6.1),[Table 15](https://arxiv.org/html/2608.21969#A6.T15.2.6.1),[§1](https://arxiv.org/html/2608.21969#S1.p3.1),[§4\.4](https://arxiv.org/html/2608.21969#S4.SS4.p1.1),[Table 3](https://arxiv.org/html/2608.21969#S4.T3.2.1.6.1),[Table 4](https://arxiv.org/html/2608.21969#S4.T4.2.1.6.1),[Table 5](https://arxiv.org/html/2608.21969#S4.T5.2.1.4.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. Christiano, J\. Leike, and R\. LoweTraining language models to follow instructions with human feedback\.External Links:2203\.02155,[Link](https://arxiv.org/abs/2203.02155)Cited by:[§1](https://arxiv.org/html/2608.21969#S1.p1.1),[§2\.3](https://arxiv.org/html/2608.21969#S2.SS3.p1.1),[§3\.4](https://arxiv.org/html/2608.21969#S3.SS4.SSS0.Px1.p1.1),[§3\.5](https://arxiv.org/html/2608.21969#S3.SS5.SSS0.Px3.p1.2),[§5\.2](https://arxiv.org/html/2608.21969#S5.SS2.p1.1)\.
- Papineniet al\.\(2002\)K\. Papineni, S\. Roukos, T\. Ward, and W\. ZhuBleu: a method for automatic evaluation of machine translation\.InProceedings of the 40th annual meeting of the Association for Computational Linguistics,pp\. 311–318\.Cited by:[§C\.2](https://arxiv.org/html/2608.21969#A3.SS2.SSS0.Px1.p1.1)\.
- Petukhovaet al\.\(2014\)V\. Petukhova, A\. Malchanau, and H\. BuntInteroperability of dialogue corpora through iso 24617\-2\-based querying\.\.InLREC,N\. Calzolari, K\. Choukri, T\. Declerck, H\. Loftsson, B\. Maegaard, J\. Mariani, A\. Moreno, J\. Odijk, and S\. Piperidis \(Eds\.\),pp\. 4407–4414\.External Links:[Link](http://dblp.uni-trier.de/db/conf/lrec/lrec2014.html#PetukhovaMB14)Cited by:[§1](https://arxiv.org/html/2608.21969#S1.p3.1)\.
- Rafailovet al\.\(2023\)R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. FinnDirect preference optimization: your language model is secretly a reward model\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 53728–53741\.Cited by:[§C\.1](https://arxiv.org/html/2608.21969#A3.SS1.SSS0.Px3.p1.1)\.
- Rashkinet al\.\(2019\)H\. Rashkin, E\. M\. Smith, M\. Li, and Y\. BoureauTowards empathetic open\-domain conversation models: a new benchmark and dataset\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 5370–5381\.External Links:[Link](https://aclanthology.org/P19-1534/),[Document](https://dx.doi.org/10.18653/v1/P19-1534)Cited by:[3rd item](https://arxiv.org/html/2608.21969#A2.I1.i3.p1.1),[§3\.1](https://arxiv.org/html/2608.21969#S3.SS1.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2608.21969#S4.SS2.SSS0.Px2.p1.1)\.
- Schulmanet al\.\(2017\)J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. KlimovProximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§2\.3](https://arxiv.org/html/2608.21969#S2.SS3.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. GuoDeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models\.arXiv\.External Links:2402\.03300,[Document](https://dx.doi.org/10.48550/arXiv.2402.03300)Cited by:[§1](https://arxiv.org/html/2608.21969#S1.p1.1)\.
- Shiet al\.\(2025\)L\. X\. Shi, B\. Ichter, M\. R\. Equi, L\. Ke, K\. Pertsch, Q\. Vuong, J\. Tanner, A\. Walling, H\. Wang, N\. Fusai, A\. Li\-Bell, D\. Driess, L\. Groom, S\. Levine, and C\. FinnHi Robot: Open\-Ended Instruction Following with Hierarchical Vision\-Language\-Action Models\.InForty\-Second International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2608.21969#S1.p4.1)\.
- Stolle and Precup \(2002\)M\. Stolle and D\. PrecupLearning options in reinforcement learning\.InProceedings of the 5th International Symposium on Abstraction, Reformulation and Approximation,pp\. 212–223\.Cited by:[§1](https://arxiv.org/html/2608.21969#S1.p5.1)\.
- Sunet al\.\(2023\)Z\. Sun, Y\. Shen, Q\. Zhou, H\. Zhang, Z\. Chen, D\. Cox, Y\. Yang, and C\. GanPrinciple\-driven self\-alignment of language models from scratch with minimal human supervision\.InProceedings of the 37th International Conference on Neural Information Processing Systems,NIPS ’23,Red Hook, NY, USA\.Cited by:[§1](https://arxiv.org/html/2608.21969#S1.p3.1)\.
- Sunet al\.\(2025\)Z\. Sun, B\. Pang, X\. Yuan, X\. Xu, Y\. Song, R\. Song, and Y\. LiHierarchical reinforcement learning with curriculum demonstrations and goal\-guided policies for sequential robotic manipulation\.Engineering Applications of Artificial Intelligence153,pp\. 110866\.External Links:[Document](https://dx.doi.org/10.1016/j.engappai.2025.110866)Cited by:[§1](https://arxiv.org/html/2608.21969#S1.p4.1)\.
- Suttonet al\.\(1999\)R\. S\. Sutton, D\. Precup, and S\. SinghBetween mdps and semi\-mdps: a framework for temporal abstraction in reinforcement learning\.Artif\. Intell\.112\(1–2\),pp\. 181–211\.External Links:ISSN 0004\-3702,[Link](https://doi.org/10.1016/S0004-3702(99)00052-1),[Document](https://dx.doi.org/10.1016/S0004-3702%2899%2900052-1)Cited by:[Appendix E](https://arxiv.org/html/2608.21969#A5.SS0.SSS0.Px1.p1.1),[Appendix E](https://arxiv.org/html/2608.21969#A5.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2608.21969#S1.p4.1),[§1](https://arxiv.org/html/2608.21969#S1.p5.1),[§3\.7](https://arxiv.org/html/2608.21969#S3.SS7.p1.1)\.
- Vezhnevetset al\.\(2017\)A\. S\. Vezhnevets, S\. Osindero, T\. Schaul, N\. Heess, M\. Jaderberg, D\. Silver, and K\. KavukcuoggluFeUal networks for hierarchical reinforcement learning\.InProceedings of the Workshop on Negative Dependence in Machine Learning at the 34th International Conference on Machine Learning,ICML ’17\.Cited by:[§5\.1](https://arxiv.org/html/2608.21969#S5.SS1.p1.1)\.
- Wanget al\.\(2023\)L\. Wang, W\. Xu, Y\. Lan, Z\. Hu, Y\. Lan, R\. K\. Lee, and E\. LimPlan\-and\-solve prompting: improving zero\-shot chain\-of\-thought reasoning by large language models\.External Links:2305\.04091,[Link](https://arxiv.org/abs/2305.04091)Cited by:[5th item](https://arxiv.org/html/2608.21969#A4.I2.i5.p1.1),[Table 14](https://arxiv.org/html/2608.21969#A6.T14.2.8.1),[Table 15](https://arxiv.org/html/2608.21969#A6.T15.2.8.1),[§4\.4](https://arxiv.org/html/2608.21969#S4.SS4.p1.1),[Table 3](https://arxiv.org/html/2608.21969#S4.T3.2.1.8.1),[Table 4](https://arxiv.org/html/2608.21969#S4.T4.2.1.8.1),[Table 5](https://arxiv.org/html/2608.21969#S4.T5.2.1.6.1)\.
- Wanget al\.\(2024\)X\. Wang, J\. He, Z\. yang, Y\. Wang, X\. Meng, K\. Pan, and Z\. SuiFSM: A Finite State Machine Based Zero\-Shot Prompting Paradigm for Multi\-Hop Question Answering\.arXiv\.External Links:2407\.02964,[Document](https://dx.doi.org/10.48550/arXiv.2407.02964)Cited by:[6th item](https://arxiv.org/html/2608.21969#A4.I2.i6.p1.1),[Table 14](https://arxiv.org/html/2608.21969#A6.T14.2.9.1),[Table 15](https://arxiv.org/html/2608.21969#A6.T15.2.9.1),[§4\.4](https://arxiv.org/html/2608.21969#S4.SS4.p1.1),[Table 3](https://arxiv.org/html/2608.21969#S4.T3.2.1.9.1),[Table 4](https://arxiv.org/html/2608.21969#S4.T4.2.1.9.1),[Table 5](https://arxiv.org/html/2608.21969#S4.T5.2.1.7.1)\.
- Wanget al\.\(2025\)X\. Wang, Y\. Zhao, Q\. Gu, Z\. Jiang, Y\. Chen, and L\. JiConvert language model into a value\-based strategic planner\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 6: Industry Track\),G\. Rehm and Y\. Li \(Eds\.\),Vienna, Austria,pp\. 1444–1456\.External Links:[Link](https://aclanthology.org/2025.acl-industry.102/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-industry.102),ISBN 979\-8\-89176\-288\-6Cited by:[§1](https://arxiv.org/html/2608.21969#S1.p3.1),[§4\.6](https://arxiv.org/html/2608.21969#S4.SS6.SSS0.Px3.p1.1)\.
- Watanabeet al\.\(2025\)T\. Watanabe, A\. Kubo, K\. Tsunoda, T\. Matsuba, S\. Akatsuka, Y\. Noda, H\. Kioka, J\. Izawa, S\. Ishii, and Y\. NakamuraHierarchical reinforcement learning with central pattern generator for enabling a quadruped robot simulator to walk on a variety of terrains\.Scientific Reports15\(1\),pp\. 11262\.External Links:[Document](https://dx.doi.org/10.1038/s41598-025-94163-2)Cited by:[§1](https://arxiv.org/html/2608.21969#S1.p4.1)\.
- Yanget al\.\(2024\)L\. Yang, Z\. Yu, T\. Zhang, S\. Cao, M\. Xu, W\. Zhang, J\. E\. Gonzalez, and B\. CuiBuffer of thoughts: thought\-augmented reasoning with large language models\.Advances in Neural Information Processing Systems\.Cited by:[§1](https://arxiv.org/html/2608.21969#S1.p3.1)\.
- Yaoet al\.\(2023\)S\. Yao, D\. Yu, J\. Zhao, I\. Shafran, T\. L\. Griffiths, Y\. Cao, and K\. R\. NarasimhanTree of thoughts: deliberate problem solving with large language models\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=5Xc1ecxO1h)Cited by:[4th item](https://arxiv.org/html/2608.21969#A4.I2.i4.p1.1),[Table 14](https://arxiv.org/html/2608.21969#A6.T14.2.7.1),[Table 15](https://arxiv.org/html/2608.21969#A6.T15.2.7.1),[§4\.4](https://arxiv.org/html/2608.21969#S4.SS4.p1.1),[Table 3](https://arxiv.org/html/2608.21969#S4.T3.2.1.7.1),[Table 4](https://arxiv.org/html/2608.21969#S4.T4.2.1.7.1),[Table 5](https://arxiv.org/html/2608.21969#S4.T5.2.1.5.1)\.
- Yueet al\.\(2025\)Y\. Yue, Z\. Chen, R\. Lu, A\. Zhao, Z\. Wang, Y\. Yue, S\. Song, and G\. HuangDoes Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2608.21969#S1.p1.1)\.
- Zeraatiet al\.\(2023\)R\. Zeraati, Y\. Shi, N\. Steinmetz, M\. Gieselmann, A\. Thiele, T\. Moore, A\. Levina, and T\. EngelIntrinsic timescales in the visual cortex change with selective attention and reflect spatial connectivity\.Nature Communications14\(1\),pp\. 1858\.External Links:[Document](https://dx.doi.org/10.1038/s41467-023-37613-7)Cited by:[§1](https://arxiv.org/html/2608.21969#S1.p3.1)\.
- Zhang and Whiteson \(2019\)S\. Zhang and S\. WhitesonDAC: the double actor\-critic architecture for learning options\.InAdvances in Neural Information Processing Systems,H\. Wallach, H\. Larochelle, A\. Beygelzimer, F\. d'Alché\-Buc, E\. Fox, and R\. Garnett \(Eds\.\),Vol\.32,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2019/file/4f284803bd0966cc24fa8683a34afc6e-Paper.pdf)Cited by:[§3\.2](https://arxiv.org/html/2608.21969#S3.SS2.SSS0.Px2.p1.1),[§5\.1](https://arxiv.org/html/2608.21969#S5.SS1.p1.1)\.
- Zhanget al\.\(2020\)Y\. Zhang, S\. Sun, M\. Galley, Y\. Chen, C\. Brockett, X\. Gao, J\. Gao, J\. Liu, and B\. DolanDialoGPT: large\-scale generative pre\-training for conversational response generation\.InACL, system demonstration,Cited by:[4th item](https://arxiv.org/html/2608.21969#A4.I4.i4.p1.1)\.
- Zhaoet al\.\(2026\)Y\. Zhao, Q\. Gu, X\. Wang, T\. Chen, H\. Li, and L\. JiEmoFSM: a finite state machine for emotional support conversation\.InData Science: Foundations and Applications: 30th Pacific\-Asia Conference on Knowledge Discovery and Data Mining, PAKDD 2026, Hong Kong, China, June 9–12, 2026, Proceedings, Part I,Berlin, Heidelberg,pp\. 115–127\.External Links:ISBN 978\-981\-92\-1925\-4,[Link](https://doi.org/10.1007/978-981-92-1926-1_10),[Document](https://dx.doi.org/10.1007/978-981-92-1926-1%5F10)Cited by:[3rd item](https://arxiv.org/html/2608.21969#A4.I3.i3.p1.1),[Table 14](https://arxiv.org/html/2608.21969#A6.T14.2.12.1),[Table 15](https://arxiv.org/html/2608.21969#A6.T15.2.12.1),[§1](https://arxiv.org/html/2608.21969#S1.p3.1),[§4\.4](https://arxiv.org/html/2608.21969#S4.SS4.p1.1),[Table 3](https://arxiv.org/html/2608.21969#S4.T3.2.1.12.1),[Table 4](https://arxiv.org/html/2608.21969#S4.T4.2.1.12.1)\.
- Zhouet al\.\(2024a\)J\. Zhou, L\. Pang, H\. Shen, and X\. ChengThink before you speak: cultivating communication skills of large language models via inner monologue\.InFindings of the Association for Computational Linguistics: NAACL 2024,K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 3925–3951\.External Links:[Link](https://aclanthology.org/2024.findings-naacl.248/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-naacl.248)Cited by:[§1](https://arxiv.org/html/2608.21969#S1.p3.1)\.
- Zhouet al\.\(2024b\)Y\. Zhou, A\. Zanette, J\. Pan, S\. Levine, and A\. KumarArCHer: training language model agents via hierarchical multi\-turn RL\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 62178–62209\.External Links:[Link](https://proceedings.mlr.press/v235/zhou24t.html)Cited by:[3rd item](https://arxiv.org/html/2608.21969#A4.I5.i3.p1.1),[Table 14](https://arxiv.org/html/2608.21969#A6.T14.2.18.1),[Table 15](https://arxiv.org/html/2608.21969#A6.T15.2.25.1),[§1](https://arxiv.org/html/2608.21969#S1.p2.1),[§1](https://arxiv.org/html/2608.21969#S1.p4.1),[§3\.1](https://arxiv.org/html/2608.21969#S3.SS1.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2608.21969#S3.SS1.SSS0.Px2.p1.2),[§3\.2](https://arxiv.org/html/2608.21969#S3.SS2.SSS0.Px2.p1.1),[§4\.4](https://arxiv.org/html/2608.21969#S4.SS4.p1.1),[Table 3](https://arxiv.org/html/2608.21969#S4.T3.2.1.15.1),[Table 4](https://arxiv.org/html/2608.21969#S4.T4.2.1.15.1),[Table 5](https://arxiv.org/html/2608.21969#S4.T5.2.1.10.1),[§5\.2](https://arxiv.org/html/2608.21969#S5.SS2.p1.1)\.

## Appendix AImplementation Details

### A\.1Detailed Prompts

#### Prompt of the high\-level agent\.

The following is a high\-level prompt\. In DailyDialog, there are 4 predefined strategies\. With this prompt, the language model is naturally inclined to generate one of the candidate options as the next token due to the structure of the instruction\.

High\-level promptYou are given a multi\-turn dialogue between a user and an assistant\. The user’s basic situation is as follows:Emotion: \{e​m​oemo\}Description: \{d​e​s​cdesc\}Below is the dialogue history between the user and the assistant:\{h​i​s​t​o​r​yhistory\}The user’s current query is:\{q​u​e​r​yquery\}Based on the above context, please select the most appropriate response strategy from the following options:strategy \#\(1\) \{a1a\_\{1\}\}…strategy \#\(k\) \{aka\_\{k\}\}Please provide your selection in the format of \(1\) through \(k\)\. Your selection is:
#### Prompt of the low\-level agent\.

The low\-level prompt derives the response strategy from the high\-level and generates an appropriate reply accordingly\.

Low\-level promptYou are given a multi\-turn dialogue between a user and an assistant\. The user’s basic situation is as follows:Emotion: \{e​m​oemo\}Description: \{d​e​s​cdesc\}Below is the dialogue history between the user and the assistant:\{h​i​s​t​o​r​yhistory\}The user’s current query is:\{q​u​e​r​yquery\}The current response strategy is:\{s​t​r​astra\}Based on the current response strategy and other information, please act as a assistant and provide the best response\. Keep replies brief without additional pronouns or extra elements\.
#### The evaluation prompt of GPT\-4o\.

The responses generated by the low\-level prompt are evaluated by GPT\-4o to simulate human judgment across five dimensions:Acceptance,Effectiveness,Sensitivity,Fluency, andEmotion\. Each dimension is scored as either 0 or 1, indicating whether the response is strong in that aspect\. The final output includes the total score and a brief explanation\.

Evaluation PromptYou are an expert evaluator simulating human assessment of dialogue responses\. You are given a multi\-turn dialogue between a user and an assistant\. The user’s basic situation is as follows:Emotion: \{e​m​oemo\}Description: \{d​e​s​cdesc\}Below is the dialogue history between the user and the assistant:\{h​i​s​t​o​r​yhistory\}The user’s current query is:\{q​u​e​r​yquery\}The assistant’s response is:\{r​e​s​p​o​n​s​eresponse\}Please evaluate the assistant’s response across five binary\-scored dimensions\.1\. Acceptance– Is the response socially appropriate and non\-offensive?2\. Effectiveness– Does the response address the user’s intent appropriately?3\. Sensitivity– Does the response consider emotional or situational context?4\. Fluency– Is the response grammatically correct and fluent?5\. Emotion– Does the response convey an appropriate emotional tone?Each dimension should be scored as0 \(unsatisfactory\)or1 \(satisfactory\)\.Output format:Acceptance: \[0/1\],Effectiveness: \[0/1\]Sensitivity: \[0/1\],Fluency: \[0/1\]Emotion: \[0/1\],Total score: \[0–5\]Explanation: \[your reasoning here\]

### A\.2More Methodology Details

#### Kullback\-Leibler \(KL\) Divergence

is a fundamental concept in information theory and machine learning that measures the difference between two probability distributions\. It is a crucial tool for comparing and approximating distributions, with far\-reaching implications in various fields\. The KL Divergence between two probability distributionsPPandQQis defined as: s

DK​L\(P\|\|Q\)=∑i=1Np\(x\)logp⁡\(x\)q⁡\(x\)D\_\{KL\}\(P\|\|Q\)=\\sum\_\{i=1\}^\{N\}p\(x\)\\log\\frac\{p\(x\)\}\{q\(x\)\}\(14\)wherep⁡\(x\)p\(x\)andq⁡\(x\)q\(x\)are the probability density functions ofPPandQQrespectively\.

## Appendix BExperiment Settings

### B\.1Datasets

#### Detailed introductions\.

We introduce our experimental datasets in more details:

- •TheDailyDialogdataset[19](https://arxiv.org/html/2608.21969#bib.bib47)is a widely used benchmark for daily\-life conversation\. It contains 13,118 multi\-turn dialogues across diverse daily topics \(e\.g\., family, work, hobbies\) with manually annotated emotions and dialogue strategies \(named ‘act’ in the original dataset\)\. As shown in Table[8](https://arxiv.org/html/2608.21969#A2.T8), it is annotated with 7 emotions and 4 strategies:Inform\(Inf\.\),Question\(Que\.\),Directives\(Dir\.\), andCommissive\(Com\.\)\.
- •TheESConvdataset[23](https://arxiv.org/html/2608.21969#bib.bib14)is a specialized corpus for emotional support research\. It contains 1,000\+ multi\-turn dialogues where users articulate personal struggles \(e\.g\., workplace stress, interpersonal conflicts, self\-esteem issues\), and support providers respond with empathy, validation, and actionable guidance\. Aligned with Table[8](https://arxiv.org/html/2608.21969#A2.T8), it is annotated with 11 emotions and 8 conversational strategies\. The strategies are structured across three stages \(I–III\)\.
- •TheEmpatheticDialoguesdataset[34](https://arxiv.org/html/2608.21969#bib.bib15)contains conversations centered on recognizing and responding to emotions \(e\.g\., joy, sadness, frustration\)\. It has annotations of user emotions but no strategies\.

DatasetStrategiesAbbr\.StageDailyDialogInformInf\.\-QuestionQue\.\-DirectivesDir\.\-CommissiveCom\.\-ESConvQuestionQue\.IRestatement or ParaphrasingRes\.& Par\.IReflection of FeelingsRef\.IISelf\-disclosureSelf\-Dis\.IIAffirmation and ReassuranceAff\.& Rea\.IIIProviding SuggestionsPro\.IIIInformationInf\.IIIOthersOthers\-

Table 8:Strategies, abbreviations, and stages within DailyDialog and ESConv\.
#### Statistics of datasets\.

Table[9](https://arxiv.org/html/2608.21969#A2.T9)summarizes the statistics of ESConv, DailyDialog, and EmpatheticDialogues\. It presents key metrics like the number of sessions, utterances, average utterance length, and counts of strategies and emotions for both speakers in each dataset\. For instance, DailyDialog has the most sessions \(13\.1k\) and utterances \(103\.0k in total for both speakers\), while ESConv shows detailed strategy annotations \(11 for Speaker1\) and a moderate number of sessions \(1\.3k\)\. EmpatheticDialogues, while not annotated with strategies, offers a rich set of 32 emotion categories, making it valuable for studying emotional diversity and text\-based evaluation\.

CategoryESconvDailyDialogEmpathetic\-\# Sessions1\.3K13\.1k2\.5K\# Utterances38K103\.0k11\.0KAverage \# Utterances28\.97\.94\.3Average Utterance Length18\.813\.616\.7Speaker1\# Utterances20K53\.8k5\.7KAvg \# Utterances15\.44\.12\.2Avg Uttr Len16\.813\.220\.8\# Strategies\-4\-\# Emotions11732Speaker2\# Utterances18K49\.2k5\.2KAvg \# Utterances13\.63\.92\.1Avg Uttr Len21\.014\.112\.3\# Strategies84\-\# Emotions\-732

Table 9:Statistics of ESConv, DailyDialog and EmpatheticDialogues \(named Empathetic\- in the table\)\.In our experiment, we use the following data splitting strategy:

- •DailyDialog: Following its official default configuration, the dataset is divided into a training set \(11118 conversations\), a validation set \(1000 conversations\), and a test set \(1000 conversations\)\.
- •ESConv: We split the dataset into the training and test sets with a random 9:1 ratio, since the original dataset does not provide the official splits\.
- •EmpatheticDialogues: Following its official default configuration, Conversations are partitioned into approximate splits of 80% training \(19533 conversations\), 10% validation \(2770 conversations\), and 10% testing \(2547 conversations\)\. To avoid data leakage, all conversations sharing the same initial situational description \(provided by a speaker\) are grouped into the same partition\.

#### Example of data samples\.

Table[10](https://arxiv.org/html/2608.21969#A2.T10)shows an example dialogue snippet from the ESConv dataset\. It illustrates a conversation where the seeker expresses anxiety about quitting a disliked job without a secure alternative\. The dialogue is annotated with the topic, the seeker’s query, the emotional state \(anxiety with high intensity\), and the empathetic strategy used by the supporter—in this case, a “reflection of feelings\.” This example highlights how ESConv captures nuanced emotional expression alongside supportive conversational strategies\.

TopicI hate my job but I am scared to quit and seek a new career\.Query\{history\} seeker:Seriously\! What I’m scared of now is how to secure another job\.EmotionAnxiety \(intensity: 5\)StrategyReflection of feelingsResponsesupporter:I can feel your pain just by chatting with you\.

Table 10:An example of ESConv\.

### B\.2Key Hyperparameters

GroupParameterDailyDialogESConvSFTlr5e\-55e\-5batch size6464ToSCAhigh\-level:lr ofQHQ^\{\\text\{H\}\}5e\-65e\-6batch size \(BHB^\{\\text\{H\}\}\)6464replay buffer size2000020000max window len10242048update freq ofϕ→ϕ¯\\phi\\rightarrow\\bar\{\\phi\}1010γ\\gamma0\.850\.85low\-level:lr ofπL\\pi^\{\\text\{L\}\}9e\-79e\-7lr ofQLQ^\{\\text\{L\}\}9e\-49e\-4batch size \(BLB^\{\\text\{L\}\}\)6464replay buffer size2000020000max window len10242048max decoding length128128β1\\beta\_\{1\}0\.010\.01β2\\beta\_\{2\}0\.010\.01γ\\gamma1\.01\.0

Table 11:Full list of hyperparameters\.#### Full hyperparameter values\.

Table[11](https://arxiv.org/html/2608.21969#A2.T11)shows the full list of hyperparameters in the training process of DailyDialog and ESConv\. The training program is running on OpenRLHF, with zero stage \(of Deepseed\) of 3, and the data type of bf16\.

For the policy inference on all three datasets, we conduct the general decoding process of LLM, with top\-p of 0\.9, top\-k of 10, and temperature of 0\.7\.

#### Sensitivity analysis onγ\\gamma\.

Figure[4](https://arxiv.org/html/2608.21969#A2.F4)\(upper\) shows ID \(onESConv\) performance curves under differentγ\\gammasettings\. One can observe the optimalγ\\gammahappens on0\.850\.85, which corresponds to higherAccandMaF1, as well as lowerbias\. The curves of generative metrics have similar trends\. Therefore, we chooseγ=0\.85\\gamma=0\.85as the formal setting\.

#### Sensitivity analysis on reward weights\.

To further explore the effectiveness of the reward components added to the low\-level reward \(rLr^\{\\text\{L\}\}\), we conduct the sensitivity analysis on their weights,β1\\beta\_\{1\}andβ2\\beta\_\{2\}\. The results are shown on Figure[4](https://arxiv.org/html/2608.21969#A2.F4)\(low\)\. First, the curves reveal that including of both loss terms \(rKLr^\{\\text\{KL\}\}andrimr^\{\\text\{im\}\}\) have positive effects, with the performance increasing from less to larger values of weights\. Second, we empirically find the optimal value ofβ1\\beta\_\{1\}happens at0\.010\.01, andβ2=0\.01\\beta\_\{2\}=0\.01has better strategy accuracy, whileβ2=0\.1\\beta\_\{2\}=0\.1has higherB−2B\-2result\. To make the setting simple, we temporally choose bothβ1=β2=0\.01\\beta\_\{1\}=\\beta\_\{2\}=0\.01in the formal experimental setting\.

Figure 4:Sensitivity plots of ToSCA\-imit on differentγ\\gammaand model sizes\. Metrics include Acc,𝒬\\mathcal\{Q\}, B2, and R\-L\.

## Appendix CDetailed Evaluation Methods

### C\.1Classification Metrics

#### Accuracy\.

We define evaluate strategy determination from the classification perspective, where the accuracy \(Acc\) is defined as the fraction of correctly predicted strategies over the entire number of strategies, comparing to the annotated strategies in the dataset:

Acc=\# of correctly predicted strategies\# of annotated strategies\\text\{Acc\}=\\frac\{\\text\{\\\# of correctly predicted strategies\}\}\{\\text\{\\\# of annotated strategies\}\}\(15\)

#### F1\-scores\.

F1\-related scores include Micro\-F1 and Macro\-F1\. Micro\-F1 considers the overall precision and recall of all instances, while Macro\-F1 equals the average F1\-score of labels\.

#### b​i​a​sbias\.

We define the preferenceb​i​a​sbiasashow much the model prefers certain labels over others\. To quantify the preference for each strategy in LLMs, we employ the Bradley\-Terry model\([3](https://arxiv.org/html/2608.21969#bib.bib24)\), which is widely used in human preference modeling\([33](https://arxiv.org/html/2608.21969#bib.bib25)\)\. Following[28](https://arxiv.org/html/2608.21969#bib.bib26), we formally derive the preferenceppfor strategyiias follows:

pi′=∑j\(wi​j​pj\)/\(pi\+pj\)∑jwj​i/\(pi\+pj\)\\normalsize p\_\{i\}^\{\\prime\}=\\frac\{\\sum\_\{j\}\(w\_\{ij\}p\_\{j\}\)/\(p\_\{i\}\+p\_\{j\}\)\}\{\\sum\_\{j\}w\_\{ji\}/\(p\_\{i\}\+p\_\{j\}\)\}\(16\)wherewi​jw\_\{ij\}represents the number of times the model predicts strategyiiwhen the ground\-truth strategy isjj\. All of the preferencepip\_\{i\}are initialized as 1 and updated through iteration of the Eq \([16](https://arxiv.org/html/2608.21969#A3.E16)\) , wherepi′p\_\{i\}^\{\\prime\}represents the preference in the next iteration\. After the final iteration, we scale the total sum ofpip\_\{i\}to 8 \(∑pi=8\\sum\{p\_\{i\}\}=8\) so that the averagep¯\\bar\{p\}becomes 1, indicating a strong preference for strategyiiifpi\>1p\_\{i\}\>1\.

We use a standard deviation of preferencespip\_\{i\}across the strategies asb​i​a​sbias\.

b​i​a​s=∑i=1N\(pi−p¯\)2N\\normalsize bias=\\sqrt\{\\frac\{\\sum\_\{i=1\}^\{N\}\(p\_\{i\}\-\\bar\{p\}\)^\{2\}\}\{N\}\}\(17\)where a higher value forb​i​a​sbiasindicates that the model exhibits a clear preference for both preferred and non\-preferred strategies\([10](https://arxiv.org/html/2608.21969#bib.bib16)\)\.

### C\.2Generative Metrics

#### Bleu\-2\.

B\-2[31](https://arxiv.org/html/2608.21969#bib.bib27)first compute the geometric average of the modifiednn\-gram precisions,pnp\_\{n\}, usingnn\-grams up to lengthNNand positive weightswnw\_\{n\}summing to one\.

Next, letccbe the length of the prediction andrrbe the reference length\. The BP and Bleu\-2 are computed as follows\.

BP=\{1if​c\>re\(1−r/c\)if​c≤r\.\\mathrm\{BP\}=\\left\\\{\\begin\{array\}\[\]\{ll\}1&\\text\{ if \}c\>r\\\\ e^\{\(1\-r/c\)\}&\\text\{ if \}c\\leq r\\end\{array\}\.\\right\.\(18\)
Bleu=BP⋅exp⁡\(∑n=1Nwn​log⁡pn\)\.\\mathrm\{Bleu\}=\\mathrm\{BP\}\\cdot\\exp\\left\(\\sum\_\{n=1\}^\{N\}w\_\{n\}\\log p\_\{n\}\\right\)\.\(19\)

#### Rouge\-L\.

R\-L[21](https://arxiv.org/html/2608.21969#bib.bib28)propose using LCS\-based F\-measure to estimate the similarity between two summariesXXof lengthmmandYYof lengthnn, assumingXXis a reference summary sentence andYYis a candidate summary sentence, as follows:

Rl​c​s=L​C​S​\(X,Y\)m\\displaystyle R\_\{lcs\}=\\frac\{LCS\(X,Y\)\}\{m\}\(20\)Pl​c​s=L​C​S​\(X,Y\)n\\displaystyle P\_\{lcs\}=\\frac\{LCS\(X,Y\)\}\{n\}Fl​c​s=\(1\+β2\)​Rl​c​s​Pl​c​sRl​c​s\+β2​Pl​c​s\\displaystyle F\_\{lcs\}=\\frac\{\\left\(1\+\\beta^\{2\}\\right\)R\_\{lcs\}P\_\{lcs\}\}\{R\_\{lcs\}\+\\beta^\{2\}P\_\{lcs\}\}
WhereLCS⁡\(X,Y\)\\operatorname\{LCS\}\(X,Y\)is the length of a longest common subsequence ofXXandYY, andβ=Pl​c​s/Rlcs\\beta=P\_\{lcs\}/R\_\{\\text\{lcs \}\}when∂Fl​c​s/∂Rl​c​s=∂Fl​c​s/∂Pl​c​s\\partial F\_\{lcs\}/\\partial R\_\{lcs\}=\\partial F\_\{lcs\}/\\partial P\_\{lcs\}\. In DUC,β\\betais set to a very big number\(→∞\)\(\\rightarrow\\infty\)\. Therefore, the LCS\-based F\-measure, i\.e\. Equation[20](https://arxiv.org/html/2608.21969#A3.E20), is Rouge\-L\.

#### Dist\-2\.

[14](https://arxiv.org/html/2608.21969#bib.bib29)report the degree of diversity by calculating the number of distinct unigrams and bigrams in generated responses\. The value is scaled by the total number of generated tokens to avoid favoring long sentences:

D​i​s​t​\(n\)=C​o​u​n​t​\(u​n​i​q​u​e​n−g​r​a​m\)C​o​u​n​t​\(n−g​r​a​m\)Dist\(n\)=\\frac\{Count\(unique\\ n\-gram\)\}\{Count\(n\-gram\)\}\(21\)

### C\.3Principle of Human Scoring

#### Details of Human Annotation\.

To systematically assess the model performance, we ask 8 human evaluators to rate the model responses across multiple dimensions\. Evaluators are required to independently evaluate each sample in strict accordance with the pre\-established criteria\. We conduct cross validation of their results to avoid personal bias\. The lowest and the highest scores are removed and the rest are averaged\.

#### Evaluation Dimensions\.

We start with the criteria proposed by[10](https://arxiv.org/html/2608.21969#bib.bib16)\. The human evaluation is aimed to algin with the ultimate purpose of emotional support conversation, the seeker’ssatisfaction\. To achieve this, the supporter’s behavior can be further classified into the following criteria:

Acceptance: Does the seeker accept without discomfort;

Effectiveness: Is it helpful in shifting negative emotions or attitudes towards a positive direction;

Sensitivity: Does it take into consideration the general state of the seeker\.

satisfaction: Does it resolve their emotional distress, leading to an overall positive perception of the conversation?

To achieve a more elaborate assessment, we consider three more dimensions addressing the generation quality:

Fluency: the level of fluency of response\.

Emotion: the emotional intensity of response which could affect the seeker’s emotion state\.

#### Details of Scoring Criteria\.

The detailed manual scoring criteria are as follows:

- •Fluency: 1: The sentence is highly incoherent, making it extremely difficult to understand and failing to convey a meaningful idea\. 2: The sentence has significant incoherence issues, with only parts of it making sense and struggling to form a complete thought\. 3: The sentence contains some incoherence and occasional errors, but can still convey the general meaning to a certain extent\. 4: The sentence is mostly fluent with only minor errors or slight awkwardness in expression, and effectively communicates the intended meaning\. 5: Perfect\. The sentence is completely fluent, free of any errors in grammar, punctuation, or expression, and clearly conveys the idea\.
- •Emotion: 1: The emotional expression is extremely inappropriate and chaotic, not in line with the content, and may convey the wrong emotions\. 2: The emotional expression has obvious flaws, either too weak or exaggerated, and is disjointed from the content\. 3: The emotional expression is average\. It can convey basic emotions but lacks depth and has minor issues\. 4: The emotional expression is good\. It can effectively convey the intended emotion with an appropriate intensity and is well integrated with the content\. 5: The emotional expression is excellent\. It is rich, nuanced, and perfectly matches the content, capable of evoking a strong and appropriate emotional response\.
- •Acceptance: 1: The response inescapably triggers emotional resistance\. 2: The response is highly likely to trigger emotional resistance\. 3: The response has a possibility of emotional resistance occurring\. 4: The response rarely provokes emotional resistance\. 5: The response has no occurrence of emotional resistance\.
- •Effectiveness: 1: The response actually worsens the seeker’s emotional distress\. 2: The response carries the risk of increasing stress levels, and this outcome varies depending on the individual user\. 3: The response fails to alter the seeker’s current emotional intensity and keeps it at the same level\. 4: The response shows promise in calming the emotional intensity; however, it is overly complicated or ambiguous for the user to fully comprehend and utilize effectively\. 5: The response appears to be highly effective in soothing the seeker’s emotions and offers valuable and practical emotional support\.
- •Sensitivity: 1: The response renders inaccurate evaluations regarding the seeker’s state\. 2: The response is characterized by rash judgments, as it lacks adequate assessment and in\-depth exploration of the seeker’s state\. 3: The response is formulated with a one\-sided judgment and a limited exploration of the seeker’s state\. 4: The response demonstrates an understanding that only covers a part of the seeker’s state\. 5: The response precisely grasps the seeker’s state and is appropriately tailored according to the seeker’s actual situation\.
- •Satisfaction: 1: The response is extremely disappointing\. It doesn’t answer the question at all and is of no help\. 2: The response is poor\. It only gives a partial answer and leaves many doubts unresolved\. 3: The response is average\. It meets the basic requirements but isn’t particularly outstanding\. 4: The response is good\. It answers the question clearly and provides some useful details\. 5: The response is excellent\. It not only answers the question perfectly but also offers valuable additional insights\.

## Appendix DIntroduction of Baselines

We categorize the baselines into five groups, and implement most of them on the same LLM backbone unless otherwise specified\.

#### Direct Inference\.

- •Direct: The model generates a response given the dialogue history without additional guidance\.
- •Direct\-Refine: A simple two\-step approach where the model first generates a response, then revises it by incorporating user feedback directly\.

#### Prompt\-based Methods\.

- •Self\-Refine[24](https://arxiv.org/html/2608.21969#bib.bib18): Automatically generates satisfaction\-oriented feedback based on its own output, and then uses this feedback to refine its initial response\.
- •Emotional Chain of Thoughts\(ECoT\)[20](https://arxiv.org/html/2608.21969#bib.bib17): An emotional analogue of Chain\-of\-Thought prompting, which first predicts the seeker’s emotion, followed by strategy and response generation guided by that emotion\.
- •Skeleton of Thoughts\(SoT\)[29](https://arxiv.org/html/2608.21969#bib.bib19): Generates a “skeleton” or outline of key points, and then completes each point in parallel to form the final response\.
- •Tree of Thoughts\(ToT\)[48](https://arxiv.org/html/2608.21969#bib.bib20): Decomposes complex problems into a tree of intermediate reasoning steps\. The model explores and evaluates branches to select the best reasoning path\.
- •Plan\-and\-Solve \(PS\)[43](https://arxiv.org/html/2608.21969#bib.bib21): First generates a structured plan of subgoals or reasoning intentions, then solves each step in sequence, especially useful for multi\-turn decision making\.
- •FSM[44](https://arxiv.org/html/2608.21969#bib.bib11): A prompt\-based method that guides the model through a finite state machine, where transitions are triggered by current dialogue context, enabling discrete strategy control\.

#### Finetuning\-based Methods\.

- •SFT: Supervised finetuning on the training set with annotated strategy labels and response\.
- •ECoT\-SFT: The finetuned version of ECoT\.
- •EmoFSM[53](https://arxiv.org/html/2608.21969#bib.bib10): The finetuned version of FSM on ESC tasks\.

#### Dataset\-specific Methods\.

- •Guyu[17](https://arxiv.org/html/2608.21969#bib.bib46): A BERT\-based encoder\-decoder model finetuned on EmpatheticDialogues with emotion and persona conditioning\.
- •MoGPT[6](https://arxiv.org/html/2608.21969#bib.bib23): A transformer\-based model specifically trained on DailyDialog for multi\-turn response generation\.
- •MPB\-ESC[10](https://arxiv.org/html/2608.21969#bib.bib16): A large\-scale model built for the emotional support conversation benchmark using multi\-perspective modeling and emotional strategy classification\.
- •DialoGPT[52](https://arxiv.org/html/2608.21969#bib.bib22): A pretrained generative dialogue model evaluated on ESConv with task\-specific tuning\. We report numbers from the original paper\.

#### Reinforcement Learning\-based Methods\.

- •DQN: A Deep Q\-Network that learns a policy over discrete response strategies using estimated action\-values as feedback\.
- •PPO: Proximal Policy Optimization is used to optimize response selection via a reward model based on emotional alignment and coherence\.
- •ArCHer[55](https://arxiv.org/html/2608.21969#bib.bib44): A hierarchical RL method that learns to generate and execute high\-level strategies via two\-stage training with simulated user feedback\.
- •DAT[16](https://arxiv.org/html/2608.21969#bib.bib31): A method that equips pretrained language models with a planner that treats each utterance as an action, enabling goal\-directed multi\-turn dialogue by training a small planner to steer generation and significantly improving performance in social simulations and strategic tasks\.

For dataset\-specific baselines, we directly report results from the original papers\. All other methods are re\-implemented using the same model backbone \(unless otherwise noted\) to ensure fair comparison\. Hyperparameter configurations and additional training details are provided in our code repository\.

## Appendix ETheoretical Proofs

#### Connections between hierarchical MDP to SMDP\.

As shown in Section[2](https://arxiv.org/html/2608.21969#S2), an MDP can be defined asM≐\(𝒮,𝒜,ℛ,𝒯,γ\)M\\doteq\(\\mathcal\{S\},\\mathcal\{A\},\\mathcal\{R\},\\mathcal\{T\},\\gamma\)and consider episodic tasks\. In our work, we extend this traditional MDP to a hierarchical, two\-level MDP, which may also be considered as a Semi\-Markov decision processes \(SMDP\) from the perspective of a single unified time scale in previous literature[41](https://arxiv.org/html/2608.21969#bib.bib38)\.

The formal MDP definition is generalized for SMDP as follows: An additional elementℱ\\mathcal\{F\}is added to the standard MDP components, resulting inSMDP\\operatorname\{SMDP\}as\(𝒮,𝒜,𝒯,ℛ,γ,ℱ\)\(\\mathcal\{S\},\\mathcal\{A\},\\mathcal\{T\},\\mathcal\{R\},\\gamma,\\mathcal\{F\}\)\.ℱ\\mathcal\{F\}is a function that defines the cumulative probability distribution over the number of time steps until the next controlled state\. The probability that the next controlled state has been reached by timettwhen actionaais taken in statessis written:ℱ⁡\(t∣s,a\)\\mathcal\{F\}\(t\\mid s,a\)\. It is assumed that the distribution over the number of time steps taken does not change on subsequent trials ofaainss\. In our work,ℱ⁡\(t∣s,a\)=L\\mathcal\{F\}\(t\\mid s,a\)=L, whereLLdenotes the max length of low\-level output\.

To obtain our theoretical conclusion based on previous derivations, we re\-formularize our high\-level MDP under the SMDP framework, where the Bellman equation becomes:

V∗​\(s\)=\\displaystyle V^\{\*\}\(s\)=maxa∈Aℛ\(s,a\)\+∑s′∈S\\displaystyle\\max\_\{a\\in A\}\\mathcal\{R\}\(s,a\)\+\\sum\_\{s^\{\\prime\}\\in S\}𝒯⁡\(s,a,s′\)​∫0∞γt​V∗​\(s′\)​ℱ​\(t∣s,a\)​𝑑t\\displaystyle\\mathcal\{T\}\\left\(s,a,s^\{\\prime\}\\right\)\\int\_\{0\}^\{\\infty\}\\gamma^\{t\}V^\{\*\}\\left\(s^\{\\prime\}\\right\)\\mathcal\{F\}\(t\\mid s,a\)dt\(22\)For simplicity of derivation, here we usess,aato representsHs^\{\\text\{H\}\}andaHa^\{\\text\{H\}\}\. Maximizing ofV∗​\(s\)V^\{\*\}\(s\)then produce the optimal strategy\.

Since the state values are essentially constants when the integral is computed,V∗​\(s′\)V^\{\*\}\(s^\{\\prime\}\)can be pulled out of the integral, yielding:

V∗​\(s\)=maxa∈A⁡ℛ⁡\(s,a\)\+Γ​∑s′∈S𝒯⁡\(s,a,s′\)​V∗​\(s′\)\\displaystyle V^\{\*\}\(s\)=\\max\_\{a\\in A\}\\mathcal\{R\}\(s,a\)\+\\Gamma\\sum\_\{s^\{\\prime\}\\in S\}\\mathcal\{T\}\\left\(s,a,s^\{\\prime\}\\right\)V^\{\*\}\\left\(s^\{\\prime\}\\right\)whereΓ=∫0∞γt​ℱ​\(t∣s,a\)​𝑑t\\Gamma=\\int\_\{0\}^\{\\infty\}\\gamma^\{t\}\\mathcal\{F\}\(t\\mid s,a\)dtand is interpreted as a discount rate that varies with the state and action\. The dynamic programming operator for a particular action is then:

Ja​\(s\)=ℛ⁡\(s,a\)\+Γ​∑s′∈S𝒯⁡\(s,a,s′\)​V​\(s′\)J\_\{a\}\(s\)=\\mathcal\{R\}\(s,a\)\+\\Gamma\\sum\_\{s^\{\\prime\}\\in S\}\\mathcal\{T\}\\left\(s,a,s^\{\\prime\}\\right\)V\\left\(s^\{\\prime\}\\right\)
and in general:

J⁡\(s\)=maxa⁡\[ℛ⁡\(s,a\)\+Γ​∑s′∈S𝒯⁡\(s,a,s′\)​V​\(s′\)\]J\(s\)=\\max\_\{a\}\\left\[\\mathcal\{R\}\(s,a\)\+\\Gamma\\sum\_\{s^\{\\prime\}\\in S\}\\mathcal\{T\}\\left\(s,a,s^\{\\prime\}\\right\)V\\left\(s^\{\\prime\}\\right\)\\right\]

#### Proof of Theorem[1](https://arxiv.org/html/2608.21969#Thmtheorem1)\.

Based on previous definitions, we first introduce Lemma[1](https://arxiv.org/html/2608.21969#Thmlemma1)from[41](https://arxiv.org/html/2608.21969#bib.bib38), which is originally proposed in \(Parr et al\., Hierarchical Control and Learning for Markov Decision Processes, Ph\.D Dissertation, 1998\)\.

###### Lemma 1\.

The SMDP Q\-learning rule converges to the optimalQ∗​\(s,a\)Q^\{\*\}\(s,a\)values if

1. 1\.The state and action spaces are finite\.
2. 2\.∑iαi​\(s,a\)=∞\\sum\_\{i\}\\alpha\_\{i\}\(s,a\)=\\inftyand∑iαi2​\(s,a\)<∞\\sum\_\{i\}\\alpha\_\{i\}^\{2\}\(s,a\)<\\inftyuniformly overssand a w\.p\.1\.
3. 3\.Var⁡\{r\}\\operatorname\{Var\}\\\{r\\\}is finite\.
4. 4\.0<maxs,a⁡Γ<10<\\max\_\{s,a\}\\Gamma<1\.

Then we prove Theorem[1](https://arxiv.org/html/2608.21969#Thmtheorem1)based on Lemma[1](https://arxiv.org/html/2608.21969#Thmlemma1):

###### Proof of Theorem[1](https://arxiv.org/html/2608.21969#Thmtheorem1)\.

Our work satisfies all of the above conditions, thereby allowing us to establish Theorem[1](https://arxiv.org/html/2608.21969#Thmtheorem1)\. The justifications are as follows:

1. 1\.According to the assumptions in our setting, the state space \(e\.g\., text sequences\) has a bounded length, and is therefore finite\. Similarly, the action space is also finite\.
2. 2\.The second condition is part of our basic assumption\.
3. 3\.The reward is an integer in the range of 0 to 5, which implies that its variance is finite\.
4. 4\.The discount factorγ\\gammais explicitly defined in our framework and clearly satisfies0<maxs,a⁡Γ<10<\\max\_\{s,a\}\\Gamma<1\.

Thus, the theorem is proved\. ∎

## Appendix FMore Experimental Results

### F\.1Consistency between GPT4o and human scoring

The main driven component of the reward in ToSCA isrsatr^\{\\text\{sat\}\}, which is judged by an oracle model,i\.e\., GPT\-4o\. Comparing to the truly ‘golden’ evaluation,i\.e\., human evaluation, their consistency might be critical for the experimental conclusions of this paper\. To validate the reward mechanism, we conduct the correlation studies \(including the Spearman coefficient𝝆\\boldsymbol\{\\rho\}and the𝒑\\boldsymbol\{p\}\-value of correlation slopes\), with result shown in Table[12](https://arxiv.org/html/2608.21969#A6.T12)\.

Dimensions↓\\downarrow𝝆\\boldsymbol\{\\rho\}𝒑\\boldsymbol\{p\}Fluency0\.89\*\*\*Acceptance0\.85\*\*\*Effectiveness0\.81\*\*Sensitivity0\.91\*\*\*Emotion0\.88\*\*Satisfaction0\.86\*\*Table 12:Correlation between GPT\-4o and human evaluation scores\. For the significance level of𝒑\\boldsymbol\{p\}\-value, we use \*\* to denotep<0\.01p<0\.01, and \*\*\* to denotep<0\.001p<0\.001\.Table[12](https://arxiv.org/html/2608.21969#A6.T12)shows that GPT\-4o’s scores on six evaluation dimensions, including Fluency, Acceptability, Efficiency, Sensitivity, Emotion, and Satisfaction\. GPT4o scores exhibit significant correlations with human judgments, with the Spearman correlation coefficients all above 0\.8, and the significance level of𝒑\\boldsymbol\{p\}all below 0\.01 or 0\.001\.

### F\.2Training Curves

Figure[5](https://arxiv.org/html/2608.21969#A6.F5)shows the training loss curve of ToSCA on DailyDialog\. TheℒQL\\mathcal\{L\}\_\{Q^\{L\}\}decreases the fastest and remains stable, and bothℒQH\\mathcal\{L\}\_\{Q^\{H\}\}andℒπL\\mathcal\{L\}\_\{\\pi^\{L\}\}decrease with fluctuations\. The KL divergence and intrinsic motivation value increase during the initial stages but are kept within a certain range\. Importantly,rs​a​tr^\{sat\}gradually increases throughout the training process\.

Figure 5:Training curves of ToSCA on the DailyDialog, including losses \(denoted byℒ\\mathcal\{L\}\) and rewards \(denoted byrr\)\.
### F\.3Human Evaluation Results

The results of human evaluations, as shown in Table[13](https://arxiv.org/html/2608.21969#A6.T13), confirm that ToSCA’s responses align with human preferences better than baselines\. To validate the inter\-agreement of different human annotators, we also calculate the Cohen’s Kappa, with result of 0\.47\.

MethodHuman AnnotationFluencyAcceptanceEffectivenessSensitivityEmotionSatisfactionDirect2\.95 ± 1\.412\.60 ± 1\.152\.40 ± 0\.922\.70 ± 1\.083\.00 ± 1\.342\.60 ± 1\.41\+ Direct\-Refine3\.09 ± 1\.252\.73 ± 1\.222\.91 ± 1\.412\.91 ± 1\.233\.09 ± 1\.162\.84 ± 1\.40\+ Self\-Refine3\.10 ± 1\.292\.80 ± 1\.192\.70 ± 1\.142\.90 ± 1\.033\.15 ± 1\.382\.80 ± 1\.20\+ ECoT3\.08 ± 1\.022\.83 ± 1\.272\.67 ± 1\.063\.00 ± 1\.273\.08 ± 1\.292\.83 ± 1\.10\+ FSM3\.30 ± 1\.322\.90 ± 1\.172\.90 ± 1\.033\.00 ± 1\.272\.93 ± 1\.193\.00 ± 1\.25SFT3\.15 ± 1\.442\.70 ± 1\.192\.70 ± 1\.202\.90 ± 1\.243\.40 ± 1\.302\.90 ± 1\.32\+ CoT3\.67 ± 1\.213\.22 ± 1\.253\.67 ± 1\.263\.56 ± 1\.133\.61 ± 1\.173\.45 ± 1\.31\+ FSM3\.30 ± 1\.322\.90 ± 1\.172\.90 ± 1\.033\.00 ± 1\.273\.55 ± 1\.163\.65 ± 1\.19\+ ToSCA\(ours\)3\.61 ± 1\.263\.86 ± 0\.953\.72 ± 1\.043\.73 ± 1\.143\.98 ± 1\.213\.91 ± 1\.06Table 13:Averaged Human evaluation of response quality on DailyDialog and ESConv\.
### F\.4Results with More Data\-Specific Baselines

Table[14](https://arxiv.org/html/2608.21969#A6.T14)and[15](https://arxiv.org/html/2608.21969#A6.T15)provide more completed results on DailyDialog and ESConv, with the inclusion of data\-specific methods\.

MethodsAcc↑\\uparrowMaF1↑\\uparrowb​i​a​sbias↓\\downarrowB\-2↑\\uparrowR\-L↑\\uparrowD\-2↑\\uparrowDirect52\.6018\.031\.663\.3510\.3344\.74\+ Direct\-Refine48\.2728\.280\.702\.568\.743\.67\+ Self\-Refine[24](https://arxiv.org/html/2608.21969#bib.bib18)49\.7622\.151\.182\.47\.7534\.01\+ ECoT[20](https://arxiv.org/html/2608.21969#bib.bib17)38\.9429\.990\.271\.786\.055\.26\+ SoT[29](https://arxiv.org/html/2608.21969#bib.bib19)N/AN/AN/A2\.537\.9759\.98\+ ToT[48](https://arxiv.org/html/2608.21969#bib.bib20)N/AN/AN/A2\.528\.8443\.19\+ PS[43](https://arxiv.org/html/2608.21969#bib.bib21)N/AN/AN/A2\.607\.7639\.73\+ FSM[44](https://arxiv.org/html/2608.21969#bib.bib11)46\.8621\.221\.32\.79\.4438\.75SFT60\.1944\.820\.826\.8118\.5243\.36\+ECoT60\.1144\.90\.666\.6118\.0742\.87\+FSM[53](https://arxiv.org/html/2608.21969#bib.bib10)60\.0346\.020\.555\.8521\.7747\.43Guyu \+ BigLM\-12[17](https://arxiv.org/html/2608.21969#bib.bib46)\-\-\-14\.56\-44\.31Guyu \+ BigLM\-24[17](https://arxiv.org/html/2608.21969#bib.bib46)\-\-\-2\.93\-61\.74MoGPT \+ VanillaGPT\-2\(354M\)[6](https://arxiv.org/html/2608.21969#bib.bib23)\-\-\-\-20\.8\-DQN54\.0150\.100\.624\.1813\.0959\.27DAT[16](https://arxiv.org/html/2608.21969#bib.bib31)N/AN/AN/A3\.4511\.800\.90ArCher[55](https://arxiv.org/html/2608.21969#bib.bib44)50\.4142\.670\.215\.1714\.3555\.16ToSCA\(ours\)63\.6458\.910\.6316\.3535\.2262\.67Table 14:Result comparisons on DailyDialog, including dataset\-specific methods\. The best results of each LLMs arebolded\.
N/A: method not available for the specific metric\.
\-: not reported in the original work\.MethodsAcc↑\\uparrowMaF1↑\\uparrowb​i​a​sbias↓\\downarrowB\-2↑\\uparrowR\-L↑\\uparrowD\-2↑\\uparrowDirect11\.8010\.261\.613\.4710\.6433\.45\+ Direct\-Refine17\.0811\.071\.273\.106\.1314\.22\+ Self\-Refine[24](https://arxiv.org/html/2608.21969#bib.bib18)17\.5813\.611\.923\.349\.7114\.61\+ ECoT[20](https://arxiv.org/html/2608.21969#bib.bib17)15\.3210\.381\.693\.1610\.5030\.38\+ SoT[29](https://arxiv.org/html/2608.21969#bib.bib19)N/AN/AN/A3\.078\.7626\.15\+ ToT[48](https://arxiv.org/html/2608.21969#bib.bib20)N/AN/AN/A2\.659\.8114\.2\+ PS[43](https://arxiv.org/html/2608.21969#bib.bib21)N/AN/AN/A2\.818\.2719\.12\+ FSM[44](https://arxiv.org/html/2608.21969#bib.bib11)17\.3711\.150\.814\.1211\.8335\.43SFT32\.4321\.291\.286\.9716\.5950\.45\+ECoT30\.8017\.701\.356\.5115\.0034\.96\+FSM[53](https://arxiv.org/html/2608.21969#bib.bib10)28\.0023\.700\.415\.8815\.3051\.48DialoGPT\-Vanilla[23](https://arxiv.org/html/2608.21969#bib.bib14)\-\-\-5\.1315\.26\-DialoGPT\-Joint[23](https://arxiv.org/html/2608.21969#bib.bib14)\-\-\-5\.0015\.09\-DialoGPT\-Oracle[23](https://arxiv.org/html/2608.21969#bib.bib14)\-\-\-5\.5215\.82\-MPB\-ESC \+ LLaMA2\-7B[10](https://arxiv.org/html/2608.21969#bib.bib16)\-13\.730\.774\.9813\.0934\.74MPB\-ESC \+ LLaMA2\-70B[10](https://arxiv.org/html/2608.21969#bib.bib16)\-14\.550\.476\.1514\.2930\.95MPB\-ESC \+ Vicuna\-13B[10](https://arxiv.org/html/2608.21969#bib.bib16)\-12\.850\.746\.5514\.4324\.15MPB\-ESC \+ Mistral\-7B[10](https://arxiv.org/html/2608.21969#bib.bib16)\-12\.230\.714\.7212\.9325\.36MPB\-ESC \+ Solar\-10\.7B[10](https://arxiv.org/html/2608.21969#bib.bib16)\-14\.170\.874\.7913\.5332\.36MPB\-ESC \+ Tulu\-70B[10](https://arxiv.org/html/2608.21969#bib.bib16)\-15\.930\.96\.913\.9423\.78MPB\-ESC \+ ChatGPT[10](https://arxiv.org/html/2608.21969#bib.bib16)\-13\.501\.386\.2715\.1836\.9MPB\-ESC \+ GPT4[10](https://arxiv.org/html/2608.21969#bib.bib16)\-18\.380\.906\.4715\.1836\.92DQN37\.6934\.570\.593\.5911\.7444\.14ArCher[55](https://arxiv.org/html/2608.21969#bib.bib44)24\.5019\.600\.505\.3013\.1054\.80ToSCA\(ours\)39\.2636\.850\.486\.9316\.2852\.42Table 15:Result comparisons on ESConv, including dataset\-specific methods\. The best results areboldedand the second best areunderlined\.
N/A: method not available for the specific metric\.
\-: not reported in the original work\.
### F\.5Detailed results of Strategies\.

On the DailyDialog test set, Table[16](https://arxiv.org/html/2608.21969#A6.T16)presents the performance of Direct and ToSCA across different dialogue strategies, which are grouped according to the ground\-truth labels\. It can be observed that ToSCA exhibits more balanced performance across strategies, without any notably low accuracy \(e\.g\., 9\.65\)\. Additionally, it can be seen that the "question" strategy tends to correspond to higher B\-2 scores and lower D\-2 scores\.

Table 16:Per\-strategy automatic metrics on DailyDialog\.StrategyAcc↑\\uparrowMaF1↑\\uparrowb​i​a​s↓bias\\downarrowB\-2↑\\uparrowR\-L↑\\uparrowD\-2↑\\uparrowDirectDir\.18\.4720\.350\.814\.4012\.7374\.96Inf\.41\.3652\.510\.4514\.4729\.8870\.56Que\.57\.6159\.130\.6220\.0939\.5966\.28Com\.9\.656\.013\.954\.2513\.0776\.18ToSCADir\.54\.1855\.830\.6614\.2531\.9777\.34Inf\.65\.6469\.890\.5717\.1838\.1571\.52Que\.58\.6362\.290\.5619\.8140\.6765\.18Com\.62\.1140\.732\.1917\.6534\.3470\.09

CharactersStrategyResponseRewardValueUser\-Good evening, Saliva\. What’s that wonderful aroma from your kitchen? What are you doing now?\-\-AssistantInformI am cooking now\!311\.7DirectQuestionBased on your question, it seems there is a pleasant aroma coming from the kitchen area\. Could you be referring to the scent of food preparation? Are you particularly interested in what is being cooked right now?28\.7ToSCAInformI’m cooking some fish with a coconut ginger marinade\. That aroma you smell is the ginger sizzling in the pan—it’s my favorite part\!515\.1User\-Are you good at cooking?\-\-AssistantCommissiveI have studied its skills recently at a training school\. I plan to run a restaurant, so I have to practise cooking\!410\.3DirectInformYes, I would say I have developed a solid foundation in culinary tasks over time, particularly due to frequent practice sessions and structured cooking routines\.47\.9ToSCACommissiveI’ve been practicing a lot lately because I want to open a small seafood restaurant someday\.411\.9User\-It is a good idea\.\-\-AssistantQuestionI am steaming fish in a pressure cooker\. Have you ever tried steamed fish?47\.4DirectQuestionAre you saying it’s a good idea because engaging in cooking activities can help improve your overall dietary quality and personal satisfaction with meals?24\.6ToSCAQuestionThank you\! Have you ever tried cooking fish at home? I’d love to hear how you prepare it\.59\.3User\-But the taste of steamed fish at that restaurant is not delicious\. The fish is not well\-done and too salty \.\-\-AssistantInformThat’s the case\. I think there is not enough time to steam it and soy sauce is too much\.44\.0DirectInformThat sounds like an unfortunate dining experience\. It’s possible that the restaurant staff didn’t allow enough time for proper steaming, or they added too much soy sauce during preparation\. You might consider exploring another restaurant next time to see if the taste is more acceptable\.33\.0ToSCADirectiveIt seems that restaurant’s style wasn’t really your thing\. Next time, try adding a slice of lemon and a little salt when you steam the fish — it’ll make the flavor much fresher\!55\.0

Table 17:Case of ToSCA in DailyDialog\.
### F\.6Per\-strategy Results from Prediction Pespective

Table[18](https://arxiv.org/html/2608.21969#A6.T18)presents automatic evaluation metrics grouped by predicted strategy labels rather than ground\-truth labels\. The overall trends are similar, withQuestionandInformstrategies showing relatively higher performance across metrics such as Acc, MaF1, and B\-2\. This indicates that precision and recall are consistent for these categories, reflecting that the model’s predictions align well with strategy patterns\.

StrategyAcc↑\\uparrowMaF1↑\\uparrowB\-2↑\\uparrowR\-L↑\\uparrowD\-2↑\\uparrowDirectDir\.32\.1438\.2212\.4325\.6666\.85Inf\.49\.8854\.2115\.9732\.1062\.75Que\.58\.9360\.1819\.8538\.7259\.92Com\.12\.4710\.056\.1214\.9069\.34ToSCADir\.55\.2056\.8013\.7529\.3270\.89Inf\.66\.4270\.2517\.0237\.5864\.10Que\.61\.8463\.1019\.9240\.0566\.75Com\.56\.7848\.6617\.4535\.2065\.66

Table 18:Per\-strategy automatic metrics on DailyDialog, grouped by predicted strategy\. Bias is omitted as it is defined only against ground\-truth labels\.
### F\.7Case analysis

#### Typical good cases\.

Table[17](https://arxiv.org/html/2608.21969#A6.T17)presents a cooking\-related daily dialogue\. Compared to the rigid style in the Direct group \(e\.g\., simple repetition of the user’s utterances\), ToSCA uses dialogue strategies more effectively to guide the user and sustain the topic\. GPT\-4o evaluation shows higher reward and long\-term value, indicating improved conversational quality\.

#### Bad case analysis\.

Despite the strong performance of ToSCA, some low\-quality outputs are inevitably observed, mainly due to two reasons\. First, the use of clearly inappropriate dialogue strategies leads to degraded conversational quality\. Second, hallucination phenomena arising from reinforcement learning training can also impair response fidelity\. As shown in Table[19](https://arxiv.org/html/2608.21969#A6.T19), when responding to a direction inquiry, theCommissivestrategy is evidently unsuitable, resulting in stiff and unnatural replies\. Even when the correctDirectivestrategy is selected, the model may still produce semantically unclear utterances \(e\.g\., “Upon completion…”\)\.

UserExcuse me\. Do you know where the Admiral Hotel is?AssistantThat’s over on Callaway Street, isn’t it?UserYes, but I can’t find Callaway Street\.case 1\(Commissive\) I promise to make sure you walk for four blocks and then find the hotel two blocks north\.case 2\(Directive\) Upon completion, initiate a northward movement for exactly two blocks to achieve destination acquisition\.

Table 19:Bad cases of ToSCA responses with wrong dialogue strategies or hallucinated response\.