Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging

# Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging
Source: [https://arxiv.org/html/2607.10428](https://arxiv.org/html/2607.10428)
Jinglan Gong1,2\*Jiefan Lu1\*Hewei Guo1\*Kehan Li1,3Zhiyuan Han1,2Jihang Jiang2 Wenwen Tong1🖂Lewei Lu1🖂 1SenseTime Research2University of Science and Technology of China3Tsinghua University

###### Abstract

Evaluating large language models \(LLMs\) as multi\-turn conversational partners requires probing capabilities that single\-turn benchmarks miss: persona consistency, evolving intent tracking, emotional dynamics, and goal completion across many turns\. We introduceEYT\-Bench, a human\-centered benchmark whose evaluation protocol is built around a*three\-party decoupled*design — a persona\-grounded user simulator, a target model that decouples intent perception from response generation, and an independent third\-party LLM judge with a configurable multi\-judge ensemble\. Personas are sampled from publicly released, human\-curated corpora — Nemotron\-Personas\-USA\(NVIDIA,[2025](https://arxiv.org/html/2607.10428#bib.bib39)\)and PersonaMem\-v2\(Jianget al\.,[2025](https://arxiv.org/html/2607.10428#bib.bib41)\)— rather than synthesised, eliminating a systematic source of LLM\-induced persona bias\. We additionally introduce two trajectory\-level metrics absent from prior work: an embedding\-based*intent\-drift*signal and a*final\-intent completion rate*\(FICR\) inspired byτ\\tau\-bench\(Yaoet al\.,[2024](https://arxiv.org/html/2607.10428#bib.bib12)\)\. Over a1717\-target×\\times200200\-dialogue evaluation, EYT\-Bench reveals four findings that previous benchmarks miss: \(i\) state\-of\-the\-art*closed\-*and*open\-source*models are statistically indistinguishable on subjective dimensions \(empathy / persona / anthropomorphism vary within≤0\.3\\leq 0\.3\), but separate by up to9×9\{\\times\}on objective intent\-tracking; \(ii\) reasoning \("thinking on"\) is a phase transition for objective tracking on long\-context personas \(\+0\.47\+0\.47–0\.500\.50latent\-intent accuracy on Gemma\-4\) but is essentially flat on subjective scores; \(iii\) persona format dominates trajectory spread — FICR saturates above0\.950\.95on Nemotron\-USA but spreads0\.53→0\.880\.53\\to 0\.88on PersonaMem\-v2; and \(iv\) the warm\-up effect is robust on all16/1716/17models \(one outlier, GPT\-5\.5, reverses the effect\), and aggregate rankings are stable across the warm\-up weightα∈\[0\.05,0\.15\]\\alpha\\\!\\in\\\!\[0\.05,0\.15\]\. A cross\-judge ablation withdeepseek\-v4\-proas a replacement judge confirms that target rankings and final\-intent satisfaction are preserved across judges\.

Enjoy Your Talk: A Human\-Centered Benchmark for Multi\-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging

Jinglan Gong1,2\*Jiefan Lu1\*Hewei Guo1\*Kehan Li1,3Zhiyuan Han1,2Jihang Jiang2Wenwen Tong1🖂Lewei Lu1🖂1SenseTime Research2University of Science and Technology of China3Tsinghua University

\{NoHyper\}††footnotetext:\*Equal contribution\.🖂Corresponding authors\.

## 1Introduction

Modern LLMs are no longer evaluated only on whether a single answer is correct: users now expect them to sustain coherent, persona\-aware, emotionally appropriate exchanges across many turns\(Miehlinget al\.,[2024](https://arxiv.org/html/2607.10428#bib.bib50); Liet al\.,[2025](https://arxiv.org/html/2607.10428#bib.bib51); Acikgozet al\.,[2025](https://arxiv.org/html/2607.10428#bib.bib49)\)\. In such long\-horizon interactions, a single\-turn metric is a poor proxy: models can drift from the persona, lose track of the user’s evolving intent, or accumulate small misalignments that only become visible several turns later\(Gooding and Grefenstette,[2025](https://arxiv.org/html/2607.10428#bib.bib47); Labanet al\.,[2025](https://arxiv.org/html/2607.10428#bib.bib25)\)\.

Existing multi\-turn benchmarks fall short of stress\-testing these behaviours along three axes \([Table˜1](https://arxiv.org/html/2607.10428#S1.T1)\)\.\(1\) Persona realism: many benchmarks generate personas with an LLM, introducing model\-specific bias and homogeneity\(Argyleet al\.,[2023](https://arxiv.org/html/2607.10428#bib.bib35)\)\.\(2\) Evaluation decoupling: a common pattern is to use the same model family as simulator, target and judge, which is known to inflate scores via self\-preference\(Zhenget al\.,[2023](https://arxiv.org/html/2607.10428#bib.bib13); Panicksseryet al\.,[2024](https://arxiv.org/html/2607.10428#bib.bib14)\)\.\(3\) Trajectory metrics: turn\-level label accuracy does not tell us whether the conversation*converges*on the user’s goal — a gap that recent goal\-oriented benchmarks likeτ\\tau\-bench\(Yaoet al\.,[2024](https://arxiv.org/html/2607.10428#bib.bib12)\)and process\-oriented frameworks like EMPA\(Zhanget al\.,[2026](https://arxiv.org/html/2607.10428#bib.bib6)\)explicitly target\.

Table 1:Coverage of four design axes across open\-domain multi\-turn dialogue benchmarks\.We proposeEYT\-Bench, a benchmark whose contributions directly respond to these gaps:

- •Two complementary public persona pools\.We sample500500\-record EN persona pools from public human\-curated sources — Nemotron\-Personas\-USA\(NVIDIA,[2025](https://arxiv.org/html/2607.10428#bib.bib39)\), a demographically grounded1818\-attribute schema; and PersonaMem\-v2\(Jianget al\.,[2025](https://arxiv.org/html/2607.10428#bib.bib41)\), extracted from long\-form real user–assistant interactions — and evaluate every model on both\. Treating the persona source as an experimental variable directly exposes how much of a “benchmark result” is an artefact of the persona format \([Table˜5](https://arxiv.org/html/2607.10428#S4.T5)\)\.
- •Three\-party decoupled evaluation\.The simulator, target and judge models are loaded from independent configs and constrained to be disjoint at the model\-family level, eliminating self\-preference confounds\(Panicksseryet al\.,[2024](https://arxiv.org/html/2607.10428#bib.bib14); Wanget al\.,[2025b](https://arxiv.org/html/2607.10428#bib.bib15)\)\. A cross\-judge ablation replaces the primary judge with a model from a third family \([Table˜14](https://arxiv.org/html/2607.10428#A10.T14)\)\.
- •Trajectory\-level objective metrics\.Beyond turn\-level intent / emotion accuracy we add \(a\) an embedding\-based*intent\-drift*measure in the spirit of EMPA\(Zhanget al\.,[2026](https://arxiv.org/html/2607.10428#bib.bib6)\), and \(b\) a*final\-intent completion rate*\(FICR\) adjudicated by the judge, an open\-domain analogue ofτ\\tau\-bench’s database\-state verification\(Yaoet al\.,[2024](https://arxiv.org/html/2607.10428#bib.bib12)\)\.
- •Turn\-aware weighting as a controlled ablation\.The warm\-up–weighted aggregation of prior work is reported alongside an unweighted version and a sensitivity sweep over the warm\-up weightα∈\{0\.05,0\.10,0\.15\}\\alpha\\\!\\in\\\!\\\{0\.05,0\.10,0\.15\\\}; we confirm cross\-model rankings are stable across the range \([Table˜8](https://arxiv.org/html/2607.10428#S4.T8)\)\.

##### Empirical highlights\.

A1717\-target×\\times22\-persona\-pool×\\times100100\-dialogue run uncovers four findings that previous benchmarks miss\. \(i\) Subjective Empathy / Persona / Anthropomorphism scores cluster tightly across closed\-source APIs \(Claude, Gemini, GPT\-5\.5, Doubao Seed\) and open\-source MoE / dense targets \(DeepSeek\-V4, Qwen3\.5, Gemma\-4\): the inter\-family spread is<0\.5<0\.5on a 0–5 scale, with onlygpt\-5\.5clearly separating \(<2\.0<2\.0on Empathy\)\. \(ii\) On objective tracking, the gap widens to9×9\\times:deepseek\-v4\-proand the thinking\-enabledgemma\-4\-31b/26bdominate \(≥0\.75\\geq 0\.75latent\-intent accuracy on PersonaMem\-v2\), while Doubao Seed and Qwen3\.5 fall to0\.080\.08–0\.150\.15\. \(iii\) Enabling reasoning is essentially a phase transition on long\-context PersonaMem\-v2 \(\+0\.47\+0\.47on Lat\. accuracy for Gemma\-4\-31B\) but provides only marginal gains on the shorter\-context Nemotron pool\. \(iv\) FICR saturates on Nemotron\-USA \(≥0\.95\\geq 0\.95for every closed\-source model exceptgpt\-5\.5\) yet spreads cleanly on PersonaMem\-v2 \(0\.53→0\.880\.53\\to 0\.88\), making the PersonaMem\-v2 trajectory the more discriminative signal\.

## 2Related Work

##### Objective dialogue evaluation\.

Classic surface\-level metrics \(BLEU, ROUGE, perplexity\)\(Papineniet al\.,[2002](https://arxiv.org/html/2607.10428#bib.bib53); Lin,[2004](https://arxiv.org/html/2607.10428#bib.bib54)\)fail in open\-ended chat\. Task\-oriented benchmarks rely on database\-state matching\(Sunet al\.,[2024](https://arxiv.org/html/2607.10428#bib.bib55); Abdulhaiet al\.,[2025](https://arxiv.org/html/2607.10428#bib.bib56); Jiaet al\.,[2025](https://arxiv.org/html/2607.10428#bib.bib57)\), which is precise but limited to closed domains\. Recent work on*goal\-oriented*agent evaluation, notablyτ\\tau\-bench\(Yaoet al\.,[2024](https://arxiv.org/html/2607.10428#bib.bib12)\), scores agents on whether the final database / world state matches a ground\-truth annotation; we borrow this principle for our FICR metric while adapting it to open\-domain personal\-support scenarios where world state is replaced by a judge\-adjudicated goal\.

##### LLM\-as\-judge\.

Zhenget al\.\([2023](https://arxiv.org/html/2607.10428#bib.bib13)\); Fuet al\.\([2024](https://arxiv.org/html/2607.10428#bib.bib17)\); Gaoet al\.\([2025](https://arxiv.org/html/2607.10428#bib.bib22)\)established LLM\-as\-judge as a viable scalable proxy for human evaluation\. Subsequent work documented systematic biases — self\-preference\(Panicksseryet al\.,[2024](https://arxiv.org/html/2607.10428#bib.bib14)\), position bias\(Wanget al\.,[2025b](https://arxiv.org/html/2607.10428#bib.bib15),[2023](https://arxiv.org/html/2607.10428#bib.bib16)\), and verbosity bias — and proposed multi\-judge ensembles\(Sunet al\.,[2025](https://arxiv.org/html/2607.10428#bib.bib21)\)together with rubric\-grounded reasoning prompts\(Zhanget al\.,[2024](https://arxiv.org/html/2607.10428#bib.bib19); Laskaret al\.,[2025](https://arxiv.org/html/2607.10428#bib.bib20)\)as mitigations\. We adopt a strong CoT\-enabled judge \(Gemini\-3\.1\-Pro\-Thinking\) and quantify the residual bias with a cross\-judge ablation \([Table˜14](https://arxiv.org/html/2607.10428#A10.T14)\) that replaces the judge with a different\-family model\.

##### User simulation\.

Rule\-based user simulators\(Schatzmannet al\.,[2007](https://arxiv.org/html/2607.10428#bib.bib32); Liet al\.,[2016](https://arxiv.org/html/2607.10428#bib.bib33)\)are deterministic; LLM\-based simulators\(Filippaset al\.,[2024](https://arxiv.org/html/2607.10428#bib.bib26); Wuet al\.,[2025](https://arxiv.org/html/2607.10428#bib.bib27); Changet al\.,[2025](https://arxiv.org/html/2607.10428#bib.bib28); Suhet al\.,[2025](https://arxiv.org/html/2607.10428#bib.bib29)\)are more diverse but tend to be cooperatively “polite”\(Zhonget al\.,[2025](https://arxiv.org/html/2607.10428#bib.bib30); Wanget al\.,[2025a](https://arxiv.org/html/2607.10428#bib.bib31); Herlihyet al\.,[2024](https://arxiv.org/html/2607.10428#bib.bib36)\)\. Recent process\-oriented evaluation \(\(Zhanget al\.,[2026](https://arxiv.org/html/2607.10428#bib.bib6)\)\) and multi\-challenge benchmarks\(Deshpandeet al\.,[2025](https://arxiv.org/html/2607.10428#bib.bib7); Baiet al\.,[2024](https://arxiv.org/html/2607.10428#bib.bib4); Denget al\.,[2025](https://arxiv.org/html/2607.10428#bib.bib5)\)push simulators to model resistance, emotional dynamics\(Hanet al\.,[2026b](https://arxiv.org/html/2607.10428#bib.bib2),[a](https://arxiv.org/html/2607.10428#bib.bib3)\), and shifting intents, while omni\-modal models extend multi\-turn interaction to audio\-visual settings\(Tonget al\.,[2025](https://arxiv.org/html/2607.10428#bib.bib1)\)\. Our simulator follows that line and additionally exposes a*final\-intent*target so the simulator can assess its own goal\-attainment progress per turn, enabling early termination and the FICR metric\.

##### Persona corpora\.

Public human\-curated persona resources include PersonaChat / ConvAI2\(Zhanget al\.,[2018](https://arxiv.org/html/2607.10428#bib.bib37)\), the DMT\-RoleBench mixture\(Yuanet al\.,[2025](https://arxiv.org/html/2607.10428#bib.bib8)\), the synthesised\-but\-large PersonaHub corpus\(Ge and others,[2024](https://arxiv.org/html/2607.10428#bib.bib38)\), the demographically\-rich Nemotron\-Personas\(NVIDIA,[2025](https://arxiv.org/html/2607.10428#bib.bib39)\), and the long\-context PersonaMem corpus\(Jianget al\.,[2025](https://arxiv.org/html/2607.10428#bib.bib41)\)\. We deliberately combine the two human\-curated corpora that capture different facets of persona \(structured demographics versus distilled conversational voice\) to expose the persona format as an experimental variable\.

## 3EYT\-Bench

### 3\.1Overview

![Refer to caption](https://arxiv.org/html/2607.10428v1/x1.png)Figure 1:EYT\-Bench framework\. ① generates a persona\-conditioned*ChatSEED*\(persona, topic, initial emotion, explicit / latent / final intent\)\. ② The target model predicts user labels \(perception stage\) and generates a response \(generation stage\) from independent prompts\. ③ An LLM user simulator emits the next turn with per\-turn final\-intent progress\. ④ A third\-party judge \(single or multi\-judge ensemble\) scores every turn against a five\-sub\-dimension rubric and adjudicates final\-intent completion\.EYT\-Bench composes three independent agents \([Figure˜1](https://arxiv.org/html/2607.10428#S3.F1)\):

1. 1\.User Simulator— emits a persona\-conditioned user turn together with a JSON annotation \{explicit intent,latent intent,emotion,final\-intent progress\}\.
2. 2\.Target Model— the model under evaluation\. At every turn it first*predicts*the user’s labels \(perception stage\) and then*generates*a response \(generation stage\); the two prompts are independent so the prediction rubric does not leak into the response distribution\.
3. 3\.Judge— an independent third\-party LLM that scores every turn against a five\-sub\-dimension rubric on each of \{empathy, persona alignment, anthropomorphic interaction\} and adjudicates final\-intent completion\. The framework accepts either a single judge or a multi\-judge ensemble; the main results use a single Gemini\-3\.1\-Pro\-Thinking judge and a cross\-judge ablation \([Table˜14](https://arxiv.org/html/2607.10428#A10.T14)\)\.

The three roles are required by config to be disjoint at the model\-family level, removing a major source of self\-preference bias\(Panicksseryet al\.,[2024](https://arxiv.org/html/2607.10428#bib.bib14)\)\.

### 3\.2Persona Pool Construction

We sample two complementary500500\-record EN persona pools from publicly available, human\-curated sources and treat the choice of pool as an experimental variable\.

##### Pool A — Nemotron\-Personas\-USA\.

A500500\-row stratified sample from the Nemotron\-Personas corpus\(NVIDIA,[2025](https://arxiv.org/html/2607.10428#bib.bib39)\), restricted tocountry == "United States"so that the1818\-attribute demographic schema \(age, sex, occupation group, marital status, education, race/ethnicity, Big\-5 personality vector, etc\.\) is fully populated\. Records are deduplicated by cosine≥0\.85\\geq 0\.85onall\-MiniLM\-L6\-v2embeddings and stratified byoccupation\_group×\\timesage\_bucket\.

##### Pool B — PersonaMem\-v2\.

A500500\-row sample from PersonaMem\-v2\(Jianget al\.,[2025](https://arxiv.org/html/2607.10428#bib.bib41)\)\. Unlike Nemotron’s demographic schema, PersonaMem\-v2 personas are paragraphs distilled from long\-form real user–assistant interactions, so they preserve the conversational voice of an actual user rather than a structured demographic snapshot\.

##### Why two pools rather than a mixture\.

An LLM\-synthesised “baseline pool” would mix two confounded effects —*persona origin*\(synthetic vs\. human\) and*persona format*\(structured attributes vs\. free\-text\)\. By keeping the two human\-curated pools separate and evaluating each target under both, we read off directly how much of a model’s score is driven by the persona format \([Table˜5](https://arxiv.org/html/2607.10428#S4.T5)\)\.

![Refer to caption](https://arxiv.org/html/2607.10428v1/x2.png)Figure 2:ChatSEED examples\. Each sample bundles topic, persona, initial emotion, first query, composite explicit / latent intent and final intent into a single structured record so dialogue generation is deterministic conditional on the random seed\.

### 3\.3ChatSEED

A*ChatSEED*fixes a dialogue’s starting point and goal:persona,topic,initial emotion,initial explicit intent,initial latent intent,final intent\([Figure˜2](https://arxiv.org/html/2607.10428#S3.F2)\)\. The*final\-intent*field is a single\-sentence description of what the user must reach by the end of the conversation\. The simulator tracks its own progress toward that goal per turn, the judge adjudicates it post\-hoc, and the loop early\-stops when the simulator reports*achieved*for two consecutive turns\. To avoid silent template fallbacks we require the simulator’s final\-intent generator toraiseon parse failure; cache rows are tagged with\_errorfor later retry\.

### 3\.4Dialogue Generation

[Algorithm˜1](https://arxiv.org/html/2607.10428#alg1)summarises the loop\. Decoupling perception \(label prediction\) from generation \(response\) inside the target model is critical: when both prompts are merged, the prediction rubric leaks into the response distribution and inflates the subjective interaction scores\.

Algorithm 1Multi\-Turn Dialogue Generation in EYT\-Bench \(single ChatSEED\)\.0:ChatSEED

ss; turn budget

TT; user simulator

ℳu\\mathcal\{M\}\_\{u\}; target model

ℳt\\mathcal\{M\}\_\{t\}; system promptsys; prediction prompt

PpP\_\{p\}
0:Dialogue trace

τ=\(C,\{l^t\},\{ltgt\},\{pt\}\)\\tau=\\bigl\(C,\\,\\\{\\hat\{l\}\_\{t\}\\\},\\,\\\{l^\{\\mathrm\{gt\}\}\_\{t\}\\\},\\,\\\{p\_\{t\}\\\}\\bigr\)
1:Initialise rolling context

C←∅C\\leftarrow\\emptysetand progress counter

k←0k\\leftarrow 0
2:Load opening user turn

\(q1,l1gt\)\(q\_\{1\},\\,l^\{\\mathrm\{gt\}\}\_\{1\}\)from

ss*ltgt=\(ite,gt,itl,gt,etgt\)l^\{\\mathrm\{gt\}\}\_\{t\}=\(i^\{e,\\mathrm\{gt\}\}\_\{t\},i^\{l,\\mathrm\{gt\}\}\_\{t\},e^\{\\mathrm\{gt\}\}\_\{t\}\)*

3:for

t=1,2,…,Tt=1,\\,2,\\,\\dots,\\,Tdo

4:

l^t←ℳt​\(C∪\{qt\};Pp\)\\hat\{l\}\_\{t\}\\leftarrow\\mathcal\{M\}\_\{t\}\\\!\\left\(C\\cup\\\{q\_\{t\}\\\};\\;P\_\{p\}\\right\)*perception over full history*

5:

rt←ℳt​\(C∪\{qt\};sys\)r\_\{t\}\\leftarrow\\mathcal\{M\}\_\{t\}\\\!\\left\(C\\cup\\\{q\_\{t\}\\\};\\;\\textsc\{sys\}\\right\)*generation, prompt\-disjoint from line 4*

6:Append

\(qt,rt\)\(q\_\{t\},r\_\{t\}\)to

CC
7:

\(qt\+1,lt\+1gt,pt\+1\)←ℳu​\(C;s\)\(q\_\{t\+1\},\\,l^\{\\mathrm\{gt\}\}\_\{t\+1\},\\,p\_\{t\+1\}\)\\leftarrow\\mathcal\{M\}\_\{u\}\(C;\\;s\)*next user turn and self\-reported progress*

8:

k←k\+1k\\leftarrow k\+1if

pt\+1=achievedp\_\{t\+1\}=\\textit\{achieved\}else

0
9:if

k≥2k\\geq 2then break*early stop on two consecutive achieved*

10:endfor

11:return

τ\\tau

### 3\.5Hybrid Evaluation Framework

#### 3\.5\.1Objective metrics

Turn\-level label accuracy\.Per turn we compare the simulator\-emitted gold labels against the target’s predictions for explicit intent \(1212classes\), latent intent \(88classes\), and emotion \(1515classes, expanded from the prior1010\-label dictionary to balance positive / negative valence; see[Appendix˜C](https://arxiv.org/html/2607.10428#A3)\)\.

#### 3\.5\.2Trajectory metrics

Turn\-level accuracy ignores whether the conversation*converges*on the user’s goal\. We add two complementary trajectory\-level signals\.

Intent drift\.For every turnttwe embed the gold intent descriptionitgti^\{\\text\{gt\}\}\_\{t\}and the predicted intent descriptioni^t\\hat\{i\}\_\{t\}and compute

Drifti=1N​∑t=1N\(1−cos⁡\(v​\(itgt\),v​\(i^t\)\)\),\\textsc\{Drift\}\_\{i\}=\\frac\{1\}\{N\}\\sum\_\{t=1\}^\{N\}\\bigl\(1\-\\cos\(v\(i^\{\\text\{gt\}\}\_\{t\}\),v\(\\hat\{i\}\_\{t\}\)\)\\bigr\),\(1\)wherev​\(⋅\)v\(\\cdot\)is a sentence\-transformer encoder\. We additionally report the Spearman correlation between turn index andcos⁡\(v​\(i^t\),v​\(final intent\)\)\\cos\(v\(\\hat\{i\}\_\{t\}\),v\(\\text\{final intent\}\)\)as a directional alignment signal in the spirit of EMPA\(Zhanget al\.,[2026](https://arxiv.org/html/2607.10428#bib.bib6)\)\.

Final\-intent completion rate \(FICR\)\.After the final turn the judge is asked, conditioned on the full transcript and the ChatSEED’s final\-intent text, to decide whether the assistant has meaningfully helped the user reach the stated goal, and to assign a11–55satisfaction score\. FICR is the resulting completion rate overNNdialogues \(majority vote when an ensemble is configured; the main results use a single judge\)\. This is the open\-domain analogue ofτ\\tau\-bench’s database\-state verification\(Yaoet al\.,[2024](https://arxiv.org/html/2607.10428#bib.bib12)\)\.

#### 3\.5\.3Subjective metrics

![Refer to caption](https://arxiv.org/html/2607.10428v1/x3.png)Figure 3:Three subjective dimensions, each decomposed into five sub\-indicators scored on a 3\-point Likert\{0,0\.5,1\}\\\{0,0\.5,1\\\}\. Empathy: recognition / attribution / resonance / response / support\. Persona alignment: style / value / culture / memory / security\. Anthropomorphic interaction: colloquial / emotional / flow / flexibility / rhythm\.The judge scores every turn on three dimensions decomposed into five sub\-indicators each \([Figure˜3](https://arxiv.org/html/2607.10428#S3.F3)\)\. Sub\-scores are on a 3\-point Likert\{0,0\.5,1\}\\\{0,0\.5,1\\\}— an upgrade from the binary\{0,1\}\\\{0,1\\\}used in prior work — giving better discriminative power while remaining additively interpretable\. Dimension scores are sums of the five sub\-scores, range\[0,5\]\[0,5\]\. The full rubric with anchor exemplars is in[Appendix˜D](https://arxiv.org/html/2607.10428#A4)\.

##### Turn\-aware weighting as a controlled ablation\.

FollowingGooding and Grefenstette \([2025](https://arxiv.org/html/2607.10428#bib.bib47)\), perceived interaction quality is disproportionately shaped by early turns\. We therefore report two aggregated subjective scores per dimension:

Scoreu\\displaystyle\\textsc\{Score\}^\{\\textsc\{u\}\}=1N​∑t=1T∑i∈𝒮tsi,t,\\displaystyle=\\tfrac\{1\}\{N\}\\sum\_\{t=1\}^\{T\}\\sum\_\{i\\in\\mathcal\{S\}\_\{t\}\}s\_\{i,t\},\(2\)Scorew\\displaystyle\\textsc\{Score\}^\{\\textsc\{w\}\}=1N​∑t=1Twt​∑i∈𝒮tsi,t,\\displaystyle=\\tfrac\{1\}\{N\}\\sum\_\{t=1\}^\{T\}w\_\{t\}\\sum\_\{i\\in\\mathcal\{S\}\_\{t\}\}s\_\{i,t\},\(3\)with the warm\-up scheme

w1=w2=α,wt=1−2​αT−2​for​t≥3,∑twt=1\.w\_\{1\}=w\_\{2\}=\\alpha,\\quad w\_\{t\}=\\tfrac\{1\-2\\alpha\}\{T\-2\}\\;\\text\{for\}\\;t\\geq 3,\\;\\sum\_\{t\}w\_\{t\}=1\.\(4\)The main results useTmax=10T\_\{\\max\}=10andα=0\.10\\alpha=0\.10\([Table˜2](https://arxiv.org/html/2607.10428#S3.T2)\)\. The simulator additionally emits a per\-turn final\-intent progress label∈\{not\_started,in\_progress,achieved\}\\in\\\{\\textit\{not\\\_started\},\\textit\{in\\\_progress\},\\textit\{achieved\}\\\}that triggers early termination after two consecutiveachievedreports\. The resultingTactualT\_\{\\text\{actual\}\}may range over\{2,…,Tmax\}\\\{2,\\ldots,T\_\{\\max\}\\\}, and we regenerate the weight vectorwtw\_\{t\}separately for each dialogue at its ownTactualT\_\{\\text\{actual\}\}rather than truncating a fixed\-length weight vector \(which would silently inflateα\\alpha\)\. A sensitivity sweep overα∈\{0\.05,0\.10,0\.15\}\\alpha\\in\\\{0\.05,0\.10,0\.15\\\}is reported in[Section˜4\.8](https://arxiv.org/html/2607.10428#S4.SS8)\.

Table 2:Concrete turn weightswtw\_\{t\}forT=10T\{=\}10andα=0\.10\\alpha\{=\}0\.10\(the configuration used in the main results\)\. The first two turns each carryα=0\.10\\alpha\{=\}0\.10; the remaining 8 turns share the residual mass1−2​α=0\.801\{\-\}2\\alpha\{=\}0\.80uniformly \(wt=0\.10w\_\{t\}\{=\}0\.10fort≥3t\\geq 3\)\. When a dialogue is early\-stopped \(the simulator reports*final\-intent achieved*for two consecutive turns\), weights are regenerated at the dialogue’sTactualT\_\{\\text\{actual\}\}so thatα\\alphastays semantically fixed\.

### 3\.6Cross\-Judge Bias Control

Simulator, judge and target are required disjoint at the model\-family level\. To quantify the residual judge\-side bias the framework supports both*ensemble*judging \(majority vote across≥2\\geq 2judges\) and a*cross\-judge*ablation in which the primary judge is swapped for a model from a third family\. We report the cross\-judge ablation in[Table˜14](https://arxiv.org/html/2607.10428#A10.T14)usingdeepseek\-v4\-proas the replacement judge\.

## 4Experiments

### 4\.1Setup

Slate\.We evaluate1717target models on100100dialogues per persona pool \(N=200N\{=\}200in total\) atTmax=10T\_\{\\max\}=10turns with early stop on two consecutiveachievedreports\. Closed\-source APIs are accessed through a unified gateway:claude\-opus\-4\.7,claude\-sonnet\-4\.6,gemini\-3\.1\-pro\-thinking,gemini\-3\-flash\-thinking,gpt\-5\.5,seed\-2\.0\-pro/mini/lite,deepseek\-v4\-pro/flash,qwen3\.5\-27b/35b\-a3b/397b\-a17b\. Open\-source Gemma\-4 checkpoints \(gemma\-4\-26b\-a4b,gemma\-4\-31b, each with a thinking\-on variant\) are served via vLLM\(Kwonet al\.,[2023](https://arxiv.org/html/2607.10428#bib.bib60)\)with FP8 weight\-and\-activation quantization on an A800\-80G cluster\. Full deployment details and per\-model tensor\-parallel sizing are in[Appendix˜F](https://arxiv.org/html/2607.10428#A6)\.

Simulator and primary judge\.Both the simulator and the primary judge aregemini\-3\.1\-pro\-preview\-thinkingwithreasoning\_effort = highandtemperature = 0\.0at the judge side\. Although simulator and judge share a base model, the simulator’s prompt is conditioned on the persona / ChatSEED rather than the rubric, and[Table˜14](https://arxiv.org/html/2607.10428#A10.T14)replaces the judge with a third\-family model \(deepseek\-v4\-pro\) to bound the residual self\-preference\. We use random seed2026052120260521throughout\.

Persona pools\.Pool A is a500500\-row stratified sample from Nemotron\-Personas\-USA\(NVIDIA,[2025](https://arxiv.org/html/2607.10428#bib.bib39)\); Pool B is a500500\-row sample from PersonaMem\-v2\(Jianget al\.,[2025](https://arxiv.org/html/2607.10428#bib.bib41)\)\. The first100100ChatSEEDs from each pool — persona, topic, initial emotion, explicit / latent / final intent — are cached so re\-runs are deterministic\.

### 4\.2Statistics of the Evaluation Slate

[Figure˜4](https://arxiv.org/html/2607.10428#S4.F4)summarises the data:200200ChatSEEDs,1717targets,\>30,000\>30\{,\}000scored turns,≥6,000\\geq 6\{,\}000judge\-adjudicated final\-intent verdicts\. Token counts per persona, topic\-tag distribution, and per\-bucket counts for both pools are in[Appendix˜B](https://arxiv.org/html/2607.10428#A2)\.

![Refer to caption](https://arxiv.org/html/2607.10428v1/x4.png)Figure 4:Persona, topic and emotion statistics across the two EYT\-Bench persona pools\.
### 4\.3Main Objective and Trajectory Results

Table 3:Objective metrics on EYT\-Bench \(N=100N\{=\}100per pool,Tmax=10T\_\{\\max\}\{=\}10\)\. For each pool we report per\-turn explicit\-intent accuracy \(Exp\.\), per\-turn latent\-intent accuracy \(Lat\.\), per\-turn emotion accuracy \(Emo\.\), the embedding intent drift \(Drift↓\\downarrow\), and the judge\-adjudicated final\-intent completion rate \(FICR\)\. Best per column inbold, second\-bestunderlined\.[Table˜3](https://arxiv.org/html/2607.10428#S4.T3)reports turn\-level latent\-intent / emotion accuracy, intent drift, and FICR for every target on both pools\. Two patterns dominate\.

Open vs\. closed parity collapses on objective tracking\.On Nemotron\-USA the spread between the best and median closed\-source model on latent\-intent accuracy is∼0\.15\\sim 0\.15, comparable to the spread between any two open\-source families\. On PersonaMem\-v2, however,deepseek\-v4\-pro\(0\.7990\.799\) and the thinking\-enabled Gemma\-4 variants \(0\.7700\.770,0\.7750\.775\) form a clear top tier, while the Doubao Seed and Qwen3\.5 families collapse to0\.080\.08–0\.150\.15— nearly10×10\\timesbehind\. The long\-context, free\-text PersonaMem input therefore acts as a discriminator that the structured Nemotron schema does not\.

FICR saturates on Nemotron\-USA\.11/1311/13API targets reachFICR≥0\.95\\text\{FICR\}\\\!\\geq\\\!0\.95on Nemotron; the only outlier isgpt\-5\.5at0\.7700\.770\. PersonaMem\-v2 spreads FICR from0\.5250\.525\(gpt\-5\.5\) to0\.8860\.886\(seed\-2\.0\-lite\), making it the more discriminative benchmark for trajectory\-level goal completion\. We treat Nemotron as a sanity check for*coverage*and PersonaMem as the primary lever for*discrimination*\.

### 4\.4Main Subjective Results

Table 4:Subjective dimensions on EYT\-Bench, rescaled to\[0,100\]\[0,100\]\.Emp\./Per\./Ant\.denote Empathy, Persona Alignment, and Anthropomorphic Interaction\. Best per column inbold, second\-bestunderlined\.[Table˜4](https://arxiv.org/html/2607.10428#S4.T4)reports the three subjective dimensions atα=0\.10\\alpha=0\.10\. Almost every closed\- and open\-source model lands in the band\[3\.45,3\.86\]\[3\.45,3\.86\]on Empathy,\[4\.42,4\.93\]\[4\.42,4\.93\]on Persona Alignment, and\[4\.20,4\.80\]\[4\.20,4\.80\]on Anthropomorphic interaction on Nemotron\-USA; PersonaMem\-v2 shifts the band downward but preserves ordering \(\[3\.15,3\.65\]\[3\.15,3\.65\]/\[4\.15,4\.76\]\[4\.15,4\.76\]/\[3\.13,4\.69\]\[3\.13,4\.69\]\)\.gpt\-5\.5is the single outlier \(<2\.0<2\.0Empathy on both pools\), driven by frequent agentic / refusal\-style turns\.seed\-2\.0\-miniis the lowest within its own family, especially on Anthropomorphic\. The dimension\-level spread is≥5×\\geq 5\\timestighter than the objective spread\.

### 4\.5Persona\-Source Comparison

Table 5:Family\-level comparison across the two persona pools\. Subjective scores \(Emp\.,Per\.,Ant\.\) are on\[0,100\]\[0,100\]; objectiveExp\.,Lat\.andDrift↓\\downarroware per\-turn explicit\-intent accuracy, latent\-intent accuracy and intent embedding drift;FICRis the judge\-adjudicated final\-intent completion rate\. Best per column inbold\.The persona format is a first\-order benchmark design lever \([Table˜5](https://arxiv.org/html/2607.10428#S4.T5)\)\. Aggregated at the family level, PersonaMem\-v2*narrows*subjective Empathy by at most0\.450\.45but moves objective latent\-intent accuracy by up to\+0\.45\+0\.45\(DeepSeek\-V4\) or−0\.20\-0\.20\(Doubao Seed, Qwen3\.5\)\. The flip is direct evidence that “which model is best” depends on the pool — PersonaMem\-v2 rewards long\-context reasoners \(DeepSeek, thinking Gemma\) and punishes models that key off persona attributes \(Qwen3\.5, Doubao Seed\)\.

### 4\.6Trajectory Metrics

Table 6:Trajectory metrics on PersonaMem\-v2 \(N=100N\{=\}100\)\.Sat\.is the judge\-assigned final\-intent satisfaction score, rescaled from11–55to\[0,100\]\[0,100\]\.On PersonaMem\-v2 the trajectory metrics rank the top tier sharply \([Table˜13](https://arxiv.org/html/2607.10428#A8.T13)\): the four lowest\-drift models —deepseek\-v4\-pro, both thinking\-enabled Gemma\-4 variants, anddeepseek\-v4\-flash— all sit atDrift≤0\.23\\textsc\{Drift\}\\leq 0\.23andFICR≥0\.76\\textsc\{FICR\}\\geq 0\.76\. The Doubao Seed and Qwen3\.5 cluster atDrift≥0\.62\\textsc\{Drift\}\\geq 0\.62but their FICR remains above0\.800\.80, illustrating the design hypothesis that drift and FICR capture*orthogonal*failure modes: a model can label intents poorly turn\-by\-turn yet still “arrive” at the user’s final goal\.

### 4\.7Thinking On/Off Asymmetry

Table 7:Reasoning on/off ablation on the open\-source Gemma\-4 family\. Subjective dimensions \(Emp\.,Per\.,Ant\.\) are on\[0,100\]\[0,100\];Exp\.,Lat\.andEmo\.are per\-turn explicit\-intent, latent\-intent and emotion accuracy\.Δ\\Deltarows report \(think−\-non\-think\); cells with\|Δ\|≥0\.3\|\\Delta\|\{\\geq\}0\.3on objective metrics are highlighted\.[Table˜7](https://arxiv.org/html/2607.10428#S4.T7)ablates reasoning \("thinking on"\) within the open\-source Gemma\-4 family\. On the long\-context PersonaMem\-v2, enabling reasoning lifts latent\-intent accuracy by\+0\.47\+0\.47–0\.500\.50and cuts drift by roughly0\.40\.4— a near phase transition\. On Nemotron\-USA the same switch buys only\+0\.08\+0\.08–0\.200\.20on latent\-intent\. Subjective dimensions are essentially unchanged \(\|Δ\|≤0\.10\|\\Delta\|\\leq 0\.10\), and reasoning even slightly hurts Empathy on the smaller Gemma\-4\-26B base\. We read this as: reasoning targets factual / intent tracking, not perceived interaction quality\. Together with §[4\.6](https://arxiv.org/html/2607.10428#S4.SS6)it suggests that PersonaMem\-v2 — not Nemotron — is the locus where reasoning gains are visible\.

### 4\.8Warm\-up Effect andα\\alphaSensitivity

Table 8:Warm\-up sensitivity for all1717targets\.Left:weighted Empathy \(Eq\.[4](https://arxiv.org/html/2607.10428#S3.E4)\) forα∈\{0\.05,0\.10,0\.15\}\\alpha\\in\\\{0\.05,0\.10,0\.15\\\}on each pool, on\[0,100\]\[0,100\]\.Right:warm\-up effect, early\-turn \(t≤2t\{\\leq\}2\) minus late\-turn \(t≥3t\{\\geq\}3\) raw Empathy on Nemotron\-Personas\-USA \(Nemo\.\) and PersonaMem\-v2 \(PM\); negativeΔ\\Deltaconfirms the warm\-up assumption\. GPT\-5\.5 is the only target with positiveΔ\\Deltaon both pools\.Nemo\. EmpathyPM EmpathyEarly−\-LateΔ\\DeltaModel\.05\.10\.15\.05\.10\.15Nemo\.PM*Closed\-source models*Claude\-Opus\-4\.775\.274\.273\.266\.265\.765\.2−8\.2\-8\.2−4\.6\-4\.6Claude\-Sonnet\-4\.676\.275\.274\.163\.463\.363\.2−9\.1\-9\.1−0\.3\-0\.3Gemini\-3\-Flash\-Thinking77\.676\.575\.472\.071\.069\.9−9\.7\-9\.7−9\.6\-9\.6Gemini\-3\.1\-Pro\-Thinking75\.174\.173\.168\.767\.967\.1−7\.8\-7\.8−7\.5\-7\.5GPT\-5\.543\.444\.144\.838\.339\.039\.7\+12\.6\+12\.6\+8\.5\+8\.5Seed\-2\.0\-Lite73\.172\.171\.069\.568\.467\.3−8\.8\-8\.8−10\.6\-10\.6Seed\-2\.0\-Mini67\.166\.265\.458\.958\.457\.9−5\.9\-5\.9−4\.3\-4\.3Seed\-2\.0\-Pro73\.572\.270\.970\.969\.868\.6−10\.0\-10\.0−11\.0\-11\.0*Open\-source models*DeepSeek\-V4\-Flash74\.072\.871\.569\.268\.467\.6−9\.3\-9\.3−7\.3\-7\.3DeepSeek\-V4\-Pro70\.970\.269\.568\.567\.566\.5−5\.7\-5\.7−9\.5\-9\.5Qwen3\.5\-27B80\.078\.777\.472\.972\.071\.0−12\.2\-12\.2−8\.7\-8\.7Qwen3\.5\-35B\-A3B80\.078\.877\.672\.070\.769\.4−10\.5\-10\.5−12\.1\-12\.1Qwen3\.5\-397B\-A17B79\.077\.576\.174\.973\.772\.4−14\.9\-14\.9−11\.8\-11\.8Gemma\-4\-26B\-A4B76\.975\.774\.568\.067\.066\.1−9\.8\-9\.8−9\.3\-9\.3Gemma\-4\-26B\-A4B\-Thinking74\.373\.171\.972\.170\.969\.7−11\.1\-11\.1−11\.5\-11\.5Gemma\-4\-31B75\.474\.273\.071\.069\.968\.8−10\.6\-10\.6−10\.3\-10\.3Gemma\-4\-31B\-Thinking74\.573\.472\.270\.369\.268\.0−11\.0\-11\.0−11\.0\-11\.0[Table˜8](https://arxiv.org/html/2607.10428#S4.T8)verifies the warm\-up assumption that motivates Eq\.[4](https://arxiv.org/html/2607.10428#S3.E4): early\-turn Empathy is uniformly lower than late\-turn Empathy \(Δ<0\\Delta<0\) on1616of the1717targets, with magnitudes−0\.01\-0\.01to−0\.74\-0\.74\. The only exception isgpt\-5\.5, whose early\-turn scores are*higher*than late\-turn scores \(\+0\.43\+0\.43to\+0\.63\+0\.63\); inspection confirms this is not a warm\-up failure but a quality degradation pattern — the model’s later turns are notably worse\. Persona Alignment exhibits the same direction; Anthropomorphic is mixed by family \(Gemma and Claude trend negative, Qwen3\.5 / Seed positive, see[Appendix˜K](https://arxiv.org/html/2607.10428#A11)\)\.

Sweepingα∈\{0\.05,0\.10,0\.15\}\\alpha\\in\\\{0\.05,0\.10,0\.15\\\}produces a monotone shift of0\.100\.10–0\.130\.13on weighted Empathy for the1616models withΔ<0\\Delta<0— i\.e\. heavier early\-turn weight pulls them down — and an opposite shift forgpt\-5\.5\. Crucially, no rank flips occur in either pool; the cross\-model ordering atα=0\.10\\alpha=0\.10is preserved at the two endpoints of the sweep\.

### 4\.9Cross\-Judge Ablation

Table 9:Cross\-judge ablation: primary judge replaced by the third\-family DeepSeek\-V4\-Pro on a stratified subsample \(n=46n\{=\}46paired dialogues,1111\(pool, model\) cells\)\. SubjectiveΔ\\Deltaare on\[0,100\]\[0,100\]; final\-intent satisfactionΔSat\.\\Delta\_\{\\textsc\{Sat\.\}\}is rescaled from11–55to\[0,100\]\[0,100\]\. The shift acts as a global calibration offset rather than re\-ordering targets: relative model rankings are preserved under both judges, andΔSat\.\\Delta\_\{\\textsc\{Sat\.\}\}is small in magnitude \(≤1\.8\\leq 1\.8on average\)\.Replacing the primary judge with a third\-family model \(deepseek\-v4\-pro\) shifts the absolute subjective scores systematically downward by0\.550\.55–0\.660\.66on the three Likert dimensions, consistent with documented cross\-judge calibration offsets\(Wanget al\.,[2025b](https://arxiv.org/html/2607.10428#bib.bib15); Sunet al\.,[2025](https://arxiv.org/html/2607.10428#bib.bib21)\)\. The relative ranking of targets within a pool is preserved \(e\.g\.seed\-2\.0\-miniis bottom anddeepseek\-v4\-flashis mid\-pack under both judges\)\. Critically, the final\-intent satisfaction signal is near\-identical across judges \(mean\|Δ\|=0\.07\|\\Delta\|\\\!=\\\!0\.07\), positioning it as the most cross\-judge\-stable scalar in EYT\-Bench and a natural anchor for cross\-paper comparison\.

### 4\.10Findings

##### Closed and open\-source models are nearly indistinguishable on subjective dimensions, but separate by up to9×9\\timeson objective tracking\.

claude\-opus/sonnet,gemini\-3\-pro/flash,qwen3\.5\-27/35/397B, andgemma\-4\-31Ball sit in\[3\.45,3\.86\]\[3\.45,3\.86\]on Empathy\. The same slate spreads from0\.090\.09to0\.800\.80on PersonaMem\-v2 latent\-intent accuracy\. Subjective Likert is therefore*not*the lever that separates the frontier of 2026 chat models — objective trajectory tracking is\.

##### Reasoning is a phase transition on long\-context PersonaMem\-v2\.

For both Gemma\-4 bases, “thinking on” lifts latent\-intent accuracy from∼0\.15\\sim 0\.15to∼0\.77\\sim 0\.77on PersonaMem and cuts drift from∼0\.57\\sim 0\.57to∼0\.17\\sim 0\.17\. On Nemotron the same switch delivers only\+0\.08\+0\.08–0\.200\.20Lat\. and−0\.07\-0\.07–−0\.12\-0\.12Drift\. We hypothesise that Nemotron’s structured demographic schema already exposes the latent intent label in the persona text — a form of label leakage — whereas PersonaMem’s long free\-text personas require genuine in\-context reasoning\.

##### Persona format dominates trajectory spread\.

FICR on Nemotron saturates \(≥0\.95\\geq 0\.95for every closed\-source model exceptgpt\-5\.5\); on PersonaMem\-v2 it spreads cleanly from0\.530\.53to0\.880\.88, with the four reasoning\-capable models clustering near the top\. We recommend reporting Nemotron FICR only as a coverage sanity check and treating PersonaMem\-v2 as the primary trajectory benchmark\.

##### The warm\-up effect is robust and ranking isα\\alpha\-stable\.

16/1716/17targets show negative Empathy / Persona\-AlignmentΔ\\Delta\(early−\-late\), validating the warm\-up assumption of Eq\.[4](https://arxiv.org/html/2607.10428#S3.E4)\.gpt\-5\.5is the sole inverter and reflects a genuine model quality regression across turns, not a warm\-up failure\. Sweepingα\\alphadoes not flip any ranking\.

##### Cross\-judge calibration applies but rankings are preserved\.

Replacing the judge with a third\-family model calibrates subjective Likert scores down by∼0\.55\\sim 0\.55–0\.660\.66but leaves the model ranking intact, and final\-intent satisfaction is near\-identical across judges\. We recommend treating subjective Likert as a within\-paper signal and final\-intent satisfaction as the appropriate cross\-paper anchor\.

## 5Conclusion

EYT\-Bench provides a config\-driven, three\-party\-decoupled implementation for human\-centered multi\-turn dialogue evaluation that is robust to the most common LLM\-as\-judge biases and quantifies whether a conversation actually converges on the user’s goal\. Across1717targets and200200dialogues, the benchmark reveals that subjective Likert metrics no longer separate frontier models, that the open\-source DeepSeek\-V4 and thinking\-enabled Gemma\-4 close or exceed the closed\-source gap on objective trajectory tracking, and that the persona format is itself a first\-order experimental lever\. We hope the released code, persona pools and judge prompts make it easy to extend EYT\-Bench to new languages, domains and metrics\.

## Limitations

First, the persona pools are EN\-only and centred on the US \(Nemotron\-Personas\-USA\) and English\-speaking online conversations \(PersonaMem\-v2\); cross\-cultural and non\-English evaluation is left to future work\. Second, although the slate covers1717targets, only the Gemma\-4 family contains paired thinking\-on/off variants, so the quantitative phase\-transition claim on reasoning is restricted to this family — extending the comparison to additional open\-source bases is left for follow\-up\. Third, the main results use a single Gemini\-3\.1\-Pro\-Thinking primary judge; the multi\-judge ensemble infrastructure is in place but full\-slate ensemble verdicts are left for the camera\-ready, and the cross\-judge ablation \([Table˜14](https://arxiv.org/html/2607.10428#A10.T14)\) is computed on a stratified4242\-pair subsample\. Fourth, FICR is adjudicated by an LLM, not by humans — a200200\-dialogue multi\-annotator study \([Appendix˜L](https://arxiv.org/html/2607.10428#A12)\) is scheduled\. Fifth, the simulator and primary judge share the Gemini\-3\.1\-Pro\-Thinking base;[Table˜14](https://arxiv.org/html/2607.10428#A10.T14)bounds the residual self\-preference at≤0\.66\\leq 0\.66Likert points but does not eliminate it\. Sixth, the1515\-class extended emotion taxonomy increases the difficulty of the emotion\-accuracy metric; we additionally report Macro\-F1 in[Appendix˜C](https://arxiv.org/html/2607.10428#A3)\. Seventh, at100100dialogues per pool, cross\-model subjective differences smaller than∼0\.05\\sim 0\.05and objective accuracy differences smaller than∼0\.03\\sim 0\.03should not be interpreted as significant\.

## Ethics Statement

EYT\-Bench targets evaluation, not deployment, and uses only pre\-existing public persona corpora released under permissive licences \(Nemotron\-Personas: CDLA\-Permissive\-2\.0; PersonaMem\-v2: research\-use, distilled from public dialogue interactions\)\. Persona attributes are kept generic; no personally identifying information is generated or released\. The judge may reflect biases of the underlying LLM provider, and we recommend treating EYT\-Bench numbers as one signal alongside human evaluation rather than as a sole quality metric\. To support responsible use we \(a\) release the full judge rubric with reasoning examples, \(b\) document the single\-judge versus ensemble configuration explicitly, and \(c\) quantify the residual judge bias with a cross\-judge ablation \([Table˜14](https://arxiv.org/html/2607.10428#A10.T14)\)\.

## References

- Consistently simulating human personas with multi\-turn reinforcement learning\.arXiv preprint arXiv:2511\.00222\.Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px1.p1.1)\.
- E\. C\. Acikgoz, C\. Qian, H\. Wang, V\. Dongre, X\. Chen, H\. Ji, D\. Hakkani\-Tür, and G\. Tur \(2025\)A desideratum for conversational agents: capabilities, challenges, and future directions\.arXiv preprint arXiv:2504\.16939\.Cited by:[§1](https://arxiv.org/html/2607.10428#S1.p1.1)\.
- L\. P\. Argyle, E\. C\. Busby, N\. Fulda, J\. R\. Gubler, C\. Rytting, and D\. Wingate \(2023\)Out of one, many: using language models to simulate human samples\.Political Analysis31\(3\),pp\. 337–351\.Cited by:[§1](https://arxiv.org/html/2607.10428#S1.p2.1)\.
- G\. Bai, J\. Liu, X\. Bu, Y\. He, J\. Liu, Z\. Zhou, Z\. Lin, W\. Su, T\. Ge, B\. Zheng,et al\.\(2024\)MT\-bench\-101: a fine\-grained benchmark for evaluating large language models in multi\-turn dialogues\.arXiv preprint arXiv:2402\.14762\.Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Chang, A\. Anderson, and J\. M\. Hofman \(2025\)ChatBench: from static benchmarks to human\-ai evaluation\.arXiv preprint arXiv:2504\.07114\.Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px3.p1.1)\.
- D\. Demszky, D\. Movshovitz\-Attias, J\. Ko, A\. Cowen, G\. Nemade, and S\. Ravi \(2020\)GoEmotions: a dataset of fine\-grained emotions\.InACL,Cited by:[Appendix C](https://arxiv.org/html/2607.10428#A3.SS0.SSS0.Px2.p1.2)\.
- Y\. Deng, G\. Hu, H\. Sun, X\. Zhang, H\. Zhang, F\. Tian, X\. Yang, G\. Yu, and E\. S\. Chng \(2025\)MULTI\-bench: a multi\-turn interactive benchmark for assessing emotional intelligence ability of spoken dialogue models\.arXiv preprint arXiv:2511\.00850\.Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px3.p1.1)\.
- K\. Deshpande, V\. Sirdeshmukh, J\. Mols, L\. Jin, E\. H\. Cardona, D\. Lee, J\. Kritz, W\. E\. Primack, S\. Yue, and C\. Xing \(2025\)MultiChallenge: a realistic multi\-turn conversation evaluation benchmark challenging to frontier llms\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 18632–18702\.Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Filippas, J\. J\. Horton, and B\. S\. Manning \(2024\)Large language models as simulated economic agents: what can we learn from homo silicus?\.InProceedings of the 25th ACM Conference on Economics and Computation \(EC ’24\),pp\. 614–615\.Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Fu, S\. Ng, Z\. Jiang, and P\. Liu \(2024\)GPTScore: evaluate as you desire\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 6556–6576\.Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Gao, C\. Chen, Y\. Jia, X\. Gong, K\. Lam, and Q\. Wang \(2025\)Evaluating and mitigating llm\-as\-a\-judge bias in communication systems\.arXiv preprint arXiv:2510\.12462\.Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px2.p1.1)\.
- T\. Geet al\.\(2024\)Scaling synthetic data creation with 1,000,000,000 personas\.arXiv preprint arXiv:2406\.20094\.Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px4.p1.1)\.
- S\. Gooding and E\. Grefenstette \(2025\)Interaction dynamics as a reward signal for llms\.arXiv preprint arXiv:2511\.08394\.Cited by:[§1](https://arxiv.org/html/2607.10428#S1.p1.1),[§3\.5\.3](https://arxiv.org/html/2607.10428#S3.SS5.SSS3.Px1.p1.10)\.
- Z\. Han, B\. Zhu, W\. Tong, C\. Qin, X\. Wang, J\. Zhang, J\. Chen, H\. Guo, D\. Ran, L\. Lu,et al\.\(2026a\)MER\-r1: multimodal emotion reasoning via slow\-fast thinking synergy\.arXiv preprint arXiv:2606\.27652\.Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px3.p1.1)\.
- Z\. Han, B\. Zhu, W\. Tong, P\. Shao, P\. Song, X\. Wang, J\. Chen, L\. Lu, and X\. Yang \(2026b\)Omni\-perception policy optimization for multimodal emotion reasoning\.arXiv preprint arXiv:2606\.25325\.Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px3.p1.1)\.
- C\. Herlihy, J\. Neville, T\. Schnabel, and A\. Swaminathan \(2024\)On overcoming miscalibrated conversational priors in llm\-based chatbots\.arXiv preprint arXiv:2406\.01633\.Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px3.p1.1)\.
- Q\. Jia, Y\. Shen, X\. Song, K\. Zhang, S\. Wang, D\. Pei, X\. Zhu, and G\. Zhai \(2025\)One battle after another: probing llms’ limits on multi\-turn instruction following with a benchmark evolving framework\.arXiv preprint arXiv:2511\.03508\.Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px1.p1.1)\.
- B\. Jiang, Y\. Yuan, M\. Shen, Z\. Hao, Z\. Xu, Z\. Chen, Z\. Liu, A\. R\. Vijjini, J\. He, H\. Yu,et al\.\(2025\)PersonaMem\-v2: towards personalized intelligence via learning implicit user personas and agentic memory\.arXiv preprint arXiv:2512\.06688\.Cited by:[1st item](https://arxiv.org/html/2607.10428#S1.I1.i1.p1.2),[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px4.p1.1),[§3\.2](https://arxiv.org/html/2607.10428#S3.SS2.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2607.10428#S4.SS1.p3.3)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the 29th Symposium on Operating Systems Principles \(SOSP\),pp\. 611–626\.Cited by:[§4\.1](https://arxiv.org/html/2607.10428#S4.SS1.p1.4)\.
- P\. Laban, H\. Hayashi, Y\. Zhou, and J\. Neville \(2025\)LLMs get lost in multi\-turn conversation\.arXiv preprint arXiv:2505\.06120\.Cited by:[§1](https://arxiv.org/html/2607.10428#S1.p1.1)\.
- M\. T\. R\. Laskar, I\. Jahan, E\. Dolatabadi, C\. Peng, E\. Hoque, and J\. Huang \(2025\)Improving automatic evaluation of large language models \(llms\) in biomedical relation extraction via llms\-as\-the\-judge\.arXiv preprint arXiv:2506\.00777\.Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px2.p1.1)\.
- X\. Li, Z\. C\. Lipton, B\. Dhingra, L\. Li, J\. Gao, and Y\. Chen \(2016\)A user simulator for task\-completion dialogues\.arXiv preprint arXiv:1612\.05688\.Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Li, X\. Shen, X\. Yao, X\. Ding, Y\. Miao, R\. Krishnan, and R\. Padman \(2025\)Beyond single\-turn: a survey on multi\-turn interactions with large language models\.arXiv preprint arXiv:2504\.04717\.Cited by:[§1](https://arxiv.org/html/2607.10428#S1.p1.1)\.
- C\. Lin \(2004\)ROUGE: a package for automatic evaluation of summaries\.InText Summarization Branches Out,pp\. 74–81\.Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px1.p1.1)\.
- E\. Miehling, M\. Nagireddy, P\. Sattigeri, E\. M\. Daly, D\. Piorkowski, and J\. T\. Richards \(2024\)Language models in dialogue: conversational maxims for human\-ai interactions\.arXiv preprint arXiv:2403\.15115\.Cited by:[§1](https://arxiv.org/html/2607.10428#S1.p1.1)\.
- NVIDIA \(2025\)Nemotron\-personas: a demographically grounded persona dataset\.Note:[https://huggingface\.co/datasets/nvidia/Nemotron\-Personas](https://huggingface.co/datasets/nvidia/Nemotron-Personas)Cited by:[1st item](https://arxiv.org/html/2607.10428#S1.I1.i1.p1.2),[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px4.p1.1),[§3\.2](https://arxiv.org/html/2607.10428#S3.SS2.SSS0.Px1.p1.4),[§4\.1](https://arxiv.org/html/2607.10428#S4.SS1.p3.3)\.
- A\. Panickssery, S\. R\. Bowman, and S\. Feng \(2024\)LLM evaluators recognize and favor their own generations\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[2nd item](https://arxiv.org/html/2607.10428#S1.I1.i2.p1.1),[§1](https://arxiv.org/html/2607.10428#S1.p2.1),[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2607.10428#S3.SS1.p3.1)\.
- K\. Papineni, S\. Roukos, T\. Ward, and W\. Zhu \(2002\)BLEU: a method for automatic evaluation of machine translation\.InProceedings of the 40th Annual Meeting of the Association for Computational Linguistics,pp\. 311–318\.Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px1.p1.1)\.
- H\. Rashkin, E\. M\. Smith, M\. Li, and Y\. Boureau \(2019\)Towards empathetic open\-domain conversation models: a new benchmark and dataset\.InACL,Cited by:[Appendix C](https://arxiv.org/html/2607.10428#A3.SS0.SSS0.Px2.p1.2)\.
- J\. Schatzmann, B\. Thomson, K\. Weilhammer, H\. Ye, and S\. Young \(2007\)Agenda\-based user simulation for bootstrapping a pomdp dialogue system\.InHuman Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers,pp\. 149–152\.Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Suh, E\. Jahanparast, S\. Moon, M\. Kang, and S\. Chang \(2025\)Language model fine\-tuning on scaled survey data for predicting distributions of public opinions\.arXiv preprint arXiv:2502\.16761\.Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px3.p1.1)\.
- G\. Sun, A\. Kagrecha, P\. Manakul, P\. Woodland, and M\. Gales \(2025\)SkillAggregation: reference\-free llm\-dependent aggregation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 15532–15548\.Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px2.p1.1),[§4\.9](https://arxiv.org/html/2607.10428#S4.SS9.p1.3)\.
- W\. Sun, S\. Guo, S\. Zhang, P\. Ren, Z\. Chen, M\. de Rijke, and Z\. Ren \(2024\)Metaphorical user simulators for evaluating task\-oriented dialogue systems\.ACM Transactions on Information Systems42\(1\),pp\. 17:1–17:29\.Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px1.p1.1)\.
- W\. Tong, H\. Guo, D\. Ran, J\. Chen, J\. Lu, K\. Wang, K\. Li, X\. Zhu, J\. Li, K\. Li,et al\.\(2025\)Interactiveomni: a unified omni\-modal model for audio\-visual multi\-turn dialogue\.arXiv preprint arXiv:2510\.13747\.Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Wang, J\. Morgenstern, and J\. P\. Dickerson \(2025a\)Large language models that replace human participants can harmfully misportray and flatten identity groups\.Nature Machine Intelligence,pp\. 1–12\.Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px3.p1.1)\.
- P\. Wang, L\. Li, L\. Chen, Z\. Cai, D\. Zhu, B\. Lin, Y\. Cao, Q\. Liu, T\. Liu, and Z\. Sui \(2023\)Large language models are not fair evaluators\.arXiv preprint arXiv:2305\.17926\.Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Wang, Y\. Song, T\. Zhu, X\. Zhang, Z\. Yu, H\. Chen, C\. Song, Q\. Wang, C\. Wang, Z\. Wu,et al\.\(2025b\)TrustJudge: inconsistencies of llm\-as\-a\-judge and how to alleviate them\.arXiv preprint arXiv:2509\.21117\.Cited by:[2nd item](https://arxiv.org/html/2607.10428#S1.I1.i2.p1.1),[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px2.p1.1),[§4\.9](https://arxiv.org/html/2607.10428#S4.SS9.p1.3)\.
- S\. Wu, M\. Galley, B\. Peng, H\. Cheng, G\. Li, Y\. Dou, W\. Cai, J\. Zou, J\. Leskovec, and J\. Gao \(2025\)CollabLLM: from passive responders to active collaborators\.arXiv preprint arXiv:2502\.00640\.Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhan \(2024\)τ\\tau\-Bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.arXiv preprint arXiv:2406\.12045\.Cited by:[3rd item](https://arxiv.org/html/2607.10428#S1.I1.i3.p1.1),[§1](https://arxiv.org/html/2607.10428#S1.p2.1),[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px1.p1.1),[§3\.5\.2](https://arxiv.org/html/2607.10428#S3.SS5.SSS2.p3.4)\.
- D\. Yuan, Y\. Chen, G\. Liu, C\. Li, C\. Tang, D\. Zhang, Z\. Wang, X\. Wang, and S\. Liu \(2025\)DMT\-rolebench: a dynamic multi\-turn dialogue based benchmark for role\-playing evaluation of large language model and agent\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 25760–25768\.Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px4.p1.1)\.
- C\. Zhang, L\. F\. D’Haro, Y\. Chen, M\. Zhang, and H\. Li \(2024\)A comprehensive analysis of the effectiveness of large language models as automatic dialogue evaluators\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 19515–19524\.Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Zhang, E\. Dinan, J\. Urbanek, A\. Szlam, D\. Kiela, and J\. Weston \(2018\)Personalizing dialogue agents: i have a dog, do you have pets too?\.InACL,Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px4.p1.1)\.
- S\. Zhang, Y\. Zhan, R\. Su, R\. Sun, Z\. Song, Z\. Chen, and X\. Zhang \(2026\)EMPA: evaluating persona\-aligned empathy as a process\.arXiv preprint arXiv:2603\.00552\.Cited by:[3rd item](https://arxiv.org/html/2607.10428#S1.I1.i3.p1.1),[§1](https://arxiv.org/html/2607.10428#S1.p2.1),[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px3.p1.1),[§3\.5\.2](https://arxiv.org/html/2607.10428#S3.SS5.SSS2.p2.5)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.\(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.36,pp\. 46595–46623\.Cited by:[§1](https://arxiv.org/html/2607.10428#S1.p2.1),[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px2.p1.1)\.
- Q\. Zhong, Z\. Li, S\. Fan, and A\. Sun \(2025\)Evaluating llm adaptation to sociodemographic factors: user profile vs\. dialogue history\.arXiv preprint arXiv:2505\.21362\.Cited by:[§2](https://arxiv.org/html/2607.10428#S2.SS0.SSS0.Px3.p1.1)\.

## Appendix ADialogue Generation Loop

[Algorithm˜1](https://arxiv.org/html/2607.10428#alg1)formalises the multi\-turn dialogue loop referenced from §[3](https://arxiv.org/html/2607.10428#S3)\. The simulator’s per\-turn progress label \(*not\_started*/*in\_progress*/*achieved*\) triggers early termination after two consecutive*achieved*reports; when early termination occurs, the warm\-up weight vectorwtw\_\{t\}is regenerated at the dialogue’s actual lengthTactualT\_\{\\text\{actual\}\}so that the warm\-up parameterα\\alpharemains semantically fixed rather than being inflated by truncating aTmaxT\_\{\\max\}\-length vector\.

## Appendix BPersona Pools — Schema and Statistics

Both pools expose the fields persona id, source∈\\in\{Nemotron\-USA, PersonaMem\-v2\}, a one\-paragraph persona description, and a set of structured attributes\.

##### Pool A — Nemotron\-Personas\-USA\.

A demographic schema with1818attributes including age, age bucket, sex, occupation group, marital status, education, race/ethnicity, and a Big\-Five personality vector\. Five hundred records are sampled after cosine deduplication \(≥0\.85\\geq 0\.85onall\-MiniLM\-L6\-v2\) and stratification by occupation group×\\timesage bucket\.

##### Pool B — PersonaMem\-v2\.

Paragraph\-form profiles distilled from long\-form user–assistant interactions, with lighter structured attributes \(topic tags and an interaction length bucket\)\. Five hundred records are sampled under the same deduplication criterion\.[Table˜10](https://arxiv.org/html/2607.10428#A2.T10)summarises the per\-bucket counts and persona token lengths of both pools\.

Table 10:Persona, topic, emotion and length statistics for the two EYT\-Bench persona pools\.

## Appendix CLabel Taxonomies

The user simulator emits, at every turn, three categorical labels — explicit intent, latent intent and emotion — that serve as gold labels for the per\-turn objective metrics in §[3\.5\.1](https://arxiv.org/html/2607.10428#S3.SS5.SSS1)\.[Table˜11](https://arxiv.org/html/2607.10428#A3.T11)lists the full label sets; the simulator selects exactly one value per field per turn and rejects out\-of\-vocabulary outputs\.

Table 11:Categorical label taxonomies used by the simulator and scored against the target’s per\-turn predictions\.##### Explicit vs\. latent intent\.

The explicit\-intent taxonomy captures*what*the user is asking the assistant to do at the current turn \(e\.g\. Seeking Advice, Task Completion\), and is largely instrumental\. The latent\-intent taxonomy captures*why*the user is engaging in the conversation at a deeper psychological level \(e\.g\. Seeking Validation, Building Connection\), and is invariant to the surface request\. The two views are deliberately non\-overlapping: a single user turn can carry, for example, explicit intentInformation Seekingtogether with latent intentNeed for Security\. Targets predict both fields independently, which lets EYT\-Bench distinguish models that handle surface task structure from models that also recognise underlying user needs\.

##### Emotion\.

We extend the prior1010\-label emotion dictionary \(nine of which were negative\) to a1515\-label balanced set: seven negative, six positive, and two neutral states\. The expanded taxonomy prevents the simulator from collapsing toward negative valence by construction; cross\-checks against the GoEmotions taxonomy\(Demszkyet al\.,[2020](https://arxiv.org/html/2607.10428#bib.bib44)\)and the empathetic\-dialogues label set\(Rashkinet al\.,[2019](https://arxiv.org/html/2607.10428#bib.bib43)\)confirm coverage of the positive valence\.

## Appendix DJudge Rubric and Anchors

Every sub\-indicator is scored on\{0,1\}\\\{0,1\\\}, so each dimension sums to an integer in\[0,5\]\[0,5\]before being rescaled to\[0,100\]\[0,100\]for reporting\.[Figure˜5](https://arxiv.org/html/2607.10428#A4.F5)contains the rubric prompts for Empathy \(E\), Persona Alignment \(P\) and Anthropomorphic Interaction \(I\)\. The primary judge is Gemini\-3\.1\-Pro\-Thinking with reasoning effort set to high and the response budget capped at8,0008\{,\}000tokens; the prompt template stored atprompts/default\_en\.yamlcontains the turn prompt, the final\-intent prompt, and the sub\-indicator anchors\.

![Refer to caption](https://arxiv.org/html/2607.10428v1/x5.png)

![Refer to caption](https://arxiv.org/html/2607.10428v1/x6.png)

![Refer to caption](https://arxiv.org/html/2607.10428v1/x7.png)

Figure 5:Judge rubric prompts for the three subjective dimensions: Empathy \(left\), Persona Alignment \(centre\), Anthropomorphic Interaction \(right\)\. Each rubric expands into five binary sub\-indicators\.![Refer to caption](https://arxiv.org/html/2607.10428v1/x8.png)Figure 6:User\-simulator prompt template\. The simulator receives the ChatSEED’s persona, topic, initial emotion and final\-intent description, and emits the next user utterance together with a structured per\-turn annotation\{ie,il,e,p\}\\\{i^\{e\},i^\{l\},e,p\\\}and a self\-reported final\-intent progress label\.![Refer to caption](https://arxiv.org/html/2607.10428v1/x9.png)Figure 7:System prompt for the target dialogue model, instructing the model toward an empathetic, conversational and concise style that matches the user’s language\.
## Appendix EPipeline Prompts

[Figure˜8](https://arxiv.org/html/2607.10428#A5.F8)shows the per\-turn prediction prompt used during the perception stage of the target model\. The prediction prompt is invoked independently of the system prompt in[Figure˜7](https://arxiv.org/html/2607.10428#A4.F7)so that the prediction rubric does not contaminate the response distribution\.

![Refer to caption](https://arxiv.org/html/2607.10428v1/x10.png)Figure 8:Perception\-stage prediction prompt: the target model emits an intent and emotion JSON conditioned on the rolling context before generating its response under the separate system prompt\.
## Appendix FModel Slate

[Table˜12](https://arxiv.org/html/2607.10428#A6.T12)lists the1717target models evaluated in EYT\-Bench\.

Table 12:The1717\-model evaluation slate\.FamilyVariants*Closed\-source \(API access\)*Anthropic ClaudeOpus\-4\.7, Sonnet\-4\.6Google Gemini3\.1\-Pro\-Thinking, 3\-Flash\-ThinkingOpenAI GPT5\.5ByteDance Seed2\.0\-Pro, 2\.0\-Mini, 2\.0\-Lite*Open\-source \(API access\)*DeepSeek\-V4Pro, FlashAlibaba Qwen3\.527B, 35B\-A3B, 397B\-A17B*Open\-source \(self\-hosted\)*Google Gemma\-426B\-A4B, 26B\-A4B\-Thinking31B, 31B\-Thinking
## Appendix GNumber of Turns

We setTmax=10T\_\{\\max\}=10as a middle\-of\-the\-road value:τ\\tau\-bench spans88–1515, MultiChallenge44–1010and MULTI\-Bench55–1010\. The simulator’s early\-stop condition \(two consecutive*achieved*reports\) allows goal\-completion dialogues to terminate in as few asTactual=4T\_\{\\text\{actual\}\}=4turns, with a meanTactual≈7\.8T\_\{\\text\{actual\}\}\\\!\\approx\\\!7\.8across the slate\.

## Appendix HTrajectory Metrics

[Table˜13](https://arxiv.org/html/2607.10428#A8.T13)reports the per\-model trajectory metrics on PersonaMem\-v2 that support §[4\.3](https://arxiv.org/html/2607.10428#S4.SS3)\.

Table 13:Trajectory metrics on PersonaMem\-v2 \(N=100N\{=\}100\)\.Sat\.is the judge\-assigned final\-intent satisfaction score, rescaled from11–55to\[0,100\]\[0,100\]\.
## Appendix IWarm\-up Sensitivity

Theα\\alpha\-sensitivity sweep that supports §[4\.8](https://arxiv.org/html/2607.10428#S4.SS8)is presented in[Table˜8](https://arxiv.org/html/2607.10428#S4.T8)in the main text\.

## Appendix JJudge Calibration

[Table˜14](https://arxiv.org/html/2607.10428#A10.T14)and[Table˜15](https://arxiv.org/html/2607.10428#A10.T15)are the calibration tables referenced from §[4\.9](https://arxiv.org/html/2607.10428#S4.SS9)\.

Table 14:Cross\-judge ablation: primary judge replaced by the third\-family DeepSeek\-V4\-Pro on a stratified subsample \(n=46n\{=\}46paired dialogues,1111\(pool, model\) cells\)\. SubjectiveΔ\\Deltaare on\[0,100\]\[0,100\]; final\-intent satisfactionΔSat\.\\Delta\_\{\\textsc\{Sat\.\}\}is rescaled from11–55to\[0,100\]\[0,100\]\. The shift acts as a global calibration offset rather than re\-ordering targets: relative model rankings are preserved under both judges, andΔSat\.\\Delta\_\{\\textsc\{Sat\.\}\}is small in magnitude \(≤1\.8\\leq 1\.8on average\)\.Table 15:Judge–human alignment on a55\-annotator pilot \(5959turn\-cells,99models\)\.H¯\\bar\{H\}is the human mean\.Bias=J¯−H¯=\\bar\{J\}\-\\bar\{H\}on\[0,100\]\[0,100\]\.κwJH\\kappa\_\{w\}^\{\\textsc\{JH\}\},κwHH\\kappa\_\{w\}^\{\\textsc\{HH\}\}are mean quadratic\-weighted Cohenκ\\kappafor judge–human and human–human pairs;αord\\alpha\_\{\\text\{ord\}\}is Krippendorff’s ordinal alpha over the55humans\.Judge vs\.H¯\\bar\{H\}AnnotatorsDim\.rrρ\\rhoBiasκwJH\\kappa\_\{w\}^\{\\textsc\{JH\}\}κwHH\\kappa\_\{w\}^\{\\textsc\{HH\}\}αord\\alpha\_\{\\text\{ord\}\}Empathy0\.870\.73\+7\.8\+7\.80\.710\.790\.78Persona Alignment0\.740\.45\+9\.0\+9\.00\.530\.670\.68Anthropomorphic0\.580\.55\+5\.4\+5\.40\.440\.670\.66*Tolerance bands \(Judge vs\.H¯\\bar\{H\}, fraction of cells\)*Empathy\|J−H¯\|≤1\|J\{\-\}\\bar\{H\}\|\{\\leq\}1:93\.2%93\.2\\%; same Low/Mid/High bucket:88\.1%88\.1\\%Persona Alignment\|J−H¯\|≤1\|J\{\-\}\\bar\{H\}\|\{\\leq\}1:94\.9%94\.9\\%; same bucket:96\.6%96\.6\\%Anthropomorphic\|J−H¯\|≤1\|J\{\-\}\\bar\{H\}\|\{\\leq\}1:94\.9%94\.9\\%; same bucket:98\.3%98\.3\\%

## Appendix KAnthropomorphic Warm\-up by Family

The Anthropomorphic dimension exhibits family\-specific warm\-up signatures that are absent from Empathy and Persona Alignment\. Gemma and Claude trend negative \(Δ≤0\\Delta\\\!\\leq\\\!0, the classical warm\-up pattern: early turns are more formal, later turns more colloquial\), whereas Gemini\-Flash, Qwen3\.5 and the Doubao Seed family trend positive \(Δ\>0\\Delta\\\!\>\\\!0, early turns are already colloquial and the late turns regress slightly\)\. The pattern is consistent with the observation that the colloquial and rhythm sub\-indicators saturate within the first two turns\.

## Appendix LHuman Alignment Study

The human alignment results in §[4\.9](https://arxiv.org/html/2607.10428#S4.SS9)and[Table˜15](https://arxiv.org/html/2607.10428#A10.T15)are drawn from a five\-annotator pilot on5959turn\-cells \(1010dialogues,99target models\)\. Each annotator independently scored every turn on the three subjective dimensions on the same11–55scale as the LLM judge; the judge’s per\-turn scores were visible on the annotation UI for direct A/B comparison, which introduces an anchoring risk that we mitigate in the camera\-ready full study\.

##### Per\-model agreement\.

Per\-model judge\-vs\.\-human\-mean agreement varies with sample size but tracks[Table˜15](https://arxiv.org/html/2607.10428#A10.T15): on Empathy the judge correlates with the human mean atr=0\.99r\{=\}0\.99for Seed\-2\.0\-Lite,r=0\.97r\{=\}0\.97for Gemini\-3\-Flash\-Thinking, andr=0\.96r\{=\}0\.96for Gemma\-4\-31B\-Thinking\. Claude\-Sonnet\-4\.6 is the single largest lenience point \(\+0\.82\+0\.82rating points /\+16\+16on0–100100\), reflecting the judge’s preference for Claude’s florid empathic style; Gemma\-4\-31B\-Thinking is the*only*target where the judge underestimates the human mean \(on Anthropomorphic Interaction,−0\.12\-0\.12\), consistent with that model’s strong objective accuracy and slightly more clinical conversational register\.

##### Position effect\.

The judge’s lenience on Empathy nearly triples between early and late turns \(\+0\.18→\+0\.50\+0\.18\\to\+0\.50rating points;\+3\.6→\+10\.0\+3\.6\\to\+10\.0on0–100100\)\. The direction is the same as the warm\-up effect in §[4\.8](https://arxiv.org/html/2607.10428#S4.SS8), but the judge amplifies it; we therefore recommend that papers built on EYT\-Bench’s LLM judge avoid reporting unweighted late\-turn\-only sub\-scores\.