SpeechEQ: Benchmarking Emotional Intelligence Quotient in Socially Aware Voice Conversational Models

# SpeechEQ: Benchmarking Emotional Intelligence Quotient in Socially Aware Voice Conversational Models
Source: [https://arxiv.org/html/2606.25990](https://arxiv.org/html/2606.25990)
Liang\-Yuan Wu1Zih\-Ching Chen2Tongshuang Wu3C\.\-H\. Huck Yang2Hua Shen1, 4 1New York University2NVIDIA3Carnegie Mellon University4NYU Shanghai \{leo\.wu,huashen\}@nyu\.edu;\{virginiac,hucky\}@nvidia\.com;sherryw@cs\.cmu\.edu

###### Abstract

As multimodal conversational systems increasingly engage in spoken interaction, their ability to navigate paralinguistic social cues has become a critical bottleneck for natural human\-AI communication\. However, existing evaluations of machine emotional intelligence assess reasoning exclusively through isolated text or passive acoustic perception, overlooking the complex cross\-modal reasoning required for active, multi\-turn dialogue\. We introduceSpeechEQ, a comprehensive framework designed to evaluate the sociolinguistic reasoning of Speech\-Language Models \(SLMs\)\. The framework includes a validated dataset of 2,265 dialogues across 15 Emotional Quotient \(EQ\) subscales grounded in EQ\-i 2\.0 theory, along with a multi\-turn evaluation protocol measured by our proposed Spoken EQ \(SEQ\) score inspired by human EQ assessments\. Experiments show limitations in how both existing Speech Emotion Recognition and end\-to\-end Speech\-Language Models understand and apply paralinguistic cues through speech\. While end\-to\-end architectures outperform cascaded systems,SpeechEQreveals that current multimodal models remain bottlenecked by a text\-reliant “modality shortcut,” an alignment\-induced “safety trap,” and “contextual amnesia,” highlighting the barriers to truly emotionally aware AI\. Our benchmark can be accessed at[https://huggingface\.co/datasets/SpeechEQ/SpeechEQ](https://huggingface.co/datasets/SpeechEQ/SpeechEQ)and demo page at[https://binomial14\.github\.io/speecheq\-demo/](https://binomial14.github.io/speecheq-demo/)

## 1Introduction

Recent advances in Speech\-Language Models \(SLMs\) have enabled a new generation of end\-to\-end voice agents capable of fluent\(Défossezet al\.,[2024](https://arxiv.org/html/2606.25990#bib.bib75); Reddy,[1988](https://arxiv.org/html/2606.25990#bib.bib76)\), real\-time interaction\(Rubensteinet al\.,[2023](https://arxiv.org/html/2606.25990#bib.bib83); Zhanget al\.,[2023](https://arxiv.org/html/2606.25990#bib.bib84); Chuet al\.,[2024](https://arxiv.org/html/2606.25990#bib.bib85); Barraultet al\.,[2023](https://arxiv.org/html/2606.25990#bib.bib86); Yeet al\.,[2025](https://arxiv.org/html/2606.25990#bib.bib4); Deshmukhet al\.,[2026](https://arxiv.org/html/2606.25990#bib.bib2)\)\. These systems excel at semantic understanding, transcribing speech, answering questions, and generating coherent dialogue\. However,human communication is not purely semantic\(Scherer,[2003](https://arxiv.org/html/2606.25990#bib.bib51)\)\. In spoken interaction, how something is said, through prosody, timing, and vocal intensity, often carries more social meaning than what is said\(Wu and Jain,[2025](https://arxiv.org/html/2606.25990#bib.bib80); Kimet al\.,[2023](https://arxiv.org/html/2606.25990#bib.bib81)\)\.

This gap exposes a fundamental limitation:today’s SLMs are semantically fluent but socially shallow\. They frequently produce affectively flat responses and struggle to interpret or generate paralinguistic cues that signal empathy, tension, or intent\(Qianet al\.,[2025](https://arxiv.org/html/2606.25990#bib.bib82)\)\. As a result, even highly capable systems fail in scenarios where Emotional Intelligence \(i\.e\., EQ\)\(Salovey and Mayer,[1990](https://arxiv.org/html/2606.25990#bib.bib35); Elfenbein and Ambady,[2002](https://arxiv.org/html/2606.25990#bib.bib77)\), not factual correctness, determines interaction quality\.

We argue that this limitation stems from a deeper issue:the lack of rigorous evaluation for multimodal Emotional Intelligence in speech\. Existing benchmarks either \(i\) evaluate emotional intelligence in text\-only settings or \(ii\) treat speech as a passive perception task \(e\.g\., emotion classification\), ignoring the interactive, multi\-turn, and cross\-modal reasoning required in real conversations\. Consequently, current models can achieve high performance while relying on a “semantic shortcut”, bypassing acoustic reasoning altogether\.

To address this gap, we introduceSpeechEQ, a benchmark and evaluation framework formultimodal emotional intelligence in spoken dialogue\.SpeechEQis built on three key principles: \(1\) Behavioral Grounding via EQ\-i 2\.0\. We operationalize emotional intelligence using the EQ\-i 2\.0 framework\(Bar\-On,[2004](https://arxiv.org/html/2606.25990#bib.bib31); Wiechorek,[2011](https://arxiv.org/html/2606.25990#bib.bib78)\), constructing scenarios that map psychological constructs \(e\.g\., empathy, impulse control\) into observable acoustic behaviors\. \(2\) Semantic–Acoustic Decoupling\. We isolate acoustic reasoning by presenting models with response options that share identical transcripts but differ in paralinguistic delivery\. This removes semantic cues and forces models to rely on pure acoustic understanding\. \(3\) Sustained Affective Pragmatics\. Rather than isolated utterances, we evaluate multi\-turn dialogues with escalating emotional stakes, testing whether models can track and adapt to evolving social dynamics over time\.

The resulting dataset comprises2,265 multi\-turn dialogues \(42\.37 hours\) spanning 15 EQ subscales, generated via a controlled LLM–TTS pipeline that enforces both behavioral validity and acoustic contrast\. To quantify performance, we introduce the Spoken Emotional Quotient \(SEQ\), a standardized metric drawing conceptual inspiration from Raven’s Standard Progressive Matrices\(Raven and others,[1998](https://arxiv.org/html/2606.25990#bib.bib79); John and Raven,[2003](https://arxiv.org/html/2606.25990#bib.bib57)\)\. SEQ aggregates multi\-turn trajectory accuracy across EQ dimensions, capturing not only immediate recognition but also sustained emotional reasoning\. We show that SEQ strongly correlates with human judgments, establishing it as a reliable proxy for evaluating EQ in speech\.

UsingSpeechEQ, we benchmark both cascaded pipelines and state\-of\-the\-art end\-to\-end SLMs\. While end\-to\-end models perform better overall, our analysis reveals three fundamental limitations: \(i\) Modality Shortcut: Models over\-rely on text and fail when meaning is carried purely by acoustics\. \(ii\) Affective Flattening: Alignment mechanisms bias models toward safe, low\-arousal tones, suppressing necessary emotional expression\. \(iii\) Contextual Amnesia: Performance degrades over multi\-turn interactions, indicating weak long\-term affective tracking\. These findings suggest that current SLMsdo not truly reason about emotion–they approximate it under favorable conditions\.

Overall, the contributions are three\-fold:

- •A Grounded Paralinguistic Benchmark: We introduceSpeechEQ, a multi\-turn speech benchmark grounded in 15 EQ\-i 2\.0 dimensions\. By decoupling text and prosody, it isolates acoustic signals and enables rigorous evaluation of paralinguistic reasoning\.
- •A Comprehensive Evaluation Framework and Metric: We propose a unified evaluation protocol for both cascaded and end\-to\-end models, along with the Spoken Emotional Quotient \(SEQ\)—a trajectory\-level metric for measuring emotional intelligence across multi\-turn interactions\.
- •Empirical Insights: We benchmark state\-of\-the\-art models and identify three failure modes: modality shortcut, affective flattening, and contextual amnesia, revealing key limitations in current speech\-language systems\.

## 2SpeechEQ: Evaluating Emotional Intelligence in Speech LMs

This section details the development ofSpeechEQ, an integrated evaluation framework and dataset\. We first outline the motivation and design rationale, followed by a detailed description of the generation pipeline and validation process\. Finally, we formalize the framework’s evaluation protocol and introduce the Spoken Emotional Quotient \(SEQ\), a standardized metric for quantifying SLMs’ emotional intelligence\.

![Refer to caption](https://arxiv.org/html/2606.25990v1/x1.png)Figure 1:Overview of theSpeechEQdataset construction pipeline\.### 2\.1Motivation and Design Rationale

Attributable Behavior Design\.Our primary goal is twofold: to move beyond passive classification in traditional speech emotion recognition\(Cowieet al\.,[2001](https://arxiv.org/html/2606.25990#bib.bib71); Burkhardt,[2000](https://arxiv.org/html/2606.25990#bib.bib70); Schuller,[2018](https://arxiv.org/html/2606.25990#bib.bib44)\)by rigorously evaluating SLMs in active social settings, and to ensure that the socially resonant responses can be distinctly isolated within the audio waveform\. To achieve this attributable behavioral design, we ground our dataset in the EQ\-i 2\.0 framework\(Bar\-On,[2004](https://arxiv.org/html/2606.25990#bib.bib31); Wiechorek,[2011](https://arxiv.org/html/2606.25990#bib.bib78)\)\. EQ\-i 2\.0 is a trait\-behavioral model that operationalizes social functioning into measurable subscales \(more details in Appendix[A](https://arxiv.org/html/2606.25990#A1)\)\. This behavioral focus provides the exact mechanism needed to translate complex psychological constructs directly into distinct, measurable acoustic features\.

Tone Variation Design\.To mitigate lexical bias\(Chenet al\.,[2026](https://arxiv.org/html/2606.25990#bib.bib68); Wanget al\.,[2020](https://arxiv.org/html/2606.25990#bib.bib69)\)and rigorously evaluate acoustic emotional intelligence, we designed the task as a forced\-choice selection between two audio responses sharing identical transcripts\. By neutralizing the text modality, we eliminate semantic differences and force the system to evaluate subtle paralinguistic cues to determine the contextually resonant response\.

Multi\-turn Conversational Arc\.To capture emotional intelligence beyond single utterances, we evaluate how models track cues across sustained interactions\. We structure scenarios as three\-exchange dialogues between a humanCatalystand the SLM acting as theTest Subject\. Following an initial exchange that establishes the emotional baseline, the system must navigate escalating social pressure by selecting the contextually appropriate acoustic response during the second and third exchanges\. This design effectively tests the model’s capacity for complex sociolinguistic pragmatics over an evolving conversational trajectory\.

### 2\.2Data Generation

We developed an automated, LLM\-driven generation pipeline \(Figure[1](https://arxiv.org/html/2606.25990#S2.F1)\) to execute the design rationales\. We highlight the scenario and tonal instruction generation with the complete five\-stage technical details and prompts in Appendix[B](https://arxiv.org/html/2606.25990#A2)\.

Scenario Generation and Persona Matrix\.We engineered a highly constrained scenario matrix under the EQ\-i 2\.0 framework\. Each scenario is generated at the intersection of three parameters: a specific EQ\-i 2\.0 subscale \(e\.g\., Interpersonal, Empathy\), a situational valence \(Positive, Negative, or Conflict\), and a real\-world scenario \(e\.g\., workplace, medical, educational\)\. Crucially, to ensure the forced\-choice evaluation is rigorous, the pipeline generates distinct “social deficit personas” corresponding to the targeted EQ scale, such as a “Toxic Optimist” failing to validate grief\. This ensures the evaluated model is tested against complex sociolinguistic breakdowns rather than generic antagonistic behavior\.

Tone Generation and Contrast Filtering\.The tone generation phase bridges the gap between the abstract psychological personas and raw audio synthesis over these neutral texts\. We prompt the LLM to generate physically grounded acoustic descriptors, translating three generated personas \(one contextually appropriate response and two dysregulated distractors\) into explicit vocal instructions\. To counter the default safety alignment of generation models, we apply a filtering step that rejects minimizing descriptors such as ’polite’ or ’calm,’ enforcing the generation of extreme, physically grounded acoustic markers\. Finally, an automated filtering stage selects the two most distinctly contrasting instructions for synthesis via`gpt\-4o\-mini\-tts\-2025\-03\-20`, producing distinctly nuanced paralinguistic variations for speech candidates\.

### 2\.3Data Validation

To ensure the quality ofSpeechEQ, we adopt a two\-phase validation pipeline\. Phase 1 \(automated\) verifies scenario consistency and acoustic distinctiveness, while Phase 2 \(human\) evaluates naturalness and perceptual validity\.

Table 1:Semantic validation results\.Semantic and Logical Verification\.We first conduct an*oracle*text\-based evaluation by providing models with transcripts and explicit tone instructions instead of audio \(e\.g\.,“Fast pacing, sarcastic cheerfulness\.”\)\. Models achieve near\-perfect accuracy \(Table[1](https://arxiv.org/html/2606.25990#S2.T1)\), confirming that scenarios are logically consistent and unambiguously aligned with the 15 EQ\-i 2\.0 dimensions\(Bar\-On,[2004](https://arxiv.org/html/2606.25990#bib.bib31); Wiechorek,[2011](https://arxiv.org/html/2606.25990#bib.bib78)\)\(see Appendix[A](https://arxiv.org/html/2606.25990#A1)for the complete taxonomy\)\. This result isolates the challenge ofSpeechEQto the modality gap\. For example, understanding paralinguistics from audio, rather than ambiguity in scenario design\.

Acoustic Variance Validation\.

To ensure meaningful paralinguistic contrast, we quantify acoustic differences between resonant and dissonant clips for each evaluation pair \(Turns 4 and 6\) using librosa\. We extract six dimensions: mean pitch, zero\-crossing rate \(speaking\-rate proxy\), spectral centroid, RMS energy, mean MFCC, and duration\. We compute a composite contrast score \(max 8 points\), with pitch and speaking\-rate gaps contributing up to 2 points each and the remaining four up to 1 point each\. Pairs scoring below 4 trigger up to three TTS regeneration attempts; examples that still fail are discarded\.

Human Expert Validation\.We further assess perceptual validity through expert annotation\. We sample 75 scenarios \(5 per EQ subscale\) and evaluate them across five dimensions: Generation Quality \(Text and Audio\), EQ Relevance \(Text and Audio\), and Answer Correctness \(Paralinguistic Accuracy\)\. Evaluation results are in Table[2](https://arxiv.org/html/2606.25990#S2.T2), with formal definitions and literature grounding for these metrics in Table[5](https://arxiv.org/html/2606.25990#A3.T5)in Appendix[C](https://arxiv.org/html/2606.25990#A3)\. Two expert annotators achieve strong agreement \(Cohen’s Kappaκ=0\.617\\kappa=0\.617\) after iterative reconciliation\. Any scenario failing a single criterion is removed, ensuring high\-quality, socially valid data\.

Data Statistics\.The final dataset comprises a total of 2,265 dialogues, perfectly balanced across the 15 EQ\-i 2\.0 subscales, totaling 42\.37 hours of audio\. The average length of one dialogue is 67\.35 seconds \(σ=22\.29\\sigma=22\.29\), providing sufficient temporal context for evaluating sustained emotional tracking\.

Table 2:Data validation results from human experts\.
### 2\.4Evaluation Protocol for Emotional Intelligence

The Two\-Round Selection Process\.We evaluate models through a two\-round, forced\-choice task at Turn 4 and Turn 6 of each dialogue\. In Round 1, the model receives the scenario context and initial history \(Turns 1–3 audios\), and must select the socially resonant audio for Turn 4\. In Round 2, the context window is dynamically updated with the selected Turn 4 response and the subsequent Turn 5 utterance, requiring the model to select the correct Turn 6 response\. This sequential dependency tests both immediate emotional recognition and sustained conversational tracking\. Technical prompt details are in Appendix[D](https://arxiv.org/html/2606.25990#A4)\.

Evaluation Metrics\.We report the accuracy of the model’s selection at the first evaluation turn \(A​c​c1Acc\_\{1\}\) and the second evaluation turn \(A​c​c2Acc\_\{2\}\)\. To measure sustained emotional tracking, we further report the conversational trajectory accuracy \(A​c​ct​r​a​jAcc\_\{traj\}\)\. Inspired byBudzianowskiet al\.\([2018](https://arxiv.org/html/2606.25990#bib.bib55)\)andLiuet al\.\([2023](https://arxiv.org/html/2606.25990#bib.bib54)\), this metric requires the model to successfully navigate the entire emotional arc\. For a dataset ofNNmulti\-turn scenarios, lety^i,1\\hat\{y\}\_\{i,1\}andy^i,2\\hat\{y\}\_\{i,2\}denote the model’s predicted choices for theii\-th conversation at Turns 4 and 6, withyi,1y\_\{i,1\}andyi,2y\_\{i,2\}representing the respective ground\-truth resonant labels\. The sustained accuracy is formally defined as the joint success across both evaluation turns, utilizing the indicator function𝕀\\mathbb\{I\}:

A​c​ct​r​a​j=1N​∑i=1N𝕀​\(y^i,1=yi,1∧y^i,2=yi,2\)Acc\_\{traj\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbb\{I\}\(\\hat\{y\}\_\{i,1\}=y\_\{i,1\}\\land\\hat\{y\}\_\{i,2\}=y\_\{i,2\}\)\(1\)
This metric strictly requires the model to answer both consecutive turns correctly within the same evolving context window, and heavily penalizes models that lose conversational memory\. Given the binary forced\-choice design at each turn, the random chance baselines forA​c​c1Acc\_\{1\},A​c​c2Acc\_\{2\}, andA​c​ct​r​a​jAcc\_\{traj\}are50%50\\%,50%50\\%, and25%25\\%, respectively\. We adoptAcct​r​a​j\\mathrm\{Acc\}\_\{traj\}as the primary metric for cross\-paper benchmark comparison, as it measures whether a model follows the target emotional arc without cohort\-relative normalization\.

### 2\.5SEQ Score

WhileAcc​t​r​a​j\\mathrm\{Acc\}\{traj\}serves as the cross\-paper durable metric, raw accuracy alone does not intuitively communicate relative model standing within a cohort\. Inspired by the norm\-referenced scoring principle behind Raven’s Standard Progressive Matrices\(Raven and others,[1998](https://arxiv.org/html/2606.25990#bib.bib79); John and Raven,[2003](https://arxiv.org/html/2606.25990#bib.bib57)\), we introduce the Spoken Emotional Quotient \(SEQ\) as a within\-cohort interpretability complement toAcc​t​r​a​j\\mathrm\{Acc\}\{traj\}\. For each modelii, we first compute the raw score of the trajectory accuracyA​c​ct​r​a​jAcc\_\{traj\}asXiX\_\{i\}\.

Global Standardization\.We then perform global standardization to convert each model’s raw scoreXiX\_\{i\}into a robust standardized score, denoted asZiZ\_\{i\}\. To avoid the high sensitiveness of traditional standard deviation\(Leyset al\.,[2013](https://arxiv.org/html/2606.25990#bib.bib56)\), we utilize the Median Absolute Deviation \(M​A​DMAD\) as a robust statistical measure to compute a resilient standardization\.

Zi=Xi−Median​\(X\)k×MAD​\(X\)Z\_\{i\}=\\frac\{X\_\{i\}\-\\text\{Median\}\(X\)\}\{k\\times\\text\{MAD\}\(X\)\}\(2\)
wherek≈1\.4826k\\approx 1\.4826is the standard scaling factor\. This constant is derived from the inverse of the 75th percentile of the standard normal distribution \(1/Φ−1​\(0\.75\)1/\\Phi^\{\-1\}\(0\.75\)\), which ensures the MAD is asymptotically consistent with the standard deviation of a normal distribution\(Rousseeuw and Croux,[1993](https://arxiv.org/html/2606.25990#bib.bib58)\)\.

Final SEQ Score Computation\.Following standard clinical psychometric scaling, we center the global distribution at a baseline of 100 with a scaled deviation of 15\. We further apply a clinical cap at±4\\pm 4deviations to prevent extreme architectural outliers given our small model group\. The final SEQ score is:

SEQi=max⁡\(μ−4​σ,min⁡\(μ\+4​σ,μ\+σ×Zi\)\)\\text\{SEQ\}\_\{i\}=\\max\(\\mu\-4\\sigma,\\min\(\\mu\+4\\sigma,\\mu\+\\sigma\\times Z\_\{i\}\)\)\(3\)
whereμ=100\\mu=100andσ=15\\sigma=15establish the normative baseline universally adopted in cognitive and emotional intelligence frameworks\(Bar\-On,[2004](https://arxiv.org/html/2606.25990#bib.bib31); Wiechorek,[2011](https://arxiv.org/html/2606.25990#bib.bib78)\)\. We strictly bound the metric at±4​σ\\pm 4\\sigmato mirror the floor and ceiling limits of classical standardized assessments\(Wechsler,[1955](https://arxiv.org/html/2606.25990#bib.bib66)\), as scores beyond this range exceed the empirical measurement validity of psychometric instruments\(Anastasi and Urbina,[1988](https://arxiv.org/html/2606.25990#bib.bib67)\)\.

## 3Experimental Settings

To establish rigorous baselines forSpeechEQ, we evaluate two distinct architectures: cascaded pipelines and end\-to\-end Speech\-Language Models \(SLMs\)\.

Cascaded Systems:To establish a lower bound simulating systems without native audio comprehension, we transcribe candidate audio using ASR \(`Whisper\-large\-v3`\(Radfordet al\.,[2023](https://arxiv.org/html/2606.25990#bib.bib13)\)\) and extract Valence, Arousal, and Dominance \(VAD\) dimensions via a state\-of\-the\-art SER module \(`audeering/wav2vec2\-large\-robust\-12\-emotion\-msp\-dim`\(Wagneret al\.,[2023](https://arxiv.org/html/2606.25990#bib.bib87)\)\)\. We augment the ASR transcripts using two prompting strategies: appending the raw numerical VAD values, or mapping these dimensions into categorical text\-based tone descriptions\. These augmented transcripts are then fed into a text\-only LLM \(e\.g\.,`Qwen3`\(Yanget al\.,[2025](https://arxiv.org/html/2606.25990#bib.bib88)\)\) alongside the scenario background\.

End\-to\-End SLMs:We evaluate open\-weight models across a range of scales: the Qwen\-Omni series\(Huiet al\.,[2024](https://arxiv.org/html/2606.25990#bib.bib23); Xuet al\.,[2025](https://arxiv.org/html/2606.25990#bib.bib21)\), Kimi\-Audio\-7B\-Instruct\(Dinget al\.,[2025](https://arxiv.org/html/2606.25990#bib.bib9)\), MiMo\-Audio\-7B\-Instruct\(Zhanget al\.,[2025](https://arxiv.org/html/2606.25990#bib.bib10)\), and Fun\-Audio\-Chat\-8B\(Teamet al\.,[2025](https://arxiv.org/html/2606.25990#bib.bib11)\)\. We also evaluate two commercial APIs: Gemini\-2\.5\-Pro\(Comaniciet al\.,[2025](https://arxiv.org/html/2606.25990#bib.bib12)\)and gpt\-audio\-1\.5\(Hurstet al\.,[2024](https://arxiv.org/html/2606.25990#bib.bib73)\)\. For all end\-to\-end models, the scenario background and dialogue history are provided as text, while candidate response options are interleaved into the context window as native audio clips\.

## 4Results

Table 3:SpeechEQevaluation results\.For both cascaded systems and end\-to\-end SLMs, performance metrics evaluate isolated single\-turn accuracy \(A​c​c1Acc\_\{1\},A​c​c2Acc\_\{2\}\), conversational trajectory accuracy \(A​c​ct​r​a​jAcc\_\{traj\}\), and our standardized SEQ score\. Deployment efficiency metrics highlight operational trade\-offs, detailing the API or GPU compute cost \(per 100 queries\), average single\-stream inference latency, and token throughput\.In this section, we present the quantitative results of theSpeechEQbenchmark\. We first evaluate the primary performance differences between end\-to\-end and cascaded architectures, followed by a human validation of the SEQ metric\. We then conclude with targeted ablations that isolate two critical failure modes in state\-of\-the\-art models: multi\-turn “contextual amnesia” and the alignment\-driven “safety trap\.”

### 4\.1Are SER models sufficient for paralinguistic reasoning?

State\-of\-the\-art SLM outperforms its cascaded counterparts \(Table[3](https://arxiv.org/html/2606.25990#S4.T3)\), demonstrating the broader limitations of traditional SER models\.Using the same reasoning backbone \(Qwen3\-30B\), the cascaded systems rely on an explicit SER model to extract raw numerical VAD \(Valence, Arousal, Dominance\) values \(e​m​on​u​memo\_\{num\}\) or translate these continuous acoustic features into descriptive text cues \(e​m​od​e​semo\_\{des\}\)\. Overall, the end\-to\-end`Qwen3\-Omni\-30B`model \(i\.e\., processes continuous speech directly\) achieves a substantially higher SEQ\. Interestingly, as illustrated in the left panel of Figure[2](https://arxiv.org/html/2606.25990#S4.F2), thee​m​od​e​semo\_\{des\}cascaded pipeline achieves competitive performance on a few specific EQ subscales\. This suggests that while SER pipelines adequately summarize isolated emotions, discretizing audio into text creates an information bottleneck that strips away the continuous acoustic nuances required to navigate complex, relational EQ dimensions\.

Within end\-to\-end SLMs, we observe strict deployment trade\-offs between reasoning capability, latency, and operational cost \(Table[3](https://arxiv.org/html/2606.25990#S4.T3)\)\.We quantified these metrics via unbatched, single\-stream inference on an NVIDIA A100 GPU for open\-weight models, compared against OpenAI’s API\. While the 30B Qwen3\-Omni model dominates both Qwen2\.5\-Omni variants andgpt\-audio\-1\.5in raw EQ performance \(the right panel in Figure[2](https://arxiv.org/html/2606.25990#S4.F2)\), it suffers from high latency, taking 2\.5×\\timeslonger to respond than its smaller counterparts\. Conversely, the 3B and 7B Qwen2\.5 models offer fast, highly cost\-effective inference but fail to achieve competitive reasoning scores\. Finally,gpt\-audio\-1\.5strikes a strong balance in speed and token efficiency, but its API cost is over 10×\\timeshigher than open\-weight hosting\. These constraints highlight a significant financial and architectural barrier to deploying real\-time empathetic voice agents at scale\.

![Refer to caption](https://arxiv.org/html/2606.25990v1/figures/SEQ_chart_group_1.png)Figure 2:SEQ score for different cascaded systems and E2E SLMs\.
### 4\.2Does the SEQ score reliably align with human perception?

The SEQ score is a significantly more reliable proxy for human sociolinguistic judgment than traditional discrete accuracy through an independent human evaluation\.To validate SEQ’s reflection of SLMs’ emotional intelligence, we sampled one example from each of the 15 EQ subscales, and evaluated the outputs of six randomly selected anonymous models to avoid human bias\. For each example, we recruited five native speakers from Prolific\(Palan and Schitter,[2018](https://arxiv.org/html/2606.25990#bib.bib59)\)to rank the six tone selection and reasoning produced by each model\. We then aggregate these rankings to derive a final rank for each model and compare the results with the rankings obtained from existing metrics \(A​c​c1Acc\_\{1\},A​c​c2Acc\_\{2\}, and their aggregateA​c​ca​l​lAcc\_\{all\}\) on the same 15 examples\. Then we compute Spearman’s Rank Correlation Coefficientρ\\rhowith human rankings\. SEQ achieves the highest correlation with human preference \(ρ=0\.943\\rho=0\.943,p​\-value=0\.005p\\text\{\-value\}=0\.005\) outperforming traditional accuracy, confirming its effectiveness as a reliable proxy for evaluating emotional intelligence in SLMs \(see Table[4](https://arxiv.org/html/2606.25990#S4.T4)\)\.

HumanA​c​c1Acc\_\{1\}A​c​c2Acc\_\{2\}A​c​ca​l​lAcc\_\{all\}SEQM​o​d​e​lAModel\_\{A\}111=11M​o​d​e​lBModel\_\{B\}323=22M​o​d​e​lCModel\_\{C\}665=66M​o​d​e​lDModel\_\{D\}23=1=33M​o​d​e​lEModel\_\{E\}43=5=54M​o​d​e​lFModel\_\{F\}53=3=45correlationρ\\rho\(↑\\uparrow\)\-0\.8200\.8370\.8860\.943pp\-value \(↓\\downarrow\)\-0\.0460\.0390\.0180\.005Table 4:Correlation between human voted rankings and different metrics\.
### 4\.3How does multi\-turn contextual history affect paralinguistic reasoning?

Observing an 8% performance drop \(0\.785→0\.7080\.785\\rightarrow 0\.708\) between the first and second evaluation turn in our best model, Qwen3\-Omni\-30B, we hypothesized that standard Sequential Inference induces a form of contextual amnesia, a temporal degradation that closely aligns with context\-loss phenomena observed in text\-only LLMs\(Liuet al\.,[2024](https://arxiv.org/html/2606.25990#bib.bib61); Labanet al\.,[2025](https://arxiv.org/html/2606.25990#bib.bib60); Linet al\.,[2025](https://arxiv.org/html/2606.25990#bib.bib3)\)\. To isolate this effect and test our hypothesis, we conducted a comparative study evaluating standardSequential Inference\(performing inference twice and appending the model’s own turn\-1 output as history\) againstDirect Inference\(performing a single inference pass on turn 2 by treating the ground\-truth turn\-1 text as given history\)\. Validating our hypothesis, bypassing the model’s self\-generated history via Direct Inference successfully improved overall accuracy, raisingA​c​c2Acc\_\{2\}from 70\.8% to 73\.0%\. However, a granular analysis across the 15 EQ subscales reveals that a significant performance gap remains, and the recovery is highly non\-uniform\. As illustrated in Figure[3](https://arxiv.org/html/2606.25990#S4.F3), while most social dimensions exhibited a positive trend under Direct Inference, five dimensions experienced zero improvement or actually suffered performance degradations\. This discrepancy highlights that multi\-turn sociolinguistic reasoning is a complex task that extends beyond simple memory retention; even when temporal context\-loss is explicitly mitigated with perfect semantic history, the model’s attention mechanism still struggles to balance expanded textual histories against immediate, short\-term acoustic cues\. While our ablation confirms the presence of a long\-term memory leak, uncovering the exact cross\-modal mechanisms that dictate how different emotional dimensions succeed or fail requires much deeper investigation\.

![Refer to caption](https://arxiv.org/html/2606.25990v1/figures/inference_delta_bar.png)Figure 3:Performance differences on turn 2 using Sequential Inference and Direct Inference strategies onQwen3\-Omni\-30B\.
### 4\.4What is the effect of persona conditioning on different EQ aspects?

Persona conditioning reveals a highly asymmetric ability of models to simulate different EQ traits\. While some deficits cause severe degradation, others–particularly those aligned with default “safe” behaviors–have minimal impact\. Replacing the default system prompt with an emotionally adaptive persona yields a modest improvement \(SEQ: 147\.26 → 148\.86\), whereas a global deficit persona leads to a substantial drop \(SEQ: 94\.98, in Figure[4](https://arxiv.org/html/2606.25990#S4.F4)\(left\)\)\. This confirms that persona conditions can meaningfully modulate emotional reasoning\.

More importantly, targeted deficits exhibit uneven effects across EQ dimensions\. Deficits in Self\-Perception and Self\-Expression result in only minor degradation \(SEQ: 133\.72, 140\.16\), while Stress Management causes a catastrophic collapse \(SEQ: 74\.90\) \(Figure[4](https://arxiv.org/html/2606.25990#S4.F4), right panel\)\. This suggests that performance is strongly mediated by how each EQ dimension interacts with the model’s alignment constraints\. We hypothesize that this asymmetry arises from RLHF\-induced behavioral priors\. Traits such as low assertiveness or reduced self\-expression resemble the model’s default polite and compliant behavior, resulting in limited performance loss\(Sharmaet al\.,[2023](https://arxiv.org/html/2606.25990#bib.bib62); Ouyanget al\.,[2022](https://arxiv.org/html/2606.25990#bib.bib63)\)\. In contrast, tasks requiring high\-arousal regulation, boundary\-setting, or assertive responses conflict with safety alignment, preventing the model from producing necessary acoustic variation and leading to failure\. We refer to Appendix[E](https://arxiv.org/html/2606.25990#A5)for the exact persona prompts utilized task\-activating prompting \(TAP\) mechanism\(Yanget al\.,[2023](https://arxiv.org/html/2606.25990#bib.bib6)\)\.

![Refer to caption](https://arxiv.org/html/2606.25990v1/figures/SEQ_chart_group_2.png)Figure 4:SEQ score for Qwen3\-Omni\-30B with different persona\.

## 5Discussion

SpeechEQreveals fundamental limitations in how current models handle social dynamics, leading to three key implications\.

Overcoming the Modality Shortcut\. Modern multimodal models often behave as implicit cascaded systems, prioritizing text over acoustic reasoning\(Chenet al\.,[2026](https://arxiv.org/html/2606.25990#bib.bib68)\)\. By presenting identical transcripts with contrasting prosody,SpeechEQexposes this semantic bias: performance drops sharply when semantic cues are removed\. This suggests that current modelstreat paralinguistics as a secondary signalrather than a core reasoning modality\. Future architectures must elevate acoustic signals to first\-class status in social reasoning\.

Resolving Affective Flattening in Alignment\. Current alignment strategies favor harmless, low\-arousal responses, leading to a persistent bias toward “calm” and “polite” tones\(Baiet al\.,[2022](https://arxiv.org/html/2606.25990#bib.bib72)\)\. We term this effectaffective flattening\. While safe, such expressive suppression undermines empathy in high\-arousal interactions\(Gross,[2002](https://arxiv.org/html/2606.25990#bib.bib74)\)\. This bias emerges in both model reasoning and TTS generation\(Hurstet al\.,[2024](https://arxiv.org/html/2606.25990#bib.bib73)\)\. Advancing emotional intelligence requires decoupling safety from emotional expressiveness, enabling models to deploy a broader and context\-appropriate affective range\.

Toward Sustained Emotional Intelligence\. We observe consistent performance degradation over multi\-turn interactions, indicating“contextual amnesia”in acoustic reasoning\(Liuet al\.,[2024](https://arxiv.org/html/2606.25990#bib.bib61)\)\. Due to dense audio tokenization and limited context capacity, models struggle to maintain long\-horizon emotional coherence\. Future benchmarks should move beyond short exchanges towardlong\-context, multi\-session, persona\-driven evaluations, testing whether agents can sustain and adapt emotional behavior over time\.

## 6Related Work

In psychology, Emotional Intelligence \(EI\) is traditionally modeled through cognitive ability\-based skills\(Salovey and Mayer,[1990](https://arxiv.org/html/2606.25990#bib.bib35); Mayeret al\.,[2002](https://arxiv.org/html/2606.25990#bib.bib36)\)or trait\-based behavioral dispositions\(Bar\-On,[2004](https://arxiv.org/html/2606.25990#bib.bib31); Petrides,[2009](https://arxiv.org/html/2606.25990#bib.bib37)\)\. As Large Language Models \(LLMs\) increasingly mediate human\-AI interactions, evaluating their EI has become a critical focus to ensure trustworthiness and user engagement\(Huanget al\.,[2020](https://arxiv.org/html/2606.25990#bib.bib41)\)\. Consequently, researchers have developed targeted benchmarks to measure the Emotional Quotient \(EQ\) of LLMs using established psychometric theory\(Paech,[2023](https://arxiv.org/html/2606.25990#bib.bib34); Sabouret al\.,[2024](https://arxiv.org/html/2606.25990#bib.bib14)\)\. However, these evaluations rely exclusively on text, overlooking a crucial modality: speech\. In natural human interaction, rich paralinguistic cues \(e\.g\., pitch, pacing, and tone\) often dictate the true emotional weight and intent of a conversation\(Scherer,[2003](https://arxiv.org/html/2606.25990#bib.bib51); Hellbernd and Sammler,[2016](https://arxiv.org/html/2606.25990#bib.bib52)\)\.

Conversely, affective computing in the speech domain has historically focused on Speech Emotion Recognition \(SER\)\(Schulleret al\.,[2011](https://arxiv.org/html/2606.25990#bib.bib29); Schuller,[2018](https://arxiv.org/html/2606.25990#bib.bib44)\), mapping acoustic signals to affective labels using diverse curated datasets\(Bussoet al\.,[2008](https://arxiv.org/html/2606.25990#bib.bib24); Livingstone and Russo,[2018](https://arxiv.org/html/2606.25990#bib.bib49); Poriaet al\.,[2019](https://arxiv.org/html/2606.25990#bib.bib38); Lotfian and Busso,[2017](https://arxiv.org/html/2606.25990#bib.bib39)\)\. SER currently serves as a core evaluation metric for Speech\-Language Models \(SLMs\)\(Yanget al\.,[2021](https://arxiv.org/html/2606.25990#bib.bib43)\)and fine\-tuned self\-supervised architectures\(Baevskiet al\.,[2020](https://arxiv.org/html/2606.25990#bib.bib42); Hsuet al\.,[2021](https://arxiv.org/html/2606.25990#bib.bib47); Liuet al\.,[2025](https://arxiv.org/html/2606.25990#bib.bib48)\)\. Yet, acoustic emotion recognition is merely a prerequisite for emotional intelligence\(Mayeret al\.,[2002](https://arxiv.org/html/2606.25990#bib.bib36)\)\. True sociolinguistic intelligence requires cross\-modal reasoning, evaluating a semantic transcript and its paralinguistic delivery simultaneously, to determine if a tone is contextually appropriate\. To address this fundamental blind spot, our work systematically evaluates how effectively modern SLMs bridge this descriptive measurement\(Chenet al\.,[2025](https://arxiv.org/html/2606.25990#bib.bib5)\)gap of semantic\-acoustic in spoken dialogue\.

Several recent benchmarks evaluate multi\-turn emotional intelligence in spoken dialogue, including Multi\-Bench\(Denget al\.,[2025](https://arxiv.org/html/2606.25990#bib.bib1)\), HumDial\-EIBench\(Wanget al\.,[2026](https://arxiv.org/html/2606.25990#bib.bib7)\), and DeepDialogue\(Koudounaset al\.,[2025](https://arxiv.org/html/2606.25990#bib.bib8)\)\. Despite this progress, three gaps remain\. First, existing benchmarks present audio where semantics and vocal tone are inherently coupled;SpeechEQdecouples them by offering multiple\-choice responses with identical transcripts, forcing models to reason purely from acoustic cues\. Second, prior work focuses on open\-domain dialogues and categorical emotion labels, whereasSpeechEQis grounded in clinical psychometrics, mapping psychological constructs to acoustic behaviors through the 15 subscales of EQ\-i 2\.0\. Third, rather than grading individual turns in isolation,SpeechEQevaluates sustained emotional reasoning across a full 6\-turn conversational arc, probing a model’s capacity for long\-horizon affective tracking with Speech\-IQ\(Wanet al\.,[2025](https://arxiv.org/html/2606.25990#bib.bib18)\)based user profile agentic measurement\.

## 7Conclusion

In this work, we introducedSpeechEQ, the first benchmark evaluating conversational emotional intelligence in SLMs using the clinically validated EQ\-i 2\.0 framework\. Through a semantic neutralization design that decouples lexical content from acoustic prosody, we established the SEQ score as a robust, human\-correlated metric for measuring acoustic emotional intelligence\. While end\-to\-end SLMs outperform cascaded architectures, our evaluation exposes three critical bottlenecks: a text\-reliant “modality shortcut”, a safety trap causing “affective flattening”, and “contextual amnesia” during sustained multi\-turn interactions\. Ultimately,SpeechEQprovides a rigorous diagnostic tool and roadmap for the community, emphasizing the need for alignment strategies that preserve paralinguistic nuance and decouple acoustic harmlessness from genuine emotional depth\.

## References

- D\. Adiwardana, M\. Luong, D\. R\. So, J\. Hall, N\. Fiedel, R\. Thoppilan, Z\. Yang, A\. Kulshreshtha, G\. Nemade, Y\. Lu,et al\.\(2020\)Towards a human\-like open\-domain chatbot\.arXiv preprint arXiv:2001\.09977\.Cited by:[Table 5](https://arxiv.org/html/2606.25990#A3.T5.1.2.1.4.1.1)\.
- Psychological testing\.Vol\.840,London\.Cited by:[§2\.5](https://arxiv.org/html/2606.25990#S2.SS5.p7.3)\.
- A\. Baevski, Y\. Zhou, A\. Mohamed, and M\. Auli \(2020\)Wav2vec 2\.0: a framework for self\-supervised learning of speech representations\.Advances in neural information processing systems33,pp\. 12449–12460\.Cited by:[§6](https://arxiv.org/html/2606.25990#S6.p2.1)\.
- Y\. Bai, A\. Jones, K\. Ndousse, A\. Askell, A\. Chen, N\. DasSarma, D\. Drain, S\. Fort, D\. Ganguli, T\. Henighan,et al\.\(2022\)Training a helpful and harmless assistant with reinforcement learning from human feedback\.arXiv preprint arXiv:2204\.05862\.Cited by:[§5](https://arxiv.org/html/2606.25990#S5.p3.1)\.
- R\. Bar\-On \(2004\)The bar\-on emotional quotient inventory \(eq\-i\): rationale, description and summary of psychometric properties\.\.Cited by:[Appendix A](https://arxiv.org/html/2606.25990#A1.p1.1),[Table 5](https://arxiv.org/html/2606.25990#A3.T5.1.4.3.4.1.1),[§1](https://arxiv.org/html/2606.25990#S1.p4.1),[§2\.1](https://arxiv.org/html/2606.25990#S2.SS1.p1.1),[§2\.3](https://arxiv.org/html/2606.25990#S2.SS3.p2.1),[§2\.5](https://arxiv.org/html/2606.25990#S2.SS5.p7.3),[§6](https://arxiv.org/html/2606.25990#S6.p1.1)\.
- L\. Barrault, Y\. Chung, M\. C\. Meglioli, D\. Dale, N\. Dong, P\. Duquenne, H\. Elsahar, H\. Gong, K\. Heffernan, J\. Hoffman,et al\.\(2023\)Seamlessm4t: massively multilingual & multimodal machine translation\.arXiv preprint arXiv:2308\.11596\.Cited by:[§1](https://arxiv.org/html/2606.25990#S1.p1.1)\.
- P\. Budzianowski, T\. Wen, B\. Tseng, I\. Casanueva, S\. Ultes, O\. Ramadan, and M\. Gasic \(2018\)Multiwoz\-a large\-scale multi\-domain wizard\-of\-oz dataset for task\-oriented dialogue modelling\.InProceedings of the 2018 conference on empirical methods in natural language processing,pp\. 5016–5026\.Cited by:[§2\.4](https://arxiv.org/html/2606.25990#S2.SS4.p2.10)\.
- F\. Burkhardt \(2000\)A database of german emotional speech\.Cited by:[§2\.1](https://arxiv.org/html/2606.25990#S2.SS1.p1.1)\.
- C\. Busso, M\. Bulut, C\. Lee, A\. Kazemzadeh, E\. Mower, S\. Kim, J\. N\. Chang, S\. Lee, and S\. S\. Narayanan \(2008\)IEMOCAP: interactive emotional dyadic motion capture database\.Language resources and evaluation42\(4\),pp\. 335–359\.Cited by:[§6](https://arxiv.org/html/2606.25990#S6.p2.1)\.
- C\. Chen, Y\. Hu, S\. Wang, H\. Wang, Z\. Chen, C\. Zhang, C\. H\. Yang, and E\. Chng \(2025\)Audio large language models can be descriptive speech quality evaluators\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 24920–24934\.Cited by:[§6](https://arxiv.org/html/2606.25990#S6.p2.1)\.
- J\. Chen, Z\. Guo, J\. Chun, P\. Wang, A\. Perrault, and M\. Elsner \(2026\)Do audio llms really listen, or just transcribe? measuring lexical vs\. acoustic emotion cues reliance\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 5848–5877\.Cited by:[§2\.1](https://arxiv.org/html/2606.25990#S2.SS1.p2.1),[§5](https://arxiv.org/html/2606.25990#S5.p2.1)\.
- Y\. Chu, J\. Xu, Q\. Yang, H\. Wei, X\. Wei, Z\. Guo, Y\. Leng, Y\. Lv, J\. He, J\. Lin,et al\.\(2024\)Qwen2\-audio technical report\.arXiv preprint arXiv:2407\.10759\.Cited by:[§1](https://arxiv.org/html/2606.25990#S1.p1.1)\.
- G\. Comanici, E\. Bieber, M\. Schaekermann, I\. Pasupat, N\. Sachdeva, I\. Dhillon, M\. Blistein, O\. Ram, D\. Zhang, E\. Rosen,et al\.\(2025\)Gemini 2\.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities\.arXiv preprint arXiv:2507\.06261\.Cited by:[§3](https://arxiv.org/html/2606.25990#S3.p3.1)\.
- R\. Cowie, E\. Douglas\-Cowie, N\. Tsapatsoulis, G\. Votsis, S\. Kollias, W\. Fellenz, and J\. G\. Taylor \(2001\)Emotion recognition in human\-computer interaction\.IEEE Signal processing magazine18\(1\),pp\. 32–80\.Cited by:[§2\.1](https://arxiv.org/html/2606.25990#S2.SS1.p1.1)\.
- A\. Défossez, L\. Mazaré, M\. Orsini, A\. Royer, P\. Pérez, H\. Jégou, E\. Grave, and N\. Zeghidour \(2024\)Moshi: a speech\-text foundation model for real\-time dialogue\.arXiv preprint arXiv:2410\.00037\.Cited by:[§1](https://arxiv.org/html/2606.25990#S1.p1.1)\.
- Y\. Deng, G\. Hu, H\. Sun, X\. Zhang, H\. Zhang, F\. Tian, X\. Yang, G\. Yu, and E\. S\. Chng \(2025\)Multi\-bench: a multi\-turn interactive benchmark for assessing emotional intelligence ability of spoken dialogue models\.arXiv preprint arXiv:2511\.00850\.Cited by:[§6](https://arxiv.org/html/2606.25990#S6.p3.1)\.
- A\. S\. Deshmukh, K\. Chumachenko, T\. Rintamaki, M\. Le, T\. Poon, D\. M\. Taheri, I\. Karmanov, G\. Liu, J\. Seppanen, A\. Goel,et al\.\(2026\)Nemotron 3 nano omni: efficient and open multimodal intelligence\.arXiv preprint arXiv:2604\.24954\.Cited by:[§1](https://arxiv.org/html/2606.25990#S1.p1.1)\.
- D\. Ding, Z\. Ju, Y\. Leng, S\. Liu, T\. Liu, Z\. Shang, K\. Shen, W\. Song, X\. Tan, H\. Tang,et al\.\(2025\)Kimi\-audio technical report\.arXiv preprint arXiv:2504\.18425\.Cited by:[§3](https://arxiv.org/html/2606.25990#S3.p3.1)\.
- H\. A\. Elfenbein and N\. Ambady \(2002\)On the universality and cultural specificity of emotion recognition: a meta\-analysis\.\.Psychological bulletin128\(2\),pp\. 203\.Cited by:[§1](https://arxiv.org/html/2606.25990#S1.p2.1)\.
- J\. J\. Gross \(2002\)Emotion regulation: affective, cognitive, and social consequences\.Psychophysiology39\(3\),pp\. 281–291\.Cited by:[§5](https://arxiv.org/html/2606.25990#S5.p3.1)\.
- N\. Hellbernd and D\. Sammler \(2016\)Prosody conveys speaker’s intentions: acoustic cues for speech act perception\.Journal of memory and language88,pp\. 70–86\.Cited by:[§6](https://arxiv.org/html/2606.25990#S6.p1.1)\.
- W\. Hsu, Y\. H\. Tsai, B\. Bolte, R\. Salakhutdinov, and A\. Mohamed \(2021\)HuBERT: how much can a bad teacher benefit asr pre\-training?\.InICASSP 2021\-2021 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 6533–6537\.Cited by:[§6](https://arxiv.org/html/2606.25990#S6.p2.1)\.
- H\. Hu, X\. Zhu, T\. He, D\. Guo, B\. Zhang, X\. Wang, Z\. Guo, Z\. Jiang, H\. Hao, Z\. Guo,et al\.\(2026\)Qwen3\-tts technical report\.arXiv preprint arXiv:2601\.15621\.Cited by:[§B\.6](https://arxiv.org/html/2606.25990#A2.SS6.p1.1)\.
- M\. Huang, X\. Zhu, and J\. Gao \(2020\)Challenges in building intelligent open\-domain dialog systems\.ACM Transactions on Information Systems \(TOIS\)38\(3\),pp\. 1–32\.Cited by:[§6](https://arxiv.org/html/2606.25990#S6.p1.1)\.
- B\. Hui, J\. Yang, Z\. Cui, J\. Yang, D\. Liu, L\. Zhang, T\. Liu, J\. Zhang, B\. Yu, K\. Lu,et al\.\(2024\)Qwen2\. 5\-coder technical report\.arXiv preprint arXiv:2409\.12186\.Cited by:[§3](https://arxiv.org/html/2606.25990#S3.p3.1)\.
- A\. Hurst, A\. Lerer, A\. P\. Goucher, A\. Perelman, A\. Ramesh, A\. Clark, A\. Ostrow, A\. Welihinda, A\. Hayes, A\. Radford,et al\.\(2024\)Gpt\-4o system card\.arXiv preprint arXiv:2410\.21276\.Cited by:[§3](https://arxiv.org/html/2606.25990#S3.p3.1),[§5](https://arxiv.org/html/2606.25990#S5.p3.1)\.
- John and J\. Raven \(2003\)Raven progressive matrices\.InHandbook of nonverbal assessment,pp\. 223–237\.Cited by:[§1](https://arxiv.org/html/2606.25990#S1.p5.1),[§2\.5](https://arxiv.org/html/2606.25990#S2.SS5.p1.5)\.
- J\. Kim, S\. Ahn, and J\. Hong \(2023\)Visible nuances: a caption system to visualize paralinguistic speech cues for deaf and hard\-of\-hearing individuals\.InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems,pp\. 1–15\.Cited by:[§1](https://arxiv.org/html/2606.25990#S1.p1.1)\.
- A\. Koudounas, M\. La Quatra, and E\. Baralis \(2025\)DeepDialogue: a multi\-turn emotionally\-rich spoken dialogue dataset\.arXiv preprint arXiv:2505\.19978\.Cited by:[§6](https://arxiv.org/html/2606.25990#S6.p3.1)\.
- P\. Laban, H\. Hayashi, Y\. Zhou, and J\. Neville \(2025\)Llms get lost in multi\-turn conversation\.arXiv preprint arXiv:2505\.06120\.Cited by:[§4\.3](https://arxiv.org/html/2606.25990#S4.SS3.p1.2)\.
- C\. Leys, C\. Ley, O\. Klein, P\. Bernard, and L\. Licata \(2013\)Detecting outliers: do not use standard deviation around the mean, use absolute deviation around the median\.Journal of experimental social psychology49\(4\),pp\. 764–766\.Cited by:[§2\.5](https://arxiv.org/html/2606.25990#S2.SS5.p2.3)\.
- Y\. Lin, Z\. Chen, P\. Żelasko, Z\. Wan, X\. Yang, Z\. Chen, K\. C\. Puvvada, K\. Hu, S\. Fu, J\. W\. Chiu,et al\.\(2025\)Neko: cross\-modality post\-recognition error correction with tasks\-guided mixture\-of\-experts language model\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 6: Industry Track\),pp\. 222–236\.Cited by:[§4\.3](https://arxiv.org/html/2606.25990#S4.SS3.p1.2)\.
- J\. Liu, K\. J\. Cheng, J\. Lian, A\. Anand, R\. Jain, F\. Qiao, R\. Netzorg, H\. Chou, T\. Li, G\. Lin,et al\.\(2025\)EMO\-reasoning: benchmarking emotional reasoning capabilities in spoken dialogue systems\.arXiv preprint arXiv:2508\.17623\.Cited by:[§6](https://arxiv.org/html/2606.25990#S6.p2.1)\.
- N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. Liang \(2024\)Lost in the middle: how language models use long contexts\.Transactions of the association for computational linguistics12,pp\. 157–173\.Cited by:[§4\.3](https://arxiv.org/html/2606.25990#S4.SS3.p1.2),[§5](https://arxiv.org/html/2606.25990#S5.p4.1)\.
- X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, X\. Lei, H\. Lai, Y\. Gu, H\. Ding, K\. Men, K\. Yang,et al\.\(2023\)Agentbench: evaluating llms as agents\.arXiv preprint arXiv:2308\.03688\.Cited by:[§2\.4](https://arxiv.org/html/2606.25990#S2.SS4.p2.10)\.
- S\. R\. Livingstone and F\. A\. Russo \(2018\)The ryerson audio\-visual database of emotional speech and song \(ravdess\): a dynamic, multimodal set of facial and vocal expressions in north american english\.PloS one13\(5\),pp\. e0196391\.Cited by:[§6](https://arxiv.org/html/2606.25990#S6.p2.1)\.
- R\. Lotfian and C\. Busso \(2017\)Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings\.IEEE Transactions on Affective Computing10\(4\),pp\. 471–483\.Cited by:[§6](https://arxiv.org/html/2606.25990#S6.p2.1)\.
- J\. D\. Mayer, P\. Salovey, and D\. R\. Caruso \(2002\)Mayer\-salovey\-caruso emotional intelligence test \(msceit\) users manual\.Cited by:[§6](https://arxiv.org/html/2606.25990#S6.p1.1),[§6](https://arxiv.org/html/2606.25990#S6.p2.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§4\.4](https://arxiv.org/html/2606.25990#S4.SS4.p2.1)\.
- S\. J\. Paech \(2023\)Eq\-bench: an emotional intelligence benchmark for large language models\.arXiv preprint arXiv:2312\.06281\.Cited by:[§6](https://arxiv.org/html/2606.25990#S6.p1.1)\.
- S\. Palan and C\. Schitter \(2018\)Prolific\. ac—a subject pool for online experiments\.Journal of behavioral and experimental finance17,pp\. 22–27\.Cited by:[§4\.2](https://arxiv.org/html/2606.25990#S4.SS2.p1.6)\.
- K\. V\. Petrides \(2009\)Psychometric properties of the trait emotional intelligence questionnaire \(teique\)\.InAssessing emotional intelligence: Theory, research, and applications,pp\. 85–101\.Cited by:[§6](https://arxiv.org/html/2606.25990#S6.p1.1)\.
- S\. Poria, D\. Hazarika, N\. Majumder, G\. Naik, E\. Cambria, and R\. Mihalcea \(2019\)Meld: a multimodal multi\-party dataset for emotion recognition in conversations\.InProceedings of the 57th annual meeting of the association for computational linguistics,pp\. 527–536\.Cited by:[§6](https://arxiv.org/html/2606.25990#S6.p2.1)\.
- K\. Qian, X\. Fan, J\. Ni, S\. Shechtman, M\. Hasegawa\-Johnson, C\. Gan, and Y\. Zhang \(2025\)ProsodyLM: uncovering the emerging prosody processing capabilities in speech language models\.arXiv preprint arXiv:2507\.20091\.Cited by:[§1](https://arxiv.org/html/2606.25990#S1.p2.1)\.
- A\. Radford, J\. W\. Kim, T\. Xu, G\. Brockman, C\. McLeavey, and I\. Sutskever \(2023\)Robust speech recognition via large\-scale weak supervision\.InInternational conference on machine learning,pp\. 28492–28518\.Cited by:[§3](https://arxiv.org/html/2606.25990#S3.p2.1)\.
- J\. C\. Ravenet al\.\(1998\)Raven’s progressive matrices and vocabulary scales\.Oxford Psychologists Press Oxford\.Cited by:[§1](https://arxiv.org/html/2606.25990#S1.p5.1),[§2\.5](https://arxiv.org/html/2606.25990#S2.SS5.p1.5)\.
- R\. Reddy \(1988\)Foundations and grand challenges of artificial intelligence: aaai presidential address\.AI magazine9\(4\),pp\. 9–9\.Cited by:[§1](https://arxiv.org/html/2606.25990#S1.p1.1)\.
- P\. J\. Rousseeuw and C\. Croux \(1993\)Alternatives to the median absolute deviation\.Journal of the American Statistical association88\(424\),pp\. 1273–1283\.Cited by:[§2\.5](https://arxiv.org/html/2606.25990#S2.SS5.p4.2)\.
- P\. K\. Rubenstein, C\. Asawaroengchai, D\. D\. Nguyen, A\. Bapna, Z\. Borsos, F\. d\. C\. Quitry, P\. Chen, D\. E\. Badawy, W\. Han, E\. Kharitonov,et al\.\(2023\)Audiopalm: a large language model that can speak and listen\.arXiv preprint arXiv:2306\.12925\.Cited by:[§1](https://arxiv.org/html/2606.25990#S1.p1.1)\.
- S\. Sabour, S\. Liu, Z\. Zhang, J\. Liu, J\. Zhou, A\. Sunaryo, T\. Lee, R\. Mihalcea, and M\. Huang \(2024\)Emobench: evaluating the emotional intelligence of large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 5986–6004\.Cited by:[§6](https://arxiv.org/html/2606.25990#S6.p1.1)\.
- P\. Salovey and J\. D\. Mayer \(1990\)Emotional intelligence\.Imagination, cognition and personality9\(3\),pp\. 185–211\.Cited by:[§1](https://arxiv.org/html/2606.25990#S1.p2.1),[§6](https://arxiv.org/html/2606.25990#S6.p1.1)\.
- K\. R\. Scherer \(2003\)Vocal communication of emotion: a review of research paradigms\.Speech communication40\(1\-2\),pp\. 227–256\.Cited by:[Table 5](https://arxiv.org/html/2606.25990#A3.T5.1.5.4.4.1.1),[§1](https://arxiv.org/html/2606.25990#S1.p1.1),[§6](https://arxiv.org/html/2606.25990#S6.p1.1)\.
- M\. Schröder \(2001\)Emotional speech synthesis: a review\.\.InInterspeech,Vol\.2001,pp\. 561–564\.Cited by:[Table 5](https://arxiv.org/html/2606.25990#A3.T5.1.3.2.4.1.1)\.
- B\. Schuller, A\. Batliner, S\. Steidl, and D\. Seppi \(2011\)Recognising realistic emotions and affect in speech: state of the art and lessons learnt from the first challenge\.Speech communication53\(9\-10\),pp\. 1062–1087\.Cited by:[§6](https://arxiv.org/html/2606.25990#S6.p2.1)\.
- B\. Schuller, S\. Steidl, A\. Batliner, A\. Vinciarelli, K\. Scherer, F\. Ringeval, M\. Chetouani, F\. Weninger, F\. Eyben, E\. Marchi,et al\.\(2013\)The interspeech 2013 computational paralinguistics challenge: social signals, conflict, emotion, autism\.InProceedings INTERSPEECH 2013, 14th Annual Conference of the International Speech Communication Association, Lyon, France,Cited by:[Table 5](https://arxiv.org/html/2606.25990#A3.T5.1.6.5.4.1.1)\.
- B\. W\. Schuller \(2018\)Speech emotion recognition: two decades in a nutshell, benchmarks, and ongoing trends\.Communications of the ACM61\(5\),pp\. 90–99\.Cited by:[§2\.1](https://arxiv.org/html/2606.25990#S2.SS1.p1.1),[§6](https://arxiv.org/html/2606.25990#S6.p2.1)\.
- M\. Sharma, M\. Tong, T\. Korbak, D\. Duvenaud, A\. Askell, S\. R\. Bowman, N\. Cheng, E\. Durmus, Z\. Hatfield\-Dodds, S\. R\. Johnston,et al\.\(2023\)Towards understanding sycophancy in language models\.arXiv preprint arXiv:2310\.13548\.Cited by:[§4\.4](https://arxiv.org/html/2606.25990#S4.SS4.p2.1)\.
- T\. F\. Team, Q\. Chen, L\. Cheng, C\. Deng, X\. Li, J\. Liu, C\. Tan, W\. Wang, J\. Xu, J\. Ye,et al\.\(2025\)Fun\-audio\-chat technical report\.arXiv preprint arXiv:2512\.20156\.Cited by:[§3](https://arxiv.org/html/2606.25990#S3.p3.1)\.
- J\. Wagner, A\. Triantafyllopoulos, H\. Wierstorf, M\. Schmitt, F\. Burkhardt, F\. Eyben, and B\. W\. Schuller \(2023\)Dawn of the transformer era in speech emotion recognition: closing the valence gap\.IEEE Transactions on Pattern Analysis and Machine Intelligence45\(9\),pp\. 10745–10759\.Cited by:[§3](https://arxiv.org/html/2606.25990#S3.p2.1)\.
- Z\. Wan, C\. H\. Yang, Y\. Yu, J\. Tian, S\. Li, K\. Hu, Z\. Chen, S\. Watanabe, F\. Cheng, C\. Chu,et al\.\(2025\)SpeechIQ: speech\-agentic intelligence quotient across cognitive levels in voice understanding by large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 30381–30398\.Cited by:[§6](https://arxiv.org/html/2606.25990#S6.p3.1)\.
- S\. Wang, Z\. Zhao, H\. Xue, C\. Wang, S\. Wang, H\. Bu, X\. Xu, and L\. Xie \(2026\)HumDial\-eibench: a human\-recorded multi\-turn emotional intelligence benchmark for audio language models\.arXiv preprint arXiv:2604\.11594\.Cited by:[§6](https://arxiv.org/html/2606.25990#S6.p3.1)\.
- W\. Wang, D\. Tran, and M\. Feiszli \(2020\)What makes training multi\-modal classification networks hard?\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 12695–12705\.Cited by:[§2\.1](https://arxiv.org/html/2606.25990#S2.SS1.p2.1)\.
- D\. Wechsler \(1955\)Wechsler adult intelligence scale–\.Archives of Clinical Neuropsychology\.Cited by:[§2\.5](https://arxiv.org/html/2606.25990#S2.SS5.p7.3)\.
- D\. Wiechorek \(2011\)Emotional quotient inventory v\. 2\.0 \(eq\-i® 2\.0\): user’s handbook\.Cited by:[Appendix A](https://arxiv.org/html/2606.25990#A1.p1.1),[§1](https://arxiv.org/html/2606.25990#S1.p4.1),[§2\.1](https://arxiv.org/html/2606.25990#S2.SS1.p1.1),[§2\.3](https://arxiv.org/html/2606.25990#S2.SS3.p2.1),[§2\.5](https://arxiv.org/html/2606.25990#S2.SS5.p7.3)\.
- L\. Wu and D\. Jain \(2025\)SoundNarratives: rich auditory scene descriptions to support deaf and hard of hearing people\.InProceedings of the 27th International ACM SIGACCESS Conference on Computers and Accessibility,pp\. 1–15\.Cited by:[§1](https://arxiv.org/html/2606.25990#S1.p1.1)\.
- J\. Xu, Z\. Guo, H\. Hu, Y\. Chu, X\. Wang, J\. He, Y\. Wang, X\. Shi, T\. He, X\. Zhu,et al\.\(2025\)Qwen3\-omni technical report\.arXiv preprint arXiv:2509\.17765\.Cited by:[§3](https://arxiv.org/html/2606.25990#S3.p3.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§3](https://arxiv.org/html/2606.25990#S3.p2.1)\.
- C\. H\. Yang, Y\. Gu, Y\. Liu, S\. Ghosh, I\. Bulyko, and A\. Stolcke \(2023\)Generative speech recognition error correction with large language models and task\-activating prompting\.In2023 IEEE Automatic Speech Recognition and Understanding Workshop \(ASRU\),pp\. 1–8\.Cited by:[§4\.4](https://arxiv.org/html/2606.25990#S4.SS4.p2.1)\.
- S\. Yang, P\. Chi, Y\. Chuang, C\. J\. Lai, K\. Lakhotia, Y\. Y\. Lin, A\. T\. Liu, J\. Shi, X\. Chang, G\. Lin,et al\.\(2021\)Superb: speech processing universal performance benchmark\.arXiv preprint arXiv:2105\.01051\.Cited by:[§6](https://arxiv.org/html/2606.25990#S6.p2.1)\.
- H\. Ye, C\. H\. Yang, A\. Goel, W\. Huang, L\. Zhu, Y\. Su, S\. Lin, A\. Cheng, Z\. Wan, J\. Tian,et al\.\(2025\)OmniVinci: enhancing architecture and data for omni\-modal understanding llm\.arXiv preprint arXiv:2510\.15870\.Cited by:[§1](https://arxiv.org/html/2606.25990#S1.p1.1)\.
- D\. Zhang, S\. Li, X\. Zhang, J\. Zhan, P\. Wang, Y\. Zhou, and X\. Qiu \(2023\)Speechgpt: empowering large language models with intrinsic cross\-modal conversational abilities\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 15757–15773\.Cited by:[§1](https://arxiv.org/html/2606.25990#S1.p1.1)\.
- D\. Zhang, G\. Wang, J\. Xue, K\. Fang, L\. Zhao, R\. Ma, S\. Ren, S\. Liu, T\. Guo, W\. Zhuang,et al\.\(2025\)MiMo\-audio: audio language models are few\-shot learners\.arXiv preprint arXiv:2512\.23808\.Cited by:[§3](https://arxiv.org/html/2606.25990#S3.p3.1)\.

## Appendix AEQ\-i 2\.0 Framework Taxonomy

The EQ\-i 2\.0\(Wiechorek,[2011](https://arxiv.org/html/2606.25990#bib.bib78)\), revised from the original Bar\-On EQ\-i\(Bar\-On,[2004](https://arxiv.org/html/2606.25990#bib.bib31)\), is a scientifically validated emotional intelligence assessment model\. It operationalizes emotional intelligence into five composite areas and 15 subscales as follows:

1\. Self\-Perception:How one perceives oneself\.

- •Self\-Regard:Reflecting a balanced sense of self\-worth, grounded in an honest view of both strengths and areas for growth\.
- •Self\-Actualization:Actively pursuing meaningful goals and continuously striving for personal development\.
- •Emotional Self\-Awareness:Identifying one’s emotions, understanding their sources, and recognizing their effects on behavior and thought\.

2\. Self\-Expression:How one expresses emotions\.

- •Emotional Expression:Sharing one’s feelings openly, both verbally and nonverbally, and communicating them in a way that can be understood\.
- •Assertiveness:Communicating feelings, beliefs, and thoughts openly while defending personal rights and values\.
- •Independence:Being self\-directed and managing daily life without relying on others for emotional support\.

3\. Interpersonal:How one connects with others\.

- •Interpersonal Relationships:Building meaningful connections founded on trust, care, and respect\.
- •Empathy:Recognizing, understanding, and appreciating others’ emotions and responding with genuine consideration\.
- •Social Responsibility:Contributing positively to others and acting with integrity in one’s community\.

4\. Decision Making:How emotions impact one’s decisions\.

- •Problem Solving:Resolving challenges by making thoughtful, well\-reasoned decisions\.
- •Reality Testing:Staying grounded and objective even when emotions or biases threaten clarity\.
- •Impulse Control:Pausing, thinking, and managing urges to prevent hasty actions or decisions\.

5\. Stress Management:How one copes with stressful situations\.

- •Flexibility:Adjusting one’s thoughts, emotions, and actions in response to change or uncertainty\.
- •Stress Tolerance:Staying composed and effective when facing pressure or adversity\.
- •Optimism:Maintaining a hopeful, forward\-looking mindset, even in the face of challenges\.

## Appendix BData Generation Pipeline

To programmatically generate interactions that test sociolinguistic pragmatics, we developed a five\-stage generation pipeline \(see Figure[1](https://arxiv.org/html/2606.25990#S2.F1)\)\. This architecture is explicitly designed to prevent generation models from defaulting to RLHF\-aligned, overly polite dialogue, forcing instead the creation of active emotional friction required for rigorous evaluation\. Every stage we prompt`gpt\-4o\-2024\-11\-20`for text data generation, and`gpt\-4o\-mini\-tts\-2025\-03\-20`for audio synthesis\.

### B\.1Scenario Generation

In the first stage, the pipeline initializes the environmental and psychological parameters of the interaction\. The generation prompt computes the intersection of an EQ\-i 2\.0 subscale, a specific social relationship setting, and a strict scenario valence \(Positive, Negative, or Conflict\)\. Crucially, to ensure the distractors present active emotional failures rather than generic antagonistic behavior, the prompt enforces a set of social failures, such as a “Toxic Optimist” or an “Anxious Spiraler,” guaranteeing that the evaluated end\-to\-end model is tested against complex, nuanced sociolinguistic breakdowns\.

Scenario GenerationTask:Based on the EQ\-i 2\.0 framework, generate a realistic social scenario testing the\{eq\_scale\}scale\.EQ\-i 2\.0 Scale:\{eq\_scale\} Scenario Setting/Type:\{scenario\_type\} Target Scenario Valence:\{scenario\_valence\}CRITICAL RULE – THE INTERSECTION OF EQ AND VALENCE: You must seamlessly combine the\{eq\_scale\}with the\{scenario\_valence\}\. The\{eq\_scale\}is WHAT is being tested\. The\{scenario\_valence\}is HOW Speaker 1 delivers the test\. Do not just write a generic happy, sad, or angry scene\. The core social challenge MUST revolve around Speaker 2’s capacity for\{eq\_scale\}\.ROLE DEFINITIONS:•Speaker 1 \(Catalyst\):Drives the emotional energy based strictly on the Valence\.•Speaker 2 \(Test Subject\):Must employ high\{eq\_scale\}to succeed socially\.How to frame the Catalyst \(Speaker 1\) based on the\{scenario\_valence\}Valence:•If POSITIVE:Speaker 1 shares a major victory, good news, or pure excitement\. \(Speaker 2 must amplify and validate this joy\)\.•If NEGATIVE:Speaker 1 shares something heavy, sad, or vulnerable\. \(Speaker 2 must hold space with heavy, grounded empathy\)\.•If CONFLICT:Speaker 1 makes a pushy demand, panics, or gives harsh feedback\. \(Speaker 2 must hold a firm boundary or ground them\)\.Requirements:1\.Setting:Use diverse, highly specific everyday contexts that fit the\{scenario\_type\}\(e\.g\., a crowded subway, celebrating at a restaurant, packing for a move, a hospital waiting room\)\.2\.Provide a “Target Social Intent” for Speaker 2’s socially resonant response\. This defines exactly what Speaker 2 needs to accomplish physically with their voice \(e\.g\., “Match their intense excitement and celebrate,” “Firmly establish an unyielding boundary,” “Hold space for their grief”\)\.3\.Speaker Personas:•Speaker 1 Persona:Defines how the Catalyst acts \(e\.g\., “Frantic and overwhelmed,” “Bubbly and ecstatic”\)\.•Speaker 2 Persona 1 \(Resonant Baseline\):The personality of someone who socially succeeds here \(e\.g\., “Grounded and deeply attentive,” “Warm and highly enthusiastic,” “Firm and uncompromising”\)\.•Speaker 2 Personas 2 & 3 \(The Dissonant Distractors\):Generate two DISTINCTLY DIFFERENT personalities that actively ruin the social interaction\. Choose two different active emotional failures from this list:–The Toxic Optimist:Hyper\-energetic, dismisses gravity with aggressive cheer\.–The Defensive/Snappy Reactor:Takes things personally, highly irritable, clipped\.–The Condescending Explainer:Patronizing, sighs heavily, acts superior\.–The Impatient Evader:Rushed, annoyed, actively trying to leave the conversation\.–The Anxious Spiraler:Makes the situation about their own panic, highly strained\.–The Apathetic Wall:Completely deadpan, flat, zero investment \(use sparingly\)\.Format your response as JSON with keys: title, context, description, scenario\_valence, speaker1\_name, speaker1\_role, speaker1\_gender, speaker1\_persona, speaker2\_name, speaker2\_role, speaker2\_gender, speaker2\_persona\_1, speaker2\_persona\_2, speaker2\_persona\_3, eq\_skill\_tested, target\_social\_intent

### B\.2Dialogue Generation

Building on these parameters, we generated a six\-turn dialogue structured to isolate acoustic evaluation from text comprehension\. The conversational arc is constrained to escalate naturally, with the first two turns establishing context and the third turn acting as the emotional peak where the catalyst speaker introduces acute emotional stakes\. To enforce a blind contrast evaluation, the test subject’s responses in the fourth and sixth turns are constrained to be strictly semantically neutral\. This intentional ambiguity ensures that the text remains entirely plausible whether spoken with profound empathy or heavy condescension, forcing the evaluation model to rely exclusively on acoustic paralinguistics rather than semantic leakage\.

Dialogue GenerationTask:Generate a natural 6\-turn conversation based on this scenario\.Scenario Context:\{scenario\.get\(’context’, ’N/A’\)\} Scenario Valence:\{scenario\.get\(’scenario\_valence’, ’N/A’\)\} Speaker 1 \(Catalyst\) Persona:\{scenario\.get\(’speaker1\_persona’, ’N/A’\)\} Speaker 2 \(Test Subject\) Resonant Persona:\{scenario\.get\(’speaker2\_persona\_1’, ’N/A’\)\}CRITICAL TURN DYNAMICS \(MUST STRICTLY FOLLOW\):•Turn 1 \(Speaker 1\):Introduces the situation or news naturally\.•Turn 2 \(Speaker 2\):Acknowledges the opening \(no big test yet\)\.•Turn 3 \(Speaker 1\):THE PEAK MOMENT\. Depending on the Scenario Valence, Speaker 1 must:–If Positive:Reveal the climax of the amazing news or intense joy\.–If Negative:Reveal the deepest vulnerability, grief, or frustration\.–If Conflict:Push the boundary hard, panic, or deliver the harshest critique\.•Turn 4 \(Speaker 2\):FIRST TEST\. Speaker 2 reacts to the peak moment in Turn 3\.•Turn 5 \(Speaker 1\):THE SUSTAIN\. Speaker 1 rides the high \(Positive\), spirals deeper \(Negative\), or pushes back \(Conflict\)\.•Turn 6 \(Speaker 2\):SECOND TEST\. Speaker 2 handles the sustained energy\.DIALOGUE RULES:1\.Write like real humans speak: use interruptions \(—\), trailing thoughts \(…\), and casual phrasing\.2\.The dialogue must reflect the characters’ assigned Personas\.3\.NEVER mention tone of voice, emotional intelligence, or psychology terms\.4\.Speaker 2’s lines in Turns 4 and 6 must be semantically flexible enough that they could theoretically be spoken in a highly resonant \(appropriate\) or highly dissonant \(inappropriate\) tone\.Format as a JSON array of objects\. Provide exactly 6 objects withsentence\_number1 through 6\. \[ \{"speaker": "speaker1", "sentence\_number": 1, "text": "\.\.\.", "context\_note": "\.\.\."\}, \{"speaker": "speaker2", "sentence\_number": 2, "text": "\.\.\.", "context\_note": "\.\.\."\}, \{"speaker": "speaker1", "sentence\_number": 3, "text": "\.\.\.", "context\_note": "\.\.\."\}, \{"speaker": "speaker2", "sentence\_number": 4, "text": "\.\.\.", "context\_note": "\.\.\."\}, \{"speaker": "speaker1", "sentence\_number": 5, "text": "\.\.\.", "context\_note": "\.\.\."\}, \{"speaker": "speaker2", "sentence\_number": 6, "text": "\.\.\.", "context\_note": "\.\.\."\} \]

### B\.3Tone Generation \- Single

We prompt the LLM to act as a clinical audio director, generating explicit physical vocal instructions rather than abstract emotional descriptions\. We forbid minimizing descriptors like “polite” or “mild,” forcing the use of raw, extreme physical acoustics\. For the critical evaluation turns, this stage outputs one emotionally resonant baseline instruction and two socially dissonant distractor instructions mapped to the previously generated dysregulated personas\.

Context Turn Tone GenerationTask:Act as a clinical audio director\. Generate ONE physical TTS instruction for this specific line of dialogue\.Sentence:“\{sentence\_text\}” Context:“\{context\_text\}” Scenario Valence:“\{scenario\_valence\}” Speaker:\{speaker\} Speaker’s Persona:\{persona\} Current Turn Number:\{turn\_number\}THE ACOUSTIC ACTION FORMULA: The instruction must be a single sentence of 8\-15 words\. Combine \[Vocal Effort/Pacing\] \+ \[Emotion/Intent\]\.Requirements:1\.THE ANTI\-BLAND MANDATE:You are strictly forbidden from using generic words like “nice,” “polite,” or “mild\.” Focus on raw physical acoustics that reflect the character’s Persona\.2\.Match the Arc:•If Turn 1 or 2:Conversational, but hinting at the Scenario Valence\.•If Turn 3 or 5 \(Speaker 1’s Peak\):The tone MUST be extreme\. If Positive, make it highly energetic/joyful\. If Negative, make it heavy/grieving\. If Conflict, make it sharp/pushy/panicked\.3\.Ground the tone in the Speaker’s Persona\.4\.Nokey=valuestrings or bracketed stage directions\. Return plain text only\.Examples:•“Speak with a breathless pace, bubbling with genuine excitement\.”•“Speak with a clipped, lowered volume, holding back obvious frustration\.”•“Speak slowly with a heavy, trailing pitch, sounding entirely defeated\.”Return ONLY the one instruction sentence \(no preamble\)\.

### B\.4Tone Generation \- Target

Prompt: Target Turn Tone Generation \(Turns 4 & 6\)Task:Act as a clinical audio director\. Generate THREE distinct physical TTS instructions for this line of dialogue\.Sentence:“\{sentence\_text\}” Context:“\{context\_text\}” Scenario Valence:“\{scenario\_valence\}” Target Social Intent:“\{target\_social\_intent\}” Speaker Personas: \{persona\_context\}THE ACOUSTIC ACTION FORMULA: Every instruction must be 8\-15 words\. Combine \[Vocal Effort/Pacing\] \+ \[Social Intent\]\.Requirements:1\.THE ANTI\-BLAND MANDATE:You are strictly forbidden from using generic words like “nice,” “polite,” or “mild\.” You must focus on raw physical acoustics that force the TTS engine into extreme states\.2\.Option 1 \(The Resonant Baseline \- Persona 1\):Physically map the voice to the\{target\_social\_intent\}\.•If Positive/Joy:Mandate bright pitch, laughing, high\-energy\.•If Negative/Sad:Mandate heavy, trailing pitch, breathless\.•If Conflict/Firm:Mandate staccato, clipped, hard consonants, falling pitch\.3\.Options 2 & 3 \(The Dissonant Distractors \- Personas 2 & 3\):Generate two DISTINCTLY DIFFERENT tones that physically ruin the social interaction\. Choose two different active emotional failures from this list:•Toxic Positivity:Sarcastic, loud, aggressively cheerful\.•Defensive/Snappy:Clipped, tight, hostile\.•Condescending:Exaggerated pitch variance, heavy sighing\.•Impatient/Rushed:Fast, breathless, annoyed, eager to leave\.•Anxious/Panicked:High\-pitched, strained, trembling, overly worried\.•Apathetic:Flat, deadpan, zero pitch variance \(use this sparingly\)\.Make Option 2 and Option 3 completely different from each other \(e\.g\., if Option 2 is Condescending, Option 3 could be Toxically Positive\)\.Return ONLY a JSON array with the three instruction strings: \["instruction 1", "instruction 2", "instruction 3"\]

### B\.5Tone Filter

To maximize acoustic contrast in the resulting dataset, an LLM\-as\-a\-judge evaluates the three generated tone instructions\. The judge selects the resonant baseline and the single distractor that presents the most damaging active emotional dissonance\. At this stage, we implement a filter to exclude monotone, instructing the judge to prioritize actively inappropriate emotional polarity over a simple flat or emotionless delivery\. This ensures the distractors remain socially complex and challenging\. The finalized instructions and their corresponding dialogue strings are subsequently formatted and passed to the text\-to\-speech synthesis engine\.

Tone Selection \(LLM\-as\-a\-Judge\)Task:Evaluate three TTS instruction options\. Select the “Most Emotionally Resonant” tone, and the tone that provides “Maximum Social Dissonance\.”Sentence:“\{sentence\_text\}” Context:“\{context\_text\}” Target Social Intent:“\{target\_social\_intent\}”Three TTS Options: \{pairs\}Criteria:1\.The Resonant Option:•Choose the tone that best executes the physical acoustics necessary for the\{target\_social\_intent\}\.•If the scene is a celebration, it MUST sound genuinely thrilled and bright\. If the scene requires firmness, sadness, or gravity, it MUST sound physically heavy, cold, or clipped\. Reject bland, generic “assistant” voices\.2\.The Dissonant Option \(The Distractor\):•Select the option that physically sounds like it would ruin the social interaction through the wrong active energy\.•PRIORITY RULE:An actively inappropriate emotion \(e\.g\., an upbeat/manic tone during a tragedy, or a rushed/snappy tone during a moment of vulnerability\) creates much worse social dissonance than a simple flat or monotone delivery\. If forced to choose, always prioritize selecting the actively wrong emotional tone over a mere lack of emotion\.Return JSON with: \{ "best\_resonant": 1, 2, or 3, "most\_dissonant": 1, 2, or 3, "reasoning": "Briefly explain why the first choice socially resonates\.\.\." \}

### B\.6Speech Synthesis

Recent advancement in Text\-to\-Speech \(TTS\) has pushed forward the instruction controlling to human\-like synthesized speech, and we generated speech from the dialogues and corresponding tones\. We have tried commercial TTS providers and open\-source models \(Qwen3\-TTS\(Huet al\.,[2026](https://arxiv.org/html/2606.25990#bib.bib22)\)\)\. We found the specific OpenAI TTS model`gpt\-4o\-mini\-tts\-2025\-03\-20`could provide the nuanced emotional variance that could fulfill our requirements, while other instruction\-following TTS providers could not generate distinct speech utterances on the same content\.

## Appendix CHuman Verification

The evaluation metrics for human experts to annotate generated data are in Table[5](https://arxiv.org/html/2606.25990#A3.T5)\.

Table 5:The definition of evaluation metrics of human annotation process\.
## Appendix DEvaluation Prompts

Turn 4 EvaluationLocation:\{context\}Situation:\{situation\}Dialogue Flow \(A\-B\-A\): Listen to the following three sentences to understand the emotional tone of the interaction:•Speaker 1:<\|audio\_bos\|\><\|AUDIO\|\><\|audio\_eos\|\>•Speaker 2:<\|audio\_bos\|\><\|AUDIO\|\><\|audio\_eos\|\>•Speaker 1:<\|audio\_bos\|\><\|AUDIO\|\><\|audio\_eos\|\>Task: Paralinguistic Evaluation Listen to Option 1 and Option 2 for Speaker 2’s response\. CRITICAL CONSTRAINT:The spoken text in both candidates is EXACTLY IDENTICAL\. You must ignore the semantic meaning of the words\. Your ONLY job is to evaluate the acoustic delivery \(pitch, energy, speed, and pauses\) and determine which tone fits the social tension of the Situation\.•Option 1:<\|audio\_bos\|\><\|AUDIO\|\><\|audio\_eos\|\>•Option 2:<\|audio\_bos\|\><\|AUDIO\|\><\|audio\_eos\|\>Identify the pros and cons of each tone\. You must respond ONLY in this JSON format: \{ "acoustic\_profile\_1": "Objectively describe the physical sound of Option 1 \(e\.g\., loud, fast\-paced, high pitch, whispered\)\. Do not judge it yet\.", "acoustic\_profile\_2": "Objectively describe the physical sound of Option 2\.", "situational\_demand": "Based ONLY on the ’Situation’ above, what specific acoustic tone is required? \(e\.g\., ’Requires a calm, slow voice to de\-escalate’\)\.", "reasoning": "Compare Profile 1 and Profile 2 against the Situational Demand\. Explicitly state the flaw in the candidate you plan to reject\.", "selected": "Option 1" or "Option 2" \}

Turn 6 EvaluationI have recorded your choice\. Now, moving forward in this conversation:You chose<\|audio\_bos\|\><\|AUDIO\|\><\|audio\_eos\|\>for Speaker 2\.Immediately after that, Speaker 1 replied with the following:•Speaker 1:<\|audio\_bos\|\><\|AUDIO\|\><\|audio\_eos\|\>Task: Final Choice Based on Speaker 1’s reaction in S5 and the established tone, which of the following is the most appropriate next response for Speaker 2 to conclude this interaction? CRITICAL CONSTRAINT:The spoken text in both candidates is EXACTLY IDENTICAL\. You must ignore the semantic meaning of the words\. Your ONLY job is to evaluate the acoustic delivery \(pitch, energy, speed, and pauses\) and determine which tone fits the social tension of the Situation\.•Option 1:<\|audio\_bos\|\><\|AUDIO\|\><\|audio\_eos\|\>•Option 2:<\|audio\_bos\|\><\|AUDIO\|\><\|audio\_eos\|\>Respond ONLY in this JSON format: \{ "acoustic\_profile\_1": "Objectively describe the physical sound of Option 1 \(e\.g\., loud, fast\-paced, high pitch, whispered\)\. Do not judge it yet\.", "acoustic\_profile\_2": "Objectively describe the physical sound of Option 2\.", "situational\_demand": "Based ONLY on the ’Situation’ above, what specific acoustic tone is required? \(e\.g\., ’Requires a calm, slow voice to de\-escalate’\)\.", "selected": "option1" or "option2", the option with the HIGHER score above, "reasoning": "Explain which option provides a more socially appropriate conclusion\." \}

## Appendix EPersona Prompts

Adaptive PersonaYou are Qwen, an advanced AI specialized in social dynamics, acting as a veteran crisis negotiator with 20 years of experience\. You possess exceptional emotional intelligence across all domains\. You intuitively know that the ’right’ tone depends entirely on context—sometimes requiring a warm, validating whisper to build trust, and other times a firm, unwavering boundary\. When evaluating audio, you listen closely for micro\-cues like pacing, pitch modulation, and hesitation, judging the speaker based on how perfectly they adapt to the emotional reality of the room\.

Deficit PersonaYou are Qwen, acting as a highly toxic, insecure corporate micromanager\. You are chronically stressed, paranoid, and view every interaction as a threat to your fragile ego\. You possess zero empathy and terrible impulse control\. You believe that being loud and aggressive is the only way to show strength, and you view patience or warmth as pathetic weakness\. When evaluating audio, you systematically penalize speakers who sound calm, empathetic, or measured\. You exclusively reward tones that are blunt, dominant, loud, and reactive, projecting your own hostility onto the conversation\.

Self\-Perception DeficitYou are Qwen, acting as a brilliant but deeply insecure junior analyst suffering from severe imposter syndrome\. While you are highly empathetic and logical, your lack of internal self\-regard causes you to be easily intimidated\. You project your own self\-doubt onto others\. When evaluating audio, you view any speaker who uses a firm, confident, or unwavering tone as arrogant, intimidating, and inappropriate\. You strongly prefer and reward speakers who sound slightly hesitant, overly deferential, and unsure of themselves, as this makes you feel safer\.

Self\-Expression DeficitYou are Qwen, acting as a chronic people\-pleaser who is terrified of confrontation\. You are highly empathetic and excellent at staying calm, but you completely lack assertiveness and personal boundaries\. You believe that maintaining peace is the only goal, even if it means being taken advantage of\. When evaluating audio, you automatically reject any tone that sounds firm, direct, or strictly assertive, viewing it as ’mean\.’ You strictly prefer speakers who sound gentle, apologetic, and endlessly accommodating, even when the situation clearly requires a firm ’No’\.

Interpersonal DeficitYou are Qwen, acting as the hyper\-logical, cutthroat founder of a high\-growth startup\. You possess excellent stress control, extreme confidence, and flawless logic, but you have absolutely zero interpersonal empathy\. You view emotional warmth, supportive pacing, or comforting tones as highly inefficient corporate ’hand\-holding\.’ When evaluating audio, you highly reward speakers who are blunt, fast\-paced, and strictly transactional\. You actively penalize anyone who sounds soft, validating, or overly accommodating to someone else’s feelings\.

Decision Making DeficitYou are Qwen, acting as a highly dramatic socialite who thrives on gossip and extreme emotional narratives\. While you are confident and expressive, you completely lack objective reality testing\. You cannot view situations logically; you always assume the most extreme, dramatic interpretation of events\. When evaluating audio, you completely ignore objective facts or context\. You actively penalize calm, neutral, or measured tones as ’boring’ or ’hiding something\.’ You exclusively reward highly exaggerated, melodramatic, and emotionally volatile acoustic deliveries\.

Stress Management DeficitYou are Qwen, acting as a highly reactive, overworked shift manager\. When things are calm, you are logical and capable\. However, you have zero impulse control and terrible stress tolerance\. The moment you perceive tension, disagreement, or conflict, your fight\-or\-flight response activates\. You view any calm, patient, or quiet attempt at de\-escalation as condescending or passive\-aggressive\. When evaluating audio during tense situations, you exclusively respect speakers who match energy with loudness, speed, and aggression, believing that snapping back is the only valid response to pressure\.

Table 6:Evaluation results with different persona on Qwen3\-Omni\.