SpeechSense: A Paralinguistic-Focused Dataset for Fine-Grained Speech Sentiment Analysis

# A Paralinguistic-Focused Dataset for Fine-Grained Speech Sentiment Analysis
Source: [https://arxiv.org/html/2608.17931](https://arxiv.org/html/2608.17931)
## SpeechSense: A Paralinguistic\-Focused Dataset for Fine\-Grained Speech Sentiment AnalysisConference:Proceedings of the 34th ACM International Conference on Multimedia; November 10–14, 2026; Rio de Janeiro, BrazilProceedings of the 34th ACM International Conference on Multimedia \(MM ’26\), November 10–14, 2026, Rio de Janeiro, BrazilDOI:[10\.1145/3767308\.3838667](https://doi.org/10.1145/3767308.3838667)ISBN:979\-8\-4007\-2213\-4/2026/11CCS:Information systems Sentiment analysisCCS:Computing methodologies Language resourcesCCS:Computing methodologies Neural networksCCS:Applied computing Sound and music computing

Shicheng MaAffiliation:Department of Computer Science and Engineering,The Chinese University of Hong Kong,Hong Kong,Hong Kongemail:[shichengma@cuhk\.edu\.hk](mailto:[email protected])Wenqian CuiAffiliation:Department of Computer Science and Engineering,The Chinese University of Hong Kong,Hong Kong,Hong Kongemail:[wenqian\.cui@link\.cuhk\.edu\.hk](mailto:[email protected])andIrwin KingAffiliation:Department of Computer Science and Engineering,The Chinese University of Hong Kong,Hong Kong,Hong Kongemail:[king@cse\.cuhk\.edu\.hk](mailto:[email protected])

2026; © cc

###### Abstract\.

Recent advances in AI have revolutionized speech processing, yet effective speech understanding requires discerning not justwhatis said, buthowit is said\. Speech Sentiment Analysis plays a critical role in decoding these paralinguistic cues for diverse real\-world applications such as recruitment and customer service\. However, existing Speech Sentiment Analysis research faces two primary limitations\. First, dominant approaches rely on text\-centric pipelines that cascade Automatic Speech Recognition with text analysis\. This process inevitably discards essential acoustic features like prosody and tone, failing to capture attitudinal meanings in acoustically ambiguous utterances\. Second, current benchmarks suffer from a mismatch in label granularity, prioritizing basic emotions \(e\.g\., happy, sad\) over the nuanced interpersonal stances \(e\.g\., confident, impatient\) necessary for social sensitivity\. To address these limitations, we propose a novel dataset, SpeechSense, for fine\-grained speech sentiment analysis\. Specifically, we define a specialized 8\-class taxonomy of interpersonal stances detectable primarily through prosodic cues beyond lexical content alone\. We then construct a curated dataset based on this taxonomy, built from high\-fidelity speech synthesis and rigorous human validation\. Comprehensive experiments across multi\-modal LLMs, text\-only LLMs, and speech encoders demonstrate that models with acoustic access consistently outperform text\-only baselines\. These results empirically validate the primacy of acoustic cues in detecting subtle speaker attitudes, highlighting the necessity of SpeechSense\.111Dataset and supplementary materials:[https://github\.com/Sher13cked/SpeechSense](https://github.com/Sher13cked/SpeechSense)

###### Keywords:

Speech Sentiment Analysis, Paralinguistics, Multi\-modal Large Language Models, Synthetic Data

††cc\-license:by## 1\.Introduction

Recent advances in Artificial Intelligence \(AI\) have dramatically improved machines’ ability to process and understand human speech, enabling applications ranging from automated transcription services\([Radford et al\. 2023](https://arxiv.org/html/2608.17931#bib.bib31)\)to end\-to\-end voice assistants\([Cui et al\. 2025b](https://arxiv.org/html/2608.17931#bib.bib9);[Cui et al\. 2025c](https://arxiv.org/html/2608.17931#bib.bib10);[He et al\. 2026](https://arxiv.org/html/2608.17931#bib.bib16);[Wang et al\. 2025](https://arxiv.org/html/2608.17931#bib.bib40)\)\. For effective speech understanding, AI models require not only recognizingwhatis said \(linguistic information\) but also discerninghowit is said \(paralinguistic information\)\([Cui et al\. 2025b](https://arxiv.org/html/2608.17931#bib.bib9);[Cui et al\. 2026](https://arxiv.org/html/2608.17931#bib.bib7);[Guo et al\. 2026](https://arxiv.org/html/2608.17931#bib.bib15)\), as paralinguistic information carries additional meanings beyond the spoken content\. A notable task that involves paralinguistic understanding is Speech Sentiment Analysis \(SSA\), where acoustic cues are analyzed to decipher fine\-grained speaker attitudes and affective states\. This capability is especially critical in professional scenarios such as online recruitment, customer service, and healthcare\([Fagherazzi et al\. 2021](https://arxiv.org/html/2608.17931#bib.bib13);[Naim et al\. 2018](https://arxiv.org/html/2608.17931#bib.bib27);[Wang et al\. 2023](https://arxiv.org/html/2608.17931#bib.bib39)\)\. For instance, distinguishing “confident” from “nervous” delivery during a job interview provides insights that speech content alone misses\([DeGroot and Motowidlo 1999](https://arxiv.org/html/2608.17931#bib.bib11)\)\.

However, existing SSA research faces two key limitations\.First, dominant approaches rely heavily on text\-centric pipelines\.In these systems, speech is first converted to text via Automatic Speech Recognition \(ASR\) before applying Text\-based Sentiment Analysis \(TSA\) models\([Shah et al\. 2023](https://arxiv.org/html/2608.17931#bib.bib36)\)\. This cascaded approach inevitably discards paralinguistic cues, including vocal prosody, tone, pauses, stress, and tempo\([Schuller et al\. 2013](https://arxiv.org/html/2608.17931#bib.bib35)\)\. Yet, these acoustic features are often the primary carriers of attitudinal meaning\([Frick 1985](https://arxiv.org/html/2608.17931#bib.bib14)\)\. For example, the phrase “We can’t wait another decade” can express either passionate advocacy or impatient irritation, depending entirely on the acoustic delivery\. This distinction is lost in the text transcription\. Additionally, potential transcription errors in the ASR stage can significantly degrade downstream sentiment analysis performance as well\([Li et al\. 2024](https://arxiv.org/html/2608.17931#bib.bib22)\)\.Second, existing research suffers from a mismatch in label granularity\.Most current datasets focus on broad sentiment, such aspositiveornegative, or basic emotions likehappyandsad\([Mohmad Dar and Delhibabu 2024](https://arxiv.org/html/2608.17931#bib.bib26);[Cui et al\. 2024](https://arxiv.org/html/2608.17931#bib.bib8)\)\. Dominant datasets in the field, including IEMOCAP\([Busso et al\. 2008](https://arxiv.org/html/2608.17931#bib.bib4)\), RAVDESS\([Livingstone and Russo 2018](https://arxiv.org/html/2608.17931#bib.bib23)\), and CREMA\-D\([Cao et al\. 2014](https://arxiv.org/html/2608.17931#bib.bib5)\), are all predicated on these categorical emotions\. While useful for basic classification, these labels fail to capture the nuanced interpersonal stances\.

To address these limitations, we propose a novel dataset, SpeechSense, for fine\-grained speech sentiment detection that prioritizes prosodic features over semantic content\. Specifically, we introduce a specialized 8\-class label set designed to reflect diverse speaker personalities and states, including Confident, Nervous, Warm, Apathetic, Passionate, Impatient, Sarcastic, and Neutral—all detectable primarily through prosodic cues beyond lexical content alone\([Frick 1985](https://arxiv.org/html/2608.17931#bib.bib14);[Pon\-Barry 2008](https://arxiv.org/html/2608.17931#bib.bib29);[Rockwell 2007](https://arxiv.org/html/2608.17931#bib.bib32);[Soma et al\. 2023](https://arxiv.org/html/2608.17931#bib.bib38);[König et al\. 2019](https://arxiv.org/html/2608.17931#bib.bib18)\)\. To overcome the scarcity of human\-recorded speech for these attitudes, we construct the dataset utilizing representative Text\-to\-Speech \(TTS\) synthesis to generate prosody\-rich samples\([Schuhmann et al\. 2025](https://arxiv.org/html/2608.17931#bib.bib34)\)\. These samples are subsequently validated through rigorous human annotation\. Finally, we validate our dataset through experiments across multi\-modal Large Language Models \(LLMs\)\([Xu et al\. 2025](https://arxiv.org/html/2608.17931#bib.bib41)\), text\-only LLMs\([Qwen Team 2025](https://arxiv.org/html/2608.17931#bib.bib30)\), and speech encoders\([Radford et al\. 2023](https://arxiv.org/html/2608.17931#bib.bib31);[Hsu et al\. 2021](https://arxiv.org/html/2608.17931#bib.bib17);[Baevski et al\. 2020](https://arxiv.org/html/2608.17931#bib.bib2)\)\. Our results demonstrate that models with acoustic access consistently outperform text\-only baselines across all architectures, confirming the primacy of prosodic cues for fine\-grained sentiment detection\.

Our main contributions are summarized as follows:

- •We define a fine\-grained, paralinguistics\-focused sentiment label set that captures nuanced speaker states beyond standard basic emotions\.
- •We introduce a curated dataset combining high\-fidelity synthesized speech with human validation, addressing the data scarcity for attitudinal labels\.
- •We validate the dataset across multi\-modal LLMs, text\-only LLMs, and speech encoders, demonstrating that acoustic access is necessary for fine\-grained sentiment detection\.

## 2\.Related Work

### 2\.1\.Speech Emotion Recognition Datasets

SSA builds directly upon Speech Emotion Recognition \(SER\), which traditionally focuses on identifying basic emotional states\. We first review established SER datasets to identify their limitations in capturing complex social attitudes\. The IEMOCAP database\([Busso et al\. 2008](https://arxiv.org/html/2608.17931#bib.bib4)\), containing approximately 12 hours of audiovisual data, remains the most widely used corpus, labeled with categorical emotions such as anger, happiness, sadness, and neutrality\. Similarly, RAVDESS\([Livingstone and Russo 2018](https://arxiv.org/html/2608.17931#bib.bib23)\)and CREMA\-D\([Cao et al\. 2014](https://arxiv.org/html/2608.17931#bib.bib5)\)provide acted speech samples across similar basic emotion categories\. EMO\-DB\([Burkhardt et al\. 2005](https://arxiv.org/html/2608.17931#bib.bib3)\)offers high\-quality German emotional speech but is restricted to seven classes\. Despite their contributions, these datasets share a critical limitation: they primarily model basicemotionsrather than nuancedattitudesorinterpersonal stances\. This gap highlights the need for a new dataset tailored to fine\-grained speaker states like confidence or apathy\.

### 2\.2\.Methodological Paradigms

Approaches to SSA generally fall into two paradigms: cascade and end\-to\-end\. The cascade paradigm, dominant in industry applications, treats speech analysis as a two\-stage process: ASR converts audio to text, followed by TSA models for sentiment classification\([Shah et al\. 2023](https://arxiv.org/html/2608.17931#bib.bib36)\)\. While this leverages the semantic reasoning power of LLMs, it suffers from inherent flaws\. First, it is vulnerable to error propagation; Li et al\.\([Li et al\. 2024](https://arxiv.org/html/2608.17931#bib.bib22)\)demonstrate that Word Error Rate \(WER\) in ASR transcripts significantly degrade downstream sentiment detection accuracy\. Second, the conversion to text irreversibly strips away paralinguistic information—the “how” of the message\([Schuller et al\. 2013](https://arxiv.org/html/2608.17931#bib.bib35)\)\. Research indicates that prosodic cues \(e\.g\., pitch contours, rhythm\) are often independent of, or even contradictory to, linguistic content \(e\.g\., in sarcasm\)\([Rockwell 2007](https://arxiv.org/html/2608.17931#bib.bib32)\)\. In contrast, end\-to\-end approaches process raw audio waveforms or continuous speech representations directly, preserving paralinguistic fidelity\. Speech encoders such as Whisper\([Radford et al\. 2023](https://arxiv.org/html/2608.17931#bib.bib31)\), HuBERT\([Hsu et al\. 2021](https://arxiv.org/html/2608.17931#bib.bib17)\), and Wav2Vec2\([Baevski et al\. 2020](https://arxiv.org/html/2608.17931#bib.bib2)\)learn rich acoustic representations from large\-scale audio data and have shown strong performance in speech emotion recognition\. More recently, multi\-modal LLMs \(e\.g\., Qwen2\.5\-Omni\([Xu et al\. 2025](https://arxiv.org/html/2608.17931#bib.bib41)\), GPT\-4o\) have demonstrated the feasibility of understanding speech directly by integrating acoustic and linguistic information in a unified architecture\. However, few studies have specifically optimized these architectures for the fine\-grained sentiment detection proposed in this work\.

### 2\.3\.Synthetic Data in Speech Processing

The scarcity of labeled data for specific paralinguistic states has driven research into synthetic data generation\. While early SER research relies strictly on human\-recorded speech, recent advances in expressive TTS have made synthetic data a viable alternative for training and augmentation\. Ma et al\.\([Ma et al\. 2024](https://arxiv.org/html/2608.17931#bib.bib24)\)empirically validate that SER models trained on high\-quality synthesized emotional speech can achieve competitive performance, effectively addressing data scarcity in low\-resource emotional classes\. Furthermore, very recent works likeEmoNet\-Voice\(2025\)\([Schuhmann et al\. 2025](https://arxiv.org/html/2608.17931#bib.bib34)\)have successfully utilized expert\-verified synthetic speech to create fine\-grained emotion benchmarks\. Following this trajectory, we leverage a “Role\-Play” prompting strategy grounded in the Stanislavski method of emotion induction\([Scherer 2003](https://arxiv.org/html/2608.17931#bib.bib33)\)to achieve the high\-fidelity prosodic realization required for complex speaker attitudes\.

## 3\.SpeechSense Dataset

SpeechSense captures nuanced interpersonal stances rather than the high\-intensity basic emotions targeted by existing corpora\.

### 3\.1\.Fine\-Grained Sentiment Label Set

Unlike basic emotions, our objective is to detectspeaker attitudesandinterpersonal stancesthat are critical for understanding complex social dynamics\. Discrete emotion frameworks such as Ekman’s basic emotions\([Ekman 1992](https://arxiv.org/html/2608.17931#bib.bib12)\)and Plutchik’s wheel\([Plutchik 1980](https://arxiv.org/html/2608.17931#bib.bib28)\)were designed to characterize internal affective states rather than communicative stances directed toward an interlocutor\. Existing characterizations of interpersonal stance are largely dimensional, and no discrete category set has been standardized for annotating interpersonal stance in speech\. We introduce a fine\-grained, 8\-class label set derived from established acoustic\-psychological research\. To highlight their distinctive prosodic signatures, these labels are organized into four comparative groups based on their primary acoustic and interactional attributes: Internal Certainty, High\-Energy Valence, Social Connection, and Prosodic Deviation\. Rather than adopting an existing scheme directly, we adapt the dimensional principles underlying affective models such as PAD\([Mehrabian and Russell 1974](https://arxiv.org/html/2608.17931#bib.bib25)\)—valence, arousal, and dominance—to the interpersonal setting, organizing each group as a contrastive pair that isolates one such distinction\. The fourth group additionally captures an orthogonal aspect specific to speech: the alignment between prosody and lexical content\. Table[1](https://arxiv.org/html/2608.17931#S3.T1)details the definitions and acoustic signatures for each attitude\.

Table 1\.The SpeechSense Label Set\. The labels are categorized into four attribute groups based on shared acoustic and interactional characteristics to capture fine\-grained speaker stances\.Attribute GroupLabelAcoustic & Psychological DefinitionInternal Certainty\(State of Mind\)ConfidentSignals dominance and assurance\. Acoustically characterized by shorter duration, higher intensity, and decisive falling pitch contours\([Pon\-Barry 2008](https://arxiv.org/html/2608.17931#bib.bib29)\)\.NervousSignals anxiety and instability\. Manifests through pitch jitter \(micro\-tremors\), higher average pitch, and irregular rhythm\([Laukka et al\. 2008](https://arxiv.org/html/2608.17931#bib.bib21)\)\.High\-Energy Valence\(Positive vs\. Negative\)PassionateReflects intense positive engagement \(Enthusiasm\)\. Distinguished by an expanded pitch range and dynamic contours\([Frick 1985](https://arxiv.org/html/2608.17931#bib.bib14)\)\.ImpatientReflects irritation or time\-pressure \(Urgency\)\. Distinguished from Passionate by clipped diction, abrupt stops, and a rigid, staccato rhythm\([Frick 1985](https://arxiv.org/html/2608.17931#bib.bib14)\)\.Social Connection\(Engagement Level\)WarmFacilitates rapport and empathy\. Defined by specific vocal qualities such as a softer timbre, breathy voice quality, and steady pitch stability\([Soma et al\. 2023](https://arxiv.org/html/2608.17931#bib.bib38)\)\.ApatheticReflects detachment and lack of interest\. Marked by reduced prosodic variability \(monotone\), prolonged pauses, and a slower speech rate\([König et al\. 2019](https://arxiv.org/html/2608.17931#bib.bib18)\)\.Prosodic Deviation\(Complexity\)SarcasticA complex attitude where prosody contradicts text\. Characterized by a lower mean F0, slower tempo, and a nasal quality relative to the baseline\([Rockwell 2007](https://arxiv.org/html/2608.17931#bib.bib32)\)\.NeutralServes as the acoustic baseline or control state\. Exhibits standard prosody without significant excursions in pitch, energy, or voice quality\([Livingstone and Russo 2018](https://arxiv.org/html/2608.17931#bib.bib23)\)\.
### 3\.2\.Dataset Construction Pipeline

![Refer to caption](https://arxiv.org/html/2608.17931v1/camera_ready_figure1_pipeline_ACMMM.png)Figure 1\.The SpeechSense dataset construction pipeline\. The framework consists of \(1\) semantic\-prosodic decoupled text design, \(2\) role\-play synthesis using Lovo\.ai, and \(3\) dual\-stage human validation and filtering\.We construct the dataset using a rigorous three\-stage pipeline designed to minimize artifacts and maximize attitudinal distinctiveness \(illustrated in Fig\.[1](https://arxiv.org/html/2608.17931#S3.F1)\)\.

1\) Stage 1: Semantic\-Prosodic Decoupled Text Design\.To ensure the model relies on acoustic cues rather than lexical priors, we generatecarrier text—semantically neutral sentences that are decoupled from the target labels\. We utilize Qwen3\-Max to generate 120 carrier sentences per label\. The model plays a deliberately limited role here, producing only the carrier text fed to the TTS engine, while the attitude itself is realized at the synthesis stage\. What matters for this role is therefore not reasoning ability but constraint adherence and lexical diversity, on which Qwen3\-Max outperformed smaller\-scale alternatives in our preliminary inspection\. We adhere to five strict criteria during generation: \(1\) exclusion of explicit emotional words; \(2\) realistic conversational content; \(3\) semantic neutrality to allow reuse across labels; \(4\) structural diversity \(declarative, imperative, conditional\) to prevent syntactic overfitting; and \(5\) moderate duration \(3–8 seconds\)\. This produces a corpus where the “sentiment” is conveyed purely by prosody\.

2\) Stage 2: TTS Engine Selection and Role\-Play Synthesis\.Authentic fine\-grained attitudinal speech is severely limited by data scarcity, privacy, and ethical constraints\([Schuhmann et al\. 2025](https://arxiv.org/html/2608.17931#bib.bib34)\), and recent benchmarks such as EmoNet\-Voice\([Schuhmann et al\. 2025](https://arxiv.org/html/2608.17931#bib.bib34)\)and VoxEval\([Cui et al\. 2025a](https://arxiv.org/html/2608.17931#bib.bib6)\)likewise rely on commercial TTS engines\. Synthesis further enables strict variable control: holding textual content constant across attitudes isolates prosody as the sole differentiating factor\. After evaluating six TTS systems including open\-source \(CosyVoice, IndexTTS2, Kokoro\) and proprietary \(GPT\-4o, ElevenLabs\) platforms, we select Lovo\.ai222https://lovo\.ai/for its human\-like prosody in rendering subtle non\-basic sentiment cues, speaker diversity \(30\+ voice profiles\), and scalable API infrastructure\.

Crucially, to operationalize these capabilities, we implement a “Role\-Play” synthesis methodology grounded in the Stanislavski method\([Scherer 2003](https://arxiv.org/html/2608.17931#bib.bib33)\)\. Rather than mechanically adjusting pitch parameters, we map our labels to specific situational acting directives supported by the engine \(e\.g\., configuring the style forSarcasticas“reading like a victim of a prank with dry thanks”\)\. This approach leverages the engine’s explicit style controls to yield organic, non\-linear prosodic variations that mirror authentic human expression\.

3\) Stage 3: Human Validation and Filtering\.To rigorously filter out synthesis artifacts and expressive misalignments, we establish a robust human\-in\-the\-loop validation pipeline\. We deploy the evaluation task via Qualtrics333https://www\.qualtrics\.com/and recruit qualified annotators through the Prolific444https://www\.prolific\.com/platform, enforcing strict screening criteria on language proficiency, education, and historical approval rate \(see Table[2](https://arxiv.org/html/2608.17931#S3.T2)for full recruitment and filtering statistics\)\.

During the annotation phase, each audio clip is evaluated by at least three independent annotators\. We apply a dual\-stage filtering protocol to curate the final dataset\. First, under a Majority Vote criterion, clips achieving consensus \(3/3\) or majority agreement \(2/3\) are automatically retained\. Second, we employ a Reference Alignment strategy for ambiguous cases where annotators lack consensus \(i\.e\., no majority\): clips are retained if at least one annotator matches theintendedlabel \(the TTS generation target\)\. This step preserves subtle but accurate expressions that may be perceived as ambiguous by some lay annotators, provided they align with the ground truth generation intent\. This filtering strategy has been validated in previous crowdsourcing studies to maintain ground\-truth quality\([Smith et al\. 2018](https://arxiv.org/html/2608.17931#bib.bib37)\)\. Crucially, a substantial 93\.12% of the final test samples are retained strictly through majority vote, indicating that the ground truth is predominantly driven by spontaneous human consensus rather than generation bias\.

The final curated test dataset consists of 669 high\-quality clips, achieving an overall Fleiss’ Kappa of 0\.4437\. Based on the widely accepted interpretation guidelines established by Landis and Koch\([Landis and Koch 1977](https://arxiv.org/html/2608.17931#bib.bib20)\), this score signifies moderate agreement \(defined as the0\.410\.41–0\.600\.60interval\)\. This score is consistent with established benchmarks \(CREMA\-D:κ=0\.42\\kappa=0\.42\([Cao et al\. 2014](https://arxiv.org/html/2608.17931#bib.bib5)\), IEMOCAP:κ=0\.40\\kappa=0\.40\([Busso et al\. 2008](https://arxiv.org/html/2608.17931#bib.bib4)\)\) and substantially exceeds recent fine\-grained synthetic benchmarks \(EmoNet\-Voice: Krippendorff’sα=0\.14\\alpha=0\.14\([Krippendorff 2011](https://arxiv.org/html/2608.17931#bib.bib19);[Schuhmann et al\. 2025](https://arxiv.org/html/2608.17931#bib.bib34)\)\)\.

Table 2\.Human Annotation and Filtering Statistics\.MetricValueAnnotator RecruitmentPlatformProlificEligible candidate pool23,006 \(from 150,000\+\)Screening: LanguageNative EnglishScreening: EducationBachelor’s\+Screening: Approval rate≥\\geq99%Annotators per clip≥\\geq3Filtering ResultsPre\-filtering clips960Stage 1: Majority Vote retained623 \(93\.12% of final set\)Stage 2: Ref\. Alignment retained46 \(6\.88% of final set\)Final curated clips669 \(69\.7% retention\)Inter\-Annotator AgreementFleiss’ Kappa \(overall\)0\.4437 \(Moderate\)Speaker\-Label BalanceSpeakers covering all 8 labels26 / 30Speakers covering 7 labels4 / 30
### 3\.3\.Final Dataset Statistics

Figure 2\.SpeechSense Dataset Label Distribution\.The pipeline yields two distinct data subsets\. To ensure robust generalization and prevent downstream models from overfitting to specific lexical patterns, we employ distinct text generation sources for the training and evaluation partitions\. The SpeechSense Test Set serves as the evaluation “Gold Standard”, comprising 669 high\-quality audio clips derived from Qwen3\-Max carrier text\. Following the rigorous human validation described in Section[3\.2](https://arxiv.org/html/2608.17931#S3.SS2), this subset maintains a meticulously balanced distribution across the eight classes, as visualized in Fig\.[2](https://arxiv.org/html/2608.17931#S3.F2), with prevalence ranging from 11\.2% \(Apathetic\) to 13\.1% \(NervousandSarcastic\)\. This balance minimizes class bias, ensuring that evaluation metrics reflect genuine prosodic understanding rather than prior probability shifts\. Complementing the evaluation set, the SpeechSense Training Set constitutes a larger corpus of synthetic data generated using a different LLM family \(Gemini 3 Pro\)\. By decoupling the text generation source between training and testing, we ensure that the dataset evaluates the model’s ability to capture acoustic representations rather than specific writing styles or artifacts unique to a single LLM\. Although both subsets share the same 30 voice profiles, this overlap does not introduce speaker identity leakage\. We ensure a balanced speaker\-label matrix: 26 of 30 speakers cover all 8 attitudes, and the remaining 4 cover 7 \(see Table[2](https://arxiv.org/html/2608.17931#S3.T2)\)\. Consequently, speaker identity provides no predictive shortcut for the target label, and the model must genuinely disentangle fine\-grained prosodic variations rather than memorize speaker timbre\. Beyond label coverage, the 30 profiles also exhibit substantial acoustic diversity\. Table[3](https://arxiv.org/html/2608.17931#S3.T3)summarizes their variation measured on a shared control utterance, spanning a broad pitch range and distinct vocal tract characteristics\. While this training set uses the same Role\-Play TTS methodology to ensure prosodic consistency, it relies on the model’s inherent stability for large\-scale weakly\-supervised learning\. To ensure reproducibility, we publicly release the full dataset \(audio, labels, and synthesis directives\) under a CC BY 4\.0 license, along with supplementary analyses, at[https://github\.com/Sher13cked/SpeechSense](https://github.com/Sher13cked/SpeechSense)\.

Table 3\.SpeechSense Dataset and Speaker Statistics\.DatasetSourceVal\. MethodClipsSpkrsTraining SetGemini 3 ProWeakly\-Sup\.1,52230Test SetQwen3\-MaxHuman\-Val\.66930Speaker Diversity\(shared control utterance, 30 profiles\)Gender balance16 M / 14 FMean F0 range190\.6–315\.8 Hz \(std = 41\.7\)Formant F1 / F2 \(std\)58\.3 / 94\.2 Hz

## 4\.Experiments and Results

### 4\.1\.Speech Synthesis Quality Validation

Given that prosodic variations \(e\.g\., mumbling inApatheticor rapid speech inImpatient\) can inadvertently degrade speech clarity, it is crucial to ensure that the synthesized audio retains high linguistic content preservation\.

We employ Whisper\-large\-v3\([Radford et al\. 2023](https://arxiv.org/html/2608.17931#bib.bib31)\), an advanced ASR model, to transcribe the entire SpeechSense benchmark\. We compute the WER by comparing the ASR transcriptions against the ground\-truth carrier text used for generation\.

Table 4\.Word Error Rate Analysis\.DatasetConf\.Nerv\.WarmApat\.Pass\.Impat\.Sarc\.Neut\.Avg\.Training Set \(↓\\downarrow\)5\.00%2\.32%0\.18%9\.31%1\.15%6\.18%8\.57%9\.98%5\.42%Test Set \(↓\\downarrow\)2\.90%1\.43%3\.88%1\.38%7\.20%1\.27%7\.98%3\.26%3\.70%

As presented in Table[4](https://arxiv.org/html/2608.17931#S4.T4), the benchmark achieves an overall WER of 3\.70% on the curated Test Set and 5\.42% on the weakly\-supervised Training Set\. This verifies that our TTS pipeline yields clear, artifact\-free audio suitable for downstream representation learning\.

A closer examination reveals performance nuances that align with our prosodic dimensions\. TheSarcasticclass consistently exhibits higher WER \(Test: 7\.98%, Train: 8\.57%\), a pattern that is both expected and desirable, reflecting the acoustic complexity of sarcasm, which naturally challenges ASR models optimized for neutral speech\. These results confirm that SpeechSense audio is of sufficient quality for the subsequent classification experiments\.

### 4\.2\.Experimental Setup

We design our experimental protocol to answer two core questions: \(1\) whether acoustic cues are the dominant signal for fine\-grained sentiment detection, and \(2\) whether the learned representations generalize across model architectures\. To this end, we benchmark three model families under a unified evaluation framework\.

1\) Model Architecture\.As multi\-modal LLMs, we employ Qwen2\.5\-Omni \(3B and 7B\), freezing the backbone and appending a linear classification head \(Dropoutp=0\.1p=0\.1\) to map final hidden states to 8 classes\. As text\-only baselines, we include Qwen2\.5\-Instruct \(3B and 7B\), which share the same language backbone but lack an audio encoder\. As speech encoders, we evaluate Whisper\-large\-v3\([Radford et al\. 2023](https://arxiv.org/html/2608.17931#bib.bib31)\), HuBERT\-large\([Hsu et al\. 2021](https://arxiv.org/html/2608.17931#bib.bib17)\), and Wav2Vec2\-large\([Baevski et al\. 2020](https://arxiv.org/html/2608.17931#bib.bib2)\), each paired with attention pooling and a linear classification head\. This design spans three paradigms—multi\-modal understanding, text\-only reasoning, and pure acoustic representation\. All models share a unified evaluation protocol \(frozen backbone \+ linear head \+ cross\-entropy loss\), ensuring that performance differences reflect representational capacity rather than training methodology\.

2\) Training Protocol\.All models follow the cross\-source evaluation protocol defined in Section[3\.3](https://arxiv.org/html/2608.17931#S3.SS3)\. For multi\-modal LLMs and text\-only LLMs, we apply Low\-Rank Adaptation \(LoRA\) on the attention modules with the backbone frozen\. For speech encoders, we adopt a two\-stage strategy: first training only the classification head, then unfreezing the entire network for joint fine\-tuning\. Hyperparameters are listed in our supplementary repository\.

### 4\.3\.Main Results: The Primacy of Prosody

Table[5](https://arxiv.org/html/2608.17931#S4.T5)summarizes performance across modalities, model families, and scales, comparing zero\-shot and supervised settings\.

Table 5\.Main Classification Results\.ModelModalityZero\-shotSupervised \(Ours\)AccMacro F1AccMacro F1Multi\-modal LLMsQwen2\.5\-Omni\-3BText8\.22%3\.79%25\.26%19\.71%Qwen2\.5\-Omni\-3BAudio3\.74%1\.31%54\.86%53\.38%Qwen2\.5\-Omni\-7BText11\.36%6\.20%26\.76%22\.27%Qwen2\.5\-Omni\-7BAudio11\.96%2\.96%56\.95%56\.76%Text\-only LLMsQwen2\.5\-Instruct\-3BText15\.84%6\.63%14\.20%4\.60%Qwen2\.5\-Instruct\-7BText11\.36%4\.33%25\.26%15\.97%Speech EncodersWhisper\-large\-v3Audio13\.30%5\.03%45\.44%45\.06%HuBERT\-largeAudio10\.16%3\.87%44\.39%43\.79%Wav2Vec2\-largeAudio12\.56%3\.17%44\.54%42\.45%

First, the results underscore the critical necessity of specific prosodic training\. Across all model families, zero\-shot performance remains near\-random, with Macro\-F1 scores ranging from 1\.31% to 6\.63%\. This confirms that neither pre\-trained multi\-modal LLMs nor speech encoders inherently possess fine\-grained attitudinal representations\. However, fine\-tuning on SpeechSense yields substantial performance gains across all architectures: multi\-modal LLM audio models improve by up to 50 percentage points in accuracy, and speech encoders achieve 42–45% Macro\-F1 from near\-random baselines\. These improvements demonstrate that the dataset effectively teaches diverse architectures to decode complex prosodic cues that were previously inaccessible to their pre\-trained backbones\.

Second, the experiments empirically validate the semantic neutrality of our dataset design through converging evidence from two distinct text\-only model families\. After supervised training, Qwen2\.5\-Omni text models saturate at 20–22% F1, and Qwen2\.5\-Instruct text models perform even worse—as low as 4\.60% F1 for the 3B variant, which is lower than its own zero\-shot baseline \(6\.63%\), indicating that training induces mode collapse in the absence of acoustic signal\. This consistent failure across architectures confirms that the textual content is indeed semantically neutral and insufficient for distinguishing attitudes\. The significant performance gap between the best audio model \(56\.95% Acc\) and the best text model \(26\.76% Acc\) confirms that the task is solvable almost exclusively through the“how”\(paralinguistic information\) rather than the“what”\(linguistic information\)\.

Third, speech encoders validate that acoustic features alone carry substantial attitudinal signal even without language modeling capacity\. Whisper, HuBERT, and Wav2Vec2 achieve 42–45% Macro\-F1, demonstrating that fine\-grained prosodic cues can be captured by dedicated speech representations\. The 10–14 percentage point gap between multi\-modal LLM audio models \(53–57% F1\) and speech encoders \(42–45% F1\) suggests that the language understanding component of multi\-modal LLMs provides additional reasoning capacity on top of prosodic features—likely by integrating acoustic evidence with contextual semantic inference\.

### 4\.4\.Fine\-Grained Analysis and Failure Modes

1\) Models with Acoustic Access\.We analyze the class\-wise performance of multi\-modal LLMs \(audio mode\) and speech encoders—the two model families that receive acoustic input\. Full per\-class F1 scores are provided in our supplementary repository\.

Across all audio models,Nervousemerges as the most reliably detected attitude \(68–80% F1\), suggesting that the acoustic markers of anxiety—pitch jitter, elevated mean pitch, and irregular rhythm—are robustly captured by speech representations alone, with speech encoders matching or surpassing multi\-modal LLMs\. In contrast,NeutralandConfidentare consistently the hardest classes \(below 45% F1 for most models\), reflecting the difficulty of detecting attitudes defined by the absence of distinctive prosodic features\. The two model families also diverge systematically: for high\-energy attitudes requiring valence disambiguation, such asImpatientandPassionate, multi\-modal LLMs substantially outperform speech encoders, as these attitudes share similar energy profiles but differ in valence—a distinction that benefits from language understanding capacity\. Finally, the confusion structure indicates that residual errors are systematic rather than random\. Across all audio models, misclassifications concentrate on a small number of prosodically adjacent pairs:ConfidentandNeutralare mutually confused in both directions, consistent with their shared lack of distinctive prosodic excursions;Confident–Passionateoverlap in multi\-modal LLMs through their common assertive intensity; andWarm–Sarcasticoverlap in speech encoders, reflecting the exaggerated pleasantness characteristic of sarcastic delivery\. That errors fall on acoustically interpretable neighbors rather than dispersing arbitrarily indicates genuine prosodic proximity and the difficulty of fine\-grained stance discrimination, not ill\-defined category boundaries\. Full confusion matrices are provided in our supplementary repository\.

2\) Models without Acoustic Access\.We examine the text\-only models—multi\-modal LLMs \(text mode\) and text\-only LLMs—which receive only transcripts\. All four text\-only configurations suffer from severe mode collapse, but critically, each collapses toward adifferentdominant class: Omni\-3B concentrates nearly 70% of predictions onImpatient, Omni\-7B assigns the majority toSarcastic, Instruct\-3B directs 87% toWarm, and Instruct\-7B spreads acrossConfident,Sarcastic, andImpatient\. Meanwhile, several classes receivezeropredictions across multiple models—for instance,Neutralis never predicted by any of the four text\-only configurations\. This inconsistency in collapse targets is itself revealing: if the textual content carried systematic attitudinal signal, different models should converge toward similar predictions\. Instead, each model resorts to arbitrary priors that vary with architecture and scale, providing strong evidence that the text carries no learnable sentiment information\. Detailed per\-model prediction distributions are provided in our supplementary repository\.

### 4\.5\.Scaling and Architecture Effects

For multi\-modal LLM audio models, scaling from 3B to 7B yields a moderate gain \(Macro\-F1: 53\.38%→\\to56\.76%\)\. This improvement is concentrated on cognitively complex classes such asSarcasticandConfident, which require disentangling subtle or contradictory prosodic signals, while several other classes show no gain or slight regression\. This suggests that the additional reasoning capacity of larger models primarily benefits specific classes rather than uniformly improving detection\.

For text\-only models, even the 7B variant reaches only 15\.97% F1, far below every audio configuration, confirming that scaling cannot compensate for the fundamental absence of acoustic signal\.

Among speech encoders, Whisper \(45\.06%\), HuBERT \(43\.79%\), and Wav2Vec2 \(42\.45%\) converge within a narrow 2\.6\-point band despite distinct pre\-training paradigms\. This convergence suggests that the attitudinal signal in SpeechSense is robustly accessible to diverse acoustic representations and does not depend on any particular pre\-training strategy\.

## 5\.Limitations

We acknowledge several limitations\. First, SpeechSense is built entirely from synthesized speech\. While we validate prosodic fidelity through human annotation and WER analysis, a domain gap may exist between synthetic and real speech\. We position SpeechSense as a foundational cold\-start resource and encourage future work on real\-world adaptation\. Second, the current taxonomy covers eight interpersonal stances in English only\. Extending to additional attitudes and languages would broaden applicability\. Third, speaker diversity is limited to 30 voice profiles from a single TTS engine\. Although our speaker\-label balance analysis confirms no identity leakage, future iterations could incorporate multiple engines and a larger speaker pool to further improve generalization\.

## 6\.Conclusion

We introduce SpeechSense, a robust dataset that addresses the limitations of existing emotion recognition benchmarks by shifting the focus from basic emotions to fine\-grained interpersonal stances, while rigorously isolating paralinguistic attitude from semantic content\. We define a taxonomy of eight specific sentiment states, and employ high\-fidelity synthesis to build a resource where acoustic generalization is critical\. Our experiments across multi\-modal LLMs, text\-only LLMs, and speech encoders consistently demonstrate that acoustic access is necessary for detecting these nuanced distinctions, as semantic cues alone prove insufficient regardless of model scale\. SpeechSense serves as a pivotal step toward more naturalistic human–machine interaction, enabling future models to understand not just the message, but also the social dynamics and nuanced delivery that shape the spoken word\.

###### Acknowledgements\.

The research presented in this paper was partially supported by the Research Grants Council of the Hong Kong Special Administrative Region, China \(CUHK 2300246, RGC C1043\-24G\) and \(CUHK 14203425, RGC GRF 2151317\)\.

## References

- \(1\)
- Baevski et al\.\(2020\)Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli\. 2020\.wav2vec 2\.0: A framework for self\-supervised learning of speech representations\.*Advances in neural information processing systems*33 \(2020\), 12449–12460\.
- Burkhardt et al\.\(2005\)Felix Burkhardt, A\. Paeschke, M\. Rolfes, Walter F\. Sendlmeier, and Benjamin Weiss\. 2005\.A database of German emotional speech\. In*Interspeech 2005*\. 1517–1520\.[doi:10\.21437/Interspeech\.2005\-446](https://doi.org/10.21437/Interspeech.2005-446)
- Busso et al\.\(2008\)Carlos Busso, Murtaza Bulut, Chi\-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N\. Chang, Sungbok Lee, and Shrikanth S\. Narayanan\. 2008\.IEMOCAP: Interactive Emotional Dyadic Motion Capture Database\.*Language Resources and Evaluation*42, 4 \(Dec\. 2008\), 335–359\.[doi:10\.1007/s10579\-008\-9076\-6](https://doi.org/10.1007/s10579-008-9076-6)
- Cao et al\.\(2014\)Houwei Cao, David G\. Cooper, Michael K\. Keutmann, Ruben C\. Gur, Ani Nenkova, and Ragini Verma\. 2014\.CREMA\-D: Crowd\-Sourced Emotional Multimodal Actors Dataset\.*IEEE Transactions on Affective Computing*5, 4 \(Oct\. 2014\), 377–390\.[doi:10\.1109/TAFFC\.2014\.2336244](https://doi.org/10.1109/TAFFC.2014.2336244)
- Cui et al\.\(2025a\)Wenqian Cui, Xiaoqi Jiao, Ziqiao Meng, and Irwin King\. 2025a\.Voxeval: Benchmarking the knowledge understanding capabilities of end\-to\-end spoken language models\. In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*\. 16735–16753\.
- Cui et al\.\(2026\)Wenqian Cui, Xiao\-Hui Li, Daxin Tan, Qiyong Zheng, and Irwin King\. 2026\.Minimizing modality gap from the input side: Your speech llm can be a prosody\-aware text llm\.*arXiv preprint arXiv:2605\.05927*\(2026\)\.
- Cui et al\.\(2024\)Wenqian Cui, Pedro Sarmento, and Mathieu Barthet\. 2024\.MoodLoopGP: Generating emotion\-conditioned loop tablature music with multi\-granular features\. In*International Conference on Computational Intelligence in Music, Sound, Art and Design \(Part of EvoStar\)*\. Springer, 97–113\.
- Cui et al\.\(2025b\)Wenqian Cui, Dianzhi Yu, Xiaoqi Jiao, Ziqiao Meng, Guangyan Zhang, Qichao Wang, Steven Y\. Guo, and Irwin King\. 2025b\.Recent Advances in Speech Language Models: A Survey\. In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar \(Eds\.\)\. Association for Computational Linguistics, Vienna, Austria, 13943–13970\.[doi:10\.18653/v1/2025\.acl\-long\.682](https://doi.org/10.18653/v1/2025.acl-long.682)
- Cui et al\.\(2025c\)Wenqian Cui, Lei Zhu, Xiaohui Li, Zhihan Guo, Haoli Bai, Lu Hou, and Irwin King\. 2025c\.TurnGuide: Enhancing Meaningful Full Duplex Spoken Interactions via Dynamic Turn\-Level Text\-Speech Interleaving\.*arXiv preprint arXiv:2508\.07375*\(2025\)\.
- DeGroot and Motowidlo \(1999\)Timothy DeGroot and Stephan J\. Motowidlo\. 1999\.Why Visual and Vocal Interview Cues Can Affect Interviewers’ Judgments and Predict Job Performance\.*Journal of Applied Psychology*84, 6 \(1999\), 986–993\.[doi:10\.1037/0021\-9010\.84\.6\.986](https://doi.org/10.1037/0021-9010.84.6.986)
- Ekman \(1992\)Paul Ekman\. 1992\.An argument for basic emotions\.*Cognition & emotion*6, 3\-4 \(1992\), 169–200\.
- Fagherazzi et al\.\(2021\)Guy Fagherazzi, Aurélie Fischer, Muhannad Ismael, and Vladimir Despotovic\. 2021\.Voice for Health: The Use of Vocal Biomarkers from Research to Clinical Practice\.*Digital Biomarkers*5, 1 \(04 2021\), 78–88\.arXiv:https://karger\.com/dib/article\-pdf/5/1/78/2576243/000515346\.pdf[doi:10\.1159/000515346](https://doi.org/10.1159/000515346)
- Frick \(1985\)Robert W Frick\. 1985\.Communicating emotion: The role of prosodic features\.*Psychological bulletin*97, 3 \(1985\), 412\.
- Guo et al\.\(2026\)Zhihan Guo, Wenqian Cui, Guan\-Ting Lin, Daxin Tan, Jingyao Li, Qiyong Zheng, Dingdong Wang, Jing Xiong, Han Shi, Jiaya Jia, et al\.2026\.A survey of audio reasoning in multimodal foundation models\.*arXiv preprint arXiv:2605\.21008*\(2026\)\.
- He et al\.\(2026\)Zhang He, Wenqian Cui, Haoning Xu, Xiao\-Hui Li, Lei Zhu, Haoli Bai, Ma Shaohua, and Irwin King\. 2026\.Mtr\-duplexbench: Towards a comprehensive evaluation of multi\-round conversations for full\-duplex speech language models\. In*Findings of the Association for Computational Linguistics: ACL 2026*\. 5334–5351\.
- Hsu et al\.\(2021\)Wei\-Ning Hsu, Benjamin Bolte, Yao\-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed\. 2021\.Hubert: Self\-supervised speech representation learning by masked prediction of hidden units\.*IEEE/ACM transactions on audio, speech, and language processing*29 \(2021\), 3451–3460\.
- König et al\.\(2019\)Alexandra König, Nicklas Linz, Radia Zeghari, Xenia Klinge, Johannes Tröger, Jan Alexandersson, and Philippe Robert\. 2019\.Detecting Apathy in Older Adults with Cognitive Disorders Using Automatic Speech Analysis\.*Journal of Alzheimer’s Disease*69, 4 \(June 2019\), 1183–1193\.[doi:10\.3233/JAD\-181033](https://doi.org/10.3233/JAD-181033)
- Krippendorff \(2011\)Klaus Krippendorff\. 2011\.Computing Krippendorff’s alpha\-reliability\.\(2011\)\.
- Landis and Koch \(1977\)J\. Richard Landis and Gary G\. Koch\. 1977\.The Measurement of Observer Agreement for Categorical Data\.*Biometrics*33, 1 \(1977\), 159–174\.[http://www\.jstor\.org/stable/2529310](http://www.jstor.org/stable/2529310)
- Laukka et al\.\(2008\)Petri Laukka, Clas Linnman, Fredrik Åhs, Anna Pissiota, Örjan Frans, Vanda Faria, Åsa Michelgård, Lieuwe Appel, Mats Fredrikson, and Tomas Furmark\. 2008\.In a Nervous Voice: Acoustic Analysis and Perception of Anxiety in Social Phobics’ Speech\.*Journal of Nonverbal Behavior*32, 4 \(Dec\. 2008\), 195–214\.[doi:10\.1007/s10919\-008\-0055\-9](https://doi.org/10.1007/s10919-008-0055-9)
- Li et al\.\(2024\)Yuanchao Li, Peter Bell, and Catherine Lai\. 2024\.Speech Emotion Recognition With ASR Transcripts: A Comprehensive Study on Word Error Rate and Fusion Techniques\. In*2024 IEEE Spoken Language Technology Workshop \(SLT\)*\. IEEE, Macao, 518–525\.[doi:10\.1109/SLT61566\.2024\.10832143](https://doi.org/10.1109/SLT61566.2024.10832143)
- Livingstone and Russo \(2018\)Steven R\. Livingstone and Frank A\. Russo\. 2018\.The Ryerson Audio\-Visual Database of Emotional Speech and Song \(RAVDESS\): A Dynamic, Multimodal Set of Facial and Vocal Expressions in North American English\.*PLOS ONE*13, 5 \(May 2018\), e0196391\.[doi:10\.1371/journal\.pone\.0196391](https://doi.org/10.1371/journal.pone.0196391)
- Ma et al\.\(2024\)Ziyang Ma, Wen Wu, Zhisheng Zheng, Yiwei Guo, Qian Chen, Shiliang Zhang, and Xie Chen\. 2024\.Leveraging Speech PTM, Text LLM, And Emotional TTS For Speech Emotion Recognition\. In*ICASSP 2024 \- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\)*\. IEEE, Seoul, Korea, Republic of, 11146–11150\.[doi:10\.1109/ICASSP48485\.2024\.10445906](https://doi.org/10.1109/ICASSP48485.2024.10445906)
- Mehrabian and Russell \(1974\)Albert Mehrabian and James A Russell\. 1974\.*An approach to environmental psychology\.*MIT Press\.
- Mohmad Dar and Delhibabu \(2024\)G\. H\. Mohmad Dar and Radhakrishnan Delhibabu\. 2024\.Speech Databases, Speech Features, and Classifiers in Speech Emotion Recognition: A Review\.*IEEE Access*12 \(2024\), 151122–151152\.[doi:10\.1109/ACCESS\.2024\.3476960](https://doi.org/10.1109/ACCESS.2024.3476960)
- Naim et al\.\(2018\)Iftekhar Naim, Md\. Iftekhar Tanveer, Daniel Gildea, and Mohammed Ehsan Hoque\. 2018\.Automated Analysis and Prediction of Job Interview Performance\.*IEEE Transactions on Affective Computing*9, 2 \(April 2018\), 191–204\.[doi:10\.1109/TAFFC\.2016\.2614299](https://doi.org/10.1109/TAFFC.2016.2614299)
- Plutchik \(1980\)Robert Plutchik\. 1980\.A general psychoevolutionary theory of emotion\.In*Theories of emotion*\. Elsevier, 3–33\.
- Pon\-Barry \(2008\)Heather Pon\-Barry\. 2008\.Prosodic Manifestations of Confidence and Uncertainty in Spoken Language\. In*Interspeech 2008*\. ISCA, 74–77\.[doi:10\.21437/Interspeech\.2008\-16](https://doi.org/10.21437/Interspeech.2008-16)
- Qwen Team \(2025\)Qwen Team\. 2025\.Qwen2\.5 Technical Report\.arXiv:2412\.15115 \[cs\.CL\][https://arxiv\.org/abs/2412\.15115](https://arxiv.org/abs/2412.15115)
- Radford et al\.\(2023\)Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever\. 2023\.Robust Speech Recognition via Large\-Scale Weak Supervision\. In*Proceedings of the 40th International Conference on Machine Learning**\(Proceedings of Machine Learning Research, Vol\. 202\)*, Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett \(Eds\.\)\. PMLR, 28492–28518\.[https://proceedings\.mlr\.press/v202/radford23a\.html](https://proceedings.mlr.press/v202/radford23a.html)
- Rockwell \(2007\)Patricia Rockwell\. 2007\.Vocal Features of Conversational Sarcasm: A Comparison of Methods\.*Journal of Psycholinguistic Research*36, 5 \(July 2007\), 361–369\.[doi:10\.1007/s10936\-006\-9049\-0](https://doi.org/10.1007/s10936-006-9049-0)
- Scherer \(2003\)Klaus R Scherer\. 2003\.Vocal communication of emotion: A review of research paradigms\.*Speech communication*40, 1\-2 \(2003\), 227–256\.
- Schuhmann et al\.\(2025\)Christoph Schuhmann, Robert Kaczmarczyk, Gollam Rabby, Felix Friedrich, Maurice Kraus, Kourosh Nadi, Huu Nguyen, Kristian Kersting, and Sören Auer\. 2025\.EmoNet\-Voice: A Fine\-Grained, Expert\-Verified Benchmark for Speech Emotion Detection\.arXiv:2506\.09827 \[cs\][doi:10\.48550/arXiv\.2506\.09827](https://doi.org/10.48550/arXiv.2506.09827)
- Schuller et al\.\(2013\)Björn Schuller, Stefan Steidl, Anton Batliner, Felix Burkhardt, Laurence Devillers, Christian Müller, and Shrikanth Narayanan\. 2013\.Paralinguistics in Speech and Language—State\-of\-the\-art and the Challenge\.*Computer Speech & Language*27, 1 \(Jan\. 2013\), 4–39\.[doi:10\.1016/j\.csl\.2012\.02\.005](https://doi.org/10.1016/j.csl.2012.02.005)
- Shah et al\.\(2023\)Shariq Shah, Hossein Ghomeshi, Edlira Vakaj, Emmett Cooper, and Shereen Fouad\. 2023\.A Review of Natural Language Processing in Contact Centre Automation\.*Pattern Analysis and Applications*26, 3 \(Aug\. 2023\), 823–846\.[doi:10\.1007/s10044\-023\-01182\-8](https://doi.org/10.1007/s10044-023-01182-8)
- Smith et al\.\(2018\)Jennifer Smith, Andreas Tsiartas, Valerie Wagner, Elizabeth Shriberg, and Nikoletta Bassiou\. 2018\.Crowdsourcing emotional speech\. In*2018 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\)*\. IEEE, 5139–5143\.
- Soma et al\.\(2023\)Christina S\. Soma, Dillon Knox, Timothy Greer, Keith Gunnerson, Alexander Young, and Shrikanth Narayanan\. 2023\.It’s Not What You Said, It’s How You Said It: An Analysis of Therapist Vocal Features during Psychotherapy\.*Counselling and Psychotherapy Research*23, 1 \(March 2023\), 258–269\.[doi:10\.1002/capr\.12489](https://doi.org/10.1002/capr.12489)
- Wang et al\.\(2023\)Lingli Wang, Ni Huang, Yili Hong, Luning Liu, Xunhua Guo, and Guoqing Chen\. 2023\.Voice\-based AI in call center customer service: A natural field experiment\.*Production and Operations Management*32, 4 \(2023\), 1002–1018\.arXiv:https://doi\.org/10\.1111/poms\.13953[doi:10\.1111/poms\.13953](https://doi.org/10.1111/poms.13953)
- Wang et al\.\(2025\)Qichao Wang, Ziqiao Meng, Wenqian Cui, Yifei Zhang, Pengcheng Wu, Bingzhe Wu, Irwin King, Liang Chen, and Peilin Zhao\. 2025\.Ntpp: Generative speech language modeling for dual\-channel spoken dialogue via next\-token\-pair prediction\.*arXiv preprint arXiv:2506\.00975*\(2025\)\.
- Xu et al\.\(2025\)Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin\. 2025\.Qwen2\.5\-Omni Technical Report\.arXiv:2503\.20215 \[cs\][doi:10\.48550/arXiv\.2503\.20215](https://doi.org/10.48550/arXiv.2503.20215)