A Benchmark for Controllable Speaking-Style Captioning in Audio-Language Models

# A Benchmark for Controllable Speaking-Style Captioning in Audio-Language Models
Source: [https://arxiv.org/html/2610.33036](https://arxiv.org/html/2610.33036)
Shreeram Suresh ChandraAffiliation:Center for Language and Speech Processing, Johns Hopkins University, USAAffiliation:The University of Texas at Dallas, USALiang LuAffiliation:Otter AIXiutian ZhaoAffiliation:Center for Language and Speech Processing, Johns Hopkins University, USALeda SariAffiliation:Otter AIBjörn SchullerAffiliation:GLAM – Group on Language, Audio, & Music, Imperial College London, UK\*Work done during an internship at Otter AI\.Berrak SismanAffiliation:Center for Language and Speech Processing, Johns Hopkins University, USA

###### Abstract

Speaking\-style captioning \(SSC\) aims to generate natural language descriptions ofhowspeech is delivered, capturing paralinguistic attributes such as vocal timbre, prosody, and expressivity\. Many downstream applications, including conversational AI agents, controllable speech generation, and large\-scale audio annotation, require dimension\-specific style captions that describe targeted aspects of speech \(e\.g\., speaker traits, emotion, or delivery style\), rather than a single undifferentiated description\. However, existing work lacks a unified formulation for controllable speaking\-style captioning\. We address this gap by formulating SSC as an instruction\-following audio\-language task, where explicit instructions specify the speaking\-style dimensions to be described\. We introduceStyleInstructCaps, the first standardized benchmark for controllable speaking\-style captioning in audio\-language models\.StyleInstructCapsprovides a dataset and evaluation framework that measures metadata groundedness, hallucination, instruction\-following ability, generalization to unseen instructions and audio datasets, and speaker\-style consistency\. Across nine state\-of\-the\-art audio\-language models, we find that strong instruction\-following ability does not necessarily imply grounded speaking\-style understanding, and that SSC\-specific instruction\-diverse training substantially improves groundedness, controllability, and speaker consistency\.111Code, data, and benchmark are available at[https://styleinstructcaps\.github\.io](https://styleinstructcaps.github.io/)

## 1Introduction

Speaking\-style captioning \(SSC\) has recently emerged as a promising direction for generating natural\-language descriptions of how speech is delivered, capturing properties such as vocal timbre, prosody, rhythm, emphasis, emotion, and other paralinguistic cues[Yamauchi et al\. \(2024\)](https://arxiv.org/html/2610.33036#bib.bib2)\. Despite growing interest, SSC still lacks a unified task formulation and evaluation paradigm\.

Researchers in audio understanding often treat SSC as an extension of speech comprehension, emphasizing communicative intent and discourse\-level meaning[Jing et al\. \(2024\)](https://arxiv.org/html/2610.33036#bib.bib7);[Chandra et al\. \(2025\)](https://arxiv.org/html/2610.33036#bib.bib3);[Zhang et al\. \(2025b\)](https://arxiv.org/html/2610.33036#bib.bib4)\. In contrast, speech synthesis and generative modeling communities typically view speaking\-style captions as controllable descriptions of vocal characteristics for expressive text\-to\-speech[Wang et al\. \(2025a\)](https://arxiv.org/html/2610.33036#bib.bib5);[Guo et al\. \(2023\)](https://arxiv.org/html/2610.33036#bib.bib6)and voice conversion systems[Kuan et al\. \(2023\)](https://arxiv.org/html/2610.33036#bib.bib9);[Hai et al\. \(2024\)](https://arxiv.org/html/2610.33036#bib.bib8)\. These differing perspectives reflect a deeper challenge:speaking\-style is inherently subjective[Adigwe et al\. \(2024\)](https://arxiv.org/html/2610.33036#bib.bib10), spanning multiple interacting paralinguistic factors such as speaker identity, affect, prosody, and contextual delivery\. As a result, existing SSC systems often capture different notions of style, making evaluation and comparison across models difficult\.

![Refer to caption](https://arxiv.org/html/2610.33036v2/CONCEPTUAL_FIG_v7.drawio.png)

Figure 1:Speaking\-style captioning as an instruction\-following audio\-language taskTo mitigate this ambiguity, we reformulate SSC as aninstruction\-following audio\-language task, where explicit instructions specify which aspect of speaking\-style the model should describe, as illustrated in Figure[1](https://arxiv.org/html/2610.33036#S1.F1)\. Conventional SSC approaches typically produce a single holistic description of speech, often entangling speaker traits, emotion, prosody, and delivery characteristics, making captions difficult to interpret or evaluate systematically\. In contrast, controllable SSC conditions generation on explicit style instructions, enabling the same utterance to yield multiple targeted and complementary descriptions depending on the task objective\.

Despite their effectiveness on narrowly defined objectives, existing SSC models often adopt a single view of speaking\-style, typically focusing on speaker\-centric or emotion\-centric attributes, which limits generalization to broader stylistic phenomena\. Since speaking\-style depends on downstream context, SSC models must be controllable and generate captions conditioned on explicit style instructions\. Motivated by this perspective, we introduceStyleInstructCaps, the first standardized benchmark for controllable SSC\.StyleInstructCapsevaluates generated captions along five dimensions: \(1\)metadata groundedness, \(2\)hallucination, \(3\)instruction\-following ability, \(4\)generalization to unseen speech data and prompts, and \(5\)speaker\-style consistency\.

To support this benchmark, we construct a multi\-style speaking\-style caption corpus built on top of the ParaSpeechCaps \(PSC\) dataset[Diwan et al\. \(2025\)](https://arxiv.org/html/2610.33036#bib.bib11)\. The dataset contains 2\.7k hours of speech, where each utterance is paired with six complementary caption types targeting distinct dimensions of speaking\-style, including speaker identity, contextual delivery, emotion, pragmatics, listener perception, and holistic style summaries\. These parallel annotations enable controlled evaluation of whether models can isolate specific stylistic dimensions, integrate multiple cues when required, and generalize across diverse speaking conditions\.

Overall, this work makes three main contributions: \(1\) We present the first formulation of controllable SSC as an instruction\-following audio\-language task, in which style descriptions are generated conditioned on explicit stylistic instructions; \(2\) We introduce StyleInstructCaps, a standardized and challenging benchmark supported by a 2\.7k\-hour multi\-style speaking\-style caption corpus with six parallel caption types per utterance; \(3\) We evaluate nine state\-of\-the\-art audio\-language models, including Audio\-Flamingo\-3[Goel et al\. \(2025\)](https://arxiv.org/html/2610.33036#bib.bib69), Voxtral[Liu et al\. \(2025\)](https://arxiv.org/html/2610.33036#bib.bib70), Qwen2\-Audio[Chu et al\. \(2024\)](https://arxiv.org/html/2610.33036#bib.bib71), Qwen3\-Omni[Xu et al\. \(2025\)](https://arxiv.org/html/2610.33036#bib.bib78), MOSS\-Audio[Gong et al\. \(2026\)](https://arxiv.org/html/2610.33036#bib.bib74), Audio Flamingo Next[Ghosh et al\. \(2026\)](https://arxiv.org/html/2610.33036#bib.bib73), MERaLiON\-AudioLLM[MERaLiON Team \(2025\)](https://arxiv.org/html/2610.33036#bib.bib72), and SALMONN[Tang et al\. \(2024\)](https://arxiv.org/html/2610.33036#bib.bib12), together with SSC\-specific instruction\-tuned variants of SALMONN\. We find a consistent gap between instruction\-following ability and grounded speaking\-style understanding: models often generate stylistically plausible captions while hallucinating unsupported paralinguistic attributes, whereas SSC\-specific, instruction\-diverse training substantially improves groundedness, controllability, and speaker consistency\.

Through this work, we aim to establish a unified foundation for controllable SSC and enable more faithful and interpretable speaking\-style understanding in audio\-language models\.

## 2Related Work

### 2\.1Speaking\-Style Captioning for Paralinguistic Speech Understanding

Speaking\-style captioning aims to generate natural\-language descriptions of*how*speech is delivered, capturing attributes such as vocal timbre, prosody, emotion, and delivery style[Yamauchi et al\. \(2024\)](https://arxiv.org/html/2610.33036#bib.bib2)\. Unlike categorical paralinguistic tasks such as speaker recognition[Hansen and Hasan \(2015\)](https://arxiv.org/html/2610.33036#bib.bib32)and speech emotion recognition[Triantafyllopoulos et al\. \(2026\)](https://arxiv.org/html/2610.33036#bib.bib33), which assign a fixed label, and content\-oriented tasks such as speech question answering[Peng et al\. \(2026\)](https://arxiv.org/html/2610.33036#bib.bib34)and speech summarization[Shang et al\. \(2024\)](https://arxiv.org/html/2610.33036#bib.bib35);[Retkowski et al\. \(2025\)](https://arxiv.org/html/2610.33036#bib.bib53), which address what is said rather than how it is said, SSC produces open\-ended descriptions that integrate multiple perceptual cues\. This capability has become increasingly relevant in large audio\-language models, where natural\-language descriptions provide a flexible and interpretable interface for expressing subjective speaking\-style phenomena[Su et al\. \(2026\)](https://arxiv.org/html/2610.33036#bib.bib52);[Dhamyal et al\. \(2024\)](https://arxiv.org/html/2610.33036#bib.bib29);[Lu et al\. \(2024\)](https://arxiv.org/html/2610.33036#bib.bib30);[Lu et al\. \(2026\)](https://arxiv.org/html/2610.33036#bib.bib31)\. In the reverse direction, speech\-generative audio\-language models increasingly accept natural\-language instructions to produce expressive speech, yet reliably realizing a requested emotion without degrading linguistic content remains challenging\([Zhang et al\., 2026](https://arxiv.org/html/2610.33036#bib.bib80);[Zhao et al\., 2026](https://arxiv.org/html/2610.33036#bib.bib65)\)\.

### 2\.2Datasets and Benchmarks

Progress in SSC depends on \(1\) speech corpora with rich speaking\-style supervision and \(2\) evaluation protocols that assess generation quality beyond lexical overlap\. Existing resources span multiple communities: expressive TTS and style\-prompted speech datasets provide speaker and delivery annotations, while paralinguistic benchmarks evaluate reasoning over cues such as emotion, prosody, and conversational pragmatics\. Appendix[A](https://arxiv.org/html/2610.33036#A1), Table[5](https://arxiv.org/html/2610.33036#A1.T5)and Table[6](https://arxiv.org/html/2610.33036#A1.T6)summarize representative datasets and benchmarks in this space\. However, most existing benchmarks emphasize*recognition*rather than SSC as a standalone*generation*task, and do not support*controllable*evaluation over which stylistic dimensions should be described\. This makes it difficult to compare SSC models consistently or evaluate whether models can selectively attend to requested style factors\.

### 2\.3Speaking\-Style Captioning Models

Advances in audio\-language models enable speech\-conditioned text generation[Cui et al\. \(2025\)](https://arxiv.org/html/2610.33036#bib.bib14), typically by coupling a speech encoder with a pretrained LLM through an adapter[Verdini et al\. \(2025\)](https://arxiv.org/html/2610.33036#bib.bib15)\. SSC\-relevant approaches fall into two families\. \(1\)*General\-purpose*audio\-language models trained via multitask or instruction tuning[Das et al\. \(2025\)](https://arxiv.org/html/2610.33036#bib.bib16);[Chu et al\. \(2023\)](https://arxiv.org/html/2610.33036#bib.bib17);[Lu et al\. \(2026\)](https://arxiv.org/html/2610.33036#bib.bib31)can produce style\-related descriptions[Huang et al\. \(2025\)](https://arxiv.org/html/2610.33036#bib.bib18), likely due to broad task coverage \(e\.g\., ASR, AC, AST, SER, SV\)[Tang et al\. \(2024\)](https://arxiv.org/html/2610.33036#bib.bib12)\. \(2\)*Task\-specialized*SSC models explicitly target style caption generation: StyleCap[Yamauchi et al\. \(2024\)](https://arxiv.org/html/2610.33036#bib.bib2)combines a self\-supervised speech encoder[Chen et al\. \(2022\)](https://arxiv.org/html/2610.33036#bib.bib19)with an auto\-regressive LM; factor\-conditioned methods introduce intermediate attribute prediction[Ando et al\. \(2024\)](https://arxiv.org/html/2610.33036#bib.bib20); emotion\-focused SSC variants emphasize free\-form generation and multimodal prompting[Xu et al\. \(2024a\)](https://arxiv.org/html/2610.33036#bib.bib21);[Xu et al\. \(2024b\)](https://arxiv.org/html/2610.33036#bib.bib22); and joint frameworks connect SSC with prompt\-based speech synthesis[Zhu et al\. \(2024\)](https://arxiv.org/html/2610.33036#bib.bib1)\. Despite this progress, prior work rarely provides a standardized setting to test instruction\-level controllability across multiple stylistic factors, motivating our benchmark design\.

## 3Methodology

We first formalize controllable SSC as an instruction\-conditioned generation task \(Section[3\.1](https://arxiv.org/html/2610.33036#S3.SS1)\), then describeStyleInstructCaps\-DB, the multi\-dimensional caption dataset underlying the benchmark \(Section[3\.2](https://arxiv.org/html/2610.33036#S3.SS2)\)\. The evaluation tasks follow in Section[4](https://arxiv.org/html/2610.33036#S4)\.

### 3\.1Problem Formulation

We definecontrollable SSCas generating, for a given speech signal, a natural\-language caption that describes only the speaking\-style dimension requested by an explicit instruction \(e\.g\., speaker traits, emotional expression, or delivery style\)\. Unlike traditional SSC, which produces a single holistic description per utterance, controllable SSC requires models to selectively ground instruction\-relevant attributes while avoiding unsupported or irrelevant ones\.

Let𝒟=\{\(xi,si,yi\)\}i=1N\\mathcal\{D\}=\\\{\(x\_\{i\},s\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{N\}contain speech utterancesxix\_\{i\}, free\-form instructionssis\_\{i\}, and reference captionsyi=\(yi,1,…,yi,Ti\)y\_\{i\}=\(y\_\{i,1\},\\dots,y\_\{i,T\_\{i\}\}\)\. An audio\-language model with parametersθ\\thetageneratesy^∼pθ​\(y∣x,s\)\\hat\{y\}\\sim p\_\{\\theta\}\(y\\mid x,s\)\.StyleInstructCaps\-DBcovers six speaking\-style dimensions \(Section[3\.2](https://arxiv.org/html/2610.33036#S3.SS2)\), and the benchmark adds unseen instructions \(Section[4](https://arxiv.org/html/2610.33036#S4)\)\.

![Refer to caption](https://arxiv.org/html/2610.33036v2/StyleinstructCaps_fig_v2.png)

Figure 2:Overview ofStyleInstructCaps, which evaluates models on metadata groundedness, hallucination, instruction\-following, generalization to unseen prompts and speech data, and speaker\-style consistency\.
### 3\.2Dataset Description

We introduceStyleInstructCaps\-DB, a multi\-style speaking\-style caption dataset built on the existing SSC corpus ParaSpeechCaps \(PSC\)[Diwan et al\. \(2025\)](https://arxiv.org/html/2610.33036#bib.bib11)\. PSC aggregates multiple speech datasets and comprises two subsets: PSC\-base, which includes Expresso[Nguyen et al\. \(2023\)](https://arxiv.org/html/2610.33036#bib.bib23), EARS[Richter et al\. \(2024\)](https://arxiv.org/html/2610.33036#bib.bib24), and VoxCeleb[Nagrani et al\. \(2017\)](https://arxiv.org/html/2610.33036#bib.bib54)and is human\-annotated, and PSC\-scaled, which consists of the Emilia dataset[He et al\. \(2024\)](https://arxiv.org/html/2610.33036#bib.bib55)and is automatically annotated\. These annotations cover diverse intrinsic and situational speaking\-style attributes across speakers and recording conditions\. A brief overview of the PSC annotation procedure is provided in Appendix[B](https://arxiv.org/html/2610.33036#A2)\.

While PSC provides a single caption for most utterances,StyleInstructCaps\-DBintroduces six complementary stylistic captions per utterance: \(1\) speaker idiosyncratic style, \(2\) situational or contextual delivery style, \(3\) expressive or emotional style, \(4\) linguistic\-pragmatic style, \(5\) perceptual listener\-centric style, and \(6\) an overall style summary\. These caption types jointly span representative dimensions of speaking\-style used in speech understanding and generation tasks\.

Captions are generated using an LLM conditioned on the rich per\-utterance metadata provided by PSC; an example metadata record is shown in the Appendix[C](https://arxiv.org/html/2610.33036#A3)Figure[4](https://arxiv.org/html/2610.33036#A3.F4)\.StyleInstructCaps\-DBis organized into train, validation, and evaluation splits\. We use Mistral\-Small\-24B\-Instruct[Mistral AI Team \(2025\)](https://arxiv.org/html/2610.33036#bib.bib75)to generate captions for the train and validation splits, and Llama\-3\.1\-8B\-Instruct[LlamaTeam \(2024\)](https://arxiv.org/html/2610.33036#bib.bib56)for the evaluation split, ensuring that benchmark evaluation does not benefit from exposure to captions generated by the same language model\. Appendix[C](https://arxiv.org/html/2610.33036#A3)Figure[5](https://arxiv.org/html/2610.33036#A3.F5)presents the captioning prompt, including explicit definitions of the six speaking\-style dimensions, while Appendix[C](https://arxiv.org/html/2610.33036#A3)Table[7](https://arxiv.org/html/2610.33036#A3.T7)reports instruction\-caption pairs for the same utterance shown in Appendix[C](https://arxiv.org/html/2610.33036#A3)Figure[4](https://arxiv.org/html/2610.33036#A3.F4)\.

## 4StyleInstructCaps Benchmark

We introduceStyleInstructCaps, a benchmark built onStyleInstructCaps\-DBfor evaluating SSC models\. The benchmark assesses captions along five evaluation dimensions \(Figure[2](https://arxiv.org/html/2610.33036#S3.F2)\): metadata groundedness, hallucination, instruction\-following, generalization to unseen instructions and speech data, and speaker\-style consistency\. The following subsections describe the setup and evaluation for each dimension\.

### 4\.1Metadata Groundedness

In the first evaluation task \(Figure[2](https://arxiv.org/html/2610.33036#S3.F2), Part A\), we assessmetadata groundedness, the extent to which a generated caption is supported by and consistent with the speech metadata\. We sample utterances from theStyleInstructCaps\-DBevaluation split with an emphasis on metadata richness, first selecting those with non\-empty situational tags and then completing the set from the remaining pool\. To ensure diversity, we cap each speaker at 10 utterances and limit any single accent to 25% of the set\. We refer to this subset asStyleInstructCaps\-MetaSet\(distribution plots for accent, emotion, signal\-to\-noise ratio \(SNR\), and speech duration in Appendix[C](https://arxiv.org/html/2610.33036#A3), Figure[6](https://arxiv.org/html/2610.33036#A3.F6)\)\. Using the overall style summary instruction, an LLM judge panel compares the generated caption against PSC ground\-truth metadata on speaker gender, accent, pitch, speaking rate, environment or noise level, and emotion or expression, with particular emphasis on prosodic alignment\. The judge assigns a 1–10 similarity score with a brief justification, and we report the mean across utterances as the meta\-score \(prompt in Appendix[H](https://arxiv.org/html/2610.33036#A8), Figure[9](https://arxiv.org/html/2610.33036#A8.F9)\)\.

### 4\.2Hallucination

In the second evaluation task \(Figure[2](https://arxiv.org/html/2610.33036#S3.F2), Part B\), we assess hallucination in generated speaking\-style captions\. We define hallucination as the inclusion of speaking\-style attributes that are unsupported by or contradictory to the input speech\. Evaluation is conducted onStyleInstructCaps\-MetaSet\.

The LLM judge compares factual claims in the generated caption against the corresponding speech metadata, focusing on attributes such as speaker identity, emotion, accent, and acoustic characteristics\. Descriptive or prosodic interpretations that do not contradict the metadata are treated leniently\. The judge outputs a hallucination score on a 0 \- 10 scale, together with a categorical severity label \(Low, Medium, or High\), reflecting both the frequency and severity of hallucinated attributes\. The prompt is provided in Appendix[H](https://arxiv.org/html/2610.33036#A8)Figure[10](https://arxiv.org/html/2610.33036#A8.F10)\.

### 4\.3Instruction\-Following

We evaluate the controllability of SSC models through an explicit instruction\-following task \(See Figure[2](https://arxiv.org/html/2610.33036#S3.F2)Part C\)\. Using theStyleInstructCaps\-MetaSetset, we assess whether models generate captions that adhere to user\-specified style instructions\. This evaluation probes instruction\-following ability across the six speaking\-style dimensions defined inStyleInstructCaps\-DB, testing whether models can selectively attend to and describe the intended stylistic attributes\. The LLM\-based judge evaluates each generated caption for compliance with the given instruction, considering content relevance, stylistic focus, and constraint satisfaction\. The judge outputs a categorical instruction\-following outcome \(Success or Failure\) and a scalar score on a 1–10 scale, accompanied by a brief justification\. We report the mean of this score as the*IFA score*\(instruction\-following accuracy, 1–10\) and the percentage of captions judged a Success as the*IFA rate*\(%\)\. The complete evaluation prompts are provided in Appendix[H](https://arxiv.org/html/2610.33036#A8)Figure[11](https://arxiv.org/html/2610.33036#A8.F11)and[12](https://arxiv.org/html/2610.33036#A8.F12)\.

### 4\.4Generalization to Unseen Instructions

Beyond the six speaking\-style dimensions inStyleInstructCaps\-MetaSet, we assess generalization to instructions that differ*both in surface form and output requirements*from those seen during training\. Training instructions are primarily narrative directives, whereas the unseen set is deliberately heterogeneous and includes short, deployment\-oriented controls: slot\-/tag\-like schemas \(e\.g\., YAML or Python\-dict templates\), strict JSON outputs, hard lexical and length constraints, numeric rating scales, and persona\-conditioned prompts \(e\.g\., call\-center evaluator\)\. This tests whether instruction tuning yields genuine instruction\-following ability rather than overfitting to the verbose narrative style of the training distribution\. We pair two utterances from each of 16 accent\-diverse VCTK speakers[Yamagishi et al\. \(2019\)](https://arxiv.org/html/2610.33036#bib.bib60)with all 36 unseen instructions \(Appendix[H](https://arxiv.org/html/2610.33036#A8), Figure[13](https://arxiv.org/html/2610.33036#A8.F13)\)\. Each of the resulting 1,152 outputs per system is scored with the instruction\-following protocol and Qwen3\-32B judge of Section[4\.3](https://arxiv.org/html/2610.33036#S4.SS3), and we report the mean 1–10 score per system \(Table[4](https://arxiv.org/html/2610.33036#S6.T4)\)\.

### 4\.5Generalization to Unseen Speech Dataset

To evaluate SSC generalization to unseen speech datasets, we construct an evaluation set using the GigaSpeech corpus[Chen et al\. \(2021\)](https://arxiv.org/html/2610.33036#bib.bib61)with corresponding captions from Speechcraft[Jin et al\. \(2024\)](https://arxiv.org/html/2610.33036#bib.bib27)\. We sample 600 speech segments \(300 from YouTube and 300 from podcasts\), ensuring each segment originates from a distinct episode and is at least 3 seconds long\. Each utterance is paired with a randomly sampled overall style summary speaking\-style instruction, enabling evaluation under realistic acoustic conditions and diverse recording environments\. Following the protocols in Sections[4\.1](https://arxiv.org/html/2610.33036#S4.SS1)and[4\.2](https://arxiv.org/html/2610.33036#S4.SS2), we assess metadata groundedness and hallucination with respect to the SpeechCraft captions\. We note that Speechcraft provides limited metadata coverage compared to PSC\.

Table 1:Metadata groundedness, hallucination severity, and instruction\-following performance evaluated onStyleInstructCaps\-MetaSet\. Blue shading indicates relative performance for scalar metrics\. Metadata groundedness and hallucination are averaged over a panel of three LLM judges \(Qwen3\-32B, Gemma\-4\-31B\-it, GPT\-OSS\-20B\) with three runs per judge \(9 runs per model\); instruction\-following is scored with a single Qwen3\-32B run\.Model nameLLM metadatagroundednessLLM hallucinationseverity \(%\)InstructionfollowingMeta score↑\\uparrowHall\.↓\\downarrowLowMidHighIFA score↑\\uparrowIFA rate\(%\)↑\\uparrowOracle Caption \(PSC\)9\.480\.0399\.50\.50\-\-Oracle caption \(SIC\-DB\)9\.860\.2299\.80\.208\.4871\.3Audio\-Flamingo\-33\.886\.4315\.420\.763\.88\.2162\.4Voxtral\-Mini\-3B4\.994\.2948\.635\.016\.48\.1568\.0Voxtral\-Small\-24B4\.381\.9973\.516\.89\.687\.5167\.8Qwen2\-Audio\-7B\-Instruct3\.627\.0019\.127\.953\.06\.5853\.6MERaLiON\-AudioLLM3\.621\.2681\.810\.67\.606\.1127\.7MOSS\-Audio6\.325\.5021\.845\.432\.78\.8282\.8Qwen3\-Omni5\.753\.8651\.230\.618\.27\.9974\.0Audio\-Flamingo\-next5\.095\.5419\.3538\.9441\.77\.5948\.5SALMONN4\.684\.6642\.128\.329\.57\.5058\.5SALMONN \(PSC\)7\.015\.3721\.246\.332\.57\.9852\.9SALMONN \(SIC\-DB\) \- ST7\.304\.3335\.237\.327\.48\.3765\.6SALMONN \(SIC\-DB\) \- MT7\.354\.0441\.338\.220\.58\.6275\.0

Closed\-source reference— OpenAI\-GPT\-Audio: Meta score↑\\uparrow5\.70, Hall\.↓\\downarrow5\.14\.

Table 2:Speaker\-style consistency evaluation using clustering and verification metrics on the ID\-balanced and UD\-VCTK sets\. Darker shading indicates better performance per column\. Higher is better for ARI, NMI, Purity, InterSep, and ROC\-AUC; lower is better for EER\.ModelID\-balancedUD\-VCTKARI↑\\uparrowNMI↑\\uparrowPurity↑\\uparrowInterSep↑\\uparrowEER↓\\downarrowAUC↑\\uparrowARI↑\\uparrowNMI↑\\uparrowPurity↑\\uparrowInterSep↑\\uparrowEER↓\\downarrowAUC↑\\uparrowAudio\-Flamingo\-30\.0090\.0460\.1000\.0460\.4760\.5320\.0110\.0480\.1210\.0570\.4630\.556Voxtral\-Mini\-3B0\.0330\.1220\.1520\.1330\.3990\.6430\.0030\.0280\.1030\.0630\.4860\.519Voxtral\-Small\-24B0\.0670\.1820\.1860\.1570\.3630\.6740\.0110\.0520\.1190\.0760\.4800\.532Qwen2\-Audio\-7B\-Instruct\-0\.0020\.0100\.0720\.0260\.4980\.499\-0\.0020\.0070\.0830\.0250\.4960\.498MERaLiON\-AudioLLM0\.0810\.2760\.2130\.2330\.4280\.6040\.0390\.1460\.1670\.1690\.4760\.534MOSS\-Audio0\.1910\.3810\.3470\.2210\.2820\.7920\.0760\.2280\.1860\.1750\.3190\.755Qwen3\-Omni0\.1520\.2970\.2430\.2040\.3760\.6300\.0690\.2160\.1840\.1650\.3110\.734Audio\-Flamingo\-Next0\.1130\.3180\.2420\.2270\.3480\.7170\.0860\.2640\.1870\.2120\.3850\.673SALMONN0\.3510\.5980\.5110\.2900\.2250\.8540\.1450\.3240\.3030\.2050\.3200\.749SALMONN \(PSC\)0\.2970\.5410\.4910\.2710\.2310\.8610\.1570\.3400\.3120\.2090\.3200\.711SALMONN\(SIC\-DB\) \- ST0\.3630\.5770\.5080\.2840\.2140\.8760\.1450\.3240\.3030\.2050\.3200\.749SALMONN\(SIC\-DB\) \- MT0\.3570\.5900\.5170\.2790\.2120\.8760\.1560\.3410\.2980\.2220\.2820\.795

### 4\.6Speaker\-Style Consistency

Speaker\-specific style captions describe inherent vocal traits and enable text\-prompted speech synthesis via natural language control rather than explicit speaker IDs[Wang et al\. \(2025a\)](https://arxiv.org/html/2610.33036#bib.bib5)\.

To evaluate speaker\-level consistency, we design two complementary evaluation schemes \(See Figure[2](https://arxiv.org/html/2610.33036#S3.F2)Part E\)\.Speaker\-caption verificationmeasures pairwise consistency by testing whether captions generated from different utterances preserve invariant speaker\-specific attributes\.Speaker\-caption clustering analysisevaluates the global structure of speaker\-related information in the caption embedding space, examining whether captions from the same speaker form coherent and well\-separated clusters\.

Experiments are conducted on two datasets\. The first is an in\-domain balanced set \(ID\-Balanced\) from theStyleInstructCaps\-DBevaluation split, consisting of 400 utterances from speakers with at least 20 utterances each, with balanced gender and accent coverage \(See Appendix[C](https://arxiv.org/html/2610.33036#A3)Figure[7](https://arxiv.org/html/2610.33036#A3.F7)for distribution plots\)\. The second is an unseen\-speaker set \(UD\-VCTK\) derived from VCTK, containing 320 utterances from 16 speakers spanning diverse accents\. In both settings, each utterance is paired with the same 10 speaker\-specific instructions applied to assess robustness to unseen speakers, accents, and recording conditions\.

#### Speaker\-Caption Verification\.

We formulate speaker\-caption verification as a binary verification task in the text embedding space, analogous to speaker verification in the speech domain[Dehak et al\. \(2011\)](https://arxiv.org/html/2610.33036#bib.bib62)\. Positive trials pair captions from different utterances of the same speaker, while negative trials pair captions from different speakers\. Captions are encoded using the pretrained E5\-base\-v2 text encoder[Wang et al\. \(2024\)](https://arxiv.org/html/2610.33036#bib.bib63), followed by attention\-masked average pooling and L2 normalization\. Speaker consistency is measured using cosine similarity between caption embeddings, with performance evaluated by Equal Error Rate \(EER\) and ROC\-AUC\.

#### Speaker\-Caption Clustering Analysis\.

To assess global speaker organization, caption embeddings are clustered using k\-means, with the number of clusters set equal to the number of speakers\. Clustering quality is evaluated using adjusted rand index \(ARI\), normalized mutual information \(NMI\), and cluster purity\. Inter\-cluster separation is additionally reported to quantify global separation between speakers\.

### 4\.7Are LLM judges reliable for the SSC task evaluation?

For metadata groundedness, hallucination, instruction\-following, and generalization, we adopt the LLM\-as\-a\-judge paradigm[Zheng et al\. \(2023\)](https://arxiv.org/html/2610.33036#bib.bib47), which prior work has shown to align better with human preferences than automatic metrics such as ROUGE[Lin \(2004\)](https://arxiv.org/html/2610.33036#bib.bib51), CIDEr[Vedantam et al\. \(2015\)](https://arxiv.org/html/2610.33036#bib.bib57), BERTScore[Zhang\* et al\. \(2020\)](https://arxiv.org/html/2610.33036#bib.bib58), and n\-gram based measurements[Mondshine et al\. \(2025\)](https://arxiv.org/html/2610.33036#bib.bib59)\. We validate this choice with three experiments\.

Adversarial gaming check\.Following the LLM judge reliability protocol of EmoSura[Jing et al\. \(2026\)](https://arxiv.org/html/2610.33036#bib.bib77), we introduce controlled poisoning perturbations into the oracle captions\. The resulting score shifts show that the judges detect such degradation in caption quality \(Appendix[D](https://arxiv.org/html/2610.33036#A4)\)\.

Panel of LLM judges\.To ensure that the reported scores and system rankings are not an artifact of a single evaluator, we use the panel of three judges described in Section[5](https://arxiv.org/html/2610.33036#S5)\. All judge pairs rank the systems consistently \(Spearmanρ≥0\.864\\rho\\geq 0\.864\), so the metadata\-groundedness and hallucination scores in Tables[1](https://arxiv.org/html/2610.33036#S4.T1)and[3](https://arxiv.org/html/2610.33036#S6.T3)are averaged over the full panel \(Appendix[F](https://arxiv.org/html/2610.33036#A6)\)\.

Human agreement with the judge\.In a pairwise preference study, thirteen listeners made forced choices between captions of different SSC systems over 100 caption pairs spanning all 14 systems in Table[1](https://arxiv.org/html/2610.33036#S4.T1), while hearing only the raw audio\. The human majority vote agrees with the judge panel on 85\.9% of pairs \(Cohen’sκ=0\.717\\kappa=0\.717\), rising from 73\.3% on near\-ties to 93\.3% on the largest score gaps, and a Bradley–Terry ranking fit to the human votes recovers the judge system ordering at Spearmanρ=0\.895\\rho=0\.895\. The protocol and full results are given in Appendix[E](https://arxiv.org/html/2610.33036#A5)\.

## 5Experiment Setup

We use a panel of three open\-weight LLM judges from different model families: Qwen3\-32B[Yang et al\. \(2025\)](https://arxiv.org/html/2610.33036#bib.bib50)\(thinking\), Gemma\-4\-31B\-it \(thinking\)[Team et al\. \(2026\)](https://arxiv.org/html/2610.33036#bib.bib49), and GPT\-OSS\-20B \(reasoning\)[OpenAI \(2025\)](https://arxiv.org/html/2610.33036#bib.bib48)\. Open\-weight judges enable reproducible, open evaluation compared to closed\-source alternatives\. Metadata groundedness and hallucination are averaged over this panel, with three runs per judge onStyleInstructCaps\-MetaSetand two runs per judge on the out\-of\-domain evaluation sets \(GigaSpeech, VCTK\)\. Instruction\-following is scored with a single Qwen3\-32B run because of its high computational cost of 2,604 judgments per audio\-language model, judge, and run \(Appendix[F](https://arxiv.org/html/2610.33036#A6)\)\.

We evaluate nine open\-source audio\-language models: Audio\-Flamingo\-3, Voxtral\-Mini\-3B, Voxtral\-Small\-24B, Qwen2\-Audio\-7B\-Instruct, MERaLiON\-AudioLLM, Audio\-Flamingo\-Next, MOSS\-Audio, Qwen3\-omni and SALMONN \(specifications in Appendix[I](https://arxiv.org/html/2610.33036#A9), Table[19](https://arxiv.org/html/2610.33036#A9.T19)\)\. As a closed\-source reference, we additionally evaluate OpenAI\-GPT\-Audio; we report only its core groundedness metrics, as the fine\-grained analyses require large\-scale inference with a paid API\. We further instruction\-tune SALMONN under three configurations: \(1\) using the original ParaSpeechCaps captions, SALMONN \(PSC\); \(2\) using the overall style summary captions fromStyleInstructCaps\-DBin a single\-task setup, SALMONN \(SIC\-DB\)\-ST; and \(3\) using the multi\-caption set from SIC\-DB with multiple instruction\-caption pairs per utterance in a multi\-task setup, SALMONN \(SIC\-DB\)\-MT\. All variants are trained with supervised fine\-tuning \(SFT\), minimizing the token\-level negative log\-likelihood over their respective training set𝒟\\mathcal\{D\}\(Section[3\.1](https://arxiv.org/html/2610.33036#S3.SS1)\):

ℒSFT​\(θ\)\\displaystyle\\mathcal\{L\}\_\{\\text\{SFT\}\}\(\\theta\)=𝔼\(x,s,y\)∼𝒟\\displaystyle=\\mathbb\{E\}\_\{\(x,s,y\)\\sim\\mathcal\{D\}\}\[∑t=1T−logpθ\(yt∣x,s,y<t\)\]\.\\displaystyle\\left\[\\sum\_\{t=1\}^\{T\}\-\\log p\_\{\\theta\}\\\!\\left\(y\_\{t\}\\mid x,s,y\_\{<t\}\\right\)\\right\]\.Additional training and inference details are in Appendix[I](https://arxiv.org/html/2610.33036#A9)\. Data pre\-processing follows the ParaSpeechCaps setup[Diwan et al\. \(2025\)](https://arxiv.org/html/2610.33036#bib.bib11), and split statistics are in Appendix[C](https://arxiv.org/html/2610.33036#A3), Table[8](https://arxiv.org/html/2610.33036#A3.T8)\. We further include an analysis of lexical diversity of the generated captions in the Appendix[J](https://arxiv.org/html/2610.33036#A10), reporting HD\-D, Self\-BLEU, Distinct\-2, Distinct\-3, and caption length statistics \(mean and standard deviation\) \- to measure the stylistic richness of the speaking style captions\.

## 6Results and Discussion

Table[1](https://arxiv.org/html/2610.33036#S4.T1)reports the results for the first three evaluation tasks – metadata groundedness, hallucination severity and instruction\-following, all evaluated by LLM judges\. Broadly, we observe thatgeneral\-purpose SOTA large audio\-language models tend to struggle with the SSC task, even though they have been trained in a large\-scale setting with iterative training strategies\. Metadata groundedness and hallucination severity are largely independent across models \(Spearmanρ≈0\\rho\\approx 0\): some weakly grounded models hallucinate rarely \(e\.g\., MERaLiON\-AudioLLM, Voxtral\-Small\-24B\) while others hallucinate heavily \(e\.g\., Qwen2\-Audio, Audio\-Flamingo\-3\), so a low hallucination score alone does not indicate faithful style description\. SSC\-specific training yields the largest gains in groundedness: all three SSC\-trained SALMONN variants exceed the best general\-purpose model, MOSS\-Audio\. Appendix[K](https://arxiv.org/html/2610.33036#A11)Figure[14](https://arxiv.org/html/2610.33036#A11.F14)provides an example of judgment for the metadata groundedness evaluation\. Pairwise significance tests over utterances are reported in \(Appendix[G](https://arxiv.org/html/2610.33036#A7)\)\.

Interestingly, models trained with SSC\-relevant audio encoders and tasks outperform those exposed only to ASR, which captureswhatis said but nothowit is said\. For instance, Audio\-Flamingo\-3 achieves state\-of\-the\-art results on MMAU[Sakshi et al\. \(2025\)](https://arxiv.org/html/2610.33036#bib.bib64)and MMAR[Ma et al\. \(2025\)](https://arxiv.org/html/2610.33036#bib.bib66)but shows a low metascore and high hallucination on SSC, while SALMONN, despite a smaller backbone, benefits from its BEATs[Chen et al\. \(2023\)](https://arxiv.org/html/2610.33036#bib.bib67)\+ Whisper[Radford et al\. \(2023\)](https://arxiv.org/html/2610.33036#bib.bib68)encoder and SSC\-relevant training tasks \(SQA, GR, SER\)\.

Next, we examine instruction\-following performance\. As shown in Table[1](https://arxiv.org/html/2610.33036#S4.T1), instruction\-following and metadata groundedness are moderately correlated across models,222Spearmanρ=0\.68\\rho=0\.68over the nine general\-purpose models \(ρ=0\.73\\rho=0\.73over all twelve systems\); between hallucination severity and instruction\-following,ρ=0\.27\\rho=0\.27\. Given the small number of systems, we report these as descriptive statistics\.but the relation is asymmetric: well\-grounded models also follow instructions well, whereas strong instruction\-following does not guarantee grounding\. Hallucination severity is largely unrelated to instruction\-following\. Performance is primarily driven by the diversity of tasks seen during instruction tuning, the capacity of the underlying LLM backbone, and hyperparameter choices that prevent overfitting to narrow instruction distributions\. MOSS\-Audio is the strongest general\-purpose model on both axes, showing that the two abilities can co\-occur\. In contrast, Audio\-Flamingo\-3 \(IFA score 8\.21\) and Voxtral\-Mini\-3B \(8\.15\) achieve strong instruction\-following despite weak faithfulness\-oriented performance \(meta scores of 3\.88 and 4\.99\)\.This dissociation indicates that generic instruction tuning suffices to produce stylistically well\-formed captions but fails to ground them in speech\-specific paralinguistic attributes\. SSC\-specific training alone does not improve instruction\-following \(IFA rate of 52\.9% for SALMONN \(PSC\) vs\. 58\.5% for SALMONN\), whereas instruction\-diverse training does: SALMONN \(SIC\-DB\)\-MT raises the IFA score from 7\.50 to 8\.62 and the IFA rate from 58\.5% to 75\.0%\.

On the unseen speech dataset \(Table[3](https://arxiv.org/html/2610.33036#S6.T3)\), SSC\-specific training brings no further groundedness gain: all SALMONN variants score 6\.25–6\.30, on par with base SALMONN \(6\.28; not significant, Appendix[G](https://arxiv.org/html/2610.33036#A7)\), while MOSS\-Audio scores highest \(6\.70\)\. Pooled instruction\-following scores also hide a large effect of output format\. We label each of the 36 unseen instructions by required format \(narrative, key\-value slots, tags and lists, minimal imperatives, persona\) and re\-score every system with the same Qwen3\-32B judge\. Table[4](https://arxiv.org/html/2610.33036#S6.T4)shows that format sensitivity is model\-specific: Audio\-Flamingo\-3 and Qwen2\-Audio lose much of their controllability on key\-value slots \(3\.34 and 4\.00 vs\. 7\.26 and 7\.27 on narrative\), the Voxtral models drop on minimal imperatives \(4\.85 and 3\.46\), and MOSS\-Audio and Qwen3\-Omni remain strong across all formats\. All SALMONN models, including the untuned baseline, score higher on key\-value and tag/list schemas than on narrative instructions, so this pattern stems from the backbone rather than SSC\-specific training\.

Table 3:Evaluation on the unseen speech dataset\. Metadata groundedness \(1–10,↑\\uparrow\) and hallucination count \(0–10,↓\\downarrow\) are averaged over the three\-judge panel \(Qwen3\-32B, Gemma\-4\-31B\-it, GPT\-OSS\-20B\) with two runs per judge \(6 runs per model\) on the GigaSpeech\-derived out\-of\-domain evaluation set\.ModelMeta score↑\\uparrowHall\.↓\\downarrowOpenAI\-GPT\-Audio4\.987\.02Audio\-Flamingo\-33\.855\.97Voxtral\-Mini\-3B4\.577\.12Voxtral\-Small\-24B3\.456\.45Qwen2\-Audio\-7B\-Instruct4\.626\.97MERaLiON\-AudioLLM3\.955\.72MOSS\-Audio6\.705\.90Qwen3\-Omni4\.403\.81Audio\-flamingo\-next5\.816\.65SALMONN6\.286\.29SALMONN \(PSC\)6\.286\.38SALMONN \(SIC\-DB\) \- ST6\.255\.93SALMONN \(SIC\-DB\) \- MT6\.306\.05

Table 4:Instruction\-following on unseen prompts, broken down by the output format each instruction requires\. Each cell aggregates 32 clips over the instructions in that family, giving 128 to 416 judgements per system and 1,152 per system in total\.ModelNarrativeKey\-valueTags/listMin\. imper\.PersonaAudio\-Flamingo\-37\.263\.344\.126\.397\.47Voxtral\-Mini\-3B6\.236\.307\.314\.857\.41Voxtral\-Small\-24B5\.437\.687\.303\.465\.82Qwen2\-Audio\-7B\-Instruct7\.274\.005\.635\.846\.96MERaLiON\-AudioLLM4\.506\.915\.705\.155\.79MOSS\-Audio8\.008\.578\.348\.398\.33Qwen3\-Omni7\.088\.899\.077\.897\.72Audio\-flamingo\-next6\.697\.627\.526\.387\.06SALMONN6\.307\.867\.456\.907\.67SALMONN \(PSC\)6\.577\.777\.525\.357\.14SALMONN \(SIC\-DB\) \- MT6\.697\.858\.286\.677\.68

![Refer to caption](https://arxiv.org/html/2610.33036v2/tsne_6speakers_singlecol_12.png)
Figure 3:t\-SNE visualization of speaker\-style caption embeddings on UD\-VCTK \(6 speaker subset\)Next, we turn to discussing speaker style consistency\. Table[2](https://arxiv.org/html/2610.33036#S4.T2)reveals clear and consistent trends in speaker\-style consistency across both the ID\-balanced set and UD\-VCTK sets\.SALMONN\-based models yield the most speaker\-consistent captions, and this advantage is already present in the untuned baseline\. All SALMONN variants outperform all other models on every metric on the ID\-balanced set and on ARI, NMI, and Purity on UD\-VCTK\. On UD\-VCTK, however, Qwen3\-Omni and MOSS\-Audio achieve lower EER than every variant except SALMONN \(SIC\-DB\)\-MT, which is best on EER \(0\.282\) and AUC \(0\.795\), and Audio\-Flamingo\-Next exceeds all variants but MT on InterSep\. These results indicate that SALMONN\-based captions encode stable, speaker\-specific paralinguistic attributes rather than surface\-level descriptions\. The t\-SNE[Maaten and Hinton \(2008\)](https://arxiv.org/html/2610.33036#bib.bib76)visualizations of speaker\-caption embeddings for a subset of six speakers from the UD\-VCTK set \(Figure[3](https://arxiv.org/html/2610.33036#S6.F3)\) show more coherent and well\-separated speaker clusters for SALMONN\-based models than for the other models\. A full t\-SNE analysis across all speakers is provided in Appendix[L](https://arxiv.org/html/2610.33036#A12)Figure[15](https://arxiv.org/html/2610.33036#A12.F15)\.

Relative to the untuned baseline, style\-caption fine\-tuning yields mixed, modest changes: SALMONN \(PSC\) improves clustering on UD\-VCTK but not on the ID\-balanced set, while theSIC\-DBvariants improve verification \(EER, AUC\) on the ID\-balanced set\. Single\-task instruction tuning \(ST\) further improves consistency on the ID\-balanced set but does not generalize to UD\-VCTK, indicating over\-specialization to a narrow instruction distribution\. In contrast,multi\-task instruction tuning \(MT\) yields the most balanced and transferable performance, with strong clustering structure and competitive verification scores across both datasets\. Finally, the strong agreement between clustering and verification metrics indicates that these improvements reflect a well\-structured caption embedding space\.Overall, speaker consistency stems mainly from the SALMONN backbone; among fine\-tuning strategies, instruction\-diverse \(MT\) training transfers best across domains\.

## 7Conclusions

We introducedStyleInstructCaps, a benchmark that formulates controllable SSC as an instruction\-following audio\-language task\. Its dataset,StyleInstructCaps\-DB, extends ParaSpeechCaps with six complementary captions per utterance, covering speaker idiosyncratic, situational, expressive, linguistic\-pragmatic, perceptual, and overall style\. The benchmark evaluates models along five axes: metadata groundedness, hallucination, instruction\-following, generalization to unseen instructions and speech datasets, and speaker\-style consistency\. For groundedness and hallucination, we designed an LLM\-as\-a\-judge protocol that checks captions against per\-utterance speech metadata\. We validated this protocol with adversarial perturbation checks, consistent system rankings across a three\-judge panel \(Spearmanρ≥0\.864\\rho\\geq 0\.864\), and a listening study in which the human majority vote agreed with the panel on 85\.9% of caption pairs\. We measured speaker\-style consistency directly in the caption embedding space, using speaker\-caption verification and clustering to test whether captions of the same speaker preserve invariant vocal traits across utterances\. Together, these components turn SSC from a loosely defined capability into a measurable research task\.

## Limitations

This study is subject to several limitations inherent to large\-scale speaking\-style captioning \(SSC\)\. First, publicly available speech corpora with rich, fine\-grained, per\-utterance speaking\-style annotations remain limited in both scale and diversity\. As a result, SSC systems are often trained and evaluated on datasets with constrained stylistic coverage, which may not fully capture the variability of real\-world expressive speech across speakers, domains, recording conditions, and cultural contexts\.

Second, the proposed evaluation framework relies substantially on LLM\-based judgment for assessing open\-ended caption quality, including groundedness, hallucination, and instruction\-following behavior\. Although recent LLM judges exhibit strong reasoning and language understanding capabilities, their evaluations remain dependent on the specific judge model, prompting strategy, and the distribution of examples encountered during pretraining and instruction tuning\. Consequently, evaluation outcomes may vary across alternative judge architectures or future generations of language models\. To reduce this dependence, we average the open\-ended metrics over a panel of three judges from different model families, whose system rankings agree closely \(pairwise Spearmanρ\\rhoof 0\.864–0\.978; Appendix[F](https://arxiv.org/html/2610.33036#A6)\)\.

Third, despite efforts to improve robustness through neutral prompt design, prompt paraphrasing, and multi\-prompt evaluation, LLM\-based assessments may still exhibit sensitivity to prompt wording, decoding configurations, or stylistic biases in generated text\. While our adversarial calibration experiments demonstrate that the judge is sensitive to targeted perturbations, such analyses cannot fully eliminate all forms of evaluation instability or hidden biases\.

Fourth, the overall pipeline introduces a potential closed\-loop evaluation concern, since metadata\-derived descriptions are used both to construct reference captions and to evaluate generated outputs through LLM\-based scoring\. Although we explicitly validate the sensitivity of the judge to adversarial manipulations and metadata inconsistencies, some degree of implicit alignment between the generation and evaluation processes may still remain\. Future work should further investigate evaluation paradigms involving independent human annotation and audio\-grounded verification\.

Finally, the current benchmark primarily evaluates metadata\-grounded captioning rather than fully audio\-grounded speaking\-style understanding\. In many cases, the reference captions reflect structured metadata annotations associated with the utterance rather than exhaustively capturing all perceptually salient properties present in the speech signal itself\. As a result, models may partially optimize toward reproducing metadata\-aligned descriptions instead of developing deeper holistic representations of expressive speech\. Bridging this gap between metadata supervision and truly audio\-grounded style understanding remains an important direction for future SSC research\.

## Ethical Considerations

In this work, we utilize open\-source audio\-language models to study the task of speaking\-style captioning\. We emphasize that the paralinguistic understanding capabilities of large language models remain an open research problem and are not yet sufficiently reliable for high\-stakes use\. Consequently, the outputs and evaluations presented in this paper should not be interpreted as substitutes for human judgment in scenarios that require factual accuracy, mental\-state inference, or sensitive assessments of speaker intent or affect\. Our results are intended to support research and benchmarking in controlled experimental settings, rather than deployment in applications where misinterpretation of paralinguistic cues could lead to harm\.

## References

- Adigweet al\.\(2024\)A\. Adigwe, S\. Wallbridge, and S\. KingWhat do people hear? Listeners’ Perception of Conversational Speech\.InInterspeech 2024,pp\. 1210–1214\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2024-1193),ISSN 2958\-1796Cited by:[§1](https://arxiv.org/html/2610.33036#S1.p2.1)\.
- Andoet al\.\(2024\)A\. Ando, T\. Moriya, S\. Horiguchi, and R\. MasumuraFactor\-Conditioned Speaking\-Style Captioning\.InInterspeech 2024,pp\. 782–786\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2024-633),ISSN 2958\-1796Cited by:[§2\.3](https://arxiv.org/html/2610.33036#S2.SS3.p1.1)\.
- Aoet al\.\(2024\)J\. Ao, Y\. Wang, X\. Tian, D\. Chen, J\. Zhang, L\. Lu, Y\. Wang, H\. Li, and Z\. WuSd\-eval: a benchmark dataset for spoken dialogue understanding beyond words\.Advances in Neural Information Processing Systems37,pp\. 56898–56918\.Cited by:[Table 6](https://arxiv.org/html/2610.33036#A1.T6.2.1.3.1.1.1)\.
- Chandraet al\.\(2025\)S\. S\. Chandra, L\. Goncalves, J\. Lu, C\. Busso, and B\. SismanEmotionRankCLAP: Bridging Natural Language Speaking Styles and Ordinal Speech Emotion via Rank\-N\-Contrast\.InInterspeech 2025,pp\. 3000–3004\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2025-1198),ISSN 2958\-1796Cited by:[§1](https://arxiv.org/html/2610.33036#S1.p2.1)\.
- Chenet al\.\(2021\)G\. Chen, S\. Chai, G\. Wang, J\. Du, W\. Zhang, C\. Weng, D\. Su, D\. Povey, J\. Trmal, J\. Zhang, M\. Jin, S\. Khudanpur, S\. Watanabe, S\. Zhao, W\. Zou, X\. Li, X\. Yao, Y\. Wang, Z\. You, and Z\. YanGigaSpeech: an evolving, multi\-domain asr corpus with 10,000 hours of transcribed audio\.InInterspeech 2021,pp\. 3670–3674\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2021-1965),ISSN 2958\-1796Cited by:[§4\.5](https://arxiv.org/html/2610.33036#S4.SS5.p1.1)\.
- Chenet al\.\(2022\)S\. Chen, C\. Wang, Z\. Chen, Y\. Wu, S\. Liu, Z\. Chen, J\. Li, N\. Kanda, T\. Yoshioka, X\. Xiao,et al\.Wavlm: large\-scale self\-supervised pre\-training for full stack speech processing\.IEEE Journal of Selected Topics in Signal Processing16\(6\),pp\. 1505–1518\.Cited by:[§2\.3](https://arxiv.org/html/2610.33036#S2.SS3.p1.1)\.
- Chenet al\.\(2023\)S\. Chen, Y\. Wu, C\. Wang, S\. Liu, D\. Tompkins, Z\. Chen, W\. Che, X\. Yu, and F\. WeiBEATs: audio pre\-training with acoustic tokenizers\.InProceedings of the 40th International Conference on Machine Learning,A\. Krause, E\. Brunskill, K\. Cho, B\. Engelhardt, S\. Sabato, and J\. Scarlett \(Eds\.\),Proceedings of Machine Learning Research, Vol\.202,pp\. 5178–5193\.External Links:[Link](https://proceedings.mlr.press/v202/chen23ag.html)Cited by:[§6](https://arxiv.org/html/2610.33036#S6.p2.1)\.
- Chuet al\.\(2024\)Y\. Chu, J\. Xu, Q\. Yang, H\. Wei, X\. Wei, Z\. Guo, Y\. Leng, Y\. Lv, J\. He, J\. Lin, C\. Zhou, and J\. ZhouQwen2\-audio technical report\.External Links:2407\.10759,[Link](https://arxiv.org/abs/2407.10759)Cited by:[§1](https://arxiv.org/html/2610.33036#S1.p6.1)\.
- Chuet al\.\(2023\)Y\. Chu, J\. Xu, X\. Zhou, Q\. Yang, S\. Zhang, Z\. Yan, C\. Zhou, and J\. ZhouQwen\-audio: advancing universal audio understanding via unified large\-scale audio\-language models\.External Links:2311\.07919,[Link](https://arxiv.org/abs/2311.07919)Cited by:[§2\.3](https://arxiv.org/html/2610.33036#S2.SS3.p1.1)\.
- Cuiet al\.\(2025\)W\. Cui, X\. Jiao, Z\. Meng, and I\. KingVoxEval: benchmarking the knowledge understanding capabilities of end\-to\-end spoken language models\.External Links:2501\.04962,[Link](https://arxiv.org/abs/2501.04962)Cited by:[§2\.3](https://arxiv.org/html/2610.33036#S2.SS3.p1.1)\.
- Daset al\.\(2025\)N\. Das, S\. Dingliwal, S\. Ronanki, R\. Paturi, Z\. Huang, P\. Mathur, J\. Yuan, D\. Bekal, X\. Niu, S\. M\. Jayanthi, X\. Li, K\. Mundnich, M\. Sunkara, S\. Bodapati, S\. Srinivasan, K\. J\. Han, and K\. KirchhoffSpeechVerse: a large\-scale generalizable audio language model\.External Links:2405\.08295,[Link](https://arxiv.org/abs/2405.08295)Cited by:[§2\.3](https://arxiv.org/html/2610.33036#S2.SS3.p1.1)\.
- Dehaket al\.\(2011\)N\. Dehak, P\. J\. Kenny, R\. Dehak, P\. Dumouchel, and P\. OuelletFront\-end factor analysis for speaker verification\.IEEE Transactions on Audio, Speech, and Language Processing19\(4\),pp\. 788–798\.External Links:[Document](https://dx.doi.org/10.1109/TASL.2010.2064307)Cited by:[§4\.6](https://arxiv.org/html/2610.33036#S4.SS6.SSS0.Px1.p1.1)\.
- Denget al\.\(2025\)Y\. Deng, G\. Hu, H\. Sun, X\. Zhang, H\. Zhang, F\. Tian, X\. Yang, G\. Yu, and E\. S\. ChngMULTI\-bench: a multi\-turn interactive benchmark for assessing emotional intelligence ability of spoken dialogue models\.External Links:2511\.00850,[Link](https://arxiv.org/abs/2511.00850)Cited by:[Table 6](https://arxiv.org/html/2610.33036#A1.T6.2.1.12.1.1.1)\.
- Dhamyalet al\.\(2024\)H\. Dhamyal, B\. Elizalde, S\. Deshmukh, H\. Wang, B\. Raj, and R\. SinghPrompting audios using acoustic properties for emotion representation\.InICASSP 2024\-2024 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 11936–11940\.Cited by:[§2\.1](https://arxiv.org/html/2610.33036#S2.SS1.p1.1)\.
- Diwanet al\.\(2025\)A\. Diwan, Z\. Zheng, D\. Harwath, and E\. ChoiScaling rich style\-prompted text\-to\-speech datasets\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 3639–3659\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.180/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.180),ISBN 979\-8\-89176\-332\-6Cited by:[Table 5](https://arxiv.org/html/2610.33036#A1.T5.2.12.1),[§1](https://arxiv.org/html/2610.33036#S1.p5.1),[§3\.2](https://arxiv.org/html/2610.33036#S3.SS2.p1.1),[§5](https://arxiv.org/html/2610.33036#S5.p2.2)\.
- Ghoshet al\.\(2026\)S\. Ghosh, A\. Goel, K\. Jayakumar, L\. Koroshinadze, N\. Anand, Z\. Kong, S\. Gururani, S\. Lee, J\. Kim, A\. Aljafari,et al\.Audio flamingo next: next\-generation open audio\-language models for speech, sound, and music\.External Links:2604\.10905,[Link](https://arxiv.org/abs/2604.10905)Cited by:[§1](https://arxiv.org/html/2610.33036#S1.p6.1)\.
- Goelet al\.\(2025\)A\. Goel, S\. Ghosh, J\. Kim, S\. Kumar, Z\. Kong, S\. Lee, C\. H\. Yang, R\. Duraiswami, D\. Manocha, R\. Valle, and B\. CatanzaroAudio flamingo 3: advancing audio intelligence with fully open large audio language models\.External Links:2507\.08128,[Link](https://arxiv.org/abs/2507.08128)Cited by:[§1](https://arxiv.org/html/2610.33036#S1.p6.1)\.
- Gonget al\.\(2026\)Y\. Gong, K\. Chen, Z\. Fei, X\. Yang, K\. Chen, Y\. Wang, K\. Huang, M\. Chen, R\. Li, Q\. Cheng,et al\.Moss\-audio\-tokenizer: scaling audio tokenizers for future audio foundation models\.External Links:2602\.10934,[Link](https://arxiv.org/abs/2602.10934)Cited by:[§1](https://arxiv.org/html/2610.33036#S1.p6.1)\.
- Guanet al\.\(2024\)W\. Guan, Y\. Li, T\. Li, H\. Huang, F\. Wang, J\. Lin, L\. Huang, L\. Li, and Q\. HongMM\-tts: multi\-modal prompt based style transfer for expressive text\-to\-speech synthesis\.Proceedings of the AAAI Conference on Artificial Intelligence38\(16\),pp\. 18117–18125\.External Links:[Link](https://ojs.aaai.org/index.php/AAAI/article/view/29769),[Document](https://dx.doi.org/10.1609/aaai.v38i16.29769)Cited by:[Table 5](https://arxiv.org/html/2610.33036#A1.T5.2.7.1)\.
- Guoet al\.\(2023\)Z\. Guo, Y\. Leng, Y\. Wu, S\. Zhao, and X\. TanPrompttts: controllable text\-to\-speech with text descriptions\.InICASSP 2023 \- 2023 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),Vol\.,pp\. 1–5\.External Links:[Document](https://dx.doi.org/10.1109/ICASSP49357.2023.10096285)Cited by:[Table 5](https://arxiv.org/html/2610.33036#A1.T5.2.3.1),[§1](https://arxiv.org/html/2610.33036#S1.p2.1)\.
- Haiet al\.\(2024\)J\. Hai, K\. Thakkar, H\. Wang, Z\. Qin, and M\. ElhilaliDreamVoice: Text\-Guided Voice Conversion\.InInterspeech 2024,pp\. 4373–4377\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2024-1432),ISSN 2958\-1796Cited by:[Table 5](https://arxiv.org/html/2610.33036#A1.T5.2.10.1),[§1](https://arxiv.org/html/2610.33036#S1.p2.1)\.
- Hansen and Hasan \(2015\)J\. H\. Hansen and T\. HasanSpeaker recognition by machines and humans: a tutorial review\.IEEE Signal processing magazine32\(6\),pp\. 74–99\.Cited by:[§2\.1](https://arxiv.org/html/2610.33036#S2.SS1.p1.1)\.
- Heet al\.\(2024\)H\. He, Z\. Shang, C\. Wang, X\. Li, Y\. Gu, H\. Hua, L\. Liu, C\. Yang, J\. Li, P\. Shi,et al\.Emilia: an extensive, multilingual, and diverse speech dataset for large\-scale speech generation\.In2024 IEEE Spoken Language Technology Workshop \(SLT\),pp\. 885–890\.Cited by:[§3\.2](https://arxiv.org/html/2610.33036#S3.SS2.p1.1)\.
- Huanget al\.\(2025\)C\. Huang, M\. Shih, K\. Lu, C\. Hsiao, and H\. LeeSpeechCaps: advancing instruction\-based universal speech models with multi\-talker speaking style captioning\.InICASSP 2025 \- 2025 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),Vol\.,pp\. 1–5\.External Links:[Document](https://dx.doi.org/10.1109/ICASSP49660.2025.10889205)Cited by:[Table 6](https://arxiv.org/html/2610.33036#A1.T6.2.1.10.1.1.1),[§2\.3](https://arxiv.org/html/2610.33036#S2.SS3.p1.1)\.
- Jiet al\.\(2024\)S\. Ji, J\. Zuo, M\. Fang, Z\. Jiang, F\. Chen, X\. Duan, B\. Huai, and Z\. ZhaoTextrolspeech: a text style control speech corpus with codec language text\-to\-speech models\.InICASSP 2024\-2024 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 10301–10305\.Cited by:[Table 5](https://arxiv.org/html/2610.33036#A1.T5.2.6.1)\.
- Jinet al\.\(2024\)Z\. Jin, J\. Jia, Q\. Wang, K\. Li, S\. Zhou, S\. Zhou, X\. Qin, and Z\. WuSpeechcraft: a fine\-grained expressive speech dataset with natural language description\.InProceedings of the 32nd ACM International Conference on Multimedia,pp\. 1255–1264\.Cited by:[Table 5](https://arxiv.org/html/2610.33036#A1.T5.2.8.1),[§4\.5](https://arxiv.org/html/2610.33036#S4.SS5.p1.1)\.
- Jinget al\.\(2024\)X\. Jing, A\. Triantafyllopoulos, and B\. SchullerParaCLAP – Towards a general language\-audio model for computational paralinguistic tasks\.InInterspeech 2024,pp\. 1155–1159\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2024-1315),ISSN 2958\-1796Cited by:[§1](https://arxiv.org/html/2610.33036#S1.p2.1)\.
- Jinget al\.\(2026\)X\. Jing, A\. Triantafyllopoulos, J\. Wang, S\. Amiriparian, J\. Luo, and B\. SchullerEmoSURA: towards accurate evaluation of detailed and long\-context emotional speech captions\.External Links:2603\.09820,[Link](https://arxiv.org/abs/2603.09820)Cited by:[§4\.7](https://arxiv.org/html/2610.33036#S4.SS7.p2.1)\.
- Kawamuraet al\.\(2024\)M\. Kawamura, R\. Yamamoto, Y\. Shirahata, T\. Hasumi, and K\. TachibanaLibriTTS\-P: A Corpus with Speaking Style and Speaker Identity Prompts for Text\-to\-Speech and Style Captioning\.InInterspeech 2024,pp\. 1850–1854\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2024-692),ISSN 2958\-1796Cited by:[Table 5](https://arxiv.org/html/2610.33036#A1.T5.2.9.1)\.
- Kuanet al\.\(2023\)C\. Kuan, C\. Li, T\. Hsu, T\. Lin, H\. Chung, K\. Chang, S\. Chang, and H\. LeeTowards general\-purpose text\-instruction\-guided voice conversion\.In2023 IEEE Automatic Speech Recognition and Understanding Workshop \(ASRU\),pp\. 1–8\.Cited by:[§1](https://arxiv.org/html/2610.33036#S1.p2.1)\.
- Lin \(2004\)C\. LinRouge: a package for automatic evaluation of summaries\.InText summarization branches out,pp\. 74–81\.Cited by:[§4\.7](https://arxiv.org/html/2610.33036#S4.SS7.p1.1)\.
- Liuet al\.\(2025\)A\. H\. Liu, A\. Ehrenberg, A\. Lo, C\. Denoix, C\. Barreau, G\. Lample, J\. Delignon, K\. R\. Chandu, P\. von Platen, P\. R\. Muddireddy, S\. Gandhi, S\. Ghosh, S\. Mishra, T\. Foubert, A\. Rastogi, A\. Yang, A\. Q\. Jiang, A\. Sablayrolles, A\. Héliou, A\. Martin, A\. Agarwal, A\. Roux, A\. Darcet, A\. Mensch, B\. Bout, B\. Rozière, B\. D\. Monicault, C\. Bamford, C\. Wallenwein, C\. Renaudin, C\. Lanfranchi, D\. Dabert, D\. S\. Chaplot, D\. Mizelle, D\. de las Casas, E\. Chane\-Sane, E\. Fugier, E\. B\. Hanna, G\. Berrada, G\. Delerce, G\. Guinet, G\. Novikov, G\. Martin, H\. Jaju, J\. Ludziejewski, J\. Rute, J\. Chabran, J\. Chudnovsky, J\. Studnia, J\. Barmentlo, J\. Amar, J\. S\. Roberts, J\. Denize, K\. Saxena, K\. Yadav, K\. Khandelwal, K\. Jain, L\. R\. Lavaud, L\. Blier, L\. Zhao, L\. Martin, L\. Saulnier, L\. Gao, M\. Pellat, M\. Guillaumin, M\. Felardos, M\. Dinot, M\. Darrin, M\. Augustin, M\. Seznec, N\. Gupta, N\. Raghuraman, O\. Duchenne, P\. Wang, P\. Saffer, P\. Jacob, P\. Wambergue, P\. Kurylowicz, P\. Chagniot, P\. Stock, P\. Agrawal, R\. Delacourt, R\. Sauvestre, R\. Soletskyi, S\. Vaze, S\. Subramanian, S\. Garg, S\. Dalal, S\. Gandhi, S\. Aithal, S\. Antoniak, T\. L\. Scao, T\. Schueller, T\. Lavril, T\. Robert, T\. Wang, T\. Lacroix, T\. Bewley, V\. Nemychnikova, V\. Paltz, V\. Richard, W\. Li, W\. Marshall, X\. Zhang, Y\. Wan, and Y\. TangVoxtral\.External Links:2507\.13264,[Link](https://arxiv.org/abs/2507.13264)Cited by:[§1](https://arxiv.org/html/2610.33036#S1.p6.1)\.
- LlamaTeam \(2024\)A\. LlamaTeamThe llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§3\.2](https://arxiv.org/html/2610.33036#S3.SS2.p3.1)\.
- Luet al\.\(2024\)K\. Lu, Z\. Chen, S\. Fu, H\. Huang, B\. Ginsburg, Y\. F\. Wang, and H\. LeeDeSTA: Enhancing Speech Language Models through Descriptive Speech\-Text Alignment\.InInterspeech 2024,pp\. 4159–4163\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2024-457),ISSN 2958\-1796Cited by:[§2\.1](https://arxiv.org/html/2610.33036#S2.SS1.p1.1)\.
- Luet al\.\(2026\)K\. Lu, Z\. Chen, S\. Fu, C\. H\. Yang, S\. Huang, C\. Yang, C\. Yu, C\. Chen, W\. Chen, C\. Huang, Y\. Lin, Y\. Lin, C\. Fu, C\. Kuan, W\. Ren, X\. Chen, W\. Huang, E\. Hu, T\. Lin, Y\. Wu, K\. Huang, H\. Huang, H\. Chou, K\. Chang, C\. Chiang, B\. Ginsburg, Y\. F\. Wang, and H\. LeeDeSTA2\.5\-audio: toward general\-purpose large audio language model with self\-generated cross\-modal alignment\.IEEE Transactions on Audio, Speech and Language Processing34\(\),pp\. 2062–2076\.External Links:[Document](https://dx.doi.org/10.1109/TASLPRO.2026.3675792)Cited by:[§2\.1](https://arxiv.org/html/2610.33036#S2.SS1.p1.1),[§2\.3](https://arxiv.org/html/2610.33036#S2.SS3.p1.1)\.
- Lyth and King \(2024\)D\. Lyth and S\. KingNatural language guidance of high\-fidelity text\-to\-speech with synthetic annotations\.External Links:2402\.01912,[Link](https://arxiv.org/abs/2402.01912)Cited by:[Table 5](https://arxiv.org/html/2610.33036#A1.T5.2.2.1)\.
- Maet al\.\(2025\)Z\. Ma, Y\. Ma, Y\. Zhu, C\. Yang, Y\. Chao, R\. Xu, W\. Chen, Y\. Chen, Z\. Chen, J\. Cong, K\. Li, K\. Li, S\. Li, X\. Li, X\. Li, Z\. Lian, Y\. Liang, M\. Liu, Z\. Niu, T\. Wang, W\. Yuping, Y\. Wang, Y\. Wu, G\. Yang, J\. Yu, R\. Yuan, Z\. Zheng, Z\. Zhou, H\. Zhu, W\. Xue, E\. Benetos, K\. Yu, E\. Chng, and X\. ChenMMAR: a challenging benchmark for deep reasoning in speech, audio, music, and their mix\.InAdvances in Neural Information Processing Systems,D\. Belgrave, C\. Zhang, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, and N\. Chen \(Eds\.\),Vol\.38, Main Conference,pp\.\.External Links:[Document](https://dx.doi.org/10.52202/085713-2252),[Link](https://proceedings.neurips.cc/paper_files/paper/2025/file/610a7d6507d55be70c6d13d0b663227d-Paper-Datasets_and_Benchmarks_Track.pdf)Cited by:[§6](https://arxiv.org/html/2610.33036#S6.p2.1)\.
- Maaten and Hinton \(2008\)L\. v\. d\. Maaten and G\. HintonVisualizing data using t\-sne\.Journal of machine learning research9\(Nov\),pp\. 2579–2605\.Cited by:[§6](https://arxiv.org/html/2610.33036#S6.p5.1)\.
- Maimonet al\.\(2025\)G\. Maimon, A\. Roth, and Y\. AdiSalmon: a suite for acoustic language model evaluation\.InICASSP 2025\-2025 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 1–5\.Cited by:[Table 6](https://arxiv.org/html/2610.33036#A1.T6.2.1.13.1.1.1)\.
- MERaLiON Team \(2025\)MERaLiON TeamMERaLiON\-audiollm: bridging audio and language with large language models\.External Links:2412\.09818,[Link](https://arxiv.org/abs/2412.09818)Cited by:[§1](https://arxiv.org/html/2610.33036#S1.p6.1)\.
- Mistral AI Team \(2025\)Mistral AI TeamMistral small 3\.External Links:[Link](https://mistral.ai/news/mistral-small-3/)Cited by:[§3\.2](https://arxiv.org/html/2610.33036#S3.SS2.p3.1)\.
- Mondshineet al\.\(2025\)I\. Mondshine, T\. Paz\-Argaman, and R\. TsarfatyBeyond n\-grams: rethinking evaluation metrics and strategies for multilingual abstractive summarization\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 19019–19035\.Cited by:[§4\.7](https://arxiv.org/html/2610.33036#S4.SS7.p1.1)\.
- Nagraniet al\.\(2017\)A\. Nagrani, J\. S\. Chung, and A\. ZissermanVoxCeleb: a large\-scale speaker identification dataset\.InInterspeech 2017,pp\. 2616–2620\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2017-950),ISSN 2958\-1796Cited by:[§3\.2](https://arxiv.org/html/2610.33036#S3.SS2.p1.1)\.
- Nguyenet al\.\(2023\)T\. A\. Nguyen, W\. Hsu, A\. D’Avirro, B\. Shi, I\. Gat, M\. Fazel\-Zarani, T\. Remez, J\. Copet, G\. Synnaeve, M\. Hassid, F\. Kreuk, Y\. Adi, and E\. DupouxExpresso: a benchmark and analysis of discrete expressive speech resynthesis\.InInterspeech 2023,pp\. 4823–4827\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2023-1905),ISSN 2958\-1796Cited by:[Table 5](https://arxiv.org/html/2610.33036#A1.T5.2.4.1),[§3\.2](https://arxiv.org/html/2610.33036#S3.SS2.p1.1)\.
- Ok and Lee \(2025\)H\. Ok and J\. LeeS2Cap: a benchmark and a baseline for singing style captioning\.InProceedings of the 34th ACM International Conference on Information and Knowledge Management,pp\. 6492–6497\.Cited by:[Table 6](https://arxiv.org/html/2610.33036#A1.T6.2.1.9.1.1.1)\.
- OpenAI \(2025\)OpenAIGpt\-oss\-120b & gpt\-oss\-20b model card\.External Links:2508\.10925,[Link](https://arxiv.org/abs/2508.10925)Cited by:[§5](https://arxiv.org/html/2610.33036#S5.p1.1)\.
- Penget al\.\(2026\)J\. Peng, Y\. Wang, B\. Li, Y\. Guo, H\. Wang, Y\. Fang, Y\. Xi, H\. Li, X\. Li, K\. Zhang, S\. Wang, and K\. YuA survey on speech large language models for understanding\.IEEE Journal of Selected Topics in Signal Processing20\(1\),pp\. 2–31\.External Links:[Document](https://dx.doi.org/10.1109/JSTSP.2025.3640535)Cited by:[§2\.1](https://arxiv.org/html/2610.33036#S2.SS1.p1.1)\.
- Radfordet al\.\(2023\)A\. Radford, J\. W\. Kim, T\. Xu, G\. Brockman, C\. McLeavey, and I\. SutskeverRobust speech recognition via large\-scale weak supervision\.InInternational conference on machine learning,pp\. 28492–28518\.Cited by:[§6](https://arxiv.org/html/2610.33036#S6.p2.1)\.
- Retkowskiet al\.\(2025\)F\. Retkowski, M\. Züfle, A\. Sudmann, D\. Pfau, S\. Watanabe, J\. Niehues, and A\. WaibelSummarizing speech: a comprehensive survey\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 27263–27294\.Cited by:[§2\.1](https://arxiv.org/html/2610.33036#S2.SS1.p1.1)\.
- Richteret al\.\(2024\)J\. Richter, Y\. Wu, S\. Krenn, S\. Welker, B\. Lay, S\. Watanabe, A\. Richard, and T\. GerkmannEARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberation\.InInterspeech 2024,pp\. 4873–4877\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2024-153),ISSN 2958\-1796Cited by:[Table 5](https://arxiv.org/html/2610.33036#A1.T5.2.5.1),[§3\.2](https://arxiv.org/html/2610.33036#S3.SS2.p1.1)\.
- Sakshiet al\.\(2025\)S\. Sakshi, U\. Tyagi, S\. Kumar, A\. Seth, R\. Selvakumar, O\. Nieto, R\. Duraiswami, S\. Ghosh, and D\. ManochaMMAU: a massive multi\-task audio understanding and reasoning benchmark\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=TeVAZXr3yv)Cited by:[§6](https://arxiv.org/html/2610.33036#S6.p2.1)\.
- Schuhmannet al\.\(2026\)C\. Schuhmann, R\. Kaczmarczyk, G\. Rabby, F\. Friedrich, M\. Kraus, K\. Nadi, H\. Nguyen, K\. Kersting, and S\. AuerEmoNet\-voice: a fine\-grained, expert\-verified benchmark for speech emotion detection\.External Links:2506\.09827,[Link](https://arxiv.org/abs/2506.09827)Cited by:[Table 6](https://arxiv.org/html/2610.33036#A1.T6.2.1.7.1.1.1)\.
- Shanget al\.\(2024\)H\. Shang, Z\. Li, J\. Guo, S\. Li, Z\. Rao, Y\. Luo, D\. Wei, and H\. YangAn End\-to\-End Speech Summarization Using Large Language Model\.InInterspeech 2024,pp\. 1950–1954\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2024-1428),ISSN 2958\-1796Cited by:[§2\.1](https://arxiv.org/html/2610.33036#S2.SS1.p1.1)\.
- StepFun \(2025\)StepFunStep\-audio 2 technical report\.External Links:2507\.16632,[Link](https://arxiv.org/abs/2507.16632)Cited by:[Table 6](https://arxiv.org/html/2610.33036#A1.T6.2.1.8.1.1.1)\.
- Suet al\.\(2026\)B\. Su, H\. Shih, J\. Tian, J\. Shi, C\. Lee, C\. Busso, and S\. WatanabeReasoning beyond majority vote: an explainable speechlm framework for speech emotion recognition\.InICASSP 2026 \- 2026 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),Vol\.,pp\. 22532–22536\.External Links:[Document](https://dx.doi.org/10.1109/ICASSP55912.2026.11464563)Cited by:[§2\.1](https://arxiv.org/html/2610.33036#S2.SS1.p1.1)\.
- Tanget al\.\(2024\)C\. Tang, W\. Yu, G\. Sun, X\. Chen, T\. Tan, W\. Li, L\. Lu, Z\. Ma, and C\. ZhangSALMONN: towards generic hearing abilities for large language models\.External Links:2310\.13289,[Link](https://arxiv.org/abs/2310.13289)Cited by:[§1](https://arxiv.org/html/2610.33036#S1.p6.1),[§2\.3](https://arxiv.org/html/2610.33036#S2.SS3.p1.1)\.
- Teamet al\.\(2026\)G\. Team, S\. E\. Abd, V\. Aggarwal, R\. Algayres, A\. Andreev, O\. Bachem, I\. Ballantyne, C\. Brick, V\. Cărbune, M\. Casbon,et al\.Gemma 4 technical report\.External Links:2607\.02770,[Link](https://arxiv.org/abs/2607.02770)Cited by:[§5](https://arxiv.org/html/2610.33036#S5.p1.1)\.
- Teamet al\.\(2024\)G\. Team, M\. Riviere, S\. Pathak, P\. G\. Sessa, C\. Hardin, S\. Bhupatiraju, L\. Hussenot, T\. Mesnard, B\. Shahriari, A\. Ramé,et al\.Gemma 2: improving open language models at a practical size\.External Links:2408\.00118,[Link](https://arxiv.org/abs/2408.00118)Cited by:[Appendix D](https://arxiv.org/html/2610.33036#A4.p1.1)\.
- Triantafyllopouloset al\.\(2026\)A\. Triantafyllopoulos, A\. Batliner, and B\. W\. SchullerCharting 15 years of progress in deep learning for speech emotion recognition: a replication study\.IEEE Transactions on Affective Computing17\(3\),pp\. 3185–3199\.External Links:[Document](https://dx.doi.org/10.1109/TAFFC.2026.3680663)Cited by:[§2\.1](https://arxiv.org/html/2610.33036#S2.SS1.p1.1)\.
- Tuttösíet al\.\(2026\)P\. Tuttösí, M\. Dhillon, L\. Sang, S\. Eastwood, P\. Bhatia, Q\. M\. Dinh, A\. Kapoor, Y\. Jin, and A\. LimBERSting at the screams: a benchmark for distanced, emotional and shouted speech recognition\.Computer Speech & Language95,pp\. 101815\.Cited by:[Table 6](https://arxiv.org/html/2610.33036#A1.T6.2.1.6.1.1.1)\.
- Vedantamet al\.\(2015\)R\. Vedantam, C\. Lawrence Zitnick, and D\. ParikhCider: consensus\-based image description evaluation\.InProceedings of the IEEE conference on computer vision and pattern recognition,pp\. 4566–4575\.Cited by:[§4\.7](https://arxiv.org/html/2610.33036#S4.SS7.p1.1)\.
- Verdiniet al\.\(2025\)F\. Verdini, P\. Melucci, S\. Perna, F\. Cariaggi, M\. Gaido, S\. Papi, S\. Mazurek, M\. Kasztelnik, L\. Bentivogli, S\. Bratières, P\. Merialdo, and S\. ScardapaneHow to Connect Speech Foundation Models and Large Language Models? What Matters and What Does Not\.InInterspeech 2025,pp\. 1813–1817\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2025-2245),ISSN 2958\-1796Cited by:[§2\.3](https://arxiv.org/html/2610.33036#S2.SS3.p1.1)\.
- Wanget al\.\(2025a\)H\. Wang, J\. Hai, D\. Chong, K\. Thakkar, T\. Feng, D\. Yang, J\. Lee, T\. Thebaud, L\. M\. Velazquez, J\. Villalba, Z\. Qin, S\. Narayanan, M\. Elhiali, and N\. DehakCapSpeech: enabling downstream applications in style\-captioned text\-to\-speech\.External Links:2506\.02863,[Link](https://arxiv.org/abs/2506.02863)Cited by:[Table 5](https://arxiv.org/html/2610.33036#A1.T5.2.11.1),[§1](https://arxiv.org/html/2610.33036#S1.p2.1),[§4\.6](https://arxiv.org/html/2610.33036#S4.SS6.p1.1)\.
- Wanget al\.\(2024\)L\. Wang, N\. Yang, X\. Huang, B\. Jiao, L\. Yang, D\. Jiang, R\. Majumder, and F\. WeiText embeddings by weakly\-supervised contrastive pre\-training\.External Links:2212\.03533,[Link](https://arxiv.org/abs/2212.03533)Cited by:[§4\.6](https://arxiv.org/html/2610.33036#S4.SS6.SSS0.Px1.p1.1)\.
- Wanget al\.\(2025b\)Q\. Wang, H\. B\. Sailor, T\. Liu, W\. Zhang, M\. Huzaifah, N\. Lertcheva, S\. Sun, N\. F\. Chen, J\. Wu, and A\. AwBenchmarking contextual and paralinguistic reasoning in speech\-LLMs: a case study with in\-the\-wild data\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 14133–14148\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.760/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.760),ISBN 979\-8\-89176\-335\-7Cited by:[Table 6](https://arxiv.org/html/2610.33036#A1.T6.2.1.2.1.1.1)\.
- Xuet al\.\(2025\)J\. Xu, Z\. Guo, H\. Hu, Y\. Chu, X\. Wang, J\. He, Y\. Wang, X\. Shi, T\. He, X\. Zhu,et al\.Qwen3\-omni technical report\.External Links:2509\.17765,[Link](https://arxiv.org/abs/2509.17765)Cited by:[§1](https://arxiv.org/html/2610.33036#S1.p6.1)\.
- Xuet al\.\(2024a\)Y\. Xu, H\. Chen, J\. Yu, Q\. Huang, Z\. Wu, S\. Zhang, G\. Li, Y\. Luo, and R\. GuSECap: speech emotion captioning with large language model\.Proceedings of the AAAI Conference on Artificial Intelligence38\(17\),pp\. 19323–19331\.External Links:[Link](https://ojs.aaai.org/index.php/AAAI/article/view/29902),[Document](https://dx.doi.org/10.1609/aaai.v38i17.29902)Cited by:[§2\.3](https://arxiv.org/html/2610.33036#S2.SS3.p1.1)\.
- Xuet al\.\(2024b\)Y\. Xu, Y\. Zhou, Y\. Cai, J\. Xie, R\. Ye, and Z\. WuMultimodal emotion captioning using large language model with prompt engineering\.InProceedings of the 2nd International Workshop on Multimodal and Responsible Affective Computing,pp\. 104–109\.Cited by:[§2\.3](https://arxiv.org/html/2610.33036#S2.SS3.p1.1)\.
- Yamagishiet al\.\(2019\)J\. Yamagishi, C\. Veaux, and K\. MacDonaldCSTR vctk corpus: english multi\-speaker corpus for cstr voice cloning toolkit \(version 0\.92\)\.The Rainbow Passage which the speakers read out can be found in the International Dialects of English Archive:\(http://web\. ku\. edu/˜ idea/readings/rainbow\. htm\)\.\.Cited by:[§4\.4](https://arxiv.org/html/2610.33036#S4.SS4.p1.1)\.
- Yamauchiet al\.\(2024\)K\. Yamauchi, Y\. Ijima, and Y\. SaitoSTYLECAP: automatic speaking\-style captioning from speech based on speech and language self\-supervised learning models\.InICASSP 2024 \- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),Vol\.,pp\. 11261–11265\.External Links:[Document](https://dx.doi.org/10.1109/ICASSP48485.2024.10445977)Cited by:[§1](https://arxiv.org/html/2610.33036#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.33036#S2.SS1.p1.1),[§2\.3](https://arxiv.org/html/2610.33036#S2.SS3.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§5](https://arxiv.org/html/2610.33036#S5.p1.1)\.
- Yanget al\.\(2026\)S\. Yang, M\. Tu, T\. Liu, X\. Qu, H\. Lee, L\. Lu, Y\. Wang, and Y\. WuParaS2S: benchmarking and aligning spoken language models for paralinguistic\-aware speech\-to\-speech interaction\.InInternational Conference on Learning Representations,C\. Vondrick, B\. Hariharan, C\. Raffel, L\. Pinto, D\. Yang, and A\. Faust \(Eds\.\),Vol\.2026,pp\. 70704–70729\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2026/file/734b2e77222680728b9ce78c573eae1e-Paper-Conference.pdf)Cited by:[Table 6](https://arxiv.org/html/2610.33036#A1.T6.2.1.4.1.1.1)\.
- Zhanget al\.\(2025a\)L\. Zhang, J\. Zhang, B\. Lei, C\. Wu, A\. Liu, W\. Jia, and X\. ZhouWildSpeech\-bench: benchmarking end\-to\-end speechllms in the wild\.External Links:2506\.21875,[Link](https://arxiv.org/abs/2506.21875)Cited by:[Table 6](https://arxiv.org/html/2610.33036#A1.T6.2.1.11.1.1.1)\.
- Zhanget al\.\(2026\)Z\. Zhang, S\. S\. Chandra, Z\. Du, X\. Zhao, A\. Mahapatra, H\. Zhang, P\. Koehn, and B\. SismanTRACE\-evc: text\-guided relative affective control for zero\-shot emotional voice conversion\.External Links:2607\.03666,[Link](https://arxiv.org/abs/2607.03666)Cited by:[§2\.1](https://arxiv.org/html/2610.33036#S2.SS1.p1.1)\.
- Zhanget al\.\(2025b\)Z\. Zhang, Y\. Wu, Z\. Dong, W\. Xiang, S\. Shen, and B\. W\. SchullerSSE: a speaking style extractor based on fine\-grained contrastive learning between speech and descriptive text\.InICASSP 2025 \- 2025 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),Vol\.,pp\. 1–5\.External Links:[Document](https://dx.doi.org/10.1109/ICASSP49660.2025.10888883)Cited by:[§1](https://arxiv.org/html/2610.33036#S1.p2.1)\.
- Zhanget al\.\(2024\)Z\. Zhang, W\. Xu, Z\. Dong, K\. Wang, Y\. Wu, J\. Peng, R\. Wang, and D\. HuangParaLBench: a large\-scale benchmark for computational paralinguistics over acoustic foundation models\.IEEE Transactions on Affective Computing\.Cited by:[Table 6](https://arxiv.org/html/2610.33036#A1.T6.2.1.5.1.1.1)\.
- Zhang\*et al\.\(2020\)T\. Zhang\*, V\. Kishore\*, F\. Wu\*, K\. Q\. Weinberger, and Y\. ArtziBERTScore: evaluating text generation with bert\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=SkeHuCVFDr)Cited by:[§4\.7](https://arxiv.org/html/2610.33036#S4.SS7.p1.1)\.
- Zhaoet al\.\(2026\)X\. Zhao, I\. R\. Ulgen, P\. Koehn, B\. Schuller, and B\. SismanNeuron\-level emotion control in speech\-generative large audio\-language models\.External Links:2603\.17231,[Link](https://arxiv.org/abs/2603.17231)Cited by:[§2\.1](https://arxiv.org/html/2610.33036#S2.SS1.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.InAdvances in neural information processing systems,Vol\.36,pp\. 46595–46623\.Cited by:[§4\.7](https://arxiv.org/html/2610.33036#S4.SS7.p1.1)\.
- Zhuet al\.\(2024\)X\. Zhu, W\. Tian, X\. Wang, L\. He, Y\. Xiao, X\. Wang, X\. Tan, S\. Zhao, and L\. XieUniStyle: unified style modeling for speaking style captioning and stylistic speech synthesis\.InProceedings of the 32nd ACM International Conference on Multimedia,MM ’24,New York, NY, USA,pp\. 7513–7522\.External Links:ISBN 9798400706868,[Link](https://doi.org/10.1145/3664647.3681465),[Document](https://dx.doi.org/10.1145/3664647.3681465)Cited by:[§2\.3](https://arxiv.org/html/2610.33036#S2.SS3.p1.1)\.

## Appendix AExisting Datasets and Benchmarks

NameSize \(hrs\)ParlerTTS[Lyth and King \(2024\)](https://arxiv.org/html/2610.33036#bib.bib13)45kPromptSpeech[Guo et al\. \(2023\)](https://arxiv.org/html/2610.33036#bib.bib6)37 \(real\)Expresso[Nguyen et al\. \(2023\)](https://arxiv.org/html/2610.33036#bib.bib23)47EARS[Richter et al\. \(2024\)](https://arxiv.org/html/2610.33036#bib.bib24)60TextrolSpeech[Ji et al\. \(2024\)](https://arxiv.org/html/2610.33036#bib.bib25)300MEAD\-TTS[Guan et al\. \(2024\)](https://arxiv.org/html/2610.33036#bib.bib26)36SpeechCraft[Jin et al\. \(2024\)](https://arxiv.org/html/2610.33036#bib.bib27)2\.4kLibriTTS\-P[Kawamura et al\. \(2024\)](https://arxiv.org/html/2610.33036#bib.bib28)600Dreamvoice[Hai et al\. \(2024\)](https://arxiv.org/html/2610.33036#bib.bib8)231Capspeech[Wang et al\. \(2025a\)](https://arxiv.org/html/2610.33036#bib.bib5)33\.6kParaSpeechCaps[Diwan et al\. \(2025\)](https://arxiv.org/html/2610.33036#bib.bib11)2\.7kTable 5:Summary of existing datasets relevant to SSC\.BenchmarkWhat Task It BenchmarksEvaluation MetricsCP\-Bench[Wang et al\. \(2025b\)](https://arxiv.org/html/2610.33036#bib.bib36)Contextual Paralinguistic Reasoning: Evaluates the model’s ability to reason over both verbal content and non\-verbal cues like emotion and prosody to answer questionsAnswer accuracy, human preference ratingsSD\-Eval[Ao et al\. \(2024\)](https://arxiv.org/html/2610.33036#bib.bib37)Spoken Dialogue Understanding: assesses understanding of dialogue that relies on paralinguistic cues \(e\.g\., sarcasm, tone\) for correct interpretationsTask accuracy, BLEU \(for generation\), MOS\-style human ratings\.ParaS2SBench[Yang et al\. \(2026\)](https://arxiv.org/html/2610.33036#bib.bib38)Paralinguistic\-Aware Speech\-to\-Speech: Evaluates if generated speech responses maintain appropriate paralinguistic alignment \(style/emotion\) with the inputContent similarity, style similarity, MOS \(Mean Opinion Score\)ParaLBench[Zhang et al\. \(2024\)](https://arxiv.org/html/2610.33036#bib.bib39)General Computational Paralinguistics: A comprehensive test suite across multiple existing paralinguistic datasets to test generalizabilityAccuracy, UAR \(Unweighted Average Recall\), F1 score, correlation metricsBERSt[Tuttösí et al\. \(2026\)](https://arxiv.org/html/2610.33036#bib.bib42)Expressive Emotion Recognition: Specifically benchmarks emotion recognition in challenging conditions like shouted or highly expressive speechSER \(Speech Emotion Recognition\) accuracy, UAR, WER \(Word Error Rate \- as a stress test\)EmoNet\-Voice\-Bench[Schuhmann et al\. \(2026\)](https://arxiv.org/html/2610.33036#bib.bib40)Fine\-grained Emotion Understanding: Tests understanding of nuanced emotions beyond basic categories \(e\.g\., distinguishing "cold anger" from "hot anger"\)Classification accuracy, macro\-F1, confusion matrix analysisStepEval\-Audio\-Paralinguistic[StepFun \(2025\)](https://arxiv.org/html/2610.33036#bib.bib41)Multidimensional paralinguistic comprehension: Evaluates 11 distinct dimensions including gender, age, pitch, rhythm, and speaking rate\.Dimension\-wise accuracy, human ratingsS2Cap[Ok and Lee \(2025\)](https://arxiv.org/html/2610.33036#bib.bib43)Singing Style Description: Focuses on describing singing voices using natural language, capturing attributes like mood, tempo, and timbreClassification accuracy on style factors, Captioning metrics: BLEU, METEOR, CIDErSpeechCaps[Huang et al\. \(2025\)](https://arxiv.org/html/2610.33036#bib.bib18)Speaking\-Style Captioning in multi\-talker scenarios: Benchmarks the ability to generate captions that describe speaker\-specific styles and prosody in multi\-talker scenarios\.Instruction\-following rate, overall accuracy, and conditional accuracy, with correctness and relevance judged via LLM\-based semantic alignment \(GPT\-4o\)WildSpeech\-Bench[Zhang et al\. \(2025a\)](https://arxiv.org/html/2610.33036#bib.bib44)Natural / In\-the\-Wild Prosody: Benchmarks how models handle real\-world paralinguistic variability, including stuttering, hesitations, interruptions, and extreme prosodic conditions \(e\.g\., whispering vs\. shouting\)\.UGPT\-4o\-based 1\-10 scoring, human\-alignment metrics \(Pearson/R²/MSE\), and UTMOS\.Multi\-Bench[Deng et al\. \(2025\)](https://arxiv.org/html/2610.33036#bib.bib45)Multi\-turn Emotional Dialogue: Tests whether models can maintain emotional consistency and empathy over a long conversation, unlike single\-turn benchmarks\.Consistency score, empathy rating \(Likert scale by human or LLM judge\)Salmon[Maimon et al\. \(2025\)](https://arxiv.org/html/2610.33036#bib.bib46)Acoustic consistency and semantic\-acoustic alignment across attributes such as sentiment, speaker identity, background noise, and room acoustics using likelihood\-based comparisons\.Uses pairwise likelihood preference scores \(accuracy\) for acoustic consistency and semantic\-acoustic alignment tasks, with sWUGGY for spoken\-content modeling and human agreement as an upper bound\.

Table 6:Benchmarks for Speech Paralinguistics Understanding
## Appendix BAnnotation reliability of PSC style metadata

The quality of PSC annotations is supported by multiple stages of annotation verification and filtering throughout the dataset construction pipeline\. For PSC\-Base, annotations were collected from qualified Amazon Mechanical Turk annotators who completed a dedicated qualification task designed around identifying speaking\-style attributes, and annotations were aggregated across five annotators per speaker with agreement\-based filtering applied before retaining tags\. For PSC\-Scaled, automatic annotation combines complementary components including perceptual speaker similarity, expressivity filtering, semantic matching, and acoustic matching using an audio LLM\-as\-a\-judge, rather than relying on a single classifier or heuristic\. Importantly, the paper further verifies scaled data quality through human evaluation, showing that PSC\-Scaled achieves tag recall comparable to PSC\-Base for both intrinsic and situational tags, supporting that the automatically scaled annotations remain aligned with human perceptual judgments\.

## Appendix CStyleInstructCaps\-DB Construction details

\{ "source": "ears", "relative\_audio\_path": "p047/emo\_amazement\_freeform\.wav", "text\_description": \[ " A female speaker’s authoritative, nasal voice is crisp and high\-pitched, conveyed in an American accent with a measured speed\. Her speech is delivered in a slightly clean environment, yet she expresses awe with a loud and slightly elevated volume\.", " An American woman speaks with a high\-pitched voice, expressing awe, in a slightly clean environment\. Her speech is measured and deliberate\." \], "transcription": " Oh, really? You were so good at that\. I love to just sit here and listen to you play\. It’s so relaxing\. I just can’t believe how talented you are\.", "intrinsic\_tags": \[ "authoritative", "crisp", "loud", "american", "nasal" \], "situational\_tags": \[ "awed" \], "basic\_tags": \[ "female", "high\-pitched", "measured speed", "slightly clean environment" \], "all\_tags": \[ "american", "authoritative", "awed", "crisp", "female", "high\-pitched", "loud", "measured speed", "nasal", "slightly clean environment" \], "speakerid": "p047", "name": "p047", "duration": 9\.8, "gender": "female", "accent": "american", "pitch": "high\-pitched", "speaking\_rate": "measured speed", "noise": "slightly clean environment", "utterance\_pitch\_mean": 220\.68133544921875, "snr": 56\.96498489379883, "phonemes": " o, li? ju w so d æt ðæt\. a lv tu dst st hi nd lsn tu ju ple\. t’s so læksŋ\. a dst kæn’ti bliv ha tælntd ju \.", "audio\_path": "datasets/ears/p047/emo\_amazement\_freeform\.wav", \}

Figure 4:Example metadata object from EARS used in our captioning/evaluation pipeline\.
\[ROLE & GOAL\]
You are an AI expert in speech prosody\. Your primary task is to generate six highly descriptive, fine\-grained speech captions from a given audio metadata object\.
\[CONTEXT: INPUT METADATA\]
Analyze the following JSON object, which contains detailed metadata for a speech utterance\.
JSON
\{metadata\}
\[TASK & INSTRUCTIONS\]
Based on the metadata, generate six unique and descriptive captions\. For each caption, combine multiple vocal characteristics using descriptive verbs and adverbs to precisely capture the essence of the speaking\-style\.
Use these general guidelines for each caption:
1\. Think in Verbs and Adverbs: Instead of "sad," think "a voice that wavers and falls off slowly\."
2\. Combine at Least Three Elements: A good caption should ideally define a pitch contour, a pace, and a vocal quality\.
3\. Specify the Context: Adding a phrase like "as if explaining a complex topic" or "like a conspiratorial whisper" can give the model a powerful contextual anchor\.
4\. Use the Tags: Leverage the intrinsic\_tags, situational\_tags, basic\_tags, and all\_tags to inform your captions\.
5\. Always associate the nationality with the accent
6\. Use the provided transcription to infer emotional and contextual elements but do not directly quote it in the captions\.
Each caption should focus on a different aspect of the speech, as outlined below\.
1\. Speaker\-Idiosyncratic Style:
Goal: Capture the speaker’s core vocal identity\.
Instruction: Generate a caption describing the speaker’s inherent qualities\. Combine the pitch , accent, gender , and timbre\-related intrinsic\_tags\.
2\. Situational/Contextual Style:
Goal: Describe the circumstances of the speech\.
Instruction: Generate a caption that sets the scene\. Combine the speaking\_rate \(measured speed\), the noise \(noisy environment\), and contextual intrinsic\_tags \(authoritative\)\. Frame it as a person speaking with purpose in a specific setting\.
3\. Expressive/Emotional Style:
Goal: Convey the inferred emotion or attitude\.
Instruction: Generate a caption describing the emotional tone\. Infer the attitude from the transcription, situational\_tags and all\_tags\. Describe the prosody \(pitch contour, rhythm\) that would create this feeling\.
4\. Linguistic/Pragmatic Style:
Goal: Detail the speech’s structural and emphasis patterns\.
Instruction: Generate a caption focused on the delivery of the words themselves\. Analyze the transcription to identify linguistic features\. Describe the pausing and stress patterns\.
5\. Perceptual/Listener\-Centric Style:
Goal: Describe the impression the voice leaves on a listener\.
Instruction: Generate a caption from the listener’s point of view\. Use impressionistic intrinsic\_tags to describe the overall effect of the voice\. How does it feel to hear this person speak?
6\. Overall style summary:
Goal: Create a rich, narrative\-style caption\.
Instruction: Combine all elements in instrinsic\-tags, situational\_tags, basic\_tags, all\_tags, gender, accent, pitch, speaking\_rate, noise into a creative, narrative\-style caption that captures the essence of the speaking\-style\.
\[OUTPUT FORMAT & CONSTRAINTS\]
Return your response as a single, valid JSON object\. Use the descriptive keys provided in the example below\. The entire output must be only the JSON object, with no additional text, explanations, or markdown formatting\.
JSON
\{
"speaker\_idiosyncratic\_style": "<Generated caption for style 1\>",
"situational\_contextual\_style": "<Generated caption for style 2\>",
"expressive\_emotional\_style": "<Generated caption for style 3\>",
"linguistic\_pragmatic\_style": "<Generated caption for style 4\>",
"perceptual\_listener\_centric\_style": "<Generated caption for style 5\>",
"overall\_style\_summary": "<Generated caption for style 6\>"
\}
Figure 5:Prompt template for generating six style\-specific captions from an input metadata object\.Instruction typeStyle instructionGenerated captionSpeaker idiosyncratic styleDescribe the speaker’s inherent vocal identity by combining their pitch, accent, gender, and timbre\-related characteristics\.A high\-pitched, crisp, and nasal voice with an American accent, delivered by a female speaker, instantly conveys a sense of authority and confidence\.Situational contextual styleSet the scene by combining the speaking rate, noise level, and contextual intrinsic tags to describe a person speaking with intention in a specific setting\.Speaking with measured speed and authority, a female voice with an American accent cuts through a slightly clean environment, commanding attention as if addressing a gathering with purpose and conviction\.Expressive emotional styleInfer the emotional tone from the transcription and situational tags, and describe the prosody that creates this feeling, capturing the speaker’s attitude and sentiment\.A voice filled with awe and admiration, conveyed through a rising pitch contour and deliberate pacing, expresses genuine astonishment, as if the speaker is witnessing something remarkable for the first time\.Linguistic pragmatic styleAnalyze the transcription to identify linguistic features, and describe the pausing and stress patterns that reveal the speaker’s emphasis and intention\.The speech unfolds with a steady, measured rhythm, punctuated by deliberate pauses that emphasize key phrases, as the speaker carefully chooses words to convey their thoughts, creating a sense of sincerity and authenticity\.Perceptual listener\-centric styleDescribe the overall effect of the voice on the listener, using impressionistic intrinsic tags to capture the impression the voice leaves on those who hear it\.Listening to this voice is like being drawn into a conversation with a charismatic and confident individual, as the speaker’s authoritative tone and crisp delivery create a sense of trust and engagement\.Holistic creative synthesisCombine all elements to create a rich, narrative\-style caption that captures the essence of the speaking\-style, weaving together the speaker’s inherent qualities, situational context, emotional tone, and linguistic features into a vivid and immersive portrait\.Imagine a female speaker with a high\-pitched, crisp, and nasal voice, speaking with measured speed and authority, her American accent shining through as she expresses awe and admiration in a slightly clean environment, conveying confidence and charisma with every word\.Table 7:Style\-specific instructions and example generated captions\.![Refer to caption](https://arxiv.org/html/2610.33036v2/emotion_distribution.png)\(a\) Emotion
![Refer to caption](https://arxiv.org/html/2610.33036v2/accent_distribution.png)\(b\) Accent
![Refer to caption](https://arxiv.org/html/2610.33036v2/snr_histogram.png)\(c\) SNR
![Refer to caption](https://arxiv.org/html/2610.33036v2/duration_histogram.png)\(d\) Duration

Figure 6:Distributions of emotion \(a\), accent \(b\) signal\-to\-noise \(SNR\) ratio \(c\), and speech duration \(d\) inStyleInstructCaps\-MetaSet\.\(a\) Accent

\(b\) Duration

\(c\) SNR

Figure 7:Distributions of \(a\) accent \(b\) duration and \(c\) signal\-to\-noise \(SNR\) ratio in StyleInstructCaps In\-domain balanced set \(ID\-Balanced\)\.Table 8:Summary of dataset splits\.Data splitNumber of utterancesNumber of hoursPSC/SIC train set10318402681\.54PSC/SIC validation set1196726\.29PSC/SIC evaluation set1475633\.04StyleInstructCaps\-MetaSet4341\.02In\-domain balanced set4000\.82Unseen Data\(UD\) \- VCTK3200\.31VCTK \(unseen prompts\)320\.03Gigaspeech \(with SpeechCraft captions\)6001\.00
## Appendix DLLM\- judge calibration : Adversarial gaming check

The adversarial gaming check is designed as a controlled stress\-test for the LLM\-as\-a\-judge evaluation pipeline used in StyleInstructCaps\. Starting from the 434 utterances in StyleInstructCaps\-MetaSet, we construct adversarial captions that are intentionally fluent, instruction\-aligned, or stylistically convincing while violating grounding constraints in different ways\. Each attack category targets a specific potential failure mode of the judge, such as rewarding verbosity, surface\-level instruction alignment, or stylistic fluency despite factual inconsistencies with the underlying metadata\. We use the gemma2\-27B[Team et al\. \(2024\)](https://arxiv.org/html/2610.33036#bib.bib79)to generate the adversarial captions\. The adversarial gaming check is performed for a single\-run of the Qwen3\-32B judge\.

The resulting evaluation setup enables us to measure whether the judge meaningfully penalizes adversarial behavior rather than assigning inflated scores to superficially plausible captions\. Table[9](https://arxiv.org/html/2610.33036#A4.T9)summarizes the five adversarial attack categories, the evaluation dimensions they target, and the corresponding failure modes being tested\.

Attack TypePrimary Target MetricFailure Mode TestedCountType 1: Attribute InversionMeta\-Score, HallucinationThe caption preserves fluency and overall structure while directly contradicting core metadata attributes such as gender, accent, or pitch\. This tests whether the judge anchors on stylistic fluency rather than factual correctness with respect to the provided metadata\.87Type 2: Instruction ParrotingInstruction\-Following Accuracy \(IFA\)The caption directly restates the instruction as if it were a caption, without providing any grounded acoustic or stylistic content\. This tests whether the judge rewards instruction\-aligned surface form rather than true instruction compliance\.87Type 3: Verbosity MaskingMeta\-Score, Hallucination, IFAThe caption uses long, fluent, impressionistic prose filled with prosodic language while deliberately avoiding verifiable factual claims\. This tests whether the judge exhibits verbosity bias by inflating scores for stylistically rich but ungrounded captions\.87Type 4: Hedged HallucinationHallucination SeverityThe caption introduces incorrect factual claims through epistemically hedged language such as “may suggest” or “could be interpreted as\.” This tests whether the judge is overly lenient toward softened or non\-committal hallucinations\.87Type 5: Cross\-Speaker SwapMeta\-ScoreThe caption uses a real oracle caption from a different speaker with mismatched demographic and emotional attributes\. This tests whether the judge grounds evaluation in the provided metadata or simply rewards any fluent, high\-quality caption\.86Table 9:Summary of the five adversarial attack categories used in the LLM judge calibration experiment\. Each attack probes a distinct potential failure mode of the LLM\-as\-a\-judge evaluation pipeline\.![Refer to caption](https://arxiv.org/html/2610.33036v2/calibration_heatmap_paper.png)

Figure 8:Calibration analysis of speaking\-style captioning models across metadata groundedness, instruction\-following and hallucination severity metrics\.For each attack type, we compare judge scores on oracle vs\. adversarial captions using a two\-tailed \(pairedtt\-test\), and report effect size as Cohen’sdd\. The results of the adversarial calibration experiment, as visualised in Figure[8](https://arxiv.org/html/2610.33036#A4.F8)demonstrates that the Qwen\-3\-32B LLM\-as\-a\-judge is highly sensitive to multiple forms of adversarial caption manipulation and does not merely reward fluent or stylistically convincing text\. Across all three evaluation axes, oracle captions consistently achieve near\-ceiling scores, whereas adversarial captions incur statistically significant degradations\.

For the metadata groundedness \(Meta\-Score\) evaluation, the judge consistently penalizes all five attack types\. Oracle captions obtain very high scores \(mean∼9\.5\\sim 9\.5\), while adversarial captions exhibit substantial drops across all conditions\. The most pronounced degradations are observed for Type 4 \(Hedged Hallucination\) and Type 5 \(Cross\-Speaker Swap\), with score gaps of 7\.46 and 6\.41 respectively and very large effect sizes \(d=3\.82d=3\.82andd=7\.26d=7\.26, bothp<0\.001p<0\.001\)\. Even the weakest attack, Type 1 \(Attribute Inversion\), yields a statistically significant degradation \(d=1\.39d=1\.39,p<0\.001p<0\.001\)\. These findings indicate that the judge is strongly grounded in metadata consistency and remains robust to both explicit and implicitly hedged factual violations\.

The instruction\-following \(IFA\) evaluation further demonstrates that the judge performs an independent assessment rather than collapsing instruction adherence with metadata correctness\. Type 2 \(Instruction Parroting\) induces the largest degradation in IFA performance, reducing scores from approximately 9\.0 to 2\.59 with a large effect size \(d=4\.23d=4\.23,p<0\.001p<0\.001\)\. Type 4 \(Hedged Hallucination\) also significantly degrades IFA performance \(d=3\.73d=3\.73,p<0\.001p<0\.001\)\. In contrast, Type 1 \(Attribute Inversion\) has only a minor impact on IFA \(d=0\.34d=0\.34,p<0\.05p<0\.05\), as the captions preserve structural compliance with the instruction despite factual corruption\. Type 3 \(Verbosity Masking\) yields a moderate reduction \(d=1\.58d=1\.58,p<0\.001p<0\.001\), indicating that stylistically rich but semantically ungrounded outputs are not rewarded as valid instruction\-following instances\. Overall, these results show that the judge disentangles instruction adherence from factual grounding\.

The hallucination severity analysis reveals a more nuanced pattern\. Type 1 \(Attribute Inversion\) and Type 5 \(Cross\-Speaker Swap\) consistently produce the highest hallucination rates, confirming that explicit factual contradictions are reliably detected\. Type 2 \(Instruction Parroting\) is also strongly penalized despite containing limited explicit factual content, suggesting that the absence of grounded information itself contributes to hallucination judgments\. Interestingly, Type 4 \(Hedged Hallucination\) is not consistently assigned high hallucination severity, despite being strongly penalized under Meta\-Score and IFA evaluations\. This indicates that epistemically hedged formulations may partially obscure hallucination detection under the dedicated hallucination rubric, even though they remain identifiable as low\-quality or weakly grounded under other evaluation axes\.

Additionally, we observe that oracle captions remain near\-saturated across metrics, supporting evaluation stability; SALMONN outputs score below oracle references but well above adversarial captions, indicating the judge does not penalize all non\-oracle outputs\.

Overall, these results provide strong evidence that the LLM\-as\-a\-judge framework is robust to a diverse set of adversarial manipulations\. The judge consistently distinguishes grounded from ungrounded captions, remains sensitive to factual inconsistencies even under hedging, and evaluates instruction\-following, hallucination, and metadata consistency as partially decoupled dimensions rather than conflating them into a single fluency\-based score\.

## Appendix EHuman\-LLM judge correlation study

Table[1](https://arxiv.org/html/2610.33036#S4.T1)ranks speaking\-style captioning systems with scores from a panel of LLM judges\. An LLM judge can reward fluent or detailed text that listeners do not hear in the speech\. We therefore test the judge against human listeners\. The study answers four questions, each addressed by one results paragraph below\.\(Q1\)Do listeners agree with each other well enough to serve as a reference?\(Q2\)Does the judge prefer the same caption as listeners, pair by pair?\(Q3\)Is the judge more often right when its score gap between two captions is larger?\(Q4\)Does the system ranking survive when it is fit to human votes alone?

### E\.1Study design

Judge score under test\.For each captioncc, the panel assigns a metadata\-groundedness scores⁡\(c\)s\(c\)on a 1–10 scale \(Section[4\.1](https://arxiv.org/html/2610.33036#S4.SS1)\)\. The*ensemble*score is the mean over three judges \(Qwen3\-32B, Gemma\-4\-31B\-it, GPT\-OSS\-20B\) and three runs per judge, nine runs in total \(Appendix[F](https://arxiv.org/html/2610.33036#A6)\)\. This is the score reported in Table[1](https://arxiv.org/html/2610.33036#S4.T1)\. The judge reads the utterance metadata and the caption and never hears the audio\. Listeners hear the audio and never see the metadata\. The study therefore tests whether a score computed from metadata agrees with what listeners perceive in the speech\.

Caption pairs and score gap\.Each of the 100 study items consists of one utterance fromStyleInstructCaps\-MetaSetand the captions of two different systems for that utterance\. The 14 systems include the oracle captions ofStyleInstructCaps\-DB, which enter the study as one system alongside the 13 captioning models \(the 12 open\-weight models and GPT\-Audio\); the PSC oracle is not included\. For an item with captionscac\_\{a\}andcbc\_\{b\}, the signed score gap isΔ=s⁡\(ca\)−s⁡\(cb\)\\Delta=s\(c\_\{a\}\)\-s\(c\_\{b\}\)\. The judge preferscac\_\{a\}whenΔ\>0\\Delta\>0andcbc\_\{b\}whenΔ<0\\Delta<0\. The absolute gap\|Δ\|\|\\Delta\|measures how strongly the judge separates the two captions\. A near\-zero gap means the judge rates both captions almost equally\. Items were sampled to cover all 14 systems defined above and the full range of\|Δ\|\|\\Delta\|\. Items with small\|Δ\|\|\\Delta\|were over\-sampled\. These are the decisions on which a ranking error is most likely, so the study spends most of its trial budget there instead of on easy pairs\. For analysis, we partition the observed range of\|Δ\|\|\\Delta\|into five equal\-width buckets, from B1 \(smallest gaps,*near\-tie*\) to B5 \(largest gaps\)\.

Task\.On each trial, a listener heard the raw audio clip, read the two captions, and chose the one that better described the speaking style\. The order of the two captions was randomized\. We use a binary forced choice for two reasons\. It matches how the benchmark is used, which is to decide which of two systems is better\. It also avoids absolute 1–10 ratings, whose scale drifts from listener to listener\. All listeners rated the same 100 items, so every agreement statistic below is computed on a fully crossed design\.

Listeners and quality control\.Fifteen English\-speaking participants took part\. Eight attention\-check trials with a known correct answer were mixed into the session\. A listener had to answer at least 6 of the 8 checks correctly \(75%\) to be retained\. Two listeners failed, with 3/8 \(37\.5%\) and 5/8 \(62\.5%\) correct, and are excluded\. All results use the remaining 13 listeners and their 1,300 votes\. For each item, the*human majority*is the caption chosen by at least 7 of the 13 listeners\. With an odd number of listeners, the majority is always defined\.

### E\.2Metrics

Agreement among listeners \(Q1\)\.Krippendorff’sα\\alphaand Fleiss’κ\\kappameasure how often listeners pick the same caption, corrected for the agreement expected by chance\. A value of 0 means chance\-level agreement and 1 means perfect agreement\. Mean pairwise agreement is the uncorrected fraction of items on which two listeners pick the same caption, averaged over all 78 listener pairs\. Its chance level is 50%\. We also count how evenly each item’s 13 votes divide\. An item is*unanimous*when all 13 listeners pick the same caption\. Non\-unanimous items are*lopsided*when one caption wins by a wide margin and*split*when the vote is close to even\.

Agreement between judge and listeners \(Q2\)\.*Agreement*is the fraction of items on which the judge prefers the same caption as the human majority\. Items on which the judge assigns both captions the same score \(Δ=0\\Delta=0\) have no judge preference and are excluded\. The number of remaining itemsnnis reported per judge\. Cohen’sκ\\kappacorrects agreement for chance, with the same 0\-to\-1 reading as above\. Thepp\-value is a two\-sided exact binomial test of agreement against the 50% expected from a judge that guesses\. All intervals are 95% confidence intervals\.

Calibration against the score gap \(Q3\)\.If\|Δ\|\|\\Delta\|reflects how confident the judge is, agreement should rise with\|Δ\|\|\\Delta\|\. We test this in two ways\. First, we report agreement per\|Δ\|\|\\Delta\|bucket\. Second, we fit a logistic regression to the 1,300 individual votes \(top\), and we fit a Bradley–Terry model for Q4 \(bottom\):

P⁡\(ca≻cb\)=σ⁡\(β0\+β1​Δ\),\\displaystyle P\(c\_\{a\}\\succ c\_\{b\}\)=\\sigma\(\\beta\_\{0\}\+\\beta\_\{1\}\\Delta\),P⁡\(i≻j\)=πiπi\+πj\.\\displaystyle P\(i\\succ j\)=\\frac\{\\pi\_\{i\}\}\{\\pi\_\{i\}\+\\pi\_\{j\}\}\.Hereσ\\sigmais the logistic function andcac\_\{a\}is the first\-presented caption\. A positive slopeβ1\\beta\_\{1\}means that listeners prefer the higher\-scored caption more often as the gap grows\. The magnitude of the score then carries information, not only its sign\. The interceptσ⁡\(β0\)\\sigma\(\\beta\_\{0\}\)is the probability of choosing the first\-presented caption when the judge sees a tie\. A value near 50% indicates no position bias\.

System\-level ranking \(Q4\)\.The Bradley–Terry model assigns each systemiia strengthπi\\pi\_\{i\}and gives the probability that systemii’s caption is preferred over systemjj’s\. We fit it by maximum likelihood to all 1,300 individual human votes, without any judge scores, and rank systems byπi\\pi\_\{i\}\. We compare this ranking over the 14 systems with the system ranking induced by each judge’s metadata\-groundedness scores\. Spearman’sρ\\rhocorrelates the two rank orders\. Kendall’sτ\\tauis the fraction of system pairs ordered the same way minus the fraction ordered differently, so it penalizes every local swap\.

### E\.3Results

Table 10:Inter\-listener reliability and quality control for the human pairwise study \(13 retained listeners, 100 items\)\. Chance level for pairwise agreement is 50%\.MeasureValueKrippendorff’sα\\alpha/ Fleiss’κ\\kappa0\.347 / 0\.346Mean pairwise agreement67\.3%Split / lopsided / unanimous items45 / 44 / 11Attention\-check pass rate \(all 15 listeners\)83\.3%

Q1: Listeners agree above chance, and the majority vote is the reference\.Table[10](https://arxiv.org/html/2610.33036#A5.T10)reports agreement among the 13 retained listeners\. Krippendorff’sα\\alphais 0\.347 and Fleiss’κ\\kappais 0\.346\. Two listeners pick the same caption on 67\.3% of items, against a 50% chance level\. Individual listeners therefore often disagree\. This is expected for two reasons\. Judging which caption better describes a speaking style is subjective, and the sampling deliberately over\-represents near\-tie items\. Accordingly, 45 of the 100 items received a split vote\. For this reason, we compare the judge with the majority of 13 listeners and not with any single listener\. The majority vote averages out individual variation\.

Table 11:Agreement between the human majority vote and the preference of each LLM judge and of the reported ensemble\. Items on which a judge assigns equal scores are excluded, leavingnnitems\. Brackets give 95% confidence intervals\.pptests agreement against a 50% chance level\.JudgennAgreement \(%\)Cohen’sκ\\kappappEnsemble9985\.9 \[77\.8, 91\.9\]0\.7171×10−131\\times 10^\{\-13\}Qwen3\-32B6583\.1 \[73\.8, 92\.3\]0\.6616×10−86\\times 10^\{\-8\}GPT\-OSS\-20B9682\.3 \[74\.0, 89\.6\]0\.6461×10−101\\times 10^\{\-10\}Gemma\-4\-31B\-it9284\.8 \[76\.1, 91\.3\]0\.6966×10−126\\times 10^\{\-12\}

Q2: The judge prefers the same caption as listeners on most items\.Table[11](https://arxiv.org/html/2610.33036#A5.T11)reports the results\. The ensemble agrees with the human majority on 85\.9% of the 99 items it decides, with Cohen’sκ=0\.717\\kappa=0\.717\(p=1×10−13p=1\\times 10^\{\-13\}\)\. Individual judges reach 82\.3% to 84\.8%\. The ensemble has the highest point estimate, but its confidence interval overlaps with those of the individual judges, so we do not claim a significant gain from ensembling\. The ensemble averages nine runs and assigns tied scores less often, so it decides more items than any single judge\. Excluding the 15 near\-tie items \(B1\) leaves 84 items, on which agreement rises to 88\.1% \(95% CI \[81\.0, 95\.2\]\)\.

Table 12:Agreement between the human majority and the ensemble judge as a function of the absolute ensemble score gap\|Δ\|\|\\Delta\|, partitioned into five equal\-width buckets\.ppis a two\-sided exact binomial test against 50%\.BucketnnAgreementppB1 \(near\-tie\)1573\.3%0\.12B22080\.0%0\.012B32487\.5%3×10−43\\times 10^\{\-4\}B42592\.0%2×10−52\\times 10^\{\-5\}B5 \(large gap\)1593\.3%1×10−31\\times 10^\{\-3\}Q3: Agreement rises with the judge’s score gap\.Table[12](https://arxiv.org/html/2610.33036#A5.T12)reports agreement per bucket\. Agreement rises monotonically from 73\.3% in the near\-tie bucket to 93\.3% in the largest\-gap bucket\. Buckets B2 to B5 are each significantly above chance \(p≤0\.012p\\leq 0\.012\)\. The near\-tie bucket is not \(p=0\.12p=0\.12, 11 of 15 items\)\. On near\-tie items the judge rates both captions almost equally, and listeners split in the same way\. The size of a score difference in Table[1](https://arxiv.org/html/2610.33036#S4.T1)therefore predicts how reliably listeners share the judge’s preference\.

Table 13:Rank correlation over the 14 systems between the ranking of a Bradley–Terry model fit to human votes alone and the ranking produced by each LLM judge\.JudgeSpearmanρ\\rhoKendallτ\\tauEnsemble0\.8950\.780Qwen3\-32B0\.8730\.707GPT\-OSS\-20B0\.8550\.692Gemma\-4\-31B\-it0\.8810\.714Q4: Human votes alone recover the judge’s system ranking\.Table[13](https://arxiv.org/html/2610.33036#A5.T13)compares the two rankings\. The ensemble reaches Spearmanρ=0\.895\\rho=0\.895and Kendallτ=0\.780\\tau=0\.780\. Individual judges reachρ\\rhobetween 0\.855 and 0\.881\. The five systems ranked lowest by listeners are the same five ranked lowest by the judge: Audio\-Flamingo\-3, Qwen2\-Audio\-7B\-Instruct, MERaLiON\-AudioLLM, Voxtral\-Small\-24B and SALMONN\. At the top of the ranking, the human\-fit model places MOSS\-Audio above the oracle captions\. The 95% confidence intervals of the two Bradley–Terry strengths overlap heavily, so listeners do not separate these two systems\.

Summary\.Listeners agree with each other above chance, and their majority vote provides a reference for the judge\. The judge panel prefers the same caption as the majority on 85\.9% of items\. Its agreement grows with its own score gap, so score magnitudes carry information that listeners confirm\. A ranking fit to human votes alone reproduces the judge’s system ranking atρ=0\.895\\rho=0\.895\. These results support the use of the LLM judge panel for the caption\-quality scores reported in Table[1](https://arxiv.org/html/2610.33036#S4.T1)\.

## Appendix FPanel of LLM judges: cross\-judge agreement

To test whether the benchmark conclusions depend on the choice of evaluator, we evaluate the open\-ended metrics with three independent open\-weight LLM judges and execute each judge multiple times\. This design quantifies cross\-judge agreement and the stability of system rankings across judges\.

Judge panel\.The panel consists of three open\-weight judges spanning different model families and scales: Qwen3\-32B, Gemma\-4\-31B\-it, and GPT\-OSS\-20B \(Table[14](https://arxiv.org/html/2610.33036#A6.T14)\)\. All judges receive the same prompts \(Appendix[H](https://arxiv.org/html/2610.33036#A8)\)\. Table[15](https://arxiv.org/html/2610.33036#A6.T15)lists the number of runs per judge and metric\. Instruction\-following is scored with a single Qwen3\-32B run because it requires 2,604 judgments per audio\-language model, judge, and run\.

Table 14:LLM judges used in the evaluation panel\.JudgeFamily / sizeRoleQwen3\-32B \(thinking\)Qwen, 32B densePanel judge; instruction\-followingGemma\-4\-31B\-it \(thinking\)Gemma, 31B densePanel judgeGPT\-OSS\-20B \(reasoning\)OpenAI OSS, 20B MoEPanel judge

Table 15:Number of runs per judge and metric\.Evaluation setMeta\. groundednessHallucinationInstr\.\-followingMetaSet3×\\timeseach judge3×\\timeseach judge1×\\timesQwen3\-32BOOD \(GigaSpeech, VCTK\)2×\\timeseach judge2×\\timeseach judge1×\\timesQwen3\-32B

Cross\-judge agreement\.Because the benchmark is used to rank systems, the central question is whether the ranking is an artifact of a specific judge\. We compute Spearman’s rank correlation between the per\-judge mean metadata\-groundedness scores of the 14 benchmarked systems onStyleInstructCaps\-MetaSet\(Table[16](https://arxiv.org/html/2610.33036#A6.T16)\)\. All three judge pairs agree strongly \(ρ≥0\.864\\rho\\geq 0\.864\), and Gemma\-4\-31B\-it and GPT\-OSS\-20B agree almost perfectly \(ρ=0\.978\\rho=0\.978\)\.

Table 16:Cross\-judge agreement: Spearman rank correlation between judges over the 14 benchmarked systems \(MetaSetmetadata groundedness, per\-judge means\)\.Judge pairSpearmanρ\\rho\(n=14n=14\)GPT\-OSS\-20B↔\\leftrightarrowGemma\-4\-31B\-it0\.978Qwen3\-32B↔\\leftrightarrowGPT\-OSS\-20B0\.921Qwen3\-32B↔\\leftrightarrowGemma\-4\-31B\-it0\.864

Reproducibility\.We release the judge model identifiers, prompts, and decoding settings for all three judges together with the benchmark\.

## Appendix GPairwise statistical significance of system rankings

Method\.Our results tables rank systems against each other\. We therefore test whether the gap between two systems reflects a real difference or only noise from the judges or the utterances\. We first compute a per\-utterance consensus score for each system, which gives one score per system for every utterance and lets us pair scores across systems\. For metadata groundedness and hallucination, this score is the mean over the system’s panel runs \(9 forMetaSetand 6 for the out\-of\-domain set; Appendix[F](https://arxiv.org/html/2610.33036#A6)\)\. For instruction\-following, it is the score from the single Qwen3\-32B run\. We then compare every pair of systems with a two\-tailed paired test over utterances\. We use both a pairedtt\-test and a Wilcoxon signed\-rank test, and the two agree in every case\. Groundedness and hallucination scores are paired by utterance index, and instruction\-following scores are paired by \(utterance, instruction type\)\. We report significance atp<0\.05p<0\.05along with effect sizes, namely the mean paired differenceΔ\\Deltaand Cohen’sdd\. Before looking at individual pairs, we run an omnibus Friedman test to check whether the systems differ at all on a given metric\.

Systems differ, and the leaderboards are statistically well separated\.The Friedman test is highly significant \(p<10−4p<10^\{\-4\}\) for every metric and every evaluation set\. Table[17](https://arxiv.org/html/2610.33036#A7.T17)shows how many of the\(n2\)\\binom\{n\}\{2\}system pairs differ atp<0\.05p<0\.05\.

Table 17:Omnibus Friedman test and number of significantly different system pairs \(p<0\.05p<0\.05\) out of all\(n2\)\\binom\{n\}\{2\}pairs, per metric and evaluation set\.nnis the number of systems compared\. TheMetaSetgroundedness and hallucination tests use all 15 systems of Table[1](https://arxiv.org/html/2610.33036#S4.T1)\(two oracle caption sets, 12 open\-weight models, and GPT\-Audio\)\. Instruction\-following excludes the PSC oracle and GPT\-Audio, which have no instruction\-following scores\. The OOD groundedness and hallucination tests use the 13 systems of Table[3](https://arxiv.org/html/2610.33036#S6.T3)\. The large majority \(86–96%\) of pairs are statistically distinguishable; the remaining pairs are statistically close\.Eval set \(table\)MetricnnFriedmanppSig\. pairs%MetaSet\(Tab\.[1](https://arxiv.org/html/2610.33036#S4.T1)\)Groundedness↑\\uparrow15<10−4<10^\{\-4\}101/10596\.2Hallucination↓\\downarrow15<10−4<10^\{\-4\}99/10594\.3Instr\.\-following↑\\uparrow13<10−4<10^\{\-4\}73/7893\.6OOD \(Tab\.[3](https://arxiv.org/html/2610.33036#S6.T3)\)Groundedness↑\\uparrow13<10−4<10^\{\-4\}68/7887\.2Hallucination↓\\downarrow13<10−4<10^\{\-4\}67/7885\.9OOD \(Tab\.[4](https://arxiv.org/html/2610.33036#S6.T4)\)Instr\.\-following↑\\uparrow14<10−4<10^\{\-4\}79/9186\.8

Our central claims are statistically significant\.Table[18](https://arxiv.org/html/2610.33036#A7.T18)shows that the paired tests confirm the key findings of the paper onMetaSetmetadata groundedness \(Table[1](https://arxiv.org/html/2610.33036#S4.T1)\)\. Our main claim is that SSC\-specific, instruction\-diverse training improves groundedness, and this gain is statistically significant\. The oracle upper bound scores significantly higher than every real model, which validates the metric\. The closed\-source GPT\-Audio scores significantly lower than the best open model on groundedness and significantly worse on hallucination on both evaluation sets \(allp<10−4p<10^\{\-4\}\)\. This is consistent with our finding that following instructions does not imply grounded understanding\.

Table 18:Headline comparisons onMetaSetmetadata groundedness \(↑\\uparrow;N=434N=434utterances\)\.Δ\\Deltais the mean per\-utterance score gap\. A Cohen’sddabove 0\.8 is conventionally a large effect, so these gaps are both significant and sizeable\.ComparisonΔ\\DeltaddWilcoxonppSSC\-tuned SALMONN \(SIC\-DB, MT\)\>\>baseline SALMONN\+2\.67\+2\.671\.51<10−4<10^\{\-4\}SSC\-tuned SALMONN \(SIC\-DB, ST\)\>\>baseline SALMONN\+2\.62\+2\.621\.53<10−4<10^\{\-4\}Oracle \(SIC\-DB\)\>\>best real model\+2\.51\+2\.511\.70<10−4<10^\{\-4\}Best open model\>\>GPT\-Audio \(closed SOTA\)\+1\.65\+1\.650\.87<10−4<10^\{\-4\}Near\-identical systems are not separated\.The tests do not distinguish systems that are nearly identical by design\. On the out\-of\-domain set, for example, the four SALMONN variants \(means 6\.25–6\.30\) do not differ significantly from one another \(SSC\-MT vs\. baseline SALMONN:Δ=\+0\.02\\Delta=\+0\.02,p=0\.69p=0\.69\)\. The non\-significant pairs in Table[17](https://arxiv.org/html/2610.33036#A7.T17)are near\-ties of this kind\.

## Appendix HPrompts used

You are an expert linguist and speech analyst tasked with comparing the speaking\-styles between two sets of audio metadata\. Your goal is to determine the similarity between a ground truth recording and a generated caption, focusing particularly on stylistic aspects of speech, intention, and prosody\. Please carefully review the following metadata: Ground Truth Metadata \(DATA1\): <ground\_truth\_metadata\> \{data1\} </ground\_truth\_metadata\> Generated Caption Metadata \(DATA2\): <generated\_caption\_metadata\> \{data2\} </generated\_caption\_metadata\> Your task is to compare these two sets of metadata and rate their similarity on a scale of 1 to 10, where 1 indicates completely different speaking\-styles and 10 indicates identical speaking\-styles\. Instructions: 1\. Analyze and compare the following attributes between DATA1 and DATA2: a\) Speaker gender b\) Accent c\) Pitch d\) Speaking rate e\) Environment/noise level f\) Emotion/expression g\) Voice quality \(e\.g\., guttural, silky\) h\) Prosody \(including rhythm, intonation, and stress\) i\) Intention of speech 2\. Inside your thinking block, wrap your attribute comparisons inside <attribute\_comparison\> tags\. For each attribute: \- Rate its similarity on a scale of 1\-10 \- Summarize key similarities and differences \- Consider how they contribute to the overall speaking\-style Pay special attention to stylistic aspects, intention, and prosody\. 3\. After analyzing all attributes, calculate an average similarity score based on your individual attribute ratings\. 4\. Don’t penalize the generation if is more specific than the ground truth\. For example, if the ground truth says "American accent" and the generation says "Boston accent", that’s fine\. 5\. Provide a comprehensive justification for your similarity rating in <justification\> tags\. Explain how the similarities and differences in each attribute contribute to your overall assessment\. 6\. In the metadata, only using the following information: \- speaker gender \- accent \- pitch \- speaking rate \- environment/noise level \- emotion/expression 7\. Finally, provide your similarity score in JSON format within <score\> tags\. The JSON should include a single key ’similarity\_score’ with a value between 1 and 10, based on your calculated average and overall assessment\. Example output structure \(do not copy the content, only the structure\): <attribute\_comparison\> a\) Speaker gender: Similarity rating: X/10 Key similarities: \[\.\.\.\] Key differences: \[\.\.\.\] Contribution to overall style: \[\.\.\.\] b\) Accent: Similarity rating: X/10 Key similarities: \[\.\.\.\] Key differences: \[\.\.\.\] Contribution to overall style: \[\.\.\.\] \[\.\.\. continue for all attributes \.\.\.\] Average similarity score: X/10 </attribute\_comparison\> <justification\> \[Your comprehensive justification for the similarity rating, synthesizing the analyses of individual attributes\] </justification\> <score\> \{\{’similarity\_score’: X\}\} </score\> Remember to focus on the stylistic aspects of speech, paying particular attention to the intention behind the speech and the nuances of prosody in your analysis and justification\. Your final output should consist only of the justification and score, without duplicating the detailed attribute comparisons from your thinking block\.

Figure 9:Prompt template for rating speaking\-style similarity between ground\-truth metadata and generated\-caption metadata\.
You are an expert evaluator assessing the quality of automatically generated audio captions\. Your task is to compare metadata from audio files with generated captions to identify and score hallucinations\.
\#\# Task Overview
Compare the \*\*metadata\*\* \(ground truth information about the audio\) with a \*\*generated caption\*\* and evaluate the presence and severity of hallucinations\.
\#\# Input Format
Ground Truth Metadata:
<ground\_truth\_metadata\>
\{data1\}
</ground\_truth\_metadata\>
Generated Caption:
<generated\_caption\>
\{data2\}
</generated\_caption\>
\#\# Evaluation Criteria
\#\#\# What Counts as a Hallucination?
A hallucination is information in the generated caption that:
\- Directly contradicts the metadata
\- Invents specific details not supported by any evidence
\- Misrepresents factual characteristics \(e\.g\., wrong gender, wrong emotion, wrong accent\)
\#\#\# What Does NOT Count as a Hallucination?
The following are \*\*acceptable creative interpretations\*\* and should NOT be penalized:
\- \*\*Reasonable emotional inferences\*\*: If metadata shows "laughing" audio, describing the speaker as "cheerful," "animated," or "enthusiastic" is acceptable
\- \*\*Stylistic embellishments\*\*: Descriptive language like "her voice carries warmth," "delivered with precision," or "words tumble out" that enhance the description without contradicting facts
\- \*\*Contextual scenarios\*\*: Adding reasonable context \(e\.g\., "as if addressing an audience," "in a quiet environment"\) when supported by acoustic characteristics
\- \*\*Prosodic interpretations\*\*: Descriptions of rhythm, cadence, emphasis, or delivery style that are reasonable given the speech characteristics
\- \*\*Minor elaborations\*\*: Details that naturally extend from the metadata without contradicting it
\#\#\# Hallucination Examples
\*\*SEVERE Hallucinations:\*\*
\- Metadata indicates male speaker \-\> Caption says "female speaker"
\- Metadata shows speaker p026 \-\> Caption invents specific name or identity
\- Metadata indicates disgust emotion \-\> Caption describes joy or happiness
\- Inventing specific events or quoted speech not in the audio
\*\*MODERATE Hallucinations:\*\*
\- Claiming "noisy environment" when metadata suggests clean audio
\- Describing specific background sounds not mentioned in metadata
\- Asserting definitive context \(e\.g\., "delivering a news broadcast"\) without support
\*\*MINOR Hallucinations:\*\*
\- Slightly exaggerating a characteristic \(e\.g\., "booming voice" for a moderately loud voice\)
\- Adding plausible but unverified details that don’t contradict metadata
\#\# Scoring Instructions
Provide two scores:
\#\#\# 1\. Hallucination Count \(0\-10 scale\)
\- \*\*0\-2\*\*: No hallucinations or only very minor embellishments
\- \*\*3\-4\*\*: 1\-2 minor hallucinations that don’t significantly distort the audio
\- \*\*5\-6\*\*: Multiple minor hallucinations OR 1 moderate hallucination
\- \*\*7\-8\*\*: Multiple moderate hallucinations OR 1 severe hallucination
\- \*\*9\-10\*\*: Multiple severe hallucinations that fundamentally misrepresent the audio
\#\#\# 2\. Severity Level
\- \*\*Low\*\*: Only minor embellishments; caption is largely accurate
\- \*\*Medium\*\*: Some factual errors that partially misrepresent the audio
\- \*\*High\*\*: Severe contradictions or fabrications that fundamentally mischaracterize the audio
\#\# Instructions
1\. In your thinking block, carefully analyze the metadata and generated caption step by step:
\- List all factual claims made in the generated caption
\- For each claim, check if it’s supported by, contradicts, or extends beyond the metadata
\- Categorize each issue as minor, moderate, or severe
\- Count the total number of hallucinations
2\. Be lenient with creative, descriptive language that doesn’t contradict facts
3\. Focus on factual accuracy regarding speaker identity, emotions, accent, and acoustic characteristics
4\. Distinguish between reasonable inference and baseless fabrication
\#\# Output Format
After your thinking, provide your evaluation in the following structure:
<analysis\>
\[Brief explanation of what you observed, noting any hallucinations found and why they are/aren’t problematic\]
</analysis\>
<justification\>
Explain your scoring in detail, referencing specific examples from the caption and how they relate to the metadata
</justification\>
<score\>
\{\{"hallucination\_count": X, "severity": "Low/Medium/High"\}\}
</score\>
Remember: Context is important \- "laughing" audio naturally suggests positive emotions\. Be strict about factual contradictions but lenient about creative descriptions that align with the metadata\.
Your final output should consist only of the analysis, justification and score, without duplicating the detailed step\-by\-step analysis from your thinking block\.Figure 10:Hallucination evaluation prompt for comparing ground\-truth metadata against a generated caption\.You are an expert evaluator specializing in assessing instruction\-following capabilities of audio\-language models, particularly in the domain of speaking\-style captioning\. Your role is to rigorously evaluate whether generated captions accurately follow given instructions while maintaining factual accuracy about the audio content\.
\#\# Context
audio\-language models are tasked with generating descriptive captions about speaking\-styles, vocal characteristics, and acoustic properties of speech recordings\. These models receive specific instructions that guide what aspects to focus on, what style to adopt, or what format to use in their captions\.
\#\# Your Task
Evaluate how well the generated caption follows the provided instruction, considering both explicit requirements and implicit expectations\.
\-\-\-
\#\# Input Materials
\#\#\# Instruction Given to the Model:
<instruction\>
\{instruction\}
</instruction\>
\#\#\# Generated Caption by the Model:
<generated\_caption\>
\{caption\}
</generated\_caption\>
\-\-\-
\#\# Evaluation Framework
\#\#\# 1\. \*\*Instruction Compliance Categories\*\*
Analyze the instruction for the following types of requirements:
\*\*A\. Content Requirements\*\*
\- Specific attributes to include \(e\.g\., "describe the pitch," "mention the accent"\)
\- Attributes to exclude or de\-emphasize
\- Level of detail requested \(brief, detailed, comprehensive\)
\- Factual accuracy requirements
\*\*B\. Structural Requirements\*\*
\- Format specifications \(paragraph, bullet points, structured description\)
\- Length constraints \(word count, sentence count\)
\- Organizational pattern \(chronological, importance\-based, categorical\)
\*\*C\. Stylistic Requirements\*\*
\- Tone \(formal, casual, technical, creative\)
\- Perspective \(objective observer, subjective interpreter, technical analyst\)
\- Language complexity \(simple, sophisticated, jargon\-heavy\)
\- Writing style \(descriptive, analytical, narrative\)
\*\*D\. Focus Requirements\*\*
\- Specific aspects to emphasize \(prosody, emotion, acoustic features\)
\- Target audience considerations
\- Use case or application context
\*\*E\. Constraint Requirements\*\*
\- Prohibitions \(avoid certain terms, don’t speculate\)
\- Boundaries \(stick to observable features only\)
\- Scope limitations \(focus on X, ignore Y\)
\#\#\# 2\. \*\*Evaluation Criteria\*\*
For each requirement identified in the instruction, assess:
\*\*Explicit Compliance \(70% weight\)\*\*
\- Does the caption directly address explicitly stated requirements?
\- Are all mandatory elements present?
\- Are any prohibited elements absent?
\- Is the specified format/structure followed?
\*\*Implicit Compliance \(20% weight\)\*\*
\- Does the caption follow reasonable interpretations of the instruction?
\- Are contextual clues and implied expectations honored?
\- Does it align with the spirit/intent of the instruction?
\*\*Quality of Execution \(10% weight\)\*\*
\- How well are the requirements implemented \(not just their presence\)?
\- Is the execution natural and coherent, or forced and awkward?
\- Does following the instruction enhance or detract from caption quality?
\#\#\# 3\. \*\*Common Instruction Types in Audio Captioning\*\*
Be aware of these typical instruction patterns:
\- \*\*Attribute\-focused\*\*: "Describe the speaker’s pitch and speaking rate"
\- \*\*Style\-focused\*\*: "Write in a technical, objective manner"
\- \*\*Audience\-focused\*\*: "Explain as if to a non\-expert"
\- \*\*Format\-focused\*\*: "Provide a structured analysis with separate sections"
\- \*\*Perspective\-focused\*\*: "Focus on perceptual qualities rather than technical measurements"
\- \*\*Comparative\*\*: "Compare this to typical conversational speech"
\- \*\*Holistic vs\. Analytical\*\*: "Provide an overall impression" vs\. "Break down individual components"Figure 11:Instruction\-following evaluation prompt \(Part 1\): context, inputs, and evaluation framework overview\.\-\-\-
\#\# Scoring Guidelines
\#\#\# Instruction\-Following Success: Success / Partial Success / Failure
\*\*Success\*\*:
\- All critical requirements met \(90%\+ compliance\)
\- No significant violations of explicit constraints
\- Natural execution that maintains caption quality
\- Minor omissions only in non\-essential elements
\*\*Partial Success\*\*:
\- Most requirements met \(60\-89% compliance\)
\- Some explicit requirements missed or violated
\- Execution may be somewhat awkward or forced
\- Core intent of instruction honored despite gaps
\*\*Failure\*\*:
\- Major requirements ignored \(<60% compliance\)
\- Explicit constraints violated
\- Instruction intent misunderstood or disregarded
\- Caption appears written without considering the instruction
\#\#\# Instruction\-Following Accuracy: 1\-\-10 Scale
\*\*9\-10 \(Exceptional\)\*\*: Near\-perfect instruction adherence\. All explicit and implicit requirements met with natural, high\-quality execution\. Any deviations are trivial\.
\*\*7\-8 \(Strong\)\*\*: Excellent instruction\-following with minor gaps\. All critical requirements met\. May miss 1\-2 minor elements or execute some aspects imperfectly, but overall very aligned\.
\*\*5\-6 \(Adequate\)\*\*: Satisfactory instruction\-following with notable gaps\. Core requirements met but several secondary elements missed or poorly executed\. Instruction intent generally honored\.
\*\*3\-4 \(Weak\)\*\*: Poor instruction\-following with major gaps\. Many requirements missed or violated\. Some alignment with instruction but significant divergence in execution\.
\*\*1\-2 \(Very Weak\)\*\*: Minimal instruction\-following\. Most requirements ignored\. Caption appears largely independent of the instruction\. Major misunderstanding or disregard of requirements\.
\-\-\-
\#\# Special Considerations
\#\#\# Edge Cases to Watch For:
1\. \*\*Conflicting Requirements\*\*: If instruction contains contradictory elements, evaluate based on how well the model navigates the conflict\.
2\. \*\*Ambiguous Instructions\*\*: If instruction is vague, give credit for reasonable interpretations that align with common sense\.
3\. \*\*Impossible Requirements\*\*: If instruction asks for information not determinable from audio alone, don’t penalize the model for not hallucinating content\.
4\. \*\*Over\-compliance\*\*: If model follows instruction so rigidly that caption quality suffers dramatically, note this in justification but still credit the instruction\-following attempt\.
5\. \*\*Creative Interpretation\*\*: Distinguish between helpful creative interpretation \(good\) and ignoring instructions \(bad\)\.
\#\#\# Common Pitfalls to Identify:
\- ✘ \*\*Instruction ignored entirely\*\*: Caption is generic and could apply to any instruction
\- ✘ \*\*Partial attention\*\*: Model addresses only the first part of a multi\-part instruction
\- ✘ \*\*Format violation\*\*: Instruction specifies structure but caption uses different format
\- ✘ \*\*Tone mismatch\*\*: Formal instruction produces casual output, or vice versa
\- ✘ \*\*Scope creep\*\*: Model includes content explicitly excluded by instruction
\- ✘ \*\*Misinterpretation\*\*: Model misunderstands key terms or intent of instruction
\-\-\-
\#\# Output Format
Provide your evaluation in the following structure:
<analysis\> \[Identify the key requirements in the instruction: What are the explicit demands? What are the implicit expectations? What constraints are specified? Categorize them as content, structural, stylistic, focus, or constraint requirements\.\] </analysis\>
<compliance\_assessment\> \[For each identified requirement, evaluate whether the caption meets it\. Mark each as: ✓ Met,△\\trianglePartially Met, or ✗ Not Met\. Provide brief evidence from the caption\.\] </compliance\_assessment\>
<strengths\> \[Highlight what the model did well in following the instruction\. What requirements were executed particularly effectively?\] </strengths\>
<weaknesses\> \[Identify gaps, violations, or missed opportunities\. What requirements were ignored, poorly executed, or misinterpreted?\] </weaknesses\>
<justification\> \[Synthesize your analysis into a coherent evaluation\. Explain your success determination and accuracy score\. Reference specific examples from both the instruction and caption\. Consider the severity of any violations and the overall alignment with instruction intent\.\] </justification\>
<ratings\>
\{ "instruction\_following\_success": "\[Success/Partial Success/Failure\]", "instruction\_following\_accuracy": \[1\-10\] \}
</ratings\>
\-\-\-
\#\# Important Reminders
1\. \*\*Focus on instruction compliance, not caption quality\*\*: A poorly written caption that follows instructions perfectly should score higher than a beautifully written caption that ignores them\.
2\. \*\*Be precise in your reasoning\*\*: Cite specific phrases from both the instruction and caption to support your evaluation\.
3\. \*\*Distinguish between critical and minor requirements\*\*: Not all instruction elements carry equal weight\. Prioritize core requirements over secondary details\.
4\. \*\*Consider context\*\*: Audio captioning has domain\-specific norms\. Evaluate within the context of speaking\-style description conventions\.
5\. \*\*Be fair but rigorous\*\*: Give credit for good\-faith attempts while maintaining high standards for what constitutes successful instruction\-following\.
6\. \*\*Avoid lenience bias\*\*: Don’t give full credit just because the caption is reasonable\. It must specifically follow the given instruction\.
Begin your evaluation now\.Figure 12:Instruction\-following evaluation prompt \(Part 2\): scoring rubric, edge cases, and required output format\.\{
"unseen\_prompt\_1": "Establish the speaker’s vocal identity without mentioning or inferring gender\. Use only acoustic cues to describe pitch range, timbre/texture, resonance, and any stable voice qualities; if a trait is uncertain, state it as unknown rather than guessing\.",
"unseen\_prompt\_2": "Write a speaker\-identity caption that avoids any accent or origin claims\. Focus on voice quality \(e\.g\., breathiness/creak/clarity\), pitch behavior, articulation style, and timbral color, relying strictly on what can be heard\.",
"unseen\_prompt\_3": "Output a Python\-dict\-style profile \(not JSON\) with exactly these keys: \{p̈itch\_character:̈ \.\.\., ẗimbre:̈ \.\.\., v̈ocal\_texture:̈ \.\.\., s̈tability:̈ \.\.\., n̈otes:̈ \.\.\.\}\. Fill values with short phrases grounded in the audio; use null for anything you cannot justify\.",
"unseen\_prompt\_4": "Provide two competing hypotheses for the speaker’s vocal identity \(Hypothesis A vs B\) based only on acoustics, then pick the better one with a confidence score from 0 to 1\. Do not mention language, name, or demographics\.",
"unseen\_prompt\_5": "Describe the speaker as a casting director evaluating a voice actor for a role\. Keep it strictly evidence\-based: every descriptive claim must correspond to an audible cue \(pitch movement, timbre, articulation, resonance\)\.",
"unseen\_prompt\_6": "Produce a single sentence that captures the speaker’s ‘voiceprint’ using neutral phrasing and no demographic attributes\. The sentence must include exactly one timbre descriptor, one pitch descriptor, and one articulation descriptor\.",
"unseen\_prompt\_7": "Infer the recording context \(environment \+ microphone distance\) and present it as: \(1\) best\-guess setting, \(2\) 2\-3 acoustic cues supporting it, \(3\) an uncertainty note\. Do not mention emotion or speaker identity\.",
"unseen\_prompt\_8": "Write a ‘field recording log’ in three bullet points: Background \(noise type \+ stability\), Channel \(reverb/roominess \+ clipping/cleanliness\), Distance \(near/far \+ evidence\)\. Avoid guessing location names\.",
"unseen\_prompt\_9": "Return a compact YAML block with keys: setting\_guess, noise\_profile, reverberation, mic\_distance, confidence\. Values must be short, and confidence must be one of: low/medium/high\.",
"unseen\_prompt\_10": "Describe how the environment changes over time across the utterance \(start → middle → end\)\. Only reference audible changes \(noise bursts, reverb shifts, level changes\); do not hallucinate events\.",
"unseen\_prompt\_11": "As an audio engineer, diagnose whether the speech sounds studio\-like or in\-the\-wild and justify your conclusion with at least three acoustic indicators\. If evidence is mixed, explicitly say so\.",
"unseen\_prompt\_12": "Give a one\-sentence context caption that must avoid these words: room, background, noise, microphone\. You may still describe context using alternative phrasing grounded in acoustics\.",
"unseen\_prompt\_13": "Describe the affect using only valence \(1\-7\) and arousal \(1\-7\), plus short acoustic evidence \(pitch dynamics, intensity, tempo, voice quality\)\. Do not use categorical emotion labels\.",
"unseen\_prompt\_14": "Return a strict JSON object with exactly these keys: \{v̈alence:̈1\-7, ärousal:̈1\-7, c̈onfidence:̈0\-1, ëvidence:̈\[\.\.\.\]\}\.",
"unseen\_prompt\_15": "Write two sentences: Sentence 1 must be purely objective \(acoustic description\)\. Sentence 2 may interpret the likely feeling state but must include a caution clause about uncertainty\.",
"unseen\_prompt\_16": "Describe the emotional trajectory across the utterance and name the specific prosodic features that signal each phase\. Avoid any mention of the textual content\.",
"unseen\_prompt\_17": "As a clinician writing a neutral intake note, summarize the speaker’s affective presentation using cautious language and explicit acoustic grounding; do not overclaim\.",
"unseen\_prompt\_18": "Generate a caption that includes exactly one metaphor for the affect, immediately followed by concrete acoustic justification\.",
"unseen\_prompt\_19": "Infer the likely communicative intent from prosody alone and provide 2\-3 prosodic cues as justification\.",
"unseen\_prompt\_20": "Output a ranked list of up to three pragmatic functions with probability estimates that sum to 1\.0\.",
"unseen\_prompt\_21": "Write as a debate judge assessing delivery: identify confidence vs hesitation and strategic emphasis\.",
"unseen\_prompt\_22": "Produce a minimal\-pair analysis contrasting the most likely intent with a plausible alternative\.",
"unseen\_prompt\_23": "Return a markdown table: Prosodic cue \| What it signals \| Confidence \(low/med/high\)\.",
"unseen\_prompt\_24": "Create a single\-sentence pragmatic caption that must not include any pitch words\.",
"unseen\_prompt\_25": "Rate warmth, urgency, and confidence \(1\-5\) with one acoustic justification per scale\.",
"unseen\_prompt\_26": "As a call\-center QA evaluator, describe perceived clarity and assertiveness, separating observation from interpretation\.",
"unseen\_prompt\_27": "Write a listener\-centric caption focusing only on intelligibility and comfort\.",
"unseen\_prompt\_28": "Return a strict JSON object with keys: impression, accessibility\_risks, most\_salient\_cue, confidence\.",
"unseen\_prompt\_29": "Advise a TTS system how to match the perceived presence and approachability using audible cues only\.",
"unseen\_prompt\_30": "Give two captions: one for a casual listener and one for a trained listener with technical descriptors\.",
"unseen\_prompt\_31": "Write director’s notes for recreating this performance, grounded strictly in acoustics\.",
"unseen\_prompt\_32": "Produce two labeled lines: FACT and IMAGERY, ensuring IMAGERY does not contradict FACT\.",
"unseen\_prompt\_33": "Return a nested JSON object summarizing speaker, delivery, and context; use null for unknowns\.",
"unseen\_prompt\_34": "Write a sports\-commentator\-style description justified by explicit acoustic cues\.",
"unseen\_prompt\_35": "Create a single sentence of exactly 18 words integrating identity, delivery, and context\.",
"unseen\_prompt\_36": "Write a three\-bullet spec sheet \(Identity / Delivery / Context\) with acoustic justification\."
\}
Figure 13:Unseen instruction prompts used to probe diverse instruction\-following behaviors for speaking\-style captioning\.
## Appendix IModel Training Details

Table 19:Specification and sources of the employed models\.#### Training and inference details\.

We fine\-tune the SALMONN variants using LoRA to control adaptation capacity and limit task\-specific collapse\. We set the LoRA rank tor=8r\{=\}8throughout\. During training, we use a stronger scaling \(α=32\\alpha\{=\}32\) to enable rapid alignment to the instruction\-following format and the speech\-conditioning interface; at inference, we reduce the scaling toα=16\\alpha\{=\}16\. This inference\-time attenuation acts as a practical “de\-overfitting” knob: it dampens overly task\-specialized update directions while preserving the base LLM’s general generation ability, improving robustness to prompt and domain shifts\. Empirically, this reduces random stylistic drift \(e\.g\., ungrounded claims about affect or speaker traits\) and encourages generations to remain anchored to the acoustic evidence rather than the most frequent training\-task patterns\.

#### Instruction\-tuning regimes and batch mixing\.

We study two instruction\-tuning regimes\.Single\-tasktuning introduces one new task, which isolates the effect of task supervision but also makes memorization easier\.Multi\-tasktuning introduces six new tasks, where tasks are sampled uniformly at random within each mini\-batch\. This stochastic task mixture increases gradient diversity and discourages the model from specializing to any one instruction template\. We additionally explored structured batch sampling strategies \(e\.g\., round\-robin task cycling and scheduled curricula\), but these did not improve performance and in several cases degraded it, likely because deterministic mixing reduces the regularizing effect of interleaving and amplifies short\-horizon task dominance within contiguous training windows\.

#### Scaling data and diagnosing task overfitting\.

Training on the PSC\-base human\-annotated subset \(282 hours\) yields a consistent failure mode: \(i\) the model overfits to the limited set of training tasks, \(ii\) it produces less expressive captions \(tending toward safe, generic phrasing\), and \(iii\) it exhibits catastrophic forgetting, struggling to answer open\-set questions that were not explicitly present in the tuning set\. We interpret this as a distributional “compression” effect: with limited diversity, the model learns a narrow mapping from acoustics to a small set of instruction\-conditioned outputs, trading off nuanced style description for high\-confidence task compliance\. Increasing scale mitigates these issues: training on 100% of PSC and expanding instruction diversity to roughly∼\\sim300 prompts per task substantially improves caption quality and stylistic richness\. However, the underlying task\-overfitting tendency persists, suggesting that capacity control and objective design \(not only data volume\) are necessary to fully address specialization\.

#### Alternative learning strategies for style grounding\.

To improve grounding and reduce spurious style attributions, we investigate two complementary strategies\. \(A\)Auxiliary\-task inclusion:we incorporate automatic speech recognition \(ASR\), gender recognition \(GR\), and accent recognition \(AR\) as additional supervised objectives\. These tasks encourage the shared representation to encode linguistically and paralinguistically salient cues, acting as an inductive bias toward acoustically\-verifiable factors\. \(B\)Factor\-conditioned captioning:we modify the training target to first predict explicit factors \(e\.g\.,gender,accent,pitch,speaking rate\) followed by the free\-form speaking\-style caption\. By forcing the model to commit to concrete attributes before generating the descriptive caption, we aim to reduce shortcut learning and improve causal alignment between acoustics and the final narrative description\.

#### Transcript as linguistic context\.

We provide the transcript as linguistic context for speaking\-style captioning\. This yields a complementary perspective: it allows the model to separate*what*was said \(lexical content\) from*how*it was said \(prosody, voice quality, pacing\), and to interpret paralinguistic signals relative to discourse intent \(e\.g\., emphasis, question intonation, sarcasm markers\)\. To our knowledge, explicitly incorporating the transcript in this way has not been standard in prior speaking\-style captioning setups, and it improves the model’s ability to generate contextually appropriate style descriptions rather than purely acoustic correlates\.

#### Decoding\.

At inference time, we use default decoding settings for all models unless otherwise noted\. For SALMONN, we use deterministic beam search to prioritize stability and reduce sampling\-induced variance:max\_new\_tokens=200,num\_beams=4,do\_sample=False,min\_length=1,temperature=1\.0,top\_p=0\.9,repetition\_penalty=1\.0,length\_penalty=1\.0\.

## Appendix JLexical Diversity

Caption setHD\-DSelf\-BLEUDistinct\-2Distinct\-3Mean captionlengthSTD ofcaption lengthsOracle caption \(PSC\)0\.732—0\.04250\.079532\.57\.97Oracle caption \(SIC\)0\.7710\.3040\.1520\.31034\.912\.5Audio\-Flamingo\-30\.7480\.08810\.1380\.24831\.823\.0Voxtral\-Mini\-3B0\.8130\.1230\.2470\.46314578\.1Voxtral\-Small\-24B0\.8070\.1450\.2810\.48472\.056\.8Qwen2\-Audio\-7B\-Instruct0\.8360\.03620\.2310\.43986\.259\.5MERaLiON\-AudioLLM0\.7820\.08260\.2590\.42929\.921\.6MOSS\-Audio0\.8360\.04780\.3420\.63156\.626\.4Qwen3\-omni0\.8560\.06410\.3430\.61013492\.0Audio\-Flamingo\-next0\.8850\.1280\.3300\.50146\.844\.4SALMONN0\.7950\.1130\.2210\.39784\.332\.4SALMONN \(PSC\)0\.7350\.3610\.09060\.19566\.418\.9SALMONN \(SIC\-DB\) \- ST0\.7860\.3150\.1350\.27172\.815\.4SALMONN \(SIC\-DB\) \- MT0\.7760\.2060\.1370\.26652\.818\.0Table 20:Lexical diversity and caption\-length statistics across different caption sets and models\.From the lexical diversity analysis in Table[20](https://arxiv.org/html/2610.33036#A10.T20), we find that task\-specific fine\-tuning generally reduces linguistic diversity, which is expected since the model is optimized to consistently express a constrained set of speaking\-style attributes rather than maximize unconstrained lexical variation\. Compared with single\-task training, multi\-task training improves faithfulness and reduces hallucinations \(Table[1](https://arxiv.org/html/2610.33036#S4.T1)\), but its effect on diversity is mixed: it clearly reduces repetition across captions \(Self\-BLEU 0\.206 vs\. 0\.315\), whereas HD\-D and Distinct\-3 are slightly lower and Distinct\-2 is essentially unchanged\. A larger LLM backbone also does not consistently increase diversity\. From Voxtral\-Mini\-3B to Voxtral\-Small\-24B, Distinct\-2 and Distinct\-3 increase, but HD\-D decreases slightly and Self\-BLEU rises, indicating more repetition\. We additionally observe that the oracle captions in StyleInstructCaps are considerably more diverse than those in ParaSpeechCaps\. Models trained on StyleInstructCaps are not uniformly more diverse than the caption set itself\. Both SALMONN \(SIC\-DB\) variants exceed the oracle captions on HD\-D, and the MT variant has lower Self\-BLEU, but both have lower Distinct\-2 and Distinct\-3\.

## Appendix KLLM Judgement

LLM judgementsimilarity\_score:7justification:The generated caption matches the ground truth in gender, accent, pitch, and speaking rate\. It captures the emotional tone of discomfort and frustration, though it’s slightly less specific than the ground truth’s pained emotion\. The generated caption introduces a nasal voice quality not mentioned in the ground truth, which is acceptable as per instructions\. The environment/noise level is missing, which is a notable difference\.Ground truth metadata\{ "source": "ears", "relative\_audio\_path": "p019/emo\_pain\_freeform\.wav", "text\_description": \[ " A male speaker with an American accent delivers a speech that is deep and high\-pitched, characterized by crisp, husky tones\. His voice is measured in speed, punctuated with slight pauses, conveying a sense of pained emotion\. The speech was recorded in a slightly noisy environment, adding a flowing, textured quality to the overall sound\.", " A male speaker with a high\-pitched voice delivers pained expressions in an American accent, speaking at a measured speed in a slightly noisy environment\." \], "transcription": " Ow, god damn it, that hurts\. Damn, I wish I hadn’t stubbed my toe on the corner of that table\. Oh, that’s kind of smart\. Oh, boy, I hate that so much\. Why did it have to be the pinky toe as well? Ow\.", "intrinsic\_tags": \["crisp", "deep", "american", "flowing", "husky", "punctuated"\], "situational\_tags": \["pained"\], "basic\_tags": \["high\-pitched", "male", "measured speed", "slightly noisy environment"\], "all\_tags": \["american", "crisp", "deep", "flowing", "high\-pitched", "husky", "male", "measured speed", "pained", "punctuated", "slightly noisy environment"\], "speakerid": "p019", "name": "p019", "duration": 16\.1, "gender": "male", "accent": "american", "pitch": "high\-pitched", "speaking\_rate": "measured speed", "noise": "slightly noisy environment", "utterance\_pitch\_mean": 159\.45840454101562, "snr": 38\.82289123535156, "holistic\_creative\_synthesis": "A high\-pitched, crisp, and husky American male voice speaks in a slightly noisy environment, conveying a sense of pained emotion through a measured pace and deliberate pauses, the speaker’s words punctuated by a rhythmic flow that adds to the overall sense of conviction, creating a sense of depth and nuance that draws the listener in\.", "holistic\_creative\_synthesis\_instruction": "Combine all elements of the speaker’s style, including their accent, pitch, timbre, and delivery, to create a rich, narrative\-style caption that captures the essence of their speaking\-style\." \}Generated captionGenerated caption: The speaker’s style is characterized by a low, guttural tone, a slow and deliberate pace, and a deep, resonant timbre\. Their accent is distinct and evocative, with a strong emphasis on certain syllables and a distinctive rhythm\. The delivery is confident and assertive, with a sense of authority and expertise\. Overall, the speaker’s style is engaging and captivating, drawing the listener in with their unique voice and delivery\.

Figure 14:Example showing an LLM judgment \(top\) and the corresponding ground\-truth metadata vs\. generated caption \(bottom\)\.
## Appendix LSpeaker Caption clustering plots

![[Uncaptioned image]](https://arxiv.org/html/2610.33036v2/tsne_all_speakers_doublecol_12.png)

Figure 15:t\-SNE visualization of speaker\-style caption embeddings on UD\-VCTK \(all speakers\)
## Appendix MUse of AI assistants

We used GPT\-5 and Gemini 3 Flash for code assistance, and the same models purely for language\-related assistance in writing the paper\.

## Appendix NLicensing

All resources used in this work are released under open licenses\. We use the VCTK corpus, which is distributed under the Open Data Commons Attribution License \(ODC\-By\) v1\.0, requiring attribution\. We also use the GigaSpeech dataset, released under the Apache License 2\.0\. ParaSpeechCaps is provided under the CC BY\-NC\-SA 4\.0 license, which restricts commercial use and requires attribution and share\-alike for derivatives\. Finally, the SALMONN codebase is released under the Apache License 2\.0\.