HEAR Who Said What: Unlocking Speaker-Attributed Reasoning via Counterfactual Voice Grounding

# HEAR Who Said What: Unlocking Speaker-Attributed Reasoning via Counterfactual Voice Grounding
Source: [https://arxiv.org/html/2608.29120](https://arxiv.org/html/2608.29120)
Sangkwon ParkAffiliation:Department of ECE, SNUEunwoo SongAffiliation:Department of EE, Yonsei UniversityChe Hyun LeeAffiliation:Department of ECE, SNUYoungho ChoJunho KimJune Young YiHeeseung Kim222Corresponding authors\.Sungroh Yoon222Corresponding authors\.Affiliation:IPAI, Seoul National University \(SNU\)Affiliation:Department of ECE, SNUAffiliation:Department of CSE, SNUAffiliation:Department of AI, University of SeoulAffiliation:AIIS, ASRI, INMC, and ISRC, SNU

###### Abstract

Speech Language Models \(SLMs\) are increasingly deployed in multi\-speaker environments, yet their ability to attribute speech to the correct speaker and reason over speaker identities remains unclear\. Hence, we introduceHEAR, a conceptually hierarchical benchmark diagnosing the foundational capabilities of speaker\-attributed reasoning, comprising 2\.4K human\-verified samples from 887 diverse multi\-party audio clips\. Evaluating 20 leading SLMs on HEAR reveals they struggle with these foundational tasks, often relying on semantic priors rather than actual vocal cues\. To address this, we presentA2R, a 30B model optimized onCounterfactual Audio with Speaker\-level Hard negatives\(CASH\), a dataset designed to guide the model to prioritize acoustic vocal cues over linguistic signals\. A2R achieves strong performance on HEAR and exhibits zero\-shot generalization to diverse multi\-speaker downstream tasks, demonstrating that learned speaker attribution unlocks the model’s latent capacity for speaker\-aware reasoning\. All resources are available on our[project page](https://attributetoreason.github.io/AttributeToReason/)\.

22footnotetext:Corresponding authors\.## 1Introduction

Understanding speech in real\-world environments often requires handling multi\-speaker scenarios, where multiple voices are present and interact dynamically\([Lee et al\., 2026](https://arxiv.org/html/2608.29120#bib.bib1)\)\. Identifying who said what, a task commonly referred to as speaker attribution[Kanda et al\. \(2021\)](https://arxiv.org/html/2608.29120#bib.bib42), is a fundamental prerequisite for any higher\-level understanding or reasoning in multi\-party scenarios\. This challenge becomes critical as speech language models \(SLMs\) are increasingly deployed in multi\-party environments\([Nguyen et al\., 2025](https://arxiv.org/html/2608.29120#bib.bib41)\)\. However, prior works[Kumar et al\. \(2025\)](https://arxiv.org/html/2608.29120#bib.bib5);[Sakshi et al\. \(2024\)](https://arxiv.org/html/2608.29120#bib.bib4)often evaluate speaker attribution only through its downstream tasks, without directly testing the foundational abilities required to support it\. This makes it difficult to localize where models actually fail: whether they struggle to discriminate between distinct voices, attribute utterances to the correct speakers, or simply lack the reasoning competency for the downstream task\.

To address this gap, we introduce a benchmark,HEAR, designed to evaluate the foundational abilities underlying speaker\-attributed reasoning in SLMs\. HEAR is grounded in Erber’s Auditory Hierarchy[Erber \(1982\)](https://arxiv.org/html/2608.29120#bib.bib2), which organizes human auditory understanding as a progression from perceptual discrimination to identification and higher\-level comprehension\. It maps Erber’s view into three dimensions:\(i\) Discrimination, the ability to distinguish distinct voice identities;\(ii\) Attribution, the capacity to bind linguistic content to the exact voice identity; and\(iii\) Reasoning, the capacity to perform basic reasoning over speaker\-attributed utterances\. By mirroring this human cognitive progression, we provide a biologically inspired taxonomy to diagnose whether SLMs’ failures stem from low\-level acoustic perception or high\-level semantic reasoning\. The benchmark consists of 887 real\-world audio clips and 2\.4K human\-verified multiple\-choice question\-answer pairs, covering diverse real\-world acoustic settings such as daily conversations, meetings, and documentaries\.

Our evaluations on the HEAR benchmark expose vulnerabilities in most leading SLMs\. These models fail at basic attribution and exhibit an over\-reliance on semantic priors, ignoring acoustic vocal cues when semantic context is available\. To mitigate this semantic over\-reliance, we introduceCounterfactual Audio with Speaker\-level Hard negatives\(CASH\), a 60K\-scale corpus spanning all task dimensions of HEAR, designed to decouple acoustic identity from semantic content\. By using voice cloning to swap speakers for specific utterances while preserving the original transcript \(Figure[2](https://arxiv.org/html/2608.29120#S2.F2)\-\(A\)\), CASH yields hard negatives where the correct answer flips solely based on vocal cues, thereby compelling models to ground their predictions in acoustic evidence rather than textual priors\.

Building on this dataset, we presentA2R\(Attribution\-to\-Reasoning\), a speech language model optimized from Qwen3\-Omni\-30B\-A3B\-Instruct[Xu et al\. \(2025b\)](https://arxiv.org/html/2608.29120#bib.bib19)via Group Relative Policy Optimization \(GRPO\)[Shao et al\. \(2024\)](https://arxiv.org/html/2608.29120#bib.bib3)to strengthen speaker attribution while preserving the base model’s strong inherent reasoning ability\. It is trained under a transcription\-first objective: it first produces a speaker\-tagged transcript and then reasons over it\.

Consequently, A2R not only excels on HEAR but also demonstrates robust zero\-shot transfer across various downstream multi\-speaker benchmarks that require speaker attribution\. These results suggest that speaker attribution is not merely an auxiliary sub\-task, but a key mechanism for resolving the acoustic bottleneck in multi\-party auditory comprehension\. By explicitly forcing the model to ground utterances in vocal cues, we show that speaker\-attributed reasoning can be elicited from the model in multi\-speaker environments\.

Our contributions can be summarized as follows:\(1\)We presentHEAR, a benchmark comprising 2\.4K human\-annotated samples, designed to evaluate the foundational capabilities for speaker\-attributed reasoning\.\(2\)We reveal that current leading SLMs exhibit*semantic hallucination*in speaker\-attributed reasoning, exploiting language priors rather than the voice cues that actually determine who said what\.\(3\)We proposeCounterfactual Audio with Speaker\-level Hard negatives\(CASH\), a 60K\-scale dataset designed to guide models to prioritize vocal cues over semantic content\.\(4\)We introduceA2R, a 30B model trained with a transcription\-first objective that rewards accurate transcriptions before reasoning, leading to strong improvements on HEAR and transfer gains on benchmarks requiring speaker attribution\.

![Refer to caption](https://arxiv.org/html/2608.29120v1/figure1_a2r.png)Figure 1:Illustration of HEAR benchmark\.Inspired by Erber’s auditory hierarchy[Erber \(1982\)](https://arxiv.org/html/2608.29120#bib.bib2), HEAR breaks speaker\-attributed reasoning down into several foundational capabilities: discrimination, attribution, and reasoning\.
## 2HEAR Benchmark

In this section, we introduceHEAR, a benchmark for Hierarchical Evaluation of Attribution and Reasoning, designed to evaluate the foundational capabilities underlying speaker\-attributed reasoning in SLMs\. The benchmark is formatted as Multiple\-Choice Question Answering \(MCQA\) and organized into a hierarchy that mirrors human auditory processing[Erber \(1982\)](https://arxiv.org/html/2608.29120#bib.bib2), as illustrated in Figure[1](https://arxiv.org/html/2608.29120#S1.F1): \(i\) Discrimination, \(ii\) Attribution, and \(iii\) Reasoning\. To analyze the impact of overlapping speech, we distinguish cases based on whether the overlapping utterance segment is needed to answer the question \(Appendix[B\.3](https://arxiv.org/html/2608.29120#A2.SS3)\)\.

### 2\.1Taxonomy

#### 2\.1\.1Discrimination

This axis evaluates the capacity to discriminate different voice identities and localize their temporal boundaries\.

##### Voice Cardinality \(VC\)

This task evaluates the model’s ability to determine the total number of unique speakers present within a given audio clip\. Representing the most fundamental level of speaker\-identity clustering, VC assesses whether the model can accurately quantify the underlying voice population of an acoustic scene\.

##### Voice Localization \(VL\)

Given a reference audio snippet of a target speaker, the task is to identify the temporal intervals in which the target speaker is present or absent\. By incorporating both active speech detection and absence verification, this task evaluates target\-voice activity detection\.

##### Voice Change Detection \(VCD\)

This task requires the model to identify speaker transition points\. For non\-overlapping speech samples, the focus is on detecting turn\-taking boundaries, whereas for overlapping samples, the task involves detecting time intervals containing simultaneous speech\.

#### 2\.1\.2Attribution

This axis evaluates the ability to bind semantic content to its corresponding voice identity in both directions: text to audio and audio to text\.

##### Content\-to\-Voice Attribution \(CVA\)

Given a transcribed utterance as a textual query, the model must identify the corresponding speaker from a set of five candidate voices\.

##### Voice\-to\-Content Attribution \(VCA\)

In this task, the model is given a brief audio sample of a reference voice and must identify which of five textual candidates was spoken by that voice\.

#### 2\.1\.3Reasoning

This axis examines basic*speaker\-attributed reasoning*capabilities\. Every Reasoning item is a semantic\-hallucination pair: each original clip is paired with a counterfactual variant in which one answer\-relevant utterance is re\-spoken in another speaker’s voice, leaving the transcript unchanged while flipping the correct answer, so that a voice\-blind model necessarily fails the pair \(Section[4](https://arxiv.org/html/2608.29120#S4.SS0.SSS0.Px2)\)\.

##### Identity Reasoning \(IR\)

It evaluates whether a model can determine which utterances were spoken by the same person\. Given an anchor utterance \(text\), the model must identify another utterance \(text\) produced by the same speaker\.

##### Temporal Reasoning \(TR\)

This task assesses the model’s ability to discern sequential relationships and temporal dynamics conditioned on speaker identity\. We decompose this evaluation into two granular levels\. \(a\)*Intra\-speaker temporal ordering*requires the model to navigate a single speaker’s history to spot the first or the last utterance of the speaker\. \(b\)*Inter\-speaker temporal alignment*evaluates the ability to align timelines across multiple participants by locating a target speaker’s utterance relative to an anchor utterance from a different speaker \(e\.g\., After someone said ‘X’, what was the first thing said by the person who said ‘Y’?\)\.

##### Quantitative Reasoning \(QR\)

This task examines the model’s capability to aggregate per\-speaker utterance statistics to deduce numerical and comparative relationships\. We evaluate this through two subtasks\. \(a\)*Conditional counting*requires the model to quantify the number of distinct speakers who utter a specific word\. \(b\)*Ordinal ranking*assesses the ability to rank speakers by cumulative speaking time and identify the speaker with thekk\-th longest total speaking time \(e\.g\., “Which of the following was said by the speaker with thekk\-th longest total speaking time?”\)\.

### 2\.2Benchmark Construction

HEAR consists of approximately 2\.4K MCQA queries derived from 887 audio clips, totaling roughly 23 hours of audio\. The clips average∼\\sim93 seconds in length, with a maximum duration of 150 seconds\. We curate audio clips exclusively from the test splits of AMI[Carletta et al\. \(2005\)](https://arxiv.org/html/2608.29120#bib.bib6), ICSI[Janin et al\. \(2003\)](https://arxiv.org/html/2608.29120#bib.bib8), and VoxMM[Kwak et al\. \(2024\)](https://arxiv.org/html/2608.29120#bib.bib7), where speaker diarization and transcripts are already annotated by humans, ensuring reliable speaker\-speech alignments across various real\-world acoustic conditions such as meetings, daily conversations, broadcast media, and lectures\. Each clip contains 4\.05 speakers on average, and all samples are annotated by human annotators as overlap or non\-overlap, based on whether answering the question requires focusing on regions with overlapping speech \(Appendix[B\.3](https://arxiv.org/html/2608.29120#A2.SS3)\)\. We generate queries through a rule\-based pipeline and conduct an additional manual verification with eight annotators\. As a result, every item in HEAR benefits from two stages of human validation: first during the construction of the source datasets, and again through our additional verification process, yielding a high\-quality dataset with strong annotation reliability\.

![Refer to caption](https://arxiv.org/html/2608.29120v1/figure2_a2r.png)Figure 2:Overview of our dataset and method on speaker\-attributed reasoning\.\(A\)Our dataset, CASH, randomly reassigns utterances to different speakers via voice cloning while preserving the whole utterance, yielding counterfactual variants whose ground\-truth answer always flips\.\(B\)We show an elementary speaker\-attribution task from theWhat Do You Like?benchmark[Wu et al\. \(2024\)](https://arxiv.org/html/2608.29120#bib.bib9), where the query is unsolvable without identifying which speaker said what\.\(C\)Our model first transcribes the multi\-party audio scene, then reasons over the transcript for more reliable speaker\-aware reasoning\.

## 3Training

### 3\.1Motivation

Unimodal collapse, where a model over\-relies on a single modality, has long been a recognized issue in vision\-language tasks[Koishigarina et al\. \(2026\)](https://arxiv.org/html/2608.29120#bib.bib15), and a similar phenomenon is observed in the speech domain\. A recent study[Lee et al\. \(2026\)](https://arxiv.org/html/2608.29120#bib.bib1)highlights that SLMs often exhibit*semantic hallucinations*, bypassing acoustic cues in favor of their textual priors, which is a critical bottleneck for speaker\-attributed reasoning in multi\-party audio scenes\. Determining “who said what” based on semantic plausibility may suggest who*might*speak, but fails in dynamic conversations where any participant can make any statement, thus requiring acoustic vocal cues to be prioritized over textual context to determine who*actually*spoke\.

### 3\.2Forcing Models to Rely on Vocal Cues

To encourage models to ground speaker attribution in acoustic vocal cues rather than semantic content, we introduceCounterfactual Audio with Speaker\-level Hard negatives\(CASH\)\. CASH is a 60K\-scale dataset for multi\-party audio, spanning all task dimensions of HEAR\. Given an original multi\-speaker clip, we construct counterfactual variants by replacing a target utterance with a voice\-converted version of the*same transcript*by a different speaker, while preserving chronological structure and other speech content \(Figure[2](https://arxiv.org/html/2608.29120#S2.F2)\-\(A\)\)\. We build on the VoxMM[Kwak et al\. \(2024\)](https://arxiv.org/html/2608.29120#bib.bib7)train split, extracting∼\\sim5\.2K real\-world audio clips drawn from diverse audio scenes\. All synthetic voices are generated with VoxCPM2[Zhou et al\. \(2026\)](https://arxiv.org/html/2608.29120#bib.bib52), yielding∼\\sim20Kmixed real\-syntheticaudio clips and∼\\sim60K queries in total\. Details are provided in Appendix[A](https://arxiv.org/html/2608.29120#A1)\.

### 3\.3Rewarding Attribution Before Reasoning

We presentA2R\(Attribution\-to\-Reasoning\), a model developed by optimizing the baseline model via GRPO to enhance speaker attribution capabilities while preserving general performance\. By bridging newly acquired vocal grounding ability with its inherent inferential capacity, we unlock the model’s ability to perform speaker\-attributed reasoning\. During training, as illustrated in Figure[2](https://arxiv.org/html/2608.29120#S2.F2)\-\(C\), the model first transcribes the input audio with speaker tags and then generates both the reasoning and the final answer based on its transcript\.

We compute the reward from three aspects of the model response: the speaker\-tagged transcription \(RtrR\_\{\\mathrm\{tr\}\}\), the final answer \(RansR\_\{\\mathrm\{ans\}\}\), and the response format \(RfmtR\_\{\\mathrm\{fmt\}\}\)\. The speaker\-tagged transcription not only captures how accurately the model attributes utterances to speakers, but also serves as the basis for the reasoning needed to answer the question, whereas the final answer captures direct task\-level correctness\. Accordingly, for a model responseYiY\_\{i\}, we parse its predicted speaker\-tagged transcription and answer asτ^i\\hat\{\\tau\}\_\{i\}anda^i\\hat\{a\}\_\{i\}, and compare them against the ground\-truth transcriptionτ⋆\\tau^\{\\star\}and answera⋆a^\{\\star\}\. The overall reward is then defined as

R⁡\(Yi,τ⋆,a⋆\)\\displaystyle R\(Y\_\{i\};\\tau^\{\\star\},a^\{\\star\}\)=Rtr​\(τ^i,τ⋆\)\+λans​Rans​\(a^i,a⋆\)\+λfmt​Rfmt​\(Yi\),\\displaystyle=R\_\{\\mathrm\{tr\}\}\(\\hat\{\\tau\}\_\{i\};\\tau^\{\\star\}\)\+\\lambda\_\{\\mathrm\{ans\}\}R\_\{\\mathrm\{ans\}\}\(\\hat\{a\}\_\{i\};a^\{\\star\}\)\+\\lambda\_\{\\mathrm\{fmt\}\}R\_\{\\mathrm\{fmt\}\}\(Y\_\{i\}\),Rtr​\(τ^i,τ⋆\)\\displaystyle R\_\{\\mathrm\{tr\}\}\(\\hat\{\\tau\}\_\{i\};\\tau^\{\\star\}\)=λcp​Rcp​\(τ^i,τ⋆\)\+λcnt​Rcnt​\(τ^i,τ⋆\)\+λord​Rord​\(τ^i,τ⋆\)\\displaystyle=\\lambda\_\{\\mathrm\{cp\}\}R\_\{\\mathrm\{cp\}\}\(\\hat\{\\tau\}\_\{i\};\\tau^\{\\star\}\)\+\\lambda\_\{\\mathrm\{cnt\}\}R\_\{\\mathrm\{cnt\}\}\(\\hat\{\\tau\}\_\{i\};\\tau^\{\\star\}\)\+\\lambda\_\{\\mathrm\{ord\}\}R\_\{\\mathrm\{ord\}\}\(\\hat\{\\tau\}\_\{i\};\\tau^\{\\star\}\)
Here, the transcription rewardRtrR\_\{\\mathrm\{tr\}\}is further decomposed into three terms: a cpWER\-based[von Neumann et al\. \(2024\)](https://arxiv.org/html/2608.29120#bib.bib24)attribution reward \(RcpR\_\{\\mathrm\{cp\}\}\), which measures whether the model correctly captures what was said by whom; a speaker\-count reward \(RcntR\_\{\\mathrm\{cnt\}\}\), which evaluates whether the predicted transcription contains the correct number of speakers; and a speaker\-order reward \(RordR\_\{\\mathrm\{ord\}\}\), which assesses whether the temporal sequence of speaker turns is preserved \(Appendix[E](https://arxiv.org/html/2608.29120#A5)\)\.

During training, we sampleGGresponses per inputxx, compute rewards, and normalize them as group\-relative advantagesAi=\(Ri−μR\)/\(σR\+δ\)A\_\{i\}=\(R\_\{i\}\-\\mu\_\{R\}\)/\(\\sigma\_\{R\}\+\\delta\), whereμR\\mu\_\{R\}andσR\\sigma\_\{R\}are the group reward mean and standard deviation\. We then optimize the current policyπθ\\pi\_\{\\theta\}using the objective:

ℒGRPO\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{GRPO\}\}=−1G∑i=1Gmin\(riAi,r¯iAi\),\\displaystyle=\-\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\min\\\!\\left\(r\_\{i\}A\_\{i\},\\bar\{r\}\_\{i\}A\_\{i\}\\right\),r¯i\\displaystyle\\bar\{r\}\_\{i\}=clip⁡\(ri,1−ϵlow,1\+ϵhigh\),\\displaystyle=\\mathrm\{clip\}\(r\_\{i\},1\-\\epsilon\_\{\\mathrm\{low\}\},1\+\\epsilon\_\{\\mathrm\{high\}\}\),whereri=πθ​\(Yi∣x\)/πθold​\(Yi∣x\)r\_\{i\}=\\pi\_\{\\theta\}\(Y\_\{i\}\\mid x\)/\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(Y\_\{i\}\\mid x\), withπθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}denoting the old policy\.

Table 1:Evaluation results of 20 leading Speech Language Models on the HEAR benchmark\.Values in parentheses in the Reasoning columns report scores under the paired setting, where a prediction is counted as correct only when both the original audio clip and its corresponding semantic hallucination counterpart are correct\. A2R results are averaged over three seeds\.

## 4Experiments

We design our experiments to answer the following three questions: \(i\) How do current leading SLMs perform on HEAR, and do they truly exploit acoustic speaker cues rather than relying on semantic priors for speaker\-attributed reasoning? \(ii\) Do the resulting gains transfer to other multi\-party audio understanding benchmarks where speaker attribution is essential, while preserving performance in dyadic conversational settings? \(iii\) How much does CASH\-60K improve speaker\-attributed reasoning, and what are the contributions of speaker\-level hard negatives and training method?

##### Baseline Models\.

We evaluate 20 leading SLMs, covering both open\-source and proprietary models\. We assess \(a\) omni\-modality models, \(b\) speech/audio language models, and \(c\) proprietary models as closed\-source baselines\. The evaluated models and their inference hyperparameters are summarized in Appendix[D](https://arxiv.org/html/2608.29120#A4)and Table[7](https://arxiv.org/html/2608.29120#A10.T7)\. Among these models, we choose Qwen3\-Omni\-30B\-A3B\-Instruct as our base model for developing speaker\-attributed reasoning capabilities, as its demonstrated zero\-shot capability on HEAR indicates a promising starting point \(Table[1](https://arxiv.org/html/2608.29120#S3.T1)\)\. We train LoRA adapters[Hu et al\. \(2021\)](https://arxiv.org/html/2608.29120#bib.bib16)withr=64r=64on NVIDIA H200 GPUs, while freezing the audio encoders and modality aligners\. Details are provided in Appendix[F](https://arxiv.org/html/2608.29120#A6)\.

##### Probing Semantic Hallucination\.

As introduced in Section[2\.1\.3](https://arxiv.org/html/2608.29120#S2.SS1.SSS3), each “Reasoning” pair consists of\(x,x′\)\(x,x^\{\\prime\}\), wherex′x^\{\\prime\}is a CASH\-style voice\-swapped variant of audio clipxx\. Pair construction enforces five constraints simultaneously: \(1\) the transcript and temporal structure ofx′x^\{\\prime\}are preserved relative toxx; \(2\) the speaker similarity[Desplanques et al\. \(2020\)](https://arxiv.org/html/2608.29120#bib.bib20)between the swapped target utterance and its reference voice exceeds0\.70\.7; \(3\) the WER between the Whisper\-large\-v3[Radford et al\. \(2022\)](https://arxiv.org/html/2608.29120#bib.bib18)transcription of the synthesized utterance and the reference transcript is00, ensuring linguistic content is preserved; \(4\) the answer is required to be different betweenxxandx′x^\{\\prime\}, so that any model that ignores acoustic identity would necessarily answer identically on both, thereby failing the question; and \(5\)\(x,x′\)\(x,x^\{\\prime\}\)pairs are retained only if human annotators verify that the synthesized utterance matches the intended target speaker\. Target utterances are synthesized with IndexTTS 2[Zhou et al\. \(2025\)](https://arxiv.org/html/2608.29120#bib.bib33), a different synthesizer from the one employed in CASH to avoid synthesizer\-specific artifacts\. Each pair is independently checked by expert annotators for acoustic naturalness, transcript fidelity, and answer\-flip validity; pairs failing any criterion are discarded \(Appendix[B](https://arxiv.org/html/2608.29120#A2)\)\.

##### Zero\-Shot Transfer and Dyadic Evaluation\.

To evaluate zero\-shot transfer beyond HEAR, we consider three held\-out benchmarks that require speaker attribution under diverse task settings\.

TheWhat Do You Like? \(WDYL\)[Wu et al\. \(2024\)](https://arxiv.org/html/2608.29120#bib.bib9)tests whether a model can associate spoken content with the correct speaker\. We focus on its Identity\-Critical Questions \(ICQ\) category, where two speakers describe their preferences and the model must determine whose preference is queried \(Figure[2](https://arxiv.org/html/2608.29120#S2.F2)\-\(B\)\)\. We synthesize the last query sentence using a reference voice identical to one of the speakers\. The model must therefore identify the queried speaker from vocal characteristics rather than relying on semantic content alone\.

We additionally evaluate onGaokao[Hu et al\. \(2024\)](https://arxiv.org/html/2608.29120#bib.bib54), a multi\-turn dialogue QA benchmark derived from English listening examinations for the Chinese college entrance exam\. To introduce an explicit speaker\-attribution requirement, we synthesize the final query using the voice of a randomly selected character from the preceding dialogue, such that answering correctly requires associating the query voice with that character’s earlier utterances\. We further filter the examples using GPT\-5\.5[OpenAI \(2026a\)](https://arxiv.org/html/2608.29120#bib.bib50)to retain only questions that cannot be solved without speaker attribution\.

We also utilize theFind the Spy \(FTS\), social deduction game[Xu et al\. \(2025c\)](https://arxiv.org/html/2608.29120#bib.bib21)\. In this task, 3\-6 players introduce themselves and describe their assigned word, while one spy receives a different word\. The players take turns giving clues about their word, and the group \(model\) must vote to identify the spy \(Figure[3](https://arxiv.org/html/2608.29120#S4.F3)\)\. Solving the task requires speaker attribution to track which player provided each description, as well as higher\-level reasoning to identify the spy whose description is semantically inconsistent with the others\.

Apart from the main audio already provided in the original benchmark, the text query segments are synthesized using a TTS model[Zhou et al\. \(2025\)](https://arxiv.org/html/2608.29120#bib.bib33), with speaker similarity constrained to be over 0\.7 \(Appendix[C](https://arxiv.org/html/2608.29120#A3)\)\.

Lastly, in order to assess whether our model preserves dyadic spoken\-language ability, we further evaluate on VoiceBench[Chen et al\. \(2024\)](https://arxiv.org/html/2608.29120#bib.bib32), which covers eight benchmarks in dyadic conversational settings \(Appendix[G](https://arxiv.org/html/2608.29120#A7)\)\.

![Refer to caption](https://arxiv.org/html/2608.29120v1/FTS.png)Figure 3:Illustration of Find the Spy, a benchmark where speaker attribution is essential\.Each player receives a word \(e\.g\.,Banana\), except for one spy who receives a different word \(e\.g\.,Apple\), and players describe their words in a random speaking order\. To identify the spy, a model must do more than recognize which clue is inconsistent with the others: it must also determine*who produced that clue*\.

## 5Results & Analysis

##### Limitations of Leading SLMs on HEAR\.

Across discrimination and attribution dimensions, most open\-source models perform close to random chance, suggesting that current speech\-language training does not reliably endow models with the basic capacity to segregate voices and localize them\. As illustrated in Figure[4](https://arxiv.org/html/2608.29120#S5.F4), this difficulty is exacerbated in scenarios with overlapping speech; performance consistently degrades across six leading baselines\. While several recent open\-source models achieve modest improvements over chance, the closed\-source Gemini series performs substantially better, indicating that the core skills evaluated by HEAR are attainable but remain underdeveloped in current open\-source SLMs\. Notably, A2R \(Ours\) largely mitigates this limitation, bringing performance substantially similar to Gemini Flash and significantly narrowing the gap between open\- and closed\-source models\.

Figure 4:Performance degradation under overlap samples and the effect of semantic hallucination\.
##### Overestimated Reasoning Performance Due to Linguistic Priors\.

Open\-source models achieve strong reasoning scores, yet their performance drops sharply under the paired accuracy metric reported in parentheses in Table[1](https://arxiv.org/html/2608.29120#S3.T1)\. The parenthesized score requires a model to answer both an original clip and its semantic\-hallucination counterpart correctly \(Appendix[H](https://arxiv.org/html/2608.29120#A8)\)\. The same pattern also appears in proprietary models, including the Gemini models, suggesting that their reasoning also partly relies on semantic cues rather than voice\-grounded inference\.

##### Transfers to Unseen Attribution\-Critical Tasks\.

As shown in Table[2](https://arxiv.org/html/2608.29120#S5.T2), even on elementary tasks such asWhat Do You Like?, the Qwen3 series collapses to near\-random performance, reflecting its lack of speaker attribution ability\. In contrast, A2R shows substantial gains of \+36\.4 \(WDYL\), \+22\.6 \(GAOKAO\), and \+19\.0 \(FTS\) percentage points\. These results suggest that learning speaker attribution provides a transferable inductive bias, enabling the model to generalize speaker\-attributed reasoning to unseen tasks without task\-specific training\.

Table 2:Generalization across three benchmarks where speaker attribution is essential\.“HN” denotes voice\-swapped hard negative variants in CASH\.
##### Transfers to Real\-World Human Voices\.

To examine whether A2R’s gains transfer to real\-world human voices, we randomly sample 100 conversations used in the unseen\-distribution evaluation above \(50 WDYL, 30 GAOKAO, and 20 FTS\) and collect corresponding human recordings for each instance\. Specifically, five participants recorded the same conversational transcripts underlying the original synthesized inputs, with speakers randomly assigned to conversational roles\. As shown in Table[3](https://arxiv.org/html/2608.29120#S5.T3), A2R’s substantial improvements over the base model persist when synthesized speech is replaced by human recordings: on WDYL, GAOKAO, and FTS, A2R improves over the base model by \+48\.0, \+20\.0, and \+30\.0 percentage points, respectively\. These results provide evidence that the attribution capability learned from CASH transfers to real\-world human voices beyond synthesized speech\.

Table 3:Evaluation on human\-recording attribution\-critical tasks\.We compare the 100 synthetic samples with corresponding recordings produced by human speakers\.
##### Ablation Study\.

To isolate the contribution of each component, we conduct ablations along three axes: \(i\) the inclusion of speaker\-level hard negatives in CASH; \(ii\) reward shaping, where the model is prompted to reason but is not required to follow our structured reasoning trace, with rewards assigned only to the final answer; and \(iii\) the training regime, where the baseline model is trained with LoRA\-based supervised fine\-tuning \(SFT\)\. As shown in Figure[5](https://arxiv.org/html/2608.29120#S5.F5), removing the hard negatives from CASH substantially reduces HEAR reasoning pair accuracy from 60\.0 to 43\.0 \(for one seed\) and also leads to a higher same\-answer rate\. These results suggest that the HEAR\-axis data itself promotes acoustic sensitivity, while the hard\-negative structure further strengthens this capability by encouraging the model to rely on speaker\-specific voice cues\. Importantly, the benefits of this structure extend beyond HEAR: as shown in Table[2](https://arxiv.org/html/2608.29120#S5.T2), incorporating hard negatives also improves performance on zero\-shot benchmarks\. For the second axis, removing the structured reasoning\-trace constraint and rewarding only the final answer similarly degrades overall HEAR performance and pair consistency, as shown in Figure[6](https://arxiv.org/html/2608.29120#S5.F6)\. Finally, replacing our training objective with LoRA\-based SFT yields weaker results not only on HEAR but also across most VoiceBench settings, as shown in Table[4](https://arxiv.org/html/2608.29120#S5.T4)\. All ablation models are trained for one epoch and evaluated after validation performance has stabilized\.

Figure 5:Results on semantic hallucination pairs in the “Reasoning” category of HEAR\.“HN” denotes voice\-swapped hard negative variants in CASH\. Lower same\-answer rates indicate higher acoustic sensitivity, as each pair is designed to elicit different answers\.Figure 6:Ablation of training regimes on HEAR\.
##### Minimal Trade\-off on General Dyadic Tasks\.

As shown in Table[4](https://arxiv.org/html/2608.29120#S5.T4), A2R achieves performance broadly on par with the baseline across eight general dyadic benchmarks, demonstrating that speaker\-attributed reasoning can be acquired with only a marginal impact on general capability\.

Table 4:Evaluation on VoiceBench\.We assess performance drop under dyadic conversational settings\.
##### Prompt Analysis\.

To assess A2R’s learned speaker attribution capability independent of transcript\-based reasoning, we remove the reasoning trace at inference time\. Figure[7](https://arxiv.org/html/2608.29120#S5.F7)comparesReasoning w/ Transcript, which explicitly generates a transcript before reasoning, with a less structuredReasoningprompt that elicits only a brief rationale\. While transcript\-first reasoning provides an additional benefit, A2R continues to outperform the baselines on the majority of benchmarks even without this scaffold\. This suggests that A2R has internalized speaker attribution rather than merely learning to exploit a particular prompting strategy\.

Figure 7:Prompt Analysis on benchmarks\.“Reasoning” denotes a prompt that asks the model to provide brief reasoning before answering, while “Reasoning w/ transcript” asks for a transcript before answering\.
##### Why A2R Works\.

In our answer\-only ablation experiments, we observed the model relying on subjective adjectives \(e\.g\., “high pitch”, “documentary\-like voice”\) to track speakers\. This mapping from continuous acoustic signals to descriptive text can induce identity hallucination\. While merely prompting baselines to use structural identifiers \(e\.g\., “Speaker 1:”\) does not help \(Figure[7](https://arxiv.org/html/2608.29120#S5.F7)\), explicitly reinforcing a 1:1 alignment between distinct voices and these tags establishes a reliable anchor for the model’s reasoning process\. Replacing vague vocal descriptors with discrete tags prevents identity confusion throughout extended chain\-of\-thought generation\.

## 6Related Works

##### Speaker Attribution: From Speaker Recognition to Reasoning\.

The question of*who said what*[Kanda et al\. \(2021\)](https://arxiv.org/html/2608.29120#bib.bib42)is traditionally addressed through speaker diarization and multi\-talker ASR, which identify*who spoke when*and*what was said*, respectively\. Recent LLM\-based recognizers improve this pipeline by incorporating speaker information or supporting instruction\-guided transcription[Wang et al\. \(2024\)](https://arxiv.org/html/2608.29120#bib.bib40);[Li et al\. \(2026\)](https://arxiv.org/html/2608.29120#bib.bib39);[Meng et al\. \(2025\)](https://arxiv.org/html/2608.29120#bib.bib13)\. Yet these systems mainly produce speaker\-labeled transcripts, leaving speaker\-content reasoning largely unexplored\. We refer to this capability as*speaker\-attributed reasoning*: answering questions that require linking what was said to who said it, such as identifying the speaker of an utterance or determining who spoke more or first\.

##### Evaluation of Speaker\-Attributed Understanding\.

Recent SLM benchmarks have expanded from single\-speaker QA to broader audio and multi\-party understanding, but speaker attribution remains only partially isolated\. Broad benchmarks such as MMAU[Sakshi et al\. \(2024\)](https://arxiv.org/html/2608.29120#bib.bib4);[Kumar et al\. \(2025\)](https://arxiv.org/html/2608.29120#bib.bib5), AudioMarathon[He et al\. \(2025\)](https://arxiv.org/html/2608.29120#bib.bib11), and ChronosAudio[Luo et al\. \(2026\)](https://arxiv.org/html/2608.29120#bib.bib14)include multi\-speaker scenarios, yet treat them as a small subset of tasks \(e\.g\., Voice Counting\) among many evaluation axes\. M3\-SLU[Kwon et al\. \(2025\)](https://arxiv.org/html/2608.29120#bib.bib10)is closest to our setting, but its evaluation is limited to a narrow task \(e\.g\., similar to IR dimension of HEAR\), mainly true/false or single utterance–speaker matching\. MSU\-Bench[Wang et al\. \(2025\)](https://arxiv.org/html/2608.29120#bib.bib12)evaluates multi\-party conversational understanding, but does not separate speaker attribution as a dedicated axis\. In contrast, HEAR isolates speaker attribution itself as the primary axis, organizing it into discrimination, attribution, and reasoning stages that prior benchmarks leave entangled\.

## 7Conclusion

We introduced HEAR, a benchmark for evaluating speaker\-attributed reasoning in SLMs, and showed that current models often fail to bind utterances to distinct voices, instead relying on semantic priors over vocal evidence\. To address this limitation, we proposed CASH, a 60K\-scale dataset of voice\-cloned hard negatives that decouples acoustic cues from semantic content, and developed A2R to explicitly learn speaker attribution\. A2R substantially improves performance on HEAR and transfers to unseen multi\-speaker tasks\. Together, we hope our findings establish speaker attribution as a key step toward multi\-party auditory comprehension\.

## Limitations

While explicit reasoning via transcript generation maximizes speaker attribution performance, it introduces latency that limits real\-time applications\. Importantly, externalizing transcripts is not essential: as shown in Figure[7](https://arxiv.org/html/2608.29120#S5.F7), A2R significantly outperforms baselines even without this step, albeit with slightly lower performance\. Bridging this latency–performance gap through implicit reasoning over internal representations, without generating explicit reasoning traces, is a promising direction for future work\.

## Ethics Statement

All verification in our work was conducted by eight authors of this paper\. Our work also involves voice cloning samples that pose risks of impersonation and speaker re\-identification\. Because dataset or copyright licenses do not themselves constitute consent for voice cloning, and no specific consent was obtained from source speakers, synthetic audio is restricted to approved, non\-commercial research under safeguards against misuse\.

##### Dual\-use risks\.

Given the potential misuse of voice cloning for impersonation, deceptive audio generation, fraud, harassment, speaker re\-identification, and other harmful applications, we treat the synthetic speech component of our work as a controlled research resource rather than an unrestricted public dataset\.

##### Voice, likeness, and consent\.

Although the source corpora provide licenses or research\-use conditions governing the underlying recordings, these conditions do not necessarily constitute consent to generate novel utterances in an individual’s voice, and we did not obtain additional task\-specific consent from the source speakers\. We also do not regard pseudonymous speaker identifiers as providing complete anonymity, because a recognizable voice can itself convey identity\.

##### Release scope and access control\.

Code, evaluation protocols, non\-identifying annotations and metadata, and aggregate statistics can be released publicly, whereas synthetic waveform data are available only through a gated research repository for approved, non\-commercial research related to speaker\-aware speech understanding and reasoning\. Requesters must provide their identity, institutional affiliation, and intended research use and agree to the Data Use Agreement \(DUA\) before access is granted\.

##### Data Use Agreement\.

The DUA restricts use of the synthetic audio to approved non\-commercial research and prohibits redistribution, attempts to re\-identify source speakers, impersonation, deceptive or misleading use, commercial voice replication, deployment of cloned voices in interactive or production systems, and other uses intended to harm or misrepresent individuals\. Access is revocable when these conditions are violated\.

##### Non\-attribution and documentation\.

All synthetic utterances are documented as artificially generated samples\. They are not statements actually made by the corresponding source speakers and must not be interpreted as representing those individuals’ views, intentions, beliefs, or endorsements\. Personally identifying metadata that are unnecessary for the research purpose, including available speaker names and source\-speaker mappings, are removed from the released resource\.

##### Takedown procedure\.

Both rights holders and individuals whose voices occur in the resource, or their authorized representatives, can request removal of corresponding entries\. Requests can identify the relevant source recording or other information sufficient to locate the affected samples\. Valid requests result in removal of the associated source\-derived and synthetic entries from subsequent distributions of the resource\. Takedown records are maintained in the dataset documentation without disclosing unnecessary personal information about the requester\.

## Acknowledgements

This work was supported by Institute of Information & Communications Technology Planning & Evaluation \(IITP\) grants funded by the Korea government \(MSIT\) \[NO\.RS\-2021\-II211343, Artificial Intelligence Graduate School Program \(Seoul National University\); No\.2022\-0\-00959, RS\-2022\-II220959\], National Research Foundation of Korea \(NRF\) grant \[No\.2022R1A3B1077720, 2022R1A5A7083908\], BK21 FOUR Program of the Education and Research Program for Future ICT Pioneers, Seoul National University in 2026, Mobile eXperience\(MX\) Business, Samsung Electronics Co\., Ltd\., NVIDIA Academic Grant Program and the Research Grant from Seoul National University\(800\-20250397\)\.

## References

- Ardilaet al\.\(2020\)R\. Ardila, M\. Branson, K\. Davis, M\. Henretty, M\. Kohler, J\. Meyer, R\. Morais, L\. Saunders, F\. M\. Tyers, and G\. WeberCommon voice: a massively\-multilingual speech corpus\.External Links:1912\.06670,[Link](https://arxiv.org/abs/1912.06670)Cited by:[Appendix G](https://arxiv.org/html/2608.29120#A7.SS0.SSS0.Px1)\.
- Carlettaet al\.\(2005\)J\. Carletta, S\. Ashby, S\. Bourban, M\. Flynn, M\. Guillemot, T\. Hain, J\. Kadlec, V\. Karaiskos, W\. Kraaij, M\. Kronenthal, G\. Lathoud, M\. Lincoln, A\. Lisowska, I\. McCowan, W\. Post, D\. Reidsma, and P\. WellnerThe ami meeting corpus: a pre\-announcement\.InProceedings of the Second International Conference on Machine Learning for Multimodal Interaction,MLMI’05,Berlin, Heidelberg,pp\. 28–39\.External Links:ISBN 3540325492,[Link](https://doi.org/10.1007/11677482_3),[Document](https://dx.doi.org/10.1007/11677482%5F3)Cited by:[Appendix I](https://arxiv.org/html/2608.29120#A9.p1.1),[§2\.2](https://arxiv.org/html/2608.29120#S2.SS2.p1.1)\.
- Chenet al\.\(2024\)Y\. Chen, X\. Yue, C\. Zhang, X\. Gao, R\. T\. Tan, and H\. LiVoiceBench: benchmarking llm\-based voice assistants\.External Links:2410\.17196,[Link](https://arxiv.org/abs/2410.17196)Cited by:[Appendix G](https://arxiv.org/html/2608.29120#A7.p1.1),[§4](https://arxiv.org/html/2608.29120#S4.SS0.SSS0.Px3.p6.1)\.
- Chuet al\.\(2024\)Y\. Chu, J\. Xu, Q\. Yang, H\. Wei, X\. Wei, Z\. Guo, Y\. Leng, Y\. Lv, J\. He, J\. Lin, C\. Zhou, and J\. ZhouQwen2\-audio technical report\.External Links:2407\.10759,[Link](https://arxiv.org/abs/2407.10759)Cited by:[2nd item](https://arxiv.org/html/2608.29120#A4.I1.i2.p1.1)\.
- Cuiet al\.\(2026\)J\. Cui, B\. Xu, C\. Wang, T\. Yu, W\. Sun, Y\. Xu, T\. Wang, Z\. He, W\. Ma, T\. Cai, J\. Gui, L\. Zhang, X\. Sun, F\. Huang, M\. Chen, Z\. Lin, H\. Liu, Q\. Gui, Q\. Han, Y\. Wen, H\. Liu, R\. Wang, Y\. Zhang, H\. Wei, C\. Chen, Y\. Li, K\. Fang, J\. Zhou, Y\. Li, G\. Zeng, C\. Xiao, Y\. Lin, X\. Han, M\. Sun, Z\. Liu, and Y\. YaoMiniCPM\-o 4\.5: towards real\-time full\-duplex omni\-modal interaction\.External Links:2604\.27393,[Link](https://arxiv.org/abs/2604.27393)Cited by:[1st item](https://arxiv.org/html/2608.29120#A4.I1.i1.p1.1)\.
- Desplanqueset al\.\(2020\)B\. Desplanques, J\. Thienpondt, and K\. DemuynckECAPA\-tdnn: emphasized channel attention, propagation and aggregation in tdnn based speaker verification\.InInterspeech 2020,interspeech 2020,pp\. 3830–3834\.External Links:[Link](http://dx.doi.org/10.21437/Interspeech.2020-2650),[Document](https://dx.doi.org/10.21437/interspeech.2020-2650)Cited by:[Appendix A](https://arxiv.org/html/2608.29120#A1.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2608.29120#S4.SS0.SSS0.Px2.p1.1)\.
- Dinkelet al\.\(2026\)H\. Dinkel, G\. Li, J\. Liu, J\. Luan, Y\. Niu, X\. Sun, T\. Wang, Q\. Xiao, J\. Zhang, and J\. ZhouMiDashengLM: efficient audio understanding with general audio captions\.External Links:2508\.03983,[Link](https://arxiv.org/abs/2508.03983)Cited by:[2nd item](https://arxiv.org/html/2608.29120#A4.I1.i2.p1.1)\.
- Duboiset al\.\(2025\)Y\. Dubois, B\. Galambosi, P\. Liang, and T\. B\. HashimotoLength\-controlled alpacaeval: a simple way to debias automatic evaluators\.External Links:2404\.04475,[Link](https://arxiv.org/abs/2404.04475)Cited by:[Appendix G](https://arxiv.org/html/2608.29120#A7.SS0.SSS0.Px1)\.
- Erber \(1982\)N\. P\. ErberAuditory training / norman p\. erber\.\.Alexander Graham Bell Association for the Deaf,Washington, D\.C\(eng\)\.External Links:ISBN 0882001493,LCCN 82173470Cited by:[Figure 1](https://arxiv.org/html/2608.29120#S1.F1),[§1](https://arxiv.org/html/2608.29120#S1.p2.1),[§2](https://arxiv.org/html/2608.29120#S2.p1.1)\.
- Gaoet al\.\(2025\)Y\. Gao, B\. Wang, C\. Wei, S\. Sun, and A\. AwIFEval\-audio: benchmarking instruction\-following capability in audio\-based large language models\.External Links:2505\.16774,[Link](https://arxiv.org/abs/2505.16774)Cited by:[Appendix G](https://arxiv.org/html/2608.29120#A7.SS0.SSS0.Px4)\.
- Goelet al\.\(2025\)A\. Goel, S\. Ghosh, J\. Kim, S\. Kumar, Z\. Kong, S\. Lee, C\. H\. Yang, R\. Duraiswami, D\. Manocha, R\. Valle, and B\. CatanzaroAudio flamingo 3: advancing audio intelligence with fully open large audio language models\.External Links:2507\.08128,[Link](https://arxiv.org/abs/2507.08128)Cited by:[2nd item](https://arxiv.org/html/2608.29120#A4.I1.i2.p1.1)\.
- Google DeepMind \(2026a\)Google DeepMindGemini 3 flash model card\.Note:Model cardAccessed: 2026\-08\-26External Links:[Link](https://deepmind.google/models/model-cards/gemini-3-flash/)Cited by:[3rd item](https://arxiv.org/html/2608.29120#A4.I1.i3.p1.1)\.
- Google DeepMind \(2026b\)Google DeepMindGemini 3\.1 pro model card\.Note:Model cardAccessed: 2026\-08\-26External Links:[Link](https://deepmind.google/models/model-cards/gemini-3-1-pro/)Cited by:[3rd item](https://arxiv.org/html/2608.29120#A4.I1.i3.p1.1)\.
- Heet al\.\(2025\)P\. He, Z\. Wen, Y\. Wang, Y\. Wang, X\. Liu, J\. Huang, Z\. Lei, Z\. Gu, X\. Jin, J\. Yang, K\. Li, Z\. Liu, W\. Li, C\. Wang, C\. He, and L\. ZhangAudioMarathon: a comprehensive benchmark for long\-context audio understanding and efficiency in audio llms\.External Links:2510\.07293,[Link](https://arxiv.org/abs/2510.07293)Cited by:[§6](https://arxiv.org/html/2608.29120#S6.SS0.SSS0.Px2.p1.1)\.
- Huet al\.\(2021\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.External Links:2106\.09685,[Link](https://arxiv.org/abs/2106.09685)Cited by:[§4](https://arxiv.org/html/2608.29120#S4.SS0.SSS0.Px1.p1.1)\.
- Huet al\.\(2024\)S\. Hu, L\. Zhou, S\. Liu, S\. Chen, L\. Meng, H\. Hao, J\. Pan, X\. Liu, J\. Li, S\. Sivasankaran, L\. Liu, and F\. WeiWavLLM: towards robust and adaptive speech large language model\.External Links:2404\.00656,[Link](https://arxiv.org/abs/2404.00656)Cited by:[Appendix C](https://arxiv.org/html/2608.29120#A3.SS0.SSS0.Px2.p2.1),[§4](https://arxiv.org/html/2608.29120#S4.SS0.SSS0.Px3.p3.1)\.
- Janinet al\.\(2003\)A\. L\. Janin, D\. Baron, J\. Edwards, D\. P\. W\. Ellis, D\. Gelbart, N\. Morgan, B\. Peskin, T\. Pfau, E\. Shriberg, A\. Stolcke, and C\. WootersThe icsi meeting corpus\.2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003\. Proceedings\. \(ICASSP ’03\)\.1,pp\. I–I\.External Links:[Link](https://api.semanticscholar.org/CorpusID:18614936)Cited by:[Appendix I](https://arxiv.org/html/2608.29120#A9.p1.1),[§2\.2](https://arxiv.org/html/2608.29120#S2.SS2.p1.1)\.
- Kandaet al\.\(2021\)N\. Kanda, X\. Xiao, J\. Wu, T\. Zhou, Y\. Gaur, X\. Wang, Z\. Meng, Z\. Chen, and T\. YoshiokaA comparative study of modular and joint approaches for speaker\-attributed asr on monaural long\-form audio\.External Links:2107\.02852,[Link](https://arxiv.org/abs/2107.02852)Cited by:[§1](https://arxiv.org/html/2608.29120#S1.p1.1),[§6](https://arxiv.org/html/2608.29120#S6.SS0.SSS0.Px1.p1.1)\.
- KimiTeamet al\.\(2025\)KimiTeam, D\. Ding, Z\. Ju, Y\. Leng, S\. Liu, T\. Liu, Z\. Shang, K\. Shen, W\. Song, X\. Tan, H\. Tang, Z\. Wang, C\. Wei, Y\. Xin, X\. Xu, J\. Yu, Y\. Zhang, X\. Zhou, Y\. Charles, J\. Chen, Y\. Chen, Y\. Du, W\. He, Z\. Hu, G\. Lai, Q\. Li, Y\. Liu, W\. Sun, J\. Wang, Y\. Wang, Y\. Wu, Y\. Wu, D\. Yang, H\. Yang, Y\. Yang, Z\. Yang, A\. Yin, R\. Yuan, Y\. Zhang, and Z\. ZhouKimi\-audio technical report\.External Links:2504\.18425,[Link](https://arxiv.org/abs/2504.18425)Cited by:[2nd item](https://arxiv.org/html/2608.29120#A4.I1.i2.p1.1)\.
- Koishigarinaet al\.\(2026\)D\. Koishigarina, A\. Uselis, and S\. J\. OhCLIP behaves like a bag\-of\-words model cross\-modally but not uni\-modally\.External Links:2502\.03566,[Link](https://arxiv.org/abs/2502.03566)Cited by:[§3\.1](https://arxiv.org/html/2608.29120#S3.SS1.p1.1)\.
- Kumaret al\.\(2025\)S\. Kumar, Š\. Sedláček, V\. Lokegaonkar, F\. López, W\. Yu, N\. Anand, H\. Ryu, L\. Chen, M\. Plička, M\. Hlaváček, W\. F\. Ellingwood, S\. Udupa, S\. Hou, A\. Ferner, S\. Barahona, C\. Bolaños, S\. Rahi, L\. Herrera\-Alarcón, S\. Dixit, S\. Patil, S\. Deshmukh, L\. Koroshinadze, Y\. Liu, L\. P\. G\. Perera, E\. Zanou, T\. Stafylakis, J\. S\. Chung, D\. Harwath, C\. Zhang, D\. Manocha, A\. Lozano\-Diez, S\. Kesiraju, S\. Ghosh, and R\. DuraiswamiMMAU\-pro: a challenging and comprehensive benchmark for holistic evaluation of audio general intelligence\.External Links:2508\.13992,[Link](https://arxiv.org/abs/2508.13992)Cited by:[§1](https://arxiv.org/html/2608.29120#S1.p1.1),[§6](https://arxiv.org/html/2608.29120#S6.SS0.SSS0.Px2.p1.1)\.
- Kwaket al\.\(2024\)D\. Kwak, J\. Jung, K\. Nam, Y\. Jang, J\. Jung, S\. Watanabe, and J\. S\. ChungVoxMM: rich transcription of conversations in the wild\.InICASSP 2024 \- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),Vol\.,pp\. 12551–12555\.External Links:[Document](https://dx.doi.org/10.1109/ICASSP48485.2024.10446300)Cited by:[Appendix A](https://arxiv.org/html/2608.29120#A1.SS0.SSS0.Px1.p1.1),[Appendix I](https://arxiv.org/html/2608.29120#A9.p1.1),[§2\.2](https://arxiv.org/html/2608.29120#S2.SS2.p1.1),[§3\.2](https://arxiv.org/html/2608.29120#S3.SS2.p1.1)\.
- Kwonet al\.\(2023\)W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. StoicaEfficient memory management for large language model serving with pagedattention\.InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles,Cited by:[Table 7](https://arxiv.org/html/2608.29120#A10.T7),[Appendix F](https://arxiv.org/html/2608.29120#A6.p2.1)\.
- Kwonet al\.\(2025\)Y\. Kwon, T\. Kang, H\. Yoon, and C\. KimM3\-slu: evaluating speaker\-attributed reasoning in multimodal large language models\.External Links:2510\.19358,[Link](https://arxiv.org/abs/2510.19358)Cited by:[§6](https://arxiv.org/html/2608.29120#S6.SS0.SSS0.Px2.p1.1)\.
- Leeet al\.\(2026\)D\. Lee, E\. Song, C\. H\. Lee, H\. Kim, and S\. YoonStill between us? evaluating and improving voice assistant robustness to third\-party interruptions\.External Links:2604\.17358,[Link](https://arxiv.org/abs/2604.17358)Cited by:[§1](https://arxiv.org/html/2608.29120#S1.p1.1),[§3\.1](https://arxiv.org/html/2608.29120#S3.SS1.p1.1)\.
- Liet al\.\(2026\)L\. Li, M\. Cheng, W\. Zhu, Y\. Wang, J\. Liu, and M\. LiDM\-asr: diarization\-aware multi\-speaker asr with large language models\.External Links:2604\.22467,[Link](https://arxiv.org/abs/2604.22467)Cited by:[§6](https://arxiv.org/html/2608.29120#S6.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2025\)A\. H\. Liu, A\. Ehrenberg, A\. Lo, C\. Denoix, C\. Barreau, G\. Lample, J\. Delignon, K\. R\. Chandu, P\. von Platen, P\. R\. Muddireddy, S\. Gandhi, S\. Ghosh, S\. Mishra, T\. Foubert, A\. Rastogi, A\. Yang, A\. Q\. Jiang, A\. Sablayrolles, A\. Héliou, A\. Martin, A\. Agarwal, A\. Roux, A\. Darcet, A\. Mensch, B\. Bout, B\. Rozière, B\. D\. Monicault, C\. Bamford, C\. Wallenwein, C\. Renaudin, C\. Lanfranchi, D\. Dabert, D\. S\. Chaplot, D\. Mizelle, D\. de las Casas, E\. Chane\-Sane, E\. Fugier, E\. B\. Hanna, G\. Berrada, G\. Delerce, G\. Guinet, G\. Novikov, G\. Martin, H\. Jaju, J\. Ludziejewski, J\. Rute, J\. Chabran, J\. Chudnovsky, J\. Studnia, J\. Barmentlo, J\. Amar, J\. S\. Roberts, J\. Denize, K\. Saxena, K\. Yadav, K\. Khandelwal, K\. Jain, L\. R\. Lavaud, L\. Blier, L\. Zhao, L\. Martin, L\. Saulnier, L\. Gao, M\. Pellat, M\. Guillaumin, M\. Felardos, M\. Dinot, M\. Darrin, M\. Augustin, M\. Seznec, N\. Gupta, N\. Raghuraman, O\. Duchenne, P\. Wang, P\. Saffer, P\. Jacob, P\. Wambergue, P\. Kurylowicz, P\. Chagniot, P\. Stock, P\. Agrawal, R\. Delacourt, R\. Sauvestre, R\. Soletskyi, S\. Vaze, S\. Subramanian, S\. Garg, S\. Dalal, S\. Gandhi, S\. Aithal, S\. Antoniak, T\. L\. Scao, T\. Schueller, T\. Lavril, T\. Robert, T\. Wang, T\. Lacroix, T\. Bewley, V\. Nemychnikova, V\. Paltz, V\. Richard, W\. Li, W\. Marshall, X\. Zhang, Y\. Wan, and Y\. TangVoxtral\.External Links:2507\.13264,[Link](https://arxiv.org/abs/2507.13264)Cited by:[2nd item](https://arxiv.org/html/2608.29120#A4.I1.i2.p1.1)\.
- Luoet al\.\(2026\)K\. Luo, L\. Lin, Y\. Zhang, M\. Aloqaily, D\. Wang, Z\. Zhou, J\. Zhang, K\. Wang, L\. Sun, and Q\. WenChronosAudio: a comprehensive long\-audio benchmark for evaluating audio\-large language models\.External Links:2601\.04876,[Link](https://arxiv.org/abs/2601.04876)Cited by:[§6](https://arxiv.org/html/2608.29120#S6.SS0.SSS0.Px2.p1.1)\.
- Menget al\.\(2025\)L\. Meng, S\. Hu, J\. Kang, Z\. Li, Y\. Wang, W\. Wu, X\. Wu, X\. Liu, and H\. MengLarge language model can transcribe speech in multi\-talker scenarios with versatile instructions\.External Links:2409\.08596,[Link](https://arxiv.org/abs/2409.08596)Cited by:[§6](https://arxiv.org/html/2608.29120#S6.SS0.SSS0.Px1.p1.1)\.
- Microsoftet al\.\(2025\)Microsoft, :, A\. Abouelenin, A\. Ashfaq, A\. Atkinson, H\. Awadalla, N\. Bach, J\. Bao, A\. Benhaim, M\. Cai, V\. Chaudhary, C\. Chen, D\. Chen, D\. Chen, J\. Chen, W\. Chen, Y\. Chen, Y\. Chen, Q\. Dai, X\. Dai, R\. Fan, M\. Gao, M\. Gao, A\. Garg, A\. Goswami, J\. Hao, A\. Hendy, Y\. Hu, X\. Jin, M\. Khademi, D\. Kim, Y\. J\. Kim, G\. Lee, J\. Li, Y\. Li, C\. Liang, X\. Lin, Z\. Lin, M\. Liu, Y\. Liu, G\. Lopez, C\. Luo, P\. Madan, V\. Mazalov, A\. Mitra, A\. Mousavi, A\. Nguyen, J\. Pan, D\. Perez\-Becker, J\. Platin, T\. Portet, K\. Qiu, B\. Ren, L\. Ren, S\. Roy, N\. Shang, Y\. Shen, S\. Singhal, S\. Som, X\. Song, T\. Sych, P\. Vaddamanu, S\. Wang, Y\. Wang, Z\. Wang, H\. Wu, H\. Xu, W\. Xu, Y\. Yang, Z\. Yang, D\. Yu, I\. Zabir, J\. Zhang, L\. L\. Zhang, Y\. Zhang, and X\. ZhouPhi\-4\-mini technical report: compact yet powerful multimodal language models via mixture\-of\-loras\.External Links:2503\.01743,[Link](https://arxiv.org/abs/2503.01743)Cited by:[1st item](https://arxiv.org/html/2608.29120#A4.I1.i1.p1.1)\.
- Mihaylovet al\.\(2018\)T\. Mihaylov, P\. Clark, T\. Khot, and A\. SabharwalCan a suit of armor conduct electricity? a new dataset for open book question answering\.External Links:1809\.02789,[Link](https://arxiv.org/abs/1809.02789)Cited by:[Appendix G](https://arxiv.org/html/2608.29120#A7.SS0.SSS0.Px3)\.
- Nguyenet al\.\(2025\)L\. T\. P\. Nguyen, Z\. Yu, S\. L\. Y\. Hang, S\. An, J\. Lee, Y\. Ban, S\. Chung, T\. Nguyen, J\. Maeng, S\. Lee, and Y\. J\. LeeSee, hear, and understand: benchmarking audiovisual human speech understanding in multimodal large language models\.External Links:2512\.02231,[Link](https://arxiv.org/abs/2512.02231)Cited by:[§1](https://arxiv.org/html/2608.29120#S1.p1.1)\.
- OpenAI \(2024\)OpenAIGPT\-4o system card\.arXiv preprint arXiv:2410\.21276\.External Links:[Link](https://arxiv.org/abs/2410.21276)Cited by:[3rd item](https://arxiv.org/html/2608.29120#A4.I1.i3.p1.1)\.
- OpenAI \(2026a\)OpenAIGPT\-5\.5 System Card\.Note:[https://openai\.com/index/gpt\-5\-5\-system\-card/](https://openai.com/index/gpt-5-5-system-card/)Accessed: 2026\-08\-27Cited by:[§4](https://arxiv.org/html/2608.29120#S4.SS0.SSS0.Px3.p3.1)\.
- OpenAI \(2026b\)OpenAIIntroducing GPT\-5\.4 mini and nano\.Note:[https://openai\.com/index/introducing\-gpt\-5\-4\-mini\-and\-nano/](https://openai.com/index/introducing-gpt-5-4-mini-and-nano/)Accessed: 2026\-08\-27Cited by:[Appendix G](https://arxiv.org/html/2608.29120#A7.p1.1)\.
- Radfordet al\.\(2022\)A\. Radford, J\. W\. Kim, T\. Xu, G\. Brockman, C\. McLeavey, and I\. SutskeverRobust speech recognition via large\-scale weak supervision\.External Links:2212\.04356,[Link](https://arxiv.org/abs/2212.04356)Cited by:[Appendix A](https://arxiv.org/html/2608.29120#A1.SS0.SSS0.Px1.p1.1),[§B\.2](https://arxiv.org/html/2608.29120#A2.SS2.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2608.29120#S4.SS0.SSS0.Px2.p1.1)\.
- Sakshiet al\.\(2024\)S\. Sakshi, U\. Tyagi, S\. Kumar, A\. Seth, R\. Selvakumar, O\. Nieto, R\. Duraiswami, S\. Ghosh, and D\. ManochaMMAU: a massive multi\-task audio understanding and reasoning benchmark\.External Links:2410\.19168,[Link](https://arxiv.org/abs/2410.19168)Cited by:[§1](https://arxiv.org/html/2608.29120#S1.p1.1),[§6](https://arxiv.org/html/2608.29120#S6.SS0.SSS0.Px2.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. GuoDeepSeekMath: pushing the limits of mathematical reasoning in open language models\.External Links:2402\.03300,[Link](https://arxiv.org/abs/2402.03300)Cited by:[§1](https://arxiv.org/html/2608.29120#S1.p4.1)\.
- Suzgunet al\.\(2022\)M\. Suzgun, N\. Scales, N\. Schärli, S\. Gehrmann, Y\. Tay, H\. W\. Chung, A\. Chowdhery, Q\. V\. Le, E\. H\. Chi, D\. Zhou, and J\. WeiChallenging big\-bench tasks and whether chain\-of\-thought can solve them\.External Links:2210\.09261,[Link](https://arxiv.org/abs/2210.09261)Cited by:[Appendix G](https://arxiv.org/html/2608.29120#A7.SS0.SSS0.Px2)\.
- Teamet al\.\(2026a\)G\. Team, S\. E\. Abd, V\. Aggarwal, R\. Algayres, A\. Andreev, O\. Bachem, I\. Ballantyne, C\. Brick, V\. Cărbune, M\. Casbon, M\. Chaturvedi, A\. Chawla, V\. Cotruta, A\. Coucke, P\. Culliton, R\. Dadashi, L\. Dixon, M\. Elhawaty, U\. Evci, C\. Farabet, J\. Ferret, F\. Galgani, S\. Girgin, J\. Grill, M\. Grootendorst, J\. Guo, C\. Hardin, Y\. He, S\. M\. Hernandez, O\. Homburger, L\. Hussenot, J\. Ji, A\. Joulin, A\. Kamath, P\. Kassraie, O\. Lacombe, P\. Lahoti, G\. Liu, G\. Martins, L\. Martins, T\. Matejovicova, R\. Merhej, N\. Momchev, S\. Mondal, R\. Mullins, S\. R\. Panyam, S\. Pathak, S\. Perrin, A\. S\. Pinto, E\. Pot, A\. Pouget, A\. Ramé, S\. Ramos, D\. Reid, D\. Rim, M\. Rivière, K\. Roth, L\. Rouillard, O\. Sanseviero, P\. G\. Sessa, S\. Settle, D\. Sinopalnikov, S\. Smoot, P\. Stanczyk, A\. Steiner, L\. Stewart, I\. Tolstikhin, M\. Tschannen, A\. Tsitsulin, N\. Vieillard, R\. Wu, P\. Xu, H\. Yang, E\. Yvinec, B\. Zhang, L\. Zhang, J\. Zou, N\. Aagnes, A\. Abdelhamed, J\. Adamek, S\. Agrawal, S\. Agrawal, I\. Alabdulmohsin, J\. B\. Alayrac, U\. Alon, C\. Amarnath, A\. Anand, C\. Anastasiou, S\. Ariafar, F\. Aubet, K\. Axiotis, F\. Barbero, J\. Barral, A\. Bendebury, U\. Bergmann, S\. Bileschi, K\. Black, M\. Blondel, S\. Borgeaud, A\. Bražinskas, R\. Burnell, R\. Busa\-Fekete, M\. Cai, D\. Calandriello, G\. Cameron, C\. Caucheteux, R\. Chaabouni, G\. Chadha, J\. Chan, B\. J\. Chen, J\. Chen, L\. Chen, X\. Chen, D\. Cheng, T\. Chien, N\. Chinaev, Y\. Chou, Z\. Chu, B\. Coleman, P\. Consul, S\. Conway\-Rahman, S\. Crowell, D\. Cutler, V\. Dani, S\. Daruki, A\. Das, D\. Deutsch, N\. Dikkala, L\. Ding, Q\. Ding, S\. Dodhia, K\. Donhauser, T\. Doshi, A\. Dragan, A\. Druinsky, S\. Dua, Z\. Egyed, D\. Eisenbud, D\. Eppens, C\. Fan, B\. Fatemi, Y\. Fathullah, V\. Feinberg, M\. Ferev, S\. Flennerhag, T\. Fujimoto, J\. G\. Oliveira, I\. Galatzer\-Levy, J\. Gante, S\. Geisler, S\. Ghosal, A\. M\. Girgis, T\. von Glehn, A\. Go, A\. Gokhale, A\. Grills, Y\. Gu, M\. Gupta, P\. Gupta, G\. Guruganesh, R\. Hadsell, H\. Harkous, J\. Harlalka, D\. Hassabis, A\. Hauth, J\. Heyward, A\. Hosseini, C\. Hsia, I\. Hsu, X\. Huang, Y\. Huang, K\. Hui, A\. Hutter, T\. I, F\. Iliopoulos, A\. Jain, G\. Jawahar, Z\. Ji, Q\. Jin, M\. Johnson, K\. Joshi, A\. Kandoor, W\. Kang, K\. Kavukcuoglu, M\. Kazemi, K\. Kenealy, A\. Khalifa, P\. Kirk, I\. Korotkov, S\. Kothawade, V\. Kovalev, N\. Kovelamudi, A\. Kraft, R\. Kumar, V\. Kumar, H\. Kuppam, J\. Lannin, C\. Lee, S\. Lee, D\. Lepikhin, A\. Levkovitch, D\. Li, Q\. Li, V\. Liévin, E\. Lin, Z\. Lin, C\. Liu, T\. Liu, T\. Liu, X\. Liu, I\. Lobov, M\. Lunayach, M\. Ma, G\. Madan, A\. Maksai, E\. Malmi, M\. Matuszak, D\. McDuff, G\. Menghani, M\. Mikuła, D\. Mirylenka, K\. Misiunas, V\. Misra, A\. Mitran, K\. Mohamed, M\. Mukha, E\. Noland, J\. O’Donnell, B\. O’Donoghue, K\. Olszewska, B\. Orlando, W\. Pan, R\. Panigrahy, U\. Parekh, N\. Perez\-Nieves, C\. Park, E\. Paskie, L\. Peng, B\. Petrini, S\. Petrov, J\. Pfeiffer, B\. Piot, M\. Plomecka, S\. Poder, O\. Ponce, A\. Pramanik, D\. Racz, A\. Rajan, M\. Ramanovich, A\. Rao, M\. Ritter, V\. Rodrigues, E\. Rosen, M\. Rybiński, N\. Sachdeva, M\. E\. Sander, R\. Sathyanarayana, S\. Savla, S\. Schmidgall, T\. Schuster, G\. Scrivener, B\. Seguin, A\. Sellergren, A\. Severyn, I\. Shafran, D\. Shah, B\. Shahriari, Y\. Shangguan, A\. Shenoy, P\. Shenoy, R\. Shivanna, P\. Sho, L\. Spangher, W\. Stokowiec, T\. Strother, Y\. Su, Y\. Sun, M\. Sundararajan, A\. Tacchetti, M\. H\. Taege, P\. Tafti, J\. Tarbouriech, C\. Tekur, S\. Thakoor, R\. Thapa, M\. Traverse, L\. Treven, T\. Tu, C\. T\. Tung, Ç\. Ünlü, P\. Veličković, M\. P\. Venkat, S\. G\. Venkatesh, V\. Venkiteswaran, F\. Visin, A\. Vitvitskyi, K\. Vodrahalli, W\. Wang, X\. Wang, T\. Warkentin, J\. Wassenberg, J\. Wieting, C\. Wu, L\. Xiao, H\. Xu, Y\. Xu, F\. Xue, A\. Yadav, J\. Yan, A\. Yang, L\. Yang, M\. Yang, Z\. Ying, J\. H\. Yoo, M\. Zadimoghaddam, S\. Zafar, F\. Zhang, J\. Zhang, J\. Zhang, X\. Zhang, C\. Zhao, D\. Zhou, and C\. ZouGemma 4 technical report\.External Links:2607\.02770,[Link](https://arxiv.org/abs/2607.02770)Cited by:[1st item](https://arxiv.org/html/2608.29120#A4.I1.i1.p1.1)\.
- Teamet al\.\(2026b\)T\. F\. Team, Q\. Chen, L\. Cheng, C\. Deng, X\. Li, J\. Liu, C\. Tan, W\. Wang, J\. Xu, J\. Ye, Q\. Zhang, Q\. Zhang, and J\. ZhouFun\-audio\-chat technical report\.External Links:2512\.20156,[Link](https://arxiv.org/abs/2512.20156)Cited by:[2nd item](https://arxiv.org/html/2608.29120#A4.I1.i2.p1.1)\.
- Tianet al\.\(2025\)F\. Tian, X\. T\. Zhang, Y\. Zhang, H\. Zhang, Y\. Li, D\. Liu, Y\. Deng, D\. Wu, J\. Chen, L\. Zhao, C\. Yao, H\. Liu, E\. S\. Chng, X\. Yang, X\. Zhang, D\. Jiang, and G\. YuStep\-audio\-r1 technical report\.External Links:2511\.15848,[Link](https://arxiv.org/abs/2511.15848)Cited by:[2nd item](https://arxiv.org/html/2608.29120#A4.I1.i2.p1.1)\.
- von Neumannet al\.\(2024\)T\. von Neumann, C\. Boeddeker, M\. Delcroix, and R\. Haeb\-UmbachMeetEval: a toolkit for computation of word error rates for meeting transcription systems\.External Links:2307\.11394,[Link](https://arxiv.org/abs/2307.11394)Cited by:[§3\.3](https://arxiv.org/html/2608.29120#S3.SS3.p3.1)\.
- Wanget al\.\(2026\)D\. Wang, J\. Li, J\. Wu, D\. Yang, X\. Chen, T\. Zhang, and H\. MengMMSU: a massive multi\-task spoken language understanding and reasoning benchmark\.External Links:2506\.04779,[Link](https://arxiv.org/abs/2506.04779)Cited by:[Appendix G](https://arxiv.org/html/2608.29120#A7.SS0.SSS0.Px3)\.
- Wanget al\.\(2024\)J\. Wang, W\. Wang, K\. Dhawan, T\. Park, M\. Kim, I\. Medennikov, H\. Huang, N\. Koluguri, J\. Balam, and B\. GinsburgMETA\-cat: speaker\-informed speech embeddings via meta information concatenation for multi\-talker asr\.External Links:2409\.12352,[Link](https://arxiv.org/abs/2409.12352)Cited by:[§6](https://arxiv.org/html/2608.29120#S6.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2025\)S\. Wang, Z\. Sun, Z\. Lin, C\. Wang, Z\. Pan, and L\. XieMSU\-bench: towards understanding the conversational multi\-talker scenarios\.External Links:2508\.08155,[Link](https://arxiv.org/abs/2508.08155)Cited by:[§6](https://arxiv.org/html/2608.29120#S6.SS0.SSS0.Px2.p1.1)\.
- Wuet al\.\(2025\)B\. Wu, C\. Yan, C\. Hu, C\. Yi, C\. Feng, F\. Tian, F\. Shen, G\. Yu, H\. Zhang, J\. Li, M\. Chen, P\. Liu, W\. You, X\. T\. Zhang, X\. Li, X\. Yang, Y\. Deng, Y\. Huang, Y\. Li, Y\. Zhang, Z\. You, B\. Li, C\. Wan, H\. Hu, J\. Zhen, S\. Chen, S\. Yuan, X\. Zhang, Y\. Jiang, Y\. Zhou, Y\. Yang, B\. Li, B\. Ma, C\. Song, D\. Pang, G\. Hu, H\. Sun, K\. An, N\. Wang, S\. Gao, W\. Ji, W\. Li, W\. Sun, X\. Wen, Y\. Ren, Y\. Ma, Y\. Lu, B\. Wang, B\. Li, C\. Miao, C\. Liu, C\. Xu, D\. Shi, D\. Hu, D\. Wu, E\. Liu, G\. Huang, G\. Yan, H\. Zhang, H\. Nie, H\. Jia, H\. Zhou, J\. Sun, J\. Wu, J\. Wu, J\. Yang, J\. Yang, J\. Lin, K\. Li, L\. Yang, L\. Shi, L\. Zhou, L\. Gu, M\. Li, M\. Li, M\. Li, N\. Wu, Q\. Han, Q\. Tan, S\. Pang, S\. Fan, S\. Liu, T\. Cao, W\. Lu, W\. He, W\. Xie, X\. Zhao, X\. Li, Y\. Yu, Y\. Yang, Y\. Liu, Y\. Lu, Y\. Wang, Y\. Ding, Y\. Liang, Y\. Lu, Y\. Luo, Y\. Yin, Y\. Zhan, Y\. Zhang, Z\. Yang, Z\. Zhang, B\. Jiao, D\. Jiang, H\. Shum, J\. Chen, J\. Li, X\. Zhang, and Y\. ZhuStep\-audio 2 technical report\.External Links:2507\.16632,[Link](https://arxiv.org/abs/2507.16632)Cited by:[2nd item](https://arxiv.org/html/2608.29120#A4.I1.i2.p1.1)\.
- Wuet al\.\(2024\)J\. Wu, X\. Fan, B\. Lu, X\. Jiang, N\. Mesgarani, M\. Hasegawa\-Johnson, and M\. OstendorfJust asr \+ llm? a study on speech large language models’ ability to identify and understand speaker in spoken dialogue\.External Links:2409\.04927,[Link](https://arxiv.org/abs/2409.04927)Cited by:[Appendix C](https://arxiv.org/html/2608.29120#A3.SS0.SSS0.Px1.p2.1),[Figure 2](https://arxiv.org/html/2608.29120#S2.F2),[§4](https://arxiv.org/html/2608.29120#S4.SS0.SSS0.Px3.p2.1)\.
- Xuet al\.\(2025a\)J\. Xu, Z\. Guo, J\. He, H\. Hu, T\. He, S\. Bai, K\. Chen, J\. Wang, Y\. Fan, K\. Dang, B\. Zhang, X\. Wang, Y\. Chu, and J\. LinQwen2\.5\-omni technical report\.External Links:2503\.20215,[Link](https://arxiv.org/abs/2503.20215)Cited by:[1st item](https://arxiv.org/html/2608.29120#A4.I1.i1.p1.1)\.
- Xuet al\.\(2025b\)J\. Xu, Z\. Guo, H\. Hu, Y\. Chu, X\. Wang, J\. He, Y\. Wang, X\. Shi, T\. He, X\. Zhu, Y\. Lv, Y\. Wang, D\. Guo, H\. Wang, L\. Ma, P\. Zhang, X\. Zhang, H\. Hao, Z\. Guo, B\. Yang, B\. Zhang, Z\. Ma, X\. Wei, S\. Bai, K\. Chen, X\. Liu, P\. Wang, M\. Yang, D\. Liu, X\. Ren, B\. Zheng, R\. Men, F\. Zhou, B\. Yu, J\. Yang, L\. Yu, J\. Zhou, and J\. LinQwen3\-omni technical report\.External Links:2509\.17765,[Link](https://arxiv.org/abs/2509.17765)Cited by:[1st item](https://arxiv.org/html/2608.29120#A4.I1.i1.p1.1),[Appendix I](https://arxiv.org/html/2608.29120#A9.p1.1),[§1](https://arxiv.org/html/2608.29120#S1.p4.1)\.
- Xuet al\.\(2025c\)Z\. Xu, Y\. Wang, Y\. Huang, J\. Ye, H\. Zhuang, Z\. Song, L\. Gao, C\. Wang, Z\. Chen, Y\. Zhou, S\. Li, W\. Pan, Y\. Zhao, J\. Zhao, X\. Zhang, and X\. ChenSocialMaze: a benchmark for evaluating social reasoning in large language models\.External Links:2505\.23713,[Link](https://arxiv.org/abs/2505.23713)Cited by:[Appendix C](https://arxiv.org/html/2608.29120#A3.SS0.SSS0.Px3.p4.1),[§4](https://arxiv.org/html/2608.29120#S4.SS0.SSS0.Px3.p4.1)\.
- Yaoet al\.\(2024\)Y\. Yao, T\. Yu, A\. Zhang, C\. Wang, J\. Cui, H\. Zhu, T\. Cai, H\. Li, W\. Zhao, Z\. He, Q\. Chen, H\. Zhou, Z\. Zou, H\. Zhang, S\. Hu, Z\. Zheng, J\. Zhou, J\. Cai, X\. Han, G\. Zeng, D\. Li, Z\. Liu, and M\. SunMiniCPM\-v: a gpt\-4v level mllm on your phone\.External Links:2408\.01800,[Link](https://arxiv.org/abs/2408.01800)Cited by:[1st item](https://arxiv.org/html/2608.29120#A4.I1.i1.p1.1)\.
- Zhaoet al\.\(2024\)Y\. Zhao, J\. Huang, J\. Hu, X\. Wang, Y\. Mao, D\. Zhang, Z\. Jiang, Z\. Wu, B\. Ai, A\. Wang, W\. Zhou, and Y\. ChenSWIFT:a scalable lightweight infrastructure for fine\-tuning\.External Links:2408\.05517,[Link](https://arxiv.org/abs/2408.05517)Cited by:[Appendix F](https://arxiv.org/html/2608.29120#A6.p2.1)\.
- Zhouet al\.\(2025\)S\. Zhou, Y\. Zhou, Y\. He, X\. Zhou, J\. Wang, W\. Deng, and J\. ShuIndexTTS2: a breakthrough in emotionally expressive and duration\-controlled auto\-regressive zero\-shot text\-to\-speech\.External Links:2506\.21619,[Link](https://arxiv.org/abs/2506.21619)Cited by:[§B\.2](https://arxiv.org/html/2608.29120#A2.SS2.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2608.29120#S4.SS0.SSS0.Px2.p1.1),[§4](https://arxiv.org/html/2608.29120#S4.SS0.SSS0.Px3.p5.1)\.
- Zhouet al\.\(2026\)Y\. Zhou, G\. Zeng, X\. Liu, X\. Li, R\. Yu, J\. Gui, J\. Wu, Z\. Wang, X\. Shen, R\. Ye, Z\. Zhang, J\. Zhou, B\. Bai, W\. Sun, M\. Deng, Q\. Shi, Z\. Wu, and Z\. LiuVoxCPM2 technical report\.External Links:2606\.06928,[Link](https://arxiv.org/abs/2606.06928)Cited by:[Appendix A](https://arxiv.org/html/2608.29120#A1.SS0.SSS0.Px1.p1.1),[§3\.2](https://arxiv.org/html/2608.29120#S3.SS2.p1.1)\.
- Zouet al\.\(2023\)A\. Zou, Z\. Wang, N\. Carlini, M\. Nasr, J\. Z\. Kolter, and M\. FredriksonUniversal and transferable adversarial attacks on aligned language models\.External Links:2307\.15043,[Link](https://arxiv.org/abs/2307.15043)Cited by:[Appendix G](https://arxiv.org/html/2608.29120#A7.SS0.SSS0.Px5)\.

## Appendix ACuration of CASH & Statistics

##### CASH\-60K\.

We build on the VoxMM[Kwak et al\. \(2024\)](https://arxiv.org/html/2608.29120#bib.bib7)train split, extracting 5,242 multi\-speaker clips drawn from diverse multi\-speaker scenarios such as daily conversations, interviews and broadcast scenes\. From these originals we synthesize 14,180 voice\-swap variant clips, about 2\.7 variants per original on average, yielding 19,422 audio clips in total\. We then use both originals and variants to instantiate the task dimensions defined in Section[2](https://arxiv.org/html/2608.29120#S2), producing 59,762 training queries \(roughly three queries per audio clip\): 31,045 for attribution, 20,094 for reasoning, and 8,623 for discrimination\. All voice cloning is performed with VoxCPM2[Zhou et al\. \(2026\)](https://arxiv.org/html/2608.29120#bib.bib52), one of the current state\-of\-the\-art TTS systems\. To ensure the linguistic and acoustic fidelity of the cloned speech, we apply a two\-stage filtering protocol to every synthesized clip\. First, the clip is transcribed with Whisper\-large\-v3[Radford et al\. \(2022\)](https://arxiv.org/html/2608.29120#bib.bib18)and we retain only samples whose word error rate \(WER\) against the target transcript falls below 0\.1\. Second, we compute the cosine similarity between ECAPA\-TDNN[Desplanques et al\. \(2020\)](https://arxiv.org/html/2608.29120#bib.bib20)embeddings of the synthesized clip and the reference voice, and keep only samples whose speaker similarity exceeds 0\.7\.

## Appendix BCuration Process of HEAR Benchmark

This section details the data curation methodology for the HEAR benchmark\. The curation pipeline operates in two primary stages: an automated process to generate a comprehensive pool of candidate questions from three established datasets, followed by a rigorous manual review phase wherein human annotators validate and filter the questions to ensure high data quality\.

### B\.1Rule\-Based Question Construction

##### Source Data and Extraction\.

We leverage three publicly available datasets characterized by human\-verified text transcripts: AMI and ICSI \(multi\-party meetings\) and VoxMM \(CC\-BY\-4\.0\-licensed YouTube videos\)\. To maintain ground\-truth fidelity, we strictly rely on the original human annotations, explicitly excluding any AI\-generated transcriptions\. The source audio is segmented into 30\- to 150\-second clips \(averaging approximately9393seconds\), with each segment curated to contain at least two distinct speakers,4\.054\.05on average and ranging from22to1212\.

##### Voice References\.

All voice samples utilized as multiple\-choice options are sourced externally to the test clip\. These external reference samples consistently feature a single, non\-overlapping speaker and range in duration from 3 to 10 seconds\.

### B\.2Taxonomy

The automated pipeline generated three categories of questions:

##### Discrimination\.

The Discrimination dimension is generated mechanically from the transcript and speaker timing\.VC\(*voice cardinality*\) asks for the number of distinct speakers in a clip\.VL\(*voice localization*\) presents a reference voice and asks the model to select a time range on the basis of that speaker’s activity\. Each item takes one of two polarities: a presence variant, whose gold option is an interval in which the target speaker is active while the four distractors are intervals in which the speaker is not, and an absence variant, which reverses this so that the gold option is the only interval containing no speech from the target\.VCD\(*voice change/overlap detection*\) asks for the time range that contains either a turn change or an overlap; the generator enumerates all eligible 3–7 s windows and rejects any that violate the regime constraint\.

##### Attribution\.

Attribution items are produced by jointly sampling a 5\-way option set on top of the per\-clip speaker pool\.CVA\(*content→\\tovoice*\) shows a transcript span and asks which voice uttered it;VCA\(*voice→\\tocontent*\) shows a voice and asks which transcript span matches it\. Both come in non\-overlap and overlap variants determined by the surrounding speech context\. Distractors are drawn from the same clip to enforce voice\-level discrimination; each option’s voice reference is the external segment described above\.

##### Reasoning\.

Reasoning items follow a hallucination\-pair design\. For every original question, we synthesize a counterfactual variant in which a single answer\-relevant utterance has been re\-spoken in a*different*speaker’s voice via voice cloning, while leaving all other audio untouched\. Voice swaps are produced by IndexTTS 2[Zhou et al\. \(2025\)](https://arxiv.org/html/2608.29120#bib.bib33)conditioned on external 3–10 s voice references from the long source clip, and only swaps whose Whisper\-large\-v3[Radford et al\. \(2022\)](https://arxiv.org/html/2608.29120#bib.bib18)ASR transcript matches the target text withWER=0\{\\rm WER\}\\\!=\\\!0are admitted\. Three sub\-types are produced:QR\(Quantitative Reasoning\),IR\(Identity Reasoning\), andTR\(Temporal Reasoning\)\. A model that ignores voice identity must answer the original and the hallucinated variant identically and therefore fails the pair; this is the mechanism by which the Reasoning split exposes voice\-blind shortcutting\.

### B\.3Overlap Labeling Criteria

To further evaluate robustness under acoustically mixed conditions, we provide overlap\-labeled subsets for all taxonomy dimensions\. Overlap samples are annotated by human evaluators based on whether the overlapping speech segment is required to derive the correct answer, along with additional criteria specific to each dimension\. Below, we describe the overlap criteria and task\-specific objectives for each category\.

##### Voice Localization \(VL\-Overlap\)

The overlap subset is defined as samples where the target speaker’s active interval temporally overlaps with speech from at least one additional speaker\. The objective is to localize the target voice despite acoustic interference and concurrent speech activity\.

##### Voice Change Detection \(VCD\-Overlap\)

For overlap samples, the target interval contains simultaneous speech activity from multiple speakers\. Unlike the standard setting, which focuses on clean turn\-taking boundaries, this subset evaluates whether the model can detect overlap onset and offset boundaries within acoustically mixed regions\.

##### Content\-to\-Voice Attribution \(CVA\-Overlap\)

The overlap subset consists of utterances spoken during intervals containing concurrent speech from multiple speakers\. The model must correctly associate the queried textual utterance with its corresponding speaker identity despite acoustic mixtures\.

##### Voice\-to\-Content Attribution \(VCA\-Overlap\)

The overlap subset is defined by cases where the ground\-truth utterance occurs within overlapping speech intervals\. The task evaluates whether the model can correctly retrieve the linguistic content associated with a target voice under concurrent speech conditions\.

##### Identity Reasoning \(IR\-Overlap\)

IR\-Overlap samples contain anchor or candidate utterances that temporally intersect with overlapping speech regions\. The model must resolve speaker identity consistently despite acoustic interference\.

##### Temporal Reasoning \(TR\-Overlap\)

The overlap subset includes temporal reasoning chains involving utterances occurring during concurrent speech\. Models must correctly infer speaker\-conditioned temporal relationships while disentangling overlapping voices\.

##### Quantitative Reasoning \(QR\-Overlap\)

QR\-Overlap samples involve speaker statistics computed from sessions containing overlapping speech intervals\. The model must aggregate speaker\-specific evidence robustly despite acoustically mixed observations\.

![Refer to caption](https://arxiv.org/html/2608.29120v1/human_filtering.png)Figure 8:Screenshot of the interface shown to human annotators during verification\.

### B\.4Human Verification

Eight members of the authors served as annotators\. All were graduate students in artificial intelligence and comfortable reading English transcripts and listening to long\-form multi\-party audio\. Annotators worked through a custom local web tool that played the clip audio, displayed the word\-level transcript, exposed every option \(including each voice reference\), and offered Pass/Fail buttons together with a free\-text reason field as described in Figure[8](https://arxiv.org/html/2608.29120#A2.F8)\.

##### Filtering criteria\.

1. 1\.Solvability\.The reviewers are requested to answer the question correctly using only the clip audio, transcript, and the offered voice references\. Items that are ambiguous, under\-specified, or require information outside the clip are rejected\.
2. 2\.Unique correct answer\.Exactly one option must be correct and the remaining four distractors must each be falsifiable from the clip\. Items with multiple defensible answers, with options that are mutual paraphrases, or whose distractors collapse onto the gold answer are rejected\.
3. 3\.Voice\-identity preservation \(Reasoning split only\)\.For each TTS\-synthesized swap, the reviewer plays both the original and the swapped clip and confirms that the swapped voice is perceptually*the same person*as the conditioning voice reference, and that the synthesis is free of obvious artifacts \(mispronunciation, prosodic discontinuity at the splice, audible bandwidth change\)\. Pairs that fail either criterion are rejected even if WER=0=\\\!0\.
4. 4\.Transcript–option consistency\.Although AMI, ICSI, and VoxMM are all distributed with human\-produced transcripts, boundary errors \(off\-by\-a\-word starts/ends, missed back\-channels, mis\-attributed overlapping speech\) still survive into our candidate pool\. Any item whose option text or query text fails to match the audio at the specified timestamps, even by a single word, is filtered out\.

After the verification pass,887887clips and2,3952\{,\}395questions survive from a candidate pool of roughly three times that size\. The resulting benchmark therefore reflects two compounded layers of human quality control: the original transcripts of the three source corpora, and the per\-question audit described above\.

### B\.5Benchmark Statistics

Table 5:HEAR per\-domain composition\.\#Src\.: distinct source recordings\.\#Spk\.: speakers, deduplicated within each corpus’s identity namespace\.Dur\.: average clip duration, in seconds\.Hours: total speech\-clip\-hours\.Table[5](https://arxiv.org/html/2608.29120#A2.T5)reports the per\-domain composition of HEAR\. The benchmark covers887887clips, totals∼\\sim22\.822\.8h of audio, and exposes a model to290290unique speakers in aggregate\. VoxMM contributes the bulk of the speaker diversity \(245245speakers across3434CC\-BY\-4\.0\-licensed videos \(audio\) in1111genre categories\), and AMI and ICSI contribute longer, denser meeting clips \(3\.93\.9–4\.74\.7speakers/clip on average\)\.

##### Domain and dimension distribution\.

Figure[9](https://arxiv.org/html/2608.29120#A2.F9)\(left\) shows that no single domain dominates the benchmark: VoxMM accounts for40\.6%40\.6\\%of clips, ICSI for32\.2%32\.2\\%, and AMI for27\.2%27\.2\\%\. The skew toward meeting\-style audio is intentional and follows the original corpus sizes; we counter\-balance it on the question side by saturating the VoxMM category budget \(Figure[11](https://arxiv.org/html/2608.29120#A2.F11)\)\. Figure[9](https://arxiv.org/html/2608.29120#A2.F9)\(right\) reports the question count per evaluation dimension:627627Discrimination,608608Attribution, and1,1601\{,\}160Reasoning items \(original\+\+swapped\), for a total of2,3952\{,\}395questions\.

Figure 9:HEAR composition\. Left: clip distribution across the three source domains \(N=887N\\\!=\\\!887\)\. Right: question distribution across the three evaluation dimensions \(N=2,395N\\\!=\\\!2\{,\}395\)\.
##### Sub\-dimension breakdown\.

Figure[10](https://arxiv.org/html/2608.29120#A2.F10)unrolls the three top\-level dimensions into the eight sub\-dimensions that the benchmark scores\. Discrimination splits into VC \(7\.9%7\.9\\%\), VL \(9\.9%9\.9\\%\), and VCD \(8\.4%8\.4\\%\); Attribution splits into CVA \(12\.8%12\.8\\%\) and VCA \(12\.6%12\.6\\%\); Reasoning splits into QR \(23\.4%23\.4\\%, the dominant Reasoning sub\-type\), IR \(15\.4%15\.4\\%\), and TR \(9\.6%9\.6\\%\)\.

Figure 10:Sub\-dimension distribution of HEAR questions \(N=2,395N\\\!=\\\!2\{,\}395\)\. VC/VL/VCD make up Discrimination \(top three slices\), CVA/VCA make up Attribution, and QR/IR/TR make up Reasoning\.
##### VoxMM category coverage\.

Figure[11](https://arxiv.org/html/2608.29120#A2.F11)reports the genre composition of the VoxMM slice:*Remote*\(104104clips\) and*Documentary*\(7272\) are the two heads, followed by*Entertainment*,*Commercial*,*Interview*, and*News*\(3232–4040each\), with*Politics*,*Conversation*,*Lecture*,*Presentation*, and*Sports*forming a long tail\.

Figure 11:Distribution of VoxMM clips in HEAR by genre category \(N=360N\\\!=\\\!360\)\.Figure 12:Per\-domain summary statistics for HEAR\.Figure 13:Statistics of clip duration for HEAR\.
##### Per\-domain summary and clip\-level distributions\.

Figure[12](https://arxiv.org/html/2608.29120#A2.F12)consolidates four headline statistics across the three domains: clip count, distinct source recordings, distinct speakers, and average clip duration\. Clip durations are evenly distributed as described in Figure[13](https://arxiv.org/html/2608.29120#A2.F13)\.

VCVOICE COUNTING▶\\blacktrianglerightMain AudioListen to the main audio and answer the following multiple\-choice question\. How many distinct speakers are present in this audio clip?Options: \(A\) 2 \(B\) 3 \(C\) 4 \(D\) 5 \(E\) 6

VLVOICE LOCALISATION▶\\blacktrianglerightMain Audio \+ Reference VoiceListen to the main audio, followed by a reference voice of the target speaker… Given the reference voice, select the time range where this speaker appears\.Options A–E are time ranges, e\.g\., \[3\.0s – 5\.5s\]\.

VLVOICE LOCALISATION▶\\blacktrianglerightMain Audio \+ Reference VoiceListen to the main audio, followed by a reference voice… Given the reference voice, select the time range where this speaker does NOT appear\.Options A–E are time ranges\.

VCDVOICE\-CHANGE DETECTION▶\\blacktrianglerightMain AudioListen to the main audio and answer the following multiple\-choice question\. In which time range does a speaker change \(voice transition\) occur?Options A–E are time ranges\.

VCDVOICE\-CHANGE DETECTION▶\\blacktrianglerightMain AudioListen to the main audio and answer the following multiple\-choice question\. In which time range does speaker overlap \(multiple speakers talking simultaneously\) occur?Options A–E are time ranges\.

Figure 14:Sample queries ofDiscriminationdimension in HEAR benchmark\.CVACONTENT\-TO\-VOICE ATTRIBUTION▶\\blacktrianglerightMain Audio \+ Voice\-Options \(A–E\)Listen to the main audio, followed by the voice options audio… Which speaker said:⟨X⟩\\langle X\\rangle?Options \(A\)–\(E\) correspond to the voice options\.

VCAVOICE\-TO\-CONTENT ATTRIBUTION▶\\blacktrianglerightMain Audio \+ Reference VoiceListen to the main audio, followed by a reference voice of the target speaker… Which of the following utterances was spoken by the given voice?\(A\)⟨X1⟩\\langle X\_\{1\}\\rangle\(B\)⟨X2⟩\\langle X\_\{2\}\\rangle\(C\)⟨X3⟩\\langle X\_\{3\}\\rangle\(D\)⟨X4⟩\\langle X\_\{4\}\\rangle\(E\)⟨X5⟩\\langle X\_\{5\}\\rangle

Figure 15:Sample queries ofAttributiondimension in HEAR benchmark\.QR1QUANTITATIVE REASONING▶\\blacktrianglerightMain AudioListen to the main audio and answer the following multiple\-choice question\. How many speakers mentioned⟨X⟩\\langle X\\rangle?Options: \(A\) 1 \(B\) 2 \(C\) 3 \(D\) 4 \(E\) 5

QR2QUANTITATIVE REASONING▶\\blacktrianglerightMain AudioListen to the main audio and answer the following multiple\-choice question\. Which of the following was said by the speaker with thekk\-th longest total speaking time?\(A\)⟨X1⟩\\langle X\_\{1\}\\rangle\(B\)⟨X2⟩\\langle X\_\{2\}\\rangle\(C\)⟨X3⟩\\langle X\_\{3\}\\rangle\(D\)⟨X4⟩\\langle X\_\{4\}\\rangle\(E\)⟨X5⟩\\langle X\_\{5\}\\rangle

Figure 16:Sample queries ofReasoningdimension in HEAR benchmark\.TR1TEMPORAL REASONING▶\\blacktrianglerightMain AudioListen to the main audio and answer the following multiple\-choice question\. What was the very first utterance spoken by the person who said⟨Y⟩\\langle Y\\rangle?Options A–E are candidate utterances\.

TR1TEMPORAL REASONING▶\\blacktrianglerightMain AudioListen to the main audio and answer the following multiple\-choice question\. What was the very last utterance spoken by the person who said⟨Y⟩\\langle Y\\rangle?Options A–E are candidate utterances\.

TR2TEMPORAL REASONING▶\\blacktrianglerightMain AudioListen to the main audio and answer the following multiple\-choice question\. After someone said⟨X⟩\\langle X\\rangle, what was the first thing said by the person who said⟨Y⟩\\langle Y\\rangle?Options A–E are candidate utterances\.

IRIDENTITY REASONING▶\\blacktrianglerightMain AudioListen to the main audio and answer the following multiple\-choice question\. Which of the following was also said by the person who said⟨Y⟩\\langle Y\\rangle?Options A–E are candidate utterances\.

Figure 17:Sample queries ofReasoningdimension in HEAR benchmark\.

## Appendix CDetails of Benchmarks Where Speaker Attribution Is Essential

We provide additional details on the construction and statistics of the three voice\-dependent benchmarks used in our evaluation: WDYL \(*What Do You Like?*\), GAOKAO, and FTS \(*Find the Spy*\)\. All three benchmarks are designed such that correctly answering the question requires not only understanding the spoken content, but also identifying and tracking*who*is speaking\.

##### WDYL \(What Do You Like?\)\.

WDYL evaluates whether a model can resolve first\-person references using speaker identity\. Each example contains a conversation between two speakers, followed by a first\-person question such as “What is my favorite city?” spoken in the cloned voice of one of the two speakers\. Because the referent of “my” depends entirely on the identity of the question speaker, the correct answer cannot be determined from the linguistic content of the question alone\.

The benchmark contains 793 questions over 793 audio clips, totaling 2\.36 hours of audio\. The average clip duration is 10\.7 s, with a range of 8\.1–13\.2 s\. Each question has three answer choices: one corresponding to each speaker’s stated preference and one distractor not mentioned by either speaker\. Since the distractor can be eliminated from the transcript alone, the effective chance accuracy is 50%; accordingly, we use 50% as the random\-choice baseline in Table[2](https://arxiv.org/html/2608.29120#S5.T2)\. The dialogue audio is taken from[Wu et al\. \(2024\)](https://arxiv.org/html/2608.29120#bib.bib9)\. For each example, we append a 0\.3 s pause followed by a question synthesized with IndexTTS 2, using a cloned voice from one of the speakers in the preceding dialogue\.

We initially synthesized 1,000 examples and retained 793 after speaker\-similarity quality control using ECAPA embeddings, requiring a speaker embedding cosine similarity \(SECS\) of at least 0\.70\. The retained examples have a mean SECS of 0\.778\. The cloned question voice corresponds to the*asker*in 476 examples and the*answerer*in 317 examples\.

##### GAOKAO\.

GAOKAO tests speaker\-conditioned interpretation in natural English listening\-comprehension conversations\. Each example consists of a short English dialogue followed by a first\-person question synthesized in the cloned voice of one of the speakers\.

The benchmark contains 93 questions over 93 clips, totaling 0\.52 hours \(30\.9 minutes\) of audio\. The average duration is 20\.0 s, ranging from 12\.0 s to 29\.9 s\. Each question has two answer choices, corresponding to a random\-guess accuracy of 50%\. The source audio consists of Gaokao English short\-conversation recordings from the ICQ subset of[Hu et al\. \(2024\)](https://arxiv.org/html/2608.29120#bib.bib54)\. We append a 0\.35 s pause and an IndexTTS 2\-generated question whose voice is cloned from a speech segment of the target speaker within the same dialogue\.

We synthesized 97 examples and retained 93 after quality control\. Examples were required to achieve an ECAPA speaker\-similarity score of at least 0\.7 and a Whisper word error rate \(WER\) of at most 0\.20\. The target speaker is male in 46 examples and female in 47 examples\. Gold\-answer positions are distributed as A: 40 and B: 53\.

##### FTS \(Find the Spy\)\.

FTS evaluates multi\-speaker voice tracking in a social\-deduction setting\. Each example is a 3–6\-player Find\-the\-Spy game in which every player receives the same secret word except for one designated spy\. The model observes the players’ spoken clues and must identify which participant is the spy\.

Crucially, players first introduce themselves by name, after which they provide their clues in a*different, randomly shuffled order*and without explicitly stating their names\. Consequently, associating each clue with the corresponding player requires tracking speaker identity across temporally separated turns; textual content alone does not reveal which player produced which clue\.

The benchmark contains 400 questions over 400 clips, totaling 3\.67 hours of audio\. The mean duration is 33\.1 s, with clips ranging from 17\.2 s to 56\.6 s\. There are 100 games for each player count from three to six, yielding chance accuracies of 33\.3%, 25\.0%, 20\.0%, and 16\.7%, respectively\.

Game configurations and clues are generated using the SocialMaze[Xu et al\. \(2025c\)](https://arxiv.org/html/2608.29120#bib.bib21)framework\. Each player’s utterances are synthesized with IndexTTS 2, while keeping a single reference voice fixed for that player throughout the game\.

## Appendix DExperiment Details

As introduced in Section[4](https://arxiv.org/html/2608.29120#S4), we evaluate 20 leading SLMs categorized into three distinct families\. The detailed list of models and their references is as follows:

- •Omni\-modality models: MiniCPM\-o series[Yao et al\. \(2024\)](https://arxiv.org/html/2608.29120#bib.bib49);[Cui et al\. \(2026\)](https://arxiv.org/html/2608.29120#bib.bib27), Qwen2\.5\-Omni series[Xu et al\. \(2025a\)](https://arxiv.org/html/2608.29120#bib.bib17), Qwen3\-Omni\-30B\-A3B series[Xu et al\. \(2025b\)](https://arxiv.org/html/2608.29120#bib.bib19), Gemma\-4\-E4B\-it[Team et al\. \(2026a\)](https://arxiv.org/html/2608.29120#bib.bib46), and Phi\-4\-Multimodal[Microsoft et al\. \(2025\)](https://arxiv.org/html/2608.29120#bib.bib53)\.
- •Spoken or Audio language models: MiDashengLM[Dinkel et al\. \(2026\)](https://arxiv.org/html/2608.29120#bib.bib30), Voxtral[Liu et al\. \(2025\)](https://arxiv.org/html/2608.29120#bib.bib31), Fun\-Audio\-Chat[Team et al\. \(2026b\)](https://arxiv.org/html/2608.29120#bib.bib43), Kimi\-Audio[KimiTeam et al\. \(2025\)](https://arxiv.org/html/2608.29120#bib.bib29), Qwen2\-Audio[Chu et al\. \(2024\)](https://arxiv.org/html/2608.29120#bib.bib25), Audio\-Flamingo 3[Goel et al\. \(2025\)](https://arxiv.org/html/2608.29120#bib.bib26), and Step\-Audio series[Tian et al\. \(2025\)](https://arxiv.org/html/2608.29120#bib.bib28);[Wu et al\. \(2025\)](https://arxiv.org/html/2608.29120#bib.bib55)\.
- •Proprietary models: Gemini\-3 series[Google DeepMind \(2026b\)](https://arxiv.org/html/2608.29120#bib.bib44);[Google DeepMind \(2026a\)](https://arxiv.org/html/2608.29120#bib.bib45)and GPT\-4o\-audio\-preview[OpenAI \(2024\)](https://arxiv.org/html/2608.29120#bib.bib48)\.

For every open\-source baseline, we adopt the inference hyperparameters specified in its official repository; for proprietary systems, we use the default API settings\. The concrete experimental settings are comprehensively described in Table[7](https://arxiv.org/html/2608.29120#A10.T7)\. All open\-weight baselines are served through vLLM or the Hugging Face inference path, while the three closed\-source endpoints \(gpt\-4o\-audio\-preview,gemini\-3\-flash\-preview,gemini\-3\.1\-pro\-preview\) are queried with default setting\. We report the value used in our configuration for reproducibility\. Table[7](https://arxiv.org/html/2608.29120#A10.T7)summarizes the per\-model defaults\.

## Appendix EReward Composition Details

This appendix details the coefficients of the composite reward \(Section[3\.3](https://arxiv.org/html/2608.29120#S3.SS3)\) and the dynamic\-programming implementation of the speaker\-order rewardRordR\_\{\\mathrm\{ord\}\}\.

### E\.1Per\-term Coefficients

The five reward terms are linearly combined with fixed coefficientsλx\\lambda\_\{x\}:

R⁡\(Yi,τ⋆,a⋆\)\\displaystyle R\(Y\_\{i\};\\tau^\{\\star\},a^\{\\star\}\)=λans​Rans​\(a^i,a⋆\)\\displaystyle=\\lambda\_\{\\mathrm\{ans\}\}R\_\{\\mathrm\{ans\}\}\(\\hat\{a\}\_\{i\};a^\{\\star\}\)\(1\)\+λfmt​Rfmt​\(Yi\)\+λcp​Rcp​\(τ^i,τ⋆\)\\displaystyle\+\\lambda\_\{\\mathrm\{fmt\}\}R\_\{\\mathrm\{fmt\}\}\(Y\_\{i\}\)\+\\lambda\_\{\\mathrm\{cp\}\}R\_\{\\mathrm\{cp\}\}\(\\hat\{\\tau\}\_\{i\};\\tau^\{\\star\}\)\+λcnt​Rcnt​\(τ^i,τ⋆\)\+λord​Rord​\(τ^i,τ⋆\)\.\\displaystyle\+\\lambda\_\{\\mathrm\{cnt\}\}R\_\{\\mathrm\{cnt\}\}\(\\hat\{\\tau\}\_\{i\};\\tau^\{\\star\}\)\+\\lambda\_\{\\mathrm\{ord\}\}R\_\{\\mathrm\{ord\}\}\(\\hat\{\\tau\}\_\{i\};\\tau^\{\\star\}\)\.
Table[6](https://arxiv.org/html/2608.29120#A5.T6)lists the coefficients used in our experiments\.

Table 6:Coefficients for the composite reward terms\.
### E\.2Speaker\-Order Reward via LCS

To computeRordR\_\{\\mathrm\{ord\}\}, we extract the sequence of speaker labels in order of appearance from both the ground\-truth \(Q=\(q1,…,qm\)Q=\(q\_\{1\},\\dots,q\_\{m\}\)\) and the prediction \(P=\(p1,…,pn\)P=\(p\_\{1\},\\dots,p\_\{n\}\)\)\.

We defineRordR\_\{\\mathrm\{ord\}\}as the length\-normalized Longest Common Subsequence \(LCS\):

Rord​\(τ^i,τ⋆\)=LCS⁡\(P,Q\)max⁡\(\|P\|,\|Q\|,1\)∈\[0,1\]\.R\_\{\\mathrm\{ord\}\}\(\\hat\{\\tau\}\_\{i\};\\tau^\{\\star\}\)=\\frac\{\\mathrm\{LCS\}\(P,Q\)\}\{\\max\(\|P\|,\|Q\|,1\)\}\\in\[0,1\]\.\(2\)
Algorithm 1Speaker\-Order RewardRordR\_\{\\mathrm\{ord\}\}\(Length\-normalized LCS\)1:Predicted speaker labels

P=\(p1,…,pn\)P=\(p\_\{1\},\\dots,p\_\{n\}\); GT speaker labels

Q=\(q1,…,qm\)Q=\(q\_\{1\},\\dots,q\_\{m\}\)
2:Reward

r∈\[0,1\]r\\in\[0,1\]
3:

D\[i,0\]←0,D\[0,j\]←0∀i,jD\[i,0\]\\leftarrow 0,\\ D\[0,j\]\\leftarrow 0\\quad\\forall i,j
4:for

i=1i=1to

nndo

5:for

j=1j=1to

mmdo

6:if

pi=qjp\_\{i\}=q\_\{j\}then

7:

D⁡\[i,j\]←D⁡\[i−1,j−1\]\+1D\[i,j\]\\leftarrow D\[i\\\!\-\\\!1,j\\\!\-\\\!1\]\+1
8:else

9:

D⁡\[i,j\]←max⁡\(D⁡\[i−1,j\],D⁡\[i,j−1\]\)D\[i,j\]\\leftarrow\\max\\\!\\bigl\(D\[i\\\!\-\\\!1,j\],\\,D\[i,j\\\!\-\\\!1\]\\bigr\)
10:endif

11:endfor

12:endfor

13:return

D⁡\[n,m\]/max⁡\(n,m,1\)D\[n,m\]\\,/\\,\\max\(n,m,1\)

LCS⁡\(P,Q\)\\mathrm\{LCS\}\(P,Q\)represents the longest sequence appearing in bothPPandQQin the same relative order\. Algorithm[1](https://arxiv.org/html/2608.29120#alg1)outlines theO⁡\(n​m\)O\(nm\)dynamic\-programming procedure\.

##### From DP to Speaker Turns\.

The LCS algorithm directly aligns predicted \(PP\) and ground\-truth \(QQ\) speaker sequences\. Whenpi=qjp\_\{i\}=q\_\{j\}, the DP matches a correctly attributed turn in temporal order \(\+1\+1\)\. Whenpi≠qjp\_\{i\}\\neq q\_\{j\}, themax\\maxoperation resolves misalignments by skipping either a hallucinated prediction or a missing GT turn\. This ensures that isolated errors do not disrupt the global turn alignment\.

##### Example\.

Suppose the maximum rewardRordR\_\{\\mathrm\{ord\}\}is normalized to 1 in this example\. Given a 5\-turn dialogueQ=\(s1,s2,s1,s3,s2\)Q=\(s\_\{1\},s\_\{2\},s\_\{1\},s\_\{3\},s\_\{2\}\), our metric gracefully penalizes conversational errors\.*Case 1 \(misattribution\):*PredictingP1=\(s1,s2,𝐬𝟐,s3,s2\)P\_\{1\}=\(s\_\{1\},s\_\{2\},\\mathbf\{s\_\{2\}\},s\_\{3\},s\_\{2\}\)yieldsLCS=4\\mathrm\{LCS\}=4andRord=4/5=0\.8R\_\{\\mathrm\{ord\}\}=4/5=0\.8\.*Case 2 \(insertion\):*Emitting an extra turn \(\|P2\|=6\|P\_\{2\}\|=6\) keepsLCS=5\\mathrm\{LCS\}=5but increases the denominator, yieldingRord=5/max⁡\(6,5\)≈0\.83R\_\{\\mathrm\{ord\}\}=5/\\max\(6,5\)\\approx 0\.83\.*Case 3 \(deletion\):*Missing a turn \(\|P3\|=4\|P\_\{3\}\|=4\) yieldsRord=4/5=0\.8R\_\{\\mathrm\{ord\}\}=4/5=0\.8\. Thus, misattributions, insertions, and deletions all incur proportional penalties, providing a robust turn\-alignment signal\.

## Appendix FTraining Details

We fine\-tune Qwen3\-Omni\-30B\-A3B\-Instruct with GRPO on4×4\{\\times\}NVIDIA H200 GPUs\. LoRA adapters are applied to all language\-model linear projections with rankr=64r\{=\}64,α=128\\alpha\{=\}128, and dropout00, while the audio/vision encoders and modality aligners remain frozen\. Expert parallelism \(EP==4\) is used to fit the MoE backbone across GPUs\.

Training runs for11epoch with an effective batch size of6464, formed by1616unique prompts and44GRPO rollouts per prompt\. We use bfloat16 precision and a learning rate of5×10−65\{\\times\}10^\{\-6\}with cosine scheduling,239239warmup iterations, and cosine decay to5×10−75\{\\times\}10^\{\-7\}\. Rollouts are generated with vLLM[Kwon et al\. \(2023\)](https://arxiv.org/html/2608.29120#bib.bib22)at temperature1\.01\.0using sequence\-level importance sampling, asymmetric PPO clipping \(εlow=3×10−4\\varepsilon\_\{\\text\{low\}\}\{=\}3\{\\times\}10^\{\-4\},εhigh=4×10−4\\varepsilon\_\{\\text\{high\}\}\{=\}4\{\\times\}10^\{\-4\}\), andβ=0\\beta\{=\}0KL regularization\. Training is implemented on top of theMegatron\-SWIFT[Zhao et al\. \(2024\)](https://arxiv.org/html/2608.29120#bib.bib23)backend, which provides the expert\-parallel MoE training support required for the Qwen3\-Omni\-30B\-A3B backbone\.

## Appendix GDescriptions of VoiceBench

To evaluate the dyadic conversational capabilities of our fine\-tuned A2R models, we utilize VoiceBench[Chen et al\. \(2024\)](https://arxiv.org/html/2608.29120#bib.bib32), a comprehensive benchmark featuring both real and synthetic spoken instructions\. Following standard protocols, AlpacaEval, CommonEval, and WildVoice samples are assessed using gpt\-5\.4\-mini[OpenAI \(2026b\)](https://arxiv.org/html/2608.29120#bib.bib51)\. We evaluate our models across five key dimensions:

##### Open\-Ended QA \(AlpacaEval[Dubois et al\. \(2025\)](https://arxiv.org/html/2608.29120#bib.bib34), CommonEval[Ardila et al\. \(2020\)](https://arxiv.org/html/2608.29120#bib.bib35), WildVoice\)

assesses general knowledge and conversational capability\. WhileAlpacaEvaluses synthetic instructions to measure response quality,CommonEvalandWildVoice\(1,000 human\-recorded samples\) evaluate information\-seeking capabilities under diverse, real\-world human speech conditions\.

##### Reference\-Based QA \(BBH\)[Suzgun et al\. \(2022\)](https://arxiv.org/html/2608.29120#bib.bib47)

measures reasoning accuracy against reference answers\. We use a human\-recorded spoken version ofBBH\(BIG\-Bench Hard\) to evaluate multi\-step reasoning \(e\.g\., logical, arithmetic, and temporal\) on long and complex real spoken instructions\.

##### Multiple\-Choice QA \(MMSU[Wang et al\. \(2026\)](https://arxiv.org/html/2608.29120#bib.bib56), OpenBookQA\)[Mihaylov et al\. \(2018\)](https://arxiv.org/html/2608.29120#bib.bib37)

evaluates general knowledge, elementary science, and common\-sense reasoning by requiring models to select the correct option from a predefined set\.

##### Instruction Following \(IFEval\)[Gao et al\. \(2025\)](https://arxiv.org/html/2608.29120#bib.bib38)

tests the model’s strict adherence to specific structural and formatting constraints using synthetic spoken instructions\.

##### Safety \(AdvBench\)[Zou et al\. \(2023\)](https://arxiv.org/html/2608.29120#bib.bib36)

measures the model’s refusal rate against malicious prompts using synthetic speech, operating on the premise that a safe assistant must firmly reject harmful instructions\.

## Appendix HPaired Reasoning Accuracy

For the reasoning axis, we evaluate each original clip jointly with its semantic\-hallucination counterpart\. Letcic\_\{i\}andc~i\\tilde\{c\}\_\{i\}denote binary correctness indicators for the original and hallucinated clips in pairii, respectively\. We define paired reasoning accuracy as

Rpair=1NR​∑i=1NRci​c~i,R\_\{\\mathrm\{pair\}\}=\\frac\{1\}\{N\_\{R\}\}\\sum\_\{i=1\}^\{N\_\{R\}\}c\_\{i\}\\tilde\{c\}\_\{i\},\(3\)whereNRN\_\{R\}is the number of reasoning clip pairs\. Thus, each pair constitutes a single evaluation unit and contributes one correct prediction only when both clips are answered correctly\.

Discrimination and attribution are evaluated with standard accuracy, whereas reasoning usesRpairR\_\{\\mathrm\{pair\}\}\. To account for the different numbers of evaluation units across the three axes, we compute the overall average by weighting each axis by its corresponding number of evaluation units:

Avg\.=ND​D\+NA​A\+NR​RpairND\+NA\+NR,\\mathrm\{Avg\.\}=\\frac\{N\_\{D\}D\+N\_\{A\}A\+N\_\{R\}R\_\{\\mathrm\{pair\}\}\}\{N\_\{D\}\+N\_\{A\}\+N\_\{R\}\},\(4\)whereNDN\_\{D\}andNAN\_\{A\}denote the numbers of discrimination and attribution examples, respectively, andNRN\_\{R\}denotes the number of reasoning clip pairs\. In particular, each reasoning pair is counted as one evaluation unit rather than as two individual clips\.

## Appendix ILicenses and Intended Use

We use all assets in accordance with their stated terms\. AMI[Carletta et al\. \(2005\)](https://arxiv.org/html/2608.29120#bib.bib6), ICSI[Janin et al\. \(2003\)](https://arxiv.org/html/2608.29120#bib.bib8), and VoxMM[Kwak et al\. \(2024\)](https://arxiv.org/html/2608.29120#bib.bib7)are released under CC BY 4\.0\. Our base model, Qwen3\-Omni\-30B\-A3B\-Instruct[Xu et al\. \(2025b\)](https://arxiv.org/html/2608.29120#bib.bib19), is released under Apache 2\.0\.

## Appendix JAI Tool Use

We used AI\-based writing assistance tools only for language editing, grammar correction, and improving the clarity of the manuscript\.

Table 7:Decoding hyperparameters used for the open\-weight baselines in our HEAR evaluation\.“–” denotes parameters left unset, using the vLLM engine’s default value \(1\.0\)\([Kwon et al\., 2023](https://arxiv.org/html/2608.29120#bib.bib22)\)\. For A2R, we use the same parameters as the Qwen3\-Omni\-30B\-A3B\-Instruct model\.A2R \(Ours\)Reason in two stages inside<reasoning\>\.\.\.</reasoning\>\. First, transcribe the multi\-speaker conversation in the main audio inside<transcript\>\.\.\.</transcript\>placed at the very start of<reasoning\>\. Use the format‘Speaker 1: \.\.\.\\nSpeaker 2: \.\.\.‘, labeling speakers in the order they first appear and joining each speaker’s utterances on a single line in chronological order\. Then continue reasoning grounded in that transcript, and finally provide the answer in JSON inside<answer\>\.\.\.</answer\>\.Example:<reasoning\><transcript\>Speaker 1: \.\.\.\\nSpeaker 2: \.\.\.</transcript\> \.\.\.your detailed reasoning grounded in the transcript\.\.\.</reasoning\><answer\>\{"Answer": "\(A\)"\}</answer\>

AblationReason briefly inside<reasoning\>\.\.\.</reasoning\>, then output your answer in JSON inside<answer\>\.\.\.</answer\>\.Example:<reasoning\>Your brief reasoning here\.</reasoning\><answer\>\{"Answer": "\(A\)"\}</answer\>

Figure 18:Instruction prompts used for GRPO experiments\.Inference Prompt \(HEAR\)VC / VCD / IR /QR / TRListen to the main audio and answer the following multiple\-choice question\.VL / VCAListen to the main audio, followed by a reference voice of the target speaker, and answer the following multiple\-choice question\.CVAListen to the main audio, followed by the voice options audio where each option \(A through E\) is announced before its voice sample, and answer the following multiple\-choice question\.Reason in two stages inside<reasoning\>\.\.\.</reasoning\>\. First, transcribe the multi\-speaker conversation in the main audio inside<transcript\>\.\.\.</transcript\>placed at the very start of<reasoning\>\. Use the format‘Speaker 1: \.\.\.\\nSpeaker 2: \.\.\.‘, labeling speakers in the order they first appear and joining each speaker’s utterances on a single line in chronological order\. Then continue reasoning grounded in that transcript, and finally provide the answer in JSON inside<answer\>\.\.\.</answer\>\. Example:<reasoning\><transcript\>Speaker 1: \.\.\.\\nSpeaker 2: \.\.\.</transcript\>
\.\.\.reasoning\.\.\.</reasoning\><answer\>\{"Answer": "\(A\) example text"\}</answer\>Figure 19:Instruction prompt template used for HEAR benchmark, consisting of a task\-specific listening prefix and a common reasoning suffix\.Inference Prompt \(What Do You Like, Gaokao\)Reason in two stages inside<reasoning\>\.\.\.</reasoning\>\. First, transcribe the multi\-speaker conversation in the main audio inside<transcript\>\.\.\.</transcript\>placed at the very start of<reasoning\>\. Use the format‘Speaker 1: \.\.\.\\nSpeaker 2: \.\.\.‘, labeling speakers in the order they first appear and joining each speaker’s utterances on a single line in chronological order\. Then continue reasoning grounded in that transcript, and finally provide the answer in JSON inside<answer\>\.\.\.</answer\>\. Example:<reasoning\><transcript\>Speaker 1: \.\.\.\\nSpeaker 2: \.\.\.</transcript\>
\.\.\.reasoning\.\.\.</reasoning\><answer\>\{"Answer": "\(A\) example text"\}</answer\>Figure 20:Instruction prompt template used for What Do You Like and Gaokao benchmarks, consisting of a task\-specific listening prefix and a common reasoning suffix\.Instruction Prompt \(Find the SPY\)You are playing the game “Find the Spy\.” There are 3 players: \{Player Name 1\}, \{Player Name 2\} …\. All players except one received the same word; one player, the spy, received a different word\. Each player gives a short description of their word without saying it directly\. Your task is to identify which player is describing a different word\.Based on a recording of the game, decide which player is the spy\.Options:
\(A\) Rachel
\(B\) Benjamin
\(C\) Grace
\(D\) …Reason in two stages inside<reasoning\>\.\.\.</reasoning\>\. First, transcribe the multi\-speaker conversation in the main audio inside<transcript\>\.\.\.</transcript\>placed at the very start of<reasoning\>\. Use the format‘Speaker 1: \.\.\.\\nSpeaker 2: \.\.\.‘, labeling speakers in the order they first appear and joining each speaker’s utterances on a single line in chronological order\. Then continue reasoning grounded in that transcript, and finally provide the answer in JSON inside<answer\>\.\.\.</answer\>\. Example:<reasoning\><transcript\>Speaker 1: \.\.\.\\nSpeaker 2: \.\.\.</transcript\>
\.\.\.reasoning\.\.\.</reasoning\><answer\>\{"Answer": "\(A\) example text"\}</answer\>Figure 21:Instruction prompt used for the Find the Spy benchmark\.