Rationale-Guided Learning for Multimodal Emotion Recognition

# Rationale-Guided Learning for Multimodal Emotion Recognition
Source: [https://arxiv.org/html/2608.10448](https://arxiv.org/html/2608.10448)
###### Abstract

Multimodal emotion recognition in conversation \(MERC\) requires understanding complex interactions between verbal and non\-verbal cues\. However, most existing approaches fundamentally treat this as a direct input\-output \(multimodal cues\-emotion labels\) mapping problem, overlooking the causal reasoning that humans use when interpreting emotions\. We propose rationale\-guided learning \(RGL\), a novel framework that transforms MERC into a cognitively\-inspired reasoning task\. Based on dual\-process theory, we decompose emotional reasoning into three facets:Intuitive\(immediate perception,System 1\),Contextual\(situational analysis,System 2\), andIntegrative\(synthesis of both\)\. We leverage a Multimodal Large Language Model \(MLLM\) to generate structured rationales, which are encoded as rationale banks to guide model training via aligning internal representations with human\-like reasoning patterns\. Our final model operates without any MLLM overheads at inference time\. Experimental results show that RGL achieves state\-of\-the\-art performance on the IEMOCAP and MELD benchmarks\. Further, for interpretation, we demonstrate that the model’s internal features effectively retrieve semantically correct rationales for unseen test samples, validating its rationale reasoning capabilities\.

Index Terms—Multimodal emotion recognition, rationale\-guided learning, reasoning patterns, representation learning, multimodal large language model

## 1INTRODUCTION

Multimodal emotion recognition in conversation \(MERC\) aims to identify the emotional state of speakers within dialogues by leveraging text, audio, and video streams\[[19](https://arxiv.org/html/2608.10448#bib.bib3)\]\. Unlike analyzing isolated clips, MERC requires a deep understanding of conversational context where emotions evolve through complex multimodal interplay\. This contextual understanding is crucial for building truly empathetic interactive systems\[[18](https://arxiv.org/html/2608.10448#bib.bib4)\]\.

The field has progressed through several architectural shifts to better capture conversational dynamics\. Initial approaches based on Recurrent Neural Networks \(RNNs\)\[[15](https://arxiv.org/html/2608.10448#bib.bib5),[6](https://arxiv.org/html/2608.10448#bib.bib6)\]modeled dialogue sequentially, but struggled with long\-range dependencies\. Transformer\-based models\[[16](https://arxiv.org/html/2608.10448#bib.bib7),[29](https://arxiv.org/html/2608.10448#bib.bib8),[30](https://arxiv.org/html/2608.10448#bib.bib9)\]addressed this by leveraging self\-attention to capture distant contextual cues\. Subsequently, Graph Neural Networks \(GNNs\)\[[21](https://arxiv.org/html/2608.10448#bib.bib10),[5](https://arxiv.org/html/2608.10448#bib.bib11),[9](https://arxiv.org/html/2608.10448#bib.bib12)\]were introduced to explicitly represent speakers and utterances as nodes, allowing more nuanced propagation of context that reflects the multi\-party nature of the conversation\. Alongside these architectural shifts, recent work has also emphasized robust multimodal fusion and generalization, including cross\-modal knowledge distillation\[[28](https://arxiv.org/html/2608.10448#bib.bib13)\], dynamic attention mechanisms\[[11](https://arxiv.org/html/2608.10448#bib.bib14)\], and context\-aware contrastive learning\[[26](https://arxiv.org/html/2608.10448#bib.bib16)\]\.

However, despite this progress, current approaches still suffer from a fundamental limitation\. They treat emotion recognition as a direct mapping problem from raw inputs to emotion labels, focusing on predicting‘what’the final emotion is\. This overlooks the causal reasoning humans use to understand‘how’verbal and non\-verbal cues interact to convey an emotional state\. For instance, wide eyes can signify fear in one context but joyful surprise in another, a nuance that rationale\-free models often miss\. Without modeling such rationale\-driven processes, they are prone to learning superficial shortcuts from spurious correlations between input cues and emotions\.

![Refer to caption](https://arxiv.org/html/2608.10448v1/x1.png)Fig\. 1:An overview of the RGL architecture’s two main stages\.\(Top\)An MLLM generates structured rationales \(Intuitive,Contextual,Integrative\) to construct rationale banks offline\.\(Bottom\)A compact model is trained with contrastive losses to align its internal representations with the rationale vectors from the banks, fostering a human\-like reasoning process\.To address this issue, we propose Rationale\-Guided Learning \(RGL\) for Multimodal Emotion Recognition, a novel framework that injects human\-like, rationale\-guided reasoning into the process using a Multimodal Large Language Model \(MLLM\)\. Crucially, the MLLM is leveraged only once during the offline training preparation step to generate rationales, allowing our final model to remain efficient without requiring any MLLM overheads at inference time\. Our approach is inspired by the dual\-process theory of human cognition\[[4](https://arxiv.org/html/2608.10448#bib.bib17)\], which distinguishes between fast, automaticSystem 1and slow, deliberateSystem 2\. Based on this theory, we decompose this reasoning into three explicit facets:Intuitive rationalesmirror the rapid perception of facial cues \(System 1\),Contextual rationalesreflect the deliberate analysis of the situation \(System 2\), andIntegrative rationalessynthesize both \(System 1 & 2\) to form a coherent conclusion\. These pre\-generated rationales are transformed into intermediate supervision signals that guide our model to internalize these reasoning pathways, prioritizing causal understanding over direct input\-output mappings\.

Our main contributions are summarized as follows:

- •We propose RGL, a novel rationale\-aware framework for MERC that leverages MLLM to inject human\-like, rationale\-guided reasoning\. It enables models to learn reasoning patterns rather than superficial predictions\.
- •We propose a three\-facet rationale decomposition:Intuitive,Contextual, andIntegrativerationales\. We utilize rationale features for intermediate guidance, enhancing the robustness without any inference overhead\.
- •Through comprehensive experiments on IEMOCAP and MELD benchmarks, we demonstrate that RGL outperforms existing state\-of\-the\-art methods, validating the effectiveness of our rationale\-aware approach\.

## 2PROPOSED METHOD

Our proposed framework RGL, is designed to explicitly guide the reasoning process using rationales\. The overall architecture is illustrated in Fig\.[1](https://arxiv.org/html/2608.10448#S1.F1)\. The process consists of two primary stages: \(1\) Offline phase for generating three types of rationale banks \(Intuitive,Contextual,Integrative\) by leveraging the reasoning of an MLLM, and \(2\) training phase for an emotion recognition model to learn rationale\-guided reasoning patterns while predicting the target emotion\.

### 2\.1Rationale generation

The cornerstone of RGL lies in structured rationale banks, generated offline by an MLLM \(GPT\-4o\[[17](https://arxiv.org/html/2608.10448#bib.bib20)\]\) to emulate the cognitive system from dual\-process theory\[[4](https://arxiv.org/html/2608.10448#bib.bib17)\]\. This one\-time offline process ensures our model operates without the overhead of running the MLLM at training and inference time\.

We prompt the MLLM with the multimodal inputs \(video frames and dialogue text\) and the ground\-truth emotion label for leveraging the reasoning power\. The prompt is meticulously designed to guide the MLLM through a three\-step analytical process, forcing it to deconstruct its reasoning into distinct, cognitively\-motivated facets:

- •Intuitive rationale\(rIr\_\{\\text\{I\}\}\): This facet is designed to captureSystem 1processing, which involves the immediate, automatic perception of evidence\. The MLLM is instructed to describe only the objective facial muscle configurations \(e\.g\., “eyebrows are lowered and drawn together”\) without using any emotional terminology\.
- •Contextual rationale\(rCr\_\{\\text\{C\}\}\): This facet modelsSystem 2reasoning, specifically the slower, more deliberate analysis of the surrounding situation\. The MLLM identifies the specific conversational event \(e\.g\., “the speaker is informed their work has been shut down”\) that likely triggered the emotion, which requires a deeper understanding of the dialogue’s narrative\.
- •Integrative rationale\(rGr\_\{\\text\{G\}\}\): This represents the final synthesis where the outputs ofSystem 1\(Intuitive\) andSystem 2\(Contextual\) are logically connected\. The MLLM formulates an explanation that justifies emotion label by combining the observed cues with the situational trigger\.

This three\-step process yields a dataset of textual descriptions for each training sample\. These texts are then encoded using a pre\-trained text embedder \(BGE\-large\-en\-v1\.5\[[25](https://arxiv.org/html/2608.10448#bib.bib18)\]\) to create dense vector representations, denoted as the rationales\{rI,rC,rG\}\\\{r\_\{\\text\{I\}\},r\_\{\\text\{C\}\},r\_\{\\text\{G\}\}\\\}\. These rationale vectors are organized into three distinct banks,ℬI\\mathcal\{B\}\_\{\\text\{I\}\},ℬC\\mathcal\{B\}\_\{\\text\{C\}\}, andℬG\\mathcal\{B\}\_\{\\text\{G\}\}, corresponding to theIntuitive,Contextual, andIntegrativefacets, respectively\. They serve as supervisory targets for our emotion recognition model\. We refer to the combination ofℬI\\mathcal\{B\}\_\{\\text\{I\}\},ℬC\\mathcal\{B\}\_\{\\text\{C\}\}, andℬG\\mathcal\{B\}\_\{\\text\{G\}\}as the rationale banksℬ\\mathcal\{B\}\.

### 2\.2Model architecture

The trainable part of RGL is a compact, end\-to\-end network consisting of unimodal encoders and a multimodal fusion module\.

Unimodal encoders\.To extract modality\-specific features, our model processes visual \(V\), textual \(T\), and audio \(A\) modalities using standard pre\-trained backbones: ViT\-base\[[3](https://arxiv.org/html/2608.10448#bib.bib22)\], RoBERTa\-large\[[14](https://arxiv.org/html/2608.10448#bib.bib21)\], and HuBERT\-base\[[7](https://arxiv.org/html/2608.10448#bib.bib23)\], respectively\. The visual and textual encoders are designed with a dual\-head architecture to output two distinct representations: \(1\) the main featurefmain,Vf\_\{\\text\{main,V\}\}andfmain,Tf\_\{\\text\{main,T\}\}for the primary emotion prediction, \(2\) rationale featurefrat,Vf\_\{\\text\{rat,V\}\}andfrat,Tf\_\{\\text\{rat,T\}\}specifically for aligning with rationale banks\. The audio encoder outputs a single main feature, denoted asfAf\_\{\\text\{A\}\}\. This dual\-head design decouples the tasks, enabling targeted rationale alignment without interfering with the main classification objective\.

Multimodal fusion\.The main features from all encoders\{fmain,V,fmain,T,fA\}\\\{f\_\{\\text\{main,V\}\},f\_\{\\text\{main,T\}\},f\_\{\\text\{A\}\}\\\}are first concatenated and then processed by a stack of Transformer encoder layers\[[23](https://arxiv.org/html/2608.10448#bib.bib24)\]to model cross\-modal interactions\. This captures complex, cross\-modal interactions through self\-attention, yielding a sequence of contextually enriched hidden states𝐇∈ℝL×D\\mathbf\{H\}\\in\\mathbb\{R\}^\{L\\times D\}, whereLLis the input sequence length andDDis the hidden dimension\. To aggregate these sequential states into a single vector,ffusedf\_\{\\text\{fused\}\}, we employ attention pooling\[[13](https://arxiv.org/html/2608.10448#bib.bib19)\], which dynamically weighs the importance of each token\. Finally, this vectorffusedf\_\{\\text\{fused\}\}is projected through two task\-specific heads: an MLP classifier for emotion prediction, and a rationale head for the rationale\-guided reasoning objective\.

### 2\.3Rationale\-guided representation learning

The core of our training is to align the model’s rationale features \(frat,Vf\_\{\\text\{rat,V\}\},frat,Tf\_\{\\text\{rat,T\}\}, andfrat,Ff\_\{\\text\{rat,F\}\}\) with their corresponding rationales from the pre\-computed banks,ℬ\\mathcal\{B\}\. This alignment is achieved through a contrastive learning objective\. The objectivepullseach model representation \(f\(i\)f^\{\(i\)\}\), referred to as the anchor, towards its corresponding target rationale \(r\(i\)r^\{\(i\)\}\) from the bank, which forms apositive pair\. Simultaneously, the objectivepushesthe anchor away from rationales of different emotions, which formnegative pairs\.

To make this process more effective, we employ ahard negative mining strategy\. For each anchorf\(i\)f^\{\(i\)\}, we construct a set of hard negatives𝒩K\(i\)\\mathcal\{N\}\_\{K\}^\{\(i\)\}by sampling from the rationale banksℬ\\mathcal\{B\}\. Specifically, we first form a candidate pool by excluding all rationales that share the same emotion label as the positive pairr\(i\)r^\{\(i\)\}\. Then, from this pool, we retrieve the top\-KK\(K=128K=128\) most similar negative samples that have the highest cosine similarity to the anchorf\(i\)f^\{\(i\)\}\. This approach forces the model to learn finer\-grained distinctions between semantically close yet emotionally distinct concepts, moving beyond simple class separation\. For brevity, letsi\+=sim​\(f\(i\),r\(i\)\)s\_\{i\}^\{\+\}=\\text\{sim\}\(f^\{\(i\)\},r^\{\(i\)\}\)denote the similarity score for the positive pair, andsi​k−=sim​\(f\(i\),rk\)s\_\{ik\}^\{\-\}=\\text\{sim\}\(f^\{\(i\)\},r\_\{k\}\)for a negative pairrk∈𝒩K\(i\)r\_\{k\}\\in\\mathcal\{N\}\_\{K\}^\{\(i\)\}\. The rationale loss for a sampleiiis then defined as:

ℒrat\(i\)=−log⁡exp⁡\(si\+/τ\)exp⁡\(si\+/τ\)\+∑k=1Kexp⁡\(si​k−/τ\),\\mathcal\{L\}\_\{\\text\{rat\}\}^\{\(i\)\}=\-\\log\\frac\{\\exp\(s\_\{i\}^\{\+\}/\\tau\)\}\{\\exp\(s\_\{i\}^\{\+\}/\\tau\)\+\\sum\_\{k=1\}^\{K\}\\exp\(s\_\{ik\}^\{\-\}/\\tau\)\},\(1\)wheresim​\(⋅,⋅\)\\text\{sim\}\(\\cdot,\\cdot\)is the cosine similarity between two vectors,𝒩K\(i\)\\mathcal\{N\}\_\{K\}^\{\(i\)\}is the set ofKKhard negatives for sampleii, andτ\\tauis a temperature hyperparameter\.

This alignment process is designed to mirror a cognitive reasoning pipeline inspired by the human dual\-process theory\.

1\. Aligning visual features withIntuitive rationale:We ground the model’s understanding in fast, intuitive rationale\. We align the visual rationale representationfrat,Vf\_\{\\text\{rat,V\}\}with theIntuitive rationalevectorrIr\_\{\\text\{I\}\}from its corresponding bankℬI\\mathcal\{B\}\_\{\\text\{I\}\}, using the lossℒrat,I\\mathcal\{L\}\_\{\\text\{rat,I\}\}\.

2\. Aligning textual features withContextual rationale:We train the model to be aware of slow, analytical reasoning of contexts from the dialogue\. We align the textual rationale representationfrat,Tf\_\{\\text\{rat,T\}\}with theContextual rationalerCr\_\{\\text\{C\}\}fromℬC\\mathcal\{B\}\_\{\\text\{C\}\}viaℒrat,C\\mathcal\{L\}\_\{\\text\{rat,C\}\}\.

3\. Aligning fused features withIntegrative rationale:Finally, we align the fused rationale representationfrat,Ff\_\{\\text\{rat,F\}\}with theIntegrative rationalevectorrGr\_\{\\text\{G\}\}fromℬG\\mathcal\{B\}\_\{\\text\{G\}\}based onℒrat,G\\mathcal\{L\}\_\{\\text\{rat,G\}\}\. This step guides the model to synthesize bothIntuitiveandContextualinsights, forming a coherent inference\. This staged alignment ensures meaningful unimodal representations are learned first, providing a robust foundation for the final synthesis\.

As a result, a final training objective can be formulated as:

ℒtotal=ℒCE⏟Emotion Classification\+λ​\(ℒrat,I\+ℒrat,C\+ℒrat,G\)⏟Rationale\-Guided Alignment,\\mathcal\{L\}\_\{\\text\{total\}\}=\\underbrace\{\\mathcal\{L\}\_\{\\text\{CE\}\}\}\_\{\\text\{Emotion Classification\}\}\+\\lambda\\underbrace\{\\bigl\(\\mathcal\{L\}\_\{\\text\{rat,I\}\}\+\\mathcal\{L\}\_\{\\text\{rat,C\}\}\+\\mathcal\{L\}\_\{\\text\{rat,G\}\}\\bigr\)\}\_\{\\text\{Rationale\-Guided Alignment\}\},\(2\)whereℒCE\\mathcal\{L\}\_\{\\text\{CE\}\}represents a cross\-entropy loss, each rationale loss termℒrat,X\\mathcal\{L\}\_\{\\text\{rat,X\}\}\(forX∈\{I,C,G\}X\\in\\\{I,C,G\\\}\) is computed as defined in Eq\. \([1](https://arxiv.org/html/2608.10448#S2.E1)\), andλ\\lambdais a hyperparameter that balances the rationale\-guided objectives, thereby training RGL to structure its embedding space in a way that mirrors a logical, human\-like reasoning process\.

Table 1:Performance comparison with existing methods on IEMOCAP and MELD datasets\.Table 2:Ablation study of RGL’s components on the IEMOCAP test set\. The full model’s performance is inbold\.Model ConfigurationW\-F1AccRGL \(Full Model\)73\.6873\.51w/oIntuitive loss\(ℒrat,I\\mathcal\{L\}\_\{\\text\{rat,I\}\}\)72\.7072\.52w/oContextual loss\(ℒrat,C\\mathcal\{L\}\_\{\\text\{rat,C\}\}\)68\.7868\.70w/oIntegrative loss\(ℒrat,G\\mathcal\{L\}\_\{\\text\{rat,G\}\}\)72\.4472\.34w/oℒrat,I\\mathcal\{L\}\_\{\\text\{rat,I\}\},ℒrat,C\\mathcal\{L\}\_\{\\text\{rat,C\}\},ℒrat,G\\mathcal\{L\}\_\{\\text\{rat,G\}\}68\.0167\.71

## 3EXPERIMENTS

### 3\.1Datasets and implementation details

Datasets\.We conduct experiments on two widely adopted datasets: IEMOCAP\[[1](https://arxiv.org/html/2608.10448#bib.bib27)\]and MELD\[[20](https://arxiv.org/html/2608.10448#bib.bib28)\]\. IEMOCAP is a dyadic dataset for which we use six standard emotion categories: ‘neutral’, ‘sad’, ‘angry’, ‘happy’, ‘excited’, and ‘frustrated’\. MELD is a multi\-party dataset extracted from the TV show “Friends”, containing seven emotion labels: ‘anger’, ‘disgust’, ‘fear’, ‘joy’, ‘neutral’, ‘sadness’, and ‘surprise’\.

Implementation details\.We train using the AdamW optimizer with a learning rate of1​e−51\\mathrm\{e\}\{\-5\}and a batch size of 4\. The temperature parameter in Eq\. \(1\) isτ=0\.07\\tau=0\.07, and the hyperparameter in Eq\. \(2\) is set toλ=0\.3\\lambda=0\.3\. For video streams, we follow FacialMMT\[[31](https://arxiv.org/html/2608.10448#bib.bib26)\]and apply TalkNet\-ASD\[[22](https://arxiv.org/html/2608.10448#bib.bib25)\]to detect the face of the active speakers based on vocal activity\.

### 3\.2Performance evaluation

We evaluate performance using two standard metrics, Weighted F1 \(W\-F1\) and accuracy \(Acc\), following\[[21](https://arxiv.org/html/2608.10448#bib.bib10),[27](https://arxiv.org/html/2608.10448#bib.bib31)\]\.

Performance comparison\.As shown in Table[1](https://arxiv.org/html/2608.10448#S2.T1), our RGL achieves state\-of\-the\-art results on both datasets\. The consistent improvements across two different settings, dyadic interactions on IEMOCAP and multi\-party conversations in MELD, provide strong evidence for our hypothesis that explicitly supervising internal representations with structured cognitive rationales is effective\.

Ablation studies\.We also conduct ablation studies to verify the contribution of our proposed rationale designs\. As shown in Table[2](https://arxiv.org/html/2608.10448#S2.T2), removing all losses simultaneously causes the most significant drop in performance, confirming their overall importance\. The results reveal thatℒrat,C\\mathcal\{L\}\_\{\\text\{rat,C\}\}is the most critical component, whileℒrat,G\\mathcal\{L\}\_\{\\text\{rat,G\}\}andℒrat,I\\mathcal\{L\}\_\{\\text\{rat,I\}\}are also effective\.

### 3\.3Reasoning interpretation

![Refer to caption](https://arxiv.org/html/2608.10448v1/x2.png)Fig\. 2:Rationale retrieval example on an unseen test sample\.To validate that RGL genuinely learns reasoning patterns, we analyze how it leverages its learned rationale banks for unseen test samples\. For a given test case, we extract the model’s internal rationale representations \(frat,Vf\_\{\\text\{rat,V\}\},frat,Tf\_\{\\text\{rat,T\}\},frat,Ff\_\{\\text\{rat,F\}\}\) and use them as queries to retrieve the most similar rationales from the training rationale banks\. Figure[2](https://arxiv.org/html/2608.10448#S3.F2)shows this capability on a challenging sample, where successful retrieval requires a deep semantic understanding beyond superficial cues\. The model accesses anIntuitive rationalefrom rationale bank that aligns with the facial expression and aContextual rationalerelevant to the “loss of people\.” Furthermore, the retrievedIntegrative rationalecorrectly synthesizes both aspects\. This confirms that RGL learns a robust mapping from raw multimodal signals to a structured, semantically meaningful rationale space, proving its internal features are grounded in human\-like reasoning\.

## 4CONCLUSION

We introduce Rationale\-Guided Learning \(RGL\) for Multimodal Emotion Recognition, a novel framework that trains a model by aligning its representations with cognitive rationales generated offline by an MLLM\. Achieving state\-of\-the\-art results on IEMOCAP and MELD, our work demonstrates that emulating cognitive reasoning is an effective approach for advancing multimodal emotion recognition\.

Acknowledgement\.This work was supported in part by the NRF grant funded by the Korea government\(MSIT\) \(RS\-2025\-00563942\), the IITP grant funded by the Korea government\(MSIT\)\(IITP\-2026\-RS\-2020\-II201819, 20%\), and the IITP\-ITRC grant funded by the Korea government\(MSIT\) \(IITP\-2026\-RS\-2023\-00258649, 30%\)\.

## References

- \[1\]C\. Busso, M\. Bulut, C\. Lee, A\. Kazemzadeh, E\. Mower, S\. Kim, J\. N\. Chang, S\. Lee, and S\. S\. Narayanan\(2008\)IEMOCAP: interactive emotional dyadic motion capture database\.InLREC,pp\. 335–339\.Cited by:[§3\.1](https://arxiv.org/html/2608.10448#S3.SS1.p1.1)\.
- \[2\]\(2025\)Dynamic interactive bimodal hypergraph networks for emotion recognition in conversations\.InAAAI,pp\. 1256–1264\.Cited by:[Table 1](https://arxiv.org/html/2608.10448#S2.T1.4.1.12.10.1)\.
- \[3\]A\. Dosovitskiy, L\. Beyer, A\. Kolesnikov, D\. Weissenborn, X\. Zhai, T\. Unterthiner, M\. Dehghani, M\. Minderer, G\. Heigold, S\. Gelly, J\. Uszkoreit, and N\. Houlsby\(2021\)An image is worth 16x16 words: transformers for image recognition at scale\.InICLR,Cited by:[§2\.2](https://arxiv.org/html/2608.10448#S2.SS2.p2.5)\.
- \[4\]J\. S\. B\. T\. Evans and K\. E\. Stanovich\(2013\)Dual\-process theories of higher cognition: advancing the debate\.Perspect\. Psychol\. Sci\.8\(3\),pp\. 223–241\.Cited by:[§1](https://arxiv.org/html/2608.10448#S1.p4.1),[§2\.1](https://arxiv.org/html/2608.10448#S2.SS1.p1.1)\.
- \[5\]D\. Ghosal, N\. Majumder, S\. Poria, N\. Chhaya, and A\. Gelbukh\(2019\)DialogueGCN: a graph convolutional neural network for emotion recognition in conversation\.InEMNLP,pp\. 154–164\.Cited by:[§1](https://arxiv.org/html/2608.10448#S1.p2.1)\.
- \[6\]D\. Hazarika, S\. Poria, R\. Mihalcea, and E\. Cambria\(2018\)ICON: interactive conversational memory network for multimodal emotion detection\.InEMNLP,pp\. 2594–2604\.Cited by:[§1](https://arxiv.org/html/2608.10448#S1.p2.1)\.
- \[7\]W\. Hsu, B\. Bolte, Y\. H\. Tsai, K\. Lakhotia, R\. Salakhutdinov, and A\. Mohamed\(2021\)HuBERT: self\-supervised speech representation learning by masked prediction of hidden units\.InACM\-TASLP,pp\. 3451–3460\.Cited by:[§2\.2](https://arxiv.org/html/2608.10448#S2.SS2.p2.5)\.
- \[8\]D\. Hu, X\. Hou, L\. Wei, L\. Jiang, and Y\. Mo\(2022\)MM\-dfn: multimodal dynamic fusion network for emotion recognition in conversations\.InICASSP,pp\. 7037–7041\.Cited by:[Table 1](https://arxiv.org/html/2608.10448#S2.T1.4.1.5.3.1)\.
- \[9\]J\. Hu, Y\. Liu, J\. Zhao, and Q\. Jin\(2021\)MMGCN: multimodal fusion via deep graph convolution network for emotion recognition in conversation\.InACL\-Long,pp\. 5666–5675\.Cited by:[§1](https://arxiv.org/html/2608.10448#S1.p2.1)\.
- \[10\]Y\. Hwang and J\. Kim\(2024\)EASUM: enhancing affective state understanding through joint sentiment and emotion modeling for multimodal tasks\.InWACV,pp\. 5668–5678\.Cited by:[Table 1](https://arxiv.org/html/2608.10448#S2.T1.4.1.8.6.1)\.
- \[11\]Y\. Jing and X\. Zhao\(2024\)DQ\-former: querying transformer with dynamic modality priority for cognitive\-aligned multimodal emotion recognition in conversation\.InACM MM,pp\. 4795–4804\.Cited by:[§1](https://arxiv.org/html/2608.10448#S1.p2.1)\.
- \[12\]J\. Li, S\. Ding, L\. Guo, and X\. Li\(2025\)Multi\-modal anchor gated transformer with knowledge distillation for emotion recognition in conversation\.InIJCAI,External Links:[Document](https://dx.doi.org/10.24963/ijcai.2025/905)Cited by:[Table 1](https://arxiv.org/html/2608.10448#S2.T1.4.1.13.11.1)\.
- \[13\]Z\. Lin, M\. Feng, C\. N\. d\. Santos, M\. Yu, B\. Xiang, B\. Zhou, and Y\. Bengio\(2017\)A structured self\-attentive sentence embedding\.InICLR,Cited by:[§2\.2](https://arxiv.org/html/2608.10448#S2.SS2.p3.6)\.
- \[14\]Y\. Liu, M\. Ott, N\. Goyal, J\. Du, M\. Joshi, D\. Chen, O\. Levy, M\. Lewis, L\. Zettlemoyer, and V\. Stoyanov\(2019\)RoBERTa: a robustly optimized BERT pretraining approach\.arXiv\.Cited by:[§2\.2](https://arxiv.org/html/2608.10448#S2.SS2.p2.5)\.
- \[15\]N\. Majumder, S\. Poria, D\. Hazarika, R\. Mihalcea, A\. Gelbukh, and E\. Cambria\(2019\)DialogueRNN: an attentive rnn for emotion detection in conversations\.InAAAI,pp\. 6818–6825\.Cited by:[§1](https://arxiv.org/html/2608.10448#S1.p2.1),[Table 1](https://arxiv.org/html/2608.10448#S2.T1.4.1.3.1.1)\.
- \[16\]Y\. Mao, G\. Liu, X\. Wang, W\. Gao, and X\. Li\(2021\)DialogueTRM: exploring multi\-modal emotional dynamics in a conversation\.InFindings of EMNLP,pp\. 2694–2704\.Cited by:[§1](https://arxiv.org/html/2608.10448#S1.p2.1),[Table 1](https://arxiv.org/html/2608.10448#S2.T1.4.1.4.2.1)\.
- \[17\]OpenAI, L\. Ahmad, A\. Hurst, A\. Lerer, A\. P\. Goucher, A\. Perelman, A\. Ramesh, A\. Clark, A\. Ostrow, A\. Welihinda, and et al\.\(2024\)GPT\-4o system card\.arXiv\.Cited by:[§2\.1](https://arxiv.org/html/2608.10448#S2.SS1.p1.1)\.
- \[18\]R\. W\. Picard\(1997\)Affective computing\.MIT Press\.Cited by:[§1](https://arxiv.org/html/2608.10448#S1.p1.1)\.
- \[19\]S\. Poria, E\. Cambria, R\. Bajpai, and A\. Hussain\(2017\)A review of affective computing: from unimodal analysis to multimodal fusion\.Information Fusion37,pp\. 98–125\.Cited by:[§1](https://arxiv.org/html/2608.10448#S1.p1.1)\.
- \[20\]S\. Poria, D\. Hazarika, N\. Majumder, G\. Naik, E\. Cambria, and R\. Mihalcea\(2019\)MELD: a multimodal multi\-party dataset for emotion recognition in conversations\.InACL,pp\. 527–536\.Cited by:[§3\.1](https://arxiv.org/html/2608.10448#S3.SS1.p1.1)\.
- \[21\]Y\. Shou, T\. Meng, W\. Ai, and K\. Li\(2025\)Dynamic graph neural ODE network for multi\-modal emotion recognition in conversation\.InCOLING,pp\. 256–268\.Cited by:[§1](https://arxiv.org/html/2608.10448#S1.p2.1),[§3\.2](https://arxiv.org/html/2608.10448#S3.SS2.p1.1)\.
- \[22\]R\. Tao, Z\. Pan, R\. K\. Das, X\. Qian, M\. Z\. Shou, and H\. Li\(2021\)Is someone speaking? exploring long\-term temporal features for audio\-visual active speaker detection\.InACM MM,pp\. 3927–3935\.Cited by:[§3\.1](https://arxiv.org/html/2608.10448#S3.SS1.p2.3)\.
- \[23\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.InNeurIPS,pp\. 6000–6010\.Cited by:[§2\.2](https://arxiv.org/html/2608.10448#S2.SS2.p3.6)\.
- \[24\]Y\. Wang, X\. Fang, H\. Yin, D\. Li, G\. Li, Q\. Xu, Y\. Xu, S\. Zhong, and M\. Xu\(2025\)BIG\-fusion: brain\-inspired global\-local context fusion framework for multimodal emotion recognition in conversations\.InAAAI,pp\. 1574–1582\.Cited by:[Table 1](https://arxiv.org/html/2608.10448#S2.T1.4.1.11.9.1)\.
- \[25\]S\. Xiao, Z\. Liu, P\. Zhang, N\. Muennighoff, D\. Lian, and J\. Nie\(2024\)C\-pack: packed resources for general chinese embeddings\.InACM SIGIR,pp\. 641–649\.Cited by:[§2\.1](https://arxiv.org/html/2608.10448#S2.SS1.p3.8)\.
- \[26\]Y\. Xie, C\. Sun, Z\. Cao, B\. Liu, Z\. Ji, Y\. Liu, and L\. Shan\(2025\)A dual contrastive learning framework for enhanced multimodal conversational emotion recognition\.InCOLING,pp\. 4055–4065\.Cited by:[§1](https://arxiv.org/html/2608.10448#S1.p2.1)\.
- \[27\]Z\. Yi, Z\. Zhao, Z\. Shen, and T\. Zhang\(2024\)Multimodal fusion via hypergraph autoencoder and contrastive learning for emotion recognition in conversation\.InACM MM,pp\. 4341–4348\.Cited by:[Table 1](https://arxiv.org/html/2608.10448#S2.T1.4.1.10.8.1),[§3\.2](https://arxiv.org/html/2608.10448#S3.SS2.p1.1)\.
- \[28\]T\. Yun, H\. Lim, J\. Lee, and M\. Song\(2024\)TelME: teacher\-leading multimodal fusion network for emotion recognition in conversation\.InNAACL\-Long,pp\. 82–95\.Cited by:[§1](https://arxiv.org/html/2608.10448#S1.p2.1),[Table 1](https://arxiv.org/html/2608.10448#S2.T1.4.1.9.7.1)\.
- \[29\]X\. Zhang and Y\. Li\(2023\)A cross\-modality context fusion and semantic refinement network for emotion recognition in conversation\.InACL\-Long,pp\. 13099–13110\.Cited by:[§1](https://arxiv.org/html/2608.10448#S1.p2.1)\.
- \[30\]H\. Zhao, B\. Li, and Z\. Zhang\(2023\)Speaker\-aware cross\-modal fusion for conversational emotion recognition\.InINTERSPEECH,pp\. 2718–2722\.Cited by:[§1](https://arxiv.org/html/2608.10448#S1.p2.1),[Table 1](https://arxiv.org/html/2608.10448#S2.T1.4.1.6.4.1)\.
- \[31\]W\. Zheng, J\. Yu, R\. Xia, and S\. Wang\(2023\)A facial expression\-aware multimodal multi\-task learning framework for emotion recognition in multi\-party conversations\.InACL\-Long,pp\. 15445–15459\.Cited by:[Table 1](https://arxiv.org/html/2608.10448#S2.T1.4.1.7.5.1),[§3\.1](https://arxiv.org/html/2608.10448#S3.SS1.p2.3)\.