Semantic-Aligned Structural Abstraction for Multimodal Sentiment Analysis

# Semantic-Aligned Structural Abstraction for Multimodal Sentiment Analysis
Source: [https://arxiv.org/html/2607.27790](https://arxiv.org/html/2607.27790)
\(2026\)

###### Abstract\.

Multimodal Sentiment Analysis \(MSA\) aims to interpret complex human emotions by integrating natural language with non\-verbal modalities\. Non\-verbal modalities share a structural isomorphism with natural language, as both can be viewed as feature sequences evolving over time\. This isomorphism enables the transformation of non\-verbal modalities into text\-like tokens for unified semantic reasoning\. Large Language Models \(LLMs\), designed to understand and generate sequential data, can thus be utilized to interpret complex affective sequences\. However, existing LLM\-based methods primarily capture low\-level superficial features, failing to model affective semantics arising from structural variations and contextual interactions\. To address this limitation, we proposeSentiLLM, a unified framework that leveragesSemantic\-Aligned Structural Abstractionto distill continuous raw signals into compact, semantically meaningful tokens\. Specifically, we introduce aDual\-Stream Salience\-Context Calibration Mechanism, which disentangles non\-verbal feature sequences into a focus stream and an ambient stream\. The focus stream captures salient sentiment shifts \(e\.g\., facial expressions\) guided by textual priors, while the ambient stream characterizes stable background states\. Through calibrating these dynamic sentiment shifts against background states, SentiLLM effectively projects non\-verbal modalities into a unified semantic space, making them naturally understandable for LLMs\. Serving as a plug\-and\-play module, SentiLLM significantly improves discriminative performance with only a small number of trainable parameters\. Our method achieves superior performance on four datasets—MOSI, MOSEI, CH\-SIMS, and CH\-SIMS v2—demonstrating the effectiveness of the structural abstraction paradigm in MSA\. Our code is available at:[https://github\.com/especiallyW/SentiLLM](https://github.com/especiallyW/SentiLLM)\.

Social Media; Multimodal Sentiment Analysis; Large Language Model; Multimodal Learning; Multimodal Fusion

††journalyear:2026††copyright:cc††conference:Proceedings of the 34th ACM International Conference on Multimedia; November 10–14, 2026; Rio de Janeiro, Brazil††booktitle:Proceedings of the 34th ACM International Conference on Multimedia \(MM ’26\), November 10–14, 2026, Rio de Janeiro, Brazil††isbn:979\-8\-4007\-2213\-4/2026/11††doi:10\.1145/3767308\.3835866††ccs:Computing methodologies Computer vision††ccs:Computing methodologies Natural language processing## 1\.Introduction

Multimodal Sentiment Analysis \(MSA\)\(Zhang2024ACS\)leverages heterogeneous data sources—video, audio, and text—to achieve precise reasoning about human sentiment states\. Driven by its pivotal value in business decisions and human\-computer interaction, the field has garnered significant attention\. Existing methods\(Hazarika2020MISAMA;li2023decoupled;tsai2019multimodal;wang2025dlf;luo2025towards;wang2025rclmufn\)typically adopt a modality decoupling and fusion paradigm\. These methods focus on designing complex network architectures to capture shared and specific modal information\. However, over\-reliance on specialized architectures constrains scalability and generalization, hindering the effective utilization of large\-scale pre\-trained knowledge\.

The advent of Multimodal Large Language Models \(MLLMs\)\(li2023blip;liu2024improved\)necessitates a re\-examination of the intrinsic nature of different modalities\. Non\-verbal modalities, such as video and audio, share a structural isomorphism with natural language, both can be viewed as feature sequences evolving over time\(sun2019videobert;hsu2021hubert;tsai2019multimodal\)\. For instance, video frames resemble words in a sentence, where semantics are defined by their predecessors and successors\. Consequently, Large Language Models \(LLMs\)\(dubey2024llama;Yang2024Qwen25TR\), serving as sequence modeling experts embedded with vast world knowledge, possess an innate potential to process non\-verbal sequences and perform unified reasoning\.

However, the direct application of LLMs to MSA faces a substantialsemantic gap\. Raw audio\-visual signals appear as continuous and redundant long sequences\. A simple linear mapping often capture only surface\-level features \(e\.g\., facial textures or acoustic frequencies\), failing to extract subtle sentiment cues, such as an angry tremble or ironic tone\. This appearance\-semantic misalignment prevents transformed features from aligning within the LLM semantic space\. Therefore, a semantic translation mechanism is required to convert non\-verbal modalities into LLM\-comprehensible sentiment tokens\.

We argue that sentiment is not a static time step mapping\. Instead, it represents the structural unity of salient semantic shifts and contextual states across time\. We proposed that sentiment judgment followsa Focus\-Ambient Calibration Mechanism:human first attend to salient sentiment shifts \(the Focus\), such as a sudden scream or raised eyebrow, and subsequently verify reliability through contextual cues \(the Ambient\)\. Based on this, an ideal sentiment token must meet two criteria: \(1\)Structural, representing a significant dynamic process rather than a static snapshot; and \(2\)Calibration, verified by context to ensure semantic stability rather than random noise\.

Guided by these cognitive insights, we introduce SentiLLM, a parameter\-efficient framework built onSemantic\-Aligned Structural Abstraction—a process that distills continuous, unaligned raw signals into compact, semantically meaningful tokens\. Transcending traditional feature extraction paradigms, SentiLLM functions as a spatial\-temporal semantic translator, converting non\-verbal signals into sentiment tokens\. Specifically, to instantiate this theoretical calibration, we design a Dual\-Stream Salience\-Context Calibration Mechanism\. This mechanism disentangles audio\-visual signals into aFocus Streamcarrying sentiment shifts and anAmbient Streamdepicting contextual states\. The focus stream utilizes textual semantic anchors to retrieve salient dynamic sentiment shifts, while the ambient stream captures global background states to calibrate focus information stability\. Through this interaction, raw signals are abstracted into compact, semantically rich sentiment tokens\. Benefiting from this structural abstraction, the generated sentiment tokens align seamlessly with the textual semantic space, making them compatible with LLM reasoning without requiring complex architectural modifications\.

To validate our method, we conducted extensive experiments on four benchmark datasets: MOSI, MOSEI, CH\-SIMS, and CH\-SIMS v2\. Experimental results demonstrate that our method achieves competitive performance without complex architecture design, requiring only a small number of trainable parameters\. Our main contributions are summarized as follows:

- ∙\\bulletLeveraging the structural isomorphism between non\-verbal modalities and natural language, we propose SentiLLM, demonstrating the feasibility of unifying multimodal sentiment reasoning within the LLM semantic space\.
- ∙\\bulletRooted in theFocus\-Ambient Calibration, we introduce a Dual\-Stream Salience\-Context Calibration Mechanism\. This mechanism achieves the structural abstraction from raw signals to compact sentiment tokens suitable for LLMs\.
- ∙\\bulletExtensive experiments on four datasets demonstrate the effectiveness of our proposed method\. We also release our code to facilitate further community research\.

## 2\.Related Work

### 2\.1\.Multimodal Sentiment Analysis

Traditional approaches prioritize intricate fusion mechanisms to capture inter\-modal interactions\. Early works\(Zadeh2017TensorFN;liu2018efficient;zadeh2018memory\)explicitly model interactions via mathematical outer products\. With the rise of Transformers\(vaswani2017attention\)and Graph Neural Networks\(scarselli2008graph\), some methods\(zadeh2018multimodal;williams2018recognizing;tsai2019multimodal\)introduce dynamic fusion graphs and architectural innovations to handle long\-range dependency issues in unaligned sequences\. Similarly,\(yu2021learning;han2021improving;rahman2020integrating\)optimize fusion characteristics through learning paradigms and theory\. Specifically, Self\-MM\(yu2021learning\)employs self\-supervised learning for multi\-task training, while MMIM\(han2021improving\)maximizes mutual information to retain key data\. MAG\-BERT\(rahman2020integrating\)integrates BERT\(devlin2019bert\)with multimodal inputs\.

Recently, text\-dominant tasks have highlighted the importance of modality disentanglement\(liu2022make;Hazarika2020MISAMA;li2023decoupled;sun2022cubemlp;sun2024mfm;zhang2025modal\)\. These methods differentiate between specific and shared modalities to enhance robustness\. For instance, DMD\(li2023decoupled\)refines representation purity via distillation\. DLF\(wang2025dlf\)separates shared and specific information for hierarchical prediction\. KAN\-MCP\(luo2025towards\)addresses modality imbalance caused by density differences using KANs\(liu2024kan\)\.

However, these methods rely on complex, task\-specific architectures\. In this paper, we treat video and audio modalities as ”temporal sentences”, leveraging the extensive sequential knowledge embedded in pre\-trained LLMs to capture robust sentiment shifts\.

### 2\.2\.Large Language Models

Recent advancements in LLMs have revolutionized sequence modeling and reasoning, such as GPT\(brown2020language\), LLaMA\(dubey2024llama\), and Qwen\(Yang2024Qwen25TR\)\. Through visual instruction tuning, some studies extended LLMs to vision\-language tasks\. Models like LLaVA\(liu2024improved\), Flamingo\(alayrac2022flamingo\), and BLIP\-2\(li2023blip\)employ MLP or Q\-Former as connectors, effectively bridging the gap between vision encoders and LLMs\.

In the MSA domain, some methods\(hu2022unimse;li2023unisa\)reframe sentiment analysis as sequence generation, converting multimodal inputs into discrete tokens for unified generation\. MSE\-Adapter\(yang2025mse\)designs a lightweight adapter, using a text\-guide\-mixer to filter non\-text features for efficient adaptation\.

However, these methods typically resort to coarse linear mapping or instruction tuning, thereby failing to capture the intrinsic essence of sentiment\. They treat audio\-visual signals as static context supplements, failing to explicitly capture structural variations across time \(e\.g\., sudden sentiment shifts\)\. This absence of a denoising\-calibration mechanism renders the model susceptible to environmental noise, limiting fine\-grained reasoning capabilities\. In this paper, we propose a Dual\-Stream Salience\-Context Calibration Mechanism\. It actively extracts and calibrates salient sentiment shifts, achieving feature translation into the LLM semantic space\.

![Refer to caption](https://arxiv.org/html/2607.27790v1/x1.png)Figure 1\.Overall architecture of SentiLLM\. The model consists of two core modules: an Contextual Temporal Encoding Module, and a Dual\-Stream Salience\-Context Calibration Mechanism\.

## 3\.Methodology

In this section, we detail the SentiLLM framework\. First, we outline the overall architecture in Section[3\.1](https://arxiv.org/html/2607.27790#S3.SS1)\. Then, we describe the contextual temporal encoding module in Section[3\.2](https://arxiv.org/html/2607.27790#S3.SS2)\. Next, we elaborate on the dual\-stream salience\-context calibration mechanism in Section[3\.3](https://arxiv.org/html/2607.27790#S3.SS3)\. Finally, we define the joint optimization objectives in Section[3\.4](https://arxiv.org/html/2607.27790#S3.SS4)\.

### 3\.1\.Overall Architecture

Problem definition\.Given a multimodal dataset𝒟\\mathcal\{D\}withNNsamples, each sample consists of text𝐗t\\mathbf\{X\}\_\{t\}, visual𝐗v\\mathbf\{X\}\_\{v\}, and audio𝐗a\\mathbf\{X\}\_\{a\}inputs alongside a sentiment labelyy\.

Figure[1](https://arxiv.org/html/2607.27790#S2.F1)\(a\) illustrates the SentiLLM architecture\. Specifically, for text input𝐗t\\mathbf\{X\}\_\{t\}, we utilize a pre\-trained tokenizer to obtain𝐅t\\mathbf\{F\}\_\{t\}and an embedding layer to transform it into vectors𝐙t∈ℝT×D\\mathbf\{Z\}\_\{t\}\\in\\mathbb\{R\}^\{T\\times D\}\. For visual𝐗v\\mathbf\{X\}\_\{v\}and audio𝐗a\\mathbf\{X\}\_\{a\}inputs, we employ specific encoders to extract raw feature sequences𝐅v\\mathbf\{F\}\_\{v\}and𝐅a\\mathbf\{F\}\_\{a\}, respectively\. Subsequently,the Semantic\-Aligned Structural Abstraction Moduleprocesses these non\-verbal modalities\. This module involves two stages: \(1\)a contextual temporal encoding modulethat transforms raw features to context\-aware sequences; and \(2\)a dual\-stream salience\-context calibration mechanismthat abstracts continuous sequences into compact, calibrated sentiment tokens𝐙v\\mathbf\{Z\}\_\{v\}and𝐙a\\mathbf\{Z\}\_\{a\}via soft salience disentanglement and dual\-query abstraction\. Finally, we concatenate these tokens with text features to form a unified sequence for LLM reasoning\.

### 3\.2\.Contextual Temporal Encoding

The raw features𝐅m\\mathbf\{F\}\_\{m\}\(m∈\{v,a\}m\\in\\\{v,a\\\}\) contain rich low\-level information, but they lack temporal causality and contextual dependencies\. To empower the model with temporal context awareness, we perform temporal modeling and positional enhancement\.

To achieve this, instead of using simple pooling operations, we employ a standard transformer block \(TFBlock\\text\{TF\}\_\{\\text\{Block\}\}\) to capture long\-range temporal dependencies\. Additionally, since subsequent disentanglement may disrupt local structure, we introduce absolute position embeddings \(PE\) to preserve sequential priors\. Formally:

\(1\)𝐇m=TFBlock​\(Linear​\(LN​\(𝐅m\)\)\+PE\)\\mathbf\{H\}\_\{m\}=\\text\{TF\}\_\{\\text\{Block\}\}\(\\text\{Linear\}\(\\text\{LN\}\(\\mathbf\{F\}\_\{m\}\)\)\+\\text\{PE\}\)
Here,Linear​\(⋅\)\\text\{Linear\}\(\\cdot\)denotes a linear mapping layer andLN​\(⋅\)\\text\{LN\}\(\\cdot\)represents Layer Normalization\(ba2016layer\)\. This transformation yields context\-aware features𝐇m∈ℝT×D\\mathbf\{H\}\_\{m\}\\in\\mathbb\{R\}^\{T\\times D\}, establishing a semantic foundation for the dual\-stream mechanism\.

### 3\.3\.Dual\-Stream Salience\-Context Calibration

We face a core challenge after obtaining context\-aware features: sequence redundancy\. Some segments carry salient sentiment shifts \(e\.g\., fleeting expressions\), while others provide necessary background states\. To filter redundancy, we propose the Dual\-Stream Salience\-Context Calibration mechanism to model both salient semantic shifts and contextual states\.

#### 3\.3\.1\.Soft Salience Disentanglement

To distinguish sentiment shifts from background states, we require time\-step level disentanglement\. We design a soft salience disentanglement mechanism, instead of using hard truncation strategies\. It learns step\-wise salience probabilities, dynamically assigning features𝐇m\\mathbf\{H\}\_\{m\}to either the focus stream carrying salient actions or the ambient stream carrying background states\.

Specifically, we use a lightweight salience scoring network to calculate the probability scoreStS\_\{t\}that thett\-th time step represents a significant moments:

\(2\)𝐒=σ​\(𝐖2⋅GELU​\(𝐖1​𝐇m\+𝐛1\)\+𝐛2\)\\mathbf\{S\}=\\sigma\(\\mathbf\{W\}\_\{2\}\\cdot\\text\{GELU\}\(\\mathbf\{W\}\_\{1\}\\mathbf\{H\}\_\{m\}\+\\mathbf\{b\}\_\{1\}\)\+\\mathbf\{b\}\_\{2\}\)Here,𝐒∈ℝT×1\\mathbf\{S\}\\in\\mathbb\{R\}^\{T\\times 1\}denotes the salience scores,St∈\[0,1\]S\_\{t\}\\in\[0,1\], andσ\\sigmais the sigmoid function\. Then we use𝐒\\mathbf\{S\}as a soft mask for weighted disentanglement:

\(3\)𝐇mfoc\\displaystyle\\mathbf\{H\}\_\{m\}^\{\\text\{foc\}\}=𝐇m⊙𝐒\\displaystyle=\\mathbf\{H\}\_\{m\}\\odot\\mathbf\{S\}𝐇mamb\\displaystyle\\mathbf\{H\}\_\{m\}^\{\\text\{amb\}\}=𝐇m⊙\(1−𝐒\)\\displaystyle=\\mathbf\{H\}\_\{m\}\\odot\(1\-\\mathbf\{S\}\)where⊙\\odotrepresents the Hadamard product\. In this way,𝐇mfoc\\mathbf\{H\}\_\{m\}^\{\\text\{foc\}\}focuses on high\-salience dynamic sentiment shifts, while𝐇mamb\\mathbf\{H\}\_\{m\}^\{\\text\{amb\}\}retains low\-salience contextual background information\. This preserves the complete temporal length and positional information, achieving adaptive information separation rather than brute\-force truncation\.

#### 3\.3\.2\.Dual\-Query Abstraction

Following soft disentanglement, we must transform the streams into compact sentiment tokens aligned with textual semantics\. To achieve this, we introduce a dual\-query mechanism that applies distinct query strategies to aggregate information via cross\-attention\.

Explicit Sentiment Extraction\.The focus stream𝐇mfoc\\mathbf\{H\}\_\{m\}^\{\\text\{foc\}\}contains salient sentiment shifts\. To capture significant variations consistent with human cognition, we utilize a set of predefined text anchors as priors\. These anchors describe modality\-specific sentiment expressions, such as”intense body language”for video and”rapid tone”for audio\. These descriptions are embedded by an LLM as initialized query vectors𝐐focm\\mathbf\{Q\}\_\{\\text\{foc\}\}^\{m\}\. Subsequently, we perform cross\-attention:

\(4\)𝐄mfoc=CrossAttn​\(𝐐focm,𝐇mfoc,𝐇mfoc\)\\mathbf\{E\}\_\{m\}^\{\\text\{foc\}\}=\\text\{CrossAttn\}\(\\mathbf\{Q\}\_\{\\text\{foc\}\}^\{m\},\\mathbf\{H\}\_\{m\}^\{\\text\{foc\}\},\\mathbf\{H\}\_\{m\}^\{\\text\{foc\}\}\)Here,𝐐focm\\mathbf\{Q\}\_\{\\text\{foc\}\}^\{m\}acts as a semantic probe, retrieving dynamic patterns from the focus stream to match the descriptions\. This extracts critical information and explicitly aligns audio\-visual signals with the LLM semantic space\.

Implicit Ambient Extraction\.The ambient stream𝐇mamb\\mathbf\{H\}\_\{m\}^\{\\text\{amb\}\}contains unstructured background information like lighting and noise, which is difficult to describe with text\. Therefore, we employ randomly initialized learnable parameters as query vectors𝐐ambm\\mathbf\{Q\}\_\{\\text\{amb\}\}^\{m\}\. The process is formulated as:

\(5\)𝐄mamb=CrossAttn​\(𝐐ambm,𝐇mamb,𝐇mamb\)\\mathbf\{E\}\_\{m\}^\{\\text\{amb\}\}=\\text\{CrossAttn\}\(\\mathbf\{Q\}\_\{\\text\{amb\}\}^\{m\},\\mathbf\{H\}\_\{m\}^\{\\text\{amb\}\},\\mathbf\{H\}\_\{m\}^\{\\text\{amb\}\}\)Through end\-to\-end training,𝐐ambm\\mathbf\{Q\}\_\{\\text\{amb\}\}^\{m\}adaptively summarizes global background states, providing necessary contextual reference for validating focus stream reliability\.

#### 3\.3\.3\.Context Calibration

Based on theFocus\-Ambient Calibrationhypothesis, salient features may contain noise and require environmental context to verify them\. To achieve this, we utilize the extracted background state𝐄mamb\\mathbf\{E\}\_\{m\}^\{\\text\{amb\}\}to calibrate the focus sentiment𝐄mfoc\\mathbf\{E\}\_\{m\}^\{\\text\{foc\}\}\.

Specifically, we design a Confidence Modulation \(CFM\)\. This module generates gating coefficients based on background information to amplify or suppress sentiment, controlling reliability\. A Feed\-Forward Network \(FFN\) then maps the results to the LLM latent space:

\(6\)α\\displaystyle\\alpha=σ​\(Linear​\(𝐄mamb\)\)\\displaystyle=\\sigma\(\\text\{Linear\}\(\\mathbf\{E\}\_\{m\}^\{\\text\{amb\}\}\)\)𝐄mmod\\displaystyle\\mathbf\{E\}\_\{m\}^\{\\text\{mod\}\}=α⋅𝐄mfoc\\displaystyle=\\alpha\\cdot\\mathbf\{E\}\_\{m\}^\{\\text\{foc\}\}𝐙m\\displaystyle\\mathbf\{Z\}\_\{m\}=FFN​\(𝐄mfoc\+𝐄mmod\)\\displaystyle=\\text\{FFN\}\(\\mathbf\{E\}\_\{m\}^\{\\text\{foc\}\}\+\\mathbf\{E\}\_\{m\}^\{\\text\{mod\}\}\)whereα\\alphais the confidence coefficient\. The final sentiment token𝐙m\\mathbf\{Z\}\_\{m\}integrates structural sentiment shifts with contextual constraints, ensuring high semantic stability and consistency\.

#### 3\.3\.4\.Unified LLM Reasoning

We obtain calibrated sentiment tokens𝐙v\\mathbf\{Z\}\_\{v\}and𝐙a\\mathbf\{Z\}\_\{a\}after processing\. To perform unified reasoning, we concatenate modality features and append a \[CLS\]\(radford2019language\)token to build the complete input sequence:

\(7\)𝐇in=Concat​\(𝐙t,𝐙v,𝐙a,\[CLS\]\)\\mathbf\{H\}\_\{\\text\{in\}\}=\\text\{Concat\}\(\\mathbf\{Z\}\_\{t\},\\mathbf\{Z\}\_\{v\},\\mathbf\{Z\}\_\{a\},\[\\text\{CLS\}\]\)Finally, we perform cross\-modal interaction by utilizing the powerful sequence modeling capabilities of LLMs\. And we extract the \[CLS\] token to obtain the sentiment predictiony^\\hat\{y\}\.

### 3\.4\.Optimization Objectives

To balance prediction accuracy and feature disentanglement, we employ a joint optimization strategy including regression loss and sparsity regularization\.

#### 3\.4\.1\.Task Loss

For sentiment prediction on datasets like MOSI/MOSEI, we use Mean Absolute Error \(MAE\) as the primary loss function:

\(8\)ℒcls=1N​∑i=1N\|yi−y^i\|\\mathcal\{L\}\_\{\\text\{cls\}\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\|y\_\{i\}\-\\hat\{y\}\_\{i\}\|

#### 3\.4\.2\.Sparsity Loss

In soft salience disentanglement, the model risks trivial solutions where allStS\_\{t\}are predicted as 1 \(keep all\) or 0 \(drop all\)\. To encourage the model to select truly salient moments, we introduce a target\-ratio sparsity loss\. This loss constrains the average activation rate near a preset sparsity ratiokk\(e\.g\.,k=0\.2k=0\.2implies only 20% significant time steps\):

\(9\)ℒsp=\|\(1T​∑t=1TSt\)−k\|\\mathcal\{L\}\_\{\\text\{sp\}\}=\\left\|\\left\(\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}S\_\{t\}\\right\)\-k\\right\|wherek∈\[0,1\]k\\in\[0,1\]\. This explicitly guides the model to focus on the informative time steps\. The total optimization objective is defined as:

\(10\)ℒtotal=ℒcls\+λ​ℒsp\\mathcal\{L\}\_\{\\text\{total\}\}=\\mathcal\{L\}\_\{\\text\{cls\}\}\+\\lambda\\mathcal\{L\}\_\{\\text\{sp\}\}whereλ\\lambdais a weighted hyperparameter\.

Table 1\.Performance comparison on the MOSI and MOSEI datasets\. We compare our method against Traditional and LLM\-based baselines\.Bold: best result;Underline: second best result\.†\{\\dagger\}: reproduced result using public code\.MethodsMOSIMOSEIMAE↓\\downarrowCorr↑\\uparrowAcc\-7↑\\uparrowAcc\-5↑\\uparrowAcc\-2↑\\uparrowF1↑\\uparrowMAE↓\\downarrowCorr↑\\uparrowAcc\-7↑\\uparrowAcc\-5↑\\uparrowAcc\-2↑\\uparrowF1↑\\uparrowTraditional MethodsTFN\(Zadeh2017TensorFN\)0\.9010\.69834\.9039\.3980\.0880\.070\.5930\.70050\.2053\.1082\.5082\.10LMF\(liu2018efficient\)0\.9170\.69533\.2038\.1382\.5082\.400\.6230\.67748\.0052\.9082\.0082\.10Graph\-MFN\(zadeh2018memory\)0\.9560\.64934\.6438\.6378\.3578\.350\.5750\.71351\.3752\.6983\.4883\.43MuLT\(tsai2019multimodal\)0\.8710\.69840\.0042\.6883\.0082\.000\.5800\.70351\.8054\.1882\.5082\.30MISA\(Hazarika2020MISAMA\)0\.7770\.77841\.3747\.0883\.5483\.580\.5580\.75252\.0553\.6384\.6784\.66MAG\-BERT\(rahman2020integrating\)0\.7270\.78143\.62\-84\.4384\.610\.5430\.75552\.67\-84\.8284\.71Self\-MM\(yu2021learning\)0\.7120\.79545\.79\-82\.5482\.680\.5290\.76753\.46\-82\.6882\.95MMIM\(han2021improving\)0\.7380\.78145\.00\-85\.1085\.000\.5470\.75253\.10\-85\.1085\.00DMD\(li2023decoupled\)0\.752\-46\.06\-83\.2383\.290\.543\-52\.78\-84\.6284\.62CubeMLP\(sun2022cubemlp\)0\.7700\.76745\.50\-85\.6085\.500\.5290\.76054\.90\-85\.1084\.50DLF†\(wang2025dlf\)0\.7220\.79446\.9453\.3585\.8285\.810\.5500\.75952\.5454\.4184\.1584\.26KAN\-MCP†\(luo2025towards\)0\.6590\.82946\.9453\.5086\.1386\.060\.5120\.79254\.9356\.7386\.8286\.82LLM\-based MethodsUniMSE\(hu2022unimse\)0\.6910\.80948\.68\-85\.8585\.830\.5230\.77354\.39\-85\.8685\.79UniSA\-GPT2\(li2023unisa\)1\.410\-15\.45\-44\.75\-0\.838\-41\.36\-71\.02\-UniSA\-T5\(li2023unisa\)0\.900\-37\.46\-76\.82\-0\.546\-52\.50\-84\.22\-UniSA\-BART\(li2023unisa\)0\.742\-48\.54\-84\.11\-0\.587\-50\.03\-84\.93\-MSE\-Qwen\-1\.8B†\(yang2025mse\)0\.9920\.61633\.6740\.3875\.6675\.670\.5600\.72251\.9253\.6484\.3583\.70MSE\-LLaMA2\-7B†\(yang2025mse\)0\.7620\.78242\.1350\.2982\.6582\.670\.5320\.76353\.5355\.5785\.9485\.51MSE\-ChatGLM\-6B†\(yang2025mse\)0\.6470\.85246\.0654\.3788\.0588\.010\.5180\.77554\.7156\.7986\.8286\.52Proposed MethodVanilla SentiLLM0\.8230\.76536\.7343\.7383\.9983\.790\.5290\.78253\.1754\.8286\.7186\.71SentiLLM \(Ours\)0\.6390\.84648\.6856\.7189\.0288\.990\.5080\.79955\.2956\.8487\.2987\.29

## 4\.Experiments

### 4\.1\.Datasets and Evaluation Metrics

Datasets\.We evaluate our approach on four datasets: CMU\-MOSI\(Zadeh2016MOSIMC\), CMU\-MOSEI\(zadeh2018multimodal\), CH\-SIMS\(yu2020ch\), and CH\-SIMS v2\(liu2022make\)\. Detailed dataset statistics are provided in theSupplementary Material\.

Evaluation Metrics\.To ensure fair comparison, we align with prior works\(wang2025dlf;yu2020ch;liu2022make\)\. For MOSI and MOSEI, we report 7\-class accuracy \(Acc\-7\), 5\-class accuracy \(Acc\-5\), binary accuracy \(Acc\-2\), F1 score \(F1\), mean absolute error \(MAE\) and Pearson correlation \(Corr\)\. For CH\-SIMS and CH\-SIMS v2, we utilize Acc\-5, Acc\-3, Acc\-2, F1, MAE, and Corr\.

### 4\.2\.Baseline Methods

We compare our method against state\-of\-the\-art models, categorized into:

1. \(1\)Traditional Methods: TFN\(Zadeh2017TensorFN\), LMF\(liu2018efficient\), Graph\-MFN\(zadeh2018multimodal\), MuLT\(tsai2019multimodal\), MISA\(Hazarika2020MISAMA\), MAG\-BERT\(rahman2020integrating\), Self\-MM\(yu2021learning\), MMIM\(han2021improving\), AV\-MC\(liu2022make\), DMD\(li2023decoupled\), CubeMLP\(sun2022cubemlp\), DLF\(wang2025dlf\), and KAN\-MCP\(luo2025towards\);
2. \(2\)LLM\-based Methods: UniMSE\(hu2022unimse\), UniSA\(li2023unisa\)and MSE\-Adapter\(yang2025mse\);
3. \(3\)Vanilla SentiLLM \(Baseline\): A variant without our proposed module, using only linear layers to project LLM inputs\.

### 4\.3\.Implementation Details

Feature Extraction\.For MOSI and MOSEI, we follow\(wang2025dlf\)to obtain raw text descriptions, video, and audio features\. For CH\-SIMS and CH\-SIMS v2, we utilize data files provided by\(yu2020ch\)and\(liu2022make\), respectively\.

Settings\.We conduct all experiments on NVIDIA RTX 5090 32GB GPUs\. For LLM, we use Qwen2\.5\-1\.5B\(Yang2024Qwen25TR\), initializing its parameters from the official release111[https://huggingface\.co/Qwen/Qwen2\.5\-1\.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B)\. The input dimensionDDis set to 256\. The hyperparameters are adjusted based on the dataset’s scale, as detailed inSupplementary Material\. Notably, SentiLLM has 1549M parameters \(Qwen2\.5\-1\.5B: 1544M, Proposed Module: 5M\),with only 0\.32% trainable, substantially fewer than mainstream approaches\.

More Experimental Results\.We defer detailed analyses on the impact of different modalities, the roles of soft salience disentanglement, the effectiveness of sparsity loss, and additional visualizations to theSupplement Material\.

### 4\.4\.Comparison with State\-of\-the\-arts

Table 2\.Performance comparison on the CH\-SIMS and CH\-SIMS v2 datasets\. We compare our method against Traditional and LLM\-based baselines\.Bold: best result;Underline: second best result\.†\{\\dagger\}: reproduced result using public code\.MethodsCH\-SIMSCH\-SIMS v2MAE↓\\downarrowCorr↑\\uparrowAcc\-5↑\\uparrowAcc\-3↑\\uparrowAcc\-2↑\\uparrowF1↑\\uparrowMAE↓\\downarrowCorr↑\\uparrowAcc\-5↑\\uparrowAcc\-3↑\\uparrowAcc\-2↑\\uparrowF1↑\\uparrowTraditional MethodsTFN\(Zadeh2017TensorFN\)0\.4370\.582––77\.1076\.900\.3220\.66253\.3070\.9078\.1078\.10LMF\(liu2018efficient\)0\.4380\.578––77\.4077\.400\.3270\.65151\.6070\.0077\.8077\.80MuLT\(tsai2019multimodal\)0\.4420\.58140\.0065\.7078\.2078\.500\.3000\.73854\.6074\.2080\.8080\.70MISA\(Hazarika2020MISAMA\)0\.4470\.563––76\.5076\.600\.3420\.67147\.5068\.9078\.2078\.30MAG\-BERT†\(rahman2020integrating\)0\.4570\.55937\.2065\.6575\.4975\.940\.3380\.65149\.3269\.5476\.5076\.63Self\-MM\(yu2021learning\)0\.4110\.60143\.1066\.1078\.6078\.600\.3150\.69153\.5072\.7078\.7078\.60MMIM\(han2021improving\)0\.4220\.59742\.0065\.5078\.3078\.200\.3390\.64150\.5070\.4077\.8077\.80AV\-MC†\(liu2022make\)0\.3780\.67545\.0870\.0280\.3180\.490\.2980\.73353\.5872\.4480\.3780\.25DLF†\(wang2025dlf\)0\.5470\.22227\.5751\.6469\.8062\.780\.4350\.43035\.2052\.1367\.7067\.40KAN\-MCP†\(luo2025towards\)0\.4930\.45235\.6759\.9672\.8773\.070\.3920\.52540\.9160\.6471\.3771\.36LLM\-based MethodsMSE\-Qwen\-1\.8B†\(yang2025mse\)––––––0\.3180\.66854\.7472\.6379\.5979\.64MSE\-LLaMA2\-7B†\(yang2025mse\)––––––0\.3930\.53747\.7869\.1575\.7375\.83MSE\-ChatGLM\-6B†\(yang2025mse\)––––––0\.2900\.73054\.0676\.0281\.9181\.87Proposed MethodVanilla SentiLLM0\.4780\.50835\.0160\.1874\.6274\.400\.3680\.63043\.2365\.4776\.1176\.19SentiLLM \(Ours\)0\.4220\.61644\.2067\.1880\.9680\.220\.3290\.68948\.5571\.1879\.8879\.91

MOSI & MOSEI\.As shown in Table[1](https://arxiv.org/html/2607.27790#S3.T1), our proposed method achieves state\-of\-the\-art performance on both MOSI and MOSEI datasets, significantly outperforming traditional baselines and recent LLM\-based approaches\. On MOSI, our method achieves the best Acc\-2 of 89\.02% and Acc\-7 of 48\.68%, surpassing the strongest competitor, MSE\-ChatGLM3\-6B, by 0\.97% in accuracy\. Notably, our method utilizes a much smaller backbone compared to LLaMA2\-7B and ChatGLM3\-6B, yet it yields superior results\. This efficiency is attributed to our dual\-stream mechanism, which extract semantically aligned tokens rather than relying solely on the LLM’s inherent capacity\. Furthermore, the significant margin between our method and the Vanilla SentiLLM highlights the critical role of our structural abstraction mechanism in bridging the modality gap, preventing the LLM from being overwhelmed by noisy, unaligned multimodal signals\.

CH\-SIMS & CH\-SIMS v2\.We further evaluate the generalization capability of our method on Chinese sentiment datasets, as shown in Table[2](https://arxiv.org/html/2607.27790#S4.T2)\. On CH\-SIMS, our method establishes a new state\-of\-the\-art Acc\-2 of 80\.96%, outperforming the robust AV\-MC baseline\. On the larger CH\-SIMS v2 dataset, while MSE\-ChatGLM3\-6B achieves the top performance due to its larger parameter size and native Chinese pre\-training, our method remains highly competitive, achieving 79\.88% Acc\-2 and surpassing MSE\-LLaMA2\-7B \(\+4\.15%\) and MSE\-Qwen\-1\.8B \(\+0\.29%\)\. Crucially, our approach demonstrates consistent improvements over the Vanilla SentiLLM across both datasets, confirming that our concept of learning structural abstraction is linguistically universal\. This suggests that our method effectively captures universal salient sentiment shifts, independent of the specific language, making it a robust solution for cross\-lingual multimodal sentiment analysis\.

### 4\.5\.Ablation Studies

Table 3\.Ablation study on the contribution of different model components on the MOSI dataset\.Model ConfigurationStepMAE↓\\downarrowCorr↑\\uparrowAcc\-7↑\\uparrowAcc\-2↑\\uparrowF1↑\\uparrowSentiLLM \(Ours\)00\.6400\.84648\.4089\.0288\.99w/oContext Calibration10\.6430\.84946\.2188\.8788\.88w/oFocus Stream20\.6820\.83243\.5987\.6587\.61w/oAmbient Stream20\.6760\.82944\.6187\.8087\.74w/oDual Stream30\.6990\.81244\.1785\.6785\.66Vanilla SentiLLM40\.8230\.76536\.7383\.9983\.79

Table 4\.Impact of different query initialization strategies on the MOSI dataset\.Prompt StrategyMAE↓\\downarrowCorr↑\\uparrowAcc\-7↑\\uparrowAcc\-5↑\\uparrowAcc\-2↑\\uparrowF1↑\\uparrowSemantic Sentences0\.6400\.84648\.4056\.7189\.0288\.99Simplified Keywords0\.6440\.84047\.5254\.9687\.3587\.31Random Vectors0\.6840\.82843\.4450\.5886\.7486\.71

Effectiveness of Model Components\.The contribution of each component is reported in Table[3](https://arxiv.org/html/2607.27790#S4.T3)\. The significant performance drop is observed when removing the ambient stream or the dual\-stream structure\. This highlights the importance of contextual background alongside salient sentiment shifts\. Additionally, the contextual calibration is crucial for fine\-grained improvements, confirming the necessity of calibrating salient sentiment shifts\.

Impact of Query Initialization\.The results in Table[4](https://arxiv.org/html/2607.27790#S4.T4)examine the impact of query initialization\. This demonstrates that our detailed semantic sentences consistently outperform simplified keywords \(e\.g\., positive/negative\) or random vectors\. The performance gap confirms that leveraging the pre\-aligned LLM semantic space is crucial for guiding the attention mechanism to capture subtle, structurally meaningful affective cues\.

Table 5\.Generalization analysis across different LLM families on MOSI and CH\-SIMS datasets\. “Δ\\Delta” denotes the performance gap between Ours and Baseline\.Greenindicates improvement, whileRedindicates degradation\.LLM BackboneSettingMOSICH\-SIMSMAE↓\\downarrowCorr↑\\uparrowAcc\-7↑\\uparrowAcc\-2↑\\uparrowF1↑\\uparrowMAE↓\\downarrowCorr↑\\uparrowAcc\-5↑\\uparrowAcc\-2↑\\uparrowF1↑\\uparrowQwen2\.5\-1\.5BBaseline0\.8230\.76536\.7383\.9983\.790\.4780\.50835\.0174\.6274\.40Ours0\.6400\.84648\.4089\.0288\.990\.4220\.61644\.2080\.9680\.22Δ\\Delta\(\-0\.183\)\(\+0\.081\)\(\+11\.67\)\(\+5\.03\)\(\+5\.20\)\(\-0\.056\)\(\+0\.108\)\(\+9\.19\)\(\+6\.34\)\(\+5\.82\)InternLM2\.5\-1\.8BBaseline0\.9580\.66231\.7878\.6678\.560\.4310\.59941\.1479\.8779\.29Ours0\.7030\.82544\.3185\.5285\.270\.4330\.60041\.9480\.3180\.58Δ\\Delta\(\-0\.255\)\(\+0\.163\)\(\+12\.53\)\(\+6\.86\)\(\+6\.71\)\(\+0\.002\)\(\+0\.001\)\(\+0\.80\)\(\+0\.44\)\(\+1\.29\)Gemma\-2\-2B\-itBaseline0\.7250\.80543\.0085\.9885\.980\.4330\.61042\.2378\.1276\.79Ours0\.5990\.86549\.5689\.1889\.120\.4280\.61242\.8977\.9077\.04Δ\\Delta\(\-0\.126\)\(\+0\.060\)\(\+6\.56\)\(\+3\.20\)\(\+3\.14\)\(\-0\.005\)\(\+0\.002\)\(\+0\.66\)\(\-0\.22\)\(\+0\.25\)Llama\-3\.2\-3BBaseline0\.7930\.76939\.6584\.4584\.430\.5690\.50336\.7674\.8469\.47Ours0\.6690\.83244\.7587\.2087\.180\.4930\.55437\.6477\.2474\.46Δ\\Delta\(\-0\.124\)\(\+0\.063\)\(\+5\.10\)\(\+2\.75\)\(\+2\.75\)\(\-0\.076\)\(\+0\.051\)\(\+0\.88\)\(\+2\.40\)\(\+4\.99\)

Analysis of sparsity ratiokk\.As illustrated in Figure[2](https://arxiv.org/html/2607.27790#S4.F2), model performance exhibits a distinct pattern regarding sparsity ratio\. Low sparsity causes severe information loss by discarding salient moments, whereas high sparsity introduces excessive noise, interfering with LLM reasoning\. An optimal sparsity ratio achieves the best trade\-off between retaining sentiment shifts and background states, validating the effectiveness of soft salience disentanglement\.

![Refer to caption](https://arxiv.org/html/2607.27790v1/x2.png)Figure 2\.Ablation study on the impact of sparsity ratiokkon the MOSI dataset\.LLMs Scalability\.We investigate scalability by varying the LLM size from 0\.5B to 7B parameters\. As shown in Figure[3](https://arxiv.org/html/2607.27790#S4.F3), our method exhibits strong scalability properties\. Although Acc\-2 fluctuates slightly at 3B, the strict monotonic decrease in MAE indicates that larger models provide more precise prediction scores\. This suggests that our method effectively leverages the capabilities of larger LLMs\.

Generalization to Different LLMs\.As presented in Table[5](https://arxiv.org/html/2607.27790#S4.T5), to demonstrates generalization capabilities, we evaluate four distinct LLMs: Qwen\(Yang2024Qwen25TR\), InternLM\(cai2024internlm2\), Gemma\(team2024gemma\), and Llama3\(dubey2024llama\)\. On the English\-dominant MOSI, our method yields consistent gains, notably boosting the Chinese\-optimized InternLM by \+6\.86% \(Acc\-2\)\. On the Chinese CH\-SIMS, results reflect linguistic characteristics: the bilingual Qwen benefits most significantly \(\+6\.34%\), demonstrating perfect synergy\. While the English\-centric Gemma faces a slight accuracy trade\-off due to cross\-lingual alignment gaps, its improved F1 score confirms structural robustness\. Even against InternLM’s exceptionally high baseline, our method maintains performance gains, verifying that SentiLLM provides a universal, semantic\-aligned structural abstraction, regardless of the underlying pre\-training distribution\.

![Refer to caption](https://arxiv.org/html/2607.27790v1/x3.png)Figure 3\.Scalability analysis with different Qwen2\.5 sizes on the MOSI dataset\.
### 4\.6\.Extended Analysis

MOSIMOSEICH\-SIMSCH\-SIMS v2

Baseline

![Refer to caption](https://arxiv.org/html/2607.27790v1/x4.png)![Refer to caption](https://arxiv.org/html/2607.27790v1/x5.png)![Refer to caption](https://arxiv.org/html/2607.27790v1/x6.png)![Refer to caption](https://arxiv.org/html/2607.27790v1/x7.png)Ours

![Refer to caption](https://arxiv.org/html/2607.27790v1/x8.png)![Refer to caption](https://arxiv.org/html/2607.27790v1/x9.png)![Refer to caption](https://arxiv.org/html/2607.27790v1/x10.png)![Refer to caption](https://arxiv.org/html/2607.27790v1/x11.png)\\lxSVG@picture

\\endlxSVG@pictureHighly Negative\\lxSVG@picture\\endlxSVG@pictureNegative\\lxSVG@picture\\endlxSVG@pictureWeak Negative\\lxSVG@picture\\endlxSVG@pictureNeutral\\lxSVG@picture\\endlxSVG@pictureWeak Positive\\lxSVG@picture\\endlxSVG@picturePositive\\lxSVG@picture\\endlxSVG@pictureHighly Positive

Figure 4\.Visualization of learned representation spaces\. Rows indicate the methods \(Baseline vs\. Ours\) and columns represent different datasets\. The color legend represents sentiment intensity classes\.Video ModalAudio ModalFocus Stream Ambient Stream

![Refer to caption](https://arxiv.org/html/2607.27790v1/x12.png)
![Refer to caption](https://arxiv.org/html/2607.27790v1/x13.png)
![Refer to caption](https://arxiv.org/html/2607.27790v1/x14.png)
![Refer to caption](https://arxiv.org/html/2607.27790v1/x15.png)

Figure 5\.Visualization of feature disentanglement on the MOSI dataset\.Visualization of Learned Representation Space\.In Figure[4](https://arxiv.org/html/2607.27790#S4.F4), we visualize the t\-SNE distribution\(linderman2017efficient\)of the final \[CLS\] token on all four datasets\. The baseline consistently exhibits a ”chaotic cloud” distribution with severe class entanglement, indicating weak semantic discriminability\. In strong contrast, our method reconstructs the feature space into a highly structured manifold across both languages\. Specifically, on MOSI and MOSEI, we observe a clear continuous gradient transitioning from highly negative \(red\) to highly positive \(purple\), reflecting the model’s precise grasp of fine\-grained sentiment intensity\. Similarly, on the Chinese CH\-SIMS and CH\-SIMS v2, the sentiment clusters become significantly more compact and distinguishable\. This distinct clustering confirms that our method successfully maps raw information into a semantically separable space, reducing classifier ambiguity\.

Visualization of Dual\-Stream Feature Distribution\.In Figure[5](https://arxiv.org/html/2607.27790#S4.F5), we qualitatively validate the dual\-stream mechanism on the MOSI dataset\. The focus stream exhibits strong linear separability under LDA projection, confirming the effective disentanglement of sentiment semantics\. Conversely, the ambient stream shows a sentiment\-agnostic distribution under t\-SNE projection, effectively absorbing background states without leaking sentiment information\.

Analysis of Fine\-grained Sentiment Prediction\.As shown in Figure[6](https://arxiv.org/html/2607.27790#S4.F6), we present the confusion matrix and corresponding accuracy of 7\-class sentiment for the MOSI dataset\. We observe that the samples for ”HN” and ”HP” yield the worst performance\. The confusion matrix reveals these classes are relatively less than other sentiments, which indicates that the long\-tail distribution limits performance on extreme sentiments\.

![Refer to caption](https://arxiv.org/html/2607.27790v1/x16.png)![Refer to caption](https://arxiv.org/html/2607.27790v1/x17.png)
Figure 6\.Confusion matrix and per\-class accuracy for 7\-class sentiment prediction on the MOSI dataset

## 5\.Conclusion

Grounded in the structural isomorphism between non\-verbal modalities and natural language, we explore the potential of LLMs for multimodal sentiment analysis\. However, the appearance\-semantic misalignment hinders effective integration within the LLM semantic space\. To address this, we propose SentiLLM, a framework incorporating a Dual\-Stream Salience\-Context Calibration Mechanism\. This framework decouples audio\-visual sequences into a focus stream containing salient sentiment shifts and an ambient stream representing contextual states\. Extensive experiments across four datasets confirm the effectiveness of our proposed method\. We anticipate this structural abstraction paradigm will offer novel insights into non\-verbal modality sequence modeling\.

###### Acknowledgements\.

This work is supported by the National Natural Science Foundation of China \(Grant No\. 62272188\)\.

## References

## Appendix

## Appendix AExperimental Details

### A\.1\.Datasets

In this section, we briefly introduce the datasets:

1. \(1\)MOSI\(Zadeh2016MOSIMC\): MOSI contains 2,199 segments from 93 videos\. Speakers express personal opinions in a monologue format\. Annotations cover a 7\-point sentiment scale from \-3 \(strongly negative\) to \+3 \(strongly positive\)\.
2. \(2\)MOSEI\(zadeh2018multimodal\): MOSEI is an extended version of MOSI with larger scale\. It contains 22,856 segments across 25 topics from YouTube\.
3. \(3\)CH\-SIMS\(yu2020ch\): CH\-SIMS is a Chinese dataset with both unimodal and multimodal annotations\. It includes 2,281 segments from movies and TV shows, annotated on a 5\-point scale \(\-2 to \+2\)\.
4. \(4\)CH\-SIMS v2\(liu2022make\): CH\-SIMS v2 extends CH\-SIMS with data from 11 scenarios, including interviews and talk shows, to simulate real\-world diversity\. This demands precise multimodal integration for correct prediction\.

### A\.2\.Baseline Methods

We compare the proposed SentiLLM with the following competitive baselines:

1. \(1\)Tensor Fusion Network \(TFN\)\(Zadeh2017TensorFN\): TFN explicitly captures unimodal, bimodal, and trimodal interactions via mathematical outer products\.
2. \(2\)Low\-rank Multimodal Fusion \(LMF\)\(liu2018efficient\): LMF addresses TFN complexity by utilizing Low\-rank Tensor Decomposition to reduce parameters from exponential to linear scale\.
3. \(3\)Graph Memory Fusion Network \(Graph\-MFN\)\(zadeh2018multimodal\): Graph\-MFN introduces a dynamic fusion graph to adaptively adjust edge weights between vertices, selecting effective fusion paths\.
4. \(4\)Multimodal Transformer \(MuLT\)\(tsai2019multimodal\): MuLT abandons alignment and fusion strategy, utilizing cross\-modal attention to directly capture long\-range dependencies\.
5. \(5\)MISA\(Hazarika2020MISAMA\): MISA projects modalities into shared and specific sub\-spaces, enforcing separation via orthogonality constraints and reconstruction losses\.
6. \(6\)Multimodal Adaptation Gate BERT \(MAG\-BERT\)\(rahman2020integrating\): MAG\-BERT injects multimodal information into BERT via a soft adaptation gate, refining text representations\.
7. \(7\)Self\-Supervised Multi\-Task Learning \(Self\-MM\)\(yu2021learning\): Self\-MM generates pseudo\-labels from multimodal targets to construct self\-supervised multi\-task learning\.
8. \(8\)Multimodal Mutual Information Maximization \(MMIM\)\(han2021improving\): MMIM maximizes mutual information between fusion results and unimodal inputs to preserve key modality\-specific information\.
9. \(9\)Decoupled Multimodal Distillation \(DMD\)\(li2023decoupled\): DMD decomposes representations into modality\-agnostic and modality\-specific parts via knowledge distillation\.
10. \(10\)MLP\-Based Fusion \(CubeMLP\)\(sun2022cubemlp\): CubeMLP treats data as a 3D cube, mixing features across temporal, modality, and channel axes using a pure MLP architecture\.
11. \(11\)Audio\-Visual Mixup Consistency \(AV\-MC\)\(liu2022make\): AV\-MC introduces Mixup strategies to mitigate model over\-reliance on the text modality\.
12. \(12\)Disentangled\-Language\-Focused Framework \(DLF\)\(wang2025dlf\): DLF revisits disentanglement with a language\-dominant perspective, using language queries to aggregate specific information from other modalities\.
13. \(13\)KAN\-MCP\(luo2025towards\): KAN\-MCP places activation functions on edges using KANs, making the fusion process transparent and interpretable\.
14. \(14\)Unified MSA and ERC \(UniMSE\)\(hu2022unimse\): UniMSE uses T5 as a backbone, incorporating cross\-modal contrastive learning guided by syntax dependency trees\.
15. \(15\)UniSA\(li2023unisa\): UniSA converts audio\-visual features into discrete tokens via clustering for LLM generation\.
16. \(16\)MSE\-Adapter\(yang2025mse\): MSE\-Adapter builds a Text\-Guide\-Mixer to explicitly connect text tokens with non\-text features for LLM analysis\.

### A\.3\.LLM Backbones

We compare the proposed SentiLLM with the following competitive baselines:

1. \(1\)Qwen2\.5\-1\.5B\(Yang2024Qwen25TR\): Qwen2\.5\-1\.5B achieves exceptional instruction following with minimal parameters, leveraging 18T tokens and Grouped\-Query Attention\.
2. \(2\)Llama\-3\.2\-3B\(dubey2024llama\): Llama\-3\.2\-3B is distilled and pruned from a 70B Llama\-3 model, inheriting complex logical reasoning capabilities\.
3. \(3\)Gemma\-2\-2B\-it\(team2024gemma\): Gemma\-2\-2B\-it combines sliding windows and global attention to capture both local details and global context\.
4. \(4\)InternLM2\.5\-1\.8B\(cai2024internlm2\): InternLM2\.5\-1\.8B features strong long\-context processing, making it ideal for analyzing temporal evolution in multi\-turn dialogue\.

### A\.4\.Training Hyperparameters

We adjusted hyperparameters based on dataset scale and complexity\. Details are provided in Table[6](https://arxiv.org/html/2607.27790#A2.T6)\.

## Appendix BExtended Ablation Studies

### B\.1\.Impact of Different Modalities

Table[7](https://arxiv.org/html/2607.27790#A2.T7)validates the necessity of multimodal integration\. The ”text only” baseline performs decently due to LLM capabilities, but incorporating visual or audio signals brings significant gains\. This indicates that non\-verbal modalities carry critical sentiment information ignored by text, and our method effectively extracts text\-aligned sentiment elements to activate LLM reasoning\.

Table 6\.Hyperparameter settings across different datasets\.ParameterMOSIMOSEICH\-SIMSCH\-SIMS v2OptimizerAdamWSchedulerCosine DecayLearning Rate1e\-32e\-35e\-41e\-3Weight Decay0\.010\.010\.050\.05Warmup Ratio0\.100\.200\.100\.10Batch Size16161616Gradient Accum2222Total Epochs50304030Sparsity Ratiokk0\.60\.40\.60\.6

Table 7\.Ablation study on modality impact on the MOSI dataset\.ModalitiesMAE\(↓\\downarrow\)Corr\(↑\\uparrow\)Acc\-7\(↑\\uparrow\)Acc\-5\(↑\\uparrow\)Acc\-2\(↑\\uparrow\)F1\(↑\\uparrow\)Text Only0\.7600\.79339\.5047\.6784\.3084\.21Text \+ Video0\.6760\.83143\.5952\.1988\.1188\.08Text \+ Audio0\.6480\.84146\.6554\.0888\.5788\.53Text \+ Video \+ Audio0\.6400\.84648\.4056\.7189\.0288\.99

Table 8\.Comparative analysis of Soft Salience Disentanglement and Hard Truncation\. Sparsity ratiokkis set to 0\.6\.Gating StrategyMAE\(↓\\downarrow\)Corr\(↑\\uparrow\)Acc\-7\(↑\\uparrow\)Acc\-5\(↑\\uparrow\)Acc\-2\(↑\\uparrow\)F1\(↑\\uparrow\)Hard Truncation0\.6640\.83546\.7954\.2387\.9687\.95Soft Salience0\.6400\.84648\.4056\.7189\.0288\.99

Table 9\.Ablation study on the effectiveness of sparsity loss on the MOSI dataset\.Loss FunctionMAE\(↓\\downarrow\)Corr\(↑\\uparrow\)Acc\-7\(↑\\uparrow\)Acc\-5\(↑\\uparrow\)Acc\-2\(↑\\uparrow\)F1\(↑\\uparrow\)w/o Sparsity Loss0\.6910\.83042\.5751\.1788\.4188\.38w/ Sparsity Loss0\.6400\.84648\.4056\.7189\.0288\.99

Table 10\.Ablation study on computational efficiency and overhead for single\-batch inferenceSettingFLOPs\(G\)Memory\(MB\)Latency\(ms\)Throughput\(samples/s\)Params\(M\)Vanilla SentiLLM2725\.216170\.3644\.48359\.681543\.91SentiLLM \(Ours\)3050\.766241\.8553\.43299\.461548\.59Incremental \(Δ\\Delta\)\+325\.55\+71\.49\+8\.95–\+4\.68
### B\.2\.Soft Salience Disentanglement and Hard Truncation

As shown in Table[8](https://arxiv.org/html/2607.27790#A2.T8), our soft salience disentanglement outperforms hard truncation across all metrics\. This indicates that soft salience disentanglement maintains differentiability and assigns continuous salience scores, ensuring flexible separation between focus and ambient streams\.

### B\.3\.Effectiveness of Sparsity Loss

Table[9](https://arxiv.org/html/2607.27790#A2.T9)highlights the critical role of sparsity loss\. A sharp drop in fine\-grained accuracy is observed when removing this constraint\. This suggests that salience scores converge to trivial solutions without explicit regularization\. And the sparsity loss forces the model to make decisive binary distinctions, ensuring meaningful separation between the two streams\.

### B\.4\.Model Efficiency

To substantiate our parameter\-efficient claim, single\-batch inference tests on an RTX 5090 \(Table[10](https://arxiv.org/html/2607.27790#A2.T10)\) demonstrate that SentiLLM introduces minimal overhead\. This confirms its viability as a lightweight reasoning enhancer\.

## Appendix CDetailed Experiment Results

### C\.1\.Semantic Sentences

As shown in Table[11](https://arxiv.org/html/2607.27790#A3.T11)and[12](https://arxiv.org/html/2607.27790#A3.T12), we detail the natural language prompts used in our semantic sentence setting\. These sentence\-level prompts cover multiple sentiment dimensions, including valence, intensity, and social cues, instead of using simplified keywords \(positive/negative\) or random vectors\.

### C\.2\.Visualization of Dual\-Stream Feature Distribution

We visualize the dual\-stream feature distributions across four datasets in Figure[7](https://arxiv.org/html/2607.27790#A3.F7)\. On smaller datasets \(MOSI and CH\-SIMS\), the focus stream exhibit strong linear separability via distinct bimodal peaks, while the ambient stream remains chaotically entangled, successfully absorbing background states\. Conversely, on larger\-scale datasets \(MOSEI and CH\-SIMS v2\), the focus stream shows significant distributional overlap between sentiment polarities\. This overlap, particularly pronounced in CH\-SIMS v2, directly reveals the performance bottleneck preventing state\-of\-the\-art results on this specific benchmark\. And these finding indicate that scaling our disentanglement mechanism to handle highly diverse, large\-scale dataset remains a vital direction for future work\.

### C\.3\.Analysis of Fine\-grained Sentiment Prediction

To provide a deeper understanding of the model’s predictive behavior, we visualize the confusion matrix and per\-class accuracies across all four datasets in Figure[8](https://arxiv.org/html/2607.27790#A3.F8)\. Consistent with our observations on MOSI, the MOSEI dataset exhibits a serve long\-tail distribution where the model struggles significantly with the extreme polarities \(”HN” and ”HP”\) due to data sparsity\. Interestingly, on the CH\-SIMS and CH\-SIMS v2 datasets, the bottleneck shifts to ”NT” class, which yields the lowest accuracy\. This discrepancy is likely because neutral expression in these datasets are highly nuanced and easily misclassified into neighboring weak sentiment classes \(e\.g\., WN or WP\)\. These results indicate that the inherent class imbalance constrains the theoretical upper bound of sentiment prediction, highlighting long\-tail multimodal learning as a critical direction for our future work\.

Table 11\.Complete prompt list of semantic sentences used for query initialization on MOSI and MOSEI datasets\.\\lxSVG@picture

Complete Prompt List of Semantic Sentences for MOSI / MOSEI\# Video Modality\>a face showing clear positive emotion \(smile, joyful\)\# Positive Valence\>a face showing clear negative emotion \(frown, sadness, anger\)\# Negative Valence\>a facial expression with moderate intensity \(subtle emotion\)\# Low Intensity\>a sudden change in facial expression or head movement\# Temporal Dynamics\>a speaker gesturing emphatically \(strong body language\)\# Body Language\>a speaker with restrained micro\-expressions or tension\# Micro\-expression\>a social interaction cue \(eye contact, mutual gaze\)\# Social Cue\>a neutral face or resting expression \(no strong emotion\)\# Neutral\# Audio Modality\>a voice with a warm, friendly tone \(positive prosody\)\# Positive Prosody\>a voice with a tense or harsh tone \(angry or stressed\)\# Negative Prosody\>low\-intensity prosodic variation \(subtle change in pitch\)\# Subtle Variation\>a sudden change in pitch or loudness \(surge or exclamation\)\# Transient Event\>vocal emphasis or strong stress on words \(emphatic speech\)\# Emphatic Speech\>vocal micro\-intonations like sighs, breathiness, or pauses\# Paralinguistics\>prosodic pattern of uncertainty or hesitation \(uh, um\)\# Hesitation\>a neutral, flat prosodic contour \(monotone\)\# Neutral\\endlxSVG@picture

Table 12\.Complete prompt list of semantic sentences used for query initialization on CH\-SIMS and CH\-SIMS v2 datasets\.\\lxSVG@picture

Complete Prompt List of Semantic Sentences for CH\-SIMS / CH\-SIMS v2\# Video Modality\>嘴角上扬、嘴唇张开或明显的嘴部肌肉运动\#笑容、说话、嘲讽\>眉毛皱起、紧锁或前额肌肉的剧烈收缩\# 愤怒、思考、痛苦\>眼睛睁大、眉毛上挑或眼周肌肉的扩张\# 惊讶、恐惧、强调\>面部肌肉松弛、平坦或缺乏明显的纹理变化\# 中性、发呆、冷漠\>头部的大幅度转动、点头或剧烈的晃动\# 否认、肯定、激动\>单侧嘴角抽动、眼神斜视或不对称的面部表情\# 轻蔑、不屑\>明显的手部挥舞、指点或身体姿态的前后移动\# 肢体语言、情绪强度\>极短时间内发生的快速面部抽动或微表情\# 压抑、掩饰、紧张\# Audio Modality\>音高较高、频率尖锐或上升的语调模式\# 兴奋、尖叫、质问\>音量巨大、能量爆发或带有冲击力的声音\# 愤怒、强调、惊叹\>低沉、沙哑、微弱或频率较低的声音特质\# 悲伤、疲惫、阴沉\>语速极快、连贯紧凑且没有停顿的说话方式\# 紧张、急切、激辩\>明显的长时间停顿、沉默或极慢的语速\# 犹豫、尴尬、思考\>呼吸声、叹气声、笑声或非语言的喉音\# 无奈、轻蔑、潜台词\>节奏平稳、单调且缺乏起伏的机械式发音\# 中性、陈述、冷漠\>音调突然转折、断裂或不稳定的颤音\# 哭腔、破音、极度激动\\endlxSVG@picture

MOSI

MOSEI

CH\-SIMS

CH\-SIMS v2

Video \(Focus Stream\)

![Refer to caption](https://arxiv.org/html/2607.27790v1/x18.png)
![Refer to caption](https://arxiv.org/html/2607.27790v1/x19.png)
![Refer to caption](https://arxiv.org/html/2607.27790v1/x20.png)
![Refer to caption](https://arxiv.org/html/2607.27790v1/x21.png)Video \(Ambient Stream\)

![Refer to caption](https://arxiv.org/html/2607.27790v1/x22.png)
![Refer to caption](https://arxiv.org/html/2607.27790v1/x23.png)
![Refer to caption](https://arxiv.org/html/2607.27790v1/x24.png)
![Refer to caption](https://arxiv.org/html/2607.27790v1/x25.png)Audio \(Focus Stream\)

![Refer to caption](https://arxiv.org/html/2607.27790v1/x26.png)
![Refer to caption](https://arxiv.org/html/2607.27790v1/x27.png)
![Refer to caption](https://arxiv.org/html/2607.27790v1/x28.png)
![Refer to caption](https://arxiv.org/html/2607.27790v1/x29.png)Audio \(Ambient Stream\)

![Refer to caption](https://arxiv.org/html/2607.27790v1/x30.png)
![Refer to caption](https://arxiv.org/html/2607.27790v1/x31.png)
![Refer to caption](https://arxiv.org/html/2607.27790v1/x32.png)
![Refer to caption](https://arxiv.org/html/2607.27790v1/x33.png)

Figure 7\.Visualization of feature disentanglement on the four datasets\.MOSIMOSEICH\-SIMSCH\-SIMS v2

![Refer to caption](https://arxiv.org/html/2607.27790v1/x34.png)
![Refer to caption](https://arxiv.org/html/2607.27790v1/x35.png)
![Refer to caption](https://arxiv.org/html/2607.27790v1/x36.png)
![Refer to caption](https://arxiv.org/html/2607.27790v1/x37.png)
![Refer to caption](https://arxiv.org/html/2607.27790v1/x38.png)
![Refer to caption](https://arxiv.org/html/2607.27790v1/x39.png)
![Refer to caption](https://arxiv.org/html/2607.27790v1/x40.png)
![Refer to caption](https://arxiv.org/html/2607.27790v1/x41.png)

Figure 8\.Confusion matrix and per\-class accuracy for 7\-class sentiment prediction on the four datasets\. HN: Highly Negative; N: Negative; WN: Weakly Negative; NT: Neutral; WP: Weak Positive; P: Positive; HP: Highly Positive\.

## Appendix DLimitations

Despite the demonstrated effectiveness, there exist some limitations:

1. \(1\)Interpretability: We currently rely on LLM reasoning for sentiment classification\. However, classification labels alone do not fully reveal the internal decision logic\. Future work could explore generating textual explanations to enhance interpretability and transparency\.
2. \(2\)Missing\-Modality Scenarios: Our method assumes complete modality availability\. In real\-world applications, data corruption often leads to missing signals\. Improving robustness under missing\-modality scenarios remains a critical direction for future work\.