Robust Multimodal Sentiment Analysis with Incomplete Modalities via Semantic-aware Completeness based Reconstruction

# Robust Multimodal Sentiment Analysis with Incomplete Modalities via Semantic-aware Completeness based Reconstruction
Source: [https://arxiv.org/html/2609.10950](https://arxiv.org/html/2609.10950)
Byunggill Joe11footnotemark:1Saim ShinJin Yea Jang††thanks:Corresponding author\.Affiliation:Korea Electronics Technology Institute \(KETI\)Affiliation:Seongnam, South KoreaAffiliation:\{hanjun\_c, byunggill, sishin, jinyea\.jang\}@keti\.re\.kr

###### Abstract

Recent multimodal sentiment analysis studies increasingly adopt text\-centric fusion approaches to exploit the rich sentiment information inherent in the textual modality\. However, these approaches often suffer from performance degradation during inference due to partially missing or noisy data in real\-world scenarios, especially when sentiment\-related cues are missing\. To address this issue, we introduce a new completeness estimation approach that quantifies the degree of sentiment\-relevant information preserved in incomplete data to guide the reconstruction of missing semantics\. Furthermore, we propose a training strategy that stabilizes multi\-task learning while jointly optimizing sentiment prediction and completeness estimation\. Extensive experiments and in\-depth analyses on three benchmark datasets demonstrate that the proposed approach enables more accurate semantic reconstruction, leading to more precise sentiment prediction\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2609.10950v1/motivation_camera_ready.png)Figure 1:Comparison between missing rate–based and semantic–based completeness\. The former is computed as1−r1\-r, wherer∈\[0,1\)r\\in\[0,1\)denotes the missing rate, while the latter is estimated by our proposed method\.Building on the rich sentiment\-relevant information in text[Hazarika et al\. \(2022\)](https://arxiv.org/html/2609.10950#bib.bib19);[Wei et al\. \(2023\)](https://arxiv.org/html/2609.10950#bib.bib20), recent Multimodal Sentiment Analysis \(MSA\) studies have increasingly explored text\-centric approaches[Han et al\. \(2021\)](https://arxiv.org/html/2609.10950#bib.bib7);[Wang et al\. \(2023a\)](https://arxiv.org/html/2609.10950#bib.bib8);[Wang et al\. \(2023b\)](https://arxiv.org/html/2609.10950#bib.bib10);[Zhang et al\. \(2023\)](https://arxiv.org/html/2609.10950#bib.bib9)that treat text as the dominant modality while considering audio and vision modalities as auxiliary sources\. Although these approaches have shown strong performance with complete data, they suffer substantial performance degradation when modalities are partially missing or noisy, particularly when the textual modality is incomplete\.

To address this issue, several reconstruction\-based approaches have been proposed[Yuan et al\. \(2021\)](https://arxiv.org/html/2609.10950#bib.bib11);[Zhang et al\. \(2024a\)](https://arxiv.org/html/2609.10950#bib.bib12);[Zhu et al\. \(2025\)](https://arxiv.org/html/2609.10950#bib.bib22)\. In particular,[Zhang et al\. \(2024a\)](https://arxiv.org/html/2609.10950#bib.bib12)proposed a missing rate\-based completeness estimation method to utilize the missing rate as a reconstruction weight, assuming that a higher missing rate indicates lower completeness, and vice versa\. However, we argue that relying solely on the missing rate may fail to accurately capture semantic informativeness\.

Figure[1](https://arxiv.org/html/2609.10950#S1.F1)highlights this limitation through two contrasting cases\. In Case 1, a missing key sentiment cue \[‘love’\] leads to substantial semantic loss despite a low missing rate, causing an overestimation of missing rate\-based completeness\. On the other hand, in Case 2, it is underestimated even though the sentiment meaning remains intact\. Such inaccurate completeness may lead to unreliable reconstruction, which highlights the necessity of a semantic\-aware completeness approach that determines completeness based on whether key sentiment cues are missing\.

Motivated by the above observations, we propose a semantic\-aware completeness estimation approach for more reliable reconstruction of missing textual semantics\. Specifically, we first estimate the completeness of incomplete text through a semantic completeness estimator\. To supervise the estimator, we introduce Target Probability\-based Semantic Completeness \(TPSC\), which is a pseudo\-labeling strategy that generates completeness labels\. Meanwhile, the Importance\-aware Proxy Feature Generator \(IPFG\) generates proxy features from the auxiliary modalities by adaptively weighting their contributions for text reconstruction\. The estimated completeness is then used as a weighting factor to adaptively combine proxy features and incomplete text, producing reconstructed textual representations\. Furthermore, we propose an Alternating Optimization Strategy \(AOS\) to mitigate gradient conflicts that stem from jointly optimizing the sentiment prediction and completeness estimation tasks on a shared encoder\. The contributions of this paper are summarized as follows:

- •To the best of our knowledge, we are the first to introduce a semantic\-aware completeness estimation approach that quantifies the degree of semantic preservation in incomplete data\.
- •We propose an optimization strategy that mitigates gradient conflicts in hierarchical multi\-task learning, thereby stabilizing training between the completeness estimation and sentiment prediction tasks\.
- •Extensive experiments on three benchmark datasets with varying missing rates show that our proposed method consistently outperforms 12 competitive baselines, demonstrating the effectiveness of the proposed completeness label in real\-world data scenarios\.

## 2Related Work

Recent studies in MSA have explored various multimodal fusion strategies to integrate textual, acoustic, and visual information[Zadeh et al\. \(2017\)](https://arxiv.org/html/2609.10950#bib.bib1);[Liu et al\. \(2018\)](https://arxiv.org/html/2609.10950#bib.bib2);[Tsai et al\. \(2019\)](https://arxiv.org/html/2609.10950#bib.bib3);[Hazarika et al\. \(2020\)](https://arxiv.org/html/2609.10950#bib.bib4);[Yu et al\. \(2021\)](https://arxiv.org/html/2609.10950#bib.bib5);[Sun et al\. \(2022\)](https://arxiv.org/html/2609.10950#bib.bib6)\. These approaches aim to capture cross\-modal interactions and leverage complementary information across modalities to improve sentiment prediction\. Another line of work focuses on text\-centric approaches[Han et al\. \(2021\)](https://arxiv.org/html/2609.10950#bib.bib7);[Wang et al\. \(2023a\)](https://arxiv.org/html/2609.10950#bib.bib8);[Wang et al\. \(2023b\)](https://arxiv.org/html/2609.10950#bib.bib10);[Zhang et al\. \(2023\)](https://arxiv.org/html/2609.10950#bib.bib9), which treat text as the dominant modality and utilize audio and visual signals as auxiliary sources to enhance textual representations\. However, most existing methods assume that all modalities are fully available during inference, which limits their robustness in real\-world scenarios\. This issue becomes particularly critical for text\-centric approaches, as the loss of important semantic information in text directly undermines the effectiveness of text\-centric fusion\.

To mitigate the above issue, recent studies have explored reconstruction\-based approaches that aim to restore semantic information from missing modalities[Yuan et al\. \(2021\)](https://arxiv.org/html/2609.10950#bib.bib11);[Lin and Hu \(2023\)](https://arxiv.org/html/2609.10950#bib.bib27);[Zhang et al\. \(2024a\)](https://arxiv.org/html/2609.10950#bib.bib12);[Zhu et al\. \(2025\)](https://arxiv.org/html/2609.10950#bib.bib22);[Yang et al\. \(2025\)](https://arxiv.org/html/2609.10950#bib.bib26);[Shi et al\. \(2025\)](https://arxiv.org/html/2609.10950#bib.bib28);[Li et al\. \(2025\)](https://arxiv.org/html/2609.10950#bib.bib29);[Lin et al\. \(2025\)](https://arxiv.org/html/2609.10950#bib.bib32)\. In particular, LNLN[Zhang et al\. \(2024a\)](https://arxiv.org/html/2609.10950#bib.bib12)introduces a reconstruction method that restores missing textual semantics using proxy features derived from audio and vision inputs, where the reconstruction weight is estimated from the missing rate of the incomplete text\. P\-RMF[Zhu et al\. \(2025\)](https://arxiv.org/html/2609.10950#bib.bib22)introduces a proxy\-driven strategy that dynamically reconstructs incomplete multimodal inputs through cross\-modal feature generation and adaptive fusion, further improving robustness under uncertain missing conditions\. Additionally, TF\-Mamba[Li et al\. \(2025\)](https://arxiv.org/html/2609.10950#bib.bib29)proposes a text\-enhanced Mamba\-based framework that reconstructs missing textual semantics by aligning and enhancing auxiliary modalities through a text\-aware modality enhancement module while modeling multimodal dependencies via efficient Mamba\-based fusion\. While these methods improve robustness to missing modalities, they do not explicitly account for the semantic information loss in incomplete text\.

![Refer to caption](https://arxiv.org/html/2609.10950v1/tcmr_overview_camera_ready.png)Figure 2:Overview of the proposed framework\. Models in the light purple boxes are used only during pretraining or pseudo\-labeling and are not part of the TCMR training pipeline\.
## 3Proposed Method

### 3\.1Problem Definition

Given multimodal inputs from text \(t\), audio \(a\), and vision \(v\) derived from the same utterance, the MSA task aims to predict a continuous sentiment scorey∈ℝy\\in\\mathbb\{R\}\.

### 3\.2Framework Overview

In this work, we introduce aTextCompleteness\-basedMissingReconstruction \(TCMR\) framework that estimates textual completeness from incomplete text and reconstructs the missing semantic information by adaptively weighting proxy and incomplete text features based on the estimated completeness\.

The overall workflow of TCMR is illustrated in[Figure 2](https://arxiv.org/html/2609.10950#S2.F2)\. First, each modality input is randomly masked and then embedded into feature representations via modality\-specific encoders \(1\)\. Next, a completeness estimator predicts how much semantic information remains in incomplete text and reconstructs missing semantics using proxy features derived from auxiliary modalities, weighted by the predicted completeness \(2–4\)\. Lastly, a multimodal fusion module performs text\-centric fusion to produce the final sentiment score \(5\)\. Note that the multimodal fusion module is adopted from LNLN, as designing a new fusion module is not the main objective of this work \(see Appendix[A\.4](https://arxiv.org/html/2609.10950#A1.SS4)\)\.

### 3\.3Input Processing

Raw multimodal inputs are processed into high\-level embeddings \(Figure[2](https://arxiv.org/html/2609.10950#S2.F2)1\)\. First, each complete multimodal inputImcI\_\{m\}^\{c\}for modalitym∈\{v,a,t\}m\\in\\\{v,a,t\\\}, is processed by modality\-specific feature extractors: BERT for text, Librosa for audio, and OpenFace for vision[Devlin et al\. \(2019\)](https://arxiv.org/html/2609.10950#bib.bib15);[McFee et al\. \(2015\)](https://arxiv.org/html/2609.10950#bib.bib14);[Baltrušaitis et al\. \(2016\)](https://arxiv.org/html/2609.10950#bib.bib13)\.

Following prior works[Zhang et al\. \(2024a\)](https://arxiv.org/html/2609.10950#bib.bib12);[Zhu et al\. \(2025\)](https://arxiv.org/html/2609.10950#bib.bib22), we adopt a random missing scenario to generate incomplete featuresXm∈ℝTm×dmX\_\{m\}\\in\\mathbb\{R\}^\{T\_\{m\}\\times d\_\{m\}\}, whereTmT\_\{m\}anddmd\_\{m\}denote the sequence length and feature dimension\. For text, a proportion of tokens inItcI\_\{t\}^\{c\}are randomly replaced with\[UNK\]according to a missing rate sampled from\[0,1\)\[0,1\)\. For audio and vision features, a proportion of the sequence is replaced with zero vectors\. Subsequently, the incomplete features are processed by modality\-specific Transformer encoders[Vaswani et al\. \(2017\)](https://arxiv.org/html/2609.10950#bib.bib16)to obtain the high\-level embeddingsHm∈ℝT×dH\_\{m\}\\in\\mathbb\{R\}^\{T\\times d\}:

Hm=Transformer​\(Xm,θmtrans\)H\_\{m\}=\\text\{Transformer\}\(X\_\{m\};\\theta^\{\\text\{trans\}\}\_\{m\}\)\(1\)

### 3\.4Missing Semantic Reconstruction

In this section, we elaborate on the completeness–guided semantic reconstruction pipeline of TCMR \(Figure[2](https://arxiv.org/html/2609.10950#S2.F2)2–4\), which is the core contribution of our work\.

#### Text Completeness Estimation

To estimate the completeness of incomplete text, we design a semantic completeness estimatorCompNet, which consists of fully connected layers followed by a sigmoid activation \(Figure[2](https://arxiv.org/html/2609.10950#S2.F2)2–1\)\.

Given the BERT\[CLS\]embeddingSt∈ℝdS\_\{t\}\\in\\mathbb\{R\}^\{d\},CompNetestimates a completeness weightw^∈\[0,1\]\\hat\{w\}\\in\[0,1\]:

w^=CompNet​\(St,θcomp\),\\hat\{w\}=\\text\{CompNet\}\(S\_\{t\};\\theta^\{\\mathrm\{comp\}\}\),\(2\)The model is trained to minimize mean squared error between the predicted scorew^\\hat\{w\}and the completeness labelww:

ℒcomp=1N​∑i=1N‖w^\(i\)−w\(i\)‖2,\\mathcal\{L\}\_\{\\text\{comp\}\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\left\\\|\\hat\{w\}^\{\(i\)\}\-w^\{\(i\)\}\\right\\\|^\{2\},\(3\)whereiidenotes the data sample index\.

#### Pseudo\-Label Generation

In our study, we define textual completeness as the extent to which sentiment\-related semantic information is preserved in incomplete text\. Following this definition, we introduce a pseudo\-labeling strategy to generate completeness labelswwfor trainingCompNet\(Figure[2](https://arxiv.org/html/2609.10950#S2.F2)2–2\)\.

To quantify the completeness, we extend the concept of True Class Probability \(TCP\)[Corbière et al\. \(2019\)](https://arxiv.org/html/2609.10950#bib.bib21), defined as the probability assigned to the target class\. Our hypothesis is that if a classifier trained on complete text assigns a high TCP to a given complete text, the TCP for the corresponding incomplete text will remain high as long as sentiment\-relevant tokens are preserved\. Conversely, the TCP may substantially decrease if key sentiment tokens are missing\.

Based on this insight, we design a sentiment polarity classifier that takes only the text modality as input\. First, the classifier is trained on the complete textItcI\_\{t\}^\{c\}:

y^sp\\displaystyle\\hat\{y\}\_\{\\text\{sp\}\}=Classifier​\(BERTcls​\(Itc\),θcls\)\\displaystyle=\\text\{Classifier\}\(\\text\{BERT\}^\{\\text\{cls\}\}\(I\_\{t\}^\{c\}\);\\theta^\{\\text\{cls\}\}\)\(4\)ℒce\\displaystyle\\mathcal\{L\}\_\{\\text\{ce\}\}=−∑c=1Cys​p\(c\)logy^s​p\(c\)\\displaystyle=\-\\sum\_\{c=1\}^\{C\}y\_\{sp\}^\{\(c\)\}\\log\\hat\{y\}\_\{sp\}^\{\(c\)\}\(5\)whereys​py\_\{sp\}denotes the polarity class obtained by discretizing the continuous sentiment score according to predefined thresholds\. After training, we perform inference on incomplete text using the pretrained classifier, and the probability assigned to the ground\-truth polarity classys​py\_\{sp\}is used as the pseudo completeness labelw∈\[0,1\]w\\in\[0,1\]:

w\(i\)≜p⁡\(ysp\(i\)∣BERTcls​\(It\);θprecls\),w^\{\(i\)\}\\triangleq p\\\!\\left\(y^\{\(i\)\}\_\{\\text\{sp\}\}\\mid\\text\{BERT\}^\{\\text\{cls\}\}\(I\_\{t\}\);\\theta^\{\\text\{cls\}\}\_\{\\text\{pre\}\}\\right\),\(6\)whereθprecls\\theta^\{\\text\{cls\}\}\_\{\\text\{pre\}\}are the pretrained classifier parameters\. The definition of sentiment polarity classes and the choice of the optimal number of classes are provided in Appendix[B\.1](https://arxiv.org/html/2609.10950#A2.SS1)and Appendix[B\.2](https://arxiv.org/html/2609.10950#A2.SS2), respectively\.

#### Proxy Feature Generation

Prior work[Zhang et al\. \(2024a\)](https://arxiv.org/html/2609.10950#bib.bib12)generates proxy features without considering the relative importance of auxiliary modalities for reconstruction\. However, the contributions of each modality to the reconstruction can vary under random missing conditions\.

Therefore, we design an Importance\-aware Proxy Feature Generator \(IPFG\), which consists of Transformer\-based modality\-specific generators and an MLP\-based gating network with a sigmoid:

Hm′p\\displaystyle H\_\{m^\{\\prime\}\}^\{p\}=Gen⁡\(Hm′,θm′gen\),\\displaystyle=\\mathrm\{Gen\}\(H\_\{m^\{\\prime\}\};\\,\\theta\_\{m^\{\\prime\}\}^\{\\mathrm\{gen\}\}\),\(7\)ρ\\displaystyle\\rho=Gate⁡\(\[Hap,Hvp\],θgate\),\\displaystyle=\\mathrm\{Gate\}\(\[H\_\{a\}^\{p\},H^\{p\}\_\{v\}\];\\,\\theta^\{\\mathrm\{gate\}\}\),\(8\)H¯t\\displaystyle\\bar\{H\}\_\{t\}=ρ⊙Hap\+\(1−ρ\)⊙Hvp\.\\displaystyle=\\rho\\odot H\_\{a\}^\{p\}\+\(1\-\\rho\)\\odot H\_\{v\}^\{p\}\.\(9\)wherem′∈\{a,v\}m^\{\\prime\}\\in\\\{a,v\\\}denotes the auxiliary modalities,⊙\\odotdenotes element\-wise multiplication, andρ∈\[0,1\]\\rho\\in\[0,1\]is a gating weight that adaptively balances the contributions of the auxiliary modalities\.

#### Semantic Reconstruction

The reconstructed text representationH^t∈ℝT×d\\hat\{H\}\_\{t\}\\in\\mathbb\{R\}^\{T\\times d\}is generated by integrating the incomplete text representationHtH\_\{t\}with the generated proxy featureH¯t\\bar\{H\}\_\{t\}, guided by the predicted completeness weightw^\\hat\{w\}fromCompNet:

H^t=w^​Ht\+\(1−w^\)​H¯t\.\\hat\{H\}\_\{t\}=\\hat\{w\}H\_\{t\}\+\(1\-\\hat\{w\}\)\\bar\{H\}\_\{t\}\.\(10\)To encourage the reconstructed features to be similar to the corresponding complete features, we minimize the mean squared error lossℒr​e​ct\\mathcal\{L\}^\{t\}\_\{rec\}\. For the auxiliary modalities, following prior work[Zhang et al\. \(2024a\)](https://arxiv.org/html/2609.10950#bib.bib12), we adopt a self\-reconstruction method that takesHm′H\_\{m^\{\\prime\}\}as input and predicts the corresponding complete representationH^m′\\hat\{H\}\_\{m^\{\\prime\}\}\. The reconstruction losses are defined as:

ℒrecm=1N​∑i=1N‖H^m\(i\)−Hmc,\(i\)‖2,\\mathcal\{L\}\_\{\\text\{rec\}\}^\{m\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\left\\\|\\hat\{H\}\_\{m\}^\{\(i\)\}\-H\_\{m\}^\{c,\(i\)\}\\right\\\|^\{2\},\(11\)whereHmc,\(i\)H\_\{m\}^\{c,\(i\)\}denotes the complete feature of modalitymmobtained from a model trained with the corresponding complete input\.

### 3\.5Training Objective

TCMR predicts the final sentiment scorey^\\hat\{y\}through the text\-centric multimodal fusion module\. The model is trained using the following loss:

ℒtask=1N​∑i=1N‖y^\(i\)−y\(i\)‖2\.\\mathcal\{L\}\_\{\\text\{task\}\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\left\\\|\\hat\{y\}^\{\(i\)\}\-y^\{\(i\)\}\\right\\\|^\{2\}\.\(12\)The overall training objective is defined as:

ℒtotal=α​ℒcomp\+β​ℒrect\+γ​ℒrecm′\+σ​ℒtask\.\\mathcal\{L\}\_\{\\text\{total\}\}=\\alpha\\mathcal\{L\}\_\{\\text\{comp\}\}\+\\beta\\mathcal\{L\}^\{t\}\_\{\\text\{rec\}\}\+\\gamma\\mathcal\{L\}\_\{\\text\{rec\}\}^\{m^\{\\prime\}\}\+\\sigma\\mathcal\{L\}\_\{\\text\{task\}\}\.\(13\)whereα\\alpha,β\\beta,γ\\gamma, andσ\\sigmaare hyperparameters\.

Algorithm 1Alternating Optimization StrategyInput:dataset𝒟\\mathcal\{D\}, epochsEE, batch sizeBB, weightsα,β,γ,σ\\alpha,\\beta,\\gamma,\\sigma\. Params:Θcomp=θcomp∪θbert\\Theta^\{\\text\{comp\}\}=\\theta^\{\\text\{comp\}\}\\cup\\theta^\{\\text\{bert\}\},Θother\\Theta^\{\\text\{other\}\}for the remaining modules includingθbert\\theta^\{\\text\{bert\}\}\. Opt:Optcomp\\mathrm\{Opt\}\_\{\\text\{comp\}\},Optother\\mathrm\{Opt\}\_\{\\text\{other\}\}\.

1:for

e=1e=1to

EEdo

2:Phase 1:optimize

Θcomp\\Theta^\{\\text\{comp\}\}to minimize

ℒcomp\\mathcal\{L\}\_\{\\text\{comp\}\}
3:formini\-batch

ℬ⊂𝒟\\mathcal\{B\}\\subset\\mathcal\{D\},

\|ℬ\|=B\|\\mathcal\{B\}\|=Bdo

4:

ℒcomp←LossComp⁡\(ℬ,Θcomp\)\\mathcal\{L\}\_\{\\text\{comp\}\}\\leftarrow\\mathrm\{LossComp\}\(\\mathcal\{B\};\\,\\Theta^\{\\text\{comp\}\}\)
5:

ℒcomp←α​ℒcomp\\mathcal\{L\}\_\{\\text\{comp\}\}\\leftarrow\\alpha\\,\\mathcal\{L\}\_\{\\text\{comp\}\}
6:

Optcomp\.step⁡\(∇Θcompℒcomp\)\\mathrm\{Opt\}\_\{\\text\{comp\}\}\.\\mathrm\{step\}\\\!\\left\(\\nabla\_\{\\Theta^\{\\text\{comp\}\}\}\\mathcal\{L\}\_\{\\text\{comp\}\}\\right\)
7:endfor

8:Phase 2:optimize

Θother\\Theta^\{\\text\{other\}\}to minimize

ℒother\\mathcal\{L\}\_\{\\text\{other\}\}
9:formini\-batch

ℬ⊂𝒟\\mathcal\{B\}\\subset\\mathcal\{D\},

\|ℬ\|=B\|\\mathcal\{B\}\|=Bdo

10:

ℒtask←LossTask⁡\(ℬ,Θother\)\\mathcal\{L\}\_\{\\text\{task\}\}\\leftarrow\\mathrm\{LossTask\}\(\\mathcal\{B\};\\,\\Theta^\{\\text\{other\}\}\)
11:

ℒrect←LossRecText⁡\(ℬ,Θother\)\\mathcal\{L\}\_\{\\text\{rec\}\}^\{t\}\\leftarrow\\mathrm\{LossRecText\}\(\\mathcal\{B\};\\,\\Theta^\{\\text\{other\}\}\)
12:

ℒrecm′←LossRecAux⁡\(ℬ,Θother\)\\mathcal\{L\}\_\{\\text\{rec\}\}^\{m^\{\\prime\}\}\\leftarrow\\mathrm\{LossRecAux\}\(\\mathcal\{B\};\\,\\Theta^\{\\text\{other\}\}\)
13:

ℒother←β​ℒrect\+γ​ℒrecm′\+σ​ℒtask\\mathcal\{L\}\_\{\\text\{other\}\}\\leftarrow\\beta\\,\\mathcal\{L\}\_\{\\text\{rec\}\}^\{t\}\+\\gamma\\,\\mathcal\{L\}\_\{\\text\{rec\}\}^\{m^\{\\prime\}\}\+\\sigma\\,\\mathcal\{L\}\_\{\\text\{task\}\}
14:

Optother\.step⁡\(∇Θotherℒother\)\\mathrm\{Opt\}\_\{\\text\{other\}\}\.\\mathrm\{step\}\\\!\\left\(\\nabla\_\{\\Theta^\{\\text\{other\}\}\}\\mathcal\{L\}\_\{\\text\{other\}\}\\right\)
15:endfor

16:endfor

### 3\.6Alternating Optimization Strategy

We found that naively minimizingℒtotal\\mathcal\{L\}\_\{\\text\{total\}\}in an end\-to\-end manner often leads to unstable training, whereℒcomp\\mathcal\{L\}\_\{\\text\{comp\}\}is effectively ignored andCompNetremains stuck at a poor local minimum\. We attribute this behavior to gradient conflict[Yu et al\. \(2020a\)](https://arxiv.org/html/2609.10950#bib.bib24)\. This phenomenon commonly arises in multi\-task learning, where the gradients of a dominant task suppress the optimization of relatively weaker tasks\. In our case,ℒtask\\mathcal\{L\}\_\{\\text\{task\}\}can also be minimized through shortcut solutions that bypass learning completeness estimation, causing the optimization ofℒcomp\\mathcal\{L\}\_\{\\text\{comp\}\}to be dominated during training\.

To address this issue, we introduce a simple yet effective training strategy called Alternating Optimization Strategy \(AOS\)\. We first partition the total objectiveℒtotal\\mathcal\{L\}\_\{\\text\{total\}\}into two groups according to their respective roles:

ℒcomp,ℒother=β​ℒrect\+γ​ℒrecm′\+σ​ℒtask\.\\mathcal\{L\}\_\{\\text\{comp\}\},\\quad\\mathcal\{L\}\_\{\\text\{other\}\}=\\beta\\,\\mathcal\{L\}\_\{\\text\{rec\}\}^\{t\}\+\\gamma\\,\\mathcal\{L\}\_\{\\text\{rec\}\}^\{m^\{\\prime\}\}\+\\sigma\\,\\mathcal\{L\}\_\{\\text\{task\}\}\.\(14\)
As shown in Algorithm[1](https://arxiv.org/html/2609.10950#alg1), AOS performs a two\-phase optimization within each training epoch\. In Phase 1, we focus on completeness learning by minimizingℒcomp\\mathcal\{L\}\_\{\\text\{comp\}\}, while fixing the remaining parameters and optimizing onlyΘcomp\\Theta^\{\\text\{comp\}\}\. In Phase 2, we fix the learnedθcomp\\theta^\{\\text\{comp\}\}and optimizeΘother\\Theta^\{\\text\{other\}\}by minimizingℒother\\mathcal\{L\}\_\{\\text\{other\}\}\. At this stage, the model focuses on reconstruction and sentiment prediction based on the completeness estimation learned in Phase 1\. By alternately optimizing the two objectives, AOS stabilizes the training ofCompNetand enables the remaining modules to be effectively trained according to their respective objectives\. Figure[3](https://arxiv.org/html/2609.10950#S4.F3)illustrates this alternating optimization process, and further experimental analysis of gradient conflicts is provided in Appendix[C](https://arxiv.org/html/2609.10950#A3)\.

## 4Experiments

### 4\.1Experimental Setup

#### Dataset and Evaluation Metrics

We conduct experiments on three MSA benchmark datasets: MOSI[Zadeh et al\. \(2016\)](https://arxiv.org/html/2609.10950#bib.bib17), MOSEI[Bagher Zadeh et al\. \(2018\)](https://arxiv.org/html/2609.10950#bib.bib18), and SIMS[Yu et al\. \(2020b\)](https://arxiv.org/html/2609.10950#bib.bib30)\. In MOSI and MOSEI, sentiment scores are annotated with continuous values ranging from−3\-3\(strongly negative\) to\+3\+3\(strongly positive\), while SIMS provides sentiment labels on a scale from−1\-1\(negative\) to\+1\+1\(positive\)\. Model performance is evaluated using Acc\-2, Acc\-5, Acc\-7, F1, MAE, and Corr on MOSI and MOSEI, and Acc\-2, Acc\-3, Acc\-5, F1, MAE, and Corr on SIMS\. Dataset statistics and metric definitions are provided in Appendix[A\.1](https://arxiv.org/html/2609.10950#A1.SS1)and Appendix[A\.3](https://arxiv.org/html/2609.10950#A1.SS3), respectively\.

![Refer to caption](https://arxiv.org/html/2609.10950v1/aos_vis.png)Figure 3:Illustration of the AOS optimization process\.
#### Training and Evaluation Settings

Following previous studies[Zhang et al\. \(2024a\)](https://arxiv.org/html/2609.10950#bib.bib12);[Zhu et al\. \(2025\)](https://arxiv.org/html/2609.10950#bib.bib22), we adopt apartial random missingscenario\. During training, each modality is randomly erased according to a missing raterrsampled from a uniform distribution over the range\[0,1\.0\)\[0,1\.0\)\. For each of the three seeds, the best model is selected based on its performance at a missing rate ofr=0\.5r=0\.5on the validation set\. For testing, we conduct experiments across missing rates ranging from00to0\.90\.9with an increment of0\.10\.1, resulting in ten evaluation settings for each seed\.

Table 1:Comparison of the overall performance on MOSI and MOSEI datasets\.MethodMOSIMOSEIAcc\-7Acc\-5Non0 Acc / F1Has0 Acc / F1MAECorrAcc\-7Acc\-5Non0 Acc / F1Has0 Acc / F1MAECorrMISA29\.0331\.6168\.77 / 68\.6767\.94 / 67\.721\.163747\.5743\.8944\.4372\.22 / 68\.2174\.16 / 71\.240\.734643\.57Self\-MM30\.3833\.6968\.83 / 68\.6368\.65 / 69\.341\.184347\.2546\.4547\.3673\.02 / 71\.4372\.66 / 71\.820\.681853\.68MMIM30\.6734\.3170\.19 / 69\.8769\.59 / 69\.151\.164949\.0144\.8945\.4274\.29 / 73\.1973\.46 / 73\.190\.703252\.75CENet29\.4932\.9069\.88 / 69\.9469\.43 / 69\.391\.180148\.2447\.3648\.2477\.01 / 77\.2675\.96 / 76\.230\.662258\.24TETFN29\.8632\.5670\.62 / 70\.6769\.85 / 69\.791\.132749\.1646\.7847\.8377\.87 / 77\.4576\.11 / 76\.370\.674158\.31TFR\-Net28\.2230\.3170\.88 / 70\.7370\.18 / 69\.931\.145450\.0546\.0846\.4775\.46 / 74\.1074\.18 / 73\.610\.678456\.24ALMT29\.5732\.0571\.51 / 71\.4870\.52 / 70\.391\.167147\.5746\.6647\.3776\.83 / 76\.4174\.47 / 74\.840\.674956\.45LNLN31\.3634\.4370\.00 / 69\.9969\.51 / 69\.411\.145048\.0846\.3647\.1477\.79 / 77\.2776\.43 / 76\.580\.669858\.17P\-RMF28\.9131\.2469\.91 / 69\.5769\.28 / 68\.841\.130250\.0747\.0147\.8278\.07 / 77\.7677\.18 / 76\.910\.666758\.45TF\-Mamba28\.8531\.4671\.05 / 71\.1170\.73 / 70\.691\.122851\.9745\.1146\.0976\.11 / 76\.1772\.54 / 73\.490\.689157\.51TCMR\-MRSC30\.3933\.3670\.84 / 70\.9569\.81 / 69\.821\.106150\.8646\.6147\.4677\.65 / 76\.7476\.91 / 76\.640\.667358\.46TCMR\-LDSC31\.6335\.3870\.61 / 70\.5569\.83 / 69\.581\.132249\.0747\.1347\.8777\.52 / 76\.9776\.16 / 76\.290\.664158\.51TCMR\-TPSC32\.8036\.7372\.01/71\.6770\.93/70\.491\.072152\.2847\.2748\.1677\.99 / 76\.9977\.59/77\.210\.661458\.63

### 4\.2Main Experimental Results

#### Baseline Models

We select baseline models with publicly available implementations to ensure reproducibility\. For a fair comparison, we reproduce all baseline models using publicly available implementations\. For MISA, Self\-MM, MMIM, CENet, TETFN, TFR\-Net, and ALMT, we use the MMSA implementations[Mao et al\. \(2022\)](https://arxiv.org/html/2609.10950#bib.bib23)\. For LNLN111[https://github\.com/Haoyu\-ha/LNLN](https://github.com/Haoyu-ha/LNLN), P\-RMF222[https://github\.com/aoqzhu/P\-RMF](https://github.com/aoqzhu/P-RMF), and TF\-Mamba333[https://github\.com/codemous/TF\-Mamba](https://github.com/codemous/TF-Mamba), we use their official repositories\. In addition, we implement three variants of TCMR with different completeness labeling strategies to analyze their effectiveness:

- •TCMR\-TPSC: estimates completeness usingTarget Probability\-based Semantic Completeness\(TPSC\), as defined in Section[3\.4](https://arxiv.org/html/2609.10950#S3.SS4)\. Note that unless otherwise specified, TCMR refers to TCMR\-TPSC for simplicity\.
- •TCMR\-MRSC: estimates completeness usingMissing Rate\-based Semantic Completeness\(MRSC\), which measures completeness according to the missing rate of text\.
- •TCMR\-LDSC: To better analyze the effectiveness of TPSC, we designLexicon\-Derived Semantic Completeness\(LDSC\)\. LDSC generates completeness labels based on sentiment scores of individual tokens obtained from SentiWordNet[Baccianella et al\. \(2010\)](https://arxiv.org/html/2609.10950#bib.bib31)\. Specifically, we first compute the sentiment strength of the complete text by accumulating the sentiment scores of all tokens\. Then, the same procedure is applied to the incomplete text\. As a result, the completeness label is defined as the ratio between the two strengths\. Further implementation details are provided in Appendix[D\.1](https://arxiv.org/html/2609.10950#A4.SS1)\.

MethodAcc\-5Acc\-3Acc\-2F1MAECorrMISA32\.9756\.0571\.2566\.950\.57310\.319Self\-MM33\.5755\.8269\.0268\.380\.52430\.363MMIM30\.3355\.0469\.4666\.740\.56290\.301CENet21\.6153\.4869\.2356\.970\.63040\.014TETFN34\.9156\.0170\.8569\.170\.54390\.342TFR\-Net27\.2154\.2669\.3756\.820\.62580\.091ALMT30\.9953\.2370\.6567\.340\.54450\.311LNLN32\.1756\.7771\.5169\.210\.54230\.345P\-RMF32\.7257\.3070\.5870\.360\.54320\.361TF\-Mamba33\.0555\.1869\.8869\.240\.52350\.359TCMR\-MRSC33\.2755\.4670\.0368\.760\.54460\.341TCMR\-TPSC31\.1557\.4572\.8168\.880\.52120\.379

Table 2:Comparison of the overall performance on the SIMS dataset\. Note: TCMR\-LDSC is excluded since it is based on an English sentiment lexicon\.Figure 4:Performance curves under various missing rates\. \(a\)–\(c\): Accuracy on MOSI \(Acc\-7\), MOSEI \(Has0 Acc\), and SIMS \(Acc\-2\)\. \(d\)–\(f\): MAE on MOSI, MOSEI, and SIMS\.
#### Overall Performance

Tables[1](https://arxiv.org/html/2609.10950#S4.T1)–[2](https://arxiv.org/html/2609.10950#S4.T2)present the evaluation results of various models on three MSA benchmark datasets\. On the MOSI dataset, TCMR\-TPSC achieves state\-of\-the\-art performance across all evaluation metrics\. In particular, TCMR\-TPSC improves Acc\-5 by 6\.68% and reduces MAE by 6\.37% compared to LNLN\. Among theTCMR\-variants, TCMR\-TPSC consistently achieves the best performance across all datasets\. This is because TCMR\-MRSC struggles to capture the actual semantic loss from the text, as it relies solely on the missing rate\. TCMR\-LDSC, on the other hand, estimates completeness based on sentiment scores assigned to individual tokens, which does not consider contextual semantics and can lead to inaccurate completeness labels\. These results further highlight the importance of semantic\-aware completeness estimation\. Further analyses with competitive baselines are provided in Appendix[E](https://arxiv.org/html/2609.10950#A5)\.

Table 3:A comprehensive ablation study of the proposed TCMR framework on the MOSI and MOSEI datasets\.MethodMOSIMOSEIAcc\-7Acc\-5Non0 Acc / F1Has0 Acc / F1MAECorrAcc\-7Acc\-5Non0 Acc / F1Has0 Acc / F1MAECorrw/o AOSCompNet\-first27\.6730\.6566\.88 / 66\.8466\.45 / 66\.291\.167143\.0643\.3343\.4865\.37 / 64\.0969\.14 / 65\.160\.782732\.61End2End29\.6832\.0768\.15 / 68\.0467\.58 / 67\.361\.184949\.1546\.8247\.7577\.97 /77\.4376\.27 / 76\.440\.661958\.71CompNet\-later32\.2436\.1771\.16 / 71\.2270\.41 / 70\.371\.094450\.9143\.1143\.8673\.75 / 71\.8374\.58 / 73\.470\.754746\.85w/o IPFG31\.3134\.8169\.02 / 68\.9168\.65 / 68\.421\.167445\.8946\.0547\.0675\.92 / 75\.8072\.71 / 73\.450\.681657\.22TCMR \(Full\)32\.8036\.7372\.01 / 71\.6770\.93 / 70\.491\.072152\.2847\.2748\.1677\.99/ 76\.9977\.59 / 77\.210\.661458\.63

On the MOSEI and SIMS datasets, TCMR\-TPSC consistently shows strong performance across most evaluation metrics\. In particular, on MOSEI, it achieves the best results on Has0 Acc / F1, MAE, and Corr, while remaining competitive on Acc\-7, Acc\-5, and Non0 Acc / F1\. On SIMS, it achieves the best results on Acc\-3, Acc\-2, MAE, and Corr\. While TCMR\-TPSC shows slightly lower performance on a few classification metrics compared to CENet and P\-RMF, this can be attributed to the discretization involved in computing such metrics\. For instance, Acc\-7 converts continuous sentiment predictions into discrete classes via rounding\. When a prediction lies close to a rounding boundary, it may fall into a different class than the ground truth, leading to misclassification despite a lower regression error\. For this reason, regression metrics more accurately reflect the precision of sentiment prediction\. A more detailed analysis is provided in Appendix[E\.1](https://arxiv.org/html/2609.10950#A5.SS1)\.

To provide a comprehensive comparison with reconstruction\-based approaches, we plot performance changes across missing rates in Figure[4](https://arxiv.org/html/2609.10950#S4.F4)\. As shown in the figure, TCMR\-TPSC consistently outperforms baseline methods, even as the missing rate increases\. These results support our motivation that accurately estimating semantic information loss in incomplete text enables effective reconstruction of missing semantics, thereby leading to improved overall performance\.

Table 4:A comprehensive ablation study of the proposed TCMR framework on the SIMS dataset\.MethodAcc\-5Acc\-3Acc\-2F1MAECorrw/o AOSCompNet\-first22\.9253\.7969\.8260\.810\.57640\.171End2End29\.2653\.9268\.5767\.710\.53740\.321CompNet\-later29\.7155\.4970\.0468\.480\.54160\.339w/o IPFG30\.1555\.5770\.0966\.930\.54510\.336TCMR \(Full\)31\.1557\.4572\.8168\.880\.52120\.379

## 5Ablation Study

As shown in Table[3](https://arxiv.org/html/2609.10950#S4.T3)and Table[4](https://arxiv.org/html/2609.10950#S4.T4), we conduct an ablation study on the MOSI, MOSEI, and SIMS datasets to analyze the contribution of each component\. In the w/o AOS setting, we design several alternative multi\-task optimization strategies\. InCompNet\-first, we first optimizeΘcomp\\Theta^\{\\text\{comp\}\}, and then optimize the remaining modules while freezing both the shared BERT encoder and the trainedCompNet\. In contrast,CompNet\-laterfirst optimizesΘother\\Theta^\{\\text\{other\}\}using the completeness labelww, and then optimizesθcomp\\theta^\{\\text\{comp\}\}while freezing the shared BERT encoder\.End2Endjointly optimizes all model parameters without any staged optimization\. In the w/o IPFG setting, we replace IPFG with a simpler proxy feature generator following prior work[Zhang et al\. \(2024a\)](https://arxiv.org/html/2609.10950#bib.bib12)\. Specifically, instead of generating modality\-specific proxy features and combining them through a gating mechanism, the audio and vision features are directly used as inputs to a single network that produces the proxy feature\.

According to Table[3](https://arxiv.org/html/2609.10950#S4.T3),CompNet\-firstconsistently achieves the lowest performance across all datasets\. One possible reason is that freezing the shared BERT encoder after completeness learning prevents it from being further adapted for sentiment prediction, leading to misalignment between the two objectives\. This result highlights the necessity of continuously updating the shared BERT encoder across both tasks, while preserving the hierarchical structure from completeness learning to sentiment learning\. In contrast,CompNet\-laterachieves relatively competitive performance withTCMR \(Full\)on MOSI, while it performs relatively worse on MOSEI\. We attribute this discrepancy to the differences in scale and characteristics between the two datasets\. Compared to MOSI, MOSEI contains substantially more samples and covers a wider range of topics, indicating that a broader range of sentiment expression patterns must be learned\. As a result, learning completeness estimation by optimizing a CompNet with relatively few parameters after the shared BERT encoder has already been optimized for ground\-truth sentiment labels may be insufficient to capture the diverse patterns present in MOSEI\. In this regard,End2Endshows relatively more competitive performance on MOSEI than on MOSI, which we attribute to MOSEI’s greater diversity reducing the influence of the dominant task, thereby naturally alleviating gradient conflict\. Nevertheless, its inherent end\-to\-end nature still results in sub\-optimal performance compared toTCMR \(Full\)\. RemovingIPFGconsistently degrades performance across all datasets\. This suggests that it adaptively regulates the contribution of each auxiliary modality for incomplete text reconstruction, effectively filtering redundant information while enhancing complementary semantic cues\.

## 6In\-depth Analysis

\(a\)MOSI\(b\)MOSEI
Figure 5:Comparison of error\-triggering sentiment clue tokens across two benchmark datasets\. Each histogram shows the frequency of missing tokens that cause misclassification in the sentiment classifier\.#### Sentiment Token Sensitivity

To further examine the effectiveness of the TPSC label, we conduct a sensitivity analysis on sentiment\-related tokens\. First, we collect samples that are correctly classified by a pretrained sentiment polarity classifier on complete textual inputs\. For each sample, we construct a perturbation token set by masking each word token in turn\. Then, we count the frequency of tokens whose masking causes a previously correct prediction to become incorrect \(detailed procedures are provided in Appendix[D\.2](https://arxiv.org/html/2609.10950#A4.SS2)\)\.

As shown in Figure[5](https://arxiv.org/html/2609.10950#S6.F5), misclassifications are concentrated on affective tokens such as \[’boring’\], \[’not’\], and \[’fun’\]\. This observation supports our hypothesis that TCP indeed reflects the preservation of sentiment\-relevant information, as discussed in Section[3\.4](https://arxiv.org/html/2609.10950#S3.SS4.SSS0.Px2)\. Consequently,CompNetprovides a more accurate estimate of semantic completeness, enabling more reliable reconstruction by maximizing complementary semantic information from auxiliary modalities\.

![Refer to caption](https://arxiv.org/html/2609.10950v1/CaseStudy.png)Figure 6:Qualitative comparison between the LNLN and TCMR models under varying missing\-text conditions\. Each cell presents an example utterance where key semantic cues are either present or missing\.
#### Case Study

Figure[6](https://arxiv.org/html/2609.10950#S6.F6)illustrates how LNLN and TCMR behave under different missing\-text conditions\. Overall, LNLN fails to accurately reconstruct missing semantics\. This stems from overestimating or underestimating the semantic information remaining in incomplete text, leading to incorrect sentiment prediction\. In contrast, TCMR produces correct predictions through more accurate completeness estimation\. This allows TCMR to adaptively balance its reliance on auxiliary modalities based on the extent of semantic loss, resulting in a more faithful reconstruction\. Notably, the bottom\-right cell provides an interesting insight\. Although LNLN can estimate completeness accurately in this case, due to the high missing rate and the loss of a key sentiment token, it still fails to produce the correct sentiment prediction\. This suggests that inaccurate completeness estimation during training may limit LNLN’s ability to reconstruct missing semantics\. Therefore, accurately estimating the semantic information retained in incomplete text is crucial not only for effective reconstruction but also for stable sentiment prediction\.

Table 5:Qualitative comparison of completeness labels\. Each case shows how MRSC, LDSC, and TPSC assign completeness labels depending on whether sentiment\-related tokens are missing\. In the TPSC column, the value in parentheses denotesP⁡\(ys​p∣complete\)P\(y\_\{sp\}\\mid\\text\{complete\}\), while the value above it denotesP⁡\(ys​p∣incomplete\)P\(y\_\{sp\}\\mid\\text\{incomplete\}\)\.✓and✗indicate correct and incorrect completeness estimation, respectively\.CaseComplete TextIncomplete TextMRSCLDSCTPSC1I was happy to see itI\[UNK\]happy to see it0\.8333✓0\.9888✓✓2The soundtrack is good its really good some really great songs\[UNK\]soundtrack\[UNK\]\[UNK\]\[UNK\]really\[UNK\]\[UNK\]\[UNK\]\[UNK\]\[UNK\]0\.1818✓0\.0044✓✓3But it was really really awesomebut it was\[UNK\]\[UNK\]\[UNK\]0\.5000✗0\.0112✓✓4Kids are gonna love the film\[UNK\]\[UNK\]\[UNK\]love the\[UNK\]0\.3333✗0\.9808✓✓5I will admit I’m a big Johnny Depp fanI will\[UNK\]I’m\[UNK\]big Johnny Depp fan0\.7931✓0\.0690✗✓6and you’re better off saving your money uh and maybe renting it when it comes out on uh video soand\[UNK\]better\[UNK\]saving\[UNK\]money\[UNK\]\[UNK\]maybe renting it when it\[UNK\]out\[UNK\]uh\[UNK\]\[UNK\]0\.5921✓0\.9928✗✓7I really dig this movieI\[UNK\]\[UNK\]this\[UNK\]0\.4000✗0\.5000✗✓
#### Comparison of Completeness Labels

Table[5](https://arxiv.org/html/2609.10950#S6.T5)provides a qualitative comparison of different completeness labeling strategies under random missing conditions\. Cases 3 and 4 highlight the limitations of MRSC\. Since MRSC estimates completeness solely based on the proportion of missing tokens, it cannot capture the actual semantic information loss in the incomplete text\. In contrast, LDSC and TPSC more appropriately reflect completeness by considering the semantic importance of sentiment\-related tokens\. Cases 5–7 reveal the limitations of LDSC\. For example, in Cases 5 and 7, words such as \[’fan’\] and \[’dig’\] are used as positive sentiment expressions\. However, as SentiWordNet assigns sentiment scores based on the most frequent sense of a word, the assigned score may differ from the sense used in the actual context\. Moreover, in Case 6, \[’better’\] is assigned a positive sentiment score in SentiWordNet, leading LDSC to yield a high completeness value even though the overall sentence conveys negative sentiment\. In contrast, TPSC produces relatively correct completeness in these challenging cases due to its ability to capture sentiment based on the context of the full sentence rather than individual words\.

## 7Conclusion

In this work, we reinterpret semantic information loss in incomplete data based on the change in target\-class probability between complete and incomplete data to address the performance degradation of text\-centric fusion models in MSA caused by noisy or missing data\. First, we generate pseudo\-labels that provide supervision for training a completeness estimator\. We further propose an optimization strategy that mitigates gradient conflicts between the completeness estimation and sentiment prediction tasks, thereby stabilizing hierarchical multi\-task learning\. We believe the proposed completeness estimation approach can be broadly applied beyond MSA, such as confidence\-aware decision making and low\-quality data detection or restoration\.

## Limitations

Although TCMR consistently outperforms existing reconstruction\-based MSA models, several limitations remain to be addressed in future work\. First, this study only focuses on reconstructing semantic information loss in the textual modality\. Extending the TPSC\-based completeness estimation to other modalities, such as audio and vision, could improve general multimodal fusion approaches beyond text\-centric fusion\. Second, TCMR relies on pseudo\-labeling, which inherently introduces potential noise in the supervision signal\. Future work could mitigate this issue by adopting confidence\-aware filtering, using only high\-confidence TPSC predictions as pseudo\-labels while relying on complementary strategies such as LDSC for less confident samples\. For samples where both TPSC and LDSC yield low confidence, human verification could be incorporated to further reduce label noise\.

## Acknowledgments

This work was supported by Institute of Information & Communications Technology Planning & Evaluation \(IITP\) grant funded by the Korea government \(MSIT\) \(No\. RS\-2022\-II220608, RS\-2022\-II220320, and RS\-2024\-00398115\); and the Korea Electronics Technology Institute \(KETI\) through the project “Development of Modular Humanoid Physical AI Technology for Performing Complex Tasks in Industrial Environments\.”

## References

- Baccianellaet al\.\(2010\)S\. Baccianella, A\. Esuli, and F\. SebastianiSentiWordNet 3\.0: an enhanced lexical resource for sentiment analysis and opinion mining\.InProceedings of the Seventh International Conference on Language Resources and Evaluation \(LREC’10\),N\. Calzolari, K\. Choukri, B\. Maegaard, J\. Mariani, J\. Odijk, S\. Piperidis, M\. Rosner, and D\. Tapias \(Eds\.\),Valletta, Malta\.External Links:[Link](https://aclanthology.org/L10-1531/)Cited by:[§D\.1](https://arxiv.org/html/2609.10950#A4.SS1.p1.1),[3rd item](https://arxiv.org/html/2609.10950#S4.I1.i3.p1.1)\.
- Bagher Zadehet al\.\(2018\)A\. Bagher Zadeh, P\. P\. Liang, S\. Poria, E\. Cambria, and L\. MorencyMultimodal language analysis in the wild: CMU\-MOSEI dataset and interpretable dynamic fusion graph\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),I\. Gurevych and Y\. Miyao \(Eds\.\),Melbourne, Australia,pp\. 2236–2246\.External Links:[Link](https://aclanthology.org/P18-1208/),[Document](https://dx.doi.org/10.18653/v1/P18-1208)Cited by:[§4\.1](https://arxiv.org/html/2609.10950#S4.SS1.SSS0.Px1.p1.1)\.
- Baltrušaitiset al\.\(2016\)T\. Baltrušaitis, P\. Robinson, and L\. MorencyOpenFace: an open source facial behavior analysis toolkit\.In2016 IEEE Winter Conference on Applications of Computer Vision \(WACV\),Vol\.,pp\. 1–10\.External Links:[Document](https://dx.doi.org/10.1109/WACV.2016.7477553)Cited by:[§3\.3](https://arxiv.org/html/2609.10950#S3.SS3.p1.1)\.
- Chenet al\.\(2018\)Z\. Chen, V\. Badrinarayanan, C\. Lee, and A\. RabinovichGradNorm: gradient normalization for adaptive loss balancing in deep multitask networks\.InProceedings of the 35th International Conference on Machine Learning,J\. Dy and A\. Krause \(Eds\.\),Proceedings of Machine Learning Research, Vol\.80,pp\. 794–803\.External Links:[Link](https://proceedings.mlr.press/v80/chen18a.html)Cited by:[Appendix C](https://arxiv.org/html/2609.10950#A3.p1.1)\.
- Corbièreet al\.\(2019\)C\. Corbière, N\. THOME, A\. Bar\-Hen, M\. Cord, and P\. PérezAddressing failure prediction by learning model confidence\.InAdvances in Neural Information Processing Systems,H\. Wallach, H\. Larochelle, A\. Beygelzimer, F\. d'Alché\-Buc, E\. Fox, and R\. Garnett \(Eds\.\),Vol\.32,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2019/file/757f843a169cc678064d9530d12a1881-Paper.pdf)Cited by:[§3\.4](https://arxiv.org/html/2609.10950#S3.SS4.SSS0.Px2.p2.1)\.
- Devlinet al\.\(2019\)J\. Devlin, M\. Chang, K\. Lee, and K\. ToutanovaBERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 4171–4186\.External Links:[Link](https://aclanthology.org/N19-1423/),[Document](https://dx.doi.org/10.18653/v1/N19-1423)Cited by:[§3\.3](https://arxiv.org/html/2609.10950#S3.SS3.p1.1)\.
- Hanet al\.\(2021\)W\. Han, H\. Chen, A\. Gelbukh, A\. Zadeh, L\. Morency, and S\. PoriaBi\-bimodal modality fusion for correlation\-controlled multimodal sentiment analysis\.InProceedings of the 2021 International Conference on Multimodal Interaction,ICMI ’21,New York, NY, USA,pp\. 6–15\.External Links:ISBN 9781450384810,[Link](https://doi.org/10.1145/3462244.3479919),[Document](https://dx.doi.org/10.1145/3462244.3479919)Cited by:[§1](https://arxiv.org/html/2609.10950#S1.p1.1),[§2](https://arxiv.org/html/2609.10950#S2.p1.1)\.
- Hazarikaet al\.\(2022\)D\. Hazarika, Y\. Li, B\. Cheng, S\. Zhao, R\. Zimmermann, and S\. PoriaAnalyzing modality robustness in multimodal sentiment analysis\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,M\. Carpuat, M\. de Marneffe, and I\. V\. Meza Ruiz \(Eds\.\),Seattle, United States,pp\. 685–696\.External Links:[Link](https://aclanthology.org/2022.naacl-main.50/),[Document](https://dx.doi.org/10.18653/v1/2022.naacl-main.50)Cited by:[§1](https://arxiv.org/html/2609.10950#S1.p1.1)\.
- Hazarikaet al\.\(2020\)D\. Hazarika, R\. Zimmermann, and S\. PoriaMISA: modality\-invariant and \-specific representations for multimodal sentiment analysis\.InProceedings of the 28th ACM International Conference on Multimedia,MM ’20,New York, NY, USA,pp\. 1122–1131\.External Links:ISBN 9781450379885,[Link](https://doi.org/10.1145/3394171.3413678),[Document](https://dx.doi.org/10.1145/3394171.3413678)Cited by:[§2](https://arxiv.org/html/2609.10950#S2.p1.1)\.
- Liet al\.\(2025\)X\. Li, X\. Cheng, D\. Miao, X\. Zhang, and Z\. LiTF\-Mamba: text\-enhanced fusion mamba with missing modalities for robust multimodal sentiment analysis\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 11252–11267\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.602/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.602),ISBN 979\-8\-89176\-335\-7Cited by:[§2](https://arxiv.org/html/2609.10950#S2.p2.1)\.
- Linet al\.\(2025\)R\. Lin, Q\. He, S\. Mai, Y\. Zeng, A\. Xiong, L\. Huang, Y\. Tan, and H\. HuCyIN: cyclic informative latent space for bridging complete and incomplete multimodal learning\.InAdvances in Neural Information Processing Systems,D\. Belgrave, C\. Zhang, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, and N\. Chen \(Eds\.\),Vol\.38, Main Conference,pp\. 17605–17642\.External Links:[Document](https://dx.doi.org/10.52202/085713-0594),[Link](https://proceedings.neurips.cc/paper_files/paper/2025/file/1994798981a8467d7d50d8603c6caecb-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2609.10950#S2.p2.1)\.
- Lin and Hu \(2023\)R\. Lin and H\. HuMissModal: increasing robustness to missing modality in multimodal sentiment analysis\.Transactions of the Association for Computational Linguistics11,pp\. 1686–1702\.External Links:ISSN 2307\-387X,[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00628),[Link](https://doi.org/10.1162/tacl_a_00628),https://direct\.mit\.edu/tacl/article\-pdf/doi/10\.1162/tacl\_a\_00628/2201959/tacl\_a\_00628\.pdfCited by:[§2](https://arxiv.org/html/2609.10950#S2.p2.1)\.
- Liuet al\.\(2018\)Z\. Liu, Y\. Shen, V\. B\. Lakshminarasimhan, P\. P\. Liang, A\. Bagher Zadeh, and L\. MorencyEfficient low\-rank multimodal fusion with modality\-specific factors\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),I\. Gurevych and Y\. Miyao \(Eds\.\),Melbourne, Australia,pp\. 2247–2256\.External Links:[Link](https://aclanthology.org/P18-1209/),[Document](https://dx.doi.org/10.18653/v1/P18-1209)Cited by:[§2](https://arxiv.org/html/2609.10950#S2.p1.1)\.
- Maoet al\.\(2022\)H\. Mao, Z\. Yuan, H\. Xu, W\. Yu, Y\. Liu, and K\. GaoM\-SENA: an integrated platform for multimodal sentiment analysis\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations,V\. Basile, Z\. Kozareva, and S\. Stajner \(Eds\.\),Dublin, Ireland,pp\. 204–213\.External Links:[Link](https://aclanthology.org/2022.acl-demo.20/),[Document](https://dx.doi.org/10.18653/v1/2022.acl-demo.20)Cited by:[§4\.2](https://arxiv.org/html/2609.10950#S4.SS2.SSS0.Px1.p1.1)\.
- McFeeet al\.\(2015\)B\. McFee, C\. Raffel, D\. Liang, D\. P\. W\. Ellis, M\. McVicar, E\. Battenberg, and O\. NietoLibrosa: audio and music signal analysis in python\.InProceedings of the 14th Python in Science Conference, SciPy 2015, Austin, Texas, USA, July 6\-12, 2015,K\. D\. Huff and J\. Bergstra \(Eds\.\),pp\. 18–24\.External Links:[Link](https://doi.org/10.25080/Majora-7b98e3ed-003),[Document](https://dx.doi.org/10.25080/MAJORA-7B98E3ED-003)Cited by:[§3\.3](https://arxiv.org/html/2609.10950#S3.SS3.p1.1)\.
- Shiet al\.\(2025\)P\. Shi, M\. Hu, S\. Nakagawa, X\. Zheng, X\. Shi, and F\. RenText\-guided reconstruction network for sentiment analysis with uncertain missing modalities\.IEEE Transactions on Affective Computing16\(3\),pp\. 1825–1838\.External Links:[Document](https://dx.doi.org/10.1109/TAFFC.2025.3541743)Cited by:[§2](https://arxiv.org/html/2609.10950#S2.p2.1)\.
- Sunet al\.\(2022\)H\. Sun, H\. Wang, J\. Liu, Y\. Chen, and L\. LinCubeMLP: an mlp\-based model for multimodal sentiment analysis and depression estimation\.InProceedings of the 30th ACM International Conference on Multimedia,MM ’22,New York, NY, USA,pp\. 3722–3729\.External Links:ISBN 9781450392037,[Link](https://doi.org/10.1145/3503161.3548025),[Document](https://dx.doi.org/10.1145/3503161.3548025)Cited by:[§2](https://arxiv.org/html/2609.10950#S2.p1.1)\.
- Tsaiet al\.\(2019\)Y\. H\. Tsai, S\. Bai, P\. P\. Liang, J\. Z\. Kolter, L\. Morency, and R\. SalakhutdinovMultimodal transformer for unaligned multimodal language sequences\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 6558–6569\.External Links:[Link](https://aclanthology.org/P19-1656/),[Document](https://dx.doi.org/10.18653/v1/P19-1656)Cited by:[§2](https://arxiv.org/html/2609.10950#S2.p1.1)\.
- Vaswaniet al\.\(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. PolosukhinAttention is all you need\.InAdvances in Neural Information Processing Systems,I\. Guyon, U\. V\. Luxburg, S\. Bengio, H\. Wallach, R\. Fergus, S\. Vishwanathan, and R\. Garnett \(Eds\.\),Vol\.30,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)Cited by:[§3\.3](https://arxiv.org/html/2609.10950#S3.SS3.p2.1)\.
- Wanget al\.\(2023a\)D\. Wang, X\. Guo, Y\. Tian, J\. Liu, L\. He, and X\. LuoTETFN: a text enhanced transformer fusion network for multimodal sentiment analysis\.Pattern Recognition136,pp\. 109259\.External Links:ISSN 0031\-3203,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.patcog.2022.109259),[Link](https://www.sciencedirect.com/science/article/pii/S0031320322007385)Cited by:[§1](https://arxiv.org/html/2609.10950#S1.p1.1),[§2](https://arxiv.org/html/2609.10950#S2.p1.1)\.
- Wanget al\.\(2023b\)D\. Wang, S\. Liu, Q\. Wang, Y\. Tian, L\. He, and X\. GaoCross\-modal enhancement network for multimodal sentiment analysis\.IEEE Transactions on Multimedia25\(\),pp\. 4909–4921\.External Links:[Document](https://dx.doi.org/10.1109/TMM.2022.3183830)Cited by:[§1](https://arxiv.org/html/2609.10950#S1.p1.1),[§2](https://arxiv.org/html/2609.10950#S2.p1.1)\.
- Weiet al\.\(2023\)Y\. Wei, S\. Yuan, R\. Yang, L\. Shen, Z\. Li, L\. Wang, and M\. ChenTackling modality heterogeneity with multi\-view calibration network for multimodal sentiment detection\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 5240–5252\.External Links:[Link](https://aclanthology.org/2023.acl-long.287/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.287)Cited by:[§1](https://arxiv.org/html/2609.10950#S1.p1.1)\.
- Yanget al\.\(2025\)M\. Yang, K\. Zhang, Y\. Ye, Y\. Zhang, R\. Yu, and M\. HouDecoupling and reconstructing: a multimodal sentiment analysis framework towards robustness\.InProceedings of the Thirty\-Fourth International Joint Conference on Artificial Intelligence, IJCAI\-25,J\. Kwok \(Ed\.\),pp\. 6803–6811\.Note:Main TrackExternal Links:[Document](https://dx.doi.org/10.24963/ijcai.2025/757),[Link](https://doi.org/10.24963/ijcai.2025/757)Cited by:[§2](https://arxiv.org/html/2609.10950#S2.p2.1)\.
- Yuet al\.\(2020a\)T\. Yu, S\. Kumar, A\. Gupta, S\. Levine, K\. Hausman, and C\. FinnGradient surgery for multi\-task learning\.InAdvances in Neural Information Processing Systems,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\.F\. Balcan, and H\. Lin \(Eds\.\),Vol\.33,pp\. 5824–5836\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/file/3fe78a8acf5fda99de95303940a2420c-Paper.pdf)Cited by:[Appendix C](https://arxiv.org/html/2609.10950#A3.p1.1),[§3\.6](https://arxiv.org/html/2609.10950#S3.SS6.p1.1)\.
- Yuet al\.\(2020b\)W\. Yu, H\. Xu, F\. Meng, Y\. Zhu, Y\. Ma, J\. Wu, J\. Zou, and K\. YangCH\-SIMS: a Chinese multimodal sentiment analysis dataset with fine\-grained annotation of modality\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 3718–3727\.External Links:[Link](https://aclanthology.org/2020.acl-main.343/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.343)Cited by:[§4\.1](https://arxiv.org/html/2609.10950#S4.SS1.SSS0.Px1.p1.1)\.
- Yuet al\.\(2021\)W\. Yu, H\. Xu, Z\. Yuan, and J\. WuLearning modality\-specific representations with self\-supervised multi\-task learning for multimodal sentiment analysis\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.35,pp\. 10790–10797\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v35i12.17289)Cited by:[§2](https://arxiv.org/html/2609.10950#S2.p1.1)\.
- Yuanet al\.\(2021\)Z\. Yuan, W\. Li, H\. Xu, and W\. YuTransformer\-based feature reconstruction network for robust multimodal sentiment analysis\.InProceedings of the 29th ACM International Conference on Multimedia,MM ’21,New York, NY, USA,pp\. 4400–4407\.External Links:ISBN 9781450386517,[Link](https://doi.org/10.1145/3474085.3475585),[Document](https://dx.doi.org/10.1145/3474085.3475585)Cited by:[§1](https://arxiv.org/html/2609.10950#S1.p2.1),[§2](https://arxiv.org/html/2609.10950#S2.p2.1)\.
- Zadehet al\.\(2017\)A\. Zadeh, M\. Chen, S\. Poria, E\. Cambria, and L\. MorencyTensor fusion network for multimodal sentiment analysis\.InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing,M\. Palmer, R\. Hwa, and S\. Riedel \(Eds\.\),Copenhagen, Denmark,pp\. 1103–1114\.External Links:[Link](https://aclanthology.org/D17-1115/),[Document](https://dx.doi.org/10.18653/v1/D17-1115)Cited by:[§2](https://arxiv.org/html/2609.10950#S2.p1.1)\.
- Zadehet al\.\(2016\)A\. Zadeh, R\. Zellers, E\. Pincus, and L\. MorencyMultimodal sentiment intensity analysis in videos: facial gestures and verbal messages\.IEEE Intelligent Systems31\(6\),pp\. 82–88\.External Links:[Document](https://dx.doi.org/10.1109/MIS.2016.94)Cited by:[§4\.1](https://arxiv.org/html/2609.10950#S4.SS1.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2024a\)H\. Zhang, W\. Wang, and T\. YuTowards robust multimodal sentiment analysis with incomplete data\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 55943–55974\.External Links:[Document](https://dx.doi.org/10.52202/079017-1779),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/65a39213d7d0e1eb5d192aa77e77eeb7-Paper-Conference.pdf)Cited by:[§A\.3](https://arxiv.org/html/2609.10950#A1.SS3.p1.1),[§A\.4](https://arxiv.org/html/2609.10950#A1.SS4.p1.1),[§1](https://arxiv.org/html/2609.10950#S1.p2.1),[§2](https://arxiv.org/html/2609.10950#S2.p2.1),[§3\.3](https://arxiv.org/html/2609.10950#S3.SS3.p2.1),[§3\.4](https://arxiv.org/html/2609.10950#S3.SS4.SSS0.Px3.p1.1),[§3\.4](https://arxiv.org/html/2609.10950#S3.SS4.SSS0.Px4.p1.2),[§4\.1](https://arxiv.org/html/2609.10950#S4.SS1.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2609.10950#S5.p1.1)\.
- Zhanget al\.\(2023\)H\. Zhang, Y\. Wang, G\. Yin, K\. Liu, Y\. Liu, and T\. YuLearning language\-guided adaptive hyper\-modality representation for multimodal sentiment analysis\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 756–767\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.49/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.49)Cited by:[§1](https://arxiv.org/html/2609.10950#S1.p1.1),[§2](https://arxiv.org/html/2609.10950#S2.p1.1)\.
- Zhanget al\.\(2024b\)Z\. Zhang, J\. Shen, C\. Cao, G\. Dai, S\. Zhou, Q\. Zhang, S\. Zhang, and E\. ShutovaProactive gradient conflict mitigation in multi\-task learning: a sparse training perspective\.arXiv preprint arXiv:2411\.18615\.External Links:[Link](https://arxiv.org/abs/2411.18615)Cited by:[Appendix C](https://arxiv.org/html/2609.10950#A3.p2.1)\.
- Zhuet al\.\(2025\)A\. Zhu, M\. Hu, X\. Wang, J\. Yang, Y\. Tang, and N\. AnProxy\-driven robust multimodal sentiment analysis with incomplete data\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 22123–22138\.External Links:[Link](https://aclanthology.org/2025.acl-long.1075/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1075),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2609.10950#S1.p2.1),[§2](https://arxiv.org/html/2609.10950#S2.p2.1),[§3\.3](https://arxiv.org/html/2609.10950#S3.SS3.p2.1),[§4\.1](https://arxiv.org/html/2609.10950#S4.SS1.SSS0.Px2.p1.1)\.

## Appendix AImplementation Details

### A\.1Dataset Statistics

Table 6:Statistics of the MOSI, MOSEI, and SIMS datasets used in our experiments\.MOSIMOSEISIMS\#Samples2,19922,8562,281Train / Val / Test1,284 / 229 / 68616,326 / 1,871 / 4,6591,368 / 456 / 457\#Speakers931,000\+–\#Topics89250\+–SourceYouTubeYouTubeMovies / TVAnnotation TypeUtterance\-levelUtterance\-levelUtterance\-levelSentiment Score–3 to \+3–3 to \+3–1 to \+1

Table[6](https://arxiv.org/html/2609.10950#A1.T6)provides key statistics of the three benchmark datasets used in our experiments\. Among them, MOSEI stands out as substantially larger and more diverse in speakers and topics than MOSI and SIMS, resulting in greater variability across acoustic, visual, and textual modalities\. Due to this diversity, MOSEI is considered a more challenging benchmark than MOSI and SIMS\.

### A\.2Model Configuration

The proposed TCMR model was implemented in PyTorch \(v2\.2\.1\) with Python 3\.11\.7 and trained on a workstation with an NVIDIA GeForce RTX 4090 GPU and an AMD Ryzen Threadripper PRO 5975WX CPU\. To offer a thorough specification of our architecture design, Table[7](https://arxiv.org/html/2609.10950#A1.T7)provides the detailed configurations of all modules within TCMR\.

Table 7:Network configurations of TCMR\. Transformer\-based modules list the number of layers, sequence lengths, token counts, input dimensions, attention heads, and hidden dimensions for both MOSI and MOSEI \(shown as MOSI / MOSEI\)\. MLP\-based modules display the input dimensions and hidden\-layer widths used in each component\.Module \(MOSI / MOSEI\)TypeNotation\#LayersSeq\. lenToken lenInput dim\#HeadsHidden dimTransformer\-based modulesBERT EncoderTransformerθbert\\theta^\{\\text\{bert\}\}1250 / 5050768 / 76812768Text EncoderTransformerθttrans\\theta^\{\\text\{trans\}\}\_\{t\}249 / 498768 / 7688128Audio EncoderTransformerθatrans\\theta^\{\\text\{trans\}\}\_\{a\}2375 / 50085 / 748128Vision EncoderTransformerθvtrans\\theta^\{\\text\{trans\}\}\_\{v\}2500 / 500820 / 358128A2T Generator \(audio→\\rightarrowtext\)Transformerθagen\\theta^\{\\text\{gen\}\}\_\{a\}216 / 16–128 / 1288128V2T Generator \(vision→\\rightarrowtext\)Transformerθvgen\\theta^\{\\text\{gen\}\}\_\{v\}216 / 16–128 / 1288128Text Refinement TransformerTransformerθtref\\theta^\{\\text\{ref\}\}\_\{t\}28 / 8–128 / 1288128CrossTransformer \(fusion head\)Transformerθcross\\theta^\{\\text\{cross\}\}28 / 8–128 / 1288128Reconstructor \(Audio\)Transformerθarecon\\theta^\{\\text\{recon\}\}\_\{a\}28 / 8–128 / 1288128Reconstructor \(Vision\)Transformerθvrecon\\theta^\{\\text\{recon\}\}\_\{v\}28 / 8–128 / 1288128MLP\-based modulesCompleteness Estimator \(CompNet\)MLPθcomp\\theta^\{\\text\{comp\}\}6––768 / 768–\[768, 768, 1536, 768, 384, 1\]Polarity Classifier \(Classifier\)MLPθpreclassifier\\theta^\{\\text\{classifier\}\}\_\{\\text\{pre\}\}2––768 / 768–\[384, 3\]Gating Network \(GateNet\)MLPθgate\\theta^\{\\text\{gate\}\}2––256 / 256–\[128, 1\]Text Regressor \(Regressor\)MLPθregressor\\theta^\{\\text\{regressor\}\}1––128 / 128–\[1\]

Table 8:Performance sensitivity to sentiment class granularity \(kk\) in TPSC label generation\.\(a\)MOSI datasetModelAcc\-7Acc\-5Non0 Acc/F1Has0 Acc/F1MAECorrTPSC\-2cls31\.4536\.0170\.01 / 69\.9169\.46 / 69\.251\.107350\.66TPSC\-3cls32\.8036\.7372\.01 / 71\.6770\.93 / 70\.491\.072152\.28TPSC\-5cls30\.9635\.1269\.51 / 69\.0968\.93 / 68\.421\.144949\.62TPSC\-7cls32\.1635\.3570\.85 / 70\.7669\.74 / 69\.541\.104049\.28

\(b\)MOSEI datasetModelAcc\-7Acc\-5Non0 Acc/F1Has0 Acc/F1MAECorrTPSC\-2cls46\.4147\.2577\.76 / 77\.3276\.31 / 76\.530\.667757\.95TPSC\-3cls47\.2748\.1677\.99/ 76\.9977\.59 / 77\.210\.661458\.63TPSC\-5cls46\.9447\.9677\.79 /77\.5675\.46 / 75\.950\.663859\.28TPSC\-7cls46\.7547\.6377\.74 / 76\.7477\.15 / 76\.790\.666757\.81

\(c\)SIMS datasetModelAcc\-5Acc\-3Acc\-2F1MAECorrTPSC\-2cls31\.1557\.4572\.8168\.880\.52120\.379TPSC\-3cls32\.4356\.2371\.7268\.960\.54780\.321TPSC\-5cls33\.1455\.4569\.1368\.060\.54650\.329

### A\.3Evaluation Metrics

Following prior work[Zhang et al\. \(2024a\)](https://arxiv.org/html/2609.10950#bib.bib12), we adopt the same evaluation metrics\. For MOSI and MOSEI, we report both classification metrics \(Acc\-7, Acc\-5, Non0 Acc / F1, Has0 Acc / F1\) and regression metrics \(MAE, Corr\)\. The classification metrics are computed by discretizing the continuous sentiment scores into predefined sentiment intervals\. For SIMS, we report Acc\-2, Acc\-3, Acc\-5, F1, MAE, and Corr\. Non0 Acc denotes negative/positive classification \(excluding zero\-valued samples\), whereas Has0 Acc denotes negative/non\-negative classification, where zero\-valued samples are treated as non\-negative\.

### A\.4Multimodal Fusion Module

The multimodal fusion module follows the design of prior work[Zhang et al\. \(2024a\)](https://arxiv.org/html/2609.10950#bib.bib12), and its key components are briefly summarized here\. Starting from the reconstructed text representationH^t\\hat\{H\}\_\{t\}, a refinement Transformer encoder applies self\-attention, producing layerwise refined featuresHt\+\(i\)H\_\{t\}^\{\+\(i\)\}withi∈\{1,…,lref\}i\\in\\\{1,\\ldots,l^\{\\text\{ref\}\}\\\}\. Based onHt\+\(i\)H\_\{t\}^\{\+\(i\)\}, cross\-modal attention integrates complementary cues from vision and audio across multiple layers\. At each layer, the current refined text featureHt\+\(i\)H\_\{t\}^\{\+\(i\)\}serves as the query and attends toHvH\_\{v\}andHaH\_\{a\}, while the fused representation updatesHf\(i\)H\_\{f\}^\{\(i\)\}through residual accumulation\. In compact form,Hf\(i\)=Hf\(i−1\)\+MHA⁡\(Ht\+\(i\),Hv\)\+MHA⁡\(Ht\+\(i\),Ha\)H\_\{f\}^\{\(i\)\}=H\_\{f\}^\{\(i\-1\)\}\+\\mathrm\{MHA\}\(H\_\{t\}^\{\+\(i\)\},H\_\{v\}\)\+\\mathrm\{MHA\}\(H\_\{t\}^\{\+\(i\)\},H\_\{a\}\)whereHf0H\_\{f\}^\{0\}is a learnable embedding andMHA⁡\(Q,K\)\\mathrm\{MHA\}\(Q,K\)denotes multi\-head attention with queryQQand key and value fromKK\. A CrossTransformer then models interactions between\[Hf\(lref\),Ht\+\(lref\)\]\[\\,H\_\{f\}^\{\(l^\{\\text\{ref\}\}\)\},\\,H\_\{t\}^\{\+\(l^\{\\text\{ref\}\}\)\}\\,\], and a regression head outputs the final sentiment predictiony^\\hat\{y\}\.

## Appendix BSentiment Polarity Classes

### B\.1Definition of Sentiment Polarity Classes

To utilize TCP, we convert continuous sentiment scores into discrete polarity classes\. As the score ranges differ across datasets, the class boundaries are defined separately for SIMS and MOSI/MOSEI\.

For the SIMS dataset, polarity classes are defined using fixed interval thresholds\. Specifically, the 2\-class setting divides the scores into\[−1\.0,0\.0\]\[\-1\.0,0\.0\]and\(0\.0,1\.0\]\(0\.0,1\.0\]\. The 3\-class setting introduces a neutral region, resulting in\[−1\.0,−0\.1\]\[\-1\.0,\-0\.1\],\(−0\.1,0\.1\]\(\-0\.1,0\.1\], and\(0\.1,1\.0\]\(0\.1,1\.0\]\. The 5\-class setting adopts a finer partition given by\[−1\.0,−0\.7\]\[\-1\.0,\-0\.7\],\(−0\.7,−0\.1\]\(\-0\.7,\-0\.1\],\(−0\.1,0\.1\]\(\-0\.1,0\.1\],\(0\.1,0\.7\]\(0\.1,0\.7\], and\(0\.7,1\.0\]\(0\.7,1\.0\]\.

For the MOSI and MOSEI datasets, polarity classes are defined by mapping sentiment scores into discrete levels\. The 2\-class setting separates negative \(<0<0\) and non\-negative \(≥0\\geq 0\) samples\. The 3\-class, 5\-class, and 7\-class settings correspond to\{−1,0,1\}\\\{\-1,0,1\\\},\{−2,−1,0,1,2\}\\\{\-2,\-1,0,1,2\\\}, and\{−3,−2,−1,0,1,2,3\}\\\{\-3,\-2,\-1,0,1,2,3\\\}, respectively\.

### B\.2Optimal Choice of Class Granularity

We conduct comparative experiments withk∈\{2,3,5,7\}k\\in\\\{2,3,5,7\\\}\(MOSI/MOSEI\) andk∈\{2,3,5\}k\\in\\\{2,3,5\\\}\(SIMS\) and select k = 3 \(MOSI/MOSEI\) and k = 2 \(SIMS\) based on validation performance\. As shown in Table[8](https://arxiv.org/html/2609.10950#A1.T8), this configuration also yields the best test\-set performance\.

## Appendix CGradient Conflict Analysis

In multi\-task learning, jointly optimizing multiple losses often leads to gradient conflicts, resulting in biased optimization where certain tasks dominate while others are under\-optimized\. To address this problem, prior work[Chen et al\. \(2018\)](https://arxiv.org/html/2609.10950#bib.bib33);[Yu et al\. \(2020a\)](https://arxiv.org/html/2609.10950#bib.bib24)has proposed methods that mitigate gradient conflicts by adjusting gradient magnitudes or directions during training\. For example,[Chen et al\. \(2018\)](https://arxiv.org/html/2609.10950#bib.bib33)propose GradNorm, a method that automatically balances multiple tasks during training by adjusting the magnitudes of gradients derived from each task\-specific loss, ensuring that all tasks learn at a similar rate\.[Yu et al\. \(2020a\)](https://arxiv.org/html/2609.10950#bib.bib24)propose PCGrad, which reduces gradient interference by projecting conflicting gradients onto the normal plane of other task gradients\.

Table 9:Gradient conflict rate \(GCR\) analysis measured during training\. F50/L50 denote the first and last 50% of training epochs, respectively\.MethodGCR \(%\)↓\\downarrowℒc​o​m​p\\mathcal\{L\}\_\{comp\}↓\\downarrowAllF50L50E2E51\.2751\.1654\.360\.1634GradNorm54\.8456\.5553\.160\.1294PCGrad52\.3357\.4348\.700\.0340AOS48\.4753\.6743\.330\.0055To validate the effectiveness of AOS, we conduct comparative experiments using existing methods as baselines\. Table[9](https://arxiv.org/html/2609.10950#A3.T9)presents the gradient conflict rates during training and the corresponding convergence behavior of the completeness estimation loss, following the protocol of prior work[Zhang et al\. \(2024b\)](https://arxiv.org/html/2609.10950#bib.bib25)\. In joint training,ℒcomp\\mathcal\{L\}\_\{\\text\{comp\}\}tends to converge to a local optimum, which leads to an increase in GCR \(see E2E in Table[9](https://arxiv.org/html/2609.10950#A3.T9)\)\. While GradNorm and PCGrad mitigate gradient conflict, they are still insufficient to escape local optima due to the fundamental limitation of joint training\. In contrast, AOS removes interference from other losses during theCompNettraining phase, thereby effectively mitigating suboptimal convergence ofℒcomp\\mathcal\{L\}\_\{\\text\{comp\}\}\.

## Appendix DAnalysis of Completeness Labels

### D\.1Lexicon\-derived Completeness Label

To further validate the effectiveness of the proposed TPSC, we introduceLexicon\-Derived Semantic Completeness\(LDSC\) as a comparison baseline\. LDSC measures completeness by explicitly quantifying how much sentiment\-related lexical information remains in the incomplete text\. Specifically, we adopt SentiWordNet[Baccianella et al\. \(2010\)](https://arxiv.org/html/2609.10950#bib.bib31), a widely used lexical resource for sentiment analysis\. SentiWordNet provides positive and negative sentiment scores for each word\.

![Refer to caption](https://arxiv.org/html/2609.10950v1/C_LDSD.png)Figure 7:Illustration of LDSC computationFigure[7](https://arxiv.org/html/2609.10950#A4.F7)illustrates the overall process of generating the LDSC label\. First, we define the sentiment intensity score of a tokenttas:

s⁡\(t\)=max⁡\(pos⁡\(t\),neg⁡\(t\)\),s\(t\)=\\max\\big\(\\mathrm\{pos\}\(t\),\\mathrm\{neg\}\(t\)\\big\),\(15\)wherepos⁡\(t\)\\mathrm\{pos\}\(t\)andneg⁡\(t\)\\mathrm\{neg\}\(t\)denote the positive and negative sentiment scores of the token\. Using this token\-level score, we compute the total sentiment\-related content of the complete and incomplete text:

Scomp=∑t∈Tcomps⁡\(t\),S\_\{\\text\{comp\}\}=\\sum\_\{t\\in T\_\{\\text\{comp\}\}\}s\(t\),\(16\)Sincomp=∑t∈Tincomps⁡\(t\),S\_\{\\text\{incomp\}\}=\\sum\_\{t\\in T\_\{\\text\{incomp\}\}\}s\(t\),\(17\)whereTcompT\_\{\\text\{comp\}\}andTincompT\_\{\\text\{incomp\}\}denote the token sets of the complete and incomplete text, respectively\. Finally, LDSC\-based completeness is computed as:

wl​d​s​c=Sincomp\+εScomp\+2​ε,w\_\{ldsc\}=\\frac\{S\_\{\\text\{incomp\}\}\+\\varepsilon\}\{S\_\{\\text\{comp\}\}\+2\\varepsilon\},\(18\)whereε\\varepsilonis a small smoothing constant introduced to avoid numerical instability and degenerate cases in which bothScompS\_\{\\text\{comp\}\}andSincompS\_\{\\text\{incomp\}\}become zero\. Intuitively, whenwl​d​s​c≈0w\_\{ldsc\}\\approx 0, most key sentiment tokens are missing, whereaswl​d​s​c≈1w\_\{ldsc\}\\approx 1indicates that the sentiment\-related information is largely preserved\.

\(a\)c≥0\.7c\\geq 0\.7\(b\)c≥0\.8c\\geq 0\.8\(c\)c≥0\.9c\\geq 0\.9\(d\)c≥0\.7c\\geq 0\.7\(e\)c≥0\.8c\\geq 0\.8\(f\)c≥0\.9c\\geq 0\.9
Figure 8:Token\-level sensitivity analysis across different confidence thresholds\. Subfigures \(a\)–\(c\) present results on the MOSI dataset, while \(d\)–\(f\) show the corresponding results on the MOSEI dataset\.
### D\.2Sentiment Token Sensitivity of TPSC

This subsection provides a more detailed description of the sentiment token sensitivity analysis introduced in Section[6](https://arxiv.org/html/2609.10950#S6.SS0.SSS0.Px1)\.

#### Analysis Procedure\.

To evaluate the sensitivity of pseudo completeness labels to key sentiment tokens, we conduct an analysis following a four\-step procedure:

- •Step 1: Correct\-sample filtering\.A pre\-trained sentiment polarity classifier is applied to the complete textual inputs, and only the correctly classified samples are selected for subsequent steps\. Note that a sample is included only when the true\-class probabilityc∈\[0,1\]c\\in\[0,1\]exceeds a predefined threshold\.
- •Step 2: Token\-wise Augmentation via Masking\.For each correctly classified sample, we generate augmented samples by sliding a window over the sequence and removing exactly one token at each position, replacing it with\[UNK\]\. For example, the sentence*“I love this movie”*produces four augmented samples: \(1\)\[UNK\]love this movie\. \(2\)I\[UNK\]this movie\. \(3\)I love\[UNK\]movie\. \(4\)I love this\[UNK\]\.
- •Step 3: Re\-evaluation of augmented samples\.Then, each augmented sample is passed through the same classifier used in Step 1, and we collect the samples in which masking a specific token causes the predicted sentiment to flip to the opposite polarity \(e\.g\., positive→\\rightarrownegative or negative→\\rightarrowpositive\)\.
- •Step 4: Identification of error\-triggering tokens\.Using the polarity\-flip samples collected in Step 3, we examine which token in each flipped sample triggers misclassification and compute how frequently each token leads to a misprediction\.

#### Analysis Results

Figure[8](https://arxiv.org/html/2609.10950#A4.F8)illustrates which tokens trigger polarity flips when masked under different thresholds\. In both datasets, we consistently observed that as the threshold increases key sentiment tokens remain at the top while semantically ambiguous tokens naturally disappear\. In MOSI, \[‘I’\] is a sentiment\-neutral token\. One possible explanation is that masking it creates an ambiguous context before the verb, which may resemble negation patterns \(e\.g\., “don’t”\) learned during training and occasionally lead to prediction flips\. In MOSEI, the token \[’two’\] is also observed among the flip\-triggering words\. This is because in the original sentence*“I give the movie two out of five stars”*the token \[’two’\] serves as an explicit negative cue\. Overall, the consistent emergence of key sentiment tokens at higher confidence thresholds suggests that the text classifier is well calibrated with respect to semantic cues\.

### D\.3Upper\-Bound Analysis

Table 10:Comparison between TCMR and its upper\-bound variant TCMR\-ub\.MethodAcc\-5Acc\-2MAECorrMOSI DatasetTCMR36\.7370\.931\.072152\.28TCMR\-ub41\.2477\.570\.930565\.03MOSEI DatasetTCMR48\.1677\.590\.661458\.63TCMR\-ub58\.7279\.830\.569771\.78SIMS DatasetTCMR31\.1572\.810\.521237\.91TCMR\-ub40\.3973\.690\.399166\.47To validate the effectiveness of the pseudo\-labels, we compare the performance of TCMR\-ub, trained with target TPSC labels\. Specifically, TCMR\-ub is trained by directly using the pseudo\-labels instead of training the confidence estimator, enabling us to assess the upper\-bound performance of our framework\. As shown in Table[10](https://arxiv.org/html/2609.10950#A4.T10), TCMR\-ub consistently outperforms TCMR across all metrics\. This indicates that the pseudo\-labels provide a meaningful and effective signal for guiding the reconstruction process\.

### D\.4Failure Cases of TPSC

DatasetAccuracy \(%\)Error Rate \(%\)MOSI74\.9225\.08MOSEI66\.9833\.02SIMS79\.0720\.93Table 11:Accuracy of the pretrained sentiment polar\- ity classifier used for TPSC pseudo\-labeling on each dataset’s complete text\. The higher error rate indicates a greater likelihood of noisy pseudo\-labels\.PatternUtteranceGTPredTCPAmbiguousThere are some funny momentsNeutralPositive0\.038Um I did enjoy itNeutralPositive0\.027SarcasmHi I’m pretty I have a giant smile I’m supposed to know things um walk of screenNegativePositive0\.131He um had all the charm of a narcissist xxx boy the whole filmNegativePositive0\.022

Table 12:Representative failure cases of the pretrained classifier in TPSC pseudo\-labeling\.DatasetTCMR vsAcc\-7Acc\-5Non0 A/FHas0 A/FMAECorrW\-LMOSICENet\+3\.31\+3\.83\+2\.13 / \+1\.73\+1\.50 / \+1\.10\+0\.108\+4\.048\-0P\-RMF\+3\.89\+5\.49\+2\.10 / \+2\.10\+1\.65 / \+1\.65\+0\.058\+2\.218\-0MOSEICENet−\-0\.09−\-0\.08\+0\.98 /−\-0\.27\+1\.63 / \+0\.98\+0\.001\+0\.395\-3P\-RMF\+0\.26\+0\.34−\-0\.08 /−\-0\.77\+0\.41 / \+0\.30\+0\.005\+0\.186\-2
DatasetTCMR vsAcc\-5Acc\-3Acc\-2F1MAECorrW\-LSIMSCENet\+9\.54\+3\.97\+3\.58\+11\.91\+0\.109\+0\.3656\-0P\-RMF−\-1\.57\+0\.15\+2\.23−\-1\.48\+0\.022\+0\.0184\-2

Table 13:Detailed comparison between TCMR\-TPSC and CENet/P\-RMF on MOSI, MOSEI, and SIMS\.Since TPSC is based on pseudo\-labeling, incorrect completeness labels may be generated when the classifier misclassifies samples even on complete data\. This is a fundamental limitation of pseudo\-label\-based approaches that do not rely on human annotations\. To assess the prevalence of such failure cases, we evaluated the performance of the pre\-trained classifier on each dataset\. Table[11](https://arxiv.org/html/2609.10950#A4.T11)shows that MOSEI exhibits relatively lower accuracy compared to other datasets, likely due to its higher complexity\. This suggests that the noisier pseudo\-labels in MOSEI can partially explain its relatively weaker generalization performance\.

We further analyzed failure cases and observed that misclassifications mainly occur in ambiguous and sarcastic expressions, as shown in Table[12](https://arxiv.org/html/2609.10950#A4.T12)\. These cases are inherently challenging to predict using only the text modality\. Despite these limitations, the impact of such mispredictions on completeness estimation is limited\. Due to their low TCP, TPSC assigns low completeness scores to misclassified samples, preventing overestimation of completeness and encouraging reliance on proxy features\.

## Appendix EAnalyses with competitive baselines

### E\.1Metric\-Level Comparison

In MSA tasks, both classification and regression performances are evaluated to provide a comprehensive assessment\. However, classification metrics may not fully represent the quality of sentiment prediction in certain cases due to discretization in their computation\.

Specifically, this discretization process discards fine\-grained differences between predictions within the same rounding interval\. For example, when computing the classification metrics \(e\.g\., Acc\-7, Acc\-5\), continuous sentiment predictions are first converted into discrete classes via rounding, which may lead to inconsistencies near class boundaries, as follows:

- •GT:1\.4→round⁡\(1\.4\)→1\.4\\to\\mathrm\{round\}\(1\.4\)\\toclass 1
- •Model A:1\.551\.55\(MAE=0\.15=0\.15\)→\\toclass 2
- •Model B:1\.101\.10\(MAE=0\.30=0\.30\)→\\toclass 1

As shown in Table[13](https://arxiv.org/html/2609.10950#A4.T13), except for a few classification metrics, TCMR\-TPSC consistently outperforms CENet and P\-RMF across all benchmark datasets\. This indicates that TCMR provides more reliable sentiment prediction overall\.

### E\.2Training Efficiency Analysis

Table 14:Comparison of model complexity and training time per epoch\.Model\# Params \(M\)Time / Epoch \(s\)LNLN115\.9715\.40P\-RMF117\.3118\.00TF\-Mamba111\.3011\.12TCMR119\.2312\.48As shown in Table[14](https://arxiv.org/html/2609.10950#A5.T14), TCMR achieves the second\-fastest training speed, following TF\-Mamba\. This is because the reconstruction modules used in LNLN and P\-RMF involve more computationally intensive operations, whereas TCMR employs a lightweight MLP\-based CompNet\. As a result, the additional optimization steps introduced by AOS do not significantly increase the overall training cost\.

Figure 9:Error bars indicate standard deviations\. Asterisks denote statistical significance \(\* p < 0\.1, \*\* p < 0\.05, \*\*\* p < 0\.01\)\.
### E\.3Statistical Analysis

To examine the statistical significance of performance differences among models, we conduct comparative experiments on the MOSI dataset with LNLN, P\-RMF, and TCMR\. As shown in Figure[9](https://arxiv.org/html/2609.10950#A5.F9), TCMR achieves statistically significant performance improvements across most evaluation metrics\.