CONFER: Conflict-Aware Evidence Negotiation for Regime-Calibrated Weak Supervision in Multimodal Emotion Recognition

# CONFER: Conflict-Aware Evidence Negotiation for Regime-Calibrated Weak Supervision in Multimodal Emotion Recognition
Source: [https://arxiv.org/html/2608.07867](https://arxiv.org/html/2608.07867)
###### Abstract

Multimodal emotion recognition often treats self\-reported labels as reliable supervision while overlooking self\-report unreliability and cross\-modal conflict\. We proposeCONFER, a graph\-based conflict\-aware evidence negotiation framework for weakly supervised multimodal emotion recognition\. CONFER represents each modality expert as a node with a predictive belief, boundary\-based uncertainty, and runtime reliability estimated from historical out\-of\-fold performance and current\-sample uncertainty\. Uncertainty\-aware compatibility and reliability\-directed asymmetric edge weights govern iterative message\-passing negotiation, followed by peer\-supported prediction readout\. Conflict reduction, residual disagreement, and mean modality uncertainty further characterize three regimes—Consensus, Dissent, and Ambiguity—for sample\-specific weak\-label calibration\. We evaluate CONFER on AMIGOS, MAHNOB\-HCI, and DEAP under subject\-dependent 10\-fold and strict leave\-one\-subject\-out \(LOSO\) protocols\. CONFER achieves competitive performance, reaching0\.873accuracy on AMIGOS\-V and0\.854accuracy on MAHNOB\-V under strict LOSO evaluation\. Further analyses show larger negotiation gains on high\-conflict samples and improved robustness to weak\-label corruption, indicating that cross\-modal conflict provides useful information for both directional modality coordination and supervision\-reliability estimation\.

## Introduction

Multimodal emotion recognition \(MER\) aims to infer latent emotional states from heterogeneous neural, physiological, acoustic, and behavioral signals\. Although these modalities provide complementary evidence, most existing methods only treat self\-reported valence and arousal scores as reliable supervision\. In practice, self\-reports are influenced by individual rating habits, cognitive interpretation, memory bias, and experimental context\(Mauss and Robinson[2009](https://arxiv.org/html/2608.07867#bib.bib15); Barrett[2017](https://arxiv.org/html/2608.07867#bib.bib16)\), and are therefore better regarded as weak observations of latent emotion\.

This challenge is further complicated bycross\-modal conflict, which we define as disagreement among modality\-specific predictions\. Such conflict may arise from heterogeneous sources of unreliability across modalities \(e\.g\., sensing noise, modality\-specific limitations, or insufficient evidence\)\(Bezirganyanet al\.[2025](https://arxiv.org/html/2608.07867#bib.bib73); Luet al\.[2025](https://arxiv.org/html/2608.07867#bib.bib74)\)\. However, cross\-modal conflict can also reveal complementary evidence and guide selective knowledge exchange between modalities\. Yet discrepancies between fused predictions and self\-reported labels alone cannot determine whether the inconsistency arises from unreliable modality predictions or weak labels\. MER must therefore jointly model modality reliability, uncertainty and cross\-modal conflict to determine when agreement or conflict between modalities is informative and how weak\-label supervision should be calibrated\.

Existing MER methods address modality heterogeneity through attention, gating, cross\-modal interaction, shared and modality\-specific representation learning, and expert routing, but forcibly unify conflicts between modalities through fusion weights, without explicitly determining whether one modality should influence another for the current sample\(Hanet al\.[2024](https://arxiv.org/html/2608.07867#bib.bib38); Xinet al\.[2025](https://arxiv.org/html/2608.07867#bib.bib41)\)\. Meanwhile, weak\-label methods estimate supervision reliability from confidence, loss, or global statistics while overlooking pairwise agreement and conflict among modality\-specific predictions\. In existing frameworks, modality reliability estimation, cross\-modal conflict modeling, and weak\-label calibration remain disconnected\. Consequently, information passing between modalities cannot be efficiently orchestrated or robustly learned\.

To address this gap, we proposeCONFER\(CONFlict\-AwareEvidence Negotiation forRegime\-Calibrated Weak Supervision\)\. Each modality expert produces a predictive belief and a boundary\-based uncertainty score, while its historical out\-of\-fold performance is combined with current uncertainty to estimate runtime reliability\. Together, these quantities constitute modality evidence\. Inspired by multi\-agent negotiation, CONFER represents modality experts as nodes in a dynamic graph, where pairwise compatibility and reliability\-aware directional edge weights govern iterative message passing\. A peer\-supported readout produces the final multimodal prediction, while residual conflict and uncertainty characterize three regimes—*Consensus*,*Dissent*, and*Ambiguity*—to calibrate each sample’s contribution to weak\-label training\. Thus, CONFER connects modality reliability, cross\-modal negotiation, and weak\-label calibration within a unified framework\.

We evaluate CONFER on AMIGOS, MAHNOB\-HCI, and DEAP under subject\-dependent 10\-fold and strict LOSO protocols\. CONFER achieves competitive performance, including 0\.873 accuracy/0\.879 F1 on AMIGOS\-V and 0\.854 accuracy on MAHNOB\-V under LOSO\. Further analyses show larger gains on high\-conflict samples, improved robustness under synthetic label corruption, and supervision\-reliability estimates that provide information beyond predictive confidence\.

The main contributions are summarized as follows:

- •We proposeCONFER, a unified graph\-based framework that negotiates cross\-modal conflict according to sample\-specific modality reliability and calibrates learning from self\-reported weak labels\.
- •We develop a dynamic graph mechanism that converts uncertainty\-aware compatibility into reliability\-directed asymmetric message passing, allowing cross\-modal conflict to guide both expert coordination and supervision\-reliability estimation\.
- •Systematic experiments on three MER benchmarks demonstrate that CONFER achieves competitive cross\-subject performance and improved robustness to weak\-label corruption\.

## Related Work

Multimodal emotion recognition integrates visual, acoustic, neural, and physiological signals\(Baltrušaitiset al\.[2019](https://arxiv.org/html/2608.07867#bib.bib25); Mittalet al\.[2020](https://arxiv.org/html/2608.07867#bib.bib32); Rayatdoostet al\.[2020](https://arxiv.org/html/2608.07867#bib.bib34); Zhanget al\.[2020](https://arxiv.org/html/2608.07867#bib.bib35)\)\. Representative fusion methods include TFN, MulT, MISA, Self\-MM, and MMIM\(Zadehet al\.[2017](https://arxiv.org/html/2608.07867#bib.bib27); Tsaiet al\.[2019](https://arxiv.org/html/2608.07867#bib.bib28); Hazarikaet al\.[2020](https://arxiv.org/html/2608.07867#bib.bib29); Yuet al\.[2021](https://arxiv.org/html/2608.07867#bib.bib30); Hanet al\.[2021a](https://arxiv.org/html/2608.07867#bib.bib31)\), while recent expert\-routing approaches adapt modality contributions to individual inputs\(Hanet al\.[2024](https://arxiv.org/html/2608.07867#bib.bib38); Fanget al\.[2025](https://arxiv.org/html/2608.07867#bib.bib40); Xinet al\.[2025](https://arxiv.org/html/2608.07867#bib.bib41); Gaoet al\.[2024](https://arxiv.org/html/2608.07867#bib.bib42)\)\. However, these methods mainly optimize feature aggregation or modality weighting without explicitly relating cross\-modal conflict to supervision reliability\.

Modality reliability varies with noise, missing information, and individual differences\(Liuet al\.[2024](https://arxiv.org/html/2608.07867#bib.bib37); Zenget al\.[2022](https://arxiv.org/html/2608.07867#bib.bib45)\), and has been modeled through attention, gating, confidence, uncertainty estimation, and uncertainty\-aware fusion\(Guoet al\.[2017](https://arxiv.org/html/2608.07867#bib.bib47); Kendall and Gal[2017](https://arxiv.org/html/2608.07867#bib.bib48); Sensoyet al\.[2018](https://arxiv.org/html/2608.07867#bib.bib49); Hanet al\.[2021b](https://arxiv.org/html/2608.07867#bib.bib52),[2023](https://arxiv.org/html/2608.07867#bib.bib53)\)\. Meanwhile, self\-reported emotion labels are affected by subjective perception and experimental conditions\(Mauss and Robinson[2009](https://arxiv.org/html/2608.07867#bib.bib15); Barrett[2017](https://arxiv.org/html/2608.07867#bib.bib16)\)\. Noisy\-label methods employ robust losses, reweighting, co\-training, sample selection, and label correction\(Songet al\.[2023](https://arxiv.org/html/2608.07867#bib.bib54); Zhang and Sabuncu[2018](https://arxiv.org/html/2608.07867#bib.bib56); Wanget al\.[2019](https://arxiv.org/html/2608.07867#bib.bib57); Hanet al\.[2018](https://arxiv.org/html/2608.07867#bib.bib58); Liet al\.[2020](https://arxiv.org/html/2608.07867#bib.bib60); Shuet al\.[2023](https://arxiv.org/html/2608.07867#bib.bib66); Wanget al\.[2024](https://arxiv.org/html/2608.07867#bib.bib68)\), typically estimating supervision reliability from confidence, loss, training dynamics, or representation neighborhoods\(Liuet al\.[2020](https://arxiv.org/html/2608.07867#bib.bib61); Xiaet al\.[2022](https://arxiv.org/html/2608.07867#bib.bib64); Liet al\.[2022](https://arxiv.org/html/2608.07867#bib.bib65); Huanget al\.[2023](https://arxiv.org/html/2608.07867#bib.bib67); Kimet al\.[2024](https://arxiv.org/html/2608.07867#bib.bib69)\)\. These directions rarely exploit pairwise cross\-modal relationships to jointly coordinate modality interaction and calibrate weak\-label supervision, which is the focus of CONFER\.

![Refer to caption](https://arxiv.org/html/2608.07867v1/Figures/framework.png)Figure 1:Overview of the CONFER framework\. \(a\) Modality\-specific encoders extract window\-level representations\. \(b\) Modality experts produce trial\-level beliefs and boundary\-based uncertainty scores\. \(c\) Dynamic compatibility and directional edges govern message\-passing negotiation and peer\-supported prediction readout\. \(d\) Conflict and uncertainty guide regime\-calibrated weak supervision\. \(e\) Frozen inner\-OOF statistics provide historical reliability priors\.Table 1:Modality\-wise prediction error and uncertainty statistics on AMIGOS\-V under strict\-LOSO evaluation\.ModalityMean Error↓\\downarrowError StdMeanu~m\\widetilde\{u\}\_\{m\}↓\\downarrowMedianu~m\\widetilde\{u\}\_\{m\}Audio0\.2250\.3311\.221\.00Face0\.2630\.4991\.551\.02EEG0\.3500\.4972\.100\.98Physio0\.3750\.4992\.001\.01
## Method

### Overview and Problem Formulation

Multimodal emotion recognition \(MER\) aims to infer latent emotional states from heterogeneous neural, physiological, acoustic, and behavioral observations\. Given a trialii, its multimodal input is represented as

𝒳i=\{Xim\}m∈ℳ,Xim=\{xi,km\}k=1Km\.\\begin\{array\}\[\]\{rcl\}\\mathcal\{X\}\_\{i\}&=&\\left\\\{X\_\{i\}^\{m\}\\right\\\}\_\{m\\in\\mathcal\{M\}\},\\\\\[4\.0pt\] X\_\{i\}^\{m\}&=&\\left\\\{x\_\{i,k\}^\{m\}\\right\\\}\_\{k=1\}^\{K\_\{m\}\}\.\\end\{array\}\(1\)
Here,ℳ\\mathcal\{M\}is the modality set,xi,kmx\_\{i,k\}^\{m\}is thekk\-th window of modalitymm, andKmK\_\{m\}is the number of windows\. Each trial has a self\-reported weak labelyiwy\_\{i\}^\{w\}\. We formulate MER as jointly estimating sample\-dependent modality reliability, negotiating cross\-modal conflict, and calibrating weak\-label supervision\.

As shown in Fig\.[1](https://arxiv.org/html/2608.07867#Sx2.F1), CONFER comprises modality\-expert construction, dynamic\-graph negotiation, and regime\-calibrated weak supervision\. Historical reliability and calibration statistics are estimated from frozen inner\-OOF predictions using only outer\-training subjects\. The trial indexiiis omitted when unambiguous\.

### Modality Expert Construction

A fixed modality weight assumes that modality quality remains stable across subjects and samples\. However, Table[1](https://arxiv.org/html/2608.07867#Sx2.T1)shows clear differences in prediction error and uncertainty across modalities under strict leave\-one\-subject\-out evaluation\. We therefore model each modality as an independent expert whose reliability depends on both historical performance and the current sample\.

As illustrated in Fig\.[1](https://arxiv.org/html/2608.07867#Sx2.F1)\(a\), the modality\-specific encoder maps the temporal windows of modalitymmto an embedding sequenceZm=\{zkm\}k=1KmZ^\{m\}=\\\{z\_\{k\}^\{m\}\\\}\_\{k=1\}^\{K\_\{m\}\}\. The modality expert constructor in Fig\.[1](https://arxiv.org/html/2608.07867#Sx2.F1)\(b\) then applies global mean pooling to capture persistent trial\-level responses, sparse top\-kkattention to preserve informative temporal events, and temporal dispersion to characterize variations hidden by average pooling\. Their outputs,gmeanmg\_\{\\mathrm\{mean\}\}^\{m\},gtopkmg\_\{\\mathrm\{topk\}\}^\{m\}, andgdispmg\_\{\\mathrm\{disp\}\}^\{m\}, are concatenated into the trial\-level representationgmg\_\{m\}\.

A belief head produces the initial predictive belief, from which the boundary\-based uncertainty score is computed:

μm\(0\)=σ​\(hμm​\(gm\)\),um=μm\(0\)​\(1−μm\(0\)\)\+ϵ\.\\begin\{array\}\[\]\{rcl\}\\mu\_\{m\}^\{\(0\)\}&=&\\sigma\\left\(h\_\{\\mu\}^\{m\}\(g\_\{m\}\)\\right\),\\\\\[6\.0pt\] u\_\{m\}&=&\\mu\_\{m\}^\{\(0\)\}\(1\-\\mu\_\{m\}^\{\(0\)\}\)\+\\epsilon\.\\end\{array\}\(2\)
Here,hμmh\_\{\\mu\}^\{m\}is the belief head,σ​\(⋅\)\\sigma\(\\cdot\)is the sigmoid function, andϵ\>0\\epsilon\>0is a small constant for numerical stability\. The initial predictive belief is denoted byμm\(0\)\\mu\_\{m\}^\{\(0\)\}, whileumu\_\{m\}is a boundary\-based uncertainty score whose value increases as the prediction approaches the decision boundary\.

### Conflict\-Aware Evidence Negotiation

CONFER represents modality experts as nodes in a dynamic negotiation graph\. At negotiation roundtt, the predictive beliefμm\(t\)\\mu\_\{m\}^\{\(t\)\}serves as the state of nodemm, while its boundary\-based uncertaintyumu\_\{m\}and runtime reliabilitycmc\_\{m\}characterize the reliability of that state\. Pairwise compatibilityAm​n\(t\)A\_\{mn\}^\{\(t\)\}defines the graph edges, and receiver\-specific message weightsγm←n\(t\)\\gamma\_\{m\\leftarrow n\}^\{\(t\)\}govern directional information exchange and node updates\. The negotiated node states are subsequently read out to produce the multimodal prediction, as illustrated in Fig\.[1](https://arxiv.org/html/2608.07867#Sx2.F1)\(c\)\. Reliability estimation alone cannot resolve sample\-level disagreement, because predictions from different experts should not be directly combined without considering whether their disagreement is consistent with their uncertainty\.

#### Modality\-Expert Node State and Reliability Characterization

As summarized in Fig\.[1](https://arxiv.org/html/2608.07867#Sx2.F1)\(c1\), each modality\-expert node is characterized by its predictive state, boundary\-based uncertainty, and runtime reliability\. To capture both historical modality quality and sample\-specific uncertainty, runtime reliability is estimated by combining the OOF reliability prior with current uncertainty\. Specifically,rd,mr\_\{d,m\}is estimated as the inner\-OOF balanced accuracy of modalitymmon the outer\-training subjects, wheredddenotes the dataset–affective\-dimension setting\. The median boundary\-based uncertainty obtained from the same OOF predictions is recorded asum,medu\_\{m,\\mathrm\{med\}\}\. At runtime, uncertainty is normalized asu~m=um/um,med\\widetilde\{u\}\_\{m\}=u\_\{m\}/u\_\{m,\\mathrm\{med\}\}and combined with the historical prior:

cm=rd,m​exp⁡\(−u~m\)\.c\_\{m\}=r\_\{d,m\}\\exp\\left\(\-\\widetilde\{u\}\_\{m\}\\right\)\.\(3\)
Here,cmc\_\{m\}is the runtime reliability of the expert associated with modalitymm\. It controls the expert’s outgoing contribution, while its max\-normalized formc¯m\\overline\{c\}\_\{m\}, obtained by dividingcmc\_\{m\}by the largest runtime reliability within the current trial, controls the acceptance of incoming peer beliefs\. All OOF statistics are estimated only from the outer\-training subjects and fixed before joint optimization, preventing held\-out\-subject leakage\. By construction, the median normalized uncertainty over the OOF calibration samples is one for each modality\. Together,μm\(0\)\\mu\_\{m\}^\{\(0\)\},umu\_\{m\}, andcmc\_\{m\}constitute the modality evidence used in subsequent negotiation\.

#### Pairwise Compatibility and Directional Edge Weighting

as shown in component \(c2\) of Fig\.[1](https://arxiv.org/html/2608.07867#Sx2.F1), at negotiation roundtt, a compatibility edge is constructed between each pair of modality\-expert nodesm,n∈ℳm,n\\in\\mathcal\{M\}from their uncertainty\-adjusted prediction disagreement:

Am​n\(t\)=exp⁡\[−\|μm\(t\)−μn\(t\)\|22​\(u~m\+u~n\+ϵ\)\]\.A\_\{mn\}^\{\(t\)\}=\\exp\\left\[\-\\frac\{\\left\|\\mu\_\{m\}^\{\(t\)\}\-\\mu\_\{n\}^\{\(t\)\}\\right\|^\{2\}\}\{2\\left\(\\widetilde\{u\}\_\{m\}\+\\widetilde\{u\}\_\{n\}\+\\epsilon\\right\)\}\\right\]\.\(4\)
BecauseAm​n\(t\)=An​m\(t\)A\_\{mn\}^\{\(t\)\}=A\_\{nm\}^\{\(t\)\}, it represents symmetric pairwise compatibility rather than the direction of information exchange\. Self\-edges are excluded by settingAm​m\(t\)=0A\_\{mm\}^\{\(t\)\}=0\. Compatibility decreases when two confident experts disagree, but remains relatively high when their disagreement can be explained by uncertainty\.

The symmetric compatibility scores are converted into receiver\-specific directional edge weights:

γm←n\(t\)=\(1−c¯m\)​cn​Am​n\(t\)\.\\gamma\_\{m\\leftarrow n\}^\{\(t\)\}=\\left\(1\-\\overline\{c\}\_\{m\}\\right\)c\_\{n\}A\_\{mn\}^\{\(t\)\}\.\(5\)Although compatibility is symmetric, the resulting edge weights are generally asymmetric because the sender and receiver reliabilities play different roles\.

#### Iterative Message\-Passing Negotiation

As shown in component \(c3\) of Fig\.[1](https://arxiv.org/html/2608.07867#Sx2.F1), for receiver nodemm,μ¯m\(t\)\\overline\{\\mu\}\_\{m\}^\{\(t\)\}is theγm←n\(t\)\\gamma\_\{m\\leftarrow n\}^\{\(t\)\}\-weighted average of the incoming sender states, andZm\(t\)Z\_\{m\}^\{\(t\)\}is their total weight\. We setλm\(t\)=η​Zm\(t\)/\(1\+Zm\(t\)\)\\lambda\_\{m\}^\{\(t\)\}=\\eta Z\_\{m\}^\{\(t\)\}/\(1\+Z\_\{m\}^\{\(t\)\}\), whereη∈\(0,1\]\\eta\\in\(0,1\]bounds the concession strength\. The node state is then updated as:

μm\(t\+1\)=\(1−λm\(t\)\)​μm\(t\)\+λm\(t\)​μ¯m\(t\)\.\\begin\{array\}\[\]\{rcl\}\\mu\_\{m\}^\{\(t\+1\)\}&=&\\left\(1\-\\lambda\_\{m\}^\{\(t\)\}\\right\)\\mu\_\{m\}^\{\(t\)\}\\\\\[4\.0pt\] &&\{\}\+\\lambda\_\{m\}^\{\(t\)\}\\overline\{\\mu\}\_\{m\}^\{\(t\)\}\.\\end\{array\}\(6\)WhenZm\(t\)=0Z\_\{m\}^\{\(t\)\}=0, the peer aggregation is omitted andλm\(t\)=0\\lambda\_\{m\}^\{\(t\)\}=0, so nodemmretains its current state\. Reliable senders contribute more, whereas less reliable receivers accept more information from compatible peers\. We perform two negotiation rounds,μm\(0\)→μm\(1\)→μm\(2\)\\mu\_\{m\}^\{\(0\)\}\\rightarrow\\mu\_\{m\}^\{\(1\)\}\\rightarrow\\mu\_\{m\}^\{\(2\)\}, recomputing compatibility and directional edge weights after each update\. The final graph state\{μm\(2\),um,cm,Am​n\(2\)\}\\\{\\mu\_\{m\}^\{\(2\)\},u\_\{m\},c\_\{m\},A\_\{mn\}^\{\(2\)\}\\\}is used for prediction readout and conflict\-regime characterization\.

#### Peer\-Supported Multimodal Prediction Readout

As shown in of Fig\.[1](https://arxiv.org/html/2608.07867#Sx2.F1)\(c4\), the negotiated graph state is read out to produce the final multimodal prediction\. Individual reliability cannot solely determine whether an expert is supported by the remaining modalities\. We therefore definesms\_\{m\}as the final compatibility between the expert of modalitymmand the experts of the other modalities, averaged using the corresponding peer reliabilitiescnc\_\{n\}as weights\. Its unnormalized readout weight is

w~m=cm​\[β\+\(1−β\)​sm\]\.\\widetilde\{w\}\_\{m\}=c\_\{m\}\\left\[\\beta\+\\left\(1\-\\beta\\right\)s\_\{m\}\\right\]\.\(7\)
The unnormalized readout weightsw~m\\widetilde\{w\}\_\{m\}are normalized across modalities to obtainwmw\_\{m\}, and the final negotiated prediction is

yneg=∑m∈ℳwm​μm2\.y\_\{\\mathrm\{neg\}\}=\\sum\_\{m\\in\\mathcal\{M\}\}w\_\{m\}\\mu\_\{m\}^\{2\}\.\(8\)
The peer\-support term assigns greater readout weights to nodes that are compatible with other reliable experts, whileβ∈\[0,1\]\\beta\\in\[0,1\]preserves individual reliability and prevents a reliable expert from being discarded solely because it disagrees with the majority\.

### Regime\-calibrated Weak Supervision

The conflict\-regime inference and regime\-based sample\-weighting process are illustrated in Fig\.[1](https://arxiv.org/html/2608.07867#Sx2.F1)\(d\)\.

#### Conflict Regime Inference

The negotiation outcome is characterized by conflict reduction, residual disagreement, and overall uncertainty\. At roundtt,D\(t\)D^\{\(t\)\}is thecm​cnc\_\{m\}c\_\{n\}\-weighted mean of pairwise incompatibility1−Am​n\(t\)1\-A\_\{mn\}^\{\(t\)\}over unordered node pairs\. The regime factors are then defined fromD\(0\)D^\{\(0\)\},D\(2\)D^\{\(2\)\}, and modality uncertainty:

R=max⁡\(D\(0\)−D\(2\),0\),P=D\(2\),U=1\|ℳ\|​∑m∈ℳu~m\.\\begin\{array\}\[\]\{rcl\}R&=&\\max\\left\(D^\{\(0\)\}\-D^\{\(2\)\},0\\right\),\\\\\[6\.0pt\] P&=&D^\{\(2\)\},\\\\\[6\.0pt\] U&=&\\displaystyle\\frac\{1\}\{\|\\mathcal\{M\}\|\}\\sum\_\{m\\in\\mathcal\{M\}\}\\widetilde\{u\}\_\{m\}\.\\end\{array\}\(9\)
Here,RR,PP, andUUrepresent resolved conflict, persistent conflict, and mean modality uncertainty, respectively\.

The regime factors are mapped to the Consensus, Dissent, and Ambiguity regimes through structured logits:

ℓC=aC​\(1−P\)\+bC​R−cC​U,ℓD=aD​P−bD​U,ℓA=aA​U\.\\begin\{array\}\[\]\{rcl\}\\ell\_\{C\}&=&a\_\{C\}\\left\(1\-P\\right\)\+b\_\{C\}R\-c\_\{C\}U,\\\\\[6\.0pt\] \\ell\_\{D\}&=&a\_\{D\}P\-b\_\{D\}U,\\\\\[6\.0pt\] \\ell\_\{A\}&=&a\_\{A\}U\.\\end\{array\}\(10\)
The coefficientsaCa\_\{C\},bCb\_\{C\},cCc\_\{C\},aDa\_\{D\},bDb\_\{D\}, andaAa\_\{A\}are learned during OOF calibration and constrained to be positive through a softplus parameterization\. A softmax produces the regime probabilitiesπC\\pi\_\{C\},πD\\pi\_\{D\}, andπA\\pi\_\{A\}, corresponding to Consensus, Dissent, and Ambiguity, respectively\.

#### Regime\-Based Sample Weighting

The inferred regime probabilities do not modify the self\-reported label, but determine the contribution of each trial:

ωi=πi,C\+αD​πi,D\+αA​πi,A\.\\omega\_\{i\}=\\pi\_\{i,C\}\+\\alpha\_\{D\}\\pi\_\{i,D\}\+\\alpha\_\{A\}\\pi\_\{i,A\}\.\(11\)
We setαD=0\.5\\alpha\_\{D\}=0\.5andαA=0\.2\\alpha\_\{A\}=0\.2, satisfying1\>αD\>αA\>01\>\\alpha\_\{D\}\>\\alpha\_\{A\}\>0, to reflect the supervision\-reliability ordering Consensus\>\>Dissent\>\>Ambiguity\. This design keepsωi\\omega\_\{i\}non\-zero for ambiguity samples while reducing their supervision contribution\.

The weak\-label lossℒweak\\mathcal\{L\}\_\{\\mathrm\{weak\}\}is theωi\\omega\_\{i\}\-weighted mean BCE betweenyi,negy\_\{i,\\mathrm\{neg\}\}andyiwy\_\{i\}^\{w\}\. When computing this loss,stopgrad⁡\(ωi\)\\operatorname\{stopgrad\}\(\\omega\_\{i\}\)is used to prevent the model from reducing the objective by artificially increasing uncertainty or assigning difficult samples to Ambiguity\.

Table 2:Subject\-independent LOSO results \(mean±\\pmstd across held\-out subjects\)\.MethodMetricAMIGOS\-VAMIGOS\-AMAHNOB\-VMAHNOB\-ADEAP\-VDEAP\-ALate FusionAcc0\.858±\\pm0\.1110\.612±\\pm0\.2130\.741±\\pm0\.1270\.671±\\pm0\.1640\.505±\\pm0\.1070\.577±\\pm0\.147F10\.866±\\pm0\.1010\.653±\\pm0\.2580\.705±\\pm0\.1970\.597±\\pm0\.2510\.562±\\pm0\.2400\.661±\\pm0\.229TFNAcc0\.846±\\pm0\.1130\.607±\\pm0\.1920\.778±\\pm0\.1030\.635±\\pm0\.1740\.516±\\pm0\.1090\.527±\\pm0\.137F10\.854±\\pm0\.0940\.617±\\pm0\.2460\.774±\\pm0\.1170\.570±\\pm0\.2910\.513±\\pm0\.2510\.604±\\pm0\.220Attention\-MILAcc0\.842±\\pm0\.1200\.626±\\pm0\.1970\.739±\\pm0\.0910\.622±\\pm0\.1580\.531±\\pm0\.1000\.563±\\pm0\.161F10\.850±\\pm0\.1070\.648±\\pm0\.2540\.736±\\pm0\.1350\.542±\\pm0\.2790\.569±\\pm0\.2160\.648±\\pm0\.233MISAAcc0\.829±\\pm0\.1440\.619±\\pm0\.1900\.698±\\pm0\.1060\.641±\\pm0\.1570\.520±\\pm0\.0880\.536±\\pm0\.157F10\.835±\\pm0\.1310\.641±\\pm0\.2450\.675±\\pm0\.1270\.582±\\pm0\.2710\.485±\\pm0\.2840\.627±\\pm0\.242FuseMoEAcc0\.807±\\pm0\.1550\.617±\\pm0\.1930\.749±\\pm0\.1130\.663±\\pm0\.1360\.522±\\pm0\.0970\.563±\\pm0\.150F10\.816±\\pm0\.1500\.628±\\pm0\.2480\.743±\\pm0\.1510\.658±\\pm0\.2110\.459±\\pm0\.2810\.665±\\pm0\.195EMoEAcc0\.856±\\pm0\.1160\.576±\\pm0\.1940\.728±\\pm0\.1240\.631±\\pm0\.1260\.518±\\pm0\.1090\.541±\\pm0\.153F10\.862±\\pm0\.1150\.594±\\pm0\.2540\.729±\\pm0\.1410\.563±\\pm0\.2520\.469±\\pm0\.3020\.648±\\pm0\.201I2MoEAcc0\.839±\\pm0\.1300\.602±\\pm0\.1970\.751±\\pm0\.0960\.670±\\pm0\.1610\.551±\\pm0\.1140\.550±\\pm0\.165F10\.852±\\pm0\.1150\.611±\\pm0\.2460\.741±\\pm0\.1100\.664±\\pm0\.2250\.566±\\pm0\.2310\.671±\\pm0\.186CONFER \(Ours\)Acc0\.873±\\pm0\.1220\.632±\\pm0\.1870\.854±\\pm0\.0860\.687±\\pm0\.1530\.559±\\pm0\.1040\.568±\\pm0\.133F10\.879±\\pm0\.1140\.657±\\pm0\.2260\.783±\\pm0\.1380\.679±\\pm0\.2010\.578±\\pm0\.2700\.650±\\pm0\.252

### Training and Optimization

CONFER is trained in three phases\. InPhase A, modality\-specific encoders and belief heads are trained to produce initial predictive beliefs, from which boundary\-based uncertainty scores are deterministically computed\.

InPhase B, as illustrated in Fig\.[1](https://arxiv.org/html/2608.07867#Sx2.F1)\(e\), the Phase\-A experts are retrained on the training portion of each inner split and evaluated on its held\-out subjects to obtain OOF predictions\. These predictions are used to estimaterd,mr\_\{d,m\}andum,medu\_\{m,\\mathrm\{med\}\}\. After message\-passing negotiation, the resulting graph states provide the OOF multimodal predictiony^iOOF\\hat\{y\}\_\{i\}^\{\\mathrm\{OOF\}\}through prediction readout and the regime factors used to computeωi\\omega\_\{i\}\. The calibration target is defined as

ti\\displaystyle t\_\{i\}=1−Norm⁡\(BCE⁡\(y^iOOF,yiw\)\),\\displaystyle=1\-\\operatorname\{Norm\}\\left\(\\operatorname\{BCE\}\(\\hat\{y\}^\{\\mathrm\{OOF\}\}\_\{i\},y\_\{i\}^\{w\}\)\\right\),\(12\)t~i\\displaystyle\\tilde\{t\}\_\{i\}=αA\+\(1−αA\)​ti\.\\displaystyle=\\alpha\_\{A\}\+\(1\-\\alpha\_\{A\}\)t\_\{i\}\.Here,tit\_\{i\}measures OOF prediction–label consistency rather than label correctness, andNorm⁡\(⋅\)\\operatorname\{Norm\}\(\\cdot\)denotes min–max normalization over the OOF calibration samples within each outer\-training split\. The regime coefficients are optimized by minimizing the mean squared error betweenωi\\omega\_\{i\}andt~i\\tilde\{t\}\_\{i\}\. All calibrated coefficients and OOF statistics are then fixed\.

InPhase C, the encoders, belief heads, negotiation module, and prediction\-readout module are jointly optimized using

ℒ=ℒweak\+λexpert​ℒexpert\.\\mathcal\{L\}=\\mathcal\{L\}\_\{\\mathrm\{weak\}\}\+\\lambda\_\{\\mathrm\{expert\}\}\\mathcal\{L\}\_\{\\mathrm\{expert\}\}\.\(13\)Here,ℒexpert\\mathcal\{L\}\_\{\\mathrm\{expert\}\}is the mean BCE over the pre\-negotiation modality beliefs,ℒweak\\mathcal\{L\}\_\{\\mathrm\{weak\}\}supervises the negotiated prediction usingstopgrad⁡\(ωi\)\\operatorname\{stopgrad\}\(\\omega\_\{i\}\), andλexpert≥0\\lambda\_\{\\mathrm\{expert\}\}\\geq 0balances the two objectives\.

Table 3:Dataset statistics after multimodal alignment\.DatasetSubjectsModalitiesTaskAMIGOS40Face, Audio, EEG, Physio\.V / ADEAP22Face, EEG, Physio\.V / AMAHNOB\-HCI28Face, Audio, EEG, Physio\.V / A

## Experiments and Results

### Experimental Setup

#### Datasets

We evaluate CONFER on AMIGOS\(Miranda\-Correaet al\.[2021](https://arxiv.org/html/2608.07867#bib.bib19)\), MAHNOB\-HCI\(Soleymaniet al\.[2012](https://arxiv.org/html/2608.07867#bib.bib18)\), and DEAP\(Koelstraet al\.[2012](https://arxiv.org/html/2608.07867#bib.bib17)\), which provide self\-reported valence/arousal ratings on a 1–9 Self\-Assessment Manikin scale\(Russell[1980](https://arxiv.org/html/2608.07867#bib.bib13); Bradley and Lang[1994](https://arxiv.org/html/2608.07867#bib.bib14)\)\. Table[3](https://arxiv.org/html/2608.07867#Sx3.T3)reports the subjects retained after multimodal alignment\.

#### Evaluation Protocol

We use subject\-independent LOSO as the primary protocol and additionally report subject\-dependent trial\-wise 10\-fold CV\(Kapoor and Narayanan[2023](https://arxiv.org/html/2608.07867#bib.bib71)\)\. Labels are binarized using training\-fold medians, and accuracy and F1\-Score are reported\. For each test fold or held\-out subject, predictions are averaged over three seeds; standard deviations are then computed across folds or subjects\.

#### Implementation Details

Feature and temporal modalities use gated residual MLPs and multi\-scale 1D\-CNNs, respectively, withd=64d=64and 48 windows per trial\. We setT=2T=2,η=0\.5\\eta=0\.5,β=0\.4\\beta=0\.4,αD=0\.5\\alpha\_\{D\}=0\.5,αA=0\.2\\alpha\_\{A\}=0\.2, andλexpert=0\.15\\lambda\_\{\\mathrm\{expert\}\}=0\.15\. AdamW uses a learning rate of5×10−45\\times 10^\{\-4\}, weight decay10−310^\{\-3\}, batch size 16, and cosine scheduling\. Phases A and C use 30 epochs, while Phase B performs OOF calibration\. Attention\-MIL followsIlseet al\.\([2018](https://arxiv.org/html/2608.07867#bib.bib70)\); complete configurations are provided in the supplementary material\.

Table 4:Subject\-dependent 10\-fold results \(mean±\\pmstd across test folds\)\.MethodMetricAMIGOS\-VAMIGOS\-AMAHNOB\-VMAHNOB\-ADEAP\-VDEAP\-ALate FusionAcc0\.869±\\pm0\.0420\.718±\\pm0\.0960\.790±\\pm0\.0670\.728±\\pm0\.0700\.713±\\pm0\.0420\.631±\\pm0\.044F10\.880±\\pm0\.0340\.768±\\pm0\.0840\.791±\\pm0\.0680\.711±\\pm0\.0950\.740±\\pm0\.0430\.712±\\pm0\.049TFNAcc0\.857±\\pm0\.0650\.731±\\pm0\.0680\.799±\\pm0\.0560\.726±\\pm0\.0450\.684±\\pm0\.0390\.613±\\pm0\.061F10\.863±\\pm0\.0650\.768±\\pm0\.0670\.801±\\pm0\.0540\.702±\\pm0\.0580\.684±\\pm0\.0640\.676±\\pm0\.076Attention\-MILAcc0\.845±\\pm0\.0650\.743±\\pm0\.0810\.801±\\pm0\.0690\.720±\\pm0\.0820\.683±\\pm0\.0560\.627±\\pm0\.055F10\.855±\\pm0\.0640\.791±\\pm0\.0830\.809±\\pm0\.0630\.711±\\pm0\.1010\.703±\\pm0\.0560\.707±\\pm0\.072MISAAcc0\.854±\\pm0\.0540\.745±\\pm0\.0530\.770±\\pm0\.1020\.762±\\pm0\.0750\.695±\\pm0\.0520\.602±\\pm0\.039F10\.861±\\pm0\.0570\.782±\\pm0\.0620\.760±\\pm0\.1130\.760±\\pm0\.0950\.710±\\pm0\.0540\.692±\\pm0\.042FuseMoEAcc0\.866±\\pm0\.0360\.738±\\pm0\.0670\.801±\\pm0\.0550\.742±\\pm0\.0500\.696±\\pm0\.0510\.634±\\pm0\.058F10\.875±\\pm0\.0300\.788±\\pm0\.0660\.800±\\pm0\.0610\.743±\\pm0\.0600\.718±\\pm0\.0580\.709±\\pm0\.068EMoEAcc0\.862±\\pm0\.0510\.760±\\pm0\.0680\.805±\\pm0\.0520\.728±\\pm0\.0590\.682±\\pm0\.0530\.595±\\pm0\.060F10\.874±\\pm0\.0410\.807±\\pm0\.0570\.800±\\pm0\.0550\.726±\\pm0\.0670\.708±\\pm0\.0520\.681±\\pm0\.066I2MoEAcc0\.861±\\pm0\.0680\.748±\\pm0\.0480\.772±\\pm0\.0670\.720±\\pm0\.0710\.685±\\pm0\.0540\.605±\\pm0\.068F10\.872±\\pm0\.0610\.796±\\pm0\.0350\.782±\\pm0\.0600\.714±\\pm0\.0950\.706±\\pm0\.0630\.691±\\pm0\.078CONFER \(Ours\)Acc0\.894±\\pm0\.0380\.752±\\pm0\.0680\.870±\\pm0\.0500\.779±\\pm0\.0600\.722±\\pm0\.0570\.674±\\pm0\.030F10\.891±\\pm0\.0370\.807±\\pm0\.0770\.810±\\pm0\.0770\.735±\\pm0\.1060\.757±\\pm0\.0730\.714±\\pm0\.055

### Main Results

Under 10\-fold CV \(Table[4](https://arxiv.org/html/2608.07867#Sx4.T4)\), CONFER achieves the best or competitive results on most tasks, including Accuracy/F1 of 0\.894/0\.891 on AMIGOS\-V and 0\.870/0\.810 on MAHNOB\-V\. Under LOSO \(Table[2](https://arxiv.org/html/2608.07867#Sx3.T2)\), it obtains 0\.873/0\.879 on AMIGOS\-V and 0\.854/0\.783 on MAHNOB\-V\. A paired Wilcoxon test over subject\-level accuracies confirms a significant improvement over the strongest baseline \(p<0\.05p<0\.05\)\.

Table 5:Ablation results on AMIGOS under LOSO evaluation\. Each entry reports Acc/F1\. Same Backbone Fusion shares the CONFER backbone but removes negotiation and regime calibration\.MethodValenceArousalSame Backbone Fusion0\.838/0\.8460\.600/0\.619Mean Pooling0\.844/0\.8510\.606/0\.625w/o Negotiation0\.854/0\.8600\.616/0\.638w/o Peer Support0\.852/0\.8580\.614/0\.633w/o Sender Reliability0\.860/0\.8660\.621/0\.644w/o Receiver Concession0\.863/0\.8680\.624/0\.647w/o Regime0\.849/0\.8550\.611/0\.632Static Reliability0\.842/0\.8490\.604/0\.624Full \(CONFER\)0\.873/0\.8790\.632/0\.657
### Ablation Study

Table[5](https://arxiv.org/html/2608.07867#Sx4.T5)shows that removing negotiation, sender reliability, or receiver concession consistently degrades LOSO performance, supporting reliability\-aware directional interaction\. Static Reliability underperforms CONFER, while mean pooling and the removal of peer support or regime calibration cause further reductions\.

![Refer to caption](https://arxiv.org/html/2608.07867v1/Figures/noise_robustness_clean.png)Figure 2:Robustness comparison under symmetric weak\-label corruption on AMIGOS\-V\.Table 6:Robustness under different label corruption settings on AMIGOS\-V\. Entries report Accuracy\.NoiseBackbonew/o Reg\.CONFERClean0\.8560\.8510\.894Sym\. 30%0\.6500\.6580\.723Subj\. 30%0\.6410\.6370\.686
### Robustness to Weak\-label Corruption

As shown in Fig\.[2](https://arxiv.org/html/2608.07867#Sx4.F2), all methods degrade as corruption increases, whereas CONFER maintains the most stable trend and achieves 0\.723 accuracy under 30% symmetric corruption\. Corrupted samples also exhibit higher Dissent/Ambiguity probability than clean samples \(mean softπD\+πA\\pi\_\{D\}\+\\pi\_\{A\}: 0\.61 vs\. 0\.32\), indicating that the inferred regimes capture supervision inconsistency\. We further evaluate subject\-level corruption by flipping all training labels from 30% randomly selected subjects while keeping test labels unchanged\. Although this setting is more challenging than symmetric noise, CONFER again achieves the best performance and smallest degradation \(Table[6](https://arxiv.org/html/2608.07867#Sx4.T6)\)\.

Table 7:Negotiation utility on MAHNOB\-HCI \(%\)\. Rescue/Harm denote corrected/disrupted predictions in the high\-conflict group\.TargetHigh\-conflictΔ\\DeltaAccLow\-conflictΔ\\DeltaAccRescueHarmValence\+5\.4\+1\.021\.90\.37Arousal\+8\.9\+0\.638\.42\.93
### Negotiation Utility Analysis

We rank held\-out samples according to their pre\-negotiation conflictD\(0\)D^\{\(0\)\}and define the upper and lower quartiles as the high\- and low\-conflict groups, respectively\. As shown in Table[7](https://arxiv.org/html/2608.07867#Sx4.T7), among high\-conflict samples negotiation corrects 21\.9%/38\.4% \(valence/arousal\) of wrong predictions while disrupting only 0\.37%/2\.93%, i\.e\. rescue exceeds harm by over an order of magnitude\. Accordingly, negotiation yields accuracy gains of 5\.4/8\.9 points in the high\-conflict quartile, compared with 1\.0/0\.6 points in the low\-conflict quartile, showing that its benefit concentrates on high\-conflict samples\.

Table 8:Prediction error across inferred conflict regimes on held\-out LOSO subjects\.DatasetConsensusDissentAmbiguityAMIGOS\-V0\.0710\.1370\.214MAHNOB\-V0\.0410\.1610\.200
### Prediction Reliability Across Conflict Regimes

Each held\-out sample is assigned to the regime with the highest posterior probability amongπC\\pi\_\{C\},πD\\pi\_\{D\}, andπA\\pi\_\{A\}\. Table[8](https://arxiv.org/html/2608.07867#Sx4.T8)reports the pooled prediction error within each inferred regime on held\-out LOSO subjects\. Consensus samples consistently exhibit the lowest error, whereas Dissent and Ambiguity correspond to substantially less reliable predictions\. This ordering supports the interpretation of the inferred conflict regimes as operational indicators of post\-negotiation prediction reliability\. Errors are pooled over held\-out samples after seed averaging rather than averaged equally over subjects; therefore, their aggregate values are not expected to equal1−Acc1\-\\mathrm\{Acc\}in Table[2](https://arxiv.org/html/2608.07867#Sx3.T2)\.

Table 9:Supervision inconsistency detection AUROC using predictive entropy and negotiation\-derived regime factors\.FeaturesAMIGOS\-VMAHNOB\-VHH\(Entropy\)0\.6170\.680UU0\.6460\.698\[R,P,U\]\[R,P,U\]\(LR\)0\.6620\.7141−ω1\-\\omega\(Regime weight\)0\.6530\.711\[𝐇,𝐑,𝐏,𝐔\]\\mathbf\{\[H,R,P,U\]\}\(LR\)0\.6740\.718
### Supervision\-Inconsistency Detection

We detect supervision\-inconsistent held\-out samples whose predictions disagree with their weak labels\. As shown in Table[9](https://arxiv.org/html/2608.07867#Sx4.T9), predictive entropyHHobtains AUROCs of 0\.617/0\.680, while1−ωi1\-\\omega\_\{i\}and\[R,P,U\]\[R,P,U\]reach 0\.653/0\.711 and 0\.662/0\.714, respectively\. Combining all features through a leave\-one\-fold\-out logistic detector achieves the best results of 0\.674/0\.718, improving over entropy by 0\.057/0\.038\. These results indicate that resolved conflict, residual dissent, and regime weights provide supervision\-reliability information beyond predictive confidence\.

The frozen Phase\-B calibration also remains stable after Phase\-C optimization: reliability estimates achieve Pearson/Spearman correlations of 0\.976/0\.899 with 100% dominant\-modality agreement\. The pre\-/post\-optimization correlation ofωi\\omega\_\{i\}is 0\.93, while its correlation witht~i\\tilde\{t\}\_\{i\}remains 0\.82, confirming limited drift in reliability estimation and supervision calibration\.

### Visualization of Negotiation Dynamics

Figure[3](https://arxiv.org/html/2608.07867#Sx4.F3)visualizes a high\-conflict sample in which initially inconsistent modality beliefs are progressively refined through two negotiation rounds\. The compatibility matrix and belief trajectories illustrate how CONFER selectively updates experts according to their compatibility and relative reliability\.

![Refer to caption](https://arxiv.org/html/2608.07867v1/Figures/fig_case_study_paper.png)Figure 3:CONFER negotiation on a high\-conflict sample: \(a\) belief evolution, \(b\) initial compatibility, and \(c\) conflict reduction\.

## Discussion

Unlike direct aggregation or adaptive expert routing\(Baltrušaitiset al\.[2019](https://arxiv.org/html/2608.07867#bib.bib25); Gaoet al\.[2024](https://arxiv.org/html/2608.07867#bib.bib42)\), CONFER determines both whether a modality provides reliable evidence and whether it should influence another modality for the current sample\. Importantly, negotiation does not indiscriminately minimize cross\-modal conflict or force experts toward identical predictions\. Instead, compatible beliefs from more reliable peers are selectively incorporated, with the larger gains on high\-conflict samples indicating that this interaction is most useful when modality evidence is inconsistent\. After negotiation, resolved conflict, persistent dissent, and modality uncertainty distinguish the Consensus, Dissent, and Ambiguity regimes\. Their different prediction\-error patterns and supervision\-inconsistency detection performance suggest that these regimes capture information beyond predictive confidence\. Together with the corruption results, these findings indicate that cross\-modal conflict can serve as an informative signal for both modality coordination and weak\-label reliability estimation\.

## Conclusion

This paper introduced CONFER, a graph\-based conflict\-aware framework for weakly supervised multimodal emotion recognition\. Reliability\-aware directional message passing refines modality predictions, while negotiation\-derived conflict regimes calibrate each sample’s contribution to weak\-label training\. Experiments on AMIGOS, MAHNOB\-HCI, and DEAP demonstrate competitive performance across evaluation protocols and improved robustness to weak\-label corruption\. These results show that cross\-modal conflict can serve not only as a source of prediction inconsistency, but also as an informative signal for modality coordination and supervision\-reliability assessment\.

## References

- T\. Baltrušaitis, C\. Ahuja, and L\. Morency \(2019\)Multimodal machine learning: a survey and taxonomy\.IEEE Transactions on Pattern Analysis and Machine Intelligence41\(2\),pp\. 423–443\.External Links:[Document](https://dx.doi.org/10.1109/TPAMI.2018.2798607),[Link](https://doi.org/10.1109/TPAMI.2018.2798607)Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p1.1),[Discussion](https://arxiv.org/html/2608.07867#Sx5.p1.1)\.
- L\. F\. Barrett \(2017\)The theory of constructed emotion: an active inference account of interoception and categorization\.Social Cognitive and Affective Neuroscience12\(1\),pp\. 1–23\.External Links:[Document](https://dx.doi.org/10.1093/scan/nsw154),[Link](https://doi.org/10.1093/scan/nsw154)Cited by:[Introduction](https://arxiv.org/html/2608.07867#Sx1.p1.1),[Related Work](https://arxiv.org/html/2608.07867#Sx2.p2.1)\.
- G\. Bezirganyan, S\. Sellami, L\. Berti\-Equille, and S\. Fournier \(2025\)Multimodal learning with uncertainty quantification based on discounted belief fusion\.InProceedings of the 28th International Conference on Artificial Intelligence and Statistics,Proceedings of Machine Learning Research, Vol\.258,pp\. 3142–3150\.External Links:[Link](https://proceedings.mlr.press/v258/bezirganyan25a.html)Cited by:[Introduction](https://arxiv.org/html/2608.07867#Sx1.p2.1)\.
- M\. M\. Bradley and P\. J\. Lang \(1994\)Measuring emotion: the self\-assessment manikin and the semantic differential\.Journal of Behavior Therapy and Experimental Psychiatry25\(1\),pp\. 49–59\.External Links:[Document](https://dx.doi.org/10.1016/0005-7916%2894%2990063-9),[Link](https://doi.org/10.1016/0005-7916(94)90063-9)Cited by:[Datasets](https://arxiv.org/html/2608.07867#Sx4.SSx1.SSSx1.p1.1)\.
- Y\. Fang, W\. Huang, G\. Wan, K\. Su, and M\. Ye \(2025\)EMOE: modality\-specific enhanced dynamic emotion experts\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 14314–14324\.External Links:[Link](https://openaccess.thecvf.com/content/CVPR2025/html/Fang_EMOE_Modality-Specific_Enhanced_Dynamic_Emotion_Experts_CVPR_2025_paper.html)Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p1.1)\.
- Z\. Gao, D\. Hu, X\. Jiang, H\. Lu, H\. T\. Shen, and X\. Xu \(2024\)Enhanced experts with uncertainty\-aware routing for multimodal sentiment analysis\.InProceedings of the 32nd ACM International Conference on Multimedia,pp\. 9650–9659\.External Links:[Document](https://dx.doi.org/10.1145/3664647.3680949),[Link](https://doi.org/10.1145/3664647.3680949)Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p1.1),[Discussion](https://arxiv.org/html/2608.07867#Sx5.p1.1)\.
- C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger \(2017\)On calibration of modern neural networks\.InProceedings of the 34th International Conference on Machine Learning,pp\. 1321–1330\.External Links:[Link](https://proceedings.mlr.press/v70/guo17a.html)Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p2.1)\.
- B\. Han, Q\. Yao, X\. Yu, G\. Niu, M\. Xu, W\. Hu, I\. Tsang, and M\. Sugiyama \(2018\)Co\-teaching: robust training of deep neural networks with extremely noisy labels\.InAdvances in Neural Information Processing Systems,Vol\.31\.External Links:[Link](https://proceedings.neurips.cc/paper/2018/hash/a19744e268754fb0148b017647355b7b-Abstract.html)Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p2.1)\.
- W\. Han, H\. Chen, and S\. Poria \(2021a\)Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 9180–9192\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.723),[Link](https://aclanthology.org/2021.emnlp-main.723/)Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p1.1)\.
- X\. Han, H\. Nguyen, C\. Harris, N\. Ho, and S\. Saria \(2024\)FuseMoE: mixture\-of\-experts transformers for fleximodal fusion\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/7d62a85ebfed2f680eb5544beae93191-Abstract-Conference.html)Cited by:[Introduction](https://arxiv.org/html/2608.07867#Sx1.p3.1),[Related Work](https://arxiv.org/html/2608.07867#Sx2.p1.1)\.
- Z\. Han, C\. Zhang, H\. Fu, and J\. T\. Zhou \(2021b\)Trusted multi\-view classification\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=OOsR8BzCnl5)Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p2.1)\.
- Z\. Han, C\. Zhang, H\. Fu, and J\. T\. Zhou \(2023\)Trusted multi\-view classification with dynamic evidential fusion\.IEEE Transactions on Pattern Analysis and Machine Intelligence45\(2\),pp\. 2551–2566\.External Links:[Document](https://dx.doi.org/10.1109/TPAMI.2022.3171983),[Link](https://doi.org/10.1109/TPAMI.2022.3171983)Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p2.1)\.
- D\. Hazarika, R\. Zimmermann, and S\. Poria \(2020\)MISA: modality\-invariant and \-specific representations for multimodal sentiment analysis\.InProceedings of the 28th ACM International Conference on Multimedia,pp\. 1122–1131\.External Links:[Document](https://dx.doi.org/10.1145/3394171.3413678),[Link](https://doi.org/10.1145/3394171.3413678)Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p1.1)\.
- Z\. Huang, J\. Zhang, and H\. Shan \(2023\)Twin contrastive learning with noisy labels\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 11661–11670\.External Links:[Link](https://openaccess.thecvf.com/content/CVPR2023/html/Huang_Twin_Contrastive_Learning_With_Noisy_Labels_CVPR_2023_paper.html)Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p2.1)\.
- M\. Ilse, J\. Tomczak, and M\. Welling \(2018\)Attention\-based deep multiple instance learning\.InProceedings of the 35th International Conference on Machine Learning,pp\. 2127–2136\.External Links:[Link](https://proceedings.mlr.press/v80/ilse18a.html)Cited by:[Implementation Details](https://arxiv.org/html/2608.07867#Sx4.SSx1.SSSx3.p1.9)\.
- S\. Kapoor and A\. Narayanan \(2023\)Leakage and the reproducibility crisis in machine\-learning\-based science\.Patterns4\(9\),pp\. 100804\.External Links:[Document](https://dx.doi.org/10.1016/j.patter.2023.100804),[Link](https://doi.org/10.1016/j.patter.2023.100804)Cited by:[Evaluation Protocol](https://arxiv.org/html/2608.07867#Sx4.SSx1.SSSx2.p1.1)\.
- A\. Kendall and Y\. Gal \(2017\)What uncertainties do we need in bayesian deep learning for computer vision?\.InAdvances in Neural Information Processing Systems,Vol\.30\.External Links:[Link](https://proceedings.neurips.cc/paper/2017/hash/2650d6089a6d640c5e85b2b88265dc2b-Abstract.html)Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p2.1)\.
- C\. D\. Kim, S\. Moon, J\. Moon, D\. Woo, and G\. Kim \(2024\)Sample selection via contrastive fragmentation for noisy label regression\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/e68c3a624d19154b28951e8690834607-Abstract-Conference.html)Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p2.1)\.
- S\. Koelstra, C\. Muehl, M\. Soleymani, J\. Lee, A\. Yazdani, T\. Ebrahimi, T\. Pun, A\. Nijholt, and I\. Patras \(2012\)DEAP: a database for emotion analysis using physiological signals\.IEEE Transactions on Affective Computing3\(1\),pp\. 18–31\.External Links:[Document](https://dx.doi.org/10.1109/T-AFFC.2011.15),[Link](https://doi.org/10.1109/T-AFFC.2011.15)Cited by:[Datasets](https://arxiv.org/html/2608.07867#Sx4.SSx1.SSSx1.p1.1)\.
- J\. Li, R\. Socher, and S\. C\. H\. Hoi \(2020\)DivideMix: learning with noisy labels as semi\-supervised learning\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=HJgExaVtwr)Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p2.1)\.
- S\. Li, X\. Xia, S\. Ge, and T\. Liu \(2022\)Selective\-supervised contrastive learning with noisy labels\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 316–325\.External Links:[Link](https://openaccess.thecvf.com/content/CVPR2022/html/Li_Selective-Supervised_Contrastive_Learning_With_Noisy_Labels_CVPR_2022_paper.html)Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p2.1)\.
- R\. Liu, H\. Zuo, Z\. Lian, B\. W\. Schuller, and H\. Li \(2024\)Contrastive learning based modality\-invariant feature acquisition for robust multimodal emotion recognition with missing modalities\.IEEE Transactions on Affective Computing15,pp\. 1856–1873\.External Links:[Link](https://dblp.org/rec/journals/taffco/LiuZLSL24)Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p2.1)\.
- S\. Liu, J\. Niles\-Weed, N\. Razavian, and C\. Fernandez\-Granda \(2020\)Early\-learning regularization prevents memorization of noisy labels\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 20331–20342\.External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/ea89621bee7c88b2c5be6681c8ef4906-Abstract.html)Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p2.1)\.
- J\. Lu, W\. Buntine, Y\. Qi, J\. Dipnall, B\. Gabbe, and L\. Du \(2025\)Navigating conflicting views: harnessing trust for learning\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 40411–40435\.External Links:[Link](https://proceedings.mlr.press/v267/lu25a.html)Cited by:[Introduction](https://arxiv.org/html/2608.07867#Sx1.p2.1)\.
- I\. B\. Mauss and M\. D\. Robinson \(2009\)Measures of emotion: a review\.Cognition and Emotion23\(2\),pp\. 209–237\.External Links:[Document](https://dx.doi.org/10.1080/02699930802204677),[Link](https://doi.org/10.1080/02699930802204677)Cited by:[Introduction](https://arxiv.org/html/2608.07867#Sx1.p1.1),[Related Work](https://arxiv.org/html/2608.07867#Sx2.p2.1)\.
- J\. A\. Miranda\-Correa, M\. K\. Abadi, N\. Sebe, and I\. Patras \(2021\)AMIGOS: a dataset for affect, personality and mood research on individuals and groups\.IEEE Transactions on Affective Computing12\(2\),pp\. 479–493\.External Links:[Document](https://dx.doi.org/10.1109/TAFFC.2018.2884461),[Link](https://doi.org/10.1109/TAFFC.2018.2884461)Cited by:[Datasets](https://arxiv.org/html/2608.07867#Sx4.SSx1.SSSx1.p1.1)\.
- T\. Mittal, U\. Bhattacharya, R\. Chandra, A\. Bera, and D\. Manocha \(2020\)M3ER: multiplicative multimodal emotion recognition using facial, textual, and speech cues\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.34,pp\. 1359–1367\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v34i02.5492),[Link](https://doi.org/10.1609/aaai.v34i02.5492)Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p1.1)\.
- S\. Rayatdoost, D\. Rudrauf, and M\. Soleymani \(2020\)Multimodal gated information fusion for emotion recognition from EEG signals and facial behaviors\.InProceedings of the 2020 International Conference on Multimodal Interaction,pp\. 655–659\.External Links:[Document](https://dx.doi.org/10.1145/3382507.3418864),[Link](https://doi.org/10.1145/3382507.3418864)Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p1.1)\.
- J\. A\. Russell \(1980\)A circumplex model of affect\.Journal of Personality and Social Psychology39\(6\),pp\. 1161–1178\.External Links:[Document](https://dx.doi.org/10.1037/h0077714),[Link](https://doi.org/10.1037/h0077714)Cited by:[Datasets](https://arxiv.org/html/2608.07867#Sx4.SSx1.SSSx1.p1.1)\.
- M\. Sensoy, L\. Kaplan, and M\. Kandemir \(2018\)Evidential deep learning to quantify classification uncertainty\.InAdvances in Neural Information Processing Systems,Vol\.31\.External Links:[Link](https://proceedings.neurips.cc/paper/2018/hash/a981f2b708044d6fb4a71a1463242520-Abstract.html)Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p2.1)\.
- J\. Shu, X\. Yuan, D\. Meng, and Z\. Xu \(2023\)Cmw\-net: learning a class\-aware sample weighting mapping for robust deep learning\.IEEE Transactions on Pattern Analysis and Machine Intelligence45\(10\),pp\. 11521–11539\.Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p2.1)\.
- M\. Soleymani, J\. Lichtenauer, T\. Pun, and M\. Pantic \(2012\)A multimodal database for affect recognition and implicit tagging\.IEEE Transactions on Affective Computing3\(1\),pp\. 42–55\.External Links:[Document](https://dx.doi.org/10.1109/T-AFFC.2011.25),[Link](https://doi.org/10.1109/T-AFFC.2011.25)Cited by:[Datasets](https://arxiv.org/html/2608.07867#Sx4.SSx1.SSSx1.p1.1)\.
- H\. Song, M\. Kim, D\. Park, Y\. Shin, and J\. Lee \(2023\)Learning from noisy labels with deep neural networks: a survey\.IEEE Transactions on Neural Networks and Learning Systems34\(11\),pp\. 8135–8153\.External Links:[Document](https://dx.doi.org/10.1109/TNNLS.2022.3152527),[Link](https://doi.org/10.1109/TNNLS.2022.3152527)Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p2.1)\.
- Y\. H\. Tsai, S\. Bai, P\. P\. Liang, J\. Z\. Kolter, L\. Morency, and R\. Salakhutdinov \(2019\)Multimodal transformer for unaligned multimodal language sequences\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\. 6558–6569\.External Links:[Document](https://dx.doi.org/10.18653/v1/P19-1656),[Link](https://aclanthology.org/P19-1656/)Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p1.1)\.
- H\. Wang, Z\. Huang, Z\. Lin, and T\. Liu \(2024\)NoiseGPT: label noise detection and rectification through probability curvature\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/d95cb79a3421e6d9b6c9a9008c4d07c5-Abstract-Conference.html)Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p2.1)\.
- Y\. Wang, X\. Ma, Z\. Chen, Y\. Luo, J\. Yi, and J\. Bailey \(2019\)Symmetric cross entropy for robust learning with noisy labels\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 322–330\.External Links:[Link](https://openaccess.thecvf.com/content_ICCV_2019/html/Wang_Symmetric_Cross_Entropy_for_Robust_Learning_With_Noisy_Labels_ICCV_2019_paper.html)Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p2.1)\.
- X\. Xia, T\. Liu, B\. Han, M\. Gong, J\. Yu, G\. Niu, and M\. Sugiyama \(2022\)Sample selection with uncertainty of losses for learning with noisy labels\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=xENf4QUL4LW)Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p2.1)\.
- J\. Xin, S\. Yun, J\. Peng, I\. Choi, J\. L\. Ballard, T\. Chen, and Q\. Long \(2025\)I2MoE: interpretable multimodal interaction\-aware mixture\-of\-experts\.InProceedings of the 42nd International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=EuJaF5QsMP)Cited by:[Introduction](https://arxiv.org/html/2608.07867#Sx1.p3.1),[Related Work](https://arxiv.org/html/2608.07867#Sx2.p1.1)\.
- W\. Yu, H\. Xu, Z\. Yuan, and J\. Wu \(2021\)Learning modality\-specific representations with self\-supervised multi\-task learning for multimodal sentiment analysis\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.35,pp\. 10790–10797\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v35i12.17289),[Link](https://doi.org/10.1609/aaai.v35i12.17289)Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p1.1)\.
- A\. Zadeh, M\. Chen, S\. Poria, E\. Cambria, and L\. Morency \(2017\)Tensor fusion network for multimodal sentiment analysis\.InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing,pp\. 1103–1114\.External Links:[Document](https://dx.doi.org/10.18653/v1/D17-1115),[Link](https://aclanthology.org/D17-1115/)Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p1.1)\.
- J\. Zeng, J\. Zhou, and T\. Liu \(2022\)Mitigating inconsistencies in multimodal sentiment analysis under uncertain missing modalities\.InProceedings of the 2022 conference on empirical methods in natural language processing,pp\. 2924–2934\.Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p2.1)\.
- X\. Zhang, J\. Liu, J\. Shen, S\. Li, K\. Hou, B\. Hu, J\. Gao, and T\. Zhang \(2020\)Emotion recognition from multimodal physiological signals using a regularized deep fusion of kernel machine\.IEEE transactions on cybernetics51\(9\),pp\. 4386–4399\.Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p1.1)\.
- Z\. Zhang and M\. R\. Sabuncu \(2018\)Generalized cross entropy loss for training deep neural networks with noisy labels\.InAdvances in Neural Information Processing Systems,Vol\.31\.External Links:[Link](https://proceedings.neurips.cc/paper/2018/hash/f2925f97bc13ad2852a7a551802feea0-Abstract.html)Cited by:[Related Work](https://arxiv.org/html/2608.07867#Sx2.p2.1)\.