Affect-Prototype Guided Fusion for Open-Vocabulary Incomplete Multi-modal Emotion Recognition

# Affect-Prototype Guided Fusion for Open-Vocabulary Incomplete Multi-modal Emotion Recognition
Source: [https://arxiv.org/html/2609.16962](https://arxiv.org/html/2609.16962)
Shenyue WangJing LuoChunyang Yu\\correspondingXinyu Yang\\corresponding

###### Abstract

Open\-vocabulary multimodal emotion recognition \(OV\-MER\) aims to generate open natural\-language emotion labels from multimodal affective cues\. In real\-world scenarios, however, complete and synchronized modal data are difficult to obtain due to limitations of acquisition devices and user privacy constraints\. Existing OV\-MER methods are largely designed for full\-modal inputs, and fail to perform effective feature fusion under modal missing conditions\. Meanwhile, current fusion approaches designed for incomplete modalities mainly focus on fixed\-label recognition context, and cannot satisfy the demand for fuse emotional cues guided with arbitrary emotion semantics in OV\-MER context\. To tackle these challenges, this paper proposes an Affect\-Prototype\-Conditioned Fusion \(APCF\) framework for incomplete open\-vocabulary emotion recognition\. As a candidate\-free generative framework, APCF extends modal contribution learning to scenarios guided by arbitrary emotional semantics\. Specifically, we construct an affect\-prototype library to explicitly model multimodal contribution characteristics corresponding to diverse emotions, which provides dynamic constraints for modal fusion under different emotional semantic perspectives\. Conditional retrieval and feature aggregation are conducted based on available modal features\. The refined fused affective representations are then fed into an LLM decoder to produce open\-vocabulary emotion labels\. Experiments on the OV\-MERD\+ and MER\-FG datasets demonstrate that APCF substantially outperforms state\-of\-the\-art baselines\.

1School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, China

2OPPO Research Institute, Shanghai, China

datasonezyc@stu\.xjtu\.edu\.cn,wangshenyue@whu\.edu\.cn,chinglo@xjtu\.edu\.cn,chunyang\.yu@ucalgary\.ca,yxyphd@mail\.xjtu\.edu\.cn

![Refer to caption](https://arxiv.org/html/2609.16962v1/main.v2.png)Figure 1:Overview of the proposed APCF framework\.## Introduction

Multimodal emotion recognition \(MER\) combines complementary audio, visual, and textual cues to infer a user’s affective state\. Most traditional approaches formulate MER as classification over a fixed inventory of emotion words as labels\([Zhang et al\. 2022](https://arxiv.org/html/2609.16962#bib.bib41)\)\. Although this formulation supports straightforward supervision and evaluation, a small label inventory compresses nuanced states and cannot express concepts outside the predefined classes\. Continuous emotion recognition instead predicts values in dimensional emotion labels to express arbitrary affect states\([Praveen et al\. 2022](https://arxiv.org/html/2609.16962#bib.bib42)\), however the dimension\-based labels naturally present huge semantic gap between the human\-understandable descriptions and dimensional descriptions, limiting dataset annotation and further usage on end\-user facing systems\. OV\-MER proposed predicting emotion phrases in natural language for balancing semantic coverage with human readability\([Lian et al\. 2025c](https://arxiv.org/html/2609.16962#bib.bib11)\)\. Such phrases can describe intensity, mixtures, and fine distinctions, but they also enlarge the semantic space that must be connected to sensory evidence\. AffectGPT proved multi\-modal large language models \(MLLMs\) is capable for generating fine\-grained affect description based on multi\-modal emotion evidence\([Lian et al\. 2025a](https://arxiv.org/html/2609.16962#bib.bib9)\)\. Recent work further advances this interface through perception\-oriented policy optimization\([Han et al\. 2026](https://arxiv.org/html/2609.16962#bib.bib5)\)and reinforcement learning\([Lian et al\. 2026b](https://arxiv.org/html/2609.16962#bib.bib10)\)\. Challenge\-oriented generative evaluation\([Lian et al\. 2026a](https://arxiv.org/html/2609.16962#bib.bib17)\)and hybrid\-evidential deduction\([Liu et al\. 2026](https://arxiv.org/html/2609.16962#bib.bib18)\)likewise support open\-vocabulary emotion understanding\.

Most current open\-vocabulary MER pipelines nevertheless assume that all modalities are available\. That assumption is fragile in practice: obtaining complete and synchronous modality data would be hard due to issues like sensor malfunction, data collection limitation and user privacy concerns\. User may not look into the camera, or even reject giving permissions on capturing visual data\. Transcriptions may not collect enough meaningful semantic information with heavily\-accented or mainly\-interjection speech\. Incomplete modality data will hamper learning and reasoning abilities of the language modal on the relationship between affect phrase semantic and multi\-modal emotional evidence\. Lack of crucial trained emotion features, such as micro\-expression, or the semantic information inside transcription, would impact emotion reasoning when fusing multi\-modal emotional cues, and reduce the emotion recognition performance\.

Incomplete\-MER research has addressed missing inputs through reconstruction\([Zhao et al\. 2021](https://arxiv.org/html/2609.16962#bib.bib39)\), learned prompts\([Guo et al\. 2024](https://arxiv.org/html/2609.16962#bib.bib3)\), and flexible expert routing\([Han et al\. 2024](https://arxiv.org/html/2609.16962#bib.bib4)\)\. Other approaches use cross\-modal queries\([Miyoshi et al\. 2026](https://arxiv.org/html/2609.16962#bib.bib14)\), balanced prompting\([He et al\. 2026](https://arxiv.org/html/2609.16962#bib.bib6)\), or uncertainty\-aware diffusion\([Qiu et al\. 2026](https://arxiv.org/html/2609.16962#bib.bib16)\)\. These methods improve the use of partial data for emotion understanding and reasoning\. However, current algorithm design focused on semantic understanding under fixed\-label context, not able to adapt its ability on learning emotional modality cues and fusion matrices to the arbitrary emotion space in open\-vocabulary context\. Suppose that a system is trained mainly withfearfulbut is evaluated onapprehensive\. A language model may know that the phrases are related, yet the sensory pathway may not know that they should draw on a similar mixture of hesitant speech, tense facial behavior, and uncertain wording\. As well as how to fully utilize the mixture of emotion cues to better instructing language models under incomplete multi\-modal circumstances\.

We proposeAffect\-Prototype–Conditioned Fusion \(APCF\)to address this problem\. APCF maintains a set of overlapping landmarks in emotion\-language space called affect\-prototypes, which are aligned between categorical emotion word semantic features, and label coordinates in the dimensional emotion space\. A prototype is a reusable query that conditions shared computation, it learns a range of similar emotion evidence\-wise, and a evidence fusion algorithm for emotions inside the prototype\. For each observed audio, video, and text stream, APCF extracts prototype\-specific evidence and combines the available modality streams with an observed\-set fusion module\. The resulting prototype memory is exposed to a pretrained language decoder\. Its causal prefix determines which semantic regions and which sample\-specific evidence slots are relevant, instructing the decoder to fully reason and generate open\-vocabulary emotion descriptions with incomplete modality cues\.

Our main contributions are listed as follows:

1. 1\.We extend open\-vocabulary MER to incomplete multi\-modal scenario\.This paper formulates incomplete open\-vocabulary MER as candidate\-free generation from any nonempty subset of modalities, bridging open\-vocabulary recognition and incomplete\-modality fusion\.
2. 2\.We construct semantic and modality\-aware affect\-prototypes\.Transferable prototypes are learned from paired categorical and dimensional source supervision, then adapted to target emotion phrases through bounded, regularized residuals\. They organize both affect semantics and modality\-specific evidence\-use patterns\.
3. 3\.We designed a novel modality fusion algorithm\.We use the prototypes to condition timestamped evidence retrieval, observed\-only set fusion, and causal routing into a language encoder for reasoning and generating open\-vocabulary emotional phrases\. The algorithm achieves state\-of\-the\-art \(SOTA\) on incomplete open\-vocabulary MER task\.

## Related Work

### Open\-vocabulary Emotion Recognition

OV\-MER replaces closed emotion classes with natural\-language terms and evaluates predictions through semantic normalization and an emotion\-wheel representation\([Lian et al\. 2025c](https://arxiv.org/html/2609.16962#bib.bib11)\)\. AffectGPT extends this formulation with emotion\-oriented multi\-modal instruction data and an audiovisual language\-model scaffold\([Lian et al\. 2025a](https://arxiv.org/html/2609.16962#bib.bib9)\)\. Emotion\-LLaMA uses multi\-modal instruction tuning to combine emotion recognition with explanation and reasoning\([Cheng et al\. 2024](https://arxiv.org/html/2609.16962#bib.bib2)\)\. Agent\-MER applies hierarchical agent deliberation\([Lai et al\. 2025](https://arxiv.org/html/2609.16962#bib.bib8)\), Clue2Emo organizes multimodal clues before prediction\([Zhang et al\. 2026](https://arxiv.org/html/2609.16962#bib.bib38)\), Nano\-EmoX develops a compact multitask affect model\([Huang et al\. 2026](https://arxiv.org/html/2609.16962#bib.bib22)\), and OPPO optimizes multimodal emotion reasoning with a perception\-oriented policy objective\([Han et al\. 2026](https://arxiv.org/html/2609.16962#bib.bib5)\)\. AffectGPT\-RL studies reinforcement learning for open\-vocabulary recognition\([Lian et al\. 2026b](https://arxiv.org/html/2609.16962#bib.bib10)\), HyDRA performs structured clue\-based deduction\([Liu et al\. 2026](https://arxiv.org/html/2609.16962#bib.bib18)\), AffectAgent coordinates retrieval\-augmented agents\([Wang et al\. 2026](https://arxiv.org/html/2609.16962#bib.bib23)\), and AffectVerse predicts latent audiovisual futures for affective reasoning\([Zhao et al\. 2026](https://arxiv.org/html/2609.16962#bib.bib24)\)\. Together, these methods strengthen open\-ended perception and reasoning\. Some include missing\-input mechanisms, but missingness is not uniformly studied across controlled modality subsets\.

### Emotion Recognition with Incomplete Modalities

The Missing Modality Imagination Network \(MMIN\) reconstructs unavailable representations through cross\-modal imagination and cycle consistency\([Zhao et al\. 2021](https://arxiv.org/html/2609.16962#bib.bib39)\)\. The multi\-modal prompt\-learning method of Guo et al\. associates learned prompts with missing\-input patterns\([Guo et al\. 2024](https://arxiv.org/html/2609.16962#bib.bib3)\), while the query\-based method of Miyoshi et al\. combines unimodal and cross\-modal evidence\([Miyoshi et al\. 2026](https://arxiv.org/html/2609.16962#bib.bib14)\)\. BALM balances training under unequal missing rates\([Nguyen et al\. 2026](https://arxiv.org/html/2609.16962#bib.bib15)\), and SimMLM trains over nested modality subsets so that additional observations do not degrade predictions\([Li et al\. 2025](https://arxiv.org/html/2609.16962#bib.bib12)\)\. FuseMoE and Flex\-MoE support arbitrary modality combinations through mixture\-of\-experts routing\([Han et al\. 2024](https://arxiv.org/html/2609.16962#bib.bib4);[Yun et al\. 2024](https://arxiv.org/html/2609.16962#bib.bib37)\)\. ComP propagates cross\-modal prompts and consensus information between modality branches to balance fixed\-label predictions\([He et al\. 2026](https://arxiv.org/html/2609.16962#bib.bib6)\)\. HyperEF uses hypergraph\-conditioned diffusion to recover latent missing features and models source\- and decision\-level uncertainty in fixed\-label conversational emotion recognition\([Qiu et al\. 2026](https://arxiv.org/html/2609.16962#bib.bib16)\)\. These approaches establish strong mechanisms for partial inputs, but their fusion modules are optimized primarily through fixed\-class decisions\. Applying them to a generator does not by itself provide an emotion\-semantic query that transfers evidence\-use patterns to previously unseen phrases\.

### Semantic Prototypes and Affect Representation

Semantic information has long been used to organize representations across labels and modalities\. UniBind uses language\-enhanced prototypes to construct a shared multi\-modal space\([Lyu et al\. 2024](https://arxiv.org/html/2609.16962#bib.bib13)\)\. The semantic prompting method of Pipoli et al\. conditions visual recognition on available\-modality scenarios\([Pipoli et al\. 2025](https://arxiv.org/html/2609.16962#bib.bib25)\), and the label\-semantic model of Gaonkar et al\. uses label descriptions to guide attention and relations between emotional reactions\([Gaonkar et al\. 2020](https://arxiv.org/html/2609.16962#bib.bib26)\)\. The label\-agnostic embeddings of Buechel et al\. bridge heterogeneous categorical inventories\([Buechel et al\. 2021](https://arxiv.org/html/2609.16962#bib.bib19)\)\. Park et al\. map categorical emotions to dimensional coordinates\([Park et al\. 2021](https://arxiv.org/html/2609.16962#bib.bib20)\), while the affective\-manifold method of Li et al\. anchors discrete terms in continuous affective structure\([Li et al\. 2026](https://arxiv.org/html/2609.16962#bib.bib21)\)\.

## Methodology

### Problem Formulation

Letℳ=\{A,V,T\}\\mathcal\{M\}=\\\{A,V,T\\\}denote audio, video, and text\. Sampleiicontains a nonempty observed subsetSi⊆ℳS\_\{i\}\\subseteq\\mathcal\{M\}and an availability vectoraia\_\{i\}that records which streams are present\. A frozen modality encoderEmE\_\{m\}maps each observed stream to

Him=\{\(hi,m,t,τi,m,t\)\}t=1Tim,m∈Si,H\_\{i\}^\{m\}=\\\{\(h\_\{i,m,t\},\\tau\_\{i,m,t\}\)\\\}\_\{t=1\}^\{T\_\{i\}^\{m\}\},\\qquad m\\in S\_\{i\},\(1\)wherehi,m,th\_\{i,m,t\}is an encoded token andτi,m,t\\tau\_\{i,m,t\}is its normalized position in the clip\. Every valid token follows this timestamp\-aware representation\. A naturally pooled stream is represented by a single valid token rather than expanded into an artificial temporal sequence\. If a modality is unavailable, its encoder output is not created and no zero or learned missing\-modality token is inserted into the evidence set\.

The targetYi=\(yi,1,…,yi,Ni\)Y\_\{i\}=\(y\_\{i,1\},\\ldots,y\_\{i,N\_\{i\}\}\)is an ordinary tokenizer sequence containing one or more free\-form emotion phrases\. APCF is therefore not given an emotion\-specific output vocabulary or a test\-time candidate list\. Given a task promptQiQ\_\{i\}and an evidence memoryZiSiZ\_\{i\}^\{S\_\{i\}\}built only from the observed streams, prediction is factorized causally as

p⁡\(Yi∣XiSi\)=∏np⁡\(yi,n∣yi,<n,Qi,ZiSi\)\.p\(Y\_\{i\}\\mid X\_\{i\}^\{S\_\{i\}\}\)=\\prod\_\{n\}p\(y\_\{i,n\}\\mid y\_\{i,<n\},Q\_\{i\},Z\_\{i\}^\{S\_\{i\}\}\)\.\(2\)This formulation separates two requirements\. The sensory pathway must reorganize whatever evidence remains underSiS\_\{i\}, while the language pathway must express that evidence using unrestricted emotion language\. The structure of APCF is shown in Figure[1](https://arxiv.org/html/2609.16962#S0.F1)\. APCF connects the two through affect prototypes: overlapping semantic queries that organize evidence without becoming output classes\. The model containsKKprototypes andLLreference\-time slots\. Throughout this section,W∗W\_\{\*\}denotes learned projections, and all unavailable streams are excluded from the corresponding operations\.

### Affect\-Prototype Bank

APCF represents affect using overlapping landmarks rather than mutually exclusive output classes\. The source bank is constructed only from samples for which categorical and dimensional annotations are paired\. Prototype slots are initialized from source emotion categories, but a sample may assign mass to several slots\. PrototypePkSP\_\{k\}^\{S\}contains a language anchorck,SLc\_\{k,S\}^\{L\}, a dimensional centerck,SDc\_\{k,S\}^\{D\}, and a robust bandwidthσk,S\\sigma\_\{k,S\}\. A frozen semantic encoder embeds the category description\. The dimensional center is estimated from the paired annotations, and the bandwidth summarizes within\-category dispersion\.

For source categoryyy, letgyg\_\{y\}be its normalized semantic embedding andv¯y\\bar\{v\}\_\{y\}its robust dimensional center\. Language and dimensional similarity induce two distributions:

αy,kL\\displaystyle\\alpha\_\{y,k\}^\{L\}=softmaxk⁡\(cos⁡\(gy,ck,SL\)τL\),\\displaystyle=\\operatorname\{softmax\}\_\{k\}\\left\(\\frac\{\\cos\(g\_\{y\},c\_\{k,S\}^\{L\}\)\}\{\\tau\_\{L\}\}\\right\),\(3\)αy,kD\\displaystyle\\alpha\_\{y,k\}^\{D\}=softmaxk⁡\(−∥v¯y−ck,SD∥2τD​σk,S2\)\.\\displaystyle=\\operatorname\{softmax\}\_\{k\}\\left\(\-\\frac\{\\lVert\\bar\{v\}\_\{y\}\-c\_\{k,S\}^\{D\}\\rVert^\{2\}\}\{\\tau\_\{D\}\\sigma\_\{k,S\}^\{2\}\}\\right\)\.\(4\)The temperaturesτL\\tau\_\{L\}andτD\\tau\_\{D\}control how broadly a category overlaps neighboring prototypes\. Their bridge target is

αyS=λB​αyL\+\(1−λB\)​αyD,\\alpha\_\{y\}^\{S\}=\\lambda\_\{B\}\\alpha\_\{y\}^\{L\}\+\(1\-\\lambda\_\{B\}\)\\alpha\_\{y\}^\{D\},\(5\)whereλB\\lambda\_\{B\}controls the relative contribution of language and dimensional geometry\. Because both annotations refer to the same source examples and share the same prototype slots, the bridge aligns categorical meaning with continuous affect structure instead of treating them as unrelated auxiliary tasks\.

The source bank remains immutable during target training\. This preserves a stable semantic coordinate system and prevents a small number of unusual target phrases from rewriting the source geometry\. Directly freezing that geometry, however, would assume that source categories and target open\-vocabulary descriptions organize affect identically\. APCF therefore derives a target bank through bounded residual adaptation:

ck,TL\\displaystyle c\_\{k,T\}^\{L\}=norm\(ck,SL\+ρLtanhΔkL\),\\displaystyle=\\operatorname\{norm\}\\\!\\left\(c\_\{k,S\}^\{L\}\+\\rho\_\{L\}\\tanh\\Delta\_\{k\}^\{L\}\\right\),\(6\)ck,TD\\displaystyle c\_\{k,T\}^\{D\}=clip\(ck,SD\+ρDtanhΔkD\)\.\\displaystyle=\\operatorname\{clip\}\\\!\\left\(c\_\{k,S\}^\{D\}\+\\rho\_\{D\}\\tanh\\Delta\_\{k\}^\{D\}\\right\)\.\(7\)The radiiρL\\rho\_\{L\}andρD\\rho\_\{D\}bound how far a target prototype can move, while normalization and clipping keep the two components in their valid spaces\. Source bandwidths remain fixed so that adaptation changes prototype location rather than silently changing what counts as a broad or narrow affect region\.

Target emotion phrases provide soft supervision for the language residual through their similarity to the adapted anchors\. No dimensional target is fabricated when a target corpus does not supply one\. Instead, the dimensional residual receives gradients from downstream generation and is constrained by prototype\-preservation terms\. These terms penalize excessive residual norms, distortion of pairwise source relations, reversal of the source dimensional ordering, and collapse between target prototypes\. The result is a target\-aware bank that may shift toward domain\-specific emotion language while remaining tethered to the source affect structure\.

### Source\-Domain Sensory Initialization

The prototype bank is also used to initialize the sensory pathway\. For each paired source example, the bridge distributionαyS\\alpha\_\{y\}^\{S\}supervises temporary prototype\-assignment heads, while a temporary regression head predicts the dimensional annotation\. These heads are attached to unimodal evidence and to the fused source representation\. Their joint objective can be summarized as

ℒsrc=ℒassign\+λdim​ℒdim\+λcross​ℒcross,\\mathcal\{L\}\_\{\\mathrm\{src\}\}=\\mathcal\{L\}\_\{\\mathrm\{assign\}\}\+\\lambda\_\{\\mathrm\{dim\}\}\\mathcal\{L\}\_\{\\mathrm\{dim\}\}\+\\lambda\_\{\\mathrm\{cross\}\}\\mathcal\{L\}\_\{\\mathrm\{cross\}\},\(8\)whereℒcross\\mathcal\{L\}\_\{\\mathrm\{cross\}\}encourages compatible evidence organization across observed modalities\. This stage teaches the shared evidence encoder that different affect regions may rely on different modality relations\. It is therefore more than prototype construction: source supervision initializes the perceptual extraction and fusion parameters as well\.

The temporary prediction heads are discarded after source training\. The learned evidence parameters are transferred to the target stage, whereas the source anchors remain fixed and the target residuals begin from the source geometry\. This separation prevents the source task from imposing a fixed output inventory on open\-vocabulary generation\.

### Prototype\-Conditioned Evidence Extraction

Each adapted prototype is transformed into a route codeckc\_\{k\}and a sensory\-conditioning codedkd\_\{k\}:

ck=Wr​\[ck,TL;ck,TD\],dk=Ws​\[sk;ck\],c\_\{k\}=W\_\{r\}\[c\_\{k,T\}^\{L\};c\_\{k,T\}^\{D\}\],\\qquad d\_\{k\}=W\_\{s\}\[s\_\{k\};c\_\{k\}\],\(9\)
wheresks\_\{k\}is a learned slot identifier without explicit semantic initialization\. The route code carries affect meaning to the decoder, while the sensory code conditions shared perceptual computation\. Keeping the slot identifier separate from the semantic code allows the architecture to preserve stable memory positions without equating a prototype with an output class\.

For reference positionτ¯ℓ\\bar\{\\tau\}\_\{\\ell\}, the pair\(k,ℓ\)\(k,\\ell\)forms a query that retrieves evidence from each observed modality:

qk,ℓ\\displaystyle q\_\{k,\\ell\}=pℓ\+Wq​dk,\\displaystyle=p\_\{\\ell\}\+W\_\{q\}d\_\{k\},\(10\)ei,k,m,ℓ\\displaystyle e\_\{i,k,m,\\ell\}=Attn⁡\(qk,ℓ,Him,bm​\(τi,m−τ¯ℓ\)\)\.\\displaystyle=\\operatorname\{Attn\}\\\!\\left\(q\_\{k,\\ell\},H\_\{i\}^\{m\};b\_\{m\}\(\\tau\_\{i,m\}\-\\bar\{\\tau\}\_\{\\ell\}\)\\right\)\.\(11\)Herepℓp\_\{\\ell\}identifies a learned reference\-time slot andbmb\_\{m\}is a modality\-specific relative\-time bias\. Timestamp embeddings are also added to the attention values\. Consequently, the same sensory tokens can yield different evidence when queried from different affect regions or temporal positions\. For example, a short vocal hesitation may be highly relevant to one prototype but weak evidence for another\.

All prototypes and modalities share the retrieval parameters; only their conditioning codes and modality embeddings differ\. This parameter sharing allows a target phrase near a source prototype to reuse computation rather than requiring a newly trained detector\. Since retrieval is evaluated only form∈Sim\\in S\_\{i\}, the evidence tensor contains no representation derived from an unavailable channel\.

### Observed\-Only Set Fusion

Retrieved evidence must be combined without assuming a fixed number or order of observed streams\. Letηm\\eta\_\{m\}be a modality identity embedding\. APCF first applies prototype\-conditioned FiLM\([Perez et al\. 2018](https://arxiv.org/html/2609.16962#bib.bib27)\), then uses a shared Set Transformer\([Lee et al\. 2019](https://arxiv.org/html/2609.16962#bib.bib28)\):

xi,k,m,ℓ\\displaystyle x\_\{i,k,m,\\ell\}=FiLM⁡\(LN⁡\(We​e\+ηm\+Wa​ai\);dk\),\\displaystyle=\\operatorname\{FiLM\}\\\!\\left\(\\operatorname\{LN\}\(W\_\{e\}e\+\\eta\_\{m\}\+W\_\{a\}a\_\{i\}\);d\_\{k\}\\right\),\(12\)ri,k,ℓ\\displaystyle r\_\{i,k,\\ell\}=Pooldk,ai⁡\(SetAttn⁡\{xi,k,m,ℓ:m∈Si\}\)\.\\displaystyle=\\operatorname\{Pool\}\_\{d\_\{k\},a\_\{i\}\}\\left\(\\operatorname\{SetAttn\}\\\{x\_\{i,k,m,\\ell\}:m\\in S\_\{i\}\\\}\\right\)\.\(13\)The availability encodingaia\_\{i\}tells the shared module which observation state produced the set, while the modality identities distinguish evidence sources\. FiLM lets the same sensory content be emphasized differently under different affect prototypes\. Set attention then models relations among only the available streams, and prototype\-conditioned pooling aggregates their contributions intori,k,ℓr\_\{i,k,\\ell\}\.

This operation is permutation invariant with respect to the presentation order of modalities and accepts every nonempty observed subset\. Missingness is not represented by a synthetic feature, and the model is not required to reconstruct a plausible but unverifiable hidden stream\. Instead, it learns how the contribution of an observed stream changes with both the affect query and the current observation state\.

The fused slots form a prototype\-by\-time memory

zi,k,ℓ=Wg​ri,k,ℓ\+ηkP\+ηℓR,z\_\{i,k,\\ell\}=W\_\{g\}r\_\{i,k,\\ell\}\+\\eta\_\{k\}^\{P\}\+\\eta\_\{\\ell\}^\{R\},\(14\)whereηkP\\eta\_\{k\}^\{P\}andηℓR\\eta\_\{\\ell\}^\{R\}are semantic\-free prototype and reference\-slot identifiers\. Semantic\-free identifiers preserve memory structure; affect meaning enters through the prototype codes that conditioned retrieval and fusion\.

Figure 2:Structure of affect\-prototype causal routing\.
### Affect Prototype Causal Routing

The structure of this module is detailed in Figure[2](https://arxiv.org/html/2609.16962#Sx3.F2)\. At a routed decoder layer, the causal statehi,n−1h\_\{i,n\-1\}contains only the prompt and previously generated tokens\. Its projectionqi,nq\_\{i,n\}produces a semantic priorβ\\betaover prototypes and content scoresssover the sample\-specific memory:

βi,n,k\\displaystyle\\beta\_\{i,n,k\}=softmaxk⁡\(cos⁡\(qi,n,WP​ck\)τG\),\\displaystyle=\\operatorname\{softmax\}\_\{k\}\\left\(\\frac\{\\cos\(q\_\{i,n\},W\_\{P\}c\_\{k\}\)\}\{\\tau\_\{G\}\}\\right\),\(15\)si,n,k,ℓ\\displaystyle s\_\{i,n,k,\\ell\}=qi,n⊤​WK​zi,k,ℓdh,\\displaystyle=\\frac\{q\_\{i,n\}^\{\\top\}W\_\{K\}z\_\{i,k,\\ell\}\}\{\\sqrt\{d\_\{h\}\}\},\(16\)ai,n,k,ℓ\\displaystyle a\_\{i,n,k,\\ell\}=softmaxk,ℓ⁡\(si,n,k,ℓ\+log⁡\(βi,n,k\+ϵ\)\)\.\\displaystyle=\\operatorname\{softmax\}\_\{k,\\ell\}\\left\(s\_\{i,n,k,\\ell\}\+\\log\(\\beta\_\{i,n,k\}\+\\epsilon\)\\right\)\.\(17\)
HereτG\\tau\_\{G\}is the routing temperature,dhd\_\{h\}is the query dimension, andϵ\\epsilonis a numerical stabilizer\. The semantic prior asks which affect regions are compatible with the current generation prefix; the content score asks which prototype\-time slots contain relevant evidence for this particular sample\. Their sum yields one joint distributionai,n,k,ℓa\_\{i,n,k,\\ell\}, so prototype choice and evidence choice cannot drift into independent decisions\.

The routed evidence is

vi,n\\displaystyle v\_\{i,n\}=∑k,ℓai,n,k,ℓ​WV​zi,k,ℓ,\\displaystyle=\\sum\_\{k,\\ell\}a\_\{i,n,k,\\ell\}W\_\{V\}z\_\{i,k,\\ell\},\(18\)h~i,n\\displaystyle\\widetilde\{h\}\_\{i,n\}=hi,n\+g​WO​vi,n,\\displaystyle=h\_\{i,n\}\+g\\,W\_\{O\}v\_\{i,n\},\(19\)
whereggis a zero\-initialized residual gate\. The value and output paths follow an identity\-preserving factorization: learned transformations reconcile dimensions, while the prototype router controls selection rather than freely rewriting memory content\. At initialization, the gate makes the augmented decoder equivalent to the frozen decoder\. Training can then introduce evidence gradually without destabilizing its language ability\.

Routing is strictly causal\. A completed target phrase is never encoded and returned to the router during its own generation\. At stepnn, both the semantic prior and the content route depend only onQiQ\_\{i\},yi,<ny\_\{i,<n\}, and observed evidence\. This prevents label leakage and permits ordinary autoregressive inference\.

### Training Process

Training proceeds from source prototype construction and sensory initialization to target\-domain adaptation and candidate\-free generation\. The modality encoders and pretrained language decoder remain frozen; optimization updates the evidence modules, observed\-set fusion, target prototype residuals, and causal router\. At inference, target\-phrase assignments, temporary source heads, and complete\-view references are absent\. APCF encodes only the observed streams, builds their prototype\-conditioned memory, and generates emotion terms autoregressively\.

Table 1:Open\-vocabulary comparison on OV\-MERD\+ and MER\-FG\. Cells report \(%\): level\-1 F1 on OV\-MERD\+ and Avg on MER\-FG\.Table 2:Comparison with missing\-modality methods adapted to open\-vocabulary emotion generation\. Cells report the dataset\-native score \(%\): level\-1 F1 on OV\-MERD\+ and Avg on MER\-FG\.

## Experiments

We evaluate released open\-vocabulary systems, controlled fusion methods, APCF component variants, and the learned prototype space, for proving our proposed framework can achieve effective emotion modelling and recognition within incomplete modality context\.

### Datasets and Evaluation Metrics

#### Data and views\.

IEMOCAP\([Busso et al\. 2008](https://arxiv.org/html/2609.16962#bib.bib1)\)supplies paired categorical and dimensional source supervision\. All 31,327 MER\-Caption\+ examples\([Lian et al\. 2025a](https://arxiv.org/html/2609.16962#bib.bib9)\)are used for target training, and evaluation uses all 532 OV\-MERD\+ examples\([Lian et al\. 2025a](https://arxiv.org/html/2609.16962#bib.bib9)\)and all 1,200 MER\-FG examples\([Lian et al\. 2025b](https://arxiv.org/html/2609.16962#bib.bib31)\)\. A fixed non\-emotional sentence replaces unavailable native transcripts for 22 and five examples, respectively; these rows are not counted as controlled missing\-text cases\. Controlled views are instead produced by deleting complete streams\. Every sample is evaluated under\{A,V,T,A​V,A​T,V​T,A​V​T\}\\\{A,V,T,AV,AT,VT,AVT\\\}modality combinations\.

#### Metrics\.

For both datasets, normalized terms are mapped through the same five emotion wheels\.S1S\_\{1\}andS2S\_\{2\}are the mean F\-scores over their coarse and fine mappings, andAvg=\(S1\+S2\)/2\\mathrm\{Avg\}=\(S\_\{1\}\+S\_\{2\}\)/2is the primary metric\. Avg is the native MER\-FG ranking score and for OV\-MERD\+, its native score \(S1S\_\{1\}F\-score\) is used\. All table entries are percentages;S1S\_\{1\}andS2S\_\{2\}remain available in the result artifacts\. A frozen parser handles raw generations, and malformed or empty outputs remain failures\.

### Experimental Setup

Audio, video, and text are encoded by frozen Chinese\-HuBERT\-large\([Hsu et al\. 2021](https://arxiv.org/html/2609.16962#bib.bib7)\), CLIP ViT\-L/14\([Radford et al\. 2021](https://arxiv.org/html/2609.16962#bib.bib32)\), and all\-mpnet\-base\-v2\([Song et al\. 2020](https://arxiv.org/html/2609.16962#bib.bib33)\), respectively\. The decoder is Qwen2\.5\-7B\-Instruct\([Yang et al\. 2024](https://arxiv.org/html/2609.16962#bib.bib29)\)with the released AffectGPT MER\-Caption\+ LoRA\([Lian et al\. 2025a](https://arxiv.org/html/2609.16962#bib.bib9);[Hu et al\. 2022](https://arxiv.org/html/2609.16962#bib.bib30)\)merged and frozen\. APCF uses eight prototypes and eight temporal slots, constructed from IEMOCAP dataset, a 384\-dimensional evidence space, and a router at decoder layer 13\. Generation is deterministic with at most 64 new tokens\. All experiments are conducted on RTX 3090 GPUs\.

Source initialization precedes eight epochs of target training\. We employ a hybrid Muon–AdamW optimizer: Muon updates eligible two\-dimensional hidden matrices with learning rate10−310^\{\-3\}, momentum0\.950\.95, Nesterov momentum, and five Newton–Schulz iterations, while AdamW\([Loshchilov and Hutter 2019](https://arxiv.org/html/2609.16962#bib.bib34)\)updates auxiliary parameters at5×10−55\\times 10^\{\-5\}and prototype residuals at10−510^\{\-5\}\. All parameter groups use weight decay0\.010\.01\. Learning rates follow a cosine schedule with 500 warm\-up steps and decay to0\.10\.1of their initial values; the global gradient norm is clipped at1\.01\.0\. Controlled methods share the prompt, decoder, parser, and scorer; released systems retain their native front ends and prompting\.

### Comparison with Open\-Vocabulary Systems

OV\-MER\([Lian et al\. 2025c](https://arxiv.org/html/2609.16962#bib.bib11)\)uses its released acoustic and visual clue checkpoints with a view\-aware merger\. Emotion\-LLaMA\([Cheng et al\. 2024](https://arxiv.org/html/2609.16962#bib.bib2)\)is evaluated zero\-shot from its released checkpoint, and AffectGPT\([Lian et al\. 2025a](https://arxiv.org/html/2609.16962#bib.bib9)\)retains its released multimodal modules without additional adaptation\. AffectAgent\([Wang et al\. 2026](https://arxiv.org/html/2609.16962#bib.bib23)\)proposed module for incomplete modal fusion and was evaluated on open\-vocabulary tasks, however, as it provides neither executable code nor a checkpoint,*AffectAgent\-R*denotes our paper\-guided implementation of its reported agent, retrieval, fusion, and optimization components\.

Table[1](https://arxiv.org/html/2609.16962#Sx3.T1)shows that APCF ranks first under every modality view on both datasets\. It gains 8\.264 points forAA, 3\.316 forA​VAV, and 2\.913 forA​TATon OV\-MERD\+ dataset, and 5\.987, 4\.879, and 5\.348 points forAA,A​VAV, andA​TATon MER\-FG dataset, respectively\. Text\-rich views are already strong for language\-decoder\-based systems because the transcript is close to the generated output space, and APCF’s larger gains onAA,A​VAV, andA​TATprovide more direct evidence that it can organize the surviving non\-textual cues\.

### Comparison with Incomplete\-Modality Fusion Methods

Under the same target data, modality encoders, language decoder, prompt, evidence budget, and scorer, we adapt MulT\([Tsai et al\. 2019](https://arxiv.org/html/2609.16962#bib.bib36)\), MMIN\([Zhao et al\. 2021](https://arxiv.org/html/2609.16962#bib.bib39)\), MPLMM\([Guo et al\. 2024](https://arxiv.org/html/2609.16962#bib.bib3)\), ComP\([He et al\. 2026](https://arxiv.org/html/2609.16962#bib.bib6)\), and BALM\-FCM\([Nguyen et al\. 2026](https://arxiv.org/html/2609.16962#bib.bib15)\)to open\-vocabulary generation\.

#### Open\-vocabulary adaptation\.

For MulT, MMIN, MPLMM, and ComP, we remove the native fixed\-label prediction head and pass the fused hidden sequenceinro the frozen language decoder\. For BALM specifically, we insert its Feature Calibration Module \(FCM\), which can be transferred without altering the task\. Gradient Rebalancing Module is not included because its design depends hevaily on fixed\-label classification\.

As shown in Table[2](https://arxiv.org/html/2609.16962#Sx3.T2), APCF leads on open\-vocabulary emotion phrases generation, attributing its advantage more directly to incomplete\-evidence organization under a common generator\. The shared language decoder makes text\-containing views a necessary but insufficient test of fusion\. For MulT, MMIN, and ComP, their text\-absent profiles are substantially weaker or unstable, particularly onVVandA​VAV\. Adding audio or video therefore contributes little once text is present; the apparently competitive text\-view results mainly reflect transcript processing by the common language backbone\. This also explains why comparing onlyA​V​TAVTwould substantially overestimate the missing\-modality capability of these methods\.

### Ablation Studies

Table[3](https://arxiv.org/html/2609.16962#Sx4.T3)evaluates missing\-view learning \(MVL\), source prototype transfer \(SPT\), prototype\-conditioned sensory fusion \(PSF\), target prototype adaptation \(TPA\), and semantic causal routing \(SCR\)\. The upper block adds these components to a common base in order, while the lower block removes one component from full APCF\. All variants retain the same backbone, dimensions, data, and evaluation protocol\.

Table 3:APCF component ablation\. Filled and open circles denote enabled and disabled components\. Values are the average over all seven modality combinations \(All7, %\), using level\-1 F1 on OV\-MERD\+ and Avg on MER\-FG\. Best results are bold\.Tested componentsAll7 \(%\)MVLSPTPSFTPASCROV\-MERD\+MER\-FG∘\\circ∘\\circ∘\\circ∘\\circ∘\\circ48\.3836\.87∙\\bullet∘\\circ∘\\circ∘\\circ∘\\circ53\.8740\.36∙\\bullet∙\\bullet∘\\circ∘\\circ∘\\circ53\.4240\.93∙\\bullet∙\\bullet∙\\bullet∘\\circ∘\\circ54\.1941\.05∙\\bullet∙\\bullet∙\\bullet∙\\bullet∘\\circ55\.8241\.08∘\\circ∙\\bullet∙\\bullet∙\\bullet∙\\bullet47\.6937\.36∙\\bullet∘\\circ∙\\bullet∙\\bullet∙\\bullet55\.0542\.18∙\\bullet∙\\bullet∘\\circ∙\\bullet∙\\bullet53\.9441\.96∙\\bullet∙\\bullet∙\\bullet∘\\circ∙\\bullet53\.8941\.84∙\\bullet∙\\bullet∙\\bullet∙\\bullet∙\\bullet57\.9042\.61MVL produces the largest cumulative improvement, adding 5\.49 points on OV\-MERD\+ and 3\.49 on MER\-FG; removing it from full APCF reduces performance by 10\.21 and 5\.25 points, respectively\. The prototype components are also beneficial in the full system\. Removing SPT, PSF, or TPA costs 2\.85/0\.43, 3\.96/0\.65, and 4\.01/0\.77 points on OV\-MERD\+/MER\-FG, while adding SCR to the otherwise complete model contributes 2\.08/1\.53 points\.

### Affect\-Prototype Adaptation Analysis

Figure[3](https://arxiv.org/html/2609.16962#Sx4.F3)analyzes the two effects of target adaptation that are central to our prototype design\. The bounded bank closely preserves the IEMOCAP pairwise anchor geometry \(ρ=\.952\\rho=\.952\), whereas unconstrained adaptation largely destroys it \(ρ=\.095\\rho=\.095\)\. On 1,024 held\-out NRC\-VAD concepts\([Mohammad 2018](https://arxiv.org/html/2609.16962#bib.bib35)\), bounded adaptation also raises CCC\([Lin 1989](https://arxiv.org/html/2609.16962#bib.bib40)\)from \.254/\.062/\.086 to \.305/\.083/\.111 for valence/arousal/dominance\. These results show that bounded adaptation improves target\-domain affect alignment while retaining the source structure\.

\(a\)Pairwise source and adapted anchor distances\.\(b\)Axis\-wise CCC on held\-out emotion concepts\.
Figure 3:Affect\-prototype adaptation diagnostics averaged over target\-training seeds 64, 65, and 71\.

## Conclusion

This paper presents APCF, a candidate\-free framework for open\-vocabulary emotion recognition when any subset of audio, video, and text may be unavailable\. APCF transfers a categorical–dimensional affect geometry from existing datasets, adapts it to target emotion phrases, uses the resulting prototypes to query and fuse only observed evidence, and routes that evidence causally into a frozen language decoder\. In this way, emotion semantics influence the sensory fusion process rather than serving only as output labels\. Comparisons on OV\-MERD\+ and MER\-FG datasets with released open\-vocabulary systems and controlled incomplete\-modality pipelines show that APCF improves robustness across missing\-input views while preserving competitive AVT performance\.

## References

- Buechelet al\.\(2021\)S\. Buechel, L\. Modersohn, and U\. HahnTowards label\-agnostic emotion embeddings\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 9231–9249\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.728)Cited by:[Semantic Prototypes and Affect Representation](https://arxiv.org/html/2609.16962#Sx2.SSx3.p1.1)\.
- Bussoet al\.\(2008\)C\. Busso, M\. Bulut, C\. Lee, A\. Kazemzadeh, E\. Mower, S\. Kim, J\. N\. Chang, S\. Lee, and S\. S\. NarayananIEMOCAP: interactive emotional dyadic motion capture database\.Language Resources and Evaluation42\(4\),pp\. 335–359\.External Links:ISSN 1574\-0218,[Document](https://dx.doi.org/10.1007/s10579-008-9076-6)Cited by:[Data and views\.](https://arxiv.org/html/2609.16962#Sx4.SSx1.SSS0.Px1.p1.1)\.
- Chenget al\.\(2024\)Z\. Cheng, Z\. Cheng, J\. He, J\. Sun, K\. Wang, Y\. Lin, Z\. Lian, X\. Peng, and A\. G\. HauptmannEmotion\-LLaMA: multimodal emotion recognition and reasoning with instruction tuning\.InProceedings of the 38th International Conference on Neural Information Processing Systems,NIPS ’24, Vol\.37,Red Hook, NY, USA,pp\. 110805–110853\.External Links:[Document](https://dx.doi.org/10.52202/079017-3518),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/c7f43ada17acc234f568dc66da527418-Abstract-Conference.html),ISBN 979\-8\-3313\-1438\-5Cited by:[Open\-vocabulary Emotion Recognition](https://arxiv.org/html/2609.16962#Sx2.SSx1.p1.1),[Table 1](https://arxiv.org/html/2609.16962#Sx3.T1.1.4.1.1.1),[Table 1](https://arxiv.org/html/2609.16962#Sx3.T1.1.9.1.1.1),[Comparison with Open\-Vocabulary Systems](https://arxiv.org/html/2609.16962#Sx4.SSx3.p1.1)\.
- Gaonkaret al\.\(2020\)R\. Gaonkar, H\. Kwon, M\. Bastan, N\. Balasubramanian, and N\. ChambersModeling label semantics for predicting emotional reactions\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 4687–4692\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.426)Cited by:[Semantic Prototypes and Affect Representation](https://arxiv.org/html/2609.16962#Sx2.SSx3.p1.1)\.
- Guoet al\.\(2024\)Z\. Guo, T\. Jin, and Z\. ZhaoMultimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 1726–1736\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.94)Cited by:[Introduction](https://arxiv.org/html/2609.16962#Sx1.p3.1),[Emotion Recognition with Incomplete Modalities](https://arxiv.org/html/2609.16962#Sx2.SSx2.p1.1),[Table 2](https://arxiv.org/html/2609.16962#Sx3.T2.1.11.1.1.1),[Table 2](https://arxiv.org/html/2609.16962#Sx3.T2.1.5.1.1.1),[Comparison with Incomplete\-Modality Fusion Methods](https://arxiv.org/html/2609.16962#Sx4.SSx4.p1.1)\.
- Hanet al\.\(2024\)X\. Han, H\. Nguyen, C\. Harris, N\. Ho, and S\. SariaFuseMoE: Mixture\-of\-Experts Transformers for Fleximodal Fusion\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 67850–67900\.External Links:[Document](https://dx.doi.org/10.52202/079017-2167)Cited by:[Introduction](https://arxiv.org/html/2609.16962#Sx1.p3.1),[Emotion Recognition with Incomplete Modalities](https://arxiv.org/html/2609.16962#Sx2.SSx2.p1.1)\.
- Hanet al\.\(2026\)Z\. Han, B\. Zhu, W\. Tong, P\. Shao, P\. Song, X\. Wang, J\. Chen, L\. Lu, and X\. YangOmni\-Perception Policy Optimization for Multimodal Emotion Reasoning\.InProceedings of the 43rd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.306,Seoul, South Korea\.External Links:2606\.25325Cited by:[Introduction](https://arxiv.org/html/2609.16962#Sx1.p1.1),[Open\-vocabulary Emotion Recognition](https://arxiv.org/html/2609.16962#Sx2.SSx1.p1.1)\.
- Heet al\.\(2026\)W\. He, X\. Zhu, and Z\. ZhangCross\-modal Prompting for Balanced Incomplete Multi\-modal Emotion Recognition\.Proceedings of the AAAI Conference on Artificial Intelligence40\(21\),pp\. 17463–17471\.External Links:ISSN 2374\-3468,[Document](https://dx.doi.org/10.1609/aaai.v40i21.38800)Cited by:[Introduction](https://arxiv.org/html/2609.16962#Sx1.p3.1),[Emotion Recognition with Incomplete Modalities](https://arxiv.org/html/2609.16962#Sx2.SSx2.p1.1),[Table 2](https://arxiv.org/html/2609.16962#Sx3.T2.1.12.1.1.1),[Table 2](https://arxiv.org/html/2609.16962#Sx3.T2.1.6.1.1.1),[Comparison with Incomplete\-Modality Fusion Methods](https://arxiv.org/html/2609.16962#Sx4.SSx4.p1.1)\.
- Hsuet al\.\(2021\)W\. Hsu, B\. Bolte, Y\. H\. Tsai, K\. Lakhotia, R\. Salakhutdinov, and A\. MohamedHuBERT: Self\-Supervised Speech Representation Learning by Masked Prediction of Hidden Units\.IEEE/ACM Transactions on Audio, Speech, and Language Processing29,pp\. 3451–3460\.External Links:[Document](https://dx.doi.org/10.1109/TASLP.2021.3122291)Cited by:[Experimental Setup](https://arxiv.org/html/2609.16962#Sx4.SSx2.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: Low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:2106\.09685Cited by:[Experimental Setup](https://arxiv.org/html/2609.16962#Sx4.SSx2.p1.1)\.
- Huanget al\.\(2026\)J\. Huang, F\. Lin, X\. Yang, C\. Feng, K\. Zhu, X\. Yang, and Z\. ChenNano\-EmoX: Unifying multimodal emotional intelligence from perception to empathy\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 22986–22997\.Cited by:[Open\-vocabulary Emotion Recognition](https://arxiv.org/html/2609.16962#Sx2.SSx1.p1.1)\.
- Laiet al\.\(2025\)Z\. Lai, Z\. Zhu, X\. Hong, and Y\. WangAgent\-MER: A Cognitive Agent with Hierarchical Deliberation for Open\-Vocabulary Multimodal Emotion Recognition\.InProceedings of the 33rd ACM International Conference on Multimedia,MM ’25,New York, NY, USA,pp\. 13864–13871\.External Links:[Document](https://dx.doi.org/10.1145/3746027.3762011),ISBN 979\-8\-4007\-2035\-2Cited by:[Open\-vocabulary Emotion Recognition](https://arxiv.org/html/2609.16962#Sx2.SSx1.p1.1)\.
- Leeet al\.\(2019\)J\. Lee, Y\. Lee, J\. Kim, A\. Kosiorek, S\. Choi, and Y\. W\. TehSet transformer: a framework for attention\-based permutation\-invariant neural networks\.InProceedings of the 36th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.97,pp\. 3744–3753\.Cited by:[Observed\-Only Set Fusion](https://arxiv.org/html/2609.16962#Sx3.SSx5.p1.1)\.
- Liet al\.\(2025\)S\. Li, C\. Chen, and J\. HanSimMLM: A Simple Framework for Multi\-modal Learning with Missing Modality\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 24068–24077\.External Links:2507\.19264Cited by:[Emotion Recognition with Incomplete Modalities](https://arxiv.org/html/2609.16962#Sx2.SSx2.p1.1)\.
- Liet al\.\(2026\)W\. Li, J\. Cheng, X\. Tang, and C\. M\. VongAnchoring the affective manifold: Learning canonical and disentangled representations via generative cross\-modal alignment\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 41605–41614\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.1929)Cited by:[Semantic Prototypes and Affect Representation](https://arxiv.org/html/2609.16962#Sx2.SSx3.p1.1)\.
- Lianet al\.\(2025a\)Z\. Lian, H\. Chen, L\. Chen, H\. Sun, L\. Sun, Y\. Ren, Z\. Cheng, B\. Liu, R\. Liu, X\. Peng, J\. Yi, and J\. TaoAffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 36993–37014\.External Links:[Link](https://proceedings.mlr.press/v267/lian25a.html)Cited by:[Introduction](https://arxiv.org/html/2609.16962#Sx1.p1.1),[Open\-vocabulary Emotion Recognition](https://arxiv.org/html/2609.16962#Sx2.SSx1.p1.1),[Table 1](https://arxiv.org/html/2609.16962#Sx3.T1.1.10.1.1.1),[Table 1](https://arxiv.org/html/2609.16962#Sx3.T1.1.5.1.1.1),[Data and views\.](https://arxiv.org/html/2609.16962#Sx4.SSx1.SSS0.Px1.p1.1),[Experimental Setup](https://arxiv.org/html/2609.16962#Sx4.SSx2.p1.1),[Comparison with Open\-Vocabulary Systems](https://arxiv.org/html/2609.16962#Sx4.SSx3.p1.1)\.
- Lianet al\.\(2025b\)Z\. Lian, R\. Liu, K\. Xu, B\. Liu, X\. Liu, Y\. Zhang, X\. Liu, Y\. Li, Z\. Cheng, H\. Zuo, Z\. Ma, X\. Peng, X\. Chen, Y\. Li, E\. Cambria, G\. Zhao, B\. W\. Schuller, and J\. TaoMER 2025: When affective computing meets large language models\.InProceedings of the 33rd ACM International Conference on Multimedia,pp\. 13837–13842\.External Links:[Document](https://dx.doi.org/10.1145/3746027.3762007)Cited by:[Data and views\.](https://arxiv.org/html/2609.16962#Sx4.SSx1.SSS0.Px1.p1.1)\.
- Lianet al\.\(2026a\)Z\. Lian, X\. Peng, K\. Xu, Z\. Jia, X\. Che, Z\. Cheng, F\. Ma, L\. Cui, Y\. Zhang, X\. Liu, L\. Yang, J\. Li, F\. Zhang, L\. Xue, E\. Cambria, G\. Zhao, B\. W\. Schuller, and J\. TaoMER 2026: From discriminative emotion recognition to generative emotion understanding\.External Links:2604\.19417,[Document](https://dx.doi.org/10.48550/arXiv.2604.19417)Cited by:[Introduction](https://arxiv.org/html/2609.16962#Sx1.p1.1)\.
- Lianet al\.\(2025c\)Z\. Lian, H\. Sun, L\. Sun, H\. Chen, L\. Chen, H\. Gu, Z\. Wen, S\. Chen, S\. Zhang, H\. Yao, B\. Liu, R\. Liu, S\. Liang, Y\. Li, J\. Yi, and J\. TaoOV\-MER: Towards Open\-Vocabulary Multimodal Emotion Recognition\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 37015–37050\.External Links:[Link](https://proceedings.mlr.press/v267/lian25b.html)Cited by:[Introduction](https://arxiv.org/html/2609.16962#Sx1.p1.1),[Open\-vocabulary Emotion Recognition](https://arxiv.org/html/2609.16962#Sx2.SSx1.p1.1),[Table 1](https://arxiv.org/html/2609.16962#Sx3.T1.1.3.2.1.1),[Table 1](https://arxiv.org/html/2609.16962#Sx3.T1.1.8.2.1.1),[Comparison with Open\-Vocabulary Systems](https://arxiv.org/html/2609.16962#Sx4.SSx3.p1.1)\.
- Lianet al\.\(2026b\)Z\. Lian, F\. Zhang, L\. Chen, Y\. Zhang, R\. Liu, J\. Wu, H\. Chen, X\. Li, X\. Peng, B\. He, and J\. TaoAffectGPT\-RL: Revealing Roles of Reinforcement Learning in Open\-Vocabulary Emotion Recognition\.arXiv\.External Links:2605\.06126,[Document](https://dx.doi.org/10.48550/arXiv.2605.06126)Cited by:[Introduction](https://arxiv.org/html/2609.16962#Sx1.p1.1),[Open\-vocabulary Emotion Recognition](https://arxiv.org/html/2609.16962#Sx2.SSx1.p1.1)\.
- Lin \(1989\)L\. I\. LinA concordance correlation coefficient to evaluate reproducibility\.Biometrics45\(1\),pp\. 255–268\.External Links:[Document](https://dx.doi.org/10.2307/2532051),[Link](https://pubmed.ncbi.nlm.nih.gov/2720055/)Cited by:[Affect\-Prototype Adaptation Analysis](https://arxiv.org/html/2609.16962#Sx4.SSx6.p1.1)\.
- Liuet al\.\(2026\)Y\. Liu, L\. Zhang, H\. Li, H\. Shi, Y\. Ding, L\. Qu, and T\. LiFollow the clues, frame the truth: Hybrid\-evidential deductive reasoning in open\-vocabulary multimodal emotion recognition\.External Links:2603\.16463,[Document](https://dx.doi.org/10.48550/arXiv.2603.16463)Cited by:[Introduction](https://arxiv.org/html/2609.16962#Sx1.p1.1),[Open\-vocabulary Emotion Recognition](https://arxiv.org/html/2609.16962#Sx2.SSx1.p1.1)\.
- Loshchilov and Hutter \(2019\)I\. Loshchilov and F\. HutterDecoupled weight decay regularization\.InInternational Conference on Learning Representations,External Links:1711\.05101Cited by:[Experimental Setup](https://arxiv.org/html/2609.16962#Sx4.SSx2.p2.1)\.
- Lyuet al\.\(2024\)Y\. Lyu, X\. Zheng, J\. Zhou, and L\. WangUniBind: LLM\-Augmented Unified and Balanced Representation Space to Bind Them All\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 26742–26752\.External Links:[Document](https://dx.doi.org/10.1109/CVPR52733.2024.02526)Cited by:[Semantic Prototypes and Affect Representation](https://arxiv.org/html/2609.16962#Sx2.SSx3.p1.1)\.
- Miyoshiet al\.\(2026\)R\. Miyoshi, M\. Otani, and Y\. OkafujiRobust Multimodal Emotion Recognition from Incomplete Modalities via Query\-Based Unimodal and Cross\-Modal Learning\.In2026 IEEE/CVF Winter Conference on Applications of Computer Vision \(WACV\),pp\. 4901–4911\.External Links:ISSN 2642\-9381,[Document](https://dx.doi.org/10.1109/WACV61042.2026.00476)Cited by:[Introduction](https://arxiv.org/html/2609.16962#Sx1.p3.1),[Emotion Recognition with Incomplete Modalities](https://arxiv.org/html/2609.16962#Sx2.SSx2.p1.1)\.
- Mohammad \(2018\)S\. M\. MohammadObtaining reliable human ratings of valence, arousal, and dominance for 20,000 english words\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 174–184\.External Links:[Document](https://dx.doi.org/10.18653/v1/P18-1017)Cited by:[Affect\-Prototype Adaptation Analysis](https://arxiv.org/html/2609.16962#Sx4.SSx6.p1.1)\.
- Nguyenet al\.\(2026\)P\. Nguyen, T\. A\. Pham, D\. Le, and C\. T\. NguyenBALM: A Model\-Agnostic Framework for Balanced Multimodal Learning under Imbalanced Missing Rates\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 30246–30256\.External Links:2603\.19718Cited by:[Emotion Recognition with Incomplete Modalities](https://arxiv.org/html/2609.16962#Sx2.SSx2.p1.1),[Table 2](https://arxiv.org/html/2609.16962#Sx3.T2.1.13.1.1.1),[Table 2](https://arxiv.org/html/2609.16962#Sx3.T2.1.7.1.1.1),[Comparison with Incomplete\-Modality Fusion Methods](https://arxiv.org/html/2609.16962#Sx4.SSx4.p1.1)\.
- Parket al\.\(2021\)S\. Park, J\. Kim, S\. Ye, J\. Jeon, H\. Y\. Park, and A\. OhDimensional emotion detection from categorical emotion\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 4367–4380\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.358)Cited by:[Semantic Prototypes and Affect Representation](https://arxiv.org/html/2609.16962#Sx2.SSx3.p1.1)\.
- Perezet al\.\(2018\)E\. Perez, F\. Strub, H\. de Vries, V\. Dumoulin, and A\. CourvilleFiLM: Visual reasoning with a general conditioning layer\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.32\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v32i1.11671)Cited by:[Observed\-Only Set Fusion](https://arxiv.org/html/2609.16962#Sx3.SSx5.p1.1)\.
- Pipoliet al\.\(2025\)V\. Pipoli, F\. Bolelli, S\. Sarto, M\. Cornia, L\. Baraldi, C\. Grana, R\. Cucchiara, and E\. FicarraSemantically conditioned prompts for visual recognition under missing modality scenarios\.InProceedings of the Winter Conference on Applications of Computer Vision,pp\. 4968–4977\.Cited by:[Semantic Prototypes and Affect Representation](https://arxiv.org/html/2609.16962#Sx2.SSx3.p1.1)\.
- Praveenet al\.\(2022\)R\. G\. Praveen, W\. C\. de Melo, N\. Ullah, H\. Aslam, O\. Zeeshan, T\. Denorme, M\. Pedersoli, A\. L\. Koerich, S\. Bacon, P\. Cardinal, and E\. GrangerA joint cross\-attention model for audio\-visual fusion in dimensional emotion recognition\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops,pp\. 2485–2494\.External Links:[Document](https://dx.doi.org/10.1109/CVPRW56347.2022.00278),[Link](https://doi.org/10.1109/CVPRW56347.2022.00278)Cited by:[Introduction](https://arxiv.org/html/2609.16962#Sx1.p1.1)\.
- Qiuet al\.\(2026\)X\. Qiu, Y\. Fang, Q\. Zhou, B\. Zhai, J\. Hong, W\. Zhang, Y\. Lu, Y\. Zhang, and C\. LiBeyond Missing Modalities: Hypergraph Conditioned Diffusion for Uncertainty\-Aware Multimodal Emotion Recognition\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 22953–22963\.Cited by:[Introduction](https://arxiv.org/html/2609.16962#Sx1.p3.1),[Emotion Recognition with Incomplete Modalities](https://arxiv.org/html/2609.16962#Sx2.SSx2.p1.1)\.
- Radfordet al\.\(2021\)A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark, G\. Krueger, and I\. SutskeverLearning transferable visual models from natural language supervision\.InProceedings of the 38th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.139,pp\. 8748–8763\.Cited by:[Experimental Setup](https://arxiv.org/html/2609.16962#Sx4.SSx2.p1.1)\.
- Songet al\.\(2020\)K\. Song, X\. Tan, T\. Qin, J\. Lu, and T\. LiuMPNet: Masked and permuted pre\-training for language understanding\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 16857–16867\.Cited by:[Experimental Setup](https://arxiv.org/html/2609.16962#Sx4.SSx2.p1.1)\.
- Tsaiet al\.\(2019\)Y\. H\. Tsai, S\. Bai, P\. P\. Liang, J\. Z\. Kolter, L\. Morency, and R\. SalakhutdinovMultimodal Transformer for Unaligned Multimodal Language Sequences\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\. 6558–6569\.External Links:[Document](https://dx.doi.org/10.18653/v1/P19-1656)Cited by:[Table 2](https://arxiv.org/html/2609.16962#Sx3.T2.1.3.2.1.1),[Table 2](https://arxiv.org/html/2609.16962#Sx3.T2.1.9.2.1.1),[Comparison with Incomplete\-Modality Fusion Methods](https://arxiv.org/html/2609.16962#Sx4.SSx4.p1.1)\.
- Wanget al\.\(2026\)Z\. Wang, Z\. Yu, Y\. Zhu, B\. Zhao, H\. Liang, T\. Wang, W\. Xia, J\. Zhang, Z\. Liu, H\. Ma, F\. Ma, and Q\. TianAffectAgent: Collaborative multi\-agent reasoning for retrieval\-augmented multimodal emotion recognition\.External Links:2604\.12735,[Document](https://dx.doi.org/10.48550/arXiv.2604.12735)Cited by:[Open\-vocabulary Emotion Recognition](https://arxiv.org/html/2609.16962#Sx2.SSx1.p1.1),[Table 1](https://arxiv.org/html/2609.16962#Sx3.T1.1.11.1.1.1),[Table 1](https://arxiv.org/html/2609.16962#Sx3.T1.1.6.1.1.1),[Comparison with Open\-Vocabulary Systems](https://arxiv.org/html/2609.16962#Sx4.SSx3.p1.1)\.
- Yanget al\.\(2024\)A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. QiuQwen2\.5 technical report\.External Links:2412\.15115,[Document](https://dx.doi.org/10.48550/arXiv.2412.15115)Cited by:[Experimental Setup](https://arxiv.org/html/2609.16962#Sx4.SSx2.p1.1)\.
- Yunet al\.\(2024\)S\. Yun, I\. Choi, J\. Peng, Y\. Wu, J\. Bao, Q\. Zhang, J\. Xin, Q\. Long, and T\. ChenFlex\-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture\-of\-Experts\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 98782–98805\.External Links:[Document](https://dx.doi.org/10.52202/079017-3135)Cited by:[Emotion Recognition with Incomplete Modalities](https://arxiv.org/html/2609.16962#Sx2.SSx2.p1.1)\.
- Zhanget al\.\(2022\)Y\. Zhang, M\. Chen, J\. Shen, and C\. WangTailor versatile multi\-modal learning for multi\-label emotion recognition\.Proceedings of the AAAI Conference on Artificial Intelligence36\(8\),pp\. 9100–9108\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v36i8.20895),[Link](https://ojs.aaai.org/index.php/AAAI/article/view/20895)Cited by:[Introduction](https://arxiv.org/html/2609.16962#Sx1.p1.1)\.
- Zhanget al\.\(2026\)Z\. Zhang, J\. Chen, Y\. Hu, Z\. Zhang, X\. Yuan, M\. Yang, X\. Zhao, E\. C\. H\. Ngai, C\. Li, and X\. HuClue2Emo: A Brain\-Inspired Framework for Open\-Vocabulary Multimodal Emotion Recognition\.InICASSP 2026 \- 2026 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 12567–12571\.External Links:ISSN 2379\-190X,[Document](https://dx.doi.org/10.1109/ICASSP55912.2026.11460916)Cited by:[Open\-vocabulary Emotion Recognition](https://arxiv.org/html/2609.16962#Sx2.SSx1.p1.1)\.
- Zhaoet al\.\(2026\)B\. Zhao, F\. Ye, Y\. Ji, S\. Zhao, X\. Peng, and Z\. YuAffectVerse: Emotional world models for multimodal affective computing\.External Links:2605\.19950,[Document](https://dx.doi.org/10.48550/arXiv.2605.19950)Cited by:[Open\-vocabulary Emotion Recognition](https://arxiv.org/html/2609.16962#Sx2.SSx1.p1.1)\.
- Zhaoet al\.\(2021\)J\. Zhao, R\. Li, and Q\. JinMissing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 2608–2618\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.203)Cited by:[Introduction](https://arxiv.org/html/2609.16962#Sx1.p3.1),[Emotion Recognition with Incomplete Modalities](https://arxiv.org/html/2609.16962#Sx2.SSx2.p1.1),[Table 2](https://arxiv.org/html/2609.16962#Sx3.T2.1.10.1.1.1),[Table 2](https://arxiv.org/html/2609.16962#Sx3.T2.1.4.1.1.1),[Comparison with Incomplete\-Modality Fusion Methods](https://arxiv.org/html/2609.16962#Sx4.SSx4.p1.1)\.