Mitigating Strong-Modality Collapse in Multimodal Learning via Inverted Asymmetric Fusion

# Mitigating Strong-Modality Collapse in Multimodal Learning via Inverted Asymmetric Fusion
Source: [https://arxiv.org/html/2608.26879](https://arxiv.org/html/2608.26879)
Mary Ogbuka KennethAffiliation:Algorithmic Human Development group, Department of ComputingImperial College London, United KingdomFoaad KhosmoodAffiliation:Computer Engineering Department, California Polytechnic State UniversitySan Luis Obispo, United StatesCorrespondence:[m\.kenneth22@imperial\.ac\.uk](mailto:[email protected])Abbas EdalatAffiliation:Algorithmic Human Development group, Department of ComputingImperial College London, United Kingdom

###### Abstract

Fusing multiple modalities is expected to improve model performance\. However, on the MultiHuSE dataset, early, late, and symmetric attention fusion often fail to outperform the best unimodal baseline \(text\)\. Pathway isolation of a symmetric attention fusion model reveals that the text\-pathway accuracy drops from 74\.9% to 56\.4% after fusion in one such setting, indicating that the dominant modality can be degraded during integration\. We term this strong\-modality collapse and argue that it helps explain why some multimodal models fail to surpass unimodal baselines\. We propose Inverted Asymmetric Fusion \(IAF\), which avoids forcing mutual attention across modalities\. The dominant modality is preserved by passing through fusion unchanged, while weaker modalities attend to it as a contextual anchor\. Before fusion, weaker modalities are strengthened using Modality\-Aware Knowledge Distillation\. We evaluate IAF on three benchmarks with different modality hierarchies: text\-dominant datasets \(MultiHuSE, UR\-FUNNY\) and an audio\-visual\-dominant dataset \(MUStARD\)\. Pathway isolation shows that IAF preserves the dominant modality’s internal accuracy at its unimodal ceiling across all tested configurations, whereas symmetric fusion degrades it by up to 18\.5% on MultiHuSE\. IAF improves over the strongest unimodal baseline by up to 8\.25%\.

## 1Introduction

Early, late, and symmetric attention fusion form the standard toolkit for integrating text, audio, and video, resting on the intuition that combining heterogeneous signals should enable models to outperform any single modality alone[Baltrusaitis et al\. \(2019\)](https://arxiv.org/html/2608.26879#bib.bib38);[Li and Tang \(2025\)](https://arxiv.org/html/2608.26879#bib.bib35)\. Yet on MultiHuSE\([Kenneth et al\., 2025b](https://arxiv.org/html/2608.26879#bib.bib33)\), we observe that these approaches often fail to surpass a text\-only baseline and can even degrade performance in some settings\. This pattern is not isolated: prior work in affective computing and action recognition reports similar findings, where multimodal models trained with conventional fusion strategies underperform their strongest unimodal component despite having access to additional modalities\([Peng et al\., 2022](https://arxiv.org/html/2608.26879#bib.bib6);[Wang et al\., 2020b](https://arxiv.org/html/2608.26879#bib.bib56)\)\.

Prior work attributes such failures to optimisation dynamics: modalities overfit and generalise at different rates, causing training to gravitate toward whichever modality converges fastest[Chaudhuri et al\. \(2025\)](https://arxiv.org/html/2608.26879#bib.bib1);[Huang et al\. \(2022\)](https://arxiv.org/html/2608.26879#bib.bib29);[Wang et al\. \(2020b\)](https://arxiv.org/html/2608.26879#bib.bib56)\. We extend this view by identifying an additional architectural mechanism that may contribute to the problem\. Pathway isolation analysis of a symmetric attention model on MultiHuSE shows that the dominant text pathway drops from 74\.9% to 56\.4% after fusion \(Figure[1](https://arxiv.org/html/2608.26879#S1.F1)\)\.

74\.956\.474\.9Unimodal\(E5\)SymmetricFusionIAF\(Ours\)−\-18\.5 pppreservedFigure 1:Text\-pathway accuracy on MultiHuSE \(E5 encoder\)\. Symmetric fusion degrades the dominant modality by 18\.5 pp; IAF preserves it \(Table[5](https://arxiv.org/html/2608.26879#S5.T5)\)\.This decline suggests that symmetric cross\-attention forces the dominant modality to attend to weaker, noisier signals, degrading the representation that initially carried the most predictive information\. We term this phenomenonstrong\-modality collapse:a failure mode in which fusion undermines the model’s strongest modality\.

This diagnosis suggests an architectural alternative\. If symmetric attention contributes to this degradation, the solution lies in restructuring the fusion pathway rather than solely reweighting gradients or losses[Wei et al\. \(2025\)](https://arxiv.org/html/2608.26879#bib.bib43);[Fan et al\. \(2023\)](https://arxiv.org/html/2608.26879#bib.bib46);[Javaloy et al\. \(2022\)](https://arxiv.org/html/2608.26879#bib.bib26), which adjust training dynamics while leaving the cross\-attention topology intact\. To this end, we propose Inverted Asymmetric Fusion \(IAF\), which imposes an explicit structural hierarchy\. In IAF, the dominant modality bypasses cross\-modal attention entirely to preserve its representation, while weaker modalities attend to it as a contextual anchor rather than as standalone classifiers\. Additionally, a modality\-aware knowledge distillation stage[Hinton et al\. \(2015\)](https://arxiv.org/html/2608.26879#bib.bib16)precedes fusion, strengthening weaker modality representations so they can contribute a more informative complementary signal to the dominant pathway\.

Testing across inverted modality hierarchies is essential to establish that IAF’s structural approach generalises beyond any single dominance configuration\. We therefore evaluate on MultiHuSE\([Kenneth et al\., 2025b](https://arxiv.org/html/2608.26879#bib.bib33)\)and UR\-FUNNY\([Kamrul Hasan et al\., 2019](https://arxiv.org/html/2608.26879#bib.bib54)\), where text is dominant, and MUStARD\([Castro et al\., 2019](https://arxiv.org/html/2608.26879#bib.bib53)\), where acoustic and visual cues carry more predictive signal\. Our contributions are as follows:

- •We identify strong\-modality collapse as an architectural failure mode, distinct from optimisation\-based methods that reweight gradients while preserving symmetric cross\-attention, and provide pathway\-level evidence in symmetric fusion\.
- •We propose IAF, a fusion framework that prevents strong\-modality collapse by design and preserves the dominant modality’s pathway accuracy at its unimodal ceiling across all tested configurations\.
- •We introduce a Modality\-Aware Knowledge Distillation pipeline that adapts distillation to each dataset’s modality hierarchy, strengthening weaker encoders prior to fusion\.
- •We demonstrate cross\-hierarchy robustness across 3 benchmarks with differing dominance structures, improving on prior baselines on UR\-FUNNY and MUStARD\.

## 2Related Work

### 2\.1Multimodal Fusion Architectures

Multimodal fusion has evolved from simple concatenation and prediction averaging to learned attention mechanisms[Baltrusaitis et al\. \(2019\)](https://arxiv.org/html/2608.26879#bib.bib38);[Pu Liang et al\. \(2021\)](https://arxiv.org/html/2608.26879#bib.bib32)\. Tensor\-based approaches explicitly model cross\-modal interactions via outer products[Zadeh et al\. \(2017\)](https://arxiv.org/html/2608.26879#bib.bib51), with later work reducing computational cost through low\-rank factorisation[Liu et al\. \(2018\)](https://arxiv.org/html/2608.26879#bib.bib17)\. The advent of attention\-based fusion, led by the Multimodal Transformer \(MulT\)[Tsai et al\. \(2020\)](https://arxiv.org/html/2608.26879#bib.bib42)established symmetric cross\-attention as the dominant paradigm\. Subsequent improvements incorporated pretrained language models[Rahman et al\. \(2020\)](https://arxiv.org/html/2608.26879#bib.bib22), bottleneck token constraints[Nagrani et al\. \(2021\)](https://arxiv.org/html/2608.26879#bib.bib4), and modality subspace disentanglement[Hazarika et al\. \(2020\)](https://arxiv.org/html/2608.26879#bib.bib25)\.

While these designs capture rich cross\-modal dynamics, they apply attention uniformly, regardless of the relative modality strengths[Feng et al\. \(2024\)](https://arxiv.org/html/2608.26879#bib.bib24)\. Forcing a dominant modality to symmetrically attend to weaker, noisier signals can corrupt its internal representation\. Recent work has begun to address this by using text to guide multimodal fusion in affective tasks[Zhang et al\. \(2025\)](https://arxiv.org/html/2608.26879#bib.bib52), though such designs typically retain mutual attention rather than fully shielding the dominant stream from cross\-modal interference\. This vulnerability motivates our inverted asymmetric approach, which preserves the dominant modality while using weaker streams as contextual anchors\.

### 2\.2Modality Collapse and Pathway Analysis

Representational degradation in fusion is one instance of a broader failure mode: modality collapse\. Empirical studies show multimodal models often exploit only a subset of their input modalities[Gong et al\. \(2025\)](https://arxiv.org/html/2608.26879#bib.bib39);[Kenneth et al\. \(2024b\)](https://arxiv.org/html/2608.26879#bib.bib50);[Zhang et al\. \(2024\)](https://arxiv.org/html/2608.26879#bib.bib40)\. Because different modalities overfit and generalise at uneven rates, training tends to favour the fastest\-converging modality[Wang et al\. \(2020b\)](https://arxiv.org/html/2608.26879#bib.bib56);[Wu et al\. \(2022\)](https://arxiv.org/html/2608.26879#bib.bib10)\. Theoretical analyses attribute this behaviour to conflicting gradient directions during joint optimisation[Huang et al\. \(2022\)](https://arxiv.org/html/2608.26879#bib.bib29);[Chaudhuri et al\. \(2025\)](https://arxiv.org/html/2608.26879#bib.bib1)\.

To mitigate modality imbalance, prior work has proposed optimisation\-based strategies such as on\-the\-fly gradient modulation \(OGM\-GE\)[Wei et al\. \(2025\)](https://arxiv.org/html/2608.26879#bib.bib43), Pareto\-based integration[Wei and Hu \(2024\)](https://arxiv.org/html/2608.26879#bib.bib28), prototype\-guided clustering \(PMR\)[Fan et al\. \(2023\)](https://arxiv.org/html/2608.26879#bib.bib46), and impartial multitask optimisation[Javaloy et al\. \(2022\)](https://arxiv.org/html/2608.26879#bib.bib26)\. These methods address imbalance by adjusting gradients or training objectives, but generally retain the same fusion topology, leaving dominant representations vulnerable to cross\-modal interference\. In contrast, our work takes a complementary architectural approach\. IAF restructures the fusion pathway to limit such interference by design\. However, structural shielding alone is insufficient unless weaker encoders are first strengthened, motivating the pre\-fusion distillation stage\.

### 2\.3Knowledge Distillation in Multimodal Learning

Knowledge distillation \(KD\) transfers generalisation by aligning a student’s predictions with a high\-capacity teacher[Hinton et al\. \(2015\)](https://arxiv.org/html/2608.26879#bib.bib16);[Gou et al\. \(2021\)](https://arxiv.org/html/2608.26879#bib.bib23)\. Cross\-modal distillation extends this across domain boundaries and has proven effective for depth, optical flow, action recognition, and emotion analysis[Gupta et al\. \(2016\)](https://arxiv.org/html/2608.26879#bib.bib13);[Thoker and Gall \(2019\)](https://arxiv.org/html/2608.26879#bib.bib14);[Albanie et al\. \(2018\)](https://arxiv.org/html/2608.26879#bib.bib19)\. In multimodal fusion, KD is commonly used within the fusion stage for compression, dropout robustness, missing\-modality learning, or representational alignment[Fang et al\. \(2021\)](https://arxiv.org/html/2608.26879#bib.bib11);[Wang et al\. \(2023\)](https://arxiv.org/html/2608.26879#bib.bib18);[Wang et al\. \(2020a\)](https://arxiv.org/html/2608.26879#bib.bib37);[Wei et al\. \(2023\)](https://arxiv.org/html/2608.26879#bib.bib27);[Li et al\. \(2023b\)](https://arxiv.org/html/2608.26879#bib.bib15);[Lin and Hu \(2024\)](https://arxiv.org/html/2608.26879#bib.bib31)\. By contrast, our Modality\-Aware Knowledge Distillation \(MAKD\) is a pre\-fusion stage that uses only the empirically dominant unimodal modality as teacher and is designed specifically to strengthen weaker encoders before asymmetric integration\. A detailed design comparison is provided in Appendix[H](https://arxiv.org/html/2608.26879#A8)\.

### 2\.4Humor and Sarcasm Detection

Humor and sarcasm benchmarks provide a useful testbed for modality imbalance because their predictive hierarchies differ substantially across datasets\. UR\-FUNNY[Kamrul Hasan et al\. \(2019\)](https://arxiv.org/html/2608.26879#bib.bib54)and MultiHuSE[Kenneth et al\. \(2025b\)](https://arxiv.org/html/2608.26879#bib.bib33);[Kenneth et al\. \(2024a\)](https://arxiv.org/html/2608.26879#bib.bib3);[Kenneth et al\. \(2025a\)](https://arxiv.org/html/2608.26879#bib.bib20)are text\-dominant, whereas MUStARD[Castro et al\. \(2019\)](https://arxiv.org/html/2608.26879#bib.bib53)places greater weight on acoustic and visual cues\. This contrast makes the domain well suited for evaluating whether a fusion architecture can preserve strong modalities without sacrificing cross\-modal contribution\.

## 3Methodology

Our framework comprises two sequential stages \(Figure[2](https://arxiv.org/html/2608.26879#S3.F2)\)\.Stage 1appliesModality\-Aware Knowledge Distillationto strengthen weaker modalities prior to fusion, adapting the strategy to each dataset’s modality hierarchy\.Stage 2performsInverted Asymmetric Fusion\(IAF\), integrating streams within a strict hierarchy that shields the dominant modality from cross\-modal interference and prevents strong\-modality collapse\.

Unimodal EncodersE5 / BERT𝐡\(t\)\\mathbf\{h\}^\{\(t\)\}dominantDasheng𝐡\(a\)\\mathbf\{h\}^\{\(a\)\}PE\-Core𝐡\(v\)\\mathbf\{h\}^\{\(v\)\}Acc⁡\(m∗\)\>Acc⁡\(m≠m∗\)\\mathrm\{Acc\}\(m^\{\*\}\)\>\\mathrm\{Acc\}\(m\{\\neq\}m^\{\*\}\)Stage 1:Knowledge DistillationTeacher\(m∗\)\(m^\{\*\}\)ℒKD=α​ℒCE\\mathcal\{L\}\_\{\\mathrm\{KD\}\}=\\alpha\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\+β​ℒsoft\\quad\{\+\}\\,\\beta\\mathcal\{L\}\_\{\\mathrm\{soft\}\}\+γ​ℒfeat\\quad\{\+\}\\,\\gamma\\mathcal\{L\}\_\{\\mathrm\{feat\}\}AudioStudentVideoStudentdistilDirection adapts todataset modality hierarchyStage 2:Inverted Asymmetric Fusionpure𝐨\(m∗\)=𝐡\(m∗\)\\mathbf\{o\}^\{\(m^\{\*\}\)\}=\\mathbf\{h\}^\{\(m^\{\*\}\)\}attend\| CMAattend\| CMAcontextψm∗\\psi\_\{m^\{\*\}\}frozenψa\\psi\_\{a\}trainableψv\\psi\_\{v\}trainableGate𝒢\\mathcal\{G\}:𝐰=softmax⁡\(𝒢⁡\(\[𝐡\(t\)​‖𝐡\(a\)‖​𝐡\(v\)\]\)\)∈ℝ3\\mathbf\{w\}=\\mathrm\{softmax\}\\\!\\left\(\\mathcal\{G\}\(\[\\mathbf\{h\}^\{\(t\)\}\\\|\\mathbf\{h\}^\{\(a\)\}\\\|\\mathbf\{h\}^\{\(v\)\}\]\)\\right\)\\in\\mathbb\{R\}^\{3\}Training Regularisation:Mixup\(α=0\.05\)\(\\alpha\{=\}0\.05\)\+ Curriculum Modality Dropout𝐲^=∑mwm⋅ψm​\(𝐨\(m\)\)\\hat\{\\mathbf\{y\}\}=\\sum\_\{m\}w\_\{m\}\\cdot\\psi\_\{m\}\(\\mathbf\{o\}^\{\(m\)\}\)Prediction𝐲^\\hat\{\\mathbf\{y\}\}Humor style\(MultiHuSE\)Humor / Sarcasm\(UR\-FUNNY, MUStARD\)Modalities:TextAudioVideopure: no cross\-attnattend: queries allKnowledge Distil\.Dominant \(preserved\)

Figure 2:Overview of the proposed two\-stage framework\. Stage 1 performs Modality\-Aware Knowledge Distillation, and Stage 2 applies the Inverted Asymmetric Fusion\.### 3\.1Problem Formulation

Let𝐟\(t\)∈ℝdt\\mathbf\{f\}^\{\(t\)\}\\in\\mathbb\{R\}^\{d\_\{t\}\},𝐟\(a\)∈ℝda\\mathbf\{f\}^\{\(a\)\}\\in\\mathbb\{R\}^\{d\_\{a\}\}, and𝐟\(v\)∈ℝdv\\mathbf\{f\}^\{\(v\)\}\\in\\mathbb\{R\}^\{d\_\{v\}\}denote pre\-extracted feature vectors for text, audio, and video modalities of a single utterance, respectively\. Each is projected to a sharedd=512d\{=\}512space via a modality\-specific encoderϕm\\phi\_\{m\}, yielding𝐡\(m\)=ϕm​\(𝐟\(m\)\)∈ℝ512\\mathbf\{h\}^\{\(m\)\}=\\phi\_\{m\}\(\\mathbf\{f\}^\{\(m\)\}\)\\in\\mathbb\{R\}^\{512\}\. Thedominant modalitym∗m^\{\*\}is defined asm∗=arg⁡maxm​Accuracy​\(ϕm\)m^\{\*\}=\\arg\\max\_\{m\}\\;\\text\{Accuracy\}\(\\phi\_\{m\}\)on the validation set\.

### 3\.2Stage 1: Modality\-Aware Knowledge Distillation

The dominant modalitym∗m^\{\*\}acts as a teacher to elevate weaker modalitiesm≠m∗m\\neq m^\{\*\}before fusion\.

#### 3\.2\.1Distillation Loss \(ℒKD\\mathcal\{L\}\_\{\\text\{KD\}\}\)\.

Each student𝒮\\mathcal\{S\}is trained against a frozen teacher𝒯\\mathcal\{T\}\(logits𝐳T∈ℝC\\mathbf\{z\}^\{T\}\\in\\mathbb\{R\}^\{C\}, bottleneck features𝐠T∈ℝ512\\mathbf\{g\}^\{T\}\\in\\mathbb\{R\}^\{512\}\) using:

ℒKD=α​ℒCE\+β​ℒsoft\+γ​ℒfeat\\mathcal\{L\}\_\{\\text\{KD\}\}\\;=\\;\\alpha\\,\\mathcal\{L\}\_\{\\text\{CE\}\}\+\\beta\\,\\mathcal\{L\}\_\{\\text\{soft\}\}\+\\gamma\\,\\mathcal\{L\}\_\{\\text\{feat\}\}\(1\)The loss terms are: hard\-label cross\-entropyℒCE\\mathcal\{L\}\_\{\\text\{CE\}\}, temperature\-scaled KL divergenceℒsoft\\mathcal\{L\}\_\{\\text\{soft\}\}\(withT=3\.5T=3\.5\), and bottleneck feature alignmentℒfeat\\mathcal\{L\}\_\{\\text\{feat\}\}:

ℒCE\\displaystyle\\mathcal\{L\}\_\{\\text\{CE\}\}=−∑c=1C𝟏\[y=c\]log\(softmax\(𝐳S\)c\)\\displaystyle=\-\\sum\_\{c=1\}^\{C\}\\mathbf\{1\}\[y=c\]\\,\\log\\\!\\bigl\(\\text\{softmax\}\(\\mathbf\{z\}^\{S\}\)\_\{c\}\\bigr\)\(2\)ℒsoft\\displaystyle\\mathcal\{L\}\_\{\\text\{soft\}\}=T2​KL​\(σ⁡\(𝐳ST\)∥σ⁡\(𝐳TT\)\)\\displaystyle=T^\{2\}\\,\\text\{KL\}\\\!\\left\(\\sigma\\\!\\left\(\\frac\{\\mathbf\{z\}^\{S\}\}\{T\}\\right\)\\,\\middle\\\|\\,\\sigma\\\!\\left\(\\frac\{\\mathbf\{z\}^\{T\}\}\{T\}\\right\)\\right\)\(3\)ℒfeat\\displaystyle\\mathcal\{L\}\_\{\\text\{feat\}\}=‖𝐠S−𝐠T‖22\\displaystyle=\\left\\\|\\mathbf\{g\}^\{S\}\-\\mathbf\{g\}^\{T\}\\right\\\|\_\{2\}^\{2\}\(4\)whereσ\\sigmadenotes the softmax function\. We use fixed coefficientsα=0\.4\\alpha=0\.4,β=0\.35\\beta=0\.35, andγ=0\.25\\gamma=0\.25, selected by manual tuning on MultiHuSE and held constant across all datasets and encoder combinations\. A sensitivity sweep of coefficients over four configurations shows that fusion accuracy varies by at most 1\.20 percentage points, with both the default and equal\-weight settings achieving the best result; see Appendix[G](https://arxiv.org/html/2608.26879#A7)for details\.

##### Weighted Dual\-Teacher Distillation \(MUStARD\)\.

When both audio and video dominate \(MUStARD\), students are supervised by weighted dual teachers\. The student is jointly supervised by a primary teacher𝒯1\\mathcal\{T\}\_\{1\}and a secondary teacher𝒯2\\mathcal\{T\}\_\{2\}, whose losses are combined via a scalar weightw1w\_\{1\}:

ℒdual=w1​ℒKD​\(𝒯1\)\+\(1−w1\)​ℒKD​\(𝒯2\)\\mathcal\{L\}\_\{\\text\{dual\}\}=w\_\{1\}\\,\\mathcal\{L\}\_\{\\text\{KD\}\}\(\\mathcal\{T\}\_\{1\}\)\+\(1\-w\_\{1\}\)\\,\\mathcal\{L\}\_\{\\text\{KD\}\}\(\\mathcal\{T\}\_\{2\}\)\(5\)wherew1w\_\{1\}is tuned per configuration \(w1=0\.8w\_\{1\}=0\.8for audio\-primary,w1=0\.2w\_\{1\}=0\.2for video\-primary\)\.

##### Cross\-Architecture Distillation \(UR\-FUNNY\)\.

On UR\-FUNNY, the narrow unimodal performance gap limits the representational surplus available for cross\-modal transfer\. This motivated a same\-modality, cross\-architecture distillation: high\-capacity static teachers \(E5, Dasheng, PE\-Core\) distill into BiLSTM students on benchmark features \(GloVe, COVAREP, OpenFace\), isolating IAF’s architectural contribution\. The same three\-term loss in Eq\. \([1](https://arxiv.org/html/2608.26879#S3.E1)\) applies, with teacher features𝐠T\\mathbf\{g\}^\{T\}projected to 512\-dim prior to alignment\.

With all modality encoders operating at an enhanced representation, Stage 2 integrates these streams using IAF\.

### 3\.3Stage 2: Inverted Asymmetric Fusion

#### 3\.3\.1CrossModal Anchor \(CMA\)

The*CrossModal Anchor*module serves as the core attention primitive in our architecture\. Given a query\-modality vector𝐡∈ℝ512\\mathbf\{h\}\\in\\mathbb\{R\}^\{512\}and a context set𝐂∈ℝK×512\\mathbf\{C\}\\in\\mathbb\{R\}^\{K\\times 512\}containingKKother modalities, the module computes:

𝐡^\\displaystyle\\hat\{\\mathbf\{h\}\}=MHA​\(𝐡,𝐂,𝐂\)\\displaystyle=\\text\{MHA\}\(\\mathbf\{h\},\\,\\mathbf\{C\},\\,\\mathbf\{C\}\)\(6\)𝐡′\\displaystyle\\mathbf\{h\}^\{\\prime\}=LayerNorm​\(𝐡\+Dropout​\(𝐡^\)\)\\displaystyle=\\text\{LayerNorm\}\\\!\\left\(\\mathbf\{h\}\+\\text\{Dropout\}\(\\hat\{\\mathbf\{h\}\}\)\\right\)\(7\)whereMHA​\(⋅\)\\text\{MHA\}\(\\cdot\)denotes multi\-head attention[Vaswani et al\. \(2017\)](https://arxiv.org/html/2608.26879#bib.bib5)withdmodel=512d\_\{\\text\{model\}\}\{=\}512andH=8H\{=\}8heads\. The residual connection Eq\. \([7](https://arxiv.org/html/2608.26879#S3.E7)\) preserves the query modality’s original representation\.

#### 3\.3\.2Structural Hierarchy:Purevs\.Attend

Letm∗m^\{\*\}denote the dominant modality \(or modalities\)\. IAF enforces the following constraint on the forward pass:

𝐨\(m\)=\{𝐡\(m\)if​m=m∗​\(pure\)CMA​\(𝐡\(m\),\[𝐡\(j\)\]j≠m\)if​m≠m∗​\(attend\)\\mathbf\{o\}^\{\(m\)\}=\\begin\{cases\}\\mathbf\{h\}^\{\(m\)\}&\\text\{if \}m=m^\{\*\}\\;\(\\textsc\{pure\}\)\\\\\[4\.0pt\] \\text\{CMA\}\\\!\\left\(\\mathbf\{h\}^\{\(m\)\},\\;\[\\mathbf\{h\}^\{\(j\)\}\]\_\{j\\neq m\}\\right\)&\\text\{if \}m\\neq m^\{\*\}\\;\(\\textsc\{attend\}\)\\end\{cases\}\(8\)
Thepurepathway remains unmodified\.Attendpathways query all other modalities \(including the shielded dominant\) as contextual filters\. On MultiHuSE/UR\-FUNNY, text ispure; on MUStARD, audio/video\.

#### 3\.3\.3Adaptive Gating and Final Classification

Each modality produces pathway logits via a dedicated classifier headψm\\psi\_\{m\}\. For thepuremodality,ψm∗\\psi\_\{m^\{\*\}\}is a frozen copy of its standalone classifier, preserving its decision boundary\. Forattendmodalities,ψm\\psi\_\{m\}is a trainable linear head suited to their contextually\-enriched representations\.

A gating network𝒢\\mathcal\{G\}computes sample\-adaptive weights from pre\-attention features:

𝐰=softmax​\(𝒢⁡\(\[𝐡\(t\)​‖𝐡\(a\)‖​𝐡\(v\)\]\)\)\\mathbf\{w\}=\\text\{softmax\}\\\!\\left\(\\mathcal\{G\}\\\!\\left\(\[\\mathbf\{h\}^\{\(t\)\}\\,\\\|\\,\\mathbf\{h\}^\{\(a\)\}\\,\\\|\\,\\mathbf\{h\}^\{\(v\)\}\]\\right\)\\right\)\(9\)where𝐰∈ℝ3\\mathbf\{w\}\\in\\mathbb\{R\}^\{3\}denotes the modality\-gate weights, and𝒢\\mathcal\{G\}is a two\-layer MLP with batch normalisation and dropout\. The final prediction is the weighted sum of per\-pathway logits:

𝐲^=∑m∈\{t,a,v\}wm⋅ψm​\(𝐨\(m\)\)\\hat\{\\mathbf\{y\}\}=\\sum\_\{m\\in\\\{t,a,v\\\}\}w\_\{m\}\\cdot\\psi\_\{m\}\(\\mathbf\{o\}^\{\(m\)\}\)\(10\)Gating on pre\-attention features preserves the textscpure/attendseparation\.

### 3\.4Training Regularisation

We apply two complementary regularisation strategies during fusion training\.

#### 3\.4\.1Mixup

For each mini\-batch, we sample a mixing coefficientλ∼Beta​\(αmix,αmix\)\\lambda\\sim\\text\{Beta\}\(\\alpha\_\{\\text\{mix\}\},\\,\\alpha\_\{\\text\{mix\}\}\)withαmix=0\.05\\alpha\_\{\\text\{mix\}\}\{=\}0\.05and interpolate pairs of feature vectors and their labels:

𝐡~i\(m\)=λ​𝐡i\(m\)\+\(1−λ\)​𝐡σ⁡\(i\)\(m\),y~i=\(λ,yi,yσ⁡\(i\)\)\\tilde\{\\mathbf\{h\}\}\_\{i\}^\{\(m\)\}=\\lambda\\,\\mathbf\{h\}\_\{i\}^\{\(m\)\}\+\(1\-\\lambda\)\\,\\mathbf\{h\}\_\{\\sigma\(i\)\}^\{\(m\)\},\\quad\\tilde\{y\}\_\{i\}=\(\\lambda,\\,y\_\{i\},\\,y\_\{\\sigma\(i\)\}\)\(11\)whereσ\\sigmais a random permutation of batch indices\. Mixup discourages overconfident predictions, which is critical for preventing overfitting \(ablation in Appendix[C](https://arxiv.org/html/2608.26879#A3)\)

#### 3\.4\.2Curriculum Modality Dropout

To prevent over\-reliance on any single modality, we apply a two\-phase stochastic masking schedule: all modalities are present during warm\-up \(epochs<<15\), after which each sample is assigned one of seven masking configurations \(trimodalp=0\.40p=0\.40, each bimodal pairp=0\.10p=0\.10, each unimodalp=0\.10p=0\.10\), with masked modalities zeroed before the forward pass\.

### 3\.5Pathway Analysis

To verify dominant pathway preservation, we isolate each modality’s internal contribution by routing𝐡\(m\)\\mathbf\{h\}^\{\(m\)\}directly throughψm\\psi\_\{m\}, bypassing gating\. For thepuremodalitym∗m^\{\*\}:

Accuracypath\(m∗\)=Accuracyuni\(m∗\)\\text\{Accuracy\}\_\{\\text\{path\}\}^\{\(m^\{\*\}\)\}=\\text\{Accuracy\}\_\{\\text\{uni\}\}^\{\(m^\{\*\}\)\}\(12\)
That is,m∗m^\{\*\}pathway accuracy equals its standalone unimodal performance\. Any deviation from this equality indicates cross\-modal interference\. Symmetric fusion violates this equality; IAF enforces it by construction\. Forattendmodalities, pathway accuracy is expected to drop below their unimodal baselines, as their representations are no longer optimised for standalone classification but instead tuned to provide contextual signal\.

## 4Experiments

We describe the datasets, feature extraction procedures, baseline models, and implementation details\.

### 4\.1Datasets

We evaluate on three multimodal benchmarks for humor and sarcasm detection, chosen to span contrasting modality hierarchies and task complexity \(binary and multiclass classification\)\.

##### MultiHuSE

[Kenneth et al\. \(2025b\)](https://arxiv.org/html/2608.26879#bib.bib33)is a 5\-class humor style classification dataset comprising 2,407 video utterances from 50 diverse actors, including 1,463 unique text scripts and 943 re\-performances\. Standard random splits risk data leakage across performances of the same script, so we introduce a strict 5\-fold cross\-validation protocol grouped by unique text ID, ensuring all performances of a given script appear exclusively in either training or evaluation\. All baseline models are re\-implemented and re\-evaluated under this protocol\. Modality hierarchy: Text≫\\ggAudio≫\\ggVideo\.

##### UR\-FUNNY

[Kamrul Hasan et al\. \(2019\)](https://arxiv.org/html/2608.26879#bib.bib54)is a binary humor detection dataset of 16,514 context–punchline pairs drawn from TED talks, using the standard train/dev/test split \(Ntest=994N\_\{\\text\{test\}\}\{=\}994\)\. Following the original benchmark, we use sequential features GloVe \(text\), COVAREP \(audio\), and OpenFace \(video\) throughout, isolating our architectural contribution from feature quality and enabling direct comparison with prior work using the same feature set\. Modality hierarchy: Text\>\>Audio≈\\approxVideo\.

##### MUStARD

[Castro et al\. \(2019\)](https://arxiv.org/html/2608.26879#bib.bib53)is a binary sarcasm detection dataset of 690 balanced utterances drawn from TV sitcoms, using 5\-fold cross\-validation \(Ntest=138N\_\{\\text\{test\}\}\{=\}138per fold\) from the original paper\. Modality hierarchy: Audio≈\\approxVideo\>\>Text, is the clearest inversion relative to MultiHuSE and motivates our cross\-hierarchy evaluation\.

### 4\.2Feature Extraction

All features are extracted offline using frozen pre\-trained encoders and serialised to disk prior to training\. This design decouples representation quality from the fusion architecture, ensuring that observed performance differences reflect architectural choices rather than effects of encoder fine\-tuning\.

##### MultiHuSE and MUStARD\.

For text, we evaluate two encoders: BERT\-based\-uncased[Devlin et al\. \(2019\)](https://arxiv.org/html/2608.26879#bib.bib8)\(768\-d\) and Multilingual\-E5\-large\-instruct[Wang et al\. \(2024\)](https://arxiv.org/html/2608.26879#bib.bib34)\(1024\-d\)\. Using both allows us to verify that our findings are not specific to a single text backbone\. For audio, we use Dasheng\-0\.6B[Dinkel et al\. \(2024\)](https://arxiv.org/html/2608.26879#bib.bib48)\(1280\-d\), a large\-scale general\-purpose audio encoder\. For video, we evaluate two encoders: VideoMAE[Tong et al\. \(2022\)](https://arxiv.org/html/2608.26879#bib.bib55)\(768\-d\) and PE\-Core[Bolya et al\. \(2025\)](https://arxiv.org/html/2608.26879#bib.bib45)\(1024\-d\), providing coverage of both masked autoencoder and contrastive video representations\.

##### UR\-FUNNY\.

We use GloVe[Pennington et al\. \(2014\)](https://arxiv.org/html/2608.26879#bib.bib21)\(text\), COVAREP[Degottex et al\. \(2014\)](https://arxiv.org/html/2608.26879#bib.bib12)\(audio\), and OpenFace[Baltrusaitis et al\. \(2018\)](https://arxiv.org/html/2608.26879#bib.bib44)\(video\), consistent with the original benchmark\. The role of modern encoders as distillation teachers within this established feature setting is described in Section[3\.2](https://arxiv.org/html/2608.26879#S3.SS2)\.

### 4\.3Implementation Details

##### Infrastructure\.

Feature extraction is performed on an HPC cluster \(NVIDIA A16, 15GB VRAM\), while all subsequent training is conducted on a consumer laptop \(NVIDIA GeForce RTX 3050, 4GB VRAM\)\. All primary results use a fixed random seed of 999\. For MultiHuSE and MUStARD, results are reported as mean±\\pmSD over 5\-fold cross\-validation; for UR\-FUNNY, we additionally report a five\-seed analysis using seeds 42, 100, 123, 999, and 600\. The unimodal pre\-evaluation used to identify the dominant modality adds minimal overhead: on our hardware, training a single unimodal encoder takes less than 15 minutes, compared with a few hours for full fusion training\.

##### Model Parameters\.

Pre\-trained encoders used for feature extraction have the following parameter counts: BERT\-base \(110M\), E5\-multilingual \(560M\), Dasheng \(600M\), VideoMAE\-base \(94M\), and PE\-Core \(320M visual encoder \+ 310M language tower\)\. These encoders are frozen throughout; no encoder parameters are updated during distillation or fusion training\.

##### Distillation and Fusion\.

We used AdamW for distillation, CosineAnnealingLR for fusion, a batch size of 64, Mixup withαmix=0\.05\\alpha\_\{\\text\{mix\}\}=0\.05, a curriculum dropout warm\-up of 15 epochs, and a CrossModal Anchor withdmodel=512d\_\{\\text\{model\}\}=512andH=8H=8heads\. Full hyperparameters are provided in Appendix[A](https://arxiv.org/html/2608.26879#A1)\.

##### Evaluation\.

We report accuracy and macro\-averaged F1\. For MultiHuSE and MUStARD, results are reported as mean±\\pmSD over 5 folds\. For UR\-FUNNY, we use the standard train/dev/test split; the result at seed 999 is reported in Table[3](https://arxiv.org/html/2608.26879#S5.T3)for direct comparability with prior published baselines, and a multi\-seed analysis is reported in Section[5](https://arxiv.org/html/2608.26879#S5)\.

### 4\.4Baselines

We compare against three internal baselines trained on identical features under identical hyperparameters: \(1\)Early Fusion: feature concatenation followed by an MLP; \(2\)Late Fusion: weighted averaging of unimodal prediction vectors; \(3\)Symmetric Fusion: mutual cross\-attention across all three modalities using the same distilled features as IAF, isolating the inverted asymmetric design\. Comparisons with prior published work appear in the results tables in Section[5](https://arxiv.org/html/2608.26879#S5)\.

## 5Results and Analysis

Table[1](https://arxiv.org/html/2608.26879#S5.T1)reports unimodal performance\. Distillation consistently improves student models \(gains of 0\.88–6\.23%, mean 2\.44%\), with the largest gain on UR\-FUNNY’s COVAREP audio \(\+\+6\.23%\), highlighting the benefit of cross\-architecture distillation\. These distilled students, together with their teachers, serve as the inputs to IAF and symmetry fusion\.

Modality / FeatureBase Acc\(%\)Base F1\(%\)Dist\. Acc\(%\)Dist\. F1\(%\)MultiHuSE— 5\-class humor style \| Text≫\\ggAudio≫\\ggVideoText: BERT68\.26±\\pm2\.256870\.21±\\pm2\.8270Text: E5⋆74\.86±\\pm3\.0875––Audio: Dasheng60\.45±\\pm1\.586062\.90±\\pm1\.8363Video: PE\-Core43\.66±\\pm1\.294345\.57±\\pm1\.2246Video: VideoMAE40\.13±\\pm0\.734041\.01±\\pm1\.6041UR\-FUNNY— binary humor \| Text\>\>Audio≈\\approxVideoText: GloVe⋆62\.476465\.1965Audio: COVAREP57\.554863\.7864Video: OpenFace57\.755858\.9559MUStARD— binary sarcasm \| Audio≈\\approxVideo\>\>TextText: BERT69\.42±\\pm2\.806972\.03±\\pm2\.4972Text: E570\.29±\\pm2\.007072\.32±\\pm2\.2672Audio: Dasheng⋆78\.12±\\pm5\.4578––Video: PE\-Core77\.10±\\pm3\.1377––Video: VideoMAE74\.49±\\pm1\.1674––

Table 1:Unimodal results across all three benchmarks\.Base: raw pre\-trained;Dist\.: knowledge\-distilled\.⋆= dominant modality\. MultiHuSE and MUStARD: mean±\\pmSD over 5\-fold CV; UR\-FUNNY: single split\.Table[2](https://arxiv.org/html/2608.26879#S5.T2)presents trimodal fusion results on MultiHuSE\. Early and late fusion fail to reliably surpass the dominant unimodal text baseline, with early fusion degrading sharply in the E5\+VideoMAE setting \(−10\.55%\-10\.55\\%\), suggesting that naive concatenation may disrupt strong text representations when combined with noisier video inputs\. Symmetric fusion also fails to recover the E5 baseline in both E5 configurations \(−\-2\.49%,−\-0\.54%\), indicating that expressive attention alone is insufficient to preserve the dominant modality\. IAF is the only architecture that consistently exceeds the unimodal ceiling across all four configurations, attaining gains of 2\.06–6\.48%\. Paired Wilcoxon signed\-rank tests across the five folds show that IAF significantly outperforms symmetric fusion in the PE\-Core configurations \(BERT:W=0W\{=\}0,p=0\.031p\{=\}0\.031; E5:W=0W\{=\}0,p=0\.031p\{=\}0\.031; one\-tailed\), a result further supported by pairedtt\-tests \(p=0\.023p\{=\}0\.023andp=0\.005p\{=\}0\.005, respectively\); Here,W=0W\{=\}0indicates IAF outperformed symmetric fusion on every fold\.

TextVideoArchitectureAccF1Δ\\DeltaAudio: Dasheng used across all configurationsText: BERT \(base 68\.26%\)BERTVideoMAEEarly Fusion67\.22±\\pm2\.0067−1\.04\-1\.04Late Fusion68\.05±\\pm2\.1268−0\.21\-0\.21Symmetric Fusion69\.42±\\pm1\.4869\+1\.16\+1\.16Proposed \(IAF\)73\.16±\\pm2\.5972\+4\.90\+4\.90BERTPE\-CoreEarly Fusion70\.38±\\pm2\.5770\+2\.12\+2\.12Late Fusion68\.63±\\pm2\.3668\+0\.37\+0\.37Symmetric Fusion72\.00±\\pm1\.2070\+3\.74\+3\.74Proposed \(IAF\)74\.74±\\pm1\.7174\+6\.48\+6\.48Text: E5 \(base 74\.86%\)E5VideoMAEEarly Fusion64\.31±\\pm1\.5164−10\.55\-10\.55Late Fusion71\.25±\\pm4\.0771−3\.61\-3\.61Symmetric Fusion72\.37±\\pm2\.7372−2\.49\-2\.49Proposed \(IAF\)76\.92±\\pm3\.5376\+2\.06\+2\.06E5PE\-CoreEarly Fusion76\.28±\\pm2\.9176\+1\.42\+1\.42Late Fusion71\.54±\\pm4\.2071−3\.32\-3\.32Symmetric Fusion74\.32±\\pm3\.0374−0\.54\-0\.54Proposed \(IAF\)78\.06±\\pm3\.8278\+3\.20\+3\.20

Table 2:Trimodal fusion results on MultiHuSE\.Δ\\Delta= gain over the strongest unimodal text baseline \(BERT = 68\.26%; E5 = 74\.86%\)\. All metrics in %\.Table[3](https://arxiv.org/html/2608.26879#S5.T3)compares IAF against prior work and our internal baselines on UR\-FUNNY\. IAF achieves 70\.72% accuracy, surpassing the strongest prior method \(MISA, 68\.60%\) by 2\.12% and symmetric fusion by 1\.81%\. Interestingly, symmetric fusion \(68\.91%\) outperforms previously published methods, demonstrating that strengthening weaker modalities before fusion is beneficial even without architectural asymmetry\. To assess initialisation sensitivity, we evaluated IAF over five random seeds \(42, 100, 123, 600, 999\), obtaining a mean accuracy of69\.70±0\.61%69\.70\\pm 0\.61\\%\(min69\.32%69\.32\\%; seed 999 is reported in Table[3](https://arxiv.org/html/2608.26879#S5.T3)for comparability with prior work\)\. A one\-samplett\-test shows that this mean significantly exceeds both symmetric fusion \(t⁡\(4\)=2\.90t\(4\)\{=\}2\.90,p=0\.022p\{=\}0\.022, one\-tailed\) and MISA \(t⁡\(4\)=4\.03t\(4\)\{=\}4\.03,p=0\.008p\{=\}0\.008, one\-tailed\), and each seed individually outperforms both baselines\.

MethodContextAcc \(%\)F1 \(%\)Δuni\\Delta\_\{\\text\{uni\}\}Prior published methodsC\-MFN[Kamrul Hasan et al\. \(2019\)](https://arxiv.org/html/2608.26879#bib.bib54)×64\.47––C\-MFN[Kamrul Hasan et al\. \(2019\)](https://arxiv.org/html/2608.26879#bib.bib54)✓65\.23––AGM\(Late fusion\)[Li et al\. \(2023a\)](https://arxiv.org/html/2608.26879#bib.bib9)×65\.97––AGM\(Early fusion\)[Li et al\. \(2023a\)](https://arxiv.org/html/2608.26879#bib.bib9)×66\.07––MULTIBENCH[Pu Liang et al\. \(2021\)](https://arxiv.org/html/2608.26879#bib.bib32)✓66\.70––BCFNet[Deng et al\. \(2025\)](https://arxiv.org/html/2608.26879#bib.bib7)×66\.7868–HF[Choube and Soleymani \(2020\)](https://arxiv.org/html/2608.26879#bib.bib47)✓67\.8469–MISA \(GloVe\)[Hazarika et al\. \(2020\)](https://arxiv.org/html/2608.26879#bib.bib25)×68\.60––Our internal baselines — Text: GloVe \(base 62\.47%\)Early Fusion✓65\.1966\+2\.72\+2\.72Late Fusion✓64\.1965\+1\.72\+1\.72Symmetric Fusion✓68\.9169\+6\.44\+6\.44Proposed \(IAF\)✓70\.72†71\+8\.25\+8\.25

Table 3:Results on UR\-FUNNY\. All methods use GloVe/COVAREP/OpenFace features[Kamrul Hasan et al\. \(2019\)](https://arxiv.org/html/2608.26879#bib.bib54)\.Δuni\\Delta\_\{\\text\{uni\}\}= gain over the GloVe baseline \(62\.47%\) for our internal baselines\. Context: ✓ = punchline \+ context; × = punchline only\.†Seed 999; mean69\.70±0\.61%69\.70\\pm 0\.61\\%over 5 seeds\.Table[4](https://arxiv.org/html/2608.26879#S5.T4)reports results on MUStARD\. Symmetric fusion already improves substantially over the strongest unimodal audio baseline \(\+3\.47% and \+4\.34% for BERT and E5, respectively\), and IAF yields further gains in both text\-encoder settings\. In particular, IAF \(E5\) attains 83\.33% accuracy, exceeding the state\-of\-the\-art method \(MHA with RoBERTa, 79\.32%\) by 4\.01%\. We note that the IAF\-over\-symmetric margin on MUStARD \(0\.87–1\.02 pp\) falls within one standard deviation of both models and should be interpreted accordingly\. Paired Wilcoxon signed\-rank tests are consistent with this pattern: neither BERT\+PE\-Core \(W=2W\{=\}2,p=0\.188p\{=\}0\.188\) nor E5\+PE\-Core \(W=1W\{=\}1,p=0\.125p\{=\}0\.125\) is significant \(one\-tailed,α=0\.05\\alpha\{=\}0\.05\)\.

MethodContextAcc\(%\)F1\(%\)Δuni\\Delta\_\{\\text\{uni\}\}Prior published methodsMULTIBENCH[Pu Liang et al\. \(2021\)](https://arxiv.org/html/2608.26879#bib.bib32)✓71\.8±\\pm0\.3—–SVM \(BERT\)[Castro et al\. \(2019\)](https://arxiv.org/html/2608.26879#bib.bib53)✓—71\.5–SWIA[Chauhan et al\. \(2020\)](https://arxiv.org/html/2608.26879#bib.bib49)✓—72\.6–CE \(BART\)[Ray et al\. \(2022\)](https://arxiv.org/html/2608.26879#bib.bib2)✓—74\.2–MuLOT \(BERT\)[Pramanick et al\. \(2022\)](https://arxiv.org/html/2608.26879#bib.bib36)×74\.52—–IWAN \(BERT\)[Wu et al\. \(2021\)](https://arxiv.org/html/2608.26879#bib.bib30)×—74\.5–IWAN \(BERT\)[Wu et al\. \(2021\)](https://arxiv.org/html/2608.26879#bib.bib30)✓—75\.1–MuLOT \(BERT\)[Pramanick et al\. \(2022\)](https://arxiv.org/html/2608.26879#bib.bib36)✓76\.82—–MHA \(RoBERTa\)[Aggarwal et al\. \(2023\)](https://arxiv.org/html/2608.26879#bib.bib41)✓79\.3277\.6–Our internal baselines – Audio: Dasheng \(base 78\.12%\)Early Fusion✓79\.71±\\pm3\.0480\.0\+1\.59\+1\.59Late Fusion \(Average\)✓79\.86±\\pm4\.7180\.0\+1\.74\+1\.74Symmetric Fusion \(BERT\)✓81\.59±\\pm4\.1782\.0\+3\.47\+3\.47Symmetric Fusion \(E5\)✓82\.46±\\pm4\.5282\.0\+4\.34\+4\.34Proposed \(IAF \(BERT\)\)✓82\.61±\\pm3\.9283\.0\+4\.49\+4\.49Proposed \(IAF \(E5\)\)✓83\.33±\\pm4\.2084\.0\+5\.21\+5\.21

Table 4:Results on MUStARD\.Δuni\\Delta\_\{\\text\{uni\}\}= gain over Dasheng audio unimodal baseline \(our internal baselines only\)\.Table[5](https://arxiv.org/html/2608.26879#S5.T5)summarises pathway isolation analysis\. Across all three datasets, IAF preserves the dominant modality’s pathway at its unimodal baseline, whereas symmetric fusion degrades the same pathways by up to 18\.5pp: the E5 text pathway drops from 74\.9% \(Table[1](https://arxiv.org/html/2608.26879#S5.T1)\) to 56\.4% under symmetric fusion \(Table[5](https://arxiv.org/html/2608.26879#S5.T5)\)\. This directly supports our claim that IAF’s structural asymmetry functions as a representation\-preservation guarantee that symmetric fusion cannot provide\. Visual summaries of this collapse pattern are provided in Appendix[E](https://arxiv.org/html/2608.26879#A5)\.

TextVideoArchi\.Full FusionText PathAudio PathVideo PathMultiHuSE – Dominant Modality: TEXT – BERT=70\.2%, E5=74\.9%BERTVideoMAESymmetric69\.42±\\pm1\.4861\.3±\\pm2\.559\.0±\\pm4\.560\.1±\\pm1\.3IAF73\.16±\\pm2\.5970\.2±\\pm2\.855\.8±\\pm7\.457\.5±\\pm5\.0PE\-CoreSymmetric72\.00±\\pm1\.2063\.5±\\pm1\.663\.6±\\pm2\.361\.6±\\pm7\.2IAF74\.74±\\pm1\.7170\.2±\\pm2\.862\.9±\\pm2\.565\.6±\\pm3\.1E5VideoMAESymmetric72\.37±\\pm2\.7356\.4±\\pm4\.663\.7±\\pm9\.267\.4±\\pm3\.3IAF76\.92±\\pm3\.5374\.9±\\pm3\.148\.4±\\pm8\.956\.7±\\pm7\.0PE\-CoreSymmetric74\.32±\\pm3\.0355\.3±\\pm8\.765\.5±\\pm2\.269\.7±\\pm4\.8IAF78\.06±\\pm3\.8274\.9±\\pm3\.154\.9±\\pm9\.662\.2±\\pm6\.6UR\-FUNNY – Dominant Modality: TEXT=65\.19%GloVeOpenFaceSymmetric68\.9163\.0849\.3050\.70IAF70\.7265\.1950\.9149\.30MUStARD– Dominant Modalities: AUDIO=78\.12%; VIDEO \(PE\)=77\.10%BERTVideoMAESymmetric79\.28±\\pm4\.2662\.0±\\pm5\.064\.4±\\pm15\.267\.8±\\pm8\.0IAF80\.43±\\pm2\.6364\.6±\\pm9\.078\.1±\\pm5\.574\.5±\\pm1\.2PE\-CoreSymmetric81\.59±\\pm4\.1764\.9±\\pm9\.569\.4±\\pm6\.373\.5±\\pm9\.0IAF82\.61±\\pm3\.9263\.9±\\pm6\.278\.1±\\pm5\.577\.1±\\pm3\.1E5VideoMAESymmetric79\.86±\\pm3\.0960\.3±\\pm12\.377\.3±\\pm5\.171\.5±\\pm6\.2IAF79\.42±\\pm4\.3466\.4±\\pm7\.678\.1±\\pm5\.574\.5±\\pm1\.2PE\-CoreSymmetric82\.46±\\pm4\.5274\.4±\\pm3\.574\.9±\\pm4\.673\.3±\\pm10\.4IAF83\.33±\\pm4\.2069\.0±\\pm7\.878\.1±\\pm5\.577\.1±\\pm3\.1

Table 5:Pathway isolation analysis\. Each score reflects accuracy using only the corresponding modality pathway, bypassing fusion aggregation\.Bold= dominant modality pathway\. All metrics in %\.### 5\.1Ablation Study

##### \(1\) Role of Modalities\.

We assess modality contribution through bimodal ablations, reported in Appendix[B](https://arxiv.org/html/2608.26879#A2)\. Across all three datasets, full trimodal IAF remains the best configuration\. The ablations are consistent with the empirical modality hierarchies identified by unimodal validation: removing text hurts most on MultiHuSE and UR\-FUNNY, whereas on MUStARD the text\-free A\+V setting remains close to full trimodal performance, reflecting the stronger audio\-visual signal\.

##### \(2\) Role of Distilled Feature Representations\.

We contrast IAF trained on raw pre\-trained features with IAF trained on distillation\-enhanced features, isolating the impact of Stage 1\. As seen in Table[6](https://arxiv.org/html/2608.26879#S5.T6), incorporating distilled representations improves performance: gains of 1\.02–2\.41% on MultiHuSE, 1\.02–1\.59% on MUStARD, and 1\.81% on UR\-FUNNY\. This underscores the importance of raising weaker modality representations before fusion, rather than relying solely on the architecture\.

DatasetConfigFeat\.Acc\(%\)±\\pmSDF1\(%\)Δ\\Delta\(%\)MultiHuSEBERT\+PECBase72\.33±\\pm2\.0272\-2\.41Dist74\.74±\\pm1\.7174–E5\+PECBase75\.90±\\pm6\.1776\-2\.16Dist78\.06±\\pm3\.8278–MUStARDBERT\+PECBase81\.59±\\pm3\.8582\-1\.02Dist82\.61±\\pm3\.9283–E5\+PECBase81\.74±\\pm4\.2482\-1\.59Dist83\.33±\\pm4\.2084–UR\-FUNNYGloVe/COV/OFBase68\.9169\-1\.81Dist70\.7271–

Table 6:Distillation ablation: raw \(Base\) vs\. distilled \(Dist\.\) features within IAF\.Δ\\Delta= Dist−\-Base\.
##### \(3\) Role of Regularisation\.

Appendix[C](https://arxiv.org/html/2608.26879#A3), Table[11](https://arxiv.org/html/2608.26879#A3.T11), reports the contribution of Mixup and Curriculum Modality Dropout\. Removing both yields the largest drops across all datasets \(up to 4\.07% on MultiHuSE, 1\.61% on UR\-FUNNY, and 2\.90% on MUStARD\), confirming their complementary roles\. Curriculum Dropout provides a consistently larger isolated gain than Mixup, suggesting that progressive difficulty scheduling is particularly effective in noisy multimodal feature spaces\.

##### \(4\) Role of the Gate Mechanism\.

We evaluate three gating strategies within IAF: aLearnablegate \(sample\-adaptive weights\), aHierarchygate \(fixed weights derived from modality ranking\), and aUniformgate \(equal weights, T=A=V=0\.33\)\. Results \(Appendix[D](https://arxiv.org/html/2608.26879#A4), Table[12](https://arxiv.org/html/2608.26879#A4.T12)\) shows that the learnable gate generally performs best, while the hierarchy gate remains competitive under strong modality imbalance\. Appendix[F](https://arxiv.org/html/2608.26879#A6)further shows that mean gate weights remain distributed across modalities in all configurations, with no modality exceeding 0\.53 on average\.

## 6Conclusion

We introduced Inverted Asymmetric Fusion \(IAF\), a two\-stage multimodal framework for mitigating strong\-modality collapse in discriminative fusion\. By preserving the dominant modality through apure/attendhierarchy and strengthening weaker modalities via knowledge distillation prior to fusion, IAF maintains the dominant pathway at its unimodal ceiling while improving fused performance in text\-dominant and audio\-visual\-dominant settings\. Across three benchmarks with different modality hierarchies, IAF consistently outperforms early, late, and symmetric fusion baselines, and surpasses prior published methods on UR\-FUNNY and MUStARD\. These findings suggest that protecting strong modalities from cross\-modal interference is an important architectural principle for robust multimodal learning\.

## Limitations

This work has several limitations\. First, IAF requires the dominant modality to be identified empirically through unimodal evaluation before fusion training\. When modalities have similar predictive strength, thepure/attendassignment becomes less clear and the benefits of structural asymmetry may diminish\. Moreover, the inferred hierarchy depends on the encoder set: a stronger text encoder may appear dominant over a weaker video encoder even when the task itself relies more heavily on visual information\. The hierarchy therefore reflects empirical encoder performance rather than intrinsic task structure\. Although the dual\-teacher design used for MUStARD partially addresses co\-dominant settings, a more principled treatment remains open, for example through dynamic hierarchy assignment at the sample level\.

Second, all experiments use frozen, pre\-extracted features\. This isolates the architectural contribution of IAF from encoder fine\-tuning, but the framework has not been evaluated end to end, where joint optimisation could alter both the modality hierarchy and the effectiveness of distillation\. In addition, our pathway analysis uses the original frozen unimodal classifier as a fixed probe, so it tests whether the dominant modality’s decision boundary is preserved rather than whether it remains recoverable in principle\. Future work could examine this more directly through probing or representational analyses such as Centred Kernel Alignment \(CKA\)\.

Finally, all datasets are English\-only and confined to humor and sarcasm\. It therefore remains unclear whether the modality hierarchies observed here, and the benefits of IAF, generalise to other languages or to tasks such as emotion recognition, visual question answering, or action recognition\. Although the inclusion of E5\-multilingual suggests potential cross\-lingual applicability, this has not been evaluated, and cross\-dataset generalisation remains for future work\.

## Ethical considerations

This work uses three publicly available datasets \(MultiHuSE, UR\-FUNNY, MUStARD\), collected and released under standard academic licences by their respective authors\. No new data collection or human annotation was conducted\. All experiments operate on pre\-extracted, anonymised feature representations with no access to personally identifiable information\. We note that automated humor and sarcasm detection systems may misclassify culturally specific communication styles and are not intended for deployment beyond controlled research settings\.

## Acknowledgments

This research was supported by the Petroleum Technology Development Fund \(PTDF\) of Nigeria\.

## References

- S\. Aggarwal, A\. Pandey, and D\. K\. VishwakarmaMultimodal Sarcasm Recognition by Fusing Textual, Visual and Acoustic content via Multi\-Headed Attention for Video Dataset\.In2023 World Conference on Communication and Computing, WCONF 2023,External Links:ISBN 9798350311204,[Document](https://dx.doi.org/10.1109/WCONF58270.2023.10235179)Cited by:[Table 4](https://arxiv.org/html/2608.26879#S5.T4.2.1.11.1)\.
- Albanieet al\.\(2018\)S\. Albanie, A\. Nagrani, A\. Vedaldi, and A\. ZissermanEmotion recognition in speech using cross\-modal transfer in the wild\.InMM 2018 \- Proceedings of the 2018 ACM Multimedia Conference,pp\. 292–301\.External Links:ISBN 9781450356657,[Document](https://dx.doi.org/10.1145/3240508.3240578)Cited by:[Table 16](https://arxiv.org/html/2608.26879#A8.T16.2.1.2.1.2.1.1.1),[§2\.3](https://arxiv.org/html/2608.26879#S2.SS3.p1.1)\.
- Baltrusaitiset al\.\(2019\)T\. Baltrusaitis, C\. Ahuja, and L\. P\. MorencyMultimodal Machine Learning: A Survey and Taxonomy\.Vol\.41,IEEE Computer Society\.External Links:[Document](https://dx.doi.org/10.1109/TPAMI.2018.2798607),ISSN 19393539Cited by:[§1](https://arxiv.org/html/2608.26879#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.26879#S2.SS1.p1.1)\.
- Baltrusaitiset al\.\(2018\)T\. Baltrusaitis, A\. Zadeh, Y\. C\. Lim, and L\. P\. MorencyOpenFace 2\.0: Facial behavior analysis toolkit\.InProceedings \- 13th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2018,pp\. 59–66\.External Links:ISBN 9781538623350,[Document](https://dx.doi.org/10.1109/FG.2018.00019)Cited by:[§4\.2](https://arxiv.org/html/2608.26879#S4.SS2.SSS0.Px2.p1.1)\.
- Bolyaet al\.\(2025\)D\. Bolya, P\. Huang, P\. Sun, J\. H\. Cho, A\. Madotto, C\. Wei, T\. Ma, J\. Zhi, J\. Rajasegaran, H\. Rasheed, J\. Wang, M\. Monteiro, H\. Xu, S\. Dong, N\. Ravi, D\. Li, P\. Dollár, and C\. FeichtenhoferPerception Encoder: The best visual embeddings are not at the output of the network\.arXiv Preprint \(Facebook research\)\.External Links:[Link](http://arxiv.org/abs/2504.13181)Cited by:[§4\.2](https://arxiv.org/html/2608.26879#S4.SS2.SSS0.Px1.p1.1)\.
- Castroet al\.\(2019\)S\. Castro, D\. Hazarika, V\. Pérez\-Rosas, R\. Zimmermann, R\. Mihalcea, and S\. PoriaTowards Multimodal Sarcasm Detection \(An Obviously Perfect Paper\)\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,Florence, Italy,pp\. 4619–4629\.External Links:[Link](https://github./)Cited by:[§1](https://arxiv.org/html/2608.26879#S1.p5.1),[§2\.4](https://arxiv.org/html/2608.26879#S2.SS4.p1.1),[§4\.1](https://arxiv.org/html/2608.26879#S4.SS1.SSS0.Px3.p1.1),[Table 4](https://arxiv.org/html/2608.26879#S5.T4.2.1.4.1)\.
- Chaudhuriet al\.\(2025\)A\. Chaudhuri, A\. Dutta, T\. Bui, and S\. GeorgescuA Closer Look at Multimodal Representation Collapse\.InProceedings of the 42nd International Conference on Machine Learning,Vancouver\.External Links:[Link](http://arxiv.org/abs/2505.22483)Cited by:[§1](https://arxiv.org/html/2608.26879#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.26879#S2.SS2.p1.1)\.
- Chauhanet al\.\(2020\)D\. S\. Chauhan, A\. Ekbal, and P\. BhattacharyyaSentiment and Emotion help Sarcasm? A Multi\-task Learning Framework for Multi\-Modal Sarcasm, Sentiment and Emotion Analysis\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 4351–4360\.External Links:[Link](http://www.iitp.ac./)Cited by:[Table 4](https://arxiv.org/html/2608.26879#S5.T4.2.1.5.1)\.
- Choube and Soleymani \(2020\)A\. Choube and M\. SoleymaniPunchline Detection using Context\-Aware Hierarchical Multimodal Fusion\.InICMI 2020 \- Proceedings of the 2020 International Conference on Multimodal Interaction,pp\. 675–679\.External Links:ISBN 9781450375818,[Document](https://dx.doi.org/10.1145/3382507.3418891)Cited by:[Table 3](https://arxiv.org/html/2608.26879#S5.T3.2.1.9.1)\.
- Degottexet al\.\(2014\)G\. Degottex, J\. Kane, T\. Drugman, T\. Raitio, and S\. SchererCOVAREP: A Collaborative Voice Analysis Repository for Speech Technologies\.InIEEE International Conference on Acoustic, Speech and Signal Processing \(ICASSP\),External Links:[Link](http://covarep.github.io/covarep),ISBN 9781479928934Cited by:[§4\.2](https://arxiv.org/html/2608.26879#S4.SS2.SSS0.Px2.p1.1)\.
- Denget al\.\(2025\)B\. Deng, J\. Li, M\. Gong, Z\. Tang, Y\. Zhang, K\. Feng, and Y\. WuBCFNet: Bi\-temporal collaborative fusion network for multi\-modal humor detection\.Pattern Recognition172\.External Links:[Document](https://dx.doi.org/10.1016/j.patcog.2025.112744),ISSN 00313203Cited by:[Table 3](https://arxiv.org/html/2608.26879#S5.T3.2.1.8.1)\.
- Devlinet al\.\(2019\)J\. Devlin, M\. Chang, K\. Lee, K\. T\. Google, and A\. I\. LanguageBERT: Pre\-training of Deep Bidirectional Transformers for Language Understanding\.InProceedings of NAACL\-HLT,Minnesota,pp\. 4171–4186\.External Links:[Link](https://github.com/tensorflow/tensor2tensor)Cited by:[§4\.2](https://arxiv.org/html/2608.26879#S4.SS2.SSS0.Px1.p1.1)\.
- Dinkelet al\.\(2024\)H\. Dinkel, Z\. Yan, Y\. Wang, J\. Zhang, Y\. Wang, and B\. WangScaling up masked audio encoder learning for general audio classification\.InProceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH,pp\. 547–551\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2024-246),ISSN 19909772Cited by:[§4\.2](https://arxiv.org/html/2608.26879#S4.SS2.SSS0.Px1.p1.1)\.
- Fanet al\.\(2023\)Y\. Fan, W\. Xu, H\. Wang, J\. Wang, and S\. GuoPMR: Prototypical Modal Rebalance for Multimodal Learning\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Vancouver,pp\. 20029–20038\.External Links:[Link](https://github.com/fanyunfeng-bit/Modal-Imbalance-PMR)Cited by:[§1](https://arxiv.org/html/2608.26879#S1.p4.1),[§2\.2](https://arxiv.org/html/2608.26879#S2.SS2.p2.1)\.
- Fanget al\.\(2021\)Z\. Fang, J\. Wang, X\. Hu, L\. Wang, Y\. Yang, and Z\. LiuCompressing Visual\-linguistic Model via Knowledge Distillation\.InProceedings of the IEEE International Conference on Computer Vision,pp\. 1408–1418\.External Links:ISBN 9781665428125,[Document](https://dx.doi.org/10.1109/ICCV48922.2021.00146),ISSN 15505499Cited by:[§2\.3](https://arxiv.org/html/2608.26879#S2.SS3.p1.1)\.
- Fenget al\.\(2024\)X\. Feng, Y\. Lin, L\. He, Y\. Li, L\. Chang, and Y\. ZhouKnowledge\-Guided Dynamic Modality Attention Fusion Framework for Multimodal Sentiment Analysis\.InFindings of the Association for Computational Linguistics: EMNLP,pp\. 14755–14766\.External Links:[Link](https://github.com/)Cited by:[§2\.1](https://arxiv.org/html/2608.26879#S2.SS1.p2.1)\.
- Gonget al\.\(2025\)B\. Gong, X\. Gao, P\. Zhu, Q\. Hu, and B\. CaoMultimodal Negative Learning\.In39th Conference on Neural Information Processing Systems,External Links:[Link](http://arxiv.org/abs/2510.20877)Cited by:[§2\.2](https://arxiv.org/html/2608.26879#S2.SS2.p1.1)\.
- Gouet al\.\(2021\)J\. Gou, B\. Yu, S\. J\. Maybank, and D\. TaoKnowledge Distillation: A Survey\.International Journal of Computer Vision129\(6\),pp\. 1789–1819\.External Links:[Document](https://dx.doi.org/10.1007/s11263-021-01453-z),ISSN 15731405Cited by:[§2\.3](https://arxiv.org/html/2608.26879#S2.SS3.p1.1)\.
- Guptaet al\.\(2016\)S\. Gupta, J\. Hoffman, and J\. MalikCross Modal Distillation for Supervision Transfer\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 2827–2836\.Cited by:[§2\.3](https://arxiv.org/html/2608.26879#S2.SS3.p1.1)\.
- Hazarikaet al\.\(2020\)D\. Hazarika, R\. Zimmermann, and S\. PoriaMISA: Modality\-Invariant and \-Specific Representations for Multimodal Sentiment Analysis\.InMM 2020 \- Proceedings of the 28th ACM International Conference on Multimedia,pp\. 1122–1131\.External Links:ISBN 9781450379885,[Document](https://dx.doi.org/10.1145/3394171.3413678)Cited by:[§2\.1](https://arxiv.org/html/2608.26879#S2.SS1.p1.1),[Table 3](https://arxiv.org/html/2608.26879#S5.T3.2.1.10.1)\.
- Hintonet al\.\(2015\)G\. Hinton, O\. Vinyals, and J\. DeanDistilling the Knowledge in a Neural Network\.arXiv Preprint\.External Links:[Link](http://arxiv.org/abs/1503.02531)Cited by:[§1](https://arxiv.org/html/2608.26879#S1.p4.1),[§2\.3](https://arxiv.org/html/2608.26879#S2.SS3.p1.1)\.
- Huanget al\.\(2022\)Y\. Huang, J\. Lin, C\. Zhou, H\. Yang, and L\. HuangModality Competition: What Makes Joint Training of Multi\-modal Network Fail in Deep Learning? \(Provably\)\.InProceedings of the 39th International Conference on Machine Learning,Maryland\.Cited by:[§1](https://arxiv.org/html/2608.26879#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.26879#S2.SS2.p1.1)\.
- Javaloyet al\.\(2022\)A\. Javaloy, M\. Meghdadi, and I\. ValeraMitigating Modality Collapse in Multimodal VAEs via Impartial Optimization\.InProceedings of the 39th International Conference on Machine Learning,,Maryland\.Cited by:[§1](https://arxiv.org/html/2608.26879#S1.p4.1),[§2\.2](https://arxiv.org/html/2608.26879#S2.SS2.p2.1)\.
- Kamrul Hasanet al\.\(2019\)M\. Kamrul Hasan, W\. Rahman, A\. Zadeh, J\. Zhong, M\. Iftekhar Tanveer, L\. Morency, and M\. HoqueUR\-FUNNY: A Multimodal Language Dataset for Understanding Humor\.InProceedings ofthe 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing,pp\. 2046–2056\.External Links:[Link](https://www.ted.com/)Cited by:[§1](https://arxiv.org/html/2608.26879#S1.p5.1),[§2\.4](https://arxiv.org/html/2608.26879#S2.SS4.p1.1),[§4\.1](https://arxiv.org/html/2608.26879#S4.SS1.SSS0.Px2.p1.1),[Table 3](https://arxiv.org/html/2608.26879#S5.T3),[Table 3](https://arxiv.org/html/2608.26879#S5.T3.2.1.3.1),[Table 3](https://arxiv.org/html/2608.26879#S5.T3.2.1.4.1)\.
- Kennethet al\.\(2024a\)M\. O\. Kenneth, F\. Khosmood, and A\. EdalatA Two\-Model Approach for Humour Style Recognition\.InProceedings of the 4th International Conference on Natural Language Processing for Digital Humanities,Miami,pp\. 259–274\.External Links:[Link](https://aclanthology.org/2024.nlp4dh-1.25/)Cited by:[§2\.4](https://arxiv.org/html/2608.26879#S2.SS4.p1.1)\.
- Kennethet al\.\(2024b\)M\. O\. Kenneth, F\. Khosmood, and A\. EdalatSystematic Literature Review: Computational Approaches for Humour Style Classification\.ArXiv\.External Links:[Link](https://arxiv.org/abs/2402.01759),[Document](https://dx.doi.org/https%3A//doi.org/10.48550/arXiv.2402.01759)Cited by:[§2\.2](https://arxiv.org/html/2608.26879#S2.SS2.p1.1)\.
- Kennethet al\.\(2025a\)M\. O\. Kenneth, F\. Khosmood, and A\. EdalatExplaining Humour Style Classifications: An XAI Approach to Understanding Computational Humour Analysis\.Journal of Data Mining & Digital Humanities, NLP4DH\.External Links:[Link](http://arxiv.org/abs/2501.02891)Cited by:[§2\.4](https://arxiv.org/html/2608.26879#S2.SS4.p1.1)\.
- Kennethet al\.\(2025b\)M\. O\. Kenneth, F\. Khosmood, and A\. EdalatMultiHuSE: A Multimodal Dataset for Humour Styles and Emotions\.InInternational Conference on Content\-Based Multimedia Indexing \(CBMI\),pp\. 1–7\.External Links:[Document](https://dx.doi.org/10.1109/cbmi66578.2025.11339313)Cited by:[§1](https://arxiv.org/html/2608.26879#S1.p1.1),[§1](https://arxiv.org/html/2608.26879#S1.p5.1),[§2\.4](https://arxiv.org/html/2608.26879#S2.SS4.p1.1),[§4\.1](https://arxiv.org/html/2608.26879#S4.SS1.SSS0.Px1.p1.1)\.
- Liet al\.\(2023a\)H\. Li, X\. Li, P\. Hu, Y\. Lei, C\. Li, and Y\. ZhouBoosting Multi\-modal Model Performance with Adaptive Gradient Modulation\.InIEEE/CVF International Conference on Computer Vision \(ICCV\),pp\. 22214–22224\.Cited by:[Table 3](https://arxiv.org/html/2608.26879#S5.T3.2.1.5.1),[Table 3](https://arxiv.org/html/2608.26879#S5.T3.2.1.6.1)\.
- Li and Tang \(2025\)S\. Li and H\. TangMultimodal Alignment and Fusion: A Survey\.International Journal of Computer Vision\.External Links:[Link](http://arxiv.org/abs/2411.17040),[Document](https://dx.doi.org/10.1007/s11263-025-02667-1),ISSN 15731405Cited by:[§1](https://arxiv.org/html/2608.26879#S1.p1.1)\.
- Liet al\.\(2023b\)Y\. Li, Y\. Wang, and Z\. CuiDecoupled Multimodal Distilling for Emotion Recognition\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 6631–6640\.External Links:[Link](https://github.com/mdswyz/DMD.)Cited by:[Table 16](https://arxiv.org/html/2608.26879#A8.T16.2.1.5.1),[§2\.3](https://arxiv.org/html/2608.26879#S2.SS3.p1.1)\.
- Lin and Hu \(2024\)R\. Lin and H\. HuMulti\-Task Momentum Distillation for Multimodal Sentiment Analysis\.IEEE Transactions on Affective Computing15\(2\),pp\. 549–565\.External Links:[Document](https://dx.doi.org/10.1109/TAFFC.2023.3282410),ISSN 19493045Cited by:[Table 16](https://arxiv.org/html/2608.26879#A8.T16.2.1.6.1),[§2\.3](https://arxiv.org/html/2608.26879#S2.SS3.p1.1)\.
- Liuet al\.\(2018\)Z\. Liu, Y\. Shen, V\. B\. Lakshminarasimhan, P\. P\. Liang, A\. Zadeh, and L\. MorencyEfficient Low\-rank Multimodal Fusion with Modality\-Specific Factors\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics,pp\. 2247–2256\.Cited by:[§2\.1](https://arxiv.org/html/2608.26879#S2.SS1.p1.1)\.
- Nagraniet al\.\(2021\)A\. Nagrani, S\. Yang, A\. Arnab, A\. Jansen, C\. Schmid, C\. Sun, and G\. ResearchAttention Bottlenecks for Multimodal Fusion\.In35th Conference on Neural Information Processing Systems,Cited by:[§2\.1](https://arxiv.org/html/2608.26879#S2.SS1.p1.1)\.
- Penget al\.\(2022\)X\. Peng, Y\. Wei, A\. Deng, D\. Wang, and D\. HuBalanced Multimodal Learning via On\-the\-fly Gradient Modulation\.InIEEE Conference on Computer Vision and Pattern Recognition,pp\. 8238–8247\.External Links:[Link](https://github.com/GeWu-)Cited by:[§1](https://arxiv.org/html/2608.26879#S1.p1.1)\.
- Penningtonet al\.\(2014\)J\. Pennington, R\. Socher, and C\. D\. ManningGloVe: Global Vectors for Word Representation\.InProceedings of the 2014 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 1532–1543\.External Links:[Link](http://nlp./)Cited by:[§4\.2](https://arxiv.org/html/2608.26879#S4.SS2.SSS0.Px2.p1.1)\.
- Pramanicket al\.\(2022\)S\. Pramanick, A\. Roy, and V\. M\. PatelMultimodal Learning using Optimal Transport for Sarcasm and Humor Detection\.InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision \(WACV\),pp\. 3930–3940\.Cited by:[Table 4](https://arxiv.org/html/2608.26879#S5.T4.2.1.10.1),[Table 4](https://arxiv.org/html/2608.26879#S5.T4.2.1.7.1)\.
- Pu Lianget al\.\(2021\)P\. Pu Liang, Y\. Lyu, X\. Fan, Z\. Wu, Y\. Cheng, J\. Wu, L\. Chen, P\. Wu, M\. A\. Lee, Y\. Zhu, R\. Salakhutdinov, L\. Morency, and J\. HopkinsMULTIBENCH: Multiscale Benchmarks for Multimodal Representation Learning\.In35th Conference on Neural Information Processing Systems, Track on Datasets and Benchmarks\.,pp\. 1–20\(English\)\.External Links:[Link](https://cmu-multicomp-lab.github.io/multibench/)Cited by:[§2\.1](https://arxiv.org/html/2608.26879#S2.SS1.p1.1),[Table 3](https://arxiv.org/html/2608.26879#S5.T3.2.1.7.1),[Table 4](https://arxiv.org/html/2608.26879#S5.T4.2.1.3.1)\.
- Rahmanet al\.\(2020\)W\. Rahman, M\. K\. Hasan, S\. Lee, A\. Zadeh, C\. Mao, L\. Morency, and E\. HoqueIntegrating Multimodal Information in Large Pretrained Transformers\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 2359–2369\.Cited by:[§2\.1](https://arxiv.org/html/2608.26879#S2.SS1.p1.1)\.
- Rayet al\.\(2022\)A\. Ray, S\. Mishra, A\. Nunna, and P\. BhattacharyyaA Multimodal Corpus for Emotion Recognition in Sarcasm\.InProceedings of the 13th Conference on Language Resources and Evaluation,pp\. 20–25\.External Links:[Link](https://github.com/apoorva-nunna/MUStARD_Plus_Plus)Cited by:[Table 4](https://arxiv.org/html/2608.26879#S5.T4.2.1.6.1)\.
- Thoker and Gall \(2019\)F\. M\. Thoker and J\. GallCross\-Modal Knowledge Distillation for Action Recognition\.In2019 IEEE International Conference on Image Processing \(ICIP\),External Links:ISBN 9781538662496Cited by:[Table 16](https://arxiv.org/html/2608.26879#A8.T16.2.1.2.1.2.1.2.1),[§2\.3](https://arxiv.org/html/2608.26879#S2.SS3.p1.1)\.
- Tonget al\.\(2022\)Z\. Tong, Y\. Song, J\. Wang, and L\. WangVideoMAE: Masked Autoencoders are Data\-Efficient Learners for Self\-Supervised Video Pre\-Training\.In36th Conference on Neural Information Processing Systems 9\(NeurIPS\),New Orleans\.External Links:[Link](https://github.com/MCG-NJU/VideoMAE.)Cited by:[§4\.2](https://arxiv.org/html/2608.26879#S4.SS2.SSS0.Px1.p1.1)\.
- Tsaiet al\.\(2020\)Y\. H\. H\. Tsai, S\. Bai, P\. P\. Liang, J\. Zico Kolter, L\. P\. Morency, and R\. SalakhutdinovMultimodal transformer for unaligned multimodal language sequences\.InACL 2019 \- 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference,pp\. 6558–6569\.External Links:ISBN 9781950737482,[Document](https://dx.doi.org/10.18653/v1/p19-1656)Cited by:[§2\.1](https://arxiv.org/html/2608.26879#S2.SS1.p1.1)\.
- Vaswaniet al\.\(2017\)A\. Vaswani, G\. Brain, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. PolosukhinAttention Is All You Need\.In31st Conference on Neural Information Processing Systems,Cited by:[§3\.3\.1](https://arxiv.org/html/2608.26879#S3.SS3.SSS1.p1.2)\.
- Wanget al\.\(2024\)L\. Wang, N\. Yang, X\. Huang, L\. Yang, R\. Majumder, and F\. WeiMultilingual E5 Text Embeddings: A Technical Report\.arXiv preprint\.External Links:[Link](http://arxiv.org/abs/2402.05672)Cited by:[§4\.2](https://arxiv.org/html/2608.26879#S4.SS2.SSS0.Px1.p1.1)\.
- Wanget al\.\(2020a\)Q\. Wang, L\. Zhan, P\. Thompson, and J\. ZhouMultimodal Learning with Incomplete Modalities by Knowledge Distillation\.InProceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,pp\. 1828–1838\.External Links:ISBN 9781450379984,[Document](https://dx.doi.org/10.1145/3394486.3403234)Cited by:[Table 16](https://arxiv.org/html/2608.26879#A8.T16.2.1.3.1),[§2\.3](https://arxiv.org/html/2608.26879#S2.SS3.p1.1)\.
- Wanget al\.\(2023\)T\. Wang, W\. Zhou, E\. Zurich, Y\. Zeng, and X\. ZhangEfficientVLM: Fast and Accurate Vision\-Language Models via Knowledge Distillation and Modal\-adaptive Pruning\.InFindings of the Association for Computational Linguistics,pp\. 13899–13913\.External Links:[Link](https://github.com/swaggy-TN/EfficientVLM.)Cited by:[§2\.3](https://arxiv.org/html/2608.26879#S2.SS3.p1.1)\.
- Wanget al\.\(2020b\)W\. Wang, D\. Tran, and M\. FeiszliWhat Makes Training Multi\-Modal Classification Networks Hard?\.InProceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition,pp\. 12692–12702\.External Links:[Document](https://dx.doi.org/10.1109/CVPR42600.2020.01271),ISSN 10636919Cited by:[§1](https://arxiv.org/html/2608.26879#S1.p1.1),[§1](https://arxiv.org/html/2608.26879#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.26879#S2.SS2.p1.1)\.
- Weiet al\.\(2023\)S\. Wei, C\. Luo, and Y\. LuoMMANet: Margin\-aware Distillation and Modality\-aware Regularization for Incomplete Multimodal Learning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 20039–20049\.External Links:[Link](https://github.com/shicaiwei123/MMANet)Cited by:[Table 16](https://arxiv.org/html/2608.26879#A8.T16.2.1.4.1),[§2\.3](https://arxiv.org/html/2608.26879#S2.SS3.p1.1)\.
- Weiet al\.\(2025\)Y\. Wei, D\. Hu, H\. Du, and J\. R\. WenOn\-the\-Fly Modulation for Balanced Multimodal Learning\.IEEE Transactions on Pattern Analysis and Machine Intelligence47\(1\),pp\. 469–485\.External Links:[Document](https://dx.doi.org/10.1109/TPAMI.2024.3468315),ISSN 19393539Cited by:[§1](https://arxiv.org/html/2608.26879#S1.p4.1),[§2\.2](https://arxiv.org/html/2608.26879#S2.SS2.p2.1)\.
- Wei and Hu \(2024\)Y\. Wei and D\. HuMMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance\.InProceedings of the 41st International Conference on Machine Learning,Vienna\.External Links:[Link](http://arxiv.org/abs/2405.17730)Cited by:[§2\.2](https://arxiv.org/html/2608.26879#S2.SS2.p2.1)\.
- Wuet al\.\(2022\)N\. Wu, S\. Jastrz˛ Ebski, K\. Cho, and K\. J\. GerasCharacterizing and Overcoming the Greedy Nature of Learning in Multi\-modal Deep Neural Networks\.InProceedings of the 39th International Conference on Machine Learning,External Links:[Link](https://github.com/nyukat/greedy_multimodal_learning)Cited by:[§2\.2](https://arxiv.org/html/2608.26879#S2.SS2.p1.1)\.
- Wuet al\.\(2021\)Y\. Wu, Y\. Zhao, X\. Lu, B\. Qin, Y\. Wu, J\. Sheng, and J\. LiModeling Incongruity between Modalities for Multimodal Sarcasm Detection\.IEEE Multimedia28\(2\),pp\. 86–95\.External Links:[Document](https://dx.doi.org/10.1109/MMUL.2021.3069097),ISSN 19410166Cited by:[Table 4](https://arxiv.org/html/2608.26879#S5.T4.2.1.8.1),[Table 4](https://arxiv.org/html/2608.26879#S5.T4.2.1.9.1)\.
- Zadehet al\.\(2017\)A\. Zadeh, M\. Chen, S\. Poria, E\. Cambria, and L\. MorencyTensor Fusion Network for Multimodal Sentiment Analysis\.InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing,Denmark,pp\. 1103–1114\(English\)\.Cited by:[§2\.1](https://arxiv.org/html/2608.26879#S2.SS1.p1.1)\.
- Zhanget al\.\(2024\)X\. Zhang, J\. Yoon, M\. Bansal, and H\. YaoMultimodal Representation Learning by Alternating Unimodal Adaptation\.InIEEE / CVF Computer Vision and Pattern Recognition Conference \(CVPR\),External Links:[Link](https://github.com/Cecile-hi/MLA.)Cited by:[§2\.2](https://arxiv.org/html/2608.26879#S2.SS2.p1.1)\.
- Zhanget al\.\(2025\)Y\. Zhang, B\. Chen, H\. Ye, Z\. Gao, T\. Wan, L\. Lan, and K\. XuText\-guided Multimodal Fusion for the Multimodal Emotion and Intent Joint Understanding\.InICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing \- Proceedings,External Links:[Document](https://dx.doi.org/10.1109/ICASSP49660.2025.10890680),ISSN 15206149Cited by:[§2\.1](https://arxiv.org/html/2608.26879#S2.SS1.p2.1)\.

## Appendix AFull Hyperparameter Configurations

Table[8](https://arxiv.org/html/2608.26879#A1.T8)and Table[7](https://arxiv.org/html/2608.26879#A1.T7)present the complete hyperparameter configurations for the distillation and fusion stages, across all datasets\. Hyperparameters were selected through manual tuning: values were adjusted iteratively based on validation performance until stable, well\-performing configurations were identified, after which a single final experiment was run per configuration and reported in the paper\. No automated search was conducted\.

HyperparameterMultiHuSEUR\-FUNNYMUStARDOptimiserAdamWAdamAdamWLearning rate6×10−56\\times 10^\{\-5\}1×10−51\\times 10^\{\-5\}1×10−31\\times 10^\{\-3\}Weight decay10−310^\{\-3\}10−310^\{\-3\}10−210^\{\-2\}Max epochs10080100Early stopping patience352040Gradient clipping1\.0—1\.0

Table 7:Dataset\-specific hyperparameters for the fusionHyperparameterValueKD loss weights\(α,β,γ\)\(\\alpha,\\beta,\\gamma\)0\.4, 0\.35, 0\.25TemperatureTT3\.5OptimiserAdamWWeight decay10−310^\{\-3\}Text/Audio LR10−410^\{\-4\}–10−310^\{\-3\}Video LR9×10−59\\times 10^\{\-5\}Table 8:Distillation stage hyperparameters\.
## Appendix BRole of Modalities Ablation

We remove one modality at a time to quantify each modality’s contribution to full trimodal performance\. Table[9](https://arxiv.org/html/2608.26879#A2.T9)shows that full trimodal IAF outperforms all bimodal variants on every dataset\. On MultiHuSE, the text\-free A\+V pair drops to 65\.85%, confirming text’s dominant role\. On UR\-FUNNY,T\+AT\{\+\}AoutperformsT\+VT\{\+\}V\(68\.91% vs\. 65\.29%\), consistent with the Text\>\>Audio≈\\approxVideo hierarchy\. On MUStARD, the text\-free A\+V pair \(82\.61%\) nearly matches full trimodal performance \(83\.33%\), reflecting the co\-dominance of audio and video and the smaller marginal benefit of text\.

ConfigurationPureAttendsAcc\(%\)±\\pmSDF1\(%\)MultiHuSE — Text dominant \(E5, Dasheng, PE\-Core\)Full TrimodalTextAudio, Video78\.06±\\pm3\.8278T\+VTextVideo76\.61±\\pm4\.5877T\+ATextAudio76\.57±\\pm2\.9276A\+V \(Text\-Free\)AudioVideo65\.85±\\pm1\.6365UR\-FUNNY — Text dominant \(GloVe, COVAREP, OpenFace\)Full TrimodalTextAudio, Video70\.7271T\+ATextAudio68\.9169T\+VTextVideo65\.2965A\+V \(Text\-Free\)AudioVideo65\.2965MUStARD — Audio/Video dominant \(E5, Dasheng, PE\-Core\)Full TrimodalAudio, VideoText83\.33±\\pm4\.2084A\+TAudioText79\.57±\\pm5\.1780V\+TVideoText79\.57±\\pm5\.4180A\+V \(Text\-Free\)AudioVideo82\.61±\\pm3\.0183

Table 9:Bimodal ablation of E5 and GloVe\-based configuration for IAFTable[10](https://arxiv.org/html/2608.26879#A2.T10)reports bimodal ablation results for BERT encoder configurations on MultiHuSE and MUStARD\. Results follow the same directional patterns as the E5 configurations in Table[9](https://arxiv.org/html/2608.26879#A2.T9), confirming that the modality hierarchy findings are consistent across text encoders\.

ConfigurationPureAttendsAcc \(%\)±\\pmSDF1\(%\)MultiHuSE — Text dominant \(BERT, Dasheng, PE\-Core\)Full TrimodalTextAudio, Video74\.74±\\pm1\.7174T\+VTextVideo72\.45±\\pm3\.2772T\+ATextAudio72\.08±\\pm2\.7172MUStARD — Audio/Video dominant \(BERT, Dasheng, PE\-Core\)Full TrimodalAudio, VideoText82\.61±\\pm3\.9283A\+TAudioText79\.71±\\pm3\.3080V\+TVideoText77\.25±\\pm4\.7577

Table 10:Bimodal fusion for BERT configurations on MultiHuSE and MUStARD\.
## Appendix CRole of Regularisation Ablation

Table[11](https://arxiv.org/html/2608.26879#A3.T11)reports the full regularisation ablation across all datasets and encoder configurations\. BERT\- and E5\-based configurations are shown for MultiHuSE and MUStARD; results are directionally consistent across both encoders\.

MixupCurriculumAcc \(%\)±\\pmSDF1Δ\\Delta\(%\)MultiHuSE\(BERT \+ Dasheng \+ PE\-Core\)✓✓74\.74±\\pm1\.7174–✗✓73\.49±\\pm3\.1573−1\.25\-1\.25✓✗72\.58±\\pm3\.8072−2\.16\-2\.16✗✗70\.75±\\pm2\.6770−3\.99\-3\.99MultiHuSE\(E5 \+ Dasheng \+ PE\-Core\)✓✓78\.06±\\pm3\.8278–✗✓76\.61±\\pm4\.4077−1\.45\-1\.45✓✗75\.74±\\pm5\.5376−2\.32\-2\.32✗✗73\.99±\\pm5\.0474−4\.07\-4\.07UR\-FUNNY\(GloVe \+ COVAREP \+ OpenFace\)✓✓70\.7271–✓✗70\.6271−0\.10\-0\.10✗✓69\.7270−1\.00\-1\.00✗✗69\.1169−1\.61\-1\.61MUStARD\(BERT \+ Dasheng \+ PE\-Core\)✓✓82\.61±\\pm3\.9283–✓✗82\.03±\\pm4\.0882−0\.58\-0\.58✗✓81\.45±\\pm4\.1181−1\.16\-1\.16✗✗79\.71±\\pm4\.2580−2\.90\-2\.90MUStARD\(E5 \+ Dasheng \+ PE\-Core\)✓✓83\.33±\\pm4\.2084–✓✗82\.03±\\pm3\.7382−1\.30\-1\.30✗✓81\.59±\\pm3\.6582−1\.74\-1\.74✗✗81\.16±\\pm4\.5181−2\.17\-2\.17

Table 11:Regularisation ablation across MultiHuSE, UR\-FUNNY, and MUStARD\. ✓✓ = both Mixup and Curriculum Dropout enabled \(full model\);Δ\\Delta= accuracy change relative to the full model\. All metrics in %\.
## Appendix DRole of Gate Mechanism Ablation

Table[12](https://arxiv.org/html/2608.26879#A4.T12)reports the effect of different gating strategies\. The learnable gate attains the highest accuracy on UR\-FUNNY and MUStARD, demonstrating the benefit of sample\-adaptive weighting\. MultiHuSE is a mild exception where the fixed hierarchy gate performs marginally better, likely due to its extreme modality imbalance: under highly skewed conditions, a learnable gate can over\-emphasize the dominant modality\. Overall differences between gates are small \(≤\\leq1\.45%\), indicating that IAF is largely insensitive to the precise gating strategy\.

ConfigGate TypeAcc\(%\)±\\pmSDΔ\\Delta\(%\)MultiHuSE Dataset\(Hierarchy: T=0\.50, A=0\.30, V=0\.20\)BERT \+ PE\-CoreLearnable74\.74±\\pm1\.71–Hierarchy74\.99±\\pm2\.68\+0\.25\+0\.25Uniform74\.86±\\pm3\.31\+0\.12\+0\.12E5 \+ PE\-CoreLearnable78\.06±\\pm3\.82–Hierarchy78\.52±\\pm3\.64\+0\.46\+0\.46Uniform78\.02±\\pm3\.81−0\.04\-0\.04UR\-FUNNY Dataset\(Hierarchy: T=0\.50, A=0\.30, V=0\.20\)GloVe/COVAREP/OpenFaceLearnable70\.72–Hierarchy70\.32−0\.40\-0\.40Uniform69\.42−1\.30\-1\.30MUStARD Dataset\(Hierarchy: T=0\.25, A=0\.40, V=0\.35\)BERT \+ PE\-CoreLearnable82\.61±\\pm3\.92–Hierarchy81\.74±\\pm4\.11−0\.87\-0\.87Uniform81\.88±\\pm3\.94−0\.73\-0\.73E5 \+ PE\-CoreLearnable83\.33±\\pm4\.20–Hierarchy82\.03±\\pm4\.48−1\.30\-1\.30Uniform81\.88±\\pm4\.42−1\.45\-1\.45

Table 12:Gate mechanism ablation across all three datasets\. Learnable = sample\-adaptive weights; Hierarchy = fixed weights derived from modality ranking; Uniform = equal weights \(T=A=V=0\.33\)\.Δ\\Delta= accuracy difference relative to the Learnable gate\.
## Appendix EPathway Collapse: Visualisation

This section visualises pathway collapse in the distilled IAF and symmetric models \(E5 \+ PE\-Core, MultiHuSE\) from two complementary perspectives: aggregate pathway accuracy across configurations and feature\-space behaviour of the text pathway\.

Figure[3](https://arxiv.org/html/2608.26879#A5.F3)summarises dominant\-modality pathway accuracy for all configurations in Table[5](https://arxiv.org/html/2608.26879#S5.T5), comparing the unimodal ceiling, symmetric fusion, and IAF\. Across all seven settings, symmetric fusion consistently reduces pathway accuracy, with a mean drop of 11\.6 pp, whereas IAF matches the unimodal ceiling in every case\. This pattern holds across datasets and encoder combinations, suggesting that strong\-modality collapse is a structural property of symmetric cross\-attention fusion rather than an artefact of any particular encoder\.

Figure 3:Dominant\-modality pathway accuracy for all configurations \(data from Table[5](https://arxiv.org/html/2608.26879#S5.T5)\)\. Shaded regions group MultiHuSE \(blue\) and MUStARD \(purple\) configurations\.##### Feature\-space visualisation of collapse \(t\-SNE\)\.

Figure[4](https://arxiv.org/html/2608.26879#A5.F4)shows a joint t\-SNE embedding of the text\-pathway features used at the final classification step, coloured by prediction correctness\. Both panels share a single embedding fitted on the combined feature matrix, so differences in error density reflect classifier behaviour rather than changes in the underlying feature geometry\.

In the IAF panel \(left\), the frozen unimodal classifierψT\\psi\_\{T\}operates on the backbone features𝒉\(T\)\\bm\{h\}^\{\(T\)\}on which it was trained; text\-pathway accuracy on fold 4 is 76\.7%\. In the symmetric fusion panel \(right\), the shared classifier operates on post\-cross\-attention features𝒉~\(T\)\\tilde\{\\bm\{h\}\}^\{\(T\)\}; accuracy drops to 40\.3%\. The residual connection preserves a broadly similar manifold structure across both panels, yet the symmetric pathway produces substantially more errors in the same geometric space\. This pattern is consistent with classifier mismatch: the shared head is not optimised for single\-pathway inputs\.

Figure 4:Joint t\-SNE of features used at the final classification step for the text pathway under IAF \(left\) and symmetric fusion \(right\), fold 4 of MultiHuSE \(E5 \+ PE\-Core, distilled\)\.Green= correctly classified by that pathway;red= misclassified\.

## Appendix FGate Weight Analysis

The IAF gate network produces per\-sample weights\(wT,wA,wV\)\(w\_\{T\},w\_\{A\},w\_\{V\}\)that sum to one and determine each modality’s contribution before classification\. Table[13](https://arxiv.org/html/2608.26879#A6.T13)reports mean gate weights averaged across cross\-validation folds at the best checkpoint for all nine configurations\. Two population\-level patterns emerge\. First, the modality assigned thepurerole consistently receives the highest mean weight: text leads on MultiHuSE \(w¯T∈\[0\.35,0\.47\]\\bar\{w\}\_\{T\}\\in\[0\.35,0\.47\]\), whereas audio and video dominate on MUStARD \(w¯A\+w¯V\>0\.80\\bar\{w\}\_\{A\}\+\\bar\{w\}\_\{V\}\>0\.80in all configurations\)\. On UR\-FUNNY, weights are more evenly distributed \(w¯T=0\.372\\bar\{w\}\_\{T\}=0\.372,w¯A=0\.310\\bar\{w\}\_\{A\}=0\.310,w¯V=0\.318\\bar\{w\}\_\{V\}=0\.318\), consistent with more balanced cross\-modal cues in that corpus\. Second, no modality exceeds a mean weight of 0\.53 in any configuration, suggesting that IAF preserves multimodal integration rather than collapsing into a regularised unimodal model\.

TextFeat\.𝚫\\bm\{\\Delta\}\(%\)w¯T\\bar\{w\}\_\{T\}w¯A\\bar\{w\}\_\{A\}w¯V\\bar\{w\}\_\{V\}MultiHuSE\(Textpure\|\|Dasheng\+PE\-Core\)BERT\+PE\-CoreBase\+4\.07\[B\]\+4\.07^\{\[\\text\{B\}\]\}0\.4740\.3550\.171Dist\.\+6\.48\[B\]\+6\.48^\{\[\\text\{B\}\]\}0\.3550\.2060\.439E5\+PE\-CoreBase\+1\.04\[E\]\+1\.04^\{\[\\text\{E\}\]\}0\.4310\.4250\.144Dist\.\+3\.20\[E\]\+3\.20^\{\[\\text\{E\}\]\}0\.4120\.1970\.391UR\-FUNNY\(Textpure\|\|COVAREP \+ OpenFace\)GloVeDist\.\+8\.25\+8\.250\.3720\.3100\.318MUStARD\(Audio\+Videopure\|\|Textattend\)BERT\+PE\-CoreBase\+3\.47\[B\]\+3\.47^\{\[\\text\{B\}\]\}0\.1940\.5300\.276Dist\.\+4\.49\[B\]\+4\.49^\{\[\\text\{B\}\]\}0\.1610\.3890\.449E5\+PE\-CoreBase\+3\.62\[E\]\+3\.62^\{\[\\text\{E\}\]\}0\.1660\.4400\.394Dist\.\+5\.21\[E\]\+5\.21^\{\[\\text\{E\}\]\}0\.1030\.4700\.428

Table 13:Mean learned gate weights across CV folds at the best checkpoint\.Δ\\Delta= gain over dominant unimodal baseline \(MultiHuSE: BERT = 68\.26%, E5 = 74\.86%; UR\-FUNNY: GloVe = 62\.47%; MUStARD: Dasheng = 78\.12%\)\.\[B\]/\[E\]: gap from BERT/E5 baseline\. Base = raw pre\-trained; Dist\. = knowledge\-distilled\.These averages, however, mask substantial within\-dataset variation\. Table[14](https://arxiv.org/html/2608.26879#A6.T14)presents representative samples from the distilled IAF model \(E5 \+ PE\-Core\) on MultiHuSE\. Although the population average isw¯T≈0\.51\\bar\{w\}\_\{T\}\\approx 0\.51, individual samples range from near\-exclusive text weighting \(wT\>0\.95w\_\{T\}\>0\.95\) to strong audio dominance \(wA\>0\.85w\_\{A\}\>0\.85\)\. Four cases illustrate this adaptive behaviour\. In Category \(i\), unambiguous headline\-style utterances labelled as neutral humour are classified primarily from text\. In Category \(ii\), deadpan or context\-dependent phrasing provides weak textual evidence, and the gate shifts weight toward audio and visual cues while preserving correct classification\. In Category \(iii\), the model shows over\-reliance on text: utterances such asAge is of no importance unless you’re a cheesedepend on delivery cues absent from the transcript, leading to misclassification\. In Category \(iv\), audio becomes dominant when the transcript is relatively uninformative; for example,Tomorrow is 2\-22\-22\. Happy Tuesday\!appears textually plain, but vocal warmth and intonation support affiliative classification, and the gate assigns about 90% of the weight to audio\. Overall, the aggregate trends align with thepure/attendassignments, while the case studies show that the gate adapts to sample\-level variation in modality reliability\.

Sample textTruePredwTw\_\{T\}wAw\_\{A\}wVw\_\{V\}\(i\) High text weight, correctly classifiedJoe Kennedy III reveals how his GOP counterparts really feel about Trump’s tweets\.NeuNeu0\.9700\.0060\.024✓Katy Perry wears American flag dress to kids in overall concerts\.NeuNeu0\.9610\.0180\.020✓Porn actress confirms Trump affair after unpublished 2011 interview\.NeuNeu0\.9590\.0200\.021✓\(ii\) Low text weight, correctly classifiedThe only positive thing about you is your HIV status\.AggAgg0\.0230\.5480\.429✓Advice to ice skaters: you can’t always tell a brick by its cover\.NeuNeu0\.0240\.6430\.333✓He who laughs last probably doesn’t understand the joke\.NeuNeu0\.0280\.6850\.287✓\(iii\) High text weight, incorrectly classifiedAge is of no importance unless you’re a cheese\.S\-enhAgg0\.8850\.0320\.082✗Don’t look back\. You’re not going that way\.S\-enhAgg0\.8630\.0490\.088✗What is the best contraceptive for old people? Nudity\.AggAff0\.8570\.0840\.058✗\(iv\) Audio\-dominant, correctly classifiedHow many Californians does it take to screw in a light bulb? None\. They screw in hot tubs\.NeuNeu0\.0390\.9000\.061✓Tomorrow is 2\-22\-22\. Happy Tuesday\!AffAff0\.1350\.8560\.009✓Why couldn’t the leopard play hide and seek? Because he was always spotted\.AffAff0\.1020\.8510\.048✓

Table 14:Representative IAF gate weight profiles \(E5 \+ PE\-Core, MultiHuSE, all folds\)\.This table contains potentially offensive or sensitive example text reproduced from the dataset for analysis\.Class labels: Aff = Affiliative, Agg = Aggressive, Neu = Neutral, S\-enh = Self\-enhancing\.wT\+wA\+wV=1w\_\{T\}\{\+\}w\_\{A\}\{\+\}w\_\{V\}=1per sample\. ✓ correct; ✗ incorrect\.
## Appendix GDistillation Coefficient Sensitivity

Table[15](https://arxiv.org/html/2608.26879#A7.T15)reports the effect of varying the three distillation loss coefficients \(α\\alpha: hard CE,β\\beta: soft KD,γ\\gamma: feature alignment\) on MultiHuSE\. We evaluate three alternative configurations alongside the default setting: equal weights, a CE\-heavy variant, and a soft\-target\-heavy variant\. All distilled students are then used as inputs to IAF under the same fusion training procedure\.

Fusion accuracy for IAF with E5, audio, and PE\-Core varies by at most 1\.20 percentage points across the four configurations\. The two best\-performing settings, the default and equal\-weight variants, achieve the same top\-line accuracy of 78\.06%\. CE\-heavy weighting yields a small improvement in video distillation \(\+0\.38 pp relative to the default\) but reduces cross\-modal soft transfer, leading to a 1\.20 pp drop in fusion accuracy\.

SettingCoeff\. \(α\\alpha/β\\beta/γ\\gamma\)BERT Dist\.Audio Dist\.Video Dist\.IAF \(BERT\)IAF \(E5\)Default†0\.40 / 0\.35 / 0\.2570\.21±\\pm2\.8262\.90±\\pm1\.8345\.57±\\pm1\.2274\.74±\\pm1\.7178\.06±\\pm3\.82Equal weights0\.33 / 0\.33 / 0\.3369\.92±\\pm2\.7963\.11±\\pm1\.5444\.66±\\pm1\.5774\.37±\\pm2\.3378\.06±\\pm3\.36CE\-heavy0\.60 / 0\.25 / 0\.1569\.63±\\pm2\.7562\.78±\\pm1\.1845\.95±\\pm0\.9874\.03±\\pm2\.4876\.86±\\pm4\.32Soft\-target\-heavy0\.25 / 0\.55 / 0\.2070\.17±\\pm2\.5862\.86±\\pm1\.1044\.74±\\pm1\.3573\.33±\\pm2\.3276\.90±\\pm4\.18Range–0\.580\.331\.291\.411\.20

Table 15:Distillation coefficient sensitivity on MultiHuSE \(5\-fold CV, mean±\\pmSD %\)\.†\\dagger= setting used in all main experiments\. IAF columns use distilled students; Audio = Dasheng, Video = PE\-Core throughout\. “Range” reports max−\-min across the four settings\.The soft\-target\-heavy variant lowers video distillation performance \(\-0\.83 pp relative to the default\), suggesting that the weakest modality depends more on direct task supervision through CE than on soft targets from a teacher trained on a different modality\. In this setting, increasing the weight on soft targets from the text teacher appears to overwhelm the video student’s task\-specific learning signal\. Audio distillation remains stable across all settings, with a range of at most 0\.33 pp, indicating that the Dasheng encoder adapts consistently regardless of coefficient choice\. Overall, these results suggest that IAF is not highly sensitive to the precise coefficient values; the default setting was selected by manual tuning and then held fixed across all datasets\.

## Appendix HKnowledge Distillation Design Comparison

Table[16](https://arxiv.org/html/2608.26879#A8.T16)situates MAKD relative to prior multimodal distillation approaches along five design axes: training stage, teacher source, objective, and optimisation procedure\. The key distinction is that MAKD is a standalone pre\-fusion stage that uses only the empirically dominant unimodal modality as teacher to strengthen weaker encoders before fusion, whereas prior multimodal KD methods typically operate within the fusion stage itself\.

MethodStageTeacherPurposeProcedureCross\-M KD[Albanie et al\. \(2018\)](https://arxiv.org/html/2608.26879#bib.bib19)[Thoker and Gall \(2019\)](https://arxiv.org/html/2608.26879#bib.bib14)Pre\-fusion†Paired modalityUnimodal transferSequentialWang et al\.[Wang et al\. \(2020a\)](https://arxiv.org/html/2608.26879#bib.bib37)Within fusionFull multimodalDropout robustnessJointMMANet[Wei et al\. \(2023\)](https://arxiv.org/html/2608.26879#bib.bib27)Within fusionFull multimodalMissing\-modalityJointDMD[Li et al\. \(2023b\)](https://arxiv.org/html/2608.26879#bib.bib15)Within fusionSubspace teachersRepr\. disentanglementConcurrentMTMD[Lin and Hu \(2024\)](https://arxiv.org/html/2608.26879#bib.bib31)Within fusionUnimodal \+ fusedHeterogeneity reductionConcurrentMAKD \(Ours\)Pre\-fusionDom\. unimodalEncoder strengtheningSequential

Table 16:Design comparison of multimodal knowledge distillation methods\.†Not designed in the context of a subsequent fusion stage\.