CAT-GS: Balanced Multimodal Learning via Calibrated Gating and Fusion Surgery

# CAT-GS: Balanced Multimodal Learning via Calibrated Gating and Fusion Surgery
Source: [https://arxiv.org/html/2608.24947](https://arxiv.org/html/2608.24947)
Mahir Shahriar TamimEmail:[mahir\.tamim@northsouth\.edu](mailto:[email protected])Corresponding author:Corresponding authorAffiliation:Department of Electrical and Computer Engineering, North South University, Dhaka, BangladeshSharjil KhanAffiliation:Department of Electrical and Computer Engineering, North South University, Dhaka, BangladeshMd\. Samiul AlimAffiliation:Department of Electrical and Computer Engineering, North South University, Dhaka, BangladeshTanvir Ahmed KhanAffiliation:Department of Electrical and Computer Engineering, North South University, Dhaka, BangladeshShafin RahmanAffiliation:Department of Electrical and Computer Engineering, North South University, Dhaka, BangladeshNabeel MohammedAffiliation:Department of Electrical and Computer Engineering, North South University, Dhaka, Bangladesh

###### Abstract

End\-to\-end training of multimodal neural networks often exhibits unstable neural dynamics characterized by three coupled failure modes that degrade learning: \(i\)*modality imbalance*, where one branch dominates gradient\-based optimization; \(ii\)*unstable gating*, where noisy confidence cues induce erratic modality selection; and \(iii\)*fusion interference*, where modality\-specific gradients conflict at the shared fusion layer\. We proposeCAT\-GS\(Calibrated, Adaptive, Thresholded Gating with Fusion Surgery\), a neural dynamics\-based optimization controller for intelligent computing applications\. CAT\-GS operates during backpropagation without modifying model architectures, fusion modules, or task losses\. Through calibration of teacher\-derived reliability via temperature scaling and EMA smoothing, CAT\-GS stabilizes neural dynamics using a margin\-thresholded policy to switch between warm\-up dropout, weak\-modality prioritization, and weak\-biased blending, stabilizes gradient magnitudes under aggressive gating via capped gradient\-budget renormalization, and applies*fusion\-only*PCGrad to reduce destructive cross\-modal interference at the primary shared bottleneck\. We evaluate CAT\-GS on audio–visual multimodal pattern recognition benchmarks \(CREMA\-D, AV\-MNIST, and VGGSound\), a tri\-modal setting \(UR\-FUNNY\), controlled synthetic data \(CG\-MNIST\),and additional cross\-domain benchmarks \(AVE and CMU\-MOSI\)\. CAT\-GS improves or matches fused multimodal accuracy against strong imbalance\-aware baselines \(including OGM\-GE, G2D, and UMT\) across settings, and yields smoother gating behavior with fewer conflicting fusion gradients\.

###### Keywords:

Neural dynamics , Multimodal learning , Intelligent computing , Gradient\-based optimization , Learning dynamics , Knowledge distillation , Calibration , Pattern recognition

## 1Introduction

Deep multimodal learning integrates heterogeneous signals \(e\.g\., audio/speech and vision\) within neural network architectures to improve pattern recognition and robustness beyond any single modality[34](https://arxiv.org/html/2608.24947#bib.bib1)\. From a neural dynamics perspective, intelligent computing systems require stable temporal evolution of gradient flows and adaptive control mechanisms throughout training\. CAT\-GS is a neural dynamics\-based optimization controller that operates during backpropagation to regulate learning dynamics without modifying model architectures, fusion modules, or task losses\. In an ideal system, the model dynamically relies on the most reliable modality*for the current sample and training stage*\. In practice, however, joint optimization is frequently derailed by the*modality imbalance phenomenon*[21](https://arxiv.org/html/2608.24947#bib.bib17): one branch \(often the easier or higher\-SNR modality\) quickly dominates gradient flow, while the weaker branch receives progressively smaller updates\. The result is brittle fusion, under\-trained modality\-specific encoders, and reduced multimodal gains\. Figure[1](https://arxiv.org/html/2608.24947#S1.F1)illustrates these failure modes on CREMA\-D, highlighting how dominant\-modality imbalance and fusion\-gradient conflicts can derail convergence under standard joint training\.

Existing approaches mitigate imbalance primarily through two families of techniques\.*Gradient\-modulation methods*attenuate the dominant modality’s gradients to amplify weaker ones[21](https://arxiv.org/html/2608.24947#bib.bib17),[37](https://arxiv.org/html/2608.24947#bib.bib14), while*confidence\-based gating methods*down\-weight or mask unreliable modalities based on calibration\-aware confidence estimates[11](https://arxiv.org/html/2608.24947#bib.bib11)\. Although effective in isolation, these strategies fail to address three tightly\-coupled challenges that arise in the learning dynamics of end\-to\-end multimodal optimization\. First, confidence signals used for modality gating are typically derived from raw logits, which are frequently miscalibrated and noisy[10](https://arxiv.org/html/2608.24947#bib.bib12)\. Without temporal stabilization, these unreliable estimates induce sharp batch\-to\-batch fluctuations in gating decisions, leading to “gate thrashing” rather than reliability\-aware control\. Second, aggressive or persistent suppression of a modality can result in*gradient starvation*: when a branch receives near\-zero gradients over extended periods, its representations stagnate relative to the dominant modality, reinforcing the imbalance and making recovery increasingly difficult[25](https://arxiv.org/html/2608.24947#bib.bib9)\. Third, even when modality\-specific gradients are individually balanced, their interaction in the shared fusion layer can remain destructive; gradients with negative cosine similarity[38](https://arxiv.org/html/2608.24947#bib.bib10)interfere at the bottleneck, impeding joint convergence despite otherwise well\-behaved unimodal updates\.

To address these limitations, we introduceCAT\-GS, a neural dynamics\-based optimization framework for intelligent computing applications that provides a unified treatment of these coupled failure modes for calibrated, balanced, and conflict\-aware multimodal learning\. CAT\-GS wraps an existing multimodal training loop and modifies*only*the backward\-stage optimization signals\. It uses a*margin\-thresholded controller*driven by stabilized teacher reliability scores to switch between warm\-up dropout, dominance suppression, and weak\-biased blending\. This produces interpretable regime transitions and avoids oscillatory gating behavior\. Empirically, CAT\-GS improves fused multimodal accuracy in diverse settings, achieving gains over strong imbalance\-aware baselines \(e\.g\., in the UR\-FUNNY A–V–TXT task; Table[2](https://arxiv.org/html/2608.24947#S4.T2)\), while also exhibiting smoother gating dynamics and fewer negative fusion\-gradient cosine events \(Figure[4](https://arxiv.org/html/2608.24947#S4.F4)\)\. Unlike prior work that addresses imbalance, gating, or conflict resolution in isolation, CAT\-GS couples calibrated reliability estimation via knowledge distillation, regime\-aware gating, budget\-preserving gradient renormalization, and localized conflict resolution within a single optimization step\.

Novelty and contributions\.We introduceCAT\-GS, an optimization\-stage multimodal backward\-pass controller that jointly addresses gate instability, modality starvation, and fusion interference within a single optimization step\. The building blocks draw on established techniques; the two new elements are restricting PCGrad to the fusion bottleneck rather than the whole network, and a budget\-preserving renormalization that prevents a gated modality from starving\. Our contributions are as follows:

- •We cast adaptive modality gating as a*regime\-based gating controller*driven by calibrated reliability margins from knowledge distillation, enabling interpretable transitions between warm\-up dropout, dominance suppression, and weak\-biased blending, without modifying model architectures or task losses\.
- •We couple gating with a*budget\-preserving gradient renormalization mechanism*that stabilizes update magnitudes under aggressive \(hard\) gating using an EMA target magnitude with a hard cap, mitigating failure modes such as prolonged imbalance and brittle convergence\.
- •We adopt a*fusion\-only gradient surgery strategy*, applying PCGrad exclusively at the shared fusion bottleneck where cross\-modal gradients first interact, thereby reducing destructive interference while preserving unimodal encoder dynamics at substantially lower cost than global projection\.
- •We provide extensive empirical validation across audio–visual, tri\-modal, and controlled synthetic benchmarks, plus additional AVE and CMU\-MOSI evaluations, demonstrating consistent improvements in fused accuracy, optimization stability, and robustness to modality imbalance\.

![Refer to caption](https://arxiv.org/html/2608.24947v1/Fig1.png)Figure 1:Optimization dynamics on CREMA\-D\.Shown are experimentally derived visualizations of training behavior under standard joint training \(left\) and CAT\-GS \(right\)\. Annotations are illustrative summaries of observed gradient statistics\. Joint training exhibits unstable dynamics with dominant\-modality imbalance and frequent fusion\-gradient conflicts, converging to a suboptimal solution \(≈\\approx67% accuracy\)\. CAT\-GS stabilizes optimization via calibrated gating, gradient\-budget preservation, and fusion\-only gradient surgery, yielding smooth convergence to a robust solution \(86\.3% accuracy\)\.
## 2Related Work

Modality Imbalance in Multimodal Learning\.The dominance of specific modalities during joint training is a pervasive issue in multimodal learning\. Early efforts address this through auxiliary losses or feature norm regularization[21](https://arxiv.org/html/2608.24947#bib.bib17)\. More recent gradient\-modulation techniques, such as on\-the\-fly rescaling[21](https://arxiv.org/html/2608.24947#bib.bib17)and multi\-loss balancing[18](https://arxiv.org/html/2608.24947#bib.bib23), dynamically adjust updates to prevent one modality from overpowering others\.Recent balancing\-oriented methods include MMCosine[35](https://arxiv.org/html/2608.24947#bib.bib2), which uses cosine\-based feature normalization for improved fine\-grained discriminability, DRL[32](https://arxiv.org/html/2608.24947#bib.bib4), which diagnoses modality learning states and applies soft re\-learning, and MCR[19](https://arxiv.org/html/2608.24947#bib.bib3), which introduces game\-theoretic regularization to mitigate modality competition\.While these methods improve balance, many still rely on predefined or coarse dominance cues and may not explicitly prevent long\-horizon gradient starvation in suppressed branches[23](https://arxiv.org/html/2608.24947#bib.bib24)\.

In contrast, CAT\-GS leverages calibrated confidence margins for context\-aware modulation, ensuring suppression is temporary and paired with budget\-preserving rescaling\.

Confidence\-Based Gating and Dynamic Fusion\.Multimodal fusion has been studied across a wide range of tasks, from sentiment analysis that combines audio, visual, and textual cues[24](https://arxiv.org/html/2608.24947#bib.bib38)to video captioning via hierarchical attention\-based fusion[33](https://arxiv.org/html/2608.24947#bib.bib39)\. Gating mechanisms have long enabled adaptive fusion in multimodal systems[1](https://arxiv.org/html/2608.24947#bib.bib28)\. Contemporary approaches use soft attention[27](https://arxiv.org/html/2608.24947#bib.bib6)or entropy\-driven gates to weigh modalities by reliability\. Teacher\-guided fusion further enhances this by distilling unimodal priors[6](https://arxiv.org/html/2608.24947#bib.bib22)\. However, reliance on uncalibrated teacher logits leads to unstable decisions[10](https://arxiv.org/html/2608.24947#bib.bib12), exacerbating gate thrashing\. Few works integrate direct calibration into the gating loop; CAT\-GS advances this by combining temperature scaling, EMA smoothing, and margin thresholding for robust, regime\-aware control\.

Gradient Conflict Resolution\.Gradient conflicts hinder multi\-task and multimodal optimization[30](https://arxiv.org/html/2608.24947#bib.bib25)\. Projection\-based methods like PCGrad and CAGrad excise destructive components, while others normalize via variance reduction\. In multimodal settings, conflicts are acute at fusion layers[30](https://arxiv.org/html/2608.24947#bib.bib25), yet prior applications are global rather than localized\. To the best of our knowledge, prior work has not systematically explored applying PCGrad selectively at the fusion head; CAT\-GS adopts this localized strategy to preserve unimodal encoder dynamics while resolving bottleneck interference\.

Knowledge Distillation in Multimodal Settings\.Distillation from unimodal teachers is standard for multimodal students[14](https://arxiv.org/html/2608.24947#bib.bib29),[6](https://arxiv.org/html/2608.24947#bib.bib22)\. Methods like UMT[6](https://arxiv.org/html/2608.24947#bib.bib22)match logits and features but treat teacher outputs as infallible, ignoring miscalibration[10](https://arxiv.org/html/2608.24947#bib.bib12)\. Recent calibration\-aware variants focus on post\-hoc adjustments, not online integration\. CAT\-GS embeds temperature scaling and EMA stabilization within the distillation\-guided gating process, yielding reliable signals for adaptive optimization\.

In summary, prior work typically tackles imbalance, confidence\-based fusion, and gradient conflicts in isolation\.CAT\-GS integrates these ideas into a single optimization\-stage controller, yielding a practical recipe for stable and balanced multimodal training across fusion designs\.The contribution is therefore the optimization\-stage integration itself, together with the fusion\-localized projection and budget\-preserving renormalization it introduces\.

## 3Methodology

Figures[2](https://arxiv.org/html/2608.24947#S3.F2)and[3](https://arxiv.org/html/2608.24947#S3.F3)summarize the overall model and the CAT\-GS training iteration\. Figure[2](https://arxiv.org/html/2608.24947#S3.F2)shows the forward architecture and supervision flow \(trainable student encoders and fusion head with frozen unimodal teachers\), while Figure[3](https://arxiv.org/html/2608.24947#S3.F3)details the optimization\-stage controller: calibrated/EMA\-stabilized teacher reliabilities drive regime\-based gating, followed by gradient\-budget stabilization and fusion\-only surgery before the optimizer update\.For readability, we present CAT\-GS in a strict control flow: reliability estimation \(Section[3\.3](https://arxiv.org/html/2608.24947#S3.SS3)\), regime selection and gating \(Section[3\.4](https://arxiv.org/html/2608.24947#S3.SS4)\), magnitude stabilization \(Section[3\.5](https://arxiv.org/html/2608.24947#S3.SS5)\), and fusion\-conflict handling \(Section[3\.6](https://arxiv.org/html/2608.24947#S3.SS6)\), followed by the unified training step in Algorithm[1](https://arxiv.org/html/2608.24947#alg1)\.

![Refer to caption](https://arxiv.org/html/2608.24947v1/Fig2-revised.png)Figure 2:High\-level forward architecture and supervision flow\.Audio and video inputs are processed by trainable student encoders and fused to produce predictions\.Frozen unimodal teachers receive the same raw modality inputs \(xax^\{a\},xvx^\{v\}\) directly \(not student\-derived features\) and remain frozen throughout training; they provide auxiliary supervision via knowledge distillation and per\-modality reliability signals utilized by CAT\-GS\.Optimization\-stage control \(CAT\-GS\) operates exclusively during backpropagation and is shown separately in Figure[3](https://arxiv.org/html/2608.24947#S3.F3)\.![Refer to caption](https://arxiv.org/html/2608.24947v1/Fig_b.png)Figure 3:Overview of CAT\-GS\.The student model contains trainable unimodal encoders and a fusion head, while unimodal teachers are frozen and used only to estimate per\-modality reliability\. Teacher confidences are calibrated and stabilized to drive margin\-threshold gating, followed by gradient\-budget stabilization and fusion\-only PCGrad\.In this section we first formalize the multimodal learning setup and training objective, and then introduce CAT\-GS as an optimization\-level controller that operates on the resulting gradients\.

### 3\.1Problem Formulation

We consider a multimodal dataset𝒟=\{\(xia,xiv,yi\)\}i=1B\\mathcal\{D\}=\\\{\(x\_\{i\}^\{a\},x\_\{i\}^\{v\},y\_\{i\}\)\\\}\_\{i=1\}^\{B\}containing paired audio–visual samples and labels\.The student model follows a standard multimodal fusion design and comprises two unimodal encodersEa​\(⋅,θa\)E\_\{a\}\(\\cdot;\\theta\_\{a\}\)andEv​\(⋅,θv\)E\_\{v\}\(\\cdot;\\theta\_\{v\}\), which extract modality\-specific features, and a fusion networkF⁡\(⋅,θf\)F\(\\cdot;\\theta\_\{f\}\)that maps the concatenated representations to final logits\. Two unimodal teacher modelsTaT^\{a\}andTvT^\{v\}provide auxiliary guidance in the form of logits and intermediate features\. This setup is intentionally generic: the encoders may be CNNs or transformers, and the fusion head may be an MLP or attention module, since CAT\-GS assumes no architectural constraints\.

Given a batch, the student produces

fsa=Ea\(xa\),fsv=Ev\(xv\),𝐬=F\(\[fsa∥fsv\]\)\.f\_\{s\}^\{a\}=E\_\{a\}\(x^\{a\}\),\\qquad f\_\{s\}^\{v\}=E\_\{v\}\(x^\{v\}\),\\qquad\\mathbf\{s\}=F\(\[f\_\{s\}^\{a\}\\\|f\_\{s\}^\{v\}\]\)\.\(1\)
Training objective\.CAT\-GS is objective\-agnostic, but in our experiments we optimize a standard supervised loss augmented with unimodal\-teacher distillation terms\. Letℳ\\mathcal\{M\}denote the set of available modalities \(e\.g\.,\{a,v\}\\\{a,v\\\}or\{a,v,t\}\\\{a,v,t\\\}\)\. For a labeled batch, the student is trained with

ℒ\\displaystyle\\mathcal\{L\}=λCE​ℒCE​\(𝐬,y\)\+∑m∈ℳλKDm​ℒKD​\(𝐬,𝐳m,τm\)\\displaystyle=\\lambda\_\{\\mathrm\{CE\}\}\\,\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\(\\mathbf\{s\},y\)\+\\sum\_\{m\\in\\mathcal\{M\}\}\\lambda\_\{\\mathrm\{KD\}\}^\{m\}\\,\\mathcal\{L\}\_\{\\mathrm\{KD\}\}\(\\mathbf\{s\},\\mathbf\{z\}^\{m\},\\tau\_\{m\}\)\(2\)\+∑m∈ℳλfeatmℒfeat\(fsm,ftm\)\.\\displaystyle\+\\sum\_\{m\\in\\mathcal\{M\}\}\\lambda\_\{\\mathrm\{feat\}\}^\{m\}\\,\\mathcal\{L\}\_\{\\mathrm\{feat\}\}\(f\_\{s\}^\{m\},f\_\{t\}^\{m\}\)\.
whereℒCE\\mathcal\{L\}\_\{\\mathrm\{CE\}\}is cross\-entropy,ftmf\_\{t\}^\{m\}and𝐳m\\mathbf\{z\}^\{m\}are the unimodal teacher features and logits, and

ℒKD\(𝐬,𝐳m,τm\)=τm2KL\(Softmax\(𝐳m/τm\)∥Softmax\(𝐬/τm\)\),\\mathcal\{L\}\_\{\\mathrm\{KD\}\}\(\\mathbf\{s\},\\mathbf\{z\}^\{m\},\\tau\_\{m\}\)=\\tau\_\{m\}^\{2\}\\,\\mathrm\{KL\}\\big\(\\mathrm\{Softmax\}\(\\mathbf\{z\}^\{m\}/\\tau\_\{m\}\)\\,\\\|\\,\\mathrm\{Softmax\}\(\\mathbf\{s\}/\\tau\_\{m\}\)\\big\),\(3\)withℒfeat\\mathcal\{L\}\_\{\\mathrm\{feat\}\}implemented as MSE\. The coefficients\{λ\}\\\{\\lambda\\\}and temperatures\{τm\}\\\{\\tau\_\{m\}\\\}match the training code and are set per experiment; when a distillation term is not used, its weight is set to zero\.

Any differentiable training objective defined over𝐬\\mathbf\{s\}, the labelsyy, and optionally the teacher outputs\{Tm​\(xm\)\}m∈\{a,v\}\\\{T^\{m\}\(x^\{m\}\)\\\}\_\{m\\in\\\{a,v\\\}\}induces gradients with respect to the modality\-specific parametersθa,θv\\theta\_\{a\},\\theta\_\{v\}and the fusion parametersθf\\theta\_\{f\}\. In a conventional setting, these gradients are aggregated and applied uniformly across modalities during backpropagation, which is precisely where modality imbalance can arise: one branch may dominate the updates, while the other receives progressively weaker signals\. CAT\-GS intervenes at this optimization stage\. Rather than changing the architecture or the underlying training objective, it modulates and stabilizes modality\-specific and fusion\-layer gradients, aiming to ensure balanced, reliability\-aware updates throughout training\.

### 3\.2CAT\-GS Overview

Multimodal models frequently suffer from*modality imbalance*, where one modality converges faster or produces cleaner gradients and consequently dominates optimization\. As training progresses, gradients for the weaker modality diminish, feature representations diverge, and the fusion layer receives conflicting update signals, ultimately leading to brittle and suboptimal fusion\. CAT\-GS addresses this issue at the*optimization level*\. Rather than modifying architectures or introducing task\-specific losses, CAT\-GS restructures how gradients are weighted, stabilized, and combined during training\. It operates transparently betweenloss\.backward\(\)andoptimizer\.step\(\), and can be integrated into any multimodal model with modality\-specific parameters\. The framework consists of four complementary components:

1. 1\.Calibrated reliability estimationto produce stable modality confidence signals;
2. 2\.Margin\-thresholded adaptive gatingto dynamically prioritize modalities;
3. 3\.Gradient\-budget stabilizationto control update magnitudes under aggressive gating;
4. 4\.Fusion\-layer gradient surgeryto remove destructive cross\-modal conflicts\.

These components are jointly necessary\. Calibration stabilizes gating decisions, adaptive gating prevents runaway dominance, budget reallocation stabilizes update magnitudes under aggressive gating, and fusion surgery resolves residual gradient interference at the bottleneck\. Together, they yield a stable optimization dynamic that adapts to evolving modality reliability without altering the forward computation\.

### 3\.3Calibrated Reliability Estimation

Teacher logits encode useful task knowledge but can be noisy and overconfident, especially early in training\. Since CAT\-GS relies on teacher reliability as a control signal, we stabilize these estimates to reduce batch\-to\-batch volatility\.

##### Temperature scaling

For the teacherTmT\_\{m\}with logitszmz^\{m\}, we compute the average probability assigned to the ground\-truth label under a temperatureTmT\_\{m\}:

ptm=1B​∑i=1BSoftmax⁡\(zim/Tm\)​\[yi\],p\_\{t\}^\{m\}=\\frac\{1\}\{B\}\\sum\_\{i=1\}^\{B\}\\mathrm\{Softmax\}\(z\_\{i\}^\{m\}/T\_\{m\}\)\[y\_\{i\}\],\(4\)whereTmT\_\{m\}controls the sharpness of the confidence signal\. In our main experiments, we setTm=1T\_\{m\}\{=\}1for all modalities \(i\.e\., no temperature scaling\) unless explicitly stated\. We include temperature scaling as an optional extension when a calibrated temperature is available, but it is not required for CAT\-GS\.Thus, temperature scaling is not a required performance driver in our default setting; it is an optional calibration switch and a stress\-test factor in Section[7](https://arxiv.org/html/2608.24947#S4.T7)\.The resulting scalarptmp\_\{t\}^\{m\}is a*batch\-level*\(step\-level\) proxy for modality reliability; we use the batch mean for stability and to avoid noisy per\-sample gating signals\.

##### EMA smoothing

To prevent noisy fluctuations from batch to batch, we maintain:

p^m←β​p^m\+\(1−β\)​ptm\.\\hat\{p\}^\{m\}\\leftarrow\\beta\\,\\hat\{p\}^\{m\}\+\(1\-\\beta\)\\,p\_\{t\}^\{m\}\.\(5\)The smoothed confidencesp^a,p^v\\hat\{p\}^\{a\},\\hat\{p\}^\{v\}are used as the continuous control signal for gating, while temperature scaling remains an optional calibration step \(defaultTm=1T\_\{m\}\{=\}1in our main experiments\)\.This reduces “gate thrashing” and enables regime decisions based on reliability trends rather than instantaneous noise\. In practice, we use a high momentum \(e\.g\.,β=0\.9\\beta=0\.9\) so that the controller reacts gradually rather than abruptly\.

### 3\.4Margin\-Thresholded Adaptive Gating

CAT\-GS uses the smoothed reliabilities to compute a*reliability margin*:

Δ=\|p^a−p^v\|\.\\Delta=\|\\hat\{p\}^\{a\}\-\\hat\{p\}^\{v\}\|\.\(6\)This margin measures how far apart the modalities are in terms of calibrated confidence\. Two thresholdsτlow\\tau\_\{\\text\{low\}\}andτhigh\\tau\_\{\\text\{high\}\}define three regimes reflecting distinct optimization needs\. Crucially, these regimes determine the gating coefficientsαm\\alpha\_\{m\}that will later be used to scale gradients flowing through each modality\.For a compact view, the gating controller in this subsection can be summarized as one regime map:

αm​\(Δ\)=\{αmdrop,Δ<τlow​and epoch<Ew,αmhard,Δ\>τhigh,αmsoft,τlow≤Δ≤τhigh,\{\\color\[rgb\]\{0,0,0\}\\alpha\_\{m\}\(\\Delta\)=\\begin\{cases\}\\alpha\_\{m\}^\{\\mathrm\{drop\}\},&\\Delta<\\tau\_\{\\mathrm\{low\}\}\\ \\text\{and epoch\}<E\_\{w\},\\\\ \\alpha\_\{m\}^\{\\mathrm\{hard\}\},&\\Delta\>\\tau\_\{\\mathrm\{high\}\},\\\\ \\alpha\_\{m\}^\{\\mathrm\{soft\}\},&\\tau\_\{\\mathrm\{low\}\}\\leq\\Delta\\leq\\tau\_\{\\mathrm\{high\}\},\\end\{cases\}\}\(7\)
#### 1\. Warm\-up Dropout \(Δ<τlow\\Delta<\\tau\_\{\\text\{low\}\}, early epochs\)

Early teacher signals are noisy, so CAT\-GS uses stochastic modality dropout \(probabilitypdropp\_\{\\text\{drop\}\}\) for a short warm\-up to prevent premature specialization and to strengthen unimodal representations\.

#### 2\. Dominance Suppression \(Δ\>τhigh\\Delta\>\\tau\_\{\\text\{high\}\}\)

When one modality becomes clearly more reliable, CAT\-GS protects the weaker one by performing hard gating:

αm=\{1,p^m<p^¬m,0,otherwise\.\\alpha\_\{m\}=\\begin\{cases\}1,&\\hat\{p\}^\{m\}<\\hat\{p\}^\{\\neg m\},\\\\ 0,&\\text\{otherwise\}\.\\end\{cases\}\(8\)The weaker branch receives the full update emphasis during this regime, counteracting runaway dominance\.This hard regime is used only when the margin is clearly large; otherwise the controller uses soft blending\.

#### 3\. Weak\-Bias Blending \(τlow≤Δ≤τhigh\\tau\_\{\\text\{low\}\}\\leq\\Delta\\leq\\tau\_\{\\text\{high\}\}\)

When reliabilities are close, CAT\-GS uses smooth weak\-biased blending rather than abrupt switching:

α~m=p^mp^a\+p^v,\\tilde\{\\alpha\}\_\{m\}=\\frac\{\\hat\{p\}^\{m\}\}\{\\hat\{p\}^\{a\}\+\\hat\{p\}^\{v\}\},\(9\)then boosts the weaker modality using a bias factorλb​i​a​s\\lambda\_\{bias\}:

αw​e​a​k=α~w​e​a​k​\(1\+λb​i​a​s\),αs​t​r​o​n​g=α~s​t​r​o​n​g​\(1−λb​i​a​s\),\\alpha\_\{weak\}=\\tilde\{\\alpha\}\_\{weak\}\(1\+\\lambda\_\{bias\}\),\\quad\\alpha\_\{strong\}=\\tilde\{\\alpha\}\_\{strong\}\(1\-\\lambda\_\{bias\}\),\(10\)Weights are then normalized with floorε\\varepsilonso both modalities remain active while the weaker branch receives a controlled advantage\.

Together, the three regimes define an interpretable controller that outputsαa,αv∈\[0,1\]\\alpha\_\{a\},\\alpha\_\{v\}\\in\[0,1\]for gradient modulation\.

∇θm←αm​∇θm\.\\nabla\_\{\\theta\_\{m\}\}\\leftarrow\\alpha\_\{m\}\\nabla\_\{\\theta\_\{m\}\}\.\(11\)

### 3\.5Gradient\-Budget Reallocation

Directly applyingαm\\alpha\_\{m\}scales gradients but does not guarantee stable optimization dynamics under aggressive gating\. A fundamental risk in gating mechanisms is*gradient starvation*: if a modality is consistently down\-weighted \(or hard\-gated\), its encoder can receive near\-zero gradients over extended periods\. This causes a vicious cycle where the suppressed modality’s features stagnate, making it even less reliable in future epochs, effectively permanently disabling a branch of the network\.

CAT\-GS mitigates these dynamics by \(i\) enforcing a nonzero floorε\\varepsilonin soft regimes \(Section[3\.4](https://arxiv.org/html/2608.24947#S3.SS4)\), and \(ii\) stabilizing gradient magnitudes during hard\-gating events via*gradient\-budget reallocation*\. We track the historical gradient norm \(per modality\) as a running reference scale:

g^m←βg​g^m\+\(1−βg\)​‖∇θm‖\.\\hat\{g\}^\{m\}\\leftarrow\\beta\_\{g\}\\hat\{g\}^\{m\}\+\(1\-\\beta\_\{g\}\)\\\|\\nabla\_\{\\theta\_\{m\}\}\\\|\.\(12\)After gating, we rescale gradients:

∇θm←αm​∇θm​min⁡\(g^m‖αm​∇θm‖\+δ,γcap\)\.\\nabla\_\{\\theta\_\{m\}\}\\leftarrow\\alpha\_\{m\}\\nabla\_\{\\theta\_\{m\}\}\\;\\min\\\!\\left\(\\frac\{\\hat\{g\}^\{m\}\}\{\\\|\\alpha\_\{m\}\\nabla\_\{\\theta\_\{m\}\}\\\|\+\\delta\},\\;\\gamma\_\{\\text\{cap\}\}\\right\)\.\(13\)Optimization view:Gating alone makes encoder updates scale linearly withαm\\alpha\_\{m\}, so whenαm≈0\\alpha\_\{m\}\\approx 0for many steps, the expected update magnitude collapses and the modality may not recover\. CAT\-GS instead usesg^m\\hat\{g\}^\{m\}as a running reference scale and \(when a modality is hard\-gated\) renormalizes the*active*modality’s encoder gradients toward its EMA magnitude, capped byγcap\\gamma\_\{\\text\{cap\}\}, to avoid brittle update collapse or instability\. Equivalently, Step 5 approximately enforces‖∇θm‖≈min⁡\(g^m,γcap​‖∇~θm‖\)\\\|\\nabla\_\{\\theta\_\{m\}\}\\\|\\approx\\min\(\\hat\{g\}^\{m\},\\,\\gamma\_\{\\text\{cap\}\}\\,\\\|\\tilde\{\\nabla\}\_\{\\theta\_\{m\}\}\\\|\), which stabilizes the magnitude of the applied update while avoiding exploding rescaling when the current gradient is extremely small\.

This stabilizes the overall optimization dynamics under aggressive gating\. When a modality is fully gated \(αm=0\\alpha\_\{m\}=0\), its encoder receives no update by design; CAT\-GS focuses on keeping the remaining active branch well\-conditioned while soft regimes maintain nonzero updates via theε\\varepsilonfloor\. The capγcap\\gamma\_\{\\text\{cap\}\}prevents excessively large rescaling when the current gradient is extremely small, andδ\\deltaavoids numerical instability\.

### 3\.6Fusion\-Layer Gradient Surgery

Even when unimodal gradients are balanced, fusion parameters often receive*conflicting*update directions\. Audio may encourage temporal invariances while visual gradients push toward spatial distinctions, producing destructive interference\. While methods like PCGrad[38](https://arxiv.org/html/2608.24947#bib.bib10)address this globally, applying projection to the entire network is computationally expensive \(O⁡\(N2\)O\(N^\{2\}\)in the number of tasks/modalities\) and may interfere with the specialized feature extraction of unimodal encoders\.

We identify the fusion output layer as the critical*informational bottleneck*where these semantic conflicts are most destructive\. Therefore, CAT\-GS applies a surgical, localized PCGrad\-like projection only to the fusion output parameters \(in our implementation, the final fusion classifier layer, e\.g\.,fc\_out\)\.

Letθf\\theta\_\{f\}denote the parameters of the fusion output layer, and let𝐨a\\mathbf\{o\}^\{a\}and𝐨v\\mathbf\{o\}^\{v\}be the modality\-specific fusion outputs \(logits\) obtained by passing only modality\-mmfeatures through the fusion head\. In the current implementation, we form modality\-specific fusion gradients via a Jacobian–vector product using an all\-ones vector \(equivalently, the gradient of the sum of logits\):

𝐠fm=∇θf⟨𝟏,𝐨m⟩,m∈\{a,v\}\.\\displaystyle\\mathbf\{g\}\_\{f\}^\{m\}=\\nabla\_\{\\theta\_\{f\}\}\\,\\langle\\mathbf\{1\},\\mathbf\{o\}^\{m\}\\rangle,\\qquad m\\in\\\{a,v\\\}\.\(14\)Ifcos⁡\(𝐠fa,𝐠fv\)<0,\\displaystyle\\text\{If \}\\quad\\cos\(\\mathbf\{g\}\_\{f\}^\{a\},\\mathbf\{g\}\_\{f\}^\{v\}\)<0,\(15\)the gradients point in conflicting directions\. CAT\-GS applies a PCGrad\-style projection \(single\-sided in the current implementation\):

𝐠fa←𝐠fa−𝐠fa⋅𝐠fv‖𝐠fv‖2\+δ​𝐠fv,\\mathbf\{g\}\_\{f\}^\{a\}\\leftarrow\\mathbf\{g\}\_\{f\}^\{a\}\-\\frac\{\\mathbf\{g\}\_\{f\}^\{a\}\\cdot\\mathbf\{g\}\_\{f\}^\{v\}\}\{\\\|\\mathbf\{g\}\_\{f\}^\{v\}\\\|^\{2\}\+\\delta\}\\,\\mathbf\{g\}\_\{f\}^\{v\},\(16\)The filtered gradients are then summed to form the fusion\-layer update, and the fusion layer’s gradient is overwritten with𝐠fa\+𝐠fv\\mathbf\{g\}\_\{f\}^\{a\}\+\\mathbf\{g\}\_\{f\}^\{v\}\. This approach resolves the "trilemma" component of gradient interference efficiently, ensuring that the shared decision boundary respects the geometric requirements of both modalities without the overhead of global gradient projection\.

Why fusion\-only PCGrad\.Gradient conflicts in multimodal networks arise predominantly at shared representational bottlenecks, where modality\-specific signals compete to shape a common decision boundary\. Applying projection\-based methods such as PCGrad globally treats unimodal encoders as competing tasks, which can inadvertently suppress modality\-specific feature learning and increase computational overhead\. CAT\-GS instead applies PCGrad\-like projection only at the fusion output layer, where cross\-modal gradients first interact\. This localized projection preserves unimodal encoder gradients while explicitly enforcing cooperative geometry at the shared fusion parameters, yielding stable convergence at substantially lower cost than full\-network gradient projection\.

Cost perspective\.ForMMmodalities, projection\-based conflict handling is pairwise \(O⁡\(M2\)O\(M^\{2\}\)\)\. Applying it globally scales with the full parameter count\|θ\|\|\\theta\|, whereas fusion\-only surgery scales with the fusion head size\|θf\|\|\\theta\_\{f\}\|\(typically\|θf\|≪\|θ\|\|\\theta\_\{f\}\|\\ll\|\\theta\|\):O⁡\(M2​\|θ\|\)O\(M^\{2\}\|\\theta\|\)vs\.O⁡\(M2​\|θf\|\)O\(M^\{2\}\|\\theta\_\{f\}\|\)\. This substantially reduces overhead and avoids unnecessarily constraining unimodal encoder learning\.

Algorithm 1CAT\-GS Training Step1:Input:

\(xa,xv,y\)\(x^\{a\},x^\{v\},y\), teachers

Ta,TvT^\{a\},T^\{v\}, student

SS
2:Hyperparameters:

τlow,τhigh,Ew,pdrop,β,βg,\\tau\_\{\\text\{low\}\},\\tau\_\{\\text\{high\}\},E\_\{w\},p\_\{\\text\{drop\}\},\\beta,\\beta\_\{g\},
3:

λb​i​a​s,γcap,ε,δ\\lambda\_\{bias\},\\gamma\_\{\\text\{cap\}\},\\varepsilon,\\delta
4:1\. Teacher Reliability

5:

za←Ta​\(xa\)z^\{a\}\\leftarrow T^\{a\}\(x^\{a\}\),

zv←Tv​\(xv\)z^\{v\}\\leftarrow T^\{v\}\(x^\{v\}\)
6:Compute calibrated confidences

pta,ptvp\_\{t\}^\{a\},p\_\{t\}^\{v\}\(Eq\. \([4](https://arxiv.org/html/2608.24947#S3.E4)\)\)

7:Update EMA reliabilities

p^a,p^v\\hat\{p\}^\{a\},\\hat\{p\}^\{v\}\(Eq\. \([5](https://arxiv.org/html/2608.24947#S3.E5)\)\)

8:2\. Student Forward & Backward

9:

fsa,fsv,𝐬←S⁡\(xa,xv\)f\_\{s\}^\{a\},f\_\{s\}^\{v\},\\mathbf\{s\}\\leftarrow S\(x^\{a\},x^\{v\}\)
10:Compute

ℒ\\mathcal\{L\}and gradients

∇θ\\nabla\_\{\\theta\}
11:3\. Gating Regime Selection

12:

Δ←\|p^a−p^v\|\\Delta\\leftarrow\|\\hat\{p\}^\{a\}\-\\hat\{p\}^\{v\}\|\(Eq\. \([6](https://arxiv.org/html/2608.24947#S3.E6)\)\)

13:if

Δ<τlow\\Delta<\\tau\_\{\\text\{low\}\}and epoch

<Ew<E\_\{w\}then

14:Apply stochastic modality dropout with probability

pdropp\_\{\\text\{drop\}\}
15:elseif

Δ\>τhigh\\Delta\>\\tau\_\{\\text\{high\}\}then

16:Apply dominance gating \(Eq\. \([8](https://arxiv.org/html/2608.24947#S3.E8)\)\)

17:else

18:Compute weak\-bias weights

αa,αv\\alpha\_\{a\},\\alpha\_\{v\}\(Eqs\. \([9](https://arxiv.org/html/2608.24947#S3.E9)\)–\([10](https://arxiv.org/html/2608.24947#S3.E10)\)\)

19:endif

20:4\. Gradient Modulation

21:

∇θm←αm​∇θm\\nabla\_\{\\theta\_\{m\}\}\\leftarrow\\alpha\_\{m\}\\nabla\_\{\\theta\_\{m\}\}\(Eq\. \([11](https://arxiv.org/html/2608.24947#S3.E11)\)\)

22:5\. Budget Reallocation

23:Update

g^m\\hat\{g\}^\{m\}and rescale gradients \(Eqs\. \([12](https://arxiv.org/html/2608.24947#S3.E12)\)–\([13](https://arxiv.org/html/2608.24947#S3.E13)\)\)

24:6\. Fusion\-Layer Surgery

25:Compute fusion\-output\-layer gradients

𝐠fa,𝐠fv\\mathbf\{g\}\_\{f\}^\{a\},\\mathbf\{g\}\_\{f\}^\{v\}via Eq\. \([14](https://arxiv.org/html/2608.24947#S3.E14)\); if

cos⁡\(𝐠fa,𝐠fv\)<0\\cos\(\\mathbf\{g\}\_\{f\}^\{a\},\\mathbf\{g\}\_\{f\}^\{v\}\)<0, apply PCGrad\-style projection \(Eqs\. \([15](https://arxiv.org/html/2608.24947#S3.E15)\)–\([16](https://arxiv.org/html/2608.24947#S3.E16)\)\) and overwrite fusion\-layer gradient with

𝐠fa\+𝐠fv\\mathbf\{g\}\_\{f\}^\{a\}\+\\mathbf\{g\}\_\{f\}^\{v\}
26:7\. Optimizer Update

27:

θ←θ−η​∇θ\\theta\\leftarrow\\theta\-\\eta\\nabla\_\{\\theta\}

Table 1:Accuracy \(%\) of unimodal teachers \(TaT^\{a\},TvT^\{v\}\) and multimodal students on audio–visual benchmarks\. For each dataset, the “Audio” and “Video” rows present the performance of the student’s modality\-specific encoders, and the “Multi” row reports the fused multimodal output\. The best and second\-best multimodal scores are highlighted in bold and underlined, respectively \(where applicable\)\.For CAT\-GS, we report mean±\\pmstd when repeated runs are available\.DatasetTaT^\{a\}TvT^\{v\}CREMA\-DAudio61\.96\-59\.9554\.8654\.8648\.5849\.1958\.6059\.2765\.4657\.7154\.3761\.0256\.4555\.38Video\-76\.4827\.4222\.5726\.3157\.8523\.2549\.0664\.9155\.2465\.2170\.8925\.4072\.7275\.23Multi\-\-67\.4760\.9964\.4278\.4859\.1372\.1879\.7075\.1379\.8283\.6267\.6185\.8986\.29±\\pm0\.15AV\-MNISTAudio42\.70\-16\.0527\.5022\.7238\.9037\.6024\.5342\.2642\.1141\.5041\.9931\.5539\.1041\.28Video\-65\.3455\.8363\.3462\.9263\.6558\.5055\.8565\.3065\.2664\.2865\.0464\.0865\.0964\.89Multi\-\-69\.7770\.6870\.6272\.1471\.8271\.0865\.3272\.6372\.4772\.1472\.3373\.0373\.21±\\pm0\.08VGGSoundAudio43\.39\-39\.2239\.5739\.1038\.1526\.3037\.9637\.5642\.4442\.3541\.5442\.1239\.4339\.30Video\-32\.3218\.7017\.8518\.6625\.657\.1222\.6432\.0217\.9418\.1223\.6523\.7729\.8828\.84Multi\-\-50\.9750\.7650\.9847\.1133\.0751\.4551\.6549\.6950\.9752\.7453\.7853\.8253\.24±\\pm0\.24

### 3\.7Extension toMMModalities and Unified Training Pipeline

CAT\-GS extends toM\>2M\{\>\}2modalities by computing per\-modality stabilized reliabilities\{p^m\}m∈ℳ\\\{\\hat\{p\}^\{m\}\\\}\_\{m\\in\\mathcal\{M\}\}and using their spread to trigger the same regime policy\. For the generalMM\-modality setting, we replace the two\-modality reliability gap of Eq\. \([6](https://arxiv.org/html/2608.24947#S3.E6)\) with the max–min confidence margin over all available modalities:

ΔM=maxm∈ℳ⁡p^m−minm∈ℳ⁡p^m,\\Delta\_\{M\}=\\max\_\{m\\in\\mathcal\{M\}\}\\hat\{p\}^\{m\}\-\\min\_\{m\\in\\mathcal\{M\}\}\\hat\{p\}^\{m\},\(17\)and identifym−=arg⁡minm⁡p^mm^\{\-\}\{=\}\\arg\\min\_\{m\}\\hat\{p\}^\{m\}andm\+=arg⁡maxm⁡p^mm^\{\+\}\{=\}\\arg\\max\_\{m\}\\hat\{p\}^\{m\}\.The same CAT\-GS controller logic is used across datasets; only the number of modalities and corresponding branch/group definitions are adapted\.The tri\-modal UR\-FUNNY benchmark \(M=3M\{=\}3, withℳ=\{a,v,t\}\\mathcal\{M\}=\\\{a,v,t\\\}\) is one instance of this general formulation; the controller itself is not dataset\-specific\. The gating coefficients\{αm\}\\\{\\alpha\_\{m\}\\\}are computed by: \(i\) warm\-up dropout \(optionally keep one random modality\), \(ii\) dominance mode ifΔM\>τhigh\\Delta\_\{M\}\>\\tau\_\{\\text\{high\}\}\(setαm−=1\\alpha\_\{m^\{\-\}\}\{=\}1, others 0\), else \(iii\) a floored proportional split with weak bias applied tom−m^\{\-\}andm\+m^\{\+\}:

αm∝max⁡\(p^m∑j∈ℳp^j,ε\)⋅\{1\+λbias,m=m−,1−λbias,m=m\+,1,otherwise,\\alpha\_\{m\}\\propto\\max\\\!\\left\(\\frac\{\\hat\{p\}^\{m\}\}\{\\sum\_\{j\\in\\mathcal\{M\}\}\\hat\{p\}^\{j\}\},\\,\\varepsilon\\right\)\\cdot\\begin\{cases\}1\+\\lambda\_\{\\text\{bias\}\},&m=m^\{\-\},\\\\ 1\-\\lambda\_\{\\text\{bias\}\},&m=m^\{\+\},\\\\ 1,&\\text\{otherwise\},\\end\{cases\}\(18\)
followed by renormalization\. Encoder gradients are then scaled as∇θm←αm​∇θm\\nabla\_\{\\theta\_\{m\}\}\\leftarrow\\alpha\_\{m\}\\nabla\_\{\\theta\_\{m\}\}with the same budget\-preserving renormalization from Section[3\.5](https://arxiv.org/html/2608.24947#S3.SS5)to stabilize update magnitudes under aggressive gating\.

Fusion\-only PCGrad generalizes by computing modality\-specific fusion gradients\{𝐠fm\}\\\{\\mathbf\{g\}\_\{f\}^\{m\}\\\}and applying standard pairwise PCGrad projections sequentially across conflicting pairs, then updating the fusion head with the summed projected gradients\.

Unified training pipeline\.The previous sections introduced the CAT\-GS components individually\. We now summarize how these components operate together during a single training iteration\. As shown in Algorithm[1](https://arxiv.org/html/2608.24947#alg1), teacher reliability estimation generates the control signal for adaptive gating, which determines modality\-wise gradient scaling\. Gradient\-budget reallocation then stabilizes the magnitude of gated gradients, and fusion\-layer surgery removes cross\-modal conflicts before parameters are updated\.

This pipeline consolidates CAT\-GS into a lightweight controller applied during optimization\. It introduces no changes to the forward computation and only modest overhead in the backward pass, making it easily applicable to a wide range of multimodal architectures\.

##### Local Stability Analysis

CAT\-GS is an optimization\-stage controller that shapes the effective update direction without altering the objective\. We provide a local descent guarantee for the induced update\.

Letℒ⁡\(θ\)\\mathcal\{L\}\(\\theta\)denote the training objective and𝐮t\\mathbf\{u\}\_\{t\}the CAT\-GS update after gating, budget reallocation, and fusion\-only surgery:

θt\+1=θt−η​𝐮t\.\\theta\_\{t\+1\}=\\theta\_\{t\}\-\\eta\\mathbf\{u\}\_\{t\}\.\(19\)
Assume: \(A1\)ℒ\\mathcal\{L\}isLL\-smooth; \(A2\)‖𝐮t‖≤C​‖∇ℒ​\(θt\)‖\\\|\\mathbf\{u\}\_\{t\}\\\|\\leq C\\\|\\nabla\\mathcal\{L\}\(\\theta\_\{t\}\)\\\|; and \(A3\)

⟨∇ℒ​\(θt\),𝐮t⟩≥ρ​‖∇ℒ​\(θt\)‖​‖𝐮t‖,ρ∈\(0,1\]\.\\langle\\nabla\\mathcal\{L\}\(\\theta\_\{t\}\),\\mathbf\{u\}\_\{t\}\\rangle\\geq\\rho\\,\\\|\\nabla\\mathcal\{L\}\(\\theta\_\{t\}\)\\\|\\,\\\|\\mathbf\{u\}\_\{t\}\\\|,\\quad\\rho\\in\(0,1\]\.\(20\)
Then

ℒ⁡\(θt\+1\)≤ℒ⁡\(θt\)−η⁡\(ρ−L​η​C2\)​‖∇ℒ​\(θt\)‖2\.\\mathcal\{L\}\(\\theta\_\{t\+1\}\)\\leq\\mathcal\{L\}\(\\theta\_\{t\}\)\-\\eta\\left\(\\rho\-\\frac\{L\\eta C\}\{2\}\\right\)\\\|\\nabla\\mathcal\{L\}\(\\theta\_\{t\}\)\\\|^\{2\}\.\(21\)
Hence, for0<η<2​ρL​C0<\\eta<\\frac\{2\\rho\}\{LC\}, CAT\-GS guarantees one\-step monotone descent\. This reflects: \(i\) EMA smoothing stabilizes regime transitions, \(ii\) budget caps controlCC, and \(iii\) fusion\-only surgery improves alignment \(largerρ\\rho\)\.This is a local, one\-step descent guarantee for the CAT\-GS update, not a global convergence result; the broader stabilization behavior is supported empirically in Section[4](https://arxiv.org/html/2608.24947#S4)\.

## 4Experiments

We evaluate CAT\-GS on audio–visual, tri\-modal, and controlled synthetic settings to answer the following questions: \(Q1\) Does CAT\-GS improve multimodal accuracy compared to state\-of\-the\-art imbalance\-aware training methods? \(Q2\) How effectively does CAT\-GS stabilize unimodal branches and mitigate modality starvation? \(Q3\) Does CAT\-GS reduce gradient conflicts and gate thrashing during optimization? \(Q4\) How do the individual components of CAT\-GS contribute to overall performance?

### 4\.1Experimental Setup

Datasets\.Following prior work on multimodal imbalance and gradient modulation, we evaluate CAT\-GS on a representative mix of audio–visual, tri\-modal, and controlled synthetic benchmarks\.CREMA\-D[3](https://arxiv.org/html/2608.24947#bib.bib30)is an audio–visual emotion recognition dataset with 7,442 clips, 6 emotion classes, and paired speech–face recordings\.AV\-MNIST[29](https://arxiv.org/html/2608.24947#bib.bib32)combines spoken\-digit audio with PCA\-compressed MNIST images for 10\-class classification, and is widely used to study modality dominance under noise\.VGGSound[5](https://arxiv.org/html/2608.24947#bib.bib8)is a large\-scale audio–visual benchmark with diverse in\-the\-wild categories\.UR\-FUNNY[12](https://arxiv.org/html/2608.24947#bib.bib31)is a tri\-modal \(audio, visual, text\) humor detection dataset with 16k labeled segments and speaker\-independent splits, enabling evaluation under stronger cross\-modal interactions\.CG\-MNIST[17](https://arxiv.org/html/2608.24947#bib.bib40)is a controlled synthetic variant of MNIST that allows explicit manipulation of modality correlations, making it suitable for analyzing imbalance and gradient starvation effects\.Audio\-Visual Event \(AVE\)[26](https://arxiv.org/html/2608.24947#bib.bib26)is an additional benchmark with 4,143 videos across 28 event categories, where videos are temporally labeled with audio–visual event boundaries\.CMU\-MOSI[39](https://arxiv.org/html/2608.24947#bib.bib27)is an additional benchmark containing 2,199 opinion video clips, each annotated with sentiment\.We use AVE and CMU\-MOSI as additional benchmarks to evaluate generalization beyond the core benchmark set\.All datasets in this work are established public benchmarks obtained from official online sources\.

Baselines\.We compare CAT\-GS with ten representative multimodal imbalance\-mitigation or gradient\-modulation methods: MSES[8](https://arxiv.org/html/2608.24947#bib.bib13), MSLR[37](https://arxiv.org/html/2608.24947#bib.bib14), AGM[20](https://arxiv.org/html/2608.24947#bib.bib15), PMR[7](https://arxiv.org/html/2608.24947#bib.bib16), OGM\-GE[21](https://arxiv.org/html/2608.24947#bib.bib17), MLA[40](https://arxiv.org/html/2608.24947#bib.bib18), MM\-Pareto[31](https://arxiv.org/html/2608.24947#bib.bib19), ReconBoost[15](https://arxiv.org/html/2608.24947#bib.bib20), DLMG[36](https://arxiv.org/html/2608.24947#bib.bib21), and UMT[6](https://arxiv.org/html/2608.24947#bib.bib22)\. These baselines span gradient\-based rebalancing, multi\-objective optimization, reconstruction\-based regularization, and unimodal\-teacher\-guided training\.

Backbone Architectures and Hyperparameters\.For audio–visual datasets \(CREMA\-D and AV\-MNIST\), we employResNet\-18encoders[13](https://arxiv.org/html/2608.24947#bib.bib35)for both audio and video modalities across student and teacher models\. For the tri\-modal UR\-FUNNY dataset, we use lightweightTransformer encoders[28](https://arxiv.org/html/2608.24947#bib.bib36)for text, audio, and visual streams to maintain consistent modality capacity\.Teacher Protocol\.To ensure high\-quality guidance, all teacher models are pre\-trained individually on their respective unimodal datasets until convergence\. During student training, these teachers arefrozenand used solely for inference to provide logits for the calibration module\. This setup ensures that the reliability estimatesptmp\_\{t\}^\{m\}are derived from stable, converged experts rather than evolving peers\.

Table 2:Accuracy \(%\) on the UR\-FUNNY dataset for bi\-modal and tri\-modal combinations \(A–V, A–TXT, V–TXT, and A–V–TXT\)\. Unimodal teacher performance for audio, visual, and text are58\.67%58\.67\\%,56\.84%56\.84\\%, and63\.48%63\.48\\%, respectively\.TypeCAT\-GSA\-VAudio57\.3459\.7661\.7760\.5354\.6359\.0558\.25Visual53\.9253\.8255\.7357\.8756\.4458\.0554\.37Multi61\.5761\.8761\.2762\.0760\.4662\.9862\.12±\\pm0\.18A\-TXTAudio50\.3054\.1258\.1550\.1855\.6359\.8659\.42Text57\.4458\.3558\.4556\.9857\.7558\.8559\.30Multi62\.1762\.4762\.8061\.0662\.4763\.2864\.49±\\pm1\.05V\-TXTVisual49\.3055\.3356\.0455\.4156\.3456\.3456\.91Text51\.2158\.9559\.1550\.9453\.8256\.0457\.60Multi62\.0762\.9861\.2760\.0763\.1863\.4865\.25±\\pm0\.97A\-V\-TXTAudio55\.0350\.3058\.0551\.6550\.7059\.1559\.60Visual54\.9355\.7356\.1455\.2654\.9355\.9455\.41Text58\.2555\.7158\.5556\.2552\.7258\.1560\.72Multi62\.5863\.6862\.8861\.3763\.3865\.4967\.55±\\pm0\.84

![Refer to caption](https://arxiv.org/html/2608.24947v1/Figure4.png)Figure 4:Gradient conflict dynamics at the fusion layer\. CAT\-GS achieves higher gradient alignment and fewer conflicting updates than Joint\-Train and G2D, indicating effective mitigation of destructive interference\.Table 3:Additional benchmark comparison on AVE and CMU\-MOSI\. We report accuracy \(%\) for representative balancing baselines and CAT\-GS under the corresponding setup family \(AVE: ResNet V–A; MOSI: Transformer V–T and V–A–T\)\.MethodAVE\(ResNet V–A\)CMU\-MOSI\(Transformer V–T\)CMU\-MOSI\(Transformer V–A–T\)MSLR[37](https://arxiv.org/html/2608.24947#bib.bib14)67\.3––MMCosine[35](https://arxiv.org/html/2608.24947#bib.bib2)65\.0––OGM[21](https://arxiv.org/html/2608.24947#bib.bib17)67\.373\.9–AGM[20](https://arxiv.org/html/2608.24947#bib.bib15)68\.474\.073\.9ReconBoost[15](https://arxiv.org/html/2608.24947#bib.bib20)68\.4––MM\-Pareto[31](https://arxiv.org/html/2608.24947#bib.bib19)73\.073\.473\.7MCR[19](https://arxiv.org/html/2608.24947#bib.bib3)73\.475\.276\.5DRL[32](https://arxiv.org/html/2608.24947#bib.bib4)––77\.99CAT\-GS \(Ours\)74\.2±\\pm0\.276\.0±\\pm1\.278\.3±\\pm1\.0

Table 4:Performance comparison on the CG\-MNIST dataset\. Modality 1: monochromatic image; Modality 2: gray\-scale image\.ModalityPMR[7](https://arxiv.org/html/2608.24947#bib.bib16)OGM\-GE[21](https://arxiv.org/html/2608.24947#bib.bib17)MSES[8](https://arxiv.org/html/2608.24947#bib.bib13)MSLR[37](https://arxiv.org/html/2608.24947#bib.bib14)G2D[25](https://arxiv.org/html/2608.24947#bib.bib9)CAT\-GSMono99\.3099\.3899\.2699\.3099\.2699\.32Gray60\.4067\.2160\.8559\.9663\.3667\.85Multi78\.5097\.3593\.4695\.0497\.0897\.42±\\pm0\.23

Training Details\.Unless otherwise noted, we train for 400 epochs with batch size 16 using SGD \(momentum 0\.9, weight decay10−410^\{\-4\}\) and an initial learning rate of10−310^\{\-3\}, decayed by a factor of 0\.1 at epoch 200\. We fix the CAT\-GS hyperparametersβ=0\.9\\beta=0\.9,βg=0\.9\\beta\_\{g\}=0\.9,γcap=1\.5\\gamma\_\{\\mathrm\{cap\}\}=1\.5,ε=0\.1\\varepsilon=0\.1,λbias=0\.2\\lambda\_\{\\mathrm\{bias\}\}=0\.2, warm\-up epochsEw=5E\_\{w\}=5, and dropout probabilitypdrop=0\.6p\_\{\\mathrm\{drop\}\}=0\.6across datasets, while tuning the regime thresholdsτlow\\tau\_\{\\mathrm\{low\}\}andτhigh\\tau\_\{\\mathrm\{high\}\}once per dataset\.In all experiments except the explicit sensitivity analysis, we use identical CAT\-GS hyperparameters across datasets\.

Evaluation Protocol and Reproducibility\.We report top\-1 classification accuracy \(%\) on the held\-out test split for each benchmark\. During training, we evaluate once per epoch on a validation split and select the checkpoint with the best validation accuracy; we then report test accuracy once using this selected checkpoint\. Unless otherwise noted, we report the average test accuracy \(%\) over three random seeds \(42, 123, and 999\)\.For CAT\-GS, we additionally report run\-to\-run variability as mean±\\pmstd in Tables[1](https://arxiv.org/html/2608.24947#S3.T1),[2](https://arxiv.org/html/2608.24947#S4.T2), and[4](https://arxiv.org/html/2608.24947#S4.T4)\.Withn=3n\{=\}3runs, 95% confidence intervals are computed asx¯±t0\.975,2​s/3\\bar\{x\}\\pm t\_\{0\.975,2\}\\,s/\\sqrt\{3\}\.CG\-MNIST experiments use small CNN encoders to isolate optimization behavior from representational capacity\.All methods use identical backbones and fusion heads to ensure fair comparison\.Experiments are conducted on NVIDIA RTX 3060 and RTX 4050 GPUs\.

### 4\.2Results

Audio–Visual Classification Benchmarks\.Table[1](https://arxiv.org/html/2608.24947#S3.T1)reports results on CREMA\-D, AV\-MNIST, and VGGSound\.In Table[1](https://arxiv.org/html/2608.24947#S3.T1),Joint\-Traindenotes the original model trained with standard end\-to\-end optimization \(CAT\-GS disabled\) under the same backbone, fusion head, optimizer schedule, and data split as CAT\-GS for fair attribution\.Across datasets, CAT\-GS consistently remains competitive with imbalance\-aware baselines spanning gradient modulation \(e\.g\., OGM\-GE[21](https://arxiv.org/html/2608.24947#bib.bib17), AGM[20](https://arxiv.org/html/2608.24947#bib.bib15), PMR[7](https://arxiv.org/html/2608.24947#bib.bib16)\) and teacher\-guided training \(UMT[6](https://arxiv.org/html/2608.24947#bib.bib22)\)\.

On CREMA\-D, where the video modality is substantially stronger than audio, CAT\-GS improves the fused prediction to 86\.29%±\\pm0\.15 \(vs\. 85\.89% for G2D\), indicating that stabilizing gating and resolving fusion conflicts can yield gains beyond confidence\-only suppression\.On AV\-MNIST, where modalities are closer in strength, CAT\-GS remains competitive and achieves the best fused accuracy \(73\.21%±\\pm0\.08\), suggesting the controller does not over\-correct when imbalance is mild\.

On VGGSound, CAT\-GS reaches 53\.24%±\\pm0\.24 fused accuracy and does not surpass the strongest baselines, falling below G2D \(53\.82%\) and UMT \(53\.78%\) by roughly 0\.5–0\.6 points\. This reflects the large\-scale in\-the\-wild regime rather than the training setup: the large label space \(∼\\sim300 classes\) and noisy clips yield low, unstable teacher reliabilities, so the marginΔ\\Deltathat drives gating is weakly informative, and accuracy at this scale is dominated by encoder capacity and data volume rather than by gradient\-level control\. CAT\-GS therefore offers limited benefit in this regime\.The unimodal branches on VGGSound follow the same pattern and remain close to G2D, with slightly lower Audio/Video accuracies \(39\.30 vs\. 39\.43 and 28\.84 vs\. 29\.88\)\.

For completeness, we also include a no\-teacher standard\-training reference on CREMA\-D: training the student with only the supervised classification objective \(i\.e\., without distillation and without CAT\-GS modulation\) reaches 64\.78% best fused accuracy under the same backbone, fusion setup, and training schedule, and remains substantially below CAT\-GS, consistent with the relative ordering reported in Table[1](https://arxiv.org/html/2608.24947#S3.T1)\.

Figure 5:Optimization diagnostics on CREMA\-D\. \(a\) Audio\-gating trajectory over 600 steps: OGM\-GE and PMR show high\-frequency oscillations \(gate thrashing\), whereas CAT\-GS remains smooth under EMA\-stabilized control\. \(b\) Cumulative gradient\-budget ratioΣ​‖∇v‖/Σ​‖∇a‖\\Sigma\\\|\\nabla\_\{v\}\\\|/\\Sigma\\\|\\nabla\_\{a\}\\\|: Joint\-Train reaches 3\.2×\\timesimbalance, G2D remains near 2\.0×\\times, and CAT\-GS reduces this to 1\.7×\\timesthrough budget\-preserving reallocation\.Figure 6:Decomposed gating weight trajectories \(αaudio\\alpha\_\{\\mathrm\{audio\}\}\) across training steps for Joint, OGM\-GE, PMR, G2D, and CAT\-GS\.![Refer to caption](https://arxiv.org/html/2608.24947v1/cm_all_datasets.png)Figure 7:Unimodal correctness versus multimodal outcome for Joint\-Train, G2D, and CAT\-GS \(columns\) on CREMA\-D, AV\-MNIST, and VGGSound \(rows\)\.Within each panel, columns group test samples by which unimodal branches are correct and rows give the multimodal outcome\.The corresponding 95% confidence intervals for CAT\-GS are \[85\.92, 86\.66\] \(CREMA\-D\), \[73\.01, 73\.41\] \(AV\-MNIST\), and \[52\.64, 53\.84\] \(VGGSound\)\.We note that fused accuracy is the primary objective; unimodal branch accuracies can change in either direction depending on how much specialization toward fusion is beneficial\.

Tri\-Modal Humor Understanding \(UR\-FUNNY\)\.Table[2](https://arxiv.org/html/2608.24947#S4.T2)reports results on UR\-FUNNY across increasing modality combinations, from audio–visual \(A–V\) to text\-inclusive settings\. In the A–V configuration, CAT\-GS performs competitively but does not outperform the strongest baseline \(G2D\), which is expected given the reduced modality interactions and weaker gradient conflicts in the bi\-modal setting\. In contrast, when textual modality is introduced, CAT\-GS consistently achieves the best multimodal accuracy in both A–TXT and V–TXT settings, and yields the largest gains in the full A–V–TXT configuration\. Specifically, CAT\-GS improves over multi\-objective optimization \(MM\-Pareto[31](https://arxiv.org/html/2608.24947#bib.bib19)\) and reconstruction\-based regularization \(ReconBoost[15](https://arxiv.org/html/2608.24947#bib.bib20)\), highlighting the effectiveness of calibrated gating, gradient\-budget stabilization, and fusion\-layer conflict mitigation under more challenging tri\-modal fusion\.For CAT\-GS on UR\-FUNNY, the 95% confidence intervals are \[61\.67, 62\.57\] \(A–V\), \[61\.88, 67\.10\] \(A–TXT\), \[62\.84, 67\.66\] \(V–TXT\), and \[65\.46, 69\.64\] \(A–V–TXT\)\.

Additional Benchmarks \(AVE and CMU\-MOSI\)\.Table[3](https://arxiv.org/html/2608.24947#S4.T3)extends evaluation to additional benchmark families and includes recent baselines such as MCR and DRL\. CAT\-GS remains strongest on AVE and on the MOSI two\-modality setting, and reaches 78\.3%±\\pm1\.0 on MOSI three\-modality classification, slightly above DRL \(77\.99\) and clearly above MCR \(76\.5\)\. The corresponding 95% confidence intervals for CAT\-GS are \[73\.70, 74\.70\] on AVE, \[73\.02, 78\.98\] on CMU\-MOSI \(V–T\), and \[75\.82, 80\.78\] on CMU\-MOSI \(V–A–T\)\. Compared with earlier methods \(e\.g\., MSLR/OGM/AGM/MM\-Pareto\), the margins are larger, while against stronger recent methods they become tighter but remain positive\. Overall, these results indicate that CAT\-GS transfers to both event\-centric and sentiment\-centric benchmarks instead of being limited to the original dataset family\.

Controlled Synthetic Benchmark \(CG\-MNIST\)\.Results on CG\-MNIST are shown in Table[4](https://arxiv.org/html/2608.24947#S4.T4)\. This benchmark allows explicit control over label correlations between the monochromatic and grayscale modalities\. Even when one modality is made spuriously more predictive early in training, CAT\-GS maintains strong fused performance and avoids brittle training dynamics observed in several baselines\. This controlled setting supports the role of margin\-thresholded gating and budget\-aware stabilization in reducing the risk that prolonged dominance suppresses the weaker modality\.For CAT\-GS on CG\-MNIST, the 95% confidence interval is \[96\.85, 97\.99\]\.

Evidence for Reduced Gradient Conflicts\.To quantify cross\-modal interference, we follow the implementation and define a modality\-specific fusion gradient surrogate𝐠fm\\mathbf\{g\}\_\{f\}^\{m\}on the fusion output layer \(e\.g\.,fc\_out\) using the logit\-sum gradients from Eq\. \([14](https://arxiv.org/html/2608.24947#S3.E14)\)\. We measure gradient alignment using the cosine similaritycos⁡\(𝐠fa,𝐠fv\)\\cos\(\\mathbf\{g\}\_\{f\}^\{a\},\\mathbf\{g\}\_\{f\}^\{v\}\)\. As shown in Figure[4](https://arxiv.org/html/2608.24947#S4.F4)\(top\), CAT\-GS maintains a positive average cosine similarity throughout training, indicating a cooperative optimization regime at the fusion layer\. Correspondingly, Figure[4](https://arxiv.org/html/2608.24947#S4.F4)\(bottom\) shows that CAT\-GS substantially reduces the fraction of training batches with negative cosine similarity, directly evidencing suppression of destructive gradient interference\.

Gate Stability and Gradient Starvation Analysis\.We quantify gating stability using the Total Variation \(TV\) of the gating coefficients\. CAT\-GS demonstrates smooth gating transitions with low total variation, confirming that calibration and EMA smoothing effectively filter out high\-frequency noise \(“gate thrashing”\)\. To assess starvation risk under gating, we track the gradient norms of the unimodal encoders\. CAT\-GS’sε\\varepsilon\-floored soft regimes reduce accidental gradient collapse, and its capped budget reallocation stabilizes update magnitudes during hard\-gating events\.

Observed failure\-mode behavior\.Figure[5](https://arxiv.org/html/2608.24947#S4.F5)summarizes optimization dynamics from CREMA\-D training logs to illustrate gate thrashing and dominant\-modality trapping\. In panel \(a\), the audio gateαaudio\\alpha\_\{\\mathrm\{audio\}\}is plotted over 600 steps: OGM\-GE and PMR exhibit pronounced oscillations under batch\-wise modulation, whereas CAT\-GS produces smooth trajectories using EMA\-smoothed reliability \(β=0\.9\\beta=0\.9\) and margin\-based regime switching\. In panel \(b\), we report the cumulative gradient budget ratioΣ​‖∇v‖/Σ​‖∇a‖\\Sigma\\\|\\nabla\_\{v\}\\\|/\\Sigma\\\|\\nabla\_\{a\}\\\|\. Joint training accumulates a3\.2×3\.2\\timesimbalance; gradient\-modulation baselines remain around∼2\.4×\\sim 2\.4\\times; G2D reaches∼2\.0×\\sim 2\.0\\times; and CAT\-GS maintains1\.7×1\.7\\timesthrough budget\-preserving reallocation \(Eq\. \([13](https://arxiv.org/html/2608.24947#S3.E13)\)\)\. Together, these diagnostics indicate that prior approaches can suffer from gate instability and persistent gradient imbalance, which CAT\-GS is designed to address\.

For clearer interpretation of panel \(a\), Figure[6](https://arxiv.org/html/2608.24947#S4.F6)shows decomposed gating trajectories \(raw and smoothed\) for Joint, OGM\-GE, PMR, G2D, and CAT\-GS\. In this decomposed view, each method is shown with a raw per\-step trajectory and a smoothed trend \(moving average\), which separates high\-frequency jitter from underlying control behavior\. Joint remains largely static, indicating weak reliability\-driven adaptation\. OGM\-GE and PMR show strong high\-frequency fluctuation and jagged trends, consistent with unstable batch\-reactive routing\. G2D reduces fluctuation amplitude relative to OGM\-GE/PMR but still exhibits frequent short\-window switching\. CAT\-GS is the smoothest, reducing oscillation while preserving gradual trend adaptation, which is consistent with its EMA\-stabilized and margin\-thresholded controller design\.

To complement this optimization\-dynamics view, we further provide a confusion\-matrix\-style qualitative analysison the three core audio–visual benchmarks \(CREMA\-D, AV\-MNIST, and VGGSound\),in which each test sample is assigned to a single cell according to which unimodal branches predicted it correctly and whether the fused prediction was correct \(Figure[7](https://arxiv.org/html/2608.24947#S4.F7)\)\. The top\-left cell holds the samples that neither branch predicted correctly and that fusion did not recover, which are limited by the unimodal encoders; the wrong row of the remaining columns holds those on which at least one branch was correct but the fused prediction was not, which are limited by the fusion stage\. Together the two account for all errors\. On CREMA\-D, CAT\-GS reduces both relative to Joint\-Train, from19\.5%19\.5\\%to7\.4%7\.4\\%and from13\.4%13\.4\\%to6\.3%6\.3\\%, consistent with the reduced negative fusion\-gradient cosine events in Figure[4](https://arxiv.org/html/2608.24947#S4.F4)\. The fusion\-stage loss is lower than that of G2D on all three benchmarks \(6\.3%6\.3\\%vs\.7\.5%7\.5\\%,10\.7%10\.7\\%vs\.11\.7%11\.7\\%,11\.8%11\.8\\%vs\.12\.4%12\.4\\%\), but on VGGSound the encoder\-limited error is larger \(35\.1%35\.1\\%vs\.33\.4%33\.4\\%\) and outweighs that advantage\. This places the VGGSound shortfall in the unimodal branches rather than at the fusion layer, consistent with the weakly informative reliability marginΔ\\Deltaand the large label space at this scale\.

### 4\.3Ablation Studies

We perform two complementary ablations: a unified component matrix with both single\-component activation and leave\-one\-out removal \(Table[5](https://arxiv.org/html/2608.24947#S4.T5)\), and architectural ablation over fusion modules \(Table[6](https://arxiv.org/html/2608.24947#S4.T6)\)\.

Table[5](https://arxiv.org/html/2608.24947#S4.T5)merges both ablation views using a tick\-mark configuration: \(i\) starting from Joint\-Train and enabling one component at a time, and \(ii\) starting from full CAT\-GS and removing one component at a time\. This gives a compact view of standalone contribution and component necessity in one place\.

Unified Component Ablation \(Table[5](https://arxiv.org/html/2608.24947#S4.T5)\)\.Gating acts as the controller’s hub: the calibrated, EMA\-smoothed reliability margin drives only the gating regime selector, the weak\-bias rule is itself a gating regime, and gradient\-budget reallocation is invoked only at hard\-gate \(α=0\\alpha\{=\}0\) events\. Calibrated gating is therefore active in every variant that improves over Joint\-Train, so we omit calibration and gating as columns and vary the four remaining components; removing gating reduces CAT\-GS to fusion\-only PCGrad, the “\+ Fusion\-only PCGrad only” row \(77\.8077\.80,\+10\.33\+10\.33\)\.In the leave\-one\-out block, “w/o Weak\-Bias” disables only the weak\-bias factor by settingλbias=0\\lambda\_\{\\text\{bias\}\}\{=\}0, while keeping the adaptive gating controller \(warm\-up, dominance, and soft regime selection viaτlow,τhigh\\tau\_\{\\text\{low\}\},\\tau\_\{\\text\{high\}\}\) unchanged\.Turning off*gradient\-budget reallocation*produces the largest drop in the removal block, showing that gating alone is insufficient to prevent long\-term update collapse\. Discarding*EMA smoothing*also causes a substantial drop, indicating strong dependence on stabilized reliability signals\. Omitting*fusion\-layer PCGrad surgery*hurts performance to a lesser extent, indicating conflict resolution is important but secondary to reliability and budget stabilization\. The activation block complements this view by showing each component helps over Joint\-Train, while the full combination remains strongest\.

Table 5:Component\-wise contribution and necessity analysis on CREMA\-D\. Checkmarks indicate enabled components\. Calibrated reliability and gating are the always\-on base controller \(not shown as columns\); we vary the four components listed\.VariantEMAWeak\-BiasBudgetPCGradMMΔJoint\\Delta\_\{\\text\{Joint\}\}Single\-component activation \(from Joint\-Train\)Joint\-Train \(baseline\)––––67\.470\.00\+ Calibrated\+EMA reliability✓–––78\.90\+11\.43\+ Adaptive gating \(incl\. weak\-bias\)–✓––77\.20\+9\.73\+ Gradient\-budget only––✓–79\.40\+11\.93\+ Fusion\-only PCGrad only–––✓77\.80\+10\.33Leave\-one\-out removal \(from full CAT\-GS\)CAT\-GS \(full\)✓✓✓✓86\.29\+18\.82w/o EMA smoothing–✓✓✓84\.60\+17\.13w/o Weak\-Bias✓–✓✓85\.50\+18\.03w/o Grad\. Budget✓✓–✓83\.20\+15\.73w/o PCGrad✓✓✓–85\.08\+17\.61

Table 6:Performance comparison ofCAT\-GSunder different fusion strategies across multiple datasets in terms of accuracy \(%\)\.FusionCREMA\-DAV\-MNISTCG\-MNISTUR\-FUNNYSum82\.1071\.8594\.2664\.02Concat84\.7272\.9395\.8865\.01FiLM[22](https://arxiv.org/html/2608.24947#bib.bib33)83\.9572\.1292\.4363\.77BiGated[16](https://arxiv.org/html/2608.24947#bib.bib34)81\.8872\.4491\.3663\.25Cross\-Attention[4](https://arxiv.org/html/2608.24947#bib.bib37)85\.0472\.9896\.7366\.41Transformer Fusion[27](https://arxiv.org/html/2608.24947#bib.bib6)85\.7372\.9597\.0666\.56Early Fusion[9](https://arxiv.org/html/2608.24947#bib.bib5)82\.6272\.1893\.4564\.02Late Fusion[9](https://arxiv.org/html/2608.24947#bib.bib5)86\.2973\.2197\.4267\.55

Fusion Strategy Analysis\(Table[6](https://arxiv.org/html/2608.24947#S4.T6)\)\.Table[6](https://arxiv.org/html/2608.24947#S4.T6)compares different fusion modules when used inside CAT\-GS\. We explore simple additive fusion \(Sum\), concatenation \(Concat\), FiLM\-style conditioning[22](https://arxiv.org/html/2608.24947#bib.bib33), bidirectional gating \(BiGated\)[16](https://arxiv.org/html/2608.24947#bib.bib34), cross\-attention[4](https://arxiv.org/html/2608.24947#bib.bib37), and explicit Early/Late Fusion variants[9](https://arxiv.org/html/2608.24947#bib.bib5),[2](https://arxiv.org/html/2608.24947#bib.bib7)\. We observe three trends\. First, naive Sum fusion is consistently weaker than more expressive alternatives, especially on CREMA\-D and CG\-MNIST, suggesting that richer cross\-modal interactions remain beneficial even when gradients are well\-balanced\. Second, cross\-attention and concatenation form a strong pair of contenders, often approaching the best performance\.

We additionally tested a lightweight transformer\-style token\-fusion variant[27](https://arxiv.org/html/2608.24947#bib.bib6)under the same training protocol; CAT\-GS remained stable and trainable, while absolute performance differences remained architecture\-dependent\.In our experiments, Late Fusion is the primary setting \(best absolute accuracy\), and we report Early Fusion and Transformer Fusion as transfer checks under the same training protocol\.

The fusion ablation isolates the role of the architecture used at the multimodal head while keeping the CAT\-GS controller fixed\. FiLM and BiGated fusion can already capitalize on CAT\-GS, but their performance is inconsistent across datasets, likely due to the stronger inductive assumptions they impose \(e\.g\., one modality modulating the other\)\.Cross\-attention provides a strong trade\-off and often matches or slightly trails the strongest\-performing variant across datasets\.To assess transfer across fusion heads, we evaluate Transformer Fusion, Early Fusion, and Late Fusion under the same training protocol\. CAT\-GS remains stable in all cases, with expected architecture\-dependent absolute differences\.Taken together, Tables[5](https://arxiv.org/html/2608.24947#S4.T5)and[6](https://arxiv.org/html/2608.24947#S4.T6)show that \(i\) each component of CAT\-GS contributes complementary gains, and \(ii\) the method is not overly sensitive to the specific fusion module\.

Overall, CAT\-GS delivers consistent gains across classification, tri\-modal understanding, and controlled synthetic settings\. The ablations confirm that its advantages arise from the combination of stabilized reliability signals, adaptive regime switching, budget\-aware gradient stabilization, and conflict\-aware fusion\.

Teacher Miscalibration Stress Test \(Table[7](https://arxiv.org/html/2608.24947#S4.T7)\)\.CAT\-GS relies on unimodal teachers to provide modality reliability signals\. To assess robustness to errors in these signals, we introduce a controlled teacher miscalibration stress test by applying a temperature mismatch at teacher inference time\. For modalitymm, teacher logits are perturbed aszm←zm/Tmismz\_\{m\}\\leftarrow z\_\{m\}/T^\{m\}\_\{\\text\{mis\}\}before computing the confidence used by the CAT\-GS controller\.

We evaluate four regimes: \(R0\) a calibrated baseline, \(R1\) an overconfident audio teacher, \(R2\) an underconfident audio teacher, and \(R3\) both modalities miscalibrated\. To isolate the role of reliability stabilization, we further evaluate the worst\-performing mismatch \(R2\) with EMA smoothing disabled \(R4\) and temperature calibration removed \(R5\)\. All other settings are kept identical\. For R5, we bypass the calibration step when computing reliability; to avoid confusion, we omit the mismatch\-temperature entries in Table[7](https://arxiv.org/html/2608.24947#S4.T7)\. Table[7](https://arxiv.org/html/2608.24947#S4.T7)reports multimodal \(MM\) accuracy under each condition\. CAT\-GS degrades gracefully under moderate miscalibration \(R1–R2\), remaining close to the calibrated baseline\. Performance drops further under the stressed setting \(R3\) but remains stable\. In contrast, removing EMA smoothing \(R4\) or temperature calibration \(R5\) leads to a larger degradation, highlighting the importance of reliability stabilization\. Overall, CAT\-GS is robust to moderate teacher miscalibration, with stabilization mechanisms playing a critical role\.R4–R5 are best interpreted as internal stress controls \(ablation\-style reliability degradation\), not as a standalone weak\-teacher/domain\-shift benchmark\. Under these stressed settings, CAT\-GS remains trainable and stable, but final accuracy decreases relative to the strong\-teacher setting\. This supports a balanced conclusion: robustness to moderate unreliability, with clear dependence on teacher quality under stronger reliability degradation\.

Table 7:Teacher miscalibration stress test on CREMA\-D dataset\. Multimodal \(MM\) accuracy of the fused model under temperature\-mismatched teachers\.RunTmisaT^\{a\}\_\{\\text\{mis\}\}TmisvT^\{v\}\_\{\\text\{mis\}\}MM Acc\. \(%\)↑\\uparrowΔ\\Delta\(%\)↓\\downarrowR0 \(Baseline\)1\.01\.086\.30\.0R1 \(Audio overconf\.\)0\.51\.085\.2−1\.1\-1\.1R2 \(Audio underconf\.\)2\.01\.085\.1−1\.2\-1\.2R3 \(Both mismatched\)2\.00\.584\.4−1\.9\-1\.9R4 \(R2 w/o EMA\)2\.01\.083\.6−2\.7\-2\.7R5 \(R2 w/o calibration\)––84\.0−2\.3\-2\.3

Modality\-Imbalance Validation \(clean vs degraded modality\)\.To directly validate robustness to modality imbalance, we run a controlled clean\-vs\-degraded protocol on CREMA\-D under identical training settings for Joint/Normal and CAT\-GS\. We evaluate three conditions: clean \(none\), audio\-degraded, and visual\-degraded\. All reported values are fused multimodal \(MM\) test accuracy \(%\)\. Results are summarized in Table[8](https://arxiv.org/html/2608.24947#S4.T8)\.Under audio degradation, Joint/Normal drops from 67\.47% to 46\.23% \(drop 21\.24\), whereas CAT\-GS drops from 86\.29% to 71\.79% \(drop 14\.50\)\. Under visual degradation, Joint/Normal drops from 67\.47% to 55\.40% \(drop 12\.07\), whereas CAT\-GS drops from 86\.29% to 78\.29% \(drop 8\.00\)\.Therefore, CAT\-GS yields smaller degradation in both stressed conditions, supporting the claim that it improves robustness under modality imbalance\.

Table 8:Modality\-imbalance validation on CREMA\-D \(clean vs degraded modality\)\. Values are fused multimodal \(MM\) test accuracy \(%\)\. Drop is measured in percentage points asMMclean−MMcondition\\mathrm\{MM\}\_\{\\text\{clean\}\}\-\\mathrm\{MM\}\_\{\\text\{condition\}\}\.ConditionJoint\(MM %\)CAT\-GS\(MM %\)JointDrop \(pp\)CAT\-GSDrop \(pp\)Clean \(none\)67\.4786\.290\.000\.00Audio\-degraded46\.2371\.7921\.2414\.50Visual\-degraded55\.4078\.2912\.078\.00Threshold Sensitivity Analysis\.We evaluate the sensitivity of CAT\-GS toτlow\\tau\_\{\\text\{low\}\},τhigh\\tau\_\{\\text\{high\}\}, andλbias\\lambda\_\{\\text\{bias\}\}with a grid search over the three thresholds\. Across all tested combinations, the maximum absolute change in fused accuracy relative to the default setting stays below0\.50\.5percentage points \(variance<0\.5%<0\.5\\%\), with stable performance forτlow∈\[0\.03,0\.08\]\\tau\_\{\\text\{low\}\}\\in\[0\.03,0\.08\],τhigh∈\[0\.10,0\.20\]\\tau\_\{\\text\{high\}\}\\in\[0\.10,0\.20\], andλbias∈\[0\.1,0\.3\]\\lambda\_\{\\text\{bias\}\}\\in\[0\.1,0\.3\], so CAT\-GS is not highly sensitive within these ranges\. These ranges translate into a simple setting procedure:\(1\) run a short pilot \(e\.g\., first 3–5 epochs\) and log the reliability marginΔ\\Deltaper mini\-batch, \(2\) inspect the empirical histogram ofΔ\\Delta, \(3\) setτlow\\tau\_\{\\text\{low\}\}near the lower\-middle mass of the histogram \(about the 40th percentile\) andτhigh\\tau\_\{\\text\{high\}\}near the clear\-dominance region \(about the 80th percentile\), while keeping a gap of at least0\.050\.05, and \(4\) keepλbias∈\[0\.1,0\.3\]\\lambda\_\{\\text\{bias\}\}\\in\[0\.1,0\.3\]unless validation indicates otherwise\.

Runtime Analysis\.CAT\-GS introduces minimal computational overhead compared to standard joint training\. The additional operations—teacher inference \(which can be precomputed or run in parallel\), reliability calibration, and gradient rescaling—are lightweight vector operations\. The fusion\-layer surgery involves only a single\-sided projection step on the fusion output layer \(e\.g\.,fc\_out\) using the logit\-sum gradients from Eq\. \([14](https://arxiv.org/html/2608.24947#S3.E14)\), avoiding the high cost of full\-network gradient projection\. Empirically, we observe a per\-epoch training time increase of approximately 2–5% across our benchmarks, which is negligible given the performance gains and improved convergence stability\.

### 4\.4Discussion

CAT\-GS is most beneficial when unimodal reliability varies across training and the fused head is a meaningful shared bottleneck where cross\-modal gradients can interfere\. On large\-scale in\-the\-wild datasets such as VGGSound, CAT\-GS yields comparable rather than dominant improvements, and in fact does not surpass the strongest baselines \(G2D, UMT\) there, suggesting that representation capacity and data scale may outweigh optimization control when the teacher\-reliability signal is weak and the label space is large\. Moreover, CAT\-GS inherits any systematic bias in the unimodal teachers: if a teacher is consistently miscalibrated, the controller may over\- or under\-prioritize a modality\. Our temperature\-mismatch stress test \(Table[7](https://arxiv.org/html/2608.24947#S4.T7)\) indicates the method degrades gracefully under moderate miscalibration, and that EMA smoothing and calibration materially improve robustness\.Under stronger reliability degradation stress settings, the method remains stable but shows lower final accuracy, reinforcing that teacher quality is an important practical factor\.At the same time, teacher guidance alone does not explain the gains: under matched training settings on CREMA\-D, Joint\-Train \(CAT\-GS disabled\) reaches 67\.47%, the no\-teacher standard\-training reference reaches 64\.78%, and CAT\-GS reaches 86\.29%±\\pm0\.15 \(Table[1](https://arxiv.org/html/2608.24947#S3.T1)\)\.In practice, we recommend CAT\-GS when unimodal teachers are reasonably strong and when training dynamics exhibit either gate thrashing, prolonged starvation, or frequent negative fusion\-gradient cosine events\.CAT\-GS is implemented at the optimization stage, so extension across fusion paradigms is operationally straightforward\. In our additional architecture checks, CAT\-GS remained stable for Early Fusion, Late Fusion, and a transformer\-style token\-fusion head \(Table[6](https://arxiv.org/html/2608.24947#S4.T6)\)\. Together, these results indicate transferability across fusion paradigms, with expected architecture\-dependent accuracy differences\. For token\-level transformer architectures, reliability signals can be computed from modality\-specific token branches, with gating/budget stabilization on modality\-specific adapters or blocks and conflict handling on shared/cross\-attention parameters\.

Limitations\.CAT\-GS relies on pre\-trained unimodal teachers to provide reliability signals\. If teachers are weak or systematically biased, the controller may inherit these failure modes\. The teacher\-miscalibration stress test \(Table[7](https://arxiv.org/html/2608.24947#S4.T7)\) is the closest evidence available: temperature mismatch perturbs the teacher signal \(R1–R3\) with only gradual accuracy loss, and the R4–R5 controls show EMA smoothing and calibration buffer it\. Since this perturbs teacher*calibration*rather than*accuracy*, a genuinely weak or domain\-shifted teacher remains untested and is left to future work\.The main remaining limitation is dependence on teacher reliability; fully teacher\-free CAT\-GS control remains an important direction for future work\.Future work should investigate teacher\-free reliability estimation and stronger domain\-shift robustness so that control quality is less dependent on teacher calibration\.Extending to many\-modality or fully entangled token\-level settings mainly requires careful definition of modality\-linked parameter groups so that gating/budget operations remain well\-posed while shared\-layer conflict handling remains localized\.From a practical deployment perspective, CAT\-GS introduces several hyperparameters \(e\.g\.,τlow\\tau\_\{\\text\{low\}\},τhigh\\tau\_\{\\text\{high\}\},λbias\\lambda\_\{\\text\{bias\}\},β\\beta,βg\\beta\_\{g\},γcap\\gamma\_\{\\text\{cap\}\},ε\\varepsilon\)\. However, most can be fixed to stable defaults across datasets \(we useβ=0\.9\\beta\{=\}0\.9,βg=0\.9\\beta\_\{g\}\{=\}0\.9,γcap=1\.5\\gamma\_\{\\text\{cap\}\}\{=\}1\.5, andε=0\.1\\varepsilon\{=\}0\.1throughout\), while the primary tuning knobs are the regime thresholds and the weak\-bias factor\. Our sensitivity analysis indicates broad plateaus where performance is stable \(Section[4\.3](https://arxiv.org/html/2608.24947#S4.SS3)\), suggesting limited tuning effort in practice\. A simple guideline is to setτlow\\tau\_\{\\text\{low\}\}andτhigh\\tau\_\{\\text\{high\}\}to separate “close” vs\. “clear\-dominance” regions of the observed reliability\-margin distribution, and then chooseλbias∈\[0\.1,0\.3\]\\lambda\_\{\\text\{bias\}\}\\in\[0\.1,0\.3\]to softly favor the weaker modality when reliabilities are comparable\.

## 5Conclusion

We introduced CAT\-GS, an optimization\-stage learning\-dynamics controller for balanced and robust multimodal learning\. Instead of modifying architectures or designing bespoke losses, CAT\-GS treats multimodal training as a regime\-switching control problem and enforces stability constraints on gradient flow, update magnitude, and fusion\-gradient geometry\.Across audio–visual benchmarks \(CREMA\-D, AV\-MNIST\), additional benchmarks \(AVE and CMU\-MOSI\), tri\-modal humor understanding \(UR\-FUNNY\), and a controlled synthetic dataset \(CG\-MNIST\), CAT\-GS improves multimodal accuracy and training stability over strong imbalance\-aware baselines while maintaining competitive unimodal performance\. The additional AVE/CMU\-MOSI comparisons further indicate that CAT\-GS transfers beyond the original dataset family to event\-centric and sentiment\-centric settings\.On the large\-scale in\-the\-wild VGGSound benchmark, however, CAT\-GS remains competitive but does not surpass the strongest baselines \(G2D, UMT\), a limitation we attribute to the weak, noisy teacher\-reliability signal and the large label space at this scale\. Scaling optimization\-stage control to such data, for instance through stronger or teacher\-free reliability estimation, is an explicit direction for future work\.Ablations show that calibration, gating, budget reallocation, and fusion surgery contribute complementary gains, and CAT\-GS remains effective across fusion architectures under a consistent late\-fusion\-centered evaluation with transfer checks to alternative fusion heads\. Future work includes scaling to more modalities, integrating token\-level/temporal gating in transformer\-based models, and studying interactions with large\-scale pretraining\.

## 6Code Availabality

## References

- J\. Arevalo, T\. Solorio, M\. Montes\-y\-Gomez, and F\. A\. GonzálezGated multimodal networks\.Neural Computing and Applications32\(14\),pp\. 10209–10228\.Cited by:[§2](https://arxiv.org/html/2608.24947#S2.p3.1)\.
- Atreyet al\.\(2010\)P\. K\. Atrey, M\. A\. Hossain, A\. El Saddik, and M\. S\. KankanhalliMultimodal fusion for multimedia analysis: a survey\.Multimedia systems16\(6\),pp\. 345–379\.Cited by:[§4\.3](https://arxiv.org/html/2608.24947#S4.SS3.p4.1.2)\.
- Caoet al\.\(2014\)H\. Cao, D\. G\. Cooper, M\. K\. Keutmann, R\. C\. Gur, A\. Nenkova, and R\. VermaCREMA\-D: crowd\-sourced emotional multimodal actors dataset\.IEEE transactions on affective computing5\(4\),pp\. 377–390\.Cited by:[§4\.1](https://arxiv.org/html/2608.24947#S4.SS1.p1.1)\.
- Chenet al\.\(2021\)C\. R\. Chen, Q\. Fan, and R\. PandaCrossViT: cross\-attention multi\-scale vision transformer for image classification\.In2021 IEEE/CVF International Conference on Computer Vision \(ICCV\),Vol\.,pp\. 347–356\.Cited by:[§4\.3](https://arxiv.org/html/2608.24947#S4.SS3.p4.1.2),[Table 6](https://arxiv.org/html/2608.24947#S4.T6.6.1.6.1)\.
- Chenet al\.\(2020\)H\. Chen, W\. Xie, A\. Vedaldi, and A\. ZissermanVGGSound: a large\-scale audio\-visual dataset\.InICASSP 2020 \- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),Vol\.,pp\. 721–725\.Cited by:[§4\.1](https://arxiv.org/html/2608.24947#S4.SS1.p1.1)\.
- Duet al\.\(2023\)C\. Du, J\. Teng, T\. Li, Y\. Liu, T\. Yuan, Y\. Wang, Y\. Yuan, and H\. ZhaoOn uni\-modal feature learning in supervised multi\-modal learning\.InProceedings of the 40th International Conference on Machine Learning,ICML’23\.Cited by:[§2](https://arxiv.org/html/2608.24947#S2.p3.1),[§2](https://arxiv.org/html/2608.24947#S2.p5.1),[Table 1](https://arxiv.org/html/2608.24947#S3.T1.6.1.1.14.1.2.1),[§4\.1](https://arxiv.org/html/2608.24947#S4.SS1.p2.1),[§4\.2](https://arxiv.org/html/2608.24947#S4.SS2.p1.1),[Table 2](https://arxiv.org/html/2608.24947#S4.T2.7.1.1.6.1.2.1)\.
- Fanet al\.\(2023\)Y\. Fan, W\. Xu, H\. Wang, J\. Wang, and S\. GuoPMR: prototypical modal rebalance for multimodal learning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 20029–20038\.Cited by:[Table 1](https://arxiv.org/html/2608.24947#S3.T1.6.1.1.8.1.2.1),[§4\.1](https://arxiv.org/html/2608.24947#S4.SS1.p2.1),[§4\.2](https://arxiv.org/html/2608.24947#S4.SS2.p1.1),[Table 4](https://arxiv.org/html/2608.24947#S4.T4.5.1.1.2)\.
- Fujimoriet al\.\(2020\)N\. Fujimori, R\. Endo, Y\. Kawai, and T\. MochizukiModality\-specific learning rate control for multimodal classification\.InPattern Recognition,S\. Palaiahnakote, G\. Sanniti di Baja, L\. Wang, and W\. Q\. Yan \(Eds\.\),Cham,pp\. 412–422\.Cited by:[Table 1](https://arxiv.org/html/2608.24947#S3.T1.6.1.1.5.1.2.1),[§4\.1](https://arxiv.org/html/2608.24947#S4.SS1.p2.1),[Table 4](https://arxiv.org/html/2608.24947#S4.T4.5.1.1.4)\.
- Gunes and Piccardi \(2005\)H\. Gunes and M\. PiccardiAffect recognition from face and body: early fusion vs\. late fusion\.In2005 IEEE International Conference on Systems, Man and Cybernetics,Vol\.4,pp\. 3437–3443 Vol\. 4\.Cited by:[§4\.3](https://arxiv.org/html/2608.24947#S4.SS3.p4.1.2),[Table 6](https://arxiv.org/html/2608.24947#S4.T6.6.1.8.1.1),[Table 6](https://arxiv.org/html/2608.24947#S4.T6.6.1.9.1.1)\.
- Guoet al\.\(2017\)C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. WeinbergerOn calibration of modern neural networks\.InProceedings of the 34th International Conference on Machine Learning \- Volume 70,ICML’17,pp\. 1321–1330\.Cited by:[§1](https://arxiv.org/html/2608.24947#S1.p2.1),[§2](https://arxiv.org/html/2608.24947#S2.p3.1),[§2](https://arxiv.org/html/2608.24947#S2.p5.1)\.
- Hanet al\.\(2022\)Z\. Han, F\. Yang, J\. Huang, C\. Zhang, and J\. YaoMultimodal dynamics: dynamical fusion for trustworthy multimodal classification\.In2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 20675–20685\.Cited by:[§1](https://arxiv.org/html/2608.24947#S1.p2.1)\.
- Hasanet al\.\(2019\)M\. K\. Hasan, W\. Rahman, A\. Bagher Zadeh, J\. Zhong, M\. I\. Tanveer, L\. Morency, and M\. \(\. HoqueUR\-FUNNY: a multimodal language dataset for understanding humor\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),Hong Kong, China,pp\. 2046–2056\.Cited by:[§4\.1](https://arxiv.org/html/2608.24947#S4.SS1.p1.1)\.
- Heet al\.\(2016\)K\. He, X\. Zhang, S\. Ren, and J\. SunDeep Residual Learning for Image Recognition\.In2016 IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),Vol\.,Los Alamitos, CA, USA,pp\. 770–778\.External Links:ISSN 1063\-6919Cited by:[§4\.1](https://arxiv.org/html/2608.24947#S4.SS1.p3.1)\.
- Hintonet al\.\(2015\)G\. Hinton, O\. Vinyals, and J\. DeanDistilling the knowledge in a neural network\.pp\.\.Cited by:[§2](https://arxiv.org/html/2608.24947#S2.p5.1)\.
- Huaet al\.\(2024\)C\. Hua, Q\. Xu, S\. Bao, Z\. Yang, and Q\. HuangReconBoost: boosting can achieve modality reconcilement\.InProceedings of the 41st International Conference on Machine Learning,ICML’24\.Cited by:[Table 1](https://arxiv.org/html/2608.24947#S3.T1.6.1.1.11.1.2.1),[§4\.1](https://arxiv.org/html/2608.24947#S4.SS1.p2.1),[§4\.2](https://arxiv.org/html/2608.24947#S4.SS2.p6.1),[Table 2](https://arxiv.org/html/2608.24947#S4.T2.7.1.1.5.1.2.1),[Table 3](https://arxiv.org/html/2608.24947#S4.T3.7.1.6.1)\.
- Kielaet al\.\(2018\)D\. Kiela, E\. Grave, A\. Joulin, and T\. MikolovEfficient large\-scale multi\-modal classification\.AAAI’18/IAAI’18/EAAI’18\.External Links:ISBN 978\-1\-57735\-800\-8Cited by:[§4\.3](https://arxiv.org/html/2608.24947#S4.SS3.p4.1.2),[Table 6](https://arxiv.org/html/2608.24947#S4.T6.6.1.5.1)\.
- Kimet al\.\(2019\)B\. Kim, H\. Kim, K\. Kim, S\. Kim, and J\. KimLearning Not to Learn: Training Deep Neural Networks With Biased Data\.In2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Vol\.,Los Alamitos, CA, USA,pp\. 9004–9012\.External Links:ISSNCited by:[§4\.1](https://arxiv.org/html/2608.24947#S4.SS1.p1.1)\.
- Kontraset al\.\(2024\)K\. Kontras, C\. Chatzichristos, M\. B\. Blaschko, and M\. D\. VosImproving multimodal learning with multi\-loss gradient modulation\.In35th British Machine Vision Conference 2024, BMVC 2024, Glasgow, UK, November 25\-28, 2024,Cited by:[§2](https://arxiv.org/html/2608.24947#S2.p1.1)\.
- Kontraset al\.\(2025\)K\. Kontras, T\. Strypsteen, C\. Chatzichristos, P\. P\. Liang, M\. B\. Blaschko, and M\. D\. VosBalancing multimodal training through game\-theoretic regularization\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2608.24947#S2.p1.1.2),[Table 3](https://arxiv.org/html/2608.24947#S4.T3.7.1.8.1)\.
- Liet al\.\(2023\)H\. Li, X\. Li, P\. Hu, Y\. Lei, C\. Li, and Y\. ZhouBoosting multi\-modal model performance with adaptive gradient modulation\.InProceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\),pp\. 22214–22224\.Cited by:[Table 1](https://arxiv.org/html/2608.24947#S3.T1.6.1.1.7.1.2.1),[§4\.1](https://arxiv.org/html/2608.24947#S4.SS1.p2.1),[§4\.2](https://arxiv.org/html/2608.24947#S4.SS2.p1.1),[Table 3](https://arxiv.org/html/2608.24947#S4.T3.7.1.5.1)\.
- Penget al\.\(2022\)X\. Peng, Y\. Wei, A\. Deng, D\. Wang, and D\. HuBalanced multimodal learning via on\-the\-fly gradient modulation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§1](https://arxiv.org/html/2608.24947#S1.p1.1),[§1](https://arxiv.org/html/2608.24947#S1.p2.1),[§2](https://arxiv.org/html/2608.24947#S2.p1.1),[Table 1](https://arxiv.org/html/2608.24947#S3.T1.6.1.1.9.1.2.1),[§4\.1](https://arxiv.org/html/2608.24947#S4.SS1.p2.1),[§4\.2](https://arxiv.org/html/2608.24947#S4.SS2.p1.1),[Table 2](https://arxiv.org/html/2608.24947#S4.T2.7.1.1.3.1.2.1),[Table 3](https://arxiv.org/html/2608.24947#S4.T3.7.1.4.1),[Table 4](https://arxiv.org/html/2608.24947#S4.T4.5.1.1.3)\.
- Perezet al\.\(2018\)E\. Perez, F\. Strub, H\. de Vries, V\. Dumoulin, and A\. CourvilleFiLM: visual reasoning with a general conditioning layer\.AAAI’18/IAAI’18/EAAI’18\.External Links:ISBN 978\-1\-57735\-800\-8Cited by:[§4\.3](https://arxiv.org/html/2608.24947#S4.SS3.p4.1.2),[Table 6](https://arxiv.org/html/2608.24947#S4.T6.6.1.4.1)\.
- Pezeshkiet al\.\(2021\)M\. Pezeshki, S\. Kaba, Y\. Bengio, A\. Courville, D\. Precup, and G\. LajoieGradient starvation: a learning proclivity in neural networks\.InProceedings of the 35th International Conference on Neural Information Processing Systems,NIPS ’21,Red Hook, NY, USA\.External Links:ISBN 9781713845393Cited by:[§2](https://arxiv.org/html/2608.24947#S2.p1.1)\.
- Poriaet al\.\(2016\)S\. Poria, E\. Cambria, N\. Howard, G\. Huang, and A\. HussainFusing audio, visual and textual clues for sentiment analysis from multimodal content\.Neurocomputing174,pp\. 50–59\.Cited by:[§2](https://arxiv.org/html/2608.24947#S2.p3.1)\.
- Rakib and Bagavathi \(2025\)M\. Rakib and A\. BagavathiG2\{\}^\{2\}D: boosting multimodal learning with gradient\-guided distillation\.InProceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\),pp\. 4059–4068\.Cited by:[§1](https://arxiv.org/html/2608.24947#S1.p2.1),[Table 1](https://arxiv.org/html/2608.24947#S3.T1.6.1.1.15.1.2.1),[Table 2](https://arxiv.org/html/2608.24947#S4.T2.7.1.1.7.1.2.1),[Table 4](https://arxiv.org/html/2608.24947#S4.T4.5.1.1.6)\.
- Tianet al\.\(2018\)Y\. Tian, J\. Shi, B\. Li, Z\. Duan, and C\. XuAudio\-visual event localization in unconstrained videos\.InECCV,Cited by:[§4\.1](https://arxiv.org/html/2608.24947#S4.SS1.p1.1.7.1)\.
- Tsaiet al\.\(2019\)Y\. H\. Tsai, S\. Bai, P\. P\. Liang, J\. Z\. Kolter, L\. Morency, and R\. SalakhutdinovMultimodal transformer for unaligned multimodal language sequences\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 6558–6569\.Cited by:[§2](https://arxiv.org/html/2608.24947#S2.p3.1),[§4\.3](https://arxiv.org/html/2608.24947#S4.SS3.p5.1.1),[Table 6](https://arxiv.org/html/2608.24947#S4.T6.6.1.7.1.1)\.
- Vaswaniet al\.\(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. PolosukhinAttention is all you need\.InAdvances in Neural Information Processing Systems,Vol\.30,pp\.\.Cited by:[§4\.1](https://arxiv.org/html/2608.24947#S4.SS1.p3.1)\.
- Vielzeufet al\.\(2019\)V\. Vielzeuf, A\. Lechervy, S\. Pateux, and F\. JurieCentralNet: a multilayer approach for multimodal fusion\.InComputer Vision – ECCV 2018 Workshops: Munich, Germany, September 8\-14, 2018, Proceedings, Part VI,Berlin, Heidelberg,pp\. 575–589\.External Links:ISBN 978\-3\-030\-11023\-9Cited by:[§4\.1](https://arxiv.org/html/2608.24947#S4.SS1.p1.1)\.
- Weiet al\.\(2025\)S\. Wei, C\. Luo, and Y\. LuoBoosting multimodal learning via disentangled gradient learning\.In2025 IEEE/CVF International Conference on Computer Vision \(ICCV\),pp\. 1–10\.Cited by:[§2](https://arxiv.org/html/2608.24947#S2.p4.1)\.
- Wei and Hu \(2024\)Y\. Wei and D\. HuMMPareto: boosting multimodal learning with innocent unimodal assistance\.ICML’24\.Cited by:[Table 1](https://arxiv.org/html/2608.24947#S3.T1.6.1.1.12.1.2.1),[§4\.1](https://arxiv.org/html/2608.24947#S4.SS1.p2.1),[§4\.2](https://arxiv.org/html/2608.24947#S4.SS2.p6.1),[Table 2](https://arxiv.org/html/2608.24947#S4.T2.7.1.1.4.1.2.1),[Table 3](https://arxiv.org/html/2608.24947#S4.T3.7.1.7.1)\.
- Weiet al\.\(2024\)Y\. Wei, S\. Li, R\. Feng, and D\. HuDiagnosing and re\-learning for balanced multimodal learning\.pp\. 71–86\.External Links:ISBN 978\-3\-031\-73038\-2Cited by:[§2](https://arxiv.org/html/2608.24947#S2.p1.1.2),[Table 3](https://arxiv.org/html/2608.24947#S4.T3.7.1.9.1)\.
- Wuet al\.\(2018\)C\. Wu, Y\. Wei, X\. Chu, W\. Sun, F\. Su, and L\. WangHierarchical attention\-based multimodal fusion for video captioning\.Neurocomputing315,pp\. 362–370\.External Links:ISSN 0925\-2312Cited by:[§2](https://arxiv.org/html/2608.24947#S2.p3.1)\.
- Xuet al\.\(2023a\)P\. Xu, X\. Zhu, and D\. A\. CliftonMultimodal learning with transformers: a survey\.IEEE Transactions on Pattern Analysis and Machine Intelligence45\(10\),pp\. 12113–12132\.Cited by:[§1](https://arxiv.org/html/2608.24947#S1.p1.1)\.
- Xuet al\.\(2023b\)R\. Xu, R\. Feng, S\. Zhang, and D\. HuMMCosine: multi\-modal cosine loss towards balanced audio\-visual fine\-grained learning\.InICASSP 2023 \- 2023 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),Vol\.,pp\. 1–5\.Cited by:[§2](https://arxiv.org/html/2608.24947#S2.p1.1.2),[Table 3](https://arxiv.org/html/2608.24947#S4.T3.7.1.3.1)\.
- Yanget al\.\(2024\)Y\. Yang, F\. Wan, Q\. Jiang, and Y\. XuFacilitating multimodal classification via dynamically learning modality gap\.InProceedings of the 38th International Conference on Neural Information Processing Systems,NIPS ’24,Red Hook, NY, USA\.External Links:ISBN 9798331314385Cited by:[Table 1](https://arxiv.org/html/2608.24947#S3.T1.6.1.1.13.1.2.1),[§4\.1](https://arxiv.org/html/2608.24947#S4.SS1.p2.1)\.
- Yao and Mihalcea \(2022\)Y\. Yao and R\. MihalceaModality\-specific learning rates for effective multimodal additive late\-fusion\.InFindings of the Association for Computational Linguistics: ACL 2022,S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 1824–1834\.Cited by:[§1](https://arxiv.org/html/2608.24947#S1.p2.1),[Table 1](https://arxiv.org/html/2608.24947#S3.T1.6.1.1.6.1.2.1),[§4\.1](https://arxiv.org/html/2608.24947#S4.SS1.p2.1),[Table 3](https://arxiv.org/html/2608.24947#S4.T3.7.1.2.1),[Table 4](https://arxiv.org/html/2608.24947#S4.T4.5.1.1.5)\.
- Yuet al\.\(2020\)T\. Yu, S\. Kumar, A\. Gupta, S\. Levine, K\. Hausman, and C\. FinnGradient surgery for multi\-task learning\.InAdvances in Neural Information Processing Systems,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\.F\. Balcan, and H\. Lin \(Eds\.\),Vol\.33,pp\. 5824–5836\.Cited by:[§1](https://arxiv.org/html/2608.24947#S1.p2.1),[§3\.6](https://arxiv.org/html/2608.24947#S3.SS6.p1.1)\.
- Zadehet al\.\(2016\)A\. Zadeh, R\. Zellers, E\. Pincus, and L\. MorencyMOSI: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos\.arXiv preprint arXiv:1606\.06259\.Cited by:[§4\.1](https://arxiv.org/html/2608.24947#S4.SS1.p1.1.8.1)\.
- Zhanget al\.\(2024\)X\. Zhang, J\. Yoon, M\. Bansal, and H\. YaoMultimodal representation learning by alternating unimodal adaptation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[Table 1](https://arxiv.org/html/2608.24947#S3.T1.6.1.1.10.1.2.1),[§4\.1](https://arxiv.org/html/2608.24947#S4.SS1.p2.1)\.