SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning

# SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning
Source: [https://arxiv.org/html/2606.09853](https://arxiv.org/html/2606.09853)
SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal LearningKonstantinos Kontras1,2, Teodora Gagaleska1, Thomas Strypsteen1, Christos Chatzichristos1, Matthew Blaschko1, Maarten De Vos1,†, Paul Pu Liang2,†1KU Leuven,2MIT,†Equal supervisionA central objective in multimodal learning is to capture synergy: task\-relevant information that arises only from the joint use of multiple modalities, and is not available from any single modality alone\. While most approaches operate at the architectural level through larger or more complex fusion models, we propose a complementary axis: shaping the training objective itself\. Standard training often emphasizes unimodal or redundant information, falling short on examples that require cross\-modal reasoning\. We formalize multimodal synergy through information theory and introduce the Synergistic Information Bottleneck \(SynIB\), a scalable objective that targets synergy directly\. To prioritize learning synergy, SynIB motivates the model to predict accurately from all modalities while penalizing confidence when information from any modality is withheld\. Alongside the standard task loss, the model runs forward passes with one modality masked at a time and is penalized for remaining confident, which would indicate reliance on unimodal cues rather than cross\-modal interactions\. We validate SynIB in two regimes\. On synthetic XOR tasks where the ground\-truth synergy is known by construction, standard training fails to recover it while SynIB does\. On five real\-world benchmarks, including three MultiBench affective tasks, Hateful Memes with CLIP\-ViT and DeBERTa backbones, and a controllable irony extension of CREMA\-D we introduce, SynIB improves accuracy on synergy\-dependent examples by up to 7\.8% and overall accuracy by up to 3\.8%\.Correspondence:[kkontras@mit\.edu](https://arxiv.org/html/2606.09853v1/mailto:[email protected])![[Uncaptioned image]](https://arxiv.org/html/2606.09853v1/Figures/logos/mit_lockup_std-three-line_rgb_black.png)![[Uncaptioned image]](https://arxiv.org/html/2606.09853v1/Figures/logos/KU-Leuven-logo.png)

## 1Introduction

Multimodal learning aims to combine information from multiple sources to improve prediction\. A central challenge is capturing synergistic information: predictive signal that arises from interactions between modalities rather than from any modality alone\(Lianget al\.,[2024b](https://arxiv.org/html/2606.09853#bib.bib3)\)\. In unimodal learning, nonlinear activations enable feature interactions and techniques such as weight decay\(Krogh and Hertz,[1991](https://arxiv.org/html/2606.09853#bib.bib108)\)and dropout\(Srivastavaet al\.,[2014](https://arxiv.org/html/2606.09853#bib.bib85)\)reduce reliance on individual features\(Goodfellowet al\.,[2016](https://arxiv.org/html/2606.09853#bib.bib109)\); analogous mechanisms for cross\-modal interactions remain underexplored\.

Multimodal models sometimes over\-rely on a single modality during training, favoring signals that yield faster optimization progress, a phenomenon known as multimodal competition\(Huanget al\.,[2022](https://arxiv.org/html/2606.09853#bib.bib19)\)\. Numerous methods address this by estimating modality contributions and rebalancing learning through gradient modulation, auxiliary losses, or related interventions, but yield limited gains on synergistic cases\(Kontraset al\.,[2025](https://arxiv.org/html/2606.09853#bib.bib110)\), indicating that correcting modality imbalance alone is insufficient to induce cross\-modal synergy\. A complementary line captures cross\-modal interactions through architectural design, from fusion mechanisms\(Zadehet al\.,[2017](https://arxiv.org/html/2606.09853#bib.bib63); Tsaiet al\.,[2019](https://arxiv.org/html/2606.09853#bib.bib62)\)to masked multimodal pretraining\(Singhet al\.,[2022](https://arxiv.org/html/2606.09853#bib.bib131)\), expanding capacity for interaction without directly targeting synergistic prediction\. Representation\-level objectives such as contrastive alignment\(Radfordet al\.,[2021](https://arxiv.org/html/2606.09853#bib.bib61)\)do shape training, but pull modalities toward agreement, a property closer to redundancy than to synergy\. What is missing across these lines is a training signal that directly prioritizes synergistic prediction on the downstream task\. Our approach fills this gap: it targets synergy at the loss level, is orthogonal to architectural and pretraining choices, and is complementary to rebalancing methods, which address a related but distinct failure mode\.

To this end, we formalize multimodal synergy through an information\-theoretic lens and introduce the*Synergistic Information Bottleneck*\(SynIB\)\. The idea follows from the definition: synergistic information is the information that vanishes when any single modality is removed\. A model that relies on synergy must therefore become uncertain when one modality is masked; a model that remains confident is relying on unimodal or redundant cues\. SynIB turns this observation into a training signal by penalizing confident predictions under modality\-wise masking, pressuring the model toward cross\-modal dependencies on the examples where they matter most\. Our main contributions are:

1. 1\.SynIB, a training objective that targets synergy by penalizing confident predictions under modality\-wise masking, requiring no prior knowledge of whether synergy exists in the data\.
2. 2\.A motivating analysis showing that the failure to learn synergy is not caused by interference from unimodal signals; rather, synergistic cues are learned but quickly overfit, limiting generalization\.
3. 3\.Extensive evaluation across real\-world benchmarks from MultiBench and HatefulMemes, including pretrained CLIP\-ViT and DeBERTa backbones, a new irony extension of CREMA\-D, and controlled Gaussian XOR tasks\. On these SynIB improves synergy\-dependent test set accuracy by up to 7\.8% and overall accuracy by up to 3\.8%\.

## 2Related Work

##### Partial Information Decomposition \(PID\)\.

PID formalizes synergy, redundancy, and unique information across multiple sources with respect to a target\(Williams and Beer,[2010](https://arxiv.org/html/2606.09853#bib.bib111); Griffith and Koch,[2014](https://arxiv.org/html/2606.09853#bib.bib112); Bertschingeret al\.,[2014](https://arxiv.org/html/2606.09853#bib.bib113)\)\. In multimodal learning, it has primarily served as a post\-hoc analytical tool to quantify information distribution in trained models\(Lianget al\.,[2023](https://arxiv.org/html/2606.09853#bib.bib114)\), where synergy is operationalized as the predictive gap between a fusion model and an ensemble of unimodal models\. Recent work incorporates PID\-inspired quantities into training objectives via contrastive factorization\(Lianget al\.,[2024a](https://arxiv.org/html/2606.09853#bib.bib83); Wenet al\.,[2025](https://arxiv.org/html/2606.09853#bib.bib119)\)and interaction\-aware architectures\(Xinet al\.,[2025](https://arxiv.org/html/2606.09853#bib.bib120)\)\. While effective for disentanglement, attribution, and interpretability, these approaches treat synergy as a structural property to be isolated rather than as a learning optimization signal\.

##### Multimodal representation learning\.

A broad field of works captures cross\-modal interactions through architectural and representational design: fusion mechanisms from tensor fusion\(Zadehet al\.,[2017](https://arxiv.org/html/2606.09853#bib.bib63)\)to cross\-attention transformers\(Tsaiet al\.,[2019](https://arxiv.org/html/2606.09853#bib.bib62); Luet al\.,[2019](https://arxiv.org/html/2606.09853#bib.bib133)\), alignment objectives such as multimodal contrastive learning\(Radfordet al\.,[2021](https://arxiv.org/html/2606.09853#bib.bib61)\), masked multimodal pretraining\(Singhet al\.,[2022](https://arxiv.org/html/2606.09853#bib.bib131); Baevskiet al\.,[2022](https://arxiv.org/html/2606.09853#bib.bib132)\), and factorized representations into modality\-invariant and modality\-specific subspaces\(Lianget al\.,[2024a](https://arxiv.org/html/2606.09853#bib.bib83)\)\. These approaches expand the capacity for cross\-modal interaction but none directly shape the training signal on the downstream task to penalize models that ignore synergistic cues, leaving even expressive architectures free to settle into unimodal or redundant solutions when those minimize loss faster\(Huanget al\.,[2022](https://arxiv.org/html/2606.09853#bib.bib19)\)\. Our objective is complementary: it operates at the loss level and can be combined with any of these architectural choices\.

##### Multimodal competition\.

A complementary line of work addresses cross\-modal learning failures as multimodal competition, estimating modality contributions and rebalancing learning accordingly\. Strategies include measuring contribution via unimodal performance\(Yao and Mihalcea,[2022](https://arxiv.org/html/2606.09853#bib.bib28); Penget al\.,[2022](https://arxiv.org/html/2606.09853#bib.bib12); Vielzeufet al\.,[2018](https://arxiv.org/html/2606.09853#bib.bib30); Kontraset al\.,[2024](https://arxiv.org/html/2606.09853#bib.bib67); Fanet al\.,[2023](https://arxiv.org/html/2606.09853#bib.bib13)\)and enforcing balance through gradient modulation or decomposition\(Wei and Hu,[2024](https://arxiv.org/html/2606.09853#bib.bib103)\), reinitializing modality\-specific components to escape suboptimal trajectories\(Weiet al\.,[2024](https://arxiv.org/html/2606.09853#bib.bib105)\), perturbation\-based attribution such as Shapley approximations\(Liet al\.,[2023](https://arxiv.org/html/2606.09853#bib.bib25)\), permutation importance\(Kontraset al\.,[2025](https://arxiv.org/html/2606.09853#bib.bib110)\), counterfactual modality removal\(Jiet al\.,[2022](https://arxiv.org/html/2606.09853#bib.bib52)\), and ensemble\-based strategies that sidestep fusion\(Huaet al\.,[2024](https://arxiv.org/html/2606.09853#bib.bib104)\)\. While effective at preventing modality collapse, these methods assume balanced contributions suffice for multimodal learning\. As MCR\(Kontraset al\.,[2025](https://arxiv.org/html/2606.09853#bib.bib110)\)shows, however, correcting modality imbalance does not by itself induce synergy\.

## 3Method

##### Problem setup\.

We consider a supervised multimodal prediction task with two input modalitiesX1X\_\{1\}andX2X\_\{2\}and a targetYY:\(X1∈ℝd1,X2∈ℝd2,Y∈𝒴\)∼p​\(x1,x2,y\)\(X\_\{1\}\\in\\mathbb\{R\}^\{d\_\{1\}\},\\,X\_\{2\}\\in\\mathbb\{R\}^\{d\_\{2\}\},\\,Y\\in\\mathcal\{Y\}\)\\sim p\(x\_\{1\},x\_\{2\},y\)\.111We present the formulation for two modalities for clarity; the SynIB objective is symmetric across modalities and extends ton\>2n\>2settings by adding counterfactual passes on the subset of modalities\.Modality\-specific encodersfθi:ℝdi→ℝmif\_\{\\theta\_\{i\}\}:\\mathbb\{R\}^\{d\_\{i\}\}\\to\\mathbb\{R\}^\{m\_\{i\}\}produceZi=fθi​\(Xi\)Z\_\{i\}=f\_\{\\theta\_\{i\}\}\(X\_\{i\}\)fori∈\{1,2\}i\\in\\\{1,2\\\}, a fusion networkfθ12:ℝm1×ℝm2→Δ​\(𝒴\)f\_\{\\theta\_\{12\}\}:\\mathbb\{R\}^\{m\_\{1\}\}\\times\\mathbb\{R\}^\{m\_\{2\}\}\\to\\Delta\(\\mathcal\{Y\}\)it’s multimodal outputsY^12=fθc12​\(Z1,Z2\)\\hat\{Y\}\_\{12\}=f\_\{\\theta\_\{c\_\{12\}\}\}\(Z\_\{1\},Z\_\{2\}\), and the unimodal encodersfθci:ℝmi×ℝf\_\{\\theta\_\{c\_\{i\}\}\}:\\mathbb\{R\}^\{m\_\{i\}\}\\times\\mathbb\{R\}the unimodal outputsY^i=fθci​\(Zi\)\\hat\{Y\}\_\{i\}=f\_\{\\theta\_\{c\_\{i\}\}\}\(Z\_\{i\}\)\. We collect all trainable parameters asθ=\{θ1,θ2​θc1,θc2,θc12\}\\theta=\\\{\\theta\_\{1\},\\theta\_\{2\}\\,\\theta\_\{c\_\{1\}\},\\theta\_\{c\_\{2\}\},\\theta\_\{c\_\{12\}\}\\\}\.

##### Synergy as a learning goal\.

Partial Information Decomposition\(Williams and Beer,[2010](https://arxiv.org/html/2606.09853#bib.bib111); Bertschingeret al\.,[2014](https://arxiv.org/html/2606.09853#bib.bib113)\)decomposes the predictive mutual information into four atoms:

I​\(Y;X1,X2\)=R​\(Y;X1,X2\)\+U1​\(Y;X1∣X2\)\+U2​\(Y;X2∣X1\)\+S​\(Y;X1,X2\),I\(Y;X\_\{1\},X\_\{2\}\)\\;=\\;R\(Y;X\_\{1\},X\_\{2\}\)\\;\+\\;U\_\{1\}\(Y;X\_\{1\}\\mid X\_\{2\}\)\\;\+\\;U\_\{2\}\(Y;X\_\{2\}\\mid X\_\{1\}\)\\;\+\\;S\(Y;X\_\{1\},X\_\{2\}\),\(1\)whereRRis redundant across modalities,UiU\_\{i\}is unique toXiX\_\{i\}, andSSis accessible only from the joint\(X1,X2\)\(X\_\{1\},X\_\{2\}\)\. FollowingBertschingeret al\.\([2014](https://arxiv.org/html/2606.09853#bib.bib113)\), synergy is the gap betweenIp​\(Y;X1,X2\)I\_\{p\}\(Y;X\_\{1\},X\_\{2\}\)and its minimum over distributions preserving the bivariate marginals\(Y,X1\)\(Y,X\_\{1\}\)and\(Y,X2\)\(Y,X\_\{2\}\):

S​\(Y;X1,X2\)=Ip​\(Y;X1,X2\)⏟predictive informationin the joint distribution−minq∈Δp​∫q​\(y,x1,x2\)​log⁡q​\(y∣x1,x2\)q​\(y\)​d​y​d​x1​d​x2⏟maximum predictive information from the unimodal marginals,S\(Y;X\_\{1\},X\_\{2\}\)=\\underbrace\{I\_\{p\}\(Y;X\_\{1\},X\_\{2\}\)\}\_\{\\begin\{subarray\}\{c\}\\text\{predictive information\}\\\\ \\text\{in the joint distribution\}\\end\{subarray\}\}\-\\underbrace\{\\min\_\{q\\in\\Delta\_\{p\}\}\\int q\(y,x\_\{1\},x\_\{2\}\)\\log\\frac\{q\(y\\mid x\_\{1\},x\_\{2\}\)\}\{q\(y\)\}\\,dy\\,dx\_\{1\}\\,dx\_\{2\}\}\_\{\\text\{maximum predictive information from the unimodal marginals\}\},\(2\)withΔp=\{q:q​\(y,xi\)=p​\(y,xi\),i∈\{1,2\}\}\\Delta\_\{p\}=\\\{\\,q:q\(y,x\_\{i\}\)=p\(y,x\_\{i\}\),\\ i\\in\\\{1,2\\\}\\,\\\}\. The first term is the predictive information available under the joint distribution; the second upper\-bounds what any predictor whose dependence on\(X1,X2\)\(X\_\{1\},X\_\{2\}\)factors through the bivariate marginals\. Standard cross\-entropy training maximizesI​\(Y;X1,X2\)I\(Y;X\_\{1\},X\_\{2\}\)as a single scalar but offers no mechanism to allocate gradient signal toSSrather thanRR,U1U\_\{1\}, orU2U\_\{2\}\(Lianget al\.,[2023](https://arxiv.org/html/2606.09853#bib.bib114)\), motivating an objective that targetsSSdirectly\.

### 3\.1A Motivating Analysis: What Makes Synergy Hard to Learn?

A common hypothesis attributes the difficulty of learning synergy to gradient interference between modalities or PID sources\(Wei and Hu,[2024](https://arxiv.org/html/2606.09853#bib.bib103)\)\.We probe this on a bimodal XOR with single\-source examples \(App\.[F\.2](https://arxiv.org/html/2606.09853#A6.SS2)\), attributing gradient signal to specific components\.

##### Observations\.

We diagnose vanilla training using two per\-source quantities derived from the empirical Neural Tangent Kernel \(NTK\)\(Jacotet al\.,[2018](https://arxiv.org/html/2606.09853#bib.bib116); Chizatet al\.,[2019](https://arxiv.org/html/2606.09853#bib.bib115)\), which characterizes training dynamics through inner products of per\-example gradients: the*learning signal strength*λg\\lambda\_\{g\}, measuring how strongly examples from PID sourceggdrive parameter updates, and the*gradient alignment*cos⁡\(g,h\)\\cos\(g,h\), measuring interference between sources \(derivations in App\.[K](https://arxiv.org/html/2606.09853#A11)\)\. Figure[1](https://arxiv.org/html/2606.09853#S3.F1)shows synergistic examples receiving the largestλg\\lambda\_\{g\}\(left\) and pairwise alignments staying near zero \(center\): synergy is neither starved of gradient signal nor in destructive competition with other sources\. Yet its training loss drops while validation loss rises \(right\), indicating overfitting on a scarce subset rather than inability to learn\. This localizes the failure to training dynamics rather than representational capacity\. SynIB is designed for this regime, motivating the model to learn from cross\-modal predictions throughout training; App\.[A\.1](https://arxiv.org/html/2606.09853#A1.SS1)reports the diagnostics under SynIB\.

![Refer to caption](https://arxiv.org/html/2606.09853v1/x1.png)Figure 1:Gradient geometry across PID sources under vanilla fusion\.A model is trained on examples drawn from the three PID sources,U1U\_\{1\},RR, andSS, withU2=0U\_\{2\}=0by construction \(details in Sec\.[4\.2](https://arxiv.org/html/2606.09853#S4.SS2)\)\.Left:Per\-group learning signal strength is substantial for all sources, meaning that the examples of that source create gradient capable of changing the parameters, with synergistic examples producing the largestλg\\lambda\_\{g\}\.Center:Gradient alignment between source pairs stays near zero or slightly positive throughout training, indicating no destructive competition\.Right:Fusion BCE by source on intact inputs \(solid train, dashed validation\)\. Synergistic training loss drops to zero while validation rises, indicating overfitting on the scarce synergistic examples and leaving test synergy accuracy at chance \(50\.1%50\.1\\%; total94\.4%94\.4\\%\)\.

### 3\.2The SynIB Training Objective

The above analysis suggests synergy fails because the model converges to unimodal or redundant shortcuts before synergistic cues can be learned\(Huanget al\.,[2022](https://arxiv.org/html/2606.09853#bib.bib19)\)\. SynIB addresses this by penalizing the model when it remains confident under a modality corruptionX~1\\tilde\{X\}\_\{1\}\(with latentZ~1=fθ1​\(X~1\)\\tilde\{Z\}\_\{1\}=f\_\{\\theta\_\{1\}\}\(\\tilde\{X\}\_\{1\}\)\): confidence under the counterfactual\(X~1,X2\)\(\\tilde\{X\}\_\{1\},X\_\{2\}\)indicates the model reached its answer without cross\-modal interaction, pushing it toward joint use of both modalities\. The objective combines cross\-entropy on intact inputs with a confidence penalty on counterfactuals\.

The remainder of this section develops each component\. Sec\.[3\.3](https://arxiv.org/html/2606.09853#S3.SS3)grounds the confidence penalty in information\-theory; Sec\.[3\.4](https://arxiv.org/html/2606.09853#S3.SS4)derives a tractable variational surrogate; Sec\.[3\.5](https://arxiv.org/html/2606.09853#S3.SS5)specifies the corruption operatorX~1\\tilde\{X\}\_\{1\}; and Sec\.[3\.6](https://arxiv.org/html/2606.09853#S3.SS6)assembles the full objective\.

### 3\.3Quantifying Complementarity from Counterfactual Predictions

The confidence penalty under corruption measures how much the model’s predictions change when information from one modality is removed\. This quantity has a principled interpretation, derived from the informationX1X\_\{1\}contributes to predictingYYbeyond what is available fromX2X\_\{2\}alone\.

The informationX1X\_\{1\}carries aboutYYgivenX2X\_\{2\}is captured by the conditional mutual informationI​\(X1;Y∣X2\)=H​\(Y∣X2\)−H​\(Y∣X1,X2\)I\(X\_\{1\};Y\\mid X\_\{2\}\)=H\(Y\\mid X\_\{2\}\)\-H\(Y\\mid X\_\{1\},X\_\{2\}\)\. Computing this quantity requires marginalizingX1X\_\{1\}out ofH​\(Y∣X2\)H\(Y\\mid X\_\{2\}\), which is intractable in high dimensions\. We replace this marginalization with a single targeted perturbationX~1\\tilde\{X\}\_\{1\}designed to removeX1X\_\{1\}’s task\-relevant content, and measure the resulting loss in conditional predictability, the*Mutual Information Perturbation Difference*:

MIPD1=H​\(Y∣X~1,X2\)−H​\(Y∣X1,X2\)=I​\(X1;Y∣X2\)−I​\(X~1;Y∣X2\)≤I​\(X1;Y∣X2\)\.\\mathrm\{MIPD\}\_\{1\}=H\(Y\\mid\\tilde\{X\}\_\{1\},X\_\{2\}\)\-H\(Y\\mid X\_\{1\},X\_\{2\}\)=I\(X\_\{1\};Y\\mid X\_\{2\}\)\-I\(\\tilde\{X\}\_\{1\};Y\\mid X\_\{2\}\)\\leq I\(X\_\{1\};Y\\mid X\_\{2\}\)\.\(3\)MaximizingMIPD1\\mathrm\{MIPD\}\_\{1\}maximizes a lower bound on the contribution ofX1X\_\{1\}beyondX2X\_\{2\}; derivation and tightness conditions are in App\.[B](https://arxiv.org/html/2606.09853#A2)\. This transforms the question of estimating CMI into the design problem of constructingX~1\\tilde\{X\}\_\{1\}, addressed in Sec\.[3\.5](https://arxiv.org/html/2606.09853#S3.SS5)\.

![Refer to caption](https://arxiv.org/html/2606.09853v1/x2.png)Figure 2:SynIB overview\.Standard multimodal fusion \(black\) trains a model to predictYYfrom\(Z1,Z2\)\(Z\_\{1\},Z\_\{2\}\), leaving optimization free to settle on unimodal or redundant cues\. SynIB \(blue\) adds counterfactual passes in which one modality is replaced with a feature\-masked versionZ~i\\tilde\{Z\}\_\{i\}that removes its task\-relevant content, and penalizes the model when its predictions remain confident under this corruption\. Confidence under masking signals reliance on a single modality; the KL penalty against a reference distributionrrpushes the model toward predictions that depend on the joint\(Z1,Z2\)\(Z\_\{1\},Z\_\{2\}\), so the cross\-entropy objective prioritizes learning from cross\-modal interactions\.
### 3\.4Variational Approximation for Tractable Cross\-Modal Optimization

Eq\. \([3](https://arxiv.org/html/2606.09853#S3.E3)\) is not directly optimizable: the conditional entropies require access to the true predictive distributionsp​\(y∣x1,x2\)p\(y\\mid x\_\{1\},x\_\{2\}\)andp​\(y∣x~1,x2\)p\(y\\mid\\tilde\{x\}\_\{1\},x\_\{2\}\)\. We obtain a tractable surrogate by approximating each entropy through variational quantities the model can compute\.

ForH​\(Y∣X1,X2\)H\(Y\\mid X\_\{1\},X\_\{2\}\), a variational decoderqθ​\(y∣x1,x2\)q\_\{\\theta\}\(y\\mid x\_\{1\},x\_\{2\}\)and the non\-negativity of KL\(Barber and Agakov,[2004](https://arxiv.org/html/2606.09853#bib.bib118)\)give the standard upper boundH​\(Y∣X1,X2\)≤𝔼p​\(x1,x2,y\)​\[−log⁡qθ​\(y∣x1,x2\)\]H\(Y\\mid X\_\{1\},X\_\{2\}\)\\leq\\mathbb\{E\}\_\{p\(x\_\{1\},x\_\{2\},y\)\}\[\-\\log q\_\{\\theta\}\(y\\mid x\_\{1\},x\_\{2\}\)\], recovering the cross\-entropy loss\. ForH​\(Y∣X~1,X2\)H\(Y\\mid\\tilde\{X\}\_\{1\},X\_\{2\}\), the reference\-based entropy identity\(Tishbyet al\.,[2000](https://arxiv.org/html/2606.09853#bib.bib73); Alemiet al\.,[2016](https://arxiv.org/html/2606.09853#bib.bib75)\)gives

H\(Y∣X~1,X2\)=Cr−𝔼p​\(x~1,x2\)DKL\(p\(⋅∣x~1,x2\)∥r\(⋅\)\),\\displaystyle H\(Y\\mid\\tilde\{X\}\_\{1\},X\_\{2\}\)=C\_\{r\}\-\\mathbb\{E\}\_\{p\(\\tilde\{x\}\_\{1\},x\_\{2\}\)\}\\,D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\(\\cdot\\mid\\tilde\{x\}\_\{1\},x\_\{2\}\)\\,\\\|\\,r\(\\cdot\)\\right\),\(4\)whereCr=𝔼p​\(y\)​\[−log⁡r​\(y\)\]C\_\{r\}=\\mathbb\{E\}\_\{p\(y\)\}\[\-\\log r\(y\)\]is constant inθ\\theta\. Substitutingqθq\_\{\\theta\}forppin Eq\. \([4](https://arxiv.org/html/2606.09853#S3.E4)\) defines the variational surrogateH^θ​\(Y∣X~1,X2\)\\widehat\{H\}\_\{\\theta\}\(Y\\mid\\tilde\{X\}\_\{1\},X\_\{2\}\), which recovers the true entropy whenqθ=pq\_\{\\theta\}=p\. CombiningH^θ\\widehat\{H\}\_\{\\theta\}with the cross\-entropy upper bound onH​\(Y∣X1,X2\)H\(Y\\mid X\_\{1\},X\_\{2\}\)gives a tractable surrogate forMIPD1\\mathrm\{MIPD\}\_\{1\}, which coincides with the true value when the variational family containspp\. The two terms together form the SynIB training objective, assembled in Sec\.[3\.6](https://arxiv.org/html/2606.09853#S3.SS6)\. Variational families, closed\-form KL divergences, and properties of the surrogate are in Appendices[C](https://arxiv.org/html/2606.09853#A3)and[B](https://arxiv.org/html/2606.09853#A2)\.

### 3\.5Counterfactual Corruption via Feature Masking

The MIPD lower bound from Sec\.[3\.3](https://arxiv.org/html/2606.09853#S3.SS3)tightens with smallerI​\(X~1;Y∣X2\)I\(\\tilde\{X\}\_\{1\};Y\\mid X\_\{2\}\), giving a concrete design criterion: chooseX~1\\tilde\{X\}\_\{1\}to minimize residual task\-relevant information while perturbing as little ofX1X\_\{1\}as possible\. We implement the corruption as feature\-wise masking through a binary maskM1∈\{0,1\}d1M\_\{1\}\\in\\\{0,1\\\}^\{d\_\{1\}\}and a gatez~1=\(1−M1\)⊙z1\+M1⊙ϵ\\tilde\{z\}\_\{1\}=\(1\-M\_\{1\}\)\\odot z\_\{1\}\+M\_\{1\}\\odot\\epsilon, whereϵ\\epsilonis noise independent ofYYand coordinates withM1,j=1M\_\{1,j\}=1are corrupted\. Feature\-level control gives more precise residual minimization than global perturbations, and the operator can be applied equivalently toxix\_\{i\}or the latentziz\_\{i\}\.

The choice ofM1M\_\{1\}determines what SynIB ends up penalizing\. The ideal choice would corrupt cross\-modal cues while leaving unimodal ones intact, so that any remaining confidence under\(X~1,X2\)\(\\tilde\{X\}\_\{1\},X\_\{2\}\)comes from unimodal reliance and gets penalized\. Since unimodal cues are not labeled, we identify them adversarially: alearned maskMψ1∈\{0,1\}d1M\_\{\\psi\_\{1\}\}\\in\\\{0,1\\\}^\{d\_\{1\}\}is trained to flag the minimum set of coordinates whose corruption collapses unimodal prediction,

ℒmask​\(ψ1\)=−CE​\(fθc1​\(\(1−Mψ1\)⊙x1\+Mψ1⊙ϵ\),y\)\+λM​‖Mψ1‖1,\\mathcal\{L\}\_\{\\text\{mask\}\}\(\\psi\_\{1\}\)=\-\\mathrm\{CE\}\\\!\\left\(f\_\{\\theta\_\{c\_\{1\}\}\}\\\!\\left\(\(1\-M\_\{\\psi\_\{1\}\}\)\\odot x\_\{1\}\+M\_\{\\psi\_\{1\}\}\\odot\\epsilon\\right\),\\,y\\right\)\+\\lambda\_\{M\}\\\|M\_\{\\psi\_\{1\}\}\\\|\_\{1\},\(5\)and the SynIB corruption mask is its inverse,M1=𝟏−Mψ1M\_\{1\}=\\mathbf\{1\}\-M\_\{\\psi\_\{1\}\}\. This isolates the features the unimodal predictor relies on: SynIB preserves them and corrupts the rest, leaving the model with only unimodal cues at the counterfactual pass\. Eq\. \([5](https://arxiv.org/html/2606.09853#S3.E5)\) is equivalent to finding the sparsest perturbation that maximally degrades the unimodal log\-likelihood \(App\.[E](https://arxiv.org/html/2606.09853#A5)\)\.

Alongside learned masking, we examine a simplerrandom maskingvariant,M1∼Bernoulli​\(π\)M\_\{1\}\\sim\\mathrm\{Bernoulli\}\(\\pi\), which corrupts coordinates independently without consulting any predictor\. This variant is parameter\-free and serves by providing a strong task\-agnostic baseline when learned masking is impractical, and it isolates the benefits of precise mask construction\. Figure[3](https://arxiv.org/html/2606.09853#S3.F3)illustrates the two constructions\.

![Refer to caption](https://arxiv.org/html/2606.09853v1/x3.png)Figure 3:Two strategies for constructing the counterfactual maskMM\.Left:Learned masking trainsMψ1M\_\{\\psi\_\{1\}\}adversarially against a unimodal predictor with a sparsity penalty, isolating the features needed for unimodal prediction; SynIB inverts this mask so the corruption removes everything else, leaving only unimodal cues intact\.Right:Random masking samples each coordinate independently asMi∼Bernoulli​\(π\)M\_\{i\}\\sim\\mathrm\{Bernoulli\}\(\\pi\), corrupting features without assumptions about their content\.
### 3\.6The SynIB Objective

Combining the surrogate \(Sec\.[3\.4](https://arxiv.org/html/2606.09853#S3.SS4)\) with the masking operator \(Sec\.[3\.5](https://arxiv.org/html/2606.09853#S3.SS5)\) yields the SynIB objective:

ℒ=𝔼p​\(x1,x2,y\)​\[−log⁡qθ​\(y∣x1,x2\)\]⏟accurate prediction with intact modalities\+λ​𝔼p​\(x1,x2\)𝔼p​\(x~1∣x1\)DKL\(qθ\(⋅∣x~1,x2\)∥r\(⋅\)\)⏟uncertainty under modality corruption\\mathcal\{L\}=\\underbrace\{\\mathbb\{E\}\_\{p\(x\_\{1\},x\_\{2\},y\)\}\[\-\\log q\_\{\\theta\}\(y\\mid x\_\{1\},x\_\{2\}\)\]\}\_\{\\text\{accurate prediction with intact modalities\}\}\+\\lambda\\,\\underbrace\{\\mathbb\{E\}\_\{p\(x\_\{1\},x\_\{2\}\)\}\\mathbb\{E\}\_\{p\(\\tilde\{x\}\_\{1\}\\mid x\_\{1\}\)\}D\_\{\\mathrm\{KL\}\}\\\!\\left\(q\_\{\\theta\}\(\\cdot\\mid\\tilde\{x\}\_\{1\},x\_\{2\}\)\\,\\\|\\,r\(\\cdot\)\\right\)\}\_\{\\text\{uncertainty under modality corruption\}\}\(6\)

whereqθ​\(y∣x1,x2\)q\_\{\\theta\}\(y\\mid x\_\{1\},x\_\{2\}\)is the model’s predictive distribution,r​\(y\)r\(y\)a fixed reference distribution, andλ∈ℝ\+\\lambda\\in\\mathbb\{R\}^\{\+\}the regularization strength\. Figure[2](https://arxiv.org/html/2606.09853#S3.F2)illustrates the resulting training procedure: at each step, the fusion model runs forward passes on the intact pair and on each counterfactual, with the cross\-entropy term applied to the first and the KL penalty to the others\.

## 4Experiments

We evaluate SynIB on five real\-world multimodal benchmarks \(Sec\.[4\.1](https://arxiv.org/html/2606.09853#S4.SS1)\), comprising three MultiBench affective tasks, Hateful Memes with large pretrained backbones, and a controllable irony extension of CREMA\-D we introduce, and on synthetic XOR tasks \(Sec\.[4\.2](https://arxiv.org/html/2606.09853#S4.SS2)\)\. We ask whether SynIB improves performance on examples requiring cross\-modal reasoning, and whether targeting synergy harms accuracy on the full test set where unimodal cues already suffice\.

### 4\.1Real\-World Multimodal Benchmarks

We evaluate on five real\-world multimodal tasks\. Three come from MultiBench\(Lianget al\.,[2021](https://arxiv.org/html/2606.09853#bib.bib95)\): UR\-Funny\(Hasanet al\.,[2019](https://arxiv.org/html/2606.09853#bib.bib123)\)\(humor detection\), MUStARD\(Castroet al\.,[2019](https://arxiv.org/html/2606.09853#bib.bib124)\)\(sarcasm detection\), and CMU\-MOSI\(Zadehet al\.,[2016](https://arxiv.org/html/2606.09853#bib.bib91)\)\(sentiment analysis\)\. The fourth is Hateful Memes\(Kielaet al\.,[2020](https://arxiv.org/html/2606.09853#bib.bib129)\), where hatefulness frequently emerges only from the joint interpretation of image and text\. The fifth is CREMA\-D\(Caoet al\.,[2014](https://arxiv.org/html/2606.09853#bib.bib8)\), a six\-class audio–visual emotion recognition benchmark whose largely congruent signals admit strong unimodal performance\. Because CREMA\-D does not stress cross\-modal integration, we further construct CREMA\-D\-Irony: for a fractionα\\alphaof examples we replace the audio with a donor clip carrying a contradicting emotion and relabel the sample as*ironic*\. The rateα\\alphacontrols synergy density \(App\.[H](https://arxiv.org/html/2606.09853#A8)\)\.

##### Baselines\.

We compare SynIB against unimodal models, late ensembling, vanilla fusion, and four modality\-balancing methods: \(1\) D&R\(Weiet al\.,[2024](https://arxiv.org/html/2606.09853#bib.bib105)\), which periodically re\-initializes encoders; \(2\) MMPareto\(Wei and Hu,[2024](https://arxiv.org/html/2606.09853#bib.bib103)\), which balances unimodal and multimodal gradients via a Pareto rule; \(3\) ReconBoost\(Huaet al\.,[2024](https://arxiv.org/html/2606.09853#bib.bib104)\), which alternates unimodal updates with KL reconciliation; and \(4\) MCR\(Kontraset al\.,[2025](https://arxiv.org/html/2606.09853#bib.bib110)\), which adds a mutual\-information regularizer to the fusion loss\. All methods share the same backbone and differ only in the training objective \(full descriptions in App\.[J](https://arxiv.org/html/2606.09853#A10)\)\. For Hateful Memes we use CLIP ViT\-B/16\(Radfordet al\.,[2021](https://arxiv.org/html/2606.09853#bib.bib61)\)\(86M\) and DeBERTa\-v3\-base\(Heet al\.,[2021](https://arxiv.org/html/2606.09853#bib.bib130)\)\(184M\) encoders feeding a fusion Transformer\.

Isolating synergy\-dependent examples\.Frequently multimodal datasets test examples are solvable from a single modality\. To isolate cases requiring multimodal integration, we define asynergy subsetof test samples misclassified by every unimodal model\. The subset size indicates each task’s reliance on cross\-modal interaction: Hateful Memes \(26\.5%\), MUStARD \(12\.7%\), UR\-Funny \(11\.6%\), and MOSI \(2\.5%\)\. With only 17 examples, the MOSI synergy subset is too small for firm conclusions; we treat MOSI as a check that synergy\-targeted training does not harm tasks where unimodal cues already suffice\. CREMA\-D\-Irony complements this post\-hoc view: the irony class is*constructed*to require cross\-modal integration, withα\\alphadirectly controlling its density\.

0\.10\.5120102030405060Irony rateα\\alphaIrony\-class F1Irony recognition0\.10\.51235404550556065Irony rateα\\alphaTotal F1Overall performanceUni\-VUni\-AEnsembleVanilla fusionD&RMMParetoReconBoostMCRSynIBFigure 4:F1 scores on the CREMA\-D irony recognition task under varying irony ratesα\\alpha\.Left:irony\-class F1\.Right:total F1\. Standard fusion and prior balancing methods yield unstable gains when irony is rare, whereasSynIBconsistently improves irony detection across all regimes with only minor trade\-offs in overall performance, often achieving the best total F1\.
##### Results on CREMA\-D\-Irony\.

Figure[4](https://arxiv.org/html/2606.09853#S4.F4)reports irony\-class F1 \(left\) and total F1 \(right\) acrossα\\alpha\. SynIB outperforms MCR, the strongest baseline, on irony F1 at everyα\\alpha, most clearly atα=0\.5\\alpha\{=\}0\.5\(21\.7 vs\. 17\.6\) and with consistent margins atα=0\.1\\alpha\{=\}0\.1\(16\.3 vs\. 14\.1\) andα=1\.0\\alpha\{=\}1\.0\(35\.3 vs\. 31\.6\)\. Vanilla fusion and ensembling become competitive only when synergy is abundant \(α≥1\\alpha\{\\geq\}1\)\. SynIB stays within 1\.0–3\.3 points of the best total F1 at everyα\\alpha\(full numbers in App\.[H\.7](https://arxiv.org/html/2606.09853#A8.SS7)\)\.

##### Results on MultiBench and Hateful Memes\.

Figure[5](https://arxiv.org/html/2606.09853#S4.F5)reports accuracy on the synergy subset and on the full test set\. On the synergy subset, SynIB achieves the largest gains across all four benchmarks, with learned masking improving over the strongest baseline by\+3\.0\+3\.0on Hateful Memes,\+7\.8\+7\.8on MUStARD, and\+3\.6\+3\.6on UR\-Funny; random masking is strong on Hateful Memes \(\+3\.0\+3\.0\) but more variable elsewhere, suggesting that the targeted mask is what reliably drives improvements on synergy\-required examples\. Modality\-balancing methods \(D&R, MMPareto, ReconBoost, MCR\) yield inconsistent gains on the synergy subset, consistent with our argument that rebalancing alone does not induce synergistic learning\. On the full test set, SynIB remains the top performer on Hateful Memes \(\+1\.1\+1\.1random,\+0\.3\+0\.3learned\) and MUStARD \(\+3\.8\+3\.8learned\), and stays within11point on UR\-Funny \(−0\.9\-0\.9\) and MOSI \(\+0\.6\+0\.6\), indicating no cost to overall accuracy\.

UR\-FunnyMUStARDMOSIHatefulMemes0101020203030404050506060Acc \(%\)−3\.8\-3\.8\+2\.4\+2\.4\+3\.9\+3\.9\+3\.0\+3\.0\+3\.6\+3\.6\+7\.8\+7\.8\+5\.8\+5\.8\+1\.1\+1\.1\(a\) Synergy Subset AccuracyUR\-FunnyMUStARDMOSIHatefulMemes505055556060656570707575Acc \(%\)==\+1\.9\+1\.9\+0\.3\+0\.3\+1\.1\+1\.1−0\.9\-0\.9\+3\.8\+3\.8\+0\.6\+0\.6\+0\.3\+0\.3\(b\) Whole\-test AccuracyEnsembleVanilla FusionD&RMMParetoReconBoostMCRSynIB​MRandom\\mathrm\{SynIB\}\\ M\_\{\\mathrm\{Random\}\}SynIB​MLearned\\mathrm\{SynIB\}\\ M\_\{\\mathrm\{Learned\}\}

Figure 5:Comparison across real\-world multimodal benchmarks\.\(a\)Synergy\-subset accuracy on UR\-Funny, MUStARD, MOSI, and Hateful Memes \(samples misclassified by every unimodal model\)\.\(b\)Whole\-test accuracy\. Baselines comprise late ensembling, vanilla fusion, and four modality\-balancing methods: D&R\(Weiet al\.,[2024](https://arxiv.org/html/2606.09853#bib.bib105)\), MMPareto\(Wei and Hu,[2024](https://arxiv.org/html/2606.09853#bib.bib103)\), ReconBoost\(Huaet al\.,[2024](https://arxiv.org/html/2606.09853#bib.bib104)\), and MCR\(Kontraset al\.,[2025](https://arxiv.org/html/2606.09853#bib.bib110)\); SynIB with \(MRandomM\_\{\\mathrm\{Random\}\}\) and \(MLearnedM\_\{\\mathrm\{Learned\}\}\) masking\. Annotations above SynIB bars give the difference relative to the best baseline\. SynIB consistently improves synergy\-subset accuracy on all four while remaining competitive on the full test set\.Across five benchmarks spanning humor, sarcasm, sentiment, hate speech, and emotion\-irony, SynIB consistently improves performance on the examples that demonstrably require cross\-modal reasoning, while remaining competitive, and often best, on the full test distribution\. The pattern is most pronounced on tasks with substantial synergy content \(CREMA\-D\-Irony, MUStARD, Hateful Memes\) and naturally smaller on tasks where unimodal shortcuts already solve most of the problem\.

### 4\.2Synthetic Experiments

We design two bimodal XOR tasks\.*Spurious XOR*probes resistance to unimodal shortcuts: the label is the XOR of two modality\-specific bits \(neither modality alone informative\), but one modality additionally carries a label\-correlated feature available only at training, letting the model fit training perfectly without the cross\-modal signal\.*PID\-Controlled XOR*sweeps the density of each PID source \(unique\-to\-X1X\_\{1\}, redundant, synergistic\)\. Full setup in App\.[F](https://arxiv.org/html/2606.09853#A6)\.

![Refer to caption](https://arxiv.org/html/2606.09853v1/x4.png)Figure 6:Test accuracy vs\. spurious correlation strengthβ\\betaon bimodal XOR\. Vanilla fusion collapses to chance as the shortcut strengthens\. SynIB with oracle masking stays at∼\\sim100%, while learned and random masks degrade gracefully, with the learned mask consistently closer to the oracle\.##### Robustness to spurious shortcuts\.

One modality contains a spurious feature linearly correlated with the label at training strengthβ\\betabut uncorrelated at test\. Asβ\\betagrows, the shortcut becomes increasingly attractive during training but provides no test\-time signal\. Figure[6](https://arxiv.org/html/2606.09853#S4.F6)shows that vanilla fusion collapses to chance onceβ≥0\.6\\beta\\geq 0\.6, locking onto the shortcut and discarding the cross\-modal signal\. SynIB with oracle masking maintains∼\\sim100% across allβ\\beta, confirming that isolating synergistic information suffices\. Without oracle annotations, both random and learned masks substantially improve robustness at everyβ\\beta, with the learned mask consistently closer to the oracle by identifying the features the unimodal head exploits and corrupting their complement\.

![Refer to caption](https://arxiv.org/html/2606.09853v1/x5.png)Figure 7:Performance across PID\-controlled data regimes on the synthetic XOR task\.Each triangle is a probability simplex over information types\(pU1,pRed,pSyn\)\(p\_\{\\mathrm\{U1\}\},p\_\{\\mathrm\{Red\}\},p\_\{\\mathrm\{Syn\}\}\); top row shows total accuracy, bottom row accuracy on synergistic examples only\. Vanilla fusion \(a\) degrades sharply as synergy dominates\. SynIB with oracle masking \(b\) resolves the task uniformly\. Random masking \(c\) provides inconsistent gains, while learned masking \(d\) closely recovers oracle performance\.
##### Coverage across PID compositions\.

Each example is generated from exactly one active source \(unique, redundant, or synergistic\), enabling controlled interpolation between unimodal\-dominated and synergy\-dominated regimes\. Figure[7](https://arxiv.org/html/2606.09853#S4.F7)reports performance across the information simplex\. Without SynIB \(Fig\.[7](https://arxiv.org/html/2606.09853#S4.F7)a\), accuracy degrades sharply as synergistic structure becomes dominant; with oracle masking \([7](https://arxiv.org/html/2606.09853#S4.F7)b\) it remains uniformly high\. Random masking \([7](https://arxiv.org/html/2606.09853#S4.F7)c\) yields inconsistent gains, while the learned mask \([7](https://arxiv.org/html/2606.09853#S4.F7)d\) closely matches the oracle even when synergy is scarce, indicating that SynIB recovers synergistic structure from data without requiring annotations\.

##### Per\-source training dynamics\.

Figure[8](https://arxiv.org/html/2606.09853#S4.F8)disaggregates accuracy by PID source across training, directly testing the diagnosis from Sec\.[3\.1](https://arxiv.org/html/2606.09853#S3.SS1): that synergy fails through overfitting on a scarce subset rather than through gradient interference or capacity limits\. Vanilla fusion fitsU1U\_\{1\}andRRon both splits, but synergy shows the signature train–validation divergence, with training accuracy climbing while validation stalls at chance\. The three SynIB variants close this gap: synergy train and validation track each other throughout training, with learned masking \(0\.890\.89\) tracking the oracle \(0\.910\.91\) within two points and random masking \(0\.870\.87\) close behind\.

![Refer to caption](https://arxiv.org/html/2606.09853v1/x6.png)Figure 8:PID\-XOR training dynamics across all four methods\.Per\-source training \(solid\) and validation \(dashed\) accuracy across 30 epochs, mean±\\pmstandard error over three seeds\. Sources are colored by type: unique\-to\-modality\-1 \(U1U\_\{1\}, blue\), redundant \(RR, green\), and synergistic \(SS, red\)\. Final synergy test accuracies \(annotated per panel\):0\.500\.50vanilla,0\.910\.91oracle,0\.870\.87random,0\.890\.89learned\. PID mixture\(pU1,pU2,pR,pS\)=\(0\.45,0,0\.45,0\.10\)\(p\_\{U\_\{1\}\},p\_\{U\_\{2\}\},p\_\{R\},p\_\{S\}\)=\(0\.45,0,0\.45,0\.10\)\. Vanilla fusion \(left\) fitsU1U\_\{1\}andRRon both splits, but synergy validation stalls at chance while training accuracy climbs, the signature of memorization on the scarce synergistic examples\. The three SynIB variants \(M⋆M^\{\\star\},MRandomM\_\{\\mathrm\{Random\}\},MLearnedM\_\{\\mathrm\{Learned\}\}\) close this gap, lifting synergy validation accuracy without degradingU1U\_\{1\}orRR\.

## 5Conclusion

Multimodal models often default to unimodal shortcuts during training, even when the task rewards combining modalities\. We argued this is an optimization problem: synergistic cues are scarce, so the model fits dominant unimodal signals first and never learns cross\-modal interaction\. SynIB addresses this by penalizing the model when its predictions remain confident under modality corruption, pushing it toward signals that emerge only from joint use of both modalities\. Across synthetic tasks, and five real\-world datasets including the controllable irony construction of CREMA\-D and the Hateful Memes with pretrained backbones, SynIB consistently improves accuracy on examples requiring cross\-modal reasoning while remaining competitive on the full test set\. See App\.[A](https://arxiv.org/html/2606.09853#A1)for limitations\.

##### Broader Impact\.

Synergistic information underlies many of the settings where multimodal learning is most consequential, and methods that reliably train models to use cross\-modal interactions, rather than defaulting to whichever modality is easiest to fit, expand the set of problems where multimodal learning can deliver on its premise\. Better synergy does not, on its own, make a model trustworthy: a model that integrates modalities more effectively can still inherit biases from its training data, fail under distribution shift, or be applied in settings where higher accuracy is not the primary concern\. We see synergy\-targeted training as one ingredient in responsible multimodal learning, useful where cross\-modal reasoning is the bottleneck and best paired with the auditing practices appropriate to each application\.

## References

- Deep variational information bottleneck\.arXiv preprint arXiv:1612\.00410\.Cited by:[§B\.1](https://arxiv.org/html/2606.09853#A2.SS1.SSS0.Px2.p1.8),[§3\.4](https://arxiv.org/html/2606.09853#S3.SS4.p2.4)\.
- A\. Baevski, W\. Hsu, Q\. Xu, A\. Babu, J\. Gu, and M\. Auli \(2022\)Data2vec: a general framework for self\-supervised learning in speech, vision and language\.InInternational conference on machine learning,pp\. 1298–1312\.Cited by:[§2](https://arxiv.org/html/2606.09853#S2.SS0.SSS0.Px2.p1.1)\.
- D\. Barber and F\. Agakov \(2004\)The im algorithm: a variational approach to information maximization\.Advances in neural information processing systems16\(320\),pp\. 201\.Cited by:[§B\.1](https://arxiv.org/html/2606.09853#A2.SS1.SSS0.Px1.p1.1),[§3\.4](https://arxiv.org/html/2606.09853#S3.SS4.p2.4)\.
- N\. Bertschinger, J\. Rauh, E\. Olbrich, J\. Jost, and N\. Ay \(2014\)Quantifying unique information\.Entropy16\(4\),pp\. 2161–2183\.Cited by:[§2](https://arxiv.org/html/2606.09853#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2606.09853#S3.SS0.SSS0.Px2.p1.17),[§3](https://arxiv.org/html/2606.09853#S3.SS0.SSS0.Px2.p1.8)\.
- H\. Cao, D\. G\. Cooper, M\. K\. Keutmann, R\. C\. Gur, A\. Nenkova, and R\. Verma \(2014\)Crema\-d: crowd\-sourced emotional multimodal actors dataset\.IEEE transactions on affective computing5\(4\),pp\. 377–390\.Cited by:[§H\.1](https://arxiv.org/html/2606.09853#A8.SS1.p1.1),[§4\.1](https://arxiv.org/html/2606.09853#S4.SS1.p1.2)\.
- S\. Castro, D\. Hazarika, V\. Pérez\-Rosas, R\. Zimmermann, R\. Mihalcea, and S\. Poria \(2019\)Towards multimodal sarcasm detection \(an \_obviously\_ perfect paper\)\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\. 4619–4629\.Cited by:[§4\.1](https://arxiv.org/html/2606.09853#S4.SS1.p1.2)\.
- L\. Chizat, E\. Oyallon, and F\. Bach \(2019\)On lazy training in differentiable programming\.Advances in neural information processing systems32\.Cited by:[Appendix K](https://arxiv.org/html/2606.09853#A11.SS0.SSS0.Px3.p1.8),[§3\.1](https://arxiv.org/html/2606.09853#S3.SS1.SSS0.Px1.p1.4)\.
- Y\. Fan, W\. Xu, H\. Wang, J\. Wang, and S\. Guo \(2023\)PMR: prototypical modal rebalance for multimodal learning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 20029–20038\.Cited by:[§2](https://arxiv.org/html/2606.09853#S2.SS0.SSS0.Px3.p1.1)\.
- B\. Ghorbani, S\. Mei, T\. Misiakiewicz, and A\. Montanari \(2019\)Limitations of lazy training of two\-layers neural network\.Advances in Neural Information Processing Systems32\.Cited by:[Appendix K](https://arxiv.org/html/2606.09853#A11.SS0.SSS0.Px3.p1.8)\.
- I\. Goodfellow, Y\. Bengio, A\. Courville, and Y\. Bengio \(2016\)Deep learning\.Vol\.1,MIT press Cambridge\.Cited by:[§1](https://arxiv.org/html/2606.09853#S1.p1.1)\.
- V\. Griffith and C\. Koch \(2014\)Quantifying synergistic mutual information\.InGuided self\-organization: inception,pp\. 159–190\.Cited by:[§2](https://arxiv.org/html/2606.09853#S2.SS0.SSS0.Px1.p1.1)\.
- M\. K\. Hasan, W\. Rahman, A\. Bagher Zadeh, J\. Zhong, M\. I\. Tanveer, L\. Morency, and M\. \(\. Hoque \(2019\)UR\-FUNNY: a multimodal language dataset for understanding humor\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),pp\. 2046–2056\.Cited by:[§4\.1](https://arxiv.org/html/2606.09853#S4.SS1.p1.2)\.
- P\. He, J\. Gao, and W\. Chen \(2021\)Debertav3: improving deberta using electra\-style pre\-training with gradient\-disentangled embedding sharing\.arXiv preprint arXiv:2111\.09543\.Cited by:[§4\.1](https://arxiv.org/html/2606.09853#S4.SS1.SSS0.Px1.p1.1)\.
- C\. Hua, Q\. Xu, S\. Bao, Z\. Yang, and Q\. Huang \(2024\)ReconBoost: boosting can achieve modality reconcilement\.InProceedings of the 41st International Conference on Machine Learning,ICML’24\.Cited by:[Appendix J](https://arxiv.org/html/2606.09853#A10.SS0.SSS0.Px6),[§2](https://arxiv.org/html/2606.09853#S2.SS0.SSS0.Px3.p1.1),[Figure 5](https://arxiv.org/html/2606.09853#S4.F5),[Figure 5](https://arxiv.org/html/2606.09853#S4.F5.5.2),[§4\.1](https://arxiv.org/html/2606.09853#S4.SS1.SSS0.Px1.p1.1)\.
- Y\. Huang, J\. Lin, C\. Zhou, H\. Yang, and L\. Huang \(2022\)Modality competition: what makes joint training of multi\-modal network fail in deep learning?\(provably\)\.InInternational Conference on Machine Learning,pp\. 9226–9259\.Cited by:[Appendix J](https://arxiv.org/html/2606.09853#A10.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2606.09853#S1.p2.1),[§2](https://arxiv.org/html/2606.09853#S2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2606.09853#S3.SS2.p1.3)\.
- A\. Jacot, F\. Gabriel, and C\. Hongler \(2018\)Neural tangent kernel: convergence and generalization in neural networks\.Advances in neural information processing systems31\.Cited by:[§3\.1](https://arxiv.org/html/2606.09853#S3.SS1.SSS0.Px1.p1.4)\.
- E\. Jang, S\. Gu, and B\. Poole \(2016\)Categorical reparameterization with gumbel\-softmax\.arXiv preprint arXiv:1611\.01144\.Cited by:[Appendix D](https://arxiv.org/html/2606.09853#A4.SS0.SSS0.Px4.p1.8)\.
- B\. Ji, T\. Zhang, Y\. Zou, B\. Hu, and S\. Shen \(2022\)Increasing visual awareness in multimodal neural machine translation from an information theoretic perspective\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 6755–6764\.Cited by:[§2](https://arxiv.org/html/2606.09853#S2.SS0.SSS0.Px3.p1.1)\.
- D\. Kiela, H\. Firooz, A\. Mohan, V\. Goswami, A\. Singh, P\. Ringshia, and D\. Testuggine \(2020\)The hateful memes challenge: detecting hate speech in multimodal memes\.Advances in neural information processing systems33,pp\. 2611–2624\.Cited by:[§4\.1](https://arxiv.org/html/2606.09853#S4.SS1.p1.2)\.
- K\. Kontras, C\. Chatzichristos, M\. Blaschko, and M\. De Vos \(2024\)Improving multimodal learning with multi\-loss gradient modulation\.arXiv preprint arXiv:2405\.07930\.Cited by:[§2](https://arxiv.org/html/2606.09853#S2.SS0.SSS0.Px3.p1.1)\.
- K\. Kontras, T\. Strypsteen, C\. Chatzichristos, P\. P\. Liang, M\. B\. Blaschko, and M\. De Vos \(2025\)Balancing multimodal training through game\-theoretic regularization\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,Cited by:[Appendix J](https://arxiv.org/html/2606.09853#A10.SS0.SSS0.Px7),[§1](https://arxiv.org/html/2606.09853#S1.p2.1),[§2](https://arxiv.org/html/2606.09853#S2.SS0.SSS0.Px3.p1.1),[Figure 5](https://arxiv.org/html/2606.09853#S4.F5),[Figure 5](https://arxiv.org/html/2606.09853#S4.F5.5.2),[§4\.1](https://arxiv.org/html/2606.09853#S4.SS1.SSS0.Px1.p1.1)\.
- A\. Krogh and J\. Hertz \(1991\)A simple weight decay can improve generalization\.Advances in neural information processing systems4\.Cited by:[§1](https://arxiv.org/html/2606.09853#S1.p1.1)\.
- H\. Li, X\. Li, P\. Hu, Y\. Lei, C\. Li, and Y\. Zhou \(2023\)Boosting multi\-modal model performance with adaptive gradient modulation\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 22214–22224\.Cited by:[§2](https://arxiv.org/html/2606.09853#S2.SS0.SSS0.Px3.p1.1)\.
- P\. P\. Liang, Y\. Cheng, X\. Fan, C\. K\. Ling, S\. Nie, R\. Chen, Z\. Deng, N\. Allen, R\. Auerbach, F\. Mahmood,et al\.\(2023\)Quantifying & modeling multimodal interactions: an information decomposition framework\.Advances in Neural Information Processing Systems36,pp\. 27351–27393\.Cited by:[§2](https://arxiv.org/html/2606.09853#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2606.09853#S3.SS0.SSS0.Px2.p1.16)\.
- P\. P\. Liang, Z\. Deng, M\. Q\. Ma, J\. Y\. Zou, L\. Morency, and R\. Salakhutdinov \(2024a\)Factorized contrastive learning: going beyond multi\-view redundancy\.Advances in Neural Information Processing Systems36\.Cited by:[§2](https://arxiv.org/html/2606.09853#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2606.09853#S2.SS0.SSS0.Px2.p1.1)\.
- P\. P\. Liang, Y\. Lyu, X\. Fan, Z\. Wu, Y\. Cheng, J\. Wu, L\. Chen, P\. Wu, M\. A\. Lee, Y\. Zhu,et al\.\(2021\)Multibench: multiscale benchmarks for multimodal representation learning\.Advances in neural information processing systems2021\(DB1\),pp\. 1\.Cited by:[§4\.1](https://arxiv.org/html/2606.09853#S4.SS1.p1.2)\.
- P\. P\. Liang, A\. Zadeh, and L\. Morency \(2024b\)Foundations & trends in multimodal machine learning: principles, challenges, and open questions\.ACM Computing Surveys56\(10\),pp\. 1–42\.Cited by:[§1](https://arxiv.org/html/2606.09853#S1.p1.1)\.
- J\. Lu, D\. Batra, D\. Parikh, and S\. Lee \(2019\)Vilbert: pretraining task\-agnostic visiolinguistic representations for vision\-and\-language tasks\.Advances in neural information processing systems32\.Cited by:[§2](https://arxiv.org/html/2606.09853#S2.SS0.SSS0.Px2.p1.1)\.
- X\. Peng, Y\. Wei, A\. Deng, D\. Wang, and D\. Hu \(2022\)Balanced multimodal learning via on\-the\-fly gradient modulation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 8238–8247\.Cited by:[§2](https://arxiv.org/html/2606.09853#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark,et al\.\(2021\)Learning transferable visual models from natural language supervision\.InInternational conference on machine learning,pp\. 8748–8763\.Cited by:[§1](https://arxiv.org/html/2606.09853#S1.p2.1),[§2](https://arxiv.org/html/2606.09853#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2606.09853#S4.SS1.SSS0.Px1.p1.1)\.
- A\. Singh, R\. Hu, V\. Goswami, G\. Couairon, W\. Galuba, M\. Rohrbach, and D\. Kiela \(2022\)Flava: a foundational language and vision alignment model\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 15638–15650\.Cited by:[§1](https://arxiv.org/html/2606.09853#S1.p2.1),[§2](https://arxiv.org/html/2606.09853#S2.SS0.SSS0.Px2.p1.1)\.
- N\. Srivastava, G\. Hinton, A\. Krizhevsky, I\. Sutskever, and R\. Salakhutdinov \(2014\)Dropout: a simple way to prevent neural networks from overfitting\.The journal of machine learning research15\(1\),pp\. 1929–1958\.Cited by:[§1](https://arxiv.org/html/2606.09853#S1.p1.1)\.
- N\. Tishby, F\. C\. Pereira, and W\. Bialek \(2000\)The information bottleneck method\.arXiv preprint physics/0004057\.Cited by:[§B\.1](https://arxiv.org/html/2606.09853#A2.SS1.SSS0.Px2.p1.8),[§3\.4](https://arxiv.org/html/2606.09853#S3.SS4.p2.4)\.
- Y\. H\. Tsai, S\. Bai, P\. P\. Liang, J\. Z\. Kolter, L\. Morency, and R\. Salakhutdinov \(2019\)Multimodal transformer for unaligned multimodal language sequences\.InProceedings of the conference\. Association for computational linguistics\. Meeting,Vol\.2019,pp\. 6558\.Cited by:[§1](https://arxiv.org/html/2606.09853#S1.p2.1),[§2](https://arxiv.org/html/2606.09853#S2.SS0.SSS0.Px2.p1.1)\.
- V\. Vielzeuf, A\. Lechervy, S\. Pateux, and F\. Jurie \(2018\)Centralnet: a multilayer approach for multimodal fusion\.InProceedings of the European Conference on Computer Vision \(ECCV\) Workshops,pp\. 0–0\.Cited by:[§2](https://arxiv.org/html/2606.09853#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Wei and D\. Hu \(2024\)MMPareto: boosting multimodal learning with innocent unimodal assistance\.InProceedings of the 41st International Conference on Machine Learning,ICML’24\.Cited by:[Appendix J](https://arxiv.org/html/2606.09853#A10.SS0.SSS0.Px5),[§2](https://arxiv.org/html/2606.09853#S2.SS0.SSS0.Px3.p1.1),[§3\.1](https://arxiv.org/html/2606.09853#S3.SS1.p1.1),[Figure 5](https://arxiv.org/html/2606.09853#S4.F5),[Figure 5](https://arxiv.org/html/2606.09853#S4.F5.5.2),[§4\.1](https://arxiv.org/html/2606.09853#S4.SS1.SSS0.Px1.p1.1)\.
- Y\. Wei, S\. Li, R\. Feng, and D\. Hu \(2024\)Diagnosing and re\-learning for balanced multimodal learning\.InEuropean Conference on Computer Vision,pp\. 71–86\.Cited by:[Appendix J](https://arxiv.org/html/2606.09853#A10.SS0.SSS0.Px4),[§2](https://arxiv.org/html/2606.09853#S2.SS0.SSS0.Px3.p1.1),[Figure 5](https://arxiv.org/html/2606.09853#S4.F5),[Figure 5](https://arxiv.org/html/2606.09853#S4.F5.5.2),[§4\.1](https://arxiv.org/html/2606.09853#S4.SS1.SSS0.Px1.p1.1)\.
- L\. Wen, Q\. Dai, J\. Liu, J\. Zheng, Y\. Dai, D\. Wang, Z\. Kang, J\. Wang, Z\. Xu, and J\. Duan \(2025\)InfMasking: unleashing synergistic information by contrastive multimodal interactions\.arXiv preprint arXiv:2509\.25270\.Cited by:[§2](https://arxiv.org/html/2606.09853#S2.SS0.SSS0.Px1.p1.1)\.
- P\. L\. Williams and R\. D\. Beer \(2010\)Nonnegative decomposition of multivariate information\.arXiv preprint arXiv:1004\.2515\.Cited by:[§2](https://arxiv.org/html/2606.09853#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2606.09853#S3.SS0.SSS0.Px2.p1.17)\.
- J\. Xin, S\. Yun, J\. Peng, I\. Choi, J\. L\. Ballard, T\. Chen, and Q\. Long \(2025\)I2MoE: interpretable multimodal interaction\-aware mixture\-of\-experts\.arXiv preprint arXiv:2505\.19190\.Cited by:[§2](https://arxiv.org/html/2606.09853#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Yao and R\. Mihalcea \(2022\)Modality\-specific learning rates for effective multimodal additive late\-fusion\.InFindings of the Association for Computational Linguistics: ACL 2022,pp\. 1824–1834\.Cited by:[§2](https://arxiv.org/html/2606.09853#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Zadeh, M\. Chen, S\. Poria, E\. Cambria, and L\. Morency \(2017\)Tensor fusion network for multimodal sentiment analysis\.arXiv preprint arXiv:1707\.07250\.Cited by:[§1](https://arxiv.org/html/2606.09853#S1.p2.1),[§2](https://arxiv.org/html/2606.09853#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Zadeh, R\. Zellers, E\. Pincus, and L\. Morency \(2016\)Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos\.arXiv preprint arXiv:1606\.06259\.Cited by:[§4\.1](https://arxiv.org/html/2606.09853#S4.SS1.p1.2)\.

Supplementary Material

## Appendix ALimitations

We discuss four limitations of SynIB that we believe are important for interpreting our results and identifying directions for future work\. Limitations of the NTK\-based motivating analysis in Sec\.[3\.1](https://arxiv.org/html/2606.09853#S3.SS1)are discussed separately in App\.[K](https://arxiv.org/html/2606.09853#A11)\.

##### Dependence on the latent representation

SynIB operates on representations produced by modality\-specific encoders, and its effectiveness ultimately depends on whether synergistic features are expressible in the latent space those encoders produce\. When the encoders strip out cross\-modal structure during their forward pass, no loss\-level objective can recover it\. Self\-supervised pretrained encoders such as CLIP partially mitigate this by preserving richer per\-modality features, and our Hateful Memes experiments suggest that SynIB transfers cleanly to this setting\. A more systematic study of how pretraining choices affect the synergy a fusion model can ultimately learn remains open\.

##### Sensitivity of the learned mask

The adversarial maskMψM\_\{\\psi\}is trained jointly with the SynIB objective and depends on hyperparameters governing sparsity \(λM\\lambda\_\{M\}\), optimization dynamics, and the unimodal predictor used to define the adversary\. In our experiments these settings transferred reasonably across datasets without per\-task tuning, but in regimes with very small synergy subsets, mask training can become unstable and may collapse to either trivial \(no features kept\) or near\-identity \(all features kept\) solutions\. Random masking, while less precise, is more robust in these cases and provides a useful fallback\.

##### Variational surrogate rather than exact decomposition

Our objective is motivated by partial information decomposition, but we optimize a tractable variational surrogate of the Mutual Information Perturbed Difference rather than computing PID quantities directly\. The surrogate coincides with MIPD when the variational family contains the true predictive distributions, and tracks the same direction in the loss landscape more generally; in particular, it is not guaranteed to be a one\-sided bound on MIPD\. SynIB therefore pushes the model toward predictions that depend on cross\-modal interaction, but does not produce a formal decomposition ofI​\(Y;X1,X2\)I\(Y;X\_\{1\},X\_\{2\}\)into unique, redundant, and synergistic components\. The surrogate is sufficient for shaping training but does not, on its own, yield an explicit attribution of the learned model’s predictions to unique, redundant, and synergistic components; that attribution is a separate analytical question we leave to future work\.

##### Training\-time overhead

Each SynIB training step requires forward passes through the fusion model on both the intact inputs and one or more counterfactual inputs \(one per masked modality\), increasing per\-step compute by roughly1\+n1\{\+\}nwherennis the number of modalities\. The learned\-masking variant additionally trains the mask network jointly\. In our experiments this overhead was tolerable, but for very large backbones or high\-modality settings, more efficient approximations, for example sampling a subset of modalities to mask per batch, would be valuable\. We leave this to future work\.

### A\.1NTK Analysis with SynIB

Section[3\.1](https://arxiv.org/html/2606.09853#S3.SS1)used the empirical NTK to diagnose why synergistic examples fail to generalize under vanilla training: the largest per\-source learning signalλg\\lambda\_\{g\}is allocated to synergy and no destructive interference appears between PID sources, yet synergistic validation loss rises while training loss drops\. This appendix repeats the same diagnostics on a model trained with SynIB, verifying that the objective shifts these quantities in the direction the motivating analysis prescribes\.

##### Setup\.

We train on the PID\-Controlled XOR data \(App\.[F\.2](https://arxiv.org/html/2606.09853#A6.SS2)\) at composition\(pU1,pR,pS\)=\(0\.45,0\.45,0\.10\)\(p\_\{U\_\{1\}\},p\_\{R\},p\_\{S\}\)=\(0\.45,0\.45,0\.10\), identical to the vanilla run in Fig\.[1](https://arxiv.org/html/2606.09853#S3.F1), and add SynIB with the learned\-mask inner loop \(λ=10\\lambda=10\)\. Architecture, optimiser, batching, and seeds are unchanged\.

##### Findings\.

Three changes appear in Fig\.[9](https://arxiv.org/html/2606.09853#A1.F9), each consistent with the SynIB design\. First, the per\-source learning signal redistributes:λU1\\lambda\_\{U\_\{1\}\}andλR\\lambda\_\{R\}rise whileλS\\lambda\_\{S\}falls \(left\), so SynIB de\-emphasizes the scarce synergistic examples that were being memorized under vanilla without starving any source\. Second, gradient alignment between sources remains near zero throughout training \(center\), confirming that the redistribution does not introduce destructive competition\. Third, and most importantly, the train–validation gap on the synergistic source closes \(right\): BCE on synergistic train and validation now track each other, in contrast to the diverging curves under vanilla\. Combined with test accuracy of88\.6%88\.6\\%on the synergy slice \(98\.3%98\.3\\%overall\), these findings indicate that SynIB reshapes the per\-source signal in the way the motivating analysis prescribes, rather than simply injecting noise into training\.

![Refer to caption](https://arxiv.org/html/2606.09853v1/x7.png)Figure 9:Gradient geometry across PID sources under SynIB\.Training augments the vanilla setup with the SynIB learned\-mask inner loop \(λ=10\\lambda\{=\}10\); data, architecture, and optimiser are otherwise identical to Fig\.[1](https://arxiv.org/html/2606.09853#S3.F1)\.Left:per\-source learning signal strengthλg\\lambda\_\{g\}over training\. SynIB redistributes mass towardU1U\_\{1\}andRR, leaving synergy with the smallest share without starving it\.Center:gradient alignment between source pairs stays near zero throughout training, ruling out destructive competition\.Right:fusion BCE by source on intact inputs \(solid train, dashed validation\)\. Train and validation curves track closely on every source; the synergistic train–validation gap that opened under vanilla is now closed\. Test accuracy: synergy88\.6%88\.6\\%, total98\.3%98\.3\\%\.

## Appendix BVariational Surrogate for MIPD

### B\.1Derivation of the SynIB Surrogate

This appendix derives the SynIB training objective as a tractable surrogate forMIPD1=H​\(Y∣X~1,X2\)−H​\(Y∣X1,X2\)\\mathrm\{MIPD\}\_\{1\}=H\(Y\\mid\\tilde\{X\}\_\{1\},X\_\{2\}\)\-H\(Y\\mid X\_\{1\},X\_\{2\}\)\. The two conditional entropies in this expression are not directly computable for high\-dimensional inputs, since they require access to the true predictive distributionsp​\(y∣x1,x2\)p\(y\\mid x\_\{1\},x\_\{2\}\)andp​\(y∣x~1,x2\)p\(y\\mid\\tilde\{x\}\_\{1\},x\_\{2\}\)\. We approximate each term using the variational distributionqθ​\(y∣⋅\)q\_\{\\theta\}\(y\\mid\\cdot\)produced by the model, withr​\(y\)r\(y\)denoting a fixed reference distribution over𝒴\\mathcal\{Y\}chosen independently of the inputs and of the model parameters\.

##### Upper bound onH​\(Y∣X1,X2\)H\(Y\\mid X\_\{1\},X\_\{2\}\)via cross\-entropy\.

For the entropy under intact inputs, the standard variational upper bound due toBarber and Agakov \([2004](https://arxiv.org/html/2606.09853#bib.bib118)\)applies\. For any variational distributionqθ​\(y∣x1,x2\)q\_\{\\theta\}\(y\\mid x\_\{1\},x\_\{2\}\),

𝔼p​\(x1,x2,y\)​\[−log⁡qθ​\(y∣x1,x2\)\]\\displaystyle\\mathbb\{E\}\_\{p\(x\_\{1\},x\_\{2\},y\)\}\[\-\\log q\_\{\\theta\}\(y\\mid x\_\{1\},x\_\{2\}\)\]=H\(Y∣X1,X2\)\+𝔼p​\(x1,x2\)DKL\(p\(⋅∣x1,x2\)∥qθ\(⋅∣x1,x2\)\),\\displaystyle=H\(Y\\mid X\_\{1\},X\_\{2\}\)\+\\mathbb\{E\}\_\{p\(x\_\{1\},x\_\{2\}\)\}\\,D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\(\\cdot\\mid x\_\{1\},x\_\{2\}\)\\,\\\|\\,q\_\{\\theta\}\(\\cdot\\mid x\_\{1\},x\_\{2\}\)\\right\),\(7\)which follows directly from expanding the cross\-entropy and identifying the KL divergence between the true and variational conditionals\. SinceDKL≥0D\_\{\\mathrm\{KL\}\}\\geq 0, this yields

H​\(Y∣X1,X2\)≤𝔼p​\(x1,x2,y\)​\[−log⁡qθ​\(y∣x1,x2\)\],\\displaystyle H\(Y\\mid X\_\{1\},X\_\{2\}\)\\leq\\mathbb\{E\}\_\{p\(x\_\{1\},x\_\{2\},y\)\}\[\-\\log q\_\{\\theta\}\(y\\mid x\_\{1\},x\_\{2\}\)\],\(8\)with equality if and only ifqθ\(⋅∣x1,x2\)=p\(⋅∣x1,x2\)q\_\{\\theta\}\(\\cdot\\mid x\_\{1\},x\_\{2\}\)=p\(\\cdot\\mid x\_\{1\},x\_\{2\}\)almost everywhere\. The bound’s tightness is governed by the variational approximation error𝔼p​\(x1,x2\)​DKL​\(p∥qθ\)\\mathbb\{E\}\_\{p\(x\_\{1\},x\_\{2\}\)\}D\_\{\\mathrm\{KL\}\}\(p\\,\\\|\\,q\_\{\\theta\}\), which the cross\-entropy training objective itself minimizes\.

##### Reference\-based identity forH​\(Y∣X~1,X2\)H\(Y\\mid\\tilde\{X\}\_\{1\},X\_\{2\}\)\.

For the entropy under modality corruption, we use an exact identity that re\-expresses conditional entropy in terms of a divergence to the referencerr\. Starting from the definition of conditional entropy and adding and subtractinglog⁡r​\(y\)\\log r\(y\),

H​\(Y∣X~1,X2\)\\displaystyle H\(Y\\mid\\tilde\{X\}\_\{1\},X\_\{2\}\)=−𝔼p​\(x~1,x2,y\)​\[log⁡p​\(y∣x~1,x2\)\]\\displaystyle=\-\\mathbb\{E\}\_\{p\(\\tilde\{x\}\_\{1\},x\_\{2\},y\)\}\[\\log p\(y\\mid\\tilde\{x\}\_\{1\},x\_\{2\}\)\]=−𝔼p​\(x~1,x2,y\)​\[log⁡p​\(y∣x~1,x2\)r​\(y\)\]−𝔼p​\(y\)​\[log⁡r​\(y\)\]\\displaystyle=\-\\mathbb\{E\}\_\{p\(\\tilde\{x\}\_\{1\},x\_\{2\},y\)\}\\\!\\left\[\\log\\frac\{p\(y\\mid\\tilde\{x\}\_\{1\},x\_\{2\}\)\}\{r\(y\)\}\\right\]\-\\mathbb\{E\}\_\{p\(y\)\}\[\\log r\(y\)\]=Cr−𝔼p​\(x~1,x2\)DKL\(p\(⋅∣x~1,x2\)∥r\(⋅\)\),\\displaystyle=C\_\{r\}\-\\mathbb\{E\}\_\{p\(\\tilde\{x\}\_\{1\},x\_\{2\}\)\}\\,D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\(\\cdot\\mid\\tilde\{x\}\_\{1\},x\_\{2\}\)\\,\\\|\\,r\(\\cdot\)\\right\),\(9\)whereCr:=𝔼p​\(y\)​\[−log⁡r​\(y\)\]C\_\{r\}:=\\mathbb\{E\}\_\{p\(y\)\}\[\-\\log r\(y\)\]is the cross\-entropy of the marginalp​\(y\)p\(y\)relative torrand is constant with respect toθ\\theta\. This identity, used in the Information Bottleneck framework\(Tishbyet al\.,[2000](https://arxiv.org/html/2606.09853#bib.bib73); Alemiet al\.,[2016](https://arxiv.org/html/2606.09853#bib.bib75)\), expresses the conditional entropy as a constant minus the average divergence between the true conditional and the reference: whenp\(⋅∣x~1,x2\)p\(\\cdot\\mid\\tilde\{x\}\_\{1\},x\_\{2\}\)is far fromrr, the conditional entropy is small \(predictions are sharp\), and conversely\.

##### Variational surrogate\.

Eq\. \([9](https://arxiv.org/html/2606.09853#A2.E9)\) is exact but not directly usable, since the KL divergence involves the unknownp\(⋅∣x~1,x2\)p\(\\cdot\\mid\\tilde\{x\}\_\{1\},x\_\{2\}\)\. We obtain a tractable expression by replacing the true conditional with the variationalqθ\(⋅∣x~1,x2\)q\_\{\\theta\}\(\\cdot\\mid\\tilde\{x\}\_\{1\},x\_\{2\}\):

H^θ​\(Y∣X~1,X2\)\\displaystyle\\widehat\{H\}\_\{\\theta\}\(Y\\mid\\tilde\{X\}\_\{1\},X\_\{2\}\):=Cr−𝔼p​\(x~1,x2\)DKL\(qθ\(⋅∣x~1,x2\)∥r\(⋅\)\)\.\\displaystyle:=C\_\{r\}\-\\mathbb\{E\}\_\{p\(\\tilde\{x\}\_\{1\},x\_\{2\}\)\}\\,D\_\{\\mathrm\{KL\}\}\\\!\\left\(q\_\{\\theta\}\(\\cdot\\mid\\tilde\{x\}\_\{1\},x\_\{2\}\)\\,\\\|\\,r\(\\cdot\)\\right\)\.\(10\)The substitution defines a surrogate forH​\(Y∣X~1,X2\)H\(Y\\mid\\tilde\{X\}\_\{1\},X\_\{2\}\)that is computable from the model’s predictions on samples drawn fromp​\(x~1,x2\)p\(\\tilde\{x\}\_\{1\},x\_\{2\}\)\. By construction,H^θ​\(Y∣X~1,X2\)=H​\(Y∣X~1,X2\)\\widehat\{H\}\_\{\\theta\}\(Y\\mid\\tilde\{X\}\_\{1\},X\_\{2\}\)=H\(Y\\mid\\tilde\{X\}\_\{1\},X\_\{2\}\)exactly whenqθ\(⋅∣x~1,x2\)=p\(⋅∣x~1,x2\)q\_\{\\theta\}\(\\cdot\\mid\\tilde\{x\}\_\{1\},x\_\{2\}\)=p\(\\cdot\\mid\\tilde\{x\}\_\{1\},x\_\{2\}\)\.

##### Combining the two terms\.

Substituting Eq\. \([8](https://arxiv.org/html/2606.09853#A2.E8)\) forH​\(Y∣X1,X2\)H\(Y\\mid X\_\{1\},X\_\{2\}\)and Eq\. \([10](https://arxiv.org/html/2606.09853#A2.E10)\) forH​\(Y∣X~1,X2\)H\(Y\\mid\\tilde\{X\}\_\{1\},X\_\{2\}\)inMIPD1\\mathrm\{MIPD\}\_\{1\}yields the SynIB surrogate

MIPD^1\\displaystyle\\widehat\{\\mathrm\{MIPD\}\}\_\{1\}=H^θ​\(Y∣X~1,X2\)−𝔼p​\(x1,x2,y\)​\[−log⁡qθ​\(y∣x1,x2\)\]\\displaystyle=\\widehat\{H\}\_\{\\theta\}\(Y\\mid\\tilde\{X\}\_\{1\},X\_\{2\}\)\-\\mathbb\{E\}\_\{p\(x\_\{1\},x\_\{2\},y\)\}\[\-\\log q\_\{\\theta\}\(y\\mid x\_\{1\},x\_\{2\}\)\]=Cr−𝔼p​\(x~1,x2\)DKL\(qθ\(⋅∣x~1,x2\)∥r\(⋅\)\)−𝔼p​\(x1,x2,y\)\[−logqθ\(y∣x1,x2\)\]\.\\displaystyle=C\_\{r\}\-\\mathbb\{E\}\_\{p\(\\tilde\{x\}\_\{1\},x\_\{2\}\)\}\\,D\_\{\\mathrm\{KL\}\}\\\!\\left\(q\_\{\\theta\}\(\\cdot\\mid\\tilde\{x\}\_\{1\},x\_\{2\}\)\\,\\\|\\,r\(\\cdot\)\\right\)\-\\mathbb\{E\}\_\{p\(x\_\{1\},x\_\{2\},y\)\}\[\-\\log q\_\{\\theta\}\(y\\mid x\_\{1\},x\_\{2\}\)\]\.\(11\)Up to the additive constantCrC\_\{r\}, maximizingMIPD^1\\widehat\{\\mathrm\{MIPD\}\}\_\{1\}is equivalent to minimizing the SynIB loss: jointly minimizing the cross\-entropy on intact inputs and the KL\-to\-reference on corrupted inputs\.

##### Properties of the surrogate\.

Three observations clarify howMIPD^1\\widehat\{\\mathrm\{MIPD\}\}\_\{1\}relates to the trueMIPD1\\mathrm\{MIPD\}\_\{1\}\.

First,MIPD^1=MIPD1\\widehat\{\\mathrm\{MIPD\}\}\_\{1\}=\\mathrm\{MIPD\}\_\{1\}wheneverqθ\(⋅∣x1,x2\)=p\(⋅∣x1,x2\)q\_\{\\theta\}\(\\cdot\\mid x\_\{1\},x\_\{2\}\)=p\(\\cdot\\mid x\_\{1\},x\_\{2\}\)andqθ\(⋅∣x~1,x2\)=p\(⋅∣x~1,x2\)q\_\{\\theta\}\(\\cdot\\mid\\tilde\{x\}\_\{1\},x\_\{2\}\)=p\(\\cdot\\mid\\tilde\{x\}\_\{1\},x\_\{2\}\)\. In this case both the upper bound in Eq\. \([8](https://arxiv.org/html/2606.09853#A2.E8)\) and the surrogate substitution in Eq\. \([10](https://arxiv.org/html/2606.09853#A2.E10)\) are tight\.

Second, the cross\-entropy term inMIPD^1\\widehat\{\\mathrm\{MIPD\}\}\_\{1\}is itself the optimization signal that drivesqθ\(⋅∣x1,x2\)q\_\{\\theta\}\(\\cdot\\mid x\_\{1\},x\_\{2\}\)towardp\(⋅∣x1,x2\)p\(\\cdot\\mid x\_\{1\},x\_\{2\}\), so the surrogate is self\-correcting on the intact\-input branch: improvements inqθq\_\{\\theta\}tighten the bound in Eq\. \([8](https://arxiv.org/html/2606.09853#A2.E8)\)\.

Third, the surrogate substitution in Eq\. \([10](https://arxiv.org/html/2606.09853#A2.E10)\) is not a one\-sided bound: the gapH^θ​\(Y∣X~1,X2\)−H​\(Y∣X~1,X2\)=𝔼p​\(x~1,x2\)​\[DKL​\(p∥r\)−DKL​\(qθ∥r\)\]\\widehat\{H\}\_\{\\theta\}\(Y\\mid\\tilde\{X\}\_\{1\},X\_\{2\}\)\-H\(Y\\mid\\tilde\{X\}\_\{1\},X\_\{2\}\)=\\mathbb\{E\}\_\{p\(\\tilde\{x\}\_\{1\},x\_\{2\}\)\}\[D\_\{\\mathrm\{KL\}\}\(p\\,\\\|\\,r\)\-D\_\{\\mathrm\{KL\}\}\(q\_\{\\theta\}\\,\\\|\\,r\)\]can take either sign depending on the relative position ofqθq\_\{\\theta\}andppwith respect torr\. The surrogate should therefore be understood as a tractable target whose sign\-correct alignment withMIPD1\\mathrm\{MIPD\}\_\{1\}relies onqθq\_\{\\theta\}approximatingppin distribution, which the cross\-entropy term promotes during training\.

### B\.2MIPD as a Proxy for CMI

Beyond serving as the optimization target,MIPD1\\mathrm\{MIPD\}\_\{1\}admits a clean information\-theoretic interpretation as a lower\-bounded proxy function for the conditional mutual informationI​\(X1;Y∣X2\)I\(X\_\{1\};Y\\mid X\_\{2\}\)\. Expanding the definition of MIPD in terms of conditional mutual informations givesMIPD1=I​\(X1;Y∣X2\)−I​\(X~1;Y∣X2\)\\mathrm\{MIPD\}\_\{1\}=I\(X\_\{1\};Y\\mid X\_\{2\}\)\-I\(\\tilde\{X\}\_\{1\};Y\\mid X\_\{2\}\), and sinceI​\(X~1;Y∣X2\)≥0I\(\\tilde\{X\}\_\{1\};Y\\mid X\_\{2\}\)\\geq 0for any random variableX~1\\tilde\{X\}\_\{1\}, it follows directly thatMIPD1≤I​\(X1;Y∣X2\)\\mathrm\{MIPD\}\_\{1\}\\leq I\(X\_\{1\};Y\\mid X\_\{2\}\), with the gap given exactly by the residual conditional information retained after perturbation:

I​\(X1;Y∣X2\)−MIPD1=I​\(X~1;Y∣X2\)\.I\(X\_\{1\};Y\\mid X\_\{2\}\)\-\\mathrm\{MIPD\}\_\{1\}=I\(\\tilde\{X\}\_\{1\};Y\\mid X\_\{2\}\)\.\(12\)This identity has a clear interpretation: MIPD recovers the full CMI exactly when the perturbationX~1\\tilde\{X\}\_\{1\}is conditionally uninformative aboutYYgivenX2X\_\{2\}, i\.e\., whenY⟂⟂X~1∣X2Y\\perp\\\!\\\!\\\!\\perp\\tilde\{X\}\_\{1\}\\mid X\_\{2\}\. In this case, all task\-relevant information thatX1X\_\{1\}contributed beyondX2X\_\{2\}has been removed by the corruption, and the drop in predictabilityMIPD1\\mathrm\{MIPD\}\_\{1\}equals the conditional informationX1X\_\{1\}originally carried\. More generally, if the corruption leaves residual informationI​\(X~1;Y∣X2\)≤εI\(\\tilde\{X\}\_\{1\};Y\\mid X\_\{2\}\)\\leq\\varepsilon, then MIPD approximates CMI to withinε\\varepsilon\.

The proxy quality thus depends entirely on how thoroughly the perturbation removes task\-relevant information fromX1X\_\{1\}beyond whatX2X\_\{2\}already provides\. DesigningX~1\\tilde\{X\}\_\{1\}to make this residual small motivates the masking constructions in Sec\.[3\.5](https://arxiv.org/html/2606.09853#S3.SS5): random masking corrupts coordinates uniformly, while learned masking targets the features the unimodal predictor relies on, both aiming to driveI​\(X~1;Y∣X2\)I\(\\tilde\{X\}\_\{1\};Y\\mid X\_\{2\}\)as close to zero as possible while preserving the rest of the distribution\.

## Appendix CVariational Families and Closed\-Form KL Divergences

The SynIB objective requires evaluating−log⁡qθ​\(y∣x1,x2\)\-\\log q\_\{\\theta\}\(y\\mid x\_\{1\},x\_\{2\}\)for the cross\-entropy term andDKL\(qθ\(⋅∣x~1,x2\)∥r\(⋅\)\)D\_\{\\mathrm\{KL\}\}\(q\_\{\\theta\}\(\\cdot\\mid\\tilde\{x\}\_\{1\},x\_\{2\}\)\\,\\\|\\,r\(\\cdot\)\)for the KL term\. Both quantities admit closed\-form expressions for standard variational families, removing the need for stochastic estimators\. We use the same family for both intact and corrupted inputs, with the conditioning input being the only difference between the two passes\.

### C\.1Gaussian Variational Family

For continuous targetsY∈ℝdY\\in\\mathbb\{R\}^\{d\}, we model the variational predictive distribution asqθ​\(y∣x~1,x2\)=𝒩​\(y∣μθ,Σθ\)q\_\{\\theta\}\(y\\mid\\tilde\{x\}\_\{1\},x\_\{2\}\)=\\mathcal\{N\}\(y\\mid\\mu\_\{\\theta\},\\Sigma\_\{\\theta\}\), with reference distributionr​\(y\)=𝒩​\(0,I\)r\(y\)=\\mathcal\{N\}\(0,I\)\. The meanμθ\\mu\_\{\\theta\}and covarianceΣθ\\Sigma\_\{\\theta\}are outputs of the variational decoder and depend on\(x~1,x2\)\(\\tilde\{x\}\_\{1\},x\_\{2\}\)\. The KL divergence admits the closed form

DKL​\(qθ∥r\)=12​\(‖μθ‖22\+tr​Σθ−log​detΣθ−d\),D\_\{\\mathrm\{KL\}\}\(q\_\{\\theta\}\\,\\\|\\,r\)=\\tfrac\{1\}\{2\}\\\!\\left\(\\\|\\mu\_\{\\theta\}\\\|\_\{2\}^\{2\}\+\\mathrm\{tr\}\\,\\Sigma\_\{\\theta\}\-\\log\\det\\Sigma\_\{\\theta\}\-d\\right\),\(13\)which is minimized whenqθ=rq\_\{\\theta\}=r\(i\.e\.,μθ=0\\mu\_\{\\theta\}=0,Σθ=I\\Sigma\_\{\\theta\}=I\) and grows with both the magnitude ofμθ\\mu\_\{\\theta\}and the deviation ofΣθ\\Sigma\_\{\\theta\}from the identity, providing a differentiable uncertainty penalty under modality corruption\.

### C\.2Categorical Variational Family

For categorical targetsY∈\{1,…,K\}Y\\in\\\{1,\\dots,K\\\}, we model the variational predictive distribution asqθ​\(y∣x~1,x2\)=Categorical​\(y;πθ\)q\_\{\\theta\}\(y\\mid\\tilde\{x\}\_\{1\},x\_\{2\}\)=\\mathrm\{Categorical\}\(y;\\pi\_\{\\theta\}\), with referencer​\(y\)=Categorical​\(y;ρ\)r\(y\)=\\mathrm\{Categorical\}\(y;\\rho\), whereπθ=πθ​\(x~1,x2\)∈ΔK−1\\pi\_\{\\theta\}=\\pi\_\{\\theta\}\(\\tilde\{x\}\_\{1\},x\_\{2\}\)\\in\\Delta^\{K\-1\}is the output of the variational decoder andρ∈ΔK−1\\rho\\in\\Delta^\{K\-1\}is a fixed reference distribution \(we useρk=1/K\\rho\_\{k\}=1/K, the uniform prior, in all experiments\)\. The KL divergence admits the closed form

DKL​\(qθ∥r\)=∑k=1Kπθ,k​log⁡πθ,kρk\.D\_\{\\mathrm\{KL\}\}\(q\_\{\\theta\}\\,\\\|\\,r\)=\\sum\_\{k=1\}^\{K\}\\pi\_\{\\theta,k\}\\log\\frac\{\\pi\_\{\\theta,k\}\}\{\\rho\_\{k\}\}\.\(14\)The penalty is minimized whenπθ=ρ\\pi\_\{\\theta\}=\\rhoand grows asπθ\\pi\_\{\\theta\}concentrates on any single class, discouraging confident predictions under modality corruption\. The binary caseK=2K=2recovers the Bernoulli formDKL​\(qθ∥r\)=πθ​log⁡\(πθ/ρ\)\+\(1−πθ\)​log⁡\(\(1−πθ\)/\(1−ρ\)\)D\_\{\\mathrm\{KL\}\}\(q\_\{\\theta\}\\,\\\|\\,r\)=\\pi\_\{\\theta\}\\log\(\\pi\_\{\\theta\}/\\rho\)\+\(1\-\\pi\_\{\\theta\}\)\\log\(\(1\-\\pi\_\{\\theta\}\)/\(1\-\\rho\)\), used for tasks with binary targets\.

## Appendix DMasking Operator Instantiations

This appendix specifies the concrete realizations of the masking operator𝒯​\(⋅;M\)\\mathcal\{T\}\(\\cdot;\\,M\)introduced in Sec\.[3\.5](https://arxiv.org/html/2606.09853#S3.SS5)\. All variants implement the same semantics, selectively corrupting a subset of representation coordinates while leaving the rest intact, and differ only in how the maskMMis constructed and how the suppression is realized within a given architecture\. Throughout this appendix we follow the convention from the main text:Mj=1M\_\{j\}=1denotes a corrupted coordinate,Mj=0M\_\{j\}=0a preserved one\.

##### Feature replacement via gating\.

In non\-attention\-based fusion networks, masking is implemented by replacing corrupted coordinates with noise through the feature\-wise gate

x~1=\(1−M\)⊙x1\+M⊙ϵ,\\tilde\{x\}\_\{1\}=\(1\-M\)\\odot x\_\{1\}\+M\\odot\\epsilon,\(15\)whereM∈\{0,1\}d1M\\in\\\{0,1\\\}^\{d\_\{1\}\}andϵ\\epsilonis sampled independently of the targetYY\. The construction preserves dimensionality and, by design, removes the task\-relevant signal carried by the corrupted features while leaving the rest of the input distribution intact\. The same expression applies when masking is performed in the latent space: substitutingz1=fθc1​\(x1\)z\_\{1\}=f\_\{\\theta\_\{c\_\{1\}\}\}\(x\_\{1\}\)forx1x\_\{1\}yields the corresponding latent\-space gate\.

##### Oracle masking\.

In synthetic settings where the synergistic coordinates ofX1X\_\{1\}are known by construction, we define an oracle mask

Mj⋆=𝟏​\[j∈Ssyn\],M^\{\\star\}\_\{j\}=\\mathbf\{1\}\[j\\in S\_\{\\mathrm\{syn\}\}\],\(16\)which corrupts exactly the designated synergistic features and preserves everything else\. Applied through Eq\. \([15](https://arxiv.org/html/2606.09853#A4.E15)\), this isolates the contribution of synergy by removing only the coordinates whose information is recoverable from the joint\(X1,X2\)\(X\_\{1\},X\_\{2\}\)but not from either modality alone\. Oracle masking is used exclusively for diagnostic experiments and serves as an upper\-bound reference point: it characterizes how SynIB would behave if it could perfectly identify synergistic features, without claiming that this information is available in real\-world settings\.

##### Random masking\.

For stochastic masking, each coordinate is independently corrupted according toMj∼Bernoulli​\(π\)M\_\{j\}\\sim\\mathrm\{Bernoulli\}\(\\pi\), whereπ∈\(0,1\)\\pi\\in\(0,1\)controls the expected corruption level\. Random masking makes no assumption about which coordinates carry which type of information, providing a task\-agnostic baseline that removes signal uniformly across dimensions and surfaces synergy whenever any corrupted subset breaks the joint prediction\.

To avoid introducing out\-of\-distribution artifacts when corrupting, the noiseϵ\\epsilonin Eq\. \([15](https://arxiv.org/html/2606.09853#A4.E15)\) is sampled to match the marginal statistics of the representation being corrupted\. Letz1=fθc1​\(x1\)z\_\{1\}=f\_\{\\theta\_\{c\_\{1\}\}\}\(x\_\{1\}\)and maintain exponential moving averages over minibatchesℬ\\mathcal\{B\}:

μt\\displaystyle\\mu\_\{t\}=α​μt−1\+\(1−α\)​𝔼ℬ​\[z1\],\\displaystyle=\\alpha\\mu\_\{t\-1\}\+\(1\-\\alpha\)\\,\\mathbb\{E\}\_\{\\mathcal\{B\}\}\[z\_\{1\}\],\(17\)σt2\\displaystyle\\sigma\_\{t\}^\{2\}=α​σt−12\+\(1−α\)​𝔼ℬ​\[\(z1−𝔼ℬ​\[z1\]\)2\]\.\\displaystyle=\\alpha\\sigma\_\{t\-1\}^\{2\}\+\(1\-\\alpha\)\\,\\mathbb\{E\}\_\{\\mathcal\{B\}\}\\\!\\left\[\(z\_\{1\}\-\\mathbb\{E\}\_\{\\mathcal\{B\}\}\[z\_\{1\}\]\)^\{2\}\\right\]\.Noise is then sampled asϵ∼𝒩​\(μt,diag​\(σt2\)\)\\epsilon\\sim\\mathcal\{N\}\(\\mu\_\{t\},\\mathrm\{diag\}\(\\sigma\_\{t\}^\{2\}\)\)\. The momentumα\\alphacontrols the smoothness of the running statistics; we useα=0\.99\\alpha=0\.99in all experiments\. To prevent the model from exploiting structure in the replacement distribution, the same noise process is applied to fully corrupted counterfactual inputs during training\.

##### Learned masking\.

The learned maskMψ1M\_\{\\psi\_\{1\}\}introduced in Sec\.[3\.5](https://arxiv.org/html/2606.09853#S3.SS5)flags the minimum set of coordinates whose corruption collapses unimodal prediction\. TrainingMψ1M\_\{\\psi\_\{1\}\}adversarially against the unimodal predictorfθc1f\_\{\\theta\_\{c\_\{1\}\}\}via Eq\. \([5](https://arxiv.org/html/2606.09853#S3.E5)\) therefore identifies the unimodally important features ofX1X\_\{1\}\. The SynIB counterfactual then uses the complementM¯ψ1=𝟏−Mψ1\\bar\{M\}\_\{\\psi\_\{1\}\}=\\mathbf\{1\}\-M\_\{\\psi\_\{1\}\}as its corruption mask in Eq\. \([15](https://arxiv.org/html/2606.09853#A4.E15)\), leaving the unimodally important coordinates intact and corrupting the rest\. Confidence under this corruption signals reliance on cues that survive the removal ofX1X\_\{1\}’s non\-unimodal signal, which the SynIB KL term penalizes\. The mask networkMψ1M\_\{\\psi\_\{1\}\}is parameterized as a small MLP producing logitsℓ∈ℝd1\\ell\\in\\mathbb\{R\}^\{d\_\{1\}\}, with binary samples drawn through a Gumbel\-softmax relaxation\(Janget al\.,[2016](https://arxiv.org/html/2606.09853#bib.bib134)\)to enable gradient flow during joint training\.

## Appendix ELikelihood Perspective on Learned Masking

Learned masking can be interpreted as a structured perturbation of the conditional likelihood implicitly defined by the unimodal predictorfθc1f\_\{\\theta\_\{c\_\{1\}\}\}\. This appendix makes the connection explicit and clarifies how the learned mask differs from the fixed alternatives \(random, oracle\)\.

Cross\-entropy training offθc1f\_\{\\theta\_\{c\_\{1\}\}\}implicitly defines a conditional likelihoodpθc1​\(y∣z1\)p\_\{\\theta\_\{c\_\{1\}\}\}\(y\\mid z\_\{1\}\)over the target given the modality\-1 representation, with the cross\-entropy loss equal to its negative log\-likelihood under the data distribution:

CE​\(fθc1​\(z1\),y\)=−𝔼p​\(y,z1\)​\[log⁡pθc1​\(y∣z1\)\]\.\\mathrm\{CE\}\\\!\\left\(f\_\{\\theta\_\{c\_\{1\}\}\}\(z\_\{1\}\),\\,y\\right\)=\-\\mathbb\{E\}\_\{p\(y,z\_\{1\}\)\}\\\!\\left\[\\log p\_\{\\theta\_\{c\_\{1\}\}\}\(y\\mid z\_\{1\}\)\\right\]\.\(18\)Perturbations of the representation that increase cross\-entropy therefore correspond directly to degradations of this conditional log\-likelihood\.

The learned maskMψ1M\_\{\\psi\_\{1\}\}exploits this relationship by seeking the sparsest perturbation that maximally degrades the likelihood\. WithZ~1=𝒯​\(Z1;Mψ1\)\\tilde\{Z\}\_\{1\}=\\mathcal\{T\}\(Z\_\{1\};\\,M\_\{\\psi\_\{1\}\}\)denoting the gated representation under the new convention \(Mψ1,j=1M\_\{\\psi\_\{1\},j\}=1corrupts coordinatejj,Mψ1,j=0M\_\{\\psi\_\{1\},j\}=0preserves it\), the mask training objective in Eq\. \([5](https://arxiv.org/html/2606.09853#S3.E5)\) can be equivalently written as

ψ1⋆=arg⁡maxψ1⁡𝔼p​\(y,z1\)​\[log⁡pθc1​\(y∣Z1\)pθc1​\(y∣Z~1\)\]−λM​‖Mψ1‖1,\\psi\_\{1\}^\{\\star\}=\\arg\\max\_\{\\psi\_\{1\}\}\\;\\mathbb\{E\}\_\{p\(y,z\_\{1\}\)\}\\\!\\left\[\\log\\frac\{p\_\{\\theta\_\{c\_\{1\}\}\}\(y\\mid Z\_\{1\}\)\}\{p\_\{\\theta\_\{c\_\{1\}\}\}\(y\\mid\\tilde\{Z\}\_\{1\}\)\}\\right\]\-\\lambda\_\{M\}\\\|M\_\{\\psi\_\{1\}\}\\\|\_\{1\},\(19\)where the first term measures the average log\-likelihood drop induced by the perturbation and the second imposes anℓ1\\ell\_\{1\}relaxation of anℓ0\\ell\_\{0\}sparsity constraint on the corruption set\. Solutions concentrate corruption on the minimal subset of representation coordinates whose removal most degrades the unimodal likelihood, identifying the featuresfθc1f\_\{\\theta\_\{c\_\{1\}\}\}relies on for prediction\.

This view places the three masking strategies on a common axis\. Random masking \(Mj∼Bernoulli​\(π\)M\_\{j\}\\sim\\mathrm\{Bernoulli\}\(\\pi\)\) corrupts coordinates uniformly without consulting the likelihood\. Oracle masking corrupts a fixed setSsynS\_\{\\mathrm\{syn\}\}defined by ground\-truth synergy structure available only in synthetic settings\. Learned masking adaptively concentrates corruption on the coordinates that most reduce the likelihood offθc1f\_\{\\theta\_\{c\_\{1\}\}\}, which we then complement in the SynIB counterfactual to leave only the unimodally\-important features intact\.

## Appendix FAdditional Synthetic Experiment Details

We use two synthetic XOR datasets, each targeting a different aspect of SynIB’s behavior\. Sec\.[F\.1](https://arxiv.org/html/2606.09853#A6.SS1)\(*Spurious XOR*\) tests robustness when one modality contains a label\-correlated shortcut whose strength sweeps continuously between absent and dominant\. Sec\.[F\.2](https://arxiv.org/html/2606.09853#A6.SS2)\(*PID\-Controlled XOR*\) generates examples whose label is determined by a single PID source per sample, allowing the dataset’s mixture of unique, redundant, and synergistic information to be controlled directly\. Both datasets use the modality\-indexing convention from the main text \(X1,X2X\_\{1\},X\_\{2\}\) and the masking conventionMj=1M\_\{j\}=1corrupts,Mj=0M\_\{j\}=0preserves\.

### F\.1Spurious XOR

#### F\.1\.1Task and Data Generation

The Spurious XOR task is a bimodal binary classification problem in which neither modality alone is informative aboutyy, but their XOR is\. Latent bitsb1,b2∼Bernoulli​\(0\.5\)b\_\{1\},b\_\{2\}\\sim\\mathrm\{Bernoulli\}\(0\.5\)are sampled independently and the label is set toy=b1⊕b2y=b\_\{1\}\\oplus b\_\{2\}, with modalityXmX\_\{m\}encoding bitbmb\_\{m\}\. ModalityX1X\_\{1\}additionally contains a spurious feature whose correlation withyyduring training is controlled by a parameterβ\\betaand is removed at test time, so a model that latches onto the shortcut fails to generalize\.

Both modalities are 64\-dimensional and partitioned into disjoint coordinate blocks\. Modality 1 contains a 4\-coordinate signal block at\[0,4\)\[0,4\)encodingb1b\_\{1\}, a 6\-coordinate spurious block at\[4,10\)\[4,10\)correlated withyyat training time, and 54 pure\-noise coordinates at\[10,64\)\[10,64\)\. Modality 2 contains a 4\-coordinate signal block at\[0,4\)\[0,4\)encodingb2b\_\{2\}and 60 pure\-noise coordinates at\[4,64\)\[4,64\)\. Pure\-noise coordinates are sampled as𝒩​\(0,1\)\\mathcal\{N\}\(0,1\)\.

Both signal blocks are mean\-shifted Gaussians,Xmsig=μ⋅\(2​bm−1\)⋅𝟏4\+σsig​εX\_\{m\}^\{\\mathrm\{sig\}\}=\\mu\\cdot\(2b\_\{m\}\-1\)\\cdot\\mathbf\{1\}\_\{4\}\+\\sigma\_\{\\mathrm\{sig\}\}\\,\\varepsilonwithε∼𝒩​\(0,I4\)\\varepsilon\\sim\\mathcal\{N\}\(0,I\_\{4\}\),μ=2\.2\\mu=2\.2, andσsig=0\.55\\sigma\_\{\\mathrm\{sig\}\}=0\.55\. The spurious block inX1X\_\{1\}is a continuous mean\-shifted Gaussian whose label\-correlation strength is controlled byβ≥0\\beta\\geq 0:

X1spur=1\.3⋅\[β⋅\(2​yeff−1\)⋅𝟏6\+max⁡\(0,1−β2\)⋅ε\],ε∼𝒩​\(0,I6\),X\_\{1\}^\{\\mathrm\{spur\}\}=1\.3\\cdot\\left\[\\,\\beta\\cdot\(2y\_\{\\mathrm\{eff\}\}\-1\)\\cdot\\mathbf\{1\}\_\{6\}\+\\sqrt\{\\max\(0,1\-\\beta^\{2\}\)\}\\cdot\\varepsilon\\,\\right\],\\quad\\varepsilon\\sim\\mathcal\{N\}\(0,I\_\{6\}\),\(20\)withyeff=yy\_\{\\mathrm\{eff\}\}=yat training time andyeff=0y\_\{\\mathrm\{eff\}\}=0at test time\. The parameterβ\\betais a continuous SNR knob, not a probability: atβ=0\\beta=0the spur is pure noise; atβ=1\\beta=1the noise term vanishes and the spur becomes deterministic inyy; themax⁡\(0,⋅\)\\max\(0,\\cdot\)clamp permits valuesβ\>1\\beta\>1, which further amplify the deterministic mean\-shift while keeping the noise term zero\. We sweepβ∈\{0,0\.2,0\.4,0\.6,0\.8,1\.0,2\.0\}\\beta\\in\\\{0,0\.2,0\.4,0\.6,0\.8,1\.0,2\.0\\\}\. The training set contains 300 samples, the validation set 120, and the test set 30,000; the small training set is deliberate and gives the spurious shortcut a real chance to dominate the cross\-modal XOR signal during fitting\.

#### F\.1\.2Model, Objective, and Mask Variants

The fusion network has modality\-specific encoders, two unimodal heads, and a fusion head\. Each encoderfθmf\_\{\\theta\_\{m\}\}is a single hidden layer,Linear​\(64→16\)→ReLU→Dropout\\mathrm\{Linear\}\(64\\to 16\)\\to\\mathrm\{ReLU\}\\to\\mathrm\{Dropout\}\. Unimodal heads areLinear​\(16→1\)\\mathrm\{Linear\}\(16\\to 1\), trained againstyywith weightλuni=0\.05\\lambda\_\{\\mathrm\{uni\}\}=0\.05\. The fusion head concatenates the two encoder outputs and appliesLinear​\(32→32\)→ReLU→Dropout→Linear​\(32→1\)\\mathrm\{Linear\}\(32\\to 32\)\\to\\mathrm\{ReLU\}\\to\\mathrm\{Dropout\}\\to\\mathrm\{Linear\}\(32\\to 1\), producing a single logit trained with binary cross\-entropy\. SynIB regularizes the fusion logit symmetrically across modalities,

ℒ=ℒCE\+λ∑i∈\{1,2\}DKL\(Pf\(⋅∣X~i,X¬i\)∥Bernoulli\(0\.5\)\),\\mathcal\{L\}=\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\+\\lambda\\sum\_\{i\\in\\\{1,2\\\}\}D\_\{\\mathrm\{KL\}\}\\\!\\left\(P\_\{f\}\(\\cdot\\mid\\tilde\{X\}\_\{i\},X\_\{\\neg i\}\)\\,\\big\\\|\\,\\mathrm\{Bernoulli\}\(0\.5\)\\right\),\(21\)whereX¬iX\_\{\\neg i\}denotes the unmasked modality\. The reference distribution is uniform over the binary outcomes; only the fusion logit is regularized, not the unimodal heads\.

Corrupted coordinates are replaced viax~=\(1−M\)⊙x\+M⊙ε\\tilde\{x\}=\(1\-M\)\\odot x\+M\\odot\\varepsilonwithε∼𝒩​\(0,I\)\\varepsilon\\sim\\mathcal\{N\}\(0,I\)\. The oracle maskM⋆M^\{\\star\}flags the signal blocks\[0,4\)\[0,4\)of both modalities \(the coordinates encodingb1,b2b\_\{1\},b\_\{2\}\); the spurious block inX1X\_\{1\}is not flagged, so any confident prediction under oracle corruption would have to rely on the shortcut\. Random masking corrupts each coordinate of each modality independently with probabilityπ=0\.4\\pi=0\.4\. Learned masking uses per\-example, per\-coordinate logits initialized at−4\-4, with an inner loop of 50 Adam steps \(lr=3\.0=3\.0, sparsityλM=0\.03\\lambda\_\{M\}=0\.03, temperatureτ=1\.0\\tau=1\.0\) finding the smallest mask whose corruption breaks the unimodal head; the discovered mask is hard\-thresholded at0\.50\.5, and the SynIB pass corrupts the complement, leaving the unimodally\-important coordinates intact\.

#### F\.1\.3Training Configuration

We train with Adam, batch size 128, for 100 epochs, across seeds\{0,1,2\}\\\{0,1,2\\\}and dropout0\. The baseline uses learning rate10−210^\{\-2\}and weight decay10−310^\{\-3\}; the SynIB variants use learning rate3×10−43\\times 10^\{\-4\}and weight decay10−510^\{\-5\}, with regularization strengthλ=10\\lambda=10\. Validation accuracy curves over training are reported in Figure[10](https://arxiv.org/html/2606.09853#A6.F10)\(left panel\) with final test accuracies\. Reported numbers are mean±\\pmstandard error across the 3 seeds\.

![Refer to caption](https://arxiv.org/html/2606.09853v1/x8.png)Figure 10:Validation accuracy over training on both synthetic benchmarks\.Mean±\\pmstandard error across three seeds\.Left: Spurious XOR \(β=1\.0\\beta=1\.0\)\. All four methods reach≈1\.0\\approx 1\.0on the in\-distribution validation set, fitting the training distribution equally well\. The gray dashed curve shows training accuracy for vanilla fusion, which saturates within the first epoch as the model locks onto the shortcut\. The annotated final out\-of\-distribution test accuracies reveal the divergence: vanilla collapses to chance \(0\.500\.50\) while oracle masking recovers fully \(1\.001\.00\); learned masking reaches0\.830\.83and random masking0\.680\.68\. Random masking shows the largest seed\-to\-seed variance \(per\-seed test accuracies0\.580\.58–0\.780\.78\), reflecting sensitivity to which coordinates the random mask happens to corrupt\.Right:PID\-Controlled XOR\. Synergy\-restricted validation accuracy at simplex point\(0\.45,0,0\.45,0\.10\)\(0\.45,0,0\.45,0\.10\)\. Vanilla plateaus at chance on the synergy slice; oracle, learned, and random masking each rise above0\.850\.85, with learned masking \(0\.890\.89\) tracking oracle \(0\.910\.91\) within three percentage points\.

### F\.2PID\-Controlled XOR

#### F\.2\.1Task and Data Generation

The PID\-Controlled XOR task is a bimodal binary classification problem in which each sample’s label is generated by a single PID source: unique to modality 1 \(U1U\_\{1\}\), unique to modality 2 \(U2U\_\{2\}\), redundant \(RR\), or synergistic \(SS\)\. The dataset’s mixture is set by a categorical distributionp​\(A\)=\(pU1,pU2,pR,pS\)p\(A\)=\(p\_\{U\_\{1\}\},p\_\{U\_\{2\}\},p\_\{R\},p\_\{S\}\), allowing direct control over the prevalence of each information type\. For Figure[7](https://arxiv.org/html/2606.09853#S4.F7)we holdpU2=0p\_\{U\_\{2\}\}=0and sweep the remaining three coordinates over the probability simplex on a regular grid with step0\.050\.05\.

Both modalities are 32\-dimensional and partitioned into four disjoint blocks of 6, 6, 6, and 14 coordinates\. The first block carries the modality’s unique signal \(U1U\_\{1\}for modality 1,U2U\_\{2\}for modality 2\), the second block carries the redundant signal, the third block carries the synergistic signal, and the fourth block is pure noise\. Block coordinate sets are fixed across samples\.

Each sample is generated by initializing all coordinates ofX1,X2X\_\{1\},X\_\{2\}as i\.i\.d\.𝒩​\(0,1\)\\mathcal\{N\}\(0,1\), samplingy∼Bernoulli​\(0\.5\)y\\sim\\mathrm\{Bernoulli\}\(0\.5\)independently, sampling an active sourceA∼p​\(A\)A\\sim p\(A\), overwriting the block coordinates of the active source with label\-aligned signal as described below, and finally standardizing each coordinate using the training\-set mean and standard deviation\. Blocks not corresponding to the active source remain at their initial𝒩​\(0,1\)\\mathcal\{N\}\(0,1\)noise\.

Three pairs of i\.i\.d\. random projection matricesPum,Prm,Psm∈ℝdm×4P\_\{u\_\{m\}\},P\_\{r\_\{m\}\},P\_\{s\_\{m\}\}\\in\\mathbb\{R\}^\{d\_\{m\}\\times 4\}are sampled once at script load \(seeded for reproducibility\) with entries i\.i\.d\.𝒩​\(0,0\.52\)\\mathcal\{N\}\(0,0\.5^\{2\}\)\. Sign\-aligned latents are constructed by samplingz∼𝒩​\(0,s2​I4\)z\\sim\\mathcal\{N\}\(0,s^\{2\}I\_\{4\}\)for the appropriate source\-specific scalessand applyingz←\(2​b−1\)⋅\|z\|z\\leftarrow\(2b\-1\)\\cdot\|z\|elementwise for the relevant bitbb\. The full projected vector is computed and only the corresponding block coordinates are written back into the modality\. We usesu=sr=ss=3s\_\{u\}=s\_\{r\}=s\_\{s\}=3\.

When sourceUmU\_\{m\}is active,zzis sampled withb=yb=yandXm​\[unique\]=\(Pum​z\)​\[unique\]X\_\{m\}\[\\mathrm\{unique\}\]=\(P\_\{u\_\{m\}\}z\)\[\\mathrm\{unique\}\]; the other modality’s unique block remains noise\. When sourceRRis active, a single shared latentzRz\_\{R\}is sign\-aligned withyyand projected through different matricesPr1,Pr2P\_\{r\_\{1\}\},P\_\{r\_\{2\}\}into the redundant blocks of both modalities, so the two redundant blocks are correlated throughzRz\_\{R\}but not identical\. When sourceSSis active, a latent bitb∼Bernoulli​\(0\.5\)b\\sim\\mathrm\{Bernoulli\}\(0\.5\)is sampled withb1=bb\_\{1\}=bandb2=b⊕yb\_\{2\}=b\\oplus yso thatb1⊕b2=yb\_\{1\}\\oplus b\_\{2\}=y; independent latentsz\(m\)z^\{\(m\)\}are sign\-aligned withbmb\_\{m\}and projected into modalitymm’s synergistic block, making each modality alone uninformative aboutyywhile their XOR recovers it\. The training set contains 1800 samples per simplex point, with 200 held out for validation and 4000 for test\.

#### F\.2\.2Model, Objective, and Mask Variants

The PID\-XOR model uses substantially more capacity than the Spurious\-XOR model, reflecting the harder task of distinguishing four signal types across more coordinates\. Each encoder is two hidden layers,Linear​\(32→1024\)→ReLU→Dropout→Linear​\(1024→1024\)→ReLU\\mathrm\{Linear\}\(32\\to 1024\)\\to\\mathrm\{ReLU\}\\to\\mathrm\{Dropout\}\\to\\mathrm\{Linear\}\(1024\\to 1024\)\\to\\mathrm\{ReLU\}\. Unimodal heads areLinear​\(1024→1024\)→ReLU→Linear​\(1024→1\)\\mathrm\{Linear\}\(1024\\to 1024\)\\to\\mathrm\{ReLU\}\\to\\mathrm\{Linear\}\(1024\\to 1\), with the unimodal\-head loss entering the total objective with weightλuni=1\.0\\lambda\_\{\\mathrm\{uni\}\}=1\.0\. The fusion head concatenates the two encoder outputs \(2048\-d\) and appliesLinear​\(2048→128\)→ReLU→Dropout→Linear​\(128→128\)→ReLU→Linear​\(128→1\)\\mathrm\{Linear\}\(2048\\to 128\)\\to\\mathrm\{ReLU\}\\to\\mathrm\{Dropout\}\\to\\mathrm\{Linear\}\(128\\to 128\)\\to\\mathrm\{ReLU\}\\to\\mathrm\{Linear\}\(128\\to 1\)\. The SynIB objective takes the same form as in Sec\.[F\.1](https://arxiv.org/html/2606.09853#A6.SS1), withBernoulli​\(0\.5\)\\mathrm\{Bernoulli\}\(0\.5\)as the reference and KL applied to the fusion logit\.

The same gating mechanism is used, with replacement noise𝒩​\(0,1\)\\mathcal\{N\}\(0,1\)per coordinate\. The oracle maskM⋆M^\{\\star\}corrupts only the synergistic blocks of both modalities; unique, redundant, and noise blocks are preserved\. Random masking corrupts each coordinate of each modality independently with probabilityπ=0\.5\\pi=0\.5\. Learned masking uses an inner loop of 20 Adam steps \(lr=0\.1=0\.1,τ=1\.0\\tau=1\.0\); the hard binary mask discovered by the inner loop is used as the corruption pattern in the outer SynIB pass\.

## Appendix GDisentangling Masking Effects from SynIB

An arising question is whether the gains from SynIB stem from the proposed information\-theoretic objective or simply from exposing the model to masked inputs during training\. To rule out the latter, we compare SynIB against a masking\-only baseline that uses the same mask\-construction mechanism as SynIB but applies it as input augmentation rather than as a counterfactual KL penalty: each batch is augmented with masked copies of the original samples, and training proceeds with cross\-entropy on the union of intact and masked inputs\.

We instantiate two variants of this baseline that mirror SynIB’s mask choices:*masked input with oracle*corrupts the unique and redundant blocks \(preserving the synergistic block\) so the model is exposed to inputs that carry only synergistic signal, and*masked input with random masking*applies a random Bernoulli mask matching the SynIB\-random configuration\.

Figure[11](https://arxiv.org/html/2606.09853#A7.F11)reports the comparison across the PID simplex\. The masking\-only baselines yield limited improvements over no regularization and fail to recover the synergistic structure, particularly when synergy is dominant\. SynIB, in contrast, achieves substantially higher accuracy across the simplex and on the synergy\-restricted subset\. The gap indicates that input augmentation alone is insufficient: SynIB’s gains depend on explicitly penalizing confident predictions under modality corruption, not on data exposure to masked inputs\.

![Refer to caption](https://arxiv.org/html/2606.09853v1/x9.png)Figure 11:Masking\-based baselines vs SynIB across PID compositions\. Columns: \(a\) no regularization, \(b\) SynIB with oracle maskingM⋆M^\{\\star\}, \(c\) SynIB with random masking, \(d\) masked\-input training with oracle, \(e\) masked\-input training with random masking\. Triangles sweep the PID simplex over\(pU1,pR,pS\)\(p\_\{U\_\{1\}\},p\_\{R\},p\_\{S\}\)\. Top row: total accuracy; bottom row: accuracy restricted to synergistic samples\. Masked\-input baselines yield limited gains, while SynIB recovers synergistic structure across the simplex\.
## Appendix HSynergy\-Driven Irony Dataset Construction

This appendix provides the construction details for the synthetic irony benchmark introduced in Sec\.[4\.1](https://arxiv.org/html/2606.09853#S4.SS1), along with the corresponding model architecture, training setup, and evaluation protocol\.

### H\.1Base Dataset and Splits

We start from CREMA\-D\(Caoet al\.,[2014](https://arxiv.org/html/2606.09853#bib.bib8)\), a crowd\-sourced audio–visual emotion recognition dataset of short clips in which actors speak sentences with a target emotion\. Following the standard 6\-class taxonomy, we use neutral \(NEU\), happy \(HAP\), sad \(SAD\), fearful \(FEA\), disgusted \(DIS\), and angry \(ANG\); we add a synthetic 7th class for irony\. We use a 5\-fold speaker\-disjoint split with no actor appearing in more than one of \(train, val, test\) within any fold\. After dropping samples missing audio, video, or face crops, each fold contains 5,820 samples \(approximately 4,043 train / 878 val / 899 test\); base classes are mildly imbalanced with HAP largest and SAD smallest at a roughly1\.5×1\.5\\timesratio\.

### H\.2Mutation Procedure

LetDoriginal=\{\(Vi,Ai,Li\)\}i=1ND\_\{\\text\{original\}\}=\\\{\(V\_\{i\},A\_\{i\},L\_\{i\}\)\\\}\_\{i=1\}^\{N\}denote the original set of \(video, audio, label\) triplets, and letN¯\\bar\{N\}denote the mean number of samples per base class within a given split\. The irony rateα\\alphacontrols the number of mutated samples,\|I\|=⌊α​N¯⌋\|I\|=\\lfloor\\alpha\\bar\{N\}\\rfloor, drawn uniformly without replacement from the base\-class pool of that split\. For each selected indexiiwith base labelLiL\_\{i\}, we sample a donor indexjjfrom a designated set of contradiction labels and form a mutated triplet\(Vi,Aj,Lirony\)\(V\_\{i\},A\_\{j\},L\_\{\\text\{irony\}\}\)\. The original audio is replaced by the donor’s audio while video and identity are preserved; the new label is the irony class\. The resulting dataset is

Dnew=\{\(Vi,Aj,Lirony\)\}i∈I∪\{\(Vi,Ai,Li\)\}i∉I\.D\_\{\\text\{new\}\}=\\\{\(V\_\{i\},A\_\{j\},L\_\{\\text\{irony\}\}\)\\\}\_\{i\\in I\}\\cup\\\{\(V\_\{i\},A\_\{i\},L\_\{i\}\)\\\}\_\{i\\notin I\}\.\(22\)Mutation is in\-place: each mutated sample replaces its original, so the total dataset size is preserved and the irony class population is controlled solely byα\\alpha\. We sweepα∈\{0\.1,0\.3,0\.5,0\.8,1\.0,2\.0\}\\alpha\\in\\\{0\.1,0\.3,0\.5,0\.8,1\.0,2\.0\\\}\.

The donor set for each base class is fixed by a semantic contradiction map: HAP is paired with one of\{\\\{SAD, FEA, DIS, ANG\}\\\}, while SAD, FEA, DIS, and ANG are each paired with HAP\. NEU samples have no semantically opposite class in this taxonomy and are paired with any non\-matching label uniformly at random\. This map ensures that ironic pairings produce strong cross\-modal contradictions rather than subtle disagreements between similar emotions\. Donors are drawn from the same split as the mutated sample \(train donors for train mutations, etc\.\) to prevent identity leakage across train and test, and the sameα\\alphais applied to all three splits\. Each mutated sample is, by construction, individually consistent with both its visual base emotion and its donor audio’s emotion: neither modality alone identifies the sample as ironic, so the irony label is recoverable only from the contradiction between modalities\.

### H\.3Feature Representations

Audio is represented as a log\-magnitude short\-time Fourier transform \(STFT\) spectrogram\. Waveforms are resampled to 22,050 Hz, tiled and cropped to 3 seconds, and clipped to\[−1,1\]\[\-1,1\]before computing an STFT withnFFT=512n\_\{\\text\{FFT\}\}=512and hop length 353\. The resulting spectrogram islog\(\|⋅\|\+10−7\)\\log\(\|\\cdot\|\+10^\{\-7\}\), producing a single\-channel257×188257\\times 188representation\.

Video is represented by 3 RGB frames sampled at 1 fps, resized to224×224224\\times 224, and normalized using ImageNet statistics\. Training\-time augmentation uses random resized crops and random horizontal flips; validation and test use deterministic resizing\.

### H\.4Model Architecture

Each modality is encoded by a ResNet18 backbone adapted for its input shape\. The audio backbone takes the single\-channel spectrogram and applies a global average pool to produce a 512\-dimensional representation; the video backbone takes the 3\-frame clip and applies a 3D global average pool to produce a 512\-dimensional representation\. Both backbones are pretrained unimodally on CREMA\-D before joint training, with the same pretrained checkpoints used as initialization across all methods\.

The fusion trunk concatenates the two 512\-dimensional encoder outputs and appliesLinear​\(1024→64\)→ReLU→Dropout​\(0\.1\)\\mathrm\{Linear\}\(1024\\to 64\)\\to\\mathrm\{ReLU\}\\to\\mathrm\{Dropout\}\(0\.1\)\. The fusion head is a singleLinear​\(64→7\)\\mathrm\{Linear\}\(64\\to 7\)producing the 7\-way classification logits\. Auxiliary unimodal heads \(Linear​\(512→64\)→ReLU→Dropout​\(0\.1\)→Linear​\(64→7\)\\mathrm\{Linear\}\(512\\to 64\)\\to\\mathrm\{ReLU\}\\to\\mathrm\{Dropout\}\(0\.1\)\\to\\mathrm\{Linear\}\(64\\to 7\)\) are attached to each encoder for the SynIB inner loop\. All baselines share the same ResNet18 backbones and pretrained initialization; they differ in the fusion mechanism according to each method’s specification\.

### H\.5Training and SynIB Configuration

We train with Adam \(β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.999\\beta\_\{2\}=0\.999\), learning rate10−510^\{\-5\}, weight decay10−510^\{\-5\}, and batch size 64, with cosine annealing and a 780\-step warmup\. Maximum training length is 1500 epochs with early stopping on validation accuracy \(patience 30 epochs\)\. The classification loss is unweighted cross\-entropy across all 7 classes, with no per\-class reweighting\.

For SynIB, we use random masking on each modality with corruption probabilityπ\\piand reference distribution given by the complementary modality’s unimodal prediction, so the KL term penalizes the fused prediction for departing from what the unmasked modality alone would predict\. The regularization strength isλ\\lambda\. SynIB\-specific values are held constant across allα\\alphafor a given run\.

### H\.6Evaluation

Models are evaluated on the test split of each fold using two metrics: macro F1 across all 7 classes \(reported as Total F1 in Figure[4](https://arxiv.org/html/2606.09853#S4.F4)\) and per\-class F1 for the irony class \(reported as Irony F1\)\. Both metrics are computed on the full test set, which contains⌊α​N¯test⌋\\lfloor\\alpha\\bar\{N\}\_\{\\text\{test\}\}\\rfloorirony samples generated by the same mutation procedure used at training time\. Reported values in Figure[4](https://arxiv.org/html/2606.09853#S4.F4)are mean and standard deviation across 3 folds\.

![Refer to caption](https://arxiv.org/html/2606.09853v1/x10.png)Figure 12:Construction of the synthetic irony class\. Each ironic sample is built by pairing a videoViV\_\{i\}from one base emotion \(e\.g\., happy\) with a donor audioAjA\_\{j\}from a contradicting emotion \(e\.g\., disgust\)\. The new label is irony\. The dataset isDnew=\{\(Vi,Aj,Lirony\)\}i∈I∪\{\(Vi,Ai,Li\)\}i∉ID\_\{\\text\{new\}\}=\\\{\(V\_\{i\},A\_\{j\},L\_\{\\text\{irony\}\}\)\\\}\_\{i\\in I\}\\cup\\\{\(V\_\{i\},A\_\{i\},L\_\{i\}\)\\\}\_\{i\\notin I\}, with\|I\|=⌊α​N¯⌋\|I\|=\\lfloor\\alpha\\bar\{N\}\\rfloorcontrolled by the irony\-rate hyperparameterα\\alpha\. Neither modality alone identifies an ironic sample; the label requires reading the cross\-modal contradiction\.
### H\.7Numerical Results

Table[1](https://arxiv.org/html/2606.09853#A8.T1)reports the per\-method F1 scores summarized in Figure[4](https://arxiv.org/html/2606.09853#S4.F4), with standard deviations across folds\. Each cell shows irony\-class F1 over total F1 \(mean±\\pmstandard deviation across 3 folds\); bold marks the best entry per column for each metric\. Uni\-Video and Uni\-Audio do not predict the irony class and report total F1 only\. Reported values cover the four irony rates used in the main figure \(α∈\{0\.1,0\.5,1\.0,2\.0\}\\alpha\\in\\\{0\.1,0\.5,1\.0,2\.0\\\}\)\.

Table 1:F1 scores on the CREMA\-D irony recognition task under varying irony ratesα\\alpha, reported as irony\-class F1 / total F1 \(mean±\\pmstandard deviation across 3 folds\)\. Bold marks the best per column for each metric\. Dashes indicate that the corresponding metric does not apply\.##### Reading the table\.

SynIB achieves the highest irony\-class F1 in the two regimes where synergistic samples are scarce \(α∈\{0\.1,0\.5\}\\alpha\\in\\\{0\.1,0\.5\\\}\), with margins of\+2\.2\+2\.2and\+3\.4\+3\.4points over the best balancing baseline \(MCR\) and consistently larger margins over the unimodal\-rebalancing methods \(D&R, MMPareto, ReconBoost\)\. When synergistic samples become abundant \(α∈\{1\.0,2\.0\}\\alpha\\in\\\{1\.0,2\.0\\\}\), simpler methods that do not regularize fusion explicitly \(vanilla fusion atα=1\.0\\alpha=1\.0, ensemble atα=2\.0\\alpha=2\.0\) match or slightly exceed SynIB on irony F1, reflecting that abundant synergy is recoverable without an explicit synergy\-targeted objective\. SynIB nevertheless remains within1\.31\.3–4\.44\.4points of the leader in these regimes and outperforms all four balancing methods on irony F1 at everyα\\alpha\.

##### On total F1\.

On macro F1 across all 7 classes, MCR achieves the best score in three of four cells; SynIB stays within0\.20\.2to3\.33\.3points of MCR across all rates and outperforms vanilla fusion atα=0\.1,0\.5\\alpha=0\.1,0\.5\. Ensemble achieves the best total F1 atα=1\.0\\alpha=1\.0but collapses atα=2\.0\\alpha=2\.0\(38\.3±\\pm21\.4\), where the high standard deviation reflects fold\-level instability rather than a stable improvement\. The trade\-off between irony\-class F1 and total F1 is small and consistent: SynIB sacrifices roughly 1–3 points of total F1 to gain22–55points on the synergy\-required class\.

##### Variance considerations\.

Standard deviations on the irony\-class metric are large for several baselines \(MCR±12\.2\\pm 12\.2atα=0\.1\\alpha=0\.1, vanilla fusion±12\.2\\pm 12\.2atα=0\.5\\alpha=0\.5, ReconBoost±10\.0\\pm 10\.0atα=2\.0\\alpha=2\.0\), indicating fold\-to\-fold instability in their irony predictions when synergistic structure is present but scarce\. SynIB’s standard deviations on irony F1 are uniformly smaller \(between±3\.6\\pm 3\.6and±6\.6\\pm 6\.6\), suggesting that the synergy\-targeted objective produces more stable irony detection across folds in addition to higher mean F1\.

## Appendix IPer\-Method Numerical Results on MultiBench and Hateful Memes

Table[2](https://arxiv.org/html/2606.09853#A9.T2)reports the per\-method numbers underlying Fig\.[5](https://arxiv.org/html/2606.09853#S4.F5)in the main text\. Synergy\-subset examples are test samples misclassified by every unimodal predictor \(see Sec\.[4\.1](https://arxiv.org/html/2606.09853#S4.SS1)for definition and subset sizes per benchmark\)\. Each cell is the mean over 3 seeds; subscripts give one standard deviation\.Boldmarks the best per column\.

Table 2:Synergy\-subset accuracy and whole\-test accuracy \(%\\%\) on four real\-world multimodal benchmarks\. Each cell is the mean over 3 seeds; subscripts give one standard deviation\.Boldmarks the best per column\.##### Reading the table\.

On the synergy subset, SynIBMLM\_\{\\mathrm\{L\}\}is the top method on three of four benchmarks \(UR\-Funny, MUStARD, MOSI\), and SynIBMRM\_\{\\mathrm\{R\}\}is the top on Hateful Memes; no baseline is best on more than one benchmark\. On the full test set, SynIB variants lead on three of four benchmarks \(MUStARD, MOSI, Hateful Memes\) and tie D&R on UR\-Funny, never trailing the strongest baseline by more than 0\.9 points\. Standard deviations on the synergy\-subset metric are large for several balancing baselines \(MCR±22\.4\\pm 22\.4on MUStARD, MMPareto±13\.4\\pm 13\.4, ReconBoost±6\.7\\pm 6\.7\), indicating fold\-to\-fold instability when synergy is scarce; SynIB’s standard deviations are smaller across all four benchmarks, suggesting the gains are also more stable\.

### I\.1Compute Resources

All experiments ran on NVIDIA A100 GPUs \(40 GB HBM2\)\. Each job was allocated a single A100, 8 CPU cores, and 80 GB of host RAM\. Final\-result runs \(3 seeds×\\times8 methods×\\times4 datasets, 96 runs total\) consumed roughly 200 GPU\-h\. Hyperparameter sweeps for SynIB and the four balancing baselines added approximately 1000 GPU\-h, the bulk on Hateful Memes due to its larger grid and longer per\-run time\. The full main\-paper table can be reproduced end\-to\-end on a single A100 in roughly 50 wall\-clock hours\. All runs use Python 3\.10, PyTorch 2\.1, CUDA 12\.1, and HuggingFacetransformers4\.38; checkpoints, logs, and reproduction scripts are released with the code\.

## Appendix JBaselines

We compare SynIB against representative multimodal learning strategies spanning unimodal models, standard fusion, and methods designed to mitigate multimodal competition\. All baselines use the same backbone architecture and differ only in their optimization objectives or training procedures\.

Let\(x1,x2,y\)∼p​\(x1,x2,y\)\(x\_\{1\},x\_\{2\},y\)\\sim p\(x\_\{1\},x\_\{2\},y\)denote multimodal samples and letfθ​\(x1,x2\)∈Δ​\(𝒴\)f\_\{\\theta\}\(x\_\{1\},x\_\{2\}\)\\in\\Delta\(\\mathcal\{Y\}\)be the joint predictor\. Unimodal predictors are denotedfθ1​\(x1\)f\_\{\\theta\_\{1\}\}\(x\_\{1\}\)andfθ2​\(x2\)f\_\{\\theta\_\{2\}\}\(x\_\{2\}\)\.

##### Unimodal models\.

Audio\-only and video\-only classifiers are trained independently using cross\-entropy:ℒuni,i=ℒtask​\(fθi​\(y∣xi\),Y\)i∈\{1,2\}\.\\mathcal\{L\}\_\{\\text\{uni\},i\}=\\mathcal\{L\}\_\{\\text\{task\}\}\(f\_\{\\theta\_\{i\}\}\(y\\mid x\_\{i\}\),Y\)\\quad i\\in\\\{1,2\\\}\.These models establish a lower bound and verify whether the task is solvable from a single modality\.

##### Late ensemble\.

Unimodal predictions are averaged at inference:fens​\(y∣x1,x2\)=12​\(fθ1​\(y∣x1\)\+fθ2​\(y∣x2\)\)\.f\_\{\\text\{ens\}\}\(y\\mid x\_\{1\},x\_\{2\}\)=\\frac\{1\}\{2\}\\left\(f\_\{\\theta\_\{1\}\}\(y\\mid x\_\{1\}\)\+f\_\{\\theta\_\{2\}\}\(y\\mid x\_\{2\}\)\\right\)\.No joint representation learning occurs\. This baseline tests whether aggregation alone can recover cross\-modal signal\.

##### Vanilla fusion\.

Standard multimodal fusion minimizes cross\-entropy on joint inputsℒfusion=ℒtask​\(fθ​\(y∣x1,x2\),Y\)\\mathcal\{L\}\_\{\\text\{fusion\}\}=\\mathcal\{L\}\_\{\\text\{task\}\}\(f\_\{\\theta\}\(y\\mid x\_\{1\},x\_\{2\}\),Y\)\. This objective does not constrain modality contributions and is known to exhibit multimodal competition, where optimization favors dominant unimodal shortcutsHuanget al\.\([2022](https://arxiv.org/html/2606.09853#bib.bib19)\)\.

##### Diagnosing & Re\-learning \(D&R\)Weiet al\.\([2024](https://arxiv.org/html/2606.09853#bib.bib105)\)\.

D&R mitigates modality collapse by periodically reinitializing modality encoders when imbalance is detected\. Letgkg\_\{k\}denote the purity\-gap diagnostic for modalitykk\. The reset strength isαk=tanh⁡\(λ​gk\)\\alpha\_\{k\}=\\tanh\(\\lambda g\_\{k\}\)\. When imbalance exceeds a threshold, parameters are softly reset viaθk←\(1−αk\)​θk\+αk​θkinit\\theta\_\{k\}\\leftarrow\(1\-\\alpha\_\{k\}\)\\theta\_\{k\}\+\\alpha\_\{k\}\\theta\_\{k\}^\{\\text\{init\}\}\. Training then resumes under the vanilla fusion objective\.

D&R prevents one modality from monopolizing learning by periodically restoring weaker modality capacity\. This stabilizes the balance of unique contributions across modalities, but it does not introduce an objective that favors joint\-only information\. Consequently, D&R addresses optimization imbalance without explicitly promoting synergistic representations\.

##### MMParetoWei and Hu \([2024](https://arxiv.org/html/2606.09853#bib.bib103)\)\.

MMPareto replaces the vanilla fusion gradient with a Pareto\-integrated gradient that balances unimodal and multimodal objectives\. Letℒ12=ℒ​\(x1,x2,y\),ℒ1=ℒ​\(x1,y\),ℒ2=ℒ​\(x2,y\)\\mathcal\{L\}\_\{12\}=\\mathcal\{L\}\(x\_\{1\},x\_\{2\},y\),\\quad\\mathcal\{L\}\_\{1\}=\\mathcal\{L\}\(x\_\{1\},y\),\\quad\\mathcal\{L\}\_\{2\}=\\mathcal\{L\}\(x\_\{2\},y\)denote multimodal and unimodal losses\. The parameter update uses the integrated gradientg∗=∑m∈\{1,2,12\}wm​∇ℒmg^\{\*\}=\\sum\_\{m\\in\\\{1,2,12\\\}\}w\_\{m\}\\nabla\\mathcal\{L\}\_\{m\}, where the weights are obtained by solving the Pareto min\-norm problem

min𝐰∈Δ⁡‖∑mwm​∇ℒm‖2,Δ=\{wm≥0,∑mwm=1\}\.\\min\_\{\\mathbf\{w\}\\in\\Delta\}\\left\\\|\\sum\_\{m\}w\_\{m\}\\nabla\\mathcal\{L\}\_\{m\}\\right\\\|\_\{2\},\\quad\\Delta=\\\{w\_\{m\}\\geq 0,\\ \\sum\_\{m\}w\_\{m\}=1\\\}\.\(23\)Parameters are updated usingg∗g^\{\*\}instead of the vanilla gradient\.

From a PID perspective, MMPareto equalizes optimization pressure across unimodal and multimodal objectives, preventing one modality from dominating learning\. This stabilizes unique and redundant information usage, but does not explicitly prioritize signals that arise only from joint cross\-modal interaction\. As a result, it balances modality competition without directly targeting synergy\.

##### ReconBoostHuaet al\.\([2024](https://arxiv.org/html/2606.09853#bib.bib104)\)\.

ReconBoost alternates unimodal updates while enforcing agreement and gradient consistency between modality predictors\. The fused prediction isp12=∑k=1Mpkp\_\{12\}=\\sum\_\{k=1\}^\{M\}p\_\{k\}, wherepk=p​\(y\|xk\)p\_\{k\}=p\(y\|x\_\{k\}\)are unimodal predictors\.At boosting roundss, modalitykkoptimizes

ℒ~s=ℒk−λ​DKL​\(p−k∥pk\)\+α​‖∇ℒk−∇ℒk−1‖2,\\tilde\{\\mathcal\{L\}\}^\{\\,s\}=\\mathcal\{L\}\_\{k\}\-\\lambda\\,D\_\{\\mathrm\{KL\}\}\(p\_\{\-k\}\\,\\\|\\,p\_\{k\}\)\+\\alpha\\\|\\nabla\\mathcal\{L\}\_\{k\}\-\\nabla\\mathcal\{L\}\_\{k\-1\}\\\|^\{2\},\(24\)whereℒk=−log⁡pk​\(y\)\\mathcal\{L\}\_\{k\}=\-\\log p\_\{k\}\(y\)andp−k=∑j≠kpjp\_\{\-k\}=\\sum\_\{j\\neq k\}p\_\{j\}\. After alternating rounds, a global rectification step minimizesℒrect=−log⁡p12​\(y\)\\mathcal\{L\}\_\{\\text\{rect\}\}=\-\\log p\_\{12\}\(y\)\.

ReconBoost alternates unimodal specialization while enforcing agreement and gradient alignment between modalities\. This stabilizes redundancy and prevents dominance by a single modality, but the objective reconciles unimodal predictors rather than rewarding information that exists only in their interaction\. Consequently, ReconBoost mitigates modality conflict without explicitly optimizing for synergistic representations\.

##### MCR \(Multimodal Competition Regularizer\)Kontraset al\.\([2025](https://arxiv.org/html/2606.09853#bib.bib110)\)\.

MCR augments the fusion loss with a regularizer derived from a mutual\-information decomposition:

∇ℒ=∇ℒfusion\+ℒCon\+λ​\(∇θ1ℒMIPD1−∇θ2ℒMIPD1−∇θ1ℒMIPD2\+∇θ2ℒMIPD2\)\.\\nabla\\mathcal\{L\}=\\nabla\\mathcal\{L\}\_\{\\text\{fusion\}\}\+\\mathcal\{L\}\_\{\\text\{Con\}\}\+\\lambda\\Big\(\\nabla\_\{\\theta\_\{1\}\}\\mathcal\{L\}\_\{\\text\{MIPD\}\_\{1\}\}\-\\nabla\_\{\\theta\_\{2\}\}\\mathcal\{L\}\_\{\\text\{MIPD\}\_\{1\}\}\-\\nabla\_\{\\theta\_\{1\}\}\\mathcal\{L\}\_\{\\text\{MIPD\}\_\{2\}\}\+\\nabla\_\{\\theta\_\{2\}\}\\mathcal\{L\}\_\{\\text\{MIPD\}\_\{2\}\}\\Big\)\.\(25\)The Mutual Information Perturbed Difference term measures modality contribution via latent perturbations:

ℒMIPD=−𝔼​\[JSD​\(p12,p1~​2\)\+JSD​\(p12,p1​2~\)\],\\mathcal\{L\}\_\{\\text\{MIPD\}\}=\-\\mathbb\{E\}\\Big\[\\mathrm\{JSD\}\(p\_\{12\},p\_\{\\tilde\{1\}2\}\)\+\\mathrm\{JSD\}\(p\_\{12\},p\_\{1\\tilde\{2\}\}\)\\Big\],\(26\)wherep12=p​\(y\|x1,x2\)p\_\{12\}=p\(y\|x\_\{1\},x\_\{2\}\),p1~​2=p​\(y\|x~1,x2\)p\_\{\\tilde\{1\}2\}=p\(y\|\\tilde\{x\}\_\{1\},x\_\{2\}\), andp1​2~=p​\(y\|x1,x~2\)p\_\{1\\tilde\{2\}\}=p\(y\|x\_\{1\},\\tilde\{x\}\_\{2\}\)and the shared task\-relevant information is preserved through supervised contrastive alignmentℒCon=−log⁡ψ​\(z1,z2\+\)∑kψ​\(z1,z2,k−\)\.\\mathcal\{L\}\_\{\\text\{Con\}\}=\-\\log\\frac\{\\psi\(z\_\{1\},z\_\{2\}^\{\+\}\)\}\{\\sum\_\{k\}\\psi\(z\_\{1\},z\_\{2,k\}^\{\-\}\)\}\.

MCR regulates how much each modality contributes individually, preventing one modality from dominating the prediction\. The perturbation term enforces sensitivity to both inputs, encouraging the model to retain unique modality information, while the contrastive alignment term preserves shared task\-relevant structure\. However, these mechanisms operate by balancing redundancy and unique contributions rather than explicitly modeling information that exists only in the joint interaction\. As a result, MCR mitigates modality collapse but does not directly optimize for synergistic representations\.

## Appendix KNTK Diagnostics for PID Learning

This appendix expands on the NTK\-based diagnostic introduced in Sec\.[3\.1](https://arxiv.org/html/2606.09853#S3.SS1)\. We restate the core definitions for self\-containedness, then provide additional intuition on the quantities involved and discuss the limitations of this analysis\.

##### Setup\.

Consider a modelf​\(xi;θ\)f\(x\_\{i\};\\theta\)with parametersθ∈ℝP\\theta\\in\\mathbb\{R\}^\{P\}, trained on examples\(xi,yi\)\(x\_\{i\},y\_\{i\}\)grouped by PID sourceg∈\{U, Red, Syn\}g\\in\\\{\\text\{U, Red, Syn\}\\\}, with𝒢\\mathcal\{G\}denoting the index set of groupgg\. As in the main text, we form residualssi\(g\)=yi−f​\(xi;θ\)s^\{\(g\)\}\_\{i\}=y\_\{i\}\-f\(x\_\{i\};\\theta\)fori∈𝒢i\\in\\mathcal\{G\}and define the gradient update direction in parameter space asvg=J​\(θ\)⊤​s\(g\)v\_\{g\}=J\(\\theta\)^\{\\top\}s^\{\(g\)\}, whereJ​\(θ\)i,k=∂f​\(xi;θ\)/∂θkJ\(\\theta\)\_\{i,k\}=\\partial f\(x\_\{i\};\\theta\)/\\partial\\theta\_\{k\}is the Jacobian\. The empirical NTK isK​\(θ\)=J​\(θ\)​J​\(θ\)⊤K\(\\theta\)=J\(\\theta\)J\(\\theta\)^\{\\top\}, with entriesKi​j=∇θf​\(xi;θ\)⋅∇θf​\(xj;θ\)K\_\{ij\}=\\nabla\_\{\\theta\}f\(x\_\{i\};\\theta\)\\cdot\\nabla\_\{\\theta\}f\(x\_\{j\};\\theta\)measuring the inner product between parameter\-space gradients of two examples, and thus capturing how coupled their predictions are through the shared parameters\.

##### NTK strength and cosine similarity\.

The two quantities reported in Sec\.[3\.1](https://arxiv.org/html/2606.09853#S3.SS1)admit natural interpretations\. The NTK strength for groupggcan be rewritten as the Rayleigh quotient ofK​\(θ\)K\(\\theta\)with respect tos\(g\)s^\{\(g\)\}:

λg=‖vg‖2‖s\(g\)‖2=s\(g\)⊤​K​\(θ\)​s\(g\)s\(g\)⊤​s\(g\),\\lambda\_\{g\}=\\frac\{\\\|v\_\{g\}\\\|^\{2\}\}\{\\\|s^\{\(g\)\}\\\|^\{2\}\}=\\frac\{s^\{\(g\)\\top\}K\(\\theta\)\\,s^\{\(g\)\}\}\{s^\{\(g\)\\top\}s^\{\(g\)\}\},which measures how efficiently the network converts prediction errors from groupgginto parameter updates, independently of error magnitude\. The cosine similarity between update directions,

cos⁡\(g,h\)=⟨vg,vh⟩‖vg‖​‖vh‖=s\(g\)⊤​K​\(θ\)​s\(h\)s\(g\)⊤​K​\(θ\)​s\(g\)⋅s\(h\)⊤​K​\(θ\)​s\(h\),\\cos\(g,h\)=\\frac\{\\langle v\_\{g\},v\_\{h\}\\rangle\}\{\\\|v\_\{g\}\\\|\\,\\\|v\_\{h\}\\\|\}=\\frac\{s^\{\(g\)\\top\}K\(\\theta\)\\,s^\{\(h\)\}\}\{\\sqrt\{s^\{\(g\)\\top\}K\(\\theta\)\\,s^\{\(g\)\}\}\\cdot\\sqrt\{s^\{\(h\)\\top\}K\(\\theta\)\\,s^\{\(h\)\}\}\},quantifies interference between two PID sources, with near\-zero values indicating that they occupy approximately orthogonal subspaces in parameter space with minimal gradient interference\.

##### Limitations\.

Three caveats are worth keeping in mind when interpreting these results\. First, the NTK characterizes training dynamics in the linearized regime whereK​\(θ\)K\(\\theta\)varies slowly\(Chizatet al\.,[2019](https://arxiv.org/html/2606.09853#bib.bib115); Ghorbaniet al\.,[2019](https://arxiv.org/html/2606.09853#bib.bib117)\), and is therefore best read as a local diagnostic of how the current parameterization processes each PID source rather than a global account of feature learning over training\. This matches our use: we trackλg\\lambda\_\{g\}andcos⁡\(g,h\)\\cos\(g,h\)throughout training to measure the per\-step gradient signal and interference each source produces, which is the quantity at issue when asking whether synergy is suppressed by competing sources\. Second, projecting the NTK along residual\-weighted directionss\(g\)s^\{\(g\)\}yields group\-level summaries that conflate kernel geometry with the current error distribution, meaning that observed differences inλg\\lambda\_\{g\}orcos⁡\(g,h\)\\cos\(g,h\)across PID sources may partly reflect differences in error magnitude or sampling frequency rather than intrinsic properties of the kernel\. Third, the PID\-controlled XOR places each source on disjoint feature blocks and activates one source per example, which biases first\-layer parameter gradients toward orthogonal supports across sources; near\-zerocos⁡\(g,h\)\\cos\(g,h\)therefore reflects an absence of destructive competition rather than a strong claim of zero coupling, since the design itself favors orthogonality\. The strength asymmetryλSyn≥λU1,λR\\lambda\_\{\\mathrm\{Syn\}\}\\geq\\lambda\_\{U\_\{1\}\},\\lambda\_\{R\}and the overfitting signature in Fig\.[1](https://arxiv.org/html/2606.09853#S3.F1)are observations about training dynamics that do not depend on this orthogonal block structure\.

Our NTK\-based analysis should therefore be read as a local diagnostic of gradient interference and learning signal strength, not as a complete characterization of how PID structure is encoded in learned representations\. Within these limits, it establishes that synergistic examples receive the strongest learning signal of any PID source and show no destructive interference with the other sources during training, evidence that complements the overfitting signature in Fig\.[1](https://arxiv.org/html/2606.09853#S3.F1)\(right\) as motivation for the SynIB objective\.