Token-Level Cross-Modal Transformer with Contrastive Multi-Task Learning for Breast Cancer Subtype Classification and Survival Prediction

# Token-Level Cross-Modal Transformer with Contrastive Multi-Task Learning for Breast Cancer Subtype Classification and Survival Prediction
Source: [https://arxiv.org/html/2607.16233](https://arxiv.org/html/2607.16233)
Byungwon Min2,‡ 1School of Digital Arts, Jiangxi Arts & Ceramics Technology Institute, Jingdezhen 330001, China 2Department of IT Engineering, Mokwon University, Daejeon 35349, South Korea ∗Correspondence: bentondoucet@gmail\.com †Current address: Department of IT Engineering, Mokwon University, Daejeon 35349, South Korea

###### Abstract

Integrating heterogeneous genomic and clinical modalities for joint cancer subtype classification and survival prediction remains a key challenge in precision oncology\. Existing approaches suffer from three limitations: \(1\) they treat each modality as a monolithic feature vector, precluding fine\-grained token\-level interactions across modalities; \(2\) cross\-modal fusion is typically performed through linear weighting or late averaging rather than structured token exchange; and \(3\) survival and classification objectives are optimized independently, missing a joint regularization signal\. We present UMMT \(Unified Multi\-Modal Transformer\), a framework that addresses all three failure points through token\-level cross\-modal fusion\. Specifically, each modality’s compressed latent representation is treated as a*modality token*in a Transformer sequence, enabling full token\-level self\-attention across modalities via a Cross\-Modal Transformer \(CMT\) encoder\. We introduce a cross\-modal contrastive learning objective \(InfoNCE\) that aligns patient representations across modalities by pulling same\-patient tokens together while pushing inter\-patient tokens apart\. A non\-linear DeepSurv survival head replaces the standard linear Cox model to capture non\-proportional hazard patterns\. All objectives, classification, survival, and contrastive alignment, are jointly optimized in a multi\-task learning framework\. On the METABRIC dataset \(n=1,981\), UMMT achieves 79\.8% accuracy and 0\.767 macro F1 for 6\-class molecular subtyping, and a concordance index of 0\.741 for survival prediction, outperforming PORPOISE \(adapted\), DeepSurv, MOFA\+, and OmiVAE across all metrics\. Paired t\-tests and bootstrap confidence intervals confirm statistical significance \(p<0\.01p<0\.01vs\. all baselines\)\. External validation on TCGA\-BRCA \(n=1,089\) yields a C\-index of 0\.718, demonstrating generalizability\. Attention analysis reveals that the CMT learns interpretable cross\-modal interaction patterns, with mRNA expression and clinical features forming the dominant inter\-token communication pathways\. Our code is available at[https://github\.com/liushuxing8888student\.usm\.my/breast\-cancer\-mt](https://github.com/liushuxing8888student.usm.my/breast-cancer-mt)\.

Keywords:Breast cancer, multi\-modal learning, Transformer, token\-level fusion, cross\-modal attention, contrastive learning, DeepSurv, survival analysis, precision oncology\.

## 1Introduction

Breast cancer remains the leading cause of cancer\-related mortality among women worldwide, with approximately 2\.3 million new cases diagnosed annually\[[1](https://arxiv.org/html/2607.16233#bib.bib1)\]\. The disease exhibits distinct molecular heterogeneity, encompassing at least six intrinsic subtypes \(Luminal A, Luminal B, HER2\-enriched, Claudin\-low, Basal\-like, and Normal\-like\), each with different clinical trajectories, treatment sensitivities, and survival outcomes\[[2](https://arxiv.org/html/2607.16233#bib.bib2)\]\. Accurate molecular subtyping and prognostic stratification are foundational to personalized therapeutic decision\-making\.

Modern clinical oncology generates rich multi\-modal data for each patient, including clinical records, mRNA expression profiles, copy number alteration \(CNA\) signatures, DNA methylation patterns, and somatic mutation data\. Each modality captures a complementary facet of tumor biology, and their integration should yield more reliable predictions than any single modality alone\. Yet existing multi\-modal approaches suffer from three fundamental failure points that this work directly addresses\.

Failure Point 1: Monolithic modality representations preclude token\-level interaction\.Current methods encode each modality as a single fixed\-dimensional vector, then concatenate or sum these vectors for downstream prediction\[[3](https://arxiv.org/html/2607.16233#bib.bib3),[4](https://arxiv.org/html/2607.16233#bib.bib4)\]\. This design treats a500500\-dimensional mRNA profile \(containing information from hundreds of genes\) as an atomic unit, making it impossible for the model to align specific genes from one modality with corresponding features from another\. Cross\-modal signals operate at a*finer granularity*: a mutation inTP53may manifest as altered expression of its transcriptional targets, and this relationship disappears when entire modalities are collapsed into monolithic embeddings\.

Failure Point 2: Fusion strategies lack structured cross\-modal exchange\.The predominant fusion paradigms \(early concatenation, late averaging, and attention\-weighted summation\[[5](https://arxiv.org/html/2607.16233#bib.bib5)\]\) are static or linear in nature\. Even attention\-based fusion mechanisms typically compute a scalar importance weight per modality and then perform weighted summation, which constitutes linear mixing in the representation space\. No existing method provides*structured exchange*where information from one modality can selectively attend to, query, and update information from another modality at the token level\. Such token\-level cross\-modal communication is precisely what the Transformer self\-attention mechanism was designed to enable\[[6](https://arxiv.org/html/2607.16233#bib.bib6)\], yet prior work in multi\-omics fusion has not fully exploited this capability\.

Failure Point 3: Disjoint optimization of classification and survival objectives\.Most existing frameworks treat molecular subtyping and survival prediction as independent tasks, training separate models for each objective\[[7](https://arxiv.org/html/2607.16233#bib.bib7),[8](https://arxiv.org/html/2607.16233#bib.bib8)\]\. This ignores the intrinsic biological coupling between cancer subtypes and prognosis: subtype is a categorical label derived from molecular signatures that inherently correlate with survival trajectories\. Jointly optimizing these objectives yields combined effects where the classification task regularizes the survival representation and vice versa\. Furthermore, no existing multi\-modal survival framework incorporates*cross\-modal contrastive alignment*to enforce representation invariance across modalities, which would be particularly beneficial when modalities are noisy or partially missing\.

Our contributions\.We propose UMMT, a structurally novel framework that replaces all three failure modes with principled alternatives:

1. \(1\)Token\-Level Cross\-Modal Representation \(Structural Innovation\)\.Rather than encoding each modality as a monolithic vector, UMMT projects each modality’s latent representation into a*sequence of modality tokens*that form the input to a Cross\-Modal Transformer \(CMT\)\. Each token corresponds to one modality \(clinical, mRNA, CNA, methylation, mutation\), and the CMT processes them through full bidirectional self\-attention, enabling fine\-grained token\-level information exchange\. This is a structural departure from concatenation\-based fusion: the CMT learns*which*modalities should attend to*which*other modalities for each patient, producing an interaction matrix rather than a scalar weight\.
2. \(2\)Cross\-Modal Contrastive Learning \(Alignment Objective\)\.We introduce a modality\-aware InfoNCE loss that pulls together the CMT outputs of the same patient across different modalities while pushing apart representations of different patients\. This explicitly enforces cross\-modal representation consistency at the patient level, grounded in the mutual information maximization principle\[[10](https://arxiv.org/html/2607.16233#bib.bib10)\]\.
3. \(3\)Unified Multi\-Task Learning with DeepSurv Head \(Joint Optimization\)\.All three objectives, molecular subtype classification via weighted cross\-entropy, survival prediction via a discrete\-time DeepSurv head \(replacing linear Cox\), and cross\-modal contrastive alignment via InfoNCE, are optimized jointly in an end\-to\-end fashion\. The DeepSurv head captures non\-proportional hazard patterns that the standard Cox model cannot express\.
4. \(4\)Rigorous Statistical Validation\.We provide bootstrap confidence intervals \(n=1,000n=1\{,\}000resamples\), pairedtt\-tests against all baselines, calibration curves with Integrated Brier Score \(IBS\), and external validation on the TCGA\-BRCA cohort \(n=1,089n=1\{,\}089\)\. Strong SOTA baselines include MOFA\+\[[23](https://arxiv.org/html/2607.16233#bib.bib23)\], DeepHit\[[21](https://arxiv.org/html/2607.16233#bib.bib21)\], and OmiVAE\[[13](https://arxiv.org/html/2607.16233#bib.bib13)\]\.

The remainder of this paper is organized as follows\. Section[2](https://arxiv.org/html/2607.16233#S2)reviews prior work and identifies the gaps that motivate our contributions\. Section[3](https://arxiv.org/html/2607.16233#S3)presents the UMMT architecture in detail\. Section[4](https://arxiv.org/html/2607.16233#S4)describes the experimental setup, datasets, baselines, and evaluation protocol\. Section[5](https://arxiv.org/html/2607.16233#S5)reports quantitative and qualitative results\. Section[6](https://arxiv.org/html/2607.16233#S6)interprets the findings, analyzes the learned interaction patterns, and discusses limitations\. Section[7](https://arxiv.org/html/2607.16233#S7)concludes\.

## 2Related Work

### 2\.1Multi\-Modal Fusion for Cancer Prognosis

Multi\-modal integration in computational oncology has progressed through several distinct paradigms\. Early work by Cheerla and Gevaert\[[3](https://arxiv.org/html/2607.16233#bib.bib3)\]concatenated clinical and genomic features into a single vector before passing them through a feed\-forward network, an*early fusion*strategy that treats all modalities symmetrically but cannot model cross\-modal interactions\. Huang et al\.\[[4](https://arxiv.org/html/2607.16233#bib.bib4)\]proposed an*attention\-based fusion*framework that computes modality\-level importance weights and performs weighted summation\. More recently, the PORPOISE framework\[[5](https://arxiv.org/html/2607.16233#bib.bib5)\]adopted a Transformer architecture for histology–genomic fusion but retained a concatenation\-based design\. Pathomic Fusion\[[11](https://arxiv.org/html/2607.16233#bib.bib11)\]employed gated attention to combine histopathological and genomic features\.

However, all these methods share a common architectural limitation: they treat each modality’s representation as an*atomic unit*that is either concatenated, summed, or linearly weighted\. None perform token\-level cross\-modal attention where individual features from one modality directly interact with features from another\. The UMMT framework breaks this pattern by treating compressed modality representations as a*token sequence*processed by a Cross\-Modal Transformer, enabling full pairwise interaction at the token level\.

### 2\.2Denoising Autoencoders for Genomic Representation Learning

Genomic data is characterized by high dimensionality \(thousands of features\), considerable measurement noise, and frequent missing values\. Denoising autoencoders \(DAEs\)\[[9](https://arxiv.org/html/2607.16233#bib.bib9)\]address these challenges by learning to reconstruct clean inputs from corrupted versions, thereby capturing reliable latent representations\. Chaudhary et al\.\[[12](https://arxiv.org/html/2607.16233#bib.bib12)\]applied DAEs to multi\-omics integration for hepatocellular carcinoma survival prediction\. Zhang et al\.\[[13](https://arxiv.org/html/2607.16233#bib.bib13)\]extended DAEs to cancer subtyping across multiple cancer types\.

Within UMMT, each modality is independently processed by its own DAE, which serves two critical purposes: \(1\) dimensionality reduction from the original feature space \(dmd\_\{m\}up to 500\) to a compact latent space \(dlatent=64d\_\{\\text\{latent\}\}=64\), and \(2\) noise resilience through Gaussian corruption during training\. The resulting latent vectors serve as the “tokens” that are fed into the CMT\.

### 2\.3Cross\-Modal Attention and Contrastive Learning

Cross\-modal attention has been extensively explored in vision–language modeling\[[15](https://arxiv.org/html/2607.16233#bib.bib15),[16](https://arxiv.org/html/2607.16233#bib.bib16)\], where Transformers process aligned sequences from different modalities\. However, its application to multi\-omics fusion remains limited\. The key insight of UMMT is that genomic modalities, despite having different dimensionalities and semantic scales, can be treated as a*set of tokens*after compression into a shared latent space, enabling Transformer\-based cross\-modal attention\.

Simultaneously, contrastive learning has significantly advanced representation learning across domains\[[17](https://arxiv.org/html/2607.16233#bib.bib17),[18](https://arxiv.org/html/2607.16233#bib.bib18)\]\. The InfoNCE loss\[[10](https://arxiv.org/html/2607.16233#bib.bib10)\]maximizes mutual information between different views of the same data point\. In the multi\-modal setting, contrastive objectives have been applied in CLIP\[[19](https://arxiv.org/html/2607.16233#bib.bib19)\]for aligning images and text\. To our knowledge, UMMT is the first framework to apply cross\-modal contrastive learning to genomic multi\-modal fusion for cancer prognosis, pulling together same\-patient representations across modalities\.

### 2\.4Deep Survival Analysis

The Cox proportional hazards model\[[20](https://arxiv.org/html/2607.16233#bib.bib20)\]remains the standard approach for survival analysis\. Deep learning extensions such as DeepSurv\[[7](https://arxiv.org/html/2607.16233#bib.bib7)\]and Cox\-nnet\[[8](https://arxiv.org/html/2607.16233#bib.bib8)\]introduced non\-linear hazard modeling while preserving the proportional hazards assumption\. More flexible approaches include DeepHit\[[21](https://arxiv.org/html/2607.16233#bib.bib21)\], which directly estimates the joint distribution of survival times and events, and discrete\-time survival models\[[22](https://arxiv.org/html/2607.16233#bib.bib22)\]\. In UMMT, we replace the standard linear Cox head with a DeepSurv\-style non\-linear head that can capture complex, non\-proportional risk patterns while maintaining the Cox partial likelihood as the training objective\.

## 3Methodology

### 3\.1Problem Formulation

Let𝒫=\{p1,p2,…,pN\}\\mathcal\{P\}=\\\{p\_\{1\},p\_\{2\},\\ldots,p\_\{N\}\\\}denote a cohort ofNNpatients\. Each patientpip\_\{i\}is associated withM=5M=5modalities: clinical features, mRNA expression, copy number alterations \(CNA\), DNA methylation, and somatic mutations\. Formally, we observe𝐗i=\{𝐱i\(1\),𝐱i\(2\),…,𝐱i\(M\)\}\\mathbf\{X\}\_\{i\}=\\\{\\mathbf\{x\}\_\{i\}^\{\(1\)\},\\mathbf\{x\}\_\{i\}^\{\(2\)\},\\ldots,\\mathbf\{x\}\_\{i\}^\{\(M\)\}\\\}where𝐱i\(m\)∈ℝdm\\mathbf\{x\}\_\{i\}^\{\(m\)\}\\in\\mathbb\{R\}^\{d\_\{m\}\}represents the feature vector for modalitymm\. For each patient, two prediction targets are defined:

- •Molecular subtype classification:yi∈\{0,1,…,C−1\}y\_\{i\}\\in\\\{0,1,\\ldots,C\-1\\\}withC=6C=6\(Luminal A, Luminal B, HER2\-enriched, Claudin\-low, Basal\-like, Normal\-like\)\.
- •Survival prediction:\(ti,δi\)\(t\_\{i\},\\delta\_\{i\}\)whereti∈ℝ\+t\_\{i\}\\in\\mathbb\{R\}^\{\+\}is the relapse\-free survival time \(months\) andδi∈\{0,1\}\\delta\_\{i\}\\in\\\{0,1\\\}is the event indicator \(δi=1\\delta\_\{i\}=1for recurrence or death\)\.

### 3\.2Architectural Overview

UMMT comprises five core components arranged in a structured pipeline \(Figure[1](https://arxiv.org/html/2607.16233#S3.F1)\):

1. 1\.Modality\-Specific Denoising Autoencoders:Each of theMMmodalities is compressed independently through its own DAE into a shared latent spaceℝdlatent\\mathbb\{R\}^\{d\_\{\\text\{latent\}\}\}\.
2. 2\.Token Projection and Embedding:TheMMlatent vectors serve as modality tokens, each projected toℝdmodel\\mathbb\{R\}^\{d\_\{\\text\{model\}\}\}and augmented with a learned token\-type embedding\.
3. 3\.Cross\-Modal Transformer \(CMT\):The token sequence passes throughLLlayers of bidirectional self\-attention, enabling full token\-level interaction across modalities\. A CLS token is prepended to aggregate cross\-modal information\.
4. 4\.Cross\-Modal Contrastive Head:An InfoNCE loss aligns the CLS representations of the same patient across modalities by maximizing agreement between augmented views\.
5. 5\.Dual Task Heads:A classification head \(weighted cross\-entropy\) and a DeepSurv survival head \(non\-linear Cox partial likelihood\) operate on the fused CLS embedding\.

![Refer to caption](https://arxiv.org/html/2607.16233v1/x1.png)Figure 1:Overview of the UMMT architecture\. The key structural innovation is the treatment of each modality’s latent representation as a token in the Cross\-Modal Transformer sequence, enabling full pairwise token\-level interaction \(rather than concatenation or weighted summation\)\.
### 3\.3Denoising Autoencoder for Modality Compression

Each modalitymmis encoded by an independent denoising autoencoderD​A​EmDAE\_\{m\}\. The encoderEm:ℝdm→ℝdlatentE\_\{m\}:\\mathbb\{R\}^\{d\_\{m\}\}\\to\\mathbb\{R\}^\{d\_\{\\text\{latent\}\}\}compresses the input into a latent representation, while the decoderDm:ℝdlatent→ℝdmD\_\{m\}:\\mathbb\{R\}^\{d\_\{\\text\{latent\}\}\}\\to\\mathbb\{R\}^\{d\_\{m\}\}reconstructs the original input\. The encoder consists of three fully\-connected layers with batch normalization, ReLU activation, and dropout \(p=0\.2p=0\.2\):

𝐡1\\displaystyle\\mathbf\{h\}\_\{1\}=Dropout​\(ReLU​\(BN​\(𝐖1​𝐱\+𝐛1\)\)\)\\displaystyle=\\text\{Dropout\}\(\\text\{ReLU\}\(\\text\{BN\}\(\\mathbf\{W\}\_\{1\}\\mathbf\{x\}\+\\mathbf\{b\}\_\{1\}\)\)\)\(1\)𝐡2\\displaystyle\\mathbf\{h\}\_\{2\}=Dropout​\(ReLU​\(BN​\(𝐖2​𝐡1\+𝐛2\)\)\)\\displaystyle=\\text\{Dropout\}\(\\text\{ReLU\}\(\\text\{BN\}\(\\mathbf\{W\}\_\{2\}\\mathbf\{h\}\_\{1\}\+\\mathbf\{b\}\_\{2\}\)\)\)𝐳\(m\)\\displaystyle\\mathbf\{z\}^\{\(m\)\}=𝐖3​𝐡2\+𝐛3\\displaystyle=\\mathbf\{W\}\_\{3\}\\mathbf\{h\}\_\{2\}\+\\mathbf\{b\}\_\{3\}
where the hidden dimensions are\(dm→512→256→dlatent\)\(d\_\{m\}\\to 512\\to 256\\to d\_\{\\text\{latent\}\}\)\. The decoder mirrors this structure in reverse:\(dlatent→256→512→dm\)\(d\_\{\\text\{latent\}\}\\to 256\\to 512\\to d\_\{m\}\)\. During training, Gaussian noise withσ=0\.1\\sigma=0\.1is injected into inputs to promote robustness\. The training objective is:

ℒrecon\(m\)=‖𝐱\(m\)−Dm​\(Em​\(𝐱~\(m\)\)\)‖22\+λctr​‖𝐖1\(m\)‖F2\\mathcal\{L\}\_\{\\text\{recon\}\}^\{\(m\)\}=\\\|\\mathbf\{x\}^\{\(m\)\}\-D\_\{m\}\(E\_\{m\}\(\\tilde\{\\mathbf\{x\}\}^\{\(m\)\}\)\)\\\|\_\{2\}^\{2\}\+\\lambda\_\{\\text\{ctr\}\}\\\|\\mathbf\{W\}\_\{1\}^\{\(m\)\}\\\|\_\{F\}^\{2\}\(2\)
where𝐱~\(m\)=𝐱\(m\)\+ϵ\\tilde\{\\mathbf\{x\}\}^\{\(m\)\}=\\mathbf\{x\}^\{\(m\)\}\+\\epsilon,ϵ∼𝒩​\(0,σ2​𝐈\)\\epsilon\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}\\mathbf\{I\}\), and the contractive penalty‖𝐖1\(m\)‖F2\\\|\\mathbf\{W\}\_\{1\}^\{\(m\)\}\\\|\_\{F\}^\{2\}\(λctr=10−4\\lambda\_\{\\text\{ctr\}\}=10^\{\-4\}\) encourages the encoder’s first\-layer Jacobian to be small, improving local stability\.

### 3\.4Token Projection and Embedding

After DAE compression, each modalitymmis represented by a latent vector𝐳\(m\)∈ℝdlatent\\mathbf\{z\}^\{\(m\)\}\\in\\mathbb\{R\}^\{d\_\{\\text\{latent\}\}\}\. This is a critical design choice: rather than concatenating these vectors, we treat them as a*token sequence*\. Each latent vector is first projected to the Transformer dimensiondmodeld\_\{\\text\{model\}\}via a learned linear projection:

𝐞\(m\)=𝐖proj​𝐳\(m\)\+𝐛proj,𝐞\(m\)∈ℝdmodel\\mathbf\{e\}^\{\(m\)\}=\\mathbf\{W\}\_\{\\text\{proj\}\}\\mathbf\{z\}^\{\(m\)\}\+\\mathbf\{b\}\_\{\\text\{proj\}\},\\quad\\mathbf\{e\}^\{\(m\)\}\\in\\mathbb\{R\}^\{d\_\{\\text\{model\}\}\}\(3\)
We then add a*learned modality\-type embedding*𝐭\(m\)∈ℝdmodel\\mathbf\{t\}^\{\(m\)\}\\in\\mathbb\{R\}^\{d\_\{\\text\{model\}\}\}to encode which modality each token represents, analogous to segment embeddings in BERT\[[14](https://arxiv.org/html/2607.16233#bib.bib14)\]\. The final token sequence is:

𝐄=\[𝐜;𝐞\(1\)\+𝐭\(1\);𝐞\(2\)\+𝐭\(2\);…;𝐞\(M\)\+𝐭\(M\)\]∈ℝ\(M\+1\)×dmodel\\mathbf\{E\}=\[\\mathbf\{c\};\\mathbf\{e\}^\{\(1\)\}\\\!\+\\\!\\mathbf\{t\}^\{\(1\)\};\\mathbf\{e\}^\{\(2\)\}\\\!\+\\\!\\mathbf\{t\}^\{\(2\)\};\\ldots;\\mathbf\{e\}^\{\(M\)\}\\\!\+\\\!\\mathbf\{t\}^\{\(M\)\}\]\\in\\mathbb\{R\}^\{\(M\+1\)\\times d\_\{\\text\{model\}\}\}\(4\)
where𝐜∈ℝdmodel\\mathbf\{c\}\\in\\mathbb\{R\}^\{d\_\{\\text\{model\}\}\}is a learnable CLS token prepended to the sequence\. Each modality is now represented as a single token, one that has been non\-linearly compressed from the originaldmd\_\{m\}\-dimensional feature space through the DAE\.

### 3\.5Cross\-Modal Transformer \(CMT\)

The Cross\-Modal Transformer processes the token sequence throughLLlayers of bidirectional multi\-head self\-attention\. Each layer performs:

𝐇ℓ′\\displaystyle\\mathbf\{H\}^\{\\prime\}\_\{\\ell\}=LayerNorm​\(𝐇ℓ−1\+MSA​\(𝐇ℓ−1\)\)\\displaystyle=\\text\{LayerNorm\}\(\\mathbf\{H\}\_\{\\ell\-1\}\+\\text\{MSA\}\(\\mathbf\{H\}\_\{\\ell\-1\}\)\)\(5\)𝐇ℓ\\displaystyle\\mathbf\{H\}\_\{\\ell\}=LayerNorm​\(𝐇ℓ′\+FFN​\(𝐇ℓ′\)\)\\displaystyle=\\text\{LayerNorm\}\(\\mathbf\{H\}^\{\\prime\}\_\{\\ell\}\+\\text\{FFN\}\(\\mathbf\{H\}^\{\\prime\}\_\{\\ell\}\)\)
where𝐇0=𝐄\\mathbf\{H\}\_\{0\}=\\mathbf\{E\},MSAdenotes multi\-head self\-attention, andFFNis a two\-layer feed\-forward network with hidden dimensiondff=512d\_\{\\text\{ff\}\}=512and ReLU activation\. The multi\-head attention withHHheads computes:

Attention​\(𝐐,𝐊,𝐕\)=softmax​\(𝐐𝐊⊤dk\)​𝐕\\text\{Attention\}\(\\mathbf\{Q\},\\mathbf\{K\},\\mathbf\{V\}\)=\\text\{softmax\}\\left\(\\frac\{\\mathbf\{Q\}\\mathbf\{K\}^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\right\)\\mathbf\{V\}\(6\)
The CLS token’s final representation𝐡CLS=𝐇L​\[0,:\]\\mathbf\{h\}\_\{\\text\{CLS\}\}=\\mathbf\{H\}\_\{L\}\[0,:\]serves as the*fused patient embedding*that has attended to all modality tokens throughLLlayers of cross\-modal interaction\.

#### 3\.5\.1Token Interaction Analysis

A key advantage of the CMT over concatenation\-based fusion is that the attention mechanism produces an interaction matrix𝐀∈ℝ\(M\+1\)×\(M\+1\)\\mathbf\{A\}\\in\\mathbb\{R\}^\{\(M\+1\)\\times\(M\+1\)\}at each layer, where𝐀i​j\\mathbf\{A\}\_\{ij\}quantifies how much tokeniiattends to tokenjj\. For the CLS token,𝐀CLS,m\\mathbf\{A\}\_\{\\text\{CLS\},m\}captures how much the fused representation relies on modalitymm\. This is a generalization of modality attention that emerges from structured cross\-modal exchange rather than from a separate learned weighting module\. For non\-CLS tokens,𝐀m1,m2\\mathbf\{A\}\_\{m\_\{1\},m\_\{2\}\}captures the direct cross\-modal interaction between modalitiesm1m\_\{1\}andm2m\_\{2\}\.

### 3\.6Cross\-Modal Contrastive Learning

To explicitly enforce representation consistency across modalities, we introduce a cross\-modal contrastive learning objective\. The intuition is that for a given patientpip\_\{i\}, the CMT output𝐡CLS\(i\)\\mathbf\{h\}\_\{\\text\{CLS\}\}^\{\(i\)\}should be close to the outputs obtained from the same patient under different modality dropout patterns, while being far from outputs of different patients\.

We construct augmented views by randomly masking a subset of modalities during training \(dropout rateρ=0\.2\\rho=0\.2\)\. For a batch ofBBpatients, let𝐡i\\mathbf\{h\}\_\{i\}and𝐡i\+\\mathbf\{h\}\_\{i\}^\{\+\}denote the CLS embeddings from two forward passes with independently sampled modality masks for patientii\. The InfoNCE loss\[[10](https://arxiv.org/html/2607.16233#bib.bib10)\]is:

ℒcont=−1B​∑i=1Blog⁡exp⁡\(sim​\(𝐡i,𝐡i\+\)/τ\)∑j=1Bexp⁡\(sim​\(𝐡i,𝐡j\+\)/τ\)\\mathcal\{L\}\_\{\\text\{cont\}\}=\-\\frac\{1\}\{B\}\\sum\_\{i=1\}^\{B\}\\log\\frac\{\\exp\(\\text\{sim\}\(\\mathbf\{h\}\_\{i\},\\mathbf\{h\}\_\{i\}^\{\+\}\)/\\tau\)\}\{\\sum\_\{j=1\}^\{B\}\\exp\(\\text\{sim\}\(\\mathbf\{h\}\_\{i\},\\mathbf\{h\}\_\{j\}^\{\+\}\)/\\tau\)\}\(7\)
wheresim​\(𝐮,𝐯\)=𝐮⊤​𝐯/‖𝐮‖​‖𝐯‖\\text\{sim\}\(\\mathbf\{u\},\\mathbf\{v\}\)=\\mathbf\{u\}^\{\\top\}\\mathbf\{v\}/\\\|\\mathbf\{u\}\\\|\\\|\\mathbf\{v\}\\\|is cosine similarity andτ=0\.1\\tau=0\.1is the temperature parameter\. This loss maximizes the mutual information between different views of the same patient \(Equation[7](https://arxiv.org/html/2607.16233#S3.E7)numerator\) while minimizing it across different patients \(denominator\)\. In the multi\-modal context, this encourages the model to extract representations that are invariant to which specific modalities are observed, a desirable property for clinical deployment where some modalities may be unavailable\.

### 3\.7Dual Task Heads

#### 3\.7\.1Classification Head

The fused patient embedding𝐡CLS\\mathbf\{h\}\_\{\\text\{CLS\}\}is passed through a classification head with one hidden layer:

𝐡cls\\displaystyle\\mathbf\{h\}\_\{\\text\{cls\}\}=Dropout​\(ReLU​\(BN​\(𝐖c\(1\)​𝐡CLS\+𝐛c\(1\)\)\)\)\\displaystyle=\\text\{Dropout\}\(\\text\{ReLU\}\(\\text\{BN\}\(\\mathbf\{W\}\_\{c\}^\{\(1\)\}\\mathbf\{h\}\_\{\\text\{CLS\}\}\+\\mathbf\{b\}\_\{c\}^\{\(1\)\}\)\)\)\(8\)𝐲^\\displaystyle\\hat\{\\mathbf\{y\}\}=softmax​\(𝐖c\(2\)​𝐡cls\+𝐛c\(2\)\)\\displaystyle=\\text\{softmax\}\(\\mathbf\{W\}\_\{c\}^\{\(2\)\}\\mathbf\{h\}\_\{\\text\{cls\}\}\+\\mathbf\{b\}\_\{c\}^\{\(2\)\}\)
The classification loss uses weighted cross\-entropy to mitigate class imbalance:

ℒcls=−∑c=1Cwc⋅yc​log⁡\(y^c\)\\mathcal\{L\}\_\{\\text\{cls\}\}=\-\\sum\_\{c=1\}^\{C\}w\_\{c\}\\cdot y\_\{c\}\\log\(\\hat\{y\}\_\{c\}\)\(9\)
wherewcw\_\{c\}are inversely proportional to class frequency:\[1\.0,1\.5,2\.9,3\.8,3\.4,4\.6\]\[1\.0,1\.5,2\.9,3\.8,3\.4,4\.6\]for the six subtypes ordered by decreasing frequency\.

#### 3\.7\.2DeepSurv Survival Head

We replace the standard linear Cox head with a*DeepSurv\-style non\-linear head*\[[7](https://arxiv.org/html/2607.16233#bib.bib7)\]that captures non\-linear risk functions while maintaining the Cox proportional hazards framework:

𝐡surv\(1\)\\displaystyle\\mathbf\{h\}\_\{\\text\{surv\}\}^\{\(1\)\}=ReLU​\(𝐖s\(1\)​𝐡CLS\+𝐛s\(1\)\)\\displaystyle=\\text\{ReLU\}\(\\mathbf\{W\}\_\{s\}^\{\(1\)\}\\mathbf\{h\}\_\{\\text\{CLS\}\}\+\\mathbf\{b\}\_\{s\}^\{\(1\)\}\)\(10\)𝐡surv\(2\)\\displaystyle\\mathbf\{h\}\_\{\\text\{surv\}\}^\{\(2\)\}=ReLU​\(𝐖s\(2\)​𝐡surv\(1\)\+𝐛s\(2\)\)\\displaystyle=\\text\{ReLU\}\(\\mathbf\{W\}\_\{s\}^\{\(2\)\}\\mathbf\{h\}\_\{\\text\{surv\}\}^\{\(1\)\}\+\\mathbf\{b\}\_\{s\}^\{\(2\)\}\)r^\\displaystyle\\hat\{r\}=𝐰s\(3\)⊤​𝐡surv\(2\)\+bs\(3\)\\displaystyle=\\mathbf\{w\}\_\{s\}^\{\(3\)\\top\}\\mathbf\{h\}\_\{\\text\{surv\}\}^\{\(2\)\}\+b\_\{s\}^\{\(3\)\}
with hidden dimensions\[dmodel→128→64→1\]\[d\_\{\\text\{model\}\}\\to 128\\to 64\\to 1\]and dropout \(p=0\.3p=0\.3\) between layers\. The hazard function ish​\(t\|𝐱\)=h0​\(t\)⋅exp⁡\(r^\)h\(t\|\\mathbf\{x\}\)=h\_\{0\}\(t\)\\cdot\\exp\(\\hat\{r\}\), whereh0​\(t\)h\_\{0\}\(t\)is the unspecified baseline hazard\. The training objective is the Cox partial log\-likelihood:

ℒsurv=−1∑iδi​∑i:δi=1\(r^i−log​∑j:tj≥tiexp⁡\(r^j\)\)\\mathcal\{L\}\_\{\\text\{surv\}\}=\-\\frac\{1\}\{\\sum\_\{i\}\\delta\_\{i\}\}\\sum\_\{i:\\delta\_\{i\}=1\}\\left\(\\hat\{r\}\_\{i\}\-\\log\\sum\_\{j:t\_\{j\}\\geq t\_\{i\}\}\\exp\(\\hat\{r\}\_\{j\}\)\\right\)\(11\)
The DeepSurv head offers two advantages over linear Cox: \(1\) it can learn non\-linear prognostic factors through its hidden layers, and \(2\) the multi\-layer architecture provides additional capacity for modeling complex interactions between the fused embedding and survival outcomes\.

### 3\.8Multi\-Task Joint Training Objective

The complete training objective combines all three losses:

ℒtotal=λcls​ℒcls\+λsurv​ℒsurv\+λcont​ℒcont\+∑m=1Mλrecon​ℒrecon\(m\)\\mathcal\{L\}\_\{\\text\{total\}\}=\\lambda\_\{\\text\{cls\}\}\\mathcal\{L\}\_\{\\text\{cls\}\}\+\\lambda\_\{\\text\{surv\}\}\\mathcal\{L\}\_\{\\text\{surv\}\}\+\\lambda\_\{\\text\{cont\}\}\\mathcal\{L\}\_\{\\text\{cont\}\}\+\\sum\_\{m=1\}^\{M\}\\lambda\_\{\\text\{recon\}\}\\mathcal\{L\}\_\{\\text\{recon\}\}^\{\(m\)\}\(12\)
whereλcls\\lambda\_\{\\text\{cls\}\},λsurv\\lambda\_\{\\text\{surv\}\},λcont\\lambda\_\{\\text\{cont\}\}, andλrecon\\lambda\_\{\\text\{recon\}\}are task\-balancing hyperparameters\. The reconstruction lossesℒrecon\(m\)\\mathcal\{L\}\_\{\\text\{recon\}\}^\{\(m\)\}are pre\-trained independently for each DAE before end\-to\-end fine\-tuning\.

Algorithm[1](https://arxiv.org/html/2607.16233#alg1)summarizes the complete training procedure\.

Algorithm 1UMMT Training Procedure0:Multi\-modal data

\{𝐱\(1\),…,𝐱\(M\)\}\\\{\\mathbf\{x\}^\{\(1\)\},\\dots,\\mathbf\{x\}^\{\(M\)\}\\\}, subtype labels

yy, survival times

tt, events

δ\\delta
0:Trained UMMT parameters

θ\\theta
1:Phase 1: DAE Pre\-training

2:for

m=1m=1to

MMdo

3:Train

D​A​EmDAE\_\{m\}via Eq\. \([2](https://arxiv.org/html/2607.16233#S3.E2)\) on

𝐱\(m\)\\mathbf\{x\}^\{\(m\)\}
4:endfor

5:Phase 2: End\-to\-End Multi\-Task Training

6:foreach epochdo

7:Sample batch of

BBpatients

8:foreach batchdo

9:Encode:

𝐳\(m\)←Em​\(𝐱\(m\)\+𝒩​\(0,σ2​𝐈\)\)\\mathbf\{z\}^\{\(m\)\}\\leftarrow E\_\{m\}\(\\mathbf\{x\}^\{\(m\)\}\+\\mathcal\{N\}\(0,\\sigma^\{2\}\\mathbf\{I\}\)\),

∀m\\forall m
10:Tokenize:

𝐞\(m\)←𝐖proj​𝐳\(m\)\+𝐛proj\\mathbf\{e\}^\{\(m\)\}\\leftarrow\\mathbf\{W\}\_\{\\text\{proj\}\}\\mathbf\{z\}^\{\(m\)\}\+\\mathbf\{b\}\_\{\\text\{proj\}\},

∀m\\forall m
11:Construct:

𝐄←\[𝐜;𝐞\(1\)\+𝐭\(1\);…;𝐞\(M\)\+𝐭\(M\)\]\\mathbf\{E\}\\leftarrow\[\\mathbf\{c\};\\mathbf\{e\}^\{\(1\)\}\\\!\+\\\!\\mathbf\{t\}^\{\(1\)\};\\ldots;\\mathbf\{e\}^\{\(M\)\}\\\!\+\\\!\\mathbf\{t\}^\{\(M\)\}\]
12:Encode:

𝐇L←CMT​\(𝐄\)\\mathbf\{H\}\_\{L\}\\leftarrow\\text\{CMT\}\(\\mathbf\{E\}\)⊳\\trianglerightCross\-modal interaction

13:Fuse:

𝐡CLS←𝐇L​\[0,:\]\\mathbf\{h\}\_\{\\text\{CLS\}\}\\leftarrow\\mathbf\{H\}\_\{L\}\[0,:\]
14:Predict:

𝐲^←ClsHead​\(𝐡CLS\)\\hat\{\\mathbf\{y\}\}\\leftarrow\\text\{ClsHead\}\(\\mathbf\{h\}\_\{\\text\{CLS\}\}\),

r^←DeepSurv​\(𝐡CLS\)\\hat\{r\}\\leftarrow\\text\{DeepSurv\}\(\\mathbf\{h\}\_\{\\text\{CLS\}\}\)
15:Multi\-task loss:

16:

ℒcls←CE​\(𝐲^,y\)\\mathcal\{L\}\_\{\\text\{cls\}\}\\leftarrow\\text\{CE\}\(\\hat\{\\mathbf\{y\}\},y\)⊳\\trianglerightWeighted cross\-entropy

17:

ℒsurv←CoxPLL​\(r^,t,δ\)\\mathcal\{L\}\_\{\\text\{surv\}\}\\leftarrow\\text\{CoxPLL\}\(\\hat\{r\},t,\\delta\)⊳\\trianglerightCox partial log\-likelihood

18:

ℒcont←InfoNCE​\(𝐡CLS\)\\mathcal\{L\}\_\{\\text\{cont\}\}\\leftarrow\\text\{InfoNCE\}\(\\mathbf\{h\}\_\{\\text\{CLS\}\}\)⊳\\trianglerightCross\-modal contrastive

19:

ℒtotal←λcls​ℒcls\+λsurv​ℒsurv\+λcont​ℒcont\\mathcal\{L\}\_\{\\text\{total\}\}\\leftarrow\\lambda\_\{\\text\{cls\}\}\\mathcal\{L\}\_\{\\text\{cls\}\}\+\\lambda\_\{\\text\{surv\}\}\\mathcal\{L\}\_\{\\text\{surv\}\}\+\\lambda\_\{\\text\{cont\}\}\\mathcal\{L\}\_\{\\text\{cont\}\}
20:

θ←θ−η​∇θℒtotal\\theta\\leftarrow\\theta\-\\eta\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\text\{total\}\}
21:endfor

22:endfor

### 3\.9Handling Missing Modalities

The CMT architecture natively supports missing modalities through the sequence masking mechanism\. For patients with missing modalitymm, we replace the corresponding token embedding with a learned\[MASK\]token vector𝐦mask∈ℝdmodel\\mathbf\{m\}\_\{\\text\{mask\}\}\\in\\mathbb\{R\}^\{d\_\{\\text\{model\}\}\}\. The self\-attention mask is adjusted to prevent the CLS and other tokens from attending to masked positions\. This design enables the model to operate even when a large fraction of modalities are absent, a critical requirement for clinical deployment\.

## 4Experimental Setup

### 4\.1Datasets

METABRIC\.The Molecular Taxonomy of Breast Cancer International Consortium \(METABRIC\) dataset\[[25](https://arxiv.org/html/2607.16233#bib.bib25)\]comprises 1,981 primary breast cancer samples with five aligned modalities:

- •Clinical\(80 features\): age, tumor size, stage, grade, lymph node status, ER/HER2/PR status, treatment history\. Categorical variables one\-hot encoded\.
- •mRNA expression\(500\): top 500 highly\-variable genes from Illumina microarrays, Z\-score normalized\.
- •Copy number alterations\(500\): top 500 highly\-variable CNA gene\-level profiles\.
- •DNA methylation\(500\): top 500 probes by variance from promoter methylation\.
- •Somatic mutations\(173\): mutation counts for the top 173 most frequently mutated genes\.

Subtype distribution: Luminal A \(490\), Luminal B \(330\), HER2\-enriched \(171\), Claudin\-low \(129\), Basal\-like \(142\), Normal\-like \(107\)\. Survival endpoint: relapse\-free survival \(RFS\) in months\. The dataset is split 70%/15%/15% stratified by subtype\.

TCGA\-BRCA \(External Validation\)\.The Cancer Genome Atlas Breast Cancer cohort provides an independent validation set of 1,089 patients with matched clinical, mRNA expression \(RNA\-seq, top 500 genes by variance\), and CNA data\. Subtype labels are derived from PAM50 classification\. Survival endpoint is overall survival \(OS\)\. We use TCGA\-BRCA specifically to assess cross\-cohort generalization\.

### 4\.2Preprocessing

Missing values are imputed with column medians for all modalities\. Numerical features are Z\-score normalized\. Categorical clinical variables are one\-hot encoded\. For TCGA\-BRCA, we map gene symbols to match METABRIC gene names and select the common intersecting genes\.

### 4\.3Implementation Details

All experiments are implemented in PyTorch 2\.0\+ and trained on a single NVIDIA GPU\. Table[1](https://arxiv.org/html/2607.16233#S4.T1)lists the full hyperparameter configuration\.

Table 1:Default and optimal hyperparameter configuration\.The Adam optimizer is used with weight decay10−510^\{\-5\}\. A ReduceLROnPlateau scheduler reduces the learning rate by factor 0\.5 after 5 epochs without validation loss improvement\. Early stopping with patience of 15 epochs is applied\. Optuna\[[27](https://arxiv.org/html/2607.16233#bib.bib27)\]with 20 trials and TPE sampling is used for hyperparameter optimization, targeting validation C\-index\.

### 4\.4Baseline Methods

We compare against a thorough set of baselines spanning traditional and state\-of\-the\-art approaches:

- •Early Fusion MLP: All modality features concatenated→\\to3\-layer MLP with ReLU\.
- •Late Fusion: Independent MLP per modality, predictions averaged\.
- •Attention Fusion: Concatenated features \+ self\-attention layer\.
- •DeepSurv\[[7](https://arxiv.org/html/2607.16233#bib.bib7)\]: 3\-layer MLP with Cox loss \(survival only, all modalities\)\.
- •PORPOISE \(adapted\)\[[5](https://arxiv.org/html/2607.16233#bib.bib5)\]: Transformer\-based multi\-modal fusion adapted for tabular genomic data\.
- •MOFA\+\[[23](https://arxiv.org/html/2607.16233#bib.bib23)\]: Probabilistic multi\-omics factor analysis \(non\-deep\-learning baseline\)\.
- •DeepHit\[[21](https://arxiv.org/html/2607.16233#bib.bib21)\]: Deep learning survival model with non\-proportional hazards\.
- •OmiVAE\[[13](https://arxiv.org/html/2607.16233#bib.bib13)\]: Variational autoencoder for multi\-omics integration\.
- •UMMT variants: Ablations removing contrastive loss \(UMMT−cont\{\}\_\{\-\\text\{cont\}\}\), removing DeepSurv head \(UMMT−DS\{\}\_\{\-\\text\{DS\}\}\), and removing CMT \(UMMT−CMT\{\}\_\{\-\\text\{CMT\}\}\)\.

### 4\.5Evaluation Metrics

Classification:Accuracy, macro\-averaged Precision, Recall, and F1\-score, macro\-averaged AUC\-ROC\.

Survival:Concordance index \(C\-index\), time\-dependent AUC at 5 and 10 years, Integrated Brier Score \(IBS\)\[[24](https://arxiv.org/html/2607.16233#bib.bib24)\], calibration curves at 5 and 10 years\.

Statistical Testing:95% confidence intervals via bootstrap \(n=1,000n=1\{,\}000resamples\), pairedtt\-test for pairwise method comparisons\. C\-index differences are evaluated via bootstrap hypothesis testing\.

## 5Results

### 5\.1Molecular Subtype Classification

Table[2](https://arxiv.org/html/2607.16233#S5.T2)reports the classification performance on the METABRIC test set\. UMMT achieves the highest accuracy \(79\.8%\) and macro F1\-score \(76\.7%\), outperforming the strongest baseline \(PORPOISE adapted\) by 4\.4 and 4\.6 percentage points, respectively\.

Table 2:6\-class molecular subtype classification performance on METABRIC test set\. Best results inbold, second\-best underlined\. All UMMT results withp<0\.01p<0\.01vs\. all baselines \(pairedtt\-test\)\.#### 5\.1\.1Ablation Analysis

The three ablation variants provide clear evidence for each component’s contribution:

- •Removing contrastive learning \(UMMT−cont\{\}\_\{\-\\text\{cont\}\}\) reduces accuracy by 1\.7 pp \(0\.798→\\to0\.781\), confirming that cross\-modal alignment improves representation quality\.
- •Replacing DeepSurv with a linear Cox head \(UMMT−DS\{\}\_\{\-\\text\{DS\}\}\) reduces accuracy by 2\.2 pp, indicating that non\-linear survival modeling also benefits classification through multi\-task sharing\.
- •Replacing CMT with attention\-weighted summation \(UMMT−CMT\{\}\_\{\-\\text\{CMT\}\}\) reduces accuracy by 2\.9 pp, the largest drop, confirming that token\-level cross\-modal attention is the most impactful structural innovation\.

### 5\.2Survival Prediction

Table[3](https://arxiv.org/html/2607.16233#S5.T3)presents survival analysis results\. UMMT achieves a C\-index of 0\.741, outperforming all baselines by statistically significant margins\.

Table 3:Survival prediction performance on METABRIC\. IBS: Integrated Brier Score \(lower is better\)\. C\-index differences: UMMT vs\. all baselinesp<0\.01p<0\.01\(bootstrap test\)\.
### 5\.3External Validation: TCGA\-BRCA

To assess cross\-cohort generalizability, we evaluate UMMT trained on METABRIC on the TCGA\-BRCA cohort \(Table[4](https://arxiv.org/html/2607.16233#S5.T4)\)\.

Table 4:External validation on TCGA\-BRCA \(n=1,089\)\. Models trained on METABRIC, evaluated on TCGA\-BRCA without fine\-tuning\.Despite domain shift \(different sequencing platforms, different survival endpoints\), UMMT achieves a C\-index of 0\.718 on TCGA\-BRCA, well above the baselines\. The accuracy drop \(79\.8%→\\to65\.8%\) is expected given platform differences and label noise in cross\-cohort PAM50 assignment, but the relative ordering of methods is preserved\.

### 5\.4Kaplan–Meier Survival Stratification

Figure[2](https://arxiv.org/html/2607.16233#S5.F2)shows Kaplan–Meier curves stratified by UMMT predicted risk \(median split\)\. The high\-risk and low\-risk groups exhibit clear separation \(log\-rankχ2=42\.3\\chi^\{2\}=42\.3,p<0\.001p<0\.001\)\. The 10\-year survival rates are 74\.2% \(95% CI: 71\.1%–77\.3%\) for the low\-risk group and 36\.8% \(95% CI: 33\.5%–40\.1%\) for the high\-risk group, corresponding to a hazard ratio of 2\.64 \(95% CI: 2\.21–3\.15\) between groups\.

![Refer to caption](https://arxiv.org/html/2607.16233v1/x2.png)Figure 2:Kaplan–Meier survival curves stratified by UMMT predicted risk score\. The median risk score is used as the stratification threshold\. Shaded regions indicate 95% confidence intervals\.
### 5\.5Calibration Analysis

Figure[3](https://arxiv.org/html/2607.16233#S5.F3)presents calibration curves at 5 and 10 years\. The Integrated Brier Score \(IBS\) of 0\.138 \(Table[3](https://arxiv.org/html/2607.16233#S5.T3)\) indicates well\-calibrated probability estimates\. The calibration slope is 0\.94 at 5 years and 0\.91 at 10 years, close to the ideal of 1\.0\.

![Refer to caption](https://arxiv.org/html/2607.16233v1/x3.png)Figure 3:Calibration curves for UMMT survival predictions at 5 and 10 years\. The diagonal dashed line represents perfect calibration\.
### 5\.6Cross\-Modal Token Interaction Analysis

A unique capability of the CMT is the ability to inspect the learned token interaction matrix\. Figure[4](https://arxiv.org/html/2607.16233#S5.F4)visualizes the average attention weights between CLS and modality tokens, as well as pairwise modality\-to\-modality attention\.

![Refer to caption](https://arxiv.org/html/2607.16233v1/x4.png)Figure 4:Average token interaction matrix from the final CMT layer\. The off\-diagonal entries quantify direct cross\-modal communication strength\. Error bars \(std across patients\) range from 0\.04 to 0\.09, indicating wide patient\-level variability\.mRNA expression receives the highest CLS attention \(0\.31\), consistent with its central role in PAM50 subtyping\. The strongest pairwise modality interaction is between mRNA and CNA \(0\.22\), reflecting the known biological coupling where copy number alterations drive gene expression changes\[[25](https://arxiv.org/html/2607.16233#bib.bib25)\]\. These interaction strengths vary widely across patients \(coefficient of variation 0\.28–0\.45\), indicating that the CMT learns patient\-specific cross\-modal communication patterns\.

### 5\.7Statistical Significance

All reported UMMT results carry 95% confidence intervals computed via bootstrap \(n=1,000n=1\{,\}000resamples\)\. For the key comparisons:

- •UMMT vs\. PORPOISE \(adapted\): C\-index improvement 0\.040, 95% CI \[0\.022, 0\.058\],p<0\.001p<0\.001\.
- •UMMT vs\. DeepHit: C\-index improvement 0\.029, 95% CI \[0\.012, 0\.046\],p=0\.002p=0\.002\.
- •UMMT vs\. OmiVAE: C\-index improvement 0\.046, 95% CI \[0\.027, 0\.065\],p<0\.001p<0\.001\.

Allpp\-values are Bonferroni\-corrected for multiple comparisons\.

### 5\.8Contrastive Loss Analysis

To verify that the InfoNCE loss indeed improves cross\-modal representation alignment, we compute the average cosine similarity between CLS embeddings of the same patient under different modality masks\. With contrastive training, mean intra\-patient similarity increases from 0\.68 to 0\.81 \(p<0\.001p<0\.001\), while inter\-patient similarity decreases from 0\.42 to 0\.31 \(p<0\.001p<0\.001\)\. This confirms that the contrastive objective achieves its intended effect of pulling same\-patient representations together while pushing different\-patient representations apart\.

## 6Discussion

### 6\.1Why Token\-Level Interaction Matters

The central thesis of this work is that multi\-modal fusion for cancer genomics should be approached as a*token interaction*problem rather than a*vector concatenation*problem\. The empirical evidence supports this claim: the full CMT variant \(UMMT\) outperforms the ablated variant that replaces CMT with weighted summation \(UMMT−CMT\{\}\_\{\-\\text\{CMT\}\}\) by 2\.9 pp in accuracy and 0\.024 in C\-index\. Beyond these performance gains, the token interaction analysis \(Figure[4](https://arxiv.org/html/2607.16233#S5.F4)\) reveals that different modality pairs have*qualitatively different*interaction strengths, and these vary across patients\. A concatenation\-based model cannot capture such structured interactions because it treats all modality pairs homogeneously through a shared weight matrix\.

The biological plausibility of these interaction patterns further validates the approach\. The strong mRNA–CNA interaction aligns with the well\-established cis\- and trans\-acting effects of copy number alterations on gene expression\[[25](https://arxiv.org/html/2607.16233#bib.bib25)\]\. The weaker methylation–mutation interaction is consistent with the fact that promoter methylation and somatic mutations are largely independent mechanisms of gene silencing\[[26](https://arxiv.org/html/2607.16233#bib.bib26)\]\. That the CMT recovers these known biological relationships purely from data is evidence that token\-level cross\-modal attention captures meaningful biological structure\.

### 6\.2Mechanism of Cross\-Modal Contrastive Learning

The InfoNCE loss operates by maximizing a lower bound on the mutual information between different views of the same patient’s multi\-modal data\. In our setting, each view corresponds to a random subset of observed modalities \(via dropout masking\)\. By enforcing that the CLS embedding is predictive of the patient identity across modality subsets, the contrastive objective effectively*regularizes*the embedding space: it penalizes representations that shift markedly when a modality is removed, and rewards representations that remain stable\. This stability is clinically desirable because a deployed model should not produce wildly different predictions for the same patient if one assay result is temporarily unavailable\.

The ablation results confirm this mechanism: removingℒcont\\mathcal\{L\}\_\{\\text\{cont\}\}\(UMMT−cont\{\}\_\{\-\\text\{cont\}\}\) degrades both classification \(1\.7 pp accuracy\) and survival \(0\.016 C\-index\)\. The fact that the contrastive loss improves both objectives, not just one, suggests that it acts as a general\-purpose regularizer for the shared embedding space\.

### 6\.3Multi\-Task Combined Effect

The joint optimization of classification, survival, and contrastive objectives yields combined effects that no single\-task variant matches\. We hypothesize three specific mechanisms of combined benefit:

1. 1\.Classification aids survivalby forcing the embedding space to preserve subtype\-discriminative information, which correlates with survival trajectories\.
2. 2\.Survival aids classificationby providing a continuous ranking signal \(risk ordering\) that regularizes the discrete classification boundary\.
3. 3\.Contrastive alignment aids bothby enforcing representation stability across modality subsets, preventing overfitting to specific modality combinations\.

The ablation studies support this: single\-task variants \(not shown\) achieve strictly lower performance on both tasks compared to the multi\-task model\.

### 6\.4Comparison with Prior Architectures

The structural difference between UMMT and prior multi\-modal frameworks is worth emphasizing\. PORPOISE\[[5](https://arxiv.org/html/2607.16233#bib.bib5)\]also uses a Transformer, but processes each modality independently through separate Transformer encoders and then concatenates their outputs\. MOFA\+\[[23](https://arxiv.org/html/2607.16233#bib.bib23)\]is a probabilistic factor model that does not support non\-linear interactions\. OmiVAE\[[13](https://arxiv.org/html/2607.16233#bib.bib13)\]uses separate variational autoencoders per modality without cross\-modal attention\. DeepHit\[[21](https://arxiv.org/html/2607.16233#bib.bib21)\]is a strong survival model but does not perform multi\-modal fusion at all \(it uses a single input vector\)\. UMMT is the first framework to combine \(1\) token\-level cross\-modal attention, \(2\) contrastive cross\-modal alignment, and \(3\) multi\-task optimization with a non\-linear survival head in a single architecture\.

### 6\.5Practical Implications for Clinical Deployment

For clinical deployment, several properties of UMMT are particularly relevant:

- •Missing modality robustness:The CMT can operate with an arbitrary subset of available modalities, enabling use in resource\-constrained settings where only clinical data and one genomic modality may be available\.
- •Interpretable cross\-modal interactions:The attention matrix provides a direct visualization of which modalities are communicating, which can guide clinical hypothesis generation\.
- •Calibrated risk estimates:The IBS of 0\.138 and calibration slopes close to 1\.0 indicate that predicted survival probabilities are reliable for decision\-making\.
- •Cross\-cohort generalizability:The TCGA\-BRCA validation demonstrates that UMMT trained on METABRIC transfers to an independent cohort despite differences in sequencing platforms and survival endpoints\.

### 6\.6Limitations

Despite these strengths, several limitations must be acknowledged:

1. 1\.Single\-institution METABRIC training\.While TCGA\-BRCA provides external validation, training on METABRIC alone may limit generalizability\. A multi\-cohort training strategy would likely improve cross\-population performance\.
2. 2\.No histopathological imaging\.Whole\-slide pathology images provide complementary morphological information that our current framework does not incorporate\. Extending UMMT with a vision Transformer branch for histology is a natural direction\.
3. 3\.Proportional hazards assumption\.While our DeepSurv head introduces non\-linearity, the Cox partial likelihood still assumes proportional hazards\. Discrete\-time survival models\[[22](https://arxiv.org/html/2607.16233#bib.bib22)\]or competing risks formulations could capture more complex time\-to\-event patterns\.
4. 4\.Token granularity\.Currently, each modality is a single token\. Increasing token granularity \(e\.g\., representing individual gene pathways as separate tokens\) could enable finer\-grained cross\-modal interaction but would introduce computational challenges\.
5. 5\.Causal interpretation\.The attention weights reflect predictive importance, not causal relationships\. Causal inference frameworks would be needed to establish directionality in cross\-modal interactions\.

### 6\.7Future Work

Building on these findings, we identify several promising directions: \(1\) incorporating whole\-slide pathology images through a vision Transformer branch with cross\-attention to genomic tokens; \(2\) self\-supervised pre\-training of the genomic DAEs on large unlabeled cohorts to improve representation quality; \(3\) extending the CMT with hierarchical tokenization \(pathways→\\tomodalities→\\topatient\) to enable multi\-scale cross\-modal interaction; \(4\) uncertainty quantification through Bayesian variants of the CMT; and \(5\) prospective clinical validation studies to assess real\-world utility\.

## 7Conclusion

We presented UMMT, a structurally novel framework for joint breast cancer molecular subtype classification and survival prediction from multi\-modal clinical and genomic data\. UMMT introduces three architectural innovations that address fundamental limitations of prior approaches: \(1\) token\-level cross\-modal representation through a Cross\-Modal Transformer that enables structured token interaction \(replacing monolithic vector concatenation\), \(2\) cross\-modal contrastive learning via InfoNCE that enforces representation consistency across modalities, and \(3\) multi\-task joint optimization of classification, non\-linear DeepSurv survival, and contrastive alignment objectives\. On the METABRIC dataset with thorough statistical validation and external TCGA\-BRCA validation, UMMT achieves superior performance across all metrics, with the CMT token interaction analysis revealing biologically interpretable cross\-modal communication patterns\. These results demonstrate that token\-level cross\-modal fusion, borrowing the core insight of Transformer\-based sequence modeling, represents a principled and effective approach to multi\-modal learning in precision oncology\.

## Code Availability

The codebase includes the full model implementation, training and evaluation pipelines, Optuna hyperparameter optimization, visualization tools \(attention matrices, calibration curves, Kaplan–Meier plots\), and METABRIC/TCGA\-BRCA preprocessing utilities\.

## Data Availability

Both METABRIC and TCGA\-BRCA datasets are publicly available\. METABRIC can be accessed through cBioPortal \([https://www\.cbioportal\.org/](https://www.cbioportal.org/)\)\. TCGA\-BRCA data is available through the GDC Data Portal \([https://portal\.gdc\.cancer\.gov/](https://portal.gdc.cancer.gov/)\)\. Preprocessed data files are included in the repository\.

## Ethics Statement

This study uses only publicly available, de\-identified retrospective datasets \(METABRIC and TCGA\-BRCA\)\. No new patient recruitment, interventions, or institutional review board approval were required\.

## AI Use Statement

AI\-assisted tools \(Claude Code\) were used during manuscript preparation for literature retrieval, code debugging, and writing quality checks\. All scientific content, data analysis, and conclusions are the author’s sole responsibility\.

## Acknowledgments

The author thanks Universiti Sains Malaysia \(USM\) for institutional support\.

## References

- \[1\]F\. Bray, M\. Laversanne, H\. Sung*et al\.*, “Global cancer statistics 2022: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries,”*CA Cancer J\. Clin\.*, vol\. 74, no\. 3, pp\. 229–263, 2024\.
- \[2\]C\. M\. Perou*et al\.*, “Molecular portraits of human breast tumours,”*Nature*, vol\. 406, no\. 6797, pp\. 747–752, 2000\.
- \[3\]A\. Cheerla and O\. Gevaert, “Deep learning with multimodal representation for pancancer prognosis prediction,”*Bioinformatics*, vol\. 35, no\. 14, pp\. i446–i454, 2019\.
- \[4\]S\. C\. Huang*et al\.*, “Multi\-modal fusion of histopathology and genomics for cancer survival prediction,”*Med\. Image Anal\.*, vol\. 67, p\. 101813, 2021\.
- \[5\]R\. J\. Chen*et al\.*, “Pan\-cancer integrative histology\-genomic analysis via multimodal deep learning,”*Cancer Cell*, vol\. 40, no\. 8, pp\. 865–878, 2022\.
- \[6\]A\. Vaswani*et al\.*, “Attention is all you need,” in*Adv\. Neural Inf\. Process\. Syst\.*, 2017, pp\. 5998–6008\.
- \[7\]J\. L\. Katzman*et al\.*, “DeepSurv: Personalized treatment recommender system using a Cox proportional hazards deep neural network,”*BMC Med\. Res\. Methodol\.*, vol\. 18, no\. 1, p\. 24, 2018\.
- \[8\]T\. Ching, X\. Zhu, and L\. X\. Garmire, “Cox\-nnet: An artificial neural network method for prognosis prediction of high\-throughput omics data,”*PLoS Comput\. Biol\.*, vol\. 14, no\. 4, p\. e1006076, 2018\.
- \[9\]P\. Vincent, H\. Larochelle, Y\. Bengio, and P\.\-A\. Manzagol, “Extracting and composing robust features with denoising autoencoders,” in*Int\. Conf\. Mach\. Learn\.*, 2008, pp\. 1096–1103\.
- \[10\]A\. v\. d\. Oord, Y\. Li, and O\. Vinyals, “Representation learning with contrastive predictive coding,”*arXiv preprint arXiv:1807\.03748*, 2018\.
- \[11\]R\. J\. Chen*et al\.*, “Pathomic fusion: An integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis,”*IEEE Trans\. Med\. Imaging*, vol\. 41, no\. 4, pp\. 757–770, 2022\.
- \[12\]K\. Chaudhary, O\. B\. Poirion, L\. Lu, and L\. X\. Garmire, “Deep learning–based multi\-omics integration robustly predicts survival in liver cancer,”*Clin\. Cancer Res\.*, vol\. 24, no\. 6, pp\. 1248–1259, 2018\.
- \[13\]X\. Zhang*et al\.*, “Multi\-omics data integration for cancer subtyping using denoising autoencoders,”*Bioinformatics*, vol\. 38, no\. 3, pp\. 750–758, 2022\.
- \[14\]J\. Devlin, M\.\-W\. Chang, K\. Lee, and K\. Toutanova, “BERT: Pre\-training of deep bidirectional Transformers for language understanding,” in*NAACL\-HLT*, 2019, pp\. 4171–4186\.
- \[15\]Y\.\-H\. H\. Tsai*et al\.*, “Multimodal Transformer for unaligned multimodal language sequences,” in*Proc\. ACL*, 2019, pp\. 6558–6569\.
- \[16\]J\. Lu, D\. Bateman, D\. Parikh, and S\. Lee, “ViLBERT: Pretraining task\-agnostic visiolinguistic representations for vision\-and\-language tasks,” in*NeurIPS*, 2019, pp\. 13–23\.
- \[17\]T\. Chen, S\. Kornblith, M\. Norouzi, and G\. Hinton, “A simple framework for contrastive learning of visual representations,” in*ICML*, 2020, pp\. 1597–1607\.
- \[18\]K\. He, H\. Fan, Y\. Wu, S\. Xie, and R\. Girshick, “Momentum contrast for unsupervised visual representation learning,” in*CVPR*, 2020, pp\. 9729–9738\.
- \[19\]A\. Radford*et al\.*, “Learning transferable visual models from natural language supervision,” in*ICML*, 2021, pp\. 8748–8763\.
- \[20\]D\. R\. Cox, “Regression models and life\-tables,”*J\. R\. Stat\. Soc\. Ser\. B*, vol\. 34, no\. 2, pp\. 187–202, 1972\.
- \[21\]C\. Lee, W\. R\. Zame, J\. Yoon, and M\. van der Schaar, “Deephit: A deep learning approach to survival analysis with competing risks,” in*AAAI*, 2018, pp\. 2314–2321\.
- \[22\]S\. G\. Zadeh and M\. Schmid, “Bias in cross\-entropy\-based training of deep survival models,”*IEEE Trans\. Pattern Anal\. Mach\. Intell\.*, vol\. 43, no\. 9, pp\. 3126–3137, 2020\.
- \[23\]R\. Argelaguet*et al\.*, “MOFA\+: a statistical framework for comprehensive integration of multi\-modal single\-cell data,”*Genome Biol\.*, vol\. 21, no\. 1, p\. 111, 2020\.
- \[24\]E\. Graf, C\. Schmoor, W\. Sauerbrei, and M\. Schumacher, “Assessment and comparison of prognostic classification schemes for survival data,”*Stat\. Med\.*, vol\. 18, no\. 17–18, pp\. 2529–2545, 1999\.
- \[25\]C\. Curtis*et al\.*, “The genomic and transcriptomic architecture of 2,000 breast tumours reveals novel subgroups,”*Nature*, vol\. 486, no\. 7403, pp\. 346–352, 2012\.
- \[26\]S\. B\. Baylin and J\. E\. Ohm, “Epigenetic gene silencing in cancer—a mechanism for early oncogenic pathway addiction?,”*Nat\. Rev\. Cancer*, vol\. 6, no\. 2, pp\. 107–116, 2006\.
- \[27\]T\. Akiba, S\. Sano, T\. Yanase, T\. Ohta, and M\. Koyama, “Optuna: A next\-generation hyperparameter optimization framework,” in*KDD*, 2019, pp\. 2623–2631\.
- \[28\]S\. M\. Lundberg and S\.\-I\. Lee, “A unified approach to interpreting model predictions,” in*NeurIPS*, 2017, pp\. 4765–4774\.
- \[29\]A\. Paszke*et al\.*, “PyTorch: An imperative style, high\-performance deep learning library,” in*NeurIPS*, 2019, pp\. 8024–8035\.
- \[30\]C\. Davidson\-Pilon, “lifelines: Survival analysis in Python,”*J\. Open Source Softw\.*, vol\. 4, no\. 40, p\. 1317, 2019\.
- \[31\]X\. Huang, A\. Khetan, M\. Cvitkovic, and Z\. Karnin, “TabTransformer: Tabular data modeling using contextual embeddings,”*arXiv:2012\.06678*, 2020\.
- \[32\]J\. Jiang*et al\.*, “Self\-attention genomic network for multi\-omics data integration,”*Bioinformatics*, vol\. 38, no\. 10, pp\. 2792–2800, 2022\.
- \[33\]Z\. Shao*et al\.*, “TransMIL: Transformer based correlated multiple instance learning for whole slide image classification,” in*NeurIPS*, 2021, pp\. 2136–2147\.
- \[34\]J\. A\. Sparano*et al\.*, “Adjuvant chemotherapy guided by a 21\-gene expression assay in breast cancer,”*N\. Engl\. J\. Med\.*, vol\. 379, no\. 2, pp\. 111–121, 2018\.
- \[35\]F\. Cardoso*et al\.*, “70\-gene signature as an aid to treatment decisions in early\-stage breast cancer,”*N\. Engl\. J\. Med\.*, vol\. 375, no\. 8, pp\. 717–729, 2016\.