Multimodal Routing for Interpretable, Robust, and Auditable Clinical Prediction

# Multimodal Routing for Interpretable, Robust, and Auditable Clinical Prediction
Source: [https://arxiv.org/html/2607.09982](https://arxiv.org/html/2607.09982)
,Zhongjie Wu[zhongjiew@smu\.edu](https://arxiv.org/html/2607.09982v1/mailto:[email protected])Southern Methodist UniversityDallasUSA,Eric C\. Larson[eclarson@smu\.edu](https://arxiv.org/html/2607.09982v1/mailto:[email protected])Southern Methodist UniversityDallasUSAandMehak Gupta[mehakg@smu\.edu](https://arxiv.org/html/2607.09982v1/mailto:[email protected])Southern Methodist UniversityDallasUSA

\(2026\)

###### Abstract\.

EHR data are inherently multimodal, and leveraging multiple modalities can improve predictive performance\. However, most existing approaches rely on deep fusion, which obscures how individual modalities contribute to predictions and limits the interpretability of multimodal reasoning\. We propose an explicit multimodal routing framework for clinical prediction that enables interpretable, robust, and auditable reasoning across three EHR modalities: structured longitudinal variables \(LL\), clinical notes \(NN\), and chest X\-rays \(II\)\. Our model constructs discrete unimodal, directional bimodal, and trimodal routes to capture both individual modality signals and asymmetric cross\-modal interactions\. To audit multimodal reasoning and assess robustness, we introduce inference\-time route masking, which simulates missing modalities and reweights the remaining routes without retraining\. We analyze changes in performance and routing weights under these scenarios to understand model decision\-making\. We evaluate our framework on multi\-label phenotype prediction \(K=25K=25\) and binary ICU mortality prediction using trimodal patient stays from MIMIC\-IV, revealing systematic differences in modality reliance across clinical condition groups\. Overall, our framework offers a transparent, auditable, and practical approach to multimodal clinical prediction, providing interpretability, robustness, and insights into how different data sources drive model decisions\.

Multimodal learning, EHR, routing module, missing modalities, ICU outcome prediction

††copyright:rightsretained††journalyear:2026††doi:XXXXXXX\.XXXXXXX††conference:Smart Connected Health; Month DD–DD, 2026; City, Country††isbn:978\-1\-4503\-XXXX\-X/26/XX††ccs:Computing methodologies Neural networks††ccs:Computing methodologies Machine learning††ccs:Applied computing Health informatics††ccs:Computing methodologies Learning paradigms††ccs:Computing methodologies Multimodal learning††ccs:Computing methodologies Model interpretability## 1\.Introduction

Clinicians in Intensive Care Units \(ICUs\) routinely make high\-stakes decisions by integrating heterogeneous information, including longitudinal vital signs and laboratory measurements, clinical notes, and medical imaging\. These diverse data sources are captured within Electronic Health Records \(EHRs\), creating a strong motivation for multimodal AI \(MAI\) systems that jointly model structured variables, free text, and images\. Compared to unimodal approaches, multimodal models can leverage complementary clinical evidence, improve predictive performance, and better tolerate noisy or incomplete inputs\(Leeet al\.,[2023](https://arxiv.org/html/2607.09982#bib.bib1); Wang and Li,[2025](https://arxiv.org/html/2607.09982#bib.bib20)\)\. By supporting structured multimodal reasoning that reflects how clinicians combine evidence across sources, multimodal systems offer a strong foundation for accurate and clinically meaningful decision support\.

Recent studies have demonstrated consistent performance gains from multimodal models \(MAI\) across a range of clinical prediction tasks\(Huanget al\.,[2020](https://arxiv.org/html/2607.09982#bib.bib8); Patilet al\.,[2025](https://arxiv.org/html/2607.09982#bib.bib23); Wanget al\.,[2025b](https://arxiv.org/html/2607.09982#bib.bib21); Sunet al\.,[2024](https://arxiv.org/html/2607.09982#bib.bib22)\)\. However, most MAI systems rely on deep fusion strategies that combine modalities into a single latent representation, rendering model behavior difficult to interpret\. This opacity is particularly problematic in clinical settings, where trust depends not only on predictive accuracy but also on understanding which sources of evidence drive a given prediction\. Because the relevance of each modality varies across clinical outcomes, effective multimodal systems must move beyond a single fixed fusion strategy and instead adaptively combine modalities in a transparent and task\-aware manner\. Without such transparency, even high\-performing MAI models may struggle to gain clinical acceptance\.

To address these challenges, we propose a multimodal routing framework that explicitly separates and evaluates the contributions of structured laboratory data \(LL\), clinical notes \(NN\), and medical images \(II\), as well as their interactions\. The framework defines ten explicit routes corresponding to unimodal, directional bimodal, and trimodal information flow\. By treating routes such asA←BA\\leftarrow BandB←AB\\leftarrow Aas distinct, the model captures asymmetric cross\-modal effects that more closely align with clinical reasoning processes\. Learnable routing weights quantify the contribution of each route to the final prediction, enabling both local and global interpretability\.

We evaluate our approach using trimodal ICU stays from the MIMIC\-IV dataset on two clinically relevant tasks: binary ICU mortality prediction and multi\-label phenotype prediction across 25 conditions\. Beyond predictive performance, we analyze routing activations and coefficients to characterize how evidence is combined across tasks\. We further assess model robustness through inference\-time route masking, simulating missing\-modality scenarios, and examining how routing weights redistribute under incomplete information\. This analysis provides an auditable view of model behavior and reliability under realistic clinical conditions\. Our contributions include:

- •We introduce a routing\-based multimodal framework that explicitly models unimodal, directional bimodal, and trimodal interactions, capturing cross\-modal clinical reasoning\.
- •We implement routing activations and coefficients to quantify the contribution of each interaction, providing local and global interpretability across 25 phenotypes\.
- •We analyze changes in routing weights and performance under missing\-modality scenarios to evaluate robustness and audit the model’s multimodal reasoning\.

## 2\.Related Work

### 2\.1\.Multimodal Fusion in EHR

A substantial body of work has explored integrating structured Electronic Health Record \(EHR\) data with unstructured modalities such as clinical notes and medical images\. For example, MINGLE models structured EHR variables as a hypergraph and injects information from other modalities to learn a shared patient representation\(Cuiet al\.,[2024](https://arxiv.org/html/2607.09982#bib.bib6)\)\. Other approaches employ multimodal transformers to jointly encode clinical time series and notes, fusing information via self\-attention across the ICU timeline for mortality prediction\(Lyuet al\.,[2023](https://arxiv.org/html/2607.09982#bib.bib10)\), or utilize LSTM\-based joint fusion modules to aggregate sequence and imaging data, as seen in MedFuse\(Hayatet al\.,[2022](https://arxiv.org/html/2607.09982#bib.bib17)\)\. While these methods demonstrate strong performance gains, they rely on deep fusion mechanisms that collapse multiple modalities into a single latent space, making it difficult to determine whether a prediction is driven by a specific modality, cross\-modal interactions, or spurious correlations\. Late fusion approaches partially address this interpretability bottleneck by combining modality\-specific models at the decision level\. For instance,Ruanet al\.\([2025](https://arxiv.org/html/2607.09982#bib.bib12)\)fuse structured data and clinical notes via late fusion, and RadFusion integrates CT imaging with EHR data for pulmonary embolism detection\(Zhouet al\.,[2021](https://arxiv.org/html/2607.09982#bib.bib18)\)\. Although these methods permit explicit analysis of unimodal contributions, they combine modalities at the decision level and do not capture how modalities interact to jointly influence predictions\.

A more recent line of work explicitly models multimodal interactions rather than treating fusion as a purely additive process\(Wörtweinet al\.,[2022](https://arxiv.org/html/2607.09982#bib.bib28); Tsaiet al\.,[2020](https://arxiv.org/html/2607.09982#bib.bib29)\)\. TriMF\(Wanget al\.,[2025a](https://arxiv.org/html/2607.09982#bib.bib7)\)constructs three bimodal pathways but sums them into a single multimodal embedding, collapsing the pathway structure before prediction\. Our framework instead preserves each unimodal, directional bimodal, and trimodal route as a separately parameterized pathway and supports inference\-time weighting, enabling post\-hoc auditing of multimodal reasoning without retraining\.

Mixture\-of\-Experts \(MoE\) architectures form another prominent direction for multimodal fusion under missing\-modality conditions\. MoE\-Health pretrains experts on specific modality combinations and applies gating to the top\-kkroutes\(Wang and Yang,[2025](https://arxiv.org/html/2607.09982#bib.bib25)\)\. FuseMoE introduces Laplace\-gated sparse MoE fusion over modality embeddings, designed for flexible handling of missing modalities and irregular temporal sampling\(Hanet al\.,[2024](https://arxiv.org/html/2607.09982#bib.bib50)\)\. Flex\-MoE replaces transformer feed\-forward layers with sparse MoE routing over modality embeddings supplemented by a learnable missing\-modality bank\(Yunet al\.,[2024](https://arxiv.org/html/2607.09982#bib.bib51)\)\. A related approach, FAME, applies fairness\-aware scalar weighting over unimodal embeddings\(Hoomanet al\.,[2025](https://arxiv.org/html/2607.09982#bib.bib15)\)\. These methods fuse modalities by concatenation, weighted sum, or stacked self\-attention before producing predictions, and do not expose directional cross\-modal pathways\. Consequently, the contribution of an interaction such as “notes conditioned on imaging” versus “imaging conditioned on notes” cannot be individually inspected or ablated, which is the mechanism underlying our route\-level auditing framework\.

### 2\.2\.Interpretability and Auditable Multimodal Reasoning

Explicit modeling of multimodal interactions naturally enables interpretability and auditability—key requirements for clinical decision support\. In contrast to deep fusion approaches, where modality contributions are entangled, routing\-based interaction modeling exposes distinct evidence pathways that align with clinical reasoning\. Learnable routing coefficients quantify the contribution of each unimodal and cross\-modal route, supporting both local interpretation at the patient level and global analysis across phenotypes\.

Prior multimodal approaches, such as Mixture\-of\-Experts methods discussed above, have addressed modality missingness to preserve predictive performance\(Yaoet al\.,[2024](https://arxiv.org/html/2607.09982#bib.bib24); Al\-Dailamiet al\.,[2025](https://arxiv.org/html/2607.09982#bib.bib27)\)\. Although effective, these methods provide limited insight into how the availability or removal of specific cross\-modal interactions affects model decision\-making\.

In our framework, missing\-modality analysis serves as an auditing mechanism rather than solely a robustness strategy\. By masking routes associated with missing modalities at inference and renormalizing over the remaining routes, we directly observe how the model redistributes reliance across evidence pathways\. Changes in routing weights and performance reveal whether predictions are supported by complementary modalities or dominated by narrow, potentially unreliable signals\. This allows clinicians and researchers to assess the reliability of automated predictions based on the structure of multimodal evidence supporting them\.

Overall, by linking multimodal interaction modeling with explicit routing and inference\-time auditing, our approach provides a transparent and clinically grounded framework for interpretable and auditable multimodal prediction in EHR data\.

## 3\.Methods

### 3\.1\.Problem Formulation

We design a multimodal AI \(MAI\) framework that jointly models three clinical modalities—structured longitudinal data \(LL\), clinical notes \(NN\), and medical images \(II\)—and is trained on multiple supervised clinical prediction tasks\. To handle these heterogeneous modalities in a unified manner, we represent each modality as a sequence of tokens or feature units\. For modalityM∈\{L,N,I\}M\\in\\\{L,N,I\\\}, the input tokens are denoted asXM=\{x1,x2,…,xTM\}X\_\{M\}=\\\{x\_\{1\},x\_\{2\},\\ldots,x\_\{T\_\{M\}\}\\\}, whereTMT\_\{M\}is the number of tokens for that modality\. For structured data, tokens correspond to time\-ordered clinical events \(e\.g\., vitals or lab measurements\), while for clinical notes, tokens correspond to words or subwords in the text\. For medical images, tokens are derived from a convolutional backbone \(e\.g\., ResNet\) and correspond to spatial feature regions of the image\.

Each modality is processed by a modality\-specific encoder \(e​n​cMenc\_\{M\}\) to capture the unique structure of each data type, producing hidden representations

\(1\)HM=e​n​cM​\(XM\)H\_\{M\}=enc\_\{M\}\(X\_\{M\}\)whereHM∈ℝBsz×TM×dMH\_\{M\}\\in\\mathbb\{R\}^\{B\_\{\\text\{sz\}\}\\times T\_\{M\}\\times d\_\{M\}\},BszB\_\{\\text\{sz\}\}is the batch size, anddMd\_\{M\}is the encoder\-specific hidden dimension\. Embeddings are pooled across the dimensionTMT\_\{M\}using a pooling operator \(e\.g\., CLS token selection, mean pooling, or max pooling\) to obtain a single global vector summarizing the entire modality,zM∈ℝBsz×dMz\_\{M\}\\in\\mathbb\{R\}^\{B\_\{\\text\{sz\}\}\\times d\_\{M\}\}\. Since the number of tokens \(TMT\_\{M\}\) may vary across patients, we define a binary mask,MM∈\{0,1\}Bsz×TMM\_\{M\}\\in\\\{0,1\\\}^\{B\_\{\\text\{sz\}\}\\times T\_\{M\}\}to indicate valid tokens \(non\-padding\) that contribute towards pooling\.

Different encoders produce latent representations with modality\-specific dimensionalities\. To align these representations, we apply a linear projection to each modality’s pooled output:

\(2\)z~M=zM​WM\+bM,z~M∈ℝBsz×d,\\tilde\{z\}\_\{M\}=z\_\{M\}W\_\{M\}\+b\_\{M\},\\quad\\tilde\{z\}\_\{M\}\\in\\mathbb\{R\}^\{B\_\{\\text\{sz\}\}\\times d\},wherezM∈ℝBsz×dMz\_\{M\}\\in\\mathbb\{R\}^\{B\_\{\\text\{sz\}\}\\times d\_\{M\}\}is the pooled encoder output for modalityMM,WM∈ℝdM×dW\_\{M\}\\in\\mathbb\{R\}^\{d\_\{M\}\\times d\}is a learnable projection matrix, andbM∈ℝdb\_\{M\}\\in\\mathbb\{R\}^\{d\}is a bias vector\. The projected representationsz~M\\tilde\{z\}\_\{M\}are then used in subsequent fusion and cross\-modal interaction modules\.

In the following sections, we detail each component of the proposed framework, as illustrated in Figure[1](https://arxiv.org/html/2607.09982#S3.F1)\.

![Refer to caption](https://arxiv.org/html/2607.09982v1/x1.png)Figure 1\.Architecture of the proposed multimodal routing framework\.
### 3\.2\.Modality\-Specific Encoders

##### Structured encoder\.

The structured modality consists of longitudinal sequences of vitals, medications, and other measurements aggregated in hourly bins\. Each clinical event is embedded into a token representation, and positional embeddings are added to maintain longitudinal order\. We pass the sequence through a BEHRT Transformer\(Liet al\.,[2020](https://arxiv.org/html/2607.09982#bib.bib14)\)\. Let the token\-level embeddings for a patient be

HL=\[hL,1,…,hL,TL\]∈ℝBsz×TL×dL,H\_\{L\}=\[h\_\{L,1\},\\ldots,h\_\{L,T\_\{L\}\}\]\\in\\mathbb\{R\}^\{B\_\{\\text\{sz\}\}\\times T\_\{L\}\\times d\_\{L\}\},whereTLT\_\{L\}is the number of hourly bins anddLd\_\{L\}is the token embedding dimension\. We obtain the patient\-level embedding by taking the\[CLS\]token output from the last layer:

zL∈ℝBsz×dL\.z\_\{L\}\\in\\mathbb\{R\}^\{B\_\{\\text\{sz\}\}\\times d\_\{L\}\}\.

##### Clinical notes encoder\.

Clinical notes within the observation window are concatenated in chronological order and split into overlapping chunks of lengthL=512L=512tokens\. Each chunk is passed through BioClinicalBERT\(Alsentzeret al\.,[2019](https://arxiv.org/html/2607.09982#bib.bib13)\)to obtain chunk\-level embeddings

hN,Ts∈ℝBsz×dNh\_\{N,T\_\{s\}\}\\in\\mathbb\{R\}^\{B\_\{\\text\{sz\}\}\\times d\_\{N\}\}whereTST\_\{S\}is the number of chunks\. Stacking the chunk embeddings gives the token\-level representation

HN=\[hN,1,…,hN,TS\]∈ℝBsz×TS×dN\.H\_\{N\}=\[h\_\{N,1\},\\ldots,h\_\{N,T\_\{S\}\}\]\\in\\mathbb\{R\}^\{B\_\{\\text\{sz\}\}\\times T\_\{S\}\\times d\_\{N\}\}\.We then compute a patient\-level embedding by mean pooling over chunks:

zN=1S​∑s=1ShN,s∈ℝBsz×dN\.z\_\{N\}=\\frac\{1\}\{S\}\\sum\_\{s=1\}^\{S\}h\_\{N,s\}\\in\\mathbb\{R\}^\{B\_\{\\text\{sz\}\}\\times d\_\{N\}\}\.

##### Image encoder\.

Chest X\-ray images are resized to224×224224\\times 224and passed through a ResNet\-34\(Heet al\.,[2016](https://arxiv.org/html/2607.09982#bib.bib33); Hayatet al\.,[2022](https://arxiv.org/html/2607.09982#bib.bib17)\)\. From the final convolutional block, we extract spatial feature maps and flatten them into image tokens

HI=\[hI,1,…,hI,TI\]∈ℝBsz×TI×dI,H\_\{I\}=\[h\_\{I,1\},\\ldots,h\_\{I,T\_\{I\}\}\]\\in\\mathbb\{R\}^\{B\_\{\\text\{sz\}\}\\times T\_\{I\}\\times d\_\{I\}\},whereTIT\_\{I\}is the number of spatial locations\. To obtain a patient\-level embedding, we apply global average pooling followed by a linear projection:

zI∈ℝBsz×dI\.z\_\{I\}\\in\\mathbb\{R\}^\{B\_\{\\text\{sz\}\}\\times d\_\{I\}\}\.

### 3\.3\.Multimodal Route Construction

We build explicit routes that represent unimodal information and cross\-modal interaction features, so we can track how each modality and each interaction contributes to the model decision\. Our approach does not assume multimodal interactions to be symmetric because conditioning modalityAAon modalityBBcan capture different evidence than conditioning modalityBBon modalityAA\. Therefore, we construct these interactions \(routes\) with directional cross\-attention, where we treat each direction as a separate route, and the model can learn to weigh each route differently when forming the final representation\.

##### Unimodal routes \(ℛ1\\mathcal\{R\}\_\{1\}\)\.

Each modality encoder returns a patient\-level embedding that summarizes a single modality information for that patient, and we use these embeddings as unimodal route vectors\. Each unimodal embedding projected into add\-dimensional representation forms the unimodal setℛ1=\{L,N,I\}\\mathcal\{R\}\_\{1\}=\\\{L,N,I\\\}\.

##### Bimodal routes \(ℛ2\\mathcal\{R\}\_\{2\}\)\.

We define a bimodal block that constructs a directed interaction route for every ordered pair of modalities\. For a bimodal routeA←BA\\leftarrow B, modalityAAis updated using contextual information from modalityBBvia cross\-attention\.

Each modality is encoded using a modality\-specific encoder to produce token\-level embeddings, projected into a shared latent space of dimensiondd,

HA∈ℝBsz×TA×d,HB∈ℝBsz×TB×dH\_\{A\}\\in\\mathbb\{R\}^\{B\_\{\\text\{sz\}\}\\times T\_\{A\}\\times d\},H\_\{B\}\\in\\mathbb\{R\}^\{B\_\{\\text\{sz\}\}\\times T\_\{B\}\\times d\}
To model directed interactionsA←BA\\leftarrow B, we use token\-level embeddings for A as queries \(Q=HA​WQQ=H\_\{A\}W\_\{Q\}\) and for B as keys \(K=HB​WKK=H\_\{B\}W\_\{K\}\) and values \(V=HB​WVV=H\_\{B\}W\_\{V\}\), whereWQ,WK,WV∈ℝd×dkW\_\{Q\},W\_\{K\},W\_\{V\}\\in\\mathbb\{R\}^\{d\\times d\_\{k\}\}are learned projection matrices\. Cross\-attention is then computed as

H^A←B=softmax​\(Q​K⊤dk\)​V∈ℝBsz×TA×d\.\\widehat\{H\}\_\{A\\leftarrow B\}=\\mathrm\{softmax\}\\\!\\Big\(\\frac\{QK^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\Big\)V\\in\\mathbb\{R\}^\{B\_\{\\text\{sz\}\}\\times T\_\{A\}\\times d\}\.
We apply a residual connection and feed\-forward network to update the tokens of modality A:

H~A←B=FFN​\(HA\+H^A←B\)∈ℝBsz×TA×d\.\\widetilde\{H\}\_\{A\\leftarrow B\}=\\mathrm\{FFN\}\(H\_\{A\}\+\\widehat\{H\}\_\{A\\leftarrow B\}\)\\in\\mathbb\{R\}^\{B\_\{\\text\{sz\}\}\\times T\_\{A\}\\times d\}\.
Finally, we obtain the fixed\-dimensional route embedding by mean pooling over the updated tokens of A:

\(3\)eA←B=1TA​∑t=1TAH~A←B​\[:,t,:\]∈ℝBsz×d\.e\_\{A\\leftarrow B\}=\\frac\{1\}\{T\_\{A\}\}\\sum\_\{t=1\}^\{T\_\{A\}\}\\widetilde\{H\}\_\{A\\leftarrow B\}\[:,t,:\]\\in\\mathbb\{R\}^\{B\_\{\\text\{sz\}\}\\times d\}\.
This formulation produces a route\-level embeddingeA←Be\_\{A\\leftarrow B\}that summarizes how modalityBBcontextualizes modalityAA\.

This block is reused for all ordered modality pairsA,B∈\{L,N,I\}A,B\\in\\\{L,N,I\\\}, enabling the construction of six directional bimodal routes,ℛ2=\{L←N,N←L,L←I,I←L,N←I,I←N\}\\mathcal\{R\}\_\{2\}=\\\{L\\\!\\leftarrow\\\!N,\\ N\\\!\\leftarrow\\\!L,\\ L\\\!\\leftarrow\\\!I,\\ I\\\!\\leftarrow\\\!L,\\ N\\\!\\leftarrow\\\!I,\\ I\\\!\\leftarrow\\\!N\\\}\.

##### Trimodal route \(ℛ3\\mathcal\{R\}\_\{3\}\)\.

Trimodal reasoning is modeled as a structured composition of pairwise cross\-modal interactions\. From the bimodal routing block, we obtain six directional embeddings:

\{eL←N,eN←L,eL←I,eI←L,eN←I,eI←N\}∈ℝBsz×d\.\\\{e\_\{L\\leftarrow N\},e\_\{N\\leftarrow L\},e\_\{L\\leftarrow I\},e\_\{I\\leftarrow L\},e\_\{N\\leftarrow I\},e\_\{I\\leftarrow N\}\\\}\\in\\mathbb\{R\}^\{B\_\{\\text\{sz\}\}\\times d\}\.We first form three modality\-pair representations by combining each directional pair:

\(4\)eL​N\\displaystyle e\_\{LN\}=ProjL​N​\(\[eL←N;eN←L\]\),\\displaystyle=\\mathrm\{Proj\}\_\{LN\}\\big\(\[e\_\{L\\leftarrow N\};e\_\{N\\leftarrow L\}\]\\big\),\(5\)eL​I\\displaystyle e\_\{LI\}=ProjL​I​\(\[eL←I;eI←L\]\),\\displaystyle=\\mathrm\{Proj\}\_\{LI\}\\big\(\[e\_\{L\\leftarrow I\};e\_\{I\\leftarrow L\}\]\\big\),\(6\)eN​I\\displaystyle e\_\{NI\}=ProjN​I​\(\[eN←I;eI←N\]\),\\displaystyle=\\mathrm\{Proj\}\_\{NI\}\\big\(\[e\_\{N\\leftarrow I\};e\_\{I\\leftarrow N\}\]\\big\),where\[;\]\[\\,;\\,\]denotes concatenation andProj∗​\(x\)\\mathrm\{Proj\}\_\{\*\}\(x\)is a learnable linear projection mapping fromℝ2​d\\mathbb\{R\}^\{2d\}toℝd\\mathbb\{R\}^\{d\}\. The trimodal route is then constructed by combining the three modality\-pair embeddings:

\(7\)eL​N​I=ProjT​\(\[eL​N;eL​I;eN​I\]\),e\_\{LNI\}=\\mathrm\{Proj\}\_\{T\}\\big\(\[e\_\{LN\};e\_\{LI\};e\_\{NI\}\]\\big\),whereeL​N​I∈ℝBsz×de\_\{LNI\}\\in\\mathbb\{R\}^\{B\_\{\\text\{sz\}\}\\times d\}, andProjT\\mathrm\{Proj\}\_\{T\}projectsℝ3​d\\mathbb\{R\}^\{3d\}toℝd\\mathbb\{R\}^\{d\}\.

This hierarchical formulation grounds trimodal reasoning in structured combinations of directional cross\-modal evidence, while preserving interpretability and enabling analysis of how multimodal interactions contribute to prediction\.

##### Final route set\.

Putting everything together, we get 3 unimodal routesℛ1\\mathcal\{R\}\_\{1\}, 6 directional bimodal routesℛ2\\mathcal\{R\}\_\{2\}, and 1 trimodal routeℛ3\\mathcal\{R\}\_\{3\}giving a total of 10 routes

\(8\)ℛ=ℛ1∪ℛ2∪ℛ3,\|ℛ\|=10\.\\mathcal\{R\}=\\mathcal\{R\}\_\{1\}\\cup\\mathcal\{R\}\_\{2\}\\cup\\mathcal\{R\}\_\{3\},\\qquad\|\\mathcal\{R\}\|=10\.

### 3\.4\.Routing Activations and Routing Coefficients

Each route embeddinger∈ℝBsz×de\_\{r\}\\in\\mathbb\{R\}^\{B\_\{\\text\{sz\}\}\\times d\}captures information from a specific unimodal, bimodal, or trimodal interaction\. However, not every route is equally relevant for every patient or outcome\. We therefore introduce two complementary quantities: \(i\)*route activations*, which measure how strongly a route is expressed for a given patient, independent of any specific prediction target, and \(ii\)*routing coefficients*, quantify how much each route contributes to each prediction target\. Together, they ensure that only routes both strongly expressed and relevant for the target influence the decision, enforcing structured aggregation of multimodal information\.

##### Route Activations\.

Each route embeddingere\_\{r\}is projected into a route vector of sizea\+1a\+1\. The firstaadimensions form the route vector𝐞~r∈ℝBsz×a\\mathbf\{\\tilde\{e\}\}\_\{r\}\\in\\mathbb\{R\}^\{B\_\{\\text\{sz\}\}\\times a\}, carries the predictive information, and the last dimension is the sigmoid activation, which acts as the route activationαp,r∈\(0,1\)\\alpha\_\{p,r\}\\in\(0,1\), indicating how strongly routerris expressed for patientpp\. These activations are patient\-specific indicators allowing the model to downweight weak routes\.

##### Routing Coefficients\.

The routing layer produces patient\- and label\-specific routing coefficientsRp,r,cR\_\{p,r,c\}, normalized over routes for each patientppand labelccsuch that∑r∈ℛRp,r,c=1\\sum\_\{r\\in\\mathcal\{R\}\}R\_\{p,r,c\}=1\. A highRp,r,cR\_\{p,r,c\}indicates strong agreement between routerrand labelccfor patientpp, whereas low values indicate less contribution\. These coefficients act as dynamic weights that modulate the influence of each route on a specific outcome, while also enabling direct inspection of modality reliance and interaction patterns at both local \(per\-patient\) and global \(across dataset\) levels\.

##### Outcome Prediction\.

The weighted aggregation of route embeddings gives the decision representation for patientppfor labelcc:

\(9\)𝐝p,c=∑r∈ℛRp,r,c​αp,r​𝐞~r\.\\mathbf\{d\}\_\{p,c\}=\\sum\_\{r\\in\\mathcal\{R\}\}R\_\{p,r,c\}\\,\\alpha\_\{p,r\}\\,\\mathbf\{\\tilde\{e\}\}\_\{r\}\.Here,𝐞~r\\mathbf\{\\tilde\{e\}\}\_\{r\}is the route content vector,αp,r\\alpha\_\{p,r\}is the route activation, andRp,r,cR\_\{p,r,c\}is the routing coefficient\. Final predictions are obtained by applying aS​o​f​t​m​a​xSoftmaxfor binary tasks or aS​i​g​m​o​i​dSigmoidfor multi\-label tasks\. This design enforces explicit, structured, and interpretable contributions from each route while producing task\-specific predictions\.

### 3\.5\.Robustness and Auditable Reasoning Under Missing Modalities

Robustness to missing modalities is critical for multimodal clinical models, especially in ICU settings where data availability is often variable\. Our framework explicitly models unimodal, directional bimodal, and trimodal routes, allowing predictions to rely on available evidence rather than failing under partial observability\.

To evaluate robustness, we mask modalities at inference time\. All routes involving the masked modality are disabled, and the routing coefficients of the remaining active routes are reweighted\. The model is not retrained, so this evaluation reflects performance under post\-deployment missing\-modality scenarios\.

This approach also provides a natural audit of multimodal reasoning\. By examining how routing weights and performance change when modalities are missing, we can assess how the model redistributes reliance across available evidence pathways\. Stable performance supported by complementary routes indicates robust and clinically coherent reasoning, whereas large drops in performance or shifts toward narrow unimodal pathways reveal fragile or potentially unreliable decision logic\.

### 3\.6\.Interpretability

By explicitly separating patient\-specific route activations from task\-specific routing coefficients, our framework supports interpretability of multimodal clinical predictions\. By combining activationsαp,r\\alpha\_\{p,r\}and routing coefficientsRp,r,cR\_\{p,r,c\}, we define an effective contribution weight for each route and label it asWp,r,c=αp,r⋅Rp,r,cW\_\{p,r,c\}=\\alpha\_\{p,r\}\\cdot R\_\{p,r,c\}, capturing both route expression and task\-specific importance\. We use these variables to provide structured interpretability across individual patients \(local\), the full dataset \(global\), and disease\-specific\.

##### Local interpretation\.

For an individual patient, we inspect route activations, routing coefficients, and their effective contributions to identify which unimodal or cross\-modal interactions dominate the prediction for different labels\. This gives patient\-level insight into modality reliance and the reasoning behind the model’s decision\.

##### Global interpretation\.

At the dataset level, we average route activations, routing coefficients, and effective weights across all patients to assess overall modality usage for each disease label\. This will help quantify which routes drive predictions across the dataset\.

##### Group\-level interpretation\.

To analyze patterns across clinically related outcomes, we group labels into disease categories \(e\.g\., all heart\-related conditions, all lung\-related conditions\)\. Effective contributions are averaged across patients within each disease group to reveal which routes are most prominent for specific types of outcomes\. This provides a more nuanced, group\-level view of the model’s structured multimodal reasoning specific to each disease group\.

## 4\.Experiment Design

### 4\.1\.Cohort and Prediction Tasks

We evaluate our proposed method on the MIMIC\-IV EHR dataset\(Johnsonet al\.,[2023a](https://arxiv.org/html/2607.09982#bib.bib3)\), which contains ICU stays with three modalities: structured longitudinal data \(LL\), clinical notes \(NN\) from the MIMIC\-IV\-Note module\(Johnsonet al\.,[2023b](https://arxiv.org/html/2607.09982#bib.bib4)\), and chest X\-ray images \(II\) from MIMIC\-CXR\(Johnsonet al\.,[2024](https://arxiv.org/html/2607.09982#bib.bib5)\)\.

We test on two prediction tasks \- ICU mortality and phenotype prediction\. ICU mortality prediction is a binary task, where we use paired EHR data for all modalities from the first 48 hours of the ICU stay \(N=6,677N=6,677\)\. Phenotype prediction is a multi\-label task \(K=25K=25\), where we identify 25 clinically meaningful phenotypes derived from ICD\-9 and ICD\-10 diagnosis codes assigned during the ICU stay, following established phenotype groupings used in prior MIMIC\-based benchmark studies\(Harutyunyanet al\.,[2019](https://arxiv.org/html/2607.09982#bib.bib9)\)\. We use full\-stay paired EHR data \(N=11,230N=11\{,\}230\)\. We exclude discharge summaries because they directly contain phenotype information, leading to data leakage\.

The dataset is split into 80% training and 20% test sets, with 5% of the training data reserved for validation\. We report all results on the test set using the best model selected via early stopping on the validation set\. Additional details on cohort construction and preprocessing are provided in Appendix[A](https://arxiv.org/html/2607.09982#A1)\.

### 4\.2\.Implementation Details

All models are implemented in PyTorch\. Structured EHR time series are encoded using a BEHRT\-style Transformer, clinical notes are encoded using BioClinicalBERT, and chest X\-ray images are encoded using a ResNet\-34 backbone\.

All modality encoders project to a shared embedding space of dimensiond=256d=256\. Route activation representations use a dimension ofa=128a=128, and the final decision representations used for classification used a dimension256256\. Models are trained using the Adam optimizer with a learning rate of1×10−41\\times 10^\{\-4\}and a batch size of 16\. Training proceeds for up to 50 epochs with early stopping based on validation loss\.

We report performance using AUROC and F1\. For the multi\-label phenotype prediction task, we report macro\-averaged F1 across the 25 phenotypes\. All weight analyses are conducted using normalized weights across different multimodal routes\. Our open\-source implementation is publicly available\.[https://anonymous\.4open\.science/r/MultimodalRouting\-85C1/README\.md](https://anonymous.4open.science/r/MultimodalRouting-85C1/README.md)

### 4\.3\.Types of Experiments

#### 4\.3\.1\.Baseline Comparison\.

We compare our routing\-based model with multiple multimodal baseline techniques and also with more sophisticated multimodal models:

- •Joint fusion \(average modality embeddings\)
- •Late fusion\(Huanget al\.,[2020](https://arxiv.org/html/2607.09982#bib.bib8)\)
- •TriMF\(Wanget al\.,[2025a](https://arxiv.org/html/2607.09982#bib.bib7)\), which implements bimodal pathways and combines them to create a final multimodal embedding,
- •MoE\(Wang and Yang,[2025](https://arxiv.org/html/2607.09982#bib.bib25)\), which routes inputs to modality\-subset experts using gating\-based selection\.

These baselines cover the dominant fusion paradigms relevant to our setting: simple aggregation \(joint and late fusion, also adopted by MedFuse\(Hayatet al\.,[2022](https://arxiv.org/html/2607.09982#bib.bib17)\), RadFusion\(Zhouet al\.,[2021](https://arxiv.org/html/2607.09982#bib.bib18)\), andRuanet al\.\([2025](https://arxiv.org/html/2607.09982#bib.bib12)\)\), multi\-pathway fusion \(TriMF\), and gated expert fusion \(MoE\-Health\)\. Graph\- and hypergraph\-based fusion methods, such as MINGLE\(Cuiet al\.,[2024](https://arxiv.org/html/2607.09982#bib.bib6)\), target a different input representation and are not directly comparable in our token\-based encoder setting\. We do not benchmark against methods whose central mechanisms target cohorts with varying modality availability, such as FuseMoE\(Hanet al\.,[2024](https://arxiv.org/html/2607.09982#bib.bib50)\), Flex\-MoE\(Yunet al\.,[2024](https://arxiv.org/html/2607.09982#bib.bib51)\), and DrFuse\(Yaoet al\.,[2024](https://arxiv.org/html/2607.09982#bib.bib24)\), as they do not apply to our fully\-trimodal training data\. All baselines use the same modality\-specific encoders, training data, and optimization settings, and differ only in the fusion mechanism\.

#### 4\.3\.2\.Ablation Analysis\.

We further conduct an ablation study to assess the contribution of key model components\. Specifically, we evaluate our model under three settings: \(i\) removing routing activations \(α\\alpha\), \(ii\) removing routing coefficients \(rr\), and \(iii\) removing cross\-attention\. In this setting, instead of modeling six directional bimodal interactions, we simply concatenate unimodal embeddings to form three undirected bimodal representations that correspond to the symmetric setting\.

#### 4\.3\.3\.Missing Modality Evaluation\.

We audit multimodal reasoning and robustness under missing\-modality settings to assess model reliability in realistic deployment scenarios\. For unimodal evaluation, each modality is assessed independently; for bimodal settings, one modality is masked at a time, and all routes involving the masked modality are removed at inference\. Structured data are always retained, as they are consistently available in ICU environments and provide core clinical signals \(vitals and laboratory measurements\), making them a natural anchor modality\. Masking structured data would therefore be clinically unrealistic and less informative\. We analyze changes in predictive performance alongside shifts in route activations and routing coefficients to characterize how the model’s reasoning pathways adapt under missing modalities\.

#### 4\.3\.4\.Interpretability Analysis\.

Finally, we conduct interpretability analyses of routing activations and coefficients under both full\- and missing\-modality settings to examine how routing behavior and reasoning pathways influence performance and reliability\. All analyses are performed using normalized routing weights across multimodal routes to ensure comparability\.

Table 1\.Baseline comparison on the test set\. We compare our model with the existing state\-of\-the\-art multimodal fusion techniques using AUROC and F1\. All Phenotype results are averages across 25 phenotypes\. Best results are in bold\.Table 2\.Ablation analysis on the test set\. We compare our multimodal routing model with different ablations by removing one component at a time\. We evaluate performance using AUROC and F1\. All Phenotype results are averages across 25 phenotypes\. Best results are shown in bold\.

## 5\.Results

### 5\.1\.Predictive Performance

#### 5\.1\.1\.Baselines Comparison\.

Table[1](https://arxiv.org/html/2607.09982#S4.T1)reports performance on ICU mortality and multi\-label phenotype prediction, comparing our approach with established multimodal fusion baselines and targeted ablations of our routing framework\. For phenotype prediction, results are averaged across 25 phenotypes\.

Among baseline methods, more advanced multimodal methods such as MoE and TriMF outperform simple joint and late fusion baselines, confirming the benefit of modeling multimodal structure\. However, our routing\-based method achieves the best performance, improving ICU mortality AUROC and F1 by 3\.9% and 4\.7%, and phenotype prediction AUROC and F1 by 4\.0% and 6\.4% over the next\-best model, highlighting the benefits of explicitly modeling unimodal and cross\-modal interactions\.

#### 5\.1\.2\.Ablation Analysis\.

The ablation results in Table[2](https://arxiv.org/html/2607.09982#S4.T2)further clarify the role of each component in enabling structured multimodal reasoning\. Removing routing activations or coefficients leads to large performance degradation \(17\-20% in F1\), confirming that these components form the core of our framework by explicitly and systematically weighting multimodal evidence\. Removing cross\-attention also degrades performance \(11% on F1\), demonstrating that directional modeling of bimodal interactions strengthens the routing framework by distinguishing how information flows between modalities, rather than assuming symmetric effects\. These results show that directional cross\-attention, patient\-specific route weighting via routing activations, and label\-specific weighting through routing coefficients play complementary roles in enabling robust, selective, and interpretable multimodal reasoning\.

#### 5\.1\.3\.Per\-Phenotype Comparison\.

Table[3](https://arxiv.org/html/2607.09982#S5.T3)summarizes per phenotype performance, along with prevalence and group\-level averages\. Overall, predictive performance varies substantially across phenotypes and disease groups, reflecting differences in prevalence, signal availability, and clinical heterogeneity\.

Across groups, cardiovascular, renal, metabolic, and infectious phenotypes achieve consistently strong performance \(meanF1F\_\{1\}between 0\.52 and 0\.56\), indicating that these conditions are well supported by multimodal EHR signals\. In particular, highly prevalent \(p​r​e​v=prev=0\.30 to 0\.43\) phenotypes such as essential hypertension, fluid and electrolyte disorders, lipid metabolism disorders, congestive heart failure, and renal failure demonstrate strong predictive performance \(F1=F\_\{1\}=0\.61 to 0\.67\), consistent with their reliance on structured longitudinal measurements and well\-documented clinical patterns\.

Respiratory and gastrointestinal phenotypes exhibit lower average performance \(meanF1=F\_\{1\}=0\.38 to 0\.40\), driven largely by low\-prevalence and heterogeneous conditions such as upper and lower respiratory disease \(p​r​e​v=prev=0\.06 to 0\.11\) and gastrointestinal hemorrhage \(p​r​e​v=prev=0\.06\)\. These results highlight the challenge of learning reliable predictors under class imbalance and variable clinical presentation, even with multimodal input\.

Notably, acute cerebrovascular disease achieves relatively high performance \(F1=0\.61F\_\{1\}=0\.61\) despite low prevalence \(0\.07\)\. This deviation from the general prevalence–performance trend motivates deeper interpretability and auditing analyses to assess whether high accuracy in low\-prevalence settings reflects robust multimodal evidence or reliance on spurious correlations\.

Table 3\.We report the per\-phenotype performance \(F1\) on the test dataset using our proposed model\. We also group each phenotype into disease categories and report the mean performance across each group\.

### 5\.2\.Missing\-Modality Performance Evaluation

Table[4](https://arxiv.org/html/2607.09982#S5.T4)evaluates our routing\-based method under different modality combinations at inference time\. Although the model is trained using all available modalities, we simulate realistic clinical settings at test time by masking the modality on the test data at inference time\. Among unimodal models, structured data achieves the best performance for ICU mortality, indicating the strong predictive signal contained in vitals and laboratory measurements\. In contrast, clinical notes yield the strongest performance for phenotype prediction, reflecting the richness of information captured in free\-text documentation for more complex clinical characterization\.

Among the bimodal combinations,L\+NL\+Nperforms best, indicating strong synergy between structured data and clinical notes\. The trimodal setting achieves the highest overall performance, but performance underL\+NL\+Ndegrades less compared toL\+IL\+I\. This trend is consistent with the unimodal results, where both notes\-only and structured\-only models outperform the image\-only model, highlighting the relative predictive strength of structured data and notes compared to imaging\.

Table 4\.Performance of our model compared across different modality combinations using AUROC and F1\. The model is trained on paired trimodal data and evaluated with modalities masked on test data\. Best results are bolded\.#### 5\.2\.1\.Phenotype Analysis

At the phenotype level, performance degradation under missing modalities reveals clinically meaningful modality dependencies \(Table[3](https://arxiv.org/html/2607.09982#S5.T3)\)\.

![Refer to caption](https://arxiv.org/html/2607.09982v1/x2.png)Figure 2\.Routing activation and effective weight across all phenotypes\. The top heatmap shows the average primary activation \(αr\\alpha\_\{r\}\) across routing primaries\. The bottom heatmaps show the effective routing weights for each phenotype\. Three heatmaps at the top and bottom correspond to the FULL setting and missing\-modality settings \(Missing\-I and Missing\-N\)\.![Refer to caption](https://arxiv.org/html/2607.09982v1/x3.png)Figure 3\.Routing activation and effective weight for mortality\. The top heatmap shows the average primary activation \(αr\\alpha\_\{r\}\) across routing primaries\. The bottom heatmaps show the effective routing weights for mortality\. Three heatmaps at the top and bottom correspond to the FULL setting and missing\-modality settings \(Missing\-I and Missing\-N\)\.At the group level, cardiovascular, renal, infectious, and gastrointestinal phenotypes show a consistent pattern where performance drops more when clinical notes are missing compared to imaging, indicating that structured data and narrative information form the core predictive signals, while imaging plays a smaller role\.

Within the cardiovascular and renal groups, some phenotypes, such as essential hypertension, cardiac dysrhythmias, and chronic kidney disease, remain largely unaffected by missing modalities, reflecting their primary reliance on structured data\. In contrast, acute myocardial infarction, heart failure, shock, and coronary atherosclerosis exhibit moderate, symmetric declines when either notes or imaging are removed, suggesting that both modalities provide complementary information\.

A similar pattern is observed in iatrogenic phenotypes, where performance declines symmetrically when either notes or imaging is removed\. This again indicates that predictions for these conditions rely on a balanced integration of multiple modalities\.

Respiratory phenotypes exhibit overall high reliance on imaging, reflecting the importance of chest X\-rays in this group compared to other phenotype categories\. Despite this, the performance drop is similar under both missing images and missing notes scenarios, suggesting that while the model depends on imaging, the clinical notes containing radiology report details provide complementary evidence for images\.

Metabolic phenotypes demonstrate the highest resilience to missing\-modality scenarios, reflecting their strong reliance on structured data \(LL\), such as laboratory values, which serve as the primary clinical indicators for conditions like diabetes and lipid disorders\.

Cerebrovascular phenotypes achieve high performance despite low prevalence, but exhibit poor robustness to missing clinical notes, with a sharp drop in F1, while performance remains largely unchanged when imaging is removed\. This asymmetric sensitivity suggests reliance on a noisy or shortcut signal rather than stable multimodal evidence\. Such behavior highlights the importance of missing\-modality analysis as an audit mechanism, where inspecting routing weights can reveal over\-reliance on specific modalities and guide closer examination of the model’s decision\-making pathways\.

### 5\.3\.Global Interpretability Analysis

Figures[2](https://arxiv.org/html/2607.09982#S5.F2)and[3](https://arxiv.org/html/2607.09982#S5.F3)summarize average route activations and effective weights across the test cohort for phenotype and mortality prediction, respectively\. Route activations, averaged across the dataset, capture overall route expressiveness, while routing coefficients are learned per label and per sample, reflecting outcome\-specific routing preferences\. Effective weights are computed as the dot product of route activations and routing coefficients, representing the final contribution of each interaction to the prediction\. Routing coefficients are reported in Appendix FigureMultimodal Routing for Interpretable, Robust, and Auditable Clinical PredictionandMultimodal Routing for Interpretable, Robust, and Auditable Clinical Predictionfor completeness, as their qualitative patterns largely overlap with the effective weights\. The dot\-product operation primarily rescales and redistributes weights without substantially altering the relative importance of dominant routes, making effective weights a more concise and interpretable summary for the main text\.

#### 5\.3\.1\.Route activations\.

In both phenotypes and mortality tasks, unimodal routes \(LLandNN\) consistently exhibit strong activations, indicating that each modality independently provides meaningful predictive signals\. In addition, several cross\-modal routes, particularlyN​LNL,N​INI, andL​N​ILNI, are highly active, demonstrating that the model frequently integrates information across modalities to gather patient\-level evidence\.

Our bimodal directional interactions, implemented via cross\-attention, further reveal systematic asymmetries\. In each bimodal setting, one directional route is consistently more activated than its reverse\. Specifically, under theNN–LLcombination,N​LNLdominates, underNN–II,N​INIshows higher activation, and underLL–II,I​LILis more active\. This behavior is expected, as the model learns that rich note representations benefit from contextual enhancement provided by structured and imaging data\. Notably, radiographic representations gain substantial value when conditioned on physiological context such as vitals and laboratory measurements, whereas structured features derive relatively limited additional information directly from visual patterns\.

Under missing\-modality scenarios, both tasks shift toward greater reliance on structured data\. Phenotype prediction additionally assigns substantial weight toN​LNLandI​LILroutes, reflecting the increased complexity and heterogeneous evidence requirements of phenotype\-level tasks\.

#### 5\.3\.2\.Effective weights\.

Routing patterns through effective weights reveal how the model integrates multimodal information for each prediction task, including 25 phenotypes \(Figure[2](https://arxiv.org/html/2607.09982#S5.F2)\) and mortality \(Figure[3](https://arxiv.org/html/2607.09982#S5.F3)\)\. Overall, trimodal interactions receive the highest weights for both mortality and phenotype prediction, indicating a strong preference for jointly leveraging all available modalities\. Among bimodal routes,N​LNLconsistently stands out, reinforcing earlier findings from route\-activation analysis that clinical notes play a central integrative role\.

![Refer to caption](https://arxiv.org/html/2607.09982v1/x4.png)Figure 4\.The figure illustrates local interpretability for three patients, showing how routing weights are distributed across the three modalities for different phenotype prediction groups\.For phenotype prediction, when imaging \(II\) is missing, the most heavily weighted effective route isN​LNL, reflecting reliance on structured data combined with narrative information\. In contrast, under missing\-notes \(NN\) scenarios, weights are more evenly distributed acrossLLandI​LIL, indicating a compensatory shift toward the remaining modalities\. For mortality prediction, missing\-modality scenarios lead to a stronger shift toward structured data \(LL\) in both cases\. However, when clinical notes \(NN\) are available, weights are more evenly distributed acrossLL,NN, andL​NLN, reinforcing the complementary importance of structured measurements and narrative information for mortality risk assessment\.

### 5\.4\.Group\-level Interpretability

As shown in Figure[2](https://arxiv.org/html/2607.09982#S5.F2), we group phenotypes into clinical categories and average effective routing weights within each group to characterize modality\-reliance patterns\. Combined with phenotype\-level analyses, this provides a transparent view of how clinical signals are integrated at both phenotype and group levels, enabling systematic robustness analysis and auditing of multimodal integration\.

Cardiovascular \(F1=0\.54F\_\{1\}=0\.54\), renal \(F1=0\.55F\_\{1\}=0\.55\), and infectious \(F1=0\.56F\_\{1\}=0\.56\) phenotypes exhibit an integrated reasoning pattern, characterized by dominant reliance on the trimodal routeL​N​ILNI\(15–17%\) and secondary weighting on clinical notesNN\(12–14%\)\. Consistent with this integration, these groups remain relatively robust under missing\-modality scenarios, with performance drops of 7–11% when imaging \(II\) is removed and 14–15% when notes \(NN\) are removed\. When imaging is removed, routing shifts primarily toward structured data \(LL\) andN​LNL, with comparable weight assigned to notes, highlighting the central role of structured measurements and the added value of narrative context\. Similarly, when notes are removed, weights shift towardLLand the strongest remaining multimodal route \(I​LIL\), reflecting the importance of preserving multimodal interactions when possible\. Directional bimodal preferences remain stable across missing\-modality scenarios, whileL​NLNandL​ILIremain consistently weak in their respective missing conditions, mirroring full\-modality behavior\.

Respiratory phenotypes also show balanced multimodal reliance, with increased contributions from imagingII\(9–17%\) while maintaining substantial weighting on clinical notesNN\(11–16%\)\. Unlike other phenotype groups that primarily leverageN​LNLinteractions, respiratory conditions show stronger reliance onN​INI\(16%\), reflecting the integration of chest X\-ray evidence with narrative documentation, including radiology reports\. Consistent with this routing behavior, performance drops most sharply when imaging is removed \(Δ​F1=22\.5%\\Delta F\_\{1\}=22\.5\\%\), highlighting the central role of radiographic evidence for respiratory phenotypes\. When notes are removed, performance also declines substantially \(Δ​F1=17\.5%\\Delta F\_\{1\}=17\.5\\%\), indicating complementary dependence on narrative context\. Under missing imaging, the model shifts weight toward structured data \(LL\), clinical notes \(NN\), and their interaction \(L​NLN\)\. Conversely, under missing notes, routing redistributes across structured data \(LL\), imaging \(II\), and their interaction \(I​LIL\), reflecting phenotype\-specific adaptive reweighting toward the strongest multimodal and unimodal pathways involving imaging\.

On the other hand, gastrointestinal phenotypes show more skewed reliance on trimodal signals under full\-modality conditions, with routing concentrated onL​N​ILNIandL​NLN\. Even in missing\-modality scenarios, weights remain concentrated on the strongest available multimodal pathways, with limited shift toward structured data \(LL\)\. Consistent with this behavior, performance drops sharply when either modality is removed \(Δ​F1=23\.7%\\Delta F\_\{1\}=23\.7\\%for missing imaging and−36\.8%\-36\.8\\%for missing notes\), reflecting strong dependence on complementary multimodal information\.

Metabolic \(F1=0\.52F\_\{1\}=0\.52\) and iatrogenic \(F1=0\.46F\_\{1\}=0\.46\) phenotypes both exhibit substantial reliance on structured dataLL\(22% and 32%, respectively\), reflecting the dominant role of laboratory and vital measurements\. For metabolic phenotypes, this reliance provides a stable core signal, with minimal degradation when notes or imaging are removed \(Δ​F1=1\.9%\\Delta F\_\{1\}=1\.9\\%for both\)\. In contrast, iatrogenic phenotypes that show more skewed weighting towardsLL\(32%\), show larger performance drops \(Δ​F1=8\.7%\\Delta F\_\{1\}=8\.7\\%missing imaging and−6\.5%\-6\.5\\%missing notes\), indicating that complementary multimodal signals are present but not effectively integrated\. This suggests gaps in multimodal reasoning where the model fails to redistribute weights toward informative auxiliary modalities\.

Cerebrovascular phenotypes assign substantial weight to a single routeI​NIN\(40%\) under full\-modality input, yet exhibit asymmetric sensitivity to missing modalities\. Performance drops minimally when imaging is removed \(Δ​F1=1\.6%\\Delta F\_\{1\}=1\.6\\%\) but collapses when notes are removed \(Δ​F1=80\.3%\\Delta F\_\{1\}=80\.3\\%\), with routing shifting toI​LILand failing to sustain predictive accuracy\. Conversely, removing imaging shifts the weight toNNwhile maintaining high performance, confirming that narrative notes are the primary predictive source\. This suggests a potential shortcut effect, in which specific keywords in clinical notes \(NN\) spuriously correlate with imaging signals \(II\), to support high performance in a full\-modality scenario\.

These patterns in iatrogenic and cerebrovascular phenotypes illustrate how combining missing\-modality analysis with routing weights provides effective audit signals\. The observed performance drops and routing shifts reveal when multimodal evidence is underutilized or when the model relies on unstable or spurious correlations\. This auditing framework therefore, exposes gaps in the model’s reasoning and highlights phenotypes in which additional training data, regularization, or architectural refinements are needed to better integrate complementary clinical evidence\.

### 5\.5\.Local interpretability

Figure[4](https://arxiv.org/html/2607.09982#S5.F4)illustrates routing weights for three ICU patients across three phenotype prediction tasks: acute myocardial infarction, pulmonary collapse, and diabetes mellitus with complications\. The figure highlights how our model dynamically distributes reliance across structured data \(LL\), clinical notes \(NN\), imaging \(II\), and their multimodal interactions\.

For the patient with acute myocardial infarction, predictions are driven primarily by structured measurements \(e\.g\., elevated heart rate, low systolic BP, acidosis\) and relevant clinical notes describing hypotension, volume overload, and pulmonary edema\. The trimodal route \(L​N​ILNI\) receives moderate weight, reflecting confirmatory support from imaging, while other modality combinations contribute minimally\.

In the pulmonary collapse patient, the model places high weight on clinical notes and imaging \(NN,II\), with strong emphasis on their interaction \(N​INI\), capturing both the narrative description of collapse symptoms and radiographic evidence\. Secondary contributions come from structured\-to\-note interactions \(N​LNL\) and the full trimodal route \(L​N​ILNI\), demonstrating the model’s balanced multimodal reasoning\.

For diabetes mellitus with complications, the patient was admitted primarily for pneumonia, yet laboratory evidence \(extreme hyperglycemia, acidosis\) supports the DM phenotype\. Here, the model relies heavily on structured data \(LL\) and clinical notes \(NN\), with lower weight to imaging\. If the task shifts to pneumonia prediction, routing adapts to emphasize imaging and narrative evidence, illustrating task\-specific interpretability\.

Overall, these examples demonstrate that our framework enables transparent, patient\-level reasoning and supports systematic auditing of evidence integration across phenotypes\.

## 6\.Discussion

We present a multimodal framework that explicitly models and exposes interactions across clinical modalities, enabling both strong predictive performance and interpretable reasoning pathways\. Unlike conventional fusion approaches that implicitly mix signals, our routing mechanism disentangles unimodal, bimodal, and trimodal evidence contributions, providing a structured representation of how clinical information is integrated for each prediction\.

Across phenotypes, the model demonstrates condition\-specific reasoning strategies that align with clinical expectations, confirming that multimodal integration improves predictive accuracy through specialized reliance on informative signals\. Importantly, routing weights offer a mechanistic explanation of model behavior, revealing whether predictions are supported by complementary evidence or dominated by a narrow signal that may be fragile or biased\.

Missing modality experiments further validate that routing weights correspond to functional reliance\. Performance degrades in clinically coherent ways when critical modalities are removed, indicating that the model is meaningfully integrating available information\. Moreover, controlled redistribution of routing weights under modality ablation suggests graceful degradation rather than catastrophic failure, a key property for real\-world clinical deployment where data incompleteness is common\.

While the inherent difficulty of the task and class imbalances result in lower absolute F1 scores, our framework consistently outperforms all baselines in both AUROC and F1\. A current limitation is that the number of directional bimodal routes scales quadratically with additional modalities \(O​\(M2\)O\(M^\{2\}\)\), which may increase computational overhead; future work will explore sparse or dynamic routing to extend this framework to higher\-modality settings\.

Overall, our framework advances multimodal clinical modeling by coupling strong predictive performance with explicit, multi\-level interpretability\. By exposing dominant evidence pathways and potential shortcut dependencies, it enables clinicians to assess reliability and identify failure modes\. This positions multimodal routing as a practical mechanism for building transparent, trustworthy clinical decision\-support systems\.

## 7\.Conclusion

We propose a multimodal routing framework that explicitly models unimodal and cross\-modal evidence pathways\. The model achieves competitive predictive performance while providing interpretable routing patterns that reveal disease\-specific modality dependence\. Combined with missing\-modality experiments and routing explanations, this framework enables systematic auditing of multimodal clinical predictions and highlights when decisions are supported by robust multimodal evidence versus potentially unstable signals\.

###### Acknowledgements\.

NH, and ZW completed experiments\. NH, EL and MG wrote the main manuscript text and prepared figures and tables\. Computational resources were provided by the SMU O’Donnell Data Science and Research Computing Institute\.

## References

- A\. Al\-Dailami, H\. Kuang, and J\. Wang \(2025\)Multimodal representation learning based on personalized graph\-based fusion for mortality prediction using electronic medical records\.Big Data Mining and Analytics8\(4\),pp\. 933–950\.Cited by:[§2\.2](https://arxiv.org/html/2607.09982#S2.SS2.p2.1)\.
- E\. Alsentzer, J\. Murphy, W\. Boag, W\. Weng, D\. Jindi, T\. Naumann, and M\. McDermott \(2019\)Publicly available clinical bert embeddings\.InProceedings of the 2nd clinical natural language processing workshop,pp\. 72–78\.Cited by:[§3\.2](https://arxiv.org/html/2607.09982#S3.SS2.SSS0.Px2.p1.1)\.
- H\. Cui, X\. Fang, R\. Xu, X\. Kan, J\. C\. Ho, and C\. Yang \(2024\)Multimodal fusion of ehr in structures and semantics: integrating clinical records and notes with hypergraph and llm\.arXiv preprint arXiv:2403\.08818\.Cited by:[§2\.1](https://arxiv.org/html/2607.09982#S2.SS1.p1.1),[§4\.3\.1](https://arxiv.org/html/2607.09982#S4.SS3.SSS1.p2.1)\.
- X\. Han, H\. Nguyen, C\. Harris, N\. Ho, and S\. Saria \(2024\)Fusemoe: mixture\-of\-experts transformers for fleximodal fusion\.Advances in Neural Information Processing Systems37,pp\. 67850–67900\.Cited by:[§2\.1](https://arxiv.org/html/2607.09982#S2.SS1.p3.1),[§4\.3\.1](https://arxiv.org/html/2607.09982#S4.SS3.SSS1.p2.1)\.
- H\. Harutyunyan, H\. Khachatrian, D\. C\. Kale, G\. Ver Steeg, and A\. Galstyan \(2019\)Multitask learning and benchmarking with clinical time series data\.Scientific data6\(1\),pp\. 96\.Cited by:[Appendix A](https://arxiv.org/html/2607.09982#A1.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2607.09982#S4.SS1.p2.3)\.
- N\. Hayat, K\. J\. Geras, and F\. E\. Shamout \(2022\)MedFuse: multi\-modal fusion with clinical time\-series data and chest x\-ray images\.InMachine Learning for Healthcare Conference,pp\. 479–503\.Cited by:[Chest X\-ray images\.](https://arxiv.org/html/2607.09982#A1.SS0.SSS0.Px3.p1.1),[§2\.1](https://arxiv.org/html/2607.09982#S2.SS1.p1.1),[§3\.2](https://arxiv.org/html/2607.09982#S3.SS2.SSS0.Px3.p1.1),[§4\.3\.1](https://arxiv.org/html/2607.09982#S4.SS3.SSS1.p2.1)\.
- K\. He, X\. Zhang, S\. Ren, and J\. Sun \(2016\)Deep residual learning for image recognition\.InProceedings of the IEEE conference on computer vision and pattern recognition,pp\. 770–778\.Cited by:[§3\.2](https://arxiv.org/html/2607.09982#S3.SS2.SSS0.Px3.p1.1)\.
- N\. Hooman, Z\. Wu, E\. C\. Larson, and M\. Gupta \(2025\)Equitable electronic health record prediction with fame: fairness\-aware multimodal embedding\.arXiv preprint arXiv:2506\.13104\.Cited by:[§2\.1](https://arxiv.org/html/2607.09982#S2.SS1.p3.1)\.
- S\. Huang, A\. Pareek, S\. Seyyedi, I\. Banerjee, and M\. P\. Lungren \(2020\)Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines\.NPJ digital medicine3\(1\),pp\. 136\.Cited by:[§1](https://arxiv.org/html/2607.09982#S1.p2.1),[2nd item](https://arxiv.org/html/2607.09982#S4.I1.i2.p1.1)\.
- A\. E\. Johnson, L\. Bulgarelli, L\. Shen, A\. Gayles, A\. Shammout, S\. Horng, T\. J\. Pollard, S\. Hao, B\. Moody, B\. Gow,et al\.\(2023a\)MIMIC\-iv, a freely accessible electronic health record dataset\.Scientific data10\(1\),pp\. 1\.Cited by:[§4\.1](https://arxiv.org/html/2607.09982#S4.SS1.p1.3)\.
- A\. Johnson, M\. Lungren, Y\. Peng, Z\. Lu, R\. Mark, S\. Berkowitz, and S\. Horng \(2024\)MIMIC\-CXR\-JPG \- chest radiographs with structured labels\.PhysioNet\.Note:Version 2\.1\.0External Links:[Document](https://dx.doi.org/10.13026/jsn5-t979),[Link](https://doi.org/10.13026/jsn5-t979)Cited by:[§4\.1](https://arxiv.org/html/2607.09982#S4.SS1.p1.3)\.
- A\. Johnson, T\. Pollard, S\. Horng, L\. A\. Celi, and R\. Mark \(2023b\)MIMIC\-IV\-Note: Deidentified free\-text clinical notes\.PhysioNet\.Note:Version 2\.2External Links:[Document](https://dx.doi.org/10.13026/1n74-ne17),[Link](https://doi.org/10.13026/1n74-ne17)Cited by:[§4\.1](https://arxiv.org/html/2607.09982#S4.SS1.p1.3)\.
- K\. Lee, S\. Lee, S\. Hahn, H\. Hyun, E\. Choi, B\. Ahn, and J\. Lee \(2023\)Learning missing modal electronic health records with unified multi\-modal data embedding and modality\-aware attention\.InMachine Learning for Healthcare Conference,pp\. 423–442\.Cited by:[§1](https://arxiv.org/html/2607.09982#S1.p1.1)\.
- Y\. Li, S\. Rao, J\. R\. A\. Solares, A\. Hassaine, R\. Ramakrishnan, D\. Canoy, Y\. Zhu, K\. Rahimi, and G\. Salimi\-Khorshidi \(2020\)BEHRT: transformer for electronic health records\.Scientific reports10\(1\),pp\. 7155\.Cited by:[§3\.2](https://arxiv.org/html/2607.09982#S3.SS2.SSS0.Px1.p1.3)\.
- W\. Lyu, X\. Dong, R\. Wong, S\. Zheng, K\. Abell\-Hart, F\. Wang, and C\. Chen \(2023\)A multimodal transformer: fusing clinical notes with structured ehr data for interpretable in\-hospital mortality prediction\.InAMIA Annual Symposium Proceedings,Vol\.2022,pp\. 719\.Cited by:[§2\.1](https://arxiv.org/html/2607.09982#S2.SS1.p1.1)\.
- A\. Patil, V\. Patil, S\. Sankpal, T\. S\. Patankar, and H\. Bhute \(2025\)Multimodal decision support system for improved diagnosis and healthcare decision making\.Journal of Biology and Health Science\.Cited by:[§1](https://arxiv.org/html/2607.09982#S1.p2.1)\.
- Y\. Ruan, D\. J\. Tan, S\. Ng, L\. Huang, and M\. Feng \(2025\)Towards accurate and reliable icu outcome prediction: a multimodal learning framework based on belief function theory using structured ehrs and free\-text notes\.Journal of Healthcare Informatics Research,pp\. 1–42\.Cited by:[§2\.1](https://arxiv.org/html/2607.09982#S2.SS1.p1.1),[§4\.3\.1](https://arxiv.org/html/2607.09982#S4.SS3.SSS1.p2.1)\.
- D\. Sun, L\. Hadjiiski, J\. Gormley, H\. Chan, E\. Caoili, R\. Cohan, A\. Alva, G\. Bruno, R\. Mihalcea, C\. Zhou,et al\.\(2024\)Outcome prediction using multi\-modal information: integrating large language model\-extracted clinical information and image analysis\.Cancers16\(13\),pp\. 2402\.Cited by:[§1](https://arxiv.org/html/2607.09982#S1.p2.1)\.
- Y\. H\. Tsai, M\. Ma, M\. Yang, R\. Salakhutdinov, and L\. Morency \(2020\)Multimodal routing: improving local and global interpretability of multimodal language analysis\.InProceedings of the 2020 conference on empirical methods in natural language processing \(emnlp\),pp\. 1823–1833\.Cited by:[§2\.1](https://arxiv.org/html/2607.09982#S2.SS1.p2.1)\.
- M\. Wang, S\. Fan, Y\. Li, Z\. Xie, and H\. Chen \(2025a\)Missing\-modality enabled multi\-modal fusion architecture for medical data\.Journal of Biomedical Informatics164,pp\. 104796\.Cited by:[§2\.1](https://arxiv.org/html/2607.09982#S2.SS1.p2.1),[3rd item](https://arxiv.org/html/2607.09982#S4.I1.i3.p1.1)\.
- X\. Wang and C\. Yang \(2025\)MoE\-health: a mixture of experts framework for robust multimodal healthcare prediction\.InProceedings of the 16th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics,pp\. 1–9\.Cited by:[§2\.1](https://arxiv.org/html/2607.09982#S2.SS1.p3.1),[4th item](https://arxiv.org/html/2607.09982#S4.I1.i4.p1.1)\.
- X\. Wang, Y\. Wang, S\. Liang, F\. Tang, C\. Liu, M\. Hu, C\. Hu, J\. He, Z\. Ge, and I\. Razzak \(2025b\)Robust multimodal learning for ophthalmic disease grading via disentangled representation\.InInternational Conference on Medical Image Computing and Computer\-Assisted Intervention,pp\. 447–456\.Cited by:[§1](https://arxiv.org/html/2607.09982#S1.p2.1)\.
- Y\. Wang and W\. Li \(2025\)Integrating multimodal ehr data for mortality prediction in icu sepsis patients\.Statistics in Medicine44\(10\-12\),pp\. e70060\.Cited by:[§1](https://arxiv.org/html/2607.09982#S1.p1.1)\.
- T\. Wörtwein, L\. Sheeber, N\. Allen, J\. Cohn, and L\. Morency \(2022\)Beyond additive fusion: learning non\-additive multimodal interactions\.InFindings of the Association for Computational Linguistics: EMNLP 2022,pp\. 4681–4696\.Cited by:[§2\.1](https://arxiv.org/html/2607.09982#S2.SS1.p2.1)\.
- W\. Yao, K\. Yin, W\. K\. Cheung, J\. Liu, and J\. Qin \(2024\)Drfuse: learning disentangled representation for clinical multi\-modal fusion with missing modality and modal inconsistency\.InProceedings of the AAAI conference on artificial intelligence,Vol\.38,pp\. 16416–16424\.Cited by:[§2\.2](https://arxiv.org/html/2607.09982#S2.SS2.p2.1),[§4\.3\.1](https://arxiv.org/html/2607.09982#S4.SS3.SSS1.p2.1)\.
- S\. Yun, I\. Choi, J\. Peng, Y\. Wu, J\. Bao, Q\. Zhang, J\. Xin, Q\. Long, and T\. Chen \(2024\)Flex\-moe: modeling arbitrary modality combination via the flexible mixture\-of\-experts\.Advances in Neural Information Processing Systems37,pp\. 98782–98805\.Cited by:[§2\.1](https://arxiv.org/html/2607.09982#S2.SS1.p3.1),[§4\.3\.1](https://arxiv.org/html/2607.09982#S4.SS3.SSS1.p2.1)\.
- Y\. Zhou, S\. Huang, J\. A\. Fries, A\. Youssef, T\. J\. Amrhein, M\. Chang, I\. Banerjee, D\. Rubin, L\. Xing, N\. Shah,et al\.\(2021\)Radfusion: benchmarking performance and fairness for multimodal pulmonary embolism detection from ct and ehr\.arXiv preprint arXiv:2111\.11665\.Cited by:[§2\.1](https://arxiv.org/html/2607.09982#S2.SS1.p1.1),[§4\.3\.1](https://arxiv.org/html/2607.09982#S4.SS3.SSS1.p2.1)\.

## Appendix AData Preprocessing

##### Structured EHR data\.

We use 17 routinely collected ICU variables, including 5 categorical variables \(e\.g\., Glasgow Coma Scale components\) and 12 continuous variables \(e\.g\., vital signs and laboratory measurements\), following prior MIMIC benchmark work\(Harutyunyanet al\.,[2019](https://arxiv.org/html/2607.09982#bib.bib9)\)\. Variables are discretized at a fixed time resolution and normalized using statistics computed from the training split\. After one\-hot encoding categorical variables, each time step is represented as a fixed\-length feature vector\.

##### Clinical notes\.

For each ICU stay, all available clinical notes within the specified observation window are collected and concatenated\.

We tokenize the text using the BioClinicalBERT tokenizer and split long documents into chunks of up to 512 tokens\. Each chunk is encoded independently, and chunk\-level embeddings are averaged to obtain a single note embedding per ICU stay\.

##### Chest X\-ray images\.

Chest X\-ray images are preprocessed following the MedFuse\(Hayatet al\.,[2022](https://arxiv.org/html/2607.09982#bib.bib17)\)pipeline\. Images are resized and normalized before being passed to the image encoder\.

## Appendix BRouting Coefficient Heatmaps

We add FigureMultimodal Routing for Interpretable, Robust, and Auditable Clinical Predictionand FigureMultimodal Routing for Interpretable, Robust, and Auditable Clinical Predictionto show routing weights for phenotypes and mortality\.

![Refer to caption](https://arxiv.org/html/2607.09982v1/x5.png)

Figure 5\.Routing coefficient heatmaps across all phenotypes\. Three heatmaps correspond to the FULL setting and missing\-modality settings \(Missing\-I and Missing\-N\)\.![Refer to caption](https://arxiv.org/html/2607.09982v1/x6.png)

Figure 6\.Routing coefficient heatmaps for mortality\. Three heatmaps correspond to the FULL setting and missing\-modality settings \(Missing\-I and Missing\-N\)\.