Mr.Dec: Daily-Scale Longitudinal Multimodal Modeling for 30-Day Readmission Prediction

arXiv cs.LG Papers
multimodal transformer healthcare readmission-prediction ehr medical-ai contrastive-learning

Summary

Mr.Dec is a multimodal Transformer-based model that sequences daily Electronic Health Record updates and chest X-ray findings to predict 30-day hospital readmissions, achieving state-of-the-art results on MIMIC datasets.

arXiv:2608.16929v1 Announce Type: new Abstract: Predicting 30-day hospital readmission is essential for assessing patient stability and optimizing healthcare resources. As clinical risk evolves with the accumulation of evidence during hospitalization, capturing these dynamic trajectories is essential. However, many existing approaches compress the complex longitudinal history into fixed representations, often losing the granular, day-level clinical signals that reflect a patient's evolving physiological state. To address this, we propose Mr.Dec (Multimodal Readmission-risk prediction Decoder), which models each admission as a natural chronological sequence of daily multimodal events. By leveraging a Transformer Decoder, Mr.Dec integrates daily Electronic Health Record(EHR) updates and intermittent Chest X-ray(CXR) findings in a time-aligned stream, reflecting the actual clinical workflow. To ensure robustness, we utilize Disease-Specific Supervised Contrastive Learning as an auxiliary regularization to induce a diagnosis-aware structure in the latent space. Evaluations on the MIMIC-IV and MIMIC-CXR datasets show that Mr.Dec achieves state-of-the-art performance by preserving the integrity of the clinical sequence. Furthermore, our model identifies "Critical Days" within an admission, providing actionable and clinically grounded interpretations for real-time risk stratification. Code is available at: https://github.com/yejix-ai/MR.DEC
Original Article
View Cached Full Text

Cached at: 08/19/26, 10:17 AM 

# Mr.Dec: Daily-Scale Longitudinal Multimodal Modeling for 30-Day Readmission Prediction
Source: [https://arxiv.org/html/2608.16929](https://arxiv.org/html/2608.16929)
11institutetext:Yeji X, Seoul, South Korea
11email:\{mj\.kim, jh\.moon\}@yejix\.ai###### Abstract

Predicting 30\-day hospital readmission is essential for assessing patient stability and optimizing healthcare resources\. As clinical risk evolves with the accumulation of evidence during hospitalization, capturing these dynamic trajectories is essential\. However, many existing approaches compress the complex longitudinal history into fixed representations, often losing the granular, day\-level clinical signals that reflect a patient’s evolving physiological state\. To address this, we proposeMr\.Dec\(MultimodalReadmission\-risk predictionDecoder\), which models each admission as a natural chronological sequence of daily multimodal events\. By leveraging a Transformer Decoder,Mr\.Decintegrates daily Electronic Health Record\(EHR\) updates and intermittent Chest X\-ray\(CXR\) findings in a time\-aligned stream, reflecting the actual clinical workflow\. To ensure robustness, we utilize Disease\-Specific Supervised Contrastive Learning as an auxiliary regularization to induce a diagnosis\-aware structure in the latent space\. Evaluations on the MIMIC\-IV and MIMIC\-CXR datasets show thatMr\.Decachieves state\-of\-the\-art performance by preserving the integrity of the clinical sequence\. Furthermore, our model identifies "Critical Days" within an admission, providing actionable and clinically grounded interpretations for real\-time risk stratification\. Code is available at: https://github\.com/yejix\-ai/MR\.DEC

## 1Introduction

![Refer to caption](https://arxiv.org/html/2608.16929v1/figure/model.png)Figure 1:Mr\.Dec\(Multimodal Readmission\-risk Prediction Decoder\)\.For each hospital daynn, textualized EHR \(e\.g\., ICD codes mapped to natural\-language descriptors\) is encoded by a clinical encoder and same\-day CXR is encoded by a vision encoder, producing modality\-specific\[CLS\]representations\{fEHRnCLS,fCXRnCLS\}\\\{f\_\{\\mathrm\{EHR\}\_\{n\}\}^\{\\mathrm\{CLS\}\},\\,f\_\{\\mathrm\{CXR\}\_\{n\}\}^\{\\mathrm\{CLS\}\}\\\}\(withfCXRnCLSf\_\{\\mathrm\{CXR\}\_\{n\}\}^\{\\mathrm\{CLS\}\}omitted when no CXR is acquired\)\. Day\-wise features are serialized in chronological order and projected with modality\-specific layers; modality and day embeddings are added to obtain the embedded token sequenceFemb=\(h1,…,hL\)F\_\{\\mathrm\{emb\}\}=\(h\_\{1\},\\ldots,h\_\{L\}\)\. We then append two learnable special tokens, yielding the decoder inputF~emb=\[Femb,\[READMIT\],\[SupCon\]\]\\tilde\{F\}\_\{\\mathrm\{emb\}\}=\[F\_\{\\mathrm\{emb\}\},\\texttt\{\[READMIT\]\},\\texttt\{\[SupCon\]\}\]\. The final hidden states of\[READMIT\]and\[SupCon\]are fed to separate MLP heads for readmission risk estimation optimized withℒreadmit\\mathcal\{L\}\_\{\\mathrm\{readmit\}\}and admission\-level representation learning optimized withℒSupCon\\mathcal\{L\}\_\{\\mathrm\{SupCon\}\}, respectively\.Predicting 30\-day all\-cause hospital readmission is a widely used benchmark of healthcare quality and patient stability\. Unplanned readmissions worsen outcomes and impose substantial economic burden, costing billions annually\. While post\-discharge environmental factors also shape riskLaxet al\.\[[2017](https://arxiv.org/html/2608.16929#bib.bib6)\]; Sterlinget al\.\[[2022](https://arxiv.org/html/2608.16929#bib.bib7)\]; Navatheet al\.\[[2018](https://arxiv.org/html/2608.16929#bib.bib8)\], a central engineering objective is to maximize discriminative signal from in\-hospital data to identify high\-risk cohorts before dischargeAlarcónet al\.\[[2024](https://arxiv.org/html/2608.16929#bib.bib4)\]; Reddyet al\.\[[2025](https://arxiv.org/html/2608.16929#bib.bib3)\]; Miao and Yu \[[2023](https://arxiv.org/html/2608.16929#bib.bib2)\]; Jianget al\.\[[2023](https://arxiv.org/html/2608.16929#bib.bib5)\]; Subasi \[[2024](https://arxiv.org/html/2608.16929#bib.bib10)\]; Lvet al\.\[[2023](https://arxiv.org/html/2608.16929#bib.bib11)\]; Tanget al\.\[[2023](https://arxiv.org/html/2608.16929#bib.bib1)\]\. The operational goal is a robust screening mechanism that interprets a patient’s evolving trajectory to support triage and resource allocation\. Accurate longitudinal modeling is essential to capture subtle signs of instability that may warrant intervention\.

Toward this end, readmission modeling has evolved toward richer patient representations\. Early approaches relied on administrative variables such as ICD codes as static categorical featuresReddyet al\.\[[2025](https://arxiv.org/html/2608.16929#bib.bib3)\]; Okolieet al\.\[[2025](https://arxiv.org/html/2608.16929#bib.bib12)\], but these lack the granularity needed to reflect complex clinical progression\. More recent work adopted multimodal learning, recognizing that structured codes alone are insufficient\. This includes incorporating clinical notes with LLMsJianget al\.\[[2023](https://arxiv.org/html/2608.16929#bib.bib5)\]; Miao and Yu \[[2023](https://arxiv.org/html/2608.16929#bib.bib2)\]and integrating CXR using deep neural networksTanget al\.\[[2023](https://arxiv.org/html/2608.16929#bib.bib1)\]; Miao and Yu \[[2023](https://arxiv.org/html/2608.16929#bib.bib2)\]; Subasi \[[2024](https://arxiv.org/html/2608.16929#bib.bib10)\]; Lvet al\.\[[2023](https://arxiv.org/html/2608.16929#bib.bib11)\]; Reddyet al\.\[[2025](https://arxiv.org/html/2608.16929#bib.bib3)\]\. Such fusion improves prediction by combining complementary signals, including semantic context from text and visual evidence from imaging\.

Within this multimodal landscape, fusing time\-series EHR and routine CXR provides a strong foundation for day\-level trajectory tracking\. Although clinical notes can add semantic detail, they are often irregular, subjective, and retrospectiveJianget al\.\[[2023](https://arxiv.org/html/2608.16929#bib.bib5)\]; Miao and Yu \[[2023](https://arxiv.org/html/2608.16929#bib.bib2)\], which can introduce reporting delays and observer bias\. In contrast, time\-series EHR \(e\.g\., labs\) and CXR serve as more direct proxies of physiological status: numerical signals provide immediate quantitative indicators, while CXR offers an observer\-independent view of internal pathology such as pulmonary edema progression that may not be captured by codesTanget al\.\[[2023](https://arxiv.org/html/2608.16929#bib.bib1)\]\. High\-fidelity tracking of daily clinical evolution therefore benefits from emphasizing these objective snapshots\. However, existing SOTA \(state\-of\-the\-art\) systems such as MM\-STGNNTanget al\.\[[2023](https://arxiv.org/html/2608.16929#bib.bib1)\]and MuSTMiao and Yu \[[2023](https://arxiv.org/html/2608.16929#bib.bib2)\]often aggregate variable\-length admissions into static representations to exploit population\-level structure, which can dilute intra\-patient temporal granularity and weaken causal dependencies\. This motivates an alternative design that prioritizes sequence fidelity over global graph topology, especially when the timing of physiological change is clinically decisive\.

To bridge this gap, we proposeMr\.Dec\(Multimodal Readmission\-Risk Prediction Decoder\), a daily\-scale causal Transformer that preserves intra\-admission temporal structure by modeling each hospitalization as a chronological token stream of daily EHR records interleaved with sporadic same\-day CXR observations\.Our contributions are threefold:\(i\) we introduce a new multimodal trajectory architecture that jointly models longitudinal EHR and intermittent CXR and achieves SOTA performance on 30\-day readmission prediction, \(ii\) we propose an efficient daily\-recording tokenization that supports longer sequences for fine\-grained risk tracking across the full hospital stay without retraining separate unimodal models, and \(iii\) we develop clinically grounded training and analysis components, including disease\-specific supervised contrastive regularization for robustness under label noise and imbalance, and gradient\-based "Critical Days" identification that links predicted risk changes to day\-level EHR and CXR\.

## 2Methods

#### Problem Formulation

We formulate 30\-day all\-cause readmission prediction as acausal in\-hospital trajectory modelingtask, motivated by evidence that readmission risk evolves as clinical evidence accumulates during hospitalization\[Jianget al\.,[2019](https://arxiv.org/html/2608.16929#bib.bib17); Davis and Greiner,[2024](https://arxiv.org/html/2608.16929#bib.bib18)\]\. A single admission is represented as a chronological sequence of hospital days𝒮=\(d1,d2,…,dN\)\\mathcal\{S\}=\(d\_\{1\},d\_\{2\},\\ldots,d\_\{N\}\)overNNdays, where each day is a multimodal recorddn:=\(xnEHR,xnCXR\)d\_\{n\}:=\\big\(x\_\{n\}^\{\\text\{EHR\}\},x\_\{n\}^\{\\text\{CXR\}\}\\big\)\. Here,xnEHRx\_\{n\}^\{\\text\{EHR\}\}denotes the structured EHR observed on daynn, andxnCXRx\_\{n\}^\{\\text\{CXR\}\}denotes the chest radiograph acquired on daynn\(if available; otherwisexnCXR=∅x\_\{n\}^\{\\text\{CXR\}\}=\\varnothing\)\. Given the full in\-hospital historyℋN=\(d1,…,dN\)\\mathcal\{H\}\_\{N\}=\(d\_\{1\},\\ldots,d\_\{N\}\), our model predicts the admission\-level outcome

y^=fθ​\(ℋN\),\\hat\{y\}=f\_\{\\theta\}\(\\mathcal\{H\}\_\{N\}\),while enforcing causality inside the decoder via masked self\-attention so that each token aggregates evidence only from preceding tokens in the interleaved stream\.

### 2\.1Mr\.Dec\(MultimodalReadmission\-risk predictionDecoder\)

#### Hierarchical Multimodal Tokenization

We encode the full admission historyℋN\\mathcal\{H\}\_\{N\}as a day\-aligned, variable\-length token stream that reflects the fact that CXR is not acquired every day \(Fig\.[1](https://arxiv.org/html/2608.16929#S1.F1)\)\. For each hospital dayn∈\{1,…,N\}n\\in\\\{1,\\ldots,N\\\}, textualized EHR is encoded by a frozen Clinical Encoder \(e\.g\., BioClinical ModernBERT\[Sounacket al\.,[2025](https://arxiv.org/html/2608.16929#bib.bib16)\]\) to obtainfEHRnCLSf\_\{\\text\{EHR\}\_\{n\}\}^\{\\text\{CLS\}\}, and same\-day CXR \(if present\) is encoded by a frozen Vision Encoder \(e\.g\., EVA\-X\-Base\[Yaoet al\.,[2025](https://arxiv.org/html/2608.16929#bib.bib15)\]\) to obtainfCXRnCLSf\_\{\\text\{CXR\}\_\{n\}\}^\{\\text\{CLS\}\}\.

We serialize day\-wise tokens in chronological order by emitting one EHR token per day and inserting the CXR token immediately after it only on imaging days\. We denote the resulting interleaved token features for the admission as

F=\(f1,…,fL\),F=\(f\_\{1\},\\ldots,f\_\{L\}\),where

F=\[fEHR1CLS,fCXR1CLS⏟Day​1,fEHR2CLS⏟Day​2,fEHR3CLS,fCXR3CLS⏟Day​3,…,fEHRNCLS,fCXRNCLS⏟Day​N\]\.\{\\scriptsize F=\\Big\[\\underbrace\{f\_\{\\text\{EHR\}\_\{1\}\}^\{\\text\{CLS\}\},\\,f\_\{\\text\{CXR\}\_\{1\}\}^\{\\text\{CLS\}\}\}\_\{\\text\{Day \}1\},\\underbrace\{f\_\{\\text\{EHR\}\_\{2\}\}^\{\\text\{CLS\}\}\}\_\{\\text\{Day \}2\},\\underbrace\{f\_\{\\text\{EHR\}\_\{3\}\}^\{\\text\{CLS\}\},\\,f\_\{\\text\{CXR\}\_\{3\}\}^\{\\text\{CLS\}\}\}\_\{\\text\{Day \}3\},\\dots,\\underbrace\{f\_\{\\text\{EHR\}\_\{N\}\}^\{\\text\{CLS\}\},\\,f\_\{\\text\{CXR\}\_\{N\}\}^\{\\text\{CLS\}\}\}\_\{\\text\{Day \}N\}\\Big\]\.\}Thus, the token length is

L=N\+∑n=1N𝕀​\[xnCXR≠∅\]\.L=N\+\\sum\_\{n=1\}^\{N\}\\mathbb\{I\}\\\!\\left\[x\_\{n\}^\{\\text\{CXR\}\}\\neq\\varnothing\\right\]\.
Each tokenfif\_\{i\}has modalitymi∈\{ehr,cxr\}m\_\{i\}\\in\\\{\\text\{ehr\},\\text\{cxr\}\\\}and hospital\-day indexni∈\{1,…,N\}n\_\{i\}\\in\\\{1,\\ldots,N\\\}\. We projectfif\_\{i\}into the model dimensiondmodeld\_\{\\text\{model\}\}with modality\-specific projections and add modality and day embeddings:

hi=GELU​\(Wprojmi​fi\+bprojmi\)\+Memb​\(mi\)\+Demb​\(ni\),h\_\{i\}=\\text\{GELU\}\\\!\\big\(W\_\{\\text\{proj\}\}^\{m\_\{i\}\}f\_\{i\}\+b\_\{\\text\{proj\}\}^\{m\_\{i\}\}\\big\)\+M\_\{\\text\{emb\}\}\(m\_\{i\}\)\+D\_\{\\text\{emb\}\}\(n\_\{i\}\),\(1\)whereMemb​\(⋅\)M\_\{\\text\{emb\}\}\(\\cdot\)andDemb​\(⋅\)D\_\{\\text\{emb\}\}\(\\cdot\)denote learnable modality and day embedding lookups, respectively\. We denote the resulting decoder input embedding sequence asFemb:=\(h1,…,hL\)F\_\{\\mathrm\{emb\}\}:=\(h\_\{1\},\\ldots,h\_\{L\}\)\.

#### Multimodal Readmission\-Risk Prediction Decoder

We model the admission trajectory by feeding the decoder inputF~emb=\[Femb,\[READMIT\],\[SupCon\]\]\\tilde\{F\}\_\{\\mathrm\{emb\}\}=\[F\_\{\\mathrm\{emb\}\},\\texttt\{\[READMIT\]\},\\texttt\{\[SupCon\]\}\]into a causal Transformer decoder \(Fig\.[1](https://arxiv.org/html/2608.16929#S1.F1)\)\. Masked self\-attention enforces chronological information flow: each position aggregates information only from preceding tokens, so the appended summary tokens can attend to the entire admission stream while respecting its temporal order\.

##### Training objective\.

We jointly optimize two objectives using the appended summary tokens inF~emb\\tilde\{F\}\_\{\\mathrm\{emb\}\}\. Let𝐳readmit\\mathbf\{z\}\_\{\\text\{readmit\}\}and𝐳supcon\\mathbf\{z\}\_\{\\text\{supcon\}\}denote the final hidden states of\[READMIT\]and\[SupCon\], respectively\. We map𝐳readmit\\mathbf\{z\}\_\{\\text\{readmit\}\}to the admission\-level probabilityy^\\hat\{y\}using an MLP classification head, and map𝐳supcon\\mathbf\{z\}\_\{\\text\{supcon\}\}to a projected representation𝐳~supcon\\tilde\{\\mathbf\{z\}\}\_\{\\text\{supcon\}\}using a separate MLP projection head\. This decouples the predictive and contrastive signals, and the model is trained end\-to\-end with

ℒtotal=ℒreadmit\+λ​ℒSupCon,\\mathcal\{L\}\_\{\\text\{total\}\}=\\mathcal\{L\}\_\{\\text\{readmit\}\}\+\\lambda\\,\\mathcal\{L\}\_\{\\text\{SupCon\}\},whereλ=0\.3\\lambda=0\.3scales the contrastive contribution\.

Readmission prediction\.We supervise the admission\-level prediction with binary cross\-entropy:

ℒreadmit=−\[y​log⁡y^\+\(1−y\)​log⁡\(1−y^\)\]\.\\mathcal\{L\}\_\{\\text\{readmit\}\}=\-\\big\[y\\log\\hat\{y\}\+\(1\-y\)\\log\(1\-\\hat\{y\}\)\\big\]\.\(2\)
Disease\-specific supervised contrastive learning\.We apply supervised contrastive loss to the projected representation𝐳~supcon\\tilde\{\\mathbf\{z\}\}\_\{\\text\{supcon\}\}to impose diagnosis\-supergroup\-aware structure in the latent space and improve robustness to noisy readmission labels\. We define positives using a coarse diagnostic labelSUPER\_GROUP111We use diagnostic super\-groups with the following categories: Circulatory \(19\.4%\), Endocrine & Metabolic \(13\.1%\), Symptoms & Signs \(8\.4%\), Respiratory \(7\.9%\), Trauma & Poisoning \(7\.6%\), Digestive \(7\.2%\), Genitourinary \(5\.9%\), Factors & Services \(5\.2%\), Blood & Immune \(5\.2%\), Mental Disorders \(4\.9%\), Nervous System \(4\.5%\), Musculoskeletal & Skin \(4\.5%\), Infectious \(3\.7%\), and Neoplasms \(2\.5%\)\.; each admission is assigned a singleSUPER\_GROUPas its most frequent coarse ICD\-10 category \(14 total\)\. For an anchor admissioniiin a batchℬ\\mathcal\{B\}, positives𝒫​\(i\)\\mathcal\{P\}\(i\)are admissions that share both the same readmission labelyyand the sameSUPER\_GROUP\. To ensure positives exist for all anchors, each mini\-batch includes at least two samples for every \(SUPER\_GROUP,yy\) combination, while combinations are sampled proportionally to their prevalence\. Here,sim​\(⋅,⋅\)\\mathrm\{sim\}\(\\cdot,\\cdot\)denotes cosine similarity andτ=0\.1\\tau=0\.1is a temperature hyperparameter\. Let𝐳~i\\tilde\{\\mathbf\{z\}\}\_\{i\}denote the projection\-head output of the\[SupCon\]token for admissionii\. The loss is

ℒSupCon=∑i∈ℬ−1\|𝒫​\(i\)\|​∑p∈𝒫​\(i\)log⁡exp⁡\(sim​\(𝐳~i,𝐳~p\)/τ\)∑a∈ℬ∖\{i\}exp⁡\(sim​\(𝐳~i,𝐳~a\)/τ\)\.\\mathcal\{L\}\_\{\\text\{SupCon\}\}=\\sum\_\{i\\in\\mathcal\{B\}\}\\frac\{\-1\}\{\|\\mathcal\{P\}\(i\)\|\}\\sum\_\{p\\in\\mathcal\{P\}\(i\)\}\\log\\frac\{\\exp\(\\mathrm\{sim\}\(\\tilde\{\\mathbf\{z\}\}\_\{i\},\\tilde\{\\mathbf\{z\}\}\_\{p\}\)/\\tau\)\}\{\\sum\_\{a\\in\\mathcal\{B\}\\setminus\\\{i\\\}\}\\exp\(\\mathrm\{sim\}\(\\tilde\{\\mathbf\{z\}\}\_\{i\},\\tilde\{\\mathbf\{z\}\}\_\{a\}\)/\\tau\)\}\.\(3\)

## 3Experiments

### 3\.1Experimental Setup

##### Dataset\.

We conducted experiments on MIMIC\-IV\[Johnsonet al\.,[2024a](https://arxiv.org/html/2608.16929#bib.bib13)\]linked with MIMIC\-CXR\[Johnsonet al\.,[2024b](https://arxiv.org/html/2608.16929#bib.bib14)\]\. Following prior studies\[Miao and Yu,[2023](https://arxiv.org/html/2608.16929#bib.bib2); Tanget al\.,[2023](https://arxiv.org/html/2608.16929#bib.bib1)\], we include adult admissions \(age≥\\geq18\) with a length of stay≥\\geq48 hours and at least two CXRs during the stay\. This yields 13,821 admissions from 11,972 patients\. We apply a stratified patient\-level split \(90%/10%\), yielding 12,438 training and 1,383 test admissions with no patient overlap\. Daily EHR summaries are constructed by concatenating demographics, ICD\-10 diagnosis subgroups, abnormal laboratory results, and medications grouped by therapeutic class, with duplicates removed\. For imaging, we restrict to frontal\-view radiographs \(AP/PA\)\.

##### Baselines\.

We compareMr\.Decagainst two categories of baselines\. For a comprehensive evaluation, we include the reported metrics of MuST\[Miao and Yu,[2023](https://arxiv.org/html/2608.16929#bib.bib2)\]as an established literature reference\. To ensure a strictly fair and identical\-cohort comparison, we prioritize a direct baseline evaluation by re\-implementing MM\-STGNN\[Tanget al\.,[2023](https://arxiv.org/html/2608.16929#bib.bib1)\]on our unified MIMIC\-IV cohort\.

LVLMs\(Qwen3\-VL\[Qwen Team,[2025](https://arxiv.org/html/2608.16929#bib.bib19)\], MedGemma\[Sellergrenet al\.,[2025](https://arxiv.org/html/2608.16929#bib.bib20)\], Lingshu\[Xuet al\.,[2025](https://arxiv.org/html/2608.16929#bib.bib21)\]\) serve as an upper\-bound probe for general\-purpose multimodal reasoning\. Given their scale \(4B–32B parameters\) and broad medical pretraining, they represent a strong zero\-shot alternative to task\-specialized trajectory modeling, assessing whether purpose\-built longitudinal architectures remain necessary in the foundation model era\.222Full prompting details for all LVLM baselines are available at https://github\.com/yejix\-ai/MR\.DEC\.

### 3\.2Performance Analysis and Clinical Robustness

Table[1](https://arxiv.org/html/2608.16929#S3.T1)comparesMr\.Decagainst all baselines across modality settings via modality masking without retraining\. In the multimodal \(CXR–EHR\) setting,Mr\.Decachieves the best overall performance \(AUC 0\.814, F1 0\.752\), outperforming the primary task\-matched baseline MM\-STGNN by \+0\.014 in AUC and \+0\.184 in F1\. These gains are especially meaningful under class imbalance, where ACC can be inflated by majority\-class predictions: MM\-STGNN attains high ACC \(0\.857\) but much lower F1 \(0\.568\), whereasMr\.Decimproves positive\-case detection while maintaining competitive ACC \(0\.744\)\. Compared to its BCE\-only ablation,Mr\.Decshows comparable AUC \(0\.814 vs\. 0\.809\) but substantially higher Rec\(N\-C\) \(0\.548 vs\. 0\.329\), confirming thatℒSupCon\\mathcal\{L\}\_\{\\text\{SupCon\}\}improves sensitivity to non\-critical readmission cases under label noise and class imbalance—precisely where clinical value is highest\.

Table 1:Performance comparison for 30\-day readmission prediction\.All models predict the same 30\-day post\-discharge outcome; the day limit refers to the length of the*input*in\-hospital sequence, not the prediction horizon\. Unless otherwise noted, models are trained on sequences of up to 10 hospital days;Mr\.Dec\(Max 30\-days\) extends the input window to 30 days\. For the CXR\+EHR setting, we report outcome\-stratified recall \(y=1y\{=\}1\):Rec\(C\)for critical cases andRec\(N\-C\)for non critical cases\. ‘Params’ and ‘Dom\.’ denote trainable parameters and training domain \(Gen\.: General, Med\.: Medical\), respectively\.CXREHRCXR\+EHRModelParamsDom\.AUCACCF1AUCACCF1AUCACCF1Rec\(C\)Rec\(N\-C\)Readmission\-specialized modelsMuST111Due to the absence of publicly available source code, performance metrics for MuST are cited from the original publication\. F1\-scores were not available and are denoted by ‘–’\.—Med\.0\.7210\.744—0\.7840\.850—0\.7990\.859———MM\-STGNN3MMed\.0\.7690\.7570\.4770\.7810\.8480\.5560\.8000\.8570\.5680\.7140\.214Large Vision\-Language Models \(LVLMs\)Qwen3\-VL32BGen\.0\.5250\.5500\.5950\.6090\.5000\.6670\.6050\.5050\.6691\.0000\.007MedGemma\-1\.5v4BMed\.0\.5460\.5540\.6250\.5170\.4960\.6590\.4610\.4620\.6000\.7530\.030MedGemma27BMed\.0\.5570\.5180\.6700\.5410\.5140\.6730\.5410\.5000\.6661\.0000\.007Lingshu32BMed\.0\.5170\.5070\.6350\.5610\.5270\.6760\.4890\.5180\.6721\.0000\.038OursMr\.Dec\(BCE only\)19MMed\.0\.7170\.6330\.4830\.6730\.6760\.6330\.8090\.7220\.6970\.8210\.329Mr\.Dec19MMed\.0\.7870\.7110\.6560\.7500\.7070\.6650\.8140\.7440\.7520\.9110\.548Mr\.Dec\(Max 30\-days\)19MMed\.0\.7780\.6890\.6290\.6970\.6610\.6720\.7930\.7260\.7360\.9170\.506

To separate critical deterioration\-driven sensitivity from non\-critical readmission detection, we report outcome\-stratified recall for readmission\-positive admissions \(y=1y=1\)\. Specifically, we stratify the positive cohort into two subgroups based on clinical severity at discharge:Criticalcases \(i\.e\., end\-of\-life\) andNon\-criticalreadmission cases, denoted as Rec\(C\) and Rec\(N\-C\), respectively\. LVLM baselines show high Rec\(C\) but near\-zero Rec\(N\-C\) \(0\.007–0\.038\), indicating that their sensitivity is heavily concentrated on critical cases\. In contrast,Mr\.Decpreserves strong Rec\(C\) \(0\.911\) while substantially improving Rec\(N\-C\) to 0\.548\.Mr\.Dec\(Max 30\-days\) extends the temporal window to 30 days for longer\-stay admissions that baselines cannot natively handle\. Despite the longer and noisier sequences, it remains highly competitive \(AUC 0\.793, F1 0\.736\) with a markedly higher F1 than MM\-STGNN \(\+0\.168\), demonstrating graceful scalability without architectural modification\.

#### 3\.2\.1Consistent Performance Across Clinical Categories

Table 2:Category\-stratified performance for 30\-day readmission prediction in the paired CXR–EHR setting\. Categories are grouped by the primary source of clinical evidence \(radiographic findings vs\. structured EHR\)\.ModelCategories with Radiographic ManifestationsEHR\-Predominant CategoriesCirculatoryRespiratoryInfectiousEndocrine & MetabolicGenitourinaryAUCACCF1AUCACCF1AUCACCF1AUCACCF1AUCACCF1Readmission\-specialized modelsMM\-STGNN0\.7980\.7660\.4990\.7940\.7450\.5140\.7990\.7330\.5430\.7910\.7630\.4990\.8120\.7370\.511Large Vision\-Language Models \(LVLMs\)Qwen3\-VL0\.5870\.5250\.6870\.5910\.5880\.7390\.5850\.5580\.7140\.5960\.5230\.6870\.5710\.5670\.724MedGemma\-1\.5v0\.4720\.4830\.6160\.4580\.5000\.6540\.4240\.4700\.6130\.4730\.4800\.6150\.4560\.5000\.639MedGemma0\.5180\.5200\.6840\.5130\.5840\.7370\.5610\.5530\.7100\.5120\.5210\.6850\.4850\.5710\.725Lingshu0\.4670\.5300\.6860\.4320\.5840\.7340\.4810\.5670\.7170\.4780\.5280\.6860\.4680\.5740\.725OursMr\.Dec\(BCE only\)0\.8030\.7160\.6990\.7910\.7010\.7140\.7940\.6920\.7200\.8000\.7120\.7010\.7870\.6880\.710Mr\.Dec0\.8150\.7470\.7610\.7980\.7320\.7710\.7810\.7240\.7700\.8000\.7350\.7560\.8000\.7430\.782Mr\.Dec\(Max 30\-days\)0\.7950\.7330\.7500\.7610\.7160\.7600\.7800\.7220\.7690\.7890\.7200\.7390\.7800\.7250\.768

Table[2](https://arxiv.org/html/2608.16929#S3.T2)reports category\-stratified results for 30\-day readmission prediction in the paired CXR–EHR setting\. The five diagnostic groups span conditions with strong radiographic correlates \(Circulatory, Respiratory, Infectious\) as well as EHR\-predominant categories where risk\-relevant evidence is primarily reflected in structured daily records \(Endocrine & Metabolic, Genitourinary\)\. Across all groups,Mr\.Decmaintains consistently strong performance \(F1: 0\.756–0\.782\), indicating that it leverages multimodal evidence in a category\-appropriate manner\. In contrast, general\-purpose LVLM baselines exhibit only moderate discrimination, with substantially lower AUC \(roughly 0\.42–0\.60\), suggesting limited stability for specialized clinical risk stratification\. Notably,Mr\.Decretains robust category\-wise performance when extending the temporal window to 30 days:Mr\.Dec\(Max 30\-days\) achieves F1 of 0\.739–0\.769 across groups, supporting its scalability to longer admission trajectories without sacrificing consistency\.

#### 3\.2\.2Interpretability and Clinical Trajectory Analysis

![Refer to caption](https://arxiv.org/html/2608.16929v1/figure/QUA_v2.png)Figure 2:Readmission prediction in a deteriorating oncology admission\.CXRs were obtained only on Hospital Days 3, 6, and 11 \(shown with modality\-specific heatmaps\)\. Risk rises from 0\.81, peaks at 0\.88 on Day 7, and remains at 0\.87 through Day 11\.Figure[2](https://arxiv.org/html/2608.16929#S3.F2)illustratesMr\.Dec’s trajectory\-level interpretability on a representative deteriorating oncology admission\. We defineCritical Daysas hospital days where token\-level attributions peak and predicted risk changes most markedly, providing temporally grounded signals for real\-time risk stratification\. Heatmaps are derived by back\-projecting gradient\-based importance scores from the\[READMIT\]token through the decoder sequence to the original EHR tokens and CXR patch embeddings\. CXRs were acquired only on Days 3, 6, and 11; modality\-specific heatmaps are shown at these time points\. The predicted risk rises from 0\.81 to 0\.87 by discharge, with EHR heatmaps emphasizing diagnosis and comorbidity tokens throughout, where Day 6 highlights anti\-infective medications and Day 11 shifts toward abnormal\-lab expressions\. In parallel, CXR heatmaps show lung\-field activations expanding from Day 3 and peaking by Day 11, together yielding temporally coherent cross\-modal explanations\.

## 4Conclusion

We presentedMr\.Dec\(Multimodalreadmission\-risk predictionDecoder\), a framework that redefines readmission prediction as a daily\-scale causal trajectory modeling task\. By jointly modeling longitudinal EHR and intermittent CXR in a single causal decoder,Mr\.Decoutperforms both graph\-based and large foundation model baselines, particularly in identifying critical deterioration trajectories\. Disease\-Specific Supervised Contrastive Learning further captures inter\-patient phenotypic structure without explicit graph construction, and token\-level interpretability provides temporally grounded signals for real\-time risk stratification\. Future work will explore extending the multimodal token stream to additional clinical modalities such as ECG and clinical notes, broadening the framework’s applicability across diverse inpatient settings\.

\{credits\}

#### 4\.0\.1Acknowledgements

This work was supported by the TIPS \(Tech Incubator Program for Startup\) Program of the Ministry of SMEs and Startups and Korea Startup Foundation under Grant No\. RS\-2025\-25467010\.

#### 4\.0\.2Disclosure of Interests

The authors have no competing interests to declare that are relevant to the content of this article\.

## References

- I\. D\. S\. Alarcón, A\. Santana\-Velásquez, M\. J\. F\. Duitama, M\. B\. Salazar\-Sánchez, and A\. Hernández\-Arango \(2024\)Prediction of readmission risk in hospital patients using artificial intelligence techniques\.In2024 3rd International Congress of Biomedical Engineering and Bioengineering \(CIIBBI\),pp\. 1–6\.Cited by:[§1](https://arxiv.org/html/2608.16929#S1.p1.1)\.
- S\. Davis and R\. Greiner \(2024\)Survival models and longitudinal medical events for hospital readmission forecasting\.BMC Health Services Research24\(1\),pp\. 1394\.Cited by:[§2](https://arxiv.org/html/2608.16929#S2.SS0.SSSx1.p1.9)\.
- L\. Y\. Jiang, X\. C\. Liu, N\. P\. Nejatian, M\. Nasir\-Moin, D\. Wang, A\. Abidin, K\. Eaton, H\. A\. Riina, I\. Laufer, P\. Punjabi,et al\.\(2023\)Health system\-scale language models are all\-purpose prediction engines\.Nature619\(7969\),pp\. 357–362\.Cited by:[§1](https://arxiv.org/html/2608.16929#S1.p1.1),[§1](https://arxiv.org/html/2608.16929#S1.p2.1),[§1](https://arxiv.org/html/2608.16929#S1.p3.1)\.
- W\. Jiang, S\. Siddiqui, S\. Barnes, L\. A\. Barouch, F\. Korley, D\. A\. Martinez, M\. Toerper, S\. Cabral, E\. Hamrock, and S\. Levin \(2019\)Readmission risk trajectories for patients with heart failure using a dynamic prediction approach: retrospective study\.JMIR Medical Informatics7\(4\),pp\. e14756\.Cited by:[§2](https://arxiv.org/html/2608.16929#S2.SS0.SSSx1.p1.9)\.
- A\. Johnson, L\. Bulgarelli, T\. Pollard, B\. Gow, B\. Moody, S\. Horng, L\. A\. Celi, and R\. Mark \(2024a\)MIMIC\-IV\.Note:PhysioNetVersion 3\.1, DOI: 10\.13026/kpb9\-mt58Cited by:[§3\.1](https://arxiv.org/html/2608.16929#S3.SS1.SSS0.Px1.p1.2)\.
- A\. Johnson, T\. Pollard, R\. Mark, S\. Berkowitz, and S\. Horng \(2024b\)MIMIC\-CXR database\.Note:PhysioNetVersion 2\.1\.0, DOI: 10\.13026/4jqj\-jw95Cited by:[§3\.1](https://arxiv.org/html/2608.16929#S3.SS1.SSS0.Px1.p1.2)\.
- Y\. Lax, M\. Martinez, and N\. M\. Brown \(2017\)Social determinants of health and hospital readmission\.Pediatrics140\(5\),pp\. e20171427\.Cited by:[§1](https://arxiv.org/html/2608.16929#S1.p1.1)\.
- J\. Lv, M\. Zhang, Y\. Fu, M\. Chen, B\. Chen, Z\. Xu, X\. Yan, S\. Hu, and N\. Zhao \(2023\)An interpretable machine learning approach for predicting 30\-day readmission after stroke\.International Journal of Medical Informatics174,pp\. 105050\.Cited by:[§1](https://arxiv.org/html/2608.16929#S1.p1.1),[§1](https://arxiv.org/html/2608.16929#S1.p2.1)\.
- Y\. Miao and L\. Yu \(2023\)MuST: multimodal spatiotemporal graph\-transformer for hospital readmission prediction\.InInternational Conference on Medical Image Computing and Computer\-Assisted Intervention,pp\. 276–285\.Cited by:[§1](https://arxiv.org/html/2608.16929#S1.p1.1),[§1](https://arxiv.org/html/2608.16929#S1.p2.1),[§1](https://arxiv.org/html/2608.16929#S1.p3.1),[§3\.1](https://arxiv.org/html/2608.16929#S3.SS1.SSS0.Px1.p1.2),[§3\.1](https://arxiv.org/html/2608.16929#S3.SS1.SSS0.Px2.p1.1)\.
- A\. S\. Navathe, F\. Zhong, V\. J\. Lei, F\. Y\. Chang, M\. Sordo, M\. Topaz, S\. B\. Navathe, R\. A\. Rocha, and L\. Zhou \(2018\)Hospital readmission and social risk factors identified from physician notes\.Health Services Research53\(2\),pp\. 1110–1136\.Cited by:[§1](https://arxiv.org/html/2608.16929#S1.p1.1)\.
- A\. Okolie, A\. Bello, M\. O\. Ikhifa, A\. O\. Ibiyeye, D\. O\. Agbeso, and P\. Alumona \(2025\)Machine learning approaches for predicting 30\-day hospital readmissions: evidence from Massachusetts healthcare data\.World Journal of Advanced Research and Reviews28\(1\),pp\. 1–12\.Cited by:[§1](https://arxiv.org/html/2608.16929#S1.p2.1)\.
- Qwen Team \(2025\)Qwen3 technical report\.External Links:2505\.09388Cited by:[§3\.1](https://arxiv.org/html/2608.16929#S3.SS1.SSS0.Px2.p2.1)\.
- N\. K\. Reddy, T\. Anirudh, T\. Varshitha, B\. C\. Kumar, N\. G\. Kumar, and M\. Jabbar \(2025\)Hospital readmission risk predictor for diabetic patients using XGBoost\.In2025 3rd World Conference on Communication & Computing \(WCONF\),pp\. 1–6\.Cited by:[§1](https://arxiv.org/html/2608.16929#S1.p1.1),[§1](https://arxiv.org/html/2608.16929#S1.p2.1)\.
- A\. Sellergren, S\. Kazemzadeh, T\. Jaroensri, A\. Kiraly, M\. Traverse, T\. Kohlberger, S\. Xu, F\. Jamil, C\. Hughes, C\. Lau,et al\.\(2025\)MedGemma technical report\.External Links:2507\.05201Cited by:[§3\.1](https://arxiv.org/html/2608.16929#S3.SS1.SSS0.Px2.p2.1)\.
- T\. Sounack, J\. Davis, B\. Durieux, A\. Chaffin, T\. J\. Pollard, E\. Lehman, A\. E\. Johnson, M\. McDermott, T\. Naumann, and C\. Lindvall \(2025\)BioClinical ModernBERT: a state\-of\-the\-art long\-context encoder for biomedical and clinical NLP\.External Links:2506\.10896Cited by:[§2\.1](https://arxiv.org/html/2608.16929#S2.SS1.SSSx1.p1.4)\.
- M\. R\. Sterling, J\. B\. Ringel, L\. C\. Pinheiro, M\. M\. Safford, E\. B\. Levitan, E\. Phillips, T\. M\. Brown, O\. K\. Nguyen, and P\. Goyal \(2022\)Social determinants of health and 30\-day readmissions among adults hospitalized for heart failure in the REGARDS study\.Circulation: Heart Failure15\(1\),pp\. e008409\.Cited by:[§1](https://arxiv.org/html/2608.16929#S1.p1.1)\.
- A\. Subasi \(2024\)Hospital readmission forecasting using artificial intelligence\.InApplications of Artificial Intelligence in Healthcare and Biomedicine,pp\. 455–520\.Cited by:[§1](https://arxiv.org/html/2608.16929#S1.p1.1),[§1](https://arxiv.org/html/2608.16929#S1.p2.1)\.
- S\. Tang, A\. Tariq, J\. A\. Dunnmon, U\. Sharma, P\. Elugunti, D\. L\. Rubin, B\. N\. Patel, and I\. Banerjee \(2023\)Predicting 30\-day all\-cause hospital readmission using multimodal spatiotemporal graph neural networks\.IEEE Journal of Biomedical and Health Informatics27\(4\),pp\. 2071–2082\.Cited by:[§1](https://arxiv.org/html/2608.16929#S1.p1.1),[§1](https://arxiv.org/html/2608.16929#S1.p2.1),[§1](https://arxiv.org/html/2608.16929#S1.p3.1),[§3\.1](https://arxiv.org/html/2608.16929#S3.SS1.SSS0.Px1.p1.2),[§3\.1](https://arxiv.org/html/2608.16929#S3.SS1.SSS0.Px2.p1.1)\.
- W\. Xu, H\. P\. Chan, L\. Li, M\. Aljunied, R\. Yuan, J\. Wang, C\. Xiao, G\. Chen, C\. Liu, Z\. Li,et al\.\(2025\)Lingshu: a generalist foundation model for unified multimodal medical understanding and reasoning\.External Links:2506\.07044Cited by:[§3\.1](https://arxiv.org/html/2608.16929#S3.SS1.SSS0.Px2.p2.1)\.
- J\. Yao, X\. Wang, Y\. Song, H\. Zhao, J\. Ma, Y\. Chen, W\. Liu, and B\. Wang \(2025\)EVA\-X: a foundation model for general chest X\-ray analysis with self\-supervised learning\.npj Digital Medicine8\(1\),pp\. 678\.Cited by:[§2\.1](https://arxiv.org/html/2608.16929#S2.SS1.SSSx1.p1.4)\.

Similar Articles