ECG幻象：揭示与缓解视觉-语言模型中ECGs在临床预测中的未充分利用

# ECG Mirage: Revealing and Mitigating the Underutilisation of ECGs in Vision-Language Models for Clinical Prediction
Source: [https://arxiv.org/html/2609.21755](https://arxiv.org/html/2609.21755)
###### Abstract

Emergency department \(ED\) decision\-making relies on heterogeneous clinical information, including patient history, vital signs, laboratory results, and electrocardiograms \(ECGs\)\. Vision–language models \(VLMs\) can jointly process these modalities, but strong predictive performance does not necessarily imply meaningful use of the correct patient’s ECG\. We term this failure mode ECG Mirage: apparent multimodal capability without useful dependence on patient\-specific ECG information\. We distinguish two forms: ECG neglect, where ECGs provide little predictive benefit, and ECG confusion, where matched ECGs outperform no\-image inputs but not mismatched ECGs\. To evaluate these behaviours, we compare predictions obtained with matched ECGs, outcome\-discordant mismatched ECGs, and no\-image inputs while holding the clinical text and prediction targets fixed\. Across four VLMs on MDS\-ED, matched ECGs provide no consistent advantage for either ICU admission or clinical deterioration prediction\. We then train four restricted visual prompts using supervised learning followed by conditional direct preference optimisation, while keeping the VLM backbone frozen\. The resulting models achieve balanced accuracies of 70\.6% for ICU admission and 67\.5% for deterioration and increase the matched\-versus\-mismatched performance gap to approximately 16\.5 and 5\.5 percentage points, respectively\. Overall, our study identifies ECG Mirage in multimodal clinical prediction and introduces visual prompt tuning as an efficient mitigation strategy\. Code is available at[https://github\.com/JasonZuu/ECG\-Mirage](https://github.com/JasonZuu/ECG-Mirage)\.

###### Index Terms:

Electrocardiography, vision\-language models, multimodal mirage, efficient optimisation, AI for healthcare\.

††address:University of Oxford, Oxford, United Kingdom
jinning\.liang@exeter\.ox\.ac\.uk## 1Introduction

![Refer to caption](https://arxiv.org/html/2609.21755v1/ECG_Mirage.png)Figure 1:Evaluation and training share a controlled ECG substitution\. The reference answer remains fixed in conditional preference training\. The no\-image input contains only EHR text and the task question and is used only for evaluation\.Predicting clinical outcomes in the emergency department \(ED\) presents an opportunity for multimodal signal processing from patient history, vital signs, laboratory measurements and electrocardiograms \(ECGs\)\[[2](https://arxiv.org/html/2609.21755#bib.bib11),[14](https://arxiv.org/html/2609.21755#bib.bib4)\]\. Vision–language models \(VLMs\) offer a flexible approach to combining these sources, processing textual electronic health records \(EHRs\) alongside rendered ECG images\[[18](https://arxiv.org/html/2609.21755#bib.bib24),[23](https://arxiv.org/html/2609.21755#bib.bib5)\]\. Related EHR\-based work has investigated efficient prompt compression for clinical prediction\[[24](https://arxiv.org/html/2609.21755#bib.bib20)\]and adaptation across clinical conditions\[[22](https://arxiv.org/html/2609.21755#bib.bib21)\]\. Recent work has demonstrated progress in ECG interpretation and in linking diagnostic predictions to waveform evidence\[[13](https://arxiv.org/html/2609.21755#bib.bib12),[11](https://arxiv.org/html/2609.21755#bib.bib13)\]\. These capabilities motivate the use of VLMs for outcome prediction, where ECGs must be interpreted alongside the broader clinical context\. However, processing both modalities does not necessarily imply that both contribute to the prediction\. This leads to a central question: do VLMs benefit from patient\-specific ECG information beyond the EHR context?

Prior work shows that VLM benchmark performance can obscure limited visual dependence, including strong reliance on accompanying text in medical image prediction\[[3](https://arxiv.org/html/2609.21755#bib.bib15),[1](https://arxiv.org/html/2609.21755#bib.bib14)\]\. For clinical outcome prediction, an improvement from adding an ECG does not necessarily indicate that the model benefits from patient\-specific information\. A model could perform similarly when given another patient’s ECG, despite outperforming a no\-image control\. We refer to this discrepancy as*ECG Mirage*, where apparent multimodal predictive capability is not accompanied by reliable benefit from the ECG information\. ECG Mirage operationalises this broader multimodal failure mode for clinical outcome prediction through patient\-level ECG substitution controls\. To examine this phenomenon, we compare matched ECGs, outcome\-discordant mismatched ECGs and a no\-image control while keeping the EHR, prediction question and target unchanged\. Across the evaluated zero\-shot VLMs and clinical tasks, matched ECGs offer no consistent predictive advantage over the controls\. These findings motivate training that encourages models to distinguish matched from mismatched ECG evidence\.

To mitigate ECG Mirage, we propose visual prompt tuning \(VPT\), a lightweight training approach that combines prompt tuning with restricted visual prompts and conditional preference optimisation\. The approach encourages VLMs to use patient\-matched ECG information when predicting clinical outcomes from multimodal clinical data\. Specifically, we train a small set of prompts while keeping the VLM backbone frozen\. An attention mask prevents textual tokens from attending directly to these prompts, allowing them to influence predictions through only image tokens\. We first train these prompts to predict clinical outcomes from matched EHR–ECG pairs\. We then use conditional direct preference optimisation\[[17](https://arxiv.org/html/2609.21755#bib.bib1),[21](https://arxiv.org/html/2609.21755#bib.bib16)\]to encourage the model to favour the correct outcome when given the patient’s own ECG over a mismatched ECG from another patient\. Experiments on ICU admission and clinical deterioration prediction show that our approach mitigates ECG Mirage, improving matched\-ECG balanced accuracy and widening the performance gap between matched\- and mismatched\-ECG cases\.

Our contributions are threefold\. \(1\) We identify and characterise ECG Mirage across four zero\-shot VLMs, showing that multimodal prediction performance does not necessarily reflect benefit from patient\-matched ECGs\. \(2\) We propose a lightweight approach that combines restricted visual prompt tuning with supervised learning and conditional preference optimisation to mitigate ECG Mirage\. \(3\) Through experiments on two clinical prediction tasks, we demonstrate improvements in matched\-ECG balanced accuracy and greater separation from mismatched ECGs and no\-image inputs\.

## 2Methodology

### 2\.1Clinical Prediction with VLMs

We consider clinical outcome prediction from electronic health records \(EHRs\) and electrocardiogram \(ECG\) images\. For each encounter, letxxdenote the EHR input available at prediction time,IIthe matched ECG image, andqqthe task question\. The target answeryyencodes either the binary ICU\-admission outcome or the six clinical deterioration labels\. We represent this answer as a token sequencey=\(y1,…,yT\)y=\(y\_\{1\},\\ldots,y\_\{T\}\)\. Given the EHR inputxx, ECG imageII, and task questionqq, a VLM with parametersθ\\thetamodels

pθ​\(y∣x,I,q\)=∏t=1Tpθ​\(yt∣y<t,x,I,q\)\.p\_\{\\theta\}\(y\\mid x,I,q\)=\\prod\_\{t=1\}^\{T\}p\_\{\\theta\}\(y\_\{t\}\\mid y\_\{<t\},x,I,q\)\.\(1\)We denote the resulting outcome prediction byy^=fθ​\(x,I,q\)\\hat\{y\}=f\_\{\\theta\}\(x,I,q\), using a fixed decoding procedure across input conditions\.

### 2\.2ECG Mirage

We define ECG Mirage as apparent multimodal predictive capability without reliable benefit from patient\-matched ECG information\. LetI\+I^\{\+\}denote the matched ECG andI−I^\{\-\}an ECG from another patient with discordant outcomes\. In the no\-image condition \(∅\\varnothing\), the VLM receives only the EHR input and task question\. The EHR inputxx, task questionqq, and targetyyremain fixed across conditions\. For a task lossℓ\\ell, the expected prediction losses are

R\+\\displaystyle R\_\{\+\}=𝔼⁡\[ℓ⁡\(fθ​\(x,I\+,q\),y\)\],\\displaystyle=\\mathbb\{E\}\\\!\\left\[\\ell\\\!\\left\(f\_\{\\theta\}\(x,I^\{\+\},q\),y\\right\)\\right\],\(2\)R−\\displaystyle R\_\{\-\}=𝔼⁡\[ℓ⁡\(fθ​\(x,I−,q\),y\)\],\\displaystyle=\\mathbb\{E\}\\\!\\left\[\\ell\\\!\\left\(f\_\{\\theta\}\(x,I^\{\-\},q\),y\\right\)\\right\],R0\\displaystyle R\_\{0\}=𝔼⁡\[ℓ⁡\(fθ​\(x,∅,q\),y\)\]\.\\displaystyle=\\mathbb\{E\}\\\!\\left\[\\ell\\\!\\left\(f\_\{\\theta\}\(x,\\varnothing,q\),y\\right\)\\right\]\.
We assess whether patient\-matched ECGs provide predictive benefit by requiring lower expected loss than both mismatched ECGs and EHR input alone:

R\+<R−andR\+<R0\.R\_\{\+\}<R\_\{\-\}\\quad\\text\{and\}\\quad R\_\{\+\}<R\_\{0\}\.\(3\)
The first comparison assesses the benefit of the correct patient’s ECG over an outcome\-discordant substitute\. The second assesses the added benefit of the ECG over the EHR text alone\.

We distinguish two illustrative ECG Mirage patterns\. ECG neglect occurs whenR\+≥R−R\_\{\+\}\\geq R\_\{\-\}andR\+≥R0R\_\{\+\}\\geq R\_\{0\}, indicating no predictive advantage from the matched ECG over either control\. ECG confusion occurs whenR−≤R\+<R0R\_\{\-\}\\leq R\_\{\+\}<R\_\{0\}, indicating an advantage over the no\-image condition without an advantage over the mismatched ECG\. Thus, improvement from including an ECG does not necessarily establish benefit from the correct patient’s ECG\. These conditions describe predictive behaviour rather than internal model processing\. Poorer control performance can inflate the matched\-ECG advantage without improving matched predictions\.

Figure 2:Zero\-shot performance on ICU admission and clinical deterioration prediction under matched ECG, mismatched ECG and no\-image conditions\. Bars show full\-test balanced accuracy for ICU admission and macro balanced accuracy for deterioration \(%\)\. Error bars indicate one standard deviation from 100 case\-bootstrap resamples\.
### 2\.3Visual Prompt Tuning \(VPT\) and Optimisation

We adapt visual prompt tuning\[[8](https://arxiv.org/html/2609.21755#bib.bib19)\]through a restricted attention pathway and two\-stage optimisation\. Specifically, let𝐇I∈ℝNI×d\\mathbf\{H\}\_\{I\}\\in\\mathbb\{R\}^\{N\_\{I\}\\times d\}denote the image token embeddings and𝐇x,q∈ℝNT×d\\mathbf\{H\}\_\{x,q\}\\in\\mathbb\{R\}^\{N\_\{T\}\\times d\}the textual token embeddings, whereddis the hidden dimension\. We introduceK=4K=4learnable visual prompt embeddings𝐇v∈ℝK×d\\mathbf\{H\}\_\{v\}\\in\\mathbb\{R\}^\{K\\times d\}before the image and textual tokens, yielding

𝐇\(0\)=\[𝐇v,𝐇I,𝐇x,q\],\\mathbf\{H\}^\{\(0\)\}=\\bigl\[\\mathbf\{H\}\_\{v\},\\,\\mathbf\{H\}\_\{I\},\\,\\mathbf\{H\}\_\{x,q\}\\bigr\],\(4\)where the brackets denote concatenation along the token dimension\. The backbone parametersθ\\thetaremain frozen, and only𝐇v\\mathbf\{H\}\_\{v\}is optimised\. To restrict direct access to the prompts, let𝒫\\mathcal\{P\},ℐ\\mathcal\{I\}, and𝒯\\mathcal\{T\}denote the positions of visual prompts, image tokens, and textual tokens, respectively\. The textual positions include both input text and answer tokens\. We define an additive attention mask

Mi​j=\{−∞,i∈𝒯​and​j∈𝒫,0,otherwise,M\_\{ij\}=\\begin\{cases\}\-\\infty,&i\\in\\mathcal\{T\}\\ \\text\{and\}\\ j\\in\\mathcal\{P\},\\\\ 0,&\\text\{otherwise\},\\end\{cases\}\(5\)whereiiandjjindex queries and keys, respectively\. For each attention head, we compute the masked attention as

Attention⁡\(𝐐,𝐊,𝐕\)=softmax⁡\(𝐐𝐊⊤dk\+𝐀\+𝐌\)​𝐕,\\operatorname\{Attention\}\(\\mathbf\{Q\},\\mathbf\{K\},\\mathbf\{V\}\)=\\operatorname\{softmax\}\\\!\\left\(\\frac\{\\mathbf\{Q\}\\mathbf\{K\}^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\+\\mathbf\{A\}\+\\mathbf\{M\}\\right\)\\mathbf\{V\},\(6\)where𝐐\\mathbf\{Q\},𝐊\\mathbf\{K\}, and𝐕\\mathbf\{V\}are the query, key, and value matrices,dkd\_\{k\}is the key dimension, and𝐀\\mathbf\{A\}is the existing attention mask\. In our implementation, image\-token queries attend to preceding visual\-prompt keys and values in full\-attention layers, whereas textual queries are masked from attending directly to the prompts\. The resulting prompt\-conditioned image representations influence answer prediction through subsequent full\-attention and recurrent linear\-attention layers\. To restrict direct prompt contributions through the recurrent pathway, we additionally zero the visual\-prompt hidden states at the input to every recurrent token mixer of Qwen3\.5\. In the first training stage, we optimise the visual prompts on matched encounters\.

ℒSFT=−log⁡pθ,𝐇v​\(y∣x,I\+,q\)\.\\mathcal\{L\}\_\{\\mathrm\{SFT\}\}=\-\\log p\_\{\\theta,\\mathbf\{H\}\_\{v\}\}\(y\\mid x,I^\{\+\},q\)\.\(7\)Only target answer tokens contribute to this loss\. This stage learns task\-specific prompts through the restricted visual pathway, but provides no explicit comparison between matched and mismatched ECGs\. In the second stage, we initialise the policy and a frozen reference from the same supervised checkpoint\. Each pair contains a matched ECGI\+I^\{\+\}and a mismatched ECGI−I^\{\-\}, with the EHR inputxx, task questionqq, and target answeryyheld fixed\. We define the policy’s corresponding log\-probability margin as

m𝐇v=log⁡pθ,𝐇v​\(y∣x,I\+,q\)−log⁡pθ,𝐇v​\(y∣x,I−,q\)\.m\_\{\\mathbf\{H\}\_\{v\}\}=\\log p\_\{\\theta,\\mathbf\{H\}\_\{v\}\}\(y\\mid x,I^\{\+\},q\)\-\\log p\_\{\\theta,\\mathbf\{H\}\_\{v\}\}\(y\\mid x,I^\{\-\},q\)\.\(8\)
Letmrefm\_\{\\mathrm\{ref\}\}denote the same margin under the frozen reference\. The second\-stage training objective is

ℒDPO=−log⁡σ⁡\(β⁡\[m𝐇v−mref\]\)\+λ​ℒSFT,\\mathcal\{L\}\_\{\\mathrm\{DPO\}\}=\-\\log\\sigma\\\!\\left\(\\beta\[m\_\{\\mathbf\{H\}\_\{v\}\}\-m\_\{\\mathrm\{ref\}\}\]\\right\)\+\\lambda\\mathcal\{L\}\_\{\\mathrm\{SFT\}\},\(9\)whereσ\\sigmais the logistic sigmoid,β=0\.1\\beta=0\.1scales the preference margin, andλ=0\.1\\lambda=0\.1weights the supervised term\. DPO favours the target answer under matched over mismatched ECGs\.

## 3Results

We investigate whether zero\-shot VLMs benefit from patient\-matched ECGs in clinical outcome prediction \(RQ1\), whether our VPT approach improves predictive performance while mitigating ECG Mirage \(RQ2\), and how conditional preference optimisation, the supervised training stage and the visual attention mask contribute to these outcomes \(RQ3\)\.

### 3\.1Experimental setup

We use MIMIC\-IV\-ED\[[9](https://arxiv.org/html/2609.21755#bib.bib2)\]with linked MIMIC\-IV\[[10](https://arxiv.org/html/2609.21755#bib.bib3)\]and MIMIC\-IV\-ECG\[[6](https://arxiv.org/html/2609.21755#bib.bib18)\]following MDS\-ED cohort construction and patient\-level partitioning\[[14](https://arxiv.org/html/2609.21755#bib.bib4)\]\. For ICU admission, the training, validation and test sets comprise 108,877/5,802/6,048 visits from 63,676/3,513/3,626 patients, respectively\. For deterioration, the corresponding sets comprise 109,299/5,819/6,077 visits from 63,929/3,525/3,644 patients\. ICU admission is assessed over the entire hospital stay, while deterioration outcomes are assessed within 24 hours of ED arrival, using the records from the first 90 minutes of observation in the ED admission\[[14](https://arxiv.org/html/2609.21755#bib.bib4)\]\. The clinical deterioration task comprises six outcomes: severe hypoxaemia, vasopressor use, mechanical ventilation, extracorporeal membrane oxygenation, inotrope use, and in\-hospital cardiac arrest\.

Prediction is performed 90 minutes after ED arrival\. All EHR information available within this window is converted into textual input following\[[4](https://arxiv.org/html/2609.21755#bib.bib6)\], with task\-specific leakage exclusions applied\. The matched ECG is the first recording from the ED encounter acquired by prediction time\. We compare matched, mismatched, and no\-image conditions while keeping the EHR input, task question, and target fixed\. The mismatched condition randomly selects an ECG from another patient with a different task label\. The no\-image condition supplies only EHR text and the task question\.

For the deep learning baseline, we use the S4–MLP model\[[14](https://arxiv.org/html/2609.21755#bib.bib4)\], which combines an ECG waveform encoder with an MLP for fusion with tabular clinical features\. For VLM adaptation baselines, we include low\-rank adaptation \(LoRA\)\[[7](https://arxiv.org/html/2609.21755#bib.bib7)\]and prompt tuning\[[12](https://arxiv.org/html/2609.21755#bib.bib8)\]\. All methods use the same cohort splits, prediction\-time cutoff, and leakage exclusions\. Models are trained separately for the two prediction tasks\. For S4–MLP, we follow the original training configuration reported in MDS\-ED\[[14](https://arxiv.org/html/2609.21755#bib.bib4)\]\. For VLM adaptation, we use AdamW\[[15](https://arxiv.org/html/2609.21755#bib.bib9)\]with a learning rate of10−310^\{\-3\}and a batch size of 32\. The learning rate increases linearly over the first 10% of training steps and follows a cosine decay schedule over the remaining 90%\. Training runs for a maximum of one epoch, with validation every 500 optimisation steps\. We select the best checkpoint by task\-specific validation F1 and stop after three consecutive checks without improvement\.

For ICU admission, we report balanced accuracy and macro F1\. For clinical deterioration, we report macro balanced accuracy and macro positive\-class F1 across the six outcomes, excluding unavailable labels from the corresponding calculations\. We report scores computed on the complete test set together with standard deviations estimated from 100 bootstrap resamples\[[5](https://arxiv.org/html/2609.21755#bib.bib10)\]\. Each resample draws test encounters with replacement, retaining their predictions and reference labels, and the same resampling indices are used across methods and input conditions\. These standard deviations describe test\-set sampling variability rather than variation across training runs\.

Table 1:Performance on ICU admission and clinical deterioration\. All VLM methods use Qwen3\.5\-4B\. S4\-MLP follows the MDS\-ED architecture and main training hyperparameters\. Subscripts indicate case\-bootstrap standard deviations over 100 resamples\.
### 3\.2RQ1: Examining ECG Mirage in zero\-shot prediction

In this study, we examined whether zero\-shot VLMs derive predictive benefit from ECGs when EHR is also available\. We evaluated Qwen3\.5\-4B\[[16](https://arxiv.org/html/2609.21755#bib.bib17)\], Qwen3\.8\-27B\[[16](https://arxiv.org/html/2609.21755#bib.bib17)\], Gemma4\-4B\[[20](https://arxiv.org/html/2609.21755#bib.bib22)\]and MedGemma1\.5\-4B\[[19](https://arxiv.org/html/2609.21755#bib.bib23)\]on ICU admission and clinical deterioration prediction under three conditions: matched ECGs, mismatched ECGs from other patients, and no image\.

Figure[2](https://arxiv.org/html/2609.21755#S2.F2)shows evidence of ECG Mirage\. For ICU admission, all four models had slightly lower balanced\-accuracy point estimates with matched ECGs than with mismatched ECGs\. Qwen3\.5\-4B achieved 52\.3%, 52\.5% and 53\.4% under matched, mismatched and no\-image conditions, respectively\. MedGemma performed better with matched ECGs than without an image, but not better than with mismatched ECGs, indicating that gains from including an ECG do not necessarily reflect a benefit from patient\-specific information\. For clinical deterioration, Gemma4\-4B also performed worse with matched ECGs than under either control, whereas both Qwen models achieved higher matched\-ECG point estimates than under either control\. Overall, matched ECGs provided no consistent predictive advantage across the evaluated models and tasks, supporting the existence of ECG Mirage\. These models may rely primarily on EHR text, consistent with prior findings that medical images add little predictive value when clinical text is informative\[[1](https://arxiv.org/html/2609.21755#bib.bib14)\]\.

### 3\.3RQ2: Mitigating ECG Mirage

We investigated whether our VPT approach mitigates ECG Mirage while improving clinical outcome prediction\. Table[1](https://arxiv.org/html/2609.21755#S3.T1)compares VPT with zero\-shot inference, LoRA and supervised prompt tuning, using Qwen3\.5\-4B as the VLM backbone\. S4–MLP served as a deep\-learning baseline for predictive performance\.

VPT improved matched\-ECG performance over zero\-shot inference on both tasks, achieving balanced accuracies of 70\.6% for ICU admission and 67\.5% for clinical deterioration, with corresponding F1 scores of 62\.8% and 21\.7% under the task\-specific definitions\. Matched ECGs outperformed mismatched ECGs by 16\.5 and 5\.5 percentage points, respectively, and no\-image inputs by 15\.1 and 11\.9 percentage points\. VPT had the largest matched–mismatched balanced\-accuracy gaps among the evaluated VLM methods\. Although LoRA achieved higher absolute predictive performance, its performance remained similar across ECG conditions\. These comparisons show that improvements in task performance need not be accompanied by greater benefit from patient\-matched ECGs\. LoRA substantially improved prediction even without an image, while the prompt\-tuning results indicate that a benefit from adding an ECG need not depend on patient matching\. Our VPT approach combined improved matched performance over zero\-shot inference with higher scores for matched ECGs than for either control, supporting mitigation of ECG Mirage in the evaluated setting\. Its main benefit was therefore improved prediction over the unadapted model combined with a clearer distinction between matched and mismatched ECGs\.

Table 2:Ablation study on clinical deterioration prediction\. Performance is measured with balanced accuracy \(%\)\.
### 3\.4RQ3: Contributions of the adaptation components

We conducted ablation experiments on clinical deterioration prediction to assess the contributions of conditional preference optimisation, supervised initialisation and the visual attention mask\. Table[2](https://arxiv.org/html/2609.21755#S3.T2)compares VPT with three variants: w/o DPO, which uses supervised training alone; w/o SFT stage, which removes the separate supervised training stage and jointly optimises the SFT and DPO losses from the outset; and w/o Visual Mask, which allows text tokens to attend directly to the learned prompts\. VPT achieved the highest matched balanced accuracy among the evaluated configurations \(67\.5%\)\. The w/o DPO variant achieved similar scores with matched \(63\.4%\) and mismatched \(63\.7%\) ECGs, although both exceeded its no\-image performance \(55\.4%\)\. Relative to this variant, VPT improved matched performance and reduced mismatched performance to 62\.1%, with both changes contributing to the larger separation\. The Joint SFT\+DPO variant achieved 56\.5% with matched ECGs and similar scores under mismatched \(56\.3%\) and no\-image \(55\.5%\) conditions, favouring a separate supervised initialisation stage over joint optimisation from the outset in this setting\. The w/o Visual Mask variant retained sequential training but achieved similar matched \(64\.3%\) and mismatched \(64\.1%\) scores\.

## 4Conclusion

In this study, we identified ECG Mirage in multimodal clinical outcome prediction, where VLM performance can obscure benefit from patient\-matched ECG information\. Across four zero\-shot VLMs and two ED prediction tasks, matched ECGs offered no consistent advantage over mismatched ECGs or EHR text alone\. We introduced a lightweight approach combining restricted visual prompts and conditional preference optimisation with a frozen backbone\. VPT improved matched\-ECG performance over zero\-shot inference and increased matched–mismatched performance gaps on both tasks\. These findings support evaluating multimodal clinical models through both predictive performance and patient\-matched modality benefit\. However, LoRA achieved higher predictive performance with matched ECGs than VPT on both tasks\. Future work aims to improve predictive performance while ensuring that models benefit from patient\-specific ECG information\.

## 5Compliance with Ethical Standards

This study retrospectively analysed de\-identified data from MIMIC\-IV, accessed through PhysioNet under the respective data use agreements\. The Beth Israel Deaconess Medical Centre Institutional Review Board approved the original MIMIC\-IV data sharing with a waiver of informed consent\. Our study involved no patient recruitment or clinical intervention\.

## 6Acknowledgements

The authors declare no conflicts of interest\.

## References

- \[1\]T\. A\. Buckley, J\. A\. Diao, C\. N\. Srivastava, P\. G\. Brodeur, P\. Rajpurkar, A\. Rodman, and A\. K\. Manrai\(2026\)Multimodal foundation models exploit text to make medical image predictions\.Nature Communications17,pp\. 7475\.Cited by:[§1](https://arxiv.org/html/2609.21755#S1.p2.1),[§3\.2](https://arxiv.org/html/2609.21755#S3.SS2.p2.1)\.
- \[2\]E\. Chenet al\.\(2023\)Multimodal clinical benchmark for emergency care \(MC\-BEC\): a comprehensive benchmark for evaluating foundation models in emergency medicine\.Advances in Neural Information Processing Systems36,pp\. 45794–45811\.Cited by:[§1](https://arxiv.org/html/2609.21755#S1.p1.1)\.
- \[3\]L\. Chen, J\. Li, X\. Dong, P\. Zhang, Y\. Zang, Z\. Chen, H\. Duan, J\. Wang, Y\. Qiao, D\. Lin,et al\.\(2024\)Are we on the right way for evaluating large vision\-language models?\.Advances in Neural Information Processing Systems37,pp\. 27056–27087\.Cited by:[§1](https://arxiv.org/html/2609.21755#S1.p2.1)\.
- \[4\]T\. Chen, M\. Zhu, Z\. Luo, and T\. Zhu\(2026\)Cross\-representation benchmarking in time\-series electronic health records for clinical outcome prediction\.InICASSP 2026\-2026 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 7076–7080\.Cited by:[§3\.1](https://arxiv.org/html/2609.21755#S3.SS1.p2.1)\.
- \[5\]B\. Efron\(1979\)Bootstrap methods: another look at the jackknife\.The Annals of Statistics7\(1\),pp\. 1–26\.External Links:[Document](https://dx.doi.org/10.1214/aos/1176344552)Cited by:[§3\.1](https://arxiv.org/html/2609.21755#S3.SS1.p4.1)\.
- \[6\]B\. Gow, T\. Pollard, L\. A\. Nathanson, A\. Johnson, B\. Moody, C\. Fernandes, N\. Greenbaum, J\. W\. Waks, P\. Eslami, T\. Carbonati,et al\.\(2023\)MIMIC\-IV\-ECG: diagnostic electrocardiogram matched subset\.PhysioNet\.Note:Version 1\.0External Links:[Document](https://dx.doi.org/10.13026/4nqg-sb35)Cited by:[§3\.1](https://arxiv.org/html/2609.21755#S3.SS1.p1.1)\.
- \[7\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§3\.1](https://arxiv.org/html/2609.21755#S3.SS1.p3.1)\.
- \[8\]M\. Jia, L\. Tang, B\. Chen, C\. Cardie, S\. Belongie, B\. Hariharan, and S\. Lim\(2022\)Visual prompt tuning\.InEuropean conference on computer vision,pp\. 709–727\.Cited by:[§2\.3](https://arxiv.org/html/2609.21755#S2.SS3.p1.1)\.
- \[9\]A\. Johnson, L\. Bulgarelli, T\. Pollard, L\. A\. Celi, R\. Mark, and S\. Horng\(2023\)MIMIC\-IV\-ED\.PhysioNet\.Note:Version 2\.2External Links:[Document](https://dx.doi.org/10.13026/5ntk-km72),[Link](https://physionet.org/content/mimic-iv-ed/2.2/)Cited by:[§3\.1](https://arxiv.org/html/2609.21755#S3.SS1.p1.1)\.
- \[10\]A\. Johnson, L\. Bulgarelli, T\. Pollard, S\. Horng, L\. A\. Celi, and R\. Mark\(2023\)MIMIC\-IV\.PhysioNet\.Note:Version 2\.2External Links:[Document](https://dx.doi.org/10.13026/6mm1-ek67),[Link](https://physionet.org/content/mimiciv/2.2/)Cited by:[§3\.1](https://arxiv.org/html/2609.21755#S3.SS1.p1.1)\.
- \[11\]X\. Lan, F\. Wu, K\. He, Q\. Zhao, S\. Hong, and M\. Feng\(2025\)GEM: empowering MLLM for grounded ECG understanding with time series and images\.Advances in Neural Information Processing Systems38,pp\. 94421–94455\.Cited by:[§1](https://arxiv.org/html/2609.21755#S1.p1.1)\.
- \[12\]B\. Lester, R\. Al\-Rfou, and N\. Constant\(2021\)The power of scale for parameter\-efficient prompt tuning\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 3045–3059\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.243),[Link](https://aclanthology.org/2021.emnlp-main.243/)Cited by:[§3\.1](https://arxiv.org/html/2609.21755#S3.SS1.p3.1)\.
- \[13\]R\. Liu, Y\. Bai, X\. Yue, and P\. Zhang\(2026\)Teaching multimodal LLMs to comprehend 12\-lead electrocardiographic images\.npj Digital Medicine9\(1\),pp\. 349\.Cited by:[§1](https://arxiv.org/html/2609.21755#S1.p1.1)\.
- \[14\]J\. M\. Lopez Alcaraz, H\. Bouma, and N\. Strodthoff\(2025\)Enhancing clinical decision support with physiological waveforms—a multimodal benchmark in emergency care\.Computers in Biology and Medicine192,pp\. 110196\.External Links:[Document](https://dx.doi.org/10.1016/j.compbiomed.2025.110196)Cited by:[§1](https://arxiv.org/html/2609.21755#S1.p1.1),[§3\.1](https://arxiv.org/html/2609.21755#S3.SS1.p1.1),[§3\.1](https://arxiv.org/html/2609.21755#S3.SS1.p3.1)\.
- \[15\]I\. Loshchilov and F\. Hutter\(2019\)Decoupled weight decay regularization\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Bkg6RiCqY7)Cited by:[§3\.1](https://arxiv.org/html/2609.21755#S3.SS1.p3.1)\.
- \[16\]Qwen Team\(2026\)Qwen3\.5: towards native multimodal agents\.Note:[https://qwen\.ai/blog?id=qwen3\.5](https://qwen.ai/blog?id=qwen3.5)Cited by:[§3\.2](https://arxiv.org/html/2609.21755#S3.SS2.p1.1)\.
- \[17\]R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn\(2023\)Direct preference optimization: your language model is secretly a reward model\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 53728–53741\.Cited by:[§1](https://arxiv.org/html/2609.21755#S1.p3.1)\.
- \[18\]T\. Razzaq, M\. Taj, and A\. Iqbal\(2026\)Multimodal ai in healthcare: review of vision\-language foundation models for real\-world medical applications\.Journal of Biomedical Informatics,pp\. 105075\.Cited by:[§1](https://arxiv.org/html/2609.21755#S1.p1.1)\.
- \[19\]A\. Sellergrenet al\.\(2026\)MedGemma 1\.5 technical report\.arXiv preprint arXiv:2604\.05081\.Cited by:[§3\.2](https://arxiv.org/html/2609.21755#S3.SS2.p1.1)\.
- \[20\]G\. Team, S\. E\. Abd, V\. Aggarwal, R\. Algayres, A\. Andreev, O\. Bachem, I\. Ballantyne, C\. Brick, V\. Cărbune, M\. Casbon,et al\.\(2026\)Gemma 4 technical report\.arXiv preprint arXiv:2607\.02770\.Cited by:[§3\.2](https://arxiv.org/html/2609.21755#S3.SS2.p1.1)\.
- \[21\]F\. Wang, W\. Zhou, J\. Y\. Huang, N\. Xu, S\. Zhang, H\. Poon, and M\. Chen\(2024\)mDPO: conditional preference optimization for multimodal large language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 8078–8088\.Cited by:[§1](https://arxiv.org/html/2609.21755#S1.p3.1)\.
- \[22\]M\. Zhu, Y\. Liu, Z\. Luo, and T\. Zhu\(2026\)Bridging data gaps of rare conditions in ICU: a multi\-disease adaptation approach for clinical prediction\.npj Digital Medicine9\(1\),pp\. 7\.Cited by:[§1](https://arxiv.org/html/2609.21755#S1.p1.1)\.
- \[23\]M\. Zhu, Y\. Liu, Z\. Luo, and T\. Zhu\(2026\)The taxonomies, training, and applications of event stream modelling for electronic health records\.arXiv preprint arXiv:2603\.14003\.Cited by:[§1](https://arxiv.org/html/2609.21755#S1.p1.1)\.
- \[24\]M\. Zhu, Z\. Luo, Y\. Liu, and T\. Zhu\(2026\)From token to token pair: efficient prompt compression for large language models in clinical prediction\.InProceedings of the 43rd International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2609.21755#S1.p1.1)\.