LLM4EHR: Aligning Clinical Time Series with Medical Event Sequences via Large Language Models

# LLM4EHR: Aligning Clinical Time Series with Medical Event Sequences via Large Language Models
Source: [https://arxiv.org/html/2607.15447](https://arxiv.org/html/2607.15447)
\[type=editor, orcid=0009\-0003\-0011\-5828\]

\\credit

Writing \- Original Draft, Writing \- Review & Editing, Data Curation, Visualization, Resources, Investigation, Formal analysis, Validation, Software, Methodology, Conceptualization

\[type=editor, orcid=0000\-0002\-3884\-8045,\]

\\credit

Writing \- Review & Editing, Methodology

\[type=editor, orcid=0009\-0008\-7252\-6622,\]

\\credit

Writing \- Review & Editing

\[type=editor, orcid=0009\-0006\-5785\-4295,\]

\\credit

Writing \- Review & Editing, Methodology

\[type=editor, orcid=0000\-0001\-5348\-9063,\]

\\credit

Writing \- Review & Editing, Supervision, Project administration, Funding acquisition

\[type=editor, orcid=0000\-0001\-8591\-9638,\]

\\credit

Writing \- Review & Editing, Methodology, Resources, Project administration, Supervision, Funding acquisition, Conceptualization

1\]organization=GOS Institute of Child Health, addressline=University College London, city=London, country=United Kingdom

2\]organization=Department of Brain Sciences, addressline=Imperial College London, city=London, country=United Kingdom

3\]organization=UK Dementia Research Institute, addressline=Care Research and Technology Centre, city=London, country=United Kingdom

4\]organization=NIHR Great Ormond Street Hospital Biomedical Research, addressline=Centre and GOSH Data Research, Innovation and Virtual Environments, city=London, country=United Kingdom

\\cortext

\[cor1\]Corresponding author\\cortext\[cor2\]Corresponding author

Alexander CapstickLouise RignyIona BiggartNeil J SebirePayam Barnaghip\.barnaghi@imperial\.ac\.uk\[\[\[\[

###### Abstract

Recent research in clinical machine learning, focusing on outcome predictions in intensive care unit \(ICU\), has shifted from bespoke supervised models to foundation models, utilising modern representation learning methods\. Here, foundation models are pre\-trained on mixtures of complex clinical data modalities, useful for various downstream tasks\. Existing works often utilise Electronic Health Records \(EHR\) to provide rich and diverse patient observations to train clinical foundation models\. However, existing methods do not sufficiently explore the shared temporal structures between clinical events and time series \(TS\) observations recorded in EHRs\. This limitation potentially leads to less robust and adaptive clinical foundation models, resulting in reduced performance on downstream tasks\. To fully exploit this temporal structure, we propose LLM4EHR, a new clinical foundation model trained on ICU EHR data\. Combining domain adapted large language models with a transformer TS encoder, we pre\-trained LLM4EHR by temporally aligning the EHR events and TS\. For this, we propose a regularised contrastive objective to learn robust EHR TS representations conditioned on EHR event embeddings produced by the domain adapted LLM\. Supported by an ablation study, we find that learnt EHR TS embeddings from LLM4EHR improve performance on various downstream clinical tasks with competitive performance\. Further, we empirically demonstrate that LLM4EHR learns transferable clinical TS embeddings that can be deployed to new cohorts via k\-shot adaptation\. These findings provide a step towards building more generalisable and performant clinical foundation models\.

###### keywords:

Electronic health records\\sepClinical predictive modeling\\sepContrastive learning\\sepRepresentation learning\\sepLarge Language Model

\{highlights\}

LLM4EHR: Introduces a multimodal fusion framework that temporally aligns LLM‑encoded Electronic Health Record \(EHR\) event sequences with a dedicated EHR Time Series \(TS\) encoder via a semantic regularised contrastive objective\.

Improved prediction outcomes: LLM4EHR achieved consistent performance gains on clinically relevant downstream tasks including mortality, phenotyping, decompensation, and remaining Length of Stay \(LoS\) prediction when compared to supervised and self\-supervised baselines\.

Improved transferability: We empirically demonstrate that LLM4EHR learns transferable clinical TS embeddings that can be deployed to new cohorts via k\-shot adaptation\.

Ablation and sensitivity: Ablation study involving different pre\-training objectives showed that our semantic regularised alignment objective enabled LLM4EHR to achieve improved prediction outcomes in downstream tasks\.

## 1Introduction

Electronic Health Records \(EHRs\) data is inherently multimodal\. Temporal patient observations, interactions and measurements are recorded as EHR events and EHR Time Series \(TS\), as shown inFigure[1](https://arxiv.org/html/2607.15447#S1.F1)aandFigure[1](https://arxiv.org/html/2607.15447#S1.F1)b\. Specifically, EHR events consist of itemised, irregularly sampled clinical interactions with specific time stamps, while EHR TS includes routinely collected patient measurements for patient monitoring\. In EHR data modelling, entries such as medication prescriptions are typically treated as events due to the lack of fine\-grained temporal dynamics\. Conversely, monitoring signals such as patient vital signs are best modelled as TS due to the clinical significance of temporal variations in these signals\. These two data modalities complement each other in representing a patient’s health\. However, the structural differences between these modalities pose a significant challenge for building versatile clinical foundation models\.

Towards overcoming this, we release a new structure for clinical foundation models, LLM4EHR, which jointly models EHR events and EHR TS via a time and semantic\-aware contrastive pre\-training objective\. We find that the resulting learnt representations can adapt to downstream clinical tasks such as mortality prediction, phenotyping and remaining Length of Stay \(LoS\) prediction, which we support with an ablation study\. Additionally, we show that LLM4EHR can be adapted to new datasets using routinely collected EHR TS variables with consistent performance\.

Traditionally, bespoke supervised methods were developed individually for clinical tasks, including survival analysis\[Ghassemi2014,Barajas2015\], risk prediction\[Alvarez2013,Cheng2016\]and phenotyping\[Albers2014,Liu2015\]\. However, experiments conducted on large multitask clinical benchmarks\[MIMICBenchmark,eICUBenchmark\]demonstrated that learnt information from one task is often relevant for others\. Furthermore, patient observations in the clinical setting form temporal sequences, making them well\-suited for language modelling techniques\. These observations, along with recent findings that Large Language Models \(LLMs\) function as unsupervised multitask learners\[GPT\-2\], have driven efforts to develop multitask clinical foundation models\[Wornow2023\]\. Recent work has thus explored using pre\-trained LLMs for modelling EHR event trajectories\[BEHRT,MedBERT,ETHOS,Foresight\]or EHR TS\[LLM4TS,AutoTimes\]\. A review byWornow2023found that clinical foundation models, pre\-trained on large unlabelled EHR datasets, can adapt to downstream tasks using limited labelled data\.

However, many current methods fail to incorporate multiple temporal EHR data modalities during pre\-training, despite evidence that combining modalities such as clinical notes and TS improves downstream performance\[king2023,MedsTsLLM,MAIN\]\. Given the interdependency between EHR events and EHR TS observations, we hypothesise that similar improvements could be achieved by jointly modelling these two temporal modalities via self\-supervised pre\-training\.

Prior work aligning these different clinical data modalities focused on contrastive representation learning between pairs of samples\[Kline2022\]\. In these cases, the alignment is often centred on shared identities between modalities, such as aligning the clinical TS and note embeddings describing the same care period\[king2023,MAIN\]\. However, this stationary, instance\-wise alignment across clinical data modalities does not consider the shared temporal structure between EHR events and EHR TS observations, which we hypothesise provide more effective learnt representations\.

![Refer to caption](https://arxiv.org/html/2607.15447v1/x1.png)Figure 1:a\) An example of EHR TS, b\) an example of EHR events and c\) the difference between our regularised contrastive objective and the InfoNCE\[InfoNCE,he2020momentum\]objective, during LLM4EHR pre\-training\. EHR events embeddings produced by a domain adapted LLM areused to guide the EHR TS embeddings via our proposed objective\.Unlike prior work, LLM4EHR learns entangled EHR TS representations with EHR events embeddings, where a trainable EHR TS encoder is guided by a pre\-trained and domain\-adapted large language model \(LLM\) via contrastive pre\-training\. We reformulate the commonly used InfoNCE\[InfoNCE,he2020momentum\]objective to temporally align the EHR TS embedding with EHR event embedding produced by a domain\-adapted LLM, where EHR TS embedding is pulled closer to EHR event embedding at each time step\. This leads to improved performance in downstream tasks that generalise to new datasets\.

Additionally, we proposed incorporating learnt semantic similarities from the pre\-trained LLM into the contrastive objective to mitigate the effects of class collision\[SimCLR,Zheng2021,Wu2024\]in contrastive learning\. Class collision arises when similar instances are pushed apart in the embedding space due to strict one\-to\-one positive sampling\. If EHR TS observations atttandt′t^\{\\prime\}correspond to similar EHR events, they should not be treated as negatives, as implied by the InfoNCE loss\. We present a semantic\-regularised InfoNCE objective\[InfoNCE,he2020momentum\], where the semantic similarities between LLM\-embedded EHR events are used to weigh the learnt EHR TS embeddings \(Figure[1](https://arxiv.org/html/2607.15447#S1.F1)c\), providing improved EHR TS embeddings\. Our contributions are as follows:

- •We present LLM4EHR, an LLM\-based framework for representation learning of EHR TS, trained on data from intensive care units\.
- •To train LLM4EHR, we propose a contrastive learning objective to temporally align EHR events and EHR TS embeddings\. With an ablation study, we find that this improves the learnt representations\.
- •We show that our method achieves superior performance in clinical classification tasks in few\-shot and cross\-dataset settings\.

![Refer to caption](https://arxiv.org/html/2607.15447v1/x2.png)Figure 2:Overview of LLM4EHR, EHR entries and clinical TS are centred on ICU admissions, where EHR operations are coloured in beige and TS operations are coloured in red, pre\-trained LLM layers \(besides the embedding weights for new tokens\) are frozen during pre\-training\.
## 2Related work

EHR foundation models are clinical foundation models trained on large\-scale EHR data\[Wornow2023\]\. Recent methods exhibit notable similarities with LLMs in that they are trained on tokenised event sequences via autoregressive\[ETHOS,Foresight,MedGPT,Cascella2023\]or masked sequence modelling\[MedBERT,BEHRT,BioClinicalBERT,CEHR\-BERT\]\. Notable examples include Foresight byForesightand ETHOS byETHOS, which repurposed the autoregressive attention layers of GPT\-2\[GPT\-2\]to simulate event transitions in patient timelines\. However, these works primarily treat patient trajectories as event sequences, which can lead to fine\-grained temporal dynamics being lost or obscured during modelling\.

Self\-supervised representation learning for TS learns transferable representations of TS features from unlabelled data for downstream tasks\[SimMTM,PatchTST,PatchTSMixer\]\. Recent work demonstrated improved performance in TS forecasting by modelling TS via transformer architectures\[Transformer\]\. Here, TS segments are tokenised to form temporal sequences, and attention layers are trained to process TS tokens via self\-supervised pretext tasks\. LLM4TS byLLM4TSused self\-supervised reconstruction to repurpose pre\-trained LLMs for TS forecasting\. Similarly, AutoTimes byAutoTimesused language prompts during self\-supervised pre\-training to allow in\-context TS forecasting via frozen LLMs\. Different from prior works, LLM4EHR maintains a dedicated transformer encoder for EHR TS, where EHR TS embeddings are guided by EHR event embeddings produced by a pre\-trained LLM during pre\-training\.

Contrastive learning can create entangled representations between pairs of samples\. Prior works in clinical machine learning explored aligning similar EHR TS samples via contrastive learning\. EBCL byEBCLused contrastive pre\-training to align TS observations before and after key index events and showed improved performances in downstream classification tasks\. CROCS byKiyasseh2021used contrastive learning to align ECG measurement embeddings with patient prototype embeddings, where patient prototypes serve as queries to retrieve similar ECG samples\.king2023explored aligning clinical time series \(TS\) with ICU notes via a shared LLM\. However, their work maintains a stationary alignment between EHR TS and clinical notes and does not consider important temporal relationships\.

## 3Background

### 3\.1EHR data representation

When patients interact with care services, each measurement, observation, and care giver action is recorded in an Electronic Health Record \(EHR\)\. While different types of EHR data share the same structured format, they are generated through different processes and often use different data modalities\. Regularly collected physiological observations are usually modelled as clinical Time Series \(TS\) due to their dynamic nature\. These variables are tracked over time to monitor a patient’s health status, with abrupt changes contributing heavily to standardised risk analyses such as SAPS\[SAPS\]or SOFA\[SOFA\]\. Supervised methods in clinical machine learning often rely on TS features as input for downstream tasks, especially for time\-sensitive predictions such as survival\[Ghassemi2014,Barajas2015\]or risk\[Albers2014,Liu2015\]predictions\.

In contrast, certain events or occurrences in EHR systems are not conveniently or logically modelled as TS variables\. For example, while the presence of a standing drug prescription at a given time is clinically meaningful, modelling it as a TS variable is not informative due to the lack of temporal variation\. Additionally, each unique event would be represented as a separate binary variable, which scales poorly given the natural diversity of unique interactions in EHR systems\. EHR events are typically tokenised to form temporal sequences for sequence modelling\[Meds\_reader,ETHOS,Foresight\], which simplifies the temporal dimension as longitudinal ordering of tokens\.

Similar to prior work on cross\-modal alignment between EHR TS and clinical notes, we define the problem of modelling multi\-modal EHR data as cross\-modal alignment between EHR TS and events, where TS variables and EHR events are generated separately but aligned in time\. We did not use clinical notes in our framework, as while clinical notes provide a high\-level summary of patient observations and treatments in the ICU, they are often documented with substantial delay and therefore cannot be assumed to align temporally with fine\-grained TS observations\.

We define clinical time series as multivariate time series consisting of time varying patient observations, primarily lab and nurse charted lab observations\. While some work does not treat irregularly sampled lab values as TS variables\[stein2025prediction\], we included them as changes in lab observations over time are clinically significant\. Formally, we define an instance of clinical TS withLLvariables overTTtime steps asx1:T=\{x1,x2,⋯,xT\},xt∈ℝLx^\{1:T\}=\\\{x^\{1\},x^\{2\},\\cdots,x^\{T\}\\\},x^\{t\}\\in\\mathbb\{R\}^\{L\}\.

EHR events, however, are represented as time\-indexed sequences\. We define an EHR event sequence withNNevents overTTask1:T=\{kn11,kn22,⋯,knTT\}k^\{1:T\}=\\\{k\_\{n\_\{1\}\}^\{1\},k\_\{n\_\{2\}\}^\{2\},\\cdots,k\_\{n\_\{T\}\}^\{T\}\\\}, wherekntt∈𝒦k\_\{n\_\{t\}\}^\{t\}\\in\\mathcal\{K\}is a set ofntn\_\{t\}clinical events observed at time steptt\. In this case, we treat time\-stamped care giver actions as EHR events\. In MIMIC\-IV, these entries include prescription orders, medication administrations, infusion orders, renal replacement therapies \(RRT/CRRT\) settings, oxygen\-delivery settings and the insertion, presence or removal of invasive lines\. Since these events are not regularly sampled or distributed evenly overtime, the number of eventsntn\_\{t\}at timettis not fixed\.

Finally, we consider data collected for a given patient throughout a single Intensive Care Unit \(ICU\) stay as an ’episode’\. An episode starts at ICU admission and ends at ICU discharge\. A full episode is defined asei=\(xi1:Ti,ki1:Ti\)e\_\{i\}=\(x\_\{i\}^\{1:T\_\{i\}\},k\_\{i\}^\{1:T\_\{i\}\}\), where bothxix\_\{i\}andkik\_\{i\}are centred on ICU admission and are aligned with the time indicatort∈Tit\\in T\_\{i\}\.

### 3\.2Multimodal pre\-training of EHR data

Pre\-training on clinical TS aims to learn general temporal patterns in patient trajectories using a task agnostic objective\. The goal of the pre\-training objective is to reward the model for capturing clinically meaningful dynamics, such as trends or sudden changes in physiological observations, without bias toward any specific downstream task\. Because EHR TS and EHR events are generated through separate processes but are temporally aligned, we hypothesise that explicitly aligning these two modalities during pre\-training allows TS encoders to learn more informative EHR TS representations, leading to better performance in context sensitive downstream tasks such as decompensation and Length\-of\-stay predictions\.

Existing multimodal pre\-training frameworks for EHR data involve stationary alignment of two EHR modalities via a contrastive objective\. Here, we focus on multimodal pre\-training methods involving non\-imaging data\. The most common form of alignment occurs between clinical TS and clinical notes, where the sequence\-wise representation of EHR TS across an episode is pulled closer to the embedding of clinical notes describing the same episode\. The best example is the multimodal pre\-training framework byking2023, where a masked attention encoder is used to encode clinical TS and the resulting embedding is aligned with the clinical note embedding produced from a domain adapted BERT model\[BioClinicalBERT\]via a contrastive objective\. We argue that stationary alignment between clinical TS and notes is insufficient due to a mismatch in temporal granularity, as aligning fine\-grained TS representations with coarse\-grained clinical note embeddings fails to adequately capture the shared temporal dynamics between the two modalities\.

A more recent work byCTPDexplored learning shared temporal patterns between clinical TS and notes, where TS and clinical notes embeddings are first aligned at the attention level via a contrastive objective, expressed over individual time steps, and then concatenated at the final level for final output\. We note that CTPD is susceptible to time mismatches between EHR TS and clinical notes, as clinical notes are not synchronised with EHR TS observations\. Additionally, CTPD explicitly assumes that clinical notes are available during prediction, which is counterintuitive for time\-sensitive predictions, as clinical notes are not produced at the point of observation and are unlikely to be available at the prediction time\.

Additionally, the semantic knowledge in pre\-trained LLMs is not utilised in either framework\. Clinical LLMs are used only as fixed encoders for clinical notes, without modifying the pre\-training objective to transfer semantic structure into the TS representation\. Therefore, we propose a similar framework that aligns EHR event summaries with EHR TS embeddings through an explicit temporal alignment objective derived from InfoNCE, while also leveraging the LLM’s learnt semantic similarity to guide the EHR TS representation\.

## 4Methods

### 4\.1LLM4EHR structure

Table 1:Example aggregated EHR summary for a 5‑hour window \(70–75 hours after ICU admission\), the patient was given Furosemide, pantoprazole by the care givers and received scheduled Insulin infusion, no invasive lines and renal replacement therapies were active\.SummaryTextPrescriptionsPrescription Orders:Prescriptions started:Furosemide \(40mg/4ml Vial\),Pantoprazole \(40mg Tablet\)Prescriptions ended:Furosemide \(40mg/4ml Vial\)emarAdministered:Furosemide,Insulin,Pantoprazole,RemdesivirInfusionInfusion Order:\- NaCl 0\.9% 250ml \(active\)\- Furosemide \(Lasix\) 40\.00mg \(active\)\- Insulin \- Humalog 4\.00 units \(active\)rrt/crrtNo active renal replacement therapyOxygenDelivery device:Oxymizer, High flow nasal cannulaOxygen flow:6\.0Additional Oxygen flow:NoneInvasive linesNo invasive lines activeTS\- albumin:mean: 3\.50; min: 2\.80; max: 3\.33; std: 0\.28; count: 2\- no measurements for fio2\- no measurements for Tropinin T……

EHR events and EHR TS provide complementary views of a patient’s ICU stay\. We hypothesise that conditioning TS representations on information from EHR events yields more informative and generalisable TS embeddings for downstream clinical tasks, such as predicting intensive care outcome or patient decompensation\. Given a patient’s full EHR episodeeie\_\{i\}, consisting of EHR event sequenceki1:Tk\_\{i\}^\{1:T\}and multivariate EHR TSxi1:Tx\_\{i\}^\{1:T\}, LLM4EHR pre\-training aligns the EHR events embeddingzi1:Tz\_\{i\}^\{1:T\}with the TS embeddingsvi1:Tv\_\{i\}^\{1:T\}via a contrastive objective applied at each time step and aggregated overTT\.

Aligning EHR events and TS embeddings directly step\-by\-step is problematic due to the structural differences between EHR events and EHR TS data\. First, as explained inSection[3\.1](https://arxiv.org/html/2607.15447#S3.SS1), EHR events are not regularly sampled, and the number of eventsntn\_\{t\}at time stepttis not fixed\. Expressing a contrastive objective directly overTTresults in an ambiguous objective\. Additionally, jointly learning EHR TS and EHR event embeddings is unstable for contrastive learning\. Since the learnt EHR TS embedding is conditioned on EHR events embedding in LLM4EHR, rapidly changing the EHR events embedding reduces the consistency of the alignment objective\[he2020momentum\]\.

We propose using a domain adapted LLM to embed EHR events, similar to prior work bySun2024\. We define the EHR event set𝒦\\mathcal\{K\}as a set of natural language phrases describing each unique EHR event, such as the name and dosage of a prescription order\. Different fromSun2024, multiple EHR events atttare aggregated into a single natural language summary using a fixed schema\. An example of the aggregated summary is shown inTable[1](https://arxiv.org/html/2607.15447#S4.T1)\. Here,zi1:Tz\_\{i\}^\{1:T\}is produced by a frozen LLM domain adapted using clinical domain texts andvi1:Tv\_\{i\}^\{1:T\}is produced by a trainable TS encoder, as shown inFigure[2](https://arxiv.org/html/2607.15447#S1.F2)\.

In MIMIC\-IV, we found that aligning EHR summary embeddings with EHR TS embeddings hourly is suboptimal due to the relative sparsity of EHR events, and therefore enforced a bin width of55hours\. In practice, a small bin would result in many instances ofvitv\_\{i\}^\{t\}with no correspondingzitz\_\{i\}^\{t\}\. Conversely, a large bin would degrade the temporal consistency of the alignment process, similar to the temporal mismatch between EHR TS and note alignment inSection[3\.2](https://arxiv.org/html/2607.15447#S3.SS2)\. EHR TS variables within each time step are aggregated with windowed features \(mean, min, max, std and count\)\. Finally, we append a plain text summary of TS variables to the end of EHR event summary atttto provide an additional layer of mutual information between the two modalities\.

### 4\.2Semantic alignment

![Refer to caption](https://arxiv.org/html/2607.15447v1/x3.png)Figure 3:Calculating the semantic alignment loss for a given episode; the figures only include the lower triangular and non\-diagonal part of the soft assignmentω\\omegasinceω\\omegais symmetric\.The attraction between elements ofvi1:Tv\_\{i\}^\{1:T\}andzi1:Tz\_\{i\}^\{1:T\}is defined at each timestamp via a modified InforNCE\[InfoNCE\]loss\. Similar tohe2020momentumandKiyasseh2021, we implemented the normalised variant of InfoNCEℒNCE\\mathcal\{L\}\_\{\\text\{NCE\}\}, with temperatureτ\\tauas:

ℒNCE=−∑t=1Tlog⁡exp⁡\(sim​\(vt,zt\)/τ\)∑t′∈T\\texp⁡\(sim​\(vt,zt′\)/τ\)\\displaystyle\\mathcal\{L\}\_\{\\text\{NCE\}\}=\-\\sum\_\{t=1\}^\{T\}\\log\\frac\{\\exp\{\(\\text\{sim\}\(v^\{t\},z^\{t\}\)/\\tau\)\}\}\{\\sum\_\{t^\{\\prime\}\\in T\\backslash t\}\\exp\{\(\\text\{sim\}\(v^\{t\},z^\{t^\{\\prime\}\}\)/\\tau\)\}\}\(1\)sim​\(vt,zt\)=vt⋅zt∥vt∥​∥zt∥\\displaystyle\\text\{sim\}\(v^\{t\},z^\{t\}\)=\\frac\{v^\{t\}\\cdot z^\{t\}\}\{\\lVert v^\{t\}\\rVert\\lVert z^\{t\}\\rVert\}\(2\)
However, we note that this objective is susceptible to class collision, where instances within same class are pushed apart due to the loss being defined between an instance and its augmented view\. In our case, similar instances of TS observations should remain closer after embedding\. Since LLMs are known to capture word similarities\[Mahajan2025\], we propose aω\\omega\-regularised variant of the InfoNCE loss to weight similarities between instances withinvi1:Tv\_\{i\}^\{1:T\}based on the semantic similarities between instances withinzi1:Tz\_\{i\}^\{1:T\}\. FollowingEquation[1](https://arxiv.org/html/2607.15447#S4.E1), the regularisation functionℒω\\mathcal\{L\}\_\{\\omega\}is given as:

ℒω=−∑t=1T∑t∗=1​t∗<tTωt,t∗​log⁡exp⁡\(sim​\(vt,vt∗\)/τ\)∑t′∈T\\texp⁡\(sim​\(vt,vt′\)/τ\)\\displaystyle\\mathcal\{L\}\_\{\\omega\}=\-\\sum\_\{t=1\}^\{T\}\\sum\_\{\\begin\{subarray\}\{c\}t^\{\*\}=1\\ t^\{\*\}<t\\end\{subarray\}\}^\{T\}\\omega\_\{t,t^\{\*\}\}\\log\\frac\{\\exp\(\\text\{sim\}\(v^\{t\},v^\{t^\{\*\}\}\)/\\tau\)\}\{\\sum\_\{t^\{\\prime\}\\in T\\backslash t\}\\exp\{\(\\text\{sim\}\(v^\{t\},v^\{t^\{\\prime\}\}\)/\\tau\}\)\}\(3\)ωt,t∗=exp⁡\(sim​\(zt,zt∗\)\)∑t′∈T\\texp⁡\(sim​\(zt,zt′\)\)\\displaystyle\\omega\_\{t,t^\{\*\}\}=\\frac\{\\exp\(\\text\{sim\}\(z^\{t\},z^\{t^\{\*\}\}\)\)\}\{\\sum\_\{t^\{\\prime\}\\in T\\backslash t\}\\exp\(\\text\{sim\}\(z^\{t\},z^\{t^\{\\prime\}\}\)\)\}\(4\)
The attraction weightωt,t∗\\omega\_\{t,t^\{\*\}\}is interpreted as the degree of agreement betweenvtv^\{t\}andvt∗v^\{t^\{\*\}\}, quantified using the similarity betweenztz^\{t\}andzt∗z^\{t^\{\*\}\}\. For a pair ofvtv^\{t\}andztz^\{t\}, the weightω\\omegais aT×TT\\times Tsymmetric matrix, as shown inFigure[3](https://arxiv.org/html/2607.15447#S4.F3)\. We thus used only the lower triangular part ofω\\omega, such thatωt,t∗\\omega\_\{t,t^\{\*\}\}is only defined fort∗<tt^\{\*\}<t\. Intuitively,Equation[3](https://arxiv.org/html/2607.15447#S4.E3)pulls TS observations corresponding to similar EHR events closer, such that similar instances withinxix\_\{i\}are consistent after embedding\. The overall alignment objective is defined as:

ℒalign=ℒNCE\+ℒω\\mathcal\{L\}\_\{\\text\{align\}\}=\\mathcal\{L\}\_\{\\text\{NCE\}\}\+\\mathcal\{L\}\_\{\\omega\}\(5\)
#### Pre\-training

The EHR summary embeddingzitz\_\{i\}^\{t\}is obtained by passing the EHR summaries for episodeeie\_\{i\}atttthrough the pre\-trained LLM, then applying masked average pooling over the hidden states after the last attention block\. Since EHR summaries contain complex clinical terminologies such as specific drug compounds or ventilator settings, we used domain adapted LLMs to embed EHR summaries\. We selected the BioClinical ModernBERT\[BioClinicalModernBert\], a version of ModernBERT\[ModernBERT\]further trained on clinical language datasets and with a long context length of 8192\. Because gradients cannot propagate through the frozen LLM, an additional dense projection layer is added for training stability\. The final EHR summary embeddingzitz\_\{i\}^\{t\}is produced by passing the average\-pooled EHR summary embedding through the projection layer\. During pre\-training, we update the weights of the projection layer with a large momentum \(m=0\.9m=0\.9\) via the Adam optimiser\[kingma2014adam\]to keepzitz\_\{i\}^\{t\}stable for calculatingℒalign\\mathcal\{L\}\_\{\\text\{align\}\}\.

As shown inFigure[2](https://arxiv.org/html/2607.15447#S1.F2), the EHR TS encoder is a separate, fully\-trainable encoder for encoding EHR TS\. We built the EHR TS encoder with 1D convolutional embedding layers, trainable positional embeddings for positional encoding, and autoregressive attention blocks\[GPT\-2\]\. Similar to LLM4TS byLLM4TSand AutoTimes byAutoTimes, we treat instances ofxitx\_\{i\}^\{t\}as tokens and employ autoregressive attention blocks to model the transition between TS observations\.

The EHR TS embeddingvi1:Tv\_\{i\}^\{1:T\}is produced by passing z\-standardised EHR TS features through the embedding layers, with added positional embeddings from passing the EHR TS positional encodings through the positional embeddings layers\. The EHR TS embeddings are then passed throughNNautoregressive attention blocks before calculating the alignment loss viaEquation[5](https://arxiv.org/html/2607.15447#S4.E5)\. To produce our final optimisation objective, we reconstruct the input TS autoregressively from the latent representationviv\_\{i\}and use the reconstruction loss as an auxiliary objective toℒalign\\mathcal\{L\}\_\{\\text\{align\}\}\. This provides the loss value:

ℒtotal=ℒalign\+ℒrecon=ℒalign\+1T∑t=2T∥xit−xi^t∥22\\mathcal\{L\}\_\{\\text\{total\}\}=\\mathcal\{L\}\_\{\\text\{align\}\}\+\\mathcal\{L\}\_\{\\text\{recon\}\}=\\mathcal\{L\}\_\{\\text\{align\}\}\+\\frac\{1\}\{T\}\\sum\_\{t=2\}^\{T\}\\rVert x\_\{i\}^\{t\}\-\\hat\{x\_\{i\}\}^\{t\}\\lVert^\{2\}\_\{2\}\(6\)
After pre\-training, LLM4EHR can be adapted to downstream clinical tasks using task\-specific linear heads, as with other TS foundation models\. During linear probing, only EHR TS observations are used to make downstream predictions\. The TS embedding layer, positional embedding layer and TS encoder attention blocks are kept frozen during fine\-tuning\. For complex longitudinal prediction tasks, such as rolling patient decompensation and length\-of\-stays predictions, we unfreeze the last TS encoder attention block to give LLM4EHR more flexibility to adapt to the unique temporal dynamics of these downstream tasks\.

## 5Results

### 5\.1Datasets

We evaluated the performance of our pre\-trained model on two public intensive care EHR datasets: MIMIC\-IV\[mimiciv\]and the Physionet Challenge 2012 \(Physionet2012\)\[Physionet2012\]\(Table[2](https://arxiv.org/html/2607.15447#S5.T2)\)\. We conducted evaluations using four downstream ICU outcome prediction tasks in MIMIC\-IV and used Physionet2012 as an external dataset to assess the transferability of models pre\-trained on MIMIC\-IV\. For MIMIC\-IV, we retained the first ICU stay for each unique hospital admission and excluded stays shorter than 48 hours\. Each ICU episode includes 32 physiological and laboratory variables as EHR TS variables, derived from nursing and laboratory observations recorded during the episode\. These variables are resampled hourly, and we imputed missing values by first front filling within each episode and then filling any remaining gaps using the population mean\. Continuous variables were z\-standardised using the observed mean and standard deviation before imputation\. A detailed summary of all EHR TS variables is available in the appendix\.

Table 2:Overview of the two datasets\. MIMIC‑IV is partitioned at the patient level; Physionet2012 uses predefined training/testing splits\. Numbers in parentheses for ICU hours indicate the average episode length \(hours\) within each partition\.StatisticsMIMIC\-IVPhysionet2012Patients \(training\)237814000Patients \(validation\)5096N/APatients \(testing\)50964000ICU hours \(training\)4082k \(145\)192k \(48\)ICU hours \(validation\)896k \(146\)N/AICU hours \(testing\)883k \(147\)192k \(48\)Mortality rate15\.4%14\.0%

We extracted prescription orders, medication administrations, infusion orders, oxygen delivery settings, invasive line operations and renal replacement therapies \(RRT/CRRT\) settings as EHR events for an ICU episode\. These events are extracted from corresponding tables and mapped to specific episodes via unique patient and ICU stay identifiers\. The EHR events and EHR TS observations for the same ICU episode are synchronised using their recorded timestamps\. We additionally processed some EHR events to LLM\-friendly phrases by mapping all medications to ATC classifications and expanding ambiguous abbreviations into full English phrases\.

### 5\.2Experimental setting

Similar to the ICU outcome prediction benchmarkMIMICBenchmark, we defined four downstream evaluation tasks based on ICU patient outcomes: \(1\) Mortality prediction uses the first 48 hours of measurements from an ICU episode to predict the risk of in\-hospital mortality; \(2\) Phenotyping classifies episodes agains 25 acute phenotypes extracted from ICD\-coded diagnoses; \(3\) Decompensation predicts immediate patient deteriorations within each episode, characterised by risk of organ dysfunctions; and \(4\) Remaining LoS is a time\-to\-event task to predict the days untile ICU discharge for a patient\. Both decompensation and remaining LoS are rolling predictions, meaning predictions are made at regular intervals throughout each episode\.

We compared LLM4EHR against supervised TS baselines and self‑supervised contrastive pre‑training methods for EHR TS modelling\. For experiments on MIMIC\-IV, we used the training set for self\-supervised pre\-training and supervised fine\-tuning, the validation set for hyperparameter tuning and reported all models’ performance on the test set\. For additional experiments on Physionet2012, we used the challenge’s pre\-defined training set for k\-shot adaptation and evaluated all models using the pre\-defined testing set\. All dataset splits were performed at the patient level, with Physionet2012 containing only the first4848hours of ICU stay per patient\. Deep learning models were trained using the Adam optimiser\[kingma2014adam\]on an Nvidia A100 GPU, and we reported performance across 10 random seeds for all models\. We performed hyperparameter searches task\-by\-task for supervised models and used pre\-training loss to determine the optimal hyperparameters for the self\-supervised models\. The hyperparameter search spaces for all baseline models are shown inTable[3](https://arxiv.org/html/2607.15447#S5.T3)\.

Table 3:Summary of the hyperparameter search space for the baseline models\.Hidden dimConfigurationSpecific parametersLRN/AN/AInverse L1 strength:\[0\.01, 0\.1, 1\.0, 3\.0, 10\.0\]XGBN/AN/AMax depth:\[3, 4, 5, 8, 10\]Subsample:\[0\.2, 0\.8, 1\.0\]N estimators:\[300, 500\]LSTM128, 256, 512N/AGRU\-D128, 256, 512N/AN/AT\-TS/T\-EHR256, 512N layers:\[4, 8\]N heads:\[4, 8\]N/ASimMTM256, 512N layers:\[4, 8\]N heads:\[4, 8\]N neighbours:\[2, 3, 4\]Mask prob:\[0\.1, 0\.3, 0\.5, 0\.8\]Temperature:\[0\.02, 0\.05, 0\.1\]King et al\.768N layers:\[4, 8\]N heads:\[4, 8\]Mask prob:\[0\.1, 0\.3, 0\.5\]Temperature:\[0\.02, 0\.05, 0\.1\]

For our implementation of LLM4EHR, we used a transformer with88autoregressive transformer layers with88attention heads per layer and a hidden dimension of512512as the EHR TS encoder\. EHR summary embeddings were extracted from BioClinical ModernBERT\[BioClinicalModernBert\], which is a domain adapted version of ModernBERT\[ModernBERT\]trained on medical texts\. To allow more efficient LLM4EHR pre\-training, the EHR embeddings were pre\-computed and are projected into LLM4EHR’s feature space via a dense projection layer, as described inSection[4\.2](https://arxiv.org/html/2607.15447#S4.SS2.SSSx1)\. We implemented the baseline models as:

\(1\) Linear Model \(LR\): Linear models included logistic regression for classifications and linear log‑normal accelerated failure time \(AFT\) model for remaining LoS predictions\. Similar tolipton2015, we extracted aggregated statistical features, including mean, minimum, maximum, standard deviation and count of measurements for each EHR TS variable\. For sequence‑level classification, we extracted features over multiple clinically relevant windows: the 24 hours before the first heart rate measurement, the 24 and 48 hour windows following it, and the full interval up to the decision point\. For rolling predictions, features were aggregated from the start of each patient trajectory up to the prediction point\.

\(2\) Gradient Boosting \(XGB\): We used the XGBoost implementation byxgboost, with the same feature extraction method as the linear models\. For classification, we trained the model using the binary logistic objective, and for LoS prediction, we used the log\-normal AFT objective

\(3\) LSTMThe input to the LSTM consisted of 32 imputed and z\-standardised EHR TS variables concatenated with 32 corresponding channel‑wise missingness indicator variables\. We found that concatenating channel\-wise missingness indicators with EHR TS variables yielded better single task performances than only using the EHR TS variables as input features\. All LSTM models were trained end\-to\-end with task specific heads\. For LoS prediction, we implemented the log\-normal AFT objective in PyTorch\[Pytorch\]following the implementation byweibullrnn\.

\(4\) GRU\-DWe implemented the GRU\-D bygrudas the second recurrent baseline\. Unlike LSTM, GRU\-D incorporates missingness indicators directly within the decayed GRU layer, allowing each partially observed variable to ’decay’ towards its observed mean during training\. GRU\-D additionally uses time delta variables as input features, which are the time since the last observation for each EHR TS variable\. Similar to LSTM, all GRU\-D models were trained end\-to\-end, and we used the log\-normal AFT objective for the remaining LoS prediction\.

\(5\) Supervised Transformers \(T\-TS & T\-EHR\)We included two supervised transformer baselines, T‑TS and T‑EHR, as ablated versions of pre\-trained LLM4EHR\. Both models share the same autoregressive transformer structures and task heads with LLM4EHR, but are trained directly with task supervision\. T\-TS uses EHR TS variables as input, and T\-EHR uses temporally aligned EHR summary embeddings as input\. These models are conceptually similar to supervised transformer models for clinical TS, such as SAnD bySAnD\. Both models use the log\-noraml AFT objective for LoS prediction\.

\(6\) SimMTMWe implemented SimMTM bySimMTMas a unimodal contrastive learning baseline\. SimMTM introduces a contrastive reconstruction objective for masked TS modelling, where each masked time step is reconstructed via a weighted aggregation fromnnneighbours\. We implemented the SimMTM pre\-training and aggregation objective with a bidirectional transformer encoder and pre\-trained SimMTM on the training set\. The input to SimMTM consists of 32 imputed and z\-standardised EHR TS variables concatenated with 32 corresponding channel‑wise missingness indicator variables\. The reconstruction loss was calculated only on the 32 EHR TS variables\. Pre\-trained SimMTM was adapted to downstream tasks using task specific heads and supervised fine\-tuning on the training labels\.

\(7\) King et al\.We implemented the pre\-training method byking2023as a multimodal contrastive learning baseline\. The model by King et al\. aligns EHR TS embeddings with clinical notes embeddings at the sequence level via a combination of cross\-model contrastive\[InfoNCE\]and masked reconstruction objectives\. In our implementation, we used the same pre\-training objective to align the embedding of the 32 EHR TS variables with clinical notes embeddings from BioClinical ModernBERT\[BioClinicalModernBert\]\. Since the model by King et al\. did not include additional projection layers for clinical notes embeddings, we kept the hidden dimension of the pre\-trained model the same as the LLM\. Similar to SimMTM, the channel\-wise missingness features were concatenated with the TS variables but were excluded from the reconstruction loss\. The pre\-trained models were then adapted to downstream tasks using task‑specific heads and supervised fine‑tuning on the training labels\.

### 5\.3Phenotyping

Similar to the MIMIC\-III\[MIMIC\-III\]benchmark byMIMICBenchmark, the phenotyping task aims to predict the presence of 25 acute care conditions during an ICU stay\. Labels were derived from ICD\-coded diagnoses using the HCUP Clinical Classifications Software \(CCS\) mappings\. For our experiments, we used the same 25 phenotypes as Harutyunyan et al\., but extracted labels via the revised HCUP CCSR mappings for ICD\-10 codes\. We note that several broad phenotypes in CCS, such as ’Acute cerebrovascular disease’, have been split into more specific subtypes in CSSR\. In this case, our label sets cover a wider range of ICD\-10 codes and therefore exhibit higher positive rates for some phenotypes\. The prevalence of specific phenotypes is available in the appendix\. Performance is summarised as macro\-averaged AUROC and AUPRC across phenotype groups: All, Acute, Chronic/Mixed and low prevalence \(Pr < 10%\)\.

Table 4:Results for 25 phenotype predictions on MIMIC\-IV with mean \(std\) over 10 runs, best results are highlighted in bold, second best results are underlined, significant results \(p<0\.05p<0\.05for paired t\-test between best and second best\) are noted with an asterisk\.ModelAllAcuteChronic/MixedPr <10%AUROCAUPRCAUROCAUPRCAUROCAUPRCAUROCAUPRCLR0\.776 \(0\.004\)0\.534 \(0\.007\)0\.782 \(0\.005\)0\.516 \(0\.011\)0\.768 \(0\.004\)0\.554 \(0\.006\)0\.702 \(0\.003\)0\.218 \(0\.005\)XGB0\.805 \(0\.004\)0\.578 \(0\.005\)0\.809 \(0\.004\)0\.549 \(0\.005\)0\.801 \(0\.005\)0\.609 \(0\.006\)0\.734 \(0\.004\)0\.242 \(0\.006\)LSTM0\.795 \(0\.011\)0\.542 \(0\.018\)0\.805 \(0\.009\)0\.520 \(0\.017\)0\.784 \(0\.011\)0\.565 \(0\.017\)0\.726 \(0\.012\)0\.209 \(0\.018\)GRU\-D0\.817 \(0\.004\)0\.580 \(0\.006\)0\.826 \(0\.005\)0\.558 \(0\.007\)0\.808 \(0\.005\)0\.603 \(0\.006\)0\.750 \(0\.005\)0\.248 \(0\.006\)T\-TS0\.796 \(0\.005\)0\.570 \(0\.006\)0\.802 \(0\.005\)0\.545 \(0\.007\)0\.790 \(0\.006\)0\.597 \(0\.006\)0\.729 \(0\.008\)0\.257 \(0\.009\)T\-EHR0\.803 \(0\.008\)0\.575 \(0\.011\)0\.809 \(0\.007\)0\.550 \(0\.009\)0\.797 \(0\.007\)0\.602 \(0\.011\)0\.737 \(0\.008\)0\.259 \(0\.010\)SimMTM0\.811 \(0\.007\)0\.558 \(0\.010\)0\.815 \(0\.007\)0\.527 \(0\.008\)0\.806 \(0\.007\)0\.592 \(0\.0010\)0\.730 \(0\.008\)0\.266 \(0\.012\)King et al\.0\.822 \(0\.005\)0\.580 \(0\.008\)0\.825 \(0\.005\)0\.550 \(0\.006\)0\.818 \(0\.006\)0\.612 \(0\.009\)0\.749 \(0\.007\)0\.250 \(0\.008\)LLM4EHR0\.844 \(0\.007\)∗0\.610 \(0\.009\)∗0\.848 \(0\.007\)∗0\.580 \(0\.008\)∗0\.840 \(0\.006\)∗0\.642 \(0\.010\)∗0\.771 \(0\.008\)∗0\.275 \(0\.009\)∗

![Refer to caption](https://arxiv.org/html/2607.15447v1/x4.png)\(a\)AUROC distribution by task
![Refer to caption](https://arxiv.org/html/2607.15447v1/x5.png)\(b\)AUPRC distribution by task

Figure 4:Box plot for mean AUROC and AUPRC across all 25 phenotypes\.As seen inTable[4](https://arxiv.org/html/2607.15447#S5.T4), LLM4EHR consistently outperforms all baselines in all phenotype groups\. LLM4EHR achieved a2\.7%2\.7\\%\(\+0\.022\+0\.022\) gain in AUROC and a more sizeable5\.1%5\.1\\%\(\+0\.030\+0\.030\) gain in AUPRC over the next strongest baseline across the full set \(All\)\. The same pattern can be observed across the Acute and Chronic/Mixed phenotypes, indicating stable performance over clinically distinct phenotype subgroups\. More importantly, LLM4EHR achieved significant gains in the low prevalence group \(Pr < 10%\) with a3\.38%3\.38\\%improvement in AUPRC, suggesting that LLM4EHR is more robust to rare phenotypes\.

However, as seen inFigure[4](https://arxiv.org/html/2607.15447#S5.F4), despite overall improvements, LLM4EHR’s performance still varies wildly across specific phenotypes\. This observation is expected, as some specific phenotypes like ’lower/upper respiratory’, ’complications of surgical/medical care’ or ’conduction disorders’ are harder to predict from general EHR data or charted observations and often require specific data such as chest X\-ray or ECG monitoring data to consistently predict\. Finally, we note that the presence of Chronic/Mixed phenotypes was harder to predict than Acute phenotypes for all models, consistent with prior observations by Harutyunyan et al\.

### 5\.4Rolling predictions

Rolling predictions include two longitudinal ICU outcome prediction tasks useful for patient management\. We used the same two tasks asMIMICBenchmarkbut defined the labels differently\. Rolling decompensation predictions aim to predict imminent physiological deterioration\. While Harutyunyan et al\. used in\-unit mortality as a proxy label, we instead use changes in SOFA score\[SOFA,SOFAdev\], which more directly reflect acute organ dysfunction for patients under intensive care\. Specifically, we computed SOFA score hourly using the worst measurements over the past 24 hour window\. Missing components are treated as 0 \(no related risk\) if no previous observations and front filled \(no change in risk\) if previous observations exist\. At timett, decompensation is marked by either patient death or a change over\+2\+2in SOFA score\. The change is SOFA score is calculate as the difference between SOFA atttand SOFA att−1t\-1\. At each prediction pointtpt\_\{p\}, we locate the nearest decompensation indicator att′\>tpt^\{\\prime\}\>t\_\{p\}within the same ICU stay and assign a positive label totpt\_\{p\}ift′−tp<=24t^\{\\prime\}\-t\_\{p\}<=24\. We report AUROC, AUPRC and F1 score for all models\.

Table 5:Summary of Decompensation and LoS labels across downsampled prediction pointsTrainingValidationTesting525k117k115k20\.67%20\.93%20\.2%3\.97%4\.74%4\.60%LoS Mean8\.488\.9010\.91LoS Max158\.43110\.80369\.75LoS Median4\.955\.225\.09LoS IQR7\.948\.548\.44LoS Skew4\.193\.188\.86

For the remaining LoS prediction, we avoided the task definition by Harutyunyan et al\. and formulated the task as time\-to\-event regression with an accelerated failure time objective\. More specifically, we assume that covariates \(EHR TS variables\) would hasten or prolong the patient’s discharge from ICU\. The aim is to predict days until discharge at at each prediction point and we imposed right censoring at 30 days\. Given that the utility of LoS prediction is mostly for ICU bed management, prediction over arbitrarily long horizon is not clinically useful due the presence of extreme outliers \(Table[5](https://arxiv.org/html/2607.15447#S5.T5)\)\. In our case, we consider 30 days to be a generous horizon as less than5%5\\%of the prediction points are censored\. Since the censor rate is low, we report MAE and R2 for uncensored predictions and C\-index for all predictions\. Due to the extreme number of prediction points \(Table[2](https://arxiv.org/html/2607.15447#S5.T2)\), we downsampled the rolling prediction to be one prediction every 5 hours\. We additionally limited the prediction period to be from 24 hours after ICU admission and 24 hours before ICU discharge to avoid trivial decompensation and LoS predictions\.

Table 6:Results for rolling decompensation and LoS predictions on MIMIC\-IV with mean \(std\) over 10 runs, best results are highlighted in bold, second best results are underlined, significant results \(p<0\.05p<0\.05for paired t\-test between best and second best\) are noted with an asterisk\.ModelsDecompensationLoSAUROCAUPRCF1MAER2C\-indexLR0\.635 \(0\.005\)0\.296 \(0\.007\)0\.370 \(0\.011\)2\.630 \(0\.006\)0\.058 \(0\.008\)0\.637 \(0\.005\)XGB0\.701 \(0\.006\)0\.407 \(0\.007\)0\.194 \(0\.009\)4\.052 \(0\.019\)0\.093 \(0\.011\)0\.652 \(0\.007\)LSTM0\.777 \(0\.008\)0\.561 \(0\.011\)0\.515 \(0\.018\)2\.574 \(0\.040\)0\.036 \(0\.027\)0\.608 \(0\.008\)GRU\-D0\.848 \(0\.007\)0\.669 \(0\.009\)0\.586 \(0\.012\)2\.352 \(0\.030\)0\.216 \(0\.021\)0\.653 \(0\.010\)T\-TS0\.864 \(0\.008\)0\.691 \(0\.009\)0\.623 \(0\.015\)2\.356 \(0\.029\)0\.148 \(0\.024\)0\.658 \(0\.009\)T\-EHR0\.808 \(0\.008\)0\.587 \(0\.011\)0\.500 \(0\.014\)2\.486 \(0\.033\)0\.132 \(0\.026\)0\.646 \(0\.009\)SimMTM0\.845 \(0\.008\)0\.658 \(0\.009\)0\.572 \(0\.015\)2\.380 \(0\.019\)0\.200 \(0\.024\)0\.659 \(0\.011\)King et al\.0\.768 \(0\.007\)0\.516 \(0\.008\)0\.378 \(0\.009\)2\.651 \(0\.027\)0\.145 \(0\.020\)0\.656 \(0\.008\)LLM4EHR0\.876 \(0\.007\)∗0\.723 \(0\.009\)∗0\.632 \(0\.012\)2\.343 \(0\.026\)0\.226 \(0\.024\)0\.675 \(0\.009\)

As shown inTable[6](https://arxiv.org/html/2607.15447#S5.T6), LLM4EHR achieves the best performance across both rolling prediction tasks, with significant gains in rolling decompensation\. Since our definition of decompensation assumes that physiological decline can occur at any point within the prediction window, the task requires models to process longitudinal observations to maintain stable performance across all metrics, as seen in models such as LR and XGB achieving substantially lower F1 scores compared with deep learning baselines\. In this case, LLM4EHR’s consistent performance gain over the next best model in all metrics suggests that pre\-training helped LLM4EHR to better capture temporal dependencies during downstream tuning, which is further supported by LLM4EHR outperforming the ablated baselines T\-TS and T\-EHR\.

In contrast, the performance differences between other baseline models and LLM4EHR in rolling LoS prediction are less significant\. LLM4EHR performed on par with GRU\-D and SimMTM in all metrics, with only marginal performance gains\. The overall R2 was poor across all models, consistent with prior work on ICU LoS regression, withlosmimicachieving an R2 of0\.240\.24on a carefully curated MIMIC\-IV cohort with outliers removed\. This outcome demonstrates that it is far easier to discriminate between short and prolonged stays than to predict the exact remaining time to discharge\. However, LLM4EHR noticeably outperformed the ablated baselines T\-TS and T\-EHR, suggesting that LLM4EHR benefited from semantic alignment pre\-training even in a complex downstream task with high label variability\.

### 5\.5Mortality prediction

Table 7:Results for full\-shot mortality prediction on MIMIC\-IV and Physionet2012 with mean \(std\) over 10 runs, best results are highlighted in bold, second best results are underlined, significant results \(p<0\.05p<0\.05for paired t\-test between best and second best\) are noted with an asterisk\.ModelMIMIC\-IVPhysionet2012AUROCAUPRCAUROCAUPRCLR0\.824 \(0\.004\)0\.354 \(0\.003\)0\.754 \(0\.003\)0\.322 \(0\.006\)XGB0\.841 \(0\.003\)0\.403 \(0\.005\)0\.827 \(0\.004\)0\.398 \(0\.005\)LSTM0\.835 \(0\.007\)0\.487 \(0\.006\)0\.804 \(0\.007\)0\.443 \(0\.008\)GRU\-D0\.850 \(0\.009\)0\.533 \(0\.012\)0\.843 \(0\.008\)0\.484 \(0\.010\)T\-TS0\.844 \(0\.008\)0\.511 \(0\.009\)0\.839 \(0\.008\)0\.452 \(0\.010\)T\-EHR0\.839 \(0\.011\)0\.509 \(0\.012\)N/AN/ASimMTM0\.852 \(0\.007\)0\.522 \(0\.009\)0\.841 \(0\.008\)0\.487 \(0\.009\)King et al\.0\.857 \(0\.007\)0\.534 \(0\.008\)0\.829 \(0\.007\)0\.484 \(0\.009\)LLM4EHR0\.870 \(0\.008\)∗0\.542 \(0\.009\)0\.856 \(0\.006\)∗0\.495 \(0\.008\)∗

![Refer to caption](https://arxiv.org/html/2607.15447v1/x6.png)Figure 5:AUROC \(upper\) and AUPRC \(lower\) for k\-shot mortality prediction on Physionet2012 with mean and95%95\\%confidence over 1000 bootstrap samples\.Mortality prediction aims to predict in\-hospital mortality using the first 48 hours of ICU observations\. We used the Physionet2012 challenge’s definition of in\-hospital mortality and assigned positive labels to patients who died before hospital discharge after spending at least 48 hours in the ICU\[Physionet2012\]\. We report AUROC and AUPRC for all models in a full‑shot setting on both MIMIC‑IV and Physionet2012\. Additionally, we used k\-shot adaptation to assess the generalisability and label efficiency of pre\-trained models\. In this case, models pre\-trained using MIMIC\-IV episodes \(SimMTM, the model by King et al\. and LLM4EHR\) were fine\-tuned usingkkpositive andkknegative samples from the Physionet2012 training set and tested on the full test set\. For eachkk, we report the mean AUROC and AUPRC with95%95\\%confidence intervals using 1000 bootstrap samples\.

The full\-shot mortality prediction results inTable[7](https://arxiv.org/html/2607.15447#S5.T7)show that models pre\-trained on MIMIC\-IV patient TS outperformed supervised models on MIMIC\-IV mortality prediction, with LLM4EHR achieving the best overall performance in both datasets\. LLM4EHR experienced a larger relative AUROC drop \(−1\.6%\-1\.6\\%\) than GRU\-D and SimMTM when adapted to Physionet2012, likely caused by LLM4EHR capturing MIMIC\-IV specific patterns during pre\-training\. This observation is additionally supported by the few\-shot results inFigure[5](https://arxiv.org/html/2607.15447#S5.F5), where LLM4EHR performed worse than SimMTM in extremely low\-k settingsk<5k<5but overtook both SimMTM and the model by King et al\. atk=10k=10and maintained a steady lead during moderate and high\-k settings\. The model by King et al\., similarly pre\-trained via cross\-modal alignment on MIMIC\-IV, experienced a larger relative performance drop than LLM4EHR \(−3\.4%\-3\.4\\%in AUROC\) when adapted to Physionet2012, indicating that our temporal alignment objective is less sensitive to data shift than the stationary alignment objective used by King et al\.

### 5\.6Further analysis of LLM4EHR

#### Ablation study of the pre\-training objective

We experimented with different pre\-training objectives to assess the impact of each component within LLM4EHR’s semantic alignment objective\. As shown inTable[8](https://arxiv.org/html/2607.15447#S5.T8), the largest performance gain comes from the combination ofℒr​e​c​o​n\\mathcal\{L\}\_\{recon\}withℒN​C​E\\mathcal\{L\}\_\{NCE\}, with the addition ofℒω\\mathcal\{L\}\_\{\\omega\}delivering marginal but consistent performance gains across all four downstream tasks\. The results demonstrate that temporal alignment is crucial for LLM4EHR, withℒω\\mathcal\{L\}\_\{\\omega\}having a regularising effect on the learned TS embeddings\. Additionally, models pre\-trained without the reconstruction objective are significantly worse at all tasks, suggesting that reconstruction is a necessary auxiliary task to contrastive alignment\.

Table 8:Ablation study on MIMIC\-IV downstream tasks using different pre\-training objectives, results are reported as mean \(std\) over 10 runs, best results are highlighted in bold, second best results are underlined, significant results \(p<0\.05p<0\.05for paired t\-test between best and second best\) are noted with an asterisk\.ObjectiveMortalityPhenotypingDecompensationLoSAUROCAUPRCAUROCAUPRCMAER2ℒr​e​c​o​n\\mathcal\{L\}\_\{recon\}0\.847 \(0\.006\)0\.520 \(0\.009\)0\.809 \(0\.008\)0\.554 \(0\.009\)0\.832 \(0\.009\)0\.662 \(0\.013\)2\.382 \(0\.021\)0\.209 \(0\.019\)ℒN​C​E\\mathcal\{L\}\_\{NCE\}0\.836 \(0\.010\)0\.499 \(0\.013\)0\.786 \(0\.009\)0\.540 \(0\.013\)0\.806 \(0\.008\)0\.580 \(0\.011\)2\.582 \(0\.031\)0\.099 \(0\.033\)ℒr​e​c​o​n\\mathcal\{L\}\_\{recon\}\+ℒN​C​E\\mathcal\{L\}\_\{NCE\}0\.863 \(0\.008\)0\.540 \(0\.010\)0\.839 \(0\.007\)0\.607 \(0\.009\)0\.868 \(0\.008\)0\.712 \(0\.009\)2\.340 \(0\.022\)0\.222 \(0\.023\)ℒa​l​i​g​n\\mathcal\{L\}\_\{align\}0\.870 \(0\.008\)∗0\.542 \(0\.009\)0\.844 \(0\.007\)∗0\.610 \(0\.009\)0\.876 \(0\.007\)∗0\.723 \(0\.009\)∗2\.343 \(0\.026\)0\.226 \(0\.024\)

Table 9:Ablation study on MIMIC\-IV downstream tasks using different LLMs for embedding EHR summaries, results are reported as mean \(std\) over 10 runs, best results are highlighted in bold, second best results are underlined, significant results \(p<0\.05p<0\.05for paired t\-test between best and second best\) are noted with an asterisk\.MortalityPhenotypeDecompensationLoSAUROCAUPRCAUROCAUPRCMAER2ModernBERT0\.850 \(0\.009\)0\.530 \(0\.009\)0\.822 \(0\.006\)0\.577 \(0\.009\)0\.869 \(0\.010\)0\.715 \(0\.009\)2\.348 \(0\.031\)0\.215 \(0\.026\)BioClinical ModernBERT0\.870 \(0\.008\)0\.542 \(0\.009\)0\.844 \(0\.007\)∗0\.610 \(0\.009\)∗0\.876 \(0\.007\)0\.723 \(0\.009\)∗2\.343 \(0\.026\)0\.226 \(0\.024\)Longformer0\.848 \(0\.008\)0\.529 \(0\.012\)0\.827 \(0\.010\)0\.582 \(0\.015\)0\.866 \(0\.008\)0\.708 \(0\.013\)2\.344 \(0\.022\)0\.208 \(0\.018\)Clinical\-Longformer0\.864 \(0\.007\)0\.541 \(0\.011\)0\.839 \(0\.011\)0\.583 \(0\.017\)0\.869 \(0\.012\)0\.707 \(0\.015\)2\.344 \(0\.023\)0\.210 \(0\.018\)RoBERTa0\.838 \(0\.011\)0\.392 \(0\.013\)0\.810 \(0\.013\)0\.552 \(0\.021\)0\.823 \(0\.006\)0\.660 \(0\.008\)2\.653 \(0\.020\)0\.060 \(0\.021\)BioMed\-RoBERTa0\.842 \(0\.009\)0\.395 \(0\.014\)0\.818 \(0\.009\)0\.572 \(0\.011\)0\.829 \(0\.009\)0\.658 \(0\.012\)2\.650 \(0\.021\)0\.072 \(0\.019\)

Table 10:Summary of LLMs for embedding EHR summariesSizeModernBERT8192150MBioClinical ModernBERT8192150MLongformer4096150MClinical\-Longformer4096150MRoBERTa512125MBioMed\-RoBERTa512125M

Table 11:The effects of the contrastive temperatureτ\\tauon the downstream task, since the optimalτ\\tauwas different for each task, we selectedτ=0\.02\\tau=0\.02for all evaluations as it achieved competitive results in all three tasks\.MortalityPhenotypeDecompensationLoSAUROCAUPRCAUROCAUPRCMAER2τ=0\.02\\tau=0\.020\.870 \(0\.008\)∗0\.542 \(0\.009\)0\.844 \(0\.007\)0\.610 \(0\.009\)0\.876 \(0\.007\)∗0\.723 \(0\.009\)∗2\.343 \(0\.026\)0\.226 \(0\.024\)τ=0\.08\\tau=0\.080\.862 \(0\.008\)0\.537 \(0\.010\)0\.846 \(0\.008\)0\.609 \(0\.011\)0\.856 \(0\.007\)0\.699 \(0\.010\)2\.351 \(0\.022\)0\.228 \(0\.020\)τ=0\.14\\tau=0\.140\.849 \(0\.007\)0\.529 \(0\.009\)0\.811 \(0\.007\)0\.592 \(0\.012\)0\.825 \(0\.005\)0\.657 \(0\.007\)2\.339 \(0\.017\)0\.208 \(0\.021\)τ=0\.20\\tau=0\.200\.852 \(0\.008\)0\.531 \(0\.011\)0\.789 \(0\.006\)0\.571 \(0\.008\)0\.805 \(0\.006\)0\.621 \(0\.009\)2\.427 \(0\.025\)0\.152 \(0\.022\)τ=0\.50\\tau=0\.500\.844 \(0\.012\)0\.522 \(0\.016\)0\.779 \(0\.007\)0\.554 \(0\.010\)0\.767 \(0\.010\)0\.576 \(0\.017\)2\.554 \(0\.031\)0\.112 \(0\.029\)

#### The choice of LLMs

Since our pre\-training framework is compatible with most LLM structures, we investigated the impacts of using different LLMs to embed EHR summaries\. We compared the results obtained from BioClinical ModernBERT\[BioClinicalModernBert\]against two other LLMs trained on medical texts: Clinical\-Longformer\[ClinicalLongformer\]and BioMed\-RoBERTa\[clinicalRoberta\]\. Additionally, we also include results for ModernBERT\[ModernBERT\], Longformer\[Longformer\]and RoBERTa\[RoBERTa\]\.Table[10](https://arxiv.org/html/2607.15447#S5.T10)summarises the context length and size of the LLMs\.Table[9](https://arxiv.org/html/2607.15447#S5.T9)shows that LLM4EHR pre\-trained with long context LLMs outperformed LLM4EHR pre\-trained with short context LLMs\. BioClinical ModernBERT and Clinical\-Longformer consistently yielded better performance in downstream evaluations than their general\-purpose counterparts, indicating that domain\-adapted LLMs are crucial for LLM4EHR pre\-training\. This observation, along with the ablation results inTable[8](https://arxiv.org/html/2607.15447#S5.T8), shows that our pre\-training objective facilitated meaning transfer of semantic knowledge from pre\-trained LLM to LLM4EHR embedded EHR TS features\.

#### The effects of the temperature𝝉\\boldsymbol\{\\tau\}

The temperatureτ\\tauis the scaling term for the similarity matrices in bothℒN​C​E\\mathcal\{L\}\_\{NCE\}andℒω\\mathcal\{L\}\_\{\\omega\}, and it is an important hyperparameter for LLM4EHR pre\-training\. Intuitively, a smallτ\\tausharpens the differences between similar and dissimilar samples, forcing the model to distinguish positive pairs from negatives aggressively\. As we formulatedℒω\\mathcal\{L\}\_\{\\omega\}to be a regularisation term for the main alignment lossℒN​C​E\\mathcal\{L\}\_\{NCE\}, we prioritised investigating the effects ofτ\\tauinℒN​C​E\\mathcal\{L\}\_\{NCE\}and keptτ=0\.14\\tau=0\.14forℒω\\mathcal\{L\}\_\{\\omega\}\. In practice, we found that usingτ<0\.08\\tau<0\.08forℒω\\mathcal\{L\}\_\{\\omega\}causesℒa​l​i​g​n\\mathcal\{L\}\_\{align\}to be dominated byℒω\\mathcal\{L\}\_\{\\omega\}, leading to unstable pre\-training\.Table[11](https://arxiv.org/html/2607.15447#S5.T11)shows that a smallerτ\\tauforℒN​C​E\\mathcal\{L\}\_\{NCE\}is overall better for pre\-training, with a noticeable drop in performance betweenτ=0\.08\\tau=0\.08andτ=0\.14\\tau=0\.14\.

## 6Discussion

We presented LLM4EHR as a multimodal pre\-training framework for clinical foundation models\. The results inSection[5](https://arxiv.org/html/2607.15447#S5)show that by jointly modelling EHR TS with EHR events, LLM4EHR achieves state\-of\-the\-art performance on clinically significant downstream tasks, including mortality prediction, patient phenotyping, ICU decompensation, and remaining LoS prediction\. The cross\-dataset mortality prediction results inSection[5\.5](https://arxiv.org/html/2607.15447#S5.SS5)show that pre\-trained LLM4EHR can efficiently generalise to new ICU patient cohorts via few\-shot learning\. The ablation studies inSection[5\.6](https://arxiv.org/html/2607.15447#S5.SS6)show that ourω\\omega\-regularised alignment objective improves the consistency of the learned EHR TS embedding and that using domain adapted LLM for embedding EHR event summaries during pre\-training was crucial for LLM4EHR\.

The major limitation of LLM4EHR is scalability to larger language models, which have considerable memory requirements to train\. Future research could consider more efficient contrastive learning structures such ashe2020momentum, which uses a memory queue to store past samples for calculating the contrastive objective\. Additionally, we focused on intensive care episodes during pre\-training; future studies should consider modelling extended patient trajectories spanning multiple stays, which would extend LLM4EHR to additional downstream tasks useful for long\-term care monitoring\. Lastly, we evaluated the performance of LLM4EHR using learned TS embeddings, given our focus on predicting ICU patient outcomes\. Future research should investigate the applicability of LLM4EHR with schema alignment\-based pre\-training\[Sun2024\]for natural language inference, such as question answering with EHR patient observations\.

\\printcredits

## Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper\.

## Acknowledgments

This work was supported by the Great Ormond Street Hospital \(GOSH\) Charity \(grant number 21PP30\)\. PB would like to acknowledge funding via theăRoyal Academy of Engineering and Great Ormond Street Hospital, the UK Dementia Research Institute \(award number UK DRI\-7002\) through UK DRI Ltd, principally funded by the Medical Research Council\.

## Data availability

The MIMIC\-IV and Physionet Challenge 2012 are publicly available\. All code used for curating the dataset and running our experiments can be found at:[https://github\.com/CrankyWilliam/LLM4EHR](https://github.com/CrankyWilliam/LLM4EHR)\.

## References

## Appendix AAppendix

### A\.1Curating the datasets

Here we describe the process for curating the MIMIC\-IV dataset, corresponding to the Python implementation provided in our code repository\. Since we did not further filter or process the Physionet Challenge 2012 dataset, we refer to the original implementation byPhysionet2012for details regarding dataset curation and cohort selection\.

#### A\.1\.1MIMIC\-IV cohort

We curated a cohort of patients with at least one recorded ICU stay\. Different from the cohort selection process byMIMICBenchmark, we included patients with multiple ICU stays per hospital admission but kept only the first ICU stay per hospital admission\.MIMICBenchmarkargues that including hospital admissions with multiple ICU stays makes the mortality and phenotyping labels ambiguous\. While doing so would be valid for predicting the onset of adverse events in a unit, such as predicting the onset of cardiac arrest in the ICU, we found no reason to remove hospital admissions with multiple ICU stays, as both phenotype and mortality labels are defined for unique hospital admissions rather than specific stays\. The labels for phenotype and mortality would still be valid for a given ICU stay if a patient was discharged then subsequently readmitted to ICU provided both stays occurs within the same hospital admission period\. Therefore, we elect to retain the first ICU stay for each hospital admission\. For direct transfers between ICU wards, when the discharge time of the first ICU ward stay overlaps with the admission time of the second ICU ward stay, we merge the two stays into a single long stay\. The process of curating MIMIC\-IV cohort is shown inFigure[6](https://arxiv.org/html/2607.15447#A1.F6)\.

![Refer to caption](https://arxiv.org/html/2607.15447v1/x7.png)Figure 6:Flowchart for curating the MIMIC\-IV ICU stay patient cohort, short stays<48<48hours are removed to allow adequate patient observations before prediction\.
#### A\.1\.2MIMIC\-IV Mortality labels

We used the Physionet2012 Challenge’s definition of in\-hospital mortality and assigned positive labels to patients who died before hospital discharge after spending at least 48 hours in the ICU\. Since we only included stays longer than 48 hours, all curated MIMIC\-IV episodes are eligible for mortality prediction\. Here, we specify that the patient mortality label is associated with unique hospital stays rather than unique ICU stays, as the hospital discharge time was used to determine in\-hospital patient mortality\. In this case, we included only the first ICU stay for each hospital admission to avoid duplicating mortality labels\.

#### A\.1\.3MIMIC\-IV Phenotype labels

Similar to the MIMIC\-III benchmark byMIMICBenchmark,c the phenotyping task aims to predict the presence of 25 acute care conditions during an ICU stay\. Labels were derived from ICD\-coded diagnoses using the HCUP Clinical Classifications Software \(CCS\) mappings\. In their original implementation Harutyunyan et al\. noted that phenotype labels are associated with hospital stays rather than ICU stays and are predicted retrospectively using an entire ICU episode\. For our experiments, we used the same 25 phenotypes as Harutyunyan et al\., but extracted labels via the revised HCUP CCSR mappings for ICD\-10 codes\. We note that several broad phenotypes in CCS, such as ’Acute cerebrovascular disease’, have been split into more specific subtypes in CSSR\.Table[12](https://arxiv.org/html/2607.15447#A1.T12)summarises the 25 phenotype labels in MIMIC\-IV\.

Table 12:Summary of 25 phenotype labels in MIMIC\-IV\.PhenotypeTypePrevalenceAcute cerebrovascular diseaseAcute0\.159Acute myocardial infarctionAcute0\.110Acute renal failureAcute0\.375Cardiac dysrhythmiasChronic/Mixed0\.375Chronic obstructive pulmonary diseaseChronic/Mixed0\.175Complications of surgical/medical careAcute0\.099Conduction disordersChronic/Mixed0\.097Congestive heart failure; nonhypertensiveChronic/Mixed0\.312Coronary atherosclerosis and relatedChronic/Mixed0\.334Diabetes mellitus with complicationsChronic/Mixed0\.151Diabetes mellitus without complicationsChronic/Mixed0\.173Disorder of lipid metabolismChronic/Mixed0\.433Essential hypertensionChronic/Mixed0\.404Fluid and electrolyte disorderAcute0\.498Gastrointestinal hemorrhageAcute0\.092Hypertension with complicationsChronic/Mixed0\.267Other liver diseaseChronic/Mixed0\.122Other lower respiratory diseaseAcute0\.093Other upper respiratory diseaseAcute0\.028Pleurisy; pneumothorax; pulmonary collapseAcute0\.169PheumoniaAcute0\.245Respiratory failure; insufficiency; arrestAcute0\.386Septicemia \(except in labour\)Acute0\.219ShockAcute0\.261

#### A\.1\.4MIMIC\-IV decompensation labels

Rolling decompensation predictions aim to predict imminent physiological deterioration\. WhileMIMICBenchmarkused in\-unit mortality as a proxy label, we instead use changes in SOFA score, which more directly reflect acute organ dysfunction\. Specifically, we computed SOFA score hourly using the worst measurements over the past 24 hour window\. Missing components are treated as 0 \(no related risk\) if no previous observations and front filled \(no change in risk\) if previous observations exist\. At timett, decompensation is marked by either patient death or a change over\+2\+2in SOFA score\. The change is SOFA score is calculate as the difference between SOFA atttand SOFA att−1t\-1\. At each prediction pointtpt\_\{p\}, we locate the nearest decompensation indicator att′\>tpt^\{\\prime\}\>t\_\{p\}within the same ICU stay and assign a positive label totpt\_\{p\}ift′−tp<=24t^\{\\prime\}\-t\_\{p\}<=24\. SOFA score is calculated using the criteria inTable[13](https://arxiv.org/html/2607.15447#A1.T13)\.

Table 13:Scoring criteria for SOFA score components summarised fromSOFA, with additional Norepinephrine equivalent dosages fromSOFAdevRespiratory systemNervous systemCardiovascular SystemLiver systemCoagulationKidneysSOFA scorePaO2/FiO2 \(mmHg\)Glasgow coma scoreMAP/VasopressorBilirubin \(mg/dl\) \[umo/L\]Platelets x1000/mlCreatinine \(mg/dl\);Urine output \(ml/day\)0\>40015MAP \>70 mmHg<1\.2 \(<20\)\>150<1\.21<40013\-14MAP <70 mmHg1\.2 \- 1\.9 \[20 \- 32\]<1501\.2 \- 1\.92<30010\-12Dopamine <= 5 ug/kg/minDobutamine \(any dosage\)2\.0 \- 5\.9 \[33 \- 101\]<1002\.0 \- 3\.43<200 with respiratorysupport6\-9Norepinephrine <= 0\.1 ug/kg/minEpinephrine <= 0\.1 ug/kg/minDopamine \>5 ug/kg/min6\.0 \- 11\.9 \[102 \- 204\]<503\.5 \- 4\.9Urine output <5004<100 with respiratorysupport<6Norepinephrine \>0\.1 ug/kg/minEpinephrine \>0\.1 ug/kg/minDopamine \>15 ug/kg/min\>12\.0 \[\>204\]<20

#### A\.1\.5MIMIC\-IV LoS labels

For the remaining LoS prediction, we avoided the task definition byMIMICBenchmarkand formulated the task as time\-to\-event regression with an accelerated failure time objective\. More specifically, we assume that covariates \(EHR TS variables\) would hasten or prolong the patient’s discharge from ICU\. The aim is to predict days until discharge at at each prediction point and we imposed right censoring at 30 days\. Given that the utility of LoS prediction is mostly for ICU bed management, prediction over arbitrarily long horizon is not clinically useful due the presence of extreme outliers \(Table[5](https://arxiv.org/html/2607.15447#S5.T5)\)\. In our case, we consider 30 days to be a generous horizon as less than5%5\\%of the prediction points are censored\.

Algorithm 1Masked negative log\-likelihood for Log\-normal AFT0:

μ,log⁡σ,y,event,mask∈ℝB×T\\mu,\\log\\sigma,y,\\mathrm\{event\},\\mathrm\{mask\}\\in\\mathbb\{R\}^\{B\\times T\},

ε\>0\\varepsilon\>0
1:

σ←exp⁡\(log⁡σ\)\\sigma\\leftarrow\\exp\(\\log\\sigma\)
2:

y←max⁡\(y,ε\)y\\leftarrow\\max\(y,\\varepsilon\)
3:

log⁡y←ln⁡\(y\)\\log y\\leftarrow\\ln\(y\)
4:

z←\(log⁡y−μ\)/σz\\leftarrow\(\\log y\-\\mu\)/\\sigma
5:

log⁡f←−log⁡\(y⋅σ​2​π\)−12​z2\\log f\\leftarrow\-\\log\\big\(y\\cdot\\sigma\\sqrt\{2\\pi\}\\big\)\-\\tfrac\{1\}\{2\}z^\{2\}
6:

log⁡S←log⁡\(1−Φ​\(z\)\+ε\)\\log S\\leftarrow\\log\\big\(1\-\\Phi\(z\)\+\\varepsilon\\big\)
7:

nll←−\(event⋅log⁡f\+\(1−event\)⋅log⁡S\)\\mathrm\{nll\}\\leftarrow\-\\big\(\\mathrm\{event\}\\cdot\\log f\+\(1\-\\mathrm\{event\}\)\\cdot\\log S\\big\)
8:

nll←nll⊙mask\\mathrm\{nll\}\\leftarrow\\mathrm\{nll\}\\odot\\mathrm\{mask\}
9:

loss←∑nll∑mask\\mathrm\{loss\}\\leftarrow\\dfrac\{\\sum\\mathrm\{nll\}\}\{\\sum\\mathrm\{mask\}\}
10:

11:return

loss\\mathrm\{loss\}

The log\-normal AFT objective for the linear and XGB baseline models were implemented bylifelinesandxgboostrespectively\. For remaining LoS prediction, LLM4EHR and other deep learning baselines are fitted with a accelerated failure prediction head that predicts log\-scale meanμ\\muand standard deviationlog⁡σ\\log\\sigmaof the expected time to discharge\. Given the observed remaining LoSyy, binary event indicatore​v​e​n​tevent, validity maskmmand batch sizeBB, the negative log\-likelihood is calculated viaAlgorithm[1](https://arxiv.org/html/2607.15447#alg1), which is derived from the implementation byweibullrnn\. The LoS prediction evaluation metrics for all baseline models were implemented bylifelines\.

#### A\.1\.6Summary EHR TS features

Table 14:Summary of EHR TS features for MIMIC\-IV \(A\) and Physionet2012 \(B\), Count refers to the number of measurements for each variable in the datasetVariableUnitCount \(A\)Median \(A\)Count \(B\)Median \(B\)heart\_ratebpm554607285\.034599886\.0sbpmmHg5195562119\.0207385118\.0dbpmmHg519488662\.020736258\.0mbpmmHg521310478\.020626778\.0sbp\_nimmHg3254534119\.0162383118\.0dbp\_nimmHg325395265\.016216757\.0mbp\_nimmHg326686579\.016008976\.0resp\_ratebreaths/min545208220\.09314219\.0temperature°C169816536\.9414127337\.1sao2%532680497\.01523397\.0glucosemg/dL1216399136\.026115127\.0albuming/dL691853\.048022\.9bicarbonatemmol/L45046724\.02716923\.0bunmg/dL44835624\.02775620\.0creatininemg/dL4490721\.1278951\.0sodiummEq/L477454139\.027227139\.0potassiummEq/L4758754\.1290414\.1phpH Unit3925767\.38463707\.38lactatemmol/L2116441\.8160512\.0fio2%28344750\.06077050\.0pao2mmHg39274098\.044346120\.0paco2mmHg39238241\.04440439\.0hematocrit%47066528\.93651230\.3altIU/L12948137\.0646943\.0alpIU/L12863599\.0629182\.0astIU/L13108452\.0647362\.0bilirubin\_totalmg/dL1294010\.965110\.9platelet103/µL403151179\.028290172\.0wbc103/µL39660410\.92589111\.4troponin\_tug/L408400\.1651010\.30gcsscore \(3–15\)159352015\.012311213\.0urineoutputmL2955401100\.026584870\.0Table[14](https://arxiv.org/html/2607.15447#A1.T14)summarises EHR TS features for MIMIC\-IV and Physionet2012, with count of measurements and median values for each variable\. The density of measurements is substantially higher in MIMIC\-IV due to our MIMIC\-IV dataset includes more patients and longer patient trajectories than Physionet2012\. The EHR TS features for MIMIC\-IV and Physionet2012 were synchronised to ICU stay hours following the curation process bymimicrepo\. Prior to analysis, physically implausible outliers were removed on a variable\-by\-variable basis\. We imputed missing values by front filling within each episode, and filled any remaining gaps with the population mean\. Continuous variables were z\-standardised using observed mean and standard deviation before imputation\.