Pretraining EHR Foundation Models with Patient-Aware Sampling

# Pretraining EHR Foundation Models with Patient-Aware Sampling
Source: [https://arxiv.org/html/2607.22114](https://arxiv.org/html/2607.22114)
###### Abstract

Autoregressive foundation models for electronic health records \(EHRs\) typically inherit pretraining methods from language modeling, where patient trajectories are concatenated into a single token stream and windows are sampled from that stream\. In EHR data, this choice is consequential: windows may mix multiple patients, and patients with longer records contribute more optimization updates, potentially introducing bias\. We propose Patient Sampling, a pretraining sequence\-construction method that allows us to control how training signal is distributed across patients\. We compare this method to the standard approach, which we refer to as Global Stream\. We show that stochastic Patient Sampling with controllable weighting improves performance on real\-world EHR data\. Across downstream clinical tasks on MIMIC\-IV v2\.2 and v3\.1, Patient Sampling improves Macro AUROC and AUPRC over the Global Stream baseline\. These results identify training and validation sequence construction as important and underexplored design choices for autoregressive EHR foundation models\.

Machine Learning, Pretraining, Autoregressive, Health, Medical, ICML

## 1Introduction

Autoregressive foundation models for electronic health records \(EHRs\) represent each patient history as a variable\-length sequence of discrete tokens and train with a next\-token prediction objective\. Recent work has shown that these models can support zero\-shot clinical prediction by simulating future trajectories from observed patient histories\(Rencet al\.,[2024](https://arxiv.org/html/2607.22114#bib.bib9); Kraljevicet al\.,[2024](https://arxiv.org/html/2607.22114#bib.bib36)\)\.

Current EHR pretraining methods largely inherit sequence construction strategies from language modeling\. In the standard setup, patient trajectories are concatenated into a global token stream and fixed\-length training windows are sampled from that stream\. In EHR data, this is more than an efficiency choice: a window may contain events from multiple patients, and patients with longer records contribute more optimization updates during training\. This is especially relevant in our setting, where patient trajectory lengths are highly unequal, forming a log\-normal distribution as shown in Figure[3](https://arxiv.org/html/2607.22114#S2.F3)\.

![Refer to caption](https://arxiv.org/html/2607.22114v1/x1.png)Figure 1:Toy example of an EHR dataset represented as variable\-length token sequencesDiD\_\{i\}\. Colored boxes denote EHR tokens, and black\-bordered boxes indicate end\-of\-sequence tokens\.In this work, we study alternative methods for pretraining sequence construction\. We identify three different approaches, which we refer to as Global Stream, Patient Chunks and Patient Sampling\.

Keeping tokenization and model architecture fixed, we compare the standard Global Stream method with patient\-aware alternatives that preserve patient boundaries and alter how training signal is distributed across patients\. Across MIMIC\-IV v2\.2 and v3\.1\(Johnsonet al\.,[2023b](https://arxiv.org/html/2607.22114#bib.bib39),[2024](https://arxiv.org/html/2607.22114#bib.bib10),[a](https://arxiv.org/html/2607.22114#bib.bib11)\), we find that Patient Sampling achieves stronger overall downstream performance than the Global Stream baseline, improving both Macro AUROC and Macro AUPRC\. Our findings highlight the impact of pretraining sequence construction methods, and to our knowledge this is the first work to explore this design choice in the context of autoregressive health foundation models\.

## 2Background

![Refer to caption](https://arxiv.org/html/2607.22114v1/x2.png)Figure 2:Comparison of the dataset constructions considered in this work\. On the left*Global Stream*forms training windows from a global concatenated token stream\.*Patient Chunks*constructs deterministic within\-patient windows\.*Patient Sampling*first samples a patient and then samples a window from that patient alone\. Colors denote different patients, and gray boxes indicate padding\.Autoregressive modeling has recently emerged as a promising paradigm for EHR data, where patient history is represented as a sequence of tokenized events\. Figure[1](https://arxiv.org/html/2607.22114#S1.F1)illustrates this setup with a toy tokenized EHR dataset\.

### EHR Foundation Models

Recent work has applied autoregressive or generative pretraining to longitudinal EHR data\. ETHOS\(Rencet al\.,[2024](https://arxiv.org/html/2607.22114#bib.bib9)\)and its follow\-up ETHOS\-ARES\(Rencet al\.,[2025](https://arxiv.org/html/2607.22114#bib.bib38)\)convert EHR data into tokenized patient timelines, train GPT\-2\(Radfordet al\.,[2019](https://arxiv.org/html/2607.22114#bib.bib7)\)style models with next\-token prediction, and evaluate them by rolling out future events from a clinical prediction point to obtain zero\-shot predictions for downstream tasks\. Foresight\(Kraljevicet al\.,[2024](https://arxiv.org/html/2607.22114#bib.bib36)\)models patient timelines with a generative pretrained transformer that combines structured EHR data and free text to predict future clinical concepts and outcomes\. CoMET\(Waxleret al\.,[2025](https://arxiv.org/html/2607.22114#bib.bib37)\)demonstrates that autoregressive medical event pretraining with a next\-token prediction objective scales to larger models and datasets with predictable scaling laws\. Together, these works establish autoregressive pretraining over patient timelines as a viable approach to EHR foundation modeling, but do not explicitly study pretraining sequence construction as a primary design variable\.

![Refer to caption](https://arxiv.org/html/2607.22114v1/x3.png)Figure 3:Distribution of patient token sequence lengths in the MIMIC\-IV v2\.2 training set, shown on a log\-scaled x\-axis\.
### Sequence Composition

The impact of sequence composition has previously been explored in language model pretraining\.Zhaoet al\.\([2024](https://arxiv.org/html/2607.22114#bib.bib2)\)show that concatenating unrelated documents into a fixed\-length sequence can introduce distracting cross\-document context, harming language modeling and downstream performance\. They demonstrate that both intra\-document masking and retrieval\-based packing of related documents improve performance\. Similarly, In\-Context Pretraining\(Shiet al\.,[2024](https://arxiv.org/html/2607.22114#bib.bib40)\)shows that re\-ordering documents so that each context window contains semantically related material improves tasks requiring stronger contextual reasoning across document boundaries\. These findings motivate studying sequence composition in EHR modelling, where patient trajectories are highly unequal in length and standard sequence construction may further amplify this imbalance\.

Dataset MethodMIMIC\-IV v2\.2 \+ ED v2\.2MIMIC\-IV v3\.1 \+ ED v2\.2TrainValidationα\\alphaICU MortalityICU ReadmissionICU MortalityICU ReadmissionMacroAUROCAUPRCAUROCAUPRCAUROCAUPRCAUROCAUPRCAUROCAUPRCGlobal StreamGlobal Stream–0\.8780\.4340\.7250\.3830\.8430\.4280\.7030\.3800\.7870\.406Global StreamPatient Chunks–0\.8660\.4280\.7170\.3890\.8520\.4390\.7120\.3820\.7860\.410Patient ChunksPatient Chunks–0\.8650\.3570\.6640\.3130\.8320\.3270\.6620\.3340\.7560\.332Patient SamplingPatient Chunks00\.8850\.4320\.7260\.3920\.8510\.4370\.7010\.3720\.7910\.4080\.20\.8480\.3500\.7210\.3820\.8390\.4020\.6900\.3650\.7740\.3750\.40\.8820\.4430\.7280\.3990\.8340\.4050\.6960\.3720\.7850\.4050\.60\.8840\.4560\.7300\.4050\.8500\.4460\.7060\.3810\.7920\.4220\.80\.8450\.3780\.7250\.3970\.8210\.3790\.7070\.3760\.7750\.38310\.8630\.4160\.7330\.4070\.8480\.4350\.7190\.3990\.7910\.414

Table 1:Comparison of pretraining data constructions on ICU Mortality and ICU Readmission across MIMIC\-IV v2\.2 \+ ED v2\.2 and MIMIC\-IV v3\.1 \+ ED v2\.2\. Results are reported as AUROC and AUPRC\. The final macro column denotes the mean across all four reported task settings, and the best value in each metric column is shown in bold\.

## 3Method

We consider a dataset ofNNpatients𝒟=\{x\(i\)\}i=1N\\mathcal\{D\}=\\\{x^\{\(i\)\}\\\}\_\{i=1\}^\{N\}, where patientiiis represented by a sequence of EHR tokens

x\(i\)=\(x1\(i\),…,xLi\(i\)\),x^\{\(i\)\}=\\left\(x^\{\(i\)\}\_\{1\},\\dots,x^\{\(i\)\}\_\{L\_\{i\}\}\\right\),andLiL\_\{i\}is the sequence length of patientii\. Following prior work on autoregressive EHR modeling, we train a decoder\-only transformer with a next\-token prediction objective on fixed\-length token windows of lengthSS\.

We compare several strategies for constructing*windows*w=\(w1,…,wS\)w=\(w\_\{1\},\\dots,w\_\{S\}\)for training and validation\. A window is a sequence of tokens that forms a single datapoint\.

### Global Stream\.

Our baseline follows the standard global stream construction used in autoregressive pretraining\. We concatenate all patient sequences into a single stream

z=x\(1\)∥x\(2\)∥⋯∥x\(N\)z=x^\{\(1\)\}\\mathbin\{\\\|\}x^\{\(2\)\}\\mathbin\{\\\|\}\\cdots\\mathbin\{\\\|\}x^\{\(N\)\}with total lengthT=∑i=1NLiT=\\sum\_\{i=1\}^\{N\}L\_\{i\}\. Given striderr, we define the set of valid start indices

𝒯GS=\{1,1\+r,1\+2​r,…,tmax\},tmax≤T−S\+1\.\\mathcal\{T\}\_\{\\mathrm\{GS\}\}=\\\{1,\\,1\+r,\\,1\+2r,\\,\\dots,\\,t\_\{\\max\}\\\},\\qquad t\_\{\\max\}\\leq T\-S\+1\.Each window is thenw\(t\)=\(zt,…,zt\+S−1\),t∈𝒯GSw^\{\(t\)\}=\(z\_\{t\},\\dots,z\_\{t\+S\-1\}\),\\qquad t\\in\\mathcal\{T\}\_\{\\mathrm\{GS\}\}\. This construction is simple and efficient, but a single window may contain tokens from multiple patients\.

### Patient Chunks\.

To preserve patient boundaries, we propose constructing windows independently within each patient sequence\. For patientii, we define a deterministic set of chunk start indices

ℬi=\{1,1\+S,1\+2​S,…\}∩\{1,…,Li\},\\mathcal\{B\}\_\{i\}=\\\{1,\\,1\+S,\\,1\+2S,\\,\\dots\\\}\\cap\\\{1,\\dots,L\_\{i\}\\\},We then add a final chunk that is right\-aligned to the end of the sequence,

bi,end=max⁡\(1,Li−S\+1\),b\_\{i,\\mathrm\{end\}\}=\\max\(1,\\,L\_\{i\}\-S\+1\),and add this to the start index setℬi←ℬi∪\{bi,end\}\\mathcal\{B\}\_\{i\}\\leftarrow\\mathcal\{B\}\_\{i\}\\cup\\\{b\_\{i,\\mathrm\{end\}\}\\\}\. Each window is then

w\(i,b\)=\(xb\(i\),…,xmin⁡\(Li,b\+S−1\)\(i\)\),b∈ℬi\.w^\{\(i,b\)\}=\\left\(x^\{\(i\)\}\_\{b\},\\dots,x^\{\(i\)\}\_\{\\min\(L\_\{i\},\\,b\+S\-1\)\}\\right\),\\qquad b\\in\\mathcal\{B\}\_\{i\}\.Windows shorter thanSSare right\-padded to lengthSS, and padded target positions are excluded from the loss and masked out\. Unlike Global Stream, Patient Chunks always produces windows containing tokens from a single patient\.

### Patient\-Aware Sampling\.

We next introduce a stochastic construction that decouples*patient*and*window*selection, to control the distribution of loss signal during training\. This construction is only used for training due to its non\-deterministic nature\.

For each patientii, let𝒲\(i\)\\mathcal\{W\}^\{\(i\)\}denote the set of valid start indices for constructing a window\. We first sample a patient according to

pα​\(i\)=\|𝒲\(i\)\|α∑j=1N\|𝒲\(j\)\|α,p\_\{\\alpha\}\(i\)=\\frac\{\|\\mathcal\{W\}^\{\(i\)\}\|^\{\\alpha\}\}\{\\sum\_\{j=1\}^\{N\}\|\\mathcal\{W\}^\{\(j\)\}\|^\{\\alpha\}\},whereα∈\[0,1\]\\alpha\\in\[0,1\]controls the weighting of patients during training\. Whenα=0\\alpha=0, patients are sampled uniformly; whenα=1\\alpha=1, patients are sampled proportionally to the number of valid windows they contain, i\.e\. a patient with twice as many possible token windows as another patient will be sampled twice as often\.

Given a sampled patientii, we then sample a start index uniformly from

𝒲\(i\)=\{1−S,2−S,…,Li−2\}\.\\mathcal\{W\}^\{\(i\)\}=\\\{1\-S,\\,2\-S,\\,\\dots,\\,L\_\{i\}\-2\\\}\.We clip the proposed start index to the first valid tokens~=max⁡\(1,s\)\\tilde\{s\}=\\max\(1,s\)\. This choice of start indices allows tokens within a patient trajectory to appear with a broader range of left\-context lengths, rather than systematically under\-sampling early positions111Due to the need for an input and a target, the first and last tokens are still slightly under\-sampled compared to interior tokens, though we believe this effect is minimal\.\. Conditioned on\(i,s~\)\(i,\\tilde\{s\}\), the resulting window isxs~:s~\+S−1\(i\)x^\{\(i\)\}\_\{\\tilde\{s\}:\\tilde\{s\}\+S\-1\}, right\-padded to lengthSSif necessary\.

Global Stream may cross patient boundaries and introduce long\-trajectory bias, Patient Chunks preserves patient boundaries but retains this imbalance, and Patient Sampling preserves boundaries while allowing the training distribution over patients to vary smoothly between patient\-uniform and length\-weighted sampling\. Each method is visualized in Figure[2](https://arxiv.org/html/2607.22114#S2.F2)\.

## 4Experiments and Results

Table 2:Comparison of Global Stream and Patient Sampling \(α=0\.6\\alpha=0\.6\) across ICU Mortality, ICU Readmission, ICU Admission, and Hospital Mortality for MIMIC\-IV v2\.2 \+ ED v2\.2 and MIMIC\-IV v3\.1 \+ ED v2\.2\. Results are reported as AUROC and AUPRC, and the Macro columns report the mean across the four tasks within each dataset version\. Best value in each column is shown in bold\.### Experimental setup\.

We evaluate the methods introduced in Section[3](https://arxiv.org/html/2607.22114#S3)using two tokenized EHR datasets derived from MIMIC\-IV: version 2\.2\(Johnsonet al\.,[2023b](https://arxiv.org/html/2607.22114#bib.bib39)\)and version 3\.1\(Johnsonet al\.,[2024](https://arxiv.org/html/2607.22114#bib.bib10)\), both accessed via PhysioNet\(Goldbergeret al\.,[2000](https://arxiv.org/html/2607.22114#bib.bib12)\)\. In both settings, we also include the Emergency Department module, MIMIC\-IV\-ED version 2\.2\(Johnsonet al\.,[2023a](https://arxiv.org/html/2607.22114#bib.bib11)\)\. Each dataset is split by patient into 80% train, 10% validation, and 10% test sets\.

We train a 6\-layer GPT\-2 model\(Radfordet al\.,[2019](https://arxiv.org/html/2607.22114#bib.bib7)\)on a single NVIDIA GH200 with sequence lengthS=2048S=2048\. Validation loss is evaluated every 10K training steps, and the checkpoint with the lowest validation loss is used for downstream evaluation\. Because Patient Chunks is deterministic and both patient\-aware methods operate on single\-patient windows, we use Patient Chunks as the validation dataset for Patient Chunks and Patient Sampling training datasets\. For Global Stream, we report results with both Global Stream and Patient Chunks as validation datasets\. We evaluate downstream performance using the rollout framework ofRencet al\.\([2025](https://arxiv.org/html/2607.22114#bib.bib38)\)on the following clinical tasks: ICU Mortality, ICU Readmission, ICU Admission, and Hospital Mortality\. See Appendix[A](https://arxiv.org/html/2607.22114#A1)for full training and evaluation details\.

![Refer to caption](https://arxiv.org/html/2607.22114v1/x4.png)Figure 4:Δ\\Deltapositive score on ICU Mortality \(MIMIC\-IV v2\.2\) for Patient Sampling models with differentα\\alphavalues, relative to a Global Stream baseline\. Positive score is the fraction of simulations that terminate with the correct end token, delta positive score is measured relative to the Global Stream baseline\. Datapoints are grouped by the number of tokens of the ground truth sequence\.
### Main results\.

We first evaluate each method on the ICU Mortality and ICU Readmission tasks, sweeping Patient Sampling overα\\alphavalues in increments of 0\.2\. Results are shown in Table[1](https://arxiv.org/html/2607.22114#S2.T1)\. We find that Patient Chunks does not improve over the Global Stream baseline, suggesting that simply preserving patient boundaries is not sufficient\. Patient Sampling withα=0\.6\\alpha=0\.6yields the strongest macro performance across both tasks in both datasets, and we therefore select this model for broader evaluation\.

Table[2](https://arxiv.org/html/2607.22114#S4.T2)shows results from further evaluation on the Hospital Mortality and ICU Admission tasks\. We find performance improvements on the Hospital Mortality task across both datasets\. Improvements on ICU Admission are marginal, with no gain on MIMIC\-IV v2\.2 and small gains on MIMIC\-IV v3\.1\. Across all four benchmarks and both datasets, Patient Sampling outperforms Global Stream on all four macro metrics\.

### Effect ofα\\alpha\.

Figure[4](https://arxiv.org/html/2607.22114#S4.F4)shows how the behavior of Patient Sampling varies with ground\-truth token length on the ICU Mortality benchmark for MIMIC\-IV v2\.2\. Smaller values ofα\\alphaproduce the strongest positive shifts relative to the Global Stream baseline, withα=0\\alpha=0showing the largest effect overall\. As hypothesized,α=0\\alpha=0improves performance on shorter sequences, but this effect is actually amplified as ground\-truth sequence length increases\. Intermediate settings such asα=0\.4\\alpha=0\.4andα=0\.6\\alpha=0\.6also remain positive across much of the range, whereas larger values ofα\\alphastay closer to parity or become negative as trajectory length increases\. Notably,α=0\.2\\alpha=0\.2underperforms relative to neighbouring values ofα\\alpha, highlighting the need for further investigation into the effect ofα\\alphaon downstream tasks\.

## 5Conclusion

We studied how pretraining sequence construction affects autoregressive EHR foundation models, proposing a new Patient Sampling method\. We show that Patient Sampling improves over the Global Stream baseline across a broad set of downstream benchmarks\. These results suggest that patient boundaries alone are insufficient, but that patient\-aware sampling and control over how training signal is distributed can improve autoregressive EHR pretraining\. More broadly, our findings identify sequence construction as an important design choice for autoregressive EHR foundation models\.

## Acknowledgements

JP is supported by UK Research and Innovation \(UKRI\) AI Centre for Doctoral Training in Digital Healthcare grant number EP/Y030974/1, and by a philanthropic BUPA PhD Scholarship\. YL is supported by UK Research and Innovation \(UKRI\) Centre for Doctoral Training in AI for Healthcare \(EP/S023283/1\)\. AAF is supported by the UK Research and Innovation \(UKRI\) Turing AI Fellowship grant \(EP/V025449/1\)\. MR & AAF acknowledge support of the UKRI AI programme, and the Engineering and Physical Sciences Research Council \(EPSRC\), for the AI Hub in Generative Models \(EP/Y028805/1\)\. AAF & MR acknowledge the use of resources provided by the Isambard\-AI National AI Research Resource \(AIRR\)\. Isambard\-AI is operated by the University of Bristol and is funded by the UK Government’s Department for Science, Innovation and Technology \(DSIT\) via UK Research and Innovation; and the Science and Technology Facilities Council \(ST/AIRR/I\-A\-I/1023\)\.

## References

- A\. L\. Goldberger, L\. A\. N\. Amaral, L\. Glass, J\. M\. Hausdorff, P\. C\. Ivanov, R\. G\. Mark, J\. E\. Mietus, G\. B\. Moody, C\. Peng, and H\. E\. Stanley \(2000\)PhysioBank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals\.Circulation101\(23\),pp\. e215–e220\.Note:RRID:SCR\_007345Cited by:[§4](https://arxiv.org/html/2607.22114#S4.SS0.SSS0.Px1.p1.1)\.
- A\. Johnson, L\. Bulgarelli, T\. Pollard, L\. A\. Celi, R\. Mark, and S\. Horng \(2023a\)Cited by:[§1](https://arxiv.org/html/2607.22114#S1.p4.1),[§4](https://arxiv.org/html/2607.22114#S4.SS0.SSS0.Px1.p1.1)\.
- A\. Johnson, L\. Bulgarelli, T\. Pollard, B\. Gow, B\. Moody, S\. Horng, L\. A\. Celi, and R\. Mark \(2024\)Cited by:[§1](https://arxiv.org/html/2607.22114#S1.p4.1),[§4](https://arxiv.org/html/2607.22114#S4.SS0.SSS0.Px1.p1.1)\.
- A\. Johnson, L\. Bulgarelli, T\. Pollard, S\. Horng, L\. A\. Celi, and R\. Mark \(2023b\)MIMIC\-IV\.PhysioNet\.Note:Version 2\.2External Links:[Document](https://dx.doi.org/10.13026/6mm1-ek67),[Link](https://doi.org/10.13026/6mm1-ek67)Cited by:[§1](https://arxiv.org/html/2607.22114#S1.p4.1),[§4](https://arxiv.org/html/2607.22114#S4.SS0.SSS0.Px1.p1.1)\.
- Z\. Kraljevic, D\. Bean, A\. Shek, R\. Bendayan, H\. Hemingway, J\. A\. Yeung, A\. Deng, A\. Balston, J\. Ross, E\. Idowu, J\. T\. Teo, and R\. J\. B\. Dobson \(2024\)Foresight&\#x2014;a generative pretrained transformer for modelling of patient timelines using electronic health records: a retrospective modelling study\.The Lancet Digital Health6\(4\),pp\. e281–e290\.Note:doi: 10\.1016/S2589\-7500\(24\)00025\-6External Links:[Document](https://dx.doi.org/10.1016/S2589-7500%2824%2900025-6),ISBN 2589\-7500,[Link](https://doi.org/10.1016/S2589-7500(24)00025-6)Cited by:[§1](https://arxiv.org/html/2607.22114#S1.p1.1),[§2](https://arxiv.org/html/2607.22114#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, and I\. Sutskever \(2019\)Language models are unsupervised multitask learners\.OpenAI Technical Report\.External Links:[Link](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)Cited by:[§2](https://arxiv.org/html/2607.22114#S2.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2607.22114#S4.SS0.SSS0.Px1.p2.1)\.
- P\. Renc, M\. K\. Grzeszczyk, N\. Oufattole, D\. Goode, Y\. Jia, S\. Bieganski, M\. B\. A\. McDermott, J\. Was, A\. E\. Samir, J\. W\. Cunningham, D\. W\. Bates, and A\. Sitek \(2025\)Foundation model of electronic medical records for adaptive risk estimation\.GigaScience14,pp\. giaf107\.External Links:ISSN 2047\-217X,[Document](https://dx.doi.org/10.1093/gigascience/giaf107),[Link](https://doi.org/10.1093/gigascience/giaf107),https://academic\.oup\.com/gigascience/article\-pdf/doi/10\.1093/gigascience/giaf107/64443371/giaf107\.pdfCited by:[§2](https://arxiv.org/html/2607.22114#S2.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2607.22114#S4.SS0.SSS0.Px1.p2.1)\.
- P\. Renc, Y\. Jia, A\. E\. Samir, J\. Was, Q\. Li, D\. W\. Bates, and A\. Sitek \(2024\)Zero\-shot health trajectory prediction using transformers\.npj Digital Medicine7\(1\),pp\. 256\.External Links:[Document](https://dx.doi.org/10.1038/s41746-024-01235-0),[Link](https://doi.org/10.1038/s41746-024-01235-0)Cited by:[§1](https://arxiv.org/html/2607.22114#S1.p1.1),[§2](https://arxiv.org/html/2607.22114#S2.SS0.SSS0.Px1.p1.1)\.
- W\. Shi, S\. Min, M\. Lomeli, C\. Zhou, M\. Li, X\. V\. Lin, N\. A\. Smith, L\. Zettlemoyer, W\. Yih, and M\. Lewis \(2024\)In\-context pretraining: language modeling beyond document boundaries\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=LXVswInHOo)Cited by:[§2](https://arxiv.org/html/2607.22114#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Waxler, P\. Blazek, D\. White, D\. Sneider, K\. Chung, M\. Nagarathnam, P\. Williams, H\. Voeller, K\. Wong, M\. Swanhorst, S\. Zhang, N\. Usuyama, C\. Wong, T\. Naumann, H\. Poon, A\. Loza, D\. Meeker, S\. Hain, and R\. Shah \(2025\)Generative medical event models improve with scale\.External Links:2508\.12104,[Link](https://arxiv.org/abs/2508.12104)Cited by:[§2](https://arxiv.org/html/2607.22114#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Zhao, Y\. Qu, K\. Staniszewski, S\. Tworkowski, W\. Liu, P\. Miłoś, Y\. Wu, and P\. Minervini \(2024\)Analysing the impact of sequence composition on language model pre\-training\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 7897–7912\.External Links:[Link](https://aclanthology.org/2024.acl-long.427/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.427)Cited by:[§2](https://arxiv.org/html/2607.22114#S2.SS0.SSS0.Px2.p1.1)\.

## Appendix AModel, Training, and Evaluation Framework

### Model\.

We use a decoder\-only transformer based on a GPT\-2 style architecture\. The model consists of a learned token embedding layer, a stack of transformer blocks with causal self\-attention and MLP sublayers, a final layer normalization, and a linear language\-modeling head\. Rotary positional embeddings \(RoPE\) are used in place of learned absolute position embeddings\. Models were trained on a single NVIDIA GH200 with a standard next\-token prediction objective over tokenized EHR trajectories\. Table[3](https://arxiv.org/html/2607.22114#A1.T3)summarizes the core model and optimization hyperparameters used in our experiments\.

Table 3:Core model and training hyperparameters used in our experiments\.
### Evaluation framework\.

Downstream evaluation is performed using a rollout\-based benchmark framework\. For each benchmark, evaluation prompts are constructed by identifying a benchmark\-specific prediction point together with two terminating outcome tokens in each patient trajectory\. The model receives the patient history up to the prediction point and is then rolled out autoregressively until a terminating token is generated or a maximum generation length of 4096 tokens is reached\. For each prompt, we run 20 stochastic rollouts\. A scalar score is computed as the fraction of rollouts that terminate with the different end tokens, and this score is used as the model output for binary classification\. Simulations that exceed the 4096\-token limit are terminated and excluded from benchmark calculations\. AUROC and AUPRC are then computed from the resulting per\-prompt scores\. Table[4](https://arxiv.org/html/2607.22114#A1.T4)presents the definitions for each benchmark we report\.

We note that the rollout\-based evaluation is stochastic and can be noisy, and due to computational constraints we were unable to repeat all experiments across multiple random seeds\.

Table 4:Benchmark definitions used in the rollout\-based evaluation framework\.