EHR-MPC: Inference-Time Control for Sepsis Treatment with Generative Patient Digital Twins

# EHR-MPC: Inference-Time Control for Sepsis Treatment with Generative Patient Digital Twins
Source: [https://arxiv.org/html/2607.08793](https://arxiv.org/html/2607.08793)
\\theorembodyfont\\theoremheaderfont\\theorempostheader

:\\theoremsep \\jmlrvolume\[VOLUME \# TBD\]\\jmlryear2026\\jmlrworkshopMachine Learning for Healthcare

\\NameWei Qi \\addrBroad Institute of MIT and Harvard\\NameNa Li\\Email \\addrHarvard University\\NameAnn Woolley\\Email \\addrBrigham and Women’s Hospital\\NameLisa Cosimi\\Email \\addrBrigham and Women’s Hospital\\NameRoy Kishony\\Email \\addrTechnion–Israel Institute of Technology\\NameDeborah Hung\\Email \\addrBroad Institute of Harvard and MIT

###### Abstract

Sepsis is a leading cause of mortality, yet optimal treatment policies remain contested\. Existing reinforcement learning \(RL\) approaches learn fixed strategies for sepsis treatment, limiting adaptability to changing clinical objectives during inference\. We proposeEHR\-MPC, a framework that decouples learning patient dynamics from optimizing treatment by training a patient digital twin in the form of a generative electronic health record \(EHR\) model\. The digital twin predicts clinical trajectories under interventions and enables model predictive control \(MPC\) to optimize treatments via inference\-time planning over simulations\. We evaluateEHR\-MPCon a multicenter ICU sepsis cohort spanning 8 hospitals in the Mass General Brigham health system using both off\-policy importance sampling and on\-policy simulation\-based evaluation\. Relative to RL baselines,EHR\-MPCachieves comparable off\-policy performance and improved simulation performance\. Unlike RL, this work frames sepsis treatment optimization as inference\-time control over learned patient dynamics, establishing a general framework for decision making with generative clinical models\.

## 1Introduction

Treatment of sepsis patients is a major healthcare challenge affecting more than 48 million people and resulting in 11 million deaths per year globally\(who2024sepsis\)\. Extensive work has explored optimal sepsis treatment policies in both clinical trials\(venkatesh2018adjunctive;annane2018hydrocortisone\)and reinforcement learning\(raghu2017deep;raghu2017continuous;raghu2018model;komorowski2018artificial;huang2022reinforcement\)\. In particular, corticosteroid administration remains a longstanding clinical question, with more than 60 randomized controlled trials conducted\(schumer1976steroids;annane2025corticosteroids\)\. Still, evidence guiding which patients should receive corticosteroids, as well as the appropriate timing and dosage, remains contested\(marik2018steroids\)\.

While both clinical trials and reinforcement learning \(RL\) aim to identify optimal policies for sepsis management, their recommendations remain largely disconnected\. For instance, one RL study suggests that “optimal treatment \[with corticosteroids\] may be more restrictive than routine clinical practice”\(bologheanu2023development\), in contrast to randomized clinical trial evidence indicating that corticosteroids “probably reduce 28\-day and hospital mortality among patients with sepsis”\(annane2025corticosteroids\)\. This discrepancy highlights a limitation of RL approaches, as despite strong retrospective performance under off\-policy evaluation, learned policies are often difficult to interpret, adapt, or validate in clinical practice\(zhang2022interpretable;frommeyer2025reinforcement\)\. Optimizing a fixed reward function without explicitly modeling patient dynamics limits the ability of RL\-based treatment strategies to accommodate individualized clinical goals or to adapt to competing objectives\(jayaraman2024primer\)\. In practice, sepsis management requires balancing trade\-offs such as short\-term hemodynamic stabilization versus long\-term organ\-system preservation, informed by clinician judgment, evolving standards of care, and patient\-specific context\(prescott2026surviving\)\. Because RL typically entangles patient dynamics and reward specification within a single model, adapting a learned policy to new clinical objectives requires retraining, limiting robustness to changing goals, deployment settings, and human\-in\-the\-loop constraints\.

The need to improve sepsis management, together with the limitations of prior RL approaches, motivates decoupling the learning of patient dynamics from the optimization of treatment decisions, so that clinical objectives are specified at inference time rather than embedded in a fixed policy\. We proposeEHR\-MPC, a framework that learns a generative digital twin of patient trajectories from electronic health record \(EHR\) data and performs treatment optimization using model predictive control \(MPC\)\. The digital twin simulates counterfactual trajectories under candidate interventions, while an MPC controller evaluates and selects action sequences according to clinically specified objectives at inference time, forming a closed\-loop system that can incorporate clinician feedback \(Fig\.[1](https://arxiv.org/html/2607.08793#S1.F1)\)\. This framework enables treatment strategies to be optimized under new objectives without retraining, supporting flexible, objective\-aware, and human\-in\-the\-loop clinical decision\-making\.

![Refer to caption](https://arxiv.org/html/2607.08793v1/x1.png)Figure 1:Digital Twin MPC for Treatment Optimization\.\(left\) A patient generates data that is observed by a clinician, tokenized, and sent to the digital twin\. \(simulation\) Using observed data, a digital twin performs forecasts on the state of the patient as if several candidate treatments were applied\. Each forecast is evaluated according to several clinical outcomes\. \(optimization\) Based upon clinical goals defined at inference time, the action token generating the best simulated outcome is translated to the proposed intervention\.Generalizable Insights about Machine Learning in Healthcare\.This work suggests three potential advantages of decoupling dynamics from policy optimization\. First, separating patient dynamics from intervention policies changes the structure of the learning problem, as modeling patient trajectories explicitly yields a digital twin that is reusable across objectives, whereas policy learning must entangle both dynamics and treatment objectives\. Second, generative EHR models support decision\-making algorithms, where pretrained patient digital twins enable downstream optimization via inference\-time methods such as model predictive control \(MPC\)\. This suggests a broader class of clinical machine learning systems that extend beyond prediction to planning over learned patient simulators\. Third, moving from training\-time policy optimization to inference\-time planning enables adaptation to changing clinical objectives without retraining, supporting more flexible and objective\-aware decision\-making\. Together, these insights point toward a paradigm in which reusable models of patient dynamics replace fixed policies as the primary object of learning, with treatment decisions obtained through inference\-time control over these models\.

## 2Related Work

We compare two existing paradigms for resolving the question of steroid use in sepsis and introduce a third framework\. Table[1](https://arxiv.org/html/2607.08793#S2.T1)summarizes this section\.

#### Optimal use of steroids in sepsis remains unresolved\.

The use of corticosteroids for sepsis and septic shock has been debated for decades, with trials dating back to the 1970s\(schumer1976steroids\)\. Despite substantial randomized controlled trials, no consensus has emerged on if, when, and for whom, steroids are beneficial\(venkatesh2018adjunctive;annane2018hydrocortisone;annane2025corticosteroids\)\. This ambiguity reflects that steroid therapy involves a trade\-off between suppressing harmful inflammation and impairing host immune response, and the balance of these effects varies across patients and over time\(wiersinga2018handbook\)\. As a result, the optimal treatment strategy is dynamic and patient\-specific, depending on the patient state and treatment goals\. These challenges make steroid use in sepsis management a longstanding, open challenge and an example of sequential decision\-making under uncertainty, motivating the use of machine learning to find improved treatment strategies\.

#### Reinforcement learning for sepsis\.

The AI Clinician was an RL model trained to recommend vasopressor and fluid dosing for sepsis patients, reporting improved off\-policy evaluation performance\(komorowski2018artificial\)\. Subsequent work expanded the action space to include steroids\(bologheanu2023development\), antibiotics\(futoma2018learning;wang2024clinical\), and continuous treatment representations\(raghu2017continuous;li2020optimizing\), while also incorporating safety constraints and uncertainty modeling\(nanayakkara2022unifying;tu2025offline\)\. A predominant RL paradigm has relied on Q\-learning–based methods, with policy evaluation commonly performed using off\-policy estimators such as importance sampling or doubly robust methods\(peng2018improving;jia2020safe;liu2021offline;huang2022reinforcement;bock2022superhuman;wu2023value;zhang2024optimizing;choi2024deep;drudi2024reinforcement\)\. Across these approaches, policies are learned from retrospective EHR data by optimizing value functions that implicitly encode both patient dynamics and clinical objectives\. Despite extensive methodological development, these approaches have had limited translation into clinical practice\. More broadly, many results are validated on a small number of benchmark datasets, such as MIMIC and eICU, raising concerns about generalizability\(pollard2018eicu;johnson2023mimic\)\. Collectively, this suggests that learning policies directly from observational data without explicitly modeling patient dynamics can yield brittle and potentially misaligned treatment recommendations\.

Table 1:Comparison of paradigms for sepsis treatment optimization\.The EHR\-MPC framework combines the interpretability of clinical trials with the data\-driven approach of RL and enables inference\-time adaptation\. Characterizations vary by implementation\.
#### Foundation models as patient digital twins for control\.

Large language models trained on clinical notes and structured EHR tokens have demonstrated strong performance on diagnostic and predictive tasks\(lee2020biobert;rasmy2021med\)\. More recently, trajectory\-level foundation models have been shown to capture temporal physiological structure, enabling prediction of patient evolution over time\(renc2024zero;renc2025foundation;makarov2025large;li2025mira\)\. These models can be interpreted as*digital twins*, that is, data\-driven simulators of patient dynamics that evolve under different clinical interventions\. This framing enables decision\-making through optimization over predicted patient trajectories\(alge2024continuous;pickard2025dynamic;prunella2026evolutionary\)\. In parallel, classical model predictive control \(MPC\) has been applied in clinical settings such as drug delivery and glucose regulation, but relies on hand\-crafted physiological models that limit flexibility and scalability\(ionescu2008robust;bruttomesso2009closed;nacscu2014advanced\)\. In contrast, learned digital twins provide a data\-driven foundation for MPC, enabling inference\-time optimization over treatments to improve patient outcomes\.

## 3EHR\-MPC: Electronic Health Record Model Predictive Control

We introduceEHR\-MPC, a framework that decouples \(i\) learning patient dynamics from real\-time EHR data and \(ii\) optimizing clinical decisions\. We formalize drug administration as a sequential decision\-making problem, then present a two\-stage approach of a digital twin patient model and an MPC controller that optimizes treatment actions during inference\.

### 3\.1Problem Formulation

We model sepsis management as a state–action–reward process\. At each decision timett, the*state*x​\(t\)x\(t\)consists of all clinical data available in the ICU \(vitals, labs, interventions, demographics, and device settings\)\. An*action*a​\(t\)a\(t\)includes any clinical intervention, such as the administration of corticosteroids\. A*reward*functionR​\(x​\(t\),a​\(t\)\)R\(x\(t\),a\(t\)\)encodes the clinical objective, such as the Sequential Organ Failure Assessment \(SOFA\) score or mortality\.

This formulation is compatible with both previous RL studies and the proposed framework\. Unlike RL, which explicitly trains a policy by entangling patient dynamics and clinical objectives during training, EHR\-MPC trains only a generative model of patient dynamics and obtains treatment decisions entirely through inference\-time planning\.

### 3\.2Learning and Simulating Patient Digital Twins

We learn a model to forecast patient trajectories by training on tokenized EHR data\. Data are pulled directly from an operational EHR database shared by several hospitals so that the model is designed for constraints that arise during the real\-time deployment at a clinical setting\.

#### Tokenized EHR\.

Each patient is represented as a sequencex1:k=\(x1,x2,…,xk\)x\_\{1:k\}=\(x\_\{1\},x\_\{2\},\\ldots,x\_\{k\}\)where tokensxix\_\{i\}are ordered in time and correspond to clinical events such as measurements of vitals, labs, device settings, or drug administrations\. At timettthe state isx​\(t\)=x1:kt\.x\(t\)=x\_\{1:k\_\{t\}\}\.The state variablex1:kx\_\{1:k\}represents both historic patient data and clinical interventions\. Continuous measurements are discretized into bins, and interventions \(e\.g\., administering corticosteroids\) are represented with action tokens\. Additional special tokens to indicate start and end of the sequence, unknown clinical events, and the passing of four hour intervals, are also included\. From timettto timet\+h,t\+h,the patient state updates as:

x1:kt\+h←StreamUpdate​\(x1:kt,t\+h\)\.x\_\{1:k\_\{t\+h\}\}\\leftarrow\\mathrm\{StreamUpdate\}\(x\_\{1:k\_\{t\}\},t\+h\)\.\(1\)There arekt\+h−ktk\_\{t\+h\}\-k\_\{t\}new tokens added to the patient state during this time interval\. The durationhhis a time horizon that represents intervals such as the time delay in streaming data from the EHR database or the forecasting horizon for predicting patient outcomes\. Tokens are indexed by both order \(kk\) and time \(tt\) to accommodate irregular sampling schedules in EHR and event triggered control\(heemels2012introduction\)\. The StreamUpdate operator is modeled by the generative patient digital twin\.

#### Simulated patient dynamics\.

Given the current patient historyx1:ktx\_\{1:k\_\{t\}\}, the digital twin defines the next\-token distribution

pθ​\(xkt\+1∣x1:kt\),p\_\{\\theta\}\(x\_\{k\_\{t\}\+1\}\\mid x\_\{1:k\_\{t\}\}\),\(2\)whereθ\\thetaare the parameters of the twin model\. This distributionpθp\_\{\\theta\}can be instantiated using sequence models such as transformers\(makarov2025large;li2025mira\)\. We train a moderate\-scale model on our cohort, though the formulation readily scales to larger pretrained models\(renc2024zero\)\.

The digital twinpθp\_\{\\theta\}is trained to forecast patient trajectories over a fixed horizonhh\(e\.g\., the next 24 hours\)\. Beginning from timett, this corresponds to generating tokens up to a horizont\+h,t\+h,which are distributed according to

pθ​\(xkt:kt\+h\|xkt\)=∏i=ktkt\+hpθ​\(xi\|x<i\)\.p\_\{\\theta\}\(x\_\{k\_\{t\}:k\_\{t\+h\}\}\\ \|\\ x\_\{k\_\{t\}\}\)=\\prod\_\{i=k\_\{t\}\}^\{k\_\{t\+h\}\}p\_\{\\theta\}\(x\_\{i\}\\ \|\\ x\_\{<i\}\)\.\(3\)The number of tokens required to reach timet\+ht\+hdepends on the realized sequence\. For example, whenh=24h=24hours, generation proceeds until six\[4 hour\]tokens have been produced\.

#### Simulation via Next\-Token Prediction\.

To generate patient trajectories under a candidate treatment sequenceakt:kt\+ha\_\{k\_\{t\}:k\_\{t\+h\}\}, we*force*the corresponding action tokens into the input and recursively generate future tokens using the autoregressive model:

p​\(xkt:kt\+h∣x1:kt,akt:kt\+h\)=∏i=ktkt\+hp​\(xi\|x<i,a<i\),p\(x\_\{k\_\{t\}:k\_\{t\+h\}\}\\mid x\_\{1:k\_\{t\}\},a\_\{k\_\{t\}:k\_\{t\+h\}\}\)=\\prod\_\{i=k\_\{t\}\}^\{k\_\{t\+h\}\}p\(x\_\{i\}\\ \|\\ x\_\{<i\},a\_\{<i\}\),\(4\)whereakta\_\{k\_\{t\}\}denotes the actions applied up to stepii\. The resulting model defines a controlled generative process\(plumerault2020controlling\)\. This procedure yields a counterfactual trajectory in which the generated tokens represent the model’s predicted evolution of patient state \(e\.g\., vitals, labs, and interventions\) under the proposed treatment\.

Eq\.[4](https://arxiv.org/html/2607.08793#S3.E4)enables the modelpθp\_\{\\theta\}to act as a*patient digital twin*: a data\-driven, virtual replica that evolves in response to real\-time EHR data and proposed interventions\(laubenbacher2024digital\)\. The digital twin can be viewed as a simulator of patient trajectories, forecasting how physiology may evolve under alternative treatment strategies and enabling comparison of candidate interventions before action\.

### 3\.3Model Predictive Control via Token\-Forced Rollouts

To derive treatment recommendations, we apply model predictive control \(MPC\) to the learned patient digital twinpθp\_\{\\theta\}at each decision time\. Rather than learning a fixed policy, the controller performs explicit planning at inference time\. MPC evaluates candidate intervention sequences by simulating their effect on future patient trajectories, scores the trajectories under a given objective, and selects the action sequence with the highest predicted utility\. Because MPC operates entirely at inference time, the clinical objective \(i\.e\., reward function\) can be specified after training\. This enables flexible optimization across different and potentially evolving clinical goals, and allows the controller to evaluate interventions that were not explicitly anticipated during training of the digital twin\.

#### Token\-forced rollouts\.

Given a candidate action sequenceat:t\+ha\_\{t:t\+h\}over a horizonhh, we simulate a counterfactual trajectory by*forcing*the corresponding action tokens into the model input and generating future tokens by sampling from the distribution defined in Eq\.[4](https://arxiv.org/html/2607.08793#S3.E4)\. At each stepktk\_\{t\}, the next token is sampled

x^kt\+1∼p​\(xkt\+1\|x1:kt,akt:kt\+h\),\\hat\{x\}\_\{k\_\{t\+1\}\}\\sim p\(x\_\{k\_\{t\+1\}\}\\ \|\\ x\_\{1:k\_\{t\}\},a\_\{k\_\{t\}:k\_\{t\+h\}\}\),\(5\)yielding a trajectoryx^t:t\+h\\hat\{x\}\_\{t:t\+h\}that represents the predicted evolution of patient state under the proposed interventions\.

This procedure simulates “what would happen” under a treatment sequence\. By explicitly modeling intervention effects, the framework yields \(i\) a mechanistic forecast of patient trajectory evolution, rather than only final outcomes, and \(ii\) a basis for evaluating and comparing outcomes across alternative treatment strategies\.

#### Trajectory evaluation\.

Each simulated trajectory is evaluated according to a reward function\. The purpose of these functions is to evaluate how well each patient trajectory fits defined clinical endpoints\. Common endpoints considered in clinical trials for learning improved sepsis treatment policies include: patient mortality, length of stay in the hospital and ICU, and organ failure, which is commonly measured with the Sequential Organ Failure Assessment \(SOFA\) score\(vincent1996sofa\)\.

A reward functionR:x1:kt→ℝR:x\_\{1:k\_\{t\}\}\\xrightarrow\{\}\\mathbb\{R\}assigns a numeric score to each patient trajectory\. Computationally,RRacts as a decoding objective over trajectories, converting predicted clinical sequences into a scalar utility that can be optimized via search over intervention candidates\. These reward functions can be specified in two ways\. In an explicit formulation, specific tokens \(e\.g\.,\[Mortality\]\) directly induce reward or penalty\. In an implicit formulation, clinically relevant outcomes such as SOFA score or length of stay are estimated from the trajectory rather than directly observed in the token sequence\. This distinction is necessary because several endpoints are not directly represented as tokens\. For example, length of stay is only known after transfer or discharge, and SOFA components may be only partially and irregularly observed during an ICU stay\. As a result, these quantities are partially observable and must be inferred from the evolving trajectory rather than read off the sequence directly\. In both formulations, constructingRRcan be achieved for any clinical objective by constructing either explicit functions operating on the token sequence or with task\-specific heads that predict clinical outcomes\.

Evaluation is performed at inference time, enabling the same learned dynamics model to support optimization over multiple, potentially changing objectives without retrainingpθp\_\{\\theta\}\. This flexibility is a consequence of decoupling patient dynamics from clinical objectives\. BecauseRRis never embedded in the modelpθp\_\{\\theta\}, new reward functions can be introduced or modified at deployment time without any retraining\. This allows a single trained digital twin to simultaneously serve clinicians with different treatment priorities, adapting to evolving standards of care or patient\-specific goals without additional model development\.

#### Action selection and receding\-horizon control\.

During inference, the controller selects a sequence of interventions by solving a planning problem over the digital twin:

at∗=arg⁡maxakt:kt\+h⁡𝔼x^kt:kt\+h∼pθ\(⋅∣x1:kt,akt:kt\+h\)​\[R​\(x^kt:kt\+h\)\]\.a\_\{t\}^\{\*\}=\\arg\\max\_\{a\_\{k\_\{t\}:k\_\{t\+h\}\}\}\\ \\mathbb\{E\}\_\{\\hat\{x\}\_\{k\_\{t\}:k\_\{t\+h\}\}\\sim p\_\{\\theta\}\(\\cdot\\mid x\_\{1:k\_\{t\}\},a\_\{k\_\{t\}:k\_\{t\+h\}\}\)\}\\left\[R\(\\hat\{x\}\_\{k\_\{t\}:k\_\{t\+h\}\}\)\\right\]\.\(6\)In practice, this optimization is intractable to solve exactly and is approximated via simulation\-based search\(garcia1989model\)\. Candidate action sequences are sampled \(or constructed via heuristic exploration\), their corresponding trajectories are generated by sampling frompθp\_\{\\theta\}, and the resulting trajectories are scored usingRR\. The action sequence with highest estimated value is selected\. See Algorithm[1](https://arxiv.org/html/2607.08793#alg1)\.

Only the first actionat∗a\_\{t\}^\{\*\}is executed, after which the system observes updated patient data and repeats the optimization\. This receding\-horizon procedure enables continual re\-planning as new information becomes available, analogous to clinical practice where treatment decisions are repeatedly updated in response to evolving patient state\.

#### Methodological interpretation\.

EHR\-MPCdecouples learning of patient dynamics from decision\-making by using a generative digital twin to explicitly simulate future trajectories under candidate intervention sequences\. At inference time, decisions are obtained through simulation\-based search, where each candidate intervention is evaluated by rolling out the learned model and scoring the resulting trajectory against a clinically specified objective\. This shifts computation from offline policy fitting to online planning, enabling clinical objectives to be modified without retraining\. This formulation enables three capabilities absent from standard RL approaches, namely \(i\) inference\-time objective specification, \(ii\) direct inspection of counterfactual patient trajectories, and \(iii\) reuse of a single dynamics model across multiple clinical goals\.

## 4Cohort Selection

### 4\.1Clinical Setting and Data Source

We assembled a multi\-site cohort of ICU patients from a large health system comprising eight hospitals\. This included patient encounters at two academic medical centers \(AMC\) and six community hospitals \(CH\) observed from 2022 onward\. Data originate from the institutional electronic health record system, which feeds into a relational database\. Patient encounter information, including lab results, vital signs, locations, medication administrations, diagnoses, and other signals were extracted from the production EHR system\.

### 4\.2Cohort Identification

We constructed the study cohort from all patient encounters with at least one ICU admission across a curated set of more than 30 critical care units, spanning medical, surgical, cardiac, and mixed ICUs across all participating sites\. From this eligible population, we sampled 36,930 patients, including both sepsis and non\-sepsis ICU admissions\. The resulting cohort is temporally uniform over the collection dates and stratified across ICU locations in proportion to their underlying patient volumes\. Including a broad ICU population was intentional, since sepsis may develop during an ICU stay and diagnostic labeling based on ICD codes is known to be imperfect and sensitive to evolving clinical definitions\(liu2022accuracy\)\. This design reflects deployment conditions in which the model operates over general ICU admissions rather than a pre\-filtered diagnostic cohort, and avoids coupling cohort construction to the same coding schemes used to define the target outcome\. Cohort characteristics stratified by site are reported in Table[2](https://arxiv.org/html/2607.08793#S4.T2)\.

Table 2:Patient cohort\.3636k ICU patients from eight hospitals, collected between 2022 and 2026\.Location\\columncolor\[gray\]0\.92TotalMGHBWHSLMNWHWDHBWFCDHMVHHospital Type\\columncolor\[gray\]0\.92AMCAMCCHCHCHCHCHCHDemographics\\columncolor\[gray\]0\.92Patients\\columncolor\[gray\]0\.9236930 \(100\.0%\)136511229135412256184914651358519LOS \(hrs\)\\columncolor\[gray\]0\.9266\.2 \(27\.6–154\.6\)83\.074\.354\.749\.038\.544\.940\.87\.8Mortality\\columncolor\[gray\]0\.9212657 \(34\.3%\)428543371370864561494629117Female \(n\)\\columncolor\[gray\]0\.9216101 \(43\.6%\)5487532216541146841746641254Median Age \(yrs\)\\columncolor\[gray\]0\.9267 \(55–76\)6666687267676874Median BMI\\columncolor\[gray\]0\.9226\.9 \(23\.1–31\.4\)27\.026\.827\.025\.727\.826\.427\.126\.0SOFA Scores\\columncolor\[gray\]0\.92SOFA IQR\\columncolor\[gray\]0\.922–52–52–52–51–42–51–51–41–3Days\>3\>3subscores \(%\)\\columncolor\[gray\]0\.9278\.879\.581\.377\.370\.674\.975\.564\.957\.0Treatments\\columncolor\[gray\]0\.92Vasopressor\\columncolor\[gray\]0\.9221097 \(57\.1%\)959470671448110584257240465Antibiotic\\columncolor\[gray\]0\.9222635 \(61\.3%\)74308260235813861112967907215Corticosteroid\\columncolor\[gray\]0\.9212616 \(34\.2%\)463344051155722551458567125Diagnoses\\columncolor\[gray\]0\.92Sepsis\\columncolor\[gray\]0\.924129 \(11\.2%\)105399466034832429641044Septic shock\\columncolor\[gray\]0\.921495 \(4\.0%\)33337921113414013514518Respiratory failure\\columncolor\[gray\]0\.926741 \(18\.3%\)18091865105635343346366795Hepatic failure\\columncolor\[gray\]0\.92831 \(2\.3%\)385136110404251598Heart failure\\columncolor\[gray\]0\.928435 \(22\.8%\)300024511069534469409384119Coagulopathy\\columncolor\[gray\]0\.921226 \(3\.3%\)406468988151564323Encephalopathy\\columncolor\[gray\]0\.921277 \(3\.5%\)2752222387413011021117Pneumonia\\columncolor\[gray\]0\.925026 \(13\.6%\)14681113916372304300424129Shock \(other\)\\columncolor\[gray\]0\.923581 \(9\.7%\)107513962361822462611778
### 4\.3Data Extraction

For each encounter, we assembled a complete longitudinal EHR record by joining across flowsheet, laboratory, medication, procedure, administrative, demographic, and diagnostic tables within the EHR relational database\. We included available records from each patient admission as well as those occurring within two days before or after the patient encounter\. Time\-unrestricted historical data for comorbidities, prior diagnoses, mortality outcomes, and admission records to other hospitals were also included\. This design ensures that each patient representation incorporates both proximal clinical dynamics and relevant longitudinal context\.

Working with a production EHR database introduces several well\-known data quality challenges\. First, the database schema evolves over time as EHR systems are updated and hospital configurations change\. Second, substantial heterogeneity exists across sites in how clinical variables are recorded, including differences in flowsheet structure, laboratory naming conventions, medication ordering systems, and unit\-level documentation practices\. Third, many clinical events have uncertain or delayed timestamps, particularly for diagnosis codes, laboratory processing times, and documentation\-based observations, which can introduce temporal ambiguity in the recorded trajectories\.

These challenges are standard in large\-scale EHR analysis and are mitigated through careful manual curation of key mappings and alignment between backend data structures and their corresponding clinical semantics\. For example, while the SOFA score is derived from more than 100 raw database fields, these map to a small number of underlying physiological measurements\. In our framework, the use of a token\-based, self\-supervised sequence model provides additional robustness to these issues by learning directly from observed event streams without requiring perfect alignment of individual fields or strict synchronization of measurement times\.

### 4\.4Tokenized EHR Representation

Following extraction and preprocessing, each patient trajectory was converted to a discrete token sequence suitable for modeling as described in Section[3](https://arxiv.org/html/2607.08793#S3)\. Followingrenc2024zero, continuous measurements were discretized into bins and medications, procedures, and administrative \(admit, transfer, discharge, etc\.\) events were mapped to dedicated vocabulary entries\. The tokenized dataset contained 469 million content tokens, 90\.3% of which indicated observational data rather than interventions or other data\. The median patient trajectory sequence length contained 20,232 tokens of which 1,058 are unique\. Detailed token\-type breakdowns and vocabulary statistics are provided in Section[A\.2](https://arxiv.org/html/2607.08793#A1.SS2)\.

## 5Experiments

Following our decoupled formulation of learning and policy optimization, experiments are organized into two stages\. First, validating the digital twin as a predictive and intervention\-aware model \(Section[5\.1](https://arxiv.org/html/2607.08793#S5.SS1)\)\. Second, evaluating treatment policies derived from the model using both off\-policy and simulation\-based methods \(Section[5\.2](https://arxiv.org/html/2607.08793#S5.SS2)\)111See the appendix for complete experimental details\.\.

### 5\.1Digital Twin Construction and Forecasting Evaluation

We train a transformer to parameterizepθp\_\{\\theta\}\(Section[3](https://arxiv.org/html/2607.08793#S3)\) and evaluate it on the following tasks\.

![Refer to caption](https://arxiv.org/html/2607.08793v1/x2.png)Figure 2:Representations and Forecasting\.Prediction of outcomes from patient embeddings and random forest classifiers\.#### Outcome Prediction from Learned Patient Representations\.

We assess representation quality learned by the digital twin according to their ability to predict several standard clinical endpoints\. We fit lightweight random forest models to predict the ICU length of stay \(time\-to\-event\), ICU readmission within the same encounter \(binary\), hospital readmission \(binary\), and all\-cause mortality \(binary\) from the internal representations learned by the digital twin of each trajectory until the first ICU day \(Fig\.[2](https://arxiv.org/html/2607.08793#S5.F2)\)\. Across hospital sites, the embeddings achieve consistent, nontrivial predictive performance, indicating that they capture meaningful signal\. We also observe modest site\-level structure in the embedding space, suggesting partial but not complete alignment across institutions\.

#### Patient Forecasting\.

We evaluate the digital twinpθp\_\{\\theta\}as a generative model of clinical time series using next\-token prediction \(NTP\)\. Performance is assessed via next\-mmtoken set recall and token\-level accuracy stratified by semantic type\(stein2023exposing\)\. For multi\-step forecasting, we measure whether true future tokens appear within the top\-iipredictions over horizonsm∈\{5,10\}m\\in\\\{5,10\\\}\(Table[3](https://arxiv.org/html/2607.08793#S5.T3), left\)\. We also report accuracy by token category \(Table[3](https://arxiv.org/html/2607.08793#S5.T3), right\), including medications, location/unit transitions, and laboratory, vital, and order events\. Medication tokens are predicted most accurately, reflecting their temporal persistence and repeated administration\. In contrast, laboratory, vital, and order tokens exhibit lower accuracy, consistent with their higher temporal variability and the stochastic nature of short\-term physiologic dynamics\.

Table 3:Digital Twin Forecasting\.Next\-token prediction \(NTP\) evaluation\. Left: Overall performance \(Set Recall, %\)\. Right: Accuracy stratified by token type\.Next\-mmtokensTop\-iim=5m=5m=10m=101086\.282\.82089\.485\.95092\.989\.9
Token TypeTop\-1Top\-5Top\-10Medications54\.874\.082\.7Location / unit31\.458\.766\.0Labs, vitals & orders27\.254\.365\.0

#### Dose\-Response Sensitivity\.

We evaluate whether the digital twin exhibits sensitivity to pharmacologic intervention tokens in line with Eq\.[4](https://arxiv.org/html/2607.08793#S3.E4)\. For each high\-acuity patient \(SOFA≥6\\geq 6\), we construct counterfactual 24\-hour trajectories by injecting repeated drug tokens into the observed context as if the drug were administered at a higher dose and then generate future tokens by sampling from the digital twinpθp\_\{\\theta\}\. The number of drug tokens is varied from 1 to 20, and estimated mortality risk is predicted using a task specific prediction head \(see Table[4](https://arxiv.org/html/2607.08793#S5.T4)\)\. Corticosteroid tokens induce a consistent, monotonic decrease in predicted mortality risk across increasing injection levels\. Antibiotics and vasopressors exhibit weaker and less monotonic responses, with more variability across dose levels\. These results suggest that the model does not respond uniformly to intervention tokens, but instead exhibits drug\-specific variation\.

The prediction and forecasting accuracy together with the intervention sensitivity satisfy empirical conditions for local controllability\. This supports the use of MPC to steer the learned digital twin\.

Table 4:Dose\-response sensitivity in the digital twin\.Predicted mortality under counterfactual rollouts with increasing numbers of injected intervention tokens\. Values correspond toP​\(mortality\)P\(\\mathrm\{mortality\}\)from a frozen prediction head\.

### 5\.2EHR\-MPC Treatment Policy Optimization

Using the learned digital twinpθp\_\{\\theta\}, we solve Eq\.[6](https://arxiv.org/html/2607.08793#S3.E6)to optimize treatment policies\. Evaluating treatment policies learned from observational EHR data presents significant statistical challenges\. Off\-policy weighted importance sampling \(WIS\) estimates policy value from observed trajectories but exhibits high variance when the learned and observed policies differ substantially; WIS is included here to enable comparison with prior RL approaches for sepsis that rely on similar protocols\. Simulation\-based evaluation enables on\-policy rollout under the learned dynamics but introduces dependence on model fidelity\. Due to these fundamental limitations, we interpret results jointly across both evaluation frameworks, treating consistency between them as stronger evidence than either alone\.

#### Off\-Policy Evaluation via Importance Sampling\.

We evaluateEHR\-MPCunder two reward functions: \(i\) minimizing SOFA score and \(ii\) minimizing mortality risk, comparing against a Q\-network baseline and the observed clinician policy using per\-decision weighted importance sampling \(WIS\)\. Althoughpθp\_\{\\theta\}is trained on the full ICU cohort, evaluation is restricted to sepsis patients\. Across both objectives,EHR\-MPCand the Q\-network achieve higher estimated value than the clinician policy under WIS \(Fig\.[3](https://arxiv.org/html/2607.08793#S5.F3), left\)\. Point estimates betweenEHR\-MPCand the Q\-network are similar with overlapping confidence intervals\.

![Refer to caption](https://arxiv.org/html/2607.08793v1/x3.png)Figure 3:Offline Evaluation of EHR\-MPC Treatment Policies\.The learned EHR\-MPC policy is evaluated with WIS \(left\) and simulated patient outcomes \(middle, right\)\. The improvement of patient outcomes according to EHR\-MPC and Q\-networks is shown per patient \(right\)\.
#### On\-Policy Evaluation via Digital Twin Simulator\.

We assess learned policies using the digital twin as a simulator\. Patient trajectories are initialized from observed tokens up to the first ICU day, after which each policy \(EHR\-MPCand Q\-networks\) selects interventions at daily intervals as the simulated patient evolves over time\. Unlike WIS, which evaluates fixed logged trajectories under distribution shift, simulation\-based evaluation enables on\-policy rollouts under the learned digital twin\. Across simulations,EHR\-MPCconsistently outperforms Q\-network policies aimed to improve daily SOFA scores and decrease mortality risk, both in terms of average effect \(Fig\.[3](https://arxiv.org/html/2607.08793#S5.F3), center\) and individual simulations \(Fig\.[3](https://arxiv.org/html/2607.08793#S5.F3), right\)\. We examine the robustness of this result to model misspecification in the following experiment\.

#### Robustness to Model Misspecification\.

A potential concern with simulator\-based evaluation is thatEHR\-MPCmay trivially optimize the evaluation metric by exploiting the same dynamics model it plans with\. To address this, we decouple the*planning model*from the*evaluation model*using checkpoints from different training epochs: MPC planning uses an early\-epoch checkpoint \(40% of training epochs\), while policy scoring and simulation use a later, more\-converged checkpoint\. The Q\-network is also trained on the representations of the planning model used by MPC\. This separates action selection from action evaluation, providing a more rigorous test of generalization under model mismatch\. Under this protocol,EHR\-MPCcontinues to outperform the clinician policy with only modest performance degradation relative to the same\-model baseline \(Fig\.[4](https://arxiv.org/html/2607.08793#S5.F4)\)\. Degradation is larger for mortality risk minimization than for SOFA, consistent with SOFA providing denser per\-step rewards that are easier to optimize\.

![Refer to caption](https://arxiv.org/html/2607.08793v1/x4.png)Figure 4:MPC Robustness to Model Misalignment\.Performance of MPC and Q\-networks are evaluated on the digital twin simulator when the MPC controller uses a less accurate version of the patient digital twin\.
#### Treatment Policy Divergence\.

To complement the off\-policy and simulation\-based evaluations, we examine differences in action distributions betweenEHR\-MPC, Q\-networks, and the observed clinical policy \(Fig\.[5](https://arxiv.org/html/2607.08793#S5.F5), top\)\. The policies are stratified by location and recommendations of individual drugs\. We further quantify policy differences using Jensen\-Shannon divergence \(JSD\) over action distributions across hospital sites \(Fig\.[5](https://arxiv.org/html/2607.08793#S5.F5), bottom\)\. The Q\-network policy remains close to the clinical policy across most sites, whileEHR\-MPCshows greater divergence from both\. Finally, the clinical policy shows the greatest heterogeneity across sites, whereas bothEHR\-MPCand Q\-network policies are more consistent across locations \(Fig\.[6](https://arxiv.org/html/2607.08793#A2.F6)\)\. Analogous analyses for SOFA score reduction are provided in Figs\.[7](https://arxiv.org/html/2607.08793#A2.F7)and[8](https://arxiv.org/html/2607.08793#A2.F8)\.

![Refer to caption](https://arxiv.org/html/2607.08793v1/x5.png)Figure 5:Policy Divergence for Mortality Optimization\.\(Top\) The distribution of recommended drugs is shown for each of the three policies across locations\. \(Bottom\) The JSD divergence between these distributions is shown\.

## 6Discussion

EHR\-MPC frames sepsis treatment optimization as an inference\-time planning problem over a learned generative model of patient dynamics\. The framework separates two distinct optimization problems of \(i\) modeling how patients evolve under interventions and \(ii\) selecting interventions that optimize clinical goals\. Previous RL for sepsis addressed these tasks jointly to learn a fixed policy\. Oncepθp\_\{\\theta\}is learned, EHR\-MPC determines treatment strategies that are flexible to new clinical objectives without retraining, providing a robust and generalizable tool for decision\-making at inference time\.

#### Interpreting Policy Evaluations\.

Based on the WIS, the EHR\-MPC’s comparable performance relative to Q\-networks demonstrates the decoupled framework for treatment optimization can achieve comparable performance to RL methods that study sepsis treatment optimization\. Moreover, the increased divergence between EHR\-MPC and the clinician while maintaining comparable performance with the Q\-networks, which have policies more similar to the clinicians, indicate that EHR\-MPC overcomes the penalty of WIS for diverging from observed data\. This is an important property for searching and optimizing the treatment space\. Based on the simulations,EHR\-MPCconsistently outperforms Q\-networks on both SOFA improvement and mortality reduction, and this advantage persists under the model misspecification protocol, where the MPC controller planned over a poorly identified simulation model\. This robustness rules out the concern thatEHR\-MPCis merely exploiting a shared dynamics model, and supports the interpretation that online planning provides a robust framework for treatment optimization\. Across both evaluation schemes, which are standard in RL and MPC respectively, the results support the use of EHR\-MPC as a framework for inference\-time treatment optimization\.

Examining the action distributions provides further context for these results\. Regarding specific treatment recommendations, the Q\-network reduces corticosteroid utilization relative to clinical practice, consistent with prior RL studies\(bologheanu2023development\)\. In contrast,EHR\-MPCrecommends increased steroid use when optimized for mortality reduction, more consistent with clinical trial evidence that corticosteroids reduce 28\-day mortality in sepsis patients\(annane2025corticosteroids\)\. The site\-level variation shows that both learned policies are more consistent than the clinical policy\.

Taken together,EHR\-MPCis supported both empirically—by strong simulation performance and off\-policy estimates—and clinically, by producing treatment recommendations that better align with the broader evidence base than competing algorithms\.

#### Capabilities enabled by inference\-time control\.

By decoupling patient dynamics from treatment optimization, EHR\-MPC supports several capabilities that are difficult to achieve with RL models\. First, treatment strategies are computed via inference\-time planning, allowing decisions to be adapted to patient state and evolving clinical objectives without retraining\. Second, a single learned dynamics model can support multiple clinical objectives, enabling trade\-offs between competing goals\. Third, safety and feasibility constraints can be incorporated directly during planning, allowing candidate interventions to be filtered or penalized without requiring changes to the underlying model\. Finally, the framework explicitly models and predicts patient trajectories under interventions, providing clinicians with interpretable forecasts of patient state rather than only scalar reward estimates\. Together, these capabilities position EHR\-MPC as a flexible decision\-support framework that aligns with the requirements of clinical practice\.

#### Limitations\.

First, the primary limitation of this work is the fidelity and validation of the learned digital twin\. Whilepθp\_\{\\theta\}is trained on intervention\-aware patient trajectories, the training process does not provide guarantees of correct system identification\. This reflects a well\-known challenge in system identification and control\(ljung1998system;bemporad2007robust\), and more broadly in off\-policy evaluation for sequential decision\-making in healthcare\(shalit2017estimating;oberst2019counterfactual\)\. Improving the reliability of learned patient dynamics, as well as validating their behavior under interventions, remains an open and active problem in data\-driven clinical modeling\.

Second, offline policy evaluation remains a substantial challenge for validating this work along with other RL for sepsis optimization\. Off\-policy estimators such as WIS exhibit high variance and sensitivity to support mismatch between policies, specifically when evaluating policies that diverge from clinician behavior\(precup2000eligibility;gottesman2018evaluating\)\. In contrast, simulation\-based evaluation alleviates these statistical issues by enabling rollout under the learned dynamics, but introduces dependence on model accuracy\. Neither evaluation framework alone is sufficient, but their consistency provides stronger evidence than either in isolation\. While EHR\-MPC enables simulation\-based, on\-policy evaluation, substantial work remains to improve the fidelity, generalizability, and validation of such simulations\.

From a computational standpoint, this work adopts standard architectures and evaluation schemes to isolate the effect of the proposed framework\. However, each component, including the generative model, the MPC planner, and off\-policy evaluation, could be further optimized\. While the results demonstrate the effectiveness of EHR\-MPC, performance may be improved through more advanced architectures and more extensive hyperparameter tuning\. Nonetheless, these results establish the utility of EHR\-MPC under standard modeling choices, suggesting opportunities for improvement through advances in model architecture and optimization that are orthogonal to the proposed framework\.

Ultimately, the value of decoupling dynamics from policy optimization depends on how faithfully the digital twin captures patient physiology under intervention\. Improving digital twin fidelity, validating learned dynamics prospectively, and developing principled evaluation frameworks for simulation\-based policies are the central open problems for translating this class of methods into clinical practice\.

## References

## Appendix ACohort and Data Description

### A\.1Patient Categorization and Clinical Rewards

#### Disease definitions\.

Sepsis and related comorbidities were identified using ICD\-10 codes as listed in Table[5](https://arxiv.org/html/2607.08793#A1.T5)\.

Table 5:ICD\-10 codes used to define clinical conditions in the study cohort\.
#### Mortality\.

In\-hospital mortality was defined as a documented death during the index hospital encounter or within 72 hours of discharge\. Mortality labels were extracted from the Epic EHR system’s outcome and encounter tables\.

#### SOFA score\.

The Sequential Organ Failure Assessment \(SOFA\) score was computed from raw EHR flowsheet and laboratory data across six organ systems \(Table[6](https://arxiv.org/html/2607.08793#A1.T6)\)\. Each patient’s stay was partitioned into non\-overlapping 24\-hour windows anchored at first event; patients with fewer than 24 hours of data were excluded\. Within each window, the worst \(most abnormal\) value was used for each component, except PaO2and platelets where the lowest observed value is the worst\. Subscores follow standard 0–4 SOFA thresholds; the total is their sum over non\-missing components\.

Where arterial PaO2was unavailable, it was estimated from SpO2via a linear approximation of the oxygen–haemoglobin dissociation curve:PaO^2=3​\(SpO2−90\)\+60\\widehat\{\\text\{PaO\}\}\_\{2\}=3\(\\text\{SpO\}\_\{2\}\-90\)\+60mmHg, valid for SpO∈2\[75,100\]%\{\}\_\{2\}\\in\[75,100\]\\%\. FiO2was resolved by hierarchy: \(i\) directly measured, \(ii\) estimated from O2flow rate as20\+4×V˙​\(L/min\)20\+4\{\\times\}\\dot\{V\}\\,\(\\text\{L/min\}\)capped at 60%, or \(iii\) assumed 21%\. Mechanical ventilation was inferred from O2\-device flowsheet values containing “vent”; SOFA respiratory subscores of 3–4 require ventilation \(else capped at 2\)\. Vasopressor detection used a fixed\-score of 2 regardless of dose, as reliable infusion\-rate data were unavailable\.

Δ​SOFA\\Delta\\text\{SOFA\}was computed component\-wise between consecutive windows and summed only over components non\-missing in*both*windows, avoiding artifacts from missing data:

Δ​SOFA\(t\)=∑k∈𝒦t\(sk\(t\+1\)−sk\(t\)\),\\Delta\\text\{SOFA\}^\{\(t\)\}=\\textstyle\\sum\_\{k\\in\\mathcal\{K\}\_\{t\}\}\\\!\\bigl\(s\_\{k\}^\{\(t\+1\)\}\-s\_\{k\}^\{\(t\)\}\\bigr\),\(7\)where𝒦t\\mathcal\{K\}\_\{t\}is the set of components with non\-missing scores in both windowttandt\+1t\{\+\}1\.

Table 6:SOFA computation rules per 24\-hour window\.All EHR field patterns are case\-insensitive regular expressions matched on flowsheet event names\. “Agg\.” denotes within\-window aggregation\.†Scores 3–4 require concurrent mechanical ventilation; otherwise subscore capped at 2\.

### A\.2Patient Tokenization

Tables[7](https://arxiv.org/html/2607.08793#A1.T7)and[8](https://arxiv.org/html/2607.08793#A1.T8)summarize the tokenization scheme\.

Table 7:Tokenization statistics\.Content tokens exclude special tokens:\[BOS\], \[EOS\], \[PAD\], \[UNK\], \[MASK\], \[4 hours\]\.Table 8:Per\-hospital tokenization statistics\.

## Appendix BExperimental Details

### B\.1Digital Twin Architecture

The patient digital twin is a decoder\-only transformer with the following architecture:

- •Model family:GPT\-2\-style causal language model trained for next token prediction on the tokenized EHR data and finetuned task heads for predicting mortality risk and daily changes in SOFA\.
- •Parameters:74\.8M total \(backbone\)\.
- •Embedding dimension:d=512d=512\.
- •Vocabulary:consists of discretized vitals, labs, medications, procedures, administrative events, demographics, and special tokens \(\[BOS\],\[EOS\],\[PAD\],\[UNK\],\[4 hour\]\)\.
- •Context length:512 tokens \(block size\)\.

The base transformer model was trained on 7 NVIDIA A6000 GPUs for approximately 18 hours\.

Patient trajectories are long, exceeding the model’s context window\. To obtain patient trajectory\-level representations for outcome prediction, we aggregate representations across sequential, non\-overlapping windows spanning the trajectory\. This design aligns with the approximately Markovian nature of short\-horizon ICU dynamics, consistent with previous sepsis simulators\(bock2022superhuman\)\.

#### Evaluating Next Token Prediction\.

Standard top\-iiaccuracy measures whether the single true next token appears in the model’s top\-iipredictions\. Set Recall generalizes this to multi\-step forecasting horizons\.

Given a context prefixx1:kx\_\{1:k\}, define the*ground\-truth set*𝒯m=\{xk\+1,xk\+2,…,xk\+m\}\\mathcal\{T\}\_\{m\}=\\\{x\_\{k\+1\},x\_\{k\+2\},\\ldots,x\_\{k\+m\}\\\}as the set of distinct token identities observed in the nextmmpositions\. Let𝒯^i\\hat\{\\mathcal\{T\}\}\_\{i\}be the set of token identities in the model’s top\-iipredictions from the last\-position logits:

𝒯^i=\{idj\|j∈argtopi​pθ​\(xk\+1∣x1:k\)\}\.\\hat\{\\mathcal\{T\}\}\_\{i\}=\\left\\\{\\,\\text\{id\}\_\{j\}\\;\\middle\|\\;j\\in\\mathrm\{argtop\}\_\{i\}\\,p\_\{\\theta\}\(x\_\{k\+1\}\\mid x\_\{1:k\}\)\\,\\right\\\}\.\(8\)Set Recall at budgetiiover horizonmmis:

SR​@​\(i,m\)=\|𝒯m∩𝒯^i\|\|𝒯m\|\.\\mathrm\{SR\}@\(i,m\)=\\frac\{\|\\mathcal\{T\}\_\{m\}\\cap\\hat\{\\mathcal\{T\}\}\_\{i\}\|\}\{\|\\mathcal\{T\}\_\{m\}\|\}\.\(9\)For each patient in the held\-out set, we sample random positions as context endpoints and collect both the immediate next token \(for top\-iiaccuracy\) and the next\-mmtoken set \(for Set Recall\)\.

### B\.2Outcome Prediction Heads

#### SOFA delta head\.

A two\-layer MLP with architecture512→256→7512\\to 256\\to 7, predicting 24\-hour per\-component SOFA deltas\. Trained with mean squared error loss on frozen backbone representations\. The seven outputs correspond to total SOFA and six organ\-system components \(respiratory, coagulation, liver, cardiovascular, CNS, renal\)\.

#### Mortality prediction head\.

A three\-layer MLP with architectureLayerNorm​\(512\)→Linear​\(512,256\)→GELU→Dropout​\(0\.2\)→Linear​\(256,128\)→GELU→Dropout​\(0\.2\)→Linear​\(128,1\)\\text\{LayerNorm\}\(512\)\\to\\text\{Linear\}\(512,256\)\\to\\text\{GELU\}\\to\\text\{Dropout\}\(0\.2\)\\to\\text\{Linear\}\(256,128\)\\to\\text\{GELU\}\\to\\text\{Dropout\}\(0\.2\)\\to\\text\{Linear\}\(128,1\), predicting log\-odds of in\-hospital mortality\. Trained with binary cross\-entropy loss with class\-weight balancing \(positive rate: 38\.5%\) for 100 epochs with early stopping \(patience 20\), learning rate10−310^\{\-3\}, batch size 256, using Adam optimizer\. Achieves AUROC = 0\.9352 and AUPRC = 0\.9142 on held\-out patients\.

Separate heads are trained for different backbone checkpoints \(early and late fine\-tuning epochs\) to support cross\-epoch evaluation\.

### B\.3MPC Optimization Procedure

The MPC planing algorithm is outlined in Algorithm[1](https://arxiv.org/html/2607.08793#alg1)\. At each decision point, four intervention options are considered: giving a combination of vasopressor, antibiotics, or steroids, or the option to not give one of these medications\. Up to 5 representative drug tokens are considered per drug class\. The final hidden state \(last\-token representation\) of the transformer after rollout is extracted and passed to the outcome prediction head\. The action minimizing the predicted outcome score is selected\.

The autoregressive generation uses a greedy \(argmax\) decoding strategy, generating future tokens token\-by\-token until six\[4 hour\]tokens have been produced \(approximately 24 hours of simulated patient state\)\.

Algorithm 1EHR\-MPC Planning via Token\-Forced Rollouts0:Current patient state

x1:ktx\_\{1:k\_\{t\}\}, digital twin

pθp\_\{\\theta\}, reward function

RR, planning horizon

hh, number of candidates

NN
1:for

i=1i=1to

NNdo

2:Sample candidate action sequence

akt:kt\+h\(i\)a\_\{k\_\{t\}:k\_\{t\+h\}\}^\{\(i\)\}
3:Initialize trajectory

x^1:kt\(i\)←x1:kt\\hat\{x\}\_\{1:k\_\{t\}\}^\{\(i\)\}\\leftarrow x\_\{1:k\_\{t\}\}
4:for

τ=kt\\tau=k\_\{t\}to

kt\+hk\_\{t\+h\}do

5:Force action token

aτ\(i\)a\_\{\\tau\}^\{\(i\)\}into model input

6:Sample next token:

x^kτ\+1\(i\)∼pθ​\(xkτ\+1∣x^1:kτ\(i\),akt:kτ\(i\)\)\\hat\{x\}\_\{k\_\{\\tau\+1\}\}^\{\(i\)\}\\sim p\_\{\\theta\}\\big\(x\_\{k\_\{\\tau\+1\}\}\\mid\\hat\{x\}\_\{1:k\_\{\\tau\}\}^\{\(i\)\},a\_\{k\_\{t\}:k\_\{\\tau\}\}^\{\(i\)\}\\big\)
7:endfor

8:Compute trajectory reward:

J\(i\)←R​\(x^kt:kt\+h\(i\)\)J^\{\(i\)\}\\leftarrow R\\big\(\\hat\{x\}\_\{k\_\{t\}:k\_\{t\+h\}\}^\{\(i\)\}\\big\)
9:endfor

10:Select best sequence:

i∗←arg⁡maxi⁡J\(i\)i^\{\*\}\\leftarrow\\arg\\max\_\{i\}J^\{\(i\)\}
11:returnfirst action

at\(i∗\)a\_\{t\}^\{\(i^\{\*\}\)\}

### B\.4Per\-Decision Weighted Importance Sampling \(PDWIS\)

The behavior policyμ​\(a∣s\)\\mu\(a\\mid s\)is estimated by fitting a multinomial logistic regression classifier on the concatenated last\-token representations of all training\-set patient decision steps, predicting which of the four action classes was taken by the clinician\. At each evaluation stepttfor trajectoryii, the per\-step IS ratio is

ρt\(i\)=π​\(at\(i\)∣st\(i\)\)μ​\(at\(i\)∣st\(i\)\),\\rho\_\{t\}^\{\(i\)\}=\\frac\{\\pi\(a\_\{t\}^\{\(i\)\}\\mid s\_\{t\}^\{\(i\)\}\)\}\{\\mu\(a\_\{t\}^\{\(i\)\}\\mid s\_\{t\}^\{\(i\)\}\)\},\(10\)whereπ\\piis the evaluation policy \(softmax over predicted scores, temperatureτ=0\.5\\tau=0\.5\)\. Cumulative weights are clipped atcmax=10c\_\{\\text\{max\}\}=10to reduce variance:

wt\(i\)=∏τ≤tmin⁡\(ρτ\(i\),cmax\)\.w\_\{t\}^\{\(i\)\}=\\prod\_\{\\tau\\leq t\}\\min\\\!\\left\(\\rho\_\{\\tau\}^\{\(i\)\},\\,c\_\{\\text\{max\}\}\\right\)\.\(11\)The PDWIS estimate is

V^π=∑i∑twt\(i\)​rt\(i\)∑i∑twt\(i\),\\hat\{V\}^\{\\pi\}=\\frac\{\\sum\_\{i\}\\sum\_\{t\}w\_\{t\}^\{\(i\)\}\\,r\_\{t\}^\{\(i\)\}\}\{\\sum\_\{i\}\\sum\_\{t\}w\_\{t\}^\{\(i\)\}\},\(12\)wherert\(i\)=−Δ​SOFAt\(i\)r\_\{t\}^\{\(i\)\}=\-\\Delta\\text\{SOFA\}\_\{t\}^\{\(i\)\}\(positive when SOFA improves\) is taken from the observed EHR data\.

### B\.5Reinforcement Learning Baselines

To situate our framework against existing methods, we compare against Q\-network\-based policies trained on the same frozen patient representations𝐡p\\mathbf\{h\}\_\{p\}\. A Q\-networkQψ:ℝ512×𝒜→ℝQ\_\{\\psi\}:\\mathbb\{R\}^\{512\}\\times\\mathcal\{A\}\\to\\mathbb\{R\}estimates the expected discounted return of each \(state, action\) pair under a fixed reward functionrr, and is trained to minimize the Bellman residual:

ℒQ​\(ψ\)=𝔼​\[\(r​\(p,a\)\+γ​maxa′⁡Qψ¯​\(𝐡p′,a′\)−Qψ​\(𝐡p,a\)\)2\],\\mathcal\{L\}\_\{\\mathrm\{Q\}\}\(\\psi\)=\\mathbb\{E\}\\\!\\left\[\\left\(r\(p,a\)\+\\gamma\\max\_\{a^\{\\prime\}\}Q\_\{\\bar\{\\psi\}\}\(\\mathbf\{h\}\_\{p^\{\\prime\}\},a^\{\\prime\}\)\-Q\_\{\\psi\}\(\\mathbf\{h\}\_\{p\},a\)\\right\)^\{\\\!2\}\\right\],\(13\)whereγ\\gammais the discount factor andQψ¯Q\_\{\\bar\{\\psi\}\}is a periodically updated target network\. Q\-networks constitute the canonical fused policy\-dynamics baseline: the value function implicitly encodes both patient dynamics and the clinical objective, and adapting to a new reward function requires full retraining ofQψQ\_\{\\psi\}\. We train separate Q\-networks for each reward function considered in our experiments\. Each Q\-network was trained on a single A6000 GPU for up to 300 epochs with a learning rate of10−3\.10^\{\-3\}\.

### B\.6Additional Results

Figures[6](https://arxiv.org/html/2607.08793#A2.F6)\-[8](https://arxiv.org/html/2607.08793#A2.F8)illustrate policy distributions across mortality and SOFA rewards and locations\.

![Refer to caption](https://arxiv.org/html/2607.08793v1/x6.png)Figure 6:Policy Variation for Mortality Risk Minimization\.![Refer to caption](https://arxiv.org/html/2607.08793v1/x7.png)Figure 7:Policy Distributions and Variation for SOFA Minimization\.![Refer to caption](https://arxiv.org/html/2607.08793v1/x8.png)Figure 8:Policy Variation for SOFA Minimization\.