Rethinking How We Evaluate Methodological Progress in Health AI

# Rethinking How We Evaluate Methodological Progress in Health AI
Source: [https://arxiv.org/html/2609.18134](https://arxiv.org/html/2609.18134)
Matthew McDermottEmail:[mm6677@cumc\.columbia\.edu](mailto:[email protected])Affiliation:Department of Biomedical Informatics, Columbia University

###### Abstract

Methodological progress in artificial intelligence \(AI\) for electronic health records \(EHRs\) depends on determining which algorithms work better, and under which conditions\. However, such progress is thought to be hindered by difficulties in reproducibility and in defining clinically meaningful evaluation tasks\. We empirically study these barriers by re\-implementing 12 historical and recent algorithms within a shared evaluation framework and evaluating them on two clinical datasets, MIMIC\-IV and NWICU\. We compare two complementary task families: expert\-authored clinically meaningful tasks and generated tasks defined from randomly sampled event codes and prediction horizons\. We ask whether relative algorithm comparisons transfer across task families and datasets, whether residual task heterogeneity contains useful methodological structure, and what a controlled comparison reveals about progress over the last decade\. Aggregate pairwise comparisons transfer strongly across evaluation settings, including from randomly generated to clinically meaningful tasks and across datasets\. At the same time, clinically meaningful tasks exhibit greater task\-algorithm interaction, providing preliminary evidence that task properties can help explain when particular modeling choices are advantageous\. Finally, newer algorithms do not consistently outperform earlier approaches: gradient\-boosted trees remain highly competitive when paired with a high\-capacity, wide and sparse representation of the EHR\. Together, these results suggest that useful methodological knowledge may require less task engineering than commonly assumed, while highlighting the need to understand the structured heterogeneity that remains across tasks and algorithms\.

††workshop:Machine Learning for Health \(ML4H\) 2026††proceedings:: Submitted to ML4H 2026:\\mlhtrackname††workshop:Machine Learning for Health \(ML4H\) 2026###### keywords

EHR, Evaluation, Health AI

## 1Introduction

Artificial Intelligence \(AI\) over structured EHR data promises to revolutionize patient care\[[30](https://arxiv.org/html/2609.18134#bib.bib3),[2](https://arxiv.org/html/2609.18134#bib.bib33),[27](https://arxiv.org/html/2609.18134#bib.bib34),[29](https://arxiv.org/html/2609.18134#bib.bib35)\]and is receiving increasing attention in both the scientific community and the media\[[32](https://arxiv.org/html/2609.18134#bib.bib4),[21](https://arxiv.org/html/2609.18134#bib.bib5)\]\. Despite this, many researchers remain concerned about the extent of methodological progress and rigor in this sub\-field\[[18](https://arxiv.org/html/2609.18134#bib.bib6),[1](https://arxiv.org/html/2609.18134#bib.bib7)\], arguing that the wealth of research in this space has not meaningfully enabled us to answer key methodological questions, such as when and why some algorithms work better than others on a given dataset and task\[[8](https://arxiv.org/html/2609.18134#bib.bib8)\]\. Commentators have identified many possible barriers to rigorous methodological progress, including: \(i\) the difficulty of defining sufficiently clinically meaningful evaluation tasks to characterize AI algorithms\[[1](https://arxiv.org/html/2609.18134#bib.bib7),[20](https://arxiv.org/html/2609.18134#bib.bib12)\]and \(ii\) the longstanding reproducibility and transportability crisis that hinders replicating task definitions or re\-using model code across studies\[[16](https://arxiv.org/html/2609.18134#bib.bib11),[11](https://arxiv.org/html/2609.18134#bib.bib13),[18](https://arxiv.org/html/2609.18134#bib.bib6)\]\.

While there is empirical evidence that the reproducibility crisis hinders the accumulation of methodological knowledge–for example,[Johnson et al\.](https://arxiv.org/html/2609.18134#bib.bib13)found that the reported performance gain of neural network algorithms over gradient\-boosted trees often reversed upon reproduction\[[11](https://arxiv.org/html/2609.18134#bib.bib13)\]–barrier \(i\) has received very limited empirical characterization\. Concretely, we do not know whether relative comparisons between algorithms \(e\.g\., does algorithm 1 outperform algorithm 2 when trained from scratch on a given dataset and task\) identified on clinically “meaningless” tasks would agree with the same comparisons on high\-quality, expert\-determined tasks\. If the conclusions drawn in these two settings were highly similar–and a growing body of evidence on evaluation in other domains of machine learning suggests this is plausible\[[26](https://arxiv.org/html/2609.18134#bib.bib17)\]–methodological progress would be more accessible in this domain than previously expected, opening new routes to robust insight into the relative performance of different algorithms across diverse clinical settings\.

In this work, we empirically study this question, and methodological progress in health AI more generally, via a shared implementation and evaluation framework\. We re\-implement111As our models are reproductions, our findings do not imply that the original authors’ results were invalid; our reproduction of any given model may itself have errors or flaws\. Authors of re\-implemented models were not consulted\.historical and recent ICU and EHR models using a common MEDS\-based interface\[[15](https://arxiv.org/html/2609.18134#bib.bib14)\], and evaluate them on two public clinical datasets, MIMIC\-IV\[[10](https://arxiv.org/html/2609.18134#bib.bib1)\]and NWICU\[[19](https://arxiv.org/html/2609.18134#bib.bib2)\], across both clinically meaningful tasks–explicitly authored by experts to capture meaningful predictions–and randomly generated tasks–which ask whether a randomly sampled code from the dataset vocabulary will occur within a randomly sampled duration, and require no clinical curation\. We use our findings to answer three key questions:Q1:Are rankings and comparisons between algorithms consistent across evaluation settings, such as across datasets or between clinically meaningful and randomly generated tasks?Q2:Beyond global ranking consistency, is there evidence of shared structure between task, algorithm, and dataset properties that could tell us when a given algorithm is likely to work well in a target setting?Q3:When algorithms developed over the last decade are evaluated in a common implementation, training, and evaluation framework, what evidence of methodological progress emerges, if any?

Our analyses suggest that: \(1\) comparative conclusions about algorithm performance are statistically significantly similar across evaluation settings, particularly between clinically meaningful and randomly generated tasks, suggesting that barrier \(i\) may not be a major barrier for methodological research; \(2\) there is preliminary but meaningful evidence of shared structure between tasks, datasets, and algorithms,*particularly on clinically meaningful tasks*, that is suggestive of future opportunities for progress; and \(3\) there is meaningful evidence of methodological progress, though not in directions aligned with typical AI development: the best\-performing algorithm in our comparison is a gradient\-boosted tree model, but only when paired with an extremely wide, sparse tabularization of the full feature space rather than the restricted, clinically motivated featurizations that dominated historical uses of such models\.

In sum, we make the following key contributions:

1. 1\.We provide reproducible, standard\-interface reimplementations of twelve AI algorithms that can be applied across MEDS datasets through a common interface, enabling robust baseline comparisons in future methodological research\.
2. 2\.We provide a rich set of comparisons of algorithm performance, in both discriminative and computational terms, across a battery of tasks and two datasets, offering concrete guidance on which algorithms are likely to perform best and at what computational cost\.
3. 3\.We show that randomly generated tasks provide highly reliable estimates of relative algorithm performance, even across datasets, suggesting a more accessible route to robust methodological research in EHR AI\.
4. 4\.We find preliminary evidence that relating task properties to algorithm and dataset properties yields further methodological insight, suggesting that relative algorithm rankings are partly predictable from characteristics of the evaluation setting\.

## 2Methods

To answer the three key questions introduced in Section[1](https://arxiv.org/html/2609.18134#S1), we use the following high\-level experimental process\. First, we re\-implement a variety of historically published algorithms such that they are trainable for any binary classification task on any MEDS dataset through a consistent interface, but are otherwise faithful to their original, published form\. We then train these aligned algorithms on a collection of target tasks from two families: expert\-defined clinically meaningful tasks and randomly generated tasks, which are randomly defined and require no clinical expertise\. We evaluate the trained algorithms by per\-task AUROC on held\-out subjects\. This process is replicated over two publicly available longitudinal critical care EHR datasets, MIMIC\-IV and NWICU, which differ substantially in scale and clinical scope \(Table[3](https://arxiv.org/html/2609.18134#A3.T3)in Appendix[C](https://arxiv.org/html/2609.18134#A3)\)\. With these evaluation results, we answer our questions directly: for Q1, we compute how consistent the AUROC ranking of these algorithms is across task families and datasets; for Q2, we assess to what extent variance in algorithm rankings is related to structured properties of the task, dataset, and algorithm combination; and for Q3, we assess how the best overall performance has changed with the first public artifact date of these algorithms and examine other properties differentiating high\- from low\-performing algorithms\. The statistical procedures used throughout \(ranking\-agreement tests, task\-level reversal rates, variance decomposition, and uncertainty quantification\) are detailed in Appendix[B\.1](https://arxiv.org/html/2609.18134#A2.SS1)\.

### 2\.1Models

#### Model selection\.

Five model families were selected arbitrarily from a set of 49 ICU and EHR modeling approaches identified through a literature review \(Appendix[A](https://arxiv.org/html/2609.18134#A1)\)\. We deliberately added several models and baselines to anchor the comparison with recent pretraining approaches\. MEDS\-Tab\[[22](https://arxiv.org/html/2609.18134#bib.bib18)\]was included as a strong tabular baseline, using both its XGBoost\-based configurations and an additional logistic\-regression variant using the same MEDS\-Tab featurization \(MEDS\-Tab\-LR\)\. We also included the XGBoost model from the mortality reproducibility study of\[[11](https://arxiv.org/html/2609.18134#bib.bib13)\]\(ICU\-XGBoost\) as a historically motivated ICU baseline\. Finally, MOTOR\[[28](https://arxiv.org/html/2609.18134#bib.bib19)\]and MEDS\-EIC\-AR\[[17](https://arxiv.org/html/2609.18134#bib.bib20)\]were added as potentially strong recent competitors\. Table[1](https://arxiv.org/html/2609.18134#S2.T1)summarizes all evaluated model variants\.

#### Model implementation\.

All twelve models were re\-implemented using agentic generative AI coding workflows \(Claude Code with Claude Opus 5; see Appendix[E](https://arxiv.org/html/2609.18134#A5)\), with one agent assigned to each model and the implementation template, verification procedure, and human review held fixed across models\. Each model therefore received roughly equal ‘‘expert attention’’ during implementation, rather than effort concentrating on any single model or family\. To reduce implementation variability and the risk that obvious re\-implementation errors affect comparisons, we used a fixed model interface and verification workflow from a common template repository222Will be released upon publication\.\. For each model, we first inspected the original paper, supplementary material, source code, configurations, preprocessing, and evaluation code, and recorded which components were ported, adapted, or omitted\. Implementations then passed a staged set of tests\. First, on a small synthetic dataset, we constructed a controlled prediction problem in which a single event code determined a binary label without sequence length or event position leaking the outcome, verified that the model recovered this signal through its actual preprocessing and featurization pipeline, and used shuffled labels as a negative control\. Local end\-to\-end tests also checked that the complete workflow produced predictions for exactly the requested examples and splits\. Second, models were executed through the isolated MEDS\-DEV interface to verify installation, configuration, and output compatibility\. Third, they were run on a MIMIC\-IV demo task to expose data\-dependent, serialization, resource, and scale\-related failures before full\-scale experiments\. For models requiring specific input features, we supplied dataset\-specific predicate files mapping available MEDS events to the required features as closely as possible\. Full\-data experiments were then run from fixed, clean commits using the same task bundles across models, recording code revision, configuration, dataset, task, and resource information for reproducibility\.

Table 1:Models considered in our evaluation, ordered by the date of their first publicly available artifact\.

### 2\.2Tasks

All tasks are binary prediction problems\. For each dataset we use 10 clinically meaningful and 10 randomly generated tasks, yielding four evaluation settings \(dataset×\\timestask family\) across which the evaluated algorithms and evaluation procedure are held fixed\. Full task definitions, cohort sizes, prevalences, and dataset\-specific differences are provided in Appendix[D](https://arxiv.org/html/2609.18134#A4)\.

#### Randomly generated tasks\.

Randomly generated tasks are defined by an event code and a prediction horizon, inspired by the EveryQuery task parametrization\[[3](https://arxiv.org/html/2609.18134#bib.bib25)\]\. For each task, prediction times are sampled from eligible patient timelines, and the label indicates whether the event occurs within the horizon after the prediction time\. Candidate tasks are retained only when they satisfy statistical adequacy and censoring requirements; in particular, we requiremin⁡\(npos,nneg\)≥150\\min\(n\_\{\\mathrm\{pos\}\},n\_\{\\mathrm\{neg\}\}\)\\geq 150\.333See Appendix[D\.2](https://arxiv.org/html/2609.18134#A4.SS2)for the derivation of the minority\-class floor and its relationship to AUROC estimation error\.Among eligible event\-code–horizon pairs, tasks are sampled subject to diversity constraints limiting repeated codes and over\-representation of individual event categories\. Ten tasks are sampled independently per dataset, so the MIMIC\-IV and NWICU randomly generated task sets are intentionally different\.

#### Clinically meaningful tasks\.

In parallel, we define clinically motivated tasks using ACES\[[34](https://arxiv.org/html/2609.18134#bib.bib26)\], with each task specified by a clinical trigger, prediction\-time anchor, prediction horizon, and, when appropriate, an exclusion criterion for patients already in the target state\. We sought to make the MIMIC\-IV and NWICU suites as similar as possible, adapting task definitions across datasets when the required information was available, and to cover diverse clinical specialties, prediction horizons, and question types, including laboratory abnormalities, acute organ dysfunction, interventions, utilization, and mortality\. Exact correspondence was not required for our main methodological question, however, and some tasks necessarily differ because the available events and predicates differ across datasets; we therefore retain dataset\-specific clinically meaningful tasks when a direct adaptation is not reliable\. Because these more constrained definitions often yield smaller cohorts, we use a reporting floor ofmin⁡\(npos,nneg\)≥40\\min\(n\_\{\\mathrm\{pos\}\},n\_\{\\mathrm\{neg\}\}\)\\geq 40and exclude tasks below it\.

## 3Results

### 3\.1Q1: Comparative conclusions generally transfer across settings

Our experiments reliably show that comparative conclusions across algorithms largely transfer from randomly generated tasks to clinically meaningful tasks and even across datasets, suggesting that intensive efforts to ensure that benchmarking tasks are highly clinically meaningful may not be as essential for methodological research as previously thought\. Figure[1](https://arxiv.org/html/2609.18134#S3.F1)shows this visually: panels A–D plot the held\-out AUROC of each algorithm on every task in each of our four evaluation settings, with algorithms colored by global macro\-AUROC on a consistent gradient, so the preservation of this gradient across tasks and settings shows directly that algorithm rankings are quite consistent\. Full numerical results are available in Appendix[F](https://arxiv.org/html/2609.18134#A6)\.

To quantify this finding formally, Figure[2](https://arxiv.org/html/2609.18134#S3.F2)shows the ranking agreement between every pair of evaluation settings \(Appendix[B\.1](https://arxiv.org/html/2609.18134#A2.SS1)\)\. Agreement ranges from 76% to 92% and is statistically significantly higher than chance in all six comparisons after Holm correction; it is strongest between MIMIC\-IV and NWICU randomly generated tasks \(92%\) and between MIMIC\-IV randomly generated and clinically meaningful tasks \(91%\)\. Figure[1](https://arxiv.org/html/2609.18134#S3.F1)E further shows that agreement is concentrated at the top and bottom of the ranking, with more variability in the middle: MEDS\-Tab\-large ranks first in every setting, MEDS\-EIC\-AR ranks second in three settings and third in the fourth by less than 0\.008 AUROC, and the same three algorithms \(TECO, MEDS\-Tab\-LR, and BEHRT\) form the bottom three in every setting\.

These results suggest that randomly generated tasks, which require no clinical curation, can serve as an efficient first\-pass substrate for determining which modeling choices tend to perform better, and that additional datasets need not yield entirely new comparative conclusions\. Clinically meaningful tasks nonetheless remain essential when the question shifts from broad algorithm comparison toward task\-specific or deployment\-oriented evaluation\[[25](https://arxiv.org/html/2609.18134#bib.bib9),[35](https://arxiv.org/html/2609.18134#bib.bib10)\], and with only two datasets our cross\-dataset conclusions remain preliminary\.

Figure 1:Task\-level performance and ranking transfer across settings\. A–D: tasks ordered from harder to easier by mean AUC across models\. E: model ranks by setting\-level mean AUC \(1 = best\), ordered MG, MC, NC, NG\. Colors identify the same models throughout; lines connect tasks or settings as visual guides\.![Refer to caption](https://arxiv.org/html/2609.18134v1/q1e_consistency_matrix_outline.png)Figure 2:Ranking consistency across evaluation settings\. Off\-diagonal cells show agreement between mean\-AUROC algorithm rankings; italic diagonal entries show one minus the within\-setting task reversal rate\. Stars mark above\-chance agreement \(one\-sided random\-ranking tests, Holm\-adjustedp<0\.05p<0\.05across six setting pairs\)\. MG/MC denote MIMIC\-IV Generated/Clinical; NC/NG denote NWICU Clinical/Generated\. Confidence intervals and Kendallτ\\taufor comparisons against MG are available in Appendix[I](https://arxiv.org/html/2609.18134#A9)\.
### 3\.2Q2: Dataset\-task\-algorithm interactions provide preliminary evidence of shared structure

While Section[3\.1](https://arxiv.org/html/2609.18134#S3.SS1)shows that comparative conclusions are globally robust across settings, our experiments are also strongly suggestive of a shared, underlying structure in task\-, dataset\-, and algorithm\-performance relationships\. This structure matters because it is precisely the kind of methodological knowledge we seek to uncover: not only which algorithm performs best on average, but which design choices are advantageous under which conditions\. Our analyses find preliminary evidence of this shared structure through two vehicles: greater algorithm–task interaction structure among clinically meaningful than randomly generated tasks, and relationships between algorithm design and task properties that are associated with comparative performance\.

#### Increased heterogeneity and algorithm–task interaction structure\.

We find three complementary, interrelated sources of evidence suggesting that the heterogeneity remaining in algorithm rankings across tasks is driven by algorithm–task relationships, especially for clinically meaningful tasks\. First, and most directly, the within\-setting ranking consistency shown on the diagonal of Figure[2](https://arxiv.org/html/2609.18134#S3.F2)\(the proportion of task\-level pairwise algorithm comparisons that agree with the setting\-level aggregate comparison\) is consistently lower for clinically meaningful tasks than for randomly generated tasks: 15\.2% and 16\.7% of task\-level comparisons reverse the aggregate ordering on MIMIC\-IV and NWICU clinically meaningful tasks, versus 8\.3% and 10\.9% on randomly generated tasks\. Second, a variance decomposition \(Appendix[I](https://arxiv.org/html/2609.18134#A9)\) estimating how much observed AUROC variance is due to algorithms alone, tasks alone, or their interaction attributes roughly 24% of variance to interaction terms for clinically meaningful tasks, versus below 10% for randomly generated tasks\. Third, comparing per\-task algorithm rankings between MIMIC\-IV and NWICU across six clinical task concepts present in both datasets \(Appendix[D\.3](https://arxiv.org/html/2609.18134#A4.SS3)\), rankings are statistically significantly more similar when the task has the same conceptual meaning in both datasets than when it does not \(mean Kendall’sτb\\tau\_\{b\}of 0\.601 versus 0\.437; exact permutationp=0\.019p=0\.019\), suggesting that a task’s conceptual identity is associated with systematic algorithm\-performance differences that persist across datasets, beyond properties of the algorithm or dataset alone\.

#### Evidence of relationships between algorithm design principles and task properties\.

We can also test whether hypothesized links between algorithm design principles and performance hold in real experiments\. Specifically, we consider whether the form of the pre\-training loss in our two highest\-performing neural algorithms \(fine\-tuned MOTOR and MEDS\-EIC\-AR\) is related to their relative performance\. MEDS\-EIC\-AR is pre\-trained for next\-token prediction, by design a very short\-horizon task, whereas MOTOR is pre\-trained to estimate time\-to\-next\-occurrence distributions over a panel of codes, a much longer\-horizon task\. We might therefore hypothesize that MOTOR\-FT’s advantage over MEDS\-EIC\-AR should grow with the task’s prediction horizon, which pushes the task farther from MEDS\-EIC\-AR’s pre\-training loss relative to MOTOR’s\. Across randomly generated tasks on our two datasets,444We use only randomly generated tasks because prediction horizon is an explicitly sampled task parameter in this family, providing broad horizon variation without tying each horizon to a clinically authored endpoint\.we find a statistically significant relationship supporting this hypothesis on NWICU \(Spearmanρ=0\.795\\rho=0\.795,pHolm=0\.016p\_\{\\mathrm\{Holm\}\}=0\.016\) and a non\-significant but directionally aligned correlation on MIMIC\-IV \(ρ=0\.483\\rho=0\.483,pHolm=0\.16p\_\{\\mathrm\{Holm\}\}=0\.16\), providing preliminary, partial evidence\. A similar hypothesis applies to the underperformance of BEHRT, which by design ingests only relatively low\-frequency input features such as diagnoses and should therefore perform better on longer\-horizon tasks, where such features may play a more dominant role\. We observe a statistically significant relationship consistent with this hypothesis on MIMIC\-IV \(ρ=0\.805\\rho=0\.805,pHolm=0\.015p\_\{\\mathrm\{Holm\}\}=0\.015\) and a non\-significant correlation on NWICU \(ρ=−0\.135\\rho=\-0\.135,pHolm=0\.71p\_\{\\mathrm\{Holm\}\}=0\.71\)\. Full tables for these and further exploratory analyses are available in Appendix[I](https://arxiv.org/html/2609.18134#A9)\.

### 3\.3Q3: Methodological progress can arise from new uses of existing techniques, not only new architectures

Our controlled comparison shows mixed evidence of methodological progress over the last decade, and that progress is poorly summarized by the recency of model architectures\. Most strikingly, the highest\-performing pipeline we evaluate uses gradient\-boosted trees, a technology available for roughly the entire period we study, but obtains its performance through a substantially newer way of representing and exposing longitudinal EHR data to that learner\.

Concretely, Figure[3](https://arxiv.org/html/2609.18134#S3.F3)shows each algorithm’s mean AUROC against the date at which, to the best of our knowledge, it was first released or published\. MEDS\-Tab\-large, an XGBoost\-derived algorithm\[[4](https://arxiv.org/html/2609.18134#bib.bib16)\], obtains best\-in\-class performance but is anchored to 2024, when MEDS\-Tab was first released\. This seems at first glance contradictory, given that XGBoost models were in use in this space well before 2024; however, MEDS\-Tab\-large uses XGBoost in a much higher\-capacity manner than many prior XGBoost baselines\. For example, ICU\-XGBoost, which substantially underperforms both MEDS\-Tab\-large and recent neural algorithms, uses a more historically typical formulation: a relatively small set of clinically curated features modeled through XGBoost\. In contrast, MEDS\-Tab\-large tabularizes the raw clinical data at an extremely wide, sparse scale, producing tens to hundreds of thousands of features that summarize every code in the dataset vocabulary over diverse aggregation functions and lookback windows\. This added representational capacity, coupled with the advantages of the XGBoost model class, yields its performance relative to other algorithms\.

Consistent with this usage, MEDS\-Tab\-large was among the most time\-intensive algorithms to train, as seen in Appendix[H](https://arxiv.org/html/2609.18134#A8), Figure[5](https://arxiv.org/html/2609.18134#A8.F5), which shows the Pareto frontier between total runtime \(preprocessing, amortized pre\-training where applicable, and training\) and overall performance\. There, MEDS\-Tab\-large occupies the end of the frontier that maximizes performance at high cost, while, contrary to typical expectations, neural algorithms such as MEDS\-EIC\-AR offer competitive performance at greatly reduced wall\-time cost, a benefit to efficiency rather than predictive power\.

Figure 3:Retrospective progress by model\-family artifact year\. Each point is one model’s mean AUROC across the 40 tasks in all four settings; the dashed step curve marks the best mean AUROC among models available in that year or earlier\. Colors and abbreviations match Figure[1](https://arxiv.org/html/2609.18134#S3.F1)\. Artifact years denote first public availability of the model family; scores are from the present evaluation and do not establish historical state of the art\.

## 4Discussion

### 4\.1Implications for methodological research in Health AI

Taken together, Q1 and Q2 suggest a two\-stage evaluation strategy for methodological research in Health AI: first estimate the broadly stable component of algorithm performance, which transfers across task families and datasets and can therefore be measured cheaply, and then study systematic departures from it, which is where task\-, dataset\-, and design\-specific methodological knowledge lives\. Under this view, the question to ask of a new algorithm is not only whether it improves a particular benchmark, but whether that improvement transfers, where it fails, and what those failures teach us about which algorithms work where\.

A natural extension is to characterize the task space itself\. With more tasks and datasets, one could ask whether subsets of tasks induce consistently similar algorithm rankings, whether such subsets can be predicted from observable properties such as horizon, prevalence, clinical category, or cohort size, and whether algorithm characteristics explain systematic departures from the aggregate ranking\. Such analyses could turn task heterogeneity from a nuisance into a source of methodological insight\.

Finally, our experience suggests that recent tooling can considerably lower the historical barriers to comparative research in Health AI\[[16](https://arxiv.org/html/2609.18134#bib.bib11),[6](https://arxiv.org/html/2609.18134#bib.bib28)\]\. Agentic generative AI made it feasible to reconstruct multiple heterogeneous model pipelines under a common interface with comparable effort per model, while data standards and shared tooling—MEDS for event\-stream representation\[[15](https://arxiv.org/html/2609.18134#bib.bib14)\], MEDS\-DEV for decentralized model execution and comparison\[[12](https://arxiv.org/html/2609.18134#bib.bib15)\], and ACES for portable task specification\[[34](https://arxiv.org/html/2609.18134#bib.bib26)\]—make such comparisons cumulative rather than one\-off\. Standardized re\-implementation is a different objective from exact reproduction of an original result, and our framework is designed for the former\. We intend to contribute the re\-implemented algorithms back to MEDS\-DEV so that future work can extend rather than recreate these comparisons\.

### 4\.2Limitations and future work

Several limitations constrain the scope of our conclusions\. First, we evaluate only two datasets, both drawn from structured hospital and ICU EHR data; extending the study to additional public datasets such as eICU and EHRSHOT, and to substantially different clinical contexts, would provide a stronger test of dataset invariance\. Second, our algorithm set is a subset of the much larger Health AI literature, and extending the common interface to the remaining candidate algorithms would reduce sensitivity to the particular algorithms sampled here\. Third, each evaluation setting contains only ten tasks, which is sufficient to identify aggregate transfer and task\-level heterogeneity but too small to characterize a latent task space; multiple independently sampled randomly generated suites and larger clinically meaningful collections would enable sensitivity analyses with respect to task count\. Fourth, clinically meaningful task definitions cannot always be transferred identically across datasets, so some observed cross\-dataset differences may conflate dataset and task\-definition changes\. Fifth, all re\-implementations are best\-effort translations of the original algorithms: our common template, behavioral tests, integration tests, and human review reduce the risk of obvious implementation errors but cannot guarantee equivalence to the original pipelines, so our conclusions concern the algorithms as instantiated within our shared framework rather than the original studies\. Finally, our uncertainty analysis does not capture all sources of variation: we do not systematically repeat model training across random seeds, and our primary analyses focus on AUROC rather than other metrics such as win rate\.

## 5Related work

Prior work on barriers to cumulative methodological progress in Health AI can be organized around three recurring themes: access to and suitability of clinical data\[[20](https://arxiv.org/html/2609.18134#bib.bib12),[35](https://arxiv.org/html/2609.18134#bib.bib10)\], the formulation of meaningful and comparable evaluation tasks\[[11](https://arxiv.org/html/2609.18134#bib.bib13),[1](https://arxiv.org/html/2609.18134#bib.bib7)\], and the reproducibility and comparability of experimental results\[[16](https://arxiv.org/html/2609.18134#bib.bib11)\]\. Donoho’s account of frictionless reproducibility identifies complementary ingredients for such cumulative comparison: accessible data, re\-executable workflows, and shared challenge problems with explicit performance criteria\[[6](https://arxiv.org/html/2609.18134#bib.bib28)\]\. Shared datasets alone, however, do not guarantee comparability\. In a mortality\-prediction case study,[Johnson et al\. \[11\]](https://arxiv.org/html/2609.18134#bib.bib13)documented substantial heterogeneity in cohort construction and task specification, motivating shared code, benchmarks, and common data\-extraction procedures\.[Harutyunyan et al\. \[9\]](https://arxiv.org/html/2609.18134#bib.bib29)subsequently introduced four standardized clinical prediction tasks on MIMIC\-III, while MIMIC\-Extract\[[31](https://arxiv.org/html/2609.18134#bib.bib32)\]further standardized data extraction and preprocessing for reproducible EHR modeling\. Yet[Bellamy et al\. \[1\]](https://arxiv.org/html/2609.18134#bib.bib7)found that only a small fraction of subsequent studies citing these benchmarks used sufficiently similar experimental setups for direct comparison, so benchmark availability and citation alone do not ensure consistent use\.

More recent benchmarks have expanded both the scope of tasks and the models being evaluated\. EHRSHOT\[[33](https://arxiv.org/html/2609.18134#bib.bib31)\]defines 15 prediction tasks over longitudinal EHR data for few\-shot evaluation of pretrained models, moving beyond ICU\-only evaluation\. FoMoH\[[23](https://arxiv.org/html/2609.18134#bib.bib30)\]similarly evaluates structured\-EHR foundation models across a diverse suite of clinically meaningful prediction tasks, emphasizing clinical relevance and cross\-model comparability\. These efforts provide increasingly rich and standardized evaluation settings, but necessarily instantiate particular choices of datasets, cohorts, and tasks\. Our work is complementary: rather than proposing another fixed benchmark, we ask how much the methodological conclusions obtained from an evaluation depend on those choices themselves\.

## References

- \[1\]D\. Bellamy, L\. Celi, and A\. L\. Beam\(2020\)Evaluating progress on machine learning for longitudinal electronic healthcare data\.arXiv preprint arXiv:2010\.01149\.External Links:[Link](https://arxiv.org/abs/2010.01149)Cited by:[§1](https://arxiv.org/html/2609.18134#S1.p1.1),[§5](https://arxiv.org/html/2609.18134#S5.p1.1)\.
- \[2\]A\. Boussina, S\. P\. Shashikumar, A\. Malhotra, R\. L\. Owens, R\. El\-Kareh, C\. A\. Longhurst, K\. Quintero, A\. Donahue, T\. C\. Chan, S\. Nemati, and G\. Wardi\(2024\)Impact of a deep learning sepsis prediction model on quality of care and survival\.npj Digital Medicine7,pp\. 14\.External Links:[Document](https://dx.doi.org/10.1038/s41746-023-00986-6)Cited by:[§1](https://arxiv.org/html/2609.18134#S1.p1.1)\.
- \[3\]P\. Chandak, G\. Kondas, L\. Antwarg Friedman, I\. Kohane, and M\. McDermott\(2026\)EveryQuery: zero\-shot clinical prediction via task\-conditioned pretraining over electronic health records\.External Links:2603\.07900,[Link](https://arxiv.org/abs/2603.07900)Cited by:[§2\.2](https://arxiv.org/html/2609.18134#S2.SS2.SSS0.Px1.p1.1)\.
- \[4\]T\. Chen and C\. Guestrin\(2016\)XGBoost: a scalable tree boosting system\.InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,pp\. 785–794\.External Links:[Document](https://dx.doi.org/10.1145/2939672.2939785)Cited by:[§3\.3](https://arxiv.org/html/2609.18134#S3.SS3.p2.1)\.
- \[5\]E\. Choi, M\. T\. Bahadori, J\. A\. Kulas, A\. Schuetz, W\. F\. Stewart, and J\. Sun\(2016\)RETAIN: an interpretable predictive model for healthcare using reverse time attention mechanism\.arXiv preprint arXiv:1608\.05745\.Cited by:[Table 1](https://arxiv.org/html/2609.18134#S2.T1.2.2.1)\.
- \[6\]D\. Donoho\(2024\)Data science at the singularity\.Harvard Data Science Review6\(1\)\.External Links:[Document](https://dx.doi.org/10.1162/99608f92.b91339ef),[Link](https://hdsr.mitpress.mit.edu/pub/g9mau4m0/release/2)Cited by:[§4\.1](https://arxiv.org/html/2609.18134#S4.SS1.p3.1),[§5](https://arxiv.org/html/2609.18134#S5.p1.1)\.
- \[7\]J\. A\. Hanley and B\. J\. McNeil\(1982\)The meaning and use of the area under a receiver operating characteristic \(ROC\) curve\.Radiology143\(1\),pp\. 29–36\.External Links:[Document](https://dx.doi.org/10.1148/radiology.143.1.7063747)Cited by:[§D\.2](https://arxiv.org/html/2609.18134#A4.SS2.p1.1)\.
- \[8\]M\. Hardt and B\. Recht\(2022\)Patterns, predictions, and actions: foundations of machine learning\.Princeton University Press\.External Links:ISBN 9780691233734Cited by:[§1](https://arxiv.org/html/2609.18134#S1.p1.1)\.
- \[9\]H\. Harutyunyan, H\. Khachatrian, D\. C\. Kale, G\. Ver Steeg, and A\. Galstyan\(2019\)Multitask learning and benchmarking with clinical time series data\.Scientific Data6\(1\),pp\. 96\.External Links:[Document](https://dx.doi.org/10.1038/s41597-019-0103-9),[Link](https://doi.org/10.1038/s41597-019-0103-9)Cited by:[§5](https://arxiv.org/html/2609.18134#S5.p1.1)\.
- \[10\]A\. Johnson, L\. Bulgarelli, T\. Pollard, B\. Gow, B\. Moody, S\. Horng, L\. A\. Celi, and R\. Mark\(2024\)MIMIC\-IV\.PhysioNet\.Note:Version 3\.1External Links:[Document](https://dx.doi.org/10.13026/kpb9-mt58),[Link](https://doi.org/10.13026/kpb9-mt58)Cited by:[§1](https://arxiv.org/html/2609.18134#S1.p3.1)\.
- \[11\]A\. E\. W\. Johnson, T\. J\. Pollard, and R\. G\. Mark\(2017\)Reproducibility in critical care: a mortality prediction case study\.InProceedings of the 2nd Machine Learning for Healthcare Conference,F\. Doshi\-Velez, J\. Fackler, D\. Kale, R\. Ranganath, B\. Wallace, and J\. Wiens \(Eds\.\),Proceedings of Machine Learning Research, Vol\.68,pp\. 361–376\.External Links:[Link](https://proceedings.mlr.press/v68/johnson17a.html)Cited by:[§1](https://arxiv.org/html/2609.18134#S1.p1.1),[§1](https://arxiv.org/html/2609.18134#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.18134#S2.SS1.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2609.18134#S2.T1.2.3.1),[§5](https://arxiv.org/html/2609.18134#S5.p1.1)\.
- \[12\]A\. Kolo, C\. Pang, E\. Choi, E\. Steinberg, H\. Jeong, J\. Gallifant, J\. A\. Fries, J\. N\. Chiang, J\. Oh, J\. Xu, K\. Stankevičiūtė, K\. V\. Klein, M\. B\. A\. McDermott, M\. Odgaard, N\. Oufattole, N\. H\. Shah, P\. Rockenschaub, P\. Renc, R\. P\. van de Water, S\. Joshi, S\. A\. Lee, T\. S\. Bergamaschi, T\. J\. Pollard, V\. Jeanselme, Y\. S\. Choi, M\. Wornow, A\. Kashyap, X\. Jiang, Y\. Li, Y\. Kobayashi, and R\. C\. King\(2024\)MEDS decentralized, extensible validation \(MEDS\-DEV\) benchmark: establishing reproducibility and comparability in ml for health\.InMachine Learning for Health \(ML4H\) 2024 Demo Track,Cited by:[§4\.1](https://arxiv.org/html/2609.18134#S4.SS1.p3.1)\.
- \[13\]A\. Labach, A\. Pokhrel, X\. S\. Huang, S\. Zuberi, S\. E\. Yi, M\. Volkovs, T\. Poutanen, and R\. G\. Krishnan\(2023\)DuETT: dual event time transformer for electronic health records\.arXiv preprint arXiv:2304\.13017\.Cited by:[Table 1](https://arxiv.org/html/2609.18134#S2.T1.2.9.1)\.
- \[14\]Y\. Li, S\. Rao, J\. R\. Ayala Solares, A\. Hassaine, D\. Canoy, Y\. Zhu, K\. Rahimi, and G\. Salimi\-Khorshidi\(2019\)BEHRT: transformer for electronic health records\.arXiv preprint arXiv:1907\.09538\.Cited by:[Table 1](https://arxiv.org/html/2609.18134#S2.T1.2.4.1)\.
- \[15\]M\. B\. A\. McDermott, E\. Steinberg, J\. A\. Fries, R\. P\. van de Water, C\. Pang, P\. Rockenschaub, P\. Renc, J\. Oh, K\. Stankevičiūtė, J\. Xu, T\. J\. Pollard, N\. Oufattole, M\. Wornow, T\. S\. Bergamaschi, H\. Jeong, S\. A\. Lee, V\. Jeanselme, K\. V\. Klein, M\. Odgaard, M\. E\. Montgomery, A\. Sitek, M\. Nielsen, J\. N\. Chiang, N\. Dagan, I\. Kohane, S\. Joshi, E\. Choi, and N\. H\. Shah\(2026\)MEDS—an emerging data standard and ecosystem for health ai research\.NEJM AI3\(6\)\.External Links:[Document](https://dx.doi.org/10.1056/AIra2501253)Cited by:[§1](https://arxiv.org/html/2609.18134#S1.p3.1),[§4\.1](https://arxiv.org/html/2609.18134#S4.SS1.p3.1)\.
- \[16\]M\. B\. A\. McDermott, S\. Wang, N\. Marinsek, R\. Ranganath, L\. Foschini, and M\. Ghassemi\(2021\)Reproducibility in machine learning for health research: still a ways to go\.Science Translational Medicine13\(586\),pp\. eabb1655\.External Links:[Document](https://dx.doi.org/10.1126/scitranslmed.abb1655)Cited by:[§1](https://arxiv.org/html/2609.18134#S1.p1.1),[§4\.1](https://arxiv.org/html/2609.18134#S4.SS1.p3.1),[§5](https://arxiv.org/html/2609.18134#S5.p1.1)\.
- \[17\]MEDS “everything\-is\-code” autoregressive modelExternal Links:[Document](https://dx.doi.org/10.5281/zenodo.17535559),[Link](https://github.com/mmcdermott/MEDS_EIC_AR)Cited by:[§2\.1](https://arxiv.org/html/2609.18134#S2.SS1.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2609.18134#S2.T1.2.14.1),[Table 1](https://arxiv.org/html/2609.18134#S2.T1.2.16.1)\.
- \[18\]M\. McDermott\(2025\)The \(lack of?\) science of machine learning for healthcare\.InProceedings of the 4th Machine Learning for Health Symposium,S\. Hegselmann, H\. Zhou, E\. Healey, T\. Chang, C\. Ellington, V\. Mhasawade, S\. Tonekaboni, P\. Argaw, and H\. Zhang \(Eds\.\),Proceedings of Machine Learning Research, Vol\.259,pp\. 19–29\.External Links:[Link](https://proceedings.mlr.press/v259/mcdermott25a.html)Cited by:[§1](https://arxiv.org/html/2609.18134#S1.p1.1)\.
- \[19\]D\. Moukheiber, W\. Temps, B\. Molgi, Y\. Li, A\. Lu, P\. Nannapaneni, A\. Chahin, S\. Hao, F\. Torres Fabregas, L\. A\. Celi, A\. Wong, M\. Lloyd, X\. Borrat Frigola, H\. Lee, D\. Schneider, T\. Pollard, Y\. Luo, A\. Kho, and R\. Mark\(2024\)Northwestern ICU \(NWICU\) database\.PhysioNet\.Note:Version 0\.1\.0External Links:[Document](https://dx.doi.org/10.13026/s84w-1829),[Link](https://doi.org/10.13026/s84w-1829)Cited by:[§1](https://arxiv.org/html/2609.18134#S1.p3.1)\.
- \[20\]S\. Mullainathan and Z\. Obermeyer\(2022\)Solving medicine’s data bottleneck: nightingale open science\.Nature Medicine28\(5\),pp\. 897–899\.External Links:[Document](https://dx.doi.org/10.1038/s41591-022-01804-4),[Link](https://doi.org/10.1038/s41591-022-01804-4)Cited by:[§1](https://arxiv.org/html/2609.18134#S1.p1.1),[§5](https://arxiv.org/html/2609.18134#S5.p1.1)\.
- \[21\]M\. Nagendran, Y\. Chen, C\. A\. Lovejoy, A\. C\. Gordon, M\. Komorowski, H\. Harvey, E\. J\. Topol, J\. P\. A\. Ioannidis, G\. S\. Collins, and M\. Maruthappu\(2020\)Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies\.BMJ368,pp\. m689\.External Links:[Document](https://dx.doi.org/10.1136/bmj.m689)Cited by:[§1](https://arxiv.org/html/2609.18134#S1.p1.1)\.
- \[22\]N\. Oufattole, T\. Bergamaschi, A\. Kolo, H\. Jeong, H\. Gaggin, C\. M\. Stultz, and M\. B\. A\. McDermott\(2024\)MEDS\-Tab: automated tabularization and baseline methods for MEDS datasets\.External Links:2411\.00200,[Link](https://arxiv.org/abs/2411.00200)Cited by:[§2\.1](https://arxiv.org/html/2609.18134#S2.SS1.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2609.18134#S2.T1.2.10.1),[Table 1](https://arxiv.org/html/2609.18134#S2.T1.2.11.1),[Table 1](https://arxiv.org/html/2609.18134#S2.T1.2.12.1)\.
- \[23\]C\. Pang, V\. Jeanselme, Y\. S\. Choi, X\. Jiang, Z\. Jing, A\. Kashyap, Y\. Kobayashi, Y\. Li, F\. Pollet, K\. Natarajan, and S\. Joshi\(2025\)FoMoH: a clinically meaningful foundation model evaluation for structured electronic health records\.arXiv preprint arXiv:2505\.16941\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2505.16941)Cited by:[§5](https://arxiv.org/html/2609.18134#S5.p2.1)\.
- \[24\]R\. Rong, Z\. Gu, H\. Lai, T\. L\. Nelson, T\. Keller, C\. Walker, K\. W\. Jin, C\. Chen, A\. M\. Navar, F\. Velasco, E\. D\. Peterson, G\. Xiao, D\. M\. Yang, and Y\. Xie\(2025\)A deep learning model for clinical outcome prediction using longitudinal inpatient electronic health records\.medRxiv\.External Links:[Document](https://dx.doi.org/10.1101/2025.01.21.25320916)Cited by:[Table 1](https://arxiv.org/html/2609.18134#S2.T1.2.13.1)\.
- \[25\]O\. Salaudeen, F\. E\. Dorner, and P\. Hase\(2025\)On evaluating methods vs\. evaluating models\.InNeurIPS 2025 Workshop on Evaluating the Evolving LLM Lifecycle: Benchmarks, Emergent Abilities, and Scaling,Cited by:[§3\.1](https://arxiv.org/html/2609.18134#S3.SS1.p3.1)\.
- \[26\]O\. Salaudeen and M\. Hardt\(2026\)ImageNot: a contrast with imagenet preserves model rankings\.Transactions on Machine Learning Research\.Cited by:[§1](https://arxiv.org/html/2609.18134#S1.p2.1)\.
- \[27\]D\. W\. Shimabukuro, C\. W\. Barton, M\. D\. Feldman, S\. J\. Mataraso, and R\. Das\(2017\)Effect of a machine learning\-based severe sepsis prediction algorithm on patient survival and hospital length of stay: a randomised clinical trial\.BMJ Open Respiratory Research4\(1\),pp\. e000234\.External Links:[Document](https://dx.doi.org/10.1136/bmjresp-2017-000234)Cited by:[§1](https://arxiv.org/html/2609.18134#S1.p1.1)\.
- \[28\]E\. Steinberg, J\. A\. Fries, Y\. Xu, and N\. Shah\(2024\)MOTOR: a time\-to\-event foundation model for structured medical records\.InInternational Conference on Learning Representations,Cited by:[§2\.1](https://arxiv.org/html/2609.18134#S2.SS1.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2609.18134#S2.T1.2.5.1),[Table 1](https://arxiv.org/html/2609.18134#S2.T1.2.7.1)\.
- \[29\]N\. Tomašev, X\. Glorot, J\. W\. Rae, M\. Zielinski, H\. Askham, A\. Saraiva, A\. Mottram, C\. Meyer, S\. Ravuri, I\. Protsyuk, A\. Connell, C\. O\. Hughes, A\. Karthikesalingam, J\. Cornebise, H\. Montgomery, G\. Rees, C\. Laing, C\. R\. Baker, K\. Peterson, R\. Reeves, D\. Hassabis, D\. King, M\. Suleyman, T\. Back, C\. Nielson, J\. R\. Ledsam, and S\. Mohamed\(2019\)A clinically applicable approach to continuous prediction of future acute kidney injury\.Nature572\(7767\),pp\. 116–119\.External Links:[Document](https://dx.doi.org/10.1038/s41586-019-1390-1)Cited by:[§1](https://arxiv.org/html/2609.18134#S1.p1.1)\.
- \[30\]E\. J\. Topol\(2019\)High\-performance medicine: the convergence of human and artificial intelligence\.Nature Medicine25\(1\),pp\. 44–56\.External Links:[Document](https://dx.doi.org/10.1038/s41591-018-0300-7)Cited by:[§1](https://arxiv.org/html/2609.18134#S1.p1.1)\.
- \[31\]S\. Wang, M\. B\. A\. McDermott, G\. Chauhan, M\. Ghassemi, M\. C\. Hughes, and T\. Naumann\(2020\)MIMIC\-Extract: a data extraction, preprocessing, and representation pipeline for MIMIC\-III\.InProceedings of the ACM Conference on Health, Inference, and Learning,pp\. 222–235\.External Links:[Document](https://dx.doi.org/10.1145/3368555.3384469)Cited by:[§5](https://arxiv.org/html/2609.18134#S5.p1.1)\.
- \[32\]J\. Wiens, S\. Saria, M\. Sendak, M\. Ghassemi, V\. X\. Liu, F\. Doshi\-Velez, K\. Jung, K\. Heller, D\. Kale, M\. Saeed, P\. N\. Ossorio, S\. Thadaney\-Israni, and A\. Goldenberg\(2019\)Do no harm: a roadmap for responsible machine learning for health care\.Nature Medicine25\(9\),pp\. 1337–1340\.External Links:[Document](https://dx.doi.org/10.1038/s41591-019-0548-6)Cited by:[§1](https://arxiv.org/html/2609.18134#S1.p1.1)\.
- \[33\]M\. Wornow, R\. Thapa, E\. Steinberg, J\. A\. Fries, and N\. H\. Shah\(2023\)EHRSHOT: an ehr benchmark for few\-shot evaluation of foundation models\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 67125–67137\.External Links:[Document](https://dx.doi.org/10.52202/075280-2933)Cited by:[§5](https://arxiv.org/html/2609.18134#S5.p2.1)\.
- \[34\]J\. Xu, J\. Gallifant, A\. E\. W\. Johnson, and M\. B\. A\. McDermott\(2025\)ACES: automatic cohort extraction system for event\-stream datasets\.InInternational Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/hash/d8542126cd3e0dd6c0a44e0aa1957072-Abstract-Conference.html)Cited by:[§2\.2](https://arxiv.org/html/2609.18134#S2.SS2.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2609.18134#S4.SS1.p3.1)\.
- \[35\]A\. Zhang, L\. Xing, J\. Zou, and J\. C\. Wu\(2022\)Shifting machine learning for healthcare from development to deployment and from models to data\.Nature Biomedical Engineering6,pp\. 1330–1345\.External Links:[Document](https://dx.doi.org/10.1038/s41551-022-00898-y)Cited by:[§3\.1](https://arxiv.org/html/2609.18134#S3.SS1.p3.1),[§5](https://arxiv.org/html/2609.18134#S5.p1.1)\.

## Appendix AList of considered models

The list of considered ICU/EHR models for our study is presented in Table[2](https://arxiv.org/html/2609.18134#A1.T2)\.

Table 2:The 49 structured\-EHR model cards surveyed for this pilot, grouped by training strategy\.*Supervised*models learn predictors directly from task\-specific labels;*Pretrained*models use a separate reusable pretraining stage, consumed either by fine\-tuning or by a frozen probe;*Other*includes zero\-shot and prior\-fitted approaches that do not require task\-specific supervised training\. For zero\-shot algorithms,*dir\.*denotes algorithms that produce predictions directly from the observed record without sampling future trajectories, whereas*mat\.*denotes algorithms that obtain predictions by sampling or materializing future trajectories and evaluating the target outcome on those trajectories\. Years follow each model card’s classification; links are to the source paper and, where available, a public code repository\. Years in this table refer to publication year, whereas years in some other tables refer to the artifact date \(e\.g\., the date of the original preprint or release\)\.∗foundation\-model configuration\.†with self\-supervised pretraining\.ModelPublication YearResourceSupervisedSTraTS\-mTAND2025[paper](https://dl.acm.org/doi/10.1145/3743689)TECO2025[paper](https://academic.oup.com/jamiaopen/article/8/2/ooaf026/8110091)GenHPF2024[paper](https://arxiv.org/abs/2207.09858)/[code](https://github.com/hoon9405/GenHPF)PRISM2024[paper](https://arxiv.org/abs/2309.04160)/[code](https://github.com/yhzhu99/PRISM)TRANS2024[paper](https://arxiv.org/abs/2405.03943)/[code](https://github.com/The-Real-JerryChen/TRANS)KerPrint2023[paper](https://ojs.aaai.org/index.php/AAAI/article/view/25667/25439)/[code](https://github.com/xyxpku/KerPrint)Chet2022[paper](https://arxiv.org/abs/2112.05195)/[code](https://github.com/LuChang-CS/Chet)UniHPF2022[paper](https://arxiv.org/abs/2211.08082)/[code](https://github.com/hoon9405/UniHPF)GRASP2021[paper](https://ojs.aaai.org/index.php/AAAI/article/view/16152)/[code](https://github.com/choczhang/GRASP)SETOR2021[paper](https://arxiv.org/abs/2109.03069)/[code](https://github.com/Xueping/SETOR)AdaCare2020[paper](https://arxiv.org/abs/1911.12205)/[code](https://github.com/Accountable-Machine-Intelligence/AdaCare)ConCare2020[paper](https://arxiv.org/abs/1911.12216)/[code](https://github.com/Accountable-Machine-Intelligence/ConCare)GCT2020[paper](https://arxiv.org/abs/1906.04716)/[code](https://github.com/lycpaul/dl4h-gp30-gct)HiTANet2020[paper](https://dl.acm.org/doi/10.1145/3394486.3403107)/[code](https://github.com/machinelearning4health/HiTANet)StageNet2020[paper](https://arxiv.org/abs/2001.10054)/[code](https://github.com/v1xerunt/StageNet)SAnD2018[paper](https://arxiv.org/abs/1711.03905)/[code](https://github.com/khirotaka/SAnD)Dipole2017[paper](https://arxiv.org/abs/1706.05764)/[code](https://github.com/sunlabuiuc/PyHealth)GRAM2017[paper](https://arxiv.org/abs/1611.07012)/[code](https://github.com/mp2893/gram)Doctor AI2016[paper](https://proceedings.mlr.press/v56/Choi16.html)/[code](https://github.com/mp2893/doctorai)RETAIN2016[paper](https://proceedings.neurips.cc/paper/2016/hash/231141b34c82aa95e48810a9d1b33a79-Abstract.html)/[code](https://github.com/mp2893/retain)Pretrained – ProbeORA2026[paper](https://arxiv.org/abs/2602.00541)PORTER2026[paper](https://arxiv.org/abs/2606.24102)CLMBR2021[paper](https://arxiv.org/abs/2001.05295)/[code](https://github.com/som-shahlab/ehr_ml)Pretrained – Fine\-tunedAID\-MAE2026[paper](https://arxiv.org/abs/2602.15159)HealthFormer2026[paper](https://www.medrxiv.org/content/10.64898/2026.03.25.26349262v2.full-text)/[code](https://github.com/renyi-ai/HealthFormer)SurvivEHR2026[paper](https://www.nature.com/articles/s41746-026-02709-z)/[code](https://github.com/cwlgadd/SurvivEHR)BAT∗2025[paper](https://arxiv.org/abs/2509.19885)/[code](https://github.com/Katja-Jagd/YAIB)CEHR\-XGPT2025[paper](https://arxiv.org/abs/2509.03643)/[code](https://github.com/knatarajan-lab/cehrgpt)PULSE\-ICU2025[paper](https://arxiv.org/abs/2511.22199)/[code](https://github.com/sejeongak/PULSE-ICU)TOO\-BERT2025[paper](https://medinform.jmir.org/2025/1/e68138)/[code](https://github.com/ali-amirahmadii/TOO-Bert)CORE\-BEHRT2024[paper](https://raw.githubusercontent.com/mlresearch/v252/main/assets/odgaard24a/odgaard24a.pdf)/[code](https://github.com/mikkelfo/CORE-BEHRT)EBCL2024[paper](https://proceedings.mlr.press/v252/oufattole24a.html)/[code](https://github.com/mit-ccrg/EBCL)MOTOR2024[paper](https://proceedings.iclr.cc/paper_files/paper/2024/hash/b36554b97da741b1c48c9de05c73993e-Abstract-Conference.html)/[code](https://github.com/som-shahlab/motor_code_release)DuETT2023[paper](https://proceedings.mlr.press/v219/labach23a.html)/[code](https://github.com/layer6ai-labs/duett)Hi\-BEHRT2023[paper](https://arxiv.org/abs/2106.11360)TransEHR2023[paper](https://proceedings.mlr.press/v225/xu23a.html)/[code](https://github.com/SigmaTsing/TransEHR)TransformEHR2023[paper](https://www.nature.com/articles/s41467-023-43715-z)/[code](https://github.com/whaleloops/TransformEHR)GenHPF†2022/2023[paper](https://arxiv.org/abs/2207.09858)/[code](https://github.com/hoon9405/GenHPF)STraTS2022[paper](https://arxiv.org/abs/2107.14293)/[code](https://github.com/sindhura97/STraTS)CEHR\-BERT2021[paper](https://proceedings.mlr.press/v158/pang21a.html)/[code](https://github.com/cumc-dbmi/cehrbert)Med\-BERT2021[paper](https://www.nature.com/articles/s41746-021-00455-y)/[code](https://github.com/ZhiGroup/Med-BERT)BEHRT2020[paper](https://www.nature.com/articles/s41598-020-62922-y)/[code](https://github.com/deepmedicine/BEHRT)G\-BERT2019[paper](https://arxiv.org/abs/1906.00346)/[code](https://github.com/jshang123/g-bert)OtherEveryQuery \(dir\.\)2026[paper](https://arxiv.org/abs/2603.07900)/[code](https://github.com/payalchandak/EveryQuery)SurvPFN2026[paper](https://arxiv.org/abs/2606.04564)/[code](https://github.com/genepi-freiburg/SurvPFN)CoMET \(mat\.\)2025[paper](https://arxiv.org/abs/2508.12104)Delphi\-2M \(dir\.\)2025[paper](https://www.nature.com/articles/s41586-025-09529-3)/[code](https://github.com/gerstung-lab/Delphi)MEDS\-EIC\-AR \(mat\.\)2025–2026[code](https://github.com/mmcdermott/MEDS_EIC_AR)ETHOS / ARES \(mat\.\)2024/2025[paper](https://www.nature.com/articles/s41746-024-01235-0)/[code](https://github.com/ipolharvard/ethos-ares)Table 2:*\(continued\)*
## Appendix BAnalysis definitions

### B\.1Statistical analysis overview

Our primary quantity of interest is the difference in held\-out AUROC between two algorithms on a given task and evaluation setting\. We use these pairwise differences, together with the aggregate rankings across tasks that they induce, to assess how comparative conclusions change when the dataset or task family changes\. Formal definitions follow in the remainder of this appendix, and uncertainty quantification is detailed in Appendix[B\.4](https://arxiv.org/html/2609.18134#A2.SS4)\.

#### Transfer across evaluation settings \(Q1\)\.

We compare the aggregate algorithm rankings for every pair of the four settings\. Pairwise agreement is the proportion of algorithm pairs whose mean\-AUROC difference has the same sign in both settings\. Agreement above chance is assessed with one\-sided random\-ranking permutation tests \(10610^\{6\}permutations, plus\-one correction\), with Holm adjustment across the six between\-setting comparisons\.

#### Task–algorithm interactions \(Q2\)\.

We quantify within\-setting task consistency as one minus the task\-level reversal rate: the proportion of task\-specific pairwise algorithm comparisons that agree with the corresponding comparison averaged across tasks in the same setting \(Appendix[B\.3](https://arxiv.org/html/2609.18134#A2.SS3)\)\. We complement this with a descriptive decomposition of each task–algorithm AUROC matrix into algorithm, task, and algorithm–task interaction components\. Further exploratory Q2 analyses are described alongside their results in Section[3](https://arxiv.org/html/2609.18134#S3); where significance is reported for these analyses, it is assessed with permutation tests, Holm\-adjusted across datasets where applicable\. Q3 relates aggregate performance to model\-family artifact dates, with component\-precedent and performance–compute analyses reported in Appendices[G](https://arxiv.org/html/2609.18134#A7)and[H](https://arxiv.org/html/2609.18134#A8)\.

### B\.2AUROC and pairwise algorithm differences

For a binary prediction task, letS\+S^\{\+\}andS−S^\{\-\}denote the scores assigned by a model to randomly sampled positive and negative examples, respectively\. The area under the receiver operating characteristic curve \(AUROC\) can be interpreted as the probability that a randomly selected positive example receives a higher score than a randomly selected negative example:

AUROC=Pr⁡\(S\+\>S−\)\+12​Pr⁡\(S\+=S−\)\.\\mathrm\{AUROC\}=\\Pr\(S^\{\+\}\>S^\{\-\}\)\+\\frac\{1\}\{2\}\\Pr\(S^\{\+\}=S^\{\-\}\)\.
For algorithmmm, tasktt, and evaluation settingss, we denote the held\-out AUROC byAm​t​sA\_\{mts\}\. Our primary comparative quantity is the difference in AUROC between two algorithmsiiandjj:

Δi​j​t​s=Ai​t​s−Aj​t​s\.\\Delta\_\{ijts\}=A\_\{its\}\-A\_\{jts\}\.Positive values indicate that algorithmiioutperforms algorithmjjon tasktt\.

When comparing algorithms at the evaluation\-setting level, we first average AUROC across theTsT\_\{s\}tasks in that setting:

A¯m​s=1Ts​∑t=1TsAm​t​s,\\bar\{A\}\_\{ms\}=\\frac\{1\}\{T\_\{s\}\}\\sum\_\{t=1\}^\{T\_\{s\}\}A\_\{mts\},and define the corresponding aggregate pairwise difference as

Δ¯i​j​s=A¯i​s−A¯j​s\.\\bar\{\\Delta\}\_\{ijs\}=\\bar\{A\}\_\{is\}\-\\bar\{A\}\_\{js\}\.

### B\.3Task\-level reversal rate

We use task\-level reversals to quantify how often an individual task favors the opposite algorithm from the aggregate comparison within the same evaluation setting\. A reversal occurs for algorithm pair\(i,j\)\(i,j\)on taskttwhen the task\-specific difference and aggregate difference have opposite signs:

Δi​j​t​s​Δ¯i​j​s<0\.\\Delta\_\{ijts\}\\,\\bar\{\\Delta\}\_\{ijs\}<0\.
The reversal rate for evaluation settingssis therefore

Rs=1Ts​\(M2\)∑t=1Ts∑i<j𝟏\[Δi​j​t​sΔ¯i​j​s<0\],R\_\{s\}=\\frac\{1\}\{T\_\{s\}\\binom\{M\}\{2\}\}\\sum\_\{t=1\}^\{T\_\{s\}\}\\sum\_\{i<j\}\\mathbf\{1\}\\left\[\\Delta\_\{ijts\}\\,\\bar\{\\Delta\}\_\{ijs\}<0\\right\],whereMMis the number of evaluated algorithms and𝟏​\[⋅\]\\mathbf\{1\}\[\\cdot\]is the indicator function\. Thus,RsR\_\{s\}is the proportion of all task–algorithm\-pair comparisons whose direction disagrees with the corresponding aggregate comparison\. Ties are not counted as reversals\.

### B\.4Uncertainty quantification

For ranking agreement, reversal rates, horizon correlations, and training\-strategy differences, we combine the 2,000 supplied joint patient\-bootstrap draws with task resampling\. Within each dataset, a common patient draw is reused across all tasks and algorithms; draw orders are paired independently across datasets\. Independently within each setting, we sample ten tasks with replacement, identically across algorithms, and recompute each statistic and its aggregate references\. For horizon correlations, each sampled task’s horizon and contrast remain paired and ranks are recomputed; undefined correlations from constant resamples are excluded and counted\. We report marginal 95% percentile intervals, assuming exchangeable tasks within settings\. These intervals condition on the fitted algorithms: repeated algorithm training was not bootstrapped because of computational cost\.

## Appendix CDataset characterization

Dataset characterization is shown in Table[3](https://arxiv.org/html/2609.18134#A3.T3)\.

Table 3:Summary of the two datasets used in our experiments\.
## Appendix DTasks

### D\.1Task specifications

Task specifications are shown in Tables[4](https://arxiv.org/html/2609.18134#A4.T4),[5](https://arxiv.org/html/2609.18134#A4.T5),[6](https://arxiv.org/html/2609.18134#A4.T6),[7](https://arxiv.org/html/2609.18134#A4.T7)\.

Table 4:Randomly generated tasks selected on MIMIC\-IV\. All statistics are measured on the held\-out split\. The randomly generated task selection requiresmin⁡\(npos,nneg\)≥150\\min\(n\_\{\\mathrm\{pos\}\},n\_\{\\mathrm\{neg\}\}\)\\geq 150\.Δ​t\\Delta tis the prediction horizon; Prev\. is the prevalence, with the minority\-class countnminn\_\{\\min\}in parentheses\. Avg AUROC is averaged across the 12 models\.Table 5:Randomly generated tasks selected on NWICU\. All statistics are measured on the held\-out split\. The randomly generated task selection requiresmin⁡\(npos,nneg\)≥150\\min\(n\_\{\\mathrm\{pos\}\},n\_\{\\mathrm\{neg\}\}\)\\geq 150\. Columns as in Table[4](https://arxiv.org/html/2609.18134#A4.T4)\.Table 6:Clinically meaningful tasks evaluated on MIMIC\-IV\. All statistics are measured on the held\-out split\. Clinical tasks are retained whenmin⁡\(npos,nneg\)≥40\\min\(n\_\{\\mathrm\{pos\}\},n\_\{\\mathrm\{neg\}\}\)\\geq 40\. Columns as in Table[4](https://arxiv.org/html/2609.18134#A4.T4)\.Table 7:Clinically meaningful tasks evaluated on NWICU\. All statistics are measured on the held\-out split\. Clinical tasks are retained whenmin⁡\(npos,nneg\)≥40\\min\(n\_\{\\mathrm\{pos\}\},n\_\{\\mathrm\{neg\}\}\)\\geq 40\. Columns as in Table[4](https://arxiv.org/html/2609.18134#A4.T4)\.
### D\.2Task adequacy threshold\.

We impose a minimum cohort size to avoid evaluating models on tasks for which the held\-out AUROC is estimated with substantial sampling uncertainty\. AUROC can be interpreted as the probability that a randomly selected positive example is ranked above a randomly selected negative example\[[7](https://arxiv.org/html/2609.18134#bib.bib27)\]\. For a Bernoulli ranking outcome with probabilityAA, the variance isA⁡\(1−A\)A\(1\-A\), which is maximized atA=0\.5A=0\.5and is therefore bounded by1/41/4\. This motivates the simple resolution heuristic

SE⁡\(AUROC\)≲12​N,N=min⁡\(npos,nneg\),\\mathrm\{SE\}\(\\mathrm\{AUROC\}\)\\lesssim\\frac\{1\}\{2\\sqrt\{N\}\},\\qquad N=\\min\(n\_\{\\mathrm\{pos\}\},n\_\{\\mathrm\{neg\}\}\),where the minority\-class size is used because it is the limiting class for estimating pairwise discrimination\.

Under this heuristic,N=150N=150corresponds to an AUROC standard\-error scale of approximately0\.0410\.041, whileN=40N=40corresponds to approximately0\.0790\.079\. We use these values as task\-selection criteria rather than as exact confidence intervals; the actual sampling variance of AUROC depends on both class sizes and the distribution of model scores\.

### D\.3Matched tasks

Matched tasks are summarized in Table[8](https://arxiv.org/html/2609.18134#A4.T8)\.

Table 8:Clinical task pairs treated as conceptually matched in the cross\-dataset ranking\-similarity analysis\.

## Appendix EModel development

We used Claude Code v2\.1 with Anthropic Claude Opus 5, assigning one agent to each model implementation\. The agents were not given access to raw PhysioNet data\. Experiments were run on a compute cluster with NVIDIA L40S GPUs\.

## Appendix FDetailed results

Detailed AUROC results are shown in Tables[9](https://arxiv.org/html/2609.18134#A6.T9),[10](https://arxiv.org/html/2609.18134#A6.T10),[11](https://arxiv.org/html/2609.18134#A6.T11),[12](https://arxiv.org/html/2609.18134#A6.T12)\.

Detailed compute cost results are shown in Tables[13](https://arxiv.org/html/2609.18134#A6.T13),[14](https://arxiv.org/html/2609.18134#A6.T14)and[15](https://arxiv.org/html/2609.18134#A6.T15)\.

Table 9:Task\-level AUROC for MIMIC\-IV Generated \(MG\)\. Mean is the unweighted task average; bold indicates the highest unrounded value in each column\. Point estimates only\.Table 10:Task\-level AUROC for MIMIC\-IV Clinical \(MC\)\. Mean is the unweighted task average; bold indicates the highest unrounded value in each column\. Point estimates only\.Table 11:Task\-level AUROC for NWICU Clinical \(NC\)\. Mean is the unweighted task average; bold indicates the highest unrounded value in each column\. Point estimates only\.Table 12:Task\-level AUROC for NWICU Generated \(NG\)\. Mean is the unweighted task average; bold indicates the highest unrounded value in each column\. Point estimates only\.Table 13:MIMIC\-IV generated tasks: macro\-AUC and mean pipeline wall time per task for the 8 retained tasks\. Preprocessing and pretraining include the allocated shared costs; extraction is required for MOTOR\-LP feature extraction\. Training denotes fine\-tuning or supervised fitting\. Total is in hours; components are in seconds\.Totals use unrounded components\. Prediction time is excluded\. Zero denotes a component not used\. Wall times are not hardware\-normalized; values describe the retained task set, not the full evaluation\.

Table 14:NWICU generated tasks: macro\-AUC and mean pipeline wall time per task for the 7 retained tasks\. Preprocessing and pretraining include the allocated shared costs; extraction is required for MOTOR\-LP feature extraction\. Training denotes fine\-tuning or supervised fitting\. Total is in hours; components are in seconds\.Totals use unrounded components\. Prediction time is excluded\. Zero denotes a component not used\. Wall times are not hardware\-normalized; values describe the retained task set, not the full evaluation\.

Table 15:Pooled generated tasks: macro\-AUC and mean pipeline wall time per task for the 15 retained tasks\. Preprocessing and pretraining include the allocated shared costs; extraction is required for MOTOR\-LP feature extraction\. Training denotes fine\-tuning or supervised fitting\. Total is in hours; components are in seconds\. Pooled means give equal weight to each retained task across both datasets\.Totals use unrounded components\. Prediction time is excluded\. Zero denotes a component not used\. Wall times are not hardware\-normalized; values describe the retained task set, not the full evaluation\.

## Appendix GModel recency and methodological progress

To assess whether newer algorithms correspond to improved performance, we relate overall mean AUROC to two notions of model “age”: the first documented public artifact of each model family and the precedent date of selected architectural or learning components, as shown in Table[16](https://arxiv.org/html/2609.18134#A7.T16)\. For each timeline, we also report the best performance achieved up to each year within the evaluated model set\. Component dates are editorial choices, and these curves should be interpreted as retrospective summaries of the models studied here rather than reconstructions of the historical state of the art\.

We consider two notions of recency\. First, we date each model family by the year of its first public artifact, that can be a paper or a GitHub repository \(Figure[3](https://arxiv.org/html/2609.18134#S3.F3)in the main text\)\. Under this view, there is some evidence of improvement over time, but the relationship is far from monotonic: several recent models perform below substantially older approaches, and the strongest overall performer is the XGBoost\-based MEDS\-Tab\-Large baseline\.

A different picture emerges when models are dated by the precedent of their main technical components \(Figure[4](https://arxiv.org/html/2609.18134#A7.F4)\)\. Strong performance is already achieved by relatively old modeling components, most notably logistic regression and XGBoost, and newer technical components do not consistently improve upon this frontier\. This suggests that model recency alone is a poor proxy for methodological progress in the settings we evaluate\.

MEDS\-Tab Large illustrates an important distinction between these two views\. Although it is a recent model artifact, its predictive model is XGBoost; its strong performance instead relies on a modern, extremely wide and sparse tabularization of the longitudinal record, rather than on a recently introduced learning algorithm\. Thus, some improvements associated with newer systems may come from how clinical data are represented and exposed to the model, rather than from the recency of the predictive architecture itself\.

Table 16:Model\-family artifact dates and selected technical\-component dates\.Figure 4:Retrospective progress by selected component\-precedent year\. Points show mean AUROC across the same 40 equally weighted tasks as Figure[3](https://arxiv.org/html/2609.18134#S3.F3), using identical model colors and AUROC limits\. The dashed step curve marks the best mean AUROC among evaluated models assigned to that year or earlier\. Component years are editorial precedents \(Table[16](https://arxiv.org/html/2609.18134#A7.T16)\) and do not establish when the complete pipelines first became feasible\. TECO’s asterisk marks a provisional Transformer\-backbone date\.
## Appendix HPerformance–compute tradeoff

To assess whether newer approaches expand the achievable tradeoff between predictive performance and computational cost, rather than considering performance alone, Figure[5](https://arxiv.org/html/2609.18134#A8.F5)compares generated\-task macro\-AUROC with amortized wall time across the evaluated algorithms\. Shared preprocessing and pretraining costs are amortized across tasks, while task\-specific preprocessing, training, and required feature extraction are included; the Pareto frontier marks algorithms that achieve higher performance for a given cost or lower cost for a given performance\. The models span a wide range of tradeoffs: inexpensive historical approaches such as ICU\-XGBoost and RETAIN achieve relatively strong performance at very low cost, while higher\-performing algorithms generally require more computation\. The frontier is formed by algorithms from several generations and model families rather than by a monotonic sequence of increasingly recent architectures: MEDS\-Tab\-large achieves the highest mean AUROC at the second\-highest cost, whereas MEDS\-EIC\-AR and RETAIN occupy different points on the frontier at lower computational cost\. Per\-model costs are tabulated in Appendix[F](https://arxiv.org/html/2609.18134#A6)\.

Figure 5:Generated\-task macro\-AUROC versus amortized wall time for 12 models on 8 MIMIC\-IV and 7 NWICU tasks\. All models use the same retained tasks within each dataset\. Shared preprocessing and pretraining are amortized over these tasks; task\-specific preprocessing, training, and required feature extraction are added\. Dashed curves show the Pareto frontier\. Wall times are not hardware\-normalized\. Pooled values weight each retained task equally\. For the compute analysis, we restrict to tasks for which compute measurements were successfully verified, yielding 8 MIMIC\-IV and 7 NWICU generated tasks\.
## Appendix IAdditional analyses

This section provides the detailed numerical results underlying the analyses in Section[3](https://arxiv.org/html/2609.18134#S3)\. For Q1, Table[17](https://arxiv.org/html/2609.18134#A9.T17)reports aggregate pairwise agreement and Kendall rank correlation between MIMIC\-IV randomly generated tasks and the other evaluation settings, including bootstrap confidence intervals and permutation\-test results\.

For Q2, Tables[18](https://arxiv.org/html/2609.18134#A9.T18)and[19](https://arxiv.org/html/2609.18134#A9.T19)provide two complementary summaries of task–algorithm interaction: the frequency with which individual\-task comparisons reverse the corresponding aggregate comparison, and the decomposition of observed AUROC variation into model, task, and model–task interaction components\. Table[20](https://arxiv.org/html/2609.18134#A9.T20)examines whether conceptually matched clinical tasks induce more similar model rankings across MIMIC\-IV and NWICU than incorrect correspondences among the same six task concepts; the broader nonmatched comparison is also reported for completeness\.

Tables[22](https://arxiv.org/html/2609.18134#A9.T22)and[21](https://arxiv.org/html/2609.18134#A9.T21)report the exploratory analyses relating prediction horizon to model performance\. The former measures how the relative performance of MOTOR\-FT and MEDS\-EIC\-AR changes with horizon, while the latter relates horizon to BEHRT’s absolute AUROC\. Finally, Table[23](https://arxiv.org/html/2609.18134#A9.T23)reports paired comparisons of fine\-tuning versus linear probing for MOTOR and pretraining versus supervised\-only training for MEDS\-EIC\-AR across each evaluation setting and overall\.

Table 17:Agreement of aggregate model rankings with MIMIC\-IV Generated \(MG;A\)\.Table 18:Task\-level reversals of aggregate model comparisons\. Each setting contains10×66=66010\\times 66=660task–model\-pair comparisons\. MG/MC denote MIMIC\-IV Generated/Clinical; NG/NC denote NWICU Generated/Clinical\.Table 19:Two\-way decomposition of observed AUROC variation within each evaluation setting\. MG/MC denote MIMIC\-IV Generated/Clinical; NG/NC denote NWICU Generated/Clinical\.Table 20:Similarity of individual\-task model rankings across datasets\. Entries are mean Kendallτb\\tau\_\{b\}with 95% patient\-and\-task bootstrap intervals\.Task pairsnnMeanτb\\tau\_\{b\}\[95% CI\]Matched60\.601 \[0\.485, 0\.712\]All nonmatched3940\.546 \[0\.480, 0\.596\]Same\-six nonmatched300\.437 \[0\.352, 0\.573\]Matched−\-all nonmatched0\.055 \[\-0\.059, 0\.182\]Matched−\-same\-six nonmatched0\.164 \[0\.025, 0\.284\]
Exact two\-sided correspondence\-permutation test within the six concepts:p=0\.0194p=0\.0194\(720 bijections\)\.

Table 21:Association between generated\-task horizon and BEHRT AUROC\. Each dataset contains 10 generated tasks\. Brackets give 95% percentile intervals from 2,000 task\-only bootstrap replicates; patient uncertainty is unavailable for absolute AUROC\. Two\-sided permutation tests use 1,000,000 permutations\. Reported p\-values are Holm\-adjusted across the two datasets within this analysis\. Positive correlation indicates better absolute performance at longer horizons\.Table 22:Association of generated\-task horizon with the AUROC difference MOTOR\-FT minus MEDS\-EIC\-AR, across 10 tasks per dataset\. Positive Spearmanρ\\rhoindicates a growing relative advantage for MOTOR\-FT at longer horizons\. Repeated horizons receive average ranks\.Table 23:Paired training\-strategy comparisons\. Entries are mean AUROC differences in percentage points\. MOTOR compares FT minus LP; EIC\-AR compares the pretrained variant minus the supervised\-only variant\.