Evaluating Retrieval-Augmented Generation vs. Long-Context Input for Clinical Reasoning over EHRs

# Evaluating Retrieval-Augmented Generation vs. Long-Context Input for Clinical Reasoning over EHRs
Source: [https://arxiv.org/html/2508.14817](https://arxiv.org/html/2508.14817)
Dmitriy DligachLoyola University ChicagoTimothy A\. MillerBoston Children’s HospitalHarvard Medical SchoolSamantha BarrUniversity of Wisconsin\-MadisonJames LandefeldUniversity of Wisconsin\-MadisonYanjun GaoUniversity of Colorado\-AnschutzMatthew M\. ChurpekUniversity of Wisconsin\-MadisonAnoop MayampurathUniversity of Wisconsin\-MadisonMajid AfsharUniversity of Wisconsin\-Madison

###### Abstract

Objective:To evaluate whether retrieval\-augmented generation \(RAG\) can serve as an efficient alternative to long\-context prompting for clinical reasoning over electronic health records \(EHRs\)\. Methods:We defined three EHR\-based tasks that are replicable across health systems and vary in reasoning complexity: 1\) extracting imaging procedures \(modality, date, and anatomic site\), 2\) generating timelines of therapeutic antibiotic use, and 3\) identifying the key diagnoses for a hospitalization\. Using real inpatient clinical notes from a US academic health system, we evaluated three large language models \(GPT\-5\.4\-mini, Mistral Medium 3, DeepSeek V3\.1\) with varying amounts of provided context, comparing targeted retrieval to using the most recent clinical notes\. Results:ForImaging Procedures, RAG strongly outperformed recent\-note inputs and exceeded long\-context performance \(by 0\.17\-9\.83 F1 across all models\) using fewer than 8K tokens\. Similar benefits were observed forAntibiotic Timelines, where <8K of retrieved tokens matched long\-context recent\-notes performance \(between \-3\.26 to \+3\.24 Jaccard\)\. Error analysis revealed that missing information in the clinical notes—often due to inter\-hospital transfers—limited performance to some extent\. However, performance on theDiagnosis Generationtask remains largely static across methods and models\. Discussion:RAG demonstrated strong token efficiency across tasks, with the clearest and most consistent gains observed for imaging extraction and antibiotic timeline reconstruction\. Diagnosis generation proved the most challenging task, suggesting ceiling effects imposed by documentation variability and evaluation constraints\. Conclusion:Our results suggest that RAG remains a competitive and efficient approach for clinical tasks over large amounts of EHR, even as newer models become capable of handling increasingly longer amounts of text\.

## 1Introduction

Electronic health records \(EHRs\) contain comprehensive documentation of patient care, including critical information for diagnosis and treatment planning\. However, the volume of clinical notes has increased substantially in recent years, driven in part by copy\-paste practices, templated documentation, and regulatory pressures—a phenomenon often referred to as “note bloat”\. For example, nearly 1 in 5 patients arrive at the emergency department with a chart the size of Moby Dick \(more than 200K words\)\[[19](https://arxiv.org/html/2508.14817#bib.bib23)\]\. As a result, clinicians must navigate increasingly lengthy and redundant records to locate key information\.

Large language models \(LLMs\) can potentially alleviate this burden by assisting clinicians in quickly extracting information and reasoning over EHRs, and have demonstrated promising capabilities in clinical summarization\[[21](https://arxiv.org/html/2508.14817#bib.bib15)\]and question answering\[[20](https://arxiv.org/html/2508.14817#bib.bib21)\]\. However, the sheer volume of clinical documentation can exceed most LLMs’ context window size\. A practical approach is to provide the most recent notes, which may suffice for some tasks but risks omitting crucial information buried in earlier documentation\.

Retrieval\-augmented generation \(RAG\) has emerged as a promising solution to using LLMs on long documents by retrieving only the most relevant text passages for a given task\. Rather than processing entire patient charts, RAG systems can selectively extract pertinent clinical information to answer specific questions \(Figure[1](https://arxiv.org/html/2508.14817#S1.F1)\)\. This approach can potentially reduce computational costs, improve accuracy by eliminating noise, and mitigate the “lost\-in\-the\-middle” effect\[[14](https://arxiv.org/html/2508.14817#bib.bib14)\], where model performance degrades when relevant information is buried within lengthy contexts\.

![Refer to caption](https://arxiv.org/html/2508.14817v2/x1.png)Figure 1:Retrieval\-augmented generation pipeline for clinical question answering over EHR\.However, there has been limited empirical evaluation on the accuracy and token efficiency of this retrieval approach for tasks that require longitudinal reasoning over real\-world EHR data\. One barrier is the scarcity of large, annotated clinical datasets due to legal and ethical constraints regarding patient privacy\. While the MIMIC datasets\[[9](https://arxiv.org/html/2508.14817#bib.bib20)\]have been further annotated for benchmarking a variety of natural language processing tasks, including question\-answering, these data are restricted to the patients’ ICU stay, as opposed to the full hospital course, limiting their potential for testing realistic use cases that stretch the token limitations of LLMs’ processing abilities\.

To address these gaps, we define three tasks that reflect different clinical reasoning demands and can be replicated in other health systems without labor\-intensive manual annotation:

- •Imaging Procedures: Produce a list of imaging procedures \(including modality, date, and anatomical location\) that occurred during a hospitalization from the raw clinical notes\.
- •Antibiotic Timelines: Generate the timelines of therapeutic antibiotic use for patients with a life\-threatening infection and, in many cases, have sepsis\.
- •Diagnosis generation: Identify the key diagnoses relevant to the hospitalization\.

TheImaging Procedurestask is a straightforward extractive task that requires the model to identify the imaging procedures performed on different days during the hospital stay\. TheAntibiotic Timelinestask requires not only identifying the antibiotics the patient was on and when they were discontinued, but also incorporating implicit medical reasoning to determine the indications for those antibiotics \(e\.g\., prophylaxis, suppression, prior treatment, current treatment, etc\.\)\. The final task,Diagnosis Generation, requires the most medical reasoning—the model is asked not just to list the diagnoses mentioned, but to determine which required active management and affected the care plan\.

These tasks allow us to investigate the following questions: \(a\) given a limited token budget, does a simple, generalizable RAG approach improve efficiency and performance compared to providing an LLM with only the most recent notes?, and \(b\) to what extent can this baseline approach approximate the performance achieved by utilizing the full long\-context window of modern models?

Using EHR data from the University of Wisconsin \(UW\) hospital, we evaluate three LLMs on these tasks using varying amounts of clinical context, including up to the models’ full context window of 120K tokens\.

Our findings suggest that while a simple RAG can provide substantial efficiency improvements over comparable amounts of recent clinical note tokens, this effect may be task\-dependent\. However, across all three tasks, we found that RAG achieved near\-parity with the full context window using a fraction of the tokens, indicating that retrieval remains a competitive approach even as newer model architectures continue to extend context windows\.

## 2Related Work

Many existing question\-answering datasets for EHRs focus on fact extraction\. Datasets such as EmrQA\[[18](https://arxiv.org/html/2508.14817#bib.bib19)\]and DrugEHRQA\[[4](https://arxiv.org/html/2508.14817#bib.bib18)\]are semi\-automatically constructed by leveraging previous annotations from National NLP Clinical Challenges \(n2c2\) to transform them into question\-answer pairs\. For this type of data, template questions are constructed where the annotation can fill a slot, such as “What is the dosage of\|\|medication\|\|?”\.

The MedAlign dataset\[[7](https://arxiv.org/html/2508.14817#bib.bib8)\]comprises clinician\-generated instruction\-answer pairs and longitudinal EHR data\. Many of the instructions are yes/no questions that can be answered by retrieving a single piece of evidence \(e\.g\. “Does she smoke?”\), but some instructions require the model to synthesize information across the EHR \(e\.g\. “Summarize this patient’s cardiac history\.”\)\. However, evaluation on open\-ended responses poses an ongoing challenge in NLP, with popular automatic metrics such as BLEU and ROUGE showing poor correlation with human judgment on natural language generation tasks in healthcare\[[5](https://arxiv.org/html/2508.14817#bib.bib17)\]\.

Retrieval\-augmented generation has been used for a variety of tasks within the medical domain, including answering open\-ended medical questions by retrieving from medical guidelines and journal articles\[[25](https://arxiv.org/html/2508.14817#bib.bib5)\]and assessing surgical fitness by retrieving from perioperative guidelines\[[10](https://arxiv.org/html/2508.14817#bib.bib7)\]\. Alkhalaf et al\.\[[3](https://arxiv.org/html/2508.14817#bib.bib3)\]used RAG to generate structured summaries by retrieving from EHR, querying for relevant text using the names of the summary fields \(such as “age” and “weight”\)\. The MIRAGE benchmark\[[23](https://arxiv.org/html/2508.14817#bib.bib25)\]is a collection of medical examination and biomedical research QA datasets to evaluate RAG systems\.

Outside the clinical domain, several studies have directly compared RAG against long\-context prompting\. Li et al\.\[[13](https://arxiv.org/html/2508.14817#bib.bib31)\]found that long\-context models generally outperformed retrieval on general\-domain question\-answering benchmarks, though at substantially higher computational cost, motivating a hybrid approach that routes queries between the two methods\. In contrast, Yu et al\.\[[24](https://arxiv.org/html/2508.14817#bib.bib32)\]demonstrated that an order\-preserving retrieval mechanism could exceed full long\-context performance with far fewer input tokens, and the LaRA benchmark\[[12](https://arxiv.org/html/2508.14817#bib.bib33)\]concluded that the optimal choice depends on the model, context length, and task type\. Notably, these comparisons rely on general\-domain corpora such as novels and academic papers, with conclusions that do not straightforwardly transfer to EHRs and reasoning tasks grounded in clinical workflows rather than single\-answer question answering\.

Our work aims to fill these gaps in the prior literature by extending the RAG\-versus\-long\-context comparison to real\-world longitudinal EHR data, using reproducible tasks that require reasoning over a patient’s hospital course\.

## 3Methods

### 3\.1Data and Models

We constructed datasets of 200 inpatient hospitalizations for each of our three tasks using data from the University of Wisconsin hospital system, comprising clinical notes from admission to discharge \(daily progress notes, specialist consultations, imaging reports, etc\.\)\. Table[1](https://arxiv.org/html/2508.14817#S3.T1)provides summary statistics\. All hospitalizations were at least 7 days long and comprised at least 15,000 tokens of clinical notes\. For theImaging ProceduresandDiagnosis Generationtasks, only the clinical notespriorto the discharge summary are used to provide information to the LLM, to avoid leaking information from the hospital course or diagnosis sections of the discharge summary\. For theAntibiotic Timelinestask, all included hospitalizations involved a consultation with Infectious Diseases, and only the notes prior to the consultation note are included in the data presented to the LLM\.

The only structured EHR data provided to the system were the notes’ timestamps and types \(e\.g\., progress note, handoff, etc\.\)\.

Table 1:Dataset statistics for each task\.We evaluated three state\-of\-the\-art LLMs capable of processing up to 128K tokens:

- •GPT\-5\.4\-mini\[[17](https://arxiv.org/html/2508.14817#bib.bib30)\]
- •Mistral Medium 3\[[2](https://arxiv.org/html/2508.14817#bib.bib29)\]
- •DeepSeek V3\.1\[[6](https://arxiv.org/html/2508.14817#bib.bib28)\]

### 3\.2RAG System

For each patient hospitalization, clinical notes were segmented into overlapping 128\-token chunks, with a sliding window of 20\. We embedded these chunks using two models: BGE\-en\-large\-v1\.5\[[22](https://arxiv.org/html/2508.14817#bib.bib4)\], a general\-purpose BERT\-based embedding model trained through contrastive learning, and Qwen3\-Embedding\-8B\[[26](https://arxiv.org/html/2508.14817#bib.bib27)\]\. We included BGE\-en\-large\-v1\.5 based on findings from Myers et al\.\[[15](https://arxiv.org/html/2508.14817#bib.bib13)\]\. This ablation study of embedding models and pooling strategies for EHR retrieval found that BGE\-en\-large\-v1\.5 significantly outperformed general\-domain and biomedical\-domain alternatives across several related retrieval tasks over EHR\. We additionally evaluated Qwen\-3\-Embedding to assess whether a more recent, larger\-scale embedding model offers further gains for these tasks\.

For each task, we manually crafted a simple query for retrieving relevant passages \(e\.g\. “What are the patient’s diagnoses?”\)\. Each query was prepended with the model’s recommended instruction prefix: “Represent this sentence for searching relevant passages:” for BGE\-en\-large\-v1\.5, and “Instruct: Given a web search query, retrieve relevant passages that answer the query\\nQuery: ” for Qwen3\-Embedding\-8B\. We used cosine similarity between the query and each chunk to retrieve the top\-N most relevant passages \(N = 20, 40, 60\)\. These chunks were inserted into the instruction prompt \(provided in Appendix B\) and passed to the LLM\.

We compared this retrieval configuration to a baseline approach of providing the most recent clinical notes in comparable amounts \(2560, 5120, or 7680 tokens of EHR text\) and long\-context inputs with up to 64K or 120K EHR tokens\. References to these token amounts throughout this study should be understood as anupper bound, as some encounters consist of fewer tokens, reflecting the underlying hospitalization distribution\. Token counts reported throughout this study were computed using an independent tokenizer rather than each model’s native tokenizer\. For each token amount evaluated \(e\.g\., 20 chunks of 128 tokens, 64K tokens\), this ensured that all models received identical input text, since tokenizers can produce different counts for the same content\.

Performance on the tasks was evaluated using either the F1 or Jaccard index, as described in the following sections, and we assessed comparative performance between the RAG and non\-RAG approaches across increasing numbers of tokens by calculating the area under the curves and reporting the normalized area difference\. Specifically, for each method we plotted task performance as a function of input token count and interpolated between the measured points to obtain continuous curves over the range of token budgets shared by both methods\. We then computed the area under each curve using the trapezoidal rule and defined the normalized area difference as:

Normalized Area Difference=AUCRAG−AUCRecent NotesAUCRecent Notes×100%\\text\{Normalized Area Difference\}=\\frac\{\\text\{AUC\}\_\{\\text\{RAG\}\}\-\\text\{AUC\}\_\{\\text\{Recent Notes\}\}\}\{\\text\{AUC\}\_\{\\text\{Recent Notes\}\}\}\\times 100\\%\(1\)A positive value indicates that RAG outperformed the recent\-notes baseline across the shared token range, with the magnitude reflecting the relative size of the gap\. A value of 100% indicates that the area under the RAG curve was twice that of the Recent Notes curve over the evaluated range, while a negative value indicates that the recent\-notes baseline performed better on average\.

### 3\.3Task 1: Imaging Procedures

TheImaging Procedurestask involves extracting structured information from unstructured clinical notes about diagnostic imaging procedures\. We focused on five common imaging modalities: Magnetic Resonance Imaging \(MRI\), Computed Tomography \(CT\), Ultrasound, X\-ray, and Nuclear Medicine \(NM\) Imaging\. The model was prompted to produce a list of imaging procedures performed during the hospitalization, specifying the modality, anatomical location, and date\. Relevant passages were retrieved using a query string listing relevant procedures: “X\-ray, CT, MRI, ultrasound, NM imaging, echocardiogram, fluoroscopy”\.

As a gold standard for this task, we used tabular procedure records from the EHR\. We mapped these procedure descriptions to imaging modality and anatomical site using simple rules and regular expressions, and manually verified the 934 resultant mappings\. Our test set contains 66 unique combinations of modality and location\. For example:

```
X-RAY CHEST 2 VIEWS
        modality: "X-ray"
        location: "chest"
   CT LUMBAR SPINE W/O IV CONTRAST
        modality: "CT"
        location: "lumbar spine"
```

Evaluation metrics are reported for three levels of strictness:

- •Modality\+Date\+Location
- •Modality\+Date
- •Modality\+Date\(±1 day\)

In the lattermost case, we allow for a reasonable tolerance in the predicted date due to observed variation in the reported metadata times for the procedure and note\. For example, the timestamp for the note may reflect the date it was filed into the system, rather than the date it was actually written\.

It should also be noted that the anatomical location is not normalized, other than for capitalization\. For example, predicting “spine” for the above example would not be deemed a positive match\.

### 3\.4Task 2: Antibiotic Timelines

This task emulates work performed by Infectious Diseases \(ID\) physicians, documenting the antibiotic regimen for an active infection in their consultation note\. When these specialists are consulted, they document the history of the present illness, including lab results and medications, and outline a treatment plan\. This note typically contains a “History of Anti\-Infectives” section, where they list the antibiotics that have been used to treat the infection of concern, omitting prophylactic or non\-relevant anti\-infectives\. For example:

```
Vancomycin: 1/16-present
       Ceftriaxone: 1/17-present
```

These medication names and date ranges were manually annotated by ID specialists after reviewing the patient’s chart and served as our ground truth for this task, after extraction using regular expressions\. No notes authored by ID physicians were included in the data that was presented to the model, and only notes that were written prior to the ground truth note were made available\. Relevant passages were retrieved using the query “What antibiotics has the patient taken?”\.

We evaluated system accuracy with the following metrics:

- •Medications \(name only\): Classification accuracy of only the medications, disregarding timespans\.
- •Timespan Overlap: The overlap between the predicted and gold date ranges for each antibiotic, reported using the Jaccard index\. A value of 1 indicates an exact match; 0 indicates no overlap, missing a medication entirely, or including a medication not present in the gold standard\. Values are averaged over the dataset\.

The medications in both the generated predictions and the gold data are normalized to their ingredients using the RxNorm\[[16](https://arxiv.org/html/2508.14817#bib.bib22)\]API provided by the National Library of Medicine and a handful of manual rules for edge cases, such as typos\. This approach allows for accurate matching of generic and brand\-name medications, such as Zosyn \(piperacillin and tazobactam\)\.

As a baseline, we used a rule\-based approach to directly extract the time ranges for all medications of the “anti\-infective” therapeutic class from the list of administered medications using the EHR’s medication administration record \(MAR\), a tabular form ubiquitous in EHRs for tracking all medications and infusions\. However, this list of medications includes those used to treat other conditions that were not the focus of the ID consultation\. By formulating this task to replicate the ID specialists’ work, rather than on replicating structured data as theImaging Proceduresdid, this task requires an additional level of medical reasoning to accurately conform to inclusion criteria\.

### 3\.5Task 3: Diagnosis Generation

The goal of this task is to generate a list of diagnoses for a given hospitalization that is of primary relevance to the clinician\.

We drew from two EHR sources to construct our gold labels for each hospitalization:

- •Discharge Summary: The free text from the discharge summary that lists the primary and secondary diagnoses\.
- •Billing Codes: The lists of International Classification of Diseases \(ICD\-10\) codes from the structured EHR for the hospitalization, manually annotated by billing coders\.

The diagnoses annotated by the coders are guaranteed to be documented within the notes, provide a high degree of specificity, and are normalized to a standard vocabulary, but these lengthy lists often include diagnoses that are not necessarily considered to be of key importance to clinicians, such as a history of smoking, obesity, or minor issues such as a contusion\. On the other hand, the discharge summary may not list specific diagnoses, such as noting post\-surgical status \(e\.g\., “S/p kidney transplant”\) or leaving some diagnoses undocumented because they can be inferred\. For example, documenting that the patient is post\-kidney transplant and has complications, but not that they have kidney disease\.

While discharge summaries reflect the information of most clinical relevance to the clinicians providing treatment, the billing codes lend themselves better to validation due to their standardization\. To produce a more balanced representation, we instructed Gemma\-3\-32B\[[8](https://arxiv.org/html/2508.14817#bib.bib24)\]to filter the billing ICD code lists to only the entries that reflect the clinician’s primary foci for the hospital stay based on the text from the discharge summary\. ThisFilteredlist of ICD\-10 codes serves as our primary evaluation target and the instruction prompt was designed to elicit this list by outlining inclusion and exclusion criteria for diagnoses \(e\.g\., include acute conditions requiring ICU care, exclude stable chronic conditions or irrelevant historical diagnoses\)\. Manual validation of a subset of these filtered lists was performed, as detailed in Appendix D, finding that they largely correlate with clinician judgment\.

To enable classification evaluation, the free text generated by the LLM and from the discharge summary need to be normalized to ICD\-10 codes\. For this process, we trained the state\-of\-the\-art SNOBERT\[[11](https://arxiv.org/html/2508.14817#bib.bib2)\]architecture to extract Systematized Nomenclature of Medicine \(SNOMED\) concepts from the text111Training details are provided in Appendix Cand used the mappings provided by the SNOMED Clinical Terms data release to convert them to ICD\-10\. A manual assessment of the accuracy of this process is described in Appendix E\.

ICD\-10 is a hierarchical vocabulary, ranging from broad concepts \(e\.g\. “Anemia, unspecified” \[D64\.9\]\) to highly specific \(e\.g\. “Age\-related osteoporosis with current pathological fracture, right shoulder, initial encounter for fracture” \[M80\.011A\]\)\. Due to this high granularity, evaluating this task requires a more fuzzy matching technique, rather than evaluating classification accuracy on the ICD codes themselves\. For this purpose, we employed the Healthcare Cost and Utilization Project’s Clinical Classifications Software Refined \(CCSR\)\[[1](https://arxiv.org/html/2508.14817#bib.bib1)\]\. The CCSR provides a mapping from ICD\-10 codes to about 530 clinically relevant categories\.

CCSR is a many\-to\-many mapping, which enables mapping very fine\-grained ICD codes such as “Hypertensive chronic kidney disease” to multiple CCSR categories: “Hypertension with complications and secondary hypertension” and “Chronic kidney disease”\. This approach allowed us to consider predicted diagnoses to be a match even if the LLM split them into “Hypertension” and “Chronic kidney disease”\.

Some broader non\-billable ICD codes are not included in the CCSR mapping \(e\.g\. “Hypotension” \[I95\]\)\. In these cases, we used the set intersection of the CCSR categories that the ICD code’ssubcategories\(e\.g\., I95\.3, I95\.89, etc\.\) are mapped to\.

## 4Results

### 4\.1Task 1: Imaging Procedures

Across all three models and evaluation methods, RAG yielded dramatic performance improvements compared to comparable token amounts of recent notes for generating a list of the imaging procedures over the hospitalization\. In Figure[2](https://arxiv.org/html/2508.14817#S4.F2), we show the classification performance forModality\+Date\+Locationacross varying amounts of provided EHR context\. We calculated the normalized area difference between the curves for the overlapping token amounts, presented in Table[2](https://arxiv.org/html/2508.14817#S4.T2)\. We found at minimum a 2\.5\-fold performance gain against using similar amounts of the most recent notes\. These results also demonstrate that targeted retrieval of passages can closely approach the performance of utilizing the full context window with only a fraction of the tokens: Using only 60 chunks retrieved with Qwen3\-Embedding, GPT\-5\.4\-mini, Mistral Medium 3, and DeepSeek V3\.1 exceeded the performance of using 120K tokens of recent notes by 0\.17, 9\.83, and 2\.15, respectively\.

![Refer to caption](https://arxiv.org/html/2508.14817v2/ImagingProcedures.png)Figure 2:Imaging Procedures: F1 scores for the three models using theModality\+Date\+Locationevaluation method, across varying amounts of provided EHR tokens\. The area between the curves used to calculate normalized area difference for RAG \(BGE\) is shown highlighted in green\.GPT\-5\.4\-miniMistral Medium 3DeepSeek V3\.1BGEQwen3BGEQwen3BGEQwen3Modality\+Date\+Location296\.80%338\.40%298\.03%327\.02%364\.86%387\.16%Modality\+Date263\.27%281\.38%266\.36%279\.80%311\.08%321\.15%Modality\+Date\(±1 day\)265\.44%273\.76%251\.52%256\.40%297\.92%294\.63%

Table 2:Normalized area difference between the RAG and Recent Notes curves for theImaging Procedurestask\.A complete listing of precision, recall, and F1 scores can be found in Appendix A\.

### 4\.2Task 2: Antibiotic Timelines

Figure[3](https://arxiv.org/html/2508.14817#S4.F3)shows the performance of the models forTimespan Overlap\. The RAG approach consistently exceeds the rule\-based baseline and demonstrates close performance to using 120K tokens of recent notes \(using 60 chunks retrieved with BGE, GPT\-5\.4\-mini: \-3\.51, Mistral Medium 3: \+3\.24, DeepSeek V3\.1: \-1\.34\)\. The performance of the RAG approach shows only slight gains with increased retrieved text\. The models show a consistent 22\.23%\-33\.16% improvement in the normalized area difference between retrieval and recent\-note methods for generating antibiotic timespans, as shown in Table[3](https://arxiv.org/html/2508.14817#S4.T3)\.

For all models, the average Jaccard index drops slightly when increasing the provided context from 64K to 120K tokens \(GPT\-5\.4\-mini: \-0\.78, Mistral Medium 3: \-1\.68, DeepSeek V3\.1: \-0\.16\)\.

On the task of predictingMedications \(name only\), the RAG approach slightly outperforms using 120K tokens of recent notes with only 60 chunks retrieved by BGE \(GPT\-5\.4\-mini: \+1\.61, Mistral Medium 3: \+5\.56, DeepSeek V3\.1: \+0\.34\. A complete listing of Jaccard index, precision, recall, and F1 scores can be found in Appendix A\.

Error analysis for this task draws attention to one of the limitations to be encountered when designing tasks for longitudinal EHR\. In examining 70 encounters, we found that in 22\.4% of the gold medications, the information needed to generate the gold standard medication and precise timespan is not present in the full clinical notes\. Most often, this occurs when the patient was transferred from another health system\. This incomplete picture of a patient’s history is hard to avoid when constructing datasets to capture longitudinal EHR, as patients don’t exclusively visit a single healthcare system and healthcare data governance creates barriers to accessing this external information\. The simple retrieval method we used also leaves further room for improvement—when retrieving 20 chunks using BGE, relevant information that could have improved performance was missed for 32% of gold medications\.

![Refer to caption](https://arxiv.org/html/2508.14817v2/AntibioticTimelines.png)Figure 3:Antibiotic Timelines: Average Jaccard index for the three models using theTimespan Overlapevaluation method, across varying amounts of provided EHR tokens\. The area between the curves used to calculate normalized area difference for RAG \(BGE\) is shown highlighted in green\.GPT\-5\.4\-miniMistral Medium 3DeepSeek V3\.1BGEQwen3BGEQwen3BGEQwen3Medications \(name only\)21\.30%17\.45%30\.00%26\.75%27\.81%24\.84%Timespan Overlap26\.04%22\.23%33\.16%30\.12%29\.72%22\.27%

Table 3:Normalized area difference between the RAG and Recent Notes curves for theAntibiotic Timelinestask\.
### 4\.3Task 3: Diagnosis Generation

![Refer to caption](https://arxiv.org/html/2508.14817v2/DiagnosisGeneration.png)Figure 4:Diagnosis Generation: F1 scores for the three models using theFilteredevaluation method, across varying amounts of provided EHR tokens\. The area between the curves used to calculate normalized area difference for RAG \(BGE\) is shown highlighted in red\.GPT\-5\.4\-miniMistral Medium 3DeepSeek V3\.1BGEQwen3BGEQwen3BGEQwen3Billing codes2\.90%7\.62%7\.23%8\.52%4\.33%7\.81%Discharge summary\-6\.71%\-5\.34%\-10\.32%\-11\.50%\-7\.02%\-6\.88%Filtered\-4\.67%\-4\.86%\-4\.80%\-7\.90%\-3\.82%\-4\.53%

Table 4:Normalized area difference between the RAG and Recent Notes curves for theDiagnosis Generationtask\.Unlike the previous two tasks, we do not see a consistent improvement in performance for using RAG compared to comparable amounts of recent notes, shown in Figure[4](https://arxiv.org/html/2508.14817#S4.F4)and Table[4](https://arxiv.org/html/2508.14817#S4.T4), but actually a slight decrease, other than evaluating against theBilling Codes\. However, the performance using very long contexts is not substantially higher than that of using fewer tokens\. Overall, performance is relatively flat across models and data selection approaches and does not exceed an F1 score of 41\.07 againstFiltered diagnosis lists, or 51\.03 against Billing codes\. For theFilteredlist that serves as our primary target, scores across all context selection methods for GPT\-5\.4\-mini, Mistral Medium 3, and DeepSeek V3\.1 all fell within the small ranges of 2\.50, 4\.04, and 4\.37, respectively\.

For bothFilteredandDischarge Summarytargets, performance using the most recent notes is detrimented by using very large context amounts, while performance onBilling Codesdemonstrates additional benefit from the additional text\. A complete listing of precision, recall, and F1 scores can be found in Appendix A\.

## 5Discussion

TheImaging Procedurestask, which requires relatively shallow extraction of information data from the clinical notes, demonstrated the clearest benefit from RAG, and the performance gains from retrieval were both substantial and consistent across models\.

TheAntibiotic Timelinestask introduces greater complexity, requiring both temporal reasoning and clinical understanding to distinguish therapeutic antibiotics from incidental medications\. While RAG also provided a significant improvement over using only recent notes, performance gains plateaued quickly—suggesting that only a limited number of passages are needed to reconstruct the key temporal history when performing targeted retrieval\.

TheDiagnosis Generationtask presented the greatest challenge, but we note that it is also the most subjective task\. Physicians can vary in documentation practices and what is chosen to be included in the discharge summary – an inherent limitation in automatic evaluation of this task\. Performance varied by the evaluation target, with theBilling Codeslist benefiting the most from additional text, likely due to this being a more exhaustive list of diagnoses to capture everything that can be billed\. The fact that all scores, regardless of retrieval method and consistently across the models, fell within low, narrow ranges points towards performance reaching a ceiling caused by limitations of the task or evaluation method\. The evaluation of this task depends on normalizing free text to ICD codes, which we do through a trained model identifying SNOMED codes before using manually written mappings\. While we conducted a manual evaluation of this normalization pipeline \(Appendix E\) and found that core diagnoses were captured with at most minor errors in the majority of cases, this process inevitably introduces some noise into the evaluation\.

We acknowledge several limitations to our study: Our evaluation draws on clinical notes from a single academic health system and the generalizability of our findings to other settings is not established by this work\. Additionally, our efficiency comparison is reported in prompt tokens and does not capture the full cost of a deployed system: RAG incurs one\-time chunking and embedding costs per patient as well as per\-query retrieval overhead, while long\-context inference can amortize repeated queries over the same record through prompt caching; end\-to\-end evaluation of these deployment costs remains for future work\.

Furthermore, our evaluated RAG implementation uses a fixed chunking strategy, query formulation, and only two embedding models\. Retrieval performance is sensitive to these parameters, and alternative configurations, or more advanced techniques such as hybrid retrieval and reranking, may yield different results\. However, across all tasks and models, we observed a consistent trend: a very basic retrieval\-augmented generation approach was able to closely match the performance of long\-context inputs with far fewer tokens\.

## 6Conclusion

In this work, we evaluated the effectiveness of retrieval\-augmented generation across varying information demands in electronic health records across three clinical tasks\. Each task was selected for its clinical relevance, reproducibility across health systems, and varying degrees of reasoning complexity\.

Our results demonstrate that a simple targeted retrieval approach can reach near parity with using up to 120K tokens of recent clinical notes on these three tasks, while requiring significantly fewer input tokens\. These findings show RAG’s continued value even as LLMs grow more capable of processing long sequences\. Further tuning the retrieval approach \(queries, embedding model, retrieving more than 60 chunks, etc\.\), may close the remaining gap\.

Future work should explore additional tasks that can be devised without extensive manual effort and informed by clinical workflows and documentation practices, as well as more granular error analyses distinguishing retrieval failures from generation failures, in order to provide a more robust assessment of models and retrieval methods over longitudinal EHR tasks\.

## Data Availability Statement

The patient data underlying this article cannot be shared publicly due to ethical and legal restrictions\. Intermediary analysis data will be shared upon reasonable request to the corresponding author\.

## Funding acknowledgement

This work was supported by the National Library of Medicine of the National Institutes of Health under award number R01LM012973 and by the National Institute of General Medical Sciences under award number R35GM145330\. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health\.

## CRediT authorship contribution statement

S\.M\.: Writing\-original draft, Methodology, Data curation, Investigation, Conceptualization, Formal analysis, Software\.D\.D\.: Conceptualization, Methodology, Writing–review & editing, Supervision\.T\.A\.M\.: Writing–review & editing, Methodology, Conceptualization, Funding acquisition, Supervision\.S\.B\.: Formal analysis, Writing–review & editing\.J\.L\.: Formal analysis, Writing–review & editing\.Y\.G\.: Methodology, Writing–review & editing\.M\.C\.: Methodology, Writing–review & editing\.A\.M\.: Methodology, Writing–review & editing\.M\.A\.: Supervision, Methodology, Formal analysis, Writing–review & editing, Investigation, Conceptualization, Funding acquisition\.

## Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper\.

## Ethics approval

This study was reviewed and approved by the University of Wisconsin–Madison Institutional Review Board, with a waiver of informed consent given the retrospective use of clinical data\. All data handling complied with the Health Insurance Portability and Accountability Act\.

## References

- \[1\]Agency for Healthcare Research and Quality\(2025\)Clinical Classifications Software Refined \(CCSR\) for ICD\-10\-CM Diagnoses — hcup\-us\.ahrq\.gov\.Note:[https://hcup\-us\.ahrq\.gov/toolssoftware/ccsr/dxccsr\.jsp](https://hcup-us.ahrq.gov/toolssoftware/ccsr/dxccsr.jsp)\[Accessed 17\-07\-2025\]Cited by:[§3\.5](https://arxiv.org/html/2508.14817#S3.SS5.p6.1)\.
- \[2\]M\. AI\(2025\)Medium is the new large\. — Mistral AI — mistral\.ai\.Note:[https://mistral\.ai/news/mistral\-medium\-3](https://mistral.ai/news/mistral-medium-3)\[Accessed 03\-05\-2026\]Cited by:[2nd item](https://arxiv.org/html/2508.14817#S3.I1.i2.p1.1)\.
- \[3\]M\. Alkhalaf, P\. Yu, M\. Yin, and C\. Deng\(2024\)Applying generative ai with retrieval augmented generation to summarize and extract key clinical information from electronic health records\.Journal of biomedical informatics156,pp\. 104662\.Cited by:[§2](https://arxiv.org/html/2508.14817#S2.p3.1)\.
- \[4\]J\. Bardhan, A\. Colas, K\. Roberts, and D\. Z\. Wang\(2022\-06\)DrugEHRQA: a question answering dataset on structured and unstructured electronic health records for medicine related queries\.InProceedings of the Thirteenth Language Resources and Evaluation Conference,N\. Calzolari, F\. Béchet, P\. Blache, K\. Choukri, C\. Cieri, T\. Declerck, S\. Goggi, H\. Isahara, B\. Maegaard, J\. Mariani, H\. Mazo, J\. Odijk, and S\. Piperidis \(Eds\.\),Marseille, France,pp\. 1083–1097\.External Links:[Link](https://aclanthology.org/2022.lrec-1.117/)Cited by:[§2](https://arxiv.org/html/2508.14817#S2.p1.2)\.
- \[5\]E\. Croxford, Y\. Gao, B\. Patterson, D\. To, S\. Tesch, D\. Dligach, A\. Mayampurath, M\. M\. Churpek, and M\. Afshar\(2024\)Development of a Human Evaluation Framework and Correlation with Automated Metrics for Natural Language Generation of Medical Diagnoses\.AMIA … Annual Symposium proceedings\. AMIA Symposium2024,pp\. 309–318\(eng\)\.External Links:ISSN 1942\-597XCited by:[§2](https://arxiv.org/html/2508.14817#S2.p2.1)\.
- \[6\]I\. DeepSeek\(2025\)DeepSeek\-V3\.1 Release — DeepSeek API Docs — api\-docs\.deepseek\.com\.Note:[https://api\-docs\.deepseek\.com/news/news250821](https://api-docs.deepseek.com/news/news250821)\[Accessed 03\-05\-2026\]Cited by:[3rd item](https://arxiv.org/html/2508.14817#S3.I1.i3.p1.1)\.
- \[7\]S\. L\. Fleming, A\. Lozano, W\. J\. Haberkorn, J\. A\. Jindal, E\. Reis, R\. Thapa, L\. Blankemeier, J\. Z\. Genkins, E\. Steinberg, A\. Nayak,et al\.\(2024\)Medalign: a clinician\-generated dataset for instruction following with electronic medical records\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 22021–22030\.Cited by:[§2](https://arxiv.org/html/2508.14817#S2.p2.1)\.
- \[8\]Google\(2025\)Introducing Gemma 3: The most capable model you can run on a single GPU or TPU — blog\.google\.Note:[https://blog\.google/technology/developers/gemma\-3/](https://blog.google/technology/developers/gemma-3/)Cited by:[§3\.5](https://arxiv.org/html/2508.14817#S3.SS5.p4.1)\.
- \[9\]A\. E\. Johnson, T\. J\. Pollard, L\. Shen, L\. H\. Lehman, M\. Feng, M\. Ghassemi, B\. Moody, P\. Szolovits, L\. Anthony Celi, and R\. G\. Mark\(2016\)MIMIC\-iii, a freely accessible critical care database\.Scientific data3\(1\),pp\. 1–9\.Cited by:[§1](https://arxiv.org/html/2508.14817#S1.p4.1)\.
- \[10\]Y\. H\. Ke, L\. Jin, K\. Elangovan, H\. R\. Abdullah, N\. Liu, A\. T\. H\. Sia, C\. R\. Soh, J\. Y\. M\. Tung, J\. C\. L\. Ong, C\. Kuo,et al\.\(2025\)Retrieval augmented generation for 10 large language models and its generalizability in assessing medical fitness\.npj Digital Medicine8\(1\),pp\. 187\.Cited by:[§2](https://arxiv.org/html/2508.14817#S2.p3.1)\.
- \[11\]M\. Kulyabin, G\. Sokolov, A\. Galaida, A\. Maier, and T\. Arias\-Vergara\(2025\)SNOBERT: a benchmark for clinical notes entity linking in the SNOMED CT clinical terminology\.InPattern Recognition,A\. Antonacopoulos, S\. Chaudhuri, R\. Chellappa, C\. Liu, S\. Bhattacharya, and U\. Pal \(Eds\.\),Cham,pp\. 154–163\.External Links:ISBN 978\-3\-031\-78119\-3Cited by:[§3\.5](https://arxiv.org/html/2508.14817#S3.SS5.p5.1)\.
- \[12\]K\. Li, L\. Zhang, Y\. Jiang, P\. Xie, F\. Huang, S\. Wang, and M\. Cheng\(2025\-13–19 Jul\)LaRA: benchmarking retrieval\-augmented generation and long\-context LLMs – no silver bullet for LC or RAG routing\.InProceedings of the 42nd International Conference on Machine Learning,A\. Singh, M\. Fazel, D\. Hsu, S\. Lacoste\-Julien, F\. Berkenkamp, T\. Maharaj, K\. Wagstaff, and J\. Zhu \(Eds\.\),Proceedings of Machine Learning Research, Vol\.267,pp\. 36846–36867\.External Links:[Link](https://proceedings.mlr.press/v267/li25dv.html)Cited by:[§2](https://arxiv.org/html/2508.14817#S2.p4.1)\.
- \[13\]Z\. Li, C\. Li, M\. Zhang, Q\. Mei, and M\. Bendersky\(2024\-11\)Retrieval augmented generation or long\-context LLMs? a comprehensive study and hybrid approach\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track,F\. Dernoncourt, D\. Preoţiuc\-Pietro, and A\. Shimorina \(Eds\.\),Miami, Florida, US,pp\. 881–893\.External Links:[Link](https://aclanthology.org/2024.emnlp-industry.66/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-industry.66)Cited by:[§2](https://arxiv.org/html/2508.14817#S2.p4.1)\.
- \[14\]N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. Liang\(2024\)Lost in the middle: how language models use long contexts\.Transactions of the Association for Computational Linguistics12,pp\. 157–173\.External Links:[Link](https://aclanthology.org/2024.tacl-1.9/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00638)Cited by:[§1](https://arxiv.org/html/2508.14817#S1.p3.1)\.
- \[15\]S\. Myers, T\. A\. Miller, Y\. Gao, M\. M\. Churpek, A\. Mayampurath, D\. Dligach, and M\. Afshar\(2025\)Lessons learned on information retrieval in electronic health records: a comparison of embedding models and pooling strategies\.Journal of the American Medical Informatics Association32\(2\),pp\. 357–364\.Cited by:[§3\.2](https://arxiv.org/html/2508.14817#S3.SS2.p1.1)\.
- \[16\]S\. J\. Nelson, K\. Zeng, J\. Kilbourne, T\. Powell, and R\. Moore\(2011\)Normalized names for clinical drugs: rxnorm at 6 years\.Journal of the American Medical Informatics Association18\(4\),pp\. 441–448\.Cited by:[§3\.4](https://arxiv.org/html/2508.14817#S3.SS4.p4.1)\.
- \[17\]OpenAI\(2026\)Introducing GPT\-5\.4 mini and nano — openai\.com\.Note:[https://openai\.com/index/introducing\-gpt\-5\-4\-mini\-and\-nano/](https://openai.com/index/introducing-gpt-5-4-mini-and-nano/)\[Accessed 03\-05\-2026\]Cited by:[1st item](https://arxiv.org/html/2508.14817#S3.I1.i1.p1.1)\.
- \[18\]A\. Pampari, P\. Raghavan, J\. Liang, and J\. Peng\(2018\-October\-November\)EmrQA: a large corpus for question answering on electronic medical records\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,E\. Riloff, D\. Chiang, J\. Hockenmaier, and J\. Tsujii \(Eds\.\),Brussels, Belgium,pp\. 2357–2368\.External Links:[Link](https://aclanthology.org/D18-1258/),[Document](https://dx.doi.org/10.18653/v1/D18-1258)Cited by:[§2](https://arxiv.org/html/2508.14817#S2.p1.2)\.
- \[19\]B\. W\. Patterson, D\. J\. Hekman, F\. J\. Liao, A\. G\. Hamedani, M\. N\. Shah, and M\. Afshar\(2024\)Call me dr ishmael: trends in electronic health record notes available at emergency department visits and admissions\.JAMIA open7\(2\),pp\. ooae039\.Cited by:[§1](https://arxiv.org/html/2508.14817#S1.p1.1)\.
- \[20\]K\. Singhal, T\. Tu, J\. Gottweis, R\. Sayres, E\. Wulczyn, M\. Amin, L\. Hou, K\. Clark, S\. R\. Pfohl, H\. Cole\-Lewis,et al\.\(2025\)Toward expert\-level medical question answering with large language models\.Nature Medicine31\(3\),pp\. 943–950\.Cited by:[§1](https://arxiv.org/html/2508.14817#S1.p2.1)\.
- \[21\]D\. Van Veen, C\. Van Uden, L\. Blankemeier, J\. Delbrouck, A\. Aali, C\. Bluethgen, A\. Pareek, M\. Polacin, E\. P\. Reis, A\. Seehofnerová,et al\.\(2024\)Adapted large language models can outperform medical experts in clinical text summarization\.Nature medicine30\(4\),pp\. 1134–1142\.Cited by:[§1](https://arxiv.org/html/2508.14817#S1.p2.1)\.
- \[22\]S\. Xiao, Z\. Liu, P\. Zhang, and N\. Muennighoff\(2023\)C\-pack: packaged resources to advance general chinese embedding\.External Links:2309\.07597Cited by:[§3\.2](https://arxiv.org/html/2508.14817#S3.SS2.p1.1)\.
- \[23\]G\. Xiong, Q\. Jin, Z\. Lu, and A\. Zhang\(2024\-08\)Benchmarking retrieval\-augmented generation for medicine\.InFindings of the Association for Computational Linguistics ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand and virtual meeting,pp\. 6233–6251\.External Links:[Link](https://aclanthology.org/2024.findings-acl.372),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.372)Cited by:[§2](https://arxiv.org/html/2508.14817#S2.p3.1)\.
- \[24\]T\. Yu, A\. Xu, and R\. Akkiraju\(2024\)In defense of rag in the era of long\-context language models\.arXiv preprint arXiv:2409\.01666\.Cited by:[§2](https://arxiv.org/html/2508.14817#S2.p4.1)\.
- \[25\]C\. Zakka, R\. Shad, A\. Chaurasia, A\. R\. Dalal, J\. L\. Kim, M\. Moor, R\. Fong, C\. Phillips, K\. Alexander, E\. Ashley,et al\.\(2024\)Almanac—retrieval\-augmented language models for clinical medicine\.Nejm ai1\(2\),pp\. AIoa2300068\.Cited by:[§2](https://arxiv.org/html/2508.14817#S2.p3.1)\.
- \[26\]Y\. Zhang, M\. Li, D\. Long, X\. Zhang, H\. Lin, B\. Yang, P\. Xie, A\. Yang, D\. Liu, J\. Lin, F\. Huang, and J\. Zhou\(2025\)Qwen3 embedding: advancing text embedding and reranking through foundation models\.arXiv preprint arXiv:2506\.05176\.Cited by:[§3\.2](https://arxiv.org/html/2508.14817#S3.SS2.p1.1)\.

## 7Appendices

### 7\.1A\. Additional Results

Modality\+Time\+LocationCategory\+TimeCategory\+Time \(±1 day\)\# chunks/tokensPRFPRFPRFGPT\-5\.4\-miniRAG \(BGE\)20 chunks48\.9329\.3436\.6865\.9939\.5749\.4780\.4748\.2560\.3340 chunks48\.7936\.5441\.7965\.4148\.9956\.0277\.3657\.9466\.2660 chunks47\.0440\.1143\.362\.8353\.5757\.8373\.0962\.3167\.27RAG \(Qwen3\)20 chunks49\.7431\.9738\.9262\.4140\.1148\.8372\.9846\.957\.1140 chunks55\.4740\.2446\.6570\.7851\.3559\.528259\.4968\.9560 chunks55\.3543\.1448\.4971\.2455\.5262\.4181\.0963\.1971\.03Recent Notes256036\.493\.636\.6158\.115\.7910\.5368\.246\.812\.36512040\.195\.7910\.1260\.758\.7515\.2971\.9610\.3618\.12768047\.018\.4814\.3763\.0611\.3719\.2773\.8813\.3222\.5864K52\.5936\.8843\.3571\.7950\.3459\.1880\.6156\.5366\.46120K50\.8646\.0348\.3269\.5262\.9266\.0579\.7872\.2175\.8Mistral Medium 3RAG \(BGE\)20 chunks51\.9635\.6742\.365\.5945\.0253\.3978\.1453\.6363\.6140 chunks52\.5243\.4147\.5367\.4355\.7261\.0277\.3663\.9370\.0160 chunks51\.9348\.7950\.3165\.9761\.9863\.9174\.9370\.3972\.59RAG \(Qwen3\)20 chunks52\.9437\.5543\.9464\.3345\.6353\.3973\.3452\.0260\.8740 chunks58\.6445\.6951\.3672\.4556\.4663\.4681\.5263\.5371\.4160 chunks58\.2151\.5554\.6872\.1963\.9367\.8181\.3172\.0176\.37Recent Notes256035\.964\.317\.6951\.126\.1210\.9458\.997\.0712\.62512037\.456\.9311\.753\.829\.9616\.8165\.8212\.1820\.56768039\.2710\.0916\.0650\.7913\.0620\.7760\.7315\.6124\.8464K43\.0840\.4441\.7256\.9953\.555\.1963\.8759\.9661\.85120K45\.9443\.8144\.8560\.8358\.0159\.3968\.8865\.6867\.24DeepSeek V3\.1RAG \(BGE\)20 chunks51\.3533\.2440\.3666\.424352\.2181\.0852\.4963\.7340 chunks55\.2940\.7846\.9471\.2652\.5660\.583\.4961\.5770\.8860 chunks57\.2545\.6950\.8272\.6858\.0164\.5283\.0566\.2973\.73RAG \(Qwen3\)20 chunks55\.2234\.1942\.2369\.5743\.0753\.28049\.5361\.1840 chunks58\.243\.9450\.0872\.6454\.8562\.582\.4462\.2570\.9460 chunks57\.8246\.551\.5573\.3158\.9565\.3582\.8566\.6273\.85Recent Notes256027\.193\.976\.9337\.335\.459\.5148\.397\.0712\.33512030\.995\.929\.9446\.138\.8214\.855\.6310\.6317\.85768035\.767\.941351\.5211\.4418\.7260\.9113\.5322\.1464K53\.1535\.842\.7874\.3350\.0759\.8384\.4256\.8667\.95120K53\.7645\.6949\.473\.6362\.5867\.6685\.1972\.4178\.28

Table A1:Scores for theImaging Procedurestask, across different models and differing amounts of provided clinical notes\.Timespan OverlapMedications \(name only\)\# chunks/tokensJaccard indexPRF1GPT\-5\.4\-miniRAG \(BGE\)20 chunks0\.46950\.72320\.84910\.781140 chunks0\.49080\.71390\.88490\.790360 chunks0\.49570\.70720\.88650\.7867RAG \(Qwen3\)20 chunks0\.44260\.67790\.81180\.738940 chunks0\.47360\.68810\.86470\.766460 chunks0\.49820\.69580\.87870\.7766Recent Notes25600\.31420\.71760\.48210\.576751200\.39760\.7510\.58160\.655676800\.43520\.75940\.66250\.707664K0\.53860\.71580\.85380\.7787120K0\.53080\.6950\.86470\.7706Mistral Medium 3RAG \(BGE\)20 chunks0\.50490\.76890\.80720\.787640 chunks0\.5620\.78890\.83670\.812160 chunks0\.57290\.77180\.85230\.8101RAG \(Qwen3\)20 chunks0\.50040\.7440\.77290\.758240 chunks0\.54570\.76220\.82740\.793460 chunks0\.55980\.75070\.84760\.7962Recent Notes25600\.37480\.77380\.40440\.531251200\.40420\.7740\.52720\.627276800\.47030\.76260\.63450\.692764K0\.55730\.76880\.81180\.7897120K0\.54050\.7660\.74340\.7545DeepSeek V3\.1RAG \(BGE\)20 chunks0\.54410\.78580\.79320\.789540 chunks0\.5990\.80030\.82270\.811360 chunks0\.60280\.79880\.80870\.8037RAG \(Qwen3\)20 chunks0\.52230\.75570\.77450\.76540 chunks0\.5530\.77460\.80720\.790660 chunks0\.58180\.78280\.80720\.7948Recent Notes25600\.37130\.7180\.47120\.56951200\.46140\.75660\.53650\.627876800\.51350\.80120\.60810\.691464K0\.61780\.79810\.78690\.7925120K0\.61620\.79060\.81030\.8003Rule\-based MAR baseline0\.4648

Table A2:Scores for theAntibiotic Timelinestask, across different models and differing amounts of provided clinical notes\.Billing CodesDischarge SummaryFiltered\# chunks/tokensPRF1PRF1PRF1GPT\-5\.4\-miniRAG \(BGE\)20 chunks0\.5620\.33690\.42130\.28150\.59440\.3820\.30240\.53250\.385740 chunks0\.56410\.37770\.45240\.27430\.64690\.38520\.29340\.57710\.38960 chunks0\.57060\.38460\.45950\.28230\.67040\.39730\.30170\.59750\.4009RAG \(Qwen3\)20 chunks0\.59550\.3590\.4480\.28430\.60380\.38650\.30240\.53560\.386540 chunks0\.60290\.38650\.4710\.28480\.64320\.39480\.29650\.55830\.387360 chunks0\.59030\.40090\.47750\.2810\.67230\.39630\.30060\.59980\.4005Recent Notes25600\.59090\.30680\.40390\.31780\.58120\.41090\.33980\.51840\.410551200\.59930\.34430\.43740\.31350\.63470\.41970\.3270\.55210\.410776800\.59450\.37050\.45650\.2990\.65630\.41080\.31650\.57950\.409464K0\.58630\.44560\.50640\.26470\.70890\.38550\.28120\.6280\.3885120K0\.59590\.44620\.51030\.27020\.71270\.39180\.28910\.63590\.3975Mistral Medium 3RAG \(BGE\)20 chunks0\.56510\.34010\.42460\.26570\.56340\.36110\.28260\.49960\.36140 chunks0\.56920\.39790\.46840\.260\.64040\.36980\.27370\.56230\.368260 chunks0\.58480\.42940\.49520\.26030\.67320\.37540\.27330\.58970\.3735RAG \(Qwen3\)20 chunks0\.56270\.36380\.44190\.25470\.58030\.35410\.26590\.50510\.348440 chunks0\.57160\.4030\.47270\.26010\.6460\.37090\.26620\.55130\.35960 chunks0\.56810\.43340\.49170\.24810\.66670\.36160\.25790\.57790\.3566Recent Notes25600\.58540\.30140\.3980\.31060\.56340\.40040\.31940\.48320\.384551200\.59540\.34360\.43570\.31270\.63570\.41920\.30850\.52310\.388176800\.5820\.3830\.4620\.2920\.6770\.4080\.29160\.56380\.384464K0\.50110\.49490\.4980\.21210\.7380\.32950\.21590\.62650\.3211120K0\.50390\.46670\.48460\.21090\.68830\.32290\.21810\.59360\.319DeepSeek V3\.1RAG \(BGE\)20 chunks0\.57190\.29370\.38810\.28910\.5230\.37230\.30460\.45970\.366440 chunks0\.58680\.33610\.42740\.28940\.5840\.38710\.30570\.51450\.383560 chunks0\.60330\.36430\.45430\.28820\.61310\.39210\.30450\.54030\.3895RAG \(Qwen3\)20 chunks0\.59640\.30170\.40070\.2950\.52580\.3780\.30980\.46050\.370440 chunks0\.60040\.35550\.44660\.28490\.59440\.38520\.29840\.51920\.37960 chunks0\.5970\.37390\.45980\.28510\.62910\.39240\.29570\.54420\.3832Recent Notes25600\.56830\.26390\.36040\.31970\.5230\.39690\.32660\.44560\.376951200\.59810\.31610\.41360\.32020\.59620\.41670\.32730\.50820\.398276800\.61870\.34040\.43910\.32170\.62350\.42440\.33190\.53640\.410164K0\.59290\.39550\.47450\.28050\.65920\.39350\.29560\.57950\.3915120K0\.58020\.39340\.46890\.27480\.65630\.38740\.29250\.58260\.3894

Table A3:Scores for theDiagnosis Generationtask, across different models and differing amounts of provided clinical notes\.
### 7\.2B\. Prompts

#### 7\.2\.1Imaging Procedures

\#Task:IdentificationofImagingProceduresfromElectronicHealthRecords

YouareanAIassistanttaskedwithidentifyingimagingproceduresperformedonapatientduringtheirhospitalstay\.YouwillbeprovidedwithrelevantpassagesfromthepatientsElectronicHealthRecord\.

\#\#Instructions

1\.CarefullyreadthroughallprovidedEHRpassagesandidentifytheimagingproceduresperformedandthelocationimaged\.

2\.Createabulletedlistoftheimagingproceduresperformedduringthiscurrenthospitalization,DONOTincludeproceduresfromtheirhistorythatoccurredpriortothisstayandDONOTincludeimagingthatwasonlyperformedforguidanceduringanotherprocedure\.

3\.ONLYincludeproceduresfromtheseprimarycategories:MRI\(MagneticResonanceImaging\),CT\(ComputedTomography\),Ultrasound\(US\),X\-ray\(Radiograph\),NMImaging\(NuclearMedicine\)

4\.Foreachprocedurefound,youmustidentify:

\-Theprimaryimagingmodality\(fromthelistabove\)

\-Whetherthereisamorespecificsubtypeofthemodality\(e\.g\.,Fluoroscopy,Mammography,Echocardiography,PET,Angiography\)

\-Thetimeoftheimaging\(asMM/DDformator"unknown"ifthiscannotbedetermined\)

\-Thespecificbodylocationthatwasimaged\(e\.g\.,brain,chest,leftankle\)

5\.Ignoreallothermedicalinformationsuchastests,medications,treatmentplans,assessments,othernon\-imagingprocedures\.

\#\#OutputFormat

\-UseEXACTLYthisformatforeachitem:"\-\(MM/DDor"unknown"\)\[PrimaryImagingModality\]\-\[Subtypeor"None"\]:\[BodyLocation\]"

\-Use"None"inplaceofsubtypewhennospecificsubtypeisnecessaryand"unknown"inplaceofthe’

dateifthetimeoftheimagingcannotbedetermined\.

\-Presentasacleanbulletedlist

\-Includenoexplanatorytext,introductions,orconclusions

\-Donotnumbertheitems

\-Ifmultipleimagingproceduresofthesametypewereperformedondifferentlocations,listeachseparately

\-ThesameimagingoccurencemaybementionedmultipletimesthroughouttheEHR,onlyincludeoneentryperoccurence\.

\-Ifnoimagingproceduresexist,outputonly:"Noimagingproceduresidentified\."

\#\#Example

2018\-03\-1215:43:00H&P

Patientadmittedwithchestpain\.Cardiacenzymeswereelevated\.AchestX\-raywasperformedon3/10showingcardiomegaly\.CTscanofthechestwascompletedtoruleoutaorticdissection\.Thepatientalsounderwentatransthoracicechocardiogramtoday\.

Exampleoutput:

\-\(03/10\)X\-ray\-None:Chest

\-\(unknown\)CT\-None:Chest

\-\(03/12\)Ultrasound\-Echocardiogram:Heart

\#\#BeginTask

EHRpassages:

\[INSERTTEXT\]

Yourresponseasalistofimagingtypesandlocations:

#### 7\.2\.2Antibiotic Timelines

\#Task:IdentificationofAdministeredAntibioticsandDateRangesfromElectronicHealthRecords\\n\\n

YouareanAIassistanttaskedwithidentifyingantibioticsadministeredtoapatientduring

theirhospitalstayandthedaterangesforeachantibiotic’suse\.Youwillbe

providedwithrelevantpassagesfromthepatient’sElectronicHealthRecord\(EHR\),eachwithan

associatedtimestamp\.\\n\\n

\#\#Instructions\\n

1\.CarefullyreadthroughallprovidedEHRpassages,notingtheirtimestamps\.\\n

2\.Createalistofantibioticsbeingadministered,prescribed,orcontinued\.\\n

3\.Donotincludeantibioticsgivenforprophylaxisorminorconditions\.OnlyincludeantibioticsbeingusedforthetreatmentofthemajoracuteconditionoftheICUpatient\.\\n

4\.Foreachantibiotic,determinethestartandenddatesofitsusebyinferringfromthetimestampsofthepassagesandanydateinformationwithinthetext\.\\n

5\.UsetheformatMM/DD\-MM/DDfordateranges\.Iftheantibioticuseisongoing,use\\"present\\"fortheenddate\.\\n

6\.Don’tincludedosagesoradministrationroutes\.\\n

7\.Ifadaterangecan’tbedeterminedwhatsoever,listtheantibioticwith\\"\(datesunclear\)\\"afterit\.\\n

\\n

\#\#OutputFormat\\n

Provideyourresponseasalistofantibioticswiththeirdaterangesinthefollowingformat:\\n

\-Antibiotic1\(MM/DD\-MM/DD\)\\n

\-Antibiotic2\(MM/DD\-present\)\\n

\#\#Example\\n

Rightnowitis2019\-09\-1514:51:00\.\\n

EHRpassages:\\n\\n

2019\-09\-1210:15:00\\n

\\"Patientadmittedwithsuspectedpneumonia\.StartedonIVceftriaxone1gdaily\.\\"\\n\\n

2019\-09\-1414:30:00\\n

\\"BloodculturespositiveforMRSA\.Ceftriaxonediscontinued\.StartedonIVvancomycin1gq12h\.\\"\\n\\n

Output:\\n

\-Ceftriaxone\(09/12\-09/14\)\\n

\-Vancomycin\(09/14\-ongoing\)\\n\\n

\#\#BeginTask\\n

Rightnowitis\[TIMESTAMP\]\.\\n\\nEHRpassages:\\n\\n\[INSERTTEXT\]

\\n\\nYourresponseasalistofantibioticnamesanddateranges:\\n

#### 7\.2\.3Diagnosis Generation

\#Task:IdentificationofClinicallyImportantDiagnoses

YouareanAIassistanttaskedwithcreatingaclinicallyrelevantproblemlistforanICUpatient’sstay\.Youwillanalyzepassagesofclinicalnotesfromtheirhospitalizationandidentifydiagnosesthatrequiredactivemanagementormonitoringduringtheirstay\.

\#Task

Reviewtheprovidedclinicalnotepassagesandgenerateastructuredlistofdiagnosesthat:

1\.Requiredactivemanagementduringthehospitalization

2\.Wereclinicallysignificanttotheircriticalcarecourse

3\.ImpactedtheirICUcareplanoroutcomes

\#InclusionCriteria

IncludediagnosesthatmeetANYofthesecriteria:

\-AcuteconditionsrequiringICU\-levelcare\(e\.g\.,NSTEMI,septicshock,acuterespiratoryfailure\)

\-ChronicconditionsrequiringactivemanagementoraffectingICUcare\(e\.g\.,atrialfibrillation,COPDexacerbation\)

\-Newdiagnosesmadeduringtheencounter

\-Complicationsthatdevelopedduringthestay

\-Conditionsrequiringmonitoringorintervention\(e\.g\.,acutekidneyinjury,severeelectrolytedisorders\)

\-Neurologic/cognitiveconditionsaffectingICUcare\(e\.g\.,delirium,acutestroke\)

\-ConditionsdirectlyrelatedtothereasonforICUadmission

\#ExclusionCriteria

DoNOTinclude:

\-Stablechronicconditionsnotrequiringactivemanagement\(e\.g\.,well\-controlleddiabetes,stablehypothyroidism\)

\-Historicaldiagnosesnotaffectingcurrentcare\(e\.g\.,"historyofappendectomy"\)

\-Socialhistoryitems\(e\.g\.,"formersmoker"\)

\-Proceduralorpost\-surgicalstatuses\(e\.g\.,"s/pCABG","post\-cholecystectomy"\)

\-Symptomswithoutcleardiagnoses

\-Conditionsthatresolvedpriortoadmission

\-Incidentalfindingsnotrequiringintervention

\#OutputFormat

Presentthediagnosesasanumberedlist,orderedbyclinicalpriority:

1\.\[Primarydiagnosis\]

2\.\[Secondarydiagnosis\]

3\.\[Tertiarydiagnosis\]

\.\.\.

\#Note

\-Everylisteditemmustbeaspecificmedicaldiagnosis\-Usestandardmedicalterminologyfordiagnoses

\#EHRpassages:\\n\[INSERTTEXT\]\\n

\#OutputthediagnosesthatrequiredactivemanagementormonitoringduringtheirICUstay,asinstructed\.Everylisteditemmustbeaspecificmedicaldiagnosis:

### 7\.3C\. SNOBERT Training

We trained SNOBERT using the configuration provided in the authors’ Github Repository as\-is with the same training data from the SNOMED CT Entity Linking Challenge, but using the International SNOMED vocabulary files from 2025, since we did not have access to the version used for the challenge\. We trained a single model, as opposed to their approach of ensembling six with varying class weights and data splits for the competition, but through expert review, we determined performance on the downstream ICD\-10 code extraction step to be acceptable\.

### 7\.4D\.FilteredDiagnoses Validation

To assess the reliability of the Gemma\-3\-32B filtering step, in which the full ICD\-10 billing code list is reduced to the diagnoses directly relevant to the text in the discharge summary, a medical physcian independently performed the same filtering task on a random sample of 20 hospitalizations\. Given the discharge summary text and the unfiltered list of billing codes, the physcian produced their own filtered list under the same instructions as Gemma\-3\-32B\. The model’s output was then evaluated against this manual annotation, resulting in 86\.72 precision, 92\.50 recall, and 89\.52 F1\. We interpret this as evidence that theFilteredgold standard reasonably approximates clinician judgment for this filtering task\.

### 7\.5E\. Free\-text to ICD\-10 Normalization

To assess the reliability of the text normalization pipeline used in theDiagnosis Generationtask \(comprising SNOBERT\-based SNOMED extraction followed by the SNOMED\-to\-ICD\-10 mapping from the SNOMED CT release\), a medical physcian manually reviewed the pipeline’s output on 40 cases drawn from our test set: 20 discharge summary normalizations and 20 LLM\-generated output normalizations\. For each case, the physcian was provided with the source free text and the resulting ICD\-10 code list, and assigned a score from 1 to 5 reflecting how faithfully the normalized codes captured the clinically significant diagnoses present in the text\. The rubric descriptions are shown in Table[A4](https://arxiv.org/html/2508.14817#S7.T4)and the distribution of scores is shown in Table[A5](https://arxiv.org/html/2508.14817#S7.T5)\.

Table A4:Rubric for manual evaluation of free\-text to ICD\-10 normalization\.ScoreDischarge SummariesLLM Outputs55 \(25%\)9 \(45%\)46 \(30%\)5 \(25%\)36 \(30%\)5 \(25%\)23 \(15%\)1 \(5%\)10 \(0%\)0 \(0%\)Mean3\.654\.10

Table A5:Manual evaluation of the SNOBERT \+ SNOMED\-to\-ICD\-10 normalization pipeline\. Scores of 4 or higher indicate that core diagnoses were captured with at most minor errors\.Cases with a score of 4 or higher \(indicating that core diagnoses were captured correctly with at most minor specificity issues\) accounted for 55% of discharge summary normalizations and 70% of LLM output normalizations\. No cases were rated as largely wrong \(score 1\)\. Notably, normalization quality was somewhat higher on LLM outputs than on discharge summaries, likely because LLM outputs tend to use more explicit, standardized diagnostic phrasing, whereas discharge summaries contain more abbreviations, narrative phrasing, and inferred diagnoses that pose challenges for the SNOBERT model\.