MedicalBench: Evaluating Large Language Models Toward Improved Medical Concept Extraction

# Evaluating Large Language Models Towards Improved Medical Concept Extraction
Source: [https://arxiv.org/html/2605.20197](https://arxiv.org/html/2605.20197)
Gregory D\. Lyng∗ Optum AISanjit Singh Batra∗ Optum AIRobert E\. Tillman Optum AICorrespondence to:gregory\.lyng, sanjit\.batra, rob\.tillman @optum\.com

###### Abstract

Medical concept extraction from electronic health records underpins many downstream applications, yet remains challenging because medically meaningful concepts, such as diagnoses, are frequently implied rather than explicitly stated in medical narratives\. Existing benchmarks with human\-annotated evidence spans underscore the importance of grounding extracted concepts in medical text\. However, they predominantly focus on explicitly stated concepts and provide limited coverage of cases in which medically relevant concepts must be inferred\. We present MedicalBench, a new benchmark for medical concept extraction with evidence grounding that evaluates implicit medical reasoning\. MedicalBench formulates medical concept extraction as a verification task over medical note–concept pairs, coupled with sentence\-level evidence identification\. Built from MIMIC\-IV discharge summaries and human\-verified ICD\-10 codes, the dataset is curated through a multi\-stage large language model \(LLM\) triage pipeline followed by medical annotation and expert review\. It deliberately includes implicit positives, semantically confusable negatives, and cases where LLM judgments disagree with medical expert assessments\. Annotators provide sentence\-level evidence spans and concise medical rationales\. The final dataset contains 823 high\-quality examples\. We define two complementary evaluation tasks: \(1\) medical concept extraction and \(2\) sentence\-level evidence retrieval, enabling assessment of both correctness and interpretability\. Benchmarking state\-of\-the\-art LLMs and a supervised baseline reveals that performance remains modest, highlighting the difficulty of extracting implicitly expressed concepts\. We further show that explicitly incorporating reasoning cues and prompting to extract implicit evidence substantially improves medical concept extractions, while performance is largely invariant to note length, indicating that MedicalBench isolates reasoning difficulty rather than superficial confounders\. MedicalBench provides the first systematic benchmark for implicit, evidence\-grounded medical concept extraction, offering a foundation for developing medical language models that can both identify medically relevant concepts and justify their predictions in a transparent and medically faithful manner\.111MedicalBench dataset is available at[PhysioNet](https://physionet.org/content/mimic-iv-ext-medicalbench/1.0.0/)\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2605.20197v1/figs/figure1_bar.png)Figure 1:Medical concept extraction F1 across medical experts and LLMs on our MedicalBench\.Medical concept extraction aims to identify and structure key medical facts from unstructured clinical narratives such as discharge summaries, progress notes, and oncology reports\(Wiestet al\.,[2025](https://arxiv.org/html/2605.20197#bib.bib11); Wuet al\.,[2024](https://arxiv.org/html/2605.20197#bib.bib36); Sushilet al\.,[2024](https://arxiv.org/html/2605.20197#bib.bib34); Landolsiet al\.,[2022](https://arxiv.org/html/2605.20197#bib.bib44)\)\. Accurate extraction of diagnostic events is fundamental for building reliable clinical knowledge systems and it powers numerous downstream applications\(Huanget al\.,[2024](https://arxiv.org/html/2605.20197#bib.bib14); Goldsteinet al\.,[2016](https://arxiv.org/html/2605.20197#bib.bib37)\)\. For example, converting free\-text notes into structured diagnoses and findings can aid cohort identification for research\(Afsharet al\.,[2025](https://arxiv.org/html/2605.20197#bib.bib39); Heinet al\.,[2025](https://arxiv.org/html/2605.20197#bib.bib9); Sarmiento and Dernoncourt,[2016](https://arxiv.org/html/2605.20197#bib.bib13)\), improve clinical outcome prediction\(Sunet al\.,[2025](https://arxiv.org/html/2605.20197#bib.bib10); Seinenet al\.,[2022](https://arxiv.org/html/2605.20197#bib.bib17); Shukla and Marlin,[2020](https://arxiv.org/html/2605.20197#bib.bib38)\), and streamline automated documentation processes\(Shahidet al\.,[2025](https://arxiv.org/html/2605.20197#bib.bib16); Daset al\.,[2025](https://arxiv.org/html/2605.20197#bib.bib8); Lauet al\.,[2021](https://arxiv.org/html/2605.20197#bib.bib43)\)\. Robust medical concept extraction underpins many secondary uses of electronic health records \(EHRs\) by making implicit information explicit and machine\-readable\(Gaoet al\.,[2024](https://arxiv.org/html/2605.20197#bib.bib4); Tsaiet al\.,[2020](https://arxiv.org/html/2605.20197#bib.bib18); Xiaoet al\.,[2018](https://arxiv.org/html/2605.20197#bib.bib15)\)\.

A core challenge in medical concept extraction is that medical concept related information in narrative text is often implicit, fragmented, and context\-dependent\(Pereraet al\.,[2015](https://arxiv.org/html/2605.20197#bib.bib33); Meystreet al\.,[2008](https://arxiv.org/html/2605.20197#bib.bib35)\)\. For instance, a note might report a “low hemoglobin level” as a finding \(suggesting concept anemia\) or list a body mass index “BMI 37” \(indicating concept obesity\) without explicitly using the words anemia or obesity\. Empirical studies have shown that such implicit documentation is widespread: although more than half of patients may meet objective criteria for obesity, only a small fraction have the condition explicitly recorded in structured problem lists\(Mattaret al\.,[2017](https://arxiv.org/html/2605.20197#bib.bib42)\)\.

One prominent line of work that exposes this limitation frames medical concept extraction as a verification and grounding problem: rather than merely identifying which concepts may apply to a patient, automated extraction models must determine whether each candidate concept is supported by the clinical narrative and identify the corresponding evidence\. Recent datasets with expert\-annotated evidence spans demonstrate the importance of grounding extracted concepts in text, but they remain limited on explicitly stated concepts and offer limited coverage of cases where concepts must be infered from indirect evidences\(DeYounget al\.,[2022](https://arxiv.org/html/2605.20197#bib.bib19); Chenget al\.,[2023](https://arxiv.org/html/2605.20197#bib.bib30); Beckhet al\.,[2025](https://arxiv.org/html/2605.20197#bib.bib5)\)\. More broadly, most existing benchmarks emphasize concept presence or absence at the document level, while providing little insight into why a concept is warranted or where the supporting evidence resides\(Mullenbachet al\.,[2018](https://arxiv.org/html/2605.20197#bib.bib12)\)\.

In this work, we introduce MedicalBench, a benchmark designed to evaluate medical concept extraction with explicit evidence grounding in realistic medical notes\. MedicalBench formulates the task as follows: given a medical note and a candidate medical concept, the model must determine whether the concept is supported by the text and, if so, locate the relevant evidence spans\. Using discharge summaries from MIMIC\-IV and medical concept labels, we construct challenging positive and negative examples through a multi\-stage large language model \(LLM\) triage pipeline followed by rigorous expert annotation\. The dataset emphasizes implicit cases and semantically confusable negatives, with expert annotators marking supporting evidence and providing explanatory rationales\. To ensure annotation quality, we incorporate control labels derived from LLM\-based heuristics to assess annotator reliability\. The resulting dataset comprises 823 high\-quality, expert\-annotated examples and, to our knowledge, represents the first benchmark to systematically evaluate implicit medical concept extraction with verified evidence grounding\. Across a range of large language models, we find that performance remains limited, with the highest extraction F1 score below 0\.6 \(Figure[1](https://arxiv.org/html/2605.20197#S1.F1)\), underscoring the difficulty of this task\. MedicalBench provides a foundation for the development and evaluation of medical language models that can both identify medically meaningful concepts and justify their predictions in a transparent and medically faithful manner\.

## 2Data collection

### 2\.1Data Source and Candidate Concepts

We constructed the dataset using the MIMIC\-IV database, which contains de\-identified electronic health records of patients admitted to the Beth Israel Deaconess Medical Center\(Johnsonet al\.,[2023a](https://arxiv.org/html/2605.20197#bib.bib23)\), through both MIMIC\-IV repository\(Johnsonet al\.,[2024](https://arxiv.org/html/2605.20197#bib.bib24)\)and MIMIC\-IV\-Note repository\(Johnsonet al\.,[2023b](https://arxiv.org/html/2605.20197#bib.bib25)\)from physionet\(PhysioBank,[2000](https://arxiv.org/html/2605.20197#bib.bib26)\)\. FollowingMullenbachet al\.\([2018](https://arxiv.org/html/2605.20197#bib.bib12)\), we extracted the discharge summary as the representative medical note for each hospital admission \(‘hadm\_id‘\)\. Each admission was then mapped to all associated ICD\-10 diagnoses in the ‘diagnoses\_icd‘ table and ICD\-10 procedures in the ‘procedures\_icd‘ table\. Because these billing codes are assigned and reviewed by professional medical coders, we treated them as the clinical concepts of interest\. Alongside each code, we preserved metadata including the ICD chapter, parent hierarchy, and textual description for prompting and analysis\.

### 2\.2LLM\-based Triage for Positive Candidates

To identify challenging cases where LLMs may disagree with human experts, we performed a two\-stage triage process\. In the first stage, a non\-reasoning LLM \(GPT\-4o\-mini\) was prompted with the discharge summary and the candidate medical concepts, and asked to classify each note–concept pair into one of three categories: Explicit \(the concept is explicitly supported by text spans in the note\), Implicit \(the concept is supported but requires clinical inference, e\.g\., “low hemoglobin level” implying anemia\), or Unrelated \(the concept is not supported\)\. Each pair is prompted 3 times, and the final answer is achieved by majority vote\. Explanations and evidence spans \(character offsets\) were stored internally\. The specific prompt is shown in Section[A\.1](https://arxiv.org/html/2605.20197#A1.SS1)\.

In the second stage, previous disagreements were re\-evaluated using a reasoning LLM \(o3\)\. We retained only those cases where both LLMs judged the concept as “Unrelated”, even though the concept \(ICD\-10 code\) was assigned in MIMIC\-IV for that admission\. This subset \(putative positive\) represents candidate false negatives for LLMs and forms the basis for challenging positive examples\.

### 2\.3Negative Sampling Strategy

To construct difficult negative examples, we designed two complementary sampling strategies\.

Prevalence\-weighted negatives: For each discharge summary, we sampled candidate medical concepts according to their prevalence in MIMIC\-IV\. To characterize their difficulty, we additionally recorded the number of hops in the ICD hierarchy between each sampled negative and the closest related concept\.

Semantically similar negatives: To increase semantic confusion, we embedded medical concept name and sampled the most similar but unrelated concept\. For example, Obesity and BMI 30–39 both capture obesity\-related conditions but belong to different categories in the ICD hierarchy \(disease vs\. BMI classification\)\. These hard negatives are designed to test fine\-grained reasoning about whether a concept is truly documented in the note\.

Together, these strategies minimize the presence of trivial negatives and enrich the dataset with medically challenging contrasts\.

### 2\.4Annotation Candidate Classes

We recruited 9 independent expert annotators with medical training to serve as the final authority\. Two annotators were presented with a discharge summary and a candidate medical concept and asked to label the pair as Related or Unrelated\. For Related cases, annotators highlighted evidence spans \(character offsets\) in the note and provided a short textual justification\. Only cases where both annotators agreed were retained in the dataset\.

To ensure label quality, all annotations underwent a two\-stage review:

- •Initial Review – An automated program compared the two independent annotations for each note–concept pair and refined highlights as needed\. The program flagged mismatches between classification \(related / unrelated\); these issues were resolved before proceeding\.
- •SME Review – A subject matter expert independently verified the selected label for correctness, completeness, and consistency, ensuring evidences were medically sound\.

We recruited 9 independent expert annotators with medical training\. For each note and concept pair, two annotators independently reviewed the discharge summary and labeled the concept asRelatedorUnrelated\. ForRelatedpairs, annotators additionally highlighted sentence\-level evidence spans \(recorded as character offsets\) and provided a short clinical justification\.

Disagreements between the two annotators were not included without adjudication\. Specifically, if the initial labels disagreed \(Relatedvs\.Unrelated\), the example was flagged for resolution and reviewed to determine a single final label; pairs that could not be resolved to a clear final decision were excluded from the release\. The final dataset therefore contains only examples with a verified gold label and accompanying evidence when applicable\.

Through this pipeline, we curated a final dataset of 352 gold positives \(explicit/implicit\) and 471 gold negatives \(unrelated\)\. The resulting dataset emphasizes implicit reasoning, semantic ambiguity, and LLM disagreement, providing a challenging benchmark for medical concept extraction models\.

## 3Evaluation metrics

To evaluate performance on this dataset, we define two subtasks\.

\(A\) Medical Concept Extraction: Given a medical note and a candidate medical concept, the system predicts whether the note documents that concept\. We compute micro\-averaged Precision, Recall, and F1 across all cases, where a true positive is defined as predicting True when the gold label is True\.

\(B\) Sentence\-Level Evidence Retrieval: For cases with expert\-annotated evidence, we measure how well a system retrieves the correct sentences\. We report macro\-averaged sentence recall, defined as the proportion of gold evidence sentences correctly retrieved\. If no sentences are returned for a case with gold evidence, then the recall for this case is 0\.

Together, these metrics assess both the correctness of concept extraction and the interpretability of the reasoning process\.

![Refer to caption](https://arxiv.org/html/2605.20197v1/figs/figure1.png)Figure 2:Performance of LLMs on the medical concept extraction task \(y\-axis\) and evidence retrieval task \(x\-axis\)\.
## 4Results

We report MedicalBench results for different models and analyze performance by concept extraction \(Section[4\.1](https://arxiv.org/html/2605.20197#S4.SS1)\), evidence retrieval \(Section[4\.2](https://arxiv.org/html/2605.20197#S4.SS2)\), relationship between concepts and evidences \(Section[4\.3](https://arxiv.org/html/2605.20197#S4.SS3)\), the need for implicit prompt \(Section[4\.4](https://arxiv.org/html/2605.20197#S4.SS4)\), relationship between extraction and note length \(Section[4\.5](https://arxiv.org/html/2605.20197#S4.SS5)\), and case studies \(Section[4\.6](https://arxiv.org/html/2605.20197#S4.SS6)\)\.

### 4\.1Medical Concept Extraction

![Refer to caption](https://arxiv.org/html/2605.20197v1/figs/figure2_prcurve.png)Figure 3:Precision–recall comparison of LLMs and a pretrained language model \(PLM\-ICD\), which is an encoder with≈125​M\\approx 125Mparameters, on the medical concept extraction task\.To assess medical concept extractions, we evaluate each model’s ability to judge whether medical note documents the candidate medical concept\. The y\-axis of Figure[2](https://arxiv.org/html/2605.20197#S3.F2)shows medical concept extraction F1\. Gemini\-3\-pro\-preview achieves the best F1 score of 0\.59, while Claude\-opus\-4\.5, GPT\-5, Gemini\-2\.5\-pro achieve F1 scores of 0\.55, 0\.54 and 0\.53\. The earlier reasoning model \(O3\) achieves F1 of 0\.48 and non\-reasoning model \(GPT\-4\.1\) achieves 0\.46\. Models with more parameters perform better\. For example, Claude\-opus\-4\.5 \(0\.55\) outperforms its sonnet version \(0\.45\) and haiku version \(0\.41\)\. Across all systems, F1 scores remain below 0\.60, reflecting the inherent difficulty of MedicalBench, which requires nuanced understanding of implicit disease mentions and fine\-grained reasoning over long notes\.

Figure[3](https://arxiv.org/html/2605.20197#S4.F3)presents a precision–recall comparison between LLMs and a supervised pretrained language model baseline fine\-tuned for medical concept extraction \(PLM\-ICD\)\. While the PLM establishes a traditional discriminative frontier, Gemini\-3\-pro\-preview, GPT\-5, and Grok\-4 surpass it even in zero\-shot settings\. They achieve the best overall balance between precision and recall, indicating strong ability to identify correct medical concepts while minimizing false positives\. The OpenAI o3 model performs comparably, slightly favoring higher precision over recall, whereas GPT\-4\.1 and GPT\-4o exhibit lower precision due to more permissive coding behavior\. The area under the precision\-recall curve is 0\.55, further confirming that MedicalBench poses a challenging benchmark for traditional BERT based methods as well\.

### 4\.2Evidence Retrieval

To assess interpretability, we evaluate each model’s ability to retrieve the sentences that support its medical concept extraction \(X\-axis of Figure[2](https://arxiv.org/html/2605.20197#S3.F2)\)\. Gemini\-3\-pro\-preview again leads, achieving the highest evidence recall, followed by Claude\-opus\-4\.5, Gemini\-2\.5\-pro, and GPT\-5, which also retrieve relevant evidence with strong precision\. Models with smaller parameter counts \(e\.g\., GPT\-5\-nano, GPT\-4\.1\-nano\) display diminished evidence recall, consistent with their weaker text comprehension and contextual grounding\.

### 4\.3Improved Medical Concept Extraction via Evidence Retrieval cues

From Figure[2](https://arxiv.org/html/2605.20197#S3.F2), models that better retrieve supporting evidence also tend to make more correct medical concept extractions, suggesting that evidence benefits concept extraction\.

To further examine this link, we paraphrased the retrieved evidence into reasoning cues \(e\.g\., “The note describes a low hemoglobin level, which indicates anemia\.”\) and incorporated these statements directly into the prompt and then re\-evaluated the medical concept extraction task\. Table[1](https://arxiv.org/html/2605.20197#S4.T1)compares GPT\-5’s performance with and without the inclusion of such reasononing cues\.

Table 1:GPT\-5 Performance metrics comparison with and without reasoning cuesProviding reasoning cues leads to a substantial performance gain: the F1 score increases from 0\.55 to 0\.71, driven by improvements in both precision \(\+12 points\) and recall \(\+18 points\)\. This improvement suggests that explicitly structured reasoning cues act as a bridge between retrieval and decision\-making, helping the model not only identify concept but also interpret meaningful cues\. Moreover, the reduction in false positives and false negatives indicates enhanced discrimination between true and confusable concept candidates\. These results demonstrate that integrating human\-interpretable reasoning cues into prompts can meaningfully strengthen extraction quality and highlight a promising direction for explainable medical LLMs\.

### 4\.4Improved medical concept extraction via extraction of implicit evidence

We next investigated the role of explicit mentions of extracting implicit evidence in the prompt, by ablating the prompt for the model to consider not only explicit disease mentions but also implicit expressions of conditions \(e\.g\., BMI 37 may indicate obesity\)\. To quantify its contribution, we remove the extraction of implicit evidence from the prompt \(Section[A\.1](https://arxiv.org/html/2605.20197#A1.SS1)\)\. The results are shown in Table[2](https://arxiv.org/html/2605.20197#S4.T2)\.

Table 2:GPT\-5 Performance metrics comparison with and without extraction of implicit evidenceWhen the implicit evidence extraction is removed from the prompt, GPT\-5’s performance declines substantially across all metrics: F1 drops from 0\.55 to 0\.32, precision decreases from 0\.63 to 0\.49, and recall nearly halves from 0\.49 to 0\.24\. The model thus becomes both less accurate and less sensitive, failing to recognize many implicit positives while also producing more false alarms\. This indicates that prompting the model to explicitly account for implicit evidence not only helps it capture hidden relationships between medical concepts but also sharpens its decision boundary, improving discrimination between true and false relations\.

These results highlight that instructions encouraging inferential reasoning are essential for accurate medical concept extraction, as many diagnoses in medical notes are only implied rather than directly stated\. Without guidance to infer such relationships, models lose both precision and recall—highlighting that implicit awareness contributes simultaneously to sensitivity and selectivity in medical reasoning\.

### 4\.5Medical concept extraction and note length

Length bias, where longer inputs reduce extraction accuracy, is well documented in text extraction\. Studies such as LongBench\(Baiet al\.,[2024](https://arxiv.org/html/2605.20197#bib.bib47)\), Lost in the Middle\(Liuet al\.,[2024](https://arxiv.org/html/2605.20197#bib.bib48)\), and Long\-Context LLMs Meet RAG\(Jinet al\.,[2024](https://arxiv.org/html/2605.20197#bib.bib46)\)have shown that as the context length grows, models become distracted by irrelevant evidence, leading to a lower extraction accuracy\. In the medical domain prior work\(Jiet al\.,[2021](https://arxiv.org/html/2605.20197#bib.bib40); Liuet al\.,[2022](https://arxiv.org/html/2605.20197#bib.bib45)\)also reported that medical concept extraction is less precise for longer discharge summaries\.

However, MedicalBench does not exhibit such a note length bias\. As shown in Figure[4](https://arxiv.org/html/2605.20197#S4.F4), precision, recall, and F1 remain stable across note length bins, with no monotonic decline\. We also conducted an equivalence test for correlation using the Two One\-Sided Tests \(TOST\) procedure with bound of 0\.1\. The observed correlation wasr=0\.02r=0\.02, and the 90% confidence interval was\[−0\.03,0\.07\]\[\-0\.03,0\.07\]\. Both one\-sided tests were significant \(TOST p\-value = 0\.0109\), supporting equivalence\. Thus, the correlation between note length and accuracy is statistically negligible within the predefined bounds\. This suggests that note length explains little of the variance in performance compared with other factors such as implicit reasoning\. This finding indicates that the MedicalBench design effectively mitigates confounding effects of note length and isolates reasoning difficulty as the main challenge\.

![Refer to caption](https://arxiv.org/html/2605.20197v1/figs/figure3_performance_length2.png)Figure 4:GPT\-5 performance across different note lengths
### 4\.6Case studies

To further illustrate the reasoning challenges posed by MedicalBench, we present three representative examples highlightingimplicit medical reasoning,temporal context, andcontextual misinterpretation\. These cases demonstrate how even high\-performing models such as GPT\-5 can make plausible but incorrect decisions, underscoring the dataset’s difficulty and the need for medically grounded interpretability\.

#### 4\.6\.1Case Study 1: Implicit Inference from Medication Patterns

Medical Concept:Unspecified kidney failure Medical Note \(excerpt\):

> calcium acetate 667 mg tablet — 2 tabs orally 3 times a day \(with meals\), Status: Active valganciclovir 450 mg tablet — 1 tab orally 3 times weekly \(Mon/Wed/Fri\), Status: Active furosemide 80 mg tablet — 1 tab orally twice daily, Status: Active

Model Output:GPT\-5 predicted that the medical concept isnot relatedto the note, reasoning that kidney failure was not explicitly mentioned\.

These pharmacological cues collectively suggest significant renal dysfunction, even in the absence of an explicit diagnosis\. This case underscores the need for implicit reasoning and domain knowledge integration, requiring models to infer conditions from treatment patterns rather than literal text\.

#### 4\.6\.2Case Study 2: Misinterpretation of Temporal Context

Medical Concept:Old myocardial infarction Medical Note \(excerpt\):

> IMPRESSION: 1\) Severe regional in setting of moderate global LV systolic dysfunction consistent with Takotsubo’s cardiomyopathy; however, myocardial infarction in proximal LAD cannot be excluded, though less likely given distribution of regionality\.

Model Output:GPT\-5 predicted that the medical concept isrelatedto the note, citing the explicit mention of “myocardial infarction”\.

Expert Judgment:The concept isnot related\. The phrase “old myocardial infarction” refers specifically to aprevious \(remote, healed\)infarction rather than a current or suspected acute event\. The medical note describes an ongoing evaluation foracute chest painwith differential diagnoses of NSTEMI versus Takotsubo’s cardiomyopathy\. Importantly, the note states that myocardial infarction wasnot confirmedand was consideredless likely\. No evidence supports a prior infarction, which is required for the concept\.

This case highlights the challenge of temporal reasoning—distinguishingpastversuscurrentdisease states—and shows that even advanced models may conflate lexical overlap \(“myocardial infarction”\) with valid evidence for medical concept extractions\.

#### 4\.6\.3Case Study 3: Scattered Evidences

Medical Concept:Excision of Thoracic Vertebra, Percutaneous Approach, Diagnostic Medical Note \(excerpt\):

> Major Surgical or Invasive Procedure: T11/L1 kyphoplasty … \(1446 tokens later\) … Assessment/Plan: \[\] F/u pathology results: bone biopsy of the L1 and T11 vertebral bodies\. \[\] Home metoprolol increased to 12\.5mg BID given afib with RVR on admission\. Please ensure cardiology follow up as an outpatient\.

Model Output:GPT\-5 predicted that the concept isnot relatedto the note\.

Expert Judgment:The concept isrelated\. There are two evidences in this note\. First, “bone biopsy of the L1 and T11 vertebral bodies” documents that tissue from T11 \(a thoracic vertebra\) was obtained for diagnostic purposes, establishing an excisional biopsy at the correct anatomical site and with diagnostic intent\. Without this, there is no support for anexcision/biopsyof a thoracic vertebra\. Second, the note also records “T11/L1 kyphoplasty” in another section of the note\. This indicates apercutaneousprocedure typically performed via percutaneous transpedicular access in the same operative session\. Without this, there is no support for percutaneous approach\.

Both evidences are essential and complementary: Evidence 1 supplies diagnostic excision at the thoracic site \(T11\), and evidence 2 supplies the percutaneous approach\. If either is missing, the concept extraction fails—either the action at T11 is not proven \(no diagnostic excision\) or the approach is not supported \(no percutaneous access\)\. The long\-range separation of these evidences \(1,446 tokens apart\) illustrates a scattered evidence failure case, where models must retrieve and integrate distant, cross\-referential details to correctly extract medical concpets\.

All examples above reveal fundamental aspects that make MedicalBench difficult:

- •Ambiguous evidences, where correctness depends on nuanced interpretation rather than explicit mention\.
- •Subtle temporal distinctions, such as differentiating old versus acute disease states\.
- •Scattered evidences, where essential evidences are scattered in the medical notes\.

Such cases explain why model F1 scores remain below 0\.6: the dataset intentionally tests beyond surface\-level keyword recognition\. Future models that integrate temporal understanding, pharmacological reasoning, and evidence\-aware explanations may better capture these nuanced relationships essential for trustworthy medical concept extractions\.

## 5Conclusion

This work introduces MedicalBench, a new benchmark designed to evaluate not only the correctness but also the reasoning transparency of automated medical concept extractions systems\. Unlike existing datasets that emphasize binary concept extraction, MedicalBench incorporates implicit and confusable diagnostic cases, along with expert\-verified evidence spans, providing the first large\-scale resource for assessing why a medical concept should be extracted\.

Our experiments across a range of LLMs demonstrate that MedicalBench is a challenging benchmark \(best F1 score<<0\.6\), requiring nuanced comprehension and reasoning beyond surface\-level pattern matching\. Models that effectively retrieve evidence and articulate reasoning achieve markedly higher concept extractions accuracy, revealing a clear link between interpretability and performance\. Incorporating explicit reasoning cues or implicit evidence extraction into prompts further enhances outcomes, underscoring the need for models that can reason medically rather than merely classify text\. Finally, MedicalBench seeks to transition the field from extraction\-as\-classification toward extraction\-as\-reasoning, a shift essential for deploying large language models responsibly in real\-world medical settings\.

## 6Acknowledgment

This project is built on de\-identified clinical data from the MIMIC\-IV database, which has been publicly released for research under a data use agreement\. All patient health information in the discharge summaries was de\-identified in accordance with HIPAA safe harbor standards by the original MIMIC\-IV data curators\. The annotations we added \(evidence and explanations\) contain no patient\-identifying information, only medical insights\. Institutional Review Board \(IRB\) approval was not required for this work, as it does not involve identifiable human subjects\. We thank Dr\. Deqi Kong for clinical input, Sepehr Janghorbani for feedback on evaluation and Elizabeth Erickson and Kalyan Guntupalli for their administrative support\. We are grateful to the MIT Laboratory for Computational Physiology and PhysioNet for providing access to the MIMIC\-IV databases used in this study\.

## References

- M\. Afshar, F\. Resnik, M\. R\. Baumann, J\. Hintzke, K\. Lemmon, A\. G\. Sullivan, T\. Shah, A\. Stordalen, M\. Oberst, J\. Dambach, L\. A\. Mrotek, M\. Quinn, K\. Abramson, P\. Kleinschmidt, T\. Brazelton, H\. Twedt, D\. Kunstman, G\. Wills, J\. Long, B\. W\. Patterson, F\. J\. Liao, S\. Rasmussen, E\. Burnside, C\. Goswami, and J\. E\. Gordon \(2025\)A novel playbook for pragmatic trial operations to monitor and evaluate ambient artificial intelligence in clinical practice\.NEJM AI2\(9\),pp\. AIdbp2401267\.External Links:[Document](https://dx.doi.org/10.1056/AIdbp2401267),[Link](https://ai.nejm.org/doi/full/10.1056/AIdbp2401267),https://ai\.nejm\.org/doi/pdf/10\.1056/AIdbp2401267Cited by:[§1](https://arxiv.org/html/2605.20197#S1.p1.1)\.
- Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou, Y\. Dong, J\. Tang, and J\. Li \(2024\)LongBench: a bilingual, multitask benchmark for long context understanding\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 3119–3137\.External Links:[Link](https://aclanthology.org/2024.acl-long.172/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.172)Cited by:[§4\.5](https://arxiv.org/html/2605.20197#S4.SS5.p1.1)\.
- K\. Beckh, E\. Studeny, S\. S\. Gannamaneni, D\. Antweiler, and S\. Rueping \(2025\)The anatomy of evidence: an investigation into explainable ICD coding\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 16840–16851\.External Links:[Link](https://aclanthology.org/2025.findings-acl.864/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.864),ISBN 979\-8\-89176\-256\-5Cited by:[§1](https://arxiv.org/html/2605.20197#S1.p3.1)\.
- H\. Cheng, R\. Jafari, A\. Russell, R\. Klopfer, E\. Lu, B\. Striner, and M\. Gormley \(2023\)MDACE: MIMIC documents annotated with code evidence\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 7534–7550\.External Links:[Link](https://aclanthology.org/2023.acl-long.416/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.416)Cited by:[§1](https://arxiv.org/html/2605.20197#S1.p3.1)\.
- A\. Das, I\. A\. Talati, J\. M\. Z\. Chaves, D\. Rubin, and I\. Banerjee \(2025\)Weakly supervised language models for automated extraction of critical findings from radiology reports\.npj Digital Medicine8\(1\),pp\. 257\.Cited by:[§1](https://arxiv.org/html/2605.20197#S1.p1.1)\.
- J\. DeYoung, H\. Shing, L\. Kong, C\. Winestock, and C\. Shivade \(2022\)Entity anchored icd coding\.arXiv preprint arXiv:2208\.07444\.Cited by:[§1](https://arxiv.org/html/2605.20197#S1.p3.1)\.
- Y\. Gao, Y\. Chen, M\. Wang, J\. Wu, Y\. Kim, K\. Zhou, M\. Li, X\. Liu, X\. Fu, J\. Wu,et al\.\(2024\)Optimising the paradigms of human ai collaborative clinical coding\.npj Digital Medicine7\(1\),pp\. 368\.Cited by:[§1](https://arxiv.org/html/2605.20197#S1.p1.1)\.
- B\. A\. Goldstein, A\. M\. Navar, M\. J\. Pencina, and J\. P\. Ioannidis \(2016\)Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review\.Journal of the American Medical Informatics Association: JAMIA24\(1\),pp\. 198\.Cited by:[§1](https://arxiv.org/html/2605.20197#S1.p1.1)\.
- D\. Hein, A\. Christie, M\. Holcomb, B\. Xie, A\. Jain, J\. Vento, N\. Rakheja, A\. H\. Shakur, S\. Christley, L\. G\. Cowell,et al\.\(2025\)Iterative refinement and goal articulation to optimize large language models for clinical information extraction\.npj Digital Medicine8\(1\),pp\. 301\.Cited by:[§1](https://arxiv.org/html/2605.20197#S1.p1.1)\.
- J\. Huang, D\. M\. Yang, R\. Rong, K\. Nezafati, C\. Treager, Z\. Chi, S\. Wang, X\. Cheng, Y\. Guo, L\. J\. Klesse, G\. Xiao, E\. D\. Peterson, X\. Zhan, and Y\. Xie \(2024\)A critical assessment of using chatgpt for extracting structured data from clinical notes\.NPJ Digital Medicine7\.Cited by:[§1](https://arxiv.org/html/2605.20197#S1.p1.1)\.
- S\. Ji, M\. Holtta, and P\. Marttinen \(2021\)Does the magic of bert apply to medical code assignment? a quantitative study\.Computers in biology and medicine139,pp\. 104998\.External Links:[Link](https://api.semanticscholar.org/CorpusID:232185406)Cited by:[§4\.5](https://arxiv.org/html/2605.20197#S4.SS5.p1.1)\.
- B\. Jin, J\. Yoon, J\. Han, and S\. Ö\. Arik \(2024\)Long\-context llms meet rag: overcoming challenges for long inputs in rag\.ArXivabs/2410\.05983\.External Links:[Link](https://api.semanticscholar.org/CorpusID:273229050)Cited by:[§4\.5](https://arxiv.org/html/2605.20197#S4.SS5.p1.1)\.
- A\. Johnson, L\. Bulgarelli, T\. Pollard, S\. Horng, L\. A\. Celi, and R\. Mark \(2024\)MIMIC\-iv \(version 3\.1\)\.PhysioNet,pp\. 49–55\.External Links:[Document](https://dx.doi.org/10.1038/s41597-022-01899-x)Cited by:[§2\.1](https://arxiv.org/html/2605.20197#S2.SS1.p1.1)\.
- A\. E\. Johnson, L\. Bulgarelli, L\. Shen, A\. Gayles, A\. Shammout, S\. Horng, T\. J\. Pollard, S\. Hao, B\. Moody, B\. Gow,et al\.\(2023a\)MIMIC\-iv, a freely accessible electronic health record dataset\.Scientific data10\(1\),pp\. 1\.Cited by:[§2\.1](https://arxiv.org/html/2605.20197#S2.SS1.p1.1)\.
- A\. Johnson, T\. Pollard, S\. Horng, L\. A\. Celi, and R\. Mark \(2023b\)MIMIC\-iv\-note: deidentified free\-text clinical notes \(version 2\.2\)\.PhysioNet\.External Links:[Document](https://dx.doi.org/10.13026/1n74-ne17)Cited by:[§2\.1](https://arxiv.org/html/2605.20197#S2.SS1.p1.1)\.
- M\. Y\. Landolsi, L\. Hlaoua, and L\. B\. Romdhane \(2022\)Information extraction from electronic medical documents: state of the art and future research directions\.Knowledge and Information Systems65,pp\. 463 – 516\.External Links:[Link](https://api.semanticscholar.org/CorpusID:253418128)Cited by:[§1](https://arxiv.org/html/2605.20197#S1.p1.1)\.
- W\. Lau, K\. Lybarger, M\. L\. Gunn, and M\. Yetisgen \(2021\)Event\-based clinical finding extraction from radiology reports with pre\-trained language model\.Journal of Digital Imaging36,pp\. 91 – 104\.External Links:[Link](https://api.semanticscholar.org/CorpusID:245501918)Cited by:[§1](https://arxiv.org/html/2605.20197#S1.p1.1)\.
- L\. Liu, O\. Perez\-Concha, A\. N\. Nguyen, V\. Bennett, and L\. R\. Jorm \(2022\)Automated icd coding using extreme multi\-label long text transformer\-based models\.Artificial intelligence in medicine144,pp\. 102662\.External Links:[Link](https://api.semanticscholar.org/CorpusID:254564396)Cited by:[§4\.5](https://arxiv.org/html/2605.20197#S4.SS5.p1.1)\.
- N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. Liang \(2024\)Lost in the middle: how language models use long contexts\.Transactions of the Association for Computational Linguistics12,pp\. 157–173\.External Links:[Link](https://aclanthology.org/2024.tacl-1.9/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00638)Cited by:[§4\.5](https://arxiv.org/html/2605.20197#S4.SS5.p1.1)\.
- A\. Mattar, D\. Carlston, G\. Sariol, T\. Yu, A\. Almustafa, G\. B\. Melton, and A\. Ahmed \(2017\)The prevalence of obesity documentation in primary care electronic medical records\.Applied Clinical Informatics08,pp\. 67 – 79\.External Links:[Link](https://api.semanticscholar.org/CorpusID:10598808)Cited by:[§1](https://arxiv.org/html/2605.20197#S1.p2.1)\.
- S\. M\. Meystre, G\. K\. Savova, K\. C\. Kipper\-Schuler, and J\. F\. Hurdle \(2008\)Extracting information from textual documents in the electronic health record: a review of recent research\.Yearbook of medical informatics17\(01\),pp\. 128–144\.Cited by:[§1](https://arxiv.org/html/2605.20197#S1.p2.1)\.
- J\. Mullenbach, S\. Wiegreffe, J\. Duke, J\. Sun, and J\. Eisenstein \(2018\)Explainable prediction of medical codes from clinical text\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long Papers\),M\. Walker, H\. Ji, and A\. Stent \(Eds\.\),New Orleans, Louisiana,pp\. 1101–1111\.External Links:[Link](https://aclanthology.org/N18-1100/),[Document](https://dx.doi.org/10.18653/v1/N18-1100)Cited by:[§1](https://arxiv.org/html/2605.20197#S1.p3.1),[§2\.1](https://arxiv.org/html/2605.20197#S2.SS1.p1.1)\.
- S\. Perera, P\. Mendes, A\. Sheth, K\. Thirunarayan, A\. Alex, C\. Heid, and G\. Mott \(2015\)Implicit entity recognition in clinical documents\.InProceedings of the Fourth Joint Conference on Lexical and Computational Semantics,M\. Palmer, G\. Boleda, and P\. Rosso \(Eds\.\),Denver, Colorado,pp\. 228–238\.External Links:[Link](https://aclanthology.org/S15-1028/),[Document](https://dx.doi.org/10.18653/v1/S15-1028)Cited by:[§1](https://arxiv.org/html/2605.20197#S1.p2.1)\.
- P\. PhysioBank \(2000\)Physionet: components of a new research resource for complex physiologic signals\.Circulation101\(23\),pp\. e215–e220\.Cited by:[§2\.1](https://arxiv.org/html/2605.20197#S2.SS1.p1.1)\.
- R\. F\. R\. Sarmiento and F\. Dernoncourt \(2016\)Improving patient cohort identification using natural language processing\.External Links:[Link](https://api.semanticscholar.org/CorpusID:63420021)Cited by:[§1](https://arxiv.org/html/2605.20197#S1.p1.1)\.
- T\. M\. Seinen, E\. A\. Fridgeirsson, S\. Ioannou, D\. Jeannetot, L\. H\. John, J\. A\. Kors, A\. F\. Markus, V\. Pera, A\. Rekkas, R\. D\. Williams,et al\.\(2022\)Use of unstructured text in prognostic clinical prediction models: a systematic review\.Journal of the American Medical Informatics Association29\(7\),pp\. 1292–1302\.Cited by:[§1](https://arxiv.org/html/2605.20197#S1.p1.1)\.
- F\. Shahid, M\. Hsu, Y\. Chang, and W\. Jian \(2025\)Using generative ai to extract structured information from free text pathology reports\.Journal of Medical Systems49\(1\),pp\. 36\.Cited by:[§1](https://arxiv.org/html/2605.20197#S1.p1.1)\.
- S\. N\. Shukla and B\. M\. Marlin \(2020\)Integrating physiological time series and clinical notes with deep learning for improved icu mortality prediction\.arXiv preprint arXiv:2003\.11059\.Cited by:[§1](https://arxiv.org/html/2605.20197#S1.p1.1)\.
- D\. Sun, L\. Hadjiiski, G\. Bruno, J\. Gormley, H\. Chan, E\. M\. Caoili, R\. H\. Cohan, A\. Alva, R\. Mihalcea, C\. Zhou,et al\.\(2025\)Evaluating the reliability of large language models for clinical data extraction in bladder cancer prognosis\.Scientific Reports\.Cited by:[§1](https://arxiv.org/html/2605.20197#S1.p1.1)\.
- M\. Sushil, V\. E\. Kennedy, D\. Mandair, B\. Y\. Miao, T\. Zack, and A\. J\. Butte \(2024\)CORAL: expert\-curated oncology reports to advance language model inference\.Nejm Ai1\(4\),pp\. AIdbp2300110\.Cited by:[§1](https://arxiv.org/html/2605.20197#S1.p1.1)\.
- C\. H\. Tsai, A\. Eghdam, N\. Davoody, G\. Wright, S\. Flowerday, and S\. Koch \(2020\)Effects of electronic health record implementation and barriers to adoption and use: a scoping review and qualitative analysis of the content\.Life10\(12\),pp\. 327\.Cited by:[§1](https://arxiv.org/html/2605.20197#S1.p1.1)\.
- I\. C\. Wiest, F\. Wolf, M\. Leßmann, M\. van Treeck, D\. Ferber, J\. Zhu, H\. Boehme, K\. K\. Bressem, H\. Ulrich, M\. P\. Ebert,et al\.\(2025\)A software pipeline for medical information extraction with large language models, open source and suitable for oncology\.npj Precision Oncology9\(1\),pp\. 313\.Cited by:[§1](https://arxiv.org/html/2605.20197#S1.p1.1)\.
- J\. Wu, X\. Liu, M\. Li, W\. Li, Z\. Su, S\. Lin, L\. Garay, Z\. Zhang, Y\. Zhang, Q\. Zeng,et al\.\(2024\)Clinical text datasets for medical artificial intelligence and large language models—a systematic review\.NEJM AI1\(6\),pp\. AIra2400012\.Cited by:[§1](https://arxiv.org/html/2605.20197#S1.p1.1)\.
- C\. Xiao, E\. Choi, and J\. Sun \(2018\)Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review\.Journal of the American Medical Informatics Association25\(10\),pp\. 1419–1428\.External Links:ISSN 1527\-974X,[Document](https://dx.doi.org/10.1093/jamia/ocy068),[Link](https://doi.org/10.1093/jamia/ocy068),https://academic\.oup\.com/jamia/article\-pdf/25/10/1419/34150605/ocy068\.pdfCited by:[§1](https://arxiv.org/html/2605.20197#S1.p1.1)\.

## Appendix AAppendix

### A\.1Prompt template

Below is the template to extract medical concept and locate evidence\. Unless otherwise specified, it is the default prompt for all LLMs in our experiments\.

Prompt to extract concept and locate evidenceYou are an expert in annotating clinical notes\. Your task is to identify the type of mention for a given medical concept within a provided clinical note\. There are three possible mention types:\- Explicit: entity that is found directly in the document\. For example, the discharge diagnosis includes ’Acute Kidney Injury’, which corresponds to the acute kidney failure with tubular necrosis entity\.\- Implicit: entity which is not directly mentioned in the document but can be inferred using context\. For example, The mention of low hemoglobin level suggests anemia\.\- Unrelated: entity which is not related to the clinical note\.Output format: A json dictionary, where the json is of the following keys‘extracted mention‘: List\[str\], extracted mentions for medical concept from clinical note\.‘mention type‘: str, either ”Explicit”, ”Implicit”, or ”Unrelated”\.Medical concept: \{label\}Clinical note: \{text\}

Below is the template for the ablation study where we remove the categories from the prompt \(Section[4\.4](https://arxiv.org/html/2605.20197#S4.SS4)\)\.

Prompt to extract concept and locate evidence without categoriesYou are an expert in annotating clinical notes\. Your task is to identify the medical concept within a provided clinical note\.Output format: A json dictionary, where the json is of the following keys\.‘extracted mention‘: List\[str\], extracted mentions for medical concept from clinical note\. If no evidence, then return empty\.‘is related‘: bool, If medical concept is related to the clinical note, return true, otherwise return false\.Medical concept: \{label\}Clinical note: \{text\}